lael-labs/linux

Documentation/ABI/testing/sysfs-block-zram+14−0

@@ -150,3 +150,17 @@ Contact: Sergey Senozhatsky <senozhatsky@chromium.org> · 149 unmodified lines
150	Description:	150	Description:
151	The algorithm_params file is write-only and is used to setup	151	The algorithm_params file is write-only and is used to setup
152	compression algorithm parameters.	152	compression algorithm parameters.
		153
		154	What: /sys/block/zram<id>/writeback_compressed
		155	Date: Decemeber 2025
		156	Contact: Richard Chang <richardycc@google.com>
		157	Description:
		158	The writeback_compressed device atrribute toggles compressed
		159	writeback feature.
		160
		161	What: /sys/block/zram<id>/writeback_batch_size
		162	Date: November 2025
		163	Contact: Sergey Senozhatsky <senozhatsky@chromium.org>
		164	Description:
		165	The writeback_batch_size device atrribute sets the maximum
		166	number of in-flight writeback operations.

Documentation/ABI/testing/sysfs-kernel-mm-damon+13−0

@@ -516,6 +516,19 @@ Contact: SeongJae Park <sj@kernel.org> · 515 unmodified lines
516	Description: Reading this file returns the number of the exceed events of	516	Description: Reading this file returns the number of the exceed events of
517	the scheme's quotas.	517	the scheme's quotas.
518		518
		519	What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/nr_snapshots
		520	Date: Dec 2025
		521	Contact: SeongJae Park <sj@kernel.org>
		522	Description: Reading this file returns the total number of DAMON snapshots
		523	that the scheme has tried to be applied.
		524
		525	What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/max_nr_snapshots
		526	Date: Dec 2025
		527	Contact: SeongJae Park <sj@kernel.org>
		528	Description: Writing a number to this file sets the upper limit of
		529	nr_snapshots that deactivates the scheme when the limit is
		530	reached or exceeded.
		531
519	What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/total_bytes	532	What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/total_bytes
520	Date: Jul 2023	533	Date: Jul 2023
521	Contact: SeongJae Park <sj@kernel.org>	534	Contact: SeongJae Park <sj@kernel.org>

Documentation/admin-guide/blockdev/zram.rst+23−1

@@ -214,6 +214,9 @@ mem_limit WO specifies the maximum amount of memory ZRAM can · 213 unmodified lines
214	writeback_limit WO specifies the maximum amount of write IO zram	214	writeback_limit WO specifies the maximum amount of write IO zram
215	can write out to backing device as 4KB unit	215	can write out to backing device as 4KB unit
216	writeback_limit_enable RW show and set writeback_limit feature	216	writeback_limit_enable RW show and set writeback_limit feature
		217	writeback_batch_size RW show and set maximum number of in-flight
		218	writeback operations
		219	writeback_compressed RW show and set compressed writeback feature
217	comp_algorithm RW show and change the compression algorithm	220	comp_algorithm RW show and change the compression algorithm
218	algorithm_params WO setup compression algorithm parameters	221	algorithm_params WO setup compression algorithm parameters
219	compact WO trigger memory compaction	222	compact WO trigger memory compaction
@@ -222,7 +225,6 @@ backing_dev RW set up backend storage for zram to write out · 2 unmodified lines
222	idle WO mark allocated slot as idle	225	idle WO mark allocated slot as idle
223	====================== ====== ===============================================	226	====================== ====== ===============================================
224		227
225
226	User space is advised to use the following files to read the device statistics.	228	User space is advised to use the following files to read the device statistics.
227		229
228	File /sys/block/zram<id>/stat	230	File /sys/block/zram<id>/stat
@@ -434,6 +436,26 @@ system reboot, echo 1 > /sys/block/zramX/reset) so keeping how many of · 205 unmodified lines
434	writeback happened until you reset the zram to allocate extra writeback	436	writeback happened until you reset the zram to allocate extra writeback
435	budget in next setting is user's job.	437	budget in next setting is user's job.
436		438
		439	By default zram stores written back pages in decompressed (raw) form, which
		440	means that writeback operation involves decompression of the page before
		441	writing it to the backing device. This behavior can be changed by enabling
		442	`writeback_compressed` feature, which causes zram to write compressed pages
		443	to the backing device, thus avoiding decompression overhead. To enable
		444	this feature, execute::
		445
		446	$ echo yes > /sys/block/zramX/writeback_compressed
		447
		448	Note that this feature should be configured before the `zramX` device is
		449	initialized.
		450
		451	Depending on backing device storage type, writeback operation may benefit
		452	from a higher number of in-flight write requests (batched writes). The
		453	number of maximum in-flight writeback operations can be configured via
		454	`writeback_batch_size` attribute. To change the default value (which is 32),
		455	execute::
		456
		457	$ echo 64 > /sys/block/zramX/writeback_batch_size
		458
437	If admin wants to measure writeback count in a certain period, they could	459	If admin wants to measure writeback count in a certain period, they could
438	know it via /sys/block/zram0/bd_stat's 3rd column.	460	know it via /sys/block/zram0/bd_stat's 3rd column.
439		461

Documentation/admin-guide/cgroup-v1/memory.rst+2−3

@@ -311,9 +311,8 @@ Lock order is as follows:: · 310 unmodified lines
311		311
312	folio_lock	312	folio_lock
313	mm->page_table_lock or split pte_lock	313	mm->page_table_lock or split pte_lock
314	folio_memcg_lock (memcg->move_lock)	314	mapping->i_pages lock
315	mapping->i_pages lock	315	lruvec->lru_lock.
316	lruvec->lru_lock.
317		316
318	Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by	317	Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by
319	lruvec->lru_lock; the folio LRU flag is cleared before	318	lruvec->lru_lock; the folio LRU flag is cleared before

Documentation/admin-guide/laptops/index.rst+0−1

@@ -10,7 +10,6 @@ Laptop Drivers · 9 unmodified lines
10	alienware-wmi	10	alienware-wmi
11	asus-laptop	11	asus-laptop
12	disk-shock-protection	12	disk-shock-protection
13	laptop-mode
14	lg-laptop	13	lg-laptop
15	samsung-galaxybook	14	samsung-galaxybook
16	sony-laptop	15	sony-laptop

Documentation/admin-guide/laptops/laptop-mode.rst+0−770

@@ -1,770 +0,0 @@
1	===============================================
2	How to conserve battery power using laptop-mode
3	===============================================
4
5	Document Author: Bart Samwel (bart@samwel.tk)
6
7	Date created: January 2, 2004
8
9	Last modified: December 06, 2004
10
11	Introduction
12	------------
13
14	Laptop mode is used to minimize the time that the hard disk needs to be spun up,
15	to conserve battery power on laptops. It has been reported to cause significant
16	power savings.
17
18	.. Contents
19
20	* Introduction
21	* Installation
22	* Caveats
23	* The Details
24	* Tips & Tricks
25	* Control script
26	* ACPI integration
27	* Monitoring tool
28
29
30	Installation
31	------------
32
33	To use laptop mode, you don't need to set any kernel configuration options
34	or anything. Simply install all the files included in this document, and
35	laptop mode will automatically be started when you're on battery. For
36	your convenience, a tarball containing an installer can be downloaded at:
37
38	http://www.samwel.tk/laptop_mode/laptop_mode/
39
40	To configure laptop mode, you need to edit the configuration file, which is
41	located in /etc/default/laptop-mode on Debian-based systems, or in
42	/etc/sysconfig/laptop-mode on other systems.
43
44	Unfortunately, automatic enabling of laptop mode does not work for
45	laptops that don't have ACPI. On those laptops, you need to start laptop
46	mode manually. To start laptop mode, run "laptop_mode start", and to
47	stop it, run "laptop_mode stop". (Note: The laptop mode tools package now
48	has experimental support for APM, you might want to try that first.)
49
50
51	Caveats
52	-------
53
54	* The downside of laptop mode is that you have a chance of losing up to 10
55	minutes of work. If you cannot afford this, don't use it! The supplied ACPI
56	scripts automatically turn off laptop mode when the battery almost runs out,
57	so that you won't lose any data at the end of your battery life.
58
59	* Most desktop hard drives have a very limited lifetime measured in spindown
60	cycles, typically about 50.000 times (it's usually listed on the spec sheet).
61	Check your drive's rating, and don't wear down your drive's lifetime if you
62	don't need to.
63
64	* If you mount some of your ext3 filesystems with the -n option, then
65	the control script will not be able to remount them correctly. You must set
66	DO_REMOUNTS=0 in the control script, otherwise it will remount them with the
67	wrong options -- or it will fail because it cannot write to /etc/mtab.
68
69	* If you have your filesystems listed as type "auto" in fstab, like I did, then
70	the control script will not recognize them as filesystems that need remounting.
71	You must list the filesystems with their true type instead.
72
73	* It has been reported that some versions of the mutt mail client use file access
74	times to determine whether a folder contains new mail. If you use mutt and
75	experience this, you must disable the noatime remounting by setting the option
76	DO_REMOUNT_NOATIME to 0 in the configuration file.
77
78
79	The Details
80	-----------
81
82	Laptop mode is controlled by the knob /proc/sys/vm/laptop_mode. This knob is
83	present for all kernels that have the laptop mode patch, regardless of any
84	configuration options. When the knob is set, any physical disk I/O (that might
85	have caused the hard disk to spin up) causes Linux to flush all dirty blocks. The
86	result of this is that after a disk has spun down, it will not be spun up
87	anymore to write dirty blocks, because those blocks had already been written
88	immediately after the most recent read operation. The value of the laptop_mode
89	knob determines the time between the occurrence of disk I/O and when the flush
90	is triggered. A sensible value for the knob is 5 seconds. Setting the knob to
91	0 disables laptop mode.
92
93	To increase the effectiveness of the laptop_mode strategy, the laptop_mode
94	control script increases dirty_expire_centisecs and dirty_writeback_centisecs in
95	/proc/sys/vm to about 10 minutes (by default), which means that pages that are
96	dirtied are not forced to be written to disk as often. The control script also
97	changes the dirty background ratio, so that background writeback of dirty pages
98	is not done anymore. Combined with a higher commit value (also 10 minutes) for
99	ext3 filesystem (also done automatically by the control script),
100	this results in concentration of disk activity in a small time interval which
101	occurs only once every 10 minutes, or whenever the disk is forced to spin up by
102	a cache miss. The disk can then be spun down in the periods of inactivity.
103
104
105	Configuration
106	-------------
107
108	The laptop mode configuration file is located in /etc/default/laptop-mode on
109	Debian-based systems, or in /etc/sysconfig/laptop-mode on other systems. It
110	contains the following options:
111
112	MAX_AGE:
113
114	Maximum time, in seconds, of hard drive spindown time that you are
115	comfortable with. Worst case, it's possible that you could lose this
116	amount of work if your battery fails while you're in laptop mode.
117
118	MINIMUM_BATTERY_MINUTES:
119
120	Automatically disable laptop mode if the remaining number of minutes of
121	battery power is less than this value. Default is 10 minutes.
122
123	AC_HD/BATT_HD:
124
125	The idle timeout that should be set on your hard drive when laptop mode
126	is active (BATT_HD) and when it is not active (AC_HD). The defaults are
127	20 seconds (value 4) for BATT_HD and 2 hours (value 244) for AC_HD. The
128	possible values are those listed in the manual page for "hdparm" for the
129	"-S" option.
130
131	HD:
132
133	The devices for which the spindown timeout should be adjusted by laptop mode.
134	Default is /dev/hda. If you specify multiple devices, separate them by a space.
135
136	READAHEAD:
137
138	Disk readahead, in 512-byte sectors, while laptop mode is active. A large
139	readahead can prevent disk accesses for things like executable pages (which are
140	loaded on demand while the application executes) and sequentially accessed data
141	(MP3s).
142
143	DO_REMOUNTS:
144
145	The control script automatically remounts any mounted journaled filesystems
146	with appropriate commit interval options. When this option is set to 0, this
147	feature is disabled.
148
149	DO_REMOUNT_NOATIME:
150
151	When remounting, should the filesystems be remounted with the noatime option?
152	Normally, this is set to "1" (enabled), but there may be programs that require
153	access time recording.
154
155	DIRTY_RATIO:
156
157	The percentage of memory that is allowed to contain "dirty" or unsaved data
158	before a writeback is forced, while laptop mode is active. Corresponds to
159	the /proc/sys/vm/dirty_ratio sysctl.
160
161	DIRTY_BACKGROUND_RATIO:
162
163	The percentage of memory that is allowed to contain "dirty" or unsaved data
164	after a forced writeback is done due to an exceeding of DIRTY_RATIO. Set
165	this nice and low. This corresponds to the /proc/sys/vm/dirty_background_ratio
166	sysctl.
167
168	Note that the behaviour of dirty_background_ratio is quite different
169	when laptop mode is active and when it isn't. When laptop mode is inactive,
170	dirty_background_ratio is the threshold percentage at which background writeouts
171	start taking place. When laptop mode is active, however, background writeouts
172	are disabled, and the dirty_background_ratio only determines how much writeback
173	is done when dirty_ratio is reached.
174
175	DO_CPU:
176
177	Enable CPU frequency scaling when in laptop mode. (Requires CPUFreq to be setup.
178	See Documentation/admin-guide/pm/cpufreq.rst for more info. Disabled by default.)
179
180	CPU_MAXFREQ:
181
182	When on battery, what is the maximum CPU speed that the system should use? Legal
183	values are "slowest" for the slowest speed that your CPU is able to operate at,
184	or a value listed in /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies.
185
186
187	Tips & Tricks
188	-------------
189
190	* Bartek Kania reports getting up to 50 minutes of extra battery life (on top
191	of his regular 3 to 3.5 hours) using a spindown time of 5 seconds (BATT_HD=1).
192
193	* You can spin down the disk while playing MP3, by setting disk readahead
194	to 8MB (READAHEAD=16384). Effectively, the disk will read a complete MP3 at
195	once, and will then spin down while the MP3 is playing. (Thanks to Bartek
196	Kania.)
197
198	* Drew Scott Daniels observed: "I don't know why, but when I decrease the number
199	of colours that my display uses it consumes less battery power. I've seen
200	this on powerbooks too. I hope that this is a piece of information that
201	might be useful to the Laptop Mode patch or its users."
202
203	* In syslog.conf, you can prefix entries with a dash `-` to omit syncing the
204	file after every logging. When you're using laptop-mode and your disk doesn't
205	spin down, this is a likely culprit.
206
207	* Richard Atterer observed that laptop mode does not work well with noflushd
208	(http://noflushd.sourceforge.net/), it seems that noflushd prevents laptop-mode
209	from doing its thing.
210
211	* If you're worried about your data, you might want to consider using a USB
212	memory stick or something like that as a "working area". (Be aware though
213	that flash memory can only handle a limited number of writes, and overuse
214	may wear out your memory stick pretty quickly. Do _not_ use journalling
215	filesystems on flash memory sticks.)
216
217
218	Configuration file for control and ACPI battery scripts
219	-------------------------------------------------------
220
221	This allows the tunables to be changed for the scripts via an external
222	configuration file
223
224	It should be installed as /etc/default/laptop-mode on Debian, and as
225	/etc/sysconfig/laptop-mode on Red Hat, SUSE, Mandrake, and other work-alikes.
226
227	Config file::
228
229	# Maximum time, in seconds, of hard drive spindown time that you are
230	# comfortable with. Worst case, it's possible that you could lose this
231	# amount of work if your battery fails you while in laptop mode.
232	#MAX_AGE=600
233
234	# Automatically disable laptop mode when the number of minutes of battery
235	# that you have left goes below this threshold.
236	MINIMUM_BATTERY_MINUTES=10
237
238	# Read-ahead, in 512-byte sectors. You can spin down the disk while playing MP3/OGG
239	# by setting the disk readahead to 8MB (READAHEAD=16384). Effectively, the disk
240	# will read a complete MP3 at once, and will then spin down while the MP3/OGG is
241	# playing.
242	#READAHEAD=4096
243
244	# Shall we remount journaled fs. with appropriate commit interval? (1=yes)
245	#DO_REMOUNTS=1
246
247	# And shall we add the "noatime" option to that as well? (1=yes)
248	#DO_REMOUNT_NOATIME=1
249
250	# Dirty synchronous ratio. At this percentage of dirty pages the process
251	# which
252	# calls write() does its own writeback
253	#DIRTY_RATIO=40
254
255	#
256	# Allowed dirty background ratio, in percent. Once DIRTY_RATIO has been
257	# exceeded, the kernel will wake flusher threads which will then reduce the
258	# amount of dirty memory to dirty_background_ratio. Set this nice and low,
259	# so once some writeout has commenced, we do a lot of it.
260	#
261	#DIRTY_BACKGROUND_RATIO=5
262
263	# kernel default dirty buffer age
264	#DEF_AGE=30
265	#DEF_UPDATE=5
266	#DEF_DIRTY_BACKGROUND_RATIO=10
267	#DEF_DIRTY_RATIO=40
268	#DEF_XFS_AGE_BUFFER=15
269	#DEF_XFS_SYNC_INTERVAL=30
270	#DEF_XFS_BUFD_INTERVAL=1
271
272	# This must be adjusted manually to the value of HZ in the running kernel
273	# on 2.4, until the XFS people change their 2.4 external interfaces to work in
274	# centisecs. This can be automated, but it's a work in progress that still
275	# needs# some fixes. On 2.6 kernels, XFS uses USER_HZ instead of HZ for
276	# external interfaces, and that is currently always set to 100. So you don't
277	# need to change this on 2.6.
278	#XFS_HZ=100
279
280	# Should the maximum CPU frequency be adjusted down while on battery?
281	# Requires CPUFreq to be setup.
282	# See Documentation/admin-guide/pm/cpufreq.rst for more info
283	#DO_CPU=0
284
285	# When on battery what is the maximum CPU speed that the system should
286	# use? Legal values are "slowest" for the slowest speed that your
287	# CPU is able to operate at, or a value listed in:
288	# /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
289	# Only applicable if DO_CPU=1.
290	#CPU_MAXFREQ=slowest
291
292	# Idle timeout for your hard drive (man hdparm for valid values, -S option)
293	# Default is 2 hours on AC (AC_HD=244) and 20 seconds for battery (BATT_HD=4).
294	#AC_HD=244
295	#BATT_HD=4
296
297	# The drives for which to adjust the idle timeout. Separate them by a space,
298	# e.g. HD="/dev/hda /dev/hdb".
299	#HD="/dev/hda"
300
301	# Set the spindown timeout on a hard drive?
302	#DO_HD=1
303
304
305	Control script
306	--------------
307
308	Please note that this control script works for the Linux 2.4 and 2.6 series (thanks
309	to Kiko Piris).
310
311	Control script::
312
313	#!/bin/bash
314
315	# start or stop laptop_mode, best run by a power management daemon when
316	# ac gets connected/disconnected from a laptop
317	#
318	# install as /sbin/laptop_mode
319	#
320	# Contributors to this script: Kiko Piris
321	# Bart Samwel
322	# Micha Feigin
323	# Andrew Morton
324	# Herve Eychenne
325	# Dax Kelson
326	#
327	# Original Linux 2.4 version by: Jens Axboe
328
329	#############################################################################
330
331	# Source config
332	if [ -f /etc/default/laptop-mode ] ; then
333	# Debian
334	. /etc/default/laptop-mode
335	elif [ -f /etc/sysconfig/laptop-mode ] ; then
336	# Others
337	. /etc/sysconfig/laptop-mode
338	fi
339
340	# Don't raise an error if the config file is incomplete
341	# set defaults instead:
342
343	# Maximum time, in seconds, of hard drive spindown time that you are
344	# comfortable with. Worst case, it's possible that you could lose this
345	# amount of work if your battery fails you while in laptop mode.
346	MAX_AGE=${MAX_AGE:-'600'}
347
348	# Read-ahead, in kilobytes
349	READAHEAD=${READAHEAD:-'4096'}
350
351	# Shall we remount journaled fs. with appropriate commit interval? (1=yes)
352	DO_REMOUNTS=${DO_REMOUNTS:-'1'}
353
354	# And shall we add the "noatime" option to that as well? (1=yes)
355	DO_REMOUNT_NOATIME=${DO_REMOUNT_NOATIME:-'1'}
356
357	# Shall we adjust the idle timeout on a hard drive?
358	DO_HD=${DO_HD:-'1'}
359
360	# Adjust idle timeout on which hard drive?
361	HD="${HD:-'/dev/hda'}"
362
363	# spindown time for HD (hdparm -S values)
364	AC_HD=${AC_HD:-'244'}
365	BATT_HD=${BATT_HD:-'4'}
366
367	# Dirty synchronous ratio. At this percentage of dirty pages the process which
368	# calls write() does its own writeback
369	DIRTY_RATIO=${DIRTY_RATIO:-'40'}
370
371	# cpu frequency scaling
372	# See Documentation/admin-guide/pm/cpufreq.rst for more info
373	DO_CPU=${CPU_MANAGE:-'0'}
374	CPU_MAXFREQ=${CPU_MAXFREQ:-'slowest'}
375
376	#
377	# Allowed dirty background ratio, in percent. Once DIRTY_RATIO has been
378	# exceeded, the kernel will wake flusher threads which will then reduce the
379	# amount of dirty memory to dirty_background_ratio. Set this nice and low,
380	# so once some writeout has commenced, we do a lot of it.
381	#
382	DIRTY_BACKGROUND_RATIO=${DIRTY_BACKGROUND_RATIO:-'5'}
383
384	# kernel default dirty buffer age
385	DEF_AGE=${DEF_AGE:-'30'}
386	DEF_UPDATE=${DEF_UPDATE:-'5'}
387	DEF_DIRTY_BACKGROUND_RATIO=${DEF_DIRTY_BACKGROUND_RATIO:-'10'}
388	DEF_DIRTY_RATIO=${DEF_DIRTY_RATIO:-'40'}
389	DEF_XFS_AGE_BUFFER=${DEF_XFS_AGE_BUFFER:-'15'}
390	DEF_XFS_SYNC_INTERVAL=${DEF_XFS_SYNC_INTERVAL:-'30'}
391	DEF_XFS_BUFD_INTERVAL=${DEF_XFS_BUFD_INTERVAL:-'1'}
392
393	# This must be adjusted manually to the value of HZ in the running kernel
394	# on 2.4, until the XFS people change their 2.4 external interfaces to work in
395	# centisecs. This can be automated, but it's a work in progress that still needs
396	# some fixes. On 2.6 kernels, XFS uses USER_HZ instead of HZ for external
397	# interfaces, and that is currently always set to 100. So you don't need to
398	# change this on 2.6.
399	XFS_HZ=${XFS_HZ:-'100'}
400
401	#############################################################################
402
403	KLEVEL="$(uname -r \|
404	{
405	IFS='.' read a b c
406	echo $a.$b
407	}
408	)"
409	case "$KLEVEL" in
410	"2.4"\|"2.6")
411	;;
412	*)
413	echo "Unhandled kernel version: $KLEVEL ('uname -r' = '$(uname -r)')" >&2
414	exit 1
415	;;
416	esac
417
418	if [ ! -e /proc/sys/vm/laptop_mode ] ; then
419	echo "Kernel is not patched with laptop_mode patch." >&2
420	exit 1
421	fi
422
423	if [ ! -w /proc/sys/vm/laptop_mode ] ; then
424	echo "You do not have enough privileges to enable laptop_mode." >&2
425	exit 1
426	fi
427
428	# Remove an option (the first parameter) of the form option=<number> from
429	# a mount options string (the rest of the parameters).
430	parse_mount_opts () {
431	OPT="$1"
432	shift
433	echo ",$*," \| sed \
434	-e 's/,'"$OPT"'=[0-9]*,/,/g' \
435	-e 's/,,*/,/g' \
436	-e 's/^,//' \
437	-e 's/,$//'
438	}
439
440	# Remove an option (the first parameter) without any arguments from
441	# a mount option string (the rest of the parameters).
442	parse_nonumber_mount_opts () {
443	OPT="$1"
444	shift
445	echo ",$*," \| sed \
446	-e 's/,'"$OPT"',/,/g' \
447	-e 's/,,*/,/g' \
448	-e 's/^,//' \
449	-e 's/,$//'
450	}
451
452	# Find out the state of a yes/no option (e.g. "atime"/"noatime") in
453	# fstab for a given filesystem, and use this state to replace the
454	# value of the option in another mount options string. The device
455	# is the first argument, the option name the second, and the default
456	# value the third. The remainder is the mount options string.
457	#
458	# Example:
459	# parse_yesno_opts_wfstab /dev/hda1 atime atime defaults,noatime
460	#
461	# If fstab contains, say, "rw" for this filesystem, then the result
462	# will be "defaults,atime".
463	parse_yesno_opts_wfstab () {
464	L_DEV="$1"
465	OPT="$2"
466	DEF_OPT="$3"
467	shift 3
468	L_OPTS="$*"
469	PARSEDOPTS1="$(parse_nonumber_mount_opts $OPT $L_OPTS)"
470	PARSEDOPTS1="$(parse_nonumber_mount_opts no$OPT $PARSEDOPTS1)"
471	# Watch for a default atime in fstab
472	FSTAB_OPTS="$(awk '$1 == "'$L_DEV'" { print $4 }' /etc/fstab)"
473	if echo "$FSTAB_OPTS" \| grep "$OPT" > /dev/null ; then
474	# option specified in fstab: extract the value and use it
475	if echo "$FSTAB_OPTS" \| grep "no$OPT" > /dev/null ; then
476	echo "$PARSEDOPTS1,no$OPT"
477	else
478	# no$OPT not found -- so we must have $OPT.
479	echo "$PARSEDOPTS1,$OPT"
480	fi
481	else
482	# option not specified in fstab -- choose the default.
483	echo "$PARSEDOPTS1,$DEF_OPT"
484	fi
485	}
486
487	# Find out the state of a numbered option (e.g. "commit=NNN") in
488	# fstab for a given filesystem, and use this state to replace the
489	# value of the option in another mount options string. The device
490	# is the first argument, and the option name the second. The
491	# remainder is the mount options string in which the replacement
492	# must be done.
493	#
494	# Example:
495	# parse_mount_opts_wfstab /dev/hda1 commit defaults,commit=7
496	#
497	# If fstab contains, say, "commit=3,rw" for this filesystem, then the
498	# result will be "rw,commit=3".
499	parse_mount_opts_wfstab () {
500	L_DEV="$1"
501	OPT="$2"
502	shift 2
503	L_OPTS="$*"
504	PARSEDOPTS1="$(parse_mount_opts $OPT $L_OPTS)"
505	# Watch for a default commit in fstab
506	FSTAB_OPTS="$(awk '$1 == "'$L_DEV'" { print $4 }' /etc/fstab)"
507	if echo "$FSTAB_OPTS" \| grep "$OPT=" > /dev/null ; then
508	# option specified in fstab: extract the value, and use it
509	echo -n "$PARSEDOPTS1,$OPT="
510	echo ",$FSTAB_OPTS," \| sed \
511	-e 's/.*,'"$OPT"'=//' \
512	-e 's/,.*//'
513	else
514	# option not specified in fstab: set it to 0
515	echo "$PARSEDOPTS1,$OPT=0"
516	fi
517	}
518
519	deduce_fstype () {
520	MP="$1"
521	# My root filesystem unfortunately has
522	# type "unknown" in /etc/mtab. If we encounter
523	# "unknown", we try to get the type from fstab.
524	cat /etc/fstab \|
525	grep -v '^#' \|
526	while read FSTAB_DEV FSTAB_MP FSTAB_FST FSTAB_OPTS FSTAB_DUMP FSTAB_DUMP ; do
527	if [ "$FSTAB_MP" = "$MP" ]; then
528	echo $FSTAB_FST
529	exit 0
530	fi
531	done
532	}
533
534	if [ $DO_REMOUNT_NOATIME -eq 1 ] ; then
535	NOATIME_OPT=",noatime"
536	fi
537
538	case "$1" in
539	start)
540	AGE=$((100*$MAX_AGE))
541	XFS_AGE=$(($XFS_HZ*$MAX_AGE))
542	echo -n "Starting laptop_mode"
543
544	if [ -d /proc/sys/vm/pagebuf ] ; then
545	# (For 2.4 and early 2.6.)
546	# This only needs to be set, not reset -- it is only used when
547	# laptop mode is enabled.
548	echo $XFS_AGE > /proc/sys/vm/pagebuf/lm_flush_age
549	echo $XFS_AGE > /proc/sys/fs/xfs/lm_sync_interval
550	elif [ -f /proc/sys/fs/xfs/lm_age_buffer ] ; then
551	# (A couple of early 2.6 laptop mode patches had these.)
552	# The same goes for these.
553	echo $XFS_AGE > /proc/sys/fs/xfs/lm_age_buffer
554	echo $XFS_AGE > /proc/sys/fs/xfs/lm_sync_interval
555	elif [ -f /proc/sys/fs/xfs/age_buffer ] ; then
556	# (2.6.6)
557	# But not for these -- they are also used in normal
558	# operation.
559	echo $XFS_AGE > /proc/sys/fs/xfs/age_buffer
560	echo $XFS_AGE > /proc/sys/fs/xfs/sync_interval
561	elif [ -f /proc/sys/fs/xfs/age_buffer_centisecs ] ; then
562	# (2.6.7 upwards)
563	# And not for these either. These are in centisecs,
564	# not USER_HZ, so we have to use $AGE, not $XFS_AGE.
565	echo $AGE > /proc/sys/fs/xfs/age_buffer_centisecs
566	echo $AGE > /proc/sys/fs/xfs/xfssyncd_centisecs
567	echo 3000 > /proc/sys/fs/xfs/xfsbufd_centisecs
568	fi
569
570	case "$KLEVEL" in
571	"2.4")
572	echo 1 > /proc/sys/vm/laptop_mode
573	echo "30 500 0 0 $AGE $AGE 60 20 0" > /proc/sys/vm/bdflush
574	;;
575	"2.6")
576	echo 5 > /proc/sys/vm/laptop_mode
577	echo "$AGE" > /proc/sys/vm/dirty_writeback_centisecs
578	echo "$AGE" > /proc/sys/vm/dirty_expire_centisecs
579	echo "$DIRTY_RATIO" > /proc/sys/vm/dirty_ratio
580	echo "$DIRTY_BACKGROUND_RATIO" > /proc/sys/vm/dirty_background_ratio
581	;;
582	esac
583	if [ $DO_REMOUNTS -eq 1 ]; then
584	cat /etc/mtab \| while read DEV MP FST OPTS DUMP PASS ; do
585	PARSEDOPTS="$(parse_mount_opts "$OPTS")"
586	if [ "$FST" = 'unknown' ]; then
587	FST=$(deduce_fstype $MP)
588	fi
589	case "$FST" in
590	"ext3")
591	PARSEDOPTS="$(parse_mount_opts commit "$OPTS")"
592	mount $DEV -t $FST $MP -o remount,$PARSEDOPTS,commit=$MAX_AGE$NOATIME_OPT
593	;;
594	"xfs")
595	mount $DEV -t $FST $MP -o remount,$OPTS$NOATIME_OPT
596	;;
597	esac
598	if [ -b $DEV ] ; then
599	blockdev --setra $(($READAHEAD * 2)) $DEV
600	fi
601	done
602	fi
603	if [ $DO_HD -eq 1 ] ; then
604	for THISHD in $HD ; do
605	/sbin/hdparm -S $BATT_HD $THISHD > /dev/null 2>&1
606	/sbin/hdparm -B 1 $THISHD > /dev/null 2>&1
607	done
608	fi
609	if [ $DO_CPU -eq 1 -a -e /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq ]; then
610	if [ $CPU_MAXFREQ = 'slowest' ]; then
611	CPU_MAXFREQ=`cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq`
612	fi
613	echo $CPU_MAXFREQ > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
614	fi
615	echo "."
616	;;
617	stop)
618	U_AGE=$((100*$DEF_UPDATE))
619	B_AGE=$((100*$DEF_AGE))
620	echo -n "Stopping laptop_mode"
621	echo 0 > /proc/sys/vm/laptop_mode
622	if [ -f /proc/sys/fs/xfs/age_buffer -a ! -f /proc/sys/fs/xfs/lm_age_buffer ] ; then
623	# These need to be restored, if there are no lm_*.
624	echo $(($XFS_HZ*$DEF_XFS_AGE_BUFFER)) > /proc/sys/fs/xfs/age_buffer
625	echo $(($XFS_HZ*$DEF_XFS_SYNC_INTERVAL)) > /proc/sys/fs/xfs/sync_interval
626	elif [ -f /proc/sys/fs/xfs/age_buffer_centisecs ] ; then
627	# These need to be restored as well.
628	echo $((100*$DEF_XFS_AGE_BUFFER)) > /proc/sys/fs/xfs/age_buffer_centisecs
629	echo $((100*$DEF_XFS_SYNC_INTERVAL)) > /proc/sys/fs/xfs/xfssyncd_centisecs
630	echo $((100*$DEF_XFS_BUFD_INTERVAL)) > /proc/sys/fs/xfs/xfsbufd_centisecs
631	fi
632	case "$KLEVEL" in
633	"2.4")
634	echo "30 500 0 0 $U_AGE $B_AGE 60 20 0" > /proc/sys/vm/bdflush
635	;;
636	"2.6")
637	echo "$U_AGE" > /proc/sys/vm/dirty_writeback_centisecs
638	echo "$B_AGE" > /proc/sys/vm/dirty_expire_centisecs
639	echo "$DEF_DIRTY_RATIO" > /proc/sys/vm/dirty_ratio
640	echo "$DEF_DIRTY_BACKGROUND_RATIO" > /proc/sys/vm/dirty_background_ratio
641	;;
642	esac
643	if [ $DO_REMOUNTS -eq 1 ] ; then
644	cat /etc/mtab \| while read DEV MP FST OPTS DUMP PASS ; do
645	# Reset commit and atime options to defaults.
646	if [ "$FST" = 'unknown' ]; then
647	FST=$(deduce_fstype $MP)
648	fi
649	case "$FST" in
650	"ext3")
651	PARSEDOPTS="$(parse_mount_opts_wfstab $DEV commit $OPTS)"
652	PARSEDOPTS="$(parse_yesno_opts_wfstab $DEV atime atime $PARSEDOPTS)"
653	mount $DEV -t $FST $MP -o remount,$PARSEDOPTS
654	;;
655	"xfs")
656	PARSEDOPTS="$(parse_yesno_opts_wfstab $DEV atime atime $OPTS)"
657	mount $DEV -t $FST $MP -o remount,$PARSEDOPTS
658	;;
659	esac
660	if [ -b $DEV ] ; then
661	blockdev --setra 256 $DEV
662	fi
663	done
664	fi
665	if [ $DO_HD -eq 1 ] ; then
666	for THISHD in $HD ; do
667	/sbin/hdparm -S $AC_HD $THISHD > /dev/null 2>&1
668	/sbin/hdparm -B 255 $THISHD > /dev/null 2>&1
669	done
670	fi
671	if [ $DO_CPU -eq 1 -a -e /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq ]; then
672	echo `cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq` > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
673	fi
674	echo "."
675	;;
676	*)
677	echo "Usage: $0 {start\|stop}" 2>&1
678	exit 1
679	;;
680
681	esac
682
683	exit 0
684
685
686	ACPI integration
687	----------------
688
689	Dax Kelson submitted this so that the ACPI acpid daemon will
690	kick off the laptop_mode script and run hdparm. The part that
691	automatically disables laptop mode when the battery is low was
692	written by Jan Topinski.
693
694	/etc/acpi/events/ac_adapter::
695
696	event=ac_adapter
697	action=/etc/acpi/actions/ac.sh %e
698
699	/etc/acpi/events/battery::
700
701	event=battery.*
702	action=/etc/acpi/actions/battery.sh %e
703
704	/etc/acpi/actions/ac.sh::
705
706	#!/bin/bash
707
708	# ac on/offline event handler
709
710	status=`awk '/^state: / { print $2 }' /proc/acpi/ac_adapter/$2/state`
711
712	case $status in
713	"on-line")
714	/sbin/laptop_mode stop
715	exit 0
716	;;
717	"off-line")
718	/sbin/laptop_mode start
719	exit 0
720	;;
721	esac
722
723
724	/etc/acpi/actions/battery.sh::
725
726	#! /bin/bash
727
728	# Automatically disable laptop mode when the battery almost runs out.
729
730	BATT_INFO=/proc/acpi/battery/$2/state
731
732	if [[ -f /proc/sys/vm/laptop_mode ]]
733	then
734	LM=`cat /proc/sys/vm/laptop_mode`
735	if [[ $LM -gt 0 ]]
736	then
737	if [[ -f $BATT_INFO ]]
738	then
739	# Source the config file only now that we know we need
740	if [ -f /etc/default/laptop-mode ] ; then
741	# Debian
742	. /etc/default/laptop-mode
743	elif [ -f /etc/sysconfig/laptop-mode ] ; then
744	# Others
745	. /etc/sysconfig/laptop-mode
746	fi
747	MINIMUM_BATTERY_MINUTES=${MINIMUM_BATTERY_MINUTES:-'10'}
748
749	ACTION="`cat $BATT_INFO \| grep charging \| cut -c 26-`"
750	if [[ ACTION -eq "discharging" ]]
751	then
752	PRESENT_RATE=`cat $BATT_INFO \| grep "present rate:" \| sed "s/.* $[0-9][0-9]* $.*/\1/" `
753	REMAINING=`cat $BATT_INFO \| grep "remaining capacity:" \| sed "s/.* $[0-9][0-9]* $.*/\1/" `
754	fi
755	if (($REMAINING * 60 / $PRESENT_RATE < $MINIMUM_BATTERY_MINUTES))
756	then
757	/sbin/laptop_mode stop
758	fi
759	else
760	logger -p daemon.warning "You are using laptop mode and your battery interface $BATT_INFO is missing. This may lead to loss of data when the battery runs out. Check kernel ACPI support and /proc/acpi/battery folder, and edit /etc/acpi/battery.sh to set BATT_INFO to the correct path."
761	fi
762	fi
763	fi
764
765
766	Monitoring tool
767	---------------
768
769	Bartek Kania submitted this, it can be used to measure how much time your disk
770	spends spun up/down. See tools/laptop/dslm/dslm.c

Documentation/admin-guide/mm/damon/lru_sort.rst+37−0

@@ -79,6 +79,43 @@ of parametrs except ``enabled`` again. Once the re-reading is done, this · 78 unmodified lines
79	parameter is set as ``N``. If invalid parameters are found while the	79	parameter is set as ``N``. If invalid parameters are found while the
80	re-reading, DAMON_LRU_SORT will be disabled.	80	re-reading, DAMON_LRU_SORT will be disabled.
81		81
		82	active_mem_bp
		83	-------------
		84
		85	Desired active to [in]active memory ratio in bp (1/10,000).
		86
		87	While keeping the caps that set by other quotas, DAMON_LRU_SORT automatically
		88	increases and decreases the effective level of the quota aiming the LRU
		89	[de]prioritizations of the hot and cold memory resulting in this active to
		90	[in]active memory ratio. Value zero means disabling this auto-tuning feature.
		91
		92	Disabled by default.
		93
		94	Auto-tune monitoring intervals
		95	------------------------------
		96
		97	If this parameter is set as ``Y``, DAMON_LRU_SORT automatically tunes DAMON's
		98	sampling and aggregation intervals. The auto-tuning aims to capture meaningful
		99	amount of access events in each DAMON-snapshot, while keeping the sampling
		100	interval 5 milliseconds in minimum, and 10 seconds in maximum. Setting this as
		101	``N`` disables the auto-tuning.
		102
		103	Disabled by default.
		104
		105	filter_young_pages
		106	------------------
		107
		108	Filter [non-]young pages accordingly for LRU [de]prioritizations.
		109
		110	If this is set, check page level access (youngness) once again before each
		111	LRU [de]prioritization operation. LRU prioritization operation is skipped
		112	if the page has not accessed since the last check (not young). LRU
		113	deprioritization operation is skipped if the page has accessed since the
		114	last check (young). The feature is enabled or disabled if this parameter is
		115	set as ``Y`` or ``N``, respectively.
		116
		117	Disabled by default.
		118
82	hot_thres_access_freq	119	hot_thres_access_freq
83	---------------------	120	---------------------
84		121

Documentation/admin-guide/mm/damon/usage.rst+14−5

@@ -6,6 +6,11 @@ Detailed Usages · 5 unmodified lines
6		6
7	DAMON provides below interfaces for different users.	7	DAMON provides below interfaces for different users.
8		8
		9	- Special-purpose DAMON modules.
		10	:ref:`This <damon_modules_special_purpose>` is for people who are building,
		11	distributing, and/or administrating the kernel with special-purpose DAMON
		12	usages. Using this, users can use DAMON's major features for the given
		13	purposes in build, boot, or runtime in simple ways.
9	- DAMON user space tool.	14	- DAMON user space tool.
10	`This <https://github.com/damonitor/damo>`_ is for privileged people such as	15	`This <https://github.com/damonitor/damo>`_ is for privileged people such as
11	system administrators who want a just-working human-friendly interface.	16	system administrators who want a just-working human-friendly interface.
@@ -87,7 +92,7 @@ comma (","). · 75 unmodified lines
87	│ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,target_idx,min,max	92	│ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,target_idx,min,max
88	│ │ │ │ │ │ │ :ref:`dests <damon_sysfs_dests>`/nr_dests	93	│ │ │ │ │ │ │ :ref:`dests <damon_sysfs_dests>`/nr_dests
89	│ │ │ │ │ │ │ │ 0/id,weight	94	│ │ │ │ │ │ │ │ 0/id,weight
90	│ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,sz_ops_filter_passed,qt_exceeds	95	│ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,sz_ops_filter_passed,qt_exceeds,nr_snapshots,max_nr_snapshots
91	│ │ │ │ │ │ │ :ref:`tried_regions <sysfs_schemes_tried_regions>`/total_bytes	96	│ │ │ │ │ │ │ :ref:`tried_regions <sysfs_schemes_tried_regions>`/total_bytes
92	│ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age,sz_filter_passed	97	│ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age,sz_filter_passed
93	│ │ │ │ │ │ │ │ ...	98	│ │ │ │ │ │ │ │ ...
@@ -543,10 +548,14 @@ online analysis or tuning of the schemes. Refer to :ref:`design doc · 449 unmodified lines
543		548
544	The statistics can be retrieved by reading the files under ``stats`` directory	549	The statistics can be retrieved by reading the files under ``stats`` directory
545	(``nr_tried``, ``sz_tried``, ``nr_applied``, ``sz_applied``,	550	(``nr_tried``, ``sz_tried``, ``nr_applied``, ``sz_applied``,
546	``sz_ops_filter_passed``, and ``qt_exceeds``), respectively. The files are not	551	``sz_ops_filter_passed``, ``qt_exceeds``, ``nr_snapshots`` and
547	updated in real time, so you should ask DAMON sysfs interface to update the	552	``max_nr_snapshots``), respectively.
548	content of the files for the stats by writing a special keyword,	553
549	``update_schemes_stats`` to the relevant ``kdamonds/<N>/state`` file.	554	The files are not updated in real time by default. Users should ask DAMON
		555	sysfs interface to periodically update those using ``refresh_ms``, or do a one
		556	time update by writing a special keyword, ``update_schemes_stats`` to the
		557	relevant ``kdamonds/<N>/state`` file. Refer to :ref:`kdamond directory
		558	<sysfs_kdamond>` for more details.
550		559
551	.. _sysfs_schemes_tried_regions:	560	.. _sysfs_schemes_tried_regions:
552		561

Documentation/admin-guide/mm/memory-hotplug.rst+16−6

   memory for metadata and page tables in the direct map; having a lot of offline
   memory blocks is not a typical case, though.
-- Memory ballooning without balloon compaction is incompatible with
+- Memory ballooning without support for balloon memory migration is incompatible
-  ZONE_MOVABLE. Only some implementations, such as virtio-balloon and
+  with ZONE_MOVABLE. Only some implementations, such as virtio-balloon and
-  pseries CMM, fully support balloon compaction.
+  pseries CMM, fully support balloon memory migration.
-  Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be
+  Further, the CONFIG_BALLOON_MIGRATION kernel configuration option might be
   disabled. In that case, balloon inflation will only perform unmovable
   allocations and silently create a zone imbalance, usually triggered by
   inflation requests from the hypervisor.
-- Gigantic pages are unmovable, resulting in user space consuming a
+- Gigantic pages are unmovable when an architecture does not support
-  lot of unmovable memory.
+  huge page migration and/or the ``movable_gigantic_pages`` sysctl is false.
+  See Documentation/admin-guide/sysctl/vm.rst for more info on this sysctl.
 - Huge pages are unmovable when an architectures does not support huge
   page migration, resulting in a similar issue as with gigantic pages.
 - Concurrent activity that operates on the same physical memory area, such as
   allocating gigantic pages, can result in temporary offlining failures.
+- When an admin sets the ``movable_gigantic_pages`` sysctl to true, gigantic
+  pages are allowed in ZONE_MOVABLE.  This only allows migratable gigantic
+  pages to be allocated; however, if there are no eligible destination gigantic
+  pages at offline, the offlining operation will fail.
+  Users leveraging ``movable_gigantic_pages`` should weigh the value of
+  ZONE_MOVABLE for increasing the reliability of gigantic page allocation
+  against the potential loss of hot-unplug reliability.
 - Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap
   Optimization (HVO) is enabled.

Documentation/admin-guide/sysctl/vm.rst+28−8

 - extfrag_threshold
 - highmem_is_dirtyable
 - hugetlb_shm_group
-- laptop_mode
 - legacy_va_layout
 - lowmem_reserve_ratio
 - max_map_count
 - mmap_min_addr
 - mmap_rnd_bits
 - mmap_rnd_compat_bits
+- movable_gigantic_pages
 - nr_hugepages
 - nr_hugepages_mempolicy
 - nr_overcommit_hugepages
 shared memory segment using hugetlb page.
-laptop_mode
-===========
-laptop_mode is a knob that controls "laptop mode". All the things that are
-controlled by this knob are discussed in Documentation/admin-guide/laptops/laptop-mode.rst.
 legacy_va_layout
 ================
 /proc/sys/vm/mmap_rnd_compat_bits tunable
+movable_gigantic_pages
+======================
+This parameter controls whether gigantic pages may be allocated from
+ZONE_MOVABLE. If set to non-zero, gigantic pages can be allocated
+from ZONE_MOVABLE. ZONE_MOVABLE memory may be created via the kernel
+boot parameter `kernelcore` or via memory hotplug as discussed in
+Documentation/admin-guide/mm/memory-hotplug.rst.
+Support may depend on specific architecture.
+Note that using ZONE_MOVABLE gigantic pages make memory hotremove unreliable.
+Memory hot-remove operations will block indefinitely until the admin reserves
+sufficient gigantic pages to service migration requests associated with the
+memory offlining process.  As HugeTLB gigantic page reservation is a manual
+process (via `nodeN/hugepages/.../nr_hugepages` interfaces) this may not be
+obvious when just attempting to offline a block of memory.
+Additionally, as multiple gigantic pages may be reserved on a single block,
+it may appear that gigantic pages are available for migration when in reality
+they are in the process of being removed. For example if `memoryN` contains
+two gigantic pages, one reserved and one allocated, and an admin attempts to
+offline that block, this operations may hang indefinitely unless another
+reserved gigantic page is available on another block `memoryM`.
 nr_hugepages
 ============

Documentation/core-api/mm-api.rst+1−1

@@ -130,5 +130,5 @@ More Memory Management Functions · 129 unmodified lines
130	.. kernel-doc:: mm/vmscan.c	130	.. kernel-doc:: mm/vmscan.c
131	.. kernel-doc:: mm/memory_hotplug.c	131	.. kernel-doc:: mm/memory_hotplug.c
132	.. kernel-doc:: mm/mmu_notifier.c	132	.. kernel-doc:: mm/mmu_notifier.c
133	.. kernel-doc:: mm/balloon_compaction.c	133	.. kernel-doc:: mm/balloon.c
134	.. kernel-doc:: mm/huge_memory.c	134	.. kernel-doc:: mm/huge_memory.c

Documentation/driver-api/cxl/linux/early-boot.rst+1−1

@@ -125,7 +125,7 @@ The contiguous memory allocator (CMA) enables reservation of contiguous memory · 124 unmodified lines
125	regions on NUMA nodes during early boot. However, CMA cannot reserve memory	125	regions on NUMA nodes during early boot. However, CMA cannot reserve memory
126	on NUMA nodes that are not online during early boot. ::	126	on NUMA nodes that are not online during early boot. ::
127		127
128	void __init hugetlb_cma_reserve(int order) {	128	void __init hugetlb_cma_reserve(void) {
129	if (!node_online(nid))	129	if (!node_online(nid))
130	/* do not allow reservations */	130	/* do not allow reservations */
131	}	131	}

Documentation/mm/damon/design.rst+29−3

   specific NUMA node, in bp (1/10,000).
 - ``node_memcg_free_bp``: Specific cgroup's node unused memory ratio for a
   specific NUMA node, in bp (1/10,000).
+- ``active_mem_bp``: Active to active + inactive (LRU) memory size ratio in bp
+  (1/10,000).
+- ``inactive_mem_bp``: Inactive to active + inactive (LRU) memory size ratio in
+  bp (1/10,000).
 ``nid`` is optionally required for only ``node_mem_used_bp``,
 ``node_mem_free_bp``, ``node_memcg_used_bp`` and ``node_memcg_free_bp`` to
 - ``nr_applied``: Total number of regions that the scheme is applied.
 - ``sz_applied``: Total size of regions that the scheme is applied.
 - ``qt_exceeds``: Total number of times the quota of the scheme has exceeded.
+- ``nr_snapshots``: Total number of DAMON snapshots that the scheme is tried to
+  be applied.
+- ``max_nr_snapshots``: Upper limit of ``nr_snapshots``.
 "A scheme is tried to be applied to a region" means DAMOS core logic determined
 the region is eligible to apply the scheme's :ref:`action
 action is ``pageout`` while all pages of the region are unreclaimable, applying
 the action to the region will fail.
+Unlike normal stats, ``max_nr_snapshots`` is set by users.  If it is set as
+non-zero and ``nr_snapshots`` be same to or greater than ``nr_snapshots``, the
+scheme is deactivated.
 To know how user-space can read the stats via :ref:`DAMON sysfs interface
 <sysfs_interface>`, refer to :ref:s`stats <sysfs_stats>` part of the
 documentation.
 rather than human beings' fingers.  Human users are recommended to use such
 user space tools.  One such Python-written user space tool is available at
 Github (https://github.com/damonitor/damo), Pypi
-(https://pypistats.org/packages/damo), and Fedora
+(https://pypistats.org/packages/damo), and multiple distros
-(https://packages.fedoraproject.org/pkgs/python-damo/damo/).
+(https://repology.org/project/damo/versions).
 Currently, one module for this type, namely 'DAMON sysfs interface' is
 available.  Please refer to the ABI :ref:`doc <sysfs_interface>` for details of
 the interfaces.
+.. _damon_modules_special_purpose:
 Special-Purpose Access-aware Kernel Modules
 -------------------------------------------
 simple and optimized user space interfaces are available.  Currently, two
 modules for proactive reclamation and LRU lists manipulation are provided.  For
 more detail, please read the usage documents for those
-(:doc:`/admin-guide/mm/damon/reclaim` and
+(:doc:`/admin-guide/mm/damon/stat`, :doc:`/admin-guide/mm/damon/reclaim` and
 :doc:`/admin-guide/mm/damon/lru_sort`).
+Sample DAMON Modules
+--------------------
+DAMON modules that provides example DAMON kernel API usages.
+kernel programmers can build their own special or general purpose DAMON modules
+using DAMON kernel API.  To help them easily understand how DAMON kernel API
+can be used, a few sample modules are provided under ``samples/damon/`` of the
+linux source tree.  Please note that these modules are not developed for being
+used on real products, but only for showing how DAMON kernel API can be used in
+simple ways.

Documentation/mm/damon/index.rst+9−22

 DAMON: Data Access MONitoring and Access-aware System Operations
 ================================================================
-DAMON is a Linux kernel subsystem that provides a framework for data access
+DAMON is a Linux kernel subsystem for efficient :ref:`data access monitoring
-monitoring and the monitoring results based system operations.  The core
+<damon_design_monitoring>` and :ref:`access-aware system operations
-monitoring :ref:`mechanisms <damon_design_monitoring>` of DAMON make it
+<damon_design_damos>`.  It is designed for being
- - *accurate* (the monitoring output is useful enough for DRAM level memory
+ - *accurate* (for DRAM level memory management),
-   management; It might not appropriate for CPU Cache levels, though),
+ - *light-weight* (for production online usages),
- - *light-weight* (the monitoring overhead is low enough to be applied online),
+ - *scalable* (in terms of memory size),
-   and
+ - *tunable* (for flexible usages), and
- - *scalable* (the upper-bound of the overhead is in constant range regardless
+ - *autoamted* (for production operation without manual tunings).
-   of the size of target workloads).
-Using this framework, therefore, the kernel can operate system in an
-access-aware fashion.  Because the features are also exposed to the :doc:`user
-space </admin-guide/mm/damon/index>`, users who have special information about
-their workloads can write personalized applications for better understanding
-and optimizations of their workloads and systems.
-For easier development of such systems, DAMON provides a feature called
-:ref:`DAMOS <damon_design_damos>` (DAMon-based Operation Schemes) in addition
-to the monitoring.  Using the feature, DAMON users in both kernel and :doc:`user
-spaces </admin-guide/mm/damon/index>` can do access-aware system operations
-with no code but simple configurations.
 .. toctree::
    :maxdepth: 2

Documentation/mm/damon/maintainer-profile.rst+3−4

 DAMON Maintainer Entry Profile
 ==============================
-The DAMON subsystem covers the files that are listed in 'DATA ACCESS MONITOR'
+The DAMON subsystem covers the files that are listed in 'DAMON' section of
-section of 'MAINTAINERS' file.
+'MAINTAINERS' file.
 The mailing lists for the subsystem are damon@lists.linux.dev and
 linux-mm@kvack.org.  Patches should be made against the `mm-new tree
 - Run `damon-tests/corr
   <https://github.com/damonitor/damon-tests/tree/master/corr>`_ for normal
   changes.
-- Run `damon-tests/perf
+- Measure impacts on benchmarks or real world workloads for performance
-  <https://github.com/damonitor/damon-tests/tree/master/perf>`_ for performance
   changes.
 Key cycle dates

Documentation/mm/memory-model.rst+0−3

@@ -97,9 +97,6 @@ sections: · 96 unmodified lines
97	`mem_section` objects and the number of rows is calculated to fit	97	`mem_section` objects and the number of rows is calculated to fit
98	all the memory sections.	98	all the memory sections.
99		99
100	The architecture setup code should call sparse_init() to
101	initialize the memory sections and the memory maps.
102
103	With SPARSEMEM there are two possible ways to convert a PFN to the	100	With SPARSEMEM there are two possible ways to convert a PFN to the
104	corresponding `struct page` - a "classic sparse" and "sparse	101	corresponding `struct page` - a "classic sparse" and "sparse
105	vmemmap". The selection is made at build time and it is determined by	102	vmemmap". The selection is made at build time and it is determined by

Documentation/translations/zh_CN/mm/memory-model.rst+0−2

@@ -83,8 +83,6 @@ SPARSEMEM模型将物理内存显示为一个部分的集合。一个区段用me · 82 unmodified lines
83	每一行包含价值 `PAGE_SIZE` 的 `mem_section` 对象，行数的计算是为了适应所有的	83	每一行包含价值 `PAGE_SIZE` 的 `mem_section` 对象，行数的计算是为了适应所有的
84	内存区。	84	内存区。
85		85
86	架构设置代码应该调用sparse_init()来初始化内存区和内存映射。
87
88	通过SPARSEMEM，有两种可能的方式将PFN转换为相应的 `struct page` --"classic sparse"和	86	通过SPARSEMEM，有两种可能的方式将PFN转换为相应的 `struct page` --"classic sparse"和
89	"sparse vmemmap"。选择是在构建时进行的，它由 `CONFIG_SPARSEMEM_VMEMMAP` 的	87	"sparse vmemmap"。选择是在构建时进行的，它由 `CONFIG_SPARSEMEM_VMEMMAP` 的
90	值决定。	88	值决定。

MAINTAINERS+11−3

 F:	mm/
 F:	tools/mm/
+MEMORY MANAGEMENT - BALLOON
+M:	Andrew Morton <akpm@linux-foundation.org>
+M:	David Hildenbrand <david@kernel.org>
+L:	linux-mm@kvack.org
+L:	virtualization@lists.linux.dev
+S:	Maintained
+W:	http://www.linux-mm.org
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
+F:	include/linux/balloon.h
+F:	mm/balloon.c
 MEMORY MANAGEMENT - CORE
 M:	Andrew Morton <akpm@linux-foundation.org>
 M:	David Hildenbrand <david@kernel.org>
 R:	Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
 L:	linux-mm@kvack.org
 S:	Maintained
-F:	mm/pt_reclaim.c
 F:	mm/vmscan.c
 F:	mm/workingset.c
 L:	virtualization@lists.linux.dev
 S:	Maintained
 F:	drivers/virtio/virtio_balloon.c
-F:	include/linux/balloon_compaction.h
 F:	include/uapi/linux/virtio_balloon.h
-F:	mm/balloon_compaction.c
 VIRTIO BLOCK AND SCSI DRIVERS
 M:	"Michael S. Tsirkin" <mst@redhat.com>

arch/alpha/Kconfig+1−0

@@ -38,6 +38,7 @@ config ALPHA · 37 unmodified lines
38	select OLD_SIGSUSPEND	38	select OLD_SIGSUSPEND
39	select CPU_NO_EFFICIENT_FFS if !ALPHA_EV67	39	select CPU_NO_EFFICIENT_FFS if !ALPHA_EV67
40	select MMU_GATHER_NO_RANGE	40	select MMU_GATHER_NO_RANGE
		41	select MMU_GATHER_RCU_TABLE_FREE
41	select SPARSEMEM_EXTREME if SPARSEMEM	42	select SPARSEMEM_EXTREME if SPARSEMEM
42	select ZONE_DMA	43	select ZONE_DMA
43	help	44	help

arch/alpha/include/asm/page.h+0−1

@@ -11,7 +11,6 @@ · 10 unmodified lines
11	#define STRICT_MM_TYPECHECKS	11	#define STRICT_MM_TYPECHECKS
12		12
13	extern void clear_page(void *page);	13	extern void clear_page(void *page);
14	#define clear_user_page(page, vaddr, pg) clear_page(page)
15		14
16	#define vma_alloc_zeroed_movable_folio(vma, vaddr) \	15	#define vma_alloc_zeroed_movable_folio(vma, vaddr) \
17	vma_alloc_folio(GFP_HIGHUSER_MOVABLE \| __GFP_ZERO, 0, vma, vaddr)	16	vma_alloc_folio(GFP_HIGHUSER_MOVABLE \| __GFP_ZERO, 0, vma, vaddr)

arch/alpha/include/asm/tlb.h+3−3

@@ -4,7 +4,7 @@ · 3 unmodified lines
4		4
5	#include <asm-generic/tlb.h>	5	#include <asm-generic/tlb.h>
6		6
7	#define __pte_free_tlb(tlb, pte, address) pte_free((tlb)->mm, pte)	7	#define __pte_free_tlb(tlb, pte, address) tlb_remove_ptdesc((tlb), page_ptdesc(pte))
8	#define __pmd_free_tlb(tlb, pmd, address) pmd_free((tlb)->mm, pmd)	8	#define __pmd_free_tlb(tlb, pmd, address) tlb_remove_ptdesc((tlb), virt_to_ptdesc(pmd))
9		9
10	#endif	10	#endif

arch/alpha/kernel/setup.c+0−1

@@ -607,7 +607,6 @@ setup_arch(char **cmdline_p) · 606 unmodified lines
607	/* Find our memory. */	607	/* Find our memory. */
608	setup_memory(kernel_end);	608	setup_memory(kernel_end);
609	memblock_set_bottom_up(true);	609	memblock_set_bottom_up(true);
610	sparse_init();
611		610
612	/* First guess at cpu cache sizes. Do this before init_arch. */	611	/* First guess at cpu cache sizes. Do this before init_arch. */
613	determine_cpu_caches(cpu->type);	612	determine_cpu_caches(cpu->type);

arch/alpha/mm/init.c+7−9

@@ -208,12 +208,8 @@ callback_init(void * kernel_end) · 207 unmodified lines
208	return kernel_end;	208	return kernel_end;
209	}	209	}
210		210
211	/*	211	void __init arch_zone_limits_init(unsigned long *max_zone_pfn)
212	* paging_init() sets up the memory map.
213	*/
214	void __init paging_init(void)
215	{	212	{
216	unsigned long max_zone_pfn[MAX_NR_ZONES] = {0, };
217	unsigned long dma_pfn;	213	unsigned long dma_pfn;
218		214
219	dma_pfn = virt_to_phys((char *)MAX_DMA_ADDRESS) >> PAGE_SHIFT;	215	dma_pfn = virt_to_phys((char *)MAX_DMA_ADDRESS) >> PAGE_SHIFT;
@@ -221,11 +217,13 @@ void __init paging_init(void) · 1 unmodified line
221		217
222	max_zone_pfn[ZONE_DMA] = dma_pfn;	218	max_zone_pfn[ZONE_DMA] = dma_pfn;
223	max_zone_pfn[ZONE_NORMAL] = max_pfn;	219	max_zone_pfn[ZONE_NORMAL] = max_pfn;
		220	}
224		221
225	/* Initialize mem_map[]. */	222	/*
226	free_area_init(max_zone_pfn);	223	* paging_init() initializes the kernel's ZERO_PGE.
227		224	*/
228	/* Initialize the kernel's ZERO_PGE. */	225	void __init paging_init(void)
		226	{
229	memset(absolute_pointer(ZERO_PGE), 0, PAGE_SIZE);	227	memset(absolute_pointer(ZERO_PGE), 0, PAGE_SIZE);
230	}	228	}
231		229

arch/arc/include/asm/page.h+2−0

@@ -32,6 +32,8 @@ struct page; · 31 unmodified lines
32		32
33	void copy_user_highpage(struct page to, struct page from,	33	void copy_user_highpage(struct page to, struct page from,
34	unsigned long u_vaddr, struct vm_area_struct *vma);	34	unsigned long u_vaddr, struct vm_area_struct *vma);
		35
		36	#define clear_user_page clear_user_page
35	void clear_user_page(void to, unsigned long u_vaddr, struct page page);	37	void clear_user_page(void to, unsigned long u_vaddr, struct page page);
36		38
37	typedef struct {	39	typedef struct {

arch/arc/mm/init.c+19−18

 		base, TO_MB(size), !in_use ? "Not used":"");
 }
+void __init arch_zone_limits_init(unsigned long *max_zone_pfn)
+{
+	/*----------------- node/zones setup --------------------------*/
+	max_zone_pfn[ZONE_NORMAL] = max_low_pfn;
+#ifdef CONFIG_HIGHMEM
+	/*
+	 * max_high_pfn should be ok here for both HIGHMEM and HIGHMEM+PAE.
+	 * For HIGHMEM without PAE max_high_pfn should be less than
+	 * min_low_pfn to guarantee that these two regions don't overlap.
+	 * For PAE case highmem is greater than lowmem, so it is natural
+	 * to use max_high_pfn.
+	 *
+	 * In both cases, holes should be handled by pfn_valid().
+	 */
+	max_zone_pfn[ZONE_HIGHMEM] = max_high_pfn;
+#endif
+}
 /*
  * First memory setup routine called from setup_arch()
  * 1. setup swapper's mm @init_mm
  */
 void __init setup_arch_memory(void)
 {
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
 	setup_initial_init_mm(_text, _etext, _edata, _end);
 	/* first page of system - kernel .vector starts here */
 	memblock_dump_all();
-	/*----------------- node/zones setup --------------------------*/
-	max_zone_pfn[ZONE_NORMAL] = max_low_pfn;
 #ifdef CONFIG_HIGHMEM
 	/*
 	 * On ARC (w/o PAE) HIGHMEM addresses are actually smaller (0 based)
 	min_high_pfn = PFN_DOWN(high_mem_start);
 	max_high_pfn = PFN_DOWN(high_mem_start + high_mem_sz);
-	/*
-	 * max_high_pfn should be ok here for both HIGHMEM and HIGHMEM+PAE.
-	 * For HIGHMEM without PAE max_high_pfn should be less than
-	 * min_low_pfn to guarantee that these two regions don't overlap.
-	 * For PAE case highmem is greater than lowmem, so it is natural
-	 * to use max_high_pfn.
-	 *
-	 * In both cases, holes should be handled by pfn_valid().
-	 */
-	max_zone_pfn[ZONE_HIGHMEM] = max_high_pfn;
 	arch_pfn_offset = min(min_low_pfn, min_high_pfn);
 	kmap_init();
 #endif /* CONFIG_HIGHMEM */
-	free_area_init(max_zone_pfn);
 }
 void __init arch_mm_preinit(void)

arch/arm/include/asm/page-nommu.h+0−1

@@ -11,7 +11,6 @@ · 10 unmodified lines
11	#define clear_page(page) memset((page), 0, PAGE_SIZE)	11	#define clear_page(page) memset((page), 0, PAGE_SIZE)
12	#define copy_page(to,from) memcpy((to), (from), PAGE_SIZE)	12	#define copy_page(to,from) memcpy((to), (from), PAGE_SIZE)
13		13
14	#define clear_user_page(page, vaddr, pg) clear_page(page)
15	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	14	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
16		15
17	/*	16	/*

arch/arm/include/asm/pgtable.h+2−2

@@ -15,8 +15,8 @@ · 14 unmodified lines
15	* ZERO_PAGE is a global shared page that is always zero: used	15	* ZERO_PAGE is a global shared page that is always zero: used
16	* for zero-mapped memory areas etc..	16	* for zero-mapped memory areas etc..
17	*/	17	*/
18	extern struct page *empty_zero_page;	18	extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
19	#define ZERO_PAGE(vaddr) (empty_zero_page)	19	#define ZERO_PAGE(vaddr) (virt_to_page(empty_zero_page))
20	#endif	20	#endif
21		21
22	#include <asm-generic/pgtable-nopud.h>	22	#include <asm-generic/pgtable-nopud.h>

arch/arm/mm/init.c+4−21

 #endif
 }
-static void __init zone_sizes_init(unsigned long min, unsigned long max_low,
+void __init arch_zone_limits_init(unsigned long *max_zone_pfn)
-	unsigned long max_high)
 {
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
 #ifdef CONFIG_ZONE_DMA
-	max_zone_pfn[ZONE_DMA] = min(arm_dma_pfn_limit, max_low);
+	max_zone_pfn[ZONE_DMA] = min(arm_dma_pfn_limit, max_low_pfn);
 #endif
-	max_zone_pfn[ZONE_NORMAL] = max_low;
+	max_zone_pfn[ZONE_NORMAL] = max_low_pfn;
 #ifdef CONFIG_HIGHMEM
-	max_zone_pfn[ZONE_HIGHMEM] = max_high;
+	max_zone_pfn[ZONE_HIGHMEM] = max_pfn;
 #endif
-	free_area_init(max_zone_pfn);
 }
 #ifdef CONFIG_HAVE_ARCH_PFN_VALID
 	early_memtest((phys_addr_t)min_low_pfn << PAGE_SHIFT,
 		      (phys_addr_t)max_low_pfn << PAGE_SHIFT);
-	/*
-	 * sparse_init() tries to allocate memory from memblock, so must be
-	 * done after the fixed reservations
-	 */
-	sparse_init();
-	/*
-	 * Now free the memory - free_area_init needs
-	 * the sparse mem_map arrays initialized by sparse_init()
-	 * for memmap_init_zone(), otherwise all PFNs are invalid.
-	 */
-	zone_sizes_init(min_low_pfn, max_low_pfn, max_pfn);
 }
 /*

arch/arm/mm/mmu.c+1−9

  * empty_zero_page is a special page that is used for
  * zero-initialized data and COW.
  */
-struct page *empty_zero_page;
+unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)] __page_aligned_bss;
 EXPORT_SYMBOL(empty_zero_page);
 /*
  */
 void __init paging_init(const struct machine_desc *mdesc)
 {
-	void *zero_page;
 #ifdef CONFIG_XIP_KERNEL
 	/* Store the kernel RW RAM region start/end in these variables */
 	kernel_sec_start = CONFIG_PHYS_OFFSET & SECTION_MASK;
 	top_pmd = pmd_off_k(0xffff0000);
-	/* allocate the zero page. */
-	zero_page = early_alloc(PAGE_SIZE);
 	bootmem_init();
-	empty_zero_page = virt_to_page(zero_page);
-	__flush_dcache_folio(NULL, page_folio(empty_zero_page));
 }
 void __init early_mm_init(const struct machine_desc *mdesc)

arch/arm/mm/nommu.c+1−9

  * empty_zero_page is a special page that is used for
  * zero-initialized data and COW.
  */
-struct page *empty_zero_page;
+unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)] __page_aligned_bss;
 EXPORT_SYMBOL(empty_zero_page);
 #ifdef CONFIG_ARM_MPU
  */
 void __init paging_init(const struct machine_desc *mdesc)
 {
-	void *zero_page;
 	early_trap_init((void *)vectors_base);
 	mpu_setup();
-	/* allocate the zero page. */
-	zero_page = (void *)memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE);
 	bootmem_init();
-	empty_zero_page = virt_to_page(zero_page);
-	flush_dcache_page(empty_zero_page);
 }
 /*

arch/arm64/Kconfig+1−0

@@ -35,6 +35,7 @@ config ARM64 · 34 unmodified lines
35	select ARCH_HAS_KCOV	35	select ARCH_HAS_KCOV
36	select ARCH_HAS_KERNEL_FPU_SUPPORT if KERNEL_MODE_NEON	36	select ARCH_HAS_KERNEL_FPU_SUPPORT if KERNEL_MODE_NEON
37	select ARCH_HAS_KEEPINITRD	37	select ARCH_HAS_KEEPINITRD
		38	select ARCH_HAS_LAZY_MMU_MODE
38	select ARCH_HAS_MEMBARRIER_SYNC_CORE	39	select ARCH_HAS_MEMBARRIER_SYNC_CORE
39	select ARCH_HAS_MEM_ENCRYPT	40	select ARCH_HAS_MEM_ENCRYPT
40	select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS	41	select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS

arch/arm64/include/asm/hugetlb.h+0−2

@@ -56,8 +56,6 @@ extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, · 55 unmodified lines
56	#define __HAVE_ARCH_HUGE_PTEP_GET	56	#define __HAVE_ARCH_HUGE_PTEP_GET
57	extern pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep);	57	extern pte_t huge_ptep_get(struct mm_struct mm, unsigned long addr, pte_t ptep);
58		58
59	void __init arm64_hugetlb_cma_reserve(void);
60
61	#define huge_ptep_modify_prot_start huge_ptep_modify_prot_start	59	#define huge_ptep_modify_prot_start huge_ptep_modify_prot_start
62	extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma,	60	extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma,
63	unsigned long addr, pte_t *ptep);	61	unsigned long addr, pte_t *ptep);

arch/arm64/include/asm/page.h+0−1

@@ -36,7 +36,6 @@ struct folio vma_alloc_zeroed_movable_folio(struct vm_area_struct vma, · 35 unmodified lines
36	bool tag_clear_highpages(struct page *to, int numpages);	36	bool tag_clear_highpages(struct page *to, int numpages);
37	#define __HAVE_ARCH_TAG_CLEAR_HIGHPAGES	37	#define __HAVE_ARCH_TAG_CLEAR_HIGHPAGES
38		38
39	#define clear_user_page(page, vaddr, pg) clear_page(page)
40	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	39	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
41		40
42	typedef struct page *pgtable_t;	41	typedef struct page *pgtable_t;

arch/arm64/include/asm/pgtable.h+26−61

 static inline void queue_pte_barriers(void)
 {
-	unsigned long flags;
+	if (is_lazy_mmu_mode_active()) {
-	if (in_interrupt()) {
-		emit_pte_barriers();
-		return;
-	}
-	flags = read_thread_flags();
-	if (flags & BIT(TIF_LAZY_MMU)) {
 		/* Avoid the atomic op if already set. */
-		if (!(flags & BIT(TIF_LAZY_MMU_PENDING)))
+		if (!test_thread_flag(TIF_LAZY_MMU_PENDING))
 			set_thread_flag(TIF_LAZY_MMU_PENDING);
 	} else {
 		emit_pte_barriers();
 	}
 }
-#define  __HAVE_ARCH_ENTER_LAZY_MMU_MODE
+static inline void arch_enter_lazy_mmu_mode(void) {}
-static inline void arch_enter_lazy_mmu_mode(void)
-{
-	/*
-	 * lazy_mmu_mode is not supposed to permit nesting. But in practice this
-	 * does happen with CONFIG_DEBUG_PAGEALLOC, where a page allocation
-	 * inside a lazy_mmu_mode section (such as zap_pte_range()) will change
-	 * permissions on the linear map with apply_to_page_range(), which
-	 * re-enters lazy_mmu_mode. So we tolerate nesting in our
-	 * implementation. The first call to arch_leave_lazy_mmu_mode() will
-	 * flush and clear the flag such that the remainder of the work in the
-	 * outer nest behaves as if outside of lazy mmu mode. This is safe and
-	 * keeps tracking simple.
-	 */
-	if (in_interrupt())
-		return;
-	set_thread_flag(TIF_LAZY_MMU);
-}
 static inline void arch_flush_lazy_mmu_mode(void)
 {
-	if (in_interrupt())
-		return;
 	if (test_and_clear_thread_flag(TIF_LAZY_MMU_PENDING))
 		emit_pte_barriers();
 }
 static inline void arch_leave_lazy_mmu_mode(void)
 {
-	if (in_interrupt())
-		return;
 	arch_flush_lazy_mmu_mode();
-	clear_thread_flag(TIF_LAZY_MMU);
 }
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud));
 }
-static inline void __set_ptes_anysz(struct mm_struct *mm, pte_t *ptep,
+static inline void __set_ptes_anysz(struct mm_struct *mm, unsigned long addr,
-				    pte_t pte, unsigned int nr,
+				    pte_t *ptep, pte_t pte, unsigned int nr,
 				    unsigned long pgsize)
 {
 	unsigned long stride = pgsize >> PAGE_SHIFT;
 	switch (pgsize) {
 	case PAGE_SIZE:
-		page_table_check_ptes_set(mm, ptep, pte, nr);
+		page_table_check_ptes_set(mm, addr, ptep, pte, nr);
 		break;
 	case PMD_SIZE:
-		page_table_check_pmds_set(mm, (pmd_t *)ptep, pte_pmd(pte), nr);
+		page_table_check_pmds_set(mm, addr, (pmd_t *)ptep,
+					  pte_pmd(pte), nr);
 		break;
 #ifndef __PAGETABLE_PMD_FOLDED
 	case PUD_SIZE:
-		page_table_check_puds_set(mm, (pud_t *)ptep, pte_pud(pte), nr);
+		page_table_check_puds_set(mm, addr, (pud_t *)ptep,
+					  pte_pud(pte), nr);
 		break;
 #endif
 	default:
 	__set_pte_complete(pte);
 }
-static inline void __set_ptes(struct mm_struct *mm,
+static inline void __set_ptes(struct mm_struct *mm, unsigned long addr,
-			      unsigned long __always_unused addr,
 			      pte_t *ptep, pte_t pte, unsigned int nr)
 {
-	__set_ptes_anysz(mm, ptep, pte, nr, PAGE_SIZE);
+	__set_ptes_anysz(mm, addr, ptep, pte, nr, PAGE_SIZE);
 }
-static inline void __set_pmds(struct mm_struct *mm,
+static inline void __set_pmds(struct mm_struct *mm, unsigned long addr,
-			      unsigned long __always_unused addr,
 			      pmd_t *pmdp, pmd_t pmd, unsigned int nr)
 {
-	__set_ptes_anysz(mm, (pte_t *)pmdp, pmd_pte(pmd), nr, PMD_SIZE);
+	__set_ptes_anysz(mm, addr, (pte_t *)pmdp, pmd_pte(pmd), nr, PMD_SIZE);
 }
 #define set_pmd_at(mm, addr, pmdp, pmd) __set_pmds(mm, addr, pmdp, pmd, 1)
-static inline void __set_puds(struct mm_struct *mm,
+static inline void __set_puds(struct mm_struct *mm, unsigned long addr,
-			      unsigned long __always_unused addr,
 			      pud_t *pudp, pud_t pud, unsigned int nr)
 {
-	__set_ptes_anysz(mm, (pte_t *)pudp, pud_pte(pud), nr, PUD_SIZE);
+	__set_ptes_anysz(mm, addr, (pte_t *)pudp, pud_pte(pud), nr, PUD_SIZE);
 }
 #define set_pud_at(mm, addr, pudp, pud) __set_puds(mm, addr, pudp, pud, 1)
 #endif
 #ifdef CONFIG_PAGE_TABLE_CHECK
-static inline bool pte_user_accessible_page(pte_t pte)
+static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr)
 {
 	return pte_valid(pte) && (pte_user(pte) || pte_user_exec(pte));
 }
-static inline bool pmd_user_accessible_page(pmd_t pmd)
+static inline bool pmd_user_accessible_page(pmd_t pmd, unsigned long addr)
 {
 	return pmd_valid(pmd) && !pmd_table(pmd) && (pmd_user(pmd) || pmd_user_exec(pmd));
 }
-static inline bool pud_user_accessible_page(pud_t pud)
+static inline bool pud_user_accessible_page(pud_t pud, unsigned long addr)
 {
 	return pud_valid(pud) && !pud_table(pud) && (pud_user(pud) || pud_user_exec(pud));
 }
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
 static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,
+					       unsigned long address,
 					       pte_t *ptep,
 					       unsigned long pgsize)
 {
 	switch (pgsize) {
 	case PAGE_SIZE:
-		page_table_check_pte_clear(mm, pte);
+		page_table_check_pte_clear(mm, address, pte);
 		break;
 	case PMD_SIZE:
-		page_table_check_pmd_clear(mm, pte_pmd(pte));
+		page_table_check_pmd_clear(mm, address, pte_pmd(pte));
 		break;
 #ifndef __PAGETABLE_PMD_FOLDED
 	case PUD_SIZE:
-		page_table_check_pud_clear(mm, pte_pud(pte));
+		page_table_check_pud_clear(mm, address, pte_pud(pte));
 		break;
 #endif
 	default:
 static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
 				       unsigned long address, pte_t *ptep)
 {
-	return __ptep_get_and_clear_anysz(mm, ptep, PAGE_SIZE);
+	return __ptep_get_and_clear_anysz(mm, address, ptep, PAGE_SIZE);
 }
 static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr,
 static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
 					    unsigned long address, pmd_t *pmdp)
 {
-	return pte_pmd(__ptep_get_and_clear_anysz(mm, (pte_t *)pmdp, PMD_SIZE));
+	return pte_pmd(__ptep_get_and_clear_anysz(mm, address, (pte_t *)pmdp, PMD_SIZE));
 }
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 		unsigned long address, pmd_t *pmdp, pmd_t pmd)
 {
-	page_table_check_pmd_set(vma->vm_mm, pmdp, pmd);
+	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
 	return __pmd(xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd)));
 }
 #endif

arch/arm64/include/asm/thread_info.h+1−2

@@ -84,8 +84,7 @@ void arch_setup_new_exec(void); · 83 unmodified lines
84	#define TIF_SME_VL_INHERIT 28 /* Inherit SME vl_onexec across exec */	84	#define TIF_SME_VL_INHERIT 28 /* Inherit SME vl_onexec across exec */
85	#define TIF_KERNEL_FPSTATE 29 /* Task is in a kernel mode FPSIMD section */	85	#define TIF_KERNEL_FPSTATE 29 /* Task is in a kernel mode FPSIMD section */
86	#define TIF_TSC_SIGSEGV 30 /* SIGSEGV on counter-timer access */	86	#define TIF_TSC_SIGSEGV 30 /* SIGSEGV on counter-timer access */
87	#define TIF_LAZY_MMU 31 /* Task in lazy mmu mode */	87	#define TIF_LAZY_MMU_PENDING 31 /* Ops pending for lazy mmu mode exit */
88	#define TIF_LAZY_MMU_PENDING 32 /* Ops pending for lazy mmu mode exit */
89		88
90	#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)	89	#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
91	#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)	90	#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)

arch/arm64/mm/hugetlbpage.c+12−15

  * huge pages could still be served from those areas.
  */
 #ifdef CONFIG_CMA
-void __init arm64_hugetlb_cma_reserve(void)
+unsigned int arch_hugetlb_cma_order(void)
 {
-	int order;
 	if (pud_sect_supported())
-		order = PUD_SHIFT - PAGE_SHIFT;
+		return PUD_SHIFT - PAGE_SHIFT;
-	else
-		order = CONT_PMD_SHIFT - PAGE_SHIFT;
-	hugetlb_cma_reserve(order);
+	return CONT_PMD_SHIFT - PAGE_SHIFT;
 }
 #endif /* CONFIG_CMA */
 	pte_t pte, tmp_pte;
 	bool present;
-	pte = __ptep_get_and_clear_anysz(mm, ptep, pgsize);
+	pte = __ptep_get_and_clear_anysz(mm, addr, ptep, pgsize);
 	present = pte_present(pte);
 	while (--ncontig) {
 		ptep++;
-		tmp_pte = __ptep_get_and_clear_anysz(mm, ptep, pgsize);
+		addr += pgsize;
+		tmp_pte = __ptep_get_and_clear_anysz(mm, addr, ptep, pgsize);
 		if (present) {
 			if (pte_dirty(tmp_pte))
 				pte = pte_mkdirty(pte);
 	unsigned long i, saddr = addr;
 	for (i = 0; i < ncontig; i++, addr += pgsize, ptep++)
-		__ptep_get_and_clear_anysz(mm, ptep, pgsize);
+		__ptep_get_and_clear_anysz(mm, addr, ptep, pgsize);
 	if (mm == &init_mm)
 		flush_tlb_kernel_range(saddr, addr);
 	ncontig = num_contig_ptes(sz, &pgsize);
 	if (!pte_present(pte)) {
-		for (i = 0; i < ncontig; i++, ptep++)
+		for (i = 0; i < ncontig; i++, ptep++, addr += pgsize)
-			__set_ptes_anysz(mm, ptep, pte, 1, pgsize);
+			__set_ptes_anysz(mm, addr, ptep, pte, 1, pgsize);
 		return;
 	}
 	if (pte_cont(pte) && pte_valid(__ptep_get(ptep)))
 		clear_flush(mm, addr, ptep, pgsize, ncontig);
-	__set_ptes_anysz(mm, ptep, pte, ncontig, pgsize);
+	__set_ptes_anysz(mm, addr, ptep, pte, ncontig, pgsize);
 }
 pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 	if (pte_young(orig_pte))
 		pte = pte_mkyoung(pte);
-	__set_ptes_anysz(mm, ptep, pte, ncontig, pgsize);
+	__set_ptes_anysz(mm, addr, ptep, pte, ncontig, pgsize);
 	return 1;
 }
 	pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig);
 	pte = pte_wrprotect(pte);
-	__set_ptes_anysz(mm, ptep, pte, ncontig, pgsize);
+	__set_ptes_anysz(mm, addr, ptep, pte, ncontig, pgsize);
 }
 pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,

arch/arm64/mm/init.c+16−23

@@ -118,9 +118,22 @@ static phys_addr_t __init max_zone_phys(phys_addr_t zone_limit) · 117 unmodified lines
118	return min(zone_limit, memblock_end_of_DRAM() - 1) + 1;	118	return min(zone_limit, memblock_end_of_DRAM() - 1) + 1;
119	}	119	}
120		120
121	static void __init zone_sizes_init(void)	121	void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
		122	{
		123	phys_addr_t __maybe_unused dma32_phys_limit =
		124	max_zone_phys(DMA_BIT_MASK(32));
		125
		126	#ifdef CONFIG_ZONE_DMA
		127	max_zone_pfns[ZONE_DMA] = PFN_DOWN(max_zone_phys(zone_dma_limit));
		128	#endif
		129	#ifdef CONFIG_ZONE_DMA32
		130	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(dma32_phys_limit);
		131	#endif
		132	max_zone_pfns[ZONE_NORMAL] = max_pfn;
		133	}
		134
		135	static void __init dma_limits_init(void)
122	{	136	{
123	unsigned long max_zone_pfns[MAX_NR_ZONES] = {0};
124	phys_addr_t __maybe_unused acpi_zone_dma_limit;	137	phys_addr_t __maybe_unused acpi_zone_dma_limit;
125	phys_addr_t __maybe_unused dt_zone_dma_limit;	138	phys_addr_t __maybe_unused dt_zone_dma_limit;
126	phys_addr_t __maybe_unused dma32_phys_limit =	139	phys_addr_t __maybe_unused dma32_phys_limit =
@@ -139,18 +152,13 @@ static void __init zone_sizes_init(void) · 12 unmodified lines
139	if (memblock_start_of_DRAM() < U32_MAX)	152	if (memblock_start_of_DRAM() < U32_MAX)
140	zone_dma_limit = min(zone_dma_limit, U32_MAX);	153	zone_dma_limit = min(zone_dma_limit, U32_MAX);
141	arm64_dma_phys_limit = max_zone_phys(zone_dma_limit);	154	arm64_dma_phys_limit = max_zone_phys(zone_dma_limit);
142	max_zone_pfns[ZONE_DMA] = PFN_DOWN(arm64_dma_phys_limit);
143	#endif	155	#endif
144	#ifdef CONFIG_ZONE_DMA32	156	#ifdef CONFIG_ZONE_DMA32
145	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(dma32_phys_limit);
146	if (!arm64_dma_phys_limit)	157	if (!arm64_dma_phys_limit)
147	arm64_dma_phys_limit = dma32_phys_limit;	158	arm64_dma_phys_limit = dma32_phys_limit;
148	#endif	159	#endif
149	if (!arm64_dma_phys_limit)	160	if (!arm64_dma_phys_limit)
150	arm64_dma_phys_limit = PHYS_MASK + 1;	161	arm64_dma_phys_limit = PHYS_MASK + 1;
151	max_zone_pfns[ZONE_NORMAL] = max_pfn;
152
153	free_area_init(max_zone_pfns);
154	}	162	}
155		163
156	int pfn_is_map_memory(unsigned long pfn)	164	int pfn_is_map_memory(unsigned long pfn)
@@ -303,23 +311,8 @@ void __init bootmem_init(void) · 146 unmodified lines
303		311
304	arch_numa_init();	312	arch_numa_init();
305		313
306	/*
307	* must be done after arch_numa_init() which calls numa_init() to
308	* initialize node_online_map that gets used in hugetlb_cma_reserve()
309	* while allocating required CMA size across online nodes.
310	*/
311	#if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA)
312	arm64_hugetlb_cma_reserve();
313	#endif
314
315	kvm_hyp_reserve();	314	kvm_hyp_reserve();
316		315	dma_limits_init();
317	/*
318	* sparse_init() tries to allocate memory from memblock, so must be
319	* done after the fixed reservations
320	*/
321	sparse_init();
322	zone_sizes_init();
323		316
324	/*	317	/*
325	* Reserve the CMA area after arm64_dma_phys_limit was initialised.	318	* Reserve the CMA area after arm64_dma_phys_limit was initialised.

arch/arm64/mm/mmu.c+4−4

@@ -800,7 +800,7 @@ int split_kernel_leaf_mapping(unsigned long start, unsigned long end) · 799 unmodified lines
800	return -EINVAL;	800	return -EINVAL;
801		801
802	mutex_lock(&pgtable_split_lock);	802	mutex_lock(&pgtable_split_lock);
803	arch_enter_lazy_mmu_mode();	803	lazy_mmu_mode_enable();
804		804
805	/*	805	/*
806	* The split_kernel_leaf_mapping_locked() may sleep, it is not a	806	* The split_kernel_leaf_mapping_locked() may sleep, it is not a
@@ -822,7 +822,7 @@ int split_kernel_leaf_mapping(unsigned long start, unsigned long end) · 15 unmodified lines
822	ret = split_kernel_leaf_mapping_locked(end);	822	ret = split_kernel_leaf_mapping_locked(end);
823	}	823	}
824		824
825	arch_leave_lazy_mmu_mode();	825	lazy_mmu_mode_disable();
826	mutex_unlock(&pgtable_split_lock);	826	mutex_unlock(&pgtable_split_lock);
827	return ret;	827	return ret;
828	}	828	}
@@ -883,10 +883,10 @@ static int range_split_to_ptes(unsigned long start, unsigned long end, gfp_t gfp · 54 unmodified lines
883	{	883	{
884	int ret;	884	int ret;
885		885
886	arch_enter_lazy_mmu_mode();	886	lazy_mmu_mode_enable();
887	ret = walk_kernel_page_table_range_lockless(start, end,	887	ret = walk_kernel_page_table_range_lockless(start, end,
888	&split_to_ptes_ops, NULL, &gfp);	888	&split_to_ptes_ops, NULL, &gfp);
889	arch_leave_lazy_mmu_mode();	889	lazy_mmu_mode_disable();
890		890
891	return ret;	891	return ret;
892	}	892	}

arch/arm64/mm/pageattr.c+2−2

@@ -110,7 +110,7 @@ static int update_range_prot(unsigned long start, unsigned long size, · 109 unmodified lines
110	if (WARN_ON_ONCE(ret))	110	if (WARN_ON_ONCE(ret))
111	return ret;	111	return ret;
112		112
113	arch_enter_lazy_mmu_mode();	113	lazy_mmu_mode_enable();
114		114
115	/*	115	/*
116	* The caller must ensure that the range we are operating on does not	116	* The caller must ensure that the range we are operating on does not
@@ -119,7 +119,7 @@ static int update_range_prot(unsigned long start, unsigned long size, · 2 unmodified lines
119	*/	119	*/
120	ret = walk_kernel_page_table_range_lockless(start, start + size,	120	ret = walk_kernel_page_table_range_lockless(start, start + size,
121	&pageattr_ops, NULL, &data);	121	&pageattr_ops, NULL, &data);
122	arch_leave_lazy_mmu_mode();	122	lazy_mmu_mode_disable();
123		123
124	return ret;	124	return ret;
125	}	125	}

arch/csky/abiv1/inc/abi/page.h+1−0

@@ -10,6 +10,7 @@ static inline unsigned long pages_do_alias(unsigned long addr1, · 9 unmodified lines
10	return (addr1 ^ addr2) & (SHMLBA-1);	10	return (addr1 ^ addr2) & (SHMLBA-1);
11	}	11	}
12		12
		13	#define clear_user_page clear_user_page
13	static inline void clear_user_page(void *addr, unsigned long vaddr,	14	static inline void clear_user_page(void *addr, unsigned long vaddr,
14	struct page *page)	15	struct page *page)
15	{	16	{

arch/csky/abiv2/inc/abi/page.h+0−7

@@ -1,11 +1,4 @@
1	/* SPDX-License-Identifier: GPL-2.0 */	1	/* SPDX-License-Identifier: GPL-2.0 */
2
3	static inline void clear_user_page(void *addr, unsigned long vaddr,
4	struct page *page)
5	{
6	clear_page(addr);
7	}
8
9	static inline void copy_user_page(void to, void from, unsigned long vaddr,	2	static inline void copy_user_page(void to, void from, unsigned long vaddr,
10	struct page *page)	3	struct page *page)
11	{	4	{

arch/csky/kernel/setup.c+8−8

 }
 #endif
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
+#ifdef CONFIG_HIGHMEM
+	max_zone_pfns[ZONE_HIGHMEM] = max_pfn;
+#endif
+}
 static void __init csky_memblock_init(void)
 {
 	unsigned long lowmem_size = PFN_DOWN(LOWMEM_LIMIT - PHYS_OFFSET_OFFSET);
 	unsigned long sseg_size = PFN_DOWN(SSEG_SIZE - PHYS_OFFSET_OFFSET);
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
 	signed long size;
 	memblock_reserve(__pa(_start), _end - _start);
 	setup_initrd();
 #endif
-	max_zone_pfn[ZONE_NORMAL] = max_low_pfn;
 	mmu_init(min_low_pfn, max_low_pfn);
 #ifdef CONFIG_HIGHMEM
-	max_zone_pfn[ZONE_HIGHMEM] = max_pfn;
 	highstart_pfn = max_low_pfn;
 	highend_pfn   = max_pfn;
 	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
 	dma_contiguous_reserve(0);
-	free_area_init(max_zone_pfn);
 }
 void __init setup_arch(char **cmdline_p)
 	setup_smp();
 #endif
-	sparse_init();
 	fixaddr_init();
 #ifdef CONFIG_HIGHMEM

arch/hexagon/include/asm/page.h+0−1

@@ -113,7 +113,6 @@ static inline void clear_page(void *page) · 112 unmodified lines
113	/*	113	/*
114	* Under assumption that kernel always "sees" user map...	114	* Under assumption that kernel always "sees" user map...
115	*/	115	*/
116	#define clear_user_page(page, vaddr, pg) clear_page(page)
117	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	116	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
118		117
119	static inline unsigned long virt_to_pfn(const void *kaddr)	118	static inline unsigned long virt_to_pfn(const void *kaddr)

arch/hexagon/mm/init.c+5−14

@@ -54,17 +54,8 @@ void sync_icache_dcache(pte_t pte) · 53 unmodified lines
54	__vmcache_idsync(addr, PAGE_SIZE);	54	__vmcache_idsync(addr, PAGE_SIZE);
55	}	55	}
56		56
57	/*	57	void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
58	* In order to set up page allocator "nodes",
59	* somebody has to call free_area_init() for UMA.
60	*
61	* In this mode, we only have one pg_data_t
62	* structure: contig_mem_data.
63	*/
64	static void __init paging_init(void)
65	{	58	{
66	unsigned long max_zone_pfn[MAX_NR_ZONES] = {0, };
67
68	/*	59	/*
69	* This is not particularly well documented anywhere, but	60	* This is not particularly well documented anywhere, but
70	* give ZONE_NORMAL all the memory, including the big holes	61	* give ZONE_NORMAL all the memory, including the big holes
@@ -72,11 +63,11 @@ static void __init paging_init(void) · 1 unmodified line
72	* in the bootmem_map; free_area_init should see those bits and	63	* in the bootmem_map; free_area_init should see those bits and
73	* adjust accordingly.	64	* adjust accordingly.
74	*/	65	*/
		66	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
		67	}
75		68
76	max_zone_pfn[ZONE_NORMAL] = max_low_pfn;	69	static void __init paging_init(void)
77		70	{
78	free_area_init(max_zone_pfn); /* sets up the zonelists and mem_map */
79
80	/*	71	/*
81	* Set the init_mm descriptors "context" value to point to the	72	* Set the init_mm descriptors "context" value to point to the
82	* initial kernel segment table's physical address.	73	* initial kernel segment table's physical address.

arch/loongarch/Kconfig+1−0

@@ -187,6 +187,7 @@ config LOONGARCH · 186 unmodified lines
187	select IRQ_LOONGARCH_CPU	187	select IRQ_LOONGARCH_CPU
188	select LOCK_MM_AND_FIND_VMA	188	select LOCK_MM_AND_FIND_VMA
189	select MMU_GATHER_MERGE_VMAS if MMU	189	select MMU_GATHER_MERGE_VMAS if MMU
		190	select MMU_GATHER_RCU_TABLE_FREE
190	select MODULES_USE_ELF_RELA if MODULES	191	select MODULES_USE_ELF_RELA if MODULES
191	select NEED_PER_CPU_EMBED_FIRST_CHUNK	192	select NEED_PER_CPU_EMBED_FIRST_CHUNK
192	select NEED_PER_CPU_PAGE_FIRST_CHUNK	193	select NEED_PER_CPU_PAGE_FIRST_CHUNK

arch/loongarch/include/asm/page.h+0−1

@@ -30,7 +30,6 @@ · 29 unmodified lines
30	extern void clear_page(void *page);	30	extern void clear_page(void *page);
31	extern void copy_page(void to, void from);	31	extern void copy_page(void to, void from);
32		32
33	#define clear_user_page(page, vaddr, pg) clear_page(page)
34	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	33	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
35		34
36	extern unsigned long shm_align_mask;	35	extern unsigned long shm_align_mask;

arch/loongarch/include/asm/pgalloc.h+3−4

 	return pte;
 }
-#define __pte_free_tlb(tlb, pte, address)	\
+#define __pte_free_tlb(tlb, pte, address)	tlb_remove_ptdesc((tlb), page_ptdesc(pte))
-	tlb_remove_ptdesc((tlb), page_ptdesc(pte))
 #ifndef __PAGETABLE_PMD_FOLDED
 	return pmd;
 }
-#define __pmd_free_tlb(tlb, x, addr)	pmd_free((tlb)->mm, x)
+#define __pmd_free_tlb(tlb, x, addr)	tlb_remove_ptdesc((tlb), virt_to_ptdesc(x))
 #endif
 	return pud;
 }
-#define __pud_free_tlb(tlb, x, addr)	pud_free((tlb)->mm, x)
+#define __pud_free_tlb(tlb, x, addr)	tlb_remove_ptdesc((tlb), virt_to_ptdesc(x))
 #endif /* __PAGETABLE_PUD_FOLDED */

arch/loongarch/include/asm/pgtable.h+0−2

@@ -353,8 +353,6 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) · 352 unmodified lines
353	return pte;	353	return pte;
354	}	354	}
355		355
356	extern void paging_init(void);
357
358	#define pte_none(pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))	356	#define pte_none(pte) (!(pte_val(pte) & ~_PAGE_GLOBAL))
359	#define pte_present(pte) (pte_val(pte) & (_PAGE_PRESENT \| _PAGE_PROTNONE))	357	#define pte_present(pte) (pte_val(pte) & (_PAGE_PRESENT \| _PAGE_PROTNONE))
360	#define pte_no_exec(pte) (pte_val(pte) & _PAGE_NO_EXEC)	358	#define pte_no_exec(pte) (pte_val(pte) & _PAGE_NO_EXEC)

arch/loongarch/kernel/setup.c+0−10

 	check_kernel_sections_mem();
-	/*
-	 * In order to reduce the possibility of kernel panic when failed to
-	 * get IO TLB memory under CONFIG_SWIOTLB, it is better to allocate
-	 * low memory as small as possible before swiotlb_init(), so make
-	 * sparse_init() using top-down allocation.
-	 */
-	memblock_set_bottom_up(false);
-	sparse_init();
 	memblock_set_bottom_up(true);
 	swiotlb_init(true, SWIOTLB_VERBOSE);
 	prefill_possible_map();
 #endif
-	paging_init();
 #ifdef CONFIG_KASAN
 	kasan_init();
 #endif

arch/loongarch/mm/init.c+1−5

 	return memblock_is_memory(addr) && !memblock_is_reserved(addr);
 }
-void __init paging_init(void)
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
 {
-	unsigned long max_zone_pfns[MAX_NR_ZONES];
 #ifdef CONFIG_ZONE_DMA32
 	max_zone_pfns[ZONE_DMA32] = MAX_DMA32_PFN;
 #endif
 	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
-	free_area_init(max_zone_pfns);
 }
 void __ref free_initmem(void)

arch/m68k/include/asm/page_no.h+0−1

@@ -10,7 +10,6 @@ extern unsigned long memory_end; · 9 unmodified lines
10	#define clear_page(page) memset((page), 0, PAGE_SIZE)	10	#define clear_page(page) memset((page), 0, PAGE_SIZE)
11	#define copy_page(to,from) memcpy((to), (from), PAGE_SIZE)	11	#define copy_page(to,from) memcpy((to), (from), PAGE_SIZE)
12		12
13	#define clear_user_page(page, vaddr, pg) clear_page(page)
14	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	13	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
15		14
16	#define vma_alloc_zeroed_movable_folio(vma, vaddr) \	15	#define vma_alloc_zeroed_movable_folio(vma, vaddr) \

arch/m68k/mm/init.c+5−3

 void *empty_zero_page;
 EXPORT_SYMBOL(empty_zero_page);
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+	max_zone_pfns[ZONE_DMA] = PFN_DOWN(memblock_end_of_DRAM());
+}
 #ifdef CONFIG_MMU
 int m68k_virt_to_node_shift;
 	 * page_alloc get different views of the world.
 	 */
 	unsigned long end_mem = memory_end & PAGE_MASK;
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0, };
 	high_memory = (void *) end_mem;
 	empty_zero_page = memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE);
-	max_zone_pfn[ZONE_DMA] = end_mem >> PAGE_SHIFT;
-	free_area_init(max_zone_pfn);
 }
 #endif /* CONFIG_MMU */

arch/m68k/mm/mcfmmu.c+0−3

 	pte_t *pg_table;
 	unsigned long address, size;
 	unsigned long next_pgtable;
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
 	int i;
 	empty_zero_page = memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE);
 	}
 	current->mm = NULL;
-	max_zone_pfn[ZONE_DMA] = PFN_DOWN(_ramend);
-	free_area_init(max_zone_pfn);
 }
 int cf_tlb_miss(struct pt_regs *regs, int write, int dtlb, int extension_word)

arch/m68k/mm/motorola.c+1−5

  */
 void __init paging_init(void)
 {
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0, };
 	unsigned long min_addr, max_addr;
 	unsigned long addr;
 	int i;
 	set_fc(USER_DATA);
 #ifdef DEBUG
-	printk ("before free_area_init\n");
+	printk ("before node_set_state\n");
 #endif
 	for (i = 0; i < m68k_num_memory; i++)
 		if (node_present_pages(i))
 			node_set_state(i, N_NORMAL_MEMORY);
-	max_zone_pfn[ZONE_DMA] = memblock_end_of_DRAM();
-	free_area_init(max_zone_pfn);
 }

arch/m68k/mm/sun3mmu.c+0−9

 	unsigned long address;
 	unsigned long next_pgtable;
 	unsigned long bootmem_end;
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0, };
 	unsigned long size;
 	empty_zero_page = memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE);
 	mmu_emu_init(bootmem_end);
 	current->mm = NULL;
-	/* memory sizing is a hack stolen from motorola.c..  hope it works for us */
-	max_zone_pfn[ZONE_DMA] = ((unsigned long)high_memory) >> PAGE_SHIFT;
-	/* I really wish I knew why the following change made things better...  -- Sam */
-	free_area_init(max_zone_pfn);
 }
 static const pgprot_t protection_map[16] = {

arch/microblaze/include/asm/page.h+0−1

@@ -45,7 +45,6 @@ typedef unsigned long pte_basic_t; · 44 unmodified lines
45	# define copy_page(to, from) memcpy((to), (from), PAGE_SIZE)	45	# define copy_page(to, from) memcpy((to), (from), PAGE_SIZE)
46	# define clear_page(pgaddr) memset((pgaddr), 0, PAGE_SIZE)	46	# define clear_page(pgaddr) memset((pgaddr), 0, PAGE_SIZE)
47		47
48	# define clear_user_page(pgaddr, vaddr, page) memset((pgaddr), 0, PAGE_SIZE)
49	# define copy_user_page(vto, vfrom, vaddr, topg) \	48	# define copy_user_page(vto, vfrom, vaddr, topg) \
50	memcpy((vto), (vfrom), PAGE_SIZE)	49	memcpy((vto), (vfrom), PAGE_SIZE)
51		50

arch/microblaze/mm/init.c+10−12

 }
 #endif /* CONFIG_HIGHMEM */
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+#ifdef CONFIG_HIGHMEM
+	max_zone_pfns[ZONE_DMA] = max_low_pfn;
+	max_zone_pfns[ZONE_HIGHMEM] = max_pfn;
+#else
+	max_zone_pfns[ZONE_DMA] = max_pfn;
+#endif
+}
 /*
  * paging_init() sets up the page tables - in fact we've already done this.
  */
 static void __init paging_init(void)
 {
-	unsigned long zones_size[MAX_NR_ZONES];
 	int idx;
 	/* Setup fixmaps */
 	for (idx = 0; idx < __end_of_fixed_addresses; idx++)
 		clear_fixmap(idx);
-	/* Clean every zones */
-	memset(zones_size, 0, sizeof(zones_size));
 #ifdef CONFIG_HIGHMEM
 	highmem_init();
-	zones_size[ZONE_DMA] = max_low_pfn;
-	zones_size[ZONE_HIGHMEM] = max_pfn;
-#else
-	zones_size[ZONE_DMA] = max_pfn;
 #endif
-	/* We don't have holes in memory map */
-	free_area_init(zones_size);
 }
 void __init setup_memory(void)

arch/mips/Kconfig+1−0

@@ -99,6 +99,7 @@ config MIPS · 98 unmodified lines
99	select IRQ_FORCED_THREADING	99	select IRQ_FORCED_THREADING
100	select ISA if EISA	100	select ISA if EISA
101	select LOCK_MM_AND_FIND_VMA	101	select LOCK_MM_AND_FIND_VMA
		102	select MMU_GATHER_RCU_TABLE_FREE
102	select MODULES_USE_ELF_REL if MODULES	103	select MODULES_USE_ELF_REL if MODULES
103	select MODULES_USE_ELF_RELA if MODULES && 64BIT	104	select MODULES_USE_ELF_RELA if MODULES && 64BIT
104	select PERF_USE_VMALLOC	105	select PERF_USE_VMALLOC

arch/mips/include/asm/page.h+1−0

@@ -90,6 +90,7 @@ static inline void clear_user_page(void *addr, unsigned long vaddr, · 89 unmodified lines
90	if (pages_do_alias((unsigned long) addr, vaddr & PAGE_MASK))	90	if (pages_do_alias((unsigned long) addr, vaddr & PAGE_MASK))
91	flush_data_cache_page((unsigned long)addr);	91	flush_data_cache_page((unsigned long)addr);
92	}	92	}
		93	#define clear_user_page clear_user_page
93		94
94	struct vm_area_struct;	95	struct vm_area_struct;
95	extern void copy_user_highpage(struct page to, struct page from,	96	extern void copy_user_highpage(struct page to, struct page from,

arch/mips/include/asm/pgalloc.h+3−6

 extern void pgd_init(void *addr);
 extern pgd_t *pgd_alloc(struct mm_struct *mm);
-#define __pte_free_tlb(tlb, pte, address)	\
+#define __pte_free_tlb(tlb, pte, address)	tlb_remove_ptdesc((tlb), page_ptdesc(pte))
-	tlb_remove_ptdesc((tlb), page_ptdesc(pte))
 #ifndef __PAGETABLE_PMD_FOLDED
 	return pmd;
 }
-#define __pmd_free_tlb(tlb, x, addr)	pmd_free((tlb)->mm, x)
+#define __pmd_free_tlb(tlb, x, addr)	tlb_remove_ptdesc((tlb), virt_to_ptdesc(x))
 #endif
 	set_p4d(p4d, __p4d((unsigned long)pud));
 }
-#define __pud_free_tlb(tlb, x, addr)	pud_free((tlb)->mm, x)
+#define __pud_free_tlb(tlb, x, addr)	tlb_remove_ptdesc((tlb), virt_to_ptdesc(x))
 #endif /* __PAGETABLE_PUD_FOLDED */
-extern void pagetable_init(void);
 #endif /* _ASM_PGALLOC_H */

arch/mips/include/asm/pgtable.h+1−1

@@ -56,7 +56,7 @@ extern unsigned long zero_page_mask; · 55 unmodified lines
56	(virt_to_page((void *)(empty_zero_page + (((unsigned long)(vaddr)) & zero_page_mask))))	56	(virt_to_page((void *)(empty_zero_page + (((unsigned long)(vaddr)) & zero_page_mask))))
57	#define __HAVE_COLOR_ZERO_PAGE	57	#define __HAVE_COLOR_ZERO_PAGE
58		58
59	extern void paging_init(void);	59	extern void pagetable_init(void);
60		60
61	/*	61	/*
62	* Conversion functions: convert a page and protection to a page entry,	62	* Conversion functions: convert a page and protection to a page entry,

arch/mips/kernel/setup.c+2−13

  * kernel but generic memory management system is still entirely uninitialized.
  *
  *  o bootmem_init()
- *  o sparse_init()
+ *  o pagetable_init()
- *  o paging_init()
  *  o dma_contiguous_reserve()
  *
  * At this stage the bootmem allocator is ready to use.
 	mips_parse_crashkernel();
 	device_tree_init();
-	/*
-	 * In order to reduce the possibility of kernel panic when failed to
-	 * get IO TLB memory under CONFIG_SWIOTLB, it is better to allocate
-	 * low memory as small as possible before plat_swiotlb_setup(), so
-	 * make sparse_init() using top-down allocation.
-	 */
-	memblock_set_bottom_up(false);
-	sparse_init();
-	memblock_set_bottom_up(true);
 	plat_swiotlb_setup();
 	dma_contiguous_reserve(PFN_PHYS(max_low_pfn));
 	prefill_possible_map();
 	cpu_cache_init();
-	paging_init();
+	pagetable_init();
 	memblock_dump_all();

arch/mips/loongson64/numa.c+3−7

@@ -154,14 +154,10 @@ static __init void prom_meminit(void) · 153 unmodified lines
154	}	154	}
155	}	155	}
156		156
157	void __init paging_init(void)	157	void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
158	{	158	{
159	unsigned long zones_size[MAX_NR_ZONES] = {0, };	159	max_zone_pfns[ZONE_DMA32] = MAX_DMA32_PFN;
160		160	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
161	pagetable_init();
162	zones_size[ZONE_DMA32] = MAX_DMA32_PFN;
163	zones_size[ZONE_NORMAL] = max_low_pfn;
164	free_area_init(zones_size);
165	}	161	}
166		162
167	/* All PCI device belongs to logical Node-0 */	163	/* All PCI device belongs to logical Node-0 */

arch/mips/mm/init.c+1−7

 }
 #ifndef CONFIG_NUMA
-void __init paging_init(void)
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
 {
-	unsigned long max_zone_pfns[MAX_NR_ZONES];
-	pagetable_init();
 #ifdef CONFIG_ZONE_DMA
 	max_zone_pfns[ZONE_DMA] = MAX_DMA_PFN;
 #endif
 		max_zone_pfns[ZONE_HIGHMEM] = max_low_pfn;
 	}
 #endif
-	free_area_init(max_zone_pfns);
 }
 #ifdef CONFIG_64BIT

arch/mips/sgi-ip27/ip27-memory.c+2−6

 	}
 }
-void __init paging_init(void)
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
 {
-	unsigned long zones_size[MAX_NR_ZONES] = {0, };
+	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
-	pagetable_init();
-	zones_size[ZONE_NORMAL] = max_low_pfn;
-	free_area_init(zones_size);
 }

arch/nios2/include/asm/page.h+1−0

@@ -45,6 +45,7 @@ · 44 unmodified lines
45		45
46	struct page;	46	struct page;
47		47
		48	#define clear_user_page clear_user_page
48	extern void clear_user_page(void addr, unsigned long vaddr, struct page page);	49	extern void clear_user_page(void addr, unsigned long vaddr, struct page page);
49	extern void copy_user_page(void vto, void vfrom, unsigned long vaddr,	50	extern void copy_user_page(void vto, void vfrom, unsigned long vaddr,
50	struct page *to);	51	struct page *to);

arch/nios2/mm/init.c+5−7

 pgd_t *pgd_current;
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
+}
 /*
  * paging_init() continues the virtual memory environment setup which
  * was begun by the code in arch/head.S.
  */
 void __init paging_init(void)
 {
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
 	pagetable_init();
 	pgd_current = swapper_pg_dir;
-	max_zone_pfn[ZONE_NORMAL] = max_low_pfn;
-	/* pass the memory from the bootmem allocator to the main allocator */
-	free_area_init(max_zone_pfn);
 	flush_dcache_range((unsigned long)empty_zero_page,
 			(unsigned long)empty_zero_page + PAGE_SIZE);
 }

arch/openrisc/include/asm/page.h+0−1

@@ -30,7 +30,6 @@ · 29 unmodified lines
30	#define clear_page(page) memset((page), 0, PAGE_SIZE)	30	#define clear_page(page) memset((page), 0, PAGE_SIZE)
31	#define copy_page(to, from) memcpy((to), (from), PAGE_SIZE)	31	#define copy_page(to, from) memcpy((to), (from), PAGE_SIZE)
32		32
33	#define clear_user_page(page, vaddr, pg) clear_page(page)
34	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	33	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
35		34
36	/*	35	/*

arch/openrisc/mm/init.c+2−8

 int mem_init_done;
-static void __init zone_sizes_init(void)
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
 {
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
 	/*
 	 * We use only ZONE_NORMAL
 	 */
-	max_zone_pfn[ZONE_NORMAL] = max_low_pfn;
+	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
-	free_area_init(max_zone_pfn);
 }
 extern const char _s_kernel_ro[], _e_kernel_ro[];
 	map_ram();
-	zone_sizes_init();
 	/* self modifying code ;) */
 	/* Since the old TLB miss handler has been running up until now,
 	 * the kernel pages are still all RW, so we can still modify the

arch/parisc/Kconfig+1−0

@@ -79,6 +79,7 @@ config PARISC · 78 unmodified lines
79	select GENERIC_CLOCKEVENTS	79	select GENERIC_CLOCKEVENTS
80	select CPU_NO_EFFICIENT_FFS	80	select CPU_NO_EFFICIENT_FFS
81	select THREAD_INFO_IN_TASK	81	select THREAD_INFO_IN_TASK
		82	select MMU_GATHER_RCU_TABLE_FREE
82	select NEED_DMA_MAP_STATE	83	select NEED_DMA_MAP_STATE
83	select NEED_SG_DMA_LENGTH	84	select NEED_SG_DMA_LENGTH
84	select HAVE_ARCH_KGDB	85	select HAVE_ARCH_KGDB

arch/parisc/include/asm/page.h+0−1

@@ -21,7 +21,6 @@ struct vm_area_struct; · 20 unmodified lines
21		21
22	void clear_page_asm(void *page);	22	void clear_page_asm(void *page);
23	void copy_page_asm(void to, void from);	23	void copy_page_asm(void to, void from);
24	#define clear_user_page(vto, vaddr, page) clear_page_asm(vto)
25	void copy_user_highpage(struct page to, struct page from, unsigned long vaddr,	24	void copy_user_highpage(struct page to, struct page from, unsigned long vaddr,
26	struct vm_area_struct *vma);	25	struct vm_area_struct *vma);
27	#define __HAVE_ARCH_COPY_USER_HIGHPAGE	26	#define __HAVE_ARCH_COPY_USER_HIGHPAGE

arch/parisc/include/asm/tlb.h+2−2

@@ -5,8 +5,8 @@ · 4 unmodified lines
5	#include <asm-generic/tlb.h>	5	#include <asm-generic/tlb.h>
6		6
7	#if CONFIG_PGTABLE_LEVELS == 3	7	#if CONFIG_PGTABLE_LEVELS == 3
8	#define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd)	8	#define __pmd_free_tlb(tlb, pmd, addr) tlb_remove_ptdesc((tlb), virt_to_ptdesc(pmd))
9	#endif	9	#endif
10	#define __pte_free_tlb(tlb, pte, addr) pte_free((tlb)->mm, pte)	10	#define __pte_free_tlb(tlb, pte, addr) tlb_remove_ptdesc((tlb), page_ptdesc(pte))
11		11
12	#endif	12	#endif

arch/parisc/mm/init.c+2−9

 	} while (addr < end);
 }
-static void __init parisc_bootmem_free(void)
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
 {
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0, };
+	max_zone_pfns[ZONE_NORMAL] = PFN_DOWN(memblock_end_of_DRAM());
-	max_zone_pfn[0] = memblock_end_of_DRAM();
-	free_area_init(max_zone_pfn);
 }
 void __init paging_init(void)
 	fixmap_init();
 	flush_cache_all_local(); /* start with known state */
 	flush_tlb_all_local(NULL);
-	sparse_init();
-	parisc_bootmem_free();
 }
 static void alloc_btlb(unsigned long start, unsigned long end, int *slot,

arch/powerpc/Kconfig+2−0

 	select ARCH_STACKWALK
 	select ARCH_SUPPORTS_ATOMIC_RMW
 	select ARCH_SUPPORTS_DEBUG_PAGEALLOC	if PPC_BOOK3S || PPC_8xx
+	select ARCH_SUPPORTS_PAGE_TABLE_CHECK	if !HUGETLB_PAGE
 	select ARCH_SUPPORTS_SCHED_MC		if SMP
 	select ARCH_SUPPORTS_SCHED_SMT		if PPC64 && SMP
 	select SCHED_MC				if ARCH_SUPPORTS_SCHED_MC
 	select LOCK_MM_AND_FIND_VMA
 	select MMU_GATHER_PAGE_SIZE
 	select MMU_GATHER_RCU_TABLE_FREE
+	select HAVE_ARCH_TLB_REMOVE_TABLE
 	select MMU_GATHER_MERGE_VMAS
 	select MMU_LAZY_TLB_SHOOTDOWN		if PPC_BOOK3S_64
 	select MODULES_USE_ELF_RELA

arch/powerpc/include/asm/book3s/32/pgtable.h+11−1

 #ifndef __ASSEMBLER__
 #include <linux/sched.h>
 #include <linux/threads.h>
+#include <linux/page_table_check.h>
 /* Bits to mask out from a PGD to get to the PUD page */
 #define PGD_MASKED_BITS		0
 static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr,
 				       pte_t *ptep)
 {
-	return __pte(pte_update(mm, addr, ptep, ~_PAGE_HASHPTE, 0, 0));
+	pte_t old_pte = __pte(pte_update(mm, addr, ptep, ~_PAGE_HASHPTE, 0, 0));
+	page_table_check_pte_clear(mm, addr, old_pte);
+	return old_pte;
 }
 #define __HAVE_ARCH_PTEP_SET_WRPROTECT
 	return true;
 }
+static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr)
+{
+	return pte_present(pte) && !is_kernel_addr(addr);
+}
 /* Conversion functions: convert a page and protection to a page entry,
  * and a page entry and page directory to the page they refer to.
  *

arch/powerpc/include/asm/book3s/64/pgtable.h+52−10

@@ -144,6 +144,8 @@ · 143 unmodified lines
144	#define PAGE_KERNEL_ROX __pgprot(_PAGE_BASE \| _PAGE_KERNEL_ROX)	144	#define PAGE_KERNEL_ROX __pgprot(_PAGE_BASE \| _PAGE_KERNEL_ROX)
145		145
146	#ifndef __ASSEMBLER__	146	#ifndef __ASSEMBLER__
		147	#include <linux/page_table_check.h>
		148
147	/*	149	/*
148	* page table defines	150	* page table defines
149	*/	151	*/
@@ -416,8 +418,11 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm, · 266 unmodified lines
416	static inline pte_t ptep_get_and_clear(struct mm_struct *mm,	418	static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
417	unsigned long addr, pte_t *ptep)	419	unsigned long addr, pte_t *ptep)
418	{	420	{
419	unsigned long old = pte_update(mm, addr, ptep, ~0UL, 0, 0);	421	pte_t old_pte = __pte(pte_update(mm, addr, ptep, ~0UL, 0, 0));
420	return __pte(old);	422
		423	page_table_check_pte_clear(mm, addr, old_pte);
		424
		425	return old_pte;
421	}	426	}
422		427
423	#define __HAVE_ARCH_PTEP_GET_AND_CLEAR_FULL	428	#define __HAVE_ARCH_PTEP_GET_AND_CLEAR_FULL
@@ -426,11 +431,16 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · 2 unmodified lines
426	pte_t *ptep, int full)	431	pte_t *ptep, int full)
427	{	432	{
428	if (full && radix_enabled()) {	433	if (full && radix_enabled()) {
		434	pte_t old_pte;
		435
429	/*	436	/*
430	* We know that this is a full mm pte clear and	437	* We know that this is a full mm pte clear and
431	* hence can be sure there is no parallel set_pte.	438	* hence can be sure there is no parallel set_pte.
432	*/	439	*/
433	return radix__ptep_get_and_clear_full(mm, addr, ptep, full);	440	old_pte = radix__ptep_get_and_clear_full(mm, addr, ptep, full);
		441	page_table_check_pte_clear(mm, addr, old_pte);
		442
		443	return old_pte;
434	}	444	}
435	return ptep_get_and_clear(mm, addr, ptep);	445	return ptep_get_and_clear(mm, addr, ptep);
436	}	446	}
@@ -539,6 +549,11 @@ static inline bool pte_access_permitted(pte_t pte, bool write) · 102 unmodified lines
539	return arch_pte_access_permitted(pte_val(pte), write, 0);	549	return arch_pte_access_permitted(pte_val(pte), write, 0);
540	}	550	}
541		551
		552	static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr)
		553	{
		554	return pte_present(pte) && pte_user(pte);
		555	}
		556
542	/*	557	/*
543	* Conversion functions: convert a page and protection to a page entry,	558	* Conversion functions: convert a page and protection to a page entry,
544	* and a page entry and page directory to the page they refer to.	559	* and a page entry and page directory to the page they refer to.
@@ -909,6 +924,12 @@ static inline bool pud_access_permitted(pud_t pud, bool write) · 364 unmodified lines
909	return pte_access_permitted(pud_pte(pud), write);	924	return pte_access_permitted(pud_pte(pud), write);
910	}	925	}
911		926
		927	#define pud_user_accessible_page pud_user_accessible_page
		928	static inline bool pud_user_accessible_page(pud_t pud, unsigned long addr)
		929	{
		930	return pud_leaf(pud) && pte_user_accessible_page(pud_pte(pud), addr);
		931	}
		932
912	#define __p4d_raw(x) ((p4d_t) { __pgd_raw(x) })	933	#define __p4d_raw(x) ((p4d_t) { __pgd_raw(x) })
913	static inline __be64 p4d_raw(p4d_t x)	934	static inline __be64 p4d_raw(p4d_t x)
914	{	935	{
@@ -1074,6 +1095,12 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write) · 159 unmodified lines
1074	return pte_access_permitted(pmd_pte(pmd), write);	1095	return pte_access_permitted(pmd_pte(pmd), write);
1075	}	1096	}
1076		1097
		1098	#define pmd_user_accessible_page pmd_user_accessible_page
		1099	static inline bool pmd_user_accessible_page(pmd_t pmd, unsigned long addr)
		1100	{
		1101	return pmd_leaf(pmd) && pte_user_accessible_page(pmd_pte(pmd), addr);
		1102	}
		1103
1077	#ifdef CONFIG_TRANSPARENT_HUGEPAGE	1104	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
1078	extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot);	1105	extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot);
1079	extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot);	1106	extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot);
@@ -1284,19 +1311,34 @@ extern int pudp_test_and_clear_young(struct vm_area_struct *vma, · 204 unmodified lines
1284	static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,	1311	static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
1285	unsigned long addr, pmd_t *pmdp)	1312	unsigned long addr, pmd_t *pmdp)
1286	{	1313	{
1287	if (radix_enabled())	1314	pmd_t old_pmd;
1288	return radix__pmdp_huge_get_and_clear(mm, addr, pmdp);	1315
1289	return hash__pmdp_huge_get_and_clear(mm, addr, pmdp);	1316	if (radix_enabled()) {
		1317	old_pmd = radix__pmdp_huge_get_and_clear(mm, addr, pmdp);
		1318	} else {
		1319	old_pmd = hash__pmdp_huge_get_and_clear(mm, addr, pmdp);
		1320	}
		1321
		1322	page_table_check_pmd_clear(mm, addr, old_pmd);
		1323
		1324	return old_pmd;
1290	}	1325	}
1291		1326
1292	#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR	1327	#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR
1293	static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,	1328	static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,
1294	unsigned long addr, pud_t *pudp)	1329	unsigned long addr, pud_t *pudp)
1295	{	1330	{
1296	if (radix_enabled())	1331	pud_t old_pud;
1297	return radix__pudp_huge_get_and_clear(mm, addr, pudp);	1332
1298	BUG();	1333	if (radix_enabled()) {
1299	return *pudp;	1334	old_pud = radix__pudp_huge_get_and_clear(mm, addr, pudp);
		1335	} else {
		1336	BUG();
		1337	}
		1338
		1339	page_table_check_pud_clear(mm, addr, old_pud);
		1340
		1341	return old_pud;
1300	}	1342	}
1301		1343
1302	static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,	1344	static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,

arch/powerpc/include/asm/book3s/64/tlbflush-hash.h+9−11

 #define PPC64_TLB_BATCH_NR 192
 struct ppc64_tlb_batch {
-	int			active;
 	unsigned long		index;
 	struct mm_struct	*mm;
 	real_pte_t		pte[PPC64_TLB_BATCH_NR];
 extern void __flush_tlb_pending(struct ppc64_tlb_batch *batch);
-#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE
 static inline void arch_enter_lazy_mmu_mode(void)
 {
-	struct ppc64_tlb_batch *batch;
 	if (radix_enabled())
 		return;
 	/*
 	 * operating on kernel page tables.
 	 */
 	preempt_disable();
-	batch = this_cpu_ptr(&ppc64_tlb_batch);
-	batch->active = 1;
 }
-static inline void arch_leave_lazy_mmu_mode(void)
+static inline void arch_flush_lazy_mmu_mode(void)
 {
 	struct ppc64_tlb_batch *batch;
 	if (batch->index)
 		__flush_tlb_pending(batch);
-	batch->active = 0;
-	preempt_enable();
 }
-#define arch_flush_lazy_mmu_mode()      do {} while (0)
+static inline void arch_leave_lazy_mmu_mode(void)
+{
+	if (radix_enabled())
+		return;
+	arch_flush_lazy_mmu_mode();
+	preempt_enable();
+}
 extern void hash__tlbiel_all(unsigned int action);

arch/powerpc/include/asm/hugetlb.h+0−5

 			       unsigned long addr, pte_t *ptep,
 			       pte_t pte, int dirty);
-void gigantic_hugetlb_cma_reserve(void) __init;
 #include <asm-generic/hugetlb.h>
 #else /* ! CONFIG_HUGETLB_PAGE */
 {
 }
-static inline void __init gigantic_hugetlb_cma_reserve(void)
-{
-}
 static inline void __init hugetlbpage_init_defaultsize(void)
 {
 }

arch/powerpc/include/asm/nohash/pgtable.h+12−1

 #ifndef __ASSEMBLER__
+#include <linux/page_table_check.h>
 extern int icache_44x_need_flush;
 #ifndef pte_huge_size
 static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr,
 				       pte_t *ptep)
 {
-	return __pte(pte_update(mm, addr, ptep, ~0UL, 0, 0));
+	pte_t old_pte = __pte(pte_update(mm, addr, ptep, ~0UL, 0, 0));
+	page_table_check_pte_clear(mm, addr, old_pte);
+	return old_pte;
 }
 #define __HAVE_ARCH_PTEP_GET_AND_CLEAR
 	return true;
 }
+static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr)
+{
+	return pte_present(pte) && !is_kernel_addr(addr);
+}
 /* Conversion functions: convert a page and protection to a page entry,
  * and a page entry and page directory to the page they refer to.
  *

arch/powerpc/include/asm/page.h+1−0

@@ -271,6 +271,7 @@ static inline const void *pfn_to_kaddr(unsigned long pfn) · 270 unmodified lines
271		271
272	struct page;	272	struct page;
273	extern void clear_user_page(void page, unsigned long vaddr, struct page pg);	273	extern void clear_user_page(void page, unsigned long vaddr, struct page pg);
		274	#define clear_user_page clear_user_page
274	extern void copy_user_page(void to, void from, unsigned long vaddr,	275	extern void copy_user_page(void to, void from, unsigned long vaddr,
275	struct page *p);	276	struct page *p);
276	extern int devmem_is_allowed(unsigned long pfn);	277	extern int devmem_is_allowed(unsigned long pfn);

arch/powerpc/include/asm/pgtable.h+10−0

 void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep,
 		pte_t pte, unsigned int nr);
 #define set_ptes set_ptes
+void set_pte_at_unchecked(struct mm_struct *mm, unsigned long addr,
+			  pte_t *ptep, pte_t pte);
 #define update_mmu_cache(vma, addr, ptep) \
 	update_mmu_cache_range(NULL, vma, addr, ptep, 1)
 #endif /* CONFIG_PPC64 */
+#ifndef pmd_user_accessible_page
+#define pmd_user_accessible_page(pmd, addr)	false
+#endif
+#ifndef pud_user_accessible_page
+#define pud_user_accessible_page(pud, addr)	false
+#endif
 #endif /* __ASSEMBLER__ */
 #endif /* _ASM_POWERPC_PGTABLE_H */

arch/powerpc/include/asm/setup.h+4−0

 void check_for_initrd(void);
 void mem_topology_setup(void);
+#ifdef CONFIG_NUMA
 void initmem_init(void);
+#else
+static inline void initmem_init(void) {}
+#endif
 void setup_panic(void);
 #define ARCH_PANIC_TIMEOUT 180

arch/powerpc/include/asm/thread_info.h+0−2

 /* Don't move TLF_NAPPING without adjusting the code in entry_32.S */
 #define TLF_NAPPING		0	/* idle thread enabled NAP mode */
 #define TLF_SLEEPING		1	/* suspend code enabled SLEEP mode */
-#define TLF_LAZY_MMU		3	/* tlb_batch is active */
 #define TLF_RUNLATCH		4	/* Is the runlatch enabled? */
 #define _TLF_NAPPING		(1 << TLF_NAPPING)
 #define _TLF_SLEEPING		(1 << TLF_SLEEPING)
-#define _TLF_LAZY_MMU		(1 << TLF_LAZY_MMU)
 #define _TLF_RUNLATCH		(1 << TLF_RUNLATCH)
 #ifndef __ASSEMBLER__

arch/powerpc/include/asm/tlb.h+0−1

@@ -37,7 +37,6 @@ extern void tlb_flush(struct mmu_gather *tlb); · 36 unmodified lines
37	*/	37	*/
38	#define tlb_needs_table_invalidate() radix_enabled()	38	#define tlb_needs_table_invalidate() radix_enabled()
39		39
40	#define __HAVE_ARCH_TLB_REMOVE_TABLE
41	/* Get the generic bits... */	40	/* Get the generic bits... */
42	#include <asm-generic/tlb.h>	41	#include <asm-generic/tlb.h>
43		42

arch/powerpc/kernel/process.c+0−25

@@ -1281,9 +1281,6 @@ struct task_struct __switch_to(struct task_struct prev, · 1280 unmodified lines
1281	{	1281	{
1282	struct thread_struct new_thread, old_thread;	1282	struct thread_struct new_thread, old_thread;
1283	struct task_struct *last;	1283	struct task_struct *last;
1284	#ifdef CONFIG_PPC_64S_HASH_MMU
1285	struct ppc64_tlb_batch *batch;
1286	#endif
1287		1284
1288	new_thread = &new->thread;	1285	new_thread = &new->thread;
1289	old_thread = &current->thread;	1286	old_thread = &current->thread;
@@ -1291,14 +1288,6 @@ struct task_struct __switch_to(struct task_struct prev, · 1 unmodified line
1291	WARN_ON(!irqs_disabled());	1288	WARN_ON(!irqs_disabled());
1292		1289
1293	#ifdef CONFIG_PPC_64S_HASH_MMU	1290	#ifdef CONFIG_PPC_64S_HASH_MMU
1294	batch = this_cpu_ptr(&ppc64_tlb_batch);
1295	if (batch->active) {
1296	current_thread_info()->local_flags \|= _TLF_LAZY_MMU;
1297	if (batch->index)
1298	__flush_tlb_pending(batch);
1299	batch->active = 0;
1300	}
1301
1302	/*	1291	/*
1303	* On POWER9 the copy-paste buffer can only paste into	1292	* On POWER9 the copy-paste buffer can only paste into
1304	* foreign real addresses, so unprivileged processes can not	1293	* foreign real addresses, so unprivileged processes can not
@@ -1369,20 +1358,6 @@ struct task_struct __switch_to(struct task_struct prev, · 64 unmodified lines
1369	*/	1358	*/
1370		1359
1371	#ifdef CONFIG_PPC_BOOK3S_64	1360	#ifdef CONFIG_PPC_BOOK3S_64
1372	#ifdef CONFIG_PPC_64S_HASH_MMU
1373	/*
1374	* This applies to a process that was context switched while inside
1375	* arch_enter_lazy_mmu_mode(), to re-activate the batch that was
1376	* deactivated above, before _switch(). This will never be the case
1377	* for new tasks.
1378	*/
1379	if (current_thread_info()->local_flags & _TLF_LAZY_MMU) {
1380	current_thread_info()->local_flags &= ~_TLF_LAZY_MMU;
1381	batch = this_cpu_ptr(&ppc64_tlb_batch);
1382	batch->active = 1;
1383	}
1384	#endif
1385
1386	/*	1361	/*
1387	* Math facilities are masked out of the child MSR in copy_thread.	1362	* Math facilities are masked out of the child MSR in copy_thread.
1388	* A new task does not need to restore_math because it will	1363	* A new task does not need to restore_math because it will

arch/powerpc/kernel/setup-common.c+0−1

@@ -1003,7 +1003,6 @@ void __init setup_arch(char **cmdline_p) · 1002 unmodified lines
1003	fadump_cma_init();	1003	fadump_cma_init();
1004	kdump_cma_reserve();	1004	kdump_cma_reserve();
1005	kvm_cma_reserve();	1005	kvm_cma_reserve();
1006	gigantic_hugetlb_cma_reserve();
1007		1006
1008	early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT);	1007	early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT);
1009		1008

arch/powerpc/mm/book3s64/hash_pgtable.c+4−0

 #include <linux/sched.h>
 #include <linux/mm_types.h>
 #include <linux/mm.h>
+#include <linux/page_table_check.h>
 #include <linux/stop_machine.h>
 #include <asm/sections.h>
 	pmd = *pmdp;
 	pmd_clear(pmdp);
+	page_table_check_pmd_clear(vma->vm_mm, address, pmd);
 	/*
 	 * Wait for all pending hash_page to finish. This is needed
 	 * in case of subpage collapse. When we collapse normal pages

arch/powerpc/mm/book3s64/hash_tlb.c+8−6

@@ -25,11 +25,12 @@ · 24 unmodified lines
25	#include <asm/tlb.h>	25	#include <asm/tlb.h>
26	#include <asm/bug.h>	26	#include <asm/bug.h>
27	#include <asm/pte-walk.h>	27	#include <asm/pte-walk.h>
28		28	#include <kunit/visibility.h>
29		29
30	#include <trace/events/thp.h>	30	#include <trace/events/thp.h>
31		31
32	DEFINE_PER_CPU(struct ppc64_tlb_batch, ppc64_tlb_batch);	32	DEFINE_PER_CPU(struct ppc64_tlb_batch, ppc64_tlb_batch);
		33	EXPORT_SYMBOL_IF_KUNIT(ppc64_tlb_batch);
33		34
34	/*	35	/*
35	* A linux PTE was changed and the corresponding hash table entry	36	* A linux PTE was changed and the corresponding hash table entry
@@ -100,7 +101,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 64 unmodified lines
100	* Check if we have an active batch on this CPU. If not, just	101	* Check if we have an active batch on this CPU. If not, just
101	* flush now and return.	102	* flush now and return.
102	*/	103	*/
103	if (!batch->active) {	104	if (!is_lazy_mmu_mode_active()) {
104	flush_hash_page(vpn, rpte, psize, ssize, mm_is_thread_local(mm));	105	flush_hash_page(vpn, rpte, psize, ssize, mm_is_thread_local(mm));
105	put_cpu_var(ppc64_tlb_batch);	106	put_cpu_var(ppc64_tlb_batch);
106	return;	107	return;
@@ -154,6 +155,7 @@ void __flush_tlb_pending(struct ppc64_tlb_batch *batch) · 47 unmodified lines
154	flush_hash_range(i, local);	155	flush_hash_range(i, local);
155	batch->index = 0;	156	batch->index = 0;
156	}	157	}
		158	EXPORT_SYMBOL_IF_KUNIT(__flush_tlb_pending);
157		159
158	void hash__tlb_flush(struct mmu_gather *tlb)	160	void hash__tlb_flush(struct mmu_gather *tlb)
159	{	161	{
@@ -205,7 +207,7 @@ void __flush_hash_table_range(unsigned long start, unsigned long end) · 45 unmodified lines
205	* way to do things but is fine for our needs here.	207	* way to do things but is fine for our needs here.
206	*/	208	*/
207	local_irq_save(flags);	209	local_irq_save(flags);
208	arch_enter_lazy_mmu_mode();	210	lazy_mmu_mode_enable();
209	for (; start < end; start += PAGE_SIZE) {	211	for (; start < end; start += PAGE_SIZE) {
210	pte_t *ptep = find_init_mm_pte(start, &hugepage_shift);	212	pte_t *ptep = find_init_mm_pte(start, &hugepage_shift);
211	unsigned long pte;	213	unsigned long pte;
@@ -217,7 +219,7 @@ void __flush_hash_table_range(unsigned long start, unsigned long end) · 5 unmodified lines
217	continue;	219	continue;
218	hpte_need_flush(&init_mm, start, ptep, pte, hugepage_shift);	220	hpte_need_flush(&init_mm, start, ptep, pte, hugepage_shift);
219	}	221	}
220	arch_leave_lazy_mmu_mode();	222	lazy_mmu_mode_disable();
221	local_irq_restore(flags);	223	local_irq_restore(flags);
222	}	224	}
223		225
@@ -237,7 +239,7 @@ void flush_hash_table_pmd_range(struct mm_struct mm, pmd_t pmd, unsigned long · 13 unmodified lines
237	* way to do things but is fine for our needs here.	239	* way to do things but is fine for our needs here.
238	*/	240	*/
239	local_irq_save(flags);	241	local_irq_save(flags);
240	arch_enter_lazy_mmu_mode();	242	lazy_mmu_mode_enable();
241	start_pte = pte_offset_map(pmd, addr);	243	start_pte = pte_offset_map(pmd, addr);
242	if (!start_pte)	244	if (!start_pte)
243	goto out;	245	goto out;
@@ -249,6 +251,6 @@ void flush_hash_table_pmd_range(struct mm_struct mm, pmd_t pmd, unsigned long · 5 unmodified lines
249	}	251	}
250	pte_unmap(start_pte);	252	pte_unmap(start_pte);
251	out:	253	out:
252	arch_leave_lazy_mmu_mode();	254	lazy_mmu_mode_disable();
253	local_irq_restore(flags);	255	local_irq_restore(flags);
254	}	256	}

arch/powerpc/mm/book3s64/pgtable.c+16−9

 #include <linux/pkeys.h>
 #include <linux/debugfs.h>
 #include <linux/proc_fs.h>
+#include <linux/page_table_check.h>
 #include <asm/pgalloc.h>
 #include <asm/tlb.h>
 	WARN_ON(!(pmd_leaf(pmd)));
 #endif
 	trace_hugepage_set_pmd(addr, pmd_val(pmd));
-	return set_pte_at(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd));
+	page_table_check_pmd_set(mm, addr, pmdp, pmd);
+	return set_pte_at_unchecked(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd));
 }
 void set_pud_at(struct mm_struct *mm, unsigned long addr,
 	WARN_ON(!(pud_leaf(pud)));
 #endif
 	trace_hugepage_set_pud(addr, pud_val(pud));
-	return set_pte_at(mm, addr, pudp_ptep(pudp), pud_pte(pud));
+	page_table_check_pud_set(mm, addr, pudp, pud);
+	return set_pte_at_unchecked(mm, addr, pudp_ptep(pudp), pud_pte(pud));
 }
 static void do_serialize(void *arg)
 pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
 		     pmd_t *pmdp)
 {
-	unsigned long old_pmd;
+	pmd_t old_pmd;
 	VM_WARN_ON_ONCE(!pmd_present(*pmdp));
-	old_pmd = pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, _PAGE_INVALID);
+	old_pmd = __pmd(pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, _PAGE_INVALID));
 	flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
-	return __pmd(old_pmd);
+	page_table_check_pmd_clear(vma->vm_mm, address, old_pmd);
+	return old_pmd;
 }
 pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address,
 		      pud_t *pudp)
 {
-	unsigned long old_pud;
+	pud_t old_pud;
 	VM_WARN_ON_ONCE(!pud_present(*pudp));
-	old_pud = pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID);
+	old_pud = __pud(pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID));
 	flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE);
-	return __pud(old_pud);
+	page_table_check_pud_clear(vma->vm_mm, address, old_pud);
+	return old_pud;
 }
 pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
 	if (radix_enabled())
 		return radix__ptep_modify_prot_commit(vma, addr,
 						      ptep, old_pte, pte);
-	set_pte_at(vma->vm_mm, addr, ptep, pte);
+	set_pte_at_unchecked(vma->vm_mm, addr, ptep, pte);
 }
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE

arch/powerpc/mm/book3s64/radix_pgtable.c+6−3

 #include <linux/of.h>
 #include <linux/of_fdt.h>
 #include <linux/mm.h>
+#include <linux/page_table_check.h>
 #include <linux/hugetlb.h>
 #include <linux/string_helpers.h>
 #include <linux/memory.h>
 	pmd = *pmdp;
 	pmd_clear(pmdp);
+	page_table_check_pmd_clear(vma->vm_mm, address, pmd);
 	radix__flush_tlb_collapsed_pmd(vma->vm_mm, address);
 	return pmd;
 	    (atomic_read(&mm->context.copros) > 0))
 		radix__flush_tlb_page(vma, addr);
-	set_pte_at(mm, addr, ptep, pte);
+	set_pte_at_unchecked(mm, addr, ptep, pte);
 }
 int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot)
 	if (!radix_enabled())
 		return 0;
-	set_pte_at(&init_mm, 0 /* radix unused */, ptep, new_pud);
+	set_pte_at_unchecked(&init_mm, 0 /* radix unused */, ptep, new_pud);
 	return 1;
 }
 	if (!radix_enabled())
 		return 0;
-	set_pte_at(&init_mm, 0 /* radix unused */, ptep, new_pmd);
+	set_pte_at_unchecked(&init_mm, 0 /* radix unused */, ptep, new_pmd);
 	return 1;
 }

arch/powerpc/mm/book3s64/subpage_prot.c+2−2

@@ -73,13 +73,13 @@ static void hpte_flush_range(struct mm_struct *mm, unsigned long addr, · 72 unmodified lines
73	pte = pte_offset_map_lock(mm, pmd, addr, &ptl);	73	pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
74	if (!pte)	74	if (!pte)
75	return;	75	return;
76	arch_enter_lazy_mmu_mode();	76	lazy_mmu_mode_enable();
77	for (; npages > 0; --npages) {	77	for (; npages > 0; --npages) {
78	pte_update(mm, addr, pte, 0, 0, 0);	78	pte_update(mm, addr, pte, 0, 0, 0);
79	addr += PAGE_SIZE;	79	addr += PAGE_SIZE;
80	++pte;	80	++pte;
81	}	81	}
82	arch_leave_lazy_mmu_mode();	82	lazy_mmu_mode_disable();
83	pte_unmap_unlock(pte - 1, ptl);	83	pte_unmap_unlock(pte - 1, ptl);
84	}	84	}
85		85

arch/powerpc/mm/hugetlbpage.c+4−7

 arch_initcall(hugetlbpage_init);
-void __init gigantic_hugetlb_cma_reserve(void)
+unsigned int __init arch_hugetlb_cma_order(void)
 {
-	unsigned long order = 0;
 	if (radix_enabled())
-		order = PUD_SHIFT - PAGE_SHIFT;
+		return PUD_SHIFT - PAGE_SHIFT;
 	else if (!firmware_has_feature(FW_FEATURE_LPAR) && mmu_psize_defs[MMU_PAGE_16G].shift)
 		/*
 		 * For pseries we do use ibm,expected#pages for reserving 16G pages.
 		 */
-		order = mmu_psize_to_shift(MMU_PAGE_16G) - PAGE_SHIFT;
+		return mmu_psize_to_shift(MMU_PAGE_16G) - PAGE_SHIFT;
-	if (order)
+	return 0;
-		hugetlb_cma_reserve(order);
 }

arch/powerpc/mm/mem.c+10−17

 	memblock_set_node(0, PHYS_ADDR_MAX, &memblock.memory, 0);
 }
-void __init initmem_init(void)
-{
-	sparse_init();
-}
 /* mark pages that don't exist as nosave */
 static int __init mark_nonram_nosave(void)
 {
  * anyway) will take a first dip into ZONE_NORMAL and get otherwise served by
  * ZONE_DMA.
  */
-static unsigned long max_zone_pfns[MAX_NR_ZONES];
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+#ifdef CONFIG_ZONE_DMA
+	max_zone_pfns[ZONE_DMA] = min((zone_dma_limit >> PAGE_SHIFT) + 1, max_low_pfn);
+#endif
+	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
+#ifdef CONFIG_HIGHMEM
+	max_zone_pfns[ZONE_HIGHMEM] = max_pfn;
+#endif
+}
 /*
  * paging_init() sets up the page tables - in fact we've already done this.
 	zone_dma_limit = DMA_BIT_MASK(zone_dma_bits);
-#ifdef CONFIG_ZONE_DMA
-	max_zone_pfns[ZONE_DMA]	= min(max_low_pfn,
-				      1UL << (zone_dma_bits - PAGE_SHIFT));
-#endif
-	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
-#ifdef CONFIG_HIGHMEM
-	max_zone_pfns[ZONE_HIGHMEM] = max_pfn;
-#endif
-	free_area_init(max_zone_pfns);
 	mark_nonram_nosave();
 }

arch/powerpc/mm/numa.c+0−2

@@ -1213,8 +1213,6 @@ void __init initmem_init(void) · 1212 unmodified lines
1213	setup_node_data(nid, start_pfn, end_pfn);	1213	setup_node_data(nid, start_pfn, end_pfn);
1214	}	1214	}
1215		1215
1216	sparse_init();
1217
1218	/*	1216	/*
1219	* We need the numa_cpu_lookup_table to be accurate for all CPUs,	1217	* We need the numa_cpu_lookup_table to be accurate for all CPUs,
1220	* even before we online them, so that we can use cpu_to_{node,mem}	1218	* even before we online them, so that we can use cpu_to_{node,mem}

arch/powerpc/mm/pgtable.c+12−0

 #include <linux/mm.h>
 #include <linux/percpu.h>
 #include <linux/hardirq.h>
+#include <linux/page_table_check.h>
 #include <linux/hugetlb.h>
 #include <asm/tlbflush.h>
 #include <asm/tlb.h>
 	 * and not hw_valid ptes. Hence there is no translation cache flush
 	 * involved that need to be batched.
 	 */
+	page_table_check_ptes_set(mm, addr, ptep, pte, nr);
 	for (;;) {
 		/*
 	}
 }
+void set_pte_at_unchecked(struct mm_struct *mm, unsigned long addr,
+			  pte_t *ptep, pte_t pte)
+{
+	VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep));
+	pte = set_pte_filter(pte, addr);
+	__set_pte_at(mm, addr, ptep, pte, 0);
+}
 void unmap_kernel_page(unsigned long va)
 {
 	pmd_t *pmdp = pmd_off_k(va);

arch/powerpc/platforms/Kconfig.cputype+1−0

@@ -93,6 +93,7 @@ config PPC_BOOK3S_64 · 92 unmodified lines
93	select IRQ_WORK	93	select IRQ_WORK
94	select PPC_64S_HASH_MMU if !PPC_RADIX_MMU	94	select PPC_64S_HASH_MMU if !PPC_RADIX_MMU
95	select KASAN_VMALLOC if KASAN	95	select KASAN_VMALLOC if KASAN
		96	select ARCH_HAS_LAZY_MMU_MODE
96		97
97	config PPC_BOOK3E_64	98	config PPC_BOOK3E_64
98	bool "Embedded processors"	99	bool "Embedded processors"

arch/powerpc/platforms/pseries/Kconfig+1−1

@@ -120,7 +120,7 @@ config PPC_SMLPAR · 119 unmodified lines
120	config CMM	120	config CMM
121	tristate "Collaborative memory management"	121	tristate "Collaborative memory management"
122	depends on PPC_SMLPAR	122	depends on PPC_SMLPAR
123	select MEMORY_BALLOON	123	select BALLOON
124	default y	124	default y
125	help	125	help
126	Select this option, if you want to enable the kernel interface	126	Select this option, if you want to enable the kernel interface

arch/powerpc/platforms/pseries/cmm.c+10−43

@@ -19,7 +19,7 @@ · 18 unmodified lines
19	#include <linux/stringify.h>	19	#include <linux/stringify.h>
20	#include <linux/swap.h>	20	#include <linux/swap.h>
21	#include <linux/device.h>	21	#include <linux/device.h>
22	#include <linux/balloon_compaction.h>	22	#include <linux/balloon.h>
23	#include <asm/firmware.h>	23	#include <asm/firmware.h>
24	#include <asm/hvcall.h>	24	#include <asm/hvcall.h>
25	#include <asm/mmu.h>	25	#include <asm/mmu.h>
@@ -165,7 +165,6 @@ static long cmm_alloc_pages(long nr) · 139 unmodified lines
165		165
166	balloon_page_enqueue(&b_dev_info, page);	166	balloon_page_enqueue(&b_dev_info, page);
167	atomic_long_inc(&loaned_pages);	167	atomic_long_inc(&loaned_pages);
168	adjust_managed_page_count(page, -1);
169	nr--;	168	nr--;
170	}	169	}
171		170
@@ -190,7 +189,6 @@ static long cmm_free_pages(long nr) · 18 unmodified lines
190	if (!page)	189	if (!page)
191	break;	190	break;
192	plpar_page_set_active(page);	191	plpar_page_set_active(page);
193	adjust_managed_page_count(page, 1);
194	__free_page(page);	192	__free_page(page);
195	atomic_long_dec(&loaned_pages);	193	atomic_long_dec(&loaned_pages);
196	nr--;	194	nr--;
@@ -496,13 +494,11 @@ static struct notifier_block cmm_mem_nb = { · 299 unmodified lines
496	.priority = CMM_MEM_HOTPLUG_PRI	494	.priority = CMM_MEM_HOTPLUG_PRI
497	};	495	};
498		496
499	#ifdef CONFIG_BALLOON_COMPACTION	497	#ifdef CONFIG_BALLOON_MIGRATION
500	static int cmm_migratepage(struct balloon_dev_info *b_dev_info,	498	static int cmm_migratepage(struct balloon_dev_info *b_dev_info,
501	struct page newpage, struct page page,	499	struct page newpage, struct page page,
502	enum migrate_mode mode)	500	enum migrate_mode mode)
503	{	501	{
504	unsigned long flags;
505
506	/*	502	/*
507	* loan/"inflate" the newpage first.	503	* loan/"inflate" the newpage first.
508	*	504	*
@@ -517,47 +513,17 @@ static int cmm_migratepage(struct balloon_dev_info *b_dev_info, · 8 unmodified lines
517	return -EBUSY;	513	return -EBUSY;
518	}	514	}
519		515
520	/* balloon page list reference */
521	get_page(newpage);
522
523	/*
524	* When we migrate a page to a different zone, we have to fixup the
525	* count of both involved zones as we adjusted the managed page count
526	* when inflating.
527	*/
528	if (page_zone(page) != page_zone(newpage)) {
529	adjust_managed_page_count(page, 1);
530	adjust_managed_page_count(newpage, -1);
531	}
532
533	spin_lock_irqsave(&b_dev_info->pages_lock, flags);
534	balloon_page_insert(b_dev_info, newpage);
535	__count_vm_event(BALLOON_MIGRATE);
536	b_dev_info->isolated_pages--;
537	spin_unlock_irqrestore(&b_dev_info->pages_lock, flags);
538
539	/*	516	/*
540	* activate/"deflate" the old page. We ignore any errors just like the	517	* activate/"deflate" the old page. We ignore any errors just like the
541	* other callers.	518	* other callers.
542	*/	519	*/
543	plpar_page_set_active(page);	520	plpar_page_set_active(page);
544
545	balloon_page_finalize(page);
546	/* balloon page list reference */
547	put_page(page);
548
549	return 0;	521	return 0;
550	}	522	}
551		523	#else /* CONFIG_BALLOON_MIGRATION */
552	static void cmm_balloon_compaction_init(void)	524	int cmm_migratepage(struct balloon_dev_info b_dev_info, struct page newpage,
553	{	525	struct page *page, enum migrate_mode mode);
554	b_dev_info.migratepage = cmm_migratepage;	526	#endif /* CONFIG_BALLOON_MIGRATION */
555	}
556	#else /* CONFIG_BALLOON_COMPACTION */
557	static void cmm_balloon_compaction_init(void)
558	{
559	}
560	#endif /* CONFIG_BALLOON_COMPACTION */
561		527
562	/**	528	/**
563	* cmm_init - Module initialization	529	* cmm_init - Module initialization
@@ -573,11 +539,13 @@ static int cmm_init(void) · 9 unmodified lines
573	return -EOPNOTSUPP;	539	return -EOPNOTSUPP;
574		540
575	balloon_devinfo_init(&b_dev_info);	541	balloon_devinfo_init(&b_dev_info);
576	cmm_balloon_compaction_init();	542	b_dev_info.adjust_managed_page_count = true;
		543	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
		544	b_dev_info.migratepage = cmm_migratepage;
577		545
578	rc = register_oom_notifier(&cmm_oom_nb);	546	rc = register_oom_notifier(&cmm_oom_nb);
579	if (rc < 0)	547	if (rc < 0)
580	goto out_balloon_compaction;	548	return rc;
581		549
582	if ((rc = register_reboot_notifier(&cmm_reboot_nb)))	550	if ((rc = register_reboot_notifier(&cmm_reboot_nb)))
583	goto out_oom_notifier;	551	goto out_oom_notifier;
@@ -606,7 +574,6 @@ out_reboot_notifier: · 22 unmodified lines
606	unregister_reboot_notifier(&cmm_reboot_nb);	574	unregister_reboot_notifier(&cmm_reboot_nb);
607	out_oom_notifier:	575	out_oom_notifier:
608	unregister_oom_notifier(&cmm_oom_nb);	576	unregister_oom_notifier(&cmm_oom_nb);
609	out_balloon_compaction:
610	return rc;	577	return rc;
611	}	578	}
612		579

arch/riscv/include/asm/page.h+0−1

@@ -50,7 +50,6 @@ void clear_page(void *page); · 49 unmodified lines
50	#endif	50	#endif
51	#define copy_page(to, from) memcpy((to), (from), PAGE_SIZE)	51	#define copy_page(to, from) memcpy((to), (from), PAGE_SIZE)
52		52
53	#define clear_user_page(pgaddr, vaddr, page) clear_page(pgaddr)
54	#define copy_user_page(vto, vfrom, vaddr, topg) \	53	#define copy_user_page(vto, vfrom, vaddr, topg) \
55	memcpy((vto), (vfrom), PAGE_SIZE)	54	memcpy((vto), (vfrom), PAGE_SIZE)
56		55

arch/riscv/include/asm/pgtable.h+11−11

@@ -627,7 +627,7 @@ static inline void __set_pte_at(struct mm_struct mm, pte_t ptep, pte_t pteval) · 626 unmodified lines
627	static inline void set_ptes(struct mm_struct *mm, unsigned long addr,	627	static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
628	pte_t *ptep, pte_t pteval, unsigned int nr)	628	pte_t *ptep, pte_t pteval, unsigned int nr)
629	{	629	{
630	page_table_check_ptes_set(mm, ptep, pteval, nr);	630	page_table_check_ptes_set(mm, addr, ptep, pteval, nr);
631		631
632	for (;;) {	632	for (;;) {
633	__set_pte_at(mm, ptep, pteval);	633	__set_pte_at(mm, ptep, pteval);
@@ -664,7 +664,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · 30 unmodified lines
664	set_pte(ptep, __pte(0));	664	set_pte(ptep, __pte(0));
665	#endif	665	#endif
666		666
667	page_table_check_pte_clear(mm, pte);	667	page_table_check_pte_clear(mm, address, pte);
668		668
669	return pte;	669	return pte;
670	}	670	}
@@ -946,29 +946,29 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) · 275 unmodified lines
946	static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,	946	static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
947	pmd_t *pmdp, pmd_t pmd)	947	pmd_t *pmdp, pmd_t pmd)
948	{	948	{
949	page_table_check_pmd_set(mm, pmdp, pmd);	949	page_table_check_pmd_set(mm, addr, pmdp, pmd);
950	return __set_pte_at(mm, (pte_t *)pmdp, pmd_pte(pmd));	950	return __set_pte_at(mm, (pte_t *)pmdp, pmd_pte(pmd));
951	}	951	}
952		952
953	static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,	953	static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
954	pud_t *pudp, pud_t pud)	954	pud_t *pudp, pud_t pud)
955	{	955	{
956	page_table_check_pud_set(mm, pudp, pud);	956	page_table_check_pud_set(mm, addr, pudp, pud);
957	return __set_pte_at(mm, (pte_t *)pudp, pud_pte(pud));	957	return __set_pte_at(mm, (pte_t *)pudp, pud_pte(pud));
958	}	958	}
959		959
960	#ifdef CONFIG_PAGE_TABLE_CHECK	960	#ifdef CONFIG_PAGE_TABLE_CHECK
961	static inline bool pte_user_accessible_page(pte_t pte)	961	static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr)
962	{	962	{
963	return pte_present(pte) && pte_user(pte);	963	return pte_present(pte) && pte_user(pte);
964	}	964	}
965		965
966	static inline bool pmd_user_accessible_page(pmd_t pmd)	966	static inline bool pmd_user_accessible_page(pmd_t pmd, unsigned long addr)
967	{	967	{
968	return pmd_leaf(pmd) && pmd_user(pmd);	968	return pmd_leaf(pmd) && pmd_user(pmd);
969	}	969	}
970		970
971	static inline bool pud_user_accessible_page(pud_t pud)	971	static inline bool pud_user_accessible_page(pud_t pud, unsigned long addr)
972	{	972	{
973	return pud_leaf(pud) && pud_user(pud);	973	return pud_leaf(pud) && pud_user(pud);
974	}	974	}
@@ -1007,7 +1007,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, · 32 unmodified lines
1007	pmd_clear(pmdp);	1007	pmd_clear(pmdp);
1008	#endif	1008	#endif
1009		1009
1010	page_table_check_pmd_clear(mm, pmd);	1010	page_table_check_pmd_clear(mm, address, pmd);
1011		1011
1012	return pmd;	1012	return pmd;
1013	}	1013	}
@@ -1023,7 +1023,7 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, · 9 unmodified lines
1023	static inline pmd_t pmdp_establish(struct vm_area_struct *vma,	1023	static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
1024	unsigned long address, pmd_t *pmdp, pmd_t pmd)	1024	unsigned long address, pmd_t *pmdp, pmd_t pmd)
1025	{	1025	{
1026	page_table_check_pmd_set(vma->vm_mm, pmdp, pmd);	1026	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
1027	return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd)));	1027	return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd)));
1028	}	1028	}
1029		1029
@@ -1101,7 +1101,7 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, · 71 unmodified lines
1101	pud_clear(pudp);	1101	pud_clear(pudp);
1102	#endif	1102	#endif
1103		1103
1104	page_table_check_pud_clear(mm, pud);	1104	page_table_check_pud_clear(mm, address, pud);
1105		1105
1106	return pud;	1106	return pud;
1107	}	1107	}
@@ -1122,7 +1122,7 @@ static inline void update_mmu_cache_pud(struct vm_area_struct *vma, · 14 unmodified lines
1122	static inline pud_t pudp_establish(struct vm_area_struct *vma,	1122	static inline pud_t pudp_establish(struct vm_area_struct *vma,
1123	unsigned long address, pud_t *pudp, pud_t pud)	1123	unsigned long address, pud_t *pudp, pud_t pud)
1124	{	1124	{
1125	page_table_check_pud_set(vma->vm_mm, pudp, pud);	1125	page_table_check_pud_set(vma->vm_mm, address, pudp, pud);
1126	return __pud(atomic_long_xchg((atomic_long_t *)pudp, pud_val(pud)));	1126	return __pud(atomic_long_xchg((atomic_long_t *)pudp, pud_val(pud)));
1127	}	1127	}
1128		1128

arch/riscv/mm/hugetlbpage.c+8−0

@@ -447,3 +447,11 @@ static __init int gigantic_pages_init(void) · 446 unmodified lines
447	}	447	}
448	arch_initcall(gigantic_pages_init);	448	arch_initcall(gigantic_pages_init);
449	#endif	449	#endif
		450
		451	unsigned int __init arch_hugetlb_cma_order(void)
		452	{
		453	if (IS_ENABLED(CONFIG_64BIT))
		454	return PUD_SHIFT - PAGE_SHIFT;
		455
		456	return 0;
		457	}

arch/riscv/mm/init.c+1−9

 phys_addr_t dma32_phys_limit __initdata;
-static void __init zone_sizes_init(void)
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
 {
-	unsigned long max_zone_pfns[MAX_NR_ZONES] = { 0, };
 #ifdef CONFIG_ZONE_DMA32
 	max_zone_pfns[ZONE_DMA32] = PFN_DOWN(dma32_phys_limit);
 #endif
 	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
-	free_area_init(max_zone_pfns);
 }
 #if defined(CONFIG_MMU) && defined(CONFIG_DEBUG_VM)
 		memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va));
 	dma_contiguous_reserve(dma32_phys_limit);
-	if (IS_ENABLED(CONFIG_64BIT))
-		hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
 }
 #ifdef CONFIG_RELOCATABLE
 {
 	early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT);
 	arch_numa_init();
-	sparse_init();
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
 	/* The entire VMEMMAP region has been populated. Flush TLB for this region */
 	local_flush_tlb_kernel_range(VMEMMAP_START, VMEMMAP_END);
 #endif
-	zone_sizes_init();
 	arch_reserve_crashkernel();
 	memblock_dump_all();
 }

arch/s390/include/asm/page.h+0−1

@@ -65,7 +65,6 @@ static inline void copy_page(void to, void from) · 64 unmodified lines
65	: : "memory", "cc");	65	: : "memory", "cc");
66	}	66	}
67		67
68	#define clear_user_page(page, vaddr, pg) clear_page(page)
69	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	68	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
70		69
71	#define vma_alloc_zeroed_movable_folio(vma, vaddr) \	70	#define vma_alloc_zeroed_movable_folio(vma, vaddr) \

arch/s390/include/asm/tlb.h+2−4

 static inline void tlb_flush(struct mmu_gather *tlb);
 static inline bool __tlb_remove_page_size(struct mmu_gather *tlb,
-		struct page *page, bool delay_rmap, int page_size);
+		struct page *page, int page_size);
 static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb,
 		struct page *page, unsigned int nr_pages, bool delay_rmap);
  * s390 doesn't delay rmap removal.
  */
 static inline bool __tlb_remove_page_size(struct mmu_gather *tlb,
-		struct page *page, bool delay_rmap, int page_size)
+		struct page *page, int page_size)
 {
-	VM_WARN_ON_ONCE(delay_rmap);
 	free_folio_and_swap_cache(page_folio(page));
 	return false;
 }

arch/s390/kernel/setup.c+0−2

@@ -963,8 +963,6 @@ void __init setup_arch(char **cmdline_p) · 962 unmodified lines
963	setup_uv();	963	setup_uv();
964	dma_contiguous_reserve(ident_map_size);	964	dma_contiguous_reserve(ident_map_size);
965	vmcp_cma_reserve();	965	vmcp_cma_reserve();
966	if (cpu_has_edat2())
967	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
968		966
969	reserve_crashkernel();	967	reserve_crashkernel();
970	#ifdef CONFIG_CRASH_DUMP	968	#ifdef CONFIG_CRASH_DUMP

arch/s390/mm/gmap_helpers.c+1−1

@@ -32,7 +32,7 @@ static void ptep_zap_softleaf_entry(struct mm_struct *mm, softleaf_t entry) · 31 unmodified lines
32	dec_mm_counter(mm, MM_SWAPENTS);	32	dec_mm_counter(mm, MM_SWAPENTS);
33	else if (softleaf_is_migration(entry))	33	else if (softleaf_is_migration(entry))
34	dec_mm_counter(mm, mm_counter(softleaf_to_folio(entry)));	34	dec_mm_counter(mm, mm_counter(softleaf_to_folio(entry)));
35	free_swap_and_cache(entry);	35	swap_put_entries_direct(entry, 1);
36	}	36	}
37		37
38	/**	38	/**

arch/s390/mm/hugetlbpage.c+8−0

@@ -255,3 +255,11 @@ bool __init arch_hugetlb_valid_size(unsigned long size) · 254 unmodified lines
255	else	255	else
256	return false;	256	return false;
257	}	257	}
		258
		259	unsigned int __init arch_hugetlb_cma_order(void)
		260	{
		261	if (cpu_has_edat2())
		262	return PUD_SHIFT - PAGE_SHIFT;
		263
		264	return 0;
		265	}

arch/s390/mm/init.c+6−7

 	zero_page_mask = ((PAGE_SIZE << order) - 1) & PAGE_MASK;
 }
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+	max_zone_pfns[ZONE_DMA] = virt_to_pfn(MAX_DMA_ADDRESS);
+	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
+}
 /*
  * paging_init() sets up the page tables
  */
 void __init paging_init(void)
 {
-	unsigned long max_zone_pfns[MAX_NR_ZONES];
 	vmem_map_init();
-	sparse_init();
 	zone_dma_limit = DMA_BIT_MASK(31);
-	memset(max_zone_pfns, 0, sizeof(max_zone_pfns));
-	max_zone_pfns[ZONE_DMA] = virt_to_pfn(MAX_DMA_ADDRESS);
-	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
-	free_area_init(max_zone_pfns);
 }
 void mark_rodata_ro(void)

arch/s390/mm/pgtable.c+1−1

@@ -682,7 +682,7 @@ static void ptep_zap_softleaf_entry(struct mm_struct *mm, softleaf_t entry) · 681 unmodified lines
682		682
683	dec_mm_counter(mm, mm_counter(folio));	683	dec_mm_counter(mm, mm_counter(folio));
684	}	684	}
685	free_swap_and_cache(entry);	685	swap_put_entries_direct(entry, 1);
686	}	686	}
687		687
688	void ptep_zap_unused(struct mm_struct *mm, unsigned long addr,	688	void ptep_zap_unused(struct mm_struct *mm, unsigned long addr,

arch/sh/mm/init.c+5−7

 	node_set_online(0);
 	plat_mem_setup();
-	sparse_init();
 }
 static void __init early_reserve_mem(void)
 	reserve_crashkernel();
 }
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
+}
 void __init paging_init(void)
 {
-	unsigned long max_zone_pfns[MAX_NR_ZONES];
 	unsigned long vaddr, end;
 	sh_mv.mv_mem_init();
 	page_table_range_init(vaddr, end, swapper_pg_dir);
 	kmap_coherent_init();
-	memset(max_zone_pfns, 0, sizeof(max_zone_pfns));
-	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
-	free_area_init(max_zone_pfns);
 }
 unsigned int mem_init_done = 0;

arch/sparc/Kconfig+2−0

 	select HAVE_KRETPROBES
 	select HAVE_KPROBES
 	select MMU_GATHER_RCU_TABLE_FREE if SMP
+	select HAVE_ARCH_TLB_REMOVE_TABLE if SMP
 	select MMU_GATHER_MERGE_VMAS
 	select MMU_GATHER_NO_FLUSH_CACHE
 	select HAVE_ARCH_TRANSPARENT_HUGEPAGE
 	select NEED_PER_CPU_PAGE_FIRST_CHUNK
 	select ARCH_SUPPORTS_SCHED_SMT if SMP
 	select ARCH_SUPPORTS_SCHED_MC  if SMP
+	select ARCH_HAS_LAZY_MMU_MODE
 config ARCH_PROC_KCORE_TEXT
 	def_bool y

arch/sparc/include/asm/page_64.h+1−0

@@ -43,6 +43,7 @@ void _clear_page(void *page); · 42 unmodified lines
43	#define clear_page(X) _clear_page((void *)(X))	43	#define clear_page(X) _clear_page((void *)(X))
44	struct page;	44	struct page;
45	void clear_user_page(void addr, unsigned long vaddr, struct page page);	45	void clear_user_page(void addr, unsigned long vaddr, struct page page);
		46	#define clear_user_page clear_user_page
46	#define copy_page(X,Y) memcpy((void )(X), (void )(Y), PAGE_SIZE)	47	#define copy_page(X,Y) memcpy((void )(X), (void )(Y), PAGE_SIZE)
47	void copy_user_page(void to, void from, unsigned long vaddr, struct page *topage);	48	void copy_user_page(void to, void from, unsigned long vaddr, struct page *topage);
48	#define __HAVE_ARCH_COPY_USER_HIGHPAGE	49	#define __HAVE_ARCH_COPY_USER_HIGHPAGE

arch/sparc/include/asm/tlb_64.h+0−1

@@ -33,7 +33,6 @@ void flush_tlb_pending(void); · 32 unmodified lines
33	#define tlb_needs_table_invalidate() (false)	33	#define tlb_needs_table_invalidate() (false)
34	#endif	34	#endif
35		35
36	#define __HAVE_ARCH_TLB_REMOVE_TABLE
37	#include <asm-generic/tlb.h>	36	#include <asm-generic/tlb.h>
38		37
39	#endif /* _SPARC64_TLB_H */	38	#endif /* _SPARC64_TLB_H */

arch/sparc/include/asm/tlbflush_64.h+1−4

 	unsigned int hugepage_shift;
 	struct mm_struct *mm;
 	unsigned long tlb_nr;
-	unsigned long active;
 	unsigned long vaddrs[TLB_BATCH_NR];
 };
 void flush_tlb_kernel_range(unsigned long start, unsigned long end);
-#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE
 void flush_tlb_pending(void);
 void arch_enter_lazy_mmu_mode(void);
+void arch_flush_lazy_mmu_mode(void);
 void arch_leave_lazy_mmu_mode(void);
-#define arch_flush_lazy_mmu_mode()      do {} while (0)
 /* Local cpu only.  */
 void __flush_tlb_all(void);

arch/sparc/mm/init_64.c+5−12

 	/* XXX cpu notifier XXX */
-	sparse_init();
 	return end_pfn;
 }
 	memblock_enforce_memory_limit(limit_ram);
 }
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+	max_zone_pfns[ZONE_NORMAL] = last_valid_pfn;
+}
 void __init paging_init(void)
 {
 	unsigned long end_pfn, shift, phys_base;
 	kernel_physical_mapping_init();
-	{
-		unsigned long max_zone_pfns[MAX_NR_ZONES];
-		memset(max_zone_pfns, 0, sizeof(max_zone_pfns));
-		max_zone_pfns[ZONE_NORMAL] = end_pfn;
-		free_area_init(max_zone_pfns);
-	}
 	printk("Booting Linux...\n");
 }

arch/sparc/mm/srmmu.c+7−10

 void (*poke_srmmu)(void) = NULL;
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
+{
+	max_zone_pfns[ZONE_DMA] = max_low_pfn;
+	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
+	max_zone_pfns[ZONE_HIGHMEM] = highend_pfn;
+}
 void __init srmmu_paging_init(void)
 {
 	int i;
 	flush_tlb_all();
 	sparc_context_init(num_contexts);
-	{
-		unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
-		max_zone_pfn[ZONE_DMA] = max_low_pfn;
-		max_zone_pfn[ZONE_NORMAL] = max_low_pfn;
-		max_zone_pfn[ZONE_HIGHMEM] = highend_pfn;
-		free_area_init(max_zone_pfn);
-	}
 }
 void mmu_info(struct seq_file *m)

arch/sparc/mm/tlb.c+13−7

 #include <linux/preempt.h>
 #include <linux/pagemap.h>
+#include <kunit/visibility.h>
 #include <asm/tlbflush.h>
 #include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
 void arch_enter_lazy_mmu_mode(void)
 {
-	struct tlb_batch *tb;
 	preempt_disable();
-	tb = this_cpu_ptr(&tlb_batch);
-	tb->active = 1;
 }
+/* For lazy_mmu_mode KUnit tests */
+EXPORT_SYMBOL_IF_KUNIT(arch_enter_lazy_mmu_mode);
-void arch_leave_lazy_mmu_mode(void)
+void arch_flush_lazy_mmu_mode(void)
 {
 	struct tlb_batch *tb = this_cpu_ptr(&tlb_batch);
 	if (tb->tlb_nr)
 		flush_tlb_pending();
-	tb->active = 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(arch_flush_lazy_mmu_mode);
+void arch_leave_lazy_mmu_mode(void)
+{
+	arch_flush_lazy_mmu_mode();
 	preempt_enable();
 }
+EXPORT_SYMBOL_IF_KUNIT(arch_leave_lazy_mmu_mode);
 static void tlb_batch_add_one(struct mm_struct *mm, unsigned long vaddr,
 			      bool exec, unsigned int hugepage_shift)
 		nr = 0;
 	}
-	if (!tb->active) {
+	if (!is_lazy_mmu_mode_active()) {
 		flush_tsb_user_page(mm, vaddr, hugepage_shift);
 		global_flush_tlb_page(mm, vaddr);
 		goto out;

arch/um/Kconfig+1−0

@@ -42,6 +42,7 @@ config UML · 41 unmodified lines
42	select HAVE_SYSCALL_TRACEPOINTS	42	select HAVE_SYSCALL_TRACEPOINTS
43	select THREAD_INFO_IN_TASK	43	select THREAD_INFO_IN_TASK
44	select SPARSE_IRQ	44	select SPARSE_IRQ
		45	select MMU_GATHER_RCU_TABLE_FREE
45		46
46	config MMU	47	config MMU
47	bool	48	bool

arch/um/include/asm/page.h+0−1

@@ -26,7 +26,6 @@ struct page; · 25 unmodified lines
26	#define clear_page(page) memset((void *)(page), 0, PAGE_SIZE)	26	#define clear_page(page) memset((void *)(page), 0, PAGE_SIZE)
27	#define copy_page(to,from) memcpy((void )(to), (void )(from), PAGE_SIZE)	27	#define copy_page(to,from) memcpy((void )(to), (void )(from), PAGE_SIZE)
28		28
29	#define clear_user_page(page, vaddr, pg) clear_page(page)
30	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	29	#define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
31		30
32	typedef struct { unsigned long pte; } pte_t;	31	typedef struct { unsigned long pte; } pte_t;

arch/um/kernel/mem.c+5−5

 	kmalloc_ok = 1;
 }
-void __init paging_init(void)
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
 {
-	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
+	max_zone_pfns[ZONE_NORMAL] = high_physmem >> PAGE_SHIFT;
+}
+void __init paging_init(void)
+{
 	empty_zero_page = (unsigned long *) memblock_alloc_low(PAGE_SIZE,
 							       PAGE_SIZE);
 	if (!empty_zero_page)
 		panic("%s: Failed to allocate %lu bytes align=%lx\n",
 		      __func__, PAGE_SIZE, PAGE_SIZE);
-	max_zone_pfn[ZONE_NORMAL] = high_physmem >> PAGE_SHIFT;
-	free_area_init(max_zone_pfn);
 }
 /*

arch/x86/Kconfig+1−1

 	select FUNCTION_ALIGNMENT_4B
 	imply IMA_SECURE_AND_OR_TRUSTED_BOOT    if EFI
 	select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
-	select ARCH_SUPPORTS_PT_RECLAIM		if X86_64
 	select ARCH_SUPPORTS_SCHED_SMT		if SMP
 	select SCHED_SMT			if SMP
 	select ARCH_SUPPORTS_SCHED_CLUSTER	if SMP
 config PARAVIRT_XXL
 	bool
 	depends on X86_64
+	select ARCH_HAS_LAZY_MMU_MODE
 config PARAVIRT_SPINLOCKS
 	bool "Paravirtualization layer for spinlocks"

arch/x86/boot/compressed/misc.h+1−0

@@ -11,6 +11,7 @@ · 10 unmodified lines
11	#undef CONFIG_PARAVIRT	11	#undef CONFIG_PARAVIRT
12	#undef CONFIG_PARAVIRT_XXL	12	#undef CONFIG_PARAVIRT_XXL
13	#undef CONFIG_PARAVIRT_SPINLOCKS	13	#undef CONFIG_PARAVIRT_SPINLOCKS
		14	#undef CONFIG_ARCH_HAS_LAZY_MMU_MODE
14	#undef CONFIG_KASAN	15	#undef CONFIG_KASAN
15	#undef CONFIG_KASAN_GENERIC	16	#undef CONFIG_KASAN_GENERIC
16		17

arch/x86/boot/startup/sme.c+1−0

@@ -24,6 +24,7 @@ · 23 unmodified lines
24	#undef CONFIG_PARAVIRT	24	#undef CONFIG_PARAVIRT
25	#undef CONFIG_PARAVIRT_XXL	25	#undef CONFIG_PARAVIRT_XXL
26	#undef CONFIG_PARAVIRT_SPINLOCKS	26	#undef CONFIG_PARAVIRT_SPINLOCKS
		27	#undef CONFIG_ARCH_HAS_LAZY_MMU_MODE
27		28
28	/*	29	/*
29	* This code runs before CPU feature bits are set. By default, the	30	* This code runs before CPU feature bits are set. By default, the

arch/x86/include/asm/page.h+0−6

 extern struct range pfn_mapped[];
 extern int nr_pfn_mapped;
-static inline void clear_user_page(void *page, unsigned long vaddr,
-				   struct page *pg)
-{
-	clear_page(page);
-}
 static inline void copy_user_page(void *to, void *from, unsigned long vaddr,
 				  struct page *topage)
 {

arch/x86/include/asm/page_32.h+6−0

 #include <linux/string.h>
+/**
+ * clear_page() - clear a page using a kernel virtual address.
+ * @page: address of kernel page
+ *
+ * Does absolutely no exception handling.
+ */
 static inline void clear_page(void *page)
 {
 	memset(page, 0, PAGE_SIZE);

arch/x86/include/asm/page_64.h+61−17

@@ -48,26 +48,70 @@ static inline unsigned long __phys_addr_symbol(unsigned long x) · 47 unmodified lines
48		48
49	#define __phys_reloc_hide(x) (x)	49	#define __phys_reloc_hide(x) (x)
50		50
51	void clear_page_orig(void *page);	51	void __clear_pages_unrolled(void *page);
52	void clear_page_rep(void *page);	52	KCFI_REFERENCE(__clear_pages_unrolled);
53	void clear_page_erms(void *page);	53
54	KCFI_REFERENCE(clear_page_orig);	54	/**
55	KCFI_REFERENCE(clear_page_rep);	55	* clear_pages() - clear a page range using a kernel virtual address.
56	KCFI_REFERENCE(clear_page_erms);	56	* @addr: start address of kernel page range
57		57	* @npages: number of pages
58	static inline void clear_page(void *page)	58	*
		59	* Switch between three implementations of page clearing based on CPU
		60	* capabilities:
		61	*
		62	* - __clear_pages_unrolled(): the oldest, slowest and universally
		63	* supported method. Zeroes via 8-byte MOV instructions unrolled 8x
		64	* to write a 64-byte cacheline in each loop iteration.
		65	*
		66	* - "REP; STOSQ": really old CPUs had crummy REP implementations.
		67	* Vendor CPU setup code sets 'REP_GOOD' on CPUs where REP can be
		68	* trusted. The instruction writes 8-byte per REP iteration but
		69	* CPUs can internally batch these together and do larger writes.
		70	*
		71	* - "REP; STOSB": used on CPUs with "enhanced REP MOVSB/STOSB",
		72	* which enumerate 'ERMS' and provide an implementation which
		73	* unlike "REP; STOSQ" above wasn't overly picky about alignment.
		74	* The instruction writes 1-byte per REP iteration with CPUs
		75	* internally batching these together into larger writes and is
		76	* generally fastest of the three.
		77	*
		78	* Note that when running as a guest, features exposed by the CPU
		79	* might be mediated by the hypervisor. So, the STOSQ variant might
		80	* be in active use on some systems even when the hardware enumerates
		81	* ERMS.
		82	*
		83	* Does absolutely no exception handling.
		84	*/
		85	static inline void clear_pages(void *addr, unsigned int npages)
59	{	86	{
		87	u64 len = npages * PAGE_SIZE;
		88	/*
		89	* Clean up KMSAN metadata for the pages being cleared. The assembly call
		90	* below clobbers @addr, so perform unpoisoning before it.
		91	*/
		92	kmsan_unpoison_memory(addr, len);
		93
60	/*	94	/*
61	* Clean up KMSAN metadata for the page being cleared. The assembly call	95	* The inline asm embeds a CALL instruction and usually that is a no-no
62	* below clobbers @page, so we perform unpoisoning before it.	96	* due to the compiler not knowing that and thus being unable to track
		97	* callee-clobbered registers.
		98	*
		99	* In this case that is fine because the registers clobbered by
		100	* __clear_pages_unrolled() are part of the inline asm register
		101	* specification.
63	*/	102	*/
64	kmsan_unpoison_memory(page, PAGE_SIZE);	103	asm volatile(ALTERNATIVE_2("call __clear_pages_unrolled",
65	alternative_call_2(clear_page_orig,	104	"shrq $3, %%rcx; rep stosq", X86_FEATURE_REP_GOOD,
66	clear_page_rep, X86_FEATURE_REP_GOOD,	105	"rep stosb", X86_FEATURE_ERMS)
67	clear_page_erms, X86_FEATURE_ERMS,	106	: "+c" (len), "+D" (addr), ASM_CALL_CONSTRAINT
68	"=D" (page),	107	: "a" (0)
69	"D" (page),	108	: "cc", "memory");
70	"cc", "memory", "rax", "rcx");	109	}
		110	#define clear_pages clear_pages
		111
		112	static inline void clear_page(void *addr)
		113	{
		114	clear_pages(addr, 1);
71	}	115	}
72		116
73	void copy_page(void to, void from);	117	void copy_page(void to, void from);

arch/x86/include/asm/paravirt.h+0−1

@@ -492,7 +492,6 @@ static inline void arch_end_context_switch(struct task_struct *next) · 491 unmodified lines
492	PVOP_VCALL1(pv_ops, cpu.end_context_switch, next);	492	PVOP_VCALL1(pv_ops, cpu.end_context_switch, next);
493	}	493	}
494		494
495	#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE
496	static inline void arch_enter_lazy_mmu_mode(void)	495	static inline void arch_enter_lazy_mmu_mode(void)
497	{	496	{
498	PVOP_VCALL0(pv_ops, mmu.lazy_mode.enter);	497	PVOP_VCALL0(pv_ops, mmu.lazy_mode.enter);

arch/x86/include/asm/pgtable.h+12−11

 #define __pte(x)	native_make_pte(x)
 #define arch_end_context_switch(prev)	do {} while(0)
+static inline void arch_flush_lazy_mmu_mode(void) {}
 #endif	/* CONFIG_PARAVIRT_XXL */
 static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set)
 static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
 			      pmd_t *pmdp, pmd_t pmd)
 {
-	page_table_check_pmd_set(mm, pmdp, pmd);
+	page_table_check_pmd_set(mm, addr, pmdp, pmd);
 	set_pmd(pmdp, pmd);
 }
 static inline void set_pud_at(struct mm_struct *mm, unsigned long addr,
 			      pud_t *pudp, pud_t pud)
 {
-	page_table_check_pud_set(mm, pudp, pud);
+	page_table_check_pud_set(mm, addr, pudp, pud);
 	native_set_pud(pudp, pud);
 }
 				       pte_t *ptep)
 {
 	pte_t pte = native_ptep_get_and_clear(ptep);
-	page_table_check_pte_clear(mm, pte);
+	page_table_check_pte_clear(mm, addr, pte);
 	return pte;
 }
 		 * care about updates and native needs no locking
 		 */
 		pte = native_local_ptep_get_and_clear(ptep);
-		page_table_check_pte_clear(mm, pte);
+		page_table_check_pte_clear(mm, addr, pte);
 	} else {
 		pte = ptep_get_and_clear(mm, addr, ptep);
 	}
 {
 	pmd_t pmd = native_pmdp_get_and_clear(pmdp);
-	page_table_check_pmd_clear(mm, pmd);
+	page_table_check_pmd_clear(mm, addr, pmd);
 	return pmd;
 }
 {
 	pud_t pud = native_pudp_get_and_clear(pudp);
-	page_table_check_pud_clear(mm, pud);
+	page_table_check_pud_clear(mm, addr, pud);
 	return pud;
 }
 static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 		unsigned long address, pmd_t *pmdp, pmd_t pmd)
 {
-	page_table_check_pmd_set(vma->vm_mm, pmdp, pmd);
+	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
 	if (IS_ENABLED(CONFIG_SMP)) {
 		return xchg(pmdp, pmd);
 	} else {
 static inline pud_t pudp_establish(struct vm_area_struct *vma,
 		unsigned long address, pud_t *pudp, pud_t pud)
 {
-	page_table_check_pud_set(vma->vm_mm, pudp, pud);
+	page_table_check_pud_set(vma->vm_mm, address, pudp, pud);
 	if (IS_ENABLED(CONFIG_SMP)) {
 		return xchg(pudp, pud);
 	} else {
 #endif
 #ifdef CONFIG_PAGE_TABLE_CHECK
-static inline bool pte_user_accessible_page(pte_t pte)
+static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr)
 {
 	return (pte_val(pte) & _PAGE_PRESENT) && (pte_val(pte) & _PAGE_USER);
 }
-static inline bool pmd_user_accessible_page(pmd_t pmd)
+static inline bool pmd_user_accessible_page(pmd_t pmd, unsigned long addr)
 {
 	return pmd_leaf(pmd) && (pmd_val(pmd) & _PAGE_PRESENT) && (pmd_val(pmd) & _PAGE_USER);
 }
-static inline bool pud_user_accessible_page(pud_t pud)
+static inline bool pud_user_accessible_page(pud_t pud, unsigned long addr)
 {
 	return pud_leaf(pud) && (pud_val(pud) & _PAGE_PRESENT) && (pud_val(pud) & _PAGE_USER);
 }

arch/x86/include/asm/thread_info.h+1−3

 #define TIF_FORCED_TF		24	/* true if TF in eflags artificially */
 #define TIF_SINGLESTEP		25	/* reenable singlestep on user return*/
 #define TIF_BLOCKSTEP		26	/* set when we want DEBUGCTLMSR_BTF */
-#define TIF_LAZY_MMU_UPDATES	27	/* task is updating the mmu lazily */
+#define TIF_ADDR32		27	/* 32-bit address space on 64 bits */
-#define TIF_ADDR32		28	/* 32-bit address space on 64 bits */
 #define _TIF_SSBD		BIT(TIF_SSBD)
 #define _TIF_SPEC_IB		BIT(TIF_SPEC_IB)
 #define _TIF_FORCED_TF		BIT(TIF_FORCED_TF)
 #define _TIF_BLOCKSTEP		BIT(TIF_BLOCKSTEP)
 #define _TIF_SINGLESTEP		BIT(TIF_SINGLESTEP)
-#define _TIF_LAZY_MMU_UPDATES	BIT(TIF_LAZY_MMU_UPDATES)
 #define _TIF_ADDR32		BIT(TIF_ADDR32)
 /* flags to check in __switch_to() */

arch/x86/kernel/setup.c+0−5

 	initmem_init();
 	dma_contiguous_reserve(max_pfn_mapped << PAGE_SHIFT);
-	if (boot_cpu_has(X86_FEATURE_GBPAGES)) {
-		hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
-		hugetlb_bootmem_alloc();
-	}
 	/*
 	 * Reserve memory for crash kernel after SRAT is parsed so that it
 	 * won't consume hotpluggable memory.

arch/x86/lib/clear_page_64.S+8−31

@@ -6,30 +6,15 @@ · 5 unmodified lines
6	#include <asm/asm.h>	6	#include <asm/asm.h>
7		7
8	/*	8	/*
9	* Most CPUs support enhanced REP MOVSB/STOSB instructions. It is	9	* Zero page aligned region.
10	* recommended to use this when possible and we do use them by default.	10	* %rdi - dest
11	* If enhanced REP MOVSB/STOSB is not available, try to use fast string.	11	* %rcx - length
12	* Otherwise, use original.
13	*/	12	*/
14		13	SYM_TYPED_FUNC_START(__clear_pages_unrolled)
15	/*	14	shrq $6, %rcx
16	* Zero a page.
17	* %rdi - page
18	*/
19	SYM_TYPED_FUNC_START(clear_page_rep)
20	movl $4096/8,%ecx
21	xorl %eax,%eax
22	rep stosq
23	RET
24	SYM_FUNC_END(clear_page_rep)
25	EXPORT_SYMBOL_GPL(clear_page_rep)
26
27	SYM_TYPED_FUNC_START(clear_page_orig)
28	xorl %eax,%eax
29	movl $4096/64,%ecx
30	.p2align 4	15	.p2align 4
31	.Lloop:	16	.Lloop:
32	decl %ecx	17	decq %rcx
33	#define PUT(x) movq %rax,x*8(%rdi)	18	#define PUT(x) movq %rax,x*8(%rdi)
34	movq %rax,(%rdi)	19	movq %rax,(%rdi)
35	PUT(1)	20	PUT(1)
@@ -43,16 +28,8 @@ SYM_TYPED_FUNC_START(clear_page_orig) · 7 unmodified lines
43	jnz .Lloop	28	jnz .Lloop
44	nop	29	nop
45	RET	30	RET
46	SYM_FUNC_END(clear_page_orig)	31	SYM_FUNC_END(__clear_pages_unrolled)
47	EXPORT_SYMBOL_GPL(clear_page_orig)	32	EXPORT_SYMBOL_GPL(__clear_pages_unrolled)
48
49	SYM_TYPED_FUNC_START(clear_page_erms)
50	movl $4096,%ecx
51	xorl %eax,%eax
52	rep stosb
53	RET
54	SYM_FUNC_END(clear_page_erms)
55	EXPORT_SYMBOL_GPL(clear_page_erms)
56		33
57	/*	34	/*
58	* Default clear user-space.	35	* Default clear user-space.

arch/x86/mm/hugetlbpage.c+8−0

@@ -42,3 +42,11 @@ static __init int gigantic_pages_init(void) · 41 unmodified lines
42	arch_initcall(gigantic_pages_init);	42	arch_initcall(gigantic_pages_init);
43	#endif	43	#endif
44	#endif	44	#endif
		45
		46	unsigned int __init arch_hugetlb_cma_order(void)
		47	{
		48	if (boot_cpu_has(X86_FEATURE_GBPAGES))
		49	return PUD_SHIFT - PAGE_SHIFT;
		50
		51	return 0;
		52	}

arch/x86/mm/init.c+1−7

 }
 #endif
-void __init zone_sizes_init(void)
+void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
 {
-	unsigned long max_zone_pfns[MAX_NR_ZONES];
-	memset(max_zone_pfns, 0, sizeof(max_zone_pfns));
 #ifdef CONFIG_ZONE_DMA
 	max_zone_pfns[ZONE_DMA]		= min(MAX_DMA_PFN, max_low_pfn);
 #endif
 #ifdef CONFIG_HIGHMEM
 	max_zone_pfns[ZONE_HIGHMEM]	= max_pfn;
 #endif
-	free_area_init(max_zone_pfns);
 }
 __visible DEFINE_PER_CPU_ALIGNED(struct tlb_state, cpu_tlbstate) = {

arch/x86/mm/init_32.c+0−2

@@ -654,8 +654,6 @@ void __init paging_init(void) · 653 unmodified lines
654	* NOTE: at this point the bootmem allocator is fully available.	654	* NOTE: at this point the bootmem allocator is fully available.
655	*/	655	*/
656	olpc_dt_build_devicetree();	656	olpc_dt_build_devicetree();
657	sparse_init();
658	zone_sizes_init();
659	}	657	}
660		658
661	/*	659	/*

arch/x86/mm/init_64.c+0−4

 void __init paging_init(void)
 {
-	sparse_init();
 	/*
 	 * clear the default setting with node 0
 	 * note: don't use nodes_clear here, that is really clearing when
 	 */
 	node_clear_state(0, N_MEMORY);
 	node_clear_state(0, N_NORMAL_MEMORY);
-	zone_sizes_init();
 }
 #define PAGE_UNUSED 0xFD

arch/x86/mm/mm_internal.h+0−1

@@ -17,7 +17,6 @@ unsigned long kernel_physical_mapping_init(unsigned long start, · 16 unmodified lines
17	unsigned long kernel_physical_mapping_change(unsigned long start,	17	unsigned long kernel_physical_mapping_change(unsigned long start,
18	unsigned long end,	18	unsigned long end,
19	unsigned long page_size_mask);	19	unsigned long page_size_mask);
20	void zone_sizes_init(void);
21		20
22	extern int after_bootmem;	21	extern int after_bootmem;
23		22

arch/x86/xen/enlighten_pv.c+1−2

 	if (this_cpu_read(xen_lazy_mode) == XEN_LAZY_MMU) {
 		arch_leave_lazy_mmu_mode();
-		set_ti_thread_flag(task_thread_info(prev), TIF_LAZY_MMU_UPDATES);
 	}
 	enter_lazy(XEN_LAZY_CPU);
 }
 	xen_mc_flush();
 	leave_lazy(XEN_LAZY_CPU);
-	if (test_and_clear_ti_thread_flag(task_thread_info(next), TIF_LAZY_MMU_UPDATES))
+	if (__task_lazy_mmu_mode_active(next))
 		arch_enter_lazy_mmu_mode();
 }

arch/x86/xen/mmu_pv.c+2−4

 {
 	preempt_disable();
-	if (xen_get_lazy_mode() == XEN_LAZY_MMU) {
+	if (xen_get_lazy_mode() == XEN_LAZY_MMU)
-		arch_leave_lazy_mmu_mode();
+		xen_mc_flush();
-		arch_enter_lazy_mmu_mode();
-	}
 	preempt_enable();
 }

arch/xtensa/include/asm/page.h+0−1

@@ -126,7 +126,6 @@ void clear_user_highpage(struct page *page, unsigned long vaddr); · 125 unmodified lines
126	void copy_user_highpage(struct page to, struct page from,	126	void copy_user_highpage(struct page to, struct page from,
127	unsigned long vaddr, struct vm_area_struct *vma);	127	unsigned long vaddr, struct vm_area_struct *vma);
128	#else	128	#else
129	# define clear_user_page(page, vaddr, pg) clear_page(page)
130	# define copy_user_page(to, from, vaddr, pg) copy_page(to, from)	129	# define copy_user_page(to, from, vaddr, pg) copy_page(to, from)
131	#endif	130	#endif
132		131

arch/xtensa/mm/init.c+7−7

@@ -116,16 +116,16 @@ static void __init print_vm_layout(void) · 115 unmodified lines
116	(unsigned long)(__bss_stop - __bss_start) >> 10);	116	(unsigned long)(__bss_stop - __bss_start) >> 10);
117	}	117	}
118		118
119	void __init zones_init(void)	119	void __init arch_zone_limits_init(unsigned long *max_zone_pfns)
120	{	120	{
121	/* All pages are DMA-able, so we put them all in the DMA zone. */	121	max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
122	unsigned long max_zone_pfn[MAX_NR_ZONES] = {
123	[ZONE_NORMAL] = max_low_pfn,
124	#ifdef CONFIG_HIGHMEM	122	#ifdef CONFIG_HIGHMEM
125	[ZONE_HIGHMEM] = max_pfn,	123	max_zone_pfns[ZONE_HIGHMEM] = max_pfn;
126	#endif	124	#endif
127	};	125	}
128	free_area_init(max_zone_pfn);	126
		127	void __init zones_init(void)
		128	{
129	print_vm_layout();	129	print_vm_layout();
130	}	130	}
131		131

block/blk-mq.c+0−3

@@ -823,9 +823,6 @@ void blk_mq_free_request(struct request *rq) · 822 unmodified lines
823		823
824	blk_mq_finish_request(rq);	824	blk_mq_finish_request(rq);
825		825
826	if (unlikely(laptop_mode && !blk_rq_is_passthrough(rq)))
827	laptop_io_completion(q->disk->bdi);
828
829	rq_qos_done(q, rq);	826	rq_qos_done(q, rq);
830		827
831	WRITE_ONCE(rq->state, MQ_RQ_IDLE);	828	WRITE_ONCE(rq->state, MQ_RQ_IDLE);

drivers/block/zram/zram_drv.c+565−434

@@ -12,8 +12,7 @@ · 11 unmodified lines
12	*	12	*
13	*/	13	*/
14		14
15	#define KMSG_COMPONENT "zram"	15	#define pr_fmt(fmt) "zram: " fmt
16	#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
17		16
18	#include <linux/module.h>	17	#include <linux/module.h>
19	#include <linux/kernel.h>	18	#include <linux/kernel.h>
@@ -56,13 +55,10 @@ static size_t huge_class_size; · 36 unmodified lines
56		55
57	static const struct block_device_operations zram_devops;	56	static const struct block_device_operations zram_devops;
58		57
59	static void zram_free_page(struct zram *zram, size_t index);	58	static void slot_free(struct zram *zram, u32 index);
60	static int zram_read_from_zspool(struct zram zram, struct page page,
61	u32 index);
62
63	#define slot_dep_map(zram, index) (&(zram)->table[(index)].dep_map)	59	#define slot_dep_map(zram, index) (&(zram)->table[(index)].dep_map)
64		60
65	static void zram_slot_lock_init(struct zram *zram, u32 index)	61	static void slot_lock_init(struct zram *zram, u32 index)
66	{	62	{
67	static struct lock_class_key __key;	63	static struct lock_class_key __key;
68		64
@@ -82,9 +78,9 @@ static void zram_slot_lock_init(struct zram *zram, u32 index) · 13 unmodified lines
82	* 4) Use TRY lock variant when in atomic context	78	* 4) Use TRY lock variant when in atomic context
83	* - must check return value and handle locking failers	79	* - must check return value and handle locking failers
84	*/	80	*/
85	static __must_check bool zram_slot_trylock(struct zram *zram, u32 index)	81	static __must_check bool slot_trylock(struct zram *zram, u32 index)
86	{	82	{
87	unsigned long *lock = &zram->table[index].flags;	83	unsigned long *lock = &zram->table[index].__lock;
88		84
89	if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) {	85	if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) {
90	mutex_acquire(slot_dep_map(zram, index), 0, 1, _RET_IP_);	86	mutex_acquire(slot_dep_map(zram, index), 0, 1, _RET_IP_);
@@ -95,18 +91,18 @@ static __must_check bool zram_slot_trylock(struct zram *zram, u32 index) · 4 unmodified lines
95	return false;	91	return false;
96	}	92	}
97		93
98	static void zram_slot_lock(struct zram *zram, u32 index)	94	static void slot_lock(struct zram *zram, u32 index)
99	{	95	{
100	unsigned long *lock = &zram->table[index].flags;	96	unsigned long *lock = &zram->table[index].__lock;
101		97
102	mutex_acquire(slot_dep_map(zram, index), 0, 0, _RET_IP_);	98	mutex_acquire(slot_dep_map(zram, index), 0, 0, _RET_IP_);
103	wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE);	99	wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE);
104	lock_acquired(slot_dep_map(zram, index), _RET_IP_);	100	lock_acquired(slot_dep_map(zram, index), _RET_IP_);
105	}	101	}
106		102
107	static void zram_slot_unlock(struct zram *zram, u32 index)	103	static void slot_unlock(struct zram *zram, u32 index)
108	{	104	{
109	unsigned long *lock = &zram->table[index].flags;	105	unsigned long *lock = &zram->table[index].__lock;
110		106
111	mutex_release(slot_dep_map(zram, index), _RET_IP_);	107	mutex_release(slot_dep_map(zram, index), _RET_IP_);
112	clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock);	108	clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock);
@@ -122,52 +118,80 @@ static inline struct zram dev_to_zram(struct device dev) · 9 unmodified lines
122	return (struct zram *)dev_to_disk(dev)->private_data;	118	return (struct zram *)dev_to_disk(dev)->private_data;
123	}	119	}
124		120
125	static unsigned long zram_get_handle(struct zram *zram, u32 index)	121	static unsigned long get_slot_handle(struct zram *zram, u32 index)
126	{	122	{
127	return zram->table[index].handle;	123	return zram->table[index].handle;
128	}	124	}
129		125
130	static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle)	126	static void set_slot_handle(struct zram *zram, u32 index, unsigned long handle)
131	{	127	{
132	zram->table[index].handle = handle;	128	zram->table[index].handle = handle;
133	}	129	}
134		130
135	static bool zram_test_flag(struct zram *zram, u32 index,	131	static bool test_slot_flag(struct zram *zram, u32 index,
136	enum zram_pageflags flag)	132	enum zram_pageflags flag)
137	{	133	{
138	return zram->table[index].flags & BIT(flag);	134	return zram->table[index].attr.flags & BIT(flag);
139	}	135	}
140		136
141	static void zram_set_flag(struct zram *zram, u32 index,	137	static void set_slot_flag(struct zram *zram, u32 index,
142	enum zram_pageflags flag)	138	enum zram_pageflags flag)
143	{	139	{
144	zram->table[index].flags \|= BIT(flag);	140	zram->table[index].attr.flags \|= BIT(flag);
145	}	141	}
146		142
147	static void zram_clear_flag(struct zram *zram, u32 index,	143	static void clear_slot_flag(struct zram *zram, u32 index,
148	enum zram_pageflags flag)	144	enum zram_pageflags flag)
149	{	145	{
150	zram->table[index].flags &= ~BIT(flag);	146	zram->table[index].attr.flags &= ~BIT(flag);
151	}	147	}
152		148
153	static size_t zram_get_obj_size(struct zram *zram, u32 index)	149	static size_t get_slot_size(struct zram *zram, u32 index)
154	{	150	{
155	return zram->table[index].flags & (BIT(ZRAM_FLAG_SHIFT) - 1);	151	return zram->table[index].attr.flags & (BIT(ZRAM_FLAG_SHIFT) - 1);
156	}	152	}
157		153
158	static void zram_set_obj_size(struct zram *zram,	154	static void set_slot_size(struct zram *zram, u32 index, size_t size)
159	u32 index, size_t size)
160	{	155	{
161	unsigned long flags = zram->table[index].flags >> ZRAM_FLAG_SHIFT;	156	unsigned long flags = zram->table[index].attr.flags >> ZRAM_FLAG_SHIFT;
		157
		158	zram->table[index].attr.flags = (flags << ZRAM_FLAG_SHIFT) \| size;
		159	}
162		160
163	zram->table[index].flags = (flags << ZRAM_FLAG_SHIFT) \| size;	161	static inline bool slot_allocated(struct zram *zram, u32 index)
		162	{
		163	return get_slot_size(zram, index) \|\|
		164	test_slot_flag(zram, index, ZRAM_SAME) \|\|
		165	test_slot_flag(zram, index, ZRAM_WB);
		166	}
		167
		168	static inline void set_slot_comp_priority(struct zram *zram, u32 index,
		169	u32 prio)
		170	{
		171	prio &= ZRAM_COMP_PRIORITY_MASK;
		172	/*
		173	* Clear previous priority value first, in case if we recompress
		174	* further an already recompressed page
		175	*/
		176	zram->table[index].attr.flags &= ~(ZRAM_COMP_PRIORITY_MASK <<
		177	ZRAM_COMP_PRIORITY_BIT1);
		178	zram->table[index].attr.flags \|= (prio << ZRAM_COMP_PRIORITY_BIT1);
164	}	179	}
165		180
166	static inline bool zram_allocated(struct zram *zram, u32 index)	181	static inline u32 get_slot_comp_priority(struct zram *zram, u32 index)
167	{	182	{
168	return zram_get_obj_size(zram, index) \|\|	183	u32 prio = zram->table[index].attr.flags >> ZRAM_COMP_PRIORITY_BIT1;
169	zram_test_flag(zram, index, ZRAM_SAME) \|\|	184
170	zram_test_flag(zram, index, ZRAM_WB);	185	return prio & ZRAM_COMP_PRIORITY_MASK;
		186	}
		187
		188	static void mark_slot_accessed(struct zram *zram, u32 index)
		189	{
		190	clear_slot_flag(zram, index, ZRAM_IDLE);
		191	clear_slot_flag(zram, index, ZRAM_PP_SLOT);
		192	#ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME
		193	zram->table[index].attr.ac_time = (u32)ktime_get_boottime_seconds();
		194	#endif
171	}	195	}
172		196
173	static inline void update_used_max(struct zram *zram, const unsigned long pages)	197	static inline void update_used_max(struct zram *zram, const unsigned long pages)
@@ -204,34 +228,6 @@ static inline bool is_partial_io(struct bio_vec *bvec) · 30 unmodified lines
204	}	228	}
205	#endif	229	#endif
206		230
207	static inline void zram_set_priority(struct zram *zram, u32 index, u32 prio)
208	{
209	prio &= ZRAM_COMP_PRIORITY_MASK;
210	/*
211	* Clear previous priority value first, in case if we recompress
212	* further an already recompressed page
213	*/
214	zram->table[index].flags &= ~(ZRAM_COMP_PRIORITY_MASK <<
215	ZRAM_COMP_PRIORITY_BIT1);
216	zram->table[index].flags \|= (prio << ZRAM_COMP_PRIORITY_BIT1);
217	}
218
219	static inline u32 zram_get_priority(struct zram *zram, u32 index)
220	{
221	u32 prio = zram->table[index].flags >> ZRAM_COMP_PRIORITY_BIT1;
222
223	return prio & ZRAM_COMP_PRIORITY_MASK;
224	}
225
226	static void zram_accessed(struct zram *zram, u32 index)
227	{
228	zram_clear_flag(zram, index, ZRAM_IDLE);
229	zram_clear_flag(zram, index, ZRAM_PP_SLOT);
230	#ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME
231	zram->table[index].ac_time = ktime_get_boottime();
232	#endif
233	}
234
235	#if defined CONFIG_ZRAM_WRITEBACK \|\| defined CONFIG_ZRAM_MULTI_COMP	231	#if defined CONFIG_ZRAM_WRITEBACK \|\| defined CONFIG_ZRAM_MULTI_COMP
236	struct zram_pp_slot {	232	struct zram_pp_slot {
237	unsigned long index;	233	unsigned long index;
@@ -267,9 +263,9 @@ static void release_pp_slot(struct zram zram, struct zram_pp_slot pps) · 29 unmodified lines
267	{	263	{
268	list_del_init(&pps->entry);	264	list_del_init(&pps->entry);
269		265
270	zram_slot_lock(zram, pps->index);	266	slot_lock(zram, pps->index);
271	zram_clear_flag(zram, pps->index, ZRAM_PP_SLOT);	267	clear_slot_flag(zram, pps->index, ZRAM_PP_SLOT);
272	zram_slot_unlock(zram, pps->index);	268	slot_unlock(zram, pps->index);
273		269
274	kfree(pps);	270	kfree(pps);
275	}	271	}
@@ -308,10 +304,10 @@ static bool place_pp_slot(struct zram zram, struct zram_pp_ctl ctl, · 32 unmodified lines
308	INIT_LIST_HEAD(&pps->entry);	304	INIT_LIST_HEAD(&pps->entry);
309	pps->index = index;	305	pps->index = index;
310		306
311	bid = zram_get_obj_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE;	307	bid = get_slot_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE;
312	list_add(&pps->entry, &ctl->pp_buckets[bid]);	308	list_add(&pps->entry, &ctl->pp_buckets[bid]);
313		309
314	zram_set_flag(zram, pps->index, ZRAM_PP_SLOT);	310	set_slot_flag(zram, pps->index, ZRAM_PP_SLOT);
315	return true;	311	return true;
316	}	312	}
317		313
@@ -363,15 +359,14 @@ static bool page_same_filled(void ptr, unsigned long element) · 45 unmodified lines
363	return true;	359	return true;
364	}	360	}
365		361
366	static ssize_t initstate_show(struct device *dev,	362	static ssize_t initstate_show(struct device dev, struct device_attribute attr,
367	struct device_attribute attr, char buf)	363	char *buf)
368	{	364	{
369	u32 val;	365	u32 val;
370	struct zram *zram = dev_to_zram(dev);	366	struct zram *zram = dev_to_zram(dev);
371		367
372	down_read(&zram->init_lock);	368	guard(rwsem_read)(&zram->dev_lock);
373	val = init_done(zram);	369	val = init_done(zram);
374	up_read(&zram->init_lock);
375		370
376	return sysfs_emit(buf, "%u\n", val);	371	return sysfs_emit(buf, "%u\n", val);
377	}	372	}
@@ -385,7 +380,8 @@ static ssize_t disksize_show(struct device *dev, · 7 unmodified lines
385	}	380	}
386		381
387	static ssize_t mem_limit_store(struct device *dev,	382	static ssize_t mem_limit_store(struct device *dev,
388	struct device_attribute attr, const char buf, size_t len)	383	struct device_attribute attr, const char buf,
		384	size_t len)
389	{	385	{
390	u64 limit;	386	u64 limit;
391	char *tmp;	387	char *tmp;
@@ -395,15 +391,15 @@ static ssize_t mem_limit_store(struct device *dev, · 3 unmodified lines
395	if (buf == tmp) /* no chars parsed, invalid input */	391	if (buf == tmp) /* no chars parsed, invalid input */
396	return -EINVAL;	392	return -EINVAL;
397		393
398	down_write(&zram->init_lock);	394	guard(rwsem_write)(&zram->dev_lock);
399	zram->limit_pages = PAGE_ALIGN(limit) >> PAGE_SHIFT;	395	zram->limit_pages = PAGE_ALIGN(limit) >> PAGE_SHIFT;
400	up_write(&zram->init_lock);
401		396
402	return len;	397	return len;
403	}	398	}
404		399
405	static ssize_t mem_used_max_store(struct device *dev,	400	static ssize_t mem_used_max_store(struct device *dev,
406	struct device_attribute attr, const char buf, size_t len)	401	struct device_attribute *attr,
		402	const char *buf, size_t len)
407	{	403	{
408	int err;	404	int err;
409	unsigned long val;	405	unsigned long val;
@@ -413,12 +409,11 @@ static ssize_t mem_used_max_store(struct device *dev, · 3 unmodified lines
413	if (err \|\| val != 0)	409	if (err \|\| val != 0)
414	return -EINVAL;	410	return -EINVAL;
415		411
416	down_read(&zram->init_lock);	412	guard(rwsem_read)(&zram->dev_lock);
417	if (init_done(zram)) {	413	if (init_done(zram)) {
418	atomic_long_set(&zram->stats.max_used_pages,	414	atomic_long_set(&zram->stats.max_used_pages,
419	zs_get_total_pages(zram->mem_pool));	415	zs_get_total_pages(zram->mem_pool));
420	}	416	}
421	up_read(&zram->init_lock);
422		417
423	return len;	418	return len;
424	}	419	}
@@ -441,67 +436,66 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) · 16 unmodified lines
441	*	436	*
442	* And ZRAM_WB slots simply cannot be ZRAM_IDLE.	437	* And ZRAM_WB slots simply cannot be ZRAM_IDLE.
443	*/	438	*/
444	zram_slot_lock(zram, index);	439	slot_lock(zram, index);
445	if (!zram_allocated(zram, index) \|\|	440	if (!slot_allocated(zram, index) \|\|
446	zram_test_flag(zram, index, ZRAM_WB) \|\|	441	test_slot_flag(zram, index, ZRAM_WB) \|\|
447	zram_test_flag(zram, index, ZRAM_SAME)) {	442	test_slot_flag(zram, index, ZRAM_SAME)) {
448	zram_slot_unlock(zram, index);	443	slot_unlock(zram, index);
449	continue;	444	continue;
450	}	445	}
451		446
452	#ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME	447	#ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME
453	is_idle = !cutoff \|\|	448	is_idle = !cutoff \|\|
454	ktime_after(cutoff, zram->table[index].ac_time);	449	ktime_after(cutoff, zram->table[index].attr.ac_time);
455	#endif	450	#endif
456	if (is_idle)	451	if (is_idle)
457	zram_set_flag(zram, index, ZRAM_IDLE);	452	set_slot_flag(zram, index, ZRAM_IDLE);
458	else	453	else
459	zram_clear_flag(zram, index, ZRAM_IDLE);	454	clear_slot_flag(zram, index, ZRAM_IDLE);
460	zram_slot_unlock(zram, index);	455	slot_unlock(zram, index);
461	}	456	}
462	}	457	}
463		458
464	static ssize_t idle_store(struct device *dev,	459	static ssize_t idle_store(struct device dev, struct device_attribute attr,
465	struct device_attribute attr, const char buf, size_t len)	460	const char *buf, size_t len)
466	{	461	{
467	struct zram *zram = dev_to_zram(dev);	462	struct zram *zram = dev_to_zram(dev);
468	ktime_t cutoff_time = 0;	463	ktime_t cutoff = 0;
469	ssize_t rv = -EINVAL;
470		464
471	if (!sysfs_streq(buf, "all")) {	465	if (!sysfs_streq(buf, "all")) {
472	/*	466	/*
473	* If it did not parse as 'all' try to treat it as an integer	467	* If it did not parse as 'all' try to treat it as an integer
474	* when we have memory tracking enabled.	468	* when we have memory tracking enabled.
475	*/	469	*/
476	u64 age_sec;	470	u32 age_sec;
477		471
478	if (IS_ENABLED(CONFIG_ZRAM_TRACK_ENTRY_ACTIME) && !kstrtoull(buf, 0, &age_sec))	472	if (IS_ENABLED(CONFIG_ZRAM_TRACK_ENTRY_ACTIME) &&
479	cutoff_time = ktime_sub(ktime_get_boottime(),	473	!kstrtouint(buf, 0, &age_sec))
480	ns_to_ktime(age_sec * NSEC_PER_SEC));	474	cutoff = ktime_sub((u32)ktime_get_boottime_seconds(),
		475	age_sec);
481	else	476	else
482	goto out;	477	return -EINVAL;
483	}	478	}
484		479
485	down_read(&zram->init_lock);	480	guard(rwsem_read)(&zram->dev_lock);
486	if (!init_done(zram))	481	if (!init_done(zram))
487	goto out_unlock;	482	return -EINVAL;
488		483
489	/*	484	/*
490	* A cutoff_time of 0 marks everything as idle, this is the	485	* A cutoff of 0 marks everything as idle, this is the
491	* "all" behavior.	486	* "all" behavior.
492	*/	487	*/
493	mark_idle(zram, cutoff_time);	488	mark_idle(zram, cutoff);
494	rv = len;	489	return len;
495
496	out_unlock:
497	up_read(&zram->init_lock);
498	out:
499	return rv;
500	}	490	}
501		491
502	#ifdef CONFIG_ZRAM_WRITEBACK	492	#ifdef CONFIG_ZRAM_WRITEBACK
503	#define INVALID_BDEV_BLOCK (~0UL)	493	#define INVALID_BDEV_BLOCK (~0UL)
504		494
		495	static int read_from_zspool_raw(struct zram zram, struct page page,
		496	u32 index);
		497	static int read_from_zspool(struct zram zram, struct page page, u32 index);
		498
505	struct zram_wb_ctl {	499	struct zram_wb_ctl {
506	/* idle list is accessed only by the writeback task, no concurency */	500	/* idle list is accessed only by the writeback task, no concurency */
507	struct list_head idle_reqs;	501	struct list_head idle_reqs;
@@ -522,23 +516,86 @@ struct zram_wb_req { · 14 unmodified lines
522	struct list_head entry;	516	struct list_head entry;
523	};	517	};
524		518
		519	struct zram_rb_req {
		520	struct work_struct work;
		521	struct zram *zram;
		522	struct page *page;
		523	/* The read bio for backing device */
		524	struct bio *bio;
		525	unsigned long blk_idx;
		526	union {
		527	/* The original bio to complete (async read) */
		528	struct bio *parent;
		529	/* error status (sync read) */
		530	int error;
		531	};
		532	u32 index;
		533	};
		534
		535	#define FOUR_K(x) ((x) * (1 << (PAGE_SHIFT - 12)))
		536	static ssize_t bd_stat_show(struct device dev, struct device_attribute attr,
		537	char *buf)
		538	{
		539	struct zram *zram = dev_to_zram(dev);
		540	ssize_t ret;
		541
		542	guard(rwsem_read)(&zram->dev_lock);
		543	ret = sysfs_emit(buf,
		544	"%8llu %8llu %8llu\n",
		545	FOUR_K((u64)atomic64_read(&zram->stats.bd_count)),
		546	FOUR_K((u64)atomic64_read(&zram->stats.bd_reads)),
		547	FOUR_K((u64)atomic64_read(&zram->stats.bd_writes)));
		548
		549	return ret;
		550	}
		551
		552	static ssize_t writeback_compressed_store(struct device *dev,
		553	struct device_attribute *attr,
		554	const char *buf, size_t len)
		555	{
		556	struct zram *zram = dev_to_zram(dev);
		557	bool val;
		558
		559	if (kstrtobool(buf, &val))
		560	return -EINVAL;
		561
		562	guard(rwsem_write)(&zram->dev_lock);
		563	if (init_done(zram)) {
		564	return -EBUSY;
		565	}
		566
		567	zram->wb_compressed = val;
		568
		569	return len;
		570	}
		571
		572	static ssize_t writeback_compressed_show(struct device *dev,
		573	struct device_attribute *attr,
		574	char *buf)
		575	{
		576	bool val;
		577	struct zram *zram = dev_to_zram(dev);
		578
		579	guard(rwsem_read)(&zram->dev_lock);
		580	val = zram->wb_compressed;
		581
		582	return sysfs_emit(buf, "%d\n", val);
		583	}
		584
525	static ssize_t writeback_limit_enable_store(struct device *dev,	585	static ssize_t writeback_limit_enable_store(struct device *dev,
526	struct device_attribute *attr,	586	struct device_attribute *attr,
527	const char *buf, size_t len)	587	const char *buf, size_t len)
528	{	588	{
529	struct zram *zram = dev_to_zram(dev);	589	struct zram *zram = dev_to_zram(dev);
530	u64 val;	590	u64 val;
531	ssize_t ret = -EINVAL;
532		591
533	if (kstrtoull(buf, 10, &val))	592	if (kstrtoull(buf, 10, &val))
534	return ret;	593	return -EINVAL;
535		594
536	down_write(&zram->init_lock);	595	guard(rwsem_write)(&zram->dev_lock);
537	zram->wb_limit_enable = val;	596	zram->wb_limit_enable = val;
538	up_write(&zram->init_lock);
539	ret = len;
540		597
541	return ret;	598	return len;
542	}	599	}
543		600
544	static ssize_t writeback_limit_enable_show(struct device *dev,	601	static ssize_t writeback_limit_enable_show(struct device *dev,
@@ -548,9 +605,8 @@ static ssize_t writeback_limit_enable_show(struct device *dev, · 3 unmodified lines
548	bool val;	605	bool val;
549	struct zram *zram = dev_to_zram(dev);	606	struct zram *zram = dev_to_zram(dev);
550		607
551	down_read(&zram->init_lock);	608	guard(rwsem_read)(&zram->dev_lock);
552	val = zram->wb_limit_enable;	609	val = zram->wb_limit_enable;
553	up_read(&zram->init_lock);
554		610
555	return sysfs_emit(buf, "%d\n", val);	611	return sysfs_emit(buf, "%d\n", val);
556	}	612	}
@@ -561,10 +617,9 @@ static ssize_t writeback_limit_store(struct device *dev, · 4 unmodified lines
561	{	617	{
562	struct zram *zram = dev_to_zram(dev);	618	struct zram *zram = dev_to_zram(dev);
563	u64 val;	619	u64 val;
564	ssize_t ret = -EINVAL;
565		620
566	if (kstrtoull(buf, 10, &val))	621	if (kstrtoull(buf, 10, &val))
567	return ret;	622	return -EINVAL;
568		623
569	/*	624	/*
570	* When the page size is greater than 4KB, if bd_wb_limit is set to	625	* When the page size is greater than 4KB, if bd_wb_limit is set to
@@ -576,12 +631,10 @@ static ssize_t writeback_limit_store(struct device *dev, · 5 unmodified lines
576	*/	631	*/
577	val = rounddown(val, PAGE_SIZE / 4096);	632	val = rounddown(val, PAGE_SIZE / 4096);
578		633
579	down_write(&zram->init_lock);	634	guard(rwsem_write)(&zram->dev_lock);
580	zram->bd_wb_limit = val;	635	zram->bd_wb_limit = val;
581	up_write(&zram->init_lock);
582	ret = len;
583		636
584	return ret;	637	return len;
585	}	638	}
586		639
587	static ssize_t writeback_limit_show(struct device *dev,	640	static ssize_t writeback_limit_show(struct device *dev,
@@ -590,9 +643,8 @@ static ssize_t writeback_limit_show(struct device *dev, · 2 unmodified lines
590	u64 val;	643	u64 val;
591	struct zram *zram = dev_to_zram(dev);	644	struct zram *zram = dev_to_zram(dev);
592		645
593	down_read(&zram->init_lock);	646	guard(rwsem_read)(&zram->dev_lock);
594	val = zram->bd_wb_limit;	647	val = zram->bd_wb_limit;
595	up_read(&zram->init_lock);
596		648
597	return sysfs_emit(buf, "%llu\n", val);	649	return sysfs_emit(buf, "%llu\n", val);
598	}	650	}
@@ -610,9 +662,8 @@ static ssize_t writeback_batch_size_store(struct device *dev, · 11 unmodified lines
610	if (!val)	662	if (!val)
611	return -EINVAL;	663	return -EINVAL;
612		664
613	down_write(&zram->init_lock);	665	guard(rwsem_write)(&zram->dev_lock);
614	zram->wb_batch_size = val;	666	zram->wb_batch_size = val;
615	up_write(&zram->init_lock);
616		667
617	return len;	668	return len;
618	}	669	}
@@ -624,9 +675,8 @@ static ssize_t writeback_batch_size_show(struct device *dev, · 5 unmodified lines
624	u32 val;	675	u32 val;
625	struct zram *zram = dev_to_zram(dev);	676	struct zram *zram = dev_to_zram(dev);
626		677
627	down_read(&zram->init_lock);	678	guard(rwsem_read)(&zram->dev_lock);
628	val = zram->wb_batch_size;	679	val = zram->wb_batch_size;
629	up_read(&zram->init_lock);
630		680
631	return sysfs_emit(buf, "%u\n", val);	681	return sysfs_emit(buf, "%u\n", val);
632	}	682	}
@@ -646,37 +696,33 @@ static void reset_bdev(struct zram *zram) · 13 unmodified lines
646	}	696	}
647		697
648	static ssize_t backing_dev_show(struct device *dev,	698	static ssize_t backing_dev_show(struct device *dev,
649	struct device_attribute attr, char buf)	699	struct device_attribute attr, char buf)
650	{	700	{
651	struct file *file;	701	struct file *file;
652	struct zram *zram = dev_to_zram(dev);	702	struct zram *zram = dev_to_zram(dev);
653	char *p;	703	char *p;
654	ssize_t ret;	704	ssize_t ret;
655		705
656	down_read(&zram->init_lock);	706	guard(rwsem_read)(&zram->dev_lock);
657	file = zram->backing_dev;	707	file = zram->backing_dev;
658	if (!file) {	708	if (!file) {
659	memcpy(buf, "none\n", 5);	709	memcpy(buf, "none\n", 5);
660	up_read(&zram->init_lock);
661	return 5;	710	return 5;
662	}	711	}
663		712
664	p = file_path(file, buf, PAGE_SIZE - 1);	713	p = file_path(file, buf, PAGE_SIZE - 1);
665	if (IS_ERR(p)) {	714	if (IS_ERR(p))
666	ret = PTR_ERR(p);	715	return PTR_ERR(p);
667	goto out;
668	}
669		716
670	ret = strlen(p);	717	ret = strlen(p);
671	memmove(buf, p, ret);	718	memmove(buf, p, ret);
672	buf[ret++] = '\n';	719	buf[ret++] = '\n';
673	out:
674	up_read(&zram->init_lock);
675	return ret;	720	return ret;
676	}	721	}
677		722
678	static ssize_t backing_dev_store(struct device *dev,	723	static ssize_t backing_dev_store(struct device *dev,
679	struct device_attribute attr, const char buf, size_t len)	724	struct device_attribute attr, const char buf,
		725	size_t len)
680	{	726	{
681	char *file_name;	727	char *file_name;
682	size_t sz;	728	size_t sz;
@@ -691,7 +737,7 @@ static ssize_t backing_dev_store(struct device *dev, · 8 unmodified lines
691	if (!file_name)	737	if (!file_name)
692	return -ENOMEM;	738	return -ENOMEM;
693		739
694	down_write(&zram->init_lock);	740	guard(rwsem_write)(&zram->dev_lock);
695	if (init_done(zram)) {	741	if (init_done(zram)) {
696	pr_info("Can't setup backing device for initialized device\n");	742	pr_info("Can't setup backing device for initialized device\n");
697	err = -EBUSY;	743	err = -EBUSY;
@@ -739,7 +785,6 @@ static ssize_t backing_dev_store(struct device *dev, · 41 unmodified lines
739	zram->backing_dev = backing_dev;	785	zram->backing_dev = backing_dev;
740	zram->bitmap = bitmap;	786	zram->bitmap = bitmap;
741	zram->nr_pages = nr_pages;	787	zram->nr_pages = nr_pages;
742	up_write(&zram->init_lock);
743		788
744	pr_info("setup backing device %s\n", file_name);	789	pr_info("setup backing device %s\n", file_name);
745	kfree(file_name);	790	kfree(file_name);
@@ -751,8 +796,6 @@ out: · 5 unmodified lines
751	if (backing_dev)	796	if (backing_dev)
752	filp_close(backing_dev, NULL);	797	filp_close(backing_dev, NULL);
753		798
754	up_write(&zram->init_lock);
755
756	kfree(file_name);	799	kfree(file_name);
757		800
758	return err;	801	return err;
@@ -780,18 +823,6 @@ static void zram_release_bdev_block(struct zram *zram, unsigned long blk_idx) · 21 unmodified lines
780	atomic64_dec(&zram->stats.bd_count);	823	atomic64_dec(&zram->stats.bd_count);
781	}	824	}
782		825
783	static void read_from_bdev_async(struct zram zram, struct page page,
784	unsigned long entry, struct bio *parent)
785	{
786	struct bio *bio;
787
788	bio = bio_alloc(zram->bdev, 1, parent->bi_opf, GFP_NOIO);
789	bio->bi_iter.bi_sector = entry * (PAGE_SIZE >> 9);
790	__bio_add_page(bio, page, PAGE_SIZE, 0);
791	bio_chain(bio, parent);
792	submit_bio(bio);
793	}
794
795	static void release_wb_req(struct zram_wb_req *req)	826	static void release_wb_req(struct zram_wb_req *req)
796	{	827	{
797	__free_page(req->page);	828	__free_page(req->page);
@@ -870,7 +901,7 @@ release_wb_ctl: · 72 unmodified lines
870		901
871	static void zram_account_writeback_rollback(struct zram *zram)	902	static void zram_account_writeback_rollback(struct zram *zram)
872	{	903	{
873	lockdep_assert_held_read(&zram->init_lock);	904	lockdep_assert_held_write(&zram->dev_lock);
874		905
875	if (zram->wb_limit_enable)	906	if (zram->wb_limit_enable)
876	zram->bd_wb_limit += 1UL << (PAGE_SHIFT - 12);	907	zram->bd_wb_limit += 1UL << (PAGE_SHIFT - 12);
@@ -878,7 +909,7 @@ static void zram_account_writeback_rollback(struct zram *zram) · 1 unmodified line
878		909
879	static void zram_account_writeback_submit(struct zram *zram)	910	static void zram_account_writeback_submit(struct zram *zram)
880	{	911	{
881	lockdep_assert_held_read(&zram->init_lock);	912	lockdep_assert_held_write(&zram->dev_lock);
882		913
883	if (zram->wb_limit_enable && zram->bd_wb_limit > 0)	914	if (zram->wb_limit_enable && zram->bd_wb_limit > 0)
884	zram->bd_wb_limit -= 1UL << (PAGE_SHIFT - 12);	915	zram->bd_wb_limit -= 1UL << (PAGE_SHIFT - 12);
@@ -886,8 +917,9 @@ static void zram_account_writeback_submit(struct zram *zram) · 1 unmodified line
886		917
887	static int zram_writeback_complete(struct zram zram, struct zram_wb_req req)	918	static int zram_writeback_complete(struct zram zram, struct zram_wb_req req)
888	{	919	{
889	u32 index = req->pps->index;	920	u32 size, index = req->pps->index;
890	int err;	921	int err, prio;
		922	bool huge;
891		923
892	err = blk_status_to_errno(req->bio.bi_status);	924	err = blk_status_to_errno(req->bio.bi_status);
893	if (err) {	925	if (err) {
@@ -901,7 +933,7 @@ static int zram_writeback_complete(struct zram zram, struct zram_wb_req req) · 7 unmodified lines
901	}	933	}
902		934
903	atomic64_inc(&zram->stats.bd_writes);	935	atomic64_inc(&zram->stats.bd_writes);
904	zram_slot_lock(zram, index);	936	slot_lock(zram, index);
905	/*	937	/*
906	* We release slot lock during writeback so slot can change under us:	938	* We release slot lock during writeback so slot can change under us:
907	* slot_free() or slot_free() and zram_write_page(). In both cases	939	* slot_free() or slot_free() and zram_write_page(). In both cases
@@ -909,18 +941,36 @@ static int zram_writeback_complete(struct zram zram, struct zram_wb_req req) · 1 unmodified line
909	* set ZRAM_PP_SLOT on such slots until current post-processing	941	* set ZRAM_PP_SLOT on such slots until current post-processing
910	* finishes.	942	* finishes.
911	*/	943	*/
912	if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) {	944	if (!test_slot_flag(zram, index, ZRAM_PP_SLOT)) {
913	zram_release_bdev_block(zram, req->blk_idx);	945	zram_release_bdev_block(zram, req->blk_idx);
914	goto out;	946	goto out;
915	}	947	}
916		948
917	zram_free_page(zram, index);	949	if (zram->wb_compressed) {
918	zram_set_flag(zram, index, ZRAM_WB);	950	/*
919	zram_set_handle(zram, index, req->blk_idx);	951	* ZRAM_WB slots get freed, we need to preserve data required
		952	* for read decompression.
		953	*/
		954	size = get_slot_size(zram, index);
		955	prio = get_slot_comp_priority(zram, index);
		956	huge = test_slot_flag(zram, index, ZRAM_HUGE);
		957	}
		958
		959	slot_free(zram, index);
		960	set_slot_flag(zram, index, ZRAM_WB);
		961	set_slot_handle(zram, index, req->blk_idx);
		962
		963	if (zram->wb_compressed) {
		964	if (huge)
		965	set_slot_flag(zram, index, ZRAM_HUGE);
		966	set_slot_size(zram, index, size);
		967	set_slot_comp_priority(zram, index, prio);
		968	}
		969
920	atomic64_inc(&zram->stats.pages_stored);	970	atomic64_inc(&zram->stats.pages_stored);
921		971
922	out:	972	out:
923	zram_slot_unlock(zram, index);	973	slot_unlock(zram, index);
924	return 0;	974	return 0;
925	}	975	}
926		976
@@ -1041,18 +1091,22 @@ static int zram_writeback_slots(struct zram *zram, · 114 unmodified lines
1041	}	1091	}
1042		1092
1043	index = pps->index;	1093	index = pps->index;
1044	zram_slot_lock(zram, index);	1094	slot_lock(zram, index);
1045	/*	1095	/*
1046	* scan_slots() sets ZRAM_PP_SLOT and releases slot lock, so	1096	* scan_slots() sets ZRAM_PP_SLOT and releases slot lock, so
1047	* slots can change in the meantime. If slots are accessed or	1097	* slots can change in the meantime. If slots are accessed or
1048	* freed they lose ZRAM_PP_SLOT flag and hence we don't	1098	* freed they lose ZRAM_PP_SLOT flag and hence we don't
1049	* post-process them.	1099	* post-process them.
1050	*/	1100	*/
1051	if (!zram_test_flag(zram, index, ZRAM_PP_SLOT))	1101	if (!test_slot_flag(zram, index, ZRAM_PP_SLOT))
1052	goto next;	1102	goto next;
1053	if (zram_read_from_zspool(zram, req->page, index))	1103	if (zram->wb_compressed)
		1104	err = read_from_zspool_raw(zram, req->page, index);
		1105	else
		1106	err = read_from_zspool(zram, req->page, index);
		1107	if (err)
1054	goto next;	1108	goto next;
1055	zram_slot_unlock(zram, index);	1109	slot_unlock(zram, index);
1056		1110
1057	/*	1111	/*
1058	* From now on pp-slot is owned by the req, remove it from	1112	* From now on pp-slot is owned by the req, remove it from
@@ -1074,7 +1128,7 @@ static int zram_writeback_slots(struct zram *zram, · 15 unmodified lines
1074	continue;	1128	continue;
1075		1129
1076	next:	1130	next:
1077	zram_slot_unlock(zram, index);	1131	slot_unlock(zram, index);
1078	release_pp_slot(zram, pps);	1132	release_pp_slot(zram, pps);
1079	}	1133	}
1080		1134
@@ -1167,27 +1221,27 @@ static int scan_slots_for_writeback(struct zram *zram, u32 mode, · 86 unmodified lines
1167	while (index < hi) {	1221	while (index < hi) {
1168	bool ok = true;	1222	bool ok = true;
1169		1223
1170	zram_slot_lock(zram, index);	1224	slot_lock(zram, index);
1171	if (!zram_allocated(zram, index))	1225	if (!slot_allocated(zram, index))
1172	goto next;	1226	goto next;
1173		1227
1174	if (zram_test_flag(zram, index, ZRAM_WB) \|\|	1228	if (test_slot_flag(zram, index, ZRAM_WB) \|\|
1175	zram_test_flag(zram, index, ZRAM_SAME))	1229	test_slot_flag(zram, index, ZRAM_SAME))
1176	goto next;	1230	goto next;
1177		1231
1178	if (mode & IDLE_WRITEBACK &&	1232	if (mode & IDLE_WRITEBACK &&
1179	!zram_test_flag(zram, index, ZRAM_IDLE))	1233	!test_slot_flag(zram, index, ZRAM_IDLE))
1180	goto next;	1234	goto next;
1181	if (mode & HUGE_WRITEBACK &&	1235	if (mode & HUGE_WRITEBACK &&
1182	!zram_test_flag(zram, index, ZRAM_HUGE))	1236	!test_slot_flag(zram, index, ZRAM_HUGE))
1183	goto next;	1237	goto next;
1184	if (mode & INCOMPRESSIBLE_WRITEBACK &&	1238	if (mode & INCOMPRESSIBLE_WRITEBACK &&
1185	!zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE))	1239	!test_slot_flag(zram, index, ZRAM_INCOMPRESSIBLE))
1186	goto next;	1240	goto next;
1187		1241
1188	ok = place_pp_slot(zram, ctl, index);	1242	ok = place_pp_slot(zram, ctl, index);
1189	next:	1243	next:
1190	zram_slot_unlock(zram, index);	1244	slot_unlock(zram, index);
1191	if (!ok)	1245	if (!ok)
1192	break;	1246	break;
1193	index++;	1247	index++;
@@ -1209,33 +1263,21 @@ static ssize_t writeback_store(struct device *dev, · 15 unmodified lines
1209	ssize_t ret = len;	1263	ssize_t ret = len;
1210	int err, mode = 0;	1264	int err, mode = 0;
1211		1265
1212	down_read(&zram->init_lock);	1266	guard(rwsem_write)(&zram->dev_lock);
1213	if (!init_done(zram)) {	1267	if (!init_done(zram))
1214	up_read(&zram->init_lock);
1215	return -EINVAL;	1268	return -EINVAL;
1216	}
1217
1218	/* Do not permit concurrent post-processing actions. */
1219	if (atomic_xchg(&zram->pp_in_progress, 1)) {
1220	up_read(&zram->init_lock);
1221	return -EAGAIN;
1222	}
1223		1269
1224	if (!zram->backing_dev) {	1270	if (!zram->backing_dev)
1225	ret = -ENODEV;	1271	return -ENODEV;
1226	goto release_init_lock;
1227	}
1228		1272
1229	pp_ctl = init_pp_ctl();	1273	pp_ctl = init_pp_ctl();
1230	if (!pp_ctl) {	1274	if (!pp_ctl)
1231	ret = -ENOMEM;	1275	return -ENOMEM;
1232	goto release_init_lock;
1233	}
1234		1276
1235	wb_ctl = init_wb_ctl(zram);	1277	wb_ctl = init_wb_ctl(zram);
1236	if (!wb_ctl) {	1278	if (!wb_ctl) {
1237	ret = -ENOMEM;	1279	ret = -ENOMEM;
1238	goto release_init_lock;	1280	goto out;
1239	}	1281	}
1240		1282
1241	args = skip_spaces(buf);	1283	args = skip_spaces(buf);
@@ -1259,7 +1301,7 @@ static ssize_t writeback_store(struct device *dev, · 17 unmodified lines
1259	err = parse_mode(param, &mode);	1301	err = parse_mode(param, &mode);
1260	if (err) {	1302	if (err) {
1261	ret = err;	1303	ret = err;
1262	goto release_init_lock;	1304	goto out;
1263	}	1305	}
1264		1306
1265	scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);	1307	scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);
@@ -1270,7 +1312,7 @@ static ssize_t writeback_store(struct device *dev, · 4 unmodified lines
1270	err = parse_mode(val, &mode);	1312	err = parse_mode(val, &mode);
1271	if (err) {	1313	if (err) {
1272	ret = err;	1314	ret = err;
1273	goto release_init_lock;	1315	goto out;
1274	}	1316	}
1275		1317
1276	scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);	1318	scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);
@@ -1281,7 +1323,7 @@ static ssize_t writeback_store(struct device *dev, · 4 unmodified lines
1281	err = parse_page_index(val, nr_pages, &lo, &hi);	1323	err = parse_page_index(val, nr_pages, &lo, &hi);
1282	if (err) {	1324	if (err) {
1283	ret = err;	1325	ret = err;
1284	goto release_init_lock;	1326	goto out;
1285	}	1327	}
1286		1328
1287	scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);	1329	scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);
@@ -1292,7 +1334,7 @@ static ssize_t writeback_store(struct device *dev, · 4 unmodified lines
1292	err = parse_page_indexes(val, nr_pages, &lo, &hi);	1334	err = parse_page_indexes(val, nr_pages, &lo, &hi);
1293	if (err) {	1335	if (err) {
1294	ret = err;	1336	ret = err;
1295	goto release_init_lock;	1337	goto out;
1296	}	1338	}
1297		1339
1298	scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);	1340	scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl);
@@ -1304,33 +1346,147 @@ static ssize_t writeback_store(struct device *dev, · 5 unmodified lines
1304	if (err)	1346	if (err)
1305	ret = err;	1347	ret = err;
1306		1348
1307	release_init_lock:	1349	out:
1308	release_pp_ctl(zram, pp_ctl);	1350	release_pp_ctl(zram, pp_ctl);
1309	release_wb_ctl(wb_ctl);	1351	release_wb_ctl(wb_ctl);
1310	atomic_set(&zram->pp_in_progress, 0);
1311	up_read(&zram->init_lock);
1312		1352
1313	return ret;	1353	return ret;
1314	}	1354	}
1315		1355
1316	struct zram_work {	1356	static int decompress_bdev_page(struct zram zram, struct page page, u32 index)
1317	struct work_struct work;	1357	{
1318	struct zram *zram;	1358	struct zcomp_strm *zstrm;
1319	unsigned long entry;	1359	unsigned int size;
1320	struct page *page;	1360	int ret, prio;
1321	int error;	1361	void *src;
1322	};	1362
		1363	slot_lock(zram, index);
		1364	/* Since slot was unlocked we need to make sure it's still ZRAM_WB */
		1365	if (!test_slot_flag(zram, index, ZRAM_WB)) {
		1366	slot_unlock(zram, index);
		1367	/* We read some stale data, zero it out */
		1368	memset_page(page, 0, 0, PAGE_SIZE);
		1369	return -EIO;
		1370	}
		1371
		1372	if (test_slot_flag(zram, index, ZRAM_HUGE)) {
		1373	slot_unlock(zram, index);
		1374	return 0;
		1375	}
		1376
		1377	size = get_slot_size(zram, index);
		1378	prio = get_slot_comp_priority(zram, index);
		1379
		1380	zstrm = zcomp_stream_get(zram->comps[prio]);
		1381	src = kmap_local_page(page);
		1382	ret = zcomp_decompress(zram->comps[prio], zstrm, src, size,
		1383	zstrm->local_copy);
		1384	if (!ret)
		1385	copy_page(src, zstrm->local_copy);
		1386	kunmap_local(src);
		1387	zcomp_stream_put(zstrm);
		1388	slot_unlock(zram, index);
		1389
		1390	return ret;
		1391	}
		1392
		1393	static void zram_deferred_decompress(struct work_struct *w)
		1394	{
		1395	struct zram_rb_req *req = container_of(w, struct zram_rb_req, work);
		1396	struct page *page = bio_first_page_all(req->bio);
		1397	struct zram *zram = req->zram;
		1398	u32 index = req->index;
		1399	int ret;
		1400
		1401	ret = decompress_bdev_page(zram, page, index);
		1402	if (ret)
		1403	req->parent->bi_status = BLK_STS_IOERR;
		1404
		1405	/* Decrement parent's ->remaining */
		1406	bio_endio(req->parent);
		1407	bio_put(req->bio);
		1408	kfree(req);
		1409	}
1323		1410
1324	static void zram_sync_read(struct work_struct *work)	1411	static void zram_async_read_endio(struct bio *bio)
1325	{	1412	{
1326	struct zram_work *zw = container_of(work, struct zram_work, work);	1413	struct zram_rb_req *req = bio->bi_private;
		1414	struct zram *zram = req->zram;
		1415
		1416	if (bio->bi_status) {
		1417	req->parent->bi_status = bio->bi_status;
		1418	bio_endio(req->parent);
		1419	bio_put(bio);
		1420	kfree(req);
		1421	return;
		1422	}
		1423
		1424	/*
		1425	* NOTE: zram_async_read_endio() is not exactly right place for this.
		1426	* Ideally, we need to do it after ZRAM_WB check, but this requires
		1427	* us to use wq path even on systems that don't enable compressed
		1428	* writeback, because we cannot take slot-lock in the current context.
		1429	*
		1430	* Keep the existing behavior for now.
		1431	*/
		1432	if (zram->wb_compressed == false) {
		1433	/* No decompression needed, complete the parent IO */
		1434	bio_endio(req->parent);
		1435	bio_put(bio);
		1436	kfree(req);
		1437	return;
		1438	}
		1439
		1440	/*
		1441	* zram decompression is sleepable, so we need to deffer it to
		1442	* a preemptible context.
		1443	*/
		1444	INIT_WORK(&req->work, zram_deferred_decompress);
		1445	queue_work(system_highpri_wq, &req->work);
		1446	}
		1447
		1448	static void read_from_bdev_async(struct zram zram, struct page page,
		1449	u32 index, unsigned long blk_idx,
		1450	struct bio *parent)
		1451	{
		1452	struct zram_rb_req *req;
		1453	struct bio *bio;
		1454
		1455	req = kmalloc(sizeof(*req), GFP_NOIO);
		1456	if (!req)
		1457	return;
		1458
		1459	bio = bio_alloc(zram->bdev, 1, parent->bi_opf, GFP_NOIO);
		1460	if (!bio) {
		1461	kfree(req);
		1462	return;
		1463	}
		1464
		1465	req->zram = zram;
		1466	req->index = index;
		1467	req->blk_idx = blk_idx;
		1468	req->bio = bio;
		1469	req->parent = parent;
		1470
		1471	bio->bi_iter.bi_sector = blk_idx * (PAGE_SIZE >> 9);
		1472	bio->bi_private = req;
		1473	bio->bi_end_io = zram_async_read_endio;
		1474
		1475	__bio_add_page(bio, page, PAGE_SIZE, 0);
		1476	bio_inc_remaining(parent);
		1477	submit_bio(bio);
		1478	}
		1479
		1480	static void zram_sync_read(struct work_struct *w)
		1481	{
		1482	struct zram_rb_req *req = container_of(w, struct zram_rb_req, work);
1327	struct bio_vec bv;	1483	struct bio_vec bv;
1328	struct bio bio;	1484	struct bio bio;
1329		1485
1330	bio_init(&bio, zw->zram->bdev, &bv, 1, REQ_OP_READ);	1486	bio_init(&bio, req->zram->bdev, &bv, 1, REQ_OP_READ);
1331	bio.bi_iter.bi_sector = zw->entry * (PAGE_SIZE >> 9);	1487	bio.bi_iter.bi_sector = req->blk_idx * (PAGE_SIZE >> 9);
1332	__bio_add_page(&bio, zw->page, PAGE_SIZE, 0);	1488	__bio_add_page(&bio, req->page, PAGE_SIZE, 0);
1333	zw->error = submit_bio_wait(&bio);	1489	req->error = submit_bio_wait(&bio);
1334	}	1490	}
1335		1491
1336	/*	1492	/*
@@ -1338,39 +1494,42 @@ static void zram_sync_read(struct work_struct *work) · 1 unmodified line
1338	* chained IO with parent IO in same context, it's a deadlock. To avoid that,	1494	* chained IO with parent IO in same context, it's a deadlock. To avoid that,
1339	* use a worker thread context.	1495	* use a worker thread context.
1340	*/	1496	*/
1341	static int read_from_bdev_sync(struct zram zram, struct page page,	1497	static int read_from_bdev_sync(struct zram zram, struct page page, u32 index,
1342	unsigned long entry)	1498	unsigned long blk_idx)
1343	{	1499	{
1344	struct zram_work work;	1500	struct zram_rb_req req;
		1501
		1502	req.page = page;
		1503	req.zram = zram;
		1504	req.blk_idx = blk_idx;
1345		1505
1346	work.page = page;	1506	INIT_WORK_ONSTACK(&req.work, zram_sync_read);
1347	work.zram = zram;	1507	queue_work(system_dfl_wq, &req.work);
1348	work.entry = entry;	1508	flush_work(&req.work);
		1509	destroy_work_on_stack(&req.work);
1349		1510
1350	INIT_WORK_ONSTACK(&work.work, zram_sync_read);	1511	if (req.error \|\| zram->wb_compressed == false)
1351	queue_work(system_dfl_wq, &work.work);	1512	return req.error;
1352	flush_work(&work.work);
1353	destroy_work_on_stack(&work.work);
1354		1513
1355	return work.error;	1514	return decompress_bdev_page(zram, page, index);
1356	}	1515	}
1357		1516
1358	static int read_from_bdev(struct zram zram, struct page page,	1517	static int read_from_bdev(struct zram zram, struct page page, u32 index,
1359	unsigned long entry, struct bio *parent)	1518	unsigned long blk_idx, struct bio *parent)
1360	{	1519	{
1361	atomic64_inc(&zram->stats.bd_reads);	1520	atomic64_inc(&zram->stats.bd_reads);
1362	if (!parent) {	1521	if (!parent) {
1363	if (WARN_ON_ONCE(!IS_ENABLED(ZRAM_PARTIAL_IO)))	1522	if (WARN_ON_ONCE(!IS_ENABLED(ZRAM_PARTIAL_IO)))
1364	return -EIO;	1523	return -EIO;
1365	return read_from_bdev_sync(zram, page, entry);	1524	return read_from_bdev_sync(zram, page, index, blk_idx);
1366	}	1525	}
1367	read_from_bdev_async(zram, page, entry, parent);	1526	read_from_bdev_async(zram, page, index, blk_idx, parent);
1368	return 0;	1527	return 0;
1369	}	1528	}
1370	#else	1529	#else
1371	static inline void reset_bdev(struct zram *zram) {};	1530	static inline void reset_bdev(struct zram *zram) {};
1372	static int read_from_bdev(struct zram zram, struct page page,	1531	static int read_from_bdev(struct zram zram, struct page page, u32 index,
1373	unsigned long entry, struct bio *parent)	1532	unsigned long blk_idx, struct bio *parent)
1374	{	1533	{
1375	return -EIO;	1534	return -EIO;
1376	}	1535	}
@@ -1401,15 +1560,13 @@ static ssize_t read_block_state(struct file file, char __user buf, · 24 unmodified lines
1401	ssize_t index, written = 0;	1560	ssize_t index, written = 0;
1402	struct zram *zram = file->private_data;	1561	struct zram *zram = file->private_data;
1403	unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;	1562	unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
1404	struct timespec64 ts;
1405		1563
1406	kbuf = kvmalloc(count, GFP_KERNEL);	1564	kbuf = kvmalloc(count, GFP_KERNEL);
1407	if (!kbuf)	1565	if (!kbuf)
1408	return -ENOMEM;	1566	return -ENOMEM;
1409		1567
1410	down_read(&zram->init_lock);	1568	guard(rwsem_read)(&zram->dev_lock);
1411	if (!init_done(zram)) {	1569	if (!init_done(zram)) {
1412	up_read(&zram->init_lock);
1413	kvfree(kbuf);	1570	kvfree(kbuf);
1414	return -EINVAL;	1571	return -EINVAL;
1415	}	1572	}
@@ -1417,35 +1574,32 @@ static ssize_t read_block_state(struct file file, char __user buf, · 1 unmodified line
1417	for (index = *ppos; index < nr_pages; index++) {	1574	for (index = *ppos; index < nr_pages; index++) {
1418	int copied;	1575	int copied;
1419		1576
1420	zram_slot_lock(zram, index);	1577	slot_lock(zram, index);
1421	if (!zram_allocated(zram, index))	1578	if (!slot_allocated(zram, index))
1422	goto next;	1579	goto next;
1423		1580
1424	ts = ktime_to_timespec64(zram->table[index].ac_time);
1425	copied = snprintf(kbuf + written, count,	1581	copied = snprintf(kbuf + written, count,
1426	"%12zd %12lld.%06lu %c%c%c%c%c%c\n",	1582	"%12zd %12u.%06d %c%c%c%c%c%c\n",
1427	index, (s64)ts.tv_sec,	1583	index, zram->table[index].attr.ac_time, 0,
1428	ts.tv_nsec / NSEC_PER_USEC,	1584	test_slot_flag(zram, index, ZRAM_SAME) ? 's' : '.',
1429	zram_test_flag(zram, index, ZRAM_SAME) ? 's' : '.',	1585	test_slot_flag(zram, index, ZRAM_WB) ? 'w' : '.',
1430	zram_test_flag(zram, index, ZRAM_WB) ? 'w' : '.',	1586	test_slot_flag(zram, index, ZRAM_HUGE) ? 'h' : '.',
1431	zram_test_flag(zram, index, ZRAM_HUGE) ? 'h' : '.',	1587	test_slot_flag(zram, index, ZRAM_IDLE) ? 'i' : '.',
1432	zram_test_flag(zram, index, ZRAM_IDLE) ? 'i' : '.',	1588	get_slot_comp_priority(zram, index) ? 'r' : '.',
1433	zram_get_priority(zram, index) ? 'r' : '.',	1589	test_slot_flag(zram, index,
1434	zram_test_flag(zram, index,
1435	ZRAM_INCOMPRESSIBLE) ? 'n' : '.');	1590	ZRAM_INCOMPRESSIBLE) ? 'n' : '.');
1436		1591
1437	if (count <= copied) {	1592	if (count <= copied) {
1438	zram_slot_unlock(zram, index);	1593	slot_unlock(zram, index);
1439	break;	1594	break;
1440	}	1595	}
1441	written += copied;	1596	written += copied;
1442	count -= copied;	1597	count -= copied;
1443	next:	1598	next:
1444	zram_slot_unlock(zram, index);	1599	slot_unlock(zram, index);
1445	*ppos += 1;	1600	*ppos += 1;
1446	}	1601	}
1447		1602
1448	up_read(&zram->init_lock);
1449	if (copy_to_user(buf, kbuf, written))	1603	if (copy_to_user(buf, kbuf, written))
1450	written = -EFAULT;	1604	written = -EFAULT;
1451	kvfree(kbuf);	1605	kvfree(kbuf);
@@ -1512,16 +1666,14 @@ static int __comp_algorithm_store(struct zram zram, u32 prio, const char buf) · 60 unmodified lines
1512	return -EINVAL;	1666	return -EINVAL;
1513	}	1667	}
1514		1668
1515	down_write(&zram->init_lock);	1669	guard(rwsem_write)(&zram->dev_lock);
1516	if (init_done(zram)) {	1670	if (init_done(zram)) {
1517	up_write(&zram->init_lock);
1518	kfree(compressor);	1671	kfree(compressor);
1519	pr_info("Can't change algorithm for initialized device\n");	1672	pr_info("Can't change algorithm for initialized device\n");
1520	return -EBUSY;	1673	return -EBUSY;
1521	}	1674	}
1522		1675
1523	comp_algorithm_set(zram, prio, compressor);	1676	comp_algorithm_set(zram, prio, compressor);
1524	up_write(&zram->init_lock);
1525	return 0;	1677	return 0;
1526	}	1678	}
1527		1679
@@ -1642,9 +1794,8 @@ static ssize_t comp_algorithm_show(struct device *dev, · 114 unmodified lines
1642	struct zram *zram = dev_to_zram(dev);	1794	struct zram *zram = dev_to_zram(dev);
1643	ssize_t sz;	1795	ssize_t sz;
1644		1796
1645	down_read(&zram->init_lock);	1797	guard(rwsem_read)(&zram->dev_lock);
1646	sz = zcomp_available_show(zram->comp_algs[ZRAM_PRIMARY_COMP], buf, 0);	1798	sz = zcomp_available_show(zram->comp_algs[ZRAM_PRIMARY_COMP], buf, 0);
1647	up_read(&zram->init_lock);
1648	return sz;	1799	return sz;
1649	}	1800	}
1650		1801
@@ -1669,7 +1820,7 @@ static ssize_t recomp_algorithm_show(struct device *dev, · 18 unmodified lines
1669	ssize_t sz = 0;	1820	ssize_t sz = 0;
1670	u32 prio;	1821	u32 prio;
1671		1822
1672	down_read(&zram->init_lock);	1823	guard(rwsem_read)(&zram->dev_lock);
1673	for (prio = ZRAM_SECONDARY_COMP; prio < ZRAM_MAX_COMPS; prio++) {	1824	for (prio = ZRAM_SECONDARY_COMP; prio < ZRAM_MAX_COMPS; prio++) {
1674	if (!zram->comp_algs[prio])	1825	if (!zram->comp_algs[prio])
1675	continue;	1826	continue;
@@ -1677,7 +1828,6 @@ static ssize_t recomp_algorithm_show(struct device *dev, · 1 unmodified line
1677	sz += sysfs_emit_at(buf, sz, "#%d: ", prio);	1828	sz += sysfs_emit_at(buf, sz, "#%d: ", prio);
1678	sz += zcomp_available_show(zram->comp_algs[prio], buf, sz);	1829	sz += zcomp_available_show(zram->comp_algs[prio], buf, sz);
1679	}	1830	}
1680	up_read(&zram->init_lock);
1681	return sz;	1831	return sz;
1682	}	1832	}
1683		1833
@@ -1723,42 +1873,38 @@ static ssize_t recomp_algorithm_store(struct device *dev, · 39 unmodified lines
1723	}	1873	}
1724	#endif	1874	#endif
1725		1875
1726	static ssize_t compact_store(struct device *dev,	1876	static ssize_t compact_store(struct device dev, struct device_attribute attr,
1727	struct device_attribute attr, const char buf, size_t len)	1877	const char *buf, size_t len)
1728	{	1878	{
1729	struct zram *zram = dev_to_zram(dev);	1879	struct zram *zram = dev_to_zram(dev);
1730		1880
1731	down_read(&zram->init_lock);	1881	guard(rwsem_read)(&zram->dev_lock);
1732	if (!init_done(zram)) {	1882	if (!init_done(zram))
1733	up_read(&zram->init_lock);
1734	return -EINVAL;	1883	return -EINVAL;
1735	}
1736		1884
1737	zs_compact(zram->mem_pool);	1885	zs_compact(zram->mem_pool);
1738	up_read(&zram->init_lock);
1739		1886
1740	return len;	1887	return len;
1741	}	1888	}
1742		1889
1743	static ssize_t io_stat_show(struct device *dev,	1890	static ssize_t io_stat_show(struct device dev, struct device_attribute attr,
1744	struct device_attribute attr, char buf)	1891	char *buf)
1745	{	1892	{
1746	struct zram *zram = dev_to_zram(dev);	1893	struct zram *zram = dev_to_zram(dev);
1747	ssize_t ret;	1894	ssize_t ret;
1748		1895
1749	down_read(&zram->init_lock);	1896	guard(rwsem_read)(&zram->dev_lock);
1750	ret = sysfs_emit(buf,	1897	ret = sysfs_emit(buf,
1751	"%8llu %8llu 0 %8llu\n",	1898	"%8llu %8llu 0 %8llu\n",
1752	(u64)atomic64_read(&zram->stats.failed_reads),	1899	(u64)atomic64_read(&zram->stats.failed_reads),
1753	(u64)atomic64_read(&zram->stats.failed_writes),	1900	(u64)atomic64_read(&zram->stats.failed_writes),
1754	(u64)atomic64_read(&zram->stats.notify_free));	1901	(u64)atomic64_read(&zram->stats.notify_free));
1755	up_read(&zram->init_lock);
1756		1902
1757	return ret;	1903	return ret;
1758	}	1904	}
1759		1905
1760	static ssize_t mm_stat_show(struct device *dev,	1906	static ssize_t mm_stat_show(struct device dev, struct device_attribute attr,
1761	struct device_attribute attr, char buf)	1907	char *buf)
1762	{	1908	{
1763	struct zram *zram = dev_to_zram(dev);	1909	struct zram *zram = dev_to_zram(dev);
1764	struct zs_pool_stats pool_stats;	1910	struct zs_pool_stats pool_stats;
@@ -1768,7 +1914,7 @@ static ssize_t mm_stat_show(struct device *dev, · 3 unmodified lines
1768		1914
1769	memset(&pool_stats, 0x00, sizeof(struct zs_pool_stats));	1915	memset(&pool_stats, 0x00, sizeof(struct zs_pool_stats));
1770		1916
1771	down_read(&zram->init_lock);	1917	guard(rwsem_read)(&zram->dev_lock);
1772	if (init_done(zram)) {	1918	if (init_done(zram)) {
1773	mem_used = zs_get_total_pages(zram->mem_pool);	1919	mem_used = zs_get_total_pages(zram->mem_pool);
1774	zs_pool_stats(zram->mem_pool, &pool_stats);	1920	zs_pool_stats(zram->mem_pool, &pool_stats);
@@ -1788,55 +1934,26 @@ static ssize_t mm_stat_show(struct device *dev, · 13 unmodified lines
1788	atomic_long_read(&pool_stats.pages_compacted),	1934	atomic_long_read(&pool_stats.pages_compacted),
1789	(u64)atomic64_read(&zram->stats.huge_pages),	1935	(u64)atomic64_read(&zram->stats.huge_pages),
1790	(u64)atomic64_read(&zram->stats.huge_pages_since));	1936	(u64)atomic64_read(&zram->stats.huge_pages_since));
1791	up_read(&zram->init_lock);
1792		1937
1793	return ret;	1938	return ret;
1794	}	1939	}
1795		1940
1796	#ifdef CONFIG_ZRAM_WRITEBACK
1797	#define FOUR_K(x) ((x) * (1 << (PAGE_SHIFT - 12)))
1798	static ssize_t bd_stat_show(struct device *dev,
1799	struct device_attribute attr, char buf)
1800	{
1801	struct zram *zram = dev_to_zram(dev);
1802	ssize_t ret;
1803
1804	down_read(&zram->init_lock);
1805	ret = sysfs_emit(buf,
1806	"%8llu %8llu %8llu\n",
1807	FOUR_K((u64)atomic64_read(&zram->stats.bd_count)),
1808	FOUR_K((u64)atomic64_read(&zram->stats.bd_reads)),
1809	FOUR_K((u64)atomic64_read(&zram->stats.bd_writes)));
1810	up_read(&zram->init_lock);
1811
1812	return ret;
1813	}
1814	#endif
1815
1816	static ssize_t debug_stat_show(struct device *dev,	1941	static ssize_t debug_stat_show(struct device *dev,
1817	struct device_attribute attr, char buf)	1942	struct device_attribute attr, char buf)
1818	{	1943	{
1819	int version = 1;	1944	int version = 1;
1820	struct zram *zram = dev_to_zram(dev);	1945	struct zram *zram = dev_to_zram(dev);
1821	ssize_t ret;	1946	ssize_t ret;
1822		1947
1823	down_read(&zram->init_lock);	1948	guard(rwsem_read)(&zram->dev_lock);
1824	ret = sysfs_emit(buf,	1949	ret = sysfs_emit(buf,
1825	"version: %d\n0 %8llu\n",	1950	"version: %d\n0 %8llu\n",
1826	version,	1951	version,
1827	(u64)atomic64_read(&zram->stats.miss_free));	1952	(u64)atomic64_read(&zram->stats.miss_free));
1828	up_read(&zram->init_lock);
1829		1953
1830	return ret;	1954	return ret;
1831	}	1955	}
1832		1956
1833	static DEVICE_ATTR_RO(io_stat);
1834	static DEVICE_ATTR_RO(mm_stat);
1835	#ifdef CONFIG_ZRAM_WRITEBACK
1836	static DEVICE_ATTR_RO(bd_stat);
1837	#endif
1838	static DEVICE_ATTR_RO(debug_stat);
1839
1840	static void zram_meta_free(struct zram *zram, u64 disksize)	1957	static void zram_meta_free(struct zram *zram, u64 disksize)
1841	{	1958	{
1842	size_t num_pages = disksize >> PAGE_SHIFT;	1959	size_t num_pages = disksize >> PAGE_SHIFT;
@@ -1847,7 +1964,7 @@ static void zram_meta_free(struct zram *zram, u64 disksize) · 4 unmodified lines
1847		1964
1848	/* Free all pages that are still in this zram device */	1965	/* Free all pages that are still in this zram device */
1849	for (index = 0; index < num_pages; index++)	1966	for (index = 0; index < num_pages; index++)
1850	zram_free_page(zram, index);	1967	slot_free(zram, index);
1851		1968
1852	zs_destroy_pool(zram->mem_pool);	1969	zs_destroy_pool(zram->mem_pool);
1853	vfree(zram->table);	1970	vfree(zram->table);
@@ -1874,32 +1991,32 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) · 20 unmodified lines
1874	huge_class_size = zs_huge_class_size(zram->mem_pool);	1991	huge_class_size = zs_huge_class_size(zram->mem_pool);
1875		1992
1876	for (index = 0; index < num_pages; index++)	1993	for (index = 0; index < num_pages; index++)
1877	zram_slot_lock_init(zram, index);	1994	slot_lock_init(zram, index);
1878		1995
1879	return true;	1996	return true;
1880	}	1997	}
1881		1998
1882	static void zram_free_page(struct zram *zram, size_t index)	1999	static void slot_free(struct zram *zram, u32 index)
1883	{	2000	{
1884	unsigned long handle;	2001	unsigned long handle;
1885		2002
1886	#ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME	2003	#ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME
1887	zram->table[index].ac_time = 0;	2004	zram->table[index].attr.ac_time = 0;
1888	#endif	2005	#endif
1889		2006
1890	zram_clear_flag(zram, index, ZRAM_IDLE);	2007	clear_slot_flag(zram, index, ZRAM_IDLE);
1891	zram_clear_flag(zram, index, ZRAM_INCOMPRESSIBLE);	2008	clear_slot_flag(zram, index, ZRAM_INCOMPRESSIBLE);
1892	zram_clear_flag(zram, index, ZRAM_PP_SLOT);	2009	clear_slot_flag(zram, index, ZRAM_PP_SLOT);
1893	zram_set_priority(zram, index, 0);	2010	set_slot_comp_priority(zram, index, 0);
1894		2011
1895	if (zram_test_flag(zram, index, ZRAM_HUGE)) {	2012	if (test_slot_flag(zram, index, ZRAM_HUGE)) {
1896	zram_clear_flag(zram, index, ZRAM_HUGE);	2013	clear_slot_flag(zram, index, ZRAM_HUGE);
1897	atomic64_dec(&zram->stats.huge_pages);	2014	atomic64_dec(&zram->stats.huge_pages);
1898	}	2015	}
1899		2016
1900	if (zram_test_flag(zram, index, ZRAM_WB)) {	2017	if (test_slot_flag(zram, index, ZRAM_WB)) {
1901	zram_clear_flag(zram, index, ZRAM_WB);	2018	clear_slot_flag(zram, index, ZRAM_WB);
1902	zram_release_bdev_block(zram, zram_get_handle(zram, index));	2019	zram_release_bdev_block(zram, get_slot_handle(zram, index));
1903	goto out;	2020	goto out;
1904	}	2021	}
1905		2022
@@ -1907,24 +2024,24 @@ static void zram_free_page(struct zram *zram, size_t index) · 1 unmodified line
1907	* No memory is allocated for same element filled pages.	2024	* No memory is allocated for same element filled pages.
1908	* Simply clear same page flag.	2025	* Simply clear same page flag.
1909	*/	2026	*/
1910	if (zram_test_flag(zram, index, ZRAM_SAME)) {	2027	if (test_slot_flag(zram, index, ZRAM_SAME)) {
1911	zram_clear_flag(zram, index, ZRAM_SAME);	2028	clear_slot_flag(zram, index, ZRAM_SAME);
1912	atomic64_dec(&zram->stats.same_pages);	2029	atomic64_dec(&zram->stats.same_pages);
1913	goto out;	2030	goto out;
1914	}	2031	}
1915		2032
1916	handle = zram_get_handle(zram, index);	2033	handle = get_slot_handle(zram, index);
1917	if (!handle)	2034	if (!handle)
1918	return;	2035	return;
1919		2036
1920	zs_free(zram->mem_pool, handle);	2037	zs_free(zram->mem_pool, handle);
1921		2038
1922	atomic64_sub(zram_get_obj_size(zram, index),	2039	atomic64_sub(get_slot_size(zram, index),
1923	&zram->stats.compr_data_size);	2040	&zram->stats.compr_data_size);
1924	out:	2041	out:
1925	atomic64_dec(&zram->stats.pages_stored);	2042	atomic64_dec(&zram->stats.pages_stored);
1926	zram_set_handle(zram, index, 0);	2043	set_slot_handle(zram, index, 0);
1927	zram_set_obj_size(zram, index, 0);	2044	set_slot_size(zram, index, 0);
1928	}	2045	}
1929		2046
1930	static int read_same_filled_page(struct zram zram, struct page page,	2047	static int read_same_filled_page(struct zram zram, struct page page,
@@ -1933,7 +2050,7 @@ static int read_same_filled_page(struct zram zram, struct page page, · 2 unmodified lines
1933	void *mem;	2050	void *mem;
1934		2051
1935	mem = kmap_local_page(page);	2052	mem = kmap_local_page(page);
1936	zram_fill_page(mem, PAGE_SIZE, zram_get_handle(zram, index));	2053	zram_fill_page(mem, PAGE_SIZE, get_slot_handle(zram, index));
1937	kunmap_local(mem);	2054	kunmap_local(mem);
1938	return 0;	2055	return 0;
1939	}	2056	}
@@ -1944,12 +2061,12 @@ static int read_incompressible_page(struct zram zram, struct page page, · 4 unmodified lines
1944	unsigned long handle;	2061	unsigned long handle;
1945	void src, dst;	2062	void src, dst;
1946		2063
1947	handle = zram_get_handle(zram, index);	2064	handle = get_slot_handle(zram, index);
1948	src = zs_obj_read_begin(zram->mem_pool, handle, NULL);	2065	src = zs_obj_read_begin(zram->mem_pool, handle, PAGE_SIZE, NULL);
1949	dst = kmap_local_page(page);	2066	dst = kmap_local_page(page);
1950	copy_page(dst, src);	2067	copy_page(dst, src);
1951	kunmap_local(dst);	2068	kunmap_local(dst);
1952	zs_obj_read_end(zram->mem_pool, handle, src);	2069	zs_obj_read_end(zram->mem_pool, handle, PAGE_SIZE, src);
1953		2070
1954	return 0;	2071	return 0;
1955	}	2072	}
@@ -1962,33 +2079,60 @@ static int read_compressed_page(struct zram zram, struct page page, u32 index) · 6 unmodified lines
1962	void src, dst;	2079	void src, dst;
1963	int ret, prio;	2080	int ret, prio;
1964		2081
1965	handle = zram_get_handle(zram, index);	2082	handle = get_slot_handle(zram, index);
1966	size = zram_get_obj_size(zram, index);	2083	size = get_slot_size(zram, index);
1967	prio = zram_get_priority(zram, index);	2084	prio = get_slot_comp_priority(zram, index);
1968		2085
1969	zstrm = zcomp_stream_get(zram->comps[prio]);	2086	zstrm = zcomp_stream_get(zram->comps[prio]);
1970	src = zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy);	2087	src = zs_obj_read_begin(zram->mem_pool, handle, size,
		2088	zstrm->local_copy);
1971	dst = kmap_local_page(page);	2089	dst = kmap_local_page(page);
1972	ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst);	2090	ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst);
1973	kunmap_local(dst);	2091	kunmap_local(dst);
1974	zs_obj_read_end(zram->mem_pool, handle, src);	2092	zs_obj_read_end(zram->mem_pool, handle, size, src);
1975	zcomp_stream_put(zstrm);	2093	zcomp_stream_put(zstrm);
1976		2094
1977	return ret;	2095	return ret;
1978	}	2096	}
1979		2097
		2098	#if defined CONFIG_ZRAM_WRITEBACK
		2099	static int read_from_zspool_raw(struct zram zram, struct page page, u32 index)
		2100	{
		2101	struct zcomp_strm *zstrm;
		2102	unsigned long handle;
		2103	unsigned int size;
		2104	void *src;
		2105
		2106	handle = get_slot_handle(zram, index);
		2107	size = get_slot_size(zram, index);
		2108
		2109	/*
		2110	* We need to get stream just for ->local_copy buffer, in
		2111	* case if object spans two physical pages. No decompression
		2112	* takes place here, as we read raw compressed data.
		2113	*/
		2114	zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]);
		2115	src = zs_obj_read_begin(zram->mem_pool, handle, size,
		2116	zstrm->local_copy);
		2117	memcpy_to_page(page, 0, src, size);
		2118	zs_obj_read_end(zram->mem_pool, handle, size, src);
		2119	zcomp_stream_put(zstrm);
		2120
		2121	return 0;
		2122	}
		2123	#endif
		2124
1980	/*	2125	/*
1981	* Reads (decompresses if needed) a page from zspool (zsmalloc).	2126	* Reads (decompresses if needed) a page from zspool (zsmalloc).
1982	* Corresponding ZRAM slot should be locked.	2127	* Corresponding ZRAM slot should be locked.
1983	*/	2128	*/
1984	static int zram_read_from_zspool(struct zram zram, struct page page,	2129	static int read_from_zspool(struct zram zram, struct page page, u32 index)
1985	u32 index)
1986	{	2130	{
1987	if (zram_test_flag(zram, index, ZRAM_SAME) \|\|	2131	if (test_slot_flag(zram, index, ZRAM_SAME) \|\|
1988	!zram_get_handle(zram, index))	2132	!get_slot_handle(zram, index))
1989	return read_same_filled_page(zram, page, index);	2133	return read_same_filled_page(zram, page, index);
1990		2134
1991	if (!zram_test_flag(zram, index, ZRAM_HUGE))	2135	if (!test_slot_flag(zram, index, ZRAM_HUGE))
1992	return read_compressed_page(zram, page, index);	2136	return read_compressed_page(zram, page, index);
1993	else	2137	else
1994	return read_incompressible_page(zram, page, index);	2138	return read_incompressible_page(zram, page, index);
@@ -1999,20 +2143,20 @@ static int zram_read_page(struct zram zram, struct page page, u32 index, · 4 unmodified lines
1999	{	2143	{
2000	int ret;	2144	int ret;
2001		2145
2002	zram_slot_lock(zram, index);	2146	slot_lock(zram, index);
2003	if (!zram_test_flag(zram, index, ZRAM_WB)) {	2147	if (!test_slot_flag(zram, index, ZRAM_WB)) {
2004	/* Slot should be locked through out the function call */	2148	/* Slot should be locked through out the function call */
2005	ret = zram_read_from_zspool(zram, page, index);	2149	ret = read_from_zspool(zram, page, index);
2006	zram_slot_unlock(zram, index);	2150	slot_unlock(zram, index);
2007	} else {	2151	} else {
2008	unsigned long blk_idx = zram_get_handle(zram, index);	2152	unsigned long blk_idx = get_slot_handle(zram, index);
2009		2153
2010	/*	2154	/*
2011	* The slot should be unlocked before reading from the backing	2155	* The slot should be unlocked before reading from the backing
2012	* device.	2156	* device.
2013	*/	2157	*/
2014	zram_slot_unlock(zram, index);	2158	slot_unlock(zram, index);
2015	ret = read_from_bdev(zram, page, blk_idx, parent);	2159	ret = read_from_bdev(zram, page, index, blk_idx, parent);
2016	}	2160	}
2017		2161
2018	/* Should NEVER happen. Return bio error if it does. */	2162	/* Should NEVER happen. Return bio error if it does. */
@@ -2052,11 +2196,11 @@ static int zram_bvec_read(struct zram zram, struct bio_vec bvec, · 33 unmodified lines
2052	static int write_same_filled_page(struct zram *zram, unsigned long fill,	2196	static int write_same_filled_page(struct zram *zram, unsigned long fill,
2053	u32 index)	2197	u32 index)
2054	{	2198	{
2055	zram_slot_lock(zram, index);	2199	slot_lock(zram, index);
2056	zram_free_page(zram, index);	2200	slot_free(zram, index);
2057	zram_set_flag(zram, index, ZRAM_SAME);	2201	set_slot_flag(zram, index, ZRAM_SAME);
2058	zram_set_handle(zram, index, fill);	2202	set_slot_handle(zram, index, fill);
2059	zram_slot_unlock(zram, index);	2203	slot_unlock(zram, index);
2060		2204
2061	atomic64_inc(&zram->stats.same_pages);	2205	atomic64_inc(&zram->stats.same_pages);
2062	atomic64_inc(&zram->stats.pages_stored);	2206	atomic64_inc(&zram->stats.pages_stored);
@@ -2090,12 +2234,12 @@ static int write_incompressible_page(struct zram zram, struct page page, · 27 unmodified lines
2090	zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE);	2234	zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE);
2091	kunmap_local(src);	2235	kunmap_local(src);
2092		2236
2093	zram_slot_lock(zram, index);	2237	slot_lock(zram, index);
2094	zram_free_page(zram, index);	2238	slot_free(zram, index);
2095	zram_set_flag(zram, index, ZRAM_HUGE);	2239	set_slot_flag(zram, index, ZRAM_HUGE);
2096	zram_set_handle(zram, index, handle);	2240	set_slot_handle(zram, index, handle);
2097	zram_set_obj_size(zram, index, PAGE_SIZE);	2241	set_slot_size(zram, index, PAGE_SIZE);
2098	zram_slot_unlock(zram, index);	2242	slot_unlock(zram, index);
2099		2243
2100	atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size);	2244	atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size);
2101	atomic64_inc(&zram->stats.huge_pages);	2245	atomic64_inc(&zram->stats.huge_pages);
@@ -2155,11 +2299,11 @@ static int zram_write_page(struct zram zram, struct page page, u32 index) · 53 unmodified lines
2155	zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len);	2299	zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len);
2156	zcomp_stream_put(zstrm);	2300	zcomp_stream_put(zstrm);
2157		2301
2158	zram_slot_lock(zram, index);	2302	slot_lock(zram, index);
2159	zram_free_page(zram, index);	2303	slot_free(zram, index);
2160	zram_set_handle(zram, index, handle);	2304	set_slot_handle(zram, index, handle);
2161	zram_set_obj_size(zram, index, comp_len);	2305	set_slot_size(zram, index, comp_len);
2162	zram_slot_unlock(zram, index);	2306	slot_unlock(zram, index);
2163		2307
2164	/* Update stats */	2308	/* Update stats */
2165	atomic64_inc(&zram->stats.pages_stored);	2309	atomic64_inc(&zram->stats.pages_stored);
@@ -2210,30 +2354,30 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio_max, · 44 unmodified lines
2210	for (index = 0; index < nr_pages; index++) {	2354	for (index = 0; index < nr_pages; index++) {
2211	bool ok = true;	2355	bool ok = true;
2212		2356
2213	zram_slot_lock(zram, index);	2357	slot_lock(zram, index);
2214	if (!zram_allocated(zram, index))	2358	if (!slot_allocated(zram, index))
2215	goto next;	2359	goto next;
2216		2360
2217	if (mode & RECOMPRESS_IDLE &&	2361	if (mode & RECOMPRESS_IDLE &&
2218	!zram_test_flag(zram, index, ZRAM_IDLE))	2362	!test_slot_flag(zram, index, ZRAM_IDLE))
2219	goto next;	2363	goto next;
2220		2364
2221	if (mode & RECOMPRESS_HUGE &&	2365	if (mode & RECOMPRESS_HUGE &&
2222	!zram_test_flag(zram, index, ZRAM_HUGE))	2366	!test_slot_flag(zram, index, ZRAM_HUGE))
2223	goto next;	2367	goto next;
2224		2368
2225	if (zram_test_flag(zram, index, ZRAM_WB) \|\|	2369	if (test_slot_flag(zram, index, ZRAM_WB) \|\|
2226	zram_test_flag(zram, index, ZRAM_SAME) \|\|	2370	test_slot_flag(zram, index, ZRAM_SAME) \|\|
2227	zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE))	2371	test_slot_flag(zram, index, ZRAM_INCOMPRESSIBLE))
2228	goto next;	2372	goto next;
2229		2373
2230	/* Already compressed with same of higher priority */	2374	/* Already compressed with same of higher priority */
2231	if (zram_get_priority(zram, index) + 1 >= prio_max)	2375	if (get_slot_comp_priority(zram, index) + 1 >= prio_max)
2232	goto next;	2376	goto next;
2233		2377
2234	ok = place_pp_slot(zram, ctl, index);	2378	ok = place_pp_slot(zram, ctl, index);
2235	next:	2379	next:
2236	zram_slot_unlock(zram, index);	2380	slot_unlock(zram, index);
2237	if (!ok)	2381	if (!ok)
2238	break;	2382	break;
2239	}	2383	}
@@ -2262,18 +2406,18 @@ static int recompress_slot(struct zram zram, u32 index, struct page page, · 22 unmodified lines
2262	void *src;	2406	void *src;
2263	int ret = 0;	2407	int ret = 0;
2264		2408
2265	handle_old = zram_get_handle(zram, index);	2409	handle_old = get_slot_handle(zram, index);
2266	if (!handle_old)	2410	if (!handle_old)
2267	return -EINVAL;	2411	return -EINVAL;
2268		2412
2269	comp_len_old = zram_get_obj_size(zram, index);	2413	comp_len_old = get_slot_size(zram, index);
2270	/*	2414	/*
2271	* Do not recompress objects that are already "small enough".	2415	* Do not recompress objects that are already "small enough".
2272	*/	2416	*/
2273	if (comp_len_old < threshold)	2417	if (comp_len_old < threshold)
2274	return 0;	2418	return 0;
2275		2419
2276	ret = zram_read_from_zspool(zram, page, index);	2420	ret = read_from_zspool(zram, page, index);
2277	if (ret)	2421	if (ret)
2278	return ret;	2422	return ret;
2279		2423
@@ -2282,11 +2426,11 @@ static int recompress_slot(struct zram zram, u32 index, struct page page, · 2 unmodified lines
2282	* we don't preserve IDLE flag and don't incorrectly pick this entry	2426	* we don't preserve IDLE flag and don't incorrectly pick this entry
2283	* for different post-processing type (e.g. writeback).	2427	* for different post-processing type (e.g. writeback).
2284	*/	2428	*/
2285	zram_clear_flag(zram, index, ZRAM_IDLE);	2429	clear_slot_flag(zram, index, ZRAM_IDLE);
2286		2430
2287	class_index_old = zs_lookup_class_index(zram->mem_pool, comp_len_old);	2431	class_index_old = zs_lookup_class_index(zram->mem_pool, comp_len_old);
2288		2432
2289	prio = max(prio, zram_get_priority(zram, index) + 1);	2433	prio = max(prio, get_slot_comp_priority(zram, index) + 1);
2290	/*	2434	/*
2291	* Recompression slots scan should not select slots that are	2435	* Recompression slots scan should not select slots that are
2292	* already compressed with a higher priority algorithm, but	2436	* already compressed with a higher priority algorithm, but
@@ -2353,7 +2497,7 @@ static int recompress_slot(struct zram zram, u32 index, struct page page, · 60 unmodified lines
2353	*/	2497	*/
2354	if (prio < zram->num_active_comps)	2498	if (prio < zram->num_active_comps)
2355	return 0;	2499	return 0;
2356	zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE);	2500	set_slot_flag(zram, index, ZRAM_INCOMPRESSIBLE);
2357	return 0;	2501	return 0;
2358	}	2502	}
2359		2503
@@ -2362,14 +2506,15 @@ static int recompress_slot(struct zram zram, u32 index, struct page page, · 2 unmodified lines
2362	* avoid direct reclaim. Allocation error is not fatal since	2506	* avoid direct reclaim. Allocation error is not fatal since
2363	* we still have the old object in the mem_pool.	2507	* we still have the old object in the mem_pool.
2364	*	2508	*
2365	* XXX: technically, the node we really want here is the node that holds	2509	* XXX: technically, the node we really want here is the node that
2366	* the original compressed data. But that would require us to modify	2510	* holds the original compressed data. But that would require us to
2367	* zsmalloc API to return this information. For now, we will make do with	2511	* modify zsmalloc API to return this information. For now, we will
2368	* the node of the page allocated for recompression.	2512	* make do with the node of the page allocated for recompression.
2369	*/	2513	*/
2370	handle_new = zs_malloc(zram->mem_pool, comp_len_new,	2514	handle_new = zs_malloc(zram->mem_pool, comp_len_new,
2371	GFP_NOIO \| __GFP_NOWARN \|	2515	GFP_NOIO \| __GFP_NOWARN \|
2372	__GFP_HIGHMEM \| __GFP_MOVABLE, page_to_nid(page));	2516	__GFP_HIGHMEM \| __GFP_MOVABLE,
		2517	page_to_nid(page));
2373	if (IS_ERR_VALUE(handle_new)) {	2518	if (IS_ERR_VALUE(handle_new)) {
2374	zcomp_stream_put(zstrm);	2519	zcomp_stream_put(zstrm);
2375	return PTR_ERR((void *)handle_new);	2520	return PTR_ERR((void *)handle_new);
@@ -2378,10 +2523,10 @@ static int recompress_slot(struct zram zram, u32 index, struct page page, · 2 unmodified lines
2378	zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new);	2523	zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new);
2379	zcomp_stream_put(zstrm);	2524	zcomp_stream_put(zstrm);
2380		2525
2381	zram_free_page(zram, index);	2526	slot_free(zram, index);
2382	zram_set_handle(zram, index, handle_new);	2527	set_slot_handle(zram, index, handle_new);
2383	zram_set_obj_size(zram, index, comp_len_new);	2528	set_slot_size(zram, index, comp_len_new);
2384	zram_set_priority(zram, index, prio);	2529	set_slot_comp_priority(zram, index, prio);
2385		2530
2386	atomic64_add(comp_len_new, &zram->stats.compr_data_size);	2531	atomic64_add(comp_len_new, &zram->stats.compr_data_size);
2387	atomic64_inc(&zram->stats.pages_stored);	2532	atomic64_inc(&zram->stats.pages_stored);
@@ -2466,17 +2611,9 @@ static ssize_t recompress_store(struct device *dev, · 78 unmodified lines
2466	if (threshold >= huge_class_size)	2611	if (threshold >= huge_class_size)
2467	return -EINVAL;	2612	return -EINVAL;
2468		2613
2469	down_read(&zram->init_lock);	2614	guard(rwsem_write)(&zram->dev_lock);
2470	if (!init_done(zram)) {	2615	if (!init_done(zram))
2471	ret = -EINVAL;	2616	return -EINVAL;
2472	goto release_init_lock;
2473	}
2474
2475	/* Do not permit concurrent post-processing actions. */
2476	if (atomic_xchg(&zram->pp_in_progress, 1)) {
2477	up_read(&zram->init_lock);
2478	return -EAGAIN;
2479	}
2480		2617
2481	if (algo) {	2618	if (algo) {
2482	bool found = false;	2619	bool found = false;
@@ -2494,26 +2631,26 @@ static ssize_t recompress_store(struct device *dev, · 11 unmodified lines
2494		2631
2495	if (!found) {	2632	if (!found) {
2496	ret = -EINVAL;	2633	ret = -EINVAL;
2497	goto release_init_lock;	2634	goto out;
2498	}	2635	}
2499	}	2636	}
2500		2637
2501	prio_max = min(prio_max, (u32)zram->num_active_comps);	2638	prio_max = min(prio_max, (u32)zram->num_active_comps);
2502	if (prio >= prio_max) {	2639	if (prio >= prio_max) {
2503	ret = -EINVAL;	2640	ret = -EINVAL;
2504	goto release_init_lock;	2641	goto out;
2505	}	2642	}
2506		2643
2507	page = alloc_page(GFP_KERNEL);	2644	page = alloc_page(GFP_KERNEL);
2508	if (!page) {	2645	if (!page) {
2509	ret = -ENOMEM;	2646	ret = -ENOMEM;
2510	goto release_init_lock;	2647	goto out;
2511	}	2648	}
2512		2649
2513	ctl = init_pp_ctl();	2650	ctl = init_pp_ctl();
2514	if (!ctl) {	2651	if (!ctl) {
2515	ret = -ENOMEM;	2652	ret = -ENOMEM;
2516	goto release_init_lock;	2653	goto out;
2517	}	2654	}
2518		2655
2519	scan_slots_for_recompress(zram, mode, prio_max, ctl);	2656	scan_slots_for_recompress(zram, mode, prio_max, ctl);
@@ -2525,15 +2662,15 @@ static ssize_t recompress_store(struct device *dev, · 5 unmodified lines
2525	if (!num_recomp_pages)	2662	if (!num_recomp_pages)
2526	break;	2663	break;
2527		2664
2528	zram_slot_lock(zram, pps->index);	2665	slot_lock(zram, pps->index);
2529	if (!zram_test_flag(zram, pps->index, ZRAM_PP_SLOT))	2666	if (!test_slot_flag(zram, pps->index, ZRAM_PP_SLOT))
2530	goto next;	2667	goto next;
2531		2668
2532	err = recompress_slot(zram, pps->index, page,	2669	err = recompress_slot(zram, pps->index, page,
2533	&num_recomp_pages, threshold,	2670	&num_recomp_pages, threshold,
2534	prio, prio_max);	2671	prio, prio_max);
2535	next:	2672	next:
2536	zram_slot_unlock(zram, pps->index);	2673	slot_unlock(zram, pps->index);
2537	release_pp_slot(zram, pps);	2674	release_pp_slot(zram, pps);
2538		2675
2539	if (err) {	2676	if (err) {
@@ -2544,12 +2681,10 @@ next: · 4 unmodified lines
2544	cond_resched();	2681	cond_resched();
2545	}	2682	}
2546		2683
2547	release_init_lock:	2684	out:
2548	if (page)	2685	if (page)
2549	__free_page(page);	2686	__free_page(page);
2550	release_pp_ctl(zram, ctl);	2687	release_pp_ctl(zram, ctl);
2551	atomic_set(&zram->pp_in_progress, 0);
2552	up_read(&zram->init_lock);
2553	return ret;	2688	return ret;
2554	}	2689	}
2555	#endif	2690	#endif
@@ -2580,9 +2715,9 @@ static void zram_bio_discard(struct zram zram, struct bio bio) · 24 unmodified lines
2580	}	2715	}
2581		2716
2582	while (n >= PAGE_SIZE) {	2717	while (n >= PAGE_SIZE) {
2583	zram_slot_lock(zram, index);	2718	slot_lock(zram, index);
2584	zram_free_page(zram, index);	2719	slot_free(zram, index);
2585	zram_slot_unlock(zram, index);	2720	slot_unlock(zram, index);
2586	atomic64_inc(&zram->stats.notify_free);	2721	atomic64_inc(&zram->stats.notify_free);
2587	index++;	2722	index++;
2588	n -= PAGE_SIZE;	2723	n -= PAGE_SIZE;
@@ -2611,9 +2746,9 @@ static void zram_bio_read(struct zram zram, struct bio bio) · 22 unmodified lines
2611	}	2746	}
2612	flush_dcache_page(bv.bv_page);	2747	flush_dcache_page(bv.bv_page);
2613		2748
2614	zram_slot_lock(zram, index);	2749	slot_lock(zram, index);
2615	zram_accessed(zram, index);	2750	mark_slot_accessed(zram, index);
2616	zram_slot_unlock(zram, index);	2751	slot_unlock(zram, index);
2617		2752
2618	bio_advance_iter_single(bio, &iter, bv.bv_len);	2753	bio_advance_iter_single(bio, &iter, bv.bv_len);
2619	} while (iter.bi_size);	2754	} while (iter.bi_size);
@@ -2641,9 +2776,9 @@ static void zram_bio_write(struct zram zram, struct bio bio) · 21 unmodified lines
2641	break;	2776	break;
2642	}	2777	}
2643		2778
2644	zram_slot_lock(zram, index);	2779	slot_lock(zram, index);
2645	zram_accessed(zram, index);	2780	mark_slot_accessed(zram, index);
2646	zram_slot_unlock(zram, index);	2781	slot_unlock(zram, index);
2647		2782
2648	bio_advance_iter_single(bio, &iter, bv.bv_len);	2783	bio_advance_iter_single(bio, &iter, bv.bv_len);
2649	} while (iter.bi_size);	2784	} while (iter.bi_size);
@@ -2684,13 +2819,13 @@ static void zram_slot_free_notify(struct block_device *bdev, · 34 unmodified lines
2684	zram = bdev->bd_disk->private_data;	2819	zram = bdev->bd_disk->private_data;
2685		2820
2686	atomic64_inc(&zram->stats.notify_free);	2821	atomic64_inc(&zram->stats.notify_free);
2687	if (!zram_slot_trylock(zram, index)) {	2822	if (!slot_trylock(zram, index)) {
2688	atomic64_inc(&zram->stats.miss_free);	2823	atomic64_inc(&zram->stats.miss_free);
2689	return;	2824	return;
2690	}	2825	}
2691		2826
2692	zram_free_page(zram, index);	2827	slot_free(zram, index);
2693	zram_slot_unlock(zram, index);	2828	slot_unlock(zram, index);
2694	}	2829	}
2695		2830
2696	static void zram_comp_params_reset(struct zram *zram)	2831	static void zram_comp_params_reset(struct zram *zram)
@@ -2728,7 +2863,7 @@ static void zram_destroy_comps(struct zram *zram) · 31 unmodified lines
2728		2863
2729	static void zram_reset_device(struct zram *zram)	2864	static void zram_reset_device(struct zram *zram)
2730	{	2865	{
2731	down_write(&zram->init_lock);	2866	guard(rwsem_write)(&zram->dev_lock);
2732		2867
2733	zram->limit_pages = 0;	2868	zram->limit_pages = 0;
2734		2869
@@ -2740,15 +2875,13 @@ static void zram_reset_device(struct zram *zram) · 5 unmodified lines
2740	zram->disksize = 0;	2875	zram->disksize = 0;
2741	zram_destroy_comps(zram);	2876	zram_destroy_comps(zram);
2742	memset(&zram->stats, 0, sizeof(zram->stats));	2877	memset(&zram->stats, 0, sizeof(zram->stats));
2743	atomic_set(&zram->pp_in_progress, 0);
2744	reset_bdev(zram);	2878	reset_bdev(zram);
2745		2879
2746	comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor);	2880	comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor);
2747	up_write(&zram->init_lock);
2748	}	2881	}
2749		2882
2750	static ssize_t disksize_store(struct device *dev,	2883	static ssize_t disksize_store(struct device dev, struct device_attribute attr,
2751	struct device_attribute attr, const char buf, size_t len)	2884	const char *buf, size_t len)
2752	{	2885	{
2753	u64 disksize;	2886	u64 disksize;
2754	struct zcomp *comp;	2887	struct zcomp *comp;
@@ -2760,18 +2893,15 @@ static ssize_t disksize_store(struct device *dev, · 5 unmodified lines
2760	if (!disksize)	2893	if (!disksize)
2761	return -EINVAL;	2894	return -EINVAL;
2762		2895
2763	down_write(&zram->init_lock);	2896	guard(rwsem_write)(&zram->dev_lock);
2764	if (init_done(zram)) {	2897	if (init_done(zram)) {
2765	pr_info("Cannot change disksize for initialized device\n");	2898	pr_info("Cannot change disksize for initialized device\n");
2766	err = -EBUSY;	2899	return -EBUSY;
2767	goto out_unlock;
2768	}	2900	}
2769		2901
2770	disksize = PAGE_ALIGN(disksize);	2902	disksize = PAGE_ALIGN(disksize);
2771	if (!zram_meta_alloc(zram, disksize)) {	2903	if (!zram_meta_alloc(zram, disksize))
2772	err = -ENOMEM;	2904	return -ENOMEM;
2773	goto out_unlock;
2774	}
2775		2905
2776	for (prio = ZRAM_PRIMARY_COMP; prio < ZRAM_MAX_COMPS; prio++) {	2906	for (prio = ZRAM_PRIMARY_COMP; prio < ZRAM_MAX_COMPS; prio++) {
2777	if (!zram->comp_algs[prio])	2907	if (!zram->comp_algs[prio])
@@ -2791,15 +2921,12 @@ static ssize_t disksize_store(struct device *dev, · 13 unmodified lines
2791	}	2921	}
2792	zram->disksize = disksize;	2922	zram->disksize = disksize;
2793	set_capacity_and_notify(zram->disk, zram->disksize >> SECTOR_SHIFT);	2923	set_capacity_and_notify(zram->disk, zram->disksize >> SECTOR_SHIFT);
2794	up_write(&zram->init_lock);
2795		2924
2796	return len;	2925	return len;
2797		2926
2798	out_free_comps:	2927	out_free_comps:
2799	zram_destroy_comps(zram);	2928	zram_destroy_comps(zram);
2800	zram_meta_free(zram, disksize);	2929	zram_meta_free(zram, disksize);
2801	out_unlock:
2802	up_write(&zram->init_lock);
2803	return err;	2930	return err;
2804	}	2931	}
2805		2932
@@ -2862,6 +2989,9 @@ static const struct block_device_operations zram_devops = { · 56 unmodified lines
2862	.owner = THIS_MODULE	2989	.owner = THIS_MODULE
2863	};	2990	};
2864		2991
		2992	static DEVICE_ATTR_RO(io_stat);
		2993	static DEVICE_ATTR_RO(mm_stat);
		2994	static DEVICE_ATTR_RO(debug_stat);
2865	static DEVICE_ATTR_WO(compact);	2995	static DEVICE_ATTR_WO(compact);
2866	static DEVICE_ATTR_RW(disksize);	2996	static DEVICE_ATTR_RW(disksize);
2867	static DEVICE_ATTR_RO(initstate);	2997	static DEVICE_ATTR_RO(initstate);
@@ -2871,11 +3001,13 @@ static DEVICE_ATTR_WO(mem_used_max); · 3 unmodified lines
2871	static DEVICE_ATTR_WO(idle);	3001	static DEVICE_ATTR_WO(idle);
2872	static DEVICE_ATTR_RW(comp_algorithm);	3002	static DEVICE_ATTR_RW(comp_algorithm);
2873	#ifdef CONFIG_ZRAM_WRITEBACK	3003	#ifdef CONFIG_ZRAM_WRITEBACK
		3004	static DEVICE_ATTR_RO(bd_stat);
2874	static DEVICE_ATTR_RW(backing_dev);	3005	static DEVICE_ATTR_RW(backing_dev);
2875	static DEVICE_ATTR_WO(writeback);	3006	static DEVICE_ATTR_WO(writeback);
2876	static DEVICE_ATTR_RW(writeback_limit);	3007	static DEVICE_ATTR_RW(writeback_limit);
2877	static DEVICE_ATTR_RW(writeback_limit_enable);	3008	static DEVICE_ATTR_RW(writeback_limit_enable);
2878	static DEVICE_ATTR_RW(writeback_batch_size);	3009	static DEVICE_ATTR_RW(writeback_batch_size);
		3010	static DEVICE_ATTR_RW(writeback_compressed);
2879	#endif	3011	#endif
2880	#ifdef CONFIG_ZRAM_MULTI_COMP	3012	#ifdef CONFIG_ZRAM_MULTI_COMP
2881	static DEVICE_ATTR_RW(recomp_algorithm);	3013	static DEVICE_ATTR_RW(recomp_algorithm);
@@ -2893,17 +3025,16 @@ static struct attribute *zram_disk_attrs[] = { · 11 unmodified lines
2893	&dev_attr_idle.attr,	3025	&dev_attr_idle.attr,
2894	&dev_attr_comp_algorithm.attr,	3026	&dev_attr_comp_algorithm.attr,
2895	#ifdef CONFIG_ZRAM_WRITEBACK	3027	#ifdef CONFIG_ZRAM_WRITEBACK
		3028	&dev_attr_bd_stat.attr,
2896	&dev_attr_backing_dev.attr,	3029	&dev_attr_backing_dev.attr,
2897	&dev_attr_writeback.attr,	3030	&dev_attr_writeback.attr,
2898	&dev_attr_writeback_limit.attr,	3031	&dev_attr_writeback_limit.attr,
2899	&dev_attr_writeback_limit_enable.attr,	3032	&dev_attr_writeback_limit_enable.attr,
2900	&dev_attr_writeback_batch_size.attr,	3033	&dev_attr_writeback_batch_size.attr,
		3034	&dev_attr_writeback_compressed.attr,
2901	#endif	3035	#endif
2902	&dev_attr_io_stat.attr,	3036	&dev_attr_io_stat.attr,
2903	&dev_attr_mm_stat.attr,	3037	&dev_attr_mm_stat.attr,
2904	#ifdef CONFIG_ZRAM_WRITEBACK
2905	&dev_attr_bd_stat.attr,
2906	#endif
2907	&dev_attr_debug_stat.attr,	3038	&dev_attr_debug_stat.attr,
2908	#ifdef CONFIG_ZRAM_MULTI_COMP	3039	#ifdef CONFIG_ZRAM_MULTI_COMP
2909	&dev_attr_recomp_algorithm.attr,	3040	&dev_attr_recomp_algorithm.attr,
@@ -2957,9 +3088,10 @@ static int zram_add(void) · 47 unmodified lines
2957	goto out_free_dev;	3088	goto out_free_dev;
2958	device_id = ret;	3089	device_id = ret;
2959		3090
2960	init_rwsem(&zram->init_lock);	3091	init_rwsem(&zram->dev_lock);
2961	#ifdef CONFIG_ZRAM_WRITEBACK	3092	#ifdef CONFIG_ZRAM_WRITEBACK
2962	zram->wb_batch_size = 32;	3093	zram->wb_batch_size = 32;
		3094	zram->wb_compressed = false;
2963	#endif	3095	#endif
2964		3096
2965	/* gendisk structure */	3097	/* gendisk structure */
@@ -2978,7 +3110,6 @@ static int zram_add(void) · 12 unmodified lines
2978	zram->disk->fops = &zram_devops;	3110	zram->disk->fops = &zram_devops;
2979	zram->disk->private_data = zram;	3111	zram->disk->private_data = zram;
2980	snprintf(zram->disk->disk_name, 16, "zram%d", device_id);	3112	snprintf(zram->disk->disk_name, 16, "zram%d", device_id);
2981	atomic_set(&zram->pp_in_progress, 0);
2982	zram_comp_params_reset(zram);	3113	zram_comp_params_reset(zram);
2983	comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor);	3114	comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor);
2984		3115
@@ -3139,7 +3270,7 @@ static int __init zram_init(void) · 154 unmodified lines
3139	struct zram_table_entry zram_te;	3270	struct zram_table_entry zram_te;
3140	int ret;	3271	int ret;
3141		3272
3142	BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.flags) * 8);	3273	BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.attr.flags) * 8);
3143		3274
3144	ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare",	3275	ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare",
3145	zcomp_cpu_up_prepare, zcomp_cpu_dead);	3276	zcomp_cpu_up_prepare, zcomp_cpu_dead);

drivers/block/zram/zram_drv.h+10−5

  */
 struct zram_table_entry {
 	unsigned long handle;
-	unsigned long flags;
+	union {
+		unsigned long __lock;
+		struct attr {
+			u32 flags;
 #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME
-	ktime_t ac_time;
+			u32 ac_time;
 #endif
+		} attr;
+	};
 	struct lockdep_map dep_map;
 };
 	struct zcomp *comps[ZRAM_MAX_COMPS];
 	struct zcomp_params params[ZRAM_MAX_COMPS];
 	struct gendisk *disk;
-	/* Prevent concurrent execution of device init */
+	/* Locks the device either in exclusive or in shared mode */
-	struct rw_semaphore init_lock;
+	struct rw_semaphore dev_lock;
 	/*
 	 * the number of pages zram can consume for storing compressed data
 	 */
 #ifdef CONFIG_ZRAM_WRITEBACK
 	struct file *backing_dev;
 	bool wb_limit_enable;
+	bool wb_compressed;
 	u32 wb_batch_size;
 	u64 bd_wb_limit;
 	struct block_device *bdev;
 #ifdef CONFIG_ZRAM_MEMORY_TRACKING
 	struct dentry *debugfs_dir;
 #endif
-	atomic_t pp_in_progress;
 };
 #endif

drivers/misc/Kconfig+1−1

@@ -410,7 +410,7 @@ config DS1682 · 409 unmodified lines
410	config VMWARE_BALLOON	410	config VMWARE_BALLOON
411	tristate "VMware Balloon Driver"	411	tristate "VMware Balloon Driver"
412	depends on VMWARE_VMCI && X86 && HYPERVISOR_GUEST	412	depends on VMWARE_VMCI && X86 && HYPERVISOR_GUEST
413	select MEMORY_BALLOON	413	select BALLOON
414	help	414	help
415	This is VMware physical memory management driver which acts	415	This is VMware physical memory management driver which acts
416	like a "balloon" that can be inflated to reclaim physical pages	416	like a "balloon" that can be inflated to reclaim physical pages

drivers/misc/vmw_balloon.c+29−76

@@ -29,7 +29,7 @@ · 28 unmodified lines
29	#include <linux/rwsem.h>	29	#include <linux/rwsem.h>
30	#include <linux/slab.h>	30	#include <linux/slab.h>
31	#include <linux/spinlock.h>	31	#include <linux/spinlock.h>
32	#include <linux/balloon_compaction.h>	32	#include <linux/balloon.h>
33	#include <linux/vmw_vmci_defs.h>	33	#include <linux/vmw_vmci_defs.h>
34	#include <linux/vmw_vmci_api.h>	34	#include <linux/vmw_vmci_api.h>
35	#include <asm/hypervisor.h>	35	#include <asm/hypervisor.h>
@@ -354,10 +354,15 @@ struct vmballoon { · 318 unmodified lines
354	/**	354	/**
355	* @huge_pages - list of the inflated 2MB pages.	355	* @huge_pages - list of the inflated 2MB pages.
356	*	356	*
357	* Protected by @b_dev_info.pages_lock .	357	* Protected by @huge_pages_lock.
358	*/	358	*/
359	struct list_head huge_pages;	359	struct list_head huge_pages;
360		360
		361	/**
		362	* @huge_pages_lock: lock for the list of inflated 2MB pages.
		363	*/
		364	spinlock_t huge_pages_lock;
		365
361	/**	366	/**
362	* @vmci_doorbell.	367	* @vmci_doorbell.
363	*	368	*
@@ -987,7 +992,6 @@ static void vmballoon_enqueue_page_list(struct vmballoon *b, · 623 unmodified lines
987	unsigned int *n_pages,	992	unsigned int *n_pages,
988	enum vmballoon_page_size_type page_size)	993	enum vmballoon_page_size_type page_size)
989	{	994	{
990	unsigned long flags;
991	struct page *page;	995	struct page *page;
992		996
993	if (page_size == VMW_BALLOON_4K_PAGE) {	997	if (page_size == VMW_BALLOON_4K_PAGE) {
@@ -995,9 +999,9 @@ static void vmballoon_enqueue_page_list(struct vmballoon *b, · 1 unmodified line
995	} else {	999	} else {
996	/*	1000	/*
997	* Keep the huge pages in a local list which is not available	1001	* Keep the huge pages in a local list which is not available
998	* for the balloon compaction mechanism.	1002	* for the balloon page migration.
999	*/	1003	*/
1000	spin_lock_irqsave(&b->b_dev_info.pages_lock, flags);	1004	spin_lock(&b->huge_pages_lock);
1001		1005
1002	list_for_each_entry(page, pages, lru) {	1006	list_for_each_entry(page, pages, lru) {
1003	vmballoon_mark_page_offline(page, VMW_BALLOON_2M_PAGE);	1007	vmballoon_mark_page_offline(page, VMW_BALLOON_2M_PAGE);
@@ -1006,7 +1010,7 @@ static void vmballoon_enqueue_page_list(struct vmballoon *b, · 2 unmodified lines
1006	list_splice_init(pages, &b->huge_pages);	1010	list_splice_init(pages, &b->huge_pages);
1007	__count_vm_events(BALLOON_INFLATE, n_pages	1011	__count_vm_events(BALLOON_INFLATE, n_pages
1008	vmballoon_page_in_frames(VMW_BALLOON_2M_PAGE));	1012	vmballoon_page_in_frames(VMW_BALLOON_2M_PAGE));
1009	spin_unlock_irqrestore(&b->b_dev_info.pages_lock, flags);	1013	spin_unlock(&b->huge_pages_lock);
1010	}	1014	}
1011		1015
1012	*n_pages = 0;	1016	*n_pages = 0;
@@ -1033,7 +1037,6 @@ static void vmballoon_dequeue_page_list(struct vmballoon *b, · 20 unmodified lines
1033	{	1037	{
1034	struct page page, tmp;	1038	struct page page, tmp;
1035	unsigned int i = 0;	1039	unsigned int i = 0;
1036	unsigned long flags;
1037		1040
1038	/* In the case of 4k pages, use the compaction infrastructure */	1041	/* In the case of 4k pages, use the compaction infrastructure */
1039	if (page_size == VMW_BALLOON_4K_PAGE) {	1042	if (page_size == VMW_BALLOON_4K_PAGE) {
@@ -1043,7 +1046,7 @@ static void vmballoon_dequeue_page_list(struct vmballoon *b, · 3 unmodified lines
1043	}	1046	}
1044		1047
1045	/* 2MB pages */	1048	/* 2MB pages */
1046	spin_lock_irqsave(&b->b_dev_info.pages_lock, flags);	1049	spin_lock(&b->huge_pages_lock);
1047	list_for_each_entry_safe(page, tmp, &b->huge_pages, lru) {	1050	list_for_each_entry_safe(page, tmp, &b->huge_pages, lru) {
1048	vmballoon_mark_page_online(page, VMW_BALLOON_2M_PAGE);	1051	vmballoon_mark_page_online(page, VMW_BALLOON_2M_PAGE);
1049		1052
@@ -1054,7 +1057,7 @@ static void vmballoon_dequeue_page_list(struct vmballoon *b, · 4 unmodified lines
1054		1057
1055	__count_vm_events(BALLOON_DEFLATE,	1058	__count_vm_events(BALLOON_DEFLATE,
1056	i * vmballoon_page_in_frames(VMW_BALLOON_2M_PAGE));	1059	i * vmballoon_page_in_frames(VMW_BALLOON_2M_PAGE));
1057	spin_unlock_irqrestore(&b->b_dev_info.pages_lock, flags);	1060	spin_unlock(&b->huge_pages_lock);
1058	*n_pages = i;	1061	*n_pages = i;
1059	}	1062	}
1060		1063
@@ -1716,7 +1719,7 @@ static inline void vmballoon_debugfs_exit(struct vmballoon *b) · 655 unmodified lines
1716	#endif /* CONFIG_DEBUG_FS */	1719	#endif /* CONFIG_DEBUG_FS */
1717		1720
1718		1721
1719	#ifdef CONFIG_BALLOON_COMPACTION	1722	#ifdef CONFIG_BALLOON_MIGRATION
1720	/**	1723	/**
1721	* vmballoon_migratepage() - migrates a balloon page.	1724	* vmballoon_migratepage() - migrates a balloon page.
1722	* @b_dev_info: balloon device information descriptor.	1725	* @b_dev_info: balloon device information descriptor.
@@ -1724,18 +1727,17 @@ static inline void vmballoon_debugfs_exit(struct vmballoon *b) · 1 unmodified line
1724	* @page: a ballooned page that should be migrated.	1727	* @page: a ballooned page that should be migrated.
1725	* @mode: migration mode, ignored.	1728	* @mode: migration mode, ignored.
1726	*	1729	*
1727	* This function is really open-coded, but that is according to the interface
1728	* that balloon_compaction provides.
1729	*
1730	* Return: zero on success, -EAGAIN when migration cannot be performed	1730	* Return: zero on success, -EAGAIN when migration cannot be performed
1731	* momentarily, and -EBUSY if migration failed and should be retried	1731	* momentarily, -EBUSY if migration failed and should be retried
1732	* with that specific page.	1732	* with that specific page, and -ENOENT when deflating @page
		1733	* succeeded but inflating @newpage failed, effectively deflating
		1734	* the balloon.
1733	*/	1735	*/
1734	static int vmballoon_migratepage(struct balloon_dev_info *b_dev_info,	1736	static int vmballoon_migratepage(struct balloon_dev_info *b_dev_info,
1735	struct page newpage, struct page page,	1737	struct page newpage, struct page page,
1736	enum migrate_mode mode)	1738	enum migrate_mode mode)
1737	{	1739	{
1738	unsigned long status, flags;	1740	unsigned long status;
1739	struct vmballoon *b;	1741	struct vmballoon *b;
1740	int ret = 0;	1742	int ret = 0;
1741		1743
@@ -1773,14 +1775,6 @@ static int vmballoon_migratepage(struct balloon_dev_info *b_dev_info, · 31 unmodified lines
1773	goto out_unlock;	1775	goto out_unlock;
1774	}	1776	}
1775		1777
1776	/*
1777	* The page is isolated, so it is safe to delete it without holding
1778	* @pages_lock . We keep holding @comm_lock since we will need it in a
1779	* second.
1780	*/
1781	balloon_page_finalize(page);
1782	put_page(page);
1783
1784	/* Inflate */	1778	/* Inflate */
1785	vmballoon_add_page(b, 0, newpage);	1779	vmballoon_add_page(b, 0, newpage);
1786	status = vmballoon_lock_op(b, 1, VMW_BALLOON_4K_PAGE,	1780	status = vmballoon_lock_op(b, 1, VMW_BALLOON_4K_PAGE,
@@ -1799,60 +1793,21 @@ static int vmballoon_migratepage(struct balloon_dev_info *b_dev_info, · 12 unmodified lines
1799	* change.	1793	* change.
1800	*/	1794	*/
1801	atomic64_dec(&b->size);	1795	atomic64_dec(&b->size);
1802	} else {
1803	/*	1796	/*
1804	* Success. Take a reference for the page, and we will add it to	1797	* Tell the core that we're deflating the old page and don't
1805	* the list after acquiring the lock.	1798	* need the new page.
1806	*/	1799	*/
1807	get_page(newpage);	1800	ret = -ENOENT;
1808	}
1809
1810	/* Update the balloon list under the @pages_lock */
1811	spin_lock_irqsave(&b->b_dev_info.pages_lock, flags);
1812
1813	/*
1814	* On inflation success, we already took a reference for the @newpage.
1815	* If we succeed just insert it to the list and update the statistics
1816	* under the lock.
1817	*/
1818	if (status == VMW_BALLOON_SUCCESS) {
1819	balloon_page_insert(&b->b_dev_info, newpage);
1820	__count_vm_event(BALLOON_MIGRATE);
1821	}	1801	}
1822
1823	/*
1824	* We deflated successfully, so regardless to the inflation success, we
1825	* need to reduce the number of isolated_pages.
1826	*/
1827	b->b_dev_info.isolated_pages--;
1828	spin_unlock_irqrestore(&b->b_dev_info.pages_lock, flags);
1829
1830	out_unlock:	1802	out_unlock:
1831	up_read(&b->conf_sem);	1803	up_read(&b->conf_sem);
1832	return ret;	1804	return ret;
1833	}	1805	}
1834		1806	#else /* CONFIG_BALLOON_MIGRATION */
1835	/**	1807	int vmballoon_migratepage(struct balloon_dev_info *b_dev_info,
1836	* vmballoon_compaction_init() - initialized compaction for the balloon.	1808	struct page newpage, struct page page,
1837	*	1809	enum migrate_mode mode);
1838	* @b: pointer to the balloon.	1810	#endif /* CONFIG_BALLOON_MIGRATION */
1839	*
1840	* If during the initialization a failure occurred, this function does not
1841	* perform cleanup. The caller must call vmballoon_compaction_deinit() in this
1842	* case.
1843	*
1844	* Return: zero on success or error code on failure.
1845	*/
1846	static __init void vmballoon_compaction_init(struct vmballoon *b)
1847	{
1848	b->b_dev_info.migratepage = vmballoon_migratepage;
1849	}
1850
1851	#else /* CONFIG_BALLOON_COMPACTION */
1852	static inline void vmballoon_compaction_init(struct vmballoon *b)
1853	{
1854	}
1855	#endif /* CONFIG_BALLOON_COMPACTION */
1856		1811
1857	static int __init vmballoon_init(void)	1812	static int __init vmballoon_init(void)
1858	{	1813	{
@@ -1871,14 +1826,12 @@ static int __init vmballoon_init(void) · 12 unmodified lines
1871	if (error)	1826	if (error)
1872	return error;	1827	return error;
1873		1828
1874	/*
1875	* Initialization of compaction must be done after the call to
1876	* balloon_devinfo_init() .
1877	*/
1878	balloon_devinfo_init(&balloon.b_dev_info);	1829	balloon_devinfo_init(&balloon.b_dev_info);
1879	vmballoon_compaction_init(&balloon);	1830	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
		1831	balloon.b_dev_info.migratepage = vmballoon_migratepage;
1880		1832
1881	INIT_LIST_HEAD(&balloon.huge_pages);	1833	INIT_LIST_HEAD(&balloon.huge_pages);
		1834	spin_lock_init(&balloon.huge_pages_lock);
1882	spin_lock_init(&balloon.comm_lock);	1835	spin_lock_init(&balloon.comm_lock);
1883	init_rwsem(&balloon.conf_sem);	1836	init_rwsem(&balloon.conf_sem);
1884	balloon.vmci_doorbell = VMCI_INVALID_HANDLE;	1837	balloon.vmci_doorbell = VMCI_INVALID_HANDLE;

drivers/virtio/Kconfig+1−1

@@ -112,7 +112,7 @@ config VIRTIO_PMEM · 111 unmodified lines
112	config VIRTIO_BALLOON	112	config VIRTIO_BALLOON
113	tristate "Virtio balloon driver"	113	tristate "Virtio balloon driver"
114	depends on VIRTIO	114	depends on VIRTIO
115	select MEMORY_BALLOON	115	select BALLOON
116	select PAGE_REPORTING	116	select PAGE_REPORTING
117	help	117	help
118	This driver supports increasing and decreasing the amount	118	This driver supports increasing and decreasing the amount

drivers/virtio/virtio_balloon.c+21−43

@@ -13,7 +13,7 @@ · 12 unmodified lines
13	#include <linux/delay.h>	13	#include <linux/delay.h>
14	#include <linux/slab.h>	14	#include <linux/slab.h>
15	#include <linux/module.h>	15	#include <linux/module.h>
16	#include <linux/balloon_compaction.h>	16	#include <linux/balloon.h>
17	#include <linux/oom.h>	17	#include <linux/oom.h>
18	#include <linux/wait.h>	18	#include <linux/wait.h>
19	#include <linux/mm.h>	19	#include <linux/mm.h>
@@ -242,8 +242,8 @@ static void set_page_pfns(struct virtio_balloon *vb, · 222 unmodified lines
242	static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)	242	static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num)
243	{	243	{
244	unsigned int num_allocated_pages;	244	unsigned int num_allocated_pages;
		245	struct page page, next;
245	unsigned int num_pfns;	246	unsigned int num_pfns;
246	struct page *page;
247	LIST_HEAD(pages);	247	LIST_HEAD(pages);
248		248
249	/* We can only do one array worth at a time. */	249	/* We can only do one array worth at a time. */
@@ -262,21 +262,19 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num) · 12 unmodified lines
262	break;	262	break;
263	}	263	}
264		264
265	balloon_page_push(&pages, page);	265	list_add(&page->lru, &pages);
266	}	266	}
267		267
268	mutex_lock(&vb->balloon_lock);	268	mutex_lock(&vb->balloon_lock);
269		269
270	vb->num_pfns = 0;	270	vb->num_pfns = 0;
271		271
272	while ((page = balloon_page_pop(&pages))) {	272	list_for_each_entry_safe(page, next, &pages, lru) {
		273	list_del(&page->lru);
273	balloon_page_enqueue(&vb->vb_dev_info, page);	274	balloon_page_enqueue(&vb->vb_dev_info, page);
274		275
275	set_page_pfns(vb, vb->pfns + vb->num_pfns, page);	276	set_page_pfns(vb, vb->pfns + vb->num_pfns, page);
276	vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;	277	vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;
277	if (!virtio_has_feature(vb->vdev,
278	VIRTIO_BALLOON_F_DEFLATE_ON_OOM))
279	adjust_managed_page_count(page, -1);
280	vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE;	278	vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE;
281	}	279	}
282		280
@@ -295,9 +293,6 @@ static void release_pages_balloon(struct virtio_balloon *vb, · 12 unmodified lines
295	struct page page, next;	293	struct page page, next;
296		294
297	list_for_each_entry_safe(page, next, pages, lru) {	295	list_for_each_entry_safe(page, next, pages, lru) {
298	if (!virtio_has_feature(vb->vdev,
299	VIRTIO_BALLOON_F_DEFLATE_ON_OOM))
300	adjust_managed_page_count(page, 1);
301	list_del(&page->lru);	296	list_del(&page->lru);
302	put_page(page); /* balloon reference */	297	put_page(page); /* balloon reference */
303	}	298	}
@@ -480,15 +475,19 @@ static inline s64 towards_target(struct virtio_balloon *vb) · 176 unmodified lines
480	static unsigned long return_free_pages_to_mm(struct virtio_balloon *vb,	475	static unsigned long return_free_pages_to_mm(struct virtio_balloon *vb,
481	unsigned long num_to_return)	476	unsigned long num_to_return)
482	{	477	{
483	struct page *page;	478	unsigned long num_returned = 0;
484	unsigned long num_returned;	479	struct page page, next;
		480
		481	if (unlikely(!num_to_return))
		482	return 0;
485		483
486	spin_lock_irq(&vb->free_page_list_lock);	484	spin_lock_irq(&vb->free_page_list_lock);
487	for (num_returned = 0; num_returned < num_to_return; num_returned++) {	485
488	page = balloon_page_pop(&vb->free_page_list);	486	list_for_each_entry_safe(page, next, &vb->free_page_list, lru) {
489	if (!page)	487	list_del(&page->lru);
490	break;
491	__free_pages(page, VIRTIO_BALLOON_HINT_BLOCK_ORDER);	488	__free_pages(page, VIRTIO_BALLOON_HINT_BLOCK_ORDER);
		489	if (++num_returned == num_to_return)
		490	break;
492	}	491	}
493	vb->num_free_page_blocks -= num_returned;	492	vb->num_free_page_blocks -= num_returned;
494	spin_unlock_irq(&vb->free_page_list_lock);	493	spin_unlock_irq(&vb->free_page_list_lock);
@@ -723,7 +722,7 @@ static int get_free_page_and_send(struct virtio_balloon *vb) · 228 unmodified lines
723	}	722	}
724	virtqueue_kick(vq);	723	virtqueue_kick(vq);
725	spin_lock_irq(&vb->free_page_list_lock);	724	spin_lock_irq(&vb->free_page_list_lock);
726	balloon_page_push(&vb->free_page_list, page);	725	list_add(&page->lru, &vb->free_page_list);
727	vb->num_free_page_blocks++;	726	vb->num_free_page_blocks++;
728	spin_unlock_irq(&vb->free_page_list_lock);	727	spin_unlock_irq(&vb->free_page_list_lock);
729	} else {	728	} else {
@@ -803,7 +802,7 @@ static void report_free_page_func(struct work_struct *work) · 73 unmodified lines
803	}	802	}
804	}	803	}
805		804
806	#ifdef CONFIG_BALLOON_COMPACTION	805	#ifdef CONFIG_BALLOON_MIGRATION
807	/*	806	/*
808	* virtballoon_migratepage - perform the balloon page migration on behalf of	807	* virtballoon_migratepage - perform the balloon page migration on behalf of
809	* a compaction thread. (called under page lock)	808	* a compaction thread. (called under page lock)
@@ -827,7 +826,6 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info, · 17 unmodified lines
827	{	826	{
828	struct virtio_balloon *vb = container_of(vb_dev_info,	827	struct virtio_balloon *vb = container_of(vb_dev_info,
829	struct virtio_balloon, vb_dev_info);	828	struct virtio_balloon, vb_dev_info);
830	unsigned long flags;
831		829
832	/*	830	/*
833	* In order to avoid lock contention while migrating pages concurrently	831	* In order to avoid lock contention while migrating pages concurrently
@@ -840,25 +838,7 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info, · 6 unmodified lines
840	if (!mutex_trylock(&vb->balloon_lock))	838	if (!mutex_trylock(&vb->balloon_lock))
841	return -EAGAIN;	839	return -EAGAIN;
842		840
843	get_page(newpage); /* balloon reference */
844
845	/*
846	* When we migrate a page to a different zone and adjusted the
847	* managed page count when inflating, we have to fixup the count of
848	* both involved zones.
849	*/
850	if (!virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM) &&
851	page_zone(page) != page_zone(newpage)) {
852	adjust_managed_page_count(page, 1);
853	adjust_managed_page_count(newpage, -1);
854	}
855
856	/* balloon's page migration 1st step -- inflate "newpage" */	841	/* balloon's page migration 1st step -- inflate "newpage" */
857	spin_lock_irqsave(&vb_dev_info->pages_lock, flags);
858	balloon_page_insert(vb_dev_info, newpage);
859	vb_dev_info->isolated_pages--;
860	__count_vm_event(BALLOON_MIGRATE);
861	spin_unlock_irqrestore(&vb_dev_info->pages_lock, flags);
862	vb->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;	842	vb->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
863	set_page_pfns(vb, vb->pfns, newpage);	843	set_page_pfns(vb, vb->pfns, newpage);
864	tell_host(vb, vb->inflate_vq);	844	tell_host(vb, vb->inflate_vq);
@@ -869,13 +849,9 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info, · 4 unmodified lines
869	tell_host(vb, vb->deflate_vq);	849	tell_host(vb, vb->deflate_vq);
870		850
871	mutex_unlock(&vb->balloon_lock);	851	mutex_unlock(&vb->balloon_lock);
872
873	balloon_page_finalize(page);
874	put_page(page); /* balloon reference */
875
876	return 0;	852	return 0;
877	}	853	}
878	#endif /* CONFIG_BALLOON_COMPACTION */	854	#endif /* CONFIG_BALLOON_MIGRATION */
879		855
880	static unsigned long shrink_free_pages(struct virtio_balloon *vb,	856	static unsigned long shrink_free_pages(struct virtio_balloon *vb,
881	unsigned long pages_to_free)	857	unsigned long pages_to_free)
@@ -970,7 +946,9 @@ static int virtballoon_probe(struct virtio_device *vdev) · 88 unmodified lines
970	if (err)	946	if (err)
971	goto out_free_vb;	947	goto out_free_vb;
972		948
973	#ifdef CONFIG_BALLOON_COMPACTION	949	if (!virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM))
		950	vb->vb_dev_info.adjust_managed_page_count = true;
		951	#ifdef CONFIG_BALLOON_MIGRATION
974	vb->vb_dev_info.migratepage = virtballoon_migratepage;	952	vb->vb_dev_info.migratepage = virtballoon_migratepage;
975	#endif	953	#endif
976	if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) {	954	if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) {

fs/ext4/inode.c+1−2

 	/*
 	 * We do something simple for now.  The filemap_flush() will
 	 * also start triggering a write of the data blocks, which is
-	 * not strictly speaking necessary (and for users of
+	 * not strictly speaking necessary.  However, to do otherwise
-	 * laptop_mode, not even desirable).  However, to do otherwise
 	 * would require replicating code paths in:
 	 *
 	 * ext4_writepages() ->

fs/proc/task_mmu.c+2−2

@@ -2751,7 +2751,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, · 2750 unmodified lines
2751	return 0;	2751	return 0;
2752	}	2752	}
2753		2753
2754	arch_enter_lazy_mmu_mode();	2754	lazy_mmu_mode_enable();
2755		2755
2756	if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) {	2756	if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) {
2757	/* Fast path for performing exclusive WP */	2757	/* Fast path for performing exclusive WP */
@@ -2821,7 +2821,7 @@ flush_and_return: · 63 unmodified lines
2821	if (flush_end)	2821	if (flush_end)
2822	flush_tlb_range(vma, start, addr);	2822	flush_tlb_range(vma, start, addr);
2823		2823
2824	arch_leave_lazy_mmu_mode();	2824	lazy_mmu_mode_disable();
2825	pte_unmap_unlock(start_pte, ptl);	2825	pte_unmap_unlock(start_pte, ptl);
2826		2826
2827	cond_resched();	2827	cond_resched();

fs/sync.c+0−2

@@ -104,8 +104,6 @@ void ksys_sync(void) · 103 unmodified lines
104	iterate_supers(sync_fs_one_sb, &wait);	104	iterate_supers(sync_fs_one_sb, &wait);
105	sync_bdevs(false);	105	sync_bdevs(false);
106	sync_bdevs(true);	106	sync_bdevs(true);
107	if (unlikely(laptop_mode))
108	laptop_sync_completion();
109	}	107	}
110		108
111	SYSCALL_DEFINE0(sync)	109	SYSCALL_DEFINE0(sync)

fs/xfs/xfs_super.c+0−9

@@ -825,15 +825,6 @@ xfs_fs_sync_fs( · 824 unmodified lines
825	if (error)	825	if (error)
826	return error;	826	return error;
827		827
828	if (laptop_mode) {
829	/*
830	* The disk must be active because we're syncing.
831	* We schedule log work now (now that the disk is
832	* active) instead of later (when it might not be).
833	*/
834	flush_delayed_work(&mp->m_log->l_work);
835	}
836
837	/*	828	/*
838	* If we are called with page faults frozen out, it means we are about	829	* If we are called with page faults frozen out, it means we are about
839	* to freeze the transaction subsystem. Take the opportunity to shut	830	* to freeze the transaction subsystem. Take the opportunity to shut

include/asm-generic/tlb.h+3−4

 #define MAX_TABLE_BATCH		\
 	((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *))
-#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE
+#ifndef CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE
 static inline void __tlb_remove_table(void *table)
 {
 	struct ptdesc *ptdesc = (struct ptdesc *)table;
  */
 #define MAX_GATHER_BATCH_COUNT	(10000UL/MAX_GATHER_BATCH)
-extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page,
+extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size);
-		bool delay_rmap, int page_size);
 bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page,
 		unsigned int nr_pages, bool delay_rmap);
 static inline void tlb_remove_page_size(struct mmu_gather *tlb,
 					struct page *page, int page_size)
 {
-	if (__tlb_remove_page_size(tlb, page, false, page_size))
+	if (__tlb_remove_page_size(tlb, page, page_size))
 		tlb_flush_mmu(tlb);
 }

include/linux/backing-dev-defs.h+0−3

 	WB_REASON_VMSCAN,
 	WB_REASON_SYNC,
 	WB_REASON_PERIODIC,
-	WB_REASON_LAPTOP_TIMER,
 	WB_REASON_FS_FREE_SPACE,
 	/*
 	 * There is no bdi forker thread any more and works are done
 	char dev_name[64];
 	struct device *owner;
-	struct timer_list laptop_mode_wb_timer;
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debug_dir;
 #endif

include/linux/balloon.h+77−0

@@ -0,0 +1,77 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	/*
	3	* Common interface for implementing a memory balloon, including support
	4	* for migration of pages inflated in a memory balloon.
	5	*
	6	* Balloon page migration makes use of the general "movable_ops page migration"
	7	* feature.
	8	*
	9	* page->private is used to reference the responsible balloon device.
	10	* That these pages have movable_ops, and which movable_ops apply,
	11	* is derived from the page type (PageOffline()) combined with the
	12	* PG_movable_ops flag (PageMovableOps()).
	13	*
	14	* Once the page type and the PG_movable_ops are set, migration code
	15	* can initiate page isolation by invoking the
	16	* movable_operations()->isolate_page() callback
	17	*
	18	* As long as page->private is set, the page is either on the balloon list
	19	* or isolated for migration. If page->private is not set, the page is
	20	* either still getting inflated, or was deflated to be freed by the balloon
	21	* driver soon. Isolation is impossible in both cases.
	22	*
	23	* As the page isolation scanning step a compaction thread does is a lockless
	24	* procedure (from a page standpoint), it might bring some racy situations while
	25	* performing balloon page migration. In order to sort out these racy scenarios
	26	* and safely perform balloon's page migration we must, always, ensure following
	27	* these simple rules:
	28	*
	29	* i. Inflation/deflation must set/clear page->private under the
	30	* balloon_pages_lock
	31	*
	32	* ii. isolation or dequeueing procedure must remove the page from balloon
	33	* device page list under balloon_pages_lock
	34	*
	35	* Copyright (C) 2012, Red Hat, Inc. Rafael Aquini <aquini@redhat.com>
	36	*/
	37	#ifndef _LINUX_BALLOON_H
	38	#define _LINUX_BALLOON_H
	39	#include <linux/pagemap.h>
	40	#include <linux/page-flags.h>
	41	#include <linux/migrate.h>
	42	#include <linux/gfp.h>
	43	#include <linux/err.h>
	44	#include <linux/list.h>
	45
	46	/*
	47	* Balloon device information descriptor.
	48	* This struct is used to allow the common balloon page migration interface
	49	* procedures to find the proper balloon device holding memory pages they'll
	50	* have to cope for page migration, as well as it serves the balloon driver as
	51	* a page book-keeper for its registered balloon devices.
	52	*/
	53	struct balloon_dev_info {
	54	unsigned long isolated_pages; /* # of isolated pages for migration */
	55	struct list_head pages; /* Pages enqueued & handled to Host */
	56	int (migratepage)(struct balloon_dev_info , struct page *newpage,
	57	struct page *page, enum migrate_mode mode);
	58	bool adjust_managed_page_count;
	59	};
	60
	61	struct page *balloon_page_alloc(void);
	62	void balloon_page_enqueue(struct balloon_dev_info *b_dev_info,
	63	struct page *page);
	64	struct page balloon_page_dequeue(struct balloon_dev_info b_dev_info);
	65	size_t balloon_page_list_enqueue(struct balloon_dev_info *b_dev_info,
	66	struct list_head *pages);
	67	size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,
	68	struct list_head *pages, size_t n_req_pages);
	69
	70	static inline void balloon_devinfo_init(struct balloon_dev_info *balloon)
	71	{
	72	balloon->isolated_pages = 0;
	73	INIT_LIST_HEAD(&balloon->pages);
	74	balloon->migratepage = NULL;
	75	balloon->adjust_managed_page_count = false;
	76	}
	77	#endif /* _LINUX_BALLOON_H */

include/linux/balloon_compaction.h+0−160

@@ -1,160 +0,0 @@
1	/* SPDX-License-Identifier: GPL-2.0 */
2	/*
3	* include/linux/balloon_compaction.h
4	*
5	* Common interface definitions for making balloon pages movable by compaction.
6	*
7	* Balloon page migration makes use of the general "movable_ops page migration"
8	* feature.
9	*
10	* page->private is used to reference the responsible balloon device.
11	* That these pages have movable_ops, and which movable_ops apply,
12	* is derived from the page type (PageOffline()) combined with the
13	* PG_movable_ops flag (PageMovableOps()).
14	*
15	* As the page isolation scanning step a compaction thread does is a lockless
16	* procedure (from a page standpoint), it might bring some racy situations while
17	* performing balloon page compaction. In order to sort out these racy scenarios
18	* and safely perform balloon's page compaction and migration we must, always,
19	* ensure following these simple rules:
20	*
21	* i. Setting the PG_movable_ops flag and page->private with the following
22	* lock order
23	* +-page_lock(page);
24	* +--spin_lock_irq(&b_dev_info->pages_lock);
25	*
26	* ii. isolation or dequeueing procedure must remove the page from balloon
27	* device page list under b_dev_info->pages_lock.
28	*
29	* The functions provided by this interface are placed to help on coping with
30	* the aforementioned balloon page corner case, as well as to ensure the simple
31	* set of exposed rules are satisfied while we are dealing with balloon pages
32	* compaction / migration.
33	*
34	* Copyright (C) 2012, Red Hat, Inc. Rafael Aquini <aquini@redhat.com>
35	*/
36	#ifndef _LINUX_BALLOON_COMPACTION_H
37	#define _LINUX_BALLOON_COMPACTION_H
38	#include <linux/pagemap.h>
39	#include <linux/page-flags.h>
40	#include <linux/migrate.h>
41	#include <linux/gfp.h>
42	#include <linux/err.h>
43	#include <linux/fs.h>
44	#include <linux/list.h>
45
46	/*
47	* Balloon device information descriptor.
48	* This struct is used to allow the common balloon compaction interface
49	* procedures to find the proper balloon device holding memory pages they'll
50	* have to cope for page compaction / migration, as well as it serves the
51	* balloon driver as a page book-keeper for its registered balloon devices.
52	*/
53	struct balloon_dev_info {
54	unsigned long isolated_pages; /* # of isolated pages for migration */
55	spinlock_t pages_lock; /* Protection to pages list */
56	struct list_head pages; /* Pages enqueued & handled to Host */
57	int (migratepage)(struct balloon_dev_info , struct page *newpage,
58	struct page *page, enum migrate_mode mode);
59	};
60
61	extern struct page *balloon_page_alloc(void);
62	extern void balloon_page_enqueue(struct balloon_dev_info *b_dev_info,
63	struct page *page);
64	extern struct page balloon_page_dequeue(struct balloon_dev_info b_dev_info);
65	extern size_t balloon_page_list_enqueue(struct balloon_dev_info *b_dev_info,
66	struct list_head *pages);
67	extern size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info,
68	struct list_head *pages, size_t n_req_pages);
69
70	static inline void balloon_devinfo_init(struct balloon_dev_info *balloon)
71	{
72	balloon->isolated_pages = 0;
73	spin_lock_init(&balloon->pages_lock);
74	INIT_LIST_HEAD(&balloon->pages);
75	balloon->migratepage = NULL;
76	}
77
78	#ifdef CONFIG_BALLOON_COMPACTION
79	extern const struct movable_operations balloon_mops;
80	/*
81	* balloon_page_device - get the b_dev_info descriptor for the balloon device
82	* that enqueues the given page.
83	*/
84	static inline struct balloon_dev_info balloon_page_device(struct page page)
85	{
86	return (struct balloon_dev_info *)page_private(page);
87	}
88	#endif /* CONFIG_BALLOON_COMPACTION */
89
90	/*
91	* balloon_page_insert - insert a page into the balloon's page list and make
92	* the page->private assignment accordingly.
93	* @balloon : pointer to balloon device
94	* @page : page to be assigned as a 'balloon page'
95	*
96	* Caller must ensure the page is locked and the spin_lock protecting balloon
97	* pages list is held before inserting a page into the balloon device.
98	*/
99	static inline void balloon_page_insert(struct balloon_dev_info *balloon,
100	struct page *page)
101	{
102	__SetPageOffline(page);
103	if (IS_ENABLED(CONFIG_BALLOON_COMPACTION)) {
104	SetPageMovableOps(page);
105	set_page_private(page, (unsigned long)balloon);
106	}
107	list_add(&page->lru, &balloon->pages);
108	}
109
110	static inline gfp_t balloon_mapping_gfp_mask(void)
111	{
112	if (IS_ENABLED(CONFIG_BALLOON_COMPACTION))
113	return GFP_HIGHUSER_MOVABLE;
114	return GFP_HIGHUSER;
115	}
116
117	/*
118	* balloon_page_finalize - prepare a balloon page that was removed from the
119	* balloon list for release to the page allocator
120	* @page: page to be released to the page allocator
121	*
122	* Caller must ensure that the page is locked.
123	*/
124	static inline void balloon_page_finalize(struct page *page)
125	{
126	if (IS_ENABLED(CONFIG_BALLOON_COMPACTION))
127	set_page_private(page, 0);
128	/* PageOffline is sticky until the page is freed to the buddy. */
129	}
130
131	/*
132	* balloon_page_push - insert a page into a page list.
133	* @head : pointer to list
134	* @page : page to be added
135	*
136	* Caller must ensure the page is private and protect the list.
137	*/
138	static inline void balloon_page_push(struct list_head pages, struct page page)
139	{
140	list_add(&page->lru, pages);
141	}
142
143	/*
144	* balloon_page_pop - remove a page from a page list.
145	* @head : pointer to list
146	* @page : page to be added
147	*
148	* Caller must ensure the page is private and protect the list.
149	*/
150	static inline struct page balloon_page_pop(struct list_head pages)
151	{
152	struct page *page = list_first_entry_or_null(pages, struct page, lru);
153
154	if (!page)
155	return NULL;
156
157	list_del(&page->lru);
158	return page;
159	}
160	#endif /* _LINUX_BALLOON_COMPACTION_H */

include/linux/cma.h+6−21

@@ -49,9 +49,14 @@ extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, · 48 unmodified lines
49	struct cma **res_cma);	49	struct cma **res_cma);
50	extern struct page cma_alloc(struct cma cma, unsigned long count, unsigned int align,	50	extern struct page cma_alloc(struct cma cma, unsigned long count, unsigned int align,
51	bool no_warn);	51	bool no_warn);
52	extern bool cma_pages_valid(struct cma cma, const struct page pages, unsigned long count);
53	extern bool cma_release(struct cma cma, const struct page pages, unsigned long count);	52	extern bool cma_release(struct cma cma, const struct page pages, unsigned long count);
54		53
		54	struct page cma_alloc_frozen(struct cma cma, unsigned long count,
		55	unsigned int align, bool no_warn);
		56	struct page cma_alloc_frozen_compound(struct cma cma, unsigned int order);
		57	bool cma_release_frozen(struct cma cma, const struct page pages,
		58	unsigned long count);
		59
55	extern int cma_for_each_area(int (it)(struct cma cma, void data), void data);	60	extern int cma_for_each_area(int (it)(struct cma cma, void data), void data);
56	extern bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end);	61	extern bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end);
57		62
@@ -66,24 +71,4 @@ static inline bool cma_skip_dt_default_reserved_mem(void) · 8 unmodified lines
66	}	71	}
67	#endif	72	#endif
68		73
69	#ifdef CONFIG_CMA
70	struct folio cma_alloc_folio(struct cma cma, int order, gfp_t gfp);
71	bool cma_free_folio(struct cma cma, const struct folio folio);
72	bool cma_validate_zones(struct cma *cma);
73	#else
74	static inline struct folio cma_alloc_folio(struct cma cma, int order, gfp_t gfp)
75	{
76	return NULL;
77	}
78
79	static inline bool cma_free_folio(struct cma cma, const struct folio folio)
80	{
81	return false;
82	}
83	static inline bool cma_validate_zones(struct cma *cma)
84	{
85	return false;
86	}
87	#endif
88
89	#endif	74	#endif

include/linux/damon.h+36−29

 #include <linux/random.h>
 /* Minimal region size.  Every damon_region is aligned by this. */
-#define DAMON_MIN_REGION	PAGE_SIZE
+#define DAMON_MIN_REGION_SZ	PAGE_SIZE
 /* Max priority score for DAMON-based operation schemes */
 #define DAMOS_MAX_SCORE		(99)
  * @DAMOS_QUOTA_NODE_MEM_FREE_BP:	MemFree ratio of a node.
  * @DAMOS_QUOTA_NODE_MEMCG_USED_BP:	MemUsed ratio of a node for a cgroup.
  * @DAMOS_QUOTA_NODE_MEMCG_FREE_BP:	MemFree ratio of a node for a cgroup.
+ * @DAMOS_QUOTA_ACTIVE_MEM_BP:		Active to total LRU memory ratio.
+ * @DAMOS_QUOTA_INACTIVE_MEM_BP:	Inactive to total LRU memory ratio.
  * @NR_DAMOS_QUOTA_GOAL_METRICS:	Number of DAMOS quota goal metrics.
  *
  * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.
 	DAMOS_QUOTA_NODE_MEM_FREE_BP,
 	DAMOS_QUOTA_NODE_MEMCG_USED_BP,
 	DAMOS_QUOTA_NODE_MEMCG_FREE_BP,
+	DAMOS_QUOTA_ACTIVE_MEM_BP,
+	DAMOS_QUOTA_INACTIVE_MEM_BP,
 	NR_DAMOS_QUOTA_GOAL_METRICS,
 };
 		u64 last_psi_total;
 		struct {
 			int nid;
-			unsigned short memcg_id;
+			u64 memcg_id;
 		};
 	};
 	struct list_head list;
  * @sz_ops_filter_passed:
  *		Total bytes that passed ops layer-handled DAMOS filters.
  * @qt_exceeds: Total number of times the quota of the scheme has exceeded.
+ * @nr_snapshots:
+ *		Total number of DAMON snapshots that the scheme has tried.
  *
  * "Tried an action to a region" in this context means the DAMOS core logic
  * determined the region as eligible to apply the action.  The access pattern
 	unsigned long sz_applied;
 	unsigned long sz_ops_filter_passed;
 	unsigned long qt_exceeds;
+	unsigned long nr_snapshots;
 };
 /**
 	bool matching;
 	bool allow;
 	union {
-		unsigned short memcg_id;
+		u64 memcg_id;
 		struct damon_addr_range addr_range;
 		int target_idx;
 		struct damon_size_range sz_range;
  * @ops_filters:	ops layer handling &struct damos_filter objects list.
  * @last_applied:	Last @action applied ops-managing entity.
  * @stat:		Statistics of this scheme.
+ * @max_nr_snapshots:	Upper limit of nr_snapshots stat.
  * @list:		List head for siblings.
  *
  * For each @apply_interval_us, DAMON finds regions which fit in the
  * unsets @last_applied when each regions walking for applying the scheme is
  * finished.
  *
- * After applying the &action to each region, &stat_count and &stat_sz is
+ * After applying the &action to each region, &stat is updated.
- * updated to reflect the number of regions and total size of regions that the
+ *
- * &action is applied.
+ * If &max_nr_snapshots is set as non-zero and &stat.nr_snapshots be same to or
+ * greater than it, the scheme is deactivated.
  */
 struct damos {
 	struct damos_access_pattern pattern;
 	struct list_head ops_filters;
 	void *last_applied;
 	struct damos_stat stat;
+	unsigned long max_nr_snapshots;
 	struct list_head list;
 };
  * @apply_scheme:		Apply a DAMON-based operation scheme.
  * @target_valid:		Determine if the target is valid.
  * @cleanup_target:		Clean up each target before deallocation.
- * @cleanup:			Clean up the context.
  *
  * DAMON can be extended for various address spaces and usages.  For this,
  * users should register the low level operations for their target address
  * @target_valid should check whether the target is still valid for the
  * monitoring.
  * @cleanup_target is called before the target will be deallocated.
- * @cleanup is called from @kdamond just before its termination.
  */
 struct damon_operations {
 	enum damon_ops_id id;
 			struct damos *scheme, unsigned long *sz_filter_passed);
 	bool (*target_valid)(struct damon_target *t);
 	void (*cleanup_target)(struct damon_target *t);
-	void (*cleanup)(struct damon_ctx *context);
 };
 /*
  * @data:		Data that will be passed to @fn.
  * @repeat:		Repeat invocations.
  * @return_code:	Return code from @fn invocation.
- * @dealloc_on_cancel:	De-allocate when canceled.
+ * @dealloc_on_cancel:	If @repeat is true, de-allocate when canceled.
  *
  * Control damon_call(), which requests specific kdamond to invoke a given
  * function.  Refer to damon_call() for more details.
  * of the monitoring.
  *
  * @attrs:		Monitoring attributes for accuracy/overhead control.
- * @kdamond:		Kernel thread who does the monitoring.
- * @kdamond_lock:	Mutex for the synchronizations with @kdamond.
  *
- * For each monitoring context, one kernel thread for the monitoring is
+ * For each monitoring context, one kernel thread for the monitoring, namely
- * created.  The pointer to the thread is stored in @kdamond.
+ * kdamond, is created.  The pid of kdamond can be retrieved using
+ * damon_kdamond_pid().
  *
- * Once started, the monitoring thread runs until explicitly required to be
+ * Once started, kdamond runs until explicitly required to be terminated or
- * terminated or every monitoring target is invalid.  The validity of the
+ * every monitoring target is invalid.  The validity of the targets is checked
- * targets is checked via the &damon_operations.target_valid of @ops.  The
+ * via the &damon_operations.target_valid of @ops.  The termination can also be
- * termination can also be explicitly requested by calling damon_stop().
+ * explicitly requested by calling damon_stop().  To know if a kdamond is
- * The thread sets @kdamond to NULL when it terminates. Therefore, users can
+ * running, damon_is_running() can be used.
- * know whether the monitoring is ongoing or terminated by reading @kdamond.
- * Reads and writes to @kdamond from outside of the monitoring thread must
- * be protected by @kdamond_lock.
  *
- * Note that the monitoring thread protects only @kdamond via @kdamond_lock.
+ * While the kdamond is running, all accesses to &struct damon_ctx from a
- * Accesses to other fields must be protected by themselves.
+ * thread other than the kdamond should be made using safe DAMON APIs,
+ * including damon_call() and damos_walk().
  *
  * @ops:	Set of monitoring operations for given use cases.
  * @addr_unit:	Scale factor for core to ops address conversion.
- * @min_sz_region:		Minimum region size.
+ * @min_region_sz:	Minimum region size.
  * @adaptive_targets:	Head of monitoring targets (&damon_target) list.
  * @schemes:		Head of schemes (&damos) list.
  */
 	struct damos_walk_control *walk_control;
 	struct mutex walk_control_lock;
-/* public: */
+	/* Working thread of the given DAMON context */
 	struct task_struct *kdamond;
+	/* Protects @kdamond field access */
 	struct mutex kdamond_lock;
+/* public: */
 	struct damon_operations ops;
 	unsigned long addr_unit;
-	unsigned long min_sz_region;
+	unsigned long min_region_sz;
 	struct list_head adaptive_targets;
 	struct list_head schemes;
 void damon_add_region(struct damon_region *r, struct damon_target *t);
 void damon_destroy_region(struct damon_region *r, struct damon_target *t);
 int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
-		unsigned int nr_ranges, unsigned long min_sz_region);
+		unsigned int nr_ranges, unsigned long min_region_sz);
 void damon_update_region_access_rate(struct damon_region *r, bool accessed,
 		struct damon_attrs *attrs);
 int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive);
 int damon_stop(struct damon_ctx **ctxs, int nr_ctxs);
 bool damon_is_running(struct damon_ctx *ctx);
+int damon_kdamond_pid(struct damon_ctx *ctx);
 int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);
 int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);
 int damon_set_region_biggest_system_ram_default(struct damon_target *t,
 				unsigned long *start, unsigned long *end,
-				unsigned long min_sz_region);
+				unsigned long min_region_sz);
 #endif	/* CONFIG_DAMON */

include/linux/gfp.h+28−32

@@ -407,9 +407,15 @@ extern gfp_t gfp_allowed_mask; · 406 unmodified lines
407	/* Returns true if the gfp_mask allows use of ALLOC_NO_WATERMARK */	407	/* Returns true if the gfp_mask allows use of ALLOC_NO_WATERMARK */
408	bool gfp_pfmemalloc_allowed(gfp_t gfp_mask);	408	bool gfp_pfmemalloc_allowed(gfp_t gfp_mask);
409		409
		410	/* A helper for checking if gfp includes all the specified flags */
		411	static inline bool gfp_has_flags(gfp_t gfp, gfp_t flags)
		412	{
		413	return (gfp & flags) == flags;
		414	}
		415
410	static inline bool gfp_has_io_fs(gfp_t gfp)	416	static inline bool gfp_has_io_fs(gfp_t gfp)
411	{	417	{
412	return (gfp & (__GFP_IO \| __GFP_FS)) == (__GFP_IO \| __GFP_FS);	418	return gfp_has_flags(gfp, __GFP_IO \| __GFP_FS);
413	}	419	}
414		420
415	/*	421	/*
@@ -430,39 +436,29 @@ typedef unsigned int __bitwise acr_flags_t; · 14 unmodified lines
430	#define ACR_FLAGS_CMA ((__force acr_flags_t)BIT(0)) // allocate for CMA	436	#define ACR_FLAGS_CMA ((__force acr_flags_t)BIT(0)) // allocate for CMA
431		437
432	/* The below functions must be run on a range from a single zone. */	438	/* The below functions must be run on a range from a single zone. */
433	extern int alloc_contig_range_noprof(unsigned long start, unsigned long end,	439	int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
434	acr_flags_t alloc_flags, gfp_t gfp_mask);	440	acr_flags_t alloc_flags, gfp_t gfp_mask);
435	#define alloc_contig_range(...) alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__))	441	#define alloc_contig_frozen_range(...) \
436		442	alloc_hooks(alloc_contig_frozen_range_noprof(__VA_ARGS__))
437	extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask,	443
438	int nid, nodemask_t *nodemask);	444	int alloc_contig_range_noprof(unsigned long start, unsigned long end,
439	#define alloc_contig_pages(...) alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__))	445	acr_flags_t alloc_flags, gfp_t gfp_mask);
440		446	#define alloc_contig_range(...) \
441	#endif	447	alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__))
		448
		449	struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,
		450	gfp_t gfp_mask, int nid, nodemask_t *nodemask);
		451	#define alloc_contig_frozen_pages(...) \
		452	alloc_hooks(alloc_contig_frozen_pages_noprof(__VA_ARGS__))
		453
		454	struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask,
		455	int nid, nodemask_t *nodemask);
		456	#define alloc_contig_pages(...) \
		457	alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__))
		458
		459	void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages);
442	void free_contig_range(unsigned long pfn, unsigned long nr_pages);	460	void free_contig_range(unsigned long pfn, unsigned long nr_pages);
443
444	#ifdef CONFIG_CONTIG_ALLOC
445	static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
446	int nid, nodemask_t *node)
447	{
448	struct page *page;
449
450	if (WARN_ON(!order \|\| !(gfp & __GFP_COMP)))
451	return NULL;
452
453	page = alloc_contig_pages_noprof(1 << order, gfp, nid, node);
454
455	return page ? page_folio(page) : NULL;
456	}
457	#else
458	static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
459	int nid, nodemask_t *node)
460	{
461	return NULL;
462	}
463	#endif	461	#endif
464	/* This should be paired with folio_put() rather than free_contig_range(). */
465	#define folio_alloc_gigantic(...) alloc_hooks(folio_alloc_gigantic_noprof(__VA_ARGS__))
466		462
467	DEFINE_FREE(free_page, void *, free_page((unsigned long)_T))	463	DEFINE_FREE(free_page, void *, free_page((unsigned long)_T))
468		464

include/linux/gfp_types.h+5−2

  *
  * %GFP_ATOMIC users can not sleep and need the allocation to succeed. A lower
  * watermark is applied to allow access to "atomic reserves".
- * The current implementation doesn't support NMI and few other strict
+ * The current implementation doesn't support NMI, nor contexts that disable
- * non-preemptive contexts (e.g. raw_spin_lock). The same applies to %GFP_NOWAIT.
+ * preemption under PREEMPT_RT. This includes raw_spin_lock() and plain
+ * preempt_disable() - see "Memory allocation" in
+ * Documentation/core-api/real-time/differences.rst for more info.
  *
  * %GFP_KERNEL is typical for kernel-internal allocations. The caller requires
  * %ZONE_NORMAL or a lower zone for direct access but can direct reclaim.
  * %GFP_NOWAIT is for kernel allocations that should not stall for direct
  * reclaim, start physical IO or use any filesystem callback.  It is very
  * likely to fail to allocate memory, even for very small allocations.
+ * The same restrictions on calling contexts apply as for %GFP_ATOMIC.
  *
  * %GFP_NOIO will use direct reclaim to discard clean pages or slab pages
  * that do not require the starting of any physical IO.

include/linux/highmem.h+97−1

@@ -197,15 +197,111 @@ static inline void invalidate_kernel_vmap_range(void *vaddr, int size) · 196 unmodified lines
197	}	197	}
198	#endif	198	#endif
199		199
200	/* when CONFIG_HIGHMEM is not set these will be plain clear/copy_page */
201	#ifndef clear_user_highpage	200	#ifndef clear_user_highpage
		201	#ifndef clear_user_page
		202	/**
		203	* clear_user_page() - clear a page to be mapped to user space
		204	* @addr: the address of the page
		205	* @vaddr: the address of the user mapping
		206	* @page: the page
		207	*
		208	* We condition the definition of clear_user_page() on the architecture
		209	* not having a custom clear_user_highpage(). That's because if there
		210	* is some special flushing needed for clear_user_highpage() then it
		211	* is likely that clear_user_page() also needs some magic. And, since
		212	* our only caller is the generic clear_user_highpage(), not defining
		213	* is not much of a loss.
		214	*/
		215	static inline void clear_user_page(void addr, unsigned long vaddr, struct page page)
		216	{
		217	clear_page(addr);
		218	}
		219	#endif
		220
		221	/**
		222	* clear_user_pages() - clear a page range to be mapped to user space
		223	* @addr: start address
		224	* @vaddr: start address of the user mapping
		225	* @page: start page
		226	* @npages: number of pages
		227	*
		228	* Assumes that the region (@addr, +@npages) has been validated
		229	* already so this does no exception handling.
		230	*
		231	* If the architecture provides a clear_user_page(), use that;
		232	* otherwise, we can safely use clear_pages().
		233	*/
		234	static inline void clear_user_pages(void *addr, unsigned long vaddr,
		235	struct page *page, unsigned int npages)
		236	{
		237
		238	#ifdef clear_user_page
		239	do {
		240	clear_user_page(addr, vaddr, page);
		241	addr += PAGE_SIZE;
		242	vaddr += PAGE_SIZE;
		243	page++;
		244	} while (--npages);
		245	#else
		246	/*
		247	* Prefer clear_pages() to allow for architectural optimizations
		248	* when operating on contiguous page ranges.
		249	*/
		250	clear_pages(addr, npages);
		251	#endif
		252	}
		253
		254	/**
		255	* clear_user_highpage() - clear a page to be mapped to user space
		256	* @page: start page
		257	* @vaddr: start address of the user mapping
		258	*
		259	* With !CONFIG_HIGHMEM this (and the copy_user_highpage() below) will
		260	* be plain clear_user_page() (and copy_user_page()).
		261	*/
202	static inline void clear_user_highpage(struct page *page, unsigned long vaddr)	262	static inline void clear_user_highpage(struct page *page, unsigned long vaddr)
203	{	263	{
204	void *addr = kmap_local_page(page);	264	void *addr = kmap_local_page(page);
205	clear_user_page(addr, vaddr, page);	265	clear_user_page(addr, vaddr, page);
206	kunmap_local(addr);	266	kunmap_local(addr);
207	}	267	}
		268	#endif /* clear_user_highpage */
		269
		270	/**
		271	* clear_user_highpages() - clear a page range to be mapped to user space
		272	* @page: start page
		273	* @vaddr: start address of the user mapping
		274	* @npages: number of pages
		275	*
		276	* Assumes that all the pages in the region (@page, +@npages) are valid
		277	* so this does no exception handling.
		278	*/
		279	static inline void clear_user_highpages(struct page *page, unsigned long vaddr,
		280	unsigned int npages)
		281	{
		282
		283	#if defined(clear_user_highpage) \|\| defined(CONFIG_HIGHMEM)
		284	/*
		285	* An architecture defined clear_user_highpage() implies special
		286	* handling is needed.
		287	*
		288	* So we use that or, the generic variant if CONFIG_HIGHMEM is
		289	* enabled.
		290	*/
		291	do {
		292	clear_user_highpage(page, vaddr);
		293	vaddr += PAGE_SIZE;
		294	page++;
		295	} while (--npages);
		296	#else
		297
		298	/*
		299	* Prefer clear_user_pages() to allow for architectural optimizations
		300	* when operating on contiguous page ranges.
		301	*/
		302	clear_user_pages(page_address(page), vaddr, page, npages);
208	#endif	303	#endif
		304	}
209		305
210	#ifndef vma_alloc_zeroed_movable_folio	306	#ifndef vma_alloc_zeroed_movable_folio
211	/**	307	/**

include/linux/hugetlb.h+6−9

 struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio);
+extern int movable_gigantic_pages __read_mostly;
 extern int sysctl_hugetlb_shm_group __read_mostly;
 extern struct list_head huge_boot_pages[MAX_NUMNODES];
 void hugetlb_bootmem_alloc(void);
-bool hugetlb_bootmem_allocated(void);
 extern nodemask_t hugetlb_bootmem_nodes;
 void hugetlb_bootmem_set_nodes(void);
 void hugetlb_split(struct vm_area_struct *vma, unsigned long addr);
 int hugetlb_vma_lock_alloc(struct vm_area_struct *vma);
+unsigned int arch_hugetlb_cma_order(void);
 #else /* !CONFIG_HUGETLB_PAGE */
 static inline void hugetlb_dup_vma_private(struct vm_area_struct *vma)
 	if (!hugepage_migration_supported(h))
 		return false;
-	if (hstate_is_gigantic(h))
+	if (hstate_is_gigantic(h) && !movable_gigantic_pages)
 		return false;
 	return true;
 }
 static inline void hugetlb_bootmem_alloc(void)
 {
 }
-static inline bool hugetlb_bootmem_allocated(void)
-{
-	return false;
-}
 #endif	/* CONFIG_HUGETLB_PAGE */
 static inline spinlock_t *huge_pte_lock(struct hstate *h,
 }
 #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA)
-extern void __init hugetlb_cma_reserve(int order);
+extern void __init hugetlb_cma_reserve(void);
 #else
-static inline __init void hugetlb_cma_reserve(int order)
+static inline __init void hugetlb_cma_reserve(void)
 {
 }
 #endif

include/linux/khugepaged.h+4−5

 				 vm_flags_t vm_flags);
 extern void khugepaged_min_free_kbytes_update(void);
 extern bool current_is_khugepaged(void);
-extern int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
+void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
-				   bool install_pmd);
+		bool install_pmd);
 static inline void khugepaged_fork(struct mm_struct *mm, struct mm_struct *oldmm)
 {
 					vm_flags_t vm_flags)
 {
 }
-static inline int collapse_pte_mapped_thp(struct mm_struct *mm,
+static inline void collapse_pte_mapped_thp(struct mm_struct *mm,
-					  unsigned long addr, bool install_pmd)
+		unsigned long addr, bool install_pmd)
 {
-	return 0;
 }
 static inline void khugepaged_min_free_kbytes_update(void)

include/linux/maple_tree.h+0−9

 	struct maple_metadata meta;
 };
-struct maple_alloc {
-	unsigned long total;
-	unsigned char node_count;
-	unsigned int request_count;
-	struct maple_alloc *slot[MAPLE_ALLOC_SLOTS];
-};
 struct maple_topiary {
 	struct maple_pnode *parent;
 	struct maple_enode *next; /* Overlaps the pivot */
 		};
 		struct maple_range_64 mr64;
 		struct maple_arange_64 ma64;
-		struct maple_alloc alloc;
 	};
 };
 void mas_pause(struct ma_state *mas);
 void maple_tree_init(void);
 void mas_destroy(struct ma_state *mas);
-int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries);
 void *mas_prev(struct ma_state *mas, unsigned long min);
 void *mas_prev_range(struct ma_state *mas, unsigned long max);

include/linux/memblock.h+2−2

@@ -598,9 +598,9 @@ extern void alloc_large_system_hash(const char tablename, · 597 unmodified lines
598	*/	598	*/
599	#ifdef CONFIG_NUMA	599	#ifdef CONFIG_NUMA
600	#define HASHDIST_DEFAULT IS_ENABLED(CONFIG_64BIT)	600	#define HASHDIST_DEFAULT IS_ENABLED(CONFIG_64BIT)
601	extern int hashdist; /* Distribute hashes across NUMA nodes? */	601	extern bool hashdist; /* Distribute hashes across NUMA nodes? */
602	#else	602	#else
603	#define hashdist (0)	603	#define hashdist (false)
604	#endif	604	#endif
605		605
606	#ifdef CONFIG_MEMTEST	606	#ifdef CONFIG_MEMTEST

include/linux/memcontrol.h+20−28

 #define MEM_CGROUP_ID_SHIFT	16
-struct mem_cgroup_id {
+struct mem_cgroup_private_id {
 	int id;
 	refcount_t ref;
 };
 	struct cgroup_subsys_state css;
 	/* Private memcg ID. Used to ID objects that outlive the cgroup */
-	struct mem_cgroup_id id;
+	struct mem_cgroup_private_id id;
 	/* Accounted resources */
 	struct page_counter memory;		/* Both v1 & v2 */
 static inline void mem_cgroup_protection(struct mem_cgroup *root,
 					 struct mem_cgroup *memcg,
 					 unsigned long *min,
-					 unsigned long *low)
+					 unsigned long *low,
+					 unsigned long *usage)
 {
-	*min = *low = 0;
+	*min = *low = *usage = 0;
 	if (mem_cgroup_disabled())
 		return;
+	*usage = page_counter_read(&memcg->memory);
 	/*
 	 * There is no reclaim protection applied to a targeted reclaim.
 	 * We are special casing this specific case here because
 void mem_cgroup_scan_tasks(struct mem_cgroup *memcg,
 			   int (*)(struct task_struct *, void *), void *arg);
-static inline unsigned short mem_cgroup_id(struct mem_cgroup *memcg)
+static inline unsigned short mem_cgroup_private_id(struct mem_cgroup *memcg)
 {
 	if (mem_cgroup_disabled())
 		return 0;
 	return memcg->id.id;
 }
-struct mem_cgroup *mem_cgroup_from_id(unsigned short id);
+struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id);
-#ifdef CONFIG_SHRINKER_DEBUG
+static inline u64 mem_cgroup_id(struct mem_cgroup *memcg)
-static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
 {
-	return memcg ? cgroup_ino(memcg->css.cgroup) : 0;
+	return memcg ? cgroup_id(memcg->css.cgroup) : 0;
 }
-struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino);
+struct mem_cgroup *mem_cgroup_get_from_id(u64 id);
-#endif
 static inline struct mem_cgroup *mem_cgroup_from_seq(struct seq_file *m)
 {
 unsigned long mem_cgroup_get_max(struct mem_cgroup *memcg);
-unsigned long mem_cgroup_size(struct mem_cgroup *memcg);
 void mem_cgroup_print_oom_context(struct mem_cgroup *memcg,
 				struct task_struct *p);
 static inline void mem_cgroup_protection(struct mem_cgroup *root,
 					 struct mem_cgroup *memcg,
 					 unsigned long *min,
-					 unsigned long *low)
+					 unsigned long *low,
+					 unsigned long *usage)
 {
-	*min = *low = 0;
+	*min = *low = *usage = 0;
 }
 static inline void mem_cgroup_calculate_protection(struct mem_cgroup *root,
 {
 }
-static inline unsigned short mem_cgroup_id(struct mem_cgroup *memcg)
+static inline unsigned short mem_cgroup_private_id(struct mem_cgroup *memcg)
 {
 	return 0;
 }
-static inline struct mem_cgroup *mem_cgroup_from_id(unsigned short id)
+static inline struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id)
 {
 	WARN_ON_ONCE(id);
 	/* XXX: This should always return root_mem_cgroup */
 	return NULL;
 }
-#ifdef CONFIG_SHRINKER_DEBUG
+static inline u64 mem_cgroup_id(struct mem_cgroup *memcg)
-static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
 {
 	return 0;
 }
-static inline struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino)
+static inline struct mem_cgroup *mem_cgroup_get_from_id(u64 id)
 {
 	return NULL;
 }
-#endif
 static inline struct mem_cgroup *mem_cgroup_from_seq(struct seq_file *m)
 {
 	return 0;
 }
-static inline unsigned long mem_cgroup_size(struct mem_cgroup *memcg)
-{
-	return 0;
-}
 static inline void
 mem_cgroup_print_oom_context(struct mem_cgroup *memcg, struct task_struct *p)
 {
 	return memcg ? memcg->kmemcg_id : -1;
 }
-struct mem_cgroup *mem_cgroup_from_slab_obj(void *p);
+struct mem_cgroup *mem_cgroup_from_virt(void *p);
 static inline void count_objcg_events(struct obj_cgroup *objcg,
 				      enum vm_event_item idx,
 	return -1;
 }
-static inline struct mem_cgroup *mem_cgroup_from_slab_obj(void *p)
+static inline struct mem_cgroup *mem_cgroup_from_virt(void *p)
 {
 	return NULL;
 }

include/linux/mm.h+66−6

@@ -46,6 +46,7 @@ struct pt_regs; · 45 unmodified lines
46	struct folio_batch;	46	struct folio_batch;
47		47
48	void arch_mm_preinit(void);	48	void arch_mm_preinit(void);
		49	void mm_core_init_early(void);
49	void mm_core_init(void);	50	void mm_core_init(void);
50	void init_mm_internals(void);	51	void init_mm_internals(void);
51		52
@@ -1008,10 +1009,7 @@ static inline void vma_flag_set_atomic(struct vm_area_struct *vma, · 956 unmodified lines
1008	{	1009	{
1009	unsigned long *bitmap = ACCESS_PRIVATE(&vma->flags, __vma_flags);	1010	unsigned long *bitmap = ACCESS_PRIVATE(&vma->flags, __vma_flags);
1010		1011
1011	/* mmap read lock/VMA read lock must be held. */	1012	vma_assert_stabilised(vma);
1012	if (!rwsem_is_locked(&vma->vm_mm->mmap_lock))
1013	vma_assert_locked(vma);
1014
1015	if (__vma_flag_atomic_valid(vma, bit))	1013	if (__vma_flag_atomic_valid(vma, bit))
1016	set_bit((__force int)bit, bitmap);	1014	set_bit((__force int)bit, bitmap);
1017	}	1015	}
@@ -2906,6 +2904,13 @@ static inline unsigned long get_mm_rss(struct mm_struct *mm) · 1888 unmodified lines
2906	get_mm_counter(mm, MM_SHMEMPAGES);	2904	get_mm_counter(mm, MM_SHMEMPAGES);
2907	}	2905	}
2908		2906
		2907	static inline unsigned long get_mm_rss_sum(struct mm_struct *mm)
		2908	{
		2909	return get_mm_counter_sum(mm, MM_FILEPAGES) +
		2910	get_mm_counter_sum(mm, MM_ANONPAGES) +
		2911	get_mm_counter_sum(mm, MM_SHMEMPAGES);
		2912	}
		2913
2909	static inline unsigned long get_mm_hiwater_rss(struct mm_struct *mm)	2914	static inline unsigned long get_mm_hiwater_rss(struct mm_struct *mm)
2910	{	2915	{
2911	return max(mm->hiwater_rss, get_mm_rss(mm));	2916	return max(mm->hiwater_rss, get_mm_rss(mm));
@@ -3518,7 +3523,7 @@ static inline unsigned long get_num_physpages(void) · 606 unmodified lines
3518	}	3523	}
3519		3524
3520	/*	3525	/*
3521	* Using memblock node mappings, an architecture may initialise its	3526	* FIXME: Using memblock node mappings, an architecture may initialise its
3522	* zones, allocate the backing mem_map and account for memory holes in an	3527	* zones, allocate the backing mem_map and account for memory holes in an
3523	* architecture independent manner.	3528	* architecture independent manner.
3524	*	3529	*
@@ -3533,7 +3538,7 @@ static inline unsigned long get_num_physpages(void) · 8 unmodified lines
3533	* memblock_add_node(base, size, nid, MEMBLOCK_NONE)	3538	* memblock_add_node(base, size, nid, MEMBLOCK_NONE)
3534	* free_area_init(max_zone_pfns);	3539	* free_area_init(max_zone_pfns);
3535	*/	3540	*/
3536	void free_area_init(unsigned long *max_zone_pfn);	3541	void arch_zone_limits_init(unsigned long *max_zone_pfn);
3537	unsigned long node_map_pfn_alignment(void);	3542	unsigned long node_map_pfn_alignment(void);
3538	extern unsigned long absent_pages_in_range(unsigned long start_pfn,	3543	extern unsigned long absent_pages_in_range(unsigned long start_pfn,
3539	unsigned long end_pfn);	3544	unsigned long end_pfn);
@@ -4180,6 +4185,61 @@ static inline void clear_page_guard(struct zone zone, struct page page, · 640 unmodified lines
4180	unsigned int order) {}	4185	unsigned int order) {}
4181	#endif /* CONFIG_DEBUG_PAGEALLOC */	4186	#endif /* CONFIG_DEBUG_PAGEALLOC */
4182		4187
		4188	#ifndef clear_pages
		4189	/**
		4190	* clear_pages() - clear a page range for kernel-internal use.
		4191	* @addr: start address
		4192	* @npages: number of pages
		4193	*
		4194	* Use clear_user_pages() instead when clearing a page range to be
		4195	* mapped to user space.
		4196	*
		4197	* Does absolutely no exception handling.
		4198	*
		4199	* Note that even though the clearing operation is preemptible, clear_pages()
		4200	* does not (and on architectures where it reduces to a few long-running
		4201	* instructions, might not be able to) call cond_resched() to check if
		4202	* rescheduling is required.
		4203	*
		4204	* When running under preemptible models this is not a problem. Under
		4205	* cooperatively scheduled models, however, the caller is expected to
		4206	* limit @npages to no more than PROCESS_PAGES_NON_PREEMPT_BATCH.
		4207	*/
		4208	static inline void clear_pages(void *addr, unsigned int npages)
		4209	{
		4210	do {
		4211	clear_page(addr);
		4212	addr += PAGE_SIZE;
		4213	} while (--npages);
		4214	}
		4215	#endif
		4216
		4217	#ifndef PROCESS_PAGES_NON_PREEMPT_BATCH
		4218	#ifdef clear_pages
		4219	/*
		4220	* The architecture defines clear_pages(), and we assume that it is
		4221	* generally "fast". So choose a batch size large enough to allow the processor
		4222	* headroom for optimizing the operation and yet small enough that we see
		4223	* reasonable preemption latency for when this optimization is not possible
		4224	* (ex. slow microarchitectures, memory bandwidth saturation.)
		4225	*
		4226	* With a value of 32MB and assuming a memory bandwidth of ~10GBps, this should
		4227	* result in worst case preemption latency of around 3ms when clearing pages.
		4228	*
		4229	* (See comment above clear_pages() for why preemption latency is a concern
		4230	* here.)
		4231	*/
		4232	#define PROCESS_PAGES_NON_PREEMPT_BATCH (SZ_32M >> PAGE_SHIFT)
		4233	#else /* !clear_pages */
		4234	/*
		4235	* The architecture does not provide a clear_pages() implementation. Assume
		4236	* that clear_page() -- which clear_pages() will fallback to -- is relatively
		4237	* slow and choose a small value for PROCESS_PAGES_NON_PREEMPT_BATCH.
		4238	*/
		4239	#define PROCESS_PAGES_NON_PREEMPT_BATCH 1
		4240	#endif
		4241	#endif
		4242
4183	#ifdef __HAVE_ARCH_GATE_AREA	4243	#ifdef __HAVE_ARCH_GATE_AREA
4184	extern struct vm_area_struct get_gate_vma(struct mm_struct mm);	4244	extern struct vm_area_struct get_gate_vma(struct mm_struct mm);
4185	extern int in_gate_area_no_mm(unsigned long addr);	4245	extern int in_gate_area_no_mm(unsigned long addr);

include/linux/mm_types.h+52−5

@@ -752,8 +752,18 @@ static inline struct anon_vma_name anon_vma_name_alloc(const char name) · 751 unmodified lines
752	}	752	}
753	#endif	753	#endif
754		754
755	#define VMA_LOCK_OFFSET 0x40000000	755	/*
756	#define VMA_REF_LIMIT (VMA_LOCK_OFFSET - 1)	756	* While __vma_enter_locked() is working to ensure are no read-locks held on a
		757	* VMA (either while acquiring a VMA write lock or marking a VMA detached) we
		758	* set the VM_REFCNT_EXCLUDE_READERS_FLAG in vma->vm_refcnt to indiciate to
		759	* vma_start_read() that the reference count should be left alone.
		760	*
		761	* See the comment describing vm_refcnt in vm_area_struct for details as to
		762	* which values the VMA reference count can be.
		763	*/
		764	#define VM_REFCNT_EXCLUDE_READERS_BIT (30)
		765	#define VM_REFCNT_EXCLUDE_READERS_FLAG (1U << VM_REFCNT_EXCLUDE_READERS_BIT)
		766	#define VM_REFCNT_LIMIT (VM_REFCNT_EXCLUDE_READERS_FLAG - 1)
757		767
758	struct vma_numab_state {	768	struct vma_numab_state {
759	/*	769	/*
@@ -935,10 +945,10 @@ struct vm_area_struct { · 175 unmodified lines
935	/*	945	/*
936	* Can only be written (using WRITE_ONCE()) while holding both:	946	* Can only be written (using WRITE_ONCE()) while holding both:
937	* - mmap_lock (in write mode)	947	* - mmap_lock (in write mode)
938	* - vm_refcnt bit at VMA_LOCK_OFFSET is set	948	* - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_FLAG is set
939	* Can be read reliably while holding one of:	949	* Can be read reliably while holding one of:
940	* - mmap_lock (in read or write mode)	950	* - mmap_lock (in read or write mode)
941	* - vm_refcnt bit at VMA_LOCK_OFFSET is set or vm_refcnt > 1	951	* - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_BIT is set or vm_refcnt > 1
942	* Can be read unreliably (using READ_ONCE()) for pessimistic bailout	952	* Can be read unreliably (using READ_ONCE()) for pessimistic bailout
943	* while holding nothing (except RCU to keep the VMA struct allocated).	953	* while holding nothing (except RCU to keep the VMA struct allocated).
944	*	954	*
@@ -980,7 +990,44 @@ struct vm_area_struct { · 35 unmodified lines
980	struct vma_numab_state numab_state; / NUMA Balancing state */	990	struct vma_numab_state numab_state; / NUMA Balancing state */
981	#endif	991	#endif
982	#ifdef CONFIG_PER_VMA_LOCK	992	#ifdef CONFIG_PER_VMA_LOCK
983	/* Unstable RCU readers are allowed to read this. */	993	/*
		994	* Used to keep track of firstly, whether the VMA is attached, secondly,
		995	* if attached, how many read locks are taken, and thirdly, if the
		996	* VM_REFCNT_EXCLUDE_READERS_FLAG is set, whether any read locks held
		997	* are currently in the process of being excluded.
		998	*
		999	* This value can be equal to:
		1000	*
		1001	* 0 - Detached. IMPORTANT: when the refcnt is zero, readers cannot
		1002	* increment it.
		1003	*
		1004	* 1 - Attached and either unlocked or write-locked. Write locks are
		1005	* identified via __is_vma_write_locked() which checks for equality of
		1006	* vma->vm_lock_seq and mm->mm_lock_seq.
		1007	*
		1008	* >1, < VM_REFCNT_EXCLUDE_READERS_FLAG - Read-locked or (unlikely)
		1009	* write-locked with other threads having temporarily incremented the
		1010	* reference count prior to determining it is write-locked and
		1011	* decrementing it again.
		1012	*
		1013	* VM_REFCNT_EXCLUDE_READERS_FLAG - Detached, pending
		1014	* __vma_end_exclude_readers() completion which will decrement the
		1015	* reference count to zero. IMPORTANT - at this stage no further readers
		1016	* can increment the reference count. It can only be reduced.
		1017	*
		1018	* VM_REFCNT_EXCLUDE_READERS_FLAG + 1 - A thread is either write-locking
		1019	* an attached VMA and has yet to invoke __vma_end_exclude_readers(),
		1020	* OR a thread is detaching a VMA and is waiting on a single spurious
		1021	* reader in order to decrement the reference count. IMPORTANT - as
		1022	* above, no further readers can increment the reference count.
		1023	*
		1024	* > VM_REFCNT_EXCLUDE_READERS_FLAG + 1 - A thread is either
		1025	* write-locking or detaching a VMA is waiting on readers to
		1026	* exit. IMPORTANT - as above, no further readers can increment the
		1027	* reference count.
		1028	*
		1029	* NOTE: Unstable RCU readers are allowed to read this.
		1030	*/
984	refcount_t vm_refcnt ____cacheline_aligned_in_smp;	1031	refcount_t vm_refcnt ____cacheline_aligned_in_smp;
985	#ifdef CONFIG_DEBUG_LOCK_ALLOC	1032	#ifdef CONFIG_DEBUG_LOCK_ALLOC
986	struct lockdep_map vmlock_dep_map;	1033	struct lockdep_map vmlock_dep_map;

include/linux/mm_types_task.h+5−0

@@ -88,4 +88,9 @@ struct tlbflush_unmap_batch { · 87 unmodified lines
88	#endif	88	#endif
89	};	89	};
90		90
		91	struct lazy_mmu_state {
		92	u8 enable_count;
		93	u8 pause_count;
		94	};
		95
91	#endif /* _LINUX_MM_TYPES_TASK_H */	96	#endif /* _LINUX_MM_TYPES_TASK_H */

include/linux/mmap_lock.h+240−39

@@ -78,6 +78,43 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm) · 77 unmodified lines
78		78
79	#ifdef CONFIG_PER_VMA_LOCK	79	#ifdef CONFIG_PER_VMA_LOCK
80		80
		81	#ifdef CONFIG_LOCKDEP
		82	#define __vma_lockdep_map(vma) (&vma->vmlock_dep_map)
		83	#else
		84	#define __vma_lockdep_map(vma) NULL
		85	#endif
		86
		87	/*
		88	* VMA locks do not behave like most ordinary locks found in the kernel, so we
		89	* cannot quite have full lockdep tracking in the way we would ideally prefer.
		90	*
		91	* Read locks act as shared locks which exclude an exclusive lock being
		92	* taken. We therefore mark these accordingly on read lock acquire/release.
		93	*
		94	* Write locks are acquired exclusively per-VMA, but released in a shared
		95	* fashion, that is upon vma_end_write_all(), we update the mmap's seqcount such
		96	* that write lock is released.
		97	*
		98	* We therefore cannot track write locks per-VMA, nor do we try. Mitigating this
		99	* is the fact that, of course, we do lockdep-track the mmap lock rwsem which
		100	* must be held when taking a VMA write lock.
		101	*
		102	* We do, however, want to indicate that during either acquisition of a VMA
		103	* write lock or detachment of a VMA that we require the lock held be exclusive,
		104	* so we utilise lockdep to do so.
		105	*/
		106	#define __vma_lockdep_acquire_read(vma) \
		107	lock_acquire_shared(__vma_lockdep_map(vma), 0, 1, NULL, _RET_IP_)
		108	#define __vma_lockdep_release_read(vma) \
		109	lock_release(__vma_lockdep_map(vma), _RET_IP_)
		110	#define __vma_lockdep_acquire_exclusive(vma) \
		111	lock_acquire_exclusive(__vma_lockdep_map(vma), 0, 0, NULL, _RET_IP_)
		112	#define __vma_lockdep_release_exclusive(vma) \
		113	lock_release(__vma_lockdep_map(vma), _RET_IP_)
		114	/* Only meaningful if CONFIG_LOCK_STAT is defined. */
		115	#define __vma_lockdep_stat_mark_acquired(vma) \
		116	lock_acquired(__vma_lockdep_map(vma), _RET_IP_)
		117
81	static inline void mm_lock_seqcount_init(struct mm_struct *mm)	118	static inline void mm_lock_seqcount_init(struct mm_struct *mm)
82	{	119	{
83	seqcount_init(&mm->mm_lock_seq);	120	seqcount_init(&mm->mm_lock_seq);
@@ -115,36 +152,81 @@ static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) · 31 unmodified lines
115	#ifdef CONFIG_DEBUG_LOCK_ALLOC	152	#ifdef CONFIG_DEBUG_LOCK_ALLOC
116	static struct lock_class_key lockdep_key;	153	static struct lock_class_key lockdep_key;
117		154
118	lockdep_init_map(&vma->vmlock_dep_map, "vm_lock", &lockdep_key, 0);	155	lockdep_init_map(__vma_lockdep_map(vma), "vm_lock", &lockdep_key, 0);
119	#endif	156	#endif
120	if (reset_refcnt)	157	if (reset_refcnt)
121	refcount_set(&vma->vm_refcnt, 0);	158	refcount_set(&vma->vm_refcnt, 0);
122	vma->vm_lock_seq = UINT_MAX;	159	vma->vm_lock_seq = UINT_MAX;
123	}	160	}
124		161
125	static inline bool is_vma_writer_only(int refcnt)	162	/*
		163	* This function determines whether the input VMA reference count describes a
		164	* VMA which has excluded all VMA read locks.
		165	*
		166	* In the case of a detached VMA, we may incorrectly indicate that readers are
		167	* excluded when one remains, because in that scenario we target a refcount of
		168	* VM_REFCNT_EXCLUDE_READERS_FLAG, rather than the attached target of
		169	* VM_REFCNT_EXCLUDE_READERS_FLAG + 1.
		170	*
		171	* However, the race window for that is very small so it is unlikely.
		172	*
		173	* Returns: true if readers are excluded, false otherwise.
		174	*/
		175	static inline bool __vma_are_readers_excluded(int refcnt)
126	{	176	{
127	/*	177	/*
128	* With a writer and no readers, refcnt is VMA_LOCK_OFFSET if the vma	178	* See the comment describing the vm_area_struct->vm_refcnt field for
129	* is detached and (VMA_LOCK_OFFSET + 1) if it is attached. Waiting on	179	* details of possible refcnt values.
130	* a detached vma happens only in vma_mark_detached() and is a rare
131	* case, therefore most of the time there will be no unnecessary wakeup.
132	*/	180	*/
133	return (refcnt & VMA_LOCK_OFFSET) && refcnt <= VMA_LOCK_OFFSET + 1;	181	return (refcnt & VM_REFCNT_EXCLUDE_READERS_FLAG) &&
		182	refcnt <= VM_REFCNT_EXCLUDE_READERS_FLAG + 1;
		183	}
		184
		185	/*
		186	* Actually decrement the VMA reference count.
		187	*
		188	* The function returns the reference count as it was immediately after the
		189	* decrement took place. If it returns zero, the VMA is now detached.
		190	*/
		191	static inline __must_check unsigned int
		192	__vma_refcount_put_return(struct vm_area_struct *vma)
		193	{
		194	int oldcnt;
		195
		196	if (__refcount_dec_and_test(&vma->vm_refcnt, &oldcnt))
		197	return 0;
		198
		199	return oldcnt - 1;
134	}	200	}
135		201
		202	/**
		203	* vma_refcount_put() - Drop reference count in VMA vm_refcnt field due to a
		204	* read-lock being dropped.
		205	* @vma: The VMA whose reference count we wish to decrement.
		206	*
		207	* If we were the last reader, wake up threads waiting to obtain an exclusive
		208	* lock.
		209	*/
136	static inline void vma_refcount_put(struct vm_area_struct *vma)	210	static inline void vma_refcount_put(struct vm_area_struct *vma)
137	{	211	{
138	/* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt */	212	/* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt. */
139	struct mm_struct *mm = vma->vm_mm;	213	struct mm_struct *mm = vma->vm_mm;
140	int oldcnt;	214	int newcnt;
141		215
142	rwsem_release(&vma->vmlock_dep_map, _RET_IP_);	216	__vma_lockdep_release_read(vma);
143	if (!__refcount_dec_and_test(&vma->vm_refcnt, &oldcnt)) {	217	newcnt = __vma_refcount_put_return(vma);
144		218
145	if (is_vma_writer_only(oldcnt - 1))	219	/*
146	rcuwait_wake_up(&mm->vma_writer_wait);	220	* __vma_start_exclude_readers() may be sleeping waiting for readers to
147	}	221	* drop their reference count, so wake it up if we were the last reader
		222	* blocking it from being acquired.
		223	*
		224	* We may be raced by other readers temporarily incrementing the
		225	* reference count, though the race window is very small, this might
		226	* cause spurious wakeups.
		227	*/
		228	if (newcnt && __vma_are_readers_excluded(newcnt))
		229	rcuwait_wake_up(&mm->vma_writer_wait);
148	}	230	}
149		231
150	/*	232	/*
@@ -159,10 +241,10 @@ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int · 8 unmodified lines
159		241
160	mmap_assert_locked(vma->vm_mm);	242	mmap_assert_locked(vma->vm_mm);
161	if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,	243	if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
162	VMA_REF_LIMIT)))	244	VM_REFCNT_LIMIT)))
163	return false;	245	return false;
164		246
165	rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_);	247	__vma_lockdep_acquire_read(vma);
166	return true;	248	return true;
167	}	249	}
168		250
@@ -182,21 +264,31 @@ static inline void vma_end_read(struct vm_area_struct *vma) · 13 unmodified lines
182	vma_refcount_put(vma);	264	vma_refcount_put(vma);
183	}	265	}
184		266
185	/* WARNING! Can only be used if mmap_lock is expected to be write-locked */	267	static inline unsigned int __vma_raw_mm_seqnum(struct vm_area_struct *vma)
186	static inline bool __is_vma_write_locked(struct vm_area_struct vma, unsigned int mm_lock_seq)
187	{	268	{
		269	const struct mm_struct *mm = vma->vm_mm;
		270
		271	/* We must hold an exclusive write lock for this access to be valid. */
188	mmap_assert_write_locked(vma->vm_mm);	272	mmap_assert_write_locked(vma->vm_mm);
		273	return mm->mm_lock_seq.sequence;
		274	}
189		275
		276	/*
		277	* Determine whether a VMA is write-locked. Must be invoked ONLY if the mmap
		278	* write lock is held.
		279	*
		280	* Returns true if write-locked, otherwise false.
		281	*/
		282	static inline bool __is_vma_write_locked(struct vm_area_struct *vma)
		283	{
190	/*	284	/*
191	* current task is holding mmap_write_lock, both vma->vm_lock_seq and	285	* current task is holding mmap_write_lock, both vma->vm_lock_seq and
192	* mm->mm_lock_seq can't be concurrently modified.	286	* mm->mm_lock_seq can't be concurrently modified.
193	*/	287	*/
194	*mm_lock_seq = vma->vm_mm->mm_lock_seq.sequence;	288	return vma->vm_lock_seq == __vma_raw_mm_seqnum(vma);
195	return (vma->vm_lock_seq == *mm_lock_seq);
196	}	289	}
197		290
198	int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq,	291	int __vma_start_write(struct vm_area_struct *vma, int state);
199	int state);
200		292
201	/*	293	/*
202	* Begin writing to a VMA.	294	* Begin writing to a VMA.
@@ -205,12 +297,10 @@ int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq, · 2 unmodified lines
205	*/	297	*/
206	static inline void vma_start_write(struct vm_area_struct *vma)	298	static inline void vma_start_write(struct vm_area_struct *vma)
207	{	299	{
208	unsigned int mm_lock_seq;	300	if (__is_vma_write_locked(vma))
209
210	if (__is_vma_write_locked(vma, &mm_lock_seq))
211	return;	301	return;
212		302
213	__vma_start_write(vma, mm_lock_seq, TASK_UNINTERRUPTIBLE);	303	__vma_start_write(vma, TASK_UNINTERRUPTIBLE);
214	}	304	}
215		305
216	/**	306	/**
@@ -229,26 +319,110 @@ static inline void vma_start_write(struct vm_area_struct *vma) · 12 unmodified lines
229	static inline __must_check	319	static inline __must_check
230	int vma_start_write_killable(struct vm_area_struct *vma)	320	int vma_start_write_killable(struct vm_area_struct *vma)
231	{	321	{
232	unsigned int mm_lock_seq;	322	if (__is_vma_write_locked(vma))
233
234	if (__is_vma_write_locked(vma, &mm_lock_seq))
235	return 0;	323	return 0;
236	return __vma_start_write(vma, mm_lock_seq, TASK_KILLABLE);	324
		325	return __vma_start_write(vma, TASK_KILLABLE);
237	}	326	}
238		327
		328	/**
		329	* vma_assert_write_locked() - assert that @vma holds a VMA write lock.
		330	* @vma: The VMA to assert.
		331	*/
239	static inline void vma_assert_write_locked(struct vm_area_struct *vma)	332	static inline void vma_assert_write_locked(struct vm_area_struct *vma)
240	{	333	{
241	unsigned int mm_lock_seq;	334	VM_WARN_ON_ONCE_VMA(!__is_vma_write_locked(vma), vma);
242
243	VM_BUG_ON_VMA(!__is_vma_write_locked(vma, &mm_lock_seq), vma);
244	}	335	}
245		336
		337	/**
		338	* vma_assert_locked() - assert that @vma holds either a VMA read or a VMA write
		339	* lock and is not detached.
		340	* @vma: The VMA to assert.
		341	*/
246	static inline void vma_assert_locked(struct vm_area_struct *vma)	342	static inline void vma_assert_locked(struct vm_area_struct *vma)
247	{	343	{
248	unsigned int mm_lock_seq;	344	unsigned int refcnt;
		345
		346	if (IS_ENABLED(CONFIG_LOCKDEP)) {
		347	if (!lock_is_held(__vma_lockdep_map(vma)))
		348	vma_assert_write_locked(vma);
		349	return;
		350	}
		351
		352	/*
		353	* See the comment describing the vm_area_struct->vm_refcnt field for
		354	* details of possible refcnt values.
		355	*/
		356	refcnt = refcount_read(&vma->vm_refcnt);
		357
		358	/*
		359	* In this case we're either read-locked, write-locked with temporary
		360	* readers, or in the midst of excluding readers, all of which means
		361	* we're locked.
		362	*/
		363	if (refcnt > 1)
		364	return;
		365
		366	/* It is a bug for the VMA to be detached here. */
		367	VM_WARN_ON_ONCE_VMA(!refcnt, vma);
		368
		369	/*
		370	* OK, the VMA has a reference count of 1 which means it is either
		371	* unlocked and attached or write-locked, so assert that it is
		372	* write-locked.
		373	*/
		374	vma_assert_write_locked(vma);
		375	}
		376
		377	/**
		378	* vma_assert_stabilised() - assert that this VMA cannot be changed from
		379	* underneath us either by having a VMA or mmap lock held.
		380	* @vma: The VMA whose stability we wish to assess.
		381	*
		382	* If lockdep is enabled we can precisely ensure stability via either an mmap
		383	* lock owned by us or a specific VMA lock.
		384	*
		385	* With lockdep disabled we may sometimes race with other threads acquiring the
		386	* mmap read lock simultaneous with our VMA read lock.
		387	*/
		388	static inline void vma_assert_stabilised(struct vm_area_struct *vma)
		389	{
		390	/*
		391	* If another thread owns an mmap lock, it may go away at any time, and
		392	* thus is no guarantee of stability.
		393	*
		394	* If lockdep is enabled we can accurately determine if an mmap lock is
		395	* held and owned by us. Otherwise we must approximate.
		396	*
		397	* It doesn't necessarily mean we are not stabilised however, as we may
		398	* hold a VMA read lock (not a write lock as this would require an owned
		399	* mmap lock).
		400	*
		401	* If (assuming lockdep is not enabled) we were to assert a VMA read
		402	* lock first we may also run into issues, as other threads can hold VMA
		403	* read locks simlutaneous to us.
		404	*
		405	* Therefore if lockdep is not enabled we risk a false negative (i.e. no
		406	* assert fired). If accurate checking is required, enable lockdep.
		407	*/
		408	if (IS_ENABLED(CONFIG_LOCKDEP)) {
		409	if (lockdep_is_held(&vma->vm_mm->mmap_lock))
		410	return;
		411	} else {
		412	if (rwsem_is_locked(&vma->vm_mm->mmap_lock))
		413	return;
		414	}
249		415
250	VM_BUG_ON_VMA(refcount_read(&vma->vm_refcnt) <= 1 &&	416	/*
251	!__is_vma_write_locked(vma, &mm_lock_seq), vma);	417	* We're not stabilised by the mmap lock, so assert that we're
		418	* stabilised by a VMA lock.
		419	*/
		420	vma_assert_locked(vma);
		421	}
		422
		423	static inline bool vma_is_attached(struct vm_area_struct *vma)
		424	{
		425	return refcount_read(&vma->vm_refcnt);
252	}	426	}
253		427
254	/*	428	/*
@@ -258,12 +432,12 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) · 3 unmodified lines
258	*/	432	*/
259	static inline void vma_assert_attached(struct vm_area_struct *vma)	433	static inline void vma_assert_attached(struct vm_area_struct *vma)
260	{	434	{
261	WARN_ON_ONCE(!refcount_read(&vma->vm_refcnt));	435	WARN_ON_ONCE(!vma_is_attached(vma));
262	}	436	}
263		437
264	static inline void vma_assert_detached(struct vm_area_struct *vma)	438	static inline void vma_assert_detached(struct vm_area_struct *vma)
265	{	439	{
266	WARN_ON_ONCE(refcount_read(&vma->vm_refcnt));	440	WARN_ON_ONCE(vma_is_attached(vma));
267	}	441	}
268		442
269	static inline void vma_mark_attached(struct vm_area_struct *vma)	443	static inline void vma_mark_attached(struct vm_area_struct *vma)
@@ -273,7 +447,28 @@ static inline void vma_mark_attached(struct vm_area_struct *vma) · 3 unmodified lines
273	refcount_set_release(&vma->vm_refcnt, 1);	447	refcount_set_release(&vma->vm_refcnt, 1);
274	}	448	}
275		449
276	void vma_mark_detached(struct vm_area_struct *vma);	450	void __vma_exclude_readers_for_detach(struct vm_area_struct *vma);
		451
		452	static inline void vma_mark_detached(struct vm_area_struct *vma)
		453	{
		454	vma_assert_write_locked(vma);
		455	vma_assert_attached(vma);
		456
		457	/*
		458	* The VMA still being attached (refcnt > 0) - is unlikely, because the
		459	* vma has been already write-locked and readers can increment vm_refcnt
		460	* only temporarily before they check vm_lock_seq, realize the vma is
		461	* locked and drop back the vm_refcnt. That is a narrow window for
		462	* observing a raised vm_refcnt.
		463	*
		464	* See the comment describing the vm_area_struct->vm_refcnt field for
		465	* details of possible refcnt values.
		466	*/
		467	if (likely(!__vma_refcount_put_return(vma)))
		468	return;
		469
		470	__vma_exclude_readers_for_detach(vma);
		471	}
277		472
278	struct vm_area_struct lock_vma_under_rcu(struct mm_struct mm,	473	struct vm_area_struct lock_vma_under_rcu(struct mm_struct mm,
279	unsigned long address);	474	unsigned long address);
@@ -327,6 +522,12 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) · 47 unmodified lines
327	mmap_assert_locked(vma->vm_mm);	522	mmap_assert_locked(vma->vm_mm);
328	}	523	}
329		524
		525	static inline void vma_assert_stabilised(struct vm_area_struct *vma)
		526	{
		527	/* If no VMA locks, then either mmap lock suffices to stabilise. */
		528	mmap_assert_locked(vma->vm_mm);
		529	}
		530
330	#endif /* CONFIG_PER_VMA_LOCK */	531	#endif /* CONFIG_PER_VMA_LOCK */
331		532
332	static inline void mmap_write_lock(struct mm_struct *mm)	533	static inline void mmap_write_lock(struct mm_struct *mm)

include/linux/mmdebug.h+10−0

 			BUG();						\
 		}							\
 	} while (0)
+#define VM_WARN_ON_PAGE(cond, page)		({			\
+	int __ret_warn = !!(cond);					\
+									\
+	if (unlikely(__ret_warn)) {					\
+		dump_page(page, "VM_WARN_ON_PAGE(" __stringify(cond)")");\
+		WARN_ON(1);						\
+	}								\
+	unlikely(__ret_warn);						\
+})
 #define VM_WARN_ON_ONCE_PAGE(cond, page)	({			\
 	static bool __section(".data..once") __warned;			\
 	int __ret_warn_once = !!(cond);					\
 #define VM_BUG_ON_MM(cond, mm) VM_BUG_ON(cond)
 #define VM_WARN_ON(cond) BUILD_BUG_ON_INVALID(cond)
 #define VM_WARN_ON_ONCE(cond) BUILD_BUG_ON_INVALID(cond)
+#define VM_WARN_ON_PAGE(cond, page)  BUILD_BUG_ON_INVALID(cond)
 #define VM_WARN_ON_ONCE_PAGE(cond, page)  BUILD_BUG_ON_INVALID(cond)
 #define VM_WARN_ON_FOLIO(cond, folio)  BUILD_BUG_ON_INVALID(cond)
 #define VM_WARN_ON_ONCE_FOLIO(cond, folio)  BUILD_BUG_ON_INVALID(cond)

include/linux/mmzone.h+15−4

 #include <linux/memory_hotplug.h>
 void build_all_zonelists(pg_data_t *pgdat);
-void wakeup_kswapd(struct zone *zone, gfp_t gfp_mask, int order,
-		   enum zone_type highest_zoneidx);
 bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
 			 int highest_zoneidx, unsigned int alloc_flags,
 			 long free_pages);
 bool zone_watermark_ok(struct zone *z, unsigned int order,
 		unsigned long mark, int highest_zoneidx,
 		unsigned int alloc_flags);
+enum kswapd_clear_hopeless_reason {
+	KSWAPD_CLEAR_HOPELESS_OTHER = 0,
+	KSWAPD_CLEAR_HOPELESS_KSWAPD,
+	KSWAPD_CLEAR_HOPELESS_DIRECT,
+	KSWAPD_CLEAR_HOPELESS_PCP,
+};
+void wakeup_kswapd(struct zone *zone, gfp_t gfp_mask, int order,
+		   enum zone_type highest_zoneidx);
+void kswapd_try_clear_hopeless(struct pglist_data *pgdat,
+			       unsigned int order, int highest_zoneidx);
+void kswapd_clear_hopeless(pg_data_t *pgdat, enum kswapd_clear_hopeless_reason reason);
+bool kswapd_test_hopeless(pg_data_t *pgdat);
 /*
  * Memory initialization context, use to differentiate memory added by
  * the platform statically or via memory hotplug interface.
 #define pfn_to_nid(pfn)		(0)
 #endif
-void sparse_init(void);
 #else
-#define sparse_init()	do {} while (0)
 #define sparse_index_init(_sec, _nid)  do {} while (0)
 #define sparse_vmemmap_init_nid_early(_nid) do {} while (0)
 #define sparse_vmemmap_init_nid_late(_nid) do {} while (0)

include/linux/nodemask.h+4−4

@@ -157,10 +157,10 @@ static __always_inline bool __node_test_and_set(int node, nodemask_t *addr) · 156 unmodified lines
157		157
158	#define nodes_and(dst, src1, src2) \	158	#define nodes_and(dst, src1, src2) \
159	__nodes_and(&(dst), &(src1), &(src2), MAX_NUMNODES)	159	__nodes_and(&(dst), &(src1), &(src2), MAX_NUMNODES)
160	static __always_inline void __nodes_and(nodemask_t dstp, const nodemask_t src1p,	160	static __always_inline bool __nodes_and(nodemask_t dstp, const nodemask_t src1p,
161	const nodemask_t *src2p, unsigned int nbits)	161	const nodemask_t *src2p, unsigned int nbits)
162	{	162	{
163	bitmap_and(dstp->bits, src1p->bits, src2p->bits, nbits);	163	return bitmap_and(dstp->bits, src1p->bits, src2p->bits, nbits);
164	}	164	}
165		165
166	#define nodes_or(dst, src1, src2) \	166	#define nodes_or(dst, src1, src2) \
@@ -181,10 +181,10 @@ static __always_inline void __nodes_xor(nodemask_t dstp, const nodemask_t src1 · 14 unmodified lines
181		181
182	#define nodes_andnot(dst, src1, src2) \	182	#define nodes_andnot(dst, src1, src2) \
183	__nodes_andnot(&(dst), &(src1), &(src2), MAX_NUMNODES)	183	__nodes_andnot(&(dst), &(src1), &(src2), MAX_NUMNODES)
184	static __always_inline void __nodes_andnot(nodemask_t dstp, const nodemask_t src1p,	184	static __always_inline bool __nodes_andnot(nodemask_t dstp, const nodemask_t src1p,
185	const nodemask_t *src2p, unsigned int nbits)	185	const nodemask_t *src2p, unsigned int nbits)
186	{	186	{
187	bitmap_andnot(dstp->bits, src1p->bits, src2p->bits, nbits);	187	return bitmap_andnot(dstp->bits, src1p->bits, src2p->bits, nbits);
188	}	188	}
189		189
190	#define nodes_copy(dst, src) __nodes_copy(&(dst), &(src), MAX_NUMNODES)	190	#define nodes_copy(dst, src) __nodes_copy(&(dst), &(src), MAX_NUMNODES)

include/linux/page-isolation.h+2−0

@@ -67,4 +67,6 @@ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn); · 66 unmodified lines
67		67
68	int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn,	68	int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn,
69	enum pb_isolate_mode mode);	69	enum pb_isolate_mode mode);
		70	bool page_is_unmovable(struct zone zone, struct page page,
		71	enum pb_isolate_mode mode, unsigned long *step);
70	#endif	72	#endif

include/linux/page_table_check.h+40−29

 extern struct page_ext_operations page_table_check_ops;
 void __page_table_check_zero(struct page *page, unsigned int order);
-void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte);
+void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr,
-void __page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd);
+				  pte_t pte);
-void __page_table_check_pud_clear(struct mm_struct *mm, pud_t pud);
+void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr,
-void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte,
+				  pmd_t pmd);
-		unsigned int nr);
+void __page_table_check_pud_clear(struct mm_struct *mm, unsigned long addr,
-void __page_table_check_pmds_set(struct mm_struct *mm, pmd_t *pmdp, pmd_t pmd,
+				  pud_t pud);
-		unsigned int nr);
+void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr,
-void __page_table_check_puds_set(struct mm_struct *mm, pud_t *pudp, pud_t pud,
+		pte_t *ptep, pte_t pte, unsigned int nr);
-		unsigned int nr);
+void __page_table_check_pmds_set(struct mm_struct *mm, unsigned long addr,
+		pmd_t *pmdp, pmd_t pmd, unsigned int nr);
+void __page_table_check_puds_set(struct mm_struct *mm, unsigned long addr,
+		pud_t *pudp, pud_t pud, unsigned int nr);
 void __page_table_check_pte_clear_range(struct mm_struct *mm,
 					unsigned long addr,
 					pmd_t pmd);
 	__page_table_check_zero(page, order);
 }
-static inline void page_table_check_pte_clear(struct mm_struct *mm, pte_t pte)
+static inline void page_table_check_pte_clear(struct mm_struct *mm,
+					      unsigned long addr, pte_t pte)
 {
 	if (static_branch_likely(&page_table_check_disabled))
 		return;
-	__page_table_check_pte_clear(mm, pte);
+	__page_table_check_pte_clear(mm, addr, pte);
 }
-static inline void page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd)
+static inline void page_table_check_pmd_clear(struct mm_struct *mm,
+					      unsigned long addr, pmd_t pmd)
 {
 	if (static_branch_likely(&page_table_check_disabled))
 		return;
-	__page_table_check_pmd_clear(mm, pmd);
+	__page_table_check_pmd_clear(mm, addr, pmd);
 }
-static inline void page_table_check_pud_clear(struct mm_struct *mm, pud_t pud)
+static inline void page_table_check_pud_clear(struct mm_struct *mm,
+					      unsigned long addr, pud_t pud)
 {
 	if (static_branch_likely(&page_table_check_disabled))
 		return;
-	__page_table_check_pud_clear(mm, pud);
+	__page_table_check_pud_clear(mm, addr, pud);
 }
 static inline void page_table_check_ptes_set(struct mm_struct *mm,
-		pte_t *ptep, pte_t pte, unsigned int nr)
+					     unsigned long addr, pte_t *ptep,
+					     pte_t pte, unsigned int nr)
 {
 	if (static_branch_likely(&page_table_check_disabled))
 		return;
-	__page_table_check_ptes_set(mm, ptep, pte, nr);
+	__page_table_check_ptes_set(mm, addr, ptep, pte, nr);
 }
 static inline void page_table_check_pmds_set(struct mm_struct *mm,
-		pmd_t *pmdp, pmd_t pmd, unsigned int nr)
+		unsigned long addr, pmd_t *pmdp, pmd_t pmd, unsigned int nr)
 {
 	if (static_branch_likely(&page_table_check_disabled))
 		return;
-	__page_table_check_pmds_set(mm, pmdp, pmd, nr);
+	__page_table_check_pmds_set(mm, addr, pmdp, pmd, nr);
 }
 static inline void page_table_check_puds_set(struct mm_struct *mm,
-		pud_t *pudp, pud_t pud, unsigned int nr)
+		unsigned long addr, pud_t *pudp, pud_t pud, unsigned int nr)
 {
 	if (static_branch_likely(&page_table_check_disabled))
 		return;
-	__page_table_check_puds_set(mm, pudp, pud, nr);
+	__page_table_check_puds_set(mm, addr, pudp, pud, nr);
 }
 static inline void page_table_check_pte_clear_range(struct mm_struct *mm,
 {
 }
-static inline void page_table_check_pte_clear(struct mm_struct *mm, pte_t pte)
+static inline void page_table_check_pte_clear(struct mm_struct *mm,
+					      unsigned long addr, pte_t pte)
 {
 }
-static inline void page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd)
+static inline void page_table_check_pmd_clear(struct mm_struct *mm,
+					      unsigned long addr, pmd_t pmd)
 {
 }
-static inline void page_table_check_pud_clear(struct mm_struct *mm, pud_t pud)
+static inline void page_table_check_pud_clear(struct mm_struct *mm,
+					      unsigned long addr, pud_t pud)
 {
 }
 static inline void page_table_check_ptes_set(struct mm_struct *mm,
-		pte_t *ptep, pte_t pte, unsigned int nr)
+					     unsigned long addr, pte_t *ptep,
+					     pte_t pte, unsigned int nr)
 {
 }
 static inline void page_table_check_pmds_set(struct mm_struct *mm,
-		pmd_t *pmdp, pmd_t pmd, unsigned int nr)
+		unsigned long addr, pmd_t *pmdp, pmd_t pmd, unsigned int nr)
 {
 }
 static inline void page_table_check_puds_set(struct mm_struct *mm,
-		pud_t *pudp, pud_t pud, unsigned int nr)
+		unsigned long addr, pud_t *pudp, pud_t pud, unsigned int nr)
 {
 }
 #endif /* CONFIG_PAGE_TABLE_CHECK */
-#define page_table_check_pmd_set(mm, pmdp, pmd)	page_table_check_pmds_set(mm, pmdp, pmd, 1)
+#define page_table_check_pmd_set(mm, addr, pmdp, pmd)	page_table_check_pmds_set(mm, addr, pmdp, pmd, 1)
-#define page_table_check_pud_set(mm, pudp, pud)	page_table_check_puds_set(mm, pudp, pud, 1)
+#define page_table_check_pud_set(mm, addr, pudp, pud)	page_table_check_puds_set(mm, addr, pudp, pud, 1)
 #endif /* __LINUX_PAGE_TABLE_CHECK_H */

include/linux/pgtable.h+154−14

@@ -225,16 +225,156 @@ static inline int pmd_dirty(pmd_t pmd) · 224 unmodified lines
225	* up to date.	225	* up to date.
226	*	226	*
227	* In the general case, no lock is guaranteed to be held between entry and exit	227	* In the general case, no lock is guaranteed to be held between entry and exit
228	* of the lazy mode. So the implementation must assume preemption may be enabled	228	* of the lazy mode. (In practice, for user PTE updates, the appropriate page
229	* and cpu migration is possible; it must take steps to be robust against this.	229	* table lock(s) are held, but for kernel PTE updates, no lock is held).
230	* (In practice, for user PTE updates, the appropriate page table lock(s) are	230	* The implementation must therefore assume preemption may be enabled upon
231	* held, but for kernel PTE updates, no lock is held). Nesting is not permitted	231	* entry to the mode and cpu migration is possible; it must take steps to be
232	* and the mode cannot be used in interrupt context.	232	* robust against this. An implementation may handle this by disabling
		233	* preemption, as a consequence generic code may not sleep while the lazy MMU
		234	* mode is active.
		235	*
		236	* The mode is disabled in interrupt context and calls to the lazy_mmu API have
		237	* no effect.
		238	*
		239	* The lazy MMU mode is enabled for a given block of code using:
		240	*
		241	* lazy_mmu_mode_enable();
		242	* <code>
		243	* lazy_mmu_mode_disable();
		244	*
		245	* Nesting is permitted: <code> may itself use an enable()/disable() pair.
		246	* A nested call to enable() has no functional effect; however disable() causes
		247	* any batched architectural state to be flushed regardless of nesting. After a
		248	* call to disable(), the caller can therefore rely on all previous page table
		249	* modifications to have taken effect, but the lazy MMU mode may still be
		250	* enabled.
		251	*
		252	* In certain cases, it may be desirable to temporarily pause the lazy MMU mode.
		253	* This can be done using:
		254	*
		255	* lazy_mmu_mode_pause();
		256	* <code>
		257	* lazy_mmu_mode_resume();
		258	*
		259	* pause() ensures that the mode is exited regardless of the nesting level;
		260	* resume() re-enters the mode at the same nesting level. Any call to the
		261	* lazy_mmu_mode_* API between those two calls has no effect. In particular,
		262	* this means that pause()/resume() pairs may nest.
		263	*
		264	* is_lazy_mmu_mode_active() can be used to check whether the lazy MMU mode is
		265	* currently enabled.
233	*/	266	*/
234	#ifndef __HAVE_ARCH_ENTER_LAZY_MMU_MODE	267	#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE
235	static inline void arch_enter_lazy_mmu_mode(void) {}	268	/**
236	static inline void arch_leave_lazy_mmu_mode(void) {}	269	* lazy_mmu_mode_enable() - Enable the lazy MMU mode.
237	static inline void arch_flush_lazy_mmu_mode(void) {}	270	*
		271	* Enters a new lazy MMU mode section; if the mode was not already enabled,
		272	* enables it and calls arch_enter_lazy_mmu_mode().
		273	*
		274	* Must be paired with a call to lazy_mmu_mode_disable().
		275	*
		276	* Has no effect if called:
		277	* - While paused - see lazy_mmu_mode_pause()
		278	* - In interrupt context
		279	*/
		280	static inline void lazy_mmu_mode_enable(void)
		281	{
		282	struct lazy_mmu_state *state = &current->lazy_mmu_state;
		283
		284	if (in_interrupt() \|\| state->pause_count > 0)
		285	return;
		286
		287	VM_WARN_ON_ONCE(state->enable_count == U8_MAX);
		288
		289	if (state->enable_count++ == 0)
		290	arch_enter_lazy_mmu_mode();
		291	}
		292
		293	/**
		294	* lazy_mmu_mode_disable() - Disable the lazy MMU mode.
		295	*
		296	* Exits the current lazy MMU mode section. If it is the outermost section,
		297	* disables the mode and calls arch_leave_lazy_mmu_mode(). Otherwise (nested
		298	* section), calls arch_flush_lazy_mmu_mode().
		299	*
		300	* Must match a call to lazy_mmu_mode_enable().
		301	*
		302	* Has no effect if called:
		303	* - While paused - see lazy_mmu_mode_pause()
		304	* - In interrupt context
		305	*/
		306	static inline void lazy_mmu_mode_disable(void)
		307	{
		308	struct lazy_mmu_state *state = &current->lazy_mmu_state;
		309
		310	if (in_interrupt() \|\| state->pause_count > 0)
		311	return;
		312
		313	VM_WARN_ON_ONCE(state->enable_count == 0);
		314
		315	if (--state->enable_count == 0)
		316	arch_leave_lazy_mmu_mode();
		317	else /* Exiting a nested section */
		318	arch_flush_lazy_mmu_mode();
		319
		320	}
		321
		322	/**
		323	* lazy_mmu_mode_pause() - Pause the lazy MMU mode.
		324	*
		325	* Pauses the lazy MMU mode; if it is currently active, disables it and calls
		326	* arch_leave_lazy_mmu_mode().
		327	*
		328	* Must be paired with a call to lazy_mmu_mode_resume(). Calls to the
		329	* lazy_mmu_mode_* API have no effect until the matching resume() call.
		330	*
		331	* Has no effect if called:
		332	* - While paused (inside another pause()/resume() pair)
		333	* - In interrupt context
		334	*/
		335	static inline void lazy_mmu_mode_pause(void)
		336	{
		337	struct lazy_mmu_state *state = &current->lazy_mmu_state;
		338
		339	if (in_interrupt())
		340	return;
		341
		342	VM_WARN_ON_ONCE(state->pause_count == U8_MAX);
		343
		344	if (state->pause_count++ == 0 && state->enable_count > 0)
		345	arch_leave_lazy_mmu_mode();
		346	}
		347
		348	/**
		349	* lazy_mmu_mode_resume() - Resume the lazy MMU mode.
		350	*
		351	* Resumes the lazy MMU mode; if it was active at the point where the matching
		352	* call to lazy_mmu_mode_pause() was made, re-enables it and calls
		353	* arch_enter_lazy_mmu_mode().
		354	*
		355	* Must match a call to lazy_mmu_mode_pause().
		356	*
		357	* Has no effect if called:
		358	* - While paused (inside another pause()/resume() pair)
		359	* - In interrupt context
		360	*/
		361	static inline void lazy_mmu_mode_resume(void)
		362	{
		363	struct lazy_mmu_state *state = &current->lazy_mmu_state;
		364
		365	if (in_interrupt())
		366	return;
		367
		368	VM_WARN_ON_ONCE(state->pause_count == 0);
		369
		370	if (--state->pause_count == 0 && state->enable_count > 0)
		371	arch_enter_lazy_mmu_mode();
		372	}
		373	#else
		374	static inline void lazy_mmu_mode_enable(void) {}
		375	static inline void lazy_mmu_mode_disable(void) {}
		376	static inline void lazy_mmu_mode_pause(void) {}
		377	static inline void lazy_mmu_mode_resume(void) {}
238	#endif	378	#endif
239		379
240	#ifndef pte_batch_hint	380	#ifndef pte_batch_hint
@@ -289,7 +429,7 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) · 48 unmodified lines
289	static inline void set_ptes(struct mm_struct *mm, unsigned long addr,	429	static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
290	pte_t *ptep, pte_t pte, unsigned int nr)	430	pte_t *ptep, pte_t pte, unsigned int nr)
291	{	431	{
292	page_table_check_ptes_set(mm, ptep, pte, nr);	432	page_table_check_ptes_set(mm, addr, ptep, pte, nr);
293		433
294	for (;;) {	434	for (;;) {
295	set_pte(ptep, pte);	435	set_pte(ptep, pte);
@@ -494,7 +634,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · 198 unmodified lines
494	{	634	{
495	pte_t pte = ptep_get(ptep);	635	pte_t pte = ptep_get(ptep);
496	pte_clear(mm, address, ptep);	636	pte_clear(mm, address, ptep);
497	page_table_check_pte_clear(mm, pte);	637	page_table_check_pte_clear(mm, address, pte);
498	return pte;	638	return pte;
499	}	639	}
500	#endif	640	#endif
@@ -553,7 +693,7 @@ static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, · 52 unmodified lines
553	* No need for ptep_get_and_clear(): page table check doesn't care about	693	* No need for ptep_get_and_clear(): page table check doesn't care about
554	* any bits that could have been set by HW concurrently.	694	* any bits that could have been set by HW concurrently.
555	*/	695	*/
556	page_table_check_pte_clear(mm, pte);	696	page_table_check_pte_clear(mm, addr, pte);
557	}	697	}
558		698
559	#ifdef CONFIG_GUP_GET_PXX_LOW_HIGH	699	#ifdef CONFIG_GUP_GET_PXX_LOW_HIGH
@@ -648,7 +788,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, · 88 unmodified lines
648	pmd_t pmd = *pmdp;	788	pmd_t pmd = *pmdp;
649		789
650	pmd_clear(pmdp);	790	pmd_clear(pmdp);
651	page_table_check_pmd_clear(mm, pmd);	791	page_table_check_pmd_clear(mm, address, pmd);
652		792
653	return pmd;	793	return pmd;
654	}	794	}
@@ -661,7 +801,7 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, · 6 unmodified lines
661	pud_t pud = *pudp;	801	pud_t pud = *pudp;
662		802
663	pud_clear(pudp);	803	pud_clear(pudp);
664	page_table_check_pud_clear(mm, pud);	804	page_table_check_pud_clear(mm, address, pud);
665		805
666	return pud;	806	return pud;
667	}	807	}

include/linux/rmap.h+3−73

@@ -92,6 +92,7 @@ struct anon_vma_chain { · 91 unmodified lines
92	};	92	};
93		93
94	enum ttu_flags {	94	enum ttu_flags {
		95	TTU_USE_SHARED_ZEROPAGE = 0x2, /* for unused pages of large folios */
95	TTU_SPLIT_HUGE_PMD = 0x4, /* split huge PMD if any */	96	TTU_SPLIT_HUGE_PMD = 0x4, /* split huge PMD if any */
96	TTU_IGNORE_MLOCK = 0x8, /* ignore mlock */	97	TTU_IGNORE_MLOCK = 0x8, /* ignore mlock */
97	TTU_SYNC = 0x10, /* avoid racy checks with PVMW_SYNC */	98	TTU_SYNC = 0x10, /* avoid racy checks with PVMW_SYNC */
@@ -104,75 +105,8 @@ enum ttu_flags { · 6 unmodified lines
104	};	105	};
105		106
106	#ifdef CONFIG_MMU	107	#ifdef CONFIG_MMU
107	static inline void get_anon_vma(struct anon_vma *anon_vma)
108	{
109	atomic_inc(&anon_vma->refcount);
110	}
111
112	void __put_anon_vma(struct anon_vma *anon_vma);
113
114	static inline void put_anon_vma(struct anon_vma *anon_vma)
115	{
116	if (atomic_dec_and_test(&anon_vma->refcount))
117	__put_anon_vma(anon_vma);
118	}
119
120	static inline void anon_vma_lock_write(struct anon_vma *anon_vma)
121	{
122	down_write(&anon_vma->root->rwsem);
123	}
124
125	static inline int anon_vma_trylock_write(struct anon_vma *anon_vma)
126	{
127	return down_write_trylock(&anon_vma->root->rwsem);
128	}
129
130	static inline void anon_vma_unlock_write(struct anon_vma *anon_vma)
131	{
132	up_write(&anon_vma->root->rwsem);
133	}
134
135	static inline void anon_vma_lock_read(struct anon_vma *anon_vma)
136	{
137	down_read(&anon_vma->root->rwsem);
138	}
139
140	static inline int anon_vma_trylock_read(struct anon_vma *anon_vma)
141	{
142	return down_read_trylock(&anon_vma->root->rwsem);
143	}
144
145	static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)
146	{
147	up_read(&anon_vma->root->rwsem);
148	}
149		108
150
151	/*
152	* anon_vma helper functions.
153	*/
154	void anon_vma_init(void); /* create anon_vma_cachep */	109	void anon_vma_init(void); /* create anon_vma_cachep */
155	int __anon_vma_prepare(struct vm_area_struct *);
156	void unlink_anon_vmas(struct vm_area_struct *);
157	int anon_vma_clone(struct vm_area_struct , struct vm_area_struct );
158	int anon_vma_fork(struct vm_area_struct , struct vm_area_struct );
159
160	static inline int anon_vma_prepare(struct vm_area_struct *vma)
161	{
162	if (likely(vma->anon_vma))
163	return 0;
164
165	return __anon_vma_prepare(vma);
166	}
167
168	static inline void anon_vma_merge(struct vm_area_struct *vma,
169	struct vm_area_struct *next)
170	{
171	VM_BUG_ON_VMA(vma->anon_vma != next->anon_vma, vma);
172	unlink_anon_vmas(next);
173	}
174
175	struct anon_vma folio_get_anon_vma(const struct folio folio);
176		110
177	#ifdef CONFIG_MM_ID	111	#ifdef CONFIG_MM_ID
178	static __always_inline void folio_lock_large_mapcount(struct folio *folio)	112	static __always_inline void folio_lock_large_mapcount(struct folio *folio)
@@ -1000,12 +934,8 @@ int mapping_wrprotect_range(struct address_space *mapping, pgoff_t pgoff, · 821 unmodified lines
1000	int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,	934	int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,
1001	struct vm_area_struct *vma);	935	struct vm_area_struct *vma);
1002		936
1003	enum rmp_flags {	937	void remove_migration_ptes(struct folio src, struct folio dst,
1004	RMP_LOCKED = 1 << 0,	938	enum ttu_flags flags);
1005	RMP_USE_SHARED_ZEROPAGE = 1 << 1,
1006	};
1007
1008	void remove_migration_ptes(struct folio src, struct folio dst, int flags);
1009		939
1010	/*	940	/*
1011	* rmap_walk_control: To control rmap traversing for specific needs	941	* rmap_walk_control: To control rmap traversing for specific needs

include/linux/sched.h+45−0

@@ -1410,6 +1410,10 @@ struct task_struct { · 1409 unmodified lines
1410		1410
1411	struct page_frag task_frag;	1411	struct page_frag task_frag;
1412		1412
		1413	#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE
		1414	struct lazy_mmu_state lazy_mmu_state;
		1415	#endif
		1416
1413	#ifdef CONFIG_TASK_DELAY_ACCT	1417	#ifdef CONFIG_TASK_DELAY_ACCT
1414	struct task_delay_info *delays;	1418	struct task_delay_info *delays;
1415	#endif	1419	#endif
@@ -1693,6 +1697,47 @@ static inline char task_state_to_char(struct task_struct *tsk) · 277 unmodified lines
1693	return task_index_to_char(task_state_index(tsk));	1697	return task_index_to_char(task_state_index(tsk));
1694	}	1698	}
1695		1699
		1700	#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE
		1701	/**
		1702	* __task_lazy_mmu_mode_active() - Test the lazy MMU mode state for a task.
		1703	* @tsk: The task to check.
		1704	*
		1705	* Test whether @tsk has its lazy MMU mode state set to active (i.e. enabled
		1706	* and not paused).
		1707	*
		1708	* This function only considers the state saved in task_struct; to test whether
		1709	* current actually is in lazy MMU mode, is_lazy_mmu_mode_active() should be
		1710	* used instead.
		1711	*
		1712	* This function is intended for architectures that implement the lazy MMU
		1713	* mode; it must not be called from generic code.
		1714	*/
		1715	static inline bool __task_lazy_mmu_mode_active(struct task_struct *tsk)
		1716	{
		1717	struct lazy_mmu_state *state = &tsk->lazy_mmu_state;
		1718
		1719	return state->enable_count > 0 && state->pause_count == 0;
		1720	}
		1721
		1722	/**
		1723	* is_lazy_mmu_mode_active() - Test whether we are currently in lazy MMU mode.
		1724	*
		1725	* Test whether the current context is in lazy MMU mode. This is true if both:
		1726	* 1. We are not in interrupt context
		1727	* 2. Lazy MMU mode is active for the current task
		1728	*
		1729	* This function is intended for architectures that implement the lazy MMU
		1730	* mode; it must not be called from generic code.
		1731	*/
		1732	static inline bool is_lazy_mmu_mode_active(void)
		1733	{
		1734	if (in_interrupt())
		1735	return false;
		1736
		1737	return __task_lazy_mmu_mode_active(current);
		1738	}
		1739	#endif
		1740
1696	extern struct pid *cad_pid;	1741	extern struct pid *cad_pid;
1697		1742
1698	/*	1743	/*

include/linux/swap.h+25−46

 #define COMPACT_CLUSTER_MAX SWAP_CLUSTER_MAX
 /* Bit flag in swap_map */
-#define SWAP_HAS_CACHE	0x40	/* Flag page is cached, in first swap_map */
 #define COUNT_CONTINUED	0x80	/* Flag swap_map continuation for full count */
 /* Special value in first swap_map */
 #define SWAP_MAP_MAX	0x3e	/* Max count */
 #define SWAP_MAP_BAD	0x3f	/* Note page is bad */
-#define SWAP_MAP_SHMEM	0xbf	/* Owned by shmem/tmpfs */
 /* Special value in each swap_map continuation */
 #define SWAP_CONT_MAX	0x7f	/* Max count */
 }
 extern void si_swapinfo(struct sysinfo *);
-int folio_alloc_swap(struct folio *folio);
-bool folio_free_swap(struct folio *folio);
-void put_swap_folio(struct folio *folio, swp_entry_t entry);
-extern swp_entry_t get_swap_page_of_type(int);
 extern int add_swap_count_continuation(swp_entry_t, gfp_t);
-extern void swap_shmem_alloc(swp_entry_t, int);
-extern int swap_duplicate(swp_entry_t);
-extern int swapcache_prepare(swp_entry_t entry, int nr);
-extern void swap_free_nr(swp_entry_t entry, int nr_pages);
-extern void free_swap_and_cache_nr(swp_entry_t entry, int nr);
 int swap_type_of(dev_t device, sector_t offset);
 int find_first_swap(dev_t *device);
 extern unsigned int count_swap_pages(int, int);
 extern struct swap_info_struct *get_swap_device(swp_entry_t entry);
 sector_t swap_folio_sector(struct folio *folio);
+/*
+ * If there is an existing swap slot reference (swap entry) and the caller
+ * guarantees that there is no race modification of it (e.g., PTL
+ * protecting the swap entry in page table; shmem's cmpxchg protects t
+ * he swap entry in shmem mapping), these two helpers below can be used
+ * to put/dup the entries directly.
+ *
+ * All entries must be allocated by folio_alloc_swap(). And they must have
+ * a swap count > 1. See comments of folio_*_swap helpers for more info.
+ */
+int swap_dup_entry_direct(swp_entry_t entry);
+void swap_put_entries_direct(swp_entry_t entry, int nr);
+/*
+ * folio_free_swap tries to free the swap entries pinned by a swap cache
+ * folio, it has to be here to be called by other components.
+ */
+bool folio_free_swap(struct folio *folio);
+/* Allocate / free (hibernation) exclusive entries */
+swp_entry_t swap_alloc_hibernation_slot(int type);
+void swap_free_hibernation_slot(swp_entry_t entry);
 static inline void put_swap_device(struct swap_info_struct *si)
 {
 	percpu_ref_put(&si->users);
 #define free_pages_and_swap_cache(pages, nr) \
 	release_pages((pages), (nr));
-static inline void free_swap_and_cache_nr(swp_entry_t entry, int nr)
-{
-}
 static inline void free_swap_cache(struct folio *folio)
 {
 }
 	return 0;
 }
-static inline void swap_shmem_alloc(swp_entry_t swp, int nr)
+static inline int swap_dup_entry_direct(swp_entry_t ent)
-{
-}
-static inline int swap_duplicate(swp_entry_t swp)
-{
-	return 0;
-}
-static inline int swapcache_prepare(swp_entry_t swp, int nr)
 {
 	return 0;
 }
-static inline void swap_free_nr(swp_entry_t entry, int nr_pages)
+static inline void swap_put_entries_direct(swp_entry_t ent, int nr)
-{
-}
-static inline void put_swap_folio(struct folio *folio, swp_entry_t swp)
 {
 }
 	return 0;
 }
-static inline int folio_alloc_swap(struct folio *folio)
-{
-	return -EINVAL;
-}
 static inline bool folio_free_swap(struct folio *folio)
 {
 	return false;
 	return -EINVAL;
 }
 #endif /* CONFIG_SWAP */
-static inline void free_swap_and_cache(swp_entry_t entry)
-{
-	free_swap_and_cache_nr(entry, 1);
-}
-static inline void swap_free(swp_entry_t entry)
-{
-	swap_free_nr(entry, 1);
-}
 #ifdef CONFIG_MEMCG
 static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
 {

include/linux/vm_event_item.h+4−4

@@ -122,13 +122,13 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, · 121 unmodified lines
122	THP_SWPOUT,	122	THP_SWPOUT,
123	THP_SWPOUT_FALLBACK,	123	THP_SWPOUT_FALLBACK,
124	#endif	124	#endif
125	#ifdef CONFIG_MEMORY_BALLOON	125	#ifdef CONFIG_BALLOON
126	BALLOON_INFLATE,	126	BALLOON_INFLATE,
127	BALLOON_DEFLATE,	127	BALLOON_DEFLATE,
128	#ifdef CONFIG_BALLOON_COMPACTION	128	#ifdef CONFIG_BALLOON_MIGRATION
129	BALLOON_MIGRATE,	129	BALLOON_MIGRATE,
130	#endif	130	#endif /* CONFIG_BALLOON_MIGRATION */
131	#endif	131	#endif /* CONFIG_BALLOON */
132	#ifdef CONFIG_DEBUG_TLBFLUSH	132	#ifdef CONFIG_DEBUG_TLBFLUSH
133	NR_TLB_REMOTE_FLUSH, /* cpu tried to flush others' tlbs */	133	NR_TLB_REMOTE_FLUSH, /* cpu tried to flush others' tlbs */
134	NR_TLB_REMOTE_FLUSH_RECEIVED,/* cpu received ipi for flush */	134	NR_TLB_REMOTE_FLUSH_RECEIVED,/* cpu received ipi for flush */

include/linux/vmstat.h+0−6

 void inc_node_page_state(struct page *, enum node_stat_item);
 void dec_node_page_state(struct page *, enum node_stat_item);
-extern void inc_node_state(struct pglist_data *, enum node_stat_item);
 extern void __inc_zone_state(struct zone *, enum zone_stat_item);
 extern void __inc_node_state(struct pglist_data *, enum node_stat_item);
-extern void dec_zone_state(struct zone *, enum zone_stat_item);
 extern void __dec_zone_state(struct zone *, enum zone_stat_item);
 extern void __dec_node_state(struct pglist_data *, enum node_stat_item);
 #define dec_node_page_state __dec_node_page_state
 #define mod_node_page_state __mod_node_page_state
-#define inc_zone_state __inc_zone_state
-#define inc_node_state __inc_node_state
-#define dec_zone_state __dec_zone_state
 #define set_pgdat_percpu_threshold(pgdat, callback) { }
 static inline void refresh_zone_stat_thresholds(void) { }

include/linux/writeback.h+0−4

 	bool			dirty_exceeded;
 };
-void laptop_io_completion(struct backing_dev_info *info);
-void laptop_sync_completion(void);
-void laptop_mode_timer_fn(struct timer_list *t);
 bool node_dirty_ok(struct pglist_data *pgdat);
 int wb_domain_init(struct wb_domain *dom, gfp_t gfp);
 #ifdef CONFIG_CGROUP_WRITEBACK
 /* These are exported to sysctl. */
 extern unsigned int dirty_writeback_interval;
 extern unsigned int dirty_expire_interval;
-extern int laptop_mode;
 void global_dirty_limits(unsigned long *pbackground, unsigned long *pdirty);
 unsigned long wb_calc_thresh(struct bdi_writeback *wb, unsigned long thresh);

include/linux/zsmalloc.h+6−2

 };
 struct zs_pool;
+struct scatterlist;
 struct zs_pool *zs_create_pool(const char *name);
 void zs_destroy_pool(struct zs_pool *pool);
 void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats);
 void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle,
-			void *local_copy);
+			size_t mem_len, void *local_copy);
 void zs_obj_read_end(struct zs_pool *pool, unsigned long handle,
-		     void *handle_mem);
+		     size_t mem_len, void *handle_mem);
+void zs_obj_read_sg_begin(struct zs_pool *pool, unsigned long handle,
+			  struct scatterlist *sg, size_t mem_len);
+void zs_obj_read_sg_end(struct zs_pool *pool, unsigned long handle);
 void zs_obj_write(struct zs_pool *pool, unsigned long handle,
 		  void *handle_mem, size_t mem_len);

include/trace/events/damon.h+41−0

@@ -9,6 +9,47 @@ · 8 unmodified lines
9	#include <linux/types.h>	9	#include <linux/types.h>
10	#include <linux/tracepoint.h>	10	#include <linux/tracepoint.h>
11		11
		12	TRACE_EVENT(damos_stat_after_apply_interval,
		13
		14	TP_PROTO(unsigned int context_idx, unsigned int scheme_idx,
		15	struct damos_stat *stat),
		16
		17	TP_ARGS(context_idx, scheme_idx, stat),
		18
		19	TP_STRUCT__entry(
		20	__field(unsigned int, context_idx)
		21	__field(unsigned int, scheme_idx)
		22	__field(unsigned long, nr_tried)
		23	__field(unsigned long, sz_tried)
		24	__field(unsigned long, nr_applied)
		25	__field(unsigned long, sz_applied)
		26	__field(unsigned long, sz_ops_filter_passed)
		27	__field(unsigned long, qt_exceeds)
		28	__field(unsigned long, nr_snapshots)
		29	),
		30
		31	TP_fast_assign(
		32	__entry->context_idx = context_idx;
		33	__entry->scheme_idx = scheme_idx;
		34	__entry->nr_tried = stat->nr_tried;
		35	__entry->sz_tried = stat->sz_tried;
		36	__entry->nr_applied = stat->nr_applied;
		37	__entry->sz_applied = stat->sz_applied;
		38	__entry->sz_ops_filter_passed = stat->sz_ops_filter_passed;
		39	__entry->qt_exceeds = stat->qt_exceeds;
		40	__entry->nr_snapshots = stat->nr_snapshots;
		41	),
		42
		43	TP_printk("ctx_idx=%u scheme_idx=%u nr_tried=%lu sz_tried=%lu "
		44	"nr_applied=%lu sz_tried=%lu sz_ops_filter_passed=%lu "
		45	"qt_exceeds=%lu nr_snapshots=%lu",
		46	__entry->context_idx, __entry->scheme_idx,
		47	__entry->nr_tried, __entry->sz_tried,
		48	__entry->nr_applied, __entry->sz_applied,
		49	__entry->sz_ops_filter_passed, __entry->qt_exceeds,
		50	__entry->nr_snapshots)
		51	);
		52
12	TRACE_EVENT(damos_esz,	53	TRACE_EVENT(damos_esz,
13		54
14	TP_PROTO(unsigned int context_idx, unsigned int scheme_idx,	55	TP_PROTO(unsigned int context_idx, unsigned int scheme_idx,

include/trace/events/huge_memory.h+2−1

@@ -37,7 +37,8 @@ · 36 unmodified lines
37	EM( SCAN_PAGE_HAS_PRIVATE, "page_has_private") \	37	EM( SCAN_PAGE_HAS_PRIVATE, "page_has_private") \
38	EM( SCAN_STORE_FAILED, "store_failed") \	38	EM( SCAN_STORE_FAILED, "store_failed") \
39	EM( SCAN_COPY_MC, "copy_poisoned_page") \	39	EM( SCAN_COPY_MC, "copy_poisoned_page") \
40	EMe(SCAN_PAGE_FILLED, "page_filled")	40	EM( SCAN_PAGE_FILLED, "page_filled") \
		41	EMe(SCAN_PAGE_DIRTY_OR_WRITEBACK, "page_dirty_or_writeback")
41		42
42	#undef EM	43	#undef EM
43	#undef EMe	44	#undef EMe

include/trace/events/vmscan.h+51−0

@@ -40,6 +40,16 @@ · 39 unmodified lines
40	{_VMSCAN_THROTTLE_CONGESTED, "VMSCAN_THROTTLE_CONGESTED"} \	40	{_VMSCAN_THROTTLE_CONGESTED, "VMSCAN_THROTTLE_CONGESTED"} \
41	) : "VMSCAN_THROTTLE_NONE"	41	) : "VMSCAN_THROTTLE_NONE"
42		42
		43	TRACE_DEFINE_ENUM(KSWAPD_CLEAR_HOPELESS_OTHER);
		44	TRACE_DEFINE_ENUM(KSWAPD_CLEAR_HOPELESS_KSWAPD);
		45	TRACE_DEFINE_ENUM(KSWAPD_CLEAR_HOPELESS_DIRECT);
		46	TRACE_DEFINE_ENUM(KSWAPD_CLEAR_HOPELESS_PCP);
		47
		48	#define kswapd_clear_hopeless_reason_ops \
		49	{KSWAPD_CLEAR_HOPELESS_KSWAPD, "KSWAPD"}, \
		50	{KSWAPD_CLEAR_HOPELESS_DIRECT, "DIRECT"}, \
		51	{KSWAPD_CLEAR_HOPELESS_PCP, "PCP"}, \
		52	{KSWAPD_CLEAR_HOPELESS_OTHER, "OTHER"}
43		53
44	#define trace_reclaim_flags(file) ( \	54	#define trace_reclaim_flags(file) ( \
45	(file ? RECLAIM_WB_FILE : RECLAIM_WB_ANON) \| \	55	(file ? RECLAIM_WB_FILE : RECLAIM_WB_ANON) \| \
@@ -535,6 +545,47 @@ TRACE_EVENT(mm_vmscan_throttled, · 489 unmodified lines
535	__entry->usec_delayed,	545	__entry->usec_delayed,
536	show_throttle_flags(__entry->reason))	546	show_throttle_flags(__entry->reason))
537	);	547	);
		548
		549	TRACE_EVENT(mm_vmscan_kswapd_reclaim_fail,
		550
		551	TP_PROTO(int nid, int failures),
		552
		553	TP_ARGS(nid, failures),
		554
		555	TP_STRUCT__entry(
		556	__field(int, nid)
		557	__field(int, failures)
		558	),
		559
		560	TP_fast_assign(
		561	__entry->nid = nid;
		562	__entry->failures = failures;
		563	),
		564
		565	TP_printk("nid=%d failures=%d",
		566	__entry->nid, __entry->failures)
		567	);
		568
		569	TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless,
		570
		571	TP_PROTO(int nid, int reason),
		572
		573	TP_ARGS(nid, reason),
		574
		575	TP_STRUCT__entry(
		576	__field(int, nid)
		577	__field(int, reason)
		578	),
		579
		580	TP_fast_assign(
		581	__entry->nid = nid;
		582	__entry->reason = reason;
		583	),
		584
		585	TP_printk("nid=%d reason=%s",
		586	__entry->nid,
		587	__print_symbolic(__entry->reason, kswapd_clear_hopeless_reason_ops))
		588	);
538	#endif /* _TRACE_VMSCAN_H */	589	#endif /* _TRACE_VMSCAN_H */
539		590
540	/* This part must be outside protection */	591	/* This part must be outside protection */

include/trace/events/writeback.h+0−1

@@ -42,7 +42,6 @@ · 41 unmodified lines
42	EM( WB_REASON_VMSCAN, "vmscan") \	42	EM( WB_REASON_VMSCAN, "vmscan") \
43	EM( WB_REASON_SYNC, "sync") \	43	EM( WB_REASON_SYNC, "sync") \
44	EM( WB_REASON_PERIODIC, "periodic") \	44	EM( WB_REASON_PERIODIC, "periodic") \
45	EM( WB_REASON_LAPTOP_TIMER, "laptop_timer") \
46	EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \	45	EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \
47	EM( WB_REASON_FORKER_THREAD, "forker_thread") \	46	EM( WB_REASON_FORKER_THREAD, "forker_thread") \
48	EMe(WB_REASON_FOREIGN_FLUSH, "foreign_flush")	47	EMe(WB_REASON_FOREIGN_FLUSH, "foreign_flush")

include/uapi/linux/mempolicy.h+3−0

@@ -39,6 +39,9 @@ enum { · 38 unmodified lines
39	#define MPOL_MODE_FLAGS \	39	#define MPOL_MODE_FLAGS \
40	(MPOL_F_STATIC_NODES \| MPOL_F_RELATIVE_NODES \| MPOL_F_NUMA_BALANCING)	40	(MPOL_F_STATIC_NODES \| MPOL_F_RELATIVE_NODES \| MPOL_F_NUMA_BALANCING)
41		41
		42	/* Whether the nodemask is specified by users */
		43	#define MPOL_USER_NODEMASK_FLAGS (MPOL_F_STATIC_NODES \| MPOL_F_RELATIVE_NODES)
		44
42	/* Flags for get_mempolicy */	45	/* Flags for get_mempolicy */
43	#define MPOL_F_NODE (1<<0) /* return next IL mode instead of node mask */	46	#define MPOL_F_NODE (1<<0) /* return next IL mode instead of node mask */
44	#define MPOL_F_ADDR (1<<1) /* look up vma using address */	47	#define MPOL_F_ADDR (1<<1) /* look up vma using address */

include/uapi/linux/sysctl.h+1−1

@@ -182,7 +182,7 @@ enum · 181 unmodified lines
182	VM_LOWMEM_RESERVE_RATIO=20,/* reservation ratio for lower memory zones */	182	VM_LOWMEM_RESERVE_RATIO=20,/* reservation ratio for lower memory zones */
183	VM_MIN_FREE_KBYTES=21, /* Minimum free kilobytes to maintain */	183	VM_MIN_FREE_KBYTES=21, /* Minimum free kilobytes to maintain */
184	VM_MAX_MAP_COUNT=22, /* int: Maximum number of mmaps/address-space */	184	VM_MAX_MAP_COUNT=22, /* int: Maximum number of mmaps/address-space */
185	VM_LAPTOP_MODE=23, /* vm laptop mode */	185
186	VM_BLOCK_DUMP=24, /* block dump mode */	186	VM_BLOCK_DUMP=24, /* block dump mode */
187	VM_HUGETLB_GROUP=25, /* permitted hugetlb group */	187	VM_HUGETLB_GROUP=25, /* permitted hugetlb group */
188	VM_VFS_CACHE_PRESSURE=26, /* dcache/icache reclaim pressure */	188	VM_VFS_CACHE_PRESSURE=26, /* dcache/icache reclaim pressure */

init/main.c+1−0

@@ -1025,6 +1025,7 @@ void start_kernel(void) · 1024 unmodified lines
1025	page_address_init();	1025	page_address_init();
1026	pr_notice("%s", linux_banner);	1026	pr_notice("%s", linux_banner);
1027	setup_arch(&command_line);	1027	setup_arch(&command_line);
		1028	mm_core_init_early();
1028	/* Static keys and static calls are needed by LSMs */	1029	/* Static keys and static calls are needed by LSMs */
1029	jump_label_init();	1030	jump_label_init();
1030	static_call_init();	1031	static_call_init();

kernel/cgroup/cpuset.c+3−4

  */
 static void guarantee_online_mems(struct cpuset *cs, nodemask_t *pmask)
 {
-	while (!nodes_intersects(cs->effective_mems, node_states[N_MEMORY]))
+	while (!nodes_and(*pmask, cs->effective_mems, node_states[N_MEMORY]))
 		cs = parent_cs(cs);
-	nodes_and(*pmask, cs->effective_mems, node_states[N_MEMORY]);
 }
 /**
 	cpuset_for_each_descendant_pre(cp, pos_css, cs) {
 		struct cpuset *parent = parent_cs(cp);
-		nodes_and(*new_mems, cp->mems_allowed, parent->effective_mems);
+		bool has_mems = nodes_and(*new_mems, cp->mems_allowed, parent->effective_mems);
 		/*
 		 * If it becomes empty, inherit the effective mask of the
 		 * parent, which is guaranteed to have some MEMs.
 		 */
-		if (is_in_v2_mode() && nodes_empty(*new_mems))
+		if (is_in_v2_mode() && !has_mems)
 			*new_mems = parent->effective_mems;
 		/* Skip the whole subtree if the nodemask remains the same. */

kernel/power/swap.c+6−4

@@ -174,10 +174,10 @@ sector_t alloc_swapdev_block(int swap) · 173 unmodified lines
174	* Allocate a swap page and register that it has been allocated, so that	174	* Allocate a swap page and register that it has been allocated, so that
175	* it can be freed in case of an error.	175	* it can be freed in case of an error.
176	*/	176	*/
177	offset = swp_offset(get_swap_page_of_type(swap));	177	offset = swp_offset(swap_alloc_hibernation_slot(swap));
178	if (offset) {	178	if (offset) {
179	if (swsusp_extents_insert(offset))	179	if (swsusp_extents_insert(offset))
180	swap_free(swp_entry(swap, offset));	180	swap_free_hibernation_slot(swp_entry(swap, offset));
181	else	181	else
182	return swapdev_block(swap, offset);	182	return swapdev_block(swap, offset);
183	}	183	}
@@ -186,6 +186,7 @@ sector_t alloc_swapdev_block(int swap) · 2 unmodified lines
186		186
187	void free_all_swap_pages(int swap)	187	void free_all_swap_pages(int swap)
188	{	188	{
		189	unsigned long offset;
189	struct rb_node *node;	190	struct rb_node *node;
190		191
191	/*	192	/*
@@ -197,8 +198,9 @@ void free_all_swap_pages(int swap) · 5 unmodified lines
197		198
198	ext = rb_entry(node, struct swsusp_extent, node);	199	ext = rb_entry(node, struct swsusp_extent, node);
199	rb_erase(node, &swsusp_extents);	200	rb_erase(node, &swsusp_extents);
200	swap_free_nr(swp_entry(swap, ext->start),	201
201	ext->end - ext->start + 1);	202	for (offset = ext->start; offset <= ext->end; offset++)
		203	swap_free_hibernation_slot(swp_entry(swap, offset));
202		204
203	kfree(ext);	205	kfree(ext);
204	}	206	}

lib/alloc_tag.c+17−10

 static int proc_mem_profiling_handler(const struct ctl_table *table, int write,
 				      void *buffer, size_t *lenp, loff_t *ppos)
 {
-	if (!mem_profiling_support && write)
+	if (write) {
-		return -EINVAL;
+		/*
+		 * Call from do_sysctl_args() which is a no-op since the same
+		 * value was already set by setup_early_mem_profiling.
+		 * Return success to avoid warnings from do_sysctl_args().
+		 */
+		if (!current->mm)
+			return 0;
+#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG
+		/* User can't toggle profiling while debugging */
+		return -EACCES;
+#endif
+		if (!mem_profiling_support)
+			return -EINVAL;
+	}
 	return proc_do_static_key(table, write, buffer, lenp, ppos);
 }
-static struct ctl_table memory_allocation_profiling_sysctls[] = {
+static const struct ctl_table memory_allocation_profiling_sysctls[] = {
 	{
 		.procname	= "mem_profiling",
 		.data		= &mem_alloc_profiling_key,
-#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG
-		.mode		= 0444,
-#else
 		.mode		= 0644,
-#endif
 		.proc_handler	= proc_mem_profiling_handler,
 	},
 };
 static void __init sysctl_init(void)
 {
-	if (!mem_profiling_support)
-		memory_allocation_profiling_sysctls[0].mode = 0444;
 	register_sysctl_init("vm", memory_allocation_profiling_sysctls);
 }
 #else /* CONFIG_SYSCTL */

lib/test_vmalloc.c+7−4

 		/* Add a new test case description here. */
 );
+__param(int, nr_pcpu_objects, 35000,
+	"Number of pcpu objects to allocate for pcpu_alloc_test");
 /*
  * This is for synchronization of setup phase.
  */
 	size_t size, align;
 	int i;
-	pcpu = vmalloc(sizeof(void __percpu *) * 35000);
+	pcpu = vmalloc(sizeof(void __percpu *) * nr_pcpu_objects);
 	if (!pcpu)
 		return -1;
-	for (i = 0; i < 35000; i++) {
+	for (i = 0; i < nr_pcpu_objects; i++) {
 		size = get_random_u32_inclusive(1, PAGE_SIZE / 4);
 		/*
 		 * Maximum PAGE_SIZE
 		 */
-		align = 1 << get_random_u32_inclusive(1, 11);
+		align = 1 << get_random_u32_inclusive(1, PAGE_SHIFT - 1);
 		pcpu[i] = __alloc_percpu(size, align);
 		if (!pcpu[i])
 			rv = -1;
 	}
-	for (i = 0; i < 35000; i++)
+	for (i = 0; i < nr_pcpu_objects; i++)
 		free_percpu(pcpu[i]);
 	vfree(pcpu);

mm/Kconfig+31−17

 #
 # support for memory balloon
-config MEMORY_BALLOON
+config BALLOON
 	bool
 #
-# support for memory balloon compaction
+# support for memory balloon page migration
-config BALLOON_COMPACTION
+config BALLOON_MIGRATION
-	bool "Allow for balloon memory compaction/migration"
+	bool "Allow for balloon memory migration"
 	default y
-	depends on COMPACTION && MEMORY_BALLOON
+	depends on MIGRATION && BALLOON
 	help
-	  Memory fragmentation introduced by ballooning might reduce
+	  Allow for migration of pages inflated in a memory balloon such that
-	  significantly the number of 2MB contiguous memory blocks that can be
+	  they can be allocated from memory areas only available for movable
-	  used within a guest, thus imposing performance penalties associated
+	  allocations (e.g., ZONE_MOVABLE, CMA) and such that they can be
-	  with the reduced number of transparent huge pages that could be used
+	  migrated for memory defragmentation purposes by memory compaction.
-	  by the guest workload. Allowing the compaction & migration for memory
-	  pages enlisted as being part of memory balloon devices avoids the
-	  scenario aforementioned and helps improving memory defragmentation.
 #
 # support for memory compaction
 	  The architecture has hardware support for userspace shadow call
           stacks (eg, x86 CET, arm64 GCS or RISC-V Zicfiss).
-config ARCH_SUPPORTS_PT_RECLAIM
+config HAVE_ARCH_TLB_REMOVE_TABLE
 	def_bool n
 config PT_RECLAIM
-	bool "reclaim empty user page table pages"
+	def_bool y
-	default y
+	depends on MMU_GATHER_RCU_TABLE_FREE && !HAVE_ARCH_TLB_REMOVE_TABLE
-	depends on ARCH_SUPPORTS_PT_RECLAIM && MMU && SMP
-	select MMU_GATHER_RCU_TABLE_FREE
 	help
 	  Try to reclaim empty user page table pages in paths other than munmap
 	  and exit_mmap path.
 config FIND_NORMAL_PAGE
 	def_bool n
+config ARCH_HAS_LAZY_MMU_MODE
+	bool
+	help
+	  The architecture uses the lazy MMU mode. This allows changes to
+	  MMU-related architectural state to be deferred until the mode is
+	  exited. See <linux/pgtable.h> for details.
+config LAZY_MMU_MODE_KUNIT_TEST
+	tristate "KUnit tests for the lazy MMU mode" if !KUNIT_ALL_TESTS
+	depends on ARCH_HAS_LAZY_MMU_MODE
+	depends on KUNIT
+	default KUNIT_ALL_TESTS
+	help
+	  Enable this option to check that the lazy MMU mode interface behaves
+	  as expected. Only tests for the generic interface are included (not
+	  architecture-specific behaviours).
+	  If unsure, say N.
 source "mm/damon/Kconfig"
 endmenu

mm/Makefile+2−2

@@ -125,7 +125,7 @@ obj-$(CONFIG_CMA) += cma.o · 124 unmodified lines
125	obj-$(CONFIG_NUMA) += numa.o	125	obj-$(CONFIG_NUMA) += numa.o
126	obj-$(CONFIG_NUMA_MEMBLKS) += numa_memblks.o	126	obj-$(CONFIG_NUMA_MEMBLKS) += numa_memblks.o
127	obj-$(CONFIG_NUMA_EMU) += numa_emulation.o	127	obj-$(CONFIG_NUMA_EMU) += numa_emulation.o
128	obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.o	128	obj-$(CONFIG_BALLOON) += balloon.o
129	obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o	129	obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o
130	obj-$(CONFIG_PAGE_TABLE_CHECK) += page_table_check.o	130	obj-$(CONFIG_PAGE_TABLE_CHECK) += page_table_check.o
131	obj-$(CONFIG_CMA_DEBUGFS) += cma_debug.o	131	obj-$(CONFIG_CMA_DEBUGFS) += cma_debug.o
@@ -149,4 +149,4 @@ obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o · 17 unmodified lines
149	obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o	149	obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
150	obj-$(CONFIG_EXECMEM) += execmem.o	150	obj-$(CONFIG_EXECMEM) += execmem.o
151	obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o	151	obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
152	obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o	152	obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o

mm/backing-dev.c+3−6

 	memcg_cgwb_list->next = NULL;	/* prevent new wb's */
 	spin_unlock_irq(&cgwb_lock);
-	queue_work(system_unbound_wq, &cleanup_offline_cgwbs_work);
+	queue_work(system_dfl_wq, &cleanup_offline_cgwbs_work);
 }
 /**
 {
 	/*
 	 * There can be many concurrent release work items overwhelming
-	 * system_wq.  Put them in a separate wq and limit concurrency.
+	 * system_percpu_wq.  Put them in a separate wq and limit concurrency.
 	 * There's no point in executing many of these in parallel.
 	 */
-	cgwb_release_wq = alloc_workqueue("cgwb_release", 0, 1);
+	cgwb_release_wq = alloc_workqueue("cgwb_release", WQ_PERCPU, 1);
 	if (!cgwb_release_wq)
 		return -ENOMEM;
 	bdi->capabilities = BDI_CAP_WRITEBACK;
 	bdi->ra_pages = VM_READAHEAD_PAGES;
 	bdi->io_pages = VM_READAHEAD_PAGES;
-	timer_setup(&bdi->laptop_mode_wb_timer, laptop_mode_timer_fn, 0);
 	return bdi;
 }
 EXPORT_SYMBOL(bdi_alloc);
 void bdi_unregister(struct backing_dev_info *bdi)
 {
-	timer_delete_sync(&bdi->laptop_mode_wb_timer);
 	/* make sure nobody finds us on the bdi_list anymore */
 	bdi_remove_from_list(bdi);
 	wb_shutdown(&bdi->wb);

mm/balloon.crenamed from mm/balloon_compaction.c+140−59

@@ -1,28 +1,62 @@
1	// SPDX-License-Identifier: GPL-2.0-only	1	// SPDX-License-Identifier: GPL-2.0-only
2	/*	2	/*
3	* mm/balloon_compaction.c	3	* Common interface for implementing a memory balloon, including support
4	*	4	* for migration of pages inflated in a memory balloon.
5	* Common interface for making balloon pages movable by compaction.
6	*	5	*
7	* Copyright (C) 2012, Red Hat, Inc. Rafael Aquini <aquini@redhat.com>	6	* Copyright (C) 2012, Red Hat, Inc. Rafael Aquini <aquini@redhat.com>
8	*/	7	*/
9	#include <linux/mm.h>	8	#include <linux/mm.h>
10	#include <linux/slab.h>	9	#include <linux/slab.h>
11	#include <linux/export.h>	10	#include <linux/export.h>
12	#include <linux/balloon_compaction.h>	11	#include <linux/balloon.h>
		12
		13	/*
		14	* Lock protecting the balloon_dev_info of all devices. We don't really
		15	* expect more than one device.
		16	*/
		17	static DEFINE_SPINLOCK(balloon_pages_lock);
		18
		19	/**
		20	* balloon_page_insert - insert a page into the balloon's page list and make
		21	* the page->private assignment accordingly.
		22	* @balloon : pointer to balloon device
		23	* @page : page to be assigned as a 'balloon page'
		24	*
		25	* Caller must ensure the balloon_pages_lock is held.
		26	*/
		27	static void balloon_page_insert(struct balloon_dev_info *balloon,
		28	struct page *page)
		29	{
		30	lockdep_assert_held(&balloon_pages_lock);
		31	__SetPageOffline(page);
		32	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) {
		33	SetPageMovableOps(page);
		34	set_page_private(page, (unsigned long)balloon);
		35	}
		36	list_add(&page->lru, &balloon->pages);
		37	}
		38
		39	/**
		40	* balloon_page_finalize - prepare a balloon page that was removed from the
		41	* balloon list for release to the page allocator
		42	* @page: page to be released to the page allocator
		43	*
		44	* Caller must ensure the balloon_pages_lock is held.
		45	*/
		46	static void balloon_page_finalize(struct page *page)
		47	{
		48	lockdep_assert_held(&balloon_pages_lock);
		49	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
		50	set_page_private(page, 0);
		51	/* PageOffline is sticky until the page is freed to the buddy. */
		52	}
13		53
14	static void balloon_page_enqueue_one(struct balloon_dev_info *b_dev_info,	54	static void balloon_page_enqueue_one(struct balloon_dev_info *b_dev_info,
15	struct page *page)	55	struct page *page)
16	{	56	{
17	/*
18	* Block others from accessing the 'page' when we get around to
19	* establishing additional references. We should be the only one
20	* holding a reference to the 'page' at this point. If we are not, then
21	* memory corruption is possible and we should stop execution.
22	*/
23	BUG_ON(!trylock_page(page));
24	balloon_page_insert(b_dev_info, page);	57	balloon_page_insert(b_dev_info, page);
25	unlock_page(page);	58	if (b_dev_info->adjust_managed_page_count)
		59	adjust_managed_page_count(page, -1);
26	__count_vm_event(BALLOON_INFLATE);	60	__count_vm_event(BALLOON_INFLATE);
27	inc_node_page_state(page, NR_BALLOON_PAGES);	61	inc_node_page_state(page, NR_BALLOON_PAGES);
28	}	62	}
@@ -45,13 +79,13 @@ size_t balloon_page_list_enqueue(struct balloon_dev_info *b_dev_info, · 16 unmodified lines
45	unsigned long flags;	79	unsigned long flags;
46	size_t n_pages = 0;	80	size_t n_pages = 0;
47		81
48	spin_lock_irqsave(&b_dev_info->pages_lock, flags);	82	spin_lock_irqsave(&balloon_pages_lock, flags);
49	list_for_each_entry_safe(page, tmp, pages, lru) {	83	list_for_each_entry_safe(page, tmp, pages, lru) {
50	list_del(&page->lru);	84	list_del(&page->lru);
51	balloon_page_enqueue_one(b_dev_info, page);	85	balloon_page_enqueue_one(b_dev_info, page);
52	n_pages++;	86	n_pages++;
53	}	87	}
54	spin_unlock_irqrestore(&b_dev_info->pages_lock, flags);	88	spin_unlock_irqrestore(&balloon_pages_lock, flags);
55	return n_pages;	89	return n_pages;
56	}	90	}
57	EXPORT_SYMBOL_GPL(balloon_page_list_enqueue);	91	EXPORT_SYMBOL_GPL(balloon_page_list_enqueue);
@@ -81,34 +115,26 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info, · 23 unmodified lines
81	unsigned long flags;	115	unsigned long flags;
82	size_t n_pages = 0;	116	size_t n_pages = 0;
83		117
84	spin_lock_irqsave(&b_dev_info->pages_lock, flags);	118	spin_lock_irqsave(&balloon_pages_lock, flags);
85	list_for_each_entry_safe(page, tmp, &b_dev_info->pages, lru) {	119	list_for_each_entry_safe(page, tmp, &b_dev_info->pages, lru) {
86	if (n_pages == n_req_pages)	120	if (n_pages == n_req_pages)
87	break;	121	break;
88
89	/*
90	* Block others from accessing the 'page' while we get around to
91	* establishing additional references and preparing the 'page'
92	* to be released by the balloon driver.
93	*/
94	if (!trylock_page(page))
95	continue;
96
97	list_del(&page->lru);	122	list_del(&page->lru);
		123	if (b_dev_info->adjust_managed_page_count)
		124	adjust_managed_page_count(page, 1);
98	balloon_page_finalize(page);	125	balloon_page_finalize(page);
99	__count_vm_event(BALLOON_DEFLATE);	126	__count_vm_event(BALLOON_DEFLATE);
100	list_add(&page->lru, pages);	127	list_add(&page->lru, pages);
101	unlock_page(page);
102	dec_node_page_state(page, NR_BALLOON_PAGES);	128	dec_node_page_state(page, NR_BALLOON_PAGES);
103	n_pages++;	129	n_pages++;
104	}	130	}
105	spin_unlock_irqrestore(&b_dev_info->pages_lock, flags);	131	spin_unlock_irqrestore(&balloon_pages_lock, flags);
106		132
107	return n_pages;	133	return n_pages;
108	}	134	}
109	EXPORT_SYMBOL_GPL(balloon_page_list_dequeue);	135	EXPORT_SYMBOL_GPL(balloon_page_list_dequeue);
110		136
111	/*	137	/**
112	* balloon_page_alloc - allocates a new page for insertion into the balloon	138	* balloon_page_alloc - allocates a new page for insertion into the balloon
113	* page list.	139	* page list.
114	*	140	*
@@ -120,14 +146,18 @@ EXPORT_SYMBOL_GPL(balloon_page_list_dequeue); · 5 unmodified lines
120	*/	146	*/
121	struct page *balloon_page_alloc(void)	147	struct page *balloon_page_alloc(void)
122	{	148	{
123	struct page *page = alloc_page(balloon_mapping_gfp_mask() \|	149	gfp_t gfp_flags = __GFP_NOMEMALLOC \| __GFP_NORETRY \| __GFP_NOWARN;
124	__GFP_NOMEMALLOC \| __GFP_NORETRY \|	150
125	__GFP_NOWARN);	151	if (IS_ENABLED(CONFIG_BALLOON_MIGRATION))
126	return page;	152	gfp_flags \|= GFP_HIGHUSER_MOVABLE;
		153	else
		154	gfp_flags \|= GFP_HIGHUSER;
		155
		156	return alloc_page(gfp_flags);
127	}	157	}
128	EXPORT_SYMBOL_GPL(balloon_page_alloc);	158	EXPORT_SYMBOL_GPL(balloon_page_alloc);
129		159
130	/*	160	/**
131	* balloon_page_enqueue - inserts a new page into the balloon page list.	161	* balloon_page_enqueue - inserts a new page into the balloon page list.
132	*	162	*
133	* @b_dev_info: balloon device descriptor where we will insert a new page	163	* @b_dev_info: balloon device descriptor where we will insert a new page
@@ -136,22 +166,21 @@ EXPORT_SYMBOL_GPL(balloon_page_alloc); · 2 unmodified lines
136	* Drivers must call this function to properly enqueue a new allocated balloon	166	* Drivers must call this function to properly enqueue a new allocated balloon
137	* page before definitively removing the page from the guest system.	167	* page before definitively removing the page from the guest system.
138	*	168	*
139	* Drivers must not call balloon_page_enqueue on pages that have been pushed to	169	* Drivers must not enqueue pages while page->lru is still in
140	* a list with balloon_page_push before removing them with balloon_page_pop. To	170	* use, and must not use page->lru until a page was unqueued again.
141	* enqueue a list of pages, use balloon_page_list_enqueue instead.
142	*/	171	*/
143	void balloon_page_enqueue(struct balloon_dev_info *b_dev_info,	172	void balloon_page_enqueue(struct balloon_dev_info *b_dev_info,
144	struct page *page)	173	struct page *page)
145	{	174	{
146	unsigned long flags;	175	unsigned long flags;
147		176
148	spin_lock_irqsave(&b_dev_info->pages_lock, flags);	177	spin_lock_irqsave(&balloon_pages_lock, flags);
149	balloon_page_enqueue_one(b_dev_info, page);	178	balloon_page_enqueue_one(b_dev_info, page);
150	spin_unlock_irqrestore(&b_dev_info->pages_lock, flags);	179	spin_unlock_irqrestore(&balloon_pages_lock, flags);
151	}	180	}
152	EXPORT_SYMBOL_GPL(balloon_page_enqueue);	181	EXPORT_SYMBOL_GPL(balloon_page_enqueue);
153		182
154	/*	183	/**
155	* balloon_page_dequeue - removes a page from balloon's page list and returns	184	* balloon_page_dequeue - removes a page from balloon's page list and returns
156	* its address to allow the driver to release the page.	185	* its address to allow the driver to release the page.
157	* @b_dev_info: balloon device descriptor where we will grab a page from.	186	* @b_dev_info: balloon device descriptor where we will grab a page from.
@@ -187,32 +216,42 @@ struct page balloon_page_dequeue(struct balloon_dev_info b_dev_info) · 29 unmodified lines
187	* BUG() here, otherwise the balloon driver may get stuck in	216	* BUG() here, otherwise the balloon driver may get stuck in
188	* an infinite loop while attempting to release all its pages.	217	* an infinite loop while attempting to release all its pages.
189	*/	218	*/
190	spin_lock_irqsave(&b_dev_info->pages_lock, flags);	219	spin_lock_irqsave(&balloon_pages_lock, flags);
191	if (unlikely(list_empty(&b_dev_info->pages) &&	220	if (unlikely(list_empty(&b_dev_info->pages) &&
192	!b_dev_info->isolated_pages))	221	!b_dev_info->isolated_pages))
193	BUG();	222	BUG();
194	spin_unlock_irqrestore(&b_dev_info->pages_lock, flags);	223	spin_unlock_irqrestore(&balloon_pages_lock, flags);
195	return NULL;	224	return NULL;
196	}	225	}
197	return list_first_entry(&pages, struct page, lru);	226	return list_first_entry(&pages, struct page, lru);
198	}	227	}
199	EXPORT_SYMBOL_GPL(balloon_page_dequeue);	228	EXPORT_SYMBOL_GPL(balloon_page_dequeue);
200		229
201	#ifdef CONFIG_BALLOON_COMPACTION	230	#ifdef CONFIG_BALLOON_MIGRATION
		231	static struct balloon_dev_info balloon_page_device(struct page page)
		232	{
		233	return (struct balloon_dev_info *)page_private(page);
		234	}
202		235
203	static bool balloon_page_isolate(struct page *page, isolate_mode_t mode)	236	static bool balloon_page_isolate(struct page *page, isolate_mode_t mode)
204		237
205	{	238	{
206	struct balloon_dev_info *b_dev_info = balloon_page_device(page);	239	struct balloon_dev_info *b_dev_info;
207	unsigned long flags;	240	unsigned long flags;
208		241
209	if (!b_dev_info)	242	spin_lock_irqsave(&balloon_pages_lock, flags);
		243	b_dev_info = balloon_page_device(page);
		244	if (!b_dev_info) {
		245	/*
		246	* The page already got deflated and removed from the
		247	* balloon list.
		248	*/
		249	spin_unlock_irqrestore(&balloon_pages_lock, flags);
210	return false;	250	return false;
211		251	}
212	spin_lock_irqsave(&b_dev_info->pages_lock, flags);
213	list_del(&page->lru);	252	list_del(&page->lru);
214	b_dev_info->isolated_pages++;	253	b_dev_info->isolated_pages++;
215	spin_unlock_irqrestore(&b_dev_info->pages_lock, flags);	254	spin_unlock_irqrestore(&balloon_pages_lock, flags);
216		255
217	return true;	256	return true;
218	}	257	}
@@ -222,33 +261,75 @@ static void balloon_page_putback(struct page *page) · 3 unmodified lines
222	struct balloon_dev_info *b_dev_info = balloon_page_device(page);	261	struct balloon_dev_info *b_dev_info = balloon_page_device(page);
223	unsigned long flags;	262	unsigned long flags;
224		263
225	/* Isolated balloon pages cannot get deflated. */	264	/*
		265	* When we isolated the page, the page was still inflated in a balloon
		266	* device. As isolated balloon pages cannot get deflated, we still have
		267	* a balloon device here.
		268	*/
226	if (WARN_ON_ONCE(!b_dev_info))	269	if (WARN_ON_ONCE(!b_dev_info))
227	return;	270	return;
228		271
229	spin_lock_irqsave(&b_dev_info->pages_lock, flags);	272	spin_lock_irqsave(&balloon_pages_lock, flags);
230	list_add(&page->lru, &b_dev_info->pages);	273	list_add(&page->lru, &b_dev_info->pages);
231	b_dev_info->isolated_pages--;	274	b_dev_info->isolated_pages--;
232	spin_unlock_irqrestore(&b_dev_info->pages_lock, flags);	275	spin_unlock_irqrestore(&balloon_pages_lock, flags);
233	}	276	}
234		277
235	/* move_to_new_page() counterpart for a ballooned page */
236	static int balloon_page_migrate(struct page newpage, struct page page,	278	static int balloon_page_migrate(struct page newpage, struct page page,
237	enum migrate_mode mode)	279	enum migrate_mode mode)
238	{	280	{
239	struct balloon_dev_info *balloon = balloon_page_device(page);	281	struct balloon_dev_info *b_dev_info = balloon_page_device(page);
240		282	unsigned long flags;
241	VM_BUG_ON_PAGE(!PageLocked(page), page);	283	int rc;
242	VM_BUG_ON_PAGE(!PageLocked(newpage), newpage);
243		284
244	/* Isolated balloon pages cannot get deflated. */	285	/*
245	if (WARN_ON_ONCE(!balloon))	286	* When we isolated the page, the page was still inflated in a balloon
		287	* device. As isolated balloon pages cannot get deflated, we still have
		288	* a balloon device here.
		289	*/
		290	if (WARN_ON_ONCE(!b_dev_info))
246	return -EAGAIN;	291	return -EAGAIN;
247		292
248	return balloon->migratepage(balloon, newpage, page, mode);	293	rc = b_dev_info->migratepage(b_dev_info, newpage, page, mode);
		294	if (rc < 0 && rc != -ENOENT)
		295	return rc;
		296
		297	spin_lock_irqsave(&balloon_pages_lock, flags);
		298	if (!rc) {
		299	/* Insert the new page into the balloon list. */
		300	get_page(newpage);
		301	balloon_page_insert(b_dev_info, newpage);
		302	__count_vm_event(BALLOON_MIGRATE);
		303
		304	if (b_dev_info->adjust_managed_page_count &&
		305	page_zone(page) != page_zone(newpage)) {
		306	/*
		307	* When we migrate a page to a different zone we
		308	* have to fixup the count of both involved zones.
		309	*/
		310	adjust_managed_page_count(page, 1);
		311	adjust_managed_page_count(newpage, -1);
		312	}
		313	} else {
		314	/* Old page was deflated but new page not inflated. */
		315	__count_vm_event(BALLOON_DEFLATE);
		316
		317	if (b_dev_info->adjust_managed_page_count)
		318	adjust_managed_page_count(page, 1);
		319	}
		320
		321	b_dev_info->isolated_pages--;
		322
		323	/* Free the now-deflated page we isolated in balloon_page_isolate(). */
		324	balloon_page_finalize(page);
		325	spin_unlock_irqrestore(&balloon_pages_lock, flags);
		326
		327	put_page(page);
		328
		329	return 0;
249	}	330	}
250		331
251	const struct movable_operations balloon_mops = {	332	static const struct movable_operations balloon_mops = {
252	.migrate_page = balloon_page_migrate,	333	.migrate_page = balloon_page_migrate,
253	.isolate_page = balloon_page_isolate,	334	.isolate_page = balloon_page_isolate,
254	.putback_page = balloon_page_putback,	335	.putback_page = balloon_page_putback,
@@ -260,4 +341,4 @@ static int __init balloon_init(void) · 5 unmodified lines
260	}	341	}
261	core_initcall(balloon_init);	342	core_initcall(balloon_init);
262		343
263	#endif /* CONFIG_BALLOON_COMPACTION */	344	#endif /* CONFIG_BALLOON_MIGRATION */

mm/cma.c+71−54

@@ -22,6 +22,7 @@ · 21 unmodified lines
22	#include <linux/mm.h>	22	#include <linux/mm.h>
23	#include <linux/sizes.h>	23	#include <linux/sizes.h>
24	#include <linux/slab.h>	24	#include <linux/slab.h>
		25	#include <linux/string.h>
25	#include <linux/string_choices.h>	26	#include <linux/string_choices.h>
26	#include <linux/log2.h>	27	#include <linux/log2.h>
27	#include <linux/cma.h>	28	#include <linux/cma.h>
@@ -233,7 +234,7 @@ static int __init cma_new_area(const char *name, phys_addr_t size, · 205 unmodified lines
233	cma_area_count++;	234	cma_area_count++;
234		235
235	if (name)	236	if (name)
236	snprintf(cma->name, CMA_MAX_NAME, "%s", name);	237	strscpy(cma->name, name);
237	else	238	else
238	snprintf(cma->name, CMA_MAX_NAME, "cma%d\n", cma_area_count);	239	snprintf(cma->name, CMA_MAX_NAME, "cma%d\n", cma_area_count);
239		240
@@ -836,7 +837,7 @@ static int cma_range_alloc(struct cma cma, struct cma_memrange cmr, · 596 unmodified lines
836	spin_unlock_irq(&cma->lock);	837	spin_unlock_irq(&cma->lock);
837		838
838	mutex_lock(&cma->alloc_mutex);	839	mutex_lock(&cma->alloc_mutex);
839	ret = alloc_contig_range(pfn, pfn + count, ACR_FLAGS_CMA, gfp);	840	ret = alloc_contig_frozen_range(pfn, pfn + count, ACR_FLAGS_CMA, gfp);
840	mutex_unlock(&cma->alloc_mutex);	841	mutex_unlock(&cma->alloc_mutex);
841	if (!ret)	842	if (!ret)
842	break;	843	break;
@@ -856,8 +857,8 @@ out: · 13 unmodified lines
856	return ret;	857	return ret;
857	}	858	}
858		859
859	static struct page __cma_alloc(struct cma cma, unsigned long count,	860	static struct page __cma_alloc_frozen(struct cma cma,
860	unsigned int align, gfp_t gfp)	861	unsigned long count, unsigned int align, gfp_t gfp)
861	{	862	{
862	struct page *page = NULL;	863	struct page *page = NULL;
863	int ret = -ENOMEM, r;	864	int ret = -ENOMEM, r;
@@ -914,6 +915,21 @@ static struct page __cma_alloc(struct cma cma, unsigned long count, · 50 unmodified lines
914	return page;	915	return page;
915	}	916	}
916		917
		918	struct page cma_alloc_frozen(struct cma cma, unsigned long count,
		919	unsigned int align, bool no_warn)
		920	{
		921	gfp_t gfp = GFP_KERNEL \| (no_warn ? __GFP_NOWARN : 0);
		922
		923	return __cma_alloc_frozen(cma, count, align, gfp);
		924	}
		925
		926	struct page cma_alloc_frozen_compound(struct cma cma, unsigned int order)
		927	{
		928	gfp_t gfp = GFP_KERNEL \| __GFP_COMP \| __GFP_NOWARN;
		929
		930	return __cma_alloc_frozen(cma, 1 << order, order, gfp);
		931	}
		932
917	/**	933	/**
918	* cma_alloc() - allocate pages from contiguous area	934	* cma_alloc() - allocate pages from contiguous area
919	* @cma: Contiguous memory region for which the allocation is performed.	935	* @cma: Contiguous memory region for which the allocation is performed.
@@ -926,50 +942,61 @@ static struct page __cma_alloc(struct cma cma, unsigned long count, · 6 unmodified lines
926	*/	942	*/
927	struct page cma_alloc(struct cma cma, unsigned long count,	943	struct page cma_alloc(struct cma cma, unsigned long count,
928	unsigned int align, bool no_warn)	944	unsigned int align, bool no_warn)
929	{
930	return __cma_alloc(cma, count, align, GFP_KERNEL \| (no_warn ? __GFP_NOWARN : 0));
931	}
932
933	struct folio cma_alloc_folio(struct cma cma, int order, gfp_t gfp)
934	{	945	{
935	struct page *page;	946	struct page *page;
936		947
937	if (WARN_ON(!order \|\| !(gfp & __GFP_COMP)))	948	page = cma_alloc_frozen(cma, count, align, no_warn);
938	return NULL;	949	if (page)
939		950	set_pages_refcounted(page, count);
940	page = __cma_alloc(cma, 1 << order, order, gfp);
941		951
942	return page ? page_folio(page) : NULL;	952	return page;
943	}	953	}
944		954
945	bool cma_pages_valid(struct cma cma, const struct page pages,	955	static struct cma_memrange find_cma_memrange(struct cma cma,
946	unsigned long count)	956	const struct page *pages, unsigned long count)
947	{	957	{
948	unsigned long pfn, end;	958	struct cma_memrange *cmr = NULL;
		959	unsigned long pfn, end_pfn;
949	int r;	960	int r;
950	struct cma_memrange *cmr;	961
951	bool ret;	962	pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count);
952		963
953	if (!cma \|\| !pages \|\| count > cma->count)	964	if (!cma \|\| !pages \|\| count > cma->count)
954	return false;	965	return NULL;
955		966
956	pfn = page_to_pfn(pages);	967	pfn = page_to_pfn(pages);
957	ret = false;
958		968
959	for (r = 0; r < cma->nranges; r++) {	969	for (r = 0; r < cma->nranges; r++) {
960	cmr = &cma->ranges[r];	970	cmr = &cma->ranges[r];
961	end = cmr->base_pfn + cmr->count;	971	end_pfn = cmr->base_pfn + cmr->count;
962	if (pfn >= cmr->base_pfn && pfn < end) {	972	if (pfn >= cmr->base_pfn && pfn < end_pfn) {
963	ret = pfn + count <= end;	973	if (pfn + count <= end_pfn)
964	break;	974	break;
		975
		976	VM_WARN_ON_ONCE(1);
965	}	977	}
966	}	978	}
967		979
968	if (!ret)	980	if (r == cma->nranges) {
969	pr_debug("%s(page %p, count %lu)\n",	981	pr_debug("%s(page %p, count %lu, no cma range matches the page range)\n",
970	__func__, (void *)pages, count);	982	__func__, (void *)pages, count);
		983	return NULL;
		984	}
971		985
972	return ret;	986	return cmr;
		987	}
		988
		989	static void __cma_release_frozen(struct cma cma, struct cma_memrange cmr,
		990	const struct page *pages, unsigned long count)
		991	{
		992	unsigned long pfn = page_to_pfn(pages);
		993
		994	pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count);
		995
		996	free_contig_frozen_range(pfn, count);
		997	cma_clear_bitmap(cma, cmr, pfn, count);
		998	cma_sysfs_account_release_pages(cma, count);
		999	trace_cma_release(cma->name, pfn, pages, count);
973	}	1000	}
974		1001
975	/**	1002	/**
@@ -986,43 +1013,33 @@ bool cma_release(struct cma cma, const struct page pages, · 10 unmodified lines
986	unsigned long count)	1013	unsigned long count)
987	{	1014	{
988	struct cma_memrange *cmr;	1015	struct cma_memrange *cmr;
989	unsigned long pfn, end_pfn;	1016	unsigned long i, pfn;
990	int r;
991		1017
992	pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count);	1018	cmr = find_cma_memrange(cma, pages, count);
993		1019	if (!cmr)
994	if (!cma_pages_valid(cma, pages, count))
995	return false;	1020	return false;
996		1021
997	pfn = page_to_pfn(pages);	1022	pfn = page_to_pfn(pages);
998	end_pfn = pfn + count;	1023	for (i = 0; i < count; i++, pfn++)
		1024	VM_WARN_ON(!put_page_testzero(pfn_to_page(pfn)));
999		1025
1000	for (r = 0; r < cma->nranges; r++) {	1026	__cma_release_frozen(cma, cmr, pages, count);
1001	cmr = &cma->ranges[r];
1002	if (pfn >= cmr->base_pfn &&
1003	pfn < (cmr->base_pfn + cmr->count)) {
1004	VM_BUG_ON(end_pfn > cmr->base_pfn + cmr->count);
1005	break;
1006	}
1007	}
1008
1009	if (r == cma->nranges)
1010	return false;
1011
1012	free_contig_range(pfn, count);
1013	cma_clear_bitmap(cma, cmr, pfn, count);
1014	cma_sysfs_account_release_pages(cma, count);
1015	trace_cma_release(cma->name, pfn, pages, count);
1016		1027
1017	return true;	1028	return true;
1018	}	1029	}
1019		1030
1020	bool cma_free_folio(struct cma cma, const struct folio folio)	1031	bool cma_release_frozen(struct cma cma, const struct page pages,
		1032	unsigned long count)
1021	{	1033	{
1022	if (WARN_ON(!folio_test_large(folio)))	1034	struct cma_memrange *cmr;
		1035
		1036	cmr = find_cma_memrange(cma, pages, count);
		1037	if (!cmr)
1023	return false;	1038	return false;
1024		1039
1025	return cma_release(cma, &folio->page, folio_nr_pages(folio));	1040	__cma_release_frozen(cma, cmr, pages, count);
		1041
		1042	return true;
1026	}	1043	}
1027		1044
1028	int cma_for_each_area(int (it)(struct cma cma, void data), void data)	1045	int cma_for_each_area(int (it)(struct cma cma, void data), void data)

mm/damon/core.c+142−100

@@ -157,6 +157,12 @@ void damon_destroy_region(struct damon_region r, struct damon_target t) · 156 unmodified lines
157	damon_free_region(r);	157	damon_free_region(r);
158	}	158	}
159		159
		160	static bool damon_is_last_region(struct damon_region *r,
		161	struct damon_target *t)
		162	{
		163	return list_is_last(&r->list, &t->regions_list);
		164	}
		165
160	/*	166	/*
161	* Check whether a region is intersecting an address range	167	* Check whether a region is intersecting an address range
162	*	168	*
@@ -197,7 +203,7 @@ static int damon_fill_regions_holes(struct damon_region *first, · 34 unmodified lines
197	* @t: the given target.	203	* @t: the given target.
198	* @ranges: array of new monitoring target ranges.	204	* @ranges: array of new monitoring target ranges.
199	* @nr_ranges: length of @ranges.	205	* @nr_ranges: length of @ranges.
200	* @min_sz_region: minimum region size.	206	* @min_region_sz: minimum region size.
201	*	207	*
202	* This function adds new regions to, or modify existing regions of a	208	* This function adds new regions to, or modify existing regions of a
203	* monitoring target to fit in specific ranges.	209	* monitoring target to fit in specific ranges.
@@ -205,7 +211,7 @@ static int damon_fill_regions_holes(struct damon_region *first, · 1 unmodified line
205	* Return: 0 if success, or negative error code otherwise.	211	* Return: 0 if success, or negative error code otherwise.
206	*/	212	*/
207	int damon_set_regions(struct damon_target t, struct damon_addr_range ranges,	213	int damon_set_regions(struct damon_target t, struct damon_addr_range ranges,
208	unsigned int nr_ranges, unsigned long min_sz_region)	214	unsigned int nr_ranges, unsigned long min_region_sz)
209	{	215	{
210	struct damon_region r, next;	216	struct damon_region r, next;
211	unsigned int i;	217	unsigned int i;
@@ -242,16 +248,16 @@ int damon_set_regions(struct damon_target t, struct damon_addr_range ranges, · 30 unmodified lines
242	/* no region intersects with this range */	248	/* no region intersects with this range */
243	newr = damon_new_region(	249	newr = damon_new_region(
244	ALIGN_DOWN(range->start,	250	ALIGN_DOWN(range->start,
245	min_sz_region),	251	min_region_sz),
246	ALIGN(range->end, min_sz_region));	252	ALIGN(range->end, min_region_sz));
247	if (!newr)	253	if (!newr)
248	return -ENOMEM;	254	return -ENOMEM;
249	damon_insert_region(newr, damon_prev_region(r), r, t);	255	damon_insert_region(newr, damon_prev_region(r), r, t);
250	} else {	256	} else {
251	/* resize intersecting regions to fit in this range */	257	/* resize intersecting regions to fit in this range */
252	first->ar.start = ALIGN_DOWN(range->start,	258	first->ar.start = ALIGN_DOWN(range->start,
253	min_sz_region);	259	min_region_sz);
254	last->ar.end = ALIGN(range->end, min_sz_region);	260	last->ar.end = ALIGN(range->end, min_region_sz);
255		261
256	/* fill possible holes in the range */	262	/* fill possible holes in the range */
257	err = damon_fill_regions_holes(first, last, t);	263	err = damon_fill_regions_holes(first, last, t);
@@ -278,7 +284,7 @@ struct damos_filter *damos_new_filter(enum damos_filter_type type, · 20 unmodified lines
278	}	284	}
279		285
280	/**	286	/**
281	* damos_filter_for_ops() - Return if the filter is ops-hndled one.	287	* damos_filter_for_ops() - Return if the filter is ops-handled one.
282	* @type: type of the filter.	288	* @type: type of the filter.
283	*	289	*
284	* Return: true if the filter of @type needs to be handled by ops layer, false	290	* Return: true if the filter of @type needs to be handled by ops layer, false
@@ -395,6 +401,7 @@ struct damos damon_new_scheme(struct damos_access_pattern pattern, · 110 unmodified lines
395	INIT_LIST_HEAD(&scheme->core_filters);	401	INIT_LIST_HEAD(&scheme->core_filters);
396	INIT_LIST_HEAD(&scheme->ops_filters);	402	INIT_LIST_HEAD(&scheme->ops_filters);
397	scheme->stat = (struct damos_stat){};	403	scheme->stat = (struct damos_stat){};
		404	scheme->max_nr_snapshots = 0;
398	INIT_LIST_HEAD(&scheme->list);	405	INIT_LIST_HEAD(&scheme->list);
399		406
400	scheme->quota = *(damos_quota_init(quota));	407	scheme->quota = *(damos_quota_init(quota));
@@ -546,7 +553,7 @@ struct damon_ctx *damon_new_ctx(void) · 145 unmodified lines
546	ctx->attrs.max_nr_regions = 1000;	553	ctx->attrs.max_nr_regions = 1000;
547		554
548	ctx->addr_unit = 1;	555	ctx->addr_unit = 1;
549	ctx->min_sz_region = DAMON_MIN_REGION;	556	ctx->min_region_sz = DAMON_MIN_REGION_SZ;
550		557
551	INIT_LIST_HEAD(&ctx->adaptive_targets);	558	INIT_LIST_HEAD(&ctx->adaptive_targets);
552	INIT_LIST_HEAD(&ctx->schemes);	559	INIT_LIST_HEAD(&ctx->schemes);
@@ -1072,7 +1079,11 @@ static int damos_commit(struct damos dst, struct damos src) · 519 unmodified lines
1072	return err;	1079	return err;
1073		1080
1074	err = damos_commit_filters(dst, src);	1081	err = damos_commit_filters(dst, src);
1075	return err;	1082	if (err)
		1083	return err;
		1084
		1085	dst->max_nr_snapshots = src->max_nr_snapshots;
		1086	return 0;
1076	}	1087	}
1077		1088
1078	static int damon_commit_schemes(struct damon_ctx dst, struct damon_ctx src)	1089	static int damon_commit_schemes(struct damon_ctx dst, struct damon_ctx src)
@@ -1131,7 +1142,7 @@ static struct damon_target damon_nth_target(int n, struct damon_ctx ctx) · 52 unmodified lines
1131	* If @src has no region, @dst keeps current regions.	1142	* If @src has no region, @dst keeps current regions.
1132	*/	1143	*/
1133	static int damon_commit_target_regions(struct damon_target *dst,	1144	static int damon_commit_target_regions(struct damon_target *dst,
1134	struct damon_target *src, unsigned long src_min_sz_region)	1145	struct damon_target *src, unsigned long src_min_region_sz)
1135	{	1146	{
1136	struct damon_region *src_region;	1147	struct damon_region *src_region;
1137	struct damon_addr_range *ranges;	1148	struct damon_addr_range *ranges;
@@ -1148,7 +1159,7 @@ static int damon_commit_target_regions(struct damon_target *dst, · 10 unmodified lines
1148	i = 0;	1159	i = 0;
1149	damon_for_each_region(src_region, src)	1160	damon_for_each_region(src_region, src)
1150	ranges[i++] = src_region->ar;	1161	ranges[i++] = src_region->ar;
1151	err = damon_set_regions(dst, ranges, i, src_min_sz_region);	1162	err = damon_set_regions(dst, ranges, i, src_min_region_sz);
1152	kfree(ranges);	1163	kfree(ranges);
1153	return err;	1164	return err;
1154	}	1165	}
@@ -1156,11 +1167,11 @@ static int damon_commit_target_regions(struct damon_target *dst, · 1 unmodified line
1156	static int damon_commit_target(	1167	static int damon_commit_target(
1157	struct damon_target *dst, bool dst_has_pid,	1168	struct damon_target *dst, bool dst_has_pid,
1158	struct damon_target *src, bool src_has_pid,	1169	struct damon_target *src, bool src_has_pid,
1159	unsigned long src_min_sz_region)	1170	unsigned long src_min_region_sz)
1160	{	1171	{
1161	int err;	1172	int err;
1162		1173
1163	err = damon_commit_target_regions(dst, src, src_min_sz_region);	1174	err = damon_commit_target_regions(dst, src, src_min_region_sz);
1164	if (err)	1175	if (err)
1165	return err;	1176	return err;
1166	if (dst_has_pid)	1177	if (dst_has_pid)
@@ -1187,7 +1198,7 @@ static int damon_commit_targets( · 20 unmodified lines
1187	err = damon_commit_target(	1198	err = damon_commit_target(
1188	dst_target, damon_target_has_pid(dst),	1199	dst_target, damon_target_has_pid(dst),
1189	src_target, damon_target_has_pid(src),	1200	src_target, damon_target_has_pid(src),
1190	src->min_sz_region);	1201	src->min_region_sz);
1191	if (err)	1202	if (err)
1192	return err;	1203	return err;
1193	} else {	1204	} else {
@@ -1214,7 +1225,7 @@ static int damon_commit_targets( · 20 unmodified lines
1214	return -ENOMEM;	1225	return -ENOMEM;
1215	err = damon_commit_target(new_target, false,	1226	err = damon_commit_target(new_target, false,
1216	src_target, damon_target_has_pid(src),	1227	src_target, damon_target_has_pid(src),
1217	src->min_sz_region);	1228	src->min_region_sz);
1218	if (err) {	1229	if (err) {
1219	damon_destroy_target(new_target, NULL);	1230	damon_destroy_target(new_target, NULL);
1220	return err;	1231	return err;
@@ -1261,7 +1272,7 @@ int damon_commit_ctx(struct damon_ctx dst, struct damon_ctx src) · 40 unmodified lines
1261	}	1272	}
1262	dst->ops = src->ops;	1273	dst->ops = src->ops;
1263	dst->addr_unit = src->addr_unit;	1274	dst->addr_unit = src->addr_unit;
1264	dst->min_sz_region = src->min_sz_region;	1275	dst->min_region_sz = src->min_region_sz;
1265		1276
1266	return 0;	1277	return 0;
1267	}	1278	}
@@ -1294,8 +1305,8 @@ static unsigned long damon_region_sz_limit(struct damon_ctx *ctx) · 26 unmodified lines
1294		1305
1295	if (ctx->attrs.min_nr_regions)	1306	if (ctx->attrs.min_nr_regions)
1296	sz /= ctx->attrs.min_nr_regions;	1307	sz /= ctx->attrs.min_nr_regions;
1297	if (sz < ctx->min_sz_region)	1308	if (sz < ctx->min_region_sz)
1298	sz = ctx->min_sz_region;	1309	sz = ctx->min_region_sz;
1299		1310
1300	return sz;	1311	return sz;
1301	}	1312	}
@@ -1431,6 +1442,23 @@ bool damon_is_running(struct damon_ctx *ctx) · 129 unmodified lines
1431	return running;	1442	return running;
1432	}	1443	}
1433		1444
		1445	/**
		1446	* damon_kdamond_pid() - Return pid of a given DAMON context's worker thread.
		1447	* @ctx: The DAMON context of the question.
		1448	*
		1449	* Return: pid if @ctx is running, negative error code otherwise.
		1450	*/
		1451	int damon_kdamond_pid(struct damon_ctx *ctx)
		1452	{
		1453	int pid = -EINVAL;
		1454
		1455	mutex_lock(&ctx->kdamond_lock);
		1456	if (ctx->kdamond)
		1457	pid = ctx->kdamond->pid;
		1458	mutex_unlock(&ctx->kdamond_lock);
		1459	return pid;
		1460	}
		1461
1434	/*	1462	/*
1435	* damon_call_handle_inactive_ctx() - handle DAMON call request that added to	1463	* damon_call_handle_inactive_ctx() - handle DAMON call request that added to
1436	* an inactive context.	1464	* an inactive context.
@@ -1604,7 +1632,7 @@ static unsigned long damon_get_intervals_adaptation_bp(struct damon_ctx *c) · 167 unmodified lines
1604	adaptation_bp = damon_feed_loop_next_input(100000000, score_bp) /	1632	adaptation_bp = damon_feed_loop_next_input(100000000, score_bp) /
1605	10000;	1633	10000;
1606	/*	1634	/*
1607	* adaptaion_bp ranges from 1 to 20,000. Avoid too rapid reduction of	1635	* adaptation_bp ranges from 1 to 20,000. Avoid too rapid reduction of
1608	* the intervals by rescaling [1,10,000] to [5000, 10,000].	1636	* the intervals by rescaling [1,10,000] to [5000, 10,000].
1609	*/	1637	*/
1610	if (adaptation_bp <= 10000)	1638	if (adaptation_bp <= 10000)
@@ -1668,7 +1696,7 @@ static bool damos_valid_target(struct damon_ctx c, struct damon_target t, · 57 unmodified lines
1668	* @t: The target of the region.	1696	* @t: The target of the region.
1669	* @rp: The pointer to the region.	1697	* @rp: The pointer to the region.
1670	* @s: The scheme to be applied.	1698	* @s: The scheme to be applied.
1671	* @min_sz_region: minimum region size.	1699	* @min_region_sz: minimum region size.
1672	*	1700	*
1673	* If a quota of a scheme has exceeded in a quota charge window, the scheme's	1701	* If a quota of a scheme has exceeded in a quota charge window, the scheme's
1674	* action would applied to only a part of the target access pattern fulfilling	1702	* action would applied to only a part of the target access pattern fulfilling
@@ -1686,7 +1714,8 @@ static bool damos_valid_target(struct damon_ctx c, struct damon_target t, · 11 unmodified lines
1686	* Return: true if the region should be entirely skipped, false otherwise.	1714	* Return: true if the region should be entirely skipped, false otherwise.
1687	*/	1715	*/
1688	static bool damos_skip_charged_region(struct damon_target *t,	1716	static bool damos_skip_charged_region(struct damon_target *t,
1689	struct damon_region *rp, struct damos s, unsigned long min_sz_region)	1717	struct damon_region *rp, struct damos s,
		1718	unsigned long min_region_sz)
1690	{	1719	{
1691	struct damon_region r = rp;	1720	struct damon_region r = rp;
1692	struct damos_quota *quota = &s->quota;	1721	struct damos_quota *quota = &s->quota;
@@ -1708,11 +1737,11 @@ static bool damos_skip_charged_region(struct damon_target *t, · 15 unmodified lines
1708	if (quota->charge_addr_from && r->ar.start <	1737	if (quota->charge_addr_from && r->ar.start <
1709	quota->charge_addr_from) {	1738	quota->charge_addr_from) {
1710	sz_to_skip = ALIGN_DOWN(quota->charge_addr_from -	1739	sz_to_skip = ALIGN_DOWN(quota->charge_addr_from -
1711	r->ar.start, min_sz_region);	1740	r->ar.start, min_region_sz);
1712	if (!sz_to_skip) {	1741	if (!sz_to_skip) {
1713	if (damon_sz_region(r) <= min_sz_region)	1742	if (damon_sz_region(r) <= min_region_sz)
1714	return true;	1743	return true;
1715	sz_to_skip = min_sz_region;	1744	sz_to_skip = min_region_sz;
1716	}	1745	}
1717	damon_split_region_at(t, r, sz_to_skip);	1746	damon_split_region_at(t, r, sz_to_skip);
1718	r = damon_next_region(r);	1747	r = damon_next_region(r);
@@ -1738,7 +1767,7 @@ static void damos_update_stat(struct damos *s, · 19 unmodified lines
1738		1767
1739	static bool damos_filter_match(struct damon_ctx ctx, struct damon_target t,	1768	static bool damos_filter_match(struct damon_ctx ctx, struct damon_target t,
1740	struct damon_region r, struct damos_filter filter,	1769	struct damon_region r, struct damos_filter filter,
1741	unsigned long min_sz_region)	1770	unsigned long min_region_sz)
1742	{	1771	{
1743	bool matched = false;	1772	bool matched = false;
1744	struct damon_target *ti;	1773	struct damon_target *ti;
@@ -1755,8 +1784,8 @@ static bool damos_filter_match(struct damon_ctx ctx, struct damon_target t, · 10 unmodified lines
1755	matched = target_idx == filter->target_idx;	1784	matched = target_idx == filter->target_idx;
1756	break;	1785	break;
1757	case DAMOS_FILTER_TYPE_ADDR:	1786	case DAMOS_FILTER_TYPE_ADDR:
1758	start = ALIGN_DOWN(filter->addr_range.start, min_sz_region);	1787	start = ALIGN_DOWN(filter->addr_range.start, min_region_sz);
1759	end = ALIGN_DOWN(filter->addr_range.end, min_sz_region);	1788	end = ALIGN_DOWN(filter->addr_range.end, min_region_sz);
1760		1789
1761	/* inside the range */	1790	/* inside the range */
1762	if (start <= r->ar.start && r->ar.end <= end) {	1791	if (start <= r->ar.start && r->ar.end <= end) {
@@ -1785,14 +1814,14 @@ static bool damos_filter_match(struct damon_ctx ctx, struct damon_target t, · 22 unmodified lines
1785	return matched == filter->matching;	1814	return matched == filter->matching;
1786	}	1815	}
1787		1816
1788	static bool damos_filter_out(struct damon_ctx ctx, struct damon_target t,	1817	static bool damos_core_filter_out(struct damon_ctx ctx, struct damon_target t,
1789	struct damon_region r, struct damos s)	1818	struct damon_region r, struct damos s)
1790	{	1819	{
1791	struct damos_filter *filter;	1820	struct damos_filter *filter;
1792		1821
1793	s->core_filters_allowed = false;	1822	s->core_filters_allowed = false;
1794	damos_for_each_core_filter(filter, s) {	1823	damos_for_each_core_filter(filter, s) {
1795	if (damos_filter_match(ctx, t, r, filter, ctx->min_sz_region)) {	1824	if (damos_filter_match(ctx, t, r, filter, ctx->min_region_sz)) {
1796	if (filter->allow)	1825	if (filter->allow)
1797	s->core_filters_allowed = true;	1826	s->core_filters_allowed = true;
1798	return !filter->allow;	1827	return !filter->allow;
@@ -1927,12 +1956,12 @@ static void damos_apply_scheme(struct damon_ctx c, struct damon_target t, · 128 unmodified lines
1927	if (c->ops.apply_scheme) {	1956	if (c->ops.apply_scheme) {
1928	if (quota->esz && quota->charged_sz + sz > quota->esz) {	1957	if (quota->esz && quota->charged_sz + sz > quota->esz) {
1929	sz = ALIGN_DOWN(quota->esz - quota->charged_sz,	1958	sz = ALIGN_DOWN(quota->esz - quota->charged_sz,
1930	c->min_sz_region);	1959	c->min_region_sz);
1931	if (!sz)	1960	if (!sz)
1932	goto update_stat;	1961	goto update_stat;
1933	damon_split_region_at(t, r, sz);	1962	damon_split_region_at(t, r, sz);
1934	}	1963	}
1935	if (damos_filter_out(c, t, r, s))	1964	if (damos_core_filter_out(c, t, r, s))
1936	return;	1965	return;
1937	ktime_get_coarse_ts64(&begin);	1966	ktime_get_coarse_ts64(&begin);
1938	trace_damos_before_apply(cidx, sidx, tidx, r,	1967	trace_damos_before_apply(cidx, sidx, tidx, r,
@@ -1975,13 +2004,18 @@ static void damon_do_apply_schemes(struct damon_ctx *c, · 36 unmodified lines
1975	if (quota->esz && quota->charged_sz >= quota->esz)	2004	if (quota->esz && quota->charged_sz >= quota->esz)
1976	continue;	2005	continue;
1977		2006
1978	if (damos_skip_charged_region(t, &r, s, c->min_sz_region))	2007	if (damos_skip_charged_region(t, &r, s, c->min_region_sz))
1979	continue;	2008	continue;
1980		2009
1981	if (!damos_valid_target(c, t, r, s))	2010	if (s->max_nr_snapshots &&
		2011	s->max_nr_snapshots <= s->stat.nr_snapshots)
1982	continue;	2012	continue;
1983		2013
1984	damos_apply_scheme(c, t, r, s);	2014	if (damos_valid_target(c, t, r, s))
		2015	damos_apply_scheme(c, t, r, s);
		2016
		2017	if (damon_is_last_region(r, t))
		2018	s->stat.nr_snapshots++;
1985	}	2019	}
1986	}	2020	}
1987		2021
@@ -2078,16 +2112,13 @@ static unsigned long damos_get_node_memcg_used_bp( · 90 unmodified lines
2078	unsigned long used_pages, numerator;	2112	unsigned long used_pages, numerator;
2079	struct sysinfo i;	2113	struct sysinfo i;
2080		2114
2081	rcu_read_lock();	2115	memcg = mem_cgroup_get_from_id(goal->memcg_id);
2082	memcg = mem_cgroup_from_id(goal->memcg_id);	2116	if (!memcg) {
2083	if (!memcg \|\| !mem_cgroup_tryget(memcg)) {
2084	rcu_read_unlock();
2085	if (goal->metric == DAMOS_QUOTA_NODE_MEMCG_USED_BP)	2117	if (goal->metric == DAMOS_QUOTA_NODE_MEMCG_USED_BP)
2086	return 0;	2118	return 0;
2087	else /* DAMOS_QUOTA_NODE_MEMCG_FREE_BP */	2119	else /* DAMOS_QUOTA_NODE_MEMCG_FREE_BP */
2088	return 10000;	2120	return 10000;
2089	}	2121	}
2090	rcu_read_unlock();
2091		2122
2092	mem_cgroup_flush_stats(memcg);	2123	mem_cgroup_flush_stats(memcg);
2093	lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(goal->nid));	2124	lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(goal->nid));
@@ -2119,6 +2150,23 @@ static unsigned long damos_get_node_memcg_used_bp( · 25 unmodified lines
2119	}	2150	}
2120	#endif	2151	#endif
2121		2152
		2153	/*
		2154	* Returns LRU-active or inactive memory to total LRU memory size ratio.
		2155	*/
		2156	static unsigned int damos_get_in_active_mem_bp(bool active_ratio)
		2157	{
		2158	unsigned long active, inactive, total;
		2159
		2160	/* This should align with /proc/meminfo output */
		2161	active = global_node_page_state(NR_LRU_BASE + LRU_ACTIVE_ANON) +
		2162	global_node_page_state(NR_LRU_BASE + LRU_ACTIVE_FILE);
		2163	inactive = global_node_page_state(NR_LRU_BASE + LRU_INACTIVE_ANON) +
		2164	global_node_page_state(NR_LRU_BASE + LRU_INACTIVE_FILE);
		2165	total = active + inactive;
		2166	if (active_ratio)
		2167	return active * 10000 / total;
		2168	return inactive * 10000 / total;
		2169	}
2122		2170
2123	static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal)	2171	static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal)
2124	{	2172	{
@@ -2141,6 +2189,11 @@ static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal) · 16 unmodified lines
2141	case DAMOS_QUOTA_NODE_MEMCG_FREE_BP:	2189	case DAMOS_QUOTA_NODE_MEMCG_FREE_BP:
2142	goal->current_value = damos_get_node_memcg_used_bp(goal);	2190	goal->current_value = damos_get_node_memcg_used_bp(goal);
2143	break;	2191	break;
		2192	case DAMOS_QUOTA_ACTIVE_MEM_BP:
		2193	case DAMOS_QUOTA_INACTIVE_MEM_BP:
		2194	goal->current_value = damos_get_in_active_mem_bp(
		2195	goal->metric == DAMOS_QUOTA_ACTIVE_MEM_BP);
		2196	break;
2144	default:	2197	default:
2145	break;	2198	break;
2146	}	2199	}
@@ -2273,6 +2326,22 @@ static void damos_adjust_quota(struct damon_ctx c, struct damos s) · 126 unmodified lines
2273	quota->min_score = score;	2326	quota->min_score = score;
2274	}	2327	}
2275		2328
		2329	static void damos_trace_stat(struct damon_ctx c, struct damos s)
		2330	{
		2331	unsigned int cidx = 0, sidx = 0;
		2332	struct damos *siter;
		2333
		2334	if (!trace_damos_stat_after_apply_interval_enabled())
		2335	return;
		2336
		2337	damon_for_each_scheme(siter, c) {
		2338	if (siter == s)
		2339	break;
		2340	sidx++;
		2341	}
		2342	trace_damos_stat_after_apply_interval(cidx, sidx, &s->stat);
		2343	}
		2344
2276	static void kdamond_apply_schemes(struct damon_ctx *c)	2345	static void kdamond_apply_schemes(struct damon_ctx *c)
2277	{	2346	{
2278	struct damon_target *t;	2347	struct damon_target *t;
@@ -2299,6 +2368,9 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 20 unmodified lines
2299		2368
2300	mutex_lock(&c->walk_control_lock);	2369	mutex_lock(&c->walk_control_lock);
2301	damon_for_each_target(t, c) {	2370	damon_for_each_target(t, c) {
		2371	if (c->ops.target_valid && c->ops.target_valid(t) == false)
		2372	continue;
		2373
2302	damon_for_each_region_safe(r, next_r, t)	2374	damon_for_each_region_safe(r, next_r, t)
2303	damon_do_apply_schemes(c, t, r);	2375	damon_do_apply_schemes(c, t, r);
2304	}	2376	}
@@ -2311,6 +2383,7 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 6 unmodified lines
2311	(s->apply_interval_us ? s->apply_interval_us :	2383	(s->apply_interval_us ? s->apply_interval_us :
2312	c->attrs.aggr_interval) / sample_interval;	2384	c->attrs.aggr_interval) / sample_interval;
2313	s->last_applied = NULL;	2385	s->last_applied = NULL;
		2386	damos_trace_stat(c, s);
2314	}	2387	}
2315	mutex_unlock(&c->walk_control_lock);	2388	mutex_unlock(&c->walk_control_lock);
2316	}	2389	}
@@ -2424,7 +2497,7 @@ static void damon_split_region_at(struct damon_target *t, · 107 unmodified lines
2424		2497
2425	/* Split every region in the given target into 'nr_subs' regions */	2498	/* Split every region in the given target into 'nr_subs' regions */
2426	static void damon_split_regions_of(struct damon_target *t, int nr_subs,	2499	static void damon_split_regions_of(struct damon_target *t, int nr_subs,
2427	unsigned long min_sz_region)	2500	unsigned long min_region_sz)
2428	{	2501	{
2429	struct damon_region r, next;	2502	struct damon_region r, next;
2430	unsigned long sz_region, sz_sub = 0;	2503	unsigned long sz_region, sz_sub = 0;
@@ -2434,13 +2507,13 @@ static void damon_split_regions_of(struct damon_target *t, int nr_subs, · 3 unmodified lines
2434	sz_region = damon_sz_region(r);	2507	sz_region = damon_sz_region(r);
2435		2508
2436	for (i = 0; i < nr_subs - 1 &&	2509	for (i = 0; i < nr_subs - 1 &&
2437	sz_region > 2 * min_sz_region; i++) {	2510	sz_region > 2 * min_region_sz; i++) {
2438	/*	2511	/*
2439	* Randomly select size of left sub-region to be at	2512	* Randomly select size of left sub-region to be at
2440	* least 10 percent and at most 90% of original region	2513	* least 10 percent and at most 90% of original region
2441	*/	2514	*/
2442	sz_sub = ALIGN_DOWN(damon_rand(1, 10) *	2515	sz_sub = ALIGN_DOWN(damon_rand(1, 10) *
2443	sz_region / 10, min_sz_region);	2516	sz_region / 10, min_region_sz);
2444	/* Do not allow blank region */	2517	/* Do not allow blank region */
2445	if (sz_sub == 0 \|\| sz_sub >= sz_region)	2518	if (sz_sub == 0 \|\| sz_sub >= sz_region)
2446	continue;	2519	continue;
@@ -2480,7 +2553,7 @@ static void kdamond_split_regions(struct damon_ctx *ctx) · 33 unmodified lines
2480	nr_subregions = 3;	2553	nr_subregions = 3;
2481		2554
2482	damon_for_each_target(t, ctx)	2555	damon_for_each_target(t, ctx)
2483	damon_split_regions_of(t, nr_subregions, ctx->min_sz_region);	2556	damon_split_regions_of(t, nr_subregions, ctx->min_region_sz);
2484		2557
2485	last_nr_regions = nr_regions;	2558	last_nr_regions = nr_regions;
2486	}	2559	}
@@ -2577,41 +2650,30 @@ static void kdamond_usleep(unsigned long usecs) · 90 unmodified lines
2577	*/	2650	*/
2578	static void kdamond_call(struct damon_ctx *ctx, bool cancel)	2651	static void kdamond_call(struct damon_ctx *ctx, bool cancel)
2579	{	2652	{
2580	struct damon_call_control *control;	2653	struct damon_call_control control, next;
2581	LIST_HEAD(repeat_controls);	2654	LIST_HEAD(controls);
2582	int ret = 0;
2583		2655
2584	while (true) {	2656	mutex_lock(&ctx->call_controls_lock);
2585	mutex_lock(&ctx->call_controls_lock);	2657	list_splice_tail_init(&ctx->call_controls, &controls);
2586	control = list_first_entry_or_null(&ctx->call_controls,	2658	mutex_unlock(&ctx->call_controls_lock);
2587	struct damon_call_control, list);	2659
2588	mutex_unlock(&ctx->call_controls_lock);	2660	list_for_each_entry_safe(control, next, &controls, list) {
2589	if (!control)	2661	if (!control->repeat \|\| cancel)
2590	break;	2662	list_del(&control->list);
2591	if (cancel) {	2663
		2664	if (cancel)
2592	control->canceled = true;	2665	control->canceled = true;
2593	} else {	2666	else
2594	ret = control->fn(control->data);	2667	control->return_code = control->fn(control->data);
2595	control->return_code = ret;	2668
2596	}	2669	if (!control->repeat)
2597	mutex_lock(&ctx->call_controls_lock);
2598	list_del(&control->list);
2599	mutex_unlock(&ctx->call_controls_lock);
2600	if (!control->repeat) {
2601	complete(&control->completion);	2670	complete(&control->completion);
2602	} else if (control->canceled && control->dealloc_on_cancel) {	2671	else if (control->canceled && control->dealloc_on_cancel)
2603	kfree(control);	2672	kfree(control);
2604	continue;
2605	} else {
2606	list_add(&control->list, &repeat_controls);
2607	}
2608	}	2673	}
2609	control = list_first_entry_or_null(&repeat_controls,	2674
2610	struct damon_call_control, list);
2611	if (!control \|\| cancel)
2612	return;
2613	mutex_lock(&ctx->call_controls_lock);	2675	mutex_lock(&ctx->call_controls_lock);
2614	list_add_tail(&control->list, &ctx->call_controls);	2676	list_splice_tail(&controls, &ctx->call_controls);
2615	mutex_unlock(&ctx->call_controls_lock);	2677	mutex_unlock(&ctx->call_controls_lock);
2616	}	2678	}
2617		2679
@@ -2670,8 +2732,6 @@ static void kdamond_init_ctx(struct damon_ctx *ctx) · 52 unmodified lines
2670	static int kdamond_fn(void *data)	2732	static int kdamond_fn(void *data)
2671	{	2733	{
2672	struct damon_ctx *ctx = data;	2734	struct damon_ctx *ctx = data;
2673	struct damon_target *t;
2674	struct damon_region r, next;
2675	unsigned int max_nr_accesses = 0;	2735	unsigned int max_nr_accesses = 0;
2676	unsigned long sz_limit = 0;	2736	unsigned long sz_limit = 0;
2677		2737
@@ -2747,7 +2807,7 @@ static int kdamond_fn(void *data) · 69 unmodified lines
2747	*	2807	*
2748	* Reset ->next_aggregation_sis to avoid that.	2808	* Reset ->next_aggregation_sis to avoid that.
2749	* It will anyway correctly updated after this	2809	* It will anyway correctly updated after this
2750	* if caluse.	2810	* if clause.
2751	*/	2811	*/
2752	ctx->next_aggregation_sis =	2812	ctx->next_aggregation_sis =
2753	next_aggregation_sis;	2813	next_aggregation_sis;
@@ -2776,47 +2836,29 @@ static int kdamond_fn(void *data) · 22 unmodified lines
2776	}	2836	}
2777	}	2837	}
2778	done:	2838	done:
2779	damon_for_each_target(t, ctx) {	2839	damon_destroy_targets(ctx);
2780	damon_for_each_region_safe(r, next, t)
2781	damon_destroy_region(r, t);
2782	}
2783		2840
2784	if (ctx->ops.cleanup)
2785	ctx->ops.cleanup(ctx);
2786	kfree(ctx->regions_score_histogram);	2841	kfree(ctx->regions_score_histogram);
2787	kdamond_call(ctx, true);	2842	kdamond_call(ctx, true);
		2843	damos_walk_cancel(ctx);
2788		2844
2789	pr_debug("kdamond (%d) finishes\n", current->pid);	2845	pr_debug("kdamond (%d) finishes\n", current->pid);
2790	mutex_lock(&ctx->kdamond_lock);	2846	mutex_lock(&ctx->kdamond_lock);
2791	ctx->kdamond = NULL;	2847	ctx->kdamond = NULL;
2792	mutex_unlock(&ctx->kdamond_lock);	2848	mutex_unlock(&ctx->kdamond_lock);
2793		2849
2794	damos_walk_cancel(ctx);
2795
2796	mutex_lock(&damon_lock);	2850	mutex_lock(&damon_lock);
2797	nr_running_ctxs--;	2851	nr_running_ctxs--;
2798	if (!nr_running_ctxs && running_exclusive_ctxs)	2852	if (!nr_running_ctxs && running_exclusive_ctxs)
2799	running_exclusive_ctxs = false;	2853	running_exclusive_ctxs = false;
2800	mutex_unlock(&damon_lock);	2854	mutex_unlock(&damon_lock);
2801		2855
2802	damon_destroy_targets(ctx);
2803	return 0;	2856	return 0;
2804	}	2857	}
2805		2858
2806	/*
2807	* struct damon_system_ram_region - System RAM resource address region of
2808	* [@start, @end).
2809	* @start: Start address of the region (inclusive).
2810	* @end: End address of the region (exclusive).
2811	*/
2812	struct damon_system_ram_region {
2813	unsigned long start;
2814	unsigned long end;
2815	};
2816
2817	static int walk_system_ram(struct resource res, void arg)	2859	static int walk_system_ram(struct resource res, void arg)
2818	{	2860	{
2819	struct damon_system_ram_region *a = arg;	2861	struct damon_addr_range *a = arg;
2820		2862
2821	if (a->end - a->start < resource_size(res)) {	2863	if (a->end - a->start < resource_size(res)) {
2822	a->start = res->start;	2864	a->start = res->start;
@@ -2833,7 +2875,7 @@ static bool damon_find_biggest_system_ram(unsigned long *start, · 10 unmodified lines
2833	unsigned long *end)	2875	unsigned long *end)
2834		2876
2835	{	2877	{
2836	struct damon_system_ram_region arg = {};	2878	struct damon_addr_range arg = {};
2837		2879
2838	walk_system_ram_res(0, ULONG_MAX, &arg, walk_system_ram);	2880	walk_system_ram_res(0, ULONG_MAX, &arg, walk_system_ram);
2839	if (arg.end <= arg.start)	2881	if (arg.end <= arg.start)
@@ -2850,7 +2892,7 @@ static bool damon_find_biggest_system_ram(unsigned long *start, · 10 unmodified lines
2850	* @t: The monitoring target to set the region.	2892	* @t: The monitoring target to set the region.
2851	* @start: The pointer to the start address of the region.	2893	* @start: The pointer to the start address of the region.
2852	* @end: The pointer to the end address of the region.	2894	* @end: The pointer to the end address of the region.
2853	* @min_sz_region: Minimum region size.	2895	* @min_region_sz: Minimum region size.
2854	*	2896	*
2855	* This function sets the region of @t as requested by @start and @end. If the	2897	* This function sets the region of @t as requested by @start and @end. If the
2856	* values of @start and @end are zero, however, this function finds the biggest	2898	* values of @start and @end are zero, however, this function finds the biggest
@@ -2862,7 +2904,7 @@ static bool damon_find_biggest_system_ram(unsigned long *start, · 5 unmodified lines
2862	*/	2904	*/
2863	int damon_set_region_biggest_system_ram_default(struct damon_target *t,	2905	int damon_set_region_biggest_system_ram_default(struct damon_target *t,
2864	unsigned long start, unsigned long end,	2906	unsigned long start, unsigned long end,
2865	unsigned long min_sz_region)	2907	unsigned long min_region_sz)
2866	{	2908	{
2867	struct damon_addr_range addr_range;	2909	struct damon_addr_range addr_range;
2868		2910
@@ -2875,7 +2917,7 @@ int damon_set_region_biggest_system_ram_default(struct damon_target *t, · 6 unmodified lines
2875		2917
2876	addr_range.start = *start;	2918	addr_range.start = *start;
2877	addr_range.end = *end;	2919	addr_range.end = *end;
2878	return damon_set_regions(t, &addr_range, 1, min_sz_region);	2920	return damon_set_regions(t, &addr_range, 1, min_region_sz);
2879	}	2921	}
2880		2922
2881	/*	2923	/*

mm/damon/lru_sort.c+112−8

@@ -34,13 +34,56 @@ static bool enabled __read_mostly; · 33 unmodified lines
34	*	34	*
35	* Input parameters that updated while DAMON_LRU_SORT is running are not	35	* Input parameters that updated while DAMON_LRU_SORT is running are not
36	* applied by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT	36	* applied by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT
37	* reads values of parametrs except ``enabled`` again. Once the re-reading is	37	* reads values of parameters except ``enabled`` again. Once the re-reading is
38	* done, this parameter is set as ``N``. If invalid parameters are found while	38	* done, this parameter is set as ``N``. If invalid parameters are found while
39	* the re-reading, DAMON_LRU_SORT will be disabled.	39	* the re-reading, DAMON_LRU_SORT will be disabled.
40	*/	40	*/
41	static bool commit_inputs __read_mostly;	41	static bool commit_inputs __read_mostly;
42	module_param(commit_inputs, bool, 0600);	42	module_param(commit_inputs, bool, 0600);
43		43
		44	/*
		45	* Desired active to [in]active memory ratio in bp (1/10,000).
		46	*
		47	* While keeping the caps that set by other quotas, DAMON_LRU_SORT
		48	* automatically increases and decreases the effective level of the quota
		49	* aiming the LRU [de]prioritizations of the hot and cold memory resulting in
		50	* this active to [in]active memory ratio. Value zero means disabling this
		51	* auto-tuning feature.
		52	*
		53	* Disabled by default.
		54	*/
		55	static unsigned long active_mem_bp __read_mostly;
		56	module_param(active_mem_bp, ulong, 0600);
		57
		58	/*
		59	* Auto-tune monitoring intervals.
		60	*
		61	* If this parameter is set as ``Y``, DAMON_LRU_SORT automatically tunes
		62	* DAMON's sampling and aggregation intervals. The auto-tuning aims to capture
		63	* meaningful amount of access events in each DAMON-snapshot, while keeping the
		64	* sampling interval 5 milliseconds in minimum, and 10 seconds in maximum.
		65	* Setting this as ``N`` disables the auto-tuning.
		66	*
		67	* Disabled by default.
		68	*/
		69	static bool autotune_monitoring_intervals __read_mostly;
		70	module_param(autotune_monitoring_intervals, bool, 0600);
		71
		72	/*
		73	* Filter [non-]young pages accordingly for LRU [de]prioritizations.
		74	*
		75	* If this is set, check page level access (youngness) once again before each
		76	* LRU [de]prioritization operation. LRU prioritization operation is skipped
		77	* if the page has not accessed since the last check (not young). LRU
		78	* deprioritization operation is skipped if the page has accessed since the
		79	* last check (young). The feature is enabled or disabled if this parameter is
		80	* set as ``Y`` or ``N``, respectively.
		81	*
		82	* Disabled by default.
		83	*/
		84	static bool filter_young_pages __read_mostly;
		85	module_param(filter_young_pages, bool, 0600);
		86
44	/*	87	/*
45	* Access frequency threshold for hot memory regions identification in permil.	88	* Access frequency threshold for hot memory regions identification in permil.
46	*	89	*
@@ -71,7 +114,7 @@ static struct damos_quota damon_lru_sort_quota = { · 24 unmodified lines
71	/* Within the quota, mark hotter regions accessed first. */	114	/* Within the quota, mark hotter regions accessed first. */
72	.weight_sz = 0,	115	.weight_sz = 0,
73	.weight_nr_accesses = 1,	116	.weight_nr_accesses = 1,
74	.weight_age = 0,	117	.weight_age = 1,
75	};	118	};
76	DEFINE_DAMON_MODULES_DAMOS_TIME_QUOTA(damon_lru_sort_quota);	119	DEFINE_DAMON_MODULES_DAMOS_TIME_QUOTA(damon_lru_sort_quota);
77		120
@@ -193,10 +236,53 @@ static struct damos *damon_lru_sort_new_cold_scheme(unsigned int cold_thres) · 115 unmodified lines
193	return damon_lru_sort_new_scheme(&pattern, DAMOS_LRU_DEPRIO);	236	return damon_lru_sort_new_scheme(&pattern, DAMOS_LRU_DEPRIO);
194	}	237	}
195		238
		239	static int damon_lru_sort_add_quota_goals(struct damos *hot_scheme,
		240	struct damos *cold_scheme)
		241	{
		242	struct damos_quota_goal *goal;
		243
		244	if (!active_mem_bp)
		245	return 0;
		246	goal = damos_new_quota_goal(DAMOS_QUOTA_ACTIVE_MEM_BP, active_mem_bp);
		247	if (!goal)
		248	return -ENOMEM;
		249	damos_add_quota_goal(&hot_scheme->quota, goal);
		250	/* aim 0.2 % goal conflict, to keep little ping pong */
		251	goal = damos_new_quota_goal(DAMOS_QUOTA_INACTIVE_MEM_BP,
		252	10000 - active_mem_bp + 2);
		253	if (!goal)
		254	return -ENOMEM;
		255	damos_add_quota_goal(&cold_scheme->quota, goal);
		256	return 0;
		257	}
		258
		259	static int damon_lru_sort_add_filters(struct damos *hot_scheme,
		260	struct damos *cold_scheme)
		261	{
		262	struct damos_filter *filter;
		263
		264	if (!filter_young_pages)
		265	return 0;
		266
		267	/* disallow prioritizing not-young pages */
		268	filter = damos_new_filter(DAMOS_FILTER_TYPE_YOUNG, false, false);
		269	if (!filter)
		270	return -ENOMEM;
		271	damos_add_filter(hot_scheme, filter);
		272
		273	/* disabllow de-prioritizing young pages */
		274	filter = damos_new_filter(DAMOS_FILTER_TYPE_YOUNG, true, false);
		275	if (!filter)
		276	return -ENOMEM;
		277	damos_add_filter(cold_scheme, filter);
		278	return 0;
		279	}
		280
196	static int damon_lru_sort_apply_parameters(void)	281	static int damon_lru_sort_apply_parameters(void)
197	{	282	{
198	struct damon_ctx *param_ctx;	283	struct damon_ctx *param_ctx;
199	struct damon_target *param_target;	284	struct damon_target *param_target;
		285	struct damon_attrs attrs;
200	struct damos hot_scheme, cold_scheme;	286	struct damos hot_scheme, cold_scheme;
201	unsigned int hot_thres, cold_thres;	287	unsigned int hot_thres, cold_thres;
202	int err;	288	int err;
@@ -212,25 +298,34 @@ static int damon_lru_sort_apply_parameters(void) · 9 unmodified lines
212	if (!monitor_region_start && !monitor_region_end)	298	if (!monitor_region_start && !monitor_region_end)
213	addr_unit = 1;	299	addr_unit = 1;
214	param_ctx->addr_unit = addr_unit;	300	param_ctx->addr_unit = addr_unit;
215	param_ctx->min_sz_region = max(DAMON_MIN_REGION / addr_unit, 1);	301	param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);
216		302
217	if (!damon_lru_sort_mon_attrs.sample_interval) {	303	if (!damon_lru_sort_mon_attrs.sample_interval) {
218	err = -EINVAL;	304	err = -EINVAL;
219	goto out;	305	goto out;
220	}	306	}
221		307
222	err = damon_set_attrs(param_ctx, &damon_lru_sort_mon_attrs);	308	attrs = damon_lru_sort_mon_attrs;
		309	if (autotune_monitoring_intervals) {
		310	attrs.sample_interval = 5000;
		311	attrs.aggr_interval = 100000;
		312	attrs.intervals_goal.access_bp = 40;
		313	attrs.intervals_goal.aggrs = 3;
		314	attrs.intervals_goal.min_sample_us = 5000;
		315	attrs.intervals_goal.max_sample_us = 10 * 1000 * 1000;
		316	}
		317	err = damon_set_attrs(param_ctx, &attrs);
223	if (err)	318	if (err)
224	goto out;	319	goto out;
225		320
226	err = -ENOMEM;	321	err = -ENOMEM;
227	hot_thres = damon_max_nr_accesses(&damon_lru_sort_mon_attrs) *	322	hot_thres = damon_max_nr_accesses(&attrs) *
228	hot_thres_access_freq / 1000;	323	hot_thres_access_freq / 1000;
229	hot_scheme = damon_lru_sort_new_hot_scheme(hot_thres);	324	hot_scheme = damon_lru_sort_new_hot_scheme(hot_thres);
230	if (!hot_scheme)	325	if (!hot_scheme)
231	goto out;	326	goto out;
232		327
233	cold_thres = cold_min_age / damon_lru_sort_mon_attrs.aggr_interval;	328	cold_thres = cold_min_age / attrs.aggr_interval;
234	cold_scheme = damon_lru_sort_new_cold_scheme(cold_thres);	329	cold_scheme = damon_lru_sort_new_cold_scheme(cold_thres);
235	if (!cold_scheme) {	330	if (!cold_scheme) {
236	damon_destroy_scheme(hot_scheme);	331	damon_destroy_scheme(hot_scheme);
@@ -240,10 +335,17 @@ static int damon_lru_sort_apply_parameters(void) · 3 unmodified lines
240	damon_set_schemes(param_ctx, &hot_scheme, 1);	335	damon_set_schemes(param_ctx, &hot_scheme, 1);
241	damon_add_scheme(param_ctx, cold_scheme);	336	damon_add_scheme(param_ctx, cold_scheme);
242		337
		338	err = damon_lru_sort_add_quota_goals(hot_scheme, cold_scheme);
		339	if (err)
		340	goto out;
		341	err = damon_lru_sort_add_filters(hot_scheme, cold_scheme);
		342	if (err)
		343	goto out;
		344
243	err = damon_set_region_biggest_system_ram_default(param_target,	345	err = damon_set_region_biggest_system_ram_default(param_target,
244	&monitor_region_start,	346	&monitor_region_start,
245	&monitor_region_end,	347	&monitor_region_end,
246	param_ctx->min_sz_region);	348	param_ctx->min_region_sz);
247	if (err)	349	if (err)
248	goto out;	350	goto out;
249	err = damon_commit_ctx(ctx, param_ctx);	351	err = damon_commit_ctx(ctx, param_ctx);
@@ -303,7 +405,9 @@ static int damon_lru_sort_turn(bool on) · 53 unmodified lines
303	err = damon_start(&ctx, 1, true);	405	err = damon_start(&ctx, 1, true);
304	if (err)	406	if (err)
305	return err;	407	return err;
306	kdamond_pid = ctx->kdamond->pid;	408	kdamond_pid = damon_kdamond_pid(ctx);
		409	if (kdamond_pid < 0)
		410	return kdamond_pid;
307	return damon_call(ctx, &call_control);	411	return damon_call(ctx, &call_control);
308	}	412	}
309		413

mm/damon/paddr.c+12−15

 	LIST_HEAD(folio_list);
 	bool install_young_filter = true;
 	struct damos_filter *filter;
-	struct folio *folio;
+	struct folio *folio = NULL;
 	/* check access in page level again by default */
 	damos_for_each_ops_filter(filter, s) {
 	return damon_pa_core_addr(applied * PAGE_SIZE, addr_unit);
 }
-static inline unsigned long damon_pa_mark_accessed_or_deactivate(
+static inline unsigned long damon_pa_de_activate(
 		struct damon_region *r, unsigned long addr_unit,
-		struct damos *s, bool mark_accessed,
+		struct damos *s, bool activate,
 		unsigned long *sz_filter_passed)
 {
 	phys_addr_t addr, applied = 0;
-	struct folio *folio;
+	struct folio *folio = NULL;
 	addr = damon_pa_phys_addr(r->ar.start, addr_unit);
 	while (addr < damon_pa_phys_addr(r->ar.end, addr_unit)) {
 		else
 			*sz_filter_passed += folio_size(folio) / addr_unit;
-		if (mark_accessed)
+		if (activate)
-			folio_mark_accessed(folio);
+			folio_activate(folio);
 		else
 			folio_deactivate(folio);
 		applied += folio_nr_pages(folio);
 	return damon_pa_core_addr(applied * PAGE_SIZE, addr_unit);
 }
-static unsigned long damon_pa_mark_accessed(struct damon_region *r,
+static unsigned long damon_pa_activate_pages(struct damon_region *r,
 		unsigned long addr_unit, struct damos *s,
 		unsigned long *sz_filter_passed)
 {
-	return damon_pa_mark_accessed_or_deactivate(r, addr_unit, s, true,
+	return damon_pa_de_activate(r, addr_unit, s, true, sz_filter_passed);
-			sz_filter_passed);
 }
 static unsigned long damon_pa_deactivate_pages(struct damon_region *r,
 		unsigned long addr_unit, struct damos *s,
 		unsigned long *sz_filter_passed)
 {
-	return damon_pa_mark_accessed_or_deactivate(r, addr_unit, s, false,
+	return damon_pa_de_activate(r, addr_unit, s, false, sz_filter_passed);
-			sz_filter_passed);
 }
 static unsigned long damon_pa_migrate(struct damon_region *r,
 {
 	phys_addr_t addr, applied;
 	LIST_HEAD(folio_list);
-	struct folio *folio;
+	struct folio *folio = NULL;
 	addr = damon_pa_phys_addr(r->ar.start, addr_unit);
 	while (addr < damon_pa_phys_addr(r->ar.end, addr_unit)) {
 		unsigned long *sz_filter_passed)
 {
 	phys_addr_t addr;
-	struct folio *folio;
+	struct folio *folio = NULL;
 	if (!damos_ops_has_filter(s))
 		return 0;
 	case DAMOS_PAGEOUT:
 		return damon_pa_pageout(r, aunit, scheme, sz_filter_passed);
 	case DAMOS_LRU_PRIO:
-		return damon_pa_mark_accessed(r, aunit, scheme,
+		return damon_pa_activate_pages(r, aunit, scheme,
 				sz_filter_passed);
 	case DAMOS_LRU_DEPRIO:
 		return damon_pa_deactivate_pages(r, aunit, scheme,
 		.prepare_access_checks = damon_pa_prepare_access_checks,
 		.check_accesses = damon_pa_check_accesses,
 		.target_valid = NULL,
-		.cleanup = NULL,
 		.apply_scheme = damon_pa_apply_scheme,
 		.get_scheme_score = damon_pa_scheme_score,
 	};

mm/damon/reclaim.c+6−4

  *
  * Input parameters that updated while DAMON_RECLAIM is running are not applied
  * by default.  Once this parameter is set as ``Y``, DAMON_RECLAIM reads values
- * of parametrs except ``enabled`` again.  Once the re-reading is done, this
+ * of parameters except ``enabled`` again.  Once the re-reading is done, this
  * parameter is set as ``N``.  If invalid parameters are found while the
  * re-reading, DAMON_RECLAIM will be disabled.
  */
 	if (!monitor_region_start && !monitor_region_end)
 		addr_unit = 1;
 	param_ctx->addr_unit = addr_unit;
-	param_ctx->min_sz_region = max(DAMON_MIN_REGION / addr_unit, 1);
+	param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);
 	if (!damon_reclaim_mon_attrs.aggr_interval) {
 		err = -EINVAL;
 	err = damon_set_region_biggest_system_ram_default(param_target,
 					&monitor_region_start,
 					&monitor_region_end,
-					param_ctx->min_sz_region);
+					param_ctx->min_region_sz);
 	if (err)
 		goto out;
 	err = damon_commit_ctx(ctx, param_ctx);
 	err = damon_start(&ctx, 1, true);
 	if (err)
 		return err;
-	kdamond_pid = ctx->kdamond->pid;
+	kdamond_pid = damon_kdamond_pid(ctx);
+	if (kdamond_pid < 0)
+		return kdamond_pid;
 	return damon_call(ctx, &call_control);
 }

mm/damon/stat.c+3−11

 // SPDX-License-Identifier: GPL-2.0
 /*
- * Shows data access monitoring resutls in simple metrics.
+ * Shows data access monitoring results in simple metrics.
  */
 #define pr_fmt(fmt) "damon-stat: " fmt
 MODULE_PARM_DESC(estimated_memory_bandwidth,
 		"Estimated memory bandwidth usage in bytes per second");
-static long memory_idle_ms_percentiles[101] __read_mostly = {0,};
+static long memory_idle_ms_percentiles[101] = {0,};
 module_param_array(memory_idle_ms_percentiles, long, NULL, 0400);
 MODULE_PARM_DESC(memory_idle_ms_percentiles,
 		"Memory idle time percentiles in milliseconds");
 	if (damon_set_attrs(ctx, &attrs))
 		goto free_out;
-	/*
-	 * auto-tune sampling and aggregation interval aiming 4% DAMON-observed
-	 * accesses ratio, keeping sampling interval in [5ms, 10s] range.
-	 */
-	ctx->attrs.intervals_goal = (struct damon_intervals_goal) {
-		.access_bp = 400, .aggrs = 3,
-		.min_sample_us = 5000, .max_sample_us = 10000000,
-	};
 	if (damon_select_ops(ctx, DAMON_OPS_PADDR))
 		goto free_out;
 		goto free_out;
 	damon_add_target(ctx, target);
 	if (damon_set_region_biggest_system_ram_default(target, &start, &end,
-							ctx->min_sz_region))
+							ctx->min_region_sz))
 		goto free_out;
 	return ctx;
 free_out:

mm/damon/sysfs-schemes.c+54−4

 	unsigned long sz_applied;
 	unsigned long sz_ops_filter_passed;
 	unsigned long qt_exceeds;
+	unsigned long nr_snapshots;
+	unsigned long max_nr_snapshots;
 };
 static struct damon_sysfs_stats *damon_sysfs_stats_alloc(void)
 	return sysfs_emit(buf, "%lu\n", stats->qt_exceeds);
 }
+static ssize_t nr_snapshots_show(struct kobject *kobj,
+		struct kobj_attribute *attr, char *buf)
+{
+	struct damon_sysfs_stats *stats = container_of(kobj,
+			struct damon_sysfs_stats, kobj);
+	return sysfs_emit(buf, "%lu\n", stats->nr_snapshots);
+}
+static ssize_t max_nr_snapshots_show(struct kobject *kobj,
+		struct kobj_attribute *attr, char *buf)
+{
+	struct damon_sysfs_stats *stats = container_of(kobj,
+			struct damon_sysfs_stats, kobj);
+	return sysfs_emit(buf, "%lu\n", stats->max_nr_snapshots);
+}
+static ssize_t max_nr_snapshots_store(struct kobject *kobj,
+		struct kobj_attribute *attr, const char *buf, size_t count)
+{
+	struct damon_sysfs_stats *stats = container_of(kobj,
+			struct damon_sysfs_stats, kobj);
+	unsigned long max_nr_snapshots, err = kstrtoul(buf, 0, &max_nr_snapshots);
+	if (err)
+		return err;
+	stats->max_nr_snapshots = max_nr_snapshots;
+	return count;
+}
 static void damon_sysfs_stats_release(struct kobject *kobj)
 {
 	kfree(container_of(kobj, struct damon_sysfs_stats, kobj));
 static struct kobj_attribute damon_sysfs_stats_qt_exceeds_attr =
 		__ATTR_RO_MODE(qt_exceeds, 0400);
+static struct kobj_attribute damon_sysfs_stats_nr_snapshots_attr =
+		__ATTR_RO_MODE(nr_snapshots, 0400);
+static struct kobj_attribute damon_sysfs_stats_max_nr_snapshots_attr =
+		__ATTR_RW_MODE(max_nr_snapshots, 0600);
 static struct attribute *damon_sysfs_stats_attrs[] = {
 	&damon_sysfs_stats_nr_tried_attr.attr,
 	&damon_sysfs_stats_sz_tried_attr.attr,
 	&damon_sysfs_stats_sz_applied_attr.attr,
 	&damon_sysfs_stats_sz_ops_filter_passed_attr.attr,
 	&damon_sysfs_stats_qt_exceeds_attr.attr,
+	&damon_sysfs_stats_nr_snapshots_attr.attr,
+	&damon_sysfs_stats_max_nr_snapshots_attr.attr,
 	NULL,
 };
 ATTRIBUTE_GROUPS(damon_sysfs_stats);
 		.metric = DAMOS_QUOTA_NODE_MEMCG_FREE_BP,
 		.name = "node_memcg_free_bp",
 	},
+	{
+		.metric = DAMOS_QUOTA_ACTIVE_MEM_BP,
+		.name = "active_mem_bp",
+	},
+	{
+		.metric = DAMOS_QUOTA_INACTIVE_MEM_BP,
+		.name = "inactive_mem_bp",
+	},
 };
 static ssize_t target_metric_show(struct kobject *kobj,
 			struct damon_sysfs_scheme, kobj);
 	int err = 0;
-	/* TODO: error handling for target_nid range. */
 	err = kstrtoint(buf, 0, &scheme->target_nid);
 	return err ? err : count;
 	return false;
 }
-static int damon_sysfs_memcg_path_to_id(char *memcg_path, unsigned short *id)
+static int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id)
 {
 	struct mem_cgroup *memcg;
 	char *path;
 	for (memcg = mem_cgroup_iter(NULL, NULL, NULL); memcg;
 			memcg = mem_cgroup_iter(NULL, memcg, NULL)) {
-		/* skip removed memcg */
+		/* skip offlined memcg */
-		if (!mem_cgroup_id(memcg))
+		if (!mem_cgroup_online(memcg))
 			continue;
 		if (damon_sysfs_memcg_path_eq(memcg, path, memcg_path)) {
 			*id = mem_cgroup_id(memcg);
 		damon_destroy_scheme(scheme);
 		return NULL;
 	}
+	scheme->max_nr_snapshots = sysfs_scheme->stats->max_nr_snapshots;
 	return scheme;
 }
 		sysfs_stats->sz_ops_filter_passed =
 			scheme->stat.sz_ops_filter_passed;
 		sysfs_stats->qt_exceeds = scheme->stat.qt_exceeds;
+		sysfs_stats->nr_snapshots = scheme->stat.nr_snapshots;
 	}
 }

mm/damon/sysfs.c+9−9

 static int damon_sysfs_set_regions(struct damon_target *t,
 		struct damon_sysfs_regions *sysfs_regions,
-		unsigned long min_sz_region)
+		unsigned long min_region_sz)
 {
 	struct damon_addr_range *ranges = kmalloc_array(sysfs_regions->nr,
 			sizeof(*ranges), GFP_KERNEL | __GFP_NOWARN);
 		if (ranges[i - 1].end > ranges[i].start)
 			goto out;
 	}
-	err = damon_set_regions(t, ranges, sysfs_regions->nr, min_sz_region);
+	err = damon_set_regions(t, ranges, sysfs_regions->nr, min_region_sz);
 out:
 	kfree(ranges);
 	return err;
 			return -EINVAL;
 	}
 	t->obsolete = sys_target->obsolete;
-	return damon_sysfs_set_regions(t, sys_target->regions, ctx->min_sz_region);
+	return damon_sysfs_set_regions(t, sys_target->regions,
+			ctx->min_region_sz);
 }
 static int damon_sysfs_add_targets(struct damon_ctx *ctx,
 	ctx->addr_unit = sys_ctx->addr_unit;
 	/* addr_unit is respected by only DAMON_OPS_PADDR */
 	if (sys_ctx->ops_id == DAMON_OPS_PADDR)
-		ctx->min_sz_region = max(
+		ctx->min_region_sz = max(
-				DAMON_MIN_REGION / sys_ctx->addr_unit, 1);
+				DAMON_MIN_REGION_SZ / sys_ctx->addr_unit, 1);
 	err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs);
 	if (err)
 		return err;
 	if (!ctx)
 		goto out;
-	mutex_lock(&ctx->kdamond_lock);
+	pid = damon_kdamond_pid(ctx);
-	if (ctx->kdamond)
+	if (pid < 0)
-		pid = ctx->kdamond->pid;
+		pid = -1;
-	mutex_unlock(&ctx->kdamond_lock);
 out:
 	mutex_unlock(&damon_sysfs_lock);
 	return sysfs_emit(buf, "%d\n", pid);

mm/damon/tests/core-kunit.h+39−12

 	r->nr_accesses_bp = 420000;
 	r->nr_accesses = 42;
 	r->last_nr_accesses = 15;
+	r->age = 10;
 	damon_add_region(r, t);
 	damon_split_region_at(t, r, 25);
 	KUNIT_EXPECT_EQ(test, r->ar.start, 0ul);
 	KUNIT_EXPECT_EQ(test, r->nr_accesses_bp, r_new->nr_accesses_bp);
 	KUNIT_EXPECT_EQ(test, r->nr_accesses, r_new->nr_accesses);
 	KUNIT_EXPECT_EQ(test, r->last_nr_accesses, r_new->last_nr_accesses);
+	KUNIT_EXPECT_EQ(test, r->age, r_new->age);
 	damon_free_target(t);
 }
 	}
 	r->nr_accesses = 10;
 	r->nr_accesses_bp = 100000;
+	r->age = 9;
 	damon_add_region(r, t);
 	r2 = damon_new_region(100, 300);
 	if (!r2) {
 	}
 	r2->nr_accesses = 20;
 	r2->nr_accesses_bp = 200000;
+	r2->age = 21;
 	damon_add_region(r2, t);
 	damon_merge_two_regions(t, r, r2);
 	KUNIT_EXPECT_EQ(test, r->ar.start, 0ul);
 	KUNIT_EXPECT_EQ(test, r->ar.end, 300ul);
 	KUNIT_EXPECT_EQ(test, r->nr_accesses, 16u);
+	KUNIT_EXPECT_EQ(test, r->nr_accesses_bp, 160000u);
+	KUNIT_EXPECT_EQ(test, r->age, 17u);
 	i = 0;
 	damon_for_each_region(r3, t) {
 {
 	struct damon_target *t;
 	struct damon_region *r;
-	unsigned long sa[] = {0, 100, 114, 122, 130, 156, 170, 184};
+	unsigned long sa[] = {0, 100, 114, 122, 130, 156, 170, 184, 230};
-	unsigned long ea[] = {100, 112, 122, 130, 156, 170, 184, 230};
+	unsigned long ea[] = {100, 112, 122, 130, 156, 170, 184, 230, 10170};
-	unsigned int nrs[] = {0, 0, 10, 10, 20, 30, 1, 2};
+	unsigned int nrs[] = {0, 0, 10, 10, 20, 30, 1, 2, 5};
-	unsigned long saddrs[] = {0, 114, 130, 156, 170};
+	unsigned long saddrs[] = {0, 114, 130, 156, 170, 230};
-	unsigned long eaddrs[] = {112, 130, 156, 170, 230};
+	unsigned long eaddrs[] = {112, 130, 156, 170, 230, 10170};
 	int i;
 	t = damon_new_target();
 	}
 	damon_merge_regions_of(t, 9, 9999);
-	/* 0-112, 114-130, 130-156, 156-170 */
+	/* 0-112, 114-130, 130-156, 156-170, 170-230, 230-10170 */
-	KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 5u);
+	KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 6u);
-	for (i = 0; i < 5; i++) {
+	for (i = 0; i < 6; i++) {
 		r = __nth_region_of(t, i);
 		KUNIT_EXPECT_EQ(test, r->ar.start, saddrs[i]);
 		KUNIT_EXPECT_EQ(test, r->ar.end, eaddrs[i]);
 {
 	struct damon_target *t;
 	struct damon_region *r;
+	unsigned long sa[] = {0, 300, 500};
+	unsigned long ea[] = {220, 400, 700};
+	int i;
 	t = damon_new_target();
 	if (!t)
 	damon_split_regions_of(t, 4, 1);
 	KUNIT_EXPECT_LE(test, damon_nr_regions(t), 4u);
 	damon_free_target(t);
+	t = damon_new_target();
+	if (!t)
+		kunit_skip(test, "third target alloc fail");
+	for (i = 0; i < ARRAY_SIZE(sa); i++) {
+		r = damon_new_region(sa[i], ea[i]);
+		if (!r) {
+			damon_free_target(t);
+			kunit_skip(test, "region alloc fail");
+		}
+		damon_add_region(r, t);
+	}
+	damon_split_regions_of(t, 4, 5);
+	KUNIT_EXPECT_LE(test, damon_nr_regions(t), 12u);
+	damon_for_each_region(r, t)
+		KUNIT_EXPECT_GE(test, damon_sz_region(r) % 5ul, 0ul);
+	damon_free_target(t);
 }
 static void damon_test_ops_registration(struct kunit *test)
 			});
 	damos_test_commit_quota_goal_for(test, &dst,
 			&(struct damos_quota_goal) {
-			.metric = DAMOS_QUOTA_USER_INPUT,
+			.metric = DAMOS_QUOTA_SOME_MEM_PSI_US,
-			.target_value = 789,
+			.target_value = 234,
-			.current_value = 12,
+			.current_value = 345,
+			.last_psi_total = 567,
 			});
 }
 	damos_set_filters_default_reject(&scheme);
 	/*
 	 * A core-handled allow-filter is installed.
-	 * Rejct by default on core layer filtering stage due to the last
+	 * Reject by default on core layer filtering stage due to the last
 	 * core-layer-filter's behavior.
 	 * Allow by default on ops layer filtering stage due to the absence of
 	 * ops layer filters.

mm/damon/tests/vaddr-kunit.h+1−1

@@ -147,7 +147,7 @@ static void damon_do_test_apply_three_regions(struct kunit *test, · 146 unmodified lines
147	damon_add_region(r, t);	147	damon_add_region(r, t);
148	}	148	}
149		149
150	damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION);	150	damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ);
151		151
152	for (i = 0; i < nr_expected / 2; i++) {	152	for (i = 0; i < nr_expected / 2; i++) {
153	r = __nth_region_of(t, i);	153	r = __nth_region_of(t, i);

mm/damon/vaddr.c+12−13

 #include "ops-common.h"
 #ifdef CONFIG_DAMON_VADDR_KUNIT_TEST
-#undef DAMON_MIN_REGION
+#undef DAMON_MIN_REGION_SZ
-#define DAMON_MIN_REGION 1
+#define DAMON_MIN_REGION_SZ 1
 #endif
 /*
 	orig_end = r->ar.end;
 	sz_orig = damon_sz_region(r);
-	sz_piece = ALIGN_DOWN(sz_orig / nr_pieces, DAMON_MIN_REGION);
+	sz_piece = ALIGN_DOWN(sz_orig / nr_pieces, DAMON_MIN_REGION_SZ);
 	if (!sz_piece)
 		return -EINVAL;
 		swap(first_gap, second_gap);
 	/* Store the result */
-	regions[0].start = ALIGN(start, DAMON_MIN_REGION);
+	regions[0].start = ALIGN(start, DAMON_MIN_REGION_SZ);
-	regions[0].end = ALIGN(first_gap.start, DAMON_MIN_REGION);
+	regions[0].end = ALIGN(first_gap.start, DAMON_MIN_REGION_SZ);
-	regions[1].start = ALIGN(first_gap.end, DAMON_MIN_REGION);
+	regions[1].start = ALIGN(first_gap.end, DAMON_MIN_REGION_SZ);
-	regions[1].end = ALIGN(second_gap.start, DAMON_MIN_REGION);
+	regions[1].end = ALIGN(second_gap.start, DAMON_MIN_REGION_SZ);
-	regions[2].start = ALIGN(second_gap.end, DAMON_MIN_REGION);
+	regions[2].start = ALIGN(second_gap.end, DAMON_MIN_REGION_SZ);
-	regions[2].end = ALIGN(prev->vm_end, DAMON_MIN_REGION);
+	regions[2].end = ALIGN(prev->vm_end, DAMON_MIN_REGION_SZ);
 	return 0;
 }
 		sz += regions[i].end - regions[i].start;
 	if (ctx->attrs.min_nr_regions)
 		sz /= ctx->attrs.min_nr_regions;
-	if (sz < DAMON_MIN_REGION)
+	if (sz < DAMON_MIN_REGION_SZ)
-		sz = DAMON_MIN_REGION;
+		sz = DAMON_MIN_REGION_SZ;
 	/* Set the initial three regions of the target */
 	for (i = 0; i < 3; i++) {
 	damon_for_each_target(t, ctx) {
 		if (damon_va_three_regions(t, three_regions))
 			continue;
-		damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION);
+		damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ);
 	}
 }
 		.check_accesses = damon_va_check_accesses,
 		.target_valid = damon_va_target_valid,
 		.cleanup_target = damon_va_cleanup_target,
-		.cleanup = NULL,
 		.apply_scheme = damon_va_apply_scheme,
 		.get_scheme_score = damon_va_scheme_score,
 	};

mm/debug_vm_pgtable.c+17−21

 	return random_vaddr;
 }
-static void __init destroy_args(struct pgtable_debug_args *args)
+static void __init
+debug_vm_pgtable_free_huge_page(struct pgtable_debug_args *args,
+		unsigned long pfn, int order)
 {
-	struct page *page = NULL;
+#ifdef CONFIG_CONTIG_ALLOC
+	if (args->is_contiguous_page) {
+		free_contig_range(pfn, 1 << order);
+		return;
+	}
+#endif
+	__free_pages(pfn_to_page(pfn), order);
+}
+static void __init destroy_args(struct pgtable_debug_args *args)
+{
 	/* Free (huge) page */
 	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
 	    has_transparent_pud_hugepage() &&
 	    args->pud_pfn != ULONG_MAX) {
-		if (args->is_contiguous_page) {
+		debug_vm_pgtable_free_huge_page(args, args->pud_pfn, HPAGE_PUD_ORDER);
-			free_contig_range(args->pud_pfn,
-					  (1 << (HPAGE_PUD_SHIFT - PAGE_SHIFT)));
-		} else {
-			page = pfn_to_page(args->pud_pfn);
-			__free_pages(page, HPAGE_PUD_SHIFT - PAGE_SHIFT);
-		}
 		args->pud_pfn = ULONG_MAX;
 		args->pmd_pfn = ULONG_MAX;
 		args->pte_pfn = ULONG_MAX;
 	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
 	    has_transparent_hugepage() &&
 	    args->pmd_pfn != ULONG_MAX) {
-		if (args->is_contiguous_page) {
+		debug_vm_pgtable_free_huge_page(args, args->pmd_pfn, HPAGE_PMD_ORDER);
-			free_contig_range(args->pmd_pfn, (1 << HPAGE_PMD_ORDER));
-		} else {
-			page = pfn_to_page(args->pmd_pfn);
-			__free_pages(page, HPAGE_PMD_ORDER);
-		}
 		args->pmd_pfn = ULONG_MAX;
 		args->pte_pfn = ULONG_MAX;
 	}
 	if (args->pte_pfn != ULONG_MAX) {
-		page = pfn_to_page(args->pte_pfn);
+		__free_page(pfn_to_page(args->pte_pfn));
-		__free_page(page);
 		args->pte_pfn = ULONG_MAX;
 	}
 	 */
 	if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
 	    has_transparent_pud_hugepage()) {
-		page = debug_vm_pgtable_alloc_huge_page(args,
+		page = debug_vm_pgtable_alloc_huge_page(args, HPAGE_PUD_ORDER);
-				HPAGE_PUD_SHIFT - PAGE_SHIFT);
 		if (page) {
 			args->pud_pfn = page_to_pfn(page);
 			args->pmd_pfn = args->pud_pfn;

mm/dmapool_test.c+1−0

@@ -1,3 +1,4 @@
		1	// SPDX-License-Identifier: GPL-2.0
1	#include <linux/device.h>	2	#include <linux/device.h>
2	#include <linux/dma-map-ops.h>	3	#include <linux/dma-map-ops.h>
3	#include <linux/dma-mapping.h>	4	#include <linux/dma-mapping.h>

mm/early_ioremap.c+12−4

 }
 early_param("early_ioremap_debug", early_ioremap_debug_setup);
+#define early_ioremap_dbg(fmt, args...)			\
+	do {						\
+		if (unlikely(early_ioremap_debug)) {	\
+			pr_warn(fmt, ##args);		\
+			dump_stack();			\
+		}					\
+	} while (0)
 static int after_paging_init __initdata;
 pgprot_t __init __weak early_memremap_pgprot_adjust(resource_size_t phys_addr,
 	if (WARN_ON(nrpages > NR_FIX_BTMAPS))
 		return NULL;
+	early_ioremap_dbg("%s(%pa, %08lx) [%d] => %08lx + %08lx\n",
+			  __func__, &phys_addr, size, slot, slot_virt[slot], offset);
 	/*
 	 * Ok, go for it..
 	 */
 		--idx;
 		--nrpages;
 	}
-	WARN(early_ioremap_debug, "%s(%pa, %08lx) [%d] => %08lx + %08lx\n",
-	     __func__, &phys_addr, size, slot, offset, slot_virt[slot]);
 	prev_map[slot] = (void __iomem *)(offset + slot_virt[slot]);
 	return prev_map[slot];
 		  __func__, addr, size, slot, prev_size[slot]))
 		return;
-	WARN(early_ioremap_debug, "%s(%p, %08lx) [%d]\n",
+	early_ioremap_dbg("%s(%p, %08lx) [%d]\n", __func__, addr, size, slot);
-	      __func__, addr, size, slot);
 	virt_addr = (unsigned long)addr;
 	if (WARN_ON(virt_addr < fix_to_virt(FIX_BTMAP_BEGIN)))

mm/folio-compat.c+1−0

@@ -1,3 +1,4 @@
		1	// SPDX-License-Identifier: GPL-2.0
1	/*	2	/*
2	* Compatibility functions which bloat the callers too much to make inline.	3	* Compatibility functions which bloat the callers too much to make inline.
3	* All of the callers of these functions should be converted to use folios	4	* All of the callers of these functions should be converted to use folios

mm/gup.c+0−11

@@ -2806,17 +2806,6 @@ static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags) · 2805 unmodified lines
2806	return !reject_file_backed \|\| shmem_mapping(mapping);	2806	return !reject_file_backed \|\| shmem_mapping(mapping);
2807	}	2807	}
2808		2808
2809	static void __maybe_unused gup_fast_undo_dev_pagemap(int *nr, int nr_start,
2810	unsigned int flags, struct page **pages)
2811	{
2812	while ((*nr) - nr_start) {
2813	struct folio folio = page_folio(pages[--(nr)]);
2814
2815	folio_clear_referenced(folio);
2816	gup_put_folio(folio, 1, flags);
2817	}
2818	}
2819
2820	#ifdef CONFIG_ARCH_HAS_PTE_SPECIAL	2809	#ifdef CONFIG_ARCH_HAS_PTE_SPECIAL
2821	/*	2810	/*
2822	* GUP-fast relies on pte change detection to avoid concurrent pgtable	2811	* GUP-fast relies on pte change detection to avoid concurrent pgtable

mm/gup_test.c+1−0

@@ -1,3 +1,4 @@
		1	// SPDX-License-Identifier: GPL-2.0
1	#include <linux/kernel.h>	2	#include <linux/kernel.h>
2	#include <linux/mm.h>	3	#include <linux/mm.h>
3	#include <linux/slab.h>	4	#include <linux/slab.h>

mm/highmem.c+2−1

 		for (i = 0; i < kctrl->idx; i++) {
 			unsigned long base_addr;
 			int idx;
+			pte_t pteval = kctrl->pteval[i];
 			idx = arch_kmap_local_map_idx(i, pte_pfn(pteval));
 			base_addr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
 			if (base_addr == base)
-				return pte_page(kctrl->pteval[i]);
+				return pte_page(pteval);
 		}
 	}

mm/huge_memory.c+4−4

@@ -3431,7 +3431,7 @@ static void remap_page(struct folio *folio, unsigned long nr, int flags) · 3430 unmodified lines
3431	if (!folio_test_anon(folio))	3431	if (!folio_test_anon(folio))
3432	return;	3432	return;
3433	for (;;) {	3433	for (;;) {
3434	remove_migration_ptes(folio, folio, RMP_LOCKED \| flags);	3434	remove_migration_ptes(folio, folio, TTU_RMAP_LOCKED \| flags);
3435	i += folio_nr_pages(folio);	3435	i += folio_nr_pages(folio);
3436	if (i >= nr)	3436	if (i >= nr)
3437	break;	3437	break;
@@ -3944,7 +3944,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, · 506 unmodified lines
3944	int old_order = folio_order(folio);	3944	int old_order = folio_order(folio);
3945	struct folio new_folio, next;	3945	struct folio new_folio, next;
3946	int nr_shmem_dropped = 0;	3946	int nr_shmem_dropped = 0;
3947	int remap_flags = 0;	3947	enum ttu_flags ttu_flags = 0;
3948	int ret;	3948	int ret;
3949	pgoff_t end = 0;	3949	pgoff_t end = 0;
3950		3950
@@ -4064,9 +4064,9 @@ fail: · 113 unmodified lines
4064	shmem_uncharge(mapping->host, nr_shmem_dropped);	4064	shmem_uncharge(mapping->host, nr_shmem_dropped);
4065		4065
4066	if (!ret && is_anon && !folio_is_device_private(folio))	4066	if (!ret && is_anon && !folio_is_device_private(folio))
4067	remap_flags = RMP_USE_SHARED_ZEROPAGE;	4067	ttu_flags = TTU_USE_SHARED_ZEROPAGE;
4068		4068
4069	remap_page(folio, 1 << old_order, remap_flags);	4069	remap_page(folio, 1 << old_order, ttu_flags);
4070		4070
4071	/*	4071	/*
4072	* Unlock all after-split folios except the one containing	4072	* Unlock all after-split folios except the one containing

mm/hugetlb.c+93−87

@@ -121,16 +121,6 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, · 120 unmodified lines
121	unsigned long start, unsigned long end, bool take_locks);	121	unsigned long start, unsigned long end, bool take_locks);
122	static struct resv_map vma_resv_map(struct vm_area_struct vma);	122	static struct resv_map vma_resv_map(struct vm_area_struct vma);
123		123
124	static void hugetlb_free_folio(struct folio *folio)
125	{
126	if (folio_test_hugetlb_cma(folio)) {
127	hugetlb_cma_free_folio(folio);
128	return;
129	}
130
131	folio_put(folio);
132	}
133
134	static inline bool subpool_is_free(struct hugepage_subpool *spool)	124	static inline bool subpool_is_free(struct hugepage_subpool *spool)
135	{	125	{
136	if (spool->count)	126	if (spool->count)
@@ -588,8 +578,9 @@ hugetlb_resv_map_add(struct resv_map map, struct list_head rg, long from, · 451 unmodified lines
588	record_hugetlb_cgroup_uncharge_info(cg, h, map, nrg);	578	record_hugetlb_cgroup_uncharge_info(cg, h, map, nrg);
589	list_add(&nrg->link, rg);	579	list_add(&nrg->link, rg);
590	coalesce_file_region(map, nrg);	580	coalesce_file_region(map, nrg);
591	} else	581	} else {
592	*regions_needed += 1;	582	*regions_needed += 1;
		583	}
593		584
594	return to - from;	585	return to - from;
595	}	586	}
@@ -1257,8 +1248,9 @@ void hugetlb_dup_vma_private(struct vm_area_struct *vma) · 661 unmodified lines
1257		1248
1258	if (vma_lock && vma_lock->vma != vma)	1249	if (vma_lock && vma_lock->vma != vma)
1259	vma->vm_private_data = NULL;	1250	vma->vm_private_data = NULL;
1260	} else	1251	} else {
1261	vma->vm_private_data = NULL;	1252	vma->vm_private_data = NULL;
		1253	}
1262	}	1254	}
1263		1255
1264	/*	1256	/*
@@ -1417,47 +1409,25 @@ err: · 152 unmodified lines
1417	return NULL;	1409	return NULL;
1418	}	1410	}
1419		1411
1420	#ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE	1412	#if defined(CONFIG_ARCH_HAS_GIGANTIC_PAGE) && defined(CONFIG_CONTIG_ALLOC)
1421	#ifdef CONFIG_CONTIG_ALLOC	1413	static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask,
1422	static struct folio *alloc_gigantic_folio(int order, gfp_t gfp_mask,
1423	int nid, nodemask_t *nodemask)	1414	int nid, nodemask_t *nodemask)
1424	{	1415	{
1425	struct folio *folio;	1416	struct folio *folio;
1426	bool retried = false;
1427
1428	retry:
1429	folio = hugetlb_cma_alloc_folio(order, gfp_mask, nid, nodemask);
1430	if (!folio) {
1431	if (hugetlb_cma_exclusive_alloc())
1432	return NULL;
1433
1434	folio = folio_alloc_gigantic(order, gfp_mask, nid, nodemask);
1435	if (!folio)
1436	return NULL;
1437	}
1438		1417
1439	if (folio_ref_freeze(folio, 1))	1418	folio = hugetlb_cma_alloc_frozen_folio(order, gfp_mask, nid, nodemask);
		1419	if (folio)
1440	return folio;	1420	return folio;
1441		1421
1442	pr_warn("HugeTLB: unexpected refcount on PFN %lu\n", folio_pfn(folio));	1422	if (hugetlb_cma_exclusive_alloc())
1443	hugetlb_free_folio(folio);	1423	return NULL;
1444	if (!retried) {
1445	retried = true;
1446	goto retry;
1447	}
1448	return NULL;
1449	}
1450		1424
1451	#else /* !CONFIG_CONTIG_ALLOC */	1425	folio = (struct folio *)alloc_contig_frozen_pages(1 << order, gfp_mask,
1452	static struct folio *alloc_gigantic_folio(int order, gfp_t gfp_mask, int nid,	1426	nid, nodemask);
1453	nodemask_t *nodemask)	1427	return folio;
1454	{
1455	return NULL;
1456	}	1428	}
1457	#endif /* CONFIG_CONTIG_ALLOC */	1429	#else /* !CONFIG_ARCH_HAS_GIGANTIC_PAGE \|\| !CONFIG_CONTIG_ALLOC */
1458		1430	static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask, int nid,
1459	#else /* !CONFIG_ARCH_HAS_GIGANTIC_PAGE */
1460	static struct folio *alloc_gigantic_folio(int order, gfp_t gfp_mask, int nid,
1461	nodemask_t *nodemask)	1431	nodemask_t *nodemask)
1462	{	1432	{
1463	return NULL;	1433	return NULL;
@@ -1587,9 +1557,11 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, · 123 unmodified lines
1587	if (unlikely(folio_test_hwpoison(folio)))	1557	if (unlikely(folio_test_hwpoison(folio)))
1588	folio_clear_hugetlb_hwpoison(folio);	1558	folio_clear_hugetlb_hwpoison(folio);
1589		1559
1590	folio_ref_unfreeze(folio, 1);	1560	VM_BUG_ON_FOLIO(folio_ref_count(folio), folio);
1591		1561	if (folio_test_hugetlb_cma(folio))
1592	hugetlb_free_folio(folio);	1562	hugetlb_cma_free_frozen_folio(folio);
		1563	else
		1564	free_frozen_pages(&folio->page, folio_order(folio));
1593	}	1565	}
1594		1566
1595	/*	1567	/*
@@ -1869,7 +1841,7 @@ struct address_space hugetlb_folio_mapping_lock_write(struct folio folio) · 273 unmodified lines
1869	return NULL;	1841	return NULL;
1870	}	1842	}
1871		1843
1872	static struct folio *alloc_buddy_hugetlb_folio(int order, gfp_t gfp_mask,	1844	static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask,
1873	int nid, nodemask_t nmask, nodemask_t node_alloc_noretry)	1845	int nid, nodemask_t nmask, nodemask_t node_alloc_noretry)
1874	{	1846	{
1875	struct folio *folio;	1847	struct folio *folio;
@@ -1925,10 +1897,10 @@ static struct folio only_alloc_fresh_hugetlb_folio(struct hstate h, · 49 unmodified lines
1925	nid = numa_mem_id();	1897	nid = numa_mem_id();
1926		1898
1927	if (order_is_gigantic(order))	1899	if (order_is_gigantic(order))
1928	folio = alloc_gigantic_folio(order, gfp_mask, nid, nmask);	1900	folio = alloc_gigantic_frozen_folio(order, gfp_mask, nid, nmask);
1929	else	1901	else
1930	folio = alloc_buddy_hugetlb_folio(order, gfp_mask, nid, nmask,	1902	folio = alloc_buddy_frozen_folio(order, gfp_mask, nid, nmask,
1931	node_alloc_noretry);	1903	node_alloc_noretry);
1932	if (folio)	1904	if (folio)
1933	init_new_hugetlb_folio(folio);	1905	init_new_hugetlb_folio(folio);
1934	return folio;	1906	return folio;
@@ -2106,8 +2078,9 @@ retry: · 171 unmodified lines
2106	h->max_huge_pages++;	2078	h->max_huge_pages++;
2107	goto out;	2079	goto out;
2108	}	2080	}
2109	} else	2081	} else {
2110	rc = 0;	2082	rc = 0;
		2083	}
2111		2084
2112	update_and_free_hugetlb_folio(h, folio, false);	2085	update_and_free_hugetlb_folio(h, folio, false);
2113	return rc;	2086	return rc;
@@ -2702,11 +2675,12 @@ void restore_reserve_on_error(struct hstate h, struct vm_area_struct vma, · 588 unmodified lines
2702	* be consumed on a subsequent allocation.	2675	* be consumed on a subsequent allocation.
2703	*/	2676	*/
2704	folio_set_hugetlb_restore_reserve(folio);	2677	folio_set_hugetlb_restore_reserve(folio);
2705	} else	2678	} else {
2706	/*	2679	/*
2707	* No reservation present, do nothing	2680	* No reservation present, do nothing
2708	*/	2681	*/
2709	vma_end_reservation(h, vma, address);	2682	vma_end_reservation(h, vma, address);
		2683	}
2710	}	2684	}
2711	}	2685	}
2712		2686
@@ -2836,23 +2810,62 @@ int isolate_or_dissolve_huge_folio(struct folio folio, struct list_head list) · 123 unmodified lines
2836	*/	2810	*/
2837	int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)	2811	int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn)
2838	{	2812	{
2839	struct folio *folio;	2813	unsigned long nr = 0;
		2814	struct page *page;
		2815	struct hstate *h;
		2816	LIST_HEAD(list);
2840	int ret = 0;	2817	int ret = 0;
2841		2818
2842	LIST_HEAD(isolate_list);	2819	/* Avoid pfn iterations if no free non-gigantic huge pages */
		2820	for_each_hstate(h) {
		2821	if (hstate_is_gigantic(h))
		2822	continue;
		2823
		2824	nr += h->free_huge_pages;
		2825	if (nr)
		2826	break;
		2827	}
		2828
		2829	if (!nr)
		2830	return 0;
2843		2831
2844	while (start_pfn < end_pfn) {	2832	while (start_pfn < end_pfn) {
2845	folio = pfn_folio(start_pfn);	2833	page = pfn_to_page(start_pfn);
		2834	nr = 1;
2846		2835
2847	/* Not to disrupt normal path by vainly holding hugetlb_lock */	2836	if (PageHuge(page) \|\| PageCompound(page)) {
2848	if (folio_test_hugetlb(folio) && !folio_ref_count(folio)) {	2837	struct folio *folio = page_folio(page);
2849	ret = alloc_and_dissolve_hugetlb_folio(folio, &isolate_list);	2838
2850	if (ret)	2839	nr = folio_nr_pages(folio) - folio_page_idx(folio, page);
2851	break;	2840
		2841	/*
		2842	* Don't disrupt normal path by vainly holding
		2843	* hugetlb_lock
		2844	*/
		2845	if (folio_test_hugetlb(folio) && !folio_ref_count(folio)) {
		2846	if (order_is_gigantic(folio_order(folio))) {
		2847	ret = -ENOMEM;
		2848	break;
		2849	}
		2850
		2851	ret = alloc_and_dissolve_hugetlb_folio(folio, &list);
		2852	if (ret)
		2853	break;
		2854
		2855	putback_movable_pages(&list);
		2856	}
		2857	} else if (PageBuddy(page)) {
		2858	/*
		2859	* Buddy order check without zone lock is unsafe and
		2860	* the order is maybe invalid, but race should be
		2861	* small, and the worst thing is skipping free hugetlb.
		2862	*/
		2863	const unsigned int order = buddy_order_unsafe(page);
2852		2864
2853	putback_movable_pages(&isolate_list);	2865	if (order <= MAX_PAGE_ORDER)
		2866	nr = 1UL << order;
2854	}	2867	}
2855	start_pfn++;	2868	start_pfn += nr;
2856	}	2869	}
2857		2870
2858	return ret;	2871	return ret;
@@ -3019,13 +3032,10 @@ struct folio alloc_hugetlb_folio(struct vm_area_struct vma, · 160 unmodified lines
3019		3032
3020	rsv_adjust = hugepage_subpool_put_pages(spool, 1);	3033	rsv_adjust = hugepage_subpool_put_pages(spool, 1);
3021	hugetlb_acct_memory(h, -rsv_adjust);	3034	hugetlb_acct_memory(h, -rsv_adjust);
3022	if (map_chg) {	3035	spin_lock_irq(&hugetlb_lock);
3023	spin_lock_irq(&hugetlb_lock);	3036	hugetlb_cgroup_uncharge_folio_rsvd(
3024	hugetlb_cgroup_uncharge_folio_rsvd(	3037	hstate_index(h), pages_per_huge_page(h), folio);
3025	hstate_index(h), pages_per_huge_page(h),	3038	spin_unlock_irq(&hugetlb_lock);
3026	folio);
3027	spin_unlock_irq(&hugetlb_lock);
3028	}
3029	}	3039	}
3030	}	3040	}
3031		3041
@@ -3425,6 +3435,13 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) · 393 unmodified lines
3425		3435
3426	folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid,	3436	folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid,
3427	&node_states[N_MEMORY], NULL);	3437	&node_states[N_MEMORY], NULL);
		3438	if (!folio && !list_empty(&folio_list) &&
		3439	hugetlb_vmemmap_optimizable_size(h)) {
		3440	prep_and_add_allocated_folios(h, &folio_list);
		3441	INIT_LIST_HEAD(&folio_list);
		3442	folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid,
		3443	&node_states[N_MEMORY], NULL);
		3444	}
3428	if (!folio)	3445	if (!folio)
3429	break;	3446	break;
3430	list_add(&folio->lru, &folio_list);	3447	list_add(&folio->lru, &folio_list);
@@ -4159,7 +4176,6 @@ static int __init hugetlb_init(void) · 728 unmodified lines
4159	}	4176	}
4160	}	4177	}
4161		4178
4162	hugetlb_cma_check();
4163	hugetlb_init_hstates();	4179	hugetlb_init_hstates();
4164	gather_bootmem_prealloc();	4180	gather_bootmem_prealloc();
4165	report_hugepages();	4181	report_hugepages();
@@ -4487,21 +4503,11 @@ void __init hugetlb_bootmem_set_nodes(void) · 321 unmodified lines
4487	}	4503	}
4488	}	4504	}
4489		4505
4490	static bool __hugetlb_bootmem_allocated __initdata;
4491
4492	bool __init hugetlb_bootmem_allocated(void)
4493	{
4494	return __hugetlb_bootmem_allocated;
4495	}
4496
4497	void __init hugetlb_bootmem_alloc(void)	4506	void __init hugetlb_bootmem_alloc(void)
4498	{	4507	{
4499	struct hstate *h;	4508	struct hstate *h;
4500	int i;	4509	int i;
4501		4510
4502	if (__hugetlb_bootmem_allocated)
4503	return;
4504
4505	hugetlb_bootmem_set_nodes();	4511	hugetlb_bootmem_set_nodes();
4506		4512
4507	for (i = 0; i < MAX_NUMNODES; i++)	4513	for (i = 0; i < MAX_NUMNODES; i++)
@@ -4515,8 +4521,6 @@ void __init hugetlb_bootmem_alloc(void) · 7 unmodified lines
4515	if (hstate_is_gigantic(h))	4521	if (hstate_is_gigantic(h))
4516	hugetlb_hstate_alloc_pages(h);	4522	hugetlb_hstate_alloc_pages(h);
4517	}	4523	}
4518
4519	__hugetlb_bootmem_allocated = true;
4520	}	4524	}
4521		4525
4522	/*	4526	/*
@@ -4718,10 +4722,12 @@ static void hugetlb_vm_op_open(struct vm_area_struct *vma) · 195 unmodified lines
4718	if (vma_lock->vma != vma) {	4722	if (vma_lock->vma != vma) {
4719	vma->vm_private_data = NULL;	4723	vma->vm_private_data = NULL;
4720	hugetlb_vma_lock_alloc(vma);	4724	hugetlb_vma_lock_alloc(vma);
4721	} else	4725	} else {
4722	pr_warn("HugeTLB: vma_lock already exists in %s.\n", __func__);	4726	pr_warn("HugeTLB: vma_lock already exists in %s.\n", __func__);
4723	} else	4727	}
		4728	} else {
4724	hugetlb_vma_lock_alloc(vma);	4729	hugetlb_vma_lock_alloc(vma);
		4730	}
4725	}	4731	}
4726	}	4732	}
4727		4733

mm/hugetlb_cgroup.c+2−9

+// SPDX-License-Identifier: LGPL-2.1
 /*
  *
  * Copyright IBM Corporation, 2012
  * Copyright (C) 2019 Red Hat, Inc.
  * Author: Giuseppe Scrivano <gscrivan@redhat.com>
  *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of version 2.1 of the GNU Lesser General Public License
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- *
  */
 #include <linux/cgroup.h>
 	for (i = 0; i < tmpl_size; cft++, tmpl++, i++) {
 		*cft = *tmpl;
 		/* rebuild the name */
-		snprintf(cft->name, MAX_CFTYPE_NAME, "%s.%s", buf, tmpl->name);
+		scnprintf(cft->name, MAX_CFTYPE_NAME, "%s.%s", buf, tmpl->name);
 		/* rebuild the private */
 		cft->private = MEMFILE_PRIVATE(idx, tmpl->private);
 		/* rebuild the file_offset */

mm/hugetlb_cma.c+37−34

@@ -13,42 +13,46 @@ · 12 unmodified lines
13	#include "hugetlb_cma.h"	13	#include "hugetlb_cma.h"
14		14
15		15
16	static struct cma *hugetlb_cma[MAX_NUMNODES];	16	static struct cma *hugetlb_cma[MAX_NUMNODES] __ro_after_init;
17	static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata;	17	static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata;
18	static bool hugetlb_cma_only;	18	static bool hugetlb_cma_only __ro_after_init;
19	static unsigned long hugetlb_cma_size __initdata;	19	static unsigned long hugetlb_cma_size __ro_after_init;
20		20
21	void hugetlb_cma_free_folio(struct folio *folio)	21	void hugetlb_cma_free_frozen_folio(struct folio *folio)
22	{	22	{
23	int nid = folio_nid(folio);	23	WARN_ON_ONCE(!cma_release_frozen(hugetlb_cma[folio_nid(folio)],
24		24	&folio->page, folio_nr_pages(folio)));
25	WARN_ON_ONCE(!cma_free_folio(hugetlb_cma[nid], folio));
26	}	25	}
27		26
28		27	struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
29	struct folio *hugetlb_cma_alloc_folio(int order, gfp_t gfp_mask,	28	int nid, nodemask_t *nodemask)
30	int nid, nodemask_t *nodemask)
31	{	29	{
32	int node;	30	int node;
33	struct folio *folio = NULL;	31	struct folio *folio;
		32	struct page *page = NULL;
		33
		34	if (!hugetlb_cma_size)
		35	return NULL;
34		36
35	if (hugetlb_cma[nid])	37	if (hugetlb_cma[nid])
36	folio = cma_alloc_folio(hugetlb_cma[nid], order, gfp_mask);	38	page = cma_alloc_frozen_compound(hugetlb_cma[nid], order);
37		39
38	if (!folio && !(gfp_mask & __GFP_THISNODE)) {	40	if (!page && !(gfp_mask & __GFP_THISNODE)) {
39	for_each_node_mask(node, *nodemask) {	41	for_each_node_mask(node, *nodemask) {
40	if (node == nid \|\| !hugetlb_cma[node])	42	if (node == nid \|\| !hugetlb_cma[node])
41	continue;	43	continue;
42		44
43	folio = cma_alloc_folio(hugetlb_cma[node], order, gfp_mask);	45	page = cma_alloc_frozen_compound(hugetlb_cma[node], order);
44	if (folio)	46	if (page)
45	break;	47	break;
46	}	48	}
47	}	49	}
48		50
49	if (folio)	51	if (!page)
50	folio_set_hugetlb_cma(folio);	52	return NULL;
51		53
		54	folio = page_folio(page);
		55	folio_set_hugetlb_cma(folio);
52	return folio;	56	return folio;
53	}	57	}
54		58
@@ -85,9 +89,6 @@ hugetlb_cma_alloc_bootmem(struct hstate h, int nid, bool node_exact) · 30 unmodified lines
85	return m;	89	return m;
86	}	90	}
87		91
88
89	static bool cma_reserve_called __initdata;
90
91	static int __init cmdline_parse_hugetlb_cma(char *p)	92	static int __init cmdline_parse_hugetlb_cma(char *p)
92	{	93	{
93	int nid, count = 0;	94	int nid, count = 0;
@@ -134,12 +135,26 @@ static int __init cmdline_parse_hugetlb_cma_only(char *p) · 40 unmodified lines
134		135
135	early_param("hugetlb_cma_only", cmdline_parse_hugetlb_cma_only);	136	early_param("hugetlb_cma_only", cmdline_parse_hugetlb_cma_only);
136		137
137	void __init hugetlb_cma_reserve(int order)	138	unsigned int __weak arch_hugetlb_cma_order(void)
138	{	139	{
139	unsigned long size, reserved, per_node;	140	return 0;
		141	}
		142
		143	void __init hugetlb_cma_reserve(void)
		144	{
		145	unsigned long size, reserved, per_node, order;
140	bool node_specific_cma_alloc = false;	146	bool node_specific_cma_alloc = false;
141	int nid;	147	int nid;
142		148
		149	if (!hugetlb_cma_size)
		150	return;
		151
		152	order = arch_hugetlb_cma_order();
		153	if (!order) {
		154	pr_warn("hugetlb_cma: the option isn't supported by current arch\n");
		155	return;
		156	}
		157
143	/*	158	/*
144	* HugeTLB CMA reservation is required for gigantic	159	* HugeTLB CMA reservation is required for gigantic
145	* huge pages which could not be allocated via the	160	* huge pages which could not be allocated via the
@@ -147,10 +162,6 @@ void __init hugetlb_cma_reserve(int order) · 1 unmodified line
147	* breaking this assumption.	162	* breaking this assumption.
148	*/	163	*/
149	VM_WARN_ON(order <= MAX_PAGE_ORDER);	164	VM_WARN_ON(order <= MAX_PAGE_ORDER);
150	cma_reserve_called = true;
151
152	if (!hugetlb_cma_size)
153	return;
154		165
155	hugetlb_bootmem_set_nodes();	166	hugetlb_bootmem_set_nodes();
156		167
@@ -244,14 +255,6 @@ void __init hugetlb_cma_reserve(int order) · 87 unmodified lines
244	hugetlb_cma_size = 0;	255	hugetlb_cma_size = 0;
245	}	256	}
246		257
247	void __init hugetlb_cma_check(void)
248	{
249	if (!hugetlb_cma_size \|\| cma_reserve_called)
250	return;
251
252	pr_warn("hugetlb_cma: the option isn't supported by current arch\n");
253	}
254
255	bool hugetlb_cma_exclusive_alloc(void)	258	bool hugetlb_cma_exclusive_alloc(void)
256	{	259	{
257	return hugetlb_cma_only;	260	return hugetlb_cma_only;

mm/hugetlb_cma.h+5−10

 #define _LINUX_HUGETLB_CMA_H
 #ifdef CONFIG_CMA
-void hugetlb_cma_free_folio(struct folio *folio);
+void hugetlb_cma_free_frozen_folio(struct folio *folio);
-struct folio *hugetlb_cma_alloc_folio(int order, gfp_t gfp_mask,
+struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask,
 				      int nid, nodemask_t *nodemask);
 struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int *nid,
 						    bool node_exact);
-void hugetlb_cma_check(void);
 bool hugetlb_cma_exclusive_alloc(void);
 unsigned long hugetlb_cma_total_size(void);
 void hugetlb_cma_validate_params(void);
 bool hugetlb_early_cma(struct hstate *h);
 #else
-static inline void hugetlb_cma_free_folio(struct folio *folio)
+static inline void hugetlb_cma_free_frozen_folio(struct folio *folio)
 {
 }
-static inline struct folio *hugetlb_cma_alloc_folio(int order, gfp_t gfp_mask,
+static inline struct folio *hugetlb_cma_alloc_frozen_folio(int order,
-		int nid, nodemask_t *nodemask)
+		gfp_t gfp_mask,	int nid, nodemask_t *nodemask)
 {
 	return NULL;
 }
 	return NULL;
 }
-static inline void hugetlb_cma_check(void)
-{
-}
 static inline bool hugetlb_cma_exclusive_alloc(void)
 {
 	return false;

mm/hugetlb_sysctl.c+11−0

 #include "hugetlb_internal.h"
+int movable_gigantic_pages;
 #ifdef CONFIG_SYSCTL
 static int proc_hugetlb_doulongvec_minmax(const struct ctl_table *table, int write,
 					  void *buffer, size_t *length,
 		.mode		= 0644,
 		.proc_handler	= hugetlb_overcommit_handler,
 	},
+#ifdef CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION
+	{
+		.procname	= "movable_gigantic_pages",
+		.data		= &movable_gigantic_pages,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec,
+	},
+#endif
 };
 void __init hugetlb_sysctl_init(void)

mm/hugetlb_vmemmap.c+0−11

 	struct huge_bootmem_page *m = NULL;
 	void *map;
-	/*
-	 * Noting to do if bootmem pages were not allocated
-	 * early in boot, or if HVO wasn't enabled in the
-	 * first place.
-	 */
-	if (!hugetlb_bootmem_allocated())
-		return;
 	if (!READ_ONCE(vmemmap_optimize_enabled))
 		return;
 	struct hstate *h;
 	void *map;
-	if (!hugetlb_bootmem_allocated())
-		return;
 	if (!READ_ONCE(vmemmap_optimize_enabled))
 		return;

mm/internal.h+87−19

@@ -171,7 +171,7 @@ static inline int mmap_file(struct file file, struct vm_area_struct vma) · 170 unmodified lines
171		171
172	/*	172	/*
173	* OK, we tried to call the file hook for mmap(), but an error	173	* OK, we tried to call the file hook for mmap(), but an error
174	* arose. The mapping is in an inconsistent state and we most not invoke	174	* arose. The mapping is in an inconsistent state and we must not invoke
175	* any further hooks on it.	175	* any further hooks on it.
176	*/	176	*/
177	vma->vm_ops = &vma_dummy_vm_ops;	177	vma->vm_ops = &vma_dummy_vm_ops;
@@ -199,6 +199,73 @@ static inline void vma_close(struct vm_area_struct *vma) · 21 unmodified lines
199		199
200	#ifdef CONFIG_MMU	200	#ifdef CONFIG_MMU
201		201
		202	static inline void get_anon_vma(struct anon_vma *anon_vma)
		203	{
		204	atomic_inc(&anon_vma->refcount);
		205	}
		206
		207	void __put_anon_vma(struct anon_vma *anon_vma);
		208
		209	static inline void put_anon_vma(struct anon_vma *anon_vma)
		210	{
		211	if (atomic_dec_and_test(&anon_vma->refcount))
		212	__put_anon_vma(anon_vma);
		213	}
		214
		215	static inline void anon_vma_lock_write(struct anon_vma *anon_vma)
		216	{
		217	down_write(&anon_vma->root->rwsem);
		218	}
		219
		220	static inline int anon_vma_trylock_write(struct anon_vma *anon_vma)
		221	{
		222	return down_write_trylock(&anon_vma->root->rwsem);
		223	}
		224
		225	static inline void anon_vma_unlock_write(struct anon_vma *anon_vma)
		226	{
		227	up_write(&anon_vma->root->rwsem);
		228	}
		229
		230	static inline void anon_vma_lock_read(struct anon_vma *anon_vma)
		231	{
		232	down_read(&anon_vma->root->rwsem);
		233	}
		234
		235	static inline int anon_vma_trylock_read(struct anon_vma *anon_vma)
		236	{
		237	return down_read_trylock(&anon_vma->root->rwsem);
		238	}
		239
		240	static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)
		241	{
		242	up_read(&anon_vma->root->rwsem);
		243	}
		244
		245	struct anon_vma folio_get_anon_vma(const struct folio folio);
		246
		247	/* Operations which modify VMAs. */
		248	enum vma_operation {
		249	VMA_OP_SPLIT,
		250	VMA_OP_MERGE_UNFAULTED,
		251	VMA_OP_REMAP,
		252	VMA_OP_FORK,
		253	};
		254
		255	int anon_vma_clone(struct vm_area_struct dst, struct vm_area_struct src,
		256	enum vma_operation operation);
		257	int anon_vma_fork(struct vm_area_struct vma, struct vm_area_struct pvma);
		258	int __anon_vma_prepare(struct vm_area_struct *vma);
		259	void unlink_anon_vmas(struct vm_area_struct *vma);
		260
		261	static inline int anon_vma_prepare(struct vm_area_struct *vma)
		262	{
		263	if (likely(vma->anon_vma))
		264	return 0;
		265
		266	return __anon_vma_prepare(vma);
		267	}
		268
202	/* Flags for folio_pte_batch(). */	269	/* Flags for folio_pte_batch(). */
203	typedef int __bitwise fpb_t;	270	typedef int __bitwise fpb_t;
204		271
@@ -513,6 +580,14 @@ static inline void set_page_refcounted(struct page *page) · 308 unmodified lines
513	set_page_count(page, 1);	580	set_page_count(page, 1);
514	}	581	}
515		582
		583	static inline void set_pages_refcounted(struct page *page, unsigned long nr_pages)
		584	{
		585	unsigned long pfn = page_to_pfn(page);
		586
		587	for (; nr_pages--; pfn++)
		588	set_page_refcounted(pfn_to_page(pfn));
		589	}
		590
516	/*	591	/*
517	* Return true if a folio needs ->release_folio() calling upon it.	592	* Return true if a folio needs ->release_folio() calling upon it.
518	*/	593	*/
@@ -853,6 +928,12 @@ void memmap_init_range(unsigned long, int, unsigned long, unsigned long, · 334 unmodified lines
853	unsigned long, enum meminit_context, struct vmem_altmap *, int,	928	unsigned long, enum meminit_context, struct vmem_altmap *, int,
854	bool);	929	bool);
855		930
		931	#ifdef CONFIG_SPARSEMEM
		932	void sparse_init(void);
		933	#else
		934	static inline void sparse_init(void) {}
		935	#endif /* CONFIG_SPARSEMEM */
		936
856	#if defined CONFIG_COMPACTION \|\| defined CONFIG_CMA	937	#if defined CONFIG_COMPACTION \|\| defined CONFIG_CMA
857		938
858	/*	939	/*
@@ -929,9 +1010,14 @@ void init_cma_reserved_pageblock(struct page *page); · 70 unmodified lines
929	struct cma;	1010	struct cma;
930		1011
931	#ifdef CONFIG_CMA	1012	#ifdef CONFIG_CMA
		1013	bool cma_validate_zones(struct cma *cma);
932	void cma_reserve_early(struct cma cma, unsigned long size);	1014	void cma_reserve_early(struct cma cma, unsigned long size);
933	void init_cma_pageblock(struct page *page);	1015	void init_cma_pageblock(struct page *page);
934	#else	1016	#else
		1017	static inline bool cma_validate_zones(struct cma *cma)
		1018	{
		1019	return false;
		1020	}
935	static inline void cma_reserve_early(struct cma cma, unsigned long size)	1021	static inline void cma_reserve_early(struct cma cma, unsigned long size)
936	{	1022	{
937	return NULL;	1023	return NULL;
@@ -1658,24 +1744,6 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, · 720 unmodified lines
1658	unsigned long end, const struct mm_walk_ops *ops,	1744	unsigned long end, const struct mm_walk_ops *ops,
1659	pgd_t pgd, void private);	1745	pgd_t pgd, void private);
1660		1746
1661	/* pt_reclaim.c */
1662	bool try_get_and_clear_pmd(struct mm_struct mm, pmd_t pmd, pmd_t *pmdval);
1663	void free_pte(struct mm_struct mm, unsigned long addr, struct mmu_gather tlb,
1664	pmd_t pmdval);
1665	void try_to_free_pte(struct mm_struct mm, pmd_t pmd, unsigned long addr,
1666	struct mmu_gather *tlb);
1667
1668	#ifdef CONFIG_PT_RECLAIM
1669	bool reclaim_pt_is_enabled(unsigned long start, unsigned long end,
1670	struct zap_details *details);
1671	#else
1672	static inline bool reclaim_pt_is_enabled(unsigned long start, unsigned long end,
1673	struct zap_details *details)
1674	{
1675	return false;
1676	}
1677	#endif /* CONFIG_PT_RECLAIM */
1678
1679	void dup_mm_exe_file(struct mm_struct mm, struct mm_struct oldmm);	1747	void dup_mm_exe_file(struct mm_struct mm, struct mm_struct oldmm);
1680	int dup_mmap(struct mm_struct mm, struct mm_struct oldmm);	1748	int dup_mmap(struct mm_struct mm, struct mm_struct oldmm);
1681		1749

mm/kasan/kasan_test_c.c+35−15

@@ -1840,6 +1840,29 @@ static void vmalloc_helpers_tags(struct kunit *test) · 1839 unmodified lines
1840	vfree(ptr);	1840	vfree(ptr);
1841	}	1841	}
1842		1842
		1843	static void vmalloc_oob_helper(struct kunit test, char v_ptr, size_t size)
		1844	{
		1845	/*
		1846	* We have to be careful not to hit the guard page in vmalloc tests.
		1847	* The MMU will catch that and crash us.
		1848	*/
		1849
		1850	/* Make sure in-bounds accesses are valid. */
		1851	v_ptr[0] = 0;
		1852	v_ptr[size - 1] = 0;
		1853
		1854	/*
		1855	* An unaligned access past the requested vmalloc size.
		1856	* Only generic KASAN can precisely detect these.
		1857	*/
		1858	if (IS_ENABLED(CONFIG_KASAN_GENERIC))
		1859	KUNIT_EXPECT_KASAN_FAIL(test, ((volatile char *)v_ptr)[size]);
		1860
		1861	/* An aligned access into the first out-of-bounds granule. */
		1862	size = round_up(size, KASAN_GRANULE_SIZE);
		1863	KUNIT_EXPECT_KASAN_FAIL_READ(test, ((volatile char *)v_ptr)[size]);
		1864	}
		1865
1843	static void vmalloc_oob(struct kunit *test)	1866	static void vmalloc_oob(struct kunit *test)
1844	{	1867	{
1845	char v_ptr, p_ptr;	1868	char v_ptr, p_ptr;
@@ -1856,24 +1879,21 @@ static void vmalloc_oob(struct kunit *test) · 10 unmodified lines
1856		1879
1857	OPTIMIZER_HIDE_VAR(v_ptr);	1880	OPTIMIZER_HIDE_VAR(v_ptr);
1858		1881
1859	/*	1882	vmalloc_oob_helper(test, v_ptr, size);
1860	* We have to be careful not to hit the guard page in vmalloc tests.
1861	* The MMU will catch that and crash us.
1862	*/
1863		1883
1864	/* Make sure in-bounds accesses are valid. */	1884	size -= KASAN_GRANULE_SIZE + 1;
1865	v_ptr[0] = 0;	1885	v_ptr = vrealloc(v_ptr, size, GFP_KERNEL);
1866	v_ptr[size - 1] = 0;	1886	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, v_ptr);
1867		1887
1868	/*	1888	OPTIMIZER_HIDE_VAR(v_ptr);
1869	* An unaligned access past the requested vmalloc size.
1870	* Only generic KASAN can precisely detect these.
1871	*/
1872	if (IS_ENABLED(CONFIG_KASAN_GENERIC))
1873	KUNIT_EXPECT_KASAN_FAIL(test, ((volatile char *)v_ptr)[size]);
1874		1889
1875	/* An aligned access into the first out-of-bounds granule. */	1890	vmalloc_oob_helper(test, v_ptr, size);
1876	KUNIT_EXPECT_KASAN_FAIL_READ(test, ((volatile char *)v_ptr)[size + 5]);	1891
		1892	size += 2 * KASAN_GRANULE_SIZE + 2;
		1893	v_ptr = vrealloc(v_ptr, size, GFP_KERNEL);
		1894	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, v_ptr);
		1895
		1896	vmalloc_oob_helper(test, v_ptr, size);
1877		1897
1878	/* Check that in-bounds accesses to the physical page are valid. */	1898	/* Check that in-bounds accesses to the physical page are valid. */
1879	page = vmalloc_to_page(v_ptr);	1899	page = vmalloc_to_page(v_ptr);

mm/kasan/report.c+4−4

@@ -203,7 +203,7 @@ static inline void fail_non_kasan_kunit_test(void) { } · 202 unmodified lines
203		203
204	static DEFINE_RAW_SPINLOCK(report_lock);	204	static DEFINE_RAW_SPINLOCK(report_lock);
205		205
206	static void start_report(unsigned long *flags, bool sync)	206	static void start_report(unsigned long *flags)
207	{	207	{
208	fail_non_kasan_kunit_test();	208	fail_non_kasan_kunit_test();
209	/* Respect the /proc/sys/kernel/traceoff_on_warning interface. */	209	/* Respect the /proc/sys/kernel/traceoff_on_warning interface. */
@@ -543,7 +543,7 @@ void kasan_report_invalid_free(void *ptr, unsigned long ip, enum kasan_report_ty · 333 unmodified lines
543	if (unlikely(!report_enabled()))	543	if (unlikely(!report_enabled()))
544	return;	544	return;
545		545
546	start_report(&flags, true);	546	start_report(&flags);
547		547
548	__memset(&info, 0, sizeof(info));	548	__memset(&info, 0, sizeof(info));
549	info.type = type;	549	info.type = type;
@@ -581,7 +581,7 @@ bool kasan_report(const void *addr, size_t size, bool is_write, · 31 unmodified lines
581	goto out;	581	goto out;
582	}	582	}
583		583
584	start_report(&irq_flags, true);	584	start_report(&irq_flags);
585		585
586	__memset(&info, 0, sizeof(info));	586	__memset(&info, 0, sizeof(info));
587	info.type = KASAN_REPORT_ACCESS;	587	info.type = KASAN_REPORT_ACCESS;
@@ -615,7 +615,7 @@ void kasan_report_async(void) · 27 unmodified lines
615	if (unlikely(!report_enabled()))	615	if (unlikely(!report_enabled()))
616	return;	616	return;
617		617
618	start_report(&flags, false);	618	start_report(&flags);
619	pr_err("BUG: KASAN: invalid-access\n");	619	pr_err("BUG: KASAN: invalid-access\n");
620	pr_err("Asynchronous fault: no details available\n");	620	pr_err("Asynchronous fault: no details available\n");
621	pr_err("\n");	621	pr_err("\n");

mm/kasan/shadow.c+4−4

@@ -305,7 +305,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 304 unmodified lines
305	pte_t pte;	305	pte_t pte;
306	int index;	306	int index;
307		307
308	arch_leave_lazy_mmu_mode();	308	lazy_mmu_mode_pause();
309		309
310	index = PFN_DOWN(addr - data->start);	310	index = PFN_DOWN(addr - data->start);
311	page = data->pages[index];	311	page = data->pages[index];
@@ -319,7 +319,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 7 unmodified lines
319	}	319	}
320	spin_unlock(&init_mm.page_table_lock);	320	spin_unlock(&init_mm.page_table_lock);
321		321
322	arch_enter_lazy_mmu_mode();	322	lazy_mmu_mode_resume();
323		323
324	return 0;	324	return 0;
325	}	325	}
@@ -471,7 +471,7 @@ static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 145 unmodified lines
471	pte_t pte;	471	pte_t pte;
472	int none;	472	int none;
473		473
474	arch_leave_lazy_mmu_mode();	474	lazy_mmu_mode_pause();
475		475
476	spin_lock(&init_mm.page_table_lock);	476	spin_lock(&init_mm.page_table_lock);
477	pte = ptep_get(ptep);	477	pte = ptep_get(ptep);
@@ -483,7 +483,7 @@ static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 5 unmodified lines
483	if (likely(!none))	483	if (likely(!none))
484	__free_page(pfn_to_page(pte_pfn(pte)));	484	__free_page(pfn_to_page(pte_pfn(pte)));
485		485
486	arch_enter_lazy_mmu_mode();	486	lazy_mmu_mode_resume();
487		487
488	return 0;	488	return 0;
489	}	489	}

mm/kfence/core.c+3−3

@@ -905,7 +905,7 @@ static void toggle_allocation_gate(struct work_struct *work) · 904 unmodified lines
905	/* Disable static key and reset timer. */	905	/* Disable static key and reset timer. */
906	static_branch_disable(&kfence_allocation_key);	906	static_branch_disable(&kfence_allocation_key);
907	#endif	907	#endif
908	queue_delayed_work(system_unbound_wq, &kfence_timer,	908	queue_delayed_work(system_dfl_wq, &kfence_timer,
909	msecs_to_jiffies(kfence_sample_interval));	909	msecs_to_jiffies(kfence_sample_interval));
910	}	910	}
911		911
@@ -955,7 +955,7 @@ static void kfence_init_enable(void) · 43 unmodified lines
955	#endif	955	#endif
956		956
957	WRITE_ONCE(kfence_enabled, true);	957	WRITE_ONCE(kfence_enabled, true);
958	queue_delayed_work(system_unbound_wq, &kfence_timer, 0);	958	queue_delayed_work(system_dfl_wq, &kfence_timer, 0);
959		959
960	pr_info("initialized - using %lu bytes for %d objects at 0x%p-0x%p\n", KFENCE_POOL_SIZE,	960	pr_info("initialized - using %lu bytes for %d objects at 0x%p-0x%p\n", KFENCE_POOL_SIZE,
961	CONFIG_KFENCE_NUM_OBJECTS, (void *)__kfence_pool,	961	CONFIG_KFENCE_NUM_OBJECTS, (void *)__kfence_pool,
@@ -1051,7 +1051,7 @@ static int kfence_enable_late(void) · 89 unmodified lines
1051	return kfence_init_late();	1051	return kfence_init_late();
1052		1052
1053	WRITE_ONCE(kfence_enabled, true);	1053	WRITE_ONCE(kfence_enabled, true);
1054	queue_delayed_work(system_unbound_wq, &kfence_timer, 0);	1054	queue_delayed_work(system_dfl_wq, &kfence_timer, 0);
1055	pr_info("re-enabled\n");	1055	pr_info("re-enabled\n");
1056	return 0;	1056	return 0;
1057	}	1057	}

mm/khugepaged.c+102−83

 #include <linux/dax.h>
 #include <linux/ksm.h>
 #include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
 #include <asm/tlb.h>
 #include "internal.h"
 	SCAN_STORE_FAILED,
 	SCAN_COPY_MC,
 	SCAN_PAGE_FILLED,
+	SCAN_PAGE_DIRTY_OR_WRITEBACK,
 };
 #define CREATE_TRACE_POINTS
 	}
 }
-static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
+static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
-					unsigned long start_addr,
+		unsigned long start_addr, pte_t *pte, struct collapse_control *cc,
-					pte_t *pte,
+		struct list_head *compound_pagelist)
-					struct collapse_control *cc,
-					struct list_head *compound_pagelist)
 {
 	struct page *page = NULL;
 	struct folio *folio = NULL;
 	unsigned long addr = start_addr;
 	pte_t *_pte;
-	int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0;
+	int none_or_zero = 0, shared = 0, referenced = 0;
+	enum scan_result result = SCAN_FAIL;
 	for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
 	     _pte++, addr += PAGE_SIZE) {
  * @ptl: lock on raw pages' PTEs
  * @compound_pagelist: list that stores compound pages
  */
-static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio,
+static enum scan_result __collapse_huge_page_copy(pte_t *pte, struct folio *folio,
 		pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma,
 		unsigned long address, spinlock_t *ptl,
 		struct list_head *compound_pagelist)
 {
 	unsigned int i;
-	int result = SCAN_SUCCEED;
+	enum scan_result result = SCAN_SUCCEED;
 	/*
 	 * Copying pages' contents is subject to memory poison at any iteration.
 	remove_wait_queue(&khugepaged_wait, &wait);
 }
-struct collapse_control khugepaged_collapse_control = {
+static struct collapse_control khugepaged_collapse_control = {
 	.is_khugepaged = true,
 };
  * Returns enum scan_result value.
  */
-static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
+static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
-				   bool expect_anon,
+		bool expect_anon, struct vm_area_struct **vmap, struct collapse_control *cc)
-				   struct vm_area_struct **vmap,
-				   struct collapse_control *cc)
 {
 	struct vm_area_struct *vma;
 	enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED :
 	return SCAN_SUCCEED;
 }
-static inline int check_pmd_state(pmd_t *pmd)
+static inline enum scan_result check_pmd_state(pmd_t *pmd)
 {
 	pmd_t pmde = pmdp_get_lockless(pmd);
 	return SCAN_SUCCEED;
 }
-static int find_pmd_or_thp_or_none(struct mm_struct *mm,
+static enum scan_result find_pmd_or_thp_or_none(struct mm_struct *mm,
-				   unsigned long address,
+		unsigned long address, pmd_t **pmd)
-				   pmd_t **pmd)
 {
 	*pmd = mm_find_pmd(mm, address);
 	if (!*pmd)
 	return check_pmd_state(*pmd);
 }
-static int check_pmd_still_valid(struct mm_struct *mm,
+static enum scan_result check_pmd_still_valid(struct mm_struct *mm,
-				 unsigned long address,
+		unsigned long address, pmd_t *pmd)
-				 pmd_t *pmd)
 {
 	pmd_t *new_pmd;
-	int result = find_pmd_or_thp_or_none(mm, address, &new_pmd);
+	enum scan_result result = find_pmd_or_thp_or_none(mm, address, &new_pmd);
 	if (result != SCAN_SUCCEED)
 		return result;
  * Called and returns without pte mapped or spinlocks held.
  * Returns result: if not SCAN_SUCCEED, mmap_lock has been released.
  */
-static int __collapse_huge_page_swapin(struct mm_struct *mm,
+static enum scan_result __collapse_huge_page_swapin(struct mm_struct *mm,
-				       struct vm_area_struct *vma,
+		struct vm_area_struct *vma, unsigned long start_addr, pmd_t *pmd,
-				       unsigned long start_addr, pmd_t *pmd,
+		int referenced)
-				       int referenced)
 {
 	int swapped_in = 0;
 	vm_fault_t ret = 0;
 	unsigned long addr, end = start_addr + (HPAGE_PMD_NR * PAGE_SIZE);
-	int result;
+	enum scan_result result;
 	pte_t *pte = NULL;
 	spinlock_t *ptl;
 	return result;
 }
-static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,
+static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_struct *mm,
-			      struct collapse_control *cc)
+		struct collapse_control *cc)
 {
 	gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
 		     GFP_TRANSHUGE);
 	return SCAN_SUCCEED;
 }
-static int collapse_huge_page(struct mm_struct *mm, unsigned long address,
+static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long address,
-			      int referenced, int unmapped,
+		int referenced, int unmapped, struct collapse_control *cc)
-			      struct collapse_control *cc)
 {
 	LIST_HEAD(compound_pagelist);
 	pmd_t *pmd, _pmd;
 	pgtable_t pgtable;
 	struct folio *folio;
 	spinlock_t *pmd_ptl, *pte_ptl;
-	int result = SCAN_FAIL;
+	enum scan_result result = SCAN_FAIL;
 	struct vm_area_struct *vma;
 	struct mmu_notifier_range range;
 	return result;
 }
-static int hpage_collapse_scan_pmd(struct mm_struct *mm,
+static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm,
-				   struct vm_area_struct *vma,
+		struct vm_area_struct *vma, unsigned long start_addr, bool *mmap_locked,
-				   unsigned long start_addr, bool *mmap_locked,
+		struct collapse_control *cc)
-				   struct collapse_control *cc)
 {
 	pmd_t *pmd;
 	pte_t *pte, *_pte;
-	int result = SCAN_FAIL, referenced = 0;
+	int none_or_zero = 0, shared = 0, referenced = 0;
-	int none_or_zero = 0, shared = 0;
+	enum scan_result result = SCAN_FAIL;
 	struct page *page = NULL;
 	struct folio *folio = NULL;
 	unsigned long addr;
 }
 /* folio must be locked, and mmap_lock must be held */
-static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,
+static enum scan_result set_huge_pmd(struct vm_area_struct *vma, unsigned long addr,
-			pmd_t *pmdp, struct folio *folio, struct page *page)
+		pmd_t *pmdp, struct folio *folio, struct page *page)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	struct vm_fault vmf = {
 	return SCAN_SUCCEED;
 }
-/**
+static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
- * collapse_pte_mapped_thp - Try to collapse a pte-mapped THP for mm at
+		bool install_pmd)
- * address haddr.
- *
- * @mm: process address space where collapse happens
- * @addr: THP collapse address
- * @install_pmd: If a huge PMD should be installed
- *
- * This function checks whether all the PTEs in the PMD are pointing to the
- * right THP. If so, retract the page table so the THP can refault in with
- * as pmd-mapped. Possibly install a huge PMD mapping the THP.
- */
-int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
-			    bool install_pmd)
 {
-	int nr_mapped_ptes = 0, result = SCAN_FAIL;
+	enum scan_result result = SCAN_FAIL;
+	int nr_mapped_ptes = 0;
 	unsigned int nr_batch_ptes;
 	struct mmu_notifier_range range;
 	bool notified = false;
 	return result;
 }
+/**
+ * collapse_pte_mapped_thp - Try to collapse a pte-mapped THP for mm at
+ * address haddr.
+ *
+ * @mm: process address space where collapse happens
+ * @addr: THP collapse address
+ * @install_pmd: If a huge PMD should be installed
+ *
+ * This function checks whether all the PTEs in the PMD are pointing to the
+ * right THP. If so, retract the page table so the THP can refault in with
+ * as pmd-mapped. Possibly install a huge PMD mapping the THP.
+ */
+void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr,
+		bool install_pmd)
+{
+	try_collapse_pte_mapped_thp(mm, addr, install_pmd);
+}
 /* Can we retract page tables for this file-backed VMA? */
 static bool file_backed_vma_is_retractable(struct vm_area_struct *vma)
 {
  *    + unlock old pages
  *    + unlock and free huge page;
  */
-static int collapse_file(struct mm_struct *mm, unsigned long addr,
+static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr,
-			 struct file *file, pgoff_t start,
+		struct file *file, pgoff_t start, struct collapse_control *cc)
-			 struct collapse_control *cc)
 {
 	struct address_space *mapping = file->f_mapping;
 	struct page *dst;
 	pgoff_t index = 0, end = start + HPAGE_PMD_NR;
 	LIST_HEAD(pagelist);
 	XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER);
-	int nr_none = 0, result = SCAN_SUCCEED;
+	enum scan_result result = SCAN_SUCCEED;
+	int nr_none = 0;
 	bool is_shmem = shmem_file(file);
 	VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem);
 				 */
 				xas_unlock_irq(&xas);
 				filemap_flush(mapping);
-				result = SCAN_FAIL;
+				result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
 				goto xa_unlocked;
 			} else if (folio_test_writeback(folio)) {
 				xas_unlock_irq(&xas);
-				result = SCAN_FAIL;
+				result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
 				goto xa_unlocked;
 			} else if (folio_trylock(folio)) {
 				folio_get(folio);
 			 * folio is dirty because it hasn't been flushed
 			 * since first write.
 			 */
-			result = SCAN_FAIL;
+			result = SCAN_PAGE_DIRTY_OR_WRITEBACK;
 			goto out_unlock;
 		}
 		xas_lock_irq(&xas);
 	}
-	if (is_shmem)
+	if (is_shmem) {
+		lruvec_stat_mod_folio(new_folio, NR_SHMEM, HPAGE_PMD_NR);
 		lruvec_stat_mod_folio(new_folio, NR_SHMEM_THPS, HPAGE_PMD_NR);
-	else
+	} else {
 		lruvec_stat_mod_folio(new_folio, NR_FILE_THPS, HPAGE_PMD_NR);
-	if (nr_none) {
-		lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, nr_none);
-		/* nr_none is always 0 for non-shmem. */
-		lruvec_stat_mod_folio(new_folio, NR_SHMEM, nr_none);
 	}
+	lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, HPAGE_PMD_NR);
 	/*
 	 * Mark new_folio as uptodate before inserting it into the
 	/*
 	 * Remove pte page tables, so we can re-fault the page as huge.
-	 * If MADV_COLLAPSE, adjust result to call collapse_pte_mapped_thp().
+	 * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp().
 	 */
 	retract_page_tables(mapping, start);
 	if (cc && !cc->is_khugepaged)
 	 */
 	list_for_each_entry_safe(folio, tmp, &pagelist, lru) {
 		list_del(&folio->lru);
+		lruvec_stat_mod_folio(folio, NR_FILE_PAGES,
+				      -folio_nr_pages(folio));
+		if (is_shmem)
+			lruvec_stat_mod_folio(folio, NR_SHMEM,
+					      -folio_nr_pages(folio));
 		folio->mapping = NULL;
 		folio_clear_active(folio);
 		folio_clear_unevictable(folio);
 	return result;
 }
-static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,
+static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr,
-				    struct file *file, pgoff_t start,
+		struct file *file, pgoff_t start, struct collapse_control *cc)
-				    struct collapse_control *cc)
 {
 	struct folio *folio = NULL;
 	struct address_space *mapping = file->f_mapping;
 	XA_STATE(xas, &mapping->i_pages, start);
 	int present, swap;
 	int node = NUMA_NO_NODE;
-	int result = SCAN_SUCCEED;
+	enum scan_result result = SCAN_SUCCEED;
 	present = 0;
 	swap = 0;
 	return result;
 }
-static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result,
+static unsigned int khugepaged_scan_mm_slot(unsigned int pages, enum scan_result *result,
 					    struct collapse_control *cc)
 	__releases(&khugepaged_mm_lock)
 	__acquires(&khugepaged_mm_lock)
 			break;
 		}
 		if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_KHUGEPAGED, PMD_ORDER)) {
-skip:
 			progress++;
 			continue;
 		}
 		hstart = round_up(vma->vm_start, HPAGE_PMD_SIZE);
 		hend = round_down(vma->vm_end, HPAGE_PMD_SIZE);
-		if (khugepaged_scan.address > hend)
+		if (khugepaged_scan.address > hend) {
-			goto skip;
+			progress++;
+			continue;
+		}
 		if (khugepaged_scan.address < hstart)
 			khugepaged_scan.address = hstart;
 		VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK);
 					mmap_read_lock(mm);
 					if (hpage_collapse_test_exit_or_disable(mm))
 						goto breakouterloop;
-					*result = collapse_pte_mapped_thp(mm,
+					*result = try_collapse_pte_mapped_thp(mm,
 						khugepaged_scan.address, false);
 					if (*result == SCAN_PMD_MAPPED)
 						*result = SCAN_SUCCEED;
 	unsigned int progress = 0, pass_through_head = 0;
 	unsigned int pages = READ_ONCE(khugepaged_pages_to_scan);
 	bool wait = true;
-	int result = SCAN_SUCCEED;
+	enum scan_result result = SCAN_SUCCEED;
 	lru_add_drain_all();
 	case SCAN_PAGE_LRU:
 	case SCAN_DEL_PAGE_LRU:
 	case SCAN_PAGE_FILLED:
+	case SCAN_PAGE_DIRTY_OR_WRITEBACK:
 		return -EAGAIN;
 	/*
 	 * Other: Trying again likely not to succeed / error intrinsic to
 	struct collapse_control *cc;
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long hstart, hend, addr;
-	int thps = 0, last_fail = SCAN_FAIL;
+	enum scan_result last_fail = SCAN_FAIL;
+	int thps = 0;
 	bool mmap_locked = true;
 	BUG_ON(vma->vm_start > start);
 	hend = end & HPAGE_PMD_MASK;
 	for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
-		int result = SCAN_FAIL;
+		enum scan_result result = SCAN_FAIL;
+		bool triggered_wb = false;
+retry:
 		if (!mmap_locked) {
 			cond_resched();
 			mmap_read_lock(mm);
 			mmap_read_unlock(mm);
 			mmap_locked = false;
+			*lock_dropped = true;
 			result = hpage_collapse_scan_file(mm, addr, file, pgoff,
 							  cc);
+			if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+			    mapping_can_writeback(file->f_mapping)) {
+				loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+				loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+				filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+				triggered_wb = true;
+				fput(file);
+				goto retry;
+			}
 			fput(file);
 		} else {
 			result = hpage_collapse_scan_pmd(mm, vma, addr,
 		case SCAN_PTE_MAPPED_HUGEPAGE:
 			BUG_ON(mmap_locked);
 			mmap_read_lock(mm);
-			result = collapse_pte_mapped_thp(mm, addr, true);
+			result = try_collapse_pte_mapped_thp(mm, addr, true);
 			mmap_read_unlock(mm);
 			goto handle_result;
 		/* Whitelisted set of results where continuing OK */

mm/kmsan/kmsan_test.c+63−1

@@ -361,7 +361,7 @@ static void test_init_vmalloc(struct kunit *test) · 360 unmodified lines
361	KUNIT_EXPECT_TRUE(test, report_matches(&expect));	361	KUNIT_EXPECT_TRUE(test, report_matches(&expect));
362	}	362	}
363		363
364	/* Test case: ensure that use-after-free reporting works. */	364	/* Test case: ensure that use-after-free reporting works for kmalloc. */
365	static void test_uaf(struct kunit *test)	365	static void test_uaf(struct kunit *test)
366	{	366	{
367	EXPECTATION_USE_AFTER_FREE(expect);	367	EXPECTATION_USE_AFTER_FREE(expect);
@@ -378,6 +378,65 @@ static void test_uaf(struct kunit *test) · 10 unmodified lines
378	KUNIT_EXPECT_TRUE(test, report_matches(&expect));	378	KUNIT_EXPECT_TRUE(test, report_matches(&expect));
379	}	379	}
380		380
		381	static void test_uninit_page(struct kunit *test)
		382	{
		383	EXPECTATION_UNINIT_VALUE(expect);
		384	struct page *page;
		385	int *ptr;
		386
		387	kunit_info(test, "uninitialized page allocation (UMR report)\n");
		388	page = alloc_pages(GFP_KERNEL, 0);
		389	ptr = page_address(page);
		390	USE(*ptr);
		391	__free_pages(page, 0);
		392	KUNIT_EXPECT_TRUE(test, report_matches(&expect));
		393	}
		394
		395	static volatile char *test_uaf_pages_helper(int order, int offset)
		396	{
		397	struct page *page;
		398	volatile char *var;
		399
		400	/* Memory is initialized up until __free_pages() thanks to __GFP_ZERO. */
		401	page = alloc_pages(GFP_KERNEL \| __GFP_ZERO, order);
		402	var = page_address(page) + offset;
		403	__free_pages(page, order);
		404
		405	return var;
		406	}
		407
		408	/* Test case: ensure that use-after-free reporting works for a freed page. */
		409	static void test_uaf_pages(struct kunit *test)
		410	{
		411	EXPECTATION_USE_AFTER_FREE(expect);
		412	volatile char value;
		413
		414	kunit_info(test, "use-after-free on a freed page (UMR report)\n");
		415	/* Allocate a single page, free it, then try to access it. */
		416	value = *test_uaf_pages_helper(0, 3);
		417	USE(value);
		418
		419	KUNIT_EXPECT_TRUE(test, report_matches(&expect));
		420	}
		421
		422	/* Test case: ensure that UAF reporting works for high order pages. */
		423	static void test_uaf_high_order_pages(struct kunit *test)
		424	{
		425	EXPECTATION_USE_AFTER_FREE(expect);
		426	volatile char value;
		427
		428	kunit_info(test,
		429	"use-after-free on a freed high-order page (UMR report)\n");
		430	/*
		431	* Create a high-order non-compound page, free it, then try to access
		432	* its tail page.
		433	*/
		434	value = *test_uaf_pages_helper(1, PAGE_SIZE + 3);
		435	USE(value);
		436
		437	KUNIT_EXPECT_TRUE(test, report_matches(&expect));
		438	}
		439
381	/*	440	/*
382	* Test case: ensure that uninitialized values are propagated through per-CPU	441	* Test case: ensure that uninitialized values are propagated through per-CPU
383	* memory.	442	* memory.
@@ -682,7 +741,10 @@ static struct kunit_case kmsan_test_cases[] = { · 298 unmodified lines
682	KUNIT_CASE(test_uninit_kmsan_check_memory),	741	KUNIT_CASE(test_uninit_kmsan_check_memory),
683	KUNIT_CASE(test_init_kmsan_vmap_vunmap),	742	KUNIT_CASE(test_init_kmsan_vmap_vunmap),
684	KUNIT_CASE(test_init_vmalloc),	743	KUNIT_CASE(test_init_vmalloc),
		744	KUNIT_CASE(test_uninit_page),
685	KUNIT_CASE(test_uaf),	745	KUNIT_CASE(test_uaf),
		746	KUNIT_CASE(test_uaf_pages),
		747	KUNIT_CASE(test_uaf_high_order_pages),
686	KUNIT_CASE(test_percpu_propagate),	748	KUNIT_CASE(test_percpu_propagate),
687	KUNIT_CASE(test_printk),	749	KUNIT_CASE(test_printk),
688	KUNIT_CASE(test_init_memcpy),	750	KUNIT_CASE(test_init_memcpy),

mm/list_lru.c+3−3

@@ -187,7 +187,7 @@ bool list_lru_add_obj(struct list_lru lru, struct list_head item) · 186 unmodified lines
187		187
188	if (list_lru_memcg_aware(lru)) {	188	if (list_lru_memcg_aware(lru)) {
189	rcu_read_lock();	189	rcu_read_lock();
190	ret = list_lru_add(lru, item, nid, mem_cgroup_from_slab_obj(item));	190	ret = list_lru_add(lru, item, nid, mem_cgroup_from_virt(item));
191	rcu_read_unlock();	191	rcu_read_unlock();
192	} else {	192	} else {
193	ret = list_lru_add(lru, item, nid, NULL);	193	ret = list_lru_add(lru, item, nid, NULL);
@@ -224,7 +224,7 @@ bool list_lru_del_obj(struct list_lru lru, struct list_head item) · 30 unmodified lines
224		224
225	if (list_lru_memcg_aware(lru)) {	225	if (list_lru_memcg_aware(lru)) {
226	rcu_read_lock();	226	rcu_read_lock();
227	ret = list_lru_del(lru, item, nid, mem_cgroup_from_slab_obj(item));	227	ret = list_lru_del(lru, item, nid, mem_cgroup_from_virt(item));
228	rcu_read_unlock();	228	rcu_read_unlock();
229	} else {	229	} else {
230	ret = list_lru_del(lru, item, nid, NULL);	230	ret = list_lru_del(lru, item, nid, NULL);
@@ -369,7 +369,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, · 138 unmodified lines
369		369
370	xa_for_each(&lru->xa, index, mlru) {	370	xa_for_each(&lru->xa, index, mlru) {
371	rcu_read_lock();	371	rcu_read_lock();
372	memcg = mem_cgroup_from_id(index);	372	memcg = mem_cgroup_from_private_id(index);
373	if (!mem_cgroup_tryget(memcg)) {	373	if (!mem_cgroup_tryget(memcg)) {
374	rcu_read_unlock();	374	rcu_read_unlock();
375	continue;	375	continue;

mm/madvise.c+12−14

 struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma)
 {
-	if (!rwsem_is_locked(&vma->vm_mm->mmap_lock))
+	vma_assert_stabilised(vma);
-		vma_assert_locked(vma);
 	return vma->anon_name;
 }
 	if (!start_pte)
 		return 0;
 	flush_tlb_batched_pending(mm);
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 	for (; addr < end; pte += nr, addr += nr * PAGE_SIZE) {
 		nr = 1;
 		ptent = ptep_get(pte);
 		if (++batch_count == SWAP_CLUSTER_MAX) {
 			batch_count = 0;
 			if (need_resched()) {
-				arch_leave_lazy_mmu_mode();
+				lazy_mmu_mode_disable();
 				pte_unmap_unlock(start_pte, ptl);
 				cond_resched();
 				goto restart;
 				if (!folio_trylock(folio))
 					continue;
 				folio_get(folio);
-				arch_leave_lazy_mmu_mode();
+				lazy_mmu_mode_disable();
 				pte_unmap_unlock(start_pte, ptl);
 				start_pte = NULL;
 				err = split_folio(folio);
 				if (!start_pte)
 					break;
 				flush_tlb_batched_pending(mm);
-				arch_enter_lazy_mmu_mode();
+				lazy_mmu_mode_enable();
 				if (!err)
 					nr = 0;
 				continue;
 	}
 	if (start_pte) {
-		arch_leave_lazy_mmu_mode();
+		lazy_mmu_mode_disable();
 		pte_unmap_unlock(start_pte, ptl);
 	}
 	if (pageout)
 	if (!start_pte)
 		return 0;
 	flush_tlb_batched_pending(mm);
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 	for (; addr != end; pte += nr, addr += PAGE_SIZE * nr) {
 		nr = 1;
 		ptent = ptep_get(pte);
 				max_nr = (end - addr) / PAGE_SIZE;
 				nr = swap_pte_batch(pte, max_nr, ptent);
 				nr_swap -= nr;
-				free_swap_and_cache_nr(entry, nr);
+				swap_put_entries_direct(entry, nr);
 				clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm);
 			} else if (softleaf_is_hwpoison(entry) ||
 				   softleaf_is_poison_marker(entry)) {
 				if (!folio_trylock(folio))
 					continue;
 				folio_get(folio);
-				arch_leave_lazy_mmu_mode();
+				lazy_mmu_mode_disable();
 				pte_unmap_unlock(start_pte, ptl);
 				start_pte = NULL;
 				err = split_folio(folio);
 				if (!start_pte)
 					break;
 				flush_tlb_batched_pending(mm);
-				arch_enter_lazy_mmu_mode();
+				lazy_mmu_mode_enable();
 				if (!err)
 					nr = 0;
 				continue;
 	if (nr_swap)
 		add_mm_counter(mm, MM_SWAPENTS, nr_swap);
 	if (start_pte) {
-		arch_leave_lazy_mmu_mode();
+		lazy_mmu_mode_disable();
 		pte_unmap_unlock(start_pte, ptl);
 	}
 	cond_resched();
  * madvise_should_skip() - Return if the request is invalid or nothing.
  * @start:	Start address of madvise-requested address range.
  * @len_in:	Length of madvise-requested address range.
- * @behavior:	Requested madvise behavor.
+ * @behavior:	Requested madvise behavior.
  * @err:	Pointer to store an error code from the check.
  *
  * If the specified behaviour is invalid or nothing would occur, we skip the

mm/memblock.c+2−2

@@ -773,7 +773,7 @@ bool __init_memblock memblock_validate_numa_coverage(unsigned long threshold_byt · 772 unmodified lines
773	unsigned long start_pfn, end_pfn, mem_size_mb;	773	unsigned long start_pfn, end_pfn, mem_size_mb;
774	int nid, i;	774	int nid, i;
775		775
776	/* calculate lose page */	776	/* calculate lost page */
777	for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) {	777	for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) {
778	if (!numa_valid_node(nid))	778	if (!numa_valid_node(nid))
779	nr_pages += end_pfn - start_pfn;	779	nr_pages += end_pfn - start_pfn;
@@ -2414,7 +2414,7 @@ EXPORT_SYMBOL_GPL(reserve_mem_find_by_name); · 1634 unmodified lines
2414		2414
2415	/**	2415	/**
2416	* reserve_mem_release_by_name - Release reserved memory region with a given name	2416	* reserve_mem_release_by_name - Release reserved memory region with a given name
2417	* @name: The name that is attatched to a reserved memory region	2417	* @name: The name that is attached to a reserved memory region
2418	*	2418	*
2419	* Forcibly release the pages in the reserved memory region so that those memory	2419	* Forcibly release the pages in the reserved memory region so that those memory
2420	* can be used as free memory. After released the reserved region size becomes 0.	2420	* can be used as free memory. After released the reserved region size becomes 0.

mm/memcontrol-v1.c+25−3

@@ -427,6 +427,28 @@ static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css, · 426 unmodified lines
427	}	427	}
428	#endif	428	#endif
429		429
		430	static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)
		431	{
		432	unsigned long val;
		433
		434	if (mem_cgroup_is_root(memcg)) {
		435	/*
		436	* Approximate root's usage from global state. This isn't
		437	* perfect, but the root usage was always an approximation.
		438	*/
		439	val = global_node_page_state(NR_FILE_PAGES) +
		440	global_node_page_state(NR_ANON_MAPPED);
		441	if (swap)
		442	val += total_swap_pages - get_nr_swap_pages();
		443	} else {
		444	if (!swap)
		445	val = page_counter_read(&memcg->memory);
		446	else
		447	val = page_counter_read(&memcg->memsw);
		448	}
		449	return val;
		450	}
		451
430	static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap)	452	static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap)
431	{	453	{
432	struct mem_cgroup_threshold_ary *t;	454	struct mem_cgroup_threshold_ary *t;
@@ -613,14 +635,14 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) · 180 unmodified lines
613	* have an ID allocated to it anymore, charge the closest online	635	* have an ID allocated to it anymore, charge the closest online
614	* ancestor for the swap instead and transfer the memory+swap charge.	636	* ancestor for the swap instead and transfer the memory+swap charge.
615	*/	637	*/
616	swap_memcg = mem_cgroup_id_get_online(memcg);	638	swap_memcg = mem_cgroup_private_id_get_online(memcg);
617	nr_entries = folio_nr_pages(folio);	639	nr_entries = folio_nr_pages(folio);
618	/* Get references for the tail pages, too */	640	/* Get references for the tail pages, too */
619	if (nr_entries > 1)	641	if (nr_entries > 1)
620	mem_cgroup_id_get_many(swap_memcg, nr_entries - 1);	642	mem_cgroup_private_id_get_many(swap_memcg, nr_entries - 1);
621	mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries);	643	mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries);
622		644
623	swap_cgroup_record(folio, mem_cgroup_id(swap_memcg), entry);	645	swap_cgroup_record(folio, mem_cgroup_private_id(swap_memcg), entry);
624		646
625	folio_unqueue_deferred_split(folio);	647	folio_unqueue_deferred_split(folio);
626	folio->memcg_data = 0;	648	folio->memcg_data = 0;

mm/memcontrol-v1.h+2−4

 	     iter != NULL;				\
 	     iter = mem_cgroup_iter(NULL, iter, NULL))
-unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap);
 void drain_all_stock(struct mem_cgroup *root_memcg);
 unsigned long memcg_events(struct mem_cgroup *memcg, int event);
 int memory_stat_show(struct seq_file *m, void *v);
-void mem_cgroup_id_get_many(struct mem_cgroup *memcg, unsigned int n);
+void mem_cgroup_private_id_get_many(struct mem_cgroup *memcg, unsigned int n);
-struct mem_cgroup *mem_cgroup_id_get_online(struct mem_cgroup *memcg);
+struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg);
 /* Cgroup v1-specific declarations */
 #ifdef CONFIG_MEMCG_V1

mm/memcontrol.c+34−65

 	 * in latency-sensitive paths is as cheap as possible.
 	 */
 	__mem_cgroup_flush_stats(root_mem_cgroup, true);
-	queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME);
+	queue_delayed_work(system_dfl_wq, &stats_flush_dwork, FLUSH_TIME);
 }
 unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx)
 	struct lruvec *lruvec;
 	rcu_read_lock();
-	memcg = mem_cgroup_from_slab_obj(p);
+	memcg = mem_cgroup_from_virt(p);
 	/*
 	 * Untracked pages have no memcg, no lruvec. Update only the
 	return max;
 }
-unsigned long mem_cgroup_size(struct mem_cgroup *memcg)
-{
-	return page_counter_read(&memcg->memory);
-}
 void __memcg_memory_event(struct mem_cgroup *memcg,
 			  enum memcg_memory_event event, bool allow_spinning)
 {
  * The caller must ensure the memcg lifetime, e.g. by taking rcu_read_lock(),
  * cgroup_mutex, etc.
  */
-struct mem_cgroup *mem_cgroup_from_slab_obj(void *p)
+struct mem_cgroup *mem_cgroup_from_virt(void *p)
 {
 	struct slab *slab;
 	css_get_many(&__folio_memcg(folio)->css, new_refs);
 }
-unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)
-{
-	unsigned long val;
-	if (mem_cgroup_is_root(memcg)) {
-		/*
-		 * Approximate root's usage from global state. This isn't
-		 * perfect, but the root usage was always an approximation.
-		 */
-		val = global_node_page_state(NR_FILE_PAGES) +
-			global_node_page_state(NR_ANON_MAPPED);
-		if (swap)
-			val += total_swap_pages - get_nr_swap_pages();
-	} else {
-		if (!swap)
-			val = page_counter_read(&memcg->memory);
-		else
-			val = page_counter_read(&memcg->memsw);
-	}
-	return val;
-}
 static int memcg_online_kmem(struct mem_cgroup *memcg)
 {
 	struct obj_cgroup *objcg;
  */
 #define MEM_CGROUP_ID_MAX	((1UL << MEM_CGROUP_ID_SHIFT) - 1)
-static DEFINE_XARRAY_ALLOC1(mem_cgroup_ids);
+static DEFINE_XARRAY_ALLOC1(mem_cgroup_private_ids);
-static void mem_cgroup_id_remove(struct mem_cgroup *memcg)
+static void mem_cgroup_private_id_remove(struct mem_cgroup *memcg)
 {
 	if (memcg->id.id > 0) {
-		xa_erase(&mem_cgroup_ids, memcg->id.id);
+		xa_erase(&mem_cgroup_private_ids, memcg->id.id);
 		memcg->id.id = 0;
 	}
 }
-void __maybe_unused mem_cgroup_id_get_many(struct mem_cgroup *memcg,
+void __maybe_unused mem_cgroup_private_id_get_many(struct mem_cgroup *memcg,
 					   unsigned int n)
 {
 	refcount_add(n, &memcg->id.ref);
 }
-static void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n)
+static void mem_cgroup_private_id_put_many(struct mem_cgroup *memcg, unsigned int n)
 {
 	if (refcount_sub_and_test(n, &memcg->id.ref)) {
-		mem_cgroup_id_remove(memcg);
+		mem_cgroup_private_id_remove(memcg);
 		/* Memcg ID pins CSS */
 		css_put(&memcg->css);
 	}
 }
-static inline void mem_cgroup_id_put(struct mem_cgroup *memcg)
+static inline void mem_cgroup_private_id_put(struct mem_cgroup *memcg)
 {
-	mem_cgroup_id_put_many(memcg, 1);
+	mem_cgroup_private_id_put_many(memcg, 1);
 }
-struct mem_cgroup *mem_cgroup_id_get_online(struct mem_cgroup *memcg)
+struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg)
 {
 	while (!refcount_inc_not_zero(&memcg->id.ref)) {
 		/*
 }
 /**
- * mem_cgroup_from_id - look up a memcg from a memcg id
+ * mem_cgroup_from_private_id - look up a memcg from a memcg id
  * @id: the memcg id to look up
  *
  * Caller must hold rcu_read_lock().
  */
-struct mem_cgroup *mem_cgroup_from_id(unsigned short id)
+struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id)
 {
 	WARN_ON_ONCE(!rcu_read_lock_held());
-	return xa_load(&mem_cgroup_ids, id);
+	return xa_load(&mem_cgroup_private_ids, id);
 }
-#ifdef CONFIG_SHRINKER_DEBUG
+struct mem_cgroup *mem_cgroup_get_from_id(u64 id)
-struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino)
 {
 	struct cgroup *cgrp;
 	struct cgroup_subsys_state *css;
-	struct mem_cgroup *memcg;
+	struct mem_cgroup *memcg = NULL;
-	cgrp = cgroup_get_from_id(ino);
+	cgrp = cgroup_get_from_id(id);
 	if (IS_ERR(cgrp))
-		return ERR_CAST(cgrp);
+		return NULL;
 	css = cgroup_get_e_css(cgrp, &memory_cgrp_subsys);
 	if (css)
 		memcg = container_of(css, struct mem_cgroup, css);
-	else
-		memcg = ERR_PTR(-ENOENT);
 	cgroup_put(cgrp);
 	return memcg;
 }
-#endif
 static void free_mem_cgroup_per_node_info(struct mem_cgroup_per_node *pn)
 {
 	if (!memcg)
 		return ERR_PTR(-ENOMEM);
-	error = xa_alloc(&mem_cgroup_ids, &memcg->id.id, NULL,
+	error = xa_alloc(&mem_cgroup_private_ids, &memcg->id.id, NULL,
 			 XA_LIMIT(1, MEM_CGROUP_ID_MAX), GFP_KERNEL);
 	if (error)
 		goto fail;
 	lru_gen_init_memcg(memcg);
 	return memcg;
 fail:
-	mem_cgroup_id_remove(memcg);
+	mem_cgroup_private_id_remove(memcg);
 	__mem_cgroup_free(memcg);
 	return ERR_PTR(error);
 }
 		goto offline_kmem;
 	if (unlikely(mem_cgroup_is_root(memcg)) && !mem_cgroup_disabled())
-		queue_delayed_work(system_unbound_wq, &stats_flush_dwork,
+		queue_delayed_work(system_dfl_wq, &stats_flush_dwork,
 				   FLUSH_TIME);
 	lru_gen_online_memcg(memcg);
 	css_get(css);
 	/*
-	 * Ensure mem_cgroup_from_id() works once we're fully online.
+	 * Ensure mem_cgroup_from_private_id() works once we're fully online.
 	 *
 	 * We could do this earlier and require callers to filter with
 	 * css_tryget_online(). But right now there are no users that
 	 * publish it here at the end of onlining. This matches the
 	 * regular ID destruction during offlining.
 	 */
-	xa_store(&mem_cgroup_ids, memcg->id.id, memcg, GFP_KERNEL);
+	xa_store(&mem_cgroup_private_ids, memcg->id.id, memcg, GFP_KERNEL);
 	return 0;
 offline_kmem:
 	memcg_offline_kmem(memcg);
 remove_id:
-	mem_cgroup_id_remove(memcg);
+	mem_cgroup_private_id_remove(memcg);
 	return -ENOMEM;
 }
 	drain_all_stock(memcg);
-	mem_cgroup_id_put(memcg);
+	mem_cgroup_private_id_put(memcg);
 }
 static void mem_cgroup_css_released(struct cgroup_subsys_state *css)
 	id = lookup_swap_cgroup_id(entry);
 	rcu_read_lock();
-	memcg = mem_cgroup_from_id(id);
+	memcg = mem_cgroup_from_private_id(id);
 	if (!memcg || !css_tryget_online(&memcg->css))
 		memcg = get_mem_cgroup_from_mm(mm);
 	rcu_read_unlock();
 	memcg = folio_memcg(old);
 	/*
 	 * Note that it is normal to see !memcg for a hugetlb folio.
-	 * For e.g, itt could have been allocated when memory_hugetlb_accounting
+	 * For e.g, it could have been allocated when memory_hugetlb_accounting
 	 * was not selected.
 	 */
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old);
 		return 0;
 	}
-	memcg = mem_cgroup_id_get_online(memcg);
+	memcg = mem_cgroup_private_id_get_online(memcg);
 	if (!mem_cgroup_is_root(memcg) &&
 	    !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
 		memcg_memory_event(memcg, MEMCG_SWAP_MAX);
 		memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
-		mem_cgroup_id_put(memcg);
+		mem_cgroup_private_id_put(memcg);
 		return -ENOMEM;
 	}
 	/* Get references for the tail pages, too */
 	if (nr_pages > 1)
-		mem_cgroup_id_get_many(memcg, nr_pages - 1);
+		mem_cgroup_private_id_get_many(memcg, nr_pages - 1);
 	mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
-	swap_cgroup_record(folio, mem_cgroup_id(memcg), entry);
+	swap_cgroup_record(folio, mem_cgroup_private_id(memcg), entry);
 	return 0;
 }
 	id = swap_cgroup_clear(entry, nr_pages);
 	rcu_read_lock();
-	memcg = mem_cgroup_from_id(id);
+	memcg = mem_cgroup_from_private_id(id);
 	if (memcg) {
 		if (!mem_cgroup_is_root(memcg)) {
 			if (do_memsw_account())
 				page_counter_uncharge(&memcg->swap, nr_pages);
 		}
 		mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
-		mem_cgroup_id_put_many(memcg, nr_pages);
+		mem_cgroup_private_id_put_many(memcg, nr_pages);
 	}
 	rcu_read_unlock();
 }

mm/memfd.c+1−2

+// SPDX-License-Identifier: GPL-2.0
 /*
  * memfd_create system call and file sealing support
  *
  * Code was originally included in shmem.c, and broken out to facilitate
  * use by hugetlbfs as well as tmpfs.
- *
- * This file is released under the GPL.
  */
 #include <linux/fs.h>

mm/memory-failure.c+1−1

@@ -868,7 +868,7 @@ static int kill_accessing_process(struct task_struct *p, unsigned long pfn, · 867 unmodified lines
868	*	868	*
869	* MF_RECOVERED - The m-f() handler marks the page as PG_hwpoisoned'ed.	869	* MF_RECOVERED - The m-f() handler marks the page as PG_hwpoisoned'ed.
870	* The page has been completely isolated, that is, unmapped, taken out of	870	* The page has been completely isolated, that is, unmapped, taken out of
871	* the buddy system, or hole-punnched out of the file mapping.	871	* the buddy system, or hole-punched out of the file mapping.
872	*/	872	*/
873	static const char *action_name[] = {	873	static const char *action_name[] = {
874	[MF_IGNORED] = "Ignored",	874	[MF_IGNORED] = "Ignored",

mm/memory-tiers.c+3−4

 	 */
 	list_for_each_entry_reverse(memtier, &memory_tiers, list) {
 		tier_nodes = get_memtier_nodemask(memtier);
-		nodes_and(tier_nodes, node_states[N_CPU], tier_nodes);
+		if (nodes_and(tier_nodes, node_states[N_CPU], tier_nodes)) {
-		if (!nodes_empty(tier_nodes)) {
 			/*
 			 * abstract distance below the max value of this memtier
 			 * is considered toptier.
 	if (node_memory_types[node].memtype == memtype || !memtype)
 		node_memory_types[node].map_count--;
 	/*
-	 * If we umapped all the attached devices to this node,
+	 * If we unmapped all the attached devices to this node,
 	 * clear the node memory type.
 	 */
 	if (!node_memory_types[node].map_count) {
 		struct pglist_data *pgdat;
 		for_each_online_pgdat(pgdat)
-			atomic_set(&pgdat->kswapd_failures, 0);
+			kswapd_clear_hopeless(pgdat, KSWAPD_CLEAR_HOPELESS_OTHER);
 	}
 	return count;

mm/memory.c+260−191

@@ -934,7 +934,7 @@ copy_nonpresent_pte(struct mm_struct dst_mm, struct mm_struct src_mm, · 933 unmodified lines
934	struct page *page;	934	struct page *page;
935		935
936	if (likely(softleaf_is_swap(entry))) {	936	if (likely(softleaf_is_swap(entry))) {
937	if (swap_duplicate(entry) < 0)	937	if (swap_dup_entry_direct(entry) < 0)
938	return -EIO;	938	return -EIO;
939		939
940	/* make sure dst_mm is on swapoff's mmlist. */	940	/* make sure dst_mm is on swapoff's mmlist. */
@@ -1256,7 +1256,7 @@ again: · 315 unmodified lines
1256	spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);	1256	spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
1257	orig_src_pte = src_pte;	1257	orig_src_pte = src_pte;
1258	orig_dst_pte = dst_pte;	1258	orig_dst_pte = dst_pte;
1259	arch_enter_lazy_mmu_mode();	1259	lazy_mmu_mode_enable();
1260		1260
1261	do {	1261	do {
1262	nr = 1;	1262	nr = 1;
@@ -1325,7 +1325,7 @@ again: · 62 unmodified lines
1325	} while (dst_pte += nr, src_pte += nr, addr += PAGE_SIZE * nr,	1325	} while (dst_pte += nr, src_pte += nr, addr += PAGE_SIZE * nr,
1326	addr != end);	1326	addr != end);
1327		1327
1328	arch_leave_lazy_mmu_mode();	1328	lazy_mmu_mode_disable();
1329	pte_unmap_unlock(orig_src_pte, src_ptl);	1329	pte_unmap_unlock(orig_src_pte, src_ptl);
1330	add_mm_rss_vec(dst_mm, rss);	1330	add_mm_rss_vec(dst_mm, rss);
1331	pte_unmap_unlock(orig_dst_pte, dst_ptl);	1331	pte_unmap_unlock(orig_dst_pte, dst_ptl);
@@ -1748,7 +1748,7 @@ static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, · 416 unmodified lines
1748		1748
1749	nr = swap_pte_batch(pte, max_nr, ptent);	1749	nr = swap_pte_batch(pte, max_nr, ptent);
1750	rss[MM_SWAPENTS] -= nr;	1750	rss[MM_SWAPENTS] -= nr;
1751	free_swap_and_cache_nr(entry, nr);	1751	swap_put_entries_direct(entry, nr);
1752	} else if (softleaf_is_migration(entry)) {	1752	} else if (softleaf_is_migration(entry)) {
1753	struct folio *folio = softleaf_to_folio(entry);	1753	struct folio *folio = softleaf_to_folio(entry);
1754		1754
@@ -1821,11 +1821,70 @@ static inline int do_zap_pte_range(struct mmu_gather *tlb, · 66 unmodified lines
1821	return nr;	1821	return nr;
1822	}	1822	}
1823		1823
		1824	static bool pte_table_reclaim_possible(unsigned long start, unsigned long end,
		1825	struct zap_details *details)
		1826	{
		1827	if (!IS_ENABLED(CONFIG_PT_RECLAIM))
		1828	return false;
		1829	/* Only zap if we are allowed to and cover the full page table. */
		1830	return details && details->reclaim_pt && (end - start >= PMD_SIZE);
		1831	}
		1832
		1833	static bool zap_empty_pte_table(struct mm_struct mm, pmd_t pmd,
		1834	spinlock_t ptl, pmd_t pmdval)
		1835	{
		1836	spinlock_t *pml = pmd_lockptr(mm, pmd);
		1837
		1838	if (ptl != pml && !spin_trylock(pml))
		1839	return false;
		1840
		1841	*pmdval = pmdp_get(pmd);
		1842	pmd_clear(pmd);
		1843	if (ptl != pml)
		1844	spin_unlock(pml);
		1845	return true;
		1846	}
		1847
		1848	static bool zap_pte_table_if_empty(struct mm_struct mm, pmd_t pmd,
		1849	unsigned long addr, pmd_t *pmdval)
		1850	{
		1851	spinlock_t pml, ptl = NULL;
		1852	pte_t start_pte, pte;
		1853	int i;
		1854
		1855	pml = pmd_lock(mm, pmd);
		1856	start_pte = pte_offset_map_rw_nolock(mm, pmd, addr, pmdval, &ptl);
		1857	if (!start_pte)
		1858	goto out_ptl;
		1859	if (ptl != pml)
		1860	spin_lock_nested(ptl, SINGLE_DEPTH_NESTING);
		1861
		1862	for (i = 0, pte = start_pte; i < PTRS_PER_PTE; i++, pte++) {
		1863	if (!pte_none(ptep_get(pte)))
		1864	goto out_ptl;
		1865	}
		1866	pte_unmap(start_pte);
		1867
		1868	pmd_clear(pmd);
		1869
		1870	if (ptl != pml)
		1871	spin_unlock(ptl);
		1872	spin_unlock(pml);
		1873	return true;
		1874	out_ptl:
		1875	if (start_pte)
		1876	pte_unmap_unlock(start_pte, ptl);
		1877	if (ptl != pml)
		1878	spin_unlock(pml);
		1879	return false;
		1880	}
		1881
1824	static unsigned long zap_pte_range(struct mmu_gather *tlb,	1882	static unsigned long zap_pte_range(struct mmu_gather *tlb,
1825	struct vm_area_struct vma, pmd_t pmd,	1883	struct vm_area_struct vma, pmd_t pmd,
1826	unsigned long addr, unsigned long end,	1884	unsigned long addr, unsigned long end,
1827	struct zap_details *details)	1885	struct zap_details *details)
1828	{	1886	{
		1887	bool can_reclaim_pt = pte_table_reclaim_possible(addr, end, details);
1829	bool force_flush = false, force_break = false;	1888	bool force_flush = false, force_break = false;
1830	struct mm_struct *mm = tlb->mm;	1889	struct mm_struct *mm = tlb->mm;
1831	int rss[NR_MM_COUNTERS];	1890	int rss[NR_MM_COUNTERS];
@@ -1834,7 +1893,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, · 2 unmodified lines
1834	pte_t *pte;	1893	pte_t *pte;
1835	pmd_t pmdval;	1894	pmd_t pmdval;
1836	unsigned long start = addr;	1895	unsigned long start = addr;
1837	bool can_reclaim_pt = reclaim_pt_is_enabled(start, end, details);
1838	bool direct_reclaim = true;	1896	bool direct_reclaim = true;
1839	int nr;	1897	int nr;
1840		1898
@@ -1846,7 +1904,7 @@ retry: · 5 unmodified lines
1846	return addr;	1904	return addr;
1847		1905
1848	flush_tlb_batched_pending(mm);	1906	flush_tlb_batched_pending(mm);
1849	arch_enter_lazy_mmu_mode();	1907	lazy_mmu_mode_enable();
1850	do {	1908	do {
1851	bool any_skipped = false;	1909	bool any_skipped = false;
1852		1910
@@ -1875,10 +1933,10 @@ retry: · 22 unmodified lines
1875	* from being repopulated by another thread.	1933	* from being repopulated by another thread.
1876	*/	1934	*/
1877	if (can_reclaim_pt && direct_reclaim && addr == end)	1935	if (can_reclaim_pt && direct_reclaim && addr == end)
1878	direct_reclaim = try_get_and_clear_pmd(mm, pmd, &pmdval);	1936	direct_reclaim = zap_empty_pte_table(mm, pmd, ptl, &pmdval);
1879		1937
1880	add_mm_rss_vec(mm, rss);	1938	add_mm_rss_vec(mm, rss);
1881	arch_leave_lazy_mmu_mode();	1939	lazy_mmu_mode_disable();
1882		1940
1883	/* Do the actual TLB flush before dropping ptl */	1941	/* Do the actual TLB flush before dropping ptl */
1884	if (force_flush) {	1942	if (force_flush) {
@@ -1904,10 +1962,10 @@ retry: · 19 unmodified lines
1904	}	1962	}
1905		1963
1906	if (can_reclaim_pt) {	1964	if (can_reclaim_pt) {
1907	if (direct_reclaim)	1965	if (direct_reclaim \|\| zap_pte_table_if_empty(mm, pmd, start, &pmdval)) {
1908	free_pte(mm, start, tlb, pmdval);	1966	pte_free_tlb(tlb, pmd_pgtable(pmdval), addr);
1909	else	1967	mm_dec_nr_ptes(mm);
1910	try_to_free_pte(mm, pmd, start, tlb);	1968	}
1911	}	1969	}
1912		1970
1913	return addr;	1971	return addr;
@@ -2499,7 +2557,6 @@ static int __vm_map_pages(struct vm_area_struct vma, struct page *pages, · 585 unmodified lines
2499	{	2557	{
2500	unsigned long count = vma_pages(vma);	2558	unsigned long count = vma_pages(vma);
2501	unsigned long uaddr = vma->vm_start;	2559	unsigned long uaddr = vma->vm_start;
2502	int ret, i;
2503		2560
2504	/* Fail if the user requested offset is beyond the end of the object */	2561	/* Fail if the user requested offset is beyond the end of the object */
2505	if (offset >= num)	2562	if (offset >= num)
@@ -2509,14 +2566,7 @@ static int __vm_map_pages(struct vm_area_struct vma, struct page *pages, · 3 unmodified lines
2509	if (count > num - offset)	2566	if (count > num - offset)
2510	return -ENXIO;	2567	return -ENXIO;
2511		2568
2512	for (i = 0; i < count; i++) {	2569	return vm_insert_pages(vma, uaddr, pages + offset, &count);
2513	ret = vm_insert_page(vma, uaddr, pages[offset + i]);
2514	if (ret < 0)
2515	return ret;
2516	uaddr += PAGE_SIZE;
2517	}
2518
2519	return 0;
2520	}	2570	}
2521		2571
2522	/**	2572	/**
@@ -2816,7 +2866,7 @@ static int remap_pte_range(struct mm_struct mm, pmd_t pmd, · 293 unmodified lines
2816	mapped_pte = pte = pte_alloc_map_lock(mm, pmd, addr, &ptl);	2866	mapped_pte = pte = pte_alloc_map_lock(mm, pmd, addr, &ptl);
2817	if (!pte)	2867	if (!pte)
2818	return -ENOMEM;	2868	return -ENOMEM;
2819	arch_enter_lazy_mmu_mode();	2869	lazy_mmu_mode_enable();
2820	do {	2870	do {
2821	BUG_ON(!pte_none(ptep_get(pte)));	2871	BUG_ON(!pte_none(ptep_get(pte)));
2822	if (!pfn_modify_allowed(pfn, prot)) {	2872	if (!pfn_modify_allowed(pfn, prot)) {
@@ -2826,7 +2876,7 @@ static int remap_pte_range(struct mm_struct mm, pmd_t pmd, · 3 unmodified lines
2826	set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot)));	2876	set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot)));
2827	pfn++;	2877	pfn++;
2828	} while (pte++, addr += PAGE_SIZE, addr != end);	2878	} while (pte++, addr += PAGE_SIZE, addr != end);
2829	arch_leave_lazy_mmu_mode();	2879	lazy_mmu_mode_disable();
2830	pte_unmap_unlock(mapped_pte, ptl);	2880	pte_unmap_unlock(mapped_pte, ptl);
2831	return err;	2881	return err;
2832	}	2882	}
@@ -3177,7 +3227,7 @@ static int apply_to_pte_range(struct mm_struct mm, pmd_t pmd, · 344 unmodified lines
3177	return -EINVAL;	3227	return -EINVAL;
3178	}	3228	}
3179		3229
3180	arch_enter_lazy_mmu_mode();	3230	lazy_mmu_mode_enable();
3181		3231
3182	if (fn) {	3232	if (fn) {
3183	do {	3233	do {
@@ -3190,7 +3240,7 @@ static int apply_to_pte_range(struct mm_struct mm, pmd_t pmd, · 6 unmodified lines
3190	}	3240	}
3191	*mask \|= PGTBL_PTE_MODIFIED;	3241	*mask \|= PGTBL_PTE_MODIFIED;
3192		3242
3193	arch_leave_lazy_mmu_mode();	3243	lazy_mmu_mode_disable();
3194		3244
3195	if (mm != &init_mm)	3245	if (mm != &init_mm)
3196	pte_unmap_unlock(mapped_pte, ptl);	3246	pte_unmap_unlock(mapped_pte, ptl);
@@ -4357,12 +4407,27 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) · 1160 unmodified lines
4357	return 0;	4407	return 0;
4358	}	4408	}
4359		4409
4360	static inline bool should_try_to_free_swap(struct folio *folio,	4410	/*
		4411	* Check if we should call folio_free_swap to free the swap cache.
		4412	* folio_free_swap only frees the swap cache to release the slot if swap
		4413	* count is zero, so we don't need to check the swap count here.
		4414	*/
		4415	static inline bool should_try_to_free_swap(struct swap_info_struct *si,
		4416	struct folio *folio,
4361	struct vm_area_struct *vma,	4417	struct vm_area_struct *vma,
		4418	unsigned int extra_refs,
4362	unsigned int fault_flags)	4419	unsigned int fault_flags)
4363	{	4420	{
4364	if (!folio_test_swapcache(folio))	4421	if (!folio_test_swapcache(folio))
4365	return false;	4422	return false;
		4423	/*
		4424	* Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap
		4425	* cache can help save some IO or memory overhead, but these devices
		4426	* are fast, and meanwhile, swap cache pinning the slot deferring the
		4427	* release of metadata or fragmentation is a more critical issue.
		4428	*/
		4429	if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
		4430	return true;
4366	if (mem_cgroup_swap_full(folio) \|\| (vma->vm_flags & VM_LOCKED) \|\|	4431	if (mem_cgroup_swap_full(folio) \|\| (vma->vm_flags & VM_LOCKED) \|\|
4367	folio_test_mlocked(folio))	4432	folio_test_mlocked(folio))
4368	return true;	4433	return true;
@@ -4373,7 +4438,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, · 4 unmodified lines
4373	* reference only in case it's likely that we'll be the exclusive user.	4438	* reference only in case it's likely that we'll be the exclusive user.
4374	*/	4439	*/
4375	return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) &&	4440	return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) &&
4376	folio_ref_count(folio) == (1 + folio_nr_pages(folio));	4441	folio_ref_count(folio) == (extra_refs + folio_nr_pages(folio));
4377	}	4442	}
4378		4443
4379	static vm_fault_t pte_marker_clear(struct vm_fault *vmf)	4444	static vm_fault_t pte_marker_clear(struct vm_fault *vmf)
@@ -4611,7 +4676,16 @@ static struct folio alloc_swap_folio(struct vm_fault vmf) · 231 unmodified lines
4611	}	4676	}
4612	#endif /* CONFIG_TRANSPARENT_HUGEPAGE */	4677	#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
4613		4678
4614	static DECLARE_WAIT_QUEUE_HEAD(swapcache_wq);	4679	/* Sanity check that a folio is fully exclusive */
		4680	static void check_swap_exclusive(struct folio *folio, swp_entry_t entry,
		4681	unsigned int nr_pages)
		4682	{
		4683	/* Called under PT locked and folio locked, the swap count is stable */
		4684	do {
		4685	VM_WARN_ON_ONCE_FOLIO(__swap_count(entry) != 1, folio);
		4686	entry.val++;
		4687	} while (--nr_pages);
		4688	}
4615		4689
4616	/*	4690	/*
4617	* We enter with non-exclusive mmap_lock (to exclude vma changes,	4691	* We enter with non-exclusive mmap_lock (to exclude vma changes,
@@ -4624,17 +4698,14 @@ static DECLARE_WAIT_QUEUE_HEAD(swapcache_wq); · 6 unmodified lines
4624	vm_fault_t do_swap_page(struct vm_fault *vmf)	4698	vm_fault_t do_swap_page(struct vm_fault *vmf)
4625	{	4699	{
4626	struct vm_area_struct *vma = vmf->vma;	4700	struct vm_area_struct *vma = vmf->vma;
4627	struct folio swapcache, folio = NULL;	4701	struct folio swapcache = NULL, folio;
4628	DECLARE_WAITQUEUE(wait, current);
4629	struct page *page;	4702	struct page *page;
4630	struct swap_info_struct *si = NULL;	4703	struct swap_info_struct *si = NULL;
4631	rmap_t rmap_flags = RMAP_NONE;	4704	rmap_t rmap_flags = RMAP_NONE;
4632	bool need_clear_cache = false;
4633	bool exclusive = false;	4705	bool exclusive = false;
4634	softleaf_t entry;	4706	softleaf_t entry;
4635	pte_t pte;	4707	pte_t pte;
4636	vm_fault_t ret = 0;	4708	vm_fault_t ret = 0;
4637	void *shadow = NULL;
4638	int nr_pages;	4709	int nr_pages;
4639	unsigned long page_idx;	4710	unsigned long page_idx;
4640	unsigned long address;	4711	unsigned long address;
@@ -4705,57 +4776,21 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 64 unmodified lines
4705	folio = swap_cache_get_folio(entry);	4776	folio = swap_cache_get_folio(entry);
4706	if (folio)	4777	if (folio)
4707	swap_update_readahead(folio, vma, vmf->address);	4778	swap_update_readahead(folio, vma, vmf->address);
4708	swapcache = folio;
4709
4710	if (!folio) {	4779	if (!folio) {
4711	if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&	4780	if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
4712	__swap_count(entry) == 1) {
4713	/* skip swapcache */
4714	folio = alloc_swap_folio(vmf);	4781	folio = alloc_swap_folio(vmf);
4715	if (folio) {	4782	if (folio) {
4716	__folio_set_locked(folio);
4717	__folio_set_swapbacked(folio);
4718
4719	nr_pages = folio_nr_pages(folio);
4720	if (folio_test_large(folio))
4721	entry.val = ALIGN_DOWN(entry.val, nr_pages);
4722	/*	4783	/*
4723	* Prevent parallel swapin from proceeding with	4784	* folio is charged, so swapin can only fail due
4724	* the cache flag. Otherwise, another thread	4785	* to raced swapin and return NULL.
4725	* may finish swapin first, free the entry, and
4726	* swapout reusing the same entry. It's
4727	* undetectable as pte_same() returns true due
4728	* to entry reuse.
4729	*/	4786	*/
4730	if (swapcache_prepare(entry, nr_pages)) {	4787	swapcache = swapin_folio(entry, folio);
4731	/*	4788	if (swapcache != folio)
4732	* Relax a bit to prevent rapid	4789	folio_put(folio);
4733	* repeated page faults.	4790	folio = swapcache;
4734	*/
4735	add_wait_queue(&swapcache_wq, &wait);
4736	schedule_timeout_uninterruptible(1);
4737	remove_wait_queue(&swapcache_wq, &wait);
4738	goto out_page;
4739	}
4740	need_clear_cache = true;
4741
4742	memcg1_swapin(entry, nr_pages);
4743
4744	shadow = swap_cache_get_shadow(entry);
4745	if (shadow)
4746	workingset_refault(folio, shadow);
4747
4748	folio_add_lru(folio);
4749
4750	/* To provide entry to swap_read_folio() */
4751	folio->swap = entry;
4752	swap_read_folio(folio, NULL);
4753	folio->private = NULL;
4754	}	4791	}
4755	} else {	4792	} else {
4756	folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE,	4793	folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf);
4757	vmf);
4758	swapcache = folio;
4759	}	4794	}
4760		4795
4761	if (!folio) {	4796	if (!folio) {
@@ -4777,60 +4812,58 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 15 unmodified lines
4777	count_memcg_event_mm(vma->vm_mm, PGMAJFAULT);	4812	count_memcg_event_mm(vma->vm_mm, PGMAJFAULT);
4778	}	4813	}
4779		4814
		4815	swapcache = folio;
4780	ret \|= folio_lock_or_retry(folio, vmf);	4816	ret \|= folio_lock_or_retry(folio, vmf);
4781	if (ret & VM_FAULT_RETRY)	4817	if (ret & VM_FAULT_RETRY)
4782	goto out_release;	4818	goto out_release;
4783		4819
4784	page = folio_file_page(folio, swp_offset(entry));	4820	page = folio_file_page(folio, swp_offset(entry));
4785	if (swapcache) {	4821	/*
4786	/*	4822	* Make sure folio_free_swap() or swapoff did not release the
4787	* Make sure folio_free_swap() or swapoff did not release the	4823	* swapcache from under us. The page pin, and pte_same test
4788	* swapcache from under us. The page pin, and pte_same test	4824	* below, are not enough to exclude that. Even if it is still
4789	* below, are not enough to exclude that. Even if it is still	4825	* swapcache, we need to check that the page's swap has not
4790	* swapcache, we need to check that the page's swap has not	4826	* changed.
4791	* changed.	4827	*/
4792	*/	4828	if (unlikely(!folio_matches_swap_entry(folio, entry)))
4793	if (unlikely(!folio_matches_swap_entry(folio, entry)))	4829	goto out_page;
4794	goto out_page;
4795
4796	if (unlikely(PageHWPoison(page))) {
4797	/*
4798	* hwpoisoned dirty swapcache pages are kept for killing
4799	* owner processes (which may be unknown at hwpoison time)
4800	*/
4801	ret = VM_FAULT_HWPOISON;
4802	goto out_page;
4803	}
4804
4805	/*
4806	* KSM sometimes has to copy on read faults, for example, if
4807	* folio->index of non-ksm folios would be nonlinear inside the
4808	* anon VMA -- the ksm flag is lost on actual swapout.
4809	*/
4810	folio = ksm_might_need_to_copy(folio, vma, vmf->address);
4811	if (unlikely(!folio)) {
4812	ret = VM_FAULT_OOM;
4813	folio = swapcache;
4814	goto out_page;
4815	} else if (unlikely(folio == ERR_PTR(-EHWPOISON))) {
4816	ret = VM_FAULT_HWPOISON;
4817	folio = swapcache;
4818	goto out_page;
4819	}
4820	if (folio != swapcache)
4821	page = folio_page(folio, 0);
4822		4830
		4831	if (unlikely(PageHWPoison(page))) {
4823	/*	4832	/*
4824	* If we want to map a page that's in the swapcache writable, we	4833	* hwpoisoned dirty swapcache pages are kept for killing
4825	* have to detect via the refcount if we're really the exclusive	4834	* owner processes (which may be unknown at hwpoison time)
4826	* owner. Try removing the extra reference from the local LRU
4827	* caches if required.
4828	*/	4835	*/
4829	if ((vmf->flags & FAULT_FLAG_WRITE) && folio == swapcache &&	4836	ret = VM_FAULT_HWPOISON;
4830	!folio_test_ksm(folio) && !folio_test_lru(folio))	4837	goto out_page;
4831	lru_add_drain();
4832	}	4838	}
4833		4839
		4840	/*
		4841	* KSM sometimes has to copy on read faults, for example, if
		4842	* folio->index of non-ksm folios would be nonlinear inside the
		4843	* anon VMA -- the ksm flag is lost on actual swapout.
		4844	*/
		4845	folio = ksm_might_need_to_copy(folio, vma, vmf->address);
		4846	if (unlikely(!folio)) {
		4847	ret = VM_FAULT_OOM;
		4848	folio = swapcache;
		4849	goto out_page;
		4850	} else if (unlikely(folio == ERR_PTR(-EHWPOISON))) {
		4851	ret = VM_FAULT_HWPOISON;
		4852	folio = swapcache;
		4853	goto out_page;
		4854	} else if (folio != swapcache)
		4855	page = folio_page(folio, 0);
		4856
		4857	/*
		4858	* If we want to map a page that's in the swapcache writable, we
		4859	* have to detect via the refcount if we're really the exclusive
		4860	* owner. Try removing the extra reference from the local LRU
		4861	* caches if required.
		4862	*/
		4863	if ((vmf->flags & FAULT_FLAG_WRITE) &&
		4864	!folio_test_ksm(folio) && !folio_test_lru(folio))
		4865	lru_add_drain();
		4866
4834	folio_throttle_swaprate(folio, GFP_KERNEL);	4867	folio_throttle_swaprate(folio, GFP_KERNEL);
4835		4868
4836	/*	4869	/*
@@ -4846,24 +4879,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 9 unmodified lines
4846	goto out_nomap;	4879	goto out_nomap;
4847	}	4880	}
4848		4881
4849	/* allocated large folios for SWP_SYNCHRONOUS_IO */
4850	if (folio_test_large(folio) && !folio_test_swapcache(folio)) {
4851	unsigned long nr = folio_nr_pages(folio);
4852	unsigned long folio_start = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE);
4853	unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE;
4854	pte_t *folio_ptep = vmf->pte - idx;
4855	pte_t folio_pte = ptep_get(folio_ptep);
4856
4857	if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) \|\|
4858	swap_pte_batch(folio_ptep, nr, folio_pte) != nr)
4859	goto out_nomap;
4860
4861	page_idx = idx;
4862	address = folio_start;
4863	ptep = folio_ptep;
4864	goto check_folio;
4865	}
4866
4867	nr_pages = 1;	4882	nr_pages = 1;
4868	page_idx = 0;	4883	page_idx = 0;
4869	address = vmf->address;	4884	address = vmf->address;
@@ -4907,12 +4922,37 @@ check_folio: · 37 unmodified lines
4907	BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));	4922	BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio));
4908	BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));	4923	BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page));
4909		4924
		4925	/*
		4926	* If a large folio already belongs to anon mapping, then we
		4927	* can just go on and map it partially.
		4928	* If not, with the large swapin check above failing, the page table
		4929	* have changed, so sub pages might got charged to the wrong cgroup,
		4930	* or even should be shmem. So we have to free it and fallback.
		4931	* Nothing should have touched it, both anon and shmem checks if a
		4932	* large folio is fully appliable before use.
		4933	*
		4934	* This will be removed once we unify folio allocation in the swap cache
		4935	* layer, where allocation of a folio stabilizes the swap entries.
		4936	*/
		4937	if (!folio_test_anon(folio) && folio_test_large(folio) &&
		4938	nr_pages != folio_nr_pages(folio)) {
		4939	if (!WARN_ON_ONCE(folio_test_dirty(folio)))
		4940	swap_cache_del_folio(folio);
		4941	goto out_nomap;
		4942	}
		4943
4910	/*	4944	/*
4911	* Check under PT lock (to protect against concurrent fork() sharing	4945	* Check under PT lock (to protect against concurrent fork() sharing
4912	* the swap entry concurrently) for certainly exclusive pages.	4946	* the swap entry concurrently) for certainly exclusive pages.
4913	*/	4947	*/
4914	if (!folio_test_ksm(folio)) {	4948	if (!folio_test_ksm(folio)) {
		4949	/*
		4950	* The can_swapin_thp check above ensures all PTE have
		4951	* same exclusiveness. Checking just one PTE is fine.
		4952	*/
4915	exclusive = pte_swp_exclusive(vmf->orig_pte);	4953	exclusive = pte_swp_exclusive(vmf->orig_pte);
		4954	if (exclusive)
		4955	check_swap_exclusive(folio, entry, nr_pages);
4916	if (folio != swapcache) {	4956	if (folio != swapcache) {
4917	/*	4957	/*
4918	* We have a fresh page that is not exposed to the	4958	* We have a fresh page that is not exposed to the
@@ -4946,19 +4986,10 @@ check_folio: · 27 unmodified lines
4946	/*	4986	/*
4947	* Some architectures may have to restore extra metadata to the page	4987	* Some architectures may have to restore extra metadata to the page
4948	* when reading from swap. This metadata may be indexed by swap entry	4988	* when reading from swap. This metadata may be indexed by swap entry
4949	* so this must be called before swap_free().	4989	* so this must be called before folio_put_swap().
4950	*/	4990	*/
4951	arch_swap_restore(folio_swap(entry, folio), folio);	4991	arch_swap_restore(folio_swap(entry, folio), folio);
4952		4992
4953	/*
4954	* Remove the swap entry and conditionally try to free up the swapcache.
4955	* We're already holding a reference on the page but haven't mapped it
4956	* yet.
4957	*/
4958	swap_free_nr(entry, nr_pages);
4959	if (should_try_to_free_swap(folio, vma, vmf->flags))
4960	folio_free_swap(folio);
4961
4962	add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages);	4993	add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages);
4963	add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages);	4994	add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages);
4964	pte = mk_pte(page, vma->vm_page_prot);	4995	pte = mk_pte(page, vma->vm_page_prot);
@@ -4990,22 +5021,24 @@ check_folio: · 25 unmodified lines
4990	vmf->orig_pte = pte_advance_pfn(pte, page_idx);	5021	vmf->orig_pte = pte_advance_pfn(pte, page_idx);
4991		5022
4992	/* ksm created a completely new copy */	5023	/* ksm created a completely new copy */
4993	if (unlikely(folio != swapcache && swapcache)) {	5024	if (unlikely(folio != swapcache)) {
4994	folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE);	5025	folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE);
4995	folio_add_lru_vma(folio, vma);	5026	folio_add_lru_vma(folio, vma);
		5027	folio_put_swap(swapcache, NULL);
4996	} else if (!folio_test_anon(folio)) {	5028	} else if (!folio_test_anon(folio)) {
4997	/*	5029	/*
4998	* We currently only expect small !anon folios which are either	5030	* We currently only expect !anon folios that are fully
4999	* fully exclusive or fully shared, or new allocated large	5031	* mappable. See the comment after can_swapin_thp above.
5000	* folios which are fully exclusive. If we ever get large
5001	* folios within swapcache here, we have to be careful.
5002	*/	5032	*/
5003	VM_WARN_ON_ONCE(folio_test_large(folio) && folio_test_swapcache(folio));	5033	VM_WARN_ON_ONCE_FOLIO(folio_nr_pages(folio) != nr_pages, folio);
5004	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);	5034	VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio);
5005	folio_add_new_anon_rmap(folio, vma, address, rmap_flags);	5035	folio_add_new_anon_rmap(folio, vma, address, rmap_flags);
		5036	folio_put_swap(folio, NULL);
5006	} else {	5037	} else {
		5038	VM_WARN_ON_ONCE(nr_pages != 1 && nr_pages != folio_nr_pages(folio));
5007	folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address,	5039	folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address,
5008	rmap_flags);	5040	rmap_flags);
		5041	folio_put_swap(folio, nr_pages == 1 ? page : NULL);
5009	}	5042	}
5010		5043
5011	VM_BUG_ON(!folio_test_anon(folio) \|\|	5044	VM_BUG_ON(!folio_test_anon(folio) \|\|
@@ -5014,13 +5047,21 @@ check_folio: · 2 unmodified lines
5014	arch_do_swap_page_nr(vma->vm_mm, vma, address,	5047	arch_do_swap_page_nr(vma->vm_mm, vma, address,
5015	pte, pte, nr_pages);	5048	pte, pte, nr_pages);
5016		5049
		5050	/*
		5051	* Remove the swap entry and conditionally try to free up the swapcache.
		5052	* Do it after mapping, so raced page faults will likely see the folio
		5053	* in swap cache and wait on the folio lock.
		5054	*/
		5055	if (should_try_to_free_swap(si, folio, vma, nr_pages, vmf->flags))
		5056	folio_free_swap(folio);
		5057
5017	folio_unlock(folio);	5058	folio_unlock(folio);
5018	if (folio != swapcache && swapcache) {	5059	if (unlikely(folio != swapcache)) {
5019	/*	5060	/*
5020	* Hold the lock to avoid the swap entry to be reused	5061	* Hold the lock to avoid the swap entry to be reused
5021	* until we take the PT lock for the pte_same() check	5062	* until we take the PT lock for the pte_same() check
5022	* (to avoid false positives from pte_same). For	5063	* (to avoid false positives from pte_same). For
5023	* further safety release the lock after the swap_free	5064	* further safety release the lock after the folio_put_swap
5024	* so that the swap count won't change under a	5065	* so that the swap count won't change under a
5025	* parallel locked swapcache.	5066	* parallel locked swapcache.
5026	*/	5067	*/
@@ -5041,12 +5082,6 @@ unlock: · 14 unmodified lines
5041	if (vmf->pte)	5082	if (vmf->pte)
5042	pte_unmap_unlock(vmf->pte, vmf->ptl);	5083	pte_unmap_unlock(vmf->pte, vmf->ptl);
5043	out:	5084	out:
5044	/* Clear the swap cache pin for direct swapin after PTL unlock */
5045	if (need_clear_cache) {
5046	swapcache_clear(si, entry, nr_pages);
5047	if (waitqueue_active(&swapcache_wq))
5048	wake_up(&swapcache_wq);
5049	}
5050	if (si)	5085	if (si)
5051	put_swap_device(si);	5086	put_swap_device(si);
5052	return ret;	5087	return ret;
@@ -5054,18 +5089,15 @@ out_nomap: · 1 unmodified line
5054	if (vmf->pte)	5089	if (vmf->pte)
5055	pte_unmap_unlock(vmf->pte, vmf->ptl);	5090	pte_unmap_unlock(vmf->pte, vmf->ptl);
5056	out_page:	5091	out_page:
		5092	if (folio_test_swapcache(folio))
		5093	folio_free_swap(folio);
5057	folio_unlock(folio);	5094	folio_unlock(folio);
5058	out_release:	5095	out_release:
5059	folio_put(folio);	5096	folio_put(folio);
5060	if (folio != swapcache && swapcache) {	5097	if (folio != swapcache) {
5061	folio_unlock(swapcache);	5098	folio_unlock(swapcache);
5062	folio_put(swapcache);	5099	folio_put(swapcache);
5063	}	5100	}
5064	if (need_clear_cache) {
5065	swapcache_clear(si, entry, nr_pages);
5066	if (waitqueue_active(&swapcache_wq))
5067	wake_up(&swapcache_wq);
5068	}
5069	if (si)	5101	if (si)
5070	put_swap_device(si);	5102	put_swap_device(si);
5071	return ret;	5103	return ret;
@@ -5935,7 +5967,7 @@ int numa_migrate_check(struct folio folio, struct vm_fault vmf, · 863 unmodified lines
5935	else	5967	else
5936	*last_cpupid = folio_last_cpupid(folio);	5968	*last_cpupid = folio_last_cpupid(folio);
5937		5969
5938	/* Record the current PID acceesing VMA */	5970	/* Record the current PID accessing VMA */
5939	vma_set_access_pid_bit(vma);	5971	vma_set_access_pid_bit(vma);
5940		5972
5941	count_vm_numa_event(NUMA_HINT_FAULTS);	5973	count_vm_numa_event(NUMA_HINT_FAULTS);
@@ -6254,7 +6286,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) · 312 unmodified lines
6254	* Use the maywrite version to indicate that vmf->pte may be	6286	* Use the maywrite version to indicate that vmf->pte may be
6255	* modified, but since we will use pte_same() to detect the	6287	* modified, but since we will use pte_same() to detect the
6256	* change of the !pte_none() entry, there is no need to recheck	6288	* change of the !pte_none() entry, there is no need to recheck
6257	* the pmdval. Here we chooes to pass a dummy variable instead	6289	* the pmdval. Here we choose to pass a dummy variable instead
6258	* of NULL, which helps new user think about why this place is	6290	* of NULL, which helps new user think about why this place is
6259	* special.	6291	* special.
6260	*/	6292	*/
@@ -7240,40 +7272,77 @@ static inline int process_huge_page( · 979 unmodified lines
7240	return 0;	7272	return 0;
7241	}	7273	}
7242		7274
7243	static void clear_gigantic_page(struct folio *folio, unsigned long addr_hint,	7275	static void clear_contig_highpages(struct page *page, unsigned long addr,
7244	unsigned int nr_pages)	7276	unsigned int nr_pages)
7245	{	7277	{
7246	unsigned long addr = ALIGN_DOWN(addr_hint, folio_size(folio));	7278	unsigned int i, count;
7247	int i;	7279	/*
		7280	* When clearing we want to operate on the largest extent possible to
		7281	* allow for architecture specific extent based optimizations.
		7282	*
		7283	* However, since clear_user_highpages() (and primitives clear_user_pages(),
		7284	* clear_pages()), do not call cond_resched(), limit the unit size when
		7285	* running under non-preemptible scheduling models.
		7286	*/
		7287	const unsigned int unit = preempt_model_preemptible() ?
		7288	nr_pages : PROCESS_PAGES_NON_PREEMPT_BATCH;
7248		7289
7249	might_sleep();	7290	might_sleep();
7250	for (i = 0; i < nr_pages; i++) {	7291
		7292	for (i = 0; i < nr_pages; i += count) {
7251	cond_resched();	7293	cond_resched();
7252	clear_user_highpage(folio_page(folio, i), addr + i * PAGE_SIZE);	7294
		7295	count = min(unit, nr_pages - i);
		7296	clear_user_highpages(page + i, addr + i * PAGE_SIZE, count);
7253	}	7297	}
7254	}	7298	}
7255		7299
7256	static int clear_subpage(unsigned long addr, int idx, void *arg)	7300	/*
7257	{	7301	* When zeroing a folio, we want to differentiate between pages in the
7258	struct folio *folio = arg;	7302	* vicinity of the faulting address where we have spatial and temporal
7259		7303	* locality, and those far away where we don't.
7260	clear_user_highpage(folio_page(folio, idx), addr);	7304	*
7261	return 0;	7305	* Use a radius of 2 for determining the local neighbourhood.
7262	}	7306	*/
		7307	#define FOLIO_ZERO_LOCALITY_RADIUS 2
7263		7308
7264	/**	7309	/**
7265	* folio_zero_user - Zero a folio which will be mapped to userspace.	7310	* folio_zero_user - Zero a folio which will be mapped to userspace.
7266	* @folio: The folio to zero.	7311	* @folio: The folio to zero.
7267	* @addr_hint: The address will be accessed or the base address if uncelar.	7312	* @addr_hint: The address accessed by the user or the base address.
7268	*/	7313	*/
7269	void folio_zero_user(struct folio *folio, unsigned long addr_hint)	7314	void folio_zero_user(struct folio *folio, unsigned long addr_hint)
7270	{	7315	{
7271	unsigned int nr_pages = folio_nr_pages(folio);	7316	const unsigned long base_addr = ALIGN_DOWN(addr_hint, folio_size(folio));
		7317	const long fault_idx = (addr_hint - base_addr) / PAGE_SIZE;
		7318	const struct range pg = DEFINE_RANGE(0, folio_nr_pages(folio) - 1);
		7319	const int radius = FOLIO_ZERO_LOCALITY_RADIUS;
		7320	struct range r[3];
		7321	int i;
7272		7322
7273	if (unlikely(nr_pages > MAX_ORDER_NR_PAGES))	7323	/*
7274	clear_gigantic_page(folio, addr_hint, nr_pages);	7324	* Faulting page and its immediate neighbourhood. Will be cleared at the
7275	else	7325	* end to keep its cachelines hot.
7276	process_huge_page(addr_hint, nr_pages, clear_subpage, folio);	7326	*/
		7327	r[2] = DEFINE_RANGE(clamp_t(s64, fault_idx - radius, pg.start, pg.end),
		7328	clamp_t(s64, fault_idx + radius, pg.start, pg.end));
		7329
		7330	/* Region to the left of the fault */
		7331	r[1] = DEFINE_RANGE(pg.start,
		7332	clamp_t(s64, r[2].start - 1, pg.start - 1, r[2].start));
		7333
		7334	/* Region to the right of the fault: always valid for the common fault_idx=0 case. */
		7335	r[0] = DEFINE_RANGE(clamp_t(s64, r[2].end + 1, r[2].end, pg.end + 1),
		7336	pg.end);
		7337
		7338	for (i = 0; i < ARRAY_SIZE(r); i++) {
		7339	const unsigned long addr = base_addr + r[i].start * PAGE_SIZE;
		7340	const unsigned int nr_pages = range_len(&r[i]);
		7341	struct page *page = folio_page(folio, r[i].start);
		7342
		7343	if (nr_pages > 0)
		7344	clear_contig_highpages(page, addr, nr_pages);
		7345	}
7277	}	7346	}
7278		7347
7279	static int copy_user_gigantic_page(struct folio dst, struct folio src,	7348	static int copy_user_gigantic_page(struct folio dst, struct folio src,

mm/memory_hotplug.c+4−4

mm/mempolicy.c+2−3

 static inline int mpol_store_user_nodemask(const struct mempolicy *pol)
 {
-	return pol->flags & MPOL_MODE_FLAGS;
+	return pol->flags & MPOL_USER_NODEMASK_FLAGS;
 }
 static void mpol_relative_nodemask(nodemask_t *ret, const nodemask_t *orig,
 	}
 	task_nodes = cpuset_mems_allowed(current);
-	nodes_and(*new, *new, task_nodes);
+	if (!nodes_and(*new, *new, task_nodes))
-	if (nodes_empty(*new))
 		goto out_put;
 	err = security_task_movememory(task);

mm/migrate.c+7−7

 	 * back to the buddy.
 	 */
 	if (PageOffline(page))
-		/* Only balloon compaction sets PageOffline pages movable. */
+		/* Only balloon page migration sets PageOffline pages movable. */
 		return offline_movable_ops;
 	if (PageZsmalloc(page))
 		return zsmalloc_movable_ops;
  * Get rid of all migration entries and replace them by
  * references to the indicated page.
  */
-void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
+void remove_migration_ptes(struct folio *src, struct folio *dst,
+		enum ttu_flags flags)
 {
 	struct rmap_walk_arg rmap_walk_arg = {
 		.folio = src,
-		.map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE,
+		.map_unused_to_zeropage = flags & TTU_USE_SHARED_ZEROPAGE,
 	};
 	struct rmap_walk_control rwc = {
 		.arg = &rmap_walk_arg,
 	};
-	VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src);
+	VM_BUG_ON_FOLIO((flags & TTU_USE_SHARED_ZEROPAGE) && (src != dst), src);
-	if (flags & RMP_LOCKED)
+	if (flags & TTU_RMAP_LOCKED)
 		rmap_walk_locked(dst, &rwc);
 	else
 		rmap_walk(dst, &rwc);
 		rc = move_to_new_folio(dst, src, mode);
 	if (page_was_mapped)
-		remove_migration_ptes(src, !rc ? dst : src,
+		remove_migration_ptes(src, !rc ? dst : src, ttu);
-				ttu ? RMP_LOCKED : 0);
 	if (ttu & TTU_RMAP_LOCKED)
 		i_mmap_unlock_write(mapping);

mm/migrate_device.c+6−6

@@ -271,7 +271,7 @@ again: · 270 unmodified lines
271	ptep = pte_offset_map_lock(mm, pmdp, start, &ptl);	271	ptep = pte_offset_map_lock(mm, pmdp, start, &ptl);
272	if (!ptep)	272	if (!ptep)
273	goto again;	273	goto again;
274	arch_enter_lazy_mmu_mode();	274	lazy_mmu_mode_enable();
275	ptep += (addr - start) / PAGE_SIZE;	275	ptep += (addr - start) / PAGE_SIZE;
276		276
277	for (; addr < end; addr += PAGE_SIZE, ptep++) {	277	for (; addr < end; addr += PAGE_SIZE, ptep++) {
@@ -313,7 +313,7 @@ again: · 35 unmodified lines
313	if (folio_test_large(folio)) {	313	if (folio_test_large(folio)) {
314	int ret;	314	int ret;
315		315
316	arch_leave_lazy_mmu_mode();	316	lazy_mmu_mode_disable();
317	pte_unmap_unlock(ptep, ptl);	317	pte_unmap_unlock(ptep, ptl);
318	ret = migrate_vma_split_folio(folio,	318	ret = migrate_vma_split_folio(folio,
319	migrate->fault_page);	319	migrate->fault_page);
@@ -356,7 +356,7 @@ again: · 36 unmodified lines
356	if (folio && folio_test_large(folio)) {	356	if (folio && folio_test_large(folio)) {
357	int ret;	357	int ret;
358		358
359	arch_leave_lazy_mmu_mode();	359	lazy_mmu_mode_disable();
360	pte_unmap_unlock(ptep, ptl);	360	pte_unmap_unlock(ptep, ptl);
361	ret = migrate_vma_split_folio(folio,	361	ret = migrate_vma_split_folio(folio,
362	migrate->fault_page);	362	migrate->fault_page);
@@ -485,7 +485,7 @@ next: · 122 unmodified lines
485	if (unmapped)	485	if (unmapped)
486	flush_tlb_range(walk->vma, start, end);	486	flush_tlb_range(walk->vma, start, end);
487		487
488	arch_leave_lazy_mmu_mode();	488	lazy_mmu_mode_disable();
489	pte_unmap_unlock(ptep - 1, ptl);	489	pte_unmap_unlock(ptep - 1, ptl);
490		490
491	return 0;	491	return 0;
@@ -1419,10 +1419,10 @@ EXPORT_SYMBOL(migrate_device_range); · 927 unmodified lines
1419		1419
1420	/**	1420	/**
1421	* migrate_device_pfns() - migrate device private pfns to normal memory.	1421	* migrate_device_pfns() - migrate device private pfns to normal memory.
1422	* @src_pfns: pre-popluated array of source device private pfns to migrate.	1422	* @src_pfns: pre-populated array of source device private pfns to migrate.
1423	* @npages: number of pages to migrate.	1423	* @npages: number of pages to migrate.
1424	*	1424	*
1425	* Similar to migrate_device_range() but supports non-contiguous pre-popluated	1425	* Similar to migrate_device_range() but supports non-contiguous pre-populated
1426	* array of device pages to migrate.	1426	* array of device pages to migrate.
1427	*/	1427	*/
1428	int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages)	1428	int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages)

mm/mm_init.c+18−17

 	/*
 	 * For policy OVERCOMMIT_NEVER, set batch size to 0.4% of
 	 * (total memory/#cpus), and lift it to 25% for other policies
-	 * to easy the possible lock contention for percpu_counter
+	 * to ease the possible lock contention for percpu_counter
 	 * vm_committed_as, while the max limit is INT_MAX
 	 */
 	if (overcommit_policy == OVERCOMMIT_NEVER)
 	return nid;
 }
-int hashdist = HASHDIST_DEFAULT;
+bool hashdist = HASHDIST_DEFAULT;
 static int __init set_hashdist(char *str)
 {
-	if (!str)
+	return kstrtobool(str, &hashdist) == 0;
-		return 0;
-	hashdist = simple_strtoul(str, &str, 0);
-	return 1;
 }
 __setup("hashdist=", set_hashdist);
 static inline void fixup_hashdist(void)
 {
 	if (num_node_state(N_MEMORY) == 1)
-		hashdist = 0;
+		hashdist = false;
 }
 #else
 static inline void fixup_hashdist(void) {}
 	lru_gen_init_pgdat(pgdat);
 }
-/* Any regular or high memory on that node ? */
+/* Any regular or high memory on that node? */
 static void __init check_for_memory(pg_data_t *pgdat)
 {
 	enum zone_type zone_type;
 /**
  * free_area_init - Initialise all pg_data_t and zone data
- * @max_zone_pfn: an array of max PFNs for each zone
  *
  * This will call free_area_init_node() for each active node in the system.
  * Using the page ranges provided by memblock_set_node(), the size of each
  * starts where the previous one ended. For example, ZONE_DMA32 starts
  * at arch_max_dma_pfn.
  */
-void __init free_area_init(unsigned long *max_zone_pfn)
+static void __init free_area_init(void)
 {
+	unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 };
 	unsigned long start_pfn, end_pfn;
 	int i, nid, zone;
 	bool descending;
-	/* Record where the zone boundaries are */
+	arch_zone_limits_init(max_zone_pfn);
-	memset(arch_zone_lowest_possible_pfn, 0,
+	sparse_init();
-				sizeof(arch_zone_lowest_possible_pfn));
-	memset(arch_zone_highest_possible_pfn, 0,
-				sizeof(arch_zone_highest_possible_pfn));
 	start_pfn = PHYS_PFN(memblock_start_of_DRAM());
 	descending = arch_has_descending_max_zone_pfns();
  * Initialize and free pages.
  *
  * At this point reserved pages and struct pages that correspond to holes in
- * memblock.memory are already intialized so every free range has a valid
+ * memblock.memory are already initialized so every free range has a valid
  * memory map around it.
  * This ensures that access of pages that are ahead of the range being
  * initialized (computing buddy page in __free_one_page()) always reads a valid
 {
 }
+void __init mm_core_init_early(void)
+{
+	hugetlb_cma_reserve();
+	hugetlb_bootmem_alloc();
+	free_area_init();
+}
 /*
  * Set up kernel memory allocators
  */
 void __init mm_core_init(void)
 {
 	arch_mm_preinit();
-	hugetlb_bootmem_alloc();
 	/* Initializations relying on SMP setup */
 	BUILD_BUG_ON(MAX_ZONELISTS > 2);

mm/mmap_lock.c+114−63

@@ -45,65 +45,111 @@ EXPORT_SYMBOL(__mmap_lock_do_trace_released); · 44 unmodified lines
45		45
46	#ifdef CONFIG_MMU	46	#ifdef CONFIG_MMU
47	#ifdef CONFIG_PER_VMA_LOCK	47	#ifdef CONFIG_PER_VMA_LOCK
		48
		49	/* State shared across __vma_[start, end]_exclude_readers. */
		50	struct vma_exclude_readers_state {
		51	/* Input parameters. */
		52	struct vm_area_struct *vma;
		53	int state; /* TASK_KILLABLE or TASK_UNINTERRUPTIBLE. */
		54	bool detaching;
		55
		56	/* Output parameters. */
		57	bool detached;
		58	bool exclusive; /* Are we exclusively locked? */
		59	};
		60
48	/*	61	/*
49	* __vma_enter_locked() returns 0 immediately if the vma is not	62	* Now that all readers have been evicted, mark the VMA as being out of the
50	* attached, otherwise it waits for any current readers to finish and	63	* 'exclude readers' state.
51	* returns 1. Returns -EINTR if a signal is received while waiting.
52	*/	64	*/
53	static inline int __vma_enter_locked(struct vm_area_struct *vma,	65	static void __vma_end_exclude_readers(struct vma_exclude_readers_state *ves)
54	bool detaching, int state)
55	{	66	{
56	int err;	67	struct vm_area_struct *vma = ves->vma;
57	unsigned int tgt_refcnt = VMA_LOCK_OFFSET;
58		68
59	mmap_assert_write_locked(vma->vm_mm);	69	VM_WARN_ON_ONCE(ves->detached);
		70
		71	ves->detached = refcount_sub_and_test(VM_REFCNT_EXCLUDE_READERS_FLAG,
		72	&vma->vm_refcnt);
		73	__vma_lockdep_release_exclusive(vma);
		74	}
		75
		76	static unsigned int get_target_refcnt(struct vma_exclude_readers_state *ves)
		77	{
		78	const unsigned int tgt = ves->detaching ? 0 : 1;
		79
		80	return tgt \| VM_REFCNT_EXCLUDE_READERS_FLAG;
		81	}
		82
		83	/*
		84	* Mark the VMA as being in a state of excluding readers, check to see if any
		85	* VMA read locks are indeed held, and if so wait for them to be released.
		86	*
		87	* Note that this function pairs with vma_refcount_put() which will wake up this
		88	* thread when it detects that the last reader has released its lock.
		89	*
		90	* The ves->state parameter ought to be set to TASK_UNINTERRUPTIBLE in cases
		91	* where we wish the thread to sleep uninterruptibly or TASK_KILLABLE if a fatal
		92	* signal is permitted to kill it.
		93	*
		94	* The function sets the ves->exclusive parameter to true if readers were
		95	* excluded, or false if the VMA was detached or an error arose on wait.
		96	*
		97	* If the function indicates an exclusive lock was acquired via ves->exclusive
		98	* the caller is required to invoke __vma_end_exclude_readers() once the
		99	* exclusive state is no longer required.
		100	*
		101	* If ves->state is set to something other than TASK_UNINTERRUPTIBLE, the
		102	* function may also return -EINTR to indicate a fatal signal was received while
		103	* waiting. Otherwise, the function returns 0.
		104	*/
		105	static int __vma_start_exclude_readers(struct vma_exclude_readers_state *ves)
		106	{
		107	struct vm_area_struct *vma = ves->vma;
		108	unsigned int tgt_refcnt = get_target_refcnt(ves);
		109	int err = 0;
60		110
61	/* Additional refcnt if the vma is attached. */	111	mmap_assert_write_locked(vma->vm_mm);
62	if (!detaching)
63	tgt_refcnt++;
64		112
65	/*	113	/*
66	* If vma is detached then only vma_mark_attached() can raise the	114	* If vma is detached then only vma_mark_attached() can raise the
67	* vm_refcnt. mmap_write_lock prevents racing with vma_mark_attached().	115	* vm_refcnt. mmap_write_lock prevents racing with vma_mark_attached().
		116	*
		117	* See the comment describing the vm_area_struct->vm_refcnt field for
		118	* details of possible refcnt values.
68	*/	119	*/
69	if (!refcount_add_not_zero(VMA_LOCK_OFFSET, &vma->vm_refcnt))	120	if (!refcount_add_not_zero(VM_REFCNT_EXCLUDE_READERS_FLAG, &vma->vm_refcnt)) {
		121	ves->detached = true;
70	return 0;	122	return 0;
		123	}
71		124
72	rwsem_acquire(&vma->vmlock_dep_map, 0, 0, _RET_IP_);	125	__vma_lockdep_acquire_exclusive(vma);
73	err = rcuwait_wait_event(&vma->vm_mm->vma_writer_wait,	126	err = rcuwait_wait_event(&vma->vm_mm->vma_writer_wait,
74	refcount_read(&vma->vm_refcnt) == tgt_refcnt,	127	refcount_read(&vma->vm_refcnt) == tgt_refcnt,
75	state);	128	ves->state);
76	if (err) {	129	if (err) {
77	if (refcount_sub_and_test(VMA_LOCK_OFFSET, &vma->vm_refcnt)) {	130	__vma_end_exclude_readers(ves);
78	/*
79	* The wait failed, but the last reader went away
80	* as well. Tell the caller the VMA is detached.
81	*/
82	WARN_ON_ONCE(!detaching);
83	err = 0;
84	}
85	rwsem_release(&vma->vmlock_dep_map, _RET_IP_);
86	return err;	131	return err;
87	}	132	}
88	lock_acquired(&vma->vmlock_dep_map, _RET_IP_);
89		133
90	return 1;	134	__vma_lockdep_stat_mark_acquired(vma);
91	}	135	ves->exclusive = true;
92		136	return 0;
93	static inline void __vma_exit_locked(struct vm_area_struct vma, bool detached)
94	{
95	*detached = refcount_sub_and_test(VMA_LOCK_OFFSET, &vma->vm_refcnt);
96	rwsem_release(&vma->vmlock_dep_map, _RET_IP_);
97	}	137	}
98		138
99	int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq,	139	int __vma_start_write(struct vm_area_struct *vma, int state)
100	int state)
101	{	140	{
102	int locked;	141	const unsigned int mm_lock_seq = __vma_raw_mm_seqnum(vma);
		142	struct vma_exclude_readers_state ves = {
		143	.vma = vma,
		144	.state = state,
		145	};
		146	int err;
103		147
104	locked = __vma_enter_locked(vma, false, state);	148	err = __vma_start_exclude_readers(&ves);
105	if (locked < 0)	149	if (err) {
106	return locked;	150	WARN_ON_ONCE(ves.detached);
		151	return err;
		152	}
107		153
108	/*	154	/*
109	* We should use WRITE_ONCE() here because we can have concurrent reads	155	* We should use WRITE_ONCE() here because we can have concurrent reads
@@ -113,39 +159,42 @@ int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq, · 3 unmodified lines
113	*/	159	*/
114	WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq);	160	WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq);
115		161
116	if (locked) {	162	if (ves.exclusive) {
117	bool detached;	163	__vma_end_exclude_readers(&ves);
118		164	/* VMA should remain attached. */
119	__vma_exit_locked(vma, &detached);	165	WARN_ON_ONCE(ves.detached);
120	WARN_ON_ONCE(detached); /* vma should remain attached */
121	}	166	}
122		167
123	return 0;	168	return 0;
124	}	169	}
125	EXPORT_SYMBOL_GPL(__vma_start_write);	170	EXPORT_SYMBOL_GPL(__vma_start_write);
126		171
127	void vma_mark_detached(struct vm_area_struct *vma)	172	void __vma_exclude_readers_for_detach(struct vm_area_struct *vma)
128	{	173	{
129	vma_assert_write_locked(vma);	174	struct vma_exclude_readers_state ves = {
130	vma_assert_attached(vma);	175	.vma = vma,
		176	.state = TASK_UNINTERRUPTIBLE,
		177	.detaching = true,
		178	};
		179	int err;
131		180
132	/*	181	/*
133	* We are the only writer, so no need to use vma_refcount_put().	182	* Wait until the VMA is detached with no readers. Since we hold the VMA
134	* The condition below is unlikely because the vma has been already	183	* write lock, the only read locks that might be present are those from
135	* write-locked and readers can increment vm_refcnt only temporarily	184	* threads trying to acquire the read lock and incrementing the
136	* before they check vm_lock_seq, realize the vma is locked and drop	185	* reference count before realising the write lock is held and
137	* back the vm_refcnt. That is a narrow window for observing a raised	186	* decrementing it.
138	* vm_refcnt.
139	*/	187	*/
140	if (unlikely(!refcount_dec_and_test(&vma->vm_refcnt))) {	188	err = __vma_start_exclude_readers(&ves);
141	/* Wait until vma is detached with no readers. */	189	if (!err && ves.exclusive) {
142	if (__vma_enter_locked(vma, true, TASK_UNINTERRUPTIBLE)) {	190	/*
143	bool detached;	191	* Once this is complete, no readers can increment the
144		192	* reference count, and the VMA is marked detached.
145	__vma_exit_locked(vma, &detached);	193	*/
146	WARN_ON_ONCE(!detached);	194	__vma_end_exclude_readers(&ves);
147	}
148	}	195	}
		196	/* If an error arose but we were detached anyway, we don't care. */
		197	WARN_ON_ONCE(!ves.detached);
149	}	198	}
150		199
151	/*	200	/*
@@ -180,19 +229,21 @@ static inline struct vm_area_struct vma_start_read(struct mm_struct mm, · 28 unmodified lines
180	}	229	}
181		230
182	/*	231	/*
183	* If VMA_LOCK_OFFSET is set, __refcount_inc_not_zero_limited_acquire()	232	* If VM_REFCNT_EXCLUDE_READERS_FLAG is set,
184	* will fail because VMA_REF_LIMIT is less than VMA_LOCK_OFFSET.	233	* __refcount_inc_not_zero_limited_acquire() will fail because
		234	* VM_REFCNT_LIMIT is less than VM_REFCNT_EXCLUDE_READERS_FLAG.
		235	*
185	* Acquire fence is required here to avoid reordering against later	236	* Acquire fence is required here to avoid reordering against later
186	* vm_lock_seq check and checks inside lock_vma_under_rcu().	237	* vm_lock_seq check and checks inside lock_vma_under_rcu().
187	*/	238	*/
188	if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,	239	if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
189	VMA_REF_LIMIT))) {	240	VM_REFCNT_LIMIT))) {
190	/* return EAGAIN if vma got detached from under us */	241	/* return EAGAIN if vma got detached from under us */
191	vma = oldcnt ? NULL : ERR_PTR(-EAGAIN);	242	vma = oldcnt ? NULL : ERR_PTR(-EAGAIN);
192	goto err;	243	goto err;
193	}	244	}
194		245
195	rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_);	246	__vma_lockdep_acquire_read(vma);
196		247
197	if (unlikely(vma->vm_mm != mm))	248	if (unlikely(vma->vm_mm != mm))
198	goto err_unstable;	249	goto err_unstable;

mm/mmu_gather.c+3−3

+// SPDX-License-Identifier: GPL-2.0
 #include <linux/gfp.h>
 #include <linux/highmem.h>
 #include <linux/kernel.h>
 					     PAGE_SIZE);
 }
-bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page,
+bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size)
-		bool delay_rmap, int page_size)
 {
-	return __tlb_remove_folio_pages_size(tlb, page, 1, delay_rmap, page_size);
+	return __tlb_remove_folio_pages_size(tlb, page, 1, false, page_size);
 }
 #endif /* MMU_GATHER_NO_GATHER */

mm/mprotect.c+2−2

@@ -233,7 +233,7 @@ static long change_pte_range(struct mmu_gather *tlb, · 232 unmodified lines
233	is_private_single_threaded = vma_is_single_threaded_private(vma);	233	is_private_single_threaded = vma_is_single_threaded_private(vma);
234		234
235	flush_tlb_batched_pending(vma->vm_mm);	235	flush_tlb_batched_pending(vma->vm_mm);
236	arch_enter_lazy_mmu_mode();	236	lazy_mmu_mode_enable();
237	do {	237	do {
238	nr_ptes = 1;	238	nr_ptes = 1;
239	oldpte = ptep_get(pte);	239	oldpte = ptep_get(pte);
@@ -379,7 +379,7 @@ static long change_pte_range(struct mmu_gather *tlb, · 139 unmodified lines
379	}	379	}
380	}	380	}
381	} while (pte += nr_ptes, addr += nr_ptes * PAGE_SIZE, addr != end);	381	} while (pte += nr_ptes, addr += nr_ptes * PAGE_SIZE, addr != end);
382	arch_leave_lazy_mmu_mode();	382	lazy_mmu_mode_disable();
383	pte_unmap_unlock(pte - 1, ptl);	383	pte_unmap_unlock(pte - 1, ptl);
384		384
385	return pages;	385	return pages;

mm/mremap.c+5−5

mm/mseal.c+2−2

@@ -21,7 +21,7 @@ · 20 unmodified lines
21	* It disallows unmapped regions from start to end whether they exist at the	21	* It disallows unmapped regions from start to end whether they exist at the
22	* start, in the middle, or at the end of the range, or any combination thereof.	22	* start, in the middle, or at the end of the range, or any combination thereof.
23	*	23	*
24	* This is because after sealng a range, there's nothing to stop memory mapping	24	* This is because after sealing a range, there's nothing to stop memory mapping
25	* of ranges in the remaining gaps later, meaning that the user might then	25	* of ranges in the remaining gaps later, meaning that the user might then
26	* wrongly consider the entirety of the mseal()'d range to be sealed when it	26	* wrongly consider the entirety of the mseal()'d range to be sealed when it
27	* in fact isn't.	27	* in fact isn't.
@@ -124,7 +124,7 @@ static int mseal_apply(struct mm_struct *mm, · 96 unmodified lines
124	* -EINVAL:	124	* -EINVAL:
125	* invalid input flags.	125	* invalid input flags.
126	* start address is not page aligned.	126	* start address is not page aligned.
127	* Address arange (start + len) overflow.	127	* Address range (start + len) overflow.
128	* -ENOMEM:	128	* -ENOMEM:
129	* addr is not a valid address (not allocated).	129	* addr is not a valid address (not allocated).
130	* end (start + len) is not a valid address.	130	* end (start + len) is not a valid address.

mm/numa_memblks.c+1−1

@@ -467,7 +467,7 @@ int __init numa_memblks_init(int (*init_func)(void), · 466 unmodified lines
467	* We reset memblock back to the top-down direction	467	* We reset memblock back to the top-down direction
468	* here because if we configured ACPI_NUMA, we have	468	* here because if we configured ACPI_NUMA, we have
469	* parsed SRAT in init_func(). It is ok to have the	469	* parsed SRAT in init_func(). It is ok to have the
470	* reset here even if we did't configure ACPI_NUMA	470	* reset here even if we didn't configure ACPI_NUMA
471	* or acpi numa init fails and fallbacks to dummy	471	* or acpi numa init fails and fallbacks to dummy
472	* numa init.	472	* numa init.
473	*/	473	*/

mm/oom_kill.c+13−13

@@ -228,7 +228,7 @@ long oom_badness(struct task_struct *p, unsigned long totalpages) · 227 unmodified lines
228	* The baseline for the badness score is the proportion of RAM that each	228	* The baseline for the badness score is the proportion of RAM that each
229	* task's rss, pagetable and swap space use.	229	* task's rss, pagetable and swap space use.
230	*/	230	*/
231	points = get_mm_rss(p->mm) + get_mm_counter(p->mm, MM_SWAPENTS) +	231	points = get_mm_rss_sum(p->mm) + get_mm_counter_sum(p->mm, MM_SWAPENTS) +
232	mm_pgtables_bytes(p->mm) / PAGE_SIZE;	232	mm_pgtables_bytes(p->mm) / PAGE_SIZE;
233	task_unlock(p);	233	task_unlock(p);
234		234
@@ -402,10 +402,10 @@ static int dump_task(struct task_struct p, void arg) · 167 unmodified lines
402		402
403	pr_info("[%7d] %5d %5d %8lu %8lu %8lu %8lu %9lu %8ld %8lu %5hd %s\n",	403	pr_info("[%7d] %5d %5d %8lu %8lu %8lu %8lu %9lu %8ld %8lu %5hd %s\n",
404	task->pid, from_kuid(&init_user_ns, task_uid(task)),	404	task->pid, from_kuid(&init_user_ns, task_uid(task)),
405	task->tgid, task->mm->total_vm, get_mm_rss(task->mm),	405	task->tgid, task->mm->total_vm, get_mm_rss_sum(task->mm),
406	get_mm_counter(task->mm, MM_ANONPAGES), get_mm_counter(task->mm, MM_FILEPAGES),	406	get_mm_counter_sum(task->mm, MM_ANONPAGES), get_mm_counter_sum(task->mm, MM_FILEPAGES),
407	get_mm_counter(task->mm, MM_SHMEMPAGES), mm_pgtables_bytes(task->mm),	407	get_mm_counter_sum(task->mm, MM_SHMEMPAGES), mm_pgtables_bytes(task->mm),
408	get_mm_counter(task->mm, MM_SWAPENTS),	408	get_mm_counter_sum(task->mm, MM_SWAPENTS),
409	task->signal->oom_score_adj, task->comm);	409	task->signal->oom_score_adj, task->comm);
410	task_unlock(task);	410	task_unlock(task);
411		411
@@ -458,7 +458,7 @@ static void dump_oom_victim(struct oom_control oc, struct task_struct victim) · 46 unmodified lines
458		458
459	static void dump_header(struct oom_control *oc)	459	static void dump_header(struct oom_control *oc)
460	{	460	{
461	pr_warn("%s invoked oom-killer: gfp_mask=%#x(%pGg), order=%d, oom_score_adj=%hd\n",	461	pr_warn("%s invoked oom-killer: gfp_mask=%#x(%pGg), order=%d, oom_score_adj=%d\n",
462	current->comm, oc->gfp_mask, &oc->gfp_mask, oc->order,	462	current->comm, oc->gfp_mask, &oc->gfp_mask, oc->order,
463	current->signal->oom_score_adj);	463	current->signal->oom_score_adj);
464	if (!IS_ENABLED(CONFIG_COMPACTION) && oc->order)	464	if (!IS_ENABLED(CONFIG_COMPACTION) && oc->order)
@@ -604,9 +604,9 @@ static bool oom_reap_task_mm(struct task_struct tsk, struct mm_struct mm) · 139 unmodified lines
604		604
605	pr_info("oom_reaper: reaped process %d (%s), now anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n",	605	pr_info("oom_reaper: reaped process %d (%s), now anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n",
606	task_pid_nr(tsk), tsk->comm,	606	task_pid_nr(tsk), tsk->comm,
607	K(get_mm_counter(mm, MM_ANONPAGES)),	607	K(get_mm_counter_sum(mm, MM_ANONPAGES)),
608	K(get_mm_counter(mm, MM_FILEPAGES)),	608	K(get_mm_counter_sum(mm, MM_FILEPAGES)),
609	K(get_mm_counter(mm, MM_SHMEMPAGES)));	609	K(get_mm_counter_sum(mm, MM_SHMEMPAGES)));
610	out_finish:	610	out_finish:
611	trace_finish_task_reaping(tsk->pid);	611	trace_finish_task_reaping(tsk->pid);
612	out_unlock:	612	out_unlock:
@@ -958,11 +958,11 @@ static void __oom_kill_process(struct task_struct victim, const char message) · 345 unmodified lines
958	*/	958	*/
959	do_send_sig_info(SIGKILL, SEND_SIG_PRIV, victim, PIDTYPE_TGID);	959	do_send_sig_info(SIGKILL, SEND_SIG_PRIV, victim, PIDTYPE_TGID);
960	mark_oom_victim(victim);	960	mark_oom_victim(victim);
961	pr_err("%s: Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB, UID:%u pgtables:%lukB oom_score_adj:%hd\n",	961	pr_err("%s: Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB, UID:%u pgtables:%lukB oom_score_adj:%d\n",
962	message, task_pid_nr(victim), victim->comm, K(mm->total_vm),	962	message, task_pid_nr(victim), victim->comm, K(mm->total_vm),
963	K(get_mm_counter(mm, MM_ANONPAGES)),	963	K(get_mm_counter_sum(mm, MM_ANONPAGES)),
964	K(get_mm_counter(mm, MM_FILEPAGES)),	964	K(get_mm_counter_sum(mm, MM_FILEPAGES)),
965	K(get_mm_counter(mm, MM_SHMEMPAGES)),	965	K(get_mm_counter_sum(mm, MM_SHMEMPAGES)),
966	from_kuid(&init_user_ns, task_uid(victim)),	966	from_kuid(&init_user_ns, task_uid(victim)),
967	mm_pgtables_bytes(mm) >> 10, victim->signal->oom_score_adj);	967	mm_pgtables_bytes(mm) >> 10, victim->signal->oom_score_adj);
968	task_unlock(victim);	968	task_unlock(victim);

mm/page-writeback.c+15−59

@@ -109,14 +109,6 @@ EXPORT_SYMBOL_GPL(dirty_writeback_interval); · 108 unmodified lines
109	*/	109	*/
110	unsigned int dirty_expire_interval = 30 * 100; /* centiseconds */	110	unsigned int dirty_expire_interval = 30 * 100; /* centiseconds */
111		111
112	/*
113	* Flag that puts the machine in "laptop mode". Doubles as a timeout in jiffies:
114	* a full sync is triggered after this time elapses without any disk activity.
115	*/
116	int laptop_mode;
117
118	EXPORT_SYMBOL(laptop_mode);
119
120	/* End of sysctl-exported parameters */	112	/* End of sysctl-exported parameters */
121		113
122	struct wb_domain global_wb_domain;	114	struct wb_domain global_wb_domain;
@@ -1843,17 +1835,7 @@ static int balance_dirty_pages(struct bdi_writeback *wb, · 1720 unmodified lines
1843	balance_domain_limits(mdtc, strictlimit);	1835	balance_domain_limits(mdtc, strictlimit);
1844	}	1836	}
1845		1837
1846	/*	1838	if (nr_dirty > gdtc->bg_thresh && !writeback_in_progress(wb))
1847	* In laptop mode, we wait until hitting the higher threshold
1848	* before starting background writeout, and then write out all
1849	* the way down to the lower threshold. So slow writers cause
1850	* minimal disk activity.
1851	*
1852	* In normal mode, we start background writeout at the lower
1853	* background_thresh, to keep the amount of dirty memory low.
1854	*/
1855	if (!laptop_mode && nr_dirty > gdtc->bg_thresh &&
1856	!writeback_in_progress(wb))
1857	wb_start_background_writeback(wb);	1839	wb_start_background_writeback(wb);
1858		1840
1859	/*	1841	/*
@@ -1876,10 +1858,6 @@ free_running: · 16 unmodified lines
1876	break;	1858	break;
1877	}	1859	}
1878		1860
1879	/* Start writeback even when in laptop mode */
1880	if (unlikely(!writeback_in_progress(wb)))
1881	wb_start_background_writeback(wb);
1882
1883	mem_cgroup_flush_foreign(wb);	1861	mem_cgroup_flush_foreign(wb);
1884		1862
1885	/*	1863	/*
@@ -2198,41 +2176,6 @@ static int dirty_writeback_centisecs_handler(const struct ctl_table *table, int · 312 unmodified lines
2198	}	2176	}
2199	#endif	2177	#endif
2200		2178
2201	void laptop_mode_timer_fn(struct timer_list *t)
2202	{
2203	struct backing_dev_info *backing_dev_info =
2204	timer_container_of(backing_dev_info, t, laptop_mode_wb_timer);
2205
2206	wakeup_flusher_threads_bdi(backing_dev_info, WB_REASON_LAPTOP_TIMER);
2207	}
2208
2209	/*
2210	* We've spun up the disk and we're in laptop mode: schedule writeback
2211	* of all dirty data a few seconds from now. If the flush is already scheduled
2212	* then push it back - the user is still using the disk.
2213	*/
2214	void laptop_io_completion(struct backing_dev_info *info)
2215	{
2216	mod_timer(&info->laptop_mode_wb_timer, jiffies + laptop_mode);
2217	}
2218
2219	/*
2220	* We're in laptop mode and we've just synced. The sync's writes will have
2221	* caused another writeback to be scheduled by laptop_io_completion.
2222	* Nothing needs to be written back anymore, so we unschedule the writeback.
2223	*/
2224	void laptop_sync_completion(void)
2225	{
2226	struct backing_dev_info *bdi;
2227
2228	rcu_read_lock();
2229
2230	list_for_each_entry_rcu(bdi, &bdi_list, bdi_list)
2231	timer_delete(&bdi->laptop_mode_wb_timer);
2232
2233	rcu_read_unlock();
2234	}
2235
2236	/*	2179	/*
2237	* If ratelimit_pages is too high then we can get into dirty-data overload	2180	* If ratelimit_pages is too high then we can get into dirty-data overload
2238	* if a large number of processes all perform writes at the same time.	2181	* if a large number of processes all perform writes at the same time.
@@ -2263,6 +2206,19 @@ static int page_writeback_cpu_online(unsigned int cpu) · 24 unmodified lines
2263		2206
2264	#ifdef CONFIG_SYSCTL	2207	#ifdef CONFIG_SYSCTL
2265		2208
		2209	static int laptop_mode;
		2210	static int laptop_mode_handler(const struct ctl_table *table, int write,
		2211	void buffer, size_t lenp, loff_t *ppos)
		2212	{
		2213	int ret = proc_dointvec_jiffies(table, write, buffer, lenp, ppos);
		2214
		2215	if (!ret && write)
		2216	pr_warn("%s: vm.laptop_mode is deprecated. Ignoring setting.\n",
		2217	current->comm);
		2218
		2219	return ret;
		2220	}
		2221
2266	/* this is needed for the proc_doulongvec_minmax of vm_dirty_bytes */	2222	/* this is needed for the proc_doulongvec_minmax of vm_dirty_bytes */
2267	static const unsigned long dirty_bytes_min = 2 * PAGE_SIZE;	2223	static const unsigned long dirty_bytes_min = 2 * PAGE_SIZE;
2268		2224
@@ -2332,7 +2288,7 @@ static const struct ctl_table vm_page_writeback_sysctls[] = { · 63 unmodified lines
2332	.data = &laptop_mode,	2288	.data = &laptop_mode,
2333	.maxlen = sizeof(laptop_mode),	2289	.maxlen = sizeof(laptop_mode),
2334	.mode = 0644,	2290	.mode = 0644,
2335	.proc_handler = proc_dointvec_jiffies,	2291	.proc_handler = laptop_mode_handler,
2336	},	2292	},
2337	};	2293	};
2338	#endif	2294	#endif

mm/page_alloc.c+275−150

@@ -1,6 +1,5 @@
1	// SPDX-License-Identifier: GPL-2.0-only	1	// SPDX-License-Identifier: GPL-2.0-only
2	/*	2	/*
3	* linux/mm/page_alloc.c
4	*	3	*
5	* Manages the free list, the system allocates free pages here.	4	* Manages the free list, the system allocates free pages here.
6	* Note that kmalloc() lives in slab.c	5	* Note that kmalloc() lives in slab.c
@@ -1853,7 +1852,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, · 1846 unmodified lines
1853		1852
1854	/*	1853	/*
1855	* As memory initialization might be integrated into KASAN,	1854	* As memory initialization might be integrated into KASAN,
1856	* KASAN unpoisoning and memory initializion code must be	1855	* KASAN unpoisoning and memory initialization code must be
1857	* kept together to avoid discrepancies in behavior.	1856	* kept together to avoid discrepancies in behavior.
1858	*/	1857	*/
1859		1858
@@ -2946,9 +2945,9 @@ static bool free_frozen_page_commit(struct zone *zone, · 1086 unmodified lines
2946	* 'hopeless node' to stay in that state for a while. Let	2945	* 'hopeless node' to stay in that state for a while. Let
2947	* kswapd work again by resetting kswapd_failures.	2946	* kswapd work again by resetting kswapd_failures.
2948	*/	2947	*/
2949	if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES &&	2948	if (kswapd_test_hopeless(pgdat) &&
2950	next_memory_node(pgdat->node_id) < MAX_NUMNODES)	2949	next_memory_node(pgdat->node_id) < MAX_NUMNODES)
2951	atomic_set(&pgdat->kswapd_failures, 0);	2950	kswapd_clear_hopeless(pgdat, KSWAPD_CLEAR_HOPELESS_PCP);
2952	}	2951	}
2953	return ret;	2952	return ret;
2954	}	2953	}
@@ -3112,6 +3111,15 @@ void free_unref_folios(struct folio_batch *folios) · 157 unmodified lines
3112	folio_batch_reinit(folios);	3111	folio_batch_reinit(folios);
3113	}	3112	}
3114		3113
		3114	static void __split_page(struct page *page, unsigned int order)
		3115	{
		3116	VM_WARN_ON_PAGE(PageCompound(page), page);
		3117
		3118	split_page_owner(page, order, 0);
		3119	pgalloc_tag_split(page_folio(page), order, 0);
		3120	split_page_memcg(page, order);
		3121	}
		3122
3115	/*	3123	/*
3116	* split_page takes a non-compound higher-order page, and splits it into	3124	* split_page takes a non-compound higher-order page, and splits it into
3117	* n (1<<order) sub-pages: page[0..n]	3125	* n (1<<order) sub-pages: page[0..n]
@@ -3124,14 +3132,12 @@ void split_page(struct page *page, unsigned int order) · 6 unmodified lines
3124	{	3132	{
3125	int i;	3133	int i;
3126		3134
3127	VM_BUG_ON_PAGE(PageCompound(page), page);	3135	VM_WARN_ON_PAGE(!page_count(page), page);
3128	VM_BUG_ON_PAGE(!page_count(page), page);
3129		3136
3130	for (i = 1; i < (1 << order); i++)	3137	for (i = 1; i < (1 << order); i++)
3131	set_page_refcounted(page + i);	3138	set_page_refcounted(page + i);
3132	split_page_owner(page, order, 0);	3139
3133	pgalloc_tag_split(page_folio(page), order, 0);	3140	__split_page(page, order);
3134	split_page_memcg(page, order);
3135	}	3141	}
3136	EXPORT_SYMBOL_GPL(split_page);	3142	EXPORT_SYMBOL_GPL(split_page);
3137		3143
@@ -4699,7 +4705,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · 1561 unmodified lines
4699	struct alloc_context *ac)	4705	struct alloc_context *ac)
4700	{	4706	{
4701	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;	4707	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
4702	bool can_compact = gfp_compaction_allowed(gfp_mask);	4708	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
4703	bool nofail = gfp_mask & __GFP_NOFAIL;	4709	bool nofail = gfp_mask & __GFP_NOFAIL;
4704	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;	4710	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
4705	struct page *page = NULL;	4711	struct page *page = NULL;
@@ -4712,6 +4718,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · 6 unmodified lines
4712	unsigned int cpuset_mems_cookie;	4718	unsigned int cpuset_mems_cookie;
4713	unsigned int zonelist_iter_cookie;	4719	unsigned int zonelist_iter_cookie;
4714	int reserve_flags;	4720	int reserve_flags;
		4721	bool compact_first = false;
		4722	bool can_retry_reserves = true;
4715		4723
4716	if (unlikely(nofail)) {	4724	if (unlikely(nofail)) {
4717	/*	4725	/*
@@ -4735,6 +4743,19 @@ restart: · 17 unmodified lines
4735	cpuset_mems_cookie = read_mems_allowed_begin();	4743	cpuset_mems_cookie = read_mems_allowed_begin();
4736	zonelist_iter_cookie = zonelist_iter_begin();	4744	zonelist_iter_cookie = zonelist_iter_begin();
4737		4745
		4746	/*
		4747	* For costly allocations, try direct compaction first, as it's likely
		4748	* that we have enough base pages and don't need to reclaim. For non-
		4749	* movable high-order allocations, do that as well, as compaction will
		4750	* try prevent permanent fragmentation by migrating from blocks of the
		4751	* same migratetype.
		4752	*/
		4753	if (can_compact && (costly_order \|\| (order > 0 &&
		4754	ac->migratetype != MIGRATE_MOVABLE))) {
		4755	compact_first = true;
		4756	compact_priority = INIT_COMPACT_PRIORITY;
		4757	}
		4758
4738	/*	4759	/*
4739	* The fast path uses conservative alloc_flags to succeed only until	4760	* The fast path uses conservative alloc_flags to succeed only until
4740	* kswapd needs to be woken up, and to avoid the cost of setting up	4761	* kswapd needs to be woken up, and to avoid the cost of setting up
@@ -4766,6 +4787,8 @@ restart: · 25 unmodified lines
4766	goto nopage;	4787	goto nopage;
4767	}	4788	}
4768		4789
		4790	retry:
		4791	/* Ensure kswapd doesn't accidentally go to sleep as long as we loop */
4769	if (alloc_flags & ALLOC_KSWAPD)	4792	if (alloc_flags & ALLOC_KSWAPD)
4770	wake_all_kswapds(order, gfp_mask, ac);	4793	wake_all_kswapds(order, gfp_mask, ac);
4771		4794
@@ -4777,74 +4800,6 @@ restart: · 5 unmodified lines
4777	if (page)	4800	if (page)
4778	goto got_pg;	4801	goto got_pg;
4779		4802
4780	/*
4781	* For costly allocations, try direct compaction first, as it's likely
4782	* that we have enough base pages and don't need to reclaim. For non-
4783	* movable high-order allocations, do that as well, as compaction will
4784	* try prevent permanent fragmentation by migrating from blocks of the
4785	* same migratetype.
4786	* Don't try this for allocations that are allowed to ignore
4787	* watermarks, as the ALLOC_NO_WATERMARKS attempt didn't yet happen.
4788	*/
4789	if (can_direct_reclaim && can_compact &&
4790	(costly_order \|\|
4791	(order > 0 && ac->migratetype != MIGRATE_MOVABLE))
4792	&& !gfp_pfmemalloc_allowed(gfp_mask)) {
4793	page = __alloc_pages_direct_compact(gfp_mask, order,
4794	alloc_flags, ac,
4795	INIT_COMPACT_PRIORITY,
4796	&compact_result);
4797	if (page)
4798	goto got_pg;
4799
4800	/*
4801	* Checks for costly allocations with __GFP_NORETRY, which
4802	* includes some THP page fault allocations
4803	*/
4804	if (costly_order && (gfp_mask & __GFP_NORETRY)) {
4805	/*
4806	* If allocating entire pageblock(s) and compaction
4807	* failed because all zones are below low watermarks
4808	* or is prohibited because it recently failed at this
4809	* order, fail immediately unless the allocator has
4810	* requested compaction and reclaim retry.
4811	*
4812	* Reclaim is
4813	* - potentially very expensive because zones are far
4814	* below their low watermarks or this is part of very
4815	* bursty high order allocations,
4816	* - not guaranteed to help because isolate_freepages()
4817	* may not iterate over freed pages as part of its
4818	* linear scan, and
4819	* - unlikely to make entire pageblocks free on its
4820	* own.
4821	*/
4822	if (compact_result == COMPACT_SKIPPED \|\|
4823	compact_result == COMPACT_DEFERRED)
4824	goto nopage;
4825
4826	/*
4827	* Looks like reclaim/compaction is worth trying, but
4828	* sync compaction could be very expensive, so keep
4829	* using async compaction.
4830	*/
4831	compact_priority = INIT_COMPACT_PRIORITY;
4832	}
4833	}
4834
4835	retry:
4836	/*
4837	* Deal with possible cpuset update races or zonelist updates to avoid
4838	* infinite retries.
4839	*/
4840	if (check_retry_cpuset(cpuset_mems_cookie, ac) \|\|
4841	check_retry_zonelist(zonelist_iter_cookie))
4842	goto restart;
4843
4844	/* Ensure kswapd doesn't accidentally go to sleep as long as we loop */
4845	if (alloc_flags & ALLOC_KSWAPD)
4846	wake_all_kswapds(order, gfp_mask, ac);
4847
4848	reserve_flags = __gfp_pfmemalloc_flags(gfp_mask);	4803	reserve_flags = __gfp_pfmemalloc_flags(gfp_mask);
4849	if (reserve_flags)	4804	if (reserve_flags)
4850	alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, reserve_flags) \|	4805	alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, reserve_flags) \|
@@ -4859,12 +4814,18 @@ retry: · 8 unmodified lines
4859	ac->nodemask = NULL;	4814	ac->nodemask = NULL;
4860	ac->preferred_zoneref = first_zones_zonelist(ac->zonelist,	4815	ac->preferred_zoneref = first_zones_zonelist(ac->zonelist,
4861	ac->highest_zoneidx, ac->nodemask);	4816	ac->highest_zoneidx, ac->nodemask);
4862	}
4863		4817
4864	/* Attempt with potentially adjusted zonelist and alloc_flags */	4818	/*
4865	page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac);	4819	* The first time we adjust anything due to being allowed to
4866	if (page)	4820	* ignore memory policies or watermarks, retry immediately. This
4867	goto got_pg;	4821	* allows us to keep the first allocation attempt optimistic so
		4822	* it can succeed in a zone that is still above watermarks.
		4823	*/
		4824	if (can_retry_reserves) {
		4825	can_retry_reserves = false;
		4826	goto retry;
		4827	}
		4828	}
4868		4829
4869	/* Caller is not willing to reclaim, we can't balance anything */	4830	/* Caller is not willing to reclaim, we can't balance anything */
4870	if (!can_direct_reclaim)	4831	if (!can_direct_reclaim)
@@ -4875,10 +4836,12 @@ retry: · 4 unmodified lines
4875	goto nopage;	4836	goto nopage;
4876		4837
4877	/* Try direct reclaim and then allocating */	4838	/* Try direct reclaim and then allocating */
4878	page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags, ac,	4839	if (!compact_first) {
4879	&did_some_progress);	4840	page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags,
4880	if (page)	4841	ac, &did_some_progress);
4881	goto got_pg;	4842	if (page)
		4843	goto got_pg;
		4844	}
4882		4845
4883	/* Try direct compaction and then allocating */	4846	/* Try direct compaction and then allocating */
4884	page = __alloc_pages_direct_compact(gfp_mask, order, alloc_flags, ac,	4847	page = __alloc_pages_direct_compact(gfp_mask, order, alloc_flags, ac,
@@ -4886,6 +4849,33 @@ retry: · 1 unmodified line
4886	if (page)	4849	if (page)
4887	goto got_pg;	4850	goto got_pg;
4888		4851
		4852	if (compact_first) {
		4853	/*
		4854	* THP page faults may attempt local node only first, but are
		4855	* then allowed to only compact, not reclaim, see
		4856	* alloc_pages_mpol().
		4857	*
		4858	* Compaction has failed above and we don't want such THP
		4859	* allocations to put reclaim pressure on a single node in a
		4860	* situation where other nodes might have plenty of available
		4861	* memory.
		4862	*/
		4863	if (gfp_has_flags(gfp_mask, __GFP_NORETRY \| __GFP_THISNODE))
		4864	goto nopage;
		4865
		4866	/*
		4867	* For the initial compaction attempt we have lowered its
		4868	* priority. Restore it for further retries, if those are
		4869	* allowed. With __GFP_NORETRY there will be a single round of
		4870	* reclaim and compaction with the lowered priority.
		4871	*/
		4872	if (!(gfp_mask & __GFP_NORETRY))
		4873	compact_priority = DEF_COMPACT_PRIORITY;
		4874
		4875	compact_first = false;
		4876	goto retry;
		4877	}
		4878
4889	/* Do not loop if specifically requested */	4879	/* Do not loop if specifically requested */
4890	if (gfp_mask & __GFP_NORETRY)	4880	if (gfp_mask & __GFP_NORETRY)
4891	goto nopage;	4881	goto nopage;
@@ -4898,6 +4888,15 @@ retry: · 6 unmodified lines
4898	!(gfp_mask & __GFP_RETRY_MAYFAIL)))	4888	!(gfp_mask & __GFP_RETRY_MAYFAIL)))
4899	goto nopage;	4889	goto nopage;
4900		4890
		4891	/*
		4892	* Deal with possible cpuset update races or zonelist updates to avoid
		4893	* infinite retries. No "goto retry;" can be placed above this check
		4894	* unless it can execute just once.
		4895	*/
		4896	if (check_retry_cpuset(cpuset_mems_cookie, ac) \|\|
		4897	check_retry_zonelist(zonelist_iter_cookie))
		4898	goto restart;
		4899
4901	if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,	4900	if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,
4902	did_some_progress > 0, &no_progress_loops))	4901	did_some_progress > 0, &no_progress_loops))
4903	goto retry;	4902	goto retry;
@@ -5401,9 +5400,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, · 497 unmodified lines
5401	struct page page = virt_to_page((void )addr);	5400	struct page page = virt_to_page((void )addr);
5402	struct page *last = page + nr;	5401	struct page *last = page + nr;
5403		5402
5404	split_page_owner(page, order, 0);	5403	__split_page(page, order);
5405	pgalloc_tag_split(page_folio(page), order, 0);
5406	split_page_memcg(page, order);
5407	while (page < --last)	5404	while (page < --last)
5408	set_page_refcounted(last);	5405	set_page_refcounted(last);
5409		5406
@@ -6896,7 +6893,7 @@ static int __alloc_contig_migrate_range(struct compact_control *cc, · 1486 unmodified lines
6896	return (ret < 0) ? ret : 0;	6893	return (ret < 0) ? ret : 0;
6897	}	6894	}
6898		6895
6899	static void split_free_pages(struct list_head *list, gfp_t gfp_mask)	6896	static void split_free_frozen_pages(struct list_head *list, gfp_t gfp_mask)
6900	{	6897	{
6901	int order;	6898	int order;
6902		6899
@@ -6908,11 +6905,10 @@ static void split_free_pages(struct list_head *list, gfp_t gfp_mask) · 5 unmodified lines
6908	int i;	6905	int i;
6909		6906
6910	post_alloc_hook(page, order, gfp_mask);	6907	post_alloc_hook(page, order, gfp_mask);
6911	set_page_refcounted(page);
6912	if (!order)	6908	if (!order)
6913	continue;	6909	continue;
6914		6910
6915	split_page(page, order);	6911	__split_page(page, order);
6916		6912
6917	/* Add all subpages to the order-0 head, in sequence. */	6913	/* Add all subpages to the order-0 head, in sequence. */
6918	list_del(&page->lru);	6914	list_del(&page->lru);
@@ -6956,8 +6952,14 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask) · 37 unmodified lines
6956	return 0;	6952	return 0;
6957	}	6953	}
6958		6954
		6955	static void __free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages)
		6956	{
		6957	for (; nr_pages--; pfn++)
		6958	free_frozen_pages(pfn_to_page(pfn), 0);
		6959	}
		6960
6959	/**	6961	/**
6960	* alloc_contig_range() -- tries to allocate given range of pages	6962	* alloc_contig_frozen_range() -- tries to allocate given range of frozen pages
6961	* @start: start PFN to allocate	6963	* @start: start PFN to allocate
6962	* @end: one-past-the-last PFN to allocate	6964	* @end: one-past-the-last PFN to allocate
6963	* @alloc_flags: allocation information	6965	* @alloc_flags: allocation information
@@ -6972,12 +6974,15 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask) · 8 unmodified lines
6972	* pageblocks in the range. Once isolated, the pageblocks should not	6974	* pageblocks in the range. Once isolated, the pageblocks should not
6973	* be modified by others.	6975	* be modified by others.
6974	*	6976	*
6975	* Return: zero on success or negative error code. On success all	6977	* All frozen pages which PFN is in [start, end) are allocated for the
6976	* pages which PFN is in [start, end) are allocated for the caller and	6978	* caller, and they could be freed with free_contig_frozen_range(),
6977	* need to be freed with free_contig_range().	6979	* free_frozen_pages() also could be used to free compound frozen pages
		6980	* directly.
		6981	*
		6982	* Return: zero on success or negative error code.
6978	*/	6983	*/
6979	int alloc_contig_range_noprof(unsigned long start, unsigned long end,	6984	int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
6980	acr_flags_t alloc_flags, gfp_t gfp_mask)	6985	acr_flags_t alloc_flags, gfp_t gfp_mask)
6981	{	6986	{
6982	const unsigned int order = ilog2(end - start);	6987	const unsigned int order = ilog2(end - start);
6983	unsigned long outer_start, outer_end;	6988	unsigned long outer_start, outer_end;
@@ -7093,19 +7098,18 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, · 109 unmodified lines
7093	}	7098	}
7094		7099
7095	if (!(gfp_mask & __GFP_COMP)) {	7100	if (!(gfp_mask & __GFP_COMP)) {
7096	split_free_pages(cc.freepages, gfp_mask);	7101	split_free_frozen_pages(cc.freepages, gfp_mask);
7097		7102
7098	/* Free head and tail (if any) */	7103	/* Free head and tail (if any) */
7099	if (start != outer_start)	7104	if (start != outer_start)
7100	free_contig_range(outer_start, start - outer_start);	7105	__free_contig_frozen_range(outer_start, start - outer_start);
7101	if (end != outer_end)	7106	if (end != outer_end)
7102	free_contig_range(end, outer_end - end);	7107	__free_contig_frozen_range(end, outer_end - end);
7103	} else if (start == outer_start && end == outer_end && is_power_of_2(end - start)) {	7108	} else if (start == outer_start && end == outer_end && is_power_of_2(end - start)) {
7104	struct page *head = pfn_to_page(start);	7109	struct page *head = pfn_to_page(start);
7105		7110
7106	check_new_pages(head, order);	7111	check_new_pages(head, order);
7107	prep_new_page(head, order, gfp_mask, 0);	7112	prep_new_page(head, order, gfp_mask, 0);
7108	set_page_refcounted(head);
7109	} else {	7113	} else {
7110	ret = -EINVAL;	7114	ret = -EINVAL;
7111	WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n",	7115	WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n",
@@ -7115,36 +7119,86 @@ done: · 3 unmodified lines
7115	undo_isolate_page_range(start, end);	7119	undo_isolate_page_range(start, end);
7116	return ret;	7120	return ret;
7117	}	7121	}
7118	EXPORT_SYMBOL(alloc_contig_range_noprof);	7122	EXPORT_SYMBOL(alloc_contig_frozen_range_noprof);
7119		7123
7120	static int __alloc_contig_pages(unsigned long start_pfn,	7124	/**
7121	unsigned long nr_pages, gfp_t gfp_mask)	7125	* alloc_contig_range() -- tries to allocate given range of pages
		7126	* @start: start PFN to allocate
		7127	* @end: one-past-the-last PFN to allocate
		7128	* @alloc_flags: allocation information
		7129	* @gfp_mask: GFP mask.
		7130	*
		7131	* This routine is a wrapper around alloc_contig_frozen_range(), it can't
		7132	* be used to allocate compound pages, the refcount of each allocated page
		7133	* will be set to one.
		7134	*
		7135	* All pages which PFN is in [start, end) are allocated for the caller,
		7136	* and should be freed with free_contig_range() or by manually calling
		7137	* __free_page() on each allocated page.
		7138	*
		7139	* Return: zero on success or negative error code.
		7140	*/
		7141	int alloc_contig_range_noprof(unsigned long start, unsigned long end,
		7142	acr_flags_t alloc_flags, gfp_t gfp_mask)
7122	{	7143	{
7123	unsigned long end_pfn = start_pfn + nr_pages;	7144	int ret;
7124		7145
7125	return alloc_contig_range_noprof(start_pfn, end_pfn, ACR_FLAGS_NONE,	7146	if (WARN_ON(gfp_mask & __GFP_COMP))
7126	gfp_mask);	7147	return -EINVAL;
		7148
		7149	ret = alloc_contig_frozen_range_noprof(start, end, alloc_flags, gfp_mask);
		7150	if (!ret)
		7151	set_pages_refcounted(pfn_to_page(start), end - start);
		7152
		7153	return ret;
7127	}	7154	}
		7155	EXPORT_SYMBOL(alloc_contig_range_noprof);
7128		7156
7129	static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn,	7157	static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn,
7130	unsigned long nr_pages)	7158	unsigned long nr_pages, bool skip_hugetlb,
		7159	bool *skipped_hugetlb)
7131	{	7160	{
7132	unsigned long i, end_pfn = start_pfn + nr_pages;	7161	unsigned long end_pfn = start_pfn + nr_pages;
7133	struct page *page;	7162	struct page *page;
7134		7163
7135	for (i = start_pfn; i < end_pfn; i++) {	7164	while (start_pfn < end_pfn) {
7136	page = pfn_to_online_page(i);	7165	unsigned long step = 1;
		7166
		7167	page = pfn_to_online_page(start_pfn);
7137	if (!page)	7168	if (!page)
7138	return false;	7169	return false;
7139		7170
7140	if (page_zone(page) != z)	7171	if (page_zone(page) != z)
7141	return false;	7172	return false;
7142		7173
7143	if (PageReserved(page))	7174	if (page_is_unmovable(z, page, PB_ISOLATE_MODE_OTHER, &step))
7144	return false;	7175	return false;
7145		7176
7146	if (PageHuge(page))	7177	/*
7147	return false;	7178	* Only consider ranges containing hugepages if those pages are
		7179	* smaller than the requested contiguous region. e.g.:
		7180	* Move 2MB pages to free up a 1GB range.
		7181	* Don't move 1GB pages to free up a 2MB range.
		7182	*
		7183	* This makes contiguous allocation more reliable if multiple
		7184	* hugepage sizes are used without causing needless movement.
		7185	*/
		7186	if (PageHuge(page)) {
		7187	unsigned int order;
		7188
		7189	if (skip_hugetlb) {
		7190	*skipped_hugetlb = true;
		7191	return false;
		7192	}
		7193
		7194	page = compound_head(page);
		7195	order = compound_order(page);
		7196	if ((order >= MAX_FOLIO_ORDER) \|\|
		7197	(nr_pages <= (1 << order)))
		7198	return false;
		7199	}
		7200
		7201	start_pfn += step;
7148	}	7202	}
7149	return true;	7203	return true;
7150	}	7204	}
@@ -7158,7 +7212,7 @@ static bool zone_spans_last_pfn(const struct zone *zone, · 7 unmodified lines
7158	}	7212	}
7159		7213
7160	/**	7214	/**
7161	* alloc_contig_pages() -- tries to find and allocate contiguous range of pages	7215	* alloc_contig_frozen_pages() -- tries to find and allocate contiguous range of frozen pages
7162	* @nr_pages: Number of contiguous pages to allocate	7216	* @nr_pages: Number of contiguous pages to allocate
7163	* @gfp_mask: GFP mask. Node/zone/placement hints limit the search; only some	7217	* @gfp_mask: GFP mask. Node/zone/placement hints limit the search; only some
7164	* action and reclaim modifiers are supported. Reclaim modifiers	7218	* action and reclaim modifiers are supported. Reclaim modifiers
@@ -7166,28 +7220,34 @@ static bool zone_spans_last_pfn(const struct zone *zone, · 1 unmodified line
7166	* @nid: Target node	7220	* @nid: Target node
7167	* @nodemask: Mask for other possible nodes	7221	* @nodemask: Mask for other possible nodes
7168	*	7222	*
7169	* This routine is a wrapper around alloc_contig_range(). It scans over zones	7223	* This routine is a wrapper around alloc_contig_frozen_range(). It scans over
7170	* on an applicable zonelist to find a contiguous pfn range which can then be	7224	* zones on an applicable zonelist to find a contiguous pfn range which can then
7171	* tried for allocation with alloc_contig_range(). This routine is intended	7225	* be tried for allocation with alloc_contig_frozen_range(). This routine is
7172	* for allocation requests which can not be fulfilled with the buddy allocator.	7226	* intended for allocation requests which can not be fulfilled with the buddy
		7227	* allocator.
7173	*	7228	*
7174	* The allocated memory is always aligned to a page boundary. If nr_pages is a	7229	* The allocated memory is always aligned to a page boundary. If nr_pages is a
7175	* power of two, then allocated range is also guaranteed to be aligned to same	7230	* power of two, then allocated range is also guaranteed to be aligned to same
7176	* nr_pages (e.g. 1GB request would be aligned to 1GB).	7231	* nr_pages (e.g. 1GB request would be aligned to 1GB).
7177	*	7232	*
7178	* Allocated pages can be freed with free_contig_range() or by manually calling	7233	* Allocated frozen pages need be freed with free_contig_frozen_range(),
7179	* __free_page() on each allocated page.	7234	* or by manually calling free_frozen_pages() on each allocated frozen
		7235	* non-compound page, for compound frozen pages could be freed with
		7236	* free_frozen_pages() directly.
7180	*	7237	*
7181	* Return: pointer to contiguous pages on success, or NULL if not successful.	7238	* Return: pointer to contiguous frozen pages on success, or NULL if not successful.
7182	*/	7239	*/
7183	struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask,	7240	struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages,
7184	int nid, nodemask_t *nodemask)	7241	gfp_t gfp_mask, int nid, nodemask_t *nodemask)
7185	{	7242	{
7186	unsigned long ret, pfn, flags;	7243	unsigned long ret, pfn, flags;
7187	struct zonelist *zonelist;	7244	struct zonelist *zonelist;
7188	struct zone *zone;	7245	struct zone *zone;
7189	struct zoneref *z;	7246	struct zoneref *z;
		7247	bool skip_hugetlb = true;
		7248	bool skipped_hugetlb = false;
7190		7249
		7250	retry:
7191	zonelist = node_zonelist(nid, gfp_mask);	7251	zonelist = node_zonelist(nid, gfp_mask);
7192	for_each_zone_zonelist_nodemask(zone, z, zonelist,	7252	for_each_zone_zonelist_nodemask(zone, z, zonelist,
7193	gfp_zone(gfp_mask), nodemask) {	7253	gfp_zone(gfp_mask), nodemask) {
@@ -7195,16 +7255,20 @@ struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, · 1 unmodified line
7195		7255
7196	pfn = ALIGN(zone->zone_start_pfn, nr_pages);	7256	pfn = ALIGN(zone->zone_start_pfn, nr_pages);
7197	while (zone_spans_last_pfn(zone, pfn, nr_pages)) {	7257	while (zone_spans_last_pfn(zone, pfn, nr_pages)) {
7198	if (pfn_range_valid_contig(zone, pfn, nr_pages)) {	7258	if (pfn_range_valid_contig(zone, pfn, nr_pages,
		7259	skip_hugetlb,
		7260	&skipped_hugetlb)) {
7199	/*	7261	/*
7200	* We release the zone lock here because	7262	* We release the zone lock here because
7201	* alloc_contig_range() will also lock the zone	7263	* alloc_contig_frozen_range() will also lock
7202	* at some point. If there's an allocation	7264	* the zone at some point. If there's an
7203	* spinning on this lock, it may win the race	7265	* allocation spinning on this lock, it may
7204	* and cause alloc_contig_range() to fail...	7266	* win the race and cause allocation to fail.
7205	*/	7267	*/
7206	spin_unlock_irqrestore(&zone->lock, flags);	7268	spin_unlock_irqrestore(&zone->lock, flags);
7207	ret = __alloc_contig_pages(pfn, nr_pages,	7269	ret = alloc_contig_frozen_range_noprof(pfn,
		7270	pfn + nr_pages,
		7271	ACR_FLAGS_NONE,
7208	gfp_mask);	7272	gfp_mask);
7209	if (!ret)	7273	if (!ret)
7210	return pfn_to_page(pfn);	7274	return pfn_to_page(pfn);
@@ -7214,35 +7278,96 @@ struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, · 3 unmodified lines
7214	}	7278	}
7215	spin_unlock_irqrestore(&zone->lock, flags);	7279	spin_unlock_irqrestore(&zone->lock, flags);
7216	}	7280	}
		7281	/*
		7282	* If we failed, retry the search, but treat regions with HugeTLB pages
		7283	* as valid targets. This retains fast-allocations on first pass
		7284	* without trying to migrate HugeTLB pages (which may fail). On the
		7285	* second pass, we will try moving HugeTLB pages when those pages are
		7286	* smaller than the requested contiguous region size.
		7287	*/
		7288	if (skip_hugetlb && skipped_hugetlb) {
		7289	skip_hugetlb = false;
		7290	goto retry;
		7291	}
7217	return NULL;	7292	return NULL;
7218	}	7293	}
7219	#endif /* CONFIG_CONTIG_ALLOC */	7294	EXPORT_SYMBOL(alloc_contig_frozen_pages_noprof);
7220		7295
7221	void free_contig_range(unsigned long pfn, unsigned long nr_pages)	7296	/**
		7297	* alloc_contig_pages() -- tries to find and allocate contiguous range of pages
		7298	* @nr_pages: Number of contiguous pages to allocate
		7299	* @gfp_mask: GFP mask.
		7300	* @nid: Target node
		7301	* @nodemask: Mask for other possible nodes
		7302	*
		7303	* This routine is a wrapper around alloc_contig_frozen_pages(), it can't
		7304	* be used to allocate compound pages, the refcount of each allocated page
		7305	* will be set to one.
		7306	*
		7307	* Allocated pages can be freed with free_contig_range() or by manually
		7308	* calling __free_page() on each allocated page.
		7309	*
		7310	* Return: pointer to contiguous pages on success, or NULL if not successful.
		7311	*/
		7312	struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask,
		7313	int nid, nodemask_t *nodemask)
7222	{	7314	{
7223	unsigned long count = 0;	7315	struct page *page;
7224	struct folio *folio = pfn_folio(pfn);
7225		7316
7226	if (folio_test_large(folio)) {	7317	if (WARN_ON(gfp_mask & __GFP_COMP))
7227	int expected = folio_nr_pages(folio);	7318	return NULL;
7228		7319
7229	if (nr_pages == expected)	7320	page = alloc_contig_frozen_pages_noprof(nr_pages, gfp_mask, nid,
7230	folio_put(folio);	7321	nodemask);
7231	else	7322	if (page)
7232	WARN(true, "PFN %lu: nr_pages %lu != expected %d\n",	7323	set_pages_refcounted(page, nr_pages);
7233	pfn, nr_pages, expected);	7324
		7325	return page;
		7326	}
		7327	EXPORT_SYMBOL(alloc_contig_pages_noprof);
		7328
		7329	/**
		7330	* free_contig_frozen_range() -- free the contiguous range of frozen pages
		7331	* @pfn: start PFN to free
		7332	* @nr_pages: Number of contiguous frozen pages to free
		7333	*
		7334	* This can be used to free the allocated compound/non-compound frozen pages.
		7335	*/
		7336	void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages)
		7337	{
		7338	struct page *first_page = pfn_to_page(pfn);
		7339	const unsigned int order = ilog2(nr_pages);
		7340
		7341	if (WARN_ON_ONCE(first_page != compound_head(first_page)))
		7342	return;
		7343
		7344	if (PageHead(first_page)) {
		7345	WARN_ON_ONCE(order != compound_order(first_page));
		7346	free_frozen_pages(first_page, order);
7234	return;	7347	return;
7235	}	7348	}
7236		7349
7237	for (; nr_pages--; pfn++) {	7350	__free_contig_frozen_range(pfn, nr_pages);
7238	struct page *page = pfn_to_page(pfn);	7351	}
		7352	EXPORT_SYMBOL(free_contig_frozen_range);
7239		7353
7240	count += page_count(page) != 1;	7354	/**
7241	__free_page(page);	7355	* free_contig_range() -- free the contiguous range of pages
7242	}	7356	* @pfn: start PFN to free
7243	WARN(count != 0, "%lu pages are still in use!\n", count);	7357	* @nr_pages: Number of contiguous pages to free
		7358	*
		7359	* This can be only used to free the allocated non-compound pages.
		7360	*/
		7361	void free_contig_range(unsigned long pfn, unsigned long nr_pages)
		7362	{
		7363	if (WARN_ON_ONCE(PageHead(pfn_to_page(pfn))))
		7364	return;
		7365
		7366	for (; nr_pages--; pfn++)
		7367	__free_page(pfn_to_page(pfn));
7244	}	7368	}
7245	EXPORT_SYMBOL(free_contig_range);	7369	EXPORT_SYMBOL(free_contig_range);
		7370	#endif /* CONFIG_CONTIG_ALLOC */
7246		7371
7247	/*	7372	/*
7248	* Effectively disable pcplists for the zone by setting the high limit to 0	7373	* Effectively disable pcplists for the zone by setting the high limit to 0
@@ -7658,7 +7783,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned · 409 unmodified lines
7658	* unsafe in NMI. If spin_trylock() is called from hard IRQ the current	7783	* unsafe in NMI. If spin_trylock() is called from hard IRQ the current
7659	* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will	7784	* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will
7660	* mark the task as the owner of another rt_spin_lock which will	7785	* mark the task as the owner of another rt_spin_lock which will
7661	* confuse PI logic, so return immediately if called form hard IRQ or	7786	* confuse PI logic, so return immediately if called from hard IRQ or
7662	* NMI.	7787	* NMI.
7663	*	7788	*
7664	* Note, irqs_disabled() case is ok. This function can be called	7789	* Note, irqs_disabled() case is ok. This function can be called

mm/page_io.c+2−2

@@ -450,14 +450,14 @@ void __swap_writepage(struct folio folio, struct swap_iocb *swap_plug) · 449 unmodified lines
450		450
451	VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);	451	VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
452	/*	452	/*
453	* ->flags can be updated non-atomicially (scan_swap_map_slots),	453	* ->flags can be updated non-atomically (scan_swap_map_slots),
454	* but that will never affect SWP_FS_OPS, so the data_race	454	* but that will never affect SWP_FS_OPS, so the data_race
455	* is safe.	455	* is safe.
456	*/	456	*/
457	if (data_race(sis->flags & SWP_FS_OPS))	457	if (data_race(sis->flags & SWP_FS_OPS))
458	swap_writepage_fs(folio, swap_plug);	458	swap_writepage_fs(folio, swap_plug);
459	/*	459	/*
460	* ->flags can be updated non-atomicially (scan_swap_map_slots),	460	* ->flags can be updated non-atomically (scan_swap_map_slots),
461	* but that will never affect SWP_SYNCHRONOUS_IO, so the data_race	461	* but that will never affect SWP_SYNCHRONOUS_IO, so the data_race
462	* is safe.	462	* is safe.
463	*/	463	*/

mm/page_isolation.c+100−89

 #define CREATE_TRACE_POINTS
 #include <trace/events/page_isolation.h>
+bool page_is_unmovable(struct zone *zone, struct page *page,
+		enum pb_isolate_mode mode, unsigned long *step)
+{
+	/*
+	 * Both, bootmem allocations and memory holes are marked
+	 * PG_reserved and are unmovable. We can even have unmovable
+	 * allocations inside ZONE_MOVABLE, for example when
+	 * specifying "movablecore".
+	 */
+	if (PageReserved(page))
+		return true;
+	/*
+	 * If the zone is movable and we have ruled out all reserved
+	 * pages then it should be reasonably safe to assume the rest
+	 * is movable.
+	 */
+	if (zone_idx(zone) == ZONE_MOVABLE)
+		return false;
+	/*
+	 * Hugepages are not in LRU lists, but they're movable.
+	 * THPs are on the LRU, but need to be counted as #small pages.
+	 * We need not scan over tail pages because we don't
+	 * handle each tail page individually in migration.
+	 */
+	if (PageHuge(page) || PageCompound(page)) {
+		struct folio *folio = page_folio(page);
+		if (folio_test_hugetlb(folio)) {
+			struct hstate *h;
+			if (!IS_ENABLED(CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION))
+				return true;
+			/*
+			 * The huge page may be freed so can not
+			 * use folio_hstate() directly.
+			 */
+			h = size_to_hstate(folio_size(folio));
+			if (h && !hugepage_migration_supported(h))
+				return true;
+		} else if (!folio_test_lru(folio)) {
+			return true;
+		}
+		*step = folio_nr_pages(folio) - folio_page_idx(folio, page);
+		return false;
+	}
+	/*
+	 * We can't use page_count without pin a page
+	 * because another CPU can free compound page.
+	 * This check already skips compound tails of THP
+	 * because their page->_refcount is zero at all time.
+	 */
+	if (!page_ref_count(page)) {
+		if (PageBuddy(page))
+			*step = (1 << buddy_order(page));
+		return false;
+	}
+	/*
+	 * The HWPoisoned page may be not in buddy system, and
+	 * page_count() is not 0.
+	 */
+	if ((mode == PB_ISOLATE_MODE_MEM_OFFLINE) && PageHWPoison(page))
+		return false;
+	/*
+	 * We treat all PageOffline() pages as movable when offlining
+	 * to give drivers a chance to decrement their reference count
+	 * in MEM_GOING_OFFLINE in order to indicate that these pages
+	 * can be offlined as there are no direct references anymore.
+	 * For actually unmovable PageOffline() where the driver does
+	 * not support this, we will fail later when trying to actually
+	 * move these pages that still have a reference count > 0.
+	 * (false negatives in this function only)
+	 */
+	if ((mode == PB_ISOLATE_MODE_MEM_OFFLINE) && PageOffline(page))
+		return false;
+	if (PageLRU(page) || page_has_movable_ops(page))
+		return false;
+	/*
+	 * If there are RECLAIMABLE pages, we need to check
+	 * it.  But now, memory offline itself doesn't call
+	 * shrink_node_slabs() and it still to be fixed.
+	 */
+	return true;
+}
 /*
  * This function checks whether the range [start_pfn, end_pfn) includes
  * unmovable pages or not. The range must fall into a single pageblock and
 {
 	struct page *page = pfn_to_page(start_pfn);
 	struct zone *zone = page_zone(page);
-	unsigned long pfn;
 	VM_BUG_ON(pageblock_start_pfn(start_pfn) !=
 		  pageblock_start_pfn(end_pfn - 1));
 		return page;
 	}
-	for (pfn = start_pfn; pfn < end_pfn; pfn++) {
+	while (start_pfn < end_pfn) {
-		page = pfn_to_page(pfn);
+		unsigned long step = 1;
-		/*
+		page = pfn_to_page(start_pfn);
-		 * Both, bootmem allocations and memory holes are marked
+		if (page_is_unmovable(zone, page, mode, &step))
-		 * PG_reserved and are unmovable. We can even have unmovable
-		 * allocations inside ZONE_MOVABLE, for example when
-		 * specifying "movablecore".
-		 */
-		if (PageReserved(page))
 			return page;
-		/*
+		start_pfn += step;
-		 * If the zone is movable and we have ruled out all reserved
-		 * pages then it should be reasonably safe to assume the rest
-		 * is movable.
-		 */
-		if (zone_idx(zone) == ZONE_MOVABLE)
-			continue;
-		/*
-		 * Hugepages are not in LRU lists, but they're movable.
-		 * THPs are on the LRU, but need to be counted as #small pages.
-		 * We need not scan over tail pages because we don't
-		 * handle each tail page individually in migration.
-		 */
-		if (PageHuge(page) || PageTransCompound(page)) {
-			struct folio *folio = page_folio(page);
-			unsigned int skip_pages;
-			if (PageHuge(page)) {
-				struct hstate *h;
-				/*
-				 * The huge page may be freed so can not
-				 * use folio_hstate() directly.
-				 */
-				h = size_to_hstate(folio_size(folio));
-				if (h && !hugepage_migration_supported(h))
-					return page;
-			} else if (!folio_test_lru(folio)) {
-				return page;
-			}
-			skip_pages = folio_nr_pages(folio) - folio_page_idx(folio, page);
-			pfn += skip_pages - 1;
-			continue;
-		}
-		/*
-		 * We can't use page_count without pin a page
-		 * because another CPU can free compound page.
-		 * This check already skips compound tails of THP
-		 * because their page->_refcount is zero at all time.
-		 */
-		if (!page_ref_count(page)) {
-			if (PageBuddy(page))
-				pfn += (1 << buddy_order(page)) - 1;
-			continue;
-		}
-		/*
-		 * The HWPoisoned page may be not in buddy system, and
-		 * page_count() is not 0.
-		 */
-		if ((mode == PB_ISOLATE_MODE_MEM_OFFLINE) && PageHWPoison(page))
-			continue;
-		/*
-		 * We treat all PageOffline() pages as movable when offlining
-		 * to give drivers a chance to decrement their reference count
-		 * in MEM_GOING_OFFLINE in order to indicate that these pages
-		 * can be offlined as there are no direct references anymore.
-		 * For actually unmovable PageOffline() where the driver does
-		 * not support this, we will fail later when trying to actually
-		 * move these pages that still have a reference count > 0.
-		 * (false negatives in this function only)
-		 */
-		if ((mode == PB_ISOLATE_MODE_MEM_OFFLINE) && PageOffline(page))
-			continue;
-		if (PageLRU(page) || page_has_movable_ops(page))
-			continue;
-		/*
-		 * If there are RECLAIMABLE pages, we need to check
-		 * it.  But now, memory offline itself doesn't call
-		 * shrink_node_slabs() and it still to be fixed.
-		 */
-		return page;
 	}
 	return NULL;
 }
  * pageblock. When not all pageblocks within a page are isolated at the same
  * time, free page accounting can go wrong. For example, in the case of
  * MAX_PAGE_ORDER = pageblock_order + 1, a MAX_PAGE_ORDER page has two
- * pagelbocks.
+ * pageblocks.
  * [      MAX_PAGE_ORDER         ]
  * [  pageblock0  |  pageblock1  ]
  * When either pageblock is isolated, if it is a free page, the page is not

mm/page_reporting.c+1−1

@@ -123,7 +123,7 @@ page_reporting_drain(struct page_reporting_dev_info *prdev, · 122 unmodified lines
123	continue;	123	continue;
124		124
125	/*	125	/*
126	* If page was not comingled with another page we can	126	* If page was not commingled with another page we can
127	* consider the result to be "reported" since the page	127	* consider the result to be "reported" since the page
128	* hasn't been modified, otherwise we will need to	128	* hasn't been modified, otherwise we will need to
129	* report on the new larger page when we make our way	129	* report on the new larger page when we make our way

mm/page_table_check.c+22−19

 	rcu_read_unlock();
 }
-void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte)
+void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr,
+				  pte_t pte)
 {
 	if (&init_mm == mm)
 		return;
-	if (pte_user_accessible_page(pte)) {
+	if (pte_user_accessible_page(pte, addr)) {
 		page_table_check_clear(pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT);
 	}
 }
 EXPORT_SYMBOL(__page_table_check_pte_clear);
-void __page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd)
+void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr,
+				  pmd_t pmd)
 {
 	if (&init_mm == mm)
 		return;
-	if (pmd_user_accessible_page(pmd)) {
+	if (pmd_user_accessible_page(pmd, addr)) {
 		page_table_check_clear(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT);
 	}
 }
 EXPORT_SYMBOL(__page_table_check_pmd_clear);
-void __page_table_check_pud_clear(struct mm_struct *mm, pud_t pud)
+void __page_table_check_pud_clear(struct mm_struct *mm, unsigned long addr,
+				  pud_t pud)
 {
 	if (&init_mm == mm)
 		return;
-	if (pud_user_accessible_page(pud)) {
+	if (pud_user_accessible_page(pud, addr)) {
 		page_table_check_clear(pud_pfn(pud), PUD_SIZE >> PAGE_SHIFT);
 	}
 }
 	}
 }
-void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte,
+void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr,
-		unsigned int nr)
+				 pte_t *ptep, pte_t pte, unsigned int nr)
 {
 	unsigned int i;
 	page_table_check_pte_flags(pte);
 	for (i = 0; i < nr; i++)
-		__page_table_check_pte_clear(mm, ptep_get(ptep + i));
+		__page_table_check_pte_clear(mm, addr + PAGE_SIZE * i, ptep_get(ptep + i));
-	if (pte_user_accessible_page(pte))
+	if (pte_user_accessible_page(pte, addr))
 		page_table_check_set(pte_pfn(pte), nr, pte_write(pte));
 }
 EXPORT_SYMBOL(__page_table_check_ptes_set);
 	}
 }
-void __page_table_check_pmds_set(struct mm_struct *mm, pmd_t *pmdp, pmd_t pmd,
+void __page_table_check_pmds_set(struct mm_struct *mm, unsigned long addr,
-		unsigned int nr)
+		pmd_t *pmdp, pmd_t pmd, unsigned int nr)
 {
 	unsigned long stride = PMD_SIZE >> PAGE_SHIFT;
 	unsigned int i;
 	page_table_check_pmd_flags(pmd);
 	for (i = 0; i < nr; i++)
-		__page_table_check_pmd_clear(mm, *(pmdp + i));
+		__page_table_check_pmd_clear(mm, addr + PMD_SIZE * i, *(pmdp + i));
-	if (pmd_user_accessible_page(pmd))
+	if (pmd_user_accessible_page(pmd, addr))
 		page_table_check_set(pmd_pfn(pmd), stride * nr, pmd_write(pmd));
 }
 EXPORT_SYMBOL(__page_table_check_pmds_set);
-void __page_table_check_puds_set(struct mm_struct *mm, pud_t *pudp, pud_t pud,
+void __page_table_check_puds_set(struct mm_struct *mm, unsigned long addr,
-		unsigned int nr)
+		pud_t *pudp, pud_t pud,	unsigned int nr)
 {
 	unsigned long stride = PUD_SIZE >> PAGE_SHIFT;
 	unsigned int i;
 		return;
 	for (i = 0; i < nr; i++)
-		__page_table_check_pud_clear(mm, *(pudp + i));
+		__page_table_check_pud_clear(mm, addr + PUD_SIZE * i, *(pudp + i));
-	if (pud_user_accessible_page(pud))
+	if (pud_user_accessible_page(pud, addr))
 		page_table_check_set(pud_pfn(pud), stride * nr, pud_write(pud));
 }
 EXPORT_SYMBOL(__page_table_check_puds_set);
 		if (WARN_ON(!ptep))
 			return;
 		for (i = 0; i < PTRS_PER_PTE; i++) {
-			__page_table_check_pte_clear(mm, ptep_get(ptep));
+			__page_table_check_pte_clear(mm, addr, ptep_get(ptep));
 			addr += PAGE_SIZE;
 			ptep++;
 		}

mm/pagewalk.c+2−1

@@ -313,7 +313,8 @@ static unsigned long hugetlb_entry_end(struct hstate *h, unsigned long addr, · 312 unmodified lines
313	unsigned long end)	313	unsigned long end)
314	{	314	{
315	unsigned long boundary = (addr & huge_page_mask(h)) + huge_page_size(h);	315	unsigned long boundary = (addr & huge_page_mask(h)) + huge_page_size(h);
316	return boundary < end ? boundary : end;	316
		317	return min(boundary, end);
317	}	318	}
318		319
319	static int walk_hugetlb_range(unsigned long addr, unsigned long end,	320	static int walk_hugetlb_range(unsigned long addr, unsigned long end,

mm/percpu.c+14−1

 	int bit_off, bits, end, oslot, freed;
 	lockdep_assert_held(&pcpu_lock);
-	pcpu_stats_area_dealloc(chunk);
 	oslot = pcpu_chunk_slot(chunk);
 	bit_off = off / PCPU_MIN_ALLOC_SIZE;
+	/* check invalid free */
+	if (!test_bit(bit_off, chunk->alloc_map) ||
+	    !test_bit(bit_off, chunk->bound_map))
+		return 0;
 	/* find end index */
 	end = find_next_bit(chunk->bound_map, pcpu_chunk_map_bits(chunk),
 			    bit_off + 1);
 	pcpu_chunk_relocate(chunk, oslot);
+	pcpu_stats_area_dealloc(chunk);
 	return freed;
 }
 	spin_lock_irqsave(&pcpu_lock, flags);
 	size = pcpu_free_area(chunk, off);
+	if (size == 0) {
+		spin_unlock_irqrestore(&pcpu_lock, flags);
+		/* invalid percpu free */
+		WARN_ON_ONCE(1);
+		return;
+	}
 	pcpu_alloc_tag_free_hook(chunk, off, size);

mm/pt_reclaim.c+0−72

@@ -1,72 +0,0 @@
1	// SPDX-License-Identifier: GPL-2.0
2	#include <linux/hugetlb.h>
3	#include <linux/pgalloc.h>
4
5	#include <asm-generic/tlb.h>
6
7	#include "internal.h"
8
9	bool reclaim_pt_is_enabled(unsigned long start, unsigned long end,
10	struct zap_details *details)
11	{
12	return details && details->reclaim_pt && (end - start >= PMD_SIZE);
13	}
14
15	bool try_get_and_clear_pmd(struct mm_struct mm, pmd_t pmd, pmd_t *pmdval)
16	{
17	spinlock_t *pml = pmd_lockptr(mm, pmd);
18
19	if (!spin_trylock(pml))
20	return false;
21
22	*pmdval = pmdp_get_lockless(pmd);
23	pmd_clear(pmd);
24	spin_unlock(pml);
25
26	return true;
27	}
28
29	void free_pte(struct mm_struct mm, unsigned long addr, struct mmu_gather tlb,
30	pmd_t pmdval)
31	{
32	pte_free_tlb(tlb, pmd_pgtable(pmdval), addr);
33	mm_dec_nr_ptes(mm);
34	}
35
36	void try_to_free_pte(struct mm_struct mm, pmd_t pmd, unsigned long addr,
37	struct mmu_gather *tlb)
38	{
39	pmd_t pmdval;
40	spinlock_t pml, ptl = NULL;
41	pte_t start_pte, pte;
42	int i;
43
44	pml = pmd_lock(mm, pmd);
45	start_pte = pte_offset_map_rw_nolock(mm, pmd, addr, &pmdval, &ptl);
46	if (!start_pte)
47	goto out_ptl;
48	if (ptl != pml)
49	spin_lock_nested(ptl, SINGLE_DEPTH_NESTING);
50
51	/* Check if it is empty PTE page */
52	for (i = 0, pte = start_pte; i < PTRS_PER_PTE; i++, pte++) {
53	if (!pte_none(ptep_get(pte)))
54	goto out_ptl;
55	}
56	pte_unmap(start_pte);
57
58	pmd_clear(pmd);
59
60	if (ptl != pml)
61	spin_unlock(ptl);
62	spin_unlock(pml);
63
64	free_pte(mm, addr, tlb, pmdval);
65
66	return;
67	out_ptl:
68	if (start_pte)
69	pte_unmap_unlock(start_pte, ptl);
70	if (ptl != pml)
71	spin_unlock(pml);
72	}

mm/readahead.c+1−1

@@ -439,7 +439,7 @@ static unsigned long get_next_ra_size(struct file_ra_state *ra, · 438 unmodified lines
439	* based on I/O request size and the max_readahead.	439	* based on I/O request size and the max_readahead.
440	*	440	*
441	* The code ramps up the readahead size aggressively at first, but slow down as	441	* The code ramps up the readahead size aggressively at first, but slow down as
442	* it approaches max_readhead.	442	* it approaches max_readahead.
443	*/	443	*/
444		444
445	static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index,	445	static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index,

mm/rmap.c+205−119

@@ -1,8 +1,8 @@
		1	// SPDX-License-Identifier: GPL-2.0
1	/*	2	/*
2	* mm/rmap.c - physical to virtual reverse mappings	3	* mm/rmap.c - physical to virtual reverse mappings
3	*	4	*
4	* Copyright 2001, Rik van Riel <riel@conectiva.com.br>	5	* Copyright 2001, Rik van Riel <riel@conectiva.com.br>
5	* Released under the General Public License (GPL).
6	*	6	*
7	* Simple, low overhead reverse mapping scheme.	7	* Simple, low overhead reverse mapping scheme.
8	* Please try to keep this thing as modular as possible.	8	* Please try to keep this thing as modular as possible.
@@ -82,6 +82,7 @@ · 73 unmodified lines
82	#include <trace/events/migrate.h>	82	#include <trace/events/migrate.h>
83		83
84	#include "internal.h"	84	#include "internal.h"
		85	#include "swap.h"
85		86
86	static struct kmem_cache *anon_vma_cachep;	87	static struct kmem_cache *anon_vma_cachep;
87	static struct kmem_cache *anon_vma_chain_cachep;	88	static struct kmem_cache *anon_vma_chain_cachep;
@@ -146,14 +147,13 @@ static void anon_vma_chain_free(struct anon_vma_chain *anon_vma_chain) · 58 unmodified lines
146	kmem_cache_free(anon_vma_chain_cachep, anon_vma_chain);	147	kmem_cache_free(anon_vma_chain_cachep, anon_vma_chain);
147	}	148	}
148		149
149	static void anon_vma_chain_link(struct vm_area_struct *vma,	150	static void anon_vma_chain_assign(struct vm_area_struct *vma,
150	struct anon_vma_chain *avc,	151	struct anon_vma_chain *avc,
151	struct anon_vma *anon_vma)	152	struct anon_vma *anon_vma)
152	{	153	{
153	avc->vma = vma;	154	avc->vma = vma;
154	avc->anon_vma = anon_vma;	155	avc->anon_vma = anon_vma;
155	list_add(&avc->same_vma, &vma->anon_vma_chain);	156	list_add(&avc->same_vma, &vma->anon_vma_chain);
156	anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
157	}	157	}
158		158
159	/**	159	/**
@@ -210,7 +210,8 @@ int __anon_vma_prepare(struct vm_area_struct *vma) · 50 unmodified lines
210	spin_lock(&mm->page_table_lock);	210	spin_lock(&mm->page_table_lock);
211	if (likely(!vma->anon_vma)) {	211	if (likely(!vma->anon_vma)) {
212	vma->anon_vma = anon_vma;	212	vma->anon_vma = anon_vma;
213	anon_vma_chain_link(vma, avc, anon_vma);	213	anon_vma_chain_assign(vma, avc, anon_vma);
		214	anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
214	anon_vma->num_active_vmas++;	215	anon_vma->num_active_vmas++;
215	allocated = NULL;	216	allocated = NULL;
216	avc = NULL;	217	avc = NULL;
@@ -231,97 +232,141 @@ int __anon_vma_prepare(struct vm_area_struct *vma) · 14 unmodified lines
231	return -ENOMEM;	232	return -ENOMEM;
232	}	233	}
233		234
234	/*	235	static void check_anon_vma_clone(struct vm_area_struct *dst,
235	* This is a useful helper function for locking the anon_vma root as	236	struct vm_area_struct *src,
236	* we traverse the vma->anon_vma_chain, looping over anon_vma's that	237	enum vma_operation operation)
237	* have the same vma.	238	{
238	*	239	/* The write lock must be held. */
239	* Such anon_vma's should have the same root, so you'd expect to see	240	mmap_assert_write_locked(src->vm_mm);
240	* just a single mutex_lock for the whole traversal.	241	/* If not a fork then must be on same mm. */
241	*/	242	VM_WARN_ON_ONCE(operation != VMA_OP_FORK && dst->vm_mm != src->vm_mm);
242	static inline struct anon_vma lock_anon_vma_root(struct anon_vma root, struct anon_vma *anon_vma)	243
243	{	244	/* If we have anything to do src->anon_vma must be provided. */
244	struct anon_vma *new_root = anon_vma->root;	245	VM_WARN_ON_ONCE(!src->anon_vma && !list_empty(&src->anon_vma_chain));
245	if (new_root != root) {	246	VM_WARN_ON_ONCE(!src->anon_vma && dst->anon_vma);
246	if (WARN_ON_ONCE(root))	247	/* We are establishing a new anon_vma_chain. */
247	up_write(&root->rwsem);	248	VM_WARN_ON_ONCE(!list_empty(&dst->anon_vma_chain));
248	root = new_root;	249	/*
249	down_write(&root->rwsem);	250	* On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma
250	}	251	* must be the same across dst and src.
251	return root;	252	*/
		253	VM_WARN_ON_ONCE(dst->anon_vma && dst->anon_vma != src->anon_vma);
		254	/*
		255	* Essentially equivalent to above - if not a no-op, we should expect
		256	* dst->anon_vma to be set for everything except a fork.
		257	*/
		258	VM_WARN_ON_ONCE(operation != VMA_OP_FORK && src->anon_vma &&
		259	!dst->anon_vma);
		260	/* For the anon_vma to be compatible, it can only be singular. */
		261	VM_WARN_ON_ONCE(operation == VMA_OP_MERGE_UNFAULTED &&
		262	!list_is_singular(&src->anon_vma_chain));
		263	#ifdef CONFIG_PER_VMA_LOCK
		264	/* Only merging an unfaulted VMA leaves the destination attached. */
		265	VM_WARN_ON_ONCE(operation != VMA_OP_MERGE_UNFAULTED &&
		266	vma_is_attached(dst));
		267	#endif
252	}	268	}
253		269
254	static inline void unlock_anon_vma_root(struct anon_vma *root)	270	static void maybe_reuse_anon_vma(struct vm_area_struct *dst,
		271	struct anon_vma *anon_vma)
255	{	272	{
256	if (root)	273	/* If already populated, nothing to do.*/
257	up_write(&root->rwsem);	274	if (dst->anon_vma)
		275	return;
		276
		277	/*
		278	* We reuse an anon_vma if any linking VMAs were unmapped and it has
		279	* only a single child at most.
		280	*/
		281	if (anon_vma->num_active_vmas > 0)
		282	return;
		283	if (anon_vma->num_children > 1)
		284	return;
		285
		286	dst->anon_vma = anon_vma;
		287	anon_vma->num_active_vmas++;
258	}	288	}
259		289
260	/*	290	static void cleanup_partial_anon_vmas(struct vm_area_struct *vma);
261	* Attach the anon_vmas from src to dst.	291
262	* Returns 0 on success, -ENOMEM on failure.	292	/**
263	*	293	* anon_vma_clone - Establishes new anon_vma_chain objects in @dst linking to
264	* anon_vma_clone() is called by vma_expand(), vma_merge(), __split_vma(),	294	* all of the anon_vma objects contained within @src anon_vma_chain's.
265	* copy_vma() and anon_vma_fork(). The first four want an exact copy of src,	295	* @dst: The destination VMA with an empty anon_vma_chain.
266	* while the last one, anon_vma_fork(), may try to reuse an existing anon_vma to	296	* @src: The source VMA we wish to duplicate.
267	* prevent endless growth of anon_vma. Since dst->anon_vma is set to NULL before	297	* @operation: The type of operation which resulted in the clone.
268	* call, we can identify this case by checking (!dst->anon_vma &&	298	*
269	* src->anon_vma).	299	* This is the heart of the VMA side of the anon_vma implementation - we invoke
270	*	300	* this function whenever we need to set up a new VMA's anon_vma state.
271	* If (!dst->anon_vma && src->anon_vma) is true, this function tries to find	301	*
272	* and reuse existing anon_vma which has no vmas and only one child anon_vma.	302	* This is invoked for:
273	* This prevents degradation of anon_vma hierarchy to endless linear chain in	303	*
274	* case of constantly forking task. On the other hand, an anon_vma with more	304	* - VMA Merge, but only when @dst is unfaulted and @src is faulted - meaning we
275	* than one child isn't reused even if there was no alive vma, thus rmap	305	* clone @src into @dst.
276	* walker has a good chance of avoiding scanning the whole hierarchy when it	306	* - VMA split.
277	* searches where page is mapped.	307	* - VMA (m)remap.
		308	* - Fork of faulted VMA.
		309	*
		310	* In all cases other than fork this is simply a duplication. Fork additionally
		311	* adds a new active anon_vma.
		312	*
		313	* ONLY in the case of fork do we try to 'reuse' existing anon_vma's in an
		314	* anon_vma hierarchy, reusing anon_vma's which have no VMA associated with them
		315	* but do have a single child. This is to avoid waste of memory when repeatedly
		316	* forking.
		317	*
		318	* Returns: 0 on success, -ENOMEM on failure.
278	*/	319	*/
279	int anon_vma_clone(struct vm_area_struct dst, struct vm_area_struct src)	320	int anon_vma_clone(struct vm_area_struct dst, struct vm_area_struct src,
		321	enum vma_operation operation)
280	{	322	{
281	struct anon_vma_chain avc, pavc;	323	struct anon_vma_chain avc, pavc;
282	struct anon_vma *root = NULL;	324	struct anon_vma *active_anon_vma = src->anon_vma;
283
284	list_for_each_entry_reverse(pavc, &src->anon_vma_chain, same_vma) {
285	struct anon_vma *anon_vma;
286
287	avc = anon_vma_chain_alloc(GFP_NOWAIT);
288	if (unlikely(!avc)) {
289	unlock_anon_vma_root(root);
290	root = NULL;
291	avc = anon_vma_chain_alloc(GFP_KERNEL);
292	if (!avc)
293	goto enomem_failure;
294	}
295	anon_vma = pavc->anon_vma;
296	root = lock_anon_vma_root(root, anon_vma);
297	anon_vma_chain_link(dst, avc, anon_vma);
298		325
299	/*	326	check_anon_vma_clone(dst, src, operation);
300	* Reuse existing anon_vma if it has no vma and only one	327
301	* anon_vma child.	328	if (!active_anon_vma)
302	*	329	return 0;
303	* Root anon_vma is never reused:	330
304	* it has self-parent reference and at least one child.	331	/*
305	*/	332	* Allocate AVCs. We don't need an anon_vma lock for this as we
306	if (!dst->anon_vma && src->anon_vma &&	333	* are not updating the anon_vma rbtree nor are we changing
307	anon_vma->num_children < 2 &&	334	* anon_vma statistics.
308	anon_vma->num_active_vmas == 0)	335	*
309	dst->anon_vma = anon_vma;	336	* Either src, dst have the same mm for which we hold an exclusive mmap
		337	* write lock, or we are forking and we hold it on src->vm_mm and dst is
		338	* not yet accessible to other threads so there's no possibliity of the
		339	* unlinked AVC's being observed yet.
		340	*/
		341	list_for_each_entry(pavc, &src->anon_vma_chain, same_vma) {
		342	avc = anon_vma_chain_alloc(GFP_KERNEL);
		343	if (!avc)
		344	goto enomem_failure;
		345
		346	anon_vma_chain_assign(dst, avc, pavc->anon_vma);
310	}	347	}
311	if (dst->anon_vma)	348
		349	/*
		350	* Now link the anon_vma's back to the newly inserted AVCs.
		351	* Note that all anon_vma's share the same root.
		352	*/
		353	anon_vma_lock_write(src->anon_vma);
		354	list_for_each_entry_reverse(avc, &dst->anon_vma_chain, same_vma) {
		355	struct anon_vma *anon_vma = avc->anon_vma;
		356
		357	anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
		358	if (operation == VMA_OP_FORK)
		359	maybe_reuse_anon_vma(dst, anon_vma);
		360	}
		361
		362	if (operation != VMA_OP_FORK)
312	dst->anon_vma->num_active_vmas++;	363	dst->anon_vma->num_active_vmas++;
313	unlock_anon_vma_root(root);	364
		365	anon_vma_unlock_write(active_anon_vma);
314	return 0;	366	return 0;
315		367
316	enomem_failure:	368	enomem_failure:
317	/*	369	cleanup_partial_anon_vmas(dst);
318	* dst->anon_vma is dropped here otherwise its num_active_vmas can
319	* be incorrectly decremented in unlink_anon_vmas().
320	* We can safely do this because callers of anon_vma_clone() don't care
321	* about dst->anon_vma if anon_vma_clone() failed.
322	*/
323	dst->anon_vma = NULL;
324	unlink_anon_vmas(dst);
325	return -ENOMEM;	370	return -ENOMEM;
326	}	371	}
327		372
@@ -334,7 +379,7 @@ int anon_vma_fork(struct vm_area_struct vma, struct vm_area_struct pvma) · 6 unmodified lines
334	{	379	{
335	struct anon_vma_chain *avc;	380	struct anon_vma_chain *avc;
336	struct anon_vma *anon_vma;	381	struct anon_vma *anon_vma;
337	int error;	382	int rc;
338		383
339	/* Don't bother if the parent process has no anon_vma here. */	384	/* Don't bother if the parent process has no anon_vma here. */
340	if (!pvma->anon_vma)	385	if (!pvma->anon_vma)
@@ -343,27 +388,35 @@ int anon_vma_fork(struct vm_area_struct vma, struct vm_area_struct pvma) · 2 unmodified lines
343	/* Drop inherited anon_vma, we'll reuse existing or allocate new. */	388	/* Drop inherited anon_vma, we'll reuse existing or allocate new. */
344	vma->anon_vma = NULL;	389	vma->anon_vma = NULL;
345		390
		391	anon_vma = anon_vma_alloc();
		392	if (!anon_vma)
		393	return -ENOMEM;
		394	avc = anon_vma_chain_alloc(GFP_KERNEL);
		395	if (!avc) {
		396	put_anon_vma(anon_vma);
		397	return -ENOMEM;
		398	}
		399
346	/*	400	/*
347	* First, attach the new VMA to the parent VMA's anon_vmas,	401	* First, attach the new VMA to the parent VMA's anon_vmas,
348	* so rmap can find non-COWed pages in child processes.	402	* so rmap can find non-COWed pages in child processes.
349	*/	403	*/
350	error = anon_vma_clone(vma, pvma);	404	rc = anon_vma_clone(vma, pvma, VMA_OP_FORK);
351	if (error)	405	/* An error arose or an existing anon_vma was reused, all done then. */
352	return error;	406	if (rc \|\| vma->anon_vma) {
353		407	put_anon_vma(anon_vma);
354	/* An existing anon_vma has been reused, all done then. */	408	anon_vma_chain_free(avc);
355	if (vma->anon_vma)	409	return rc;
356	return 0;	410	}
357		411
358	/* Then add our own anon_vma. */	412	/*
359	anon_vma = anon_vma_alloc();	413	* OK no reuse, so add our own anon_vma.
360	if (!anon_vma)	414	*
361	goto out_error;	415	* Since it is not linked anywhere we can safely manipulate anon_vma
362	anon_vma->num_active_vmas++;	416	* fields without a lock.
363	avc = anon_vma_chain_alloc(GFP_KERNEL);	417	*/
364	if (!avc)
365	goto out_error_free_anon_vma;
366		418
		419	anon_vma->num_active_vmas = 1;
367	/*	420	/*
368	* The root anon_vma's rwsem is the lock actually used when we	421	* The root anon_vma's rwsem is the lock actually used when we
369	* lock any of the anon_vmas in this anon_vma tree.	422	* lock any of the anon_vmas in this anon_vma tree.
@@ -378,24 +431,59 @@ int anon_vma_fork(struct vm_area_struct vma, struct vm_area_struct pvma) · 8 unmodified lines
378	get_anon_vma(anon_vma->root);	431	get_anon_vma(anon_vma->root);
379	/* Mark this anon_vma as the one where our new (COWed) pages go. */	432	/* Mark this anon_vma as the one where our new (COWed) pages go. */
380	vma->anon_vma = anon_vma;	433	vma->anon_vma = anon_vma;
		434	anon_vma_chain_assign(vma, avc, anon_vma);
		435	/* Now let rmap see it. */
381	anon_vma_lock_write(anon_vma);	436	anon_vma_lock_write(anon_vma);
382	anon_vma_chain_link(vma, avc, anon_vma);	437	anon_vma_interval_tree_insert(avc, &anon_vma->rb_root);
383	anon_vma->parent->num_children++;	438	anon_vma->parent->num_children++;
384	anon_vma_unlock_write(anon_vma);	439	anon_vma_unlock_write(anon_vma);
385		440
386	return 0;	441	return 0;
		442	}
387		443
388	out_error_free_anon_vma:	444	/*
389	put_anon_vma(anon_vma);	445	* In the unfortunate case of anon_vma_clone() failing to allocate memory we
390	out_error:	446	* have to clean things up.
391	unlink_anon_vmas(vma);	447	*
392	return -ENOMEM;	448	* Since we allocate anon_vma_chain's before we insert them into the interval
		449	* trees, we simply have to free up the AVC's and remove the entries from the
		450	* VMA's anon_vma_chain.
		451	*/
		452	static void cleanup_partial_anon_vmas(struct vm_area_struct *vma)
		453	{
		454	struct anon_vma_chain avc, next;
		455
		456	list_for_each_entry_safe(avc, next, &vma->anon_vma_chain, same_vma) {
		457	list_del(&avc->same_vma);
		458	anon_vma_chain_free(avc);
		459	}
393	}	460	}
394		461
		462	/**
		463	* unlink_anon_vmas() - remove all links between a VMA and anon_vma's, freeing
		464	* anon_vma_chain objects.
		465	* @vma: The VMA whose links to anon_vma objects is to be severed.
		466	*
		467	* As part of the process anon_vma_chain's are freed,
		468	* anon_vma->num_children,num_active_vmas is updated as required and, if the
		469	* relevant anon_vma references no further VMAs, its reference count is
		470	* decremented.
		471	*/
395	void unlink_anon_vmas(struct vm_area_struct *vma)	472	void unlink_anon_vmas(struct vm_area_struct *vma)
396	{	473	{
397	struct anon_vma_chain avc, next;	474	struct anon_vma_chain avc, next;
398	struct anon_vma *root = NULL;	475	struct anon_vma *active_anon_vma = vma->anon_vma;
		476
		477	/* Always hold mmap lock, read-lock on unmap possibly. */
		478	mmap_assert_locked(vma->vm_mm);
		479
		480	/* Unfaulted is a no-op. */
		481	if (!active_anon_vma) {
		482	VM_WARN_ON_ONCE(!list_empty(&vma->anon_vma_chain));
		483	return;
		484	}
		485
		486	anon_vma_lock_write(active_anon_vma);
399		487
400	/*	488	/*
401	* Unlink each anon_vma chained to the VMA. This list is ordered	489	* Unlink each anon_vma chained to the VMA. This list is ordered
@@ -404,7 +492,6 @@ void unlink_anon_vmas(struct vm_area_struct *vma) · 2 unmodified lines
404	list_for_each_entry_safe(avc, next, &vma->anon_vma_chain, same_vma) {	492	list_for_each_entry_safe(avc, next, &vma->anon_vma_chain, same_vma) {
405	struct anon_vma *anon_vma = avc->anon_vma;	493	struct anon_vma *anon_vma = avc->anon_vma;
406		494
407	root = lock_anon_vma_root(root, anon_vma);
408	anon_vma_interval_tree_remove(avc, &anon_vma->rb_root);	495	anon_vma_interval_tree_remove(avc, &anon_vma->rb_root);
409		496
410	/*	497	/*
@@ -419,16 +506,15 @@ void unlink_anon_vmas(struct vm_area_struct *vma) · 8 unmodified lines
419	list_del(&avc->same_vma);	506	list_del(&avc->same_vma);
420	anon_vma_chain_free(avc);	507	anon_vma_chain_free(avc);
421	}	508	}
422	if (vma->anon_vma) {
423	vma->anon_vma->num_active_vmas--;
424		509
425	/*	510	active_anon_vma->num_active_vmas--;
426	* vma would still be needed after unlink, and anon_vma will be prepared	511	/*
427	* when handle fault.	512	* vma would still be needed after unlink, and anon_vma will be prepared
428	*/	513	* when handle fault.
429	vma->anon_vma = NULL;	514	*/
430	}	515	vma->anon_vma = NULL;
431	unlock_anon_vma_root(root);	516	anon_vma_unlock_write(active_anon_vma);
		517
432		518
433	/*	519	/*
434	* Iterate the list once more, it now only contains empty and unlinked	520	* Iterate the list once more, it now only contains empty and unlinked
@@ -2147,7 +2233,7 @@ static bool try_to_unmap_one(struct folio folio, struct vm_area_struct vma, · 1712 unmodified lines
2147	goto discard;	2233	goto discard;
2148	}	2234	}
2149		2235
2150	if (swap_duplicate(entry) < 0) {	2236	if (folio_dup_swap(folio, subpage) < 0) {
2151	set_pte_at(mm, address, pvmw.pte, pteval);	2237	set_pte_at(mm, address, pvmw.pte, pteval);
2152	goto walk_abort;	2238	goto walk_abort;
2153	}	2239	}
@@ -2158,7 +2244,7 @@ static bool try_to_unmap_one(struct folio folio, struct vm_area_struct vma, · 4 unmodified lines
2158	* so we'll not check/care.	2244	* so we'll not check/care.
2159	*/	2245	*/
2160	if (arch_unmap_one(mm, vma, address, pteval) < 0) {	2246	if (arch_unmap_one(mm, vma, address, pteval) < 0) {
2161	swap_free(entry);	2247	folio_put_swap(folio, subpage);
2162	set_pte_at(mm, address, pvmw.pte, pteval);	2248	set_pte_at(mm, address, pvmw.pte, pteval);
2163	goto walk_abort;	2249	goto walk_abort;
2164	}	2250	}
@@ -2166,7 +2252,7 @@ static bool try_to_unmap_one(struct folio folio, struct vm_area_struct vma, · 1 unmodified line
2166	/* See folio_try_share_anon_rmap(): clear PTE first. */	2252	/* See folio_try_share_anon_rmap(): clear PTE first. */
2167	if (anon_exclusive &&	2253	if (anon_exclusive &&
2168	folio_try_share_anon_rmap_pte(folio, subpage)) {	2254	folio_try_share_anon_rmap_pte(folio, subpage)) {
2169	swap_free(entry);	2255	folio_put_swap(folio, subpage);
2170	set_pte_at(mm, address, pvmw.pte, pteval);	2256	set_pte_at(mm, address, pvmw.pte, pteval);
2171	goto walk_abort;	2257	goto walk_abort;
2172	}	2258	}

mm/shmem.c+36−54

@@ -1,3 +1,4 @@
		1	// SPDX-License-Identifier: GPL-2.0
1	/*	2	/*
2	* Resizable virtual memory filesystem for Linux.	3	* Resizable virtual memory filesystem for Linux.
3	*	4	*
@@ -17,8 +18,6 @@ · 13 unmodified lines
17	*	18	*
18	* tiny-shmem:	19	* tiny-shmem:
19	* Copyright (c) 2004, 2008 Matt Mackall <mpm@selenic.com>	20	* Copyright (c) 2004, 2008 Matt Mackall <mpm@selenic.com>
20	*
21	* This file is released under the GPL.
22	*/	21	*/
23		22
24	#include <linux/fs.h>	23	#include <linux/fs.h>
@@ -983,7 +982,7 @@ static long shmem_free_swap(struct address_space *mapping, · 958 unmodified lines
983	xas_unlock_irq(&xas);	982	xas_unlock_irq(&xas);
984		983
985	if (nr_pages)	984	if (nr_pages)
986	free_swap_and_cache_nr(radix_to_swp_entry(radswap), nr_pages);	985	swap_put_entries_direct(radix_to_swp_entry(radswap), nr_pages);
987		986
988	return nr_pages;	987	return nr_pages;
989	}	988	}
@@ -1622,11 +1621,23 @@ int shmem_writeout(struct folio folio, struct swap_iocb *plug, · 632 unmodified lines
1622	}	1621	}
1623		1622
1624	if (split) {	1623	if (split) {
		1624	int order;
		1625
1625	try_split:	1626	try_split:
		1627	order = folio_order(folio);
1626	/* Ensure the subpages are still dirty */	1628	/* Ensure the subpages are still dirty */
1627	folio_test_set_dirty(folio);	1629	folio_test_set_dirty(folio);
1628	if (split_folio_to_list(folio, folio_list))	1630	if (split_folio_to_list(folio, folio_list))
1629	goto redirty;	1631	goto redirty;
		1632
		1633	#ifdef CONFIG_TRANSPARENT_HUGEPAGE
		1634	if (order >= HPAGE_PMD_ORDER) {
		1635	count_memcg_folio_events(folio, THP_SWPOUT_FALLBACK, 1);
		1636	count_vm_event(THP_SWPOUT_FALLBACK);
		1637	}
		1638	#endif
		1639	count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK);
		1640
1630	folio_clear_dirty(folio);	1641	folio_clear_dirty(folio);
1631	}	1642	}
1632		1643
@@ -1684,7 +1695,7 @@ try_split: · 51 unmodified lines
1684	spin_unlock(&shmem_swaplist_lock);	1695	spin_unlock(&shmem_swaplist_lock);
1685	}	1696	}
1686		1697
1687	swap_shmem_alloc(folio->swap, nr_pages);	1698	folio_dup_swap(folio, NULL);
1688	shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));	1699	shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));
1689		1700
1690	BUG_ON(folio_mapped(folio));	1701	BUG_ON(folio_mapped(folio));
@@ -1705,7 +1716,7 @@ try_split: · 14 unmodified lines
1705	/* Swap entry might be erased by racing shmem_free_swap() */	1716	/* Swap entry might be erased by racing shmem_free_swap() */
1706	if (!error) {	1717	if (!error) {
1707	shmem_recalc_inode(inode, 0, -nr_pages);	1718	shmem_recalc_inode(inode, 0, -nr_pages);
1708	swap_free_nr(folio->swap, nr_pages);	1719	folio_put_swap(folio, NULL);
1709	}	1720	}
1710		1721
1711	/*	1722	/*
@@ -2031,10 +2042,9 @@ static struct folio shmem_swap_alloc_folio(struct inode inode, · 319 unmodified lines
2031	swp_entry_t entry, int order, gfp_t gfp)	2042	swp_entry_t entry, int order, gfp_t gfp)
2032	{	2043	{
2033	struct shmem_inode_info *info = SHMEM_I(inode);	2044	struct shmem_inode_info *info = SHMEM_I(inode);
		2045	struct folio new, swapcache;
2034	int nr_pages = 1 << order;	2046	int nr_pages = 1 << order;
2035	struct folio *new;
2036	gfp_t alloc_gfp;	2047	gfp_t alloc_gfp;
2037	void *shadow;
2038		2048
2039	/*	2049	/*
2040	* We have arrived here because our zones are constrained, so don't	2050	* We have arrived here because our zones are constrained, so don't
@@ -2074,34 +2084,19 @@ retry: · 33 unmodified lines
2074	goto fallback;	2084	goto fallback;
2075	}	2085	}
2076		2086
2077	/*	2087	swapcache = swapin_folio(entry, new);
2078	* Prevent parallel swapin from proceeding with the swap cache flag.	2088	if (swapcache != new) {
2079	*
2080	* Of course there is another possible concurrent scenario as well,
2081	* that is to say, the swap cache flag of a large folio has already
2082	* been set by swapcache_prepare(), while another thread may have
2083	* already split the large swap entry stored in the shmem mapping.
2084	* In this case, shmem_add_to_page_cache() will help identify the
2085	* concurrent swapin and return -EEXIST.
2086	*/
2087	if (swapcache_prepare(entry, nr_pages)) {
2088	folio_put(new);	2089	folio_put(new);
2089	new = ERR_PTR(-EEXIST);	2090	if (!swapcache) {
2090	/* Try smaller folio to avoid cache conflict */	2091	/*
2091	goto fallback;	2092	* The new folio is charged already, swapin can
		2093	* only fail due to another raced swapin.
		2094	*/
		2095	new = ERR_PTR(-EEXIST);
		2096	goto fallback;
		2097	}
2092	}	2098	}
2093		2099	return swapcache;
2094	__folio_set_locked(new);
2095	__folio_set_swapbacked(new);
2096	new->swap = entry;
2097
2098	memcg1_swapin(entry, nr_pages);
2099	shadow = swap_cache_get_shadow(entry);
2100	if (shadow)
2101	workingset_refault(new, shadow);
2102	folio_add_lru(new);
2103	swap_read_folio(new, NULL);
2104	return new;
2105	fallback:	2100	fallback:
2106	/* Order 0 swapin failed, nothing to fallback to, abort */	2101	/* Order 0 swapin failed, nothing to fallback to, abort */
2107	if (!order)	2102	if (!order)
@@ -2191,8 +2186,7 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, · 83 unmodified lines
2191	}	2186	}
2192		2187
2193	static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,	2188	static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,
2194	struct folio *folio, swp_entry_t swap,	2189	struct folio *folio, swp_entry_t swap)
2195	bool skip_swapcache)
2196	{	2190	{
2197	struct address_space *mapping = inode->i_mapping;	2191	struct address_space *mapping = inode->i_mapping;
2198	swp_entry_t swapin_error;	2192	swp_entry_t swapin_error;
@@ -2208,15 +2202,14 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index, · 9 unmodified lines
2208		2202
2209	nr_pages = folio_nr_pages(folio);	2203	nr_pages = folio_nr_pages(folio);
2210	folio_wait_writeback(folio);	2204	folio_wait_writeback(folio);
2211	if (!skip_swapcache)	2205	folio_put_swap(folio, NULL);
2212	swap_cache_del_folio(folio);	2206	swap_cache_del_folio(folio);
2213	/*	2207	/*
2214	* Don't treat swapin error folio as alloced. Otherwise inode->i_blocks	2208	* Don't treat swapin error folio as alloced. Otherwise inode->i_blocks
2215	* won't be 0 when inode is released and thus trigger WARN_ON(i_blocks)	2209	* won't be 0 when inode is released and thus trigger WARN_ON(i_blocks)
2216	* in shmem_evict_inode().	2210	* in shmem_evict_inode().
2217	*/	2211	*/
2218	shmem_recalc_inode(inode, -nr_pages, -nr_pages);	2212	shmem_recalc_inode(inode, -nr_pages, -nr_pages);
2219	swap_free_nr(swap, nr_pages);
2220	}	2213	}
2221		2214
2222	static int shmem_split_large_entry(struct inode *inode, pgoff_t index,	2215	static int shmem_split_large_entry(struct inode *inode, pgoff_t index,
@@ -2309,7 +2302,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 86 unmodified lines
2309	softleaf_t index_entry;	2302	softleaf_t index_entry;
2310	struct swap_info_struct *si;	2303	struct swap_info_struct *si;
2311	struct folio *folio = NULL;	2304	struct folio *folio = NULL;
2312	bool skip_swapcache = false;
2313	int error, nr_pages, order;	2305	int error, nr_pages, order;
2314	pgoff_t offset;	2306	pgoff_t offset;
2315		2307
@@ -2352,7 +2344,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 36 unmodified lines
2352	folio = NULL;	2344	folio = NULL;
2353	goto failed;	2345	goto failed;
2354	}	2346	}
2355	skip_swapcache = true;
2356	} else {	2347	} else {
2357	/* Cached swapin only supports order 0 folio */	2348	/* Cached swapin only supports order 0 folio */
2358	folio = shmem_swapin_cluster(swap, gfp, info, index);	2349	folio = shmem_swapin_cluster(swap, gfp, info, index);
@@ -2408,9 +2399,8 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 49 unmodified lines
2408	* and swap cache folios are never partially freed.	2399	* and swap cache folios are never partially freed.
2409	*/	2400	*/
2410	folio_lock(folio);	2401	folio_lock(folio);
2411	if ((!skip_swapcache && !folio_test_swapcache(folio)) \|\|	2402	if (!folio_matches_swap_entry(folio, swap) \|\|
2412	shmem_confirm_swap(mapping, index, swap) < 0 \|\|	2403	shmem_confirm_swap(mapping, index, swap) < 0) {
2413	folio->swap.val != swap.val) {
2414	error = -EEXIST;	2404	error = -EEXIST;
2415	goto unlock;	2405	goto unlock;
2416	}	2406	}
@@ -2442,14 +2432,9 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 25 unmodified lines
2442	if (sgp == SGP_WRITE)	2432	if (sgp == SGP_WRITE)
2443	folio_mark_accessed(folio);	2433	folio_mark_accessed(folio);
2444		2434
2445	if (skip_swapcache) {	2435	folio_put_swap(folio, NULL);
2446	folio->swap.val = 0;	2436	swap_cache_del_folio(folio);
2447	swapcache_clear(si, swap, nr_pages);
2448	} else {
2449	swap_cache_del_folio(folio);
2450	}
2451	folio_mark_dirty(folio);	2437	folio_mark_dirty(folio);
2452	swap_free_nr(swap, nr_pages);
2453	put_swap_device(si);	2438	put_swap_device(si);
2454		2439
2455	*foliop = folio;	2440	*foliop = folio;
@@ -2458,14 +2443,11 @@ failed: · 2 unmodified lines
2458	if (shmem_confirm_swap(mapping, index, swap) < 0)	2443	if (shmem_confirm_swap(mapping, index, swap) < 0)
2459	error = -EEXIST;	2444	error = -EEXIST;
2460	if (error == -EIO)	2445	if (error == -EIO)
2461	shmem_set_folio_swapin_error(inode, index, folio, swap,	2446	shmem_set_folio_swapin_error(inode, index, folio, swap);
2462	skip_swapcache);
2463	unlock:	2447	unlock:
2464	if (folio)	2448	if (folio)
2465	folio_unlock(folio);	2449	folio_unlock(folio);
2466	failed_nolock:	2450	failed_nolock:
2467	if (skip_swapcache)
2468	swapcache_clear(si, folio->swap, folio_nr_pages(folio));
2469	if (folio)	2451	if (folio)
2470	folio_put(folio);	2452	folio_put(folio);
2471	put_swap_device(si);	2453	put_swap_device(si);

mm/show_mem.c+1−2

@@ -278,8 +278,7 @@ static void show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_z · 277 unmodified lines
278	#endif	278	#endif
279	K(node_page_state(pgdat, NR_PAGETABLE)),	279	K(node_page_state(pgdat, NR_PAGETABLE)),
280	K(node_page_state(pgdat, NR_SECONDARY_PAGETABLE)),	280	K(node_page_state(pgdat, NR_SECONDARY_PAGETABLE)),
281	str_yes_no(atomic_read(&pgdat->kswapd_failures) >=	281	str_yes_no(kswapd_test_hopeless(pgdat)),
282	MAX_RECLAIM_RETRIES),
283	K(node_page_state(pgdat, NR_BALLOON_PAGES)));	282	K(node_page_state(pgdat, NR_BALLOON_PAGES)));
284	}	283	}
285		284

mm/shrinker_debug.c+7−6

 					       memcg_aware ? memcg : NULL,
 					       count_per_node);
 		if (total) {
-			seq_printf(m, "%lu", mem_cgroup_ino(memcg));
+			seq_printf(m, "%llu", mem_cgroup_id(memcg));
 			for_each_node(nid)
 				seq_printf(m, " %lu", count_per_node[nid]);
 			seq_putc(m, '\n');
 					   size_t size, loff_t *pos)
 {
 	struct shrinker *shrinker = file->private_data;
-	unsigned long nr_to_scan = 0, ino, read_len;
+	unsigned long nr_to_scan = 0, read_len;
+	u64 id;
 	struct shrink_control sc = {
 		.gfp_mask = GFP_KERNEL,
 	};
 		return -EFAULT;
 	kbuf[read_len] = '\0';
-	if (sscanf(kbuf, "%lu %d %lu", &ino, &nid, &nr_to_scan) != 3)
+	if (sscanf(kbuf, "%llu %d %lu", &id, &nid, &nr_to_scan) != 3)
 		return -EINVAL;
 	if (nid < 0 || nid >= nr_node_ids)
 		return size;
 	if (shrinker->flags & SHRINKER_MEMCG_AWARE) {
-		memcg = mem_cgroup_get_from_ino(ino);
+		memcg = mem_cgroup_get_from_id(id);
-		if (!memcg || IS_ERR(memcg))
+		if (!memcg)
 			return -ENOENT;
 		if (!mem_cgroup_online(memcg)) {
 			mem_cgroup_put(memcg);
 			return -ENOENT;
 		}
-	} else if (ino != 0) {
+	} else if (id != 0) {
 		return -EINVAL;
 	}

mm/slub.c+2−1

@@ -8334,7 +8334,8 @@ void __init kmem_cache_init(void) · 8333 unmodified lines
8334		8334
8335	void __init kmem_cache_init_late(void)	8335	void __init kmem_cache_init_late(void)
8336	{	8336	{
8337	flushwq = alloc_workqueue("slub_flushwq", WQ_MEM_RECLAIM, 0);	8337	flushwq = alloc_workqueue("slub_flushwq", WQ_MEM_RECLAIM \| WQ_PERCPU,
		8338	0);
8338	WARN_ON(!flushwq);	8339	WARN_ON(!flushwq);
8339	}	8340	}
8340		8341

mm/swap.c+1−1

@@ -513,7 +513,7 @@ void folio_add_lru(struct folio *folio) · 512 unmodified lines
513	EXPORT_SYMBOL(folio_add_lru);	513	EXPORT_SYMBOL(folio_add_lru);
514		514
515	/**	515	/**
516	* folio_add_lru_vma() - Add a folio to the appropate LRU list for this VMA.	516	* folio_add_lru_vma() - Add a folio to the appropriate LRU list for this VMA.
517	* @folio: The folio to be added to the LRU.	517	* @folio: The folio to be added to the LRU.
518	* @vma: VMA in which the folio is mapped.	518	* @vma: VMA in which the folio is mapped.
519	*	519	*

mm/swap.h+59−13

 	spin_unlock_irq(&ci->lock);
 }
+/*
+ * Below are the core routines for doing swap for a folio.
+ * All helpers requires the folio to be locked, and a locked folio
+ * in the swap cache pins the swap entries / slots allocated to the
+ * folio, swap relies heavily on the swap cache and folio lock for
+ * synchronization.
+ *
+ * folio_alloc_swap(): the entry point for a folio to be swapped
+ * out. It allocates swap slots and pins the slots with swap cache.
+ * The slots start with a swap count of zero.
+ *
+ * folio_dup_swap(): increases the swap count of a folio, usually
+ * during it gets unmapped and a swap entry is installed to replace
+ * it (e.g., swap entry in page table). A swap slot with swap
+ * count == 0 should only be increasd by this helper.
+ *
+ * folio_put_swap(): does the opposite thing of folio_dup_swap().
+ */
+int folio_alloc_swap(struct folio *folio);
+int folio_dup_swap(struct folio *folio, struct page *subpage);
+void folio_put_swap(struct folio *folio, struct page *subpage);
+/* For internal use */
+extern void swap_entries_free(struct swap_info_struct *si,
+			      struct swap_cluster_info *ci,
+			      unsigned long offset, unsigned int nr_pages);
 /* linux/mm/page_io.c */
 int sio_pool_init(void);
 struct swap_iocb;
 /*
  * All swap cache helpers below require the caller to ensure the swap entries
- * used are valid and stablize the device by any of the following ways:
+ * used are valid and stabilize the device by any of the following ways:
  * - Hold a reference by get_swap_device(): this ensures a single entry is
  *   valid and increases the swap device's refcount.
  * - Locking a folio in the swap cache: this ensures the folio's swap entries
  *   swap entries in the page table, similar to locking swap cache folio.
  * - See the comment of get_swap_device() for more complex usage.
  */
+bool swap_cache_has_folio(swp_entry_t entry);
 struct folio *swap_cache_get_folio(swp_entry_t entry);
 void *swap_cache_get_shadow(swp_entry_t entry);
-void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadow);
 void swap_cache_del_folio(struct folio *folio);
+struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags,
+				     struct mempolicy *mpol, pgoff_t ilx,
+				     bool *alloced);
 /* Below helpers require the caller to lock and pass in the swap cluster. */
+void __swap_cache_add_folio(struct swap_cluster_info *ci,
+			    struct folio *folio, swp_entry_t entry);
 void __swap_cache_del_folio(struct swap_cluster_info *ci,
 			    struct folio *folio, swp_entry_t entry, void *shadow);
 void __swap_cache_replace_folio(struct swap_cluster_info *ci,
 struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
 		struct vm_area_struct *vma, unsigned long addr,
 		struct swap_iocb **plug);
-struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags,
-		struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated,
-		bool skip_if_exists);
 struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag,
 		struct mempolicy *mpol, pgoff_t ilx);
 struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag,
 		struct vm_fault *vmf);
+struct folio *swapin_folio(swp_entry_t entry, struct folio *folio);
 void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
 			   unsigned long addr);
 static inline int non_swapcache_batch(swp_entry_t entry, int max_nr)
 {
-	struct swap_info_struct *si = __swap_entry_to_info(entry);
-	pgoff_t offset = swp_offset(entry);
 	int i;
 	/*
 	 * be in conflict with the folio in swap cache.
 	 */
 	for (i = 0; i < max_nr; i++) {
-		if ((si->swap_map[offset + i] & SWAP_HAS_CACHE))
+		if (swap_cache_has_folio(entry))
 			return i;
+		entry.val++;
 	}
 	return i;
 	return NULL;
 }
+static inline int folio_alloc_swap(struct folio *folio)
+{
+	return -EINVAL;
+}
+static inline int folio_dup_swap(struct folio *folio, struct page *page)
+{
+	return -EINVAL;
+}
+static inline void folio_put_swap(struct folio *folio, struct page *page)
+{
+}
 static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
 {
 }
 static inline void swap_write_unplug(struct swap_iocb *sio)
 {
 }
 	return NULL;
 }
+static inline struct folio *swapin_folio(swp_entry_t entry, struct folio *folio)
+{
+	return NULL;
+}
 static inline void swap_update_readahead(struct folio *folio,
 		struct vm_area_struct *vma, unsigned long addr)
 {
 	return 0;
 }
-static inline void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr)
+static inline bool swap_cache_has_folio(swp_entry_t entry)
 {
+	return false;
 }
 static inline struct folio *swap_cache_get_folio(swp_entry_t entry)
 	return NULL;
 }
-static inline void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadow)
-{
-}
 static inline void swap_cache_del_folio(struct folio *folio)
 {
 }

mm/swap_state.c+223−119

@@ -81,7 +81,7 @@ void show_swap_cache_info(void) · 80 unmodified lines
81	* Context: Caller must ensure @entry is valid and protect the swap device	81	* Context: Caller must ensure @entry is valid and protect the swap device
82	* with reference count or locks.	82	* with reference count or locks.
83	* Return: Returns the found folio on success, NULL otherwise. The caller	83	* Return: Returns the found folio on success, NULL otherwise. The caller
84	* must lock nd check if the folio still matches the swap entry before	84	* must lock and check if the folio still matches the swap entry before
85	* use (e.g., folio_matches_swap_entry).	85	* use (e.g., folio_matches_swap_entry).
86	*/	86	*/
87	struct folio *swap_cache_get_folio(swp_entry_t entry)	87	struct folio *swap_cache_get_folio(swp_entry_t entry)
@@ -102,6 +102,22 @@ struct folio *swap_cache_get_folio(swp_entry_t entry) · 14 unmodified lines
102	return NULL;	102	return NULL;
103	}	103	}
104		104
		105	/**
		106	* swap_cache_has_folio - Check if a swap slot has cache.
		107	* @entry: swap entry indicating the slot.
		108	*
		109	* Context: Caller must ensure @entry is valid and protect the swap
		110	* device with reference count or locks.
		111	*/
		112	bool swap_cache_has_folio(swp_entry_t entry)
		113	{
		114	unsigned long swp_tb;
		115
		116	swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
		117	swp_cluster_offset(entry));
		118	return swp_tb_is_folio(swp_tb);
		119	}
		120
105	/**	121	/**
106	* swap_cache_get_shadow - Looks up a shadow in the swap cache.	122	* swap_cache_get_shadow - Looks up a shadow in the swap cache.
107	* @entry: swap entry used for the lookup.	123	* @entry: swap entry used for the lookup.
@@ -121,6 +137,34 @@ void *swap_cache_get_shadow(swp_entry_t entry) · 13 unmodified lines
121	return NULL;	137	return NULL;
122	}	138	}
123		139
		140	void __swap_cache_add_folio(struct swap_cluster_info *ci,
		141	struct folio *folio, swp_entry_t entry)
		142	{
		143	unsigned long new_tb;
		144	unsigned int ci_start, ci_off, ci_end;
		145	unsigned long nr_pages = folio_nr_pages(folio);
		146
		147	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
		148	VM_WARN_ON_ONCE_FOLIO(folio_test_swapcache(folio), folio);
		149	VM_WARN_ON_ONCE_FOLIO(!folio_test_swapbacked(folio), folio);
		150
		151	new_tb = folio_to_swp_tb(folio);
		152	ci_start = swp_cluster_offset(entry);
		153	ci_off = ci_start;
		154	ci_end = ci_start + nr_pages;
		155	do {
		156	VM_WARN_ON_ONCE(swp_tb_is_folio(__swap_table_get(ci, ci_off)));
		157	__swap_table_set(ci, ci_off, new_tb);
		158	} while (++ci_off < ci_end);
		159
		160	folio_ref_add(folio, nr_pages);
		161	folio_set_swapcache(folio);
		162	folio->swap = entry;
		163
		164	node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
		165	lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
		166	}
		167
124	/**	168	/**
125	* swap_cache_add_folio - Add a folio into the swap cache.	169	* swap_cache_add_folio - Add a folio into the swap cache.
126	* @folio: The folio to be added.	170	* @folio: The folio to be added.
@@ -130,43 +174,51 @@ void *swap_cache_get_shadow(swp_entry_t entry) · 3 unmodified lines
130	*	174	*
131	* Context: Caller must ensure @entry is valid and protect the swap device	175	* Context: Caller must ensure @entry is valid and protect the swap device
132	* with reference count or locks.	176	* with reference count or locks.
133	* The caller also needs to update the corresponding swap_map slots with
134	* SWAP_HAS_CACHE bit to avoid race or conflict.
135	*/	177	*/
136	void swap_cache_add_folio(struct folio folio, swp_entry_t entry, void *shadowp)	178	static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,
		179	void **shadowp)
137	{	180	{
		181	int err;
138	void *shadow = NULL;	182	void *shadow = NULL;
139	unsigned long old_tb, new_tb;	183	unsigned long old_tb;
		184	struct swap_info_struct *si;
140	struct swap_cluster_info *ci;	185	struct swap_cluster_info *ci;
141	unsigned int ci_start, ci_off, ci_end;	186	unsigned int ci_start, ci_off, ci_end, offset;
142	unsigned long nr_pages = folio_nr_pages(folio);	187	unsigned long nr_pages = folio_nr_pages(folio);
143		188
144	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);	189	si = __swap_entry_to_info(entry);
145	VM_WARN_ON_ONCE_FOLIO(folio_test_swapcache(folio), folio);
146	VM_WARN_ON_ONCE_FOLIO(!folio_test_swapbacked(folio), folio);
147
148	new_tb = folio_to_swp_tb(folio);
149	ci_start = swp_cluster_offset(entry);	190	ci_start = swp_cluster_offset(entry);
150	ci_end = ci_start + nr_pages;	191	ci_end = ci_start + nr_pages;
151	ci_off = ci_start;	192	ci_off = ci_start;
152	ci = swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry));	193	offset = swp_offset(entry);
		194	ci = swap_cluster_lock(si, swp_offset(entry));
		195	if (unlikely(!ci->table)) {
		196	err = -ENOENT;
		197	goto failed;
		198	}
153	do {	199	do {
154	old_tb = __swap_table_xchg(ci, ci_off, new_tb);	200	old_tb = __swap_table_get(ci, ci_off);
155	WARN_ON_ONCE(swp_tb_is_folio(old_tb));	201	if (unlikely(swp_tb_is_folio(old_tb))) {
		202	err = -EEXIST;
		203	goto failed;
		204	}
		205	if (unlikely(!__swap_count(swp_entry(swp_type(entry), offset)))) {
		206	err = -ENOENT;
		207	goto failed;
		208	}
156	if (swp_tb_is_shadow(old_tb))	209	if (swp_tb_is_shadow(old_tb))
157	shadow = swp_tb_to_shadow(old_tb);	210	shadow = swp_tb_to_shadow(old_tb);
		211	offset++;
158	} while (++ci_off < ci_end);	212	} while (++ci_off < ci_end);
159		213	__swap_cache_add_folio(ci, folio, entry);
160	folio_ref_add(folio, nr_pages);
161	folio_set_swapcache(folio);
162	folio->swap = entry;
163	swap_cluster_unlock(ci);	214	swap_cluster_unlock(ci);
164
165	node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
166	lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
167
168	if (shadowp)	215	if (shadowp)
169	*shadowp = shadow;	216	*shadowp = shadow;
		217	return 0;
		218
		219	failed:
		220	swap_cluster_unlock(ci);
		221	return err;
170	}	222	}
171		223
172	/**	224	/**
@@ -185,8 +237,10 @@ void swap_cache_add_folio(struct folio folio, swp_entry_t entry, void *shadowp · 12 unmodified lines
185	void __swap_cache_del_folio(struct swap_cluster_info ci, struct folio folio,	237	void __swap_cache_del_folio(struct swap_cluster_info ci, struct folio folio,
186	swp_entry_t entry, void *shadow)	238	swp_entry_t entry, void *shadow)
187	{	239	{
		240	struct swap_info_struct *si;
188	unsigned long old_tb, new_tb;	241	unsigned long old_tb, new_tb;
189	unsigned int ci_start, ci_off, ci_end;	242	unsigned int ci_start, ci_off, ci_end;
		243	bool folio_swapped = false, need_free = false;
190	unsigned long nr_pages = folio_nr_pages(folio);	244	unsigned long nr_pages = folio_nr_pages(folio);
191		245
192	VM_WARN_ON_ONCE(__swap_entry_to_cluster(entry) != ci);	246	VM_WARN_ON_ONCE(__swap_entry_to_cluster(entry) != ci);
@@ -194,6 +248,7 @@ void __swap_cache_del_folio(struct swap_cluster_info ci, struct folio folio, · 1 unmodified line
194	VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);	248	VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
195	VM_WARN_ON_ONCE_FOLIO(folio_test_writeback(folio), folio);	249	VM_WARN_ON_ONCE_FOLIO(folio_test_writeback(folio), folio);
196		250
		251	si = __swap_entry_to_info(entry);
197	new_tb = shadow_swp_to_tb(shadow);	252	new_tb = shadow_swp_to_tb(shadow);
198	ci_start = swp_cluster_offset(entry);	253	ci_start = swp_cluster_offset(entry);
199	ci_end = ci_start + nr_pages;	254	ci_end = ci_start + nr_pages;
@@ -203,12 +258,27 @@ void __swap_cache_del_folio(struct swap_cluster_info ci, struct folio folio, · 3 unmodified lines
203	old_tb = __swap_table_xchg(ci, ci_off, new_tb);	258	old_tb = __swap_table_xchg(ci, ci_off, new_tb);
204	WARN_ON_ONCE(!swp_tb_is_folio(old_tb) \|\|	259	WARN_ON_ONCE(!swp_tb_is_folio(old_tb) \|\|
205	swp_tb_to_folio(old_tb) != folio);	260	swp_tb_to_folio(old_tb) != folio);
		261	if (__swap_count(swp_entry(si->type,
		262	swp_offset(entry) + ci_off - ci_start)))
		263	folio_swapped = true;
		264	else
		265	need_free = true;
206	} while (++ci_off < ci_end);	266	} while (++ci_off < ci_end);
207		267
208	folio->swap.val = 0;	268	folio->swap.val = 0;
209	folio_clear_swapcache(folio);	269	folio_clear_swapcache(folio);
210	node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages);	270	node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages);
211	lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages);	271	lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages);
		272
		273	if (!folio_swapped) {
		274	swap_entries_free(si, ci, swp_offset(entry), nr_pages);
		275	} else if (need_free) {
		276	do {
		277	if (!__swap_count(entry))
		278	swap_entries_free(si, ci, swp_offset(entry), 1);
		279	entry.val++;
		280	} while (--nr_pages);
		281	}
212	}	282	}
213		283
214	/**	284	/**
@@ -230,7 +300,6 @@ void swap_cache_del_folio(struct folio *folio) · 15 unmodified lines
230	__swap_cache_del_folio(ci, folio, entry, NULL);	300	__swap_cache_del_folio(ci, folio, entry, NULL);
231	swap_cluster_unlock(ci);	301	swap_cluster_unlock(ci);
232		302
233	put_swap_folio(folio, entry);
234	folio_ref_sub(folio, folio_nr_pages(folio));	303	folio_ref_sub(folio, folio_nr_pages(folio));
235	}	304	}
236		305
@@ -283,7 +352,7 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci, · 46 unmodified lines
283	}	352	}
284		353
285	/**	354	/**
286	* swap_cache_clear_shadow - Clears a set of shadows in the swap cache.	355	* __swap_cache_clear_shadow - Clears a set of shadows in the swap cache.
287	* @entry: The starting index entry.	356	* @entry: The starting index entry.
288	* @nr_ents: How many slots need to be cleared.	357	* @nr_ents: How many slots need to be cleared.
289	*	358	*
@@ -401,108 +470,143 @@ void swap_update_readahead(struct folio folio, struct vm_area_struct vma, · 111 unmodified lines
401	}	470	}
402	}	471	}
403		472
404	struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,	473	/**
405	struct mempolicy mpol, pgoff_t ilx, bool new_page_allocated,	474	* __swap_cache_prepare_and_add - Prepare the folio and add it to swap cache.
406	bool skip_if_exists)	475	* @entry: swap entry to be bound to the folio.
		476	* @folio: folio to be added.
		477	* @gfp: memory allocation flags for charge, can be 0 if @charged if true.
		478	* @charged: if the folio is already charged.
		479	*
		480	* Update the swap_map and add folio as swap cache, typically before swapin.
		481	* All swap slots covered by the folio must have a non-zero swap count.
		482	*
		483	* Context: Caller must protect the swap device with reference count or locks.
		484	* Return: Returns the folio being added on success. Returns the existing folio
		485	* if @entry is already cached. Returns NULL if raced with swapin or swapoff.
		486	*/
		487	static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
		488	struct folio *folio,
		489	gfp_t gfp, bool charged)
407	{	490	{
408	struct swap_info_struct *si = __swap_entry_to_info(entry);	491	struct folio *swapcache = NULL;
409	struct folio *folio;	492	void *shadow;
410	struct folio *new_folio = NULL;	493	int ret;
411	struct folio *result = NULL;
412	void *shadow = NULL;
413		494
414	*new_page_allocated = false;	495	__folio_set_locked(folio);
		496	__folio_set_swapbacked(folio);
415	for (;;) {	497	for (;;) {
416	int err;	498	ret = swap_cache_add_folio(folio, entry, &shadow);
		499	if (!ret)
		500	break;
417		501
418	/*	502	/*
419	* Check the swap cache first, if a cached folio is found,	503	* Large order allocation needs special handling on
420	* return it unlocked. The caller will lock and check it.	504	* race: if a smaller folio exists in cache, swapin needs
		505	* to fallback to order 0, and doing a swap cache lookup
		506	* might return a folio that is irrelevant to the faulting
		507	* entry because @entry is aligned down. Just return NULL.
421	*/	508	*/
422	folio = swap_cache_get_folio(entry);	509	if (ret != -EEXIST \|\| folio_test_large(folio))
423	if (folio)	510	goto failed;
424	goto got_folio;
425		511
426	/*	512	swapcache = swap_cache_get_folio(entry);
427	* Just skip read ahead for unused swap slot.	513	if (swapcache)
428	*/	514	goto failed;
429	if (!swap_entry_swapped(si, entry))	515	}
430	goto put_and_return;
431		516
432	/*	517	if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry)) {
433	* Get a new folio to read into from swap. Allocate it now if	518	swap_cache_del_folio(folio);
434	* new_folio not exist, before marking swap_map SWAP_HAS_CACHE,	519	goto failed;
435	* when -EEXIST will cause any racers to loop around until we	520	}
436	* add it to cache.
437	*/
438	if (!new_folio) {
439	new_folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
440	if (!new_folio)
441	goto put_and_return;
442	}
443		521
444	/*	522	memcg1_swapin(entry, folio_nr_pages(folio));
445	* Swap entry may have been freed since our caller observed it.	523	if (shadow)
446	*/	524	workingset_refault(folio, shadow);
447	err = swapcache_prepare(entry, 1);
448	if (!err)
449	break;
450	else if (err != -EEXIST)
451	goto put_and_return;
452		525
453	/*	526	/* Caller will initiate read into locked folio */
454	* Protect against a recursive call to __read_swap_cache_async()	527	folio_add_lru(folio);
455	* on the same entry waiting forever here because SWAP_HAS_CACHE	528	return folio;
456	* is set but the folio is not the swap cache yet. This can
457	* happen today if mem_cgroup_swapin_charge_folio() below
458	* triggers reclaim through zswap, which may call
459	* __read_swap_cache_async() in the writeback path.
460	*/
461	if (skip_if_exists)
462	goto put_and_return;
463		529
464	/*	530	failed:
465	* We might race against __swap_cache_del_folio(), and	531	folio_unlock(folio);
466	* stumble across a swap_map entry whose SWAP_HAS_CACHE	532	return swapcache;
467	* has not yet been cleared. Or race against another	533	}
468	* __read_swap_cache_async(), which has set SWAP_HAS_CACHE
469	* in swap_map, but not yet added its folio to swap cache.
470	*/
471	schedule_timeout_uninterruptible(1);
472	}
473		534
474	/*	535	/**
475	* The swap entry is ours to swap in. Prepare the new folio.	536	* swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
476	*/	537	* @entry: the swapped out swap entry to be binded to the folio.
477	__folio_set_locked(new_folio);	538	* @gfp_mask: memory allocation flags
478	__folio_set_swapbacked(new_folio);	539	* @mpol: NUMA memory allocation policy to be applied
		540	* @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
		541	* @new_page_allocated: sets true if allocation happened, false otherwise
		542	*
		543	* Allocate a folio in the swap cache for one swap slot, typically before
		544	* doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
		545	* @entry must have a non-zero swap count (swapped out).
		546	* Currently only supports order 0.
		547	*
		548	* Context: Caller must protect the swap device with reference count or locks.
		549	* Return: Returns the existing folio if @entry is cached already. Returns
		550	* NULL if failed due to -ENOMEM or @entry have a swap count < 1.
		551	*/
		552	struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,
		553	struct mempolicy *mpol, pgoff_t ilx,
		554	bool *new_page_allocated)
		555	{
		556	struct swap_info_struct *si = __swap_entry_to_info(entry);
		557	struct folio *folio;
		558	struct folio *result = NULL;
479		559
480	if (mem_cgroup_swapin_charge_folio(new_folio, NULL, gfp_mask, entry))	560	*new_page_allocated = false;
481	goto fail_unlock;	561	/* Check the swap cache again for readahead path. */
		562	folio = swap_cache_get_folio(entry);
		563	if (folio)
		564	return folio;
482		565
483	swap_cache_add_folio(new_folio, entry, &shadow);	566	/* Skip allocation for unused and bad swap slot for readahead. */
484	memcg1_swapin(entry, 1);	567	if (!swap_entry_swapped(si, entry))
		568	return NULL;
485		569
486	if (shadow)	570	/* Allocate a new folio to be added into the swap cache. */
487	workingset_refault(new_folio, shadow);	571	folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
488		572	if (!folio)
489	/* Caller will initiate read into locked new_folio */	573	return NULL;
490	folio_add_lru(new_folio);	574	/* Try add the new folio, returns existing folio or NULL on failure. */
491	*new_page_allocated = true;	575	result = __swap_cache_prepare_and_add(entry, folio, gfp_mask, false);
492	folio = new_folio;	576	if (result == folio)
493	got_folio:	577	*new_page_allocated = true;
494	result = folio;	578	else
495	goto put_and_return;	579	folio_put(folio);
496
497	fail_unlock:
498	put_swap_folio(new_folio, entry);
499	folio_unlock(new_folio);
500	put_and_return:
501	if (!(*new_page_allocated) && new_folio)
502	folio_put(new_folio);
503	return result;	580	return result;
504	}	581	}
505		582
		583	/**
		584	* swapin_folio - swap-in one or multiple entries skipping readahead.
		585	* @entry: starting swap entry to swap in
		586	* @folio: a new allocated and charged folio
		587	*
		588	* Reads @entry into @folio, @folio will be added to the swap cache.
		589	* If @folio is a large folio, the @entry will be rounded down to align
		590	* with the folio size.
		591	*
		592	* Return: returns pointer to @folio on success. If folio is a large folio
		593	* and this raced with another swapin, NULL will be returned to allow fallback
		594	* to order 0. Else, if another folio was already added to the swap cache,
		595	* return that swap cache folio instead.
		596	*/
		597	struct folio swapin_folio(swp_entry_t entry, struct folio folio)
		598	{
		599	struct folio *swapcache;
		600	pgoff_t offset = swp_offset(entry);
		601	unsigned long nr_pages = folio_nr_pages(folio);
		602
		603	entry = swp_entry(swp_type(entry), round_down(offset, nr_pages));
		604	swapcache = __swap_cache_prepare_and_add(entry, folio, 0, true);
		605	if (swapcache == folio)
		606	swap_read_folio(folio, NULL);
		607	return swapcache;
		608	}
		609
506	/*	610	/*
507	* Locate a page of swap in physical memory, reserving swap cache space	611	* Locate a page of swap in physical memory, reserving swap cache space
508	* and reading the disk if it is not already cached.	612	* and reading the disk if it is not already cached.
@@ -524,8 +628,8 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · 15 unmodified lines
524	return NULL;	628	return NULL;
525		629
526	mpol = get_vma_policy(vma, addr, 0, &ilx);	630	mpol = get_vma_policy(vma, addr, 0, &ilx);
527	folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx,	631	folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
528	&page_allocated, false);	632	&page_allocated);
529	mpol_cond_put(mpol);	633	mpol_cond_put(mpol);
530		634
531	if (page_allocated)	635	if (page_allocated)
@@ -642,9 +746,9 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 110 unmodified lines
642	blk_start_plug(&plug);	746	blk_start_plug(&plug);
643	for (offset = start_offset; offset <= end_offset ; offset++) {	747	for (offset = start_offset; offset <= end_offset ; offset++) {
644	/* Ok, do the async read-ahead now */	748	/* Ok, do the async read-ahead now */
645	folio = __read_swap_cache_async(	749	folio = swap_cache_alloc_folio(
646	swp_entry(swp_type(entry), offset),	750	swp_entry(swp_type(entry), offset), gfp_mask, mpol, ilx,
647	gfp_mask, mpol, ilx, &page_allocated, false);	751	&page_allocated);
648	if (!folio)	752	if (!folio)
649	continue;	753	continue;
650	if (page_allocated) {	754	if (page_allocated) {
@@ -661,8 +765,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 10 unmodified lines
661	lru_add_drain(); /* Push any new pages onto the LRU now */	765	lru_add_drain(); /* Push any new pages onto the LRU now */
662	skip:	766	skip:
663	/* The page was likely read above, so no need for plugging here */	767	/* The page was likely read above, so no need for plugging here */
664	folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx,	768	folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
665	&page_allocated, false);	769	&page_allocated);
666	if (unlikely(page_allocated))	770	if (unlikely(page_allocated))
667	swap_read_folio(folio, NULL);	771	swap_read_folio(folio, NULL);
668	return folio;	772	return folio;
@@ -766,8 +870,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 97 unmodified lines
766	if (!si)	870	if (!si)
767	continue;	871	continue;
768	}	872	}
769	folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx,	873	folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
770	&page_allocated, false);	874	&page_allocated);
771	if (si)	875	if (si)
772	put_swap_device(si);	876	put_swap_device(si);
773	if (!folio)	877	if (!folio)
@@ -788,8 +892,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 14 unmodified lines
788	lru_add_drain();	892	lru_add_drain();
789	skip:	893	skip:
790	/* The folio was likely read above, so no need for plugging here */	894	/* The folio was likely read above, so no need for plugging here */
791	folio = __read_swap_cache_async(targ_entry, gfp_mask, mpol, targ_ilx,	895	folio = swap_cache_alloc_folio(targ_entry, gfp_mask, mpol, targ_ilx,
792	&page_allocated, false);	896	&page_allocated);
793	if (unlikely(page_allocated))	897	if (unlikely(page_allocated))
794	swap_read_folio(folio, NULL);	898	swap_read_folio(folio, NULL);
795	return folio;	899	return folio;

mm/swapfile.c+384−478

@@ -48,16 +48,18 @@ · 47 unmodified lines
48	#include <linux/swap_cgroup.h>	48	#include <linux/swap_cgroup.h>
49	#include "swap_table.h"	49	#include "swap_table.h"
50	#include "internal.h"	50	#include "internal.h"
		51	#include "swap_table.h"
51	#include "swap.h"	52	#include "swap.h"
52		53
53	static bool swap_count_continued(struct swap_info_struct *, pgoff_t,	54	static bool swap_count_continued(struct swap_info_struct *, pgoff_t,
54	unsigned char);	55	unsigned char);
55	static void free_swap_count_continuations(struct swap_info_struct *);	56	static void free_swap_count_continuations(struct swap_info_struct *);
56	static void swap_entries_free(struct swap_info_struct *si,
57	struct swap_cluster_info *ci,
58	swp_entry_t entry, unsigned int nr_pages);
59	static void swap_range_alloc(struct swap_info_struct *si,	57	static void swap_range_alloc(struct swap_info_struct *si,
60	unsigned int nr_entries);	58	unsigned int nr_entries);
		59	static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr);
		60	static void swap_put_entry_locked(struct swap_info_struct *si,
		61	struct swap_cluster_info *ci,
		62	unsigned long offset);
61	static bool folio_swapcache_freeable(struct folio *folio);	63	static bool folio_swapcache_freeable(struct folio *folio);
62	static void move_cluster(struct swap_info_struct *si,	64	static void move_cluster(struct swap_info_struct *si,
63	struct swap_cluster_info ci, struct list_head list,	65	struct swap_cluster_info ci, struct list_head list,
@@ -81,9 +83,7 @@ bool swap_migration_ad_supported; · 17 unmodified lines
81	#endif /* CONFIG_MIGRATION */	83	#endif /* CONFIG_MIGRATION */
82		84
83	static const char Bad_file[] = "Bad swap file entry ";	85	static const char Bad_file[] = "Bad swap file entry ";
84	static const char Unused_file[] = "Unused swap file entry ";
85	static const char Bad_offset[] = "Bad swap offset entry ";	86	static const char Bad_offset[] = "Bad swap offset entry ";
86	static const char Unused_offset[] = "Unused swap offset entry ";
87		87
88	/*	88	/*
89	* all active swap_info_structs	89	* all active swap_info_structs
@@ -144,11 +144,6 @@ static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry) · 54 unmodified lines
144	return swap_type_to_info(swp_type(entry));	144	return swap_type_to_info(swp_type(entry));
145	}	145	}
146		146
147	static inline unsigned char swap_count(unsigned char ent)
148	{
149	return ent & ~SWAP_HAS_CACHE; /* may include COUNT_CONTINUED flag */
150	}
151
152	/*	147	/*
153	* Use the second highest bit of inuse_pages counter as the indicator	148	* Use the second highest bit of inuse_pages counter as the indicator
154	* if one swap device is on the available plist, so the atomic can	149	* if one swap device is on the available plist, so the atomic can
@@ -180,39 +175,25 @@ static long swap_usage_in_pages(struct swap_info_struct *si) · 25 unmodified lines
180	#define TTRS_FULL 0x4	175	#define TTRS_FULL 0x4
181		176
182	static bool swap_only_has_cache(struct swap_info_struct *si,	177	static bool swap_only_has_cache(struct swap_info_struct *si,
183	unsigned long offset, int nr_pages)	178	struct swap_cluster_info *ci,
		179	unsigned long offset, int nr_pages)
184	{	180	{
		181	unsigned int ci_off = offset % SWAPFILE_CLUSTER;
185	unsigned char *map = si->swap_map + offset;	182	unsigned char *map = si->swap_map + offset;
186	unsigned char *map_end = map + nr_pages;	183	unsigned char *map_end = map + nr_pages;
		184	unsigned long swp_tb;
187		185
188	do {	186	do {
189	VM_BUG_ON(!(*map & SWAP_HAS_CACHE));	187	swp_tb = __swap_table_get(ci, ci_off);
190	if (*map != SWAP_HAS_CACHE)	188	VM_WARN_ON_ONCE(!swp_tb_is_folio(swp_tb));
		189	if (*map)
191	return false;	190	return false;
		191	++ci_off;
192	} while (++map < map_end);	192	} while (++map < map_end);
193		193
194	return true;	194	return true;
195	}	195	}
196		196
197	static bool swap_is_last_map(struct swap_info_struct *si,
198	unsigned long offset, int nr_pages, bool *has_cache)
199	{
200	unsigned char *map = si->swap_map + offset;
201	unsigned char *map_end = map + nr_pages;
202	unsigned char count = *map;
203
204	if (swap_count(count) != 1 && swap_count(count) != SWAP_MAP_SHMEM)
205	return false;
206
207	while (++map < map_end) {
208	if (*map != count)
209	return false;
210	}
211
212	*has_cache = !!(count & SWAP_HAS_CACHE);
213	return true;
214	}
215
216	/*	197	/*
217	* returns number of pages in the folio that backs the swap entry. If positive,	198	* returns number of pages in the folio that backs the swap entry. If positive,
218	* the folio was reclaimed. If negative, the folio was not reclaimed. If 0, no	199	* the folio was reclaimed. If negative, the folio was not reclaimed. If 0, no
@@ -262,12 +243,12 @@ again: · 43 unmodified lines
262	goto out_unlock;	243	goto out_unlock;
263		244
264	/*	245	/*
265	* It's safe to delete the folio from swap cache only if the folio's	246	* It's safe to delete the folio from swap cache only if the folio
266	* swap_map is HAS_CACHE only, which means the slots have no page table	247	* is in swap cache with swap count == 0. The slots have no page table
267	* reference or pending writeback, and can't be allocated to others.	248	* reference or pending writeback, and can't be allocated to others.
268	*/	249	*/
269	ci = swap_cluster_lock(si, offset);	250	ci = swap_cluster_lock(si, offset);
270	need_reclaim = swap_only_has_cache(si, offset, nr_pages);	251	need_reclaim = swap_only_has_cache(si, ci, offset, nr_pages);
271	swap_cluster_unlock(ci);	252	swap_cluster_unlock(ci);
272	if (!need_reclaim)	253	if (!need_reclaim)
273	goto out_unlock;	254	goto out_unlock;
@@ -777,68 +758,84 @@ static int swap_cluster_setup_bad_slot(struct swap_cluster_info *cluster_info, · 503 unmodified lines
777	return 0;	758	return 0;
778	}	759	}
779		760
		761	/*
		762	* Reclaim drops the ci lock, so the cluster may become unusable (freed or
		763	* stolen by a lower order). @usable will be set to false if that happens.
		764	*/
780	static bool cluster_reclaim_range(struct swap_info_struct *si,	765	static bool cluster_reclaim_range(struct swap_info_struct *si,
781	struct swap_cluster_info *ci,	766	struct swap_cluster_info *ci,
782	unsigned long start, unsigned long end)	767	unsigned long start, unsigned int order,
		768	bool *usable)
783	{	769	{
		770	unsigned int nr_pages = 1 << order;
		771	unsigned long offset = start, end = start + nr_pages;
784	unsigned char *map = si->swap_map;	772	unsigned char *map = si->swap_map;
785	unsigned long offset = start;	773	unsigned long swp_tb;
786	int nr_reclaim;
787		774
788	spin_unlock(&ci->lock);	775	spin_unlock(&ci->lock);
789	do {	776	do {
790	switch (READ_ONCE(map[offset])) {	777	if (READ_ONCE(map[offset]))
791	case 0:
792	offset++;
793	break;
794	case SWAP_HAS_CACHE:
795	nr_reclaim = __try_to_reclaim_swap(si, offset, TTRS_ANYWAY);
796	if (nr_reclaim > 0)
797	offset += nr_reclaim;
798	else
799	goto out;
800	break;	778	break;
801	default:	779	swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);
802	goto out;	780	if (swp_tb_is_folio(swp_tb)) {
		781	if (__try_to_reclaim_swap(si, offset, TTRS_ANYWAY) < 0)
		782	break;
803	}	783	}
804	} while (offset < end);	784	} while (++offset < end);
805	out:
806	spin_lock(&ci->lock);	785	spin_lock(&ci->lock);
		786
		787	/*
		788	* We just dropped ci->lock so cluster could be used by another
		789	* order or got freed, check if it's still usable or empty.
		790	*/
		791	if (!cluster_is_usable(ci, order)) {
		792	*usable = false;
		793	return false;
		794	}
		795	*usable = true;
		796
		797	/* Fast path, no need to scan if the whole cluster is empty */
		798	if (cluster_is_empty(ci))
		799	return true;
		800
807	/*	801	/*
808	* Recheck the range no matter reclaim succeeded or not, the slot	802	* Recheck the range no matter reclaim succeeded or not, the slot
809	* could have been be freed while we are not holding the lock.	803	* could have been be freed while we are not holding the lock.
810	*/	804	*/
811	for (offset = start; offset < end; offset++)	805	for (offset = start; offset < end; offset++) {
812	if (READ_ONCE(map[offset]))	806	swp_tb = __swap_table_get(ci, offset % SWAPFILE_CLUSTER);
		807	if (map[offset] \|\| !swp_tb_is_null(swp_tb))
813	return false;	808	return false;
		809	}
814		810
815	return true;	811	return true;
816	}	812	}
817		813
818	static bool cluster_scan_range(struct swap_info_struct *si,	814	static bool cluster_scan_range(struct swap_info_struct *si,
819	struct swap_cluster_info *ci,	815	struct swap_cluster_info *ci,
820	unsigned long start, unsigned int nr_pages,	816	unsigned long offset, unsigned int nr_pages,
821	bool *need_reclaim)	817	bool *need_reclaim)
822	{	818	{
823	unsigned long offset, end = start + nr_pages;	819	unsigned long end = offset + nr_pages;
824	unsigned char *map = si->swap_map;	820	unsigned char *map = si->swap_map;
		821	unsigned long swp_tb;
825		822
826	if (cluster_is_empty(ci))	823	if (cluster_is_empty(ci))
827	return true;	824	return true;
828		825
829	for (offset = start; offset < end; offset++) {	826	do {
830	switch (READ_ONCE(map[offset])) {	827	if (map[offset])
831	case 0:	828	return false;
832	continue;	829	swp_tb = __swap_table_get(ci, offset % SWAPFILE_CLUSTER);
833	case SWAP_HAS_CACHE:	830	if (swp_tb_is_folio(swp_tb)) {
834	if (!vm_swap_full())	831	if (!vm_swap_full())
835	return false;	832	return false;
836	*need_reclaim = true;	833	*need_reclaim = true;
837	continue;	834	} else {
838	default:	835	/* A entry with no count and no cache must be null */
839	return false;	836	VM_WARN_ON_ONCE(!swp_tb_is_null(swp_tb));
840	}	837	}
841	}	838	} while (++offset < end);
842		839
843	return true;	840	return true;
844	}	841	}
@@ -863,28 +860,52 @@ static void swap_cluster_assert_table_empty(struct swap_cluster_info *ci, · 18 unmodified lines
863	}	860	}
864	}	861	}
865		862
866	static bool cluster_alloc_range(struct swap_info_struct si, struct swap_cluster_info ci,	863	static bool cluster_alloc_range(struct swap_info_struct *si,
867	unsigned int start, unsigned char usage,	864	struct swap_cluster_info *ci,
868	unsigned int order)	865	struct folio *folio,
		866	unsigned int offset)
869	{	867	{
870	unsigned int nr_pages = 1 << order;	868	unsigned long nr_pages;
		869	unsigned int order;
871		870
872	lockdep_assert_held(&ci->lock);	871	lockdep_assert_held(&ci->lock);
873		872
874	if (!(si->flags & SWP_WRITEOK))	873	if (!(si->flags & SWP_WRITEOK))
875	return false;	874	return false;
876		875
		876	/*
		877	* All mm swap allocation starts with a folio (folio_alloc_swap),
		878	* it's also the only allocation path for large orders allocation.
		879	* Such swap slots starts with count == 0 and will be increased
		880	* upon folio unmap.
		881	*
		882	* Else, it's a exclusive order 0 allocation for hibernation.
		883	* The slot starts with count == 1 and never increases.
		884	*/
		885	if (likely(folio)) {
		886	order = folio_order(folio);
		887	nr_pages = 1 << order;
		888	__swap_cache_add_folio(ci, folio, swp_entry(si->type, offset));
		889	} else if (IS_ENABLED(CONFIG_HIBERNATION)) {
		890	order = 0;
		891	nr_pages = 1;
		892	WARN_ON_ONCE(si->swap_map[offset]);
		893	si->swap_map[offset] = 1;
		894	swap_cluster_assert_table_empty(ci, offset, 1);
		895	} else {
		896	/* Allocation without folio is only possible with hibernation */
		897	WARN_ON_ONCE(1);
		898	return false;
		899	}
		900
877	/*	901	/*
878	* The first allocation in a cluster makes the	902	* The first allocation in a cluster makes the
879	* cluster exclusive to this order	903	* cluster exclusive to this order
880	*/	904	*/
881	if (cluster_is_empty(ci))	905	if (cluster_is_empty(ci))
882	ci->order = order;	906	ci->order = order;
883
884	memset(si->swap_map + start, usage, nr_pages);
885	swap_cluster_assert_table_empty(ci, start, nr_pages);
886	swap_range_alloc(si, nr_pages);
887	ci->count += nr_pages;	907	ci->count += nr_pages;
		908	swap_range_alloc(si, nr_pages);
888		909
889	return true;	910	return true;
890	}	911	}
@@ -892,17 +913,17 @@ static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster · 1 unmodified line
892	/* Try use a new cluster for current CPU and allocate from it. */	913	/* Try use a new cluster for current CPU and allocate from it. */
893	static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,	914	static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
894	struct swap_cluster_info *ci,	915	struct swap_cluster_info *ci,
895	unsigned long offset,	916	struct folio *folio, unsigned long offset)
896	unsigned int order,
897	unsigned char usage)
898	{	917	{
899	unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;	918	unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
900	unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);	919	unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);
901	unsigned long end = min(start + SWAPFILE_CLUSTER, si->max);	920	unsigned long end = min(start + SWAPFILE_CLUSTER, si->max);
		921	unsigned int order = likely(folio) ? folio_order(folio) : 0;
902	unsigned int nr_pages = 1 << order;	922	unsigned int nr_pages = 1 << order;
903	bool need_reclaim, ret;	923	bool need_reclaim, ret, usable;
904		924
905	lockdep_assert_held(&ci->lock);	925	lockdep_assert_held(&ci->lock);
		926	VM_WARN_ON(!cluster_is_usable(ci, order));
906		927
907	if (end < nr_pages \|\| ci->count + nr_pages > SWAPFILE_CLUSTER)	928	if (end < nr_pages \|\| ci->count + nr_pages > SWAPFILE_CLUSTER)
908	goto out;	929	goto out;
@@ -912,14 +933,8 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, · 3 unmodified lines
912	if (!cluster_scan_range(si, ci, offset, nr_pages, &need_reclaim))	933	if (!cluster_scan_range(si, ci, offset, nr_pages, &need_reclaim))
913	continue;	934	continue;
914	if (need_reclaim) {	935	if (need_reclaim) {
915	ret = cluster_reclaim_range(si, ci, offset, offset + nr_pages);	936	ret = cluster_reclaim_range(si, ci, offset, order, &usable);
916	/*	937	if (!usable)
917	* Reclaim drops ci->lock and cluster could be used
918	* by another order. Not checking flag as off-list
919	* cluster has no flag set, and change of list
920	* won't cause fragmentation.
921	*/
922	if (!cluster_is_usable(ci, order))
923	goto out;	938	goto out;
924	if (cluster_is_empty(ci))	939	if (cluster_is_empty(ci))
925	offset = start;	940	offset = start;
@@ -927,7 +942,7 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, · 1 unmodified line
927	if (!ret)	942	if (!ret)
928	continue;	943	continue;
929	}	944	}
930	if (!cluster_alloc_range(si, ci, offset, usage, order))	945	if (!cluster_alloc_range(si, ci, folio, offset))
931	break;	946	break;
932	found = offset;	947	found = offset;
933	offset += nr_pages;	948	offset += nr_pages;
@@ -949,8 +964,7 @@ out: · 15 unmodified lines
949		964
950	static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,	965	static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
951	struct list_head *list,	966	struct list_head *list,
952	unsigned int order,	967	struct folio *folio,
953	unsigned char usage,
954	bool scan_all)	968	bool scan_all)
955	{	969	{
956	unsigned int found = SWAP_ENTRY_INVALID;	970	unsigned int found = SWAP_ENTRY_INVALID;
@@ -962,7 +976,7 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si, · 5 unmodified lines
962	if (!ci)	976	if (!ci)
963	break;	977	break;
964	offset = cluster_offset(si, ci);	978	offset = cluster_offset(si, ci);
965	found = alloc_swap_scan_cluster(si, ci, offset, order, usage);	979	found = alloc_swap_scan_cluster(si, ci, folio, offset);
966	if (found)	980	if (found)
967	break;	981	break;
968	} while (scan_all);	982	} while (scan_all);
@@ -987,7 +1001,8 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) · 18 unmodified lines
987	to_scan--;	1001	to_scan--;
988		1002
989	while (offset < end) {	1003	while (offset < end) {
990	if (READ_ONCE(map[offset]) == SWAP_HAS_CACHE) {	1004	if (!READ_ONCE(map[offset]) &&
		1005	swp_tb_is_folio(swap_table_get(ci, offset % SWAPFILE_CLUSTER))) {
991	spin_unlock(&ci->lock);	1006	spin_unlock(&ci->lock);
992	nr_reclaim = __try_to_reclaim_swap(si, offset,	1007	nr_reclaim = __try_to_reclaim_swap(si, offset,
993	TTRS_ANYWAY);	1008	TTRS_ANYWAY);
@@ -1023,10 +1038,11 @@ static void swap_reclaim_work(struct work_struct *work) · 29 unmodified lines
1023	* Try to allocate swap entries with specified order and try set a new	1038	* Try to allocate swap entries with specified order and try set a new
1024	* cluster for current CPU too.	1039	* cluster for current CPU too.
1025	*/	1040	*/
1026	static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int order,	1041	static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
1027	unsigned char usage)	1042	struct folio *folio)
1028	{	1043	{
1029	struct swap_cluster_info *ci;	1044	struct swap_cluster_info *ci;
		1045	unsigned int order = likely(folio) ? folio_order(folio) : 0;
1030	unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;	1046	unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
1031		1047
1032	/*	1048	/*
@@ -1048,8 +1064,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o · 15 unmodified lines
1048	if (cluster_is_usable(ci, order)) {	1064	if (cluster_is_usable(ci, order)) {
1049	if (cluster_is_empty(ci))	1065	if (cluster_is_empty(ci))
1050	offset = cluster_offset(si, ci);	1066	offset = cluster_offset(si, ci);
1051	found = alloc_swap_scan_cluster(si, ci, offset,	1067	found = alloc_swap_scan_cluster(si, ci, folio, offset);
1052	order, usage);
1053	} else {	1068	} else {
1054	swap_cluster_unlock(ci);	1069	swap_cluster_unlock(ci);
1055	}	1070	}
@@ -1063,22 +1078,19 @@ new_cluster: · 7 unmodified lines
1063	* to spread out the writes.	1078	* to spread out the writes.
1064	*/	1079	*/
1065	if (si->flags & SWP_PAGE_DISCARD) {	1080	if (si->flags & SWP_PAGE_DISCARD) {
1066	found = alloc_swap_scan_list(si, &si->free_clusters, order, usage,	1081	found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
1067	false);
1068	if (found)	1082	if (found)
1069	goto done;	1083	goto done;
1070	}	1084	}
1071		1085
1072	if (order < PMD_ORDER) {	1086	if (order < PMD_ORDER) {
1073	found = alloc_swap_scan_list(si, &si->nonfull_clusters[order],	1087	found = alloc_swap_scan_list(si, &si->nonfull_clusters[order], folio, true);
1074	order, usage, true);
1075	if (found)	1088	if (found)
1076	goto done;	1089	goto done;
1077	}	1090	}
1078		1091
1079	if (!(si->flags & SWP_PAGE_DISCARD)) {	1092	if (!(si->flags & SWP_PAGE_DISCARD)) {
1080	found = alloc_swap_scan_list(si, &si->free_clusters, order, usage,	1093	found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
1081	false);
1082	if (found)	1094	if (found)
1083	goto done;	1095	goto done;
1084	}	1096	}
@@ -1092,10 +1104,9 @@ new_cluster: · 7 unmodified lines
1092	* Scan only one fragment cluster is good enough. Order 0	1104	* Scan only one fragment cluster is good enough. Order 0
1093	* allocation will surely success, and large allocation	1105	* allocation will surely success, and large allocation
1094	* failure is not critical. Scanning one cluster still	1106	* failure is not critical. Scanning one cluster still
1095	* keeps the list rotated and reclaimed (for HAS_CACHE).	1107	* keeps the list rotated and reclaimed (for clean swap cache).
1096	*/	1108	*/
1097	found = alloc_swap_scan_list(si, &si->frag_clusters[order], order,	1109	found = alloc_swap_scan_list(si, &si->frag_clusters[order], folio, false);
1098	usage, false);
1099	if (found)	1110	if (found)
1100	goto done;	1111	goto done;
1101	}	1112	}
@@ -1109,13 +1120,11 @@ new_cluster: · 7 unmodified lines
1109	* Clusters here have at least one usable slots and can't fail order 0	1120	* Clusters here have at least one usable slots and can't fail order 0
1110	* allocation, but reclaim may drop si->lock and race with another user.	1121	* allocation, but reclaim may drop si->lock and race with another user.
1111	*/	1122	*/
1112	found = alloc_swap_scan_list(si, &si->frag_clusters[o],	1123	found = alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true);
1113	0, usage, true);
1114	if (found)	1124	if (found)
1115	goto done;	1125	goto done;
1116		1126
1117	found = alloc_swap_scan_list(si, &si->nonfull_clusters[o],	1127	found = alloc_swap_scan_list(si, &si->nonfull_clusters[o], folio, true);
1118	0, usage, true);
1119	if (found)	1128	if (found)
1120	goto done;	1129	goto done;
1121	}	1130	}
@@ -1306,12 +1315,12 @@ static bool get_swap_device_info(struct swap_info_struct *si) · 184 unmodified lines
1306	* Fast path try to get swap entries with specified order from current	1315	* Fast path try to get swap entries with specified order from current
1307	* CPU's swap entry pool (a cluster).	1316	* CPU's swap entry pool (a cluster).
1308	*/	1317	*/
1309	static bool swap_alloc_fast(swp_entry_t *entry,	1318	static bool swap_alloc_fast(struct folio *folio)
1310	int order)
1311	{	1319	{
		1320	unsigned int order = folio_order(folio);
1312	struct swap_cluster_info *ci;	1321	struct swap_cluster_info *ci;
1313	struct swap_info_struct *si;	1322	struct swap_info_struct *si;
1314	unsigned int offset, found = SWAP_ENTRY_INVALID;	1323	unsigned int offset;
1315		1324
1316	/*	1325	/*
1317	* Once allocated, swap_info_struct will never be completely freed,	1326	* Once allocated, swap_info_struct will never be completely freed,
@@ -1326,22 +1335,18 @@ static bool swap_alloc_fast(swp_entry_t *entry, · 8 unmodified lines
1326	if (cluster_is_usable(ci, order)) {	1335	if (cluster_is_usable(ci, order)) {
1327	if (cluster_is_empty(ci))	1336	if (cluster_is_empty(ci))
1328	offset = cluster_offset(si, ci);	1337	offset = cluster_offset(si, ci);
1329	found = alloc_swap_scan_cluster(si, ci, offset, order, SWAP_HAS_CACHE);	1338	alloc_swap_scan_cluster(si, ci, folio, offset);
1330	if (found)
1331	*entry = swp_entry(si->type, found);
1332	} else {	1339	} else {
1333	swap_cluster_unlock(ci);	1340	swap_cluster_unlock(ci);
1334	}	1341	}
1335		1342
1336	put_swap_device(si);	1343	put_swap_device(si);
1337	return !!found;	1344	return folio_test_swapcache(folio);
1338	}	1345	}
1339		1346
1340	/* Rotate the device and switch to a new cluster */	1347	/* Rotate the device and switch to a new cluster */
1341	static void swap_alloc_slow(swp_entry_t *entry,	1348	static void swap_alloc_slow(struct folio *folio)
1342	int order)
1343	{	1349	{
1344	unsigned long offset;
1345	struct swap_info_struct si, next;	1350	struct swap_info_struct si, next;
1346		1351
1347	spin_lock(&swap_avail_lock);	1352	spin_lock(&swap_avail_lock);
@@ -1351,13 +1356,11 @@ start_over: · 3 unmodified lines
1351	plist_requeue(&si->avail_list, &swap_avail_head);	1356	plist_requeue(&si->avail_list, &swap_avail_head);
1352	spin_unlock(&swap_avail_lock);	1357	spin_unlock(&swap_avail_lock);
1353	if (get_swap_device_info(si)) {	1358	if (get_swap_device_info(si)) {
1354	offset = cluster_alloc_swap_entry(si, order, SWAP_HAS_CACHE);	1359	cluster_alloc_swap_entry(si, folio);
1355	put_swap_device(si);	1360	put_swap_device(si);
1356	if (offset) {	1361	if (folio_test_swapcache(folio))
1357	*entry = swp_entry(si->type, offset);
1358	return;	1362	return;
1359	}	1363	if (folio_test_large(folio))
1360	if (order)
1361	return;	1364	return;
1362	}	1365	}
1363		1366
@@ -1408,6 +1411,75 @@ start_over: · 44 unmodified lines
1408	return false;	1411	return false;
1409	}	1412	}
1410		1413
		1414	/**
		1415	* swap_put_entries_cluster - Decrease the swap count of a set of slots.
		1416	* @si: The swap device.
		1417	* @start: start offset of slots.
		1418	* @nr: number of slots.
		1419	* @reclaim_cache: if true, also reclaim the swap cache.
		1420	*
		1421	* This helper decreases the swap count of a set of slots and tries to
		1422	* batch free them. Also reclaims the swap cache if @reclaim_cache is true.
		1423	* Context: The caller must ensure that all slots belong to the same
		1424	* cluster and their swap count doesn't go underflow.
		1425	*/
		1426	static void swap_put_entries_cluster(struct swap_info_struct *si,
		1427	unsigned long start, int nr,
		1428	bool reclaim_cache)
		1429	{
		1430	unsigned long offset = start, end = start + nr;
		1431	unsigned long batch_start = SWAP_ENTRY_INVALID;
		1432	struct swap_cluster_info *ci;
		1433	bool need_reclaim = false;
		1434	unsigned int nr_reclaimed;
		1435	unsigned long swp_tb;
		1436	unsigned int count;
		1437
		1438	ci = swap_cluster_lock(si, offset);
		1439	do {
		1440	swp_tb = __swap_table_get(ci, offset % SWAPFILE_CLUSTER);
		1441	count = si->swap_map[offset];
		1442	VM_WARN_ON(count < 1 \|\| count == SWAP_MAP_BAD);
		1443	if (count == 1) {
		1444	/* count == 1 and non-cached slots will be batch freed. */
		1445	if (!swp_tb_is_folio(swp_tb)) {
		1446	if (!batch_start)
		1447	batch_start = offset;
		1448	continue;
		1449	}
		1450	/* count will be 0 after put, slot can be reclaimed */
		1451	need_reclaim = true;
		1452	}
		1453	/*
		1454	* A count != 1 or cached slot can't be freed. Put its swap
		1455	* count and then free the interrupted pending batch. Cached
		1456	* slots will be freed when folio is removed from swap cache
		1457	* (__swap_cache_del_folio).
		1458	*/
		1459	swap_put_entry_locked(si, ci, offset);
		1460	if (batch_start) {
		1461	swap_entries_free(si, ci, batch_start, offset - batch_start);
		1462	batch_start = SWAP_ENTRY_INVALID;
		1463	}
		1464	} while (++offset < end);
		1465
		1466	if (batch_start)
		1467	swap_entries_free(si, ci, batch_start, offset - batch_start);
		1468	swap_cluster_unlock(ci);
		1469
		1470	if (!need_reclaim \|\| !reclaim_cache)
		1471	return;
		1472
		1473	offset = start;
		1474	do {
		1475	nr_reclaimed = __try_to_reclaim_swap(si, offset,
		1476	TTRS_UNMAPPED \| TTRS_FULL);
		1477	offset++;
		1478	if (nr_reclaimed)
		1479	offset = round_up(offset, abs(nr_reclaimed));
		1480	} while (offset < end);
		1481	}
		1482
1411	/**	1483	/**
1412	* folio_alloc_swap - allocate swap space for a folio	1484	* folio_alloc_swap - allocate swap space for a folio
1413	* @folio: folio we want to move to swap	1485	* @folio: folio we want to move to swap
@@ -1422,7 +1494,6 @@ int folio_alloc_swap(struct folio *folio) · 8 unmodified lines
1422	{	1494	{
1423	unsigned int order = folio_order(folio);	1495	unsigned int order = folio_order(folio);
1424	unsigned int size = 1 << order;	1496	unsigned int size = 1 << order;
1425	swp_entry_t entry = {};
1426		1497
1427	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);	1498	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
1428	VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);	1499	VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -1447,89 +1518,94 @@ int folio_alloc_swap(struct folio *folio) · 18 unmodified lines
1447		1518
1448	again:	1519	again:
1449	local_lock(&percpu_swap_cluster.lock);	1520	local_lock(&percpu_swap_cluster.lock);
1450	if (!swap_alloc_fast(&entry, order))	1521	if (!swap_alloc_fast(folio))
1451	swap_alloc_slow(&entry, order);	1522	swap_alloc_slow(folio);
1452	local_unlock(&percpu_swap_cluster.lock);	1523	local_unlock(&percpu_swap_cluster.lock);
1453		1524
1454	if (unlikely(!order && !entry.val)) {	1525	if (!order && unlikely(!folio_test_swapcache(folio))) {
1455	if (swap_sync_discard())	1526	if (swap_sync_discard())
1456	goto again;	1527	goto again;
1457	}	1528	}
1458		1529
1459	/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */	1530	/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
1460	if (mem_cgroup_try_charge_swap(folio, entry))	1531	if (unlikely(mem_cgroup_try_charge_swap(folio, folio->swap)))
1461	goto out_free;	1532	swap_cache_del_folio(folio);
1462		1533
1463	if (!entry.val)	1534	if (unlikely(!folio_test_swapcache(folio)))
1464	return -ENOMEM;	1535	return -ENOMEM;
1465		1536
1466	swap_cache_add_folio(folio, entry, NULL);
1467
1468	return 0;	1537	return 0;
		1538	}
		1539
		1540	/**
		1541	* folio_dup_swap() - Increase swap count of swap entries of a folio.
		1542	* @folio: folio with swap entries bounded.
		1543	* @subpage: if not NULL, only increase the swap count of this subpage.
		1544	*
		1545	* Typically called when the folio is unmapped and have its swap entry to
		1546	* take its palce.
		1547	*
		1548	* Context: Caller must ensure the folio is locked and in the swap cache.
		1549	* NOTE: The caller also has to ensure there is no raced call to
		1550	* swap_put_entries_direct on its swap entry before this helper returns, or
		1551	* the swap map may underflow. Currently, we only accept @subpage == NULL
		1552	* for shmem due to the limitation of swap continuation: shmem always
		1553	* duplicates the swap entry only once, so there is no such issue for it.
		1554	*/
		1555	int folio_dup_swap(struct folio folio, struct page subpage)
		1556	{
		1557	int err = 0;
		1558	swp_entry_t entry = folio->swap;
		1559	unsigned long nr_pages = folio_nr_pages(folio);
		1560
		1561	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
		1562	VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio);
		1563
		1564	if (subpage) {
		1565	entry.val += folio_page_idx(folio, subpage);
		1566	nr_pages = 1;
		1567	}
		1568
		1569	while (!err && __swap_duplicate(entry, 1, nr_pages) == -ENOMEM)
		1570	err = add_swap_count_continuation(entry, GFP_ATOMIC);
1469		1571
1470	out_free:	1572	return err;
1471	put_swap_folio(folio, entry);
1472	return -ENOMEM;
1473	}	1573	}
1474		1574
1475	static struct swap_info_struct *_swap_info_get(swp_entry_t entry)	1575	/**
		1576	* folio_put_swap() - Decrease swap count of swap entries of a folio.
		1577	* @folio: folio with swap entries bounded, must be in swap cache and locked.
		1578	* @subpage: if not NULL, only decrease the swap count of this subpage.
		1579	*
		1580	* This won't free the swap slots even if swap count drops to zero, they are
		1581	* still pinned by the swap cache. User may call folio_free_swap to free them.
		1582	* Context: Caller must ensure the folio is locked and in the swap cache.
		1583	*/
		1584	void folio_put_swap(struct folio folio, struct page subpage)
1476	{	1585	{
1477	struct swap_info_struct *si;	1586	swp_entry_t entry = folio->swap;
1478	unsigned long offset;	1587	unsigned long nr_pages = folio_nr_pages(folio);
		1588	struct swap_info_struct *si = __swap_entry_to_info(entry);
1479		1589
1480	if (!entry.val)	1590	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
1481	goto out;	1591	VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio);
1482	si = swap_entry_to_info(entry);
1483	if (!si)
1484	goto bad_nofile;
1485	if (data_race(!(si->flags & SWP_USED)))
1486	goto bad_device;
1487	offset = swp_offset(entry);
1488	if (offset >= si->max)
1489	goto bad_offset;
1490	if (data_race(!si->swap_map[swp_offset(entry)]))
1491	goto bad_free;
1492	return si;
1493		1592
1494	bad_free:	1593	if (subpage) {
1495	pr_err("%s: %s%08lx\n", __func__, Unused_offset, entry.val);	1594	entry.val += folio_page_idx(folio, subpage);
1496	goto out;	1595	nr_pages = 1;
1497	bad_offset:	1596	}
1498	pr_err("%s: %s%08lx\n", __func__, Bad_offset, entry.val);	1597
1499	goto out;	1598	swap_put_entries_cluster(si, swp_offset(entry), nr_pages, false);
1500	bad_device:
1501	pr_err("%s: %s%08lx\n", __func__, Unused_file, entry.val);
1502	goto out;
1503	bad_nofile:
1504	pr_err("%s: %s%08lx\n", __func__, Bad_file, entry.val);
1505	out:
1506	return NULL;
1507	}	1599	}
1508		1600
1509	static unsigned char swap_entry_put_locked(struct swap_info_struct *si,	1601	static void swap_put_entry_locked(struct swap_info_struct *si,
1510	struct swap_cluster_info *ci,	1602	struct swap_cluster_info *ci,
1511	swp_entry_t entry,	1603	unsigned long offset)
1512	unsigned char usage)
1513	{	1604	{
1514	unsigned long offset = swp_offset(entry);
1515	unsigned char count;	1605	unsigned char count;
1516	unsigned char has_cache;
1517		1606
1518	count = si->swap_map[offset];	1607	count = si->swap_map[offset];
1519		1608	if ((count & ~COUNT_CONTINUED) <= SWAP_MAP_MAX) {
1520	has_cache = count & SWAP_HAS_CACHE;
1521	count &= ~SWAP_HAS_CACHE;
1522
1523	if (usage == SWAP_HAS_CACHE) {
1524	VM_BUG_ON(!has_cache);
1525	has_cache = 0;
1526	} else if (count == SWAP_MAP_SHMEM) {
1527	/*
1528	* Or we could insist on shmem.c using a special
1529	* swap_shmem_free() and free_shmem_swap_and_cache()...
1530	*/
1531	count = 0;
1532	} else if ((count & ~COUNT_CONTINUED) <= SWAP_MAP_MAX) {
1533	if (count == COUNT_CONTINUED) {	1609	if (count == COUNT_CONTINUED) {
1534	if (swap_count_continued(si, offset, count))	1610	if (swap_count_continued(si, offset, count))
1535	count = SWAP_MAP_MAX \| COUNT_CONTINUED;	1611	count = SWAP_MAP_MAX \| COUNT_CONTINUED;
@@ -1539,13 +1615,9 @@ static unsigned char swap_entry_put_locked(struct swap_info_struct *si, · 3 unmodified lines
1539	count--;	1615	count--;
1540	}	1616	}
1541		1617
1542	usage = count \| has_cache;	1618	WRITE_ONCE(si->swap_map[offset], count);
1543	if (usage)	1619	if (!count && !swp_tb_is_folio(__swap_table_get(ci, offset % SWAPFILE_CLUSTER)))
1544	WRITE_ONCE(si->swap_map[offset], usage);	1620	swap_entries_free(si, ci, offset, 1);
1545	else
1546	swap_entries_free(si, ci, entry, 1);
1547
1548	return usage;
1549	}	1621	}
1550		1622
1551	/*	1623	/*
@@ -1574,10 +1646,9 @@ static unsigned char swap_entry_put_locked(struct swap_info_struct *si, · 22 unmodified lines
1574	* CPU1 CPU2	1646	* CPU1 CPU2
1575	* do_swap_page()	1647	* do_swap_page()
1576	* ... swapoff+swapon	1648	* ... swapoff+swapon
1577	* __read_swap_cache_async()	1649	* swap_cache_alloc_folio()
1578	* swapcache_prepare()	1650	* swap_cache_add_folio()
1579	* __swap_duplicate()	1651	* // check swap_map
1580	* // check swap_map
1581	* // verify PTE not changed	1652	* // verify PTE not changed
1582	*	1653	*
1583	* In __swap_duplicate(), the swap_map need to be checked before	1654	* In __swap_duplicate(), the swap_map need to be checked before
@@ -1614,105 +1685,15 @@ put_out: · 30 unmodified lines
1614	return NULL;	1685	return NULL;
1615	}	1686	}
1616		1687
1617	static void swap_entries_put_cache(struct swap_info_struct *si,
1618	swp_entry_t entry, int nr)
1619	{
1620	unsigned long offset = swp_offset(entry);
1621	struct swap_cluster_info *ci;
1622
1623	ci = swap_cluster_lock(si, offset);
1624	if (swap_only_has_cache(si, offset, nr)) {
1625	swap_entries_free(si, ci, entry, nr);
1626	} else {
1627	for (int i = 0; i < nr; i++, entry.val++)
1628	swap_entry_put_locked(si, ci, entry, SWAP_HAS_CACHE);
1629	}
1630	swap_cluster_unlock(ci);
1631	}
1632
1633	static bool swap_entries_put_map(struct swap_info_struct *si,
1634	swp_entry_t entry, int nr)
1635	{
1636	unsigned long offset = swp_offset(entry);
1637	struct swap_cluster_info *ci;
1638	bool has_cache = false;
1639	unsigned char count;
1640	int i;
1641
1642	if (nr <= 1)
1643	goto fallback;
1644	count = swap_count(data_race(si->swap_map[offset]));
1645	if (count != 1 && count != SWAP_MAP_SHMEM)
1646	goto fallback;
1647
1648	ci = swap_cluster_lock(si, offset);
1649	if (!swap_is_last_map(si, offset, nr, &has_cache)) {
1650	goto locked_fallback;
1651	}
1652	if (!has_cache)
1653	swap_entries_free(si, ci, entry, nr);
1654	else
1655	for (i = 0; i < nr; i++)
1656	WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE);
1657	swap_cluster_unlock(ci);
1658
1659	return has_cache;
1660
1661	fallback:
1662	ci = swap_cluster_lock(si, offset);
1663	locked_fallback:
1664	for (i = 0; i < nr; i++, entry.val++) {
1665	count = swap_entry_put_locked(si, ci, entry, 1);
1666	if (count == SWAP_HAS_CACHE)
1667	has_cache = true;
1668	}
1669	swap_cluster_unlock(ci);
1670	return has_cache;
1671	}
1672
1673	/*
1674	* Only functions with "_nr" suffix are able to free entries spanning
1675	* cross multi clusters, so ensure the range is within a single cluster
1676	* when freeing entries with functions without "_nr" suffix.
1677	*/
1678	static bool swap_entries_put_map_nr(struct swap_info_struct *si,
1679	swp_entry_t entry, int nr)
1680	{
1681	int cluster_nr, cluster_rest;
1682	unsigned long offset = swp_offset(entry);
1683	bool has_cache = false;
1684
1685	cluster_rest = SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER;
1686	while (nr) {
1687	cluster_nr = min(nr, cluster_rest);
1688	has_cache \|= swap_entries_put_map(si, entry, cluster_nr);
1689	cluster_rest = SWAPFILE_CLUSTER;
1690	nr -= cluster_nr;
1691	entry.val += cluster_nr;
1692	}
1693
1694	return has_cache;
1695	}
1696
1697	/*
1698	* Check if it's the last ref of swap entry in the freeing path.
1699	* Qualified value includes 1, SWAP_HAS_CACHE or SWAP_MAP_SHMEM.
1700	*/
1701	static inline bool __maybe_unused swap_is_last_ref(unsigned char count)
1702	{
1703	return (count == SWAP_HAS_CACHE) \|\| (count == 1) \|\|
1704	(count == SWAP_MAP_SHMEM);
1705	}
1706
1707	/*	1688	/*
1708	* Drop the last ref of swap entries, caller have to ensure all entries	1689	* Drop the last ref of swap entries, caller have to ensure all entries
1709	* belong to the same cgroup and cluster.	1690	* belong to the same cgroup and cluster.
1710	*/	1691	*/
1711	static void swap_entries_free(struct swap_info_struct *si,	1692	void swap_entries_free(struct swap_info_struct *si,
1712	struct swap_cluster_info *ci,	1693	struct swap_cluster_info *ci,
1713	swp_entry_t entry, unsigned int nr_pages)	1694	unsigned long offset, unsigned int nr_pages)
1714	{	1695	{
1715	unsigned long offset = swp_offset(entry);	1696	swp_entry_t entry = swp_entry(si->type, offset);
1716	unsigned char *map = si->swap_map + offset;	1697	unsigned char *map = si->swap_map + offset;
1717	unsigned char *map_end = map + nr_pages;	1698	unsigned char *map_end = map + nr_pages;
1718		1699
@@ -1723,7 +1704,7 @@ static void swap_entries_free(struct swap_info_struct *si, · 4 unmodified lines
1723		1704
1724	ci->count -= nr_pages;	1705	ci->count -= nr_pages;
1725	do {	1706	do {
1726	VM_BUG_ON(!swap_is_last_ref(*map));	1707	VM_WARN_ON(*map > 1);
1727	*map = 0;	1708	*map = 0;
1728	} while (++map < map_end);	1709	} while (++map < map_end);
1729		1710
@@ -1737,55 +1718,18 @@ static void swap_entries_free(struct swap_info_struct *si, · 7 unmodified lines
1737	partial_free_cluster(si, ci);	1718	partial_free_cluster(si, ci);
1738	}	1719	}
1739		1720
1740	/*
1741	* Caller has made sure that the swap device corresponding to entry
1742	* is still around or has not been recycled.
1743	*/
1744	void swap_free_nr(swp_entry_t entry, int nr_pages)
1745	{
1746	int nr;
1747	struct swap_info_struct *sis;
1748	unsigned long offset = swp_offset(entry);
1749
1750	sis = _swap_info_get(entry);
1751	if (!sis)
1752	return;
1753
1754	while (nr_pages) {
1755	nr = min_t(int, nr_pages, SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER);
1756	swap_entries_put_map(sis, swp_entry(sis->type, offset), nr);
1757	offset += nr;
1758	nr_pages -= nr;
1759	}
1760	}
1761
1762	/*
1763	* Called after dropping swapcache to decrease refcnt to swap entries.
1764	*/
1765	void put_swap_folio(struct folio *folio, swp_entry_t entry)
1766	{
1767	struct swap_info_struct *si;
1768	int size = 1 << swap_entry_order(folio_order(folio));
1769
1770	si = _swap_info_get(entry);
1771	if (!si)
1772	return;
1773
1774	swap_entries_put_cache(si, entry, size);
1775	}
1776
1777	int __swap_count(swp_entry_t entry)	1721	int __swap_count(swp_entry_t entry)
1778	{	1722	{
1779	struct swap_info_struct *si = __swap_entry_to_info(entry);	1723	struct swap_info_struct *si = __swap_entry_to_info(entry);
1780	pgoff_t offset = swp_offset(entry);	1724	pgoff_t offset = swp_offset(entry);
1781		1725
1782	return swap_count(si->swap_map[offset]);	1726	return si->swap_map[offset];
1783	}	1727	}
1784		1728
1785	/*	1729	/**
1786	* How many references to @entry are currently swapped out?	1730	* swap_entry_swapped - Check if the swap entry is swapped.
1787	* This does not give an exact answer when swap count is continued,	1731	* @si: the swap device.
1788	* but does include the high COUNT_CONTINUED flag to allow for that.	1732	* @entry: the swap entry.
1789	*/	1733	*/
1790	bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry)	1734	bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry)
1791	{	1735	{
@@ -1794,9 +1738,10 @@ bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry) · 2 unmodified lines
1794	int count;	1738	int count;
1795		1739
1796	ci = swap_cluster_lock(si, offset);	1740	ci = swap_cluster_lock(si, offset);
1797	count = swap_count(si->swap_map[offset]);	1741	count = si->swap_map[offset];
1798	swap_cluster_unlock(ci);	1742	swap_cluster_unlock(ci);
1799	return !!count;	1743
		1744	return count && count != SWAP_MAP_BAD;
1800	}	1745	}
1801		1746
1802	/*	1747	/*
@@ -1812,7 +1757,7 @@ int swp_swapcount(swp_entry_t entry) · 9 unmodified lines
1812	pgoff_t offset;	1757	pgoff_t offset;
1813	unsigned char *map;	1758	unsigned char *map;
1814		1759
1815	si = _swap_info_get(entry);	1760	si = get_swap_device(entry);
1816	if (!si)	1761	if (!si)
1817	return 0;	1762	return 0;
1818		1763
@@ -1820,7 +1765,7 @@ int swp_swapcount(swp_entry_t entry) · 1 unmodified line
1820		1765
1821	ci = swap_cluster_lock(si, offset);	1766	ci = swap_cluster_lock(si, offset);
1822		1767
1823	count = swap_count(si->swap_map[offset]);	1768	count = si->swap_map[offset];
1824	if (!(count & COUNT_CONTINUED))	1769	if (!(count & COUNT_CONTINUED))
1825	goto out;	1770	goto out;
1826		1771
@@ -1842,6 +1787,7 @@ int swp_swapcount(swp_entry_t entry) · 15 unmodified lines
1842	} while (tmp_count & COUNT_CONTINUED);	1787	} while (tmp_count & COUNT_CONTINUED);
1843	out:	1788	out:
1844	swap_cluster_unlock(ci);	1789	swap_cluster_unlock(ci);
		1790	put_swap_device(si);
1845	return count;	1791	return count;
1846	}	1792	}
1847		1793
@@ -1858,12 +1804,12 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, · 10 unmodified lines
1858		1804
1859	ci = swap_cluster_lock(si, offset);	1805	ci = swap_cluster_lock(si, offset);
1860	if (nr_pages == 1) {	1806	if (nr_pages == 1) {
1861	if (swap_count(map[roffset]))	1807	if (map[roffset])
1862	ret = true;	1808	ret = true;
1863	goto unlock_out;	1809	goto unlock_out;
1864	}	1810	}
1865	for (i = 0; i < nr_pages; i++) {	1811	for (i = 0; i < nr_pages; i++) {
1866	if (swap_count(map[offset + i])) {	1812	if (map[offset + i]) {
1867	ret = true;	1813	ret = true;
1868	break;	1814	break;
1869	}	1815	}
@@ -1876,11 +1822,12 @@ unlock_out: · 6 unmodified lines
1876	static bool folio_swapped(struct folio *folio)	1822	static bool folio_swapped(struct folio *folio)
1877	{	1823	{
1878	swp_entry_t entry = folio->swap;	1824	swp_entry_t entry = folio->swap;
1879	struct swap_info_struct *si = _swap_info_get(entry);	1825	struct swap_info_struct *si;
1880		1826
1881	if (!si)	1827	VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
1882	return false;	1828	VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
1883		1829
		1830	si = __swap_entry_to_info(entry);
1884	if (!IS_ENABLED(CONFIG_THP_SWAP) \|\| likely(!folio_test_large(folio)))	1831	if (!IS_ENABLED(CONFIG_THP_SWAP) \|\| likely(!folio_test_large(folio)))
1885	return swap_entry_swapped(si, entry);	1832	return swap_entry_swapped(si, entry);
1886		1833
@@ -1939,73 +1886,45 @@ bool folio_free_swap(struct folio *folio) · 52 unmodified lines
1939	}	1886	}
1940		1887
1941	/**	1888	/**
1942	* free_swap_and_cache_nr() - Release reference on range of swap entries and	1889	* swap_put_entries_direct() - Release reference on range of swap entries and
1943	* reclaim their cache if no more references remain.	1890	* reclaim their cache if no more references remain.
1944	* @entry: First entry of range.	1891	* @entry: First entry of range.
1945	* @nr: Number of entries in range.	1892	* @nr: Number of entries in range.
1946	*	1893	*
1947	* For each swap entry in the contiguous range, release a reference. If any swap	1894	* For each swap entry in the contiguous range, release a reference. If any swap
1948	* entries become free, try to reclaim their underlying folios, if present. The	1895	* entries become free, try to reclaim their underlying folios, if present. The
1949	* offset range is defined by [entry.offset, entry.offset + nr).	1896	* offset range is defined by [entry.offset, entry.offset + nr).
		1897	*
		1898	* Context: Caller must ensure there is no race condition on the reference
		1899	* owner. e.g., locking the PTL of a PTE containing the entry being released.
1950	*/	1900	*/
1951	void free_swap_and_cache_nr(swp_entry_t entry, int nr)	1901	void swap_put_entries_direct(swp_entry_t entry, int nr)
1952	{	1902	{
1953	const unsigned long start_offset = swp_offset(entry);	1903	const unsigned long start_offset = swp_offset(entry);
1954	const unsigned long end_offset = start_offset + nr;	1904	const unsigned long end_offset = start_offset + nr;
		1905	unsigned long offset, cluster_end;
1955	struct swap_info_struct *si;	1906	struct swap_info_struct *si;
1956	bool any_only_cache = false;
1957	unsigned long offset;
1958		1907
1959	si = get_swap_device(entry);	1908	si = get_swap_device(entry);
1960	if (!si)	1909	if (WARN_ON_ONCE(!si))
1961	return;	1910	return;
1962		1911	if (WARN_ON_ONCE(end_offset > si->max))
1963	if (WARN_ON(end_offset > si->max))
1964	goto out;
1965
1966	/*
1967	* First free all entries in the range.
1968	*/
1969	any_only_cache = swap_entries_put_map_nr(si, entry, nr);
1970
1971	/*
1972	* Short-circuit the below loop if none of the entries had their
1973	* reference drop to zero.
1974	*/
1975	if (!any_only_cache)
1976	goto out;	1912	goto out;
1977		1913
1978	/*	1914	/* Put entries and reclaim cache in each cluster */
1979	* Now go back over the range trying to reclaim the swap cache.	1915	offset = start_offset;
1980	*/	1916	do {
1981	for (offset = start_offset; offset < end_offset; offset += nr) {	1917	cluster_end = min(round_up(offset + 1, SWAPFILE_CLUSTER), end_offset);
1982	nr = 1;	1918	swap_put_entries_cluster(si, offset, cluster_end - offset, true);
1983	if (READ_ONCE(si->swap_map[offset]) == SWAP_HAS_CACHE) {	1919	offset = cluster_end;
1984	/*	1920	} while (offset < end_offset);
1985	* Folios are always naturally aligned in swap so
1986	* advance forward to the next boundary. Zero means no
1987	* folio was found for the swap entry, so advance by 1
1988	* in this case. Negative value means folio was found
1989	* but could not be reclaimed. Here we can still advance
1990	* to the next boundary.
1991	*/
1992	nr = __try_to_reclaim_swap(si, offset,
1993	TTRS_UNMAPPED \| TTRS_FULL);
1994	if (nr == 0)
1995	nr = 1;
1996	else if (nr < 0)
1997	nr = -nr;
1998	nr = ALIGN(offset + 1, nr) - offset;
1999	}
2000	}
2001
2002	out:	1921	out:
2003	put_swap_device(si);	1922	put_swap_device(si);
2004	}	1923	}
2005		1924
2006	#ifdef CONFIG_HIBERNATION	1925	#ifdef CONFIG_HIBERNATION
2007		1926	/* Allocate a slot for hibernation */
2008	swp_entry_t get_swap_page_of_type(int type)	1927	swp_entry_t swap_alloc_hibernation_slot(int type)
2009	{	1928	{
2010	struct swap_info_struct *si = swap_type_to_info(type);	1929	struct swap_info_struct *si = swap_type_to_info(type);
2011	unsigned long offset;	1930	unsigned long offset;
@@ -2018,11 +1937,11 @@ swp_entry_t get_swap_page_of_type(int type) · 6 unmodified lines
2018	if (get_swap_device_info(si)) {	1937	if (get_swap_device_info(si)) {
2019	if (si->flags & SWP_WRITEOK) {	1938	if (si->flags & SWP_WRITEOK) {
2020	/*	1939	/*
2021	* Grab the local lock to be complaint	1940	* Grab the local lock to be compliant
2022	* with swap table allocation.	1941	* with swap table allocation.
2023	*/	1942	*/
2024	local_lock(&percpu_swap_cluster.lock);	1943	local_lock(&percpu_swap_cluster.lock);
2025	offset = cluster_alloc_swap_entry(si, 0, 1);	1944	offset = cluster_alloc_swap_entry(si, NULL);
2026	local_unlock(&percpu_swap_cluster.lock);	1945	local_unlock(&percpu_swap_cluster.lock);
2027	if (offset)	1946	if (offset)
2028	entry = swp_entry(si->type, offset);	1947	entry = swp_entry(si->type, offset);
@@ -2033,6 +1952,26 @@ fail: · 4 unmodified lines
2033	return entry;	1952	return entry;
2034	}	1953	}
2035		1954
		1955	/* Free a slot allocated by swap_alloc_hibernation_slot */
		1956	void swap_free_hibernation_slot(swp_entry_t entry)
		1957	{
		1958	struct swap_info_struct *si;
		1959	struct swap_cluster_info *ci;
		1960	pgoff_t offset = swp_offset(entry);
		1961
		1962	si = get_swap_device(entry);
		1963	if (WARN_ON(!si))
		1964	return;
		1965
		1966	ci = swap_cluster_lock(si, offset);
		1967	swap_put_entry_locked(si, ci, offset);
		1968	swap_cluster_unlock(ci);
		1969
		1970	/* In theory readahead might add it to the swap cache by accident */
		1971	__try_to_reclaim_swap(si, offset, TTRS_ANYWAY);
		1972	put_swap_device(si);
		1973	}
		1974
2036	/*	1975	/*
2037	* Find the swap type that corresponds to given device (if any).	1976	* Find the swap type that corresponds to given device (if any).
2038	*	1977	*
@@ -2194,7 +2133,7 @@ static int unuse_pte(struct vm_area_struct vma, pmd_t pmd, · 155 unmodified lines
2194	/*	2133	/*
2195	* Some architectures may have to restore extra metadata to the page	2134	* Some architectures may have to restore extra metadata to the page
2196	* when reading from swap. This metadata may be indexed by swap entry	2135	* when reading from swap. This metadata may be indexed by swap entry
2197	* so this must be called before swap_free().	2136	* so this must be called before folio_put_swap().
2198	*/	2137	*/
2199	arch_swap_restore(folio_swap(entry, folio), folio);	2138	arch_swap_restore(folio_swap(entry, folio), folio);
2200		2139
@@ -2235,7 +2174,7 @@ static int unuse_pte(struct vm_area_struct vma, pmd_t pmd, · 34 unmodified lines
2235	new_pte = pte_mkuffd_wp(new_pte);	2174	new_pte = pte_mkuffd_wp(new_pte);
2236	setpte:	2175	setpte:
2237	set_pte_at(vma->vm_mm, addr, pte, new_pte);	2176	set_pte_at(vma->vm_mm, addr, pte, new_pte);
2238	swap_free(entry);	2177	folio_put_swap(swapcache, folio_file_page(swapcache, swp_offset(entry)));
2239	out:	2178	out:
2240	if (pte)	2179	if (pte)
2241	pte_unmap_unlock(pte, ptl);	2180	pte_unmap_unlock(pte, ptl);
@@ -2430,6 +2369,7 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si, · 188 unmodified lines
2430	unsigned int prev)	2369	unsigned int prev)
2431	{	2370	{
2432	unsigned int i;	2371	unsigned int i;
		2372	unsigned long swp_tb;
2433	unsigned char count;	2373	unsigned char count;
2434		2374
2435	/*	2375	/*
@@ -2440,7 +2380,11 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si, · 4 unmodified lines
2440	*/	2380	*/
2441	for (i = prev + 1; i < si->max; i++) {	2381	for (i = prev + 1; i < si->max; i++) {
2442	count = READ_ONCE(si->swap_map[i]);	2382	count = READ_ONCE(si->swap_map[i]);
2443	if (count && swap_count(count) != SWAP_MAP_BAD)	2383	swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
		2384	i % SWAPFILE_CLUSTER);
		2385	if (count == SWAP_MAP_BAD)
		2386	continue;
		2387	if (count \|\| swp_tb_is_folio(swp_tb))
2444	break;	2388	break;
2445	if ((i % LATENCY_LIMIT) == 0)	2389	if ((i % LATENCY_LIMIT) == 0)
2446	cond_resched();	2390	cond_resched();
@@ -3650,67 +3594,39 @@ void si_swapinfo(struct sysinfo *val) · 1203 unmodified lines
3650	* Returns error code in following case.	3594	* Returns error code in following case.
3651	* - success -> 0	3595	* - success -> 0
3652	* - swp_entry is invalid -> EINVAL	3596	* - swp_entry is invalid -> EINVAL
3653	* - swap-cache reference is requested but there is already one. -> EEXIST	3597	* - swap-mapped reference is requested but the entry is not used. -> ENOENT
3654	* - swap-cache reference is requested but the entry is not used. -> ENOENT
3655	* - swap-mapped reference requested but needs continued swap count. -> ENOMEM	3598	* - swap-mapped reference requested but needs continued swap count. -> ENOMEM
3656	*/	3599	*/
3657	static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr)	3600	static int swap_dup_entries(struct swap_info_struct *si,
		3601	struct swap_cluster_info *ci,
		3602	unsigned long offset,
		3603	unsigned char usage, int nr)
3658	{	3604	{
3659	struct swap_info_struct *si;	3605	int i;
3660	struct swap_cluster_info *ci;
3661	unsigned long offset;
3662	unsigned char count;	3606	unsigned char count;
3663	unsigned char has_cache;
3664	int err, i;
3665
3666	si = swap_entry_to_info(entry);
3667	if (WARN_ON_ONCE(!si)) {
3668	pr_err("%s%08lx\n", Bad_file, entry.val);
3669	return -EINVAL;
3670	}
3671
3672	offset = swp_offset(entry);
3673	VM_WARN_ON(nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER);
3674	VM_WARN_ON(usage == 1 && nr > 1);
3675	ci = swap_cluster_lock(si, offset);
3676		3607
3677	err = 0;
3678	for (i = 0; i < nr; i++) {	3608	for (i = 0; i < nr; i++) {
3679	count = si->swap_map[offset + i];	3609	count = si->swap_map[offset + i];
3680
3681	/*	3610	/*
3682	* swapin_readahead() doesn't check if a swap entry is valid, so the	3611	* For swapin out, allocator never allocates bad slots. for
3683	* swap entry could be SWAP_MAP_BAD. Check here with lock held.	3612	* swapin, readahead is guarded by swap_entry_swapped.
3684	*/	3613	*/
3685	if (unlikely(swap_count(count) == SWAP_MAP_BAD)) {	3614	if (WARN_ON(count == SWAP_MAP_BAD))
3686	err = -ENOENT;	3615	return -ENOENT;
3687	goto unlock_out;	3616	/*
3688	}	3617	* Swap count duplication must be guarded by either swap cache folio (from
3689		3618	* folio_dup_swap) or external lock of existing entry (from swap_dup_entry_direct).
3690	has_cache = count & SWAP_HAS_CACHE;	3619	*/
3691	count &= ~SWAP_HAS_CACHE;	3620	if (WARN_ON(!count &&
3692		3621	!swp_tb_is_folio(__swap_table_get(ci, offset % SWAPFILE_CLUSTER))))
3693	if (!count && !has_cache) {	3622	return -ENOENT;
3694	err = -ENOENT;	3623	if (WARN_ON((count & ~COUNT_CONTINUED) > SWAP_MAP_MAX))
3695	} else if (usage == SWAP_HAS_CACHE) {	3624	return -EINVAL;
3696	if (has_cache)
3697	err = -EEXIST;
3698	} else if ((count & ~COUNT_CONTINUED) > SWAP_MAP_MAX) {
3699	err = -EINVAL;
3700	}
3701
3702	if (err)
3703	goto unlock_out;
3704	}	3625	}
3705		3626
3706	for (i = 0; i < nr; i++) {	3627	for (i = 0; i < nr; i++) {
3707	count = si->swap_map[offset + i];	3628	count = si->swap_map[offset + i];
3708	has_cache = count & SWAP_HAS_CACHE;	3629	if ((count & ~COUNT_CONTINUED) < SWAP_MAP_MAX)
3709	count &= ~SWAP_HAS_CACHE;
3710
3711	if (usage == SWAP_HAS_CACHE)
3712	has_cache = SWAP_HAS_CACHE;
3713	else if ((count & ~COUNT_CONTINUED) < SWAP_MAP_MAX)
3714	count += usage;	3630	count += usage;
3715	else if (swap_count_continued(si, offset + i, count))	3631	else if (swap_count_continued(si, offset + i, count))
3716	count = COUNT_CONTINUED;	3632	count = COUNT_CONTINUED;
@@ -3719,65 +3635,55 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) · 2 unmodified lines
3719	* Don't need to rollback changes, because if	3635	* Don't need to rollback changes, because if
3720	* usage == 1, there must be nr == 1.	3636	* usage == 1, there must be nr == 1.
3721	*/	3637	*/
3722	err = -ENOMEM;	3638	return -ENOMEM;
3723	goto unlock_out;
3724	}	3639	}
3725		3640
3726	WRITE_ONCE(si->swap_map[offset + i], count \| has_cache);	3641	WRITE_ONCE(si->swap_map[offset + i], count);
3727	}	3642	}
3728		3643
3729	unlock_out:	3644	return 0;
3730	swap_cluster_unlock(ci);
3731	return err;
3732	}	3645	}
3733		3646
3734	/*	3647	static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr)
3735	* Help swapoff by noting that swap entry belongs to shmem/tmpfs
3736	* (in which case its reference count is never incremented).
3737	*/
3738	void swap_shmem_alloc(swp_entry_t entry, int nr)
3739	{	3648	{
3740	__swap_duplicate(entry, SWAP_MAP_SHMEM, nr);	3649	int err;
		3650	struct swap_info_struct *si;
		3651	struct swap_cluster_info *ci;
		3652	unsigned long offset = swp_offset(entry);
		3653
		3654	si = swap_entry_to_info(entry);
		3655	if (WARN_ON_ONCE(!si)) {
		3656	pr_err("%s%08lx\n", Bad_file, entry.val);
		3657	return -EINVAL;
		3658	}
		3659
		3660	VM_WARN_ON(nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER);
		3661	ci = swap_cluster_lock(si, offset);
		3662	err = swap_dup_entries(si, ci, offset, usage, nr);
		3663	swap_cluster_unlock(ci);
		3664	return err;
3741	}	3665	}
3742		3666
3743	/*	3667	/*
3744	* Increase reference count of swap entry by 1.	3668	* swap_dup_entry_direct() - Increase reference count of a swap entry by one.
		3669	* @entry: first swap entry from which we want to increase the refcount.
		3670	*
3745	* Returns 0 for success, or -ENOMEM if a swap_count_continuation is required	3671	* Returns 0 for success, or -ENOMEM if a swap_count_continuation is required
3746	* but could not be atomically allocated. Returns 0, just as if it succeeded,	3672	* but could not be atomically allocated. Returns 0, just as if it succeeded,
3747	* if __swap_duplicate() fails for another reason (-EINVAL or -ENOENT), which	3673	* if __swap_duplicate() fails for another reason (-EINVAL or -ENOENT), which
3748	* might occur if a page table entry has got corrupted.	3674	* might occur if a page table entry has got corrupted.
		3675	*
		3676	* Context: Caller must ensure there is no race condition on the reference
		3677	* owner. e.g., locking the PTL of a PTE containing the entry being increased.
3749	*/	3678	*/
3750	int swap_duplicate(swp_entry_t entry)	3679	int swap_dup_entry_direct(swp_entry_t entry)
3751	{	3680	{
3752	int err = 0;	3681	int err = 0;
3753
3754	while (!err && __swap_duplicate(entry, 1, 1) == -ENOMEM)	3682	while (!err && __swap_duplicate(entry, 1, 1) == -ENOMEM)
3755	err = add_swap_count_continuation(entry, GFP_ATOMIC);	3683	err = add_swap_count_continuation(entry, GFP_ATOMIC);
3756	return err;	3684	return err;
3757	}	3685	}
3758		3686
3759	/*
3760	* @entry: first swap entry from which we allocate nr swap cache.
3761	*
3762	* Called when allocating swap cache for existing swap entries,
3763	* This can return error codes. Returns 0 at success.
3764	* -EEXIST means there is a swap cache.
3765	* Note: return code is different from swap_duplicate().
3766	*/
3767	int swapcache_prepare(swp_entry_t entry, int nr)
3768	{
3769	return __swap_duplicate(entry, SWAP_HAS_CACHE, nr);
3770	}
3771
3772	/*
3773	* Caller should ensure entries belong to the same folio so
3774	* the entries won't span cross cluster boundary.
3775	*/
3776	void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr)
3777	{
3778	swap_entries_put_cache(si, entry, nr);
3779	}
3780
3781	/*	3687	/*
3782	* add_swap_count_continuation - called when a swap count is duplicated	3688	* add_swap_count_continuation - called when a swap count is duplicated
3783	* beyond SWAP_MAP_MAX, it allocates a new page and links that to the entry's	3689	* beyond SWAP_MAP_MAX, it allocates a new page and links that to the entry's
@@ -3823,7 +3729,7 @@ int add_swap_count_continuation(swp_entry_t entry, gfp_t gfp_mask) · 39 unmodified lines
3823		3729
3824	ci = swap_cluster_lock(si, offset);	3730	ci = swap_cluster_lock(si, offset);
3825		3731
3826	count = swap_count(si->swap_map[offset]);	3732	count = si->swap_map[offset];
3827		3733
3828	if ((count & ~COUNT_CONTINUED) != SWAP_MAP_MAX) {	3734	if ((count & ~COUNT_CONTINUED) != SWAP_MAP_MAX) {
3829	/*	3735	/*
@@ -3895,7 +3801,7 @@ outer: · 65 unmodified lines
3895	* into, carry if so, or else fail until a new continuation page is allocated;	3801	* into, carry if so, or else fail until a new continuation page is allocated;
3896	* when the original swap_map count is decremented from 0 with continuation,	3802	* when the original swap_map count is decremented from 0 with continuation,
3897	* borrow from the continuation and report whether it still holds more.	3803	* borrow from the continuation and report whether it still holds more.
3898	* Called while __swap_duplicate() or caller of swap_entry_put_locked()	3804	* Called while __swap_duplicate() or caller of swap_put_entry_locked()
3899	* holds cluster lock.	3805	* holds cluster lock.
3900	*/	3806	*/
3901	static bool swap_count_continued(struct swap_info_struct *si,	3807	static bool swap_count_continued(struct swap_info_struct *si,

mm/tests/lazy_mmu_mode_kunit.c+73−0

@@ -0,0 +1,73 @@
	1	// SPDX-License-Identifier: GPL-2.0-only
	2	#include <kunit/test.h>
	3	#include <linux/pgtable.h>
	4
	5	MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
	6
	7	static void expect_not_active(struct kunit *test)
	8	{
	9	KUNIT_EXPECT_FALSE(test, is_lazy_mmu_mode_active());
	10	}
	11
	12	static void expect_active(struct kunit *test)
	13	{
	14	KUNIT_EXPECT_TRUE(test, is_lazy_mmu_mode_active());
	15	}
	16
	17	static void lazy_mmu_mode_active(struct kunit *test)
	18	{
	19	expect_not_active(test);
	20
	21	lazy_mmu_mode_enable();
	22	expect_active(test);
	23
	24	{
	25	/* Nested section */
	26	lazy_mmu_mode_enable();
	27	expect_active(test);
	28
	29	lazy_mmu_mode_disable();
	30	expect_active(test);
	31	}
	32
	33	{
	34	/* Paused section */
	35	lazy_mmu_mode_pause();
	36	expect_not_active(test);
	37
	38	{
	39	/* No effect (paused) */
	40	lazy_mmu_mode_enable();
	41	expect_not_active(test);
	42
	43	lazy_mmu_mode_disable();
	44	expect_not_active(test);
	45
	46	lazy_mmu_mode_pause();
	47	expect_not_active(test);
	48
	49	lazy_mmu_mode_resume();
	50	expect_not_active(test);
	51	}
	52
	53	lazy_mmu_mode_resume();
	54	expect_active(test);
	55	}
	56
	57	lazy_mmu_mode_disable();
	58	expect_not_active(test);
	59	}
	60
	61	static struct kunit_case lazy_mmu_mode_test_cases[] = {
	62	KUNIT_CASE(lazy_mmu_mode_active),
	63	{}
	64	};
	65
	66	static struct kunit_suite lazy_mmu_mode_test_suite = {
	67	.name = "lazy_mmu_mode",
	68	.test_cases = lazy_mmu_mode_test_cases,
	69	};
	70	kunit_test_suite(lazy_mmu_mode_test_suite);
	71
	72	MODULE_DESCRIPTION("Tests for the lazy MMU mode");
	73	MODULE_LICENSE("GPL");

mm/userfaultfd.c+7−11

 	/* It's safe to drop the reference now as the page-table is holding one. */
 	folio_put(*first_src_folio);
 	*first_src_folio = NULL;
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 	while (true) {
 		orig_src_pte = ptep_get_and_clear(mm, src_addr, src_pte);
 			break;
 	}
-	arch_leave_lazy_mmu_mode();
+	lazy_mmu_mode_disable();
 	if (src_addr > src_start)
 		flush_tlb_range(src_vma, src_start, src_addr);
 		 * Check if the swap entry is cached after acquiring the src_pte
 		 * lock. Otherwise, we might miss a newly loaded swap cache folio.
 		 *
-		 * Check swap_map directly to minimize overhead, READ_ONCE is sufficient.
 		 * We are trying to catch newly added swap cache, the only possible case is
 		 * when a folio is swapped in and out again staying in swap cache, using the
 		 * same entry before the PTE check above. The PTL is acquired and released
-		 * twice, each time after updating the swap_map's flag. So holding
+		 * twice, each time after updating the swap table. So holding
-		 * the PTL here ensures we see the updated value. False positive is possible,
+		 * the PTL here ensures we see the updated value.
-		 * e.g. SWP_SYNCHRONOUS_IO swapin may set the flag without touching the
-		 * cache, or during the tiny synchronization window between swap cache and
-		 * swap_map, but it will be gone very quickly, worst result is retry jitters.
 		 */
-		if (READ_ONCE(si->swap_map[swp_offset(entry)]) & SWAP_HAS_CACHE) {
+		if (swap_cache_has_folio(entry)) {
 			double_pt_unlock(dst_ptl, src_ptl);
 			return -EAGAIN;
 		}
 	 * Use the maywrite version to indicate that dst_pte will be modified,
 	 * since dst_pte needs to be none, the subsequent pte_same() check
 	 * cannot prevent the dst_pte page from being freed concurrently, so we
-	 * also need to abtain dst_pmdval and recheck pmd_same() later.
+	 * also need to obtain dst_pmdval and recheck pmd_same() later.
 	 */
 	dst_pte = pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dst_pmdval,
 					   &dst_ptl);
 		goto out;
 	}
-	/* If PTE changed after we locked the folio them start over */
+	/* If PTE changed after we locked the folio then start over */
 	if (src_folio && unlikely(!pte_same(src_folio_pte, orig_src_pte))) {
 		ret = -EAGAIN;
 		goto out;

mm/vma.c+8−8

@@ -381,7 +381,7 @@ again: · 380 unmodified lines
381	fput(vp->file);	381	fput(vp->file);
382	}	382	}
383	if (vp->remove->anon_vma)	383	if (vp->remove->anon_vma)
384	anon_vma_merge(vp->vma, vp->remove);	384	unlink_anon_vmas(vp->remove);
385	mm->map_count--;	385	mm->map_count--;
386	mpol_put(vma_policy(vp->remove));	386	mpol_put(vma_policy(vp->remove));
387	if (!vp->remove2)	387	if (!vp->remove2)
@@ -530,7 +530,7 @@ __split_vma(struct vma_iterator vmi, struct vm_area_struct vma, · 142 unmodified lines
530	if (err)	530	if (err)
531	goto out_free_vmi;	531	goto out_free_vmi;
532		532
533	err = anon_vma_clone(new, vma);	533	err = anon_vma_clone(new, vma, VMA_OP_SPLIT);
534	if (err)	534	if (err)
535	goto out_free_mpol;	535	goto out_free_mpol;
536		536
@@ -628,7 +628,7 @@ static int dup_anon_vma(struct vm_area_struct *dst, · 91 unmodified lines
628		628
629	vma_assert_write_locked(dst);	629	vma_assert_write_locked(dst);
630	dst->anon_vma = src->anon_vma;	630	dst->anon_vma = src->anon_vma;
631	ret = anon_vma_clone(dst, src);	631	ret = anon_vma_clone(dst, src, VMA_OP_MERGE_UNFAULTED);
632	if (ret)	632	if (ret)
633	return ret;	633	return ret;
634		634
@@ -1901,7 +1901,7 @@ struct vm_area_struct copy_vma(struct vm_area_struct *vmap, · 1266 unmodified lines
1901	vma_set_range(new_vma, addr, addr + len, pgoff);	1901	vma_set_range(new_vma, addr, addr + len, pgoff);
1902	if (vma_dup_policy(vma, new_vma))	1902	if (vma_dup_policy(vma, new_vma))
1903	goto out_free_vma;	1903	goto out_free_vma;
1904	if (anon_vma_clone(new_vma, vma))	1904	if (anon_vma_clone(new_vma, vma, VMA_OP_REMAP))
1905	goto out_free_mempol;	1905	goto out_free_mempol;
1906	if (new_vma->vm_file)	1906	if (new_vma->vm_file)
1907	get_file(new_vma->vm_file);	1907	get_file(new_vma->vm_file);
@@ -2951,10 +2951,10 @@ retry: · 1043 unmodified lines
2951	return -ENOMEM;	2951	return -ENOMEM;
2952		2952
2953	/*	2953	/*
2954	* Adjust for the gap first so it doesn't interfere with the	2954	* Adjust for the gap first so it doesn't interfere with the later
2955	* later alignment. The first step is the minimum needed to	2955	* alignment. The first step is the minimum needed to fulfill the start
2956	* fulill the start gap, the next steps is the minimum to align	2956	* gap, the next step is the minimum to align that. It is the minimum
2957	* that. It is the minimum needed to fulill both.	2957	* needed to fulfill both.
2958	*/	2958	*/
2959	gap = vma_iter_addr(&vmi) + info->start_gap;	2959	gap = vma_iter_addr(&vmi) + info->start_gap;
2960	gap += (info->align_offset - gap) & info->align_mask;	2960	gap += (info->align_offset - gap) & info->align_mask;

mm/vma.h+4−10

 		struct vm_area_struct *prev, struct vm_area_struct *next);
 /**
- * vma_modify_flags() - Peform any necessary split/merge in preparation for
+ * vma_modify_flags() - Perform any necessary split/merge in preparation for
  * setting VMA flags to *@vm_flags in the range @start to @end contained within
  * @vma.
  * @vmi: Valid VMA iterator positioned at @vma.
 		vm_flags_t *vm_flags_ptr);
 /**
- * vma_modify_name() - Peform any necessary split/merge in preparation for
+ * vma_modify_name() - Perform any necessary split/merge in preparation for
  * setting anonymous VMA name to @new_name in the range @start to @end contained
  * within @vma.
  * @vmi: Valid VMA iterator positioned at @vma.
 		struct anon_vma_name *new_name);
 /**
- * vma_modify_policy() - Peform any necessary split/merge in preparation for
+ * vma_modify_policy() - Perform any necessary split/merge in preparation for
  * setting NUMA policy to @new_pol in the range @start to @end contained
  * within @vma.
  * @vmi: Valid VMA iterator positioned at @vma.
 		   struct mempolicy *new_pol);
 /**
- * vma_modify_flags_uffd() - Peform any necessary split/merge in preparation for
+ * vma_modify_flags_uffd() - Perform any necessary split/merge in preparation for
  * setting VMA flags to @vm_flags and UFFD context to @new_ctx in the range
  * @start to @end contained within @vma.
  * @vmi: Valid VMA iterator positioned at @vma.
 	return vmi->mas.last + 1;
 }
-static inline int vma_iter_bulk_alloc(struct vma_iterator *vmi,
-				      unsigned long count)
-{
-	return mas_expected_entries(&vmi->mas, count);
-}
 static inline
 struct vm_area_struct *vma_iter_prev_range(struct vma_iterator *vmi)
 {

mm/vmalloc.c+20−6

 	if (!pte)
 		return -ENOMEM;
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 	do {
 		if (unlikely(!pte_none(ptep_get(pte)))) {
 		pfn++;
 	} while (pte += PFN_DOWN(size), addr += size, addr != end);
-	arch_leave_lazy_mmu_mode();
+	lazy_mmu_mode_disable();
 	*mask |= PGTBL_PTE_MODIFIED;
 	return 0;
 }
 	int err;
 	pgtbl_mod_mask mask = 0;
+	/*
+	 * Might allocate pagetables (for most archs a more precise annotation
+	 * would be might_alloc(GFP_PGTABLE_KERNEL)). Also might shootdown TLB
+	 * (requires IRQs enabled on x86).
+	 */
 	might_sleep();
 	BUG_ON(addr >= end);
 	unsigned long size = PAGE_SIZE;
 	pte = pte_offset_kernel(pmd, addr);
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 	do {
 #ifdef CONFIG_HUGETLB_PAGE
 		WARN_ON(!pte_none(ptent) && !pte_present(ptent));
 	} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
-	arch_leave_lazy_mmu_mode();
+	lazy_mmu_mode_disable();
 	*mask |= PGTBL_PTE_MODIFIED;
 }
 	if (!pte)
 		return -ENOMEM;
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 	do {
 		struct page *page = pages[*nr];
 		(*nr)++;
 	} while (pte++, addr += PAGE_SIZE, addr != end);
-	arch_leave_lazy_mmu_mode();
+	lazy_mmu_mode_disable();
 	*mask |= PGTBL_PTE_MODIFIED;
 	return err;
 	reclaim_list_global(&decay_list);
 }
+#define KASAN_RELEASE_BATCH_SIZE 32
 static void
 kasan_release_vmalloc_node(struct vmap_node *vn)
 {
 	struct vmap_area *va;
 	unsigned long start, end;
+	unsigned int batch_count = 0;
 	start = list_first_entry(&vn->purge_list, struct vmap_area, list)->va_start;
 	end = list_last_entry(&vn->purge_list, struct vmap_area, list)->va_end;
 			kasan_release_vmalloc(va->va_start, va->va_end,
 				va->va_start, va->va_end,
 				KASAN_VMALLOC_PAGE_RANGE);
+		if (need_resched() || (++batch_count >= KASAN_RELEASE_BATCH_SIZE)) {
+			cond_resched();
+			batch_count = 0;
+		}
 	}
 	kasan_release_vmalloc(start, end, start, end, KASAN_VMALLOC_TLB_FLUSH);
 	return n;
 }
+EXPORT_SYMBOL(vrealloc_node_align_noprof);
 #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32)
 #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL)

mm/vmscan.c+113−108

 #include <asm/div64.h>
 #include <linux/swapops.h>
-#include <linux/balloon_compaction.h>
 #include <linux/sched/sysctl.h>
 #include "internal.h"
 	unsigned int force_deactivate:1;
 	unsigned int skipped_deactivate:1;
-	/* Writepage batching in laptop mode; RECLAIM_WRITE */
+	/* zone_reclaim_mode, boost reclaim */
 	unsigned int may_writepage:1;
-	/* Can mapped folios be reclaimed? */
+	/* zone_reclaim_mode */
 	unsigned int may_unmap:1;
-	/* Can folios be swapped as part of reclaim? */
+	/* zome_reclaim_mode, boost reclaim, cgroup restrictions */
 	unsigned int may_swap:1;
 	/* Not allow cache_trim_mode to be turned on as part of reclaim? */
 	 * If kswapd is disabled, reschedule if necessary but do not
 	 * throttle as the system is likely near OOM.
 	 */
-	if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES)
+	if (kswapd_test_hopeless(pgdat))
 		return true;
 	/*
 		if (reclaimed && !mapping_exiting(mapping))
 			shadow = workingset_eviction(folio, target_memcg);
-		__swap_cache_del_folio(ci, folio, swap, shadow);
 		memcg1_swapout(folio, swap);
+		__swap_cache_del_folio(ci, folio, swap, shadow);
 		swap_cluster_unlock_irq(ci);
-		put_swap_folio(folio, swap);
 	} else {
 		void (*free_folio)(struct folio *);
 	/*
 	 * We can "enter_fs" for swap-cache with only __GFP_IO
 	 * providing this isn't SWP_FS_OPS.
-	 * ->flags can be updated non-atomicially (scan_swap_map_slots),
+	 * ->flags can be updated non-atomically (scan_swap_map_slots),
 	 * but that will never affect SWP_FS_OPS, so the data_race
 	 * is safe.
 	 */
 		 * Try to allocate it some swap space here.
 		 * Lazyfree folio could be freed directly
 		 */
-		if (folio_test_anon(folio) && folio_test_swapbacked(folio)) {
+		if (folio_test_anon(folio) && folio_test_swapbacked(folio) &&
-			if (!folio_test_swapcache(folio)) {
+				!folio_test_swapcache(folio)) {
-				if (!(sc->gfp_mask & __GFP_IO))
+			if (!(sc->gfp_mask & __GFP_IO))
-					goto keep_locked;
+				goto keep_locked;
-				if (folio_maybe_dma_pinned(folio))
+			if (folio_maybe_dma_pinned(folio))
-					goto keep_locked;
+				goto keep_locked;
-				if (folio_test_large(folio)) {
+			if (folio_test_large(folio)) {
-					/* cannot split folio, skip it */
+				/* cannot split folio, skip it */
-					if (folio_expected_ref_count(folio) !=
+				if (folio_expected_ref_count(folio) !=
-					    folio_ref_count(folio) - 1)
+				    folio_ref_count(folio) - 1)
-						goto activate_locked;
+					goto activate_locked;
-					/*
-					 * Split partially mapped folios right away.
-					 * We can free the unmapped pages without IO.
-					 */
-					if (data_race(!list_empty(&folio->_deferred_list) &&
-					    folio_test_partially_mapped(folio)) &&
-					    split_folio_to_list(folio, folio_list))
-						goto activate_locked;
-				}
-				if (folio_alloc_swap(folio)) {
-					int __maybe_unused order = folio_order(folio);
-					if (!folio_test_large(folio))
-						goto activate_locked_split;
-					/* Fallback to swap normal pages */
-					if (split_folio_to_list(folio, folio_list))
-						goto activate_locked;
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-					if (nr_pages >= HPAGE_PMD_NR) {
-						count_memcg_folio_events(folio,
-							THP_SWPOUT_FALLBACK, 1);
-						count_vm_event(THP_SWPOUT_FALLBACK);
-					}
-#endif
-					count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK);
-					if (folio_alloc_swap(folio))
-						goto activate_locked_split;
-				}
 				/*
-				 * Normally the folio will be dirtied in unmap because its
+				 * Split partially mapped folios right away.
-				 * pte should be dirty. A special case is MADV_FREE page. The
+				 * We can free the unmapped pages without IO.
-				 * page's pte could have dirty bit cleared but the folio's
-				 * SwapBacked flag is still set because clearing the dirty bit
-				 * and SwapBacked flag has no lock protected. For such folio,
-				 * unmap will not set dirty bit for it, so folio reclaim will
-				 * not write the folio out. This can cause data corruption when
-				 * the folio is swapped in later. Always setting the dirty flag
-				 * for the folio solves the problem.
 				 */
-				folio_mark_dirty(folio);
+				if (data_race(!list_empty(&folio->_deferred_list) &&
+				    folio_test_partially_mapped(folio)) &&
+				    split_folio_to_list(folio, folio_list))
+					goto activate_locked;
 			}
+			if (folio_alloc_swap(folio)) {
+				int __maybe_unused order = folio_order(folio);
+				if (!folio_test_large(folio))
+					goto activate_locked_split;
+				/* Fallback to swap normal pages */
+				if (split_folio_to_list(folio, folio_list))
+					goto activate_locked;
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+				if (nr_pages >= HPAGE_PMD_NR) {
+					count_memcg_folio_events(folio,
+						THP_SWPOUT_FALLBACK, 1);
+					count_vm_event(THP_SWPOUT_FALLBACK);
+				}
+#endif
+				count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK);
+				if (folio_alloc_swap(folio))
+					goto activate_locked_split;
+			}
+			/*
+			 * Normally the folio will be dirtied in unmap because
+			 * its pte should be dirty. A special case is MADV_FREE
+			 * page. The page's pte could have dirty bit cleared but
+			 * the folio's SwapBacked flag is still set because
+			 * clearing the dirty bit and SwapBacked flag has no
+			 * lock protected. For such folio, unmap will not set
+			 * dirty bit for it, so folio reclaim will not write the
+			 * folio out. This can cause data corruption when the
+			 * folio is swapped in later. Always setting the dirty
+			 * flag for the folio solves the problem.
+			 */
+			folio_mark_dirty(folio);
 		}
 		/*
 static unsigned long apply_proportional_protection(struct mem_cgroup *memcg,
 		struct scan_control *sc, unsigned long scan)
 {
-	unsigned long min, low;
+	unsigned long min, low, usage;
-	mem_cgroup_protection(sc->target_mem_cgroup, memcg, &min, &low);
+	mem_cgroup_protection(sc->target_mem_cgroup, memcg, &min, &low, &usage);
 	if (min || low) {
 		/*
 		 * again by how much of the total memory used is under
 		 * hard protection.
 		 */
-		unsigned long cgroup_size = mem_cgroup_size(memcg);
 		unsigned long protection;
 		/* memory.low scaling, make sure we retry before OOM */
 		}
 		/* Avoid TOCTOU with earlier protection check */
-		cgroup_size = max(cgroup_size, protection);
+		usage = max(usage, protection);
-		scan -= scan * protection / (cgroup_size + 1);
+		scan -= scan * protection / (usage + 1);
 		/*
 		 * Minimally target SWAP_CLUSTER_MAX pages to keep
 		return false;
 	}
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 restart:
 	for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) {
 		unsigned long pfn;
 	if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end))
 		goto restart;
-	arch_leave_lazy_mmu_mode();
+	lazy_mmu_mode_disable();
 	pte_unmap_unlock(pte, ptl);
 	return suitable_to_scan(total, young);
 	if (!spin_trylock(ptl))
 		goto done;
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 	do {
 		unsigned long pfn;
 	walk_update_folio(walk, last, gen, dirty);
-	arch_leave_lazy_mmu_mode();
+	lazy_mmu_mode_disable();
 	spin_unlock(ptl);
 done:
 	*first = -1;
 		}
 	}
-	arch_enter_lazy_mmu_mode();
+	lazy_mmu_mode_enable();
 	pte -= (addr - start) / PAGE_SIZE;
 	walk_update_folio(walk, last, gen, dirty);
-	arch_leave_lazy_mmu_mode();
+	lazy_mmu_mode_disable();
 	/* feedback from rmap walkers to page table walkers */
 	if (mm_state && suitable_to_scan(i, young))
 	blk_finish_plug(&plug);
 done:
 	if (sc->nr_reclaimed > reclaimed)
-		atomic_set(&pgdat->kswapd_failures, 0);
+		kswapd_try_clear_hopeless(pgdat, sc->order, sc->reclaim_idx);
 }
 /******************************************************************************
 		if (memcg)
 			cgroup_path(memcg->css.cgroup, m->private, PATH_MAX);
 #endif
-		seq_printf(m, "memcg %5hu %s\n", mem_cgroup_id(memcg), path);
+		seq_printf(m, "memcg %llu %s\n", mem_cgroup_id(memcg), path);
 	}
 	seq_printf(m, " node %5d\n", nid);
 	return -EINTR;
 }
-static int run_cmd(char cmd, int memcg_id, int nid, unsigned long seq,
+static int run_cmd(char cmd, u64 memcg_id, int nid, unsigned long seq,
 		   struct scan_control *sc, int swappiness, unsigned long opt)
 {
 	struct lruvec *lruvec;
 		return -EINVAL;
 	if (!mem_cgroup_disabled()) {
-		rcu_read_lock();
+		memcg = mem_cgroup_get_from_id(memcg_id);
-		memcg = mem_cgroup_from_id(memcg_id);
-		if (!mem_cgroup_tryget(memcg))
-			memcg = NULL;
-		rcu_read_unlock();
 		if (!memcg)
 			return -EINVAL;
 	}
 		int n;
 		int end;
 		char cmd, swap_string[5];
-		unsigned int memcg_id;
+		u64 memcg_id;
 		unsigned int nid;
 		unsigned long seq;
 		unsigned int swappiness;
 		if (!*cur)
 			continue;
-		n = sscanf(cur, "%c %u %u %lu %n %4s %n %lu %n", &cmd, &memcg_id, &nid,
+		n = sscanf(cur, "%c %llu %u %lu %n %4s %n %lu %n", &cmd, &memcg_id, &nid,
 			   &seq, &end, swap_string, &end, &opt, &end);
 		if (n < 4 || cur[end]) {
 			err = -EINVAL;
 	 * successful direct reclaim run will revive a dormant kswapd.
 	 */
 	if (reclaimable)
-		atomic_set(&pgdat->kswapd_failures, 0);
+		kswapd_try_clear_hopeless(pgdat, sc->order, sc->reclaim_idx);
 	else if (sc->cache_trim_mode)
 		sc->cache_trim_mode_failed = 1;
 }
 		if (sc->compaction_ready)
 			break;
-		/*
-		 * If we're getting trouble reclaiming, start doing
-		 * writepage even in laptop mode.
-		 */
-		if (sc->priority < DEF_PRIORITY - 2)
-			sc->may_writepage = 1;
 	} while (--sc->priority >= 0);
 	last_pgdat = NULL;
 	int i;
 	bool wmark_ok;
-	if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES)
+	if (kswapd_test_hopeless(pgdat))
 		return true;
 	for_each_managed_zone_pgdat(zone, pgdat, i, ZONE_NORMAL) {
 		.order = order,
 		.nodemask = nodemask,
 		.priority = DEF_PRIORITY,
-		.may_writepage = !laptop_mode,
+		.may_writepage = 1,
 		.may_unmap = 1,
 		.may_swap = 1,
 	};
 	struct scan_control sc = {
 		.nr_to_reclaim = SWAP_CLUSTER_MAX,
 		.target_mem_cgroup = memcg,
-		.may_writepage = !laptop_mode,
+		.may_writepage = 1,
 		.may_unmap = 1,
 		.reclaim_idx = MAX_NR_ZONES - 1,
 		.may_swap = !noswap,
 		.reclaim_idx = MAX_NR_ZONES - 1,
 		.target_mem_cgroup = memcg,
 		.priority = DEF_PRIORITY,
-		.may_writepage = !laptop_mode,
+		.may_writepage = 1,
 		.may_unmap = 1,
 		.may_swap = !!(reclaim_options & MEMCG_RECLAIM_MAY_SWAP),
 		.proactive = !!(reclaim_options & MEMCG_RECLAIM_PROACTIVE),
 		wake_up_all(&pgdat->pfmemalloc_wait);
 	/* Hopeless node, leave it to direct reclaim */
-	if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES)
+	if (kswapd_test_hopeless(pgdat))

102 more lines aren’t shown here.

mm/vmstat.c+8−22

This diff is large: its 75 lines aren’t shown here.

mm/workingset.c+4−4

This diff is large: its 36 lines aren’t shown here.

mm/zsmalloc.c+125−67

This diff is large: its 357 lines aren’t shown here.

mm/zswap.c+23−35

This diff is large: its 117 lines aren’t shown here.

tools/mm/slabinfo.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/mm/thp_swap_allocator_test.c+1−1

This diff is large: its 9 lines aren’t shown here.

tools/testing/selftests/damon/access_memory.c+21−8

This diff is large: its 55 lines aren’t shown here.

tools/testing/selftests/damon/sysfs_memcg_path_leak.sh+14−12

This diff is large: its 44 lines aren’t shown here.

tools/testing/selftests/damon/sysfs_update_schemes_tried_regions_wss_estimation.py+32−9

This diff is large: its 72 lines aren’t shown here.

tools/testing/selftests/mm/.gitignore+0−1

This diff is large: its 8 lines aren’t shown here.

tools/testing/selftests/mm/Makefile+34−10

This diff is large: its 85 lines aren’t shown here.

tools/testing/selftests/mm/charge_reserved_hugetlb.sh+32−23

This diff is large: its 98 lines aren’t shown here.

tools/testing/selftests/mm/check_config.sh+1−2

This diff is large: its 10 lines aren’t shown here.

tools/testing/selftests/mm/cow.c+33−10

This diff is large: its 85 lines aren’t shown here.

tools/testing/selftests/mm/hugetlb-madvise.c+1−8

This diff is large: its 16 lines aren’t shown here.

tools/testing/selftests/mm/ksft_compaction.sh+4−0