Pull MM updates from Andrew Morton: - "powerpc/64s: do not re-activate batched TLB flush" makes arch_{enter|leave}_lazy_mmu_mode() nest properly (Alexander Gordeev) It adds a generic enter/leave layer and switches architectures to use it. Various hacks were removed in the process. - "zram: introduce compressed data writeback" implements data compression for zram writeback (Richard Chang and Sergey Senozhatsky) - "mm: folio_zero_user: clear page ranges" adds clearing of contiguous page ranges for hugepages. Large improvements during demand faulting are demonstrated (David Hildenbrand) - "memcg cleanups" tidies up some memcg code (Chen Ridong) - "mm/damon: introduce {,max_}nr_snapshots and tracepoint for damos stats" improves DAMOS stat's provided information, deterministic control, and readability (SeongJae Park) - "selftests/mm: hugetlb cgroup charging: robustness fixes" fixes a few issues in the hugetlb cgroup charging selftests (Li Wang) - "Fix va_high_addr_switch.sh test failure - again" addresses several issues in the va_high_addr_switch test (Chunyu Hu) - "mm/damon/tests/core-kunit: extend existing test scenarios" improves the KUnit test coverage for DAMON (Shu Anzai) - "mm/khugepaged: fix dirty page handling for MADV_COLLAPSE" fixes a glitch in khugepaged which was causing madvise(MADV_COLLAPSE) to transiently return -EAGAIN (Shivank Garg) - "arch, mm: consolidate hugetlb early reservation" reworks and consolidates a pile of straggly code related to reservation of hugetlb memory from bootmem and creation of CMA areas for hugetlb (Mike Rapoport) - "mm: clean up anon_vma implementation" cleans up the anon_vma implementation in various ways (Lorenzo Stoakes) - "tweaks for __alloc_pages_slowpath()" does a little streamlining of the page allocator's slowpath code (Vlastimil Babka) - "memcg: separate private and public ID namespaces" cleans up the memcg ID code and prevents the internal-only private IDs from being exposed to userspace (Shakeel Butt) - "mm: hugetlb: allocate frozen gigantic folio" cleans up the allocation of frozen folios and avoids some atomic refcount operations (Kefeng Wang) - "mm/damon: advance DAMOS-based LRU sorting" improves DAMOS's movement of memory betewwn the active and inactive LRUs and adds auto-tuning of the ratio-based quotas and of monitoring intervals (SeongJae Park) - "Support page table check on PowerPC" makes CONFIG_PAGE_TABLE_CHECK_ENFORCED work on powerpc (Andrew Donnellan) - "nodemask: align nodes_and{,not} with underlying bitmap ops" makes nodes_and() and nodes_andnot() propagate the return values from the underlying bit operations, enabling some cleanup in calling code (Yury Norov) - "mm/damon: hide kdamond and kdamond_lock from API callers" cleans up some DAMON internal interfaces (SeongJae Park) - "mm/khugepaged: cleanups and scan limit fix" does some cleanup work in khupaged and fixes a scan limit accounting issue (Shivank Garg) - "mm: balloon infrastructure cleanups" goes to town on the balloon infrastructure and its page migration function. Mainly cleanups, also some locking simplification (David Hildenbrand) - "mm/vmscan: add tracepoint and reason for kswapd_failures reset" adds additional tracepoints to the page reclaim code (Jiayuan Chen) - "Replace wq users and add WQ_PERCPU to alloc_workqueue() users" is part of Marco's kernel-wide migration from the legacy workqueue APIs over to the preferred unbound workqueues (Marco Crivellari) - "Various mm kselftests improvements/fixes" provides various unrelated improvements/fixes for the mm kselftests (Kevin Brodsky) - "mm: accelerate gigantic folio allocation" greatly speeds up gigantic folio allocation, mainly by avoiding unnecessary work in pfn_range_valid_contig() (Kefeng Wang) - "selftests/damon: improve leak detection and wss estimation reliability" improves the reliability of two of the DAMON selftests (SeongJae Park) - "mm/damon: cleanup kdamond, damon_call(), damos filter and DAMON_MIN_REGION" does some cleanup work in the core DAMON code (SeongJae Park) - "Docs/mm/damon: update intro, modules, maintainer profile, and misc" performs maintenance work on the DAMON documentation (SeongJae Park) - "mm: add and use vma_assert_stabilised() helper" refactors and cleans up the core VMA code. The main aim here is to be able to use the mmap write lock's lockdep state to perform various assertions regarding the locking which the VMA code requires (Lorenzo Stoakes) - "mm, swap: swap table phase II: unify swapin use" removes some old swap code (swap cache bypassing and swap synchronization) which wasn't working very well. Various other cleanups and simplifications were made. The end result is a 20% speedup in one benchmark (Kairui Song) - "enable PT_RECLAIM on more 64-bit architectures" makes PT_RECLAIM available on 64-bit alpha, loongarch, mips, parisc, and um. Various cleanups were performed along the way (Qi Zheng) * tag 'mm-stable-2026-02-11-19-22' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (325 commits) mm/memory: handle non-split locks correctly in zap_empty_pte_table() mm: move pte table reclaim code to memory.c mm: make PT_RECLAIM depends on MMU_GATHER_RCU_TABLE_FREE mm: convert __HAVE_ARCH_TLB_REMOVE_TABLE to CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE config um: mm: enable MMU_GATHER_RCU_TABLE_FREE parisc: mm: enable MMU_GATHER_RCU_TABLE_FREE mips: mm: enable MMU_GATHER_RCU_TABLE_FREE LoongArch: mm: enable MMU_GATHER_RCU_TABLE_FREE alpha: mm: enable MMU_GATHER_RCU_TABLE_FREE mm: change mm/pt_reclaim.c to use asm/tlb.h instead of asm-generic/tlb.h mm/damon/stat: remove __read_mostly from memory_idle_ms_percentiles zsmalloc: make common caches global mm: add SPDX id lines to some mm source files mm/zswap: use %pe to print error pointers mm/vmscan: use %pe to print error pointers mm/readahead: fix typo in comment mm: khugepaged: fix NR_FILE_PAGES and NR_SHMEM in collapse_file() mm: refactor vma_map_pages to use vm_insert_pages mm/damon: unify address range representation with damon_addr_range mm/cma: replace snprintf with strscpy in cma_new_area ...
Unified · Split
| @@ -150,3 +150,17 @@ Contact: Sergey Senozhatsky <senozhatsky@chromium.org> · 149 unmodified lines | ||
| 150 | 150 | Description: |
| 151 | 151 | The algorithm_params file is write-only and is used to setup |
| 152 | 152 | compression algorithm parameters. |
| 153 | + | |
| 154 | +What: /sys/block/zram<id>/writeback_compressed | |
| 155 | +Date: Decemeber 2025 | |
| 156 | +Contact: Richard Chang <richardycc@google.com> | |
| 157 | +Description: | |
| 158 | + The writeback_compressed device atrribute toggles compressed | |
| 159 | + writeback feature. | |
| 160 | + | |
| 161 | +What: /sys/block/zram<id>/writeback_batch_size | |
| 162 | +Date: November 2025 | |
| 163 | +Contact: Sergey Senozhatsky <senozhatsky@chromium.org> | |
| 164 | +Description: | |
| 165 | + The writeback_batch_size device atrribute sets the maximum | |
| 166 | + number of in-flight writeback operations. | |
| @@ -516,6 +516,19 @@ Contact: SeongJae Park <sj@kernel.org> · 515 unmodified lines | ||
| 516 | 516 | Description: Reading this file returns the number of the exceed events of |
| 517 | 517 | the scheme's quotas. |
| 518 | 518 | |
| 519 | +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/nr_snapshots | |
| 520 | +Date: Dec 2025 | |
| 521 | +Contact: SeongJae Park <sj@kernel.org> | |
| 522 | +Description: Reading this file returns the total number of DAMON snapshots | |
| 523 | + that the scheme has tried to be applied. | |
| 524 | + | |
| 525 | +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/stats/max_nr_snapshots | |
| 526 | +Date: Dec 2025 | |
| 527 | +Contact: SeongJae Park <sj@kernel.org> | |
| 528 | +Description: Writing a number to this file sets the upper limit of | |
| 529 | + nr_snapshots that deactivates the scheme when the limit is | |
| 530 | + reached or exceeded. | |
| 531 | + | |
| 519 | 532 | What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/tried_regions/total_bytes |
| 520 | 533 | Date: Jul 2023 |
| 521 | 534 | Contact: SeongJae Park <sj@kernel.org> |
| @@ -214,6 +214,9 @@ mem_limit WO specifies the maximum amount of memory ZRAM can · 213 unmodified lines | ||
| 214 | 214 | writeback_limit WO specifies the maximum amount of write IO zram |
| 215 | 215 | can write out to backing device as 4KB unit |
| 216 | 216 | writeback_limit_enable RW show and set writeback_limit feature |
| 217 | +writeback_batch_size RW show and set maximum number of in-flight | |
| 218 | + writeback operations | |
| 219 | +writeback_compressed RW show and set compressed writeback feature | |
| 217 | 220 | comp_algorithm RW show and change the compression algorithm |
| 218 | 221 | algorithm_params WO setup compression algorithm parameters |
| 219 | 222 | compact WO trigger memory compaction |
| @@ -222,7 +225,6 @@ backing_dev RW set up backend storage for zram to write out · 2 unmodified lines | ||
| 222 | 225 | idle WO mark allocated slot as idle |
| 223 | 226 | ====================== ====== =============================================== |
| 224 | 227 | |
| 225 | − | |
| 226 | 228 | User space is advised to use the following files to read the device statistics. |
| 227 | 229 | |
| 228 | 230 | File /sys/block/zram<id>/stat |
| @@ -434,6 +436,26 @@ system reboot, echo 1 > /sys/block/zramX/reset) so keeping how many of · 205 unmodified lines | ||
| 434 | 436 | writeback happened until you reset the zram to allocate extra writeback |
| 435 | 437 | budget in next setting is user's job. |
| 436 | 438 | |
| 439 | +By default zram stores written back pages in decompressed (raw) form, which | |
| 440 | +means that writeback operation involves decompression of the page before | |
| 441 | +writing it to the backing device. This behavior can be changed by enabling | |
| 442 | +`writeback_compressed` feature, which causes zram to write compressed pages | |
| 443 | +to the backing device, thus avoiding decompression overhead. To enable | |
| 444 | +this feature, execute:: | |
| 445 | + | |
| 446 | + $ echo yes > /sys/block/zramX/writeback_compressed | |
| 447 | + | |
| 448 | +Note that this feature should be configured before the `zramX` device is | |
| 449 | +initialized. | |
| 450 | + | |
| 451 | +Depending on backing device storage type, writeback operation may benefit | |
| 452 | +from a higher number of in-flight write requests (batched writes). The | |
| 453 | +number of maximum in-flight writeback operations can be configured via | |
| 454 | +`writeback_batch_size` attribute. To change the default value (which is 32), | |
| 455 | +execute:: | |
| 456 | + | |
| 457 | + $ echo 64 > /sys/block/zramX/writeback_batch_size | |
| 458 | + | |
| 437 | 459 | If admin wants to measure writeback count in a certain period, they could |
| 438 | 460 | know it via /sys/block/zram0/bd_stat's 3rd column. |
| 439 | 461 | |
| @@ -311,9 +311,8 @@ Lock order is as follows:: · 310 unmodified lines | ||
| 311 | 311 | |
| 312 | 312 | folio_lock |
| 313 | 313 | mm->page_table_lock or split pte_lock |
| 314 | − folio_memcg_lock (memcg->move_lock) | |
| 315 | − mapping->i_pages lock | |
| 316 | − lruvec->lru_lock. | |
| 314 | + mapping->i_pages lock | |
| 315 | + lruvec->lru_lock. | |
| 317 | 316 | |
| 318 | 317 | Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by |
| 319 | 318 | lruvec->lru_lock; the folio LRU flag is cleared before |
| @@ -10,7 +10,6 @@ Laptop Drivers · 9 unmodified lines | ||
| 10 | 10 | alienware-wmi |
| 11 | 11 | asus-laptop |
| 12 | 12 | disk-shock-protection |
| 13 | − laptop-mode | |
| 14 | 13 | lg-laptop |
| 15 | 14 | samsung-galaxybook |
| 16 | 15 | sony-laptop |
| @@ -1,770 +0,0 @@ | ||
| 1 | −=============================================== | |
| 2 | −How to conserve battery power using laptop-mode | |
| 3 | −=============================================== | |
| 4 | − | |
| 5 | −Document Author: Bart Samwel (bart@samwel.tk) | |
| 6 | − | |
| 7 | −Date created: January 2, 2004 | |
| 8 | − | |
| 9 | −Last modified: December 06, 2004 | |
| 10 | − | |
| 11 | −Introduction | |
| 12 | −------------ | |
| 13 | − | |
| 14 | −Laptop mode is used to minimize the time that the hard disk needs to be spun up, | |
| 15 | −to conserve battery power on laptops. It has been reported to cause significant | |
| 16 | −power savings. | |
| 17 | − | |
| 18 | −.. Contents | |
| 19 | − | |
| 20 | − * Introduction | |
| 21 | − * Installation | |
| 22 | − * Caveats | |
| 23 | − * The Details | |
| 24 | − * Tips & Tricks | |
| 25 | − * Control script | |
| 26 | − * ACPI integration | |
| 27 | − * Monitoring tool | |
| 28 | − | |
| 29 | − | |
| 30 | −Installation | |
| 31 | −------------ | |
| 32 | − | |
| 33 | −To use laptop mode, you don't need to set any kernel configuration options | |
| 34 | −or anything. Simply install all the files included in this document, and | |
| 35 | −laptop mode will automatically be started when you're on battery. For | |
| 36 | −your convenience, a tarball containing an installer can be downloaded at: | |
| 37 | − | |
| 38 | − http://www.samwel.tk/laptop_mode/laptop_mode/ | |
| 39 | − | |
| 40 | −To configure laptop mode, you need to edit the configuration file, which is | |
| 41 | −located in /etc/default/laptop-mode on Debian-based systems, or in | |
| 42 | −/etc/sysconfig/laptop-mode on other systems. | |
| 43 | − | |
| 44 | −Unfortunately, automatic enabling of laptop mode does not work for | |
| 45 | −laptops that don't have ACPI. On those laptops, you need to start laptop | |
| 46 | −mode manually. To start laptop mode, run "laptop_mode start", and to | |
| 47 | −stop it, run "laptop_mode stop". (Note: The laptop mode tools package now | |
| 48 | −has experimental support for APM, you might want to try that first.) | |
| 49 | − | |
| 50 | − | |
| 51 | −Caveats | |
| 52 | −------- | |
| 53 | − | |
| 54 | −* The downside of laptop mode is that you have a chance of losing up to 10 | |
| 55 | − minutes of work. If you cannot afford this, don't use it! The supplied ACPI | |
| 56 | − scripts automatically turn off laptop mode when the battery almost runs out, | |
| 57 | − so that you won't lose any data at the end of your battery life. | |
| 58 | − | |
| 59 | −* Most desktop hard drives have a very limited lifetime measured in spindown | |
| 60 | − cycles, typically about 50.000 times (it's usually listed on the spec sheet). | |
| 61 | − Check your drive's rating, and don't wear down your drive's lifetime if you | |
| 62 | − don't need to. | |
| 63 | − | |
| 64 | −* If you mount some of your ext3 filesystems with the -n option, then | |
| 65 | − the control script will not be able to remount them correctly. You must set | |
| 66 | − DO_REMOUNTS=0 in the control script, otherwise it will remount them with the | |
| 67 | − wrong options -- or it will fail because it cannot write to /etc/mtab. | |
| 68 | − | |
| 69 | −* If you have your filesystems listed as type "auto" in fstab, like I did, then | |
| 70 | − the control script will not recognize them as filesystems that need remounting. | |
| 71 | − You must list the filesystems with their true type instead. | |
| 72 | − | |
| 73 | −* It has been reported that some versions of the mutt mail client use file access | |
| 74 | − times to determine whether a folder contains new mail. If you use mutt and | |
| 75 | − experience this, you must disable the noatime remounting by setting the option | |
| 76 | − DO_REMOUNT_NOATIME to 0 in the configuration file. | |
| 77 | − | |
| 78 | − | |
| 79 | −The Details | |
| 80 | −----------- | |
| 81 | − | |
| 82 | −Laptop mode is controlled by the knob /proc/sys/vm/laptop_mode. This knob is | |
| 83 | −present for all kernels that have the laptop mode patch, regardless of any | |
| 84 | −configuration options. When the knob is set, any physical disk I/O (that might | |
| 85 | −have caused the hard disk to spin up) causes Linux to flush all dirty blocks. The | |
| 86 | −result of this is that after a disk has spun down, it will not be spun up | |
| 87 | −anymore to write dirty blocks, because those blocks had already been written | |
| 88 | −immediately after the most recent read operation. The value of the laptop_mode | |
| 89 | −knob determines the time between the occurrence of disk I/O and when the flush | |
| 90 | −is triggered. A sensible value for the knob is 5 seconds. Setting the knob to | |
| 91 | −0 disables laptop mode. | |
| 92 | − | |
| 93 | −To increase the effectiveness of the laptop_mode strategy, the laptop_mode | |
| 94 | −control script increases dirty_expire_centisecs and dirty_writeback_centisecs in | |
| 95 | −/proc/sys/vm to about 10 minutes (by default), which means that pages that are | |
| 96 | −dirtied are not forced to be written to disk as often. The control script also | |
| 97 | −changes the dirty background ratio, so that background writeback of dirty pages | |
| 98 | −is not done anymore. Combined with a higher commit value (also 10 minutes) for | |
| 99 | −ext3 filesystem (also done automatically by the control script), | |
| 100 | −this results in concentration of disk activity in a small time interval which | |
| 101 | −occurs only once every 10 minutes, or whenever the disk is forced to spin up by | |
| 102 | −a cache miss. The disk can then be spun down in the periods of inactivity. | |
| 103 | − | |
| 104 | − | |
| 105 | −Configuration | |
| 106 | −------------- | |
| 107 | − | |
| 108 | −The laptop mode configuration file is located in /etc/default/laptop-mode on | |
| 109 | −Debian-based systems, or in /etc/sysconfig/laptop-mode on other systems. It | |
| 110 | −contains the following options: | |
| 111 | − | |
| 112 | −MAX_AGE: | |
| 113 | − | |
| 114 | −Maximum time, in seconds, of hard drive spindown time that you are | |
| 115 | −comfortable with. Worst case, it's possible that you could lose this | |
| 116 | −amount of work if your battery fails while you're in laptop mode. | |
| 117 | − | |
| 118 | −MINIMUM_BATTERY_MINUTES: | |
| 119 | − | |
| 120 | −Automatically disable laptop mode if the remaining number of minutes of | |
| 121 | −battery power is less than this value. Default is 10 minutes. | |
| 122 | − | |
| 123 | −AC_HD/BATT_HD: | |
| 124 | − | |
| 125 | −The idle timeout that should be set on your hard drive when laptop mode | |
| 126 | −is active (BATT_HD) and when it is not active (AC_HD). The defaults are | |
| 127 | −20 seconds (value 4) for BATT_HD and 2 hours (value 244) for AC_HD. The | |
| 128 | −possible values are those listed in the manual page for "hdparm" for the | |
| 129 | −"-S" option. | |
| 130 | − | |
| 131 | −HD: | |
| 132 | − | |
| 133 | −The devices for which the spindown timeout should be adjusted by laptop mode. | |
| 134 | −Default is /dev/hda. If you specify multiple devices, separate them by a space. | |
| 135 | − | |
| 136 | −READAHEAD: | |
| 137 | − | |
| 138 | −Disk readahead, in 512-byte sectors, while laptop mode is active. A large | |
| 139 | −readahead can prevent disk accesses for things like executable pages (which are | |
| 140 | −loaded on demand while the application executes) and sequentially accessed data | |
| 141 | −(MP3s). | |
| 142 | − | |
| 143 | −DO_REMOUNTS: | |
| 144 | − | |
| 145 | −The control script automatically remounts any mounted journaled filesystems | |
| 146 | −with appropriate commit interval options. When this option is set to 0, this | |
| 147 | −feature is disabled. | |
| 148 | − | |
| 149 | −DO_REMOUNT_NOATIME: | |
| 150 | − | |
| 151 | −When remounting, should the filesystems be remounted with the noatime option? | |
| 152 | −Normally, this is set to "1" (enabled), but there may be programs that require | |
| 153 | −access time recording. | |
| 154 | − | |
| 155 | −DIRTY_RATIO: | |
| 156 | − | |
| 157 | −The percentage of memory that is allowed to contain "dirty" or unsaved data | |
| 158 | −before a writeback is forced, while laptop mode is active. Corresponds to | |
| 159 | −the /proc/sys/vm/dirty_ratio sysctl. | |
| 160 | − | |
| 161 | −DIRTY_BACKGROUND_RATIO: | |
| 162 | − | |
| 163 | −The percentage of memory that is allowed to contain "dirty" or unsaved data | |
| 164 | −after a forced writeback is done due to an exceeding of DIRTY_RATIO. Set | |
| 165 | −this nice and low. This corresponds to the /proc/sys/vm/dirty_background_ratio | |
| 166 | −sysctl. | |
| 167 | − | |
| 168 | −Note that the behaviour of dirty_background_ratio is quite different | |
| 169 | −when laptop mode is active and when it isn't. When laptop mode is inactive, | |
| 170 | −dirty_background_ratio is the threshold percentage at which background writeouts | |
| 171 | −start taking place. When laptop mode is active, however, background writeouts | |
| 172 | −are disabled, and the dirty_background_ratio only determines how much writeback | |
| 173 | −is done when dirty_ratio is reached. | |
| 174 | − | |
| 175 | −DO_CPU: | |
| 176 | − | |
| 177 | −Enable CPU frequency scaling when in laptop mode. (Requires CPUFreq to be setup. | |
| 178 | −See Documentation/admin-guide/pm/cpufreq.rst for more info. Disabled by default.) | |
| 179 | − | |
| 180 | −CPU_MAXFREQ: | |
| 181 | − | |
| 182 | −When on battery, what is the maximum CPU speed that the system should use? Legal | |
| 183 | −values are "slowest" for the slowest speed that your CPU is able to operate at, | |
| 184 | −or a value listed in /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies. | |
| 185 | − | |
| 186 | − | |
| 187 | −Tips & Tricks | |
| 188 | −------------- | |
| 189 | − | |
| 190 | −* Bartek Kania reports getting up to 50 minutes of extra battery life (on top | |
| 191 | − of his regular 3 to 3.5 hours) using a spindown time of 5 seconds (BATT_HD=1). | |
| 192 | − | |
| 193 | −* You can spin down the disk while playing MP3, by setting disk readahead | |
| 194 | − to 8MB (READAHEAD=16384). Effectively, the disk will read a complete MP3 at | |
| 195 | − once, and will then spin down while the MP3 is playing. (Thanks to Bartek | |
| 196 | − Kania.) | |
| 197 | − | |
| 198 | −* Drew Scott Daniels observed: "I don't know why, but when I decrease the number | |
| 199 | − of colours that my display uses it consumes less battery power. I've seen | |
| 200 | − this on powerbooks too. I hope that this is a piece of information that | |
| 201 | − might be useful to the Laptop Mode patch or its users." | |
| 202 | − | |
| 203 | −* In syslog.conf, you can prefix entries with a dash `-` to omit syncing the | |
| 204 | − file after every logging. When you're using laptop-mode and your disk doesn't | |
| 205 | − spin down, this is a likely culprit. | |
| 206 | − | |
| 207 | −* Richard Atterer observed that laptop mode does not work well with noflushd | |
| 208 | − (http://noflushd.sourceforge.net/), it seems that noflushd prevents laptop-mode | |
| 209 | − from doing its thing. | |
| 210 | − | |
| 211 | −* If you're worried about your data, you might want to consider using a USB | |
| 212 | − memory stick or something like that as a "working area". (Be aware though | |
| 213 | − that flash memory can only handle a limited number of writes, and overuse | |
| 214 | − may wear out your memory stick pretty quickly. Do _not_ use journalling | |
| 215 | − filesystems on flash memory sticks.) | |
| 216 | − | |
| 217 | − | |
| 218 | −Configuration file for control and ACPI battery scripts | |
| 219 | −------------------------------------------------------- | |
| 220 | − | |
| 221 | −This allows the tunables to be changed for the scripts via an external | |
| 222 | −configuration file | |
| 223 | − | |
| 224 | −It should be installed as /etc/default/laptop-mode on Debian, and as | |
| 225 | −/etc/sysconfig/laptop-mode on Red Hat, SUSE, Mandrake, and other work-alikes. | |
| 226 | − | |
| 227 | −Config file:: | |
| 228 | − | |
| 229 | − # Maximum time, in seconds, of hard drive spindown time that you are | |
| 230 | − # comfortable with. Worst case, it's possible that you could lose this | |
| 231 | − # amount of work if your battery fails you while in laptop mode. | |
| 232 | − #MAX_AGE=600 | |
| 233 | − | |
| 234 | − # Automatically disable laptop mode when the number of minutes of battery | |
| 235 | − # that you have left goes below this threshold. | |
| 236 | − MINIMUM_BATTERY_MINUTES=10 | |
| 237 | − | |
| 238 | − # Read-ahead, in 512-byte sectors. You can spin down the disk while playing MP3/OGG | |
| 239 | − # by setting the disk readahead to 8MB (READAHEAD=16384). Effectively, the disk | |
| 240 | − # will read a complete MP3 at once, and will then spin down while the MP3/OGG is | |
| 241 | − # playing. | |
| 242 | − #READAHEAD=4096 | |
| 243 | − | |
| 244 | − # Shall we remount journaled fs. with appropriate commit interval? (1=yes) | |
| 245 | − #DO_REMOUNTS=1 | |
| 246 | − | |
| 247 | − # And shall we add the "noatime" option to that as well? (1=yes) | |
| 248 | − #DO_REMOUNT_NOATIME=1 | |
| 249 | − | |
| 250 | − # Dirty synchronous ratio. At this percentage of dirty pages the process | |
| 251 | − # which | |
| 252 | − # calls write() does its own writeback | |
| 253 | − #DIRTY_RATIO=40 | |
| 254 | − | |
| 255 | − # | |
| 256 | − # Allowed dirty background ratio, in percent. Once DIRTY_RATIO has been | |
| 257 | − # exceeded, the kernel will wake flusher threads which will then reduce the | |
| 258 | − # amount of dirty memory to dirty_background_ratio. Set this nice and low, | |
| 259 | − # so once some writeout has commenced, we do a lot of it. | |
| 260 | − # | |
| 261 | − #DIRTY_BACKGROUND_RATIO=5 | |
| 262 | − | |
| 263 | − # kernel default dirty buffer age | |
| 264 | − #DEF_AGE=30 | |
| 265 | − #DEF_UPDATE=5 | |
| 266 | − #DEF_DIRTY_BACKGROUND_RATIO=10 | |
| 267 | − #DEF_DIRTY_RATIO=40 | |
| 268 | − #DEF_XFS_AGE_BUFFER=15 | |
| 269 | − #DEF_XFS_SYNC_INTERVAL=30 | |
| 270 | − #DEF_XFS_BUFD_INTERVAL=1 | |
| 271 | − | |
| 272 | − # This must be adjusted manually to the value of HZ in the running kernel | |
| 273 | − # on 2.4, until the XFS people change their 2.4 external interfaces to work in | |
| 274 | − # centisecs. This can be automated, but it's a work in progress that still | |
| 275 | − # needs# some fixes. On 2.6 kernels, XFS uses USER_HZ instead of HZ for | |
| 276 | − # external interfaces, and that is currently always set to 100. So you don't | |
| 277 | − # need to change this on 2.6. | |
| 278 | − #XFS_HZ=100 | |
| 279 | − | |
| 280 | − # Should the maximum CPU frequency be adjusted down while on battery? | |
| 281 | − # Requires CPUFreq to be setup. | |
| 282 | − # See Documentation/admin-guide/pm/cpufreq.rst for more info | |
| 283 | − #DO_CPU=0 | |
| 284 | − | |
| 285 | − # When on battery what is the maximum CPU speed that the system should | |
| 286 | − # use? Legal values are "slowest" for the slowest speed that your | |
| 287 | − # CPU is able to operate at, or a value listed in: | |
| 288 | − # /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies | |
| 289 | − # Only applicable if DO_CPU=1. | |
| 290 | − #CPU_MAXFREQ=slowest | |
| 291 | − | |
| 292 | − # Idle timeout for your hard drive (man hdparm for valid values, -S option) | |
| 293 | − # Default is 2 hours on AC (AC_HD=244) and 20 seconds for battery (BATT_HD=4). | |
| 294 | − #AC_HD=244 | |
| 295 | − #BATT_HD=4 | |
| 296 | − | |
| 297 | − # The drives for which to adjust the idle timeout. Separate them by a space, | |
| 298 | − # e.g. HD="/dev/hda /dev/hdb". | |
| 299 | − #HD="/dev/hda" | |
| 300 | − | |
| 301 | − # Set the spindown timeout on a hard drive? | |
| 302 | − #DO_HD=1 | |
| 303 | − | |
| 304 | − | |
| 305 | −Control script | |
| 306 | −-------------- | |
| 307 | − | |
| 308 | −Please note that this control script works for the Linux 2.4 and 2.6 series (thanks | |
| 309 | −to Kiko Piris). | |
| 310 | − | |
| 311 | −Control script:: | |
| 312 | − | |
| 313 | − #!/bin/bash | |
| 314 | − | |
| 315 | − # start or stop laptop_mode, best run by a power management daemon when | |
| 316 | − # ac gets connected/disconnected from a laptop | |
| 317 | − # | |
| 318 | − # install as /sbin/laptop_mode | |
| 319 | − # | |
| 320 | − # Contributors to this script: Kiko Piris | |
| 321 | − # Bart Samwel | |
| 322 | − # Micha Feigin | |
| 323 | − # Andrew Morton | |
| 324 | − # Herve Eychenne | |
| 325 | − # Dax Kelson | |
| 326 | − # | |
| 327 | − # Original Linux 2.4 version by: Jens Axboe | |
| 328 | − | |
| 329 | − ############################################################################# | |
| 330 | − | |
| 331 | − # Source config | |
| 332 | − if [ -f /etc/default/laptop-mode ] ; then | |
| 333 | − # Debian | |
| 334 | − . /etc/default/laptop-mode | |
| 335 | − elif [ -f /etc/sysconfig/laptop-mode ] ; then | |
| 336 | − # Others | |
| 337 | − . /etc/sysconfig/laptop-mode | |
| 338 | − fi | |
| 339 | − | |
| 340 | − # Don't raise an error if the config file is incomplete | |
| 341 | − # set defaults instead: | |
| 342 | − | |
| 343 | − # Maximum time, in seconds, of hard drive spindown time that you are | |
| 344 | − # comfortable with. Worst case, it's possible that you could lose this | |
| 345 | − # amount of work if your battery fails you while in laptop mode. | |
| 346 | − MAX_AGE=${MAX_AGE:-'600'} | |
| 347 | − | |
| 348 | − # Read-ahead, in kilobytes | |
| 349 | − READAHEAD=${READAHEAD:-'4096'} | |
| 350 | − | |
| 351 | − # Shall we remount journaled fs. with appropriate commit interval? (1=yes) | |
| 352 | − DO_REMOUNTS=${DO_REMOUNTS:-'1'} | |
| 353 | − | |
| 354 | − # And shall we add the "noatime" option to that as well? (1=yes) | |
| 355 | − DO_REMOUNT_NOATIME=${DO_REMOUNT_NOATIME:-'1'} | |
| 356 | − | |
| 357 | − # Shall we adjust the idle timeout on a hard drive? | |
| 358 | − DO_HD=${DO_HD:-'1'} | |
| 359 | − | |
| 360 | − # Adjust idle timeout on which hard drive? | |
| 361 | − HD="${HD:-'/dev/hda'}" | |
| 362 | − | |
| 363 | − # spindown time for HD (hdparm -S values) | |
| 364 | − AC_HD=${AC_HD:-'244'} | |
| 365 | − BATT_HD=${BATT_HD:-'4'} | |
| 366 | − | |
| 367 | − # Dirty synchronous ratio. At this percentage of dirty pages the process which | |
| 368 | − # calls write() does its own writeback | |
| 369 | − DIRTY_RATIO=${DIRTY_RATIO:-'40'} | |
| 370 | − | |
| 371 | − # cpu frequency scaling | |
| 372 | − # See Documentation/admin-guide/pm/cpufreq.rst for more info | |
| 373 | − DO_CPU=${CPU_MANAGE:-'0'} | |
| 374 | − CPU_MAXFREQ=${CPU_MAXFREQ:-'slowest'} | |
| 375 | − | |
| 376 | − # | |
| 377 | − # Allowed dirty background ratio, in percent. Once DIRTY_RATIO has been | |
| 378 | − # exceeded, the kernel will wake flusher threads which will then reduce the | |
| 379 | − # amount of dirty memory to dirty_background_ratio. Set this nice and low, | |
| 380 | − # so once some writeout has commenced, we do a lot of it. | |
| 381 | − # | |
| 382 | − DIRTY_BACKGROUND_RATIO=${DIRTY_BACKGROUND_RATIO:-'5'} | |
| 383 | − | |
| 384 | − # kernel default dirty buffer age | |
| 385 | − DEF_AGE=${DEF_AGE:-'30'} | |
| 386 | − DEF_UPDATE=${DEF_UPDATE:-'5'} | |
| 387 | − DEF_DIRTY_BACKGROUND_RATIO=${DEF_DIRTY_BACKGROUND_RATIO:-'10'} | |
| 388 | − DEF_DIRTY_RATIO=${DEF_DIRTY_RATIO:-'40'} | |
| 389 | − DEF_XFS_AGE_BUFFER=${DEF_XFS_AGE_BUFFER:-'15'} | |
| 390 | − DEF_XFS_SYNC_INTERVAL=${DEF_XFS_SYNC_INTERVAL:-'30'} | |
| 391 | − DEF_XFS_BUFD_INTERVAL=${DEF_XFS_BUFD_INTERVAL:-'1'} | |
| 392 | − | |
| 393 | − # This must be adjusted manually to the value of HZ in the running kernel | |
| 394 | − # on 2.4, until the XFS people change their 2.4 external interfaces to work in | |
| 395 | − # centisecs. This can be automated, but it's a work in progress that still needs | |
| 396 | − # some fixes. On 2.6 kernels, XFS uses USER_HZ instead of HZ for external | |
| 397 | − # interfaces, and that is currently always set to 100. So you don't need to | |
| 398 | − # change this on 2.6. | |
| 399 | − XFS_HZ=${XFS_HZ:-'100'} | |
| 400 | − | |
| 401 | − ############################################################################# | |
| 402 | − | |
| 403 | − KLEVEL="$(uname -r | | |
| 404 | − { | |
| 405 | − IFS='.' read a b c | |
| 406 | − echo $a.$b | |
| 407 | − } | |
| 408 | − )" | |
| 409 | − case "$KLEVEL" in | |
| 410 | − "2.4"|"2.6") | |
| 411 | − ;; | |
| 412 | − *) | |
| 413 | − echo "Unhandled kernel version: $KLEVEL ('uname -r' = '$(uname -r)')" >&2 | |
| 414 | − exit 1 | |
| 415 | − ;; | |
| 416 | − esac | |
| 417 | − | |
| 418 | − if [ ! -e /proc/sys/vm/laptop_mode ] ; then | |
| 419 | − echo "Kernel is not patched with laptop_mode patch." >&2 | |
| 420 | − exit 1 | |
| 421 | − fi | |
| 422 | − | |
| 423 | − if [ ! -w /proc/sys/vm/laptop_mode ] ; then | |
| 424 | − echo "You do not have enough privileges to enable laptop_mode." >&2 | |
| 425 | − exit 1 | |
| 426 | − fi | |
| 427 | − | |
| 428 | − # Remove an option (the first parameter) of the form option=<number> from | |
| 429 | − # a mount options string (the rest of the parameters). | |
| 430 | − parse_mount_opts () { | |
| 431 | − OPT="$1" | |
| 432 | − shift | |
| 433 | − echo ",$*," | sed \ | |
| 434 | − -e 's/,'"$OPT"'=[0-9]*,/,/g' \ | |
| 435 | − -e 's/,,*/,/g' \ | |
| 436 | − -e 's/^,//' \ | |
| 437 | − -e 's/,$//' | |
| 438 | − } | |
| 439 | − | |
| 440 | − # Remove an option (the first parameter) without any arguments from | |
| 441 | − # a mount option string (the rest of the parameters). | |
| 442 | − parse_nonumber_mount_opts () { | |
| 443 | − OPT="$1" | |
| 444 | − shift | |
| 445 | − echo ",$*," | sed \ | |
| 446 | − -e 's/,'"$OPT"',/,/g' \ | |
| 447 | − -e 's/,,*/,/g' \ | |
| 448 | − -e 's/^,//' \ | |
| 449 | − -e 's/,$//' | |
| 450 | − } | |
| 451 | − | |
| 452 | − # Find out the state of a yes/no option (e.g. "atime"/"noatime") in | |
| 453 | − # fstab for a given filesystem, and use this state to replace the | |
| 454 | − # value of the option in another mount options string. The device | |
| 455 | − # is the first argument, the option name the second, and the default | |
| 456 | − # value the third. The remainder is the mount options string. | |
| 457 | − # | |
| 458 | − # Example: | |
| 459 | − # parse_yesno_opts_wfstab /dev/hda1 atime atime defaults,noatime | |
| 460 | − # | |
| 461 | − # If fstab contains, say, "rw" for this filesystem, then the result | |
| 462 | − # will be "defaults,atime". | |
| 463 | − parse_yesno_opts_wfstab () { | |
| 464 | − L_DEV="$1" | |
| 465 | − OPT="$2" | |
| 466 | − DEF_OPT="$3" | |
| 467 | − shift 3 | |
| 468 | − L_OPTS="$*" | |
| 469 | − PARSEDOPTS1="$(parse_nonumber_mount_opts $OPT $L_OPTS)" | |
| 470 | − PARSEDOPTS1="$(parse_nonumber_mount_opts no$OPT $PARSEDOPTS1)" | |
| 471 | − # Watch for a default atime in fstab | |
| 472 | − FSTAB_OPTS="$(awk '$1 == "'$L_DEV'" { print $4 }' /etc/fstab)" | |
| 473 | − if echo "$FSTAB_OPTS" | grep "$OPT" > /dev/null ; then | |
| 474 | − # option specified in fstab: extract the value and use it | |
| 475 | − if echo "$FSTAB_OPTS" | grep "no$OPT" > /dev/null ; then | |
| 476 | − echo "$PARSEDOPTS1,no$OPT" | |
| 477 | − else | |
| 478 | − # no$OPT not found -- so we must have $OPT. | |
| 479 | − echo "$PARSEDOPTS1,$OPT" | |
| 480 | − fi | |
| 481 | − else | |
| 482 | − # option not specified in fstab -- choose the default. | |
| 483 | − echo "$PARSEDOPTS1,$DEF_OPT" | |
| 484 | − fi | |
| 485 | − } | |
| 486 | − | |
| 487 | − # Find out the state of a numbered option (e.g. "commit=NNN") in | |
| 488 | − # fstab for a given filesystem, and use this state to replace the | |
| 489 | − # value of the option in another mount options string. The device | |
| 490 | − # is the first argument, and the option name the second. The | |
| 491 | − # remainder is the mount options string in which the replacement | |
| 492 | − # must be done. | |
| 493 | − # | |
| 494 | − # Example: | |
| 495 | − # parse_mount_opts_wfstab /dev/hda1 commit defaults,commit=7 | |
| 496 | − # | |
| 497 | − # If fstab contains, say, "commit=3,rw" for this filesystem, then the | |
| 498 | − # result will be "rw,commit=3". | |
| 499 | − parse_mount_opts_wfstab () { | |
| 500 | − L_DEV="$1" | |
| 501 | − OPT="$2" | |
| 502 | − shift 2 | |
| 503 | − L_OPTS="$*" | |
| 504 | − PARSEDOPTS1="$(parse_mount_opts $OPT $L_OPTS)" | |
| 505 | − # Watch for a default commit in fstab | |
| 506 | − FSTAB_OPTS="$(awk '$1 == "'$L_DEV'" { print $4 }' /etc/fstab)" | |
| 507 | − if echo "$FSTAB_OPTS" | grep "$OPT=" > /dev/null ; then | |
| 508 | − # option specified in fstab: extract the value, and use it | |
| 509 | − echo -n "$PARSEDOPTS1,$OPT=" | |
| 510 | − echo ",$FSTAB_OPTS," | sed \ | |
| 511 | − -e 's/.*,'"$OPT"'=//' \ | |
| 512 | − -e 's/,.*//' | |
| 513 | − else | |
| 514 | − # option not specified in fstab: set it to 0 | |
| 515 | − echo "$PARSEDOPTS1,$OPT=0" | |
| 516 | − fi | |
| 517 | − } | |
| 518 | − | |
| 519 | − deduce_fstype () { | |
| 520 | − MP="$1" | |
| 521 | − # My root filesystem unfortunately has | |
| 522 | − # type "unknown" in /etc/mtab. If we encounter | |
| 523 | − # "unknown", we try to get the type from fstab. | |
| 524 | − cat /etc/fstab | | |
| 525 | − grep -v '^#' | | |
| 526 | − while read FSTAB_DEV FSTAB_MP FSTAB_FST FSTAB_OPTS FSTAB_DUMP FSTAB_DUMP ; do | |
| 527 | − if [ "$FSTAB_MP" = "$MP" ]; then | |
| 528 | − echo $FSTAB_FST | |
| 529 | − exit 0 | |
| 530 | − fi | |
| 531 | − done | |
| 532 | − } | |
| 533 | − | |
| 534 | − if [ $DO_REMOUNT_NOATIME -eq 1 ] ; then | |
| 535 | − NOATIME_OPT=",noatime" | |
| 536 | − fi | |
| 537 | − | |
| 538 | − case "$1" in | |
| 539 | − start) | |
| 540 | − AGE=$((100*$MAX_AGE)) | |
| 541 | − XFS_AGE=$(($XFS_HZ*$MAX_AGE)) | |
| 542 | − echo -n "Starting laptop_mode" | |
| 543 | − | |
| 544 | − if [ -d /proc/sys/vm/pagebuf ] ; then | |
| 545 | − # (For 2.4 and early 2.6.) | |
| 546 | − # This only needs to be set, not reset -- it is only used when | |
| 547 | − # laptop mode is enabled. | |
| 548 | − echo $XFS_AGE > /proc/sys/vm/pagebuf/lm_flush_age | |
| 549 | − echo $XFS_AGE > /proc/sys/fs/xfs/lm_sync_interval | |
| 550 | − elif [ -f /proc/sys/fs/xfs/lm_age_buffer ] ; then | |
| 551 | − # (A couple of early 2.6 laptop mode patches had these.) | |
| 552 | − # The same goes for these. | |
| 553 | − echo $XFS_AGE > /proc/sys/fs/xfs/lm_age_buffer | |
| 554 | − echo $XFS_AGE > /proc/sys/fs/xfs/lm_sync_interval | |
| 555 | − elif [ -f /proc/sys/fs/xfs/age_buffer ] ; then | |
| 556 | − # (2.6.6) | |
| 557 | − # But not for these -- they are also used in normal | |
| 558 | − # operation. | |
| 559 | − echo $XFS_AGE > /proc/sys/fs/xfs/age_buffer | |
| 560 | − echo $XFS_AGE > /proc/sys/fs/xfs/sync_interval | |
| 561 | − elif [ -f /proc/sys/fs/xfs/age_buffer_centisecs ] ; then | |
| 562 | − # (2.6.7 upwards) | |
| 563 | − # And not for these either. These are in centisecs, | |
| 564 | − # not USER_HZ, so we have to use $AGE, not $XFS_AGE. | |
| 565 | − echo $AGE > /proc/sys/fs/xfs/age_buffer_centisecs | |
| 566 | − echo $AGE > /proc/sys/fs/xfs/xfssyncd_centisecs | |
| 567 | − echo 3000 > /proc/sys/fs/xfs/xfsbufd_centisecs | |
| 568 | − fi | |
| 569 | − | |
| 570 | − case "$KLEVEL" in | |
| 571 | − "2.4") | |
| 572 | − echo 1 > /proc/sys/vm/laptop_mode | |
| 573 | − echo "30 500 0 0 $AGE $AGE 60 20 0" > /proc/sys/vm/bdflush | |
| 574 | − ;; | |
| 575 | − "2.6") | |
| 576 | − echo 5 > /proc/sys/vm/laptop_mode | |
| 577 | − echo "$AGE" > /proc/sys/vm/dirty_writeback_centisecs | |
| 578 | − echo "$AGE" > /proc/sys/vm/dirty_expire_centisecs | |
| 579 | − echo "$DIRTY_RATIO" > /proc/sys/vm/dirty_ratio | |
| 580 | − echo "$DIRTY_BACKGROUND_RATIO" > /proc/sys/vm/dirty_background_ratio | |
| 581 | − ;; | |
| 582 | − esac | |
| 583 | − if [ $DO_REMOUNTS -eq 1 ]; then | |
| 584 | − cat /etc/mtab | while read DEV MP FST OPTS DUMP PASS ; do | |
| 585 | − PARSEDOPTS="$(parse_mount_opts "$OPTS")" | |
| 586 | − if [ "$FST" = 'unknown' ]; then | |
| 587 | − FST=$(deduce_fstype $MP) | |
| 588 | − fi | |
| 589 | − case "$FST" in | |
| 590 | − "ext3") | |
| 591 | − PARSEDOPTS="$(parse_mount_opts commit "$OPTS")" | |
| 592 | − mount $DEV -t $FST $MP -o remount,$PARSEDOPTS,commit=$MAX_AGE$NOATIME_OPT | |
| 593 | − ;; | |
| 594 | − "xfs") | |
| 595 | − mount $DEV -t $FST $MP -o remount,$OPTS$NOATIME_OPT | |
| 596 | − ;; | |
| 597 | − esac | |
| 598 | − if [ -b $DEV ] ; then | |
| 599 | − blockdev --setra $(($READAHEAD * 2)) $DEV | |
| 600 | − fi | |
| 601 | − done | |
| 602 | − fi | |
| 603 | − if [ $DO_HD -eq 1 ] ; then | |
| 604 | − for THISHD in $HD ; do | |
| 605 | − /sbin/hdparm -S $BATT_HD $THISHD > /dev/null 2>&1 | |
| 606 | − /sbin/hdparm -B 1 $THISHD > /dev/null 2>&1 | |
| 607 | − done | |
| 608 | − fi | |
| 609 | − if [ $DO_CPU -eq 1 -a -e /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq ]; then | |
| 610 | − if [ $CPU_MAXFREQ = 'slowest' ]; then | |
| 611 | − CPU_MAXFREQ=`cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq` | |
| 612 | − fi | |
| 613 | − echo $CPU_MAXFREQ > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq | |
| 614 | − fi | |
| 615 | − echo "." | |
| 616 | − ;; | |
| 617 | − stop) | |
| 618 | − U_AGE=$((100*$DEF_UPDATE)) | |
| 619 | − B_AGE=$((100*$DEF_AGE)) | |
| 620 | − echo -n "Stopping laptop_mode" | |
| 621 | − echo 0 > /proc/sys/vm/laptop_mode | |
| 622 | − if [ -f /proc/sys/fs/xfs/age_buffer -a ! -f /proc/sys/fs/xfs/lm_age_buffer ] ; then | |
| 623 | − # These need to be restored, if there are no lm_*. | |
| 624 | − echo $(($XFS_HZ*$DEF_XFS_AGE_BUFFER)) > /proc/sys/fs/xfs/age_buffer | |
| 625 | − echo $(($XFS_HZ*$DEF_XFS_SYNC_INTERVAL)) > /proc/sys/fs/xfs/sync_interval | |
| 626 | − elif [ -f /proc/sys/fs/xfs/age_buffer_centisecs ] ; then | |
| 627 | − # These need to be restored as well. | |
| 628 | − echo $((100*$DEF_XFS_AGE_BUFFER)) > /proc/sys/fs/xfs/age_buffer_centisecs | |
| 629 | − echo $((100*$DEF_XFS_SYNC_INTERVAL)) > /proc/sys/fs/xfs/xfssyncd_centisecs | |
| 630 | − echo $((100*$DEF_XFS_BUFD_INTERVAL)) > /proc/sys/fs/xfs/xfsbufd_centisecs | |
| 631 | − fi | |
| 632 | − case "$KLEVEL" in | |
| 633 | − "2.4") | |
| 634 | − echo "30 500 0 0 $U_AGE $B_AGE 60 20 0" > /proc/sys/vm/bdflush | |
| 635 | − ;; | |
| 636 | − "2.6") | |
| 637 | − echo "$U_AGE" > /proc/sys/vm/dirty_writeback_centisecs | |
| 638 | − echo "$B_AGE" > /proc/sys/vm/dirty_expire_centisecs | |
| 639 | − echo "$DEF_DIRTY_RATIO" > /proc/sys/vm/dirty_ratio | |
| 640 | − echo "$DEF_DIRTY_BACKGROUND_RATIO" > /proc/sys/vm/dirty_background_ratio | |
| 641 | − ;; | |
| 642 | − esac | |
| 643 | − if [ $DO_REMOUNTS -eq 1 ] ; then | |
| 644 | − cat /etc/mtab | while read DEV MP FST OPTS DUMP PASS ; do | |
| 645 | − # Reset commit and atime options to defaults. | |
| 646 | − if [ "$FST" = 'unknown' ]; then | |
| 647 | − FST=$(deduce_fstype $MP) | |
| 648 | − fi | |
| 649 | − case "$FST" in | |
| 650 | − "ext3") | |
| 651 | − PARSEDOPTS="$(parse_mount_opts_wfstab $DEV commit $OPTS)" | |
| 652 | − PARSEDOPTS="$(parse_yesno_opts_wfstab $DEV atime atime $PARSEDOPTS)" | |
| 653 | − mount $DEV -t $FST $MP -o remount,$PARSEDOPTS | |
| 654 | − ;; | |
| 655 | − "xfs") | |
| 656 | − PARSEDOPTS="$(parse_yesno_opts_wfstab $DEV atime atime $OPTS)" | |
| 657 | − mount $DEV -t $FST $MP -o remount,$PARSEDOPTS | |
| 658 | − ;; | |
| 659 | − esac | |
| 660 | − if [ -b $DEV ] ; then | |
| 661 | − blockdev --setra 256 $DEV | |
| 662 | − fi | |
| 663 | − done | |
| 664 | − fi | |
| 665 | − if [ $DO_HD -eq 1 ] ; then | |
| 666 | − for THISHD in $HD ; do | |
| 667 | − /sbin/hdparm -S $AC_HD $THISHD > /dev/null 2>&1 | |
| 668 | − /sbin/hdparm -B 255 $THISHD > /dev/null 2>&1 | |
| 669 | − done | |
| 670 | − fi | |
| 671 | − if [ $DO_CPU -eq 1 -a -e /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq ]; then | |
| 672 | − echo `cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq` > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq | |
| 673 | − fi | |
| 674 | − echo "." | |
| 675 | − ;; | |
| 676 | − *) | |
| 677 | − echo "Usage: $0 {start|stop}" 2>&1 | |
| 678 | − exit 1 | |
| 679 | − ;; | |
| 680 | − | |
| 681 | − esac | |
| 682 | − | |
| 683 | − exit 0 | |
| 684 | − | |
| 685 | − | |
| 686 | −ACPI integration | |
| 687 | −---------------- | |
| 688 | − | |
| 689 | −Dax Kelson submitted this so that the ACPI acpid daemon will | |
| 690 | −kick off the laptop_mode script and run hdparm. The part that | |
| 691 | −automatically disables laptop mode when the battery is low was | |
| 692 | −written by Jan Topinski. | |
| 693 | − | |
| 694 | −/etc/acpi/events/ac_adapter:: | |
| 695 | − | |
| 696 | − event=ac_adapter | |
| 697 | − action=/etc/acpi/actions/ac.sh %e | |
| 698 | − | |
| 699 | −/etc/acpi/events/battery:: | |
| 700 | − | |
| 701 | − event=battery.* | |
| 702 | − action=/etc/acpi/actions/battery.sh %e | |
| 703 | − | |
| 704 | −/etc/acpi/actions/ac.sh:: | |
| 705 | − | |
| 706 | − #!/bin/bash | |
| 707 | − | |
| 708 | − # ac on/offline event handler | |
| 709 | − | |
| 710 | − status=`awk '/^state: / { print $2 }' /proc/acpi/ac_adapter/$2/state` | |
| 711 | − | |
| 712 | − case $status in | |
| 713 | − "on-line") | |
| 714 | − /sbin/laptop_mode stop | |
| 715 | − exit 0 | |
| 716 | − ;; | |
| 717 | − "off-line") | |
| 718 | − /sbin/laptop_mode start | |
| 719 | − exit 0 | |
| 720 | − ;; | |
| 721 | − esac | |
| 722 | − | |
| 723 | − | |
| 724 | −/etc/acpi/actions/battery.sh:: | |
| 725 | − | |
| 726 | − #! /bin/bash | |
| 727 | − | |
| 728 | − # Automatically disable laptop mode when the battery almost runs out. | |
| 729 | − | |
| 730 | − BATT_INFO=/proc/acpi/battery/$2/state | |
| 731 | − | |
| 732 | − if [[ -f /proc/sys/vm/laptop_mode ]] | |
| 733 | − then | |
| 734 | − LM=`cat /proc/sys/vm/laptop_mode` | |
| 735 | − if [[ $LM -gt 0 ]] | |
| 736 | − then | |
| 737 | − if [[ -f $BATT_INFO ]] | |
| 738 | − then | |
| 739 | − # Source the config file only now that we know we need | |
| 740 | − if [ -f /etc/default/laptop-mode ] ; then | |
| 741 | − # Debian | |
| 742 | − . /etc/default/laptop-mode | |
| 743 | − elif [ -f /etc/sysconfig/laptop-mode ] ; then | |
| 744 | − # Others | |
| 745 | − . /etc/sysconfig/laptop-mode | |
| 746 | − fi | |
| 747 | − MINIMUM_BATTERY_MINUTES=${MINIMUM_BATTERY_MINUTES:-'10'} | |
| 748 | − | |
| 749 | − ACTION="`cat $BATT_INFO | grep charging | cut -c 26-`" | |
| 750 | − if [[ ACTION -eq "discharging" ]] | |
| 751 | − then | |
| 752 | − PRESENT_RATE=`cat $BATT_INFO | grep "present rate:" | sed "s/.* \([0-9][0-9]* \).*/\1/" ` | |
| 753 | − REMAINING=`cat $BATT_INFO | grep "remaining capacity:" | sed "s/.* \([0-9][0-9]* \).*/\1/" ` | |
| 754 | − fi | |
| 755 | − if (($REMAINING * 60 / $PRESENT_RATE < $MINIMUM_BATTERY_MINUTES)) | |
| 756 | − then | |
| 757 | − /sbin/laptop_mode stop | |
| 758 | − fi | |
| 759 | − else | |
| 760 | − logger -p daemon.warning "You are using laptop mode and your battery interface $BATT_INFO is missing. This may lead to loss of data when the battery runs out. Check kernel ACPI support and /proc/acpi/battery folder, and edit /etc/acpi/battery.sh to set BATT_INFO to the correct path." | |
| 761 | − fi | |
| 762 | − fi | |
| 763 | − fi | |
| 764 | − | |
| 765 | − | |
| 766 | −Monitoring tool | |
| 767 | −--------------- | |
| 768 | − | |
| 769 | −Bartek Kania submitted this, it can be used to measure how much time your disk | |
| 770 | −spends spun up/down. See tools/laptop/dslm/dslm.c | |
| @@ -79,6 +79,43 @@ of parametrs except ``enabled`` again. Once the re-reading is done, this · 78 unmodified lines | ||
| 79 | 79 | parameter is set as ``N``. If invalid parameters are found while the |
| 80 | 80 | re-reading, DAMON_LRU_SORT will be disabled. |
| 81 | 81 | |
| 82 | +active_mem_bp | |
| 83 | +------------- | |
| 84 | + | |
| 85 | +Desired active to [in]active memory ratio in bp (1/10,000). | |
| 86 | + | |
| 87 | +While keeping the caps that set by other quotas, DAMON_LRU_SORT automatically | |
| 88 | +increases and decreases the effective level of the quota aiming the LRU | |
| 89 | +[de]prioritizations of the hot and cold memory resulting in this active to | |
| 90 | +[in]active memory ratio. Value zero means disabling this auto-tuning feature. | |
| 91 | + | |
| 92 | +Disabled by default. | |
| 93 | + | |
| 94 | +Auto-tune monitoring intervals | |
| 95 | +------------------------------ | |
| 96 | + | |
| 97 | +If this parameter is set as ``Y``, DAMON_LRU_SORT automatically tunes DAMON's | |
| 98 | +sampling and aggregation intervals. The auto-tuning aims to capture meaningful | |
| 99 | +amount of access events in each DAMON-snapshot, while keeping the sampling | |
| 100 | +interval 5 milliseconds in minimum, and 10 seconds in maximum. Setting this as | |
| 101 | +``N`` disables the auto-tuning. | |
| 102 | + | |
| 103 | +Disabled by default. | |
| 104 | + | |
| 105 | +filter_young_pages | |
| 106 | +------------------ | |
| 107 | + | |
| 108 | +Filter [non-]young pages accordingly for LRU [de]prioritizations. | |
| 109 | + | |
| 110 | +If this is set, check page level access (youngness) once again before each | |
| 111 | +LRU [de]prioritization operation. LRU prioritization operation is skipped | |
| 112 | +if the page has not accessed since the last check (not young). LRU | |
| 113 | +deprioritization operation is skipped if the page has accessed since the | |
| 114 | +last check (young). The feature is enabled or disabled if this parameter is | |
| 115 | +set as ``Y`` or ``N``, respectively. | |
| 116 | + | |
| 117 | +Disabled by default. | |
| 118 | + | |
| 82 | 119 | hot_thres_access_freq |
| 83 | 120 | --------------------- |
| 84 | 121 | |
| @@ -6,6 +6,11 @@ Detailed Usages · 5 unmodified lines | ||
| 6 | 6 | |
| 7 | 7 | DAMON provides below interfaces for different users. |
| 8 | 8 | |
| 9 | +- *Special-purpose DAMON modules.* | |
| 10 | + :ref:`This <damon_modules_special_purpose>` is for people who are building, | |
| 11 | + distributing, and/or administrating the kernel with special-purpose DAMON | |
| 12 | + usages. Using this, users can use DAMON's major features for the given | |
| 13 | + purposes in build, boot, or runtime in simple ways. | |
| 9 | 14 | - *DAMON user space tool.* |
| 10 | 15 | `This <https://github.com/damonitor/damo>`_ is for privileged people such as |
| 11 | 16 | system administrators who want a just-working human-friendly interface. |
| @@ -87,7 +92,7 @@ comma (","). · 75 unmodified lines | ||
| 87 | 92 | │ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,target_idx,min,max |
| 88 | 93 | │ │ │ │ │ │ │ :ref:`dests <damon_sysfs_dests>`/nr_dests |
| 89 | 94 | │ │ │ │ │ │ │ │ 0/id,weight |
| 90 | − │ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,sz_ops_filter_passed,qt_exceeds | |
| 95 | + │ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,sz_ops_filter_passed,qt_exceeds,nr_snapshots,max_nr_snapshots | |
| 91 | 96 | │ │ │ │ │ │ │ :ref:`tried_regions <sysfs_schemes_tried_regions>`/total_bytes |
| 92 | 97 | │ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age,sz_filter_passed |
| 93 | 98 | │ │ │ │ │ │ │ │ ... |
| @@ -543,10 +548,14 @@ online analysis or tuning of the schemes. Refer to :ref:`design doc · 449 unmodified lines | ||
| 543 | 548 | |
| 544 | 549 | The statistics can be retrieved by reading the files under ``stats`` directory |
| 545 | 550 | (``nr_tried``, ``sz_tried``, ``nr_applied``, ``sz_applied``, |
| 546 | −``sz_ops_filter_passed``, and ``qt_exceeds``), respectively. The files are not | |
| 547 | −updated in real time, so you should ask DAMON sysfs interface to update the | |
| 548 | −content of the files for the stats by writing a special keyword, | |
| 549 | −``update_schemes_stats`` to the relevant ``kdamonds/<N>/state`` file. | |
| 551 | +``sz_ops_filter_passed``, ``qt_exceeds``, ``nr_snapshots`` and | |
| 552 | +``max_nr_snapshots``), respectively. | |
| 553 | + | |
| 554 | +The files are not updated in real time by default. Users should ask DAMON | |
| 555 | +sysfs interface to periodically update those using ``refresh_ms``, or do a one | |
| 556 | +time update by writing a special keyword, ``update_schemes_stats`` to the | |
| 557 | +relevant ``kdamonds/<N>/state`` file. Refer to :ref:`kdamond directory | |
| 558 | +<sysfs_kdamond>` for more details. | |
| 550 | 559 | |
| 551 | 560 | .. _sysfs_schemes_tried_regions: |
| 552 | 561 | |
| @@ -603,17 +603,18 @@ ZONE_MOVABLE, especially when fine-tuning zone ratios: · 602 unmodified lines | ||
| 603 | 603 | memory for metadata and page tables in the direct map; having a lot of offline |
| 604 | 604 | memory blocks is not a typical case, though. |
| 605 | 605 | |
| 606 | −- Memory ballooning without balloon compaction is incompatible with | |
| 607 | − ZONE_MOVABLE. Only some implementations, such as virtio-balloon and | |
| 608 | − pseries CMM, fully support balloon compaction. | |
| 606 | +- Memory ballooning without support for balloon memory migration is incompatible | |
| 607 | + with ZONE_MOVABLE. Only some implementations, such as virtio-balloon and | |
| 608 | + pseries CMM, fully support balloon memory migration. | |
| 609 | 609 | |
| 610 | − Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be | |
| 610 | + Further, the CONFIG_BALLOON_MIGRATION kernel configuration option might be | |
| 611 | 611 | disabled. In that case, balloon inflation will only perform unmovable |
| 612 | 612 | allocations and silently create a zone imbalance, usually triggered by |
| 613 | 613 | inflation requests from the hypervisor. |
| 614 | 614 | |
| 615 | −- Gigantic pages are unmovable, resulting in user space consuming a | |
| 616 | − lot of unmovable memory. | |
| 615 | +- Gigantic pages are unmovable when an architecture does not support | |
| 616 | + huge page migration and/or the ``movable_gigantic_pages`` sysctl is false. | |
| 617 | + See Documentation/admin-guide/sysctl/vm.rst for more info on this sysctl. | |
| 617 | 618 | |
| 618 | 619 | - Huge pages are unmovable when an architectures does not support huge |
| 619 | 620 | page migration, resulting in a similar issue as with gigantic pages. |
| @@ -672,6 +673,15 @@ block might fail: · 52 unmodified lines | ||
| 672 | 673 | - Concurrent activity that operates on the same physical memory area, such as |
| 673 | 674 | allocating gigantic pages, can result in temporary offlining failures. |
| 674 | 675 | |
| 676 | +- When an admin sets the ``movable_gigantic_pages`` sysctl to true, gigantic | |
| 677 | + pages are allowed in ZONE_MOVABLE. This only allows migratable gigantic | |
| 678 | + pages to be allocated; however, if there are no eligible destination gigantic | |
| 679 | + pages at offline, the offlining operation will fail. | |
| 680 | + | |
| 681 | + Users leveraging ``movable_gigantic_pages`` should weigh the value of | |
| 682 | + ZONE_MOVABLE for increasing the reliability of gigantic page allocation | |
| 683 | + against the potential loss of hot-unplug reliability. | |
| 684 | + | |
| 675 | 685 | - Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap |
| 676 | 686 | Optimization (HVO) is enabled. |
| 677 | 687 | |
| @@ -41,7 +41,6 @@ Currently, these files are in /proc/sys/vm: · 40 unmodified lines | ||
| 41 | 41 | - extfrag_threshold |
| 42 | 42 | - highmem_is_dirtyable |
| 43 | 43 | - hugetlb_shm_group |
| 44 | −- laptop_mode | |
| 45 | 44 | - legacy_va_layout |
| 46 | 45 | - lowmem_reserve_ratio |
| 47 | 46 | - max_map_count |
| @@ -54,6 +53,7 @@ Currently, these files are in /proc/sys/vm: · 6 unmodified lines | ||
| 54 | 53 | - mmap_min_addr |
| 55 | 54 | - mmap_rnd_bits |
| 56 | 55 | - mmap_rnd_compat_bits |
| 56 | +- movable_gigantic_pages | |
| 57 | 57 | - nr_hugepages |
| 58 | 58 | - nr_hugepages_mempolicy |
| 59 | 59 | - nr_overcommit_hugepages |
| @@ -365,13 +365,6 @@ hugetlb_shm_group contains group id that is allowed to create SysV · 305 unmodified lines | ||
| 365 | 365 | shared memory segment using hugetlb page. |
| 366 | 366 | |
| 367 | 367 | |
| 368 | −laptop_mode | |
| 369 | −=========== | |
| 370 | − | |
| 371 | −laptop_mode is a knob that controls "laptop mode". All the things that are | |
| 372 | −controlled by this knob are discussed in Documentation/admin-guide/laptops/laptop-mode.rst. | |
| 373 | − | |
| 374 | − | |
| 375 | 368 | legacy_va_layout |
| 376 | 369 | ================ |
| 377 | 370 | |
| @@ -630,6 +623,33 @@ This value can be changed after boot using the · 252 unmodified lines | ||
| 630 | 623 | /proc/sys/vm/mmap_rnd_compat_bits tunable |
| 631 | 624 | |
| 632 | 625 | |
| 626 | +movable_gigantic_pages | |
| 627 | +====================== | |
| 628 | + | |
| 629 | +This parameter controls whether gigantic pages may be allocated from | |
| 630 | +ZONE_MOVABLE. If set to non-zero, gigantic pages can be allocated | |
| 631 | +from ZONE_MOVABLE. ZONE_MOVABLE memory may be created via the kernel | |
| 632 | +boot parameter `kernelcore` or via memory hotplug as discussed in | |
| 633 | +Documentation/admin-guide/mm/memory-hotplug.rst. | |
| 634 | + | |
| 635 | +Support may depend on specific architecture. | |
| 636 | + | |
| 637 | +Note that using ZONE_MOVABLE gigantic pages make memory hotremove unreliable. | |
| 638 | + | |
| 639 | +Memory hot-remove operations will block indefinitely until the admin reserves | |
| 640 | +sufficient gigantic pages to service migration requests associated with the | |
| 641 | +memory offlining process. As HugeTLB gigantic page reservation is a manual | |
| 642 | +process (via `nodeN/hugepages/.../nr_hugepages` interfaces) this may not be | |
| 643 | +obvious when just attempting to offline a block of memory. | |
| 644 | + | |
| 645 | +Additionally, as multiple gigantic pages may be reserved on a single block, | |
| 646 | +it may appear that gigantic pages are available for migration when in reality | |
| 647 | +they are in the process of being removed. For example if `memoryN` contains | |
| 648 | +two gigantic pages, one reserved and one allocated, and an admin attempts to | |
| 649 | +offline that block, this operations may hang indefinitely unless another | |
| 650 | +reserved gigantic page is available on another block `memoryM`. | |
| 651 | + | |
| 652 | + | |
| 633 | 653 | nr_hugepages |
| 634 | 654 | ============ |
| 635 | 655 | |
| @@ -130,5 +130,5 @@ More Memory Management Functions · 129 unmodified lines | ||
| 130 | 130 | .. kernel-doc:: mm/vmscan.c |
| 131 | 131 | .. kernel-doc:: mm/memory_hotplug.c |
| 132 | 132 | .. kernel-doc:: mm/mmu_notifier.c |
| 133 | −.. kernel-doc:: mm/balloon_compaction.c | |
| 133 | +.. kernel-doc:: mm/balloon.c | |
| 134 | 134 | .. kernel-doc:: mm/huge_memory.c |
| @@ -125,7 +125,7 @@ The contiguous memory allocator (CMA) enables reservation of contiguous memory · 124 unmodified lines | ||
| 125 | 125 | regions on NUMA nodes during early boot. However, CMA cannot reserve memory |
| 126 | 126 | on NUMA nodes that are not online during early boot. :: |
| 127 | 127 | |
| 128 | − void __init hugetlb_cma_reserve(int order) { | |
| 128 | + void __init hugetlb_cma_reserve(void) { | |
| 129 | 129 | if (!node_online(nid)) |
| 130 | 130 | /* do not allow reservations */ |
| 131 | 131 | } |
| @@ -585,6 +585,10 @@ mechanism tries to make ``current_value`` of ``target_metric`` be same to · 584 unmodified lines | ||
| 585 | 585 | specific NUMA node, in bp (1/10,000). |
| 586 | 586 | - ``node_memcg_free_bp``: Specific cgroup's node unused memory ratio for a |
| 587 | 587 | specific NUMA node, in bp (1/10,000). |
| 588 | +- ``active_mem_bp``: Active to active + inactive (LRU) memory size ratio in bp | |
| 589 | + (1/10,000). | |
| 590 | +- ``inactive_mem_bp``: Inactive to active + inactive (LRU) memory size ratio in | |
| 591 | + bp (1/10,000). | |
| 588 | 592 | |
| 589 | 593 | ``nid`` is optionally required for only ``node_mem_used_bp``, |
| 590 | 594 | ``node_mem_free_bp``, ``node_memcg_used_bp`` and ``node_memcg_free_bp`` to |
| @@ -718,6 +722,9 @@ scheme's execution. · 127 unmodified lines | ||
| 718 | 722 | - ``nr_applied``: Total number of regions that the scheme is applied. |
| 719 | 723 | - ``sz_applied``: Total size of regions that the scheme is applied. |
| 720 | 724 | - ``qt_exceeds``: Total number of times the quota of the scheme has exceeded. |
| 725 | +- ``nr_snapshots``: Total number of DAMON snapshots that the scheme is tried to | |
| 726 | + be applied. | |
| 727 | +- ``max_nr_snapshots``: Upper limit of ``nr_snapshots``. | |
| 721 | 728 | |
| 722 | 729 | "A scheme is tried to be applied to a region" means DAMOS core logic determined |
| 723 | 730 | the region is eligible to apply the scheme's :ref:`action |
| @@ -739,6 +746,10 @@ to exclude anonymous pages and the region has only anonymous pages, or if the · 15 unmodified lines | ||
| 739 | 746 | action is ``pageout`` while all pages of the region are unreclaimable, applying |
| 740 | 747 | the action to the region will fail. |
| 741 | 748 | |
| 749 | +Unlike normal stats, ``max_nr_snapshots`` is set by users. If it is set as | |
| 750 | +non-zero and ``nr_snapshots`` be same to or greater than ``nr_snapshots``, the | |
| 751 | +scheme is deactivated. | |
| 752 | + | |
| 742 | 753 | To know how user-space can read the stats via :ref:`DAMON sysfs interface |
| 743 | 754 | <sysfs_interface>`, refer to :ref:s`stats <sysfs_stats>` part of the |
| 744 | 755 | documentation. |
| @@ -798,14 +809,16 @@ The ABIs are designed to be used for user space applications development, · 53 unmodified lines | ||
| 798 | 809 | rather than human beings' fingers. Human users are recommended to use such |
| 799 | 810 | user space tools. One such Python-written user space tool is available at |
| 800 | 811 | Github (https://github.com/damonitor/damo), Pypi |
| 801 | −(https://pypistats.org/packages/damo), and Fedora | |
| 802 | −(https://packages.fedoraproject.org/pkgs/python-damo/damo/). | |
| 812 | +(https://pypistats.org/packages/damo), and multiple distros | |
| 813 | +(https://repology.org/project/damo/versions). | |
| 803 | 814 | |
| 804 | 815 | Currently, one module for this type, namely 'DAMON sysfs interface' is |
| 805 | 816 | available. Please refer to the ABI :ref:`doc <sysfs_interface>` for details of |
| 806 | 817 | the interfaces. |
| 807 | 818 | |
| 808 | 819 | |
| 820 | +.. _damon_modules_special_purpose: | |
| 821 | + | |
| 809 | 822 | Special-Purpose Access-aware Kernel Modules |
| 810 | 823 | ------------------------------------------- |
| 811 | 824 | |
| @@ -823,5 +836,18 @@ To support such cases, yet more DAMON API user kernel modules that provide more · 11 unmodified lines | ||
| 823 | 836 | simple and optimized user space interfaces are available. Currently, two |
| 824 | 837 | modules for proactive reclamation and LRU lists manipulation are provided. For |
| 825 | 838 | more detail, please read the usage documents for those |
| 826 | −(:doc:`/admin-guide/mm/damon/reclaim` and | |
| 839 | +(:doc:`/admin-guide/mm/damon/stat`, :doc:`/admin-guide/mm/damon/reclaim` and | |
| 827 | 840 | :doc:`/admin-guide/mm/damon/lru_sort`). |
| 841 | + | |
| 842 | + | |
| 843 | +Sample DAMON Modules | |
| 844 | +-------------------- | |
| 845 | + | |
| 846 | +DAMON modules that provides example DAMON kernel API usages. | |
| 847 | + | |
| 848 | +kernel programmers can build their own special or general purpose DAMON modules | |
| 849 | +using DAMON kernel API. To help them easily understand how DAMON kernel API | |
| 850 | +can be used, a few sample modules are provided under ``samples/damon/`` of the | |
| 851 | +linux source tree. Please note that these modules are not developed for being | |
| 852 | +used on real products, but only for showing how DAMON kernel API can be used in | |
| 853 | +simple ways. | |
| @@ -4,28 +4,15 @@ · 3 unmodified lines | ||
| 4 | 4 | DAMON: Data Access MONitoring and Access-aware System Operations |
| 5 | 5 | ================================================================ |
| 6 | 6 | |
| 7 | −DAMON is a Linux kernel subsystem that provides a framework for data access | |
| 8 | −monitoring and the monitoring results based system operations. The core | |
| 9 | −monitoring :ref:`mechanisms <damon_design_monitoring>` of DAMON make it | |
| 10 | − | |
| 11 | − - *accurate* (the monitoring output is useful enough for DRAM level memory | |
| 12 | − management; It might not appropriate for CPU Cache levels, though), | |
| 13 | − - *light-weight* (the monitoring overhead is low enough to be applied online), | |
| 14 | − and | |
| 15 | − - *scalable* (the upper-bound of the overhead is in constant range regardless | |
| 16 | − of the size of target workloads). | |
| 17 | − | |
| 18 | −Using this framework, therefore, the kernel can operate system in an | |
| 19 | −access-aware fashion. Because the features are also exposed to the :doc:`user | |
| 20 | −space </admin-guide/mm/damon/index>`, users who have special information about | |
| 21 | −their workloads can write personalized applications for better understanding | |
| 22 | −and optimizations of their workloads and systems. | |
| 23 | − | |
| 24 | −For easier development of such systems, DAMON provides a feature called | |
| 25 | −:ref:`DAMOS <damon_design_damos>` (DAMon-based Operation Schemes) in addition | |
| 26 | −to the monitoring. Using the feature, DAMON users in both kernel and :doc:`user | |
| 27 | −spaces </admin-guide/mm/damon/index>` can do access-aware system operations | |
| 28 | −with no code but simple configurations. | |
| 7 | +DAMON is a Linux kernel subsystem for efficient :ref:`data access monitoring | |
| 8 | +<damon_design_monitoring>` and :ref:`access-aware system operations | |
| 9 | +<damon_design_damos>`. It is designed for being | |
| 10 | + | |
| 11 | + - *accurate* (for DRAM level memory management), | |
| 12 | + - *light-weight* (for production online usages), | |
| 13 | + - *scalable* (in terms of memory size), | |
| 14 | + - *tunable* (for flexible usages), and | |
| 15 | + - *autoamted* (for production operation without manual tunings). | |
| 29 | 16 | |
| 30 | 17 | .. toctree:: |
| 31 | 18 | :maxdepth: 2 |
| @@ -3,8 +3,8 @@ · 2 unmodified lines | ||
| 3 | 3 | DAMON Maintainer Entry Profile |
| 4 | 4 | ============================== |
| 5 | 5 | |
| 6 | −The DAMON subsystem covers the files that are listed in 'DATA ACCESS MONITOR' | |
| 7 | −section of 'MAINTAINERS' file. | |
| 6 | +The DAMON subsystem covers the files that are listed in 'DAMON' section of | |
| 7 | +'MAINTAINERS' file. | |
| 8 | 8 | |
| 9 | 9 | The mailing lists for the subsystem are damon@lists.linux.dev and |
| 10 | 10 | linux-mm@kvack.org. Patches should be made against the `mm-new tree |
| @@ -48,8 +48,7 @@ Further doing below and putting the results will be helpful. · 37 unmodified lines | ||
| 48 | 48 | - Run `damon-tests/corr |
| 49 | 49 | <https://github.com/damonitor/damon-tests/tree/master/corr>`_ for normal |
| 50 | 50 | changes. |
| 51 | −- Run `damon-tests/perf | |
| 52 | − <https://github.com/damonitor/damon-tests/tree/master/perf>`_ for performance | |
| 51 | +- Measure impacts on benchmarks or real world workloads for performance | |
| 53 | 52 | changes. |
| 54 | 53 | |
| 55 | 54 | Key cycle dates |
| @@ -97,9 +97,6 @@ sections: · 96 unmodified lines | ||
| 97 | 97 | `mem_section` objects and the number of rows is calculated to fit |
| 98 | 98 | all the memory sections. |
| 99 | 99 | |
| 100 | −The architecture setup code should call sparse_init() to | |
| 101 | −initialize the memory sections and the memory maps. | |
| 102 | − | |
| 103 | 100 | With SPARSEMEM there are two possible ways to convert a PFN to the |
| 104 | 101 | corresponding `struct page` - a "classic sparse" and "sparse |
| 105 | 102 | vmemmap". The selection is made at build time and it is determined by |
| @@ -83,8 +83,6 @@ SPARSEMEM模型将物理内存显示为一个部分的集合。一个区段用me · 82 unmodified lines | ||
| 83 | 83 | 每一行包含价值 `PAGE_SIZE` 的 `mem_section` 对象，行数的计算是为了适应所有的 |
| 84 | 84 | 内存区。 |
| 85 | 85 | |
| 86 | −架构设置代码应该调用sparse_init()来初始化内存区和内存映射。 | |
| 87 | − | |
| 88 | 86 | 通过SPARSEMEM，有两种可能的方式将PFN转换为相应的 `struct page` --"classic sparse"和 |
| 89 | 87 | "sparse vmemmap"。选择是在构建时进行的，它由 `CONFIG_SPARSEMEM_VMEMMAP` 的 |
| 90 | 88 | 值决定。 |
| @@ -16583,6 +16583,17 @@ T: quilt git://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new · 16582 unmodified lines | ||
| 16583 | 16583 | F: mm/ |
| 16584 | 16584 | F: tools/mm/ |
| 16585 | 16585 | |
| 16586 | +MEMORY MANAGEMENT - BALLOON | |
| 16587 | +M: Andrew Morton <akpm@linux-foundation.org> | |
| 16588 | +M: David Hildenbrand <david@kernel.org> | |
| 16589 | +L: linux-mm@kvack.org | |
| 16590 | +L: virtualization@lists.linux.dev | |
| 16591 | +S: Maintained | |
| 16592 | +W: http://www.linux-mm.org | |
| 16593 | +T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm | |
| 16594 | +F: include/linux/balloon.h | |
| 16595 | +F: mm/balloon.c | |
| 16596 | + | |
| 16586 | 16597 | MEMORY MANAGEMENT - CORE |
| 16587 | 16598 | M: Andrew Morton <akpm@linux-foundation.org> |
| 16588 | 16599 | M: David Hildenbrand <david@kernel.org> |
| @@ -16810,7 +16821,6 @@ R: Shakeel Butt <shakeel.butt@linux.dev> · 221 unmodified lines | ||
| 16810 | 16821 | R: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> |
| 16811 | 16822 | L: linux-mm@kvack.org |
| 16812 | 16823 | S: Maintained |
| 16813 | −F: mm/pt_reclaim.c | |
| 16814 | 16824 | F: mm/vmscan.c |
| 16815 | 16825 | F: mm/workingset.c |
| 16816 | 16826 | |
| @@ -27776,9 +27786,7 @@ M: David Hildenbrand <david@kernel.org> · 10959 unmodified lines | ||
| 27776 | 27786 | L: virtualization@lists.linux.dev |
| 27777 | 27787 | S: Maintained |
| 27778 | 27788 | F: drivers/virtio/virtio_balloon.c |
| 27779 | −F: include/linux/balloon_compaction.h | |
| 27780 | 27789 | F: include/uapi/linux/virtio_balloon.h |
| 27781 | −F: mm/balloon_compaction.c | |
| 27782 | 27790 | |
| 27783 | 27791 | VIRTIO BLOCK AND SCSI DRIVERS |
| 27784 | 27792 | M: "Michael S. Tsirkin" <mst@redhat.com> |
| @@ -38,6 +38,7 @@ config ALPHA · 37 unmodified lines | ||
| 38 | 38 | select OLD_SIGSUSPEND |
| 39 | 39 | select CPU_NO_EFFICIENT_FFS if !ALPHA_EV67 |
| 40 | 40 | select MMU_GATHER_NO_RANGE |
| 41 | + select MMU_GATHER_RCU_TABLE_FREE | |
| 41 | 42 | select SPARSEMEM_EXTREME if SPARSEMEM |
| 42 | 43 | select ZONE_DMA |
| 43 | 44 | help |
| @@ -11,7 +11,6 @@ · 10 unmodified lines | ||
| 11 | 11 | #define STRICT_MM_TYPECHECKS |
| 12 | 12 | |
| 13 | 13 | extern void clear_page(void *page); |
| 14 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 15 | 14 | |
| 16 | 15 | #define vma_alloc_zeroed_movable_folio(vma, vaddr) \ |
| 17 | 16 | vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr) |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | |
| 5 | 5 | #include <asm-generic/tlb.h> |
| 6 | 6 | |
| 7 | −#define __pte_free_tlb(tlb, pte, address) pte_free((tlb)->mm, pte) | |
| 8 | −#define __pmd_free_tlb(tlb, pmd, address) pmd_free((tlb)->mm, pmd) | |
| 9 | − | |
| 7 | +#define __pte_free_tlb(tlb, pte, address) tlb_remove_ptdesc((tlb), page_ptdesc(pte)) | |
| 8 | +#define __pmd_free_tlb(tlb, pmd, address) tlb_remove_ptdesc((tlb), virt_to_ptdesc(pmd)) | |
| 9 | + | |
| 10 | 10 | #endif |
| @@ -607,7 +607,6 @@ setup_arch(char **cmdline_p) · 606 unmodified lines | ||
| 607 | 607 | /* Find our memory. */ |
| 608 | 608 | setup_memory(kernel_end); |
| 609 | 609 | memblock_set_bottom_up(true); |
| 610 | − sparse_init(); | |
| 611 | 610 | |
| 612 | 611 | /* First guess at cpu cache sizes. Do this before init_arch. */ |
| 613 | 612 | determine_cpu_caches(cpu->type); |
| @@ -208,12 +208,8 @@ callback_init(void * kernel_end) · 207 unmodified lines | ||
| 208 | 208 | return kernel_end; |
| 209 | 209 | } |
| 210 | 210 | |
| 211 | −/* | |
| 212 | − * paging_init() sets up the memory map. | |
| 213 | − */ | |
| 214 | −void __init paging_init(void) | |
| 211 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfn) | |
| 215 | 212 | { |
| 216 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = {0, }; | |
| 217 | 213 | unsigned long dma_pfn; |
| 218 | 214 | |
| 219 | 215 | dma_pfn = virt_to_phys((char *)MAX_DMA_ADDRESS) >> PAGE_SHIFT; |
| @@ -221,11 +217,13 @@ void __init paging_init(void) · 1 unmodified line | ||
| 221 | 217 | |
| 222 | 218 | max_zone_pfn[ZONE_DMA] = dma_pfn; |
| 223 | 219 | max_zone_pfn[ZONE_NORMAL] = max_pfn; |
| 220 | +} | |
| 224 | 221 | |
| 225 | − /* Initialize mem_map[]. */ | |
| 226 | − free_area_init(max_zone_pfn); | |
| 227 | − | |
| 228 | − /* Initialize the kernel's ZERO_PGE. */ | |
| 222 | +/* | |
| 223 | + * paging_init() initializes the kernel's ZERO_PGE. | |
| 224 | + */ | |
| 225 | +void __init paging_init(void) | |
| 226 | +{ | |
| 229 | 227 | memset(absolute_pointer(ZERO_PGE), 0, PAGE_SIZE); |
| 230 | 228 | } |
| 231 | 229 | |
| @@ -32,6 +32,8 @@ struct page; · 31 unmodified lines | ||
| 32 | 32 | |
| 33 | 33 | void copy_user_highpage(struct page *to, struct page *from, |
| 34 | 34 | unsigned long u_vaddr, struct vm_area_struct *vma); |
| 35 | + | |
| 36 | +#define clear_user_page clear_user_page | |
| 35 | 37 | void clear_user_page(void *to, unsigned long u_vaddr, struct page *page); |
| 36 | 38 | |
| 37 | 39 | typedef struct { |
| @@ -75,6 +75,25 @@ void __init early_init_dt_add_memory_arch(u64 base, u64 size) · 74 unmodified lines | ||
| 75 | 75 | base, TO_MB(size), !in_use ? "Not used":""); |
| 76 | 76 | } |
| 77 | 77 | |
| 78 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfn) | |
| 79 | +{ | |
| 80 | + /*----------------- node/zones setup --------------------------*/ | |
| 81 | + max_zone_pfn[ZONE_NORMAL] = max_low_pfn; | |
| 82 | + | |
| 83 | +#ifdef CONFIG_HIGHMEM | |
| 84 | + /* | |
| 85 | + * max_high_pfn should be ok here for both HIGHMEM and HIGHMEM+PAE. | |
| 86 | + * For HIGHMEM without PAE max_high_pfn should be less than | |
| 87 | + * min_low_pfn to guarantee that these two regions don't overlap. | |
| 88 | + * For PAE case highmem is greater than lowmem, so it is natural | |
| 89 | + * to use max_high_pfn. | |
| 90 | + * | |
| 91 | + * In both cases, holes should be handled by pfn_valid(). | |
| 92 | + */ | |
| 93 | + max_zone_pfn[ZONE_HIGHMEM] = max_high_pfn; | |
| 94 | +#endif | |
| 95 | +} | |
| 96 | + | |
| 78 | 97 | /* |
| 79 | 98 | * First memory setup routine called from setup_arch() |
| 80 | 99 | * 1. setup swapper's mm @init_mm |
| @@ -83,8 +102,6 @@ void __init early_init_dt_add_memory_arch(u64 base, u64 size) · 2 unmodified lines | ||
| 83 | 102 | */ |
| 84 | 103 | void __init setup_arch_memory(void) |
| 85 | 104 | { |
| 86 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 87 | − | |
| 88 | 105 | setup_initial_init_mm(_text, _etext, _edata, _end); |
| 89 | 106 | |
| 90 | 107 | /* first page of system - kernel .vector starts here */ |
| @@ -122,9 +139,6 @@ void __init setup_arch_memory(void) · 31 unmodified lines | ||
| 122 | 139 | |
| 123 | 140 | memblock_dump_all(); |
| 124 | 141 | |
| 125 | − /*----------------- node/zones setup --------------------------*/ | |
| 126 | − max_zone_pfn[ZONE_NORMAL] = max_low_pfn; | |
| 127 | − | |
| 128 | 142 | #ifdef CONFIG_HIGHMEM |
| 129 | 143 | /* |
| 130 | 144 | * On ARC (w/o PAE) HIGHMEM addresses are actually smaller (0 based) |
| @@ -139,22 +153,9 @@ void __init setup_arch_memory(void) · 8 unmodified lines | ||
| 139 | 153 | min_high_pfn = PFN_DOWN(high_mem_start); |
| 140 | 154 | max_high_pfn = PFN_DOWN(high_mem_start + high_mem_sz); |
| 141 | 155 | |
| 142 | − /* | |
| 143 | − * max_high_pfn should be ok here for both HIGHMEM and HIGHMEM+PAE. | |
| 144 | − * For HIGHMEM without PAE max_high_pfn should be less than | |
| 145 | − * min_low_pfn to guarantee that these two regions don't overlap. | |
| 146 | − * For PAE case highmem is greater than lowmem, so it is natural | |
| 147 | − * to use max_high_pfn. | |
| 148 | − * | |
| 149 | − * In both cases, holes should be handled by pfn_valid(). | |
| 150 | − */ | |
| 151 | − max_zone_pfn[ZONE_HIGHMEM] = max_high_pfn; | |
| 152 | − | |
| 153 | 156 | arch_pfn_offset = min(min_low_pfn, min_high_pfn); |
| 154 | 157 | kmap_init(); |
| 155 | 158 | #endif /* CONFIG_HIGHMEM */ |
| 156 | − | |
| 157 | − free_area_init(max_zone_pfn); | |
| 158 | 159 | } |
| 159 | 160 | |
| 160 | 161 | void __init arch_mm_preinit(void) |
| @@ -11,7 +11,6 @@ · 10 unmodified lines | ||
| 11 | 11 | #define clear_page(page) memset((page), 0, PAGE_SIZE) |
| 12 | 12 | #define copy_page(to,from) memcpy((to), (from), PAGE_SIZE) |
| 13 | 13 | |
| 14 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 15 | 14 | #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 16 | 15 | |
| 17 | 16 | /* |
| @@ -15,8 +15,8 @@ · 14 unmodified lines | ||
| 15 | 15 | * ZERO_PAGE is a global shared page that is always zero: used |
| 16 | 16 | * for zero-mapped memory areas etc.. |
| 17 | 17 | */ |
| 18 | −extern struct page *empty_zero_page; | |
| 19 | −#define ZERO_PAGE(vaddr) (empty_zero_page) | |
| 18 | +extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; | |
| 19 | +#define ZERO_PAGE(vaddr) (virt_to_page(empty_zero_page)) | |
| 20 | 20 | #endif |
| 21 | 21 | |
| 22 | 22 | #include <asm-generic/pgtable-nopud.h> |
| @@ -107,19 +107,15 @@ void __init setup_dma_zone(const struct machine_desc *mdesc) · 106 unmodified lines | ||
| 107 | 107 | #endif |
| 108 | 108 | } |
| 109 | 109 | |
| 110 | −static void __init zone_sizes_init(unsigned long min, unsigned long max_low, | |
| 111 | − unsigned long max_high) | |
| 110 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfn) | |
| 112 | 111 | { |
| 113 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 114 | − | |
| 115 | 112 | #ifdef CONFIG_ZONE_DMA |
| 116 | − max_zone_pfn[ZONE_DMA] = min(arm_dma_pfn_limit, max_low); | |
| 113 | + max_zone_pfn[ZONE_DMA] = min(arm_dma_pfn_limit, max_low_pfn); | |
| 117 | 114 | #endif |
| 118 | − max_zone_pfn[ZONE_NORMAL] = max_low; | |
| 115 | + max_zone_pfn[ZONE_NORMAL] = max_low_pfn; | |
| 119 | 116 | #ifdef CONFIG_HIGHMEM |
| 120 | − max_zone_pfn[ZONE_HIGHMEM] = max_high; | |
| 117 | + max_zone_pfn[ZONE_HIGHMEM] = max_pfn; | |
| 121 | 118 | #endif |
| 122 | − free_area_init(max_zone_pfn); | |
| 123 | 119 | } |
| 124 | 120 | |
| 125 | 121 | #ifdef CONFIG_HAVE_ARCH_PFN_VALID |
| @@ -211,19 +207,6 @@ void __init bootmem_init(void) · 85 unmodified lines | ||
| 211 | 207 | |
| 212 | 208 | early_memtest((phys_addr_t)min_low_pfn << PAGE_SHIFT, |
| 213 | 209 | (phys_addr_t)max_low_pfn << PAGE_SHIFT); |
| 214 | − | |
| 215 | − /* | |
| 216 | − * sparse_init() tries to allocate memory from memblock, so must be | |
| 217 | − * done after the fixed reservations | |
| 218 | − */ | |
| 219 | − sparse_init(); | |
| 220 | − | |
| 221 | − /* | |
| 222 | − * Now free the memory - free_area_init needs | |
| 223 | − * the sparse mem_map arrays initialized by sparse_init() | |
| 224 | − * for memmap_init_zone(), otherwise all PFNs are invalid. | |
| 225 | − */ | |
| 226 | − zone_sizes_init(min_low_pfn, max_low_pfn, max_pfn); | |
| 227 | 210 | } |
| 228 | 211 | |
| 229 | 212 | /* |
| @@ -45,7 +45,7 @@ extern unsigned long __atags_pointer; · 44 unmodified lines | ||
| 45 | 45 | * empty_zero_page is a special page that is used for |
| 46 | 46 | * zero-initialized data and COW. |
| 47 | 47 | */ |
| 48 | −struct page *empty_zero_page; | |
| 48 | +unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)] __page_aligned_bss; | |
| 49 | 49 | EXPORT_SYMBOL(empty_zero_page); |
| 50 | 50 | |
| 51 | 51 | /* |
| @@ -1754,8 +1754,6 @@ static void __init early_fixmap_shutdown(void) · 1702 unmodified lines | ||
| 1754 | 1754 | */ |
| 1755 | 1755 | void __init paging_init(const struct machine_desc *mdesc) |
| 1756 | 1756 | { |
| 1757 | − void *zero_page; | |
| 1758 | − | |
| 1759 | 1757 | #ifdef CONFIG_XIP_KERNEL |
| 1760 | 1758 | /* Store the kernel RW RAM region start/end in these variables */ |
| 1761 | 1759 | kernel_sec_start = CONFIG_PHYS_OFFSET & SECTION_MASK; |
| @@ -1781,13 +1779,7 @@ void __init paging_init(const struct machine_desc *mdesc) · 19 unmodified lines | ||
| 1781 | 1779 | |
| 1782 | 1780 | top_pmd = pmd_off_k(0xffff0000); |
| 1783 | 1781 | |
| 1784 | − /* allocate the zero page. */ | |
| 1785 | − zero_page = early_alloc(PAGE_SIZE); | |
| 1786 | − | |
| 1787 | 1782 | bootmem_init(); |
| 1788 | − | |
| 1789 | − empty_zero_page = virt_to_page(zero_page); | |
| 1790 | − __flush_dcache_folio(NULL, page_folio(empty_zero_page)); | |
| 1791 | 1783 | } |
| 1792 | 1784 | |
| 1793 | 1785 | void __init early_mm_init(const struct machine_desc *mdesc) |
| @@ -31,7 +31,7 @@ unsigned long vectors_base; · 30 unmodified lines | ||
| 31 | 31 | * empty_zero_page is a special page that is used for |
| 32 | 32 | * zero-initialized data and COW. |
| 33 | 33 | */ |
| 34 | −struct page *empty_zero_page; | |
| 34 | +unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)] __page_aligned_bss; | |
| 35 | 35 | EXPORT_SYMBOL(empty_zero_page); |
| 36 | 36 | |
| 37 | 37 | #ifdef CONFIG_ARM_MPU |
| @@ -156,18 +156,10 @@ void __init adjust_lowmem_bounds(void) · 118 unmodified lines | ||
| 156 | 156 | */ |
| 157 | 157 | void __init paging_init(const struct machine_desc *mdesc) |
| 158 | 158 | { |
| 159 | − void *zero_page; | |
| 160 | − | |
| 161 | 159 | early_trap_init((void *)vectors_base); |
| 162 | 160 | mpu_setup(); |
| 163 | 161 | |
| 164 | − /* allocate the zero page. */ | |
| 165 | − zero_page = (void *)memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE); | |
| 166 | − | |
| 167 | 162 | bootmem_init(); |
| 168 | − | |
| 169 | − empty_zero_page = virt_to_page(zero_page); | |
| 170 | − flush_dcache_page(empty_zero_page); | |
| 171 | 163 | } |
| 172 | 164 | |
| 173 | 165 | /* |
| @@ -35,6 +35,7 @@ config ARM64 · 34 unmodified lines | ||
| 35 | 35 | select ARCH_HAS_KCOV |
| 36 | 36 | select ARCH_HAS_KERNEL_FPU_SUPPORT if KERNEL_MODE_NEON |
| 37 | 37 | select ARCH_HAS_KEEPINITRD |
| 38 | + select ARCH_HAS_LAZY_MMU_MODE | |
| 38 | 39 | select ARCH_HAS_MEMBARRIER_SYNC_CORE |
| 39 | 40 | select ARCH_HAS_MEM_ENCRYPT |
| 40 | 41 | select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS |
| @@ -56,8 +56,6 @@ extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, · 55 unmodified lines | ||
| 56 | 56 | #define __HAVE_ARCH_HUGE_PTEP_GET |
| 57 | 57 | extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); |
| 58 | 58 | |
| 59 | −void __init arm64_hugetlb_cma_reserve(void); | |
| 60 | − | |
| 61 | 59 | #define huge_ptep_modify_prot_start huge_ptep_modify_prot_start |
| 62 | 60 | extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, |
| 63 | 61 | unsigned long addr, pte_t *ptep); |
| @@ -36,7 +36,6 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, · 35 unmodified lines | ||
| 36 | 36 | bool tag_clear_highpages(struct page *to, int numpages); |
| 37 | 37 | #define __HAVE_ARCH_TAG_CLEAR_HIGHPAGES |
| 38 | 38 | |
| 39 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 40 | 39 | #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 41 | 40 | |
| 42 | 41 | typedef struct page *pgtable_t; |
| @@ -62,61 +62,26 @@ static inline void emit_pte_barriers(void) · 61 unmodified lines | ||
| 62 | 62 | |
| 63 | 63 | static inline void queue_pte_barriers(void) |
| 64 | 64 | { |
| 65 | − unsigned long flags; | |
| 66 | − | |
| 67 | − if (in_interrupt()) { | |
| 68 | − emit_pte_barriers(); | |
| 69 | − return; | |
| 70 | − } | |
| 71 | − | |
| 72 | − flags = read_thread_flags(); | |
| 73 | − | |
| 74 | − if (flags & BIT(TIF_LAZY_MMU)) { | |
| 65 | + if (is_lazy_mmu_mode_active()) { | |
| 75 | 66 | /* Avoid the atomic op if already set. */ |
| 76 | − if (!(flags & BIT(TIF_LAZY_MMU_PENDING))) | |
| 67 | + if (!test_thread_flag(TIF_LAZY_MMU_PENDING)) | |
| 77 | 68 | set_thread_flag(TIF_LAZY_MMU_PENDING); |
| 78 | 69 | } else { |
| 79 | 70 | emit_pte_barriers(); |
| 80 | 71 | } |
| 81 | 72 | } |
| 82 | 73 | |
| 83 | −#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE | |
| 84 | −static inline void arch_enter_lazy_mmu_mode(void) | |
| 85 | −{ | |
| 86 | − /* | |
| 87 | − * lazy_mmu_mode is not supposed to permit nesting. But in practice this | |
| 88 | − * does happen with CONFIG_DEBUG_PAGEALLOC, where a page allocation | |
| 89 | − * inside a lazy_mmu_mode section (such as zap_pte_range()) will change | |
| 90 | − * permissions on the linear map with apply_to_page_range(), which | |
| 91 | − * re-enters lazy_mmu_mode. So we tolerate nesting in our | |
| 92 | − * implementation. The first call to arch_leave_lazy_mmu_mode() will | |
| 93 | − * flush and clear the flag such that the remainder of the work in the | |
| 94 | − * outer nest behaves as if outside of lazy mmu mode. This is safe and | |
| 95 | − * keeps tracking simple. | |
| 96 | − */ | |
| 97 | − | |
| 98 | − if (in_interrupt()) | |
| 99 | − return; | |
| 100 | − | |
| 101 | − set_thread_flag(TIF_LAZY_MMU); | |
| 102 | −} | |
| 74 | +static inline void arch_enter_lazy_mmu_mode(void) {} | |
| 103 | 75 | |
| 104 | 76 | static inline void arch_flush_lazy_mmu_mode(void) |
| 105 | 77 | { |
| 106 | − if (in_interrupt()) | |
| 107 | − return; | |
| 108 | − | |
| 109 | 78 | if (test_and_clear_thread_flag(TIF_LAZY_MMU_PENDING)) |
| 110 | 79 | emit_pte_barriers(); |
| 111 | 80 | } |
| 112 | 81 | |
| 113 | 82 | static inline void arch_leave_lazy_mmu_mode(void) |
| 114 | 83 | { |
| 115 | − if (in_interrupt()) | |
| 116 | − return; | |
| 117 | − | |
| 118 | 84 | arch_flush_lazy_mmu_mode(); |
| 119 | − clear_thread_flag(TIF_LAZY_MMU); | |
| 120 | 85 | } |
| 121 | 86 | |
| 122 | 87 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| @@ -708,22 +673,24 @@ static inline pgprot_t pud_pgprot(pud_t pud) · 585 unmodified lines | ||
| 708 | 673 | return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); |
| 709 | 674 | } |
| 710 | 675 | |
| 711 | −static inline void __set_ptes_anysz(struct mm_struct *mm, pte_t *ptep, | |
| 712 | − pte_t pte, unsigned int nr, | |
| 676 | +static inline void __set_ptes_anysz(struct mm_struct *mm, unsigned long addr, | |
| 677 | + pte_t *ptep, pte_t pte, unsigned int nr, | |
| 713 | 678 | unsigned long pgsize) |
| 714 | 679 | { |
| 715 | 680 | unsigned long stride = pgsize >> PAGE_SHIFT; |
| 716 | 681 | |
| 717 | 682 | switch (pgsize) { |
| 718 | 683 | case PAGE_SIZE: |
| 719 | − page_table_check_ptes_set(mm, ptep, pte, nr); | |
| 684 | + page_table_check_ptes_set(mm, addr, ptep, pte, nr); | |
| 720 | 685 | break; |
| 721 | 686 | case PMD_SIZE: |
| 722 | − page_table_check_pmds_set(mm, (pmd_t *)ptep, pte_pmd(pte), nr); | |
| 687 | + page_table_check_pmds_set(mm, addr, (pmd_t *)ptep, | |
| 688 | + pte_pmd(pte), nr); | |
| 723 | 689 | break; |
| 724 | 690 | #ifndef __PAGETABLE_PMD_FOLDED |
| 725 | 691 | case PUD_SIZE: |
| 726 | − page_table_check_puds_set(mm, (pud_t *)ptep, pte_pud(pte), nr); | |
| 692 | + page_table_check_puds_set(mm, addr, (pud_t *)ptep, | |
| 693 | + pte_pud(pte), nr); | |
| 727 | 694 | break; |
| 728 | 695 | #endif |
| 729 | 696 | default: |
| @@ -744,26 +711,23 @@ static inline void __set_ptes_anysz(struct mm_struct *mm, pte_t *ptep, · 14 unmodified lines | ||
| 744 | 711 | __set_pte_complete(pte); |
| 745 | 712 | } |
| 746 | 713 | |
| 747 | −static inline void __set_ptes(struct mm_struct *mm, | |
| 748 | − unsigned long __always_unused addr, | |
| 714 | +static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, | |
| 749 | 715 | pte_t *ptep, pte_t pte, unsigned int nr) |
| 750 | 716 | { |
| 751 | − __set_ptes_anysz(mm, ptep, pte, nr, PAGE_SIZE); | |
| 717 | + __set_ptes_anysz(mm, addr, ptep, pte, nr, PAGE_SIZE); | |
| 752 | 718 | } |
| 753 | 719 | |
| 754 | −static inline void __set_pmds(struct mm_struct *mm, | |
| 755 | − unsigned long __always_unused addr, | |
| 720 | +static inline void __set_pmds(struct mm_struct *mm, unsigned long addr, | |
| 756 | 721 | pmd_t *pmdp, pmd_t pmd, unsigned int nr) |
| 757 | 722 | { |
| 758 | − __set_ptes_anysz(mm, (pte_t *)pmdp, pmd_pte(pmd), nr, PMD_SIZE); | |
| 723 | + __set_ptes_anysz(mm, addr, (pte_t *)pmdp, pmd_pte(pmd), nr, PMD_SIZE); | |
| 759 | 724 | } |
| 760 | 725 | #define set_pmd_at(mm, addr, pmdp, pmd) __set_pmds(mm, addr, pmdp, pmd, 1) |
| 761 | 726 | |
| 762 | −static inline void __set_puds(struct mm_struct *mm, | |
| 763 | − unsigned long __always_unused addr, | |
| 727 | +static inline void __set_puds(struct mm_struct *mm, unsigned long addr, | |
| 764 | 728 | pud_t *pudp, pud_t pud, unsigned int nr) |
| 765 | 729 | { |
| 766 | − __set_ptes_anysz(mm, (pte_t *)pudp, pud_pte(pud), nr, PUD_SIZE); | |
| 730 | + __set_ptes_anysz(mm, addr, (pte_t *)pudp, pud_pte(pud), nr, PUD_SIZE); | |
| 767 | 731 | } |
| 768 | 732 | #define set_pud_at(mm, addr, pudp, pud) __set_puds(mm, addr, pudp, pud, 1) |
| 769 | 733 | |
| @@ -1301,17 +1265,17 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, · 531 unmodified lines | ||
| 1301 | 1265 | #endif |
| 1302 | 1266 | |
| 1303 | 1267 | #ifdef CONFIG_PAGE_TABLE_CHECK |
| 1304 | −static inline bool pte_user_accessible_page(pte_t pte) | |
| 1268 | +static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr) | |
| 1305 | 1269 | { |
| 1306 | 1270 | return pte_valid(pte) && (pte_user(pte) || pte_user_exec(pte)); |
| 1307 | 1271 | } |
| 1308 | 1272 | |
| 1309 | −static inline bool pmd_user_accessible_page(pmd_t pmd) | |
| 1273 | +static inline bool pmd_user_accessible_page(pmd_t pmd, unsigned long addr) | |
| 1310 | 1274 | { |
| 1311 | 1275 | return pmd_valid(pmd) && !pmd_table(pmd) && (pmd_user(pmd) || pmd_user_exec(pmd)); |
| 1312 | 1276 | } |
| 1313 | 1277 | |
| 1314 | −static inline bool pud_user_accessible_page(pud_t pud) | |
| 1278 | +static inline bool pud_user_accessible_page(pud_t pud, unsigned long addr) | |
| 1315 | 1279 | { |
| 1316 | 1280 | return pud_valid(pud) && !pud_table(pud) && (pud_user(pud) || pud_user_exec(pud)); |
| 1317 | 1281 | } |
| @@ -1370,6 +1334,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, · 52 unmodified lines | ||
| 1370 | 1334 | #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ |
| 1371 | 1335 | |
| 1372 | 1336 | static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm, |
| 1337 | + unsigned long address, | |
| 1373 | 1338 | pte_t *ptep, |
| 1374 | 1339 | unsigned long pgsize) |
| 1375 | 1340 | { |
| @@ -1377,14 +1342,14 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm, · 1 unmodified line | ||
| 1377 | 1342 | |
| 1378 | 1343 | switch (pgsize) { |
| 1379 | 1344 | case PAGE_SIZE: |
| 1380 | − page_table_check_pte_clear(mm, pte); | |
| 1345 | + page_table_check_pte_clear(mm, address, pte); | |
| 1381 | 1346 | break; |
| 1382 | 1347 | case PMD_SIZE: |
| 1383 | − page_table_check_pmd_clear(mm, pte_pmd(pte)); | |
| 1348 | + page_table_check_pmd_clear(mm, address, pte_pmd(pte)); | |
| 1384 | 1349 | break; |
| 1385 | 1350 | #ifndef __PAGETABLE_PMD_FOLDED |
| 1386 | 1351 | case PUD_SIZE: |
| 1387 | − page_table_check_pud_clear(mm, pte_pud(pte)); | |
| 1352 | + page_table_check_pud_clear(mm, address, pte_pud(pte)); | |
| 1388 | 1353 | break; |
| 1389 | 1354 | #endif |
| 1390 | 1355 | default: |
| @@ -1397,7 +1362,7 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm, · 6 unmodified lines | ||
| 1397 | 1362 | static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, |
| 1398 | 1363 | unsigned long address, pte_t *ptep) |
| 1399 | 1364 | { |
| 1400 | − return __ptep_get_and_clear_anysz(mm, ptep, PAGE_SIZE); | |
| 1365 | + return __ptep_get_and_clear_anysz(mm, address, ptep, PAGE_SIZE); | |
| 1401 | 1366 | } |
| 1402 | 1367 | |
| 1403 | 1368 | static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, |
| @@ -1436,7 +1401,7 @@ static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, · 32 unmodified lines | ||
| 1436 | 1401 | static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, |
| 1437 | 1402 | unsigned long address, pmd_t *pmdp) |
| 1438 | 1403 | { |
| 1439 | − return pte_pmd(__ptep_get_and_clear_anysz(mm, (pte_t *)pmdp, PMD_SIZE)); | |
| 1404 | + return pte_pmd(__ptep_get_and_clear_anysz(mm, address, (pte_t *)pmdp, PMD_SIZE)); | |
| 1440 | 1405 | } |
| 1441 | 1406 | #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ |
| 1442 | 1407 | |
| @@ -1525,7 +1490,7 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, · 82 unmodified lines | ||
| 1525 | 1490 | static inline pmd_t pmdp_establish(struct vm_area_struct *vma, |
| 1526 | 1491 | unsigned long address, pmd_t *pmdp, pmd_t pmd) |
| 1527 | 1492 | { |
| 1528 | − page_table_check_pmd_set(vma->vm_mm, pmdp, pmd); | |
| 1493 | + page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd); | |
| 1529 | 1494 | return __pmd(xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd))); |
| 1530 | 1495 | } |
| 1531 | 1496 | #endif |
| @@ -84,8 +84,7 @@ void arch_setup_new_exec(void); · 83 unmodified lines | ||
| 84 | 84 | #define TIF_SME_VL_INHERIT 28 /* Inherit SME vl_onexec across exec */ |
| 85 | 85 | #define TIF_KERNEL_FPSTATE 29 /* Task is in a kernel mode FPSIMD section */ |
| 86 | 86 | #define TIF_TSC_SIGSEGV 30 /* SIGSEGV on counter-timer access */ |
| 87 | −#define TIF_LAZY_MMU 31 /* Task in lazy mmu mode */ | |
| 88 | −#define TIF_LAZY_MMU_PENDING 32 /* Ops pending for lazy mmu mode exit */ | |
| 87 | +#define TIF_LAZY_MMU_PENDING 31 /* Ops pending for lazy mmu mode exit */ | |
| 89 | 88 | |
| 90 | 89 | #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) |
| 91 | 90 | #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) |
| @@ -36,16 +36,12 @@ · 35 unmodified lines | ||
| 36 | 36 | * huge pages could still be served from those areas. |
| 37 | 37 | */ |
| 38 | 38 | #ifdef CONFIG_CMA |
| 39 | −void __init arm64_hugetlb_cma_reserve(void) | |
| 39 | +unsigned int arch_hugetlb_cma_order(void) | |
| 40 | 40 | { |
| 41 | − int order; | |
| 42 | − | |
| 43 | 41 | if (pud_sect_supported()) |
| 44 | − order = PUD_SHIFT - PAGE_SHIFT; | |
| 45 | − else | |
| 46 | − order = CONT_PMD_SHIFT - PAGE_SHIFT; | |
| 42 | + return PUD_SHIFT - PAGE_SHIFT; | |
| 47 | 43 | |
| 48 | − hugetlb_cma_reserve(order); | |
| 44 | + return CONT_PMD_SHIFT - PAGE_SHIFT; | |
| 49 | 45 | } |
| 50 | 46 | #endif /* CONFIG_CMA */ |
| 51 | 47 | |
| @@ -159,11 +155,12 @@ static pte_t get_clear_contig(struct mm_struct *mm, · 107 unmodified lines | ||
| 159 | 155 | pte_t pte, tmp_pte; |
| 160 | 156 | bool present; |
| 161 | 157 | |
| 162 | − pte = __ptep_get_and_clear_anysz(mm, ptep, pgsize); | |
| 158 | + pte = __ptep_get_and_clear_anysz(mm, addr, ptep, pgsize); | |
| 163 | 159 | present = pte_present(pte); |
| 164 | 160 | while (--ncontig) { |
| 165 | 161 | ptep++; |
| 166 | − tmp_pte = __ptep_get_and_clear_anysz(mm, ptep, pgsize); | |
| 162 | + addr += pgsize; | |
| 163 | + tmp_pte = __ptep_get_and_clear_anysz(mm, addr, ptep, pgsize); | |
| 167 | 164 | if (present) { |
| 168 | 165 | if (pte_dirty(tmp_pte)) |
| 169 | 166 | pte = pte_mkdirty(pte); |
| @@ -207,7 +204,7 @@ static void clear_flush(struct mm_struct *mm, · 37 unmodified lines | ||
| 207 | 204 | unsigned long i, saddr = addr; |
| 208 | 205 | |
| 209 | 206 | for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) |
| 210 | − __ptep_get_and_clear_anysz(mm, ptep, pgsize); | |
| 207 | + __ptep_get_and_clear_anysz(mm, addr, ptep, pgsize); | |
| 211 | 208 | |
| 212 | 209 | if (mm == &init_mm) |
| 213 | 210 | flush_tlb_kernel_range(saddr, addr); |
| @@ -225,8 +222,8 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 11 unmodified lines | ||
| 225 | 222 | ncontig = num_contig_ptes(sz, &pgsize); |
| 226 | 223 | |
| 227 | 224 | if (!pte_present(pte)) { |
| 228 | − for (i = 0; i < ncontig; i++, ptep++) | |
| 229 | − __set_ptes_anysz(mm, ptep, pte, 1, pgsize); | |
| 225 | + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) | |
| 226 | + __set_ptes_anysz(mm, addr, ptep, pte, 1, pgsize); | |
| 230 | 227 | return; |
| 231 | 228 | } |
| 232 | 229 | |
| @@ -234,7 +231,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 1 unmodified line | ||
| 234 | 231 | if (pte_cont(pte) && pte_valid(__ptep_get(ptep))) |
| 235 | 232 | clear_flush(mm, addr, ptep, pgsize, ncontig); |
| 236 | 233 | |
| 237 | − __set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); | |
| 234 | + __set_ptes_anysz(mm, addr, ptep, pte, ncontig, pgsize); | |
| 238 | 235 | } |
| 239 | 236 | |
| 240 | 237 | pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, |
| @@ -449,7 +446,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 208 unmodified lines | ||
| 449 | 446 | if (pte_young(orig_pte)) |
| 450 | 447 | pte = pte_mkyoung(pte); |
| 451 | 448 | |
| 452 | − __set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); | |
| 449 | + __set_ptes_anysz(mm, addr, ptep, pte, ncontig, pgsize); | |
| 453 | 450 | return 1; |
| 454 | 451 | } |
| 455 | 452 | |
| @@ -473,7 +470,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, · 17 unmodified lines | ||
| 473 | 470 | pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); |
| 474 | 471 | pte = pte_wrprotect(pte); |
| 475 | 472 | |
| 476 | − __set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); | |
| 473 | + __set_ptes_anysz(mm, addr, ptep, pte, ncontig, pgsize); | |
| 477 | 474 | } |
| 478 | 475 | |
| 479 | 476 | pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, |
| @@ -118,9 +118,22 @@ static phys_addr_t __init max_zone_phys(phys_addr_t zone_limit) · 117 unmodified lines | ||
| 118 | 118 | return min(zone_limit, memblock_end_of_DRAM() - 1) + 1; |
| 119 | 119 | } |
| 120 | 120 | |
| 121 | −static void __init zone_sizes_init(void) | |
| 121 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 122 | +{ | |
| 123 | + phys_addr_t __maybe_unused dma32_phys_limit = | |
| 124 | + max_zone_phys(DMA_BIT_MASK(32)); | |
| 125 | + | |
| 126 | +#ifdef CONFIG_ZONE_DMA | |
| 127 | + max_zone_pfns[ZONE_DMA] = PFN_DOWN(max_zone_phys(zone_dma_limit)); | |
| 128 | +#endif | |
| 129 | +#ifdef CONFIG_ZONE_DMA32 | |
| 130 | + max_zone_pfns[ZONE_DMA32] = PFN_DOWN(dma32_phys_limit); | |
| 131 | +#endif | |
| 132 | + max_zone_pfns[ZONE_NORMAL] = max_pfn; | |
| 133 | +} | |
| 134 | + | |
| 135 | +static void __init dma_limits_init(void) | |
| 122 | 136 | { |
| 123 | − unsigned long max_zone_pfns[MAX_NR_ZONES] = {0}; | |
| 124 | 137 | phys_addr_t __maybe_unused acpi_zone_dma_limit; |
| 125 | 138 | phys_addr_t __maybe_unused dt_zone_dma_limit; |
| 126 | 139 | phys_addr_t __maybe_unused dma32_phys_limit = |
| @@ -139,18 +152,13 @@ static void __init zone_sizes_init(void) · 12 unmodified lines | ||
| 139 | 152 | if (memblock_start_of_DRAM() < U32_MAX) |
| 140 | 153 | zone_dma_limit = min(zone_dma_limit, U32_MAX); |
| 141 | 154 | arm64_dma_phys_limit = max_zone_phys(zone_dma_limit); |
| 142 | − max_zone_pfns[ZONE_DMA] = PFN_DOWN(arm64_dma_phys_limit); | |
| 143 | 155 | #endif |
| 144 | 156 | #ifdef CONFIG_ZONE_DMA32 |
| 145 | − max_zone_pfns[ZONE_DMA32] = PFN_DOWN(dma32_phys_limit); | |
| 146 | 157 | if (!arm64_dma_phys_limit) |
| 147 | 158 | arm64_dma_phys_limit = dma32_phys_limit; |
| 148 | 159 | #endif |
| 149 | 160 | if (!arm64_dma_phys_limit) |
| 150 | 161 | arm64_dma_phys_limit = PHYS_MASK + 1; |
| 151 | − max_zone_pfns[ZONE_NORMAL] = max_pfn; | |
| 152 | − | |
| 153 | − free_area_init(max_zone_pfns); | |
| 154 | 162 | } |
| 155 | 163 | |
| 156 | 164 | int pfn_is_map_memory(unsigned long pfn) |
| @@ -303,23 +311,8 @@ void __init bootmem_init(void) · 146 unmodified lines | ||
| 303 | 311 | |
| 304 | 312 | arch_numa_init(); |
| 305 | 313 | |
| 306 | − /* | |
| 307 | − * must be done after arch_numa_init() which calls numa_init() to | |
| 308 | − * initialize node_online_map that gets used in hugetlb_cma_reserve() | |
| 309 | − * while allocating required CMA size across online nodes. | |
| 310 | − */ | |
| 311 | −#if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA) | |
| 312 | − arm64_hugetlb_cma_reserve(); | |
| 313 | −#endif | |
| 314 | − | |
| 315 | 314 | kvm_hyp_reserve(); |
| 316 | − | |
| 317 | − /* | |
| 318 | − * sparse_init() tries to allocate memory from memblock, so must be | |
| 319 | − * done after the fixed reservations | |
| 320 | − */ | |
| 321 | − sparse_init(); | |
| 322 | − zone_sizes_init(); | |
| 315 | + dma_limits_init(); | |
| 323 | 316 | |
| 324 | 317 | /* |
| 325 | 318 | * Reserve the CMA area after arm64_dma_phys_limit was initialised. |
| @@ -800,7 +800,7 @@ int split_kernel_leaf_mapping(unsigned long start, unsigned long end) · 799 unmodified lines | ||
| 800 | 800 | return -EINVAL; |
| 801 | 801 | |
| 802 | 802 | mutex_lock(&pgtable_split_lock); |
| 803 | − arch_enter_lazy_mmu_mode(); | |
| 803 | + lazy_mmu_mode_enable(); | |
| 804 | 804 | |
| 805 | 805 | /* |
| 806 | 806 | * The split_kernel_leaf_mapping_locked() may sleep, it is not a |
| @@ -822,7 +822,7 @@ int split_kernel_leaf_mapping(unsigned long start, unsigned long end) · 15 unmodified lines | ||
| 822 | 822 | ret = split_kernel_leaf_mapping_locked(end); |
| 823 | 823 | } |
| 824 | 824 | |
| 825 | − arch_leave_lazy_mmu_mode(); | |
| 825 | + lazy_mmu_mode_disable(); | |
| 826 | 826 | mutex_unlock(&pgtable_split_lock); |
| 827 | 827 | return ret; |
| 828 | 828 | } |
| @@ -883,10 +883,10 @@ static int range_split_to_ptes(unsigned long start, unsigned long end, gfp_t gfp · 54 unmodified lines | ||
| 883 | 883 | { |
| 884 | 884 | int ret; |
| 885 | 885 | |
| 886 | − arch_enter_lazy_mmu_mode(); | |
| 886 | + lazy_mmu_mode_enable(); | |
| 887 | 887 | ret = walk_kernel_page_table_range_lockless(start, end, |
| 888 | 888 | &split_to_ptes_ops, NULL, &gfp); |
| 889 | − arch_leave_lazy_mmu_mode(); | |
| 889 | + lazy_mmu_mode_disable(); | |
| 890 | 890 | |
| 891 | 891 | return ret; |
| 892 | 892 | } |
| @@ -110,7 +110,7 @@ static int update_range_prot(unsigned long start, unsigned long size, · 109 unmodified lines | ||
| 110 | 110 | if (WARN_ON_ONCE(ret)) |
| 111 | 111 | return ret; |
| 112 | 112 | |
| 113 | − arch_enter_lazy_mmu_mode(); | |
| 113 | + lazy_mmu_mode_enable(); | |
| 114 | 114 | |
| 115 | 115 | /* |
| 116 | 116 | * The caller must ensure that the range we are operating on does not |
| @@ -119,7 +119,7 @@ static int update_range_prot(unsigned long start, unsigned long size, · 2 unmodified lines | ||
| 119 | 119 | */ |
| 120 | 120 | ret = walk_kernel_page_table_range_lockless(start, start + size, |
| 121 | 121 | &pageattr_ops, NULL, &data); |
| 122 | − arch_leave_lazy_mmu_mode(); | |
| 122 | + lazy_mmu_mode_disable(); | |
| 123 | 123 | |
| 124 | 124 | return ret; |
| 125 | 125 | } |
| @@ -10,6 +10,7 @@ static inline unsigned long pages_do_alias(unsigned long addr1, · 9 unmodified lines | ||
| 10 | 10 | return (addr1 ^ addr2) & (SHMLBA-1); |
| 11 | 11 | } |
| 12 | 12 | |
| 13 | +#define clear_user_page clear_user_page | |
| 13 | 14 | static inline void clear_user_page(void *addr, unsigned long vaddr, |
| 14 | 15 | struct page *page) |
| 15 | 16 | { |
| @@ -1,11 +1,4 @@ | ||
| 1 | 1 | /* SPDX-License-Identifier: GPL-2.0 */ |
| 2 | − | |
| 3 | −static inline void clear_user_page(void *addr, unsigned long vaddr, | |
| 4 | − struct page *page) | |
| 5 | −{ | |
| 6 | − clear_page(addr); | |
| 7 | −} | |
| 8 | − | |
| 9 | 2 | static inline void copy_user_page(void *to, void *from, unsigned long vaddr, |
| 10 | 3 | struct page *page) |
| 11 | 4 | { |
| @@ -51,11 +51,18 @@ disable: · 50 unmodified lines | ||
| 51 | 51 | } |
| 52 | 52 | #endif |
| 53 | 53 | |
| 54 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 55 | +{ | |
| 56 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 57 | +#ifdef CONFIG_HIGHMEM | |
| 58 | + max_zone_pfns[ZONE_HIGHMEM] = max_pfn; | |
| 59 | +#endif | |
| 60 | +} | |
| 61 | + | |
| 54 | 62 | static void __init csky_memblock_init(void) |
| 55 | 63 | { |
| 56 | 64 | unsigned long lowmem_size = PFN_DOWN(LOWMEM_LIMIT - PHYS_OFFSET_OFFSET); |
| 57 | 65 | unsigned long sseg_size = PFN_DOWN(SSEG_SIZE - PHYS_OFFSET_OFFSET); |
| 58 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 59 | 66 | signed long size; |
| 60 | 67 | |
| 61 | 68 | memblock_reserve(__pa(_start), _end - _start); |
| @@ -83,12 +90,9 @@ static void __init csky_memblock_init(void) · 21 unmodified lines | ||
| 83 | 90 | setup_initrd(); |
| 84 | 91 | #endif |
| 85 | 92 | |
| 86 | − max_zone_pfn[ZONE_NORMAL] = max_low_pfn; | |
| 87 | − | |
| 88 | 93 | mmu_init(min_low_pfn, max_low_pfn); |
| 89 | 94 | |
| 90 | 95 | #ifdef CONFIG_HIGHMEM |
| 91 | − max_zone_pfn[ZONE_HIGHMEM] = max_pfn; | |
| 92 | 96 | |
| 93 | 97 | highstart_pfn = max_low_pfn; |
| 94 | 98 | highend_pfn = max_pfn; |
| @@ -96,8 +100,6 @@ static void __init csky_memblock_init(void) · 1 unmodified line | ||
| 96 | 100 | memblock_set_current_limit(PFN_PHYS(max_low_pfn)); |
| 97 | 101 | |
| 98 | 102 | dma_contiguous_reserve(0); |
| 99 | − | |
| 100 | − free_area_init(max_zone_pfn); | |
| 101 | 103 | } |
| 102 | 104 | |
| 103 | 105 | void __init setup_arch(char **cmdline_p) |
| @@ -121,8 +123,6 @@ void __init setup_arch(char **cmdline_p) · 17 unmodified lines | ||
| 121 | 123 | setup_smp(); |
| 122 | 124 | #endif |
| 123 | 125 | |
| 124 | − sparse_init(); | |
| 125 | − | |
| 126 | 126 | fixaddr_init(); |
| 127 | 127 | |
| 128 | 128 | #ifdef CONFIG_HIGHMEM |
| @@ -113,7 +113,6 @@ static inline void clear_page(void *page) · 112 unmodified lines | ||
| 113 | 113 | /* |
| 114 | 114 | * Under assumption that kernel always "sees" user map... |
| 115 | 115 | */ |
| 116 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 117 | 116 | #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 118 | 117 | |
| 119 | 118 | static inline unsigned long virt_to_pfn(const void *kaddr) |
| @@ -54,17 +54,8 @@ void sync_icache_dcache(pte_t pte) · 53 unmodified lines | ||
| 54 | 54 | __vmcache_idsync(addr, PAGE_SIZE); |
| 55 | 55 | } |
| 56 | 56 | |
| 57 | −/* | |
| 58 | − * In order to set up page allocator "nodes", | |
| 59 | − * somebody has to call free_area_init() for UMA. | |
| 60 | − * | |
| 61 | − * In this mode, we only have one pg_data_t | |
| 62 | − * structure: contig_mem_data. | |
| 63 | − */ | |
| 64 | −static void __init paging_init(void) | |
| 57 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 65 | 58 | { |
| 66 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = {0, }; | |
| 67 | − | |
| 68 | 59 | /* |
| 69 | 60 | * This is not particularly well documented anywhere, but |
| 70 | 61 | * give ZONE_NORMAL all the memory, including the big holes |
| @@ -72,11 +63,11 @@ static void __init paging_init(void) · 1 unmodified line | ||
| 72 | 63 | * in the bootmem_map; free_area_init should see those bits and |
| 73 | 64 | * adjust accordingly. |
| 74 | 65 | */ |
| 66 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 67 | +} | |
| 75 | 68 | |
| 76 | − max_zone_pfn[ZONE_NORMAL] = max_low_pfn; | |
| 77 | − | |
| 78 | − free_area_init(max_zone_pfn); /* sets up the zonelists and mem_map */ | |
| 79 | − | |
| 69 | +static void __init paging_init(void) | |
| 70 | +{ | |
| 80 | 71 | /* |
| 81 | 72 | * Set the init_mm descriptors "context" value to point to the |
| 82 | 73 | * initial kernel segment table's physical address. |
| @@ -187,6 +187,7 @@ config LOONGARCH · 186 unmodified lines | ||
| 187 | 187 | select IRQ_LOONGARCH_CPU |
| 188 | 188 | select LOCK_MM_AND_FIND_VMA |
| 189 | 189 | select MMU_GATHER_MERGE_VMAS if MMU |
| 190 | + select MMU_GATHER_RCU_TABLE_FREE | |
| 190 | 191 | select MODULES_USE_ELF_RELA if MODULES |
| 191 | 192 | select NEED_PER_CPU_EMBED_FIRST_CHUNK |
| 192 | 193 | select NEED_PER_CPU_PAGE_FIRST_CHUNK |
| @@ -30,7 +30,6 @@ · 29 unmodified lines | ||
| 30 | 30 | extern void clear_page(void *page); |
| 31 | 31 | extern void copy_page(void *to, void *from); |
| 32 | 32 | |
| 33 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 34 | 33 | #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 35 | 34 | |
| 36 | 35 | extern unsigned long shm_align_mask; |
| @@ -55,8 +55,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) · 54 unmodified lines | ||
| 55 | 55 | return pte; |
| 56 | 56 | } |
| 57 | 57 | |
| 58 | −#define __pte_free_tlb(tlb, pte, address) \ | |
| 59 | − tlb_remove_ptdesc((tlb), page_ptdesc(pte)) | |
| 58 | +#define __pte_free_tlb(tlb, pte, address) tlb_remove_ptdesc((tlb), page_ptdesc(pte)) | |
| 60 | 59 | |
| 61 | 60 | #ifndef __PAGETABLE_PMD_FOLDED |
| 62 | 61 | |
| @@ -79,7 +78,7 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) · 16 unmodified lines | ||
| 79 | 78 | return pmd; |
| 80 | 79 | } |
| 81 | 80 | |
| 82 | −#define __pmd_free_tlb(tlb, x, addr) pmd_free((tlb)->mm, x) | |
| 81 | +#define __pmd_free_tlb(tlb, x, addr) tlb_remove_ptdesc((tlb), virt_to_ptdesc(x)) | |
| 83 | 82 | |
| 84 | 83 | #endif |
| 85 | 84 | |
| @@ -99,7 +98,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) · 13 unmodified lines | ||
| 99 | 98 | return pud; |
| 100 | 99 | } |
| 101 | 100 | |
| 102 | −#define __pud_free_tlb(tlb, x, addr) pud_free((tlb)->mm, x) | |
| 101 | +#define __pud_free_tlb(tlb, x, addr) tlb_remove_ptdesc((tlb), virt_to_ptdesc(x)) | |
| 103 | 102 | |
| 104 | 103 | #endif /* __PAGETABLE_PUD_FOLDED */ |
| 105 | 104 | |
| @@ -353,8 +353,6 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) · 352 unmodified lines | ||
| 353 | 353 | return pte; |
| 354 | 354 | } |
| 355 | 355 | |
| 356 | −extern void paging_init(void); | |
| 357 | − | |
| 358 | 356 | #define pte_none(pte) (!(pte_val(pte) & ~_PAGE_GLOBAL)) |
| 359 | 357 | #define pte_present(pte) (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROTNONE)) |
| 360 | 358 | #define pte_no_exec(pte) (pte_val(pte) & _PAGE_NO_EXEC) |
| @@ -402,14 +402,6 @@ static void __init arch_mem_init(char **cmdline_p) · 401 unmodified lines | ||
| 402 | 402 | |
| 403 | 403 | check_kernel_sections_mem(); |
| 404 | 404 | |
| 405 | − /* | |
| 406 | − * In order to reduce the possibility of kernel panic when failed to | |
| 407 | − * get IO TLB memory under CONFIG_SWIOTLB, it is better to allocate | |
| 408 | − * low memory as small as possible before swiotlb_init(), so make | |
| 409 | − * sparse_init() using top-down allocation. | |
| 410 | − */ | |
| 411 | − memblock_set_bottom_up(false); | |
| 412 | − sparse_init(); | |
| 413 | 405 | memblock_set_bottom_up(true); |
| 414 | 406 | |
| 415 | 407 | swiotlb_init(true, SWIOTLB_VERBOSE); |
| @@ -621,8 +613,6 @@ void __init setup_arch(char **cmdline_p) · 205 unmodified lines | ||
| 621 | 613 | prefill_possible_map(); |
| 622 | 614 | #endif |
| 623 | 615 | |
| 624 | − paging_init(); | |
| 625 | − | |
| 626 | 616 | #ifdef CONFIG_KASAN |
| 627 | 617 | kasan_init(); |
| 628 | 618 | #endif |
| @@ -60,16 +60,12 @@ int __ref page_is_ram(unsigned long pfn) · 59 unmodified lines | ||
| 60 | 60 | return memblock_is_memory(addr) && !memblock_is_reserved(addr); |
| 61 | 61 | } |
| 62 | 62 | |
| 63 | −void __init paging_init(void) | |
| 63 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 64 | 64 | { |
| 65 | − unsigned long max_zone_pfns[MAX_NR_ZONES]; | |
| 66 | − | |
| 67 | 65 | #ifdef CONFIG_ZONE_DMA32 |
| 68 | 66 | max_zone_pfns[ZONE_DMA32] = MAX_DMA32_PFN; |
| 69 | 67 | #endif |
| 70 | 68 | max_zone_pfns[ZONE_NORMAL] = max_low_pfn; |
| 71 | − | |
| 72 | − free_area_init(max_zone_pfns); | |
| 73 | 69 | } |
| 74 | 70 | |
| 75 | 71 | void __ref free_initmem(void) |
| @@ -10,7 +10,6 @@ extern unsigned long memory_end; · 9 unmodified lines | ||
| 10 | 10 | #define clear_page(page) memset((page), 0, PAGE_SIZE) |
| 11 | 11 | #define copy_page(to,from) memcpy((to), (from), PAGE_SIZE) |
| 12 | 12 | |
| 13 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 14 | 13 | #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 15 | 14 | |
| 16 | 15 | #define vma_alloc_zeroed_movable_folio(vma, vaddr) \ |
| @@ -40,6 +40,11 @@ · 39 unmodified lines | ||
| 40 | 40 | void *empty_zero_page; |
| 41 | 41 | EXPORT_SYMBOL(empty_zero_page); |
| 42 | 42 | |
| 43 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 44 | +{ | |
| 45 | + max_zone_pfns[ZONE_DMA] = PFN_DOWN(memblock_end_of_DRAM()); | |
| 46 | +} | |
| 47 | + | |
| 43 | 48 | #ifdef CONFIG_MMU |
| 44 | 49 | |
| 45 | 50 | int m68k_virt_to_node_shift; |
| @@ -64,13 +69,10 @@ void __init paging_init(void) · 18 unmodified lines | ||
| 64 | 69 | * page_alloc get different views of the world. |
| 65 | 70 | */ |
| 66 | 71 | unsigned long end_mem = memory_end & PAGE_MASK; |
| 67 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0, }; | |
| 68 | 72 | |
| 69 | 73 | high_memory = (void *) end_mem; |
| 70 | 74 | |
| 71 | 75 | empty_zero_page = memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE); |
| 72 | − max_zone_pfn[ZONE_DMA] = end_mem >> PAGE_SHIFT; | |
| 73 | − free_area_init(max_zone_pfn); | |
| 74 | 76 | } |
| 75 | 77 | |
| 76 | 78 | #endif /* CONFIG_MMU */ |
| @@ -39,7 +39,6 @@ void __init paging_init(void) · 38 unmodified lines | ||
| 39 | 39 | pte_t *pg_table; |
| 40 | 40 | unsigned long address, size; |
| 41 | 41 | unsigned long next_pgtable; |
| 42 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 43 | 42 | int i; |
| 44 | 43 | |
| 45 | 44 | empty_zero_page = memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE); |
| @@ -73,8 +72,6 @@ void __init paging_init(void) · 27 unmodified lines | ||
| 73 | 72 | } |
| 74 | 73 | |
| 75 | 74 | current->mm = NULL; |
| 76 | − max_zone_pfn[ZONE_DMA] = PFN_DOWN(_ramend); | |
| 77 | − free_area_init(max_zone_pfn); | |
| 78 | 75 | } |
| 79 | 76 | |
| 80 | 77 | int cf_tlb_miss(struct pt_regs *regs, int write, int dtlb, int extension_word) |
| @@ -429,7 +429,6 @@ DECLARE_VM_GET_PAGE_PROT · 428 unmodified lines | ||
| 429 | 429 | */ |
| 430 | 430 | void __init paging_init(void) |
| 431 | 431 | { |
| 432 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0, }; | |
| 433 | 432 | unsigned long min_addr, max_addr; |
| 434 | 433 | unsigned long addr; |
| 435 | 434 | int i; |
| @@ -511,12 +510,9 @@ void __init paging_init(void) · 75 unmodified lines | ||
| 511 | 510 | set_fc(USER_DATA); |
| 512 | 511 | |
| 513 | 512 | #ifdef DEBUG |
| 514 | − printk ("before free_area_init\n"); | |
| 513 | + printk ("before node_set_state\n"); | |
| 515 | 514 | #endif |
| 516 | 515 | for (i = 0; i < m68k_num_memory; i++) |
| 517 | 516 | if (node_present_pages(i)) |
| 518 | 517 | node_set_state(i, N_NORMAL_MEMORY); |
| 519 | − | |
| 520 | − max_zone_pfn[ZONE_DMA] = memblock_end_of_DRAM(); | |
| 521 | − free_area_init(max_zone_pfn); | |
| 522 | 518 | } |
| @@ -41,7 +41,6 @@ void __init paging_init(void) · 40 unmodified lines | ||
| 41 | 41 | unsigned long address; |
| 42 | 42 | unsigned long next_pgtable; |
| 43 | 43 | unsigned long bootmem_end; |
| 44 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0, }; | |
| 45 | 44 | unsigned long size; |
| 46 | 45 | |
| 47 | 46 | empty_zero_page = memblock_alloc_or_panic(PAGE_SIZE, PAGE_SIZE); |
| @@ -80,14 +79,6 @@ void __init paging_init(void) · 32 unmodified lines | ||
| 80 | 79 | mmu_emu_init(bootmem_end); |
| 81 | 80 | |
| 82 | 81 | current->mm = NULL; |
| 83 | − | |
| 84 | − /* memory sizing is a hack stolen from motorola.c.. hope it works for us */ | |
| 85 | − max_zone_pfn[ZONE_DMA] = ((unsigned long)high_memory) >> PAGE_SHIFT; | |
| 86 | − | |
| 87 | − /* I really wish I knew why the following change made things better... -- Sam */ | |
| 88 | − free_area_init(max_zone_pfn); | |
| 89 | − | |
| 90 | − | |
| 91 | 82 | } |
| 92 | 83 | |
| 93 | 84 | static const pgprot_t protection_map[16] = { |
| @@ -45,7 +45,6 @@ typedef unsigned long pte_basic_t; · 44 unmodified lines | ||
| 45 | 45 | # define copy_page(to, from) memcpy((to), (from), PAGE_SIZE) |
| 46 | 46 | # define clear_page(pgaddr) memset((pgaddr), 0, PAGE_SIZE) |
| 47 | 47 | |
| 48 | −# define clear_user_page(pgaddr, vaddr, page) memset((pgaddr), 0, PAGE_SIZE) | |
| 49 | 48 | # define copy_user_page(vto, vfrom, vaddr, topg) \ |
| 50 | 49 | memcpy((vto), (vfrom), PAGE_SIZE) |
| 51 | 50 | |
| @@ -54,32 +54,30 @@ static void __init highmem_init(void) · 53 unmodified lines | ||
| 54 | 54 | } |
| 55 | 55 | #endif /* CONFIG_HIGHMEM */ |
| 56 | 56 | |
| 57 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 58 | +{ | |
| 59 | +#ifdef CONFIG_HIGHMEM | |
| 60 | + max_zone_pfns[ZONE_DMA] = max_low_pfn; | |
| 61 | + max_zone_pfns[ZONE_HIGHMEM] = max_pfn; | |
| 62 | +#else | |
| 63 | + max_zone_pfns[ZONE_DMA] = max_pfn; | |
| 64 | +#endif | |
| 65 | +} | |
| 66 | + | |
| 57 | 67 | /* |
| 58 | 68 | * paging_init() sets up the page tables - in fact we've already done this. |
| 59 | 69 | */ |
| 60 | 70 | static void __init paging_init(void) |
| 61 | 71 | { |
| 62 | − unsigned long zones_size[MAX_NR_ZONES]; | |
| 63 | 72 | int idx; |
| 64 | 73 | |
| 65 | 74 | /* Setup fixmaps */ |
| 66 | 75 | for (idx = 0; idx < __end_of_fixed_addresses; idx++) |
| 67 | 76 | clear_fixmap(idx); |
| 68 | 77 | |
| 69 | − /* Clean every zones */ | |
| 70 | − memset(zones_size, 0, sizeof(zones_size)); | |
| 71 | − | |
| 72 | 78 | #ifdef CONFIG_HIGHMEM |
| 73 | 79 | highmem_init(); |
| 74 | − | |
| 75 | − zones_size[ZONE_DMA] = max_low_pfn; | |
| 76 | − zones_size[ZONE_HIGHMEM] = max_pfn; | |
| 77 | −#else | |
| 78 | − zones_size[ZONE_DMA] = max_pfn; | |
| 79 | 80 | #endif |
| 80 | − | |
| 81 | − /* We don't have holes in memory map */ | |
| 82 | − free_area_init(zones_size); | |
| 83 | 81 | } |
| 84 | 82 | |
| 85 | 83 | void __init setup_memory(void) |
| @@ -99,6 +99,7 @@ config MIPS · 98 unmodified lines | ||
| 99 | 99 | select IRQ_FORCED_THREADING |
| 100 | 100 | select ISA if EISA |
| 101 | 101 | select LOCK_MM_AND_FIND_VMA |
| 102 | + select MMU_GATHER_RCU_TABLE_FREE | |
| 102 | 103 | select MODULES_USE_ELF_REL if MODULES |
| 103 | 104 | select MODULES_USE_ELF_RELA if MODULES && 64BIT |
| 104 | 105 | select PERF_USE_VMALLOC |
| @@ -90,6 +90,7 @@ static inline void clear_user_page(void *addr, unsigned long vaddr, · 89 unmodified lines | ||
| 90 | 90 | if (pages_do_alias((unsigned long) addr, vaddr & PAGE_MASK)) |
| 91 | 91 | flush_data_cache_page((unsigned long)addr); |
| 92 | 92 | } |
| 93 | +#define clear_user_page clear_user_page | |
| 93 | 94 | |
| 94 | 95 | struct vm_area_struct; |
| 95 | 96 | extern void copy_user_highpage(struct page *to, struct page *from, |
| @@ -48,8 +48,7 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) · 47 unmodified lines | ||
| 48 | 48 | extern void pgd_init(void *addr); |
| 49 | 49 | extern pgd_t *pgd_alloc(struct mm_struct *mm); |
| 50 | 50 | |
| 51 | −#define __pte_free_tlb(tlb, pte, address) \ | |
| 52 | − tlb_remove_ptdesc((tlb), page_ptdesc(pte)) | |
| 51 | +#define __pte_free_tlb(tlb, pte, address) tlb_remove_ptdesc((tlb), page_ptdesc(pte)) | |
| 53 | 52 | |
| 54 | 53 | #ifndef __PAGETABLE_PMD_FOLDED |
| 55 | 54 | |
| @@ -72,7 +71,7 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) · 16 unmodified lines | ||
| 72 | 71 | return pmd; |
| 73 | 72 | } |
| 74 | 73 | |
| 75 | −#define __pmd_free_tlb(tlb, x, addr) pmd_free((tlb)->mm, x) | |
| 74 | +#define __pmd_free_tlb(tlb, x, addr) tlb_remove_ptdesc((tlb), virt_to_ptdesc(x)) | |
| 76 | 75 | |
| 77 | 76 | #endif |
| 78 | 77 | |
| @@ -97,10 +96,8 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud) · 18 unmodified lines | ||
| 97 | 96 | set_p4d(p4d, __p4d((unsigned long)pud)); |
| 98 | 97 | } |
| 99 | 98 | |
| 100 | −#define __pud_free_tlb(tlb, x, addr) pud_free((tlb)->mm, x) | |
| 99 | +#define __pud_free_tlb(tlb, x, addr) tlb_remove_ptdesc((tlb), virt_to_ptdesc(x)) | |
| 101 | 100 | |
| 102 | 101 | #endif /* __PAGETABLE_PUD_FOLDED */ |
| 103 | 102 | |
| 104 | −extern void pagetable_init(void); | |
| 105 | − | |
| 106 | 103 | #endif /* _ASM_PGALLOC_H */ |
| @@ -56,7 +56,7 @@ extern unsigned long zero_page_mask; · 55 unmodified lines | ||
| 56 | 56 | (virt_to_page((void *)(empty_zero_page + (((unsigned long)(vaddr)) & zero_page_mask)))) |
| 57 | 57 | #define __HAVE_COLOR_ZERO_PAGE |
| 58 | 58 | |
| 59 | −extern void paging_init(void); | |
| 59 | +extern void pagetable_init(void); | |
| 60 | 60 | |
| 61 | 61 | /* |
| 62 | 62 | * Conversion functions: convert a page and protection to a page entry, |
| @@ -614,8 +614,7 @@ static void __init bootcmdline_init(void) · 613 unmodified lines | ||
| 614 | 614 | * kernel but generic memory management system is still entirely uninitialized. |
| 615 | 615 | * |
| 616 | 616 | * o bootmem_init() |
| 617 | − * o sparse_init() | |
| 618 | − * o paging_init() | |
| 617 | + * o pagetable_init() | |
| 619 | 618 | * o dma_contiguous_reserve() |
| 620 | 619 | * |
| 621 | 620 | * At this stage the bootmem allocator is ready to use. |
| @@ -665,16 +664,6 @@ static void __init arch_mem_init(char **cmdline_p) · 43 unmodified lines | ||
| 665 | 664 | mips_parse_crashkernel(); |
| 666 | 665 | device_tree_init(); |
| 667 | 666 | |
| 668 | − /* | |
| 669 | − * In order to reduce the possibility of kernel panic when failed to | |
| 670 | − * get IO TLB memory under CONFIG_SWIOTLB, it is better to allocate | |
| 671 | − * low memory as small as possible before plat_swiotlb_setup(), so | |
| 672 | − * make sparse_init() using top-down allocation. | |
| 673 | − */ | |
| 674 | − memblock_set_bottom_up(false); | |
| 675 | − sparse_init(); | |
| 676 | − memblock_set_bottom_up(true); | |
| 677 | − | |
| 678 | 667 | plat_swiotlb_setup(); |
| 679 | 668 | |
| 680 | 669 | dma_contiguous_reserve(PFN_PHYS(max_low_pfn)); |
| @@ -789,7 +778,7 @@ void __init setup_arch(char **cmdline_p) · 108 unmodified lines | ||
| 789 | 778 | prefill_possible_map(); |
| 790 | 779 | |
| 791 | 780 | cpu_cache_init(); |
| 792 | − paging_init(); | |
| 781 | + pagetable_init(); | |
| 793 | 782 | |
| 794 | 783 | memblock_dump_all(); |
| 795 | 784 | |
| @@ -154,14 +154,10 @@ static __init void prom_meminit(void) · 153 unmodified lines | ||
| 154 | 154 | } |
| 155 | 155 | } |
| 156 | 156 | |
| 157 | −void __init paging_init(void) | |
| 157 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 158 | 158 | { |
| 159 | − unsigned long zones_size[MAX_NR_ZONES] = {0, }; | |
| 160 | − | |
| 161 | − pagetable_init(); | |
| 162 | − zones_size[ZONE_DMA32] = MAX_DMA32_PFN; | |
| 163 | − zones_size[ZONE_NORMAL] = max_low_pfn; | |
| 164 | − free_area_init(zones_size); | |
| 159 | + max_zone_pfns[ZONE_DMA32] = MAX_DMA32_PFN; | |
| 160 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 165 | 161 | } |
| 166 | 162 | |
| 167 | 163 | /* All PCI device belongs to logical Node-0 */ |
| @@ -394,12 +394,8 @@ void maar_init(void) · 393 unmodified lines | ||
| 394 | 394 | } |
| 395 | 395 | |
| 396 | 396 | #ifndef CONFIG_NUMA |
| 397 | −void __init paging_init(void) | |
| 397 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 398 | 398 | { |
| 399 | − unsigned long max_zone_pfns[MAX_NR_ZONES]; | |
| 400 | − | |
| 401 | − pagetable_init(); | |
| 402 | − | |
| 403 | 399 | #ifdef CONFIG_ZONE_DMA |
| 404 | 400 | max_zone_pfns[ZONE_DMA] = MAX_DMA_PFN; |
| 405 | 401 | #endif |
| @@ -417,8 +413,6 @@ void __init paging_init(void) · 11 unmodified lines | ||
| 417 | 413 | max_zone_pfns[ZONE_HIGHMEM] = max_low_pfn; |
| 418 | 414 | } |
| 419 | 415 | #endif |
| 420 | − | |
| 421 | − free_area_init(max_zone_pfns); | |
| 422 | 416 | } |
| 423 | 417 | |
| 424 | 418 | #ifdef CONFIG_64BIT |
| @@ -406,11 +406,7 @@ void __init prom_meminit(void) · 405 unmodified lines | ||
| 406 | 406 | } |
| 407 | 407 | } |
| 408 | 408 | |
| 409 | −void __init paging_init(void) | |
| 409 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 410 | 410 | { |
| 411 | − unsigned long zones_size[MAX_NR_ZONES] = {0, }; | |
| 412 | − | |
| 413 | − pagetable_init(); | |
| 414 | − zones_size[ZONE_NORMAL] = max_low_pfn; | |
| 415 | − free_area_init(zones_size); | |
| 411 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 416 | 412 | } |
| @@ -45,6 +45,7 @@ · 44 unmodified lines | ||
| 45 | 45 | |
| 46 | 46 | struct page; |
| 47 | 47 | |
| 48 | +#define clear_user_page clear_user_page | |
| 48 | 49 | extern void clear_user_page(void *addr, unsigned long vaddr, struct page *page); |
| 49 | 50 | extern void copy_user_page(void *vto, void *vfrom, unsigned long vaddr, |
| 50 | 51 | struct page *to); |
| @@ -38,6 +38,11 @@ · 37 unmodified lines | ||
| 38 | 38 | |
| 39 | 39 | pgd_t *pgd_current; |
| 40 | 40 | |
| 41 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 42 | +{ | |
| 43 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 44 | +} | |
| 45 | + | |
| 41 | 46 | /* |
| 42 | 47 | * paging_init() continues the virtual memory environment setup which |
| 43 | 48 | * was begun by the code in arch/head.S. |
| @@ -46,16 +51,9 @@ pgd_t *pgd_current; · 2 unmodified lines | ||
| 46 | 51 | */ |
| 47 | 52 | void __init paging_init(void) |
| 48 | 53 | { |
| 49 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 50 | − | |
| 51 | 54 | pagetable_init(); |
| 52 | 55 | pgd_current = swapper_pg_dir; |
| 53 | 56 | |
| 54 | − max_zone_pfn[ZONE_NORMAL] = max_low_pfn; | |
| 55 | − | |
| 56 | − /* pass the memory from the bootmem allocator to the main allocator */ | |
| 57 | − free_area_init(max_zone_pfn); | |
| 58 | − | |
| 59 | 57 | flush_dcache_range((unsigned long)empty_zero_page, |
| 60 | 58 | (unsigned long)empty_zero_page + PAGE_SIZE); |
| 61 | 59 | } |
| @@ -30,7 +30,6 @@ · 29 unmodified lines | ||
| 30 | 30 | #define clear_page(page) memset((page), 0, PAGE_SIZE) |
| 31 | 31 | #define copy_page(to, from) memcpy((to), (from), PAGE_SIZE) |
| 32 | 32 | |
| 33 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 34 | 33 | #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 35 | 34 | |
| 36 | 35 | /* |
| @@ -39,16 +39,12 @@ · 38 unmodified lines | ||
| 39 | 39 | |
| 40 | 40 | int mem_init_done; |
| 41 | 41 | |
| 42 | −static void __init zone_sizes_init(void) | |
| 42 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 43 | 43 | { |
| 44 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 45 | − | |
| 46 | 44 | /* |
| 47 | 45 | * We use only ZONE_NORMAL |
| 48 | 46 | */ |
| 49 | − max_zone_pfn[ZONE_NORMAL] = max_low_pfn; | |
| 50 | − | |
| 51 | − free_area_init(max_zone_pfn); | |
| 47 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 52 | 48 | } |
| 53 | 49 | |
| 54 | 50 | extern const char _s_kernel_ro[], _e_kernel_ro[]; |
| @@ -141,8 +137,6 @@ void __init paging_init(void) · 86 unmodified lines | ||
| 141 | 137 | |
| 142 | 138 | map_ram(); |
| 143 | 139 | |
| 144 | − zone_sizes_init(); | |
| 145 | − | |
| 146 | 140 | /* self modifying code ;) */ |
| 147 | 141 | /* Since the old TLB miss handler has been running up until now, |
| 148 | 142 | * the kernel pages are still all RW, so we can still modify the |
| @@ -79,6 +79,7 @@ config PARISC · 78 unmodified lines | ||
| 79 | 79 | select GENERIC_CLOCKEVENTS |
| 80 | 80 | select CPU_NO_EFFICIENT_FFS |
| 81 | 81 | select THREAD_INFO_IN_TASK |
| 82 | + select MMU_GATHER_RCU_TABLE_FREE | |
| 82 | 83 | select NEED_DMA_MAP_STATE |
| 83 | 84 | select NEED_SG_DMA_LENGTH |
| 84 | 85 | select HAVE_ARCH_KGDB |
| @@ -21,7 +21,6 @@ struct vm_area_struct; · 20 unmodified lines | ||
| 21 | 21 | |
| 22 | 22 | void clear_page_asm(void *page); |
| 23 | 23 | void copy_page_asm(void *to, void *from); |
| 24 | −#define clear_user_page(vto, vaddr, page) clear_page_asm(vto) | |
| 25 | 24 | void copy_user_highpage(struct page *to, struct page *from, unsigned long vaddr, |
| 26 | 25 | struct vm_area_struct *vma); |
| 27 | 26 | #define __HAVE_ARCH_COPY_USER_HIGHPAGE |
| @@ -5,8 +5,8 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <asm-generic/tlb.h> |
| 6 | 6 | |
| 7 | 7 | #if CONFIG_PGTABLE_LEVELS == 3 |
| 8 | −#define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd) | |
| 8 | +#define __pmd_free_tlb(tlb, pmd, addr) tlb_remove_ptdesc((tlb), virt_to_ptdesc(pmd)) | |
| 9 | 9 | #endif |
| 10 | −#define __pte_free_tlb(tlb, pte, addr) pte_free((tlb)->mm, pte) | |
| 10 | +#define __pte_free_tlb(tlb, pte, addr) tlb_remove_ptdesc((tlb), page_ptdesc(pte)) | |
| 11 | 11 | |
| 12 | 12 | #endif |
| @@ -693,13 +693,9 @@ static void __init fixmap_init(void) · 692 unmodified lines | ||
| 693 | 693 | } while (addr < end); |
| 694 | 694 | } |
| 695 | 695 | |
| 696 | −static void __init parisc_bootmem_free(void) | |
| 696 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 697 | 697 | { |
| 698 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0, }; | |
| 699 | − | |
| 700 | − max_zone_pfn[0] = memblock_end_of_DRAM(); | |
| 701 | − | |
| 702 | − free_area_init(max_zone_pfn); | |
| 698 | + max_zone_pfns[ZONE_NORMAL] = PFN_DOWN(memblock_end_of_DRAM()); | |
| 703 | 699 | } |
| 704 | 700 | |
| 705 | 701 | void __init paging_init(void) |
| @@ -710,9 +706,6 @@ void __init paging_init(void) · 4 unmodified lines | ||
| 710 | 706 | fixmap_init(); |
| 711 | 707 | flush_cache_all_local(); /* start with known state */ |
| 712 | 708 | flush_tlb_all_local(NULL); |
| 713 | − | |
| 714 | − sparse_init(); | |
| 715 | − parisc_bootmem_free(); | |
| 716 | 709 | } |
| 717 | 710 | |
| 718 | 711 | static void alloc_btlb(unsigned long start, unsigned long end, int *slot, |
| @@ -172,6 +172,7 @@ config PPC · 171 unmodified lines | ||
| 172 | 172 | select ARCH_STACKWALK |
| 173 | 173 | select ARCH_SUPPORTS_ATOMIC_RMW |
| 174 | 174 | select ARCH_SUPPORTS_DEBUG_PAGEALLOC if PPC_BOOK3S || PPC_8xx |
| 175 | + select ARCH_SUPPORTS_PAGE_TABLE_CHECK if !HUGETLB_PAGE | |
| 175 | 176 | select ARCH_SUPPORTS_SCHED_MC if SMP |
| 176 | 177 | select ARCH_SUPPORTS_SCHED_SMT if PPC64 && SMP |
| 177 | 178 | select SCHED_MC if ARCH_SUPPORTS_SCHED_MC |
| @@ -304,6 +305,7 @@ config PPC · 126 unmodified lines | ||
| 304 | 305 | select LOCK_MM_AND_FIND_VMA |
| 305 | 306 | select MMU_GATHER_PAGE_SIZE |
| 306 | 307 | select MMU_GATHER_RCU_TABLE_FREE |
| 308 | + select HAVE_ARCH_TLB_REMOVE_TABLE | |
| 307 | 309 | select MMU_GATHER_MERGE_VMAS |
| 308 | 310 | select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 |
| 309 | 311 | select MODULES_USE_ELF_RELA |
| @@ -198,6 +198,7 @@ void unmap_kernel_page(unsigned long va); · 197 unmodified lines | ||
| 198 | 198 | #ifndef __ASSEMBLER__ |
| 199 | 199 | #include <linux/sched.h> |
| 200 | 200 | #include <linux/threads.h> |
| 201 | +#include <linux/page_table_check.h> | |
| 201 | 202 | |
| 202 | 203 | /* Bits to mask out from a PGD to get to the PUD page */ |
| 203 | 204 | #define PGD_MASKED_BITS 0 |
| @@ -311,7 +312,11 @@ static inline int __ptep_test_and_clear_young(struct mm_struct *mm, · 107 unmodified lines | ||
| 311 | 312 | static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, |
| 312 | 313 | pte_t *ptep) |
| 313 | 314 | { |
| 314 | − return __pte(pte_update(mm, addr, ptep, ~_PAGE_HASHPTE, 0, 0)); | |
| 315 | + pte_t old_pte = __pte(pte_update(mm, addr, ptep, ~_PAGE_HASHPTE, 0, 0)); | |
| 316 | + | |
| 317 | + page_table_check_pte_clear(mm, addr, old_pte); | |
| 318 | + | |
| 319 | + return old_pte; | |
| 315 | 320 | } |
| 316 | 321 | |
| 317 | 322 | #define __HAVE_ARCH_PTEP_SET_WRPROTECT |
| @@ -433,6 +438,11 @@ static inline bool pte_access_permitted(pte_t pte, bool write) · 115 unmodified lines | ||
| 433 | 438 | return true; |
| 434 | 439 | } |
| 435 | 440 | |
| 441 | +static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr) | |
| 442 | +{ | |
| 443 | + return pte_present(pte) && !is_kernel_addr(addr); | |
| 444 | +} | |
| 445 | + | |
| 436 | 446 | /* Conversion functions: convert a page and protection to a page entry, |
| 437 | 447 | * and a page entry and page directory to the page they refer to. |
| 438 | 448 | * |
| @@ -144,6 +144,8 @@ · 143 unmodified lines | ||
| 144 | 144 | #define PAGE_KERNEL_ROX __pgprot(_PAGE_BASE | _PAGE_KERNEL_ROX) |
| 145 | 145 | |
| 146 | 146 | #ifndef __ASSEMBLER__ |
| 147 | +#include <linux/page_table_check.h> | |
| 148 | + | |
| 147 | 149 | /* |
| 148 | 150 | * page table defines |
| 149 | 151 | */ |
| @@ -416,8 +418,11 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm, · 266 unmodified lines | ||
| 416 | 418 | static inline pte_t ptep_get_and_clear(struct mm_struct *mm, |
| 417 | 419 | unsigned long addr, pte_t *ptep) |
| 418 | 420 | { |
| 419 | − unsigned long old = pte_update(mm, addr, ptep, ~0UL, 0, 0); | |
| 420 | − return __pte(old); | |
| 421 | + pte_t old_pte = __pte(pte_update(mm, addr, ptep, ~0UL, 0, 0)); | |
| 422 | + | |
| 423 | + page_table_check_pte_clear(mm, addr, old_pte); | |
| 424 | + | |
| 425 | + return old_pte; | |
| 421 | 426 | } |
| 422 | 427 | |
| 423 | 428 | #define __HAVE_ARCH_PTEP_GET_AND_CLEAR_FULL |
| @@ -426,11 +431,16 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · 2 unmodified lines | ||
| 426 | 431 | pte_t *ptep, int full) |
| 427 | 432 | { |
| 428 | 433 | if (full && radix_enabled()) { |
| 434 | + pte_t old_pte; | |
| 435 | + | |
| 429 | 436 | /* |
| 430 | 437 | * We know that this is a full mm pte clear and |
| 431 | 438 | * hence can be sure there is no parallel set_pte. |
| 432 | 439 | */ |
| 433 | − return radix__ptep_get_and_clear_full(mm, addr, ptep, full); | |
| 440 | + old_pte = radix__ptep_get_and_clear_full(mm, addr, ptep, full); | |
| 441 | + page_table_check_pte_clear(mm, addr, old_pte); | |
| 442 | + | |
| 443 | + return old_pte; | |
| 434 | 444 | } |
| 435 | 445 | return ptep_get_and_clear(mm, addr, ptep); |
| 436 | 446 | } |
| @@ -539,6 +549,11 @@ static inline bool pte_access_permitted(pte_t pte, bool write) · 102 unmodified lines | ||
| 539 | 549 | return arch_pte_access_permitted(pte_val(pte), write, 0); |
| 540 | 550 | } |
| 541 | 551 | |
| 552 | +static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr) | |
| 553 | +{ | |
| 554 | + return pte_present(pte) && pte_user(pte); | |
| 555 | +} | |
| 556 | + | |
| 542 | 557 | /* |
| 543 | 558 | * Conversion functions: convert a page and protection to a page entry, |
| 544 | 559 | * and a page entry and page directory to the page they refer to. |
| @@ -909,6 +924,12 @@ static inline bool pud_access_permitted(pud_t pud, bool write) · 364 unmodified lines | ||
| 909 | 924 | return pte_access_permitted(pud_pte(pud), write); |
| 910 | 925 | } |
| 911 | 926 | |
| 927 | +#define pud_user_accessible_page pud_user_accessible_page | |
| 928 | +static inline bool pud_user_accessible_page(pud_t pud, unsigned long addr) | |
| 929 | +{ | |
| 930 | + return pud_leaf(pud) && pte_user_accessible_page(pud_pte(pud), addr); | |
| 931 | +} | |
| 932 | + | |
| 912 | 933 | #define __p4d_raw(x) ((p4d_t) { __pgd_raw(x) }) |
| 913 | 934 | static inline __be64 p4d_raw(p4d_t x) |
| 914 | 935 | { |
| @@ -1074,6 +1095,12 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write) · 159 unmodified lines | ||
| 1074 | 1095 | return pte_access_permitted(pmd_pte(pmd), write); |
| 1075 | 1096 | } |
| 1076 | 1097 | |
| 1098 | +#define pmd_user_accessible_page pmd_user_accessible_page | |
| 1099 | +static inline bool pmd_user_accessible_page(pmd_t pmd, unsigned long addr) | |
| 1100 | +{ | |
| 1101 | + return pmd_leaf(pmd) && pte_user_accessible_page(pmd_pte(pmd), addr); | |
| 1102 | +} | |
| 1103 | + | |
| 1077 | 1104 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| 1078 | 1105 | extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); |
| 1079 | 1106 | extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); |
| @@ -1284,19 +1311,34 @@ extern int pudp_test_and_clear_young(struct vm_area_struct *vma, · 204 unmodified lines | ||
| 1284 | 1311 | static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, |
| 1285 | 1312 | unsigned long addr, pmd_t *pmdp) |
| 1286 | 1313 | { |
| 1287 | − if (radix_enabled()) | |
| 1288 | − return radix__pmdp_huge_get_and_clear(mm, addr, pmdp); | |
| 1289 | − return hash__pmdp_huge_get_and_clear(mm, addr, pmdp); | |
| 1314 | + pmd_t old_pmd; | |
| 1315 | + | |
| 1316 | + if (radix_enabled()) { | |
| 1317 | + old_pmd = radix__pmdp_huge_get_and_clear(mm, addr, pmdp); | |
| 1318 | + } else { | |
| 1319 | + old_pmd = hash__pmdp_huge_get_and_clear(mm, addr, pmdp); | |
| 1320 | + } | |
| 1321 | + | |
| 1322 | + page_table_check_pmd_clear(mm, addr, old_pmd); | |
| 1323 | + | |
| 1324 | + return old_pmd; | |
| 1290 | 1325 | } |
| 1291 | 1326 | |
| 1292 | 1327 | #define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR |
| 1293 | 1328 | static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, |
| 1294 | 1329 | unsigned long addr, pud_t *pudp) |
| 1295 | 1330 | { |
| 1296 | − if (radix_enabled()) | |
| 1297 | − return radix__pudp_huge_get_and_clear(mm, addr, pudp); | |
| 1298 | − BUG(); | |
| 1299 | − return *pudp; | |
| 1331 | + pud_t old_pud; | |
| 1332 | + | |
| 1333 | + if (radix_enabled()) { | |
| 1334 | + old_pud = radix__pudp_huge_get_and_clear(mm, addr, pudp); | |
| 1335 | + } else { | |
| 1336 | + BUG(); | |
| 1337 | + } | |
| 1338 | + | |
| 1339 | + page_table_check_pud_clear(mm, addr, old_pud); | |
| 1340 | + | |
| 1341 | + return old_pud; | |
| 1300 | 1342 | } |
| 1301 | 1343 | |
| 1302 | 1344 | static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, |
| @@ -12,7 +12,6 @@ · 11 unmodified lines | ||
| 12 | 12 | #define PPC64_TLB_BATCH_NR 192 |
| 13 | 13 | |
| 14 | 14 | struct ppc64_tlb_batch { |
| 15 | − int active; | |
| 16 | 15 | unsigned long index; |
| 17 | 16 | struct mm_struct *mm; |
| 18 | 17 | real_pte_t pte[PPC64_TLB_BATCH_NR]; |
| @@ -24,12 +23,8 @@ DECLARE_PER_CPU(struct ppc64_tlb_batch, ppc64_tlb_batch); · 5 unmodified lines | ||
| 24 | 23 | |
| 25 | 24 | extern void __flush_tlb_pending(struct ppc64_tlb_batch *batch); |
| 26 | 25 | |
| 27 | −#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE | |
| 28 | − | |
| 29 | 26 | static inline void arch_enter_lazy_mmu_mode(void) |
| 30 | 27 | { |
| 31 | − struct ppc64_tlb_batch *batch; | |
| 32 | − | |
| 33 | 28 | if (radix_enabled()) |
| 34 | 29 | return; |
| 35 | 30 | /* |
| @@ -37,11 +32,9 @@ static inline void arch_enter_lazy_mmu_mode(void) · 1 unmodified line | ||
| 37 | 32 | * operating on kernel page tables. |
| 38 | 33 | */ |
| 39 | 34 | preempt_disable(); |
| 40 | − batch = this_cpu_ptr(&ppc64_tlb_batch); | |
| 41 | − batch->active = 1; | |
| 42 | 35 | } |
| 43 | 36 | |
| 44 | −static inline void arch_leave_lazy_mmu_mode(void) | |
| 37 | +static inline void arch_flush_lazy_mmu_mode(void) | |
| 45 | 38 | { |
| 46 | 39 | struct ppc64_tlb_batch *batch; |
| 47 | 40 | |
| @@ -51,11 +44,16 @@ static inline void arch_leave_lazy_mmu_mode(void) · 3 unmodified lines | ||
| 51 | 44 | |
| 52 | 45 | if (batch->index) |
| 53 | 46 | __flush_tlb_pending(batch); |
| 54 | − batch->active = 0; | |
| 55 | − preempt_enable(); | |
| 56 | 47 | } |
| 57 | 48 | |
| 58 | −#define arch_flush_lazy_mmu_mode() do {} while (0) | |
| 49 | +static inline void arch_leave_lazy_mmu_mode(void) | |
| 50 | +{ | |
| 51 | + if (radix_enabled()) | |
| 52 | + return; | |
| 53 | + | |
| 54 | + arch_flush_lazy_mmu_mode(); | |
| 55 | + preempt_enable(); | |
| 56 | +} | |
| 59 | 57 | |
| 60 | 58 | extern void hash__tlbiel_all(unsigned int action); |
| 61 | 59 | |
| @@ -68,7 +68,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 67 unmodified lines | ||
| 68 | 68 | unsigned long addr, pte_t *ptep, |
| 69 | 69 | pte_t pte, int dirty); |
| 70 | 70 | |
| 71 | −void gigantic_hugetlb_cma_reserve(void) __init; | |
| 72 | 71 | #include <asm-generic/hugetlb.h> |
| 73 | 72 | |
| 74 | 73 | #else /* ! CONFIG_HUGETLB_PAGE */ |
| @@ -77,10 +76,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, · 2 unmodified lines | ||
| 77 | 76 | { |
| 78 | 77 | } |
| 79 | 78 | |
| 80 | −static inline void __init gigantic_hugetlb_cma_reserve(void) | |
| 81 | −{ | |
| 82 | −} | |
| 83 | − | |
| 84 | 79 | static inline void __init hugetlbpage_init_defaultsize(void) |
| 85 | 80 | { |
| 86 | 81 | } |
| @@ -29,6 +29,8 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p · 28 unmodified lines | ||
| 29 | 29 | |
| 30 | 30 | #ifndef __ASSEMBLER__ |
| 31 | 31 | |
| 32 | +#include <linux/page_table_check.h> | |
| 33 | + | |
| 32 | 34 | extern int icache_44x_need_flush; |
| 33 | 35 | |
| 34 | 36 | #ifndef pte_huge_size |
| @@ -122,7 +124,11 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, · 87 unmodified lines | ||
| 122 | 124 | static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, |
| 123 | 125 | pte_t *ptep) |
| 124 | 126 | { |
| 125 | − return __pte(pte_update(mm, addr, ptep, ~0UL, 0, 0)); | |
| 127 | + pte_t old_pte = __pte(pte_update(mm, addr, ptep, ~0UL, 0, 0)); | |
| 128 | + | |
| 129 | + page_table_check_pte_clear(mm, addr, old_pte); | |
| 130 | + | |
| 131 | + return old_pte; | |
| 126 | 132 | } |
| 127 | 133 | #define __HAVE_ARCH_PTEP_GET_AND_CLEAR |
| 128 | 134 | |
| @@ -243,6 +249,11 @@ static inline bool pte_access_permitted(pte_t pte, bool write) · 114 unmodified lines | ||
| 243 | 249 | return true; |
| 244 | 250 | } |
| 245 | 251 | |
| 252 | +static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr) | |
| 253 | +{ | |
| 254 | + return pte_present(pte) && !is_kernel_addr(addr); | |
| 255 | +} | |
| 256 | + | |
| 246 | 257 | /* Conversion functions: convert a page and protection to a page entry, |
| 247 | 258 | * and a page entry and page directory to the page they refer to. |
| 248 | 259 | * |
| @@ -271,6 +271,7 @@ static inline const void *pfn_to_kaddr(unsigned long pfn) · 270 unmodified lines | ||
| 271 | 271 | |
| 272 | 272 | struct page; |
| 273 | 273 | extern void clear_user_page(void *page, unsigned long vaddr, struct page *pg); |
| 274 | +#define clear_user_page clear_user_page | |
| 274 | 275 | extern void copy_user_page(void *to, void *from, unsigned long vaddr, |
| 275 | 276 | struct page *p); |
| 276 | 277 | extern int devmem_is_allowed(unsigned long pfn); |
| @@ -34,6 +34,8 @@ struct mm_struct; · 33 unmodified lines | ||
| 34 | 34 | void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, |
| 35 | 35 | pte_t pte, unsigned int nr); |
| 36 | 36 | #define set_ptes set_ptes |
| 37 | +void set_pte_at_unchecked(struct mm_struct *mm, unsigned long addr, | |
| 38 | + pte_t *ptep, pte_t pte); | |
| 37 | 39 | #define update_mmu_cache(vma, addr, ptep) \ |
| 38 | 40 | update_mmu_cache_range(NULL, vma, addr, ptep, 1) |
| 39 | 41 | |
| @@ -202,6 +204,14 @@ static inline bool arch_supports_memmap_on_memory(unsigned long vmemmap_size) · 162 unmodified lines | ||
| 202 | 204 | |
| 203 | 205 | #endif /* CONFIG_PPC64 */ |
| 204 | 206 | |
| 207 | +#ifndef pmd_user_accessible_page | |
| 208 | +#define pmd_user_accessible_page(pmd, addr) false | |
| 209 | +#endif | |
| 210 | + | |
| 211 | +#ifndef pud_user_accessible_page | |
| 212 | +#define pud_user_accessible_page(pud, addr) false | |
| 213 | +#endif | |
| 214 | + | |
| 205 | 215 | #endif /* __ASSEMBLER__ */ |
| 206 | 216 | |
| 207 | 217 | #endif /* _ASM_POWERPC_PGTABLE_H */ |
| @@ -20,7 +20,11 @@ extern void reloc_got2(unsigned long); · 19 unmodified lines | ||
| 20 | 20 | |
| 21 | 21 | void check_for_initrd(void); |
| 22 | 22 | void mem_topology_setup(void); |
| 23 | +#ifdef CONFIG_NUMA | |
| 23 | 24 | void initmem_init(void); |
| 25 | +#else | |
| 26 | +static inline void initmem_init(void) {} | |
| 27 | +#endif | |
| 24 | 28 | void setup_panic(void); |
| 25 | 29 | #define ARCH_PANIC_TIMEOUT 180 |
| 26 | 30 | |
| @@ -154,12 +154,10 @@ void arch_setup_new_exec(void); · 153 unmodified lines | ||
| 154 | 154 | /* Don't move TLF_NAPPING without adjusting the code in entry_32.S */ |
| 155 | 155 | #define TLF_NAPPING 0 /* idle thread enabled NAP mode */ |
| 156 | 156 | #define TLF_SLEEPING 1 /* suspend code enabled SLEEP mode */ |
| 157 | −#define TLF_LAZY_MMU 3 /* tlb_batch is active */ | |
| 158 | 157 | #define TLF_RUNLATCH 4 /* Is the runlatch enabled? */ |
| 159 | 158 | |
| 160 | 159 | #define _TLF_NAPPING (1 << TLF_NAPPING) |
| 161 | 160 | #define _TLF_SLEEPING (1 << TLF_SLEEPING) |
| 162 | −#define _TLF_LAZY_MMU (1 << TLF_LAZY_MMU) | |
| 163 | 161 | #define _TLF_RUNLATCH (1 << TLF_RUNLATCH) |
| 164 | 162 | |
| 165 | 163 | #ifndef __ASSEMBLER__ |
| @@ -37,7 +37,6 @@ extern void tlb_flush(struct mmu_gather *tlb); · 36 unmodified lines | ||
| 37 | 37 | */ |
| 38 | 38 | #define tlb_needs_table_invalidate() radix_enabled() |
| 39 | 39 | |
| 40 | −#define __HAVE_ARCH_TLB_REMOVE_TABLE | |
| 41 | 40 | /* Get the generic bits... */ |
| 42 | 41 | #include <asm-generic/tlb.h> |
| 43 | 42 | |
| @@ -1281,9 +1281,6 @@ struct task_struct *__switch_to(struct task_struct *prev, · 1280 unmodified lines | ||
| 1281 | 1281 | { |
| 1282 | 1282 | struct thread_struct *new_thread, *old_thread; |
| 1283 | 1283 | struct task_struct *last; |
| 1284 | −#ifdef CONFIG_PPC_64S_HASH_MMU | |
| 1285 | − struct ppc64_tlb_batch *batch; | |
| 1286 | −#endif | |
| 1287 | 1284 | |
| 1288 | 1285 | new_thread = &new->thread; |
| 1289 | 1286 | old_thread = ¤t->thread; |
| @@ -1291,14 +1288,6 @@ struct task_struct *__switch_to(struct task_struct *prev, · 1 unmodified line | ||
| 1291 | 1288 | WARN_ON(!irqs_disabled()); |
| 1292 | 1289 | |
| 1293 | 1290 | #ifdef CONFIG_PPC_64S_HASH_MMU |
| 1294 | − batch = this_cpu_ptr(&ppc64_tlb_batch); | |
| 1295 | − if (batch->active) { | |
| 1296 | − current_thread_info()->local_flags |= _TLF_LAZY_MMU; | |
| 1297 | − if (batch->index) | |
| 1298 | − __flush_tlb_pending(batch); | |
| 1299 | − batch->active = 0; | |
| 1300 | − } | |
| 1301 | − | |
| 1302 | 1291 | /* |
| 1303 | 1292 | * On POWER9 the copy-paste buffer can only paste into |
| 1304 | 1293 | * foreign real addresses, so unprivileged processes can not |
| @@ -1369,20 +1358,6 @@ struct task_struct *__switch_to(struct task_struct *prev, · 64 unmodified lines | ||
| 1369 | 1358 | */ |
| 1370 | 1359 | |
| 1371 | 1360 | #ifdef CONFIG_PPC_BOOK3S_64 |
| 1372 | −#ifdef CONFIG_PPC_64S_HASH_MMU | |
| 1373 | − /* | |
| 1374 | − * This applies to a process that was context switched while inside | |
| 1375 | − * arch_enter_lazy_mmu_mode(), to re-activate the batch that was | |
| 1376 | − * deactivated above, before _switch(). This will never be the case | |
| 1377 | − * for new tasks. | |
| 1378 | − */ | |
| 1379 | − if (current_thread_info()->local_flags & _TLF_LAZY_MMU) { | |
| 1380 | − current_thread_info()->local_flags &= ~_TLF_LAZY_MMU; | |
| 1381 | − batch = this_cpu_ptr(&ppc64_tlb_batch); | |
| 1382 | − batch->active = 1; | |
| 1383 | − } | |
| 1384 | −#endif | |
| 1385 | − | |
| 1386 | 1361 | /* |
| 1387 | 1362 | * Math facilities are masked out of the child MSR in copy_thread. |
| 1388 | 1363 | * A new task does not need to restore_math because it will |
| @@ -1003,7 +1003,6 @@ void __init setup_arch(char **cmdline_p) · 1002 unmodified lines | ||
| 1003 | 1003 | fadump_cma_init(); |
| 1004 | 1004 | kdump_cma_reserve(); |
| 1005 | 1005 | kvm_cma_reserve(); |
| 1006 | − gigantic_hugetlb_cma_reserve(); | |
| 1007 | 1006 | |
| 1008 | 1007 | early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT); |
| 1009 | 1008 | |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/sched.h> |
| 9 | 9 | #include <linux/mm_types.h> |
| 10 | 10 | #include <linux/mm.h> |
| 11 | +#include <linux/page_table_check.h> | |
| 11 | 12 | #include <linux/stop_machine.h> |
| 12 | 13 | |
| 13 | 14 | #include <asm/sections.h> |
| @@ -230,6 +231,9 @@ pmd_t hash__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long addres · 216 unmodified lines | ||
| 230 | 231 | |
| 231 | 232 | pmd = *pmdp; |
| 232 | 233 | pmd_clear(pmdp); |
| 234 | + | |
| 235 | + page_table_check_pmd_clear(vma->vm_mm, address, pmd); | |
| 236 | + | |
| 233 | 237 | /* |
| 234 | 238 | * Wait for all pending hash_page to finish. This is needed |
| 235 | 239 | * in case of subpage collapse. When we collapse normal pages |
| @@ -25,11 +25,12 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <asm/tlb.h> |
| 26 | 26 | #include <asm/bug.h> |
| 27 | 27 | #include <asm/pte-walk.h> |
| 28 | − | |
| 28 | +#include <kunit/visibility.h> | |
| 29 | 29 | |
| 30 | 30 | #include <trace/events/thp.h> |
| 31 | 31 | |
| 32 | 32 | DEFINE_PER_CPU(struct ppc64_tlb_batch, ppc64_tlb_batch); |
| 33 | +EXPORT_SYMBOL_IF_KUNIT(ppc64_tlb_batch); | |
| 33 | 34 | |
| 34 | 35 | /* |
| 35 | 36 | * A linux PTE was changed and the corresponding hash table entry |
| @@ -100,7 +101,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 64 unmodified lines | ||
| 100 | 101 | * Check if we have an active batch on this CPU. If not, just |
| 101 | 102 | * flush now and return. |
| 102 | 103 | */ |
| 103 | − if (!batch->active) { | |
| 104 | + if (!is_lazy_mmu_mode_active()) { | |
| 104 | 105 | flush_hash_page(vpn, rpte, psize, ssize, mm_is_thread_local(mm)); |
| 105 | 106 | put_cpu_var(ppc64_tlb_batch); |
| 106 | 107 | return; |
| @@ -154,6 +155,7 @@ void __flush_tlb_pending(struct ppc64_tlb_batch *batch) · 47 unmodified lines | ||
| 154 | 155 | flush_hash_range(i, local); |
| 155 | 156 | batch->index = 0; |
| 156 | 157 | } |
| 158 | +EXPORT_SYMBOL_IF_KUNIT(__flush_tlb_pending); | |
| 157 | 159 | |
| 158 | 160 | void hash__tlb_flush(struct mmu_gather *tlb) |
| 159 | 161 | { |
| @@ -205,7 +207,7 @@ void __flush_hash_table_range(unsigned long start, unsigned long end) · 45 unmodified lines | ||
| 205 | 207 | * way to do things but is fine for our needs here. |
| 206 | 208 | */ |
| 207 | 209 | local_irq_save(flags); |
| 208 | − arch_enter_lazy_mmu_mode(); | |
| 210 | + lazy_mmu_mode_enable(); | |
| 209 | 211 | for (; start < end; start += PAGE_SIZE) { |
| 210 | 212 | pte_t *ptep = find_init_mm_pte(start, &hugepage_shift); |
| 211 | 213 | unsigned long pte; |
| @@ -217,7 +219,7 @@ void __flush_hash_table_range(unsigned long start, unsigned long end) · 5 unmodified lines | ||
| 217 | 219 | continue; |
| 218 | 220 | hpte_need_flush(&init_mm, start, ptep, pte, hugepage_shift); |
| 219 | 221 | } |
| 220 | − arch_leave_lazy_mmu_mode(); | |
| 222 | + lazy_mmu_mode_disable(); | |
| 221 | 223 | local_irq_restore(flags); |
| 222 | 224 | } |
| 223 | 225 | |
| @@ -237,7 +239,7 @@ void flush_hash_table_pmd_range(struct mm_struct *mm, pmd_t *pmd, unsigned long · 13 unmodified lines | ||
| 237 | 239 | * way to do things but is fine for our needs here. |
| 238 | 240 | */ |
| 239 | 241 | local_irq_save(flags); |
| 240 | − arch_enter_lazy_mmu_mode(); | |
| 242 | + lazy_mmu_mode_enable(); | |
| 241 | 243 | start_pte = pte_offset_map(pmd, addr); |
| 242 | 244 | if (!start_pte) |
| 243 | 245 | goto out; |
| @@ -249,6 +251,6 @@ void flush_hash_table_pmd_range(struct mm_struct *mm, pmd_t *pmd, unsigned long · 5 unmodified lines | ||
| 249 | 251 | } |
| 250 | 252 | pte_unmap(start_pte); |
| 251 | 253 | out: |
| 252 | − arch_leave_lazy_mmu_mode(); | |
| 254 | + lazy_mmu_mode_disable(); | |
| 253 | 255 | local_irq_restore(flags); |
| 254 | 256 | } |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/pkeys.h> |
| 11 | 11 | #include <linux/debugfs.h> |
| 12 | 12 | #include <linux/proc_fs.h> |
| 13 | +#include <linux/page_table_check.h> | |
| 13 | 14 | |
| 14 | 15 | #include <asm/pgalloc.h> |
| 15 | 16 | #include <asm/tlb.h> |
| @@ -127,7 +128,8 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr, · 111 unmodified lines | ||
| 127 | 128 | WARN_ON(!(pmd_leaf(pmd))); |
| 128 | 129 | #endif |
| 129 | 130 | trace_hugepage_set_pmd(addr, pmd_val(pmd)); |
| 130 | − return set_pte_at(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd)); | |
| 131 | + page_table_check_pmd_set(mm, addr, pmdp, pmd); | |
| 132 | + return set_pte_at_unchecked(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd)); | |
| 131 | 133 | } |
| 132 | 134 | |
| 133 | 135 | void set_pud_at(struct mm_struct *mm, unsigned long addr, |
| @@ -144,7 +146,8 @@ void set_pud_at(struct mm_struct *mm, unsigned long addr, · 10 unmodified lines | ||
| 144 | 146 | WARN_ON(!(pud_leaf(pud))); |
| 145 | 147 | #endif |
| 146 | 148 | trace_hugepage_set_pud(addr, pud_val(pud)); |
| 147 | − return set_pte_at(mm, addr, pudp_ptep(pudp), pud_pte(pud)); | |
| 149 | + page_table_check_pud_set(mm, addr, pudp, pud); | |
| 150 | + return set_pte_at_unchecked(mm, addr, pudp_ptep(pudp), pud_pte(pud)); | |
| 148 | 151 | } |
| 149 | 152 | |
| 150 | 153 | static void do_serialize(void *arg) |
| @@ -179,23 +182,27 @@ void serialize_against_pte_lookup(struct mm_struct *mm) · 28 unmodified lines | ||
| 179 | 182 | pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, |
| 180 | 183 | pmd_t *pmdp) |
| 181 | 184 | { |
| 182 | − unsigned long old_pmd; | |
| 185 | + pmd_t old_pmd; | |
| 183 | 186 | |
| 184 | 187 | VM_WARN_ON_ONCE(!pmd_present(*pmdp)); |
| 185 | − old_pmd = pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, _PAGE_INVALID); | |
| 188 | + old_pmd = __pmd(pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, _PAGE_INVALID)); | |
| 186 | 189 | flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); |
| 187 | − return __pmd(old_pmd); | |
| 190 | + page_table_check_pmd_clear(vma->vm_mm, address, old_pmd); | |
| 191 | + | |
| 192 | + return old_pmd; | |
| 188 | 193 | } |
| 189 | 194 | |
| 190 | 195 | pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, |
| 191 | 196 | pud_t *pudp) |
| 192 | 197 | { |
| 193 | − unsigned long old_pud; | |
| 198 | + pud_t old_pud; | |
| 194 | 199 | |
| 195 | 200 | VM_WARN_ON_ONCE(!pud_present(*pudp)); |
| 196 | − old_pud = pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID); | |
| 201 | + old_pud = __pud(pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID)); | |
| 197 | 202 | flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); |
| 198 | − return __pud(old_pud); | |
| 203 | + page_table_check_pud_clear(vma->vm_mm, address, old_pud); | |
| 204 | + | |
| 205 | + return old_pud; | |
| 199 | 206 | } |
| 200 | 207 | |
| 201 | 208 | pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, |
| @@ -550,7 +557,7 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, · 348 unmodified lines | ||
| 550 | 557 | if (radix_enabled()) |
| 551 | 558 | return radix__ptep_modify_prot_commit(vma, addr, |
| 552 | 559 | ptep, old_pte, pte); |
| 553 | − set_pte_at(vma->vm_mm, addr, ptep, pte); | |
| 560 | + set_pte_at_unchecked(vma->vm_mm, addr, ptep, pte); | |
| 554 | 561 | } |
| 555 | 562 | |
| 556 | 563 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| @@ -14,6 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <linux/of.h> |
| 15 | 15 | #include <linux/of_fdt.h> |
| 16 | 16 | #include <linux/mm.h> |
| 17 | +#include <linux/page_table_check.h> | |
| 17 | 18 | #include <linux/hugetlb.h> |
| 18 | 19 | #include <linux/string_helpers.h> |
| 19 | 20 | #include <linux/memory.h> |
| @@ -1474,6 +1475,8 @@ pmd_t radix__pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long addre · 1454 unmodified lines | ||
| 1474 | 1475 | pmd = *pmdp; |
| 1475 | 1476 | pmd_clear(pmdp); |
| 1476 | 1477 | |
| 1478 | + page_table_check_pmd_clear(vma->vm_mm, address, pmd); | |
| 1479 | + | |
| 1477 | 1480 | radix__flush_tlb_collapsed_pmd(vma->vm_mm, address); |
| 1478 | 1481 | |
| 1479 | 1482 | return pmd; |
| @@ -1606,7 +1609,7 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, · 126 unmodified lines | ||
| 1606 | 1609 | (atomic_read(&mm->context.copros) > 0)) |
| 1607 | 1610 | radix__flush_tlb_page(vma, addr); |
| 1608 | 1611 | |
| 1609 | − set_pte_at(mm, addr, ptep, pte); | |
| 1612 | + set_pte_at_unchecked(mm, addr, ptep, pte); | |
| 1610 | 1613 | } |
| 1611 | 1614 | |
| 1612 | 1615 | int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) |
| @@ -1617,7 +1620,7 @@ int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) · 4 unmodified lines | ||
| 1617 | 1620 | if (!radix_enabled()) |
| 1618 | 1621 | return 0; |
| 1619 | 1622 | |
| 1620 | − set_pte_at(&init_mm, 0 /* radix unused */, ptep, new_pud); | |
| 1623 | + set_pte_at_unchecked(&init_mm, 0 /* radix unused */, ptep, new_pud); | |
| 1621 | 1624 | |
| 1622 | 1625 | return 1; |
| 1623 | 1626 | } |
| @@ -1664,7 +1667,7 @@ int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot) · 40 unmodified lines | ||
| 1664 | 1667 | if (!radix_enabled()) |
| 1665 | 1668 | return 0; |
| 1666 | 1669 | |
| 1667 | − set_pte_at(&init_mm, 0 /* radix unused */, ptep, new_pmd); | |
| 1670 | + set_pte_at_unchecked(&init_mm, 0 /* radix unused */, ptep, new_pmd); | |
| 1668 | 1671 | |
| 1669 | 1672 | return 1; |
| 1670 | 1673 | } |
| @@ -73,13 +73,13 @@ static void hpte_flush_range(struct mm_struct *mm, unsigned long addr, · 72 unmodified lines | ||
| 73 | 73 | pte = pte_offset_map_lock(mm, pmd, addr, &ptl); |
| 74 | 74 | if (!pte) |
| 75 | 75 | return; |
| 76 | − arch_enter_lazy_mmu_mode(); | |
| 76 | + lazy_mmu_mode_enable(); | |
| 77 | 77 | for (; npages > 0; --npages) { |
| 78 | 78 | pte_update(mm, addr, pte, 0, 0, 0); |
| 79 | 79 | addr += PAGE_SIZE; |
| 80 | 80 | ++pte; |
| 81 | 81 | } |
| 82 | − arch_leave_lazy_mmu_mode(); | |
| 82 | + lazy_mmu_mode_disable(); | |
| 83 | 83 | pte_unmap_unlock(pte - 1, ptl); |
| 84 | 84 | } |
| 85 | 85 | |
| @@ -200,18 +200,15 @@ static int __init hugetlbpage_init(void) · 199 unmodified lines | ||
| 200 | 200 | |
| 201 | 201 | arch_initcall(hugetlbpage_init); |
| 202 | 202 | |
| 203 | −void __init gigantic_hugetlb_cma_reserve(void) | |
| 203 | +unsigned int __init arch_hugetlb_cma_order(void) | |
| 204 | 204 | { |
| 205 | − unsigned long order = 0; | |
| 206 | − | |
| 207 | 205 | if (radix_enabled()) |
| 208 | − order = PUD_SHIFT - PAGE_SHIFT; | |
| 206 | + return PUD_SHIFT - PAGE_SHIFT; | |
| 209 | 207 | else if (!firmware_has_feature(FW_FEATURE_LPAR) && mmu_psize_defs[MMU_PAGE_16G].shift) |
| 210 | 208 | /* |
| 211 | 209 | * For pseries we do use ibm,expected#pages for reserving 16G pages. |
| 212 | 210 | */ |
| 213 | − order = mmu_psize_to_shift(MMU_PAGE_16G) - PAGE_SHIFT; | |
| 211 | + return mmu_psize_to_shift(MMU_PAGE_16G) - PAGE_SHIFT; | |
| 214 | 212 | |
| 215 | − if (order) | |
| 216 | − hugetlb_cma_reserve(order); | |
| 213 | + return 0; | |
| 217 | 214 | } |
| @@ -182,11 +182,6 @@ void __init mem_topology_setup(void) · 181 unmodified lines | ||
| 182 | 182 | memblock_set_node(0, PHYS_ADDR_MAX, &memblock.memory, 0); |
| 183 | 183 | } |
| 184 | 184 | |
| 185 | −void __init initmem_init(void) | |
| 186 | −{ | |
| 187 | − sparse_init(); | |
| 188 | −} | |
| 189 | − | |
| 190 | 185 | /* mark pages that don't exist as nosave */ |
| 191 | 186 | static int __init mark_nonram_nosave(void) |
| 192 | 187 | { |
| @@ -221,7 +216,16 @@ static int __init mark_nonram_nosave(void) · 28 unmodified lines | ||
| 221 | 216 | * anyway) will take a first dip into ZONE_NORMAL and get otherwise served by |
| 222 | 217 | * ZONE_DMA. |
| 223 | 218 | */ |
| 224 | −static unsigned long max_zone_pfns[MAX_NR_ZONES]; | |
| 219 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 220 | +{ | |
| 221 | +#ifdef CONFIG_ZONE_DMA | |
| 222 | + max_zone_pfns[ZONE_DMA] = min((zone_dma_limit >> PAGE_SHIFT) + 1, max_low_pfn); | |
| 223 | +#endif | |
| 224 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 225 | +#ifdef CONFIG_HIGHMEM | |
| 226 | + max_zone_pfns[ZONE_HIGHMEM] = max_pfn; | |
| 227 | +#endif | |
| 228 | +} | |
| 225 | 229 | |
| 226 | 230 | /* |
| 227 | 231 | * paging_init() sets up the page tables - in fact we've already done this. |
| @@ -259,17 +263,6 @@ void __init paging_init(void) · 31 unmodified lines | ||
| 259 | 263 | |
| 260 | 264 | zone_dma_limit = DMA_BIT_MASK(zone_dma_bits); |
| 261 | 265 | |
| 262 | −#ifdef CONFIG_ZONE_DMA | |
| 263 | − max_zone_pfns[ZONE_DMA] = min(max_low_pfn, | |
| 264 | − 1UL << (zone_dma_bits - PAGE_SHIFT)); | |
| 265 | −#endif | |
| 266 | − max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 267 | −#ifdef CONFIG_HIGHMEM | |
| 268 | − max_zone_pfns[ZONE_HIGHMEM] = max_pfn; | |
| 269 | −#endif | |
| 270 | − | |
| 271 | − free_area_init(max_zone_pfns); | |
| 272 | − | |
| 273 | 266 | mark_nonram_nosave(); |
| 274 | 267 | } |
| 275 | 268 | |
| @@ -1213,8 +1213,6 @@ void __init initmem_init(void) · 1212 unmodified lines | ||
| 1213 | 1213 | setup_node_data(nid, start_pfn, end_pfn); |
| 1214 | 1214 | } |
| 1215 | 1215 | |
| 1216 | − sparse_init(); | |
| 1217 | − | |
| 1218 | 1216 | /* |
| 1219 | 1217 | * We need the numa_cpu_lookup_table to be accurate for all CPUs, |
| 1220 | 1218 | * even before we online them, so that we can use cpu_to_{node,mem} |
| @@ -22,6 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <linux/mm.h> |
| 23 | 23 | #include <linux/percpu.h> |
| 24 | 24 | #include <linux/hardirq.h> |
| 25 | +#include <linux/page_table_check.h> | |
| 25 | 26 | #include <linux/hugetlb.h> |
| 26 | 27 | #include <asm/tlbflush.h> |
| 27 | 28 | #include <asm/tlb.h> |
| @@ -206,6 +207,9 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, · 178 unmodified lines | ||
| 206 | 207 | * and not hw_valid ptes. Hence there is no translation cache flush |
| 207 | 208 | * involved that need to be batched. |
| 208 | 209 | */ |
| 210 | + | |
| 211 | + page_table_check_ptes_set(mm, addr, ptep, pte, nr); | |
| 212 | + | |
| 209 | 213 | for (;;) { |
| 210 | 214 | |
| 211 | 215 | /* |
| @@ -224,6 +228,14 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, · 12 unmodified lines | ||
| 224 | 228 | } |
| 225 | 229 | } |
| 226 | 230 | |
| 231 | +void set_pte_at_unchecked(struct mm_struct *mm, unsigned long addr, | |
| 232 | + pte_t *ptep, pte_t pte) | |
| 233 | +{ | |
| 234 | + VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep)); | |
| 235 | + pte = set_pte_filter(pte, addr); | |
| 236 | + __set_pte_at(mm, addr, ptep, pte, 0); | |
| 237 | +} | |
| 238 | + | |
| 227 | 239 | void unmap_kernel_page(unsigned long va) |
| 228 | 240 | { |
| 229 | 241 | pmd_t *pmdp = pmd_off_k(va); |
| @@ -93,6 +93,7 @@ config PPC_BOOK3S_64 · 92 unmodified lines | ||
| 93 | 93 | select IRQ_WORK |
| 94 | 94 | select PPC_64S_HASH_MMU if !PPC_RADIX_MMU |
| 95 | 95 | select KASAN_VMALLOC if KASAN |
| 96 | + select ARCH_HAS_LAZY_MMU_MODE | |
| 96 | 97 | |
| 97 | 98 | config PPC_BOOK3E_64 |
| 98 | 99 | bool "Embedded processors" |
| @@ -120,7 +120,7 @@ config PPC_SMLPAR · 119 unmodified lines | ||
| 120 | 120 | config CMM |
| 121 | 121 | tristate "Collaborative memory management" |
| 122 | 122 | depends on PPC_SMLPAR |
| 123 | − select MEMORY_BALLOON | |
| 123 | + select BALLOON | |
| 124 | 124 | default y |
| 125 | 125 | help |
| 126 | 126 | Select this option, if you want to enable the kernel interface |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <linux/stringify.h> |
| 20 | 20 | #include <linux/swap.h> |
| 21 | 21 | #include <linux/device.h> |
| 22 | −#include <linux/balloon_compaction.h> | |
| 22 | +#include <linux/balloon.h> | |
| 23 | 23 | #include <asm/firmware.h> |
| 24 | 24 | #include <asm/hvcall.h> |
| 25 | 25 | #include <asm/mmu.h> |
| @@ -165,7 +165,6 @@ static long cmm_alloc_pages(long nr) · 139 unmodified lines | ||
| 165 | 165 | |
| 166 | 166 | balloon_page_enqueue(&b_dev_info, page); |
| 167 | 167 | atomic_long_inc(&loaned_pages); |
| 168 | − adjust_managed_page_count(page, -1); | |
| 169 | 168 | nr--; |
| 170 | 169 | } |
| 171 | 170 | |
| @@ -190,7 +189,6 @@ static long cmm_free_pages(long nr) · 18 unmodified lines | ||
| 190 | 189 | if (!page) |
| 191 | 190 | break; |
| 192 | 191 | plpar_page_set_active(page); |
| 193 | − adjust_managed_page_count(page, 1); | |
| 194 | 192 | __free_page(page); |
| 195 | 193 | atomic_long_dec(&loaned_pages); |
| 196 | 194 | nr--; |
| @@ -496,13 +494,11 @@ static struct notifier_block cmm_mem_nb = { · 299 unmodified lines | ||
| 496 | 494 | .priority = CMM_MEM_HOTPLUG_PRI |
| 497 | 495 | }; |
| 498 | 496 | |
| 499 | −#ifdef CONFIG_BALLOON_COMPACTION | |
| 497 | +#ifdef CONFIG_BALLOON_MIGRATION | |
| 500 | 498 | static int cmm_migratepage(struct balloon_dev_info *b_dev_info, |
| 501 | 499 | struct page *newpage, struct page *page, |
| 502 | 500 | enum migrate_mode mode) |
| 503 | 501 | { |
| 504 | − unsigned long flags; | |
| 505 | − | |
| 506 | 502 | /* |
| 507 | 503 | * loan/"inflate" the newpage first. |
| 508 | 504 | * |
| @@ -517,47 +513,17 @@ static int cmm_migratepage(struct balloon_dev_info *b_dev_info, · 8 unmodified lines | ||
| 517 | 513 | return -EBUSY; |
| 518 | 514 | } |
| 519 | 515 | |
| 520 | − /* balloon page list reference */ | |
| 521 | − get_page(newpage); | |
| 522 | − | |
| 523 | − /* | |
| 524 | − * When we migrate a page to a different zone, we have to fixup the | |
| 525 | − * count of both involved zones as we adjusted the managed page count | |
| 526 | − * when inflating. | |
| 527 | − */ | |
| 528 | − if (page_zone(page) != page_zone(newpage)) { | |
| 529 | − adjust_managed_page_count(page, 1); | |
| 530 | − adjust_managed_page_count(newpage, -1); | |
| 531 | − } | |
| 532 | − | |
| 533 | − spin_lock_irqsave(&b_dev_info->pages_lock, flags); | |
| 534 | − balloon_page_insert(b_dev_info, newpage); | |
| 535 | − __count_vm_event(BALLOON_MIGRATE); | |
| 536 | − b_dev_info->isolated_pages--; | |
| 537 | − spin_unlock_irqrestore(&b_dev_info->pages_lock, flags); | |
| 538 | − | |
| 539 | 516 | /* |
| 540 | 517 | * activate/"deflate" the old page. We ignore any errors just like the |
| 541 | 518 | * other callers. |
| 542 | 519 | */ |
| 543 | 520 | plpar_page_set_active(page); |
| 544 | − | |
| 545 | − balloon_page_finalize(page); | |
| 546 | − /* balloon page list reference */ | |
| 547 | − put_page(page); | |
| 548 | − | |
| 549 | 521 | return 0; |
| 550 | 522 | } |
| 551 | − | |
| 552 | −static void cmm_balloon_compaction_init(void) | |
| 553 | −{ | |
| 554 | − b_dev_info.migratepage = cmm_migratepage; | |
| 555 | −} | |
| 556 | −#else /* CONFIG_BALLOON_COMPACTION */ | |
| 557 | −static void cmm_balloon_compaction_init(void) | |
| 558 | −{ | |
| 559 | −} | |
| 560 | −#endif /* CONFIG_BALLOON_COMPACTION */ | |
| 523 | +#else /* CONFIG_BALLOON_MIGRATION */ | |
| 524 | +int cmm_migratepage(struct balloon_dev_info *b_dev_info, struct page *newpage, | |
| 525 | + struct page *page, enum migrate_mode mode); | |
| 526 | +#endif /* CONFIG_BALLOON_MIGRATION */ | |
| 561 | 527 | |
| 562 | 528 | /** |
| 563 | 529 | * cmm_init - Module initialization |
| @@ -573,11 +539,13 @@ static int cmm_init(void) · 9 unmodified lines | ||
| 573 | 539 | return -EOPNOTSUPP; |
| 574 | 540 | |
| 575 | 541 | balloon_devinfo_init(&b_dev_info); |
| 576 | − cmm_balloon_compaction_init(); | |
| 542 | + b_dev_info.adjust_managed_page_count = true; | |
| 543 | + if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) | |
| 544 | + b_dev_info.migratepage = cmm_migratepage; | |
| 577 | 545 | |
| 578 | 546 | rc = register_oom_notifier(&cmm_oom_nb); |
| 579 | 547 | if (rc < 0) |
| 580 | − goto out_balloon_compaction; | |
| 548 | + return rc; | |
| 581 | 549 | |
| 582 | 550 | if ((rc = register_reboot_notifier(&cmm_reboot_nb))) |
| 583 | 551 | goto out_oom_notifier; |
| @@ -606,7 +574,6 @@ out_reboot_notifier: · 22 unmodified lines | ||
| 606 | 574 | unregister_reboot_notifier(&cmm_reboot_nb); |
| 607 | 575 | out_oom_notifier: |
| 608 | 576 | unregister_oom_notifier(&cmm_oom_nb); |
| 609 | −out_balloon_compaction: | |
| 610 | 577 | return rc; |
| 611 | 578 | } |
| 612 | 579 | |
| @@ -50,7 +50,6 @@ void clear_page(void *page); · 49 unmodified lines | ||
| 50 | 50 | #endif |
| 51 | 51 | #define copy_page(to, from) memcpy((to), (from), PAGE_SIZE) |
| 52 | 52 | |
| 53 | −#define clear_user_page(pgaddr, vaddr, page) clear_page(pgaddr) | |
| 54 | 53 | #define copy_user_page(vto, vfrom, vaddr, topg) \ |
| 55 | 54 | memcpy((vto), (vfrom), PAGE_SIZE) |
| 56 | 55 | |
| @@ -627,7 +627,7 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) · 626 unmodified lines | ||
| 627 | 627 | static inline void set_ptes(struct mm_struct *mm, unsigned long addr, |
| 628 | 628 | pte_t *ptep, pte_t pteval, unsigned int nr) |
| 629 | 629 | { |
| 630 | − page_table_check_ptes_set(mm, ptep, pteval, nr); | |
| 630 | + page_table_check_ptes_set(mm, addr, ptep, pteval, nr); | |
| 631 | 631 | |
| 632 | 632 | for (;;) { |
| 633 | 633 | __set_pte_at(mm, ptep, pteval); |
| @@ -664,7 +664,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · 30 unmodified lines | ||
| 664 | 664 | set_pte(ptep, __pte(0)); |
| 665 | 665 | #endif |
| 666 | 666 | |
| 667 | − page_table_check_pte_clear(mm, pte); | |
| 667 | + page_table_check_pte_clear(mm, address, pte); | |
| 668 | 668 | |
| 669 | 669 | return pte; |
| 670 | 670 | } |
| @@ -946,29 +946,29 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd) · 275 unmodified lines | ||
| 946 | 946 | static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, |
| 947 | 947 | pmd_t *pmdp, pmd_t pmd) |
| 948 | 948 | { |
| 949 | − page_table_check_pmd_set(mm, pmdp, pmd); | |
| 949 | + page_table_check_pmd_set(mm, addr, pmdp, pmd); | |
| 950 | 950 | return __set_pte_at(mm, (pte_t *)pmdp, pmd_pte(pmd)); |
| 951 | 951 | } |
| 952 | 952 | |
| 953 | 953 | static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, |
| 954 | 954 | pud_t *pudp, pud_t pud) |
| 955 | 955 | { |
| 956 | − page_table_check_pud_set(mm, pudp, pud); | |
| 956 | + page_table_check_pud_set(mm, addr, pudp, pud); | |
| 957 | 957 | return __set_pte_at(mm, (pte_t *)pudp, pud_pte(pud)); |
| 958 | 958 | } |
| 959 | 959 | |
| 960 | 960 | #ifdef CONFIG_PAGE_TABLE_CHECK |
| 961 | −static inline bool pte_user_accessible_page(pte_t pte) | |
| 961 | +static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr) | |
| 962 | 962 | { |
| 963 | 963 | return pte_present(pte) && pte_user(pte); |
| 964 | 964 | } |
| 965 | 965 | |
| 966 | −static inline bool pmd_user_accessible_page(pmd_t pmd) | |
| 966 | +static inline bool pmd_user_accessible_page(pmd_t pmd, unsigned long addr) | |
| 967 | 967 | { |
| 968 | 968 | return pmd_leaf(pmd) && pmd_user(pmd); |
| 969 | 969 | } |
| 970 | 970 | |
| 971 | −static inline bool pud_user_accessible_page(pud_t pud) | |
| 971 | +static inline bool pud_user_accessible_page(pud_t pud, unsigned long addr) | |
| 972 | 972 | { |
| 973 | 973 | return pud_leaf(pud) && pud_user(pud); |
| 974 | 974 | } |
| @@ -1007,7 +1007,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, · 32 unmodified lines | ||
| 1007 | 1007 | pmd_clear(pmdp); |
| 1008 | 1008 | #endif |
| 1009 | 1009 | |
| 1010 | − page_table_check_pmd_clear(mm, pmd); | |
| 1010 | + page_table_check_pmd_clear(mm, address, pmd); | |
| 1011 | 1011 | |
| 1012 | 1012 | return pmd; |
| 1013 | 1013 | } |
| @@ -1023,7 +1023,7 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, · 9 unmodified lines | ||
| 1023 | 1023 | static inline pmd_t pmdp_establish(struct vm_area_struct *vma, |
| 1024 | 1024 | unsigned long address, pmd_t *pmdp, pmd_t pmd) |
| 1025 | 1025 | { |
| 1026 | − page_table_check_pmd_set(vma->vm_mm, pmdp, pmd); | |
| 1026 | + page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd); | |
| 1027 | 1027 | return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd))); |
| 1028 | 1028 | } |
| 1029 | 1029 | |
| @@ -1101,7 +1101,7 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, · 71 unmodified lines | ||
| 1101 | 1101 | pud_clear(pudp); |
| 1102 | 1102 | #endif |
| 1103 | 1103 | |
| 1104 | − page_table_check_pud_clear(mm, pud); | |
| 1104 | + page_table_check_pud_clear(mm, address, pud); | |
| 1105 | 1105 | |
| 1106 | 1106 | return pud; |
| 1107 | 1107 | } |
| @@ -1122,7 +1122,7 @@ static inline void update_mmu_cache_pud(struct vm_area_struct *vma, · 14 unmodified lines | ||
| 1122 | 1122 | static inline pud_t pudp_establish(struct vm_area_struct *vma, |
| 1123 | 1123 | unsigned long address, pud_t *pudp, pud_t pud) |
| 1124 | 1124 | { |
| 1125 | − page_table_check_pud_set(vma->vm_mm, pudp, pud); | |
| 1125 | + page_table_check_pud_set(vma->vm_mm, address, pudp, pud); | |
| 1126 | 1126 | return __pud(atomic_long_xchg((atomic_long_t *)pudp, pud_val(pud))); |
| 1127 | 1127 | } |
| 1128 | 1128 | |
| @@ -447,3 +447,11 @@ static __init int gigantic_pages_init(void) · 446 unmodified lines | ||
| 447 | 447 | } |
| 448 | 448 | arch_initcall(gigantic_pages_init); |
| 449 | 449 | #endif |
| 450 | + | |
| 451 | +unsigned int __init arch_hugetlb_cma_order(void) | |
| 452 | +{ | |
| 453 | + if (IS_ENABLED(CONFIG_64BIT)) | |
| 454 | + return PUD_SHIFT - PAGE_SHIFT; | |
| 455 | + | |
| 456 | + return 0; | |
| 457 | +} | |
| @@ -79,16 +79,12 @@ uintptr_t _dtb_early_pa __initdata; · 78 unmodified lines | ||
| 79 | 79 | |
| 80 | 80 | phys_addr_t dma32_phys_limit __initdata; |
| 81 | 81 | |
| 82 | −static void __init zone_sizes_init(void) | |
| 82 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 83 | 83 | { |
| 84 | − unsigned long max_zone_pfns[MAX_NR_ZONES] = { 0, }; | |
| 85 | − | |
| 86 | 84 | #ifdef CONFIG_ZONE_DMA32 |
| 87 | 85 | max_zone_pfns[ZONE_DMA32] = PFN_DOWN(dma32_phys_limit); |
| 88 | 86 | #endif |
| 89 | 87 | max_zone_pfns[ZONE_NORMAL] = max_low_pfn; |
| 90 | − | |
| 91 | − free_area_init(max_zone_pfns); | |
| 92 | 88 | } |
| 93 | 89 | |
| 94 | 90 | #if defined(CONFIG_MMU) && defined(CONFIG_DEBUG_VM) |
| @@ -315,8 +311,6 @@ static void __init setup_bootmem(void) · 220 unmodified lines | ||
| 315 | 311 | memblock_reserve(dtb_early_pa, fdt_totalsize(dtb_early_va)); |
| 316 | 312 | |
| 317 | 313 | dma_contiguous_reserve(dma32_phys_limit); |
| 318 | − if (IS_ENABLED(CONFIG_64BIT)) | |
| 319 | − hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); | |
| 320 | 314 | } |
| 321 | 315 | |
| 322 | 316 | #ifdef CONFIG_RELOCATABLE |
| @@ -1434,12 +1428,10 @@ void __init misc_mem_init(void) · 1111 unmodified lines | ||
| 1434 | 1428 | { |
| 1435 | 1429 | early_memtest(min_low_pfn << PAGE_SHIFT, max_low_pfn << PAGE_SHIFT); |
| 1436 | 1430 | arch_numa_init(); |
| 1437 | − sparse_init(); | |
| 1438 | 1431 | #ifdef CONFIG_SPARSEMEM_VMEMMAP |
| 1439 | 1432 | /* The entire VMEMMAP region has been populated. Flush TLB for this region */ |
| 1440 | 1433 | local_flush_tlb_kernel_range(VMEMMAP_START, VMEMMAP_END); |
| 1441 | 1434 | #endif |
| 1442 | − zone_sizes_init(); | |
| 1443 | 1435 | arch_reserve_crashkernel(); |
| 1444 | 1436 | memblock_dump_all(); |
| 1445 | 1437 | } |
| @@ -65,7 +65,6 @@ static inline void copy_page(void *to, void *from) · 64 unmodified lines | ||
| 65 | 65 | : : "memory", "cc"); |
| 66 | 66 | } |
| 67 | 67 | |
| 68 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 69 | 68 | #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 70 | 69 | |
| 71 | 70 | #define vma_alloc_zeroed_movable_folio(vma, vaddr) \ |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | |
| 25 | 25 | static inline void tlb_flush(struct mmu_gather *tlb); |
| 26 | 26 | static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, |
| 27 | − struct page *page, bool delay_rmap, int page_size); | |
| 27 | + struct page *page, int page_size); | |
| 28 | 28 | static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb, |
| 29 | 29 | struct page *page, unsigned int nr_pages, bool delay_rmap); |
| 30 | 30 | |
| @@ -46,10 +46,8 @@ static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb, · 15 unmodified lines | ||
| 46 | 46 | * s390 doesn't delay rmap removal. |
| 47 | 47 | */ |
| 48 | 48 | static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, |
| 49 | − struct page *page, bool delay_rmap, int page_size) | |
| 49 | + struct page *page, int page_size) | |
| 50 | 50 | { |
| 51 | − VM_WARN_ON_ONCE(delay_rmap); | |
| 52 | − | |
| 53 | 51 | free_folio_and_swap_cache(page_folio(page)); |
| 54 | 52 | return false; |
| 55 | 53 | } |
| @@ -963,8 +963,6 @@ void __init setup_arch(char **cmdline_p) · 962 unmodified lines | ||
| 963 | 963 | setup_uv(); |
| 964 | 964 | dma_contiguous_reserve(ident_map_size); |
| 965 | 965 | vmcp_cma_reserve(); |
| 966 | − if (cpu_has_edat2()) | |
| 967 | − hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); | |
| 968 | 966 | |
| 969 | 967 | reserve_crashkernel(); |
| 970 | 968 | #ifdef CONFIG_CRASH_DUMP |
| @@ -32,7 +32,7 @@ static void ptep_zap_softleaf_entry(struct mm_struct *mm, softleaf_t entry) · 31 unmodified lines | ||
| 32 | 32 | dec_mm_counter(mm, MM_SWAPENTS); |
| 33 | 33 | else if (softleaf_is_migration(entry)) |
| 34 | 34 | dec_mm_counter(mm, mm_counter(softleaf_to_folio(entry))); |
| 35 | − free_swap_and_cache(entry); | |
| 35 | + swap_put_entries_direct(entry, 1); | |
| 36 | 36 | } |
| 37 | 37 | |
| 38 | 38 | /** |
| @@ -255,3 +255,11 @@ bool __init arch_hugetlb_valid_size(unsigned long size) · 254 unmodified lines | ||
| 255 | 255 | else |
| 256 | 256 | return false; |
| 257 | 257 | } |
| 258 | + | |
| 259 | +unsigned int __init arch_hugetlb_cma_order(void) | |
| 260 | +{ | |
| 261 | + if (cpu_has_edat2()) | |
| 262 | + return PUD_SHIFT - PAGE_SHIFT; | |
| 263 | + | |
| 264 | + return 0; | |
| 265 | +} | |
| @@ -86,20 +86,19 @@ static void __init setup_zero_pages(void) · 85 unmodified lines | ||
| 86 | 86 | zero_page_mask = ((PAGE_SIZE << order) - 1) & PAGE_MASK; |
| 87 | 87 | } |
| 88 | 88 | |
| 89 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 90 | +{ | |
| 91 | + max_zone_pfns[ZONE_DMA] = virt_to_pfn(MAX_DMA_ADDRESS); | |
| 92 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 93 | +} | |
| 94 | + | |
| 89 | 95 | /* |
| 90 | 96 | * paging_init() sets up the page tables |
| 91 | 97 | */ |
| 92 | 98 | void __init paging_init(void) |
| 93 | 99 | { |
| 94 | − unsigned long max_zone_pfns[MAX_NR_ZONES]; | |
| 95 | − | |
| 96 | 100 | vmem_map_init(); |
| 97 | − sparse_init(); | |
| 98 | 101 | zone_dma_limit = DMA_BIT_MASK(31); |
| 99 | − memset(max_zone_pfns, 0, sizeof(max_zone_pfns)); | |
| 100 | − max_zone_pfns[ZONE_DMA] = virt_to_pfn(MAX_DMA_ADDRESS); | |
| 101 | − max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 102 | − free_area_init(max_zone_pfns); | |
| 103 | 102 | } |
| 104 | 103 | |
| 105 | 104 | void mark_rodata_ro(void) |
| @@ -682,7 +682,7 @@ static void ptep_zap_softleaf_entry(struct mm_struct *mm, softleaf_t entry) · 681 unmodified lines | ||
| 682 | 682 | |
| 683 | 683 | dec_mm_counter(mm, mm_counter(folio)); |
| 684 | 684 | } |
| 685 | − free_swap_and_cache(entry); | |
| 685 | + swap_put_entries_direct(entry, 1); | |
| 686 | 686 | } |
| 687 | 687 | |
| 688 | 688 | void ptep_zap_unused(struct mm_struct *mm, unsigned long addr, |
| @@ -227,8 +227,6 @@ static void __init do_init_bootmem(void) · 226 unmodified lines | ||
| 227 | 227 | node_set_online(0); |
| 228 | 228 | |
| 229 | 229 | plat_mem_setup(); |
| 230 | − | |
| 231 | − sparse_init(); | |
| 232 | 230 | } |
| 233 | 231 | |
| 234 | 232 | static void __init early_reserve_mem(void) |
| @@ -264,9 +262,13 @@ static void __init early_reserve_mem(void) · 29 unmodified lines | ||
| 264 | 262 | reserve_crashkernel(); |
| 265 | 263 | } |
| 266 | 264 | |
| 265 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 266 | +{ | |
| 267 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 268 | +} | |
| 269 | + | |
| 267 | 270 | void __init paging_init(void) |
| 268 | 271 | { |
| 269 | − unsigned long max_zone_pfns[MAX_NR_ZONES]; | |
| 270 | 272 | unsigned long vaddr, end; |
| 271 | 273 | |
| 272 | 274 | sh_mv.mv_mem_init(); |
| @@ -320,10 +322,6 @@ void __init paging_init(void) · 47 unmodified lines | ||
| 320 | 322 | page_table_range_init(vaddr, end, swapper_pg_dir); |
| 321 | 323 | |
| 322 | 324 | kmap_coherent_init(); |
| 323 | − | |
| 324 | − memset(max_zone_pfns, 0, sizeof(max_zone_pfns)); | |
| 325 | − max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 326 | − free_area_init(max_zone_pfns); | |
| 327 | 325 | } |
| 328 | 326 | |
| 329 | 327 | unsigned int mem_init_done = 0; |
| @@ -75,6 +75,7 @@ config SPARC64 · 74 unmodified lines | ||
| 75 | 75 | select HAVE_KRETPROBES |
| 76 | 76 | select HAVE_KPROBES |
| 77 | 77 | select MMU_GATHER_RCU_TABLE_FREE if SMP |
| 78 | + select HAVE_ARCH_TLB_REMOVE_TABLE if SMP | |
| 78 | 79 | select MMU_GATHER_MERGE_VMAS |
| 79 | 80 | select MMU_GATHER_NO_FLUSH_CACHE |
| 80 | 81 | select HAVE_ARCH_TRANSPARENT_HUGEPAGE |
| @@ -113,6 +114,7 @@ config SPARC64 · 32 unmodified lines | ||
| 113 | 114 | select NEED_PER_CPU_PAGE_FIRST_CHUNK |
| 114 | 115 | select ARCH_SUPPORTS_SCHED_SMT if SMP |
| 115 | 116 | select ARCH_SUPPORTS_SCHED_MC if SMP |
| 117 | + select ARCH_HAS_LAZY_MMU_MODE | |
| 116 | 118 | |
| 117 | 119 | config ARCH_PROC_KCORE_TEXT |
| 118 | 120 | def_bool y |
| @@ -43,6 +43,7 @@ void _clear_page(void *page); · 42 unmodified lines | ||
| 43 | 43 | #define clear_page(X) _clear_page((void *)(X)) |
| 44 | 44 | struct page; |
| 45 | 45 | void clear_user_page(void *addr, unsigned long vaddr, struct page *page); |
| 46 | +#define clear_user_page clear_user_page | |
| 46 | 47 | #define copy_page(X,Y) memcpy((void *)(X), (void *)(Y), PAGE_SIZE) |
| 47 | 48 | void copy_user_page(void *to, void *from, unsigned long vaddr, struct page *topage); |
| 48 | 49 | #define __HAVE_ARCH_COPY_USER_HIGHPAGE |
| @@ -33,7 +33,6 @@ void flush_tlb_pending(void); · 32 unmodified lines | ||
| 33 | 33 | #define tlb_needs_table_invalidate() (false) |
| 34 | 34 | #endif |
| 35 | 35 | |
| 36 | −#define __HAVE_ARCH_TLB_REMOVE_TABLE | |
| 37 | 36 | #include <asm-generic/tlb.h> |
| 38 | 37 | |
| 39 | 38 | #endif /* _SPARC64_TLB_H */ |
| @@ -12,7 +12,6 @@ struct tlb_batch { · 11 unmodified lines | ||
| 12 | 12 | unsigned int hugepage_shift; |
| 13 | 13 | struct mm_struct *mm; |
| 14 | 14 | unsigned long tlb_nr; |
| 15 | − unsigned long active; | |
| 16 | 15 | unsigned long vaddrs[TLB_BATCH_NR]; |
| 17 | 16 | }; |
| 18 | 17 | |
| @@ -39,12 +38,10 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, · 20 unmodified lines | ||
| 39 | 38 | |
| 40 | 39 | void flush_tlb_kernel_range(unsigned long start, unsigned long end); |
| 41 | 40 | |
| 42 | −#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE | |
| 43 | − | |
| 44 | 41 | void flush_tlb_pending(void); |
| 45 | 42 | void arch_enter_lazy_mmu_mode(void); |
| 43 | +void arch_flush_lazy_mmu_mode(void); | |
| 46 | 44 | void arch_leave_lazy_mmu_mode(void); |
| 47 | −#define arch_flush_lazy_mmu_mode() do {} while (0) | |
| 48 | 45 | |
| 49 | 46 | /* Local cpu only. */ |
| 50 | 47 | void __flush_tlb_all(void); |
| @@ -1609,8 +1609,6 @@ static unsigned long __init bootmem_init(unsigned long phys_base) · 1608 unmodified lines | ||
| 1609 | 1609 | |
| 1610 | 1610 | /* XXX cpu notifier XXX */ |
| 1611 | 1611 | |
| 1612 | − sparse_init(); | |
| 1613 | − | |
| 1614 | 1612 | return end_pfn; |
| 1615 | 1613 | } |
| 1616 | 1614 | |
| @@ -2273,6 +2271,11 @@ static void __init reduce_memory(phys_addr_t limit_ram) · 656 unmodified lines | ||
| 2273 | 2271 | memblock_enforce_memory_limit(limit_ram); |
| 2274 | 2272 | } |
| 2275 | 2273 | |
| 2274 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 2275 | +{ | |
| 2276 | + max_zone_pfns[ZONE_NORMAL] = last_valid_pfn; | |
| 2277 | +} | |
| 2278 | + | |
| 2276 | 2279 | void __init paging_init(void) |
| 2277 | 2280 | { |
| 2278 | 2281 | unsigned long end_pfn, shift, phys_base; |
| @@ -2448,16 +2451,6 @@ void __init paging_init(void) · 169 unmodified lines | ||
| 2448 | 2451 | |
| 2449 | 2452 | kernel_physical_mapping_init(); |
| 2450 | 2453 | |
| 2451 | − { | |
| 2452 | − unsigned long max_zone_pfns[MAX_NR_ZONES]; | |
| 2453 | − | |
| 2454 | − memset(max_zone_pfns, 0, sizeof(max_zone_pfns)); | |
| 2455 | − | |
| 2456 | − max_zone_pfns[ZONE_NORMAL] = end_pfn; | |
| 2457 | − | |
| 2458 | − free_area_init(max_zone_pfns); | |
| 2459 | − } | |
| 2460 | − | |
| 2461 | 2454 | printk("Booting Linux...\n"); |
| 2462 | 2455 | } |
| 2463 | 2456 | |
| @@ -884,6 +884,13 @@ static void __init map_kernel(void) · 883 unmodified lines | ||
| 884 | 884 | |
| 885 | 885 | void (*poke_srmmu)(void) = NULL; |
| 886 | 886 | |
| 887 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 888 | +{ | |
| 889 | + max_zone_pfns[ZONE_DMA] = max_low_pfn; | |
| 890 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 891 | + max_zone_pfns[ZONE_HIGHMEM] = highend_pfn; | |
| 892 | +} | |
| 893 | + | |
| 887 | 894 | void __init srmmu_paging_init(void) |
| 888 | 895 | { |
| 889 | 896 | int i; |
| @@ -963,16 +970,6 @@ void __init srmmu_paging_init(void) · 73 unmodified lines | ||
| 963 | 970 | flush_tlb_all(); |
| 964 | 971 | |
| 965 | 972 | sparc_context_init(num_contexts); |
| 966 | − | |
| 967 | − { | |
| 968 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 969 | − | |
| 970 | − max_zone_pfn[ZONE_DMA] = max_low_pfn; | |
| 971 | − max_zone_pfn[ZONE_NORMAL] = max_low_pfn; | |
| 972 | − max_zone_pfn[ZONE_HIGHMEM] = highend_pfn; | |
| 973 | − | |
| 974 | − free_area_init(max_zone_pfn); | |
| 975 | − } | |
| 976 | 973 | } |
| 977 | 974 | |
| 978 | 975 | void mmu_info(struct seq_file *m) |
| @@ -11,6 +11,8 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/preempt.h> |
| 12 | 12 | #include <linux/pagemap.h> |
| 13 | 13 | |
| 14 | +#include <kunit/visibility.h> | |
| 15 | + | |
| 14 | 16 | #include <asm/tlbflush.h> |
| 15 | 17 | #include <asm/cacheflush.h> |
| 16 | 18 | #include <asm/mmu_context.h> |
| @@ -52,22 +54,26 @@ out: · 35 unmodified lines | ||
| 52 | 54 | |
| 53 | 55 | void arch_enter_lazy_mmu_mode(void) |
| 54 | 56 | { |
| 55 | − struct tlb_batch *tb; | |
| 56 | − | |
| 57 | 57 | preempt_disable(); |
| 58 | − tb = this_cpu_ptr(&tlb_batch); | |
| 59 | − tb->active = 1; | |
| 60 | 58 | } |
| 59 | +/* For lazy_mmu_mode KUnit tests */ | |
| 60 | +EXPORT_SYMBOL_IF_KUNIT(arch_enter_lazy_mmu_mode); | |
| 61 | 61 | |
| 62 | −void arch_leave_lazy_mmu_mode(void) | |
| 62 | +void arch_flush_lazy_mmu_mode(void) | |
| 63 | 63 | { |
| 64 | 64 | struct tlb_batch *tb = this_cpu_ptr(&tlb_batch); |
| 65 | 65 | |
| 66 | 66 | if (tb->tlb_nr) |
| 67 | 67 | flush_tlb_pending(); |
| 68 | − tb->active = 0; | |
| 68 | +} | |
| 69 | +EXPORT_SYMBOL_IF_KUNIT(arch_flush_lazy_mmu_mode); | |
| 70 | + | |
| 71 | +void arch_leave_lazy_mmu_mode(void) | |
| 72 | +{ | |
| 73 | + arch_flush_lazy_mmu_mode(); | |
| 69 | 74 | preempt_enable(); |
| 70 | 75 | } |
| 76 | +EXPORT_SYMBOL_IF_KUNIT(arch_leave_lazy_mmu_mode); | |
| 71 | 77 | |
| 72 | 78 | static void tlb_batch_add_one(struct mm_struct *mm, unsigned long vaddr, |
| 73 | 79 | bool exec, unsigned int hugepage_shift) |
| @@ -86,7 +92,7 @@ static void tlb_batch_add_one(struct mm_struct *mm, unsigned long vaddr, · 12 unmodified lines | ||
| 86 | 92 | nr = 0; |
| 87 | 93 | } |
| 88 | 94 | |
| 89 | − if (!tb->active) { | |
| 95 | + if (!is_lazy_mmu_mode_active()) { | |
| 90 | 96 | flush_tsb_user_page(mm, vaddr, hugepage_shift); |
| 91 | 97 | global_flush_tlb_page(mm, vaddr); |
| 92 | 98 | goto out; |
| @@ -42,6 +42,7 @@ config UML · 41 unmodified lines | ||
| 42 | 42 | select HAVE_SYSCALL_TRACEPOINTS |
| 43 | 43 | select THREAD_INFO_IN_TASK |
| 44 | 44 | select SPARSE_IRQ |
| 45 | + select MMU_GATHER_RCU_TABLE_FREE | |
| 45 | 46 | |
| 46 | 47 | config MMU |
| 47 | 48 | bool |
| @@ -26,7 +26,6 @@ struct page; · 25 unmodified lines | ||
| 26 | 26 | #define clear_page(page) memset((void *)(page), 0, PAGE_SIZE) |
| 27 | 27 | #define copy_page(to,from) memcpy((void *)(to), (void *)(from), PAGE_SIZE) |
| 28 | 28 | |
| 29 | −#define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 30 | 29 | #define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 31 | 30 | |
| 32 | 31 | typedef struct { unsigned long pte; } pte_t; |
| @@ -84,18 +84,18 @@ void __init mem_init(void) · 83 unmodified lines | ||
| 84 | 84 | kmalloc_ok = 1; |
| 85 | 85 | } |
| 86 | 86 | |
| 87 | −void __init paging_init(void) | |
| 87 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 88 | 88 | { |
| 89 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 89 | + max_zone_pfns[ZONE_NORMAL] = high_physmem >> PAGE_SHIFT; | |
| 90 | +} | |
| 90 | 91 | |
| 92 | +void __init paging_init(void) | |
| 93 | +{ | |
| 91 | 94 | empty_zero_page = (unsigned long *) memblock_alloc_low(PAGE_SIZE, |
| 92 | 95 | PAGE_SIZE); |
| 93 | 96 | if (!empty_zero_page) |
| 94 | 97 | panic("%s: Failed to allocate %lu bytes align=%lx\n", |
| 95 | 98 | __func__, PAGE_SIZE, PAGE_SIZE); |
| 96 | − | |
| 97 | − max_zone_pfn[ZONE_NORMAL] = high_physmem >> PAGE_SHIFT; | |
| 98 | − free_area_init(max_zone_pfn); | |
| 99 | 99 | } |
| 100 | 100 | |
| 101 | 101 | /* |
| @@ -331,7 +331,6 @@ config X86 · 330 unmodified lines | ||
| 331 | 331 | select FUNCTION_ALIGNMENT_4B |
| 332 | 332 | imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI |
| 333 | 333 | select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE |
| 334 | − select ARCH_SUPPORTS_PT_RECLAIM if X86_64 | |
| 335 | 334 | select ARCH_SUPPORTS_SCHED_SMT if SMP |
| 336 | 335 | select SCHED_SMT if SMP |
| 337 | 336 | select ARCH_SUPPORTS_SCHED_CLUSTER if SMP |
| @@ -823,6 +822,7 @@ config PARAVIRT · 485 unmodified lines | ||
| 823 | 822 | config PARAVIRT_XXL |
| 824 | 823 | bool |
| 825 | 824 | depends on X86_64 |
| 825 | + select ARCH_HAS_LAZY_MMU_MODE | |
| 826 | 826 | |
| 827 | 827 | config PARAVIRT_SPINLOCKS |
| 828 | 828 | bool "Paravirtualization layer for spinlocks" |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #undef CONFIG_PARAVIRT |
| 12 | 12 | #undef CONFIG_PARAVIRT_XXL |
| 13 | 13 | #undef CONFIG_PARAVIRT_SPINLOCKS |
| 14 | +#undef CONFIG_ARCH_HAS_LAZY_MMU_MODE | |
| 14 | 15 | #undef CONFIG_KASAN |
| 15 | 16 | #undef CONFIG_KASAN_GENERIC |
| 16 | 17 | |
| @@ -24,6 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #undef CONFIG_PARAVIRT |
| 25 | 25 | #undef CONFIG_PARAVIRT_XXL |
| 26 | 26 | #undef CONFIG_PARAVIRT_SPINLOCKS |
| 27 | +#undef CONFIG_ARCH_HAS_LAZY_MMU_MODE | |
| 27 | 28 | |
| 28 | 29 | /* |
| 29 | 30 | * This code runs before CPU feature bits are set. By default, the |
| @@ -22,12 +22,6 @@ struct page; · 21 unmodified lines | ||
| 22 | 22 | extern struct range pfn_mapped[]; |
| 23 | 23 | extern int nr_pfn_mapped; |
| 24 | 24 | |
| 25 | −static inline void clear_user_page(void *page, unsigned long vaddr, | |
| 26 | − struct page *pg) | |
| 27 | −{ | |
| 28 | − clear_page(page); | |
| 29 | −} | |
| 30 | − | |
| 31 | 25 | static inline void copy_user_page(void *to, void *from, unsigned long vaddr, |
| 32 | 26 | struct page *topage) |
| 33 | 27 | { |
| @@ -17,6 +17,12 @@ extern unsigned long __phys_addr(unsigned long); · 16 unmodified lines | ||
| 17 | 17 | |
| 18 | 18 | #include <linux/string.h> |
| 19 | 19 | |
| 20 | +/** | |
| 21 | + * clear_page() - clear a page using a kernel virtual address. | |
| 22 | + * @page: address of kernel page | |
| 23 | + * | |
| 24 | + * Does absolutely no exception handling. | |
| 25 | + */ | |
| 20 | 26 | static inline void clear_page(void *page) |
| 21 | 27 | { |
| 22 | 28 | memset(page, 0, PAGE_SIZE); |
| @@ -48,26 +48,70 @@ static inline unsigned long __phys_addr_symbol(unsigned long x) · 47 unmodified lines | ||
| 48 | 48 | |
| 49 | 49 | #define __phys_reloc_hide(x) (x) |
| 50 | 50 | |
| 51 | −void clear_page_orig(void *page); | |
| 52 | −void clear_page_rep(void *page); | |
| 53 | −void clear_page_erms(void *page); | |
| 54 | −KCFI_REFERENCE(clear_page_orig); | |
| 55 | −KCFI_REFERENCE(clear_page_rep); | |
| 56 | −KCFI_REFERENCE(clear_page_erms); | |
| 57 | − | |
| 58 | −static inline void clear_page(void *page) | |
| 51 | +void __clear_pages_unrolled(void *page); | |
| 52 | +KCFI_REFERENCE(__clear_pages_unrolled); | |
| 53 | + | |
| 54 | +/** | |
| 55 | + * clear_pages() - clear a page range using a kernel virtual address. | |
| 56 | + * @addr: start address of kernel page range | |
| 57 | + * @npages: number of pages | |
| 58 | + * | |
| 59 | + * Switch between three implementations of page clearing based on CPU | |
| 60 | + * capabilities: | |
| 61 | + * | |
| 62 | + * - __clear_pages_unrolled(): the oldest, slowest and universally | |
| 63 | + * supported method. Zeroes via 8-byte MOV instructions unrolled 8x | |
| 64 | + * to write a 64-byte cacheline in each loop iteration. | |
| 65 | + * | |
| 66 | + * - "REP; STOSQ": really old CPUs had crummy REP implementations. | |
| 67 | + * Vendor CPU setup code sets 'REP_GOOD' on CPUs where REP can be | |
| 68 | + * trusted. The instruction writes 8-byte per REP iteration but | |
| 69 | + * CPUs can internally batch these together and do larger writes. | |
| 70 | + * | |
| 71 | + * - "REP; STOSB": used on CPUs with "enhanced REP MOVSB/STOSB", | |
| 72 | + * which enumerate 'ERMS' and provide an implementation which | |
| 73 | + * unlike "REP; STOSQ" above wasn't overly picky about alignment. | |
| 74 | + * The instruction writes 1-byte per REP iteration with CPUs | |
| 75 | + * internally batching these together into larger writes and is | |
| 76 | + * generally fastest of the three. | |
| 77 | + * | |
| 78 | + * Note that when running as a guest, features exposed by the CPU | |
| 79 | + * might be mediated by the hypervisor. So, the STOSQ variant might | |
| 80 | + * be in active use on some systems even when the hardware enumerates | |
| 81 | + * ERMS. | |
| 82 | + * | |
| 83 | + * Does absolutely no exception handling. | |
| 84 | + */ | |
| 85 | +static inline void clear_pages(void *addr, unsigned int npages) | |
| 59 | 86 | { |
| 87 | + u64 len = npages * PAGE_SIZE; | |
| 88 | + /* | |
| 89 | + * Clean up KMSAN metadata for the pages being cleared. The assembly call | |
| 90 | + * below clobbers @addr, so perform unpoisoning before it. | |
| 91 | + */ | |
| 92 | + kmsan_unpoison_memory(addr, len); | |
| 93 | + | |
| 60 | 94 | /* |
| 61 | − * Clean up KMSAN metadata for the page being cleared. The assembly call | |
| 62 | − * below clobbers @page, so we perform unpoisoning before it. | |
| 95 | + * The inline asm embeds a CALL instruction and usually that is a no-no | |
| 96 | + * due to the compiler not knowing that and thus being unable to track | |
| 97 | + * callee-clobbered registers. | |
| 98 | + * | |
| 99 | + * In this case that is fine because the registers clobbered by | |
| 100 | + * __clear_pages_unrolled() are part of the inline asm register | |
| 101 | + * specification. | |
| 63 | 102 | */ |
| 64 | − kmsan_unpoison_memory(page, PAGE_SIZE); | |
| 65 | − alternative_call_2(clear_page_orig, | |
| 66 | − clear_page_rep, X86_FEATURE_REP_GOOD, | |
| 67 | − clear_page_erms, X86_FEATURE_ERMS, | |
| 68 | − "=D" (page), | |
| 69 | − "D" (page), | |
| 70 | − "cc", "memory", "rax", "rcx"); | |
| 103 | + asm volatile(ALTERNATIVE_2("call __clear_pages_unrolled", | |
| 104 | + "shrq $3, %%rcx; rep stosq", X86_FEATURE_REP_GOOD, | |
| 105 | + "rep stosb", X86_FEATURE_ERMS) | |
| 106 | + : "+c" (len), "+D" (addr), ASM_CALL_CONSTRAINT | |
| 107 | + : "a" (0) | |
| 108 | + : "cc", "memory"); | |
| 109 | +} | |
| 110 | +#define clear_pages clear_pages | |
| 111 | + | |
| 112 | +static inline void clear_page(void *addr) | |
| 113 | +{ | |
| 114 | + clear_pages(addr, 1); | |
| 71 | 115 | } |
| 72 | 116 | |
| 73 | 117 | void copy_page(void *to, void *from); |
| @@ -492,7 +492,6 @@ static inline void arch_end_context_switch(struct task_struct *next) · 491 unmodified lines | ||
| 492 | 492 | PVOP_VCALL1(pv_ops, cpu.end_context_switch, next); |
| 493 | 493 | } |
| 494 | 494 | |
| 495 | −#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE | |
| 496 | 495 | static inline void arch_enter_lazy_mmu_mode(void) |
| 497 | 496 | { |
| 498 | 497 | PVOP_VCALL0(pv_ops, mmu.lazy_mode.enter); |
| @@ -118,6 +118,7 @@ extern pmdval_t early_pmd_flags; · 117 unmodified lines | ||
| 118 | 118 | #define __pte(x) native_make_pte(x) |
| 119 | 119 | |
| 120 | 120 | #define arch_end_context_switch(prev) do {} while(0) |
| 121 | +static inline void arch_flush_lazy_mmu_mode(void) {} | |
| 121 | 122 | #endif /* CONFIG_PARAVIRT_XXL */ |
| 122 | 123 | |
| 123 | 124 | static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) |
| @@ -1213,14 +1214,14 @@ static inline pud_t native_local_pudp_get_and_clear(pud_t *pudp) · 1089 unmodified lines | ||
| 1213 | 1214 | static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, |
| 1214 | 1215 | pmd_t *pmdp, pmd_t pmd) |
| 1215 | 1216 | { |
| 1216 | − page_table_check_pmd_set(mm, pmdp, pmd); | |
| 1217 | + page_table_check_pmd_set(mm, addr, pmdp, pmd); | |
| 1217 | 1218 | set_pmd(pmdp, pmd); |
| 1218 | 1219 | } |
| 1219 | 1220 | |
| 1220 | 1221 | static inline void set_pud_at(struct mm_struct *mm, unsigned long addr, |
| 1221 | 1222 | pud_t *pudp, pud_t pud) |
| 1222 | 1223 | { |
| 1223 | − page_table_check_pud_set(mm, pudp, pud); | |
| 1224 | + page_table_check_pud_set(mm, addr, pudp, pud); | |
| 1224 | 1225 | native_set_pud(pudp, pud); |
| 1225 | 1226 | } |
| 1226 | 1227 | |
| @@ -1251,7 +1252,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, · 24 unmodified lines | ||
| 1251 | 1252 | pte_t *ptep) |
| 1252 | 1253 | { |
| 1253 | 1254 | pte_t pte = native_ptep_get_and_clear(ptep); |
| 1254 | − page_table_check_pte_clear(mm, pte); | |
| 1255 | + page_table_check_pte_clear(mm, addr, pte); | |
| 1255 | 1256 | return pte; |
| 1256 | 1257 | } |
| 1257 | 1258 | |
| @@ -1267,7 +1268,7 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · 9 unmodified lines | ||
| 1267 | 1268 | * care about updates and native needs no locking |
| 1268 | 1269 | */ |
| 1269 | 1270 | pte = native_local_ptep_get_and_clear(ptep); |
| 1270 | − page_table_check_pte_clear(mm, pte); | |
| 1271 | + page_table_check_pte_clear(mm, addr, pte); | |
| 1271 | 1272 | } else { |
| 1272 | 1273 | pte = ptep_get_and_clear(mm, addr, ptep); |
| 1273 | 1274 | } |
| @@ -1318,7 +1319,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long · 44 unmodified lines | ||
| 1318 | 1319 | { |
| 1319 | 1320 | pmd_t pmd = native_pmdp_get_and_clear(pmdp); |
| 1320 | 1321 | |
| 1321 | − page_table_check_pmd_clear(mm, pmd); | |
| 1322 | + page_table_check_pmd_clear(mm, addr, pmd); | |
| 1322 | 1323 | |
| 1323 | 1324 | return pmd; |
| 1324 | 1325 | } |
| @@ -1329,7 +1330,7 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, · 4 unmodified lines | ||
| 1329 | 1330 | { |
| 1330 | 1331 | pud_t pud = native_pudp_get_and_clear(pudp); |
| 1331 | 1332 | |
| 1332 | − page_table_check_pud_clear(mm, pud); | |
| 1333 | + page_table_check_pud_clear(mm, addr, pud); | |
| 1333 | 1334 | |
| 1334 | 1335 | return pud; |
| 1335 | 1336 | } |
| @@ -1356,7 +1357,7 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, · 20 unmodified lines | ||
| 1356 | 1357 | static inline pmd_t pmdp_establish(struct vm_area_struct *vma, |
| 1357 | 1358 | unsigned long address, pmd_t *pmdp, pmd_t pmd) |
| 1358 | 1359 | { |
| 1359 | − page_table_check_pmd_set(vma->vm_mm, pmdp, pmd); | |
| 1360 | + page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd); | |
| 1360 | 1361 | if (IS_ENABLED(CONFIG_SMP)) { |
| 1361 | 1362 | return xchg(pmdp, pmd); |
| 1362 | 1363 | } else { |
| @@ -1371,7 +1372,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · 8 unmodified lines | ||
| 1371 | 1372 | static inline pud_t pudp_establish(struct vm_area_struct *vma, |
| 1372 | 1373 | unsigned long address, pud_t *pudp, pud_t pud) |
| 1373 | 1374 | { |
| 1374 | − page_table_check_pud_set(vma->vm_mm, pudp, pud); | |
| 1375 | + page_table_check_pud_set(vma->vm_mm, address, pudp, pud); | |
| 1375 | 1376 | if (IS_ENABLED(CONFIG_SMP)) { |
| 1376 | 1377 | return xchg(pudp, pud); |
| 1377 | 1378 | } else { |
| @@ -1679,17 +1680,17 @@ static inline bool arch_has_hw_nonleaf_pmd_young(void) · 301 unmodified lines | ||
| 1679 | 1680 | #endif |
| 1680 | 1681 | |
| 1681 | 1682 | #ifdef CONFIG_PAGE_TABLE_CHECK |
| 1682 | −static inline bool pte_user_accessible_page(pte_t pte) | |
| 1683 | +static inline bool pte_user_accessible_page(pte_t pte, unsigned long addr) | |
| 1683 | 1684 | { |
| 1684 | 1685 | return (pte_val(pte) & _PAGE_PRESENT) && (pte_val(pte) & _PAGE_USER); |
| 1685 | 1686 | } |
| 1686 | 1687 | |
| 1687 | −static inline bool pmd_user_accessible_page(pmd_t pmd) | |
| 1688 | +static inline bool pmd_user_accessible_page(pmd_t pmd, unsigned long addr) | |
| 1688 | 1689 | { |
| 1689 | 1690 | return pmd_leaf(pmd) && (pmd_val(pmd) & _PAGE_PRESENT) && (pmd_val(pmd) & _PAGE_USER); |
| 1690 | 1691 | } |
| 1691 | 1692 | |
| 1692 | −static inline bool pud_user_accessible_page(pud_t pud) | |
| 1693 | +static inline bool pud_user_accessible_page(pud_t pud, unsigned long addr) | |
| 1693 | 1694 | { |
| 1694 | 1695 | return pud_leaf(pud) && (pud_val(pud) & _PAGE_PRESENT) && (pud_val(pud) & _PAGE_USER); |
| 1695 | 1696 | } |
| @@ -100,8 +100,7 @@ struct thread_info { · 99 unmodified lines | ||
| 100 | 100 | #define TIF_FORCED_TF 24 /* true if TF in eflags artificially */ |
| 101 | 101 | #define TIF_SINGLESTEP 25 /* reenable singlestep on user return*/ |
| 102 | 102 | #define TIF_BLOCKSTEP 26 /* set when we want DEBUGCTLMSR_BTF */ |
| 103 | −#define TIF_LAZY_MMU_UPDATES 27 /* task is updating the mmu lazily */ | |
| 104 | −#define TIF_ADDR32 28 /* 32-bit address space on 64 bits */ | |
| 103 | +#define TIF_ADDR32 27 /* 32-bit address space on 64 bits */ | |
| 105 | 104 | |
| 106 | 105 | #define _TIF_SSBD BIT(TIF_SSBD) |
| 107 | 106 | #define _TIF_SPEC_IB BIT(TIF_SPEC_IB) |
| @@ -114,7 +113,6 @@ struct thread_info { · 6 unmodified lines | ||
| 114 | 113 | #define _TIF_FORCED_TF BIT(TIF_FORCED_TF) |
| 115 | 114 | #define _TIF_BLOCKSTEP BIT(TIF_BLOCKSTEP) |
| 116 | 115 | #define _TIF_SINGLESTEP BIT(TIF_SINGLESTEP) |
| 117 | −#define _TIF_LAZY_MMU_UPDATES BIT(TIF_LAZY_MMU_UPDATES) | |
| 118 | 116 | #define _TIF_ADDR32 BIT(TIF_ADDR32) |
| 119 | 117 | |
| 120 | 118 | /* flags to check in __switch_to() */ |
| @@ -1185,11 +1185,6 @@ void __init setup_arch(char **cmdline_p) · 1184 unmodified lines | ||
| 1185 | 1185 | initmem_init(); |
| 1186 | 1186 | dma_contiguous_reserve(max_pfn_mapped << PAGE_SHIFT); |
| 1187 | 1187 | |
| 1188 | − if (boot_cpu_has(X86_FEATURE_GBPAGES)) { | |
| 1189 | − hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); | |
| 1190 | − hugetlb_bootmem_alloc(); | |
| 1191 | − } | |
| 1192 | − | |
| 1193 | 1188 | /* |
| 1194 | 1189 | * Reserve memory for crash kernel after SRAT is parsed so that it |
| 1195 | 1190 | * won't consume hotpluggable memory. |
| @@ -6,30 +6,15 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <asm/asm.h> |
| 7 | 7 | |
| 8 | 8 | /* |
| 9 | − * Most CPUs support enhanced REP MOVSB/STOSB instructions. It is | |
| 10 | − * recommended to use this when possible and we do use them by default. | |
| 11 | − * If enhanced REP MOVSB/STOSB is not available, try to use fast string. | |
| 12 | − * Otherwise, use original. | |
| 9 | + * Zero page aligned region. | |
| 10 | + * %rdi - dest | |
| 11 | + * %rcx - length | |
| 13 | 12 | */ |
| 14 | − | |
| 15 | −/* | |
| 16 | − * Zero a page. | |
| 17 | − * %rdi - page | |
| 18 | − */ | |
| 19 | −SYM_TYPED_FUNC_START(clear_page_rep) | |
| 20 | − movl $4096/8,%ecx | |
| 21 | − xorl %eax,%eax | |
| 22 | − rep stosq | |
| 23 | − RET | |
| 24 | −SYM_FUNC_END(clear_page_rep) | |
| 25 | −EXPORT_SYMBOL_GPL(clear_page_rep) | |
| 26 | − | |
| 27 | −SYM_TYPED_FUNC_START(clear_page_orig) | |
| 28 | − xorl %eax,%eax | |
| 29 | − movl $4096/64,%ecx | |
| 13 | +SYM_TYPED_FUNC_START(__clear_pages_unrolled) | |
| 14 | + shrq $6, %rcx | |
| 30 | 15 | .p2align 4 |
| 31 | 16 | .Lloop: |
| 32 | − decl %ecx | |
| 17 | + decq %rcx | |
| 33 | 18 | #define PUT(x) movq %rax,x*8(%rdi) |
| 34 | 19 | movq %rax,(%rdi) |
| 35 | 20 | PUT(1) |
| @@ -43,16 +28,8 @@ SYM_TYPED_FUNC_START(clear_page_orig) · 7 unmodified lines | ||
| 43 | 28 | jnz .Lloop |
| 44 | 29 | nop |
| 45 | 30 | RET |
| 46 | −SYM_FUNC_END(clear_page_orig) | |
| 47 | −EXPORT_SYMBOL_GPL(clear_page_orig) | |
| 48 | − | |
| 49 | −SYM_TYPED_FUNC_START(clear_page_erms) | |
| 50 | − movl $4096,%ecx | |
| 51 | − xorl %eax,%eax | |
| 52 | − rep stosb | |
| 53 | − RET | |
| 54 | −SYM_FUNC_END(clear_page_erms) | |
| 55 | −EXPORT_SYMBOL_GPL(clear_page_erms) | |
| 31 | +SYM_FUNC_END(__clear_pages_unrolled) | |
| 32 | +EXPORT_SYMBOL_GPL(__clear_pages_unrolled) | |
| 56 | 33 | |
| 57 | 34 | /* |
| 58 | 35 | * Default clear user-space. |
| @@ -42,3 +42,11 @@ static __init int gigantic_pages_init(void) · 41 unmodified lines | ||
| 42 | 42 | arch_initcall(gigantic_pages_init); |
| 43 | 43 | #endif |
| 44 | 44 | #endif |
| 45 | + | |
| 46 | +unsigned int __init arch_hugetlb_cma_order(void) | |
| 47 | +{ | |
| 48 | + if (boot_cpu_has(X86_FEATURE_GBPAGES)) | |
| 49 | + return PUD_SHIFT - PAGE_SHIFT; | |
| 50 | + | |
| 51 | + return 0; | |
| 52 | +} | |
| @@ -996,12 +996,8 @@ void __init free_initrd_mem(unsigned long start, unsigned long end) · 995 unmodified lines | ||
| 996 | 996 | } |
| 997 | 997 | #endif |
| 998 | 998 | |
| 999 | −void __init zone_sizes_init(void) | |
| 999 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 1000 | 1000 | { |
| 1001 | − unsigned long max_zone_pfns[MAX_NR_ZONES]; | |
| 1002 | − | |
| 1003 | − memset(max_zone_pfns, 0, sizeof(max_zone_pfns)); | |
| 1004 | − | |
| 1005 | 1001 | #ifdef CONFIG_ZONE_DMA |
| 1006 | 1002 | max_zone_pfns[ZONE_DMA] = min(MAX_DMA_PFN, max_low_pfn); |
| 1007 | 1003 | #endif |
| @@ -1012,8 +1008,6 @@ void __init zone_sizes_init(void) · 4 unmodified lines | ||
| 1012 | 1008 | #ifdef CONFIG_HIGHMEM |
| 1013 | 1009 | max_zone_pfns[ZONE_HIGHMEM] = max_pfn; |
| 1014 | 1010 | #endif |
| 1015 | − | |
| 1016 | − free_area_init(max_zone_pfns); | |
| 1017 | 1011 | } |
| 1018 | 1012 | |
| 1019 | 1013 | __visible DEFINE_PER_CPU_ALIGNED(struct tlb_state, cpu_tlbstate) = { |
| @@ -654,8 +654,6 @@ void __init paging_init(void) · 653 unmodified lines | ||
| 654 | 654 | * NOTE: at this point the bootmem allocator is fully available. |
| 655 | 655 | */ |
| 656 | 656 | olpc_dt_build_devicetree(); |
| 657 | − sparse_init(); | |
| 658 | − zone_sizes_init(); | |
| 659 | 657 | } |
| 660 | 658 | |
| 661 | 659 | /* |
| @@ -833,8 +833,6 @@ void __init initmem_init(void) · 832 unmodified lines | ||
| 833 | 833 | |
| 834 | 834 | void __init paging_init(void) |
| 835 | 835 | { |
| 836 | − sparse_init(); | |
| 837 | − | |
| 838 | 836 | /* |
| 839 | 837 | * clear the default setting with node 0 |
| 840 | 838 | * note: don't use nodes_clear here, that is really clearing when |
| @@ -843,8 +841,6 @@ void __init paging_init(void) · 2 unmodified lines | ||
| 843 | 841 | */ |
| 844 | 842 | node_clear_state(0, N_MEMORY); |
| 845 | 843 | node_clear_state(0, N_NORMAL_MEMORY); |
| 846 | − | |
| 847 | − zone_sizes_init(); | |
| 848 | 844 | } |
| 849 | 845 | |
| 850 | 846 | #define PAGE_UNUSED 0xFD |
| @@ -17,7 +17,6 @@ unsigned long kernel_physical_mapping_init(unsigned long start, · 16 unmodified lines | ||
| 17 | 17 | unsigned long kernel_physical_mapping_change(unsigned long start, |
| 18 | 18 | unsigned long end, |
| 19 | 19 | unsigned long page_size_mask); |
| 20 | −void zone_sizes_init(void); | |
| 21 | 20 | |
| 22 | 21 | extern int after_bootmem; |
| 23 | 22 | |
| @@ -426,7 +426,6 @@ static void xen_start_context_switch(struct task_struct *prev) · 425 unmodified lines | ||
| 426 | 426 | |
| 427 | 427 | if (this_cpu_read(xen_lazy_mode) == XEN_LAZY_MMU) { |
| 428 | 428 | arch_leave_lazy_mmu_mode(); |
| 429 | − set_ti_thread_flag(task_thread_info(prev), TIF_LAZY_MMU_UPDATES); | |
| 430 | 429 | } |
| 431 | 430 | enter_lazy(XEN_LAZY_CPU); |
| 432 | 431 | } |
| @@ -437,7 +436,7 @@ static void xen_end_context_switch(struct task_struct *next) · 4 unmodified lines | ||
| 437 | 436 | |
| 438 | 437 | xen_mc_flush(); |
| 439 | 438 | leave_lazy(XEN_LAZY_CPU); |
| 440 | − if (test_and_clear_ti_thread_flag(task_thread_info(next), TIF_LAZY_MMU_UPDATES)) | |
| 439 | + if (__task_lazy_mmu_mode_active(next)) | |
| 441 | 440 | arch_enter_lazy_mmu_mode(); |
| 442 | 441 | } |
| 443 | 442 | |
| @@ -2139,10 +2139,8 @@ static void xen_flush_lazy_mmu(void) · 2138 unmodified lines | ||
| 2139 | 2139 | { |
| 2140 | 2140 | preempt_disable(); |
| 2141 | 2141 | |
| 2142 | − if (xen_get_lazy_mode() == XEN_LAZY_MMU) { | |
| 2143 | − arch_leave_lazy_mmu_mode(); | |
| 2144 | − arch_enter_lazy_mmu_mode(); | |
| 2145 | − } | |
| 2142 | + if (xen_get_lazy_mode() == XEN_LAZY_MMU) | |
| 2143 | + xen_mc_flush(); | |
| 2146 | 2144 | |
| 2147 | 2145 | preempt_enable(); |
| 2148 | 2146 | } |
| @@ -126,7 +126,6 @@ void clear_user_highpage(struct page *page, unsigned long vaddr); · 125 unmodified lines | ||
| 126 | 126 | void copy_user_highpage(struct page *to, struct page *from, |
| 127 | 127 | unsigned long vaddr, struct vm_area_struct *vma); |
| 128 | 128 | #else |
| 129 | −# define clear_user_page(page, vaddr, pg) clear_page(page) | |
| 130 | 129 | # define copy_user_page(to, from, vaddr, pg) copy_page(to, from) |
| 131 | 130 | #endif |
| 132 | 131 | |
| @@ -116,16 +116,16 @@ static void __init print_vm_layout(void) · 115 unmodified lines | ||
| 116 | 116 | (unsigned long)(__bss_stop - __bss_start) >> 10); |
| 117 | 117 | } |
| 118 | 118 | |
| 119 | −void __init zones_init(void) | |
| 119 | +void __init arch_zone_limits_init(unsigned long *max_zone_pfns) | |
| 120 | 120 | { |
| 121 | − /* All pages are DMA-able, so we put them all in the DMA zone. */ | |
| 122 | − unsigned long max_zone_pfn[MAX_NR_ZONES] = { | |
| 123 | − [ZONE_NORMAL] = max_low_pfn, | |
| 121 | + max_zone_pfns[ZONE_NORMAL] = max_low_pfn; | |
| 124 | 122 | #ifdef CONFIG_HIGHMEM |
| 125 | − [ZONE_HIGHMEM] = max_pfn, | |
| 123 | + max_zone_pfns[ZONE_HIGHMEM] = max_pfn; | |
| 126 | 124 | #endif |
| 127 | − }; | |
| 128 | − free_area_init(max_zone_pfn); | |
| 125 | +} | |
| 126 | + | |
| 127 | +void __init zones_init(void) | |
| 128 | +{ | |
| 129 | 129 | print_vm_layout(); |
| 130 | 130 | } |
| 131 | 131 | |
| @@ -823,9 +823,6 @@ void blk_mq_free_request(struct request *rq) · 822 unmodified lines | ||
| 823 | 823 | |
| 824 | 824 | blk_mq_finish_request(rq); |
| 825 | 825 | |
| 826 | − if (unlikely(laptop_mode && !blk_rq_is_passthrough(rq))) | |
| 827 | − laptop_io_completion(q->disk->bdi); | |
| 828 | − | |
| 829 | 826 | rq_qos_done(q, rq); |
| 830 | 827 | |
| 831 | 828 | WRITE_ONCE(rq->state, MQ_RQ_IDLE); |
| @@ -12,8 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | * |
| 13 | 13 | */ |
| 14 | 14 | |
| 15 | −#define KMSG_COMPONENT "zram" | |
| 16 | −#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt | |
| 15 | +#define pr_fmt(fmt) "zram: " fmt | |
| 17 | 16 | |
| 18 | 17 | #include <linux/module.h> |
| 19 | 18 | #include <linux/kernel.h> |
| @@ -56,13 +55,10 @@ static size_t huge_class_size; · 36 unmodified lines | ||
| 56 | 55 | |
| 57 | 56 | static const struct block_device_operations zram_devops; |
| 58 | 57 | |
| 59 | −static void zram_free_page(struct zram *zram, size_t index); | |
| 60 | −static int zram_read_from_zspool(struct zram *zram, struct page *page, | |
| 61 | − u32 index); | |
| 62 | − | |
| 58 | +static void slot_free(struct zram *zram, u32 index); | |
| 63 | 59 | #define slot_dep_map(zram, index) (&(zram)->table[(index)].dep_map) |
| 64 | 60 | |
| 65 | −static void zram_slot_lock_init(struct zram *zram, u32 index) | |
| 61 | +static void slot_lock_init(struct zram *zram, u32 index) | |
| 66 | 62 | { |
| 67 | 63 | static struct lock_class_key __key; |
| 68 | 64 | |
| @@ -82,9 +78,9 @@ static void zram_slot_lock_init(struct zram *zram, u32 index) · 13 unmodified lines | ||
| 82 | 78 | * 4) Use TRY lock variant when in atomic context |
| 83 | 79 | * - must check return value and handle locking failers |
| 84 | 80 | */ |
| 85 | −static __must_check bool zram_slot_trylock(struct zram *zram, u32 index) | |
| 81 | +static __must_check bool slot_trylock(struct zram *zram, u32 index) | |
| 86 | 82 | { |
| 87 | − unsigned long *lock = &zram->table[index].flags; | |
| 83 | + unsigned long *lock = &zram->table[index].__lock; | |
| 88 | 84 | |
| 89 | 85 | if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) { |
| 90 | 86 | mutex_acquire(slot_dep_map(zram, index), 0, 1, _RET_IP_); |
| @@ -95,18 +91,18 @@ static __must_check bool zram_slot_trylock(struct zram *zram, u32 index) · 4 unmodified lines | ||
| 95 | 91 | return false; |
| 96 | 92 | } |
| 97 | 93 | |
| 98 | −static void zram_slot_lock(struct zram *zram, u32 index) | |
| 94 | +static void slot_lock(struct zram *zram, u32 index) | |
| 99 | 95 | { |
| 100 | − unsigned long *lock = &zram->table[index].flags; | |
| 96 | + unsigned long *lock = &zram->table[index].__lock; | |
| 101 | 97 | |
| 102 | 98 | mutex_acquire(slot_dep_map(zram, index), 0, 0, _RET_IP_); |
| 103 | 99 | wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); |
| 104 | 100 | lock_acquired(slot_dep_map(zram, index), _RET_IP_); |
| 105 | 101 | } |
| 106 | 102 | |
| 107 | −static void zram_slot_unlock(struct zram *zram, u32 index) | |
| 103 | +static void slot_unlock(struct zram *zram, u32 index) | |
| 108 | 104 | { |
| 109 | − unsigned long *lock = &zram->table[index].flags; | |
| 105 | + unsigned long *lock = &zram->table[index].__lock; | |
| 110 | 106 | |
| 111 | 107 | mutex_release(slot_dep_map(zram, index), _RET_IP_); |
| 112 | 108 | clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); |
| @@ -122,52 +118,80 @@ static inline struct zram *dev_to_zram(struct device *dev) · 9 unmodified lines | ||
| 122 | 118 | return (struct zram *)dev_to_disk(dev)->private_data; |
| 123 | 119 | } |
| 124 | 120 | |
| 125 | −static unsigned long zram_get_handle(struct zram *zram, u32 index) | |
| 121 | +static unsigned long get_slot_handle(struct zram *zram, u32 index) | |
| 126 | 122 | { |
| 127 | 123 | return zram->table[index].handle; |
| 128 | 124 | } |
| 129 | 125 | |
| 130 | −static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) | |
| 126 | +static void set_slot_handle(struct zram *zram, u32 index, unsigned long handle) | |
| 131 | 127 | { |
| 132 | 128 | zram->table[index].handle = handle; |
| 133 | 129 | } |
| 134 | 130 | |
| 135 | −static bool zram_test_flag(struct zram *zram, u32 index, | |
| 136 | − enum zram_pageflags flag) | |
| 131 | +static bool test_slot_flag(struct zram *zram, u32 index, | |
| 132 | + enum zram_pageflags flag) | |
| 137 | 133 | { |
| 138 | − return zram->table[index].flags & BIT(flag); | |
| 134 | + return zram->table[index].attr.flags & BIT(flag); | |
| 139 | 135 | } |
| 140 | 136 | |
| 141 | −static void zram_set_flag(struct zram *zram, u32 index, | |
| 142 | − enum zram_pageflags flag) | |
| 137 | +static void set_slot_flag(struct zram *zram, u32 index, | |
| 138 | + enum zram_pageflags flag) | |
| 143 | 139 | { |
| 144 | − zram->table[index].flags |= BIT(flag); | |
| 140 | + zram->table[index].attr.flags |= BIT(flag); | |
| 145 | 141 | } |
| 146 | 142 | |
| 147 | −static void zram_clear_flag(struct zram *zram, u32 index, | |
| 148 | − enum zram_pageflags flag) | |
| 143 | +static void clear_slot_flag(struct zram *zram, u32 index, | |
| 144 | + enum zram_pageflags flag) | |
| 149 | 145 | { |
| 150 | − zram->table[index].flags &= ~BIT(flag); | |
| 146 | + zram->table[index].attr.flags &= ~BIT(flag); | |
| 151 | 147 | } |
| 152 | 148 | |
| 153 | −static size_t zram_get_obj_size(struct zram *zram, u32 index) | |
| 149 | +static size_t get_slot_size(struct zram *zram, u32 index) | |
| 154 | 150 | { |
| 155 | − return zram->table[index].flags & (BIT(ZRAM_FLAG_SHIFT) - 1); | |
| 151 | + return zram->table[index].attr.flags & (BIT(ZRAM_FLAG_SHIFT) - 1); | |
| 156 | 152 | } |
| 157 | 153 | |
| 158 | −static void zram_set_obj_size(struct zram *zram, | |
| 159 | − u32 index, size_t size) | |
| 154 | +static void set_slot_size(struct zram *zram, u32 index, size_t size) | |
| 160 | 155 | { |
| 161 | − unsigned long flags = zram->table[index].flags >> ZRAM_FLAG_SHIFT; | |
| 156 | + unsigned long flags = zram->table[index].attr.flags >> ZRAM_FLAG_SHIFT; | |
| 157 | + | |
| 158 | + zram->table[index].attr.flags = (flags << ZRAM_FLAG_SHIFT) | size; | |
| 159 | +} | |
| 162 | 160 | |
| 163 | − zram->table[index].flags = (flags << ZRAM_FLAG_SHIFT) | size; | |
| 161 | +static inline bool slot_allocated(struct zram *zram, u32 index) | |
| 162 | +{ | |
| 163 | + return get_slot_size(zram, index) || | |
| 164 | + test_slot_flag(zram, index, ZRAM_SAME) || | |
| 165 | + test_slot_flag(zram, index, ZRAM_WB); | |
| 166 | +} | |
| 167 | + | |
| 168 | +static inline void set_slot_comp_priority(struct zram *zram, u32 index, | |
| 169 | + u32 prio) | |
| 170 | +{ | |
| 171 | + prio &= ZRAM_COMP_PRIORITY_MASK; | |
| 172 | + /* | |
| 173 | + * Clear previous priority value first, in case if we recompress | |
| 174 | + * further an already recompressed page | |
| 175 | + */ | |
| 176 | + zram->table[index].attr.flags &= ~(ZRAM_COMP_PRIORITY_MASK << | |
| 177 | + ZRAM_COMP_PRIORITY_BIT1); | |
| 178 | + zram->table[index].attr.flags |= (prio << ZRAM_COMP_PRIORITY_BIT1); | |
| 164 | 179 | } |
| 165 | 180 | |
| 166 | −static inline bool zram_allocated(struct zram *zram, u32 index) | |
| 181 | +static inline u32 get_slot_comp_priority(struct zram *zram, u32 index) | |
| 167 | 182 | { |
| 168 | − return zram_get_obj_size(zram, index) || | |
| 169 | − zram_test_flag(zram, index, ZRAM_SAME) || | |
| 170 | − zram_test_flag(zram, index, ZRAM_WB); | |
| 183 | + u32 prio = zram->table[index].attr.flags >> ZRAM_COMP_PRIORITY_BIT1; | |
| 184 | + | |
| 185 | + return prio & ZRAM_COMP_PRIORITY_MASK; | |
| 186 | +} | |
| 187 | + | |
| 188 | +static void mark_slot_accessed(struct zram *zram, u32 index) | |
| 189 | +{ | |
| 190 | + clear_slot_flag(zram, index, ZRAM_IDLE); | |
| 191 | + clear_slot_flag(zram, index, ZRAM_PP_SLOT); | |
| 192 | +#ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME | |
| 193 | + zram->table[index].attr.ac_time = (u32)ktime_get_boottime_seconds(); | |
| 194 | +#endif | |
| 171 | 195 | } |
| 172 | 196 | |
| 173 | 197 | static inline void update_used_max(struct zram *zram, const unsigned long pages) |
| @@ -204,34 +228,6 @@ static inline bool is_partial_io(struct bio_vec *bvec) · 30 unmodified lines | ||
| 204 | 228 | } |
| 205 | 229 | #endif |
| 206 | 230 | |
| 207 | −static inline void zram_set_priority(struct zram *zram, u32 index, u32 prio) | |
| 208 | −{ | |
| 209 | − prio &= ZRAM_COMP_PRIORITY_MASK; | |
| 210 | − /* | |
| 211 | − * Clear previous priority value first, in case if we recompress | |
| 212 | − * further an already recompressed page | |
| 213 | − */ | |
| 214 | − zram->table[index].flags &= ~(ZRAM_COMP_PRIORITY_MASK << | |
| 215 | − ZRAM_COMP_PRIORITY_BIT1); | |
| 216 | − zram->table[index].flags |= (prio << ZRAM_COMP_PRIORITY_BIT1); | |
| 217 | −} | |
| 218 | − | |
| 219 | −static inline u32 zram_get_priority(struct zram *zram, u32 index) | |
| 220 | −{ | |
| 221 | − u32 prio = zram->table[index].flags >> ZRAM_COMP_PRIORITY_BIT1; | |
| 222 | − | |
| 223 | − return prio & ZRAM_COMP_PRIORITY_MASK; | |
| 224 | −} | |
| 225 | − | |
| 226 | −static void zram_accessed(struct zram *zram, u32 index) | |
| 227 | −{ | |
| 228 | − zram_clear_flag(zram, index, ZRAM_IDLE); | |
| 229 | − zram_clear_flag(zram, index, ZRAM_PP_SLOT); | |
| 230 | −#ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME | |
| 231 | − zram->table[index].ac_time = ktime_get_boottime(); | |
| 232 | −#endif | |
| 233 | −} | |
| 234 | − | |
| 235 | 231 | #if defined CONFIG_ZRAM_WRITEBACK || defined CONFIG_ZRAM_MULTI_COMP |
| 236 | 232 | struct zram_pp_slot { |
| 237 | 233 | unsigned long index; |
| @@ -267,9 +263,9 @@ static void release_pp_slot(struct zram *zram, struct zram_pp_slot *pps) · 29 unmodified lines | ||
| 267 | 263 | { |
| 268 | 264 | list_del_init(&pps->entry); |
| 269 | 265 | |
| 270 | − zram_slot_lock(zram, pps->index); | |
| 271 | − zram_clear_flag(zram, pps->index, ZRAM_PP_SLOT); | |
| 272 | − zram_slot_unlock(zram, pps->index); | |
| 266 | + slot_lock(zram, pps->index); | |
| 267 | + clear_slot_flag(zram, pps->index, ZRAM_PP_SLOT); | |
| 268 | + slot_unlock(zram, pps->index); | |
| 273 | 269 | |
| 274 | 270 | kfree(pps); |
| 275 | 271 | } |
| @@ -308,10 +304,10 @@ static bool place_pp_slot(struct zram *zram, struct zram_pp_ctl *ctl, · 32 unmodified lines | ||
| 308 | 304 | INIT_LIST_HEAD(&pps->entry); |
| 309 | 305 | pps->index = index; |
| 310 | 306 | |
| 311 | − bid = zram_get_obj_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE; | |
| 307 | + bid = get_slot_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE; | |
| 312 | 308 | list_add(&pps->entry, &ctl->pp_buckets[bid]); |
| 313 | 309 | |
| 314 | − zram_set_flag(zram, pps->index, ZRAM_PP_SLOT); | |
| 310 | + set_slot_flag(zram, pps->index, ZRAM_PP_SLOT); | |
| 315 | 311 | return true; |
| 316 | 312 | } |
| 317 | 313 | |
| @@ -363,15 +359,14 @@ static bool page_same_filled(void *ptr, unsigned long *element) · 45 unmodified lines | ||
| 363 | 359 | return true; |
| 364 | 360 | } |
| 365 | 361 | |
| 366 | −static ssize_t initstate_show(struct device *dev, | |
| 367 | − struct device_attribute *attr, char *buf) | |
| 362 | +static ssize_t initstate_show(struct device *dev, struct device_attribute *attr, | |
| 363 | + char *buf) | |
| 368 | 364 | { |
| 369 | 365 | u32 val; |
| 370 | 366 | struct zram *zram = dev_to_zram(dev); |
| 371 | 367 | |
| 372 | − down_read(&zram->init_lock); | |
| 368 | + guard(rwsem_read)(&zram->dev_lock); | |
| 373 | 369 | val = init_done(zram); |
| 374 | − up_read(&zram->init_lock); | |
| 375 | 370 | |
| 376 | 371 | return sysfs_emit(buf, "%u\n", val); |
| 377 | 372 | } |
| @@ -385,7 +380,8 @@ static ssize_t disksize_show(struct device *dev, · 7 unmodified lines | ||
| 385 | 380 | } |
| 386 | 381 | |
| 387 | 382 | static ssize_t mem_limit_store(struct device *dev, |
| 388 | − struct device_attribute *attr, const char *buf, size_t len) | |
| 383 | + struct device_attribute *attr, const char *buf, | |
| 384 | + size_t len) | |
| 389 | 385 | { |
| 390 | 386 | u64 limit; |
| 391 | 387 | char *tmp; |
| @@ -395,15 +391,15 @@ static ssize_t mem_limit_store(struct device *dev, · 3 unmodified lines | ||
| 395 | 391 | if (buf == tmp) /* no chars parsed, invalid input */ |
| 396 | 392 | return -EINVAL; |
| 397 | 393 | |
| 398 | − down_write(&zram->init_lock); | |
| 394 | + guard(rwsem_write)(&zram->dev_lock); | |
| 399 | 395 | zram->limit_pages = PAGE_ALIGN(limit) >> PAGE_SHIFT; |
| 400 | − up_write(&zram->init_lock); | |
| 401 | 396 | |
| 402 | 397 | return len; |
| 403 | 398 | } |
| 404 | 399 | |
| 405 | 400 | static ssize_t mem_used_max_store(struct device *dev, |
| 406 | − struct device_attribute *attr, const char *buf, size_t len) | |
| 401 | + struct device_attribute *attr, | |
| 402 | + const char *buf, size_t len) | |
| 407 | 403 | { |
| 408 | 404 | int err; |
| 409 | 405 | unsigned long val; |
| @@ -413,12 +409,11 @@ static ssize_t mem_used_max_store(struct device *dev, · 3 unmodified lines | ||
| 413 | 409 | if (err || val != 0) |
| 414 | 410 | return -EINVAL; |
| 415 | 411 | |
| 416 | − down_read(&zram->init_lock); | |
| 412 | + guard(rwsem_read)(&zram->dev_lock); | |
| 417 | 413 | if (init_done(zram)) { |
| 418 | 414 | atomic_long_set(&zram->stats.max_used_pages, |
| 419 | 415 | zs_get_total_pages(zram->mem_pool)); |
| 420 | 416 | } |
| 421 | − up_read(&zram->init_lock); | |
| 422 | 417 | |
| 423 | 418 | return len; |
| 424 | 419 | } |
| @@ -441,67 +436,66 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) · 16 unmodified lines | ||
| 441 | 436 | * |
| 442 | 437 | * And ZRAM_WB slots simply cannot be ZRAM_IDLE. |
| 443 | 438 | */ |
| 444 | − zram_slot_lock(zram, index); | |
| 445 | − if (!zram_allocated(zram, index) || | |
| 446 | − zram_test_flag(zram, index, ZRAM_WB) || | |
| 447 | − zram_test_flag(zram, index, ZRAM_SAME)) { | |
| 448 | − zram_slot_unlock(zram, index); | |
| 439 | + slot_lock(zram, index); | |
| 440 | + if (!slot_allocated(zram, index) || | |
| 441 | + test_slot_flag(zram, index, ZRAM_WB) || | |
| 442 | + test_slot_flag(zram, index, ZRAM_SAME)) { | |
| 443 | + slot_unlock(zram, index); | |
| 449 | 444 | continue; |
| 450 | 445 | } |
| 451 | 446 | |
| 452 | 447 | #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME |
| 453 | 448 | is_idle = !cutoff || |
| 454 | − ktime_after(cutoff, zram->table[index].ac_time); | |
| 449 | + ktime_after(cutoff, zram->table[index].attr.ac_time); | |
| 455 | 450 | #endif |
| 456 | 451 | if (is_idle) |
| 457 | − zram_set_flag(zram, index, ZRAM_IDLE); | |
| 452 | + set_slot_flag(zram, index, ZRAM_IDLE); | |
| 458 | 453 | else |
| 459 | − zram_clear_flag(zram, index, ZRAM_IDLE); | |
| 460 | − zram_slot_unlock(zram, index); | |
| 454 | + clear_slot_flag(zram, index, ZRAM_IDLE); | |
| 455 | + slot_unlock(zram, index); | |
| 461 | 456 | } |
| 462 | 457 | } |
| 463 | 458 | |
| 464 | −static ssize_t idle_store(struct device *dev, | |
| 465 | − struct device_attribute *attr, const char *buf, size_t len) | |
| 459 | +static ssize_t idle_store(struct device *dev, struct device_attribute *attr, | |
| 460 | + const char *buf, size_t len) | |
| 466 | 461 | { |
| 467 | 462 | struct zram *zram = dev_to_zram(dev); |
| 468 | − ktime_t cutoff_time = 0; | |
| 469 | − ssize_t rv = -EINVAL; | |
| 463 | + ktime_t cutoff = 0; | |
| 470 | 464 | |
| 471 | 465 | if (!sysfs_streq(buf, "all")) { |
| 472 | 466 | /* |
| 473 | 467 | * If it did not parse as 'all' try to treat it as an integer |
| 474 | 468 | * when we have memory tracking enabled. |
| 475 | 469 | */ |
| 476 | − u64 age_sec; | |
| 470 | + u32 age_sec; | |
| 477 | 471 | |
| 478 | − if (IS_ENABLED(CONFIG_ZRAM_TRACK_ENTRY_ACTIME) && !kstrtoull(buf, 0, &age_sec)) | |
| 479 | − cutoff_time = ktime_sub(ktime_get_boottime(), | |
| 480 | − ns_to_ktime(age_sec * NSEC_PER_SEC)); | |
| 472 | + if (IS_ENABLED(CONFIG_ZRAM_TRACK_ENTRY_ACTIME) && | |
| 473 | + !kstrtouint(buf, 0, &age_sec)) | |
| 474 | + cutoff = ktime_sub((u32)ktime_get_boottime_seconds(), | |
| 475 | + age_sec); | |
| 481 | 476 | else |
| 482 | − goto out; | |
| 477 | + return -EINVAL; | |
| 483 | 478 | } |
| 484 | 479 | |
| 485 | − down_read(&zram->init_lock); | |
| 480 | + guard(rwsem_read)(&zram->dev_lock); | |
| 486 | 481 | if (!init_done(zram)) |
| 487 | − goto out_unlock; | |
| 482 | + return -EINVAL; | |
| 488 | 483 | |
| 489 | 484 | /* |
| 490 | − * A cutoff_time of 0 marks everything as idle, this is the | |
| 485 | + * A cutoff of 0 marks everything as idle, this is the | |
| 491 | 486 | * "all" behavior. |
| 492 | 487 | */ |
| 493 | − mark_idle(zram, cutoff_time); | |
| 494 | − rv = len; | |
| 495 | − | |
| 496 | −out_unlock: | |
| 497 | − up_read(&zram->init_lock); | |
| 498 | −out: | |
| 499 | − return rv; | |
| 488 | + mark_idle(zram, cutoff); | |
| 489 | + return len; | |
| 500 | 490 | } |
| 501 | 491 | |
| 502 | 492 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 503 | 493 | #define INVALID_BDEV_BLOCK (~0UL) |
| 504 | 494 | |
| 495 | +static int read_from_zspool_raw(struct zram *zram, struct page *page, | |
| 496 | + u32 index); | |
| 497 | +static int read_from_zspool(struct zram *zram, struct page *page, u32 index); | |
| 498 | + | |
| 505 | 499 | struct zram_wb_ctl { |
| 506 | 500 | /* idle list is accessed only by the writeback task, no concurency */ |
| 507 | 501 | struct list_head idle_reqs; |
| @@ -522,23 +516,86 @@ struct zram_wb_req { · 14 unmodified lines | ||
| 522 | 516 | struct list_head entry; |
| 523 | 517 | }; |
| 524 | 518 | |
| 519 | +struct zram_rb_req { | |
| 520 | + struct work_struct work; | |
| 521 | + struct zram *zram; | |
| 522 | + struct page *page; | |
| 523 | + /* The read bio for backing device */ | |
| 524 | + struct bio *bio; | |
| 525 | + unsigned long blk_idx; | |
| 526 | + union { | |
| 527 | + /* The original bio to complete (async read) */ | |
| 528 | + struct bio *parent; | |
| 529 | + /* error status (sync read) */ | |
| 530 | + int error; | |
| 531 | + }; | |
| 532 | + u32 index; | |
| 533 | +}; | |
| 534 | + | |
| 535 | +#define FOUR_K(x) ((x) * (1 << (PAGE_SHIFT - 12))) | |
| 536 | +static ssize_t bd_stat_show(struct device *dev, struct device_attribute *attr, | |
| 537 | + char *buf) | |
| 538 | +{ | |
| 539 | + struct zram *zram = dev_to_zram(dev); | |
| 540 | + ssize_t ret; | |
| 541 | + | |
| 542 | + guard(rwsem_read)(&zram->dev_lock); | |
| 543 | + ret = sysfs_emit(buf, | |
| 544 | + "%8llu %8llu %8llu\n", | |
| 545 | + FOUR_K((u64)atomic64_read(&zram->stats.bd_count)), | |
| 546 | + FOUR_K((u64)atomic64_read(&zram->stats.bd_reads)), | |
| 547 | + FOUR_K((u64)atomic64_read(&zram->stats.bd_writes))); | |
| 548 | + | |
| 549 | + return ret; | |
| 550 | +} | |
| 551 | + | |
| 552 | +static ssize_t writeback_compressed_store(struct device *dev, | |
| 553 | + struct device_attribute *attr, | |
| 554 | + const char *buf, size_t len) | |
| 555 | +{ | |
| 556 | + struct zram *zram = dev_to_zram(dev); | |
| 557 | + bool val; | |
| 558 | + | |
| 559 | + if (kstrtobool(buf, &val)) | |
| 560 | + return -EINVAL; | |
| 561 | + | |
| 562 | + guard(rwsem_write)(&zram->dev_lock); | |
| 563 | + if (init_done(zram)) { | |
| 564 | + return -EBUSY; | |
| 565 | + } | |
| 566 | + | |
| 567 | + zram->wb_compressed = val; | |
| 568 | + | |
| 569 | + return len; | |
| 570 | +} | |
| 571 | + | |
| 572 | +static ssize_t writeback_compressed_show(struct device *dev, | |
| 573 | + struct device_attribute *attr, | |
| 574 | + char *buf) | |
| 575 | +{ | |
| 576 | + bool val; | |
| 577 | + struct zram *zram = dev_to_zram(dev); | |
| 578 | + | |
| 579 | + guard(rwsem_read)(&zram->dev_lock); | |
| 580 | + val = zram->wb_compressed; | |
| 581 | + | |
| 582 | + return sysfs_emit(buf, "%d\n", val); | |
| 583 | +} | |
| 584 | + | |
| 525 | 585 | static ssize_t writeback_limit_enable_store(struct device *dev, |
| 526 | 586 | struct device_attribute *attr, |
| 527 | 587 | const char *buf, size_t len) |
| 528 | 588 | { |
| 529 | 589 | struct zram *zram = dev_to_zram(dev); |
| 530 | 590 | u64 val; |
| 531 | − ssize_t ret = -EINVAL; | |
| 532 | 591 | |
| 533 | 592 | if (kstrtoull(buf, 10, &val)) |
| 534 | − return ret; | |
| 593 | + return -EINVAL; | |
| 535 | 594 | |
| 536 | − down_write(&zram->init_lock); | |
| 595 | + guard(rwsem_write)(&zram->dev_lock); | |
| 537 | 596 | zram->wb_limit_enable = val; |
| 538 | − up_write(&zram->init_lock); | |
| 539 | − ret = len; | |
| 540 | 597 | |
| 541 | − return ret; | |
| 598 | + return len; | |
| 542 | 599 | } |
| 543 | 600 | |
| 544 | 601 | static ssize_t writeback_limit_enable_show(struct device *dev, |
| @@ -548,9 +605,8 @@ static ssize_t writeback_limit_enable_show(struct device *dev, · 3 unmodified lines | ||
| 548 | 605 | bool val; |
| 549 | 606 | struct zram *zram = dev_to_zram(dev); |
| 550 | 607 | |
| 551 | − down_read(&zram->init_lock); | |
| 608 | + guard(rwsem_read)(&zram->dev_lock); | |
| 552 | 609 | val = zram->wb_limit_enable; |
| 553 | − up_read(&zram->init_lock); | |
| 554 | 610 | |
| 555 | 611 | return sysfs_emit(buf, "%d\n", val); |
| 556 | 612 | } |
| @@ -561,10 +617,9 @@ static ssize_t writeback_limit_store(struct device *dev, · 4 unmodified lines | ||
| 561 | 617 | { |
| 562 | 618 | struct zram *zram = dev_to_zram(dev); |
| 563 | 619 | u64 val; |
| 564 | − ssize_t ret = -EINVAL; | |
| 565 | 620 | |
| 566 | 621 | if (kstrtoull(buf, 10, &val)) |
| 567 | − return ret; | |
| 622 | + return -EINVAL; | |
| 568 | 623 | |
| 569 | 624 | /* |
| 570 | 625 | * When the page size is greater than 4KB, if bd_wb_limit is set to |
| @@ -576,12 +631,10 @@ static ssize_t writeback_limit_store(struct device *dev, · 5 unmodified lines | ||
| 576 | 631 | */ |
| 577 | 632 | val = rounddown(val, PAGE_SIZE / 4096); |
| 578 | 633 | |
| 579 | − down_write(&zram->init_lock); | |
| 634 | + guard(rwsem_write)(&zram->dev_lock); | |
| 580 | 635 | zram->bd_wb_limit = val; |
| 581 | − up_write(&zram->init_lock); | |
| 582 | − ret = len; | |
| 583 | 636 | |
| 584 | − return ret; | |
| 637 | + return len; | |
| 585 | 638 | } |
| 586 | 639 | |
| 587 | 640 | static ssize_t writeback_limit_show(struct device *dev, |
| @@ -590,9 +643,8 @@ static ssize_t writeback_limit_show(struct device *dev, · 2 unmodified lines | ||
| 590 | 643 | u64 val; |
| 591 | 644 | struct zram *zram = dev_to_zram(dev); |
| 592 | 645 | |
| 593 | − down_read(&zram->init_lock); | |
| 646 | + guard(rwsem_read)(&zram->dev_lock); | |
| 594 | 647 | val = zram->bd_wb_limit; |
| 595 | − up_read(&zram->init_lock); | |
| 596 | 648 | |
| 597 | 649 | return sysfs_emit(buf, "%llu\n", val); |
| 598 | 650 | } |
| @@ -610,9 +662,8 @@ static ssize_t writeback_batch_size_store(struct device *dev, · 11 unmodified lines | ||
| 610 | 662 | if (!val) |
| 611 | 663 | return -EINVAL; |
| 612 | 664 | |
| 613 | − down_write(&zram->init_lock); | |
| 665 | + guard(rwsem_write)(&zram->dev_lock); | |
| 614 | 666 | zram->wb_batch_size = val; |
| 615 | − up_write(&zram->init_lock); | |
| 616 | 667 | |
| 617 | 668 | return len; |
| 618 | 669 | } |
| @@ -624,9 +675,8 @@ static ssize_t writeback_batch_size_show(struct device *dev, · 5 unmodified lines | ||
| 624 | 675 | u32 val; |
| 625 | 676 | struct zram *zram = dev_to_zram(dev); |
| 626 | 677 | |
| 627 | − down_read(&zram->init_lock); | |
| 678 | + guard(rwsem_read)(&zram->dev_lock); | |
| 628 | 679 | val = zram->wb_batch_size; |
| 629 | − up_read(&zram->init_lock); | |
| 630 | 680 | |
| 631 | 681 | return sysfs_emit(buf, "%u\n", val); |
| 632 | 682 | } |
| @@ -646,37 +696,33 @@ static void reset_bdev(struct zram *zram) · 13 unmodified lines | ||
| 646 | 696 | } |
| 647 | 697 | |
| 648 | 698 | static ssize_t backing_dev_show(struct device *dev, |
| 649 | − struct device_attribute *attr, char *buf) | |
| 699 | + struct device_attribute *attr, char *buf) | |
| 650 | 700 | { |
| 651 | 701 | struct file *file; |
| 652 | 702 | struct zram *zram = dev_to_zram(dev); |
| 653 | 703 | char *p; |
| 654 | 704 | ssize_t ret; |
| 655 | 705 | |
| 656 | − down_read(&zram->init_lock); | |
| 706 | + guard(rwsem_read)(&zram->dev_lock); | |
| 657 | 707 | file = zram->backing_dev; |
| 658 | 708 | if (!file) { |
| 659 | 709 | memcpy(buf, "none\n", 5); |
| 660 | − up_read(&zram->init_lock); | |
| 661 | 710 | return 5; |
| 662 | 711 | } |
| 663 | 712 | |
| 664 | 713 | p = file_path(file, buf, PAGE_SIZE - 1); |
| 665 | − if (IS_ERR(p)) { | |
| 666 | − ret = PTR_ERR(p); | |
| 667 | − goto out; | |
| 668 | − } | |
| 714 | + if (IS_ERR(p)) | |
| 715 | + return PTR_ERR(p); | |
| 669 | 716 | |
| 670 | 717 | ret = strlen(p); |
| 671 | 718 | memmove(buf, p, ret); |
| 672 | 719 | buf[ret++] = '\n'; |
| 673 | −out: | |
| 674 | − up_read(&zram->init_lock); | |
| 675 | 720 | return ret; |
| 676 | 721 | } |
| 677 | 722 | |
| 678 | 723 | static ssize_t backing_dev_store(struct device *dev, |
| 679 | − struct device_attribute *attr, const char *buf, size_t len) | |
| 724 | + struct device_attribute *attr, const char *buf, | |
| 725 | + size_t len) | |
| 680 | 726 | { |
| 681 | 727 | char *file_name; |
| 682 | 728 | size_t sz; |
| @@ -691,7 +737,7 @@ static ssize_t backing_dev_store(struct device *dev, · 8 unmodified lines | ||
| 691 | 737 | if (!file_name) |
| 692 | 738 | return -ENOMEM; |
| 693 | 739 | |
| 694 | − down_write(&zram->init_lock); | |
| 740 | + guard(rwsem_write)(&zram->dev_lock); | |
| 695 | 741 | if (init_done(zram)) { |
| 696 | 742 | pr_info("Can't setup backing device for initialized device\n"); |
| 697 | 743 | err = -EBUSY; |
| @@ -739,7 +785,6 @@ static ssize_t backing_dev_store(struct device *dev, · 41 unmodified lines | ||
| 739 | 785 | zram->backing_dev = backing_dev; |
| 740 | 786 | zram->bitmap = bitmap; |
| 741 | 787 | zram->nr_pages = nr_pages; |
| 742 | − up_write(&zram->init_lock); | |
| 743 | 788 | |
| 744 | 789 | pr_info("setup backing device %s\n", file_name); |
| 745 | 790 | kfree(file_name); |
| @@ -751,8 +796,6 @@ out: · 5 unmodified lines | ||
| 751 | 796 | if (backing_dev) |
| 752 | 797 | filp_close(backing_dev, NULL); |
| 753 | 798 | |
| 754 | − up_write(&zram->init_lock); | |
| 755 | − | |
| 756 | 799 | kfree(file_name); |
| 757 | 800 | |
| 758 | 801 | return err; |
| @@ -780,18 +823,6 @@ static void zram_release_bdev_block(struct zram *zram, unsigned long blk_idx) · 21 unmodified lines | ||
| 780 | 823 | atomic64_dec(&zram->stats.bd_count); |
| 781 | 824 | } |
| 782 | 825 | |
| 783 | −static void read_from_bdev_async(struct zram *zram, struct page *page, | |
| 784 | − unsigned long entry, struct bio *parent) | |
| 785 | −{ | |
| 786 | − struct bio *bio; | |
| 787 | − | |
| 788 | − bio = bio_alloc(zram->bdev, 1, parent->bi_opf, GFP_NOIO); | |
| 789 | − bio->bi_iter.bi_sector = entry * (PAGE_SIZE >> 9); | |
| 790 | − __bio_add_page(bio, page, PAGE_SIZE, 0); | |
| 791 | − bio_chain(bio, parent); | |
| 792 | − submit_bio(bio); | |
| 793 | −} | |
| 794 | − | |
| 795 | 826 | static void release_wb_req(struct zram_wb_req *req) |
| 796 | 827 | { |
| 797 | 828 | __free_page(req->page); |
| @@ -870,7 +901,7 @@ release_wb_ctl: · 72 unmodified lines | ||
| 870 | 901 | |
| 871 | 902 | static void zram_account_writeback_rollback(struct zram *zram) |
| 872 | 903 | { |
| 873 | − lockdep_assert_held_read(&zram->init_lock); | |
| 904 | + lockdep_assert_held_write(&zram->dev_lock); | |
| 874 | 905 | |
| 875 | 906 | if (zram->wb_limit_enable) |
| 876 | 907 | zram->bd_wb_limit += 1UL << (PAGE_SHIFT - 12); |
| @@ -878,7 +909,7 @@ static void zram_account_writeback_rollback(struct zram *zram) · 1 unmodified line | ||
| 878 | 909 | |
| 879 | 910 | static void zram_account_writeback_submit(struct zram *zram) |
| 880 | 911 | { |
| 881 | − lockdep_assert_held_read(&zram->init_lock); | |
| 912 | + lockdep_assert_held_write(&zram->dev_lock); | |
| 882 | 913 | |
| 883 | 914 | if (zram->wb_limit_enable && zram->bd_wb_limit > 0) |
| 884 | 915 | zram->bd_wb_limit -= 1UL << (PAGE_SHIFT - 12); |
| @@ -886,8 +917,9 @@ static void zram_account_writeback_submit(struct zram *zram) · 1 unmodified line | ||
| 886 | 917 | |
| 887 | 918 | static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) |
| 888 | 919 | { |
| 889 | − u32 index = req->pps->index; | |
| 890 | − int err; | |
| 920 | + u32 size, index = req->pps->index; | |
| 921 | + int err, prio; | |
| 922 | + bool huge; | |
| 891 | 923 | |
| 892 | 924 | err = blk_status_to_errno(req->bio.bi_status); |
| 893 | 925 | if (err) { |
| @@ -901,7 +933,7 @@ static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) · 7 unmodified lines | ||
| 901 | 933 | } |
| 902 | 934 | |
| 903 | 935 | atomic64_inc(&zram->stats.bd_writes); |
| 904 | − zram_slot_lock(zram, index); | |
| 936 | + slot_lock(zram, index); | |
| 905 | 937 | /* |
| 906 | 938 | * We release slot lock during writeback so slot can change under us: |
| 907 | 939 | * slot_free() or slot_free() and zram_write_page(). In both cases |
| @@ -909,18 +941,36 @@ static int zram_writeback_complete(struct zram *zram, struct zram_wb_req *req) · 1 unmodified line | ||
| 909 | 941 | * set ZRAM_PP_SLOT on such slots until current post-processing |
| 910 | 942 | * finishes. |
| 911 | 943 | */ |
| 912 | − if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { | |
| 944 | + if (!test_slot_flag(zram, index, ZRAM_PP_SLOT)) { | |
| 913 | 945 | zram_release_bdev_block(zram, req->blk_idx); |
| 914 | 946 | goto out; |
| 915 | 947 | } |
| 916 | 948 | |
| 917 | − zram_free_page(zram, index); | |
| 918 | − zram_set_flag(zram, index, ZRAM_WB); | |
| 919 | − zram_set_handle(zram, index, req->blk_idx); | |
| 949 | + if (zram->wb_compressed) { | |
| 950 | + /* | |
| 951 | + * ZRAM_WB slots get freed, we need to preserve data required | |
| 952 | + * for read decompression. | |
| 953 | + */ | |
| 954 | + size = get_slot_size(zram, index); | |
| 955 | + prio = get_slot_comp_priority(zram, index); | |
| 956 | + huge = test_slot_flag(zram, index, ZRAM_HUGE); | |
| 957 | + } | |
| 958 | + | |
| 959 | + slot_free(zram, index); | |
| 960 | + set_slot_flag(zram, index, ZRAM_WB); | |
| 961 | + set_slot_handle(zram, index, req->blk_idx); | |
| 962 | + | |
| 963 | + if (zram->wb_compressed) { | |
| 964 | + if (huge) | |
| 965 | + set_slot_flag(zram, index, ZRAM_HUGE); | |
| 966 | + set_slot_size(zram, index, size); | |
| 967 | + set_slot_comp_priority(zram, index, prio); | |
| 968 | + } | |
| 969 | + | |
| 920 | 970 | atomic64_inc(&zram->stats.pages_stored); |
| 921 | 971 | |
| 922 | 972 | out: |
| 923 | − zram_slot_unlock(zram, index); | |
| 973 | + slot_unlock(zram, index); | |
| 924 | 974 | return 0; |
| 925 | 975 | } |
| 926 | 976 | |
| @@ -1041,18 +1091,22 @@ static int zram_writeback_slots(struct zram *zram, · 114 unmodified lines | ||
| 1041 | 1091 | } |
| 1042 | 1092 | |
| 1043 | 1093 | index = pps->index; |
| 1044 | − zram_slot_lock(zram, index); | |
| 1094 | + slot_lock(zram, index); | |
| 1045 | 1095 | /* |
| 1046 | 1096 | * scan_slots() sets ZRAM_PP_SLOT and releases slot lock, so |
| 1047 | 1097 | * slots can change in the meantime. If slots are accessed or |
| 1048 | 1098 | * freed they lose ZRAM_PP_SLOT flag and hence we don't |
| 1049 | 1099 | * post-process them. |
| 1050 | 1100 | */ |
| 1051 | − if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) | |
| 1101 | + if (!test_slot_flag(zram, index, ZRAM_PP_SLOT)) | |
| 1052 | 1102 | goto next; |
| 1053 | − if (zram_read_from_zspool(zram, req->page, index)) | |
| 1103 | + if (zram->wb_compressed) | |
| 1104 | + err = read_from_zspool_raw(zram, req->page, index); | |
| 1105 | + else | |
| 1106 | + err = read_from_zspool(zram, req->page, index); | |
| 1107 | + if (err) | |
| 1054 | 1108 | goto next; |
| 1055 | − zram_slot_unlock(zram, index); | |
| 1109 | + slot_unlock(zram, index); | |
| 1056 | 1110 | |
| 1057 | 1111 | /* |
| 1058 | 1112 | * From now on pp-slot is owned by the req, remove it from |
| @@ -1074,7 +1128,7 @@ static int zram_writeback_slots(struct zram *zram, · 15 unmodified lines | ||
| 1074 | 1128 | continue; |
| 1075 | 1129 | |
| 1076 | 1130 | next: |
| 1077 | − zram_slot_unlock(zram, index); | |
| 1131 | + slot_unlock(zram, index); | |
| 1078 | 1132 | release_pp_slot(zram, pps); |
| 1079 | 1133 | } |
| 1080 | 1134 | |
| @@ -1167,27 +1221,27 @@ static int scan_slots_for_writeback(struct zram *zram, u32 mode, · 86 unmodified lines | ||
| 1167 | 1221 | while (index < hi) { |
| 1168 | 1222 | bool ok = true; |
| 1169 | 1223 | |
| 1170 | − zram_slot_lock(zram, index); | |
| 1171 | − if (!zram_allocated(zram, index)) | |
| 1224 | + slot_lock(zram, index); | |
| 1225 | + if (!slot_allocated(zram, index)) | |
| 1172 | 1226 | goto next; |
| 1173 | 1227 | |
| 1174 | − if (zram_test_flag(zram, index, ZRAM_WB) || | |
| 1175 | − zram_test_flag(zram, index, ZRAM_SAME)) | |
| 1228 | + if (test_slot_flag(zram, index, ZRAM_WB) || | |
| 1229 | + test_slot_flag(zram, index, ZRAM_SAME)) | |
| 1176 | 1230 | goto next; |
| 1177 | 1231 | |
| 1178 | 1232 | if (mode & IDLE_WRITEBACK && |
| 1179 | − !zram_test_flag(zram, index, ZRAM_IDLE)) | |
| 1233 | + !test_slot_flag(zram, index, ZRAM_IDLE)) | |
| 1180 | 1234 | goto next; |
| 1181 | 1235 | if (mode & HUGE_WRITEBACK && |
| 1182 | − !zram_test_flag(zram, index, ZRAM_HUGE)) | |
| 1236 | + !test_slot_flag(zram, index, ZRAM_HUGE)) | |
| 1183 | 1237 | goto next; |
| 1184 | 1238 | if (mode & INCOMPRESSIBLE_WRITEBACK && |
| 1185 | − !zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) | |
| 1239 | + !test_slot_flag(zram, index, ZRAM_INCOMPRESSIBLE)) | |
| 1186 | 1240 | goto next; |
| 1187 | 1241 | |
| 1188 | 1242 | ok = place_pp_slot(zram, ctl, index); |
| 1189 | 1243 | next: |
| 1190 | − zram_slot_unlock(zram, index); | |
| 1244 | + slot_unlock(zram, index); | |
| 1191 | 1245 | if (!ok) |
| 1192 | 1246 | break; |
| 1193 | 1247 | index++; |
| @@ -1209,33 +1263,21 @@ static ssize_t writeback_store(struct device *dev, · 15 unmodified lines | ||
| 1209 | 1263 | ssize_t ret = len; |
| 1210 | 1264 | int err, mode = 0; |
| 1211 | 1265 | |
| 1212 | − down_read(&zram->init_lock); | |
| 1213 | − if (!init_done(zram)) { | |
| 1214 | − up_read(&zram->init_lock); | |
| 1266 | + guard(rwsem_write)(&zram->dev_lock); | |
| 1267 | + if (!init_done(zram)) | |
| 1215 | 1268 | return -EINVAL; |
| 1216 | − } | |
| 1217 | − | |
| 1218 | − /* Do not permit concurrent post-processing actions. */ | |
| 1219 | − if (atomic_xchg(&zram->pp_in_progress, 1)) { | |
| 1220 | − up_read(&zram->init_lock); | |
| 1221 | − return -EAGAIN; | |
| 1222 | − } | |
| 1223 | 1269 | |
| 1224 | − if (!zram->backing_dev) { | |
| 1225 | − ret = -ENODEV; | |
| 1226 | − goto release_init_lock; | |
| 1227 | − } | |
| 1270 | + if (!zram->backing_dev) | |
| 1271 | + return -ENODEV; | |
| 1228 | 1272 | |
| 1229 | 1273 | pp_ctl = init_pp_ctl(); |
| 1230 | − if (!pp_ctl) { | |
| 1231 | − ret = -ENOMEM; | |
| 1232 | − goto release_init_lock; | |
| 1233 | − } | |
| 1274 | + if (!pp_ctl) | |
| 1275 | + return -ENOMEM; | |
| 1234 | 1276 | |
| 1235 | 1277 | wb_ctl = init_wb_ctl(zram); |
| 1236 | 1278 | if (!wb_ctl) { |
| 1237 | 1279 | ret = -ENOMEM; |
| 1238 | − goto release_init_lock; | |
| 1280 | + goto out; | |
| 1239 | 1281 | } |
| 1240 | 1282 | |
| 1241 | 1283 | args = skip_spaces(buf); |
| @@ -1259,7 +1301,7 @@ static ssize_t writeback_store(struct device *dev, · 17 unmodified lines | ||
| 1259 | 1301 | err = parse_mode(param, &mode); |
| 1260 | 1302 | if (err) { |
| 1261 | 1303 | ret = err; |
| 1262 | − goto release_init_lock; | |
| 1304 | + goto out; | |
| 1263 | 1305 | } |
| 1264 | 1306 | |
| 1265 | 1307 | scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl); |
| @@ -1270,7 +1312,7 @@ static ssize_t writeback_store(struct device *dev, · 4 unmodified lines | ||
| 1270 | 1312 | err = parse_mode(val, &mode); |
| 1271 | 1313 | if (err) { |
| 1272 | 1314 | ret = err; |
| 1273 | − goto release_init_lock; | |
| 1315 | + goto out; | |
| 1274 | 1316 | } |
| 1275 | 1317 | |
| 1276 | 1318 | scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl); |
| @@ -1281,7 +1323,7 @@ static ssize_t writeback_store(struct device *dev, · 4 unmodified lines | ||
| 1281 | 1323 | err = parse_page_index(val, nr_pages, &lo, &hi); |
| 1282 | 1324 | if (err) { |
| 1283 | 1325 | ret = err; |
| 1284 | − goto release_init_lock; | |
| 1326 | + goto out; | |
| 1285 | 1327 | } |
| 1286 | 1328 | |
| 1287 | 1329 | scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl); |
| @@ -1292,7 +1334,7 @@ static ssize_t writeback_store(struct device *dev, · 4 unmodified lines | ||
| 1292 | 1334 | err = parse_page_indexes(val, nr_pages, &lo, &hi); |
| 1293 | 1335 | if (err) { |
| 1294 | 1336 | ret = err; |
| 1295 | − goto release_init_lock; | |
| 1337 | + goto out; | |
| 1296 | 1338 | } |
| 1297 | 1339 | |
| 1298 | 1340 | scan_slots_for_writeback(zram, mode, lo, hi, pp_ctl); |
| @@ -1304,33 +1346,147 @@ static ssize_t writeback_store(struct device *dev, · 5 unmodified lines | ||
| 1304 | 1346 | if (err) |
| 1305 | 1347 | ret = err; |
| 1306 | 1348 | |
| 1307 | −release_init_lock: | |
| 1349 | +out: | |
| 1308 | 1350 | release_pp_ctl(zram, pp_ctl); |
| 1309 | 1351 | release_wb_ctl(wb_ctl); |
| 1310 | − atomic_set(&zram->pp_in_progress, 0); | |
| 1311 | − up_read(&zram->init_lock); | |
| 1312 | 1352 | |
| 1313 | 1353 | return ret; |
| 1314 | 1354 | } |
| 1315 | 1355 | |
| 1316 | −struct zram_work { | |
| 1317 | − struct work_struct work; | |
| 1318 | − struct zram *zram; | |
| 1319 | − unsigned long entry; | |
| 1320 | − struct page *page; | |
| 1321 | − int error; | |
| 1322 | −}; | |
| 1356 | +static int decompress_bdev_page(struct zram *zram, struct page *page, u32 index) | |
| 1357 | +{ | |
| 1358 | + struct zcomp_strm *zstrm; | |
| 1359 | + unsigned int size; | |
| 1360 | + int ret, prio; | |
| 1361 | + void *src; | |
| 1362 | + | |
| 1363 | + slot_lock(zram, index); | |
| 1364 | + /* Since slot was unlocked we need to make sure it's still ZRAM_WB */ | |
| 1365 | + if (!test_slot_flag(zram, index, ZRAM_WB)) { | |
| 1366 | + slot_unlock(zram, index); | |
| 1367 | + /* We read some stale data, zero it out */ | |
| 1368 | + memset_page(page, 0, 0, PAGE_SIZE); | |
| 1369 | + return -EIO; | |
| 1370 | + } | |
| 1371 | + | |
| 1372 | + if (test_slot_flag(zram, index, ZRAM_HUGE)) { | |
| 1373 | + slot_unlock(zram, index); | |
| 1374 | + return 0; | |
| 1375 | + } | |
| 1376 | + | |
| 1377 | + size = get_slot_size(zram, index); | |
| 1378 | + prio = get_slot_comp_priority(zram, index); | |
| 1379 | + | |
| 1380 | + zstrm = zcomp_stream_get(zram->comps[prio]); | |
| 1381 | + src = kmap_local_page(page); | |
| 1382 | + ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, | |
| 1383 | + zstrm->local_copy); | |
| 1384 | + if (!ret) | |
| 1385 | + copy_page(src, zstrm->local_copy); | |
| 1386 | + kunmap_local(src); | |
| 1387 | + zcomp_stream_put(zstrm); | |
| 1388 | + slot_unlock(zram, index); | |
| 1389 | + | |
| 1390 | + return ret; | |
| 1391 | +} | |
| 1392 | + | |
| 1393 | +static void zram_deferred_decompress(struct work_struct *w) | |
| 1394 | +{ | |
| 1395 | + struct zram_rb_req *req = container_of(w, struct zram_rb_req, work); | |
| 1396 | + struct page *page = bio_first_page_all(req->bio); | |
| 1397 | + struct zram *zram = req->zram; | |
| 1398 | + u32 index = req->index; | |
| 1399 | + int ret; | |
| 1400 | + | |
| 1401 | + ret = decompress_bdev_page(zram, page, index); | |
| 1402 | + if (ret) | |
| 1403 | + req->parent->bi_status = BLK_STS_IOERR; | |
| 1404 | + | |
| 1405 | + /* Decrement parent's ->remaining */ | |
| 1406 | + bio_endio(req->parent); | |
| 1407 | + bio_put(req->bio); | |
| 1408 | + kfree(req); | |
| 1409 | +} | |
| 1323 | 1410 | |
| 1324 | −static void zram_sync_read(struct work_struct *work) | |
| 1411 | +static void zram_async_read_endio(struct bio *bio) | |
| 1325 | 1412 | { |
| 1326 | − struct zram_work *zw = container_of(work, struct zram_work, work); | |
| 1413 | + struct zram_rb_req *req = bio->bi_private; | |
| 1414 | + struct zram *zram = req->zram; | |
| 1415 | + | |
| 1416 | + if (bio->bi_status) { | |
| 1417 | + req->parent->bi_status = bio->bi_status; | |
| 1418 | + bio_endio(req->parent); | |
| 1419 | + bio_put(bio); | |
| 1420 | + kfree(req); | |
| 1421 | + return; | |
| 1422 | + } | |
| 1423 | + | |
| 1424 | + /* | |
| 1425 | + * NOTE: zram_async_read_endio() is not exactly right place for this. | |
| 1426 | + * Ideally, we need to do it after ZRAM_WB check, but this requires | |
| 1427 | + * us to use wq path even on systems that don't enable compressed | |
| 1428 | + * writeback, because we cannot take slot-lock in the current context. | |
| 1429 | + * | |
| 1430 | + * Keep the existing behavior for now. | |
| 1431 | + */ | |
| 1432 | + if (zram->wb_compressed == false) { | |
| 1433 | + /* No decompression needed, complete the parent IO */ | |
| 1434 | + bio_endio(req->parent); | |
| 1435 | + bio_put(bio); | |
| 1436 | + kfree(req); | |
| 1437 | + return; | |
| 1438 | + } | |
| 1439 | + | |
| 1440 | + /* | |
| 1441 | + * zram decompression is sleepable, so we need to deffer it to | |
| 1442 | + * a preemptible context. | |
| 1443 | + */ | |
| 1444 | + INIT_WORK(&req->work, zram_deferred_decompress); | |
| 1445 | + queue_work(system_highpri_wq, &req->work); | |
| 1446 | +} | |
| 1447 | + | |
| 1448 | +static void read_from_bdev_async(struct zram *zram, struct page *page, | |
| 1449 | + u32 index, unsigned long blk_idx, | |
| 1450 | + struct bio *parent) | |
| 1451 | +{ | |
| 1452 | + struct zram_rb_req *req; | |
| 1453 | + struct bio *bio; | |
| 1454 | + | |
| 1455 | + req = kmalloc(sizeof(*req), GFP_NOIO); | |
| 1456 | + if (!req) | |
| 1457 | + return; | |
| 1458 | + | |
| 1459 | + bio = bio_alloc(zram->bdev, 1, parent->bi_opf, GFP_NOIO); | |
| 1460 | + if (!bio) { | |
| 1461 | + kfree(req); | |
| 1462 | + return; | |
| 1463 | + } | |
| 1464 | + | |
| 1465 | + req->zram = zram; | |
| 1466 | + req->index = index; | |
| 1467 | + req->blk_idx = blk_idx; | |
| 1468 | + req->bio = bio; | |
| 1469 | + req->parent = parent; | |
| 1470 | + | |
| 1471 | + bio->bi_iter.bi_sector = blk_idx * (PAGE_SIZE >> 9); | |
| 1472 | + bio->bi_private = req; | |
| 1473 | + bio->bi_end_io = zram_async_read_endio; | |
| 1474 | + | |
| 1475 | + __bio_add_page(bio, page, PAGE_SIZE, 0); | |
| 1476 | + bio_inc_remaining(parent); | |
| 1477 | + submit_bio(bio); | |
| 1478 | +} | |
| 1479 | + | |
| 1480 | +static void zram_sync_read(struct work_struct *w) | |
| 1481 | +{ | |
| 1482 | + struct zram_rb_req *req = container_of(w, struct zram_rb_req, work); | |
| 1327 | 1483 | struct bio_vec bv; |
| 1328 | 1484 | struct bio bio; |
| 1329 | 1485 | |
| 1330 | − bio_init(&bio, zw->zram->bdev, &bv, 1, REQ_OP_READ); | |
| 1331 | − bio.bi_iter.bi_sector = zw->entry * (PAGE_SIZE >> 9); | |
| 1332 | − __bio_add_page(&bio, zw->page, PAGE_SIZE, 0); | |
| 1333 | − zw->error = submit_bio_wait(&bio); | |
| 1486 | + bio_init(&bio, req->zram->bdev, &bv, 1, REQ_OP_READ); | |
| 1487 | + bio.bi_iter.bi_sector = req->blk_idx * (PAGE_SIZE >> 9); | |
| 1488 | + __bio_add_page(&bio, req->page, PAGE_SIZE, 0); | |
| 1489 | + req->error = submit_bio_wait(&bio); | |
| 1334 | 1490 | } |
| 1335 | 1491 | |
| 1336 | 1492 | /* |
| @@ -1338,39 +1494,42 @@ static void zram_sync_read(struct work_struct *work) · 1 unmodified line | ||
| 1338 | 1494 | * chained IO with parent IO in same context, it's a deadlock. To avoid that, |
| 1339 | 1495 | * use a worker thread context. |
| 1340 | 1496 | */ |
| 1341 | −static int read_from_bdev_sync(struct zram *zram, struct page *page, | |
| 1342 | − unsigned long entry) | |
| 1497 | +static int read_from_bdev_sync(struct zram *zram, struct page *page, u32 index, | |
| 1498 | + unsigned long blk_idx) | |
| 1343 | 1499 | { |
| 1344 | − struct zram_work work; | |
| 1500 | + struct zram_rb_req req; | |
| 1501 | + | |
| 1502 | + req.page = page; | |
| 1503 | + req.zram = zram; | |
| 1504 | + req.blk_idx = blk_idx; | |
| 1345 | 1505 | |
| 1346 | − work.page = page; | |
| 1347 | − work.zram = zram; | |
| 1348 | − work.entry = entry; | |
| 1506 | + INIT_WORK_ONSTACK(&req.work, zram_sync_read); | |
| 1507 | + queue_work(system_dfl_wq, &req.work); | |
| 1508 | + flush_work(&req.work); | |
| 1509 | + destroy_work_on_stack(&req.work); | |
| 1349 | 1510 | |
| 1350 | − INIT_WORK_ONSTACK(&work.work, zram_sync_read); | |
| 1351 | − queue_work(system_dfl_wq, &work.work); | |
| 1352 | − flush_work(&work.work); | |
| 1353 | − destroy_work_on_stack(&work.work); | |
| 1511 | + if (req.error || zram->wb_compressed == false) | |
| 1512 | + return req.error; | |
| 1354 | 1513 | |
| 1355 | − return work.error; | |
| 1514 | + return decompress_bdev_page(zram, page, index); | |
| 1356 | 1515 | } |
| 1357 | 1516 | |
| 1358 | −static int read_from_bdev(struct zram *zram, struct page *page, | |
| 1359 | − unsigned long entry, struct bio *parent) | |
| 1517 | +static int read_from_bdev(struct zram *zram, struct page *page, u32 index, | |
| 1518 | + unsigned long blk_idx, struct bio *parent) | |
| 1360 | 1519 | { |
| 1361 | 1520 | atomic64_inc(&zram->stats.bd_reads); |
| 1362 | 1521 | if (!parent) { |
| 1363 | 1522 | if (WARN_ON_ONCE(!IS_ENABLED(ZRAM_PARTIAL_IO))) |
| 1364 | 1523 | return -EIO; |
| 1365 | − return read_from_bdev_sync(zram, page, entry); | |
| 1524 | + return read_from_bdev_sync(zram, page, index, blk_idx); | |
| 1366 | 1525 | } |
| 1367 | − read_from_bdev_async(zram, page, entry, parent); | |
| 1526 | + read_from_bdev_async(zram, page, index, blk_idx, parent); | |
| 1368 | 1527 | return 0; |
| 1369 | 1528 | } |
| 1370 | 1529 | #else |
| 1371 | 1530 | static inline void reset_bdev(struct zram *zram) {}; |
| 1372 | −static int read_from_bdev(struct zram *zram, struct page *page, | |
| 1373 | − unsigned long entry, struct bio *parent) | |
| 1531 | +static int read_from_bdev(struct zram *zram, struct page *page, u32 index, | |
| 1532 | + unsigned long blk_idx, struct bio *parent) | |
| 1374 | 1533 | { |
| 1375 | 1534 | return -EIO; |
| 1376 | 1535 | } |
| @@ -1401,15 +1560,13 @@ static ssize_t read_block_state(struct file *file, char __user *buf, · 24 unmodified lines | ||
| 1401 | 1560 | ssize_t index, written = 0; |
| 1402 | 1561 | struct zram *zram = file->private_data; |
| 1403 | 1562 | unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; |
| 1404 | − struct timespec64 ts; | |
| 1405 | 1563 | |
| 1406 | 1564 | kbuf = kvmalloc(count, GFP_KERNEL); |
| 1407 | 1565 | if (!kbuf) |
| 1408 | 1566 | return -ENOMEM; |
| 1409 | 1567 | |
| 1410 | − down_read(&zram->init_lock); | |
| 1568 | + guard(rwsem_read)(&zram->dev_lock); | |
| 1411 | 1569 | if (!init_done(zram)) { |
| 1412 | − up_read(&zram->init_lock); | |
| 1413 | 1570 | kvfree(kbuf); |
| 1414 | 1571 | return -EINVAL; |
| 1415 | 1572 | } |
| @@ -1417,35 +1574,32 @@ static ssize_t read_block_state(struct file *file, char __user *buf, · 1 unmodified line | ||
| 1417 | 1574 | for (index = *ppos; index < nr_pages; index++) { |
| 1418 | 1575 | int copied; |
| 1419 | 1576 | |
| 1420 | − zram_slot_lock(zram, index); | |
| 1421 | − if (!zram_allocated(zram, index)) | |
| 1577 | + slot_lock(zram, index); | |
| 1578 | + if (!slot_allocated(zram, index)) | |
| 1422 | 1579 | goto next; |
| 1423 | 1580 | |
| 1424 | − ts = ktime_to_timespec64(zram->table[index].ac_time); | |
| 1425 | 1581 | copied = snprintf(kbuf + written, count, |
| 1426 | − "%12zd %12lld.%06lu %c%c%c%c%c%c\n", | |
| 1427 | − index, (s64)ts.tv_sec, | |
| 1428 | − ts.tv_nsec / NSEC_PER_USEC, | |
| 1429 | − zram_test_flag(zram, index, ZRAM_SAME) ? 's' : '.', | |
| 1430 | − zram_test_flag(zram, index, ZRAM_WB) ? 'w' : '.', | |
| 1431 | − zram_test_flag(zram, index, ZRAM_HUGE) ? 'h' : '.', | |
| 1432 | − zram_test_flag(zram, index, ZRAM_IDLE) ? 'i' : '.', | |
| 1433 | − zram_get_priority(zram, index) ? 'r' : '.', | |
| 1434 | − zram_test_flag(zram, index, | |
| 1582 | + "%12zd %12u.%06d %c%c%c%c%c%c\n", | |
| 1583 | + index, zram->table[index].attr.ac_time, 0, | |
| 1584 | + test_slot_flag(zram, index, ZRAM_SAME) ? 's' : '.', | |
| 1585 | + test_slot_flag(zram, index, ZRAM_WB) ? 'w' : '.', | |
| 1586 | + test_slot_flag(zram, index, ZRAM_HUGE) ? 'h' : '.', | |
| 1587 | + test_slot_flag(zram, index, ZRAM_IDLE) ? 'i' : '.', | |
| 1588 | + get_slot_comp_priority(zram, index) ? 'r' : '.', | |
| 1589 | + test_slot_flag(zram, index, | |
| 1435 | 1590 | ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); |
| 1436 | 1591 | |
| 1437 | 1592 | if (count <= copied) { |
| 1438 | − zram_slot_unlock(zram, index); | |
| 1593 | + slot_unlock(zram, index); | |
| 1439 | 1594 | break; |
| 1440 | 1595 | } |
| 1441 | 1596 | written += copied; |
| 1442 | 1597 | count -= copied; |
| 1443 | 1598 | next: |
| 1444 | − zram_slot_unlock(zram, index); | |
| 1599 | + slot_unlock(zram, index); | |
| 1445 | 1600 | *ppos += 1; |
| 1446 | 1601 | } |
| 1447 | 1602 | |
| 1448 | − up_read(&zram->init_lock); | |
| 1449 | 1603 | if (copy_to_user(buf, kbuf, written)) |
| 1450 | 1604 | written = -EFAULT; |
| 1451 | 1605 | kvfree(kbuf); |
| @@ -1512,16 +1666,14 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf) · 60 unmodified lines | ||
| 1512 | 1666 | return -EINVAL; |
| 1513 | 1667 | } |
| 1514 | 1668 | |
| 1515 | − down_write(&zram->init_lock); | |
| 1669 | + guard(rwsem_write)(&zram->dev_lock); | |
| 1516 | 1670 | if (init_done(zram)) { |
| 1517 | − up_write(&zram->init_lock); | |
| 1518 | 1671 | kfree(compressor); |
| 1519 | 1672 | pr_info("Can't change algorithm for initialized device\n"); |
| 1520 | 1673 | return -EBUSY; |
| 1521 | 1674 | } |
| 1522 | 1675 | |
| 1523 | 1676 | comp_algorithm_set(zram, prio, compressor); |
| 1524 | − up_write(&zram->init_lock); | |
| 1525 | 1677 | return 0; |
| 1526 | 1678 | } |
| 1527 | 1679 | |
| @@ -1642,9 +1794,8 @@ static ssize_t comp_algorithm_show(struct device *dev, · 114 unmodified lines | ||
| 1642 | 1794 | struct zram *zram = dev_to_zram(dev); |
| 1643 | 1795 | ssize_t sz; |
| 1644 | 1796 | |
| 1645 | − down_read(&zram->init_lock); | |
| 1797 | + guard(rwsem_read)(&zram->dev_lock); | |
| 1646 | 1798 | sz = zcomp_available_show(zram->comp_algs[ZRAM_PRIMARY_COMP], buf, 0); |
| 1647 | − up_read(&zram->init_lock); | |
| 1648 | 1799 | return sz; |
| 1649 | 1800 | } |
| 1650 | 1801 | |
| @@ -1669,7 +1820,7 @@ static ssize_t recomp_algorithm_show(struct device *dev, · 18 unmodified lines | ||
| 1669 | 1820 | ssize_t sz = 0; |
| 1670 | 1821 | u32 prio; |
| 1671 | 1822 | |
| 1672 | − down_read(&zram->init_lock); | |
| 1823 | + guard(rwsem_read)(&zram->dev_lock); | |
| 1673 | 1824 | for (prio = ZRAM_SECONDARY_COMP; prio < ZRAM_MAX_COMPS; prio++) { |
| 1674 | 1825 | if (!zram->comp_algs[prio]) |
| 1675 | 1826 | continue; |
| @@ -1677,7 +1828,6 @@ static ssize_t recomp_algorithm_show(struct device *dev, · 1 unmodified line | ||
| 1677 | 1828 | sz += sysfs_emit_at(buf, sz, "#%d: ", prio); |
| 1678 | 1829 | sz += zcomp_available_show(zram->comp_algs[prio], buf, sz); |
| 1679 | 1830 | } |
| 1680 | − up_read(&zram->init_lock); | |
| 1681 | 1831 | return sz; |
| 1682 | 1832 | } |
| 1683 | 1833 | |
| @@ -1723,42 +1873,38 @@ static ssize_t recomp_algorithm_store(struct device *dev, · 39 unmodified lines | ||
| 1723 | 1873 | } |
| 1724 | 1874 | #endif |
| 1725 | 1875 | |
| 1726 | −static ssize_t compact_store(struct device *dev, | |
| 1727 | − struct device_attribute *attr, const char *buf, size_t len) | |
| 1876 | +static ssize_t compact_store(struct device *dev, struct device_attribute *attr, | |
| 1877 | + const char *buf, size_t len) | |
| 1728 | 1878 | { |
| 1729 | 1879 | struct zram *zram = dev_to_zram(dev); |
| 1730 | 1880 | |
| 1731 | − down_read(&zram->init_lock); | |
| 1732 | − if (!init_done(zram)) { | |
| 1733 | − up_read(&zram->init_lock); | |
| 1881 | + guard(rwsem_read)(&zram->dev_lock); | |
| 1882 | + if (!init_done(zram)) | |
| 1734 | 1883 | return -EINVAL; |
| 1735 | − } | |
| 1736 | 1884 | |
| 1737 | 1885 | zs_compact(zram->mem_pool); |
| 1738 | − up_read(&zram->init_lock); | |
| 1739 | 1886 | |
| 1740 | 1887 | return len; |
| 1741 | 1888 | } |
| 1742 | 1889 | |
| 1743 | −static ssize_t io_stat_show(struct device *dev, | |
| 1744 | − struct device_attribute *attr, char *buf) | |
| 1890 | +static ssize_t io_stat_show(struct device *dev, struct device_attribute *attr, | |
| 1891 | + char *buf) | |
| 1745 | 1892 | { |
| 1746 | 1893 | struct zram *zram = dev_to_zram(dev); |
| 1747 | 1894 | ssize_t ret; |
| 1748 | 1895 | |
| 1749 | − down_read(&zram->init_lock); | |
| 1896 | + guard(rwsem_read)(&zram->dev_lock); | |
| 1750 | 1897 | ret = sysfs_emit(buf, |
| 1751 | 1898 | "%8llu %8llu 0 %8llu\n", |
| 1752 | 1899 | (u64)atomic64_read(&zram->stats.failed_reads), |
| 1753 | 1900 | (u64)atomic64_read(&zram->stats.failed_writes), |
| 1754 | 1901 | (u64)atomic64_read(&zram->stats.notify_free)); |
| 1755 | − up_read(&zram->init_lock); | |
| 1756 | 1902 | |
| 1757 | 1903 | return ret; |
| 1758 | 1904 | } |
| 1759 | 1905 | |
| 1760 | −static ssize_t mm_stat_show(struct device *dev, | |
| 1761 | − struct device_attribute *attr, char *buf) | |
| 1906 | +static ssize_t mm_stat_show(struct device *dev, struct device_attribute *attr, | |
| 1907 | + char *buf) | |
| 1762 | 1908 | { |
| 1763 | 1909 | struct zram *zram = dev_to_zram(dev); |
| 1764 | 1910 | struct zs_pool_stats pool_stats; |
| @@ -1768,7 +1914,7 @@ static ssize_t mm_stat_show(struct device *dev, · 3 unmodified lines | ||
| 1768 | 1914 | |
| 1769 | 1915 | memset(&pool_stats, 0x00, sizeof(struct zs_pool_stats)); |
| 1770 | 1916 | |
| 1771 | − down_read(&zram->init_lock); | |
| 1917 | + guard(rwsem_read)(&zram->dev_lock); | |
| 1772 | 1918 | if (init_done(zram)) { |
| 1773 | 1919 | mem_used = zs_get_total_pages(zram->mem_pool); |
| 1774 | 1920 | zs_pool_stats(zram->mem_pool, &pool_stats); |
| @@ -1788,55 +1934,26 @@ static ssize_t mm_stat_show(struct device *dev, · 13 unmodified lines | ||
| 1788 | 1934 | atomic_long_read(&pool_stats.pages_compacted), |
| 1789 | 1935 | (u64)atomic64_read(&zram->stats.huge_pages), |
| 1790 | 1936 | (u64)atomic64_read(&zram->stats.huge_pages_since)); |
| 1791 | − up_read(&zram->init_lock); | |
| 1792 | 1937 | |
| 1793 | 1938 | return ret; |
| 1794 | 1939 | } |
| 1795 | 1940 | |
| 1796 | −#ifdef CONFIG_ZRAM_WRITEBACK | |
| 1797 | −#define FOUR_K(x) ((x) * (1 << (PAGE_SHIFT - 12))) | |
| 1798 | −static ssize_t bd_stat_show(struct device *dev, | |
| 1799 | − struct device_attribute *attr, char *buf) | |
| 1800 | −{ | |
| 1801 | − struct zram *zram = dev_to_zram(dev); | |
| 1802 | − ssize_t ret; | |
| 1803 | − | |
| 1804 | − down_read(&zram->init_lock); | |
| 1805 | − ret = sysfs_emit(buf, | |
| 1806 | − "%8llu %8llu %8llu\n", | |
| 1807 | − FOUR_K((u64)atomic64_read(&zram->stats.bd_count)), | |
| 1808 | − FOUR_K((u64)atomic64_read(&zram->stats.bd_reads)), | |
| 1809 | − FOUR_K((u64)atomic64_read(&zram->stats.bd_writes))); | |
| 1810 | − up_read(&zram->init_lock); | |
| 1811 | − | |
| 1812 | − return ret; | |
| 1813 | −} | |
| 1814 | −#endif | |
| 1815 | − | |
| 1816 | 1941 | static ssize_t debug_stat_show(struct device *dev, |
| 1817 | − struct device_attribute *attr, char *buf) | |
| 1942 | + struct device_attribute *attr, char *buf) | |
| 1818 | 1943 | { |
| 1819 | 1944 | int version = 1; |
| 1820 | 1945 | struct zram *zram = dev_to_zram(dev); |
| 1821 | 1946 | ssize_t ret; |
| 1822 | 1947 | |
| 1823 | − down_read(&zram->init_lock); | |
| 1948 | + guard(rwsem_read)(&zram->dev_lock); | |
| 1824 | 1949 | ret = sysfs_emit(buf, |
| 1825 | 1950 | "version: %d\n0 %8llu\n", |
| 1826 | 1951 | version, |
| 1827 | 1952 | (u64)atomic64_read(&zram->stats.miss_free)); |
| 1828 | − up_read(&zram->init_lock); | |
| 1829 | 1953 | |
| 1830 | 1954 | return ret; |
| 1831 | 1955 | } |
| 1832 | 1956 | |
| 1833 | −static DEVICE_ATTR_RO(io_stat); | |
| 1834 | −static DEVICE_ATTR_RO(mm_stat); | |
| 1835 | −#ifdef CONFIG_ZRAM_WRITEBACK | |
| 1836 | −static DEVICE_ATTR_RO(bd_stat); | |
| 1837 | −#endif | |
| 1838 | −static DEVICE_ATTR_RO(debug_stat); | |
| 1839 | − | |
| 1840 | 1957 | static void zram_meta_free(struct zram *zram, u64 disksize) |
| 1841 | 1958 | { |
| 1842 | 1959 | size_t num_pages = disksize >> PAGE_SHIFT; |
| @@ -1847,7 +1964,7 @@ static void zram_meta_free(struct zram *zram, u64 disksize) · 4 unmodified lines | ||
| 1847 | 1964 | |
| 1848 | 1965 | /* Free all pages that are still in this zram device */ |
| 1849 | 1966 | for (index = 0; index < num_pages; index++) |
| 1850 | − zram_free_page(zram, index); | |
| 1967 | + slot_free(zram, index); | |
| 1851 | 1968 | |
| 1852 | 1969 | zs_destroy_pool(zram->mem_pool); |
| 1853 | 1970 | vfree(zram->table); |
| @@ -1874,32 +1991,32 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) · 20 unmodified lines | ||
| 1874 | 1991 | huge_class_size = zs_huge_class_size(zram->mem_pool); |
| 1875 | 1992 | |
| 1876 | 1993 | for (index = 0; index < num_pages; index++) |
| 1877 | − zram_slot_lock_init(zram, index); | |
| 1994 | + slot_lock_init(zram, index); | |
| 1878 | 1995 | |
| 1879 | 1996 | return true; |
| 1880 | 1997 | } |
| 1881 | 1998 | |
| 1882 | −static void zram_free_page(struct zram *zram, size_t index) | |
| 1999 | +static void slot_free(struct zram *zram, u32 index) | |
| 1883 | 2000 | { |
| 1884 | 2001 | unsigned long handle; |
| 1885 | 2002 | |
| 1886 | 2003 | #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME |
| 1887 | − zram->table[index].ac_time = 0; | |
| 2004 | + zram->table[index].attr.ac_time = 0; | |
| 1888 | 2005 | #endif |
| 1889 | 2006 | |
| 1890 | − zram_clear_flag(zram, index, ZRAM_IDLE); | |
| 1891 | − zram_clear_flag(zram, index, ZRAM_INCOMPRESSIBLE); | |
| 1892 | − zram_clear_flag(zram, index, ZRAM_PP_SLOT); | |
| 1893 | − zram_set_priority(zram, index, 0); | |
| 2007 | + clear_slot_flag(zram, index, ZRAM_IDLE); | |
| 2008 | + clear_slot_flag(zram, index, ZRAM_INCOMPRESSIBLE); | |
| 2009 | + clear_slot_flag(zram, index, ZRAM_PP_SLOT); | |
| 2010 | + set_slot_comp_priority(zram, index, 0); | |
| 1894 | 2011 | |
| 1895 | − if (zram_test_flag(zram, index, ZRAM_HUGE)) { | |
| 1896 | − zram_clear_flag(zram, index, ZRAM_HUGE); | |
| 2012 | + if (test_slot_flag(zram, index, ZRAM_HUGE)) { | |
| 2013 | + clear_slot_flag(zram, index, ZRAM_HUGE); | |
| 1897 | 2014 | atomic64_dec(&zram->stats.huge_pages); |
| 1898 | 2015 | } |
| 1899 | 2016 | |
| 1900 | − if (zram_test_flag(zram, index, ZRAM_WB)) { | |
| 1901 | − zram_clear_flag(zram, index, ZRAM_WB); | |
| 1902 | − zram_release_bdev_block(zram, zram_get_handle(zram, index)); | |
| 2017 | + if (test_slot_flag(zram, index, ZRAM_WB)) { | |
| 2018 | + clear_slot_flag(zram, index, ZRAM_WB); | |
| 2019 | + zram_release_bdev_block(zram, get_slot_handle(zram, index)); | |
| 1903 | 2020 | goto out; |
| 1904 | 2021 | } |
| 1905 | 2022 | |
| @@ -1907,24 +2024,24 @@ static void zram_free_page(struct zram *zram, size_t index) · 1 unmodified line | ||
| 1907 | 2024 | * No memory is allocated for same element filled pages. |
| 1908 | 2025 | * Simply clear same page flag. |
| 1909 | 2026 | */ |
| 1910 | − if (zram_test_flag(zram, index, ZRAM_SAME)) { | |
| 1911 | − zram_clear_flag(zram, index, ZRAM_SAME); | |
| 2027 | + if (test_slot_flag(zram, index, ZRAM_SAME)) { | |
| 2028 | + clear_slot_flag(zram, index, ZRAM_SAME); | |
| 1912 | 2029 | atomic64_dec(&zram->stats.same_pages); |
| 1913 | 2030 | goto out; |
| 1914 | 2031 | } |
| 1915 | 2032 | |
| 1916 | − handle = zram_get_handle(zram, index); | |
| 2033 | + handle = get_slot_handle(zram, index); | |
| 1917 | 2034 | if (!handle) |
| 1918 | 2035 | return; |
| 1919 | 2036 | |
| 1920 | 2037 | zs_free(zram->mem_pool, handle); |
| 1921 | 2038 | |
| 1922 | − atomic64_sub(zram_get_obj_size(zram, index), | |
| 2039 | + atomic64_sub(get_slot_size(zram, index), | |
| 1923 | 2040 | &zram->stats.compr_data_size); |
| 1924 | 2041 | out: |
| 1925 | 2042 | atomic64_dec(&zram->stats.pages_stored); |
| 1926 | − zram_set_handle(zram, index, 0); | |
| 1927 | − zram_set_obj_size(zram, index, 0); | |
| 2043 | + set_slot_handle(zram, index, 0); | |
| 2044 | + set_slot_size(zram, index, 0); | |
| 1928 | 2045 | } |
| 1929 | 2046 | |
| 1930 | 2047 | static int read_same_filled_page(struct zram *zram, struct page *page, |
| @@ -1933,7 +2050,7 @@ static int read_same_filled_page(struct zram *zram, struct page *page, · 2 unmodified lines | ||
| 1933 | 2050 | void *mem; |
| 1934 | 2051 | |
| 1935 | 2052 | mem = kmap_local_page(page); |
| 1936 | − zram_fill_page(mem, PAGE_SIZE, zram_get_handle(zram, index)); | |
| 2053 | + zram_fill_page(mem, PAGE_SIZE, get_slot_handle(zram, index)); | |
| 1937 | 2054 | kunmap_local(mem); |
| 1938 | 2055 | return 0; |
| 1939 | 2056 | } |
| @@ -1944,12 +2061,12 @@ static int read_incompressible_page(struct zram *zram, struct page *page, · 4 unmodified lines | ||
| 1944 | 2061 | unsigned long handle; |
| 1945 | 2062 | void *src, *dst; |
| 1946 | 2063 | |
| 1947 | − handle = zram_get_handle(zram, index); | |
| 1948 | − src = zs_obj_read_begin(zram->mem_pool, handle, NULL); | |
| 2064 | + handle = get_slot_handle(zram, index); | |
| 2065 | + src = zs_obj_read_begin(zram->mem_pool, handle, PAGE_SIZE, NULL); | |
| 1949 | 2066 | dst = kmap_local_page(page); |
| 1950 | 2067 | copy_page(dst, src); |
| 1951 | 2068 | kunmap_local(dst); |
| 1952 | − zs_obj_read_end(zram->mem_pool, handle, src); | |
| 2069 | + zs_obj_read_end(zram->mem_pool, handle, PAGE_SIZE, src); | |
| 1953 | 2070 | |
| 1954 | 2071 | return 0; |
| 1955 | 2072 | } |
| @@ -1962,33 +2079,60 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) · 6 unmodified lines | ||
| 1962 | 2079 | void *src, *dst; |
| 1963 | 2080 | int ret, prio; |
| 1964 | 2081 | |
| 1965 | − handle = zram_get_handle(zram, index); | |
| 1966 | − size = zram_get_obj_size(zram, index); | |
| 1967 | − prio = zram_get_priority(zram, index); | |
| 2082 | + handle = get_slot_handle(zram, index); | |
| 2083 | + size = get_slot_size(zram, index); | |
| 2084 | + prio = get_slot_comp_priority(zram, index); | |
| 1968 | 2085 | |
| 1969 | 2086 | zstrm = zcomp_stream_get(zram->comps[prio]); |
| 1970 | − src = zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); | |
| 2087 | + src = zs_obj_read_begin(zram->mem_pool, handle, size, | |
| 2088 | + zstrm->local_copy); | |
| 1971 | 2089 | dst = kmap_local_page(page); |
| 1972 | 2090 | ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); |
| 1973 | 2091 | kunmap_local(dst); |
| 1974 | − zs_obj_read_end(zram->mem_pool, handle, src); | |
| 2092 | + zs_obj_read_end(zram->mem_pool, handle, size, src); | |
| 1975 | 2093 | zcomp_stream_put(zstrm); |
| 1976 | 2094 | |
| 1977 | 2095 | return ret; |
| 1978 | 2096 | } |
| 1979 | 2097 | |
| 2098 | +#if defined CONFIG_ZRAM_WRITEBACK | |
| 2099 | +static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index) | |
| 2100 | +{ | |
| 2101 | + struct zcomp_strm *zstrm; | |
| 2102 | + unsigned long handle; | |
| 2103 | + unsigned int size; | |
| 2104 | + void *src; | |
| 2105 | + | |
| 2106 | + handle = get_slot_handle(zram, index); | |
| 2107 | + size = get_slot_size(zram, index); | |
| 2108 | + | |
| 2109 | + /* | |
| 2110 | + * We need to get stream just for ->local_copy buffer, in | |
| 2111 | + * case if object spans two physical pages. No decompression | |
| 2112 | + * takes place here, as we read raw compressed data. | |
| 2113 | + */ | |
| 2114 | + zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); | |
| 2115 | + src = zs_obj_read_begin(zram->mem_pool, handle, size, | |
| 2116 | + zstrm->local_copy); | |
| 2117 | + memcpy_to_page(page, 0, src, size); | |
| 2118 | + zs_obj_read_end(zram->mem_pool, handle, size, src); | |
| 2119 | + zcomp_stream_put(zstrm); | |
| 2120 | + | |
| 2121 | + return 0; | |
| 2122 | +} | |
| 2123 | +#endif | |
| 2124 | + | |
| 1980 | 2125 | /* |
| 1981 | 2126 | * Reads (decompresses if needed) a page from zspool (zsmalloc). |
| 1982 | 2127 | * Corresponding ZRAM slot should be locked. |
| 1983 | 2128 | */ |
| 1984 | −static int zram_read_from_zspool(struct zram *zram, struct page *page, | |
| 1985 | − u32 index) | |
| 2129 | +static int read_from_zspool(struct zram *zram, struct page *page, u32 index) | |
| 1986 | 2130 | { |
| 1987 | − if (zram_test_flag(zram, index, ZRAM_SAME) || | |
| 1988 | − !zram_get_handle(zram, index)) | |
| 2131 | + if (test_slot_flag(zram, index, ZRAM_SAME) || | |
| 2132 | + !get_slot_handle(zram, index)) | |
| 1989 | 2133 | return read_same_filled_page(zram, page, index); |
| 1990 | 2134 | |
| 1991 | − if (!zram_test_flag(zram, index, ZRAM_HUGE)) | |
| 2135 | + if (!test_slot_flag(zram, index, ZRAM_HUGE)) | |
| 1992 | 2136 | return read_compressed_page(zram, page, index); |
| 1993 | 2137 | else |
| 1994 | 2138 | return read_incompressible_page(zram, page, index); |
| @@ -1999,20 +2143,20 @@ static int zram_read_page(struct zram *zram, struct page *page, u32 index, · 4 unmodified lines | ||
| 1999 | 2143 | { |
| 2000 | 2144 | int ret; |
| 2001 | 2145 | |
| 2002 | − zram_slot_lock(zram, index); | |
| 2003 | − if (!zram_test_flag(zram, index, ZRAM_WB)) { | |
| 2146 | + slot_lock(zram, index); | |
| 2147 | + if (!test_slot_flag(zram, index, ZRAM_WB)) { | |
| 2004 | 2148 | /* Slot should be locked through out the function call */ |
| 2005 | − ret = zram_read_from_zspool(zram, page, index); | |
| 2006 | − zram_slot_unlock(zram, index); | |
| 2149 | + ret = read_from_zspool(zram, page, index); | |
| 2150 | + slot_unlock(zram, index); | |
| 2007 | 2151 | } else { |
| 2008 | − unsigned long blk_idx = zram_get_handle(zram, index); | |
| 2152 | + unsigned long blk_idx = get_slot_handle(zram, index); | |
| 2009 | 2153 | |
| 2010 | 2154 | /* |
| 2011 | 2155 | * The slot should be unlocked before reading from the backing |
| 2012 | 2156 | * device. |
| 2013 | 2157 | */ |
| 2014 | − zram_slot_unlock(zram, index); | |
| 2015 | − ret = read_from_bdev(zram, page, blk_idx, parent); | |
| 2158 | + slot_unlock(zram, index); | |
| 2159 | + ret = read_from_bdev(zram, page, index, blk_idx, parent); | |
| 2016 | 2160 | } |
| 2017 | 2161 | |
| 2018 | 2162 | /* Should NEVER happen. Return bio error if it does. */ |
| @@ -2052,11 +2196,11 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec, · 33 unmodified lines | ||
| 2052 | 2196 | static int write_same_filled_page(struct zram *zram, unsigned long fill, |
| 2053 | 2197 | u32 index) |
| 2054 | 2198 | { |
| 2055 | − zram_slot_lock(zram, index); | |
| 2056 | − zram_free_page(zram, index); | |
| 2057 | − zram_set_flag(zram, index, ZRAM_SAME); | |
| 2058 | − zram_set_handle(zram, index, fill); | |
| 2059 | − zram_slot_unlock(zram, index); | |
| 2199 | + slot_lock(zram, index); | |
| 2200 | + slot_free(zram, index); | |
| 2201 | + set_slot_flag(zram, index, ZRAM_SAME); | |
| 2202 | + set_slot_handle(zram, index, fill); | |
| 2203 | + slot_unlock(zram, index); | |
| 2060 | 2204 | |
| 2061 | 2205 | atomic64_inc(&zram->stats.same_pages); |
| 2062 | 2206 | atomic64_inc(&zram->stats.pages_stored); |
| @@ -2090,12 +2234,12 @@ static int write_incompressible_page(struct zram *zram, struct page *page, · 27 unmodified lines | ||
| 2090 | 2234 | zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); |
| 2091 | 2235 | kunmap_local(src); |
| 2092 | 2236 | |
| 2093 | − zram_slot_lock(zram, index); | |
| 2094 | − zram_free_page(zram, index); | |
| 2095 | − zram_set_flag(zram, index, ZRAM_HUGE); | |
| 2096 | − zram_set_handle(zram, index, handle); | |
| 2097 | − zram_set_obj_size(zram, index, PAGE_SIZE); | |
| 2098 | − zram_slot_unlock(zram, index); | |
| 2237 | + slot_lock(zram, index); | |
| 2238 | + slot_free(zram, index); | |
| 2239 | + set_slot_flag(zram, index, ZRAM_HUGE); | |
| 2240 | + set_slot_handle(zram, index, handle); | |
| 2241 | + set_slot_size(zram, index, PAGE_SIZE); | |
| 2242 | + slot_unlock(zram, index); | |
| 2099 | 2243 | |
| 2100 | 2244 | atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); |
| 2101 | 2245 | atomic64_inc(&zram->stats.huge_pages); |
| @@ -2155,11 +2299,11 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) · 53 unmodified lines | ||
| 2155 | 2299 | zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); |
| 2156 | 2300 | zcomp_stream_put(zstrm); |
| 2157 | 2301 | |
| 2158 | − zram_slot_lock(zram, index); | |
| 2159 | − zram_free_page(zram, index); | |
| 2160 | − zram_set_handle(zram, index, handle); | |
| 2161 | − zram_set_obj_size(zram, index, comp_len); | |
| 2162 | − zram_slot_unlock(zram, index); | |
| 2302 | + slot_lock(zram, index); | |
| 2303 | + slot_free(zram, index); | |
| 2304 | + set_slot_handle(zram, index, handle); | |
| 2305 | + set_slot_size(zram, index, comp_len); | |
| 2306 | + slot_unlock(zram, index); | |
| 2163 | 2307 | |
| 2164 | 2308 | /* Update stats */ |
| 2165 | 2309 | atomic64_inc(&zram->stats.pages_stored); |
| @@ -2210,30 +2354,30 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio_max, · 44 unmodified lines | ||
| 2210 | 2354 | for (index = 0; index < nr_pages; index++) { |
| 2211 | 2355 | bool ok = true; |
| 2212 | 2356 | |
| 2213 | − zram_slot_lock(zram, index); | |
| 2214 | − if (!zram_allocated(zram, index)) | |
| 2357 | + slot_lock(zram, index); | |
| 2358 | + if (!slot_allocated(zram, index)) | |
| 2215 | 2359 | goto next; |
| 2216 | 2360 | |
| 2217 | 2361 | if (mode & RECOMPRESS_IDLE && |
| 2218 | − !zram_test_flag(zram, index, ZRAM_IDLE)) | |
| 2362 | + !test_slot_flag(zram, index, ZRAM_IDLE)) | |
| 2219 | 2363 | goto next; |
| 2220 | 2364 | |
| 2221 | 2365 | if (mode & RECOMPRESS_HUGE && |
| 2222 | − !zram_test_flag(zram, index, ZRAM_HUGE)) | |
| 2366 | + !test_slot_flag(zram, index, ZRAM_HUGE)) | |
| 2223 | 2367 | goto next; |
| 2224 | 2368 | |
| 2225 | − if (zram_test_flag(zram, index, ZRAM_WB) || | |
| 2226 | − zram_test_flag(zram, index, ZRAM_SAME) || | |
| 2227 | − zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) | |
| 2369 | + if (test_slot_flag(zram, index, ZRAM_WB) || | |
| 2370 | + test_slot_flag(zram, index, ZRAM_SAME) || | |
| 2371 | + test_slot_flag(zram, index, ZRAM_INCOMPRESSIBLE)) | |
| 2228 | 2372 | goto next; |
| 2229 | 2373 | |
| 2230 | 2374 | /* Already compressed with same of higher priority */ |
| 2231 | − if (zram_get_priority(zram, index) + 1 >= prio_max) | |
| 2375 | + if (get_slot_comp_priority(zram, index) + 1 >= prio_max) | |
| 2232 | 2376 | goto next; |
| 2233 | 2377 | |
| 2234 | 2378 | ok = place_pp_slot(zram, ctl, index); |
| 2235 | 2379 | next: |
| 2236 | − zram_slot_unlock(zram, index); | |
| 2380 | + slot_unlock(zram, index); | |
| 2237 | 2381 | if (!ok) |
| 2238 | 2382 | break; |
| 2239 | 2383 | } |
| @@ -2262,18 +2406,18 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, · 22 unmodified lines | ||
| 2262 | 2406 | void *src; |
| 2263 | 2407 | int ret = 0; |
| 2264 | 2408 | |
| 2265 | − handle_old = zram_get_handle(zram, index); | |
| 2409 | + handle_old = get_slot_handle(zram, index); | |
| 2266 | 2410 | if (!handle_old) |
| 2267 | 2411 | return -EINVAL; |
| 2268 | 2412 | |
| 2269 | − comp_len_old = zram_get_obj_size(zram, index); | |
| 2413 | + comp_len_old = get_slot_size(zram, index); | |
| 2270 | 2414 | /* |
| 2271 | 2415 | * Do not recompress objects that are already "small enough". |
| 2272 | 2416 | */ |
| 2273 | 2417 | if (comp_len_old < threshold) |
| 2274 | 2418 | return 0; |
| 2275 | 2419 | |
| 2276 | − ret = zram_read_from_zspool(zram, page, index); | |
| 2420 | + ret = read_from_zspool(zram, page, index); | |
| 2277 | 2421 | if (ret) |
| 2278 | 2422 | return ret; |
| 2279 | 2423 | |
| @@ -2282,11 +2426,11 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, · 2 unmodified lines | ||
| 2282 | 2426 | * we don't preserve IDLE flag and don't incorrectly pick this entry |
| 2283 | 2427 | * for different post-processing type (e.g. writeback). |
| 2284 | 2428 | */ |
| 2285 | − zram_clear_flag(zram, index, ZRAM_IDLE); | |
| 2429 | + clear_slot_flag(zram, index, ZRAM_IDLE); | |
| 2286 | 2430 | |
| 2287 | 2431 | class_index_old = zs_lookup_class_index(zram->mem_pool, comp_len_old); |
| 2288 | 2432 | |
| 2289 | − prio = max(prio, zram_get_priority(zram, index) + 1); | |
| 2433 | + prio = max(prio, get_slot_comp_priority(zram, index) + 1); | |
| 2290 | 2434 | /* |
| 2291 | 2435 | * Recompression slots scan should not select slots that are |
| 2292 | 2436 | * already compressed with a higher priority algorithm, but |
| @@ -2353,7 +2497,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, · 60 unmodified lines | ||
| 2353 | 2497 | */ |
| 2354 | 2498 | if (prio < zram->num_active_comps) |
| 2355 | 2499 | return 0; |
| 2356 | − zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); | |
| 2500 | + set_slot_flag(zram, index, ZRAM_INCOMPRESSIBLE); | |
| 2357 | 2501 | return 0; |
| 2358 | 2502 | } |
| 2359 | 2503 | |
| @@ -2362,14 +2506,15 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, · 2 unmodified lines | ||
| 2362 | 2506 | * avoid direct reclaim. Allocation error is not fatal since |
| 2363 | 2507 | * we still have the old object in the mem_pool. |
| 2364 | 2508 | * |
| 2365 | − * XXX: technically, the node we really want here is the node that holds | |
| 2366 | − * the original compressed data. But that would require us to modify | |
| 2367 | − * zsmalloc API to return this information. For now, we will make do with | |
| 2368 | − * the node of the page allocated for recompression. | |
| 2509 | + * XXX: technically, the node we really want here is the node that | |
| 2510 | + * holds the original compressed data. But that would require us to | |
| 2511 | + * modify zsmalloc API to return this information. For now, we will | |
| 2512 | + * make do with the node of the page allocated for recompression. | |
| 2369 | 2513 | */ |
| 2370 | 2514 | handle_new = zs_malloc(zram->mem_pool, comp_len_new, |
| 2371 | 2515 | GFP_NOIO | __GFP_NOWARN | |
| 2372 | − __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page)); | |
| 2516 | + __GFP_HIGHMEM | __GFP_MOVABLE, | |
| 2517 | + page_to_nid(page)); | |
| 2373 | 2518 | if (IS_ERR_VALUE(handle_new)) { |
| 2374 | 2519 | zcomp_stream_put(zstrm); |
| 2375 | 2520 | return PTR_ERR((void *)handle_new); |
| @@ -2378,10 +2523,10 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, · 2 unmodified lines | ||
| 2378 | 2523 | zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); |
| 2379 | 2524 | zcomp_stream_put(zstrm); |
| 2380 | 2525 | |
| 2381 | − zram_free_page(zram, index); | |
| 2382 | − zram_set_handle(zram, index, handle_new); | |
| 2383 | − zram_set_obj_size(zram, index, comp_len_new); | |
| 2384 | − zram_set_priority(zram, index, prio); | |
| 2526 | + slot_free(zram, index); | |
| 2527 | + set_slot_handle(zram, index, handle_new); | |
| 2528 | + set_slot_size(zram, index, comp_len_new); | |
| 2529 | + set_slot_comp_priority(zram, index, prio); | |
| 2385 | 2530 | |
| 2386 | 2531 | atomic64_add(comp_len_new, &zram->stats.compr_data_size); |
| 2387 | 2532 | atomic64_inc(&zram->stats.pages_stored); |
| @@ -2466,17 +2611,9 @@ static ssize_t recompress_store(struct device *dev, · 78 unmodified lines | ||
| 2466 | 2611 | if (threshold >= huge_class_size) |
| 2467 | 2612 | return -EINVAL; |
| 2468 | 2613 | |
| 2469 | − down_read(&zram->init_lock); | |
| 2470 | − if (!init_done(zram)) { | |
| 2471 | − ret = -EINVAL; | |
| 2472 | − goto release_init_lock; | |
| 2473 | − } | |
| 2474 | − | |
| 2475 | − /* Do not permit concurrent post-processing actions. */ | |
| 2476 | − if (atomic_xchg(&zram->pp_in_progress, 1)) { | |
| 2477 | − up_read(&zram->init_lock); | |
| 2478 | − return -EAGAIN; | |
| 2479 | − } | |
| 2614 | + guard(rwsem_write)(&zram->dev_lock); | |
| 2615 | + if (!init_done(zram)) | |
| 2616 | + return -EINVAL; | |
| 2480 | 2617 | |
| 2481 | 2618 | if (algo) { |
| 2482 | 2619 | bool found = false; |
| @@ -2494,26 +2631,26 @@ static ssize_t recompress_store(struct device *dev, · 11 unmodified lines | ||
| 2494 | 2631 | |
| 2495 | 2632 | if (!found) { |
| 2496 | 2633 | ret = -EINVAL; |
| 2497 | − goto release_init_lock; | |
| 2634 | + goto out; | |
| 2498 | 2635 | } |
| 2499 | 2636 | } |
| 2500 | 2637 | |
| 2501 | 2638 | prio_max = min(prio_max, (u32)zram->num_active_comps); |
| 2502 | 2639 | if (prio >= prio_max) { |
| 2503 | 2640 | ret = -EINVAL; |
| 2504 | − goto release_init_lock; | |
| 2641 | + goto out; | |
| 2505 | 2642 | } |
| 2506 | 2643 | |
| 2507 | 2644 | page = alloc_page(GFP_KERNEL); |
| 2508 | 2645 | if (!page) { |
| 2509 | 2646 | ret = -ENOMEM; |
| 2510 | − goto release_init_lock; | |
| 2647 | + goto out; | |
| 2511 | 2648 | } |
| 2512 | 2649 | |
| 2513 | 2650 | ctl = init_pp_ctl(); |
| 2514 | 2651 | if (!ctl) { |
| 2515 | 2652 | ret = -ENOMEM; |
| 2516 | − goto release_init_lock; | |
| 2653 | + goto out; | |
| 2517 | 2654 | } |
| 2518 | 2655 | |
| 2519 | 2656 | scan_slots_for_recompress(zram, mode, prio_max, ctl); |
| @@ -2525,15 +2662,15 @@ static ssize_t recompress_store(struct device *dev, · 5 unmodified lines | ||
| 2525 | 2662 | if (!num_recomp_pages) |
| 2526 | 2663 | break; |
| 2527 | 2664 | |
| 2528 | − zram_slot_lock(zram, pps->index); | |
| 2529 | − if (!zram_test_flag(zram, pps->index, ZRAM_PP_SLOT)) | |
| 2665 | + slot_lock(zram, pps->index); | |
| 2666 | + if (!test_slot_flag(zram, pps->index, ZRAM_PP_SLOT)) | |
| 2530 | 2667 | goto next; |
| 2531 | 2668 | |
| 2532 | 2669 | err = recompress_slot(zram, pps->index, page, |
| 2533 | 2670 | &num_recomp_pages, threshold, |
| 2534 | 2671 | prio, prio_max); |
| 2535 | 2672 | next: |
| 2536 | − zram_slot_unlock(zram, pps->index); | |
| 2673 | + slot_unlock(zram, pps->index); | |
| 2537 | 2674 | release_pp_slot(zram, pps); |
| 2538 | 2675 | |
| 2539 | 2676 | if (err) { |
| @@ -2544,12 +2681,10 @@ next: · 4 unmodified lines | ||
| 2544 | 2681 | cond_resched(); |
| 2545 | 2682 | } |
| 2546 | 2683 | |
| 2547 | −release_init_lock: | |
| 2684 | +out: | |
| 2548 | 2685 | if (page) |
| 2549 | 2686 | __free_page(page); |
| 2550 | 2687 | release_pp_ctl(zram, ctl); |
| 2551 | − atomic_set(&zram->pp_in_progress, 0); | |
| 2552 | − up_read(&zram->init_lock); | |
| 2553 | 2688 | return ret; |
| 2554 | 2689 | } |
| 2555 | 2690 | #endif |
| @@ -2580,9 +2715,9 @@ static void zram_bio_discard(struct zram *zram, struct bio *bio) · 24 unmodified lines | ||
| 2580 | 2715 | } |
| 2581 | 2716 | |
| 2582 | 2717 | while (n >= PAGE_SIZE) { |
| 2583 | − zram_slot_lock(zram, index); | |
| 2584 | − zram_free_page(zram, index); | |
| 2585 | − zram_slot_unlock(zram, index); | |
| 2718 | + slot_lock(zram, index); | |
| 2719 | + slot_free(zram, index); | |
| 2720 | + slot_unlock(zram, index); | |
| 2586 | 2721 | atomic64_inc(&zram->stats.notify_free); |
| 2587 | 2722 | index++; |
| 2588 | 2723 | n -= PAGE_SIZE; |
| @@ -2611,9 +2746,9 @@ static void zram_bio_read(struct zram *zram, struct bio *bio) · 22 unmodified lines | ||
| 2611 | 2746 | } |
| 2612 | 2747 | flush_dcache_page(bv.bv_page); |
| 2613 | 2748 | |
| 2614 | − zram_slot_lock(zram, index); | |
| 2615 | − zram_accessed(zram, index); | |
| 2616 | − zram_slot_unlock(zram, index); | |
| 2749 | + slot_lock(zram, index); | |
| 2750 | + mark_slot_accessed(zram, index); | |
| 2751 | + slot_unlock(zram, index); | |
| 2617 | 2752 | |
| 2618 | 2753 | bio_advance_iter_single(bio, &iter, bv.bv_len); |
| 2619 | 2754 | } while (iter.bi_size); |
| @@ -2641,9 +2776,9 @@ static void zram_bio_write(struct zram *zram, struct bio *bio) · 21 unmodified lines | ||
| 2641 | 2776 | break; |
| 2642 | 2777 | } |
| 2643 | 2778 | |
| 2644 | − zram_slot_lock(zram, index); | |
| 2645 | − zram_accessed(zram, index); | |
| 2646 | − zram_slot_unlock(zram, index); | |
| 2779 | + slot_lock(zram, index); | |
| 2780 | + mark_slot_accessed(zram, index); | |
| 2781 | + slot_unlock(zram, index); | |
| 2647 | 2782 | |
| 2648 | 2783 | bio_advance_iter_single(bio, &iter, bv.bv_len); |
| 2649 | 2784 | } while (iter.bi_size); |
| @@ -2684,13 +2819,13 @@ static void zram_slot_free_notify(struct block_device *bdev, · 34 unmodified lines | ||
| 2684 | 2819 | zram = bdev->bd_disk->private_data; |
| 2685 | 2820 | |
| 2686 | 2821 | atomic64_inc(&zram->stats.notify_free); |
| 2687 | − if (!zram_slot_trylock(zram, index)) { | |
| 2822 | + if (!slot_trylock(zram, index)) { | |
| 2688 | 2823 | atomic64_inc(&zram->stats.miss_free); |
| 2689 | 2824 | return; |
| 2690 | 2825 | } |
| 2691 | 2826 | |
| 2692 | − zram_free_page(zram, index); | |
| 2693 | − zram_slot_unlock(zram, index); | |
| 2827 | + slot_free(zram, index); | |
| 2828 | + slot_unlock(zram, index); | |
| 2694 | 2829 | } |
| 2695 | 2830 | |
| 2696 | 2831 | static void zram_comp_params_reset(struct zram *zram) |
| @@ -2728,7 +2863,7 @@ static void zram_destroy_comps(struct zram *zram) · 31 unmodified lines | ||
| 2728 | 2863 | |
| 2729 | 2864 | static void zram_reset_device(struct zram *zram) |
| 2730 | 2865 | { |
| 2731 | − down_write(&zram->init_lock); | |
| 2866 | + guard(rwsem_write)(&zram->dev_lock); | |
| 2732 | 2867 | |
| 2733 | 2868 | zram->limit_pages = 0; |
| 2734 | 2869 | |
| @@ -2740,15 +2875,13 @@ static void zram_reset_device(struct zram *zram) · 5 unmodified lines | ||
| 2740 | 2875 | zram->disksize = 0; |
| 2741 | 2876 | zram_destroy_comps(zram); |
| 2742 | 2877 | memset(&zram->stats, 0, sizeof(zram->stats)); |
| 2743 | − atomic_set(&zram->pp_in_progress, 0); | |
| 2744 | 2878 | reset_bdev(zram); |
| 2745 | 2879 | |
| 2746 | 2880 | comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); |
| 2747 | − up_write(&zram->init_lock); | |
| 2748 | 2881 | } |
| 2749 | 2882 | |
| 2750 | −static ssize_t disksize_store(struct device *dev, | |
| 2751 | − struct device_attribute *attr, const char *buf, size_t len) | |
| 2883 | +static ssize_t disksize_store(struct device *dev, struct device_attribute *attr, | |
| 2884 | + const char *buf, size_t len) | |
| 2752 | 2885 | { |
| 2753 | 2886 | u64 disksize; |
| 2754 | 2887 | struct zcomp *comp; |
| @@ -2760,18 +2893,15 @@ static ssize_t disksize_store(struct device *dev, · 5 unmodified lines | ||
| 2760 | 2893 | if (!disksize) |
| 2761 | 2894 | return -EINVAL; |
| 2762 | 2895 | |
| 2763 | − down_write(&zram->init_lock); | |
| 2896 | + guard(rwsem_write)(&zram->dev_lock); | |
| 2764 | 2897 | if (init_done(zram)) { |
| 2765 | 2898 | pr_info("Cannot change disksize for initialized device\n"); |
| 2766 | − err = -EBUSY; | |
| 2767 | − goto out_unlock; | |
| 2899 | + return -EBUSY; | |
| 2768 | 2900 | } |
| 2769 | 2901 | |
| 2770 | 2902 | disksize = PAGE_ALIGN(disksize); |
| 2771 | − if (!zram_meta_alloc(zram, disksize)) { | |
| 2772 | − err = -ENOMEM; | |
| 2773 | − goto out_unlock; | |
| 2774 | − } | |
| 2903 | + if (!zram_meta_alloc(zram, disksize)) | |
| 2904 | + return -ENOMEM; | |
| 2775 | 2905 | |
| 2776 | 2906 | for (prio = ZRAM_PRIMARY_COMP; prio < ZRAM_MAX_COMPS; prio++) { |
| 2777 | 2907 | if (!zram->comp_algs[prio]) |
| @@ -2791,15 +2921,12 @@ static ssize_t disksize_store(struct device *dev, · 13 unmodified lines | ||
| 2791 | 2921 | } |
| 2792 | 2922 | zram->disksize = disksize; |
| 2793 | 2923 | set_capacity_and_notify(zram->disk, zram->disksize >> SECTOR_SHIFT); |
| 2794 | − up_write(&zram->init_lock); | |
| 2795 | 2924 | |
| 2796 | 2925 | return len; |
| 2797 | 2926 | |
| 2798 | 2927 | out_free_comps: |
| 2799 | 2928 | zram_destroy_comps(zram); |
| 2800 | 2929 | zram_meta_free(zram, disksize); |
| 2801 | −out_unlock: | |
| 2802 | − up_write(&zram->init_lock); | |
| 2803 | 2930 | return err; |
| 2804 | 2931 | } |
| 2805 | 2932 | |
| @@ -2862,6 +2989,9 @@ static const struct block_device_operations zram_devops = { · 56 unmodified lines | ||
| 2862 | 2989 | .owner = THIS_MODULE |
| 2863 | 2990 | }; |
| 2864 | 2991 | |
| 2992 | +static DEVICE_ATTR_RO(io_stat); | |
| 2993 | +static DEVICE_ATTR_RO(mm_stat); | |
| 2994 | +static DEVICE_ATTR_RO(debug_stat); | |
| 2865 | 2995 | static DEVICE_ATTR_WO(compact); |
| 2866 | 2996 | static DEVICE_ATTR_RW(disksize); |
| 2867 | 2997 | static DEVICE_ATTR_RO(initstate); |
| @@ -2871,11 +3001,13 @@ static DEVICE_ATTR_WO(mem_used_max); · 3 unmodified lines | ||
| 2871 | 3001 | static DEVICE_ATTR_WO(idle); |
| 2872 | 3002 | static DEVICE_ATTR_RW(comp_algorithm); |
| 2873 | 3003 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 3004 | +static DEVICE_ATTR_RO(bd_stat); | |
| 2874 | 3005 | static DEVICE_ATTR_RW(backing_dev); |
| 2875 | 3006 | static DEVICE_ATTR_WO(writeback); |
| 2876 | 3007 | static DEVICE_ATTR_RW(writeback_limit); |
| 2877 | 3008 | static DEVICE_ATTR_RW(writeback_limit_enable); |
| 2878 | 3009 | static DEVICE_ATTR_RW(writeback_batch_size); |
| 3010 | +static DEVICE_ATTR_RW(writeback_compressed); | |
| 2879 | 3011 | #endif |
| 2880 | 3012 | #ifdef CONFIG_ZRAM_MULTI_COMP |
| 2881 | 3013 | static DEVICE_ATTR_RW(recomp_algorithm); |
| @@ -2893,17 +3025,16 @@ static struct attribute *zram_disk_attrs[] = { · 11 unmodified lines | ||
| 2893 | 3025 | &dev_attr_idle.attr, |
| 2894 | 3026 | &dev_attr_comp_algorithm.attr, |
| 2895 | 3027 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 3028 | + &dev_attr_bd_stat.attr, | |
| 2896 | 3029 | &dev_attr_backing_dev.attr, |
| 2897 | 3030 | &dev_attr_writeback.attr, |
| 2898 | 3031 | &dev_attr_writeback_limit.attr, |
| 2899 | 3032 | &dev_attr_writeback_limit_enable.attr, |
| 2900 | 3033 | &dev_attr_writeback_batch_size.attr, |
| 3034 | + &dev_attr_writeback_compressed.attr, | |
| 2901 | 3035 | #endif |
| 2902 | 3036 | &dev_attr_io_stat.attr, |
| 2903 | 3037 | &dev_attr_mm_stat.attr, |
| 2904 | −#ifdef CONFIG_ZRAM_WRITEBACK | |
| 2905 | − &dev_attr_bd_stat.attr, | |
| 2906 | −#endif | |
| 2907 | 3038 | &dev_attr_debug_stat.attr, |
| 2908 | 3039 | #ifdef CONFIG_ZRAM_MULTI_COMP |
| 2909 | 3040 | &dev_attr_recomp_algorithm.attr, |
| @@ -2957,9 +3088,10 @@ static int zram_add(void) · 47 unmodified lines | ||
| 2957 | 3088 | goto out_free_dev; |
| 2958 | 3089 | device_id = ret; |
| 2959 | 3090 | |
| 2960 | − init_rwsem(&zram->init_lock); | |
| 3091 | + init_rwsem(&zram->dev_lock); | |
| 2961 | 3092 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 2962 | 3093 | zram->wb_batch_size = 32; |
| 3094 | + zram->wb_compressed = false; | |
| 2963 | 3095 | #endif |
| 2964 | 3096 | |
| 2965 | 3097 | /* gendisk structure */ |
| @@ -2978,7 +3110,6 @@ static int zram_add(void) · 12 unmodified lines | ||
| 2978 | 3110 | zram->disk->fops = &zram_devops; |
| 2979 | 3111 | zram->disk->private_data = zram; |
| 2980 | 3112 | snprintf(zram->disk->disk_name, 16, "zram%d", device_id); |
| 2981 | − atomic_set(&zram->pp_in_progress, 0); | |
| 2982 | 3113 | zram_comp_params_reset(zram); |
| 2983 | 3114 | comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); |
| 2984 | 3115 | |
| @@ -3139,7 +3270,7 @@ static int __init zram_init(void) · 154 unmodified lines | ||
| 3139 | 3270 | struct zram_table_entry zram_te; |
| 3140 | 3271 | int ret; |
| 3141 | 3272 | |
| 3142 | − BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.flags) * 8); | |
| 3273 | + BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.attr.flags) * 8); | |
| 3143 | 3274 | |
| 3144 | 3275 | ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare", |
| 3145 | 3276 | zcomp_cpu_up_prepare, zcomp_cpu_dead); |
| @@ -65,10 +65,15 @@ enum zram_pageflags { · 64 unmodified lines | ||
| 65 | 65 | */ |
| 66 | 66 | struct zram_table_entry { |
| 67 | 67 | unsigned long handle; |
| 68 | − unsigned long flags; | |
| 68 | + union { | |
| 69 | + unsigned long __lock; | |
| 70 | + struct attr { | |
| 71 | + u32 flags; | |
| 69 | 72 | #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME |
| 70 | − ktime_t ac_time; | |
| 73 | + u32 ac_time; | |
| 71 | 74 | #endif |
| 75 | + } attr; | |
| 76 | + }; | |
| 72 | 77 | struct lockdep_map dep_map; |
| 73 | 78 | }; |
| 74 | 79 | |
| @@ -106,8 +111,8 @@ struct zram { · 31 unmodified lines | ||
| 106 | 111 | struct zcomp *comps[ZRAM_MAX_COMPS]; |
| 107 | 112 | struct zcomp_params params[ZRAM_MAX_COMPS]; |
| 108 | 113 | struct gendisk *disk; |
| 109 | − /* Prevent concurrent execution of device init */ | |
| 110 | − struct rw_semaphore init_lock; | |
| 114 | + /* Locks the device either in exclusive or in shared mode */ | |
| 115 | + struct rw_semaphore dev_lock; | |
| 111 | 116 | /* |
| 112 | 117 | * the number of pages zram can consume for storing compressed data |
| 113 | 118 | */ |
| @@ -128,6 +133,7 @@ struct zram { · 14 unmodified lines | ||
| 128 | 133 | #ifdef CONFIG_ZRAM_WRITEBACK |
| 129 | 134 | struct file *backing_dev; |
| 130 | 135 | bool wb_limit_enable; |
| 136 | + bool wb_compressed; | |
| 131 | 137 | u32 wb_batch_size; |
| 132 | 138 | u64 bd_wb_limit; |
| 133 | 139 | struct block_device *bdev; |
| @@ -137,6 +143,5 @@ struct zram { · 3 unmodified lines | ||
| 137 | 143 | #ifdef CONFIG_ZRAM_MEMORY_TRACKING |
| 138 | 144 | struct dentry *debugfs_dir; |
| 139 | 145 | #endif |
| 140 | − atomic_t pp_in_progress; | |
| 141 | 146 | }; |
| 142 | 147 | #endif |
| @@ -410,7 +410,7 @@ config DS1682 · 409 unmodified lines | ||
| 410 | 410 | config VMWARE_BALLOON |
| 411 | 411 | tristate "VMware Balloon Driver" |
| 412 | 412 | depends on VMWARE_VMCI && X86 && HYPERVISOR_GUEST |
| 413 | − select MEMORY_BALLOON | |
| 413 | + select BALLOON | |
| 414 | 414 | help |
| 415 | 415 | This is VMware physical memory management driver which acts |
| 416 | 416 | like a "balloon" that can be inflated to reclaim physical pages |
| @@ -29,7 +29,7 @@ · 28 unmodified lines | ||
| 29 | 29 | #include <linux/rwsem.h> |
| 30 | 30 | #include <linux/slab.h> |
| 31 | 31 | #include <linux/spinlock.h> |
| 32 | −#include <linux/balloon_compaction.h> | |
| 32 | +#include <linux/balloon.h> | |
| 33 | 33 | #include <linux/vmw_vmci_defs.h> |
| 34 | 34 | #include <linux/vmw_vmci_api.h> |
| 35 | 35 | #include <asm/hypervisor.h> |
| @@ -354,10 +354,15 @@ struct vmballoon { · 318 unmodified lines | ||
| 354 | 354 | /** |
| 355 | 355 | * @huge_pages - list of the inflated 2MB pages. |
| 356 | 356 | * |
| 357 | − * Protected by @b_dev_info.pages_lock . | |
| 357 | + * Protected by @huge_pages_lock. | |
| 358 | 358 | */ |
| 359 | 359 | struct list_head huge_pages; |
| 360 | 360 | |
| 361 | + /** | |
| 362 | + * @huge_pages_lock: lock for the list of inflated 2MB pages. | |
| 363 | + */ | |
| 364 | + spinlock_t huge_pages_lock; | |
| 365 | + | |
| 361 | 366 | /** |
| 362 | 367 | * @vmci_doorbell. |
| 363 | 368 | * |
| @@ -987,7 +992,6 @@ static void vmballoon_enqueue_page_list(struct vmballoon *b, · 623 unmodified lines | ||
| 987 | 992 | unsigned int *n_pages, |
| 988 | 993 | enum vmballoon_page_size_type page_size) |
| 989 | 994 | { |
| 990 | − unsigned long flags; | |
| 991 | 995 | struct page *page; |
| 992 | 996 | |
| 993 | 997 | if (page_size == VMW_BALLOON_4K_PAGE) { |
| @@ -995,9 +999,9 @@ static void vmballoon_enqueue_page_list(struct vmballoon *b, · 1 unmodified line | ||
| 995 | 999 | } else { |
| 996 | 1000 | /* |
| 997 | 1001 | * Keep the huge pages in a local list which is not available |
| 998 | − * for the balloon compaction mechanism. | |
| 1002 | + * for the balloon page migration. | |
| 999 | 1003 | */ |
| 1000 | − spin_lock_irqsave(&b->b_dev_info.pages_lock, flags); | |
| 1004 | + spin_lock(&b->huge_pages_lock); | |
| 1001 | 1005 | |
| 1002 | 1006 | list_for_each_entry(page, pages, lru) { |
| 1003 | 1007 | vmballoon_mark_page_offline(page, VMW_BALLOON_2M_PAGE); |
| @@ -1006,7 +1010,7 @@ static void vmballoon_enqueue_page_list(struct vmballoon *b, · 2 unmodified lines | ||
| 1006 | 1010 | list_splice_init(pages, &b->huge_pages); |
| 1007 | 1011 | __count_vm_events(BALLOON_INFLATE, *n_pages * |
| 1008 | 1012 | vmballoon_page_in_frames(VMW_BALLOON_2M_PAGE)); |
| 1009 | − spin_unlock_irqrestore(&b->b_dev_info.pages_lock, flags); | |
| 1013 | + spin_unlock(&b->huge_pages_lock); | |
| 1010 | 1014 | } |
| 1011 | 1015 | |
| 1012 | 1016 | *n_pages = 0; |
| @@ -1033,7 +1037,6 @@ static void vmballoon_dequeue_page_list(struct vmballoon *b, · 20 unmodified lines | ||
| 1033 | 1037 | { |
| 1034 | 1038 | struct page *page, *tmp; |
| 1035 | 1039 | unsigned int i = 0; |
| 1036 | − unsigned long flags; | |
| 1037 | 1040 | |
| 1038 | 1041 | /* In the case of 4k pages, use the compaction infrastructure */ |
| 1039 | 1042 | if (page_size == VMW_BALLOON_4K_PAGE) { |
| @@ -1043,7 +1046,7 @@ static void vmballoon_dequeue_page_list(struct vmballoon *b, · 3 unmodified lines | ||
| 1043 | 1046 | } |
| 1044 | 1047 | |
| 1045 | 1048 | /* 2MB pages */ |
| 1046 | − spin_lock_irqsave(&b->b_dev_info.pages_lock, flags); | |
| 1049 | + spin_lock(&b->huge_pages_lock); | |
| 1047 | 1050 | list_for_each_entry_safe(page, tmp, &b->huge_pages, lru) { |
| 1048 | 1051 | vmballoon_mark_page_online(page, VMW_BALLOON_2M_PAGE); |
| 1049 | 1052 | |
| @@ -1054,7 +1057,7 @@ static void vmballoon_dequeue_page_list(struct vmballoon *b, · 4 unmodified lines | ||
| 1054 | 1057 | |
| 1055 | 1058 | __count_vm_events(BALLOON_DEFLATE, |
| 1056 | 1059 | i * vmballoon_page_in_frames(VMW_BALLOON_2M_PAGE)); |
| 1057 | − spin_unlock_irqrestore(&b->b_dev_info.pages_lock, flags); | |
| 1060 | + spin_unlock(&b->huge_pages_lock); | |
| 1058 | 1061 | *n_pages = i; |
| 1059 | 1062 | } |
| 1060 | 1063 | |
| @@ -1716,7 +1719,7 @@ static inline void vmballoon_debugfs_exit(struct vmballoon *b) · 655 unmodified lines | ||
| 1716 | 1719 | #endif /* CONFIG_DEBUG_FS */ |
| 1717 | 1720 | |
| 1718 | 1721 | |
| 1719 | −#ifdef CONFIG_BALLOON_COMPACTION | |
| 1722 | +#ifdef CONFIG_BALLOON_MIGRATION | |
| 1720 | 1723 | /** |
| 1721 | 1724 | * vmballoon_migratepage() - migrates a balloon page. |
| 1722 | 1725 | * @b_dev_info: balloon device information descriptor. |
| @@ -1724,18 +1727,17 @@ static inline void vmballoon_debugfs_exit(struct vmballoon *b) · 1 unmodified line | ||
| 1724 | 1727 | * @page: a ballooned page that should be migrated. |
| 1725 | 1728 | * @mode: migration mode, ignored. |
| 1726 | 1729 | * |
| 1727 | − * This function is really open-coded, but that is according to the interface | |
| 1728 | − * that balloon_compaction provides. | |
| 1729 | − * | |
| 1730 | 1730 | * Return: zero on success, -EAGAIN when migration cannot be performed |
| 1731 | − * momentarily, and -EBUSY if migration failed and should be retried | |
| 1732 | − * with that specific page. | |
| 1731 | + * momentarily, -EBUSY if migration failed and should be retried | |
| 1732 | + * with that specific page, and -ENOENT when deflating @page | |
| 1733 | + * succeeded but inflating @newpage failed, effectively deflating | |
| 1734 | + * the balloon. | |
| 1733 | 1735 | */ |
| 1734 | 1736 | static int vmballoon_migratepage(struct balloon_dev_info *b_dev_info, |
| 1735 | 1737 | struct page *newpage, struct page *page, |
| 1736 | 1738 | enum migrate_mode mode) |
| 1737 | 1739 | { |
| 1738 | − unsigned long status, flags; | |
| 1740 | + unsigned long status; | |
| 1739 | 1741 | struct vmballoon *b; |
| 1740 | 1742 | int ret = 0; |
| 1741 | 1743 | |
| @@ -1773,14 +1775,6 @@ static int vmballoon_migratepage(struct balloon_dev_info *b_dev_info, · 31 unmodified lines | ||
| 1773 | 1775 | goto out_unlock; |
| 1774 | 1776 | } |
| 1775 | 1777 | |
| 1776 | − /* | |
| 1777 | − * The page is isolated, so it is safe to delete it without holding | |
| 1778 | − * @pages_lock . We keep holding @comm_lock since we will need it in a | |
| 1779 | − * second. | |
| 1780 | − */ | |
| 1781 | − balloon_page_finalize(page); | |
| 1782 | − put_page(page); | |
| 1783 | − | |
| 1784 | 1778 | /* Inflate */ |
| 1785 | 1779 | vmballoon_add_page(b, 0, newpage); |
| 1786 | 1780 | status = vmballoon_lock_op(b, 1, VMW_BALLOON_4K_PAGE, |
| @@ -1799,60 +1793,21 @@ static int vmballoon_migratepage(struct balloon_dev_info *b_dev_info, · 12 unmodified lines | ||
| 1799 | 1793 | * change. |
| 1800 | 1794 | */ |
| 1801 | 1795 | atomic64_dec(&b->size); |
| 1802 | − } else { | |
| 1803 | 1796 | /* |
| 1804 | − * Success. Take a reference for the page, and we will add it to | |
| 1805 | − * the list after acquiring the lock. | |
| 1797 | + * Tell the core that we're deflating the old page and don't | |
| 1798 | + * need the new page. | |
| 1806 | 1799 | */ |
| 1807 | − get_page(newpage); | |
| 1808 | − } | |
| 1809 | − | |
| 1810 | − /* Update the balloon list under the @pages_lock */ | |
| 1811 | − spin_lock_irqsave(&b->b_dev_info.pages_lock, flags); | |
| 1812 | − | |
| 1813 | − /* | |
| 1814 | − * On inflation success, we already took a reference for the @newpage. | |
| 1815 | − * If we succeed just insert it to the list and update the statistics | |
| 1816 | − * under the lock. | |
| 1817 | − */ | |
| 1818 | − if (status == VMW_BALLOON_SUCCESS) { | |
| 1819 | − balloon_page_insert(&b->b_dev_info, newpage); | |
| 1820 | − __count_vm_event(BALLOON_MIGRATE); | |
| 1800 | + ret = -ENOENT; | |
| 1821 | 1801 | } |
| 1822 | − | |
| 1823 | − /* | |
| 1824 | − * We deflated successfully, so regardless to the inflation success, we | |
| 1825 | − * need to reduce the number of isolated_pages. | |
| 1826 | − */ | |
| 1827 | − b->b_dev_info.isolated_pages--; | |
| 1828 | − spin_unlock_irqrestore(&b->b_dev_info.pages_lock, flags); | |
| 1829 | − | |
| 1830 | 1802 | out_unlock: |
| 1831 | 1803 | up_read(&b->conf_sem); |
| 1832 | 1804 | return ret; |
| 1833 | 1805 | } |
| 1834 | − | |
| 1835 | −/** | |
| 1836 | − * vmballoon_compaction_init() - initialized compaction for the balloon. | |
| 1837 | − * | |
| 1838 | − * @b: pointer to the balloon. | |
| 1839 | − * | |
| 1840 | − * If during the initialization a failure occurred, this function does not | |
| 1841 | − * perform cleanup. The caller must call vmballoon_compaction_deinit() in this | |
| 1842 | − * case. | |
| 1843 | − * | |
| 1844 | − * Return: zero on success or error code on failure. | |
| 1845 | − */ | |
| 1846 | −static __init void vmballoon_compaction_init(struct vmballoon *b) | |
| 1847 | −{ | |
| 1848 | − b->b_dev_info.migratepage = vmballoon_migratepage; | |
| 1849 | −} | |
| 1850 | − | |
| 1851 | −#else /* CONFIG_BALLOON_COMPACTION */ | |
| 1852 | −static inline void vmballoon_compaction_init(struct vmballoon *b) | |
| 1853 | −{ | |
| 1854 | −} | |
| 1855 | −#endif /* CONFIG_BALLOON_COMPACTION */ | |
| 1806 | +#else /* CONFIG_BALLOON_MIGRATION */ | |
| 1807 | +int vmballoon_migratepage(struct balloon_dev_info *b_dev_info, | |
| 1808 | + struct page *newpage, struct page *page, | |
| 1809 | + enum migrate_mode mode); | |
| 1810 | +#endif /* CONFIG_BALLOON_MIGRATION */ | |
| 1856 | 1811 | |
| 1857 | 1812 | static int __init vmballoon_init(void) |
| 1858 | 1813 | { |
| @@ -1871,14 +1826,12 @@ static int __init vmballoon_init(void) · 12 unmodified lines | ||
| 1871 | 1826 | if (error) |
| 1872 | 1827 | return error; |
| 1873 | 1828 | |
| 1874 | − /* | |
| 1875 | − * Initialization of compaction must be done after the call to | |
| 1876 | − * balloon_devinfo_init() . | |
| 1877 | − */ | |
| 1878 | 1829 | balloon_devinfo_init(&balloon.b_dev_info); |
| 1879 | − vmballoon_compaction_init(&balloon); | |
| 1830 | + if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) | |
| 1831 | + balloon.b_dev_info.migratepage = vmballoon_migratepage; | |
| 1880 | 1832 | |
| 1881 | 1833 | INIT_LIST_HEAD(&balloon.huge_pages); |
| 1834 | + spin_lock_init(&balloon.huge_pages_lock); | |
| 1882 | 1835 | spin_lock_init(&balloon.comm_lock); |
| 1883 | 1836 | init_rwsem(&balloon.conf_sem); |
| 1884 | 1837 | balloon.vmci_doorbell = VMCI_INVALID_HANDLE; |
| @@ -112,7 +112,7 @@ config VIRTIO_PMEM · 111 unmodified lines | ||
| 112 | 112 | config VIRTIO_BALLOON |
| 113 | 113 | tristate "Virtio balloon driver" |
| 114 | 114 | depends on VIRTIO |
| 115 | − select MEMORY_BALLOON | |
| 115 | + select BALLOON | |
| 116 | 116 | select PAGE_REPORTING |
| 117 | 117 | help |
| 118 | 118 | This driver supports increasing and decreasing the amount |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/delay.h> |
| 14 | 14 | #include <linux/slab.h> |
| 15 | 15 | #include <linux/module.h> |
| 16 | −#include <linux/balloon_compaction.h> | |
| 16 | +#include <linux/balloon.h> | |
| 17 | 17 | #include <linux/oom.h> |
| 18 | 18 | #include <linux/wait.h> |
| 19 | 19 | #include <linux/mm.h> |
| @@ -242,8 +242,8 @@ static void set_page_pfns(struct virtio_balloon *vb, · 222 unmodified lines | ||
| 242 | 242 | static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num) |
| 243 | 243 | { |
| 244 | 244 | unsigned int num_allocated_pages; |
| 245 | + struct page *page, *next; | |
| 245 | 246 | unsigned int num_pfns; |
| 246 | − struct page *page; | |
| 247 | 247 | LIST_HEAD(pages); |
| 248 | 248 | |
| 249 | 249 | /* We can only do one array worth at a time. */ |
| @@ -262,21 +262,19 @@ static unsigned int fill_balloon(struct virtio_balloon *vb, size_t num) · 12 unmodified lines | ||
| 262 | 262 | break; |
| 263 | 263 | } |
| 264 | 264 | |
| 265 | − balloon_page_push(&pages, page); | |
| 265 | + list_add(&page->lru, &pages); | |
| 266 | 266 | } |
| 267 | 267 | |
| 268 | 268 | mutex_lock(&vb->balloon_lock); |
| 269 | 269 | |
| 270 | 270 | vb->num_pfns = 0; |
| 271 | 271 | |
| 272 | − while ((page = balloon_page_pop(&pages))) { | |
| 272 | + list_for_each_entry_safe(page, next, &pages, lru) { | |
| 273 | + list_del(&page->lru); | |
| 273 | 274 | balloon_page_enqueue(&vb->vb_dev_info, page); |
| 274 | 275 | |
| 275 | 276 | set_page_pfns(vb, vb->pfns + vb->num_pfns, page); |
| 276 | 277 | vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE; |
| 277 | − if (!virtio_has_feature(vb->vdev, | |
| 278 | − VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) | |
| 279 | − adjust_managed_page_count(page, -1); | |
| 280 | 278 | vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE; |
| 281 | 279 | } |
| 282 | 280 | |
| @@ -295,9 +293,6 @@ static void release_pages_balloon(struct virtio_balloon *vb, · 12 unmodified lines | ||
| 295 | 293 | struct page *page, *next; |
| 296 | 294 | |
| 297 | 295 | list_for_each_entry_safe(page, next, pages, lru) { |
| 298 | − if (!virtio_has_feature(vb->vdev, | |
| 299 | − VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) | |
| 300 | − adjust_managed_page_count(page, 1); | |
| 301 | 296 | list_del(&page->lru); |
| 302 | 297 | put_page(page); /* balloon reference */ |
| 303 | 298 | } |
| @@ -480,15 +475,19 @@ static inline s64 towards_target(struct virtio_balloon *vb) · 176 unmodified lines | ||
| 480 | 475 | static unsigned long return_free_pages_to_mm(struct virtio_balloon *vb, |
| 481 | 476 | unsigned long num_to_return) |
| 482 | 477 | { |
| 483 | − struct page *page; | |
| 484 | − unsigned long num_returned; | |
| 478 | + unsigned long num_returned = 0; | |
| 479 | + struct page *page, *next; | |
| 480 | + | |
| 481 | + if (unlikely(!num_to_return)) | |
| 482 | + return 0; | |
| 485 | 483 | |
| 486 | 484 | spin_lock_irq(&vb->free_page_list_lock); |
| 487 | − for (num_returned = 0; num_returned < num_to_return; num_returned++) { | |
| 488 | − page = balloon_page_pop(&vb->free_page_list); | |
| 489 | − if (!page) | |
| 490 | − break; | |
| 485 | + | |
| 486 | + list_for_each_entry_safe(page, next, &vb->free_page_list, lru) { | |
| 487 | + list_del(&page->lru); | |
| 491 | 488 | __free_pages(page, VIRTIO_BALLOON_HINT_BLOCK_ORDER); |
| 489 | + if (++num_returned == num_to_return) | |
| 490 | + break; | |
| 492 | 491 | } |
| 493 | 492 | vb->num_free_page_blocks -= num_returned; |
| 494 | 493 | spin_unlock_irq(&vb->free_page_list_lock); |
| @@ -723,7 +722,7 @@ static int get_free_page_and_send(struct virtio_balloon *vb) · 228 unmodified lines | ||
| 723 | 722 | } |
| 724 | 723 | virtqueue_kick(vq); |
| 725 | 724 | spin_lock_irq(&vb->free_page_list_lock); |
| 726 | − balloon_page_push(&vb->free_page_list, page); | |
| 725 | + list_add(&page->lru, &vb->free_page_list); | |
| 727 | 726 | vb->num_free_page_blocks++; |
| 728 | 727 | spin_unlock_irq(&vb->free_page_list_lock); |
| 729 | 728 | } else { |
| @@ -803,7 +802,7 @@ static void report_free_page_func(struct work_struct *work) · 73 unmodified lines | ||
| 803 | 802 | } |
| 804 | 803 | } |
| 805 | 804 | |
| 806 | −#ifdef CONFIG_BALLOON_COMPACTION | |
| 805 | +#ifdef CONFIG_BALLOON_MIGRATION | |
| 807 | 806 | /* |
| 808 | 807 | * virtballoon_migratepage - perform the balloon page migration on behalf of |
| 809 | 808 | * a compaction thread. (called under page lock) |
| @@ -827,7 +826,6 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info, · 17 unmodified lines | ||
| 827 | 826 | { |
| 828 | 827 | struct virtio_balloon *vb = container_of(vb_dev_info, |
| 829 | 828 | struct virtio_balloon, vb_dev_info); |
| 830 | − unsigned long flags; | |
| 831 | 829 | |
| 832 | 830 | /* |
| 833 | 831 | * In order to avoid lock contention while migrating pages concurrently |
| @@ -840,25 +838,7 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info, · 6 unmodified lines | ||
| 840 | 838 | if (!mutex_trylock(&vb->balloon_lock)) |
| 841 | 839 | return -EAGAIN; |
| 842 | 840 | |
| 843 | − get_page(newpage); /* balloon reference */ | |
| 844 | − | |
| 845 | − /* | |
| 846 | − * When we migrate a page to a different zone and adjusted the | |
| 847 | − * managed page count when inflating, we have to fixup the count of | |
| 848 | − * both involved zones. | |
| 849 | − */ | |
| 850 | − if (!virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM) && | |
| 851 | − page_zone(page) != page_zone(newpage)) { | |
| 852 | − adjust_managed_page_count(page, 1); | |
| 853 | − adjust_managed_page_count(newpage, -1); | |
| 854 | − } | |
| 855 | − | |
| 856 | 841 | /* balloon's page migration 1st step -- inflate "newpage" */ |
| 857 | − spin_lock_irqsave(&vb_dev_info->pages_lock, flags); | |
| 858 | − balloon_page_insert(vb_dev_info, newpage); | |
| 859 | − vb_dev_info->isolated_pages--; | |
| 860 | − __count_vm_event(BALLOON_MIGRATE); | |
| 861 | − spin_unlock_irqrestore(&vb_dev_info->pages_lock, flags); | |
| 862 | 842 | vb->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE; |
| 863 | 843 | set_page_pfns(vb, vb->pfns, newpage); |
| 864 | 844 | tell_host(vb, vb->inflate_vq); |
| @@ -869,13 +849,9 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info, · 4 unmodified lines | ||
| 869 | 849 | tell_host(vb, vb->deflate_vq); |
| 870 | 850 | |
| 871 | 851 | mutex_unlock(&vb->balloon_lock); |
| 872 | − | |
| 873 | − balloon_page_finalize(page); | |
| 874 | − put_page(page); /* balloon reference */ | |
| 875 | − | |
| 876 | 852 | return 0; |
| 877 | 853 | } |
| 878 | −#endif /* CONFIG_BALLOON_COMPACTION */ | |
| 854 | +#endif /* CONFIG_BALLOON_MIGRATION */ | |
| 879 | 855 | |
| 880 | 856 | static unsigned long shrink_free_pages(struct virtio_balloon *vb, |
| 881 | 857 | unsigned long pages_to_free) |
| @@ -970,7 +946,9 @@ static int virtballoon_probe(struct virtio_device *vdev) · 88 unmodified lines | ||
| 970 | 946 | if (err) |
| 971 | 947 | goto out_free_vb; |
| 972 | 948 | |
| 973 | −#ifdef CONFIG_BALLOON_COMPACTION | |
| 949 | + if (!virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM)) | |
| 950 | + vb->vb_dev_info.adjust_managed_page_count = true; | |
| 951 | +#ifdef CONFIG_BALLOON_MIGRATION | |
| 974 | 952 | vb->vb_dev_info.migratepage = virtballoon_migratepage; |
| 975 | 953 | #endif |
| 976 | 954 | if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) { |
| @@ -3298,8 +3298,7 @@ int ext4_alloc_da_blocks(struct inode *inode) · 3297 unmodified lines | ||
| 3298 | 3298 | /* |
| 3299 | 3299 | * We do something simple for now. The filemap_flush() will |
| 3300 | 3300 | * also start triggering a write of the data blocks, which is |
| 3301 | − * not strictly speaking necessary (and for users of | |
| 3302 | − * laptop_mode, not even desirable). However, to do otherwise | |
| 3301 | + * not strictly speaking necessary. However, to do otherwise | |
| 3303 | 3302 | * would require replicating code paths in: |
| 3304 | 3303 | * |
| 3305 | 3304 | * ext4_writepages() -> |
| @@ -2751,7 +2751,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, · 2750 unmodified lines | ||
| 2751 | 2751 | return 0; |
| 2752 | 2752 | } |
| 2753 | 2753 | |
| 2754 | − arch_enter_lazy_mmu_mode(); | |
| 2754 | + lazy_mmu_mode_enable(); | |
| 2755 | 2755 | |
| 2756 | 2756 | if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) { |
| 2757 | 2757 | /* Fast path for performing exclusive WP */ |
| @@ -2821,7 +2821,7 @@ flush_and_return: · 63 unmodified lines | ||
| 2821 | 2821 | if (flush_end) |
| 2822 | 2822 | flush_tlb_range(vma, start, addr); |
| 2823 | 2823 | |
| 2824 | − arch_leave_lazy_mmu_mode(); | |
| 2824 | + lazy_mmu_mode_disable(); | |
| 2825 | 2825 | pte_unmap_unlock(start_pte, ptl); |
| 2826 | 2826 | |
| 2827 | 2827 | cond_resched(); |
| @@ -104,8 +104,6 @@ void ksys_sync(void) · 103 unmodified lines | ||
| 104 | 104 | iterate_supers(sync_fs_one_sb, &wait); |
| 105 | 105 | sync_bdevs(false); |
| 106 | 106 | sync_bdevs(true); |
| 107 | − if (unlikely(laptop_mode)) | |
| 108 | − laptop_sync_completion(); | |
| 109 | 107 | } |
| 110 | 108 | |
| 111 | 109 | SYSCALL_DEFINE0(sync) |
| @@ -825,15 +825,6 @@ xfs_fs_sync_fs( · 824 unmodified lines | ||
| 825 | 825 | if (error) |
| 826 | 826 | return error; |
| 827 | 827 | |
| 828 | − if (laptop_mode) { | |
| 829 | − /* | |
| 830 | − * The disk must be active because we're syncing. | |
| 831 | − * We schedule log work now (now that the disk is | |
| 832 | − * active) instead of later (when it might not be). | |
| 833 | − */ | |
| 834 | − flush_delayed_work(&mp->m_log->l_work); | |
| 835 | − } | |
| 836 | − | |
| 837 | 828 | /* |
| 838 | 829 | * If we are called with page faults frozen out, it means we are about |
| 839 | 830 | * to freeze the transaction subsystem. Take the opportunity to shut |
| @@ -213,7 +213,7 @@ struct mmu_table_batch { · 212 unmodified lines | ||
| 213 | 213 | #define MAX_TABLE_BATCH \ |
| 214 | 214 | ((PAGE_SIZE - sizeof(struct mmu_table_batch)) / sizeof(void *)) |
| 215 | 215 | |
| 216 | −#ifndef __HAVE_ARCH_TLB_REMOVE_TABLE | |
| 216 | +#ifndef CONFIG_HAVE_ARCH_TLB_REMOVE_TABLE | |
| 217 | 217 | static inline void __tlb_remove_table(void *table) |
| 218 | 218 | { |
| 219 | 219 | struct ptdesc *ptdesc = (struct ptdesc *)table; |
| @@ -287,8 +287,7 @@ struct mmu_gather_batch { · 67 unmodified lines | ||
| 287 | 287 | */ |
| 288 | 288 | #define MAX_GATHER_BATCH_COUNT (10000UL/MAX_GATHER_BATCH) |
| 289 | 289 | |
| 290 | −extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, | |
| 291 | − bool delay_rmap, int page_size); | |
| 290 | +extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size); | |
| 292 | 291 | bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page, |
| 293 | 292 | unsigned int nr_pages, bool delay_rmap); |
| 294 | 293 | |
| @@ -510,7 +509,7 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) · 215 unmodified lines | ||
| 510 | 509 | static inline void tlb_remove_page_size(struct mmu_gather *tlb, |
| 511 | 510 | struct page *page, int page_size) |
| 512 | 511 | { |
| 513 | − if (__tlb_remove_page_size(tlb, page, false, page_size)) | |
| 512 | + if (__tlb_remove_page_size(tlb, page, page_size)) | |
| 514 | 513 | tlb_flush_mmu(tlb); |
| 515 | 514 | } |
| 516 | 515 | |
| @@ -46,7 +46,6 @@ enum wb_reason { · 45 unmodified lines | ||
| 46 | 46 | WB_REASON_VMSCAN, |
| 47 | 47 | WB_REASON_SYNC, |
| 48 | 48 | WB_REASON_PERIODIC, |
| 49 | − WB_REASON_LAPTOP_TIMER, | |
| 50 | 49 | WB_REASON_FS_FREE_SPACE, |
| 51 | 50 | /* |
| 52 | 51 | * There is no bdi forker thread any more and works are done |
| @@ -204,8 +203,6 @@ struct backing_dev_info { · 151 unmodified lines | ||
| 204 | 203 | char dev_name[64]; |
| 205 | 204 | struct device *owner; |
| 206 | 205 | |
| 207 | − struct timer_list laptop_mode_wb_timer; | |
| 208 | − | |
| 209 | 206 | #ifdef CONFIG_DEBUG_FS |
| 210 | 207 | struct dentry *debug_dir; |
| 211 | 208 | #endif |
| @@ -0,0 +1,77 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +/* | |
| 3 | + * Common interface for implementing a memory balloon, including support | |
| 4 | + * for migration of pages inflated in a memory balloon. | |
| 5 | + * | |
| 6 | + * Balloon page migration makes use of the general "movable_ops page migration" | |
| 7 | + * feature. | |
| 8 | + * | |
| 9 | + * page->private is used to reference the responsible balloon device. | |
| 10 | + * That these pages have movable_ops, and which movable_ops apply, | |
| 11 | + * is derived from the page type (PageOffline()) combined with the | |
| 12 | + * PG_movable_ops flag (PageMovableOps()). | |
| 13 | + * | |
| 14 | + * Once the page type and the PG_movable_ops are set, migration code | |
| 15 | + * can initiate page isolation by invoking the | |
| 16 | + * movable_operations()->isolate_page() callback | |
| 17 | + * | |
| 18 | + * As long as page->private is set, the page is either on the balloon list | |
| 19 | + * or isolated for migration. If page->private is not set, the page is | |
| 20 | + * either still getting inflated, or was deflated to be freed by the balloon | |
| 21 | + * driver soon. Isolation is impossible in both cases. | |
| 22 | + * | |
| 23 | + * As the page isolation scanning step a compaction thread does is a lockless | |
| 24 | + * procedure (from a page standpoint), it might bring some racy situations while | |
| 25 | + * performing balloon page migration. In order to sort out these racy scenarios | |
| 26 | + * and safely perform balloon's page migration we must, always, ensure following | |
| 27 | + * these simple rules: | |
| 28 | + * | |
| 29 | + * i. Inflation/deflation must set/clear page->private under the | |
| 30 | + * balloon_pages_lock | |
| 31 | + * | |
| 32 | + * ii. isolation or dequeueing procedure must remove the page from balloon | |
| 33 | + * device page list under balloon_pages_lock | |
| 34 | + * | |
| 35 | + * Copyright (C) 2012, Red Hat, Inc. Rafael Aquini <aquini@redhat.com> | |
| 36 | + */ | |
| 37 | +#ifndef _LINUX_BALLOON_H | |
| 38 | +#define _LINUX_BALLOON_H | |
| 39 | +#include <linux/pagemap.h> | |
| 40 | +#include <linux/page-flags.h> | |
| 41 | +#include <linux/migrate.h> | |
| 42 | +#include <linux/gfp.h> | |
| 43 | +#include <linux/err.h> | |
| 44 | +#include <linux/list.h> | |
| 45 | + | |
| 46 | +/* | |
| 47 | + * Balloon device information descriptor. | |
| 48 | + * This struct is used to allow the common balloon page migration interface | |
| 49 | + * procedures to find the proper balloon device holding memory pages they'll | |
| 50 | + * have to cope for page migration, as well as it serves the balloon driver as | |
| 51 | + * a page book-keeper for its registered balloon devices. | |
| 52 | + */ | |
| 53 | +struct balloon_dev_info { | |
| 54 | + unsigned long isolated_pages; /* # of isolated pages for migration */ | |
| 55 | + struct list_head pages; /* Pages enqueued & handled to Host */ | |
| 56 | + int (*migratepage)(struct balloon_dev_info *, struct page *newpage, | |
| 57 | + struct page *page, enum migrate_mode mode); | |
| 58 | + bool adjust_managed_page_count; | |
| 59 | +}; | |
| 60 | + | |
| 61 | +struct page *balloon_page_alloc(void); | |
| 62 | +void balloon_page_enqueue(struct balloon_dev_info *b_dev_info, | |
| 63 | + struct page *page); | |
| 64 | +struct page *balloon_page_dequeue(struct balloon_dev_info *b_dev_info); | |
| 65 | +size_t balloon_page_list_enqueue(struct balloon_dev_info *b_dev_info, | |
| 66 | + struct list_head *pages); | |
| 67 | +size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info, | |
| 68 | + struct list_head *pages, size_t n_req_pages); | |
| 69 | + | |
| 70 | +static inline void balloon_devinfo_init(struct balloon_dev_info *balloon) | |
| 71 | +{ | |
| 72 | + balloon->isolated_pages = 0; | |
| 73 | + INIT_LIST_HEAD(&balloon->pages); | |
| 74 | + balloon->migratepage = NULL; | |
| 75 | + balloon->adjust_managed_page_count = false; | |
| 76 | +} | |
| 77 | +#endif /* _LINUX_BALLOON_H */ | |
| @@ -1,160 +0,0 @@ | ||
| 1 | −/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | −/* | |
| 3 | − * include/linux/balloon_compaction.h | |
| 4 | − * | |
| 5 | − * Common interface definitions for making balloon pages movable by compaction. | |
| 6 | − * | |
| 7 | − * Balloon page migration makes use of the general "movable_ops page migration" | |
| 8 | − * feature. | |
| 9 | − * | |
| 10 | − * page->private is used to reference the responsible balloon device. | |
| 11 | − * That these pages have movable_ops, and which movable_ops apply, | |
| 12 | − * is derived from the page type (PageOffline()) combined with the | |
| 13 | − * PG_movable_ops flag (PageMovableOps()). | |
| 14 | − * | |
| 15 | − * As the page isolation scanning step a compaction thread does is a lockless | |
| 16 | − * procedure (from a page standpoint), it might bring some racy situations while | |
| 17 | − * performing balloon page compaction. In order to sort out these racy scenarios | |
| 18 | − * and safely perform balloon's page compaction and migration we must, always, | |
| 19 | − * ensure following these simple rules: | |
| 20 | − * | |
| 21 | − * i. Setting the PG_movable_ops flag and page->private with the following | |
| 22 | − * lock order | |
| 23 | − * +-page_lock(page); | |
| 24 | − * +--spin_lock_irq(&b_dev_info->pages_lock); | |
| 25 | − * | |
| 26 | − * ii. isolation or dequeueing procedure must remove the page from balloon | |
| 27 | − * device page list under b_dev_info->pages_lock. | |
| 28 | − * | |
| 29 | − * The functions provided by this interface are placed to help on coping with | |
| 30 | − * the aforementioned balloon page corner case, as well as to ensure the simple | |
| 31 | − * set of exposed rules are satisfied while we are dealing with balloon pages | |
| 32 | − * compaction / migration. | |
| 33 | − * | |
| 34 | − * Copyright (C) 2012, Red Hat, Inc. Rafael Aquini <aquini@redhat.com> | |
| 35 | − */ | |
| 36 | −#ifndef _LINUX_BALLOON_COMPACTION_H | |
| 37 | −#define _LINUX_BALLOON_COMPACTION_H | |
| 38 | −#include <linux/pagemap.h> | |
| 39 | −#include <linux/page-flags.h> | |
| 40 | −#include <linux/migrate.h> | |
| 41 | −#include <linux/gfp.h> | |
| 42 | −#include <linux/err.h> | |
| 43 | −#include <linux/fs.h> | |
| 44 | −#include <linux/list.h> | |
| 45 | − | |
| 46 | −/* | |
| 47 | − * Balloon device information descriptor. | |
| 48 | − * This struct is used to allow the common balloon compaction interface | |
| 49 | − * procedures to find the proper balloon device holding memory pages they'll | |
| 50 | − * have to cope for page compaction / migration, as well as it serves the | |
| 51 | − * balloon driver as a page book-keeper for its registered balloon devices. | |
| 52 | − */ | |
| 53 | −struct balloon_dev_info { | |
| 54 | − unsigned long isolated_pages; /* # of isolated pages for migration */ | |
| 55 | − spinlock_t pages_lock; /* Protection to pages list */ | |
| 56 | − struct list_head pages; /* Pages enqueued & handled to Host */ | |
| 57 | − int (*migratepage)(struct balloon_dev_info *, struct page *newpage, | |
| 58 | − struct page *page, enum migrate_mode mode); | |
| 59 | −}; | |
| 60 | − | |
| 61 | −extern struct page *balloon_page_alloc(void); | |
| 62 | −extern void balloon_page_enqueue(struct balloon_dev_info *b_dev_info, | |
| 63 | − struct page *page); | |
| 64 | −extern struct page *balloon_page_dequeue(struct balloon_dev_info *b_dev_info); | |
| 65 | −extern size_t balloon_page_list_enqueue(struct balloon_dev_info *b_dev_info, | |
| 66 | − struct list_head *pages); | |
| 67 | −extern size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info, | |
| 68 | − struct list_head *pages, size_t n_req_pages); | |
| 69 | − | |
| 70 | −static inline void balloon_devinfo_init(struct balloon_dev_info *balloon) | |
| 71 | −{ | |
| 72 | − balloon->isolated_pages = 0; | |
| 73 | − spin_lock_init(&balloon->pages_lock); | |
| 74 | − INIT_LIST_HEAD(&balloon->pages); | |
| 75 | − balloon->migratepage = NULL; | |
| 76 | −} | |
| 77 | − | |
| 78 | −#ifdef CONFIG_BALLOON_COMPACTION | |
| 79 | −extern const struct movable_operations balloon_mops; | |
| 80 | −/* | |
| 81 | − * balloon_page_device - get the b_dev_info descriptor for the balloon device | |
| 82 | − * that enqueues the given page. | |
| 83 | − */ | |
| 84 | −static inline struct balloon_dev_info *balloon_page_device(struct page *page) | |
| 85 | −{ | |
| 86 | − return (struct balloon_dev_info *)page_private(page); | |
| 87 | −} | |
| 88 | −#endif /* CONFIG_BALLOON_COMPACTION */ | |
| 89 | − | |
| 90 | −/* | |
| 91 | − * balloon_page_insert - insert a page into the balloon's page list and make | |
| 92 | − * the page->private assignment accordingly. | |
| 93 | − * @balloon : pointer to balloon device | |
| 94 | − * @page : page to be assigned as a 'balloon page' | |
| 95 | − * | |
| 96 | − * Caller must ensure the page is locked and the spin_lock protecting balloon | |
| 97 | − * pages list is held before inserting a page into the balloon device. | |
| 98 | − */ | |
| 99 | −static inline void balloon_page_insert(struct balloon_dev_info *balloon, | |
| 100 | − struct page *page) | |
| 101 | −{ | |
| 102 | − __SetPageOffline(page); | |
| 103 | − if (IS_ENABLED(CONFIG_BALLOON_COMPACTION)) { | |
| 104 | − SetPageMovableOps(page); | |
| 105 | − set_page_private(page, (unsigned long)balloon); | |
| 106 | − } | |
| 107 | − list_add(&page->lru, &balloon->pages); | |
| 108 | −} | |
| 109 | − | |
| 110 | −static inline gfp_t balloon_mapping_gfp_mask(void) | |
| 111 | −{ | |
| 112 | − if (IS_ENABLED(CONFIG_BALLOON_COMPACTION)) | |
| 113 | − return GFP_HIGHUSER_MOVABLE; | |
| 114 | − return GFP_HIGHUSER; | |
| 115 | −} | |
| 116 | − | |
| 117 | −/* | |
| 118 | − * balloon_page_finalize - prepare a balloon page that was removed from the | |
| 119 | − * balloon list for release to the page allocator | |
| 120 | − * @page: page to be released to the page allocator | |
| 121 | − * | |
| 122 | − * Caller must ensure that the page is locked. | |
| 123 | − */ | |
| 124 | −static inline void balloon_page_finalize(struct page *page) | |
| 125 | −{ | |
| 126 | − if (IS_ENABLED(CONFIG_BALLOON_COMPACTION)) | |
| 127 | − set_page_private(page, 0); | |
| 128 | − /* PageOffline is sticky until the page is freed to the buddy. */ | |
| 129 | −} | |
| 130 | − | |
| 131 | −/* | |
| 132 | − * balloon_page_push - insert a page into a page list. | |
| 133 | − * @head : pointer to list | |
| 134 | − * @page : page to be added | |
| 135 | − * | |
| 136 | − * Caller must ensure the page is private and protect the list. | |
| 137 | − */ | |
| 138 | −static inline void balloon_page_push(struct list_head *pages, struct page *page) | |
| 139 | −{ | |
| 140 | − list_add(&page->lru, pages); | |
| 141 | −} | |
| 142 | − | |
| 143 | −/* | |
| 144 | − * balloon_page_pop - remove a page from a page list. | |
| 145 | − * @head : pointer to list | |
| 146 | − * @page : page to be added | |
| 147 | − * | |
| 148 | − * Caller must ensure the page is private and protect the list. | |
| 149 | − */ | |
| 150 | −static inline struct page *balloon_page_pop(struct list_head *pages) | |
| 151 | −{ | |
| 152 | − struct page *page = list_first_entry_or_null(pages, struct page, lru); | |
| 153 | − | |
| 154 | − if (!page) | |
| 155 | − return NULL; | |
| 156 | − | |
| 157 | − list_del(&page->lru); | |
| 158 | − return page; | |
| 159 | −} | |
| 160 | −#endif /* _LINUX_BALLOON_COMPACTION_H */ | |
| @@ -49,9 +49,14 @@ extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, · 48 unmodified lines | ||
| 49 | 49 | struct cma **res_cma); |
| 50 | 50 | extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsigned int align, |
| 51 | 51 | bool no_warn); |
| 52 | −extern bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count); | |
| 53 | 52 | extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long count); |
| 54 | 53 | |
| 54 | +struct page *cma_alloc_frozen(struct cma *cma, unsigned long count, | |
| 55 | + unsigned int align, bool no_warn); | |
| 56 | +struct page *cma_alloc_frozen_compound(struct cma *cma, unsigned int order); | |
| 57 | +bool cma_release_frozen(struct cma *cma, const struct page *pages, | |
| 58 | + unsigned long count); | |
| 59 | + | |
| 55 | 60 | extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data); |
| 56 | 61 | extern bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end); |
| 57 | 62 | |
| @@ -66,24 +71,4 @@ static inline bool cma_skip_dt_default_reserved_mem(void) · 8 unmodified lines | ||
| 66 | 71 | } |
| 67 | 72 | #endif |
| 68 | 73 | |
| 69 | −#ifdef CONFIG_CMA | |
| 70 | −struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp); | |
| 71 | −bool cma_free_folio(struct cma *cma, const struct folio *folio); | |
| 72 | −bool cma_validate_zones(struct cma *cma); | |
| 73 | −#else | |
| 74 | −static inline struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) | |
| 75 | −{ | |
| 76 | − return NULL; | |
| 77 | −} | |
| 78 | − | |
| 79 | −static inline bool cma_free_folio(struct cma *cma, const struct folio *folio) | |
| 80 | −{ | |
| 81 | − return false; | |
| 82 | −} | |
| 83 | −static inline bool cma_validate_zones(struct cma *cma) | |
| 84 | −{ | |
| 85 | − return false; | |
| 86 | −} | |
| 87 | −#endif | |
| 88 | − | |
| 89 | 74 | #endif |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/random.h> |
| 16 | 16 | |
| 17 | 17 | /* Minimal region size. Every damon_region is aligned by this. */ |
| 18 | −#define DAMON_MIN_REGION PAGE_SIZE | |
| 18 | +#define DAMON_MIN_REGION_SZ PAGE_SIZE | |
| 19 | 19 | /* Max priority score for DAMON-based operation schemes */ |
| 20 | 20 | #define DAMOS_MAX_SCORE (99) |
| 21 | 21 | |
| @@ -155,6 +155,8 @@ enum damos_action { · 133 unmodified lines | ||
| 155 | 155 | * @DAMOS_QUOTA_NODE_MEM_FREE_BP: MemFree ratio of a node. |
| 156 | 156 | * @DAMOS_QUOTA_NODE_MEMCG_USED_BP: MemUsed ratio of a node for a cgroup. |
| 157 | 157 | * @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup. |
| 158 | + * @DAMOS_QUOTA_ACTIVE_MEM_BP: Active to total LRU memory ratio. | |
| 159 | + * @DAMOS_QUOTA_INACTIVE_MEM_BP: Inactive to total LRU memory ratio. | |
| 158 | 160 | * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics. |
| 159 | 161 | * |
| 160 | 162 | * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported. |
| @@ -166,6 +168,8 @@ enum damos_quota_goal_metric { · 5 unmodified lines | ||
| 166 | 168 | DAMOS_QUOTA_NODE_MEM_FREE_BP, |
| 167 | 169 | DAMOS_QUOTA_NODE_MEMCG_USED_BP, |
| 168 | 170 | DAMOS_QUOTA_NODE_MEMCG_FREE_BP, |
| 171 | + DAMOS_QUOTA_ACTIVE_MEM_BP, | |
| 172 | + DAMOS_QUOTA_INACTIVE_MEM_BP, | |
| 169 | 173 | NR_DAMOS_QUOTA_GOAL_METRICS, |
| 170 | 174 | }; |
| 171 | 175 | |
| @@ -203,7 +207,7 @@ struct damos_quota_goal { · 31 unmodified lines | ||
| 203 | 207 | u64 last_psi_total; |
| 204 | 208 | struct { |
| 205 | 209 | int nid; |
| 206 | − unsigned short memcg_id; | |
| 210 | + u64 memcg_id; | |
| 207 | 211 | }; |
| 208 | 212 | }; |
| 209 | 213 | struct list_head list; |
| @@ -330,6 +334,8 @@ struct damos_watermarks { · 120 unmodified lines | ||
| 330 | 334 | * @sz_ops_filter_passed: |
| 331 | 335 | * Total bytes that passed ops layer-handled DAMOS filters. |
| 332 | 336 | * @qt_exceeds: Total number of times the quota of the scheme has exceeded. |
| 337 | + * @nr_snapshots: | |
| 338 | + * Total number of DAMON snapshots that the scheme has tried. | |
| 333 | 339 | * |
| 334 | 340 | * "Tried an action to a region" in this context means the DAMOS core logic |
| 335 | 341 | * determined the region as eligible to apply the action. The access pattern |
| @@ -355,6 +361,7 @@ struct damos_stat { · 19 unmodified lines | ||
| 355 | 361 | unsigned long sz_applied; |
| 356 | 362 | unsigned long sz_ops_filter_passed; |
| 357 | 363 | unsigned long qt_exceeds; |
| 364 | + unsigned long nr_snapshots; | |
| 358 | 365 | }; |
| 359 | 366 | |
| 360 | 367 | /** |
| @@ -416,7 +423,7 @@ struct damos_filter { · 55 unmodified lines | ||
| 416 | 423 | bool matching; |
| 417 | 424 | bool allow; |
| 418 | 425 | union { |
| 419 | − unsigned short memcg_id; | |
| 426 | + u64 memcg_id; | |
| 420 | 427 | struct damon_addr_range addr_range; |
| 421 | 428 | int target_idx; |
| 422 | 429 | struct damon_size_range sz_range; |
| @@ -496,6 +503,7 @@ struct damos_migrate_dests { · 73 unmodified lines | ||
| 496 | 503 | * @ops_filters: ops layer handling &struct damos_filter objects list. |
| 497 | 504 | * @last_applied: Last @action applied ops-managing entity. |
| 498 | 505 | * @stat: Statistics of this scheme. |
| 506 | + * @max_nr_snapshots: Upper limit of nr_snapshots stat. | |
| 499 | 507 | * @list: List head for siblings. |
| 500 | 508 | * |
| 501 | 509 | * For each @apply_interval_us, DAMON finds regions which fit in the |
| @@ -529,9 +537,10 @@ struct damos_migrate_dests { · 27 unmodified lines | ||
| 529 | 537 | * unsets @last_applied when each regions walking for applying the scheme is |
| 530 | 538 | * finished. |
| 531 | 539 | * |
| 532 | − * After applying the &action to each region, &stat_count and &stat_sz is | |
| 533 | − * updated to reflect the number of regions and total size of regions that the | |
| 534 | − * &action is applied. | |
| 540 | + * After applying the &action to each region, &stat is updated. | |
| 541 | + * | |
| 542 | + * If &max_nr_snapshots is set as non-zero and &stat.nr_snapshots be same to or | |
| 543 | + * greater than it, the scheme is deactivated. | |
| 535 | 544 | */ |
| 536 | 545 | struct damos { |
| 537 | 546 | struct damos_access_pattern pattern; |
| @@ -566,6 +575,7 @@ struct damos { · 28 unmodified lines | ||
| 566 | 575 | struct list_head ops_filters; |
| 567 | 576 | void *last_applied; |
| 568 | 577 | struct damos_stat stat; |
| 578 | + unsigned long max_nr_snapshots; | |
| 569 | 579 | struct list_head list; |
| 570 | 580 | }; |
| 571 | 581 | |
| @@ -597,7 +607,6 @@ enum damon_ops_id { · 25 unmodified lines | ||
| 597 | 607 | * @apply_scheme: Apply a DAMON-based operation scheme. |
| 598 | 608 | * @target_valid: Determine if the target is valid. |
| 599 | 609 | * @cleanup_target: Clean up each target before deallocation. |
| 600 | − * @cleanup: Clean up the context. | |
| 601 | 610 | * |
| 602 | 611 | * DAMON can be extended for various address spaces and usages. For this, |
| 603 | 612 | * users should register the low level operations for their target address |
| @@ -630,7 +639,6 @@ enum damon_ops_id { · 26 unmodified lines | ||
| 630 | 639 | * @target_valid should check whether the target is still valid for the |
| 631 | 640 | * monitoring. |
| 632 | 641 | * @cleanup_target is called before the target will be deallocated. |
| 633 | − * @cleanup is called from @kdamond just before its termination. | |
| 634 | 642 | */ |
| 635 | 643 | struct damon_operations { |
| 636 | 644 | enum damon_ops_id id; |
| @@ -646,7 +654,6 @@ struct damon_operations { · 9 unmodified lines | ||
| 646 | 654 | struct damos *scheme, unsigned long *sz_filter_passed); |
| 647 | 655 | bool (*target_valid)(struct damon_target *t); |
| 648 | 656 | void (*cleanup_target)(struct damon_target *t); |
| 649 | − void (*cleanup)(struct damon_ctx *context); | |
| 650 | 657 | }; |
| 651 | 658 | |
| 652 | 659 | /* |
| @@ -656,7 +663,7 @@ struct damon_operations { · 3 unmodified lines | ||
| 656 | 663 | * @data: Data that will be passed to @fn. |
| 657 | 664 | * @repeat: Repeat invocations. |
| 658 | 665 | * @return_code: Return code from @fn invocation. |
| 659 | − * @dealloc_on_cancel: De-allocate when canceled. | |
| 666 | + * @dealloc_on_cancel: If @repeat is true, de-allocate when canceled. | |
| 660 | 667 | * |
| 661 | 668 | * Control damon_call(), which requests specific kdamond to invoke a given |
| 662 | 669 | * function. Refer to damon_call() for more details. |
| @@ -749,27 +756,24 @@ struct damon_attrs { · 86 unmodified lines | ||
| 749 | 756 | * of the monitoring. |
| 750 | 757 | * |
| 751 | 758 | * @attrs: Monitoring attributes for accuracy/overhead control. |
| 752 | − * @kdamond: Kernel thread who does the monitoring. | |
| 753 | − * @kdamond_lock: Mutex for the synchronizations with @kdamond. | |
| 754 | 759 | * |
| 755 | − * For each monitoring context, one kernel thread for the monitoring is | |
| 756 | − * created. The pointer to the thread is stored in @kdamond. | |
| 760 | + * For each monitoring context, one kernel thread for the monitoring, namely | |
| 761 | + * kdamond, is created. The pid of kdamond can be retrieved using | |
| 762 | + * damon_kdamond_pid(). | |
| 757 | 763 | * |
| 758 | − * Once started, the monitoring thread runs until explicitly required to be | |
| 759 | − * terminated or every monitoring target is invalid. The validity of the | |
| 760 | − * targets is checked via the &damon_operations.target_valid of @ops. The | |
| 761 | − * termination can also be explicitly requested by calling damon_stop(). | |
| 762 | − * The thread sets @kdamond to NULL when it terminates. Therefore, users can | |
| 763 | − * know whether the monitoring is ongoing or terminated by reading @kdamond. | |
| 764 | − * Reads and writes to @kdamond from outside of the monitoring thread must | |
| 765 | − * be protected by @kdamond_lock. | |
| 764 | + * Once started, kdamond runs until explicitly required to be terminated or | |
| 765 | + * every monitoring target is invalid. The validity of the targets is checked | |
| 766 | + * via the &damon_operations.target_valid of @ops. The termination can also be | |
| 767 | + * explicitly requested by calling damon_stop(). To know if a kdamond is | |
| 768 | + * running, damon_is_running() can be used. | |
| 766 | 769 | * |
| 767 | − * Note that the monitoring thread protects only @kdamond via @kdamond_lock. | |
| 768 | − * Accesses to other fields must be protected by themselves. | |
| 770 | + * While the kdamond is running, all accesses to &struct damon_ctx from a | |
| 771 | + * thread other than the kdamond should be made using safe DAMON APIs, | |
| 772 | + * including damon_call() and damos_walk(). | |
| 769 | 773 | * |
| 770 | 774 | * @ops: Set of monitoring operations for given use cases. |
| 771 | 775 | * @addr_unit: Scale factor for core to ops address conversion. |
| 772 | − * @min_sz_region: Minimum region size. | |
| 776 | + * @min_region_sz: Minimum region size. | |
| 773 | 777 | * @adaptive_targets: Head of monitoring targets (&damon_target) list. |
| 774 | 778 | * @schemes: Head of schemes (&damos) list. |
| 775 | 779 | */ |
| @@ -806,13 +810,15 @@ struct damon_ctx { · 30 unmodified lines | ||
| 806 | 810 | struct damos_walk_control *walk_control; |
| 807 | 811 | struct mutex walk_control_lock; |
| 808 | 812 | |
| 809 | −/* public: */ | |
| 813 | + /* Working thread of the given DAMON context */ | |
| 810 | 814 | struct task_struct *kdamond; |
| 815 | + /* Protects @kdamond field access */ | |
| 811 | 816 | struct mutex kdamond_lock; |
| 812 | 817 | |
| 818 | +/* public: */ | |
| 813 | 819 | struct damon_operations ops; |
| 814 | 820 | unsigned long addr_unit; |
| 815 | − unsigned long min_sz_region; | |
| 821 | + unsigned long min_region_sz; | |
| 816 | 822 | |
| 817 | 823 | struct list_head adaptive_targets; |
| 818 | 824 | struct list_head schemes; |
| @@ -901,7 +907,7 @@ static inline void damon_insert_region(struct damon_region *r, · 82 unmodified lines | ||
| 901 | 907 | void damon_add_region(struct damon_region *r, struct damon_target *t); |
| 902 | 908 | void damon_destroy_region(struct damon_region *r, struct damon_target *t); |
| 903 | 909 | int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, |
| 904 | − unsigned int nr_ranges, unsigned long min_sz_region); | |
| 910 | + unsigned int nr_ranges, unsigned long min_region_sz); | |
| 905 | 911 | void damon_update_region_access_rate(struct damon_region *r, bool accessed, |
| 906 | 912 | struct damon_attrs *attrs); |
| 907 | 913 | |
| @@ -962,13 +968,14 @@ bool damon_initialized(void); · 54 unmodified lines | ||
| 962 | 968 | int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive); |
| 963 | 969 | int damon_stop(struct damon_ctx **ctxs, int nr_ctxs); |
| 964 | 970 | bool damon_is_running(struct damon_ctx *ctx); |
| 971 | +int damon_kdamond_pid(struct damon_ctx *ctx); | |
| 965 | 972 | |
| 966 | 973 | int damon_call(struct damon_ctx *ctx, struct damon_call_control *control); |
| 967 | 974 | int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control); |
| 968 | 975 | |
| 969 | 976 | int damon_set_region_biggest_system_ram_default(struct damon_target *t, |
| 970 | 977 | unsigned long *start, unsigned long *end, |
| 971 | − unsigned long min_sz_region); | |
| 978 | + unsigned long min_region_sz); | |
| 972 | 979 | |
| 973 | 980 | #endif /* CONFIG_DAMON */ |
| 974 | 981 | |
| @@ -407,9 +407,15 @@ extern gfp_t gfp_allowed_mask; · 406 unmodified lines | ||
| 407 | 407 | /* Returns true if the gfp_mask allows use of ALLOC_NO_WATERMARK */ |
| 408 | 408 | bool gfp_pfmemalloc_allowed(gfp_t gfp_mask); |
| 409 | 409 | |
| 410 | +/* A helper for checking if gfp includes all the specified flags */ | |
| 411 | +static inline bool gfp_has_flags(gfp_t gfp, gfp_t flags) | |
| 412 | +{ | |
| 413 | + return (gfp & flags) == flags; | |
| 414 | +} | |
| 415 | + | |
| 410 | 416 | static inline bool gfp_has_io_fs(gfp_t gfp) |
| 411 | 417 | { |
| 412 | − return (gfp & (__GFP_IO | __GFP_FS)) == (__GFP_IO | __GFP_FS); | |
| 418 | + return gfp_has_flags(gfp, __GFP_IO | __GFP_FS); | |
| 413 | 419 | } |
| 414 | 420 | |
| 415 | 421 | /* |
| @@ -430,39 +436,29 @@ typedef unsigned int __bitwise acr_flags_t; · 14 unmodified lines | ||
| 430 | 436 | #define ACR_FLAGS_CMA ((__force acr_flags_t)BIT(0)) // allocate for CMA |
| 431 | 437 | |
| 432 | 438 | /* The below functions must be run on a range from a single zone. */ |
| 433 | −extern int alloc_contig_range_noprof(unsigned long start, unsigned long end, | |
| 434 | − acr_flags_t alloc_flags, gfp_t gfp_mask); | |
| 435 | −#define alloc_contig_range(...) alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__)) | |
| 436 | − | |
| 437 | −extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, | |
| 438 | − int nid, nodemask_t *nodemask); | |
| 439 | −#define alloc_contig_pages(...) alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__)) | |
| 440 | − | |
| 441 | −#endif | |
| 439 | +int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end, | |
| 440 | + acr_flags_t alloc_flags, gfp_t gfp_mask); | |
| 441 | +#define alloc_contig_frozen_range(...) \ | |
| 442 | + alloc_hooks(alloc_contig_frozen_range_noprof(__VA_ARGS__)) | |
| 443 | + | |
| 444 | +int alloc_contig_range_noprof(unsigned long start, unsigned long end, | |
| 445 | + acr_flags_t alloc_flags, gfp_t gfp_mask); | |
| 446 | +#define alloc_contig_range(...) \ | |
| 447 | + alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__)) | |
| 448 | + | |
| 449 | +struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages, | |
| 450 | + gfp_t gfp_mask, int nid, nodemask_t *nodemask); | |
| 451 | +#define alloc_contig_frozen_pages(...) \ | |
| 452 | + alloc_hooks(alloc_contig_frozen_pages_noprof(__VA_ARGS__)) | |
| 453 | + | |
| 454 | +struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, | |
| 455 | + int nid, nodemask_t *nodemask); | |
| 456 | +#define alloc_contig_pages(...) \ | |
| 457 | + alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__)) | |
| 458 | + | |
| 459 | +void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages); | |
| 442 | 460 | void free_contig_range(unsigned long pfn, unsigned long nr_pages); |
| 443 | − | |
| 444 | −#ifdef CONFIG_CONTIG_ALLOC | |
| 445 | −static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, | |
| 446 | − int nid, nodemask_t *node) | |
| 447 | −{ | |
| 448 | − struct page *page; | |
| 449 | − | |
| 450 | − if (WARN_ON(!order || !(gfp & __GFP_COMP))) | |
| 451 | − return NULL; | |
| 452 | − | |
| 453 | − page = alloc_contig_pages_noprof(1 << order, gfp, nid, node); | |
| 454 | − | |
| 455 | − return page ? page_folio(page) : NULL; | |
| 456 | −} | |
| 457 | −#else | |
| 458 | −static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, | |
| 459 | − int nid, nodemask_t *node) | |
| 460 | −{ | |
| 461 | − return NULL; | |
| 462 | −} | |
| 463 | 461 | #endif |
| 464 | −/* This should be paired with folio_put() rather than free_contig_range(). */ | |
| 465 | −#define folio_alloc_gigantic(...) alloc_hooks(folio_alloc_gigantic_noprof(__VA_ARGS__)) | |
| 466 | 462 | |
| 467 | 463 | DEFINE_FREE(free_page, void *, free_page((unsigned long)_T)) |
| 468 | 464 | |
| @@ -309,8 +309,10 @@ enum { · 308 unmodified lines | ||
| 309 | 309 | * |
| 310 | 310 | * %GFP_ATOMIC users can not sleep and need the allocation to succeed. A lower |
| 311 | 311 | * watermark is applied to allow access to "atomic reserves". |
| 312 | − * The current implementation doesn't support NMI and few other strict | |
| 313 | − * non-preemptive contexts (e.g. raw_spin_lock). The same applies to %GFP_NOWAIT. | |
| 312 | + * The current implementation doesn't support NMI, nor contexts that disable | |
| 313 | + * preemption under PREEMPT_RT. This includes raw_spin_lock() and plain | |
| 314 | + * preempt_disable() - see "Memory allocation" in | |
| 315 | + * Documentation/core-api/real-time/differences.rst for more info. | |
| 314 | 316 | * |
| 315 | 317 | * %GFP_KERNEL is typical for kernel-internal allocations. The caller requires |
| 316 | 318 | * %ZONE_NORMAL or a lower zone for direct access but can direct reclaim. |
| @@ -321,6 +323,7 @@ enum { · 4 unmodified lines | ||
| 321 | 323 | * %GFP_NOWAIT is for kernel allocations that should not stall for direct |
| 322 | 324 | * reclaim, start physical IO or use any filesystem callback. It is very |
| 323 | 325 | * likely to fail to allocate memory, even for very small allocations. |
| 326 | + * The same restrictions on calling contexts apply as for %GFP_ATOMIC. | |
| 324 | 327 | * |
| 325 | 328 | * %GFP_NOIO will use direct reclaim to discard clean pages or slab pages |
| 326 | 329 | * that do not require the starting of any physical IO. |
| @@ -197,15 +197,111 @@ static inline void invalidate_kernel_vmap_range(void *vaddr, int size) · 196 unmodified lines | ||
| 197 | 197 | } |
| 198 | 198 | #endif |
| 199 | 199 | |
| 200 | −/* when CONFIG_HIGHMEM is not set these will be plain clear/copy_page */ | |
| 201 | 200 | #ifndef clear_user_highpage |
| 201 | +#ifndef clear_user_page | |
| 202 | +/** | |
| 203 | + * clear_user_page() - clear a page to be mapped to user space | |
| 204 | + * @addr: the address of the page | |
| 205 | + * @vaddr: the address of the user mapping | |
| 206 | + * @page: the page | |
| 207 | + * | |
| 208 | + * We condition the definition of clear_user_page() on the architecture | |
| 209 | + * not having a custom clear_user_highpage(). That's because if there | |
| 210 | + * is some special flushing needed for clear_user_highpage() then it | |
| 211 | + * is likely that clear_user_page() also needs some magic. And, since | |
| 212 | + * our only caller is the generic clear_user_highpage(), not defining | |
| 213 | + * is not much of a loss. | |
| 214 | + */ | |
| 215 | +static inline void clear_user_page(void *addr, unsigned long vaddr, struct page *page) | |
| 216 | +{ | |
| 217 | + clear_page(addr); | |
| 218 | +} | |
| 219 | +#endif | |
| 220 | + | |
| 221 | +/** | |
| 222 | + * clear_user_pages() - clear a page range to be mapped to user space | |
| 223 | + * @addr: start address | |
| 224 | + * @vaddr: start address of the user mapping | |
| 225 | + * @page: start page | |
| 226 | + * @npages: number of pages | |
| 227 | + * | |
| 228 | + * Assumes that the region (@addr, +@npages) has been validated | |
| 229 | + * already so this does no exception handling. | |
| 230 | + * | |
| 231 | + * If the architecture provides a clear_user_page(), use that; | |
| 232 | + * otherwise, we can safely use clear_pages(). | |
| 233 | + */ | |
| 234 | +static inline void clear_user_pages(void *addr, unsigned long vaddr, | |
| 235 | + struct page *page, unsigned int npages) | |
| 236 | +{ | |
| 237 | + | |
| 238 | +#ifdef clear_user_page | |
| 239 | + do { | |
| 240 | + clear_user_page(addr, vaddr, page); | |
| 241 | + addr += PAGE_SIZE; | |
| 242 | + vaddr += PAGE_SIZE; | |
| 243 | + page++; | |
| 244 | + } while (--npages); | |
| 245 | +#else | |
| 246 | + /* | |
| 247 | + * Prefer clear_pages() to allow for architectural optimizations | |
| 248 | + * when operating on contiguous page ranges. | |
| 249 | + */ | |
| 250 | + clear_pages(addr, npages); | |
| 251 | +#endif | |
| 252 | +} | |
| 253 | + | |
| 254 | +/** | |
| 255 | + * clear_user_highpage() - clear a page to be mapped to user space | |
| 256 | + * @page: start page | |
| 257 | + * @vaddr: start address of the user mapping | |
| 258 | + * | |
| 259 | + * With !CONFIG_HIGHMEM this (and the copy_user_highpage() below) will | |
| 260 | + * be plain clear_user_page() (and copy_user_page()). | |
| 261 | + */ | |
| 202 | 262 | static inline void clear_user_highpage(struct page *page, unsigned long vaddr) |
| 203 | 263 | { |
| 204 | 264 | void *addr = kmap_local_page(page); |
| 205 | 265 | clear_user_page(addr, vaddr, page); |
| 206 | 266 | kunmap_local(addr); |
| 207 | 267 | } |
| 268 | +#endif /* clear_user_highpage */ | |
| 269 | + | |
| 270 | +/** | |
| 271 | + * clear_user_highpages() - clear a page range to be mapped to user space | |
| 272 | + * @page: start page | |
| 273 | + * @vaddr: start address of the user mapping | |
| 274 | + * @npages: number of pages | |
| 275 | + * | |
| 276 | + * Assumes that all the pages in the region (@page, +@npages) are valid | |
| 277 | + * so this does no exception handling. | |
| 278 | + */ | |
| 279 | +static inline void clear_user_highpages(struct page *page, unsigned long vaddr, | |
| 280 | + unsigned int npages) | |
| 281 | +{ | |
| 282 | + | |
| 283 | +#if defined(clear_user_highpage) || defined(CONFIG_HIGHMEM) | |
| 284 | + /* | |
| 285 | + * An architecture defined clear_user_highpage() implies special | |
| 286 | + * handling is needed. | |
| 287 | + * | |
| 288 | + * So we use that or, the generic variant if CONFIG_HIGHMEM is | |
| 289 | + * enabled. | |
| 290 | + */ | |
| 291 | + do { | |
| 292 | + clear_user_highpage(page, vaddr); | |
| 293 | + vaddr += PAGE_SIZE; | |
| 294 | + page++; | |
| 295 | + } while (--npages); | |
| 296 | +#else | |
| 297 | + | |
| 298 | + /* | |
| 299 | + * Prefer clear_user_pages() to allow for architectural optimizations | |
| 300 | + * when operating on contiguous page ranges. | |
| 301 | + */ | |
| 302 | + clear_user_pages(page_address(page), vaddr, page, npages); | |
| 208 | 303 | #endif |
| 304 | +} | |
| 209 | 305 | |
| 210 | 306 | #ifndef vma_alloc_zeroed_movable_folio |
| 211 | 307 | /** |
| @@ -171,11 +171,11 @@ bool hugetlbfs_pagecache_present(struct hstate *h, · 170 unmodified lines | ||
| 171 | 171 | |
| 172 | 172 | struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio); |
| 173 | 173 | |
| 174 | +extern int movable_gigantic_pages __read_mostly; | |
| 174 | 175 | extern int sysctl_hugetlb_shm_group __read_mostly; |
| 175 | 176 | extern struct list_head huge_boot_pages[MAX_NUMNODES]; |
| 176 | 177 | |
| 177 | 178 | void hugetlb_bootmem_alloc(void); |
| 178 | −bool hugetlb_bootmem_allocated(void); | |
| 179 | 179 | extern nodemask_t hugetlb_bootmem_nodes; |
| 180 | 180 | void hugetlb_bootmem_set_nodes(void); |
| 181 | 181 | |
| @@ -280,6 +280,8 @@ void fixup_hugetlb_reservations(struct vm_area_struct *vma); · 98 unmodified lines | ||
| 280 | 280 | void hugetlb_split(struct vm_area_struct *vma, unsigned long addr); |
| 281 | 281 | int hugetlb_vma_lock_alloc(struct vm_area_struct *vma); |
| 282 | 282 | |
| 283 | +unsigned int arch_hugetlb_cma_order(void); | |
| 284 | + | |
| 283 | 285 | #else /* !CONFIG_HUGETLB_PAGE */ |
| 284 | 286 | |
| 285 | 287 | static inline void hugetlb_dup_vma_private(struct vm_area_struct *vma) |
| @@ -929,7 +931,7 @@ static inline bool hugepage_movable_supported(struct hstate *h) · 643 unmodified lines | ||
| 929 | 931 | if (!hugepage_migration_supported(h)) |
| 930 | 932 | return false; |
| 931 | 933 | |
| 932 | − if (hstate_is_gigantic(h)) | |
| 934 | + if (hstate_is_gigantic(h) && !movable_gigantic_pages) | |
| 933 | 935 | return false; |
| 934 | 936 | return true; |
| 935 | 937 | } |
| @@ -1303,11 +1305,6 @@ static inline bool hugetlbfs_pagecache_present( · 367 unmodified lines | ||
| 1303 | 1305 | static inline void hugetlb_bootmem_alloc(void) |
| 1304 | 1306 | { |
| 1305 | 1307 | } |
| 1306 | − | |
| 1307 | −static inline bool hugetlb_bootmem_allocated(void) | |
| 1308 | −{ | |
| 1309 | − return false; | |
| 1310 | −} | |
| 1311 | 1308 | #endif /* CONFIG_HUGETLB_PAGE */ |
| 1312 | 1309 | |
| 1313 | 1310 | static inline spinlock_t *huge_pte_lock(struct hstate *h, |
| @@ -1321,9 +1318,9 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, · 7 unmodified lines | ||
| 1321 | 1318 | } |
| 1322 | 1319 | |
| 1323 | 1320 | #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA) |
| 1324 | −extern void __init hugetlb_cma_reserve(int order); | |
| 1321 | +extern void __init hugetlb_cma_reserve(void); | |
| 1325 | 1322 | #else |
| 1326 | −static inline __init void hugetlb_cma_reserve(int order) | |
| 1323 | +static inline __init void hugetlb_cma_reserve(void) | |
| 1327 | 1324 | { |
| 1328 | 1325 | } |
| 1329 | 1326 | #endif |
| @@ -17,8 +17,8 @@ extern void khugepaged_enter_vma(struct vm_area_struct *vma, · 16 unmodified lines | ||
| 17 | 17 | vm_flags_t vm_flags); |
| 18 | 18 | extern void khugepaged_min_free_kbytes_update(void); |
| 19 | 19 | extern bool current_is_khugepaged(void); |
| 20 | −extern int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, | |
| 21 | − bool install_pmd); | |
| 20 | +void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, | |
| 21 | + bool install_pmd); | |
| 22 | 22 | |
| 23 | 23 | static inline void khugepaged_fork(struct mm_struct *mm, struct mm_struct *oldmm) |
| 24 | 24 | { |
| @@ -42,10 +42,9 @@ static inline void khugepaged_enter_vma(struct vm_area_struct *vma, · 17 unmodified lines | ||
| 42 | 42 | vm_flags_t vm_flags) |
| 43 | 43 | { |
| 44 | 44 | } |
| 45 | −static inline int collapse_pte_mapped_thp(struct mm_struct *mm, | |
| 46 | − unsigned long addr, bool install_pmd) | |
| 45 | +static inline void collapse_pte_mapped_thp(struct mm_struct *mm, | |
| 46 | + unsigned long addr, bool install_pmd) | |
| 47 | 47 | { |
| 48 | − return 0; | |
| 49 | 48 | } |
| 50 | 49 | |
| 51 | 50 | static inline void khugepaged_min_free_kbytes_update(void) |
| @@ -129,13 +129,6 @@ struct maple_arange_64 { · 128 unmodified lines | ||
| 129 | 129 | struct maple_metadata meta; |
| 130 | 130 | }; |
| 131 | 131 | |
| 132 | −struct maple_alloc { | |
| 133 | − unsigned long total; | |
| 134 | − unsigned char node_count; | |
| 135 | − unsigned int request_count; | |
| 136 | − struct maple_alloc *slot[MAPLE_ALLOC_SLOTS]; | |
| 137 | −}; | |
| 138 | − | |
| 139 | 132 | struct maple_topiary { |
| 140 | 133 | struct maple_pnode *parent; |
| 141 | 134 | struct maple_enode *next; /* Overlaps the pivot */ |
| @@ -306,7 +299,6 @@ struct maple_node { · 164 unmodified lines | ||
| 306 | 299 | }; |
| 307 | 300 | struct maple_range_64 mr64; |
| 308 | 301 | struct maple_arange_64 ma64; |
| 309 | − struct maple_alloc alloc; | |
| 310 | 302 | }; |
| 311 | 303 | }; |
| 312 | 304 | |
| @@ -536,7 +528,6 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp); · 223 unmodified lines | ||
| 536 | 528 | void mas_pause(struct ma_state *mas); |
| 537 | 529 | void maple_tree_init(void); |
| 538 | 530 | void mas_destroy(struct ma_state *mas); |
| 539 | −int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries); | |
| 540 | 531 | |
| 541 | 532 | void *mas_prev(struct ma_state *mas, unsigned long min); |
| 542 | 533 | void *mas_prev_range(struct ma_state *mas, unsigned long max); |
| @@ -598,9 +598,9 @@ extern void *alloc_large_system_hash(const char *tablename, · 597 unmodified lines | ||
| 598 | 598 | */ |
| 599 | 599 | #ifdef CONFIG_NUMA |
| 600 | 600 | #define HASHDIST_DEFAULT IS_ENABLED(CONFIG_64BIT) |
| 601 | −extern int hashdist; /* Distribute hashes across NUMA nodes? */ | |
| 601 | +extern bool hashdist; /* Distribute hashes across NUMA nodes? */ | |
| 602 | 602 | #else |
| 603 | −#define hashdist (0) | |
| 603 | +#define hashdist (false) | |
| 604 | 604 | #endif |
| 605 | 605 | |
| 606 | 606 | #ifdef CONFIG_MEMTEST |
| @@ -65,7 +65,7 @@ struct mem_cgroup_reclaim_cookie { · 64 unmodified lines | ||
| 65 | 65 | |
| 66 | 66 | #define MEM_CGROUP_ID_SHIFT 16 |
| 67 | 67 | |
| 68 | −struct mem_cgroup_id { | |
| 68 | +struct mem_cgroup_private_id { | |
| 69 | 69 | int id; |
| 70 | 70 | refcount_t ref; |
| 71 | 71 | }; |
| @@ -191,7 +191,7 @@ struct mem_cgroup { · 119 unmodified lines | ||
| 191 | 191 | struct cgroup_subsys_state css; |
| 192 | 192 | |
| 193 | 193 | /* Private memcg ID. Used to ID objects that outlive the cgroup */ |
| 194 | − struct mem_cgroup_id id; | |
| 194 | + struct mem_cgroup_private_id id; | |
| 195 | 195 | |
| 196 | 196 | /* Accounted resources */ |
| 197 | 197 | struct page_counter memory; /* Both v1 & v2 */ |
| @@ -557,13 +557,15 @@ static inline bool mem_cgroup_disabled(void) · 359 unmodified lines | ||
| 557 | 557 | static inline void mem_cgroup_protection(struct mem_cgroup *root, |
| 558 | 558 | struct mem_cgroup *memcg, |
| 559 | 559 | unsigned long *min, |
| 560 | − unsigned long *low) | |
| 560 | + unsigned long *low, | |
| 561 | + unsigned long *usage) | |
| 561 | 562 | { |
| 562 | − *min = *low = 0; | |
| 563 | + *min = *low = *usage = 0; | |
| 563 | 564 | |
| 564 | 565 | if (mem_cgroup_disabled()) |
| 565 | 566 | return; |
| 566 | 567 | |
| 568 | + *usage = page_counter_read(&memcg->memory); | |
| 567 | 569 | /* |
| 568 | 570 | * There is no reclaim protection applied to a targeted reclaim. |
| 569 | 571 | * We are special casing this specific case here because |
| @@ -819,23 +821,21 @@ void mem_cgroup_iter_break(struct mem_cgroup *, struct mem_cgroup *); · 249 unmodified lines | ||
| 819 | 821 | void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, |
| 820 | 822 | int (*)(struct task_struct *, void *), void *arg); |
| 821 | 823 | |
| 822 | −static inline unsigned short mem_cgroup_id(struct mem_cgroup *memcg) | |
| 824 | +static inline unsigned short mem_cgroup_private_id(struct mem_cgroup *memcg) | |
| 823 | 825 | { |
| 824 | 826 | if (mem_cgroup_disabled()) |
| 825 | 827 | return 0; |
| 826 | 828 | |
| 827 | 829 | return memcg->id.id; |
| 828 | 830 | } |
| 829 | −struct mem_cgroup *mem_cgroup_from_id(unsigned short id); | |
| 831 | +struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id); | |
| 830 | 832 | |
| 831 | −#ifdef CONFIG_SHRINKER_DEBUG | |
| 832 | −static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg) | |
| 833 | +static inline u64 mem_cgroup_id(struct mem_cgroup *memcg) | |
| 833 | 834 | { |
| 834 | − return memcg ? cgroup_ino(memcg->css.cgroup) : 0; | |
| 835 | + return memcg ? cgroup_id(memcg->css.cgroup) : 0; | |
| 835 | 836 | } |
| 836 | 837 | |
| 837 | −struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino); | |
| 838 | −#endif | |
| 838 | +struct mem_cgroup *mem_cgroup_get_from_id(u64 id); | |
| 839 | 839 | |
| 840 | 840 | static inline struct mem_cgroup *mem_cgroup_from_seq(struct seq_file *m) |
| 841 | 841 | { |
| @@ -919,8 +919,6 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask) · 77 unmodified lines | ||
| 919 | 919 | |
| 920 | 920 | unsigned long mem_cgroup_get_max(struct mem_cgroup *memcg); |
| 921 | 921 | |
| 922 | −unsigned long mem_cgroup_size(struct mem_cgroup *memcg); | |
| 923 | − | |
| 924 | 922 | void mem_cgroup_print_oom_context(struct mem_cgroup *memcg, |
| 925 | 923 | struct task_struct *p); |
| 926 | 924 | |
| @@ -1108,9 +1106,10 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, · 181 unmodified lines | ||
| 1108 | 1106 | static inline void mem_cgroup_protection(struct mem_cgroup *root, |
| 1109 | 1107 | struct mem_cgroup *memcg, |
| 1110 | 1108 | unsigned long *min, |
| 1111 | − unsigned long *low) | |
| 1109 | + unsigned long *low, | |
| 1110 | + unsigned long *usage) | |
| 1112 | 1111 | { |
| 1113 | − *min = *low = 0; | |
| 1112 | + *min = *low = *usage = 0; | |
| 1114 | 1113 | } |
| 1115 | 1114 | |
| 1116 | 1115 | static inline void mem_cgroup_calculate_protection(struct mem_cgroup *root, |
| @@ -1283,29 +1282,27 @@ static inline void mem_cgroup_scan_tasks(struct mem_cgroup *memcg, · 166 unmodified lines | ||
| 1283 | 1282 | { |
| 1284 | 1283 | } |
| 1285 | 1284 | |
| 1286 | −static inline unsigned short mem_cgroup_id(struct mem_cgroup *memcg) | |
| 1285 | +static inline unsigned short mem_cgroup_private_id(struct mem_cgroup *memcg) | |
| 1287 | 1286 | { |
| 1288 | 1287 | return 0; |
| 1289 | 1288 | } |
| 1290 | 1289 | |
| 1291 | −static inline struct mem_cgroup *mem_cgroup_from_id(unsigned short id) | |
| 1290 | +static inline struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id) | |
| 1292 | 1291 | { |
| 1293 | 1292 | WARN_ON_ONCE(id); |
| 1294 | 1293 | /* XXX: This should always return root_mem_cgroup */ |
| 1295 | 1294 | return NULL; |
| 1296 | 1295 | } |
| 1297 | 1296 | |
| 1298 | −#ifdef CONFIG_SHRINKER_DEBUG | |
| 1299 | −static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg) | |
| 1297 | +static inline u64 mem_cgroup_id(struct mem_cgroup *memcg) | |
| 1300 | 1298 | { |
| 1301 | 1299 | return 0; |
| 1302 | 1300 | } |
| 1303 | 1301 | |
| 1304 | −static inline struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino) | |
| 1302 | +static inline struct mem_cgroup *mem_cgroup_get_from_id(u64 id) | |
| 1305 | 1303 | { |
| 1306 | 1304 | return NULL; |
| 1307 | 1305 | } |
| 1308 | −#endif | |
| 1309 | 1306 | |
| 1310 | 1307 | static inline struct mem_cgroup *mem_cgroup_from_seq(struct seq_file *m) |
| 1311 | 1308 | { |
| @@ -1334,11 +1331,6 @@ static inline unsigned long mem_cgroup_get_max(struct mem_cgroup *memcg) · 22 unmodified lines | ||
| 1334 | 1331 | return 0; |
| 1335 | 1332 | } |
| 1336 | 1333 | |
| 1337 | −static inline unsigned long mem_cgroup_size(struct mem_cgroup *memcg) | |
| 1338 | −{ | |
| 1339 | − return 0; | |
| 1340 | −} | |
| 1341 | − | |
| 1342 | 1334 | static inline void |
| 1343 | 1335 | mem_cgroup_print_oom_context(struct mem_cgroup *memcg, struct task_struct *p) |
| 1344 | 1336 | { |
| @@ -1750,7 +1742,7 @@ static inline int memcg_kmem_id(struct mem_cgroup *memcg) · 405 unmodified lines | ||
| 1750 | 1742 | return memcg ? memcg->kmemcg_id : -1; |
| 1751 | 1743 | } |
| 1752 | 1744 | |
| 1753 | −struct mem_cgroup *mem_cgroup_from_slab_obj(void *p); | |
| 1745 | +struct mem_cgroup *mem_cgroup_from_virt(void *p); | |
| 1754 | 1746 | |
| 1755 | 1747 | static inline void count_objcg_events(struct obj_cgroup *objcg, |
| 1756 | 1748 | enum vm_event_item idx, |
| @@ -1822,7 +1814,7 @@ static inline int memcg_kmem_id(struct mem_cgroup *memcg) · 65 unmodified lines | ||
| 1822 | 1814 | return -1; |
| 1823 | 1815 | } |
| 1824 | 1816 | |
| 1825 | −static inline struct mem_cgroup *mem_cgroup_from_slab_obj(void *p) | |
| 1817 | +static inline struct mem_cgroup *mem_cgroup_from_virt(void *p) | |
| 1826 | 1818 | { |
| 1827 | 1819 | return NULL; |
| 1828 | 1820 | } |
| @@ -46,6 +46,7 @@ struct pt_regs; · 45 unmodified lines | ||
| 46 | 46 | struct folio_batch; |
| 47 | 47 | |
| 48 | 48 | void arch_mm_preinit(void); |
| 49 | +void mm_core_init_early(void); | |
| 49 | 50 | void mm_core_init(void); |
| 50 | 51 | void init_mm_internals(void); |
| 51 | 52 | |
| @@ -1008,10 +1009,7 @@ static inline void vma_flag_set_atomic(struct vm_area_struct *vma, · 956 unmodified lines | ||
| 1008 | 1009 | { |
| 1009 | 1010 | unsigned long *bitmap = ACCESS_PRIVATE(&vma->flags, __vma_flags); |
| 1010 | 1011 | |
| 1011 | − /* mmap read lock/VMA read lock must be held. */ | |
| 1012 | − if (!rwsem_is_locked(&vma->vm_mm->mmap_lock)) | |
| 1013 | − vma_assert_locked(vma); | |
| 1014 | − | |
| 1012 | + vma_assert_stabilised(vma); | |
| 1015 | 1013 | if (__vma_flag_atomic_valid(vma, bit)) |
| 1016 | 1014 | set_bit((__force int)bit, bitmap); |
| 1017 | 1015 | } |
| @@ -2906,6 +2904,13 @@ static inline unsigned long get_mm_rss(struct mm_struct *mm) · 1888 unmodified lines | ||
| 2906 | 2904 | get_mm_counter(mm, MM_SHMEMPAGES); |
| 2907 | 2905 | } |
| 2908 | 2906 | |
| 2907 | +static inline unsigned long get_mm_rss_sum(struct mm_struct *mm) | |
| 2908 | +{ | |
| 2909 | + return get_mm_counter_sum(mm, MM_FILEPAGES) + | |
| 2910 | + get_mm_counter_sum(mm, MM_ANONPAGES) + | |
| 2911 | + get_mm_counter_sum(mm, MM_SHMEMPAGES); | |
| 2912 | +} | |
| 2913 | + | |
| 2909 | 2914 | static inline unsigned long get_mm_hiwater_rss(struct mm_struct *mm) |
| 2910 | 2915 | { |
| 2911 | 2916 | return max(mm->hiwater_rss, get_mm_rss(mm)); |
| @@ -3518,7 +3523,7 @@ static inline unsigned long get_num_physpages(void) · 606 unmodified lines | ||
| 3518 | 3523 | } |
| 3519 | 3524 | |
| 3520 | 3525 | /* |
| 3521 | − * Using memblock node mappings, an architecture may initialise its | |
| 3526 | + * FIXME: Using memblock node mappings, an architecture may initialise its | |
| 3522 | 3527 | * zones, allocate the backing mem_map and account for memory holes in an |
| 3523 | 3528 | * architecture independent manner. |
| 3524 | 3529 | * |
| @@ -3533,7 +3538,7 @@ static inline unsigned long get_num_physpages(void) · 8 unmodified lines | ||
| 3533 | 3538 | * memblock_add_node(base, size, nid, MEMBLOCK_NONE) |
| 3534 | 3539 | * free_area_init(max_zone_pfns); |
| 3535 | 3540 | */ |
| 3536 | −void free_area_init(unsigned long *max_zone_pfn); | |
| 3541 | +void arch_zone_limits_init(unsigned long *max_zone_pfn); | |
| 3537 | 3542 | unsigned long node_map_pfn_alignment(void); |
| 3538 | 3543 | extern unsigned long absent_pages_in_range(unsigned long start_pfn, |
| 3539 | 3544 | unsigned long end_pfn); |
| @@ -4180,6 +4185,61 @@ static inline void clear_page_guard(struct zone *zone, struct page *page, · 640 unmodified lines | ||
| 4180 | 4185 | unsigned int order) {} |
| 4181 | 4186 | #endif /* CONFIG_DEBUG_PAGEALLOC */ |
| 4182 | 4187 | |
| 4188 | +#ifndef clear_pages | |
| 4189 | +/** | |
| 4190 | + * clear_pages() - clear a page range for kernel-internal use. | |
| 4191 | + * @addr: start address | |
| 4192 | + * @npages: number of pages | |
| 4193 | + * | |
| 4194 | + * Use clear_user_pages() instead when clearing a page range to be | |
| 4195 | + * mapped to user space. | |
| 4196 | + * | |
| 4197 | + * Does absolutely no exception handling. | |
| 4198 | + * | |
| 4199 | + * Note that even though the clearing operation is preemptible, clear_pages() | |
| 4200 | + * does not (and on architectures where it reduces to a few long-running | |
| 4201 | + * instructions, might not be able to) call cond_resched() to check if | |
| 4202 | + * rescheduling is required. | |
| 4203 | + * | |
| 4204 | + * When running under preemptible models this is not a problem. Under | |
| 4205 | + * cooperatively scheduled models, however, the caller is expected to | |
| 4206 | + * limit @npages to no more than PROCESS_PAGES_NON_PREEMPT_BATCH. | |
| 4207 | + */ | |
| 4208 | +static inline void clear_pages(void *addr, unsigned int npages) | |
| 4209 | +{ | |
| 4210 | + do { | |
| 4211 | + clear_page(addr); | |
| 4212 | + addr += PAGE_SIZE; | |
| 4213 | + } while (--npages); | |
| 4214 | +} | |
| 4215 | +#endif | |
| 4216 | + | |
| 4217 | +#ifndef PROCESS_PAGES_NON_PREEMPT_BATCH | |
| 4218 | +#ifdef clear_pages | |
| 4219 | +/* | |
| 4220 | + * The architecture defines clear_pages(), and we assume that it is | |
| 4221 | + * generally "fast". So choose a batch size large enough to allow the processor | |
| 4222 | + * headroom for optimizing the operation and yet small enough that we see | |
| 4223 | + * reasonable preemption latency for when this optimization is not possible | |
| 4224 | + * (ex. slow microarchitectures, memory bandwidth saturation.) | |
| 4225 | + * | |
| 4226 | + * With a value of 32MB and assuming a memory bandwidth of ~10GBps, this should | |
| 4227 | + * result in worst case preemption latency of around 3ms when clearing pages. | |
| 4228 | + * | |
| 4229 | + * (See comment above clear_pages() for why preemption latency is a concern | |
| 4230 | + * here.) | |
| 4231 | + */ | |
| 4232 | +#define PROCESS_PAGES_NON_PREEMPT_BATCH (SZ_32M >> PAGE_SHIFT) | |
| 4233 | +#else /* !clear_pages */ | |
| 4234 | +/* | |
| 4235 | + * The architecture does not provide a clear_pages() implementation. Assume | |
| 4236 | + * that clear_page() -- which clear_pages() will fallback to -- is relatively | |
| 4237 | + * slow and choose a small value for PROCESS_PAGES_NON_PREEMPT_BATCH. | |
| 4238 | + */ | |
| 4239 | +#define PROCESS_PAGES_NON_PREEMPT_BATCH 1 | |
| 4240 | +#endif | |
| 4241 | +#endif | |
| 4242 | + | |
| 4183 | 4243 | #ifdef __HAVE_ARCH_GATE_AREA |
| 4184 | 4244 | extern struct vm_area_struct *get_gate_vma(struct mm_struct *mm); |
| 4185 | 4245 | extern int in_gate_area_no_mm(unsigned long addr); |
| @@ -752,8 +752,18 @@ static inline struct anon_vma_name *anon_vma_name_alloc(const char *name) · 751 unmodified lines | ||
| 752 | 752 | } |
| 753 | 753 | #endif |
| 754 | 754 | |
| 755 | −#define VMA_LOCK_OFFSET 0x40000000 | |
| 756 | −#define VMA_REF_LIMIT (VMA_LOCK_OFFSET - 1) | |
| 755 | +/* | |
| 756 | + * While __vma_enter_locked() is working to ensure are no read-locks held on a | |
| 757 | + * VMA (either while acquiring a VMA write lock or marking a VMA detached) we | |
| 758 | + * set the VM_REFCNT_EXCLUDE_READERS_FLAG in vma->vm_refcnt to indiciate to | |
| 759 | + * vma_start_read() that the reference count should be left alone. | |
| 760 | + * | |
| 761 | + * See the comment describing vm_refcnt in vm_area_struct for details as to | |
| 762 | + * which values the VMA reference count can be. | |
| 763 | + */ | |
| 764 | +#define VM_REFCNT_EXCLUDE_READERS_BIT (30) | |
| 765 | +#define VM_REFCNT_EXCLUDE_READERS_FLAG (1U << VM_REFCNT_EXCLUDE_READERS_BIT) | |
| 766 | +#define VM_REFCNT_LIMIT (VM_REFCNT_EXCLUDE_READERS_FLAG - 1) | |
| 757 | 767 | |
| 758 | 768 | struct vma_numab_state { |
| 759 | 769 | /* |
| @@ -935,10 +945,10 @@ struct vm_area_struct { · 175 unmodified lines | ||
| 935 | 945 | /* |
| 936 | 946 | * Can only be written (using WRITE_ONCE()) while holding both: |
| 937 | 947 | * - mmap_lock (in write mode) |
| 938 | − * - vm_refcnt bit at VMA_LOCK_OFFSET is set | |
| 948 | + * - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_FLAG is set | |
| 939 | 949 | * Can be read reliably while holding one of: |
| 940 | 950 | * - mmap_lock (in read or write mode) |
| 941 | − * - vm_refcnt bit at VMA_LOCK_OFFSET is set or vm_refcnt > 1 | |
| 951 | + * - vm_refcnt bit at VM_REFCNT_EXCLUDE_READERS_BIT is set or vm_refcnt > 1 | |
| 942 | 952 | * Can be read unreliably (using READ_ONCE()) for pessimistic bailout |
| 943 | 953 | * while holding nothing (except RCU to keep the VMA struct allocated). |
| 944 | 954 | * |
| @@ -980,7 +990,44 @@ struct vm_area_struct { · 35 unmodified lines | ||
| 980 | 990 | struct vma_numab_state *numab_state; /* NUMA Balancing state */ |
| 981 | 991 | #endif |
| 982 | 992 | #ifdef CONFIG_PER_VMA_LOCK |
| 983 | − /* Unstable RCU readers are allowed to read this. */ | |
| 993 | + /* | |
| 994 | + * Used to keep track of firstly, whether the VMA is attached, secondly, | |
| 995 | + * if attached, how many read locks are taken, and thirdly, if the | |
| 996 | + * VM_REFCNT_EXCLUDE_READERS_FLAG is set, whether any read locks held | |
| 997 | + * are currently in the process of being excluded. | |
| 998 | + * | |
| 999 | + * This value can be equal to: | |
| 1000 | + * | |
| 1001 | + * 0 - Detached. IMPORTANT: when the refcnt is zero, readers cannot | |
| 1002 | + * increment it. | |
| 1003 | + * | |
| 1004 | + * 1 - Attached and either unlocked or write-locked. Write locks are | |
| 1005 | + * identified via __is_vma_write_locked() which checks for equality of | |
| 1006 | + * vma->vm_lock_seq and mm->mm_lock_seq. | |
| 1007 | + * | |
| 1008 | + * >1, < VM_REFCNT_EXCLUDE_READERS_FLAG - Read-locked or (unlikely) | |
| 1009 | + * write-locked with other threads having temporarily incremented the | |
| 1010 | + * reference count prior to determining it is write-locked and | |
| 1011 | + * decrementing it again. | |
| 1012 | + * | |
| 1013 | + * VM_REFCNT_EXCLUDE_READERS_FLAG - Detached, pending | |
| 1014 | + * __vma_end_exclude_readers() completion which will decrement the | |
| 1015 | + * reference count to zero. IMPORTANT - at this stage no further readers | |
| 1016 | + * can increment the reference count. It can only be reduced. | |
| 1017 | + * | |
| 1018 | + * VM_REFCNT_EXCLUDE_READERS_FLAG + 1 - A thread is either write-locking | |
| 1019 | + * an attached VMA and has yet to invoke __vma_end_exclude_readers(), | |
| 1020 | + * OR a thread is detaching a VMA and is waiting on a single spurious | |
| 1021 | + * reader in order to decrement the reference count. IMPORTANT - as | |
| 1022 | + * above, no further readers can increment the reference count. | |
| 1023 | + * | |
| 1024 | + * > VM_REFCNT_EXCLUDE_READERS_FLAG + 1 - A thread is either | |
| 1025 | + * write-locking or detaching a VMA is waiting on readers to | |
| 1026 | + * exit. IMPORTANT - as above, no further readers can increment the | |
| 1027 | + * reference count. | |
| 1028 | + * | |
| 1029 | + * NOTE: Unstable RCU readers are allowed to read this. | |
| 1030 | + */ | |
| 984 | 1031 | refcount_t vm_refcnt ____cacheline_aligned_in_smp; |
| 985 | 1032 | #ifdef CONFIG_DEBUG_LOCK_ALLOC |
| 986 | 1033 | struct lockdep_map vmlock_dep_map; |
| @@ -88,4 +88,9 @@ struct tlbflush_unmap_batch { · 87 unmodified lines | ||
| 88 | 88 | #endif |
| 89 | 89 | }; |
| 90 | 90 | |
| 91 | +struct lazy_mmu_state { | |
| 92 | + u8 enable_count; | |
| 93 | + u8 pause_count; | |
| 94 | +}; | |
| 95 | + | |
| 91 | 96 | #endif /* _LINUX_MM_TYPES_TASK_H */ |
| @@ -78,6 +78,43 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm) · 77 unmodified lines | ||
| 78 | 78 | |
| 79 | 79 | #ifdef CONFIG_PER_VMA_LOCK |
| 80 | 80 | |
| 81 | +#ifdef CONFIG_LOCKDEP | |
| 82 | +#define __vma_lockdep_map(vma) (&vma->vmlock_dep_map) | |
| 83 | +#else | |
| 84 | +#define __vma_lockdep_map(vma) NULL | |
| 85 | +#endif | |
| 86 | + | |
| 87 | +/* | |
| 88 | + * VMA locks do not behave like most ordinary locks found in the kernel, so we | |
| 89 | + * cannot quite have full lockdep tracking in the way we would ideally prefer. | |
| 90 | + * | |
| 91 | + * Read locks act as shared locks which exclude an exclusive lock being | |
| 92 | + * taken. We therefore mark these accordingly on read lock acquire/release. | |
| 93 | + * | |
| 94 | + * Write locks are acquired exclusively per-VMA, but released in a shared | |
| 95 | + * fashion, that is upon vma_end_write_all(), we update the mmap's seqcount such | |
| 96 | + * that write lock is released. | |
| 97 | + * | |
| 98 | + * We therefore cannot track write locks per-VMA, nor do we try. Mitigating this | |
| 99 | + * is the fact that, of course, we do lockdep-track the mmap lock rwsem which | |
| 100 | + * must be held when taking a VMA write lock. | |
| 101 | + * | |
| 102 | + * We do, however, want to indicate that during either acquisition of a VMA | |
| 103 | + * write lock or detachment of a VMA that we require the lock held be exclusive, | |
| 104 | + * so we utilise lockdep to do so. | |
| 105 | + */ | |
| 106 | +#define __vma_lockdep_acquire_read(vma) \ | |
| 107 | + lock_acquire_shared(__vma_lockdep_map(vma), 0, 1, NULL, _RET_IP_) | |
| 108 | +#define __vma_lockdep_release_read(vma) \ | |
| 109 | + lock_release(__vma_lockdep_map(vma), _RET_IP_) | |
| 110 | +#define __vma_lockdep_acquire_exclusive(vma) \ | |
| 111 | + lock_acquire_exclusive(__vma_lockdep_map(vma), 0, 0, NULL, _RET_IP_) | |
| 112 | +#define __vma_lockdep_release_exclusive(vma) \ | |
| 113 | + lock_release(__vma_lockdep_map(vma), _RET_IP_) | |
| 114 | +/* Only meaningful if CONFIG_LOCK_STAT is defined. */ | |
| 115 | +#define __vma_lockdep_stat_mark_acquired(vma) \ | |
| 116 | + lock_acquired(__vma_lockdep_map(vma), _RET_IP_) | |
| 117 | + | |
| 81 | 118 | static inline void mm_lock_seqcount_init(struct mm_struct *mm) |
| 82 | 119 | { |
| 83 | 120 | seqcount_init(&mm->mm_lock_seq); |
| @@ -115,36 +152,81 @@ static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt) · 31 unmodified lines | ||
| 115 | 152 | #ifdef CONFIG_DEBUG_LOCK_ALLOC |
| 116 | 153 | static struct lock_class_key lockdep_key; |
| 117 | 154 | |
| 118 | − lockdep_init_map(&vma->vmlock_dep_map, "vm_lock", &lockdep_key, 0); | |
| 155 | + lockdep_init_map(__vma_lockdep_map(vma), "vm_lock", &lockdep_key, 0); | |
| 119 | 156 | #endif |
| 120 | 157 | if (reset_refcnt) |
| 121 | 158 | refcount_set(&vma->vm_refcnt, 0); |
| 122 | 159 | vma->vm_lock_seq = UINT_MAX; |
| 123 | 160 | } |
| 124 | 161 | |
| 125 | −static inline bool is_vma_writer_only(int refcnt) | |
| 162 | +/* | |
| 163 | + * This function determines whether the input VMA reference count describes a | |
| 164 | + * VMA which has excluded all VMA read locks. | |
| 165 | + * | |
| 166 | + * In the case of a detached VMA, we may incorrectly indicate that readers are | |
| 167 | + * excluded when one remains, because in that scenario we target a refcount of | |
| 168 | + * VM_REFCNT_EXCLUDE_READERS_FLAG, rather than the attached target of | |
| 169 | + * VM_REFCNT_EXCLUDE_READERS_FLAG + 1. | |
| 170 | + * | |
| 171 | + * However, the race window for that is very small so it is unlikely. | |
| 172 | + * | |
| 173 | + * Returns: true if readers are excluded, false otherwise. | |
| 174 | + */ | |
| 175 | +static inline bool __vma_are_readers_excluded(int refcnt) | |
| 126 | 176 | { |
| 127 | 177 | /* |
| 128 | − * With a writer and no readers, refcnt is VMA_LOCK_OFFSET if the vma | |
| 129 | − * is detached and (VMA_LOCK_OFFSET + 1) if it is attached. Waiting on | |
| 130 | − * a detached vma happens only in vma_mark_detached() and is a rare | |
| 131 | − * case, therefore most of the time there will be no unnecessary wakeup. | |
| 178 | + * See the comment describing the vm_area_struct->vm_refcnt field for | |
| 179 | + * details of possible refcnt values. | |
| 132 | 180 | */ |
| 133 | − return (refcnt & VMA_LOCK_OFFSET) && refcnt <= VMA_LOCK_OFFSET + 1; | |
| 181 | + return (refcnt & VM_REFCNT_EXCLUDE_READERS_FLAG) && | |
| 182 | + refcnt <= VM_REFCNT_EXCLUDE_READERS_FLAG + 1; | |
| 183 | +} | |
| 184 | + | |
| 185 | +/* | |
| 186 | + * Actually decrement the VMA reference count. | |
| 187 | + * | |
| 188 | + * The function returns the reference count as it was immediately after the | |
| 189 | + * decrement took place. If it returns zero, the VMA is now detached. | |
| 190 | + */ | |
| 191 | +static inline __must_check unsigned int | |
| 192 | +__vma_refcount_put_return(struct vm_area_struct *vma) | |
| 193 | +{ | |
| 194 | + int oldcnt; | |
| 195 | + | |
| 196 | + if (__refcount_dec_and_test(&vma->vm_refcnt, &oldcnt)) | |
| 197 | + return 0; | |
| 198 | + | |
| 199 | + return oldcnt - 1; | |
| 134 | 200 | } |
| 135 | 201 | |
| 202 | +/** | |
| 203 | + * vma_refcount_put() - Drop reference count in VMA vm_refcnt field due to a | |
| 204 | + * read-lock being dropped. | |
| 205 | + * @vma: The VMA whose reference count we wish to decrement. | |
| 206 | + * | |
| 207 | + * If we were the last reader, wake up threads waiting to obtain an exclusive | |
| 208 | + * lock. | |
| 209 | + */ | |
| 136 | 210 | static inline void vma_refcount_put(struct vm_area_struct *vma) |
| 137 | 211 | { |
| 138 | − /* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt */ | |
| 212 | + /* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt. */ | |
| 139 | 213 | struct mm_struct *mm = vma->vm_mm; |
| 140 | − int oldcnt; | |
| 214 | + int newcnt; | |
| 141 | 215 | |
| 142 | − rwsem_release(&vma->vmlock_dep_map, _RET_IP_); | |
| 143 | − if (!__refcount_dec_and_test(&vma->vm_refcnt, &oldcnt)) { | |
| 216 | + __vma_lockdep_release_read(vma); | |
| 217 | + newcnt = __vma_refcount_put_return(vma); | |
| 144 | 218 | |
| 145 | − if (is_vma_writer_only(oldcnt - 1)) | |
| 146 | − rcuwait_wake_up(&mm->vma_writer_wait); | |
| 147 | − } | |
| 219 | + /* | |
| 220 | + * __vma_start_exclude_readers() may be sleeping waiting for readers to | |
| 221 | + * drop their reference count, so wake it up if we were the last reader | |
| 222 | + * blocking it from being acquired. | |
| 223 | + * | |
| 224 | + * We may be raced by other readers temporarily incrementing the | |
| 225 | + * reference count, though the race window is very small, this might | |
| 226 | + * cause spurious wakeups. | |
| 227 | + */ | |
| 228 | + if (newcnt && __vma_are_readers_excluded(newcnt)) | |
| 229 | + rcuwait_wake_up(&mm->vma_writer_wait); | |
| 148 | 230 | } |
| 149 | 231 | |
| 150 | 232 | /* |
| @@ -159,10 +241,10 @@ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int · 8 unmodified lines | ||
| 159 | 241 | |
| 160 | 242 | mmap_assert_locked(vma->vm_mm); |
| 161 | 243 | if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt, |
| 162 | − VMA_REF_LIMIT))) | |
| 244 | + VM_REFCNT_LIMIT))) | |
| 163 | 245 | return false; |
| 164 | 246 | |
| 165 | − rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_); | |
| 247 | + __vma_lockdep_acquire_read(vma); | |
| 166 | 248 | return true; |
| 167 | 249 | } |
| 168 | 250 | |
| @@ -182,21 +264,31 @@ static inline void vma_end_read(struct vm_area_struct *vma) · 13 unmodified lines | ||
| 182 | 264 | vma_refcount_put(vma); |
| 183 | 265 | } |
| 184 | 266 | |
| 185 | −/* WARNING! Can only be used if mmap_lock is expected to be write-locked */ | |
| 186 | −static inline bool __is_vma_write_locked(struct vm_area_struct *vma, unsigned int *mm_lock_seq) | |
| 267 | +static inline unsigned int __vma_raw_mm_seqnum(struct vm_area_struct *vma) | |
| 187 | 268 | { |
| 269 | + const struct mm_struct *mm = vma->vm_mm; | |
| 270 | + | |
| 271 | + /* We must hold an exclusive write lock for this access to be valid. */ | |
| 188 | 272 | mmap_assert_write_locked(vma->vm_mm); |
| 273 | + return mm->mm_lock_seq.sequence; | |
| 274 | +} | |
| 189 | 275 | |
| 276 | +/* | |
| 277 | + * Determine whether a VMA is write-locked. Must be invoked ONLY if the mmap | |
| 278 | + * write lock is held. | |
| 279 | + * | |
| 280 | + * Returns true if write-locked, otherwise false. | |
| 281 | + */ | |
| 282 | +static inline bool __is_vma_write_locked(struct vm_area_struct *vma) | |
| 283 | +{ | |
| 190 | 284 | /* |
| 191 | 285 | * current task is holding mmap_write_lock, both vma->vm_lock_seq and |
| 192 | 286 | * mm->mm_lock_seq can't be concurrently modified. |
| 193 | 287 | */ |
| 194 | − *mm_lock_seq = vma->vm_mm->mm_lock_seq.sequence; | |
| 195 | − return (vma->vm_lock_seq == *mm_lock_seq); | |
| 288 | + return vma->vm_lock_seq == __vma_raw_mm_seqnum(vma); | |
| 196 | 289 | } |
| 197 | 290 | |
| 198 | −int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq, | |
| 199 | − int state); | |
| 291 | +int __vma_start_write(struct vm_area_struct *vma, int state); | |
| 200 | 292 | |
| 201 | 293 | /* |
| 202 | 294 | * Begin writing to a VMA. |
| @@ -205,12 +297,10 @@ int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq, · 2 unmodified lines | ||
| 205 | 297 | */ |
| 206 | 298 | static inline void vma_start_write(struct vm_area_struct *vma) |
| 207 | 299 | { |
| 208 | − unsigned int mm_lock_seq; | |
| 209 | − | |
| 210 | − if (__is_vma_write_locked(vma, &mm_lock_seq)) | |
| 300 | + if (__is_vma_write_locked(vma)) | |
| 211 | 301 | return; |
| 212 | 302 | |
| 213 | − __vma_start_write(vma, mm_lock_seq, TASK_UNINTERRUPTIBLE); | |
| 303 | + __vma_start_write(vma, TASK_UNINTERRUPTIBLE); | |
| 214 | 304 | } |
| 215 | 305 | |
| 216 | 306 | /** |
| @@ -229,26 +319,110 @@ static inline void vma_start_write(struct vm_area_struct *vma) · 12 unmodified lines | ||
| 229 | 319 | static inline __must_check |
| 230 | 320 | int vma_start_write_killable(struct vm_area_struct *vma) |
| 231 | 321 | { |
| 232 | − unsigned int mm_lock_seq; | |
| 233 | − | |
| 234 | − if (__is_vma_write_locked(vma, &mm_lock_seq)) | |
| 322 | + if (__is_vma_write_locked(vma)) | |
| 235 | 323 | return 0; |
| 236 | − return __vma_start_write(vma, mm_lock_seq, TASK_KILLABLE); | |
| 324 | + | |
| 325 | + return __vma_start_write(vma, TASK_KILLABLE); | |
| 237 | 326 | } |
| 238 | 327 | |
| 328 | +/** | |
| 329 | + * vma_assert_write_locked() - assert that @vma holds a VMA write lock. | |
| 330 | + * @vma: The VMA to assert. | |
| 331 | + */ | |
| 239 | 332 | static inline void vma_assert_write_locked(struct vm_area_struct *vma) |
| 240 | 333 | { |
| 241 | − unsigned int mm_lock_seq; | |
| 242 | − | |
| 243 | − VM_BUG_ON_VMA(!__is_vma_write_locked(vma, &mm_lock_seq), vma); | |
| 334 | + VM_WARN_ON_ONCE_VMA(!__is_vma_write_locked(vma), vma); | |
| 244 | 335 | } |
| 245 | 336 | |
| 337 | +/** | |
| 338 | + * vma_assert_locked() - assert that @vma holds either a VMA read or a VMA write | |
| 339 | + * lock and is not detached. | |
| 340 | + * @vma: The VMA to assert. | |
| 341 | + */ | |
| 246 | 342 | static inline void vma_assert_locked(struct vm_area_struct *vma) |
| 247 | 343 | { |
| 248 | − unsigned int mm_lock_seq; | |
| 344 | + unsigned int refcnt; | |
| 345 | + | |
| 346 | + if (IS_ENABLED(CONFIG_LOCKDEP)) { | |
| 347 | + if (!lock_is_held(__vma_lockdep_map(vma))) | |
| 348 | + vma_assert_write_locked(vma); | |
| 349 | + return; | |
| 350 | + } | |
| 351 | + | |
| 352 | + /* | |
| 353 | + * See the comment describing the vm_area_struct->vm_refcnt field for | |
| 354 | + * details of possible refcnt values. | |
| 355 | + */ | |
| 356 | + refcnt = refcount_read(&vma->vm_refcnt); | |
| 357 | + | |
| 358 | + /* | |
| 359 | + * In this case we're either read-locked, write-locked with temporary | |
| 360 | + * readers, or in the midst of excluding readers, all of which means | |
| 361 | + * we're locked. | |
| 362 | + */ | |
| 363 | + if (refcnt > 1) | |
| 364 | + return; | |
| 365 | + | |
| 366 | + /* It is a bug for the VMA to be detached here. */ | |
| 367 | + VM_WARN_ON_ONCE_VMA(!refcnt, vma); | |
| 368 | + | |
| 369 | + /* | |
| 370 | + * OK, the VMA has a reference count of 1 which means it is either | |
| 371 | + * unlocked and attached or write-locked, so assert that it is | |
| 372 | + * write-locked. | |
| 373 | + */ | |
| 374 | + vma_assert_write_locked(vma); | |
| 375 | +} | |
| 376 | + | |
| 377 | +/** | |
| 378 | + * vma_assert_stabilised() - assert that this VMA cannot be changed from | |
| 379 | + * underneath us either by having a VMA or mmap lock held. | |
| 380 | + * @vma: The VMA whose stability we wish to assess. | |
| 381 | + * | |
| 382 | + * If lockdep is enabled we can precisely ensure stability via either an mmap | |
| 383 | + * lock owned by us or a specific VMA lock. | |
| 384 | + * | |
| 385 | + * With lockdep disabled we may sometimes race with other threads acquiring the | |
| 386 | + * mmap read lock simultaneous with our VMA read lock. | |
| 387 | + */ | |
| 388 | +static inline void vma_assert_stabilised(struct vm_area_struct *vma) | |
| 389 | +{ | |
| 390 | + /* | |
| 391 | + * If another thread owns an mmap lock, it may go away at any time, and | |
| 392 | + * thus is no guarantee of stability. | |
| 393 | + * | |
| 394 | + * If lockdep is enabled we can accurately determine if an mmap lock is | |
| 395 | + * held and owned by us. Otherwise we must approximate. | |
| 396 | + * | |
| 397 | + * It doesn't necessarily mean we are not stabilised however, as we may | |
| 398 | + * hold a VMA read lock (not a write lock as this would require an owned | |
| 399 | + * mmap lock). | |
| 400 | + * | |
| 401 | + * If (assuming lockdep is not enabled) we were to assert a VMA read | |
| 402 | + * lock first we may also run into issues, as other threads can hold VMA | |
| 403 | + * read locks simlutaneous to us. | |
| 404 | + * | |
| 405 | + * Therefore if lockdep is not enabled we risk a false negative (i.e. no | |
| 406 | + * assert fired). If accurate checking is required, enable lockdep. | |
| 407 | + */ | |
| 408 | + if (IS_ENABLED(CONFIG_LOCKDEP)) { | |
| 409 | + if (lockdep_is_held(&vma->vm_mm->mmap_lock)) | |
| 410 | + return; | |
| 411 | + } else { | |
| 412 | + if (rwsem_is_locked(&vma->vm_mm->mmap_lock)) | |
| 413 | + return; | |
| 414 | + } | |
| 249 | 415 | |
| 250 | − VM_BUG_ON_VMA(refcount_read(&vma->vm_refcnt) <= 1 && | |
| 251 | − !__is_vma_write_locked(vma, &mm_lock_seq), vma); | |
| 416 | + /* | |
| 417 | + * We're not stabilised by the mmap lock, so assert that we're | |
| 418 | + * stabilised by a VMA lock. | |
| 419 | + */ | |
| 420 | + vma_assert_locked(vma); | |
| 421 | +} | |
| 422 | + | |
| 423 | +static inline bool vma_is_attached(struct vm_area_struct *vma) | |
| 424 | +{ | |
| 425 | + return refcount_read(&vma->vm_refcnt); | |
| 252 | 426 | } |
| 253 | 427 | |
| 254 | 428 | /* |
| @@ -258,12 +432,12 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) · 3 unmodified lines | ||
| 258 | 432 | */ |
| 259 | 433 | static inline void vma_assert_attached(struct vm_area_struct *vma) |
| 260 | 434 | { |
| 261 | − WARN_ON_ONCE(!refcount_read(&vma->vm_refcnt)); | |
| 435 | + WARN_ON_ONCE(!vma_is_attached(vma)); | |
| 262 | 436 | } |
| 263 | 437 | |
| 264 | 438 | static inline void vma_assert_detached(struct vm_area_struct *vma) |
| 265 | 439 | { |
| 266 | − WARN_ON_ONCE(refcount_read(&vma->vm_refcnt)); | |
| 440 | + WARN_ON_ONCE(vma_is_attached(vma)); | |
| 267 | 441 | } |
| 268 | 442 | |
| 269 | 443 | static inline void vma_mark_attached(struct vm_area_struct *vma) |
| @@ -273,7 +447,28 @@ static inline void vma_mark_attached(struct vm_area_struct *vma) · 3 unmodified lines | ||
| 273 | 447 | refcount_set_release(&vma->vm_refcnt, 1); |
| 274 | 448 | } |
| 275 | 449 | |
| 276 | −void vma_mark_detached(struct vm_area_struct *vma); | |
| 450 | +void __vma_exclude_readers_for_detach(struct vm_area_struct *vma); | |
| 451 | + | |
| 452 | +static inline void vma_mark_detached(struct vm_area_struct *vma) | |
| 453 | +{ | |
| 454 | + vma_assert_write_locked(vma); | |
| 455 | + vma_assert_attached(vma); | |
| 456 | + | |
| 457 | + /* | |
| 458 | + * The VMA still being attached (refcnt > 0) - is unlikely, because the | |
| 459 | + * vma has been already write-locked and readers can increment vm_refcnt | |
| 460 | + * only temporarily before they check vm_lock_seq, realize the vma is | |
| 461 | + * locked and drop back the vm_refcnt. That is a narrow window for | |
| 462 | + * observing a raised vm_refcnt. | |
| 463 | + * | |
| 464 | + * See the comment describing the vm_area_struct->vm_refcnt field for | |
| 465 | + * details of possible refcnt values. | |
| 466 | + */ | |
| 467 | + if (likely(!__vma_refcount_put_return(vma))) | |
| 468 | + return; | |
| 469 | + | |
| 470 | + __vma_exclude_readers_for_detach(vma); | |
| 471 | +} | |
| 277 | 472 | |
| 278 | 473 | struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, |
| 279 | 474 | unsigned long address); |
| @@ -327,6 +522,12 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) · 47 unmodified lines | ||
| 327 | 522 | mmap_assert_locked(vma->vm_mm); |
| 328 | 523 | } |
| 329 | 524 | |
| 525 | +static inline void vma_assert_stabilised(struct vm_area_struct *vma) | |
| 526 | +{ | |
| 527 | + /* If no VMA locks, then either mmap lock suffices to stabilise. */ | |
| 528 | + mmap_assert_locked(vma->vm_mm); | |
| 529 | +} | |
| 530 | + | |
| 330 | 531 | #endif /* CONFIG_PER_VMA_LOCK */ |
| 331 | 532 | |
| 332 | 533 | static inline void mmap_write_lock(struct mm_struct *mm) |
| @@ -47,6 +47,15 @@ void vma_iter_dump_tree(const struct vma_iterator *vmi); · 46 unmodified lines | ||
| 47 | 47 | BUG(); \ |
| 48 | 48 | } \ |
| 49 | 49 | } while (0) |
| 50 | +#define VM_WARN_ON_PAGE(cond, page) ({ \ | |
| 51 | + int __ret_warn = !!(cond); \ | |
| 52 | + \ | |
| 53 | + if (unlikely(__ret_warn)) { \ | |
| 54 | + dump_page(page, "VM_WARN_ON_PAGE(" __stringify(cond)")");\ | |
| 55 | + WARN_ON(1); \ | |
| 56 | + } \ | |
| 57 | + unlikely(__ret_warn); \ | |
| 58 | +}) | |
| 50 | 59 | #define VM_WARN_ON_ONCE_PAGE(cond, page) ({ \ |
| 51 | 60 | static bool __section(".data..once") __warned; \ |
| 52 | 61 | int __ret_warn_once = !!(cond); \ |
| @@ -122,6 +131,7 @@ void vma_iter_dump_tree(const struct vma_iterator *vmi); · 69 unmodified lines | ||
| 122 | 131 | #define VM_BUG_ON_MM(cond, mm) VM_BUG_ON(cond) |
| 123 | 132 | #define VM_WARN_ON(cond) BUILD_BUG_ON_INVALID(cond) |
| 124 | 133 | #define VM_WARN_ON_ONCE(cond) BUILD_BUG_ON_INVALID(cond) |
| 134 | +#define VM_WARN_ON_PAGE(cond, page) BUILD_BUG_ON_INVALID(cond) | |
| 125 | 135 | #define VM_WARN_ON_ONCE_PAGE(cond, page) BUILD_BUG_ON_INVALID(cond) |
| 126 | 136 | #define VM_WARN_ON_FOLIO(cond, folio) BUILD_BUG_ON_INVALID(cond) |
| 127 | 137 | #define VM_WARN_ON_ONCE_FOLIO(cond, folio) BUILD_BUG_ON_INVALID(cond) |
| @@ -1534,14 +1534,27 @@ static inline unsigned long pgdat_end_pfn(pg_data_t *pgdat) · 1533 unmodified lines | ||
| 1534 | 1534 | #include <linux/memory_hotplug.h> |
| 1535 | 1535 | |
| 1536 | 1536 | void build_all_zonelists(pg_data_t *pgdat); |
| 1537 | −void wakeup_kswapd(struct zone *zone, gfp_t gfp_mask, int order, | |
| 1538 | − enum zone_type highest_zoneidx); | |
| 1539 | 1537 | bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, |
| 1540 | 1538 | int highest_zoneidx, unsigned int alloc_flags, |
| 1541 | 1539 | long free_pages); |
| 1542 | 1540 | bool zone_watermark_ok(struct zone *z, unsigned int order, |
| 1543 | 1541 | unsigned long mark, int highest_zoneidx, |
| 1544 | 1542 | unsigned int alloc_flags); |
| 1543 | + | |
| 1544 | +enum kswapd_clear_hopeless_reason { | |
| 1545 | + KSWAPD_CLEAR_HOPELESS_OTHER = 0, | |
| 1546 | + KSWAPD_CLEAR_HOPELESS_KSWAPD, | |
| 1547 | + KSWAPD_CLEAR_HOPELESS_DIRECT, | |
| 1548 | + KSWAPD_CLEAR_HOPELESS_PCP, | |
| 1549 | +}; | |
| 1550 | + | |
| 1551 | +void wakeup_kswapd(struct zone *zone, gfp_t gfp_mask, int order, | |
| 1552 | + enum zone_type highest_zoneidx); | |
| 1553 | +void kswapd_try_clear_hopeless(struct pglist_data *pgdat, | |
| 1554 | + unsigned int order, int highest_zoneidx); | |
| 1555 | +void kswapd_clear_hopeless(pg_data_t *pgdat, enum kswapd_clear_hopeless_reason reason); | |
| 1556 | +bool kswapd_test_hopeless(pg_data_t *pgdat); | |
| 1557 | + | |
| 1545 | 1558 | /* |
| 1546 | 1559 | * Memory initialization context, use to differentiate memory added by |
| 1547 | 1560 | * the platform statically or via memory hotplug interface. |
| @@ -2286,9 +2299,7 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) · 738 unmodified lines | ||
| 2286 | 2299 | #define pfn_to_nid(pfn) (0) |
| 2287 | 2300 | #endif |
| 2288 | 2301 | |
| 2289 | −void sparse_init(void); | |
| 2290 | 2302 | #else |
| 2291 | −#define sparse_init() do {} while (0) | |
| 2292 | 2303 | #define sparse_index_init(_sec, _nid) do {} while (0) |
| 2293 | 2304 | #define sparse_vmemmap_init_nid_early(_nid) do {} while (0) |
| 2294 | 2305 | #define sparse_vmemmap_init_nid_late(_nid) do {} while (0) |
| @@ -157,10 +157,10 @@ static __always_inline bool __node_test_and_set(int node, nodemask_t *addr) · 156 unmodified lines | ||
| 157 | 157 | |
| 158 | 158 | #define nodes_and(dst, src1, src2) \ |
| 159 | 159 | __nodes_and(&(dst), &(src1), &(src2), MAX_NUMNODES) |
| 160 | −static __always_inline void __nodes_and(nodemask_t *dstp, const nodemask_t *src1p, | |
| 160 | +static __always_inline bool __nodes_and(nodemask_t *dstp, const nodemask_t *src1p, | |
| 161 | 161 | const nodemask_t *src2p, unsigned int nbits) |
| 162 | 162 | { |
| 163 | − bitmap_and(dstp->bits, src1p->bits, src2p->bits, nbits); | |
| 163 | + return bitmap_and(dstp->bits, src1p->bits, src2p->bits, nbits); | |
| 164 | 164 | } |
| 165 | 165 | |
| 166 | 166 | #define nodes_or(dst, src1, src2) \ |
| @@ -181,10 +181,10 @@ static __always_inline void __nodes_xor(nodemask_t *dstp, const nodemask_t *src1 · 14 unmodified lines | ||
| 181 | 181 | |
| 182 | 182 | #define nodes_andnot(dst, src1, src2) \ |
| 183 | 183 | __nodes_andnot(&(dst), &(src1), &(src2), MAX_NUMNODES) |
| 184 | −static __always_inline void __nodes_andnot(nodemask_t *dstp, const nodemask_t *src1p, | |
| 184 | +static __always_inline bool __nodes_andnot(nodemask_t *dstp, const nodemask_t *src1p, | |
| 185 | 185 | const nodemask_t *src2p, unsigned int nbits) |
| 186 | 186 | { |
| 187 | − bitmap_andnot(dstp->bits, src1p->bits, src2p->bits, nbits); | |
| 187 | + return bitmap_andnot(dstp->bits, src1p->bits, src2p->bits, nbits); | |
| 188 | 188 | } |
| 189 | 189 | |
| 190 | 190 | #define nodes_copy(dst, src) __nodes_copy(&(dst), &(src), MAX_NUMNODES) |
| @@ -67,4 +67,6 @@ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn); · 66 unmodified lines | ||
| 67 | 67 | |
| 68 | 68 | int test_pages_isolated(unsigned long start_pfn, unsigned long end_pfn, |
| 69 | 69 | enum pb_isolate_mode mode); |
| 70 | +bool page_is_unmovable(struct zone *zone, struct page *page, | |
| 71 | + enum pb_isolate_mode mode, unsigned long *step); | |
| 70 | 72 | #endif |
| @@ -14,15 +14,18 @@ extern struct static_key_true page_table_check_disabled; · 13 unmodified lines | ||
| 14 | 14 | extern struct page_ext_operations page_table_check_ops; |
| 15 | 15 | |
| 16 | 16 | void __page_table_check_zero(struct page *page, unsigned int order); |
| 17 | −void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte); | |
| 18 | −void __page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd); | |
| 19 | −void __page_table_check_pud_clear(struct mm_struct *mm, pud_t pud); | |
| 20 | −void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte, | |
| 21 | − unsigned int nr); | |
| 22 | −void __page_table_check_pmds_set(struct mm_struct *mm, pmd_t *pmdp, pmd_t pmd, | |
| 23 | − unsigned int nr); | |
| 24 | −void __page_table_check_puds_set(struct mm_struct *mm, pud_t *pudp, pud_t pud, | |
| 25 | − unsigned int nr); | |
| 17 | +void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr, | |
| 18 | + pte_t pte); | |
| 19 | +void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr, | |
| 20 | + pmd_t pmd); | |
| 21 | +void __page_table_check_pud_clear(struct mm_struct *mm, unsigned long addr, | |
| 22 | + pud_t pud); | |
| 23 | +void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr, | |
| 24 | + pte_t *ptep, pte_t pte, unsigned int nr); | |
| 25 | +void __page_table_check_pmds_set(struct mm_struct *mm, unsigned long addr, | |
| 26 | + pmd_t *pmdp, pmd_t pmd, unsigned int nr); | |
| 27 | +void __page_table_check_puds_set(struct mm_struct *mm, unsigned long addr, | |
| 28 | + pud_t *pudp, pud_t pud, unsigned int nr); | |
| 26 | 29 | void __page_table_check_pte_clear_range(struct mm_struct *mm, |
| 27 | 30 | unsigned long addr, |
| 28 | 31 | pmd_t pmd); |
| @@ -43,55 +46,59 @@ static inline void page_table_check_free(struct page *page, unsigned int order) · 14 unmodified lines | ||
| 43 | 46 | __page_table_check_zero(page, order); |
| 44 | 47 | } |
| 45 | 48 | |
| 46 | −static inline void page_table_check_pte_clear(struct mm_struct *mm, pte_t pte) | |
| 49 | +static inline void page_table_check_pte_clear(struct mm_struct *mm, | |
| 50 | + unsigned long addr, pte_t pte) | |
| 47 | 51 | { |
| 48 | 52 | if (static_branch_likely(&page_table_check_disabled)) |
| 49 | 53 | return; |
| 50 | 54 | |
| 51 | − __page_table_check_pte_clear(mm, pte); | |
| 55 | + __page_table_check_pte_clear(mm, addr, pte); | |
| 52 | 56 | } |
| 53 | 57 | |
| 54 | −static inline void page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd) | |
| 58 | +static inline void page_table_check_pmd_clear(struct mm_struct *mm, | |
| 59 | + unsigned long addr, pmd_t pmd) | |
| 55 | 60 | { |
| 56 | 61 | if (static_branch_likely(&page_table_check_disabled)) |
| 57 | 62 | return; |
| 58 | 63 | |
| 59 | − __page_table_check_pmd_clear(mm, pmd); | |
| 64 | + __page_table_check_pmd_clear(mm, addr, pmd); | |
| 60 | 65 | } |
| 61 | 66 | |
| 62 | −static inline void page_table_check_pud_clear(struct mm_struct *mm, pud_t pud) | |
| 67 | +static inline void page_table_check_pud_clear(struct mm_struct *mm, | |
| 68 | + unsigned long addr, pud_t pud) | |
| 63 | 69 | { |
| 64 | 70 | if (static_branch_likely(&page_table_check_disabled)) |
| 65 | 71 | return; |
| 66 | 72 | |
| 67 | − __page_table_check_pud_clear(mm, pud); | |
| 73 | + __page_table_check_pud_clear(mm, addr, pud); | |
| 68 | 74 | } |
| 69 | 75 | |
| 70 | 76 | static inline void page_table_check_ptes_set(struct mm_struct *mm, |
| 71 | − pte_t *ptep, pte_t pte, unsigned int nr) | |
| 77 | + unsigned long addr, pte_t *ptep, | |
| 78 | + pte_t pte, unsigned int nr) | |
| 72 | 79 | { |
| 73 | 80 | if (static_branch_likely(&page_table_check_disabled)) |
| 74 | 81 | return; |
| 75 | 82 | |
| 76 | − __page_table_check_ptes_set(mm, ptep, pte, nr); | |
| 83 | + __page_table_check_ptes_set(mm, addr, ptep, pte, nr); | |
| 77 | 84 | } |
| 78 | 85 | |
| 79 | 86 | static inline void page_table_check_pmds_set(struct mm_struct *mm, |
| 80 | − pmd_t *pmdp, pmd_t pmd, unsigned int nr) | |
| 87 | + unsigned long addr, pmd_t *pmdp, pmd_t pmd, unsigned int nr) | |
| 81 | 88 | { |
| 82 | 89 | if (static_branch_likely(&page_table_check_disabled)) |
| 83 | 90 | return; |
| 84 | 91 | |
| 85 | − __page_table_check_pmds_set(mm, pmdp, pmd, nr); | |
| 92 | + __page_table_check_pmds_set(mm, addr, pmdp, pmd, nr); | |
| 86 | 93 | } |
| 87 | 94 | |
| 88 | 95 | static inline void page_table_check_puds_set(struct mm_struct *mm, |
| 89 | − pud_t *pudp, pud_t pud, unsigned int nr) | |
| 96 | + unsigned long addr, pud_t *pudp, pud_t pud, unsigned int nr) | |
| 90 | 97 | { |
| 91 | 98 | if (static_branch_likely(&page_table_check_disabled)) |
| 92 | 99 | return; |
| 93 | 100 | |
| 94 | − __page_table_check_puds_set(mm, pudp, pud, nr); | |
| 101 | + __page_table_check_puds_set(mm, addr, pudp, pud, nr); | |
| 95 | 102 | } |
| 96 | 103 | |
| 97 | 104 | static inline void page_table_check_pte_clear_range(struct mm_struct *mm, |
| @@ -114,30 +121,34 @@ static inline void page_table_check_free(struct page *page, unsigned int order) · 16 unmodified lines | ||
| 114 | 121 | { |
| 115 | 122 | } |
| 116 | 123 | |
| 117 | −static inline void page_table_check_pte_clear(struct mm_struct *mm, pte_t pte) | |
| 124 | +static inline void page_table_check_pte_clear(struct mm_struct *mm, | |
| 125 | + unsigned long addr, pte_t pte) | |
| 118 | 126 | { |
| 119 | 127 | } |
| 120 | 128 | |
| 121 | −static inline void page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd) | |
| 129 | +static inline void page_table_check_pmd_clear(struct mm_struct *mm, | |
| 130 | + unsigned long addr, pmd_t pmd) | |
| 122 | 131 | { |
| 123 | 132 | } |
| 124 | 133 | |
| 125 | −static inline void page_table_check_pud_clear(struct mm_struct *mm, pud_t pud) | |
| 134 | +static inline void page_table_check_pud_clear(struct mm_struct *mm, | |
| 135 | + unsigned long addr, pud_t pud) | |
| 126 | 136 | { |
| 127 | 137 | } |
| 128 | 138 | |
| 129 | 139 | static inline void page_table_check_ptes_set(struct mm_struct *mm, |
| 130 | − pte_t *ptep, pte_t pte, unsigned int nr) | |
| 140 | + unsigned long addr, pte_t *ptep, | |
| 141 | + pte_t pte, unsigned int nr) | |
| 131 | 142 | { |
| 132 | 143 | } |
| 133 | 144 | |
| 134 | 145 | static inline void page_table_check_pmds_set(struct mm_struct *mm, |
| 135 | − pmd_t *pmdp, pmd_t pmd, unsigned int nr) | |
| 146 | + unsigned long addr, pmd_t *pmdp, pmd_t pmd, unsigned int nr) | |
| 136 | 147 | { |
| 137 | 148 | } |
| 138 | 149 | |
| 139 | 150 | static inline void page_table_check_puds_set(struct mm_struct *mm, |
| 140 | − pud_t *pudp, pud_t pud, unsigned int nr) | |
| 151 | + unsigned long addr, pud_t *pudp, pud_t pud, unsigned int nr) | |
| 141 | 152 | { |
| 142 | 153 | } |
| 143 | 154 | |
| @@ -149,7 +160,7 @@ static inline void page_table_check_pte_clear_range(struct mm_struct *mm, · 5 unmodified lines | ||
| 149 | 160 | |
| 150 | 161 | #endif /* CONFIG_PAGE_TABLE_CHECK */ |
| 151 | 162 | |
| 152 | −#define page_table_check_pmd_set(mm, pmdp, pmd) page_table_check_pmds_set(mm, pmdp, pmd, 1) | |
| 153 | −#define page_table_check_pud_set(mm, pudp, pud) page_table_check_puds_set(mm, pudp, pud, 1) | |
| 163 | +#define page_table_check_pmd_set(mm, addr, pmdp, pmd) page_table_check_pmds_set(mm, addr, pmdp, pmd, 1) | |
| 164 | +#define page_table_check_pud_set(mm, addr, pudp, pud) page_table_check_puds_set(mm, addr, pudp, pud, 1) | |
| 154 | 165 | |
| 155 | 166 | #endif /* __LINUX_PAGE_TABLE_CHECK_H */ |
| @@ -225,16 +225,156 @@ static inline int pmd_dirty(pmd_t pmd) · 224 unmodified lines | ||
| 225 | 225 | * up to date. |
| 226 | 226 | * |
| 227 | 227 | * In the general case, no lock is guaranteed to be held between entry and exit |
| 228 | − * of the lazy mode. So the implementation must assume preemption may be enabled | |
| 229 | − * and cpu migration is possible; it must take steps to be robust against this. | |
| 230 | − * (In practice, for user PTE updates, the appropriate page table lock(s) are | |
| 231 | − * held, but for kernel PTE updates, no lock is held). Nesting is not permitted | |
| 232 | − * and the mode cannot be used in interrupt context. | |
| 228 | + * of the lazy mode. (In practice, for user PTE updates, the appropriate page | |
| 229 | + * table lock(s) are held, but for kernel PTE updates, no lock is held). | |
| 230 | + * The implementation must therefore assume preemption may be enabled upon | |
| 231 | + * entry to the mode and cpu migration is possible; it must take steps to be | |
| 232 | + * robust against this. An implementation may handle this by disabling | |
| 233 | + * preemption, as a consequence generic code may not sleep while the lazy MMU | |
| 234 | + * mode is active. | |
| 235 | + * | |
| 236 | + * The mode is disabled in interrupt context and calls to the lazy_mmu API have | |
| 237 | + * no effect. | |
| 238 | + * | |
| 239 | + * The lazy MMU mode is enabled for a given block of code using: | |
| 240 | + * | |
| 241 | + * lazy_mmu_mode_enable(); | |
| 242 | + * <code> | |
| 243 | + * lazy_mmu_mode_disable(); | |
| 244 | + * | |
| 245 | + * Nesting is permitted: <code> may itself use an enable()/disable() pair. | |
| 246 | + * A nested call to enable() has no functional effect; however disable() causes | |
| 247 | + * any batched architectural state to be flushed regardless of nesting. After a | |
| 248 | + * call to disable(), the caller can therefore rely on all previous page table | |
| 249 | + * modifications to have taken effect, but the lazy MMU mode may still be | |
| 250 | + * enabled. | |
| 251 | + * | |
| 252 | + * In certain cases, it may be desirable to temporarily pause the lazy MMU mode. | |
| 253 | + * This can be done using: | |
| 254 | + * | |
| 255 | + * lazy_mmu_mode_pause(); | |
| 256 | + * <code> | |
| 257 | + * lazy_mmu_mode_resume(); | |
| 258 | + * | |
| 259 | + * pause() ensures that the mode is exited regardless of the nesting level; | |
| 260 | + * resume() re-enters the mode at the same nesting level. Any call to the | |
| 261 | + * lazy_mmu_mode_* API between those two calls has no effect. In particular, | |
| 262 | + * this means that pause()/resume() pairs may nest. | |
| 263 | + * | |
| 264 | + * is_lazy_mmu_mode_active() can be used to check whether the lazy MMU mode is | |
| 265 | + * currently enabled. | |
| 233 | 266 | */ |
| 234 | −#ifndef __HAVE_ARCH_ENTER_LAZY_MMU_MODE | |
| 235 | −static inline void arch_enter_lazy_mmu_mode(void) {} | |
| 236 | −static inline void arch_leave_lazy_mmu_mode(void) {} | |
| 237 | −static inline void arch_flush_lazy_mmu_mode(void) {} | |
| 267 | +#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE | |
| 268 | +/** | |
| 269 | + * lazy_mmu_mode_enable() - Enable the lazy MMU mode. | |
| 270 | + * | |
| 271 | + * Enters a new lazy MMU mode section; if the mode was not already enabled, | |
| 272 | + * enables it and calls arch_enter_lazy_mmu_mode(). | |
| 273 | + * | |
| 274 | + * Must be paired with a call to lazy_mmu_mode_disable(). | |
| 275 | + * | |
| 276 | + * Has no effect if called: | |
| 277 | + * - While paused - see lazy_mmu_mode_pause() | |
| 278 | + * - In interrupt context | |
| 279 | + */ | |
| 280 | +static inline void lazy_mmu_mode_enable(void) | |
| 281 | +{ | |
| 282 | + struct lazy_mmu_state *state = ¤t->lazy_mmu_state; | |
| 283 | + | |
| 284 | + if (in_interrupt() || state->pause_count > 0) | |
| 285 | + return; | |
| 286 | + | |
| 287 | + VM_WARN_ON_ONCE(state->enable_count == U8_MAX); | |
| 288 | + | |
| 289 | + if (state->enable_count++ == 0) | |
| 290 | + arch_enter_lazy_mmu_mode(); | |
| 291 | +} | |
| 292 | + | |
| 293 | +/** | |
| 294 | + * lazy_mmu_mode_disable() - Disable the lazy MMU mode. | |
| 295 | + * | |
| 296 | + * Exits the current lazy MMU mode section. If it is the outermost section, | |
| 297 | + * disables the mode and calls arch_leave_lazy_mmu_mode(). Otherwise (nested | |
| 298 | + * section), calls arch_flush_lazy_mmu_mode(). | |
| 299 | + * | |
| 300 | + * Must match a call to lazy_mmu_mode_enable(). | |
| 301 | + * | |
| 302 | + * Has no effect if called: | |
| 303 | + * - While paused - see lazy_mmu_mode_pause() | |
| 304 | + * - In interrupt context | |
| 305 | + */ | |
| 306 | +static inline void lazy_mmu_mode_disable(void) | |
| 307 | +{ | |
| 308 | + struct lazy_mmu_state *state = ¤t->lazy_mmu_state; | |
| 309 | + | |
| 310 | + if (in_interrupt() || state->pause_count > 0) | |
| 311 | + return; | |
| 312 | + | |
| 313 | + VM_WARN_ON_ONCE(state->enable_count == 0); | |
| 314 | + | |
| 315 | + if (--state->enable_count == 0) | |
| 316 | + arch_leave_lazy_mmu_mode(); | |
| 317 | + else /* Exiting a nested section */ | |
| 318 | + arch_flush_lazy_mmu_mode(); | |
| 319 | + | |
| 320 | +} | |
| 321 | + | |
| 322 | +/** | |
| 323 | + * lazy_mmu_mode_pause() - Pause the lazy MMU mode. | |
| 324 | + * | |
| 325 | + * Pauses the lazy MMU mode; if it is currently active, disables it and calls | |
| 326 | + * arch_leave_lazy_mmu_mode(). | |
| 327 | + * | |
| 328 | + * Must be paired with a call to lazy_mmu_mode_resume(). Calls to the | |
| 329 | + * lazy_mmu_mode_* API have no effect until the matching resume() call. | |
| 330 | + * | |
| 331 | + * Has no effect if called: | |
| 332 | + * - While paused (inside another pause()/resume() pair) | |
| 333 | + * - In interrupt context | |
| 334 | + */ | |
| 335 | +static inline void lazy_mmu_mode_pause(void) | |
| 336 | +{ | |
| 337 | + struct lazy_mmu_state *state = ¤t->lazy_mmu_state; | |
| 338 | + | |
| 339 | + if (in_interrupt()) | |
| 340 | + return; | |
| 341 | + | |
| 342 | + VM_WARN_ON_ONCE(state->pause_count == U8_MAX); | |
| 343 | + | |
| 344 | + if (state->pause_count++ == 0 && state->enable_count > 0) | |
| 345 | + arch_leave_lazy_mmu_mode(); | |
| 346 | +} | |
| 347 | + | |
| 348 | +/** | |
| 349 | + * lazy_mmu_mode_resume() - Resume the lazy MMU mode. | |
| 350 | + * | |
| 351 | + * Resumes the lazy MMU mode; if it was active at the point where the matching | |
| 352 | + * call to lazy_mmu_mode_pause() was made, re-enables it and calls | |
| 353 | + * arch_enter_lazy_mmu_mode(). | |
| 354 | + * | |
| 355 | + * Must match a call to lazy_mmu_mode_pause(). | |
| 356 | + * | |
| 357 | + * Has no effect if called: | |
| 358 | + * - While paused (inside another pause()/resume() pair) | |
| 359 | + * - In interrupt context | |
| 360 | + */ | |
| 361 | +static inline void lazy_mmu_mode_resume(void) | |
| 362 | +{ | |
| 363 | + struct lazy_mmu_state *state = ¤t->lazy_mmu_state; | |
| 364 | + | |
| 365 | + if (in_interrupt()) | |
| 366 | + return; | |
| 367 | + | |
| 368 | + VM_WARN_ON_ONCE(state->pause_count == 0); | |
| 369 | + | |
| 370 | + if (--state->pause_count == 0 && state->enable_count > 0) | |
| 371 | + arch_enter_lazy_mmu_mode(); | |
| 372 | +} | |
| 373 | +#else | |
| 374 | +static inline void lazy_mmu_mode_enable(void) {} | |
| 375 | +static inline void lazy_mmu_mode_disable(void) {} | |
| 376 | +static inline void lazy_mmu_mode_pause(void) {} | |
| 377 | +static inline void lazy_mmu_mode_resume(void) {} | |
| 238 | 378 | #endif |
| 239 | 379 | |
| 240 | 380 | #ifndef pte_batch_hint |
| @@ -289,7 +429,7 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) · 48 unmodified lines | ||
| 289 | 429 | static inline void set_ptes(struct mm_struct *mm, unsigned long addr, |
| 290 | 430 | pte_t *ptep, pte_t pte, unsigned int nr) |
| 291 | 431 | { |
| 292 | − page_table_check_ptes_set(mm, ptep, pte, nr); | |
| 432 | + page_table_check_ptes_set(mm, addr, ptep, pte, nr); | |
| 293 | 433 | |
| 294 | 434 | for (;;) { |
| 295 | 435 | set_pte(ptep, pte); |
| @@ -494,7 +634,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · 198 unmodified lines | ||
| 494 | 634 | { |
| 495 | 635 | pte_t pte = ptep_get(ptep); |
| 496 | 636 | pte_clear(mm, address, ptep); |
| 497 | − page_table_check_pte_clear(mm, pte); | |
| 637 | + page_table_check_pte_clear(mm, address, pte); | |
| 498 | 638 | return pte; |
| 499 | 639 | } |
| 500 | 640 | #endif |
| @@ -553,7 +693,7 @@ static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, · 52 unmodified lines | ||
| 553 | 693 | * No need for ptep_get_and_clear(): page table check doesn't care about |
| 554 | 694 | * any bits that could have been set by HW concurrently. |
| 555 | 695 | */ |
| 556 | − page_table_check_pte_clear(mm, pte); | |
| 696 | + page_table_check_pte_clear(mm, addr, pte); | |
| 557 | 697 | } |
| 558 | 698 | |
| 559 | 699 | #ifdef CONFIG_GUP_GET_PXX_LOW_HIGH |
| @@ -648,7 +788,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, · 88 unmodified lines | ||
| 648 | 788 | pmd_t pmd = *pmdp; |
| 649 | 789 | |
| 650 | 790 | pmd_clear(pmdp); |
| 651 | − page_table_check_pmd_clear(mm, pmd); | |
| 791 | + page_table_check_pmd_clear(mm, address, pmd); | |
| 652 | 792 | |
| 653 | 793 | return pmd; |
| 654 | 794 | } |
| @@ -661,7 +801,7 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, · 6 unmodified lines | ||
| 661 | 801 | pud_t pud = *pudp; |
| 662 | 802 | |
| 663 | 803 | pud_clear(pudp); |
| 664 | − page_table_check_pud_clear(mm, pud); | |
| 804 | + page_table_check_pud_clear(mm, address, pud); | |
| 665 | 805 | |
| 666 | 806 | return pud; |
| 667 | 807 | } |
| @@ -92,6 +92,7 @@ struct anon_vma_chain { · 91 unmodified lines | ||
| 92 | 92 | }; |
| 93 | 93 | |
| 94 | 94 | enum ttu_flags { |
| 95 | + TTU_USE_SHARED_ZEROPAGE = 0x2, /* for unused pages of large folios */ | |
| 95 | 96 | TTU_SPLIT_HUGE_PMD = 0x4, /* split huge PMD if any */ |
| 96 | 97 | TTU_IGNORE_MLOCK = 0x8, /* ignore mlock */ |
| 97 | 98 | TTU_SYNC = 0x10, /* avoid racy checks with PVMW_SYNC */ |
| @@ -104,75 +105,8 @@ enum ttu_flags { · 6 unmodified lines | ||
| 104 | 105 | }; |
| 105 | 106 | |
| 106 | 107 | #ifdef CONFIG_MMU |
| 107 | −static inline void get_anon_vma(struct anon_vma *anon_vma) | |
| 108 | −{ | |
| 109 | − atomic_inc(&anon_vma->refcount); | |
| 110 | −} | |
| 111 | − | |
| 112 | −void __put_anon_vma(struct anon_vma *anon_vma); | |
| 113 | − | |
| 114 | −static inline void put_anon_vma(struct anon_vma *anon_vma) | |
| 115 | −{ | |
| 116 | − if (atomic_dec_and_test(&anon_vma->refcount)) | |
| 117 | − __put_anon_vma(anon_vma); | |
| 118 | −} | |
| 119 | − | |
| 120 | −static inline void anon_vma_lock_write(struct anon_vma *anon_vma) | |
| 121 | −{ | |
| 122 | − down_write(&anon_vma->root->rwsem); | |
| 123 | −} | |
| 124 | − | |
| 125 | −static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) | |
| 126 | −{ | |
| 127 | − return down_write_trylock(&anon_vma->root->rwsem); | |
| 128 | −} | |
| 129 | − | |
| 130 | −static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) | |
| 131 | −{ | |
| 132 | − up_write(&anon_vma->root->rwsem); | |
| 133 | −} | |
| 134 | − | |
| 135 | −static inline void anon_vma_lock_read(struct anon_vma *anon_vma) | |
| 136 | −{ | |
| 137 | − down_read(&anon_vma->root->rwsem); | |
| 138 | −} | |
| 139 | − | |
| 140 | −static inline int anon_vma_trylock_read(struct anon_vma *anon_vma) | |
| 141 | −{ | |
| 142 | − return down_read_trylock(&anon_vma->root->rwsem); | |
| 143 | −} | |
| 144 | − | |
| 145 | −static inline void anon_vma_unlock_read(struct anon_vma *anon_vma) | |
| 146 | −{ | |
| 147 | − up_read(&anon_vma->root->rwsem); | |
| 148 | −} | |
| 149 | 108 | |
| 150 | − | |
| 151 | −/* | |
| 152 | − * anon_vma helper functions. | |
| 153 | − */ | |
| 154 | 109 | void anon_vma_init(void); /* create anon_vma_cachep */ |
| 155 | −int __anon_vma_prepare(struct vm_area_struct *); | |
| 156 | −void unlink_anon_vmas(struct vm_area_struct *); | |
| 157 | −int anon_vma_clone(struct vm_area_struct *, struct vm_area_struct *); | |
| 158 | −int anon_vma_fork(struct vm_area_struct *, struct vm_area_struct *); | |
| 159 | − | |
| 160 | −static inline int anon_vma_prepare(struct vm_area_struct *vma) | |
| 161 | −{ | |
| 162 | − if (likely(vma->anon_vma)) | |
| 163 | − return 0; | |
| 164 | − | |
| 165 | − return __anon_vma_prepare(vma); | |
| 166 | −} | |
| 167 | − | |
| 168 | −static inline void anon_vma_merge(struct vm_area_struct *vma, | |
| 169 | − struct vm_area_struct *next) | |
| 170 | −{ | |
| 171 | − VM_BUG_ON_VMA(vma->anon_vma != next->anon_vma, vma); | |
| 172 | − unlink_anon_vmas(next); | |
| 173 | −} | |
| 174 | − | |
| 175 | −struct anon_vma *folio_get_anon_vma(const struct folio *folio); | |
| 176 | 110 | |
| 177 | 111 | #ifdef CONFIG_MM_ID |
| 178 | 112 | static __always_inline void folio_lock_large_mapcount(struct folio *folio) |
| @@ -1000,12 +934,8 @@ int mapping_wrprotect_range(struct address_space *mapping, pgoff_t pgoff, · 821 unmodified lines | ||
| 1000 | 934 | int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, |
| 1001 | 935 | struct vm_area_struct *vma); |
| 1002 | 936 | |
| 1003 | −enum rmp_flags { | |
| 1004 | − RMP_LOCKED = 1 << 0, | |
| 1005 | − RMP_USE_SHARED_ZEROPAGE = 1 << 1, | |
| 1006 | −}; | |
| 1007 | − | |
| 1008 | −void remove_migration_ptes(struct folio *src, struct folio *dst, int flags); | |
| 937 | +void remove_migration_ptes(struct folio *src, struct folio *dst, | |
| 938 | + enum ttu_flags flags); | |
| 1009 | 939 | |
| 1010 | 940 | /* |
| 1011 | 941 | * rmap_walk_control: To control rmap traversing for specific needs |
| @@ -1410,6 +1410,10 @@ struct task_struct { · 1409 unmodified lines | ||
| 1410 | 1410 | |
| 1411 | 1411 | struct page_frag task_frag; |
| 1412 | 1412 | |
| 1413 | +#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE | |
| 1414 | + struct lazy_mmu_state lazy_mmu_state; | |
| 1415 | +#endif | |
| 1416 | + | |
| 1413 | 1417 | #ifdef CONFIG_TASK_DELAY_ACCT |
| 1414 | 1418 | struct task_delay_info *delays; |
| 1415 | 1419 | #endif |
| @@ -1693,6 +1697,47 @@ static inline char task_state_to_char(struct task_struct *tsk) · 277 unmodified lines | ||
| 1693 | 1697 | return task_index_to_char(task_state_index(tsk)); |
| 1694 | 1698 | } |
| 1695 | 1699 | |
| 1700 | +#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE | |
| 1701 | +/** | |
| 1702 | + * __task_lazy_mmu_mode_active() - Test the lazy MMU mode state for a task. | |
| 1703 | + * @tsk: The task to check. | |
| 1704 | + * | |
| 1705 | + * Test whether @tsk has its lazy MMU mode state set to active (i.e. enabled | |
| 1706 | + * and not paused). | |
| 1707 | + * | |
| 1708 | + * This function only considers the state saved in task_struct; to test whether | |
| 1709 | + * current actually is in lazy MMU mode, is_lazy_mmu_mode_active() should be | |
| 1710 | + * used instead. | |
| 1711 | + * | |
| 1712 | + * This function is intended for architectures that implement the lazy MMU | |
| 1713 | + * mode; it must not be called from generic code. | |
| 1714 | + */ | |
| 1715 | +static inline bool __task_lazy_mmu_mode_active(struct task_struct *tsk) | |
| 1716 | +{ | |
| 1717 | + struct lazy_mmu_state *state = &tsk->lazy_mmu_state; | |
| 1718 | + | |
| 1719 | + return state->enable_count > 0 && state->pause_count == 0; | |
| 1720 | +} | |
| 1721 | + | |
| 1722 | +/** | |
| 1723 | + * is_lazy_mmu_mode_active() - Test whether we are currently in lazy MMU mode. | |
| 1724 | + * | |
| 1725 | + * Test whether the current context is in lazy MMU mode. This is true if both: | |
| 1726 | + * 1. We are not in interrupt context | |
| 1727 | + * 2. Lazy MMU mode is active for the current task | |
| 1728 | + * | |
| 1729 | + * This function is intended for architectures that implement the lazy MMU | |
| 1730 | + * mode; it must not be called from generic code. | |
| 1731 | + */ | |
| 1732 | +static inline bool is_lazy_mmu_mode_active(void) | |
| 1733 | +{ | |
| 1734 | + if (in_interrupt()) | |
| 1735 | + return false; | |
| 1736 | + | |
| 1737 | + return __task_lazy_mmu_mode_active(current); | |
| 1738 | +} | |
| 1739 | +#endif | |
| 1740 | + | |
| 1696 | 1741 | extern struct pid *cad_pid; |
| 1697 | 1742 | |
| 1698 | 1743 | /* |
| @@ -224,13 +224,11 @@ enum { · 223 unmodified lines | ||
| 224 | 224 | #define COMPACT_CLUSTER_MAX SWAP_CLUSTER_MAX |
| 225 | 225 | |
| 226 | 226 | /* Bit flag in swap_map */ |
| 227 | −#define SWAP_HAS_CACHE 0x40 /* Flag page is cached, in first swap_map */ | |
| 228 | 227 | #define COUNT_CONTINUED 0x80 /* Flag swap_map continuation for full count */ |
| 229 | 228 | |
| 230 | 229 | /* Special value in first swap_map */ |
| 231 | 230 | #define SWAP_MAP_MAX 0x3e /* Max count */ |
| 232 | 231 | #define SWAP_MAP_BAD 0x3f /* Note page is bad */ |
| 233 | −#define SWAP_MAP_SHMEM 0xbf /* Owned by shmem/tmpfs */ | |
| 234 | 232 | |
| 235 | 233 | /* Special value in each swap_map continuation */ |
| 236 | 234 | #define SWAP_CONT_MAX 0x7f /* Max count */ |
| @@ -453,16 +451,7 @@ static inline long get_nr_swap_pages(void) · 216 unmodified lines | ||
| 453 | 451 | } |
| 454 | 452 | |
| 455 | 453 | extern void si_swapinfo(struct sysinfo *); |
| 456 | −int folio_alloc_swap(struct folio *folio); | |
| 457 | −bool folio_free_swap(struct folio *folio); | |
| 458 | −void put_swap_folio(struct folio *folio, swp_entry_t entry); | |
| 459 | −extern swp_entry_t get_swap_page_of_type(int); | |
| 460 | 454 | extern int add_swap_count_continuation(swp_entry_t, gfp_t); |
| 461 | −extern void swap_shmem_alloc(swp_entry_t, int); | |
| 462 | −extern int swap_duplicate(swp_entry_t); | |
| 463 | −extern int swapcache_prepare(swp_entry_t entry, int nr); | |
| 464 | −extern void swap_free_nr(swp_entry_t entry, int nr_pages); | |
| 465 | −extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); | |
| 466 | 455 | int swap_type_of(dev_t device, sector_t offset); |
| 467 | 456 | int find_first_swap(dev_t *device); |
| 468 | 457 | extern unsigned int count_swap_pages(int, int); |
| @@ -474,6 +463,29 @@ struct backing_dev_info; · 5 unmodified lines | ||
| 474 | 463 | extern struct swap_info_struct *get_swap_device(swp_entry_t entry); |
| 475 | 464 | sector_t swap_folio_sector(struct folio *folio); |
| 476 | 465 | |
| 466 | +/* | |
| 467 | + * If there is an existing swap slot reference (swap entry) and the caller | |
| 468 | + * guarantees that there is no race modification of it (e.g., PTL | |
| 469 | + * protecting the swap entry in page table; shmem's cmpxchg protects t | |
| 470 | + * he swap entry in shmem mapping), these two helpers below can be used | |
| 471 | + * to put/dup the entries directly. | |
| 472 | + * | |
| 473 | + * All entries must be allocated by folio_alloc_swap(). And they must have | |
| 474 | + * a swap count > 1. See comments of folio_*_swap helpers for more info. | |
| 475 | + */ | |
| 476 | +int swap_dup_entry_direct(swp_entry_t entry); | |
| 477 | +void swap_put_entries_direct(swp_entry_t entry, int nr); | |
| 478 | + | |
| 479 | +/* | |
| 480 | + * folio_free_swap tries to free the swap entries pinned by a swap cache | |
| 481 | + * folio, it has to be here to be called by other components. | |
| 482 | + */ | |
| 483 | +bool folio_free_swap(struct folio *folio); | |
| 484 | + | |
| 485 | +/* Allocate / free (hibernation) exclusive entries */ | |
| 486 | +swp_entry_t swap_alloc_hibernation_slot(int type); | |
| 487 | +void swap_free_hibernation_slot(swp_entry_t entry); | |
| 488 | + | |
| 477 | 489 | static inline void put_swap_device(struct swap_info_struct *si) |
| 478 | 490 | { |
| 479 | 491 | percpu_ref_put(&si->users); |
| @@ -501,10 +513,6 @@ static inline void put_swap_device(struct swap_info_struct *si) · 21 unmodified lines | ||
| 501 | 513 | #define free_pages_and_swap_cache(pages, nr) \ |
| 502 | 514 | release_pages((pages), (nr)); |
| 503 | 515 | |
| 504 | −static inline void free_swap_and_cache_nr(swp_entry_t entry, int nr) | |
| 505 | −{ | |
| 506 | −} | |
| 507 | − | |
| 508 | 516 | static inline void free_swap_cache(struct folio *folio) |
| 509 | 517 | { |
| 510 | 518 | } |
| @@ -514,25 +522,12 @@ static inline int add_swap_count_continuation(swp_entry_t swp, gfp_t gfp_mask) · 3 unmodified lines | ||
| 514 | 522 | return 0; |
| 515 | 523 | } |
| 516 | 524 | |
| 517 | −static inline void swap_shmem_alloc(swp_entry_t swp, int nr) | |
| 518 | −{ | |
| 519 | −} | |
| 520 | − | |
| 521 | −static inline int swap_duplicate(swp_entry_t swp) | |
| 522 | −{ | |
| 523 | − return 0; | |
| 524 | −} | |
| 525 | − | |
| 526 | −static inline int swapcache_prepare(swp_entry_t swp, int nr) | |
| 525 | +static inline int swap_dup_entry_direct(swp_entry_t ent) | |
| 527 | 526 | { |
| 528 | 527 | return 0; |
| 529 | 528 | } |
| 530 | 529 | |
| 531 | −static inline void swap_free_nr(swp_entry_t entry, int nr_pages) | |
| 532 | −{ | |
| 533 | −} | |
| 534 | − | |
| 535 | −static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) | |
| 530 | +static inline void swap_put_entries_direct(swp_entry_t ent, int nr) | |
| 536 | 531 | { |
| 537 | 532 | } |
| 538 | 533 | |
| @@ -551,11 +546,6 @@ static inline int swp_swapcount(swp_entry_t entry) · 12 unmodified lines | ||
| 551 | 546 | return 0; |
| 552 | 547 | } |
| 553 | 548 | |
| 554 | −static inline int folio_alloc_swap(struct folio *folio) | |
| 555 | −{ | |
| 556 | − return -EINVAL; | |
| 557 | −} | |
| 558 | − | |
| 559 | 549 | static inline bool folio_free_swap(struct folio *folio) |
| 560 | 550 | { |
| 561 | 551 | return false; |
| @@ -568,17 +558,6 @@ static inline int add_swap_extent(struct swap_info_struct *sis, · 6 unmodified lines | ||
| 568 | 558 | return -EINVAL; |
| 569 | 559 | } |
| 570 | 560 | #endif /* CONFIG_SWAP */ |
| 571 | − | |
| 572 | −static inline void free_swap_and_cache(swp_entry_t entry) | |
| 573 | −{ | |
| 574 | − free_swap_and_cache_nr(entry, 1); | |
| 575 | −} | |
| 576 | − | |
| 577 | −static inline void swap_free(swp_entry_t entry) | |
| 578 | −{ | |
| 579 | − swap_free_nr(entry, 1); | |
| 580 | −} | |
| 581 | − | |
| 582 | 561 | #ifdef CONFIG_MEMCG |
| 583 | 562 | static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) |
| 584 | 563 | { |
| @@ -122,13 +122,13 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, · 121 unmodified lines | ||
| 122 | 122 | THP_SWPOUT, |
| 123 | 123 | THP_SWPOUT_FALLBACK, |
| 124 | 124 | #endif |
| 125 | −#ifdef CONFIG_MEMORY_BALLOON | |
| 125 | +#ifdef CONFIG_BALLOON | |
| 126 | 126 | BALLOON_INFLATE, |
| 127 | 127 | BALLOON_DEFLATE, |
| 128 | −#ifdef CONFIG_BALLOON_COMPACTION | |
| 128 | +#ifdef CONFIG_BALLOON_MIGRATION | |
| 129 | 129 | BALLOON_MIGRATE, |
| 130 | −#endif | |
| 131 | −#endif | |
| 130 | +#endif /* CONFIG_BALLOON_MIGRATION */ | |
| 131 | +#endif /* CONFIG_BALLOON */ | |
| 132 | 132 | #ifdef CONFIG_DEBUG_TLBFLUSH |
| 133 | 133 | NR_TLB_REMOTE_FLUSH, /* cpu tried to flush others' tlbs */ |
| 134 | 134 | NR_TLB_REMOTE_FLUSH_RECEIVED,/* cpu received ipi for flush */ |
| @@ -286,10 +286,8 @@ void mod_node_page_state(struct pglist_data *, enum node_stat_item, long); · 285 unmodified lines | ||
| 286 | 286 | void inc_node_page_state(struct page *, enum node_stat_item); |
| 287 | 287 | void dec_node_page_state(struct page *, enum node_stat_item); |
| 288 | 288 | |
| 289 | −extern void inc_node_state(struct pglist_data *, enum node_stat_item); | |
| 290 | 289 | extern void __inc_zone_state(struct zone *, enum zone_stat_item); |
| 291 | 290 | extern void __inc_node_state(struct pglist_data *, enum node_stat_item); |
| 292 | −extern void dec_zone_state(struct zone *, enum zone_stat_item); | |
| 293 | 291 | extern void __dec_zone_state(struct zone *, enum zone_stat_item); |
| 294 | 292 | extern void __dec_node_state(struct pglist_data *, enum node_stat_item); |
| 295 | 293 | |
| @@ -395,10 +393,6 @@ static inline void __dec_node_page_state(struct page *page, · 99 unmodified lines | ||
| 395 | 393 | #define dec_node_page_state __dec_node_page_state |
| 396 | 394 | #define mod_node_page_state __mod_node_page_state |
| 397 | 395 | |
| 398 | −#define inc_zone_state __inc_zone_state | |
| 399 | −#define inc_node_state __inc_node_state | |
| 400 | −#define dec_zone_state __dec_zone_state | |
| 401 | − | |
| 402 | 396 | #define set_pgdat_percpu_threshold(pgdat, callback) { } |
| 403 | 397 | |
| 404 | 398 | static inline void refresh_zone_stat_thresholds(void) { } |
| @@ -328,9 +328,6 @@ struct dirty_throttle_control { · 327 unmodified lines | ||
| 328 | 328 | bool dirty_exceeded; |
| 329 | 329 | }; |
| 330 | 330 | |
| 331 | −void laptop_io_completion(struct backing_dev_info *info); | |
| 332 | −void laptop_sync_completion(void); | |
| 333 | −void laptop_mode_timer_fn(struct timer_list *t); | |
| 334 | 331 | bool node_dirty_ok(struct pglist_data *pgdat); |
| 335 | 332 | int wb_domain_init(struct wb_domain *dom, gfp_t gfp); |
| 336 | 333 | #ifdef CONFIG_CGROUP_WRITEBACK |
| @@ -342,7 +339,6 @@ extern struct wb_domain global_wb_domain; · 5 unmodified lines | ||
| 342 | 339 | /* These are exported to sysctl. */ |
| 343 | 340 | extern unsigned int dirty_writeback_interval; |
| 344 | 341 | extern unsigned int dirty_expire_interval; |
| 345 | −extern int laptop_mode; | |
| 346 | 342 | |
| 347 | 343 | void global_dirty_limits(unsigned long *pbackground, unsigned long *pdirty); |
| 348 | 344 | unsigned long wb_calc_thresh(struct bdi_writeback *wb, unsigned long thresh); |
| @@ -22,6 +22,7 @@ struct zs_pool_stats { · 21 unmodified lines | ||
| 22 | 22 | }; |
| 23 | 23 | |
| 24 | 24 | struct zs_pool; |
| 25 | +struct scatterlist; | |
| 25 | 26 | |
| 26 | 27 | struct zs_pool *zs_create_pool(const char *name); |
| 27 | 28 | void zs_destroy_pool(struct zs_pool *pool); |
| @@ -40,9 +41,12 @@ unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size); · 12 unmodified lines | ||
| 40 | 41 | void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); |
| 41 | 42 | |
| 42 | 43 | void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, |
| 43 | − void *local_copy); | |
| 44 | + size_t mem_len, void *local_copy); | |
| 44 | 45 | void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, |
| 45 | − void *handle_mem); | |
| 46 | + size_t mem_len, void *handle_mem); | |
| 47 | +void zs_obj_read_sg_begin(struct zs_pool *pool, unsigned long handle, | |
| 48 | + struct scatterlist *sg, size_t mem_len); | |
| 49 | +void zs_obj_read_sg_end(struct zs_pool *pool, unsigned long handle); | |
| 46 | 50 | void zs_obj_write(struct zs_pool *pool, unsigned long handle, |
| 47 | 51 | void *handle_mem, size_t mem_len); |
| 48 | 52 | |
| @@ -9,6 +9,47 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <linux/types.h> |
| 10 | 10 | #include <linux/tracepoint.h> |
| 11 | 11 | |
| 12 | +TRACE_EVENT(damos_stat_after_apply_interval, | |
| 13 | + | |
| 14 | + TP_PROTO(unsigned int context_idx, unsigned int scheme_idx, | |
| 15 | + struct damos_stat *stat), | |
| 16 | + | |
| 17 | + TP_ARGS(context_idx, scheme_idx, stat), | |
| 18 | + | |
| 19 | + TP_STRUCT__entry( | |
| 20 | + __field(unsigned int, context_idx) | |
| 21 | + __field(unsigned int, scheme_idx) | |
| 22 | + __field(unsigned long, nr_tried) | |
| 23 | + __field(unsigned long, sz_tried) | |
| 24 | + __field(unsigned long, nr_applied) | |
| 25 | + __field(unsigned long, sz_applied) | |
| 26 | + __field(unsigned long, sz_ops_filter_passed) | |
| 27 | + __field(unsigned long, qt_exceeds) | |
| 28 | + __field(unsigned long, nr_snapshots) | |
| 29 | + ), | |
| 30 | + | |
| 31 | + TP_fast_assign( | |
| 32 | + __entry->context_idx = context_idx; | |
| 33 | + __entry->scheme_idx = scheme_idx; | |
| 34 | + __entry->nr_tried = stat->nr_tried; | |
| 35 | + __entry->sz_tried = stat->sz_tried; | |
| 36 | + __entry->nr_applied = stat->nr_applied; | |
| 37 | + __entry->sz_applied = stat->sz_applied; | |
| 38 | + __entry->sz_ops_filter_passed = stat->sz_ops_filter_passed; | |
| 39 | + __entry->qt_exceeds = stat->qt_exceeds; | |
| 40 | + __entry->nr_snapshots = stat->nr_snapshots; | |
| 41 | + ), | |
| 42 | + | |
| 43 | + TP_printk("ctx_idx=%u scheme_idx=%u nr_tried=%lu sz_tried=%lu " | |
| 44 | + "nr_applied=%lu sz_tried=%lu sz_ops_filter_passed=%lu " | |
| 45 | + "qt_exceeds=%lu nr_snapshots=%lu", | |
| 46 | + __entry->context_idx, __entry->scheme_idx, | |
| 47 | + __entry->nr_tried, __entry->sz_tried, | |
| 48 | + __entry->nr_applied, __entry->sz_applied, | |
| 49 | + __entry->sz_ops_filter_passed, __entry->qt_exceeds, | |
| 50 | + __entry->nr_snapshots) | |
| 51 | +); | |
| 52 | + | |
| 12 | 53 | TRACE_EVENT(damos_esz, |
| 13 | 54 | |
| 14 | 55 | TP_PROTO(unsigned int context_idx, unsigned int scheme_idx, |
| @@ -37,7 +37,8 @@ · 36 unmodified lines | ||
| 37 | 37 | EM( SCAN_PAGE_HAS_PRIVATE, "page_has_private") \ |
| 38 | 38 | EM( SCAN_STORE_FAILED, "store_failed") \ |
| 39 | 39 | EM( SCAN_COPY_MC, "copy_poisoned_page") \ |
| 40 | − EMe(SCAN_PAGE_FILLED, "page_filled") | |
| 40 | + EM( SCAN_PAGE_FILLED, "page_filled") \ | |
| 41 | + EMe(SCAN_PAGE_DIRTY_OR_WRITEBACK, "page_dirty_or_writeback") | |
| 41 | 42 | |
| 42 | 43 | #undef EM |
| 43 | 44 | #undef EMe |
| @@ -40,6 +40,16 @@ · 39 unmodified lines | ||
| 40 | 40 | {_VMSCAN_THROTTLE_CONGESTED, "VMSCAN_THROTTLE_CONGESTED"} \ |
| 41 | 41 | ) : "VMSCAN_THROTTLE_NONE" |
| 42 | 42 | |
| 43 | +TRACE_DEFINE_ENUM(KSWAPD_CLEAR_HOPELESS_OTHER); | |
| 44 | +TRACE_DEFINE_ENUM(KSWAPD_CLEAR_HOPELESS_KSWAPD); | |
| 45 | +TRACE_DEFINE_ENUM(KSWAPD_CLEAR_HOPELESS_DIRECT); | |
| 46 | +TRACE_DEFINE_ENUM(KSWAPD_CLEAR_HOPELESS_PCP); | |
| 47 | + | |
| 48 | +#define kswapd_clear_hopeless_reason_ops \ | |
| 49 | + {KSWAPD_CLEAR_HOPELESS_KSWAPD, "KSWAPD"}, \ | |
| 50 | + {KSWAPD_CLEAR_HOPELESS_DIRECT, "DIRECT"}, \ | |
| 51 | + {KSWAPD_CLEAR_HOPELESS_PCP, "PCP"}, \ | |
| 52 | + {KSWAPD_CLEAR_HOPELESS_OTHER, "OTHER"} | |
| 43 | 53 | |
| 44 | 54 | #define trace_reclaim_flags(file) ( \ |
| 45 | 55 | (file ? RECLAIM_WB_FILE : RECLAIM_WB_ANON) | \ |
| @@ -535,6 +545,47 @@ TRACE_EVENT(mm_vmscan_throttled, · 489 unmodified lines | ||
| 535 | 545 | __entry->usec_delayed, |
| 536 | 546 | show_throttle_flags(__entry->reason)) |
| 537 | 547 | ); |
| 548 | + | |
| 549 | +TRACE_EVENT(mm_vmscan_kswapd_reclaim_fail, | |
| 550 | + | |
| 551 | + TP_PROTO(int nid, int failures), | |
| 552 | + | |
| 553 | + TP_ARGS(nid, failures), | |
| 554 | + | |
| 555 | + TP_STRUCT__entry( | |
| 556 | + __field(int, nid) | |
| 557 | + __field(int, failures) | |
| 558 | + ), | |
| 559 | + | |
| 560 | + TP_fast_assign( | |
| 561 | + __entry->nid = nid; | |
| 562 | + __entry->failures = failures; | |
| 563 | + ), | |
| 564 | + | |
| 565 | + TP_printk("nid=%d failures=%d", | |
| 566 | + __entry->nid, __entry->failures) | |
| 567 | +); | |
| 568 | + | |
| 569 | +TRACE_EVENT(mm_vmscan_kswapd_clear_hopeless, | |
| 570 | + | |
| 571 | + TP_PROTO(int nid, int reason), | |
| 572 | + | |
| 573 | + TP_ARGS(nid, reason), | |
| 574 | + | |
| 575 | + TP_STRUCT__entry( | |
| 576 | + __field(int, nid) | |
| 577 | + __field(int, reason) | |
| 578 | + ), | |
| 579 | + | |
| 580 | + TP_fast_assign( | |
| 581 | + __entry->nid = nid; | |
| 582 | + __entry->reason = reason; | |
| 583 | + ), | |
| 584 | + | |
| 585 | + TP_printk("nid=%d reason=%s", | |
| 586 | + __entry->nid, | |
| 587 | + __print_symbolic(__entry->reason, kswapd_clear_hopeless_reason_ops)) | |
| 588 | +); | |
| 538 | 589 | #endif /* _TRACE_VMSCAN_H */ |
| 539 | 590 | |
| 540 | 591 | /* This part must be outside protection */ |
| @@ -42,7 +42,6 @@ · 41 unmodified lines | ||
| 42 | 42 | EM( WB_REASON_VMSCAN, "vmscan") \ |
| 43 | 43 | EM( WB_REASON_SYNC, "sync") \ |
| 44 | 44 | EM( WB_REASON_PERIODIC, "periodic") \ |
| 45 | − EM( WB_REASON_LAPTOP_TIMER, "laptop_timer") \ | |
| 46 | 45 | EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \ |
| 47 | 46 | EM( WB_REASON_FORKER_THREAD, "forker_thread") \ |
| 48 | 47 | EMe(WB_REASON_FOREIGN_FLUSH, "foreign_flush") |
| @@ -39,6 +39,9 @@ enum { · 38 unmodified lines | ||
| 39 | 39 | #define MPOL_MODE_FLAGS \ |
| 40 | 40 | (MPOL_F_STATIC_NODES | MPOL_F_RELATIVE_NODES | MPOL_F_NUMA_BALANCING) |
| 41 | 41 | |
| 42 | +/* Whether the nodemask is specified by users */ | |
| 43 | +#define MPOL_USER_NODEMASK_FLAGS (MPOL_F_STATIC_NODES | MPOL_F_RELATIVE_NODES) | |
| 44 | + | |
| 42 | 45 | /* Flags for get_mempolicy */ |
| 43 | 46 | #define MPOL_F_NODE (1<<0) /* return next IL mode instead of node mask */ |
| 44 | 47 | #define MPOL_F_ADDR (1<<1) /* look up vma using address */ |
| @@ -182,7 +182,7 @@ enum · 181 unmodified lines | ||
| 182 | 182 | VM_LOWMEM_RESERVE_RATIO=20,/* reservation ratio for lower memory zones */ |
| 183 | 183 | VM_MIN_FREE_KBYTES=21, /* Minimum free kilobytes to maintain */ |
| 184 | 184 | VM_MAX_MAP_COUNT=22, /* int: Maximum number of mmaps/address-space */ |
| 185 | − VM_LAPTOP_MODE=23, /* vm laptop mode */ | |
| 185 | + | |
| 186 | 186 | VM_BLOCK_DUMP=24, /* block dump mode */ |
| 187 | 187 | VM_HUGETLB_GROUP=25, /* permitted hugetlb group */ |
| 188 | 188 | VM_VFS_CACHE_PRESSURE=26, /* dcache/icache reclaim pressure */ |
| @@ -1025,6 +1025,7 @@ void start_kernel(void) · 1024 unmodified lines | ||
| 1025 | 1025 | page_address_init(); |
| 1026 | 1026 | pr_notice("%s", linux_banner); |
| 1027 | 1027 | setup_arch(&command_line); |
| 1028 | + mm_core_init_early(); | |
| 1028 | 1029 | /* Static keys and static calls are needed by LSMs */ |
| 1029 | 1030 | jump_label_init(); |
| 1030 | 1031 | static_call_init(); |
| @@ -458,9 +458,8 @@ static void guarantee_active_cpus(struct task_struct *tsk, · 457 unmodified lines | ||
| 458 | 458 | */ |
| 459 | 459 | static void guarantee_online_mems(struct cpuset *cs, nodemask_t *pmask) |
| 460 | 460 | { |
| 461 | − while (!nodes_intersects(cs->effective_mems, node_states[N_MEMORY])) | |
| 461 | + while (!nodes_and(*pmask, cs->effective_mems, node_states[N_MEMORY])) | |
| 462 | 462 | cs = parent_cs(cs); |
| 463 | − nodes_and(*pmask, cs->effective_mems, node_states[N_MEMORY]); | |
| 464 | 463 | } |
| 465 | 464 | |
| 466 | 465 | /** |
| @@ -2622,13 +2621,13 @@ static void update_nodemasks_hier(struct cpuset *cs, nodemask_t *new_mems) · 2155 unmodified lines | ||
| 2622 | 2621 | cpuset_for_each_descendant_pre(cp, pos_css, cs) { |
| 2623 | 2622 | struct cpuset *parent = parent_cs(cp); |
| 2624 | 2623 | |
| 2625 | − nodes_and(*new_mems, cp->mems_allowed, parent->effective_mems); | |
| 2624 | + bool has_mems = nodes_and(*new_mems, cp->mems_allowed, parent->effective_mems); | |
| 2626 | 2625 | |
| 2627 | 2626 | /* |
| 2628 | 2627 | * If it becomes empty, inherit the effective mask of the |
| 2629 | 2628 | * parent, which is guaranteed to have some MEMs. |
| 2630 | 2629 | */ |
| 2631 | − if (is_in_v2_mode() && nodes_empty(*new_mems)) | |
| 2630 | + if (is_in_v2_mode() && !has_mems) | |
| 2632 | 2631 | *new_mems = parent->effective_mems; |
| 2633 | 2632 | |
| 2634 | 2633 | /* Skip the whole subtree if the nodemask remains the same. */ |
| @@ -174,10 +174,10 @@ sector_t alloc_swapdev_block(int swap) · 173 unmodified lines | ||
| 174 | 174 | * Allocate a swap page and register that it has been allocated, so that |
| 175 | 175 | * it can be freed in case of an error. |
| 176 | 176 | */ |
| 177 | − offset = swp_offset(get_swap_page_of_type(swap)); | |
| 177 | + offset = swp_offset(swap_alloc_hibernation_slot(swap)); | |
| 178 | 178 | if (offset) { |
| 179 | 179 | if (swsusp_extents_insert(offset)) |
| 180 | − swap_free(swp_entry(swap, offset)); | |
| 180 | + swap_free_hibernation_slot(swp_entry(swap, offset)); | |
| 181 | 181 | else |
| 182 | 182 | return swapdev_block(swap, offset); |
| 183 | 183 | } |
| @@ -186,6 +186,7 @@ sector_t alloc_swapdev_block(int swap) · 2 unmodified lines | ||
| 186 | 186 | |
| 187 | 187 | void free_all_swap_pages(int swap) |
| 188 | 188 | { |
| 189 | + unsigned long offset; | |
| 189 | 190 | struct rb_node *node; |
| 190 | 191 | |
| 191 | 192 | /* |
| @@ -197,8 +198,9 @@ void free_all_swap_pages(int swap) · 5 unmodified lines | ||
| 197 | 198 | |
| 198 | 199 | ext = rb_entry(node, struct swsusp_extent, node); |
| 199 | 200 | rb_erase(node, &swsusp_extents); |
| 200 | − swap_free_nr(swp_entry(swap, ext->start), | |
| 201 | − ext->end - ext->start + 1); | |
| 201 | + | |
| 202 | + for (offset = ext->start; offset <= ext->end; offset++) | |
| 203 | + swap_free_hibernation_slot(swp_entry(swap, offset)); | |
| 202 | 204 | |
| 203 | 205 | kfree(ext); |
| 204 | 206 | } |
| @@ -776,31 +776,38 @@ EXPORT_SYMBOL(page_alloc_tagging_ops); · 775 unmodified lines | ||
| 776 | 776 | static int proc_mem_profiling_handler(const struct ctl_table *table, int write, |
| 777 | 777 | void *buffer, size_t *lenp, loff_t *ppos) |
| 778 | 778 | { |
| 779 | − if (!mem_profiling_support && write) | |
| 780 | − return -EINVAL; | |
| 779 | + if (write) { | |
| 780 | + /* | |
| 781 | + * Call from do_sysctl_args() which is a no-op since the same | |
| 782 | + * value was already set by setup_early_mem_profiling. | |
| 783 | + * Return success to avoid warnings from do_sysctl_args(). | |
| 784 | + */ | |
| 785 | + if (!current->mm) | |
| 786 | + return 0; | |
| 787 | + | |
| 788 | +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG | |
| 789 | + /* User can't toggle profiling while debugging */ | |
| 790 | + return -EACCES; | |
| 791 | +#endif | |
| 792 | + if (!mem_profiling_support) | |
| 793 | + return -EINVAL; | |
| 794 | + } | |
| 781 | 795 | |
| 782 | 796 | return proc_do_static_key(table, write, buffer, lenp, ppos); |
| 783 | 797 | } |
| 784 | 798 | |
| 785 | 799 | |
| 786 | −static struct ctl_table memory_allocation_profiling_sysctls[] = { | |
| 800 | +static const struct ctl_table memory_allocation_profiling_sysctls[] = { | |
| 787 | 801 | { |
| 788 | 802 | .procname = "mem_profiling", |
| 789 | 803 | .data = &mem_alloc_profiling_key, |
| 790 | −#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG | |
| 791 | − .mode = 0444, | |
| 792 | −#else | |
| 793 | 804 | .mode = 0644, |
| 794 | −#endif | |
| 795 | 805 | .proc_handler = proc_mem_profiling_handler, |
| 796 | 806 | }, |
| 797 | 807 | }; |
| 798 | 808 | |
| 799 | 809 | static void __init sysctl_init(void) |
| 800 | 810 | { |
| 801 | − if (!mem_profiling_support) | |
| 802 | − memory_allocation_profiling_sysctls[0].mode = 0444; | |
| 803 | − | |
| 804 | 811 | register_sysctl_init("vm", memory_allocation_profiling_sysctls); |
| 805 | 812 | } |
| 806 | 813 | #else /* CONFIG_SYSCTL */ |
| @@ -58,6 +58,9 @@ __param(int, run_test_mask, 7, · 57 unmodified lines | ||
| 58 | 58 | /* Add a new test case description here. */ |
| 59 | 59 | ); |
| 60 | 60 | |
| 61 | +__param(int, nr_pcpu_objects, 35000, | |
| 62 | + "Number of pcpu objects to allocate for pcpu_alloc_test"); | |
| 63 | + | |
| 61 | 64 | /* |
| 62 | 65 | * This is for synchronization of setup phase. |
| 63 | 66 | */ |
| @@ -317,24 +320,24 @@ pcpu_alloc_test(void) · 253 unmodified lines | ||
| 317 | 320 | size_t size, align; |
| 318 | 321 | int i; |
| 319 | 322 | |
| 320 | − pcpu = vmalloc(sizeof(void __percpu *) * 35000); | |
| 323 | + pcpu = vmalloc(sizeof(void __percpu *) * nr_pcpu_objects); | |
| 321 | 324 | if (!pcpu) |
| 322 | 325 | return -1; |
| 323 | 326 | |
| 324 | − for (i = 0; i < 35000; i++) { | |
| 327 | + for (i = 0; i < nr_pcpu_objects; i++) { | |
| 325 | 328 | size = get_random_u32_inclusive(1, PAGE_SIZE / 4); |
| 326 | 329 | |
| 327 | 330 | /* |
| 328 | 331 | * Maximum PAGE_SIZE |
| 329 | 332 | */ |
| 330 | − align = 1 << get_random_u32_inclusive(1, 11); | |
| 333 | + align = 1 << get_random_u32_inclusive(1, PAGE_SHIFT - 1); | |
| 331 | 334 | |
| 332 | 335 | pcpu[i] = __alloc_percpu(size, align); |
| 333 | 336 | if (!pcpu[i]) |
| 334 | 337 | rv = -1; |
| 335 | 338 | } |
| 336 | 339 | |
| 337 | − for (i = 0; i < 35000; i++) | |
| 340 | + for (i = 0; i < nr_pcpu_objects; i++) | |
| 338 | 341 | free_percpu(pcpu[i]); |
| 339 | 342 | |
| 340 | 343 | vfree(pcpu); |
| @@ -582,23 +582,20 @@ config SPLIT_PMD_PTLOCKS · 581 unmodified lines | ||
| 582 | 582 | |
| 583 | 583 | # |
| 584 | 584 | # support for memory balloon |
| 585 | −config MEMORY_BALLOON | |
| 585 | +config BALLOON | |
| 586 | 586 | bool |
| 587 | 587 | |
| 588 | 588 | # |
| 589 | −# support for memory balloon compaction | |
| 590 | −config BALLOON_COMPACTION | |
| 591 | − bool "Allow for balloon memory compaction/migration" | |
| 589 | +# support for memory balloon page migration | |
| 590 | +config BALLOON_MIGRATION | |
| 591 | + bool "Allow for balloon memory migration" | |
| 592 | 592 | default y |
| 593 | − depends on COMPACTION && MEMORY_BALLOON | |
| 593 | + depends on MIGRATION && BALLOON | |
| 594 | 594 | help |
| 595 | − Memory fragmentation introduced by ballooning might reduce | |
| 596 | − significantly the number of 2MB contiguous memory blocks that can be | |
| 597 | − used within a guest, thus imposing performance penalties associated | |
| 598 | − with the reduced number of transparent huge pages that could be used | |
| 599 | − by the guest workload. Allowing the compaction & migration for memory | |
| 600 | − pages enlisted as being part of memory balloon devices avoids the | |
| 601 | − scenario aforementioned and helps improving memory defragmentation. | |
| 595 | + Allow for migration of pages inflated in a memory balloon such that | |
| 596 | + they can be allocated from memory areas only available for movable | |
| 597 | + allocations (e.g., ZONE_MOVABLE, CMA) and such that they can be | |
| 598 | + migrated for memory defragmentation purposes by memory compaction. | |
| 602 | 599 | |
| 603 | 600 | # |
| 604 | 601 | # support for memory compaction |
| @@ -1440,14 +1437,12 @@ config ARCH_HAS_USER_SHADOW_STACK · 835 unmodified lines | ||
| 1440 | 1437 | The architecture has hardware support for userspace shadow call |
| 1441 | 1438 | stacks (eg, x86 CET, arm64 GCS or RISC-V Zicfiss). |
| 1442 | 1439 | |
| 1443 | −config ARCH_SUPPORTS_PT_RECLAIM | |
| 1440 | +config HAVE_ARCH_TLB_REMOVE_TABLE | |
| 1444 | 1441 | def_bool n |
| 1445 | 1442 | |
| 1446 | 1443 | config PT_RECLAIM |
| 1447 | − bool "reclaim empty user page table pages" | |
| 1448 | − default y | |
| 1449 | − depends on ARCH_SUPPORTS_PT_RECLAIM && MMU && SMP | |
| 1450 | − select MMU_GATHER_RCU_TABLE_FREE | |
| 1444 | + def_bool y | |
| 1445 | + depends on MMU_GATHER_RCU_TABLE_FREE && !HAVE_ARCH_TLB_REMOVE_TABLE | |
| 1451 | 1446 | help |
| 1452 | 1447 | Try to reclaim empty user page table pages in paths other than munmap |
| 1453 | 1448 | and exit_mmap path. |
| @@ -1457,6 +1452,25 @@ config PT_RECLAIM · 3 unmodified lines | ||
| 1457 | 1452 | config FIND_NORMAL_PAGE |
| 1458 | 1453 | def_bool n |
| 1459 | 1454 | |
| 1455 | +config ARCH_HAS_LAZY_MMU_MODE | |
| 1456 | + bool | |
| 1457 | + help | |
| 1458 | + The architecture uses the lazy MMU mode. This allows changes to | |
| 1459 | + MMU-related architectural state to be deferred until the mode is | |
| 1460 | + exited. See <linux/pgtable.h> for details. | |
| 1461 | + | |
| 1462 | +config LAZY_MMU_MODE_KUNIT_TEST | |
| 1463 | + tristate "KUnit tests for the lazy MMU mode" if !KUNIT_ALL_TESTS | |
| 1464 | + depends on ARCH_HAS_LAZY_MMU_MODE | |
| 1465 | + depends on KUNIT | |
| 1466 | + default KUNIT_ALL_TESTS | |
| 1467 | + help | |
| 1468 | + Enable this option to check that the lazy MMU mode interface behaves | |
| 1469 | + as expected. Only tests for the generic interface are included (not | |
| 1470 | + architecture-specific behaviours). | |
| 1471 | + | |
| 1472 | + If unsure, say N. | |
| 1473 | + | |
| 1460 | 1474 | source "mm/damon/Kconfig" |
| 1461 | 1475 | |
| 1462 | 1476 | endmenu |
| @@ -125,7 +125,7 @@ obj-$(CONFIG_CMA) += cma.o · 124 unmodified lines | ||
| 125 | 125 | obj-$(CONFIG_NUMA) += numa.o |
| 126 | 126 | obj-$(CONFIG_NUMA_MEMBLKS) += numa_memblks.o |
| 127 | 127 | obj-$(CONFIG_NUMA_EMU) += numa_emulation.o |
| 128 | −obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.o | |
| 128 | +obj-$(CONFIG_BALLOON) += balloon.o | |
| 129 | 129 | obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o |
| 130 | 130 | obj-$(CONFIG_PAGE_TABLE_CHECK) += page_table_check.o |
| 131 | 131 | obj-$(CONFIG_CMA_DEBUGFS) += cma_debug.o |
| @@ -149,4 +149,4 @@ obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o · 17 unmodified lines | ||
| 149 | 149 | obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o |
| 150 | 150 | obj-$(CONFIG_EXECMEM) += execmem.o |
| 151 | 151 | obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o |
| 152 | −obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o | |
| 152 | +obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o | |
| @@ -939,7 +939,7 @@ void wb_memcg_offline(struct mem_cgroup *memcg) · 938 unmodified lines | ||
| 939 | 939 | memcg_cgwb_list->next = NULL; /* prevent new wb's */ |
| 940 | 940 | spin_unlock_irq(&cgwb_lock); |
| 941 | 941 | |
| 942 | − queue_work(system_unbound_wq, &cleanup_offline_cgwbs_work); | |
| 942 | + queue_work(system_dfl_wq, &cleanup_offline_cgwbs_work); | |
| 943 | 943 | } |
| 944 | 944 | |
| 945 | 945 | /** |
| @@ -971,10 +971,10 @@ static int __init cgwb_init(void) · 25 unmodified lines | ||
| 971 | 971 | { |
| 972 | 972 | /* |
| 973 | 973 | * There can be many concurrent release work items overwhelming |
| 974 | − * system_wq. Put them in a separate wq and limit concurrency. | |
| 974 | + * system_percpu_wq. Put them in a separate wq and limit concurrency. | |
| 975 | 975 | * There's no point in executing many of these in parallel. |
| 976 | 976 | */ |
| 977 | − cgwb_release_wq = alloc_workqueue("cgwb_release", 0, 1); | |
| 977 | + cgwb_release_wq = alloc_workqueue("cgwb_release", WQ_PERCPU, 1); | |
| 978 | 978 | if (!cgwb_release_wq) |
| 979 | 979 | return -ENOMEM; |
| 980 | 980 | |
| @@ -1034,7 +1034,6 @@ struct backing_dev_info *bdi_alloc(int node_id) · 53 unmodified lines | ||
| 1034 | 1034 | bdi->capabilities = BDI_CAP_WRITEBACK; |
| 1035 | 1035 | bdi->ra_pages = VM_READAHEAD_PAGES; |
| 1036 | 1036 | bdi->io_pages = VM_READAHEAD_PAGES; |
| 1037 | − timer_setup(&bdi->laptop_mode_wb_timer, laptop_mode_timer_fn, 0); | |
| 1038 | 1037 | return bdi; |
| 1039 | 1038 | } |
| 1040 | 1039 | EXPORT_SYMBOL(bdi_alloc); |
| @@ -1156,8 +1155,6 @@ static void bdi_remove_from_list(struct backing_dev_info *bdi) · 115 unmodified lines | ||
| 1156 | 1155 | |
| 1157 | 1156 | void bdi_unregister(struct backing_dev_info *bdi) |
| 1158 | 1157 | { |
| 1159 | − timer_delete_sync(&bdi->laptop_mode_wb_timer); | |
| 1160 | − | |
| 1161 | 1158 | /* make sure nobody finds us on the bdi_list anymore */ |
| 1162 | 1159 | bdi_remove_from_list(bdi); |
| 1163 | 1160 | wb_shutdown(&bdi->wb); |
| @@ -1,28 +1,62 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | /* |
| 3 | − * mm/balloon_compaction.c | |
| 4 | − * | |
| 5 | − * Common interface for making balloon pages movable by compaction. | |
| 3 | + * Common interface for implementing a memory balloon, including support | |
| 4 | + * for migration of pages inflated in a memory balloon. | |
| 6 | 5 | * |
| 7 | 6 | * Copyright (C) 2012, Red Hat, Inc. Rafael Aquini <aquini@redhat.com> |
| 8 | 7 | */ |
| 9 | 8 | #include <linux/mm.h> |
| 10 | 9 | #include <linux/slab.h> |
| 11 | 10 | #include <linux/export.h> |
| 12 | −#include <linux/balloon_compaction.h> | |
| 11 | +#include <linux/balloon.h> | |
| 12 | + | |
| 13 | +/* | |
| 14 | + * Lock protecting the balloon_dev_info of all devices. We don't really | |
| 15 | + * expect more than one device. | |
| 16 | + */ | |
| 17 | +static DEFINE_SPINLOCK(balloon_pages_lock); | |
| 18 | + | |
| 19 | +/** | |
| 20 | + * balloon_page_insert - insert a page into the balloon's page list and make | |
| 21 | + * the page->private assignment accordingly. | |
| 22 | + * @balloon : pointer to balloon device | |
| 23 | + * @page : page to be assigned as a 'balloon page' | |
| 24 | + * | |
| 25 | + * Caller must ensure the balloon_pages_lock is held. | |
| 26 | + */ | |
| 27 | +static void balloon_page_insert(struct balloon_dev_info *balloon, | |
| 28 | + struct page *page) | |
| 29 | +{ | |
| 30 | + lockdep_assert_held(&balloon_pages_lock); | |
| 31 | + __SetPageOffline(page); | |
| 32 | + if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) { | |
| 33 | + SetPageMovableOps(page); | |
| 34 | + set_page_private(page, (unsigned long)balloon); | |
| 35 | + } | |
| 36 | + list_add(&page->lru, &balloon->pages); | |
| 37 | +} | |
| 38 | + | |
| 39 | +/** | |
| 40 | + * balloon_page_finalize - prepare a balloon page that was removed from the | |
| 41 | + * balloon list for release to the page allocator | |
| 42 | + * @page: page to be released to the page allocator | |
| 43 | + * | |
| 44 | + * Caller must ensure the balloon_pages_lock is held. | |
| 45 | + */ | |
| 46 | +static void balloon_page_finalize(struct page *page) | |
| 47 | +{ | |
| 48 | + lockdep_assert_held(&balloon_pages_lock); | |
| 49 | + if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) | |
| 50 | + set_page_private(page, 0); | |
| 51 | + /* PageOffline is sticky until the page is freed to the buddy. */ | |
| 52 | +} | |
| 13 | 53 | |
| 14 | 54 | static void balloon_page_enqueue_one(struct balloon_dev_info *b_dev_info, |
| 15 | 55 | struct page *page) |
| 16 | 56 | { |
| 17 | − /* | |
| 18 | − * Block others from accessing the 'page' when we get around to | |
| 19 | − * establishing additional references. We should be the only one | |
| 20 | − * holding a reference to the 'page' at this point. If we are not, then | |
| 21 | − * memory corruption is possible and we should stop execution. | |
| 22 | − */ | |
| 23 | − BUG_ON(!trylock_page(page)); | |
| 24 | 57 | balloon_page_insert(b_dev_info, page); |
| 25 | − unlock_page(page); | |
| 58 | + if (b_dev_info->adjust_managed_page_count) | |
| 59 | + adjust_managed_page_count(page, -1); | |
| 26 | 60 | __count_vm_event(BALLOON_INFLATE); |
| 27 | 61 | inc_node_page_state(page, NR_BALLOON_PAGES); |
| 28 | 62 | } |
| @@ -45,13 +79,13 @@ size_t balloon_page_list_enqueue(struct balloon_dev_info *b_dev_info, · 16 unmodified lines | ||
| 45 | 79 | unsigned long flags; |
| 46 | 80 | size_t n_pages = 0; |
| 47 | 81 | |
| 48 | − spin_lock_irqsave(&b_dev_info->pages_lock, flags); | |
| 82 | + spin_lock_irqsave(&balloon_pages_lock, flags); | |
| 49 | 83 | list_for_each_entry_safe(page, tmp, pages, lru) { |
| 50 | 84 | list_del(&page->lru); |
| 51 | 85 | balloon_page_enqueue_one(b_dev_info, page); |
| 52 | 86 | n_pages++; |
| 53 | 87 | } |
| 54 | − spin_unlock_irqrestore(&b_dev_info->pages_lock, flags); | |
| 88 | + spin_unlock_irqrestore(&balloon_pages_lock, flags); | |
| 55 | 89 | return n_pages; |
| 56 | 90 | } |
| 57 | 91 | EXPORT_SYMBOL_GPL(balloon_page_list_enqueue); |
| @@ -81,34 +115,26 @@ size_t balloon_page_list_dequeue(struct balloon_dev_info *b_dev_info, · 23 unmodified lines | ||
| 81 | 115 | unsigned long flags; |
| 82 | 116 | size_t n_pages = 0; |
| 83 | 117 | |
| 84 | − spin_lock_irqsave(&b_dev_info->pages_lock, flags); | |
| 118 | + spin_lock_irqsave(&balloon_pages_lock, flags); | |
| 85 | 119 | list_for_each_entry_safe(page, tmp, &b_dev_info->pages, lru) { |
| 86 | 120 | if (n_pages == n_req_pages) |
| 87 | 121 | break; |
| 88 | − | |
| 89 | − /* | |
| 90 | − * Block others from accessing the 'page' while we get around to | |
| 91 | − * establishing additional references and preparing the 'page' | |
| 92 | − * to be released by the balloon driver. | |
| 93 | − */ | |
| 94 | − if (!trylock_page(page)) | |
| 95 | − continue; | |
| 96 | − | |
| 97 | 122 | list_del(&page->lru); |
| 123 | + if (b_dev_info->adjust_managed_page_count) | |
| 124 | + adjust_managed_page_count(page, 1); | |
| 98 | 125 | balloon_page_finalize(page); |
| 99 | 126 | __count_vm_event(BALLOON_DEFLATE); |
| 100 | 127 | list_add(&page->lru, pages); |
| 101 | − unlock_page(page); | |
| 102 | 128 | dec_node_page_state(page, NR_BALLOON_PAGES); |
| 103 | 129 | n_pages++; |
| 104 | 130 | } |
| 105 | − spin_unlock_irqrestore(&b_dev_info->pages_lock, flags); | |
| 131 | + spin_unlock_irqrestore(&balloon_pages_lock, flags); | |
| 106 | 132 | |
| 107 | 133 | return n_pages; |
| 108 | 134 | } |
| 109 | 135 | EXPORT_SYMBOL_GPL(balloon_page_list_dequeue); |
| 110 | 136 | |
| 111 | −/* | |
| 137 | +/** | |
| 112 | 138 | * balloon_page_alloc - allocates a new page for insertion into the balloon |
| 113 | 139 | * page list. |
| 114 | 140 | * |
| @@ -120,14 +146,18 @@ EXPORT_SYMBOL_GPL(balloon_page_list_dequeue); · 5 unmodified lines | ||
| 120 | 146 | */ |
| 121 | 147 | struct page *balloon_page_alloc(void) |
| 122 | 148 | { |
| 123 | − struct page *page = alloc_page(balloon_mapping_gfp_mask() | | |
| 124 | − __GFP_NOMEMALLOC | __GFP_NORETRY | | |
| 125 | − __GFP_NOWARN); | |
| 126 | − return page; | |
| 149 | + gfp_t gfp_flags = __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN; | |
| 150 | + | |
| 151 | + if (IS_ENABLED(CONFIG_BALLOON_MIGRATION)) | |
| 152 | + gfp_flags |= GFP_HIGHUSER_MOVABLE; | |
| 153 | + else | |
| 154 | + gfp_flags |= GFP_HIGHUSER; | |
| 155 | + | |
| 156 | + return alloc_page(gfp_flags); | |
| 127 | 157 | } |
| 128 | 158 | EXPORT_SYMBOL_GPL(balloon_page_alloc); |
| 129 | 159 | |
| 130 | −/* | |
| 160 | +/** | |
| 131 | 161 | * balloon_page_enqueue - inserts a new page into the balloon page list. |
| 132 | 162 | * |
| 133 | 163 | * @b_dev_info: balloon device descriptor where we will insert a new page |
| @@ -136,22 +166,21 @@ EXPORT_SYMBOL_GPL(balloon_page_alloc); · 2 unmodified lines | ||
| 136 | 166 | * Drivers must call this function to properly enqueue a new allocated balloon |
| 137 | 167 | * page before definitively removing the page from the guest system. |
| 138 | 168 | * |
| 139 | − * Drivers must not call balloon_page_enqueue on pages that have been pushed to | |
| 140 | − * a list with balloon_page_push before removing them with balloon_page_pop. To | |
| 141 | − * enqueue a list of pages, use balloon_page_list_enqueue instead. | |
| 169 | + * Drivers must not enqueue pages while page->lru is still in | |
| 170 | + * use, and must not use page->lru until a page was unqueued again. | |
| 142 | 171 | */ |
| 143 | 172 | void balloon_page_enqueue(struct balloon_dev_info *b_dev_info, |
| 144 | 173 | struct page *page) |
| 145 | 174 | { |
| 146 | 175 | unsigned long flags; |
| 147 | 176 | |
| 148 | − spin_lock_irqsave(&b_dev_info->pages_lock, flags); | |
| 177 | + spin_lock_irqsave(&balloon_pages_lock, flags); | |
| 149 | 178 | balloon_page_enqueue_one(b_dev_info, page); |
| 150 | − spin_unlock_irqrestore(&b_dev_info->pages_lock, flags); | |
| 179 | + spin_unlock_irqrestore(&balloon_pages_lock, flags); | |
| 151 | 180 | } |
| 152 | 181 | EXPORT_SYMBOL_GPL(balloon_page_enqueue); |
| 153 | 182 | |
| 154 | −/* | |
| 183 | +/** | |
| 155 | 184 | * balloon_page_dequeue - removes a page from balloon's page list and returns |
| 156 | 185 | * its address to allow the driver to release the page. |
| 157 | 186 | * @b_dev_info: balloon device descriptor where we will grab a page from. |
| @@ -187,32 +216,42 @@ struct page *balloon_page_dequeue(struct balloon_dev_info *b_dev_info) · 29 unmodified lines | ||
| 187 | 216 | * BUG() here, otherwise the balloon driver may get stuck in |
| 188 | 217 | * an infinite loop while attempting to release all its pages. |
| 189 | 218 | */ |
| 190 | − spin_lock_irqsave(&b_dev_info->pages_lock, flags); | |
| 219 | + spin_lock_irqsave(&balloon_pages_lock, flags); | |
| 191 | 220 | if (unlikely(list_empty(&b_dev_info->pages) && |
| 192 | 221 | !b_dev_info->isolated_pages)) |
| 193 | 222 | BUG(); |
| 194 | − spin_unlock_irqrestore(&b_dev_info->pages_lock, flags); | |
| 223 | + spin_unlock_irqrestore(&balloon_pages_lock, flags); | |
| 195 | 224 | return NULL; |
| 196 | 225 | } |
| 197 | 226 | return list_first_entry(&pages, struct page, lru); |
| 198 | 227 | } |
| 199 | 228 | EXPORT_SYMBOL_GPL(balloon_page_dequeue); |
| 200 | 229 | |
| 201 | −#ifdef CONFIG_BALLOON_COMPACTION | |
| 230 | +#ifdef CONFIG_BALLOON_MIGRATION | |
| 231 | +static struct balloon_dev_info *balloon_page_device(struct page *page) | |
| 232 | +{ | |
| 233 | + return (struct balloon_dev_info *)page_private(page); | |
| 234 | +} | |
| 202 | 235 | |
| 203 | 236 | static bool balloon_page_isolate(struct page *page, isolate_mode_t mode) |
| 204 | 237 | |
| 205 | 238 | { |
| 206 | − struct balloon_dev_info *b_dev_info = balloon_page_device(page); | |
| 239 | + struct balloon_dev_info *b_dev_info; | |
| 207 | 240 | unsigned long flags; |
| 208 | 241 | |
| 209 | − if (!b_dev_info) | |
| 242 | + spin_lock_irqsave(&balloon_pages_lock, flags); | |
| 243 | + b_dev_info = balloon_page_device(page); | |
| 244 | + if (!b_dev_info) { | |
| 245 | + /* | |
| 246 | + * The page already got deflated and removed from the | |
| 247 | + * balloon list. | |
| 248 | + */ | |
| 249 | + spin_unlock_irqrestore(&balloon_pages_lock, flags); | |
| 210 | 250 | return false; |
| 211 | − | |
| 212 | − spin_lock_irqsave(&b_dev_info->pages_lock, flags); | |
| 251 | + } | |
| 213 | 252 | list_del(&page->lru); |
| 214 | 253 | b_dev_info->isolated_pages++; |
| 215 | − spin_unlock_irqrestore(&b_dev_info->pages_lock, flags); | |
| 254 | + spin_unlock_irqrestore(&balloon_pages_lock, flags); | |
| 216 | 255 | |
| 217 | 256 | return true; |
| 218 | 257 | } |
| @@ -222,33 +261,75 @@ static void balloon_page_putback(struct page *page) · 3 unmodified lines | ||
| 222 | 261 | struct balloon_dev_info *b_dev_info = balloon_page_device(page); |
| 223 | 262 | unsigned long flags; |
| 224 | 263 | |
| 225 | − /* Isolated balloon pages cannot get deflated. */ | |
| 264 | + /* | |
| 265 | + * When we isolated the page, the page was still inflated in a balloon | |
| 266 | + * device. As isolated balloon pages cannot get deflated, we still have | |
| 267 | + * a balloon device here. | |
| 268 | + */ | |
| 226 | 269 | if (WARN_ON_ONCE(!b_dev_info)) |
| 227 | 270 | return; |
| 228 | 271 | |
| 229 | − spin_lock_irqsave(&b_dev_info->pages_lock, flags); | |
| 272 | + spin_lock_irqsave(&balloon_pages_lock, flags); | |
| 230 | 273 | list_add(&page->lru, &b_dev_info->pages); |
| 231 | 274 | b_dev_info->isolated_pages--; |
| 232 | − spin_unlock_irqrestore(&b_dev_info->pages_lock, flags); | |
| 275 | + spin_unlock_irqrestore(&balloon_pages_lock, flags); | |
| 233 | 276 | } |
| 234 | 277 | |
| 235 | −/* move_to_new_page() counterpart for a ballooned page */ | |
| 236 | 278 | static int balloon_page_migrate(struct page *newpage, struct page *page, |
| 237 | 279 | enum migrate_mode mode) |
| 238 | 280 | { |
| 239 | − struct balloon_dev_info *balloon = balloon_page_device(page); | |
| 240 | − | |
| 241 | − VM_BUG_ON_PAGE(!PageLocked(page), page); | |
| 242 | − VM_BUG_ON_PAGE(!PageLocked(newpage), newpage); | |
| 281 | + struct balloon_dev_info *b_dev_info = balloon_page_device(page); | |
| 282 | + unsigned long flags; | |
| 283 | + int rc; | |
| 243 | 284 | |
| 244 | − /* Isolated balloon pages cannot get deflated. */ | |
| 245 | − if (WARN_ON_ONCE(!balloon)) | |
| 285 | + /* | |
| 286 | + * When we isolated the page, the page was still inflated in a balloon | |
| 287 | + * device. As isolated balloon pages cannot get deflated, we still have | |
| 288 | + * a balloon device here. | |
| 289 | + */ | |
| 290 | + if (WARN_ON_ONCE(!b_dev_info)) | |
| 246 | 291 | return -EAGAIN; |
| 247 | 292 | |
| 248 | − return balloon->migratepage(balloon, newpage, page, mode); | |
| 293 | + rc = b_dev_info->migratepage(b_dev_info, newpage, page, mode); | |
| 294 | + if (rc < 0 && rc != -ENOENT) | |
| 295 | + return rc; | |
| 296 | + | |
| 297 | + spin_lock_irqsave(&balloon_pages_lock, flags); | |
| 298 | + if (!rc) { | |
| 299 | + /* Insert the new page into the balloon list. */ | |
| 300 | + get_page(newpage); | |
| 301 | + balloon_page_insert(b_dev_info, newpage); | |
| 302 | + __count_vm_event(BALLOON_MIGRATE); | |
| 303 | + | |
| 304 | + if (b_dev_info->adjust_managed_page_count && | |
| 305 | + page_zone(page) != page_zone(newpage)) { | |
| 306 | + /* | |
| 307 | + * When we migrate a page to a different zone we | |
| 308 | + * have to fixup the count of both involved zones. | |
| 309 | + */ | |
| 310 | + adjust_managed_page_count(page, 1); | |
| 311 | + adjust_managed_page_count(newpage, -1); | |
| 312 | + } | |
| 313 | + } else { | |
| 314 | + /* Old page was deflated but new page not inflated. */ | |
| 315 | + __count_vm_event(BALLOON_DEFLATE); | |
| 316 | + | |
| 317 | + if (b_dev_info->adjust_managed_page_count) | |
| 318 | + adjust_managed_page_count(page, 1); | |
| 319 | + } | |
| 320 | + | |
| 321 | + b_dev_info->isolated_pages--; | |
| 322 | + | |
| 323 | + /* Free the now-deflated page we isolated in balloon_page_isolate(). */ | |
| 324 | + balloon_page_finalize(page); | |
| 325 | + spin_unlock_irqrestore(&balloon_pages_lock, flags); | |
| 326 | + | |
| 327 | + put_page(page); | |
| 328 | + | |
| 329 | + return 0; | |
| 249 | 330 | } |
| 250 | 331 | |
| 251 | −const struct movable_operations balloon_mops = { | |
| 332 | +static const struct movable_operations balloon_mops = { | |
| 252 | 333 | .migrate_page = balloon_page_migrate, |
| 253 | 334 | .isolate_page = balloon_page_isolate, |
| 254 | 335 | .putback_page = balloon_page_putback, |
| @@ -260,4 +341,4 @@ static int __init balloon_init(void) · 5 unmodified lines | ||
| 260 | 341 | } |
| 261 | 342 | core_initcall(balloon_init); |
| 262 | 343 | |
| 263 | −#endif /* CONFIG_BALLOON_COMPACTION */ | |
| 344 | +#endif /* CONFIG_BALLOON_MIGRATION */ | |
| @@ -22,6 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <linux/mm.h> |
| 23 | 23 | #include <linux/sizes.h> |
| 24 | 24 | #include <linux/slab.h> |
| 25 | +#include <linux/string.h> | |
| 25 | 26 | #include <linux/string_choices.h> |
| 26 | 27 | #include <linux/log2.h> |
| 27 | 28 | #include <linux/cma.h> |
| @@ -233,7 +234,7 @@ static int __init cma_new_area(const char *name, phys_addr_t size, · 205 unmodified lines | ||
| 233 | 234 | cma_area_count++; |
| 234 | 235 | |
| 235 | 236 | if (name) |
| 236 | − snprintf(cma->name, CMA_MAX_NAME, "%s", name); | |
| 237 | + strscpy(cma->name, name); | |
| 237 | 238 | else |
| 238 | 239 | snprintf(cma->name, CMA_MAX_NAME, "cma%d\n", cma_area_count); |
| 239 | 240 | |
| @@ -836,7 +837,7 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr, · 596 unmodified lines | ||
| 836 | 837 | spin_unlock_irq(&cma->lock); |
| 837 | 838 | |
| 838 | 839 | mutex_lock(&cma->alloc_mutex); |
| 839 | − ret = alloc_contig_range(pfn, pfn + count, ACR_FLAGS_CMA, gfp); | |
| 840 | + ret = alloc_contig_frozen_range(pfn, pfn + count, ACR_FLAGS_CMA, gfp); | |
| 840 | 841 | mutex_unlock(&cma->alloc_mutex); |
| 841 | 842 | if (!ret) |
| 842 | 843 | break; |
| @@ -856,8 +857,8 @@ out: · 13 unmodified lines | ||
| 856 | 857 | return ret; |
| 857 | 858 | } |
| 858 | 859 | |
| 859 | −static struct page *__cma_alloc(struct cma *cma, unsigned long count, | |
| 860 | − unsigned int align, gfp_t gfp) | |
| 860 | +static struct page *__cma_alloc_frozen(struct cma *cma, | |
| 861 | + unsigned long count, unsigned int align, gfp_t gfp) | |
| 861 | 862 | { |
| 862 | 863 | struct page *page = NULL; |
| 863 | 864 | int ret = -ENOMEM, r; |
| @@ -914,6 +915,21 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, · 50 unmodified lines | ||
| 914 | 915 | return page; |
| 915 | 916 | } |
| 916 | 917 | |
| 918 | +struct page *cma_alloc_frozen(struct cma *cma, unsigned long count, | |
| 919 | + unsigned int align, bool no_warn) | |
| 920 | +{ | |
| 921 | + gfp_t gfp = GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0); | |
| 922 | + | |
| 923 | + return __cma_alloc_frozen(cma, count, align, gfp); | |
| 924 | +} | |
| 925 | + | |
| 926 | +struct page *cma_alloc_frozen_compound(struct cma *cma, unsigned int order) | |
| 927 | +{ | |
| 928 | + gfp_t gfp = GFP_KERNEL | __GFP_COMP | __GFP_NOWARN; | |
| 929 | + | |
| 930 | + return __cma_alloc_frozen(cma, 1 << order, order, gfp); | |
| 931 | +} | |
| 932 | + | |
| 917 | 933 | /** |
| 918 | 934 | * cma_alloc() - allocate pages from contiguous area |
| 919 | 935 | * @cma: Contiguous memory region for which the allocation is performed. |
| @@ -926,50 +942,61 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, · 6 unmodified lines | ||
| 926 | 942 | */ |
| 927 | 943 | struct page *cma_alloc(struct cma *cma, unsigned long count, |
| 928 | 944 | unsigned int align, bool no_warn) |
| 929 | −{ | |
| 930 | − return __cma_alloc(cma, count, align, GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0)); | |
| 931 | −} | |
| 932 | − | |
| 933 | −struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) | |
| 934 | 945 | { |
| 935 | 946 | struct page *page; |
| 936 | 947 | |
| 937 | − if (WARN_ON(!order || !(gfp & __GFP_COMP))) | |
| 938 | − return NULL; | |
| 939 | − | |
| 940 | − page = __cma_alloc(cma, 1 << order, order, gfp); | |
| 948 | + page = cma_alloc_frozen(cma, count, align, no_warn); | |
| 949 | + if (page) | |
| 950 | + set_pages_refcounted(page, count); | |
| 941 | 951 | |
| 942 | − return page ? page_folio(page) : NULL; | |
| 952 | + return page; | |
| 943 | 953 | } |
| 944 | 954 | |
| 945 | −bool cma_pages_valid(struct cma *cma, const struct page *pages, | |
| 946 | − unsigned long count) | |
| 955 | +static struct cma_memrange *find_cma_memrange(struct cma *cma, | |
| 956 | + const struct page *pages, unsigned long count) | |
| 947 | 957 | { |
| 948 | − unsigned long pfn, end; | |
| 958 | + struct cma_memrange *cmr = NULL; | |
| 959 | + unsigned long pfn, end_pfn; | |
| 949 | 960 | int r; |
| 950 | − struct cma_memrange *cmr; | |
| 951 | − bool ret; | |
| 961 | + | |
| 962 | + pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count); | |
| 952 | 963 | |
| 953 | 964 | if (!cma || !pages || count > cma->count) |
| 954 | − return false; | |
| 965 | + return NULL; | |
| 955 | 966 | |
| 956 | 967 | pfn = page_to_pfn(pages); |
| 957 | − ret = false; | |
| 958 | 968 | |
| 959 | 969 | for (r = 0; r < cma->nranges; r++) { |
| 960 | 970 | cmr = &cma->ranges[r]; |
| 961 | − end = cmr->base_pfn + cmr->count; | |
| 962 | − if (pfn >= cmr->base_pfn && pfn < end) { | |
| 963 | − ret = pfn + count <= end; | |
| 964 | − break; | |
| 971 | + end_pfn = cmr->base_pfn + cmr->count; | |
| 972 | + if (pfn >= cmr->base_pfn && pfn < end_pfn) { | |
| 973 | + if (pfn + count <= end_pfn) | |
| 974 | + break; | |
| 975 | + | |
| 976 | + VM_WARN_ON_ONCE(1); | |
| 965 | 977 | } |
| 966 | 978 | } |
| 967 | 979 | |
| 968 | − if (!ret) | |
| 969 | − pr_debug("%s(page %p, count %lu)\n", | |
| 970 | − __func__, (void *)pages, count); | |
| 980 | + if (r == cma->nranges) { | |
| 981 | + pr_debug("%s(page %p, count %lu, no cma range matches the page range)\n", | |
| 982 | + __func__, (void *)pages, count); | |
| 983 | + return NULL; | |
| 984 | + } | |
| 971 | 985 | |
| 972 | − return ret; | |
| 986 | + return cmr; | |
| 987 | +} | |
| 988 | + | |
| 989 | +static void __cma_release_frozen(struct cma *cma, struct cma_memrange *cmr, | |
| 990 | + const struct page *pages, unsigned long count) | |
| 991 | +{ | |
| 992 | + unsigned long pfn = page_to_pfn(pages); | |
| 993 | + | |
| 994 | + pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count); | |
| 995 | + | |
| 996 | + free_contig_frozen_range(pfn, count); | |
| 997 | + cma_clear_bitmap(cma, cmr, pfn, count); | |
| 998 | + cma_sysfs_account_release_pages(cma, count); | |
| 999 | + trace_cma_release(cma->name, pfn, pages, count); | |
| 973 | 1000 | } |
| 974 | 1001 | |
| 975 | 1002 | /** |
| @@ -986,43 +1013,33 @@ bool cma_release(struct cma *cma, const struct page *pages, · 10 unmodified lines | ||
| 986 | 1013 | unsigned long count) |
| 987 | 1014 | { |
| 988 | 1015 | struct cma_memrange *cmr; |
| 989 | − unsigned long pfn, end_pfn; | |
| 990 | − int r; | |
| 1016 | + unsigned long i, pfn; | |
| 991 | 1017 | |
| 992 | − pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count); | |
| 993 | − | |
| 994 | − if (!cma_pages_valid(cma, pages, count)) | |
| 1018 | + cmr = find_cma_memrange(cma, pages, count); | |
| 1019 | + if (!cmr) | |
| 995 | 1020 | return false; |
| 996 | 1021 | |
| 997 | 1022 | pfn = page_to_pfn(pages); |
| 998 | − end_pfn = pfn + count; | |
| 1023 | + for (i = 0; i < count; i++, pfn++) | |
| 1024 | + VM_WARN_ON(!put_page_testzero(pfn_to_page(pfn))); | |
| 999 | 1025 | |
| 1000 | − for (r = 0; r < cma->nranges; r++) { | |
| 1001 | − cmr = &cma->ranges[r]; | |
| 1002 | − if (pfn >= cmr->base_pfn && | |
| 1003 | − pfn < (cmr->base_pfn + cmr->count)) { | |
| 1004 | − VM_BUG_ON(end_pfn > cmr->base_pfn + cmr->count); | |
| 1005 | − break; | |
| 1006 | − } | |
| 1007 | − } | |
| 1008 | − | |
| 1009 | − if (r == cma->nranges) | |
| 1010 | − return false; | |
| 1011 | − | |
| 1012 | − free_contig_range(pfn, count); | |
| 1013 | − cma_clear_bitmap(cma, cmr, pfn, count); | |
| 1014 | − cma_sysfs_account_release_pages(cma, count); | |
| 1015 | − trace_cma_release(cma->name, pfn, pages, count); | |
| 1026 | + __cma_release_frozen(cma, cmr, pages, count); | |
| 1016 | 1027 | |
| 1017 | 1028 | return true; |
| 1018 | 1029 | } |
| 1019 | 1030 | |
| 1020 | −bool cma_free_folio(struct cma *cma, const struct folio *folio) | |
| 1031 | +bool cma_release_frozen(struct cma *cma, const struct page *pages, | |
| 1032 | + unsigned long count) | |
| 1021 | 1033 | { |
| 1022 | − if (WARN_ON(!folio_test_large(folio))) | |
| 1034 | + struct cma_memrange *cmr; | |
| 1035 | + | |
| 1036 | + cmr = find_cma_memrange(cma, pages, count); | |
| 1037 | + if (!cmr) | |
| 1023 | 1038 | return false; |
| 1024 | 1039 | |
| 1025 | − return cma_release(cma, &folio->page, folio_nr_pages(folio)); | |
| 1040 | + __cma_release_frozen(cma, cmr, pages, count); | |
| 1041 | + | |
| 1042 | + return true; | |
| 1026 | 1043 | } |
| 1027 | 1044 | |
| 1028 | 1045 | int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data) |
| @@ -157,6 +157,12 @@ void damon_destroy_region(struct damon_region *r, struct damon_target *t) · 156 unmodified lines | ||
| 157 | 157 | damon_free_region(r); |
| 158 | 158 | } |
| 159 | 159 | |
| 160 | +static bool damon_is_last_region(struct damon_region *r, | |
| 161 | + struct damon_target *t) | |
| 162 | +{ | |
| 163 | + return list_is_last(&r->list, &t->regions_list); | |
| 164 | +} | |
| 165 | + | |
| 160 | 166 | /* |
| 161 | 167 | * Check whether a region is intersecting an address range |
| 162 | 168 | * |
| @@ -197,7 +203,7 @@ static int damon_fill_regions_holes(struct damon_region *first, · 34 unmodified lines | ||
| 197 | 203 | * @t: the given target. |
| 198 | 204 | * @ranges: array of new monitoring target ranges. |
| 199 | 205 | * @nr_ranges: length of @ranges. |
| 200 | − * @min_sz_region: minimum region size. | |
| 206 | + * @min_region_sz: minimum region size. | |
| 201 | 207 | * |
| 202 | 208 | * This function adds new regions to, or modify existing regions of a |
| 203 | 209 | * monitoring target to fit in specific ranges. |
| @@ -205,7 +211,7 @@ static int damon_fill_regions_holes(struct damon_region *first, · 1 unmodified line | ||
| 205 | 211 | * Return: 0 if success, or negative error code otherwise. |
| 206 | 212 | */ |
| 207 | 213 | int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, |
| 208 | − unsigned int nr_ranges, unsigned long min_sz_region) | |
| 214 | + unsigned int nr_ranges, unsigned long min_region_sz) | |
| 209 | 215 | { |
| 210 | 216 | struct damon_region *r, *next; |
| 211 | 217 | unsigned int i; |
| @@ -242,16 +248,16 @@ int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, · 30 unmodified lines | ||
| 242 | 248 | /* no region intersects with this range */ |
| 243 | 249 | newr = damon_new_region( |
| 244 | 250 | ALIGN_DOWN(range->start, |
| 245 | − min_sz_region), | |
| 246 | − ALIGN(range->end, min_sz_region)); | |
| 251 | + min_region_sz), | |
| 252 | + ALIGN(range->end, min_region_sz)); | |
| 247 | 253 | if (!newr) |
| 248 | 254 | return -ENOMEM; |
| 249 | 255 | damon_insert_region(newr, damon_prev_region(r), r, t); |
| 250 | 256 | } else { |
| 251 | 257 | /* resize intersecting regions to fit in this range */ |
| 252 | 258 | first->ar.start = ALIGN_DOWN(range->start, |
| 253 | − min_sz_region); | |
| 254 | − last->ar.end = ALIGN(range->end, min_sz_region); | |
| 259 | + min_region_sz); | |
| 260 | + last->ar.end = ALIGN(range->end, min_region_sz); | |
| 255 | 261 | |
| 256 | 262 | /* fill possible holes in the range */ |
| 257 | 263 | err = damon_fill_regions_holes(first, last, t); |
| @@ -278,7 +284,7 @@ struct damos_filter *damos_new_filter(enum damos_filter_type type, · 20 unmodified lines | ||
| 278 | 284 | } |
| 279 | 285 | |
| 280 | 286 | /** |
| 281 | − * damos_filter_for_ops() - Return if the filter is ops-hndled one. | |
| 287 | + * damos_filter_for_ops() - Return if the filter is ops-handled one. | |
| 282 | 288 | * @type: type of the filter. |
| 283 | 289 | * |
| 284 | 290 | * Return: true if the filter of @type needs to be handled by ops layer, false |
| @@ -395,6 +401,7 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 110 unmodified lines | ||
| 395 | 401 | INIT_LIST_HEAD(&scheme->core_filters); |
| 396 | 402 | INIT_LIST_HEAD(&scheme->ops_filters); |
| 397 | 403 | scheme->stat = (struct damos_stat){}; |
| 404 | + scheme->max_nr_snapshots = 0; | |
| 398 | 405 | INIT_LIST_HEAD(&scheme->list); |
| 399 | 406 | |
| 400 | 407 | scheme->quota = *(damos_quota_init(quota)); |
| @@ -546,7 +553,7 @@ struct damon_ctx *damon_new_ctx(void) · 145 unmodified lines | ||
| 546 | 553 | ctx->attrs.max_nr_regions = 1000; |
| 547 | 554 | |
| 548 | 555 | ctx->addr_unit = 1; |
| 549 | − ctx->min_sz_region = DAMON_MIN_REGION; | |
| 556 | + ctx->min_region_sz = DAMON_MIN_REGION_SZ; | |
| 550 | 557 | |
| 551 | 558 | INIT_LIST_HEAD(&ctx->adaptive_targets); |
| 552 | 559 | INIT_LIST_HEAD(&ctx->schemes); |
| @@ -1072,7 +1079,11 @@ static int damos_commit(struct damos *dst, struct damos *src) · 519 unmodified lines | ||
| 1072 | 1079 | return err; |
| 1073 | 1080 | |
| 1074 | 1081 | err = damos_commit_filters(dst, src); |
| 1075 | − return err; | |
| 1082 | + if (err) | |
| 1083 | + return err; | |
| 1084 | + | |
| 1085 | + dst->max_nr_snapshots = src->max_nr_snapshots; | |
| 1086 | + return 0; | |
| 1076 | 1087 | } |
| 1077 | 1088 | |
| 1078 | 1089 | static int damon_commit_schemes(struct damon_ctx *dst, struct damon_ctx *src) |
| @@ -1131,7 +1142,7 @@ static struct damon_target *damon_nth_target(int n, struct damon_ctx *ctx) · 52 unmodified lines | ||
| 1131 | 1142 | * If @src has no region, @dst keeps current regions. |
| 1132 | 1143 | */ |
| 1133 | 1144 | static int damon_commit_target_regions(struct damon_target *dst, |
| 1134 | − struct damon_target *src, unsigned long src_min_sz_region) | |
| 1145 | + struct damon_target *src, unsigned long src_min_region_sz) | |
| 1135 | 1146 | { |
| 1136 | 1147 | struct damon_region *src_region; |
| 1137 | 1148 | struct damon_addr_range *ranges; |
| @@ -1148,7 +1159,7 @@ static int damon_commit_target_regions(struct damon_target *dst, · 10 unmodified lines | ||
| 1148 | 1159 | i = 0; |
| 1149 | 1160 | damon_for_each_region(src_region, src) |
| 1150 | 1161 | ranges[i++] = src_region->ar; |
| 1151 | − err = damon_set_regions(dst, ranges, i, src_min_sz_region); | |
| 1162 | + err = damon_set_regions(dst, ranges, i, src_min_region_sz); | |
| 1152 | 1163 | kfree(ranges); |
| 1153 | 1164 | return err; |
| 1154 | 1165 | } |
| @@ -1156,11 +1167,11 @@ static int damon_commit_target_regions(struct damon_target *dst, · 1 unmodified line | ||
| 1156 | 1167 | static int damon_commit_target( |
| 1157 | 1168 | struct damon_target *dst, bool dst_has_pid, |
| 1158 | 1169 | struct damon_target *src, bool src_has_pid, |
| 1159 | − unsigned long src_min_sz_region) | |
| 1170 | + unsigned long src_min_region_sz) | |
| 1160 | 1171 | { |
| 1161 | 1172 | int err; |
| 1162 | 1173 | |
| 1163 | − err = damon_commit_target_regions(dst, src, src_min_sz_region); | |
| 1174 | + err = damon_commit_target_regions(dst, src, src_min_region_sz); | |
| 1164 | 1175 | if (err) |
| 1165 | 1176 | return err; |
| 1166 | 1177 | if (dst_has_pid) |
| @@ -1187,7 +1198,7 @@ static int damon_commit_targets( · 20 unmodified lines | ||
| 1187 | 1198 | err = damon_commit_target( |
| 1188 | 1199 | dst_target, damon_target_has_pid(dst), |
| 1189 | 1200 | src_target, damon_target_has_pid(src), |
| 1190 | − src->min_sz_region); | |
| 1201 | + src->min_region_sz); | |
| 1191 | 1202 | if (err) |
| 1192 | 1203 | return err; |
| 1193 | 1204 | } else { |
| @@ -1214,7 +1225,7 @@ static int damon_commit_targets( · 20 unmodified lines | ||
| 1214 | 1225 | return -ENOMEM; |
| 1215 | 1226 | err = damon_commit_target(new_target, false, |
| 1216 | 1227 | src_target, damon_target_has_pid(src), |
| 1217 | − src->min_sz_region); | |
| 1228 | + src->min_region_sz); | |
| 1218 | 1229 | if (err) { |
| 1219 | 1230 | damon_destroy_target(new_target, NULL); |
| 1220 | 1231 | return err; |
| @@ -1261,7 +1272,7 @@ int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src) · 40 unmodified lines | ||
| 1261 | 1272 | } |
| 1262 | 1273 | dst->ops = src->ops; |
| 1263 | 1274 | dst->addr_unit = src->addr_unit; |
| 1264 | − dst->min_sz_region = src->min_sz_region; | |
| 1275 | + dst->min_region_sz = src->min_region_sz; | |
| 1265 | 1276 | |
| 1266 | 1277 | return 0; |
| 1267 | 1278 | } |
| @@ -1294,8 +1305,8 @@ static unsigned long damon_region_sz_limit(struct damon_ctx *ctx) · 26 unmodified lines | ||
| 1294 | 1305 | |
| 1295 | 1306 | if (ctx->attrs.min_nr_regions) |
| 1296 | 1307 | sz /= ctx->attrs.min_nr_regions; |
| 1297 | − if (sz < ctx->min_sz_region) | |
| 1298 | − sz = ctx->min_sz_region; | |
| 1308 | + if (sz < ctx->min_region_sz) | |
| 1309 | + sz = ctx->min_region_sz; | |
| 1299 | 1310 | |
| 1300 | 1311 | return sz; |
| 1301 | 1312 | } |
| @@ -1431,6 +1442,23 @@ bool damon_is_running(struct damon_ctx *ctx) · 129 unmodified lines | ||
| 1431 | 1442 | return running; |
| 1432 | 1443 | } |
| 1433 | 1444 | |
| 1445 | +/** | |
| 1446 | + * damon_kdamond_pid() - Return pid of a given DAMON context's worker thread. | |
| 1447 | + * @ctx: The DAMON context of the question. | |
| 1448 | + * | |
| 1449 | + * Return: pid if @ctx is running, negative error code otherwise. | |
| 1450 | + */ | |
| 1451 | +int damon_kdamond_pid(struct damon_ctx *ctx) | |
| 1452 | +{ | |
| 1453 | + int pid = -EINVAL; | |
| 1454 | + | |
| 1455 | + mutex_lock(&ctx->kdamond_lock); | |
| 1456 | + if (ctx->kdamond) | |
| 1457 | + pid = ctx->kdamond->pid; | |
| 1458 | + mutex_unlock(&ctx->kdamond_lock); | |
| 1459 | + return pid; | |
| 1460 | +} | |
| 1461 | + | |
| 1434 | 1462 | /* |
| 1435 | 1463 | * damon_call_handle_inactive_ctx() - handle DAMON call request that added to |
| 1436 | 1464 | * an inactive context. |
| @@ -1604,7 +1632,7 @@ static unsigned long damon_get_intervals_adaptation_bp(struct damon_ctx *c) · 167 unmodified lines | ||
| 1604 | 1632 | adaptation_bp = damon_feed_loop_next_input(100000000, score_bp) / |
| 1605 | 1633 | 10000; |
| 1606 | 1634 | /* |
| 1607 | − * adaptaion_bp ranges from 1 to 20,000. Avoid too rapid reduction of | |
| 1635 | + * adaptation_bp ranges from 1 to 20,000. Avoid too rapid reduction of | |
| 1608 | 1636 | * the intervals by rescaling [1,10,000] to [5000, 10,000]. |
| 1609 | 1637 | */ |
| 1610 | 1638 | if (adaptation_bp <= 10000) |
| @@ -1668,7 +1696,7 @@ static bool damos_valid_target(struct damon_ctx *c, struct damon_target *t, · 57 unmodified lines | ||
| 1668 | 1696 | * @t: The target of the region. |
| 1669 | 1697 | * @rp: The pointer to the region. |
| 1670 | 1698 | * @s: The scheme to be applied. |
| 1671 | − * @min_sz_region: minimum region size. | |
| 1699 | + * @min_region_sz: minimum region size. | |
| 1672 | 1700 | * |
| 1673 | 1701 | * If a quota of a scheme has exceeded in a quota charge window, the scheme's |
| 1674 | 1702 | * action would applied to only a part of the target access pattern fulfilling |
| @@ -1686,7 +1714,8 @@ static bool damos_valid_target(struct damon_ctx *c, struct damon_target *t, · 11 unmodified lines | ||
| 1686 | 1714 | * Return: true if the region should be entirely skipped, false otherwise. |
| 1687 | 1715 | */ |
| 1688 | 1716 | static bool damos_skip_charged_region(struct damon_target *t, |
| 1689 | − struct damon_region **rp, struct damos *s, unsigned long min_sz_region) | |
| 1717 | + struct damon_region **rp, struct damos *s, | |
| 1718 | + unsigned long min_region_sz) | |
| 1690 | 1719 | { |
| 1691 | 1720 | struct damon_region *r = *rp; |
| 1692 | 1721 | struct damos_quota *quota = &s->quota; |
| @@ -1708,11 +1737,11 @@ static bool damos_skip_charged_region(struct damon_target *t, · 15 unmodified lines | ||
| 1708 | 1737 | if (quota->charge_addr_from && r->ar.start < |
| 1709 | 1738 | quota->charge_addr_from) { |
| 1710 | 1739 | sz_to_skip = ALIGN_DOWN(quota->charge_addr_from - |
| 1711 | − r->ar.start, min_sz_region); | |
| 1740 | + r->ar.start, min_region_sz); | |
| 1712 | 1741 | if (!sz_to_skip) { |
| 1713 | − if (damon_sz_region(r) <= min_sz_region) | |
| 1742 | + if (damon_sz_region(r) <= min_region_sz) | |
| 1714 | 1743 | return true; |
| 1715 | − sz_to_skip = min_sz_region; | |
| 1744 | + sz_to_skip = min_region_sz; | |
| 1716 | 1745 | } |
| 1717 | 1746 | damon_split_region_at(t, r, sz_to_skip); |
| 1718 | 1747 | r = damon_next_region(r); |
| @@ -1738,7 +1767,7 @@ static void damos_update_stat(struct damos *s, · 19 unmodified lines | ||
| 1738 | 1767 | |
| 1739 | 1768 | static bool damos_filter_match(struct damon_ctx *ctx, struct damon_target *t, |
| 1740 | 1769 | struct damon_region *r, struct damos_filter *filter, |
| 1741 | − unsigned long min_sz_region) | |
| 1770 | + unsigned long min_region_sz) | |
| 1742 | 1771 | { |
| 1743 | 1772 | bool matched = false; |
| 1744 | 1773 | struct damon_target *ti; |
| @@ -1755,8 +1784,8 @@ static bool damos_filter_match(struct damon_ctx *ctx, struct damon_target *t, · 10 unmodified lines | ||
| 1755 | 1784 | matched = target_idx == filter->target_idx; |
| 1756 | 1785 | break; |
| 1757 | 1786 | case DAMOS_FILTER_TYPE_ADDR: |
| 1758 | − start = ALIGN_DOWN(filter->addr_range.start, min_sz_region); | |
| 1759 | − end = ALIGN_DOWN(filter->addr_range.end, min_sz_region); | |
| 1787 | + start = ALIGN_DOWN(filter->addr_range.start, min_region_sz); | |
| 1788 | + end = ALIGN_DOWN(filter->addr_range.end, min_region_sz); | |
| 1760 | 1789 | |
| 1761 | 1790 | /* inside the range */ |
| 1762 | 1791 | if (start <= r->ar.start && r->ar.end <= end) { |
| @@ -1785,14 +1814,14 @@ static bool damos_filter_match(struct damon_ctx *ctx, struct damon_target *t, · 22 unmodified lines | ||
| 1785 | 1814 | return matched == filter->matching; |
| 1786 | 1815 | } |
| 1787 | 1816 | |
| 1788 | −static bool damos_filter_out(struct damon_ctx *ctx, struct damon_target *t, | |
| 1817 | +static bool damos_core_filter_out(struct damon_ctx *ctx, struct damon_target *t, | |
| 1789 | 1818 | struct damon_region *r, struct damos *s) |
| 1790 | 1819 | { |
| 1791 | 1820 | struct damos_filter *filter; |
| 1792 | 1821 | |
| 1793 | 1822 | s->core_filters_allowed = false; |
| 1794 | 1823 | damos_for_each_core_filter(filter, s) { |
| 1795 | − if (damos_filter_match(ctx, t, r, filter, ctx->min_sz_region)) { | |
| 1824 | + if (damos_filter_match(ctx, t, r, filter, ctx->min_region_sz)) { | |
| 1796 | 1825 | if (filter->allow) |
| 1797 | 1826 | s->core_filters_allowed = true; |
| 1798 | 1827 | return !filter->allow; |
| @@ -1927,12 +1956,12 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 128 unmodified lines | ||
| 1927 | 1956 | if (c->ops.apply_scheme) { |
| 1928 | 1957 | if (quota->esz && quota->charged_sz + sz > quota->esz) { |
| 1929 | 1958 | sz = ALIGN_DOWN(quota->esz - quota->charged_sz, |
| 1930 | − c->min_sz_region); | |
| 1959 | + c->min_region_sz); | |
| 1931 | 1960 | if (!sz) |
| 1932 | 1961 | goto update_stat; |
| 1933 | 1962 | damon_split_region_at(t, r, sz); |
| 1934 | 1963 | } |
| 1935 | − if (damos_filter_out(c, t, r, s)) | |
| 1964 | + if (damos_core_filter_out(c, t, r, s)) | |
| 1936 | 1965 | return; |
| 1937 | 1966 | ktime_get_coarse_ts64(&begin); |
| 1938 | 1967 | trace_damos_before_apply(cidx, sidx, tidx, r, |
| @@ -1975,13 +2004,18 @@ static void damon_do_apply_schemes(struct damon_ctx *c, · 36 unmodified lines | ||
| 1975 | 2004 | if (quota->esz && quota->charged_sz >= quota->esz) |
| 1976 | 2005 | continue; |
| 1977 | 2006 | |
| 1978 | − if (damos_skip_charged_region(t, &r, s, c->min_sz_region)) | |
| 2007 | + if (damos_skip_charged_region(t, &r, s, c->min_region_sz)) | |
| 1979 | 2008 | continue; |
| 1980 | 2009 | |
| 1981 | − if (!damos_valid_target(c, t, r, s)) | |
| 2010 | + if (s->max_nr_snapshots && | |
| 2011 | + s->max_nr_snapshots <= s->stat.nr_snapshots) | |
| 1982 | 2012 | continue; |
| 1983 | 2013 | |
| 1984 | − damos_apply_scheme(c, t, r, s); | |
| 2014 | + if (damos_valid_target(c, t, r, s)) | |
| 2015 | + damos_apply_scheme(c, t, r, s); | |
| 2016 | + | |
| 2017 | + if (damon_is_last_region(r, t)) | |
| 2018 | + s->stat.nr_snapshots++; | |
| 1985 | 2019 | } |
| 1986 | 2020 | } |
| 1987 | 2021 | |
| @@ -2078,16 +2112,13 @@ static unsigned long damos_get_node_memcg_used_bp( · 90 unmodified lines | ||
| 2078 | 2112 | unsigned long used_pages, numerator; |
| 2079 | 2113 | struct sysinfo i; |
| 2080 | 2114 | |
| 2081 | − rcu_read_lock(); | |
| 2082 | − memcg = mem_cgroup_from_id(goal->memcg_id); | |
| 2083 | − if (!memcg || !mem_cgroup_tryget(memcg)) { | |
| 2084 | − rcu_read_unlock(); | |
| 2115 | + memcg = mem_cgroup_get_from_id(goal->memcg_id); | |
| 2116 | + if (!memcg) { | |
| 2085 | 2117 | if (goal->metric == DAMOS_QUOTA_NODE_MEMCG_USED_BP) |
| 2086 | 2118 | return 0; |
| 2087 | 2119 | else /* DAMOS_QUOTA_NODE_MEMCG_FREE_BP */ |
| 2088 | 2120 | return 10000; |
| 2089 | 2121 | } |
| 2090 | − rcu_read_unlock(); | |
| 2091 | 2122 | |
| 2092 | 2123 | mem_cgroup_flush_stats(memcg); |
| 2093 | 2124 | lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(goal->nid)); |
| @@ -2119,6 +2150,23 @@ static unsigned long damos_get_node_memcg_used_bp( · 25 unmodified lines | ||
| 2119 | 2150 | } |
| 2120 | 2151 | #endif |
| 2121 | 2152 | |
| 2153 | +/* | |
| 2154 | + * Returns LRU-active or inactive memory to total LRU memory size ratio. | |
| 2155 | + */ | |
| 2156 | +static unsigned int damos_get_in_active_mem_bp(bool active_ratio) | |
| 2157 | +{ | |
| 2158 | + unsigned long active, inactive, total; | |
| 2159 | + | |
| 2160 | + /* This should align with /proc/meminfo output */ | |
| 2161 | + active = global_node_page_state(NR_LRU_BASE + LRU_ACTIVE_ANON) + | |
| 2162 | + global_node_page_state(NR_LRU_BASE + LRU_ACTIVE_FILE); | |
| 2163 | + inactive = global_node_page_state(NR_LRU_BASE + LRU_INACTIVE_ANON) + | |
| 2164 | + global_node_page_state(NR_LRU_BASE + LRU_INACTIVE_FILE); | |
| 2165 | + total = active + inactive; | |
| 2166 | + if (active_ratio) | |
| 2167 | + return active * 10000 / total; | |
| 2168 | + return inactive * 10000 / total; | |
| 2169 | +} | |
| 2122 | 2170 | |
| 2123 | 2171 | static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal) |
| 2124 | 2172 | { |
| @@ -2141,6 +2189,11 @@ static void damos_set_quota_goal_current_value(struct damos_quota_goal *goal) · 16 unmodified lines | ||
| 2141 | 2189 | case DAMOS_QUOTA_NODE_MEMCG_FREE_BP: |
| 2142 | 2190 | goal->current_value = damos_get_node_memcg_used_bp(goal); |
| 2143 | 2191 | break; |
| 2192 | + case DAMOS_QUOTA_ACTIVE_MEM_BP: | |
| 2193 | + case DAMOS_QUOTA_INACTIVE_MEM_BP: | |
| 2194 | + goal->current_value = damos_get_in_active_mem_bp( | |
| 2195 | + goal->metric == DAMOS_QUOTA_ACTIVE_MEM_BP); | |
| 2196 | + break; | |
| 2144 | 2197 | default: |
| 2145 | 2198 | break; |
| 2146 | 2199 | } |
| @@ -2273,6 +2326,22 @@ static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) · 126 unmodified lines | ||
| 2273 | 2326 | quota->min_score = score; |
| 2274 | 2327 | } |
| 2275 | 2328 | |
| 2329 | +static void damos_trace_stat(struct damon_ctx *c, struct damos *s) | |
| 2330 | +{ | |
| 2331 | + unsigned int cidx = 0, sidx = 0; | |
| 2332 | + struct damos *siter; | |
| 2333 | + | |
| 2334 | + if (!trace_damos_stat_after_apply_interval_enabled()) | |
| 2335 | + return; | |
| 2336 | + | |
| 2337 | + damon_for_each_scheme(siter, c) { | |
| 2338 | + if (siter == s) | |
| 2339 | + break; | |
| 2340 | + sidx++; | |
| 2341 | + } | |
| 2342 | + trace_damos_stat_after_apply_interval(cidx, sidx, &s->stat); | |
| 2343 | +} | |
| 2344 | + | |
| 2276 | 2345 | static void kdamond_apply_schemes(struct damon_ctx *c) |
| 2277 | 2346 | { |
| 2278 | 2347 | struct damon_target *t; |
| @@ -2299,6 +2368,9 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 20 unmodified lines | ||
| 2299 | 2368 | |
| 2300 | 2369 | mutex_lock(&c->walk_control_lock); |
| 2301 | 2370 | damon_for_each_target(t, c) { |
| 2371 | + if (c->ops.target_valid && c->ops.target_valid(t) == false) | |
| 2372 | + continue; | |
| 2373 | + | |
| 2302 | 2374 | damon_for_each_region_safe(r, next_r, t) |
| 2303 | 2375 | damon_do_apply_schemes(c, t, r); |
| 2304 | 2376 | } |
| @@ -2311,6 +2383,7 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 6 unmodified lines | ||
| 2311 | 2383 | (s->apply_interval_us ? s->apply_interval_us : |
| 2312 | 2384 | c->attrs.aggr_interval) / sample_interval; |
| 2313 | 2385 | s->last_applied = NULL; |
| 2386 | + damos_trace_stat(c, s); | |
| 2314 | 2387 | } |
| 2315 | 2388 | mutex_unlock(&c->walk_control_lock); |
| 2316 | 2389 | } |
| @@ -2424,7 +2497,7 @@ static void damon_split_region_at(struct damon_target *t, · 107 unmodified lines | ||
| 2424 | 2497 | |
| 2425 | 2498 | /* Split every region in the given target into 'nr_subs' regions */ |
| 2426 | 2499 | static void damon_split_regions_of(struct damon_target *t, int nr_subs, |
| 2427 | − unsigned long min_sz_region) | |
| 2500 | + unsigned long min_region_sz) | |
| 2428 | 2501 | { |
| 2429 | 2502 | struct damon_region *r, *next; |
| 2430 | 2503 | unsigned long sz_region, sz_sub = 0; |
| @@ -2434,13 +2507,13 @@ static void damon_split_regions_of(struct damon_target *t, int nr_subs, · 3 unmodified lines | ||
| 2434 | 2507 | sz_region = damon_sz_region(r); |
| 2435 | 2508 | |
| 2436 | 2509 | for (i = 0; i < nr_subs - 1 && |
| 2437 | − sz_region > 2 * min_sz_region; i++) { | |
| 2510 | + sz_region > 2 * min_region_sz; i++) { | |
| 2438 | 2511 | /* |
| 2439 | 2512 | * Randomly select size of left sub-region to be at |
| 2440 | 2513 | * least 10 percent and at most 90% of original region |
| 2441 | 2514 | */ |
| 2442 | 2515 | sz_sub = ALIGN_DOWN(damon_rand(1, 10) * |
| 2443 | − sz_region / 10, min_sz_region); | |
| 2516 | + sz_region / 10, min_region_sz); | |
| 2444 | 2517 | /* Do not allow blank region */ |
| 2445 | 2518 | if (sz_sub == 0 || sz_sub >= sz_region) |
| 2446 | 2519 | continue; |
| @@ -2480,7 +2553,7 @@ static void kdamond_split_regions(struct damon_ctx *ctx) · 33 unmodified lines | ||
| 2480 | 2553 | nr_subregions = 3; |
| 2481 | 2554 | |
| 2482 | 2555 | damon_for_each_target(t, ctx) |
| 2483 | − damon_split_regions_of(t, nr_subregions, ctx->min_sz_region); | |
| 2556 | + damon_split_regions_of(t, nr_subregions, ctx->min_region_sz); | |
| 2484 | 2557 | |
| 2485 | 2558 | last_nr_regions = nr_regions; |
| 2486 | 2559 | } |
| @@ -2577,41 +2650,30 @@ static void kdamond_usleep(unsigned long usecs) · 90 unmodified lines | ||
| 2577 | 2650 | */ |
| 2578 | 2651 | static void kdamond_call(struct damon_ctx *ctx, bool cancel) |
| 2579 | 2652 | { |
| 2580 | − struct damon_call_control *control; | |
| 2581 | − LIST_HEAD(repeat_controls); | |
| 2582 | − int ret = 0; | |
| 2653 | + struct damon_call_control *control, *next; | |
| 2654 | + LIST_HEAD(controls); | |
| 2583 | 2655 | |
| 2584 | − while (true) { | |
| 2585 | − mutex_lock(&ctx->call_controls_lock); | |
| 2586 | − control = list_first_entry_or_null(&ctx->call_controls, | |
| 2587 | − struct damon_call_control, list); | |
| 2588 | − mutex_unlock(&ctx->call_controls_lock); | |
| 2589 | − if (!control) | |
| 2590 | − break; | |
| 2591 | − if (cancel) { | |
| 2656 | + mutex_lock(&ctx->call_controls_lock); | |
| 2657 | + list_splice_tail_init(&ctx->call_controls, &controls); | |
| 2658 | + mutex_unlock(&ctx->call_controls_lock); | |
| 2659 | + | |
| 2660 | + list_for_each_entry_safe(control, next, &controls, list) { | |
| 2661 | + if (!control->repeat || cancel) | |
| 2662 | + list_del(&control->list); | |
| 2663 | + | |
| 2664 | + if (cancel) | |
| 2592 | 2665 | control->canceled = true; |
| 2593 | − } else { | |
| 2594 | − ret = control->fn(control->data); | |
| 2595 | − control->return_code = ret; | |
| 2596 | − } | |
| 2597 | − mutex_lock(&ctx->call_controls_lock); | |
| 2598 | − list_del(&control->list); | |
| 2599 | − mutex_unlock(&ctx->call_controls_lock); | |
| 2600 | − if (!control->repeat) { | |
| 2666 | + else | |
| 2667 | + control->return_code = control->fn(control->data); | |
| 2668 | + | |
| 2669 | + if (!control->repeat) | |
| 2601 | 2670 | complete(&control->completion); |
| 2602 | − } else if (control->canceled && control->dealloc_on_cancel) { | |
| 2671 | + else if (control->canceled && control->dealloc_on_cancel) | |
| 2603 | 2672 | kfree(control); |
| 2604 | − continue; | |
| 2605 | − } else { | |
| 2606 | − list_add(&control->list, &repeat_controls); | |
| 2607 | − } | |
| 2608 | 2673 | } |
| 2609 | − control = list_first_entry_or_null(&repeat_controls, | |
| 2610 | − struct damon_call_control, list); | |
| 2611 | − if (!control || cancel) | |
| 2612 | − return; | |
| 2674 | + | |
| 2613 | 2675 | mutex_lock(&ctx->call_controls_lock); |
| 2614 | − list_add_tail(&control->list, &ctx->call_controls); | |
| 2676 | + list_splice_tail(&controls, &ctx->call_controls); | |
| 2615 | 2677 | mutex_unlock(&ctx->call_controls_lock); |
| 2616 | 2678 | } |
| 2617 | 2679 | |
| @@ -2670,8 +2732,6 @@ static void kdamond_init_ctx(struct damon_ctx *ctx) · 52 unmodified lines | ||
| 2670 | 2732 | static int kdamond_fn(void *data) |
| 2671 | 2733 | { |
| 2672 | 2734 | struct damon_ctx *ctx = data; |
| 2673 | − struct damon_target *t; | |
| 2674 | − struct damon_region *r, *next; | |
| 2675 | 2735 | unsigned int max_nr_accesses = 0; |
| 2676 | 2736 | unsigned long sz_limit = 0; |
| 2677 | 2737 | |
| @@ -2747,7 +2807,7 @@ static int kdamond_fn(void *data) · 69 unmodified lines | ||
| 2747 | 2807 | * |
| 2748 | 2808 | * Reset ->next_aggregation_sis to avoid that. |
| 2749 | 2809 | * It will anyway correctly updated after this |
| 2750 | − * if caluse. | |
| 2810 | + * if clause. | |
| 2751 | 2811 | */ |
| 2752 | 2812 | ctx->next_aggregation_sis = |
| 2753 | 2813 | next_aggregation_sis; |
| @@ -2776,47 +2836,29 @@ static int kdamond_fn(void *data) · 22 unmodified lines | ||
| 2776 | 2836 | } |
| 2777 | 2837 | } |
| 2778 | 2838 | done: |
| 2779 | − damon_for_each_target(t, ctx) { | |
| 2780 | − damon_for_each_region_safe(r, next, t) | |
| 2781 | − damon_destroy_region(r, t); | |
| 2782 | − } | |
| 2839 | + damon_destroy_targets(ctx); | |
| 2783 | 2840 | |
| 2784 | − if (ctx->ops.cleanup) | |
| 2785 | − ctx->ops.cleanup(ctx); | |
| 2786 | 2841 | kfree(ctx->regions_score_histogram); |
| 2787 | 2842 | kdamond_call(ctx, true); |
| 2843 | + damos_walk_cancel(ctx); | |
| 2788 | 2844 | |
| 2789 | 2845 | pr_debug("kdamond (%d) finishes\n", current->pid); |
| 2790 | 2846 | mutex_lock(&ctx->kdamond_lock); |
| 2791 | 2847 | ctx->kdamond = NULL; |
| 2792 | 2848 | mutex_unlock(&ctx->kdamond_lock); |
| 2793 | 2849 | |
| 2794 | − damos_walk_cancel(ctx); | |
| 2795 | − | |
| 2796 | 2850 | mutex_lock(&damon_lock); |
| 2797 | 2851 | nr_running_ctxs--; |
| 2798 | 2852 | if (!nr_running_ctxs && running_exclusive_ctxs) |
| 2799 | 2853 | running_exclusive_ctxs = false; |
| 2800 | 2854 | mutex_unlock(&damon_lock); |
| 2801 | 2855 | |
| 2802 | − damon_destroy_targets(ctx); | |
| 2803 | 2856 | return 0; |
| 2804 | 2857 | } |
| 2805 | 2858 | |
| 2806 | −/* | |
| 2807 | − * struct damon_system_ram_region - System RAM resource address region of | |
| 2808 | − * [@start, @end). | |
| 2809 | − * @start: Start address of the region (inclusive). | |
| 2810 | − * @end: End address of the region (exclusive). | |
| 2811 | − */ | |
| 2812 | −struct damon_system_ram_region { | |
| 2813 | − unsigned long start; | |
| 2814 | − unsigned long end; | |
| 2815 | −}; | |
| 2816 | − | |
| 2817 | 2859 | static int walk_system_ram(struct resource *res, void *arg) |
| 2818 | 2860 | { |
| 2819 | − struct damon_system_ram_region *a = arg; | |
| 2861 | + struct damon_addr_range *a = arg; | |
| 2820 | 2862 | |
| 2821 | 2863 | if (a->end - a->start < resource_size(res)) { |
| 2822 | 2864 | a->start = res->start; |
| @@ -2833,7 +2875,7 @@ static bool damon_find_biggest_system_ram(unsigned long *start, · 10 unmodified lines | ||
| 2833 | 2875 | unsigned long *end) |
| 2834 | 2876 | |
| 2835 | 2877 | { |
| 2836 | − struct damon_system_ram_region arg = {}; | |
| 2878 | + struct damon_addr_range arg = {}; | |
| 2837 | 2879 | |
| 2838 | 2880 | walk_system_ram_res(0, ULONG_MAX, &arg, walk_system_ram); |
| 2839 | 2881 | if (arg.end <= arg.start) |
| @@ -2850,7 +2892,7 @@ static bool damon_find_biggest_system_ram(unsigned long *start, · 10 unmodified lines | ||
| 2850 | 2892 | * @t: The monitoring target to set the region. |
| 2851 | 2893 | * @start: The pointer to the start address of the region. |
| 2852 | 2894 | * @end: The pointer to the end address of the region. |
| 2853 | − * @min_sz_region: Minimum region size. | |
| 2895 | + * @min_region_sz: Minimum region size. | |
| 2854 | 2896 | * |
| 2855 | 2897 | * This function sets the region of @t as requested by @start and @end. If the |
| 2856 | 2898 | * values of @start and @end are zero, however, this function finds the biggest |
| @@ -2862,7 +2904,7 @@ static bool damon_find_biggest_system_ram(unsigned long *start, · 5 unmodified lines | ||
| 2862 | 2904 | */ |
| 2863 | 2905 | int damon_set_region_biggest_system_ram_default(struct damon_target *t, |
| 2864 | 2906 | unsigned long *start, unsigned long *end, |
| 2865 | − unsigned long min_sz_region) | |
| 2907 | + unsigned long min_region_sz) | |
| 2866 | 2908 | { |
| 2867 | 2909 | struct damon_addr_range addr_range; |
| 2868 | 2910 | |
| @@ -2875,7 +2917,7 @@ int damon_set_region_biggest_system_ram_default(struct damon_target *t, · 6 unmodified lines | ||
| 2875 | 2917 | |
| 2876 | 2918 | addr_range.start = *start; |
| 2877 | 2919 | addr_range.end = *end; |
| 2878 | − return damon_set_regions(t, &addr_range, 1, min_sz_region); | |
| 2920 | + return damon_set_regions(t, &addr_range, 1, min_region_sz); | |
| 2879 | 2921 | } |
| 2880 | 2922 | |
| 2881 | 2923 | /* |
| @@ -34,13 +34,56 @@ static bool enabled __read_mostly; · 33 unmodified lines | ||
| 34 | 34 | * |
| 35 | 35 | * Input parameters that updated while DAMON_LRU_SORT is running are not |
| 36 | 36 | * applied by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT |
| 37 | − * reads values of parametrs except ``enabled`` again. Once the re-reading is | |
| 37 | + * reads values of parameters except ``enabled`` again. Once the re-reading is | |
| 38 | 38 | * done, this parameter is set as ``N``. If invalid parameters are found while |
| 39 | 39 | * the re-reading, DAMON_LRU_SORT will be disabled. |
| 40 | 40 | */ |
| 41 | 41 | static bool commit_inputs __read_mostly; |
| 42 | 42 | module_param(commit_inputs, bool, 0600); |
| 43 | 43 | |
| 44 | +/* | |
| 45 | + * Desired active to [in]active memory ratio in bp (1/10,000). | |
| 46 | + * | |
| 47 | + * While keeping the caps that set by other quotas, DAMON_LRU_SORT | |
| 48 | + * automatically increases and decreases the effective level of the quota | |
| 49 | + * aiming the LRU [de]prioritizations of the hot and cold memory resulting in | |
| 50 | + * this active to [in]active memory ratio. Value zero means disabling this | |
| 51 | + * auto-tuning feature. | |
| 52 | + * | |
| 53 | + * Disabled by default. | |
| 54 | + */ | |
| 55 | +static unsigned long active_mem_bp __read_mostly; | |
| 56 | +module_param(active_mem_bp, ulong, 0600); | |
| 57 | + | |
| 58 | +/* | |
| 59 | + * Auto-tune monitoring intervals. | |
| 60 | + * | |
| 61 | + * If this parameter is set as ``Y``, DAMON_LRU_SORT automatically tunes | |
| 62 | + * DAMON's sampling and aggregation intervals. The auto-tuning aims to capture | |
| 63 | + * meaningful amount of access events in each DAMON-snapshot, while keeping the | |
| 64 | + * sampling interval 5 milliseconds in minimum, and 10 seconds in maximum. | |
| 65 | + * Setting this as ``N`` disables the auto-tuning. | |
| 66 | + * | |
| 67 | + * Disabled by default. | |
| 68 | + */ | |
| 69 | +static bool autotune_monitoring_intervals __read_mostly; | |
| 70 | +module_param(autotune_monitoring_intervals, bool, 0600); | |
| 71 | + | |
| 72 | +/* | |
| 73 | + * Filter [non-]young pages accordingly for LRU [de]prioritizations. | |
| 74 | + * | |
| 75 | + * If this is set, check page level access (youngness) once again before each | |
| 76 | + * LRU [de]prioritization operation. LRU prioritization operation is skipped | |
| 77 | + * if the page has not accessed since the last check (not young). LRU | |
| 78 | + * deprioritization operation is skipped if the page has accessed since the | |
| 79 | + * last check (young). The feature is enabled or disabled if this parameter is | |
| 80 | + * set as ``Y`` or ``N``, respectively. | |
| 81 | + * | |
| 82 | + * Disabled by default. | |
| 83 | + */ | |
| 84 | +static bool filter_young_pages __read_mostly; | |
| 85 | +module_param(filter_young_pages, bool, 0600); | |
| 86 | + | |
| 44 | 87 | /* |
| 45 | 88 | * Access frequency threshold for hot memory regions identification in permil. |
| 46 | 89 | * |
| @@ -71,7 +114,7 @@ static struct damos_quota damon_lru_sort_quota = { · 24 unmodified lines | ||
| 71 | 114 | /* Within the quota, mark hotter regions accessed first. */ |
| 72 | 115 | .weight_sz = 0, |
| 73 | 116 | .weight_nr_accesses = 1, |
| 74 | − .weight_age = 0, | |
| 117 | + .weight_age = 1, | |
| 75 | 118 | }; |
| 76 | 119 | DEFINE_DAMON_MODULES_DAMOS_TIME_QUOTA(damon_lru_sort_quota); |
| 77 | 120 | |
| @@ -193,10 +236,53 @@ static struct damos *damon_lru_sort_new_cold_scheme(unsigned int cold_thres) · 115 unmodified lines | ||
| 193 | 236 | return damon_lru_sort_new_scheme(&pattern, DAMOS_LRU_DEPRIO); |
| 194 | 237 | } |
| 195 | 238 | |
| 239 | +static int damon_lru_sort_add_quota_goals(struct damos *hot_scheme, | |
| 240 | + struct damos *cold_scheme) | |
| 241 | +{ | |
| 242 | + struct damos_quota_goal *goal; | |
| 243 | + | |
| 244 | + if (!active_mem_bp) | |
| 245 | + return 0; | |
| 246 | + goal = damos_new_quota_goal(DAMOS_QUOTA_ACTIVE_MEM_BP, active_mem_bp); | |
| 247 | + if (!goal) | |
| 248 | + return -ENOMEM; | |
| 249 | + damos_add_quota_goal(&hot_scheme->quota, goal); | |
| 250 | + /* aim 0.2 % goal conflict, to keep little ping pong */ | |
| 251 | + goal = damos_new_quota_goal(DAMOS_QUOTA_INACTIVE_MEM_BP, | |
| 252 | + 10000 - active_mem_bp + 2); | |
| 253 | + if (!goal) | |
| 254 | + return -ENOMEM; | |
| 255 | + damos_add_quota_goal(&cold_scheme->quota, goal); | |
| 256 | + return 0; | |
| 257 | +} | |
| 258 | + | |
| 259 | +static int damon_lru_sort_add_filters(struct damos *hot_scheme, | |
| 260 | + struct damos *cold_scheme) | |
| 261 | +{ | |
| 262 | + struct damos_filter *filter; | |
| 263 | + | |
| 264 | + if (!filter_young_pages) | |
| 265 | + return 0; | |
| 266 | + | |
| 267 | + /* disallow prioritizing not-young pages */ | |
| 268 | + filter = damos_new_filter(DAMOS_FILTER_TYPE_YOUNG, false, false); | |
| 269 | + if (!filter) | |
| 270 | + return -ENOMEM; | |
| 271 | + damos_add_filter(hot_scheme, filter); | |
| 272 | + | |
| 273 | + /* disabllow de-prioritizing young pages */ | |
| 274 | + filter = damos_new_filter(DAMOS_FILTER_TYPE_YOUNG, true, false); | |
| 275 | + if (!filter) | |
| 276 | + return -ENOMEM; | |
| 277 | + damos_add_filter(cold_scheme, filter); | |
| 278 | + return 0; | |
| 279 | +} | |
| 280 | + | |
| 196 | 281 | static int damon_lru_sort_apply_parameters(void) |
| 197 | 282 | { |
| 198 | 283 | struct damon_ctx *param_ctx; |
| 199 | 284 | struct damon_target *param_target; |
| 285 | + struct damon_attrs attrs; | |
| 200 | 286 | struct damos *hot_scheme, *cold_scheme; |
| 201 | 287 | unsigned int hot_thres, cold_thres; |
| 202 | 288 | int err; |
| @@ -212,25 +298,34 @@ static int damon_lru_sort_apply_parameters(void) · 9 unmodified lines | ||
| 212 | 298 | if (!monitor_region_start && !monitor_region_end) |
| 213 | 299 | addr_unit = 1; |
| 214 | 300 | param_ctx->addr_unit = addr_unit; |
| 215 | − param_ctx->min_sz_region = max(DAMON_MIN_REGION / addr_unit, 1); | |
| 301 | + param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1); | |
| 216 | 302 | |
| 217 | 303 | if (!damon_lru_sort_mon_attrs.sample_interval) { |
| 218 | 304 | err = -EINVAL; |
| 219 | 305 | goto out; |
| 220 | 306 | } |
| 221 | 307 | |
| 222 | − err = damon_set_attrs(param_ctx, &damon_lru_sort_mon_attrs); | |
| 308 | + attrs = damon_lru_sort_mon_attrs; | |
| 309 | + if (autotune_monitoring_intervals) { | |
| 310 | + attrs.sample_interval = 5000; | |
| 311 | + attrs.aggr_interval = 100000; | |
| 312 | + attrs.intervals_goal.access_bp = 40; | |
| 313 | + attrs.intervals_goal.aggrs = 3; | |
| 314 | + attrs.intervals_goal.min_sample_us = 5000; | |
| 315 | + attrs.intervals_goal.max_sample_us = 10 * 1000 * 1000; | |
| 316 | + } | |
| 317 | + err = damon_set_attrs(param_ctx, &attrs); | |
| 223 | 318 | if (err) |
| 224 | 319 | goto out; |
| 225 | 320 | |
| 226 | 321 | err = -ENOMEM; |
| 227 | − hot_thres = damon_max_nr_accesses(&damon_lru_sort_mon_attrs) * | |
| 322 | + hot_thres = damon_max_nr_accesses(&attrs) * | |
| 228 | 323 | hot_thres_access_freq / 1000; |
| 229 | 324 | hot_scheme = damon_lru_sort_new_hot_scheme(hot_thres); |
| 230 | 325 | if (!hot_scheme) |
| 231 | 326 | goto out; |
| 232 | 327 | |
| 233 | − cold_thres = cold_min_age / damon_lru_sort_mon_attrs.aggr_interval; | |
| 328 | + cold_thres = cold_min_age / attrs.aggr_interval; | |
| 234 | 329 | cold_scheme = damon_lru_sort_new_cold_scheme(cold_thres); |
| 235 | 330 | if (!cold_scheme) { |
| 236 | 331 | damon_destroy_scheme(hot_scheme); |
| @@ -240,10 +335,17 @@ static int damon_lru_sort_apply_parameters(void) · 3 unmodified lines | ||
| 240 | 335 | damon_set_schemes(param_ctx, &hot_scheme, 1); |
| 241 | 336 | damon_add_scheme(param_ctx, cold_scheme); |
| 242 | 337 | |
| 338 | + err = damon_lru_sort_add_quota_goals(hot_scheme, cold_scheme); | |
| 339 | + if (err) | |
| 340 | + goto out; | |
| 341 | + err = damon_lru_sort_add_filters(hot_scheme, cold_scheme); | |
| 342 | + if (err) | |
| 343 | + goto out; | |
| 344 | + | |
| 243 | 345 | err = damon_set_region_biggest_system_ram_default(param_target, |
| 244 | 346 | &monitor_region_start, |
| 245 | 347 | &monitor_region_end, |
| 246 | − param_ctx->min_sz_region); | |
| 348 | + param_ctx->min_region_sz); | |
| 247 | 349 | if (err) |
| 248 | 350 | goto out; |
| 249 | 351 | err = damon_commit_ctx(ctx, param_ctx); |
| @@ -303,7 +405,9 @@ static int damon_lru_sort_turn(bool on) · 53 unmodified lines | ||
| 303 | 405 | err = damon_start(&ctx, 1, true); |
| 304 | 406 | if (err) |
| 305 | 407 | return err; |
| 306 | − kdamond_pid = ctx->kdamond->pid; | |
| 408 | + kdamond_pid = damon_kdamond_pid(ctx); | |
| 409 | + if (kdamond_pid < 0) | |
| 410 | + return kdamond_pid; | |
| 307 | 411 | return damon_call(ctx, &call_control); |
| 308 | 412 | } |
| 309 | 413 | |
| @@ -156,7 +156,7 @@ static unsigned long damon_pa_pageout(struct damon_region *r, · 155 unmodified lines | ||
| 156 | 156 | LIST_HEAD(folio_list); |
| 157 | 157 | bool install_young_filter = true; |
| 158 | 158 | struct damos_filter *filter; |
| 159 | − struct folio *folio; | |
| 159 | + struct folio *folio = NULL; | |
| 160 | 160 | |
| 161 | 161 | /* check access in page level again by default */ |
| 162 | 162 | damos_for_each_ops_filter(filter, s) { |
| @@ -206,13 +206,13 @@ put_folio: · 43 unmodified lines | ||
| 206 | 206 | return damon_pa_core_addr(applied * PAGE_SIZE, addr_unit); |
| 207 | 207 | } |
| 208 | 208 | |
| 209 | −static inline unsigned long damon_pa_mark_accessed_or_deactivate( | |
| 209 | +static inline unsigned long damon_pa_de_activate( | |
| 210 | 210 | struct damon_region *r, unsigned long addr_unit, |
| 211 | − struct damos *s, bool mark_accessed, | |
| 211 | + struct damos *s, bool activate, | |
| 212 | 212 | unsigned long *sz_filter_passed) |
| 213 | 213 | { |
| 214 | 214 | phys_addr_t addr, applied = 0; |
| 215 | − struct folio *folio; | |
| 215 | + struct folio *folio = NULL; | |
| 216 | 216 | |
| 217 | 217 | addr = damon_pa_phys_addr(r->ar.start, addr_unit); |
| 218 | 218 | while (addr < damon_pa_phys_addr(r->ar.end, addr_unit)) { |
| @@ -227,8 +227,8 @@ static inline unsigned long damon_pa_mark_accessed_or_deactivate( · 8 unmodified lines | ||
| 227 | 227 | else |
| 228 | 228 | *sz_filter_passed += folio_size(folio) / addr_unit; |
| 229 | 229 | |
| 230 | − if (mark_accessed) | |
| 231 | − folio_mark_accessed(folio); | |
| 230 | + if (activate) | |
| 231 | + folio_activate(folio); | |
| 232 | 232 | else |
| 233 | 233 | folio_deactivate(folio); |
| 234 | 234 | applied += folio_nr_pages(folio); |
| @@ -240,20 +240,18 @@ put_folio: · 5 unmodified lines | ||
| 240 | 240 | return damon_pa_core_addr(applied * PAGE_SIZE, addr_unit); |
| 241 | 241 | } |
| 242 | 242 | |
| 243 | −static unsigned long damon_pa_mark_accessed(struct damon_region *r, | |
| 243 | +static unsigned long damon_pa_activate_pages(struct damon_region *r, | |
| 244 | 244 | unsigned long addr_unit, struct damos *s, |
| 245 | 245 | unsigned long *sz_filter_passed) |
| 246 | 246 | { |
| 247 | − return damon_pa_mark_accessed_or_deactivate(r, addr_unit, s, true, | |
| 248 | − sz_filter_passed); | |
| 247 | + return damon_pa_de_activate(r, addr_unit, s, true, sz_filter_passed); | |
| 249 | 248 | } |
| 250 | 249 | |
| 251 | 250 | static unsigned long damon_pa_deactivate_pages(struct damon_region *r, |
| 252 | 251 | unsigned long addr_unit, struct damos *s, |
| 253 | 252 | unsigned long *sz_filter_passed) |
| 254 | 253 | { |
| 255 | − return damon_pa_mark_accessed_or_deactivate(r, addr_unit, s, false, | |
| 256 | − sz_filter_passed); | |
| 254 | + return damon_pa_de_activate(r, addr_unit, s, false, sz_filter_passed); | |
| 257 | 255 | } |
| 258 | 256 | |
| 259 | 257 | static unsigned long damon_pa_migrate(struct damon_region *r, |
| @@ -262,7 +260,7 @@ static unsigned long damon_pa_migrate(struct damon_region *r, · 2 unmodified lines | ||
| 262 | 260 | { |
| 263 | 261 | phys_addr_t addr, applied; |
| 264 | 262 | LIST_HEAD(folio_list); |
| 265 | − struct folio *folio; | |
| 263 | + struct folio *folio = NULL; | |
| 266 | 264 | |
| 267 | 265 | addr = damon_pa_phys_addr(r->ar.start, addr_unit); |
| 268 | 266 | while (addr < damon_pa_phys_addr(r->ar.end, addr_unit)) { |
| @@ -295,7 +293,7 @@ static unsigned long damon_pa_stat(struct damon_region *r, · 26 unmodified lines | ||
| 295 | 293 | unsigned long *sz_filter_passed) |
| 296 | 294 | { |
| 297 | 295 | phys_addr_t addr; |
| 298 | − struct folio *folio; | |
| 296 | + struct folio *folio = NULL; | |
| 299 | 297 | |
| 300 | 298 | if (!damos_ops_has_filter(s)) |
| 301 | 299 | return 0; |
| @@ -327,7 +325,7 @@ static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx, · 25 unmodified lines | ||
| 327 | 325 | case DAMOS_PAGEOUT: |
| 328 | 326 | return damon_pa_pageout(r, aunit, scheme, sz_filter_passed); |
| 329 | 327 | case DAMOS_LRU_PRIO: |
| 330 | − return damon_pa_mark_accessed(r, aunit, scheme, | |
| 328 | + return damon_pa_activate_pages(r, aunit, scheme, | |
| 331 | 329 | sz_filter_passed); |
| 332 | 330 | case DAMOS_LRU_DEPRIO: |
| 333 | 331 | return damon_pa_deactivate_pages(r, aunit, scheme, |
| @@ -375,7 +373,6 @@ static int __init damon_pa_initcall(void) · 41 unmodified lines | ||
| 375 | 373 | .prepare_access_checks = damon_pa_prepare_access_checks, |
| 376 | 374 | .check_accesses = damon_pa_check_accesses, |
| 377 | 375 | .target_valid = NULL, |
| 378 | − .cleanup = NULL, | |
| 379 | 376 | .apply_scheme = damon_pa_apply_scheme, |
| 380 | 377 | .get_scheme_score = damon_pa_scheme_score, |
| 381 | 378 | }; |
| @@ -34,7 +34,7 @@ static bool enabled __read_mostly; · 33 unmodified lines | ||
| 34 | 34 | * |
| 35 | 35 | * Input parameters that updated while DAMON_RECLAIM is running are not applied |
| 36 | 36 | * by default. Once this parameter is set as ``Y``, DAMON_RECLAIM reads values |
| 37 | − * of parametrs except ``enabled`` again. Once the re-reading is done, this | |
| 37 | + * of parameters except ``enabled`` again. Once the re-reading is done, this | |
| 38 | 38 | * parameter is set as ``N``. If invalid parameters are found while the |
| 39 | 39 | * re-reading, DAMON_RECLAIM will be disabled. |
| 40 | 40 | */ |
| @@ -208,7 +208,7 @@ static int damon_reclaim_apply_parameters(void) · 167 unmodified lines | ||
| 208 | 208 | if (!monitor_region_start && !monitor_region_end) |
| 209 | 209 | addr_unit = 1; |
| 210 | 210 | param_ctx->addr_unit = addr_unit; |
| 211 | − param_ctx->min_sz_region = max(DAMON_MIN_REGION / addr_unit, 1); | |
| 211 | + param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1); | |
| 212 | 212 | |
| 213 | 213 | if (!damon_reclaim_mon_attrs.aggr_interval) { |
| 214 | 214 | err = -EINVAL; |
| @@ -251,7 +251,7 @@ static int damon_reclaim_apply_parameters(void) · 36 unmodified lines | ||
| 251 | 251 | err = damon_set_region_biggest_system_ram_default(param_target, |
| 252 | 252 | &monitor_region_start, |
| 253 | 253 | &monitor_region_end, |
| 254 | − param_ctx->min_sz_region); | |
| 254 | + param_ctx->min_region_sz); | |
| 255 | 255 | if (err) |
| 256 | 256 | goto out; |
| 257 | 257 | err = damon_commit_ctx(ctx, param_ctx); |
| @@ -307,7 +307,9 @@ static int damon_reclaim_turn(bool on) · 49 unmodified lines | ||
| 307 | 307 | err = damon_start(&ctx, 1, true); |
| 308 | 308 | if (err) |
| 309 | 309 | return err; |
| 310 | − kdamond_pid = ctx->kdamond->pid; | |
| 310 | + kdamond_pid = damon_kdamond_pid(ctx); | |
| 311 | + if (kdamond_pid < 0) | |
| 312 | + return kdamond_pid; | |
| 311 | 313 | return damon_call(ctx, &call_control); |
| 312 | 314 | } |
| 313 | 315 | |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | /* |
| 3 | − * Shows data access monitoring resutls in simple metrics. | |
| 3 | + * Shows data access monitoring results in simple metrics. | |
| 4 | 4 | */ |
| 5 | 5 | |
| 6 | 6 | #define pr_fmt(fmt) "damon-stat: " fmt |
| @@ -34,7 +34,7 @@ module_param(estimated_memory_bandwidth, ulong, 0400); · 27 unmodified lines | ||
| 34 | 34 | MODULE_PARM_DESC(estimated_memory_bandwidth, |
| 35 | 35 | "Estimated memory bandwidth usage in bytes per second"); |
| 36 | 36 | |
| 37 | −static long memory_idle_ms_percentiles[101] __read_mostly = {0,}; | |
| 37 | +static long memory_idle_ms_percentiles[101] = {0,}; | |
| 38 | 38 | module_param_array(memory_idle_ms_percentiles, long, NULL, 0400); |
| 39 | 39 | MODULE_PARM_DESC(memory_idle_ms_percentiles, |
| 40 | 40 | "Memory idle time percentiles in milliseconds"); |
| @@ -173,14 +173,6 @@ static struct damon_ctx *damon_stat_build_ctx(void) · 132 unmodified lines | ||
| 173 | 173 | if (damon_set_attrs(ctx, &attrs)) |
| 174 | 174 | goto free_out; |
| 175 | 175 | |
| 176 | − /* | |
| 177 | − * auto-tune sampling and aggregation interval aiming 4% DAMON-observed | |
| 178 | − * accesses ratio, keeping sampling interval in [5ms, 10s] range. | |
| 179 | − */ | |
| 180 | − ctx->attrs.intervals_goal = (struct damon_intervals_goal) { | |
| 181 | − .access_bp = 400, .aggrs = 3, | |
| 182 | − .min_sample_us = 5000, .max_sample_us = 10000000, | |
| 183 | − }; | |
| 184 | 176 | if (damon_select_ops(ctx, DAMON_OPS_PADDR)) |
| 185 | 177 | goto free_out; |
| 186 | 178 | |
| @@ -189,7 +181,7 @@ static struct damon_ctx *damon_stat_build_ctx(void) · 2 unmodified lines | ||
| 189 | 181 | goto free_out; |
| 190 | 182 | damon_add_target(ctx, target); |
| 191 | 183 | if (damon_set_region_biggest_system_ram_default(target, &start, &end, |
| 192 | − ctx->min_sz_region)) | |
| 184 | + ctx->min_region_sz)) | |
| 193 | 185 | goto free_out; |
| 194 | 186 | return ctx; |
| 195 | 187 | free_out: |
| @@ -204,6 +204,8 @@ struct damon_sysfs_stats { · 203 unmodified lines | ||
| 204 | 204 | unsigned long sz_applied; |
| 205 | 205 | unsigned long sz_ops_filter_passed; |
| 206 | 206 | unsigned long qt_exceeds; |
| 207 | + unsigned long nr_snapshots; | |
| 208 | + unsigned long max_nr_snapshots; | |
| 207 | 209 | }; |
| 208 | 210 | |
| 209 | 211 | static struct damon_sysfs_stats *damon_sysfs_stats_alloc(void) |
| @@ -265,6 +267,37 @@ static ssize_t qt_exceeds_show(struct kobject *kobj, · 55 unmodified lines | ||
| 265 | 267 | return sysfs_emit(buf, "%lu\n", stats->qt_exceeds); |
| 266 | 268 | } |
| 267 | 269 | |
| 270 | +static ssize_t nr_snapshots_show(struct kobject *kobj, | |
| 271 | + struct kobj_attribute *attr, char *buf) | |
| 272 | +{ | |
| 273 | + struct damon_sysfs_stats *stats = container_of(kobj, | |
| 274 | + struct damon_sysfs_stats, kobj); | |
| 275 | + | |
| 276 | + return sysfs_emit(buf, "%lu\n", stats->nr_snapshots); | |
| 277 | +} | |
| 278 | + | |
| 279 | +static ssize_t max_nr_snapshots_show(struct kobject *kobj, | |
| 280 | + struct kobj_attribute *attr, char *buf) | |
| 281 | +{ | |
| 282 | + struct damon_sysfs_stats *stats = container_of(kobj, | |
| 283 | + struct damon_sysfs_stats, kobj); | |
| 284 | + | |
| 285 | + return sysfs_emit(buf, "%lu\n", stats->max_nr_snapshots); | |
| 286 | +} | |
| 287 | + | |
| 288 | +static ssize_t max_nr_snapshots_store(struct kobject *kobj, | |
| 289 | + struct kobj_attribute *attr, const char *buf, size_t count) | |
| 290 | +{ | |
| 291 | + struct damon_sysfs_stats *stats = container_of(kobj, | |
| 292 | + struct damon_sysfs_stats, kobj); | |
| 293 | + unsigned long max_nr_snapshots, err = kstrtoul(buf, 0, &max_nr_snapshots); | |
| 294 | + | |
| 295 | + if (err) | |
| 296 | + return err; | |
| 297 | + stats->max_nr_snapshots = max_nr_snapshots; | |
| 298 | + return count; | |
| 299 | +} | |
| 300 | + | |
| 268 | 301 | static void damon_sysfs_stats_release(struct kobject *kobj) |
| 269 | 302 | { |
| 270 | 303 | kfree(container_of(kobj, struct damon_sysfs_stats, kobj)); |
| @@ -288,6 +321,12 @@ static struct kobj_attribute damon_sysfs_stats_sz_ops_filter_passed_attr = · 17 unmodified lines | ||
| 288 | 321 | static struct kobj_attribute damon_sysfs_stats_qt_exceeds_attr = |
| 289 | 322 | __ATTR_RO_MODE(qt_exceeds, 0400); |
| 290 | 323 | |
| 324 | +static struct kobj_attribute damon_sysfs_stats_nr_snapshots_attr = | |
| 325 | + __ATTR_RO_MODE(nr_snapshots, 0400); | |
| 326 | + | |
| 327 | +static struct kobj_attribute damon_sysfs_stats_max_nr_snapshots_attr = | |
| 328 | + __ATTR_RW_MODE(max_nr_snapshots, 0600); | |
| 329 | + | |
| 291 | 330 | static struct attribute *damon_sysfs_stats_attrs[] = { |
| 292 | 331 | &damon_sysfs_stats_nr_tried_attr.attr, |
| 293 | 332 | &damon_sysfs_stats_sz_tried_attr.attr, |
| @@ -295,6 +334,8 @@ static struct attribute *damon_sysfs_stats_attrs[] = { · 1 unmodified line | ||
| 295 | 334 | &damon_sysfs_stats_sz_applied_attr.attr, |
| 296 | 335 | &damon_sysfs_stats_sz_ops_filter_passed_attr.attr, |
| 297 | 336 | &damon_sysfs_stats_qt_exceeds_attr.attr, |
| 337 | + &damon_sysfs_stats_nr_snapshots_attr.attr, | |
| 338 | + &damon_sysfs_stats_max_nr_snapshots_attr.attr, | |
| 298 | 339 | NULL, |
| 299 | 340 | }; |
| 300 | 341 | ATTRIBUTE_GROUPS(damon_sysfs_stats); |
| @@ -1038,6 +1079,14 @@ struct damos_sysfs_qgoal_metric_name damos_sysfs_qgoal_metric_names[] = { · 737 unmodified lines | ||
| 1038 | 1079 | .metric = DAMOS_QUOTA_NODE_MEMCG_FREE_BP, |
| 1039 | 1080 | .name = "node_memcg_free_bp", |
| 1040 | 1081 | }, |
| 1082 | + { | |
| 1083 | + .metric = DAMOS_QUOTA_ACTIVE_MEM_BP, | |
| 1084 | + .name = "active_mem_bp", | |
| 1085 | + }, | |
| 1086 | + { | |
| 1087 | + .metric = DAMOS_QUOTA_INACTIVE_MEM_BP, | |
| 1088 | + .name = "inactive_mem_bp", | |
| 1089 | + }, | |
| 1041 | 1090 | }; |
| 1042 | 1091 | |
| 1043 | 1092 | static ssize_t target_metric_show(struct kobject *kobj, |
| @@ -2288,7 +2337,6 @@ static ssize_t target_nid_store(struct kobject *kobj, · 1244 unmodified lines | ||
| 2288 | 2337 | struct damon_sysfs_scheme, kobj); |
| 2289 | 2338 | int err = 0; |
| 2290 | 2339 | |
| 2291 | − /* TODO: error handling for target_nid range. */ | |
| 2292 | 2340 | err = kstrtoint(buf, 0, &scheme->target_nid); |
| 2293 | 2341 | |
| 2294 | 2342 | return err ? err : count; |
| @@ -2454,7 +2502,7 @@ static bool damon_sysfs_memcg_path_eq(struct mem_cgroup *memcg, · 159 unmodified lines | ||
| 2454 | 2502 | return false; |
| 2455 | 2503 | } |
| 2456 | 2504 | |
| 2457 | −static int damon_sysfs_memcg_path_to_id(char *memcg_path, unsigned short *id) | |
| 2505 | +static int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id) | |
| 2458 | 2506 | { |
| 2459 | 2507 | struct mem_cgroup *memcg; |
| 2460 | 2508 | char *path; |
| @@ -2469,8 +2517,8 @@ static int damon_sysfs_memcg_path_to_id(char *memcg_path, unsigned short *id) · 8 unmodified lines | ||
| 2469 | 2517 | |
| 2470 | 2518 | for (memcg = mem_cgroup_iter(NULL, NULL, NULL); memcg; |
| 2471 | 2519 | memcg = mem_cgroup_iter(NULL, memcg, NULL)) { |
| 2472 | − /* skip removed memcg */ | |
| 2473 | − if (!mem_cgroup_id(memcg)) | |
| 2520 | + /* skip offlined memcg */ | |
| 2521 | + if (!mem_cgroup_online(memcg)) | |
| 2474 | 2522 | continue; |
| 2475 | 2523 | if (damon_sysfs_memcg_path_eq(memcg, path, memcg_path)) { |
| 2476 | 2524 | *id = mem_cgroup_id(memcg); |
| @@ -2719,6 +2767,7 @@ static struct damos *damon_sysfs_mk_scheme( · 242 unmodified lines | ||
| 2719 | 2767 | damon_destroy_scheme(scheme); |
| 2720 | 2768 | return NULL; |
| 2721 | 2769 | } |
| 2770 | + scheme->max_nr_snapshots = sysfs_scheme->stats->max_nr_snapshots; | |
| 2722 | 2771 | return scheme; |
| 2723 | 2772 | } |
| 2724 | 2773 | |
| @@ -2763,6 +2812,7 @@ void damon_sysfs_schemes_update_stats( · 38 unmodified lines | ||
| 2763 | 2812 | sysfs_stats->sz_ops_filter_passed = |
| 2764 | 2813 | scheme->stat.sz_ops_filter_passed; |
| 2765 | 2814 | sysfs_stats->qt_exceeds = scheme->stat.qt_exceeds; |
| 2815 | + sysfs_stats->nr_snapshots = scheme->stat.nr_snapshots; | |
| 2766 | 2816 | } |
| 2767 | 2817 | } |
| 2768 | 2818 | |
| @@ -1365,7 +1365,7 @@ static int damon_sysfs_set_attrs(struct damon_ctx *ctx, · 1364 unmodified lines | ||
| 1365 | 1365 | |
| 1366 | 1366 | static int damon_sysfs_set_regions(struct damon_target *t, |
| 1367 | 1367 | struct damon_sysfs_regions *sysfs_regions, |
| 1368 | − unsigned long min_sz_region) | |
| 1368 | + unsigned long min_region_sz) | |
| 1369 | 1369 | { |
| 1370 | 1370 | struct damon_addr_range *ranges = kmalloc_array(sysfs_regions->nr, |
| 1371 | 1371 | sizeof(*ranges), GFP_KERNEL | __GFP_NOWARN); |
| @@ -1387,7 +1387,7 @@ static int damon_sysfs_set_regions(struct damon_target *t, · 15 unmodified lines | ||
| 1387 | 1387 | if (ranges[i - 1].end > ranges[i].start) |
| 1388 | 1388 | goto out; |
| 1389 | 1389 | } |
| 1390 | − err = damon_set_regions(t, ranges, sysfs_regions->nr, min_sz_region); | |
| 1390 | + err = damon_set_regions(t, ranges, sysfs_regions->nr, min_region_sz); | |
| 1391 | 1391 | out: |
| 1392 | 1392 | kfree(ranges); |
| 1393 | 1393 | return err; |
| @@ -1409,7 +1409,8 @@ static int damon_sysfs_add_target(struct damon_sysfs_target *sys_target, · 15 unmodified lines | ||
| 1409 | 1409 | return -EINVAL; |
| 1410 | 1410 | } |
| 1411 | 1411 | t->obsolete = sys_target->obsolete; |
| 1412 | − return damon_sysfs_set_regions(t, sys_target->regions, ctx->min_sz_region); | |
| 1412 | + return damon_sysfs_set_regions(t, sys_target->regions, | |
| 1413 | + ctx->min_region_sz); | |
| 1413 | 1414 | } |
| 1414 | 1415 | |
| 1415 | 1416 | static int damon_sysfs_add_targets(struct damon_ctx *ctx, |
| @@ -1469,8 +1470,8 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx, · 53 unmodified lines | ||
| 1469 | 1470 | ctx->addr_unit = sys_ctx->addr_unit; |
| 1470 | 1471 | /* addr_unit is respected by only DAMON_OPS_PADDR */ |
| 1471 | 1472 | if (sys_ctx->ops_id == DAMON_OPS_PADDR) |
| 1472 | − ctx->min_sz_region = max( | |
| 1473 | − DAMON_MIN_REGION / sys_ctx->addr_unit, 1); | |
| 1473 | + ctx->min_region_sz = max( | |
| 1474 | + DAMON_MIN_REGION_SZ / sys_ctx->addr_unit, 1); | |
| 1474 | 1475 | err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs); |
| 1475 | 1476 | if (err) |
| 1476 | 1477 | return err; |
| @@ -1819,10 +1820,9 @@ static ssize_t pid_show(struct kobject *kobj, · 342 unmodified lines | ||
| 1819 | 1820 | if (!ctx) |
| 1820 | 1821 | goto out; |
| 1821 | 1822 | |
| 1822 | − mutex_lock(&ctx->kdamond_lock); | |
| 1823 | − if (ctx->kdamond) | |
| 1824 | − pid = ctx->kdamond->pid; | |
| 1825 | − mutex_unlock(&ctx->kdamond_lock); | |
| 1823 | + pid = damon_kdamond_pid(ctx); | |
| 1824 | + if (pid < 0) | |
| 1825 | + pid = -1; | |
| 1826 | 1826 | out: |
| 1827 | 1827 | mutex_unlock(&damon_sysfs_lock); |
| 1828 | 1828 | return sysfs_emit(buf, "%d\n", pid); |
| @@ -158,6 +158,7 @@ static void damon_test_split_at(struct kunit *test) · 157 unmodified lines | ||
| 158 | 158 | r->nr_accesses_bp = 420000; |
| 159 | 159 | r->nr_accesses = 42; |
| 160 | 160 | r->last_nr_accesses = 15; |
| 161 | + r->age = 10; | |
| 161 | 162 | damon_add_region(r, t); |
| 162 | 163 | damon_split_region_at(t, r, 25); |
| 163 | 164 | KUNIT_EXPECT_EQ(test, r->ar.start, 0ul); |
| @@ -170,6 +171,7 @@ static void damon_test_split_at(struct kunit *test) · 6 unmodified lines | ||
| 170 | 171 | KUNIT_EXPECT_EQ(test, r->nr_accesses_bp, r_new->nr_accesses_bp); |
| 171 | 172 | KUNIT_EXPECT_EQ(test, r->nr_accesses, r_new->nr_accesses); |
| 172 | 173 | KUNIT_EXPECT_EQ(test, r->last_nr_accesses, r_new->last_nr_accesses); |
| 174 | + KUNIT_EXPECT_EQ(test, r->age, r_new->age); | |
| 173 | 175 | |
| 174 | 176 | damon_free_target(t); |
| 175 | 177 | } |
| @@ -190,6 +192,7 @@ static void damon_test_merge_two(struct kunit *test) · 14 unmodified lines | ||
| 190 | 192 | } |
| 191 | 193 | r->nr_accesses = 10; |
| 192 | 194 | r->nr_accesses_bp = 100000; |
| 195 | + r->age = 9; | |
| 193 | 196 | damon_add_region(r, t); |
| 194 | 197 | r2 = damon_new_region(100, 300); |
| 195 | 198 | if (!r2) { |
| @@ -198,12 +201,15 @@ static void damon_test_merge_two(struct kunit *test) · 2 unmodified lines | ||
| 198 | 201 | } |
| 199 | 202 | r2->nr_accesses = 20; |
| 200 | 203 | r2->nr_accesses_bp = 200000; |
| 204 | + r2->age = 21; | |
| 201 | 205 | damon_add_region(r2, t); |
| 202 | 206 | |
| 203 | 207 | damon_merge_two_regions(t, r, r2); |
| 204 | 208 | KUNIT_EXPECT_EQ(test, r->ar.start, 0ul); |
| 205 | 209 | KUNIT_EXPECT_EQ(test, r->ar.end, 300ul); |
| 206 | 210 | KUNIT_EXPECT_EQ(test, r->nr_accesses, 16u); |
| 211 | + KUNIT_EXPECT_EQ(test, r->nr_accesses_bp, 160000u); | |
| 212 | + KUNIT_EXPECT_EQ(test, r->age, 17u); | |
| 207 | 213 | |
| 208 | 214 | i = 0; |
| 209 | 215 | damon_for_each_region(r3, t) { |
| @@ -232,12 +238,12 @@ static void damon_test_merge_regions_of(struct kunit *test) · 22 unmodified lines | ||
| 232 | 238 | { |
| 233 | 239 | struct damon_target *t; |
| 234 | 240 | struct damon_region *r; |
| 235 | − unsigned long sa[] = {0, 100, 114, 122, 130, 156, 170, 184}; | |
| 236 | − unsigned long ea[] = {100, 112, 122, 130, 156, 170, 184, 230}; | |
| 237 | − unsigned int nrs[] = {0, 0, 10, 10, 20, 30, 1, 2}; | |
| 241 | + unsigned long sa[] = {0, 100, 114, 122, 130, 156, 170, 184, 230}; | |
| 242 | + unsigned long ea[] = {100, 112, 122, 130, 156, 170, 184, 230, 10170}; | |
| 243 | + unsigned int nrs[] = {0, 0, 10, 10, 20, 30, 1, 2, 5}; | |
| 238 | 244 | |
| 239 | − unsigned long saddrs[] = {0, 114, 130, 156, 170}; | |
| 240 | − unsigned long eaddrs[] = {112, 130, 156, 170, 230}; | |
| 245 | + unsigned long saddrs[] = {0, 114, 130, 156, 170, 230}; | |
| 246 | + unsigned long eaddrs[] = {112, 130, 156, 170, 230, 10170}; | |
| 241 | 247 | int i; |
| 242 | 248 | |
| 243 | 249 | t = damon_new_target(); |
| @@ -255,9 +261,9 @@ static void damon_test_merge_regions_of(struct kunit *test) · 11 unmodified lines | ||
| 255 | 261 | } |
| 256 | 262 | |
| 257 | 263 | damon_merge_regions_of(t, 9, 9999); |
| 258 | − /* 0-112, 114-130, 130-156, 156-170 */ | |
| 259 | − KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 5u); | |
| 260 | − for (i = 0; i < 5; i++) { | |
| 264 | + /* 0-112, 114-130, 130-156, 156-170, 170-230, 230-10170 */ | |
| 265 | + KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 6u); | |
| 266 | + for (i = 0; i < 6; i++) { | |
| 261 | 267 | r = __nth_region_of(t, i); |
| 262 | 268 | KUNIT_EXPECT_EQ(test, r->ar.start, saddrs[i]); |
| 263 | 269 | KUNIT_EXPECT_EQ(test, r->ar.end, eaddrs[i]); |
| @@ -269,6 +275,9 @@ static void damon_test_split_regions_of(struct kunit *test) · 5 unmodified lines | ||
| 269 | 275 | { |
| 270 | 276 | struct damon_target *t; |
| 271 | 277 | struct damon_region *r; |
| 278 | + unsigned long sa[] = {0, 300, 500}; | |
| 279 | + unsigned long ea[] = {220, 400, 700}; | |
| 280 | + int i; | |
| 272 | 281 | |
| 273 | 282 | t = damon_new_target(); |
| 274 | 283 | if (!t) |
| @@ -295,6 +304,23 @@ static void damon_test_split_regions_of(struct kunit *test) · 20 unmodified lines | ||
| 295 | 304 | damon_split_regions_of(t, 4, 1); |
| 296 | 305 | KUNIT_EXPECT_LE(test, damon_nr_regions(t), 4u); |
| 297 | 306 | damon_free_target(t); |
| 307 | + | |
| 308 | + t = damon_new_target(); | |
| 309 | + if (!t) | |
| 310 | + kunit_skip(test, "third target alloc fail"); | |
| 311 | + for (i = 0; i < ARRAY_SIZE(sa); i++) { | |
| 312 | + r = damon_new_region(sa[i], ea[i]); | |
| 313 | + if (!r) { | |
| 314 | + damon_free_target(t); | |
| 315 | + kunit_skip(test, "region alloc fail"); | |
| 316 | + } | |
| 317 | + damon_add_region(r, t); | |
| 318 | + } | |
| 319 | + damon_split_regions_of(t, 4, 5); | |
| 320 | + KUNIT_EXPECT_LE(test, damon_nr_regions(t), 12u); | |
| 321 | + damon_for_each_region(r, t) | |
| 322 | + KUNIT_EXPECT_GE(test, damon_sz_region(r) % 5ul, 0ul); | |
| 323 | + damon_free_target(t); | |
| 298 | 324 | } |
| 299 | 325 | |
| 300 | 326 | static void damon_test_ops_registration(struct kunit *test) |
| @@ -574,9 +600,10 @@ static void damos_test_commit_quota_goal(struct kunit *test) · 273 unmodified lines | ||
| 574 | 600 | }); |
| 575 | 601 | damos_test_commit_quota_goal_for(test, &dst, |
| 576 | 602 | &(struct damos_quota_goal) { |
| 577 | − .metric = DAMOS_QUOTA_USER_INPUT, | |
| 578 | − .target_value = 789, | |
| 579 | − .current_value = 12, | |
| 603 | + .metric = DAMOS_QUOTA_SOME_MEM_PSI_US, | |
| 604 | + .target_value = 234, | |
| 605 | + .current_value = 345, | |
| 606 | + .last_psi_total = 567, | |
| 580 | 607 | }); |
| 581 | 608 | } |
| 582 | 609 | |
| @@ -1159,7 +1186,7 @@ static void damon_test_set_filters_default_reject(struct kunit *test) · 576 unmodified lines | ||
| 1159 | 1186 | damos_set_filters_default_reject(&scheme); |
| 1160 | 1187 | /* |
| 1161 | 1188 | * A core-handled allow-filter is installed. |
| 1162 | − * Rejct by default on core layer filtering stage due to the last | |
| 1189 | + * Reject by default on core layer filtering stage due to the last | |
| 1163 | 1190 | * core-layer-filter's behavior. |
| 1164 | 1191 | * Allow by default on ops layer filtering stage due to the absence of |
| 1165 | 1192 | * ops layer filters. |
| @@ -147,7 +147,7 @@ static void damon_do_test_apply_three_regions(struct kunit *test, · 146 unmodified lines | ||
| 147 | 147 | damon_add_region(r, t); |
| 148 | 148 | } |
| 149 | 149 | |
| 150 | − damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION); | |
| 150 | + damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ); | |
| 151 | 151 | |
| 152 | 152 | for (i = 0; i < nr_expected / 2; i++) { |
| 153 | 153 | r = __nth_region_of(t, i); |
| @@ -19,8 +19,8 @@ · 18 unmodified lines | ||
| 19 | 19 | #include "ops-common.h" |
| 20 | 20 | |
| 21 | 21 | #ifdef CONFIG_DAMON_VADDR_KUNIT_TEST |
| 22 | −#undef DAMON_MIN_REGION | |
| 23 | −#define DAMON_MIN_REGION 1 | |
| 22 | +#undef DAMON_MIN_REGION_SZ | |
| 23 | +#define DAMON_MIN_REGION_SZ 1 | |
| 24 | 24 | #endif |
| 25 | 25 | |
| 26 | 26 | /* |
| @@ -78,7 +78,7 @@ static int damon_va_evenly_split_region(struct damon_target *t, · 51 unmodified lines | ||
| 78 | 78 | |
| 79 | 79 | orig_end = r->ar.end; |
| 80 | 80 | sz_orig = damon_sz_region(r); |
| 81 | − sz_piece = ALIGN_DOWN(sz_orig / nr_pieces, DAMON_MIN_REGION); | |
| 81 | + sz_piece = ALIGN_DOWN(sz_orig / nr_pieces, DAMON_MIN_REGION_SZ); | |
| 82 | 82 | |
| 83 | 83 | if (!sz_piece) |
| 84 | 84 | return -EINVAL; |
| @@ -161,12 +161,12 @@ next: · 76 unmodified lines | ||
| 161 | 161 | swap(first_gap, second_gap); |
| 162 | 162 | |
| 163 | 163 | /* Store the result */ |
| 164 | − regions[0].start = ALIGN(start, DAMON_MIN_REGION); | |
| 165 | − regions[0].end = ALIGN(first_gap.start, DAMON_MIN_REGION); | |
| 166 | − regions[1].start = ALIGN(first_gap.end, DAMON_MIN_REGION); | |
| 167 | − regions[1].end = ALIGN(second_gap.start, DAMON_MIN_REGION); | |
| 168 | − regions[2].start = ALIGN(second_gap.end, DAMON_MIN_REGION); | |
| 169 | − regions[2].end = ALIGN(prev->vm_end, DAMON_MIN_REGION); | |
| 164 | + regions[0].start = ALIGN(start, DAMON_MIN_REGION_SZ); | |
| 165 | + regions[0].end = ALIGN(first_gap.start, DAMON_MIN_REGION_SZ); | |
| 166 | + regions[1].start = ALIGN(first_gap.end, DAMON_MIN_REGION_SZ); | |
| 167 | + regions[1].end = ALIGN(second_gap.start, DAMON_MIN_REGION_SZ); | |
| 168 | + regions[2].start = ALIGN(second_gap.end, DAMON_MIN_REGION_SZ); | |
| 169 | + regions[2].end = ALIGN(prev->vm_end, DAMON_MIN_REGION_SZ); | |
| 170 | 170 | |
| 171 | 171 | return 0; |
| 172 | 172 | } |
| @@ -259,8 +259,8 @@ static void __damon_va_init_regions(struct damon_ctx *ctx, · 86 unmodified lines | ||
| 259 | 259 | sz += regions[i].end - regions[i].start; |
| 260 | 260 | if (ctx->attrs.min_nr_regions) |
| 261 | 261 | sz /= ctx->attrs.min_nr_regions; |
| 262 | − if (sz < DAMON_MIN_REGION) | |
| 263 | − sz = DAMON_MIN_REGION; | |
| 262 | + if (sz < DAMON_MIN_REGION_SZ) | |
| 263 | + sz = DAMON_MIN_REGION_SZ; | |
| 264 | 264 | |
| 265 | 265 | /* Set the initial three regions of the target */ |
| 266 | 266 | for (i = 0; i < 3; i++) { |
| @@ -299,7 +299,7 @@ static void damon_va_update(struct damon_ctx *ctx) · 32 unmodified lines | ||
| 299 | 299 | damon_for_each_target(t, ctx) { |
| 300 | 300 | if (damon_va_three_regions(t, three_regions)) |
| 301 | 301 | continue; |
| 302 | − damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION); | |
| 302 | + damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ); | |
| 303 | 303 | } |
| 304 | 304 | } |
| 305 | 305 | |
| @@ -1014,7 +1014,6 @@ static int __init damon_va_initcall(void) · 708 unmodified lines | ||
| 1014 | 1014 | .check_accesses = damon_va_check_accesses, |
| 1015 | 1015 | .target_valid = damon_va_target_valid, |
| 1016 | 1016 | .cleanup_target = damon_va_cleanup_target, |
| 1017 | − .cleanup = NULL, | |
| 1018 | 1017 | .apply_scheme = damon_va_apply_scheme, |
| 1019 | 1018 | .get_scheme_score = damon_va_scheme_score, |
| 1020 | 1019 | }; |
| @@ -971,22 +971,26 @@ static unsigned long __init get_random_vaddr(void) · 970 unmodified lines | ||
| 971 | 971 | return random_vaddr; |
| 972 | 972 | } |
| 973 | 973 | |
| 974 | −static void __init destroy_args(struct pgtable_debug_args *args) | |
| 974 | +static void __init | |
| 975 | +debug_vm_pgtable_free_huge_page(struct pgtable_debug_args *args, | |
| 976 | + unsigned long pfn, int order) | |
| 975 | 977 | { |
| 976 | − struct page *page = NULL; | |
| 978 | +#ifdef CONFIG_CONTIG_ALLOC | |
| 979 | + if (args->is_contiguous_page) { | |
| 980 | + free_contig_range(pfn, 1 << order); | |
| 981 | + return; | |
| 982 | + } | |
| 983 | +#endif | |
| 984 | + __free_pages(pfn_to_page(pfn), order); | |
| 985 | +} | |
| 977 | 986 | |
| 987 | +static void __init destroy_args(struct pgtable_debug_args *args) | |
| 988 | +{ | |
| 978 | 989 | /* Free (huge) page */ |
| 979 | 990 | if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && |
| 980 | 991 | has_transparent_pud_hugepage() && |
| 981 | 992 | args->pud_pfn != ULONG_MAX) { |
| 982 | − if (args->is_contiguous_page) { | |
| 983 | − free_contig_range(args->pud_pfn, | |
| 984 | − (1 << (HPAGE_PUD_SHIFT - PAGE_SHIFT))); | |
| 985 | − } else { | |
| 986 | − page = pfn_to_page(args->pud_pfn); | |
| 987 | − __free_pages(page, HPAGE_PUD_SHIFT - PAGE_SHIFT); | |
| 988 | − } | |
| 989 | − | |
| 993 | + debug_vm_pgtable_free_huge_page(args, args->pud_pfn, HPAGE_PUD_ORDER); | |
| 990 | 994 | args->pud_pfn = ULONG_MAX; |
| 991 | 995 | args->pmd_pfn = ULONG_MAX; |
| 992 | 996 | args->pte_pfn = ULONG_MAX; |
| @@ -995,20 +999,13 @@ static void __init destroy_args(struct pgtable_debug_args *args) · 2 unmodified lines | ||
| 995 | 999 | if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && |
| 996 | 1000 | has_transparent_hugepage() && |
| 997 | 1001 | args->pmd_pfn != ULONG_MAX) { |
| 998 | − if (args->is_contiguous_page) { | |
| 999 | − free_contig_range(args->pmd_pfn, (1 << HPAGE_PMD_ORDER)); | |
| 1000 | − } else { | |
| 1001 | − page = pfn_to_page(args->pmd_pfn); | |
| 1002 | − __free_pages(page, HPAGE_PMD_ORDER); | |
| 1003 | − } | |
| 1004 | − | |
| 1002 | + debug_vm_pgtable_free_huge_page(args, args->pmd_pfn, HPAGE_PMD_ORDER); | |
| 1005 | 1003 | args->pmd_pfn = ULONG_MAX; |
| 1006 | 1004 | args->pte_pfn = ULONG_MAX; |
| 1007 | 1005 | } |
| 1008 | 1006 | |
| 1009 | 1007 | if (args->pte_pfn != ULONG_MAX) { |
| 1010 | − page = pfn_to_page(args->pte_pfn); | |
| 1011 | − __free_page(page); | |
| 1008 | + __free_page(pfn_to_page(args->pte_pfn)); | |
| 1012 | 1009 | |
| 1013 | 1010 | args->pte_pfn = ULONG_MAX; |
| 1014 | 1011 | } |
| @@ -1242,8 +1239,7 @@ static int __init init_args(struct pgtable_debug_args *args) · 227 unmodified lines | ||
| 1242 | 1239 | */ |
| 1243 | 1240 | if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && |
| 1244 | 1241 | has_transparent_pud_hugepage()) { |
| 1245 | − page = debug_vm_pgtable_alloc_huge_page(args, | |
| 1246 | − HPAGE_PUD_SHIFT - PAGE_SHIFT); | |
| 1242 | + page = debug_vm_pgtable_alloc_huge_page(args, HPAGE_PUD_ORDER); | |
| 1247 | 1243 | if (page) { |
| 1248 | 1244 | args->pud_pfn = page_to_pfn(page); |
| 1249 | 1245 | args->pmd_pfn = args->pud_pfn; |
| @@ -1,3 +1,4 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 1 | 2 | #include <linux/device.h> |
| 2 | 3 | #include <linux/dma-map-ops.h> |
| 3 | 4 | #include <linux/dma-mapping.h> |
| @@ -30,6 +30,14 @@ static int __init early_ioremap_debug_setup(char *str) · 29 unmodified lines | ||
| 30 | 30 | } |
| 31 | 31 | early_param("early_ioremap_debug", early_ioremap_debug_setup); |
| 32 | 32 | |
| 33 | +#define early_ioremap_dbg(fmt, args...) \ | |
| 34 | + do { \ | |
| 35 | + if (unlikely(early_ioremap_debug)) { \ | |
| 36 | + pr_warn(fmt, ##args); \ | |
| 37 | + dump_stack(); \ | |
| 38 | + } \ | |
| 39 | + } while (0) | |
| 40 | + | |
| 33 | 41 | static int after_paging_init __initdata; |
| 34 | 42 | |
| 35 | 43 | pgprot_t __init __weak early_memremap_pgprot_adjust(resource_size_t phys_addr, |
| @@ -139,6 +147,9 @@ __early_ioremap(resource_size_t phys_addr, unsigned long size, pgprot_t prot) · 103 unmodified lines | ||
| 139 | 147 | if (WARN_ON(nrpages > NR_FIX_BTMAPS)) |
| 140 | 148 | return NULL; |
| 141 | 149 | |
| 150 | + early_ioremap_dbg("%s(%pa, %08lx) [%d] => %08lx + %08lx\n", | |
| 151 | + __func__, &phys_addr, size, slot, slot_virt[slot], offset); | |
| 152 | + | |
| 142 | 153 | /* |
| 143 | 154 | * Ok, go for it.. |
| 144 | 155 | */ |
| @@ -152,8 +163,6 @@ __early_ioremap(resource_size_t phys_addr, unsigned long size, pgprot_t prot) · 7 unmodified lines | ||
| 152 | 163 | --idx; |
| 153 | 164 | --nrpages; |
| 154 | 165 | } |
| 155 | − WARN(early_ioremap_debug, "%s(%pa, %08lx) [%d] => %08lx + %08lx\n", | |
| 156 | − __func__, &phys_addr, size, slot, offset, slot_virt[slot]); | |
| 157 | 166 | |
| 158 | 167 | prev_map[slot] = (void __iomem *)(offset + slot_virt[slot]); |
| 159 | 168 | return prev_map[slot]; |
| @@ -184,8 +193,7 @@ void __init early_iounmap(void __iomem *addr, unsigned long size) · 24 unmodified lines | ||
| 184 | 193 | __func__, addr, size, slot, prev_size[slot])) |
| 185 | 194 | return; |
| 186 | 195 | |
| 187 | − WARN(early_ioremap_debug, "%s(%p, %08lx) [%d]\n", | |
| 188 | − __func__, addr, size, slot); | |
| 196 | + early_ioremap_dbg("%s(%p, %08lx) [%d]\n", __func__, addr, size, slot); | |
| 189 | 197 | |
| 190 | 198 | virt_addr = (unsigned long)addr; |
| 191 | 199 | if (WARN_ON(virt_addr < fix_to_virt(FIX_BTMAP_BEGIN))) |
| @@ -1,3 +1,4 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 1 | 2 | /* |
| 2 | 3 | * Compatibility functions which bloat the callers too much to make inline. |
| 3 | 4 | * All of the callers of these functions should be converted to use folios |
| @@ -2806,17 +2806,6 @@ static bool gup_fast_folio_allowed(struct folio *folio, unsigned int flags) · 2805 unmodified lines | ||
| 2806 | 2806 | return !reject_file_backed || shmem_mapping(mapping); |
| 2807 | 2807 | } |
| 2808 | 2808 | |
| 2809 | −static void __maybe_unused gup_fast_undo_dev_pagemap(int *nr, int nr_start, | |
| 2810 | − unsigned int flags, struct page **pages) | |
| 2811 | −{ | |
| 2812 | − while ((*nr) - nr_start) { | |
| 2813 | − struct folio *folio = page_folio(pages[--(*nr)]); | |
| 2814 | − | |
| 2815 | − folio_clear_referenced(folio); | |
| 2816 | − gup_put_folio(folio, 1, flags); | |
| 2817 | − } | |
| 2818 | −} | |
| 2819 | − | |
| 2820 | 2809 | #ifdef CONFIG_ARCH_HAS_PTE_SPECIAL |
| 2821 | 2810 | /* |
| 2822 | 2811 | * GUP-fast relies on pte change detection to avoid concurrent pgtable |
| @@ -1,3 +1,4 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 1 | 2 | #include <linux/kernel.h> |
| 2 | 3 | #include <linux/mm.h> |
| 3 | 4 | #include <linux/slab.h> |
| @@ -180,12 +180,13 @@ struct page *__kmap_to_page(void *vaddr) · 179 unmodified lines | ||
| 180 | 180 | for (i = 0; i < kctrl->idx; i++) { |
| 181 | 181 | unsigned long base_addr; |
| 182 | 182 | int idx; |
| 183 | + pte_t pteval = kctrl->pteval[i]; | |
| 183 | 184 | |
| 184 | 185 | idx = arch_kmap_local_map_idx(i, pte_pfn(pteval)); |
| 185 | 186 | base_addr = __fix_to_virt(FIX_KMAP_BEGIN + idx); |
| 186 | 187 | |
| 187 | 188 | if (base_addr == base) |
| 188 | − return pte_page(kctrl->pteval[i]); | |
| 189 | + return pte_page(pteval); | |
| 189 | 190 | } |
| 190 | 191 | } |
| 191 | 192 | |
| @@ -3431,7 +3431,7 @@ static void remap_page(struct folio *folio, unsigned long nr, int flags) · 3430 unmodified lines | ||
| 3431 | 3431 | if (!folio_test_anon(folio)) |
| 3432 | 3432 | return; |
| 3433 | 3433 | for (;;) { |
| 3434 | − remove_migration_ptes(folio, folio, RMP_LOCKED | flags); | |
| 3434 | + remove_migration_ptes(folio, folio, TTU_RMAP_LOCKED | flags); | |
| 3435 | 3435 | i += folio_nr_pages(folio); |
| 3436 | 3436 | if (i >= nr) |
| 3437 | 3437 | break; |
| @@ -3944,7 +3944,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, · 506 unmodified lines | ||
| 3944 | 3944 | int old_order = folio_order(folio); |
| 3945 | 3945 | struct folio *new_folio, *next; |
| 3946 | 3946 | int nr_shmem_dropped = 0; |
| 3947 | − int remap_flags = 0; | |
| 3947 | + enum ttu_flags ttu_flags = 0; | |
| 3948 | 3948 | int ret; |
| 3949 | 3949 | pgoff_t end = 0; |
| 3950 | 3950 | |
| @@ -4064,9 +4064,9 @@ fail: · 113 unmodified lines | ||
| 4064 | 4064 | shmem_uncharge(mapping->host, nr_shmem_dropped); |
| 4065 | 4065 | |
| 4066 | 4066 | if (!ret && is_anon && !folio_is_device_private(folio)) |
| 4067 | − remap_flags = RMP_USE_SHARED_ZEROPAGE; | |
| 4067 | + ttu_flags = TTU_USE_SHARED_ZEROPAGE; | |
| 4068 | 4068 | |
| 4069 | − remap_page(folio, 1 << old_order, remap_flags); | |
| 4069 | + remap_page(folio, 1 << old_order, ttu_flags); | |
| 4070 | 4070 | |
| 4071 | 4071 | /* |
| 4072 | 4072 | * Unlock all after-split folios except the one containing |
| @@ -121,16 +121,6 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, · 120 unmodified lines | ||
| 121 | 121 | unsigned long start, unsigned long end, bool take_locks); |
| 122 | 122 | static struct resv_map *vma_resv_map(struct vm_area_struct *vma); |
| 123 | 123 | |
| 124 | −static void hugetlb_free_folio(struct folio *folio) | |
| 125 | −{ | |
| 126 | − if (folio_test_hugetlb_cma(folio)) { | |
| 127 | − hugetlb_cma_free_folio(folio); | |
| 128 | − return; | |
| 129 | − } | |
| 130 | − | |
| 131 | − folio_put(folio); | |
| 132 | −} | |
| 133 | − | |
| 134 | 124 | static inline bool subpool_is_free(struct hugepage_subpool *spool) |
| 135 | 125 | { |
| 136 | 126 | if (spool->count) |
| @@ -588,8 +578,9 @@ hugetlb_resv_map_add(struct resv_map *map, struct list_head *rg, long from, · 451 unmodified lines | ||
| 588 | 578 | record_hugetlb_cgroup_uncharge_info(cg, h, map, nrg); |
| 589 | 579 | list_add(&nrg->link, rg); |
| 590 | 580 | coalesce_file_region(map, nrg); |
| 591 | − } else | |
| 581 | + } else { | |
| 592 | 582 | *regions_needed += 1; |
| 583 | + } | |
| 593 | 584 | |
| 594 | 585 | return to - from; |
| 595 | 586 | } |
| @@ -1257,8 +1248,9 @@ void hugetlb_dup_vma_private(struct vm_area_struct *vma) · 661 unmodified lines | ||
| 1257 | 1248 | |
| 1258 | 1249 | if (vma_lock && vma_lock->vma != vma) |
| 1259 | 1250 | vma->vm_private_data = NULL; |
| 1260 | − } else | |
| 1251 | + } else { | |
| 1261 | 1252 | vma->vm_private_data = NULL; |
| 1253 | + } | |
| 1262 | 1254 | } |
| 1263 | 1255 | |
| 1264 | 1256 | /* |
| @@ -1417,47 +1409,25 @@ err: · 152 unmodified lines | ||
| 1417 | 1409 | return NULL; |
| 1418 | 1410 | } |
| 1419 | 1411 | |
| 1420 | −#ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE | |
| 1421 | −#ifdef CONFIG_CONTIG_ALLOC | |
| 1422 | −static struct folio *alloc_gigantic_folio(int order, gfp_t gfp_mask, | |
| 1412 | +#if defined(CONFIG_ARCH_HAS_GIGANTIC_PAGE) && defined(CONFIG_CONTIG_ALLOC) | |
| 1413 | +static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask, | |
| 1423 | 1414 | int nid, nodemask_t *nodemask) |
| 1424 | 1415 | { |
| 1425 | 1416 | struct folio *folio; |
| 1426 | − bool retried = false; | |
| 1427 | − | |
| 1428 | −retry: | |
| 1429 | − folio = hugetlb_cma_alloc_folio(order, gfp_mask, nid, nodemask); | |
| 1430 | − if (!folio) { | |
| 1431 | − if (hugetlb_cma_exclusive_alloc()) | |
| 1432 | − return NULL; | |
| 1433 | − | |
| 1434 | − folio = folio_alloc_gigantic(order, gfp_mask, nid, nodemask); | |
| 1435 | − if (!folio) | |
| 1436 | − return NULL; | |
| 1437 | − } | |
| 1438 | 1417 | |
| 1439 | − if (folio_ref_freeze(folio, 1)) | |
| 1418 | + folio = hugetlb_cma_alloc_frozen_folio(order, gfp_mask, nid, nodemask); | |
| 1419 | + if (folio) | |
| 1440 | 1420 | return folio; |
| 1441 | 1421 | |
| 1442 | − pr_warn("HugeTLB: unexpected refcount on PFN %lu\n", folio_pfn(folio)); | |
| 1443 | − hugetlb_free_folio(folio); | |
| 1444 | − if (!retried) { | |
| 1445 | − retried = true; | |
| 1446 | − goto retry; | |
| 1447 | − } | |
| 1448 | − return NULL; | |
| 1449 | −} | |
| 1422 | + if (hugetlb_cma_exclusive_alloc()) | |
| 1423 | + return NULL; | |
| 1450 | 1424 | |
| 1451 | −#else /* !CONFIG_CONTIG_ALLOC */ | |
| 1452 | −static struct folio *alloc_gigantic_folio(int order, gfp_t gfp_mask, int nid, | |
| 1453 | − nodemask_t *nodemask) | |
| 1454 | −{ | |
| 1455 | − return NULL; | |
| 1425 | + folio = (struct folio *)alloc_contig_frozen_pages(1 << order, gfp_mask, | |
| 1426 | + nid, nodemask); | |
| 1427 | + return folio; | |
| 1456 | 1428 | } |
| 1457 | −#endif /* CONFIG_CONTIG_ALLOC */ | |
| 1458 | − | |
| 1459 | −#else /* !CONFIG_ARCH_HAS_GIGANTIC_PAGE */ | |
| 1460 | −static struct folio *alloc_gigantic_folio(int order, gfp_t gfp_mask, int nid, | |
| 1429 | +#else /* !CONFIG_ARCH_HAS_GIGANTIC_PAGE || !CONFIG_CONTIG_ALLOC */ | |
| 1430 | +static struct folio *alloc_gigantic_frozen_folio(int order, gfp_t gfp_mask, int nid, | |
| 1461 | 1431 | nodemask_t *nodemask) |
| 1462 | 1432 | { |
| 1463 | 1433 | return NULL; |
| @@ -1587,9 +1557,11 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, · 123 unmodified lines | ||
| 1587 | 1557 | if (unlikely(folio_test_hwpoison(folio))) |
| 1588 | 1558 | folio_clear_hugetlb_hwpoison(folio); |
| 1589 | 1559 | |
| 1590 | − folio_ref_unfreeze(folio, 1); | |
| 1591 | − | |
| 1592 | − hugetlb_free_folio(folio); | |
| 1560 | + VM_BUG_ON_FOLIO(folio_ref_count(folio), folio); | |
| 1561 | + if (folio_test_hugetlb_cma(folio)) | |
| 1562 | + hugetlb_cma_free_frozen_folio(folio); | |
| 1563 | + else | |
| 1564 | + free_frozen_pages(&folio->page, folio_order(folio)); | |
| 1593 | 1565 | } |
| 1594 | 1566 | |
| 1595 | 1567 | /* |
| @@ -1869,7 +1841,7 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio) · 273 unmodified lines | ||
| 1869 | 1841 | return NULL; |
| 1870 | 1842 | } |
| 1871 | 1843 | |
| 1872 | −static struct folio *alloc_buddy_hugetlb_folio(int order, gfp_t gfp_mask, | |
| 1844 | +static struct folio *alloc_buddy_frozen_folio(int order, gfp_t gfp_mask, | |
| 1873 | 1845 | int nid, nodemask_t *nmask, nodemask_t *node_alloc_noretry) |
| 1874 | 1846 | { |
| 1875 | 1847 | struct folio *folio; |
| @@ -1925,10 +1897,10 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, · 49 unmodified lines | ||
| 1925 | 1897 | nid = numa_mem_id(); |
| 1926 | 1898 | |
| 1927 | 1899 | if (order_is_gigantic(order)) |
| 1928 | − folio = alloc_gigantic_folio(order, gfp_mask, nid, nmask); | |
| 1900 | + folio = alloc_gigantic_frozen_folio(order, gfp_mask, nid, nmask); | |
| 1929 | 1901 | else |
| 1930 | − folio = alloc_buddy_hugetlb_folio(order, gfp_mask, nid, nmask, | |
| 1931 | − node_alloc_noretry); | |
| 1902 | + folio = alloc_buddy_frozen_folio(order, gfp_mask, nid, nmask, | |
| 1903 | + node_alloc_noretry); | |
| 1932 | 1904 | if (folio) |
| 1933 | 1905 | init_new_hugetlb_folio(folio); |
| 1934 | 1906 | return folio; |
| @@ -2106,8 +2078,9 @@ retry: · 171 unmodified lines | ||
| 2106 | 2078 | h->max_huge_pages++; |
| 2107 | 2079 | goto out; |
| 2108 | 2080 | } |
| 2109 | − } else | |
| 2081 | + } else { | |
| 2110 | 2082 | rc = 0; |
| 2083 | + } | |
| 2111 | 2084 | |
| 2112 | 2085 | update_and_free_hugetlb_folio(h, folio, false); |
| 2113 | 2086 | return rc; |
| @@ -2702,11 +2675,12 @@ void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, · 588 unmodified lines | ||
| 2702 | 2675 | * be consumed on a subsequent allocation. |
| 2703 | 2676 | */ |
| 2704 | 2677 | folio_set_hugetlb_restore_reserve(folio); |
| 2705 | − } else | |
| 2678 | + } else { | |
| 2706 | 2679 | /* |
| 2707 | 2680 | * No reservation present, do nothing |
| 2708 | 2681 | */ |
| 2709 | − vma_end_reservation(h, vma, address); | |
| 2682 | + vma_end_reservation(h, vma, address); | |
| 2683 | + } | |
| 2710 | 2684 | } |
| 2711 | 2685 | } |
| 2712 | 2686 | |
| @@ -2836,23 +2810,62 @@ int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *list) · 123 unmodified lines | ||
| 2836 | 2810 | */ |
| 2837 | 2811 | int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn) |
| 2838 | 2812 | { |
| 2839 | − struct folio *folio; | |
| 2813 | + unsigned long nr = 0; | |
| 2814 | + struct page *page; | |
| 2815 | + struct hstate *h; | |
| 2816 | + LIST_HEAD(list); | |
| 2840 | 2817 | int ret = 0; |
| 2841 | 2818 | |
| 2842 | − LIST_HEAD(isolate_list); | |
| 2819 | + /* Avoid pfn iterations if no free non-gigantic huge pages */ | |
| 2820 | + for_each_hstate(h) { | |
| 2821 | + if (hstate_is_gigantic(h)) | |
| 2822 | + continue; | |
| 2823 | + | |
| 2824 | + nr += h->free_huge_pages; | |
| 2825 | + if (nr) | |
| 2826 | + break; | |
| 2827 | + } | |
| 2828 | + | |
| 2829 | + if (!nr) | |
| 2830 | + return 0; | |
| 2843 | 2831 | |
| 2844 | 2832 | while (start_pfn < end_pfn) { |
| 2845 | − folio = pfn_folio(start_pfn); | |
| 2833 | + page = pfn_to_page(start_pfn); | |
| 2834 | + nr = 1; | |
| 2846 | 2835 | |
| 2847 | − /* Not to disrupt normal path by vainly holding hugetlb_lock */ | |
| 2848 | − if (folio_test_hugetlb(folio) && !folio_ref_count(folio)) { | |
| 2849 | − ret = alloc_and_dissolve_hugetlb_folio(folio, &isolate_list); | |
| 2850 | − if (ret) | |
| 2851 | − break; | |
| 2836 | + if (PageHuge(page) || PageCompound(page)) { | |
| 2837 | + struct folio *folio = page_folio(page); | |
| 2838 | + | |
| 2839 | + nr = folio_nr_pages(folio) - folio_page_idx(folio, page); | |
| 2840 | + | |
| 2841 | + /* | |
| 2842 | + * Don't disrupt normal path by vainly holding | |
| 2843 | + * hugetlb_lock | |
| 2844 | + */ | |
| 2845 | + if (folio_test_hugetlb(folio) && !folio_ref_count(folio)) { | |
| 2846 | + if (order_is_gigantic(folio_order(folio))) { | |
| 2847 | + ret = -ENOMEM; | |
| 2848 | + break; | |
| 2849 | + } | |
| 2850 | + | |
| 2851 | + ret = alloc_and_dissolve_hugetlb_folio(folio, &list); | |
| 2852 | + if (ret) | |
| 2853 | + break; | |
| 2854 | + | |
| 2855 | + putback_movable_pages(&list); | |
| 2856 | + } | |
| 2857 | + } else if (PageBuddy(page)) { | |
| 2858 | + /* | |
| 2859 | + * Buddy order check without zone lock is unsafe and | |
| 2860 | + * the order is maybe invalid, but race should be | |
| 2861 | + * small, and the worst thing is skipping free hugetlb. | |
| 2862 | + */ | |
| 2863 | + const unsigned int order = buddy_order_unsafe(page); | |
| 2852 | 2864 | |
| 2853 | − putback_movable_pages(&isolate_list); | |
| 2865 | + if (order <= MAX_PAGE_ORDER) | |
| 2866 | + nr = 1UL << order; | |
| 2854 | 2867 | } |
| 2855 | − start_pfn++; | |
| 2868 | + start_pfn += nr; | |
| 2856 | 2869 | } |
| 2857 | 2870 | |
| 2858 | 2871 | return ret; |
| @@ -3019,13 +3032,10 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 160 unmodified lines | ||
| 3019 | 3032 | |
| 3020 | 3033 | rsv_adjust = hugepage_subpool_put_pages(spool, 1); |
| 3021 | 3034 | hugetlb_acct_memory(h, -rsv_adjust); |
| 3022 | − if (map_chg) { | |
| 3023 | − spin_lock_irq(&hugetlb_lock); | |
| 3024 | − hugetlb_cgroup_uncharge_folio_rsvd( | |
| 3025 | − hstate_index(h), pages_per_huge_page(h), | |
| 3026 | − folio); | |
| 3027 | − spin_unlock_irq(&hugetlb_lock); | |
| 3028 | − } | |
| 3035 | + spin_lock_irq(&hugetlb_lock); | |
| 3036 | + hugetlb_cgroup_uncharge_folio_rsvd( | |
| 3037 | + hstate_index(h), pages_per_huge_page(h), folio); | |
| 3038 | + spin_unlock_irq(&hugetlb_lock); | |
| 3029 | 3039 | } |
| 3030 | 3040 | } |
| 3031 | 3041 | |
| @@ -3425,6 +3435,13 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) · 393 unmodified lines | ||
| 3425 | 3435 | |
| 3426 | 3436 | folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid, |
| 3427 | 3437 | &node_states[N_MEMORY], NULL); |
| 3438 | + if (!folio && !list_empty(&folio_list) && | |
| 3439 | + hugetlb_vmemmap_optimizable_size(h)) { | |
| 3440 | + prep_and_add_allocated_folios(h, &folio_list); | |
| 3441 | + INIT_LIST_HEAD(&folio_list); | |
| 3442 | + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid, | |
| 3443 | + &node_states[N_MEMORY], NULL); | |
| 3444 | + } | |
| 3428 | 3445 | if (!folio) |
| 3429 | 3446 | break; |
| 3430 | 3447 | list_add(&folio->lru, &folio_list); |
| @@ -4159,7 +4176,6 @@ static int __init hugetlb_init(void) · 728 unmodified lines | ||
| 4159 | 4176 | } |
| 4160 | 4177 | } |
| 4161 | 4178 | |
| 4162 | − hugetlb_cma_check(); | |
| 4163 | 4179 | hugetlb_init_hstates(); |
| 4164 | 4180 | gather_bootmem_prealloc(); |
| 4165 | 4181 | report_hugepages(); |
| @@ -4487,21 +4503,11 @@ void __init hugetlb_bootmem_set_nodes(void) · 321 unmodified lines | ||
| 4487 | 4503 | } |
| 4488 | 4504 | } |
| 4489 | 4505 | |
| 4490 | −static bool __hugetlb_bootmem_allocated __initdata; | |
| 4491 | − | |
| 4492 | −bool __init hugetlb_bootmem_allocated(void) | |
| 4493 | −{ | |
| 4494 | − return __hugetlb_bootmem_allocated; | |
| 4495 | −} | |
| 4496 | − | |
| 4497 | 4506 | void __init hugetlb_bootmem_alloc(void) |
| 4498 | 4507 | { |
| 4499 | 4508 | struct hstate *h; |
| 4500 | 4509 | int i; |
| 4501 | 4510 | |
| 4502 | − if (__hugetlb_bootmem_allocated) | |
| 4503 | − return; | |
| 4504 | − | |
| 4505 | 4511 | hugetlb_bootmem_set_nodes(); |
| 4506 | 4512 | |
| 4507 | 4513 | for (i = 0; i < MAX_NUMNODES; i++) |
| @@ -4515,8 +4521,6 @@ void __init hugetlb_bootmem_alloc(void) · 7 unmodified lines | ||
| 4515 | 4521 | if (hstate_is_gigantic(h)) |
| 4516 | 4522 | hugetlb_hstate_alloc_pages(h); |
| 4517 | 4523 | } |
| 4518 | − | |
| 4519 | − __hugetlb_bootmem_allocated = true; | |
| 4520 | 4524 | } |
| 4521 | 4525 | |
| 4522 | 4526 | /* |
| @@ -4718,10 +4722,12 @@ static void hugetlb_vm_op_open(struct vm_area_struct *vma) · 195 unmodified lines | ||
| 4718 | 4722 | if (vma_lock->vma != vma) { |
| 4719 | 4723 | vma->vm_private_data = NULL; |
| 4720 | 4724 | hugetlb_vma_lock_alloc(vma); |
| 4721 | − } else | |
| 4725 | + } else { | |
| 4722 | 4726 | pr_warn("HugeTLB: vma_lock already exists in %s.\n", __func__); |
| 4723 | − } else | |
| 4727 | + } | |
| 4728 | + } else { | |
| 4724 | 4729 | hugetlb_vma_lock_alloc(vma); |
| 4730 | + } | |
| 4725 | 4731 | } |
| 4726 | 4732 | } |
| 4727 | 4733 | |
| @@ -1,3 +1,4 @@ | ||
| 1 | +// SPDX-License-Identifier: LGPL-2.1 | |
| 1 | 2 | /* |
| 2 | 3 | * |
| 3 | 4 | * Copyright IBM Corporation, 2012 |
| @@ -7,14 +8,6 @@ · 3 unmodified lines | ||
| 7 | 8 | * Copyright (C) 2019 Red Hat, Inc. |
| 8 | 9 | * Author: Giuseppe Scrivano <gscrivan@redhat.com> |
| 9 | 10 | * |
| 10 | − * This program is free software; you can redistribute it and/or modify it | |
| 11 | − * under the terms of version 2.1 of the GNU Lesser General Public License | |
| 12 | − * as published by the Free Software Foundation. | |
| 13 | − * | |
| 14 | − * This program is distributed in the hope that it would be useful, but | |
| 15 | − * WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 16 | − * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. | |
| 17 | − * | |
| 18 | 11 | */ |
| 19 | 12 | |
| 20 | 13 | #include <linux/cgroup.h> |
| @@ -822,7 +815,7 @@ hugetlb_cgroup_cfttypes_init(struct hstate *h, struct cftype *cft, · 801 unmodified lines | ||
| 822 | 815 | for (i = 0; i < tmpl_size; cft++, tmpl++, i++) { |
| 823 | 816 | *cft = *tmpl; |
| 824 | 817 | /* rebuild the name */ |
| 825 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.%s", buf, tmpl->name); | |
| 818 | + scnprintf(cft->name, MAX_CFTYPE_NAME, "%s.%s", buf, tmpl->name); | |
| 826 | 819 | /* rebuild the private */ |
| 827 | 820 | cft->private = MEMFILE_PRIVATE(idx, tmpl->private); |
| 828 | 821 | /* rebuild the file_offset */ |
| @@ -13,42 +13,46 @@ · 12 unmodified lines | ||
| 13 | 13 | #include "hugetlb_cma.h" |
| 14 | 14 | |
| 15 | 15 | |
| 16 | −static struct cma *hugetlb_cma[MAX_NUMNODES]; | |
| 16 | +static struct cma *hugetlb_cma[MAX_NUMNODES] __ro_after_init; | |
| 17 | 17 | static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; |
| 18 | −static bool hugetlb_cma_only; | |
| 19 | −static unsigned long hugetlb_cma_size __initdata; | |
| 18 | +static bool hugetlb_cma_only __ro_after_init; | |
| 19 | +static unsigned long hugetlb_cma_size __ro_after_init; | |
| 20 | 20 | |
| 21 | −void hugetlb_cma_free_folio(struct folio *folio) | |
| 21 | +void hugetlb_cma_free_frozen_folio(struct folio *folio) | |
| 22 | 22 | { |
| 23 | − int nid = folio_nid(folio); | |
| 24 | − | |
| 25 | − WARN_ON_ONCE(!cma_free_folio(hugetlb_cma[nid], folio)); | |
| 23 | + WARN_ON_ONCE(!cma_release_frozen(hugetlb_cma[folio_nid(folio)], | |
| 24 | + &folio->page, folio_nr_pages(folio))); | |
| 26 | 25 | } |
| 27 | 26 | |
| 28 | − | |
| 29 | −struct folio *hugetlb_cma_alloc_folio(int order, gfp_t gfp_mask, | |
| 30 | − int nid, nodemask_t *nodemask) | |
| 27 | +struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask, | |
| 28 | + int nid, nodemask_t *nodemask) | |
| 31 | 29 | { |
| 32 | 30 | int node; |
| 33 | − struct folio *folio = NULL; | |
| 31 | + struct folio *folio; | |
| 32 | + struct page *page = NULL; | |
| 33 | + | |
| 34 | + if (!hugetlb_cma_size) | |
| 35 | + return NULL; | |
| 34 | 36 | |
| 35 | 37 | if (hugetlb_cma[nid]) |
| 36 | − folio = cma_alloc_folio(hugetlb_cma[nid], order, gfp_mask); | |
| 38 | + page = cma_alloc_frozen_compound(hugetlb_cma[nid], order); | |
| 37 | 39 | |
| 38 | − if (!folio && !(gfp_mask & __GFP_THISNODE)) { | |
| 40 | + if (!page && !(gfp_mask & __GFP_THISNODE)) { | |
| 39 | 41 | for_each_node_mask(node, *nodemask) { |
| 40 | 42 | if (node == nid || !hugetlb_cma[node]) |
| 41 | 43 | continue; |
| 42 | 44 | |
| 43 | − folio = cma_alloc_folio(hugetlb_cma[node], order, gfp_mask); | |
| 44 | − if (folio) | |
| 45 | + page = cma_alloc_frozen_compound(hugetlb_cma[node], order); | |
| 46 | + if (page) | |
| 45 | 47 | break; |
| 46 | 48 | } |
| 47 | 49 | } |
| 48 | 50 | |
| 49 | − if (folio) | |
| 50 | − folio_set_hugetlb_cma(folio); | |
| 51 | + if (!page) | |
| 52 | + return NULL; | |
| 51 | 53 | |
| 54 | + folio = page_folio(page); | |
| 55 | + folio_set_hugetlb_cma(folio); | |
| 52 | 56 | return folio; |
| 53 | 57 | } |
| 54 | 58 | |
| @@ -85,9 +89,6 @@ hugetlb_cma_alloc_bootmem(struct hstate *h, int *nid, bool node_exact) · 30 unmodified lines | ||
| 85 | 89 | return m; |
| 86 | 90 | } |
| 87 | 91 | |
| 88 | − | |
| 89 | −static bool cma_reserve_called __initdata; | |
| 90 | − | |
| 91 | 92 | static int __init cmdline_parse_hugetlb_cma(char *p) |
| 92 | 93 | { |
| 93 | 94 | int nid, count = 0; |
| @@ -134,12 +135,26 @@ static int __init cmdline_parse_hugetlb_cma_only(char *p) · 40 unmodified lines | ||
| 134 | 135 | |
| 135 | 136 | early_param("hugetlb_cma_only", cmdline_parse_hugetlb_cma_only); |
| 136 | 137 | |
| 137 | −void __init hugetlb_cma_reserve(int order) | |
| 138 | +unsigned int __weak arch_hugetlb_cma_order(void) | |
| 138 | 139 | { |
| 139 | − unsigned long size, reserved, per_node; | |
| 140 | + return 0; | |
| 141 | +} | |
| 142 | + | |
| 143 | +void __init hugetlb_cma_reserve(void) | |
| 144 | +{ | |
| 145 | + unsigned long size, reserved, per_node, order; | |
| 140 | 146 | bool node_specific_cma_alloc = false; |
| 141 | 147 | int nid; |
| 142 | 148 | |
| 149 | + if (!hugetlb_cma_size) | |
| 150 | + return; | |
| 151 | + | |
| 152 | + order = arch_hugetlb_cma_order(); | |
| 153 | + if (!order) { | |
| 154 | + pr_warn("hugetlb_cma: the option isn't supported by current arch\n"); | |
| 155 | + return; | |
| 156 | + } | |
| 157 | + | |
| 143 | 158 | /* |
| 144 | 159 | * HugeTLB CMA reservation is required for gigantic |
| 145 | 160 | * huge pages which could not be allocated via the |
| @@ -147,10 +162,6 @@ void __init hugetlb_cma_reserve(int order) · 1 unmodified line | ||
| 147 | 162 | * breaking this assumption. |
| 148 | 163 | */ |
| 149 | 164 | VM_WARN_ON(order <= MAX_PAGE_ORDER); |
| 150 | − cma_reserve_called = true; | |
| 151 | − | |
| 152 | − if (!hugetlb_cma_size) | |
| 153 | − return; | |
| 154 | 165 | |
| 155 | 166 | hugetlb_bootmem_set_nodes(); |
| 156 | 167 | |
| @@ -244,14 +255,6 @@ void __init hugetlb_cma_reserve(int order) · 87 unmodified lines | ||
| 244 | 255 | hugetlb_cma_size = 0; |
| 245 | 256 | } |
| 246 | 257 | |
| 247 | −void __init hugetlb_cma_check(void) | |
| 248 | −{ | |
| 249 | − if (!hugetlb_cma_size || cma_reserve_called) | |
| 250 | − return; | |
| 251 | − | |
| 252 | − pr_warn("hugetlb_cma: the option isn't supported by current arch\n"); | |
| 253 | −} | |
| 254 | − | |
| 255 | 258 | bool hugetlb_cma_exclusive_alloc(void) |
| 256 | 259 | { |
| 257 | 260 | return hugetlb_cma_only; |
| @@ -3,23 +3,22 @@ · 2 unmodified lines | ||
| 3 | 3 | #define _LINUX_HUGETLB_CMA_H |
| 4 | 4 | |
| 5 | 5 | #ifdef CONFIG_CMA |
| 6 | −void hugetlb_cma_free_folio(struct folio *folio); | |
| 7 | −struct folio *hugetlb_cma_alloc_folio(int order, gfp_t gfp_mask, | |
| 6 | +void hugetlb_cma_free_frozen_folio(struct folio *folio); | |
| 7 | +struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask, | |
| 8 | 8 | int nid, nodemask_t *nodemask); |
| 9 | 9 | struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int *nid, |
| 10 | 10 | bool node_exact); |
| 11 | −void hugetlb_cma_check(void); | |
| 12 | 11 | bool hugetlb_cma_exclusive_alloc(void); |
| 13 | 12 | unsigned long hugetlb_cma_total_size(void); |
| 14 | 13 | void hugetlb_cma_validate_params(void); |
| 15 | 14 | bool hugetlb_early_cma(struct hstate *h); |
| 16 | 15 | #else |
| 17 | −static inline void hugetlb_cma_free_folio(struct folio *folio) | |
| 16 | +static inline void hugetlb_cma_free_frozen_folio(struct folio *folio) | |
| 18 | 17 | { |
| 19 | 18 | } |
| 20 | 19 | |
| 21 | −static inline struct folio *hugetlb_cma_alloc_folio(int order, gfp_t gfp_mask, | |
| 22 | − int nid, nodemask_t *nodemask) | |
| 20 | +static inline struct folio *hugetlb_cma_alloc_frozen_folio(int order, | |
| 21 | + gfp_t gfp_mask, int nid, nodemask_t *nodemask) | |
| 23 | 22 | { |
| 24 | 23 | return NULL; |
| 25 | 24 | } |
| @@ -31,10 +30,6 @@ struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int *nid, · 5 unmodified lines | ||
| 31 | 30 | return NULL; |
| 32 | 31 | } |
| 33 | 32 | |
| 34 | −static inline void hugetlb_cma_check(void) | |
| 35 | −{ | |
| 36 | −} | |
| 37 | − | |
| 38 | 33 | static inline bool hugetlb_cma_exclusive_alloc(void) |
| 39 | 34 | { |
| 40 | 35 | return false; |
| @@ -8,6 +8,8 @@ · 7 unmodified lines | ||
| 8 | 8 | |
| 9 | 9 | #include "hugetlb_internal.h" |
| 10 | 10 | |
| 11 | +int movable_gigantic_pages; | |
| 12 | + | |
| 11 | 13 | #ifdef CONFIG_SYSCTL |
| 12 | 14 | static int proc_hugetlb_doulongvec_minmax(const struct ctl_table *table, int write, |
| 13 | 15 | void *buffer, size_t *length, |
| @@ -125,6 +127,15 @@ static const struct ctl_table hugetlb_table[] = { · 111 unmodified lines | ||
| 125 | 127 | .mode = 0644, |
| 126 | 128 | .proc_handler = hugetlb_overcommit_handler, |
| 127 | 129 | }, |
| 130 | +#ifdef CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION | |
| 131 | + { | |
| 132 | + .procname = "movable_gigantic_pages", | |
| 133 | + .data = &movable_gigantic_pages, | |
| 134 | + .maxlen = sizeof(int), | |
| 135 | + .mode = 0644, | |
| 136 | + .proc_handler = proc_dointvec, | |
| 137 | + }, | |
| 138 | +#endif | |
| 128 | 139 | }; |
| 129 | 140 | |
| 130 | 141 | void __init hugetlb_sysctl_init(void) |
| @@ -794,14 +794,6 @@ void __init hugetlb_vmemmap_init_early(int nid) · 793 unmodified lines | ||
| 794 | 794 | struct huge_bootmem_page *m = NULL; |
| 795 | 795 | void *map; |
| 796 | 796 | |
| 797 | − /* | |
| 798 | − * Noting to do if bootmem pages were not allocated | |
| 799 | − * early in boot, or if HVO wasn't enabled in the | |
| 800 | − * first place. | |
| 801 | − */ | |
| 802 | − if (!hugetlb_bootmem_allocated()) | |
| 803 | − return; | |
| 804 | − | |
| 805 | 797 | if (!READ_ONCE(vmemmap_optimize_enabled)) |
| 806 | 798 | return; |
| 807 | 799 | |
| @@ -847,9 +839,6 @@ void __init hugetlb_vmemmap_init_late(int nid) · 39 unmodified lines | ||
| 847 | 839 | struct hstate *h; |
| 848 | 840 | void *map; |
| 849 | 841 | |
| 850 | − if (!hugetlb_bootmem_allocated()) | |
| 851 | − return; | |
| 852 | − | |
| 853 | 842 | if (!READ_ONCE(vmemmap_optimize_enabled)) |
| 854 | 843 | return; |
| 855 | 844 | |
| @@ -171,7 +171,7 @@ static inline int mmap_file(struct file *file, struct vm_area_struct *vma) · 170 unmodified lines | ||
| 171 | 171 | |
| 172 | 172 | /* |
| 173 | 173 | * OK, we tried to call the file hook for mmap(), but an error |
| 174 | − * arose. The mapping is in an inconsistent state and we most not invoke | |
| 174 | + * arose. The mapping is in an inconsistent state and we must not invoke | |
| 175 | 175 | * any further hooks on it. |
| 176 | 176 | */ |
| 177 | 177 | vma->vm_ops = &vma_dummy_vm_ops; |
| @@ -199,6 +199,73 @@ static inline void vma_close(struct vm_area_struct *vma) · 21 unmodified lines | ||
| 199 | 199 | |
| 200 | 200 | #ifdef CONFIG_MMU |
| 201 | 201 | |
| 202 | +static inline void get_anon_vma(struct anon_vma *anon_vma) | |
| 203 | +{ | |
| 204 | + atomic_inc(&anon_vma->refcount); | |
| 205 | +} | |
| 206 | + | |
| 207 | +void __put_anon_vma(struct anon_vma *anon_vma); | |
| 208 | + | |
| 209 | +static inline void put_anon_vma(struct anon_vma *anon_vma) | |
| 210 | +{ | |
| 211 | + if (atomic_dec_and_test(&anon_vma->refcount)) | |
| 212 | + __put_anon_vma(anon_vma); | |
| 213 | +} | |
| 214 | + | |
| 215 | +static inline void anon_vma_lock_write(struct anon_vma *anon_vma) | |
| 216 | +{ | |
| 217 | + down_write(&anon_vma->root->rwsem); | |
| 218 | +} | |
| 219 | + | |
| 220 | +static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) | |
| 221 | +{ | |
| 222 | + return down_write_trylock(&anon_vma->root->rwsem); | |
| 223 | +} | |
| 224 | + | |
| 225 | +static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) | |
| 226 | +{ | |
| 227 | + up_write(&anon_vma->root->rwsem); | |
| 228 | +} | |
| 229 | + | |
| 230 | +static inline void anon_vma_lock_read(struct anon_vma *anon_vma) | |
| 231 | +{ | |
| 232 | + down_read(&anon_vma->root->rwsem); | |
| 233 | +} | |
| 234 | + | |
| 235 | +static inline int anon_vma_trylock_read(struct anon_vma *anon_vma) | |
| 236 | +{ | |
| 237 | + return down_read_trylock(&anon_vma->root->rwsem); | |
| 238 | +} | |
| 239 | + | |
| 240 | +static inline void anon_vma_unlock_read(struct anon_vma *anon_vma) | |
| 241 | +{ | |
| 242 | + up_read(&anon_vma->root->rwsem); | |
| 243 | +} | |
| 244 | + | |
| 245 | +struct anon_vma *folio_get_anon_vma(const struct folio *folio); | |
| 246 | + | |
| 247 | +/* Operations which modify VMAs. */ | |
| 248 | +enum vma_operation { | |
| 249 | + VMA_OP_SPLIT, | |
| 250 | + VMA_OP_MERGE_UNFAULTED, | |
| 251 | + VMA_OP_REMAP, | |
| 252 | + VMA_OP_FORK, | |
| 253 | +}; | |
| 254 | + | |
| 255 | +int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, | |
| 256 | + enum vma_operation operation); | |
| 257 | +int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma); | |
| 258 | +int __anon_vma_prepare(struct vm_area_struct *vma); | |
| 259 | +void unlink_anon_vmas(struct vm_area_struct *vma); | |
| 260 | + | |
| 261 | +static inline int anon_vma_prepare(struct vm_area_struct *vma) | |
| 262 | +{ | |
| 263 | + if (likely(vma->anon_vma)) | |
| 264 | + return 0; | |
| 265 | + | |
| 266 | + return __anon_vma_prepare(vma); | |
| 267 | +} | |
| 268 | + | |
| 202 | 269 | /* Flags for folio_pte_batch(). */ |
| 203 | 270 | typedef int __bitwise fpb_t; |
| 204 | 271 | |
| @@ -513,6 +580,14 @@ static inline void set_page_refcounted(struct page *page) · 308 unmodified lines | ||
| 513 | 580 | set_page_count(page, 1); |
| 514 | 581 | } |
| 515 | 582 | |
| 583 | +static inline void set_pages_refcounted(struct page *page, unsigned long nr_pages) | |
| 584 | +{ | |
| 585 | + unsigned long pfn = page_to_pfn(page); | |
| 586 | + | |
| 587 | + for (; nr_pages--; pfn++) | |
| 588 | + set_page_refcounted(pfn_to_page(pfn)); | |
| 589 | +} | |
| 590 | + | |
| 516 | 591 | /* |
| 517 | 592 | * Return true if a folio needs ->release_folio() calling upon it. |
| 518 | 593 | */ |
| @@ -853,6 +928,12 @@ void memmap_init_range(unsigned long, int, unsigned long, unsigned long, · 334 unmodified lines | ||
| 853 | 928 | unsigned long, enum meminit_context, struct vmem_altmap *, int, |
| 854 | 929 | bool); |
| 855 | 930 | |
| 931 | +#ifdef CONFIG_SPARSEMEM | |
| 932 | +void sparse_init(void); | |
| 933 | +#else | |
| 934 | +static inline void sparse_init(void) {} | |
| 935 | +#endif /* CONFIG_SPARSEMEM */ | |
| 936 | + | |
| 856 | 937 | #if defined CONFIG_COMPACTION || defined CONFIG_CMA |
| 857 | 938 | |
| 858 | 939 | /* |
| @@ -929,9 +1010,14 @@ void init_cma_reserved_pageblock(struct page *page); · 70 unmodified lines | ||
| 929 | 1010 | struct cma; |
| 930 | 1011 | |
| 931 | 1012 | #ifdef CONFIG_CMA |
| 1013 | +bool cma_validate_zones(struct cma *cma); | |
| 932 | 1014 | void *cma_reserve_early(struct cma *cma, unsigned long size); |
| 933 | 1015 | void init_cma_pageblock(struct page *page); |
| 934 | 1016 | #else |
| 1017 | +static inline bool cma_validate_zones(struct cma *cma) | |
| 1018 | +{ | |
| 1019 | + return false; | |
| 1020 | +} | |
| 935 | 1021 | static inline void *cma_reserve_early(struct cma *cma, unsigned long size) |
| 936 | 1022 | { |
| 937 | 1023 | return NULL; |
| @@ -1658,24 +1744,6 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, · 720 unmodified lines | ||
| 1658 | 1744 | unsigned long end, const struct mm_walk_ops *ops, |
| 1659 | 1745 | pgd_t *pgd, void *private); |
| 1660 | 1746 | |
| 1661 | −/* pt_reclaim.c */ | |
| 1662 | −bool try_get_and_clear_pmd(struct mm_struct *mm, pmd_t *pmd, pmd_t *pmdval); | |
| 1663 | −void free_pte(struct mm_struct *mm, unsigned long addr, struct mmu_gather *tlb, | |
| 1664 | − pmd_t pmdval); | |
| 1665 | −void try_to_free_pte(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, | |
| 1666 | − struct mmu_gather *tlb); | |
| 1667 | − | |
| 1668 | −#ifdef CONFIG_PT_RECLAIM | |
| 1669 | −bool reclaim_pt_is_enabled(unsigned long start, unsigned long end, | |
| 1670 | − struct zap_details *details); | |
| 1671 | −#else | |
| 1672 | −static inline bool reclaim_pt_is_enabled(unsigned long start, unsigned long end, | |
| 1673 | − struct zap_details *details) | |
| 1674 | −{ | |
| 1675 | − return false; | |
| 1676 | −} | |
| 1677 | −#endif /* CONFIG_PT_RECLAIM */ | |
| 1678 | − | |
| 1679 | 1747 | void dup_mm_exe_file(struct mm_struct *mm, struct mm_struct *oldmm); |
| 1680 | 1748 | int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm); |
| 1681 | 1749 | |
| @@ -1840,6 +1840,29 @@ static void vmalloc_helpers_tags(struct kunit *test) · 1839 unmodified lines | ||
| 1840 | 1840 | vfree(ptr); |
| 1841 | 1841 | } |
| 1842 | 1842 | |
| 1843 | +static void vmalloc_oob_helper(struct kunit *test, char *v_ptr, size_t size) | |
| 1844 | +{ | |
| 1845 | + /* | |
| 1846 | + * We have to be careful not to hit the guard page in vmalloc tests. | |
| 1847 | + * The MMU will catch that and crash us. | |
| 1848 | + */ | |
| 1849 | + | |
| 1850 | + /* Make sure in-bounds accesses are valid. */ | |
| 1851 | + v_ptr[0] = 0; | |
| 1852 | + v_ptr[size - 1] = 0; | |
| 1853 | + | |
| 1854 | + /* | |
| 1855 | + * An unaligned access past the requested vmalloc size. | |
| 1856 | + * Only generic KASAN can precisely detect these. | |
| 1857 | + */ | |
| 1858 | + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) | |
| 1859 | + KUNIT_EXPECT_KASAN_FAIL(test, ((volatile char *)v_ptr)[size]); | |
| 1860 | + | |
| 1861 | + /* An aligned access into the first out-of-bounds granule. */ | |
| 1862 | + size = round_up(size, KASAN_GRANULE_SIZE); | |
| 1863 | + KUNIT_EXPECT_KASAN_FAIL_READ(test, ((volatile char *)v_ptr)[size]); | |
| 1864 | +} | |
| 1865 | + | |
| 1843 | 1866 | static void vmalloc_oob(struct kunit *test) |
| 1844 | 1867 | { |
| 1845 | 1868 | char *v_ptr, *p_ptr; |
| @@ -1856,24 +1879,21 @@ static void vmalloc_oob(struct kunit *test) · 10 unmodified lines | ||
| 1856 | 1879 | |
| 1857 | 1880 | OPTIMIZER_HIDE_VAR(v_ptr); |
| 1858 | 1881 | |
| 1859 | − /* | |
| 1860 | − * We have to be careful not to hit the guard page in vmalloc tests. | |
| 1861 | − * The MMU will catch that and crash us. | |
| 1862 | − */ | |
| 1882 | + vmalloc_oob_helper(test, v_ptr, size); | |
| 1863 | 1883 | |
| 1864 | − /* Make sure in-bounds accesses are valid. */ | |
| 1865 | − v_ptr[0] = 0; | |
| 1866 | − v_ptr[size - 1] = 0; | |
| 1884 | + size -= KASAN_GRANULE_SIZE + 1; | |
| 1885 | + v_ptr = vrealloc(v_ptr, size, GFP_KERNEL); | |
| 1886 | + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, v_ptr); | |
| 1867 | 1887 | |
| 1868 | − /* | |
| 1869 | − * An unaligned access past the requested vmalloc size. | |
| 1870 | − * Only generic KASAN can precisely detect these. | |
| 1871 | − */ | |
| 1872 | − if (IS_ENABLED(CONFIG_KASAN_GENERIC)) | |
| 1873 | − KUNIT_EXPECT_KASAN_FAIL(test, ((volatile char *)v_ptr)[size]); | |
| 1888 | + OPTIMIZER_HIDE_VAR(v_ptr); | |
| 1874 | 1889 | |
| 1875 | − /* An aligned access into the first out-of-bounds granule. */ | |
| 1876 | − KUNIT_EXPECT_KASAN_FAIL_READ(test, ((volatile char *)v_ptr)[size + 5]); | |
| 1890 | + vmalloc_oob_helper(test, v_ptr, size); | |
| 1891 | + | |
| 1892 | + size += 2 * KASAN_GRANULE_SIZE + 2; | |
| 1893 | + v_ptr = vrealloc(v_ptr, size, GFP_KERNEL); | |
| 1894 | + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, v_ptr); | |
| 1895 | + | |
| 1896 | + vmalloc_oob_helper(test, v_ptr, size); | |
| 1877 | 1897 | |
| 1878 | 1898 | /* Check that in-bounds accesses to the physical page are valid. */ |
| 1879 | 1899 | page = vmalloc_to_page(v_ptr); |
| @@ -203,7 +203,7 @@ static inline void fail_non_kasan_kunit_test(void) { } · 202 unmodified lines | ||
| 203 | 203 | |
| 204 | 204 | static DEFINE_RAW_SPINLOCK(report_lock); |
| 205 | 205 | |
| 206 | −static void start_report(unsigned long *flags, bool sync) | |
| 206 | +static void start_report(unsigned long *flags) | |
| 207 | 207 | { |
| 208 | 208 | fail_non_kasan_kunit_test(); |
| 209 | 209 | /* Respect the /proc/sys/kernel/traceoff_on_warning interface. */ |
| @@ -543,7 +543,7 @@ void kasan_report_invalid_free(void *ptr, unsigned long ip, enum kasan_report_ty · 333 unmodified lines | ||
| 543 | 543 | if (unlikely(!report_enabled())) |
| 544 | 544 | return; |
| 545 | 545 | |
| 546 | − start_report(&flags, true); | |
| 546 | + start_report(&flags); | |
| 547 | 547 | |
| 548 | 548 | __memset(&info, 0, sizeof(info)); |
| 549 | 549 | info.type = type; |
| @@ -581,7 +581,7 @@ bool kasan_report(const void *addr, size_t size, bool is_write, · 31 unmodified lines | ||
| 581 | 581 | goto out; |
| 582 | 582 | } |
| 583 | 583 | |
| 584 | − start_report(&irq_flags, true); | |
| 584 | + start_report(&irq_flags); | |
| 585 | 585 | |
| 586 | 586 | __memset(&info, 0, sizeof(info)); |
| 587 | 587 | info.type = KASAN_REPORT_ACCESS; |
| @@ -615,7 +615,7 @@ void kasan_report_async(void) · 27 unmodified lines | ||
| 615 | 615 | if (unlikely(!report_enabled())) |
| 616 | 616 | return; |
| 617 | 617 | |
| 618 | − start_report(&flags, false); | |
| 618 | + start_report(&flags); | |
| 619 | 619 | pr_err("BUG: KASAN: invalid-access\n"); |
| 620 | 620 | pr_err("Asynchronous fault: no details available\n"); |
| 621 | 621 | pr_err("\n"); |
| @@ -305,7 +305,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 304 unmodified lines | ||
| 305 | 305 | pte_t pte; |
| 306 | 306 | int index; |
| 307 | 307 | |
| 308 | − arch_leave_lazy_mmu_mode(); | |
| 308 | + lazy_mmu_mode_pause(); | |
| 309 | 309 | |
| 310 | 310 | index = PFN_DOWN(addr - data->start); |
| 311 | 311 | page = data->pages[index]; |
| @@ -319,7 +319,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 7 unmodified lines | ||
| 319 | 319 | } |
| 320 | 320 | spin_unlock(&init_mm.page_table_lock); |
| 321 | 321 | |
| 322 | − arch_enter_lazy_mmu_mode(); | |
| 322 | + lazy_mmu_mode_resume(); | |
| 323 | 323 | |
| 324 | 324 | return 0; |
| 325 | 325 | } |
| @@ -471,7 +471,7 @@ static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 145 unmodified lines | ||
| 471 | 471 | pte_t pte; |
| 472 | 472 | int none; |
| 473 | 473 | |
| 474 | − arch_leave_lazy_mmu_mode(); | |
| 474 | + lazy_mmu_mode_pause(); | |
| 475 | 475 | |
| 476 | 476 | spin_lock(&init_mm.page_table_lock); |
| 477 | 477 | pte = ptep_get(ptep); |
| @@ -483,7 +483,7 @@ static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 5 unmodified lines | ||
| 483 | 483 | if (likely(!none)) |
| 484 | 484 | __free_page(pfn_to_page(pte_pfn(pte))); |
| 485 | 485 | |
| 486 | − arch_enter_lazy_mmu_mode(); | |
| 486 | + lazy_mmu_mode_resume(); | |
| 487 | 487 | |
| 488 | 488 | return 0; |
| 489 | 489 | } |
| @@ -905,7 +905,7 @@ static void toggle_allocation_gate(struct work_struct *work) · 904 unmodified lines | ||
| 905 | 905 | /* Disable static key and reset timer. */ |
| 906 | 906 | static_branch_disable(&kfence_allocation_key); |
| 907 | 907 | #endif |
| 908 | − queue_delayed_work(system_unbound_wq, &kfence_timer, | |
| 908 | + queue_delayed_work(system_dfl_wq, &kfence_timer, | |
| 909 | 909 | msecs_to_jiffies(kfence_sample_interval)); |
| 910 | 910 | } |
| 911 | 911 | |
| @@ -955,7 +955,7 @@ static void kfence_init_enable(void) · 43 unmodified lines | ||
| 955 | 955 | #endif |
| 956 | 956 | |
| 957 | 957 | WRITE_ONCE(kfence_enabled, true); |
| 958 | − queue_delayed_work(system_unbound_wq, &kfence_timer, 0); | |
| 958 | + queue_delayed_work(system_dfl_wq, &kfence_timer, 0); | |
| 959 | 959 | |
| 960 | 960 | pr_info("initialized - using %lu bytes for %d objects at 0x%p-0x%p\n", KFENCE_POOL_SIZE, |
| 961 | 961 | CONFIG_KFENCE_NUM_OBJECTS, (void *)__kfence_pool, |
| @@ -1051,7 +1051,7 @@ static int kfence_enable_late(void) · 89 unmodified lines | ||
| 1051 | 1051 | return kfence_init_late(); |
| 1052 | 1052 | |
| 1053 | 1053 | WRITE_ONCE(kfence_enabled, true); |
| 1054 | − queue_delayed_work(system_unbound_wq, &kfence_timer, 0); | |
| 1054 | + queue_delayed_work(system_dfl_wq, &kfence_timer, 0); | |
| 1055 | 1055 | pr_info("re-enabled\n"); |
| 1056 | 1056 | return 0; |
| 1057 | 1057 | } |
| @@ -22,6 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <linux/dax.h> |
| 23 | 23 | #include <linux/ksm.h> |
| 24 | 24 | #include <linux/pgalloc.h> |
| 25 | +#include <linux/backing-dev.h> | |
| 25 | 26 | |
| 26 | 27 | #include <asm/tlb.h> |
| 27 | 28 | #include "internal.h" |
| @@ -58,6 +59,7 @@ enum scan_result { · 30 unmodified lines | ||
| 58 | 59 | SCAN_STORE_FAILED, |
| 59 | 60 | SCAN_COPY_MC, |
| 60 | 61 | SCAN_PAGE_FILLED, |
| 62 | + SCAN_PAGE_DIRTY_OR_WRITEBACK, | |
| 61 | 63 | }; |
| 62 | 64 | |
| 63 | 65 | #define CREATE_TRACE_POINTS |
| @@ -535,17 +537,16 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, · 471 unmodified lines | ||
| 535 | 537 | } |
| 536 | 538 | } |
| 537 | 539 | |
| 538 | −static int __collapse_huge_page_isolate(struct vm_area_struct *vma, | |
| 539 | − unsigned long start_addr, | |
| 540 | − pte_t *pte, | |
| 541 | − struct collapse_control *cc, | |
| 542 | − struct list_head *compound_pagelist) | |
| 540 | +static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, | |
| 541 | + unsigned long start_addr, pte_t *pte, struct collapse_control *cc, | |
| 542 | + struct list_head *compound_pagelist) | |
| 543 | 543 | { |
| 544 | 544 | struct page *page = NULL; |
| 545 | 545 | struct folio *folio = NULL; |
| 546 | 546 | unsigned long addr = start_addr; |
| 547 | 547 | pte_t *_pte; |
| 548 | − int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; | |
| 548 | + int none_or_zero = 0, shared = 0, referenced = 0; | |
| 549 | + enum scan_result result = SCAN_FAIL; | |
| 549 | 550 | |
| 550 | 551 | for (_pte = pte; _pte < pte + HPAGE_PMD_NR; |
| 551 | 552 | _pte++, addr += PAGE_SIZE) { |
| @@ -778,13 +779,13 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, · 226 unmodified lines | ||
| 778 | 779 | * @ptl: lock on raw pages' PTEs |
| 779 | 780 | * @compound_pagelist: list that stores compound pages |
| 780 | 781 | */ |
| 781 | −static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, | |
| 782 | +static enum scan_result __collapse_huge_page_copy(pte_t *pte, struct folio *folio, | |
| 782 | 783 | pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, |
| 783 | 784 | unsigned long address, spinlock_t *ptl, |
| 784 | 785 | struct list_head *compound_pagelist) |
| 785 | 786 | { |
| 786 | 787 | unsigned int i; |
| 787 | − int result = SCAN_SUCCEED; | |
| 788 | + enum scan_result result = SCAN_SUCCEED; | |
| 788 | 789 | |
| 789 | 790 | /* |
| 790 | 791 | * Copying pages' contents is subject to memory poison at any iteration. |
| @@ -826,7 +827,7 @@ static void khugepaged_alloc_sleep(void) · 35 unmodified lines | ||
| 826 | 827 | remove_wait_queue(&khugepaged_wait, &wait); |
| 827 | 828 | } |
| 828 | 829 | |
| 829 | −struct collapse_control khugepaged_collapse_control = { | |
| 830 | +static struct collapse_control khugepaged_collapse_control = { | |
| 830 | 831 | .is_khugepaged = true, |
| 831 | 832 | }; |
| 832 | 833 | |
| @@ -896,10 +897,8 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) · 63 unmodified lines | ||
| 896 | 897 | * Returns enum scan_result value. |
| 897 | 898 | */ |
| 898 | 899 | |
| 899 | −static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, | |
| 900 | − bool expect_anon, | |
| 901 | − struct vm_area_struct **vmap, | |
| 902 | − struct collapse_control *cc) | |
| 900 | +static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, | |
| 901 | + bool expect_anon, struct vm_area_struct **vmap, struct collapse_control *cc) | |
| 903 | 902 | { |
| 904 | 903 | struct vm_area_struct *vma; |
| 905 | 904 | enum tva_type type = cc->is_khugepaged ? TVA_KHUGEPAGED : |
| @@ -928,7 +927,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, · 22 unmodified lines | ||
| 928 | 927 | return SCAN_SUCCEED; |
| 929 | 928 | } |
| 930 | 929 | |
| 931 | −static inline int check_pmd_state(pmd_t *pmd) | |
| 930 | +static inline enum scan_result check_pmd_state(pmd_t *pmd) | |
| 932 | 931 | { |
| 933 | 932 | pmd_t pmde = pmdp_get_lockless(pmd); |
| 934 | 933 | |
| @@ -951,9 +950,8 @@ static inline int check_pmd_state(pmd_t *pmd) · 16 unmodified lines | ||
| 951 | 950 | return SCAN_SUCCEED; |
| 952 | 951 | } |
| 953 | 952 | |
| 954 | −static int find_pmd_or_thp_or_none(struct mm_struct *mm, | |
| 955 | − unsigned long address, | |
| 956 | − pmd_t **pmd) | |
| 953 | +static enum scan_result find_pmd_or_thp_or_none(struct mm_struct *mm, | |
| 954 | + unsigned long address, pmd_t **pmd) | |
| 957 | 955 | { |
| 958 | 956 | *pmd = mm_find_pmd(mm, address); |
| 959 | 957 | if (!*pmd) |
| @@ -962,12 +960,11 @@ static int find_pmd_or_thp_or_none(struct mm_struct *mm, · 2 unmodified lines | ||
| 962 | 960 | return check_pmd_state(*pmd); |
| 963 | 961 | } |
| 964 | 962 | |
| 965 | −static int check_pmd_still_valid(struct mm_struct *mm, | |
| 966 | − unsigned long address, | |
| 967 | − pmd_t *pmd) | |
| 963 | +static enum scan_result check_pmd_still_valid(struct mm_struct *mm, | |
| 964 | + unsigned long address, pmd_t *pmd) | |
| 968 | 965 | { |
| 969 | 966 | pmd_t *new_pmd; |
| 970 | − int result = find_pmd_or_thp_or_none(mm, address, &new_pmd); | |
| 967 | + enum scan_result result = find_pmd_or_thp_or_none(mm, address, &new_pmd); | |
| 971 | 968 | |
| 972 | 969 | if (result != SCAN_SUCCEED) |
| 973 | 970 | return result; |
| @@ -983,15 +980,14 @@ static int check_pmd_still_valid(struct mm_struct *mm, · 9 unmodified lines | ||
| 983 | 980 | * Called and returns without pte mapped or spinlocks held. |
| 984 | 981 | * Returns result: if not SCAN_SUCCEED, mmap_lock has been released. |
| 985 | 982 | */ |
| 986 | −static int __collapse_huge_page_swapin(struct mm_struct *mm, | |
| 987 | − struct vm_area_struct *vma, | |
| 988 | − unsigned long start_addr, pmd_t *pmd, | |
| 989 | − int referenced) | |
| 983 | +static enum scan_result __collapse_huge_page_swapin(struct mm_struct *mm, | |
| 984 | + struct vm_area_struct *vma, unsigned long start_addr, pmd_t *pmd, | |
| 985 | + int referenced) | |
| 990 | 986 | { |
| 991 | 987 | int swapped_in = 0; |
| 992 | 988 | vm_fault_t ret = 0; |
| 993 | 989 | unsigned long addr, end = start_addr + (HPAGE_PMD_NR * PAGE_SIZE); |
| 994 | − int result; | |
| 990 | + enum scan_result result; | |
| 995 | 991 | pte_t *pte = NULL; |
| 996 | 992 | spinlock_t *ptl; |
| 997 | 993 | |
| @@ -1060,8 +1056,8 @@ out: · 62 unmodified lines | ||
| 1060 | 1056 | return result; |
| 1061 | 1057 | } |
| 1062 | 1058 | |
| 1063 | −static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, | |
| 1064 | − struct collapse_control *cc) | |
| 1059 | +static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, | |
| 1060 | + struct collapse_control *cc) | |
| 1065 | 1061 | { |
| 1066 | 1062 | gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : |
| 1067 | 1063 | GFP_TRANSHUGE); |
| @@ -1088,9 +1084,8 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, · 20 unmodified lines | ||
| 1088 | 1084 | return SCAN_SUCCEED; |
| 1089 | 1085 | } |
| 1090 | 1086 | |
| 1091 | −static int collapse_huge_page(struct mm_struct *mm, unsigned long address, | |
| 1092 | − int referenced, int unmapped, | |
| 1093 | − struct collapse_control *cc) | |
| 1087 | +static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long address, | |
| 1088 | + int referenced, int unmapped, struct collapse_control *cc) | |
| 1094 | 1089 | { |
| 1095 | 1090 | LIST_HEAD(compound_pagelist); |
| 1096 | 1091 | pmd_t *pmd, _pmd; |
| @@ -1098,7 +1093,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · 1 unmodified line | ||
| 1098 | 1093 | pgtable_t pgtable; |
| 1099 | 1094 | struct folio *folio; |
| 1100 | 1095 | spinlock_t *pmd_ptl, *pte_ptl; |
| 1101 | − int result = SCAN_FAIL; | |
| 1096 | + enum scan_result result = SCAN_FAIL; | |
| 1102 | 1097 | struct vm_area_struct *vma; |
| 1103 | 1098 | struct mmu_notifier_range range; |
| 1104 | 1099 | |
| @@ -1244,15 +1239,14 @@ out_nolock: · 139 unmodified lines | ||
| 1244 | 1239 | return result; |
| 1245 | 1240 | } |
| 1246 | 1241 | |
| 1247 | −static int hpage_collapse_scan_pmd(struct mm_struct *mm, | |
| 1248 | − struct vm_area_struct *vma, | |
| 1249 | − unsigned long start_addr, bool *mmap_locked, | |
| 1250 | − struct collapse_control *cc) | |
| 1242 | +static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm, | |
| 1243 | + struct vm_area_struct *vma, unsigned long start_addr, bool *mmap_locked, | |
| 1244 | + struct collapse_control *cc) | |
| 1251 | 1245 | { |
| 1252 | 1246 | pmd_t *pmd; |
| 1253 | 1247 | pte_t *pte, *_pte; |
| 1254 | − int result = SCAN_FAIL, referenced = 0; | |
| 1255 | − int none_or_zero = 0, shared = 0; | |
| 1248 | + int none_or_zero = 0, shared = 0, referenced = 0; | |
| 1249 | + enum scan_result result = SCAN_FAIL; | |
| 1256 | 1250 | struct page *page = NULL; |
| 1257 | 1251 | struct folio *folio = NULL; |
| 1258 | 1252 | unsigned long addr; |
| @@ -1439,8 +1433,8 @@ static void collect_mm_slot(struct mm_slot *slot) · 180 unmodified lines | ||
| 1439 | 1433 | } |
| 1440 | 1434 | |
| 1441 | 1435 | /* folio must be locked, and mmap_lock must be held */ |
| 1442 | −static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, | |
| 1443 | − pmd_t *pmdp, struct folio *folio, struct page *page) | |
| 1436 | +static enum scan_result set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, | |
| 1437 | + pmd_t *pmdp, struct folio *folio, struct page *page) | |
| 1444 | 1438 | { |
| 1445 | 1439 | struct mm_struct *mm = vma->vm_mm; |
| 1446 | 1440 | struct vm_fault vmf = { |
| @@ -1475,22 +1469,11 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, · 28 unmodified lines | ||
| 1475 | 1469 | return SCAN_SUCCEED; |
| 1476 | 1470 | } |
| 1477 | 1471 | |
| 1478 | −/** | |
| 1479 | − * collapse_pte_mapped_thp - Try to collapse a pte-mapped THP for mm at | |
| 1480 | − * address haddr. | |
| 1481 | − * | |
| 1482 | − * @mm: process address space where collapse happens | |
| 1483 | − * @addr: THP collapse address | |
| 1484 | − * @install_pmd: If a huge PMD should be installed | |
| 1485 | − * | |
| 1486 | − * This function checks whether all the PTEs in the PMD are pointing to the | |
| 1487 | − * right THP. If so, retract the page table so the THP can refault in with | |
| 1488 | − * as pmd-mapped. Possibly install a huge PMD mapping the THP. | |
| 1489 | − */ | |
| 1490 | −int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, | |
| 1491 | − bool install_pmd) | |
| 1472 | +static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, | |
| 1473 | + bool install_pmd) | |
| 1492 | 1474 | { |
| 1493 | − int nr_mapped_ptes = 0, result = SCAN_FAIL; | |
| 1475 | + enum scan_result result = SCAN_FAIL; | |
| 1476 | + int nr_mapped_ptes = 0; | |
| 1494 | 1477 | unsigned int nr_batch_ptes; |
| 1495 | 1478 | struct mmu_notifier_range range; |
| 1496 | 1479 | bool notified = false; |
| @@ -1709,6 +1692,24 @@ drop_folio: · 212 unmodified lines | ||
| 1709 | 1692 | return result; |
| 1710 | 1693 | } |
| 1711 | 1694 | |
| 1695 | +/** | |
| 1696 | + * collapse_pte_mapped_thp - Try to collapse a pte-mapped THP for mm at | |
| 1697 | + * address haddr. | |
| 1698 | + * | |
| 1699 | + * @mm: process address space where collapse happens | |
| 1700 | + * @addr: THP collapse address | |
| 1701 | + * @install_pmd: If a huge PMD should be installed | |
| 1702 | + * | |
| 1703 | + * This function checks whether all the PTEs in the PMD are pointing to the | |
| 1704 | + * right THP. If so, retract the page table so the THP can refault in with | |
| 1705 | + * as pmd-mapped. Possibly install a huge PMD mapping the THP. | |
| 1706 | + */ | |
| 1707 | +void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, | |
| 1708 | + bool install_pmd) | |
| 1709 | +{ | |
| 1710 | + try_collapse_pte_mapped_thp(mm, addr, install_pmd); | |
| 1711 | +} | |
| 1712 | + | |
| 1712 | 1713 | /* Can we retract page tables for this file-backed VMA? */ |
| 1713 | 1714 | static bool file_backed_vma_is_retractable(struct vm_area_struct *vma) |
| 1714 | 1715 | { |
| @@ -1854,9 +1855,8 @@ drop_pml: · 139 unmodified lines | ||
| 1854 | 1855 | * + unlock old pages |
| 1855 | 1856 | * + unlock and free huge page; |
| 1856 | 1857 | */ |
| 1857 | −static int collapse_file(struct mm_struct *mm, unsigned long addr, | |
| 1858 | − struct file *file, pgoff_t start, | |
| 1859 | − struct collapse_control *cc) | |
| 1858 | +static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, | |
| 1859 | + struct file *file, pgoff_t start, struct collapse_control *cc) | |
| 1860 | 1860 | { |
| 1861 | 1861 | struct address_space *mapping = file->f_mapping; |
| 1862 | 1862 | struct page *dst; |
| @@ -1864,7 +1864,8 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, · 1 unmodified line | ||
| 1864 | 1864 | pgoff_t index = 0, end = start + HPAGE_PMD_NR; |
| 1865 | 1865 | LIST_HEAD(pagelist); |
| 1866 | 1866 | XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); |
| 1867 | − int nr_none = 0, result = SCAN_SUCCEED; | |
| 1867 | + enum scan_result result = SCAN_SUCCEED; | |
| 1868 | + int nr_none = 0; | |
| 1868 | 1869 | bool is_shmem = shmem_file(file); |
| 1869 | 1870 | |
| 1870 | 1871 | VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem); |
| @@ -1967,11 +1968,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, · 96 unmodified lines | ||
| 1967 | 1968 | */ |
| 1968 | 1969 | xas_unlock_irq(&xas); |
| 1969 | 1970 | filemap_flush(mapping); |
| 1970 | − result = SCAN_FAIL; | |
| 1971 | + result = SCAN_PAGE_DIRTY_OR_WRITEBACK; | |
| 1971 | 1972 | goto xa_unlocked; |
| 1972 | 1973 | } else if (folio_test_writeback(folio)) { |
| 1973 | 1974 | xas_unlock_irq(&xas); |
| 1974 | − result = SCAN_FAIL; | |
| 1975 | + result = SCAN_PAGE_DIRTY_OR_WRITEBACK; | |
| 1975 | 1976 | goto xa_unlocked; |
| 1976 | 1977 | } else if (folio_trylock(folio)) { |
| 1977 | 1978 | folio_get(folio); |
| @@ -2018,7 +2019,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, · 40 unmodified lines | ||
| 2018 | 2019 | * folio is dirty because it hasn't been flushed |
| 2019 | 2020 | * since first write. |
| 2020 | 2021 | */ |
| 2021 | − result = SCAN_FAIL; | |
| 2022 | + result = SCAN_PAGE_DIRTY_OR_WRITEBACK; | |
| 2022 | 2023 | goto out_unlock; |
| 2023 | 2024 | } |
| 2024 | 2025 | |
| @@ -2194,16 +2195,13 @@ immap_locked: · 169 unmodified lines | ||
| 2194 | 2195 | xas_lock_irq(&xas); |
| 2195 | 2196 | } |
| 2196 | 2197 | |
| 2197 | − if (is_shmem) | |
| 2198 | + if (is_shmem) { | |
| 2199 | + lruvec_stat_mod_folio(new_folio, NR_SHMEM, HPAGE_PMD_NR); | |
| 2198 | 2200 | lruvec_stat_mod_folio(new_folio, NR_SHMEM_THPS, HPAGE_PMD_NR); |
| 2199 | − else | |
| 2201 | + } else { | |
| 2200 | 2202 | lruvec_stat_mod_folio(new_folio, NR_FILE_THPS, HPAGE_PMD_NR); |
| 2201 | − | |
| 2202 | − if (nr_none) { | |
| 2203 | − lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, nr_none); | |
| 2204 | − /* nr_none is always 0 for non-shmem. */ | |
| 2205 | − lruvec_stat_mod_folio(new_folio, NR_SHMEM, nr_none); | |
| 2206 | 2203 | } |
| 2204 | + lruvec_stat_mod_folio(new_folio, NR_FILE_PAGES, HPAGE_PMD_NR); | |
| 2207 | 2205 | |
| 2208 | 2206 | /* |
| 2209 | 2207 | * Mark new_folio as uptodate before inserting it into the |
| @@ -2225,7 +2223,7 @@ immap_locked: · 15 unmodified lines | ||
| 2225 | 2223 | |
| 2226 | 2224 | /* |
| 2227 | 2225 | * Remove pte page tables, so we can re-fault the page as huge. |
| 2228 | − * If MADV_COLLAPSE, adjust result to call collapse_pte_mapped_thp(). | |
| 2226 | + * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp(). | |
| 2229 | 2227 | */ |
| 2230 | 2228 | retract_page_tables(mapping, start); |
| 2231 | 2229 | if (cc && !cc->is_khugepaged) |
| @@ -2237,6 +2235,11 @@ immap_locked: · 5 unmodified lines | ||
| 2237 | 2235 | */ |
| 2238 | 2236 | list_for_each_entry_safe(folio, tmp, &pagelist, lru) { |
| 2239 | 2237 | list_del(&folio->lru); |
| 2238 | + lruvec_stat_mod_folio(folio, NR_FILE_PAGES, | |
| 2239 | + -folio_nr_pages(folio)); | |
| 2240 | + if (is_shmem) | |
| 2241 | + lruvec_stat_mod_folio(folio, NR_SHMEM, | |
| 2242 | + -folio_nr_pages(folio)); | |
| 2240 | 2243 | folio->mapping = NULL; |
| 2241 | 2244 | folio_clear_active(folio); |
| 2242 | 2245 | folio_clear_unevictable(folio); |
| @@ -2285,16 +2288,15 @@ out: · 42 unmodified lines | ||
| 2285 | 2288 | return result; |
| 2286 | 2289 | } |
| 2287 | 2290 | |
| 2288 | −static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, | |
| 2289 | − struct file *file, pgoff_t start, | |
| 2290 | − struct collapse_control *cc) | |
| 2291 | +static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, | |
| 2292 | + struct file *file, pgoff_t start, struct collapse_control *cc) | |
| 2291 | 2293 | { |
| 2292 | 2294 | struct folio *folio = NULL; |
| 2293 | 2295 | struct address_space *mapping = file->f_mapping; |
| 2294 | 2296 | XA_STATE(xas, &mapping->i_pages, start); |
| 2295 | 2297 | int present, swap; |
| 2296 | 2298 | int node = NUMA_NO_NODE; |
| 2297 | − int result = SCAN_SUCCEED; | |
| 2299 | + enum scan_result result = SCAN_SUCCEED; | |
| 2298 | 2300 | |
| 2299 | 2301 | present = 0; |
| 2300 | 2302 | swap = 0; |
| @@ -2392,7 +2394,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, · 91 unmodified lines | ||
| 2392 | 2394 | return result; |
| 2393 | 2395 | } |
| 2394 | 2396 | |
| 2395 | −static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, | |
| 2397 | +static unsigned int khugepaged_scan_mm_slot(unsigned int pages, enum scan_result *result, | |
| 2396 | 2398 | struct collapse_control *cc) |
| 2397 | 2399 | __releases(&khugepaged_mm_lock) |
| 2398 | 2400 | __acquires(&khugepaged_mm_lock) |
| @@ -2440,14 +2442,15 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, · 41 unmodified lines | ||
| 2440 | 2442 | break; |
| 2441 | 2443 | } |
| 2442 | 2444 | if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_KHUGEPAGED, PMD_ORDER)) { |
| 2443 | −skip: | |
| 2444 | 2445 | progress++; |
| 2445 | 2446 | continue; |
| 2446 | 2447 | } |
| 2447 | 2448 | hstart = round_up(vma->vm_start, HPAGE_PMD_SIZE); |
| 2448 | 2449 | hend = round_down(vma->vm_end, HPAGE_PMD_SIZE); |
| 2449 | − if (khugepaged_scan.address > hend) | |
| 2450 | − goto skip; | |
| 2450 | + if (khugepaged_scan.address > hend) { | |
| 2451 | + progress++; | |
| 2452 | + continue; | |
| 2453 | + } | |
| 2451 | 2454 | if (khugepaged_scan.address < hstart) |
| 2452 | 2455 | khugepaged_scan.address = hstart; |
| 2453 | 2456 | VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK); |
| @@ -2476,7 +2479,7 @@ skip: · 22 unmodified lines | ||
| 2476 | 2479 | mmap_read_lock(mm); |
| 2477 | 2480 | if (hpage_collapse_test_exit_or_disable(mm)) |
| 2478 | 2481 | goto breakouterloop; |
| 2479 | − *result = collapse_pte_mapped_thp(mm, | |
| 2482 | + *result = try_collapse_pte_mapped_thp(mm, | |
| 2480 | 2483 | khugepaged_scan.address, false); |
| 2481 | 2484 | if (*result == SCAN_PMD_MAPPED) |
| 2482 | 2485 | *result = SCAN_SUCCEED; |
| @@ -2552,7 +2555,7 @@ static void khugepaged_do_scan(struct collapse_control *cc) · 69 unmodified lines | ||
| 2552 | 2555 | unsigned int progress = 0, pass_through_head = 0; |
| 2553 | 2556 | unsigned int pages = READ_ONCE(khugepaged_pages_to_scan); |
| 2554 | 2557 | bool wait = true; |
| 2555 | − int result = SCAN_SUCCEED; | |
| 2558 | + enum scan_result result = SCAN_SUCCEED; | |
| 2556 | 2559 | |
| 2557 | 2560 | lru_add_drain_all(); |
| 2558 | 2561 | |
| @@ -2747,6 +2750,7 @@ static int madvise_collapse_errno(enum scan_result r) · 188 unmodified lines | ||
| 2747 | 2750 | case SCAN_PAGE_LRU: |
| 2748 | 2751 | case SCAN_DEL_PAGE_LRU: |
| 2749 | 2752 | case SCAN_PAGE_FILLED: |
| 2753 | + case SCAN_PAGE_DIRTY_OR_WRITEBACK: | |
| 2750 | 2754 | return -EAGAIN; |
| 2751 | 2755 | /* |
| 2752 | 2756 | * Other: Trying again likely not to succeed / error intrinsic to |
| @@ -2764,7 +2768,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · 11 unmodified lines | ||
| 2764 | 2768 | struct collapse_control *cc; |
| 2765 | 2769 | struct mm_struct *mm = vma->vm_mm; |
| 2766 | 2770 | unsigned long hstart, hend, addr; |
| 2767 | − int thps = 0, last_fail = SCAN_FAIL; | |
| 2771 | + enum scan_result last_fail = SCAN_FAIL; | |
| 2772 | + int thps = 0; | |
| 2768 | 2773 | bool mmap_locked = true; |
| 2769 | 2774 | |
| 2770 | 2775 | BUG_ON(vma->vm_start > start); |
| @@ -2785,8 +2790,10 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · 14 unmodified lines | ||
| 2785 | 2790 | hend = end & HPAGE_PMD_MASK; |
| 2786 | 2791 | |
| 2787 | 2792 | for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) { |
| 2788 | − int result = SCAN_FAIL; | |
| 2793 | + enum scan_result result = SCAN_FAIL; | |
| 2794 | + bool triggered_wb = false; | |
| 2789 | 2795 | |
| 2796 | +retry: | |
| 2790 | 2797 | if (!mmap_locked) { |
| 2791 | 2798 | cond_resched(); |
| 2792 | 2799 | mmap_read_lock(mm); |
| @@ -2807,8 +2814,20 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, · 14 unmodified lines | ||
| 2807 | 2814 | |
| 2808 | 2815 | mmap_read_unlock(mm); |
| 2809 | 2816 | mmap_locked = false; |
| 2817 | + *lock_dropped = true; | |
| 2810 | 2818 | result = hpage_collapse_scan_file(mm, addr, file, pgoff, |
| 2811 | 2819 | cc); |
| 2820 | + | |
| 2821 | + if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && | |
| 2822 | + mapping_can_writeback(file->f_mapping)) { | |
| 2823 | + loff_t lstart = (loff_t)pgoff << PAGE_SHIFT; | |
| 2824 | + loff_t lend = lstart + HPAGE_PMD_SIZE - 1; | |
| 2825 | + | |
| 2826 | + filemap_write_and_wait_range(file->f_mapping, lstart, lend); | |
| 2827 | + triggered_wb = true; | |
| 2828 | + fput(file); | |
| 2829 | + goto retry; | |
| 2830 | + } | |
| 2812 | 2831 | fput(file); |
| 2813 | 2832 | } else { |
| 2814 | 2833 | result = hpage_collapse_scan_pmd(mm, vma, addr, |
| @@ -2826,7 +2845,7 @@ handle_result: · 11 unmodified lines | ||
| 2826 | 2845 | case SCAN_PTE_MAPPED_HUGEPAGE: |
| 2827 | 2846 | BUG_ON(mmap_locked); |
| 2828 | 2847 | mmap_read_lock(mm); |
| 2829 | − result = collapse_pte_mapped_thp(mm, addr, true); | |
| 2848 | + result = try_collapse_pte_mapped_thp(mm, addr, true); | |
| 2830 | 2849 | mmap_read_unlock(mm); |
| 2831 | 2850 | goto handle_result; |
| 2832 | 2851 | /* Whitelisted set of results where continuing OK */ |
| @@ -361,7 +361,7 @@ static void test_init_vmalloc(struct kunit *test) · 360 unmodified lines | ||
| 361 | 361 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); |
| 362 | 362 | } |
| 363 | 363 | |
| 364 | −/* Test case: ensure that use-after-free reporting works. */ | |
| 364 | +/* Test case: ensure that use-after-free reporting works for kmalloc. */ | |
| 365 | 365 | static void test_uaf(struct kunit *test) |
| 366 | 366 | { |
| 367 | 367 | EXPECTATION_USE_AFTER_FREE(expect); |
| @@ -378,6 +378,65 @@ static void test_uaf(struct kunit *test) · 10 unmodified lines | ||
| 378 | 378 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); |
| 379 | 379 | } |
| 380 | 380 | |
| 381 | +static void test_uninit_page(struct kunit *test) | |
| 382 | +{ | |
| 383 | + EXPECTATION_UNINIT_VALUE(expect); | |
| 384 | + struct page *page; | |
| 385 | + int *ptr; | |
| 386 | + | |
| 387 | + kunit_info(test, "uninitialized page allocation (UMR report)\n"); | |
| 388 | + page = alloc_pages(GFP_KERNEL, 0); | |
| 389 | + ptr = page_address(page); | |
| 390 | + USE(*ptr); | |
| 391 | + __free_pages(page, 0); | |
| 392 | + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); | |
| 393 | +} | |
| 394 | + | |
| 395 | +static volatile char *test_uaf_pages_helper(int order, int offset) | |
| 396 | +{ | |
| 397 | + struct page *page; | |
| 398 | + volatile char *var; | |
| 399 | + | |
| 400 | + /* Memory is initialized up until __free_pages() thanks to __GFP_ZERO. */ | |
| 401 | + page = alloc_pages(GFP_KERNEL | __GFP_ZERO, order); | |
| 402 | + var = page_address(page) + offset; | |
| 403 | + __free_pages(page, order); | |
| 404 | + | |
| 405 | + return var; | |
| 406 | +} | |
| 407 | + | |
| 408 | +/* Test case: ensure that use-after-free reporting works for a freed page. */ | |
| 409 | +static void test_uaf_pages(struct kunit *test) | |
| 410 | +{ | |
| 411 | + EXPECTATION_USE_AFTER_FREE(expect); | |
| 412 | + volatile char value; | |
| 413 | + | |
| 414 | + kunit_info(test, "use-after-free on a freed page (UMR report)\n"); | |
| 415 | + /* Allocate a single page, free it, then try to access it. */ | |
| 416 | + value = *test_uaf_pages_helper(0, 3); | |
| 417 | + USE(value); | |
| 418 | + | |
| 419 | + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); | |
| 420 | +} | |
| 421 | + | |
| 422 | +/* Test case: ensure that UAF reporting works for high order pages. */ | |
| 423 | +static void test_uaf_high_order_pages(struct kunit *test) | |
| 424 | +{ | |
| 425 | + EXPECTATION_USE_AFTER_FREE(expect); | |
| 426 | + volatile char value; | |
| 427 | + | |
| 428 | + kunit_info(test, | |
| 429 | + "use-after-free on a freed high-order page (UMR report)\n"); | |
| 430 | + /* | |
| 431 | + * Create a high-order non-compound page, free it, then try to access | |
| 432 | + * its tail page. | |
| 433 | + */ | |
| 434 | + value = *test_uaf_pages_helper(1, PAGE_SIZE + 3); | |
| 435 | + USE(value); | |
| 436 | + | |
| 437 | + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); | |
| 438 | +} | |
| 439 | + | |
| 381 | 440 | /* |
| 382 | 441 | * Test case: ensure that uninitialized values are propagated through per-CPU |
| 383 | 442 | * memory. |
| @@ -682,7 +741,10 @@ static struct kunit_case kmsan_test_cases[] = { · 298 unmodified lines | ||
| 682 | 741 | KUNIT_CASE(test_uninit_kmsan_check_memory), |
| 683 | 742 | KUNIT_CASE(test_init_kmsan_vmap_vunmap), |
| 684 | 743 | KUNIT_CASE(test_init_vmalloc), |
| 744 | + KUNIT_CASE(test_uninit_page), | |
| 685 | 745 | KUNIT_CASE(test_uaf), |
| 746 | + KUNIT_CASE(test_uaf_pages), | |
| 747 | + KUNIT_CASE(test_uaf_high_order_pages), | |
| 686 | 748 | KUNIT_CASE(test_percpu_propagate), |
| 687 | 749 | KUNIT_CASE(test_printk), |
| 688 | 750 | KUNIT_CASE(test_init_memcpy), |
| @@ -187,7 +187,7 @@ bool list_lru_add_obj(struct list_lru *lru, struct list_head *item) · 186 unmodified lines | ||
| 187 | 187 | |
| 188 | 188 | if (list_lru_memcg_aware(lru)) { |
| 189 | 189 | rcu_read_lock(); |
| 190 | − ret = list_lru_add(lru, item, nid, mem_cgroup_from_slab_obj(item)); | |
| 190 | + ret = list_lru_add(lru, item, nid, mem_cgroup_from_virt(item)); | |
| 191 | 191 | rcu_read_unlock(); |
| 192 | 192 | } else { |
| 193 | 193 | ret = list_lru_add(lru, item, nid, NULL); |
| @@ -224,7 +224,7 @@ bool list_lru_del_obj(struct list_lru *lru, struct list_head *item) · 30 unmodified lines | ||
| 224 | 224 | |
| 225 | 225 | if (list_lru_memcg_aware(lru)) { |
| 226 | 226 | rcu_read_lock(); |
| 227 | − ret = list_lru_del(lru, item, nid, mem_cgroup_from_slab_obj(item)); | |
| 227 | + ret = list_lru_del(lru, item, nid, mem_cgroup_from_virt(item)); | |
| 228 | 228 | rcu_read_unlock(); |
| 229 | 229 | } else { |
| 230 | 230 | ret = list_lru_del(lru, item, nid, NULL); |
| @@ -369,7 +369,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, · 138 unmodified lines | ||
| 369 | 369 | |
| 370 | 370 | xa_for_each(&lru->xa, index, mlru) { |
| 371 | 371 | rcu_read_lock(); |
| 372 | − memcg = mem_cgroup_from_id(index); | |
| 372 | + memcg = mem_cgroup_from_private_id(index); | |
| 373 | 373 | if (!mem_cgroup_tryget(memcg)) { |
| 374 | 374 | rcu_read_unlock(); |
| 375 | 375 | continue; |
| @@ -109,9 +109,7 @@ void anon_vma_name_free(struct kref *kref) · 108 unmodified lines | ||
| 109 | 109 | |
| 110 | 110 | struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma) |
| 111 | 111 | { |
| 112 | − if (!rwsem_is_locked(&vma->vm_mm->mmap_lock)) | |
| 113 | − vma_assert_locked(vma); | |
| 114 | − | |
| 112 | + vma_assert_stabilised(vma); | |
| 115 | 113 | return vma->anon_name; |
| 116 | 114 | } |
| 117 | 115 | |
| @@ -453,7 +451,7 @@ restart: · 335 unmodified lines | ||
| 453 | 451 | if (!start_pte) |
| 454 | 452 | return 0; |
| 455 | 453 | flush_tlb_batched_pending(mm); |
| 456 | − arch_enter_lazy_mmu_mode(); | |
| 454 | + lazy_mmu_mode_enable(); | |
| 457 | 455 | for (; addr < end; pte += nr, addr += nr * PAGE_SIZE) { |
| 458 | 456 | nr = 1; |
| 459 | 457 | ptent = ptep_get(pte); |
| @@ -461,7 +459,7 @@ restart: · 1 unmodified line | ||
| 461 | 459 | if (++batch_count == SWAP_CLUSTER_MAX) { |
| 462 | 460 | batch_count = 0; |
| 463 | 461 | if (need_resched()) { |
| 464 | − arch_leave_lazy_mmu_mode(); | |
| 462 | + lazy_mmu_mode_disable(); | |
| 465 | 463 | pte_unmap_unlock(start_pte, ptl); |
| 466 | 464 | cond_resched(); |
| 467 | 465 | goto restart; |
| @@ -497,7 +495,7 @@ restart: · 29 unmodified lines | ||
| 497 | 495 | if (!folio_trylock(folio)) |
| 498 | 496 | continue; |
| 499 | 497 | folio_get(folio); |
| 500 | − arch_leave_lazy_mmu_mode(); | |
| 498 | + lazy_mmu_mode_disable(); | |
| 501 | 499 | pte_unmap_unlock(start_pte, ptl); |
| 502 | 500 | start_pte = NULL; |
| 503 | 501 | err = split_folio(folio); |
| @@ -508,7 +506,7 @@ restart: · 4 unmodified lines | ||
| 508 | 506 | if (!start_pte) |
| 509 | 507 | break; |
| 510 | 508 | flush_tlb_batched_pending(mm); |
| 511 | − arch_enter_lazy_mmu_mode(); | |
| 509 | + lazy_mmu_mode_enable(); | |
| 512 | 510 | if (!err) |
| 513 | 511 | nr = 0; |
| 514 | 512 | continue; |
| @@ -556,7 +554,7 @@ restart: · 41 unmodified lines | ||
| 556 | 554 | } |
| 557 | 555 | |
| 558 | 556 | if (start_pte) { |
| 559 | − arch_leave_lazy_mmu_mode(); | |
| 557 | + lazy_mmu_mode_disable(); | |
| 560 | 558 | pte_unmap_unlock(start_pte, ptl); |
| 561 | 559 | } |
| 562 | 560 | if (pageout) |
| @@ -675,7 +673,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, · 112 unmodified lines | ||
| 675 | 673 | if (!start_pte) |
| 676 | 674 | return 0; |
| 677 | 675 | flush_tlb_batched_pending(mm); |
| 678 | − arch_enter_lazy_mmu_mode(); | |
| 676 | + lazy_mmu_mode_enable(); | |
| 679 | 677 | for (; addr != end; pte += nr, addr += PAGE_SIZE * nr) { |
| 680 | 678 | nr = 1; |
| 681 | 679 | ptent = ptep_get(pte); |
| @@ -694,7 +692,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, · 12 unmodified lines | ||
| 694 | 692 | max_nr = (end - addr) / PAGE_SIZE; |
| 695 | 693 | nr = swap_pte_batch(pte, max_nr, ptent); |
| 696 | 694 | nr_swap -= nr; |
| 697 | − free_swap_and_cache_nr(entry, nr); | |
| 695 | + swap_put_entries_direct(entry, nr); | |
| 698 | 696 | clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); |
| 699 | 697 | } else if (softleaf_is_hwpoison(entry) || |
| 700 | 698 | softleaf_is_poison_marker(entry)) { |
| @@ -724,7 +722,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, · 23 unmodified lines | ||
| 724 | 722 | if (!folio_trylock(folio)) |
| 725 | 723 | continue; |
| 726 | 724 | folio_get(folio); |
| 727 | − arch_leave_lazy_mmu_mode(); | |
| 725 | + lazy_mmu_mode_disable(); | |
| 728 | 726 | pte_unmap_unlock(start_pte, ptl); |
| 729 | 727 | start_pte = NULL; |
| 730 | 728 | err = split_folio(folio); |
| @@ -735,7 +733,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, · 4 unmodified lines | ||
| 735 | 733 | if (!start_pte) |
| 736 | 734 | break; |
| 737 | 735 | flush_tlb_batched_pending(mm); |
| 738 | − arch_enter_lazy_mmu_mode(); | |
| 736 | + lazy_mmu_mode_enable(); | |
| 739 | 737 | if (!err) |
| 740 | 738 | nr = 0; |
| 741 | 739 | continue; |
| @@ -775,7 +773,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, · 33 unmodified lines | ||
| 775 | 773 | if (nr_swap) |
| 776 | 774 | add_mm_counter(mm, MM_SWAPENTS, nr_swap); |
| 777 | 775 | if (start_pte) { |
| 778 | − arch_leave_lazy_mmu_mode(); | |
| 776 | + lazy_mmu_mode_disable(); | |
| 779 | 777 | pte_unmap_unlock(start_pte, ptl); |
| 780 | 778 | } |
| 781 | 779 | cond_resched(); |
| @@ -1867,7 +1865,7 @@ static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior) · 1085 unmodified lines | ||
| 1867 | 1865 | * madvise_should_skip() - Return if the request is invalid or nothing. |
| 1868 | 1866 | * @start: Start address of madvise-requested address range. |
| 1869 | 1867 | * @len_in: Length of madvise-requested address range. |
| 1870 | − * @behavior: Requested madvise behavor. | |
| 1868 | + * @behavior: Requested madvise behavior. | |
| 1871 | 1869 | * @err: Pointer to store an error code from the check. |
| 1872 | 1870 | * |
| 1873 | 1871 | * If the specified behaviour is invalid or nothing would occur, we skip the |
| @@ -773,7 +773,7 @@ bool __init_memblock memblock_validate_numa_coverage(unsigned long threshold_byt · 772 unmodified lines | ||
| 773 | 773 | unsigned long start_pfn, end_pfn, mem_size_mb; |
| 774 | 774 | int nid, i; |
| 775 | 775 | |
| 776 | − /* calculate lose page */ | |
| 776 | + /* calculate lost page */ | |
| 777 | 777 | for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { |
| 778 | 778 | if (!numa_valid_node(nid)) |
| 779 | 779 | nr_pages += end_pfn - start_pfn; |
| @@ -2414,7 +2414,7 @@ EXPORT_SYMBOL_GPL(reserve_mem_find_by_name); · 1634 unmodified lines | ||
| 2414 | 2414 | |
| 2415 | 2415 | /** |
| 2416 | 2416 | * reserve_mem_release_by_name - Release reserved memory region with a given name |
| 2417 | − * @name: The name that is attatched to a reserved memory region | |
| 2417 | + * @name: The name that is attached to a reserved memory region | |
| 2418 | 2418 | * |
| 2419 | 2419 | * Forcibly release the pages in the reserved memory region so that those memory |
| 2420 | 2420 | * can be used as free memory. After released the reserved region size becomes 0. |
| @@ -427,6 +427,28 @@ static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css, · 426 unmodified lines | ||
| 427 | 427 | } |
| 428 | 428 | #endif |
| 429 | 429 | |
| 430 | +static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) | |
| 431 | +{ | |
| 432 | + unsigned long val; | |
| 433 | + | |
| 434 | + if (mem_cgroup_is_root(memcg)) { | |
| 435 | + /* | |
| 436 | + * Approximate root's usage from global state. This isn't | |
| 437 | + * perfect, but the root usage was always an approximation. | |
| 438 | + */ | |
| 439 | + val = global_node_page_state(NR_FILE_PAGES) + | |
| 440 | + global_node_page_state(NR_ANON_MAPPED); | |
| 441 | + if (swap) | |
| 442 | + val += total_swap_pages - get_nr_swap_pages(); | |
| 443 | + } else { | |
| 444 | + if (!swap) | |
| 445 | + val = page_counter_read(&memcg->memory); | |
| 446 | + else | |
| 447 | + val = page_counter_read(&memcg->memsw); | |
| 448 | + } | |
| 449 | + return val; | |
| 450 | +} | |
| 451 | + | |
| 430 | 452 | static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap) |
| 431 | 453 | { |
| 432 | 454 | struct mem_cgroup_threshold_ary *t; |
| @@ -613,14 +635,14 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry) · 180 unmodified lines | ||
| 613 | 635 | * have an ID allocated to it anymore, charge the closest online |
| 614 | 636 | * ancestor for the swap instead and transfer the memory+swap charge. |
| 615 | 637 | */ |
| 616 | − swap_memcg = mem_cgroup_id_get_online(memcg); | |
| 638 | + swap_memcg = mem_cgroup_private_id_get_online(memcg); | |
| 617 | 639 | nr_entries = folio_nr_pages(folio); |
| 618 | 640 | /* Get references for the tail pages, too */ |
| 619 | 641 | if (nr_entries > 1) |
| 620 | − mem_cgroup_id_get_many(swap_memcg, nr_entries - 1); | |
| 642 | + mem_cgroup_private_id_get_many(swap_memcg, nr_entries - 1); | |
| 621 | 643 | mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries); |
| 622 | 644 | |
| 623 | − swap_cgroup_record(folio, mem_cgroup_id(swap_memcg), entry); | |
| 645 | + swap_cgroup_record(folio, mem_cgroup_private_id(swap_memcg), entry); | |
| 624 | 646 | |
| 625 | 647 | folio_unqueue_deferred_split(folio); |
| 626 | 648 | folio->memcg_data = 0; |
| @@ -22,15 +22,13 @@ · 21 unmodified lines | ||
| 22 | 22 | iter != NULL; \ |
| 23 | 23 | iter = mem_cgroup_iter(NULL, iter, NULL)) |
| 24 | 24 | |
| 25 | −unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap); | |
| 26 | − | |
| 27 | 25 | void drain_all_stock(struct mem_cgroup *root_memcg); |
| 28 | 26 | |
| 29 | 27 | unsigned long memcg_events(struct mem_cgroup *memcg, int event); |
| 30 | 28 | int memory_stat_show(struct seq_file *m, void *v); |
| 31 | 29 | |
| 32 | −void mem_cgroup_id_get_many(struct mem_cgroup *memcg, unsigned int n); | |
| 33 | −struct mem_cgroup *mem_cgroup_id_get_online(struct mem_cgroup *memcg); | |
| 30 | +void mem_cgroup_private_id_get_many(struct mem_cgroup *memcg, unsigned int n); | |
| 31 | +struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg); | |
| 34 | 32 | |
| 35 | 33 | /* Cgroup v1-specific declarations */ |
| 36 | 34 | #ifdef CONFIG_MEMCG_V1 |
| @@ -646,7 +646,7 @@ static void flush_memcg_stats_dwork(struct work_struct *w) · 645 unmodified lines | ||
| 646 | 646 | * in latency-sensitive paths is as cheap as possible. |
| 647 | 647 | */ |
| 648 | 648 | __mem_cgroup_flush_stats(root_mem_cgroup, true); |
| 649 | − queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); | |
| 649 | + queue_delayed_work(system_dfl_wq, &stats_flush_dwork, FLUSH_TIME); | |
| 650 | 650 | } |
| 651 | 651 | |
| 652 | 652 | unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) |
| @@ -816,7 +816,7 @@ void mod_lruvec_kmem_state(void *p, enum node_stat_item idx, int val) · 163 unmodified lines | ||
| 816 | 816 | struct lruvec *lruvec; |
| 817 | 817 | |
| 818 | 818 | rcu_read_lock(); |
| 819 | − memcg = mem_cgroup_from_slab_obj(p); | |
| 819 | + memcg = mem_cgroup_from_virt(p); | |
| 820 | 820 | |
| 821 | 821 | /* |
| 822 | 822 | * Untracked pages have no memcg, no lruvec. Update only the |
| @@ -1639,11 +1639,6 @@ unsigned long mem_cgroup_get_max(struct mem_cgroup *memcg) · 816 unmodified lines | ||
| 1639 | 1639 | return max; |
| 1640 | 1640 | } |
| 1641 | 1641 | |
| 1642 | −unsigned long mem_cgroup_size(struct mem_cgroup *memcg) | |
| 1643 | −{ | |
| 1644 | − return page_counter_read(&memcg->memory); | |
| 1645 | −} | |
| 1646 | − | |
| 1647 | 1642 | void __memcg_memory_event(struct mem_cgroup *memcg, |
| 1648 | 1643 | enum memcg_memory_event event, bool allow_spinning) |
| 1649 | 1644 | { |
| @@ -2658,7 +2653,7 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p) · 1008 unmodified lines | ||
| 2658 | 2653 | * The caller must ensure the memcg lifetime, e.g. by taking rcu_read_lock(), |
| 2659 | 2654 | * cgroup_mutex, etc. |
| 2660 | 2655 | */ |
| 2661 | −struct mem_cgroup *mem_cgroup_from_slab_obj(void *p) | |
| 2656 | +struct mem_cgroup *mem_cgroup_from_virt(void *p) | |
| 2662 | 2657 | { |
| 2663 | 2658 | struct slab *slab; |
| 2664 | 2659 | |
| @@ -3320,28 +3315,6 @@ void folio_split_memcg_refs(struct folio *folio, unsigned old_order, · 655 unmodified lines | ||
| 3320 | 3315 | css_get_many(&__folio_memcg(folio)->css, new_refs); |
| 3321 | 3316 | } |
| 3322 | 3317 | |
| 3323 | −unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) | |
| 3324 | −{ | |
| 3325 | − unsigned long val; | |
| 3326 | − | |
| 3327 | − if (mem_cgroup_is_root(memcg)) { | |
| 3328 | − /* | |
| 3329 | − * Approximate root's usage from global state. This isn't | |
| 3330 | − * perfect, but the root usage was always an approximation. | |
| 3331 | − */ | |
| 3332 | − val = global_node_page_state(NR_FILE_PAGES) + | |
| 3333 | − global_node_page_state(NR_ANON_MAPPED); | |
| 3334 | − if (swap) | |
| 3335 | − val += total_swap_pages - get_nr_swap_pages(); | |
| 3336 | − } else { | |
| 3337 | − if (!swap) | |
| 3338 | − val = page_counter_read(&memcg->memory); | |
| 3339 | − else | |
| 3340 | − val = page_counter_read(&memcg->memsw); | |
| 3341 | − } | |
| 3342 | − return val; | |
| 3343 | −} | |
| 3344 | − | |
| 3345 | 3318 | static int memcg_online_kmem(struct mem_cgroup *memcg) |
| 3346 | 3319 | { |
| 3347 | 3320 | struct obj_cgroup *objcg; |
| @@ -3629,38 +3602,38 @@ static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg) · 281 unmodified lines | ||
| 3629 | 3602 | */ |
| 3630 | 3603 | |
| 3631 | 3604 | #define MEM_CGROUP_ID_MAX ((1UL << MEM_CGROUP_ID_SHIFT) - 1) |
| 3632 | −static DEFINE_XARRAY_ALLOC1(mem_cgroup_ids); | |
| 3605 | +static DEFINE_XARRAY_ALLOC1(mem_cgroup_private_ids); | |
| 3633 | 3606 | |
| 3634 | −static void mem_cgroup_id_remove(struct mem_cgroup *memcg) | |
| 3607 | +static void mem_cgroup_private_id_remove(struct mem_cgroup *memcg) | |
| 3635 | 3608 | { |
| 3636 | 3609 | if (memcg->id.id > 0) { |
| 3637 | − xa_erase(&mem_cgroup_ids, memcg->id.id); | |
| 3610 | + xa_erase(&mem_cgroup_private_ids, memcg->id.id); | |
| 3638 | 3611 | memcg->id.id = 0; |
| 3639 | 3612 | } |
| 3640 | 3613 | } |
| 3641 | 3614 | |
| 3642 | −void __maybe_unused mem_cgroup_id_get_many(struct mem_cgroup *memcg, | |
| 3615 | +void __maybe_unused mem_cgroup_private_id_get_many(struct mem_cgroup *memcg, | |
| 3643 | 3616 | unsigned int n) |
| 3644 | 3617 | { |
| 3645 | 3618 | refcount_add(n, &memcg->id.ref); |
| 3646 | 3619 | } |
| 3647 | 3620 | |
| 3648 | −static void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n) | |
| 3621 | +static void mem_cgroup_private_id_put_many(struct mem_cgroup *memcg, unsigned int n) | |
| 3649 | 3622 | { |
| 3650 | 3623 | if (refcount_sub_and_test(n, &memcg->id.ref)) { |
| 3651 | − mem_cgroup_id_remove(memcg); | |
| 3624 | + mem_cgroup_private_id_remove(memcg); | |
| 3652 | 3625 | |
| 3653 | 3626 | /* Memcg ID pins CSS */ |
| 3654 | 3627 | css_put(&memcg->css); |
| 3655 | 3628 | } |
| 3656 | 3629 | } |
| 3657 | 3630 | |
| 3658 | −static inline void mem_cgroup_id_put(struct mem_cgroup *memcg) | |
| 3631 | +static inline void mem_cgroup_private_id_put(struct mem_cgroup *memcg) | |
| 3659 | 3632 | { |
| 3660 | − mem_cgroup_id_put_many(memcg, 1); | |
| 3633 | + mem_cgroup_private_id_put_many(memcg, 1); | |
| 3661 | 3634 | } |
| 3662 | 3635 | |
| 3663 | −struct mem_cgroup *mem_cgroup_id_get_online(struct mem_cgroup *memcg) | |
| 3636 | +struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg) | |
| 3664 | 3637 | { |
| 3665 | 3638 | while (!refcount_inc_not_zero(&memcg->id.ref)) { |
| 3666 | 3639 | /* |
| @@ -3679,39 +3652,35 @@ struct mem_cgroup *mem_cgroup_id_get_online(struct mem_cgroup *memcg) · 12 unmodified lines | ||
| 3679 | 3652 | } |
| 3680 | 3653 | |
| 3681 | 3654 | /** |
| 3682 | − * mem_cgroup_from_id - look up a memcg from a memcg id | |
| 3655 | + * mem_cgroup_from_private_id - look up a memcg from a memcg id | |
| 3683 | 3656 | * @id: the memcg id to look up |
| 3684 | 3657 | * |
| 3685 | 3658 | * Caller must hold rcu_read_lock(). |
| 3686 | 3659 | */ |
| 3687 | −struct mem_cgroup *mem_cgroup_from_id(unsigned short id) | |
| 3660 | +struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id) | |
| 3688 | 3661 | { |
| 3689 | 3662 | WARN_ON_ONCE(!rcu_read_lock_held()); |
| 3690 | − return xa_load(&mem_cgroup_ids, id); | |
| 3663 | + return xa_load(&mem_cgroup_private_ids, id); | |
| 3691 | 3664 | } |
| 3692 | 3665 | |
| 3693 | −#ifdef CONFIG_SHRINKER_DEBUG | |
| 3694 | −struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino) | |
| 3666 | +struct mem_cgroup *mem_cgroup_get_from_id(u64 id) | |
| 3695 | 3667 | { |
| 3696 | 3668 | struct cgroup *cgrp; |
| 3697 | 3669 | struct cgroup_subsys_state *css; |
| 3698 | − struct mem_cgroup *memcg; | |
| 3670 | + struct mem_cgroup *memcg = NULL; | |
| 3699 | 3671 | |
| 3700 | − cgrp = cgroup_get_from_id(ino); | |
| 3672 | + cgrp = cgroup_get_from_id(id); | |
| 3701 | 3673 | if (IS_ERR(cgrp)) |
| 3702 | − return ERR_CAST(cgrp); | |
| 3674 | + return NULL; | |
| 3703 | 3675 | |
| 3704 | 3676 | css = cgroup_get_e_css(cgrp, &memory_cgrp_subsys); |
| 3705 | 3677 | if (css) |
| 3706 | 3678 | memcg = container_of(css, struct mem_cgroup, css); |
| 3707 | − else | |
| 3708 | − memcg = ERR_PTR(-ENOENT); | |
| 3709 | 3679 | |
| 3710 | 3680 | cgroup_put(cgrp); |
| 3711 | 3681 | |
| 3712 | 3682 | return memcg; |
| 3713 | 3683 | } |
| 3714 | −#endif | |
| 3715 | 3684 | |
| 3716 | 3685 | static void free_mem_cgroup_per_node_info(struct mem_cgroup_per_node *pn) |
| 3717 | 3686 | { |
| @@ -3786,7 +3755,7 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent) · 68 unmodified lines | ||
| 3786 | 3755 | if (!memcg) |
| 3787 | 3756 | return ERR_PTR(-ENOMEM); |
| 3788 | 3757 | |
| 3789 | − error = xa_alloc(&mem_cgroup_ids, &memcg->id.id, NULL, | |
| 3758 | + error = xa_alloc(&mem_cgroup_private_ids, &memcg->id.id, NULL, | |
| 3790 | 3759 | XA_LIMIT(1, MEM_CGROUP_ID_MAX), GFP_KERNEL); |
| 3791 | 3760 | if (error) |
| 3792 | 3761 | goto fail; |
| @@ -3846,7 +3815,7 @@ static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent) · 53 unmodified lines | ||
| 3846 | 3815 | lru_gen_init_memcg(memcg); |
| 3847 | 3816 | return memcg; |
| 3848 | 3817 | fail: |
| 3849 | − mem_cgroup_id_remove(memcg); | |
| 3818 | + mem_cgroup_private_id_remove(memcg); | |
| 3850 | 3819 | __mem_cgroup_free(memcg); |
| 3851 | 3820 | return ERR_PTR(error); |
| 3852 | 3821 | } |
| @@ -3920,7 +3889,7 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) · 67 unmodified lines | ||
| 3920 | 3889 | goto offline_kmem; |
| 3921 | 3890 | |
| 3922 | 3891 | if (unlikely(mem_cgroup_is_root(memcg)) && !mem_cgroup_disabled()) |
| 3923 | − queue_delayed_work(system_unbound_wq, &stats_flush_dwork, | |
| 3892 | + queue_delayed_work(system_dfl_wq, &stats_flush_dwork, | |
| 3924 | 3893 | FLUSH_TIME); |
| 3925 | 3894 | lru_gen_online_memcg(memcg); |
| 3926 | 3895 | |
| @@ -3929,7 +3898,7 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) · 2 unmodified lines | ||
| 3929 | 3898 | css_get(css); |
| 3930 | 3899 | |
| 3931 | 3900 | /* |
| 3932 | − * Ensure mem_cgroup_from_id() works once we're fully online. | |
| 3901 | + * Ensure mem_cgroup_from_private_id() works once we're fully online. | |
| 3933 | 3902 | * |
| 3934 | 3903 | * We could do this earlier and require callers to filter with |
| 3935 | 3904 | * css_tryget_online(). But right now there are no users that |
| @@ -3938,13 +3907,13 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) · 2 unmodified lines | ||
| 3938 | 3907 | * publish it here at the end of onlining. This matches the |
| 3939 | 3908 | * regular ID destruction during offlining. |
| 3940 | 3909 | */ |
| 3941 | − xa_store(&mem_cgroup_ids, memcg->id.id, memcg, GFP_KERNEL); | |
| 3910 | + xa_store(&mem_cgroup_private_ids, memcg->id.id, memcg, GFP_KERNEL); | |
| 3942 | 3911 | |
| 3943 | 3912 | return 0; |
| 3944 | 3913 | offline_kmem: |
| 3945 | 3914 | memcg_offline_kmem(memcg); |
| 3946 | 3915 | remove_id: |
| 3947 | − mem_cgroup_id_remove(memcg); | |
| 3916 | + mem_cgroup_private_id_remove(memcg); | |
| 3948 | 3917 | return -ENOMEM; |
| 3949 | 3918 | } |
| 3950 | 3919 | |
| @@ -3967,7 +3936,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) · 16 unmodified lines | ||
| 3967 | 3936 | |
| 3968 | 3937 | drain_all_stock(memcg); |
| 3969 | 3938 | |
| 3970 | − mem_cgroup_id_put(memcg); | |
| 3939 | + mem_cgroup_private_id_put(memcg); | |
| 3971 | 3940 | } |
| 3972 | 3941 | |
| 3973 | 3942 | static void mem_cgroup_css_released(struct cgroup_subsys_state *css) |
| @@ -4854,7 +4823,7 @@ int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, · 880 unmodified lines | ||
| 4854 | 4823 | |
| 4855 | 4824 | id = lookup_swap_cgroup_id(entry); |
| 4856 | 4825 | rcu_read_lock(); |
| 4857 | − memcg = mem_cgroup_from_id(id); | |
| 4826 | + memcg = mem_cgroup_from_private_id(id); | |
| 4858 | 4827 | if (!memcg || !css_tryget_online(&memcg->css)) |
| 4859 | 4828 | memcg = get_mem_cgroup_from_mm(mm); |
| 4860 | 4829 | rcu_read_unlock(); |
| @@ -5051,7 +5020,7 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) · 190 unmodified lines | ||
| 5051 | 5020 | memcg = folio_memcg(old); |
| 5052 | 5021 | /* |
| 5053 | 5022 | * Note that it is normal to see !memcg for a hugetlb folio. |
| 5054 | − * For e.g, itt could have been allocated when memory_hugetlb_accounting | |
| 5023 | + * For e.g, it could have been allocated when memory_hugetlb_accounting | |
| 5055 | 5024 | * was not selected. |
| 5056 | 5025 | */ |
| 5057 | 5026 | VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old); |
| @@ -5257,22 +5226,22 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) · 199 unmodified lines | ||
| 5257 | 5226 | return 0; |
| 5258 | 5227 | } |
| 5259 | 5228 | |
| 5260 | − memcg = mem_cgroup_id_get_online(memcg); | |
| 5229 | + memcg = mem_cgroup_private_id_get_online(memcg); | |
| 5261 | 5230 | |
| 5262 | 5231 | if (!mem_cgroup_is_root(memcg) && |
| 5263 | 5232 | !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { |
| 5264 | 5233 | memcg_memory_event(memcg, MEMCG_SWAP_MAX); |
| 5265 | 5234 | memcg_memory_event(memcg, MEMCG_SWAP_FAIL); |
| 5266 | − mem_cgroup_id_put(memcg); | |
| 5235 | + mem_cgroup_private_id_put(memcg); | |
| 5267 | 5236 | return -ENOMEM; |
| 5268 | 5237 | } |
| 5269 | 5238 | |
| 5270 | 5239 | /* Get references for the tail pages, too */ |
| 5271 | 5240 | if (nr_pages > 1) |
| 5272 | − mem_cgroup_id_get_many(memcg, nr_pages - 1); | |
| 5241 | + mem_cgroup_private_id_get_many(memcg, nr_pages - 1); | |
| 5273 | 5242 | mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); |
| 5274 | 5243 | |
| 5275 | − swap_cgroup_record(folio, mem_cgroup_id(memcg), entry); | |
| 5244 | + swap_cgroup_record(folio, mem_cgroup_private_id(memcg), entry); | |
| 5276 | 5245 | |
| 5277 | 5246 | return 0; |
| 5278 | 5247 | } |
| @@ -5289,7 +5258,7 @@ void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages) · 10 unmodified lines | ||
| 5289 | 5258 | |
| 5290 | 5259 | id = swap_cgroup_clear(entry, nr_pages); |
| 5291 | 5260 | rcu_read_lock(); |
| 5292 | − memcg = mem_cgroup_from_id(id); | |
| 5261 | + memcg = mem_cgroup_from_private_id(id); | |
| 5293 | 5262 | if (memcg) { |
| 5294 | 5263 | if (!mem_cgroup_is_root(memcg)) { |
| 5295 | 5264 | if (do_memsw_account()) |
| @@ -5298,7 +5267,7 @@ void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages) · 2 unmodified lines | ||
| 5298 | 5267 | page_counter_uncharge(&memcg->swap, nr_pages); |
| 5299 | 5268 | } |
| 5300 | 5269 | mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages); |
| 5301 | − mem_cgroup_id_put_many(memcg, nr_pages); | |
| 5270 | + mem_cgroup_private_id_put_many(memcg, nr_pages); | |
| 5302 | 5271 | } |
| 5303 | 5272 | rcu_read_unlock(); |
| 5304 | 5273 | } |
| @@ -1,10 +1,9 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 1 | 2 | /* |
| 2 | 3 | * memfd_create system call and file sealing support |
| 3 | 4 | * |
| 4 | 5 | * Code was originally included in shmem.c, and broken out to facilitate |
| 5 | 6 | * use by hugetlbfs as well as tmpfs. |
| 6 | − * | |
| 7 | − * This file is released under the GPL. | |
| 8 | 7 | */ |
| 9 | 8 | |
| 10 | 9 | #include <linux/fs.h> |
| @@ -868,7 +868,7 @@ static int kill_accessing_process(struct task_struct *p, unsigned long pfn, · 867 unmodified lines | ||
| 868 | 868 | * |
| 869 | 869 | * MF_RECOVERED - The m-f() handler marks the page as PG_hwpoisoned'ed. |
| 870 | 870 | * The page has been completely isolated, that is, unmapped, taken out of |
| 871 | − * the buddy system, or hole-punnched out of the file mapping. | |
| 871 | + * the buddy system, or hole-punched out of the file mapping. | |
| 872 | 872 | */ |
| 873 | 873 | static const char *action_name[] = { |
| 874 | 874 | [MF_IGNORED] = "Ignored", |
| @@ -475,8 +475,7 @@ static void establish_demotion_targets(void) · 474 unmodified lines | ||
| 475 | 475 | */ |
| 476 | 476 | list_for_each_entry_reverse(memtier, &memory_tiers, list) { |
| 477 | 477 | tier_nodes = get_memtier_nodemask(memtier); |
| 478 | − nodes_and(tier_nodes, node_states[N_CPU], tier_nodes); | |
| 479 | − if (!nodes_empty(tier_nodes)) { | |
| 478 | + if (nodes_and(tier_nodes, node_states[N_CPU], tier_nodes)) { | |
| 480 | 479 | /* |
| 481 | 480 | * abstract distance below the max value of this memtier |
| 482 | 481 | * is considered toptier. |
| @@ -648,7 +647,7 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype) · 165 unmodified lines | ||
| 648 | 647 | if (node_memory_types[node].memtype == memtype || !memtype) |
| 649 | 648 | node_memory_types[node].map_count--; |
| 650 | 649 | /* |
| 651 | − * If we umapped all the attached devices to this node, | |
| 650 | + * If we unmapped all the attached devices to this node, | |
| 652 | 651 | * clear the node memory type. |
| 653 | 652 | */ |
| 654 | 653 | if (!node_memory_types[node].map_count) { |
| @@ -956,7 +955,7 @@ static ssize_t demotion_enabled_store(struct kobject *kobj, · 301 unmodified lines | ||
| 956 | 955 | struct pglist_data *pgdat; |
| 957 | 956 | |
| 958 | 957 | for_each_online_pgdat(pgdat) |
| 959 | − atomic_set(&pgdat->kswapd_failures, 0); | |
| 958 | + kswapd_clear_hopeless(pgdat, KSWAPD_CLEAR_HOPELESS_OTHER); | |
| 960 | 959 | } |
| 961 | 960 | |
| 962 | 961 | return count; |
| @@ -934,7 +934,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, · 933 unmodified lines | ||
| 934 | 934 | struct page *page; |
| 935 | 935 | |
| 936 | 936 | if (likely(softleaf_is_swap(entry))) { |
| 937 | − if (swap_duplicate(entry) < 0) | |
| 937 | + if (swap_dup_entry_direct(entry) < 0) | |
| 938 | 938 | return -EIO; |
| 939 | 939 | |
| 940 | 940 | /* make sure dst_mm is on swapoff's mmlist. */ |
| @@ -1256,7 +1256,7 @@ again: · 315 unmodified lines | ||
| 1256 | 1256 | spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); |
| 1257 | 1257 | orig_src_pte = src_pte; |
| 1258 | 1258 | orig_dst_pte = dst_pte; |
| 1259 | − arch_enter_lazy_mmu_mode(); | |
| 1259 | + lazy_mmu_mode_enable(); | |
| 1260 | 1260 | |
| 1261 | 1261 | do { |
| 1262 | 1262 | nr = 1; |
| @@ -1325,7 +1325,7 @@ again: · 62 unmodified lines | ||
| 1325 | 1325 | } while (dst_pte += nr, src_pte += nr, addr += PAGE_SIZE * nr, |
| 1326 | 1326 | addr != end); |
| 1327 | 1327 | |
| 1328 | − arch_leave_lazy_mmu_mode(); | |
| 1328 | + lazy_mmu_mode_disable(); | |
| 1329 | 1329 | pte_unmap_unlock(orig_src_pte, src_ptl); |
| 1330 | 1330 | add_mm_rss_vec(dst_mm, rss); |
| 1331 | 1331 | pte_unmap_unlock(orig_dst_pte, dst_ptl); |
| @@ -1748,7 +1748,7 @@ static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, · 416 unmodified lines | ||
| 1748 | 1748 | |
| 1749 | 1749 | nr = swap_pte_batch(pte, max_nr, ptent); |
| 1750 | 1750 | rss[MM_SWAPENTS] -= nr; |
| 1751 | − free_swap_and_cache_nr(entry, nr); | |
| 1751 | + swap_put_entries_direct(entry, nr); | |
| 1752 | 1752 | } else if (softleaf_is_migration(entry)) { |
| 1753 | 1753 | struct folio *folio = softleaf_to_folio(entry); |
| 1754 | 1754 | |
| @@ -1821,11 +1821,70 @@ static inline int do_zap_pte_range(struct mmu_gather *tlb, · 66 unmodified lines | ||
| 1821 | 1821 | return nr; |
| 1822 | 1822 | } |
| 1823 | 1823 | |
| 1824 | +static bool pte_table_reclaim_possible(unsigned long start, unsigned long end, | |
| 1825 | + struct zap_details *details) | |
| 1826 | +{ | |
| 1827 | + if (!IS_ENABLED(CONFIG_PT_RECLAIM)) | |
| 1828 | + return false; | |
| 1829 | + /* Only zap if we are allowed to and cover the full page table. */ | |
| 1830 | + return details && details->reclaim_pt && (end - start >= PMD_SIZE); | |
| 1831 | +} | |
| 1832 | + | |
| 1833 | +static bool zap_empty_pte_table(struct mm_struct *mm, pmd_t *pmd, | |
| 1834 | + spinlock_t *ptl, pmd_t *pmdval) | |
| 1835 | +{ | |
| 1836 | + spinlock_t *pml = pmd_lockptr(mm, pmd); | |
| 1837 | + | |
| 1838 | + if (ptl != pml && !spin_trylock(pml)) | |
| 1839 | + return false; | |
| 1840 | + | |
| 1841 | + *pmdval = pmdp_get(pmd); | |
| 1842 | + pmd_clear(pmd); | |
| 1843 | + if (ptl != pml) | |
| 1844 | + spin_unlock(pml); | |
| 1845 | + return true; | |
| 1846 | +} | |
| 1847 | + | |
| 1848 | +static bool zap_pte_table_if_empty(struct mm_struct *mm, pmd_t *pmd, | |
| 1849 | + unsigned long addr, pmd_t *pmdval) | |
| 1850 | +{ | |
| 1851 | + spinlock_t *pml, *ptl = NULL; | |
| 1852 | + pte_t *start_pte, *pte; | |
| 1853 | + int i; | |
| 1854 | + | |
| 1855 | + pml = pmd_lock(mm, pmd); | |
| 1856 | + start_pte = pte_offset_map_rw_nolock(mm, pmd, addr, pmdval, &ptl); | |
| 1857 | + if (!start_pte) | |
| 1858 | + goto out_ptl; | |
| 1859 | + if (ptl != pml) | |
| 1860 | + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); | |
| 1861 | + | |
| 1862 | + for (i = 0, pte = start_pte; i < PTRS_PER_PTE; i++, pte++) { | |
| 1863 | + if (!pte_none(ptep_get(pte))) | |
| 1864 | + goto out_ptl; | |
| 1865 | + } | |
| 1866 | + pte_unmap(start_pte); | |
| 1867 | + | |
| 1868 | + pmd_clear(pmd); | |
| 1869 | + | |
| 1870 | + if (ptl != pml) | |
| 1871 | + spin_unlock(ptl); | |
| 1872 | + spin_unlock(pml); | |
| 1873 | + return true; | |
| 1874 | +out_ptl: | |
| 1875 | + if (start_pte) | |
| 1876 | + pte_unmap_unlock(start_pte, ptl); | |
| 1877 | + if (ptl != pml) | |
| 1878 | + spin_unlock(pml); | |
| 1879 | + return false; | |
| 1880 | +} | |
| 1881 | + | |
| 1824 | 1882 | static unsigned long zap_pte_range(struct mmu_gather *tlb, |
| 1825 | 1883 | struct vm_area_struct *vma, pmd_t *pmd, |
| 1826 | 1884 | unsigned long addr, unsigned long end, |
| 1827 | 1885 | struct zap_details *details) |
| 1828 | 1886 | { |
| 1887 | + bool can_reclaim_pt = pte_table_reclaim_possible(addr, end, details); | |
| 1829 | 1888 | bool force_flush = false, force_break = false; |
| 1830 | 1889 | struct mm_struct *mm = tlb->mm; |
| 1831 | 1890 | int rss[NR_MM_COUNTERS]; |
| @@ -1834,7 +1893,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, · 2 unmodified lines | ||
| 1834 | 1893 | pte_t *pte; |
| 1835 | 1894 | pmd_t pmdval; |
| 1836 | 1895 | unsigned long start = addr; |
| 1837 | − bool can_reclaim_pt = reclaim_pt_is_enabled(start, end, details); | |
| 1838 | 1896 | bool direct_reclaim = true; |
| 1839 | 1897 | int nr; |
| 1840 | 1898 | |
| @@ -1846,7 +1904,7 @@ retry: · 5 unmodified lines | ||
| 1846 | 1904 | return addr; |
| 1847 | 1905 | |
| 1848 | 1906 | flush_tlb_batched_pending(mm); |
| 1849 | − arch_enter_lazy_mmu_mode(); | |
| 1907 | + lazy_mmu_mode_enable(); | |
| 1850 | 1908 | do { |
| 1851 | 1909 | bool any_skipped = false; |
| 1852 | 1910 | |
| @@ -1875,10 +1933,10 @@ retry: · 22 unmodified lines | ||
| 1875 | 1933 | * from being repopulated by another thread. |
| 1876 | 1934 | */ |
| 1877 | 1935 | if (can_reclaim_pt && direct_reclaim && addr == end) |
| 1878 | − direct_reclaim = try_get_and_clear_pmd(mm, pmd, &pmdval); | |
| 1936 | + direct_reclaim = zap_empty_pte_table(mm, pmd, ptl, &pmdval); | |
| 1879 | 1937 | |
| 1880 | 1938 | add_mm_rss_vec(mm, rss); |
| 1881 | − arch_leave_lazy_mmu_mode(); | |
| 1939 | + lazy_mmu_mode_disable(); | |
| 1882 | 1940 | |
| 1883 | 1941 | /* Do the actual TLB flush before dropping ptl */ |
| 1884 | 1942 | if (force_flush) { |
| @@ -1904,10 +1962,10 @@ retry: · 19 unmodified lines | ||
| 1904 | 1962 | } |
| 1905 | 1963 | |
| 1906 | 1964 | if (can_reclaim_pt) { |
| 1907 | − if (direct_reclaim) | |
| 1908 | − free_pte(mm, start, tlb, pmdval); | |
| 1909 | − else | |
| 1910 | − try_to_free_pte(mm, pmd, start, tlb); | |
| 1965 | + if (direct_reclaim || zap_pte_table_if_empty(mm, pmd, start, &pmdval)) { | |
| 1966 | + pte_free_tlb(tlb, pmd_pgtable(pmdval), addr); | |
| 1967 | + mm_dec_nr_ptes(mm); | |
| 1968 | + } | |
| 1911 | 1969 | } |
| 1912 | 1970 | |
| 1913 | 1971 | return addr; |
| @@ -2499,7 +2557,6 @@ static int __vm_map_pages(struct vm_area_struct *vma, struct page **pages, · 585 unmodified lines | ||
| 2499 | 2557 | { |
| 2500 | 2558 | unsigned long count = vma_pages(vma); |
| 2501 | 2559 | unsigned long uaddr = vma->vm_start; |
| 2502 | − int ret, i; | |
| 2503 | 2560 | |
| 2504 | 2561 | /* Fail if the user requested offset is beyond the end of the object */ |
| 2505 | 2562 | if (offset >= num) |
| @@ -2509,14 +2566,7 @@ static int __vm_map_pages(struct vm_area_struct *vma, struct page **pages, · 3 unmodified lines | ||
| 2509 | 2566 | if (count > num - offset) |
| 2510 | 2567 | return -ENXIO; |
| 2511 | 2568 | |
| 2512 | − for (i = 0; i < count; i++) { | |
| 2513 | − ret = vm_insert_page(vma, uaddr, pages[offset + i]); | |
| 2514 | − if (ret < 0) | |
| 2515 | − return ret; | |
| 2516 | − uaddr += PAGE_SIZE; | |
| 2517 | − } | |
| 2518 | − | |
| 2519 | − return 0; | |
| 2569 | + return vm_insert_pages(vma, uaddr, pages + offset, &count); | |
| 2520 | 2570 | } |
| 2521 | 2571 | |
| 2522 | 2572 | /** |
| @@ -2816,7 +2866,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, · 293 unmodified lines | ||
| 2816 | 2866 | mapped_pte = pte = pte_alloc_map_lock(mm, pmd, addr, &ptl); |
| 2817 | 2867 | if (!pte) |
| 2818 | 2868 | return -ENOMEM; |
| 2819 | − arch_enter_lazy_mmu_mode(); | |
| 2869 | + lazy_mmu_mode_enable(); | |
| 2820 | 2870 | do { |
| 2821 | 2871 | BUG_ON(!pte_none(ptep_get(pte))); |
| 2822 | 2872 | if (!pfn_modify_allowed(pfn, prot)) { |
| @@ -2826,7 +2876,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, · 3 unmodified lines | ||
| 2826 | 2876 | set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot))); |
| 2827 | 2877 | pfn++; |
| 2828 | 2878 | } while (pte++, addr += PAGE_SIZE, addr != end); |
| 2829 | − arch_leave_lazy_mmu_mode(); | |
| 2879 | + lazy_mmu_mode_disable(); | |
| 2830 | 2880 | pte_unmap_unlock(mapped_pte, ptl); |
| 2831 | 2881 | return err; |
| 2832 | 2882 | } |
| @@ -3177,7 +3227,7 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, · 344 unmodified lines | ||
| 3177 | 3227 | return -EINVAL; |
| 3178 | 3228 | } |
| 3179 | 3229 | |
| 3180 | − arch_enter_lazy_mmu_mode(); | |
| 3230 | + lazy_mmu_mode_enable(); | |
| 3181 | 3231 | |
| 3182 | 3232 | if (fn) { |
| 3183 | 3233 | do { |
| @@ -3190,7 +3240,7 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, · 6 unmodified lines | ||
| 3190 | 3240 | } |
| 3191 | 3241 | *mask |= PGTBL_PTE_MODIFIED; |
| 3192 | 3242 | |
| 3193 | − arch_leave_lazy_mmu_mode(); | |
| 3243 | + lazy_mmu_mode_disable(); | |
| 3194 | 3244 | |
| 3195 | 3245 | if (mm != &init_mm) |
| 3196 | 3246 | pte_unmap_unlock(mapped_pte, ptl); |
| @@ -4357,12 +4407,27 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) · 1160 unmodified lines | ||
| 4357 | 4407 | return 0; |
| 4358 | 4408 | } |
| 4359 | 4409 | |
| 4360 | −static inline bool should_try_to_free_swap(struct folio *folio, | |
| 4410 | +/* | |
| 4411 | + * Check if we should call folio_free_swap to free the swap cache. | |
| 4412 | + * folio_free_swap only frees the swap cache to release the slot if swap | |
| 4413 | + * count is zero, so we don't need to check the swap count here. | |
| 4414 | + */ | |
| 4415 | +static inline bool should_try_to_free_swap(struct swap_info_struct *si, | |
| 4416 | + struct folio *folio, | |
| 4361 | 4417 | struct vm_area_struct *vma, |
| 4418 | + unsigned int extra_refs, | |
| 4362 | 4419 | unsigned int fault_flags) |
| 4363 | 4420 | { |
| 4364 | 4421 | if (!folio_test_swapcache(folio)) |
| 4365 | 4422 | return false; |
| 4423 | + /* | |
| 4424 | + * Always try to free swap cache for SWP_SYNCHRONOUS_IO devices. Swap | |
| 4425 | + * cache can help save some IO or memory overhead, but these devices | |
| 4426 | + * are fast, and meanwhile, swap cache pinning the slot deferring the | |
| 4427 | + * release of metadata or fragmentation is a more critical issue. | |
| 4428 | + */ | |
| 4429 | + if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) | |
| 4430 | + return true; | |
| 4366 | 4431 | if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) || |
| 4367 | 4432 | folio_test_mlocked(folio)) |
| 4368 | 4433 | return true; |
| @@ -4373,7 +4438,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, · 4 unmodified lines | ||
| 4373 | 4438 | * reference only in case it's likely that we'll be the exclusive user. |
| 4374 | 4439 | */ |
| 4375 | 4440 | return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && |
| 4376 | − folio_ref_count(folio) == (1 + folio_nr_pages(folio)); | |
| 4441 | + folio_ref_count(folio) == (extra_refs + folio_nr_pages(folio)); | |
| 4377 | 4442 | } |
| 4378 | 4443 | |
| 4379 | 4444 | static vm_fault_t pte_marker_clear(struct vm_fault *vmf) |
| @@ -4611,7 +4676,16 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) · 231 unmodified lines | ||
| 4611 | 4676 | } |
| 4612 | 4677 | #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ |
| 4613 | 4678 | |
| 4614 | −static DECLARE_WAIT_QUEUE_HEAD(swapcache_wq); | |
| 4679 | +/* Sanity check that a folio is fully exclusive */ | |
| 4680 | +static void check_swap_exclusive(struct folio *folio, swp_entry_t entry, | |
| 4681 | + unsigned int nr_pages) | |
| 4682 | +{ | |
| 4683 | + /* Called under PT locked and folio locked, the swap count is stable */ | |
| 4684 | + do { | |
| 4685 | + VM_WARN_ON_ONCE_FOLIO(__swap_count(entry) != 1, folio); | |
| 4686 | + entry.val++; | |
| 4687 | + } while (--nr_pages); | |
| 4688 | +} | |
| 4615 | 4689 | |
| 4616 | 4690 | /* |
| 4617 | 4691 | * We enter with non-exclusive mmap_lock (to exclude vma changes, |
| @@ -4624,17 +4698,14 @@ static DECLARE_WAIT_QUEUE_HEAD(swapcache_wq); · 6 unmodified lines | ||
| 4624 | 4698 | vm_fault_t do_swap_page(struct vm_fault *vmf) |
| 4625 | 4699 | { |
| 4626 | 4700 | struct vm_area_struct *vma = vmf->vma; |
| 4627 | − struct folio *swapcache, *folio = NULL; | |
| 4628 | − DECLARE_WAITQUEUE(wait, current); | |
| 4701 | + struct folio *swapcache = NULL, *folio; | |
| 4629 | 4702 | struct page *page; |
| 4630 | 4703 | struct swap_info_struct *si = NULL; |
| 4631 | 4704 | rmap_t rmap_flags = RMAP_NONE; |
| 4632 | − bool need_clear_cache = false; | |
| 4633 | 4705 | bool exclusive = false; |
| 4634 | 4706 | softleaf_t entry; |
| 4635 | 4707 | pte_t pte; |
| 4636 | 4708 | vm_fault_t ret = 0; |
| 4637 | − void *shadow = NULL; | |
| 4638 | 4709 | int nr_pages; |
| 4639 | 4710 | unsigned long page_idx; |
| 4640 | 4711 | unsigned long address; |
| @@ -4705,57 +4776,21 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 64 unmodified lines | ||
| 4705 | 4776 | folio = swap_cache_get_folio(entry); |
| 4706 | 4777 | if (folio) |
| 4707 | 4778 | swap_update_readahead(folio, vma, vmf->address); |
| 4708 | − swapcache = folio; | |
| 4709 | − | |
| 4710 | 4779 | if (!folio) { |
| 4711 | − if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && | |
| 4712 | − __swap_count(entry) == 1) { | |
| 4713 | − /* skip swapcache */ | |
| 4780 | + if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) { | |
| 4714 | 4781 | folio = alloc_swap_folio(vmf); |
| 4715 | 4782 | if (folio) { |
| 4716 | − __folio_set_locked(folio); | |
| 4717 | − __folio_set_swapbacked(folio); | |
| 4718 | − | |
| 4719 | − nr_pages = folio_nr_pages(folio); | |
| 4720 | − if (folio_test_large(folio)) | |
| 4721 | − entry.val = ALIGN_DOWN(entry.val, nr_pages); | |
| 4722 | 4783 | /* |
| 4723 | − * Prevent parallel swapin from proceeding with | |
| 4724 | − * the cache flag. Otherwise, another thread | |
| 4725 | − * may finish swapin first, free the entry, and | |
| 4726 | − * swapout reusing the same entry. It's | |
| 4727 | − * undetectable as pte_same() returns true due | |
| 4728 | − * to entry reuse. | |
| 4784 | + * folio is charged, so swapin can only fail due | |
| 4785 | + * to raced swapin and return NULL. | |
| 4729 | 4786 | */ |
| 4730 | − if (swapcache_prepare(entry, nr_pages)) { | |
| 4731 | − /* | |
| 4732 | − * Relax a bit to prevent rapid | |
| 4733 | − * repeated page faults. | |
| 4734 | − */ | |
| 4735 | − add_wait_queue(&swapcache_wq, &wait); | |
| 4736 | − schedule_timeout_uninterruptible(1); | |
| 4737 | − remove_wait_queue(&swapcache_wq, &wait); | |
| 4738 | − goto out_page; | |
| 4739 | − } | |
| 4740 | − need_clear_cache = true; | |
| 4741 | − | |
| 4742 | − memcg1_swapin(entry, nr_pages); | |
| 4743 | − | |
| 4744 | − shadow = swap_cache_get_shadow(entry); | |
| 4745 | − if (shadow) | |
| 4746 | − workingset_refault(folio, shadow); | |
| 4747 | − | |
| 4748 | − folio_add_lru(folio); | |
| 4749 | − | |
| 4750 | − /* To provide entry to swap_read_folio() */ | |
| 4751 | − folio->swap = entry; | |
| 4752 | − swap_read_folio(folio, NULL); | |
| 4753 | − folio->private = NULL; | |
| 4787 | + swapcache = swapin_folio(entry, folio); | |
| 4788 | + if (swapcache != folio) | |
| 4789 | + folio_put(folio); | |
| 4790 | + folio = swapcache; | |
| 4754 | 4791 | } |
| 4755 | 4792 | } else { |
| 4756 | − folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, | |
| 4757 | − vmf); | |
| 4758 | − swapcache = folio; | |
| 4793 | + folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); | |
| 4759 | 4794 | } |
| 4760 | 4795 | |
| 4761 | 4796 | if (!folio) { |
| @@ -4777,60 +4812,58 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 15 unmodified lines | ||
| 4777 | 4812 | count_memcg_event_mm(vma->vm_mm, PGMAJFAULT); |
| 4778 | 4813 | } |
| 4779 | 4814 | |
| 4815 | + swapcache = folio; | |
| 4780 | 4816 | ret |= folio_lock_or_retry(folio, vmf); |
| 4781 | 4817 | if (ret & VM_FAULT_RETRY) |
| 4782 | 4818 | goto out_release; |
| 4783 | 4819 | |
| 4784 | 4820 | page = folio_file_page(folio, swp_offset(entry)); |
| 4785 | − if (swapcache) { | |
| 4786 | − /* | |
| 4787 | − * Make sure folio_free_swap() or swapoff did not release the | |
| 4788 | − * swapcache from under us. The page pin, and pte_same test | |
| 4789 | − * below, are not enough to exclude that. Even if it is still | |
| 4790 | − * swapcache, we need to check that the page's swap has not | |
| 4791 | − * changed. | |
| 4792 | − */ | |
| 4793 | − if (unlikely(!folio_matches_swap_entry(folio, entry))) | |
| 4794 | − goto out_page; | |
| 4795 | − | |
| 4796 | − if (unlikely(PageHWPoison(page))) { | |
| 4797 | − /* | |
| 4798 | − * hwpoisoned dirty swapcache pages are kept for killing | |
| 4799 | − * owner processes (which may be unknown at hwpoison time) | |
| 4800 | − */ | |
| 4801 | − ret = VM_FAULT_HWPOISON; | |
| 4802 | − goto out_page; | |
| 4803 | − } | |
| 4804 | − | |
| 4805 | − /* | |
| 4806 | − * KSM sometimes has to copy on read faults, for example, if | |
| 4807 | − * folio->index of non-ksm folios would be nonlinear inside the | |
| 4808 | − * anon VMA -- the ksm flag is lost on actual swapout. | |
| 4809 | − */ | |
| 4810 | − folio = ksm_might_need_to_copy(folio, vma, vmf->address); | |
| 4811 | − if (unlikely(!folio)) { | |
| 4812 | − ret = VM_FAULT_OOM; | |
| 4813 | − folio = swapcache; | |
| 4814 | − goto out_page; | |
| 4815 | − } else if (unlikely(folio == ERR_PTR(-EHWPOISON))) { | |
| 4816 | − ret = VM_FAULT_HWPOISON; | |
| 4817 | − folio = swapcache; | |
| 4818 | − goto out_page; | |
| 4819 | − } | |
| 4820 | − if (folio != swapcache) | |
| 4821 | − page = folio_page(folio, 0); | |
| 4821 | + /* | |
| 4822 | + * Make sure folio_free_swap() or swapoff did not release the | |
| 4823 | + * swapcache from under us. The page pin, and pte_same test | |
| 4824 | + * below, are not enough to exclude that. Even if it is still | |
| 4825 | + * swapcache, we need to check that the page's swap has not | |
| 4826 | + * changed. | |
| 4827 | + */ | |
| 4828 | + if (unlikely(!folio_matches_swap_entry(folio, entry))) | |
| 4829 | + goto out_page; | |
| 4822 | 4830 | |
| 4831 | + if (unlikely(PageHWPoison(page))) { | |
| 4823 | 4832 | /* |
| 4824 | − * If we want to map a page that's in the swapcache writable, we | |
| 4825 | − * have to detect via the refcount if we're really the exclusive | |
| 4826 | − * owner. Try removing the extra reference from the local LRU | |
| 4827 | − * caches if required. | |
| 4833 | + * hwpoisoned dirty swapcache pages are kept for killing | |
| 4834 | + * owner processes (which may be unknown at hwpoison time) | |
| 4828 | 4835 | */ |
| 4829 | − if ((vmf->flags & FAULT_FLAG_WRITE) && folio == swapcache && | |
| 4830 | − !folio_test_ksm(folio) && !folio_test_lru(folio)) | |
| 4831 | − lru_add_drain(); | |
| 4836 | + ret = VM_FAULT_HWPOISON; | |
| 4837 | + goto out_page; | |
| 4832 | 4838 | } |
| 4833 | 4839 | |
| 4840 | + /* | |
| 4841 | + * KSM sometimes has to copy on read faults, for example, if | |
| 4842 | + * folio->index of non-ksm folios would be nonlinear inside the | |
| 4843 | + * anon VMA -- the ksm flag is lost on actual swapout. | |
| 4844 | + */ | |
| 4845 | + folio = ksm_might_need_to_copy(folio, vma, vmf->address); | |
| 4846 | + if (unlikely(!folio)) { | |
| 4847 | + ret = VM_FAULT_OOM; | |
| 4848 | + folio = swapcache; | |
| 4849 | + goto out_page; | |
| 4850 | + } else if (unlikely(folio == ERR_PTR(-EHWPOISON))) { | |
| 4851 | + ret = VM_FAULT_HWPOISON; | |
| 4852 | + folio = swapcache; | |
| 4853 | + goto out_page; | |
| 4854 | + } else if (folio != swapcache) | |
| 4855 | + page = folio_page(folio, 0); | |
| 4856 | + | |
| 4857 | + /* | |
| 4858 | + * If we want to map a page that's in the swapcache writable, we | |
| 4859 | + * have to detect via the refcount if we're really the exclusive | |
| 4860 | + * owner. Try removing the extra reference from the local LRU | |
| 4861 | + * caches if required. | |
| 4862 | + */ | |
| 4863 | + if ((vmf->flags & FAULT_FLAG_WRITE) && | |
| 4864 | + !folio_test_ksm(folio) && !folio_test_lru(folio)) | |
| 4865 | + lru_add_drain(); | |
| 4866 | + | |
| 4834 | 4867 | folio_throttle_swaprate(folio, GFP_KERNEL); |
| 4835 | 4868 | |
| 4836 | 4869 | /* |
| @@ -4846,24 +4879,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) · 9 unmodified lines | ||
| 4846 | 4879 | goto out_nomap; |
| 4847 | 4880 | } |
| 4848 | 4881 | |
| 4849 | − /* allocated large folios for SWP_SYNCHRONOUS_IO */ | |
| 4850 | − if (folio_test_large(folio) && !folio_test_swapcache(folio)) { | |
| 4851 | − unsigned long nr = folio_nr_pages(folio); | |
| 4852 | − unsigned long folio_start = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE); | |
| 4853 | − unsigned long idx = (vmf->address - folio_start) / PAGE_SIZE; | |
| 4854 | − pte_t *folio_ptep = vmf->pte - idx; | |
| 4855 | − pte_t folio_pte = ptep_get(folio_ptep); | |
| 4856 | − | |
| 4857 | − if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || | |
| 4858 | − swap_pte_batch(folio_ptep, nr, folio_pte) != nr) | |
| 4859 | − goto out_nomap; | |
| 4860 | − | |
| 4861 | − page_idx = idx; | |
| 4862 | − address = folio_start; | |
| 4863 | − ptep = folio_ptep; | |
| 4864 | − goto check_folio; | |
| 4865 | − } | |
| 4866 | − | |
| 4867 | 4882 | nr_pages = 1; |
| 4868 | 4883 | page_idx = 0; |
| 4869 | 4884 | address = vmf->address; |
| @@ -4907,12 +4922,37 @@ check_folio: · 37 unmodified lines | ||
| 4907 | 4922 | BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio)); |
| 4908 | 4923 | BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page)); |
| 4909 | 4924 | |
| 4925 | + /* | |
| 4926 | + * If a large folio already belongs to anon mapping, then we | |
| 4927 | + * can just go on and map it partially. | |
| 4928 | + * If not, with the large swapin check above failing, the page table | |
| 4929 | + * have changed, so sub pages might got charged to the wrong cgroup, | |
| 4930 | + * or even should be shmem. So we have to free it and fallback. | |
| 4931 | + * Nothing should have touched it, both anon and shmem checks if a | |
| 4932 | + * large folio is fully appliable before use. | |
| 4933 | + * | |
| 4934 | + * This will be removed once we unify folio allocation in the swap cache | |
| 4935 | + * layer, where allocation of a folio stabilizes the swap entries. | |
| 4936 | + */ | |
| 4937 | + if (!folio_test_anon(folio) && folio_test_large(folio) && | |
| 4938 | + nr_pages != folio_nr_pages(folio)) { | |
| 4939 | + if (!WARN_ON_ONCE(folio_test_dirty(folio))) | |
| 4940 | + swap_cache_del_folio(folio); | |
| 4941 | + goto out_nomap; | |
| 4942 | + } | |
| 4943 | + | |
| 4910 | 4944 | /* |
| 4911 | 4945 | * Check under PT lock (to protect against concurrent fork() sharing |
| 4912 | 4946 | * the swap entry concurrently) for certainly exclusive pages. |
| 4913 | 4947 | */ |
| 4914 | 4948 | if (!folio_test_ksm(folio)) { |
| 4949 | + /* | |
| 4950 | + * The can_swapin_thp check above ensures all PTE have | |
| 4951 | + * same exclusiveness. Checking just one PTE is fine. | |
| 4952 | + */ | |
| 4915 | 4953 | exclusive = pte_swp_exclusive(vmf->orig_pte); |
| 4954 | + if (exclusive) | |
| 4955 | + check_swap_exclusive(folio, entry, nr_pages); | |
| 4916 | 4956 | if (folio != swapcache) { |
| 4917 | 4957 | /* |
| 4918 | 4958 | * We have a fresh page that is not exposed to the |
| @@ -4946,19 +4986,10 @@ check_folio: · 27 unmodified lines | ||
| 4946 | 4986 | /* |
| 4947 | 4987 | * Some architectures may have to restore extra metadata to the page |
| 4948 | 4988 | * when reading from swap. This metadata may be indexed by swap entry |
| 4949 | − * so this must be called before swap_free(). | |
| 4989 | + * so this must be called before folio_put_swap(). | |
| 4950 | 4990 | */ |
| 4951 | 4991 | arch_swap_restore(folio_swap(entry, folio), folio); |
| 4952 | 4992 | |
| 4953 | − /* | |
| 4954 | − * Remove the swap entry and conditionally try to free up the swapcache. | |
| 4955 | − * We're already holding a reference on the page but haven't mapped it | |
| 4956 | − * yet. | |
| 4957 | − */ | |
| 4958 | − swap_free_nr(entry, nr_pages); | |
| 4959 | − if (should_try_to_free_swap(folio, vma, vmf->flags)) | |
| 4960 | − folio_free_swap(folio); | |
| 4961 | − | |
| 4962 | 4993 | add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); |
| 4963 | 4994 | add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); |
| 4964 | 4995 | pte = mk_pte(page, vma->vm_page_prot); |
| @@ -4990,22 +5021,24 @@ check_folio: · 25 unmodified lines | ||
| 4990 | 5021 | vmf->orig_pte = pte_advance_pfn(pte, page_idx); |
| 4991 | 5022 | |
| 4992 | 5023 | /* ksm created a completely new copy */ |
| 4993 | − if (unlikely(folio != swapcache && swapcache)) { | |
| 5024 | + if (unlikely(folio != swapcache)) { | |
| 4994 | 5025 | folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE); |
| 4995 | 5026 | folio_add_lru_vma(folio, vma); |
| 5027 | + folio_put_swap(swapcache, NULL); | |
| 4996 | 5028 | } else if (!folio_test_anon(folio)) { |
| 4997 | 5029 | /* |
| 4998 | − * We currently only expect small !anon folios which are either | |
| 4999 | − * fully exclusive or fully shared, or new allocated large | |
| 5000 | − * folios which are fully exclusive. If we ever get large | |
| 5001 | − * folios within swapcache here, we have to be careful. | |
| 5030 | + * We currently only expect !anon folios that are fully | |
| 5031 | + * mappable. See the comment after can_swapin_thp above. | |
| 5002 | 5032 | */ |
| 5003 | − VM_WARN_ON_ONCE(folio_test_large(folio) && folio_test_swapcache(folio)); | |
| 5004 | − VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); | |
| 5033 | + VM_WARN_ON_ONCE_FOLIO(folio_nr_pages(folio) != nr_pages, folio); | |
| 5034 | + VM_WARN_ON_ONCE_FOLIO(folio_mapped(folio), folio); | |
| 5005 | 5035 | folio_add_new_anon_rmap(folio, vma, address, rmap_flags); |
| 5036 | + folio_put_swap(folio, NULL); | |
| 5006 | 5037 | } else { |
| 5038 | + VM_WARN_ON_ONCE(nr_pages != 1 && nr_pages != folio_nr_pages(folio)); | |
| 5007 | 5039 | folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, |
| 5008 | − rmap_flags); | |
| 5040 | + rmap_flags); | |
| 5041 | + folio_put_swap(folio, nr_pages == 1 ? page : NULL); | |
| 5009 | 5042 | } |
| 5010 | 5043 | |
| 5011 | 5044 | VM_BUG_ON(!folio_test_anon(folio) || |
| @@ -5014,13 +5047,21 @@ check_folio: · 2 unmodified lines | ||
| 5014 | 5047 | arch_do_swap_page_nr(vma->vm_mm, vma, address, |
| 5015 | 5048 | pte, pte, nr_pages); |
| 5016 | 5049 | |
| 5050 | + /* | |
| 5051 | + * Remove the swap entry and conditionally try to free up the swapcache. | |
| 5052 | + * Do it after mapping, so raced page faults will likely see the folio | |
| 5053 | + * in swap cache and wait on the folio lock. | |
| 5054 | + */ | |
| 5055 | + if (should_try_to_free_swap(si, folio, vma, nr_pages, vmf->flags)) | |
| 5056 | + folio_free_swap(folio); | |
| 5057 | + | |
| 5017 | 5058 | folio_unlock(folio); |
| 5018 | − if (folio != swapcache && swapcache) { | |
| 5059 | + if (unlikely(folio != swapcache)) { | |
| 5019 | 5060 | /* |
| 5020 | 5061 | * Hold the lock to avoid the swap entry to be reused |
| 5021 | 5062 | * until we take the PT lock for the pte_same() check |
| 5022 | 5063 | * (to avoid false positives from pte_same). For |
| 5023 | − * further safety release the lock after the swap_free | |
| 5064 | + * further safety release the lock after the folio_put_swap | |
| 5024 | 5065 | * so that the swap count won't change under a |
| 5025 | 5066 | * parallel locked swapcache. |
| 5026 | 5067 | */ |
| @@ -5041,12 +5082,6 @@ unlock: · 14 unmodified lines | ||
| 5041 | 5082 | if (vmf->pte) |
| 5042 | 5083 | pte_unmap_unlock(vmf->pte, vmf->ptl); |
| 5043 | 5084 | out: |
| 5044 | − /* Clear the swap cache pin for direct swapin after PTL unlock */ | |
| 5045 | − if (need_clear_cache) { | |
| 5046 | − swapcache_clear(si, entry, nr_pages); | |
| 5047 | − if (waitqueue_active(&swapcache_wq)) | |
| 5048 | − wake_up(&swapcache_wq); | |
| 5049 | − } | |
| 5050 | 5085 | if (si) |
| 5051 | 5086 | put_swap_device(si); |
| 5052 | 5087 | return ret; |
| @@ -5054,18 +5089,15 @@ out_nomap: · 1 unmodified line | ||
| 5054 | 5089 | if (vmf->pte) |
| 5055 | 5090 | pte_unmap_unlock(vmf->pte, vmf->ptl); |
| 5056 | 5091 | out_page: |
| 5092 | + if (folio_test_swapcache(folio)) | |
| 5093 | + folio_free_swap(folio); | |
| 5057 | 5094 | folio_unlock(folio); |
| 5058 | 5095 | out_release: |
| 5059 | 5096 | folio_put(folio); |
| 5060 | − if (folio != swapcache && swapcache) { | |
| 5097 | + if (folio != swapcache) { | |
| 5061 | 5098 | folio_unlock(swapcache); |
| 5062 | 5099 | folio_put(swapcache); |
| 5063 | 5100 | } |
| 5064 | − if (need_clear_cache) { | |
| 5065 | − swapcache_clear(si, entry, nr_pages); | |
| 5066 | − if (waitqueue_active(&swapcache_wq)) | |
| 5067 | − wake_up(&swapcache_wq); | |
| 5068 | − } | |
| 5069 | 5101 | if (si) |
| 5070 | 5102 | put_swap_device(si); |
| 5071 | 5103 | return ret; |
| @@ -5935,7 +5967,7 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf, · 863 unmodified lines | ||
| 5935 | 5967 | else |
| 5936 | 5968 | *last_cpupid = folio_last_cpupid(folio); |
| 5937 | 5969 | |
| 5938 | − /* Record the current PID acceesing VMA */ | |
| 5970 | + /* Record the current PID accessing VMA */ | |
| 5939 | 5971 | vma_set_access_pid_bit(vma); |
| 5940 | 5972 | |
| 5941 | 5973 | count_vm_numa_event(NUMA_HINT_FAULTS); |
| @@ -6254,7 +6286,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) · 312 unmodified lines | ||
| 6254 | 6286 | * Use the maywrite version to indicate that vmf->pte may be |
| 6255 | 6287 | * modified, but since we will use pte_same() to detect the |
| 6256 | 6288 | * change of the !pte_none() entry, there is no need to recheck |
| 6257 | − * the pmdval. Here we chooes to pass a dummy variable instead | |
| 6289 | + * the pmdval. Here we choose to pass a dummy variable instead | |
| 6258 | 6290 | * of NULL, which helps new user think about why this place is |
| 6259 | 6291 | * special. |
| 6260 | 6292 | */ |
| @@ -7240,40 +7272,77 @@ static inline int process_huge_page( · 979 unmodified lines | ||
| 7240 | 7272 | return 0; |
| 7241 | 7273 | } |
| 7242 | 7274 | |
| 7243 | −static void clear_gigantic_page(struct folio *folio, unsigned long addr_hint, | |
| 7244 | − unsigned int nr_pages) | |
| 7275 | +static void clear_contig_highpages(struct page *page, unsigned long addr, | |
| 7276 | + unsigned int nr_pages) | |
| 7245 | 7277 | { |
| 7246 | − unsigned long addr = ALIGN_DOWN(addr_hint, folio_size(folio)); | |
| 7247 | − int i; | |
| 7278 | + unsigned int i, count; | |
| 7279 | + /* | |
| 7280 | + * When clearing we want to operate on the largest extent possible to | |
| 7281 | + * allow for architecture specific extent based optimizations. | |
| 7282 | + * | |
| 7283 | + * However, since clear_user_highpages() (and primitives clear_user_pages(), | |
| 7284 | + * clear_pages()), do not call cond_resched(), limit the unit size when | |
| 7285 | + * running under non-preemptible scheduling models. | |
| 7286 | + */ | |
| 7287 | + const unsigned int unit = preempt_model_preemptible() ? | |
| 7288 | + nr_pages : PROCESS_PAGES_NON_PREEMPT_BATCH; | |
| 7248 | 7289 | |
| 7249 | 7290 | might_sleep(); |
| 7250 | − for (i = 0; i < nr_pages; i++) { | |
| 7291 | + | |
| 7292 | + for (i = 0; i < nr_pages; i += count) { | |
| 7251 | 7293 | cond_resched(); |
| 7252 | − clear_user_highpage(folio_page(folio, i), addr + i * PAGE_SIZE); | |
| 7294 | + | |
| 7295 | + count = min(unit, nr_pages - i); | |
| 7296 | + clear_user_highpages(page + i, addr + i * PAGE_SIZE, count); | |
| 7253 | 7297 | } |
| 7254 | 7298 | } |
| 7255 | 7299 | |
| 7256 | −static int clear_subpage(unsigned long addr, int idx, void *arg) | |
| 7257 | −{ | |
| 7258 | − struct folio *folio = arg; | |
| 7259 | − | |
| 7260 | − clear_user_highpage(folio_page(folio, idx), addr); | |
| 7261 | − return 0; | |
| 7262 | −} | |
| 7300 | +/* | |
| 7301 | + * When zeroing a folio, we want to differentiate between pages in the | |
| 7302 | + * vicinity of the faulting address where we have spatial and temporal | |
| 7303 | + * locality, and those far away where we don't. | |
| 7304 | + * | |
| 7305 | + * Use a radius of 2 for determining the local neighbourhood. | |
| 7306 | + */ | |
| 7307 | +#define FOLIO_ZERO_LOCALITY_RADIUS 2 | |
| 7263 | 7308 | |
| 7264 | 7309 | /** |
| 7265 | 7310 | * folio_zero_user - Zero a folio which will be mapped to userspace. |
| 7266 | 7311 | * @folio: The folio to zero. |
| 7267 | − * @addr_hint: The address will be accessed or the base address if uncelar. | |
| 7312 | + * @addr_hint: The address accessed by the user or the base address. | |
| 7268 | 7313 | */ |
| 7269 | 7314 | void folio_zero_user(struct folio *folio, unsigned long addr_hint) |
| 7270 | 7315 | { |
| 7271 | − unsigned int nr_pages = folio_nr_pages(folio); | |
| 7316 | + const unsigned long base_addr = ALIGN_DOWN(addr_hint, folio_size(folio)); | |
| 7317 | + const long fault_idx = (addr_hint - base_addr) / PAGE_SIZE; | |
| 7318 | + const struct range pg = DEFINE_RANGE(0, folio_nr_pages(folio) - 1); | |
| 7319 | + const int radius = FOLIO_ZERO_LOCALITY_RADIUS; | |
| 7320 | + struct range r[3]; | |
| 7321 | + int i; | |
| 7272 | 7322 | |
| 7273 | − if (unlikely(nr_pages > MAX_ORDER_NR_PAGES)) | |
| 7274 | − clear_gigantic_page(folio, addr_hint, nr_pages); | |
| 7275 | − else | |
| 7276 | − process_huge_page(addr_hint, nr_pages, clear_subpage, folio); | |
| 7323 | + /* | |
| 7324 | + * Faulting page and its immediate neighbourhood. Will be cleared at the | |
| 7325 | + * end to keep its cachelines hot. | |
| 7326 | + */ | |
| 7327 | + r[2] = DEFINE_RANGE(clamp_t(s64, fault_idx - radius, pg.start, pg.end), | |
| 7328 | + clamp_t(s64, fault_idx + radius, pg.start, pg.end)); | |
| 7329 | + | |
| 7330 | + /* Region to the left of the fault */ | |
| 7331 | + r[1] = DEFINE_RANGE(pg.start, | |
| 7332 | + clamp_t(s64, r[2].start - 1, pg.start - 1, r[2].start)); | |
| 7333 | + | |
| 7334 | + /* Region to the right of the fault: always valid for the common fault_idx=0 case. */ | |
| 7335 | + r[0] = DEFINE_RANGE(clamp_t(s64, r[2].end + 1, r[2].end, pg.end + 1), | |
| 7336 | + pg.end); | |
| 7337 | + | |
| 7338 | + for (i = 0; i < ARRAY_SIZE(r); i++) { | |
| 7339 | + const unsigned long addr = base_addr + r[i].start * PAGE_SIZE; | |
| 7340 | + const unsigned int nr_pages = range_len(&r[i]); | |
| 7341 | + struct page *page = folio_page(folio, r[i].start); | |
| 7342 | + | |
| 7343 | + if (nr_pages > 0) | |
| 7344 | + clear_contig_highpages(page, addr, nr_pages); | |
| 7345 | + } | |
| 7277 | 7346 | } |
| 7278 | 7347 | |
| 7279 | 7348 | static int copy_user_gigantic_page(struct folio *dst, struct folio *src, |
| @@ -926,7 +926,7 @@ static struct zone *default_kernel_zone_for_pfn(int nid, unsigned long start_pfn · 925 unmodified lines | ||
| 926 | 926 | * |
| 927 | 927 | * MOVABLE : KERNEL_EARLY |
| 928 | 928 | * |
| 929 | − * Whereby KERNEL_EARLY is memory in one of the kernel zones, available sinze | |
| 929 | + * Whereby KERNEL_EARLY is memory in one of the kernel zones, available since | |
| 930 | 930 | * boot. We base our calculation on KERNEL_EARLY internally, because: |
| 931 | 931 | * |
| 932 | 932 | * a) Hotplugged memory in one of the kernel zones can sometimes still get |
| @@ -946,8 +946,8 @@ static struct zone *default_kernel_zone_for_pfn(int nid, unsigned long start_pfn · 13 unmodified lines | ||
| 946 | 946 | * We rely on "present pages" instead of "managed pages", as the latter is |
| 947 | 947 | * highly unreliable and dynamic in virtualized environments, and does not |
| 948 | 948 | * consider boot time allocations. For example, memory ballooning adjusts the |
| 949 | − * managed pages when inflating/deflating the balloon, and balloon compaction | |
| 950 | − * can even migrate inflated pages between zones. | |
| 949 | + * managed pages when inflating/deflating the balloon, and balloon page | |
| 950 | + * migration can even migrate inflated pages between zones. | |
| 951 | 951 | * |
| 952 | 952 | * Using "present pages" is better but some things to keep in mind are: |
| 953 | 953 | * |
| @@ -1258,7 +1258,7 @@ static pg_data_t *hotadd_init_pgdat(int nid) · 304 unmodified lines | ||
| 1258 | 1258 | * NODE_DATA is preallocated (free_area_init) but its internal |
| 1259 | 1259 | * state is not allocated completely. Add missing pieces. |
| 1260 | 1260 | * Completely offline nodes stay around and they just need |
| 1261 | − * reintialization. | |
| 1261 | + * reinitialization. | |
| 1262 | 1262 | */ |
| 1263 | 1263 | pgdat = NODE_DATA(nid); |
| 1264 | 1264 | |
| @@ -365,7 +365,7 @@ static const struct mempolicy_operations { · 364 unmodified lines | ||
| 365 | 365 | |
| 366 | 366 | static inline int mpol_store_user_nodemask(const struct mempolicy *pol) |
| 367 | 367 | { |
| 368 | − return pol->flags & MPOL_MODE_FLAGS; | |
| 368 | + return pol->flags & MPOL_USER_NODEMASK_FLAGS; | |
| 369 | 369 | } |
| 370 | 370 | |
| 371 | 371 | static void mpol_relative_nodemask(nodemask_t *ret, const nodemask_t *orig, |
| @@ -1909,8 +1909,7 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode, · 1537 unmodified lines | ||
| 1909 | 1909 | } |
| 1910 | 1910 | |
| 1911 | 1911 | task_nodes = cpuset_mems_allowed(current); |
| 1912 | − nodes_and(*new, *new, task_nodes); | |
| 1913 | − if (nodes_empty(*new)) | |
| 1912 | + if (!nodes_and(*new, *new, task_nodes)) | |
| 1914 | 1913 | goto out_put; |
| 1915 | 1914 | |
| 1916 | 1915 | err = security_task_movememory(task); |
| @@ -88,7 +88,7 @@ static const struct movable_operations *page_movable_ops(struct page *page) · 87 unmodified lines | ||
| 88 | 88 | * back to the buddy. |
| 89 | 89 | */ |
| 90 | 90 | if (PageOffline(page)) |
| 91 | − /* Only balloon compaction sets PageOffline pages movable. */ | |
| 91 | + /* Only balloon page migration sets PageOffline pages movable. */ | |
| 92 | 92 | return offline_movable_ops; |
| 93 | 93 | if (PageZsmalloc(page)) |
| 94 | 94 | return zsmalloc_movable_ops; |
| @@ -452,11 +452,12 @@ static bool remove_migration_pte(struct folio *folio, · 357 unmodified lines | ||
| 452 | 452 | * Get rid of all migration entries and replace them by |
| 453 | 453 | * references to the indicated page. |
| 454 | 454 | */ |
| 455 | −void remove_migration_ptes(struct folio *src, struct folio *dst, int flags) | |
| 455 | +void remove_migration_ptes(struct folio *src, struct folio *dst, | |
| 456 | + enum ttu_flags flags) | |
| 456 | 457 | { |
| 457 | 458 | struct rmap_walk_arg rmap_walk_arg = { |
| 458 | 459 | .folio = src, |
| 459 | − .map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE, | |
| 460 | + .map_unused_to_zeropage = flags & TTU_USE_SHARED_ZEROPAGE, | |
| 460 | 461 | }; |
| 461 | 462 | |
| 462 | 463 | struct rmap_walk_control rwc = { |
| @@ -464,9 +465,9 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, int flags) · 1 unmodified line | ||
| 464 | 465 | .arg = &rmap_walk_arg, |
| 465 | 466 | }; |
| 466 | 467 | |
| 467 | − VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src); | |
| 468 | + VM_BUG_ON_FOLIO((flags & TTU_USE_SHARED_ZEROPAGE) && (src != dst), src); | |
| 468 | 469 | |
| 469 | − if (flags & RMP_LOCKED) | |
| 470 | + if (flags & TTU_RMAP_LOCKED) | |
| 470 | 471 | rmap_walk_locked(dst, &rwc); |
| 471 | 472 | else |
| 472 | 473 | rmap_walk(dst, &rwc); |
| @@ -1521,8 +1522,7 @@ static int unmap_and_move_huge_page(new_folio_t get_new_folio, · 1048 unmodified lines | ||
| 1521 | 1522 | rc = move_to_new_folio(dst, src, mode); |
| 1522 | 1523 | |
| 1523 | 1524 | if (page_was_mapped) |
| 1524 | − remove_migration_ptes(src, !rc ? dst : src, | |
| 1525 | − ttu ? RMP_LOCKED : 0); | |
| 1525 | + remove_migration_ptes(src, !rc ? dst : src, ttu); | |
| 1526 | 1526 | |
| 1527 | 1527 | if (ttu & TTU_RMAP_LOCKED) |
| 1528 | 1528 | i_mmap_unlock_write(mapping); |
| @@ -271,7 +271,7 @@ again: · 270 unmodified lines | ||
| 271 | 271 | ptep = pte_offset_map_lock(mm, pmdp, start, &ptl); |
| 272 | 272 | if (!ptep) |
| 273 | 273 | goto again; |
| 274 | − arch_enter_lazy_mmu_mode(); | |
| 274 | + lazy_mmu_mode_enable(); | |
| 275 | 275 | ptep += (addr - start) / PAGE_SIZE; |
| 276 | 276 | |
| 277 | 277 | for (; addr < end; addr += PAGE_SIZE, ptep++) { |
| @@ -313,7 +313,7 @@ again: · 35 unmodified lines | ||
| 313 | 313 | if (folio_test_large(folio)) { |
| 314 | 314 | int ret; |
| 315 | 315 | |
| 316 | − arch_leave_lazy_mmu_mode(); | |
| 316 | + lazy_mmu_mode_disable(); | |
| 317 | 317 | pte_unmap_unlock(ptep, ptl); |
| 318 | 318 | ret = migrate_vma_split_folio(folio, |
| 319 | 319 | migrate->fault_page); |
| @@ -356,7 +356,7 @@ again: · 36 unmodified lines | ||
| 356 | 356 | if (folio && folio_test_large(folio)) { |
| 357 | 357 | int ret; |
| 358 | 358 | |
| 359 | − arch_leave_lazy_mmu_mode(); | |
| 359 | + lazy_mmu_mode_disable(); | |
| 360 | 360 | pte_unmap_unlock(ptep, ptl); |
| 361 | 361 | ret = migrate_vma_split_folio(folio, |
| 362 | 362 | migrate->fault_page); |
| @@ -485,7 +485,7 @@ next: · 122 unmodified lines | ||
| 485 | 485 | if (unmapped) |
| 486 | 486 | flush_tlb_range(walk->vma, start, end); |
| 487 | 487 | |
| 488 | − arch_leave_lazy_mmu_mode(); | |
| 488 | + lazy_mmu_mode_disable(); | |
| 489 | 489 | pte_unmap_unlock(ptep - 1, ptl); |
| 490 | 490 | |
| 491 | 491 | return 0; |
| @@ -1419,10 +1419,10 @@ EXPORT_SYMBOL(migrate_device_range); · 927 unmodified lines | ||
| 1419 | 1419 | |
| 1420 | 1420 | /** |
| 1421 | 1421 | * migrate_device_pfns() - migrate device private pfns to normal memory. |
| 1422 | − * @src_pfns: pre-popluated array of source device private pfns to migrate. | |
| 1422 | + * @src_pfns: pre-populated array of source device private pfns to migrate. | |
| 1423 | 1423 | * @npages: number of pages to migrate. |
| 1424 | 1424 | * |
| 1425 | − * Similar to migrate_device_range() but supports non-contiguous pre-popluated | |
| 1425 | + * Similar to migrate_device_range() but supports non-contiguous pre-populated | |
| 1426 | 1426 | * array of device pages to migrate. |
| 1427 | 1427 | */ |
| 1428 | 1428 | int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages) |
| @@ -187,7 +187,7 @@ void mm_compute_batch(int overcommit_policy) · 186 unmodified lines | ||
| 187 | 187 | /* |
| 188 | 188 | * For policy OVERCOMMIT_NEVER, set batch size to 0.4% of |
| 189 | 189 | * (total memory/#cpus), and lift it to 25% for other policies |
| 190 | − * to easy the possible lock contention for percpu_counter | |
| 190 | + * to ease the possible lock contention for percpu_counter | |
| 191 | 191 | * vm_committed_as, while the max limit is INT_MAX |
| 192 | 192 | */ |
| 193 | 193 | if (overcommit_policy == OVERCOMMIT_NEVER) |
| @@ -646,21 +646,18 @@ int __meminit early_pfn_to_nid(unsigned long pfn) · 452 unmodified lines | ||
| 646 | 646 | return nid; |
| 647 | 647 | } |
| 648 | 648 | |
| 649 | −int hashdist = HASHDIST_DEFAULT; | |
| 649 | +bool hashdist = HASHDIST_DEFAULT; | |
| 650 | 650 | |
| 651 | 651 | static int __init set_hashdist(char *str) |
| 652 | 652 | { |
| 653 | − if (!str) | |
| 654 | − return 0; | |
| 655 | − hashdist = simple_strtoul(str, &str, 0); | |
| 656 | − return 1; | |
| 653 | + return kstrtobool(str, &hashdist) == 0; | |
| 657 | 654 | } |
| 658 | 655 | __setup("hashdist=", set_hashdist); |
| 659 | 656 | |
| 660 | 657 | static inline void fixup_hashdist(void) |
| 661 | 658 | { |
| 662 | 659 | if (num_node_state(N_MEMORY) == 1) |
| 663 | − hashdist = 0; | |
| 660 | + hashdist = false; | |
| 664 | 661 | } |
| 665 | 662 | #else |
| 666 | 663 | static inline void fixup_hashdist(void) {} |
| @@ -1748,7 +1745,7 @@ static void __init free_area_init_node(int nid) · 1081 unmodified lines | ||
| 1748 | 1745 | lru_gen_init_pgdat(pgdat); |
| 1749 | 1746 | } |
| 1750 | 1747 | |
| 1751 | −/* Any regular or high memory on that node ? */ | |
| 1748 | +/* Any regular or high memory on that node? */ | |
| 1752 | 1749 | static void __init check_for_memory(pg_data_t *pgdat) |
| 1753 | 1750 | { |
| 1754 | 1751 | enum zone_type zone_type; |
| @@ -1810,7 +1807,6 @@ static void __init set_high_memory(void) · 55 unmodified lines | ||
| 1810 | 1807 | |
| 1811 | 1808 | /** |
| 1812 | 1809 | * free_area_init - Initialise all pg_data_t and zone data |
| 1813 | − * @max_zone_pfn: an array of max PFNs for each zone | |
| 1814 | 1810 | * |
| 1815 | 1811 | * This will call free_area_init_node() for each active node in the system. |
| 1816 | 1812 | * Using the page ranges provided by memblock_set_node(), the size of each |
| @@ -1821,17 +1817,15 @@ static void __init set_high_memory(void) · 4 unmodified lines | ||
| 1821 | 1817 | * starts where the previous one ended. For example, ZONE_DMA32 starts |
| 1822 | 1818 | * at arch_max_dma_pfn. |
| 1823 | 1819 | */ |
| 1824 | −void __init free_area_init(unsigned long *max_zone_pfn) | |
| 1820 | +static void __init free_area_init(void) | |
| 1825 | 1821 | { |
| 1822 | + unsigned long max_zone_pfn[MAX_NR_ZONES] = { 0 }; | |
| 1826 | 1823 | unsigned long start_pfn, end_pfn; |
| 1827 | 1824 | int i, nid, zone; |
| 1828 | 1825 | bool descending; |
| 1829 | 1826 | |
| 1830 | − /* Record where the zone boundaries are */ | |
| 1831 | − memset(arch_zone_lowest_possible_pfn, 0, | |
| 1832 | − sizeof(arch_zone_lowest_possible_pfn)); | |
| 1833 | − memset(arch_zone_highest_possible_pfn, 0, | |
| 1834 | − sizeof(arch_zone_highest_possible_pfn)); | |
| 1827 | + arch_zone_limits_init(max_zone_pfn); | |
| 1828 | + sparse_init(); | |
| 1835 | 1829 | |
| 1836 | 1830 | start_pfn = PHYS_PFN(memblock_start_of_DRAM()); |
| 1837 | 1831 | descending = arch_has_descending_max_zone_pfns(); |
| @@ -2048,7 +2042,7 @@ static unsigned long __init deferred_init_pages(struct zone *zone, · 210 unmodified lines | ||
| 2048 | 2042 | * Initialize and free pages. |
| 2049 | 2043 | * |
| 2050 | 2044 | * At this point reserved pages and struct pages that correspond to holes in |
| 2051 | − * memblock.memory are already intialized so every free range has a valid | |
| 2045 | + * memblock.memory are already initialized so every free range has a valid | |
| 2052 | 2046 | * memory map around it. |
| 2053 | 2047 | * This ensures that access of pages that are ahead of the range being |
| 2054 | 2048 | * initialized (computing buddy page in __free_one_page()) always reads a valid |
| @@ -2681,13 +2675,20 @@ void __init __weak mem_init(void) · 626 unmodified lines | ||
| 2681 | 2675 | { |
| 2682 | 2676 | } |
| 2683 | 2677 | |
| 2678 | +void __init mm_core_init_early(void) | |
| 2679 | +{ | |
| 2680 | + hugetlb_cma_reserve(); | |
| 2681 | + hugetlb_bootmem_alloc(); | |
| 2682 | + | |
| 2683 | + free_area_init(); | |
| 2684 | +} | |
| 2685 | + | |
| 2684 | 2686 | /* |
| 2685 | 2687 | * Set up kernel memory allocators |
| 2686 | 2688 | */ |
| 2687 | 2689 | void __init mm_core_init(void) |
| 2688 | 2690 | { |
| 2689 | 2691 | arch_mm_preinit(); |
| 2690 | − hugetlb_bootmem_alloc(); | |
| 2691 | 2692 | |
| 2692 | 2693 | /* Initializations relying on SMP setup */ |
| 2693 | 2694 | BUILD_BUG_ON(MAX_ZONELISTS > 2); |
| @@ -45,65 +45,111 @@ EXPORT_SYMBOL(__mmap_lock_do_trace_released); · 44 unmodified lines | ||
| 45 | 45 | |
| 46 | 46 | #ifdef CONFIG_MMU |
| 47 | 47 | #ifdef CONFIG_PER_VMA_LOCK |
| 48 | + | |
| 49 | +/* State shared across __vma_[start, end]_exclude_readers. */ | |
| 50 | +struct vma_exclude_readers_state { | |
| 51 | + /* Input parameters. */ | |
| 52 | + struct vm_area_struct *vma; | |
| 53 | + int state; /* TASK_KILLABLE or TASK_UNINTERRUPTIBLE. */ | |
| 54 | + bool detaching; | |
| 55 | + | |
| 56 | + /* Output parameters. */ | |
| 57 | + bool detached; | |
| 58 | + bool exclusive; /* Are we exclusively locked? */ | |
| 59 | +}; | |
| 60 | + | |
| 48 | 61 | /* |
| 49 | − * __vma_enter_locked() returns 0 immediately if the vma is not | |
| 50 | − * attached, otherwise it waits for any current readers to finish and | |
| 51 | − * returns 1. Returns -EINTR if a signal is received while waiting. | |
| 62 | + * Now that all readers have been evicted, mark the VMA as being out of the | |
| 63 | + * 'exclude readers' state. | |
| 52 | 64 | */ |
| 53 | −static inline int __vma_enter_locked(struct vm_area_struct *vma, | |
| 54 | − bool detaching, int state) | |
| 65 | +static void __vma_end_exclude_readers(struct vma_exclude_readers_state *ves) | |
| 55 | 66 | { |
| 56 | − int err; | |
| 57 | − unsigned int tgt_refcnt = VMA_LOCK_OFFSET; | |
| 67 | + struct vm_area_struct *vma = ves->vma; | |
| 58 | 68 | |
| 59 | − mmap_assert_write_locked(vma->vm_mm); | |
| 69 | + VM_WARN_ON_ONCE(ves->detached); | |
| 70 | + | |
| 71 | + ves->detached = refcount_sub_and_test(VM_REFCNT_EXCLUDE_READERS_FLAG, | |
| 72 | + &vma->vm_refcnt); | |
| 73 | + __vma_lockdep_release_exclusive(vma); | |
| 74 | +} | |
| 75 | + | |
| 76 | +static unsigned int get_target_refcnt(struct vma_exclude_readers_state *ves) | |
| 77 | +{ | |
| 78 | + const unsigned int tgt = ves->detaching ? 0 : 1; | |
| 79 | + | |
| 80 | + return tgt | VM_REFCNT_EXCLUDE_READERS_FLAG; | |
| 81 | +} | |
| 82 | + | |
| 83 | +/* | |
| 84 | + * Mark the VMA as being in a state of excluding readers, check to see if any | |
| 85 | + * VMA read locks are indeed held, and if so wait for them to be released. | |
| 86 | + * | |
| 87 | + * Note that this function pairs with vma_refcount_put() which will wake up this | |
| 88 | + * thread when it detects that the last reader has released its lock. | |
| 89 | + * | |
| 90 | + * The ves->state parameter ought to be set to TASK_UNINTERRUPTIBLE in cases | |
| 91 | + * where we wish the thread to sleep uninterruptibly or TASK_KILLABLE if a fatal | |
| 92 | + * signal is permitted to kill it. | |
| 93 | + * | |
| 94 | + * The function sets the ves->exclusive parameter to true if readers were | |
| 95 | + * excluded, or false if the VMA was detached or an error arose on wait. | |
| 96 | + * | |
| 97 | + * If the function indicates an exclusive lock was acquired via ves->exclusive | |
| 98 | + * the caller is required to invoke __vma_end_exclude_readers() once the | |
| 99 | + * exclusive state is no longer required. | |
| 100 | + * | |
| 101 | + * If ves->state is set to something other than TASK_UNINTERRUPTIBLE, the | |
| 102 | + * function may also return -EINTR to indicate a fatal signal was received while | |
| 103 | + * waiting. Otherwise, the function returns 0. | |
| 104 | + */ | |
| 105 | +static int __vma_start_exclude_readers(struct vma_exclude_readers_state *ves) | |
| 106 | +{ | |
| 107 | + struct vm_area_struct *vma = ves->vma; | |
| 108 | + unsigned int tgt_refcnt = get_target_refcnt(ves); | |
| 109 | + int err = 0; | |
| 60 | 110 | |
| 61 | − /* Additional refcnt if the vma is attached. */ | |
| 62 | − if (!detaching) | |
| 63 | − tgt_refcnt++; | |
| 111 | + mmap_assert_write_locked(vma->vm_mm); | |
| 64 | 112 | |
| 65 | 113 | /* |
| 66 | 114 | * If vma is detached then only vma_mark_attached() can raise the |
| 67 | 115 | * vm_refcnt. mmap_write_lock prevents racing with vma_mark_attached(). |
| 116 | + * | |
| 117 | + * See the comment describing the vm_area_struct->vm_refcnt field for | |
| 118 | + * details of possible refcnt values. | |
| 68 | 119 | */ |
| 69 | − if (!refcount_add_not_zero(VMA_LOCK_OFFSET, &vma->vm_refcnt)) | |
| 120 | + if (!refcount_add_not_zero(VM_REFCNT_EXCLUDE_READERS_FLAG, &vma->vm_refcnt)) { | |
| 121 | + ves->detached = true; | |
| 70 | 122 | return 0; |
| 123 | + } | |
| 71 | 124 | |
| 72 | − rwsem_acquire(&vma->vmlock_dep_map, 0, 0, _RET_IP_); | |
| 125 | + __vma_lockdep_acquire_exclusive(vma); | |
| 73 | 126 | err = rcuwait_wait_event(&vma->vm_mm->vma_writer_wait, |
| 74 | 127 | refcount_read(&vma->vm_refcnt) == tgt_refcnt, |
| 75 | − state); | |
| 128 | + ves->state); | |
| 76 | 129 | if (err) { |
| 77 | − if (refcount_sub_and_test(VMA_LOCK_OFFSET, &vma->vm_refcnt)) { | |
| 78 | − /* | |
| 79 | − * The wait failed, but the last reader went away | |
| 80 | − * as well. Tell the caller the VMA is detached. | |
| 81 | − */ | |
| 82 | − WARN_ON_ONCE(!detaching); | |
| 83 | − err = 0; | |
| 84 | − } | |
| 85 | − rwsem_release(&vma->vmlock_dep_map, _RET_IP_); | |
| 130 | + __vma_end_exclude_readers(ves); | |
| 86 | 131 | return err; |
| 87 | 132 | } |
| 88 | − lock_acquired(&vma->vmlock_dep_map, _RET_IP_); | |
| 89 | 133 | |
| 90 | − return 1; | |
| 91 | −} | |
| 92 | − | |
| 93 | −static inline void __vma_exit_locked(struct vm_area_struct *vma, bool *detached) | |
| 94 | −{ | |
| 95 | − *detached = refcount_sub_and_test(VMA_LOCK_OFFSET, &vma->vm_refcnt); | |
| 96 | − rwsem_release(&vma->vmlock_dep_map, _RET_IP_); | |
| 134 | + __vma_lockdep_stat_mark_acquired(vma); | |
| 135 | + ves->exclusive = true; | |
| 136 | + return 0; | |
| 97 | 137 | } |
| 98 | 138 | |
| 99 | −int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq, | |
| 100 | − int state) | |
| 139 | +int __vma_start_write(struct vm_area_struct *vma, int state) | |
| 101 | 140 | { |
| 102 | − int locked; | |
| 141 | + const unsigned int mm_lock_seq = __vma_raw_mm_seqnum(vma); | |
| 142 | + struct vma_exclude_readers_state ves = { | |
| 143 | + .vma = vma, | |
| 144 | + .state = state, | |
| 145 | + }; | |
| 146 | + int err; | |
| 103 | 147 | |
| 104 | − locked = __vma_enter_locked(vma, false, state); | |
| 105 | − if (locked < 0) | |
| 106 | − return locked; | |
| 148 | + err = __vma_start_exclude_readers(&ves); | |
| 149 | + if (err) { | |
| 150 | + WARN_ON_ONCE(ves.detached); | |
| 151 | + return err; | |
| 152 | + } | |
| 107 | 153 | |
| 108 | 154 | /* |
| 109 | 155 | * We should use WRITE_ONCE() here because we can have concurrent reads |
| @@ -113,39 +159,42 @@ int __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq, · 3 unmodified lines | ||
| 113 | 159 | */ |
| 114 | 160 | WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); |
| 115 | 161 | |
| 116 | − if (locked) { | |
| 117 | − bool detached; | |
| 118 | − | |
| 119 | − __vma_exit_locked(vma, &detached); | |
| 120 | − WARN_ON_ONCE(detached); /* vma should remain attached */ | |
| 162 | + if (ves.exclusive) { | |
| 163 | + __vma_end_exclude_readers(&ves); | |
| 164 | + /* VMA should remain attached. */ | |
| 165 | + WARN_ON_ONCE(ves.detached); | |
| 121 | 166 | } |
| 122 | 167 | |
| 123 | 168 | return 0; |
| 124 | 169 | } |
| 125 | 170 | EXPORT_SYMBOL_GPL(__vma_start_write); |
| 126 | 171 | |
| 127 | −void vma_mark_detached(struct vm_area_struct *vma) | |
| 172 | +void __vma_exclude_readers_for_detach(struct vm_area_struct *vma) | |
| 128 | 173 | { |
| 129 | − vma_assert_write_locked(vma); | |
| 130 | − vma_assert_attached(vma); | |
| 174 | + struct vma_exclude_readers_state ves = { | |
| 175 | + .vma = vma, | |
| 176 | + .state = TASK_UNINTERRUPTIBLE, | |
| 177 | + .detaching = true, | |
| 178 | + }; | |
| 179 | + int err; | |
| 131 | 180 | |
| 132 | 181 | /* |
| 133 | − * We are the only writer, so no need to use vma_refcount_put(). | |
| 134 | − * The condition below is unlikely because the vma has been already | |
| 135 | − * write-locked and readers can increment vm_refcnt only temporarily | |
| 136 | − * before they check vm_lock_seq, realize the vma is locked and drop | |
| 137 | − * back the vm_refcnt. That is a narrow window for observing a raised | |
| 138 | − * vm_refcnt. | |
| 182 | + * Wait until the VMA is detached with no readers. Since we hold the VMA | |
| 183 | + * write lock, the only read locks that might be present are those from | |
| 184 | + * threads trying to acquire the read lock and incrementing the | |
| 185 | + * reference count before realising the write lock is held and | |
| 186 | + * decrementing it. | |
| 139 | 187 | */ |
| 140 | − if (unlikely(!refcount_dec_and_test(&vma->vm_refcnt))) { | |
| 141 | − /* Wait until vma is detached with no readers. */ | |
| 142 | − if (__vma_enter_locked(vma, true, TASK_UNINTERRUPTIBLE)) { | |
| 143 | − bool detached; | |
| 144 | − | |
| 145 | − __vma_exit_locked(vma, &detached); | |
| 146 | − WARN_ON_ONCE(!detached); | |
| 147 | − } | |
| 188 | + err = __vma_start_exclude_readers(&ves); | |
| 189 | + if (!err && ves.exclusive) { | |
| 190 | + /* | |
| 191 | + * Once this is complete, no readers can increment the | |
| 192 | + * reference count, and the VMA is marked detached. | |
| 193 | + */ | |
| 194 | + __vma_end_exclude_readers(&ves); | |
| 148 | 195 | } |
| 196 | + /* If an error arose but we were detached anyway, we don't care. */ | |
| 197 | + WARN_ON_ONCE(!ves.detached); | |
| 149 | 198 | } |
| 150 | 199 | |
| 151 | 200 | /* |
| @@ -180,19 +229,21 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm, · 28 unmodified lines | ||
| 180 | 229 | } |
| 181 | 230 | |
| 182 | 231 | /* |
| 183 | − * If VMA_LOCK_OFFSET is set, __refcount_inc_not_zero_limited_acquire() | |
| 184 | − * will fail because VMA_REF_LIMIT is less than VMA_LOCK_OFFSET. | |
| 232 | + * If VM_REFCNT_EXCLUDE_READERS_FLAG is set, | |
| 233 | + * __refcount_inc_not_zero_limited_acquire() will fail because | |
| 234 | + * VM_REFCNT_LIMIT is less than VM_REFCNT_EXCLUDE_READERS_FLAG. | |
| 235 | + * | |
| 185 | 236 | * Acquire fence is required here to avoid reordering against later |
| 186 | 237 | * vm_lock_seq check and checks inside lock_vma_under_rcu(). |
| 187 | 238 | */ |
| 188 | 239 | if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt, |
| 189 | − VMA_REF_LIMIT))) { | |
| 240 | + VM_REFCNT_LIMIT))) { | |
| 190 | 241 | /* return EAGAIN if vma got detached from under us */ |
| 191 | 242 | vma = oldcnt ? NULL : ERR_PTR(-EAGAIN); |
| 192 | 243 | goto err; |
| 193 | 244 | } |
| 194 | 245 | |
| 195 | − rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_); | |
| 246 | + __vma_lockdep_acquire_read(vma); | |
| 196 | 247 | |
| 197 | 248 | if (unlikely(vma->vm_mm != mm)) |
| 198 | 249 | goto err_unstable; |
| @@ -1,3 +1,4 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 1 | 2 | #include <linux/gfp.h> |
| 2 | 3 | #include <linux/highmem.h> |
| 3 | 4 | #include <linux/kernel.h> |
| @@ -210,10 +211,9 @@ bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page, · 206 unmodified lines | ||
| 210 | 211 | PAGE_SIZE); |
| 211 | 212 | } |
| 212 | 213 | |
| 213 | −bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, | |
| 214 | − bool delay_rmap, int page_size) | |
| 214 | +bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) | |
| 215 | 215 | { |
| 216 | − return __tlb_remove_folio_pages_size(tlb, page, 1, delay_rmap, page_size); | |
| 216 | + return __tlb_remove_folio_pages_size(tlb, page, 1, false, page_size); | |
| 217 | 217 | } |
| 218 | 218 | |
| 219 | 219 | #endif /* MMU_GATHER_NO_GATHER */ |
| @@ -233,7 +233,7 @@ static long change_pte_range(struct mmu_gather *tlb, · 232 unmodified lines | ||
| 233 | 233 | is_private_single_threaded = vma_is_single_threaded_private(vma); |
| 234 | 234 | |
| 235 | 235 | flush_tlb_batched_pending(vma->vm_mm); |
| 236 | − arch_enter_lazy_mmu_mode(); | |
| 236 | + lazy_mmu_mode_enable(); | |
| 237 | 237 | do { |
| 238 | 238 | nr_ptes = 1; |
| 239 | 239 | oldpte = ptep_get(pte); |
| @@ -379,7 +379,7 @@ static long change_pte_range(struct mmu_gather *tlb, · 139 unmodified lines | ||
| 379 | 379 | } |
| 380 | 380 | } |
| 381 | 381 | } while (pte += nr_ptes, addr += nr_ptes * PAGE_SIZE, addr != end); |
| 382 | − arch_leave_lazy_mmu_mode(); | |
| 382 | + lazy_mmu_mode_disable(); | |
| 383 | 383 | pte_unmap_unlock(pte - 1, ptl); |
| 384 | 384 | |
| 385 | 385 | return pages; |
| @@ -260,7 +260,7 @@ static int move_ptes(struct pagetable_move_control *pmc, · 259 unmodified lines | ||
| 260 | 260 | if (new_ptl != old_ptl) |
| 261 | 261 | spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); |
| 262 | 262 | flush_tlb_batched_pending(vma->vm_mm); |
| 263 | − arch_enter_lazy_mmu_mode(); | |
| 263 | + lazy_mmu_mode_enable(); | |
| 264 | 264 | |
| 265 | 265 | for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes * PAGE_SIZE, |
| 266 | 266 | new_ptep += nr_ptes, new_addr += nr_ptes * PAGE_SIZE) { |
| @@ -305,7 +305,7 @@ static int move_ptes(struct pagetable_move_control *pmc, · 38 unmodified lines | ||
| 305 | 305 | } |
| 306 | 306 | } |
| 307 | 307 | |
| 308 | − arch_leave_lazy_mmu_mode(); | |
| 308 | + lazy_mmu_mode_disable(); | |
| 309 | 309 | if (force_flush) |
| 310 | 310 | flush_tlb_range(vma, old_end - len, old_end); |
| 311 | 311 | if (new_ptl != old_ptl) |
| @@ -678,7 +678,7 @@ static bool can_realign_addr(struct pagetable_move_control *pmc, · 366 unmodified lines | ||
| 678 | 678 | /* |
| 679 | 679 | * We don't want to have to go hunting for VMAs from the end of the old |
| 680 | 680 | * VMA to the next page table boundary, also we want to make sure the |
| 681 | − * operation is wortwhile. | |
| 681 | + * operation is worthwhile. | |
| 682 | 682 | * |
| 683 | 683 | * So ensure that we only perform this realignment if the end of the |
| 684 | 684 | * range being copied reaches or crosses the page table boundary. |
| @@ -926,7 +926,7 @@ static bool vrm_overlaps(struct vma_remap_struct *vrm) · 241 unmodified lines | ||
| 926 | 926 | /* |
| 927 | 927 | * Will a new address definitely be assigned? This either if the user specifies |
| 928 | 928 | * it via MREMAP_FIXED, or if MREMAP_DONTUNMAP is used, indicating we will |
| 929 | − * always detemrine a target address. | |
| 929 | + * always determine a target address. | |
| 930 | 930 | */ |
| 931 | 931 | static bool vrm_implies_new_addr(struct vma_remap_struct *vrm) |
| 932 | 932 | { |
| @@ -1806,7 +1806,7 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm) · 873 unmodified lines | ||
| 1806 | 1806 | /* |
| 1807 | 1807 | * move_vma() need us to stay 4 maps below the threshold, otherwise |
| 1808 | 1808 | * it will bail out at the very beginning. |
| 1809 | − * That is a problem if we have already unmaped the regions here | |
| 1809 | + * That is a problem if we have already unmapped the regions here | |
| 1810 | 1810 | * (new_addr, and old_addr), because userspace will not know the |
| 1811 | 1811 | * state of the vma's after it gets -ENOMEM. |
| 1812 | 1812 | * So, to avoid such scenario we can pre-compute if the whole |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | * It disallows unmapped regions from start to end whether they exist at the |
| 22 | 22 | * start, in the middle, or at the end of the range, or any combination thereof. |
| 23 | 23 | * |
| 24 | − * This is because after sealng a range, there's nothing to stop memory mapping | |
| 24 | + * This is because after sealing a range, there's nothing to stop memory mapping | |
| 25 | 25 | * of ranges in the remaining gaps later, meaning that the user might then |
| 26 | 26 | * wrongly consider the entirety of the mseal()'d range to be sealed when it |
| 27 | 27 | * in fact isn't. |
| @@ -124,7 +124,7 @@ static int mseal_apply(struct mm_struct *mm, · 96 unmodified lines | ||
| 124 | 124 | * -EINVAL: |
| 125 | 125 | * invalid input flags. |
| 126 | 126 | * start address is not page aligned. |
| 127 | − * Address arange (start + len) overflow. | |
| 127 | + * Address range (start + len) overflow. | |
| 128 | 128 | * -ENOMEM: |
| 129 | 129 | * addr is not a valid address (not allocated). |
| 130 | 130 | * end (start + len) is not a valid address. |
| @@ -467,7 +467,7 @@ int __init numa_memblks_init(int (*init_func)(void), · 466 unmodified lines | ||
| 467 | 467 | * We reset memblock back to the top-down direction |
| 468 | 468 | * here because if we configured ACPI_NUMA, we have |
| 469 | 469 | * parsed SRAT in init_func(). It is ok to have the |
| 470 | − * reset here even if we did't configure ACPI_NUMA | |
| 470 | + * reset here even if we didn't configure ACPI_NUMA | |
| 471 | 471 | * or acpi numa init fails and fallbacks to dummy |
| 472 | 472 | * numa init. |
| 473 | 473 | */ |
| @@ -228,7 +228,7 @@ long oom_badness(struct task_struct *p, unsigned long totalpages) · 227 unmodified lines | ||
| 228 | 228 | * The baseline for the badness score is the proportion of RAM that each |
| 229 | 229 | * task's rss, pagetable and swap space use. |
| 230 | 230 | */ |
| 231 | − points = get_mm_rss(p->mm) + get_mm_counter(p->mm, MM_SWAPENTS) + | |
| 231 | + points = get_mm_rss_sum(p->mm) + get_mm_counter_sum(p->mm, MM_SWAPENTS) + | |
| 232 | 232 | mm_pgtables_bytes(p->mm) / PAGE_SIZE; |
| 233 | 233 | task_unlock(p); |
| 234 | 234 | |
| @@ -402,10 +402,10 @@ static int dump_task(struct task_struct *p, void *arg) · 167 unmodified lines | ||
| 402 | 402 | |
| 403 | 403 | pr_info("[%7d] %5d %5d %8lu %8lu %8lu %8lu %9lu %8ld %8lu %5hd %s\n", |
| 404 | 404 | task->pid, from_kuid(&init_user_ns, task_uid(task)), |
| 405 | − task->tgid, task->mm->total_vm, get_mm_rss(task->mm), | |
| 406 | − get_mm_counter(task->mm, MM_ANONPAGES), get_mm_counter(task->mm, MM_FILEPAGES), | |
| 407 | − get_mm_counter(task->mm, MM_SHMEMPAGES), mm_pgtables_bytes(task->mm), | |
| 408 | − get_mm_counter(task->mm, MM_SWAPENTS), | |
| 405 | + task->tgid, task->mm->total_vm, get_mm_rss_sum(task->mm), | |
| 406 | + get_mm_counter_sum(task->mm, MM_ANONPAGES), get_mm_counter_sum(task->mm, MM_FILEPAGES), | |
| 407 | + get_mm_counter_sum(task->mm, MM_SHMEMPAGES), mm_pgtables_bytes(task->mm), | |
| 408 | + get_mm_counter_sum(task->mm, MM_SWAPENTS), | |
| 409 | 409 | task->signal->oom_score_adj, task->comm); |
| 410 | 410 | task_unlock(task); |
| 411 | 411 | |
| @@ -458,7 +458,7 @@ static void dump_oom_victim(struct oom_control *oc, struct task_struct *victim) · 46 unmodified lines | ||
| 458 | 458 | |
| 459 | 459 | static void dump_header(struct oom_control *oc) |
| 460 | 460 | { |
| 461 | − pr_warn("%s invoked oom-killer: gfp_mask=%#x(%pGg), order=%d, oom_score_adj=%hd\n", | |
| 461 | + pr_warn("%s invoked oom-killer: gfp_mask=%#x(%pGg), order=%d, oom_score_adj=%d\n", | |
| 462 | 462 | current->comm, oc->gfp_mask, &oc->gfp_mask, oc->order, |
| 463 | 463 | current->signal->oom_score_adj); |
| 464 | 464 | if (!IS_ENABLED(CONFIG_COMPACTION) && oc->order) |
| @@ -604,9 +604,9 @@ static bool oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm) · 139 unmodified lines | ||
| 604 | 604 | |
| 605 | 605 | pr_info("oom_reaper: reaped process %d (%s), now anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB\n", |
| 606 | 606 | task_pid_nr(tsk), tsk->comm, |
| 607 | − K(get_mm_counter(mm, MM_ANONPAGES)), | |
| 608 | − K(get_mm_counter(mm, MM_FILEPAGES)), | |
| 609 | − K(get_mm_counter(mm, MM_SHMEMPAGES))); | |
| 607 | + K(get_mm_counter_sum(mm, MM_ANONPAGES)), | |
| 608 | + K(get_mm_counter_sum(mm, MM_FILEPAGES)), | |
| 609 | + K(get_mm_counter_sum(mm, MM_SHMEMPAGES))); | |
| 610 | 610 | out_finish: |
| 611 | 611 | trace_finish_task_reaping(tsk->pid); |
| 612 | 612 | out_unlock: |
| @@ -958,11 +958,11 @@ static void __oom_kill_process(struct task_struct *victim, const char *message) · 345 unmodified lines | ||
| 958 | 958 | */ |
| 959 | 959 | do_send_sig_info(SIGKILL, SEND_SIG_PRIV, victim, PIDTYPE_TGID); |
| 960 | 960 | mark_oom_victim(victim); |
| 961 | − pr_err("%s: Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB, UID:%u pgtables:%lukB oom_score_adj:%hd\n", | |
| 961 | + pr_err("%s: Killed process %d (%s) total-vm:%lukB, anon-rss:%lukB, file-rss:%lukB, shmem-rss:%lukB, UID:%u pgtables:%lukB oom_score_adj:%d\n", | |
| 962 | 962 | message, task_pid_nr(victim), victim->comm, K(mm->total_vm), |
| 963 | − K(get_mm_counter(mm, MM_ANONPAGES)), | |
| 964 | − K(get_mm_counter(mm, MM_FILEPAGES)), | |
| 965 | − K(get_mm_counter(mm, MM_SHMEMPAGES)), | |
| 963 | + K(get_mm_counter_sum(mm, MM_ANONPAGES)), | |
| 964 | + K(get_mm_counter_sum(mm, MM_FILEPAGES)), | |
| 965 | + K(get_mm_counter_sum(mm, MM_SHMEMPAGES)), | |
| 966 | 966 | from_kuid(&init_user_ns, task_uid(victim)), |
| 967 | 967 | mm_pgtables_bytes(mm) >> 10, victim->signal->oom_score_adj); |
| 968 | 968 | task_unlock(victim); |
| @@ -109,14 +109,6 @@ EXPORT_SYMBOL_GPL(dirty_writeback_interval); · 108 unmodified lines | ||
| 109 | 109 | */ |
| 110 | 110 | unsigned int dirty_expire_interval = 30 * 100; /* centiseconds */ |
| 111 | 111 | |
| 112 | −/* | |
| 113 | − * Flag that puts the machine in "laptop mode". Doubles as a timeout in jiffies: | |
| 114 | − * a full sync is triggered after this time elapses without any disk activity. | |
| 115 | − */ | |
| 116 | −int laptop_mode; | |
| 117 | − | |
| 118 | −EXPORT_SYMBOL(laptop_mode); | |
| 119 | − | |
| 120 | 112 | /* End of sysctl-exported parameters */ |
| 121 | 113 | |
| 122 | 114 | struct wb_domain global_wb_domain; |
| @@ -1843,17 +1835,7 @@ static int balance_dirty_pages(struct bdi_writeback *wb, · 1720 unmodified lines | ||
| 1843 | 1835 | balance_domain_limits(mdtc, strictlimit); |
| 1844 | 1836 | } |
| 1845 | 1837 | |
| 1846 | − /* | |
| 1847 | − * In laptop mode, we wait until hitting the higher threshold | |
| 1848 | − * before starting background writeout, and then write out all | |
| 1849 | − * the way down to the lower threshold. So slow writers cause | |
| 1850 | − * minimal disk activity. | |
| 1851 | − * | |
| 1852 | − * In normal mode, we start background writeout at the lower | |
| 1853 | − * background_thresh, to keep the amount of dirty memory low. | |
| 1854 | − */ | |
| 1855 | − if (!laptop_mode && nr_dirty > gdtc->bg_thresh && | |
| 1856 | − !writeback_in_progress(wb)) | |
| 1838 | + if (nr_dirty > gdtc->bg_thresh && !writeback_in_progress(wb)) | |
| 1857 | 1839 | wb_start_background_writeback(wb); |
| 1858 | 1840 | |
| 1859 | 1841 | /* |
| @@ -1876,10 +1858,6 @@ free_running: · 16 unmodified lines | ||
| 1876 | 1858 | break; |
| 1877 | 1859 | } |
| 1878 | 1860 | |
| 1879 | − /* Start writeback even when in laptop mode */ | |
| 1880 | − if (unlikely(!writeback_in_progress(wb))) | |
| 1881 | − wb_start_background_writeback(wb); | |
| 1882 | − | |
| 1883 | 1861 | mem_cgroup_flush_foreign(wb); |
| 1884 | 1862 | |
| 1885 | 1863 | /* |
| @@ -2198,41 +2176,6 @@ static int dirty_writeback_centisecs_handler(const struct ctl_table *table, int · 312 unmodified lines | ||
| 2198 | 2176 | } |
| 2199 | 2177 | #endif |
| 2200 | 2178 | |
| 2201 | −void laptop_mode_timer_fn(struct timer_list *t) | |
| 2202 | −{ | |
| 2203 | − struct backing_dev_info *backing_dev_info = | |
| 2204 | − timer_container_of(backing_dev_info, t, laptop_mode_wb_timer); | |
| 2205 | − | |
| 2206 | − wakeup_flusher_threads_bdi(backing_dev_info, WB_REASON_LAPTOP_TIMER); | |
| 2207 | −} | |
| 2208 | − | |
| 2209 | −/* | |
| 2210 | − * We've spun up the disk and we're in laptop mode: schedule writeback | |
| 2211 | − * of all dirty data a few seconds from now. If the flush is already scheduled | |
| 2212 | − * then push it back - the user is still using the disk. | |
| 2213 | − */ | |
| 2214 | −void laptop_io_completion(struct backing_dev_info *info) | |
| 2215 | −{ | |
| 2216 | − mod_timer(&info->laptop_mode_wb_timer, jiffies + laptop_mode); | |
| 2217 | −} | |
| 2218 | − | |
| 2219 | −/* | |
| 2220 | − * We're in laptop mode and we've just synced. The sync's writes will have | |
| 2221 | − * caused another writeback to be scheduled by laptop_io_completion. | |
| 2222 | − * Nothing needs to be written back anymore, so we unschedule the writeback. | |
| 2223 | − */ | |
| 2224 | −void laptop_sync_completion(void) | |
| 2225 | −{ | |
| 2226 | − struct backing_dev_info *bdi; | |
| 2227 | − | |
| 2228 | − rcu_read_lock(); | |
| 2229 | − | |
| 2230 | − list_for_each_entry_rcu(bdi, &bdi_list, bdi_list) | |
| 2231 | − timer_delete(&bdi->laptop_mode_wb_timer); | |
| 2232 | − | |
| 2233 | − rcu_read_unlock(); | |
| 2234 | −} | |
| 2235 | − | |
| 2236 | 2179 | /* |
| 2237 | 2180 | * If ratelimit_pages is too high then we can get into dirty-data overload |
| 2238 | 2181 | * if a large number of processes all perform writes at the same time. |
| @@ -2263,6 +2206,19 @@ static int page_writeback_cpu_online(unsigned int cpu) · 24 unmodified lines | ||
| 2263 | 2206 | |
| 2264 | 2207 | #ifdef CONFIG_SYSCTL |
| 2265 | 2208 | |
| 2209 | +static int laptop_mode; | |
| 2210 | +static int laptop_mode_handler(const struct ctl_table *table, int write, | |
| 2211 | + void *buffer, size_t *lenp, loff_t *ppos) | |
| 2212 | +{ | |
| 2213 | + int ret = proc_dointvec_jiffies(table, write, buffer, lenp, ppos); | |
| 2214 | + | |
| 2215 | + if (!ret && write) | |
| 2216 | + pr_warn("%s: vm.laptop_mode is deprecated. Ignoring setting.\n", | |
| 2217 | + current->comm); | |
| 2218 | + | |
| 2219 | + return ret; | |
| 2220 | +} | |
| 2221 | + | |
| 2266 | 2222 | /* this is needed for the proc_doulongvec_minmax of vm_dirty_bytes */ |
| 2267 | 2223 | static const unsigned long dirty_bytes_min = 2 * PAGE_SIZE; |
| 2268 | 2224 | |
| @@ -2332,7 +2288,7 @@ static const struct ctl_table vm_page_writeback_sysctls[] = { · 63 unmodified lines | ||
| 2332 | 2288 | .data = &laptop_mode, |
| 2333 | 2289 | .maxlen = sizeof(laptop_mode), |
| 2334 | 2290 | .mode = 0644, |
| 2335 | − .proc_handler = proc_dointvec_jiffies, | |
| 2291 | + .proc_handler = laptop_mode_handler, | |
| 2336 | 2292 | }, |
| 2337 | 2293 | }; |
| 2338 | 2294 | #endif |
| @@ -1,6 +1,5 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | /* |
| 3 | − * linux/mm/page_alloc.c | |
| 4 | 3 | * |
| 5 | 4 | * Manages the free list, the system allocates free pages here. |
| 6 | 5 | * Note that kmalloc() lives in slab.c |
| @@ -1853,7 +1852,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, · 1846 unmodified lines | ||
| 1853 | 1852 | |
| 1854 | 1853 | /* |
| 1855 | 1854 | * As memory initialization might be integrated into KASAN, |
| 1856 | − * KASAN unpoisoning and memory initializion code must be | |
| 1855 | + * KASAN unpoisoning and memory initialization code must be | |
| 1857 | 1856 | * kept together to avoid discrepancies in behavior. |
| 1858 | 1857 | */ |
| 1859 | 1858 | |
| @@ -2946,9 +2945,9 @@ static bool free_frozen_page_commit(struct zone *zone, · 1086 unmodified lines | ||
| 2946 | 2945 | * 'hopeless node' to stay in that state for a while. Let |
| 2947 | 2946 | * kswapd work again by resetting kswapd_failures. |
| 2948 | 2947 | */ |
| 2949 | − if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES && | |
| 2948 | + if (kswapd_test_hopeless(pgdat) && | |
| 2950 | 2949 | next_memory_node(pgdat->node_id) < MAX_NUMNODES) |
| 2951 | − atomic_set(&pgdat->kswapd_failures, 0); | |
| 2950 | + kswapd_clear_hopeless(pgdat, KSWAPD_CLEAR_HOPELESS_PCP); | |
| 2952 | 2951 | } |
| 2953 | 2952 | return ret; |
| 2954 | 2953 | } |
| @@ -3112,6 +3111,15 @@ void free_unref_folios(struct folio_batch *folios) · 157 unmodified lines | ||
| 3112 | 3111 | folio_batch_reinit(folios); |
| 3113 | 3112 | } |
| 3114 | 3113 | |
| 3114 | +static void __split_page(struct page *page, unsigned int order) | |
| 3115 | +{ | |
| 3116 | + VM_WARN_ON_PAGE(PageCompound(page), page); | |
| 3117 | + | |
| 3118 | + split_page_owner(page, order, 0); | |
| 3119 | + pgalloc_tag_split(page_folio(page), order, 0); | |
| 3120 | + split_page_memcg(page, order); | |
| 3121 | +} | |
| 3122 | + | |
| 3115 | 3123 | /* |
| 3116 | 3124 | * split_page takes a non-compound higher-order page, and splits it into |
| 3117 | 3125 | * n (1<<order) sub-pages: page[0..n] |
| @@ -3124,14 +3132,12 @@ void split_page(struct page *page, unsigned int order) · 6 unmodified lines | ||
| 3124 | 3132 | { |
| 3125 | 3133 | int i; |
| 3126 | 3134 | |
| 3127 | − VM_BUG_ON_PAGE(PageCompound(page), page); | |
| 3128 | − VM_BUG_ON_PAGE(!page_count(page), page); | |
| 3135 | + VM_WARN_ON_PAGE(!page_count(page), page); | |
| 3129 | 3136 | |
| 3130 | 3137 | for (i = 1; i < (1 << order); i++) |
| 3131 | 3138 | set_page_refcounted(page + i); |
| 3132 | − split_page_owner(page, order, 0); | |
| 3133 | − pgalloc_tag_split(page_folio(page), order, 0); | |
| 3134 | − split_page_memcg(page, order); | |
| 3139 | + | |
| 3140 | + __split_page(page, order); | |
| 3135 | 3141 | } |
| 3136 | 3142 | EXPORT_SYMBOL_GPL(split_page); |
| 3137 | 3143 | |
| @@ -4699,7 +4705,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · 1561 unmodified lines | ||
| 4699 | 4705 | struct alloc_context *ac) |
| 4700 | 4706 | { |
| 4701 | 4707 | bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM; |
| 4702 | − bool can_compact = gfp_compaction_allowed(gfp_mask); | |
| 4708 | + bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask); | |
| 4703 | 4709 | bool nofail = gfp_mask & __GFP_NOFAIL; |
| 4704 | 4710 | const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER; |
| 4705 | 4711 | struct page *page = NULL; |
| @@ -4712,6 +4718,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, · 6 unmodified lines | ||
| 4712 | 4718 | unsigned int cpuset_mems_cookie; |
| 4713 | 4719 | unsigned int zonelist_iter_cookie; |
| 4714 | 4720 | int reserve_flags; |
| 4721 | + bool compact_first = false; | |
| 4722 | + bool can_retry_reserves = true; | |
| 4715 | 4723 | |
| 4716 | 4724 | if (unlikely(nofail)) { |
| 4717 | 4725 | /* |
| @@ -4735,6 +4743,19 @@ restart: · 17 unmodified lines | ||
| 4735 | 4743 | cpuset_mems_cookie = read_mems_allowed_begin(); |
| 4736 | 4744 | zonelist_iter_cookie = zonelist_iter_begin(); |
| 4737 | 4745 | |
| 4746 | + /* | |
| 4747 | + * For costly allocations, try direct compaction first, as it's likely | |
| 4748 | + * that we have enough base pages and don't need to reclaim. For non- | |
| 4749 | + * movable high-order allocations, do that as well, as compaction will | |
| 4750 | + * try prevent permanent fragmentation by migrating from blocks of the | |
| 4751 | + * same migratetype. | |
| 4752 | + */ | |
| 4753 | + if (can_compact && (costly_order || (order > 0 && | |
| 4754 | + ac->migratetype != MIGRATE_MOVABLE))) { | |
| 4755 | + compact_first = true; | |
| 4756 | + compact_priority = INIT_COMPACT_PRIORITY; | |
| 4757 | + } | |
| 4758 | + | |
| 4738 | 4759 | /* |
| 4739 | 4760 | * The fast path uses conservative alloc_flags to succeed only until |
| 4740 | 4761 | * kswapd needs to be woken up, and to avoid the cost of setting up |
| @@ -4766,6 +4787,8 @@ restart: · 25 unmodified lines | ||
| 4766 | 4787 | goto nopage; |
| 4767 | 4788 | } |
| 4768 | 4789 | |
| 4790 | +retry: | |
| 4791 | + /* Ensure kswapd doesn't accidentally go to sleep as long as we loop */ | |
| 4769 | 4792 | if (alloc_flags & ALLOC_KSWAPD) |
| 4770 | 4793 | wake_all_kswapds(order, gfp_mask, ac); |
| 4771 | 4794 | |
| @@ -4777,74 +4800,6 @@ restart: · 5 unmodified lines | ||
| 4777 | 4800 | if (page) |
| 4778 | 4801 | goto got_pg; |
| 4779 | 4802 | |
| 4780 | − /* | |
| 4781 | − * For costly allocations, try direct compaction first, as it's likely | |
| 4782 | − * that we have enough base pages and don't need to reclaim. For non- | |
| 4783 | − * movable high-order allocations, do that as well, as compaction will | |
| 4784 | − * try prevent permanent fragmentation by migrating from blocks of the | |
| 4785 | − * same migratetype. | |
| 4786 | − * Don't try this for allocations that are allowed to ignore | |
| 4787 | − * watermarks, as the ALLOC_NO_WATERMARKS attempt didn't yet happen. | |
| 4788 | − */ | |
| 4789 | − if (can_direct_reclaim && can_compact && | |
| 4790 | − (costly_order || | |
| 4791 | − (order > 0 && ac->migratetype != MIGRATE_MOVABLE)) | |
| 4792 | − && !gfp_pfmemalloc_allowed(gfp_mask)) { | |
| 4793 | − page = __alloc_pages_direct_compact(gfp_mask, order, | |
| 4794 | − alloc_flags, ac, | |
| 4795 | − INIT_COMPACT_PRIORITY, | |
| 4796 | − &compact_result); | |
| 4797 | − if (page) | |
| 4798 | − goto got_pg; | |
| 4799 | − | |
| 4800 | − /* | |
| 4801 | − * Checks for costly allocations with __GFP_NORETRY, which | |
| 4802 | − * includes some THP page fault allocations | |
| 4803 | − */ | |
| 4804 | − if (costly_order && (gfp_mask & __GFP_NORETRY)) { | |
| 4805 | − /* | |
| 4806 | − * If allocating entire pageblock(s) and compaction | |
| 4807 | − * failed because all zones are below low watermarks | |
| 4808 | − * or is prohibited because it recently failed at this | |
| 4809 | − * order, fail immediately unless the allocator has | |
| 4810 | − * requested compaction and reclaim retry. | |
| 4811 | − * | |
| 4812 | − * Reclaim is | |
| 4813 | − * - potentially very expensive because zones are far | |
| 4814 | − * below their low watermarks or this is part of very | |
| 4815 | − * bursty high order allocations, | |
| 4816 | − * - not guaranteed to help because isolate_freepages() | |
| 4817 | − * may not iterate over freed pages as part of its | |
| 4818 | − * linear scan, and | |
| 4819 | − * - unlikely to make entire pageblocks free on its | |
| 4820 | − * own. | |
| 4821 | − */ | |
| 4822 | − if (compact_result == COMPACT_SKIPPED || | |
| 4823 | − compact_result == COMPACT_DEFERRED) | |
| 4824 | − goto nopage; | |
| 4825 | − | |
| 4826 | − /* | |
| 4827 | − * Looks like reclaim/compaction is worth trying, but | |
| 4828 | − * sync compaction could be very expensive, so keep | |
| 4829 | − * using async compaction. | |
| 4830 | − */ | |
| 4831 | − compact_priority = INIT_COMPACT_PRIORITY; | |
| 4832 | − } | |
| 4833 | − } | |
| 4834 | − | |
| 4835 | −retry: | |
| 4836 | − /* | |
| 4837 | − * Deal with possible cpuset update races or zonelist updates to avoid | |
| 4838 | − * infinite retries. | |
| 4839 | − */ | |
| 4840 | − if (check_retry_cpuset(cpuset_mems_cookie, ac) || | |
| 4841 | − check_retry_zonelist(zonelist_iter_cookie)) | |
| 4842 | − goto restart; | |
| 4843 | − | |
| 4844 | − /* Ensure kswapd doesn't accidentally go to sleep as long as we loop */ | |
| 4845 | − if (alloc_flags & ALLOC_KSWAPD) | |
| 4846 | − wake_all_kswapds(order, gfp_mask, ac); | |
| 4847 | − | |
| 4848 | 4803 | reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); |
| 4849 | 4804 | if (reserve_flags) |
| 4850 | 4805 | alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, reserve_flags) | |
| @@ -4859,12 +4814,18 @@ retry: · 8 unmodified lines | ||
| 4859 | 4814 | ac->nodemask = NULL; |
| 4860 | 4815 | ac->preferred_zoneref = first_zones_zonelist(ac->zonelist, |
| 4861 | 4816 | ac->highest_zoneidx, ac->nodemask); |
| 4862 | − } | |
| 4863 | 4817 | |
| 4864 | − /* Attempt with potentially adjusted zonelist and alloc_flags */ | |
| 4865 | − page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); | |
| 4866 | − if (page) | |
| 4867 | − goto got_pg; | |
| 4818 | + /* | |
| 4819 | + * The first time we adjust anything due to being allowed to | |
| 4820 | + * ignore memory policies or watermarks, retry immediately. This | |
| 4821 | + * allows us to keep the first allocation attempt optimistic so | |
| 4822 | + * it can succeed in a zone that is still above watermarks. | |
| 4823 | + */ | |
| 4824 | + if (can_retry_reserves) { | |
| 4825 | + can_retry_reserves = false; | |
| 4826 | + goto retry; | |
| 4827 | + } | |
| 4828 | + } | |
| 4868 | 4829 | |
| 4869 | 4830 | /* Caller is not willing to reclaim, we can't balance anything */ |
| 4870 | 4831 | if (!can_direct_reclaim) |
| @@ -4875,10 +4836,12 @@ retry: · 4 unmodified lines | ||
| 4875 | 4836 | goto nopage; |
| 4876 | 4837 | |
| 4877 | 4838 | /* Try direct reclaim and then allocating */ |
| 4878 | − page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags, ac, | |
| 4879 | − &did_some_progress); | |
| 4880 | − if (page) | |
| 4881 | − goto got_pg; | |
| 4839 | + if (!compact_first) { | |
| 4840 | + page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags, | |
| 4841 | + ac, &did_some_progress); | |
| 4842 | + if (page) | |
| 4843 | + goto got_pg; | |
| 4844 | + } | |
| 4882 | 4845 | |
| 4883 | 4846 | /* Try direct compaction and then allocating */ |
| 4884 | 4847 | page = __alloc_pages_direct_compact(gfp_mask, order, alloc_flags, ac, |
| @@ -4886,6 +4849,33 @@ retry: · 1 unmodified line | ||
| 4886 | 4849 | if (page) |
| 4887 | 4850 | goto got_pg; |
| 4888 | 4851 | |
| 4852 | + if (compact_first) { | |
| 4853 | + /* | |
| 4854 | + * THP page faults may attempt local node only first, but are | |
| 4855 | + * then allowed to only compact, not reclaim, see | |
| 4856 | + * alloc_pages_mpol(). | |
| 4857 | + * | |
| 4858 | + * Compaction has failed above and we don't want such THP | |
| 4859 | + * allocations to put reclaim pressure on a single node in a | |
| 4860 | + * situation where other nodes might have plenty of available | |
| 4861 | + * memory. | |
| 4862 | + */ | |
| 4863 | + if (gfp_has_flags(gfp_mask, __GFP_NORETRY | __GFP_THISNODE)) | |
| 4864 | + goto nopage; | |
| 4865 | + | |
| 4866 | + /* | |
| 4867 | + * For the initial compaction attempt we have lowered its | |
| 4868 | + * priority. Restore it for further retries, if those are | |
| 4869 | + * allowed. With __GFP_NORETRY there will be a single round of | |
| 4870 | + * reclaim and compaction with the lowered priority. | |
| 4871 | + */ | |
| 4872 | + if (!(gfp_mask & __GFP_NORETRY)) | |
| 4873 | + compact_priority = DEF_COMPACT_PRIORITY; | |
| 4874 | + | |
| 4875 | + compact_first = false; | |
| 4876 | + goto retry; | |
| 4877 | + } | |
| 4878 | + | |
| 4889 | 4879 | /* Do not loop if specifically requested */ |
| 4890 | 4880 | if (gfp_mask & __GFP_NORETRY) |
| 4891 | 4881 | goto nopage; |
| @@ -4898,6 +4888,15 @@ retry: · 6 unmodified lines | ||
| 4898 | 4888 | !(gfp_mask & __GFP_RETRY_MAYFAIL))) |
| 4899 | 4889 | goto nopage; |
| 4900 | 4890 | |
| 4891 | + /* | |
| 4892 | + * Deal with possible cpuset update races or zonelist updates to avoid | |
| 4893 | + * infinite retries. No "goto retry;" can be placed above this check | |
| 4894 | + * unless it can execute just once. | |
| 4895 | + */ | |
| 4896 | + if (check_retry_cpuset(cpuset_mems_cookie, ac) || | |
| 4897 | + check_retry_zonelist(zonelist_iter_cookie)) | |
| 4898 | + goto restart; | |
| 4899 | + | |
| 4901 | 4900 | if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags, |
| 4902 | 4901 | did_some_progress > 0, &no_progress_loops)) |
| 4903 | 4902 | goto retry; |
| @@ -5401,9 +5400,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, · 497 unmodified lines | ||
| 5401 | 5400 | struct page *page = virt_to_page((void *)addr); |
| 5402 | 5401 | struct page *last = page + nr; |
| 5403 | 5402 | |
| 5404 | − split_page_owner(page, order, 0); | |
| 5405 | − pgalloc_tag_split(page_folio(page), order, 0); | |
| 5406 | − split_page_memcg(page, order); | |
| 5403 | + __split_page(page, order); | |
| 5407 | 5404 | while (page < --last) |
| 5408 | 5405 | set_page_refcounted(last); |
| 5409 | 5406 | |
| @@ -6896,7 +6893,7 @@ static int __alloc_contig_migrate_range(struct compact_control *cc, · 1486 unmodified lines | ||
| 6896 | 6893 | return (ret < 0) ? ret : 0; |
| 6897 | 6894 | } |
| 6898 | 6895 | |
| 6899 | −static void split_free_pages(struct list_head *list, gfp_t gfp_mask) | |
| 6896 | +static void split_free_frozen_pages(struct list_head *list, gfp_t gfp_mask) | |
| 6900 | 6897 | { |
| 6901 | 6898 | int order; |
| 6902 | 6899 | |
| @@ -6908,11 +6905,10 @@ static void split_free_pages(struct list_head *list, gfp_t gfp_mask) · 5 unmodified lines | ||
| 6908 | 6905 | int i; |
| 6909 | 6906 | |
| 6910 | 6907 | post_alloc_hook(page, order, gfp_mask); |
| 6911 | − set_page_refcounted(page); | |
| 6912 | 6908 | if (!order) |
| 6913 | 6909 | continue; |
| 6914 | 6910 | |
| 6915 | − split_page(page, order); | |
| 6911 | + __split_page(page, order); | |
| 6916 | 6912 | |
| 6917 | 6913 | /* Add all subpages to the order-0 head, in sequence. */ |
| 6918 | 6914 | list_del(&page->lru); |
| @@ -6956,8 +6952,14 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask) · 37 unmodified lines | ||
| 6956 | 6952 | return 0; |
| 6957 | 6953 | } |
| 6958 | 6954 | |
| 6955 | +static void __free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages) | |
| 6956 | +{ | |
| 6957 | + for (; nr_pages--; pfn++) | |
| 6958 | + free_frozen_pages(pfn_to_page(pfn), 0); | |
| 6959 | +} | |
| 6960 | + | |
| 6959 | 6961 | /** |
| 6960 | − * alloc_contig_range() -- tries to allocate given range of pages | |
| 6962 | + * alloc_contig_frozen_range() -- tries to allocate given range of frozen pages | |
| 6961 | 6963 | * @start: start PFN to allocate |
| 6962 | 6964 | * @end: one-past-the-last PFN to allocate |
| 6963 | 6965 | * @alloc_flags: allocation information |
| @@ -6972,12 +6974,15 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask) · 8 unmodified lines | ||
| 6972 | 6974 | * pageblocks in the range. Once isolated, the pageblocks should not |
| 6973 | 6975 | * be modified by others. |
| 6974 | 6976 | * |
| 6975 | − * Return: zero on success or negative error code. On success all | |
| 6976 | − * pages which PFN is in [start, end) are allocated for the caller and | |
| 6977 | − * need to be freed with free_contig_range(). | |
| 6977 | + * All frozen pages which PFN is in [start, end) are allocated for the | |
| 6978 | + * caller, and they could be freed with free_contig_frozen_range(), | |
| 6979 | + * free_frozen_pages() also could be used to free compound frozen pages | |
| 6980 | + * directly. | |
| 6981 | + * | |
| 6982 | + * Return: zero on success or negative error code. | |
| 6978 | 6983 | */ |
| 6979 | −int alloc_contig_range_noprof(unsigned long start, unsigned long end, | |
| 6980 | − acr_flags_t alloc_flags, gfp_t gfp_mask) | |
| 6984 | +int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end, | |
| 6985 | + acr_flags_t alloc_flags, gfp_t gfp_mask) | |
| 6981 | 6986 | { |
| 6982 | 6987 | const unsigned int order = ilog2(end - start); |
| 6983 | 6988 | unsigned long outer_start, outer_end; |
| @@ -7093,19 +7098,18 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, · 109 unmodified lines | ||
| 7093 | 7098 | } |
| 7094 | 7099 | |
| 7095 | 7100 | if (!(gfp_mask & __GFP_COMP)) { |
| 7096 | − split_free_pages(cc.freepages, gfp_mask); | |
| 7101 | + split_free_frozen_pages(cc.freepages, gfp_mask); | |
| 7097 | 7102 | |
| 7098 | 7103 | /* Free head and tail (if any) */ |
| 7099 | 7104 | if (start != outer_start) |
| 7100 | − free_contig_range(outer_start, start - outer_start); | |
| 7105 | + __free_contig_frozen_range(outer_start, start - outer_start); | |
| 7101 | 7106 | if (end != outer_end) |
| 7102 | − free_contig_range(end, outer_end - end); | |
| 7107 | + __free_contig_frozen_range(end, outer_end - end); | |
| 7103 | 7108 | } else if (start == outer_start && end == outer_end && is_power_of_2(end - start)) { |
| 7104 | 7109 | struct page *head = pfn_to_page(start); |
| 7105 | 7110 | |
| 7106 | 7111 | check_new_pages(head, order); |
| 7107 | 7112 | prep_new_page(head, order, gfp_mask, 0); |
| 7108 | − set_page_refcounted(head); | |
| 7109 | 7113 | } else { |
| 7110 | 7114 | ret = -EINVAL; |
| 7111 | 7115 | WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n", |
| @@ -7115,36 +7119,86 @@ done: · 3 unmodified lines | ||
| 7115 | 7119 | undo_isolate_page_range(start, end); |
| 7116 | 7120 | return ret; |
| 7117 | 7121 | } |
| 7118 | −EXPORT_SYMBOL(alloc_contig_range_noprof); | |
| 7122 | +EXPORT_SYMBOL(alloc_contig_frozen_range_noprof); | |
| 7119 | 7123 | |
| 7120 | −static int __alloc_contig_pages(unsigned long start_pfn, | |
| 7121 | − unsigned long nr_pages, gfp_t gfp_mask) | |
| 7124 | +/** | |
| 7125 | + * alloc_contig_range() -- tries to allocate given range of pages | |
| 7126 | + * @start: start PFN to allocate | |
| 7127 | + * @end: one-past-the-last PFN to allocate | |
| 7128 | + * @alloc_flags: allocation information | |
| 7129 | + * @gfp_mask: GFP mask. | |
| 7130 | + * | |
| 7131 | + * This routine is a wrapper around alloc_contig_frozen_range(), it can't | |
| 7132 | + * be used to allocate compound pages, the refcount of each allocated page | |
| 7133 | + * will be set to one. | |
| 7134 | + * | |
| 7135 | + * All pages which PFN is in [start, end) are allocated for the caller, | |
| 7136 | + * and should be freed with free_contig_range() or by manually calling | |
| 7137 | + * __free_page() on each allocated page. | |
| 7138 | + * | |
| 7139 | + * Return: zero on success or negative error code. | |
| 7140 | + */ | |
| 7141 | +int alloc_contig_range_noprof(unsigned long start, unsigned long end, | |
| 7142 | + acr_flags_t alloc_flags, gfp_t gfp_mask) | |
| 7122 | 7143 | { |
| 7123 | − unsigned long end_pfn = start_pfn + nr_pages; | |
| 7144 | + int ret; | |
| 7124 | 7145 | |
| 7125 | − return alloc_contig_range_noprof(start_pfn, end_pfn, ACR_FLAGS_NONE, | |
| 7126 | − gfp_mask); | |
| 7146 | + if (WARN_ON(gfp_mask & __GFP_COMP)) | |
| 7147 | + return -EINVAL; | |
| 7148 | + | |
| 7149 | + ret = alloc_contig_frozen_range_noprof(start, end, alloc_flags, gfp_mask); | |
| 7150 | + if (!ret) | |
| 7151 | + set_pages_refcounted(pfn_to_page(start), end - start); | |
| 7152 | + | |
| 7153 | + return ret; | |
| 7127 | 7154 | } |
| 7155 | +EXPORT_SYMBOL(alloc_contig_range_noprof); | |
| 7128 | 7156 | |
| 7129 | 7157 | static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, |
| 7130 | − unsigned long nr_pages) | |
| 7158 | + unsigned long nr_pages, bool skip_hugetlb, | |
| 7159 | + bool *skipped_hugetlb) | |
| 7131 | 7160 | { |
| 7132 | − unsigned long i, end_pfn = start_pfn + nr_pages; | |
| 7161 | + unsigned long end_pfn = start_pfn + nr_pages; | |
| 7133 | 7162 | struct page *page; |
| 7134 | 7163 | |
| 7135 | − for (i = start_pfn; i < end_pfn; i++) { | |
| 7136 | − page = pfn_to_online_page(i); | |
| 7164 | + while (start_pfn < end_pfn) { | |
| 7165 | + unsigned long step = 1; | |
| 7166 | + | |
| 7167 | + page = pfn_to_online_page(start_pfn); | |
| 7137 | 7168 | if (!page) |
| 7138 | 7169 | return false; |
| 7139 | 7170 | |
| 7140 | 7171 | if (page_zone(page) != z) |
| 7141 | 7172 | return false; |
| 7142 | 7173 | |
| 7143 | − if (PageReserved(page)) | |
| 7174 | + if (page_is_unmovable(z, page, PB_ISOLATE_MODE_OTHER, &step)) | |
| 7144 | 7175 | return false; |
| 7145 | 7176 | |
| 7146 | − if (PageHuge(page)) | |
| 7147 | − return false; | |
| 7177 | + /* | |
| 7178 | + * Only consider ranges containing hugepages if those pages are | |
| 7179 | + * smaller than the requested contiguous region. e.g.: | |
| 7180 | + * Move 2MB pages to free up a 1GB range. | |
| 7181 | + * Don't move 1GB pages to free up a 2MB range. | |
| 7182 | + * | |
| 7183 | + * This makes contiguous allocation more reliable if multiple | |
| 7184 | + * hugepage sizes are used without causing needless movement. | |
| 7185 | + */ | |
| 7186 | + if (PageHuge(page)) { | |
| 7187 | + unsigned int order; | |
| 7188 | + | |
| 7189 | + if (skip_hugetlb) { | |
| 7190 | + *skipped_hugetlb = true; | |
| 7191 | + return false; | |
| 7192 | + } | |
| 7193 | + | |
| 7194 | + page = compound_head(page); | |
| 7195 | + order = compound_order(page); | |
| 7196 | + if ((order >= MAX_FOLIO_ORDER) || | |
| 7197 | + (nr_pages <= (1 << order))) | |
| 7198 | + return false; | |
| 7199 | + } | |
| 7200 | + | |
| 7201 | + start_pfn += step; | |
| 7148 | 7202 | } |
| 7149 | 7203 | return true; |
| 7150 | 7204 | } |
| @@ -7158,7 +7212,7 @@ static bool zone_spans_last_pfn(const struct zone *zone, · 7 unmodified lines | ||
| 7158 | 7212 | } |
| 7159 | 7213 | |
| 7160 | 7214 | /** |
| 7161 | − * alloc_contig_pages() -- tries to find and allocate contiguous range of pages | |
| 7215 | + * alloc_contig_frozen_pages() -- tries to find and allocate contiguous range of frozen pages | |
| 7162 | 7216 | * @nr_pages: Number of contiguous pages to allocate |
| 7163 | 7217 | * @gfp_mask: GFP mask. Node/zone/placement hints limit the search; only some |
| 7164 | 7218 | * action and reclaim modifiers are supported. Reclaim modifiers |
| @@ -7166,28 +7220,34 @@ static bool zone_spans_last_pfn(const struct zone *zone, · 1 unmodified line | ||
| 7166 | 7220 | * @nid: Target node |
| 7167 | 7221 | * @nodemask: Mask for other possible nodes |
| 7168 | 7222 | * |
| 7169 | − * This routine is a wrapper around alloc_contig_range(). It scans over zones | |
| 7170 | − * on an applicable zonelist to find a contiguous pfn range which can then be | |
| 7171 | − * tried for allocation with alloc_contig_range(). This routine is intended | |
| 7172 | − * for allocation requests which can not be fulfilled with the buddy allocator. | |
| 7223 | + * This routine is a wrapper around alloc_contig_frozen_range(). It scans over | |
| 7224 | + * zones on an applicable zonelist to find a contiguous pfn range which can then | |
| 7225 | + * be tried for allocation with alloc_contig_frozen_range(). This routine is | |
| 7226 | + * intended for allocation requests which can not be fulfilled with the buddy | |
| 7227 | + * allocator. | |
| 7173 | 7228 | * |
| 7174 | 7229 | * The allocated memory is always aligned to a page boundary. If nr_pages is a |
| 7175 | 7230 | * power of two, then allocated range is also guaranteed to be aligned to same |
| 7176 | 7231 | * nr_pages (e.g. 1GB request would be aligned to 1GB). |
| 7177 | 7232 | * |
| 7178 | − * Allocated pages can be freed with free_contig_range() or by manually calling | |
| 7179 | − * __free_page() on each allocated page. | |
| 7233 | + * Allocated frozen pages need be freed with free_contig_frozen_range(), | |
| 7234 | + * or by manually calling free_frozen_pages() on each allocated frozen | |
| 7235 | + * non-compound page, for compound frozen pages could be freed with | |
| 7236 | + * free_frozen_pages() directly. | |
| 7180 | 7237 | * |
| 7181 | − * Return: pointer to contiguous pages on success, or NULL if not successful. | |
| 7238 | + * Return: pointer to contiguous frozen pages on success, or NULL if not successful. | |
| 7182 | 7239 | */ |
| 7183 | −struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, | |
| 7184 | − int nid, nodemask_t *nodemask) | |
| 7240 | +struct page *alloc_contig_frozen_pages_noprof(unsigned long nr_pages, | |
| 7241 | + gfp_t gfp_mask, int nid, nodemask_t *nodemask) | |
| 7185 | 7242 | { |
| 7186 | 7243 | unsigned long ret, pfn, flags; |
| 7187 | 7244 | struct zonelist *zonelist; |
| 7188 | 7245 | struct zone *zone; |
| 7189 | 7246 | struct zoneref *z; |
| 7247 | + bool skip_hugetlb = true; | |
| 7248 | + bool skipped_hugetlb = false; | |
| 7190 | 7249 | |
| 7250 | +retry: | |
| 7191 | 7251 | zonelist = node_zonelist(nid, gfp_mask); |
| 7192 | 7252 | for_each_zone_zonelist_nodemask(zone, z, zonelist, |
| 7193 | 7253 | gfp_zone(gfp_mask), nodemask) { |
| @@ -7195,16 +7255,20 @@ struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, · 1 unmodified line | ||
| 7195 | 7255 | |
| 7196 | 7256 | pfn = ALIGN(zone->zone_start_pfn, nr_pages); |
| 7197 | 7257 | while (zone_spans_last_pfn(zone, pfn, nr_pages)) { |
| 7198 | − if (pfn_range_valid_contig(zone, pfn, nr_pages)) { | |
| 7258 | + if (pfn_range_valid_contig(zone, pfn, nr_pages, | |
| 7259 | + skip_hugetlb, | |
| 7260 | + &skipped_hugetlb)) { | |
| 7199 | 7261 | /* |
| 7200 | 7262 | * We release the zone lock here because |
| 7201 | − * alloc_contig_range() will also lock the zone | |
| 7202 | − * at some point. If there's an allocation | |
| 7203 | − * spinning on this lock, it may win the race | |
| 7204 | − * and cause alloc_contig_range() to fail... | |
| 7263 | + * alloc_contig_frozen_range() will also lock | |
| 7264 | + * the zone at some point. If there's an | |
| 7265 | + * allocation spinning on this lock, it may | |
| 7266 | + * win the race and cause allocation to fail. | |
| 7205 | 7267 | */ |
| 7206 | 7268 | spin_unlock_irqrestore(&zone->lock, flags); |
| 7207 | − ret = __alloc_contig_pages(pfn, nr_pages, | |
| 7269 | + ret = alloc_contig_frozen_range_noprof(pfn, | |
| 7270 | + pfn + nr_pages, | |
| 7271 | + ACR_FLAGS_NONE, | |
| 7208 | 7272 | gfp_mask); |
| 7209 | 7273 | if (!ret) |
| 7210 | 7274 | return pfn_to_page(pfn); |
| @@ -7214,35 +7278,96 @@ struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, · 3 unmodified lines | ||
| 7214 | 7278 | } |
| 7215 | 7279 | spin_unlock_irqrestore(&zone->lock, flags); |
| 7216 | 7280 | } |
| 7281 | + /* | |
| 7282 | + * If we failed, retry the search, but treat regions with HugeTLB pages | |
| 7283 | + * as valid targets. This retains fast-allocations on first pass | |
| 7284 | + * without trying to migrate HugeTLB pages (which may fail). On the | |
| 7285 | + * second pass, we will try moving HugeTLB pages when those pages are | |
| 7286 | + * smaller than the requested contiguous region size. | |
| 7287 | + */ | |
| 7288 | + if (skip_hugetlb && skipped_hugetlb) { | |
| 7289 | + skip_hugetlb = false; | |
| 7290 | + goto retry; | |
| 7291 | + } | |
| 7217 | 7292 | return NULL; |
| 7218 | 7293 | } |
| 7219 | −#endif /* CONFIG_CONTIG_ALLOC */ | |
| 7294 | +EXPORT_SYMBOL(alloc_contig_frozen_pages_noprof); | |
| 7220 | 7295 | |
| 7221 | −void free_contig_range(unsigned long pfn, unsigned long nr_pages) | |
| 7296 | +/** | |
| 7297 | + * alloc_contig_pages() -- tries to find and allocate contiguous range of pages | |
| 7298 | + * @nr_pages: Number of contiguous pages to allocate | |
| 7299 | + * @gfp_mask: GFP mask. | |
| 7300 | + * @nid: Target node | |
| 7301 | + * @nodemask: Mask for other possible nodes | |
| 7302 | + * | |
| 7303 | + * This routine is a wrapper around alloc_contig_frozen_pages(), it can't | |
| 7304 | + * be used to allocate compound pages, the refcount of each allocated page | |
| 7305 | + * will be set to one. | |
| 7306 | + * | |
| 7307 | + * Allocated pages can be freed with free_contig_range() or by manually | |
| 7308 | + * calling __free_page() on each allocated page. | |
| 7309 | + * | |
| 7310 | + * Return: pointer to contiguous pages on success, or NULL if not successful. | |
| 7311 | + */ | |
| 7312 | +struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, | |
| 7313 | + int nid, nodemask_t *nodemask) | |
| 7222 | 7314 | { |
| 7223 | − unsigned long count = 0; | |
| 7224 | − struct folio *folio = pfn_folio(pfn); | |
| 7315 | + struct page *page; | |
| 7225 | 7316 | |
| 7226 | − if (folio_test_large(folio)) { | |
| 7227 | − int expected = folio_nr_pages(folio); | |
| 7317 | + if (WARN_ON(gfp_mask & __GFP_COMP)) | |
| 7318 | + return NULL; | |
| 7228 | 7319 | |
| 7229 | − if (nr_pages == expected) | |
| 7230 | − folio_put(folio); | |
| 7231 | − else | |
| 7232 | − WARN(true, "PFN %lu: nr_pages %lu != expected %d\n", | |
| 7233 | − pfn, nr_pages, expected); | |
| 7320 | + page = alloc_contig_frozen_pages_noprof(nr_pages, gfp_mask, nid, | |
| 7321 | + nodemask); | |
| 7322 | + if (page) | |
| 7323 | + set_pages_refcounted(page, nr_pages); | |
| 7324 | + | |
| 7325 | + return page; | |
| 7326 | +} | |
| 7327 | +EXPORT_SYMBOL(alloc_contig_pages_noprof); | |
| 7328 | + | |
| 7329 | +/** | |
| 7330 | + * free_contig_frozen_range() -- free the contiguous range of frozen pages | |
| 7331 | + * @pfn: start PFN to free | |
| 7332 | + * @nr_pages: Number of contiguous frozen pages to free | |
| 7333 | + * | |
| 7334 | + * This can be used to free the allocated compound/non-compound frozen pages. | |
| 7335 | + */ | |
| 7336 | +void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages) | |
| 7337 | +{ | |
| 7338 | + struct page *first_page = pfn_to_page(pfn); | |
| 7339 | + const unsigned int order = ilog2(nr_pages); | |
| 7340 | + | |
| 7341 | + if (WARN_ON_ONCE(first_page != compound_head(first_page))) | |
| 7342 | + return; | |
| 7343 | + | |
| 7344 | + if (PageHead(first_page)) { | |
| 7345 | + WARN_ON_ONCE(order != compound_order(first_page)); | |
| 7346 | + free_frozen_pages(first_page, order); | |
| 7234 | 7347 | return; |
| 7235 | 7348 | } |
| 7236 | 7349 | |
| 7237 | − for (; nr_pages--; pfn++) { | |
| 7238 | − struct page *page = pfn_to_page(pfn); | |
| 7350 | + __free_contig_frozen_range(pfn, nr_pages); | |
| 7351 | +} | |
| 7352 | +EXPORT_SYMBOL(free_contig_frozen_range); | |
| 7239 | 7353 | |
| 7240 | − count += page_count(page) != 1; | |
| 7241 | − __free_page(page); | |
| 7242 | − } | |
| 7243 | − WARN(count != 0, "%lu pages are still in use!\n", count); | |
| 7354 | +/** | |
| 7355 | + * free_contig_range() -- free the contiguous range of pages | |
| 7356 | + * @pfn: start PFN to free | |
| 7357 | + * @nr_pages: Number of contiguous pages to free | |
| 7358 | + * | |
| 7359 | + * This can be only used to free the allocated non-compound pages. | |
| 7360 | + */ | |
| 7361 | +void free_contig_range(unsigned long pfn, unsigned long nr_pages) | |
| 7362 | +{ | |
| 7363 | + if (WARN_ON_ONCE(PageHead(pfn_to_page(pfn)))) | |
| 7364 | + return; | |
| 7365 | + | |
| 7366 | + for (; nr_pages--; pfn++) | |
| 7367 | + __free_page(pfn_to_page(pfn)); | |
| 7244 | 7368 | } |
| 7245 | 7369 | EXPORT_SYMBOL(free_contig_range); |
| 7370 | +#endif /* CONFIG_CONTIG_ALLOC */ | |
| 7246 | 7371 | |
| 7247 | 7372 | /* |
| 7248 | 7373 | * Effectively disable pcplists for the zone by setting the high limit to 0 |
| @@ -7658,7 +7783,7 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t gfp_flags, int nid, unsigned · 409 unmodified lines | ||
| 7658 | 7783 | * unsafe in NMI. If spin_trylock() is called from hard IRQ the current |
| 7659 | 7784 | * task may be waiting for one rt_spin_lock, but rt_spin_trylock() will |
| 7660 | 7785 | * mark the task as the owner of another rt_spin_lock which will |
| 7661 | − * confuse PI logic, so return immediately if called form hard IRQ or | |
| 7786 | + * confuse PI logic, so return immediately if called from hard IRQ or | |
| 7662 | 7787 | * NMI. |
| 7663 | 7788 | * |
| 7664 | 7789 | * Note, irqs_disabled() case is ok. This function can be called |
| @@ -450,14 +450,14 @@ void __swap_writepage(struct folio *folio, struct swap_iocb **swap_plug) · 449 unmodified lines | ||
| 450 | 450 | |
| 451 | 451 | VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio); |
| 452 | 452 | /* |
| 453 | − * ->flags can be updated non-atomicially (scan_swap_map_slots), | |
| 453 | + * ->flags can be updated non-atomically (scan_swap_map_slots), | |
| 454 | 454 | * but that will never affect SWP_FS_OPS, so the data_race |
| 455 | 455 | * is safe. |
| 456 | 456 | */ |
| 457 | 457 | if (data_race(sis->flags & SWP_FS_OPS)) |
| 458 | 458 | swap_writepage_fs(folio, swap_plug); |
| 459 | 459 | /* |
| 460 | − * ->flags can be updated non-atomicially (scan_swap_map_slots), | |
| 460 | + * ->flags can be updated non-atomically (scan_swap_map_slots), | |
| 461 | 461 | * but that will never affect SWP_SYNCHRONOUS_IO, so the data_race |
| 462 | 462 | * is safe. |
| 463 | 463 | */ |
| @@ -15,6 +15,100 @@ · 14 unmodified lines | ||
| 15 | 15 | #define CREATE_TRACE_POINTS |
| 16 | 16 | #include <trace/events/page_isolation.h> |
| 17 | 17 | |
| 18 | +bool page_is_unmovable(struct zone *zone, struct page *page, | |
| 19 | + enum pb_isolate_mode mode, unsigned long *step) | |
| 20 | +{ | |
| 21 | + /* | |
| 22 | + * Both, bootmem allocations and memory holes are marked | |
| 23 | + * PG_reserved and are unmovable. We can even have unmovable | |
| 24 | + * allocations inside ZONE_MOVABLE, for example when | |
| 25 | + * specifying "movablecore". | |
| 26 | + */ | |
| 27 | + if (PageReserved(page)) | |
| 28 | + return true; | |
| 29 | + | |
| 30 | + /* | |
| 31 | + * If the zone is movable and we have ruled out all reserved | |
| 32 | + * pages then it should be reasonably safe to assume the rest | |
| 33 | + * is movable. | |
| 34 | + */ | |
| 35 | + if (zone_idx(zone) == ZONE_MOVABLE) | |
| 36 | + return false; | |
| 37 | + | |
| 38 | + /* | |
| 39 | + * Hugepages are not in LRU lists, but they're movable. | |
| 40 | + * THPs are on the LRU, but need to be counted as #small pages. | |
| 41 | + * We need not scan over tail pages because we don't | |
| 42 | + * handle each tail page individually in migration. | |
| 43 | + */ | |
| 44 | + if (PageHuge(page) || PageCompound(page)) { | |
| 45 | + struct folio *folio = page_folio(page); | |
| 46 | + | |
| 47 | + if (folio_test_hugetlb(folio)) { | |
| 48 | + struct hstate *h; | |
| 49 | + | |
| 50 | + if (!IS_ENABLED(CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION)) | |
| 51 | + return true; | |
| 52 | + | |
| 53 | + /* | |
| 54 | + * The huge page may be freed so can not | |
| 55 | + * use folio_hstate() directly. | |
| 56 | + */ | |
| 57 | + h = size_to_hstate(folio_size(folio)); | |
| 58 | + if (h && !hugepage_migration_supported(h)) | |
| 59 | + return true; | |
| 60 | + | |
| 61 | + } else if (!folio_test_lru(folio)) { | |
| 62 | + return true; | |
| 63 | + } | |
| 64 | + | |
| 65 | + *step = folio_nr_pages(folio) - folio_page_idx(folio, page); | |
| 66 | + return false; | |
| 67 | + } | |
| 68 | + | |
| 69 | + /* | |
| 70 | + * We can't use page_count without pin a page | |
| 71 | + * because another CPU can free compound page. | |
| 72 | + * This check already skips compound tails of THP | |
| 73 | + * because their page->_refcount is zero at all time. | |
| 74 | + */ | |
| 75 | + if (!page_ref_count(page)) { | |
| 76 | + if (PageBuddy(page)) | |
| 77 | + *step = (1 << buddy_order(page)); | |
| 78 | + return false; | |
| 79 | + } | |
| 80 | + | |
| 81 | + /* | |
| 82 | + * The HWPoisoned page may be not in buddy system, and | |
| 83 | + * page_count() is not 0. | |
| 84 | + */ | |
| 85 | + if ((mode == PB_ISOLATE_MODE_MEM_OFFLINE) && PageHWPoison(page)) | |
| 86 | + return false; | |
| 87 | + | |
| 88 | + /* | |
| 89 | + * We treat all PageOffline() pages as movable when offlining | |
| 90 | + * to give drivers a chance to decrement their reference count | |
| 91 | + * in MEM_GOING_OFFLINE in order to indicate that these pages | |
| 92 | + * can be offlined as there are no direct references anymore. | |
| 93 | + * For actually unmovable PageOffline() where the driver does | |
| 94 | + * not support this, we will fail later when trying to actually | |
| 95 | + * move these pages that still have a reference count > 0. | |
| 96 | + * (false negatives in this function only) | |
| 97 | + */ | |
| 98 | + if ((mode == PB_ISOLATE_MODE_MEM_OFFLINE) && PageOffline(page)) | |
| 99 | + return false; | |
| 100 | + | |
| 101 | + if (PageLRU(page) || page_has_movable_ops(page)) | |
| 102 | + return false; | |
| 103 | + | |
| 104 | + /* | |
| 105 | + * If there are RECLAIMABLE pages, we need to check | |
| 106 | + * it. But now, memory offline itself doesn't call | |
| 107 | + * shrink_node_slabs() and it still to be fixed. | |
| 108 | + */ | |
| 109 | + return true; | |
| 110 | +} | |
| 111 | + | |
| 18 | 112 | /* |
| 19 | 113 | * This function checks whether the range [start_pfn, end_pfn) includes |
| 20 | 114 | * unmovable pages or not. The range must fall into a single pageblock and |
| @@ -35,7 +129,6 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long e · 14 unmodified lines | ||
| 35 | 129 | { |
| 36 | 130 | struct page *page = pfn_to_page(start_pfn); |
| 37 | 131 | struct zone *zone = page_zone(page); |
| 38 | − unsigned long pfn; | |
| 39 | 132 | |
| 40 | 133 | VM_BUG_ON(pageblock_start_pfn(start_pfn) != |
| 41 | 134 | pageblock_start_pfn(end_pfn - 1)); |
| @@ -52,96 +145,14 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long e · 10 unmodified lines | ||
| 52 | 145 | return page; |
| 53 | 146 | } |
| 54 | 147 | |
| 55 | − for (pfn = start_pfn; pfn < end_pfn; pfn++) { | |
| 56 | − page = pfn_to_page(pfn); | |
| 148 | + while (start_pfn < end_pfn) { | |
| 149 | + unsigned long step = 1; | |
| 57 | 150 | |
| 58 | − /* | |
| 59 | − * Both, bootmem allocations and memory holes are marked | |
| 60 | − * PG_reserved and are unmovable. We can even have unmovable | |
| 61 | − * allocations inside ZONE_MOVABLE, for example when | |
| 62 | − * specifying "movablecore". | |
| 63 | − */ | |
| 64 | − if (PageReserved(page)) | |
| 151 | + page = pfn_to_page(start_pfn); | |
| 152 | + if (page_is_unmovable(zone, page, mode, &step)) | |
| 65 | 153 | return page; |
| 66 | 154 | |
| 67 | − /* | |
| 68 | − * If the zone is movable and we have ruled out all reserved | |
| 69 | − * pages then it should be reasonably safe to assume the rest | |
| 70 | − * is movable. | |
| 71 | − */ | |
| 72 | − if (zone_idx(zone) == ZONE_MOVABLE) | |
| 73 | − continue; | |
| 74 | − | |
| 75 | − /* | |
| 76 | − * Hugepages are not in LRU lists, but they're movable. | |
| 77 | − * THPs are on the LRU, but need to be counted as #small pages. | |
| 78 | − * We need not scan over tail pages because we don't | |
| 79 | − * handle each tail page individually in migration. | |
| 80 | − */ | |
| 81 | − if (PageHuge(page) || PageTransCompound(page)) { | |
| 82 | − struct folio *folio = page_folio(page); | |
| 83 | − unsigned int skip_pages; | |
| 84 | − | |
| 85 | − if (PageHuge(page)) { | |
| 86 | − struct hstate *h; | |
| 87 | − | |
| 88 | − /* | |
| 89 | − * The huge page may be freed so can not | |
| 90 | − * use folio_hstate() directly. | |
| 91 | − */ | |
| 92 | − h = size_to_hstate(folio_size(folio)); | |
| 93 | − if (h && !hugepage_migration_supported(h)) | |
| 94 | − return page; | |
| 95 | − } else if (!folio_test_lru(folio)) { | |
| 96 | − return page; | |
| 97 | − } | |
| 98 | − | |
| 99 | − skip_pages = folio_nr_pages(folio) - folio_page_idx(folio, page); | |
| 100 | − pfn += skip_pages - 1; | |
| 101 | − continue; | |
| 102 | − } | |
| 103 | − | |
| 104 | − /* | |
| 105 | − * We can't use page_count without pin a page | |
| 106 | − * because another CPU can free compound page. | |
| 107 | − * This check already skips compound tails of THP | |
| 108 | − * because their page->_refcount is zero at all time. | |
| 109 | − */ | |
| 110 | − if (!page_ref_count(page)) { | |
| 111 | − if (PageBuddy(page)) | |
| 112 | − pfn += (1 << buddy_order(page)) - 1; | |
| 113 | − continue; | |
| 114 | − } | |
| 115 | − | |
| 116 | − /* | |
| 117 | − * The HWPoisoned page may be not in buddy system, and | |
| 118 | − * page_count() is not 0. | |
| 119 | − */ | |
| 120 | − if ((mode == PB_ISOLATE_MODE_MEM_OFFLINE) && PageHWPoison(page)) | |
| 121 | − continue; | |
| 122 | − | |
| 123 | − /* | |
| 124 | − * We treat all PageOffline() pages as movable when offlining | |
| 125 | − * to give drivers a chance to decrement their reference count | |
| 126 | − * in MEM_GOING_OFFLINE in order to indicate that these pages | |
| 127 | − * can be offlined as there are no direct references anymore. | |
| 128 | − * For actually unmovable PageOffline() where the driver does | |
| 129 | − * not support this, we will fail later when trying to actually | |
| 130 | − * move these pages that still have a reference count > 0. | |
| 131 | − * (false negatives in this function only) | |
| 132 | − */ | |
| 133 | − if ((mode == PB_ISOLATE_MODE_MEM_OFFLINE) && PageOffline(page)) | |
| 134 | − continue; | |
| 135 | − | |
| 136 | − if (PageLRU(page) || page_has_movable_ops(page)) | |
| 137 | − continue; | |
| 138 | − | |
| 139 | − /* | |
| 140 | − * If there are RECLAIMABLE pages, we need to check | |
| 141 | − * it. But now, memory offline itself doesn't call | |
| 142 | − * shrink_node_slabs() and it still to be fixed. | |
| 143 | − */ | |
| 144 | − return page; | |
| 155 | + start_pfn += step; | |
| 145 | 156 | } |
| 146 | 157 | return NULL; |
| 147 | 158 | } |
| @@ -301,7 +312,7 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) · 153 unmodified lines | ||
| 301 | 312 | * pageblock. When not all pageblocks within a page are isolated at the same |
| 302 | 313 | * time, free page accounting can go wrong. For example, in the case of |
| 303 | 314 | * MAX_PAGE_ORDER = pageblock_order + 1, a MAX_PAGE_ORDER page has two |
| 304 | − * pagelbocks. | |
| 315 | + * pageblocks. | |
| 305 | 316 | * [ MAX_PAGE_ORDER ] |
| 306 | 317 | * [ pageblock0 | pageblock1 ] |
| 307 | 318 | * When either pageblock is isolated, if it is a free page, the page is not |
| @@ -123,7 +123,7 @@ page_reporting_drain(struct page_reporting_dev_info *prdev, · 122 unmodified lines | ||
| 123 | 123 | continue; |
| 124 | 124 | |
| 125 | 125 | /* |
| 126 | − * If page was not comingled with another page we can | |
| 126 | + * If page was not commingled with another page we can | |
| 127 | 127 | * consider the result to be "reported" since the page |
| 128 | 128 | * hasn't been modified, otherwise we will need to |
| 129 | 129 | * report on the new larger page when we make our way |
| @@ -145,34 +145,37 @@ void __page_table_check_zero(struct page *page, unsigned int order) · 144 unmodified lines | ||
| 145 | 145 | rcu_read_unlock(); |
| 146 | 146 | } |
| 147 | 147 | |
| 148 | −void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte) | |
| 148 | +void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr, | |
| 149 | + pte_t pte) | |
| 149 | 150 | { |
| 150 | 151 | if (&init_mm == mm) |
| 151 | 152 | return; |
| 152 | 153 | |
| 153 | − if (pte_user_accessible_page(pte)) { | |
| 154 | + if (pte_user_accessible_page(pte, addr)) { | |
| 154 | 155 | page_table_check_clear(pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT); |
| 155 | 156 | } |
| 156 | 157 | } |
| 157 | 158 | EXPORT_SYMBOL(__page_table_check_pte_clear); |
| 158 | 159 | |
| 159 | −void __page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd) | |
| 160 | +void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr, | |
| 161 | + pmd_t pmd) | |
| 160 | 162 | { |
| 161 | 163 | if (&init_mm == mm) |
| 162 | 164 | return; |
| 163 | 165 | |
| 164 | − if (pmd_user_accessible_page(pmd)) { | |
| 166 | + if (pmd_user_accessible_page(pmd, addr)) { | |
| 165 | 167 | page_table_check_clear(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT); |
| 166 | 168 | } |
| 167 | 169 | } |
| 168 | 170 | EXPORT_SYMBOL(__page_table_check_pmd_clear); |
| 169 | 171 | |
| 170 | −void __page_table_check_pud_clear(struct mm_struct *mm, pud_t pud) | |
| 172 | +void __page_table_check_pud_clear(struct mm_struct *mm, unsigned long addr, | |
| 173 | + pud_t pud) | |
| 171 | 174 | { |
| 172 | 175 | if (&init_mm == mm) |
| 173 | 176 | return; |
| 174 | 177 | |
| 175 | − if (pud_user_accessible_page(pud)) { | |
| 178 | + if (pud_user_accessible_page(pud, addr)) { | |
| 176 | 179 | page_table_check_clear(pud_pfn(pud), PUD_SIZE >> PAGE_SHIFT); |
| 177 | 180 | } |
| 178 | 181 | } |
| @@ -196,8 +199,8 @@ static void page_table_check_pte_flags(pte_t pte) · 17 unmodified lines | ||
| 196 | 199 | } |
| 197 | 200 | } |
| 198 | 201 | |
| 199 | −void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte, | |
| 200 | − unsigned int nr) | |
| 202 | +void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr, | |
| 203 | + pte_t *ptep, pte_t pte, unsigned int nr) | |
| 201 | 204 | { |
| 202 | 205 | unsigned int i; |
| 203 | 206 | |
| @@ -207,8 +210,8 @@ void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte, · 3 unmodified lines | ||
| 207 | 210 | page_table_check_pte_flags(pte); |
| 208 | 211 | |
| 209 | 212 | for (i = 0; i < nr; i++) |
| 210 | − __page_table_check_pte_clear(mm, ptep_get(ptep + i)); | |
| 211 | − if (pte_user_accessible_page(pte)) | |
| 213 | + __page_table_check_pte_clear(mm, addr + PAGE_SIZE * i, ptep_get(ptep + i)); | |
| 214 | + if (pte_user_accessible_page(pte, addr)) | |
| 212 | 215 | page_table_check_set(pte_pfn(pte), nr, pte_write(pte)); |
| 213 | 216 | } |
| 214 | 217 | EXPORT_SYMBOL(__page_table_check_ptes_set); |
| @@ -225,8 +228,8 @@ static inline void page_table_check_pmd_flags(pmd_t pmd) · 10 unmodified lines | ||
| 225 | 228 | } |
| 226 | 229 | } |
| 227 | 230 | |
| 228 | −void __page_table_check_pmds_set(struct mm_struct *mm, pmd_t *pmdp, pmd_t pmd, | |
| 229 | − unsigned int nr) | |
| 231 | +void __page_table_check_pmds_set(struct mm_struct *mm, unsigned long addr, | |
| 232 | + pmd_t *pmdp, pmd_t pmd, unsigned int nr) | |
| 230 | 233 | { |
| 231 | 234 | unsigned long stride = PMD_SIZE >> PAGE_SHIFT; |
| 232 | 235 | unsigned int i; |
| @@ -237,14 +240,14 @@ void __page_table_check_pmds_set(struct mm_struct *mm, pmd_t *pmdp, pmd_t pmd, · 4 unmodified lines | ||
| 237 | 240 | page_table_check_pmd_flags(pmd); |
| 238 | 241 | |
| 239 | 242 | for (i = 0; i < nr; i++) |
| 240 | − __page_table_check_pmd_clear(mm, *(pmdp + i)); | |
| 241 | − if (pmd_user_accessible_page(pmd)) | |
| 243 | + __page_table_check_pmd_clear(mm, addr + PMD_SIZE * i, *(pmdp + i)); | |
| 244 | + if (pmd_user_accessible_page(pmd, addr)) | |
| 242 | 245 | page_table_check_set(pmd_pfn(pmd), stride * nr, pmd_write(pmd)); |
| 243 | 246 | } |
| 244 | 247 | EXPORT_SYMBOL(__page_table_check_pmds_set); |
| 245 | 248 | |
| 246 | −void __page_table_check_puds_set(struct mm_struct *mm, pud_t *pudp, pud_t pud, | |
| 247 | − unsigned int nr) | |
| 249 | +void __page_table_check_puds_set(struct mm_struct *mm, unsigned long addr, | |
| 250 | + pud_t *pudp, pud_t pud, unsigned int nr) | |
| 248 | 251 | { |
| 249 | 252 | unsigned long stride = PUD_SIZE >> PAGE_SHIFT; |
| 250 | 253 | unsigned int i; |
| @@ -253,8 +256,8 @@ void __page_table_check_puds_set(struct mm_struct *mm, pud_t *pudp, pud_t pud, · 2 unmodified lines | ||
| 253 | 256 | return; |
| 254 | 257 | |
| 255 | 258 | for (i = 0; i < nr; i++) |
| 256 | − __page_table_check_pud_clear(mm, *(pudp + i)); | |
| 257 | − if (pud_user_accessible_page(pud)) | |
| 259 | + __page_table_check_pud_clear(mm, addr + PUD_SIZE * i, *(pudp + i)); | |
| 260 | + if (pud_user_accessible_page(pud, addr)) | |
| 258 | 261 | page_table_check_set(pud_pfn(pud), stride * nr, pud_write(pud)); |
| 259 | 262 | } |
| 260 | 263 | EXPORT_SYMBOL(__page_table_check_puds_set); |
| @@ -273,7 +276,7 @@ void __page_table_check_pte_clear_range(struct mm_struct *mm, · 12 unmodified lines | ||
| 273 | 276 | if (WARN_ON(!ptep)) |
| 274 | 277 | return; |
| 275 | 278 | for (i = 0; i < PTRS_PER_PTE; i++) { |
| 276 | − __page_table_check_pte_clear(mm, ptep_get(ptep)); | |
| 279 | + __page_table_check_pte_clear(mm, addr, ptep_get(ptep)); | |
| 277 | 280 | addr += PAGE_SIZE; |
| 278 | 281 | ptep++; |
| 279 | 282 | } |
| @@ -313,7 +313,8 @@ static unsigned long hugetlb_entry_end(struct hstate *h, unsigned long addr, · 312 unmodified lines | ||
| 313 | 313 | unsigned long end) |
| 314 | 314 | { |
| 315 | 315 | unsigned long boundary = (addr & huge_page_mask(h)) + huge_page_size(h); |
| 316 | − return boundary < end ? boundary : end; | |
| 316 | + | |
| 317 | + return min(boundary, end); | |
| 317 | 318 | } |
| 318 | 319 | |
| 319 | 320 | static int walk_hugetlb_range(unsigned long addr, unsigned long end, |
| @@ -1279,12 +1279,16 @@ static int pcpu_free_area(struct pcpu_chunk *chunk, int off) · 1278 unmodified lines | ||
| 1279 | 1279 | int bit_off, bits, end, oslot, freed; |
| 1280 | 1280 | |
| 1281 | 1281 | lockdep_assert_held(&pcpu_lock); |
| 1282 | − pcpu_stats_area_dealloc(chunk); | |
| 1283 | 1282 | |
| 1284 | 1283 | oslot = pcpu_chunk_slot(chunk); |
| 1285 | 1284 | |
| 1286 | 1285 | bit_off = off / PCPU_MIN_ALLOC_SIZE; |
| 1287 | 1286 | |
| 1287 | + /* check invalid free */ | |
| 1288 | + if (!test_bit(bit_off, chunk->alloc_map) || | |
| 1289 | + !test_bit(bit_off, chunk->bound_map)) | |
| 1290 | + return 0; | |
| 1291 | + | |
| 1288 | 1292 | /* find end index */ |
| 1289 | 1293 | end = find_next_bit(chunk->bound_map, pcpu_chunk_map_bits(chunk), |
| 1290 | 1294 | bit_off + 1); |
| @@ -1303,6 +1307,8 @@ static int pcpu_free_area(struct pcpu_chunk *chunk, int off) · 12 unmodified lines | ||
| 1303 | 1307 | |
| 1304 | 1308 | pcpu_chunk_relocate(chunk, oslot); |
| 1305 | 1309 | |
| 1310 | + pcpu_stats_area_dealloc(chunk); | |
| 1311 | + | |
| 1306 | 1312 | return freed; |
| 1307 | 1313 | } |
| 1308 | 1314 | |
| @@ -2242,6 +2248,13 @@ void free_percpu(void __percpu *ptr) · 933 unmodified lines | ||
| 2242 | 2248 | |
| 2243 | 2249 | spin_lock_irqsave(&pcpu_lock, flags); |
| 2244 | 2250 | size = pcpu_free_area(chunk, off); |
| 2251 | + if (size == 0) { | |
| 2252 | + spin_unlock_irqrestore(&pcpu_lock, flags); | |
| 2253 | + | |
| 2254 | + /* invalid percpu free */ | |
| 2255 | + WARN_ON_ONCE(1); | |
| 2256 | + return; | |
| 2257 | + } | |
| 2245 | 2258 | |
| 2246 | 2259 | pcpu_alloc_tag_free_hook(chunk, off, size); |
| 2247 | 2260 | |
| @@ -1,72 +0,0 @@ | ||
| 1 | −// SPDX-License-Identifier: GPL-2.0 | |
| 2 | −#include <linux/hugetlb.h> | |
| 3 | −#include <linux/pgalloc.h> | |
| 4 | − | |
| 5 | −#include <asm-generic/tlb.h> | |
| 6 | − | |
| 7 | −#include "internal.h" | |
| 8 | − | |
| 9 | −bool reclaim_pt_is_enabled(unsigned long start, unsigned long end, | |
| 10 | − struct zap_details *details) | |
| 11 | −{ | |
| 12 | − return details && details->reclaim_pt && (end - start >= PMD_SIZE); | |
| 13 | −} | |
| 14 | − | |
| 15 | −bool try_get_and_clear_pmd(struct mm_struct *mm, pmd_t *pmd, pmd_t *pmdval) | |
| 16 | −{ | |
| 17 | − spinlock_t *pml = pmd_lockptr(mm, pmd); | |
| 18 | − | |
| 19 | − if (!spin_trylock(pml)) | |
| 20 | − return false; | |
| 21 | − | |
| 22 | − *pmdval = pmdp_get_lockless(pmd); | |
| 23 | − pmd_clear(pmd); | |
| 24 | − spin_unlock(pml); | |
| 25 | − | |
| 26 | − return true; | |
| 27 | −} | |
| 28 | − | |
| 29 | −void free_pte(struct mm_struct *mm, unsigned long addr, struct mmu_gather *tlb, | |
| 30 | − pmd_t pmdval) | |
| 31 | −{ | |
| 32 | − pte_free_tlb(tlb, pmd_pgtable(pmdval), addr); | |
| 33 | − mm_dec_nr_ptes(mm); | |
| 34 | −} | |
| 35 | − | |
| 36 | −void try_to_free_pte(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, | |
| 37 | − struct mmu_gather *tlb) | |
| 38 | −{ | |
| 39 | − pmd_t pmdval; | |
| 40 | − spinlock_t *pml, *ptl = NULL; | |
| 41 | − pte_t *start_pte, *pte; | |
| 42 | − int i; | |
| 43 | − | |
| 44 | − pml = pmd_lock(mm, pmd); | |
| 45 | − start_pte = pte_offset_map_rw_nolock(mm, pmd, addr, &pmdval, &ptl); | |
| 46 | − if (!start_pte) | |
| 47 | − goto out_ptl; | |
| 48 | − if (ptl != pml) | |
| 49 | − spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); | |
| 50 | − | |
| 51 | − /* Check if it is empty PTE page */ | |
| 52 | − for (i = 0, pte = start_pte; i < PTRS_PER_PTE; i++, pte++) { | |
| 53 | − if (!pte_none(ptep_get(pte))) | |
| 54 | − goto out_ptl; | |
| 55 | − } | |
| 56 | − pte_unmap(start_pte); | |
| 57 | − | |
| 58 | − pmd_clear(pmd); | |
| 59 | − | |
| 60 | − if (ptl != pml) | |
| 61 | − spin_unlock(ptl); | |
| 62 | − spin_unlock(pml); | |
| 63 | − | |
| 64 | − free_pte(mm, addr, tlb, pmdval); | |
| 65 | − | |
| 66 | − return; | |
| 67 | −out_ptl: | |
| 68 | − if (start_pte) | |
| 69 | − pte_unmap_unlock(start_pte, ptl); | |
| 70 | − if (ptl != pml) | |
| 71 | − spin_unlock(pml); | |
| 72 | −} | |
| @@ -439,7 +439,7 @@ static unsigned long get_next_ra_size(struct file_ra_state *ra, · 438 unmodified lines | ||
| 439 | 439 | * based on I/O request size and the max_readahead. |
| 440 | 440 | * |
| 441 | 441 | * The code ramps up the readahead size aggressively at first, but slow down as |
| 442 | − * it approaches max_readhead. | |
| 442 | + * it approaches max_readahead. | |
| 443 | 443 | */ |
| 444 | 444 | |
| 445 | 445 | static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index, |
| @@ -1,8 +1,8 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 1 | 2 | /* |
| 2 | 3 | * mm/rmap.c - physical to virtual reverse mappings |
| 3 | 4 | * |
| 4 | 5 | * Copyright 2001, Rik van Riel <riel@conectiva.com.br> |
| 5 | − * Released under the General Public License (GPL). | |
| 6 | 6 | * |
| 7 | 7 | * Simple, low overhead reverse mapping scheme. |
| 8 | 8 | * Please try to keep this thing as modular as possible. |
| @@ -82,6 +82,7 @@ · 73 unmodified lines | ||
| 82 | 82 | #include <trace/events/migrate.h> |
| 83 | 83 | |
| 84 | 84 | #include "internal.h" |
| 85 | +#include "swap.h" | |
| 85 | 86 | |
| 86 | 87 | static struct kmem_cache *anon_vma_cachep; |
| 87 | 88 | static struct kmem_cache *anon_vma_chain_cachep; |
| @@ -146,14 +147,13 @@ static void anon_vma_chain_free(struct anon_vma_chain *anon_vma_chain) · 58 unmodified lines | ||
| 146 | 147 | kmem_cache_free(anon_vma_chain_cachep, anon_vma_chain); |
| 147 | 148 | } |
| 148 | 149 | |
| 149 | −static void anon_vma_chain_link(struct vm_area_struct *vma, | |
| 150 | − struct anon_vma_chain *avc, | |
| 151 | − struct anon_vma *anon_vma) | |
| 150 | +static void anon_vma_chain_assign(struct vm_area_struct *vma, | |
| 151 | + struct anon_vma_chain *avc, | |
| 152 | + struct anon_vma *anon_vma) | |
| 152 | 153 | { |
| 153 | 154 | avc->vma = vma; |
| 154 | 155 | avc->anon_vma = anon_vma; |
| 155 | 156 | list_add(&avc->same_vma, &vma->anon_vma_chain); |
| 156 | − anon_vma_interval_tree_insert(avc, &anon_vma->rb_root); | |
| 157 | 157 | } |
| 158 | 158 | |
| 159 | 159 | /** |
| @@ -210,7 +210,8 @@ int __anon_vma_prepare(struct vm_area_struct *vma) · 50 unmodified lines | ||
| 210 | 210 | spin_lock(&mm->page_table_lock); |
| 211 | 211 | if (likely(!vma->anon_vma)) { |
| 212 | 212 | vma->anon_vma = anon_vma; |
| 213 | − anon_vma_chain_link(vma, avc, anon_vma); | |
| 213 | + anon_vma_chain_assign(vma, avc, anon_vma); | |
| 214 | + anon_vma_interval_tree_insert(avc, &anon_vma->rb_root); | |
| 214 | 215 | anon_vma->num_active_vmas++; |
| 215 | 216 | allocated = NULL; |
| 216 | 217 | avc = NULL; |
| @@ -231,97 +232,141 @@ int __anon_vma_prepare(struct vm_area_struct *vma) · 14 unmodified lines | ||
| 231 | 232 | return -ENOMEM; |
| 232 | 233 | } |
| 233 | 234 | |
| 234 | −/* | |
| 235 | − * This is a useful helper function for locking the anon_vma root as | |
| 236 | − * we traverse the vma->anon_vma_chain, looping over anon_vma's that | |
| 237 | − * have the same vma. | |
| 238 | − * | |
| 239 | − * Such anon_vma's should have the same root, so you'd expect to see | |
| 240 | − * just a single mutex_lock for the whole traversal. | |
| 241 | − */ | |
| 242 | −static inline struct anon_vma *lock_anon_vma_root(struct anon_vma *root, struct anon_vma *anon_vma) | |
| 243 | −{ | |
| 244 | − struct anon_vma *new_root = anon_vma->root; | |
| 245 | − if (new_root != root) { | |
| 246 | − if (WARN_ON_ONCE(root)) | |
| 247 | − up_write(&root->rwsem); | |
| 248 | − root = new_root; | |
| 249 | − down_write(&root->rwsem); | |
| 250 | − } | |
| 251 | − return root; | |
| 235 | +static void check_anon_vma_clone(struct vm_area_struct *dst, | |
| 236 | + struct vm_area_struct *src, | |
| 237 | + enum vma_operation operation) | |
| 238 | +{ | |
| 239 | + /* The write lock must be held. */ | |
| 240 | + mmap_assert_write_locked(src->vm_mm); | |
| 241 | + /* If not a fork then must be on same mm. */ | |
| 242 | + VM_WARN_ON_ONCE(operation != VMA_OP_FORK && dst->vm_mm != src->vm_mm); | |
| 243 | + | |
| 244 | + /* If we have anything to do src->anon_vma must be provided. */ | |
| 245 | + VM_WARN_ON_ONCE(!src->anon_vma && !list_empty(&src->anon_vma_chain)); | |
| 246 | + VM_WARN_ON_ONCE(!src->anon_vma && dst->anon_vma); | |
| 247 | + /* We are establishing a new anon_vma_chain. */ | |
| 248 | + VM_WARN_ON_ONCE(!list_empty(&dst->anon_vma_chain)); | |
| 249 | + /* | |
| 250 | + * On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma | |
| 251 | + * must be the same across dst and src. | |
| 252 | + */ | |
| 253 | + VM_WARN_ON_ONCE(dst->anon_vma && dst->anon_vma != src->anon_vma); | |
| 254 | + /* | |
| 255 | + * Essentially equivalent to above - if not a no-op, we should expect | |
| 256 | + * dst->anon_vma to be set for everything except a fork. | |
| 257 | + */ | |
| 258 | + VM_WARN_ON_ONCE(operation != VMA_OP_FORK && src->anon_vma && | |
| 259 | + !dst->anon_vma); | |
| 260 | + /* For the anon_vma to be compatible, it can only be singular. */ | |
| 261 | + VM_WARN_ON_ONCE(operation == VMA_OP_MERGE_UNFAULTED && | |
| 262 | + !list_is_singular(&src->anon_vma_chain)); | |
| 263 | +#ifdef CONFIG_PER_VMA_LOCK | |
| 264 | + /* Only merging an unfaulted VMA leaves the destination attached. */ | |
| 265 | + VM_WARN_ON_ONCE(operation != VMA_OP_MERGE_UNFAULTED && | |
| 266 | + vma_is_attached(dst)); | |
| 267 | +#endif | |
| 252 | 268 | } |
| 253 | 269 | |
| 254 | −static inline void unlock_anon_vma_root(struct anon_vma *root) | |
| 270 | +static void maybe_reuse_anon_vma(struct vm_area_struct *dst, | |
| 271 | + struct anon_vma *anon_vma) | |
| 255 | 272 | { |
| 256 | − if (root) | |
| 257 | − up_write(&root->rwsem); | |
| 273 | + /* If already populated, nothing to do.*/ | |
| 274 | + if (dst->anon_vma) | |
| 275 | + return; | |
| 276 | + | |
| 277 | + /* | |
| 278 | + * We reuse an anon_vma if any linking VMAs were unmapped and it has | |
| 279 | + * only a single child at most. | |
| 280 | + */ | |
| 281 | + if (anon_vma->num_active_vmas > 0) | |
| 282 | + return; | |
| 283 | + if (anon_vma->num_children > 1) | |
| 284 | + return; | |
| 285 | + | |
| 286 | + dst->anon_vma = anon_vma; | |
| 287 | + anon_vma->num_active_vmas++; | |
| 258 | 288 | } |
| 259 | 289 | |
| 260 | −/* | |
| 261 | − * Attach the anon_vmas from src to dst. | |
| 262 | − * Returns 0 on success, -ENOMEM on failure. | |
| 263 | − * | |
| 264 | − * anon_vma_clone() is called by vma_expand(), vma_merge(), __split_vma(), | |
| 265 | − * copy_vma() and anon_vma_fork(). The first four want an exact copy of src, | |
| 266 | − * while the last one, anon_vma_fork(), may try to reuse an existing anon_vma to | |
| 267 | − * prevent endless growth of anon_vma. Since dst->anon_vma is set to NULL before | |
| 268 | − * call, we can identify this case by checking (!dst->anon_vma && | |
| 269 | − * src->anon_vma). | |
| 270 | − * | |
| 271 | − * If (!dst->anon_vma && src->anon_vma) is true, this function tries to find | |
| 272 | − * and reuse existing anon_vma which has no vmas and only one child anon_vma. | |
| 273 | − * This prevents degradation of anon_vma hierarchy to endless linear chain in | |
| 274 | − * case of constantly forking task. On the other hand, an anon_vma with more | |
| 275 | − * than one child isn't reused even if there was no alive vma, thus rmap | |
| 276 | − * walker has a good chance of avoiding scanning the whole hierarchy when it | |
| 277 | − * searches where page is mapped. | |
| 290 | +static void cleanup_partial_anon_vmas(struct vm_area_struct *vma); | |
| 291 | + | |
| 292 | +/** | |
| 293 | + * anon_vma_clone - Establishes new anon_vma_chain objects in @dst linking to | |
| 294 | + * all of the anon_vma objects contained within @src anon_vma_chain's. | |
| 295 | + * @dst: The destination VMA with an empty anon_vma_chain. | |
| 296 | + * @src: The source VMA we wish to duplicate. | |
| 297 | + * @operation: The type of operation which resulted in the clone. | |
| 298 | + * | |
| 299 | + * This is the heart of the VMA side of the anon_vma implementation - we invoke | |
| 300 | + * this function whenever we need to set up a new VMA's anon_vma state. | |
| 301 | + * | |
| 302 | + * This is invoked for: | |
| 303 | + * | |
| 304 | + * - VMA Merge, but only when @dst is unfaulted and @src is faulted - meaning we | |
| 305 | + * clone @src into @dst. | |
| 306 | + * - VMA split. | |
| 307 | + * - VMA (m)remap. | |
| 308 | + * - Fork of faulted VMA. | |
| 309 | + * | |
| 310 | + * In all cases other than fork this is simply a duplication. Fork additionally | |
| 311 | + * adds a new active anon_vma. | |
| 312 | + * | |
| 313 | + * ONLY in the case of fork do we try to 'reuse' existing anon_vma's in an | |
| 314 | + * anon_vma hierarchy, reusing anon_vma's which have no VMA associated with them | |
| 315 | + * but do have a single child. This is to avoid waste of memory when repeatedly | |
| 316 | + * forking. | |
| 317 | + * | |
| 318 | + * Returns: 0 on success, -ENOMEM on failure. | |
| 278 | 319 | */ |
| 279 | −int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) | |
| 320 | +int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src, | |
| 321 | + enum vma_operation operation) | |
| 280 | 322 | { |
| 281 | 323 | struct anon_vma_chain *avc, *pavc; |
| 282 | − struct anon_vma *root = NULL; | |
| 283 | − | |
| 284 | − list_for_each_entry_reverse(pavc, &src->anon_vma_chain, same_vma) { | |
| 285 | − struct anon_vma *anon_vma; | |
| 286 | − | |
| 287 | − avc = anon_vma_chain_alloc(GFP_NOWAIT); | |
| 288 | − if (unlikely(!avc)) { | |
| 289 | − unlock_anon_vma_root(root); | |
| 290 | − root = NULL; | |
| 291 | − avc = anon_vma_chain_alloc(GFP_KERNEL); | |
| 292 | − if (!avc) | |
| 293 | − goto enomem_failure; | |
| 294 | − } | |
| 295 | − anon_vma = pavc->anon_vma; | |
| 296 | − root = lock_anon_vma_root(root, anon_vma); | |
| 297 | − anon_vma_chain_link(dst, avc, anon_vma); | |
| 324 | + struct anon_vma *active_anon_vma = src->anon_vma; | |
| 298 | 325 | |
| 299 | − /* | |
| 300 | − * Reuse existing anon_vma if it has no vma and only one | |
| 301 | − * anon_vma child. | |
| 302 | − * | |
| 303 | − * Root anon_vma is never reused: | |
| 304 | − * it has self-parent reference and at least one child. | |
| 305 | − */ | |
| 306 | − if (!dst->anon_vma && src->anon_vma && | |
| 307 | − anon_vma->num_children < 2 && | |
| 308 | − anon_vma->num_active_vmas == 0) | |
| 309 | − dst->anon_vma = anon_vma; | |
| 326 | + check_anon_vma_clone(dst, src, operation); | |
| 327 | + | |
| 328 | + if (!active_anon_vma) | |
| 329 | + return 0; | |
| 330 | + | |
| 331 | + /* | |
| 332 | + * Allocate AVCs. We don't need an anon_vma lock for this as we | |
| 333 | + * are not updating the anon_vma rbtree nor are we changing | |
| 334 | + * anon_vma statistics. | |
| 335 | + * | |
| 336 | + * Either src, dst have the same mm for which we hold an exclusive mmap | |
| 337 | + * write lock, or we are forking and we hold it on src->vm_mm and dst is | |
| 338 | + * not yet accessible to other threads so there's no possibliity of the | |
| 339 | + * unlinked AVC's being observed yet. | |
| 340 | + */ | |
| 341 | + list_for_each_entry(pavc, &src->anon_vma_chain, same_vma) { | |
| 342 | + avc = anon_vma_chain_alloc(GFP_KERNEL); | |
| 343 | + if (!avc) | |
| 344 | + goto enomem_failure; | |
| 345 | + | |
| 346 | + anon_vma_chain_assign(dst, avc, pavc->anon_vma); | |
| 310 | 347 | } |
| 311 | − if (dst->anon_vma) | |
| 348 | + | |
| 349 | + /* | |
| 350 | + * Now link the anon_vma's back to the newly inserted AVCs. | |
| 351 | + * Note that all anon_vma's share the same root. | |
| 352 | + */ | |
| 353 | + anon_vma_lock_write(src->anon_vma); | |
| 354 | + list_for_each_entry_reverse(avc, &dst->anon_vma_chain, same_vma) { | |
| 355 | + struct anon_vma *anon_vma = avc->anon_vma; | |
| 356 | + | |
| 357 | + anon_vma_interval_tree_insert(avc, &anon_vma->rb_root); | |
| 358 | + if (operation == VMA_OP_FORK) | |
| 359 | + maybe_reuse_anon_vma(dst, anon_vma); | |
| 360 | + } | |
| 361 | + | |
| 362 | + if (operation != VMA_OP_FORK) | |
| 312 | 363 | dst->anon_vma->num_active_vmas++; |
| 313 | − unlock_anon_vma_root(root); | |
| 364 | + | |
| 365 | + anon_vma_unlock_write(active_anon_vma); | |
| 314 | 366 | return 0; |
| 315 | 367 | |
| 316 | 368 | enomem_failure: |
| 317 | − /* | |
| 318 | − * dst->anon_vma is dropped here otherwise its num_active_vmas can | |
| 319 | − * be incorrectly decremented in unlink_anon_vmas(). | |
| 320 | − * We can safely do this because callers of anon_vma_clone() don't care | |
| 321 | − * about dst->anon_vma if anon_vma_clone() failed. | |
| 322 | − */ | |
| 323 | − dst->anon_vma = NULL; | |
| 324 | − unlink_anon_vmas(dst); | |
| 369 | + cleanup_partial_anon_vmas(dst); | |
| 325 | 370 | return -ENOMEM; |
| 326 | 371 | } |
| 327 | 372 | |
| @@ -334,7 +379,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · 6 unmodified lines | ||
| 334 | 379 | { |
| 335 | 380 | struct anon_vma_chain *avc; |
| 336 | 381 | struct anon_vma *anon_vma; |
| 337 | − int error; | |
| 382 | + int rc; | |
| 338 | 383 | |
| 339 | 384 | /* Don't bother if the parent process has no anon_vma here. */ |
| 340 | 385 | if (!pvma->anon_vma) |
| @@ -343,27 +388,35 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · 2 unmodified lines | ||
| 343 | 388 | /* Drop inherited anon_vma, we'll reuse existing or allocate new. */ |
| 344 | 389 | vma->anon_vma = NULL; |
| 345 | 390 | |
| 391 | + anon_vma = anon_vma_alloc(); | |
| 392 | + if (!anon_vma) | |
| 393 | + return -ENOMEM; | |
| 394 | + avc = anon_vma_chain_alloc(GFP_KERNEL); | |
| 395 | + if (!avc) { | |
| 396 | + put_anon_vma(anon_vma); | |
| 397 | + return -ENOMEM; | |
| 398 | + } | |
| 399 | + | |
| 346 | 400 | /* |
| 347 | 401 | * First, attach the new VMA to the parent VMA's anon_vmas, |
| 348 | 402 | * so rmap can find non-COWed pages in child processes. |
| 349 | 403 | */ |
| 350 | − error = anon_vma_clone(vma, pvma); | |
| 351 | − if (error) | |
| 352 | − return error; | |
| 353 | − | |
| 354 | − /* An existing anon_vma has been reused, all done then. */ | |
| 355 | − if (vma->anon_vma) | |
| 356 | − return 0; | |
| 404 | + rc = anon_vma_clone(vma, pvma, VMA_OP_FORK); | |
| 405 | + /* An error arose or an existing anon_vma was reused, all done then. */ | |
| 406 | + if (rc || vma->anon_vma) { | |
| 407 | + put_anon_vma(anon_vma); | |
| 408 | + anon_vma_chain_free(avc); | |
| 409 | + return rc; | |
| 410 | + } | |
| 357 | 411 | |
| 358 | − /* Then add our own anon_vma. */ | |
| 359 | − anon_vma = anon_vma_alloc(); | |
| 360 | − if (!anon_vma) | |
| 361 | − goto out_error; | |
| 362 | − anon_vma->num_active_vmas++; | |
| 363 | − avc = anon_vma_chain_alloc(GFP_KERNEL); | |
| 364 | − if (!avc) | |
| 365 | − goto out_error_free_anon_vma; | |
| 412 | + /* | |
| 413 | + * OK no reuse, so add our own anon_vma. | |
| 414 | + * | |
| 415 | + * Since it is not linked anywhere we can safely manipulate anon_vma | |
| 416 | + * fields without a lock. | |
| 417 | + */ | |
| 366 | 418 | |
| 419 | + anon_vma->num_active_vmas = 1; | |
| 367 | 420 | /* |
| 368 | 421 | * The root anon_vma's rwsem is the lock actually used when we |
| 369 | 422 | * lock any of the anon_vmas in this anon_vma tree. |
| @@ -378,24 +431,59 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · 8 unmodified lines | ||
| 378 | 431 | get_anon_vma(anon_vma->root); |
| 379 | 432 | /* Mark this anon_vma as the one where our new (COWed) pages go. */ |
| 380 | 433 | vma->anon_vma = anon_vma; |
| 434 | + anon_vma_chain_assign(vma, avc, anon_vma); | |
| 435 | + /* Now let rmap see it. */ | |
| 381 | 436 | anon_vma_lock_write(anon_vma); |
| 382 | − anon_vma_chain_link(vma, avc, anon_vma); | |
| 437 | + anon_vma_interval_tree_insert(avc, &anon_vma->rb_root); | |
| 383 | 438 | anon_vma->parent->num_children++; |
| 384 | 439 | anon_vma_unlock_write(anon_vma); |
| 385 | 440 | |
| 386 | 441 | return 0; |
| 442 | +} | |
| 387 | 443 | |
| 388 | − out_error_free_anon_vma: | |
| 389 | − put_anon_vma(anon_vma); | |
| 390 | − out_error: | |
| 391 | − unlink_anon_vmas(vma); | |
| 392 | − return -ENOMEM; | |
| 444 | +/* | |
| 445 | + * In the unfortunate case of anon_vma_clone() failing to allocate memory we | |
| 446 | + * have to clean things up. | |
| 447 | + * | |
| 448 | + * Since we allocate anon_vma_chain's before we insert them into the interval | |
| 449 | + * trees, we simply have to free up the AVC's and remove the entries from the | |
| 450 | + * VMA's anon_vma_chain. | |
| 451 | + */ | |
| 452 | +static void cleanup_partial_anon_vmas(struct vm_area_struct *vma) | |
| 453 | +{ | |
| 454 | + struct anon_vma_chain *avc, *next; | |
| 455 | + | |
| 456 | + list_for_each_entry_safe(avc, next, &vma->anon_vma_chain, same_vma) { | |
| 457 | + list_del(&avc->same_vma); | |
| 458 | + anon_vma_chain_free(avc); | |
| 459 | + } | |
| 393 | 460 | } |
| 394 | 461 | |
| 462 | +/** | |
| 463 | + * unlink_anon_vmas() - remove all links between a VMA and anon_vma's, freeing | |
| 464 | + * anon_vma_chain objects. | |
| 465 | + * @vma: The VMA whose links to anon_vma objects is to be severed. | |
| 466 | + * | |
| 467 | + * As part of the process anon_vma_chain's are freed, | |
| 468 | + * anon_vma->num_children,num_active_vmas is updated as required and, if the | |
| 469 | + * relevant anon_vma references no further VMAs, its reference count is | |
| 470 | + * decremented. | |
| 471 | + */ | |
| 395 | 472 | void unlink_anon_vmas(struct vm_area_struct *vma) |
| 396 | 473 | { |
| 397 | 474 | struct anon_vma_chain *avc, *next; |
| 398 | − struct anon_vma *root = NULL; | |
| 475 | + struct anon_vma *active_anon_vma = vma->anon_vma; | |
| 476 | + | |
| 477 | + /* Always hold mmap lock, read-lock on unmap possibly. */ | |
| 478 | + mmap_assert_locked(vma->vm_mm); | |
| 479 | + | |
| 480 | + /* Unfaulted is a no-op. */ | |
| 481 | + if (!active_anon_vma) { | |
| 482 | + VM_WARN_ON_ONCE(!list_empty(&vma->anon_vma_chain)); | |
| 483 | + return; | |
| 484 | + } | |
| 485 | + | |
| 486 | + anon_vma_lock_write(active_anon_vma); | |
| 399 | 487 | |
| 400 | 488 | /* |
| 401 | 489 | * Unlink each anon_vma chained to the VMA. This list is ordered |
| @@ -404,7 +492,6 @@ void unlink_anon_vmas(struct vm_area_struct *vma) · 2 unmodified lines | ||
| 404 | 492 | list_for_each_entry_safe(avc, next, &vma->anon_vma_chain, same_vma) { |
| 405 | 493 | struct anon_vma *anon_vma = avc->anon_vma; |
| 406 | 494 | |
| 407 | − root = lock_anon_vma_root(root, anon_vma); | |
| 408 | 495 | anon_vma_interval_tree_remove(avc, &anon_vma->rb_root); |
| 409 | 496 | |
| 410 | 497 | /* |
| @@ -419,16 +506,15 @@ void unlink_anon_vmas(struct vm_area_struct *vma) · 8 unmodified lines | ||
| 419 | 506 | list_del(&avc->same_vma); |
| 420 | 507 | anon_vma_chain_free(avc); |
| 421 | 508 | } |
| 422 | − if (vma->anon_vma) { | |
| 423 | − vma->anon_vma->num_active_vmas--; | |
| 424 | 509 | |
| 425 | − /* | |
| 426 | − * vma would still be needed after unlink, and anon_vma will be prepared | |
| 427 | − * when handle fault. | |
| 428 | − */ | |
| 429 | − vma->anon_vma = NULL; | |
| 430 | − } | |
| 431 | − unlock_anon_vma_root(root); | |
| 510 | + active_anon_vma->num_active_vmas--; | |
| 511 | + /* | |
| 512 | + * vma would still be needed after unlink, and anon_vma will be prepared | |
| 513 | + * when handle fault. | |
| 514 | + */ | |
| 515 | + vma->anon_vma = NULL; | |
| 516 | + anon_vma_unlock_write(active_anon_vma); | |
| 517 | + | |
| 432 | 518 | |
| 433 | 519 | /* |
| 434 | 520 | * Iterate the list once more, it now only contains empty and unlinked |
| @@ -2147,7 +2233,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · 1712 unmodified lines | ||
| 2147 | 2233 | goto discard; |
| 2148 | 2234 | } |
| 2149 | 2235 | |
| 2150 | − if (swap_duplicate(entry) < 0) { | |
| 2236 | + if (folio_dup_swap(folio, subpage) < 0) { | |
| 2151 | 2237 | set_pte_at(mm, address, pvmw.pte, pteval); |
| 2152 | 2238 | goto walk_abort; |
| 2153 | 2239 | } |
| @@ -2158,7 +2244,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · 4 unmodified lines | ||
| 2158 | 2244 | * so we'll not check/care. |
| 2159 | 2245 | */ |
| 2160 | 2246 | if (arch_unmap_one(mm, vma, address, pteval) < 0) { |
| 2161 | − swap_free(entry); | |
| 2247 | + folio_put_swap(folio, subpage); | |
| 2162 | 2248 | set_pte_at(mm, address, pvmw.pte, pteval); |
| 2163 | 2249 | goto walk_abort; |
| 2164 | 2250 | } |
| @@ -2166,7 +2252,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · 1 unmodified line | ||
| 2166 | 2252 | /* See folio_try_share_anon_rmap(): clear PTE first. */ |
| 2167 | 2253 | if (anon_exclusive && |
| 2168 | 2254 | folio_try_share_anon_rmap_pte(folio, subpage)) { |
| 2169 | − swap_free(entry); | |
| 2255 | + folio_put_swap(folio, subpage); | |
| 2170 | 2256 | set_pte_at(mm, address, pvmw.pte, pteval); |
| 2171 | 2257 | goto walk_abort; |
| 2172 | 2258 | } |
| @@ -1,3 +1,4 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 1 | 2 | /* |
| 2 | 3 | * Resizable virtual memory filesystem for Linux. |
| 3 | 4 | * |
| @@ -17,8 +18,6 @@ · 13 unmodified lines | ||
| 17 | 18 | * |
| 18 | 19 | * tiny-shmem: |
| 19 | 20 | * Copyright (c) 2004, 2008 Matt Mackall <mpm@selenic.com> |
| 20 | − * | |
| 21 | − * This file is released under the GPL. | |
| 22 | 21 | */ |
| 23 | 22 | |
| 24 | 23 | #include <linux/fs.h> |
| @@ -983,7 +982,7 @@ static long shmem_free_swap(struct address_space *mapping, · 958 unmodified lines | ||
| 983 | 982 | xas_unlock_irq(&xas); |
| 984 | 983 | |
| 985 | 984 | if (nr_pages) |
| 986 | − free_swap_and_cache_nr(radix_to_swp_entry(radswap), nr_pages); | |
| 985 | + swap_put_entries_direct(radix_to_swp_entry(radswap), nr_pages); | |
| 987 | 986 | |
| 988 | 987 | return nr_pages; |
| 989 | 988 | } |
| @@ -1622,11 +1621,23 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug, · 632 unmodified lines | ||
| 1622 | 1621 | } |
| 1623 | 1622 | |
| 1624 | 1623 | if (split) { |
| 1624 | + int order; | |
| 1625 | + | |
| 1625 | 1626 | try_split: |
| 1627 | + order = folio_order(folio); | |
| 1626 | 1628 | /* Ensure the subpages are still dirty */ |
| 1627 | 1629 | folio_test_set_dirty(folio); |
| 1628 | 1630 | if (split_folio_to_list(folio, folio_list)) |
| 1629 | 1631 | goto redirty; |
| 1632 | + | |
| 1633 | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 1634 | + if (order >= HPAGE_PMD_ORDER) { | |
| 1635 | + count_memcg_folio_events(folio, THP_SWPOUT_FALLBACK, 1); | |
| 1636 | + count_vm_event(THP_SWPOUT_FALLBACK); | |
| 1637 | + } | |
| 1638 | +#endif | |
| 1639 | + count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK); | |
| 1640 | + | |
| 1630 | 1641 | folio_clear_dirty(folio); |
| 1631 | 1642 | } |
| 1632 | 1643 | |
| @@ -1684,7 +1695,7 @@ try_split: · 51 unmodified lines | ||
| 1684 | 1695 | spin_unlock(&shmem_swaplist_lock); |
| 1685 | 1696 | } |
| 1686 | 1697 | |
| 1687 | − swap_shmem_alloc(folio->swap, nr_pages); | |
| 1698 | + folio_dup_swap(folio, NULL); | |
| 1688 | 1699 | shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap)); |
| 1689 | 1700 | |
| 1690 | 1701 | BUG_ON(folio_mapped(folio)); |
| @@ -1705,7 +1716,7 @@ try_split: · 14 unmodified lines | ||
| 1705 | 1716 | /* Swap entry might be erased by racing shmem_free_swap() */ |
| 1706 | 1717 | if (!error) { |
| 1707 | 1718 | shmem_recalc_inode(inode, 0, -nr_pages); |
| 1708 | − swap_free_nr(folio->swap, nr_pages); | |
| 1719 | + folio_put_swap(folio, NULL); | |
| 1709 | 1720 | } |
| 1710 | 1721 | |
| 1711 | 1722 | /* |
| @@ -2031,10 +2042,9 @@ static struct folio *shmem_swap_alloc_folio(struct inode *inode, · 319 unmodified lines | ||
| 2031 | 2042 | swp_entry_t entry, int order, gfp_t gfp) |
| 2032 | 2043 | { |
| 2033 | 2044 | struct shmem_inode_info *info = SHMEM_I(inode); |
| 2045 | + struct folio *new, *swapcache; | |
| 2034 | 2046 | int nr_pages = 1 << order; |
| 2035 | − struct folio *new; | |
| 2036 | 2047 | gfp_t alloc_gfp; |
| 2037 | − void *shadow; | |
| 2038 | 2048 | |
| 2039 | 2049 | /* |
| 2040 | 2050 | * We have arrived here because our zones are constrained, so don't |
| @@ -2074,34 +2084,19 @@ retry: · 33 unmodified lines | ||
| 2074 | 2084 | goto fallback; |
| 2075 | 2085 | } |
| 2076 | 2086 | |
| 2077 | − /* | |
| 2078 | − * Prevent parallel swapin from proceeding with the swap cache flag. | |
| 2079 | − * | |
| 2080 | − * Of course there is another possible concurrent scenario as well, | |
| 2081 | − * that is to say, the swap cache flag of a large folio has already | |
| 2082 | − * been set by swapcache_prepare(), while another thread may have | |
| 2083 | − * already split the large swap entry stored in the shmem mapping. | |
| 2084 | − * In this case, shmem_add_to_page_cache() will help identify the | |
| 2085 | − * concurrent swapin and return -EEXIST. | |
| 2086 | − */ | |
| 2087 | − if (swapcache_prepare(entry, nr_pages)) { | |
| 2087 | + swapcache = swapin_folio(entry, new); | |
| 2088 | + if (swapcache != new) { | |
| 2088 | 2089 | folio_put(new); |
| 2089 | − new = ERR_PTR(-EEXIST); | |
| 2090 | − /* Try smaller folio to avoid cache conflict */ | |
| 2091 | − goto fallback; | |
| 2090 | + if (!swapcache) { | |
| 2091 | + /* | |
| 2092 | + * The new folio is charged already, swapin can | |
| 2093 | + * only fail due to another raced swapin. | |
| 2094 | + */ | |
| 2095 | + new = ERR_PTR(-EEXIST); | |
| 2096 | + goto fallback; | |
| 2097 | + } | |
| 2092 | 2098 | } |
| 2093 | − | |
| 2094 | − __folio_set_locked(new); | |
| 2095 | − __folio_set_swapbacked(new); | |
| 2096 | − new->swap = entry; | |
| 2097 | − | |
| 2098 | − memcg1_swapin(entry, nr_pages); | |
| 2099 | − shadow = swap_cache_get_shadow(entry); | |
| 2100 | − if (shadow) | |
| 2101 | − workingset_refault(new, shadow); | |
| 2102 | − folio_add_lru(new); | |
| 2103 | − swap_read_folio(new, NULL); | |
| 2104 | − return new; | |
| 2099 | + return swapcache; | |
| 2105 | 2100 | fallback: |
| 2106 | 2101 | /* Order 0 swapin failed, nothing to fallback to, abort */ |
| 2107 | 2102 | if (!order) |
| @@ -2191,8 +2186,7 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, · 83 unmodified lines | ||
| 2191 | 2186 | } |
| 2192 | 2187 | |
| 2193 | 2188 | static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index, |
| 2194 | − struct folio *folio, swp_entry_t swap, | |
| 2195 | − bool skip_swapcache) | |
| 2189 | + struct folio *folio, swp_entry_t swap) | |
| 2196 | 2190 | { |
| 2197 | 2191 | struct address_space *mapping = inode->i_mapping; |
| 2198 | 2192 | swp_entry_t swapin_error; |
| @@ -2208,15 +2202,14 @@ static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index, · 9 unmodified lines | ||
| 2208 | 2202 | |
| 2209 | 2203 | nr_pages = folio_nr_pages(folio); |
| 2210 | 2204 | folio_wait_writeback(folio); |
| 2211 | − if (!skip_swapcache) | |
| 2212 | − swap_cache_del_folio(folio); | |
| 2205 | + folio_put_swap(folio, NULL); | |
| 2206 | + swap_cache_del_folio(folio); | |
| 2213 | 2207 | /* |
| 2214 | 2208 | * Don't treat swapin error folio as alloced. Otherwise inode->i_blocks |
| 2215 | 2209 | * won't be 0 when inode is released and thus trigger WARN_ON(i_blocks) |
| 2216 | 2210 | * in shmem_evict_inode(). |
| 2217 | 2211 | */ |
| 2218 | 2212 | shmem_recalc_inode(inode, -nr_pages, -nr_pages); |
| 2219 | − swap_free_nr(swap, nr_pages); | |
| 2220 | 2213 | } |
| 2221 | 2214 | |
| 2222 | 2215 | static int shmem_split_large_entry(struct inode *inode, pgoff_t index, |
| @@ -2309,7 +2302,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 86 unmodified lines | ||
| 2309 | 2302 | softleaf_t index_entry; |
| 2310 | 2303 | struct swap_info_struct *si; |
| 2311 | 2304 | struct folio *folio = NULL; |
| 2312 | − bool skip_swapcache = false; | |
| 2313 | 2305 | int error, nr_pages, order; |
| 2314 | 2306 | pgoff_t offset; |
| 2315 | 2307 | |
| @@ -2352,7 +2344,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 36 unmodified lines | ||
| 2352 | 2344 | folio = NULL; |
| 2353 | 2345 | goto failed; |
| 2354 | 2346 | } |
| 2355 | − skip_swapcache = true; | |
| 2356 | 2347 | } else { |
| 2357 | 2348 | /* Cached swapin only supports order 0 folio */ |
| 2358 | 2349 | folio = shmem_swapin_cluster(swap, gfp, info, index); |
| @@ -2408,9 +2399,8 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 49 unmodified lines | ||
| 2408 | 2399 | * and swap cache folios are never partially freed. |
| 2409 | 2400 | */ |
| 2410 | 2401 | folio_lock(folio); |
| 2411 | − if ((!skip_swapcache && !folio_test_swapcache(folio)) || | |
| 2412 | − shmem_confirm_swap(mapping, index, swap) < 0 || | |
| 2413 | − folio->swap.val != swap.val) { | |
| 2402 | + if (!folio_matches_swap_entry(folio, swap) || | |
| 2403 | + shmem_confirm_swap(mapping, index, swap) < 0) { | |
| 2414 | 2404 | error = -EEXIST; |
| 2415 | 2405 | goto unlock; |
| 2416 | 2406 | } |
| @@ -2442,14 +2432,9 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, · 25 unmodified lines | ||
| 2442 | 2432 | if (sgp == SGP_WRITE) |
| 2443 | 2433 | folio_mark_accessed(folio); |
| 2444 | 2434 | |
| 2445 | − if (skip_swapcache) { | |
| 2446 | − folio->swap.val = 0; | |
| 2447 | − swapcache_clear(si, swap, nr_pages); | |
| 2448 | − } else { | |
| 2449 | − swap_cache_del_folio(folio); | |
| 2450 | − } | |
| 2435 | + folio_put_swap(folio, NULL); | |
| 2436 | + swap_cache_del_folio(folio); | |
| 2451 | 2437 | folio_mark_dirty(folio); |
| 2452 | − swap_free_nr(swap, nr_pages); | |
| 2453 | 2438 | put_swap_device(si); |
| 2454 | 2439 | |
| 2455 | 2440 | *foliop = folio; |
| @@ -2458,14 +2443,11 @@ failed: · 2 unmodified lines | ||
| 2458 | 2443 | if (shmem_confirm_swap(mapping, index, swap) < 0) |
| 2459 | 2444 | error = -EEXIST; |
| 2460 | 2445 | if (error == -EIO) |
| 2461 | − shmem_set_folio_swapin_error(inode, index, folio, swap, | |
| 2462 | − skip_swapcache); | |
| 2446 | + shmem_set_folio_swapin_error(inode, index, folio, swap); | |
| 2463 | 2447 | unlock: |
| 2464 | 2448 | if (folio) |
| 2465 | 2449 | folio_unlock(folio); |
| 2466 | 2450 | failed_nolock: |
| 2467 | − if (skip_swapcache) | |
| 2468 | − swapcache_clear(si, folio->swap, folio_nr_pages(folio)); | |
| 2469 | 2451 | if (folio) |
| 2470 | 2452 | folio_put(folio); |
| 2471 | 2453 | put_swap_device(si); |
| @@ -278,8 +278,7 @@ static void show_free_areas(unsigned int filter, nodemask_t *nodemask, int max_z · 277 unmodified lines | ||
| 278 | 278 | #endif |
| 279 | 279 | K(node_page_state(pgdat, NR_PAGETABLE)), |
| 280 | 280 | K(node_page_state(pgdat, NR_SECONDARY_PAGETABLE)), |
| 281 | − str_yes_no(atomic_read(&pgdat->kswapd_failures) >= | |
| 282 | − MAX_RECLAIM_RETRIES), | |
| 281 | + str_yes_no(kswapd_test_hopeless(pgdat)), | |
| 283 | 282 | K(node_page_state(pgdat, NR_BALLOON_PAGES))); |
| 284 | 283 | } |
| 285 | 284 | |
| @@ -70,7 +70,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) · 69 unmodified lines | ||
| 70 | 70 | memcg_aware ? memcg : NULL, |
| 71 | 71 | count_per_node); |
| 72 | 72 | if (total) { |
| 73 | − seq_printf(m, "%lu", mem_cgroup_ino(memcg)); | |
| 73 | + seq_printf(m, "%llu", mem_cgroup_id(memcg)); | |
| 74 | 74 | for_each_node(nid) |
| 75 | 75 | seq_printf(m, " %lu", count_per_node[nid]); |
| 76 | 76 | seq_putc(m, '\n'); |
| @@ -106,7 +106,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, · 29 unmodified lines | ||
| 106 | 106 | size_t size, loff_t *pos) |
| 107 | 107 | { |
| 108 | 108 | struct shrinker *shrinker = file->private_data; |
| 109 | − unsigned long nr_to_scan = 0, ino, read_len; | |
| 109 | + unsigned long nr_to_scan = 0, read_len; | |
| 110 | + u64 id; | |
| 110 | 111 | struct shrink_control sc = { |
| 111 | 112 | .gfp_mask = GFP_KERNEL, |
| 112 | 113 | }; |
| @@ -119,7 +120,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, · 6 unmodified lines | ||
| 119 | 120 | return -EFAULT; |
| 120 | 121 | kbuf[read_len] = '\0'; |
| 121 | 122 | |
| 122 | − if (sscanf(kbuf, "%lu %d %lu", &ino, &nid, &nr_to_scan) != 3) | |
| 123 | + if (sscanf(kbuf, "%llu %d %lu", &id, &nid, &nr_to_scan) != 3) | |
| 123 | 124 | return -EINVAL; |
| 124 | 125 | |
| 125 | 126 | if (nid < 0 || nid >= nr_node_ids) |
| @@ -129,15 +130,15 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, · 3 unmodified lines | ||
| 129 | 130 | return size; |
| 130 | 131 | |
| 131 | 132 | if (shrinker->flags & SHRINKER_MEMCG_AWARE) { |
| 132 | − memcg = mem_cgroup_get_from_ino(ino); | |
| 133 | − if (!memcg || IS_ERR(memcg)) | |
| 133 | + memcg = mem_cgroup_get_from_id(id); | |
| 134 | + if (!memcg) | |
| 134 | 135 | return -ENOENT; |
| 135 | 136 | |
| 136 | 137 | if (!mem_cgroup_online(memcg)) { |
| 137 | 138 | mem_cgroup_put(memcg); |
| 138 | 139 | return -ENOENT; |
| 139 | 140 | } |
| 140 | − } else if (ino != 0) { | |
| 141 | + } else if (id != 0) { | |
| 141 | 142 | return -EINVAL; |
| 142 | 143 | } |
| 143 | 144 | |
| @@ -8334,7 +8334,8 @@ void __init kmem_cache_init(void) · 8333 unmodified lines | ||
| 8334 | 8334 | |
| 8335 | 8335 | void __init kmem_cache_init_late(void) |
| 8336 | 8336 | { |
| 8337 | − flushwq = alloc_workqueue("slub_flushwq", WQ_MEM_RECLAIM, 0); | |
| 8337 | + flushwq = alloc_workqueue("slub_flushwq", WQ_MEM_RECLAIM | WQ_PERCPU, | |
| 8338 | + 0); | |
| 8338 | 8339 | WARN_ON(!flushwq); |
| 8339 | 8340 | } |
| 8340 | 8341 | |
| @@ -513,7 +513,7 @@ void folio_add_lru(struct folio *folio) · 512 unmodified lines | ||
| 513 | 513 | EXPORT_SYMBOL(folio_add_lru); |
| 514 | 514 | |
| 515 | 515 | /** |
| 516 | − * folio_add_lru_vma() - Add a folio to the appropate LRU list for this VMA. | |
| 516 | + * folio_add_lru_vma() - Add a folio to the appropriate LRU list for this VMA. | |
| 517 | 517 | * @folio: The folio to be added to the LRU. |
| 518 | 518 | * @vma: VMA in which the folio is mapped. |
| 519 | 519 | * |
| @@ -183,6 +183,33 @@ static inline void swap_cluster_unlock_irq(struct swap_cluster_info *ci) · 182 unmodified lines | ||
| 183 | 183 | spin_unlock_irq(&ci->lock); |
| 184 | 184 | } |
| 185 | 185 | |
| 186 | +/* | |
| 187 | + * Below are the core routines for doing swap for a folio. | |
| 188 | + * All helpers requires the folio to be locked, and a locked folio | |
| 189 | + * in the swap cache pins the swap entries / slots allocated to the | |
| 190 | + * folio, swap relies heavily on the swap cache and folio lock for | |
| 191 | + * synchronization. | |
| 192 | + * | |
| 193 | + * folio_alloc_swap(): the entry point for a folio to be swapped | |
| 194 | + * out. It allocates swap slots and pins the slots with swap cache. | |
| 195 | + * The slots start with a swap count of zero. | |
| 196 | + * | |
| 197 | + * folio_dup_swap(): increases the swap count of a folio, usually | |
| 198 | + * during it gets unmapped and a swap entry is installed to replace | |
| 199 | + * it (e.g., swap entry in page table). A swap slot with swap | |
| 200 | + * count == 0 should only be increasd by this helper. | |
| 201 | + * | |
| 202 | + * folio_put_swap(): does the opposite thing of folio_dup_swap(). | |
| 203 | + */ | |
| 204 | +int folio_alloc_swap(struct folio *folio); | |
| 205 | +int folio_dup_swap(struct folio *folio, struct page *subpage); | |
| 206 | +void folio_put_swap(struct folio *folio, struct page *subpage); | |
| 207 | + | |
| 208 | +/* For internal use */ | |
| 209 | +extern void swap_entries_free(struct swap_info_struct *si, | |
| 210 | + struct swap_cluster_info *ci, | |
| 211 | + unsigned long offset, unsigned int nr_pages); | |
| 212 | + | |
| 186 | 213 | /* linux/mm/page_io.c */ |
| 187 | 214 | int sio_pool_init(void); |
| 188 | 215 | struct swap_iocb; |
| @@ -236,7 +263,7 @@ static inline bool folio_matches_swap_entry(const struct folio *folio, · 47 unmodified lines | ||
| 236 | 263 | |
| 237 | 264 | /* |
| 238 | 265 | * All swap cache helpers below require the caller to ensure the swap entries |
| 239 | − * used are valid and stablize the device by any of the following ways: | |
| 266 | + * used are valid and stabilize the device by any of the following ways: | |
| 240 | 267 | * - Hold a reference by get_swap_device(): this ensures a single entry is |
| 241 | 268 | * valid and increases the swap device's refcount. |
| 242 | 269 | * - Locking a folio in the swap cache: this ensures the folio's swap entries |
| @@ -245,11 +272,16 @@ static inline bool folio_matches_swap_entry(const struct folio *folio, · 2 unmodified lines | ||
| 245 | 272 | * swap entries in the page table, similar to locking swap cache folio. |
| 246 | 273 | * - See the comment of get_swap_device() for more complex usage. |
| 247 | 274 | */ |
| 275 | +bool swap_cache_has_folio(swp_entry_t entry); | |
| 248 | 276 | struct folio *swap_cache_get_folio(swp_entry_t entry); |
| 249 | 277 | void *swap_cache_get_shadow(swp_entry_t entry); |
| 250 | −void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadow); | |
| 251 | 278 | void swap_cache_del_folio(struct folio *folio); |
| 279 | +struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags, | |
| 280 | + struct mempolicy *mpol, pgoff_t ilx, | |
| 281 | + bool *alloced); | |
| 252 | 282 | /* Below helpers require the caller to lock and pass in the swap cluster. */ |
| 283 | +void __swap_cache_add_folio(struct swap_cluster_info *ci, | |
| 284 | + struct folio *folio, swp_entry_t entry); | |
| 253 | 285 | void __swap_cache_del_folio(struct swap_cluster_info *ci, |
| 254 | 286 | struct folio *folio, swp_entry_t entry, void *shadow); |
| 255 | 287 | void __swap_cache_replace_folio(struct swap_cluster_info *ci, |
| @@ -261,13 +293,11 @@ void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr); · 5 unmodified lines | ||
| 261 | 293 | struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, |
| 262 | 294 | struct vm_area_struct *vma, unsigned long addr, |
| 263 | 295 | struct swap_iocb **plug); |
| 264 | −struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, | |
| 265 | − struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, | |
| 266 | − bool skip_if_exists); | |
| 267 | 296 | struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, |
| 268 | 297 | struct mempolicy *mpol, pgoff_t ilx); |
| 269 | 298 | struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag, |
| 270 | 299 | struct vm_fault *vmf); |
| 300 | +struct folio *swapin_folio(swp_entry_t entry, struct folio *folio); | |
| 271 | 301 | void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, |
| 272 | 302 | unsigned long addr); |
| 273 | 303 | |
| @@ -303,8 +333,6 @@ static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr, · 29 unmodified lines | ||
| 303 | 333 | |
| 304 | 334 | static inline int non_swapcache_batch(swp_entry_t entry, int max_nr) |
| 305 | 335 | { |
| 306 | − struct swap_info_struct *si = __swap_entry_to_info(entry); | |
| 307 | − pgoff_t offset = swp_offset(entry); | |
| 308 | 336 | int i; |
| 309 | 337 | |
| 310 | 338 | /* |
| @@ -313,8 +341,9 @@ static inline int non_swapcache_batch(swp_entry_t entry, int max_nr) · 2 unmodified lines | ||
| 313 | 341 | * be in conflict with the folio in swap cache. |
| 314 | 342 | */ |
| 315 | 343 | for (i = 0; i < max_nr; i++) { |
| 316 | − if ((si->swap_map[offset + i] & SWAP_HAS_CACHE)) | |
| 344 | + if (swap_cache_has_folio(entry)) | |
| 317 | 345 | return i; |
| 346 | + entry.val++; | |
| 318 | 347 | } |
| 319 | 348 | |
| 320 | 349 | return i; |
| @@ -353,9 +382,24 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry) · 32 unmodified lines | ||
| 353 | 382 | return NULL; |
| 354 | 383 | } |
| 355 | 384 | |
| 385 | +static inline int folio_alloc_swap(struct folio *folio) | |
| 386 | +{ | |
| 387 | + return -EINVAL; | |
| 388 | +} | |
| 389 | + | |
| 390 | +static inline int folio_dup_swap(struct folio *folio, struct page *page) | |
| 391 | +{ | |
| 392 | + return -EINVAL; | |
| 393 | +} | |
| 394 | + | |
| 395 | +static inline void folio_put_swap(struct folio *folio, struct page *page) | |
| 396 | +{ | |
| 397 | +} | |
| 398 | + | |
| 356 | 399 | static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug) |
| 357 | 400 | { |
| 358 | 401 | } |
| 402 | + | |
| 359 | 403 | static inline void swap_write_unplug(struct swap_iocb *sio) |
| 360 | 404 | { |
| 361 | 405 | } |
| @@ -386,6 +430,11 @@ static inline struct folio *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, · 24 unmodified lines | ||
| 386 | 430 | return NULL; |
| 387 | 431 | } |
| 388 | 432 | |
| 433 | +static inline struct folio *swapin_folio(swp_entry_t entry, struct folio *folio) | |
| 434 | +{ | |
| 435 | + return NULL; | |
| 436 | +} | |
| 437 | + | |
| 389 | 438 | static inline void swap_update_readahead(struct folio *folio, |
| 390 | 439 | struct vm_area_struct *vma, unsigned long addr) |
| 391 | 440 | { |
| @@ -397,8 +446,9 @@ static inline int swap_writeout(struct folio *folio, · 5 unmodified lines | ||
| 397 | 446 | return 0; |
| 398 | 447 | } |
| 399 | 448 | |
| 400 | −static inline void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr) | |
| 449 | +static inline bool swap_cache_has_folio(swp_entry_t entry) | |
| 401 | 450 | { |
| 451 | + return false; | |
| 402 | 452 | } |
| 403 | 453 | |
| 404 | 454 | static inline struct folio *swap_cache_get_folio(swp_entry_t entry) |
| @@ -411,10 +461,6 @@ static inline void *swap_cache_get_shadow(swp_entry_t entry) · 6 unmodified lines | ||
| 411 | 461 | return NULL; |
| 412 | 462 | } |
| 413 | 463 | |
| 414 | −static inline void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadow) | |
| 415 | −{ | |
| 416 | −} | |
| 417 | − | |
| 418 | 464 | static inline void swap_cache_del_folio(struct folio *folio) |
| 419 | 465 | { |
| 420 | 466 | } |
| @@ -81,7 +81,7 @@ void show_swap_cache_info(void) · 80 unmodified lines | ||
| 81 | 81 | * Context: Caller must ensure @entry is valid and protect the swap device |
| 82 | 82 | * with reference count or locks. |
| 83 | 83 | * Return: Returns the found folio on success, NULL otherwise. The caller |
| 84 | − * must lock nd check if the folio still matches the swap entry before | |
| 84 | + * must lock and check if the folio still matches the swap entry before | |
| 85 | 85 | * use (e.g., folio_matches_swap_entry). |
| 86 | 86 | */ |
| 87 | 87 | struct folio *swap_cache_get_folio(swp_entry_t entry) |
| @@ -102,6 +102,22 @@ struct folio *swap_cache_get_folio(swp_entry_t entry) · 14 unmodified lines | ||
| 102 | 102 | return NULL; |
| 103 | 103 | } |
| 104 | 104 | |
| 105 | +/** | |
| 106 | + * swap_cache_has_folio - Check if a swap slot has cache. | |
| 107 | + * @entry: swap entry indicating the slot. | |
| 108 | + * | |
| 109 | + * Context: Caller must ensure @entry is valid and protect the swap | |
| 110 | + * device with reference count or locks. | |
| 111 | + */ | |
| 112 | +bool swap_cache_has_folio(swp_entry_t entry) | |
| 113 | +{ | |
| 114 | + unsigned long swp_tb; | |
| 115 | + | |
| 116 | + swp_tb = swap_table_get(__swap_entry_to_cluster(entry), | |
| 117 | + swp_cluster_offset(entry)); | |
| 118 | + return swp_tb_is_folio(swp_tb); | |
| 119 | +} | |
| 120 | + | |
| 105 | 121 | /** |
| 106 | 122 | * swap_cache_get_shadow - Looks up a shadow in the swap cache. |
| 107 | 123 | * @entry: swap entry used for the lookup. |
| @@ -121,6 +137,34 @@ void *swap_cache_get_shadow(swp_entry_t entry) · 13 unmodified lines | ||
| 121 | 137 | return NULL; |
| 122 | 138 | } |
| 123 | 139 | |
| 140 | +void __swap_cache_add_folio(struct swap_cluster_info *ci, | |
| 141 | + struct folio *folio, swp_entry_t entry) | |
| 142 | +{ | |
| 143 | + unsigned long new_tb; | |
| 144 | + unsigned int ci_start, ci_off, ci_end; | |
| 145 | + unsigned long nr_pages = folio_nr_pages(folio); | |
| 146 | + | |
| 147 | + VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); | |
| 148 | + VM_WARN_ON_ONCE_FOLIO(folio_test_swapcache(folio), folio); | |
| 149 | + VM_WARN_ON_ONCE_FOLIO(!folio_test_swapbacked(folio), folio); | |
| 150 | + | |
| 151 | + new_tb = folio_to_swp_tb(folio); | |
| 152 | + ci_start = swp_cluster_offset(entry); | |
| 153 | + ci_off = ci_start; | |
| 154 | + ci_end = ci_start + nr_pages; | |
| 155 | + do { | |
| 156 | + VM_WARN_ON_ONCE(swp_tb_is_folio(__swap_table_get(ci, ci_off))); | |
| 157 | + __swap_table_set(ci, ci_off, new_tb); | |
| 158 | + } while (++ci_off < ci_end); | |
| 159 | + | |
| 160 | + folio_ref_add(folio, nr_pages); | |
| 161 | + folio_set_swapcache(folio); | |
| 162 | + folio->swap = entry; | |
| 163 | + | |
| 164 | + node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages); | |
| 165 | + lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages); | |
| 166 | +} | |
| 167 | + | |
| 124 | 168 | /** |
| 125 | 169 | * swap_cache_add_folio - Add a folio into the swap cache. |
| 126 | 170 | * @folio: The folio to be added. |
| @@ -130,43 +174,51 @@ void *swap_cache_get_shadow(swp_entry_t entry) · 3 unmodified lines | ||
| 130 | 174 | * |
| 131 | 175 | * Context: Caller must ensure @entry is valid and protect the swap device |
| 132 | 176 | * with reference count or locks. |
| 133 | − * The caller also needs to update the corresponding swap_map slots with | |
| 134 | − * SWAP_HAS_CACHE bit to avoid race or conflict. | |
| 135 | 177 | */ |
| 136 | −void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadowp) | |
| 178 | +static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry, | |
| 179 | + void **shadowp) | |
| 137 | 180 | { |
| 181 | + int err; | |
| 138 | 182 | void *shadow = NULL; |
| 139 | − unsigned long old_tb, new_tb; | |
| 183 | + unsigned long old_tb; | |
| 184 | + struct swap_info_struct *si; | |
| 140 | 185 | struct swap_cluster_info *ci; |
| 141 | − unsigned int ci_start, ci_off, ci_end; | |
| 186 | + unsigned int ci_start, ci_off, ci_end, offset; | |
| 142 | 187 | unsigned long nr_pages = folio_nr_pages(folio); |
| 143 | 188 | |
| 144 | − VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); | |
| 145 | − VM_WARN_ON_ONCE_FOLIO(folio_test_swapcache(folio), folio); | |
| 146 | − VM_WARN_ON_ONCE_FOLIO(!folio_test_swapbacked(folio), folio); | |
| 147 | − | |
| 148 | − new_tb = folio_to_swp_tb(folio); | |
| 189 | + si = __swap_entry_to_info(entry); | |
| 149 | 190 | ci_start = swp_cluster_offset(entry); |
| 150 | 191 | ci_end = ci_start + nr_pages; |
| 151 | 192 | ci_off = ci_start; |
| 152 | − ci = swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry)); | |
| 193 | + offset = swp_offset(entry); | |
| 194 | + ci = swap_cluster_lock(si, swp_offset(entry)); | |
| 195 | + if (unlikely(!ci->table)) { | |
| 196 | + err = -ENOENT; | |
| 197 | + goto failed; | |
| 198 | + } | |
| 153 | 199 | do { |
| 154 | − old_tb = __swap_table_xchg(ci, ci_off, new_tb); | |
| 155 | − WARN_ON_ONCE(swp_tb_is_folio(old_tb)); | |
| 200 | + old_tb = __swap_table_get(ci, ci_off); | |
| 201 | + if (unlikely(swp_tb_is_folio(old_tb))) { | |
| 202 | + err = -EEXIST; | |
| 203 | + goto failed; | |
| 204 | + } | |
| 205 | + if (unlikely(!__swap_count(swp_entry(swp_type(entry), offset)))) { | |
| 206 | + err = -ENOENT; | |
| 207 | + goto failed; | |
| 208 | + } | |
| 156 | 209 | if (swp_tb_is_shadow(old_tb)) |
| 157 | 210 | shadow = swp_tb_to_shadow(old_tb); |
| 211 | + offset++; | |
| 158 | 212 | } while (++ci_off < ci_end); |
| 159 | − | |
| 160 | − folio_ref_add(folio, nr_pages); | |
| 161 | − folio_set_swapcache(folio); | |
| 162 | − folio->swap = entry; | |
| 213 | + __swap_cache_add_folio(ci, folio, entry); | |
| 163 | 214 | swap_cluster_unlock(ci); |
| 164 | − | |
| 165 | − node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages); | |
| 166 | − lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages); | |
| 167 | − | |
| 168 | 215 | if (shadowp) |
| 169 | 216 | *shadowp = shadow; |
| 217 | + return 0; | |
| 218 | + | |
| 219 | +failed: | |
| 220 | + swap_cluster_unlock(ci); | |
| 221 | + return err; | |
| 170 | 222 | } |
| 171 | 223 | |
| 172 | 224 | /** |
| @@ -185,8 +237,10 @@ void swap_cache_add_folio(struct folio *folio, swp_entry_t entry, void **shadowp · 12 unmodified lines | ||
| 185 | 237 | void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio, |
| 186 | 238 | swp_entry_t entry, void *shadow) |
| 187 | 239 | { |
| 240 | + struct swap_info_struct *si; | |
| 188 | 241 | unsigned long old_tb, new_tb; |
| 189 | 242 | unsigned int ci_start, ci_off, ci_end; |
| 243 | + bool folio_swapped = false, need_free = false; | |
| 190 | 244 | unsigned long nr_pages = folio_nr_pages(folio); |
| 191 | 245 | |
| 192 | 246 | VM_WARN_ON_ONCE(__swap_entry_to_cluster(entry) != ci); |
| @@ -194,6 +248,7 @@ void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio, · 1 unmodified line | ||
| 194 | 248 | VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio); |
| 195 | 249 | VM_WARN_ON_ONCE_FOLIO(folio_test_writeback(folio), folio); |
| 196 | 250 | |
| 251 | + si = __swap_entry_to_info(entry); | |
| 197 | 252 | new_tb = shadow_swp_to_tb(shadow); |
| 198 | 253 | ci_start = swp_cluster_offset(entry); |
| 199 | 254 | ci_end = ci_start + nr_pages; |
| @@ -203,12 +258,27 @@ void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio, · 3 unmodified lines | ||
| 203 | 258 | old_tb = __swap_table_xchg(ci, ci_off, new_tb); |
| 204 | 259 | WARN_ON_ONCE(!swp_tb_is_folio(old_tb) || |
| 205 | 260 | swp_tb_to_folio(old_tb) != folio); |
| 261 | + if (__swap_count(swp_entry(si->type, | |
| 262 | + swp_offset(entry) + ci_off - ci_start))) | |
| 263 | + folio_swapped = true; | |
| 264 | + else | |
| 265 | + need_free = true; | |
| 206 | 266 | } while (++ci_off < ci_end); |
| 207 | 267 | |
| 208 | 268 | folio->swap.val = 0; |
| 209 | 269 | folio_clear_swapcache(folio); |
| 210 | 270 | node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages); |
| 211 | 271 | lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages); |
| 272 | + | |
| 273 | + if (!folio_swapped) { | |
| 274 | + swap_entries_free(si, ci, swp_offset(entry), nr_pages); | |
| 275 | + } else if (need_free) { | |
| 276 | + do { | |
| 277 | + if (!__swap_count(entry)) | |
| 278 | + swap_entries_free(si, ci, swp_offset(entry), 1); | |
| 279 | + entry.val++; | |
| 280 | + } while (--nr_pages); | |
| 281 | + } | |
| 212 | 282 | } |
| 213 | 283 | |
| 214 | 284 | /** |
| @@ -230,7 +300,6 @@ void swap_cache_del_folio(struct folio *folio) · 15 unmodified lines | ||
| 230 | 300 | __swap_cache_del_folio(ci, folio, entry, NULL); |
| 231 | 301 | swap_cluster_unlock(ci); |
| 232 | 302 | |
| 233 | − put_swap_folio(folio, entry); | |
| 234 | 303 | folio_ref_sub(folio, folio_nr_pages(folio)); |
| 235 | 304 | } |
| 236 | 305 | |
| @@ -283,7 +352,7 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci, · 46 unmodified lines | ||
| 283 | 352 | } |
| 284 | 353 | |
| 285 | 354 | /** |
| 286 | − * swap_cache_clear_shadow - Clears a set of shadows in the swap cache. | |
| 355 | + * __swap_cache_clear_shadow - Clears a set of shadows in the swap cache. | |
| 287 | 356 | * @entry: The starting index entry. |
| 288 | 357 | * @nr_ents: How many slots need to be cleared. |
| 289 | 358 | * |
| @@ -401,108 +470,143 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma, · 111 unmodified lines | ||
| 401 | 470 | } |
| 402 | 471 | } |
| 403 | 472 | |
| 404 | −struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, | |
| 405 | − struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, | |
| 406 | − bool skip_if_exists) | |
| 473 | +/** | |
| 474 | + * __swap_cache_prepare_and_add - Prepare the folio and add it to swap cache. | |
| 475 | + * @entry: swap entry to be bound to the folio. | |
| 476 | + * @folio: folio to be added. | |
| 477 | + * @gfp: memory allocation flags for charge, can be 0 if @charged if true. | |
| 478 | + * @charged: if the folio is already charged. | |
| 479 | + * | |
| 480 | + * Update the swap_map and add folio as swap cache, typically before swapin. | |
| 481 | + * All swap slots covered by the folio must have a non-zero swap count. | |
| 482 | + * | |
| 483 | + * Context: Caller must protect the swap device with reference count or locks. | |
| 484 | + * Return: Returns the folio being added on success. Returns the existing folio | |
| 485 | + * if @entry is already cached. Returns NULL if raced with swapin or swapoff. | |
| 486 | + */ | |
| 487 | +static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry, | |
| 488 | + struct folio *folio, | |
| 489 | + gfp_t gfp, bool charged) | |
| 407 | 490 | { |
| 408 | − struct swap_info_struct *si = __swap_entry_to_info(entry); | |
| 409 | − struct folio *folio; | |
| 410 | − struct folio *new_folio = NULL; | |
| 411 | − struct folio *result = NULL; | |
| 412 | − void *shadow = NULL; | |
| 491 | + struct folio *swapcache = NULL; | |
| 492 | + void *shadow; | |
| 493 | + int ret; | |
| 413 | 494 | |
| 414 | − *new_page_allocated = false; | |
| 495 | + __folio_set_locked(folio); | |
| 496 | + __folio_set_swapbacked(folio); | |
| 415 | 497 | for (;;) { |
| 416 | − int err; | |
| 498 | + ret = swap_cache_add_folio(folio, entry, &shadow); | |
| 499 | + if (!ret) | |
| 500 | + break; | |
| 417 | 501 | |
| 418 | 502 | /* |
| 419 | − * Check the swap cache first, if a cached folio is found, | |
| 420 | − * return it unlocked. The caller will lock and check it. | |
| 503 | + * Large order allocation needs special handling on | |
| 504 | + * race: if a smaller folio exists in cache, swapin needs | |
| 505 | + * to fallback to order 0, and doing a swap cache lookup | |
| 506 | + * might return a folio that is irrelevant to the faulting | |
| 507 | + * entry because @entry is aligned down. Just return NULL. | |
| 421 | 508 | */ |
| 422 | − folio = swap_cache_get_folio(entry); | |
| 423 | − if (folio) | |
| 424 | − goto got_folio; | |
| 509 | + if (ret != -EEXIST || folio_test_large(folio)) | |
| 510 | + goto failed; | |
| 425 | 511 | |
| 426 | − /* | |
| 427 | − * Just skip read ahead for unused swap slot. | |
| 428 | − */ | |
| 429 | − if (!swap_entry_swapped(si, entry)) | |
| 430 | − goto put_and_return; | |
| 512 | + swapcache = swap_cache_get_folio(entry); | |
| 513 | + if (swapcache) | |
| 514 | + goto failed; | |
| 515 | + } | |
| 431 | 516 | |
| 432 | − /* | |
| 433 | − * Get a new folio to read into from swap. Allocate it now if | |
| 434 | − * new_folio not exist, before marking swap_map SWAP_HAS_CACHE, | |
| 435 | − * when -EEXIST will cause any racers to loop around until we | |
| 436 | − * add it to cache. | |
| 437 | − */ | |
| 438 | − if (!new_folio) { | |
| 439 | − new_folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id()); | |
| 440 | − if (!new_folio) | |
| 441 | − goto put_and_return; | |
| 442 | − } | |
| 517 | + if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry)) { | |
| 518 | + swap_cache_del_folio(folio); | |
| 519 | + goto failed; | |
| 520 | + } | |
| 443 | 521 | |
| 444 | − /* | |
| 445 | − * Swap entry may have been freed since our caller observed it. | |
| 446 | − */ | |
| 447 | − err = swapcache_prepare(entry, 1); | |
| 448 | − if (!err) | |
| 449 | − break; | |
| 450 | − else if (err != -EEXIST) | |
| 451 | − goto put_and_return; | |
| 522 | + memcg1_swapin(entry, folio_nr_pages(folio)); | |
| 523 | + if (shadow) | |
| 524 | + workingset_refault(folio, shadow); | |
| 452 | 525 | |
| 453 | − /* | |
| 454 | − * Protect against a recursive call to __read_swap_cache_async() | |
| 455 | − * on the same entry waiting forever here because SWAP_HAS_CACHE | |
| 456 | − * is set but the folio is not the swap cache yet. This can | |
| 457 | − * happen today if mem_cgroup_swapin_charge_folio() below | |
| 458 | − * triggers reclaim through zswap, which may call | |
| 459 | − * __read_swap_cache_async() in the writeback path. | |
| 460 | − */ | |
| 461 | − if (skip_if_exists) | |
| 462 | − goto put_and_return; | |
| 526 | + /* Caller will initiate read into locked folio */ | |
| 527 | + folio_add_lru(folio); | |
| 528 | + return folio; | |
| 463 | 529 | |
| 464 | − /* | |
| 465 | − * We might race against __swap_cache_del_folio(), and | |
| 466 | − * stumble across a swap_map entry whose SWAP_HAS_CACHE | |
| 467 | − * has not yet been cleared. Or race against another | |
| 468 | − * __read_swap_cache_async(), which has set SWAP_HAS_CACHE | |
| 469 | − * in swap_map, but not yet added its folio to swap cache. | |
| 470 | − */ | |
| 471 | − schedule_timeout_uninterruptible(1); | |
| 472 | − } | |
| 530 | +failed: | |
| 531 | + folio_unlock(folio); | |
| 532 | + return swapcache; | |
| 533 | +} | |
| 473 | 534 | |
| 474 | − /* | |
| 475 | − * The swap entry is ours to swap in. Prepare the new folio. | |
| 476 | − */ | |
| 477 | − __folio_set_locked(new_folio); | |
| 478 | − __folio_set_swapbacked(new_folio); | |
| 535 | +/** | |
| 536 | + * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache. | |
| 537 | + * @entry: the swapped out swap entry to be binded to the folio. | |
| 538 | + * @gfp_mask: memory allocation flags | |
| 539 | + * @mpol: NUMA memory allocation policy to be applied | |
| 540 | + * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE | |
| 541 | + * @new_page_allocated: sets true if allocation happened, false otherwise | |
| 542 | + * | |
| 543 | + * Allocate a folio in the swap cache for one swap slot, typically before | |
| 544 | + * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by | |
| 545 | + * @entry must have a non-zero swap count (swapped out). | |
| 546 | + * Currently only supports order 0. | |
| 547 | + * | |
| 548 | + * Context: Caller must protect the swap device with reference count or locks. | |
| 549 | + * Return: Returns the existing folio if @entry is cached already. Returns | |
| 550 | + * NULL if failed due to -ENOMEM or @entry have a swap count < 1. | |
| 551 | + */ | |
| 552 | +struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask, | |
| 553 | + struct mempolicy *mpol, pgoff_t ilx, | |
| 554 | + bool *new_page_allocated) | |
| 555 | +{ | |
| 556 | + struct swap_info_struct *si = __swap_entry_to_info(entry); | |
| 557 | + struct folio *folio; | |
| 558 | + struct folio *result = NULL; | |
| 479 | 559 | |
| 480 | − if (mem_cgroup_swapin_charge_folio(new_folio, NULL, gfp_mask, entry)) | |
| 481 | − goto fail_unlock; | |
| 560 | + *new_page_allocated = false; | |
| 561 | + /* Check the swap cache again for readahead path. */ | |
| 562 | + folio = swap_cache_get_folio(entry); | |
| 563 | + if (folio) | |
| 564 | + return folio; | |
| 482 | 565 | |
| 483 | − swap_cache_add_folio(new_folio, entry, &shadow); | |
| 484 | − memcg1_swapin(entry, 1); | |
| 566 | + /* Skip allocation for unused and bad swap slot for readahead. */ | |
| 567 | + if (!swap_entry_swapped(si, entry)) | |
| 568 | + return NULL; | |
| 485 | 569 | |
| 486 | − if (shadow) | |
| 487 | − workingset_refault(new_folio, shadow); | |
| 488 | − | |
| 489 | − /* Caller will initiate read into locked new_folio */ | |
| 490 | − folio_add_lru(new_folio); | |
| 491 | − *new_page_allocated = true; | |
| 492 | − folio = new_folio; | |
| 493 | −got_folio: | |
| 494 | − result = folio; | |
| 495 | − goto put_and_return; | |
| 496 | − | |
| 497 | −fail_unlock: | |
| 498 | − put_swap_folio(new_folio, entry); | |
| 499 | − folio_unlock(new_folio); | |
| 500 | −put_and_return: | |
| 501 | − if (!(*new_page_allocated) && new_folio) | |
| 502 | − folio_put(new_folio); | |
| 570 | + /* Allocate a new folio to be added into the swap cache. */ | |
| 571 | + folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id()); | |
| 572 | + if (!folio) | |
| 573 | + return NULL; | |
| 574 | + /* Try add the new folio, returns existing folio or NULL on failure. */ | |
| 575 | + result = __swap_cache_prepare_and_add(entry, folio, gfp_mask, false); | |
| 576 | + if (result == folio) | |
| 577 | + *new_page_allocated = true; | |
| 578 | + else | |
| 579 | + folio_put(folio); | |
| 503 | 580 | return result; |
| 504 | 581 | } |
| 505 | 582 | |
| 583 | +/** | |
| 584 | + * swapin_folio - swap-in one or multiple entries skipping readahead. | |
| 585 | + * @entry: starting swap entry to swap in | |
| 586 | + * @folio: a new allocated and charged folio | |
| 587 | + * | |
| 588 | + * Reads @entry into @folio, @folio will be added to the swap cache. | |
| 589 | + * If @folio is a large folio, the @entry will be rounded down to align | |
| 590 | + * with the folio size. | |
| 591 | + * | |
| 592 | + * Return: returns pointer to @folio on success. If folio is a large folio | |
| 593 | + * and this raced with another swapin, NULL will be returned to allow fallback | |
| 594 | + * to order 0. Else, if another folio was already added to the swap cache, | |
| 595 | + * return that swap cache folio instead. | |
| 596 | + */ | |
| 597 | +struct folio *swapin_folio(swp_entry_t entry, struct folio *folio) | |
| 598 | +{ | |
| 599 | + struct folio *swapcache; | |
| 600 | + pgoff_t offset = swp_offset(entry); | |
| 601 | + unsigned long nr_pages = folio_nr_pages(folio); | |
| 602 | + | |
| 603 | + entry = swp_entry(swp_type(entry), round_down(offset, nr_pages)); | |
| 604 | + swapcache = __swap_cache_prepare_and_add(entry, folio, 0, true); | |
| 605 | + if (swapcache == folio) | |
| 606 | + swap_read_folio(folio, NULL); | |
| 607 | + return swapcache; | |
| 608 | +} | |
| 609 | + | |
| 506 | 610 | /* |
| 507 | 611 | * Locate a page of swap in physical memory, reserving swap cache space |
| 508 | 612 | * and reading the disk if it is not already cached. |
| @@ -524,8 +628,8 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, · 15 unmodified lines | ||
| 524 | 628 | return NULL; |
| 525 | 629 | |
| 526 | 630 | mpol = get_vma_policy(vma, addr, 0, &ilx); |
| 527 | − folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, | |
| 528 | − &page_allocated, false); | |
| 631 | + folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx, | |
| 632 | + &page_allocated); | |
| 529 | 633 | mpol_cond_put(mpol); |
| 530 | 634 | |
| 531 | 635 | if (page_allocated) |
| @@ -642,9 +746,9 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 110 unmodified lines | ||
| 642 | 746 | blk_start_plug(&plug); |
| 643 | 747 | for (offset = start_offset; offset <= end_offset ; offset++) { |
| 644 | 748 | /* Ok, do the async read-ahead now */ |
| 645 | − folio = __read_swap_cache_async( | |
| 646 | − swp_entry(swp_type(entry), offset), | |
| 647 | − gfp_mask, mpol, ilx, &page_allocated, false); | |
| 749 | + folio = swap_cache_alloc_folio( | |
| 750 | + swp_entry(swp_type(entry), offset), gfp_mask, mpol, ilx, | |
| 751 | + &page_allocated); | |
| 648 | 752 | if (!folio) |
| 649 | 753 | continue; |
| 650 | 754 | if (page_allocated) { |
| @@ -661,8 +765,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, · 10 unmodified lines | ||
| 661 | 765 | lru_add_drain(); /* Push any new pages onto the LRU now */ |
| 662 | 766 | skip: |
| 663 | 767 | /* The page was likely read above, so no need for plugging here */ |
| 664 | − folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, | |
| 665 | − &page_allocated, false); | |
| 768 | + folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx, | |
| 769 | + &page_allocated); | |
| 666 | 770 | if (unlikely(page_allocated)) |
| 667 | 771 | swap_read_folio(folio, NULL); |
| 668 | 772 | return folio; |
| @@ -766,8 +870,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 97 unmodified lines | ||
| 766 | 870 | if (!si) |
| 767 | 871 | continue; |
| 768 | 872 | } |
| 769 | − folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, | |
| 770 | − &page_allocated, false); | |
| 873 | + folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx, | |
| 874 | + &page_allocated); | |
| 771 | 875 | if (si) |
| 772 | 876 | put_swap_device(si); |
| 773 | 877 | if (!folio) |
| @@ -788,8 +892,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, · 14 unmodified lines | ||
| 788 | 892 | lru_add_drain(); |
| 789 | 893 | skip: |
| 790 | 894 | /* The folio was likely read above, so no need for plugging here */ |
| 791 | − folio = __read_swap_cache_async(targ_entry, gfp_mask, mpol, targ_ilx, | |
| 792 | − &page_allocated, false); | |
| 895 | + folio = swap_cache_alloc_folio(targ_entry, gfp_mask, mpol, targ_ilx, | |
| 896 | + &page_allocated); | |
| 793 | 897 | if (unlikely(page_allocated)) |
| 794 | 898 | swap_read_folio(folio, NULL); |
| 795 | 899 | return folio; |
| @@ -48,16 +48,18 @@ · 47 unmodified lines | ||
| 48 | 48 | #include <linux/swap_cgroup.h> |
| 49 | 49 | #include "swap_table.h" |
| 50 | 50 | #include "internal.h" |
| 51 | +#include "swap_table.h" | |
| 51 | 52 | #include "swap.h" |
| 52 | 53 | |
| 53 | 54 | static bool swap_count_continued(struct swap_info_struct *, pgoff_t, |
| 54 | 55 | unsigned char); |
| 55 | 56 | static void free_swap_count_continuations(struct swap_info_struct *); |
| 56 | −static void swap_entries_free(struct swap_info_struct *si, | |
| 57 | − struct swap_cluster_info *ci, | |
| 58 | − swp_entry_t entry, unsigned int nr_pages); | |
| 59 | 57 | static void swap_range_alloc(struct swap_info_struct *si, |
| 60 | 58 | unsigned int nr_entries); |
| 59 | +static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr); | |
| 60 | +static void swap_put_entry_locked(struct swap_info_struct *si, | |
| 61 | + struct swap_cluster_info *ci, | |
| 62 | + unsigned long offset); | |
| 61 | 63 | static bool folio_swapcache_freeable(struct folio *folio); |
| 62 | 64 | static void move_cluster(struct swap_info_struct *si, |
| 63 | 65 | struct swap_cluster_info *ci, struct list_head *list, |
| @@ -81,9 +83,7 @@ bool swap_migration_ad_supported; · 17 unmodified lines | ||
| 81 | 83 | #endif /* CONFIG_MIGRATION */ |
| 82 | 84 | |
| 83 | 85 | static const char Bad_file[] = "Bad swap file entry "; |
| 84 | −static const char Unused_file[] = "Unused swap file entry "; | |
| 85 | 86 | static const char Bad_offset[] = "Bad swap offset entry "; |
| 86 | −static const char Unused_offset[] = "Unused swap offset entry "; | |
| 87 | 87 | |
| 88 | 88 | /* |
| 89 | 89 | * all active swap_info_structs |
| @@ -144,11 +144,6 @@ static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry) · 54 unmodified lines | ||
| 144 | 144 | return swap_type_to_info(swp_type(entry)); |
| 145 | 145 | } |
| 146 | 146 | |
| 147 | −static inline unsigned char swap_count(unsigned char ent) | |
| 148 | −{ | |
| 149 | − return ent & ~SWAP_HAS_CACHE; /* may include COUNT_CONTINUED flag */ | |
| 150 | −} | |
| 151 | − | |
| 152 | 147 | /* |
| 153 | 148 | * Use the second highest bit of inuse_pages counter as the indicator |
| 154 | 149 | * if one swap device is on the available plist, so the atomic can |
| @@ -180,39 +175,25 @@ static long swap_usage_in_pages(struct swap_info_struct *si) · 25 unmodified lines | ||
| 180 | 175 | #define TTRS_FULL 0x4 |
| 181 | 176 | |
| 182 | 177 | static bool swap_only_has_cache(struct swap_info_struct *si, |
| 183 | − unsigned long offset, int nr_pages) | |
| 178 | + struct swap_cluster_info *ci, | |
| 179 | + unsigned long offset, int nr_pages) | |
| 184 | 180 | { |
| 181 | + unsigned int ci_off = offset % SWAPFILE_CLUSTER; | |
| 185 | 182 | unsigned char *map = si->swap_map + offset; |
| 186 | 183 | unsigned char *map_end = map + nr_pages; |
| 184 | + unsigned long swp_tb; | |
| 187 | 185 | |
| 188 | 186 | do { |
| 189 | − VM_BUG_ON(!(*map & SWAP_HAS_CACHE)); | |
| 190 | − if (*map != SWAP_HAS_CACHE) | |
| 187 | + swp_tb = __swap_table_get(ci, ci_off); | |
| 188 | + VM_WARN_ON_ONCE(!swp_tb_is_folio(swp_tb)); | |
| 189 | + if (*map) | |
| 191 | 190 | return false; |
| 191 | + ++ci_off; | |
| 192 | 192 | } while (++map < map_end); |
| 193 | 193 | |
| 194 | 194 | return true; |
| 195 | 195 | } |
| 196 | 196 | |
| 197 | −static bool swap_is_last_map(struct swap_info_struct *si, | |
| 198 | − unsigned long offset, int nr_pages, bool *has_cache) | |
| 199 | −{ | |
| 200 | − unsigned char *map = si->swap_map + offset; | |
| 201 | − unsigned char *map_end = map + nr_pages; | |
| 202 | − unsigned char count = *map; | |
| 203 | − | |
| 204 | − if (swap_count(count) != 1 && swap_count(count) != SWAP_MAP_SHMEM) | |
| 205 | − return false; | |
| 206 | − | |
| 207 | − while (++map < map_end) { | |
| 208 | − if (*map != count) | |
| 209 | − return false; | |
| 210 | − } | |
| 211 | − | |
| 212 | − *has_cache = !!(count & SWAP_HAS_CACHE); | |
| 213 | − return true; | |
| 214 | −} | |
| 215 | − | |
| 216 | 197 | /* |
| 217 | 198 | * returns number of pages in the folio that backs the swap entry. If positive, |
| 218 | 199 | * the folio was reclaimed. If negative, the folio was not reclaimed. If 0, no |
| @@ -262,12 +243,12 @@ again: · 43 unmodified lines | ||
| 262 | 243 | goto out_unlock; |
| 263 | 244 | |
| 264 | 245 | /* |
| 265 | − * It's safe to delete the folio from swap cache only if the folio's | |
| 266 | − * swap_map is HAS_CACHE only, which means the slots have no page table | |
| 246 | + * It's safe to delete the folio from swap cache only if the folio | |
| 247 | + * is in swap cache with swap count == 0. The slots have no page table | |
| 267 | 248 | * reference or pending writeback, and can't be allocated to others. |
| 268 | 249 | */ |
| 269 | 250 | ci = swap_cluster_lock(si, offset); |
| 270 | − need_reclaim = swap_only_has_cache(si, offset, nr_pages); | |
| 251 | + need_reclaim = swap_only_has_cache(si, ci, offset, nr_pages); | |
| 271 | 252 | swap_cluster_unlock(ci); |
| 272 | 253 | if (!need_reclaim) |
| 273 | 254 | goto out_unlock; |
| @@ -777,68 +758,84 @@ static int swap_cluster_setup_bad_slot(struct swap_cluster_info *cluster_info, · 503 unmodified lines | ||
| 777 | 758 | return 0; |
| 778 | 759 | } |
| 779 | 760 | |
| 761 | +/* | |
| 762 | + * Reclaim drops the ci lock, so the cluster may become unusable (freed or | |
| 763 | + * stolen by a lower order). @usable will be set to false if that happens. | |
| 764 | + */ | |
| 780 | 765 | static bool cluster_reclaim_range(struct swap_info_struct *si, |
| 781 | 766 | struct swap_cluster_info *ci, |
| 782 | − unsigned long start, unsigned long end) | |
| 767 | + unsigned long start, unsigned int order, | |
| 768 | + bool *usable) | |
| 783 | 769 | { |
| 770 | + unsigned int nr_pages = 1 << order; | |
| 771 | + unsigned long offset = start, end = start + nr_pages; | |
| 784 | 772 | unsigned char *map = si->swap_map; |
| 785 | − unsigned long offset = start; | |
| 786 | − int nr_reclaim; | |
| 773 | + unsigned long swp_tb; | |
| 787 | 774 | |
| 788 | 775 | spin_unlock(&ci->lock); |
| 789 | 776 | do { |
| 790 | − switch (READ_ONCE(map[offset])) { | |
| 791 | − case 0: | |
| 792 | − offset++; | |
| 793 | − break; | |
| 794 | − case SWAP_HAS_CACHE: | |
| 795 | − nr_reclaim = __try_to_reclaim_swap(si, offset, TTRS_ANYWAY); | |
| 796 | − if (nr_reclaim > 0) | |
| 797 | − offset += nr_reclaim; | |
| 798 | − else | |
| 799 | − goto out; | |
| 777 | + if (READ_ONCE(map[offset])) | |
| 800 | 778 | break; |
| 801 | − default: | |
| 802 | − goto out; | |
| 779 | + swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER); | |
| 780 | + if (swp_tb_is_folio(swp_tb)) { | |
| 781 | + if (__try_to_reclaim_swap(si, offset, TTRS_ANYWAY) < 0) | |
| 782 | + break; | |
| 803 | 783 | } |
| 804 | − } while (offset < end); | |
| 805 | −out: | |
| 784 | + } while (++offset < end); | |
| 806 | 785 | spin_lock(&ci->lock); |
| 786 | + | |
| 787 | + /* | |
| 788 | + * We just dropped ci->lock so cluster could be used by another | |
| 789 | + * order or got freed, check if it's still usable or empty. | |
| 790 | + */ | |
| 791 | + if (!cluster_is_usable(ci, order)) { | |
| 792 | + *usable = false; | |
| 793 | + return false; | |
| 794 | + } | |
| 795 | + *usable = true; | |
| 796 | + | |
| 797 | + /* Fast path, no need to scan if the whole cluster is empty */ | |
| 798 | + if (cluster_is_empty(ci)) | |
| 799 | + return true; | |
| 800 | + | |
| 807 | 801 | /* |
| 808 | 802 | * Recheck the range no matter reclaim succeeded or not, the slot |
| 809 | 803 | * could have been be freed while we are not holding the lock. |
| 810 | 804 | */ |
| 811 | − for (offset = start; offset < end; offset++) | |
| 812 | − if (READ_ONCE(map[offset])) | |
| 805 | + for (offset = start; offset < end; offset++) { | |
| 806 | + swp_tb = __swap_table_get(ci, offset % SWAPFILE_CLUSTER); | |
| 807 | + if (map[offset] || !swp_tb_is_null(swp_tb)) | |
| 813 | 808 | return false; |
| 809 | + } | |
| 814 | 810 | |
| 815 | 811 | return true; |
| 816 | 812 | } |
| 817 | 813 | |
| 818 | 814 | static bool cluster_scan_range(struct swap_info_struct *si, |
| 819 | 815 | struct swap_cluster_info *ci, |
| 820 | − unsigned long start, unsigned int nr_pages, | |
| 816 | + unsigned long offset, unsigned int nr_pages, | |
| 821 | 817 | bool *need_reclaim) |
| 822 | 818 | { |
| 823 | − unsigned long offset, end = start + nr_pages; | |
| 819 | + unsigned long end = offset + nr_pages; | |
| 824 | 820 | unsigned char *map = si->swap_map; |
| 821 | + unsigned long swp_tb; | |
| 825 | 822 | |
| 826 | 823 | if (cluster_is_empty(ci)) |
| 827 | 824 | return true; |
| 828 | 825 | |
| 829 | − for (offset = start; offset < end; offset++) { | |
| 830 | − switch (READ_ONCE(map[offset])) { | |
| 831 | − case 0: | |
| 832 | − continue; | |
| 833 | − case SWAP_HAS_CACHE: | |
| 826 | + do { | |
| 827 | + if (map[offset]) | |
| 828 | + return false; | |
| 829 | + swp_tb = __swap_table_get(ci, offset % SWAPFILE_CLUSTER); | |
| 830 | + if (swp_tb_is_folio(swp_tb)) { | |
| 834 | 831 | if (!vm_swap_full()) |
| 835 | 832 | return false; |
| 836 | 833 | *need_reclaim = true; |
| 837 | − continue; | |
| 838 | − default: | |
| 839 | − return false; | |
| 834 | + } else { | |
| 835 | + /* A entry with no count and no cache must be null */ | |
| 836 | + VM_WARN_ON_ONCE(!swp_tb_is_null(swp_tb)); | |
| 840 | 837 | } |
| 841 | − } | |
| 838 | + } while (++offset < end); | |
| 842 | 839 | |
| 843 | 840 | return true; |
| 844 | 841 | } |
| @@ -863,28 +860,52 @@ static void swap_cluster_assert_table_empty(struct swap_cluster_info *ci, · 18 unmodified lines | ||
| 863 | 860 | } |
| 864 | 861 | } |
| 865 | 862 | |
| 866 | −static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster_info *ci, | |
| 867 | − unsigned int start, unsigned char usage, | |
| 868 | − unsigned int order) | |
| 863 | +static bool cluster_alloc_range(struct swap_info_struct *si, | |
| 864 | + struct swap_cluster_info *ci, | |
| 865 | + struct folio *folio, | |
| 866 | + unsigned int offset) | |
| 869 | 867 | { |
| 870 | − unsigned int nr_pages = 1 << order; | |
| 868 | + unsigned long nr_pages; | |
| 869 | + unsigned int order; | |
| 871 | 870 | |
| 872 | 871 | lockdep_assert_held(&ci->lock); |
| 873 | 872 | |
| 874 | 873 | if (!(si->flags & SWP_WRITEOK)) |
| 875 | 874 | return false; |
| 876 | 875 | |
| 876 | + /* | |
| 877 | + * All mm swap allocation starts with a folio (folio_alloc_swap), | |
| 878 | + * it's also the only allocation path for large orders allocation. | |
| 879 | + * Such swap slots starts with count == 0 and will be increased | |
| 880 | + * upon folio unmap. | |
| 881 | + * | |
| 882 | + * Else, it's a exclusive order 0 allocation for hibernation. | |
| 883 | + * The slot starts with count == 1 and never increases. | |
| 884 | + */ | |
| 885 | + if (likely(folio)) { | |
| 886 | + order = folio_order(folio); | |
| 887 | + nr_pages = 1 << order; | |
| 888 | + __swap_cache_add_folio(ci, folio, swp_entry(si->type, offset)); | |
| 889 | + } else if (IS_ENABLED(CONFIG_HIBERNATION)) { | |
| 890 | + order = 0; | |
| 891 | + nr_pages = 1; | |
| 892 | + WARN_ON_ONCE(si->swap_map[offset]); | |
| 893 | + si->swap_map[offset] = 1; | |
| 894 | + swap_cluster_assert_table_empty(ci, offset, 1); | |
| 895 | + } else { | |
| 896 | + /* Allocation without folio is only possible with hibernation */ | |
| 897 | + WARN_ON_ONCE(1); | |
| 898 | + return false; | |
| 899 | + } | |
| 900 | + | |
| 877 | 901 | /* |
| 878 | 902 | * The first allocation in a cluster makes the |
| 879 | 903 | * cluster exclusive to this order |
| 880 | 904 | */ |
| 881 | 905 | if (cluster_is_empty(ci)) |
| 882 | 906 | ci->order = order; |
| 883 | − | |
| 884 | − memset(si->swap_map + start, usage, nr_pages); | |
| 885 | − swap_cluster_assert_table_empty(ci, start, nr_pages); | |
| 886 | − swap_range_alloc(si, nr_pages); | |
| 887 | 907 | ci->count += nr_pages; |
| 908 | + swap_range_alloc(si, nr_pages); | |
| 888 | 909 | |
| 889 | 910 | return true; |
| 890 | 911 | } |
| @@ -892,17 +913,17 @@ static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster · 1 unmodified line | ||
| 892 | 913 | /* Try use a new cluster for current CPU and allocate from it. */ |
| 893 | 914 | static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, |
| 894 | 915 | struct swap_cluster_info *ci, |
| 895 | − unsigned long offset, | |
| 896 | − unsigned int order, | |
| 897 | − unsigned char usage) | |
| 916 | + struct folio *folio, unsigned long offset) | |
| 898 | 917 | { |
| 899 | 918 | unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID; |
| 900 | 919 | unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER); |
| 901 | 920 | unsigned long end = min(start + SWAPFILE_CLUSTER, si->max); |
| 921 | + unsigned int order = likely(folio) ? folio_order(folio) : 0; | |
| 902 | 922 | unsigned int nr_pages = 1 << order; |
| 903 | − bool need_reclaim, ret; | |
| 923 | + bool need_reclaim, ret, usable; | |
| 904 | 924 | |
| 905 | 925 | lockdep_assert_held(&ci->lock); |
| 926 | + VM_WARN_ON(!cluster_is_usable(ci, order)); | |
| 906 | 927 | |
| 907 | 928 | if (end < nr_pages || ci->count + nr_pages > SWAPFILE_CLUSTER) |
| 908 | 929 | goto out; |
| @@ -912,14 +933,8 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, · 3 unmodified lines | ||
| 912 | 933 | if (!cluster_scan_range(si, ci, offset, nr_pages, &need_reclaim)) |
| 913 | 934 | continue; |
| 914 | 935 | if (need_reclaim) { |
| 915 | − ret = cluster_reclaim_range(si, ci, offset, offset + nr_pages); | |
| 916 | − /* | |
| 917 | − * Reclaim drops ci->lock and cluster could be used | |
| 918 | − * by another order. Not checking flag as off-list | |
| 919 | − * cluster has no flag set, and change of list | |
| 920 | − * won't cause fragmentation. | |
| 921 | − */ | |
| 922 | − if (!cluster_is_usable(ci, order)) | |
| 936 | + ret = cluster_reclaim_range(si, ci, offset, order, &usable); | |
| 937 | + if (!usable) | |
| 923 | 938 | goto out; |
| 924 | 939 | if (cluster_is_empty(ci)) |
| 925 | 940 | offset = start; |
| @@ -927,7 +942,7 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, · 1 unmodified line | ||
| 927 | 942 | if (!ret) |
| 928 | 943 | continue; |
| 929 | 944 | } |
| 930 | − if (!cluster_alloc_range(si, ci, offset, usage, order)) | |
| 945 | + if (!cluster_alloc_range(si, ci, folio, offset)) | |
| 931 | 946 | break; |
| 932 | 947 | found = offset; |
| 933 | 948 | offset += nr_pages; |
| @@ -949,8 +964,7 @@ out: · 15 unmodified lines | ||
| 949 | 964 | |
| 950 | 965 | static unsigned int alloc_swap_scan_list(struct swap_info_struct *si, |
| 951 | 966 | struct list_head *list, |
| 952 | − unsigned int order, | |
| 953 | − unsigned char usage, | |
| 967 | + struct folio *folio, | |
| 954 | 968 | bool scan_all) |
| 955 | 969 | { |
| 956 | 970 | unsigned int found = SWAP_ENTRY_INVALID; |
| @@ -962,7 +976,7 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si, · 5 unmodified lines | ||
| 962 | 976 | if (!ci) |
| 963 | 977 | break; |
| 964 | 978 | offset = cluster_offset(si, ci); |
| 965 | − found = alloc_swap_scan_cluster(si, ci, offset, order, usage); | |
| 979 | + found = alloc_swap_scan_cluster(si, ci, folio, offset); | |
| 966 | 980 | if (found) |
| 967 | 981 | break; |
| 968 | 982 | } while (scan_all); |
| @@ -987,7 +1001,8 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) · 18 unmodified lines | ||
| 987 | 1001 | to_scan--; |
| 988 | 1002 | |
| 989 | 1003 | while (offset < end) { |
| 990 | − if (READ_ONCE(map[offset]) == SWAP_HAS_CACHE) { | |
| 1004 | + if (!READ_ONCE(map[offset]) && | |
| 1005 | + swp_tb_is_folio(swap_table_get(ci, offset % SWAPFILE_CLUSTER))) { | |
| 991 | 1006 | spin_unlock(&ci->lock); |
| 992 | 1007 | nr_reclaim = __try_to_reclaim_swap(si, offset, |
| 993 | 1008 | TTRS_ANYWAY); |
| @@ -1023,10 +1038,11 @@ static void swap_reclaim_work(struct work_struct *work) · 29 unmodified lines | ||
| 1023 | 1038 | * Try to allocate swap entries with specified order and try set a new |
| 1024 | 1039 | * cluster for current CPU too. |
| 1025 | 1040 | */ |
| 1026 | −static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int order, | |
| 1027 | − unsigned char usage) | |
| 1041 | +static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, | |
| 1042 | + struct folio *folio) | |
| 1028 | 1043 | { |
| 1029 | 1044 | struct swap_cluster_info *ci; |
| 1045 | + unsigned int order = likely(folio) ? folio_order(folio) : 0; | |
| 1030 | 1046 | unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID; |
| 1031 | 1047 | |
| 1032 | 1048 | /* |
| @@ -1048,8 +1064,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o · 15 unmodified lines | ||
| 1048 | 1064 | if (cluster_is_usable(ci, order)) { |
| 1049 | 1065 | if (cluster_is_empty(ci)) |
| 1050 | 1066 | offset = cluster_offset(si, ci); |
| 1051 | − found = alloc_swap_scan_cluster(si, ci, offset, | |
| 1052 | − order, usage); | |
| 1067 | + found = alloc_swap_scan_cluster(si, ci, folio, offset); | |
| 1053 | 1068 | } else { |
| 1054 | 1069 | swap_cluster_unlock(ci); |
| 1055 | 1070 | } |
| @@ -1063,22 +1078,19 @@ new_cluster: · 7 unmodified lines | ||
| 1063 | 1078 | * to spread out the writes. |
| 1064 | 1079 | */ |
| 1065 | 1080 | if (si->flags & SWP_PAGE_DISCARD) { |
| 1066 | − found = alloc_swap_scan_list(si, &si->free_clusters, order, usage, | |
| 1067 | − false); | |
| 1081 | + found = alloc_swap_scan_list(si, &si->free_clusters, folio, false); | |
| 1068 | 1082 | if (found) |
| 1069 | 1083 | goto done; |
| 1070 | 1084 | } |
| 1071 | 1085 | |
| 1072 | 1086 | if (order < PMD_ORDER) { |
| 1073 | − found = alloc_swap_scan_list(si, &si->nonfull_clusters[order], | |
| 1074 | − order, usage, true); | |
| 1087 | + found = alloc_swap_scan_list(si, &si->nonfull_clusters[order], folio, true); | |
| 1075 | 1088 | if (found) |
| 1076 | 1089 | goto done; |
| 1077 | 1090 | } |
| 1078 | 1091 | |
| 1079 | 1092 | if (!(si->flags & SWP_PAGE_DISCARD)) { |
| 1080 | − found = alloc_swap_scan_list(si, &si->free_clusters, order, usage, | |
| 1081 | − false); | |
| 1093 | + found = alloc_swap_scan_list(si, &si->free_clusters, folio, false); | |
| 1082 | 1094 | if (found) |
| 1083 | 1095 | goto done; |
| 1084 | 1096 | } |
| @@ -1092,10 +1104,9 @@ new_cluster: · 7 unmodified lines | ||
| 1092 | 1104 | * Scan only one fragment cluster is good enough. Order 0 |
| 1093 | 1105 | * allocation will surely success, and large allocation |
| 1094 | 1106 | * failure is not critical. Scanning one cluster still |
| 1095 | − * keeps the list rotated and reclaimed (for HAS_CACHE). | |
| 1107 | + * keeps the list rotated and reclaimed (for clean swap cache). | |
| 1096 | 1108 | */ |
| 1097 | − found = alloc_swap_scan_list(si, &si->frag_clusters[order], order, | |
| 1098 | − usage, false); | |
| 1109 | + found = alloc_swap_scan_list(si, &si->frag_clusters[order], folio, false); | |
| 1099 | 1110 | if (found) |
| 1100 | 1111 | goto done; |
| 1101 | 1112 | } |
| @@ -1109,13 +1120,11 @@ new_cluster: · 7 unmodified lines | ||
| 1109 | 1120 | * Clusters here have at least one usable slots and can't fail order 0 |
| 1110 | 1121 | * allocation, but reclaim may drop si->lock and race with another user. |
| 1111 | 1122 | */ |
| 1112 | − found = alloc_swap_scan_list(si, &si->frag_clusters[o], | |
| 1113 | − 0, usage, true); | |
| 1123 | + found = alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true); | |
| 1114 | 1124 | if (found) |
| 1115 | 1125 | goto done; |
| 1116 | 1126 | |
| 1117 | − found = alloc_swap_scan_list(si, &si->nonfull_clusters[o], | |
| 1118 | − 0, usage, true); | |
| 1127 | + found = alloc_swap_scan_list(si, &si->nonfull_clusters[o], folio, true); | |
| 1119 | 1128 | if (found) |
| 1120 | 1129 | goto done; |
| 1121 | 1130 | } |
| @@ -1306,12 +1315,12 @@ static bool get_swap_device_info(struct swap_info_struct *si) · 184 unmodified lines | ||
| 1306 | 1315 | * Fast path try to get swap entries with specified order from current |
| 1307 | 1316 | * CPU's swap entry pool (a cluster). |
| 1308 | 1317 | */ |
| 1309 | −static bool swap_alloc_fast(swp_entry_t *entry, | |
| 1310 | − int order) | |
| 1318 | +static bool swap_alloc_fast(struct folio *folio) | |
| 1311 | 1319 | { |
| 1320 | + unsigned int order = folio_order(folio); | |
| 1312 | 1321 | struct swap_cluster_info *ci; |
| 1313 | 1322 | struct swap_info_struct *si; |
| 1314 | − unsigned int offset, found = SWAP_ENTRY_INVALID; | |
| 1323 | + unsigned int offset; | |
| 1315 | 1324 | |
| 1316 | 1325 | /* |
| 1317 | 1326 | * Once allocated, swap_info_struct will never be completely freed, |
| @@ -1326,22 +1335,18 @@ static bool swap_alloc_fast(swp_entry_t *entry, · 8 unmodified lines | ||
| 1326 | 1335 | if (cluster_is_usable(ci, order)) { |
| 1327 | 1336 | if (cluster_is_empty(ci)) |
| 1328 | 1337 | offset = cluster_offset(si, ci); |
| 1329 | − found = alloc_swap_scan_cluster(si, ci, offset, order, SWAP_HAS_CACHE); | |
| 1330 | − if (found) | |
| 1331 | − *entry = swp_entry(si->type, found); | |
| 1338 | + alloc_swap_scan_cluster(si, ci, folio, offset); | |
| 1332 | 1339 | } else { |
| 1333 | 1340 | swap_cluster_unlock(ci); |
| 1334 | 1341 | } |
| 1335 | 1342 | |
| 1336 | 1343 | put_swap_device(si); |
| 1337 | − return !!found; | |
| 1344 | + return folio_test_swapcache(folio); | |
| 1338 | 1345 | } |
| 1339 | 1346 | |
| 1340 | 1347 | /* Rotate the device and switch to a new cluster */ |
| 1341 | −static void swap_alloc_slow(swp_entry_t *entry, | |
| 1342 | − int order) | |
| 1348 | +static void swap_alloc_slow(struct folio *folio) | |
| 1343 | 1349 | { |
| 1344 | − unsigned long offset; | |
| 1345 | 1350 | struct swap_info_struct *si, *next; |
| 1346 | 1351 | |
| 1347 | 1352 | spin_lock(&swap_avail_lock); |
| @@ -1351,13 +1356,11 @@ start_over: · 3 unmodified lines | ||
| 1351 | 1356 | plist_requeue(&si->avail_list, &swap_avail_head); |
| 1352 | 1357 | spin_unlock(&swap_avail_lock); |
| 1353 | 1358 | if (get_swap_device_info(si)) { |
| 1354 | − offset = cluster_alloc_swap_entry(si, order, SWAP_HAS_CACHE); | |
| 1359 | + cluster_alloc_swap_entry(si, folio); | |
| 1355 | 1360 | put_swap_device(si); |
| 1356 | − if (offset) { | |
| 1357 | − *entry = swp_entry(si->type, offset); | |
| 1361 | + if (folio_test_swapcache(folio)) | |
| 1358 | 1362 | return; |
| 1359 | − } | |
| 1360 | − if (order) | |
| 1363 | + if (folio_test_large(folio)) | |
| 1361 | 1364 | return; |
| 1362 | 1365 | } |
| 1363 | 1366 | |
| @@ -1408,6 +1411,75 @@ start_over: · 44 unmodified lines | ||
| 1408 | 1411 | return false; |
| 1409 | 1412 | } |
| 1410 | 1413 | |
| 1414 | +/** | |
| 1415 | + * swap_put_entries_cluster - Decrease the swap count of a set of slots. | |
| 1416 | + * @si: The swap device. | |
| 1417 | + * @start: start offset of slots. | |
| 1418 | + * @nr: number of slots. | |
| 1419 | + * @reclaim_cache: if true, also reclaim the swap cache. | |
| 1420 | + * | |
| 1421 | + * This helper decreases the swap count of a set of slots and tries to | |
| 1422 | + * batch free them. Also reclaims the swap cache if @reclaim_cache is true. | |
| 1423 | + * Context: The caller must ensure that all slots belong to the same | |
| 1424 | + * cluster and their swap count doesn't go underflow. | |
| 1425 | + */ | |
| 1426 | +static void swap_put_entries_cluster(struct swap_info_struct *si, | |
| 1427 | + unsigned long start, int nr, | |
| 1428 | + bool reclaim_cache) | |
| 1429 | +{ | |
| 1430 | + unsigned long offset = start, end = start + nr; | |
| 1431 | + unsigned long batch_start = SWAP_ENTRY_INVALID; | |
| 1432 | + struct swap_cluster_info *ci; | |
| 1433 | + bool need_reclaim = false; | |
| 1434 | + unsigned int nr_reclaimed; | |
| 1435 | + unsigned long swp_tb; | |
| 1436 | + unsigned int count; | |
| 1437 | + | |
| 1438 | + ci = swap_cluster_lock(si, offset); | |
| 1439 | + do { | |
| 1440 | + swp_tb = __swap_table_get(ci, offset % SWAPFILE_CLUSTER); | |
| 1441 | + count = si->swap_map[offset]; | |
| 1442 | + VM_WARN_ON(count < 1 || count == SWAP_MAP_BAD); | |
| 1443 | + if (count == 1) { | |
| 1444 | + /* count == 1 and non-cached slots will be batch freed. */ | |
| 1445 | + if (!swp_tb_is_folio(swp_tb)) { | |
| 1446 | + if (!batch_start) | |
| 1447 | + batch_start = offset; | |
| 1448 | + continue; | |
| 1449 | + } | |
| 1450 | + /* count will be 0 after put, slot can be reclaimed */ | |
| 1451 | + need_reclaim = true; | |
| 1452 | + } | |
| 1453 | + /* | |
| 1454 | + * A count != 1 or cached slot can't be freed. Put its swap | |
| 1455 | + * count and then free the interrupted pending batch. Cached | |
| 1456 | + * slots will be freed when folio is removed from swap cache | |
| 1457 | + * (__swap_cache_del_folio). | |
| 1458 | + */ | |
| 1459 | + swap_put_entry_locked(si, ci, offset); | |
| 1460 | + if (batch_start) { | |
| 1461 | + swap_entries_free(si, ci, batch_start, offset - batch_start); | |
| 1462 | + batch_start = SWAP_ENTRY_INVALID; | |
| 1463 | + } | |
| 1464 | + } while (++offset < end); | |
| 1465 | + | |
| 1466 | + if (batch_start) | |
| 1467 | + swap_entries_free(si, ci, batch_start, offset - batch_start); | |
| 1468 | + swap_cluster_unlock(ci); | |
| 1469 | + | |
| 1470 | + if (!need_reclaim || !reclaim_cache) | |
| 1471 | + return; | |
| 1472 | + | |
| 1473 | + offset = start; | |
| 1474 | + do { | |
| 1475 | + nr_reclaimed = __try_to_reclaim_swap(si, offset, | |
| 1476 | + TTRS_UNMAPPED | TTRS_FULL); | |
| 1477 | + offset++; | |
| 1478 | + if (nr_reclaimed) | |
| 1479 | + offset = round_up(offset, abs(nr_reclaimed)); | |
| 1480 | + } while (offset < end); | |
| 1481 | +} | |
| 1482 | + | |
| 1411 | 1483 | /** |
| 1412 | 1484 | * folio_alloc_swap - allocate swap space for a folio |
| 1413 | 1485 | * @folio: folio we want to move to swap |
| @@ -1422,7 +1494,6 @@ int folio_alloc_swap(struct folio *folio) · 8 unmodified lines | ||
| 1422 | 1494 | { |
| 1423 | 1495 | unsigned int order = folio_order(folio); |
| 1424 | 1496 | unsigned int size = 1 << order; |
| 1425 | − swp_entry_t entry = {}; | |
| 1426 | 1497 | |
| 1427 | 1498 | VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); |
| 1428 | 1499 | VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio); |
| @@ -1447,89 +1518,94 @@ int folio_alloc_swap(struct folio *folio) · 18 unmodified lines | ||
| 1447 | 1518 | |
| 1448 | 1519 | again: |
| 1449 | 1520 | local_lock(&percpu_swap_cluster.lock); |
| 1450 | − if (!swap_alloc_fast(&entry, order)) | |
| 1451 | − swap_alloc_slow(&entry, order); | |
| 1521 | + if (!swap_alloc_fast(folio)) | |
| 1522 | + swap_alloc_slow(folio); | |
| 1452 | 1523 | local_unlock(&percpu_swap_cluster.lock); |
| 1453 | 1524 | |
| 1454 | − if (unlikely(!order && !entry.val)) { | |
| 1525 | + if (!order && unlikely(!folio_test_swapcache(folio))) { | |
| 1455 | 1526 | if (swap_sync_discard()) |
| 1456 | 1527 | goto again; |
| 1457 | 1528 | } |
| 1458 | 1529 | |
| 1459 | 1530 | /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */ |
| 1460 | − if (mem_cgroup_try_charge_swap(folio, entry)) | |
| 1461 | − goto out_free; | |
| 1531 | + if (unlikely(mem_cgroup_try_charge_swap(folio, folio->swap))) | |
| 1532 | + swap_cache_del_folio(folio); | |
| 1462 | 1533 | |
| 1463 | − if (!entry.val) | |
| 1534 | + if (unlikely(!folio_test_swapcache(folio))) | |
| 1464 | 1535 | return -ENOMEM; |
| 1465 | 1536 | |
| 1466 | − swap_cache_add_folio(folio, entry, NULL); | |
| 1467 | − | |
| 1468 | 1537 | return 0; |
| 1538 | +} | |
| 1539 | + | |
| 1540 | +/** | |
| 1541 | + * folio_dup_swap() - Increase swap count of swap entries of a folio. | |
| 1542 | + * @folio: folio with swap entries bounded. | |
| 1543 | + * @subpage: if not NULL, only increase the swap count of this subpage. | |
| 1544 | + * | |
| 1545 | + * Typically called when the folio is unmapped and have its swap entry to | |
| 1546 | + * take its palce. | |
| 1547 | + * | |
| 1548 | + * Context: Caller must ensure the folio is locked and in the swap cache. | |
| 1549 | + * NOTE: The caller also has to ensure there is no raced call to | |
| 1550 | + * swap_put_entries_direct on its swap entry before this helper returns, or | |
| 1551 | + * the swap map may underflow. Currently, we only accept @subpage == NULL | |
| 1552 | + * for shmem due to the limitation of swap continuation: shmem always | |
| 1553 | + * duplicates the swap entry only once, so there is no such issue for it. | |
| 1554 | + */ | |
| 1555 | +int folio_dup_swap(struct folio *folio, struct page *subpage) | |
| 1556 | +{ | |
| 1557 | + int err = 0; | |
| 1558 | + swp_entry_t entry = folio->swap; | |
| 1559 | + unsigned long nr_pages = folio_nr_pages(folio); | |
| 1560 | + | |
| 1561 | + VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); | |
| 1562 | + VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio); | |
| 1563 | + | |
| 1564 | + if (subpage) { | |
| 1565 | + entry.val += folio_page_idx(folio, subpage); | |
| 1566 | + nr_pages = 1; | |
| 1567 | + } | |
| 1568 | + | |
| 1569 | + while (!err && __swap_duplicate(entry, 1, nr_pages) == -ENOMEM) | |
| 1570 | + err = add_swap_count_continuation(entry, GFP_ATOMIC); | |
| 1469 | 1571 | |
| 1470 | −out_free: | |
| 1471 | − put_swap_folio(folio, entry); | |
| 1472 | − return -ENOMEM; | |
| 1572 | + return err; | |
| 1473 | 1573 | } |
| 1474 | 1574 | |
| 1475 | −static struct swap_info_struct *_swap_info_get(swp_entry_t entry) | |
| 1575 | +/** | |
| 1576 | + * folio_put_swap() - Decrease swap count of swap entries of a folio. | |
| 1577 | + * @folio: folio with swap entries bounded, must be in swap cache and locked. | |
| 1578 | + * @subpage: if not NULL, only decrease the swap count of this subpage. | |
| 1579 | + * | |
| 1580 | + * This won't free the swap slots even if swap count drops to zero, they are | |
| 1581 | + * still pinned by the swap cache. User may call folio_free_swap to free them. | |
| 1582 | + * Context: Caller must ensure the folio is locked and in the swap cache. | |
| 1583 | + */ | |
| 1584 | +void folio_put_swap(struct folio *folio, struct page *subpage) | |
| 1476 | 1585 | { |
| 1477 | − struct swap_info_struct *si; | |
| 1478 | − unsigned long offset; | |
| 1586 | + swp_entry_t entry = folio->swap; | |
| 1587 | + unsigned long nr_pages = folio_nr_pages(folio); | |
| 1588 | + struct swap_info_struct *si = __swap_entry_to_info(entry); | |
| 1479 | 1589 | |
| 1480 | − if (!entry.val) | |
| 1481 | − goto out; | |
| 1482 | − si = swap_entry_to_info(entry); | |
| 1483 | − if (!si) | |
| 1484 | − goto bad_nofile; | |
| 1485 | − if (data_race(!(si->flags & SWP_USED))) | |
| 1486 | − goto bad_device; | |
| 1487 | − offset = swp_offset(entry); | |
| 1488 | − if (offset >= si->max) | |
| 1489 | − goto bad_offset; | |
| 1490 | − if (data_race(!si->swap_map[swp_offset(entry)])) | |
| 1491 | − goto bad_free; | |
| 1492 | − return si; | |
| 1590 | + VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); | |
| 1591 | + VM_WARN_ON_FOLIO(!folio_test_swapcache(folio), folio); | |
| 1493 | 1592 | |
| 1494 | −bad_free: | |
| 1495 | − pr_err("%s: %s%08lx\n", __func__, Unused_offset, entry.val); | |
| 1496 | − goto out; | |
| 1497 | −bad_offset: | |
| 1498 | − pr_err("%s: %s%08lx\n", __func__, Bad_offset, entry.val); | |
| 1499 | − goto out; | |
| 1500 | −bad_device: | |
| 1501 | − pr_err("%s: %s%08lx\n", __func__, Unused_file, entry.val); | |
| 1502 | − goto out; | |
| 1503 | −bad_nofile: | |
| 1504 | − pr_err("%s: %s%08lx\n", __func__, Bad_file, entry.val); | |
| 1505 | −out: | |
| 1506 | − return NULL; | |
| 1593 | + if (subpage) { | |
| 1594 | + entry.val += folio_page_idx(folio, subpage); | |
| 1595 | + nr_pages = 1; | |
| 1596 | + } | |
| 1597 | + | |
| 1598 | + swap_put_entries_cluster(si, swp_offset(entry), nr_pages, false); | |
| 1507 | 1599 | } |
| 1508 | 1600 | |
| 1509 | −static unsigned char swap_entry_put_locked(struct swap_info_struct *si, | |
| 1510 | − struct swap_cluster_info *ci, | |
| 1511 | − swp_entry_t entry, | |
| 1512 | − unsigned char usage) | |
| 1601 | +static void swap_put_entry_locked(struct swap_info_struct *si, | |
| 1602 | + struct swap_cluster_info *ci, | |
| 1603 | + unsigned long offset) | |
| 1513 | 1604 | { |
| 1514 | − unsigned long offset = swp_offset(entry); | |
| 1515 | 1605 | unsigned char count; |
| 1516 | − unsigned char has_cache; | |
| 1517 | 1606 | |
| 1518 | 1607 | count = si->swap_map[offset]; |
| 1519 | − | |
| 1520 | − has_cache = count & SWAP_HAS_CACHE; | |
| 1521 | − count &= ~SWAP_HAS_CACHE; | |
| 1522 | − | |
| 1523 | − if (usage == SWAP_HAS_CACHE) { | |
| 1524 | − VM_BUG_ON(!has_cache); | |
| 1525 | − has_cache = 0; | |
| 1526 | − } else if (count == SWAP_MAP_SHMEM) { | |
| 1527 | − /* | |
| 1528 | − * Or we could insist on shmem.c using a special | |
| 1529 | − * swap_shmem_free() and free_shmem_swap_and_cache()... | |
| 1530 | − */ | |
| 1531 | − count = 0; | |
| 1532 | − } else if ((count & ~COUNT_CONTINUED) <= SWAP_MAP_MAX) { | |
| 1608 | + if ((count & ~COUNT_CONTINUED) <= SWAP_MAP_MAX) { | |
| 1533 | 1609 | if (count == COUNT_CONTINUED) { |
| 1534 | 1610 | if (swap_count_continued(si, offset, count)) |
| 1535 | 1611 | count = SWAP_MAP_MAX | COUNT_CONTINUED; |
| @@ -1539,13 +1615,9 @@ static unsigned char swap_entry_put_locked(struct swap_info_struct *si, · 3 unmodified lines | ||
| 1539 | 1615 | count--; |
| 1540 | 1616 | } |
| 1541 | 1617 | |
| 1542 | − usage = count | has_cache; | |
| 1543 | − if (usage) | |
| 1544 | − WRITE_ONCE(si->swap_map[offset], usage); | |
| 1545 | − else | |
| 1546 | − swap_entries_free(si, ci, entry, 1); | |
| 1547 | − | |
| 1548 | − return usage; | |
| 1618 | + WRITE_ONCE(si->swap_map[offset], count); | |
| 1619 | + if (!count && !swp_tb_is_folio(__swap_table_get(ci, offset % SWAPFILE_CLUSTER))) | |
| 1620 | + swap_entries_free(si, ci, offset, 1); | |
| 1549 | 1621 | } |
| 1550 | 1622 | |
| 1551 | 1623 | /* |
| @@ -1574,10 +1646,9 @@ static unsigned char swap_entry_put_locked(struct swap_info_struct *si, · 22 unmodified lines | ||
| 1574 | 1646 | * CPU1 CPU2 |
| 1575 | 1647 | * do_swap_page() |
| 1576 | 1648 | * ... swapoff+swapon |
| 1577 | − * __read_swap_cache_async() | |
| 1578 | − * swapcache_prepare() | |
| 1579 | − * __swap_duplicate() | |
| 1580 | − * // check swap_map | |
| 1649 | + * swap_cache_alloc_folio() | |
| 1650 | + * swap_cache_add_folio() | |
| 1651 | + * // check swap_map | |
| 1581 | 1652 | * // verify PTE not changed |
| 1582 | 1653 | * |
| 1583 | 1654 | * In __swap_duplicate(), the swap_map need to be checked before |
| @@ -1614,105 +1685,15 @@ put_out: · 30 unmodified lines | ||
| 1614 | 1685 | return NULL; |
| 1615 | 1686 | } |
| 1616 | 1687 | |
| 1617 | −static void swap_entries_put_cache(struct swap_info_struct *si, | |
| 1618 | − swp_entry_t entry, int nr) | |
| 1619 | −{ | |
| 1620 | − unsigned long offset = swp_offset(entry); | |
| 1621 | − struct swap_cluster_info *ci; | |
| 1622 | − | |
| 1623 | − ci = swap_cluster_lock(si, offset); | |
| 1624 | − if (swap_only_has_cache(si, offset, nr)) { | |
| 1625 | − swap_entries_free(si, ci, entry, nr); | |
| 1626 | − } else { | |
| 1627 | − for (int i = 0; i < nr; i++, entry.val++) | |
| 1628 | − swap_entry_put_locked(si, ci, entry, SWAP_HAS_CACHE); | |
| 1629 | − } | |
| 1630 | − swap_cluster_unlock(ci); | |
| 1631 | −} | |
| 1632 | − | |
| 1633 | −static bool swap_entries_put_map(struct swap_info_struct *si, | |
| 1634 | − swp_entry_t entry, int nr) | |
| 1635 | −{ | |
| 1636 | − unsigned long offset = swp_offset(entry); | |
| 1637 | − struct swap_cluster_info *ci; | |
| 1638 | − bool has_cache = false; | |
| 1639 | − unsigned char count; | |
| 1640 | − int i; | |
| 1641 | − | |
| 1642 | − if (nr <= 1) | |
| 1643 | − goto fallback; | |
| 1644 | − count = swap_count(data_race(si->swap_map[offset])); | |
| 1645 | − if (count != 1 && count != SWAP_MAP_SHMEM) | |
| 1646 | − goto fallback; | |
| 1647 | − | |
| 1648 | − ci = swap_cluster_lock(si, offset); | |
| 1649 | − if (!swap_is_last_map(si, offset, nr, &has_cache)) { | |
| 1650 | − goto locked_fallback; | |
| 1651 | − } | |
| 1652 | − if (!has_cache) | |
| 1653 | − swap_entries_free(si, ci, entry, nr); | |
| 1654 | − else | |
| 1655 | − for (i = 0; i < nr; i++) | |
| 1656 | − WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE); | |
| 1657 | − swap_cluster_unlock(ci); | |
| 1658 | − | |
| 1659 | − return has_cache; | |
| 1660 | − | |
| 1661 | −fallback: | |
| 1662 | − ci = swap_cluster_lock(si, offset); | |
| 1663 | −locked_fallback: | |
| 1664 | − for (i = 0; i < nr; i++, entry.val++) { | |
| 1665 | − count = swap_entry_put_locked(si, ci, entry, 1); | |
| 1666 | − if (count == SWAP_HAS_CACHE) | |
| 1667 | − has_cache = true; | |
| 1668 | − } | |
| 1669 | − swap_cluster_unlock(ci); | |
| 1670 | − return has_cache; | |
| 1671 | −} | |
| 1672 | − | |
| 1673 | −/* | |
| 1674 | − * Only functions with "_nr" suffix are able to free entries spanning | |
| 1675 | − * cross multi clusters, so ensure the range is within a single cluster | |
| 1676 | − * when freeing entries with functions without "_nr" suffix. | |
| 1677 | − */ | |
| 1678 | −static bool swap_entries_put_map_nr(struct swap_info_struct *si, | |
| 1679 | − swp_entry_t entry, int nr) | |
| 1680 | −{ | |
| 1681 | − int cluster_nr, cluster_rest; | |
| 1682 | − unsigned long offset = swp_offset(entry); | |
| 1683 | − bool has_cache = false; | |
| 1684 | − | |
| 1685 | − cluster_rest = SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER; | |
| 1686 | − while (nr) { | |
| 1687 | − cluster_nr = min(nr, cluster_rest); | |
| 1688 | − has_cache |= swap_entries_put_map(si, entry, cluster_nr); | |
| 1689 | − cluster_rest = SWAPFILE_CLUSTER; | |
| 1690 | − nr -= cluster_nr; | |
| 1691 | − entry.val += cluster_nr; | |
| 1692 | − } | |
| 1693 | − | |
| 1694 | − return has_cache; | |
| 1695 | −} | |
| 1696 | − | |
| 1697 | −/* | |
| 1698 | − * Check if it's the last ref of swap entry in the freeing path. | |
| 1699 | − * Qualified value includes 1, SWAP_HAS_CACHE or SWAP_MAP_SHMEM. | |
| 1700 | − */ | |
| 1701 | −static inline bool __maybe_unused swap_is_last_ref(unsigned char count) | |
| 1702 | −{ | |
| 1703 | − return (count == SWAP_HAS_CACHE) || (count == 1) || | |
| 1704 | − (count == SWAP_MAP_SHMEM); | |
| 1705 | −} | |
| 1706 | − | |
| 1707 | 1688 | /* |
| 1708 | 1689 | * Drop the last ref of swap entries, caller have to ensure all entries |
| 1709 | 1690 | * belong to the same cgroup and cluster. |
| 1710 | 1691 | */ |
| 1711 | −static void swap_entries_free(struct swap_info_struct *si, | |
| 1712 | − struct swap_cluster_info *ci, | |
| 1713 | − swp_entry_t entry, unsigned int nr_pages) | |
| 1692 | +void swap_entries_free(struct swap_info_struct *si, | |
| 1693 | + struct swap_cluster_info *ci, | |
| 1694 | + unsigned long offset, unsigned int nr_pages) | |
| 1714 | 1695 | { |
| 1715 | − unsigned long offset = swp_offset(entry); | |
| 1696 | + swp_entry_t entry = swp_entry(si->type, offset); | |
| 1716 | 1697 | unsigned char *map = si->swap_map + offset; |
| 1717 | 1698 | unsigned char *map_end = map + nr_pages; |
| 1718 | 1699 | |
| @@ -1723,7 +1704,7 @@ static void swap_entries_free(struct swap_info_struct *si, · 4 unmodified lines | ||
| 1723 | 1704 | |
| 1724 | 1705 | ci->count -= nr_pages; |
| 1725 | 1706 | do { |
| 1726 | − VM_BUG_ON(!swap_is_last_ref(*map)); | |
| 1707 | + VM_WARN_ON(*map > 1); | |
| 1727 | 1708 | *map = 0; |
| 1728 | 1709 | } while (++map < map_end); |
| 1729 | 1710 | |
| @@ -1737,55 +1718,18 @@ static void swap_entries_free(struct swap_info_struct *si, · 7 unmodified lines | ||
| 1737 | 1718 | partial_free_cluster(si, ci); |
| 1738 | 1719 | } |
| 1739 | 1720 | |
| 1740 | −/* | |
| 1741 | − * Caller has made sure that the swap device corresponding to entry | |
| 1742 | − * is still around or has not been recycled. | |
| 1743 | − */ | |
| 1744 | −void swap_free_nr(swp_entry_t entry, int nr_pages) | |
| 1745 | −{ | |
| 1746 | − int nr; | |
| 1747 | − struct swap_info_struct *sis; | |
| 1748 | − unsigned long offset = swp_offset(entry); | |
| 1749 | − | |
| 1750 | − sis = _swap_info_get(entry); | |
| 1751 | − if (!sis) | |
| 1752 | − return; | |
| 1753 | − | |
| 1754 | − while (nr_pages) { | |
| 1755 | − nr = min_t(int, nr_pages, SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER); | |
| 1756 | − swap_entries_put_map(sis, swp_entry(sis->type, offset), nr); | |
| 1757 | − offset += nr; | |
| 1758 | − nr_pages -= nr; | |
| 1759 | − } | |
| 1760 | −} | |
| 1761 | − | |
| 1762 | −/* | |
| 1763 | − * Called after dropping swapcache to decrease refcnt to swap entries. | |
| 1764 | − */ | |
| 1765 | −void put_swap_folio(struct folio *folio, swp_entry_t entry) | |
| 1766 | −{ | |
| 1767 | − struct swap_info_struct *si; | |
| 1768 | − int size = 1 << swap_entry_order(folio_order(folio)); | |
| 1769 | − | |
| 1770 | − si = _swap_info_get(entry); | |
| 1771 | − if (!si) | |
| 1772 | − return; | |
| 1773 | − | |
| 1774 | − swap_entries_put_cache(si, entry, size); | |
| 1775 | −} | |
| 1776 | − | |
| 1777 | 1721 | int __swap_count(swp_entry_t entry) |
| 1778 | 1722 | { |
| 1779 | 1723 | struct swap_info_struct *si = __swap_entry_to_info(entry); |
| 1780 | 1724 | pgoff_t offset = swp_offset(entry); |
| 1781 | 1725 | |
| 1782 | − return swap_count(si->swap_map[offset]); | |
| 1726 | + return si->swap_map[offset]; | |
| 1783 | 1727 | } |
| 1784 | 1728 | |
| 1785 | −/* | |
| 1786 | − * How many references to @entry are currently swapped out? | |
| 1787 | − * This does not give an exact answer when swap count is continued, | |
| 1788 | − * but does include the high COUNT_CONTINUED flag to allow for that. | |
| 1729 | +/** | |
| 1730 | + * swap_entry_swapped - Check if the swap entry is swapped. | |
| 1731 | + * @si: the swap device. | |
| 1732 | + * @entry: the swap entry. | |
| 1789 | 1733 | */ |
| 1790 | 1734 | bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry) |
| 1791 | 1735 | { |
| @@ -1794,9 +1738,10 @@ bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry) · 2 unmodified lines | ||
| 1794 | 1738 | int count; |
| 1795 | 1739 | |
| 1796 | 1740 | ci = swap_cluster_lock(si, offset); |
| 1797 | − count = swap_count(si->swap_map[offset]); | |
| 1741 | + count = si->swap_map[offset]; | |
| 1798 | 1742 | swap_cluster_unlock(ci); |
| 1799 | − return !!count; | |
| 1743 | + | |
| 1744 | + return count && count != SWAP_MAP_BAD; | |
| 1800 | 1745 | } |
| 1801 | 1746 | |
| 1802 | 1747 | /* |
| @@ -1812,7 +1757,7 @@ int swp_swapcount(swp_entry_t entry) · 9 unmodified lines | ||
| 1812 | 1757 | pgoff_t offset; |
| 1813 | 1758 | unsigned char *map; |
| 1814 | 1759 | |
| 1815 | − si = _swap_info_get(entry); | |
| 1760 | + si = get_swap_device(entry); | |
| 1816 | 1761 | if (!si) |
| 1817 | 1762 | return 0; |
| 1818 | 1763 | |
| @@ -1820,7 +1765,7 @@ int swp_swapcount(swp_entry_t entry) · 1 unmodified line | ||
| 1820 | 1765 | |
| 1821 | 1766 | ci = swap_cluster_lock(si, offset); |
| 1822 | 1767 | |
| 1823 | − count = swap_count(si->swap_map[offset]); | |
| 1768 | + count = si->swap_map[offset]; | |
| 1824 | 1769 | if (!(count & COUNT_CONTINUED)) |
| 1825 | 1770 | goto out; |
| 1826 | 1771 | |
| @@ -1842,6 +1787,7 @@ int swp_swapcount(swp_entry_t entry) · 15 unmodified lines | ||
| 1842 | 1787 | } while (tmp_count & COUNT_CONTINUED); |
| 1843 | 1788 | out: |
| 1844 | 1789 | swap_cluster_unlock(ci); |
| 1790 | + put_swap_device(si); | |
| 1845 | 1791 | return count; |
| 1846 | 1792 | } |
| 1847 | 1793 | |
| @@ -1858,12 +1804,12 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, · 10 unmodified lines | ||
| 1858 | 1804 | |
| 1859 | 1805 | ci = swap_cluster_lock(si, offset); |
| 1860 | 1806 | if (nr_pages == 1) { |
| 1861 | − if (swap_count(map[roffset])) | |
| 1807 | + if (map[roffset]) | |
| 1862 | 1808 | ret = true; |
| 1863 | 1809 | goto unlock_out; |
| 1864 | 1810 | } |
| 1865 | 1811 | for (i = 0; i < nr_pages; i++) { |
| 1866 | − if (swap_count(map[offset + i])) { | |
| 1812 | + if (map[offset + i]) { | |
| 1867 | 1813 | ret = true; |
| 1868 | 1814 | break; |
| 1869 | 1815 | } |
| @@ -1876,11 +1822,12 @@ unlock_out: · 6 unmodified lines | ||
| 1876 | 1822 | static bool folio_swapped(struct folio *folio) |
| 1877 | 1823 | { |
| 1878 | 1824 | swp_entry_t entry = folio->swap; |
| 1879 | − struct swap_info_struct *si = _swap_info_get(entry); | |
| 1825 | + struct swap_info_struct *si; | |
| 1880 | 1826 | |
| 1881 | − if (!si) | |
| 1882 | − return false; | |
| 1827 | + VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); | |
| 1828 | + VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio); | |
| 1883 | 1829 | |
| 1830 | + si = __swap_entry_to_info(entry); | |
| 1884 | 1831 | if (!IS_ENABLED(CONFIG_THP_SWAP) || likely(!folio_test_large(folio))) |
| 1885 | 1832 | return swap_entry_swapped(si, entry); |
| 1886 | 1833 | |
| @@ -1939,73 +1886,45 @@ bool folio_free_swap(struct folio *folio) · 52 unmodified lines | ||
| 1939 | 1886 | } |
| 1940 | 1887 | |
| 1941 | 1888 | /** |
| 1942 | − * free_swap_and_cache_nr() - Release reference on range of swap entries and | |
| 1943 | − * reclaim their cache if no more references remain. | |
| 1889 | + * swap_put_entries_direct() - Release reference on range of swap entries and | |
| 1890 | + * reclaim their cache if no more references remain. | |
| 1944 | 1891 | * @entry: First entry of range. |
| 1945 | 1892 | * @nr: Number of entries in range. |
| 1946 | 1893 | * |
| 1947 | 1894 | * For each swap entry in the contiguous range, release a reference. If any swap |
| 1948 | 1895 | * entries become free, try to reclaim their underlying folios, if present. The |
| 1949 | 1896 | * offset range is defined by [entry.offset, entry.offset + nr). |
| 1897 | + * | |
| 1898 | + * Context: Caller must ensure there is no race condition on the reference | |
| 1899 | + * owner. e.g., locking the PTL of a PTE containing the entry being released. | |
| 1950 | 1900 | */ |
| 1951 | −void free_swap_and_cache_nr(swp_entry_t entry, int nr) | |
| 1901 | +void swap_put_entries_direct(swp_entry_t entry, int nr) | |
| 1952 | 1902 | { |
| 1953 | 1903 | const unsigned long start_offset = swp_offset(entry); |
| 1954 | 1904 | const unsigned long end_offset = start_offset + nr; |
| 1905 | + unsigned long offset, cluster_end; | |
| 1955 | 1906 | struct swap_info_struct *si; |
| 1956 | − bool any_only_cache = false; | |
| 1957 | − unsigned long offset; | |
| 1958 | 1907 | |
| 1959 | 1908 | si = get_swap_device(entry); |
| 1960 | − if (!si) | |
| 1909 | + if (WARN_ON_ONCE(!si)) | |
| 1961 | 1910 | return; |
| 1962 | − | |
| 1963 | − if (WARN_ON(end_offset > si->max)) | |
| 1964 | − goto out; | |
| 1965 | − | |
| 1966 | − /* | |
| 1967 | − * First free all entries in the range. | |
| 1968 | − */ | |
| 1969 | − any_only_cache = swap_entries_put_map_nr(si, entry, nr); | |
| 1970 | − | |
| 1971 | − /* | |
| 1972 | − * Short-circuit the below loop if none of the entries had their | |
| 1973 | − * reference drop to zero. | |
| 1974 | − */ | |
| 1975 | − if (!any_only_cache) | |
| 1911 | + if (WARN_ON_ONCE(end_offset > si->max)) | |
| 1976 | 1912 | goto out; |
| 1977 | 1913 | |
| 1978 | − /* | |
| 1979 | − * Now go back over the range trying to reclaim the swap cache. | |
| 1980 | − */ | |
| 1981 | − for (offset = start_offset; offset < end_offset; offset += nr) { | |
| 1982 | − nr = 1; | |
| 1983 | − if (READ_ONCE(si->swap_map[offset]) == SWAP_HAS_CACHE) { | |
| 1984 | − /* | |
| 1985 | − * Folios are always naturally aligned in swap so | |
| 1986 | − * advance forward to the next boundary. Zero means no | |
| 1987 | − * folio was found for the swap entry, so advance by 1 | |
| 1988 | − * in this case. Negative value means folio was found | |
| 1989 | − * but could not be reclaimed. Here we can still advance | |
| 1990 | − * to the next boundary. | |
| 1991 | − */ | |
| 1992 | − nr = __try_to_reclaim_swap(si, offset, | |
| 1993 | − TTRS_UNMAPPED | TTRS_FULL); | |
| 1994 | − if (nr == 0) | |
| 1995 | − nr = 1; | |
| 1996 | − else if (nr < 0) | |
| 1997 | − nr = -nr; | |
| 1998 | − nr = ALIGN(offset + 1, nr) - offset; | |
| 1999 | − } | |
| 2000 | − } | |
| 2001 | − | |
| 1914 | + /* Put entries and reclaim cache in each cluster */ | |
| 1915 | + offset = start_offset; | |
| 1916 | + do { | |
| 1917 | + cluster_end = min(round_up(offset + 1, SWAPFILE_CLUSTER), end_offset); | |
| 1918 | + swap_put_entries_cluster(si, offset, cluster_end - offset, true); | |
| 1919 | + offset = cluster_end; | |
| 1920 | + } while (offset < end_offset); | |
| 2002 | 1921 | out: |
| 2003 | 1922 | put_swap_device(si); |
| 2004 | 1923 | } |
| 2005 | 1924 | |
| 2006 | 1925 | #ifdef CONFIG_HIBERNATION |
| 2007 | − | |
| 2008 | −swp_entry_t get_swap_page_of_type(int type) | |
| 1926 | +/* Allocate a slot for hibernation */ | |
| 1927 | +swp_entry_t swap_alloc_hibernation_slot(int type) | |
| 2009 | 1928 | { |
| 2010 | 1929 | struct swap_info_struct *si = swap_type_to_info(type); |
| 2011 | 1930 | unsigned long offset; |
| @@ -2018,11 +1937,11 @@ swp_entry_t get_swap_page_of_type(int type) · 6 unmodified lines | ||
| 2018 | 1937 | if (get_swap_device_info(si)) { |
| 2019 | 1938 | if (si->flags & SWP_WRITEOK) { |
| 2020 | 1939 | /* |
| 2021 | − * Grab the local lock to be complaint | |
| 1940 | + * Grab the local lock to be compliant | |
| 2022 | 1941 | * with swap table allocation. |
| 2023 | 1942 | */ |
| 2024 | 1943 | local_lock(&percpu_swap_cluster.lock); |
| 2025 | − offset = cluster_alloc_swap_entry(si, 0, 1); | |
| 1944 | + offset = cluster_alloc_swap_entry(si, NULL); | |
| 2026 | 1945 | local_unlock(&percpu_swap_cluster.lock); |
| 2027 | 1946 | if (offset) |
| 2028 | 1947 | entry = swp_entry(si->type, offset); |
| @@ -2033,6 +1952,26 @@ fail: · 4 unmodified lines | ||
| 2033 | 1952 | return entry; |
| 2034 | 1953 | } |
| 2035 | 1954 | |
| 1955 | +/* Free a slot allocated by swap_alloc_hibernation_slot */ | |
| 1956 | +void swap_free_hibernation_slot(swp_entry_t entry) | |
| 1957 | +{ | |
| 1958 | + struct swap_info_struct *si; | |
| 1959 | + struct swap_cluster_info *ci; | |
| 1960 | + pgoff_t offset = swp_offset(entry); | |
| 1961 | + | |
| 1962 | + si = get_swap_device(entry); | |
| 1963 | + if (WARN_ON(!si)) | |
| 1964 | + return; | |
| 1965 | + | |
| 1966 | + ci = swap_cluster_lock(si, offset); | |
| 1967 | + swap_put_entry_locked(si, ci, offset); | |
| 1968 | + swap_cluster_unlock(ci); | |
| 1969 | + | |
| 1970 | + /* In theory readahead might add it to the swap cache by accident */ | |
| 1971 | + __try_to_reclaim_swap(si, offset, TTRS_ANYWAY); | |
| 1972 | + put_swap_device(si); | |
| 1973 | +} | |
| 1974 | + | |
| 2036 | 1975 | /* |
| 2037 | 1976 | * Find the swap type that corresponds to given device (if any). |
| 2038 | 1977 | * |
| @@ -2194,7 +2133,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, · 155 unmodified lines | ||
| 2194 | 2133 | /* |
| 2195 | 2134 | * Some architectures may have to restore extra metadata to the page |
| 2196 | 2135 | * when reading from swap. This metadata may be indexed by swap entry |
| 2197 | − * so this must be called before swap_free(). | |
| 2136 | + * so this must be called before folio_put_swap(). | |
| 2198 | 2137 | */ |
| 2199 | 2138 | arch_swap_restore(folio_swap(entry, folio), folio); |
| 2200 | 2139 | |
| @@ -2235,7 +2174,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, · 34 unmodified lines | ||
| 2235 | 2174 | new_pte = pte_mkuffd_wp(new_pte); |
| 2236 | 2175 | setpte: |
| 2237 | 2176 | set_pte_at(vma->vm_mm, addr, pte, new_pte); |
| 2238 | − swap_free(entry); | |
| 2177 | + folio_put_swap(swapcache, folio_file_page(swapcache, swp_offset(entry))); | |
| 2239 | 2178 | out: |
| 2240 | 2179 | if (pte) |
| 2241 | 2180 | pte_unmap_unlock(pte, ptl); |
| @@ -2430,6 +2369,7 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si, · 188 unmodified lines | ||
| 2430 | 2369 | unsigned int prev) |
| 2431 | 2370 | { |
| 2432 | 2371 | unsigned int i; |
| 2372 | + unsigned long swp_tb; | |
| 2433 | 2373 | unsigned char count; |
| 2434 | 2374 | |
| 2435 | 2375 | /* |
| @@ -2440,7 +2380,11 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si, · 4 unmodified lines | ||
| 2440 | 2380 | */ |
| 2441 | 2381 | for (i = prev + 1; i < si->max; i++) { |
| 2442 | 2382 | count = READ_ONCE(si->swap_map[i]); |
| 2443 | − if (count && swap_count(count) != SWAP_MAP_BAD) | |
| 2383 | + swp_tb = swap_table_get(__swap_offset_to_cluster(si, i), | |
| 2384 | + i % SWAPFILE_CLUSTER); | |
| 2385 | + if (count == SWAP_MAP_BAD) | |
| 2386 | + continue; | |
| 2387 | + if (count || swp_tb_is_folio(swp_tb)) | |
| 2444 | 2388 | break; |
| 2445 | 2389 | if ((i % LATENCY_LIMIT) == 0) |
| 2446 | 2390 | cond_resched(); |
| @@ -3650,67 +3594,39 @@ void si_swapinfo(struct sysinfo *val) · 1203 unmodified lines | ||
| 3650 | 3594 | * Returns error code in following case. |
| 3651 | 3595 | * - success -> 0 |
| 3652 | 3596 | * - swp_entry is invalid -> EINVAL |
| 3653 | − * - swap-cache reference is requested but there is already one. -> EEXIST | |
| 3654 | − * - swap-cache reference is requested but the entry is not used. -> ENOENT | |
| 3597 | + * - swap-mapped reference is requested but the entry is not used. -> ENOENT | |
| 3655 | 3598 | * - swap-mapped reference requested but needs continued swap count. -> ENOMEM |
| 3656 | 3599 | */ |
| 3657 | −static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) | |
| 3600 | +static int swap_dup_entries(struct swap_info_struct *si, | |
| 3601 | + struct swap_cluster_info *ci, | |
| 3602 | + unsigned long offset, | |
| 3603 | + unsigned char usage, int nr) | |
| 3658 | 3604 | { |
| 3659 | − struct swap_info_struct *si; | |
| 3660 | − struct swap_cluster_info *ci; | |
| 3661 | − unsigned long offset; | |
| 3605 | + int i; | |
| 3662 | 3606 | unsigned char count; |
| 3663 | − unsigned char has_cache; | |
| 3664 | − int err, i; | |
| 3665 | − | |
| 3666 | − si = swap_entry_to_info(entry); | |
| 3667 | − if (WARN_ON_ONCE(!si)) { | |
| 3668 | − pr_err("%s%08lx\n", Bad_file, entry.val); | |
| 3669 | − return -EINVAL; | |
| 3670 | − } | |
| 3671 | − | |
| 3672 | − offset = swp_offset(entry); | |
| 3673 | − VM_WARN_ON(nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER); | |
| 3674 | − VM_WARN_ON(usage == 1 && nr > 1); | |
| 3675 | − ci = swap_cluster_lock(si, offset); | |
| 3676 | 3607 | |
| 3677 | − err = 0; | |
| 3678 | 3608 | for (i = 0; i < nr; i++) { |
| 3679 | 3609 | count = si->swap_map[offset + i]; |
| 3680 | − | |
| 3681 | 3610 | /* |
| 3682 | − * swapin_readahead() doesn't check if a swap entry is valid, so the | |
| 3683 | − * swap entry could be SWAP_MAP_BAD. Check here with lock held. | |
| 3611 | + * For swapin out, allocator never allocates bad slots. for | |
| 3612 | + * swapin, readahead is guarded by swap_entry_swapped. | |
| 3684 | 3613 | */ |
| 3685 | − if (unlikely(swap_count(count) == SWAP_MAP_BAD)) { | |
| 3686 | − err = -ENOENT; | |
| 3687 | − goto unlock_out; | |
| 3688 | − } | |
| 3689 | − | |
| 3690 | − has_cache = count & SWAP_HAS_CACHE; | |
| 3691 | − count &= ~SWAP_HAS_CACHE; | |
| 3692 | − | |
| 3693 | − if (!count && !has_cache) { | |
| 3694 | − err = -ENOENT; | |
| 3695 | − } else if (usage == SWAP_HAS_CACHE) { | |
| 3696 | − if (has_cache) | |
| 3697 | − err = -EEXIST; | |
| 3698 | − } else if ((count & ~COUNT_CONTINUED) > SWAP_MAP_MAX) { | |
| 3699 | − err = -EINVAL; | |
| 3700 | − } | |
| 3701 | − | |
| 3702 | − if (err) | |
| 3703 | − goto unlock_out; | |
| 3614 | + if (WARN_ON(count == SWAP_MAP_BAD)) | |
| 3615 | + return -ENOENT; | |
| 3616 | + /* | |
| 3617 | + * Swap count duplication must be guarded by either swap cache folio (from | |
| 3618 | + * folio_dup_swap) or external lock of existing entry (from swap_dup_entry_direct). | |
| 3619 | + */ | |
| 3620 | + if (WARN_ON(!count && | |
| 3621 | + !swp_tb_is_folio(__swap_table_get(ci, offset % SWAPFILE_CLUSTER)))) | |
| 3622 | + return -ENOENT; | |
| 3623 | + if (WARN_ON((count & ~COUNT_CONTINUED) > SWAP_MAP_MAX)) | |
| 3624 | + return -EINVAL; | |
| 3704 | 3625 | } |
| 3705 | 3626 | |
| 3706 | 3627 | for (i = 0; i < nr; i++) { |
| 3707 | 3628 | count = si->swap_map[offset + i]; |
| 3708 | − has_cache = count & SWAP_HAS_CACHE; | |
| 3709 | − count &= ~SWAP_HAS_CACHE; | |
| 3710 | − | |
| 3711 | − if (usage == SWAP_HAS_CACHE) | |
| 3712 | − has_cache = SWAP_HAS_CACHE; | |
| 3713 | − else if ((count & ~COUNT_CONTINUED) < SWAP_MAP_MAX) | |
| 3629 | + if ((count & ~COUNT_CONTINUED) < SWAP_MAP_MAX) | |
| 3714 | 3630 | count += usage; |
| 3715 | 3631 | else if (swap_count_continued(si, offset + i, count)) |
| 3716 | 3632 | count = COUNT_CONTINUED; |
| @@ -3719,65 +3635,55 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) · 2 unmodified lines | ||
| 3719 | 3635 | * Don't need to rollback changes, because if |
| 3720 | 3636 | * usage == 1, there must be nr == 1. |
| 3721 | 3637 | */ |
| 3722 | − err = -ENOMEM; | |
| 3723 | − goto unlock_out; | |
| 3638 | + return -ENOMEM; | |
| 3724 | 3639 | } |
| 3725 | 3640 | |
| 3726 | − WRITE_ONCE(si->swap_map[offset + i], count | has_cache); | |
| 3641 | + WRITE_ONCE(si->swap_map[offset + i], count); | |
| 3727 | 3642 | } |
| 3728 | 3643 | |
| 3729 | −unlock_out: | |
| 3730 | − swap_cluster_unlock(ci); | |
| 3731 | − return err; | |
| 3644 | + return 0; | |
| 3732 | 3645 | } |
| 3733 | 3646 | |
| 3734 | −/* | |
| 3735 | − * Help swapoff by noting that swap entry belongs to shmem/tmpfs | |
| 3736 | − * (in which case its reference count is never incremented). | |
| 3737 | − */ | |
| 3738 | −void swap_shmem_alloc(swp_entry_t entry, int nr) | |
| 3647 | +static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) | |
| 3739 | 3648 | { |
| 3740 | − __swap_duplicate(entry, SWAP_MAP_SHMEM, nr); | |
| 3649 | + int err; | |
| 3650 | + struct swap_info_struct *si; | |
| 3651 | + struct swap_cluster_info *ci; | |
| 3652 | + unsigned long offset = swp_offset(entry); | |
| 3653 | + | |
| 3654 | + si = swap_entry_to_info(entry); | |
| 3655 | + if (WARN_ON_ONCE(!si)) { | |
| 3656 | + pr_err("%s%08lx\n", Bad_file, entry.val); | |
| 3657 | + return -EINVAL; | |
| 3658 | + } | |
| 3659 | + | |
| 3660 | + VM_WARN_ON(nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER); | |
| 3661 | + ci = swap_cluster_lock(si, offset); | |
| 3662 | + err = swap_dup_entries(si, ci, offset, usage, nr); | |
| 3663 | + swap_cluster_unlock(ci); | |
| 3664 | + return err; | |
| 3741 | 3665 | } |
| 3742 | 3666 | |
| 3743 | 3667 | /* |
| 3744 | − * Increase reference count of swap entry by 1. | |
| 3668 | + * swap_dup_entry_direct() - Increase reference count of a swap entry by one. | |
| 3669 | + * @entry: first swap entry from which we want to increase the refcount. | |
| 3670 | + * | |
| 3745 | 3671 | * Returns 0 for success, or -ENOMEM if a swap_count_continuation is required |
| 3746 | 3672 | * but could not be atomically allocated. Returns 0, just as if it succeeded, |
| 3747 | 3673 | * if __swap_duplicate() fails for another reason (-EINVAL or -ENOENT), which |
| 3748 | 3674 | * might occur if a page table entry has got corrupted. |
| 3675 | + * | |
| 3676 | + * Context: Caller must ensure there is no race condition on the reference | |
| 3677 | + * owner. e.g., locking the PTL of a PTE containing the entry being increased. | |
| 3749 | 3678 | */ |
| 3750 | −int swap_duplicate(swp_entry_t entry) | |
| 3679 | +int swap_dup_entry_direct(swp_entry_t entry) | |
| 3751 | 3680 | { |
| 3752 | 3681 | int err = 0; |
| 3753 | − | |
| 3754 | 3682 | while (!err && __swap_duplicate(entry, 1, 1) == -ENOMEM) |
| 3755 | 3683 | err = add_swap_count_continuation(entry, GFP_ATOMIC); |
| 3756 | 3684 | return err; |
| 3757 | 3685 | } |
| 3758 | 3686 | |
| 3759 | −/* | |
| 3760 | − * @entry: first swap entry from which we allocate nr swap cache. | |
| 3761 | − * | |
| 3762 | − * Called when allocating swap cache for existing swap entries, | |
| 3763 | − * This can return error codes. Returns 0 at success. | |
| 3764 | − * -EEXIST means there is a swap cache. | |
| 3765 | − * Note: return code is different from swap_duplicate(). | |
| 3766 | − */ | |
| 3767 | −int swapcache_prepare(swp_entry_t entry, int nr) | |
| 3768 | −{ | |
| 3769 | − return __swap_duplicate(entry, SWAP_HAS_CACHE, nr); | |
| 3770 | −} | |
| 3771 | − | |
| 3772 | −/* | |
| 3773 | − * Caller should ensure entries belong to the same folio so | |
| 3774 | − * the entries won't span cross cluster boundary. | |
| 3775 | − */ | |
| 3776 | −void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry, int nr) | |
| 3777 | −{ | |
| 3778 | − swap_entries_put_cache(si, entry, nr); | |
| 3779 | −} | |
| 3780 | − | |
| 3781 | 3687 | /* |
| 3782 | 3688 | * add_swap_count_continuation - called when a swap count is duplicated |
| 3783 | 3689 | * beyond SWAP_MAP_MAX, it allocates a new page and links that to the entry's |
| @@ -3823,7 +3729,7 @@ int add_swap_count_continuation(swp_entry_t entry, gfp_t gfp_mask) · 39 unmodified lines | ||
| 3823 | 3729 | |
| 3824 | 3730 | ci = swap_cluster_lock(si, offset); |
| 3825 | 3731 | |
| 3826 | − count = swap_count(si->swap_map[offset]); | |
| 3732 | + count = si->swap_map[offset]; | |
| 3827 | 3733 | |
| 3828 | 3734 | if ((count & ~COUNT_CONTINUED) != SWAP_MAP_MAX) { |
| 3829 | 3735 | /* |
| @@ -3895,7 +3801,7 @@ outer: · 65 unmodified lines | ||
| 3895 | 3801 | * into, carry if so, or else fail until a new continuation page is allocated; |
| 3896 | 3802 | * when the original swap_map count is decremented from 0 with continuation, |
| 3897 | 3803 | * borrow from the continuation and report whether it still holds more. |
| 3898 | − * Called while __swap_duplicate() or caller of swap_entry_put_locked() | |
| 3804 | + * Called while __swap_duplicate() or caller of swap_put_entry_locked() | |
| 3899 | 3805 | * holds cluster lock. |
| 3900 | 3806 | */ |
| 3901 | 3807 | static bool swap_count_continued(struct swap_info_struct *si, |
| @@ -0,0 +1,73 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-only | |
| 2 | +#include <kunit/test.h> | |
| 3 | +#include <linux/pgtable.h> | |
| 4 | + | |
| 5 | +MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING"); | |
| 6 | + | |
| 7 | +static void expect_not_active(struct kunit *test) | |
| 8 | +{ | |
| 9 | + KUNIT_EXPECT_FALSE(test, is_lazy_mmu_mode_active()); | |
| 10 | +} | |
| 11 | + | |
| 12 | +static void expect_active(struct kunit *test) | |
| 13 | +{ | |
| 14 | + KUNIT_EXPECT_TRUE(test, is_lazy_mmu_mode_active()); | |
| 15 | +} | |
| 16 | + | |
| 17 | +static void lazy_mmu_mode_active(struct kunit *test) | |
| 18 | +{ | |
| 19 | + expect_not_active(test); | |
| 20 | + | |
| 21 | + lazy_mmu_mode_enable(); | |
| 22 | + expect_active(test); | |
| 23 | + | |
| 24 | + { | |
| 25 | + /* Nested section */ | |
| 26 | + lazy_mmu_mode_enable(); | |
| 27 | + expect_active(test); | |
| 28 | + | |
| 29 | + lazy_mmu_mode_disable(); | |
| 30 | + expect_active(test); | |
| 31 | + } | |
| 32 | + | |
| 33 | + { | |
| 34 | + /* Paused section */ | |
| 35 | + lazy_mmu_mode_pause(); | |
| 36 | + expect_not_active(test); | |
| 37 | + | |
| 38 | + { | |
| 39 | + /* No effect (paused) */ | |
| 40 | + lazy_mmu_mode_enable(); | |
| 41 | + expect_not_active(test); | |
| 42 | + | |
| 43 | + lazy_mmu_mode_disable(); | |
| 44 | + expect_not_active(test); | |
| 45 | + | |
| 46 | + lazy_mmu_mode_pause(); | |
| 47 | + expect_not_active(test); | |
| 48 | + | |
| 49 | + lazy_mmu_mode_resume(); | |
| 50 | + expect_not_active(test); | |
| 51 | + } | |
| 52 | + | |
| 53 | + lazy_mmu_mode_resume(); | |
| 54 | + expect_active(test); | |
| 55 | + } | |
| 56 | + | |
| 57 | + lazy_mmu_mode_disable(); | |
| 58 | + expect_not_active(test); | |
| 59 | +} | |
| 60 | + | |
| 61 | +static struct kunit_case lazy_mmu_mode_test_cases[] = { | |
| 62 | + KUNIT_CASE(lazy_mmu_mode_active), | |
| 63 | + {} | |
| 64 | +}; | |
| 65 | + | |
| 66 | +static struct kunit_suite lazy_mmu_mode_test_suite = { | |
| 67 | + .name = "lazy_mmu_mode", | |
| 68 | + .test_cases = lazy_mmu_mode_test_cases, | |
| 69 | +}; | |
| 70 | +kunit_test_suite(lazy_mmu_mode_test_suite); | |
| 71 | + | |
| 72 | +MODULE_DESCRIPTION("Tests for the lazy MMU mode"); | |
| 73 | +MODULE_LICENSE("GPL"); | |
| @@ -1103,7 +1103,7 @@ static long move_present_ptes(struct mm_struct *mm, · 1102 unmodified lines | ||
| 1103 | 1103 | /* It's safe to drop the reference now as the page-table is holding one. */ |
| 1104 | 1104 | folio_put(*first_src_folio); |
| 1105 | 1105 | *first_src_folio = NULL; |
| 1106 | − arch_enter_lazy_mmu_mode(); | |
| 1106 | + lazy_mmu_mode_enable(); | |
| 1107 | 1107 | |
| 1108 | 1108 | while (true) { |
| 1109 | 1109 | orig_src_pte = ptep_get_and_clear(mm, src_addr, src_pte); |
| @@ -1140,7 +1140,7 @@ static long move_present_ptes(struct mm_struct *mm, · 30 unmodified lines | ||
| 1140 | 1140 | break; |
| 1141 | 1141 | } |
| 1142 | 1142 | |
| 1143 | − arch_leave_lazy_mmu_mode(); | |
| 1143 | + lazy_mmu_mode_disable(); | |
| 1144 | 1144 | if (src_addr > src_start) |
| 1145 | 1145 | flush_tlb_range(src_vma, src_start, src_addr); |
| 1146 | 1146 | |
| @@ -1190,17 +1190,13 @@ static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma, · 43 unmodified lines | ||
| 1190 | 1190 | * Check if the swap entry is cached after acquiring the src_pte |
| 1191 | 1191 | * lock. Otherwise, we might miss a newly loaded swap cache folio. |
| 1192 | 1192 | * |
| 1193 | − * Check swap_map directly to minimize overhead, READ_ONCE is sufficient. | |
| 1194 | 1193 | * We are trying to catch newly added swap cache, the only possible case is |
| 1195 | 1194 | * when a folio is swapped in and out again staying in swap cache, using the |
| 1196 | 1195 | * same entry before the PTE check above. The PTL is acquired and released |
| 1197 | − * twice, each time after updating the swap_map's flag. So holding | |
| 1198 | − * the PTL here ensures we see the updated value. False positive is possible, | |
| 1199 | − * e.g. SWP_SYNCHRONOUS_IO swapin may set the flag without touching the | |
| 1200 | − * cache, or during the tiny synchronization window between swap cache and | |
| 1201 | − * swap_map, but it will be gone very quickly, worst result is retry jitters. | |
| 1196 | + * twice, each time after updating the swap table. So holding | |
| 1197 | + * the PTL here ensures we see the updated value. | |
| 1202 | 1198 | */ |
| 1203 | − if (READ_ONCE(si->swap_map[swp_offset(entry)]) & SWAP_HAS_CACHE) { | |
| 1199 | + if (swap_cache_has_folio(entry)) { | |
| 1204 | 1200 | double_pt_unlock(dst_ptl, src_ptl); |
| 1205 | 1201 | return -EAGAIN; |
| 1206 | 1202 | } |
| @@ -1274,7 +1270,7 @@ retry: · 67 unmodified lines | ||
| 1274 | 1270 | * Use the maywrite version to indicate that dst_pte will be modified, |
| 1275 | 1271 | * since dst_pte needs to be none, the subsequent pte_same() check |
| 1276 | 1272 | * cannot prevent the dst_pte page from being freed concurrently, so we |
| 1277 | − * also need to abtain dst_pmdval and recheck pmd_same() later. | |
| 1273 | + * also need to obtain dst_pmdval and recheck pmd_same() later. | |
| 1278 | 1274 | */ |
| 1279 | 1275 | dst_pte = pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dst_pmdval, |
| 1280 | 1276 | &dst_ptl); |
| @@ -1330,7 +1326,7 @@ retry: · 49 unmodified lines | ||
| 1330 | 1326 | goto out; |
| 1331 | 1327 | } |
| 1332 | 1328 | |
| 1333 | − /* If PTE changed after we locked the folio them start over */ | |
| 1329 | + /* If PTE changed after we locked the folio then start over */ | |
| 1334 | 1330 | if (src_folio && unlikely(!pte_same(src_folio_pte, orig_src_pte))) { |
| 1335 | 1331 | ret = -EAGAIN; |
| 1336 | 1332 | goto out; |
| @@ -381,7 +381,7 @@ again: · 380 unmodified lines | ||
| 381 | 381 | fput(vp->file); |
| 382 | 382 | } |
| 383 | 383 | if (vp->remove->anon_vma) |
| 384 | − anon_vma_merge(vp->vma, vp->remove); | |
| 384 | + unlink_anon_vmas(vp->remove); | |
| 385 | 385 | mm->map_count--; |
| 386 | 386 | mpol_put(vma_policy(vp->remove)); |
| 387 | 387 | if (!vp->remove2) |
| @@ -530,7 +530,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, · 142 unmodified lines | ||
| 530 | 530 | if (err) |
| 531 | 531 | goto out_free_vmi; |
| 532 | 532 | |
| 533 | − err = anon_vma_clone(new, vma); | |
| 533 | + err = anon_vma_clone(new, vma, VMA_OP_SPLIT); | |
| 534 | 534 | if (err) |
| 535 | 535 | goto out_free_mpol; |
| 536 | 536 | |
| @@ -628,7 +628,7 @@ static int dup_anon_vma(struct vm_area_struct *dst, · 91 unmodified lines | ||
| 628 | 628 | |
| 629 | 629 | vma_assert_write_locked(dst); |
| 630 | 630 | dst->anon_vma = src->anon_vma; |
| 631 | − ret = anon_vma_clone(dst, src); | |
| 631 | + ret = anon_vma_clone(dst, src, VMA_OP_MERGE_UNFAULTED); | |
| 632 | 632 | if (ret) |
| 633 | 633 | return ret; |
| 634 | 634 | |
| @@ -1901,7 +1901,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, · 1266 unmodified lines | ||
| 1901 | 1901 | vma_set_range(new_vma, addr, addr + len, pgoff); |
| 1902 | 1902 | if (vma_dup_policy(vma, new_vma)) |
| 1903 | 1903 | goto out_free_vma; |
| 1904 | − if (anon_vma_clone(new_vma, vma)) | |
| 1904 | + if (anon_vma_clone(new_vma, vma, VMA_OP_REMAP)) | |
| 1905 | 1905 | goto out_free_mempol; |
| 1906 | 1906 | if (new_vma->vm_file) |
| 1907 | 1907 | get_file(new_vma->vm_file); |
| @@ -2951,10 +2951,10 @@ retry: · 1043 unmodified lines | ||
| 2951 | 2951 | return -ENOMEM; |
| 2952 | 2952 | |
| 2953 | 2953 | /* |
| 2954 | − * Adjust for the gap first so it doesn't interfere with the | |
| 2955 | − * later alignment. The first step is the minimum needed to | |
| 2956 | − * fulill the start gap, the next steps is the minimum to align | |
| 2957 | − * that. It is the minimum needed to fulill both. | |
| 2954 | + * Adjust for the gap first so it doesn't interfere with the later | |
| 2955 | + * alignment. The first step is the minimum needed to fulfill the start | |
| 2956 | + * gap, the next step is the minimum to align that. It is the minimum | |
| 2957 | + * needed to fulfill both. | |
| 2958 | 2958 | */ |
| 2959 | 2959 | gap = vma_iter_addr(&vmi) + info->start_gap; |
| 2960 | 2960 | gap += (info->align_offset - gap) & info->align_mask; |
| @@ -267,7 +267,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma, · 266 unmodified lines | ||
| 267 | 267 | struct vm_area_struct *prev, struct vm_area_struct *next); |
| 268 | 268 | |
| 269 | 269 | /** |
| 270 | − * vma_modify_flags() - Peform any necessary split/merge in preparation for | |
| 270 | + * vma_modify_flags() - Perform any necessary split/merge in preparation for | |
| 271 | 271 | * setting VMA flags to *@vm_flags in the range @start to @end contained within |
| 272 | 272 | * @vma. |
| 273 | 273 | * @vmi: Valid VMA iterator positioned at @vma. |
| @@ -295,7 +295,7 @@ __must_check struct vm_area_struct *vma_modify_flags(struct vma_iterator *vmi, · 21 unmodified lines | ||
| 295 | 295 | vm_flags_t *vm_flags_ptr); |
| 296 | 296 | |
| 297 | 297 | /** |
| 298 | − * vma_modify_name() - Peform any necessary split/merge in preparation for | |
| 298 | + * vma_modify_name() - Perform any necessary split/merge in preparation for | |
| 299 | 299 | * setting anonymous VMA name to @new_name in the range @start to @end contained |
| 300 | 300 | * within @vma. |
| 301 | 301 | * @vmi: Valid VMA iterator positioned at @vma. |
| @@ -319,7 +319,7 @@ __must_check struct vm_area_struct *vma_modify_name(struct vma_iterator *vmi, · 17 unmodified lines | ||
| 319 | 319 | struct anon_vma_name *new_name); |
| 320 | 320 | |
| 321 | 321 | /** |
| 322 | − * vma_modify_policy() - Peform any necessary split/merge in preparation for | |
| 322 | + * vma_modify_policy() - Perform any necessary split/merge in preparation for | |
| 323 | 323 | * setting NUMA policy to @new_pol in the range @start to @end contained |
| 324 | 324 | * within @vma. |
| 325 | 325 | * @vmi: Valid VMA iterator positioned at @vma. |
| @@ -343,7 +343,7 @@ __must_check struct vm_area_struct *vma_modify_policy(struct vma_iterator *vmi, · 17 unmodified lines | ||
| 343 | 343 | struct mempolicy *new_pol); |
| 344 | 344 | |
| 345 | 345 | /** |
| 346 | − * vma_modify_flags_uffd() - Peform any necessary split/merge in preparation for | |
| 346 | + * vma_modify_flags_uffd() - Perform any necessary split/merge in preparation for | |
| 347 | 347 | * setting VMA flags to @vm_flags and UFFD context to @new_ctx in the range |
| 348 | 348 | * @start to @end contained within @vma. |
| 349 | 349 | * @vmi: Valid VMA iterator positioned at @vma. |
| @@ -561,12 +561,6 @@ static inline unsigned long vma_iter_end(struct vma_iterator *vmi) · 211 unmodified lines | ||
| 561 | 561 | return vmi->mas.last + 1; |
| 562 | 562 | } |
| 563 | 563 | |
| 564 | −static inline int vma_iter_bulk_alloc(struct vma_iterator *vmi, | |
| 565 | − unsigned long count) | |
| 566 | −{ | |
| 567 | − return mas_expected_entries(&vmi->mas, count); | |
| 568 | −} | |
| 569 | − | |
| 570 | 564 | static inline |
| 571 | 565 | struct vm_area_struct *vma_iter_prev_range(struct vma_iterator *vmi) |
| 572 | 566 | { |
| @@ -108,7 +108,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, · 107 unmodified lines | ||
| 108 | 108 | if (!pte) |
| 109 | 109 | return -ENOMEM; |
| 110 | 110 | |
| 111 | − arch_enter_lazy_mmu_mode(); | |
| 111 | + lazy_mmu_mode_enable(); | |
| 112 | 112 | |
| 113 | 113 | do { |
| 114 | 114 | if (unlikely(!pte_none(ptep_get(pte)))) { |
| @@ -134,7 +134,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, · 19 unmodified lines | ||
| 134 | 134 | pfn++; |
| 135 | 135 | } while (pte += PFN_DOWN(size), addr += size, addr != end); |
| 136 | 136 | |
| 137 | − arch_leave_lazy_mmu_mode(); | |
| 137 | + lazy_mmu_mode_disable(); | |
| 138 | 138 | *mask |= PGTBL_PTE_MODIFIED; |
| 139 | 139 | return 0; |
| 140 | 140 | } |
| @@ -305,6 +305,11 @@ static int vmap_range_noflush(unsigned long addr, unsigned long end, · 164 unmodified lines | ||
| 305 | 305 | int err; |
| 306 | 306 | pgtbl_mod_mask mask = 0; |
| 307 | 307 | |
| 308 | + /* | |
| 309 | + * Might allocate pagetables (for most archs a more precise annotation | |
| 310 | + * would be might_alloc(GFP_PGTABLE_KERNEL)). Also might shootdown TLB | |
| 311 | + * (requires IRQs enabled on x86). | |
| 312 | + */ | |
| 308 | 313 | might_sleep(); |
| 309 | 314 | BUG_ON(addr >= end); |
| 310 | 315 | |
| @@ -366,7 +371,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, · 55 unmodified lines | ||
| 366 | 371 | unsigned long size = PAGE_SIZE; |
| 367 | 372 | |
| 368 | 373 | pte = pte_offset_kernel(pmd, addr); |
| 369 | − arch_enter_lazy_mmu_mode(); | |
| 374 | + lazy_mmu_mode_enable(); | |
| 370 | 375 | |
| 371 | 376 | do { |
| 372 | 377 | #ifdef CONFIG_HUGETLB_PAGE |
| @@ -385,7 +390,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, · 12 unmodified lines | ||
| 385 | 390 | WARN_ON(!pte_none(ptent) && !pte_present(ptent)); |
| 386 | 391 | } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); |
| 387 | 392 | |
| 388 | − arch_leave_lazy_mmu_mode(); | |
| 393 | + lazy_mmu_mode_disable(); | |
| 389 | 394 | *mask |= PGTBL_PTE_MODIFIED; |
| 390 | 395 | } |
| 391 | 396 | |
| @@ -533,7 +538,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, · 141 unmodified lines | ||
| 533 | 538 | if (!pte) |
| 534 | 539 | return -ENOMEM; |
| 535 | 540 | |
| 536 | − arch_enter_lazy_mmu_mode(); | |
| 541 | + lazy_mmu_mode_enable(); | |
| 537 | 542 | |
| 538 | 543 | do { |
| 539 | 544 | struct page *page = pages[*nr]; |
| @@ -555,7 +560,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, · 15 unmodified lines | ||
| 555 | 560 | (*nr)++; |
| 556 | 561 | } while (pte++, addr += PAGE_SIZE, addr != end); |
| 557 | 562 | |
| 558 | − arch_leave_lazy_mmu_mode(); | |
| 563 | + lazy_mmu_mode_disable(); | |
| 559 | 564 | *mask |= PGTBL_PTE_MODIFIED; |
| 560 | 565 | |
| 561 | 566 | return err; |
| @@ -2268,11 +2273,14 @@ decay_va_pool_node(struct vmap_node *vn, bool full_decay) · 1706 unmodified lines | ||
| 2268 | 2273 | reclaim_list_global(&decay_list); |
| 2269 | 2274 | } |
| 2270 | 2275 | |
| 2276 | +#define KASAN_RELEASE_BATCH_SIZE 32 | |
| 2277 | + | |
| 2271 | 2278 | static void |
| 2272 | 2279 | kasan_release_vmalloc_node(struct vmap_node *vn) |
| 2273 | 2280 | { |
| 2274 | 2281 | struct vmap_area *va; |
| 2275 | 2282 | unsigned long start, end; |
| 2283 | + unsigned int batch_count = 0; | |
| 2276 | 2284 | |
| 2277 | 2285 | start = list_first_entry(&vn->purge_list, struct vmap_area, list)->va_start; |
| 2278 | 2286 | end = list_last_entry(&vn->purge_list, struct vmap_area, list)->va_end; |
| @@ -2282,6 +2290,11 @@ kasan_release_vmalloc_node(struct vmap_node *vn) · 3 unmodified lines | ||
| 2282 | 2290 | kasan_release_vmalloc(va->va_start, va->va_end, |
| 2283 | 2291 | va->va_start, va->va_end, |
| 2284 | 2292 | KASAN_VMALLOC_PAGE_RANGE); |
| 2293 | + | |
| 2294 | + if (need_resched() || (++batch_count >= KASAN_RELEASE_BATCH_SIZE)) { | |
| 2295 | + cond_resched(); | |
| 2296 | + batch_count = 0; | |
| 2297 | + } | |
| 2285 | 2298 | } |
| 2286 | 2299 | |
| 2287 | 2300 | kasan_release_vmalloc(start, end, start, end, KASAN_VMALLOC_TLB_FLUSH); |
| @@ -4354,6 +4367,7 @@ need_realloc: · 2066 unmodified lines | ||
| 4354 | 4367 | |
| 4355 | 4368 | return n; |
| 4356 | 4369 | } |
| 4370 | +EXPORT_SYMBOL(vrealloc_node_align_noprof); | |
| 4357 | 4371 | |
| 4358 | 4372 | #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) |
| 4359 | 4373 | #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL) |
| @@ -63,7 +63,6 @@ · 62 unmodified lines | ||
| 63 | 63 | #include <asm/div64.h> |
| 64 | 64 | |
| 65 | 65 | #include <linux/swapops.h> |
| 66 | −#include <linux/balloon_compaction.h> | |
| 67 | 66 | #include <linux/sched/sysctl.h> |
| 68 | 67 | |
| 69 | 68 | #include "internal.h" |
| @@ -104,13 +103,13 @@ struct scan_control { · 34 unmodified lines | ||
| 104 | 103 | unsigned int force_deactivate:1; |
| 105 | 104 | unsigned int skipped_deactivate:1; |
| 106 | 105 | |
| 107 | − /* Writepage batching in laptop mode; RECLAIM_WRITE */ | |
| 106 | + /* zone_reclaim_mode, boost reclaim */ | |
| 108 | 107 | unsigned int may_writepage:1; |
| 109 | 108 | |
| 110 | − /* Can mapped folios be reclaimed? */ | |
| 109 | + /* zone_reclaim_mode */ | |
| 111 | 110 | unsigned int may_unmap:1; |
| 112 | 111 | |
| 113 | − /* Can folios be swapped as part of reclaim? */ | |
| 112 | + /* zome_reclaim_mode, boost reclaim, cgroup restrictions */ | |
| 114 | 113 | unsigned int may_swap:1; |
| 115 | 114 | |
| 116 | 115 | /* Not allow cache_trim_mode to be turned on as part of reclaim? */ |
| @@ -507,7 +506,7 @@ static bool skip_throttle_noprogress(pg_data_t *pgdat) · 390 unmodified lines | ||
| 507 | 506 | * If kswapd is disabled, reschedule if necessary but do not |
| 508 | 507 | * throttle as the system is likely near OOM. |
| 509 | 508 | */ |
| 510 | − if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES) | |
| 509 | + if (kswapd_test_hopeless(pgdat)) | |
| 511 | 510 | return true; |
| 512 | 511 | |
| 513 | 512 | /* |
| @@ -758,10 +757,9 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, · 244 unmodified lines | ||
| 758 | 757 | |
| 759 | 758 | if (reclaimed && !mapping_exiting(mapping)) |
| 760 | 759 | shadow = workingset_eviction(folio, target_memcg); |
| 761 | − __swap_cache_del_folio(ci, folio, swap, shadow); | |
| 762 | 760 | memcg1_swapout(folio, swap); |
| 761 | + __swap_cache_del_folio(ci, folio, swap, shadow); | |
| 763 | 762 | swap_cluster_unlock_irq(ci); |
| 764 | − put_swap_folio(folio, swap); | |
| 765 | 763 | } else { |
| 766 | 764 | void (*free_folio)(struct folio *); |
| 767 | 765 | |
| @@ -1063,7 +1061,7 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask) · 295 unmodified lines | ||
| 1063 | 1061 | /* |
| 1064 | 1062 | * We can "enter_fs" for swap-cache with only __GFP_IO |
| 1065 | 1063 | * providing this isn't SWP_FS_OPS. |
| 1066 | − * ->flags can be updated non-atomicially (scan_swap_map_slots), | |
| 1064 | + * ->flags can be updated non-atomically (scan_swap_map_slots), | |
| 1067 | 1065 | * but that will never affect SWP_FS_OPS, so the data_race |
| 1068 | 1066 | * is safe. |
| 1069 | 1067 | */ |
| @@ -1276,58 +1274,58 @@ retry: · 206 unmodified lines | ||
| 1276 | 1274 | * Try to allocate it some swap space here. |
| 1277 | 1275 | * Lazyfree folio could be freed directly |
| 1278 | 1276 | */ |
| 1279 | − if (folio_test_anon(folio) && folio_test_swapbacked(folio)) { | |
| 1280 | − if (!folio_test_swapcache(folio)) { | |
| 1281 | − if (!(sc->gfp_mask & __GFP_IO)) | |
| 1282 | − goto keep_locked; | |
| 1283 | − if (folio_maybe_dma_pinned(folio)) | |
| 1284 | − goto keep_locked; | |
| 1285 | − if (folio_test_large(folio)) { | |
| 1286 | − /* cannot split folio, skip it */ | |
| 1287 | − if (folio_expected_ref_count(folio) != | |
| 1288 | − folio_ref_count(folio) - 1) | |
| 1289 | − goto activate_locked; | |
| 1290 | − /* | |
| 1291 | − * Split partially mapped folios right away. | |
| 1292 | − * We can free the unmapped pages without IO. | |
| 1293 | − */ | |
| 1294 | − if (data_race(!list_empty(&folio->_deferred_list) && | |
| 1295 | − folio_test_partially_mapped(folio)) && | |
| 1296 | − split_folio_to_list(folio, folio_list)) | |
| 1297 | − goto activate_locked; | |
| 1298 | − } | |
| 1299 | − if (folio_alloc_swap(folio)) { | |
| 1300 | − int __maybe_unused order = folio_order(folio); | |
| 1301 | − | |
| 1302 | − if (!folio_test_large(folio)) | |
| 1303 | − goto activate_locked_split; | |
| 1304 | − /* Fallback to swap normal pages */ | |
| 1305 | − if (split_folio_to_list(folio, folio_list)) | |
| 1306 | − goto activate_locked; | |
| 1307 | −#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 1308 | − if (nr_pages >= HPAGE_PMD_NR) { | |
| 1309 | − count_memcg_folio_events(folio, | |
| 1310 | − THP_SWPOUT_FALLBACK, 1); | |
| 1311 | − count_vm_event(THP_SWPOUT_FALLBACK); | |
| 1312 | − } | |
| 1313 | −#endif | |
| 1314 | − count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK); | |
| 1315 | − if (folio_alloc_swap(folio)) | |
| 1316 | − goto activate_locked_split; | |
| 1317 | − } | |
| 1277 | + if (folio_test_anon(folio) && folio_test_swapbacked(folio) && | |
| 1278 | + !folio_test_swapcache(folio)) { | |
| 1279 | + if (!(sc->gfp_mask & __GFP_IO)) | |
| 1280 | + goto keep_locked; | |
| 1281 | + if (folio_maybe_dma_pinned(folio)) | |
| 1282 | + goto keep_locked; | |
| 1283 | + if (folio_test_large(folio)) { | |
| 1284 | + /* cannot split folio, skip it */ | |
| 1285 | + if (folio_expected_ref_count(folio) != | |
| 1286 | + folio_ref_count(folio) - 1) | |
| 1287 | + goto activate_locked; | |
| 1318 | 1288 | /* |
| 1319 | − * Normally the folio will be dirtied in unmap because its | |
| 1320 | − * pte should be dirty. A special case is MADV_FREE page. The | |
| 1321 | − * page's pte could have dirty bit cleared but the folio's | |
| 1322 | − * SwapBacked flag is still set because clearing the dirty bit | |
| 1323 | − * and SwapBacked flag has no lock protected. For such folio, | |
| 1324 | − * unmap will not set dirty bit for it, so folio reclaim will | |
| 1325 | − * not write the folio out. This can cause data corruption when | |
| 1326 | − * the folio is swapped in later. Always setting the dirty flag | |
| 1327 | − * for the folio solves the problem. | |
| 1289 | + * Split partially mapped folios right away. | |
| 1290 | + * We can free the unmapped pages without IO. | |
| 1328 | 1291 | */ |
| 1329 | − folio_mark_dirty(folio); | |
| 1292 | + if (data_race(!list_empty(&folio->_deferred_list) && | |
| 1293 | + folio_test_partially_mapped(folio)) && | |
| 1294 | + split_folio_to_list(folio, folio_list)) | |
| 1295 | + goto activate_locked; | |
| 1330 | 1296 | } |
| 1297 | + if (folio_alloc_swap(folio)) { | |
| 1298 | + int __maybe_unused order = folio_order(folio); | |
| 1299 | + | |
| 1300 | + if (!folio_test_large(folio)) | |
| 1301 | + goto activate_locked_split; | |
| 1302 | + /* Fallback to swap normal pages */ | |
| 1303 | + if (split_folio_to_list(folio, folio_list)) | |
| 1304 | + goto activate_locked; | |
| 1305 | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 1306 | + if (nr_pages >= HPAGE_PMD_NR) { | |
| 1307 | + count_memcg_folio_events(folio, | |
| 1308 | + THP_SWPOUT_FALLBACK, 1); | |
| 1309 | + count_vm_event(THP_SWPOUT_FALLBACK); | |
| 1310 | + } | |
| 1311 | +#endif | |
| 1312 | + count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK); | |
| 1313 | + if (folio_alloc_swap(folio)) | |
| 1314 | + goto activate_locked_split; | |
| 1315 | + } | |
| 1316 | + /* | |
| 1317 | + * Normally the folio will be dirtied in unmap because | |
| 1318 | + * its pte should be dirty. A special case is MADV_FREE | |
| 1319 | + * page. The page's pte could have dirty bit cleared but | |
| 1320 | + * the folio's SwapBacked flag is still set because | |
| 1321 | + * clearing the dirty bit and SwapBacked flag has no | |
| 1322 | + * lock protected. For such folio, unmap will not set | |
| 1323 | + * dirty bit for it, so folio reclaim will not write the | |
| 1324 | + * folio out. This can cause data corruption when the | |
| 1325 | + * folio is swapped in later. Always setting the dirty | |
| 1326 | + * flag for the folio solves the problem. | |
| 1327 | + */ | |
| 1328 | + folio_mark_dirty(folio); | |
| 1331 | 1329 | } |
| 1332 | 1330 | |
| 1333 | 1331 | /* |
| @@ -2451,9 +2449,9 @@ static inline void calculate_pressure_balance(struct scan_control *sc, · 1117 unmodified lines | ||
| 2451 | 2449 | static unsigned long apply_proportional_protection(struct mem_cgroup *memcg, |
| 2452 | 2450 | struct scan_control *sc, unsigned long scan) |
| 2453 | 2451 | { |
| 2454 | − unsigned long min, low; | |
| 2452 | + unsigned long min, low, usage; | |
| 2455 | 2453 | |
| 2456 | − mem_cgroup_protection(sc->target_mem_cgroup, memcg, &min, &low); | |
| 2454 | + mem_cgroup_protection(sc->target_mem_cgroup, memcg, &min, &low, &usage); | |
| 2457 | 2455 | |
| 2458 | 2456 | if (min || low) { |
| 2459 | 2457 | /* |
| @@ -2485,7 +2483,6 @@ static unsigned long apply_proportional_protection(struct mem_cgroup *memcg, · 25 unmodified lines | ||
| 2485 | 2483 | * again by how much of the total memory used is under |
| 2486 | 2484 | * hard protection. |
| 2487 | 2485 | */ |
| 2488 | − unsigned long cgroup_size = mem_cgroup_size(memcg); | |
| 2489 | 2486 | unsigned long protection; |
| 2490 | 2487 | |
| 2491 | 2488 | /* memory.low scaling, make sure we retry before OOM */ |
| @@ -2497,9 +2494,9 @@ static unsigned long apply_proportional_protection(struct mem_cgroup *memcg, · 5 unmodified lines | ||
| 2497 | 2494 | } |
| 2498 | 2495 | |
| 2499 | 2496 | /* Avoid TOCTOU with earlier protection check */ |
| 2500 | − cgroup_size = max(cgroup_size, protection); | |
| 2497 | + usage = max(usage, protection); | |
| 2501 | 2498 | |
| 2502 | − scan -= scan * protection / (cgroup_size + 1); | |
| 2499 | + scan -= scan * protection / (usage + 1); | |
| 2503 | 2500 | |
| 2504 | 2501 | /* |
| 2505 | 2502 | * Minimally target SWAP_CLUSTER_MAX pages to keep |
| @@ -3516,7 +3513,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, · 1010 unmodified lines | ||
| 3516 | 3513 | return false; |
| 3517 | 3514 | } |
| 3518 | 3515 | |
| 3519 | − arch_enter_lazy_mmu_mode(); | |
| 3516 | + lazy_mmu_mode_enable(); | |
| 3520 | 3517 | restart: |
| 3521 | 3518 | for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { |
| 3522 | 3519 | unsigned long pfn; |
| @@ -3557,7 +3554,7 @@ restart: · 34 unmodified lines | ||
| 3557 | 3554 | if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end)) |
| 3558 | 3555 | goto restart; |
| 3559 | 3556 | |
| 3560 | − arch_leave_lazy_mmu_mode(); | |
| 3557 | + lazy_mmu_mode_disable(); | |
| 3561 | 3558 | pte_unmap_unlock(pte, ptl); |
| 3562 | 3559 | |
| 3563 | 3560 | return suitable_to_scan(total, young); |
| @@ -3598,7 +3595,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area · 34 unmodified lines | ||
| 3598 | 3595 | if (!spin_trylock(ptl)) |
| 3599 | 3596 | goto done; |
| 3600 | 3597 | |
| 3601 | − arch_enter_lazy_mmu_mode(); | |
| 3598 | + lazy_mmu_mode_enable(); | |
| 3602 | 3599 | |
| 3603 | 3600 | do { |
| 3604 | 3601 | unsigned long pfn; |
| @@ -3645,7 +3642,7 @@ next: · 40 unmodified lines | ||
| 3645 | 3642 | |
| 3646 | 3643 | walk_update_folio(walk, last, gen, dirty); |
| 3647 | 3644 | |
| 3648 | − arch_leave_lazy_mmu_mode(); | |
| 3645 | + lazy_mmu_mode_disable(); | |
| 3649 | 3646 | spin_unlock(ptl); |
| 3650 | 3647 | done: |
| 3651 | 3648 | *first = -1; |
| @@ -4244,7 +4241,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) · 592 unmodified lines | ||
| 4244 | 4241 | } |
| 4245 | 4242 | } |
| 4246 | 4243 | |
| 4247 | − arch_enter_lazy_mmu_mode(); | |
| 4244 | + lazy_mmu_mode_enable(); | |
| 4248 | 4245 | |
| 4249 | 4246 | pte -= (addr - start) / PAGE_SIZE; |
| 4250 | 4247 | |
| @@ -4278,7 +4275,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) · 27 unmodified lines | ||
| 4278 | 4275 | |
| 4279 | 4276 | walk_update_folio(walk, last, gen, dirty); |
| 4280 | 4277 | |
| 4281 | − arch_leave_lazy_mmu_mode(); | |
| 4278 | + lazy_mmu_mode_disable(); | |
| 4282 | 4279 | |
| 4283 | 4280 | /* feedback from rmap walkers to page table walkers */ |
| 4284 | 4281 | if (mm_state && suitable_to_scan(i, young)) |
| @@ -5067,7 +5064,7 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * · 782 unmodified lines | ||
| 5067 | 5064 | blk_finish_plug(&plug); |
| 5068 | 5065 | done: |
| 5069 | 5066 | if (sc->nr_reclaimed > reclaimed) |
| 5070 | − atomic_set(&pgdat->kswapd_failures, 0); | |
| 5067 | + kswapd_try_clear_hopeless(pgdat, sc->order, sc->reclaim_idx); | |
| 5071 | 5068 | } |
| 5072 | 5069 | |
| 5073 | 5070 | /****************************************************************************** |
| @@ -5417,7 +5414,7 @@ static int lru_gen_seq_show(struct seq_file *m, void *v) · 343 unmodified lines | ||
| 5417 | 5414 | if (memcg) |
| 5418 | 5415 | cgroup_path(memcg->css.cgroup, m->private, PATH_MAX); |
| 5419 | 5416 | #endif |
| 5420 | − seq_printf(m, "memcg %5hu %s\n", mem_cgroup_id(memcg), path); | |
| 5417 | + seq_printf(m, "memcg %llu %s\n", mem_cgroup_id(memcg), path); | |
| 5421 | 5418 | } |
| 5422 | 5419 | |
| 5423 | 5420 | seq_printf(m, " node %5d\n", nid); |
| @@ -5502,7 +5499,7 @@ static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct scan_co · 78 unmodified lines | ||
| 5502 | 5499 | return -EINTR; |
| 5503 | 5500 | } |
| 5504 | 5501 | |
| 5505 | −static int run_cmd(char cmd, int memcg_id, int nid, unsigned long seq, | |
| 5502 | +static int run_cmd(char cmd, u64 memcg_id, int nid, unsigned long seq, | |
| 5506 | 5503 | struct scan_control *sc, int swappiness, unsigned long opt) |
| 5507 | 5504 | { |
| 5508 | 5505 | struct lruvec *lruvec; |
| @@ -5513,14 +5510,7 @@ static int run_cmd(char cmd, int memcg_id, int nid, unsigned long seq, · 4 unmodified lines | ||
| 5513 | 5510 | return -EINVAL; |
| 5514 | 5511 | |
| 5515 | 5512 | if (!mem_cgroup_disabled()) { |
| 5516 | − rcu_read_lock(); | |
| 5517 | − | |
| 5518 | − memcg = mem_cgroup_from_id(memcg_id); | |
| 5519 | − if (!mem_cgroup_tryget(memcg)) | |
| 5520 | − memcg = NULL; | |
| 5521 | − | |
| 5522 | − rcu_read_unlock(); | |
| 5523 | − | |
| 5513 | + memcg = mem_cgroup_get_from_id(memcg_id); | |
| 5524 | 5514 | if (!memcg) |
| 5525 | 5515 | return -EINVAL; |
| 5526 | 5516 | } |
| @@ -5592,7 +5582,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, · 65 unmodified lines | ||
| 5592 | 5582 | int n; |
| 5593 | 5583 | int end; |
| 5594 | 5584 | char cmd, swap_string[5]; |
| 5595 | − unsigned int memcg_id; | |
| 5585 | + u64 memcg_id; | |
| 5596 | 5586 | unsigned int nid; |
| 5597 | 5587 | unsigned long seq; |
| 5598 | 5588 | unsigned int swappiness; |
| @@ -5602,7 +5592,7 @@ static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, · 3 unmodified lines | ||
| 5602 | 5592 | if (!*cur) |
| 5603 | 5593 | continue; |
| 5604 | 5594 | |
| 5605 | − n = sscanf(cur, "%c %u %u %lu %n %4s %n %lu %n", &cmd, &memcg_id, &nid, | |
| 5595 | + n = sscanf(cur, "%c %llu %u %lu %n %4s %n %lu %n", &cmd, &memcg_id, &nid, | |
| 5606 | 5596 | &seq, &end, swap_string, &end, &opt, &end); |
| 5607 | 5597 | if (n < 4 || cur[end]) { |
| 5608 | 5598 | err = -EINVAL; |
| @@ -6141,7 +6131,7 @@ again: · 532 unmodified lines | ||
| 6141 | 6131 | * successful direct reclaim run will revive a dormant kswapd. |
| 6142 | 6132 | */ |
| 6143 | 6133 | if (reclaimable) |
| 6144 | − atomic_set(&pgdat->kswapd_failures, 0); | |
| 6134 | + kswapd_try_clear_hopeless(pgdat, sc->order, sc->reclaim_idx); | |
| 6145 | 6135 | else if (sc->cache_trim_mode) |
| 6146 | 6136 | sc->cache_trim_mode_failed = 1; |
| 6147 | 6137 | } |
| @@ -6366,13 +6356,6 @@ retry: · 218 unmodified lines | ||
| 6366 | 6356 | |
| 6367 | 6357 | if (sc->compaction_ready) |
| 6368 | 6358 | break; |
| 6369 | − | |
| 6370 | − /* | |
| 6371 | − * If we're getting trouble reclaiming, start doing | |
| 6372 | − * writepage even in laptop mode. | |
| 6373 | − */ | |
| 6374 | − if (sc->priority < DEF_PRIORITY - 2) | |
| 6375 | − sc->may_writepage = 1; | |
| 6376 | 6359 | } while (--sc->priority >= 0); |
| 6377 | 6360 | |
| 6378 | 6361 | last_pgdat = NULL; |
| @@ -6453,7 +6436,7 @@ static bool allow_direct_reclaim(pg_data_t *pgdat) · 74 unmodified lines | ||
| 6453 | 6436 | int i; |
| 6454 | 6437 | bool wmark_ok; |
| 6455 | 6438 | |
| 6456 | − if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES) | |
| 6439 | + if (kswapd_test_hopeless(pgdat)) | |
| 6457 | 6440 | return true; |
| 6458 | 6441 | |
| 6459 | 6442 | for_each_managed_zone_pgdat(zone, pgdat, i, ZONE_NORMAL) { |
| @@ -6581,7 +6564,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order, · 121 unmodified lines | ||
| 6581 | 6564 | .order = order, |
| 6582 | 6565 | .nodemask = nodemask, |
| 6583 | 6566 | .priority = DEF_PRIORITY, |
| 6584 | − .may_writepage = !laptop_mode, | |
| 6567 | + .may_writepage = 1, | |
| 6585 | 6568 | .may_unmap = 1, |
| 6586 | 6569 | .may_swap = 1, |
| 6587 | 6570 | }; |
| @@ -6625,7 +6608,7 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, · 37 unmodified lines | ||
| 6625 | 6608 | struct scan_control sc = { |
| 6626 | 6609 | .nr_to_reclaim = SWAP_CLUSTER_MAX, |
| 6627 | 6610 | .target_mem_cgroup = memcg, |
| 6628 | − .may_writepage = !laptop_mode, | |
| 6611 | + .may_writepage = 1, | |
| 6629 | 6612 | .may_unmap = 1, |
| 6630 | 6613 | .reclaim_idx = MAX_NR_ZONES - 1, |
| 6631 | 6614 | .may_swap = !noswap, |
| @@ -6671,7 +6654,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, · 39 unmodified lines | ||
| 6671 | 6654 | .reclaim_idx = MAX_NR_ZONES - 1, |
| 6672 | 6655 | .target_mem_cgroup = memcg, |
| 6673 | 6656 | .priority = DEF_PRIORITY, |
| 6674 | − .may_writepage = !laptop_mode, | |
| 6657 | + .may_writepage = 1, | |
| 6675 | 6658 | .may_unmap = 1, |
| 6676 | 6659 | .may_swap = !!(reclaim_options & MEMCG_RECLAIM_MAY_SWAP), |
| 6677 | 6660 | .proactive = !!(reclaim_options & MEMCG_RECLAIM_PROACTIVE), |
| @@ -6862,7 +6845,7 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, · 184 unmodified lines | ||
| 6862 | 6845 | wake_up_all(&pgdat->pfmemalloc_wait); |
| 6863 | 6846 | |
| 6864 | 6847 | /* Hopeless node, leave it to direct reclaim */ |
| 6865 | − if (atomic_read(&pgdat->kswapd_failures) >= MAX_RECLAIM_RETRIES) | |
| 6848 | + if (kswapd_test_hopeless(pgdat)) | |
102 more lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 357 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 162 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 261 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.