Pull MM updates from Andrew Morton: - In the series "mm: Avoid possible overflows in dirty throttling" Jan Kara addresses a couple of issues in the writeback throttling code. These fixes are also targetted at -stable kernels. - Ryusuke Konishi's series "nilfs2: fix potential issues related to reserved inodes" does that. This should actually be in the mm-nonmm-stable tree, along with the many other nilfs2 patches. My bad. - More folio conversions from Kefeng Wang in the series "mm: convert to folio_alloc_mpol()" - Kemeng Shi has sent some cleanups to the writeback code in the series "Add helper functions to remove repeated code and improve readability of cgroup writeback" - Kairui Song has made the swap code a little smaller and a little faster in the series "mm/swap: clean up and optimize swap cache index". - In the series "mm/memory: cleanly support zeropage in vm_insert_page*(), vm_map_pages*() and vmf_insert_mixed()" David Hildenbrand has reworked the rather sketchy handling of the use of the zeropage in MAP_SHARED mappings. I don't see any runtime effects here - more a cleanup/understandability/maintainablity thing. - Dev Jain has improved selftests/mm/va_high_addr_switch.c's handling of higher addresses, for aarch64. The (poorly named) series is "Restructure va_high_addr_switch". - The core TLB handling code gets some cleanups and possible slight optimizations in Bang Li's series "Add update_mmu_tlb_range() to simplify code". - Jane Chu has improved the handling of our fake-an-unrecoverable-memory-error testing feature MADV_HWPOISON in the series "Enhance soft hwpoison handling and injection". - Jeff Johnson has sent a billion patches everywhere to add MODULE_DESCRIPTION() to everything. Some landed in this pull. - In the series "mm: cleanup MIGRATE_SYNC_NO_COPY mode", Kefeng Wang has simplified migration's use of hardware-offload memory copying. - Yosry Ahmed performs more folio API conversions in his series "mm: zswap: trivial folio conversions". - In the series "large folios swap-in: handle refault cases first", Chuanhua Han inches us forward in the handling of large pages in the swap code. This is a cleanup and optimization, working toward the end objective of full support of large folio swapin/out. - In the series "mm,swap: cleanup VMA based swap readahead window calculation", Huang Ying has contributed some cleanups and a possible fixlet to his VMA based swap readahead code. - In the series "add mTHP support for anonymous shmem" Baolin Wang has taught anonymous shmem mappings to use multisize THP. By default this is a no-op - users must opt in vis sysfs controls. Dramatic improvements in pagefault latency are realized. - David Hildenbrand has some cleanups to our remaining use of page_mapcount() in the series "fs/proc: move page_mapcount() to fs/proc/internal.h". - David also has some highmem accounting cleanups in the series "mm/highmem: don't track highmem pages manually". - Build-time fixes and cleanups from John Hubbard in the series "cleanups, fixes, and progress towards avoiding "make headers"". - Cleanups and consolidation of the core pagemap handling from Barry Song in the series "mm: introduce pmd|pte_needs_soft_dirty_wp helpers and utilize them". - Lance Yang's series "Reclaim lazyfree THP without splitting" has reduced the latency of the reclaim of pmd-mapped THPs under fairly common circumstances. A 10x speedup is seen in a microbenchmark. It does this by punting to aother CPU but I guess that's a win unless all CPUs are pegged. - hugetlb_cgroup cleanups from Xiu Jianfeng in the series "mm/hugetlb_cgroup: rework on cftypes". - Miaohe Lin's series "Some cleanups for memory-failure" does just that thing. - Someone other than SeongJae has developed a DAMON feature in Honggyu Kim's series "DAMON based tiered memory management for CXL memory". This adds DAMON features which may be used to help determine the efficiency of our placement of CXL/PCIe attached DRAM. - DAMON user API centralization and simplificatio work in SeongJae Park's series "mm/damon: introduce DAMON parameters online commit function". - In the series "mm: page_type, zsmalloc and page_mapcount_reset()" David Hildenbrand does some maintenance work on zsmalloc - partially modernizing its use of pageframe fields. - Kefeng Wang provides more folio conversions in the series "mm: remove page_maybe_dma_pinned() and page_mkclean()". - More cleanup from David Hildenbrand, this time in the series "mm/memory_hotplug: use PageOffline() instead of PageReserved() for !ZONE_DEVICE". It "enlightens memory hotplug more about PageOffline() pages" and permits the removal of some virtio-mem hacks. - Barry Song's series "mm: clarify folio_add_new_anon_rmap() and __folio_add_anon_rmap()" is a cleanup to the anon folio handling in preparation for mTHP (multisize THP) swapin. - Kefeng Wang's series "mm: improve clear and copy user folio" implements more folio conversions, this time in the area of large folio userspace copying. - The series "Docs/mm/damon/maintaier-profile: document a mailing tool and community meetup series" tells people how to get better involved with other DAMON developers. From SeongJae Park. - A large series ("kmsan: Enable on s390") from Ilya Leoshkevich does that. - David Hildenbrand sends along more cleanups, this time against the migration code. The series is "mm/migrate: move NUMA hinting fault folio isolation + checks under PTL". - Jan Kara has found quite a lot of strangenesses and minor errors in the readahead code. He addresses this in the series "mm: Fix various readahead quirks". - SeongJae Park's series "selftests/damon: test DAMOS tried regions and {min,max}_nr_regions" adds features and addresses errors in DAMON's self testing code. - Gavin Shan has found a userspace-triggerable WARN in the pagecache code. The series "mm/filemap: Limit page cache size to that supported by xarray" addresses this. The series is marked cc:stable. - Chengming Zhou's series "mm/ksm: cmp_and_merge_page() optimizations and cleanup" cleans up and slightly optimizes KSM. - Roman Gushchin has separated the memcg-v1 and memcg-v2 code - lots of code motion. The series (which also makes the memcg-v1 code Kconfigurable) are "mm: memcg: separate legacy cgroup v1 code and put under config option" and "mm: memcg: put cgroup v1-specific memcg data under CONFIG_MEMCG_V1" - Dan Schatzberg's series "Add swappiness argument to memory.reclaim" adds an additional feature to this cgroup-v2 control file. - The series "Userspace controls soft-offline pages" from Jiaqi Yan permits userspace to stop the kernel's automatic treatment of excessive correctable memory errors. In order to permit userspace to monitor and handle this situation. - Kefeng Wang's series "mm: migrate: support poison recover from migrate folio" teaches the kernel to appropriately handle migration from poisoned source folios rather than simply panicing. - SeongJae Park's series "Docs/damon: minor fixups and improvements" does those things. - In the series "mm/zsmalloc: change back to per-size_class lock" Chengming Zhou improves zsmalloc's scalability and memory utilization. - Vivek Kasireddy's series "mm/gup: Introduce memfd_pin_folios() for pinning memfd folios" makes the GUP code use FOLL_PIN rather than bare refcount increments. So these paes can first be moved aside if they reside in the movable zone or a CMA block. - Andrii Nakryiko has added a binary ioctl()-based API to /proc/pid/maps for much faster reading of vma information. The series is "query VMAs from /proc/<pid>/maps". - In the series "mm: introduce per-order mTHP split counters" Lance Yang improves the kernel's presentation of developer information related to multisize THP splitting. - Michael Ellerman has developed the series "Reimplement huge pages without hugepd on powerpc (8xx, e500, book3s/64)". This permits userspace to use all available huge page sizes. - In the series "revert unconditional slab and page allocator fault injection calls" Vlastimil Babka removes a performance-affecting and not very useful feature from slab fault injection. * tag 'mm-stable-2024-07-21-14-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (411 commits) mm/mglru: fix ineffective protection calculation mm/zswap: fix a white space issue mm/hugetlb: fix kernel NULL pointer dereference when migrating hugetlb folio mm/hugetlb: fix possible recursive locking detected warning mm/gup: clear the LRU flag of a page before adding to LRU batch mm/numa_balancing: teach mpol_to_str about the balancing mode mm: memcg1: convert charge move flags to unsigned long long alloc_tag: fix page_ext_get/page_ext_put sequence during page splitting lib: reuse page_ext_data() to obtain codetag_ref lib: add missing newline character in the warning message mm/mglru: fix overshooting shrinker memory mm/mglru: fix div-by-zero in vmpressure_calc_level() mm/kmemleak: replace strncpy() with strscpy() mm, page_alloc: put should_fail_alloc_page() back behing CONFIG_FAIL_PAGE_ALLOC mm, slab: put should_failslab() back behind CONFIG_SHOULD_FAILSLAB mm: ignore data-race in __swap_writepage hugetlbfs: ensure generic_hugetlb_get_unmapped_area() returns higher address than mmap_min_addr mm: shmem: rename mTHP shmem counters mm: swap_state: use folio_alloc_mpol() in __read_swap_cache_async() mm/migrate: putback split folios when numa hint migration fails ...
Unified · Split
| @@ -155,6 +155,12 @@ Contact: SeongJae Park <sj@kernel.org> · 154 unmodified lines | ||
| 155 | 155 | Description: Writing to and reading from this file sets and gets the action |
| 156 | 156 | of the scheme. |
| 157 | 157 | |
| 158 | +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/target_nid | |
| 159 | +Date: Jun 2024 | |
| 160 | +Contact: SeongJae Park <sj@kernel.org> | |
| 161 | +Description: Action's target NUMA node id. Supported by only relevant | |
| 162 | + actions. | |
| 163 | + | |
| 158 | 164 | What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/apply_interval_us |
| 159 | 165 | Date: Sep 2023 |
| 160 | 166 | Contact: SeongJae Park <sj@kernel.org> |
| @@ -1306,17 +1306,10 @@ PAGE_SIZE multiple when read back. · 1305 unmodified lines | ||
| 1306 | 1306 | This is a simple interface to trigger memory reclaim in the |
| 1307 | 1307 | target cgroup. |
| 1308 | 1308 | |
| 1309 | − This file accepts a single key, the number of bytes to reclaim. | |
| 1310 | − No nested keys are currently supported. | |
| 1311 | − | |
| 1312 | 1309 | Example:: |
| 1313 | 1310 | |
| 1314 | 1311 | echo "1G" > memory.reclaim |
| 1315 | 1312 | |
| 1316 | − The interface can be later extended with nested keys to | |
| 1317 | − configure the reclaim behavior. For example, specify the | |
| 1318 | − type of memory to reclaim from (anon, file, ..). | |
| 1319 | − | |
| 1320 | 1313 | Please note that the kernel can over or under reclaim from |
| 1321 | 1314 | the target cgroup. If less bytes are reclaimed than the |
| 1322 | 1315 | specified amount, -EAGAIN is returned. |
| @@ -1328,6 +1321,17 @@ PAGE_SIZE multiple when read back. · 5 unmodified lines | ||
| 1328 | 1321 | This means that the networking layer will not adapt based on |
| 1329 | 1322 | reclaim induced by memory.reclaim. |
| 1330 | 1323 | |
| 1324 | +The following nested keys are defined. | |
| 1325 | + | |
| 1326 | + ========== ================================ | |
| 1327 | + swappiness Swappiness value to reclaim with | |
| 1328 | + ========== ================================ | |
| 1329 | + | |
| 1330 | + Specifying a swappiness value instructs the kernel to perform | |
| 1331 | + the reclaim with that swappiness value. Note that this has the | |
| 1332 | + same semantics as vm.swappiness applied to memcg reclaim with | |
| 1333 | + all the existing limitations and potential future extensions. | |
| 1334 | + | |
| 1331 | 1335 | memory.peak |
| 1332 | 1336 | A read-only single value file which exists on non-root |
| 1333 | 1337 | cgroups. |
| @@ -7239,9 +7239,12 @@ · 7238 unmodified lines | ||
| 7239 | 7239 | |
| 7240 | 7240 | vmalloc=nn[KMG] [KNL,BOOT,EARLY] Forces the vmalloc area to have an |
| 7241 | 7241 | exact size of <nn>. This can be used to increase |
| 7242 | − the minimum size (128MB on x86). It can also be | |
| 7243 | − used to decrease the size and leave more room | |
| 7244 | − for directly mapped kernel RAM. | |
| 7242 | + the minimum size (128MB on x86, arm32 platforms). | |
| 7243 | + It can also be used to decrease the size and leave more room | |
| 7244 | + for directly mapped kernel RAM. Note that this parameter does | |
| 7245 | + not exist on many other platforms (including arm64, alpha, | |
| 7246 | + loongarch, arc, csky, hexagon, microblaze, mips, nios2, openrisc, | |
| 7247 | + parisc, m64k, powerpc, riscv, sh, um, xtensa, s390, sparc). | |
| 7245 | 7248 | |
| 7246 | 7249 | vmcp_cma=nn[MG] [KNL,S390,EARLY] |
| 7247 | 7250 | Sets the memory size reserved for contiguous memory |
| @@ -34,18 +34,56 @@ detail) of DAMON, you should ensure :doc:`sysfs </filesystems/sysfs>` is · 33 unmodified lines | ||
| 34 | 34 | mounted. |
| 35 | 35 | |
| 36 | 36 | |
| 37 | +Snapshot Data Access Patterns | |
| 38 | +============================= | |
| 39 | + | |
| 40 | +The commands below show the memory access pattern of a program at the moment of | |
| 41 | +the execution. :: | |
| 42 | + | |
| 43 | + $ git clone https://github.com/sjp38/masim; cd masim; make | |
| 44 | + $ sudo damo start "./masim ./configs/stairs.cfg --quiet" | |
| 45 | + $ sudo ./damo show | |
| 46 | + 0 addr [85.541 TiB , 85.541 TiB ) (57.707 MiB ) access 0 % age 10.400 s | |
| 47 | + 1 addr [85.541 TiB , 85.542 TiB ) (413.285 MiB) access 0 % age 11.400 s | |
| 48 | + 2 addr [127.649 TiB , 127.649 TiB) (57.500 MiB ) access 0 % age 1.600 s | |
| 49 | + 3 addr [127.649 TiB , 127.649 TiB) (32.500 MiB ) access 0 % age 500 ms | |
| 50 | + 4 addr [127.649 TiB , 127.649 TiB) (9.535 MiB ) access 100 % age 300 ms | |
| 51 | + 5 addr [127.649 TiB , 127.649 TiB) (8.000 KiB ) access 60 % age 0 ns | |
| 52 | + 6 addr [127.649 TiB , 127.649 TiB) (6.926 MiB ) access 0 % age 1 s | |
| 53 | + 7 addr [127.998 TiB , 127.998 TiB) (120.000 KiB) access 0 % age 11.100 s | |
| 54 | + 8 addr [127.998 TiB , 127.998 TiB) (8.000 KiB ) access 40 % age 100 ms | |
| 55 | + 9 addr [127.998 TiB , 127.998 TiB) (4.000 KiB ) access 0 % age 11 s | |
| 56 | + total size: 577.590 MiB | |
| 57 | + $ sudo ./damo stop | |
| 58 | + | |
| 59 | +The first command of the above example downloads and builds an artificial | |
| 60 | +memory access generator program called ``masim``. The second command asks DAMO | |
| 61 | +to execute the artificial generator process start via the given command and | |
| 62 | +make DAMON monitors the generator process. The third command retrieves the | |
| 63 | +current snapshot of the monitored access pattern of the process from DAMON and | |
| 64 | +shows the pattern in a human readable format. | |
| 65 | + | |
| 66 | +Each line of the output shows which virtual address range (``addr [XX, XX)``) | |
| 67 | +of the process is how frequently (``access XX %``) accessed for how long time | |
| 68 | +(``age XX``). For example, the fifth region of ~9 MiB size is being most | |
| 69 | +frequently accessed for last 300 milliseconds. Finally, the fourth command | |
| 70 | +stops DAMON. | |
| 71 | + | |
| 72 | +Note that DAMON can monitor not only virtual address spaces but multiple types | |
| 73 | +of address spaces including the physical address space. | |
| 74 | + | |
| 75 | + | |
| 37 | 76 | Recording Data Access Patterns |
| 38 | 77 | ============================== |
| 39 | 78 | |
| 40 | 79 | The commands below record the memory access patterns of a program and save the |
| 41 | 80 | monitoring results to a file. :: |
| 42 | 81 | |
| 43 | − $ git clone https://github.com/sjp38/masim | |
| 44 | − $ cd masim; make; ./masim ./configs/zigzag.cfg & | |
| 82 | + $ ./masim ./configs/zigzag.cfg & | |
| 45 | 83 | $ sudo damo record -o damon.data $(pidof masim) |
| 46 | 84 | |
| 47 | −The first two lines of the commands download an artificial memory access | |
| 48 | −generator program and run it in the background. The generator will repeatedly | |
| 85 | +The line of the commands run the artificial memory access | |
| 86 | +generator program again. The generator will repeatedly | |
| 49 | 87 | access two 100 MiB sized memory regions one by one. You can substitute this |
| 50 | 88 | with your real workload. The last line asks ``damo`` to record the access |
| 51 | 89 | pattern in the ``damon.data`` file. |
| @@ -78,7 +78,7 @@ comma (","). · 77 unmodified lines | ||
| 78 | 78 | │ │ │ │ │ │ │ │ ... |
| 79 | 79 | │ │ │ │ │ │ ... |
| 80 | 80 | │ │ │ │ │ :ref:`schemes <sysfs_schemes>`/nr_schemes |
| 81 | − │ │ │ │ │ │ :ref:`0 <sysfs_scheme>`/action,apply_interval_us | |
| 81 | + │ │ │ │ │ │ :ref:`0 <sysfs_scheme>`/action,target_nid,apply_interval_us | |
| 82 | 82 | │ │ │ │ │ │ │ :ref:`access_pattern <sysfs_access_pattern>`/ |
| 83 | 83 | │ │ │ │ │ │ │ │ sz/min,max |
| 84 | 84 | │ │ │ │ │ │ │ │ nr_accesses/min,max |
| @@ -289,14 +289,18 @@ schemes/<N>/ · 204 unmodified lines | ||
| 289 | 289 | ------------ |
| 290 | 290 | |
| 291 | 291 | In each scheme directory, five directories (``access_pattern``, ``quotas``, |
| 292 | −``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and two files | |
| 293 | −(``action`` and ``apply_interval``) exist. | |
| 292 | +``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and three files | |
| 293 | +(``action``, ``target_nid`` and ``apply_interval``) exist. | |
| 294 | 294 | |
| 295 | 295 | The ``action`` file is for setting and getting the scheme's :ref:`action |
| 296 | 296 | <damon_design_damos_action>`. The keywords that can be written to and read |
| 297 | 297 | from the file and their meaning are same to those of the list on |
| 298 | 298 | :ref:`design doc <damon_design_damos_action>`. |
| 299 | 299 | |
| 300 | +The ``target_nid`` file is for setting the migration target node, which is | |
| 301 | +only meaningful when the ``action`` is either ``migrate_hot`` or | |
| 302 | +``migrate_cold``. | |
| 303 | + | |
| 300 | 304 | The ``apply_interval_us`` file is for setting and getting the scheme's |
| 301 | 305 | :ref:`apply_interval <damon_design_damos>` in microseconds. |
| 302 | 306 | |
| @@ -118,7 +118,7 @@ Short descriptions to the page flags · 117 unmodified lines | ||
| 118 | 118 | 21 - KSM |
| 119 | 119 | Identical memory pages dynamically shared between one or more processes. |
| 120 | 120 | 22 - THP |
| 121 | − Contiguous pages which construct transparent hugepages. | |
| 121 | + Contiguous pages which construct THP of any size and mapped by any granularity. | |
| 122 | 122 | 23 - OFFLINE |
| 123 | 123 | The page is logically offline. |
| 124 | 124 | 24 - ZERO_PAGE |
| @@ -173,27 +173,6 @@ LRU related page flags · 48 unmodified lines | ||
| 173 | 173 | The page-types tool in the tools/mm directory can be used to query the |
| 174 | 174 | above flags. |
| 175 | 175 | |
| 176 | −Using pagemap to do something useful | |
| 177 | −==================================== | |
| 178 | − | |
| 179 | −The general procedure for using pagemap to find out about a process' memory | |
| 180 | −usage goes like this: | |
| 181 | − | |
| 182 | − 1. Read ``/proc/pid/maps`` to determine which parts of the memory space are | |
| 183 | − mapped to what. | |
| 184 | − 2. Select the maps you are interested in -- all of them, or a particular | |
| 185 | − library, or the stack or the heap, etc. | |
| 186 | − 3. Open ``/proc/pid/pagemap`` and seek to the pages you would like to examine. | |
| 187 | − 4. Read a u64 for each page from pagemap. | |
| 188 | − 5. Open ``/proc/kpagecount`` and/or ``/proc/kpageflags``. For each PFN you | |
| 189 | − just read, seek to that entry in the file, and read the data you want. | |
| 190 | − | |
| 191 | −For example, to find the "unique set size" (USS), which is the amount of | |
| 192 | −memory that a process is using that is not shared with any other process, | |
| 193 | −you can go through every map in the process, find the PFNs, look those up | |
| 194 | −in kpagecount, and tally up the number of pages that are only referenced | |
| 195 | −once. | |
| 196 | − | |
| 197 | 176 | Exceptions for Shared Memory |
| 198 | 177 | ============================ |
| 199 | 178 | |
| @@ -252,7 +231,7 @@ Following flags about pages are currently supported: · 52 unmodified lines | ||
| 252 | 231 | - ``PAGE_IS_PRESENT`` - Page is present in the memory |
| 253 | 232 | - ``PAGE_IS_SWAPPED`` - Page is in swapped |
| 254 | 233 | - ``PAGE_IS_PFNZERO`` - Page has zero PFN |
| 255 | −- ``PAGE_IS_HUGE`` - Page is THP or Hugetlb backed | |
| 234 | +- ``PAGE_IS_HUGE`` - Page is PMD-mapped THP or Hugetlb backed | |
| 256 | 235 | - ``PAGE_IS_SOFT_DIRTY`` - Page is soft-dirty |
| 257 | 236 | |
| 258 | 237 | The ``struct pm_scan_arg`` is used as the argument of the IOCTL. |
| @@ -202,12 +202,11 @@ PMD-mappable transparent hugepage:: · 201 unmodified lines | ||
| 202 | 202 | |
| 203 | 203 | cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size |
| 204 | 204 | |
| 205 | −khugepaged will be automatically started when one or more hugepage | |
| 206 | −sizes are enabled (either by directly setting "always" or "madvise", | |
| 207 | −or by setting "inherit" while the top-level enabled is set to "always" | |
| 208 | −or "madvise"), and it'll be automatically shutdown when the last | |
| 209 | −hugepage size is disabled (either by directly setting "never", or by | |
| 210 | −setting "inherit" while the top-level enabled is set to "never"). | |
| 205 | +khugepaged will be automatically started when PMD-sized THP is enabled | |
| 206 | +(either of the per-size anon control or the top-level control are set | |
| 207 | +to "always" or "madvise"), and it'll be automatically shutdown when | |
| 208 | +PMD-sized THP is disabled (when both the per-size anon control and the | |
| 209 | +top-level control are "never") | |
| 211 | 210 | |
| 212 | 211 | Khugepaged controls |
| 213 | 212 | ------------------- |
| @@ -332,6 +331,31 @@ deny · 118 unmodified lines | ||
| 332 | 331 | force |
| 333 | 332 | Force the huge option on for all - very useful for testing; |
| 334 | 333 | |
| 334 | +Shmem can also use "multi-size THP" (mTHP) by adding a new sysfs knob to | |
| 335 | +control mTHP allocation: | |
| 336 | +'/sys/kernel/mm/transparent_hugepage/hugepages-<size>kB/shmem_enabled', | |
| 337 | +and its value for each mTHP is essentially consistent with the global | |
| 338 | +setting. An 'inherit' option is added to ensure compatibility with these | |
| 339 | +global settings. Conversely, the options 'force' and 'deny' are dropped, | |
| 340 | +which are rather testing artifacts from the old ages. | |
| 341 | + | |
| 342 | +always | |
| 343 | + Attempt to allocate <size> huge pages every time we need a new page; | |
| 344 | + | |
| 345 | +inherit | |
| 346 | + Inherit the top-level "shmem_enabled" value. By default, PMD-sized hugepages | |
| 347 | + have enabled="inherit" and all other hugepage sizes have enabled="never"; | |
| 348 | + | |
| 349 | +never | |
| 350 | + Do not allocate <size> huge pages; | |
| 351 | + | |
| 352 | +within_size | |
| 353 | + Only allocate <size> huge page if it will be fully within i_size. | |
| 354 | + Also respect fadvise()/madvise() hints; | |
| 355 | + | |
| 356 | +advise | |
| 357 | + Only allocate <size> huge pages if requested with fadvise()/madvise(); | |
| 358 | + | |
| 335 | 359 | Need of application restart |
| 336 | 360 | =========================== |
| 337 | 361 | |
| @@ -344,10 +368,6 @@ also applies to the regions registered in khugepaged. · 6 unmodified lines | ||
| 344 | 368 | Monitoring usage |
| 345 | 369 | ================ |
| 346 | 370 | |
| 347 | −.. note:: | |
| 348 | − Currently the below counters only record events relating to | |
| 349 | − PMD-sized THP. Events relating to other THP sizes are not included. | |
| 350 | − | |
| 351 | 371 | The number of PMD-sized anonymous transparent huge pages currently used by the |
| 352 | 372 | system is available by reading the AnonHugePages field in ``/proc/meminfo``. |
| 353 | 373 | To identify what applications are using PMD-sized anonymous transparent huge |
| @@ -392,20 +412,23 @@ thp_collapse_alloc_failed · 38 unmodified lines | ||
| 392 | 412 | the allocation. |
| 393 | 413 | |
| 394 | 414 | thp_file_alloc |
| 395 | − is incremented every time a file huge page is successfully | |
| 396 | − allocated. | |
| 415 | + is incremented every time a shmem huge page is successfully | |
| 416 | + allocated (Note that despite being named after "file", the counter | |
| 417 | + measures only shmem). | |
| 397 | 418 | |
| 398 | 419 | thp_file_fallback |
| 399 | − is incremented if a file huge page is attempted to be allocated | |
| 400 | − but fails and instead falls back to using small pages. | |
| 420 | + is incremented if a shmem huge page is attempted to be allocated | |
| 421 | + but fails and instead falls back to using small pages. (Note that | |
| 422 | + despite being named after "file", the counter measures only shmem). | |
| 401 | 423 | |
| 402 | 424 | thp_file_fallback_charge |
| 403 | − is incremented if a file huge page cannot be charged and instead | |
| 425 | + is incremented if a shmem huge page cannot be charged and instead | |
| 404 | 426 | falls back to using small pages even though the allocation was |
| 405 | − successful. | |
| 427 | + successful. (Note that despite being named after "file", the | |
| 428 | + counter measures only shmem). | |
| 406 | 429 | |
| 407 | 430 | thp_file_mapped |
| 408 | − is incremented every time a file huge page is mapped into | |
| 431 | + is incremented every time a file or shmem huge page is mapped into | |
| 409 | 432 | user address space. |
| 410 | 433 | |
| 411 | 434 | thp_split_page |
| @@ -476,6 +499,34 @@ swpout_fallback · 64 unmodified lines | ||
| 476 | 499 | Usually because failed to allocate some continuous swap space |
| 477 | 500 | for the huge page. |
| 478 | 501 | |
| 502 | +shmem_alloc | |
| 503 | + is incremented every time a shmem huge page is successfully | |
| 504 | + allocated. | |
| 505 | + | |
| 506 | +shmem_fallback | |
| 507 | + is incremented if a shmem huge page is attempted to be allocated | |
| 508 | + but fails and instead falls back to using small pages. | |
| 509 | + | |
| 510 | +shmem_fallback_charge | |
| 511 | + is incremented if a shmem huge page cannot be charged and instead | |
| 512 | + falls back to using small pages even though the allocation was | |
| 513 | + successful. | |
| 514 | + | |
| 515 | +split | |
| 516 | + is incremented every time a huge page is successfully split into | |
| 517 | + smaller orders. This can happen for a variety of reasons but a | |
| 518 | + common reason is that a huge page is old and is being reclaimed. | |
| 519 | + | |
| 520 | +split_failed | |
| 521 | + is incremented if kernel fails to split huge | |
| 522 | + page. This can happen if the page was pinned by somebody. | |
| 523 | + | |
| 524 | +split_deferred | |
| 525 | + is incremented when a huge page is put onto split queue. | |
| 526 | + This happens when a huge page is partially unmapped and splitting | |
| 527 | + it would free up some memory. Pages on split queue are going to | |
| 528 | + be split under memory pressure, if splitting is possible. | |
| 529 | + | |
| 479 | 530 | As the system ages, allocating huge pages may be expensive as the |
| 480 | 531 | system uses memory compaction to copy data around memory to free a |
| 481 | 532 | huge page for use. There are some counters in ``/proc/vmstat`` to help |
| @@ -36,6 +36,7 @@ Currently, these files are in /proc/sys/vm: · 35 unmodified lines | ||
| 36 | 36 | - dirtytime_expire_seconds |
| 37 | 37 | - dirty_writeback_centisecs |
| 38 | 38 | - drop_caches |
| 39 | +- enable_soft_offline | |
| 39 | 40 | - extfrag_threshold |
| 40 | 41 | - highmem_is_dirtyable |
| 41 | 42 | - hugetlb_shm_group |
| @@ -267,6 +268,43 @@ used:: · 225 unmodified lines | ||
| 267 | 268 | These are informational only. They do not mean that anything is wrong |
| 268 | 269 | with your system. To disable them, echo 4 (bit 2) into drop_caches. |
| 269 | 270 | |
| 271 | +enable_soft_offline | |
| 272 | +=================== | |
| 273 | +Correctable memory errors are very common on servers. Soft-offline is kernel's | |
| 274 | +solution for memory pages having (excessive) corrected memory errors. | |
| 275 | + | |
| 276 | +For different types of page, soft-offline has different behaviors / costs. | |
| 277 | + | |
| 278 | +- For a raw error page, soft-offline migrates the in-use page's content to | |
| 279 | + a new raw page. | |
| 280 | + | |
| 281 | +- For a page that is part of a transparent hugepage, soft-offline splits the | |
| 282 | + transparent hugepage into raw pages, then migrates only the raw error page. | |
| 283 | + As a result, user is transparently backed by 1 less hugepage, impacting | |
| 284 | + memory access performance. | |
| 285 | + | |
| 286 | +- For a page that is part of a HugeTLB hugepage, soft-offline first migrates | |
| 287 | + the entire HugeTLB hugepage, during which a free hugepage will be consumed | |
| 288 | + as migration target. Then the original hugepage is dissolved into raw | |
| 289 | + pages without compensation, reducing the capacity of the HugeTLB pool by 1. | |
| 290 | + | |
| 291 | +It is user's call to choose between reliability (staying away from fragile | |
| 292 | +physical memory) vs performance / capacity implications in transparent and | |
| 293 | +HugeTLB cases. | |
| 294 | + | |
| 295 | +For all architectures, enable_soft_offline controls whether to soft offline | |
| 296 | +memory pages. When set to 1, kernel attempts to soft offline the pages | |
| 297 | +whenever it thinks needed. When set to 0, kernel returns EOPNOTSUPP to | |
| 298 | +the request to soft offline the pages. Its default value is 1. | |
| 299 | + | |
| 300 | +It is worth mentioning that after setting enable_soft_offline to 0, the | |
| 301 | +following requests to soft offline pages will not be performed: | |
| 302 | + | |
| 303 | +- Request to soft offline pages from RAS Correctable Errors Collector. | |
| 304 | + | |
| 305 | +- On ARM, the request to soft offline pages from GHES driver. | |
| 306 | + | |
| 307 | +- On PARISC, the request to soft offline pages from Page Deallocation Table. | |
| 270 | 308 | |
| 271 | 309 | extfrag_threshold |
| 272 | 310 | ================= |
| @@ -132,7 +132,7 @@ CASE 1: Direct IO (DIO) · 131 unmodified lines | ||
| 132 | 132 | ----------------------- |
| 133 | 133 | There are GUP references to pages that are serving |
| 134 | 134 | as DIO buffers. These buffers are needed for a relatively short time (so they |
| 135 | −are not "long term"). No special synchronization with page_mkclean() or | |
| 135 | +are not "long term"). No special synchronization with folio_mkclean() or | |
| 136 | 136 | munmap() is provided. Therefore, flags to set at the call site are: :: |
| 137 | 137 | |
| 138 | 138 | FOLL_PIN |
| @@ -144,7 +144,7 @@ CASE 2: RDMA · 5 unmodified lines | ||
| 144 | 144 | ------------ |
| 145 | 145 | There are GUP references to pages that are serving as DMA |
| 146 | 146 | buffers. These buffers are needed for a long time ("long term"). No special |
| 147 | −synchronization with page_mkclean() or munmap() is provided. Therefore, flags | |
| 147 | +synchronization with folio_mkclean() or munmap() is provided. Therefore, flags | |
| 148 | 148 | to set at the call site are: :: |
| 149 | 149 | |
| 150 | 150 | FOLL_PIN | FOLL_LONGTERM |
| @@ -170,7 +170,7 @@ callback, simply remove the range from the device's page tables. · 19 unmodified lines | ||
| 170 | 170 | |
| 171 | 171 | Either way, as long as the driver unpins the pages upon mmu notifier callback, |
| 172 | 172 | then there is proper synchronization with both filesystem and mm |
| 173 | −(page_mkclean(), munmap(), etc). Therefore, neither flag needs to be set. | |
| 173 | +(folio_mkclean(), munmap(), etc). Therefore, neither flag needs to be set. | |
| 174 | 174 | |
| 175 | 175 | CASE 4: Pinning for struct page manipulation only |
| 176 | 176 | ------------------------------------------------- |
| @@ -196,20 +196,20 @@ INCORRECT (uses FOLL_GET calls): · 19 unmodified lines | ||
| 196 | 196 | write to the data within the pages |
| 197 | 197 | put_page() |
| 198 | 198 | |
| 199 | −page_maybe_dma_pinned(): the whole point of pinning | |
| 200 | −=================================================== | |
| 199 | +folio_maybe_dma_pinned(): the whole point of pinning | |
| 200 | +==================================================== | |
| 201 | 201 | |
| 202 | −The whole point of marking pages as "DMA-pinned" or "gup-pinned" is to be able | |
| 203 | −to query, "is this page DMA-pinned?" That allows code such as page_mkclean() | |
| 202 | +The whole point of marking folios as "DMA-pinned" or "gup-pinned" is to be able | |
| 203 | +to query, "is this folio DMA-pinned?" That allows code such as folio_mkclean() | |
| 204 | 204 | (and file system writeback code in general) to make informed decisions about |
| 205 | −what to do when a page cannot be unmapped due to such pins. | |
| 205 | +what to do when a folio cannot be unmapped due to such pins. | |
| 206 | 206 | |
| 207 | 207 | What to do in those cases is the subject of a years-long series of discussions |
| 208 | 208 | and debates (see the References at the end of this document). It's a TODO item |
| 209 | 209 | here: fill in the details once that's worked out. Meanwhile, it's safe to say |
| 210 | 210 | that having this available: :: |
| 211 | 211 | |
| 212 | − static inline bool page_maybe_dma_pinned(struct page *page) | |
| 212 | + static inline bool folio_maybe_dma_pinned(struct folio *folio) | |
| 213 | 213 | |
| 214 | 214 | ...is a prerequisite to solving the long-running gup+DMA problem. |
| 215 | 215 | |
| @@ -110,6 +110,13 @@ in the Makefile. Think of this as applying ``__no_sanitize_memory`` to every · 109 unmodified lines | ||
| 110 | 110 | function in the file or directory. Most users won't need KMSAN_SANITIZE, unless |
| 111 | 111 | their code gets broken by KMSAN (e.g. runs at early boot time). |
| 112 | 112 | |
| 113 | +KMSAN checks can also be temporarily disabled for the current task using | |
| 114 | +``kmsan_disable_current()`` and ``kmsan_enable_current()`` calls. Each | |
| 115 | +``kmsan_enable_current()`` call must be preceded by a | |
| 116 | +``kmsan_disable_current()`` call; these call pairs may be nested. One needs to | |
| 117 | +be careful with these calls, keeping the regions short and preferring other | |
| 118 | +ways to disable instrumentation, where possible. | |
| 119 | + | |
| 113 | 120 | Support |
| 114 | 121 | ======= |
| 115 | 122 | |
| @@ -338,11 +345,11 @@ Per-task KMSAN state · 222 unmodified lines | ||
| 338 | 345 | ~~~~~~~~~~~~~~~~~~~~ |
| 339 | 346 | |
| 340 | 347 | Every task_struct has an associated KMSAN task state that holds the KMSAN |
| 341 | −context (see above) and a per-task flag disallowing KMSAN reports:: | |
| 348 | +context (see above) and a per-task counter disallowing KMSAN reports:: | |
| 342 | 349 | |
| 343 | 350 | struct kmsan_context { |
| 344 | 351 | ... |
| 345 | − bool allow_reporting; | |
| 352 | + unsigned int depth; | |
| 346 | 353 | struct kmsan_context_state cstate; |
| 347 | 354 | ... |
| 348 | 355 | } |
| @@ -443,6 +443,15 @@ is not associated with a file: · 442 unmodified lines | ||
| 443 | 443 | |
| 444 | 444 | or if empty, the mapping is anonymous. |
| 445 | 445 | |
| 446 | +Starting with 6.11 kernel, /proc/PID/maps provides an alternative | |
| 447 | +ioctl()-based API that gives ability to flexibly and efficiently query and | |
| 448 | +filter individual VMAs. This interface is binary and is meant for more | |
| 449 | +efficient and easy programmatic use. `struct procmap_query`, defined in | |
| 450 | +linux/fs.h UAPI header, serves as an input/output argument to the | |
| 451 | +`PROCMAP_QUERY` ioctl() command. See comments in linus/fs.h UAPI header for | |
| 452 | +details on query semantics, supported flags, data returned, and general API | |
| 453 | +usage information. | |
| 454 | + | |
| 446 | 455 | The /proc/PID/smaps is an extension based on maps, showing the memory |
| 447 | 456 | consumption for each of the process's mappings. For each mapping (aka Virtual |
| 448 | 457 | Memory Area, or VMA) there is a series of lines such as the following:: |
| @@ -90,8 +90,6 @@ PMD Page Table Helpers · 89 unmodified lines | ||
| 90 | 90 | +---------------------------+--------------------------------------------------+ |
| 91 | 91 | | pmd_leaf | Tests a leaf mapped PMD | |
| 92 | 92 | +---------------------------+--------------------------------------------------+ |
| 93 | −| pmd_huge | Tests a HugeTLB mapped PMD | | |
| 94 | −+---------------------------+--------------------------------------------------+ | |
| 95 | 93 | | pmd_trans_huge | Tests a Transparent Huge Page (THP) at PMD | |
| 96 | 94 | +---------------------------+--------------------------------------------------+ |
| 97 | 95 | | pmd_present | Tests whether pmd_page() points to valid memory | |
| @@ -169,8 +167,6 @@ PUD Page Table Helpers · 71 unmodified lines | ||
| 169 | 167 | +---------------------------+--------------------------------------------------+ |
| 170 | 168 | | pud_leaf | Tests a leaf mapped PUD | |
| 171 | 169 | +---------------------------+--------------------------------------------------+ |
| 172 | −| pud_huge | Tests a HugeTLB mapped PUD | | |
| 173 | −+---------------------------+--------------------------------------------------+ | |
| 174 | 170 | | pud_trans_huge | Tests a Transparent Huge Page (THP) at PUD | |
| 175 | 171 | +---------------------------+--------------------------------------------------+ |
| 176 | 172 | | pud_present | Tests a valid mapped PUD | |
| @@ -16,53 +16,24 @@ called DAMON ``context``. DAMON executes each context with a kernel thread · 15 unmodified lines | ||
| 16 | 16 | called ``kdamond``. Multiple kdamonds could run in parallel, for different |
| 17 | 17 | types of monitoring. |
| 18 | 18 | |
| 19 | +To know how user-space can do the configurations and start/stop DAMON, refer to | |
| 20 | +:ref:`DAMON sysfs interface <sysfs_interface>` documentation. | |
| 21 | + | |
| 19 | 22 | |
| 20 | 23 | Overall Architecture |
| 21 | 24 | ==================== |
| 22 | 25 | |
| 23 | 26 | DAMON subsystem is configured with three layers including |
| 24 | 27 | |
| 25 | −- Operations Set: Implements fundamental operations for DAMON that depends on | |
| 26 | − the given monitoring target address-space and available set of | |
| 27 | − software/hardware primitives, | |
| 28 | −- Core: Implements core logics including monitoring overhead/accurach control | |
| 29 | − and access-aware system operations on top of the operations set layer, and | |
| 30 | −- Modules: Implements kernel modules for various purposes that provides | |
| 31 | − interfaces for the user space, on top of the core layer. | |
| 32 | − | |
| 33 | − | |
| 34 | −.. _damon_design_configurable_operations_set: | |
| 35 | − | |
| 36 | −Configurable Operations Set | |
| 37 | −--------------------------- | |
| 38 | − | |
| 39 | −For data access monitoring and additional low level work, DAMON needs a set of | |
| 40 | −implementations for specific operations that are dependent on and optimized for | |
| 41 | −the given target address space. On the other hand, the accuracy and overhead | |
| 42 | −tradeoff mechanism, which is the core logic of DAMON, is in the pure logic | |
| 43 | −space. DAMON separates the two parts in different layers, namely DAMON | |
| 44 | −Operations Set and DAMON Core Logics Layers, respectively. It further defines | |
| 45 | −the interface between the layers to allow various operations sets to be | |
| 46 | −configured with the core logic. | |
| 47 | − | |
| 48 | −Due to this design, users can extend DAMON for any address space by configuring | |
| 49 | −the core logic to use the appropriate operations set. If any appropriate set | |
| 50 | −is unavailable, users can implement one on their own. | |
| 51 | − | |
| 52 | −For example, physical memory, virtual memory, swap space, those for specific | |
| 53 | −processes, NUMA nodes, files, and backing memory devices would be supportable. | |
| 54 | −Also, if some architectures or devices supporting special optimized access | |
| 55 | −check primitives, those will be easily configurable. | |
| 56 | − | |
| 57 | − | |
| 58 | −Programmable Modules | |
| 59 | −-------------------- | |
| 60 | − | |
| 61 | −Core layer of DAMON is implemented as a framework, and exposes its application | |
| 62 | −programming interface to all kernel space components such as subsystems and | |
| 63 | −modules. For common use cases of DAMON, DAMON subsystem provides kernel | |
| 64 | −modules that built on top of the core layer using the API, which can be easily | |
| 65 | −used by the user space end users. | |
| 28 | +- :ref:`Operations Set <damon_operations_set>`: Implements fundamental | |
| 29 | + operations for DAMON that depends on the given monitoring target | |
| 30 | + address-space and available set of software/hardware primitives, | |
| 31 | +- :ref:`Core <damon_core_logic>`: Implements core logics including monitoring | |
| 32 | + overhead/accuracy control and access-aware system operations on top of the | |
| 33 | + operations set layer, and | |
| 34 | +- :ref:`Modules <damon_modules>`: Implements kernel modules for various | |
| 35 | + purposes that provides interfaces for the user space, on top of the core | |
| 36 | + layer. | |
| 66 | 37 | |
| 67 | 38 | |
| 68 | 39 | .. _damon_operations_set: |
| @@ -70,11 +41,32 @@ used by the user space end users. · 1 unmodified line | ||
| 70 | 41 | Operations Set Layer |
| 71 | 42 | ==================== |
| 72 | 43 | |
| 73 | −The monitoring operations are defined in two parts: | |
| 44 | +.. _damon_design_configurable_operations_set: | |
| 45 | + | |
| 46 | +For data access monitoring and additional low level work, DAMON needs a set of | |
| 47 | +implementations for specific operations that are dependent on and optimized for | |
| 48 | +the given target address space. For example, below two operations for access | |
| 49 | +monitoring are address-space dependent. | |
| 74 | 50 | |
| 75 | 51 | 1. Identification of the monitoring target address range for the address space. |
| 76 | 52 | 2. Access check of specific address range in the target space. |
| 77 | 53 | |
| 54 | +DAMON consolidates these implementations in a layer called DAMON Operations | |
| 55 | +Set, and defines the interface between it and the upper layer. The upper layer | |
| 56 | +is dedicated for DAMON's core logics including the mechanism for control of the | |
| 57 | +monitoring accruracy and the overhead. | |
| 58 | + | |
| 59 | +Hence, DAMON can easily be extended for any address space and/or available | |
| 60 | +hardware features by configuring the core logic to use the appropriate | |
| 61 | +operations set. If there is no available operations set for a given purpose, a | |
| 62 | +new operations set can be implemented following the interface between the | |
| 63 | +layers. | |
| 64 | + | |
| 65 | +For example, physical memory, virtual memory, swap space, those for specific | |
| 66 | +processes, NUMA nodes, files, and backing memory devices would be supportable. | |
| 67 | +Also, if some architectures or devices support special optimized access check | |
| 68 | +features, those will be easily configurable. | |
| 69 | + | |
| 78 | 70 | DAMON currently provides below three operation sets. Below two subsections |
| 79 | 71 | describe how those work. |
| 80 | 72 | |
| @@ -82,6 +74,10 @@ describe how those work. · 1 unmodified line | ||
| 82 | 74 | - fvaddr: Monitor fixed virtual address ranges |
| 83 | 75 | - paddr: Monitor the physical address space of the system |
| 84 | 76 | |
| 77 | +To know how user-space can do the configuration via :ref:`DAMON sysfs interface | |
| 78 | +<sysfs_interface>`, refer to :ref:`operations <sysfs_context>` file part of the | |
| 79 | +documentation. | |
| 80 | + | |
| 85 | 81 | |
| 86 | 82 | .. _damon_design_vaddr_target_regions_construction: |
| 87 | 83 | |
| @@ -140,9 +136,12 @@ conflict with the reclaim logic using ``PG_idle`` and ``PG_young`` page flags, · 52 unmodified lines | ||
| 140 | 136 | as Idle page tracking does. |
| 141 | 137 | |
| 142 | 138 | |
| 139 | +.. _damon_core_logic: | |
| 140 | + | |
| 143 | 141 | Core Logics |
| 144 | 142 | =========== |
| 145 | 143 | |
| 144 | +.. _damon_design_monitoring: | |
| 146 | 145 | |
| 147 | 146 | Monitoring |
| 148 | 147 | ---------- |
| @@ -152,6 +151,10 @@ monitoring attributes, ``sampling interval``, ``aggregation interval``, · 3 unmodified lines | ||
| 152 | 151 | ``update interval``, ``minimum number of regions``, and ``maximum number of |
| 153 | 152 | regions``. |
| 154 | 153 | |
| 154 | +To know how user-space can set the attributes via :ref:`DAMON sysfs interface | |
| 155 | +<sysfs_interface>`, refer to :ref:`monitoring_attrs <sysfs_monitoring_attrs>` | |
| 156 | +part of the documentation. | |
| 157 | + | |
| 155 | 158 | |
| 156 | 159 | Access Frequency Monitoring |
| 157 | 160 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| @@ -192,7 +195,7 @@ one page in the region is required to be checked. Thus, for each ``sampling · 34 unmodified lines | ||
| 192 | 195 | interval``, DAMON randomly picks one page in each region, waits for one |
| 193 | 196 | ``sampling interval``, checks whether the page is accessed meanwhile, and |
| 194 | 197 | increases the access frequency counter of the region if so. The counter is |
| 195 | −called ``nr_regions`` of the region. Therefore, the monitoring overhead is | |
| 198 | +called ``nr_accesses`` of the region. Therefore, the monitoring overhead is | |
| 196 | 199 | controllable by setting the number of regions. DAMON allows users to set the |
| 197 | 200 | minimum and the maximum number of regions for the trade-off. |
| 198 | 201 | |
| @@ -209,11 +212,18 @@ the data access pattern can be dynamically changed. This will result in low · 10 unmodified lines | ||
| 209 | 212 | monitoring quality. To keep the assumption as much as possible, DAMON |
| 210 | 213 | adaptively merges and splits each region based on their access frequency. |
| 211 | 214 | |
| 212 | −For each ``aggregation interval``, it compares the access frequencies of | |
| 213 | −adjacent regions and merges those if the frequency difference is small. Then, | |
| 214 | −after it reports and clears the aggregated access frequency of each region, it | |
| 215 | −splits each region into two or three regions if the total number of regions | |
| 216 | −will not exceed the user-specified maximum number of regions after the split. | |
| 215 | +For each ``aggregation interval``, it compares the access frequencies | |
| 216 | +(``nr_accesses``) of adjacent regions. If the difference is small, and if the | |
| 217 | +sum of the two regions' sizes is smaller than the size of total regions divided | |
| 218 | +by the ``minimum number of regions``, DAMON merges the two regions. If the | |
| 219 | +resulting number of total regions is still higher than ``maximum number of | |
| 220 | +regions``, it repeats the merging with increasing access frequenceis difference | |
| 221 | +threshold until the upper-limit of the number of regions is met, or the | |
| 222 | +threshold becomes higher than possible maximum value (``aggregation interval`` | |
| 223 | +divided by ``sampling interval``). Then, after it reports and clears the | |
| 224 | +aggregated access frequency of each region, it splits each region into two or | |
| 225 | +three regions if the total number of regions will not exceed the user-specified | |
| 226 | +maximum number of regions after the split. | |
| 217 | 227 | |
| 218 | 228 | In this way, DAMON provides its best-effort quality and minimal overhead while |
| 219 | 229 | keeping the bounds users set for their trade-off. |
| @@ -248,6 +258,11 @@ and applies it to monitoring operations-related data structures such as the · 28 unmodified lines | ||
| 248 | 258 | abstracted monitoring target memory area only for each of a user-specified time |
| 249 | 259 | interval (``update interval``). |
| 250 | 260 | |
| 261 | +User-space can get the monitoring results via DAMON sysfs interface and/or | |
| 262 | +tracepoints. For more details, please refer to the documentations for | |
| 263 | +:ref:`DAMOS tried regions <sysfs_schemes_tried_regions>` and :ref:`tracepoint`, | |
| 264 | +respectively. | |
| 265 | + | |
| 251 | 266 | |
| 252 | 267 | .. _damon_design_damos: |
| 253 | 268 | |
| @@ -288,6 +303,10 @@ the access pattern of interest, and applies the user-desired operation actions · 34 unmodified lines | ||
| 288 | 303 | to the regions, for every user-specified time interval called |
| 289 | 304 | ``apply_interval``. |
| 290 | 305 | |
| 306 | +To know how user-space can set ``apply_interval`` via :ref:`DAMON sysfs | |
| 307 | +interface <sysfs_interface>`, refer to :ref:`apply_interval_us <sysfs_scheme>` | |
| 308 | +part of the documentation. | |
| 309 | + | |
| 291 | 310 | |
| 292 | 311 | .. _damon_design_damos_action: |
| 293 | 312 | |
| @@ -325,6 +344,10 @@ that supports each action are as below. · 31 unmodified lines | ||
| 325 | 344 | Supported by ``paddr`` operations set. |
| 326 | 345 | - ``lru_deprio``: Deprioritize the region on its LRU lists. |
| 327 | 346 | Supported by ``paddr`` operations set. |
| 347 | + - ``migrate_hot``: Migrate the regions prioritizing warmer regions. | |
| 348 | + Supported by ``paddr`` operations set. | |
| 349 | + - ``migrate_cold``: Migrate the regions prioritizing colder regions. | |
| 350 | + Supported by ``paddr`` operations set. | |
| 328 | 351 | - ``stat``: Do nothing but count the statistics. |
| 329 | 352 | Supported by all operations sets. |
| 330 | 353 | |
| @@ -332,6 +355,10 @@ Applying the actions except ``stat`` to a region is considered as changing the · 1 unmodified line | ||
| 332 | 355 | region's characteristics. Hence, DAMOS resets the age of regions when any such |
| 333 | 356 | actions are applied to those. |
| 334 | 357 | |
| 358 | +To know how user-space can set the action via :ref:`DAMON sysfs interface | |
| 359 | +<sysfs_interface>`, refer to :ref:`action <sysfs_scheme>` part of the | |
| 360 | +documentation. | |
| 361 | + | |
| 335 | 362 | |
| 336 | 363 | .. _damon_design_damos_access_pattern: |
| 337 | 364 | |
| @@ -345,6 +372,10 @@ interest by setting minimum and maximum values of the three properties. If a · 7 unmodified lines | ||
| 345 | 372 | region's three properties are in the ranges, DAMOS classifies it as one of the |
| 346 | 373 | regions that the scheme is having an interest in. |
| 347 | 374 | |
| 375 | +To know how user-space can set the access pattern via :ref:`DAMON sysfs | |
| 376 | +interface <sysfs_interface>`, refer to :ref:`access_pattern | |
| 377 | +<sysfs_access_pattern>` part of the documentation. | |
| 378 | + | |
| 348 | 379 | |
| 349 | 380 | .. _damon_design_damos_quotas: |
| 350 | 381 | |
| @@ -364,6 +395,10 @@ feature called quotas. It lets users specify an upper limit of time that DAMOS · 13 unmodified lines | ||
| 364 | 395 | can use for applying the action, and/or a maximum bytes of memory regions that |
| 365 | 396 | the action can be applied within a user-specified time duration. |
| 366 | 397 | |
| 398 | +To know how user-space can set the basic quotas via :ref:`DAMON sysfs interface | |
| 399 | +<sysfs_interface>`, refer to :ref:`quotas <sysfs_quotas>` part of the | |
| 400 | +documentation. | |
| 401 | + | |
| 367 | 402 | |
| 368 | 403 | .. _damon_design_damos_quotas_prioritization: |
| 369 | 404 | |
| @@ -391,6 +426,10 @@ information to the underlying mechanism. Nevertheless, how and even whether · 21 unmodified lines | ||
| 391 | 426 | the weight will be respected are up to the underlying prioritization mechanism |
| 392 | 427 | implementation. |
| 393 | 428 | |
| 429 | +To know how user-space can set the prioritization weights via :ref:`DAMON sysfs | |
| 430 | +interface <sysfs_interface>`, refer to :ref:`weights <sysfs_quotas>` part of | |
| 431 | +the documentation. | |
| 432 | + | |
| 394 | 433 | |
| 395 | 434 | .. _damon_design_damos_quotas_auto_tuning: |
| 396 | 435 | |
| @@ -420,6 +459,10 @@ Currently, two ``target_metric`` are provided. · 23 unmodified lines | ||
| 420 | 459 | DAMOS does the measurement on its own, so only ``target_value`` need to be |
| 421 | 460 | set by users at the initial time. In other words, DAMOS does self-feedback. |
| 422 | 461 | |
| 462 | +To know how user-space can set the tuning goal metric, the target value, and/or | |
| 463 | +the current value via :ref:`DAMON sysfs interface <sysfs_interface>`, refer to | |
| 464 | +:ref:`quota goals <sysfs_schemes_quota_goals>` part of the documentation. | |
| 465 | + | |
| 423 | 466 | |
| 424 | 467 | .. _damon_design_damos_watermarks: |
| 425 | 468 | |
| @@ -442,6 +485,10 @@ is activated. If all schemes are deactivated by the watermarks, the monitoring · 16 unmodified lines | ||
| 442 | 485 | is also deactivated. In this case, the DAMON worker thread only periodically |
| 443 | 486 | checks the watermarks and therefore incurs nearly zero overhead. |
| 444 | 487 | |
| 488 | +To know how user-space can set the watermarks via :ref:`DAMON sysfs interface | |
| 489 | +<sysfs_interface>`, refer to :ref:`watermarks <sysfs_watermarks>` part of the | |
| 490 | +documentation. | |
| 491 | + | |
| 445 | 492 | |
| 446 | 493 | .. _damon_design_damos_filters: |
| 447 | 494 | |
| @@ -488,6 +535,10 @@ Below types of filters are currently supported. · 40 unmodified lines | ||
| 488 | 535 | - Applied to pages that belonging to a given DAMON monitoring target. |
| 489 | 536 | - Handled by the core logic. |
| 490 | 537 | |
| 538 | +To know how user-space can set the watermarks via :ref:`DAMON sysfs interface | |
| 539 | +<sysfs_interface>`, refer to :ref:`filters <sysfs_filters>` part of the | |
| 540 | +documentation. | |
| 541 | + | |
| 491 | 542 | |
| 492 | 543 | Application Programming Interface |
| 493 | 544 | --------------------------------- |
| @@ -501,6 +552,8 @@ interface, namely ``include/linux/damon.h``. Please refer to the API · 7 unmodified lines | ||
| 501 | 552 | :doc:`document </mm/damon/api>` for details of the interface. |
| 502 | 553 | |
| 503 | 554 | |
| 555 | +.. _damon_modules: | |
| 556 | + | |
| 504 | 557 | Modules |
| 505 | 558 | ======= |
| 506 | 559 | |
| @@ -6,7 +6,7 @@ DAMON: Data Access MONitor · 5 unmodified lines | ||
| 6 | 6 | |
| 7 | 7 | DAMON is a Linux kernel subsystem that provides a framework for data access |
| 8 | 8 | monitoring and the monitoring results based system operations. The core |
| 9 | −monitoring mechanisms of DAMON (refer to :doc:`design` for the detail) make it | |
| 9 | +monitoring :ref:`mechanisms <damon_design_monitoring>` of DAMON make it | |
| 10 | 10 | |
| 11 | 11 | - *accurate* (the monitoring output is useful enough for DRAM level memory |
| 12 | 12 | management; It might not appropriate for CPU Cache levels, though), |
| @@ -16,15 +16,16 @@ monitoring mechanisms of DAMON (refer to :doc:`design` for the detail) make it · 3 unmodified lines | ||
| 16 | 16 | of the size of target workloads). |
| 17 | 17 | |
| 18 | 18 | Using this framework, therefore, the kernel can operate system in an |
| 19 | −access-aware fashion. Because the features are also exposed to the user space, | |
| 20 | −users who have special information about their workloads can write personalized | |
| 21 | −applications for better understanding and optimizations of their workloads and | |
| 22 | −systems. | |
| 19 | +access-aware fashion. Because the features are also exposed to the :doc:`user | |
| 20 | +space </admin-guide/mm/damon/index>`, users who have special information about | |
| 21 | +their workloads can write personalized applications for better understanding | |
| 22 | +and optimizations of their workloads and systems. | |
| 23 | 23 | |
| 24 | −For easier development of such systems, DAMON provides a feature called DAMOS | |
| 25 | −(DAMon-based Operation Schemes) in addition to the monitoring. Using the | |
| 26 | −feature, DAMON users in both kernel and user spaces can do access-aware system | |
| 27 | −operations with no code but simple configurations. | |
| 24 | +For easier development of such systems, DAMON provides a feature called | |
| 25 | +:ref:`DAMOS <damon_design_damos>` (DAMon-based Operation Schemes) in addition | |
| 26 | +to the monitoring. Using the feature, DAMON users in both kernel and :doc:`user | |
| 27 | +spaces </admin-guide/mm/damon/index>` can do access-aware system operations | |
| 28 | +with no code but simple configurations. | |
| 28 | 29 | |
| 29 | 30 | .. toctree:: |
| 30 | 31 | :maxdepth: 2 |
| @@ -33,3 +34,6 @@ operations with no code but simple configurations. · 2 unmodified lines | ||
| 33 | 34 | design |
| 34 | 35 | api |
| 35 | 36 | maintainer-profile |
| 37 | + | |
| 38 | +To utilize and control DAMON from the user-space, please refer to the | |
| 39 | +administration :doc:`guide </admin-guide/mm/damon/index>`. | |
| @@ -53,6 +53,40 @@ Mon-Fri) in PT (Pacific Time). The response to patches will occasionally be · 52 unmodified lines | ||
| 53 | 53 | slow. Do not hesitate to send a ping if you have not heard back within a week |
| 54 | 54 | of sending a patch. |
| 55 | 55 | |
| 56 | +Mailing tool | |
| 57 | +------------ | |
| 58 | + | |
| 59 | +Like many other Linux kernel subsystems, DAMON uses the mailing lists | |
| 60 | +(damon@lists.linux.dev and linux-mm@kvack.org) as the major communication | |
| 61 | +channel. There is a simple tool called HacKerMaiL (``hkml``) [8]_ , which is | |
| 62 | +for people who are not very familiar with the mailing lists based | |
| 63 | +communication. The tool could be particularly helpful for DAMON community | |
| 64 | +members since it is developed and maintained by DAMON maintainer. The tool is | |
| 65 | +also officially announced to support DAMON and general Linux kernel development | |
| 66 | +workflow. | |
| 67 | + | |
| 68 | +In other words, ``hkml`` [8]_ is a mailing tool for DAMON community, which | |
| 69 | +DAMON maintainer is committed to support. Please feel free to try and report | |
| 70 | +issues or feature requests for the tool to the maintainer. | |
| 71 | + | |
| 72 | +Community meetup | |
| 73 | +---------------- | |
| 74 | + | |
| 75 | +DAMON community is maintaining two bi-weekly meetup series for community | |
| 76 | +members who prefer synchronous conversations over mails. | |
| 77 | + | |
| 78 | +The first one is for any discussion between every community member. No | |
| 79 | +reservation is needed. | |
| 80 | + | |
| 81 | +The seconds one is for discussions on specific topics between restricted | |
| 82 | +members including the maintainer. The maintainer shares the available time | |
| 83 | +slots, and attendees should reserve one of those at least 24 hours before the | |
| 84 | +time slot, by reaching out to the maintainer. | |
| 85 | + | |
| 86 | +Schedules and available reservation time slots are available at the Google doc | |
| 87 | +[9]_ . DAMON maintainer will also provide periodic reminder to the mailing | |
| 88 | +list (damon@lists.linux.dev). | |
| 89 | + | |
| 56 | 90 | |
| 57 | 91 | .. [1] https://git.kernel.org/akpm/mm/h/mm-unstable |
| 58 | 92 | .. [2] https://git.kernel.org/sj/h/damon/next |
| @@ -61,3 +95,5 @@ of sending a patch. · 2 unmodified lines | ||
| 61 | 95 | .. [5] https://github.com/awslabs/damon-tests/blob/master/corr/tests/kunit.sh |
| 62 | 96 | .. [6] https://github.com/awslabs/damon-tests/tree/master/corr |
| 63 | 97 | .. [7] https://github.com/awslabs/damon-tests/tree/master/perf |
| 98 | +.. [8] https://github.com/damonitor/hackermail | |
| 99 | +.. [9] https://docs.google.com/document/d/1v43Kcj3ly4CYqmAkMaZzLiM2GEnWfgdGbZAH3mi2vpM/edit?usp=sharing | |
| @@ -191,13 +191,13 @@ have become evictable again (via munlock() for example) and have been "rescued" · 190 unmodified lines | ||
| 191 | 191 | from the unevictable list. However, there may be situations where we decide, |
| 192 | 192 | for the sake of expediency, to leave an unevictable folio on one of the regular |
| 193 | 193 | active/inactive LRU lists for vmscan to deal with. vmscan checks for such |
| 194 | −folios in all of the shrink_{active|inactive|page}_list() functions and will | |
| 194 | +folios in all of the shrink_{active|inactive|folio}_list() functions and will | |
| 195 | 195 | "cull" such folios that it encounters: that is, it diverts those folios to the |
| 196 | 196 | unevictable list for the memory cgroup and node being scanned. |
| 197 | 197 | |
| 198 | 198 | There may be situations where a folio is mapped into a VM_LOCKED VMA, |
| 199 | 199 | but the folio does not have the mlocked flag set. Such folios will make |
| 200 | −it all the way to shrink_active_list() or shrink_page_list() where they | |
| 200 | +it all the way to shrink_active_list() or shrink_folio_list() where they | |
| 201 | 201 | will be detected when vmscan walks the reverse map in folio_referenced() |
| 202 | 202 | or try_to_unmap(). The folio is culled to the unevictable list when it |
| 203 | 203 | is released by the shrinker. |
| @@ -269,7 +269,7 @@ the LRU. Such pages can be "noticed" by memory management in several places: · 65 unmodified lines | ||
| 269 | 269 | |
| 270 | 270 | (4) in the fault path and when a VM_LOCKED stack segment is expanded; or |
| 271 | 271 | |
| 272 | − (5) as mentioned above, in vmscan:shrink_page_list() when attempting to | |
| 272 | + (5) as mentioned above, in vmscan:shrink_folio_list() when attempting to | |
| 273 | 273 | reclaim a page in a VM_LOCKED VMA by folio_referenced() or try_to_unmap(). |
| 274 | 274 | |
| 275 | 275 | mlocked pages become unlocked and rescued from the unevictable list when: |
| @@ -548,12 +548,12 @@ Some examples of these unevictable pages on the LRU lists are: · 272 unmodified lines | ||
| 548 | 548 | (3) pages still mapped into VM_LOCKED VMAs, which should be marked mlocked, |
| 549 | 549 | but events left mlock_count too low, so they were munlocked too early. |
| 550 | 550 | |
| 551 | −vmscan's shrink_inactive_list() and shrink_page_list() also divert obviously | |
| 551 | +vmscan's shrink_inactive_list() and shrink_folio_list() also divert obviously | |
| 552 | 552 | unevictable pages found on the inactive lists to the appropriate memory cgroup |
| 553 | 553 | and node unevictable list. |
| 554 | 554 | |
| 555 | 555 | rmap's folio_referenced_one(), called via vmscan's shrink_active_list() or |
| 556 | −shrink_page_list(), and rmap's try_to_unmap_one() called via shrink_page_list(), | |
| 556 | +shrink_folio_list(), and rmap's try_to_unmap_one() called via shrink_folio_list(), | |
| 557 | 557 | check for (3) pages still mapped into VM_LOCKED VMAs, and call mlock_vma_folio() |
| 558 | 558 | to correct them. Such pages are culled to the unevictable list when released |
| 559 | 559 | by the shrinker. |
| @@ -5701,6 +5701,8 @@ L: linux-mm@kvack.org · 5700 unmodified lines | ||
| 5701 | 5701 | S: Maintained |
| 5702 | 5702 | F: include/linux/memcontrol.h |
| 5703 | 5703 | F: mm/memcontrol.c |
| 5704 | +F: mm/memcontrol-v1.c | |
| 5705 | +F: mm/memcontrol-v1.h | |
| 5704 | 5706 | F: mm/swap_cgroup.c |
| 5705 | 5707 | F: samples/cgroup/* |
| 5706 | 5708 | F: tools/testing/selftests/cgroup/memcg_protection.m |
| @@ -283,7 +283,7 @@ void flush_cache_pages(struct vm_area_struct *vma, unsigned long user_addr, · 282 unmodified lines | ||
| 283 | 283 | * flush_dcache_page is used when the kernel has written to the page |
| 284 | 284 | * cache page at virtual address page->virtual. |
| 285 | 285 | * |
| 286 | − * If this page isn't mapped (ie, page_mapping == NULL), or it might | |
| 286 | + * If this page isn't mapped (ie, folio_mapping == NULL), or it might | |
| 287 | 287 | * have userspace mappings, then we _must_ always clean + invalidate |
| 288 | 288 | * the dcache entries associated with the kernel mapping. |
| 289 | 289 | * |
| @@ -13,12 +13,12 @@ · 12 unmodified lines | ||
| 13 | 13 | |
| 14 | 14 | /* |
| 15 | 15 | * If our huge pte is non-zero then mark the valid bit. |
| 16 | − * This allows pte_present(huge_ptep_get(ptep)) to return true for non-zero | |
| 16 | + * This allows pte_present(huge_ptep_get(mm,addr,ptep)) to return true for non-zero | |
| 17 | 17 | * ptes. |
| 18 | 18 | * (The valid bit is automatically cleared by set_pte_at for PROT_NONE ptes). |
| 19 | 19 | */ |
| 20 | 20 | #define __HAVE_ARCH_HUGE_PTEP_GET |
| 21 | −static inline pte_t huge_ptep_get(pte_t *ptep) | |
| 21 | +static inline pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) | |
| 22 | 22 | { |
| 23 | 23 | pte_t retval = *ptep; |
| 24 | 24 | if (pte_val(retval)) |
| @@ -117,7 +117,7 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *, · 116 unmodified lines | ||
| 117 | 117 | * flush_dcache_folio is used when the kernel has written to the page |
| 118 | 118 | * cache page at virtual address page->virtual. |
| 119 | 119 | * |
| 120 | − * If this page isn't mapped (ie, page_mapping == NULL), or it might | |
| 120 | + * If this page isn't mapped (ie, folio_mapping == NULL), or it might | |
| 121 | 121 | * have userspace mappings, then we _must_ always clean + invalidate |
| 122 | 122 | * the dcache entries associated with the kernel mapping. |
| 123 | 123 | * |
| @@ -46,7 +46,7 @@ extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, · 45 unmodified lines | ||
| 46 | 46 | extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, |
| 47 | 47 | pte_t *ptep, unsigned long sz); |
| 48 | 48 | #define __HAVE_ARCH_HUGE_PTEP_GET |
| 49 | −extern pte_t huge_ptep_get(pte_t *ptep); | |
| 49 | +extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); | |
| 50 | 50 | |
| 51 | 51 | void __init arm64_hugetlb_cma_reserve(void); |
| 52 | 52 | |
| @@ -127,7 +127,7 @@ static inline int num_contig_ptes(unsigned long size, size_t *pgsize) · 126 unmodified lines | ||
| 127 | 127 | return contig_ptes; |
| 128 | 128 | } |
| 129 | 129 | |
| 130 | −pte_t huge_ptep_get(pte_t *ptep) | |
| 130 | +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) | |
| 131 | 131 | { |
| 132 | 132 | int ncontig, i; |
| 133 | 133 | size_t pgsize; |
| @@ -467,8 +467,8 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, · 466 unmodified lines | ||
| 467 | 467 | #define update_mmu_cache(vma, addr, ptep) \ |
| 468 | 468 | update_mmu_cache_range(NULL, vma, addr, ptep, 1) |
| 469 | 469 | |
| 470 | −#define __HAVE_ARCH_UPDATE_MMU_TLB | |
| 471 | −#define update_mmu_tlb update_mmu_cache | |
| 470 | +#define update_mmu_tlb_range(vma, addr, ptep, nr) \ | |
| 471 | + update_mmu_cache_range(NULL, vma, addr, ptep, nr) | |
| 472 | 472 | |
| 473 | 473 | static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, |
| 474 | 474 | unsigned long address, pmd_t *pmdp) |
| @@ -594,8 +594,8 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, · 593 unmodified lines | ||
| 594 | 594 | #define update_mmu_cache(vma, address, ptep) \ |
| 595 | 595 | update_mmu_cache_range(NULL, vma, address, ptep, 1) |
| 596 | 596 | |
| 597 | −#define __HAVE_ARCH_UPDATE_MMU_TLB | |
| 598 | −#define update_mmu_tlb update_mmu_cache | |
| 597 | +#define update_mmu_tlb_range(vma, address, ptep, nr) \ | |
| 598 | + update_mmu_cache_range(NULL, vma, address, ptep, nr) | |
| 599 | 599 | |
| 600 | 600 | static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, |
| 601 | 601 | unsigned long address, pmd_t *pmdp) |
| @@ -112,7 +112,7 @@ void __flush_dcache_pages(struct page *page, unsigned int nr) · 111 unmodified lines | ||
| 112 | 112 | } |
| 113 | 113 | |
| 114 | 114 | /* |
| 115 | − * We could delay the flush for the !page_mapping case too. But that | |
| 115 | + * We could delay the flush for the !folio_mapping case too. But that | |
| 116 | 116 | * case is for exec env/arg pages and those are %99 certainly going to |
| 117 | 117 | * get faulted into the tlb (and thus flushed) anyways. |
| 118 | 118 | */ |
| @@ -135,7 +135,6 @@ config PPC · 134 unmodified lines | ||
| 135 | 135 | select ARCH_HAS_DMA_MAP_DIRECT if PPC_PSERIES |
| 136 | 136 | select ARCH_HAS_FORTIFY_SOURCE |
| 137 | 137 | select ARCH_HAS_GCOV_PROFILE_ALL |
| 138 | − select ARCH_HAS_HUGEPD if HUGETLB_PAGE | |
| 139 | 138 | select ARCH_HAS_KCOV |
| 140 | 139 | select ARCH_HAS_KERNEL_FPU_SUPPORT if PPC64 && PPC_FPU |
| 141 | 140 | select ARCH_HAS_MEMBARRIER_CALLBACKS |
| @@ -47,8 +47,6 @@ static inline void pgtable_free(void *table, unsigned index_size) · 46 unmodified lines | ||
| 47 | 47 | } |
| 48 | 48 | } |
| 49 | 49 | |
| 50 | −#define get_hugepd_cache_index(x) (x) | |
| 51 | − | |
| 52 | 50 | static inline void pgtable_free_tlb(struct mmu_gather *tlb, |
| 53 | 51 | void *table, int shift) |
| 54 | 52 | { |
| @@ -74,21 +74,6 @@ · 73 unmodified lines | ||
| 74 | 74 | #define remap_4k_pfn(vma, addr, pfn, prot) \ |
| 75 | 75 | remap_pfn_range((vma), (addr), (pfn), PAGE_SIZE, (prot)) |
| 76 | 76 | |
| 77 | −#ifdef CONFIG_HUGETLB_PAGE | |
| 78 | −static inline int hash__hugepd_ok(hugepd_t hpd) | |
| 79 | −{ | |
| 80 | − unsigned long hpdval = hpd_val(hpd); | |
| 81 | − /* | |
| 82 | − * if it is not a pte and have hugepd shift mask | |
| 83 | − * set, then it is a hugepd directory pointer | |
| 84 | − */ | |
| 85 | − if (!(hpdval & _PAGE_PTE) && (hpdval & _PAGE_PRESENT) && | |
| 86 | − ((hpdval & HUGEPD_SHIFT_MASK) != 0)) | |
| 87 | − return true; | |
| 88 | − return false; | |
| 89 | −} | |
| 90 | −#endif | |
| 91 | − | |
| 92 | 77 | /* |
| 93 | 78 | * 4K PTE format is different from 64K PTE format. Saving the hash_slot is just |
| 94 | 79 | * a matter of returning the PTE bits that need to be modified. On 64K PTE, |
| @@ -4,6 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | #ifdef __KERNEL__ |
| 5 | 5 | |
| 6 | 6 | #include <asm/asm-const.h> |
| 7 | +#include <asm/book3s/64/slice.h> | |
| 7 | 8 | |
| 8 | 9 | /* |
| 9 | 10 | * Common bits between 4K and 64K pages in a linux-style PTE. |
| @@ -161,14 +162,10 @@ extern void hpte_need_flush(struct mm_struct *mm, unsigned long addr, · 151 unmodified lines | ||
| 161 | 162 | pte_t *ptep, unsigned long pte, int huge); |
| 162 | 163 | unsigned long htab_convert_pte_flags(unsigned long pteflags, unsigned long flags); |
| 163 | 164 | /* Atomic PTE updates */ |
| 164 | −static inline unsigned long hash__pte_update(struct mm_struct *mm, | |
| 165 | − unsigned long addr, | |
| 166 | − pte_t *ptep, unsigned long clr, | |
| 167 | − unsigned long set, | |
| 168 | − int huge) | |
| 165 | +static inline unsigned long hash__pte_update_one(pte_t *ptep, unsigned long clr, | |
| 166 | + unsigned long set) | |
| 169 | 167 | { |
| 170 | 168 | __be64 old_be, tmp_be; |
| 171 | − unsigned long old; | |
| 172 | 169 | |
| 173 | 170 | __asm__ __volatile__( |
| 174 | 171 | "1: ldarx %0,0,%3 # pte_update\n\ |
| @@ -182,11 +179,40 @@ static inline unsigned long hash__pte_update(struct mm_struct *mm, · 7 unmodified lines | ||
| 182 | 179 | : "r" (ptep), "r" (cpu_to_be64(clr)), "m" (*ptep), |
| 183 | 180 | "r" (cpu_to_be64(H_PAGE_BUSY)), "r" (cpu_to_be64(set)) |
| 184 | 181 | : "cc" ); |
| 182 | + | |
| 183 | + return be64_to_cpu(old_be); | |
| 184 | +} | |
| 185 | + | |
| 186 | +static inline unsigned long hash__pte_update(struct mm_struct *mm, | |
| 187 | + unsigned long addr, | |
| 188 | + pte_t *ptep, unsigned long clr, | |
| 189 | + unsigned long set, | |
| 190 | + int huge) | |
| 191 | +{ | |
| 192 | + unsigned long old; | |
| 193 | + | |
| 194 | + old = hash__pte_update_one(ptep, clr, set); | |
| 195 | + | |
| 196 | + if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && huge) { | |
| 197 | + unsigned int psize = get_slice_psize(mm, addr); | |
| 198 | + int nb, i; | |
| 199 | + | |
| 200 | + if (psize == MMU_PAGE_16M) | |
| 201 | + nb = SZ_16M / PMD_SIZE; | |
| 202 | + else if (psize == MMU_PAGE_16G) | |
| 203 | + nb = SZ_16G / PUD_SIZE; | |
| 204 | + else | |
| 205 | + nb = 1; | |
| 206 | + | |
| 207 | + WARN_ON_ONCE(nb == 1); /* Should never happen */ | |
| 208 | + | |
| 209 | + for (i = 1; i < nb; i++) | |
| 210 | + hash__pte_update_one(ptep + i, clr, set); | |
| 211 | + } | |
| 185 | 212 | /* huge pages use the old page table lock */ |
| 186 | 213 | if (!huge) |
| 187 | 214 | assert_pte_locked(mm, addr); |
| 188 | 215 | |
| 189 | − old = be64_to_cpu(old_be); | |
| 190 | 216 | if (old & H_PAGE_HASHPTE) |
| 191 | 217 | hpte_need_flush(mm, addr, ptep, old, huge); |
| 192 | 218 | |
| @@ -49,9 +49,6 @@ static inline bool gigantic_page_runtime_supported(void) · 48 unmodified lines | ||
| 49 | 49 | return true; |
| 50 | 50 | } |
| 51 | 51 | |
| 52 | −/* hugepd entry valid bit */ | |
| 53 | −#define HUGEPD_VAL_BITS (0x8000000000000000UL) | |
| 54 | − | |
| 55 | 52 | #define huge_ptep_modify_prot_start huge_ptep_modify_prot_start |
| 56 | 53 | extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, |
| 57 | 54 | unsigned long addr, pte_t *ptep); |
| @@ -60,29 +57,7 @@ extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, · 2 unmodified lines | ||
| 60 | 57 | extern void huge_ptep_modify_prot_commit(struct vm_area_struct *vma, |
| 61 | 58 | unsigned long addr, pte_t *ptep, |
| 62 | 59 | pte_t old_pte, pte_t new_pte); |
| 63 | −/* | |
| 64 | − * This should work for other subarchs too. But right now we use the | |
| 65 | − * new format only for 64bit book3s | |
| 66 | − */ | |
| 67 | −static inline pte_t *hugepd_page(hugepd_t hpd) | |
| 68 | −{ | |
| 69 | − BUG_ON(!hugepd_ok(hpd)); | |
| 70 | − /* | |
| 71 | − * We have only four bits to encode, MMU page size | |
| 72 | − */ | |
| 73 | − BUILD_BUG_ON((MMU_PAGE_COUNT - 1) > 0xf); | |
| 74 | − return __va(hpd_val(hpd) & HUGEPD_ADDR_MASK); | |
| 75 | −} | |
| 76 | − | |
| 77 | −static inline unsigned int hugepd_mmu_psize(hugepd_t hpd) | |
| 78 | −{ | |
| 79 | − return (hpd_val(hpd) & HUGEPD_SHIFT_MASK) >> 2; | |
| 80 | −} | |
| 81 | 60 | |
| 82 | −static inline unsigned int hugepd_shift(hugepd_t hpd) | |
| 83 | −{ | |
| 84 | − return mmu_psize_to_shift(hugepd_mmu_psize(hpd)); | |
| 85 | −} | |
| 86 | 61 | static inline void flush_hugetlb_page(struct vm_area_struct *vma, |
| 87 | 62 | unsigned long vmaddr) |
| 88 | 63 | { |
| @@ -90,19 +65,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, · 1 unmodified line | ||
| 90 | 65 | return radix__flush_hugetlb_page(vma, vmaddr); |
| 91 | 66 | } |
| 92 | 67 | |
| 93 | −static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, | |
| 94 | − unsigned int pdshift) | |
| 95 | −{ | |
| 96 | − unsigned long idx = (addr & ((1UL << pdshift) - 1)) >> hugepd_shift(hpd); | |
| 97 | − | |
| 98 | − return hugepd_page(hpd) + idx; | |
| 99 | −} | |
| 100 | − | |
| 101 | −static inline void hugepd_populate(hugepd_t *hpdp, pte_t *new, unsigned int pshift) | |
| 102 | −{ | |
| 103 | − *hpdp = __hugepd(__pa(new) | HUGEPD_VAL_BITS | (shift_to_mmu_psize(pshift) << 2)); | |
| 104 | −} | |
| 105 | − | |
| 106 | 68 | void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr); |
| 107 | 69 | |
| 108 | 70 | static inline int check_and_get_huge_psize(int shift) |
| @@ -1,47 +0,0 @@ | ||
| 1 | −/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | −#ifndef _ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H | |
| 3 | −#define _ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H | |
| 4 | −/* | |
| 5 | − * hash 4k can't share hugetlb and also doesn't support THP | |
| 6 | − */ | |
| 7 | −#ifndef __ASSEMBLY__ | |
| 8 | −#ifdef CONFIG_HUGETLB_PAGE | |
| 9 | −/* | |
| 10 | − * With radix , we have hugepage ptes in the pud and pmd entries. We don't | |
| 11 | − * need to setup hugepage directory for them. Our pte and page directory format | |
| 12 | − * enable us to have this enabled. | |
| 13 | − */ | |
| 14 | −static inline int hugepd_ok(hugepd_t hpd) | |
| 15 | −{ | |
| 16 | − if (radix_enabled()) | |
| 17 | − return 0; | |
| 18 | − return hash__hugepd_ok(hpd); | |
| 19 | −} | |
| 20 | −#define is_hugepd(hpd) (hugepd_ok(hpd)) | |
| 21 | − | |
| 22 | −/* | |
| 23 | − * 16M and 16G huge page directory tables are allocated from slab cache | |
| 24 | − * | |
| 25 | − */ | |
| 26 | −#define H_16M_CACHE_INDEX (PAGE_SHIFT + H_PTE_INDEX_SIZE + H_PMD_INDEX_SIZE - 24) | |
| 27 | −#define H_16G_CACHE_INDEX \ | |
| 28 | − (PAGE_SHIFT + H_PTE_INDEX_SIZE + H_PMD_INDEX_SIZE + H_PUD_INDEX_SIZE - 34) | |
| 29 | − | |
| 30 | −static inline int get_hugepd_cache_index(int index) | |
| 31 | −{ | |
| 32 | − switch (index) { | |
| 33 | − case H_16M_CACHE_INDEX: | |
| 34 | − return HTLB_16M_INDEX; | |
| 35 | − case H_16G_CACHE_INDEX: | |
| 36 | − return HTLB_16G_INDEX; | |
| 37 | − default: | |
| 38 | − BUG(); | |
| 39 | − } | |
| 40 | − /* should not reach */ | |
| 41 | −} | |
| 42 | − | |
| 43 | −#endif /* CONFIG_HUGETLB_PAGE */ | |
| 44 | − | |
| 45 | −#endif /* __ASSEMBLY__ */ | |
| 46 | − | |
| 47 | −#endif /*_ASM_POWERPC_BOOK3S_64_PGTABLE_4K_H */ | |
| @@ -5,26 +5,6 @@ · 4 unmodified lines | ||
| 5 | 5 | #ifndef __ASSEMBLY__ |
| 6 | 6 | #ifdef CONFIG_HUGETLB_PAGE |
| 7 | 7 | |
| 8 | −/* | |
| 9 | − * With 64k page size, we have hugepage ptes in the pgd and pmd entries. We don't | |
| 10 | − * need to setup hugepage directory for them. Our pte and page directory format | |
| 11 | − * enable us to have this enabled. | |
| 12 | − */ | |
| 13 | −static inline int hugepd_ok(hugepd_t hpd) | |
| 14 | −{ | |
| 15 | − return 0; | |
| 16 | −} | |
| 17 | − | |
| 18 | −#define is_hugepd(pdep) 0 | |
| 19 | − | |
| 20 | −/* | |
| 21 | − * This should never get called | |
| 22 | − */ | |
| 23 | −static __always_inline int get_hugepd_cache_index(int index) | |
| 24 | −{ | |
| 25 | − BUILD_BUG(); | |
| 26 | −} | |
| 27 | − | |
| 28 | 8 | #endif /* CONFIG_HUGETLB_PAGE */ |
| 29 | 9 | |
| 30 | 10 | static inline int remap_4k_pfn(struct vm_area_struct *vma, unsigned long addr, |
| @@ -274,6 +274,24 @@ static inline bool pud_leaf(pud_t pud) · 273 unmodified lines | ||
| 274 | 274 | { |
| 275 | 275 | return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE)); |
| 276 | 276 | } |
| 277 | + | |
| 278 | +#define pmd_leaf_size pmd_leaf_size | |
| 279 | +static inline unsigned long pmd_leaf_size(pmd_t pmd) | |
| 280 | +{ | |
| 281 | + if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled()) | |
| 282 | + return SZ_16M; | |
| 283 | + else | |
| 284 | + return PMD_SIZE; | |
| 285 | +} | |
| 286 | + | |
| 287 | +#define pud_leaf_size pud_leaf_size | |
| 288 | +static inline unsigned long pud_leaf_size(pud_t pud) | |
| 289 | +{ | |
| 290 | + if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled()) | |
| 291 | + return SZ_16G; | |
| 292 | + else | |
| 293 | + return PUD_SIZE; | |
| 294 | +} | |
| 277 | 295 | #endif /* __ASSEMBLY__ */ |
| 278 | 296 | |
| 279 | 297 | #include <asm/book3s/64/hash.h> |
| @@ -285,11 +303,9 @@ static inline bool pud_leaf(pud_t pud) · 5 unmodified lines | ||
| 285 | 303 | #define MAX_PHYSMEM_BITS R_MAX_PHYSMEM_BITS |
| 286 | 304 | #endif |
| 287 | 305 | |
| 288 | − | |
| 306 | +/* hash 4k can't share hugetlb and also doesn't support THP */ | |
| 289 | 307 | #ifdef CONFIG_PPC_64K_PAGES |
| 290 | 308 | #include <asm/book3s/64/pgtable-64k.h> |
| 291 | −#else | |
| 292 | −#include <asm/book3s/64/pgtable-4k.h> | |
| 293 | 309 | #endif |
| 294 | 310 | |
| 295 | 311 | #include <asm/barrier.h> |
| @@ -30,10 +30,9 @@ static inline int is_hugepage_only_range(struct mm_struct *mm, · 29 unmodified lines | ||
| 30 | 30 | } |
| 31 | 31 | #define is_hugepage_only_range is_hugepage_only_range |
| 32 | 32 | |
| 33 | −#define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGE | |
| 34 | −void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr, | |
| 35 | − unsigned long end, unsigned long floor, | |
| 36 | − unsigned long ceiling); | |
| 33 | +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT | |
| 34 | +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, | |
| 35 | + pte_t pte, unsigned long sz); | |
| 37 | 36 | |
| 38 | 37 | #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR |
| 39 | 38 | static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm, |
| @@ -67,14 +66,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, · 27 unmodified lines | ||
| 67 | 66 | { |
| 68 | 67 | } |
| 69 | 68 | |
| 70 | −#define hugepd_shift(x) 0 | |
| 71 | −static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, | |
| 72 | − unsigned pdshift) | |
| 73 | −{ | |
| 74 | − return NULL; | |
| 75 | −} | |
| 76 | − | |
| 77 | − | |
| 78 | 69 | static inline void __init gigantic_hugetlb_cma_reserve(void) |
| 79 | 70 | { |
| 80 | 71 | } |
| @@ -4,42 +4,12 @@ · 3 unmodified lines | ||
| 4 | 4 | |
| 5 | 5 | #define PAGE_SHIFT_8M 23 |
| 6 | 6 | |
| 7 | −static inline pte_t *hugepd_page(hugepd_t hpd) | |
| 8 | −{ | |
| 9 | − BUG_ON(!hugepd_ok(hpd)); | |
| 10 | − | |
| 11 | − return (pte_t *)__va(hpd_val(hpd) & ~HUGEPD_SHIFT_MASK); | |
| 12 | −} | |
| 13 | − | |
| 14 | −static inline unsigned int hugepd_shift(hugepd_t hpd) | |
| 15 | −{ | |
| 16 | − return PAGE_SHIFT_8M; | |
| 17 | −} | |
| 18 | − | |
| 19 | −static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, | |
| 20 | − unsigned int pdshift) | |
| 21 | −{ | |
| 22 | − unsigned long idx = (addr & (SZ_4M - 1)) >> PAGE_SHIFT; | |
| 23 | − | |
| 24 | − return hugepd_page(hpd) + idx; | |
| 25 | −} | |
| 26 | − | |
| 27 | 7 | static inline void flush_hugetlb_page(struct vm_area_struct *vma, |
| 28 | 8 | unsigned long vmaddr) |
| 29 | 9 | { |
| 30 | 10 | flush_tlb_page(vma, vmaddr); |
| 31 | 11 | } |
| 32 | 12 | |
| 33 | −static inline void hugepd_populate(hugepd_t *hpdp, pte_t *new, unsigned int pshift) | |
| 34 | −{ | |
| 35 | − *hpdp = __hugepd(__pa(new) | _PMD_USER | _PMD_PRESENT | _PMD_PAGE_8M); | |
| 36 | −} | |
| 37 | − | |
| 38 | −static inline void hugepd_populate_kernel(hugepd_t *hpdp, pte_t *new, unsigned int pshift) | |
| 39 | −{ | |
| 40 | − *hpdp = __hugepd(__pa(new) | _PMD_PRESENT | _PMD_PAGE_8M); | |
| 41 | −} | |
| 42 | − | |
| 43 | 13 | static inline int check_and_get_huge_psize(int shift) |
| 44 | 14 | { |
| 45 | 15 | return shift_to_mmu_psize(shift); |
| @@ -49,6 +19,14 @@ static inline int check_and_get_huge_psize(int shift) · 3 unmodified lines | ||
| 49 | 19 | void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, |
| 50 | 20 | pte_t pte, unsigned long sz); |
| 51 | 21 | |
| 22 | +#define __HAVE_ARCH_HUGE_PTEP_GET | |
| 23 | +static inline pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) | |
| 24 | +{ | |
| 25 | + if (ptep_is_8m_pmdp(mm, addr, ptep)) | |
| 26 | + ptep = pte_offset_kernel((pmd_t *)ptep, ALIGN_DOWN(addr, SZ_8M)); | |
| 27 | + return ptep_get(ptep); | |
| 28 | +} | |
| 29 | + | |
| 52 | 30 | #define __HAVE_ARCH_HUGE_PTE_CLEAR |
| 53 | 31 | static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr, |
| 54 | 32 | pte_t *ptep, unsigned long sz) |
| @@ -189,19 +189,14 @@ typedef struct { · 188 unmodified lines | ||
| 189 | 189 | |
| 190 | 190 | #define PHYS_IMMR_BASE (mfspr(SPRN_IMMR) & 0xfff80000) |
| 191 | 191 | |
| 192 | −/* Page size definitions, common between 32 and 64-bit | |
| 192 | +/* | |
| 193 | + * Page size definitions for 8xx | |
| 193 | 194 | * |
| 194 | 195 | * shift : is the "PAGE_SHIFT" value for that page size |
| 195 | − * penc : is the pte encoding mask | |
| 196 | 196 | * |
| 197 | 197 | */ |
| 198 | 198 | struct mmu_psize_def { |
| 199 | 199 | unsigned int shift; /* number of bits */ |
| 200 | − unsigned int enc; /* PTE encoding */ | |
| 201 | − unsigned int ind; /* Corresponding indirect page size shift */ | |
| 202 | − unsigned int flags; | |
| 203 | −#define MMU_PAGE_SIZE_DIRECT 0x1 /* Supported as a direct size */ | |
| 204 | −#define MMU_PAGE_SIZE_INDIRECT 0x2 /* Supported as an indirect size */ | |
| 205 | 200 | }; |
| 206 | 201 | |
| 207 | 202 | extern struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT]; |
| @@ -75,9 +75,6 @@ · 74 unmodified lines | ||
| 75 | 75 | #define _PAGE_NO_CACHE 0x00000400 /* H: I bit */ |
| 76 | 76 | #define _PAGE_WRITETHRU 0x00000800 /* H: W bit */ |
| 77 | 77 | |
| 78 | −/* No page size encoding in the linux PTE */ | |
| 79 | −#define _PAGE_PSIZE 0 | |
| 80 | − | |
| 81 | 78 | /* TODO: Add large page lowmem mapping support */ |
| 82 | 79 | #define _PMD_PRESENT 0 |
| 83 | 80 | #define _PMD_PRESENT_MASK (PAGE_MASK) |
| @@ -31,9 +31,6 @@ · 30 unmodified lines | ||
| 31 | 31 | #define _PAGE_WRITETHRU 0x00400 /* H: W bit */ |
| 32 | 32 | #define _PAGE_SPECIAL 0x00800 /* S: Special page */ |
| 33 | 33 | |
| 34 | −/* No page size encoding in the linux PTE */ | |
| 35 | −#define _PAGE_PSIZE 0 | |
| 36 | − | |
| 37 | 34 | #define _PMD_PRESENT 0 |
| 38 | 35 | #define _PMD_PRESENT_MASK (PAGE_MASK) |
| 39 | 36 | #define _PMD_BAD (~PAGE_MASK) |
| @@ -74,12 +74,11 @@ · 73 unmodified lines | ||
| 74 | 74 | #define _PTE_NONE_MASK 0 |
| 75 | 75 | |
| 76 | 76 | #ifdef CONFIG_PPC_16K_PAGES |
| 77 | −#define _PAGE_PSIZE _PAGE_SPS | |
| 77 | +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_SPS) | |
| 78 | 78 | #else |
| 79 | −#define _PAGE_PSIZE 0 | |
| 79 | +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED) | |
| 80 | 80 | #endif |
| 81 | 81 | |
| 82 | −#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE) | |
| 83 | 82 | #define _PAGE_BASE (_PAGE_BASE_NC) |
| 84 | 83 | |
| 85 | 84 | #include <asm/pgtable-masks.h> |
| @@ -120,7 +119,7 @@ static inline pte_t pte_mkhuge(pte_t pte) · 34 unmodified lines | ||
| 120 | 119 | |
| 121 | 120 | #define pte_mkhuge pte_mkhuge |
| 122 | 121 | |
| 123 | −static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p, | |
| 122 | +static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, | |
| 124 | 123 | unsigned long clr, unsigned long set, int huge); |
| 125 | 124 | |
| 126 | 125 | static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) |
| @@ -142,19 +141,12 @@ static inline void __ptep_set_access_flags(struct vm_area_struct *vma, pte_t *pt · 15 unmodified lines | ||
| 142 | 141 | } |
| 143 | 142 | #define __ptep_set_access_flags __ptep_set_access_flags |
| 144 | 143 | |
| 145 | −static inline unsigned long pgd_leaf_size(pgd_t pgd) | |
| 146 | −{ | |
| 147 | − if (pgd_val(pgd) & _PMD_PAGE_8M) | |
| 148 | − return SZ_8M; | |
| 149 | − return SZ_4M; | |
| 150 | −} | |
| 151 | − | |
| 152 | −#define pgd_leaf_size pgd_leaf_size | |
| 153 | − | |
| 154 | −static inline unsigned long pte_leaf_size(pte_t pte) | |
| 144 | +static inline unsigned long __pte_leaf_size(pmd_t pmd, pte_t pte) | |
| 155 | 145 | { |
| 156 | 146 | pte_basic_t val = pte_val(pte); |
| 157 | 147 | |
| 148 | + if (pmd_val(pmd) & _PMD_PAGE_8M) | |
| 149 | + return SZ_8M; | |
| 158 | 150 | if (val & _PAGE_HUGE) |
| 159 | 151 | return SZ_512K; |
| 160 | 152 | if (val & _PAGE_SPS) |
| @@ -162,31 +154,38 @@ static inline unsigned long pte_leaf_size(pte_t pte) · 1 unmodified line | ||
| 162 | 154 | return SZ_4K; |
| 163 | 155 | } |
| 164 | 156 | |
| 165 | −#define pte_leaf_size pte_leaf_size | |
| 157 | +#define __pte_leaf_size __pte_leaf_size | |
| 166 | 158 | |
| 167 | 159 | /* |
| 168 | 160 | * On the 8xx, the page tables are a bit special. For 16k pages, we have |
| 169 | 161 | * 4 identical entries. For 512k pages, we have 128 entries as if it was |
| 170 | 162 | * 4k pages, but they are flagged as 512k pages for the hardware. |
| 171 | − * For other page sizes, we have a single entry in the table. | |
| 163 | + * For 8M pages, we have 1024 entries as if it was 4M pages (PMD_SIZE) | |
| 164 | + * but they are flagged as 8M pages for the hardware. | |
| 165 | + * For 4k pages, we have a single entry in the table. | |
| 172 | 166 | */ |
| 173 | 167 | static pmd_t *pmd_off(struct mm_struct *mm, unsigned long addr); |
| 174 | −static int hugepd_ok(hugepd_t hpd); | |
| 168 | +static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address); | |
| 169 | + | |
| 170 | +static inline bool ptep_is_8m_pmdp(struct mm_struct *mm, unsigned long addr, pte_t *ptep) | |
| 171 | +{ | |
| 172 | + return (pmd_t *)ptep == pmd_off(mm, ALIGN_DOWN(addr, SZ_8M)); | |
| 173 | +} | |
| 175 | 174 | |
| 176 | 175 | static inline int number_of_cells_per_pte(pmd_t *pmd, pte_basic_t val, int huge) |
| 177 | 176 | { |
| 178 | 177 | if (!huge) |
| 179 | 178 | return PAGE_SIZE / SZ_4K; |
| 180 | − else if (hugepd_ok(*((hugepd_t *)pmd))) | |
| 181 | − return 1; | |
| 179 | + else if ((pmd_val(*pmd) & _PMD_PAGE_MASK) == _PMD_PAGE_8M) | |
| 180 | + return SZ_4M / SZ_4K; | |
| 182 | 181 | else if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !(val & _PAGE_HUGE)) |
| 183 | 182 | return SZ_16K / SZ_4K; |
| 184 | 183 | else |
| 185 | 184 | return SZ_512K / SZ_4K; |
| 186 | 185 | } |
| 187 | 186 | |
| 188 | −static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p, | |
| 189 | − unsigned long clr, unsigned long set, int huge) | |
| 187 | +static inline pte_basic_t __pte_update(struct mm_struct *mm, unsigned long addr, pte_t *p, | |
| 188 | + unsigned long clr, unsigned long set, int huge) | |
| 190 | 189 | { |
| 191 | 190 | pte_basic_t *entry = (pte_basic_t *)p; |
| 192 | 191 | pte_basic_t old = pte_val(*p); |
| @@ -198,7 +197,7 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p · 5 unmodified lines | ||
| 198 | 197 | |
| 199 | 198 | for (i = 0; i < num; i += PAGE_SIZE / SZ_4K, new += PAGE_SIZE) { |
| 200 | 199 | *entry++ = new; |
| 201 | − if (IS_ENABLED(CONFIG_PPC_16K_PAGES) && num != 1) { | |
| 200 | + if (IS_ENABLED(CONFIG_PPC_16K_PAGES)) { | |
| 202 | 201 | *entry++ = new; |
| 203 | 202 | *entry++ = new; |
| 204 | 203 | *entry++ = new; |
| @@ -208,6 +207,21 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p · 3 unmodified lines | ||
| 208 | 207 | return old; |
| 209 | 208 | } |
| 210 | 209 | |
| 210 | +static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, | |
| 211 | + unsigned long clr, unsigned long set, int huge) | |
| 212 | +{ | |
| 213 | + pte_basic_t old; | |
| 214 | + | |
| 215 | + if (huge && ptep_is_8m_pmdp(mm, addr, ptep)) { | |
| 216 | + pmd_t *pmdp = (pmd_t *)ptep; | |
| 217 | + | |
| 218 | + old = __pte_update(mm, addr, pte_offset_kernel(pmdp, 0), clr, set, huge); | |
| 219 | + __pte_update(mm, addr, pte_offset_kernel(pmdp + 1, 0), clr, set, huge); | |
| 220 | + } else { | |
| 221 | + old = __pte_update(mm, addr, ptep, clr, set, huge); | |
| 222 | + } | |
| 223 | + return old; | |
| 224 | +} | |
| 211 | 225 | #define pte_update pte_update |
| 212 | 226 | |
| 213 | 227 | #ifdef CONFIG_PPC_16K_PAGES |
| @@ -2,38 +2,8 @@ · 1 unmodified line | ||
| 2 | 2 | #ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H |
| 3 | 3 | #define _ASM_POWERPC_NOHASH_HUGETLB_E500_H |
| 4 | 4 | |
| 5 | −static inline pte_t *hugepd_page(hugepd_t hpd) | |
| 6 | −{ | |
| 7 | − if (WARN_ON(!hugepd_ok(hpd))) | |
| 8 | − return NULL; | |
| 9 | − | |
| 10 | − return (pte_t *)((hpd_val(hpd) & ~HUGEPD_SHIFT_MASK) | PD_HUGE); | |
| 11 | −} | |
| 12 | − | |
| 13 | −static inline unsigned int hugepd_shift(hugepd_t hpd) | |
| 14 | −{ | |
| 15 | − return hpd_val(hpd) & HUGEPD_SHIFT_MASK; | |
| 16 | −} | |
| 17 | − | |
| 18 | −static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, | |
| 19 | − unsigned int pdshift) | |
| 20 | −{ | |
| 21 | − /* | |
| 22 | − * On FSL BookE, we have multiple higher-level table entries that | |
| 23 | − * point to the same hugepte. Just use the first one since they're all | |
| 24 | − * identical. So for that case, idx=0. | |
| 25 | − */ | |
| 26 | − return hugepd_page(hpd); | |
| 27 | −} | |
| 28 | − | |
| 29 | 5 | void flush_hugetlb_page(struct vm_area_struct *vma, unsigned long vmaddr); |
| 30 | 6 | |
| 31 | −static inline void hugepd_populate(hugepd_t *hpdp, pte_t *new, unsigned int pshift) | |
| 32 | −{ | |
| 33 | − /* We use the old format for PPC_E500 */ | |
| 34 | − *hpdp = __hugepd(((unsigned long)new & ~PD_HUGE) | pshift); | |
| 35 | −} | |
| 36 | − | |
| 37 | 7 | static inline int check_and_get_huge_psize(int shift) |
| 38 | 8 | { |
| 39 | 9 | if (shift & 1) /* Not a power of 4 */ |
| @@ -42,4 +12,13 @@ static inline int check_and_get_huge_psize(int shift) · 2 unmodified lines | ||
| 42 | 12 | return shift_to_mmu_psize(shift); |
| 43 | 13 | } |
| 44 | 14 | |
| 15 | +static inline pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) | |
| 16 | +{ | |
| 17 | + unsigned int tsize = shift - _PAGE_PSIZE_SHIFT_OFFSET; | |
| 18 | + pte_basic_t val = (tsize << _PAGE_PSIZE_SHIFT) & _PAGE_PSIZE_MSK; | |
| 19 | + | |
| 20 | + return __pte((pte_val(entry) & ~(pte_basic_t)_PAGE_PSIZE_MSK) | val); | |
| 21 | +} | |
| 22 | +#define arch_make_huge_pte arch_make_huge_pte | |
| 23 | + | |
| 45 | 24 | #endif /* _ASM_POWERPC_NOHASH_HUGETLB_E500_H */ |
| @@ -244,14 +244,11 @@ typedef struct { · 243 unmodified lines | ||
| 244 | 244 | /* Page size definitions, common between 32 and 64-bit |
| 245 | 245 | * |
| 246 | 246 | * shift : is the "PAGE_SHIFT" value for that page size |
| 247 | − * penc : is the pte encoding mask | |
| 248 | 247 | * |
| 249 | 248 | */ |
| 250 | 249 | struct mmu_psize_def |
| 251 | 250 | { |
| 252 | 251 | unsigned int shift; /* number of bits */ |
| 253 | − unsigned int enc; /* PTE encoding */ | |
| 254 | − unsigned int ind; /* Corresponding indirect page size shift */ | |
| 255 | 252 | unsigned int flags; |
| 256 | 253 | #define MMU_PAGE_SIZE_DIRECT 0x1 /* Supported as a direct size */ |
| 257 | 254 | #define MMU_PAGE_SIZE_INDIRECT 0x2 /* Supported as an indirect size */ |
| @@ -303,8 +300,7 @@ extern unsigned long linear_map_top; · 45 unmodified lines | ||
| 303 | 300 | extern int book3e_htw_mode; |
| 304 | 301 | |
| 305 | 302 | #define PPC_HTW_NONE 0 |
| 306 | −#define PPC_HTW_IBM 1 | |
| 307 | −#define PPC_HTW_E6500 2 | |
| 303 | +#define PPC_HTW_E6500 1 | |
| 308 | 304 | |
| 309 | 305 | /* |
| 310 | 306 | * 64-bit booke platforms don't load the tlb in the tlb miss handler code. |
| @@ -44,8 +44,6 @@ static inline void pgtable_free(void *table, int shift) · 43 unmodified lines | ||
| 44 | 44 | } |
| 45 | 45 | } |
| 46 | 46 | |
| 47 | −#define get_hugepd_cache_index(x) (x) | |
| 48 | − | |
| 49 | 47 | static inline void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift) |
| 50 | 48 | { |
| 51 | 49 | unsigned long pgf = (unsigned long)table; |
| @@ -31,6 +31,13 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p · 30 unmodified lines | ||
| 31 | 31 | |
| 32 | 32 | extern int icache_44x_need_flush; |
| 33 | 33 | |
| 34 | +#ifndef pte_huge_size | |
| 35 | +static inline unsigned long pte_huge_size(pte_t pte) | |
| 36 | +{ | |
| 37 | + return PAGE_SIZE; | |
| 38 | +} | |
| 39 | +#endif | |
| 40 | + | |
| 34 | 41 | /* |
| 35 | 42 | * PTE updates. This function is called whenever an existing |
| 36 | 43 | * valid PTE is updated. This does -not- include set_pte_at() |
| @@ -52,11 +59,34 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p · 15 unmodified lines | ||
| 52 | 59 | { |
| 53 | 60 | pte_basic_t old = pte_val(*p); |
| 54 | 61 | pte_basic_t new = (old & ~(pte_basic_t)clr) | set; |
| 62 | + unsigned long sz; | |
| 63 | + unsigned long pdsize; | |
| 64 | + int i; | |
| 55 | 65 | |
| 56 | 66 | if (new == old) |
| 57 | 67 | return old; |
| 58 | 68 | |
| 59 | − *p = __pte(new); | |
| 69 | + if (huge) | |
| 70 | + sz = pte_huge_size(__pte(old)); | |
| 71 | + else | |
| 72 | + sz = PAGE_SIZE; | |
| 73 | + | |
| 74 | + if (sz < PMD_SIZE) | |
| 75 | + pdsize = PAGE_SIZE; | |
| 76 | + else if (sz < PUD_SIZE) | |
| 77 | + pdsize = PMD_SIZE; | |
| 78 | + else if (sz < P4D_SIZE) | |
| 79 | + pdsize = PUD_SIZE; | |
| 80 | + else if (sz < PGDIR_SIZE) | |
| 81 | + pdsize = P4D_SIZE; | |
| 82 | + else | |
| 83 | + pdsize = PGDIR_SIZE; | |
| 84 | + | |
| 85 | + for (i = 0; i < sz / pdsize; i++, p++) { | |
| 86 | + *p = __pte(new); | |
| 87 | + if (new) | |
| 88 | + new += (unsigned long long)(pdsize / PAGE_SIZE) << PTE_RPN_SHIFT; | |
| 89 | + } | |
| 60 | 90 | |
| 61 | 91 | if (IS_ENABLED(CONFIG_44x) && !is_kernel_addr(addr) && (old & _PAGE_EXEC)) |
| 62 | 92 | icache_44x_need_flush = 1; |
| @@ -340,20 +370,6 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, · 277 unmodified lines | ||
| 340 | 370 | |
| 341 | 371 | #define pgprot_writecombine pgprot_noncached_wc |
| 342 | 372 | |
| 343 | −#ifdef CONFIG_HUGETLB_PAGE | |
| 344 | −static inline int hugepd_ok(hugepd_t hpd) | |
| 345 | −{ | |
| 346 | −#ifdef CONFIG_PPC_8xx | |
| 347 | − return ((hpd_val(hpd) & _PMD_PAGE_MASK) == _PMD_PAGE_8M); | |
| 348 | −#else | |
| 349 | − /* We clear the top bit to indicate hugepd */ | |
| 350 | − return (hpd_val(hpd) && (hpd_val(hpd) & PD_HUGE) == 0); | |
| 351 | −#endif | |
| 352 | −} | |
| 353 | − | |
| 354 | −#define is_hugepd(hpd) (hugepd_ok(hpd)) | |
| 355 | −#endif | |
| 356 | − | |
| 357 | 373 | int map_kernel_page(unsigned long va, phys_addr_t pa, pgprot_t prot); |
| 358 | 374 | void unmap_kernel_page(unsigned long va); |
| 359 | 375 | |
| @@ -19,20 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #define _PAGE_BAP_SX 0x000040 |
| 20 | 20 | #define _PAGE_BAP_UX 0x000080 |
| 21 | 21 | #define _PAGE_PSIZE_MSK 0x000f00 |
| 22 | −#define _PAGE_PSIZE_4K 0x000200 | |
| 23 | −#define _PAGE_PSIZE_8K 0x000300 | |
| 24 | −#define _PAGE_PSIZE_16K 0x000400 | |
| 25 | −#define _PAGE_PSIZE_32K 0x000500 | |
| 26 | −#define _PAGE_PSIZE_64K 0x000600 | |
| 27 | −#define _PAGE_PSIZE_128K 0x000700 | |
| 28 | −#define _PAGE_PSIZE_256K 0x000800 | |
| 29 | −#define _PAGE_PSIZE_512K 0x000900 | |
| 30 | −#define _PAGE_PSIZE_1M 0x000a00 | |
| 31 | −#define _PAGE_PSIZE_2M 0x000b00 | |
| 32 | −#define _PAGE_PSIZE_4M 0x000c00 | |
| 33 | −#define _PAGE_PSIZE_8M 0x000d00 | |
| 34 | −#define _PAGE_PSIZE_16M 0x000e00 | |
| 35 | −#define _PAGE_PSIZE_32M 0x000f00 | |
| 22 | +#define _PAGE_TSIZE_4K 0x000100 | |
| 36 | 23 | #define _PAGE_DIRTY 0x001000 /* C: page changed */ |
| 37 | 24 | #define _PAGE_SW0 0x002000 |
| 38 | 25 | #define _PAGE_U3 0x004000 |
| @@ -46,6 +33,9 @@ · 7 unmodified lines | ||
| 46 | 33 | #define _PAGE_NO_CACHE 0x400000 /* I: cache inhibit */ |
| 47 | 34 | #define _PAGE_WRITETHRU 0x800000 /* W: cache write-through */ |
| 48 | 35 | |
| 36 | +#define _PAGE_PSIZE_SHIFT 7 | |
| 37 | +#define _PAGE_PSIZE_SHIFT_OFFSET 10 | |
| 38 | + | |
| 49 | 39 | /* "Higher level" linux bit combinations */ |
| 50 | 40 | #define _PAGE_EXEC (_PAGE_BAP_SX | _PAGE_BAP_UX) /* .. and was cache cleaned */ |
| 51 | 41 | #define _PAGE_READ (_PAGE_BAP_SR | _PAGE_BAP_UR) /* User read permission */ |
| @@ -65,8 +55,6 @@ · 13 unmodified lines | ||
| 65 | 55 | |
| 66 | 56 | #define _PAGE_SPECIAL _PAGE_SW0 |
| 67 | 57 | |
| 68 | −/* Base page size */ | |
| 69 | −#define _PAGE_PSIZE _PAGE_PSIZE_4K | |
| 70 | 58 | #define PTE_RPN_SHIFT (24) |
| 71 | 59 | |
| 72 | 60 | #define PTE_WIMGE_SHIFT (19) |
| @@ -89,7 +77,7 @@ · 16 unmodified lines | ||
| 89 | 77 | * pages. We always set _PAGE_COHERENT when SMP is enabled or |
| 90 | 78 | * the processor might need it for DMA coherency. |
| 91 | 79 | */ |
| 92 | −#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE) | |
| 80 | +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_TSIZE_4K) | |
| 93 | 81 | #if defined(CONFIG_SMP) |
| 94 | 82 | #define _PAGE_BASE (_PAGE_BASE_NC | _PAGE_COHERENT) |
| 95 | 83 | #else |
| @@ -105,6 +93,47 @@ static inline pte_t pte_mkexec(pte_t pte) · 9 unmodified lines | ||
| 105 | 93 | } |
| 106 | 94 | #define pte_mkexec pte_mkexec |
| 107 | 95 | |
| 96 | +static inline unsigned long pte_huge_size(pte_t pte) | |
| 97 | +{ | |
| 98 | + pte_basic_t val = pte_val(pte); | |
| 99 | + | |
| 100 | + return 1UL << (((val & _PAGE_PSIZE_MSK) >> _PAGE_PSIZE_SHIFT) + _PAGE_PSIZE_SHIFT_OFFSET); | |
| 101 | +} | |
| 102 | +#define pte_huge_size pte_huge_size | |
| 103 | + | |
| 104 | +static inline int pmd_leaf(pmd_t pmd) | |
| 105 | +{ | |
| 106 | + if (IS_ENABLED(CONFIG_PPC64)) | |
| 107 | + return (long)pmd_val(pmd) > 0; | |
| 108 | + else | |
| 109 | + return pmd_val(pmd) & _PAGE_PSIZE_MSK; | |
| 110 | +} | |
| 111 | +#define pmd_leaf pmd_leaf | |
| 112 | + | |
| 113 | +static inline unsigned long pmd_leaf_size(pmd_t pmd) | |
| 114 | +{ | |
| 115 | + return pte_huge_size(__pte(pmd_val(pmd))); | |
| 116 | +} | |
| 117 | +#define pmd_leaf_size pmd_leaf_size | |
| 118 | + | |
| 119 | +#ifdef CONFIG_PPC64 | |
| 120 | +static inline int pud_leaf(pud_t pud) | |
| 121 | +{ | |
| 122 | + if (IS_ENABLED(CONFIG_PPC64)) | |
| 123 | + return (long)pud_val(pud) > 0; | |
| 124 | + else | |
| 125 | + return pud_val(pud) & _PAGE_PSIZE_MSK; | |
| 126 | +} | |
| 127 | +#define pud_leaf pud_leaf | |
| 128 | + | |
| 129 | +static inline unsigned long pud_leaf_size(pud_t pud) | |
| 130 | +{ | |
| 131 | + return pte_huge_size(__pte(pud_val(pud))); | |
| 132 | +} | |
| 133 | +#define pud_leaf_size pud_leaf_size | |
| 134 | + | |
| 135 | +#endif | |
| 136 | + | |
| 108 | 137 | #endif /* __ASSEMBLY__ */ |
| 109 | 138 | |
| 110 | 139 | #endif /* __KERNEL__ */ |
| @@ -269,38 +269,6 @@ static inline const void *pfn_to_kaddr(unsigned long pfn) · 268 unmodified lines | ||
| 269 | 269 | #define is_kernel_addr(x) ((x) >= TASK_SIZE) |
| 270 | 270 | #endif |
| 271 | 271 | |
| 272 | −#ifndef CONFIG_PPC_BOOK3S_64 | |
| 273 | −/* | |
| 274 | − * Use the top bit of the higher-level page table entries to indicate whether | |
| 275 | − * the entries we point to contain hugepages. This works because we know that | |
| 276 | − * the page tables live in kernel space. If we ever decide to support having | |
| 277 | − * page tables at arbitrary addresses, this breaks and will have to change. | |
| 278 | − */ | |
| 279 | −#ifdef CONFIG_PPC64 | |
| 280 | −#define PD_HUGE 0x8000000000000000UL | |
| 281 | −#else | |
| 282 | −#define PD_HUGE 0x80000000 | |
| 283 | −#endif | |
| 284 | − | |
| 285 | −#else /* CONFIG_PPC_BOOK3S_64 */ | |
| 286 | −/* | |
| 287 | − * Book3S 64 stores real addresses in the hugepd entries to | |
| 288 | − * avoid overlaps with _PAGE_PRESENT and _PAGE_PTE. | |
| 289 | − */ | |
| 290 | −#define HUGEPD_ADDR_MASK (0x0ffffffffffffffful & ~HUGEPD_SHIFT_MASK) | |
| 291 | −#endif /* CONFIG_PPC_BOOK3S_64 */ | |
| 292 | − | |
| 293 | −/* | |
| 294 | − * Some number of bits at the level of the page table that points to | |
| 295 | − * a hugepte are used to encode the size. This masks those bits. | |
| 296 | − * On 8xx, HW assistance requires 4k alignment for the hugepte. | |
| 297 | − */ | |
| 298 | −#ifdef CONFIG_PPC_8xx | |
| 299 | −#define HUGEPD_SHIFT_MASK 0xfff | |
| 300 | −#else | |
| 301 | −#define HUGEPD_SHIFT_MASK 0x3f | |
| 302 | −#endif | |
| 303 | − | |
| 304 | 272 | #ifndef __ASSEMBLY__ |
| 305 | 273 | |
| 306 | 274 | #ifdef CONFIG_PPC_BOOK3S_64 |
| @@ -101,14 +101,4 @@ static inline bool pmd_xchg(pmd_t *pmdp, pmd_t old, pmd_t new) · 100 unmodified lines | ||
| 101 | 101 | return pmd_raw(old) == prev; |
| 102 | 102 | } |
| 103 | 103 | |
| 104 | −#ifdef CONFIG_ARCH_HAS_HUGEPD | |
| 105 | −typedef struct { __be64 pdbe; } hugepd_t; | |
| 106 | −#define __hugepd(x) ((hugepd_t) { cpu_to_be64(x) }) | |
| 107 | − | |
| 108 | −static inline unsigned long hpd_val(hugepd_t x) | |
| 109 | −{ | |
| 110 | − return be64_to_cpu(x.pdbe); | |
| 111 | −} | |
| 112 | −#endif | |
| 113 | − | |
| 114 | 104 | #endif /* _ASM_POWERPC_PGTABLE_BE_TYPES_H */ |
| @@ -49,7 +49,11 @@ static inline unsigned long pud_val(pud_t x) · 48 unmodified lines | ||
| 49 | 49 | #endif /* CONFIG_PPC64 */ |
| 50 | 50 | |
| 51 | 51 | /* PGD level */ |
| 52 | +#if defined(CONFIG_PPC_E500) && defined(CONFIG_PTE_64BIT) | |
| 53 | +typedef struct { unsigned long long pgd; } pgd_t; | |
| 54 | +#else | |
| 52 | 55 | typedef struct { unsigned long pgd; } pgd_t; |
| 56 | +#endif | |
| 53 | 57 | #define __pgd(x) ((pgd_t) { (x) }) |
| 54 | 58 | static inline unsigned long pgd_val(pgd_t x) |
| 55 | 59 | { |
| @@ -83,13 +87,4 @@ static inline bool pte_xchg(pte_t *ptep, pte_t old, pte_t new) · 27 unmodified lines | ||
| 83 | 87 | } |
| 84 | 88 | #endif |
| 85 | 89 | |
| 86 | −#ifdef CONFIG_ARCH_HAS_HUGEPD | |
| 87 | −typedef struct { unsigned long pd; } hugepd_t; | |
| 88 | −#define __hugepd(x) ((hugepd_t) { (x) }) | |
| 89 | −static inline unsigned long hpd_val(hugepd_t x) | |
| 90 | −{ | |
| 91 | − return x.pd; | |
| 92 | −} | |
| 93 | −#endif | |
| 94 | − | |
| 95 | 90 | #endif /* _ASM_POWERPC_PGTABLE_TYPES_H */ |
| @@ -106,6 +106,9 @@ unsigned long vmalloc_to_phys(void *vmalloc_addr); · 105 unmodified lines | ||
| 106 | 106 | |
| 107 | 107 | void pgtable_cache_add(unsigned int shift); |
| 108 | 108 | |
| 109 | +#ifdef CONFIG_PPC32 | |
| 110 | +void __init *early_alloc_pgtable(unsigned long size); | |
| 111 | +#endif | |
| 109 | 112 | pte_t *early_pte_alloc_kernel(pmd_t *pmdp, unsigned long va); |
| 110 | 113 | |
| 111 | 114 | #if defined(CONFIG_STRICT_KERNEL_RWX) || defined(CONFIG_PPC32) |
| @@ -485,8 +485,8 @@ interrupt_base_book3e: /* fake trap */ · 484 unmodified lines | ||
| 485 | 485 | EXCEPTION_STUB(0x160, decrementer) /* 0x0900 */ |
| 486 | 486 | EXCEPTION_STUB(0x180, fixed_interval) /* 0x0980 */ |
| 487 | 487 | EXCEPTION_STUB(0x1a0, watchdog) /* 0x09f0 */ |
| 488 | − EXCEPTION_STUB(0x1c0, data_tlb_miss) | |
| 489 | − EXCEPTION_STUB(0x1e0, instruction_tlb_miss) | |
| 488 | + EXCEPTION_STUB(0x1c0, data_tlb_miss_bolted) | |
| 489 | + EXCEPTION_STUB(0x1e0, instruction_tlb_miss_bolted) | |
| 490 | 490 | EXCEPTION_STUB(0x200, altivec_unavailable) |
| 491 | 491 | EXCEPTION_STUB(0x220, altivec_assist) |
| 492 | 492 | EXCEPTION_STUB(0x260, perfmon) |
| @@ -294,9 +294,10 @@ set_ivor: · 293 unmodified lines | ||
| 294 | 294 | /* Macros to hide the PTE size differences |
| 295 | 295 | * |
| 296 | 296 | * FIND_PTE -- walks the page tables given EA & pgdir pointer |
| 297 | − * r10 -- EA of fault | |
| 297 | + * r10 -- free | |
| 298 | 298 | * r11 -- PGDIR pointer |
| 299 | 299 | * r12 -- free |
| 300 | + * r13 -- EA of fault | |
| 300 | 301 | * label 2: is the bailout case |
| 301 | 302 | * |
| 302 | 303 | * if we find the pte (fall through): |
| @@ -307,34 +308,34 @@ set_ivor: · 4 unmodified lines | ||
| 307 | 308 | #ifdef CONFIG_PTE_64BIT |
| 308 | 309 | #ifdef CONFIG_HUGETLB_PAGE |
| 309 | 310 | #define FIND_PTE \ |
| 310 | − rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \ | |
| 311 | − lwzx r11, r12, r11; /* Get pgd/pmd entry */ \ | |
| 311 | + rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ | |
| 312 | + add r12, r11, r12; \ | |
| 313 | + lwz r11, 4(r12); /* Get pgd/pmd entry */ \ | |
| 314 | + rlwinm. r10, r11, 32 - _PAGE_PSIZE_SHIFT, 0x1e; /* get tsize*/ \ | |
| 315 | + bne 1000f; /* Huge page (leaf entry) */ \ | |
| 312 | 316 | rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ |
| 313 | − blt 1000f; /* Normal non-huge page */ \ | |
| 314 | 317 | beq 2f; /* Bail if no table */ \ |
| 315 | − oris r11, r11, PD_HUGE@h; /* Put back address bit */ \ | |
| 316 | − andi. r10, r11, HUGEPD_SHIFT_MASK@l; /* extract size field */ \ | |
| 317 | − xor r12, r10, r11; /* drop size bits from pointer */ \ | |
| 318 | − b 1001f; \ | |
| 319 | −1000: rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ | |
| 318 | + rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ | |
| 320 | 319 | li r10, 0; /* clear r10 */ \ |
| 321 | −1001: lwz r11, 4(r12); /* Get pte entry */ | |
| 320 | + lwz r11, 4(r12); /* Get pte entry */ \ | |
| 321 | +1000: | |
| 322 | 322 | #else |
| 323 | 323 | #define FIND_PTE \ |
| 324 | − rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \ | |
| 325 | − lwzx r11, r12, r11; /* Get pgd/pmd entry */ \ | |
| 324 | + rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ | |
| 325 | + add r12, r11, r12; \ | |
| 326 | + lwz r11, 4(r12); /* Get pgd/pmd entry */ \ | |
| 326 | 327 | rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ |
| 327 | 328 | beq 2f; /* Bail if no table */ \ |
| 328 | − rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ | |
| 329 | + rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ | |
| 329 | 330 | lwz r11, 4(r12); /* Get pte entry */ |
| 330 | 331 | #endif /* HUGEPAGE */ |
| 331 | 332 | #else /* !PTE_64BIT */ |
| 332 | 333 | #define FIND_PTE \ |
| 333 | − rlwimi r11, r10, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \ | |
| 334 | + rlwimi r11, r13, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \ | |
| 334 | 335 | lwz r11, 0(r11); /* Get L1 entry */ \ |
| 335 | 336 | rlwinm. r12, r11, 0, 0, 19; /* Extract L2 (pte) base address */ \ |
| 336 | 337 | beq 2f; /* Bail if no table */ \ |
| 337 | − rlwimi r12, r10, 22, 20, 29; /* Compute PTE address */ \ | |
| 338 | + rlwimi r12, r13, 22, 20, 29; /* Compute PTE address */ \ | |
| 338 | 339 | lwz r11, 0(r12); /* Get Linux PTE */ |
| 339 | 340 | #endif |
| 340 | 341 | |
| @@ -441,13 +442,13 @@ START_BTB_FLUSH_SECTION · 100 unmodified lines | ||
| 441 | 442 | BTB_FLUSH(r10) |
| 442 | 443 | 1: |
| 443 | 444 | END_BTB_FLUSH_SECTION |
| 444 | − mfspr r10, SPRN_DEAR /* Get faulting address */ | |
| 445 | + mfspr r13, SPRN_DEAR /* Get faulting address */ | |
| 445 | 446 | |
| 446 | 447 | /* If we are faulting a kernel address, we have to use the |
| 447 | 448 | * kernel page tables. |
| 448 | 449 | */ |
| 449 | 450 | lis r11, PAGE_OFFSET@h |
| 450 | − cmplw 5, r10, r11 | |
| 451 | + cmplw 5, r13, r11 | |
| 451 | 452 | blt 5, 3f |
| 452 | 453 | lis r11, swapper_pg_dir@h |
| 453 | 454 | ori r11, r11, swapper_pg_dir@l |
| @@ -470,29 +471,14 @@ END_BTB_FLUSH_SECTION · 16 unmodified lines | ||
| 470 | 471 | #endif |
| 471 | 472 | |
| 472 | 473 | 4: |
| 473 | − /* Mask of required permission bits. Note that while we | |
| 474 | − * do copy ESR:ST to _PAGE_WRITE position as trying to write | |
| 475 | − * to an RO page is pretty common, we don't do it with | |
| 476 | − * _PAGE_DIRTY. We could do it, but it's a fairly rare | |
| 477 | − * event so I'd rather take the overhead when it happens | |
| 478 | − * rather than adding an instruction here. We should measure | |
| 479 | − * whether the whole thing is worth it in the first place | |
| 480 | − * as we could avoid loading SPRN_ESR completely in the first | |
| 481 | − * place... | |
| 482 | − * | |
| 483 | − * TODO: Is it worth doing that mfspr & rlwimi in the first | |
| 484 | − * place or can we save a couple of instructions here ? | |
| 485 | − */ | |
| 486 | − mfspr r12,SPRN_ESR | |
| 474 | + FIND_PTE | |
| 475 | + | |
| 487 | 476 | #ifdef CONFIG_PTE_64BIT |
| 488 | 477 | li r13,_PAGE_PRESENT|_PAGE_BAP_SR |
| 489 | 478 | oris r13,r13,_PAGE_ACCESSED@h |
| 490 | 479 | #else |
| 491 | 480 | li r13,_PAGE_PRESENT|_PAGE_READ|_PAGE_ACCESSED |
| 492 | 481 | #endif |
| 493 | − rlwimi r13,r12,11,29,29 | |
| 494 | − | |
| 495 | − FIND_PTE | |
| 496 | 482 | andc. r13,r13,r11 /* Check permission */ |
| 497 | 483 | |
| 498 | 484 | #ifdef CONFIG_PTE_64BIT |
| @@ -549,13 +535,13 @@ START_BTB_FLUSH_SECTION · 50 unmodified lines | ||
| 549 | 535 | 1: |
| 550 | 536 | END_BTB_FLUSH_SECTION |
| 551 | 537 | |
| 552 | − mfspr r10, SPRN_SRR0 /* Get faulting address */ | |
| 538 | + mfspr r13, SPRN_SRR0 /* Get faulting address */ | |
| 553 | 539 | |
| 554 | 540 | /* If we are faulting a kernel address, we have to use the |
| 555 | 541 | * kernel page tables. |
| 556 | 542 | */ |
| 557 | 543 | lis r11, PAGE_OFFSET@h |
| 558 | − cmplw 5, r10, r11 | |
| 544 | + cmplw 5, r13, r11 | |
| 559 | 545 | blt 5, 3f |
| 560 | 546 | lis r11, swapper_pg_dir@h |
| 561 | 547 | ori r11, r11, swapper_pg_dir@l |
| @@ -564,6 +550,7 @@ END_BTB_FLUSH_SECTION · 2 unmodified lines | ||
| 564 | 550 | rlwinm r12,r12,0,16,1 |
| 565 | 551 | mtspr SPRN_MAS1,r12 |
| 566 | 552 | |
| 553 | + FIND_PTE | |
| 567 | 554 | /* Make up the required permissions for kernel code */ |
| 568 | 555 | #ifdef CONFIG_PTE_64BIT |
| 569 | 556 | li r13,_PAGE_PRESENT | _PAGE_BAP_SX |
| @@ -584,6 +571,7 @@ END_BTB_FLUSH_SECTION · 14 unmodified lines | ||
| 584 | 571 | beq 2f /* KUAP fault */ |
| 585 | 572 | #endif |
| 586 | 573 | |
| 574 | + FIND_PTE | |
| 587 | 575 | /* Make up the required permissions for user code */ |
| 588 | 576 | #ifdef CONFIG_PTE_64BIT |
| 589 | 577 | li r13,_PAGE_PRESENT | _PAGE_BAP_UX |
| @@ -593,7 +581,6 @@ END_BTB_FLUSH_SECTION · 3 unmodified lines | ||
| 593 | 581 | #endif |
| 594 | 582 | |
| 595 | 583 | 4: |
| 596 | − FIND_PTE | |
| 597 | 584 | andc. r13,r13,r11 /* Check permission */ |
| 598 | 585 | |
| 599 | 586 | #ifdef CONFIG_PTE_64BIT |
| @@ -746,17 +733,12 @@ finish_tlb_load: · 146 unmodified lines | ||
| 746 | 733 | lwz r15, 0(r14) |
| 747 | 734 | 100: stw r15, 0(r17) |
| 748 | 735 | |
| 749 | − /* | |
| 750 | − * Calc MAS1_TSIZE from r10 (which has pshift encoded) | |
| 751 | − * tlb_enc = (pshift - 10). | |
| 752 | − */ | |
| 753 | − subi r15, r10, 10 | |
| 754 | 736 | mfspr r16, SPRN_MAS1 |
| 755 | − rlwimi r16, r15, 7, 20, 24 | |
| 737 | + rlwimi r16, r10, MAS1_TSIZE_SHIFT, MAS1_TSIZE_MASK | |
| 756 | 738 | mtspr SPRN_MAS1, r16 |
| 757 | 739 | |
| 758 | 740 | /* copy the pshift for use later */ |
| 759 | − mr r14, r10 | |
| 741 | + addi r14, r10, _PAGE_PSIZE_SHIFT_OFFSET | |
| 760 | 742 | |
| 761 | 743 | /* fall through */ |
| 762 | 744 | |
| @@ -415,14 +415,13 @@ FixupDAR:/* Entry point for dcbx workaround. */ · 414 unmodified lines | ||
| 415 | 415 | oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha |
| 416 | 416 | 3: |
| 417 | 417 | lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ |
| 418 | + rlwinm r11, r11, 0, ~_PMD_PAGE_8M | |
| 418 | 419 | mtspr SPRN_MD_TWC, r11 |
| 419 | − mtcrf 0x01, r11 | |
| 420 | 420 | mfspr r11, SPRN_MD_TWC |
| 421 | 421 | lwz r11, 0(r11) /* Get the pte */ |
| 422 | − bt 28,200f /* bit 28 = Large page (8M) */ | |
| 423 | 422 | /* concat physical page address(r11) and page offset(r10) */ |
| 424 | 423 | rlwimi r11, r10, 0, 32 - PAGE_SHIFT, 31 |
| 425 | −201: lwz r11,0(r11) | |
| 424 | + lwz r11,0(r11) | |
| 426 | 425 | /* Check if it really is a dcbx instruction. */ |
| 427 | 426 | /* dcbt and dcbtst does not generate DTLB Misses/Errors, |
| 428 | 427 | * no need to include them here */ |
| @@ -441,11 +440,6 @@ FixupDAR:/* Entry point for dcbx workaround. */ · 12 unmodified lines | ||
| 441 | 440 | 141: mfspr r10,SPRN_M_TW |
| 442 | 441 | b DARFixed /* Nope, go back to normal TLB processing */ |
| 443 | 442 | |
| 444 | −200: | |
| 445 | − /* concat physical page address(r11) and page offset(r10) */ | |
| 446 | − rlwimi r11, r10, 0, 32 - PAGE_SHIFT_8M, 31 | |
| 447 | − b 201b | |
| 448 | − | |
| 449 | 443 | 144: mfspr r10, SPRN_DSISR |
| 450 | 444 | rlwinm r10, r10,0,7,5 /* Clear store bit for buggy dcbst insn */ |
| 451 | 445 | mtspr SPRN_DSISR, r10 |
| @@ -696,11 +696,7 @@ __init u64 ppc64_bolted_size(void) · 695 unmodified lines | ||
| 696 | 696 | { |
| 697 | 697 | #ifdef CONFIG_PPC_BOOK3E_64 |
| 698 | 698 | /* Freescale BookE bolts the entire linear mapping */ |
| 699 | − /* XXX: BookE ppc64_rma_limit setup seems to disagree? */ | |
| 700 | − if (early_mmu_has_feature(MMU_FTR_TYPE_FSL_E)) | |
| 701 | − return linear_map_top; | |
| 702 | − /* Other BookE, we assume the first GB is bolted */ | |
| 703 | − return 1ul << 30; | |
| 699 | + return linear_map_top; | |
| 704 | 700 | #else |
| 705 | 701 | /* BookS radix, does not take faults on linear mapping */ |
| 706 | 702 | if (early_radix_enabled()) |
| @@ -1233,10 +1233,6 @@ void __init hash__early_init_mmu(void) · 1232 unmodified lines | ||
| 1233 | 1233 | __pmd_table_size = H_PMD_TABLE_SIZE; |
| 1234 | 1234 | __pud_table_size = H_PUD_TABLE_SIZE; |
| 1235 | 1235 | __pgd_table_size = H_PGD_TABLE_SIZE; |
| 1236 | − /* | |
| 1237 | − * 4k use hugepd format, so for hash set then to | |
| 1238 | − * zero | |
| 1239 | − */ | |
| 1240 | 1236 | __pmd_val_bits = HASH_PMD_VAL_BITS; |
| 1241 | 1237 | __pud_val_bits = HASH_PUD_VAL_BITS; |
| 1242 | 1238 | __pgd_val_bits = HASH_PGD_VAL_BITS; |
| @@ -1546,6 +1542,13 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, · 303 unmodified lines | ||
| 1546 | 1542 | goto bail; |
| 1547 | 1543 | } |
| 1548 | 1544 | |
| 1545 | + if (IS_ENABLED(CONFIG_PPC_4K_PAGES) && !radix_enabled()) { | |
| 1546 | + if (hugeshift == PMD_SHIFT && psize == MMU_PAGE_16M) | |
| 1547 | + hugeshift = mmu_psize_defs[MMU_PAGE_16M].shift; | |
| 1548 | + if (hugeshift == PUD_SHIFT && psize == MMU_PAGE_16G) | |
| 1549 | + hugeshift = mmu_psize_defs[MMU_PAGE_16G].shift; | |
| 1550 | + } | |
| 1551 | + | |
| 1549 | 1552 | /* |
| 1550 | 1553 | * Add _PAGE_PRESENT to the required access perm. If there are parallel |
| 1551 | 1554 | * updates to the pte that can possibly clear _PAGE_PTE, catch that too. |
| @@ -53,6 +53,16 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · 52 unmodified lines | ||
| 53 | 53 | /* If PTE permissions don't match, take page fault */ |
| 54 | 54 | if (unlikely(!check_pte_access(access, old_pte))) |
| 55 | 55 | return 1; |
| 56 | + /* | |
| 57 | + * If hash-4k, hugepages use seeral contiguous PxD entries | |
| 58 | + * so bail out and let mm make the page young or dirty | |
| 59 | + */ | |
| 60 | + if (IS_ENABLED(CONFIG_PPC_4K_PAGES)) { | |
| 61 | + if (!(old_pte & _PAGE_ACCESSED)) | |
| 62 | + return 1; | |
| 63 | + if ((access & _PAGE_WRITE) && !(old_pte & _PAGE_DIRTY)) | |
| 64 | + return 1; | |
| 65 | + } | |
| 56 | 66 | |
| 57 | 67 | /* |
| 58 | 68 | * Try to lock the PTE, add ACCESSED and DIRTY if it was |
| @@ -461,18 +461,6 @@ static inline void pgtable_free(void *table, int index) · 460 unmodified lines | ||
| 461 | 461 | case PUD_INDEX: |
| 462 | 462 | __pud_free(table); |
| 463 | 463 | break; |
| 464 | −#if defined(CONFIG_PPC_4K_PAGES) && defined(CONFIG_HUGETLB_PAGE) | |
| 465 | − /* 16M hugepd directory at pud level */ | |
| 466 | − case HTLB_16M_INDEX: | |
| 467 | − BUILD_BUG_ON(H_16M_CACHE_INDEX <= 0); | |
| 468 | − kmem_cache_free(PGT_CACHE(H_16M_CACHE_INDEX), table); | |
| 469 | − break; | |
| 470 | − /* 16G hugepd directory at the pgd level */ | |
| 471 | − case HTLB_16G_INDEX: | |
| 472 | − BUILD_BUG_ON(H_16G_CACHE_INDEX <= 0); | |
| 473 | − kmem_cache_free(PGT_CACHE(H_16G_CACHE_INDEX), table); | |
| 474 | − break; | |
| 475 | −#endif | |
| 476 | 464 | /* We don't free pgd table via RCU callback */ |
| 477 | 465 | default: |
| 478 | 466 | BUG(); |
| @@ -28,8 +28,6 @@ · 27 unmodified lines | ||
| 28 | 28 | |
| 29 | 29 | bool hugetlb_disabled = false; |
| 30 | 30 | |
| 31 | −#define hugepd_none(hpd) (hpd_val(hpd) == 0) | |
| 32 | − | |
| 33 | 31 | #define PTE_T_ORDER (__builtin_ffs(sizeof(pte_basic_t)) - \ |
| 34 | 32 | __builtin_ffs(sizeof(void *))) |
| 35 | 33 | |
| @@ -42,156 +40,43 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long s · 6 unmodified lines | ||
| 42 | 40 | return __find_linux_pte(mm->pgd, addr, NULL, NULL); |
| 43 | 41 | } |
| 44 | 42 | |
| 45 | −static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp, | |
| 46 | − unsigned long address, unsigned int pdshift, | |
| 47 | − unsigned int pshift, spinlock_t *ptl) | |
| 43 | +pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, | |
| 44 | + unsigned long addr, unsigned long sz) | |
| 48 | 45 | { |
| 49 | − struct kmem_cache *cachep; | |
| 50 | − pte_t *new; | |
| 51 | − int i; | |
| 52 | − int num_hugepd; | |
| 53 | − | |
| 54 | − if (pshift >= pdshift) { | |
| 55 | − cachep = PGT_CACHE(PTE_T_ORDER); | |
| 56 | − num_hugepd = 1 << (pshift - pdshift); | |
| 57 | − } else { | |
| 58 | − cachep = PGT_CACHE(pdshift - pshift); | |
| 59 | − num_hugepd = 1; | |
| 60 | − } | |
| 61 | − | |
| 62 | − if (!cachep) { | |
| 63 | − WARN_ONCE(1, "No page table cache created for hugetlb tables"); | |
| 64 | − return -ENOMEM; | |
| 65 | − } | |
| 66 | − | |
| 67 | − new = kmem_cache_alloc(cachep, pgtable_gfp_flags(mm, GFP_KERNEL)); | |
| 46 | + p4d_t *p4d; | |
| 47 | + pud_t *pud; | |
| 48 | + pmd_t *pmd; | |
| 68 | 49 | |
| 69 | − BUG_ON(pshift > HUGEPD_SHIFT_MASK); | |
| 70 | − BUG_ON((unsigned long)new & HUGEPD_SHIFT_MASK); | |
| 50 | + addr &= ~(sz - 1); | |
| 71 | 51 | |
| 72 | − if (!new) | |
| 73 | − return -ENOMEM; | |
| 52 | + p4d = p4d_offset(pgd_offset(mm, addr), addr); | |
| 53 | + if (!mm_pud_folded(mm) && sz >= P4D_SIZE) | |
| 54 | + return (pte_t *)p4d; | |
| 74 | 55 | |
| 75 | − /* | |
| 76 | − * Make sure other cpus find the hugepd set only after a | |
| 77 | − * properly initialized page table is visible to them. | |
| 78 | − * For more details look for comment in __pte_alloc(). | |
| 79 | − */ | |
| 80 | − smp_wmb(); | |
| 56 | + pud = pud_alloc(mm, p4d, addr); | |
| 57 | + if (!pud) | |
| 58 | + return NULL; | |
| 59 | + if (!mm_pmd_folded(mm) && sz >= PUD_SIZE) | |
| 60 | + return (pte_t *)pud; | |
| 81 | 61 | |
| 82 | − spin_lock(ptl); | |
| 83 | − /* | |
| 84 | − * We have multiple higher-level entries that point to the same | |
| 85 | − * actual pte location. Fill in each as we go and backtrack on error. | |
| 86 | − * We need all of these so the DTLB pgtable walk code can find the | |
| 87 | − * right higher-level entry without knowing if it's a hugepage or not. | |
| 88 | − */ | |
| 89 | − for (i = 0; i < num_hugepd; i++, hpdp++) { | |
| 90 | − if (unlikely(!hugepd_none(*hpdp))) | |
| 91 | − break; | |
| 92 | − hugepd_populate(hpdp, new, pshift); | |
| 93 | − } | |
| 94 | − /* If we bailed from the for loop early, an error occurred, clean up */ | |
| 95 | − if (i < num_hugepd) { | |
| 96 | − for (i = i - 1 ; i >= 0; i--, hpdp--) | |
| 97 | − *hpdp = __hugepd(0); | |
| 98 | − kmem_cache_free(cachep, new); | |
| 99 | − } else { | |
| 100 | − kmemleak_ignore(new); | |
| 101 | − } | |
| 102 | − spin_unlock(ptl); | |
| 103 | − return 0; | |
| 104 | −} | |
| 62 | + pmd = pmd_alloc(mm, pud, addr); | |
| 63 | + if (!pmd) | |
| 64 | + return NULL; | |
| 105 | 65 | |
| 106 | −/* | |
| 107 | − * At this point we do the placement change only for BOOK3S 64. This would | |
| 108 | − * possibly work on other subarchs. | |
| 109 | − */ | |
| 110 | −pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, | |
| 111 | − unsigned long addr, unsigned long sz) | |
| 112 | −{ | |
| 113 | − pgd_t *pg; | |
| 114 | − p4d_t *p4; | |
| 115 | − pud_t *pu; | |
| 116 | − pmd_t *pm; | |
| 117 | − hugepd_t *hpdp = NULL; | |
| 118 | − unsigned pshift = __ffs(sz); | |
| 119 | − unsigned pdshift = PGDIR_SHIFT; | |
| 120 | − spinlock_t *ptl; | |
| 121 | − | |
| 122 | − addr &= ~(sz-1); | |
| 123 | − pg = pgd_offset(mm, addr); | |
| 124 | − p4 = p4d_offset(pg, addr); | |
| 66 | + if (sz >= PMD_SIZE) { | |
| 67 | + /* On 8xx, all hugepages are handled as contiguous PTEs */ | |
| 68 | + if (IS_ENABLED(CONFIG_PPC_8xx)) { | |
| 69 | + int i; | |
| 125 | 70 | |
| 126 | −#ifdef CONFIG_PPC_BOOK3S_64 | |
| 127 | − if (pshift == PGDIR_SHIFT) | |
| 128 | − /* 16GB huge page */ | |
| 129 | − return (pte_t *) p4; | |
| 130 | − else if (pshift > PUD_SHIFT) { | |
| 131 | − /* | |
| 132 | − * We need to use hugepd table | |
| 133 | − */ | |
| 134 | − ptl = &mm->page_table_lock; | |
| 135 | − hpdp = (hugepd_t *)p4; | |
| 136 | − } else { | |
| 137 | − pdshift = PUD_SHIFT; | |
| 138 | − pu = pud_alloc(mm, p4, addr); | |
| 139 | − if (!pu) | |
| 140 | − return NULL; | |
| 141 | − if (pshift == PUD_SHIFT) | |
| 142 | − return (pte_t *)pu; | |
| 143 | − else if (pshift > PMD_SHIFT) { | |
| 144 | − ptl = pud_lockptr(mm, pu); | |
| 145 | − hpdp = (hugepd_t *)pu; | |
| 146 | − } else { | |
| 147 | − pdshift = PMD_SHIFT; | |
| 148 | − pm = pmd_alloc(mm, pu, addr); | |
| 149 | − if (!pm) | |
| 150 | − return NULL; | |
| 151 | − if (pshift == PMD_SHIFT) | |
| 152 | − /* 16MB hugepage */ | |
| 153 | − return (pte_t *)pm; | |
| 154 | − else { | |
| 155 | − ptl = pmd_lockptr(mm, pm); | |
| 156 | − hpdp = (hugepd_t *)pm; | |
| 71 | + for (i = 0; i < sz / PMD_SIZE; i++) { | |
| 72 | + if (!pte_alloc_huge(mm, pmd + i, addr)) | |
| 73 | + return NULL; | |
| 157 | 74 | } |
| 158 | 75 | } |
| 76 | + return (pte_t *)pmd; | |
| 159 | 77 | } |
| 160 | −#else | |
| 161 | − if (pshift >= PGDIR_SHIFT) { | |
| 162 | − ptl = &mm->page_table_lock; | |
| 163 | − hpdp = (hugepd_t *)p4; | |
| 164 | − } else { | |
| 165 | − pdshift = PUD_SHIFT; | |
| 166 | − pu = pud_alloc(mm, p4, addr); | |
| 167 | − if (!pu) | |
| 168 | − return NULL; | |
| 169 | − if (pshift >= PUD_SHIFT) { | |
| 170 | − ptl = pud_lockptr(mm, pu); | |
| 171 | − hpdp = (hugepd_t *)pu; | |
| 172 | − } else { | |
| 173 | − pdshift = PMD_SHIFT; | |
| 174 | − pm = pmd_alloc(mm, pu, addr); | |
| 175 | − if (!pm) | |
| 176 | − return NULL; | |
| 177 | − ptl = pmd_lockptr(mm, pm); | |
| 178 | − hpdp = (hugepd_t *)pm; | |
| 179 | − } | |
| 180 | − } | |
| 181 | −#endif | |
| 182 | − if (!hpdp) | |
| 183 | − return NULL; | |
| 184 | − | |
| 185 | − if (IS_ENABLED(CONFIG_PPC_8xx) && pshift < PMD_SHIFT) | |
| 186 | − return pte_alloc_huge(mm, (pmd_t *)hpdp, addr); | |
| 187 | − | |
| 188 | − BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp)); | |
| 189 | 78 | |
| 190 | − if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, | |
| 191 | − pdshift, pshift, ptl)) | |
| 192 | − return NULL; | |
| 193 | − | |
| 194 | − return hugepte_offset(*hpdp, addr, pdshift); | |
| 79 | + return pte_alloc_huge(mm, pmd, addr); | |
| 195 | 80 | } |
| 196 | 81 | |
| 197 | 82 | #ifdef CONFIG_PPC_BOOK3S_64 |
| @@ -248,264 +133,6 @@ int __init alloc_bootmem_huge_page(struct hstate *h, int nid) · 50 unmodified lines | ||
| 248 | 133 | return __alloc_bootmem_huge_page(h, nid); |
| 249 | 134 | } |
| 250 | 135 | |
| 251 | −#ifndef CONFIG_PPC_BOOK3S_64 | |
| 252 | −#define HUGEPD_FREELIST_SIZE \ | |
| 253 | − ((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t)) | |
| 254 | − | |
| 255 | −struct hugepd_freelist { | |
| 256 | − struct rcu_head rcu; | |
| 257 | − unsigned int index; | |
| 258 | − void *ptes[]; | |
| 259 | −}; | |
| 260 | − | |
| 261 | −static DEFINE_PER_CPU(struct hugepd_freelist *, hugepd_freelist_cur); | |
| 262 | − | |
| 263 | −static void hugepd_free_rcu_callback(struct rcu_head *head) | |
| 264 | −{ | |
| 265 | − struct hugepd_freelist *batch = | |
| 266 | − container_of(head, struct hugepd_freelist, rcu); | |
| 267 | − unsigned int i; | |
| 268 | − | |
| 269 | − for (i = 0; i < batch->index; i++) | |
| 270 | − kmem_cache_free(PGT_CACHE(PTE_T_ORDER), batch->ptes[i]); | |
| 271 | − | |
| 272 | − free_page((unsigned long)batch); | |
| 273 | −} | |
| 274 | − | |
| 275 | −static void hugepd_free(struct mmu_gather *tlb, void *hugepte) | |
| 276 | −{ | |
| 277 | − struct hugepd_freelist **batchp; | |
| 278 | − | |
| 279 | − batchp = &get_cpu_var(hugepd_freelist_cur); | |
| 280 | − | |
| 281 | − if (atomic_read(&tlb->mm->mm_users) < 2 || | |
| 282 | − mm_is_thread_local(tlb->mm)) { | |
| 283 | − kmem_cache_free(PGT_CACHE(PTE_T_ORDER), hugepte); | |
| 284 | − put_cpu_var(hugepd_freelist_cur); | |
| 285 | − return; | |
| 286 | − } | |
| 287 | − | |
| 288 | − if (*batchp == NULL) { | |
| 289 | − *batchp = (struct hugepd_freelist *)__get_free_page(GFP_ATOMIC); | |
| 290 | − (*batchp)->index = 0; | |
| 291 | − } | |
| 292 | − | |
| 293 | − (*batchp)->ptes[(*batchp)->index++] = hugepte; | |
| 294 | − if ((*batchp)->index == HUGEPD_FREELIST_SIZE) { | |
| 295 | − call_rcu(&(*batchp)->rcu, hugepd_free_rcu_callback); | |
| 296 | − *batchp = NULL; | |
| 297 | − } | |
| 298 | − put_cpu_var(hugepd_freelist_cur); | |
| 299 | −} | |
| 300 | −#else | |
| 301 | −static inline void hugepd_free(struct mmu_gather *tlb, void *hugepte) {} | |
| 302 | −#endif | |
| 303 | − | |
| 304 | −/* Return true when the entry to be freed maps more than the area being freed */ | |
| 305 | −static bool range_is_outside_limits(unsigned long start, unsigned long end, | |
| 306 | − unsigned long floor, unsigned long ceiling, | |
| 307 | − unsigned long mask) | |
| 308 | −{ | |
| 309 | − if ((start & mask) < floor) | |
| 310 | − return true; | |
| 311 | − if (ceiling) { | |
| 312 | − ceiling &= mask; | |
| 313 | − if (!ceiling) | |
| 314 | − return true; | |
| 315 | − } | |
| 316 | − return end - 1 > ceiling - 1; | |
| 317 | −} | |
| 318 | − | |
| 319 | −static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshift, | |
| 320 | − unsigned long start, unsigned long end, | |
| 321 | − unsigned long floor, unsigned long ceiling) | |
| 322 | −{ | |
| 323 | − pte_t *hugepte = hugepd_page(*hpdp); | |
| 324 | − int i; | |
| 325 | − | |
| 326 | − unsigned long pdmask = ~((1UL << pdshift) - 1); | |
| 327 | − unsigned int num_hugepd = 1; | |
| 328 | − unsigned int shift = hugepd_shift(*hpdp); | |
| 329 | − | |
| 330 | − /* Note: On fsl the hpdp may be the first of several */ | |
| 331 | − if (shift > pdshift) | |
| 332 | − num_hugepd = 1 << (shift - pdshift); | |
| 333 | − | |
| 334 | − if (range_is_outside_limits(start, end, floor, ceiling, pdmask)) | |
| 335 | − return; | |
| 336 | − | |
| 337 | − for (i = 0; i < num_hugepd; i++, hpdp++) | |
| 338 | − *hpdp = __hugepd(0); | |
| 339 | − | |
| 340 | − if (shift >= pdshift) | |
| 341 | − hugepd_free(tlb, hugepte); | |
| 342 | − else | |
| 343 | − pgtable_free_tlb(tlb, hugepte, | |
| 344 | − get_hugepd_cache_index(pdshift - shift)); | |
| 345 | −} | |
| 346 | − | |
| 347 | −static void hugetlb_free_pte_range(struct mmu_gather *tlb, pmd_t *pmd, | |
| 348 | − unsigned long addr, unsigned long end, | |
| 349 | − unsigned long floor, unsigned long ceiling) | |
| 350 | −{ | |
| 351 | − pgtable_t token = pmd_pgtable(*pmd); | |
| 352 | − | |
| 353 | − if (range_is_outside_limits(addr, end, floor, ceiling, PMD_MASK)) | |
| 354 | − return; | |
| 355 | − | |
| 356 | − pmd_clear(pmd); | |
| 357 | − pte_free_tlb(tlb, token, addr); | |
| 358 | − mm_dec_nr_ptes(tlb->mm); | |
| 359 | −} | |
| 360 | − | |
| 361 | −static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud, | |
| 362 | − unsigned long addr, unsigned long end, | |
| 363 | − unsigned long floor, unsigned long ceiling) | |
| 364 | −{ | |
| 365 | − pmd_t *pmd; | |
| 366 | − unsigned long next; | |
| 367 | − unsigned long start; | |
| 368 | − | |
| 369 | − start = addr; | |
| 370 | − do { | |
| 371 | − unsigned long more; | |
| 372 | − | |
| 373 | − pmd = pmd_offset(pud, addr); | |
| 374 | − next = pmd_addr_end(addr, end); | |
| 375 | − if (!is_hugepd(__hugepd(pmd_val(*pmd)))) { | |
| 376 | − if (pmd_none_or_clear_bad(pmd)) | |
| 377 | − continue; | |
| 378 | − | |
| 379 | − /* | |
| 380 | − * if it is not hugepd pointer, we should already find | |
| 381 | − * it cleared. | |
| 382 | − */ | |
| 383 | − WARN_ON(!IS_ENABLED(CONFIG_PPC_8xx)); | |
| 384 | − | |
| 385 | − hugetlb_free_pte_range(tlb, pmd, addr, end, floor, ceiling); | |
| 386 | − | |
| 387 | − continue; | |
| 388 | − } | |
| 389 | − /* | |
| 390 | − * Increment next by the size of the huge mapping since | |
| 391 | − * there may be more than one entry at this level for a | |
| 392 | − * single hugepage, but all of them point to | |
| 393 | − * the same kmem cache that holds the hugepte. | |
| 394 | − */ | |
| 395 | − more = addr + (1UL << hugepd_shift(*(hugepd_t *)pmd)); | |
| 396 | − if (more > next) | |
| 397 | − next = more; | |
| 398 | − | |
| 399 | − free_hugepd_range(tlb, (hugepd_t *)pmd, PMD_SHIFT, | |
| 400 | − addr, next, floor, ceiling); | |
| 401 | − } while (addr = next, addr != end); | |
| 402 | − | |
| 403 | − if (range_is_outside_limits(start, end, floor, ceiling, PUD_MASK)) | |
| 404 | − return; | |
| 405 | − | |
| 406 | − pmd = pmd_offset(pud, start & PUD_MASK); | |
| 407 | − pud_clear(pud); | |
| 408 | − pmd_free_tlb(tlb, pmd, start & PUD_MASK); | |
| 409 | − mm_dec_nr_pmds(tlb->mm); | |
| 410 | −} | |
| 411 | − | |
| 412 | −static void hugetlb_free_pud_range(struct mmu_gather *tlb, p4d_t *p4d, | |
| 413 | − unsigned long addr, unsigned long end, | |
| 414 | − unsigned long floor, unsigned long ceiling) | |
| 415 | −{ | |
| 416 | − pud_t *pud; | |
| 417 | − unsigned long next; | |
| 418 | − unsigned long start; | |
| 419 | − | |
| 420 | − start = addr; | |
| 421 | − do { | |
| 422 | − pud = pud_offset(p4d, addr); | |
| 423 | − next = pud_addr_end(addr, end); | |
| 424 | − if (!is_hugepd(__hugepd(pud_val(*pud)))) { | |
| 425 | − if (pud_none_or_clear_bad(pud)) | |
| 426 | − continue; | |
| 427 | − hugetlb_free_pmd_range(tlb, pud, addr, next, floor, | |
| 428 | − ceiling); | |
| 429 | − } else { | |
| 430 | − unsigned long more; | |
| 431 | − /* | |
| 432 | − * Increment next by the size of the huge mapping since | |
| 433 | − * there may be more than one entry at this level for a | |
| 434 | − * single hugepage, but all of them point to | |
| 435 | − * the same kmem cache that holds the hugepte. | |
| 436 | − */ | |
| 437 | − more = addr + (1UL << hugepd_shift(*(hugepd_t *)pud)); | |
| 438 | − if (more > next) | |
| 439 | − next = more; | |
| 440 | − | |
| 441 | − free_hugepd_range(tlb, (hugepd_t *)pud, PUD_SHIFT, | |
| 442 | − addr, next, floor, ceiling); | |
| 443 | − } | |
| 444 | − } while (addr = next, addr != end); | |
| 445 | − | |
| 446 | − if (range_is_outside_limits(start, end, floor, ceiling, PGDIR_MASK)) | |
| 447 | − return; | |
| 448 | − | |
| 449 | − pud = pud_offset(p4d, start & PGDIR_MASK); | |
| 450 | − p4d_clear(p4d); | |
| 451 | − pud_free_tlb(tlb, pud, start & PGDIR_MASK); | |
| 452 | − mm_dec_nr_puds(tlb->mm); | |
| 453 | −} | |
| 454 | − | |
| 455 | −/* | |
| 456 | − * This function frees user-level page tables of a process. | |
| 457 | − */ | |
| 458 | −void hugetlb_free_pgd_range(struct mmu_gather *tlb, | |
| 459 | − unsigned long addr, unsigned long end, | |
| 460 | − unsigned long floor, unsigned long ceiling) | |
| 461 | −{ | |
| 462 | − pgd_t *pgd; | |
| 463 | − p4d_t *p4d; | |
| 464 | − unsigned long next; | |
| 465 | − | |
| 466 | − /* | |
| 467 | − * Because there are a number of different possible pagetable | |
| 468 | − * layouts for hugepage ranges, we limit knowledge of how | |
| 469 | − * things should be laid out to the allocation path | |
| 470 | − * (huge_pte_alloc(), above). Everything else works out the | |
| 471 | − * structure as it goes from information in the hugepd | |
| 472 | − * pointers. That means that we can't here use the | |
| 473 | − * optimization used in the normal page free_pgd_range(), of | |
| 474 | − * checking whether we're actually covering a large enough | |
| 475 | − * range to have to do anything at the top level of the walk | |
| 476 | − * instead of at the bottom. | |
| 477 | − * | |
| 478 | − * To make sense of this, you should probably go read the big | |
| 479 | − * block comment at the top of the normal free_pgd_range(), | |
| 480 | − * too. | |
| 481 | − */ | |
| 482 | − | |
| 483 | − do { | |
| 484 | − next = pgd_addr_end(addr, end); | |
| 485 | − pgd = pgd_offset(tlb->mm, addr); | |
| 486 | − p4d = p4d_offset(pgd, addr); | |
| 487 | − if (!is_hugepd(__hugepd(pgd_val(*pgd)))) { | |
| 488 | − if (p4d_none_or_clear_bad(p4d)) | |
| 489 | − continue; | |
| 490 | − hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling); | |
| 491 | − } else { | |
| 492 | − unsigned long more; | |
| 493 | − /* | |
| 494 | − * Increment next by the size of the huge mapping since | |
| 495 | − * there may be more than one entry at the pgd level | |
| 496 | − * for a single hugepage, but all of them point to the | |
| 497 | − * same kmem cache that holds the hugepte. | |
| 498 | − */ | |
| 499 | − more = addr + (1UL << hugepd_shift(*(hugepd_t *)pgd)); | |
| 500 | − if (more > next) | |
| 501 | − next = more; | |
| 502 | − | |
| 503 | − free_hugepd_range(tlb, (hugepd_t *)p4d, PGDIR_SHIFT, | |
| 504 | − addr, next, floor, ceiling); | |
| 505 | − } | |
| 506 | − } while (addr = next, addr != end); | |
| 507 | −} | |
| 508 | − | |
| 509 | 136 | bool __init arch_hugetlb_valid_size(unsigned long size) |
| 510 | 137 | { |
| 511 | 138 | int shift = __ffs(size); |
| @@ -552,44 +179,14 @@ static int __init hugetlbpage_init(void) · 40 unmodified lines | ||
| 552 | 179 | |
| 553 | 180 | for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { |
| 554 | 181 | unsigned shift; |
| 555 | − unsigned pdshift; | |
| 556 | 182 | |
| 557 | 183 | if (!mmu_psize_defs[psize].shift) |
| 558 | 184 | continue; |
| 559 | 185 | |
| 560 | 186 | shift = mmu_psize_to_shift(psize); |
| 561 | 187 | |
| 562 | −#ifdef CONFIG_PPC_BOOK3S_64 | |
| 563 | − if (shift > PGDIR_SHIFT) | |
| 564 | − continue; | |
| 565 | − else if (shift > PUD_SHIFT) | |
| 566 | − pdshift = PGDIR_SHIFT; | |
| 567 | − else if (shift > PMD_SHIFT) | |
| 568 | − pdshift = PUD_SHIFT; | |
| 569 | − else | |
| 570 | − pdshift = PMD_SHIFT; | |
| 571 | −#else | |
| 572 | − if (shift < PUD_SHIFT) | |
| 573 | − pdshift = PMD_SHIFT; | |
| 574 | − else if (shift < PGDIR_SHIFT) | |
| 575 | − pdshift = PUD_SHIFT; | |
| 576 | − else | |
| 577 | − pdshift = PGDIR_SHIFT; | |
| 578 | −#endif | |
| 579 | − | |
| 580 | 188 | if (add_huge_page_size(1ULL << shift) < 0) |
| 581 | 189 | continue; |
| 582 | − /* | |
| 583 | − * if we have pdshift and shift value same, we don't | |
| 584 | − * use pgt cache for hugepd. | |
| 585 | − */ | |
| 586 | − if (pdshift > shift) { | |
| 587 | − if (!IS_ENABLED(CONFIG_PPC_8xx)) | |
| 588 | − pgtable_cache_add(pdshift - shift); | |
| 589 | − } else if (IS_ENABLED(CONFIG_PPC_E500) || | |
| 590 | − IS_ENABLED(CONFIG_PPC_8xx)) { | |
| 591 | − pgtable_cache_add(PTE_T_ORDER); | |
| 592 | − } | |
| 593 | 190 | |
| 594 | 191 | configured = true; |
| 595 | 192 | } |
| @@ -123,12 +123,8 @@ void pgtable_cache_add(unsigned int shift) · 122 unmodified lines | ||
| 123 | 123 | /* When batching pgtable pointers for RCU freeing, we store |
| 124 | 124 | * the index size in the low bits. Table alignment must be |
| 125 | 125 | * big enough to fit it. |
| 126 | − * | |
| 127 | − * Likewise, hugeapge pagetable pointers contain a (different) | |
| 128 | − * shift value in the low bits. All tables must be aligned so | |
| 129 | − * as to leave enough 0 bits in the address to contain it. */ | |
| 130 | − unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1, | |
| 131 | − HUGEPD_SHIFT_MASK + 1); | |
| 126 | + */ | |
| 127 | + unsigned long minalign = MAX_PGTABLE_INDEX_SIZE + 1; | |
| 132 | 128 | struct kmem_cache *new = NULL; |
| 133 | 129 | |
| 134 | 130 | /* It would be nice if this was a BUILD_BUG_ON(), but at the |
| @@ -6,28 +6,33 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <linux/memblock.h> |
| 7 | 7 | #include <linux/hugetlb.h> |
| 8 | 8 | |
| 9 | +#include <asm/pgalloc.h> | |
| 10 | + | |
| 9 | 11 | static int __init |
| 10 | 12 | kasan_init_shadow_8M(unsigned long k_start, unsigned long k_end, void *block) |
| 11 | 13 | { |
| 12 | 14 | pmd_t *pmd = pmd_off_k(k_start); |
| 13 | 15 | unsigned long k_cur, k_next; |
| 14 | 16 | |
| 15 | − for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd += 2, block += SZ_8M) { | |
| 16 | − pte_basic_t *new; | |
| 17 | + for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++, block += SZ_4M) { | |
| 18 | + pte_t *ptep; | |
| 19 | + int i; | |
| 17 | 20 | |
| 18 | 21 | k_next = pgd_addr_end(k_cur, k_end); |
| 19 | − k_next = pgd_addr_end(k_next, k_end); | |
| 20 | 22 | if ((void *)pmd_page_vaddr(*pmd) != kasan_early_shadow_pte) |
| 21 | 23 | continue; |
| 22 | 24 | |
| 23 | − new = memblock_alloc(sizeof(pte_basic_t), SZ_4K); | |
| 24 | − if (!new) | |
| 25 | + ptep = memblock_alloc(PTE_FRAG_SIZE, PTE_FRAG_SIZE); | |
| 26 | + if (!ptep) | |
| 25 | 27 | return -ENOMEM; |
| 26 | 28 | |
| 27 | − *new = pte_val(pte_mkhuge(pfn_pte(PHYS_PFN(__pa(block)), PAGE_KERNEL))); | |
| 29 | + for (i = 0; i < PTRS_PER_PTE; i++) { | |
| 30 | + pte_t pte = pte_mkhuge(pfn_pte(PHYS_PFN(__pa(block + i * PAGE_SIZE)), PAGE_KERNEL)); | |
| 28 | 31 | |
| 29 | − hugepd_populate_kernel((hugepd_t *)pmd, (pte_t *)new, PAGE_SHIFT_8M); | |
| 30 | − hugepd_populate_kernel((hugepd_t *)pmd + 1, (pte_t *)new, PAGE_SHIFT_8M); | |
| 32 | + __set_pte_at(&init_mm, k_cur, ptep + i, pte, 1); | |
| 33 | + } | |
| 34 | + pmd_populate_kernel(&init_mm, pmd, ptep); | |
| 35 | + *pmd = __pmd(pmd_val(*pmd) | _PMD_PAGE_8M); | |
| 31 | 36 | } |
| 32 | 37 | return 0; |
| 33 | 38 | } |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/hugetlb.h> |
| 12 | 12 | |
| 13 | 13 | #include <asm/fixmap.h> |
| 14 | +#include <asm/pgalloc.h> | |
| 14 | 15 | |
| 15 | 16 | #include <mm/mmu_decl.h> |
| 16 | 17 | |
| @@ -48,20 +49,6 @@ unsigned long p_block_mapped(phys_addr_t pa) · 31 unmodified lines | ||
| 48 | 49 | return 0; |
| 49 | 50 | } |
| 50 | 51 | |
| 51 | −static pte_t __init *early_hugepd_alloc_kernel(hugepd_t *pmdp, unsigned long va) | |
| 52 | −{ | |
| 53 | − if (hpd_val(*pmdp) == 0) { | |
| 54 | − pte_t *ptep = memblock_alloc(sizeof(pte_basic_t), SZ_4K); | |
| 55 | − | |
| 56 | − if (!ptep) | |
| 57 | − return NULL; | |
| 58 | − | |
| 59 | − hugepd_populate_kernel((hugepd_t *)pmdp, ptep, PAGE_SHIFT_8M); | |
| 60 | − hugepd_populate_kernel((hugepd_t *)pmdp + 1, ptep, PAGE_SHIFT_8M); | |
| 61 | − } | |
| 62 | − return hugepte_offset(*(hugepd_t *)pmdp, va, PGDIR_SHIFT); | |
| 63 | −} | |
| 64 | − | |
| 65 | 52 | static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa, |
| 66 | 53 | pgprot_t prot, int psize, bool new) |
| 67 | 54 | { |
| @@ -75,26 +62,36 @@ static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa, · 7 unmodified lines | ||
| 75 | 62 | if (WARN_ON(slab_is_available())) |
| 76 | 63 | return -EINVAL; |
| 77 | 64 | |
| 78 | − if (psize == MMU_PAGE_512K) | |
| 65 | + if (psize == MMU_PAGE_512K) { | |
| 79 | 66 | ptep = early_pte_alloc_kernel(pmdp, va); |
| 80 | − else | |
| 81 | − ptep = early_hugepd_alloc_kernel((hugepd_t *)pmdp, va); | |
| 67 | + /* The PTE should never be already present */ | |
| 68 | + if (WARN_ON(pte_present(*ptep) && pgprot_val(prot))) | |
| 69 | + return -EINVAL; | |
| 70 | + } else { | |
| 71 | + if (WARN_ON(!pmd_none(*pmdp) || !pmd_none(*(pmdp + 1)))) | |
| 72 | + return -EINVAL; | |
| 73 | + | |
| 74 | + ptep = early_alloc_pgtable(PTE_FRAG_SIZE); | |
| 75 | + pmd_populate_kernel(&init_mm, pmdp, ptep); | |
| 76 | + | |
| 77 | + ptep = early_alloc_pgtable(PTE_FRAG_SIZE); | |
| 78 | + pmd_populate_kernel(&init_mm, pmdp + 1, ptep); | |
| 79 | + | |
| 80 | + ptep = (pte_t *)pmdp; | |
| 81 | + } | |
| 82 | 82 | } else { |
| 83 | 83 | if (psize == MMU_PAGE_512K) |
| 84 | 84 | ptep = pte_offset_kernel(pmdp, va); |
| 85 | 85 | else |
| 86 | − ptep = hugepte_offset(*(hugepd_t *)pmdp, va, PGDIR_SHIFT); | |
| 86 | + ptep = (pte_t *)pmdp; | |
| 87 | 87 | } |
| 88 | 88 | |
| 89 | 89 | if (WARN_ON(!ptep)) |
| 90 | 90 | return -ENOMEM; |
| 91 | 91 | |
| 92 | − /* The PTE should never be already present */ | |
| 93 | − if (new && WARN_ON(pte_present(*ptep) && pgprot_val(prot))) | |
| 94 | − return -EINVAL; | |
| 95 | − | |
| 96 | 92 | set_huge_pte_at(&init_mm, va, ptep, |
| 97 | − pte_mkhuge(pfn_pte(pa >> PAGE_SHIFT, prot)), psize); | |
| 93 | + pte_mkhuge(pfn_pte(pa >> PAGE_SHIFT, prot)), | |
| 94 | + 1UL << mmu_psize_to_shift(psize)); | |
| 98 | 95 | |
| 99 | 96 | return 0; |
| 100 | 97 | } |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | |
| 3 | 3 | obj-y += mmu_context.o tlb.o tlb_low.o kup.o |
| 4 | −obj-$(CONFIG_PPC_BOOK3E_64) += tlb_low_64e.o book3e_pgtable.o | |
| 4 | +obj-$(CONFIG_PPC_BOOK3E_64) += tlb_64e.o tlb_low_64e.o book3e_pgtable.o | |
| 5 | 5 | obj-$(CONFIG_44x) += 44x.o |
| 6 | 6 | obj-$(CONFIG_PPC_8xx) += 8xx.o |
| 7 | 7 | obj-$(CONFIG_PPC_E500) += e500.o |
| @@ -29,10 +29,10 @@ int __meminit vmemmap_create_mapping(unsigned long start, · 28 unmodified lines | ||
| 29 | 29 | _PAGE_KERNEL_RW; |
| 30 | 30 | |
| 31 | 31 | /* PTEs only contain page size encodings up to 32M */ |
| 32 | − BUG_ON(mmu_psize_defs[mmu_vmemmap_psize].enc > 0xf); | |
| 32 | + BUG_ON(mmu_psize_defs[mmu_vmemmap_psize].shift - 10 > 0xf); | |
| 33 | 33 | |
| 34 | 34 | /* Encode the size in the PTE */ |
| 35 | − flags |= mmu_psize_defs[mmu_vmemmap_psize].enc << 8; | |
| 35 | + flags |= (mmu_psize_defs[mmu_vmemmap_psize].shift - 10) << 8; | |
| 36 | 36 | |
| 37 | 37 | /* For each PTE for that area, map things. Note that we don't |
| 38 | 38 | * increment phys because all PTEs are of the large size and |
| @@ -53,37 +53,30 @@ · 52 unmodified lines | ||
| 53 | 53 | struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { |
| 54 | 54 | [MMU_PAGE_4K] = { |
| 55 | 55 | .shift = 12, |
| 56 | − .enc = BOOK3E_PAGESZ_4K, | |
| 57 | 56 | }, |
| 58 | 57 | [MMU_PAGE_2M] = { |
| 59 | 58 | .shift = 21, |
| 60 | − .enc = BOOK3E_PAGESZ_2M, | |
| 61 | 59 | }, |
| 62 | 60 | [MMU_PAGE_4M] = { |
| 63 | 61 | .shift = 22, |
| 64 | − .enc = BOOK3E_PAGESZ_4M, | |
| 65 | 62 | }, |
| 66 | 63 | [MMU_PAGE_16M] = { |
| 67 | 64 | .shift = 24, |
| 68 | − .enc = BOOK3E_PAGESZ_16M, | |
| 69 | 65 | }, |
| 70 | 66 | [MMU_PAGE_64M] = { |
| 71 | 67 | .shift = 26, |
| 72 | − .enc = BOOK3E_PAGESZ_64M, | |
| 73 | 68 | }, |
| 74 | 69 | [MMU_PAGE_256M] = { |
| 75 | 70 | .shift = 28, |
| 76 | − .enc = BOOK3E_PAGESZ_256M, | |
| 77 | 71 | }, |
| 78 | 72 | [MMU_PAGE_1G] = { |
| 79 | 73 | .shift = 30, |
| 80 | − .enc = BOOK3E_PAGESZ_1GB, | |
| 81 | 74 | }, |
| 82 | 75 | }; |
| 83 | 76 | |
| 84 | 77 | static inline int mmu_get_tsize(int psize) |
| 85 | 78 | { |
| 86 | − return mmu_psize_defs[psize].enc; | |
| 79 | + return mmu_psize_defs[psize].shift - 10; | |
| 87 | 80 | } |
| 88 | 81 | #else |
| 89 | 82 | static inline int mmu_get_tsize(int psize) |
| @@ -110,28 +103,6 @@ struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { · 20 unmodified lines | ||
| 110 | 103 | }; |
| 111 | 104 | #endif |
| 112 | 105 | |
| 113 | −/* The variables below are currently only used on 64-bit Book3E | |
| 114 | − * though this will probably be made common with other nohash | |
| 115 | − * implementations at some point | |
| 116 | − */ | |
| 117 | −#ifdef CONFIG_PPC64 | |
| 118 | − | |
| 119 | −int mmu_pte_psize; /* Page size used for PTE pages */ | |
| 120 | −int mmu_vmemmap_psize; /* Page size used for the virtual mem map */ | |
| 121 | −int book3e_htw_mode; /* HW tablewalk? Value is PPC_HTW_* */ | |
| 122 | −unsigned long linear_map_top; /* Top of linear mapping */ | |
| 123 | − | |
| 124 | − | |
| 125 | −/* | |
| 126 | − * Number of bytes to add to SPRN_SPRG_TLB_EXFRAME on crit/mcheck/debug | |
| 127 | − * exceptions. This is used for bolted and e6500 TLB miss handlers which | |
| 128 | − * do not modify this SPRG in the TLB miss code; for other TLB miss handlers, | |
| 129 | − * this is set to zero. | |
| 130 | − */ | |
| 131 | −int extlb_level_exc; | |
| 132 | − | |
| 133 | −#endif /* CONFIG_PPC64 */ | |
| 134 | − | |
| 135 | 106 | #ifdef CONFIG_PPC_E500 |
| 136 | 107 | /* next_tlbcam_idx is used to round-robin tlbcam entry assignment */ |
| 137 | 108 | DEFINE_PER_CPU(int, next_tlbcam_idx); |
| @@ -358,381 +329,7 @@ void tlb_flush(struct mmu_gather *tlb) · 220 unmodified lines | ||
| 358 | 329 | flush_tlb_mm(tlb->mm); |
| 359 | 330 | } |
| 360 | 331 | |
| 361 | −/* | |
| 362 | − * Below are functions specific to the 64-bit variant of Book3E though that | |
| 363 | − * may change in the future | |
| 364 | − */ | |
| 365 | − | |
| 366 | −#ifdef CONFIG_PPC64 | |
| 367 | − | |
| 368 | −/* | |
| 369 | − * Handling of virtual linear page tables or indirect TLB entries | |
| 370 | − * flushing when PTE pages are freed | |
| 371 | − */ | |
| 372 | −void tlb_flush_pgtable(struct mmu_gather *tlb, unsigned long address) | |
| 373 | −{ | |
| 374 | − int tsize = mmu_psize_defs[mmu_pte_psize].enc; | |
| 375 | − | |
| 376 | − if (book3e_htw_mode != PPC_HTW_NONE) { | |
| 377 | − unsigned long start = address & PMD_MASK; | |
| 378 | − unsigned long end = address + PMD_SIZE; | |
| 379 | − unsigned long size = 1UL << mmu_psize_defs[mmu_pte_psize].shift; | |
| 380 | − | |
| 381 | − /* This isn't the most optimal, ideally we would factor out the | |
| 382 | − * while preempt & CPU mask mucking around, or even the IPI but | |
| 383 | − * it will do for now | |
| 384 | − */ | |
| 385 | − while (start < end) { | |
| 386 | − __flush_tlb_page(tlb->mm, start, tsize, 1); | |
| 387 | − start += size; | |
| 388 | − } | |
| 389 | − } else { | |
| 390 | − unsigned long rmask = 0xf000000000000000ul; | |
| 391 | − unsigned long rid = (address & rmask) | 0x1000000000000000ul; | |
| 392 | − unsigned long vpte = address & ~rmask; | |
| 393 | − | |
| 394 | − vpte = (vpte >> (PAGE_SHIFT - 3)) & ~0xffful; | |
| 395 | − vpte |= rid; | |
| 396 | − __flush_tlb_page(tlb->mm, vpte, tsize, 0); | |
| 397 | − } | |
| 398 | −} | |
| 399 | − | |
| 400 | −static void __init setup_page_sizes(void) | |
| 401 | −{ | |
| 402 | − unsigned int tlb0cfg; | |
| 403 | − unsigned int tlb0ps; | |
| 404 | − unsigned int eptcfg; | |
| 405 | − int i, psize; | |
| 406 | − | |
| 407 | −#ifdef CONFIG_PPC_E500 | |
| 408 | − unsigned int mmucfg = mfspr(SPRN_MMUCFG); | |
| 409 | − int fsl_mmu = mmu_has_feature(MMU_FTR_TYPE_FSL_E); | |
| 410 | − | |
| 411 | − if (fsl_mmu && (mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V1) { | |
| 412 | − unsigned int tlb1cfg = mfspr(SPRN_TLB1CFG); | |
| 413 | − unsigned int min_pg, max_pg; | |
| 414 | − | |
| 415 | − min_pg = (tlb1cfg & TLBnCFG_MINSIZE) >> TLBnCFG_MINSIZE_SHIFT; | |
| 416 | − max_pg = (tlb1cfg & TLBnCFG_MAXSIZE) >> TLBnCFG_MAXSIZE_SHIFT; | |
| 417 | − | |
| 418 | − for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { | |
| 419 | − struct mmu_psize_def *def; | |
| 420 | − unsigned int shift; | |
| 421 | − | |
| 422 | − def = &mmu_psize_defs[psize]; | |
| 423 | − shift = def->shift; | |
| 424 | − | |
| 425 | − if (shift == 0 || shift & 1) | |
| 426 | − continue; | |
| 427 | − | |
| 428 | − /* adjust to be in terms of 4^shift Kb */ | |
| 429 | − shift = (shift - 10) >> 1; | |
| 430 | − | |
| 431 | − if ((shift >= min_pg) && (shift <= max_pg)) | |
| 432 | − def->flags |= MMU_PAGE_SIZE_DIRECT; | |
| 433 | − } | |
| 434 | − | |
| 435 | − goto out; | |
| 436 | − } | |
| 437 | − | |
| 438 | − if (fsl_mmu && (mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V2) { | |
| 439 | − u32 tlb1cfg, tlb1ps; | |
| 440 | − | |
| 441 | − tlb0cfg = mfspr(SPRN_TLB0CFG); | |
| 442 | − tlb1cfg = mfspr(SPRN_TLB1CFG); | |
| 443 | − tlb1ps = mfspr(SPRN_TLB1PS); | |
| 444 | − eptcfg = mfspr(SPRN_EPTCFG); | |
| 445 | − | |
| 446 | − if ((tlb1cfg & TLBnCFG_IND) && (tlb0cfg & TLBnCFG_PT)) | |
| 447 | − book3e_htw_mode = PPC_HTW_E6500; | |
| 448 | − | |
| 449 | − /* | |
| 450 | − * We expect 4K subpage size and unrestricted indirect size. | |
| 451 | − * The lack of a restriction on indirect size is a Freescale | |
| 452 | − * extension, indicated by PSn = 0 but SPSn != 0. | |
| 453 | − */ | |
| 454 | − if (eptcfg != 2) | |
| 455 | − book3e_htw_mode = PPC_HTW_NONE; | |
| 456 | − | |
| 457 | − for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { | |
| 458 | − struct mmu_psize_def *def = &mmu_psize_defs[psize]; | |
| 459 | − | |
| 460 | − if (!def->shift) | |
| 461 | − continue; | |
| 462 | − | |
| 463 | − if (tlb1ps & (1U << (def->shift - 10))) { | |
| 464 | − def->flags |= MMU_PAGE_SIZE_DIRECT; | |
| 465 | − | |
| 466 | − if (book3e_htw_mode && psize == MMU_PAGE_2M) | |
| 467 | − def->flags |= MMU_PAGE_SIZE_INDIRECT; | |
| 468 | − } | |
| 469 | − } | |
| 470 | − | |
| 471 | − goto out; | |
| 472 | − } | |
| 473 | −#endif | |
| 474 | − | |
| 475 | − tlb0cfg = mfspr(SPRN_TLB0CFG); | |
| 476 | − tlb0ps = mfspr(SPRN_TLB0PS); | |
| 477 | − eptcfg = mfspr(SPRN_EPTCFG); | |
| 478 | − | |
| 479 | − /* Look for supported direct sizes */ | |
| 480 | − for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { | |
| 481 | − struct mmu_psize_def *def = &mmu_psize_defs[psize]; | |
| 482 | − | |
| 483 | − if (tlb0ps & (1U << (def->shift - 10))) | |
| 484 | − def->flags |= MMU_PAGE_SIZE_DIRECT; | |
| 485 | − } | |
| 486 | − | |
| 487 | − /* Indirect page sizes supported ? */ | |
| 488 | − if ((tlb0cfg & TLBnCFG_IND) == 0 || | |
| 489 | − (tlb0cfg & TLBnCFG_PT) == 0) | |
| 490 | − goto out; | |
| 491 | − | |
| 492 | − book3e_htw_mode = PPC_HTW_IBM; | |
| 493 | − | |
| 494 | − /* Now, we only deal with one IND page size for each | |
| 495 | − * direct size. Hopefully all implementations today are | |
| 496 | − * unambiguous, but we might want to be careful in the | |
| 497 | − * future. | |
| 498 | − */ | |
| 499 | − for (i = 0; i < 3; i++) { | |
| 500 | − unsigned int ps, sps; | |
| 501 | − | |
| 502 | − sps = eptcfg & 0x1f; | |
| 503 | − eptcfg >>= 5; | |
| 504 | − ps = eptcfg & 0x1f; | |
| 505 | − eptcfg >>= 5; | |
| 506 | − if (!ps || !sps) | |
| 507 | − continue; | |
| 508 | − for (psize = 0; psize < MMU_PAGE_COUNT; psize++) { | |
| 509 | − struct mmu_psize_def *def = &mmu_psize_defs[psize]; | |
| 510 | − | |
| 511 | − if (ps == (def->shift - 10)) | |
| 512 | − def->flags |= MMU_PAGE_SIZE_INDIRECT; | |
| 513 | − if (sps == (def->shift - 10)) | |
| 514 | − def->ind = ps + 10; | |
| 515 | − } | |
| 516 | − } | |
| 517 | − | |
| 518 | −out: | |
| 519 | − /* Cleanup array and print summary */ | |
| 520 | − pr_info("MMU: Supported page sizes\n"); | |
| 521 | − for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { | |
| 522 | − struct mmu_psize_def *def = &mmu_psize_defs[psize]; | |
| 523 | − const char *__page_type_names[] = { | |
| 524 | − "unsupported", | |
| 525 | − "direct", | |
| 526 | − "indirect", | |
| 527 | − "direct & indirect" | |
| 528 | − }; | |
| 529 | − if (def->flags == 0) { | |
| 530 | − def->shift = 0; | |
| 531 | − continue; | |
| 532 | − } | |
| 533 | − pr_info(" %8ld KB as %s\n", 1ul << (def->shift - 10), | |
| 534 | − __page_type_names[def->flags & 0x3]); | |
| 535 | − } | |
| 536 | −} | |
| 537 | − | |
| 538 | −static void __init setup_mmu_htw(void) | |
| 539 | −{ | |
| 540 | − /* | |
| 541 | − * If we want to use HW tablewalk, enable it by patching the TLB miss | |
| 542 | − * handlers to branch to the one dedicated to it. | |
| 543 | − */ | |
| 544 | − | |
| 545 | − switch (book3e_htw_mode) { | |
| 546 | − case PPC_HTW_IBM: | |
| 547 | − patch_exception(0x1c0, exc_data_tlb_miss_htw_book3e); | |
| 548 | − patch_exception(0x1e0, exc_instruction_tlb_miss_htw_book3e); | |
| 549 | − break; | |
| 550 | −#ifdef CONFIG_PPC_E500 | |
| 551 | − case PPC_HTW_E6500: | |
| 552 | − extlb_level_exc = EX_TLB_SIZE; | |
| 553 | − patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e); | |
| 554 | − patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e); | |
| 555 | − break; | |
| 556 | −#endif | |
| 557 | − } | |
| 558 | − pr_info("MMU: Book3E HW tablewalk %s\n", | |
| 559 | − book3e_htw_mode != PPC_HTW_NONE ? "enabled" : "not supported"); | |
| 560 | −} | |
| 561 | − | |
| 562 | −/* | |
| 563 | − * Early initialization of the MMU TLB code | |
| 564 | − */ | |
| 565 | −static void early_init_this_mmu(void) | |
| 566 | −{ | |
| 567 | − unsigned int mas4; | |
| 568 | − | |
| 569 | − /* Set MAS4 based on page table setting */ | |
| 570 | − | |
| 571 | − mas4 = 0x4 << MAS4_WIMGED_SHIFT; | |
| 572 | − switch (book3e_htw_mode) { | |
| 573 | − case PPC_HTW_E6500: | |
| 574 | − mas4 |= MAS4_INDD; | |
| 575 | − mas4 |= BOOK3E_PAGESZ_2M << MAS4_TSIZED_SHIFT; | |
| 576 | − mas4 |= MAS4_TLBSELD(1); | |
| 577 | − mmu_pte_psize = MMU_PAGE_2M; | |
| 578 | − break; | |
| 579 | − | |
| 580 | − case PPC_HTW_IBM: | |
| 581 | − mas4 |= MAS4_INDD; | |
| 582 | − mas4 |= BOOK3E_PAGESZ_1M << MAS4_TSIZED_SHIFT; | |
| 583 | − mmu_pte_psize = MMU_PAGE_1M; | |
| 584 | − break; | |
| 585 | − | |
| 586 | − case PPC_HTW_NONE: | |
| 587 | − mas4 |= BOOK3E_PAGESZ_4K << MAS4_TSIZED_SHIFT; | |
| 588 | − mmu_pte_psize = mmu_virtual_psize; | |
| 589 | − break; | |
| 590 | − } | |
| 591 | − mtspr(SPRN_MAS4, mas4); | |
| 592 | − | |
| 593 | −#ifdef CONFIG_PPC_E500 | |
| 594 | − if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { | |
| 595 | − unsigned int num_cams; | |
| 596 | − bool map = true; | |
| 597 | − | |
| 598 | − /* use a quarter of the TLBCAM for bolted linear map */ | |
| 599 | − num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; | |
| 600 | − | |
| 601 | − /* | |
| 602 | − * Only do the mapping once per core, or else the | |
| 603 | − * transient mapping would cause problems. | |
| 604 | − */ | |
| 605 | −#ifdef CONFIG_SMP | |
| 606 | − if (hweight32(get_tensr()) > 1) | |
| 607 | − map = false; | |
| 608 | −#endif | |
| 609 | − | |
| 610 | − if (map) | |
| 611 | − linear_map_top = map_mem_in_cams(linear_map_top, | |
| 612 | − num_cams, false, true); | |
| 613 | − } | |
| 614 | −#endif | |
| 615 | − | |
| 616 | − /* A sync won't hurt us after mucking around with | |
| 617 | − * the MMU configuration | |
| 618 | − */ | |
| 619 | − mb(); | |
| 620 | −} | |
| 621 | − | |
| 622 | −static void __init early_init_mmu_global(void) | |
| 623 | −{ | |
| 624 | − /* XXX This should be decided at runtime based on supported | |
| 625 | − * page sizes in the TLB, but for now let's assume 16M is | |
| 626 | − * always there and a good fit (which it probably is) | |
| 627 | − * | |
| 628 | − * Freescale booke only supports 4K pages in TLB0, so use that. | |
| 629 | − */ | |
| 630 | − if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) | |
| 631 | − mmu_vmemmap_psize = MMU_PAGE_4K; | |
| 632 | − else | |
| 633 | − mmu_vmemmap_psize = MMU_PAGE_16M; | |
| 634 | − | |
| 635 | − /* XXX This code only checks for TLB 0 capabilities and doesn't | |
| 636 | − * check what page size combos are supported by the HW. It | |
| 637 | − * also doesn't handle the case where a separate array holds | |
| 638 | − * the IND entries from the array loaded by the PT. | |
| 639 | − */ | |
| 640 | − /* Look for supported page sizes */ | |
| 641 | − setup_page_sizes(); | |
| 642 | − | |
| 643 | − /* Look for HW tablewalk support */ | |
| 644 | − setup_mmu_htw(); | |
| 645 | − | |
| 646 | −#ifdef CONFIG_PPC_E500 | |
| 647 | − if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { | |
| 648 | − if (book3e_htw_mode == PPC_HTW_NONE) { | |
| 649 | − extlb_level_exc = EX_TLB_SIZE; | |
| 650 | − patch_exception(0x1c0, exc_data_tlb_miss_bolted_book3e); | |
| 651 | − patch_exception(0x1e0, | |
| 652 | − exc_instruction_tlb_miss_bolted_book3e); | |
| 653 | − } | |
| 654 | − } | |
| 655 | −#endif | |
| 656 | − | |
| 657 | − /* Set the global containing the top of the linear mapping | |
| 658 | − * for use by the TLB miss code | |
| 659 | − */ | |
| 660 | − linear_map_top = memblock_end_of_DRAM(); | |
| 661 | − | |
| 662 | − ioremap_bot = IOREMAP_BASE; | |
| 663 | −} | |
| 664 | − | |
| 665 | −static void __init early_mmu_set_memory_limit(void) | |
| 666 | −{ | |
| 667 | −#ifdef CONFIG_PPC_E500 | |
| 668 | − if (mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { | |
| 669 | − /* | |
| 670 | − * Limit memory so we dont have linear faults. | |
| 671 | − * Unlike memblock_set_current_limit, which limits | |
| 672 | − * memory available during early boot, this permanently | |
| 673 | − * reduces the memory available to Linux. We need to | |
| 674 | − * do this because highmem is not supported on 64-bit. | |
| 675 | − */ | |
| 676 | − memblock_enforce_memory_limit(linear_map_top); | |
| 677 | − } | |
| 678 | −#endif | |
| 679 | − | |
| 680 | − memblock_set_current_limit(linear_map_top); | |
| 681 | −} | |
| 682 | − | |
| 683 | −/* boot cpu only */ | |
| 684 | −void __init early_init_mmu(void) | |
| 685 | −{ | |
| 686 | − early_init_mmu_global(); | |
| 687 | − early_init_this_mmu(); | |
| 688 | − early_mmu_set_memory_limit(); | |
| 689 | −} | |
| 690 | − | |
| 691 | −void early_init_mmu_secondary(void) | |
| 692 | −{ | |
| 693 | − early_init_this_mmu(); | |
| 694 | −} | |
| 695 | − | |
| 696 | −void setup_initial_memory_limit(phys_addr_t first_memblock_base, | |
| 697 | − phys_addr_t first_memblock_size) | |
| 698 | −{ | |
| 699 | − /* On non-FSL Embedded 64-bit, we adjust the RMA size to match | |
| 700 | − * the bolted TLB entry. We know for now that only 1G | |
| 701 | − * entries are supported though that may eventually | |
| 702 | − * change. | |
| 703 | − * | |
| 704 | − * on FSL Embedded 64-bit, usually all RAM is bolted, but with | |
| 705 | − * unusual memory sizes it's possible for some RAM to not be mapped | |
| 706 | − * (such RAM is not used at all by Linux, since we don't support | |
| 707 | − * highmem on 64-bit). We limit ppc64_rma_size to what would be | |
| 708 | − * mappable if this memblock is the only one. Additional memblocks | |
| 709 | − * can only increase, not decrease, the amount that ends up getting | |
| 710 | − * mapped. We still limit max to 1G even if we'll eventually map | |
| 711 | − * more. This is due to what the early init code is set up to do. | |
| 712 | − * | |
| 713 | − * We crop it to the size of the first MEMBLOCK to | |
| 714 | − * avoid going over total available memory just in case... | |
| 715 | − */ | |
| 716 | −#ifdef CONFIG_PPC_E500 | |
| 717 | − if (early_mmu_has_feature(MMU_FTR_TYPE_FSL_E)) { | |
| 718 | − unsigned long linear_sz; | |
| 719 | − unsigned int num_cams; | |
| 720 | − | |
| 721 | − /* use a quarter of the TLBCAM for bolted linear map */ | |
| 722 | − num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; | |
| 723 | − | |
| 724 | − linear_sz = map_mem_in_cams(first_memblock_size, num_cams, | |
| 725 | − true, true); | |
| 726 | − | |
| 727 | − ppc64_rma_size = min_t(u64, linear_sz, 0x40000000); | |
| 728 | − } else | |
| 729 | −#endif | |
| 730 | − ppc64_rma_size = min_t(u64, first_memblock_size, 0x40000000); | |
| 731 | − | |
| 732 | − /* Finally limit subsequent allocations */ | |
| 733 | − memblock_set_current_limit(first_memblock_base + ppc64_rma_size); | |
| 734 | −} | |
| 735 | −#else /* ! CONFIG_PPC64 */ | |
| 332 | +#ifndef CONFIG_PPC64 | |
| 736 | 333 | void __init early_init_mmu(void) |
| 737 | 334 | { |
| 738 | 335 | unsigned long root = of_get_flat_dt_root(); |
| @@ -0,0 +1,314 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-or-later | |
| 2 | +/* | |
| 3 | + * Copyright 2008,2009 Ben Herrenschmidt <benh@kernel.crashing.org> | |
| 4 | + * IBM Corp. | |
| 5 | + * | |
| 6 | + * Derived from arch/ppc/mm/init.c: | |
| 7 | + * Copyright (C) 1995-1996 Gary Thomas (gdt@linuxppc.org) | |
| 8 | + * | |
| 9 | + * Modifications by Paul Mackerras (PowerMac) (paulus@cs.anu.edu.au) | |
| 10 | + * and Cort Dougan (PReP) (cort@cs.nmt.edu) | |
| 11 | + * Copyright (C) 1996 Paul Mackerras | |
| 12 | + * | |
| 13 | + * Derived from "arch/i386/mm/init.c" | |
| 14 | + * Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds | |
| 15 | + */ | |
| 16 | + | |
| 17 | +#include <linux/kernel.h> | |
| 18 | +#include <linux/export.h> | |
| 19 | +#include <linux/mm.h> | |
| 20 | +#include <linux/init.h> | |
| 21 | +#include <linux/pagemap.h> | |
| 22 | +#include <linux/memblock.h> | |
| 23 | + | |
| 24 | +#include <asm/pgalloc.h> | |
| 25 | +#include <asm/tlbflush.h> | |
| 26 | +#include <asm/tlb.h> | |
| 27 | +#include <asm/code-patching.h> | |
| 28 | +#include <asm/cputhreads.h> | |
| 29 | + | |
| 30 | +#include <mm/mmu_decl.h> | |
| 31 | + | |
| 32 | +/* The variables below are currently only used on 64-bit Book3E | |
| 33 | + * though this will probably be made common with other nohash | |
| 34 | + * implementations at some point | |
| 35 | + */ | |
| 36 | +int mmu_pte_psize; /* Page size used for PTE pages */ | |
| 37 | +int mmu_vmemmap_psize; /* Page size used for the virtual mem map */ | |
| 38 | +int book3e_htw_mode; /* HW tablewalk? Value is PPC_HTW_* */ | |
| 39 | +unsigned long linear_map_top; /* Top of linear mapping */ | |
| 40 | + | |
| 41 | + | |
| 42 | +/* | |
| 43 | + * Number of bytes to add to SPRN_SPRG_TLB_EXFRAME on crit/mcheck/debug | |
| 44 | + * exceptions. This is used for bolted and e6500 TLB miss handlers which | |
| 45 | + * do not modify this SPRG in the TLB miss code; for other TLB miss handlers, | |
| 46 | + * this is set to zero. | |
| 47 | + */ | |
| 48 | +int extlb_level_exc; | |
| 49 | + | |
| 50 | +/* | |
| 51 | + * Handling of virtual linear page tables or indirect TLB entries | |
| 52 | + * flushing when PTE pages are freed | |
| 53 | + */ | |
| 54 | +void tlb_flush_pgtable(struct mmu_gather *tlb, unsigned long address) | |
| 55 | +{ | |
| 56 | + int tsize = mmu_psize_defs[mmu_pte_psize].shift - 10; | |
| 57 | + | |
| 58 | + if (book3e_htw_mode != PPC_HTW_NONE) { | |
| 59 | + unsigned long start = address & PMD_MASK; | |
| 60 | + unsigned long end = address + PMD_SIZE; | |
| 61 | + unsigned long size = 1UL << mmu_psize_defs[mmu_pte_psize].shift; | |
| 62 | + | |
| 63 | + /* This isn't the most optimal, ideally we would factor out the | |
| 64 | + * while preempt & CPU mask mucking around, or even the IPI but | |
| 65 | + * it will do for now | |
| 66 | + */ | |
| 67 | + while (start < end) { | |
| 68 | + __flush_tlb_page(tlb->mm, start, tsize, 1); | |
| 69 | + start += size; | |
| 70 | + } | |
| 71 | + } else { | |
| 72 | + unsigned long rmask = 0xf000000000000000ul; | |
| 73 | + unsigned long rid = (address & rmask) | 0x1000000000000000ul; | |
| 74 | + unsigned long vpte = address & ~rmask; | |
| 75 | + | |
| 76 | + vpte = (vpte >> (PAGE_SHIFT - 3)) & ~0xffful; | |
| 77 | + vpte |= rid; | |
| 78 | + __flush_tlb_page(tlb->mm, vpte, tsize, 0); | |
| 79 | + } | |
| 80 | +} | |
| 81 | + | |
| 82 | +static void __init setup_page_sizes(void) | |
| 83 | +{ | |
| 84 | + unsigned int tlb0cfg; | |
| 85 | + unsigned int eptcfg; | |
| 86 | + int psize; | |
| 87 | + | |
| 88 | + unsigned int mmucfg = mfspr(SPRN_MMUCFG); | |
| 89 | + | |
| 90 | + if ((mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V1) { | |
| 91 | + unsigned int tlb1cfg = mfspr(SPRN_TLB1CFG); | |
| 92 | + unsigned int min_pg, max_pg; | |
| 93 | + | |
| 94 | + min_pg = (tlb1cfg & TLBnCFG_MINSIZE) >> TLBnCFG_MINSIZE_SHIFT; | |
| 95 | + max_pg = (tlb1cfg & TLBnCFG_MAXSIZE) >> TLBnCFG_MAXSIZE_SHIFT; | |
| 96 | + | |
| 97 | + for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { | |
| 98 | + struct mmu_psize_def *def; | |
| 99 | + unsigned int shift; | |
| 100 | + | |
| 101 | + def = &mmu_psize_defs[psize]; | |
| 102 | + shift = def->shift; | |
| 103 | + | |
| 104 | + if (shift == 0 || shift & 1) | |
| 105 | + continue; | |
| 106 | + | |
| 107 | + /* adjust to be in terms of 4^shift Kb */ | |
| 108 | + shift = (shift - 10) >> 1; | |
| 109 | + | |
| 110 | + if ((shift >= min_pg) && (shift <= max_pg)) | |
| 111 | + def->flags |= MMU_PAGE_SIZE_DIRECT; | |
| 112 | + } | |
| 113 | + | |
| 114 | + goto out; | |
| 115 | + } | |
| 116 | + | |
| 117 | + if ((mmucfg & MMUCFG_MAVN) == MMUCFG_MAVN_V2) { | |
| 118 | + u32 tlb1cfg, tlb1ps; | |
| 119 | + | |
| 120 | + tlb0cfg = mfspr(SPRN_TLB0CFG); | |
| 121 | + tlb1cfg = mfspr(SPRN_TLB1CFG); | |
| 122 | + tlb1ps = mfspr(SPRN_TLB1PS); | |
| 123 | + eptcfg = mfspr(SPRN_EPTCFG); | |
| 124 | + | |
| 125 | + if ((tlb1cfg & TLBnCFG_IND) && (tlb0cfg & TLBnCFG_PT)) | |
| 126 | + book3e_htw_mode = PPC_HTW_E6500; | |
| 127 | + | |
| 128 | + /* | |
| 129 | + * We expect 4K subpage size and unrestricted indirect size. | |
| 130 | + * The lack of a restriction on indirect size is a Freescale | |
| 131 | + * extension, indicated by PSn = 0 but SPSn != 0. | |
| 132 | + */ | |
| 133 | + if (eptcfg != 2) | |
| 134 | + book3e_htw_mode = PPC_HTW_NONE; | |
| 135 | + | |
| 136 | + for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { | |
| 137 | + struct mmu_psize_def *def = &mmu_psize_defs[psize]; | |
| 138 | + | |
| 139 | + if (!def->shift) | |
| 140 | + continue; | |
| 141 | + | |
| 142 | + if (tlb1ps & (1U << (def->shift - 10))) { | |
| 143 | + def->flags |= MMU_PAGE_SIZE_DIRECT; | |
| 144 | + | |
| 145 | + if (book3e_htw_mode && psize == MMU_PAGE_2M) | |
| 146 | + def->flags |= MMU_PAGE_SIZE_INDIRECT; | |
| 147 | + } | |
| 148 | + } | |
| 149 | + | |
| 150 | + goto out; | |
| 151 | + } | |
| 152 | +out: | |
| 153 | + /* Cleanup array and print summary */ | |
| 154 | + pr_info("MMU: Supported page sizes\n"); | |
| 155 | + for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) { | |
| 156 | + struct mmu_psize_def *def = &mmu_psize_defs[psize]; | |
| 157 | + const char *__page_type_names[] = { | |
| 158 | + "unsupported", | |
| 159 | + "direct", | |
| 160 | + "indirect", | |
| 161 | + "direct & indirect" | |
| 162 | + }; | |
| 163 | + if (def->flags == 0) { | |
| 164 | + def->shift = 0; | |
| 165 | + continue; | |
| 166 | + } | |
| 167 | + pr_info(" %8ld KB as %s\n", 1ul << (def->shift - 10), | |
| 168 | + __page_type_names[def->flags & 0x3]); | |
| 169 | + } | |
| 170 | +} | |
| 171 | + | |
| 172 | +/* | |
| 173 | + * Early initialization of the MMU TLB code | |
| 174 | + */ | |
| 175 | +static void early_init_this_mmu(void) | |
| 176 | +{ | |
| 177 | + unsigned int mas4; | |
| 178 | + | |
| 179 | + /* Set MAS4 based on page table setting */ | |
| 180 | + | |
| 181 | + mas4 = 0x4 << MAS4_WIMGED_SHIFT; | |
| 182 | + switch (book3e_htw_mode) { | |
| 183 | + case PPC_HTW_E6500: | |
| 184 | + mas4 |= MAS4_INDD; | |
| 185 | + mas4 |= BOOK3E_PAGESZ_2M << MAS4_TSIZED_SHIFT; | |
| 186 | + mas4 |= MAS4_TLBSELD(1); | |
| 187 | + mmu_pte_psize = MMU_PAGE_2M; | |
| 188 | + break; | |
| 189 | + | |
| 190 | + case PPC_HTW_NONE: | |
| 191 | + mas4 |= BOOK3E_PAGESZ_4K << MAS4_TSIZED_SHIFT; | |
| 192 | + mmu_pte_psize = mmu_virtual_psize; | |
| 193 | + break; | |
| 194 | + } | |
| 195 | + mtspr(SPRN_MAS4, mas4); | |
| 196 | + | |
| 197 | + unsigned int num_cams; | |
| 198 | + bool map = true; | |
| 199 | + | |
| 200 | + /* use a quarter of the TLBCAM for bolted linear map */ | |
| 201 | + num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; | |
| 202 | + | |
| 203 | + /* | |
| 204 | + * Only do the mapping once per core, or else the | |
| 205 | + * transient mapping would cause problems. | |
| 206 | + */ | |
| 207 | +#ifdef CONFIG_SMP | |
| 208 | + if (hweight32(get_tensr()) > 1) | |
| 209 | + map = false; | |
| 210 | +#endif | |
| 211 | + | |
| 212 | + if (map) | |
| 213 | + linear_map_top = map_mem_in_cams(linear_map_top, | |
| 214 | + num_cams, false, true); | |
| 215 | + | |
| 216 | + /* A sync won't hurt us after mucking around with | |
| 217 | + * the MMU configuration | |
| 218 | + */ | |
| 219 | + mb(); | |
| 220 | +} | |
| 221 | + | |
| 222 | +static void __init early_init_mmu_global(void) | |
| 223 | +{ | |
| 224 | + /* | |
| 225 | + * Freescale booke only supports 4K pages in TLB0, so use that. | |
| 226 | + */ | |
| 227 | + mmu_vmemmap_psize = MMU_PAGE_4K; | |
| 228 | + | |
| 229 | + /* XXX This code only checks for TLB 0 capabilities and doesn't | |
| 230 | + * check what page size combos are supported by the HW. It | |
| 231 | + * also doesn't handle the case where a separate array holds | |
| 232 | + * the IND entries from the array loaded by the PT. | |
| 233 | + */ | |
| 234 | + /* Look for supported page sizes */ | |
| 235 | + setup_page_sizes(); | |
| 236 | + | |
| 237 | + /* | |
| 238 | + * If we want to use HW tablewalk, enable it by patching the TLB miss | |
| 239 | + * handlers to branch to the one dedicated to it. | |
| 240 | + */ | |
| 241 | + extlb_level_exc = EX_TLB_SIZE; | |
| 242 | + switch (book3e_htw_mode) { | |
| 243 | + case PPC_HTW_E6500: | |
| 244 | + patch_exception(0x1c0, exc_data_tlb_miss_e6500_book3e); | |
| 245 | + patch_exception(0x1e0, exc_instruction_tlb_miss_e6500_book3e); | |
| 246 | + break; | |
| 247 | + } | |
| 248 | + | |
| 249 | + pr_info("MMU: Book3E HW tablewalk %s\n", | |
| 250 | + book3e_htw_mode != PPC_HTW_NONE ? "enabled" : "not supported"); | |
| 251 | + | |
| 252 | + /* Set the global containing the top of the linear mapping | |
| 253 | + * for use by the TLB miss code | |
| 254 | + */ | |
| 255 | + linear_map_top = memblock_end_of_DRAM(); | |
| 256 | + | |
| 257 | + ioremap_bot = IOREMAP_BASE; | |
| 258 | +} | |
| 259 | + | |
| 260 | +static void __init early_mmu_set_memory_limit(void) | |
| 261 | +{ | |
| 262 | + /* | |
| 263 | + * Limit memory so we dont have linear faults. | |
| 264 | + * Unlike memblock_set_current_limit, which limits | |
| 265 | + * memory available during early boot, this permanently | |
| 266 | + * reduces the memory available to Linux. We need to | |
| 267 | + * do this because highmem is not supported on 64-bit. | |
| 268 | + */ | |
| 269 | + memblock_enforce_memory_limit(linear_map_top); | |
| 270 | + | |
| 271 | + memblock_set_current_limit(linear_map_top); | |
| 272 | +} | |
| 273 | + | |
| 274 | +/* boot cpu only */ | |
| 275 | +void __init early_init_mmu(void) | |
| 276 | +{ | |
| 277 | + early_init_mmu_global(); | |
| 278 | + early_init_this_mmu(); | |
| 279 | + early_mmu_set_memory_limit(); | |
| 280 | +} | |
| 281 | + | |
| 282 | +void early_init_mmu_secondary(void) | |
| 283 | +{ | |
| 284 | + early_init_this_mmu(); | |
| 285 | +} | |
| 286 | + | |
| 287 | +void setup_initial_memory_limit(phys_addr_t first_memblock_base, | |
| 288 | + phys_addr_t first_memblock_size) | |
| 289 | +{ | |
| 290 | + /* | |
| 291 | + * On FSL Embedded 64-bit, usually all RAM is bolted, but with | |
| 292 | + * unusual memory sizes it's possible for some RAM to not be mapped | |
| 293 | + * (such RAM is not used at all by Linux, since we don't support | |
| 294 | + * highmem on 64-bit). We limit ppc64_rma_size to what would be | |
| 295 | + * mappable if this memblock is the only one. Additional memblocks | |
| 296 | + * can only increase, not decrease, the amount that ends up getting | |
| 297 | + * mapped. We still limit max to 1G even if we'll eventually map | |
| 298 | + * more. This is due to what the early init code is set up to do. | |
| 299 | + * | |
| 300 | + * We crop it to the size of the first MEMBLOCK to | |
| 301 | + * avoid going over total available memory just in case... | |
| 302 | + */ | |
| 303 | + unsigned long linear_sz; | |
| 304 | + unsigned int num_cams; | |
| 305 | + | |
| 306 | + /* use a quarter of the TLBCAM for bolted linear map */ | |
| 307 | + num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4; | |
| 308 | + | |
| 309 | + linear_sz = map_mem_in_cams(first_memblock_size, num_cams, true, true); | |
| 310 | + ppc64_rma_size = min_t(u64, linear_sz, 0x40000000); | |
| 311 | + | |
| 312 | + /* Finally limit subsequent allocations */ | |
| 313 | + memblock_set_current_limit(first_memblock_base + ppc64_rma_size); | |
| 314 | +} | |
| @@ -450,11 +450,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_SMT) · 449 unmodified lines | ||
| 450 | 450 | |
| 451 | 451 | tlb_miss_huge_e6500: |
| 452 | 452 | beq tlb_miss_fault_e6500 |
| 453 | − li r10,1 | |
| 454 | − andi. r15,r14,HUGEPD_SHIFT_MASK@l /* r15 = psize */ | |
| 455 | − rldimi r14,r10,63,0 /* Set PD_HUGE */ | |
| 456 | − xor r14,r14,r15 /* Clear size bits */ | |
| 457 | − ldx r14,0,r14 | |
| 453 | + rlwinm r15,r14,32-_PAGE_PSIZE_SHIFT,0x1e | |
| 458 | 454 | |
| 459 | 455 | /* |
| 460 | 456 | * Now we build the MAS for a huge page. |
| @@ -465,7 +461,6 @@ tlb_miss_huge_e6500: · 4 unmodified lines | ||
| 465 | 461 | * MAS 2,3+7: Needs to be redone similar to non-tablewalk handler |
| 466 | 462 | */ |
| 467 | 463 | |
| 468 | − subi r15,r15,10 /* Convert psize to tsize */ | |
| 469 | 464 | mfspr r10,SPRN_MAS1 |
| 470 | 465 | rlwinm r10,r10,0,~MAS1_IND |
| 471 | 466 | rlwimi r10,r15,MAS1_TSIZE_SHIFT,MAS1_TSIZE_MASK |
| @@ -511,232 +506,6 @@ itlb_miss_fault_e6500: · 39 unmodified lines | ||
| 511 | 506 | tlb_epilog_bolted |
| 512 | 507 | b exc_instruction_storage_book3e |
| 513 | 508 | |
| 514 | −/********************************************************************** | |
| 515 | − * * | |
| 516 | − * TLB miss handling for Book3E with TLB reservation and HES support * | |
| 517 | − * * | |
| 518 | − **********************************************************************/ | |
| 519 | − | |
| 520 | − | |
| 521 | −/* Data TLB miss */ | |
| 522 | − START_EXCEPTION(data_tlb_miss) | |
| 523 | − TLB_MISS_PROLOG | |
| 524 | − | |
| 525 | − /* Now we handle the fault proper. We only save DEAR in normal | |
| 526 | − * fault case since that's the only interesting values here. | |
| 527 | − * We could probably also optimize by not saving SRR0/1 in the | |
| 528 | − * linear mapping case but I'll leave that for later | |
| 529 | − */ | |
| 530 | − mfspr r14,SPRN_ESR | |
| 531 | − mfspr r16,SPRN_DEAR /* get faulting address */ | |
| 532 | − srdi r15,r16,44 /* get region */ | |
| 533 | − xoris r15,r15,0xc | |
| 534 | − cmpldi cr0,r15,0 /* linear mapping ? */ | |
| 535 | − beq tlb_load_linear /* yes -> go to linear map load */ | |
| 536 | − cmpldi cr1,r15,1 /* vmalloc mapping ? */ | |
| 537 | − | |
| 538 | − /* The page tables are mapped virtually linear. At this point, though, | |
| 539 | − * we don't know whether we are trying to fault in a first level | |
| 540 | − * virtual address or a virtual page table address. We can get that | |
| 541 | − * from bit 0x1 of the region ID which we have set for a page table | |
| 542 | − */ | |
| 543 | − andis. r10,r15,0x1 | |
| 544 | − bne- virt_page_table_tlb_miss | |
| 545 | − | |
| 546 | − std r14,EX_TLB_ESR(r12); /* save ESR */ | |
| 547 | − std r16,EX_TLB_DEAR(r12); /* save DEAR */ | |
| 548 | − | |
| 549 | − /* We need _PAGE_PRESENT and _PAGE_ACCESSED set */ | |
| 550 | − li r11,_PAGE_PRESENT | |
| 551 | − oris r11,r11,_PAGE_ACCESSED@h | |
| 552 | − | |
| 553 | − /* We do the user/kernel test for the PID here along with the RW test | |
| 554 | − */ | |
| 555 | − srdi. r15,r16,60 /* Check for user region */ | |
| 556 | − | |
| 557 | − /* We pre-test some combination of permissions to avoid double | |
| 558 | − * faults: | |
| 559 | − * | |
| 560 | − * We move the ESR:ST bit into the position of _PAGE_BAP_SW in the PTE | |
| 561 | − * ESR_ST is 0x00800000 | |
| 562 | − * _PAGE_BAP_SW is 0x00000010 | |
| 563 | − * So the shift is >> 19. This tests for supervisor writeability. | |
| 564 | − * If the page happens to be supervisor writeable and not user | |
| 565 | − * writeable, we will take a new fault later, but that should be | |
| 566 | − * a rare enough case. | |
| 567 | − * | |
| 568 | − * We also move ESR_ST in _PAGE_DIRTY position | |
| 569 | − * _PAGE_DIRTY is 0x00001000 so the shift is >> 11 | |
| 570 | − * | |
| 571 | − * MAS1 is preset for all we need except for TID that needs to | |
| 572 | − * be cleared for kernel translations | |
| 573 | − */ | |
| 574 | − rlwimi r11,r14,32-19,27,27 | |
| 575 | − rlwimi r11,r14,32-16,19,19 | |
| 576 | − beq normal_tlb_miss_user | |
| 577 | − /* XXX replace the RMW cycles with immediate loads + writes */ | |
| 578 | −1: mfspr r10,SPRN_MAS1 | |
| 579 | − rlwinm r10,r10,0,16,1 /* Clear TID */ | |
| 580 | − mtspr SPRN_MAS1,r10 | |
| 581 | − beq+ cr1,normal_tlb_miss | |
| 582 | − | |
| 583 | − /* We got a crappy address, just fault with whatever DEAR and ESR | |
| 584 | − * are here | |
| 585 | − */ | |
| 586 | − TLB_MISS_EPILOG_ERROR | |
| 587 | − b exc_data_storage_book3e | |
| 588 | − | |
| 589 | −/* Instruction TLB miss */ | |
| 590 | − START_EXCEPTION(instruction_tlb_miss) | |
| 591 | − TLB_MISS_PROLOG | |
| 592 | − | |
| 593 | − /* If we take a recursive fault, the second level handler may need | |
| 594 | − * to know whether we are handling a data or instruction fault in | |
| 595 | − * order to get to the right store fault handler. We provide that | |
| 596 | − * info by writing a crazy value in ESR in our exception frame | |
| 597 | − */ | |
| 598 | − li r14,-1 /* store to exception frame is done later */ | |
| 599 | − | |
| 600 | − /* Now we handle the fault proper. We only save DEAR in the non | |
| 601 | − * linear mapping case since we know the linear mapping case will | |
| 602 | − * not re-enter. We could indeed optimize and also not save SRR0/1 | |
| 603 | − * in the linear mapping case but I'll leave that for later | |
| 604 | − * | |
| 605 | − * Faulting address is SRR0 which is already in r16 | |
| 606 | − */ | |
| 607 | − srdi r15,r16,44 /* get region */ | |
| 608 | − xoris r15,r15,0xc | |
| 609 | − cmpldi cr0,r15,0 /* linear mapping ? */ | |
| 610 | − beq tlb_load_linear /* yes -> go to linear map load */ | |
| 611 | − cmpldi cr1,r15,1 /* vmalloc mapping ? */ | |
| 612 | − | |
| 613 | − /* We do the user/kernel test for the PID here along with the RW test | |
| 614 | − */ | |
| 615 | − li r11,_PAGE_PRESENT|_PAGE_BAP_UX /* Base perm */ | |
| 616 | − oris r11,r11,_PAGE_ACCESSED@h | |
| 617 | − | |
| 618 | − srdi. r15,r16,60 /* Check for user region */ | |
| 619 | − std r14,EX_TLB_ESR(r12) /* write crazy -1 to frame */ | |
| 620 | − beq normal_tlb_miss_user | |
| 621 | − | |
| 622 | − li r11,_PAGE_PRESENT|_PAGE_BAP_SX /* Base perm */ | |
| 623 | − oris r11,r11,_PAGE_ACCESSED@h | |
| 624 | − /* XXX replace the RMW cycles with immediate loads + writes */ | |
| 625 | − mfspr r10,SPRN_MAS1 | |
| 626 | − rlwinm r10,r10,0,16,1 /* Clear TID */ | |
| 627 | − mtspr SPRN_MAS1,r10 | |
| 628 | − beq+ cr1,normal_tlb_miss | |
| 629 | − | |
| 630 | − /* We got a crappy address, just fault */ | |
| 631 | − TLB_MISS_EPILOG_ERROR | |
| 632 | − b exc_instruction_storage_book3e | |
| 633 | − | |
| 634 | −/* | |
| 635 | − * This is the guts of the first-level TLB miss handler for direct | |
| 636 | − * misses. We are entered with: | |
| 637 | − * | |
| 638 | − * r16 = faulting address | |
| 639 | − * r15 = region ID | |
| 640 | − * r14 = crap (free to use) | |
| 641 | − * r13 = PACA | |
| 642 | − * r12 = TLB exception frame in PACA | |
| 643 | − * r11 = PTE permission mask | |
| 644 | − * r10 = crap (free to use) | |
| 645 | − */ | |
| 646 | −normal_tlb_miss_user: | |
| 647 | −#ifdef CONFIG_PPC_KUAP | |
| 648 | − mfspr r14,SPRN_MAS1 | |
| 649 | − rlwinm. r14,r14,0,0x3fff0000 | |
| 650 | − beq- normal_tlb_miss_access_fault /* KUAP fault */ | |
| 651 | −#endif | |
| 652 | −normal_tlb_miss: | |
| 653 | − /* So we first construct the page table address. We do that by | |
| 654 | − * shifting the bottom of the address (not the region ID) by | |
| 655 | − * PAGE_SHIFT-3, clearing the bottom 3 bits (get a PTE ptr) and | |
| 656 | − * or'ing the fourth high bit. | |
| 657 | − * | |
| 658 | − * NOTE: For 64K pages, we do things slightly differently in | |
| 659 | − * order to handle the weird page table format used by linux | |
| 660 | − */ | |
| 661 | − srdi r15,r16,44 | |
| 662 | − oris r10,r15,0x1 | |
| 663 | − rldicl r14,r16,64-(PAGE_SHIFT-3),PAGE_SHIFT-3+4 | |
| 664 | − sldi r15,r10,44 | |
| 665 | − clrrdi r14,r14,19 | |
| 666 | − or r10,r15,r14 | |
| 667 | − | |
| 668 | − ld r14,0(r10) | |
| 669 | − | |
| 670 | −finish_normal_tlb_miss: | |
| 671 | − /* Check if required permissions are met */ | |
| 672 | − andc. r15,r11,r14 | |
| 673 | − bne- normal_tlb_miss_access_fault | |
| 674 | − | |
| 675 | − /* Now we build the MAS: | |
| 676 | − * | |
| 677 | − * MAS 0 : Fully setup with defaults in MAS4 and TLBnCFG | |
| 678 | − * MAS 1 : Almost fully setup | |
| 679 | − * - PID already updated by caller if necessary | |
| 680 | − * - TSIZE need change if !base page size, not | |
| 681 | − * yet implemented for now | |
| 682 | − * MAS 2 : Defaults not useful, need to be redone | |
| 683 | − * MAS 3+7 : Needs to be done | |
| 684 | − * | |
| 685 | − * TODO: mix up code below for better scheduling | |
| 686 | − */ | |
| 687 | − clrrdi r10,r16,12 /* Clear low crap in EA */ | |
| 688 | − rlwimi r10,r14,32-19,27,31 /* Insert WIMGE */ | |
| 689 | − mtspr SPRN_MAS2,r10 | |
| 690 | − | |
| 691 | − /* Check page size, if not standard, update MAS1 */ | |
| 692 | − rldicl r10,r14,64-8,64-8 | |
| 693 | − cmpldi cr0,r10,BOOK3E_PAGESZ_4K | |
| 694 | − beq- 1f | |
| 695 | − mfspr r11,SPRN_MAS1 | |
| 696 | − rlwimi r11,r14,31,21,24 | |
| 697 | − rlwinm r11,r11,0,21,19 | |
| 698 | − mtspr SPRN_MAS1,r11 | |
| 699 | −1: | |
| 700 | − /* Move RPN in position */ | |
| 701 | − rldicr r11,r14,64-(PTE_RPN_SHIFT-PAGE_SHIFT),63-PAGE_SHIFT | |
| 702 | − clrldi r15,r11,12 /* Clear crap at the top */ | |
| 703 | − rlwimi r15,r14,32-8,22,25 /* Move in U bits */ | |
| 704 | − rlwimi r15,r14,32-2,26,31 /* Move in BAP bits */ | |
| 705 | − | |
| 706 | − /* Mask out SW and UW if !DIRTY (XXX optimize this !) */ | |
| 707 | − andi. r11,r14,_PAGE_DIRTY | |
| 708 | − bne 1f | |
| 709 | − li r11,MAS3_SW|MAS3_UW | |
| 710 | − andc r15,r15,r11 | |
| 711 | −1: | |
| 712 | − srdi r16,r15,32 | |
| 713 | − mtspr SPRN_MAS3,r15 | |
| 714 | − mtspr SPRN_MAS7,r16 | |
| 715 | − | |
| 716 | − tlbwe | |
| 717 | − | |
| 718 | −normal_tlb_miss_done: | |
| 719 | − /* We don't bother with restoring DEAR or ESR since we know we are | |
| 720 | − * level 0 and just going back to userland. They are only needed | |
| 721 | − * if you are going to take an access fault | |
| 722 | − */ | |
| 723 | − TLB_MISS_EPILOG_SUCCESS | |
| 724 | − rfi | |
| 725 | − | |
| 726 | −normal_tlb_miss_access_fault: | |
| 727 | − /* We need to check if it was an instruction miss */ | |
| 728 | − andi. r10,r11,_PAGE_BAP_UX | |
| 729 | − bne 1f | |
| 730 | − ld r14,EX_TLB_DEAR(r12) | |
| 731 | − ld r15,EX_TLB_ESR(r12) | |
| 732 | − mtspr SPRN_DEAR,r14 | |
| 733 | − mtspr SPRN_ESR,r15 | |
| 734 | − TLB_MISS_EPILOG_ERROR | |
| 735 | − b exc_data_storage_book3e | |
| 736 | −1: TLB_MISS_EPILOG_ERROR | |
| 737 | − b exc_instruction_storage_book3e | |
| 738 | − | |
| 739 | − | |
| 740 | 509 | /* |
| 741 | 510 | * This is the guts of the second-level TLB miss handler for direct |
| 742 | 511 | * misses. We are entered with: |
| @@ -893,201 +662,6 @@ virt_page_table_tlb_miss_whacko_fault: · 150 unmodified lines | ||
| 893 | 662 | TLB_MISS_EPILOG_ERROR |
| 894 | 663 | b exc_data_storage_book3e |
| 895 | 664 | |
| 896 | − | |
| 897 | −/************************************************************** | |
| 898 | − * * | |
| 899 | − * TLB miss handling for Book3E with hw page table support * | |
| 900 | − * * | |
| 901 | − **************************************************************/ | |
| 902 | − | |
| 903 | − | |
| 904 | −/* Data TLB miss */ | |
| 905 | − START_EXCEPTION(data_tlb_miss_htw) | |
| 906 | − TLB_MISS_PROLOG | |
| 907 | − | |
| 908 | − /* Now we handle the fault proper. We only save DEAR in normal | |
| 909 | − * fault case since that's the only interesting values here. | |
| 910 | − * We could probably also optimize by not saving SRR0/1 in the | |
| 911 | − * linear mapping case but I'll leave that for later | |
| 912 | − */ | |
| 913 | − mfspr r14,SPRN_ESR | |
| 914 | − mfspr r16,SPRN_DEAR /* get faulting address */ | |
| 915 | − srdi r11,r16,44 /* get region */ | |
| 916 | − xoris r11,r11,0xc | |
| 917 | − cmpldi cr0,r11,0 /* linear mapping ? */ | |
| 918 | − beq tlb_load_linear /* yes -> go to linear map load */ | |
| 919 | − cmpldi cr1,r11,1 /* vmalloc mapping ? */ | |
| 920 | − | |
| 921 | − /* We do the user/kernel test for the PID here along with the RW test | |
| 922 | − */ | |
| 923 | − srdi. r11,r16,60 /* Check for user region */ | |
| 924 | − ld r15,PACAPGD(r13) /* Load user pgdir */ | |
| 925 | − beq htw_tlb_miss | |
| 926 | − | |
| 927 | − /* XXX replace the RMW cycles with immediate loads + writes */ | |
| 928 | −1: mfspr r10,SPRN_MAS1 | |
| 929 | − rlwinm r10,r10,0,16,1 /* Clear TID */ | |
| 930 | − mtspr SPRN_MAS1,r10 | |
| 931 | − ld r15,PACA_KERNELPGD(r13) /* Load kernel pgdir */ | |
| 932 | − beq+ cr1,htw_tlb_miss | |
| 933 | − | |
| 934 | − /* We got a crappy address, just fault with whatever DEAR and ESR | |
| 935 | − * are here | |
| 936 | − */ | |
| 937 | − TLB_MISS_EPILOG_ERROR | |
| 938 | − b exc_data_storage_book3e | |
| 939 | − | |
| 940 | −/* Instruction TLB miss */ | |
| 941 | − START_EXCEPTION(instruction_tlb_miss_htw) | |
| 942 | − TLB_MISS_PROLOG | |
| 943 | − | |
| 944 | − /* If we take a recursive fault, the second level handler may need | |
| 945 | − * to know whether we are handling a data or instruction fault in | |
| 946 | − * order to get to the right store fault handler. We provide that | |
| 947 | − * info by keeping a crazy value for ESR in r14 | |
| 948 | − */ | |
| 949 | − li r14,-1 /* store to exception frame is done later */ | |
| 950 | − | |
| 951 | − /* Now we handle the fault proper. We only save DEAR in the non | |
| 952 | − * linear mapping case since we know the linear mapping case will | |
| 953 | − * not re-enter. We could indeed optimize and also not save SRR0/1 | |
| 954 | − * in the linear mapping case but I'll leave that for later | |
| 955 | − * | |
| 956 | − * Faulting address is SRR0 which is already in r16 | |
| 957 | − */ | |
| 958 | − srdi r11,r16,44 /* get region */ | |
| 959 | − xoris r11,r11,0xc | |
| 960 | − cmpldi cr0,r11,0 /* linear mapping ? */ | |
| 961 | − beq tlb_load_linear /* yes -> go to linear map load */ | |
| 962 | − cmpldi cr1,r11,1 /* vmalloc mapping ? */ | |
| 963 | − | |
| 964 | − /* We do the user/kernel test for the PID here along with the RW test | |
| 965 | − */ | |
| 966 | − srdi. r11,r16,60 /* Check for user region */ | |
| 967 | − ld r15,PACAPGD(r13) /* Load user pgdir */ | |
| 968 | − beq htw_tlb_miss | |
| 969 | − | |
| 970 | − /* XXX replace the RMW cycles with immediate loads + writes */ | |
| 971 | −1: mfspr r10,SPRN_MAS1 | |
| 972 | − rlwinm r10,r10,0,16,1 /* Clear TID */ | |
| 973 | − mtspr SPRN_MAS1,r10 | |
| 974 | − ld r15,PACA_KERNELPGD(r13) /* Load kernel pgdir */ | |
| 975 | − beq+ htw_tlb_miss | |
| 976 | − | |
| 977 | − /* We got a crappy address, just fault */ | |
| 978 | − TLB_MISS_EPILOG_ERROR | |
| 979 | − b exc_instruction_storage_book3e | |
| 980 | − | |
| 981 | − | |
| 982 | −/* | |
| 983 | − * This is the guts of the second-level TLB miss handler for direct | |
| 984 | − * misses. We are entered with: | |
| 985 | − * | |
| 986 | − * r16 = virtual page table faulting address | |
| 987 | − * r15 = PGD pointer | |
| 988 | − * r14 = ESR | |
| 989 | − * r13 = PACA | |
| 990 | − * r12 = TLB exception frame in PACA | |
| 991 | − * r11 = crap (free to use) | |
| 992 | − * r10 = crap (free to use) | |
| 993 | − * | |
| 994 | − * It can be re-entered by the linear mapping miss handler. However, to | |
| 995 | − * avoid too much complication, it will save/restore things for us | |
| 996 | − */ | |
| 997 | −htw_tlb_miss: | |
| 998 | −#ifdef CONFIG_PPC_KUAP | |
| 999 | − mfspr r10,SPRN_MAS1 | |
| 1000 | − rlwinm. r10,r10,0,0x3fff0000 | |
| 1001 | − beq- htw_tlb_miss_fault /* KUAP fault */ | |
| 1002 | −#endif | |
| 1003 | − /* Search if we already have a TLB entry for that virtual address, and | |
| 1004 | − * if we do, bail out. | |
| 1005 | − * | |
| 1006 | − * MAS1:IND should be already set based on MAS4 | |
| 1007 | − */ | |
| 1008 | − PPC_TLBSRX_DOT(0,R16) | |
| 1009 | − beq htw_tlb_miss_done | |
| 1010 | − | |
| 1011 | − /* Now, we need to walk the page tables. First check if we are in | |
| 1012 | − * range. | |
| 1013 | − */ | |
| 1014 | − rldicl. r10,r16,64-PGTABLE_EADDR_SIZE,PGTABLE_EADDR_SIZE+4 | |
| 1015 | − bne- htw_tlb_miss_fault | |
| 1016 | − | |
| 1017 | − /* Get the PGD pointer */ | |
| 1018 | − cmpldi cr0,r15,0 | |
| 1019 | − beq- htw_tlb_miss_fault | |
| 1020 | − | |
| 1021 | − /* Get to PGD entry */ | |
| 1022 | − rldicl r11,r16,64-(PGDIR_SHIFT-3),64-PGD_INDEX_SIZE-3 | |
| 1023 | − clrrdi r10,r11,3 | |
| 1024 | − ldx r15,r10,r15 | |
| 1025 | − cmpdi cr0,r15,0 | |
| 1026 | − bge htw_tlb_miss_fault | |
| 1027 | − | |
| 1028 | − /* Get to PUD entry */ | |
| 1029 | − rldicl r11,r16,64-(PUD_SHIFT-3),64-PUD_INDEX_SIZE-3 | |
| 1030 | − clrrdi r10,r11,3 | |
| 1031 | − ldx r15,r10,r15 | |
| 1032 | − cmpdi cr0,r15,0 | |
| 1033 | − bge htw_tlb_miss_fault | |
| 1034 | − | |
| 1035 | − /* Get to PMD entry */ | |
| 1036 | − rldicl r11,r16,64-(PMD_SHIFT-3),64-PMD_INDEX_SIZE-3 | |
| 1037 | − clrrdi r10,r11,3 | |
| 1038 | − ldx r15,r10,r15 | |
| 1039 | − cmpdi cr0,r15,0 | |
| 1040 | − bge htw_tlb_miss_fault | |
| 1041 | − | |
| 1042 | − /* Ok, we're all right, we can now create an indirect entry for | |
| 1043 | − * a 1M or 256M page. | |
| 1044 | − * | |
| 1045 | − * The last trick is now that because we use "half" pages for | |
| 1046 | − * the HTW (1M IND is 2K and 256M IND is 32K) we need to account | |
| 1047 | − * for an added LSB bit to the RPN. For 64K pages, there is no | |
| 1048 | − * problem as we already use 32K arrays (half PTE pages), but for | |
| 1049 | − * 4K page we need to extract a bit from the virtual address and | |
| 1050 | − * insert it into the "PA52" bit of the RPN. | |
| 1051 | − */ | |
| 1052 | − rlwimi r15,r16,32-9,20,20 | |
| 1053 | − /* Now we build the MAS: | |
| 1054 | − * | |
| 1055 | − * MAS 0 : Fully setup with defaults in MAS4 and TLBnCFG | |
| 1056 | − * MAS 1 : Almost fully setup | |
| 1057 | − * - PID already updated by caller if necessary | |
| 1058 | − * - TSIZE for now is base ind page size always | |
| 1059 | − * MAS 2 : Use defaults | |
| 1060 | − * MAS 3+7 : Needs to be done | |
| 1061 | − */ | |
| 1062 | − ori r10,r15,(BOOK3E_PAGESZ_4K << MAS3_SPSIZE_SHIFT) | |
| 1063 | − | |
| 1064 | − srdi r16,r10,32 | |
| 1065 | − mtspr SPRN_MAS3,r10 | |
| 1066 | − mtspr SPRN_MAS7,r16 | |
| 1067 | − | |
| 1068 | − tlbwe | |
| 1069 | − | |
| 1070 | −htw_tlb_miss_done: | |
| 1071 | − /* We don't bother with restoring DEAR or ESR since we know we are | |
| 1072 | − * level 0 and just going back to userland. They are only needed | |
| 1073 | − * if you are going to take an access fault | |
| 1074 | − */ | |
| 1075 | − TLB_MISS_EPILOG_SUCCESS | |
| 1076 | − rfi | |
| 1077 | − | |
| 1078 | −htw_tlb_miss_fault: | |
| 1079 | − /* We need to check if it was an instruction miss. We know this | |
| 1080 | − * though because r14 would contain -1 | |
| 1081 | − */ | |
| 1082 | − cmpdi cr0,r14,-1 | |
| 1083 | − beq 1f | |
| 1084 | − mtspr SPRN_DEAR,r16 | |
| 1085 | − mtspr SPRN_ESR,r14 | |
| 1086 | − TLB_MISS_EPILOG_ERROR | |
| 1087 | − b exc_data_storage_book3e | |
| 1088 | −1: TLB_MISS_EPILOG_ERROR | |
| 1089 | − b exc_instruction_storage_book3e | |
| 1090 | − | |
| 1091 | 665 | /* |
| 1092 | 666 | * This is the guts of "any" level TLB miss handler for kernel linear |
| 1093 | 667 | * mapping misses. We are entered with: |
| @@ -297,11 +297,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 296 unmodified lines | ||
| 297 | 297 | } |
| 298 | 298 | |
| 299 | 299 | #if defined(CONFIG_PPC_8xx) |
| 300 | −void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, | |
| 301 | − pte_t pte, unsigned long sz) | |
| 300 | +static void __set_huge_pte_at(pmd_t *pmd, pte_t *ptep, pte_basic_t val) | |
| 302 | 301 | { |
| 303 | − pmd_t *pmd = pmd_off(mm, addr); | |
| 304 | − pte_basic_t val; | |
| 305 | 302 | pte_basic_t *entry = (pte_basic_t *)ptep; |
| 306 | 303 | int num, i; |
| 307 | 304 | |
| @@ -311,15 +308,60 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, · 3 unmodified lines | ||
| 311 | 308 | */ |
| 312 | 309 | VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep)); |
| 313 | 310 | |
| 314 | − pte = set_pte_filter(pte, addr); | |
| 315 | − | |
| 316 | − val = pte_val(pte); | |
| 317 | − | |
| 318 | 311 | num = number_of_cells_per_pte(pmd, val, 1); |
| 319 | 312 | |
| 320 | 313 | for (i = 0; i < num; i++, entry++, val += SZ_4K) |
| 321 | 314 | *entry = val; |
| 322 | 315 | } |
| 316 | + | |
| 317 | +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, | |
| 318 | + pte_t pte, unsigned long sz) | |
| 319 | +{ | |
| 320 | + pmd_t *pmdp = pmd_off(mm, addr); | |
| 321 | + | |
| 322 | + pte = set_pte_filter(pte, addr); | |
| 323 | + | |
| 324 | + if (sz == SZ_8M) { /* Flag both PMD entries as 8M and fill both page tables */ | |
| 325 | + *pmdp = __pmd(pmd_val(*pmdp) | _PMD_PAGE_8M); | |
| 326 | + *(pmdp + 1) = __pmd(pmd_val(*(pmdp + 1)) | _PMD_PAGE_8M); | |
| 327 | + | |
| 328 | + __set_huge_pte_at(pmdp, pte_offset_kernel(pmdp, 0), pte_val(pte)); | |
| 329 | + __set_huge_pte_at(pmdp, pte_offset_kernel(pmdp + 1, 0), pte_val(pte) + SZ_4M); | |
| 330 | + } else { | |
| 331 | + __set_huge_pte_at(pmdp, ptep, pte_val(pte)); | |
| 332 | + } | |
| 333 | +} | |
| 334 | +#else | |
| 335 | +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, | |
| 336 | + pte_t pte, unsigned long sz) | |
| 337 | +{ | |
| 338 | + unsigned long pdsize; | |
| 339 | + int i; | |
| 340 | + | |
| 341 | + pte = set_pte_filter(pte, addr); | |
| 342 | + | |
| 343 | + /* | |
| 344 | + * Make sure hardware valid bit is not set. We don't do | |
| 345 | + * tlb flush for this update. | |
| 346 | + */ | |
| 347 | + VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep)); | |
| 348 | + | |
| 349 | + if (sz < PMD_SIZE) | |
| 350 | + pdsize = PAGE_SIZE; | |
| 351 | + else if (sz < PUD_SIZE) | |
| 352 | + pdsize = PMD_SIZE; | |
| 353 | + else if (sz < P4D_SIZE) | |
| 354 | + pdsize = PUD_SIZE; | |
| 355 | + else if (sz < PGDIR_SIZE) | |
| 356 | + pdsize = P4D_SIZE; | |
| 357 | + else | |
| 358 | + pdsize = PGDIR_SIZE; | |
| 359 | + | |
| 360 | + for (i = 0; i < sz / pdsize; i++, ptep++, addr += pdsize) { | |
| 361 | + __set_pte_at(mm, addr, ptep, pte, 0); | |
| 362 | + pte = __pte(pte_val(pte) + ((unsigned long long)pdsize / PAGE_SIZE << PFN_PTE_SHIFT)); | |
| 363 | + } | |
| 364 | +} | |
| 323 | 365 | #endif |
| 324 | 366 | #endif /* CONFIG_HUGETLB_PAGE */ |
| 325 | 367 | |
| @@ -367,11 +409,10 @@ unsigned long vmalloc_to_phys(void *va) · 41 unmodified lines | ||
| 367 | 409 | EXPORT_SYMBOL_GPL(vmalloc_to_phys); |
| 368 | 410 | |
| 369 | 411 | /* |
| 370 | − * We have 4 cases for pgds and pmds: | |
| 412 | + * We have 3 cases for pgds and pmds: | |
| 371 | 413 | * (1) invalid (all zeroes) |
| 372 | 414 | * (2) pointer to next table, as normal; bottom 6 bits == 0 |
| 373 | 415 | * (3) leaf pte for huge page _PAGE_PTE set |
| 374 | − * (4) hugepd pointer, _PAGE_PTE = 0 and bits [2..6] indicate size of table | |
| 375 | 416 | * |
| 376 | 417 | * So long as we atomically load page table pointers we are safe against teardown, |
| 377 | 418 | * we can follow the address down to the page and take a ref on it. |
| @@ -382,11 +423,12 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, · 4 unmodified lines | ||
| 382 | 423 | bool *is_thp, unsigned *hpage_shift) |
| 383 | 424 | { |
| 384 | 425 | pgd_t *pgdp; |
| 426 | +#ifdef CONFIG_PPC64 | |
| 385 | 427 | p4d_t p4d, *p4dp; |
| 386 | 428 | pud_t pud, *pudp; |
| 429 | +#endif | |
| 387 | 430 | pmd_t pmd, *pmdp; |
| 388 | 431 | pte_t *ret_pte; |
| 389 | − hugepd_t *hpdp = NULL; | |
| 390 | 432 | unsigned pdshift; |
| 391 | 433 | |
| 392 | 434 | if (hpage_shift) |
| @@ -401,8 +443,12 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, · 8 unmodified lines | ||
| 401 | 443 | * page fault or a page unmap. The return pte_t * is still not |
| 402 | 444 | * stable. So should be checked there for above conditions. |
| 403 | 445 | * Top level is an exception because it is folded into p4d. |
| 446 | + * | |
| 447 | + * On PPC32, P4D/PUD/PMD are folded into PGD so go straight to | |
| 448 | + * PMD level. | |
| 404 | 449 | */ |
| 405 | 450 | pgdp = pgdir + pgd_index(ea); |
| 451 | +#ifdef CONFIG_PPC64 | |
| 406 | 452 | p4dp = p4d_offset(pgdp, ea); |
| 407 | 453 | p4d = READ_ONCE(*p4dp); |
| 408 | 454 | pdshift = P4D_SHIFT; |
| @@ -415,11 +461,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, · 6 unmodified lines | ||
| 415 | 461 | goto out; |
| 416 | 462 | } |
| 417 | 463 | |
| 418 | − if (is_hugepd(__hugepd(p4d_val(p4d)))) { | |
| 419 | − hpdp = (hugepd_t *)&p4d; | |
| 420 | − goto out_huge; | |
| 421 | − } | |
| 422 | − | |
| 423 | 464 | /* |
| 424 | 465 | * Even if we end up with an unmap, the pgtable will not |
| 425 | 466 | * be freed, because we do an rcu free and here we are |
| @@ -437,13 +478,11 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, · 11 unmodified lines | ||
| 437 | 478 | goto out; |
| 438 | 479 | } |
| 439 | 480 | |
| 440 | − if (is_hugepd(__hugepd(pud_val(pud)))) { | |
| 441 | − hpdp = (hugepd_t *)&pud; | |
| 442 | − goto out_huge; | |
| 443 | − } | |
| 444 | − | |
| 445 | − pdshift = PMD_SHIFT; | |
| 446 | 481 | pmdp = pmd_offset(&pud, ea); |
| 482 | +#else | |
| 483 | + pmdp = pmd_offset(pud_offset(p4d_offset(pgdp, ea), ea), ea); | |
| 484 | +#endif | |
| 485 | + pdshift = PMD_SHIFT; | |
| 447 | 486 | pmd = READ_ONCE(*pmdp); |
| 448 | 487 | |
| 449 | 488 | /* |
| @@ -476,19 +515,8 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, · 26 unmodified lines | ||
| 476 | 515 | goto out; |
| 477 | 516 | } |
| 478 | 517 | |
| 479 | − if (is_hugepd(__hugepd(pmd_val(pmd)))) { | |
| 480 | − hpdp = (hugepd_t *)&pmd; | |
| 481 | − goto out_huge; | |
| 482 | − } | |
| 483 | − | |
| 484 | 518 | return pte_offset_kernel(&pmd, ea); |
| 485 | 519 | |
| 486 | −out_huge: | |
| 487 | − if (!hpdp) | |
| 488 | − return NULL; | |
| 489 | − | |
| 490 | − ret_pte = hugepte_offset(*hpdp, ea, pdshift); | |
| 491 | − pdshift = hugepd_shift(*hpdp); | |
| 492 | 520 | out: |
| 493 | 521 | if (hpage_shift) |
| 494 | 522 | *hpage_shift = pdshift; |
| @@ -48,7 +48,7 @@ notrace void __init early_ioremap_init(void) · 47 unmodified lines | ||
| 48 | 48 | early_ioremap_setup(); |
| 49 | 49 | } |
| 50 | 50 | |
| 51 | −static void __init *early_alloc_pgtable(unsigned long size) | |
| 51 | +void __init *early_alloc_pgtable(unsigned long size) | |
| 52 | 52 | { |
| 53 | 53 | void *ptr = memblock_alloc(size, size); |
| 54 | 54 | |
| @@ -44,7 +44,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 43 unmodified lines | ||
| 44 | 44 | pte_t pte, int dirty); |
| 45 | 45 | |
| 46 | 46 | #define __HAVE_ARCH_HUGE_PTEP_GET |
| 47 | −pte_t huge_ptep_get(pte_t *ptep); | |
| 47 | +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); | |
| 48 | 48 | |
| 49 | 49 | pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); |
| 50 | 50 | #define arch_make_huge_pte arch_make_huge_pte |
| @@ -514,8 +514,8 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, · 513 unmodified lines | ||
| 514 | 514 | #define update_mmu_cache(vma, addr, ptep) \ |
| 515 | 515 | update_mmu_cache_range(NULL, vma, addr, ptep, 1) |
| 516 | 516 | |
| 517 | −#define __HAVE_ARCH_UPDATE_MMU_TLB | |
| 518 | −#define update_mmu_tlb update_mmu_cache | |
| 517 | +#define update_mmu_tlb_range(vma, addr, ptep, nr) \ | |
| 518 | + update_mmu_cache_range(NULL, vma, addr, ptep, nr) | |
| 519 | 519 | |
| 520 | 520 | static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, |
| 521 | 521 | unsigned long address, pmd_t *pmdp) |
| @@ -3,7 +3,7 @@ · 2 unmodified lines | ||
| 3 | 3 | #include <linux/err.h> |
| 4 | 4 | |
| 5 | 5 | #ifdef CONFIG_RISCV_ISA_SVNAPOT |
| 6 | −pte_t huge_ptep_get(pte_t *ptep) | |
| 6 | +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) | |
| 7 | 7 | { |
| 8 | 8 | unsigned long pte_num; |
| 9 | 9 | int i; |
| @@ -159,6 +159,7 @@ config S390 · 158 unmodified lines | ||
| 159 | 159 | select HAVE_ARCH_KASAN |
| 160 | 160 | select HAVE_ARCH_KASAN_VMALLOC |
| 161 | 161 | select HAVE_ARCH_KCSAN |
| 162 | + select HAVE_ARCH_KMSAN | |
| 162 | 163 | select HAVE_ARCH_KFENCE |
| 163 | 164 | select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET |
| 164 | 165 | select HAVE_ARCH_SECCOMP_FILTER |
| @@ -36,7 +36,7 @@ KBUILD_CFLAGS_DECOMPRESSOR += $(if $(CONFIG_DEBUG_INFO_DWARF4), $(call cc-option · 35 unmodified lines | ||
| 36 | 36 | KBUILD_CFLAGS_DECOMPRESSOR += $(if $(CONFIG_CC_NO_ARRAY_BOUNDS),-Wno-array-bounds) |
| 37 | 37 | |
| 38 | 38 | UTS_MACHINE := s390x |
| 39 | −STACK_SIZE := $(if $(CONFIG_KASAN),65536,16384) | |
| 39 | +STACK_SIZE := $(if $(CONFIG_KASAN),65536,$(if $(CONFIG_KMSAN),65536,16384)) | |
| 40 | 40 | CHECKFLAGS += -D__s390__ -D__s390x__ |
| 41 | 41 | |
| 42 | 42 | export LD_BFD |
| @@ -3,11 +3,13 @@ · 2 unmodified lines | ||
| 3 | 3 | # Makefile for the linux s390-specific parts of the memory manager. |
| 4 | 4 | # |
| 5 | 5 | |
| 6 | +# Tooling runtimes are unavailable and cannot be linked for early boot code | |
| 6 | 7 | KCOV_INSTRUMENT := n |
| 7 | 8 | GCOV_PROFILE := n |
| 8 | 9 | UBSAN_SANITIZE := n |
| 9 | 10 | KASAN_SANITIZE := n |
| 10 | 11 | KCSAN_SANITIZE := n |
| 12 | +KMSAN_SANITIZE := n | |
| 11 | 13 | |
| 12 | 14 | KBUILD_AFLAGS := $(KBUILD_AFLAGS_DECOMPRESSOR) |
| 13 | 15 | KBUILD_CFLAGS := $(KBUILD_CFLAGS_DECOMPRESSOR) |
| @@ -42,6 +44,7 @@ obj-$(findstring y, $(CONFIG_PROTECTED_VIRTUALIZATION_GUEST) $(CONFIG_PGSTE)) += · 28 unmodified lines | ||
| 42 | 44 | obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o |
| 43 | 45 | obj-y += $(if $(CONFIG_KERNEL_UNCOMPRESSED),,decompressor.o) info.o |
| 44 | 46 | obj-$(CONFIG_KERNEL_ZSTD) += clz_ctz.o |
| 47 | +obj-$(CONFIG_KMSAN) += kmsan.o | |
| 45 | 48 | obj-all := $(obj-y) piggy.o syms.o |
| 46 | 49 | |
| 47 | 50 | targets := bzImage section_cmp.boot.data section_cmp.boot.preserved.data $(obj-y) |
| @@ -0,0 +1,6 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | +#include <linux/kmsan-checks.h> | |
| 3 | + | |
| 4 | +void kmsan_unpoison_memory(const void *address, size_t size) | |
| 5 | +{ | |
| 6 | +} | |
| @@ -304,11 +304,18 @@ static unsigned long setup_kernel_memory_layout(unsigned long kernel_size) · 303 unmodified lines | ||
| 304 | 304 | MODULES_END = round_down(kernel_start, _SEGMENT_SIZE); |
| 305 | 305 | MODULES_VADDR = MODULES_END - MODULES_LEN; |
| 306 | 306 | VMALLOC_END = MODULES_VADDR; |
| 307 | + if (IS_ENABLED(CONFIG_KMSAN)) | |
| 308 | + VMALLOC_END -= MODULES_LEN * 2; | |
| 307 | 309 | |
| 308 | 310 | /* allow vmalloc area to occupy up to about 1/2 of the rest virtual space left */ |
| 309 | 311 | vsize = (VMALLOC_END - FIXMAP_SIZE) / 2; |
| 310 | 312 | vsize = round_down(vsize, _SEGMENT_SIZE); |
| 311 | 313 | vmalloc_size = min(vmalloc_size, vsize); |
| 314 | + if (IS_ENABLED(CONFIG_KMSAN)) { | |
| 315 | + /* take 2/3 of vmalloc area for KMSAN shadow and origins */ | |
| 316 | + vmalloc_size = round_down(vmalloc_size / 3, _SEGMENT_SIZE); | |
| 317 | + VMALLOC_END -= vmalloc_size * 2; | |
| 318 | + } | |
| 312 | 319 | VMALLOC_START = VMALLOC_END - vmalloc_size; |
| 313 | 320 | |
| 314 | 321 | __memcpy_real_area = round_down(VMALLOC_START - MEMCPY_REAL_SIZE, PAGE_SIZE); |
| @@ -1,11 +1,18 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | +#define IN_BOOT_STRING_C 1 | |
| 2 | 3 | #include <linux/ctype.h> |
| 3 | 4 | #include <linux/kernel.h> |
| 4 | 5 | #include <linux/errno.h> |
| 5 | 6 | #undef CONFIG_KASAN |
| 6 | 7 | #undef CONFIG_KASAN_GENERIC |
| 8 | +#undef CONFIG_KMSAN | |
| 7 | 9 | #include "../lib/string.c" |
| 8 | 10 | |
| 11 | +/* | |
| 12 | + * Duplicate some functions from the common lib/string.c | |
| 13 | + * instead of fully including it. | |
| 14 | + */ | |
| 15 | + | |
| 9 | 16 | int strncmp(const char *cs, const char *ct, size_t count) |
| 10 | 17 | { |
| 11 | 18 | unsigned char c1, c2; |
| @@ -22,6 +29,15 @@ int strncmp(const char *cs, const char *ct, size_t count) · 10 unmodified lines | ||
| 22 | 29 | return 0; |
| 23 | 30 | } |
| 24 | 31 | |
| 32 | +void *memset64(uint64_t *s, uint64_t v, size_t count) | |
| 33 | +{ | |
| 34 | + uint64_t *xs = s; | |
| 35 | + | |
| 36 | + while (count--) | |
| 37 | + *xs++ = v; | |
| 38 | + return s; | |
| 39 | +} | |
| 40 | + | |
| 25 | 41 | char *skip_spaces(const char *str) |
| 26 | 42 | { |
| 27 | 43 | while (isspace(*str)) |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #define _S390_CHECKSUM_H |
| 14 | 14 | |
| 15 | 15 | #include <linux/instrumented.h> |
| 16 | +#include <linux/kmsan-checks.h> | |
| 16 | 17 | #include <linux/in6.h> |
| 17 | 18 | |
| 18 | 19 | static inline __wsum cksm(const void *buff, int len, __wsum sum) |
| @@ -23,6 +24,7 @@ static inline __wsum cksm(const void *buff, int len, __wsum sum) · 4 unmodified lines | ||
| 23 | 24 | }; |
| 24 | 25 | |
| 25 | 26 | instrument_read(buff, len); |
| 27 | + kmsan_check_memory(buff, len); | |
| 26 | 28 | asm volatile("\n" |
| 27 | 29 | "0: cksm %[sum],%[rp]\n" |
| 28 | 30 | " jo 0b\n" |
| @@ -12,6 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #define _ASM_S390_CPACF_H |
| 13 | 13 | |
| 14 | 14 | #include <asm/facility.h> |
| 15 | +#include <linux/kmsan-checks.h> | |
| 15 | 16 | |
| 16 | 17 | /* |
| 17 | 18 | * Instruction opcodes for the CPACF instructions |
| @@ -542,6 +543,8 @@ static inline void cpacf_trng(u8 *ucbuf, unsigned long ucbuf_len, · 524 unmodified lines | ||
| 542 | 543 | : [ucbuf] "+&d" (u.pair), [cbuf] "+&d" (c.pair) |
| 543 | 544 | : [fc] "K" (CPACF_PRNO_TRNG), [opc] "i" (CPACF_PRNO) |
| 544 | 545 | : "cc", "memory", "0"); |
| 546 | + kmsan_unpoison_memory(ucbuf, ucbuf_len); | |
| 547 | + kmsan_unpoison_memory(cbuf, cbuf_len); | |
| 545 | 548 | } |
| 546 | 549 | |
| 547 | 550 | /** |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #define _ASM_S390_CPU_MF_H |
| 11 | 11 | |
| 12 | 12 | #include <linux/errno.h> |
| 13 | +#include <linux/kmsan-checks.h> | |
| 13 | 14 | #include <asm/asm-extable.h> |
| 14 | 15 | #include <asm/facility.h> |
| 15 | 16 | |
| @@ -239,6 +240,11 @@ static __always_inline int stcctm(enum stcctm_ctr_set set, u64 range, u64 *dest) · 223 unmodified lines | ||
| 239 | 240 | : "=d" (cc) |
| 240 | 241 | : "Q" (*dest), "d" (range), "i" (set) |
| 241 | 242 | : "cc", "memory"); |
| 243 | + /* | |
| 244 | + * If cc == 2, less than RANGE counters are stored, but it's not easy | |
| 245 | + * to tell how many. Always unpoison the whole range for simplicity. | |
| 246 | + */ | |
| 247 | + kmsan_unpoison_memory(dest, range * sizeof(u64)); | |
| 242 | 248 | return cc; |
| 243 | 249 | } |
| 244 | 250 | |
| @@ -19,7 +19,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 18 unmodified lines | ||
| 19 | 19 | pte_t *ptep, pte_t pte, unsigned long sz); |
| 20 | 20 | void __set_huge_pte_at(struct mm_struct *mm, unsigned long addr, |
| 21 | 21 | pte_t *ptep, pte_t pte); |
| 22 | −pte_t huge_ptep_get(pte_t *ptep); | |
| 22 | +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); | |
| 23 | 23 | pte_t huge_ptep_get_and_clear(struct mm_struct *mm, |
| 24 | 24 | unsigned long addr, pte_t *ptep); |
| 25 | 25 | |
| @@ -64,7 +64,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 38 unmodified lines | ||
| 64 | 64 | unsigned long addr, pte_t *ptep, |
| 65 | 65 | pte_t pte, int dirty) |
| 66 | 66 | { |
| 67 | − int changed = !pte_same(huge_ptep_get(ptep), pte); | |
| 67 | + int changed = !pte_same(huge_ptep_get(vma->vm_mm, addr, ptep), pte); | |
| 68 | 68 | if (changed) { |
| 69 | 69 | huge_ptep_get_and_clear(vma->vm_mm, addr, ptep); |
| 70 | 70 | __set_huge_pte_at(vma->vm_mm, addr, ptep, pte); |
| @@ -37,12 +37,18 @@ static __always_inline void __arch_local_irq_ssm(unsigned long flags) · 36 unmodified lines | ||
| 37 | 37 | asm volatile("ssm %0" : : "Q" (flags) : "memory"); |
| 38 | 38 | } |
| 39 | 39 | |
| 40 | −static __always_inline unsigned long arch_local_save_flags(void) | |
| 40 | +#ifdef CONFIG_KMSAN | |
| 41 | +#define arch_local_irq_attributes noinline notrace __no_sanitize_memory __maybe_unused | |
| 42 | +#else | |
| 43 | +#define arch_local_irq_attributes __always_inline | |
| 44 | +#endif | |
| 45 | + | |
| 46 | +static arch_local_irq_attributes unsigned long arch_local_save_flags(void) | |
| 41 | 47 | { |
| 42 | 48 | return __arch_local_irq_stnsm(0xff); |
| 43 | 49 | } |
| 44 | 50 | |
| 45 | −static __always_inline unsigned long arch_local_irq_save(void) | |
| 51 | +static arch_local_irq_attributes unsigned long arch_local_irq_save(void) | |
| 46 | 52 | { |
| 47 | 53 | return __arch_local_irq_stnsm(0xfc); |
| 48 | 54 | } |
| @@ -52,7 +58,12 @@ static __always_inline void arch_local_irq_disable(void) · 3 unmodified lines | ||
| 52 | 58 | arch_local_irq_save(); |
| 53 | 59 | } |
| 54 | 60 | |
| 55 | −static __always_inline void arch_local_irq_enable(void) | |
| 61 | +static arch_local_irq_attributes void arch_local_irq_enable_external(void) | |
| 62 | +{ | |
| 63 | + __arch_local_irq_stosm(0x01); | |
| 64 | +} | |
| 65 | + | |
| 66 | +static arch_local_irq_attributes void arch_local_irq_enable(void) | |
| 56 | 67 | { |
| 57 | 68 | __arch_local_irq_stosm(0x03); |
| 58 | 69 | } |
| @@ -0,0 +1,59 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +#ifndef _ASM_S390_KMSAN_H | |
| 3 | +#define _ASM_S390_KMSAN_H | |
| 4 | + | |
| 5 | +#include <asm/lowcore.h> | |
| 6 | +#include <asm/page.h> | |
| 7 | +#include <linux/kmsan.h> | |
| 8 | +#include <linux/mmzone.h> | |
| 9 | +#include <linux/stddef.h> | |
| 10 | + | |
| 11 | +#ifndef MODULE | |
| 12 | + | |
| 13 | +static inline bool is_lowcore_addr(void *addr) | |
| 14 | +{ | |
| 15 | + return addr >= (void *)&S390_lowcore && | |
| 16 | + addr < (void *)(&S390_lowcore + 1); | |
| 17 | +} | |
| 18 | + | |
| 19 | +static inline void *arch_kmsan_get_meta_or_null(void *addr, bool is_origin) | |
| 20 | +{ | |
| 21 | + if (is_lowcore_addr(addr)) { | |
| 22 | + /* | |
| 23 | + * Different lowcores accessed via S390_lowcore are described | |
| 24 | + * by the same struct page. Resolve the prefix manually in | |
| 25 | + * order to get a distinct struct page. | |
| 26 | + */ | |
| 27 | + addr += (void *)lowcore_ptr[raw_smp_processor_id()] - | |
| 28 | + (void *)&S390_lowcore; | |
| 29 | + if (KMSAN_WARN_ON(is_lowcore_addr(addr))) | |
| 30 | + return NULL; | |
| 31 | + return kmsan_get_metadata(addr, is_origin); | |
| 32 | + } | |
| 33 | + return NULL; | |
| 34 | +} | |
| 35 | + | |
| 36 | +static inline bool kmsan_virt_addr_valid(void *addr) | |
| 37 | +{ | |
| 38 | + bool ret; | |
| 39 | + | |
| 40 | + /* | |
| 41 | + * pfn_valid() relies on RCU, and may call into the scheduler on exiting | |
| 42 | + * the critical section. However, this would result in recursion with | |
| 43 | + * KMSAN. Therefore, disable preemption here, and re-enable preemption | |
| 44 | + * below while suppressing reschedules to avoid recursion. | |
| 45 | + * | |
| 46 | + * Note, this sacrifices occasionally breaking scheduling guarantees. | |
| 47 | + * Although, a kernel compiled with KMSAN has already given up on any | |
| 48 | + * performance guarantees due to being heavily instrumented. | |
| 49 | + */ | |
| 50 | + preempt_disable(); | |
| 51 | + ret = virt_addr_valid(addr); | |
| 52 | + preempt_enable_no_resched(); | |
| 53 | + | |
| 54 | + return ret; | |
| 55 | +} | |
| 56 | + | |
| 57 | +#endif /* !MODULE */ | |
| 58 | + | |
| 59 | +#endif /* _ASM_S390_KMSAN_H */ | |
| @@ -107,6 +107,18 @@ static inline int is_module_addr(void *addr) · 106 unmodified lines | ||
| 107 | 107 | return 1; |
| 108 | 108 | } |
| 109 | 109 | |
| 110 | +#ifdef CONFIG_KMSAN | |
| 111 | +#define KMSAN_VMALLOC_SIZE (VMALLOC_END - VMALLOC_START) | |
| 112 | +#define KMSAN_VMALLOC_SHADOW_START VMALLOC_END | |
| 113 | +#define KMSAN_VMALLOC_SHADOW_END (KMSAN_VMALLOC_SHADOW_START + KMSAN_VMALLOC_SIZE) | |
| 114 | +#define KMSAN_VMALLOC_ORIGIN_START KMSAN_VMALLOC_SHADOW_END | |
| 115 | +#define KMSAN_VMALLOC_ORIGIN_END (KMSAN_VMALLOC_ORIGIN_START + KMSAN_VMALLOC_SIZE) | |
| 116 | +#define KMSAN_MODULES_SHADOW_START KMSAN_VMALLOC_ORIGIN_END | |
| 117 | +#define KMSAN_MODULES_SHADOW_END (KMSAN_MODULES_SHADOW_START + MODULES_LEN) | |
| 118 | +#define KMSAN_MODULES_ORIGIN_START KMSAN_MODULES_SHADOW_END | |
| 119 | +#define KMSAN_MODULES_ORIGIN_END (KMSAN_MODULES_ORIGIN_START + MODULES_LEN) | |
| 120 | +#endif | |
| 121 | + | |
| 110 | 122 | #ifdef CONFIG_RANDOMIZE_BASE |
| 111 | 123 | #define KASLR_LEN (1UL << 31) |
| 112 | 124 | #else |
| @@ -15,15 +15,12 @@ · 14 unmodified lines | ||
| 15 | 15 | #define __HAVE_ARCH_MEMCPY /* gcc builtin & arch function */ |
| 16 | 16 | #define __HAVE_ARCH_MEMMOVE /* gcc builtin & arch function */ |
| 17 | 17 | #define __HAVE_ARCH_MEMSET /* gcc builtin & arch function */ |
| 18 | −#define __HAVE_ARCH_MEMSET16 /* arch function */ | |
| 19 | −#define __HAVE_ARCH_MEMSET32 /* arch function */ | |
| 20 | −#define __HAVE_ARCH_MEMSET64 /* arch function */ | |
| 21 | 18 | |
| 22 | 19 | void *memcpy(void *dest, const void *src, size_t n); |
| 23 | 20 | void *memset(void *s, int c, size_t n); |
| 24 | 21 | void *memmove(void *dest, const void *src, size_t n); |
| 25 | 22 | |
| 26 | −#ifndef CONFIG_KASAN | |
| 23 | +#if !defined(CONFIG_KASAN) && !defined(CONFIG_KMSAN) | |
| 27 | 24 | #define __HAVE_ARCH_MEMCHR /* inline & arch function */ |
| 28 | 25 | #define __HAVE_ARCH_MEMCMP /* arch function */ |
| 29 | 26 | #define __HAVE_ARCH_MEMSCAN /* inline & arch function */ |
| @@ -36,6 +33,9 @@ void *memmove(void *dest, const void *src, size_t n); · 6 unmodified lines | ||
| 36 | 33 | #define __HAVE_ARCH_STRNCPY /* arch function */ |
| 37 | 34 | #define __HAVE_ARCH_STRNLEN /* inline & arch function */ |
| 38 | 35 | #define __HAVE_ARCH_STRSTR /* arch function */ |
| 36 | +#define __HAVE_ARCH_MEMSET16 /* arch function */ | |
| 37 | +#define __HAVE_ARCH_MEMSET32 /* arch function */ | |
| 38 | +#define __HAVE_ARCH_MEMSET64 /* arch function */ | |
| 39 | 39 | |
| 40 | 40 | /* Prototypes for non-inlined arch strings functions. */ |
| 41 | 41 | int memcmp(const void *s1, const void *s2, size_t n); |
| @@ -44,7 +44,7 @@ size_t strlcat(char *dest, const char *src, size_t n); · 2 unmodified lines | ||
| 44 | 44 | char *strncat(char *dest, const char *src, size_t n); |
| 45 | 45 | char *strncpy(char *dest, const char *src, size_t n); |
| 46 | 46 | char *strstr(const char *s1, const char *s2); |
| 47 | −#endif /* !CONFIG_KASAN */ | |
| 47 | +#endif /* !defined(CONFIG_KASAN) && !defined(CONFIG_KMSAN) */ | |
| 48 | 48 | |
| 49 | 49 | #undef __HAVE_ARCH_STRCHR |
| 50 | 50 | #undef __HAVE_ARCH_STRNCHR |
| @@ -74,20 +74,30 @@ void *__memset16(uint16_t *s, uint16_t v, size_t count); · 23 unmodified lines | ||
| 74 | 74 | void *__memset32(uint32_t *s, uint32_t v, size_t count); |
| 75 | 75 | void *__memset64(uint64_t *s, uint64_t v, size_t count); |
| 76 | 76 | |
| 77 | +#ifdef __HAVE_ARCH_MEMSET16 | |
| 77 | 78 | static inline void *memset16(uint16_t *s, uint16_t v, size_t count) |
| 78 | 79 | { |
| 79 | 80 | return __memset16(s, v, count * sizeof(v)); |
| 80 | 81 | } |
| 82 | +#endif | |
| 81 | 83 | |
| 84 | +#ifdef __HAVE_ARCH_MEMSET32 | |
| 82 | 85 | static inline void *memset32(uint32_t *s, uint32_t v, size_t count) |
| 83 | 86 | { |
| 84 | 87 | return __memset32(s, v, count * sizeof(v)); |
| 85 | 88 | } |
| 89 | +#endif | |
| 86 | 90 | |
| 91 | +#ifdef __HAVE_ARCH_MEMSET64 | |
| 92 | +#ifdef IN_BOOT_STRING_C | |
| 93 | +void *memset64(uint64_t *s, uint64_t v, size_t count); | |
| 94 | +#else | |
| 87 | 95 | static inline void *memset64(uint64_t *s, uint64_t v, size_t count) |
| 88 | 96 | { |
| 89 | 97 | return __memset64(s, v, count * sizeof(v)); |
| 90 | 98 | } |
| 99 | +#endif | |
| 100 | +#endif | |
| 91 | 101 | |
| 92 | 102 | #if !defined(IN_ARCH_STRING_C) && (!defined(CONFIG_FORTIFY_SOURCE) || defined(__NO_FORTIFY)) |
| 93 | 103 | |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | /* |
| 17 | 17 | * General size of kernel stacks |
| 18 | 18 | */ |
| 19 | −#ifdef CONFIG_KASAN | |
| 19 | +#if defined(CONFIG_KASAN) || defined(CONFIG_KMSAN) | |
| 20 | 20 | #define THREAD_SIZE_ORDER 4 |
| 21 | 21 | #else |
| 22 | 22 | #define THREAD_SIZE_ORDER 2 |
| @@ -18,6 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <asm/extable.h> |
| 19 | 19 | #include <asm/facility.h> |
| 20 | 20 | #include <asm-generic/access_ok.h> |
| 21 | +#include <linux/instrumented.h> | |
| 21 | 22 | |
| 22 | 23 | void debug_user_asce(int exit); |
| 23 | 24 | |
| @@ -78,13 +79,24 @@ union oac { · 54 unmodified lines | ||
| 78 | 79 | |
| 79 | 80 | int __noreturn __put_user_bad(void); |
| 80 | 81 | |
| 81 | −#define __put_user_asm(to, from, size) \ | |
| 82 | −({ \ | |
| 82 | +#ifdef CONFIG_KMSAN | |
| 83 | +#define get_put_user_noinstr_attributes \ | |
| 84 | + noinline __maybe_unused __no_sanitize_memory | |
| 85 | +#else | |
| 86 | +#define get_put_user_noinstr_attributes __always_inline | |
| 87 | +#endif | |
| 88 | + | |
| 89 | +#define DEFINE_PUT_USER(type) \ | |
| 90 | +static get_put_user_noinstr_attributes int \ | |
| 91 | +__put_user_##type##_noinstr(unsigned type __user *to, \ | |
| 92 | + unsigned type *from, \ | |
| 93 | + unsigned long size) \ | |
| 94 | +{ \ | |
| 83 | 95 | union oac __oac_spec = { \ |
| 84 | 96 | .oac1.as = PSW_BITS_AS_SECONDARY, \ |
| 85 | 97 | .oac1.a = 1, \ |
| 86 | 98 | }; \ |
| 87 | − int __rc; \ | |
| 99 | + int rc; \ | |
| 88 | 100 | \ |
| 89 | 101 | asm volatile( \ |
| 90 | 102 | " lr 0,%[spec]\n" \ |
| @@ -93,12 +105,28 @@ int __noreturn __put_user_bad(void); · 2 unmodified lines | ||
| 93 | 105 | "2:\n" \ |
| 94 | 106 | EX_TABLE_UA_STORE(0b, 2b, %[rc]) \ |
| 95 | 107 | EX_TABLE_UA_STORE(1b, 2b, %[rc]) \ |
| 96 | − : [rc] "=&d" (__rc), [_to] "+Q" (*(to)) \ | |
| 108 | + : [rc] "=&d" (rc), [_to] "+Q" (*(to)) \ | |
| 97 | 109 | : [_size] "d" (size), [_from] "Q" (*(from)), \ |
| 98 | 110 | [spec] "d" (__oac_spec.val) \ |
| 99 | 111 | : "cc", "0"); \ |
| 100 | − __rc; \ | |
| 101 | −}) | |
| 112 | + return rc; \ | |
| 113 | +} \ | |
| 114 | + \ | |
| 115 | +static __always_inline int \ | |
| 116 | +__put_user_##type(unsigned type __user *to, unsigned type *from, \ | |
| 117 | + unsigned long size) \ | |
| 118 | +{ \ | |
| 119 | + int rc; \ | |
| 120 | + \ | |
| 121 | + rc = __put_user_##type##_noinstr(to, from, size); \ | |
| 122 | + instrument_put_user(*from, to, size); \ | |
| 123 | + return rc; \ | |
| 124 | +} | |
| 125 | + | |
| 126 | +DEFINE_PUT_USER(char); | |
| 127 | +DEFINE_PUT_USER(short); | |
| 128 | +DEFINE_PUT_USER(int); | |
| 129 | +DEFINE_PUT_USER(long); | |
| 102 | 130 | |
| 103 | 131 | static __always_inline int __put_user_fn(void *x, void __user *ptr, unsigned long size) |
| 104 | 132 | { |
| @@ -106,24 +134,24 @@ static __always_inline int __put_user_fn(void *x, void __user *ptr, unsigned lon · 1 unmodified line | ||
| 106 | 134 | |
| 107 | 135 | switch (size) { |
| 108 | 136 | case 1: |
| 109 | − rc = __put_user_asm((unsigned char __user *)ptr, | |
| 110 | − (unsigned char *)x, | |
| 111 | − size); | |
| 137 | + rc = __put_user_char((unsigned char __user *)ptr, | |
| 138 | + (unsigned char *)x, | |
| 139 | + size); | |
| 112 | 140 | break; |
| 113 | 141 | case 2: |
| 114 | − rc = __put_user_asm((unsigned short __user *)ptr, | |
| 115 | − (unsigned short *)x, | |
| 116 | − size); | |
| 142 | + rc = __put_user_short((unsigned short __user *)ptr, | |
| 143 | + (unsigned short *)x, | |
| 144 | + size); | |
| 117 | 145 | break; |
| 118 | 146 | case 4: |
| 119 | − rc = __put_user_asm((unsigned int __user *)ptr, | |
| 147 | + rc = __put_user_int((unsigned int __user *)ptr, | |
| 120 | 148 | (unsigned int *)x, |
| 121 | 149 | size); |
| 122 | 150 | break; |
| 123 | 151 | case 8: |
| 124 | − rc = __put_user_asm((unsigned long __user *)ptr, | |
| 125 | − (unsigned long *)x, | |
| 126 | − size); | |
| 152 | + rc = __put_user_long((unsigned long __user *)ptr, | |
| 153 | + (unsigned long *)x, | |
| 154 | + size); | |
| 127 | 155 | break; |
| 128 | 156 | default: |
| 129 | 157 | __put_user_bad(); |
| @@ -134,13 +162,17 @@ static __always_inline int __put_user_fn(void *x, void __user *ptr, unsigned lon · 4 unmodified lines | ||
| 134 | 162 | |
| 135 | 163 | int __noreturn __get_user_bad(void); |
| 136 | 164 | |
| 137 | −#define __get_user_asm(to, from, size) \ | |
| 138 | −({ \ | |
| 165 | +#define DEFINE_GET_USER(type) \ | |
| 166 | +static get_put_user_noinstr_attributes int \ | |
| 167 | +__get_user_##type##_noinstr(unsigned type *to, \ | |
| 168 | + unsigned type __user *from, \ | |
| 169 | + unsigned long size) \ | |
| 170 | +{ \ | |
| 139 | 171 | union oac __oac_spec = { \ |
| 140 | 172 | .oac2.as = PSW_BITS_AS_SECONDARY, \ |
| 141 | 173 | .oac2.a = 1, \ |
| 142 | 174 | }; \ |
| 143 | − int __rc; \ | |
| 175 | + int rc; \ | |
| 144 | 176 | \ |
| 145 | 177 | asm volatile( \ |
| 146 | 178 | " lr 0,%[spec]\n" \ |
| @@ -149,13 +181,29 @@ int __noreturn __get_user_bad(void); · 2 unmodified lines | ||
| 149 | 181 | "2:\n" \ |
| 150 | 182 | EX_TABLE_UA_LOAD_MEM(0b, 2b, %[rc], %[_to], %[_ksize]) \ |
| 151 | 183 | EX_TABLE_UA_LOAD_MEM(1b, 2b, %[rc], %[_to], %[_ksize]) \ |
| 152 | − : [rc] "=&d" (__rc), "=Q" (*(to)) \ | |
| 184 | + : [rc] "=&d" (rc), "=Q" (*(to)) \ | |
| 153 | 185 | : [_size] "d" (size), [_from] "Q" (*(from)), \ |
| 154 | 186 | [spec] "d" (__oac_spec.val), [_to] "a" (to), \ |
| 155 | 187 | [_ksize] "K" (size) \ |
| 156 | 188 | : "cc", "0"); \ |
| 157 | − __rc; \ | |
| 158 | −}) | |
| 189 | + return rc; \ | |
| 190 | +} \ | |
| 191 | + \ | |
| 192 | +static __always_inline int \ | |
| 193 | +__get_user_##type(unsigned type *to, unsigned type __user *from, \ | |
| 194 | + unsigned long size) \ | |
| 195 | +{ \ | |
| 196 | + int rc; \ | |
| 197 | + \ | |
| 198 | + rc = __get_user_##type##_noinstr(to, from, size); \ | |
| 199 | + instrument_get_user(*to); \ | |
| 200 | + return rc; \ | |
| 201 | +} | |
| 202 | + | |
| 203 | +DEFINE_GET_USER(char); | |
| 204 | +DEFINE_GET_USER(short); | |
| 205 | +DEFINE_GET_USER(int); | |
| 206 | +DEFINE_GET_USER(long); | |
| 159 | 207 | |
| 160 | 208 | static __always_inline int __get_user_fn(void *x, const void __user *ptr, unsigned long size) |
| 161 | 209 | { |
| @@ -163,24 +211,24 @@ static __always_inline int __get_user_fn(void *x, const void __user *ptr, unsign · 1 unmodified line | ||
| 163 | 211 | |
| 164 | 212 | switch (size) { |
| 165 | 213 | case 1: |
| 166 | − rc = __get_user_asm((unsigned char *)x, | |
| 167 | − (unsigned char __user *)ptr, | |
| 168 | − size); | |
| 214 | + rc = __get_user_char((unsigned char *)x, | |
| 215 | + (unsigned char __user *)ptr, | |
| 216 | + size); | |
| 169 | 217 | break; |
| 170 | 218 | case 2: |
| 171 | − rc = __get_user_asm((unsigned short *)x, | |
| 172 | − (unsigned short __user *)ptr, | |
| 173 | − size); | |
| 219 | + rc = __get_user_short((unsigned short *)x, | |
| 220 | + (unsigned short __user *)ptr, | |
| 221 | + size); | |
| 174 | 222 | break; |
| 175 | 223 | case 4: |
| 176 | − rc = __get_user_asm((unsigned int *)x, | |
| 224 | + rc = __get_user_int((unsigned int *)x, | |
| 177 | 225 | (unsigned int __user *)ptr, |
| 178 | 226 | size); |
| 179 | 227 | break; |
| 180 | 228 | case 8: |
| 181 | − rc = __get_user_asm((unsigned long *)x, | |
| 182 | − (unsigned long __user *)ptr, | |
| 183 | − size); | |
| 229 | + rc = __get_user_long((unsigned long *)x, | |
| 230 | + (unsigned long __user *)ptr, | |
| 231 | + size); | |
| 184 | 232 | break; |
| 185 | 233 | default: |
| 186 | 234 | __get_user_bad(); |
| @@ -282,12 +282,14 @@ int diag224(void *ptr) · 281 unmodified lines | ||
| 282 | 282 | int rc = -EOPNOTSUPP; |
| 283 | 283 | |
| 284 | 284 | diag_stat_inc(DIAG_STAT_X224); |
| 285 | − asm volatile( | |
| 286 | − " diag %1,%2,0x224\n" | |
| 287 | − "0: lhi %0,0x0\n" | |
| 285 | + asm volatile("\n" | |
| 286 | + " diag %[type],%[addr],0x224\n" | |
| 287 | + "0: lhi %[rc],0\n" | |
| 288 | 288 | "1:\n" |
| 289 | 289 | EX_TABLE(0b,1b) |
| 290 | − : "+d" (rc) :"d" (0), "d" (addr) : "memory"); | |
| 290 | + : [rc] "+d" (rc) | |
| 291 | + , "=m" (*(struct { char buf[PAGE_SIZE]; } *)ptr) | |
| 292 | + : [type] "d" (0), [addr] "d" (addr)); | |
| 291 | 293 | return rc; |
| 292 | 294 | } |
| 293 | 295 | EXPORT_SYMBOL(diag224); |
| @@ -12,6 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <linux/ftrace.h> |
| 13 | 13 | #include <linux/kernel.h> |
| 14 | 14 | #include <linux/types.h> |
| 15 | +#include <linux/kmsan-checks.h> | |
| 15 | 16 | #include <linux/kprobes.h> |
| 16 | 17 | #include <linux/execmem.h> |
| 17 | 18 | #include <trace/syscall.h> |
| @@ -303,6 +304,7 @@ void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip, · 285 unmodified lines | ||
| 303 | 304 | if (bit < 0) |
| 304 | 305 | return; |
| 305 | 306 | |
| 307 | + kmsan_unpoison_memory(fregs, sizeof(*fregs)); | |
| 306 | 308 | regs = ftrace_get_regs(fregs); |
| 307 | 309 | p = get_kprobe((kprobe_opcode_t *)ip); |
| 308 | 310 | if (!regs || unlikely(!p) || kprobe_disabled(p)) |
| @@ -27,6 +27,7 @@ · 26 unmodified lines | ||
| 27 | 27 | #include <linux/uaccess.h> |
| 28 | 28 | #include <linux/cpu.h> |
| 29 | 29 | #include <linux/entry-common.h> |
| 30 | +#include <linux/kmsan.h> | |
| 30 | 31 | #include <asm/asm-extable.h> |
| 31 | 32 | #include <asm/vtime.h> |
| 32 | 33 | #include <asm/fpu.h> |
| @@ -262,6 +263,11 @@ static void monitor_event_exception(struct pt_regs *regs) · 229 unmodified lines | ||
| 262 | 263 | |
| 263 | 264 | void kernel_stack_overflow(struct pt_regs *regs) |
| 264 | 265 | { |
| 266 | + /* | |
| 267 | + * Normally regs are unpoisoned by the generic entry code, but | |
| 268 | + * kernel_stack_overflow() is a rare case that is called bypassing it. | |
| 269 | + */ | |
| 270 | + kmsan_unpoison_entry_regs(regs); | |
| 265 | 271 | bust_spinlocks(1); |
| 266 | 272 | printk("Kernel stack overflow.\n"); |
| 267 | 273 | show_regs(regs); |
| @@ -49,6 +49,8 @@ static inline bool is_final_pt_regs(struct unwind_state *state, · 48 unmodified lines | ||
| 49 | 49 | READ_ONCE_NOCHECK(regs->psw.mask) & PSW_MASK_PSTATE; |
| 50 | 50 | } |
| 51 | 51 | |
| 52 | +/* Avoid KMSAN false positives from touching uninitialized frames. */ | |
| 53 | +__no_kmsan_checks | |
| 52 | 54 | bool unwind_next_frame(struct unwind_state *state) |
| 53 | 55 | { |
| 54 | 56 | struct stack_info *info = &state->stack_info; |
| @@ -118,6 +120,8 @@ out_stop: · 63 unmodified lines | ||
| 118 | 120 | } |
| 119 | 121 | EXPORT_SYMBOL_GPL(unwind_next_frame); |
| 120 | 122 | |
| 123 | +/* Avoid KMSAN false positives from touching uninitialized frames. */ | |
| 124 | +__no_kmsan_checks | |
| 121 | 125 | void __unwind_start(struct unwind_state *state, struct task_struct *task, |
| 122 | 126 | struct pt_regs *regs, unsigned long first_frame) |
| 123 | 127 | { |
| @@ -169,7 +169,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 168 unmodified lines | ||
| 169 | 169 | __set_huge_pte_at(mm, addr, ptep, pte); |
| 170 | 170 | } |
| 171 | 171 | |
| 172 | −pte_t huge_ptep_get(pte_t *ptep) | |
| 172 | +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) | |
| 173 | 173 | { |
| 174 | 174 | return __rste_to_pte(pte_val(*ptep)); |
| 175 | 175 | } |
| @@ -177,7 +177,7 @@ pte_t huge_ptep_get(pte_t *ptep) · 1 unmodified line | ||
| 177 | 177 | pte_t huge_ptep_get_and_clear(struct mm_struct *mm, |
| 178 | 178 | unsigned long addr, pte_t *ptep) |
| 179 | 179 | { |
| 180 | − pte_t pte = huge_ptep_get(ptep); | |
| 180 | + pte_t pte = huge_ptep_get(mm, addr, ptep); | |
| 181 | 181 | pmd_t *pmdp = (pmd_t *) ptep; |
| 182 | 182 | pud_t *pudp = (pud_t *) ptep; |
| 183 | 183 | |
| @@ -490,7 +490,7 @@ void flush_dcache_folio(struct folio *folio) · 489 unmodified lines | ||
| 490 | 490 | } |
| 491 | 491 | set_dcache_dirty(folio, this_cpu); |
| 492 | 492 | } else { |
| 493 | − /* We could delay the flush for the !page_mapping | |
| 493 | + /* We could delay the flush for the !folio_mapping | |
| 494 | 494 | * case too. But that case is for exec env/arg |
| 495 | 495 | * pages and those are %99 certainly going to get |
| 496 | 496 | * faulted into the tlb (and thus flushed) anyways. |
| @@ -988,8 +988,6 @@ static void __meminit free_pagetable(struct page *page, int order) · 987 unmodified lines | ||
| 988 | 988 | |
| 989 | 989 | /* bootmem page has reserved flag */ |
| 990 | 990 | if (PageReserved(page)) { |
| 991 | − __ClearPageReserved(page); | |
| 992 | − | |
| 993 | 991 | magic = page->index; |
| 994 | 992 | if (magic == SECTION_INFO || magic == MIX_SECTION_INFO) { |
| 995 | 993 | while (nr_pages--) |
| @@ -1362,18 +1360,6 @@ void __init mem_init(void) · 366 unmodified lines | ||
| 1362 | 1360 | preallocate_vmalloc_pages(); |
| 1363 | 1361 | } |
| 1364 | 1362 | |
| 1365 | −#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT | |
| 1366 | −int __init deferred_page_init_max_threads(const struct cpumask *node_cpumask) | |
| 1367 | −{ | |
| 1368 | − /* | |
| 1369 | − * More CPUs always led to greater speedups on tested systems, up to | |
| 1370 | − * all the nodes' CPUs. Use all since the system is otherwise idle | |
| 1371 | − * now. | |
| 1372 | − */ | |
| 1373 | − return max_t(int, cpumask_weight(node_cpumask), 1); | |
| 1374 | −} | |
| 1375 | −#endif | |
| 1376 | − | |
| 1377 | 1363 | int kernel_set_to_readonly; |
| 1378 | 1364 | |
| 1379 | 1365 | void mark_rodata_ro(void) |
| @@ -1119,8 +1119,8 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, · 1118 unmodified lines | ||
| 1119 | 1119 | lpinc = PMD_SIZE; |
| 1120 | 1120 | /* |
| 1121 | 1121 | * Clear the PSE flags if the PRESENT flag is not set |
| 1122 | − * otherwise pmd_present/pmd_huge will return true | |
| 1123 | − * even on a non present pmd. | |
| 1122 | + * otherwise pmd_present() will return true even on a non | |
| 1123 | + * present pmd. | |
| 1124 | 1124 | */ |
| 1125 | 1125 | if (!(pgprot_val(ref_prot) & _PAGE_PRESENT)) |
| 1126 | 1126 | pgprot_val(ref_prot) &= ~_PAGE_PSE; |
| @@ -410,9 +410,9 @@ void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, · 409 unmodified lines | ||
| 410 | 410 | |
| 411 | 411 | typedef pte_t *pte_addr_t; |
| 412 | 412 | |
| 413 | −void update_mmu_tlb(struct vm_area_struct *vma, | |
| 414 | − unsigned long address, pte_t *ptep); | |
| 415 | −#define __HAVE_ARCH_UPDATE_MMU_TLB | |
| 413 | +void update_mmu_tlb_range(struct vm_area_struct *vma, | |
| 414 | + unsigned long address, pte_t *ptep, unsigned int nr); | |
| 415 | +#define update_mmu_tlb_range update_mmu_tlb_range | |
| 416 | 416 | |
| 417 | 417 | #endif /* !defined (__ASSEMBLY__) */ |
| 418 | 418 | |
| @@ -163,10 +163,10 @@ void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) · 162 unmodified lines | ||
| 163 | 163 | } |
| 164 | 164 | } |
| 165 | 165 | |
| 166 | −void update_mmu_tlb(struct vm_area_struct *vma, | |
| 167 | − unsigned long address, pte_t *ptep) | |
| 166 | +void update_mmu_tlb_range(struct vm_area_struct *vma, | |
| 167 | + unsigned long address, pte_t *ptep, unsigned int nr) | |
| 168 | 168 | { |
| 169 | − local_flush_tlb_page(vma, address); | |
| 169 | + local_flush_tlb_range(vma, address, address + PAGE_SIZE * nr); | |
| 170 | 170 | } |
| 171 | 171 | |
| 172 | 172 | #ifdef CONFIG_DEBUG_TLB_SANITY |
| @@ -933,17 +933,14 @@ static int hmat_callback(struct notifier_block *self, · 932 unmodified lines | ||
| 933 | 933 | return NOTIFY_OK; |
| 934 | 934 | } |
| 935 | 935 | |
| 936 | −static int hmat_set_default_dram_perf(void) | |
| 936 | +static int __init hmat_set_default_dram_perf(void) | |
| 937 | 937 | { |
| 938 | 938 | int rc; |
| 939 | 939 | int nid, pxm; |
| 940 | 940 | struct memory_target *target; |
| 941 | 941 | struct access_coordinate *attrs; |
| 942 | 942 | |
| 943 | − if (!default_dram_type) | |
| 944 | − return -EIO; | |
| 945 | − | |
| 946 | − for_each_node_mask(nid, default_dram_type->nodes) { | |
| 943 | + for_each_node_mask(nid, default_dram_nodes) { | |
| 947 | 944 | pxm = node_to_pxm(nid); |
| 948 | 945 | target = find_mem_target(pxm); |
| 949 | 946 | if (!target) |
| @@ -2,6 +2,7 @@ · 1 unmodified line | ||
| 2 | 2 | config ZRAM |
| 3 | 3 | tristate "Compressed RAM block device support" |
| 4 | 4 | depends on BLOCK && SYSFS && MMU |
| 5 | + depends on HAVE_ZSMALLOC | |
| 5 | 6 | depends on CRYPTO_LZO || CRYPTO_ZSTD || CRYPTO_LZ4 || CRYPTO_LZ4HC || CRYPTO_842 |
| 6 | 7 | select ZSMALLOC |
| 7 | 8 | help |
| @@ -35,6 +35,7 @@ config UDMABUF · 34 unmodified lines | ||
| 35 | 35 | default n |
| 36 | 36 | depends on DMA_SHARED_BUFFER |
| 37 | 37 | depends on MEMFD_CREATE || COMPILE_TEST |
| 38 | + depends on MMU | |
| 38 | 39 | help |
| 39 | 40 | A driver to let userspace turn memfd regions into dma-bufs. |
| 40 | 41 | Qemu can use this to create host dmabufs for guest framebuffers. |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/miscdevice.h> |
| 11 | 11 | #include <linux/module.h> |
| 12 | 12 | #include <linux/shmem_fs.h> |
| 13 | +#include <linux/hugetlb.h> | |
| 13 | 14 | #include <linux/slab.h> |
| 14 | 15 | #include <linux/udmabuf.h> |
| 15 | 16 | #include <linux/vmalloc.h> |
| @@ -25,9 +26,16 @@ MODULE_PARM_DESC(size_limit_mb, "Max size of a dmabuf, in megabytes. Default is · 9 unmodified lines | ||
| 25 | 26 | |
| 26 | 27 | struct udmabuf { |
| 27 | 28 | pgoff_t pagecount; |
| 28 | − struct page **pages; | |
| 29 | + struct folio **folios; | |
| 29 | 30 | struct sg_table *sg; |
| 30 | 31 | struct miscdevice *device; |
| 32 | + pgoff_t *offsets; | |
| 33 | + struct list_head unpin_list; | |
| 34 | +}; | |
| 35 | + | |
| 36 | +struct udmabuf_folio { | |
| 37 | + struct folio *folio; | |
| 38 | + struct list_head list; | |
| 31 | 39 | }; |
| 32 | 40 | |
| 33 | 41 | static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) |
| @@ -35,12 +43,15 @@ static vm_fault_t udmabuf_vm_fault(struct vm_fault *vmf) · 1 unmodified line | ||
| 35 | 43 | struct vm_area_struct *vma = vmf->vma; |
| 36 | 44 | struct udmabuf *ubuf = vma->vm_private_data; |
| 37 | 45 | pgoff_t pgoff = vmf->pgoff; |
| 46 | + unsigned long pfn; | |
| 38 | 47 | |
| 39 | 48 | if (pgoff >= ubuf->pagecount) |
| 40 | 49 | return VM_FAULT_SIGBUS; |
| 41 | − vmf->page = ubuf->pages[pgoff]; | |
| 42 | − get_page(vmf->page); | |
| 43 | − return 0; | |
| 50 | + | |
| 51 | + pfn = folio_pfn(ubuf->folios[pgoff]); | |
| 52 | + pfn += ubuf->offsets[pgoff] >> PAGE_SHIFT; | |
| 53 | + | |
| 54 | + return vmf_insert_pfn(vma, vmf->address, pfn); | |
| 44 | 55 | } |
| 45 | 56 | |
| 46 | 57 | static const struct vm_operations_struct udmabuf_vm_ops = { |
| @@ -56,17 +67,28 @@ static int mmap_udmabuf(struct dma_buf *buf, struct vm_area_struct *vma) · 9 unmodified lines | ||
| 56 | 67 | |
| 57 | 68 | vma->vm_ops = &udmabuf_vm_ops; |
| 58 | 69 | vma->vm_private_data = ubuf; |
| 70 | + vm_flags_set(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); | |
| 59 | 71 | return 0; |
| 60 | 72 | } |
| 61 | 73 | |
| 62 | 74 | static int vmap_udmabuf(struct dma_buf *buf, struct iosys_map *map) |
| 63 | 75 | { |
| 64 | 76 | struct udmabuf *ubuf = buf->priv; |
| 77 | + struct page **pages; | |
| 65 | 78 | void *vaddr; |
| 79 | + pgoff_t pg; | |
| 66 | 80 | |
| 67 | 81 | dma_resv_assert_held(buf->resv); |
| 68 | 82 | |
| 69 | − vaddr = vm_map_ram(ubuf->pages, ubuf->pagecount, -1); | |
| 83 | + pages = kmalloc_array(ubuf->pagecount, sizeof(*pages), GFP_KERNEL); | |
| 84 | + if (!pages) | |
| 85 | + return -ENOMEM; | |
| 86 | + | |
| 87 | + for (pg = 0; pg < ubuf->pagecount; pg++) | |
| 88 | + pages[pg] = &ubuf->folios[pg]->page; | |
| 89 | + | |
| 90 | + vaddr = vm_map_ram(pages, ubuf->pagecount, -1); | |
| 91 | + kfree(pages); | |
| 70 | 92 | if (!vaddr) |
| 71 | 93 | return -EINVAL; |
| 72 | 94 | |
| @@ -88,23 +110,30 @@ static struct sg_table *get_sg_table(struct device *dev, struct dma_buf *buf, · 15 unmodified lines | ||
| 88 | 110 | { |
| 89 | 111 | struct udmabuf *ubuf = buf->priv; |
| 90 | 112 | struct sg_table *sg; |
| 113 | + struct scatterlist *sgl; | |
| 114 | + unsigned int i = 0; | |
| 91 | 115 | int ret; |
| 92 | 116 | |
| 93 | 117 | sg = kzalloc(sizeof(*sg), GFP_KERNEL); |
| 94 | 118 | if (!sg) |
| 95 | 119 | return ERR_PTR(-ENOMEM); |
| 96 | − ret = sg_alloc_table_from_pages(sg, ubuf->pages, ubuf->pagecount, | |
| 97 | − 0, ubuf->pagecount << PAGE_SHIFT, | |
| 98 | − GFP_KERNEL); | |
| 120 | + | |
| 121 | + ret = sg_alloc_table(sg, ubuf->pagecount, GFP_KERNEL); | |
| 99 | 122 | if (ret < 0) |
| 100 | − goto err; | |
| 123 | + goto err_alloc; | |
| 124 | + | |
| 125 | + for_each_sg(sg->sgl, sgl, ubuf->pagecount, i) | |
| 126 | + sg_set_folio(sgl, ubuf->folios[i], PAGE_SIZE, | |
| 127 | + ubuf->offsets[i]); | |
| 128 | + | |
| 101 | 129 | ret = dma_map_sgtable(dev, sg, direction, 0); |
| 102 | 130 | if (ret < 0) |
| 103 | − goto err; | |
| 131 | + goto err_map; | |
| 104 | 132 | return sg; |
| 105 | 133 | |
| 106 | −err: | |
| 134 | +err_map: | |
| 107 | 135 | sg_free_table(sg); |
| 136 | +err_alloc: | |
| 108 | 137 | kfree(sg); |
| 109 | 138 | return ERR_PTR(ret); |
| 110 | 139 | } |
| @@ -130,18 +159,45 @@ static void unmap_udmabuf(struct dma_buf_attachment *at, · 19 unmodified lines | ||
| 130 | 159 | return put_sg_table(at->dev, sg, direction); |
| 131 | 160 | } |
| 132 | 161 | |
| 162 | +static void unpin_all_folios(struct list_head *unpin_list) | |
| 163 | +{ | |
| 164 | + struct udmabuf_folio *ubuf_folio; | |
| 165 | + | |
| 166 | + while (!list_empty(unpin_list)) { | |
| 167 | + ubuf_folio = list_first_entry(unpin_list, | |
| 168 | + struct udmabuf_folio, list); | |
| 169 | + unpin_folio(ubuf_folio->folio); | |
| 170 | + | |
| 171 | + list_del(&ubuf_folio->list); | |
| 172 | + kfree(ubuf_folio); | |
| 173 | + } | |
| 174 | +} | |
| 175 | + | |
| 176 | +static int add_to_unpin_list(struct list_head *unpin_list, | |
| 177 | + struct folio *folio) | |
| 178 | +{ | |
| 179 | + struct udmabuf_folio *ubuf_folio; | |
| 180 | + | |
| 181 | + ubuf_folio = kzalloc(sizeof(*ubuf_folio), GFP_KERNEL); | |
| 182 | + if (!ubuf_folio) | |
| 183 | + return -ENOMEM; | |
| 184 | + | |
| 185 | + ubuf_folio->folio = folio; | |
| 186 | + list_add_tail(&ubuf_folio->list, unpin_list); | |
| 187 | + return 0; | |
| 188 | +} | |
| 189 | + | |
| 133 | 190 | static void release_udmabuf(struct dma_buf *buf) |
| 134 | 191 | { |
| 135 | 192 | struct udmabuf *ubuf = buf->priv; |
| 136 | 193 | struct device *dev = ubuf->device->this_device; |
| 137 | − pgoff_t pg; | |
| 138 | 194 | |
| 139 | 195 | if (ubuf->sg) |
| 140 | 196 | put_sg_table(dev, ubuf->sg, DMA_BIDIRECTIONAL); |
| 141 | 197 | |
| 142 | − for (pg = 0; pg < ubuf->pagecount; pg++) | |
| 143 | − put_page(ubuf->pages[pg]); | |
| 144 | − kfree(ubuf->pages); | |
| 198 | + unpin_all_folios(&ubuf->unpin_list); | |
| 199 | + kfree(ubuf->offsets); | |
| 200 | + kfree(ubuf->folios); | |
| 145 | 201 | kfree(ubuf); |
| 146 | 202 | } |
| 147 | 203 | |
| @@ -194,24 +250,64 @@ static const struct dma_buf_ops udmabuf_ops = { · 46 unmodified lines | ||
| 194 | 250 | #define SEALS_WANTED (F_SEAL_SHRINK) |
| 195 | 251 | #define SEALS_DENIED (F_SEAL_WRITE) |
| 196 | 252 | |
| 253 | +static int check_memfd_seals(struct file *memfd) | |
| 254 | +{ | |
| 255 | + int seals; | |
| 256 | + | |
| 257 | + if (!memfd) | |
| 258 | + return -EBADFD; | |
| 259 | + | |
| 260 | + if (!shmem_file(memfd) && !is_file_hugepages(memfd)) | |
| 261 | + return -EBADFD; | |
| 262 | + | |
| 263 | + seals = memfd_fcntl(memfd, F_GET_SEALS, 0); | |
| 264 | + if (seals == -EINVAL) | |
| 265 | + return -EBADFD; | |
| 266 | + | |
| 267 | + if ((seals & SEALS_WANTED) != SEALS_WANTED || | |
| 268 | + (seals & SEALS_DENIED) != 0) | |
| 269 | + return -EINVAL; | |
| 270 | + | |
| 271 | + return 0; | |
| 272 | +} | |
| 273 | + | |
| 274 | +static int export_udmabuf(struct udmabuf *ubuf, | |
| 275 | + struct miscdevice *device, | |
| 276 | + u32 flags) | |
| 277 | +{ | |
| 278 | + DEFINE_DMA_BUF_EXPORT_INFO(exp_info); | |
| 279 | + struct dma_buf *buf; | |
| 280 | + | |
| 281 | + ubuf->device = device; | |
| 282 | + exp_info.ops = &udmabuf_ops; | |
| 283 | + exp_info.size = ubuf->pagecount << PAGE_SHIFT; | |
| 284 | + exp_info.priv = ubuf; | |
| 285 | + exp_info.flags = O_RDWR; | |
| 286 | + | |
| 287 | + buf = dma_buf_export(&exp_info); | |
| 288 | + if (IS_ERR(buf)) | |
| 289 | + return PTR_ERR(buf); | |
| 290 | + | |
| 291 | + return dma_buf_fd(buf, flags); | |
| 292 | +} | |
| 293 | + | |
| 197 | 294 | static long udmabuf_create(struct miscdevice *device, |
| 198 | 295 | struct udmabuf_create_list *head, |
| 199 | 296 | struct udmabuf_create_item *list) |
| 200 | 297 | { |
| 201 | − DEFINE_DMA_BUF_EXPORT_INFO(exp_info); | |
| 298 | + pgoff_t pgoff, pgcnt, pglimit, pgbuf = 0; | |
| 299 | + long nr_folios, ret = -EINVAL; | |
| 202 | 300 | struct file *memfd = NULL; |
| 203 | − struct address_space *mapping = NULL; | |
| 301 | + struct folio **folios; | |
| 204 | 302 | struct udmabuf *ubuf; |
| 205 | − struct dma_buf *buf; | |
| 206 | − pgoff_t pgoff, pgcnt, pgidx, pgbuf = 0, pglimit; | |
| 207 | − struct page *page; | |
| 208 | − int seals, ret = -EINVAL; | |
| 209 | − u32 i, flags; | |
| 303 | + u32 i, j, k, flags; | |
| 304 | + loff_t end; | |
| 210 | 305 | |
| 211 | 306 | ubuf = kzalloc(sizeof(*ubuf), GFP_KERNEL); |
| 212 | 307 | if (!ubuf) |
| 213 | 308 | return -ENOMEM; |
| 214 | 309 | |
| 310 | + INIT_LIST_HEAD(&ubuf->unpin_list); | |
| 215 | 311 | pglimit = (size_limit_mb * 1024 * 1024) >> PAGE_SHIFT; |
| 216 | 312 | for (i = 0; i < head->count; i++) { |
| 217 | 313 | if (!IS_ALIGNED(list[i].offset, PAGE_SIZE)) |
| @@ -226,66 +322,84 @@ static long udmabuf_create(struct miscdevice *device, · 8 unmodified lines | ||
| 226 | 322 | if (!ubuf->pagecount) |
| 227 | 323 | goto err; |
| 228 | 324 | |
| 229 | − ubuf->pages = kmalloc_array(ubuf->pagecount, sizeof(*ubuf->pages), | |
| 325 | + ubuf->folios = kmalloc_array(ubuf->pagecount, sizeof(*ubuf->folios), | |
| 230 | 326 | GFP_KERNEL); |
| 231 | − if (!ubuf->pages) { | |
| 327 | + if (!ubuf->folios) { | |
| 328 | + ret = -ENOMEM; | |
| 329 | + goto err; | |
| 330 | + } | |
| 331 | + ubuf->offsets = kcalloc(ubuf->pagecount, sizeof(*ubuf->offsets), | |
| 332 | + GFP_KERNEL); | |
| 333 | + if (!ubuf->offsets) { | |
| 232 | 334 | ret = -ENOMEM; |
| 233 | 335 | goto err; |
| 234 | 336 | } |
| 235 | 337 | |
| 236 | 338 | pgbuf = 0; |
| 237 | 339 | for (i = 0; i < head->count; i++) { |
| 238 | − ret = -EBADFD; | |
| 239 | 340 | memfd = fget(list[i].memfd); |
| 240 | − if (!memfd) | |
| 341 | + ret = check_memfd_seals(memfd); | |
| 342 | + if (ret < 0) | |
| 241 | 343 | goto err; |
| 242 | − mapping = memfd->f_mapping; | |
| 243 | − if (!shmem_mapping(mapping)) | |
| 244 | − goto err; | |
| 245 | − seals = memfd_fcntl(memfd, F_GET_SEALS, 0); | |
| 246 | − if (seals == -EINVAL) | |
| 344 | + | |
| 345 | + pgcnt = list[i].size >> PAGE_SHIFT; | |
| 346 | + folios = kmalloc_array(pgcnt, sizeof(*folios), GFP_KERNEL); | |
| 347 | + if (!folios) { | |
| 348 | + ret = -ENOMEM; | |
| 247 | 349 | goto err; |
| 248 | − ret = -EINVAL; | |
| 249 | − if ((seals & SEALS_WANTED) != SEALS_WANTED || | |
| 250 | − (seals & SEALS_DENIED) != 0) | |
| 350 | + } | |
| 351 | + | |
| 352 | + end = list[i].offset + (pgcnt << PAGE_SHIFT) - 1; | |
| 353 | + ret = memfd_pin_folios(memfd, list[i].offset, end, | |
| 354 | + folios, pgcnt, &pgoff); | |
| 355 | + if (ret <= 0) { | |
| 356 | + kfree(folios); | |
| 357 | + if (!ret) | |
| 358 | + ret = -EINVAL; | |
| 251 | 359 | goto err; |
| 252 | − pgoff = list[i].offset >> PAGE_SHIFT; | |
| 253 | − pgcnt = list[i].size >> PAGE_SHIFT; | |
| 254 | − for (pgidx = 0; pgidx < pgcnt; pgidx++) { | |
| 255 | − page = shmem_read_mapping_page(mapping, pgoff + pgidx); | |
| 256 | − if (IS_ERR(page)) { | |
| 257 | − ret = PTR_ERR(page); | |
| 258 | − goto err; | |
| 360 | + } | |
| 361 | + | |
| 362 | + nr_folios = ret; | |
| 363 | + pgoff >>= PAGE_SHIFT; | |
| 364 | + for (j = 0, k = 0; j < pgcnt; j++) { | |
| 365 | + ubuf->folios[pgbuf] = folios[k]; | |
| 366 | + ubuf->offsets[pgbuf] = pgoff << PAGE_SHIFT; | |
| 367 | + | |
| 368 | + if (j == 0 || ubuf->folios[pgbuf-1] != folios[k]) { | |
| 369 | + ret = add_to_unpin_list(&ubuf->unpin_list, | |
| 370 | + folios[k]); | |
| 371 | + if (ret < 0) { | |
| 372 | + kfree(folios); | |
| 373 | + goto err; | |
| 374 | + } | |
| 375 | + } | |
| 376 | + | |
| 377 | + pgbuf++; | |
| 378 | + if (++pgoff == folio_nr_pages(folios[k])) { | |
| 379 | + pgoff = 0; | |
| 380 | + if (++k == nr_folios) | |
| 381 | + break; | |
| 259 | 382 | } |
| 260 | − ubuf->pages[pgbuf++] = page; | |
| 261 | 383 | } |
| 384 | + | |
| 385 | + kfree(folios); | |
| 262 | 386 | fput(memfd); |
| 263 | 387 | memfd = NULL; |
| 264 | 388 | } |
| 265 | 389 | |
| 266 | − exp_info.ops = &udmabuf_ops; | |
| 267 | − exp_info.size = ubuf->pagecount << PAGE_SHIFT; | |
| 268 | − exp_info.priv = ubuf; | |
| 269 | − exp_info.flags = O_RDWR; | |
| 270 | − | |
| 271 | − ubuf->device = device; | |
| 272 | − buf = dma_buf_export(&exp_info); | |
| 273 | − if (IS_ERR(buf)) { | |
| 274 | − ret = PTR_ERR(buf); | |
| 390 | + flags = head->flags & UDMABUF_FLAGS_CLOEXEC ? O_CLOEXEC : 0; | |
| 391 | + ret = export_udmabuf(ubuf, device, flags); | |
| 392 | + if (ret < 0) | |
| 275 | 393 | goto err; |
| 276 | − } | |
| 277 | 394 | |
| 278 | − flags = 0; | |
| 279 | − if (head->flags & UDMABUF_FLAGS_CLOEXEC) | |
| 280 | − flags |= O_CLOEXEC; | |
| 281 | − return dma_buf_fd(buf, flags); | |
| 395 | + return ret; | |
| 282 | 396 | |
| 283 | 397 | err: |
| 284 | − while (pgbuf > 0) | |
| 285 | − put_page(ubuf->pages[--pgbuf]); | |
| 286 | 398 | if (memfd) |
| 287 | 399 | fput(memfd); |
| 288 | − kfree(ubuf->pages); | |
| 400 | + unpin_all_folios(&ubuf->unpin_list); | |
| 401 | + kfree(ubuf->offsets); | |
| 402 | + kfree(ubuf->folios); | |
| 289 | 403 | kfree(ubuf); |
| 290 | 404 | return ret; |
| 291 | 405 | } |
| @@ -683,9 +683,8 @@ static void hv_page_online_one(struct hv_hotadd_state *has, struct page *pg) · 682 unmodified lines | ||
| 683 | 683 | if (!PageOffline(pg)) |
| 684 | 684 | __SetPageOffline(pg); |
| 685 | 685 | return; |
| 686 | − } | |
| 687 | − if (PageOffline(pg)) | |
| 688 | − __ClearPageOffline(pg); | |
| 686 | + } else if (!PageOffline(pg)) | |
| 687 | + return; | |
| 689 | 688 | |
| 690 | 689 | /* This frame is currently backed; online the page. */ |
| 691 | 690 | generic_online_page(pg, 0); |
| @@ -736,7 +736,7 @@ sclp_sync_wait(void) · 735 unmodified lines | ||
| 736 | 736 | cr0_sync.val = cr0.val & ~CR0_IRQ_SUBCLASS_MASK; |
| 737 | 737 | cr0_sync.val |= 1UL << (63 - 54); |
| 738 | 738 | local_ctl_load(0, &cr0_sync); |
| 739 | − __arch_local_irq_stosm(0x01); | |
| 739 | + arch_local_irq_enable_external(); | |
| 740 | 740 | /* Loop until driver state indicates finished request */ |
| 741 | 741 | while (sclp_running_state != sclp_running_state_idle) { |
| 742 | 742 | /* Check for expired request timer */ |
| @@ -146,7 +146,7 @@ static vm_fault_t fb_deferred_io_fault(struct vm_fault *vmf) · 145 unmodified lines | ||
| 146 | 146 | printk(KERN_ERR "no mapping available\n"); |
| 147 | 147 | |
| 148 | 148 | BUG_ON(!page->mapping); |
| 149 | − page->index = vmf->pgoff; /* for page_mkclean() */ | |
| 149 | + page->index = vmf->pgoff; /* for folio_mkclean() */ | |
| 150 | 150 | |
| 151 | 151 | vmf->page = page; |
| 152 | 152 | return 0; |
| @@ -194,7 +194,7 @@ static vm_fault_t fb_deferred_io_track_page(struct fb_info *info, unsigned long · 41 unmodified lines | ||
| 194 | 194 | |
| 195 | 195 | /* |
| 196 | 196 | * We want the page to remain locked from ->page_mkwrite until |
| 197 | − * the PTE is marked dirty to avoid page_mkclean() being called | |
| 197 | + * the PTE is marked dirty to avoid folio_mkclean() being called | |
| 198 | 198 | * before the PTE is updated, which would leave the page ignored |
| 199 | 199 | * by defio. |
| 200 | 200 | * Do this by locking the page here and informing the caller |
| @@ -277,10 +277,11 @@ static void fb_deferred_io_work(struct work_struct *work) · 76 unmodified lines | ||
| 277 | 277 | /* here we mkclean the pages, then do all deferred IO */ |
| 278 | 278 | mutex_lock(&fbdefio->lock); |
| 279 | 279 | list_for_each_entry(pageref, &fbdefio->pagereflist, list) { |
| 280 | − struct page *cur = pageref->page; | |
| 281 | − lock_page(cur); | |
| 282 | − page_mkclean(cur); | |
| 283 | − unlock_page(cur); | |
| 280 | + struct folio *folio = page_folio(pageref->page); | |
| 281 | + | |
| 282 | + folio_lock(folio); | |
| 283 | + folio_mkclean(folio); | |
| 284 | + folio_unlock(folio); | |
| 284 | 285 | } |
| 285 | 286 | |
| 286 | 287 | /* driver's callback with pagereflist */ |
| @@ -1146,12 +1146,16 @@ static void virtio_mem_set_fake_offline(unsigned long pfn, · 1145 unmodified lines | ||
| 1146 | 1146 | for (; nr_pages--; pfn++) { |
| 1147 | 1147 | struct page *page = pfn_to_page(pfn); |
| 1148 | 1148 | |
| 1149 | − __SetPageOffline(page); | |
| 1150 | − if (!onlined) { | |
| 1149 | + if (!onlined) | |
| 1150 | + /* | |
| 1151 | + * Pages that have not been onlined yet were initialized | |
| 1152 | + * to PageOffline(). Remember that we have to route them | |
| 1153 | + * through generic_online_page(). | |
| 1154 | + */ | |
| 1151 | 1155 | SetPageDirty(page); |
| 1152 | − /* FIXME: remove after cleanups */ | |
| 1153 | − ClearPageReserved(page); | |
| 1154 | − } | |
| 1156 | + else | |
| 1157 | + __SetPageOffline(page); | |
| 1158 | + VM_WARN_ON_ONCE(!PageOffline(page)); | |
| 1155 | 1159 | } |
| 1156 | 1160 | page_offline_end(); |
| 1157 | 1161 | } |
| @@ -1166,9 +1170,11 @@ static void virtio_mem_clear_fake_offline(unsigned long pfn, · 8 unmodified lines | ||
| 1166 | 1170 | for (; nr_pages--; pfn++) { |
| 1167 | 1171 | struct page *page = pfn_to_page(pfn); |
| 1168 | 1172 | |
| 1169 | − __ClearPageOffline(page); | |
| 1170 | 1173 | if (!onlined) |
| 1174 | + /* generic_online_page() will clear PageOffline(). */ | |
| 1171 | 1175 | ClearPageDirty(page); |
| 1176 | + else | |
| 1177 | + __ClearPageOffline(page); | |
| 1172 | 1178 | } |
| 1173 | 1179 | } |
| 1174 | 1180 | |
| @@ -1263,12 +1269,6 @@ static void virtio_mem_fake_offline_going_offline(unsigned long pfn, · 88 unmodified lines | ||
| 1263 | 1269 | struct page *page; |
| 1264 | 1270 | unsigned long i; |
| 1265 | 1271 | |
| 1266 | − /* | |
| 1267 | − * Drop our reference to the pages so the memory can get offlined | |
| 1268 | − * and add the unplugged pages to the managed page counters (so | |
| 1269 | − * offlining code can correctly subtract them again). | |
| 1270 | − */ | |
| 1271 | − adjust_managed_page_count(pfn_to_page(pfn), nr_pages); | |
| 1272 | 1272 | /* Drop our reference to the pages so the memory can get offlined. */ |
| 1273 | 1273 | for (i = 0; i < nr_pages; i++) { |
| 1274 | 1274 | page = pfn_to_page(pfn + i); |
| @@ -1287,10 +1287,9 @@ static void virtio_mem_fake_offline_cancel_offline(unsigned long pfn, · 12 unmodified lines | ||
| 1287 | 1287 | unsigned long i; |
| 1288 | 1288 | |
| 1289 | 1289 | /* |
| 1290 | − * Get the reference we dropped when going offline and subtract the | |
| 1291 | − * unplugged pages from the managed page counters. | |
| 1290 | + * Get the reference again that we dropped via page_ref_dec_and_test() | |
| 1291 | + * when going offline. | |
| 1292 | 1292 | */ |
| 1293 | − adjust_managed_page_count(pfn_to_page(pfn), -nr_pages); | |
| 1294 | 1293 | for (i = 0; i < nr_pages; i++) |
| 1295 | 1294 | page_ref_inc(pfn_to_page(pfn + i)); |
| 1296 | 1295 | } |
| @@ -146,7 +146,8 @@ static DECLARE_WAIT_QUEUE_HEAD(balloon_wq); · 145 unmodified lines | ||
| 146 | 146 | /* balloon_append: add the given page to the balloon. */ |
| 147 | 147 | static void balloon_append(struct page *page) |
| 148 | 148 | { |
| 149 | − __SetPageOffline(page); | |
| 149 | + if (!PageOffline(page)) | |
| 150 | + __SetPageOffline(page); | |
| 150 | 151 | |
| 151 | 152 | /* Lowmem is re-populated first, so highmem pages go at list tail. */ |
| 152 | 153 | if (PageHighMem(page)) { |
| @@ -412,7 +413,11 @@ static enum bp_state increase_reservation(unsigned long nr_pages) · 259 unmodified lines | ||
| 412 | 413 | |
| 413 | 414 | xenmem_reservation_va_mapping_update(1, &page, &frame_list[i]); |
| 414 | 415 | |
| 415 | − /* Relinquish the page back to the allocator. */ | |
| 416 | + /* | |
| 417 | + * Relinquish the page back to the allocator. Note that | |
| 418 | + * some pages, including ones added via xen_online_page(), might | |
| 419 | + * not be marked reserved; free_reserved_page() will handle that. | |
| 420 | + */ | |
| 416 | 421 | free_reserved_page(page); |
| 417 | 422 | } |
| 418 | 423 | |
| @@ -533,14 +533,14 @@ static int afs_dir_iterate(struct inode *dir, struct dir_context *ctx, · 532 unmodified lines | ||
| 533 | 533 | break; |
| 534 | 534 | } |
| 535 | 535 | |
| 536 | − offset = round_down(ctx->pos, sizeof(*dblock)) - folio_file_pos(folio); | |
| 536 | + offset = round_down(ctx->pos, sizeof(*dblock)) - folio_pos(folio); | |
| 537 | 537 | size = min_t(loff_t, folio_size(folio), |
| 538 | − req->actual_len - folio_file_pos(folio)); | |
| 538 | + req->actual_len - folio_pos(folio)); | |
| 539 | 539 | |
| 540 | 540 | do { |
| 541 | 541 | dblock = kmap_local_folio(folio, offset); |
| 542 | 542 | ret = afs_dir_iterate_block(dvnode, ctx, dblock, |
| 543 | − folio_file_pos(folio) + offset); | |
| 543 | + folio_pos(folio) + offset); | |
| 544 | 544 | kunmap_local(dblock); |
| 545 | 545 | if (ret != 1) |
| 546 | 546 | goto out; |
| @@ -256,7 +256,7 @@ void afs_edit_dir_add(struct afs_vnode *vnode, · 255 unmodified lines | ||
| 256 | 256 | folio = folio0; |
| 257 | 257 | } |
| 258 | 258 | |
| 259 | − block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_file_pos(folio)); | |
| 259 | + block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_pos(folio)); | |
| 260 | 260 | |
| 261 | 261 | /* Abandon the edit if we got a callback break. */ |
| 262 | 262 | if (!test_bit(AFS_VNODE_DIR_VALID, &vnode->flags)) |
| @@ -417,7 +417,7 @@ void afs_edit_dir_remove(struct afs_vnode *vnode, · 154 unmodified lines | ||
| 417 | 417 | folio = folio0; |
| 418 | 418 | } |
| 419 | 419 | |
| 420 | − block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_file_pos(folio)); | |
| 420 | + block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_pos(folio)); | |
| 421 | 421 | |
| 422 | 422 | /* Abandon the edit if we got a callback break. */ |
| 423 | 423 | if (!test_bit(AFS_VNODE_DIR_VALID, &vnode->flags)) |
| @@ -410,17 +410,7 @@ static int aio_migrate_folio(struct address_space *mapping, struct folio *dst, · 409 unmodified lines | ||
| 410 | 410 | struct kioctx *ctx; |
| 411 | 411 | unsigned long flags; |
| 412 | 412 | pgoff_t idx; |
| 413 | − int rc; | |
| 414 | − | |
| 415 | − /* | |
| 416 | − * We cannot support the _NO_COPY case here, because copy needs to | |
| 417 | − * happen under the ctx->completion_lock. That does not work with the | |
| 418 | − * migration workflow of MIGRATE_SYNC_NO_COPY. | |
| 419 | − */ | |
| 420 | − if (mode == MIGRATE_SYNC_NO_COPY) | |
| 421 | − return -EINVAL; | |
| 422 | − | |
| 423 | − rc = 0; | |
| 413 | + int rc = 0; | |
| 424 | 414 | |
| 425 | 415 | /* mapping->i_private_lock here protects against the kioctx teardown. */ |
| 426 | 416 | spin_lock(&mapping->i_private_lock); |
| @@ -465,7 +455,8 @@ static int aio_migrate_folio(struct address_space *mapping, struct folio *dst, · 38 unmodified lines | ||
| 465 | 455 | * events from being lost. |
| 466 | 456 | */ |
| 467 | 457 | spin_lock_irqsave(&ctx->completion_lock, flags); |
| 468 | − folio_migrate_copy(dst, src); | |
| 458 | + folio_copy(dst, src); | |
| 459 | + folio_migrate_flags(dst, src); | |
| 469 | 460 | BUG_ON(ctx->ring_folios[idx] != src); |
| 470 | 461 | ctx->ring_folios[idx] = dst; |
| 471 | 462 | spin_unlock_irqrestore(&ctx->completion_lock, flags); |
| @@ -2981,8 +2981,7 @@ static int relocate_one_folio(struct reloc_control *rc, · 2980 unmodified lines | ||
| 2981 | 2981 | |
| 2982 | 2982 | if (folio_test_readahead(folio)) |
| 2983 | 2983 | page_cache_async_readahead(inode->i_mapping, ra, NULL, |
| 2984 | − folio, index, | |
| 2985 | − last_index + 1 - index); | |
| 2984 | + folio, last_index + 1 - index); | |
| 2986 | 2985 | |
| 2987 | 2986 | if (!folio_test_uptodate(folio)) { |
| 2988 | 2987 | btrfs_read_folio(NULL, folio); |
| @@ -5306,7 +5306,7 @@ static int put_file_data(struct send_ctx *sctx, u64 offset, u32 len) · 5305 unmodified lines | ||
| 5306 | 5306 | |
| 5307 | 5307 | if (folio_test_readahead(folio)) |
| 5308 | 5308 | page_cache_async_readahead(mapping, &sctx->ra, NULL, folio, |
| 5309 | − index, last_index + 1 - index); | |
| 5309 | + last_index + 1 - index); | |
| 5310 | 5310 | |
| 5311 | 5311 | if (!folio_test_uptodate(folio)) { |
| 5312 | 5312 | btrfs_read_folio(NULL, folio); |
| @@ -141,7 +141,7 @@ __dcache_find_get_entry(struct dentry *parent, u64 idx, · 140 unmodified lines | ||
| 141 | 141 | if (ptr_pos >= i_size_read(dir)) |
| 142 | 142 | return NULL; |
| 143 | 143 | |
| 144 | − if (!cache_ctl->page || ptr_pgoff != page_index(cache_ctl->page)) { | |
| 144 | + if (!cache_ctl->page || ptr_pgoff != cache_ctl->page->index) { | |
| 145 | 145 | ceph_readdir_cache_release(cache_ctl); |
| 146 | 146 | cache_ctl->page = find_lock_page(&dir->i_data, ptr_pgoff); |
| 147 | 147 | if (!cache_ctl->page) { |
| @@ -1863,7 +1863,7 @@ static int fill_readdir_cache(struct inode *dir, struct dentry *dn, · 1862 unmodified lines | ||
| 1863 | 1863 | unsigned idx = ctl->index % nsize; |
| 1864 | 1864 | pgoff_t pgoff = ctl->index / nsize; |
| 1865 | 1865 | |
| 1866 | − if (!ctl->page || pgoff != page_index(ctl->page)) { | |
| 1866 | + if (!ctl->page || pgoff != ctl->page->index) { | |
| 1867 | 1867 | ceph_readdir_cache_release(ctl); |
| 1868 | 1868 | if (idx == 0) |
| 1869 | 1869 | ctl->page = grab_cache_page(&dir->i_data, pgoff); |
| @@ -222,13 +222,13 @@ generic_hugetlb_get_unmapped_area(struct file *file, unsigned long addr, · 221 unmodified lines | ||
| 222 | 222 | unsigned long flags) |
| 223 | 223 | { |
| 224 | 224 | struct mm_struct *mm = current->mm; |
| 225 | − struct vm_area_struct *vma; | |
| 225 | + struct vm_area_struct *vma, *prev; | |
| 226 | 226 | struct hstate *h = hstate_file(file); |
| 227 | 227 | const unsigned long mmap_end = arch_get_mmap_end(addr, len, flags); |
| 228 | 228 | |
| 229 | 229 | if (len & ~huge_page_mask(h)) |
| 230 | 230 | return -EINVAL; |
| 231 | − if (len > TASK_SIZE) | |
| 231 | + if (len > mmap_end - mmap_min_addr) | |
| 232 | 232 | return -ENOMEM; |
| 233 | 233 | |
| 234 | 234 | if (flags & MAP_FIXED) { |
| @@ -239,9 +239,10 @@ generic_hugetlb_get_unmapped_area(struct file *file, unsigned long addr, · 4 unmodified lines | ||
| 239 | 239 | |
| 240 | 240 | if (addr) { |
| 241 | 241 | addr = ALIGN(addr, huge_page_size(h)); |
| 242 | − vma = find_vma(mm, addr); | |
| 243 | − if (mmap_end - len >= addr && | |
| 244 | − (!vma || addr + len <= vm_start_gap(vma))) | |
| 242 | + vma = find_vma_prev(mm, addr, &prev); | |
| 243 | + if (mmap_end - len >= addr && addr >= mmap_min_addr && | |
| 244 | + (!vma || addr + len <= vm_start_gap(vma)) && | |
| 245 | + (!prev || addr >= vm_end_gap(prev))) | |
| 245 | 246 | return addr; |
| 246 | 247 | } |
| 247 | 248 | |
| @@ -422,7 +423,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, · 174 unmodified lines | ||
| 422 | 423 | if (!ptep) |
| 423 | 424 | return false; |
| 424 | 425 | |
| 425 | − pte = huge_ptep_get(ptep); | |
| 426 | + pte = huge_ptep_get(vma->vm_mm, addr, ptep); | |
| 426 | 427 | if (huge_pte_none(pte) || !pte_present(pte)) |
| 427 | 428 | return false; |
| 428 | 429 | |
| @@ -892,7 +893,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 463 unmodified lines | ||
| 892 | 893 | error = PTR_ERR(folio); |
| 893 | 894 | goto out; |
| 894 | 895 | } |
| 895 | − clear_huge_page(&folio->page, addr, pages_per_huge_page(h)); | |
| 896 | + folio_zero_user(folio, ALIGN_DOWN(addr, hpage_size)); | |
| 896 | 897 | __folio_mark_uptodate(folio); |
| 897 | 898 | error = hugetlb_add_to_page_cache(folio, mapping, index); |
| 898 | 899 | if (unlikely(error)) { |
| @@ -1128,10 +1129,7 @@ static int hugetlbfs_migrate_folio(struct address_space *mapping, · 229 unmodified lines | ||
| 1128 | 1129 | hugetlb_set_folio_subpool(src, NULL); |
| 1129 | 1130 | } |
| 1130 | 1131 | |
| 1131 | − if (mode != MIGRATE_SYNC_NO_COPY) | |
| 1132 | − folio_migrate_copy(dst, src); | |
| 1133 | − else | |
| 1134 | − folio_migrate_flags(dst, src); | |
| 1132 | + folio_migrate_flags(dst, src); | |
| 1135 | 1133 | |
| 1136 | 1134 | return MIGRATEPAGE_SUCCESS; |
| 1137 | 1135 | } |
| @@ -271,7 +271,7 @@ int netfs_read_folio(struct file *file, struct folio *folio) · 270 unmodified lines | ||
| 271 | 271 | kenter("%lx", folio->index); |
| 272 | 272 | |
| 273 | 273 | rreq = netfs_alloc_request(mapping, file, |
| 274 | − folio_file_pos(folio), folio_size(folio), | |
| 274 | + folio_pos(folio), folio_size(folio), | |
| 275 | 275 | NETFS_READPAGE); |
| 276 | 276 | if (IS_ERR(rreq)) { |
| 277 | 277 | ret = PTR_ERR(rreq); |
| @@ -470,7 +470,7 @@ retry: · 192 unmodified lines | ||
| 470 | 470 | } |
| 471 | 471 | |
| 472 | 472 | rreq = netfs_alloc_request(mapping, file, |
| 473 | − folio_file_pos(folio), folio_size(folio), | |
| 473 | + folio_pos(folio), folio_size(folio), | |
| 474 | 474 | NETFS_READ_FOR_WRITE); |
| 475 | 475 | if (IS_ERR(rreq)) { |
| 476 | 476 | ret = PTR_ERR(rreq); |
| @@ -54,7 +54,7 @@ static enum netfs_how_to_modify netfs_how_to_modify(struct netfs_inode *ctx, · 53 unmodified lines | ||
| 54 | 54 | { |
| 55 | 55 | struct netfs_folio *finfo = netfs_folio_info(folio); |
| 56 | 56 | struct netfs_group *group = netfs_folio_group(folio); |
| 57 | − loff_t pos = folio_file_pos(folio); | |
| 57 | + loff_t pos = folio_pos(folio); | |
| 58 | 58 | |
| 59 | 59 | kenter(""); |
| 60 | 60 | |
| @@ -591,7 +591,7 @@ static vm_fault_t nfs_vm_page_mkwrite(struct vm_fault *vmf) · 590 unmodified lines | ||
| 591 | 591 | |
| 592 | 592 | dfprintk(PAGECACHE, "NFS: vm_page_mkwrite(%pD2(%lu), offset %lld)\n", |
| 593 | 593 | filp, filp->f_mapping->host->i_ino, |
| 594 | − (long long)folio_file_pos(folio)); | |
| 594 | + (long long)folio_pos(folio)); | |
| 595 | 595 | |
| 596 | 596 | sb_start_pagefault(inode->i_sb); |
| 597 | 597 | |
| @@ -46,6 +46,10 @@ static inline void nfs_add_stats(const struct inode *inode, · 45 unmodified lines | ||
| 46 | 46 | nfs_add_server_stats(NFS_SERVER(inode), stat, addend); |
| 47 | 47 | } |
| 48 | 48 | |
| 49 | +/* | |
| 50 | + * This specialized allocator has to be a macro for its allocations to be | |
| 51 | + * accounted separately (to have a separate alloc_tag). | |
| 52 | + */ | |
| 49 | 53 | #define nfs_alloc_iostats() alloc_percpu(struct nfs_iostats) |
| 50 | 54 | |
| 51 | 55 | static inline void nfs_free_iostats(struct nfs_iostats __percpu *stats) |
| @@ -182,7 +182,7 @@ static void nfs_grow_file(struct folio *folio, unsigned int offset, · 181 unmodified lines | ||
| 182 | 182 | end_index = ((i_size - 1) >> folio_shift(folio)) << folio_order(folio); |
| 183 | 183 | if (i_size > 0 && folio->index < end_index) |
| 184 | 184 | goto out; |
| 185 | − end = folio_file_pos(folio) + (loff_t)offset + (loff_t)count; | |
| 185 | + end = folio_pos(folio) + (loff_t)offset + (loff_t)count; | |
| 186 | 186 | if (i_size >= end) |
| 187 | 187 | goto out; |
| 188 | 188 | trace_nfs_size_grow(inode, end); |
| @@ -1344,7 +1344,7 @@ int nfs_update_folio(struct file *file, struct folio *folio, · 1155 unmodified lines | ||
| 1344 | 1344 | nfs_inc_stats(inode, NFSIOS_VFSUPDATEPAGE); |
| 1345 | 1345 | |
| 1346 | 1346 | dprintk("NFS: nfs_update_folio(%pD2 %d@%lld)\n", file, count, |
| 1347 | − (long long)(folio_file_pos(folio) + offset)); | |
| 1347 | + (long long)(folio_pos(folio) + offset)); | |
| 1348 | 1348 | |
| 1349 | 1349 | if (!count) |
| 1350 | 1350 | goto out; |
| @@ -450,15 +450,9 @@ int nilfs_bmap_test_and_clear_dirty(struct nilfs_bmap *bmap) · 449 unmodified lines | ||
| 450 | 450 | __u64 nilfs_bmap_data_get_key(const struct nilfs_bmap *bmap, |
| 451 | 451 | const struct buffer_head *bh) |
| 452 | 452 | { |
| 453 | − struct buffer_head *pbh; | |
| 454 | − __u64 key; | |
| 453 | + loff_t pos = folio_pos(bh->b_folio) + bh_offset(bh); | |
| 455 | 454 | |
| 456 | − key = page_index(bh->b_page) << (PAGE_SHIFT - | |
| 457 | − bmap->b_inode->i_blkbits); | |
| 458 | − for (pbh = page_buffers(bh->b_page); pbh != bh; pbh = pbh->b_this_page) | |
| 459 | − key++; | |
| 460 | − | |
| 461 | − return key; | |
| 455 | + return pos >> bmap->b_inode->i_blkbits; | |
| 462 | 456 | } |
| 463 | 457 | |
| 464 | 458 | __u64 nilfs_bmap_find_target_seq(const struct nilfs_bmap *bmap, __u64 key) |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/binfmts.h> |
| 14 | 14 | #include <linux/sched/coredump.h> |
| 15 | 15 | #include <linux/sched/task.h> |
| 16 | +#include <linux/mm.h> | |
| 16 | 17 | |
| 17 | 18 | struct ctl_table_header; |
| 18 | 19 | struct mempolicy; |
| @@ -142,6 +143,38 @@ unsigned name_to_int(const struct qstr *qstr); · 123 unmodified lines | ||
| 142 | 143 | /* Worst case buffer size needed for holding an integer. */ |
| 143 | 144 | #define PROC_NUMBUF 13 |
| 144 | 145 | |
| 146 | +/** | |
| 147 | + * folio_precise_page_mapcount() - Number of mappings of this folio page. | |
| 148 | + * @folio: The folio. | |
| 149 | + * @page: The page. | |
| 150 | + * | |
| 151 | + * The number of present user page table entries that reference this page | |
| 152 | + * as tracked via the RMAP: either referenced directly (PTE) or as part of | |
| 153 | + * a larger area that covers this page (e.g., PMD). | |
| 154 | + * | |
| 155 | + * Use this function only for the calculation of existing statistics | |
| 156 | + * (USS, PSS, mapcount_max) and for debugging purposes (/proc/kpagecount). | |
| 157 | + * | |
| 158 | + * Do not add new users. | |
| 159 | + * | |
| 160 | + * Returns: The number of mappings of this folio page. 0 for | |
| 161 | + * folios that are not mapped to user space or are not tracked via the RMAP | |
| 162 | + * (e.g., shared zeropage). | |
| 163 | + */ | |
| 164 | +static inline int folio_precise_page_mapcount(struct folio *folio, | |
| 165 | + struct page *page) | |
| 166 | +{ | |
| 167 | + int mapcount = atomic_read(&page->_mapcount) + 1; | |
| 168 | + | |
| 169 | + /* Handle page_has_type() pages */ | |
| 170 | + if (mapcount < PAGE_MAPCOUNT_RESERVE + 1) | |
| 171 | + mapcount = 0; | |
| 172 | + if (folio_test_large(folio)) | |
| 173 | + mapcount += folio_entire_mapcount(folio); | |
| 174 | + | |
| 175 | + return mapcount; | |
| 176 | +} | |
| 177 | + | |
| 145 | 178 | /* |
| 146 | 179 | * array.c |
| 147 | 180 | */ |
| @@ -37,21 +37,19 @@ static inline unsigned long get_max_dump_pfn(void) · 36 unmodified lines | ||
| 37 | 37 | #endif |
| 38 | 38 | } |
| 39 | 39 | |
| 40 | −/* /proc/kpagecount - an array exposing page counts | |
| 40 | +/* /proc/kpagecount - an array exposing page mapcounts | |
| 41 | 41 | * |
| 42 | 42 | * Each entry is a u64 representing the corresponding |
| 43 | − * physical page count. | |
| 43 | + * physical page mapcount. | |
| 44 | 44 | */ |
| 45 | 45 | static ssize_t kpagecount_read(struct file *file, char __user *buf, |
| 46 | 46 | size_t count, loff_t *ppos) |
| 47 | 47 | { |
| 48 | 48 | const unsigned long max_dump_pfn = get_max_dump_pfn(); |
| 49 | 49 | u64 __user *out = (u64 __user *)buf; |
| 50 | − struct page *ppage; | |
| 51 | 50 | unsigned long src = *ppos; |
| 52 | 51 | unsigned long pfn; |
| 53 | 52 | ssize_t ret = 0; |
| 54 | − u64 pcount; | |
| 55 | 53 | |
| 56 | 54 | pfn = src / KPMSIZE; |
| 57 | 55 | if (src & KPMMASK || count & KPMMASK) |
| @@ -61,18 +59,19 @@ static ssize_t kpagecount_read(struct file *file, char __user *buf, · 3 unmodified lines | ||
| 61 | 59 | count = min_t(unsigned long, count, (max_dump_pfn * KPMSIZE) - src); |
| 62 | 60 | |
| 63 | 61 | while (count > 0) { |
| 62 | + struct page *page; | |
| 63 | + u64 mapcount = 0; | |
| 64 | + | |
| 64 | 65 | /* |
| 65 | 66 | * TODO: ZONE_DEVICE support requires to identify |
| 66 | 67 | * memmaps that were actually initialized. |
| 67 | 68 | */ |
| 68 | − ppage = pfn_to_online_page(pfn); | |
| 69 | − | |
| 70 | − if (!ppage) | |
| 71 | − pcount = 0; | |
| 72 | − else | |
| 73 | − pcount = page_mapcount(ppage); | |
| 69 | + page = pfn_to_online_page(pfn); | |
| 70 | + if (page) | |
| 71 | + mapcount = folio_precise_page_mapcount(page_folio(page), | |
| 72 | + page); | |
| 74 | 73 | |
| 75 | − if (put_user(pcount, out)) { | |
| 74 | + if (put_user(mapcount, out)) { | |
| 76 | 75 | ret = -EFAULT; |
| 77 | 76 | break; |
| 78 | 77 | } |
| @@ -148,19 +147,16 @@ u64 stable_page_flags(const struct page *page) · 69 unmodified lines | ||
| 148 | 147 | u |= 1 << KPF_COMPOUND_TAIL; |
| 149 | 148 | if (folio_test_hugetlb(folio)) |
| 150 | 149 | u |= 1 << KPF_HUGE; |
| 151 | − /* | |
| 152 | − * We need to check PageLRU/PageAnon | |
| 153 | − * to make sure a given page is a thp, not a non-huge compound page. | |
| 154 | − */ | |
| 155 | − else if (folio_test_large(folio)) { | |
| 156 | − if ((k & (1 << PG_lru)) || is_anon) | |
| 157 | − u |= 1 << KPF_THP; | |
| 158 | − else if (is_huge_zero_folio(folio)) { | |
| 159 | − u |= 1 << KPF_ZERO_PAGE; | |
| 160 | − u |= 1 << KPF_THP; | |
| 161 | − } | |
| 162 | − } else if (is_zero_pfn(page_to_pfn(page))) | |
| 150 | + else if (folio_test_large(folio) && | |
| 151 | + folio_test_large_rmappable(folio)) { | |
| 152 | + /* Note: we indicate any THPs here, not just PMD-sized ones */ | |
| 153 | + u |= 1 << KPF_THP; | |
| 154 | + } else if (is_huge_zero_folio(folio)) { | |
| 163 | 155 | u |= 1 << KPF_ZERO_PAGE; |
| 156 | + u |= 1 << KPF_THP; | |
| 157 | + } else if (is_zero_folio(folio)) { | |
| 158 | + u |= 1 << KPF_ZERO_PAGE; | |
| 159 | + } | |
| 164 | 160 | |
| 165 | 161 | /* |
| 166 | 162 | * Caveats on high order pages: PG_buddy and PG_slab will only be set |
| @@ -22,6 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <linux/pkeys.h> |
| 23 | 23 | #include <linux/minmax.h> |
| 24 | 24 | #include <linux/overflow.h> |
| 25 | +#include <linux/buildid.h> | |
| 25 | 26 | |
| 26 | 27 | #include <asm/elf.h> |
| 27 | 28 | #include <asm/tlb.h> |
| @@ -239,6 +240,67 @@ static int do_maps_open(struct inode *inode, struct file *file, · 211 unmodified lines | ||
| 239 | 240 | sizeof(struct proc_maps_private)); |
| 240 | 241 | } |
| 241 | 242 | |
| 243 | +static void get_vma_name(struct vm_area_struct *vma, | |
| 244 | + const struct path **path, | |
| 245 | + const char **name, | |
| 246 | + const char **name_fmt) | |
| 247 | +{ | |
| 248 | + struct anon_vma_name *anon_name = vma->vm_mm ? anon_vma_name(vma) : NULL; | |
| 249 | + | |
| 250 | + *name = NULL; | |
| 251 | + *path = NULL; | |
| 252 | + *name_fmt = NULL; | |
| 253 | + | |
| 254 | + /* | |
| 255 | + * Print the dentry name for named mappings, and a | |
| 256 | + * special [heap] marker for the heap: | |
| 257 | + */ | |
| 258 | + if (vma->vm_file) { | |
| 259 | + /* | |
| 260 | + * If user named this anon shared memory via | |
| 261 | + * prctl(PR_SET_VMA ..., use the provided name. | |
| 262 | + */ | |
| 263 | + if (anon_name) { | |
| 264 | + *name_fmt = "[anon_shmem:%s]"; | |
| 265 | + *name = anon_name->name; | |
| 266 | + } else { | |
| 267 | + *path = file_user_path(vma->vm_file); | |
| 268 | + } | |
| 269 | + return; | |
| 270 | + } | |
| 271 | + | |
| 272 | + if (vma->vm_ops && vma->vm_ops->name) { | |
| 273 | + *name = vma->vm_ops->name(vma); | |
| 274 | + if (*name) | |
| 275 | + return; | |
| 276 | + } | |
| 277 | + | |
| 278 | + *name = arch_vma_name(vma); | |
| 279 | + if (*name) | |
| 280 | + return; | |
| 281 | + | |
| 282 | + if (!vma->vm_mm) { | |
| 283 | + *name = "[vdso]"; | |
| 284 | + return; | |
| 285 | + } | |
| 286 | + | |
| 287 | + if (vma_is_initial_heap(vma)) { | |
| 288 | + *name = "[heap]"; | |
| 289 | + return; | |
| 290 | + } | |
| 291 | + | |
| 292 | + if (vma_is_initial_stack(vma)) { | |
| 293 | + *name = "[stack]"; | |
| 294 | + return; | |
| 295 | + } | |
| 296 | + | |
| 297 | + if (anon_name) { | |
| 298 | + *name_fmt = "[anon:%s]"; | |
| 299 | + *name = anon_name->name; | |
| 300 | + return; | |
| 301 | + } | |
| 302 | +} | |
| 303 | + | |
| 242 | 304 | static void show_vma_header_prefix(struct seq_file *m, |
| 243 | 305 | unsigned long start, unsigned long end, |
| 244 | 306 | vm_flags_t flags, unsigned long long pgoff, |
| @@ -262,17 +324,15 @@ static void show_vma_header_prefix(struct seq_file *m, · 17 unmodified lines | ||
| 262 | 324 | static void |
| 263 | 325 | show_map_vma(struct seq_file *m, struct vm_area_struct *vma) |
| 264 | 326 | { |
| 265 | − struct anon_vma_name *anon_name = NULL; | |
| 266 | − struct mm_struct *mm = vma->vm_mm; | |
| 267 | − struct file *file = vma->vm_file; | |
| 327 | + const struct path *path; | |
| 328 | + const char *name_fmt, *name; | |
| 268 | 329 | vm_flags_t flags = vma->vm_flags; |
| 269 | 330 | unsigned long ino = 0; |
| 270 | 331 | unsigned long long pgoff = 0; |
| 271 | 332 | unsigned long start, end; |
| 272 | 333 | dev_t dev = 0; |
| 273 | − const char *name = NULL; | |
| 274 | 334 | |
| 275 | − if (file) { | |
| 335 | + if (vma->vm_file) { | |
| 276 | 336 | const struct inode *inode = file_user_inode(vma->vm_file); |
| 277 | 337 | |
| 278 | 338 | dev = inode->i_sb->s_dev; |
| @@ -283,57 +343,15 @@ show_map_vma(struct seq_file *m, struct vm_area_struct *vma) · 4 unmodified lines | ||
| 283 | 343 | start = vma->vm_start; |
| 284 | 344 | end = vma->vm_end; |
| 285 | 345 | show_vma_header_prefix(m, start, end, flags, pgoff, dev, ino); |
| 286 | − if (mm) | |
| 287 | − anon_name = anon_vma_name(vma); | |
| 288 | 346 | |
| 289 | − /* | |
| 290 | − * Print the dentry name for named mappings, and a | |
| 291 | − * special [heap] marker for the heap: | |
| 292 | − */ | |
| 293 | − if (file) { | |
| 347 | + get_vma_name(vma, &path, &name, &name_fmt); | |
| 348 | + if (path) { | |
| 294 | 349 | seq_pad(m, ' '); |
| 295 | − /* | |
| 296 | − * If user named this anon shared memory via | |
| 297 | − * prctl(PR_SET_VMA ..., use the provided name. | |
| 298 | − */ | |
| 299 | − if (anon_name) | |
| 300 | − seq_printf(m, "[anon_shmem:%s]", anon_name->name); | |
| 301 | − else | |
| 302 | − seq_path(m, file_user_path(file), "\n"); | |
| 303 | − goto done; | |
| 304 | − } | |
| 305 | − | |
| 306 | − if (vma->vm_ops && vma->vm_ops->name) { | |
| 307 | − name = vma->vm_ops->name(vma); | |
| 308 | − if (name) | |
| 309 | − goto done; | |
| 310 | − } | |
| 311 | − | |
| 312 | − name = arch_vma_name(vma); | |
| 313 | − if (!name) { | |
| 314 | − if (!mm) { | |
| 315 | − name = "[vdso]"; | |
| 316 | − goto done; | |
| 317 | − } | |
| 318 | − | |
| 319 | − if (vma_is_initial_heap(vma)) { | |
| 320 | − name = "[heap]"; | |
| 321 | − goto done; | |
| 322 | − } | |
| 323 | − | |
| 324 | − if (vma_is_initial_stack(vma)) { | |
| 325 | − name = "[stack]"; | |
| 326 | − goto done; | |
| 327 | − } | |
| 328 | − | |
| 329 | − if (anon_name) { | |
| 330 | − seq_pad(m, ' '); | |
| 331 | − seq_printf(m, "[anon:%s]", anon_name->name); | |
| 332 | − } | |
| 333 | − } | |
| 334 | − | |
| 335 | −done: | |
| 336 | − if (name) { | |
| 350 | + seq_path(m, path, "\n"); | |
| 351 | + } else if (name_fmt) { | |
| 352 | + seq_pad(m, ' '); | |
| 353 | + seq_printf(m, name_fmt, name); | |
| 354 | + } else if (name) { | |
| 337 | 355 | seq_pad(m, ' '); |
| 338 | 356 | seq_puts(m, name); |
| 339 | 357 | } |
| @@ -358,11 +376,268 @@ static int pid_maps_open(struct inode *inode, struct file *file) · 18 unmodified lines | ||
| 358 | 376 | return do_maps_open(inode, file, &proc_pid_maps_op); |
| 359 | 377 | } |
| 360 | 378 | |
| 379 | +#define PROCMAP_QUERY_VMA_FLAGS ( \ | |
| 380 | + PROCMAP_QUERY_VMA_READABLE | \ | |
| 381 | + PROCMAP_QUERY_VMA_WRITABLE | \ | |
| 382 | + PROCMAP_QUERY_VMA_EXECUTABLE | \ | |
| 383 | + PROCMAP_QUERY_VMA_SHARED \ | |
| 384 | +) | |
| 385 | + | |
| 386 | +#define PROCMAP_QUERY_VALID_FLAGS_MASK ( \ | |
| 387 | + PROCMAP_QUERY_COVERING_OR_NEXT_VMA | \ | |
| 388 | + PROCMAP_QUERY_FILE_BACKED_VMA | \ | |
| 389 | + PROCMAP_QUERY_VMA_FLAGS \ | |
| 390 | +) | |
| 391 | + | |
| 392 | +static int query_vma_setup(struct mm_struct *mm) | |
| 393 | +{ | |
| 394 | + return mmap_read_lock_killable(mm); | |
| 395 | +} | |
| 396 | + | |
| 397 | +static void query_vma_teardown(struct mm_struct *mm, struct vm_area_struct *vma) | |
| 398 | +{ | |
| 399 | + mmap_read_unlock(mm); | |
| 400 | +} | |
| 401 | + | |
| 402 | +static struct vm_area_struct *query_vma_find_by_addr(struct mm_struct *mm, unsigned long addr) | |
| 403 | +{ | |
| 404 | + return find_vma(mm, addr); | |
| 405 | +} | |
| 406 | + | |
| 407 | +static struct vm_area_struct *query_matching_vma(struct mm_struct *mm, | |
| 408 | + unsigned long addr, u32 flags) | |
| 409 | +{ | |
| 410 | + struct vm_area_struct *vma; | |
| 411 | + | |
| 412 | +next_vma: | |
| 413 | + vma = query_vma_find_by_addr(mm, addr); | |
| 414 | + if (!vma) | |
| 415 | + goto no_vma; | |
| 416 | + | |
| 417 | + /* user requested only file-backed VMA, keep iterating */ | |
| 418 | + if ((flags & PROCMAP_QUERY_FILE_BACKED_VMA) && !vma->vm_file) | |
| 419 | + goto skip_vma; | |
| 420 | + | |
| 421 | + /* VMA permissions should satisfy query flags */ | |
| 422 | + if (flags & PROCMAP_QUERY_VMA_FLAGS) { | |
| 423 | + u32 perm = 0; | |
| 424 | + | |
| 425 | + if (flags & PROCMAP_QUERY_VMA_READABLE) | |
| 426 | + perm |= VM_READ; | |
| 427 | + if (flags & PROCMAP_QUERY_VMA_WRITABLE) | |
| 428 | + perm |= VM_WRITE; | |
| 429 | + if (flags & PROCMAP_QUERY_VMA_EXECUTABLE) | |
| 430 | + perm |= VM_EXEC; | |
| 431 | + if (flags & PROCMAP_QUERY_VMA_SHARED) | |
| 432 | + perm |= VM_MAYSHARE; | |
| 433 | + | |
| 434 | + if ((vma->vm_flags & perm) != perm) | |
| 435 | + goto skip_vma; | |
| 436 | + } | |
| 437 | + | |
| 438 | + /* found covering VMA or user is OK with the matching next VMA */ | |
| 439 | + if ((flags & PROCMAP_QUERY_COVERING_OR_NEXT_VMA) || vma->vm_start <= addr) | |
| 440 | + return vma; | |
| 441 | + | |
| 442 | +skip_vma: | |
| 443 | + /* | |
| 444 | + * If the user needs closest matching VMA, keep iterating. | |
| 445 | + */ | |
| 446 | + addr = vma->vm_end; | |
| 447 | + if (flags & PROCMAP_QUERY_COVERING_OR_NEXT_VMA) | |
| 448 | + goto next_vma; | |
| 449 | + | |
| 450 | +no_vma: | |
| 451 | + return ERR_PTR(-ENOENT); | |
| 452 | +} | |
| 453 | + | |
| 454 | +static int do_procmap_query(struct proc_maps_private *priv, void __user *uarg) | |
| 455 | +{ | |
| 456 | + struct procmap_query karg; | |
| 457 | + struct vm_area_struct *vma; | |
| 458 | + struct mm_struct *mm; | |
| 459 | + const char *name = NULL; | |
| 460 | + char build_id_buf[BUILD_ID_SIZE_MAX], *name_buf = NULL; | |
| 461 | + __u64 usize; | |
| 462 | + int err; | |
| 463 | + | |
| 464 | + if (copy_from_user(&usize, (void __user *)uarg, sizeof(usize))) | |
| 465 | + return -EFAULT; | |
| 466 | + /* argument struct can never be that large, reject abuse */ | |
| 467 | + if (usize > PAGE_SIZE) | |
| 468 | + return -E2BIG; | |
| 469 | + /* argument struct should have at least query_flags and query_addr fields */ | |
| 470 | + if (usize < offsetofend(struct procmap_query, query_addr)) | |
| 471 | + return -EINVAL; | |
| 472 | + err = copy_struct_from_user(&karg, sizeof(karg), uarg, usize); | |
| 473 | + if (err) | |
| 474 | + return err; | |
| 475 | + | |
| 476 | + /* reject unknown flags */ | |
| 477 | + if (karg.query_flags & ~PROCMAP_QUERY_VALID_FLAGS_MASK) | |
| 478 | + return -EINVAL; | |
| 479 | + /* either both buffer address and size are set, or both should be zero */ | |
| 480 | + if (!!karg.vma_name_size != !!karg.vma_name_addr) | |
| 481 | + return -EINVAL; | |
| 482 | + if (!!karg.build_id_size != !!karg.build_id_addr) | |
| 483 | + return -EINVAL; | |
| 484 | + | |
| 485 | + mm = priv->mm; | |
| 486 | + if (!mm || !mmget_not_zero(mm)) | |
| 487 | + return -ESRCH; | |
| 488 | + | |
| 489 | + err = query_vma_setup(mm); | |
| 490 | + if (err) { | |
| 491 | + mmput(mm); | |
| 492 | + return err; | |
| 493 | + } | |
| 494 | + | |
| 495 | + vma = query_matching_vma(mm, karg.query_addr, karg.query_flags); | |
| 496 | + if (IS_ERR(vma)) { | |
| 497 | + err = PTR_ERR(vma); | |
| 498 | + vma = NULL; | |
| 499 | + goto out; | |
| 500 | + } | |
| 501 | + | |
| 502 | + karg.vma_start = vma->vm_start; | |
| 503 | + karg.vma_end = vma->vm_end; | |
| 504 | + | |
| 505 | + karg.vma_flags = 0; | |
| 506 | + if (vma->vm_flags & VM_READ) | |
| 507 | + karg.vma_flags |= PROCMAP_QUERY_VMA_READABLE; | |
| 508 | + if (vma->vm_flags & VM_WRITE) | |
| 509 | + karg.vma_flags |= PROCMAP_QUERY_VMA_WRITABLE; | |
| 510 | + if (vma->vm_flags & VM_EXEC) | |
| 511 | + karg.vma_flags |= PROCMAP_QUERY_VMA_EXECUTABLE; | |
| 512 | + if (vma->vm_flags & VM_MAYSHARE) | |
| 513 | + karg.vma_flags |= PROCMAP_QUERY_VMA_SHARED; | |
| 514 | + | |
| 515 | + karg.vma_page_size = vma_kernel_pagesize(vma); | |
| 516 | + | |
| 517 | + if (vma->vm_file) { | |
| 518 | + const struct inode *inode = file_user_inode(vma->vm_file); | |
| 519 | + | |
| 520 | + karg.vma_offset = ((__u64)vma->vm_pgoff) << PAGE_SHIFT; | |
| 521 | + karg.dev_major = MAJOR(inode->i_sb->s_dev); | |
| 522 | + karg.dev_minor = MINOR(inode->i_sb->s_dev); | |
| 523 | + karg.inode = inode->i_ino; | |
| 524 | + } else { | |
| 525 | + karg.vma_offset = 0; | |
| 526 | + karg.dev_major = 0; | |
| 527 | + karg.dev_minor = 0; | |
| 528 | + karg.inode = 0; | |
| 529 | + } | |
| 530 | + | |
| 531 | + if (karg.build_id_size) { | |
| 532 | + __u32 build_id_sz; | |
| 533 | + | |
| 534 | + err = build_id_parse(vma, build_id_buf, &build_id_sz); | |
| 535 | + if (err) { | |
| 536 | + karg.build_id_size = 0; | |
| 537 | + } else { | |
| 538 | + if (karg.build_id_size < build_id_sz) { | |
| 539 | + err = -ENAMETOOLONG; | |
| 540 | + goto out; | |
| 541 | + } | |
| 542 | + karg.build_id_size = build_id_sz; | |
| 543 | + } | |
| 544 | + } | |
| 545 | + | |
| 546 | + if (karg.build_id_size) { | |
| 547 | + __u32 build_id_sz; | |
| 548 | + | |
| 549 | + err = build_id_parse(vma, build_id_buf, &build_id_sz); | |
| 550 | + if (err) { | |
| 551 | + karg.build_id_size = 0; | |
| 552 | + } else { | |
| 553 | + if (karg.build_id_size < build_id_sz) { | |
| 554 | + err = -ENAMETOOLONG; | |
| 555 | + goto out; | |
| 556 | + } | |
| 557 | + karg.build_id_size = build_id_sz; | |
| 558 | + } | |
| 559 | + } | |
| 560 | + | |
| 561 | + if (karg.vma_name_size) { | |
| 562 | + size_t name_buf_sz = min_t(size_t, PATH_MAX, karg.vma_name_size); | |
| 563 | + const struct path *path; | |
| 564 | + const char *name_fmt; | |
| 565 | + size_t name_sz = 0; | |
| 566 | + | |
| 567 | + get_vma_name(vma, &path, &name, &name_fmt); | |
| 568 | + | |
| 569 | + if (path || name_fmt || name) { | |
| 570 | + name_buf = kmalloc(name_buf_sz, GFP_KERNEL); | |
| 571 | + if (!name_buf) { | |
| 572 | + err = -ENOMEM; | |
| 573 | + goto out; | |
| 574 | + } | |
| 575 | + } | |
| 576 | + if (path) { | |
| 577 | + name = d_path(path, name_buf, name_buf_sz); | |
| 578 | + if (IS_ERR(name)) { | |
| 579 | + err = PTR_ERR(name); | |
| 580 | + goto out; | |
| 581 | + } | |
| 582 | + name_sz = name_buf + name_buf_sz - name; | |
| 583 | + } else if (name || name_fmt) { | |
| 584 | + name_sz = 1 + snprintf(name_buf, name_buf_sz, name_fmt ?: "%s", name); | |
| 585 | + name = name_buf; | |
| 586 | + } | |
| 587 | + if (name_sz > name_buf_sz) { | |
| 588 | + err = -ENAMETOOLONG; | |
| 589 | + goto out; | |
| 590 | + } | |
| 591 | + karg.vma_name_size = name_sz; | |
| 592 | + } | |
| 593 | + | |
| 594 | + /* unlock vma or mmap_lock, and put mm_struct before copying data to user */ | |
| 595 | + query_vma_teardown(mm, vma); | |
| 596 | + mmput(mm); | |
| 597 | + | |
| 598 | + if (karg.vma_name_size && copy_to_user(u64_to_user_ptr(karg.vma_name_addr), | |
| 599 | + name, karg.vma_name_size)) { | |
| 600 | + kfree(name_buf); | |
| 601 | + return -EFAULT; | |
| 602 | + } | |
| 603 | + kfree(name_buf); | |
| 604 | + | |
| 605 | + if (karg.build_id_size && copy_to_user(u64_to_user_ptr(karg.build_id_addr), | |
| 606 | + build_id_buf, karg.build_id_size)) | |
| 607 | + return -EFAULT; | |
| 608 | + | |
| 609 | + if (copy_to_user(uarg, &karg, min_t(size_t, sizeof(karg), usize))) | |
| 610 | + return -EFAULT; | |
| 611 | + | |
| 612 | + return 0; | |
| 613 | + | |
| 614 | +out: | |
| 615 | + query_vma_teardown(mm, vma); | |
| 616 | + mmput(mm); | |
| 617 | + kfree(name_buf); | |
| 618 | + return err; | |
| 619 | +} | |
| 620 | + | |
| 621 | +static long procfs_procmap_ioctl(struct file *file, unsigned int cmd, unsigned long arg) | |
| 622 | +{ | |
| 623 | + struct seq_file *seq = file->private_data; | |
| 624 | + struct proc_maps_private *priv = seq->private; | |
| 625 | + | |
| 626 | + switch (cmd) { | |
| 627 | + case PROCMAP_QUERY: | |
| 628 | + return do_procmap_query(priv, (void __user *)arg); | |
| 629 | + default: | |
| 630 | + return -ENOIOCTLCMD; | |
| 631 | + } | |
| 632 | +} | |
| 633 | + | |
| 361 | 634 | const struct file_operations proc_pid_maps_operations = { |
| 362 | 635 | .open = pid_maps_open, |
| 363 | 636 | .read = seq_read, |
| 364 | 637 | .llseek = seq_lseek, |
| 365 | 638 | .release = proc_map_release, |
| 639 | + .unlocked_ioctl = procfs_procmap_ioctl, | |
| 640 | + .compat_ioctl = compat_ptr_ioctl, | |
| 366 | 641 | }; |
| 367 | 642 | |
| 368 | 643 | /* |
| @@ -442,7 +717,7 @@ static void smaps_page_accumulate(struct mem_size_stats *mss, · 73 unmodified lines | ||
| 442 | 717 | |
| 443 | 718 | static void smaps_account(struct mem_size_stats *mss, struct page *page, |
| 444 | 719 | bool compound, bool young, bool dirty, bool locked, |
| 445 | − bool migration) | |
| 720 | + bool present) | |
| 446 | 721 | { |
| 447 | 722 | struct folio *folio = page_folio(page); |
| 448 | 723 | int i, nr = compound ? compound_nr(page) : 1; |
| @@ -471,24 +746,29 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page, · 22 unmodified lines | ||
| 471 | 746 | * Then accumulate quantities that may depend on sharing, or that may |
| 472 | 747 | * differ page-by-page. |
| 473 | 748 | * |
| 474 | − * refcount == 1 guarantees the page is mapped exactly once. | |
| 475 | − * If any subpage of the compound page mapped with PTE it would elevate | |
| 476 | − * the refcount. | |
| 749 | + * refcount == 1 for present entries guarantees that the folio is mapped | |
| 750 | + * exactly once. For large folios this implies that exactly one | |
| 751 | + * PTE/PMD/... maps (a part of) this folio. | |
| 477 | 752 | * |
| 478 | − * The page_mapcount() is called to get a snapshot of the mapcount. | |
| 479 | − * Without holding the page lock this snapshot can be slightly wrong as | |
| 480 | − * we cannot always read the mapcount atomically. It is not safe to | |
| 481 | − * call page_mapcount() even with PTL held if the page is not mapped, | |
| 482 | − * especially for migration entries. Treat regular migration entries | |
| 483 | − * as mapcount == 1. | |
| 753 | + * Treat all non-present entries (where relying on the mapcount and | |
| 754 | + * refcount doesn't make sense) as "maybe shared, but not sure how | |
| 755 | + * often". We treat device private entries as being fake-present. | |
| 756 | + * | |
| 757 | + * Note that it would not be safe to read the mapcount especially for | |
| 758 | + * pages referenced by migration entries, even with the PTL held. | |
| 484 | 759 | */ |
| 485 | − if ((folio_ref_count(folio) == 1) || migration) { | |
| 760 | + if (folio_ref_count(folio) == 1 || !present) { | |
| 486 | 761 | smaps_page_accumulate(mss, folio, size, size << PSS_SHIFT, |
| 487 | − dirty, locked, true); | |
| 762 | + dirty, locked, present); | |
| 488 | 763 | return; |
| 489 | 764 | } |
| 765 | + /* | |
| 766 | + * We obtain a snapshot of the mapcount. Without holding the folio lock | |
| 767 | + * this snapshot can be slightly wrong as we cannot always read the | |
| 768 | + * mapcount atomically. | |
| 769 | + */ | |
| 490 | 770 | for (i = 0; i < nr; i++, page++) { |
| 491 | − int mapcount = page_mapcount(page); | |
| 771 | + int mapcount = folio_precise_page_mapcount(folio, page); | |
| 492 | 772 | unsigned long pss = PAGE_SIZE << PSS_SHIFT; |
| 493 | 773 | if (mapcount >= 2) |
| 494 | 774 | pss /= mapcount; |
| @@ -531,13 +811,14 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, · 36 unmodified lines | ||
| 531 | 811 | struct vm_area_struct *vma = walk->vma; |
| 532 | 812 | bool locked = !!(vma->vm_flags & VM_LOCKED); |
| 533 | 813 | struct page *page = NULL; |
| 534 | − bool migration = false, young = false, dirty = false; | |
| 814 | + bool present = false, young = false, dirty = false; | |
| 535 | 815 | pte_t ptent = ptep_get(pte); |
| 536 | 816 | |
| 537 | 817 | if (pte_present(ptent)) { |
| 538 | 818 | page = vm_normal_page(vma, addr, ptent); |
| 539 | 819 | young = pte_young(ptent); |
| 540 | 820 | dirty = pte_dirty(ptent); |
| 821 | + present = true; | |
| 541 | 822 | } else if (is_swap_pte(ptent)) { |
| 542 | 823 | swp_entry_t swpent = pte_to_swp_entry(ptent); |
| 543 | 824 | |
| @@ -555,8 +836,8 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, · 11 unmodified lines | ||
| 555 | 836 | mss->swap_pss += (u64)PAGE_SIZE << PSS_SHIFT; |
| 556 | 837 | } |
| 557 | 838 | } else if (is_pfn_swap_entry(swpent)) { |
| 558 | − if (is_migration_entry(swpent)) | |
| 559 | − migration = true; | |
| 839 | + if (is_device_private_entry(swpent)) | |
| 840 | + present = true; | |
| 560 | 841 | page = pfn_swap_entry_to_page(swpent); |
| 561 | 842 | } |
| 562 | 843 | } else { |
| @@ -567,7 +848,7 @@ static void smaps_pte_entry(pte_t *pte, unsigned long addr, · 4 unmodified lines | ||
| 567 | 848 | if (!page) |
| 568 | 849 | return; |
| 569 | 850 | |
| 570 | − smaps_account(mss, page, false, young, dirty, locked, migration); | |
| 851 | + smaps_account(mss, page, false, young, dirty, locked, present); | |
| 571 | 852 | } |
| 572 | 853 | |
| 573 | 854 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| @@ -578,18 +859,17 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, · 4 unmodified lines | ||
| 578 | 859 | struct vm_area_struct *vma = walk->vma; |
| 579 | 860 | bool locked = !!(vma->vm_flags & VM_LOCKED); |
| 580 | 861 | struct page *page = NULL; |
| 862 | + bool present = false; | |
| 581 | 863 | struct folio *folio; |
| 582 | − bool migration = false; | |
| 583 | 864 | |
| 584 | 865 | if (pmd_present(*pmd)) { |
| 585 | 866 | page = vm_normal_page_pmd(vma, addr, *pmd); |
| 867 | + present = true; | |
| 586 | 868 | } else if (unlikely(thp_migration_supported() && is_swap_pmd(*pmd))) { |
| 587 | 869 | swp_entry_t entry = pmd_to_swp_entry(*pmd); |
| 588 | 870 | |
| 589 | − if (is_migration_entry(entry)) { | |
| 590 | − migration = true; | |
| 871 | + if (is_pfn_swap_entry(entry)) | |
| 591 | 872 | page = pfn_swap_entry_to_page(entry); |
| 592 | − } | |
| 593 | 873 | } |
| 594 | 874 | if (IS_ERR_OR_NULL(page)) |
| 595 | 875 | return; |
| @@ -604,7 +884,7 @@ static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, · 8 unmodified lines | ||
| 604 | 884 | mss->file_thp += HPAGE_PMD_SIZE; |
| 605 | 885 | |
| 606 | 886 | smaps_account(mss, page, true, pmd_young(*pmd), pmd_dirty(*pmd), |
| 607 | − locked, migration); | |
| 887 | + locked, present); | |
| 608 | 888 | } |
| 609 | 889 | #else |
| 610 | 890 | static void smaps_pmd_entry(pmd_t *pmd, unsigned long addr, |
| @@ -733,19 +1013,23 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, · 122 unmodified lines | ||
| 733 | 1013 | { |
| 734 | 1014 | struct mem_size_stats *mss = walk->private; |
| 735 | 1015 | struct vm_area_struct *vma = walk->vma; |
| 736 | − pte_t ptent = huge_ptep_get(pte); | |
| 1016 | + pte_t ptent = huge_ptep_get(walk->mm, addr, pte); | |
| 737 | 1017 | struct folio *folio = NULL; |
| 1018 | + bool present = false; | |
| 738 | 1019 | |
| 739 | 1020 | if (pte_present(ptent)) { |
| 740 | 1021 | folio = page_folio(pte_page(ptent)); |
| 1022 | + present = true; | |
| 741 | 1023 | } else if (is_swap_pte(ptent)) { |
| 742 | 1024 | swp_entry_t swpent = pte_to_swp_entry(ptent); |
| 743 | 1025 | |
| 744 | 1026 | if (is_pfn_swap_entry(swpent)) |
| 745 | 1027 | folio = pfn_swap_entry_folio(swpent); |
| 746 | 1028 | } |
| 1029 | + | |
| 747 | 1030 | if (folio) { |
| 748 | − if (folio_likely_mapped_shared(folio) || | |
| 1031 | + /* We treat non-present entries as "maybe shared". */ | |
| 1032 | + if (!present || folio_likely_mapped_shared(folio) || | |
| 749 | 1033 | hugetlb_pmd_shared(pte)) |
| 750 | 1034 | mss->shared_hugetlb += huge_page_size(hstate_vma(vma)); |
| 751 | 1035 | else |
| @@ -1091,7 +1375,7 @@ struct clear_refs_private { · 339 unmodified lines | ||
| 1091 | 1375 | |
| 1092 | 1376 | static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, pte_t pte) |
| 1093 | 1377 | { |
| 1094 | − struct page *page; | |
| 1378 | + struct folio *folio; | |
| 1095 | 1379 | |
| 1096 | 1380 | if (!pte_write(pte)) |
| 1097 | 1381 | return false; |
| @@ -1099,10 +1383,10 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, · 1 unmodified line | ||
| 1099 | 1383 | return false; |
| 1100 | 1384 | if (likely(!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags))) |
| 1101 | 1385 | return false; |
| 1102 | − page = vm_normal_page(vma, addr, pte); | |
| 1103 | − if (!page) | |
| 1386 | + folio = vm_normal_folio(vma, addr, pte); | |
| 1387 | + if (!folio) | |
| 1104 | 1388 | return false; |
| 1105 | − return page_maybe_dma_pinned(page); | |
| 1389 | + return folio_maybe_dma_pinned(folio); | |
| 1106 | 1390 | } |
| 1107 | 1391 | |
| 1108 | 1392 | static inline void clear_soft_dirty(struct vm_area_struct *vma, |
| @@ -1418,7 +1702,7 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, · 309 unmodified lines | ||
| 1418 | 1702 | { |
| 1419 | 1703 | u64 frame = 0, flags = 0; |
| 1420 | 1704 | struct page *page = NULL; |
| 1421 | − bool migration = false; | |
| 1705 | + struct folio *folio; | |
| 1422 | 1706 | |
| 1423 | 1707 | if (pte_present(pte)) { |
| 1424 | 1708 | if (pm->show_pfn) |
| @@ -1450,17 +1734,20 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, · 25 unmodified lines | ||
| 1450 | 1734 | (offset << MAX_SWAPFILES_SHIFT); |
| 1451 | 1735 | } |
| 1452 | 1736 | flags |= PM_SWAP; |
| 1453 | − migration = is_migration_entry(entry); | |
| 1454 | 1737 | if (is_pfn_swap_entry(entry)) |
| 1455 | 1738 | page = pfn_swap_entry_to_page(entry); |
| 1456 | 1739 | if (pte_marker_entry_uffd_wp(entry)) |
| 1457 | 1740 | flags |= PM_UFFD_WP; |
| 1458 | 1741 | } |
| 1459 | 1742 | |
| 1460 | − if (page && !PageAnon(page)) | |
| 1461 | − flags |= PM_FILE; | |
| 1462 | − if (page && !migration && page_mapcount(page) == 1) | |
| 1463 | − flags |= PM_MMAP_EXCLUSIVE; | |
| 1743 | + if (page) { | |
| 1744 | + folio = page_folio(page); | |
| 1745 | + if (!folio_test_anon(folio)) | |
| 1746 | + flags |= PM_FILE; | |
| 1747 | + if ((flags & PM_PRESENT) && | |
| 1748 | + folio_precise_page_mapcount(folio, page) == 1) | |
| 1749 | + flags |= PM_MMAP_EXCLUSIVE; | |
| 1750 | + } | |
| 1464 | 1751 | if (vma->vm_flags & VM_SOFTDIRTY) |
| 1465 | 1752 | flags |= PM_SOFT_DIRTY; |
| 1466 | 1753 | |
| @@ -1476,13 +1763,14 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 9 unmodified lines | ||
| 1476 | 1763 | pte_t *pte, *orig_pte; |
| 1477 | 1764 | int err = 0; |
| 1478 | 1765 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| 1479 | − bool migration = false; | |
| 1480 | 1766 | |
| 1481 | 1767 | ptl = pmd_trans_huge_lock(pmdp, vma); |
| 1482 | 1768 | if (ptl) { |
| 1769 | + unsigned int idx = (addr & ~PMD_MASK) >> PAGE_SHIFT; | |
| 1483 | 1770 | u64 flags = 0, frame = 0; |
| 1484 | 1771 | pmd_t pmd = *pmdp; |
| 1485 | 1772 | struct page *page = NULL; |
| 1773 | + struct folio *folio = NULL; | |
| 1486 | 1774 | |
| 1487 | 1775 | if (vma->vm_flags & VM_SOFTDIRTY) |
| 1488 | 1776 | flags |= PM_SOFT_DIRTY; |
| @@ -1496,8 +1784,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 7 unmodified lines | ||
| 1496 | 1784 | if (pmd_uffd_wp(pmd)) |
| 1497 | 1785 | flags |= PM_UFFD_WP; |
| 1498 | 1786 | if (pm->show_pfn) |
| 1499 | − frame = pmd_pfn(pmd) + | |
| 1500 | − ((addr & ~PMD_MASK) >> PAGE_SHIFT); | |
| 1787 | + frame = pmd_pfn(pmd) + idx; | |
| 1501 | 1788 | } |
| 1502 | 1789 | #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION |
| 1503 | 1790 | else if (is_swap_pmd(pmd)) { |
| @@ -1506,11 +1793,9 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 2 unmodified lines | ||
| 1506 | 1793 | |
| 1507 | 1794 | if (pm->show_pfn) { |
| 1508 | 1795 | if (is_pfn_swap_entry(entry)) |
| 1509 | − offset = swp_offset_pfn(entry); | |
| 1796 | + offset = swp_offset_pfn(entry) + idx; | |
| 1510 | 1797 | else |
| 1511 | − offset = swp_offset(entry); | |
| 1512 | − offset = offset + | |
| 1513 | − ((addr & ~PMD_MASK) >> PAGE_SHIFT); | |
| 1798 | + offset = swp_offset(entry) + idx; | |
| 1514 | 1799 | frame = swp_type(entry) | |
| 1515 | 1800 | (offset << MAX_SWAPFILES_SHIFT); |
| 1516 | 1801 | } |
| @@ -1520,17 +1805,25 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 3 unmodified lines | ||
| 1520 | 1805 | if (pmd_swp_uffd_wp(pmd)) |
| 1521 | 1806 | flags |= PM_UFFD_WP; |
| 1522 | 1807 | VM_BUG_ON(!is_pmd_migration_entry(pmd)); |
| 1523 | − migration = is_migration_entry(entry); | |
| 1524 | 1808 | page = pfn_swap_entry_to_page(entry); |
| 1525 | 1809 | } |
| 1526 | 1810 | #endif |
| 1527 | 1811 | |
| 1528 | − if (page && !migration && page_mapcount(page) == 1) | |
| 1529 | − flags |= PM_MMAP_EXCLUSIVE; | |
| 1812 | + if (page) { | |
| 1813 | + folio = page_folio(page); | |
| 1814 | + if (!folio_test_anon(folio)) | |
| 1815 | + flags |= PM_FILE; | |
| 1816 | + } | |
| 1817 | + | |
| 1818 | + for (; addr != end; addr += PAGE_SIZE, idx++) { | |
| 1819 | + unsigned long cur_flags = flags; | |
| 1820 | + pagemap_entry_t pme; | |
| 1530 | 1821 | |
| 1531 | − for (; addr != end; addr += PAGE_SIZE) { | |
| 1532 | − pagemap_entry_t pme = make_pme(frame, flags); | |
| 1822 | + if (folio && (flags & PM_PRESENT) && | |
| 1823 | + folio_precise_page_mapcount(folio, page + idx) == 1) | |
| 1824 | + cur_flags |= PM_MMAP_EXCLUSIVE; | |
| 1533 | 1825 | |
| 1826 | + pme = make_pme(frame, cur_flags); | |
| 1534 | 1827 | err = add_to_pagemap(&pme, pm); |
| 1535 | 1828 | if (err) |
| 1536 | 1829 | break; |
| @@ -1585,7 +1878,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, · 48 unmodified lines | ||
| 1585 | 1878 | if (vma->vm_flags & VM_SOFTDIRTY) |
| 1586 | 1879 | flags |= PM_SOFT_DIRTY; |
| 1587 | 1880 | |
| 1588 | − pte = huge_ptep_get(ptep); | |
| 1881 | + pte = huge_ptep_get(walk->mm, addr, ptep); | |
| 1589 | 1882 | if (pte_present(pte)) { |
| 1590 | 1883 | struct folio *folio = page_folio(pte_page(pte)); |
| 1591 | 1884 | |
| @@ -2274,7 +2567,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, · 682 unmodified lines | ||
| 2274 | 2567 | if (~p->arg.flags & PM_SCAN_WP_MATCHING) { |
| 2275 | 2568 | /* Go the short route when not write-protecting pages. */ |
| 2276 | 2569 | |
| 2277 | − pte = huge_ptep_get(ptep); | |
| 2570 | + pte = huge_ptep_get(walk->mm, start, ptep); | |
| 2278 | 2571 | categories = p->cur_vma_category | pagemap_hugetlb_category(pte); |
| 2279 | 2572 | |
| 2280 | 2573 | if (!pagemap_scan_is_interesting_page(categories, p)) |
| @@ -2286,7 +2579,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, · 5 unmodified lines | ||
| 2286 | 2579 | i_mmap_lock_write(vma->vm_file->f_mapping); |
| 2287 | 2580 | ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep); |
| 2288 | 2581 | |
| 2289 | − pte = huge_ptep_get(ptep); | |
| 2582 | + pte = huge_ptep_get(walk->mm, start, ptep); | |
| 2290 | 2583 | categories = p->cur_vma_category | pagemap_hugetlb_category(pte); |
| 2291 | 2584 | |
| 2292 | 2585 | if (!pagemap_scan_is_interesting_page(categories, p)) |
| @@ -2566,7 +2859,7 @@ static void gather_stats(struct page *page, struct numa_maps *md, int pte_dirty, · 273 unmodified lines | ||
| 2566 | 2859 | unsigned long nr_pages) |
| 2567 | 2860 | { |
| 2568 | 2861 | struct folio *folio = page_folio(page); |
| 2569 | − int count = page_mapcount(page); | |
| 2862 | + int count = folio_precise_page_mapcount(folio, page); | |
| 2570 | 2863 | |
| 2571 | 2864 | md->pages += nr_pages; |
| 2572 | 2865 | if (pte_dirty || folio_test_dirty(folio)) |
| @@ -2682,7 +2975,7 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, · 109 unmodified lines | ||
| 2682 | 2975 | static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, |
| 2683 | 2976 | unsigned long addr, unsigned long end, struct mm_walk *walk) |
| 2684 | 2977 | { |
| 2685 | − pte_t huge_pte = huge_ptep_get(pte); | |
| 2978 | + pte_t huge_pte = huge_ptep_get(walk->mm, addr, pte); | |
| 2686 | 2979 | struct numa_maps *md; |
| 2687 | 2980 | struct page *page; |
| 2688 | 2981 | |
| @@ -257,7 +257,7 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, · 256 unmodified lines | ||
| 257 | 257 | goto out; |
| 258 | 258 | |
| 259 | 259 | ret = false; |
| 260 | − pte = huge_ptep_get(ptep); | |
| 260 | + pte = huge_ptep_get(vma->vm_mm, vmf->address, ptep); | |
| 261 | 261 | |
| 262 | 262 | /* |
| 263 | 263 | * Lockless access: we're in a wait_event so it's ok if it |
| @@ -46,6 +46,9 @@ acpi_status acpi_os_terminate(void); · 45 unmodified lines | ||
| 46 | 46 | * Interrupts are off during resume, just like they are for boot. |
| 47 | 47 | * However, boot has (system_state != SYSTEM_RUNNING) |
| 48 | 48 | * to quiet __might_sleep() in kmalloc() and resume does not. |
| 49 | + * | |
| 50 | + * These specialized allocators have to be macros for their allocations to be | |
| 51 | + * accounted separately (to have separate alloc_tag). | |
| 49 | 52 | */ |
| 50 | 53 | #define acpi_os_allocate(_size) \ |
| 51 | 54 | kmalloc(_size, irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL) |
| @@ -53,14 +56,14 @@ acpi_status acpi_os_terminate(void); · 1 unmodified line | ||
| 53 | 56 | #define acpi_os_allocate_zeroed(_size) \ |
| 54 | 57 | kzalloc(_size, irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL) |
| 55 | 58 | |
| 59 | +#define acpi_os_acquire_object(_cache) \ | |
| 60 | + kmem_cache_zalloc(_cache, irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL) | |
| 61 | + | |
| 56 | 62 | static inline void acpi_os_free(void *memory) |
| 57 | 63 | { |
| 58 | 64 | kfree(memory); |
| 59 | 65 | } |
| 60 | 66 | |
| 61 | −#define acpi_os_acquire_object(_cache) \ | |
| 62 | − kmem_cache_zalloc(_cache, irqs_disabled() ? GFP_ATOMIC : GFP_KERNEL) | |
| 63 | − | |
| 64 | 67 | static inline acpi_thread_id acpi_os_get_thread_id(void) |
| 65 | 68 | { |
| 66 | 69 | return (acpi_thread_id) (unsigned long)current; |
| @@ -144,7 +144,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, · 143 unmodified lines | ||
| 144 | 144 | #endif |
| 145 | 145 | |
| 146 | 146 | #ifndef __HAVE_ARCH_HUGE_PTEP_GET |
| 147 | −static inline pte_t huge_ptep_get(pte_t *ptep) | |
| 147 | +static inline pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) | |
| 148 | 148 | { |
| 149 | 149 | return ptep_get(ptep); |
| 150 | 150 | } |
| @@ -122,7 +122,7 @@ static inline void alloc_tag_add_check(union codetag_ref *ref, struct alloc_tag · 121 unmodified lines | ||
| 122 | 122 | "alloc_tag was not cleared (got tag for %s:%u)\n", |
| 123 | 123 | ref->ct->filename, ref->ct->lineno); |
| 124 | 124 | |
| 125 | − WARN_ONCE(!tag, "current->alloc_tag not set"); | |
| 125 | + WARN_ONCE(!tag, "current->alloc_tag not set\n"); | |
| 126 | 126 | } |
| 127 | 127 | |
| 128 | 128 | static inline void alloc_tag_sub_check(union codetag_ref *ref) |
| @@ -275,7 +275,7 @@ struct bpf_map { · 274 unmodified lines | ||
| 275 | 275 | u32 btf_value_type_id; |
| 276 | 276 | u32 btf_vmlinux_value_type_id; |
| 277 | 277 | struct btf *btf; |
| 278 | −#ifdef CONFIG_MEMCG_KMEM | |
| 278 | +#ifdef CONFIG_MEMCG | |
| 279 | 279 | struct obj_cgroup *objcg; |
| 280 | 280 | #endif |
| 281 | 281 | char name[BPF_OBJ_NAME_LEN]; |
| @@ -2253,7 +2253,7 @@ struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); · 1971 unmodified lines | ||
| 2253 | 2253 | |
| 2254 | 2254 | int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, |
| 2255 | 2255 | unsigned long nr_pages, struct page **page_array); |
| 2256 | −#ifdef CONFIG_MEMCG_KMEM | |
| 2256 | +#ifdef CONFIG_MEMCG | |
| 2257 | 2257 | void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, |
| 2258 | 2258 | int node); |
| 2259 | 2259 | void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags); |
| @@ -2262,6 +2262,10 @@ void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, · 2 unmodified lines | ||
| 2262 | 2262 | void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, |
| 2263 | 2263 | size_t align, gfp_t flags); |
| 2264 | 2264 | #else |
| 2265 | +/* | |
| 2266 | + * These specialized allocators have to be macros for their allocations to be | |
| 2267 | + * accounted separately (to have separate alloc_tag). | |
| 2268 | + */ | |
| 2265 | 2269 | #define bpf_map_kmalloc_node(_map, _size, _flags, _node) \ |
| 2266 | 2270 | kmalloc_node(_size, _flags, _node) |
| 2267 | 2271 | #define bpf_map_kzalloc(_map, _size, _flags) \ |
| @@ -53,7 +53,7 @@ typedef void (bh_end_io_t)(struct buffer_head *bh, int uptodate); · 52 unmodified lines | ||
| 53 | 53 | * filesystem and block layers. Nowadays the basic I/O unit |
| 54 | 54 | * is the bio, and buffer_heads are used for extracting block |
| 55 | 55 | * mappings (via a get_block_t call), for tracking state within |
| 56 | − * a page (via a page_mapping) and for wrapping bio submission | |
| 56 | + * a folio (via a folio_mapping) and for wrapping bio submission | |
| 57 | 57 | * for backward compatibility reasons (e.g. submit_bh). |
| 58 | 58 | */ |
| 59 | 59 | struct buffer_head { |
| @@ -681,9 +681,7 @@ struct cftype { · 680 unmodified lines | ||
| 681 | 681 | __poll_t (*poll)(struct kernfs_open_file *of, |
| 682 | 682 | struct poll_table_struct *pt); |
| 683 | 683 | |
| 684 | −#ifdef CONFIG_DEBUG_LOCK_ALLOC | |
| 685 | 684 | struct lock_class_key lockdep_key; |
| 686 | −#endif | |
| 687 | 685 | }; |
| 688 | 686 | |
| 689 | 687 | /* |
| @@ -105,6 +105,8 @@ struct damon_target { · 104 unmodified lines | ||
| 105 | 105 | * @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE. |
| 106 | 106 | * @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists. |
| 107 | 107 | * @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists. |
| 108 | + * @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions. | |
| 109 | + * @DAMOS_MIGRATE_COLD: Migrate the regions prioritizing colder regions. | |
| 108 | 110 | * @DAMOS_STAT: Do nothing but count the stat. |
| 109 | 111 | * @NR_DAMOS_ACTIONS: Total number of DAMOS actions |
| 110 | 112 | * |
| @@ -122,6 +124,8 @@ enum damos_action { · 11 unmodified lines | ||
| 122 | 124 | DAMOS_NOHUGEPAGE, |
| 123 | 125 | DAMOS_LRU_PRIO, |
| 124 | 126 | DAMOS_LRU_DEPRIO, |
| 127 | + DAMOS_MIGRATE_HOT, | |
| 128 | + DAMOS_MIGRATE_COLD, | |
| 125 | 129 | DAMOS_STAT, /* Do nothing but only record the stat */ |
| 126 | 130 | NR_DAMOS_ACTIONS, |
| 127 | 131 | }; |
| @@ -374,6 +378,7 @@ struct damos_access_pattern { · 246 unmodified lines | ||
| 374 | 378 | * @apply_interval_us: The time between applying the @action. |
| 375 | 379 | * @quota: Control the aggressiveness of this scheme. |
| 376 | 380 | * @wmarks: Watermarks for automated (in)activation of this scheme. |
| 381 | + * @target_nid: Destination node if @action is "migrate_{hot,cold}". | |
| 377 | 382 | * @filters: Additional set of &struct damos_filter for &action. |
| 378 | 383 | * @stat: Statistics of this scheme. |
| 379 | 384 | * @list: List head for siblings. |
| @@ -389,6 +394,10 @@ struct damos_access_pattern { · 9 unmodified lines | ||
| 389 | 394 | * monitoring context are inactive, DAMON stops monitoring either, and just |
| 390 | 395 | * repeatedly checks the watermarks. |
| 391 | 396 | * |
| 397 | + * @target_nid is used to set the migration target node for migrate_hot or | |
| 398 | + * migrate_cold actions, which means it's only meaningful when @action is either | |
| 399 | + * "migrate_hot" or "migrate_cold". | |
| 400 | + * | |
| 392 | 401 | * Before applying the &action to a memory region, &struct damon_operations |
| 393 | 402 | * implementation could check pages of the region and skip &action to respect |
| 394 | 403 | * &filters |
| @@ -410,6 +419,9 @@ struct damos { · 15 unmodified lines | ||
| 410 | 419 | /* public: */ |
| 411 | 420 | struct damos_quota quota; |
| 412 | 421 | struct damos_watermarks wmarks; |
| 422 | + union { | |
| 423 | + int target_nid; | |
| 424 | + }; | |
| 413 | 425 | struct list_head filters; |
| 414 | 426 | struct damos_stat stat; |
| 415 | 427 | struct list_head list; |
| @@ -726,9 +738,11 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 310 unmodified lines | ||
| 726 | 738 | enum damos_action action, |
| 727 | 739 | unsigned long apply_interval_us, |
| 728 | 740 | struct damos_quota *quota, |
| 729 | − struct damos_watermarks *wmarks); | |
| 741 | + struct damos_watermarks *wmarks, | |
| 742 | + int target_nid); | |
| 730 | 743 | void damon_add_scheme(struct damon_ctx *ctx, struct damos *s); |
| 731 | 744 | void damon_destroy_scheme(struct damos *s); |
| 745 | +int damos_commit_quota_goals(struct damos_quota *dst, struct damos_quota *src); | |
| 732 | 746 | |
| 733 | 747 | struct damon_target *damon_new_target(void); |
| 734 | 748 | void damon_add_target(struct damon_ctx *ctx, struct damon_target *t); |
| @@ -742,6 +756,7 @@ void damon_destroy_ctx(struct damon_ctx *ctx); · 7 unmodified lines | ||
| 742 | 756 | int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs); |
| 743 | 757 | void damon_set_schemes(struct damon_ctx *ctx, |
| 744 | 758 | struct damos **schemes, ssize_t nr_schemes); |
| 759 | +int damon_commit_ctx(struct damon_ctx *old_ctx, struct damon_ctx *new_ctx); | |
| 745 | 760 | int damon_nr_running_ctxs(void); |
| 746 | 761 | bool damon_is_registered_ops(enum damon_ops_id id); |
| 747 | 762 | int damon_register_ops(struct damon_operations *ops); |
| @@ -85,6 +85,10 @@ dma_fence_chain_contained(struct dma_fence *fence) · 84 unmodified lines | ||
| 85 | 85 | * dma_fence_chain_alloc |
| 86 | 86 | * |
| 87 | 87 | * Returns a new struct dma_fence_chain object or NULL on failure. |
| 88 | + * | |
| 89 | + * This specialized allocator has to be a macro for its allocations to be | |
| 90 | + * accounted separately (to have a separate alloc_tag). The typecast is | |
| 91 | + * intentional to enforce typesafety. | |
| 88 | 92 | */ |
| 89 | 93 | #define dma_fence_chain_alloc() \ |
| 90 | 94 | ((struct dma_fence_chain *)kmalloc(sizeof(struct dma_fence_chain), GFP_KERNEL)) |
| @@ -91,22 +91,19 @@ static inline void fault_config_init(struct fault_config *config, · 90 unmodified lines | ||
| 91 | 91 | |
| 92 | 92 | struct kmem_cache; |
| 93 | 93 | |
| 94 | −bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order); | |
| 95 | − | |
| 96 | 94 | #ifdef CONFIG_FAIL_PAGE_ALLOC |
| 97 | −bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order); | |
| 95 | +bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order); | |
| 98 | 96 | #else |
| 99 | −static inline bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) | |
| 97 | +static inline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) | |
| 100 | 98 | { |
| 101 | 99 | return false; |
| 102 | 100 | } |
| 103 | 101 | #endif /* CONFIG_FAIL_PAGE_ALLOC */ |
| 104 | 102 | |
| 105 | −int should_failslab(struct kmem_cache *s, gfp_t gfpflags); | |
| 106 | 103 | #ifdef CONFIG_FAILSLAB |
| 107 | −extern bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags); | |
| 104 | +int should_failslab(struct kmem_cache *s, gfp_t gfpflags); | |
| 108 | 105 | #else |
| 109 | −static inline bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags) | |
| 106 | +static inline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) | |
| 110 | 107 | { |
| 111 | 108 | return false; |
| 112 | 109 | } |
| @@ -303,6 +303,8 @@ struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order); · 302 unmodified lines | ||
| 303 | 303 | struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, |
| 304 | 304 | struct mempolicy *mpol, pgoff_t ilx, int nid); |
| 305 | 305 | struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order); |
| 306 | +struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order, | |
| 307 | + struct mempolicy *mpol, pgoff_t ilx, int nid); | |
| 306 | 308 | struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, |
| 307 | 309 | unsigned long addr, bool hugepage); |
| 308 | 310 | #else |
| @@ -319,6 +321,11 @@ static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) · 10 unmodified lines | ||
| 319 | 321 | { |
| 320 | 322 | return __folio_alloc_node(gfp, order, numa_node_id()); |
| 321 | 323 | } |
| 324 | +static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order, | |
| 325 | + struct mempolicy *mpol, pgoff_t ilx, int nid) | |
| 326 | +{ | |
| 327 | + return folio_alloc_noprof(gfp, order); | |
| 328 | +} | |
| 322 | 329 | #define vma_alloc_folio_noprof(gfp, order, vma, addr, hugepage) \ |
| 323 | 330 | folio_alloc_noprof(gfp, order) |
| 324 | 331 | #endif |
| @@ -326,6 +333,7 @@ static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) · 1 unmodified line | ||
| 326 | 333 | #define alloc_pages(...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__)) |
| 327 | 334 | #define alloc_pages_mpol(...) alloc_hooks(alloc_pages_mpol_noprof(__VA_ARGS__)) |
| 328 | 335 | #define folio_alloc(...) alloc_hooks(folio_alloc_noprof(__VA_ARGS__)) |
| 336 | +#define folio_alloc_mpol(...) alloc_hooks(folio_alloc_mpol_noprof(__VA_ARGS__)) | |
| 329 | 337 | #define vma_alloc_folio(...) alloc_hooks(vma_alloc_folio_noprof(__VA_ARGS__)) |
| 330 | 338 | |
| 331 | 339 | #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0) |
| @@ -228,6 +228,11 @@ static inline int hid_bpf_connect_device(struct hid_device *hdev) { return 0; } · 227 unmodified lines | ||
| 228 | 228 | static inline void hid_bpf_disconnect_device(struct hid_device *hdev) {} |
| 229 | 229 | static inline void hid_bpf_destroy_device(struct hid_device *hid) {} |
| 230 | 230 | static inline int hid_bpf_device_init(struct hid_device *hid) { return 0; } |
| 231 | +/* | |
| 232 | + * This specialized allocator has to be a macro for its allocations to be | |
| 233 | + * accounted separately (to have a separate alloc_tag). The typecast is | |
| 234 | + * intentional to enforce typesafety. | |
| 235 | + */ | |
| 231 | 236 | #define call_hid_bpf_rdesc_fixup(_hdev, _rdesc, _size) \ |
| 232 | 237 | ((u8 *)kmemdup(_rdesc, *(_size), GFP_KERNEL)) |
| 233 | 238 | |
| @@ -131,22 +131,17 @@ static inline void __kunmap_atomic(const void *addr) · 130 unmodified lines | ||
| 131 | 131 | preempt_enable(); |
| 132 | 132 | } |
| 133 | 133 | |
| 134 | −unsigned int __nr_free_highpages(void); | |
| 135 | −extern atomic_long_t _totalhigh_pages; | |
| 134 | +unsigned long __nr_free_highpages(void); | |
| 135 | +unsigned long __totalhigh_pages(void); | |
| 136 | 136 | |
| 137 | −static inline unsigned int nr_free_highpages(void) | |
| 137 | +static inline unsigned long nr_free_highpages(void) | |
| 138 | 138 | { |
| 139 | 139 | return __nr_free_highpages(); |
| 140 | 140 | } |
| 141 | 141 | |
| 142 | 142 | static inline unsigned long totalhigh_pages(void) |
| 143 | 143 | { |
| 144 | − return (unsigned long)atomic_long_read(&_totalhigh_pages); | |
| 145 | −} | |
| 146 | − | |
| 147 | −static inline void totalhigh_pages_add(long count) | |
| 148 | −{ | |
| 149 | − atomic_long_add(count, &_totalhigh_pages); | |
| 144 | + return __totalhigh_pages(); | |
| 150 | 145 | } |
| 151 | 146 | |
| 152 | 147 | static inline bool is_kmap_addr(const void *x) |
| @@ -239,8 +234,8 @@ static inline void __kunmap_atomic(const void *addr) · 86 unmodified lines | ||
| 239 | 234 | preempt_enable(); |
| 240 | 235 | } |
| 241 | 236 | |
| 242 | −static inline unsigned int nr_free_highpages(void) { return 0; } | |
| 243 | −static inline unsigned long totalhigh_pages(void) { return 0UL; } | |
| 237 | +static inline unsigned long nr_free_highpages(void) { return 0; } | |
| 238 | +static inline unsigned long totalhigh_pages(void) { return 0; } | |
| 244 | 239 | |
| 245 | 240 | static inline bool is_kmap_addr(const void *x) |
| 246 | 241 | { |
| @@ -179,7 +179,7 @@ static inline void *kmap_local_folio(struct folio *folio, size_t offset); · 178 unmodified lines | ||
| 179 | 179 | static inline void *kmap_atomic(struct page *page); |
| 180 | 180 | |
| 181 | 181 | /* Highmem related interfaces for management code */ |
| 182 | −static inline unsigned int nr_free_highpages(void); | |
| 182 | +static inline unsigned long nr_free_highpages(void); | |
| 183 | 183 | static inline unsigned long totalhigh_pages(void); |
| 184 | 184 | |
| 185 | 185 | #ifndef ARCH_HAS_FLUSH_ANON_PAGE |
| @@ -352,6 +352,9 @@ static inline int copy_mc_user_highpage(struct page *to, struct page *from, · 166 unmodified lines | ||
| 352 | 352 | kunmap_local(vto); |
| 353 | 353 | kunmap_local(vfrom); |
| 354 | 354 | |
| 355 | + if (ret) | |
| 356 | + memory_failure_queue(page_to_pfn(from), 0); | |
| 357 | + | |
| 355 | 358 | return ret; |
| 356 | 359 | } |
| 357 | 360 | |
| @@ -368,6 +371,9 @@ static inline int copy_mc_highpage(struct page *to, struct page *from) · 10 unmodified lines | ||
| 368 | 371 | kunmap_local(vto); |
| 369 | 372 | kunmap_local(vfrom); |
| 370 | 373 | |
| 374 | + if (ret) | |
| 375 | + memory_failure_queue(page_to_pfn(from), 0); | |
| 376 | + | |
| 371 | 377 | return ret; |
| 372 | 378 | } |
| 373 | 379 | #else |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <linux/mm_types.h> |
| 7 | 7 | |
| 8 | 8 | #include <linux/fs.h> /* only for vma_is_dax() */ |
| 9 | +#include <linux/kobject.h> | |
| 9 | 10 | |
| 10 | 11 | vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf); |
| 11 | 12 | int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, |
| @@ -63,6 +64,7 @@ ssize_t single_hugepage_flag_show(struct kobject *kobj, · 51 unmodified lines | ||
| 63 | 64 | struct kobj_attribute *attr, char *buf, |
| 64 | 65 | enum transparent_hugepage_flag flag); |
| 65 | 66 | extern struct kobj_attribute shmem_enabled_attr; |
| 67 | +extern struct kobj_attribute thpsize_shmem_enabled_attr; | |
| 66 | 68 | |
| 67 | 69 | /* |
| 68 | 70 | * Mask of all large folio orders supported for anonymous THP; all orders up to |
| @@ -126,18 +128,6 @@ static inline bool hugepage_global_always(void) · 57 unmodified lines | ||
| 126 | 128 | (1<<TRANSPARENT_HUGEPAGE_FLAG); |
| 127 | 129 | } |
| 128 | 130 | |
| 129 | −static inline bool hugepage_flags_enabled(void) | |
| 130 | −{ | |
| 131 | − /* | |
| 132 | − * We cover both the anon and the file-backed case here; we must return | |
| 133 | − * true if globally enabled, even when all anon sizes are set to never. | |
| 134 | − * So we don't need to look at huge_anon_orders_inherit. | |
| 135 | − */ | |
| 136 | − return hugepage_global_enabled() || | |
| 137 | − huge_anon_orders_always || | |
| 138 | − huge_anon_orders_madvise; | |
| 139 | −} | |
| 140 | − | |
| 141 | 131 | static inline int highest_order(unsigned long orders) |
| 142 | 132 | { |
| 143 | 133 | return fls_long(orders) - 1; |
| @@ -265,12 +255,26 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, · 121 unmodified lines | ||
| 265 | 255 | return __thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders); |
| 266 | 256 | } |
| 267 | 257 | |
| 258 | +struct thpsize { | |
| 259 | + struct kobject kobj; | |
| 260 | + struct list_head node; | |
| 261 | + int order; | |
| 262 | +}; | |
| 263 | + | |
| 264 | +#define to_thpsize(kobj) container_of(kobj, struct thpsize, kobj) | |
| 265 | + | |
| 268 | 266 | enum mthp_stat_item { |
| 269 | 267 | MTHP_STAT_ANON_FAULT_ALLOC, |
| 270 | 268 | MTHP_STAT_ANON_FAULT_FALLBACK, |
| 271 | 269 | MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, |
| 272 | 270 | MTHP_STAT_SWPOUT, |
| 273 | 271 | MTHP_STAT_SWPOUT_FALLBACK, |
| 272 | + MTHP_STAT_SHMEM_ALLOC, | |
| 273 | + MTHP_STAT_SHMEM_FALLBACK, | |
| 274 | + MTHP_STAT_SHMEM_FALLBACK_CHARGE, | |
| 275 | + MTHP_STAT_SPLIT, | |
| 276 | + MTHP_STAT_SPLIT_FAILED, | |
| 277 | + MTHP_STAT_SPLIT_DEFERRED, | |
| 274 | 278 | __MTHP_STAT_COUNT |
| 275 | 279 | }; |
| 276 | 280 | |
| @@ -415,6 +419,11 @@ static inline bool thp_migration_supported(void) · 138 unmodified lines | ||
| 415 | 419 | return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); |
| 416 | 420 | } |
| 417 | 421 | |
| 422 | +void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, | |
| 423 | + pmd_t *pmd, bool freeze, struct folio *folio); | |
| 424 | +bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, | |
| 425 | + pmd_t *pmdp, struct folio *folio); | |
| 426 | + | |
| 418 | 427 | #else /* CONFIG_TRANSPARENT_HUGEPAGE */ |
| 419 | 428 | |
| 420 | 429 | static inline bool folio_test_pmd_mappable(struct folio *folio) |
| @@ -477,6 +486,16 @@ static inline void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, · 56 unmodified lines | ||
| 477 | 486 | unsigned long address, bool freeze, struct folio *folio) {} |
| 478 | 487 | static inline void split_huge_pmd_address(struct vm_area_struct *vma, |
| 479 | 488 | unsigned long address, bool freeze, struct folio *folio) {} |
| 489 | +static inline void split_huge_pmd_locked(struct vm_area_struct *vma, | |
| 490 | + unsigned long address, pmd_t *pmd, | |
| 491 | + bool freeze, struct folio *folio) {} | |
| 492 | + | |
| 493 | +static inline bool unmap_huge_pmd_locked(struct vm_area_struct *vma, | |
| 494 | + unsigned long addr, pmd_t *pmdp, | |
| 495 | + struct folio *folio) | |
| 496 | +{ | |
| 497 | + return false; | |
| 498 | +} | |
| 480 | 499 | |
| 481 | 500 | #define split_huge_pud(__vma, __pmd, __address) \ |
| 482 | 501 | do { } while (0) |
| @@ -550,6 +569,16 @@ static inline bool thp_migration_supported(void) · 67 unmodified lines | ||
| 550 | 569 | { |
| 551 | 570 | return false; |
| 552 | 571 | } |
| 572 | + | |
| 573 | +static inline int highest_order(unsigned long orders) | |
| 574 | +{ | |
| 575 | + return 0; | |
| 576 | +} | |
| 577 | + | |
| 578 | +static inline int next_order(unsigned long *orders, int prev) | |
| 579 | +{ | |
| 580 | + return 0; | |
| 581 | +} | |
| 553 | 582 | #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ |
| 554 | 583 | |
| 555 | 584 | static inline int split_folio_to_list_to_order(struct folio *folio, |
| @@ -20,12 +20,6 @@ struct user_struct; · 19 unmodified lines | ||
| 20 | 20 | struct mmu_gather; |
| 21 | 21 | struct node; |
| 22 | 22 | |
| 23 | −#ifndef CONFIG_ARCH_HAS_HUGEPD | |
| 24 | −typedef struct { unsigned long pd; } hugepd_t; | |
| 25 | −#define is_hugepd(hugepd) (0) | |
| 26 | −#define __hugepd(x) ((hugepd_t) { (x) }) | |
| 27 | −#endif | |
| 28 | − | |
| 29 | 23 | void free_huge_folio(struct folio *folio); |
| 30 | 24 | |
| 31 | 25 | #ifdef CONFIG_HUGETLB_PAGE |
| @@ -616,47 +610,35 @@ static __always_inline \ · 584 unmodified lines | ||
| 616 | 610 | bool folio_test_hugetlb_##flname(struct folio *folio) \ |
| 617 | 611 | { void *private = &folio->private; \ |
| 618 | 612 | return test_bit(HPG_##flname, private); \ |
| 619 | − } \ | |
| 620 | −static inline int HPage##uname(struct page *page) \ | |
| 621 | − { return test_bit(HPG_##flname, &(page->private)); } | |
| 613 | + } | |
| 622 | 614 | |
| 623 | 615 | #define SETHPAGEFLAG(uname, flname) \ |
| 624 | 616 | static __always_inline \ |
| 625 | 617 | void folio_set_hugetlb_##flname(struct folio *folio) \ |
| 626 | 618 | { void *private = &folio->private; \ |
| 627 | 619 | set_bit(HPG_##flname, private); \ |
| 628 | − } \ | |
| 629 | −static inline void SetHPage##uname(struct page *page) \ | |
| 630 | − { set_bit(HPG_##flname, &(page->private)); } | |
| 620 | + } | |
| 631 | 621 | |
| 632 | 622 | #define CLEARHPAGEFLAG(uname, flname) \ |
| 633 | 623 | static __always_inline \ |
| 634 | 624 | void folio_clear_hugetlb_##flname(struct folio *folio) \ |
| 635 | 625 | { void *private = &folio->private; \ |
| 636 | 626 | clear_bit(HPG_##flname, private); \ |
| 637 | − } \ | |
| 638 | −static inline void ClearHPage##uname(struct page *page) \ | |
| 639 | − { clear_bit(HPG_##flname, &(page->private)); } | |
| 627 | + } | |
| 640 | 628 | #else |
| 641 | 629 | #define TESTHPAGEFLAG(uname, flname) \ |
| 642 | 630 | static inline bool \ |
| 643 | 631 | folio_test_hugetlb_##flname(struct folio *folio) \ |
| 644 | − { return 0; } \ | |
| 645 | −static inline int HPage##uname(struct page *page) \ | |
| 646 | 632 | { return 0; } |
| 647 | 633 | |
| 648 | 634 | #define SETHPAGEFLAG(uname, flname) \ |
| 649 | 635 | static inline void \ |
| 650 | 636 | folio_set_hugetlb_##flname(struct folio *folio) \ |
| 651 | − { } \ | |
| 652 | −static inline void SetHPage##uname(struct page *page) \ | |
| 653 | 637 | { } |
| 654 | 638 | |
| 655 | 639 | #define CLEARHPAGEFLAG(uname, flname) \ |
| 656 | 640 | static inline void \ |
| 657 | 641 | folio_clear_hugetlb_##flname(struct folio *folio) \ |
| 658 | − { } \ | |
| 659 | −static inline void ClearHPage##uname(struct page *page) \ | |
| 660 | 642 | { } |
| 661 | 643 | #endif |
| 662 | 644 | |
| @@ -681,6 +663,7 @@ HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) · 18 unmodified lines | ||
| 681 | 663 | /* Defines one hugetlb page size */ |
| 682 | 664 | struct hstate { |
| 683 | 665 | struct mutex resize_lock; |
| 666 | + struct lock_class_key resize_key; | |
| 684 | 667 | int next_nid_to_alloc; |
| 685 | 668 | int next_nid_to_free; |
| 686 | 669 | unsigned int order; |
| @@ -698,11 +681,6 @@ struct hstate { · 11 unmodified lines | ||
| 698 | 681 | unsigned int nr_huge_pages_node[MAX_NUMNODES]; |
| 699 | 682 | unsigned int free_huge_pages_node[MAX_NUMNODES]; |
| 700 | 683 | unsigned int surplus_huge_pages_node[MAX_NUMNODES]; |
| 701 | −#ifdef CONFIG_CGROUP_HUGETLB | |
| 702 | − /* cgroup control files */ | |
| 703 | − struct cftype cgroup_files_dfl[8]; | |
| 704 | − struct cftype cgroup_files_legacy[10]; | |
| 705 | −#endif | |
| 706 | 684 | char name[HSTATE_NAME_LEN]; |
| 707 | 685 | }; |
| 708 | 686 | |
| @@ -1595,6 +1595,11 @@ void jbd2_journal_put_journal_head(struct journal_head *jh); · 1594 unmodified lines | ||
| 1595 | 1595 | */ |
| 1596 | 1596 | extern struct kmem_cache *jbd2_handle_cache; |
| 1597 | 1597 | |
| 1598 | +/* | |
| 1599 | + * This specialized allocator has to be a macro for its allocations to be | |
| 1600 | + * accounted separately (to have a separate alloc_tag). The typecast is | |
| 1601 | + * intentional to enforce typesafety. | |
| 1602 | + */ | |
| 1598 | 1603 | #define jbd2_alloc_handle(_gfp_flags) \ |
| 1599 | 1604 | ((handle_t *)kmem_cache_zalloc(jbd2_handle_cache, _gfp_flags)) |
| 1600 | 1605 | |
| @@ -1609,6 +1614,11 @@ static inline void jbd2_free_handle(handle_t *handle) · 8 unmodified lines | ||
| 1609 | 1614 | */ |
| 1610 | 1615 | extern struct kmem_cache *jbd2_inode_cache; |
| 1611 | 1616 | |
| 1617 | +/* | |
| 1618 | + * This specialized allocator has to be a macro for its allocations to be | |
| 1619 | + * accounted separately (to have a separate alloc_tag). The typecast is | |
| 1620 | + * intentional to enforce typesafety. | |
| 1621 | + */ | |
| 1612 | 1622 | #define jbd2_alloc_inode(_gfp_flags) \ |
| 1613 | 1623 | ((struct jbd2_inode *)kmem_cache_alloc(jbd2_inode_cache, _gfp_flags)) |
| 1614 | 1624 | |
| @@ -230,6 +230,67 @@ void kmsan_handle_urb(const struct urb *urb, bool is_out); · 229 unmodified lines | ||
| 230 | 230 | */ |
| 231 | 231 | void kmsan_unpoison_entry_regs(const struct pt_regs *regs); |
| 232 | 232 | |
| 233 | +/** | |
| 234 | + * kmsan_get_metadata() - Return a pointer to KMSAN shadow or origins. | |
| 235 | + * @addr: kernel address. | |
| 236 | + * @is_origin: whether to return origins or shadow. | |
| 237 | + * | |
| 238 | + * Return NULL if metadata cannot be found. | |
| 239 | + */ | |
| 240 | +void *kmsan_get_metadata(void *addr, bool is_origin); | |
| 241 | + | |
| 242 | +/** | |
| 243 | + * kmsan_enable_current(): Enable KMSAN for the current task. | |
| 244 | + * | |
| 245 | + * Each kmsan_enable_current() current call must be preceded by a | |
| 246 | + * kmsan_disable_current() call. These call pairs may be nested. | |
| 247 | + */ | |
| 248 | +void kmsan_enable_current(void); | |
| 249 | + | |
| 250 | +/** | |
| 251 | + * kmsan_disable_current(): Disable KMSAN for the current task. | |
| 252 | + * | |
| 253 | + * Each kmsan_disable_current() current call must be followed by a | |
| 254 | + * kmsan_enable_current() call. These call pairs may be nested. | |
| 255 | + */ | |
| 256 | +void kmsan_disable_current(void); | |
| 257 | + | |
| 258 | +/** | |
| 259 | + * memset_no_sanitize_memory(): Fill memory without KMSAN instrumentation. | |
| 260 | + * @s: address of kernel memory to fill. | |
| 261 | + * @c: constant byte to fill the memory with. | |
| 262 | + * @n: number of bytes to fill. | |
| 263 | + * | |
| 264 | + * This is like memset(), but without KMSAN instrumentation. | |
| 265 | + */ | |
| 266 | +static inline void *memset_no_sanitize_memory(void *s, int c, size_t n) | |
| 267 | +{ | |
| 268 | + return __memset(s, c, n); | |
| 269 | +} | |
| 270 | + | |
| 271 | +extern bool kmsan_enabled; | |
| 272 | +extern int panic_on_kmsan; | |
| 273 | + | |
| 274 | +/* | |
| 275 | + * KMSAN performs a lot of consistency checks that are currently enabled by | |
| 276 | + * default. BUG_ON is normally discouraged in the kernel, unless used for | |
| 277 | + * debugging, but KMSAN itself is a debugging tool, so it makes little sense to | |
| 278 | + * recover if something goes wrong. | |
| 279 | + */ | |
| 280 | +#define KMSAN_WARN_ON(cond) \ | |
| 281 | + ({ \ | |
| 282 | + const bool __cond = WARN_ON(cond); \ | |
| 283 | + if (unlikely(__cond)) { \ | |
| 284 | + WRITE_ONCE(kmsan_enabled, false); \ | |
| 285 | + if (panic_on_kmsan) { \ | |
| 286 | + /* Can't call panic() here because */ \ | |
| 287 | + /* of uaccess checks. */ \ | |
| 288 | + BUG(); \ | |
| 289 | + } \ | |
| 290 | + } \ | |
| 291 | + __cond; \ | |
| 292 | + }) | |
| 293 | + | |
| 233 | 294 | #else |
| 234 | 295 | |
| 235 | 296 | static inline void kmsan_init_shadow(void) |
| @@ -329,6 +390,21 @@ static inline void kmsan_unpoison_entry_regs(const struct pt_regs *regs) · 93 unmodified lines | ||
| 329 | 390 | { |
| 330 | 391 | } |
| 331 | 392 | |
| 393 | +static inline void kmsan_enable_current(void) | |
| 394 | +{ | |
| 395 | +} | |
| 396 | + | |
| 397 | +static inline void kmsan_disable_current(void) | |
| 398 | +{ | |
| 399 | +} | |
| 400 | + | |
| 401 | +static inline void *memset_no_sanitize_memory(void *s, int c, size_t n) | |
| 402 | +{ | |
| 403 | + return memset(s, c, n); | |
| 404 | +} | |
| 405 | + | |
| 406 | +#define KMSAN_WARN_ON WARN_ON | |
| 407 | + | |
| 332 | 408 | #endif |
| 333 | 409 | |
| 334 | 410 | #endif /* _LINUX_KMSAN_H */ |
| @@ -31,7 +31,7 @@ struct kmsan_context_state { · 30 unmodified lines | ||
| 31 | 31 | struct kmsan_ctx { |
| 32 | 32 | struct kmsan_context_state cstate; |
| 33 | 33 | int kmsan_in_runtime; |
| 34 | − bool allow_reporting; | |
| 34 | + unsigned int depth; | |
| 35 | 35 | }; |
| 36 | 36 | |
| 37 | 37 | #endif /* _LINUX_KMSAN_TYPES_H */ |
| @@ -50,7 +50,7 @@ struct list_lru_node { · 49 unmodified lines | ||
| 50 | 50 | |
| 51 | 51 | struct list_lru { |
| 52 | 52 | struct list_lru_node *node; |
| 53 | −#ifdef CONFIG_MEMCG_KMEM | |
| 53 | +#ifdef CONFIG_MEMCG | |
| 54 | 54 | struct list_head list; |
| 55 | 55 | int shrinker_id; |
| 56 | 56 | bool memcg_aware; |
| @@ -316,8 +316,6 @@ void __next_mem_pfn_range_in_zone(u64 *idx, struct zone *zone, · 315 unmodified lines | ||
| 316 | 316 | for (; i != U64_MAX; \ |
| 317 | 317 | __next_mem_pfn_range_in_zone(&i, zone, p_start, p_end)) |
| 318 | 318 | |
| 319 | −int __init deferred_page_init_max_threads(const struct cpumask *node_cpumask); | |
| 320 | − | |
| 321 | 319 | #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ |
| 322 | 320 | |
| 323 | 321 | /** |
| @@ -69,18 +69,6 @@ struct mem_cgroup_id { · 68 unmodified lines | ||
| 69 | 69 | refcount_t ref; |
| 70 | 70 | }; |
| 71 | 71 | |
| 72 | −/* | |
| 73 | − * Per memcg event counter is incremented at every pagein/pageout. With THP, | |
| 74 | − * it will be incremented by the number of pages. This counter is used | |
| 75 | − * to trigger some periodic events. This is straightforward and better | |
| 76 | − * than using jiffies etc. to handle periodic memcg event. | |
| 77 | − */ | |
| 78 | −enum mem_cgroup_events_target { | |
| 79 | − MEM_CGROUP_TARGET_THRESH, | |
| 80 | − MEM_CGROUP_TARGET_SOFTLIMIT, | |
| 81 | − MEM_CGROUP_NTARGETS, | |
| 82 | −}; | |
| 83 | − | |
| 84 | 72 | struct memcg_vmstats_percpu; |
| 85 | 73 | struct memcg_vmstats; |
| 86 | 74 | struct lruvec_stats_percpu; |
| @@ -96,23 +84,33 @@ struct mem_cgroup_reclaim_iter { · 9 unmodified lines | ||
| 96 | 84 | * per-node information in memory controller. |
| 97 | 85 | */ |
| 98 | 86 | struct mem_cgroup_per_node { |
| 99 | − struct lruvec lruvec; | |
| 87 | + /* Keep the read-only fields at the start */ | |
| 88 | + struct mem_cgroup *memcg; /* Back pointer, we cannot */ | |
| 89 | + /* use container_of */ | |
| 100 | 90 | |
| 101 | 91 | struct lruvec_stats_percpu __percpu *lruvec_stats_percpu; |
| 102 | 92 | struct lruvec_stats *lruvec_stats; |
| 103 | − | |
| 104 | − unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; | |
| 105 | − | |
| 106 | − struct mem_cgroup_reclaim_iter iter; | |
| 107 | − | |
| 108 | 93 | struct shrinker_info __rcu *shrinker_info; |
| 109 | 94 | |
| 95 | +#ifdef CONFIG_MEMCG_V1 | |
| 96 | + /* | |
| 97 | + * Memcg-v1 only stuff in middle as buffer between read mostly fields | |
| 98 | + * and update often fields to avoid false sharing. If v1 stuff is | |
| 99 | + * not present, an explicit padding is needed. | |
| 100 | + */ | |
| 101 | + | |
| 110 | 102 | struct rb_node tree_node; /* RB tree node */ |
| 111 | 103 | unsigned long usage_in_excess;/* Set to the value by which */ |
| 112 | 104 | /* the soft limit is exceeded*/ |
| 113 | 105 | bool on_tree; |
| 114 | − struct mem_cgroup *memcg; /* Back pointer, we cannot */ | |
| 115 | − /* use container_of */ | |
| 106 | +#else | |
| 107 | + CACHELINE_PADDING(_pad1_); | |
| 108 | +#endif | |
| 109 | + | |
| 110 | + /* Fields which get updated often at the end. */ | |
| 111 | + struct lruvec lruvec; | |
| 112 | + unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; | |
| 113 | + struct mem_cgroup_reclaim_iter iter; | |
| 116 | 114 | }; |
| 117 | 115 | |
| 118 | 116 | struct mem_cgroup_threshold { |
| @@ -194,14 +192,10 @@ struct mem_cgroup { · 75 unmodified lines | ||
| 194 | 192 | struct page_counter memsw; /* v1 only */ |
| 195 | 193 | }; |
| 196 | 194 | |
| 197 | − /* Legacy consumer-oriented counters */ | |
| 198 | − struct page_counter kmem; /* v1 only */ | |
| 199 | − struct page_counter tcpmem; /* v1 only */ | |
| 200 | − | |
| 201 | 195 | /* Range enforcement for interrupt charges */ |
| 202 | 196 | struct work_struct high_work; |
| 203 | 197 | |
| 204 | −#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) | |
| 198 | +#ifdef CONFIG_ZSWAP | |
| 205 | 199 | unsigned long zswap_max; |
| 206 | 200 | |
| 207 | 201 | /* |
| @@ -211,8 +205,6 @@ struct mem_cgroup { · 3 unmodified lines | ||
| 211 | 205 | bool zswap_writeback; |
| 212 | 206 | #endif |
| 213 | 207 | |
| 214 | − unsigned long soft_limit; | |
| 215 | − | |
| 216 | 208 | /* vmpressure notifications */ |
| 217 | 209 | struct vmpressure vmpressure; |
| 218 | 210 | |
| @@ -221,13 +213,7 @@ struct mem_cgroup { · 2 unmodified lines | ||
| 221 | 213 | */ |
| 222 | 214 | bool oom_group; |
| 223 | 215 | |
| 224 | − /* protected by memcg_oom_lock */ | |
| 225 | − bool oom_lock; | |
| 226 | − int under_oom; | |
| 227 | − | |
| 228 | − int swappiness; | |
| 229 | − /* OOM-Killer disable */ | |
| 230 | − int oom_kill_disable; | |
| 216 | + int swappiness; | |
| 231 | 217 | |
| 232 | 218 | /* memory.events and memory.events.local */ |
| 233 | 219 | struct cgroup_file events_file; |
| @@ -236,29 +222,6 @@ struct mem_cgroup { · 2 unmodified lines | ||
| 236 | 222 | /* handle for "memory.swap.events" */ |
| 237 | 223 | struct cgroup_file swap_events_file; |
| 238 | 224 | |
| 239 | − /* protect arrays of thresholds */ | |
| 240 | − struct mutex thresholds_lock; | |
| 241 | − | |
| 242 | − /* thresholds for memory usage. RCU-protected */ | |
| 243 | − struct mem_cgroup_thresholds thresholds; | |
| 244 | − | |
| 245 | − /* thresholds for mem+swap usage. RCU-protected */ | |
| 246 | − struct mem_cgroup_thresholds memsw_thresholds; | |
| 247 | − | |
| 248 | − /* For oom notifier event fd */ | |
| 249 | − struct list_head oom_notify; | |
| 250 | − | |
| 251 | − /* | |
| 252 | − * Should we move charges of a task when a task is moved into this | |
| 253 | − * mem_cgroup ? And what type of charges should we move ? | |
| 254 | − */ | |
| 255 | − unsigned long move_charge_at_immigrate; | |
| 256 | − /* taken only while moving_account > 0 */ | |
| 257 | − spinlock_t move_lock; | |
| 258 | − unsigned long move_lock_flags; | |
| 259 | − | |
| 260 | − CACHELINE_PADDING(_pad1_); | |
| 261 | − | |
| 262 | 225 | /* memory.stat */ |
| 263 | 226 | struct memcg_vmstats *vmstats; |
| 264 | 227 | |
| @@ -273,11 +236,6 @@ struct mem_cgroup { · 8 unmodified lines | ||
| 273 | 236 | */ |
| 274 | 237 | unsigned long socket_pressure; |
| 275 | 238 | |
| 276 | − /* Legacy tcp memory accounting */ | |
| 277 | − bool tcpmem_active; | |
| 278 | − int tcpmem_pressure; | |
| 279 | − | |
| 280 | −#ifdef CONFIG_MEMCG_KMEM | |
| 281 | 239 | int kmemcg_id; |
| 282 | 240 | /* |
| 283 | 241 | * memcg->objcg is wiped out as a part of the objcg repaprenting |
| @@ -288,15 +246,6 @@ struct mem_cgroup { · 4 unmodified lines | ||
| 288 | 246 | struct obj_cgroup *orig_objcg; |
| 289 | 247 | /* list of inherited objcgs, protected by objcg_lock */ |
| 290 | 248 | struct list_head objcg_list; |
| 291 | −#endif | |
| 292 | − | |
| 293 | − CACHELINE_PADDING(_pad2_); | |
| 294 | − | |
| 295 | − /* | |
| 296 | − * set > 0 if pages under this cgroup are moving to other cgroup. | |
| 297 | − */ | |
| 298 | − atomic_t moving_account; | |
| 299 | − struct task_struct *move_lock_task; | |
| 300 | 249 | |
| 301 | 250 | struct memcg_vmstats_percpu __percpu *vmstats_percpu; |
| 302 | 251 | |
| @@ -306,10 +255,6 @@ struct mem_cgroup { · 3 unmodified lines | ||
| 306 | 255 | struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT]; |
| 307 | 256 | #endif |
| 308 | 257 | |
| 309 | − /* List of events which userspace want to receive */ | |
| 310 | − struct list_head event_list; | |
| 311 | − spinlock_t event_list_lock; | |
| 312 | − | |
| 313 | 258 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| 314 | 259 | struct deferred_split deferred_split_queue; |
| 315 | 260 | #endif |
| @@ -319,6 +264,56 @@ struct mem_cgroup { · 3 unmodified lines | ||
| 319 | 264 | struct lru_gen_mm_list mm_list; |
| 320 | 265 | #endif |
| 321 | 266 | |
| 267 | +#ifdef CONFIG_MEMCG_V1 | |
| 268 | + /* Legacy consumer-oriented counters */ | |
| 269 | + struct page_counter kmem; /* v1 only */ | |
| 270 | + struct page_counter tcpmem; /* v1 only */ | |
| 271 | + | |
| 272 | + unsigned long soft_limit; | |
| 273 | + | |
| 274 | + /* protected by memcg_oom_lock */ | |
| 275 | + bool oom_lock; | |
| 276 | + int under_oom; | |
| 277 | + | |
| 278 | + /* OOM-Killer disable */ | |
| 279 | + int oom_kill_disable; | |
| 280 | + | |
| 281 | + /* protect arrays of thresholds */ | |
| 282 | + struct mutex thresholds_lock; | |
| 283 | + | |
| 284 | + /* thresholds for memory usage. RCU-protected */ | |
| 285 | + struct mem_cgroup_thresholds thresholds; | |
| 286 | + | |
| 287 | + /* thresholds for mem+swap usage. RCU-protected */ | |
| 288 | + struct mem_cgroup_thresholds memsw_thresholds; | |
| 289 | + | |
| 290 | + /* For oom notifier event fd */ | |
| 291 | + struct list_head oom_notify; | |
| 292 | + | |
| 293 | + /* | |
| 294 | + * Should we move charges of a task when a task is moved into this | |
| 295 | + * mem_cgroup ? And what type of charges should we move ? | |
| 296 | + */ | |
| 297 | + unsigned long move_charge_at_immigrate; | |
| 298 | + /* taken only while moving_account > 0 */ | |
| 299 | + spinlock_t move_lock; | |
| 300 | + unsigned long move_lock_flags; | |
| 301 | + | |
| 302 | + /* Legacy tcp memory accounting */ | |
| 303 | + bool tcpmem_active; | |
| 304 | + int tcpmem_pressure; | |
| 305 | + | |
| 306 | + /* | |
| 307 | + * set > 0 if pages under this cgroup are moving to other cgroup. | |
| 308 | + */ | |
| 309 | + atomic_t moving_account; | |
| 310 | + struct task_struct *move_lock_task; | |
| 311 | + | |
| 312 | + /* List of events which userspace want to receive */ | |
| 313 | + struct list_head event_list; | |
| 314 | + spinlock_t event_list_lock; | |
| 315 | +#endif /* CONFIG_MEMCG_V1 */ | |
| 316 | + | |
| 322 | 317 | struct mem_cgroup_per_node *nodeinfo[]; |
| 323 | 318 | }; |
| 324 | 319 | |
| @@ -443,11 +438,6 @@ static inline struct mem_cgroup *folio_memcg(struct folio *folio) · 118 unmodified lines | ||
| 443 | 438 | return __folio_memcg(folio); |
| 444 | 439 | } |
| 445 | 440 | |
| 446 | −static inline struct mem_cgroup *page_memcg(struct page *page) | |
| 447 | −{ | |
| 448 | − return folio_memcg(page_folio(page)); | |
| 449 | −} | |
| 450 | − | |
| 451 | 441 | /** |
| 452 | 442 | * folio_memcg_rcu - Locklessly get the memory cgroup associated with a folio. |
| 453 | 443 | * @folio: Pointer to the folio. |
| @@ -540,7 +530,6 @@ retry: · 86 unmodified lines | ||
| 540 | 530 | return memcg; |
| 541 | 531 | } |
| 542 | 532 | |
| 543 | −#ifdef CONFIG_MEMCG_KMEM | |
| 544 | 533 | /* |
| 545 | 534 | * folio_memcg_kmem - Check if the folio has the memcg_kmem flag set. |
| 546 | 535 | * @folio: Pointer to the folio. |
| @@ -556,15 +545,6 @@ static inline bool folio_memcg_kmem(struct folio *folio) · 9 unmodified lines | ||
| 556 | 545 | return folio->memcg_data & MEMCG_DATA_KMEM; |
| 557 | 546 | } |
| 558 | 547 | |
| 559 | − | |
| 560 | −#else | |
| 561 | −static inline bool folio_memcg_kmem(struct folio *folio) | |
| 562 | −{ | |
| 563 | − return false; | |
| 564 | −} | |
| 565 | − | |
| 566 | −#endif | |
| 567 | − | |
| 568 | 548 | static inline bool PageMemcgKmem(struct page *page) |
| 569 | 549 | { |
| 570 | 550 | return folio_memcg_kmem(page_folio(page)); |
| @@ -949,51 +929,13 @@ void mem_cgroup_print_oom_context(struct mem_cgroup *memcg, · 378 unmodified lines | ||
| 949 | 929 | |
| 950 | 930 | void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg); |
| 951 | 931 | |
| 952 | −static inline void mem_cgroup_enter_user_fault(void) | |
| 953 | −{ | |
| 954 | − WARN_ON(current->in_user_fault); | |
| 955 | − current->in_user_fault = 1; | |
| 956 | −} | |
| 957 | − | |
| 958 | −static inline void mem_cgroup_exit_user_fault(void) | |
| 959 | −{ | |
| 960 | − WARN_ON(!current->in_user_fault); | |
| 961 | − current->in_user_fault = 0; | |
| 962 | −} | |
| 963 | − | |
| 964 | −static inline bool task_in_memcg_oom(struct task_struct *p) | |
| 965 | −{ | |
| 966 | − return p->memcg_in_oom; | |
| 967 | −} | |
| 968 | − | |
| 969 | −bool mem_cgroup_oom_synchronize(bool wait); | |
| 970 | 932 | struct mem_cgroup *mem_cgroup_get_oom_group(struct task_struct *victim, |
| 971 | 933 | struct mem_cgroup *oom_domain); |
| 972 | 934 | void mem_cgroup_print_oom_group(struct mem_cgroup *memcg); |
| 973 | 935 | |
| 974 | −void folio_memcg_lock(struct folio *folio); | |
| 975 | −void folio_memcg_unlock(struct folio *folio); | |
| 976 | − | |
| 977 | 936 | void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, |
| 978 | 937 | int val); |
| 979 | 938 | |
| 980 | −/* try to stablize folio_memcg() for all the pages in a memcg */ | |
| 981 | −static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg) | |
| 982 | −{ | |
| 983 | − rcu_read_lock(); | |
| 984 | − | |
| 985 | − if (mem_cgroup_disabled() || !atomic_read(&memcg->moving_account)) | |
| 986 | − return true; | |
| 987 | − | |
| 988 | − rcu_read_unlock(); | |
| 989 | − return false; | |
| 990 | −} | |
| 991 | − | |
| 992 | −static inline void mem_cgroup_unlock_pages(void) | |
| 993 | −{ | |
| 994 | − rcu_read_unlock(); | |
| 995 | −} | |
| 996 | − | |
| 997 | 939 | /* idx can be of type enum memcg_stat_item or node_stat_item */ |
| 998 | 940 | static inline void mod_memcg_state(struct mem_cgroup *memcg, |
| 999 | 941 | enum memcg_stat_item idx, int val) |
| @@ -1014,7 +956,7 @@ static inline void mod_memcg_page_state(struct page *page, · 14 unmodified lines | ||
| 1014 | 956 | return; |
| 1015 | 957 | |
| 1016 | 958 | rcu_read_lock(); |
| 1017 | − memcg = page_memcg(page); | |
| 959 | + memcg = folio_memcg(page_folio(page)); | |
| 1018 | 960 | if (memcg) |
| 1019 | 961 | mod_memcg_state(memcg, idx, val); |
| 1020 | 962 | rcu_read_unlock(); |
| @@ -1120,10 +1062,6 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, · 99 unmodified lines | ||
| 1120 | 1062 | |
| 1121 | 1063 | void split_page_memcg(struct page *head, int old_order, int new_order); |
| 1122 | 1064 | |
| 1123 | −unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, | |
| 1124 | − gfp_t gfp_mask, | |
| 1125 | − unsigned long *total_scanned); | |
| 1126 | − | |
| 1127 | 1065 | #else /* CONFIG_MEMCG */ |
| 1128 | 1066 | |
| 1129 | 1067 | #define MEM_CGROUP_ID_SHIFT 0 |
| @@ -1133,11 +1071,6 @@ static inline struct mem_cgroup *folio_memcg(struct folio *folio) · 3 unmodified lines | ||
| 1133 | 1071 | return NULL; |
| 1134 | 1072 | } |
| 1135 | 1073 | |
| 1136 | −static inline struct mem_cgroup *page_memcg(struct page *page) | |
| 1137 | −{ | |
| 1138 | − return NULL; | |
| 1139 | −} | |
| 1140 | − | |
| 1141 | 1074 | static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) |
| 1142 | 1075 | { |
| 1143 | 1076 | WARN_ON_ONCE(!rcu_read_lock_held()); |
| @@ -1439,48 +1372,10 @@ mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) · 295 unmodified lines | ||
| 1439 | 1372 | { |
| 1440 | 1373 | } |
| 1441 | 1374 | |
| 1442 | −static inline void folio_memcg_lock(struct folio *folio) | |
| 1443 | −{ | |
| 1444 | −} | |
| 1445 | − | |
| 1446 | −static inline void folio_memcg_unlock(struct folio *folio) | |
| 1447 | −{ | |
| 1448 | −} | |
| 1449 | − | |
| 1450 | −static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg) | |
| 1451 | −{ | |
| 1452 | − /* to match folio_memcg_rcu() */ | |
| 1453 | − rcu_read_lock(); | |
| 1454 | − return true; | |
| 1455 | −} | |
| 1456 | − | |
| 1457 | −static inline void mem_cgroup_unlock_pages(void) | |
| 1458 | −{ | |
| 1459 | − rcu_read_unlock(); | |
| 1460 | −} | |
| 1461 | − | |
| 1462 | 1375 | static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask) |
| 1463 | 1376 | { |
| 1464 | 1377 | } |
| 1465 | 1378 | |
| 1466 | −static inline void mem_cgroup_enter_user_fault(void) | |
| 1467 | −{ | |
| 1468 | −} | |
| 1469 | − | |
| 1470 | −static inline void mem_cgroup_exit_user_fault(void) | |
| 1471 | −{ | |
| 1472 | −} | |
| 1473 | − | |
| 1474 | −static inline bool task_in_memcg_oom(struct task_struct *p) | |
| 1475 | −{ | |
| 1476 | − return false; | |
| 1477 | −} | |
| 1478 | − | |
| 1479 | −static inline bool mem_cgroup_oom_synchronize(bool wait) | |
| 1480 | −{ | |
| 1481 | − return false; | |
| 1482 | −} | |
| 1483 | − | |
| 1484 | 1379 | static inline struct mem_cgroup *mem_cgroup_get_oom_group( |
| 1485 | 1380 | struct task_struct *victim, struct mem_cgroup *oom_domain) |
| 1486 | 1381 | { |
| @@ -1574,14 +1469,6 @@ void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) · 87 unmodified lines | ||
| 1574 | 1469 | static inline void split_page_memcg(struct page *head, int old_order, int new_order) |
| 1575 | 1470 | { |
| 1576 | 1471 | } |
| 1577 | − | |
| 1578 | −static inline | |
| 1579 | −unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, | |
| 1580 | − gfp_t gfp_mask, | |
| 1581 | − unsigned long *total_scanned) | |
| 1582 | −{ | |
| 1583 | − return 0; | |
| 1584 | −} | |
| 1585 | 1472 | #endif /* CONFIG_MEMCG */ |
| 1586 | 1473 | |
| 1587 | 1474 | /* |
| @@ -1589,7 +1476,7 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, · 1 unmodified line | ||
| 1589 | 1476 | * if MEMCG_DATA_OBJEXTS is set. |
| 1590 | 1477 | */ |
| 1591 | 1478 | struct slabobj_ext { |
| 1592 | −#ifdef CONFIG_MEMCG_KMEM | |
| 1479 | +#ifdef CONFIG_MEMCG | |
| 1593 | 1480 | struct obj_cgroup *objcg; |
| 1594 | 1481 | #endif |
| 1595 | 1482 | #ifdef CONFIG_MEM_ALLOC_PROFILING |
| @@ -1636,7 +1523,7 @@ static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec, · 40 unmodified lines | ||
| 1636 | 1523 | spin_unlock_irqrestore(&lruvec->lru_lock, flags); |
| 1637 | 1524 | } |
| 1638 | 1525 | |
| 1639 | −/* Test requires a stable page->memcg binding, see page_memcg() */ | |
| 1526 | +/* Test requires a stable folio->memcg binding, see folio_memcg() */ | |
| 1640 | 1527 | static inline bool folio_matches_lruvec(struct folio *folio, |
| 1641 | 1528 | struct lruvec *lruvec) |
| 1642 | 1529 | { |
| @@ -1734,8 +1621,10 @@ void mem_cgroup_sk_alloc(struct sock *sk); · 91 unmodified lines | ||
| 1734 | 1621 | void mem_cgroup_sk_free(struct sock *sk); |
| 1735 | 1622 | static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) |
| 1736 | 1623 | { |
| 1624 | +#ifdef CONFIG_MEMCG_V1 | |
| 1737 | 1625 | if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) |
| 1738 | 1626 | return !!memcg->tcpmem_pressure; |
| 1627 | +#endif /* CONFIG_MEMCG_V1 */ | |
| 1739 | 1628 | do { |
| 1740 | 1629 | if (time_before(jiffies, READ_ONCE(memcg->socket_pressure))) |
| 1741 | 1630 | return true; |
| @@ -1762,7 +1651,7 @@ static inline void set_shrinker_bit(struct mem_cgroup *memcg, · 20 unmodified lines | ||
| 1762 | 1651 | } |
| 1763 | 1652 | #endif |
| 1764 | 1653 | |
| 1765 | −#ifdef CONFIG_MEMCG_KMEM | |
| 1654 | +#ifdef CONFIG_MEMCG | |
| 1766 | 1655 | bool mem_cgroup_kmem_disabled(void); |
| 1767 | 1656 | int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); |
| 1768 | 1657 | void __memcg_kmem_uncharge_page(struct page *page, int order); |
| @@ -1905,9 +1794,9 @@ static inline void count_objcg_event(struct obj_cgroup *objcg, · 136 unmodified lines | ||
| 1905 | 1794 | { |
| 1906 | 1795 | } |
| 1907 | 1796 | |
| 1908 | −#endif /* CONFIG_MEMCG_KMEM */ | |
| 1797 | +#endif /* CONFIG_MEMCG */ | |
| 1909 | 1798 | |
| 1910 | −#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) | |
| 1799 | +#if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP) | |
| 1911 | 1800 | bool obj_cgroup_may_zswap(struct obj_cgroup *objcg); |
| 1912 | 1801 | void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size); |
| 1913 | 1802 | void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size); |
| @@ -1932,4 +1821,100 @@ static inline bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg) · 18 unmodified lines | ||
| 1932 | 1821 | } |
| 1933 | 1822 | #endif |
| 1934 | 1823 | |
| 1824 | + | |
| 1825 | +/* Cgroup v1-related declarations */ | |
| 1826 | + | |
| 1827 | +#ifdef CONFIG_MEMCG_V1 | |
| 1828 | +unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order, | |
| 1829 | + gfp_t gfp_mask, | |
| 1830 | + unsigned long *total_scanned); | |
| 1831 | + | |
| 1832 | +bool mem_cgroup_oom_synchronize(bool wait); | |
| 1833 | + | |
| 1834 | +static inline bool task_in_memcg_oom(struct task_struct *p) | |
| 1835 | +{ | |
| 1836 | + return p->memcg_in_oom; | |
| 1837 | +} | |
| 1838 | + | |
| 1839 | +void folio_memcg_lock(struct folio *folio); | |
| 1840 | +void folio_memcg_unlock(struct folio *folio); | |
| 1841 | + | |
| 1842 | +/* try to stablize folio_memcg() for all the pages in a memcg */ | |
| 1843 | +static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg) | |
| 1844 | +{ | |
| 1845 | + rcu_read_lock(); | |
| 1846 | + | |
| 1847 | + if (mem_cgroup_disabled() || !atomic_read(&memcg->moving_account)) | |
| 1848 | + return true; | |
| 1849 | + | |
| 1850 | + rcu_read_unlock(); | |
| 1851 | + return false; | |
| 1852 | +} | |
| 1853 | + | |
| 1854 | +static inline void mem_cgroup_unlock_pages(void) | |
| 1855 | +{ | |
| 1856 | + rcu_read_unlock(); | |
| 1857 | +} | |
| 1858 | + | |
| 1859 | +static inline void mem_cgroup_enter_user_fault(void) | |
| 1860 | +{ | |
| 1861 | + WARN_ON(current->in_user_fault); | |
| 1862 | + current->in_user_fault = 1; | |
| 1863 | +} | |
| 1864 | + | |
| 1865 | +static inline void mem_cgroup_exit_user_fault(void) | |
| 1866 | +{ | |
| 1867 | + WARN_ON(!current->in_user_fault); | |
| 1868 | + current->in_user_fault = 0; | |
| 1869 | +} | |
| 1870 | + | |
| 1871 | +#else /* CONFIG_MEMCG_V1 */ | |
| 1872 | +static inline | |
| 1873 | +unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order, | |
| 1874 | + gfp_t gfp_mask, | |
| 1875 | + unsigned long *total_scanned) | |
| 1876 | +{ | |
| 1877 | + return 0; | |
| 1878 | +} | |
| 1879 | + | |
| 1880 | +static inline void folio_memcg_lock(struct folio *folio) | |
| 1881 | +{ | |
| 1882 | +} | |
| 1883 | + | |
| 1884 | +static inline void folio_memcg_unlock(struct folio *folio) | |
| 1885 | +{ | |
| 1886 | +} | |
| 1887 | + | |
| 1888 | +static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg) | |
| 1889 | +{ | |
| 1890 | + /* to match folio_memcg_rcu() */ | |
| 1891 | + rcu_read_lock(); | |
| 1892 | + return true; | |
| 1893 | +} | |
| 1894 | + | |
| 1895 | +static inline void mem_cgroup_unlock_pages(void) | |
| 1896 | +{ | |
| 1897 | + rcu_read_unlock(); | |
| 1898 | +} | |
| 1899 | + | |
| 1900 | +static inline bool task_in_memcg_oom(struct task_struct *p) | |
| 1901 | +{ | |
| 1902 | + return false; | |
| 1903 | +} | |
| 1904 | + | |
| 1905 | +static inline bool mem_cgroup_oom_synchronize(bool wait) | |
| 1906 | +{ | |
| 1907 | + return false; | |
| 1908 | +} | |
| 1909 | + | |
| 1910 | +static inline void mem_cgroup_enter_user_fault(void) | |
| 1911 | +{ | |
| 1912 | +} | |
| 1913 | + | |
| 1914 | +static inline void mem_cgroup_exit_user_fault(void) | |
| 1915 | +{ | |
| 1916 | +} | |
| 1917 | + | |
| 1918 | +#endif /* CONFIG_MEMCG_V1 */ | |
| 1919 | + | |
| 1935 | 1920 | #endif /* _LINUX_MEMCONTROL_H */ |
| @@ -6,11 +6,16 @@ · 5 unmodified lines | ||
| 6 | 6 | |
| 7 | 7 | #ifdef CONFIG_MEMFD_CREATE |
| 8 | 8 | extern long memfd_fcntl(struct file *file, unsigned int cmd, unsigned int arg); |
| 9 | +struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx); | |
| 9 | 10 | #else |
| 10 | 11 | static inline long memfd_fcntl(struct file *f, unsigned int c, unsigned int a) |
| 11 | 12 | { |
| 12 | 13 | return -EINVAL; |
| 13 | 14 | } |
| 15 | +static inline struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) | |
| 16 | +{ | |
| 17 | + return ERR_PTR(-EINVAL); | |
| 18 | +} | |
| 14 | 19 | #endif |
| 15 | 20 | |
| 16 | 21 | #endif /* __LINUX_MEMFD_H */ |
| @@ -38,6 +38,7 @@ struct access_coordinate; · 37 unmodified lines | ||
| 38 | 38 | #ifdef CONFIG_NUMA |
| 39 | 39 | extern bool numa_demotion_enabled; |
| 40 | 40 | extern struct memory_dev_type *default_dram_type; |
| 41 | +extern nodemask_t default_dram_nodes; | |
| 41 | 42 | struct memory_dev_type *alloc_memory_type(int adistance); |
| 42 | 43 | void put_memory_type(struct memory_dev_type *memtype); |
| 43 | 44 | void init_node_memory_type(int node, struct memory_dev_type *default_type); |
| @@ -76,6 +77,7 @@ static inline bool node_is_toptier(int node) · 32 unmodified lines | ||
| 76 | 77 | |
| 77 | 78 | #define numa_demotion_enabled false |
| 78 | 79 | #define default_dram_type NULL |
| 80 | +#define default_dram_nodes NODE_MASK_NONE | |
| 79 | 81 | /* |
| 80 | 82 | * CONFIG_NUMA implementation returns non NULL error. |
| 81 | 83 | */ |
| @@ -175,8 +175,8 @@ extern int mhp_init_memmap_on_memory(unsigned long pfn, unsigned long nr_pages, · 174 unmodified lines | ||
| 175 | 175 | extern void mhp_deinit_memmap_on_memory(unsigned long pfn, unsigned long nr_pages); |
| 176 | 176 | extern int online_pages(unsigned long pfn, unsigned long nr_pages, |
| 177 | 177 | struct zone *zone, struct memory_group *group); |
| 178 | −extern void __offline_isolated_pages(unsigned long start_pfn, | |
| 179 | − unsigned long end_pfn); | |
| 178 | +extern unsigned long __offline_isolated_pages(unsigned long start_pfn, | |
| 179 | + unsigned long end_pfn); | |
| 180 | 180 | |
| 181 | 181 | typedef void (*online_page_callback_t)(struct page *page, unsigned int order); |
| 182 | 182 | |
| @@ -63,8 +63,6 @@ extern const char *migrate_reason_names[MR_TYPES]; · 62 unmodified lines | ||
| 63 | 63 | #ifdef CONFIG_MIGRATION |
| 64 | 64 | |
| 65 | 65 | void putback_movable_pages(struct list_head *l); |
| 66 | −int migrate_folio_extra(struct address_space *mapping, struct folio *dst, | |
| 67 | − struct folio *src, enum migrate_mode mode, int extra_count); | |
| 68 | 66 | int migrate_folio(struct address_space *mapping, struct folio *dst, |
| 69 | 67 | struct folio *src, enum migrate_mode mode); |
| 70 | 68 | int migrate_pages(struct list_head *l, new_folio_t new, free_folio_t free, |
| @@ -78,7 +76,6 @@ int migrate_huge_page_move_mapping(struct address_space *mapping, · 7 unmodified lines | ||
| 78 | 76 | void migration_entry_wait_on_locked(swp_entry_t entry, spinlock_t *ptl) |
| 79 | 77 | __releases(ptl); |
| 80 | 78 | void folio_migrate_flags(struct folio *newfolio, struct folio *folio); |
| 81 | −void folio_migrate_copy(struct folio *newfolio, struct folio *folio); | |
| 82 | 79 | int folio_migrate_mapping(struct address_space *mapping, |
| 83 | 80 | struct folio *newfolio, struct folio *folio, int extra_count); |
| 84 | 81 | |
| @@ -142,9 +139,16 @@ const struct movable_operations *page_movable_ops(struct page *page) · 57 unmodified lines | ||
| 142 | 139 | } |
| 143 | 140 | |
| 144 | 141 | #ifdef CONFIG_NUMA_BALANCING |
| 142 | +int migrate_misplaced_folio_prepare(struct folio *folio, | |
| 143 | + struct vm_area_struct *vma, int node); | |
| 145 | 144 | int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma, |
| 146 | 145 | int node); |
| 147 | 146 | #else |
| 147 | +static inline int migrate_misplaced_folio_prepare(struct folio *folio, | |
| 148 | + struct vm_area_struct *vma, int node) | |
| 149 | +{ | |
| 150 | + return -EAGAIN; /* can't migrate now */ | |
| 151 | +} | |
| 148 | 152 | static inline int migrate_misplaced_folio(struct folio *folio, |
| 149 | 153 | struct vm_area_struct *vma, int node) |
| 150 | 154 | { |
| @@ -7,16 +7,11 @@ · 6 unmodified lines | ||
| 7 | 7 | * on most operations but not ->writepage as the potential stall time |
| 8 | 8 | * is too significant |
| 9 | 9 | * MIGRATE_SYNC will block when migrating pages |
| 10 | − * MIGRATE_SYNC_NO_COPY will block when migrating pages but will not copy pages | |
| 11 | − * with the CPU. Instead, page copy happens outside the migratepage() | |
| 12 | − * callback and is likely using a DMA engine. See migrate_vma() and HMM | |
| 13 | − * (mm/hmm.c) for users of this mode. | |
| 14 | 10 | */ |
| 15 | 11 | enum migrate_mode { |
| 16 | 12 | MIGRATE_ASYNC, |
| 17 | 13 | MIGRATE_SYNC_LIGHT, |
| 18 | 14 | MIGRATE_SYNC, |
| 19 | − MIGRATE_SYNC_NO_COPY, | |
| 20 | 15 | }; |
| 21 | 16 | |
| 22 | 17 | enum migrate_reason { |
| @@ -29,6 +24,7 @@ enum migrate_reason { · 6 unmodified lines | ||
| 29 | 24 | MR_CONTIG_RANGE, |
| 30 | 25 | MR_LONGTERM_PIN, |
| 31 | 26 | MR_DEMOTION, |
| 27 | + MR_DAMON, | |
| 32 | 28 | MR_TYPES |
| 33 | 29 | }; |
| 34 | 30 | |
| @@ -1202,8 +1202,7 @@ static inline int is_vmalloc_or_module_addr(const void *x) · 1201 unmodified lines | ||
| 1202 | 1202 | /* |
| 1203 | 1203 | * How many times the entire folio is mapped as a single unit (eg by a |
| 1204 | 1204 | * PMD or PUD entry). This is probably not what you want, except for |
| 1205 | − * debugging purposes - it does not include PTE-mapped sub-pages; look | |
| 1206 | − * at folio_mapcount() or page_mapcount() instead. | |
| 1205 | + * debugging purposes or implementation of other core folio_*() primitives. | |
| 1207 | 1206 | */ |
| 1208 | 1207 | static inline int folio_entire_mapcount(const struct folio *folio) |
| 1209 | 1208 | { |
| @@ -1211,40 +1210,6 @@ static inline int folio_entire_mapcount(const struct folio *folio) · 1 unmodified line | ||
| 1211 | 1210 | return atomic_read(&folio->_entire_mapcount) + 1; |
| 1212 | 1211 | } |
| 1213 | 1212 | |
| 1214 | −/* | |
| 1215 | − * The atomic page->_mapcount, starts from -1: so that transitions | |
| 1216 | − * both from it and to it can be tracked, using atomic_inc_and_test | |
| 1217 | − * and atomic_add_negative(-1). | |
| 1218 | − */ | |
| 1219 | −static inline void page_mapcount_reset(struct page *page) | |
| 1220 | −{ | |
| 1221 | − atomic_set(&(page)->_mapcount, -1); | |
| 1222 | −} | |
| 1223 | − | |
| 1224 | −/** | |
| 1225 | − * page_mapcount() - Number of times this precise page is mapped. | |
| 1226 | − * @page: The page. | |
| 1227 | − * | |
| 1228 | − * The number of times this page is mapped. If this page is part of | |
| 1229 | − * a large folio, it includes the number of times this page is mapped | |
| 1230 | − * as part of that folio. | |
| 1231 | − * | |
| 1232 | − * Will report 0 for pages which cannot be mapped into userspace, eg | |
| 1233 | − * slab, page tables and similar. | |
| 1234 | − */ | |
| 1235 | −static inline int page_mapcount(struct page *page) | |
| 1236 | −{ | |
| 1237 | − int mapcount = atomic_read(&page->_mapcount) + 1; | |
| 1238 | − | |
| 1239 | − /* Handle page_has_type() pages */ | |
| 1240 | − if (mapcount < PAGE_MAPCOUNT_RESERVE + 1) | |
| 1241 | − mapcount = 0; | |
| 1242 | − if (unlikely(PageCompound(page))) | |
| 1243 | − mapcount += folio_entire_mapcount(page_folio(page)); | |
| 1244 | − | |
| 1245 | − return mapcount; | |
| 1246 | −} | |
| 1247 | − | |
| 1248 | 1213 | static inline int folio_large_mapcount(const struct folio *folio) |
| 1249 | 1214 | { |
| 1250 | 1215 | VM_WARN_ON_FOLIO(!folio_test_large(folio), folio); |
| @@ -1326,6 +1291,7 @@ void put_pages_list(struct list_head *pages); · 75 unmodified lines | ||
| 1326 | 1291 | |
| 1327 | 1292 | void split_page(struct page *page, unsigned int order); |
| 1328 | 1293 | void folio_copy(struct folio *dst, struct folio *src); |
| 1294 | +int folio_mc_copy(struct folio *dst, struct folio *src); | |
| 1329 | 1295 | |
| 1330 | 1296 | unsigned long nr_free_buffer_pages(void); |
| 1331 | 1297 | |
| @@ -1612,17 +1578,19 @@ static inline void put_page(struct page *page) · 280 unmodified lines | ||
| 1612 | 1578 | * issue. |
| 1613 | 1579 | * |
| 1614 | 1580 | * Locking: the lockless algorithm described in folio_try_get_rcu() |
| 1615 | − * provides safe operation for get_user_pages(), page_mkclean() and | |
| 1581 | + * provides safe operation for get_user_pages(), folio_mkclean() and | |
| 1616 | 1582 | * other calls that race to set up page table entries. |
| 1617 | 1583 | */ |
| 1618 | 1584 | #define GUP_PIN_COUNTING_BIAS (1U << 10) |
| 1619 | 1585 | |
| 1620 | 1586 | void unpin_user_page(struct page *page); |
| 1587 | +void unpin_folio(struct folio *folio); | |
| 1621 | 1588 | void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, |
| 1622 | 1589 | bool make_dirty); |
| 1623 | 1590 | void unpin_user_page_range_dirty_lock(struct page *page, unsigned long npages, |
| 1624 | 1591 | bool make_dirty); |
| 1625 | 1592 | void unpin_user_pages(struct page **pages, unsigned long npages); |
| 1593 | +void unpin_folios(struct folio **folios, unsigned long nfolios); | |
| 1626 | 1594 | |
| 1627 | 1595 | static inline bool is_cow_mapping(vm_flags_t flags) |
| 1628 | 1596 | { |
| @@ -1953,8 +1921,8 @@ static inline struct folio *pfn_folio(unsigned long pfn) · 324 unmodified lines | ||
| 1953 | 1921 | * |
| 1954 | 1922 | * For more information, please see Documentation/core-api/pin_user_pages.rst. |
| 1955 | 1923 | * |
| 1956 | − * Return: True, if it is likely that the page has been "dma-pinned". | |
| 1957 | − * False, if the page is definitely not dma-pinned. | |
| 1924 | + * Return: True, if it is likely that the folio has been "dma-pinned". | |
| 1925 | + * False, if the folio is definitely not dma-pinned. | |
| 1958 | 1926 | */ |
| 1959 | 1927 | static inline bool folio_maybe_dma_pinned(struct folio *folio) |
| 1960 | 1928 | { |
| @@ -1973,11 +1941,6 @@ static inline bool folio_maybe_dma_pinned(struct folio *folio) · 12 unmodified lines | ||
| 1973 | 1941 | GUP_PIN_COUNTING_BIAS; |
| 1974 | 1942 | } |
| 1975 | 1943 | |
| 1976 | −static inline bool page_maybe_dma_pinned(struct page *page) | |
| 1977 | −{ | |
| 1978 | − return folio_maybe_dma_pinned(page_folio(page)); | |
| 1979 | −} | |
| 1980 | − | |
| 1981 | 1944 | /* |
| 1982 | 1945 | * This should most likely only be called during fork() to see whether we |
| 1983 | 1946 | * should break the cow immediately for an anon page on the src mm. |
| @@ -2295,19 +2258,6 @@ static inline void *folio_address(const struct folio *folio) · 311 unmodified lines | ||
| 2295 | 2258 | return page_address(&folio->page); |
| 2296 | 2259 | } |
| 2297 | 2260 | |
| 2298 | −extern pgoff_t __page_file_index(struct page *page); | |
| 2299 | − | |
| 2300 | −/* | |
| 2301 | − * Return the pagecache index of the passed page. Regular pagecache pages | |
| 2302 | − * use ->index whereas swapcache pages use swp_offset(->private) | |
| 2303 | − */ | |
| 2304 | −static inline pgoff_t page_index(struct page *page) | |
| 2305 | −{ | |
| 2306 | − if (unlikely(PageSwapCache(page))) | |
| 2307 | − return __page_file_index(page); | |
| 2308 | − return page->index; | |
| 2309 | −} | |
| 2310 | − | |
| 2311 | 2261 | /* |
| 2312 | 2262 | * Return true only if the page has been allocated with |
| 2313 | 2263 | * ALLOC_NO_WATERMARKS and the low watermark was not |
| @@ -2550,6 +2500,9 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, · 236 unmodified lines | ||
| 2550 | 2500 | struct page **pages, unsigned int gup_flags); |
| 2551 | 2501 | long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, |
| 2552 | 2502 | struct page **pages, unsigned int gup_flags); |
| 2503 | +long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, | |
| 2504 | + struct folio **folios, unsigned int max_folios, | |
| 2505 | + pgoff_t *offset); | |
| 2553 | 2506 | |
| 2554 | 2507 | int get_user_pages_fast(unsigned long start, int nr_pages, |
| 2555 | 2508 | unsigned int gup_flags, struct page **pages); |
| @@ -4038,7 +3991,6 @@ extern int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, · 1482 unmodified lines | ||
| 4038 | 3991 | bool *migratable_cleared); |
| 4039 | 3992 | void num_poisoned_pages_inc(unsigned long pfn); |
| 4040 | 3993 | void num_poisoned_pages_sub(unsigned long pfn, long i); |
| 4041 | −struct task_struct *task_early_kill(struct task_struct *tsk, int force_early); | |
| 4042 | 3994 | #else |
| 4043 | 3995 | static inline void memory_failure_queue(unsigned long pfn, int flags) |
| 4044 | 3996 | { |
| @@ -4059,12 +4011,6 @@ static inline void num_poisoned_pages_sub(unsigned long pfn, long i) · 14 unmodified lines | ||
| 4059 | 4011 | } |
| 4060 | 4012 | #endif |
| 4061 | 4013 | |
| 4062 | −#if defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_KSM) | |
| 4063 | −void add_to_kill_ksm(struct task_struct *tsk, struct page *p, | |
| 4064 | − struct vm_area_struct *vma, struct list_head *to_kill, | |
| 4065 | − unsigned long ksm_addr); | |
| 4066 | −#endif | |
| 4067 | − | |
| 4068 | 4014 | #if defined(CONFIG_MEMORY_FAILURE) && defined(CONFIG_MEMORY_HOTPLUG) |
| 4069 | 4015 | extern void memblk_nr_poison_inc(unsigned long pfn); |
| 4070 | 4016 | extern void memblk_nr_poison_sub(unsigned long pfn, long i); |
| @@ -4105,10 +4051,10 @@ enum mf_result { · 34 unmodified lines | ||
| 4105 | 4051 | enum mf_action_page_type { |
| 4106 | 4052 | MF_MSG_KERNEL, |
| 4107 | 4053 | MF_MSG_KERNEL_HIGH_ORDER, |
| 4108 | − MF_MSG_SLAB, | |
| 4109 | 4054 | MF_MSG_DIFFERENT_COMPOUND, |
| 4110 | 4055 | MF_MSG_HUGE, |
| 4111 | 4056 | MF_MSG_FREE_HUGE, |
| 4057 | + MF_MSG_GET_HWPOISON, | |
| 4112 | 4058 | MF_MSG_UNMAP_FAILED, |
| 4113 | 4059 | MF_MSG_DIRTY_SWAPCACHE, |
| 4114 | 4060 | MF_MSG_CLEAN_SWAPCACHE, |
| @@ -4122,13 +4068,12 @@ enum mf_action_page_type { · 7 unmodified lines | ||
| 4122 | 4068 | MF_MSG_BUDDY, |
| 4123 | 4069 | MF_MSG_DAX, |
| 4124 | 4070 | MF_MSG_UNSPLIT_THP, |
| 4071 | + MF_MSG_ALREADY_POISONED, | |
| 4125 | 4072 | MF_MSG_UNKNOWN, |
| 4126 | 4073 | }; |
| 4127 | 4074 | |
| 4128 | 4075 | #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) |
| 4129 | −extern void clear_huge_page(struct page *page, | |
| 4130 | − unsigned long addr_hint, | |
| 4131 | − unsigned int pages_per_huge_page); | |
| 4076 | +void folio_zero_user(struct folio *folio, unsigned long addr_hint); | |
| 4132 | 4077 | int copy_user_large_folio(struct folio *dst, struct folio *src, |
| 4133 | 4078 | unsigned long addr_hint, |
| 4134 | 4079 | struct vm_area_struct *vma); |
| @@ -46,9 +46,7 @@ struct mem_cgroup; · 45 unmodified lines | ||
| 46 | 46 | * which is guaranteed to be aligned. If you use the same storage as |
| 47 | 47 | * page->mapping, you must restore it to NULL before freeing the page. |
| 48 | 48 | * |
| 49 | − * If your page will not be mapped to userspace, you can also use the four | |
| 50 | − * bytes in the mapcount union, but you must call page_mapcount_reset() | |
| 51 | − * before freeing it. | |
| 49 | + * The mapcount field must not be used for own purposes. | |
| 52 | 50 | * |
| 53 | 51 | * If you want to use the refcount field, it must be used in such a way |
| 54 | 52 | * that other CPUs temporarily incrementing and then decrementing the |
| @@ -152,18 +150,31 @@ struct page { · 97 unmodified lines | ||
| 152 | 150 | |
| 153 | 151 | union { /* This union is 4 bytes in size. */ |
| 154 | 152 | /* |
| 155 | − * If the page can be mapped to userspace, encodes the number | |
| 156 | − * of times this page is referenced by a page table. | |
| 153 | + * For head pages of typed folios, the value stored here | |
| 154 | + * allows for determining what this page is used for. The | |
| 155 | + * tail pages of typed folios will not store a type | |
| 156 | + * (page_type == _mapcount == -1). | |
| 157 | + * | |
| 158 | + * See page-flags.h for a list of page types which are currently | |
| 159 | + * stored here. | |
| 160 | + * | |
| 161 | + * Owners of typed folios may reuse the lower 16 bit of the | |
| 162 | + * head page page_type field after setting the page type, | |
| 163 | + * but must reset these 16 bit to -1 before clearing the | |
| 164 | + * page type. | |
| 157 | 165 | */ |
| 158 | − atomic_t _mapcount; | |
| 166 | + unsigned int page_type; | |
| 159 | 167 | |
| 160 | 168 | /* |
| 161 | − * If the page is neither PageSlab nor mappable to userspace, | |
| 162 | − * the value stored here may help determine what this page | |
| 163 | − * is used for. See page-flags.h for a list of page types | |
| 164 | − * which are currently stored here. | |
| 169 | + * For pages that are part of non-typed folios for which mappings | |
| 170 | + * are tracked via the RMAP, encodes the number of times this page | |
| 171 | + * is directly referenced by a page table. | |
| 172 | + * | |
| 173 | + * Note that the mapcount is always initialized to -1, so that | |
| 174 | + * transitions both from it and to it can be tracked, using | |
| 175 | + * atomic_inc_and_test() and atomic_add_negative(-1). | |
| 165 | 176 | */ |
| 166 | − unsigned int page_type; | |
| 177 | + atomic_t _mapcount; | |
| 167 | 178 | }; |
| 168 | 179 | |
| 169 | 180 | /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ |
| @@ -220,6 +220,8 @@ enum node_stat_item { · 219 unmodified lines | ||
| 220 | 220 | PGDEMOTE_KSWAPD, |
| 221 | 221 | PGDEMOTE_DIRECT, |
| 222 | 222 | PGDEMOTE_KHUGEPAGED, |
| 223 | + NR_MEMMAP, /* page metadata allocated through buddy allocator */ | |
| 224 | + NR_MEMMAP_BOOT, /* page metadata allocated through boot allocator */ | |
| 223 | 225 | NR_VM_NODE_STAT_ITEMS |
| 224 | 226 | }; |
| 225 | 227 | |
| @@ -30,16 +30,11 @@ · 29 unmodified lines | ||
| 30 | 30 | * - Pages falling into physical memory gaps - not IORESOURCE_SYSRAM. Trying |
| 31 | 31 | * to read/write these pages might end badly. Don't touch! |
| 32 | 32 | * - The zero page(s) |
| 33 | − * - Pages not added to the page allocator when onlining a section because | |
| 34 | − * they were excluded via the online_page_callback() or because they are | |
| 35 | − * PG_hwpoison. | |
| 36 | 33 | * - Pages allocated in the context of kexec/kdump (loaded kernel image, |
| 37 | 34 | * control pages, vmcoreinfo) |
| 38 | 35 | * - MMIO/DMA pages. Some architectures don't allow to ioremap pages that are |
| 39 | 36 | * not marked PG_reserved (as they might be in use by somebody else who does |
| 40 | 37 | * not respect the caching strategy). |
| 41 | − * - Pages part of an offline section (struct pages of offline sections should | |
| 42 | − * not be trusted as they will be initialized when first onlined). | |
| 43 | 38 | * - MCA pages on ia64 |
| 44 | 39 | * - Pages holding CPU notes for POWER Firmware Assisted Dump |
| 45 | 40 | * - Device memory (e.g. PMEM, DAX, HMM) |
| @@ -616,11 +611,6 @@ PAGEFLAG_FALSE(Uncached, uncached) · 570 unmodified lines | ||
| 616 | 611 | PAGEFLAG(HWPoison, hwpoison, PF_ANY) |
| 617 | 612 | TESTSCFLAG(HWPoison, hwpoison, PF_ANY) |
| 618 | 613 | #define __PG_HWPOISON (1UL << PG_hwpoison) |
| 619 | −#define MAGIC_HWPOISON 0x48575053U /* HWPS */ | |
| 620 | −extern void SetPageHWPoisonTakenOff(struct page *page); | |
| 621 | −extern void ClearPageHWPoisonTakenOff(struct page *page); | |
| 622 | −extern bool take_page_off_buddy(struct page *page); | |
| 623 | −extern bool put_page_back_buddy(struct page *page); | |
| 624 | 614 | #else |
| 625 | 615 | PAGEFLAG_FALSE(HWPoison, hwpoison) |
| 626 | 616 | #define __PG_HWPOISON 0 |
| @@ -655,27 +645,28 @@ PAGEFLAG_FALSE(VmemmapSelfHosted, vmemmap_self_hosted) · 28 unmodified lines | ||
| 655 | 645 | #endif |
| 656 | 646 | |
| 657 | 647 | /* |
| 658 | − * On an anonymous page mapped into a user virtual memory area, | |
| 659 | − * page->mapping points to its anon_vma, not to a struct address_space; | |
| 648 | + * On an anonymous folio mapped into a user virtual memory area, | |
| 649 | + * folio->mapping points to its anon_vma, not to a struct address_space; | |
| 660 | 650 | * with the PAGE_MAPPING_ANON bit set to distinguish it. See rmap.h. |
| 661 | 651 | * |
| 662 | 652 | * On an anonymous page in a VM_MERGEABLE area, if CONFIG_KSM is enabled, |
| 663 | 653 | * the PAGE_MAPPING_MOVABLE bit may be set along with the PAGE_MAPPING_ANON |
| 664 | − * bit; and then page->mapping points, not to an anon_vma, but to a private | |
| 654 | + * bit; and then folio->mapping points, not to an anon_vma, but to a private | |
| 665 | 655 | * structure which KSM associates with that merged page. See ksm.h. |
| 666 | 656 | * |
| 667 | 657 | * PAGE_MAPPING_KSM without PAGE_MAPPING_ANON is used for non-lru movable |
| 668 | − * page and then page->mapping points to a struct movable_operations. | |
| 658 | + * page and then folio->mapping points to a struct movable_operations. | |
| 669 | 659 | * |
| 670 | − * Please note that, confusingly, "page_mapping" refers to the inode | |
| 671 | − * address_space which maps the page from disk; whereas "page_mapped" | |
| 672 | − * refers to user virtual address space into which the page is mapped. | |
| 660 | + * Please note that, confusingly, "folio_mapping" refers to the inode | |
| 661 | + * address_space which maps the folio from disk; whereas "folio_mapped" | |
| 662 | + * refers to user virtual address space into which the folio is mapped. | |
| 673 | 663 | * |
| 674 | 664 | * For slab pages, since slab reuses the bits in struct page to store its |
| 675 | − * internal states, the page->mapping does not exist as such, nor do these | |
| 676 | − * flags below. So in order to avoid testing non-existent bits, please | |
| 677 | − * make sure that PageSlab(page) actually evaluates to false before calling | |
| 678 | − * the following functions (e.g., PageAnon). See mm/slab.h. | |
| 665 | + * internal states, the folio->mapping does not exist as such, nor do | |
| 666 | + * these flags below. So in order to avoid testing non-existent bits, | |
| 667 | + * please make sure that folio_test_slab(folio) actually evaluates to | |
| 668 | + * false before calling the following functions (e.g., folio_test_anon). | |
| 669 | + * See mm/slab.h. | |
| 679 | 670 | */ |
| 680 | 671 | #define PAGE_MAPPING_ANON 0x1 |
| 681 | 672 | #define PAGE_MAPPING_MOVABLE 0x2 |
| @@ -945,22 +936,28 @@ PAGEFLAG_FALSE(HasHWPoisoned, has_hwpoisoned) · 263 unmodified lines | ||
| 945 | 936 | */ |
| 946 | 937 | |
| 947 | 938 | enum pagetype { |
| 948 | − PG_buddy = 0x00000080, | |
| 949 | − PG_offline = 0x00000100, | |
| 950 | − PG_table = 0x00000200, | |
| 951 | − PG_guard = 0x00000400, | |
| 952 | − PG_hugetlb = 0x00000800, | |
| 953 | − PG_slab = 0x00001000, | |
| 954 | − | |
| 955 | − PAGE_TYPE_BASE = 0xf0000000, | |
| 956 | − /* Reserve 0x0000007f to catch underflows of _mapcount */ | |
| 957 | − PAGE_MAPCOUNT_RESERVE = -128, | |
| 939 | + PG_buddy = 0x40000000, | |
| 940 | + PG_offline = 0x20000000, | |
| 941 | + PG_table = 0x10000000, | |
| 942 | + PG_guard = 0x08000000, | |
| 943 | + PG_hugetlb = 0x04000000, | |
| 944 | + PG_slab = 0x02000000, | |
| 945 | + PG_zsmalloc = 0x01000000, | |
| 946 | + | |
| 947 | + PAGE_TYPE_BASE = 0x80000000, | |
| 948 | + | |
| 949 | + /* | |
| 950 | + * Reserve 0xffff0000 - 0xfffffffe to catch _mapcount underflows and | |
| 951 | + * allow owners that set a type to reuse the lower 16 bit for their own | |
| 952 | + * purposes. | |
| 953 | + */ | |
| 954 | + PAGE_MAPCOUNT_RESERVE = ~0x0000ffff, | |
| 958 | 955 | }; |
| 959 | 956 | |
| 960 | 957 | #define PageType(page, flag) \ |
| 961 | − ((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE) | |
| 958 | + ((READ_ONCE(page->page_type) & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE) | |
| 962 | 959 | #define folio_test_type(folio, flag) \ |
| 963 | − ((folio->page.page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE) | |
| 960 | + ((READ_ONCE(folio->page.page_type) & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE) | |
| 964 | 961 | |
| 965 | 962 | static inline int page_type_has_type(unsigned int page_type) |
| 966 | 963 | { |
| @@ -969,7 +966,7 @@ static inline int page_type_has_type(unsigned int page_type) · 2 unmodified lines | ||
| 969 | 966 | |
| 970 | 967 | static inline int page_has_type(const struct page *page) |
| 971 | 968 | { |
| 972 | − return page_type_has_type(page->page_type); | |
| 969 | + return page_type_has_type(READ_ONCE(page->page_type)); | |
| 973 | 970 | } |
| 974 | 971 | |
| 975 | 972 | #define FOLIO_TYPE_OPS(lname, fname) \ |
| @@ -1018,15 +1015,22 @@ PAGE_TYPE_OPS(Buddy, buddy, buddy) · 42 unmodified lines | ||
| 1018 | 1015 | * The content of these pages is effectively stale. Such pages should not |
| 1019 | 1016 | * be touched (read/write/dump/save) except by their owner. |
| 1020 | 1017 | * |
| 1018 | + * When a memory block gets onlined, all pages are initialized with a | |
| 1019 | + * refcount of 1 and PageOffline(). generic_online_page() will | |
| 1020 | + * take care of clearing PageOffline(). | |
| 1021 | + * | |
| 1021 | 1022 | * If a driver wants to allow to offline unmovable PageOffline() pages without |
| 1022 | 1023 | * putting them back to the buddy, it can do so via the memory notifier by |
| 1023 | 1024 | * decrementing the reference count in MEM_GOING_OFFLINE and incrementing the |
| 1024 | 1025 | * reference count in MEM_CANCEL_OFFLINE. When offlining, the PageOffline() |
| 1025 | − * pages (now with a reference count of zero) are treated like free pages, | |
| 1026 | − * allowing the containing memory block to get offlined. A driver that | |
| 1026 | + * pages (now with a reference count of zero) are treated like free (unmanaged) | |
| 1027 | + * pages, allowing the containing memory block to get offlined. A driver that | |
| 1027 | 1028 | * relies on this feature is aware that re-onlining the memory block will |
| 1028 | − * require to re-set the pages PageOffline() and not giving them to the | |
| 1029 | − * buddy via online_page_callback_t. | |
| 1029 | + * require not giving them to the buddy via generic_online_page(). | |
| 1030 | + * | |
| 1031 | + * Memory offlining code will not adjust the managed page count for any | |
| 1032 | + * PageOffline() pages, treating them like they were never exposed to the | |
| 1033 | + * buddy using generic_online_page(). | |
| 1030 | 1034 | * |
| 1031 | 1035 | * There are drivers that mark a page PageOffline() and expect there won't be |
| 1032 | 1036 | * any further access to page content. PFN walkers that read content of random |
| @@ -1070,6 +1074,8 @@ FOLIO_TYPE_OPS(hugetlb, hugetlb) · 37 unmodified lines | ||
| 1070 | 1074 | FOLIO_TEST_FLAG_FALSE(hugetlb) |
| 1071 | 1075 | #endif |
| 1072 | 1076 | |
| 1077 | +PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc) | |
| 1078 | + | |
| 1073 | 1079 | /** |
| 1074 | 1080 | * PageHuge - Determine if the page belongs to hugetlbfs |
| 1075 | 1081 | * @page: The page to test. |
| @@ -81,4 +81,8 @@ static inline void page_counter_reset_watermark(struct page_counter *counter) · 80 unmodified lines | ||
| 81 | 81 | counter->watermark = page_counter_read(counter); |
| 82 | 82 | } |
| 83 | 83 | |
| 84 | +void page_counter_calculate_protection(struct page_counter *root, | |
| 85 | + struct page_counter *counter, | |
| 86 | + bool recursive_protection); | |
| 87 | + | |
| 84 | 88 | #endif /* _LINUX_PAGE_COUNTER_H */ |
| @@ -434,7 +434,6 @@ static inline void filemap_nr_thps_dec(struct address_space *mapping) · 433 unmodified lines | ||
| 434 | 434 | #endif |
| 435 | 435 | } |
| 436 | 436 | |
| 437 | −struct address_space *page_mapping(struct page *); | |
| 438 | 437 | struct address_space *folio_mapping(struct folio *); |
| 439 | 438 | struct address_space *swapcache_mapping(struct folio *); |
| 440 | 439 | |
| @@ -800,7 +799,7 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping, · 359 unmodified lines | ||
| 800 | 799 | mapping_gfp_mask(mapping)); |
| 801 | 800 | } |
| 802 | 801 | |
| 803 | −#define swapcache_index(folio) __page_file_index(&(folio)->page) | |
| 802 | +extern pgoff_t __folio_swap_cache_index(struct folio *folio); | |
| 804 | 803 | |
| 805 | 804 | /** |
| 806 | 805 | * folio_index - File index of a folio. |
| @@ -815,9 +814,9 @@ static inline struct page *grab_cache_page_nowait(struct address_space *mapping, · 8 unmodified lines | ||
| 815 | 814 | */ |
| 816 | 815 | static inline pgoff_t folio_index(struct folio *folio) |
| 817 | 816 | { |
| 818 | − if (unlikely(folio_test_swapcache(folio))) | |
| 819 | − return swapcache_index(folio); | |
| 820 | − return folio->index; | |
| 817 | + if (unlikely(folio_test_swapcache(folio))) | |
| 818 | + return __folio_swap_cache_index(folio); | |
| 819 | + return folio->index; | |
| 821 | 820 | } |
| 822 | 821 | |
| 823 | 822 | /** |
| @@ -940,11 +939,6 @@ static inline loff_t page_offset(struct page *page) · 116 unmodified lines | ||
| 940 | 939 | return ((loff_t)page->index) << PAGE_SHIFT; |
| 941 | 940 | } |
| 942 | 941 | |
| 943 | −static inline loff_t page_file_offset(struct page *page) | |
| 944 | −{ | |
| 945 | − return ((loff_t)page_index(page)) << PAGE_SHIFT; | |
| 946 | −} | |
| 947 | − | |
| 948 | 942 | /** |
| 949 | 943 | * folio_pos - Returns the byte position of this folio in its file. |
| 950 | 944 | * @folio: The folio. |
| @@ -954,18 +948,6 @@ static inline loff_t folio_pos(struct folio *folio) · 3 unmodified lines | ||
| 954 | 948 | return page_offset(&folio->page); |
| 955 | 949 | } |
| 956 | 950 | |
| 957 | −/** | |
| 958 | − * folio_file_pos - Returns the byte position of this folio in its file. | |
| 959 | − * @folio: The folio. | |
| 960 | − * | |
| 961 | − * This differs from folio_pos() for folios which belong to a swap file. | |
| 962 | − * NFS is the only filesystem today which needs to use folio_file_pos(). | |
| 963 | − */ | |
| 964 | −static inline loff_t folio_file_pos(struct folio *folio) | |
| 965 | −{ | |
| 966 | − return page_file_offset(&folio->page); | |
| 967 | −} | |
| 968 | − | |
| 969 | 951 | /* |
| 970 | 952 | * Get the offset in PAGE_SIZE (even for hugetlb folios). |
| 971 | 953 | */ |
| @@ -1319,8 +1301,7 @@ void page_cache_sync_readahead(struct address_space *mapping, · 347 unmodified lines | ||
| 1319 | 1301 | * @mapping: address_space which holds the pagecache and I/O vectors |
| 1320 | 1302 | * @ra: file_ra_state which holds the readahead state |
| 1321 | 1303 | * @file: Used by the filesystem for authentication. |
| 1322 | − * @folio: The folio at @index which triggered the readahead call. | |
| 1323 | − * @index: Index of first page to be read. | |
| 1304 | + * @folio: The folio which triggered the readahead call. | |
| 1324 | 1305 | * @req_count: Total number of pages being read by the caller. |
| 1325 | 1306 | * |
| 1326 | 1307 | * page_cache_async_readahead() should be called when a page is used which |
| @@ -1331,9 +1312,9 @@ void page_cache_sync_readahead(struct address_space *mapping, · 4 unmodified lines | ||
| 1331 | 1312 | static inline |
| 1332 | 1313 | void page_cache_async_readahead(struct address_space *mapping, |
| 1333 | 1314 | struct file_ra_state *ra, struct file *file, |
| 1334 | − struct folio *folio, pgoff_t index, unsigned long req_count) | |
| 1315 | + struct folio *folio, unsigned long req_count) | |
| 1335 | 1316 | { |
| 1336 | − DEFINE_READAHEAD(ractl, file, ra, mapping, index); | |
| 1317 | + DEFINE_READAHEAD(ractl, file, ra, mapping, folio->index); | |
| 1337 | 1318 | page_cache_async_ra(&ractl, folio, req_count); |
| 1338 | 1319 | } |
| 1339 | 1320 | |
| @@ -475,6 +475,12 @@ do { \ · 474 unmodified lines | ||
| 475 | 475 | raw_cpu_cmpxchg(pcp, oval, nval); \ |
| 476 | 476 | }) |
| 477 | 477 | |
| 478 | +#define __this_cpu_try_cmpxchg(pcp, ovalp, nval) \ | |
| 479 | +({ \ | |
| 480 | + __this_cpu_preempt_check("try_cmpxchg"); \ | |
| 481 | + raw_cpu_try_cmpxchg(pcp, ovalp, nval); \ | |
| 482 | +}) | |
| 483 | + | |
| 478 | 484 | #define __this_cpu_sub(pcp, val) __this_cpu_add(pcp, -(typeof(pcp))(val)) |
| 479 | 485 | #define __this_cpu_inc(pcp) __this_cpu_add(pcp, 1) |
| 480 | 486 | #define __this_cpu_dec(pcp) __this_cpu_sub(pcp, 1) |
| @@ -15,7 +15,7 @@ extern struct page_ext_operations page_alloc_tagging_ops; · 14 unmodified lines | ||
| 15 | 15 | |
| 16 | 16 | static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) |
| 17 | 17 | { |
| 18 | − return (void *)page_ext + page_alloc_tagging_ops.offset; | |
| 18 | + return (union codetag_ref *)page_ext_data(page_ext, &page_alloc_tagging_ops); | |
| 19 | 19 | } |
| 20 | 20 | |
| 21 | 21 | static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) |
| @@ -71,6 +71,7 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) · 49 unmodified lines | ||
| 71 | 71 | static inline void pgalloc_tag_split(struct page *page, unsigned int nr) |
| 72 | 72 | { |
| 73 | 73 | int i; |
| 74 | + struct page_ext *first_page_ext; | |
| 74 | 75 | struct page_ext *page_ext; |
| 75 | 76 | union codetag_ref *ref; |
| 76 | 77 | struct alloc_tag *tag; |
| @@ -78,7 +79,7 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) · 1 unmodified line | ||
| 78 | 79 | if (!mem_alloc_profiling_enabled()) |
| 79 | 80 | return; |
| 80 | 81 | |
| 81 | − page_ext = page_ext_get(page); | |
| 82 | + first_page_ext = page_ext = page_ext_get(page); | |
| 82 | 83 | if (unlikely(!page_ext)) |
| 83 | 84 | return; |
| 84 | 85 | |
| @@ -94,7 +95,7 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) · 9 unmodified lines | ||
| 94 | 95 | page_ext = page_ext_next(page_ext); |
| 95 | 96 | } |
| 96 | 97 | out: |
| 97 | − page_ext_put(page_ext); | |
| 98 | + page_ext_put(first_page_ext); | |
| 98 | 99 | } |
| 99 | 100 | |
| 100 | 101 | static inline struct alloc_tag *pgalloc_tag_get(struct page *page) |
| @@ -729,13 +729,18 @@ static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, · 728 unmodified lines | ||
| 729 | 729 | * fault. This function updates TLB only, do nothing with cache or others. |
| 730 | 730 | * It is the difference with function update_mmu_cache. |
| 731 | 731 | */ |
| 732 | −#ifndef __HAVE_ARCH_UPDATE_MMU_TLB | |
| 732 | +#ifndef update_mmu_tlb_range | |
| 733 | +static inline void update_mmu_tlb_range(struct vm_area_struct *vma, | |
| 734 | + unsigned long address, pte_t *ptep, unsigned int nr) | |
| 735 | +{ | |
| 736 | +} | |
| 737 | +#endif | |
| 738 | + | |
| 733 | 739 | static inline void update_mmu_tlb(struct vm_area_struct *vma, |
| 734 | 740 | unsigned long address, pte_t *ptep) |
| 735 | 741 | { |
| 742 | + update_mmu_tlb_range(vma, address, ptep, 1); | |
| 736 | 743 | } |
| 737 | −#define __HAVE_ARCH_UPDATE_MMU_TLB | |
| 738 | −#endif | |
| 739 | 744 | |
| 740 | 745 | /* |
| 741 | 746 | * Some architectures may be able to avoid expensive synchronization |
| @@ -1084,6 +1089,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) · 342 unmodified lines | ||
| 1084 | 1089 | }) |
| 1085 | 1090 | |
| 1086 | 1091 | #ifndef __HAVE_ARCH_DO_SWAP_PAGE |
| 1092 | +static inline void arch_do_swap_page_nr(struct mm_struct *mm, | |
| 1093 | + struct vm_area_struct *vma, | |
| 1094 | + unsigned long addr, | |
| 1095 | + pte_t pte, pte_t oldpte, | |
| 1096 | + int nr) | |
| 1097 | +{ | |
| 1098 | + | |
| 1099 | +} | |
| 1100 | +#else | |
| 1087 | 1101 | /* |
| 1088 | 1102 | * Some architectures support metadata associated with a page. When a |
| 1089 | 1103 | * page is being swapped out, this metadata must be saved so it can be |
| @@ -1092,12 +1106,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) · 2 unmodified lines | ||
| 1092 | 1106 | * page as metadata for the page. arch_do_swap_page() can restore this |
| 1093 | 1107 | * metadata when a page is swapped back in. |
| 1094 | 1108 | */ |
| 1095 | −static inline void arch_do_swap_page(struct mm_struct *mm, | |
| 1096 | − struct vm_area_struct *vma, | |
| 1097 | − unsigned long addr, | |
| 1098 | − pte_t pte, pte_t oldpte) | |
| 1099 | −{ | |
| 1100 | − | |
| 1109 | +static inline void arch_do_swap_page_nr(struct mm_struct *mm, | |
| 1110 | + struct vm_area_struct *vma, | |
| 1111 | + unsigned long addr, | |
| 1112 | + pte_t pte, pte_t oldpte, | |
| 1113 | + int nr) | |
| 1114 | +{ | |
| 1115 | + for (int i = 0; i < nr; i++) { | |
| 1116 | + arch_do_swap_page(vma->vm_mm, vma, addr + i * PAGE_SIZE, | |
| 1117 | + pte_advance_pfn(pte, i), | |
| 1118 | + pte_advance_pfn(oldpte, i)); | |
| 1119 | + } | |
| 1101 | 1120 | } |
| 1102 | 1121 | #endif |
| 1103 | 1122 | |
| @@ -1888,9 +1907,12 @@ typedef unsigned int pgtbl_mod_mask; · 784 unmodified lines | ||
| 1888 | 1907 | #ifndef pmd_leaf_size |
| 1889 | 1908 | #define pmd_leaf_size(x) PMD_SIZE |
| 1890 | 1909 | #endif |
| 1910 | +#ifndef __pte_leaf_size | |
| 1891 | 1911 | #ifndef pte_leaf_size |
| 1892 | 1912 | #define pte_leaf_size(x) PAGE_SIZE |
| 1893 | 1913 | #endif |
| 1914 | +#define __pte_leaf_size(x,y) pte_leaf_size(y) | |
| 1915 | +#endif | |
| 1894 | 1916 | |
| 1895 | 1917 | /* |
| 1896 | 1918 | * We always define pmd_pfn for all archs as it's used in lots of generic |
| @@ -200,6 +200,9 @@ static inline void __folio_rmap_sanity_checks(struct folio *folio, · 199 unmodified lines | ||
| 200 | 200 | /* hugetlb folios are handled separately. */ |
| 201 | 201 | VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); |
| 202 | 202 | |
| 203 | + /* When (un)mapping zeropages, we should never touch ref+mapcount. */ | |
| 204 | + VM_WARN_ON_FOLIO(is_zero_folio(folio), folio); | |
| 205 | + | |
| 203 | 206 | /* |
| 204 | 207 | * TODO: we get driver-allocated folios that have nothing to do with |
| 205 | 208 | * the rmap using vm_insert_page(); therefore, we cannot assume that |
| @@ -241,7 +244,7 @@ void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, · 35 unmodified lines | ||
| 241 | 244 | void folio_add_anon_rmap_pmd(struct folio *, struct page *, |
| 242 | 245 | struct vm_area_struct *, unsigned long address, rmap_t flags); |
| 243 | 246 | void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, |
| 244 | − unsigned long address); | |
| 247 | + unsigned long address, rmap_t flags); | |
| 245 | 248 | void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, |
| 246 | 249 | struct vm_area_struct *); |
| 247 | 250 | #define folio_add_file_rmap_pte(folio, page, vma) \ |
| @@ -681,16 +684,6 @@ struct page_vma_mapped_walk { · 433 unmodified lines | ||
| 681 | 684 | unsigned int flags; |
| 682 | 685 | }; |
| 683 | 686 | |
| 684 | −#define DEFINE_PAGE_VMA_WALK(name, _page, _vma, _address, _flags) \ | |
| 685 | − struct page_vma_mapped_walk name = { \ | |
| 686 | − .pfn = page_to_pfn(_page), \ | |
| 687 | − .nr_pages = compound_nr(_page), \ | |
| 688 | − .pgoff = page_to_pgoff(_page), \ | |
| 689 | − .vma = _vma, \ | |
| 690 | − .address = _address, \ | |
| 691 | − .flags = _flags, \ | |
| 692 | − } | |
| 693 | − | |
| 694 | 687 | #define DEFINE_FOLIO_VMA_WALK(name, _folio, _vma, _address, _flags) \ |
| 695 | 688 | struct page_vma_mapped_walk name = { \ |
| 696 | 689 | .pfn = folio_pfn(_folio), \ |
| @@ -710,6 +703,30 @@ static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) · 13 unmodified lines | ||
| 710 | 703 | spin_unlock(pvmw->ptl); |
| 711 | 704 | } |
| 712 | 705 | |
| 706 | +/** | |
| 707 | + * page_vma_mapped_walk_restart - Restart the page table walk. | |
| 708 | + * @pvmw: Pointer to struct page_vma_mapped_walk. | |
| 709 | + * | |
| 710 | + * It restarts the page table walk when changes occur in the page | |
| 711 | + * table, such as splitting a PMD. Ensures that the PTL held during | |
| 712 | + * the previous walk is released and resets the state to allow for | |
| 713 | + * a new walk starting at the current address stored in pvmw->address. | |
| 714 | + */ | |
| 715 | +static inline void | |
| 716 | +page_vma_mapped_walk_restart(struct page_vma_mapped_walk *pvmw) | |
| 717 | +{ | |
| 718 | + WARN_ON_ONCE(!pvmw->pmd && !pvmw->pte); | |
| 719 | + | |
| 720 | + if (likely(pvmw->ptl)) | |
| 721 | + spin_unlock(pvmw->ptl); | |
| 722 | + else | |
| 723 | + WARN_ON_ONCE(1); | |
| 724 | + | |
| 725 | + pvmw->ptl = NULL; | |
| 726 | + pvmw->pmd = NULL; | |
| 727 | + pvmw->pte = NULL; | |
| 728 | +} | |
| 729 | + | |
| 713 | 730 | bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw); |
| 714 | 731 | |
| 715 | 732 | /* |
| @@ -730,8 +747,6 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, · 14 unmodified lines | ||
| 730 | 747 | |
| 731 | 748 | void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked); |
| 732 | 749 | |
| 733 | −unsigned long page_mapped_in_vma(struct page *page, struct vm_area_struct *vma); | |
| 734 | − | |
| 735 | 750 | /* |
| 736 | 751 | * rmap_walk_control: To control rmap traversing for specific needs |
| 737 | 752 | * |
| @@ -787,8 +802,4 @@ static inline int folio_mkclean(struct folio *folio) · 49 unmodified lines | ||
| 787 | 802 | } |
| 788 | 803 | #endif /* CONFIG_MMU */ |
| 789 | 804 | |
| 790 | −static inline int page_mkclean(struct page *page) | |
| 791 | −{ | |
| 792 | − return folio_mkclean(page_folio(page)); | |
| 793 | −} | |
| 794 | 805 | #endif /* _LINUX_RMAP_H */ |
| @@ -942,7 +942,7 @@ struct task_struct { · 941 unmodified lines | ||
| 942 | 942 | #ifndef TIF_RESTORE_SIGMASK |
| 943 | 943 | unsigned restore_sigmask:1; |
| 944 | 944 | #endif |
| 945 | −#ifdef CONFIG_MEMCG | |
| 945 | +#ifdef CONFIG_MEMCG_V1 | |
| 946 | 946 | unsigned in_user_fault:1; |
| 947 | 947 | #endif |
| 948 | 948 | #ifdef CONFIG_LRU_GEN |
| @@ -1458,17 +1458,18 @@ struct task_struct { · 509 unmodified lines | ||
| 1458 | 1458 | unsigned int kcov_softirq; |
| 1459 | 1459 | #endif |
| 1460 | 1460 | |
| 1461 | −#ifdef CONFIG_MEMCG | |
| 1461 | +#ifdef CONFIG_MEMCG_V1 | |
| 1462 | 1462 | struct mem_cgroup *memcg_in_oom; |
| 1463 | +#endif | |
| 1463 | 1464 | |
| 1465 | +#ifdef CONFIG_MEMCG | |
| 1464 | 1466 | /* Number of pages to reclaim on returning to userland: */ |
| 1465 | 1467 | unsigned int memcg_nr_pages_over_high; |
| 1466 | 1468 | |
| 1467 | 1469 | /* Used by memcontrol for targeted memcg charge: */ |
| 1468 | 1470 | struct mem_cgroup *active_memcg; |
| 1469 | −#endif | |
| 1470 | 1471 | |
| 1471 | −#ifdef CONFIG_MEMCG_KMEM | |
| 1472 | + /* Cache for current->cgroups->memcg->objcg lookups: */ | |
| 1472 | 1473 | struct obj_cgroup *objcg; |
| 1473 | 1474 | #endif |
| 1474 | 1475 | |
| @@ -113,12 +113,21 @@ int shmem_unuse(unsigned int type); · 112 unmodified lines | ||
| 113 | 113 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| 114 | 114 | extern bool shmem_is_huge(struct inode *inode, pgoff_t index, bool shmem_huge_force, |
| 115 | 115 | struct mm_struct *mm, unsigned long vm_flags); |
| 116 | +unsigned long shmem_allowable_huge_orders(struct inode *inode, | |
| 117 | + struct vm_area_struct *vma, pgoff_t index, | |
| 118 | + bool global_huge); | |
| 116 | 119 | #else |
| 117 | 120 | static __always_inline bool shmem_is_huge(struct inode *inode, pgoff_t index, bool shmem_huge_force, |
| 118 | 121 | struct mm_struct *mm, unsigned long vm_flags) |
| 119 | 122 | { |
| 120 | 123 | return false; |
| 121 | 124 | } |
| 125 | +static inline unsigned long shmem_allowable_huge_orders(struct inode *inode, | |
| 126 | + struct vm_area_struct *vma, pgoff_t index, | |
| 127 | + bool global_huge) | |
| 128 | +{ | |
| 129 | + return 0; | |
| 130 | +} | |
| 122 | 131 | #endif |
| 123 | 132 | |
| 124 | 133 | #ifdef CONFIG_SHMEM |
| @@ -3429,6 +3429,10 @@ static inline struct page *__dev_alloc_pages_noprof(gfp_t gfp_mask, · 3428 unmodified lines | ||
| 3429 | 3429 | } |
| 3430 | 3430 | #define __dev_alloc_pages(...) alloc_hooks(__dev_alloc_pages_noprof(__VA_ARGS__)) |
| 3431 | 3431 | |
| 3432 | +/* | |
| 3433 | + * This specialized allocator has to be a macro for its allocations to be | |
| 3434 | + * accounted separately (to have a separate alloc_tag). | |
| 3435 | + */ | |
| 3432 | 3436 | #define dev_alloc_pages(_order) __dev_alloc_pages(GFP_ATOMIC | __GFP_NOWARN, _order) |
| 3433 | 3437 | |
| 3434 | 3438 | /** |
| @@ -3445,6 +3449,10 @@ static inline struct page *__dev_alloc_page_noprof(gfp_t gfp_mask) · 10 unmodified lines | ||
| 3445 | 3449 | } |
| 3446 | 3450 | #define __dev_alloc_page(...) alloc_hooks(__dev_alloc_page_noprof(__VA_ARGS__)) |
| 3447 | 3451 | |
| 3452 | +/* | |
| 3453 | + * This specialized allocator has to be a macro for its allocations to be | |
| 3454 | + * accounted separately (to have a separate alloc_tag). | |
| 3455 | + */ | |
| 3448 | 3456 | #define dev_alloc_page() dev_alloc_pages(0) |
| 3449 | 3457 | |
| 3450 | 3458 | /** |
| @@ -414,6 +414,11 @@ void sk_psock_stop_verdict(struct sock *sk, struct sk_psock *psock); · 413 unmodified lines | ||
| 414 | 414 | int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock, |
| 415 | 415 | struct sk_msg *msg); |
| 416 | 416 | |
| 417 | +/* | |
| 418 | + * This specialized allocator has to be a macro for its allocations to be | |
| 419 | + * accounted separately (to have a separate alloc_tag). The typecast is | |
| 420 | + * intentional to enforce typesafety. | |
| 421 | + */ | |
| 417 | 422 | #define sk_psock_init_link() \ |
| 418 | 423 | ((struct sk_psock_link *)kzalloc(sizeof(struct sk_psock_link), \ |
| 419 | 424 | GFP_ATOMIC | __GFP_NOWARN)) |
| @@ -41,7 +41,7 @@ enum _slab_flag_bits { · 40 unmodified lines | ||
| 41 | 41 | #ifdef CONFIG_FAILSLAB |
| 42 | 42 | _SLAB_FAILSLAB, |
| 43 | 43 | #endif |
| 44 | −#ifdef CONFIG_MEMCG_KMEM | |
| 44 | +#ifdef CONFIG_MEMCG | |
| 45 | 45 | _SLAB_ACCOUNT, |
| 46 | 46 | #endif |
| 47 | 47 | #ifdef CONFIG_KASAN_GENERIC |
| @@ -171,7 +171,7 @@ enum _slab_flag_bits { · 123 unmodified lines | ||
| 171 | 171 | # define SLAB_FAILSLAB __SLAB_FLAG_UNUSED |
| 172 | 172 | #endif |
| 173 | 173 | /* Account to memcg */ |
| 174 | −#ifdef CONFIG_MEMCG_KMEM | |
| 174 | +#ifdef CONFIG_MEMCG | |
| 175 | 175 | # define SLAB_ACCOUNT __SLAB_FLAG_BIT(_SLAB_ACCOUNT) |
| 176 | 176 | #else |
| 177 | 177 | # define SLAB_ACCOUNT __SLAB_FLAG_UNUSED |
| @@ -407,7 +407,7 @@ enum kmalloc_cache_type { · 229 unmodified lines | ||
| 407 | 407 | #ifndef CONFIG_ZONE_DMA |
| 408 | 408 | KMALLOC_DMA = KMALLOC_NORMAL, |
| 409 | 409 | #endif |
| 410 | −#ifndef CONFIG_MEMCG_KMEM | |
| 410 | +#ifndef CONFIG_MEMCG | |
| 411 | 411 | KMALLOC_CGROUP = KMALLOC_NORMAL, |
| 412 | 412 | #endif |
| 413 | 413 | KMALLOC_RANDOM_START = KMALLOC_NORMAL, |
| @@ -420,7 +420,7 @@ enum kmalloc_cache_type { · 6 unmodified lines | ||
| 420 | 420 | #ifdef CONFIG_ZONE_DMA |
| 421 | 421 | KMALLOC_DMA, |
| 422 | 422 | #endif |
| 423 | −#ifdef CONFIG_MEMCG_KMEM | |
| 423 | +#ifdef CONFIG_MEMCG | |
| 424 | 424 | KMALLOC_CGROUP, |
| 425 | 425 | #endif |
| 426 | 426 | NR_KMALLOC_TYPES |
| @@ -436,7 +436,7 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; · 9 unmodified lines | ||
| 436 | 436 | #define KMALLOC_NOT_NORMAL_BITS \ |
| 437 | 437 | (__GFP_RECLAIMABLE | \ |
| 438 | 438 | (IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \ |
| 439 | − (IS_ENABLED(CONFIG_MEMCG_KMEM) ? __GFP_ACCOUNT : 0)) | |
| 439 | + (IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0)) | |
| 440 | 440 | |
| 441 | 441 | extern unsigned long random_kmalloc_seed; |
| 442 | 442 | |
| @@ -464,7 +464,7 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne · 21 unmodified lines | ||
| 464 | 464 | */ |
| 465 | 465 | if (IS_ENABLED(CONFIG_ZONE_DMA) && (flags & __GFP_DMA)) |
| 466 | 466 | return KMALLOC_DMA; |
| 467 | − if (!IS_ENABLED(CONFIG_MEMCG_KMEM) || (flags & __GFP_RECLAIMABLE)) | |
| 467 | + if (!IS_ENABLED(CONFIG_MEMCG) || (flags & __GFP_RECLAIMABLE)) | |
| 468 | 468 | return KMALLOC_RECLAIM; |
| 469 | 469 | else |
| 470 | 470 | return KMALLOC_CGROUP; |
| @@ -405,10 +405,13 @@ extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order, · 404 unmodified lines | ||
| 405 | 405 | |
| 406 | 406 | #define MEMCG_RECLAIM_MAY_SWAP (1 << 1) |
| 407 | 407 | #define MEMCG_RECLAIM_PROACTIVE (1 << 2) |
| 408 | +#define MIN_SWAPPINESS 0 | |
| 409 | +#define MAX_SWAPPINESS 200 | |
| 408 | 410 | extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, |
| 409 | 411 | unsigned long nr_pages, |
| 410 | 412 | gfp_t gfp_mask, |
| 411 | − unsigned int reclaim_options); | |
| 413 | + unsigned int reclaim_options, | |
| 414 | + int *swappiness); | |
| 412 | 415 | extern unsigned long mem_cgroup_shrink_node(struct mem_cgroup *mem, |
| 413 | 416 | gfp_t gfp_mask, bool noswap, |
| 414 | 417 | pg_data_t *pgdat, |
| @@ -478,7 +481,7 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t); · 63 unmodified lines | ||
| 478 | 481 | extern void swap_shmem_alloc(swp_entry_t); |
| 479 | 482 | extern int swap_duplicate(swp_entry_t); |
| 480 | 483 | extern int swapcache_prepare(swp_entry_t); |
| 481 | −extern void swap_free(swp_entry_t); | |
| 484 | +extern void swap_free_nr(swp_entry_t entry, int nr_pages); | |
| 482 | 485 | extern void swapcache_free_entries(swp_entry_t *entries, int n); |
| 483 | 486 | extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); |
| 484 | 487 | int swap_type_of(dev_t device, sector_t offset); |
| @@ -556,7 +559,7 @@ static inline int swapcache_prepare(swp_entry_t swp) · 71 unmodified lines | ||
| 556 | 559 | return 0; |
| 557 | 560 | } |
| 558 | 561 | |
| 559 | −static inline void swap_free(swp_entry_t swp) | |
| 562 | +static inline void swap_free_nr(swp_entry_t entry, int nr_pages) | |
| 560 | 563 | { |
| 561 | 564 | } |
| 562 | 565 | |
| @@ -604,6 +607,11 @@ static inline void free_swap_and_cache(swp_entry_t entry) · 41 unmodified lines | ||
| 604 | 607 | free_swap_and_cache_nr(entry, 1); |
| 605 | 608 | } |
| 606 | 609 | |
| 610 | +static inline void swap_free(swp_entry_t entry) | |
| 611 | +{ | |
| 612 | + swap_free_nr(entry, 1); | |
| 613 | +} | |
| 614 | + | |
| 607 | 615 | #ifdef CONFIG_MEMCG |
| 608 | 616 | static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) |
| 609 | 617 | { |
| @@ -334,7 +334,7 @@ static inline bool is_migration_entry_dirty(swp_entry_t entry) · 333 unmodified lines | ||
| 334 | 334 | |
| 335 | 335 | extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, |
| 336 | 336 | unsigned long address); |
| 337 | −extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); | |
| 337 | +extern void migration_entry_wait_huge(struct vm_area_struct *vma, unsigned long addr, pte_t *pte); | |
| 338 | 338 | #else /* CONFIG_MIGRATION */ |
| 339 | 339 | static inline swp_entry_t make_readable_migration_entry(pgoff_t offset) |
| 340 | 340 | { |
| @@ -359,7 +359,7 @@ static inline int is_migration_entry(swp_entry_t swp) · 18 unmodified lines | ||
| 359 | 359 | static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, |
| 360 | 360 | unsigned long address) { } |
| 361 | 361 | static inline void migration_entry_wait_huge(struct vm_area_struct *vma, |
| 362 | − pte_t *pte) { } | |
| 362 | + unsigned long addr, pte_t *pte) { } | |
| 363 | 363 | static inline int is_writable_migration_entry(swp_entry_t entry) |
| 364 | 364 | { |
| 365 | 365 | return 0; |
| @@ -624,4 +624,8 @@ static inline void lruvec_stat_sub_folio(struct folio *folio, · 623 unmodified lines | ||
| 624 | 624 | { |
| 625 | 625 | lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio)); |
| 626 | 626 | } |
| 627 | + | |
| 628 | +void __meminit mod_node_early_perpage_metadata(int nid, long delta); | |
| 629 | +void __meminit store_early_perpage_metadata(void); | |
| 630 | + | |
| 627 | 631 | #endif /* _LINUX_VMSTAT_H */ |
| @@ -35,7 +35,8 @@ void zswap_swapoff(int type); · 34 unmodified lines | ||
| 35 | 35 | void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); |
| 36 | 36 | void zswap_lruvec_state_init(struct lruvec *lruvec); |
| 37 | 37 | void zswap_folio_swapin(struct folio *folio); |
| 38 | −bool is_zswap_enabled(void); | |
| 38 | +bool zswap_is_enabled(void); | |
| 39 | +bool zswap_never_enabled(void); | |
| 39 | 40 | #else |
| 40 | 41 | |
| 41 | 42 | struct zswap_lruvec_state {}; |
| @@ -60,11 +61,16 @@ static inline void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) {} · 18 unmodified lines | ||
| 60 | 61 | static inline void zswap_lruvec_state_init(struct lruvec *lruvec) {} |
| 61 | 62 | static inline void zswap_folio_swapin(struct folio *folio) {} |
| 62 | 63 | |
| 63 | −static inline bool is_zswap_enabled(void) | |
| 64 | +static inline bool zswap_is_enabled(void) | |
| 64 | 65 | { |
| 65 | 66 | return false; |
| 66 | 67 | } |
| 67 | 68 | |
| 69 | +static inline bool zswap_never_enabled(void) | |
| 70 | +{ | |
| 71 | + return true; | |
| 72 | +} | |
| 73 | + | |
| 68 | 74 | #endif |
| 69 | 75 | |
| 70 | 76 | #endif /* _LINUX_ZSWAP_H */ |
| @@ -356,10 +356,9 @@ TRACE_EVENT(aer_event, · 355 unmodified lines | ||
| 356 | 356 | #define MF_PAGE_TYPE \ |
| 357 | 357 | EM ( MF_MSG_KERNEL, "reserved kernel page" ) \ |
| 358 | 358 | EM ( MF_MSG_KERNEL_HIGH_ORDER, "high-order kernel page" ) \ |
| 359 | − EM ( MF_MSG_SLAB, "kernel slab page" ) \ | |
| 360 | − EM ( MF_MSG_DIFFERENT_COMPOUND, "different compound page after locking" ) \ | |
| 361 | 359 | EM ( MF_MSG_HUGE, "huge page" ) \ |
| 362 | 360 | EM ( MF_MSG_FREE_HUGE, "free huge page" ) \ |
| 361 | + EM ( MF_MSG_GET_HWPOISON, "get hwpoison page" ) \ | |
| 363 | 362 | EM ( MF_MSG_UNMAP_FAILED, "unmapping failed page" ) \ |
| 364 | 363 | EM ( MF_MSG_DIRTY_SWAPCACHE, "dirty swapcache page" ) \ |
| 365 | 364 | EM ( MF_MSG_CLEAN_SWAPCACHE, "clean swapcache page" ) \ |
| @@ -373,6 +372,7 @@ TRACE_EVENT(aer_event, · 7 unmodified lines | ||
| 373 | 372 | EM ( MF_MSG_BUDDY, "free buddy page" ) \ |
| 374 | 373 | EM ( MF_MSG_DAX, "dax page" ) \ |
| 375 | 374 | EM ( MF_MSG_UNSPLIT_THP, "unsplit thp" ) \ |
| 375 | + EM ( MF_MSG_ALREADY_POISONED, "already poisoned" ) \ | |
| 376 | 376 | EMe ( MF_MSG_UNKNOWN, "unknown page" ) |
| 377 | 377 | |
| 378 | 378 | /* |
| @@ -36,7 +36,7 @@ TRACE_EVENT(kmem_cache_alloc, · 35 unmodified lines | ||
| 36 | 36 | __entry->bytes_alloc = s->size; |
| 37 | 37 | __entry->gfp_flags = (__force unsigned long)gfp_flags; |
| 38 | 38 | __entry->node = node; |
| 39 | − __entry->accounted = IS_ENABLED(CONFIG_MEMCG_KMEM) ? | |
| 39 | + __entry->accounted = IS_ENABLED(CONFIG_MEMCG) ? | |
| 40 | 40 | ((gfp_flags & __GFP_ACCOUNT) || |
| 41 | 41 | (s->flags & SLAB_ACCOUNT)) : false; |
| 42 | 42 | ), |
| @@ -87,7 +87,7 @@ TRACE_EVENT(kmalloc, · 44 unmodified lines | ||
| 87 | 87 | __entry->bytes_alloc, |
| 88 | 88 | show_gfp_flags(__entry->gfp_flags), |
| 89 | 89 | __entry->node, |
| 90 | − (IS_ENABLED(CONFIG_MEMCG_KMEM) && | |
| 90 | + (IS_ENABLED(CONFIG_MEMCG) && | |
| 91 | 91 | (__entry->gfp_flags & (__force unsigned long)__GFP_ACCOUNT)) ? "true" : "false") |
| 92 | 92 | ); |
| 93 | 93 | |
| @@ -22,7 +22,8 @@ · 21 unmodified lines | ||
| 22 | 22 | EM( MR_NUMA_MISPLACED, "numa_misplaced") \ |
| 23 | 23 | EM( MR_CONTIG_RANGE, "contig_range") \ |
| 24 | 24 | EM( MR_LONGTERM_PIN, "longterm_pin") \ |
| 25 | − EMe(MR_DEMOTION, "demotion") | |
| 25 | + EM( MR_DEMOTION, "demotion") \ | |
| 26 | + EMe(MR_DAMON, "damon") | |
| 26 | 27 | |
| 27 | 28 | /* |
| 28 | 29 | * First define the enums in the above macros to be exported to userspace |
| @@ -336,8 +336,10 @@ typedef int __bitwise __kernel_rwf_t; · 335 unmodified lines | ||
| 336 | 336 | #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ |
| 337 | 337 | RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC) |
| 338 | 338 | |
| 339 | +#define PROCFS_IOCTL_MAGIC 'f' | |
| 340 | + | |
| 339 | 341 | /* Pagemap ioctl */ |
| 340 | −#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg) | |
| 342 | +#define PAGEMAP_SCAN _IOWR(PROCFS_IOCTL_MAGIC, 16, struct pm_scan_arg) | |
| 341 | 343 | |
| 342 | 344 | /* Bitmasks provided in pm_scan_args masks and reported in page_region.categories. */ |
| 343 | 345 | #define PAGE_IS_WPALLOWED (1 << 0) |
| @@ -396,4 +398,158 @@ struct pm_scan_arg { · 52 unmodified lines | ||
| 396 | 398 | __u64 return_mask; |
| 397 | 399 | }; |
| 398 | 400 | |
| 401 | +/* /proc/<pid>/maps ioctl */ | |
| 402 | +#define PROCMAP_QUERY _IOWR(PROCFS_IOCTL_MAGIC, 17, struct procmap_query) | |
| 403 | + | |
| 404 | +enum procmap_query_flags { | |
| 405 | + /* | |
| 406 | + * VMA permission flags. | |
| 407 | + * | |
| 408 | + * Can be used as part of procmap_query.query_flags field to look up | |
| 409 | + * only VMAs satisfying specified subset of permissions. E.g., specifying | |
| 410 | + * PROCMAP_QUERY_VMA_READABLE only will return both readable and read/write VMAs, | |
| 411 | + * while having PROCMAP_QUERY_VMA_READABLE | PROCMAP_QUERY_VMA_WRITABLE will only | |
| 412 | + * return read/write VMAs, though both executable/non-executable and | |
| 413 | + * private/shared will be ignored. | |
| 414 | + * | |
| 415 | + * PROCMAP_QUERY_VMA_* flags are also returned in procmap_query.vma_flags | |
| 416 | + * field to specify actual VMA permissions. | |
| 417 | + */ | |
| 418 | + PROCMAP_QUERY_VMA_READABLE = 0x01, | |
| 419 | + PROCMAP_QUERY_VMA_WRITABLE = 0x02, | |
| 420 | + PROCMAP_QUERY_VMA_EXECUTABLE = 0x04, | |
| 421 | + PROCMAP_QUERY_VMA_SHARED = 0x08, | |
| 422 | + /* | |
| 423 | + * Query modifier flags. | |
| 424 | + * | |
| 425 | + * By default VMA that covers provided address is returned, or -ENOENT | |
| 426 | + * is returned. With PROCMAP_QUERY_COVERING_OR_NEXT_VMA flag set, closest | |
| 427 | + * VMA with vma_start > addr will be returned if no covering VMA is | |
| 428 | + * found. | |
| 429 | + * | |
| 430 | + * PROCMAP_QUERY_FILE_BACKED_VMA instructs query to consider only VMAs that | |
| 431 | + * have file backing. Can be combined with PROCMAP_QUERY_COVERING_OR_NEXT_VMA | |
| 432 | + * to iterate all VMAs with file backing. | |
| 433 | + */ | |
| 434 | + PROCMAP_QUERY_COVERING_OR_NEXT_VMA = 0x10, | |
| 435 | + PROCMAP_QUERY_FILE_BACKED_VMA = 0x20, | |
| 436 | +}; | |
| 437 | + | |
| 438 | +/* | |
| 439 | + * Input/output argument structured passed into ioctl() call. It can be used | |
| 440 | + * to query a set of VMAs (Virtual Memory Areas) of a process. | |
| 441 | + * | |
| 442 | + * Each field can be one of three kinds, marked in a short comment to the | |
| 443 | + * right of the field: | |
| 444 | + * - "in", input argument, user has to provide this value, kernel doesn't modify it; | |
| 445 | + * - "out", output argument, kernel sets this field with VMA data; | |
| 446 | + * - "in/out", input and output argument; user provides initial value (used | |
| 447 | + * to specify maximum allowable buffer size), and kernel sets it to actual | |
| 448 | + * amount of data written (or zero, if there is no data). | |
| 449 | + * | |
| 450 | + * If matching VMA is found (according to criterias specified by | |
| 451 | + * query_addr/query_flags, all the out fields are filled out, and ioctl() | |
| 452 | + * returns 0. If there is no matching VMA, -ENOENT will be returned. | |
| 453 | + * In case of any other error, negative error code other than -ENOENT is | |
| 454 | + * returned. | |
| 455 | + * | |
| 456 | + * Most of the data is similar to the one returned as text in /proc/<pid>/maps | |
| 457 | + * file, but procmap_query provides more querying flexibility. There are no | |
| 458 | + * consistency guarantees between subsequent ioctl() calls, but data returned | |
| 459 | + * for matched VMA is self-consistent. | |
| 460 | + */ | |
| 461 | +struct procmap_query { | |
| 462 | + /* Query struct size, for backwards/forward compatibility */ | |
| 463 | + __u64 size; | |
| 464 | + /* | |
| 465 | + * Query flags, a combination of enum procmap_query_flags values. | |
| 466 | + * Defines query filtering and behavior, see enum procmap_query_flags. | |
| 467 | + * | |
| 468 | + * Input argument, provided by user. Kernel doesn't modify it. | |
| 469 | + */ | |
| 470 | + __u64 query_flags; /* in */ | |
| 471 | + /* | |
| 472 | + * Query address. By default, VMA that covers this address will | |
| 473 | + * be looked up. PROCMAP_QUERY_* flags above modify this default | |
| 474 | + * behavior further. | |
| 475 | + * | |
| 476 | + * Input argument, provided by user. Kernel doesn't modify it. | |
| 477 | + */ | |
| 478 | + __u64 query_addr; /* in */ | |
| 479 | + /* VMA starting (inclusive) and ending (exclusive) address, if VMA is found. */ | |
| 480 | + __u64 vma_start; /* out */ | |
| 481 | + __u64 vma_end; /* out */ | |
| 482 | + /* VMA permissions flags. A combination of PROCMAP_QUERY_VMA_* flags. */ | |
| 483 | + __u64 vma_flags; /* out */ | |
| 484 | + /* VMA backing page size granularity. */ | |
| 485 | + __u64 vma_page_size; /* out */ | |
| 486 | + /* | |
| 487 | + * VMA file offset. If VMA has file backing, this specifies offset | |
| 488 | + * within the file that VMA's start address corresponds to. | |
| 489 | + * Is set to zero if VMA has no backing file. | |
| 490 | + */ | |
| 491 | + __u64 vma_offset; /* out */ | |
| 492 | + /* Backing file's inode number, or zero, if VMA has no backing file. */ | |
| 493 | + __u64 inode; /* out */ | |
| 494 | + /* Backing file's device major/minor number, or zero, if VMA has no backing file. */ | |
| 495 | + __u32 dev_major; /* out */ | |
| 496 | + __u32 dev_minor; /* out */ | |
| 497 | + /* | |
| 498 | + * If set to non-zero value, signals the request to return VMA name | |
| 499 | + * (i.e., VMA's backing file's absolute path, with " (deleted)" suffix | |
| 500 | + * appended, if file was unlinked from FS) for matched VMA. VMA name | |
| 501 | + * can also be some special name (e.g., "[heap]", "[stack]") or could | |
| 502 | + * be even user-supplied with prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME). | |
| 503 | + * | |
| 504 | + * Kernel will set this field to zero, if VMA has no associated name. | |
| 505 | + * Otherwise kernel will return actual amount of bytes filled in | |
| 506 | + * user-supplied buffer (see vma_name_addr field below), including the | |
| 507 | + * terminating zero. | |
| 508 | + * | |
| 509 | + * If VMA name is longer that user-supplied maximum buffer size, | |
| 510 | + * -E2BIG error is returned. | |
| 511 | + * | |
| 512 | + * If this field is set to non-zero value, vma_name_addr should point | |
| 513 | + * to valid user space memory buffer of at least vma_name_size bytes. | |
| 514 | + * If set to zero, vma_name_addr should be set to zero as well | |
| 515 | + */ | |
| 516 | + __u32 vma_name_size; /* in/out */ | |
| 517 | + /* | |
| 518 | + * If set to non-zero value, signals the request to extract and return | |
| 519 | + * VMA's backing file's build ID, if the backing file is an ELF file | |
| 520 | + * and it contains embedded build ID. | |
| 521 | + * | |
| 522 | + * Kernel will set this field to zero, if VMA has no backing file, | |
| 523 | + * backing file is not an ELF file, or ELF file has no build ID | |
| 524 | + * embedded. | |
| 525 | + * | |
| 526 | + * Build ID is a binary value (not a string). Kernel will set | |
| 527 | + * build_id_size field to exact number of bytes used for build ID. | |
| 528 | + * If build ID is requested and present, but needs more bytes than | |
| 529 | + * user-supplied maximum buffer size (see build_id_addr field below), | |
| 530 | + * -E2BIG error will be returned. | |
| 531 | + * | |
| 532 | + * If this field is set to non-zero value, build_id_addr should point | |
| 533 | + * to valid user space memory buffer of at least build_id_size bytes. | |
| 534 | + * If set to zero, build_id_addr should be set to zero as well | |
| 535 | + */ | |
| 536 | + __u32 build_id_size; /* in/out */ | |
| 537 | + /* | |
| 538 | + * User-supplied address of a buffer of at least vma_name_size bytes | |
| 539 | + * for kernel to fill with matched VMA's name (see vma_name_size field | |
| 540 | + * description above for details). | |
| 541 | + * | |
| 542 | + * Should be set to zero if VMA name should not be returned. | |
| 543 | + */ | |
| 544 | + __u64 vma_name_addr; /* in */ | |
| 545 | + /* | |
| 546 | + * User-supplied address of a buffer of at least build_id_size bytes | |
| 547 | + * for kernel to fill with matched VMA's ELF build ID, if available | |
| 548 | + * (see build_id_size field description above for details). | |
| 549 | + * | |
| 550 | + * Should be set to zero if build ID should not be returned. | |
| 551 | + */ | |
| 552 | + __u64 build_id_addr; /* in */ | |
| 553 | +}; | |
| 554 | + | |
| 399 | 555 | #endif /* _UAPI_LINUX_FS_H */ |
| @@ -971,10 +971,22 @@ config MEMCG · 970 unmodified lines | ||
| 971 | 971 | help |
| 972 | 972 | Provides control over the memory footprint of tasks in a cgroup. |
| 973 | 973 | |
| 974 | −config MEMCG_KMEM | |
| 975 | − bool | |
| 974 | +config MEMCG_V1 | |
| 975 | + bool "Legacy cgroup v1 memory controller" | |
| 976 | 976 | depends on MEMCG |
| 977 | − default y | |
| 977 | + default n | |
| 978 | + help | |
| 979 | + Legacy cgroup v1 memory controller which has been deprecated by | |
| 980 | + cgroup v2 implementation. The v1 is there for legacy applications | |
| 981 | + which haven't migrated to the new cgroup v2 interface yet. If you | |
| 982 | + do not have any such application then you are completely fine leaving | |
| 983 | + this option disabled. | |
| 984 | + | |
| 985 | + Please note that feature set of the legacy memory controller is likely | |
| 986 | + going to shrink due to deprecation process. New deployments with v1 | |
| 987 | + controller are highly discouraged. | |
| 988 | + | |
| 989 | + San N is unsure. | |
| 978 | 990 | |
| 979 | 991 | config BLK_CGROUP |
| 980 | 992 | bool "IO controller" |
| @@ -155,12 +155,9 @@ static void *__alloc(struct bpf_mem_cache *c, int node, gfp_t flags) · 154 unmodified lines | ||
| 155 | 155 | |
| 156 | 156 | static struct mem_cgroup *get_memcg(const struct bpf_mem_cache *c) |
| 157 | 157 | { |
| 158 | −#ifdef CONFIG_MEMCG_KMEM | |
| 158 | +#ifdef CONFIG_MEMCG | |
| 159 | 159 | if (c->objcg) |
| 160 | 160 | return get_mem_cgroup_from_objcg(c->objcg); |
| 161 | −#endif | |
| 162 | − | |
| 163 | −#ifdef CONFIG_MEMCG | |
| 164 | 161 | return root_mem_cgroup; |
| 165 | 162 | #else |
| 166 | 163 | return NULL; |
| @@ -534,7 +531,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu) · 367 unmodified lines | ||
| 534 | 531 | size += LLIST_NODE_SZ; /* room for llist_node */ |
| 535 | 532 | unit_size = size; |
| 536 | 533 | |
| 537 | −#ifdef CONFIG_MEMCG_KMEM | |
| 534 | +#ifdef CONFIG_MEMCG | |
| 538 | 535 | if (memcg_bpf_enabled()) |
| 539 | 536 | objcg = get_obj_cgroup_from_current(); |
| 540 | 537 | #endif |
| @@ -556,7 +553,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu) · 15 unmodified lines | ||
| 556 | 553 | pcc = __alloc_percpu_gfp(sizeof(*cc), 8, GFP_KERNEL); |
| 557 | 554 | if (!pcc) |
| 558 | 555 | return -ENOMEM; |
| 559 | −#ifdef CONFIG_MEMCG_KMEM | |
| 556 | +#ifdef CONFIG_MEMCG | |
| 560 | 557 | objcg = get_obj_cgroup_from_current(); |
| 561 | 558 | #endif |
| 562 | 559 | ma->objcg = objcg; |
| @@ -385,7 +385,7 @@ void bpf_map_free_id(struct bpf_map *map) · 384 unmodified lines | ||
| 385 | 385 | spin_unlock_irqrestore(&map_idr_lock, flags); |
| 386 | 386 | } |
| 387 | 387 | |
| 388 | −#ifdef CONFIG_MEMCG_KMEM | |
| 388 | +#ifdef CONFIG_MEMCG | |
| 389 | 389 | static void bpf_map_save_memcg(struct bpf_map *map) |
| 390 | 390 | { |
| 391 | 391 | /* Currently if a map is created by a process belonging to the root |
| @@ -486,7 +486,7 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, · 94 unmodified lines | ||
| 486 | 486 | unsigned long i, j; |
| 487 | 487 | struct page *pg; |
| 488 | 488 | int ret = 0; |
| 489 | −#ifdef CONFIG_MEMCG_KMEM | |
| 489 | +#ifdef CONFIG_MEMCG | |
| 490 | 490 | struct mem_cgroup *memcg, *old_memcg; |
| 491 | 491 | |
| 492 | 492 | memcg = bpf_map_get_memcg(map); |
| @@ -505,7 +505,7 @@ int bpf_map_alloc_pages(const struct bpf_map *map, gfp_t gfp, int nid, · 12 unmodified lines | ||
| 505 | 505 | break; |
| 506 | 506 | } |
| 507 | 507 | |
| 508 | −#ifdef CONFIG_MEMCG_KMEM | |
| 508 | +#ifdef CONFIG_MEMCG | |
| 509 | 509 | set_active_memcg(old_memcg); |
| 510 | 510 | mem_cgroup_put(memcg); |
| 511 | 511 | #endif |
| @@ -21132,8 +21132,12 @@ BTF_SET_START(btf_non_sleepable_error_inject) · 21131 unmodified lines | ||
| 21132 | 21132 | * Assume non-sleepable from bpf safety point of view. |
| 21133 | 21133 | */ |
| 21134 | 21134 | BTF_ID(func, __filemap_add_folio) |
| 21135 | +#ifdef CONFIG_FAIL_PAGE_ALLOC | |
| 21135 | 21136 | BTF_ID(func, should_fail_alloc_page) |
| 21137 | +#endif | |
| 21138 | +#ifdef CONFIG_FAILSLAB | |
| 21136 | 21139 | BTF_ID(func, should_failslab) |
| 21140 | +#endif | |
| 21137 | 21141 | BTF_SET_END(btf_non_sleepable_error_inject) |
| 21138 | 21142 | |
| 21139 | 21143 | static int check_non_sleepable_error_inject(u32 btf_id) |
| @@ -7634,7 +7634,7 @@ again: · 7633 unmodified lines | ||
| 7634 | 7634 | |
| 7635 | 7635 | pte = ptep_get_lockless(ptep); |
| 7636 | 7636 | if (pte_present(pte)) |
| 7637 | − size = pte_leaf_size(pte); | |
| 7637 | + size = __pte_leaf_size(pmd, pte); | |
| 7638 | 7638 | pte_unmap(ptep); |
| 7639 | 7639 | #endif /* CONFIG_HAVE_GUP_FAST */ |
| 7640 | 7640 | |
| @@ -181,7 +181,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, · 180 unmodified lines | ||
| 181 | 181 | |
| 182 | 182 | if (new_page) { |
| 183 | 183 | folio_get(new_folio); |
| 184 | − folio_add_new_anon_rmap(new_folio, vma, addr); | |
| 184 | + folio_add_new_anon_rmap(new_folio, vma, addr, RMAP_EXCLUSIVE); | |
| 185 | 185 | folio_add_lru_vma(new_folio, vma); |
| 186 | 186 | } else |
| 187 | 187 | /* no new page, just dec_mm_counter for old_page */ |
| @@ -438,14 +438,46 @@ static void coredump_task_exit(struct task_struct *tsk) · 437 unmodified lines | ||
| 438 | 438 | } |
| 439 | 439 | |
| 440 | 440 | #ifdef CONFIG_MEMCG |
| 441 | +/* drops tasklist_lock if succeeds */ | |
| 442 | +static bool __try_to_set_owner(struct task_struct *tsk, struct mm_struct *mm) | |
| 443 | +{ | |
| 444 | + bool ret = false; | |
| 445 | + | |
| 446 | + task_lock(tsk); | |
| 447 | + if (likely(tsk->mm == mm)) { | |
| 448 | + /* tsk can't pass exit_mm/exec_mmap and exit */ | |
| 449 | + read_unlock(&tasklist_lock); | |
| 450 | + WRITE_ONCE(mm->owner, tsk); | |
| 451 | + lru_gen_migrate_mm(mm); | |
| 452 | + ret = true; | |
| 453 | + } | |
| 454 | + task_unlock(tsk); | |
| 455 | + return ret; | |
| 456 | +} | |
| 457 | + | |
| 458 | +static bool try_to_set_owner(struct task_struct *g, struct mm_struct *mm) | |
| 459 | +{ | |
| 460 | + struct task_struct *t; | |
| 461 | + | |
| 462 | + for_each_thread(g, t) { | |
| 463 | + struct mm_struct *t_mm = READ_ONCE(t->mm); | |
| 464 | + if (t_mm == mm) { | |
| 465 | + if (__try_to_set_owner(t, mm)) | |
| 466 | + return true; | |
| 467 | + } else if (t_mm) | |
| 468 | + break; | |
| 469 | + } | |
| 470 | + | |
| 471 | + return false; | |
| 472 | +} | |
| 473 | + | |
| 441 | 474 | /* |
| 442 | 475 | * A task is exiting. If it owned this mm, find a new owner for the mm. |
| 443 | 476 | */ |
| 444 | 477 | void mm_update_next_owner(struct mm_struct *mm) |
| 445 | 478 | { |
| 446 | − struct task_struct *c, *g, *p = current; | |
| 479 | + struct task_struct *g, *p = current; | |
| 447 | 480 | |
| 448 | −retry: | |
| 449 | 481 | /* |
| 450 | 482 | * If the exiting or execing task is not the owner, it's |
| 451 | 483 | * someone else's problem. |
| @@ -466,19 +498,17 @@ retry: · 14 unmodified lines | ||
| 466 | 498 | /* |
| 467 | 499 | * Search in the children |
| 468 | 500 | */ |
| 469 | − list_for_each_entry(c, &p->children, sibling) { | |
| 470 | − if (c->mm == mm) | |
| 471 | − goto assign_new_owner; | |
| 501 | + list_for_each_entry(g, &p->children, sibling) { | |
| 502 | + if (try_to_set_owner(g, mm)) | |
| 503 | + goto ret; | |
| 472 | 504 | } |
| 473 | − | |
| 474 | 505 | /* |
| 475 | 506 | * Search in the siblings |
| 476 | 507 | */ |
| 477 | − list_for_each_entry(c, &p->real_parent->children, sibling) { | |
| 478 | − if (c->mm == mm) | |
| 479 | − goto assign_new_owner; | |
| 508 | + list_for_each_entry(g, &p->real_parent->children, sibling) { | |
| 509 | + if (try_to_set_owner(g, mm)) | |
| 510 | + goto ret; | |
| 480 | 511 | } |
| 481 | − | |
| 482 | 512 | /* |
| 483 | 513 | * Search through everything else, we should not get here often. |
| 484 | 514 | */ |
| @@ -487,12 +517,8 @@ retry: · 2 unmodified lines | ||
| 487 | 517 | break; |
| 488 | 518 | if (g->flags & PF_KTHREAD) |
| 489 | 519 | continue; |
| 490 | − for_each_thread(g, c) { | |
| 491 | − if (c->mm == mm) | |
| 492 | − goto assign_new_owner; | |
| 493 | − if (c->mm) | |
| 494 | − break; | |
| 495 | − } | |
| 520 | + if (try_to_set_owner(g, mm)) | |
| 521 | + goto ret; | |
| 496 | 522 | } |
| 497 | 523 | read_unlock(&tasklist_lock); |
| 498 | 524 | /* |
| @@ -501,30 +527,9 @@ retry: · 2 unmodified lines | ||
| 501 | 527 | * ptrace or page migration (get_task_mm()). Mark owner as NULL. |
| 502 | 528 | */ |
| 503 | 529 | WRITE_ONCE(mm->owner, NULL); |
| 530 | + ret: | |
| 504 | 531 | return; |
| 505 | 532 | |
| 506 | −assign_new_owner: | |
| 507 | − BUG_ON(c == p); | |
| 508 | − get_task_struct(c); | |
| 509 | − /* | |
| 510 | − * The task_lock protects c->mm from changing. | |
| 511 | − * We always want mm->owner->mm == mm | |
| 512 | − */ | |
| 513 | − task_lock(c); | |
| 514 | − /* | |
| 515 | − * Delay read_unlock() till we have the task_lock() | |
| 516 | − * to ensure that c does not slip away underneath us | |
| 517 | − */ | |
| 518 | − read_unlock(&tasklist_lock); | |
| 519 | − if (c->mm != mm) { | |
| 520 | − task_unlock(c); | |
| 521 | − put_task_struct(c); | |
| 522 | − goto retry; | |
| 523 | − } | |
| 524 | − WRITE_ONCE(mm->owner, c); | |
| 525 | − lru_gen_migrate_mm(mm); | |
| 526 | − task_unlock(c); | |
| 527 | − put_task_struct(c); | |
| 528 | 533 | } |
| 529 | 534 | #endif /* CONFIG_MEMCG */ |
| 530 | 535 | |
| @@ -44,6 +44,7 @@ · 43 unmodified lines | ||
| 44 | 44 | #include <linux/fs.h> |
| 45 | 45 | #include <linux/mm.h> |
| 46 | 46 | #include <linux/mm_inline.h> |
| 47 | +#include <linux/memblock.h> | |
| 47 | 48 | #include <linux/nsproxy.h> |
| 48 | 49 | #include <linux/capability.h> |
| 49 | 50 | #include <linux/cpu.h> |
| @@ -992,10 +993,10 @@ void __init __weak arch_task_cache_init(void) { } · 942 unmodified lines | ||
| 992 | 993 | /* |
| 993 | 994 | * set_max_threads |
| 994 | 995 | */ |
| 995 | −static void set_max_threads(unsigned int max_threads_suggested) | |
| 996 | +static void __init set_max_threads(unsigned int max_threads_suggested) | |
| 996 | 997 | { |
| 997 | 998 | u64 threads; |
| 998 | − unsigned long nr_pages = totalram_pages(); | |
| 999 | + unsigned long nr_pages = PHYS_PFN(memblock_phys_mem_size() - memblock_reserved_size()); | |
| 999 | 1000 | |
| 1000 | 1001 | /* |
| 1001 | 1002 | * The number of threads shall be limited such that the thread |
| @@ -1018,7 +1019,7 @@ static void set_max_threads(unsigned int max_threads_suggested) · 16 unmodified lines | ||
| 1018 | 1019 | int arch_task_struct_size __read_mostly; |
| 1019 | 1020 | #endif |
| 1020 | 1021 | |
| 1021 | −static void task_struct_whitelist(unsigned long *offset, unsigned long *size) | |
| 1022 | +static void __init task_struct_whitelist(unsigned long *offset, unsigned long *size) | |
| 1022 | 1023 | { |
| 1023 | 1024 | /* Fetch thread_struct whitelist for the architecture. */ |
| 1024 | 1025 | arch_thread_struct_whitelist(offset, size); |
| @@ -1519,14 +1520,13 @@ struct mm_struct *get_task_mm(struct task_struct *task) · 494 unmodified lines | ||
| 1519 | 1520 | { |
| 1520 | 1521 | struct mm_struct *mm; |
| 1521 | 1522 | |
| 1523 | + if (task->flags & PF_KTHREAD) | |
| 1524 | + return NULL; | |
| 1525 | + | |
| 1522 | 1526 | task_lock(task); |
| 1523 | 1527 | mm = task->mm; |
| 1524 | − if (mm) { | |
| 1525 | − if (task->flags & PF_KTHREAD) | |
| 1526 | − mm = NULL; | |
| 1527 | − else | |
| 1528 | − mmget(mm); | |
| 1529 | − } | |
| 1528 | + if (mm) | |
| 1529 | + mmget(mm); | |
| 1530 | 1530 | task_unlock(task); |
| 1531 | 1531 | return mm; |
| 1532 | 1532 | } |
| @@ -200,12 +200,11 @@ void free_all_swap_pages(int swap) · 199 unmodified lines | ||
| 200 | 200 | |
| 201 | 201 | while ((node = swsusp_extents.rb_node)) { |
| 202 | 202 | struct swsusp_extent *ext; |
| 203 | − unsigned long offset; | |
| 204 | 203 | |
| 205 | 204 | ext = rb_entry(node, struct swsusp_extent, node); |
| 206 | 205 | rb_erase(node, &swsusp_extents); |
| 207 | − for (offset = ext->start; offset <= ext->end; offset++) | |
| 208 | − swap_free(swp_entry(swap, offset)); | |
| 206 | + swap_free_nr(swp_entry(swap, ext->start), | |
| 207 | + ext->end - ext->start + 1); | |
| 209 | 208 | |
| 210 | 209 | kfree(ext); |
| 211 | 210 | } |
| @@ -7920,6 +7920,7 @@ out: · 7919 unmodified lines | ||
| 7920 | 7920 | void arch_ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip, |
| 7921 | 7921 | struct ftrace_ops *op, struct ftrace_regs *fregs) |
| 7922 | 7922 | { |
| 7923 | + kmsan_unpoison_memory(fregs, sizeof(*fregs)); | |
| 7923 | 7924 | __ftrace_ops_list_func(ip, parent_ip, NULL, fregs); |
| 7924 | 7925 | } |
| 7925 | 7926 | #else |
| @@ -4203,31 +4203,28 @@ slow_path: · 4202 unmodified lines | ||
| 4203 | 4203 | * |
| 4204 | 4204 | * Return: The contents that was stored at the index. |
| 4205 | 4205 | */ |
| 4206 | −static inline void *mas_wr_store_entry(struct ma_wr_state *wr_mas) | |
| 4206 | +static inline void mas_wr_store_entry(struct ma_wr_state *wr_mas) | |
| 4207 | 4207 | { |
| 4208 | 4208 | struct ma_state *mas = wr_mas->mas; |
| 4209 | 4209 | |
| 4210 | 4210 | wr_mas->content = mas_start(mas); |
| 4211 | 4211 | if (mas_is_none(mas) || mas_is_ptr(mas)) { |
| 4212 | 4212 | mas_store_root(mas, wr_mas->entry); |
| 4213 | − return wr_mas->content; | |
| 4213 | + return; | |
| 4214 | 4214 | } |
| 4215 | 4215 | |
| 4216 | 4216 | if (unlikely(!mas_wr_walk(wr_mas))) { |
| 4217 | 4217 | mas_wr_spanning_store(wr_mas); |
| 4218 | − return wr_mas->content; | |
| 4218 | + return; | |
| 4219 | 4219 | } |
| 4220 | 4220 | |
| 4221 | 4221 | /* At this point, we are at the leaf node that needs to be altered. */ |
| 4222 | 4222 | mas_wr_end_piv(wr_mas); |
| 4223 | 4223 | /* New root for a single pointer */ |
| 4224 | − if (unlikely(!mas->index && mas->last == ULONG_MAX)) { | |
| 4224 | + if (unlikely(!mas->index && mas->last == ULONG_MAX)) | |
| 4225 | 4225 | mas_new_root(mas, wr_mas->entry); |
| 4226 | − return wr_mas->content; | |
| 4227 | − } | |
| 4228 | − | |
| 4229 | − mas_wr_modify(wr_mas); | |
| 4230 | − return wr_mas->content; | |
| 4226 | + else | |
| 4227 | + mas_wr_modify(wr_mas); | |
| 4231 | 4228 | } |
| 4232 | 4229 | |
| 4233 | 4230 | /** |
| @@ -1550,4 +1550,5 @@ static void __exit hmm_dmirror_exit(void) · 1549 unmodified lines | ||
| 1550 | 1550 | |
| 1551 | 1551 | module_init(hmm_dmirror_init); |
| 1552 | 1552 | module_exit(hmm_dmirror_exit); |
| 1553 | +MODULE_DESCRIPTION("HMM (Heterogeneous Memory Management) test module"); | |
| 1553 | 1554 | MODULE_LICENSE("GPL"); |
| @@ -3946,4 +3946,5 @@ static void __exit maple_tree_harvest(void) · 3945 unmodified lines | ||
| 3946 | 3946 | module_init(maple_tree_seed); |
| 3947 | 3947 | module_exit(maple_tree_harvest); |
| 3948 | 3948 | MODULE_AUTHOR("Liam R. Howlett <Liam.Howlett@Oracle.com>"); |
| 3949 | +MODULE_DESCRIPTION("maple tree API test module"); | |
| 3949 | 3950 | MODULE_LICENSE("GPL"); |
| @@ -156,4 +156,5 @@ static void __exit test_ubsan_exit(void) · 155 unmodified lines | ||
| 156 | 156 | module_exit(test_ubsan_exit); |
| 157 | 157 | |
| 158 | 158 | MODULE_AUTHOR("Jinbum Park <jinb.park7@gmail.com>"); |
| 159 | +MODULE_DESCRIPTION("UBSAN unit test"); | |
| 159 | 160 | MODULE_LICENSE("GPL v2"); |
| @@ -2173,4 +2173,5 @@ static void xarray_exit(void) · 2172 unmodified lines | ||
| 2173 | 2173 | module_init(xarray_checks); |
| 2174 | 2174 | module_exit(xarray_exit); |
| 2175 | 2175 | MODULE_AUTHOR("Matthew Wilcox <willy@infradead.org>"); |
| 2176 | +MODULE_DESCRIPTION("XArray API test module"); | |
| 2176 | 2177 | MODULE_LICENSE("GPL"); |
| @@ -80,6 +80,7 @@ struct dfltcc_param_v0 { · 79 unmodified lines | ||
| 80 | 80 | uint8_t csb[1152]; |
| 81 | 81 | }; |
| 82 | 82 | |
| 83 | +static_assert(offsetof(struct dfltcc_param_v0, csb) == 384); | |
| 83 | 84 | static_assert(sizeof(struct dfltcc_param_v0) == 1536); |
| 84 | 85 | |
| 85 | 86 | #define CVT_CRC32 0 |
| @@ -2,6 +2,8 @@ · 1 unmodified line | ||
| 2 | 2 | #ifndef DFLTCC_UTIL_H |
| 3 | 3 | #define DFLTCC_UTIL_H |
| 4 | 4 | |
| 5 | +#include "dfltcc.h" | |
| 6 | +#include <linux/kmsan-checks.h> | |
| 5 | 7 | #include <linux/zutil.h> |
| 6 | 8 | |
| 7 | 9 | /* |
| @@ -20,6 +22,7 @@ typedef enum { · 12 unmodified lines | ||
| 20 | 22 | #define DFLTCC_CMPR 2 |
| 21 | 23 | #define DFLTCC_XPND 4 |
| 22 | 24 | #define HBT_CIRCULAR (1 << 7) |
| 25 | +#define DFLTCC_FN_MASK ((1 << 7) - 1) | |
| 23 | 26 | #define HB_BITS 15 |
| 24 | 27 | #define HB_SIZE (1 << HB_BITS) |
| 25 | 28 | |
| @@ -34,6 +37,7 @@ static inline dfltcc_cc dfltcc( · 8 unmodified lines | ||
| 34 | 37 | ) |
| 35 | 38 | { |
| 36 | 39 | Byte *t2 = op1 ? *op1 : NULL; |
| 40 | + unsigned char *orig_t2 = t2; | |
| 37 | 41 | size_t t3 = len1 ? *len1 : 0; |
| 38 | 42 | const Byte *t4 = op2 ? *op2 : NULL; |
| 39 | 43 | size_t t5 = len2 ? *len2 : 0; |
| @@ -59,6 +63,30 @@ static inline dfltcc_cc dfltcc( · 19 unmodified lines | ||
| 59 | 63 | : "cc", "memory"); |
| 60 | 64 | t2 = r2; t3 = r3; t4 = r4; t5 = r5; |
| 61 | 65 | |
| 66 | + /* | |
| 67 | + * Unpoison the parameter block and the output buffer. | |
| 68 | + * This is a no-op in non-KMSAN builds. | |
| 69 | + */ | |
| 70 | + switch (fn & DFLTCC_FN_MASK) { | |
| 71 | + case DFLTCC_QAF: | |
| 72 | + kmsan_unpoison_memory(param, sizeof(struct dfltcc_qaf_param)); | |
| 73 | + break; | |
| 74 | + case DFLTCC_GDHT: | |
| 75 | + kmsan_unpoison_memory(param, offsetof(struct dfltcc_param_v0, csb)); | |
| 76 | + break; | |
| 77 | + case DFLTCC_CMPR: | |
| 78 | + kmsan_unpoison_memory(param, sizeof(struct dfltcc_param_v0)); | |
| 79 | + kmsan_unpoison_memory( | |
| 80 | + orig_t2, | |
| 81 | + t2 - orig_t2 + | |
| 82 | + (((struct dfltcc_param_v0 *)param)->sbb == 0 ? 0 : 1)); | |
| 83 | + break; | |
| 84 | + case DFLTCC_XPND: | |
| 85 | + kmsan_unpoison_memory(param, sizeof(struct dfltcc_param_v0)); | |
| 86 | + kmsan_unpoison_memory(orig_t2, t2 - orig_t2); | |
| 87 | + break; | |
| 88 | + } | |
| 89 | + | |
| 62 | 90 | if (op1) |
| 63 | 91 | *op1 = t2; |
| 64 | 92 | if (len1) |
| @@ -128,7 +128,7 @@ config ZSWAP_COMPRESSOR_DEFAULT · 127 unmodified lines | ||
| 128 | 128 | choice |
| 129 | 129 | prompt "Default allocator" |
| 130 | 130 | depends on ZSWAP |
| 131 | − default ZSWAP_ZPOOL_DEFAULT_ZSMALLOC if MMU | |
| 131 | + default ZSWAP_ZPOOL_DEFAULT_ZSMALLOC if HAVE_ZSMALLOC | |
| 132 | 132 | default ZSWAP_ZPOOL_DEFAULT_ZBUD |
| 133 | 133 | help |
| 134 | 134 | Selects the default allocator for the compressed cache for |
| @@ -154,6 +154,7 @@ config ZSWAP_ZPOOL_DEFAULT_Z3FOLD · 19 unmodified lines | ||
| 154 | 154 | |
| 155 | 155 | config ZSWAP_ZPOOL_DEFAULT_ZSMALLOC |
| 156 | 156 | bool "zsmalloc" |
| 157 | + depends on HAVE_ZSMALLOC | |
| 157 | 158 | select ZSMALLOC |
| 158 | 159 | help |
| 159 | 160 | Use the zsmalloc allocator as the default allocator. |
| @@ -186,10 +187,15 @@ config Z3FOLD · 26 unmodified lines | ||
| 186 | 187 | page. It is a ZBUD derivative so the simplicity and determinism are |
| 187 | 188 | still there. |
| 188 | 189 | |
| 190 | +config HAVE_ZSMALLOC | |
| 191 | + def_bool y | |
| 192 | + depends on MMU | |
| 193 | + depends on PAGE_SIZE_LESS_THAN_256KB # we want <= 64 KiB | |
| 194 | + | |
| 189 | 195 | config ZSMALLOC |
| 190 | 196 | tristate |
| 191 | 197 | prompt "N:1 compression allocator (zsmalloc)" if ZSWAP |
| 192 | − depends on MMU | |
| 198 | + depends on HAVE_ZSMALLOC | |
| 193 | 199 | help |
| 194 | 200 | zsmalloc is a slab-based memory allocator designed to store |
| 195 | 201 | pages of various compression levels efficiently. It achieves |
| @@ -731,7 +737,7 @@ config DEFAULT_MMAP_MIN_ADDR · 535 unmodified lines | ||
| 731 | 737 | from userspace allocation. Keeping a user from writing to low pages |
| 732 | 738 | can help reduce the impact of kernel NULL pointer bugs. |
| 733 | 739 | |
| 734 | − For most ppc64 and x86 users with lots of address space | |
| 740 | + For most arm64, ppc64 and x86 users with lots of address space | |
| 735 | 741 | a value of 65536 is reasonable and should cause no problems. |
| 736 | 742 | On arm and other archs it should not be higher than 32768. |
| 737 | 743 | Programs which use vm86 functionality or have some need to map |
| @@ -963,6 +969,7 @@ config DEFERRED_STRUCT_PAGE_INIT · 225 unmodified lines | ||
| 963 | 969 | depends on SPARSEMEM |
| 964 | 970 | depends on !NEED_PER_CPU_KM |
| 965 | 971 | depends on 64BIT |
| 972 | + depends on !KMSAN | |
| 966 | 973 | select PADATA |
| 967 | 974 | help |
| 968 | 975 | Ordinarily all struct pages are initialised during early boot in a |
| @@ -1136,16 +1143,6 @@ config DMAPOOL_TEST · 167 unmodified lines | ||
| 1136 | 1143 | config ARCH_HAS_PTE_SPECIAL |
| 1137 | 1144 | bool |
| 1138 | 1145 | |
| 1139 | −# | |
| 1140 | −# Some architectures require a special hugepage directory format that is | |
| 1141 | −# required to support multiple hugepage sizes. For example a4fe3ce76 | |
| 1142 | −# "powerpc/mm: Allow more flexible layouts for hugepage pagetables" | |
| 1143 | −# introduced it on powerpc. This allows for a more flexible hugepage | |
| 1144 | −# pagetable layouts. | |
| 1145 | −# | |
| 1146 | −config ARCH_HAS_HUGEPD | |
| 1147 | − bool | |
| 1148 | − | |
| 1149 | 1146 | config MAPPING_DIRTY_HELPERS |
| 1150 | 1147 | bool |
| 1151 | 1148 | |
| @@ -26,6 +26,7 @@ KCOV_INSTRUMENT_page_alloc.o := n · 25 unmodified lines | ||
| 26 | 26 | KCOV_INSTRUMENT_debug-pagealloc.o := n |
| 27 | 27 | KCOV_INSTRUMENT_kmemleak.o := n |
| 28 | 28 | KCOV_INSTRUMENT_memcontrol.o := n |
| 29 | +KCOV_INSTRUMENT_memcontrol-v1.o := n | |
| 29 | 30 | KCOV_INSTRUMENT_mmzone.o := n |
| 30 | 31 | KCOV_INSTRUMENT_vmstat.o := n |
| 31 | 32 | KCOV_INSTRUMENT_failslab.o := n |
| @@ -95,6 +96,7 @@ obj-$(CONFIG_NUMA) += memory-tiers.o · 63 unmodified lines | ||
| 95 | 96 | obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o |
| 96 | 97 | obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o |
| 97 | 98 | obj-$(CONFIG_PAGE_COUNTER) += page_counter.o |
| 99 | +obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o | |
| 98 | 100 | obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o |
| 99 | 101 | ifdef CONFIG_SWAP |
| 100 | 102 | obj-$(CONFIG_MEMCG) += swap_cgroup.o |
| @@ -234,14 +234,6 @@ static int balloon_page_migrate(struct page *newpage, struct page *page, · 233 unmodified lines | ||
| 234 | 234 | { |
| 235 | 235 | struct balloon_dev_info *balloon = balloon_page_device(page); |
| 236 | 236 | |
| 237 | − /* | |
| 238 | − * We can not easily support the no copy case here so ignore it as it | |
| 239 | − * is unlikely to be used with balloon pages. See include/linux/hmm.h | |
| 240 | − * for a user of the MIGRATE_SYNC_NO_COPY mode. | |
| 241 | − */ | |
| 242 | − if (mode == MIGRATE_SYNC_NO_COPY) | |
| 243 | − return -EINVAL; | |
| 244 | − | |
| 245 | 237 | VM_BUG_ON_PAGE(!PageLocked(page), page); |
| 246 | 238 | VM_BUG_ON_PAGE(!PageLocked(newpage), newpage); |
| 247 | 239 | |
| @@ -354,7 +354,8 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 353 unmodified lines | ||
| 354 | 354 | enum damos_action action, |
| 355 | 355 | unsigned long apply_interval_us, |
| 356 | 356 | struct damos_quota *quota, |
| 357 | − struct damos_watermarks *wmarks) | |
| 357 | + struct damos_watermarks *wmarks, | |
| 358 | + int target_nid) | |
| 358 | 359 | { |
| 359 | 360 | struct damos *scheme; |
| 360 | 361 | |
| @@ -381,6 +382,8 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 20 unmodified lines | ||
| 381 | 382 | scheme->wmarks = *wmarks; |
| 382 | 383 | scheme->wmarks.activated = true; |
| 383 | 384 | |
| 385 | + scheme->target_nid = target_nid; | |
| 386 | + | |
| 384 | 387 | return scheme; |
| 385 | 388 | } |
| 386 | 389 | |
| @@ -663,6 +666,339 @@ void damon_set_schemes(struct damon_ctx *ctx, struct damos **schemes, · 276 unmodified lines | ||
| 663 | 666 | damon_add_scheme(ctx, schemes[i]); |
| 664 | 667 | } |
| 665 | 668 | |
| 669 | +static struct damos_quota_goal *damos_nth_quota_goal( | |
| 670 | + int n, struct damos_quota *q) | |
| 671 | +{ | |
| 672 | + struct damos_quota_goal *goal; | |
| 673 | + int i = 0; | |
| 674 | + | |
| 675 | + damos_for_each_quota_goal(goal, q) { | |
| 676 | + if (i++ == n) | |
| 677 | + return goal; | |
| 678 | + } | |
| 679 | + return NULL; | |
| 680 | +} | |
| 681 | + | |
| 682 | +static void damos_commit_quota_goal( | |
| 683 | + struct damos_quota_goal *dst, struct damos_quota_goal *src) | |
| 684 | +{ | |
| 685 | + dst->metric = src->metric; | |
| 686 | + dst->target_value = src->target_value; | |
| 687 | + if (dst->metric == DAMOS_QUOTA_USER_INPUT) | |
| 688 | + dst->current_value = src->current_value; | |
| 689 | + /* keep last_psi_total as is, since it will be updated in next cycle */ | |
| 690 | +} | |
| 691 | + | |
| 692 | +/** | |
| 693 | + * damos_commit_quota_goals() - Commit DAMOS quota goals to another quota. | |
| 694 | + * @dst: The commit destination DAMOS quota. | |
| 695 | + * @src: The commit source DAMOS quota. | |
| 696 | + * | |
| 697 | + * Copies user-specified parameters for quota goals from @src to @dst. Users | |
| 698 | + * should use this function for quota goals-level parameters update of running | |
| 699 | + * DAMON contexts, instead of manual in-place updates. | |
| 700 | + * | |
| 701 | + * This function should be called from parameters-update safe context, like | |
| 702 | + * DAMON callbacks. | |
| 703 | + */ | |
| 704 | +int damos_commit_quota_goals(struct damos_quota *dst, struct damos_quota *src) | |
| 705 | +{ | |
| 706 | + struct damos_quota_goal *dst_goal, *next, *src_goal, *new_goal; | |
| 707 | + int i = 0, j = 0; | |
| 708 | + | |
| 709 | + damos_for_each_quota_goal_safe(dst_goal, next, dst) { | |
| 710 | + src_goal = damos_nth_quota_goal(i++, src); | |
| 711 | + if (src_goal) | |
| 712 | + damos_commit_quota_goal(dst_goal, src_goal); | |
| 713 | + else | |
| 714 | + damos_destroy_quota_goal(dst_goal); | |
| 715 | + } | |
| 716 | + damos_for_each_quota_goal_safe(src_goal, next, src) { | |
| 717 | + if (j++ < i) | |
| 718 | + continue; | |
| 719 | + new_goal = damos_new_quota_goal( | |
| 720 | + src_goal->metric, src_goal->target_value); | |
| 721 | + if (!new_goal) | |
| 722 | + return -ENOMEM; | |
| 723 | + damos_add_quota_goal(dst, new_goal); | |
| 724 | + } | |
| 725 | + return 0; | |
| 726 | +} | |
| 727 | + | |
| 728 | +static int damos_commit_quota(struct damos_quota *dst, struct damos_quota *src) | |
| 729 | +{ | |
| 730 | + int err; | |
| 731 | + | |
| 732 | + dst->reset_interval = src->reset_interval; | |
| 733 | + dst->ms = src->ms; | |
| 734 | + dst->sz = src->sz; | |
| 735 | + err = damos_commit_quota_goals(dst, src); | |
| 736 | + if (err) | |
| 737 | + return err; | |
| 738 | + dst->weight_sz = src->weight_sz; | |
| 739 | + dst->weight_nr_accesses = src->weight_nr_accesses; | |
| 740 | + dst->weight_age = src->weight_age; | |
| 741 | + return 0; | |
| 742 | +} | |
| 743 | + | |
| 744 | +static struct damos_filter *damos_nth_filter(int n, struct damos *s) | |
| 745 | +{ | |
| 746 | + struct damos_filter *filter; | |
| 747 | + int i = 0; | |
| 748 | + | |
| 749 | + damos_for_each_filter(filter, s) { | |
| 750 | + if (i++ == n) | |
| 751 | + return filter; | |
| 752 | + } | |
| 753 | + return NULL; | |
| 754 | +} | |
| 755 | + | |
| 756 | +static void damos_commit_filter_arg( | |
| 757 | + struct damos_filter *dst, struct damos_filter *src) | |
| 758 | +{ | |
| 759 | + switch (dst->type) { | |
| 760 | + case DAMOS_FILTER_TYPE_MEMCG: | |
| 761 | + dst->memcg_id = src->memcg_id; | |
| 762 | + break; | |
| 763 | + case DAMOS_FILTER_TYPE_ADDR: | |
| 764 | + dst->addr_range = src->addr_range; | |
| 765 | + break; | |
| 766 | + case DAMOS_FILTER_TYPE_TARGET: | |
| 767 | + dst->target_idx = src->target_idx; | |
| 768 | + break; | |
| 769 | + default: | |
| 770 | + break; | |
| 771 | + } | |
| 772 | +} | |
| 773 | + | |
| 774 | +static void damos_commit_filter( | |
| 775 | + struct damos_filter *dst, struct damos_filter *src) | |
| 776 | +{ | |
| 777 | + dst->type = src->type; | |
| 778 | + dst->matching = src->matching; | |
| 779 | + damos_commit_filter_arg(dst, src); | |
| 780 | +} | |
| 781 | + | |
| 782 | +static int damos_commit_filters(struct damos *dst, struct damos *src) | |
| 783 | +{ | |
| 784 | + struct damos_filter *dst_filter, *next, *src_filter, *new_filter; | |
| 785 | + int i = 0, j = 0; | |
| 786 | + | |
| 787 | + damos_for_each_filter_safe(dst_filter, next, dst) { | |
| 788 | + src_filter = damos_nth_filter(i++, src); | |
| 789 | + if (src_filter) | |
| 790 | + damos_commit_filter(dst_filter, src_filter); | |
| 791 | + else | |
| 792 | + damos_destroy_filter(dst_filter); | |
| 793 | + } | |
| 794 | + | |
| 795 | + damos_for_each_filter_safe(src_filter, next, src) { | |
| 796 | + if (j++ < i) | |
| 797 | + continue; | |
| 798 | + | |
| 799 | + new_filter = damos_new_filter( | |
| 800 | + src_filter->type, src_filter->matching); | |
| 801 | + if (!new_filter) | |
| 802 | + return -ENOMEM; | |
| 803 | + damos_commit_filter_arg(new_filter, src_filter); | |
| 804 | + damos_add_filter(dst, new_filter); | |
| 805 | + } | |
| 806 | + return 0; | |
| 807 | +} | |
| 808 | + | |
| 809 | +static struct damos *damon_nth_scheme(int n, struct damon_ctx *ctx) | |
| 810 | +{ | |
| 811 | + struct damos *s; | |
| 812 | + int i = 0; | |
| 813 | + | |
| 814 | + damon_for_each_scheme(s, ctx) { | |
| 815 | + if (i++ == n) | |
| 816 | + return s; | |
| 817 | + } | |
| 818 | + return NULL; | |
| 819 | +} | |
| 820 | + | |
| 821 | +static int damos_commit(struct damos *dst, struct damos *src) | |
| 822 | +{ | |
| 823 | + int err; | |
| 824 | + | |
| 825 | + dst->pattern = src->pattern; | |
| 826 | + dst->action = src->action; | |
| 827 | + dst->apply_interval_us = src->apply_interval_us; | |
| 828 | + | |
| 829 | + err = damos_commit_quota(&dst->quota, &src->quota); | |
| 830 | + if (err) | |
| 831 | + return err; | |
| 832 | + | |
| 833 | + dst->wmarks = src->wmarks; | |
| 834 | + | |
| 835 | + err = damos_commit_filters(dst, src); | |
| 836 | + return err; | |
| 837 | +} | |
| 838 | + | |
| 839 | +static int damon_commit_schemes(struct damon_ctx *dst, struct damon_ctx *src) | |
| 840 | +{ | |
| 841 | + struct damos *dst_scheme, *next, *src_scheme, *new_scheme; | |
| 842 | + int i = 0, j = 0, err; | |
| 843 | + | |
| 844 | + damon_for_each_scheme_safe(dst_scheme, next, dst) { | |
| 845 | + src_scheme = damon_nth_scheme(i++, src); | |
| 846 | + if (src_scheme) { | |
| 847 | + err = damos_commit(dst_scheme, src_scheme); | |
| 848 | + if (err) | |
| 849 | + return err; | |
| 850 | + } else { | |
| 851 | + damon_destroy_scheme(dst_scheme); | |
| 852 | + } | |
| 853 | + } | |
| 854 | + | |
| 855 | + damon_for_each_scheme_safe(src_scheme, next, src) { | |
| 856 | + if (j++ < i) | |
| 857 | + continue; | |
| 858 | + new_scheme = damon_new_scheme(&src_scheme->pattern, | |
| 859 | + src_scheme->action, | |
| 860 | + src_scheme->apply_interval_us, | |
| 861 | + &src_scheme->quota, &src_scheme->wmarks, | |
| 862 | + NUMA_NO_NODE); | |
| 863 | + if (!new_scheme) | |
| 864 | + return -ENOMEM; | |
| 865 | + damon_add_scheme(dst, new_scheme); | |
| 866 | + } | |
| 867 | + return 0; | |
| 868 | +} | |
| 869 | + | |
| 870 | +static struct damon_target *damon_nth_target(int n, struct damon_ctx *ctx) | |
| 871 | +{ | |
| 872 | + struct damon_target *t; | |
| 873 | + int i = 0; | |
| 874 | + | |
| 875 | + damon_for_each_target(t, ctx) { | |
| 876 | + if (i++ == n) | |
| 877 | + return t; | |
| 878 | + } | |
| 879 | + return NULL; | |
| 880 | +} | |
| 881 | + | |
| 882 | +/* | |
| 883 | + * The caller should ensure the regions of @src are | |
| 884 | + * 1. valid (end >= src) and | |
| 885 | + * 2. sorted by starting address. | |
| 886 | + * | |
| 887 | + * If @src has no region, @dst keeps current regions. | |
| 888 | + */ | |
| 889 | +static int damon_commit_target_regions( | |
| 890 | + struct damon_target *dst, struct damon_target *src) | |
| 891 | +{ | |
| 892 | + struct damon_region *src_region; | |
| 893 | + struct damon_addr_range *ranges; | |
| 894 | + int i = 0, err; | |
| 895 | + | |
| 896 | + damon_for_each_region(src_region, src) | |
| 897 | + i++; | |
| 898 | + if (!i) | |
| 899 | + return 0; | |
| 900 | + | |
| 901 | + ranges = kmalloc_array(i, sizeof(*ranges), GFP_KERNEL | __GFP_NOWARN); | |
| 902 | + if (!ranges) | |
| 903 | + return -ENOMEM; | |
| 904 | + i = 0; | |
| 905 | + damon_for_each_region(src_region, src) | |
| 906 | + ranges[i++] = src_region->ar; | |
| 907 | + err = damon_set_regions(dst, ranges, i); | |
| 908 | + kfree(ranges); | |
| 909 | + return err; | |
| 910 | +} | |
| 911 | + | |
| 912 | +static int damon_commit_target( | |
| 913 | + struct damon_target *dst, bool dst_has_pid, | |
| 914 | + struct damon_target *src, bool src_has_pid) | |
| 915 | +{ | |
| 916 | + int err; | |
| 917 | + | |
| 918 | + err = damon_commit_target_regions(dst, src); | |
| 919 | + if (err) | |
| 920 | + return err; | |
| 921 | + if (dst_has_pid) | |
| 922 | + put_pid(dst->pid); | |
| 923 | + if (src_has_pid) | |
| 924 | + get_pid(src->pid); | |
| 925 | + dst->pid = src->pid; | |
| 926 | + return 0; | |
| 927 | +} | |
| 928 | + | |
| 929 | +static int damon_commit_targets( | |
| 930 | + struct damon_ctx *dst, struct damon_ctx *src) | |
| 931 | +{ | |
| 932 | + struct damon_target *dst_target, *next, *src_target, *new_target; | |
| 933 | + int i = 0, j = 0, err; | |
| 934 | + | |
| 935 | + damon_for_each_target_safe(dst_target, next, dst) { | |
| 936 | + src_target = damon_nth_target(i++, src); | |
| 937 | + if (src_target) { | |
| 938 | + err = damon_commit_target( | |
| 939 | + dst_target, damon_target_has_pid(dst), | |
| 940 | + src_target, damon_target_has_pid(src)); | |
| 941 | + if (err) | |
| 942 | + return err; | |
| 943 | + } else { | |
| 944 | + if (damon_target_has_pid(dst)) | |
| 945 | + put_pid(dst_target->pid); | |
| 946 | + damon_destroy_target(dst_target); | |
| 947 | + } | |
| 948 | + } | |
| 949 | + | |
| 950 | + damon_for_each_target_safe(src_target, next, src) { | |
| 951 | + if (j++ < i) | |
| 952 | + continue; | |
| 953 | + new_target = damon_new_target(); | |
| 954 | + if (!new_target) | |
| 955 | + return -ENOMEM; | |
| 956 | + err = damon_commit_target(new_target, false, | |
| 957 | + src_target, damon_target_has_pid(src)); | |
| 958 | + if (err) | |
| 959 | + return err; | |
| 960 | + } | |
| 961 | + return 0; | |
| 962 | +} | |
| 963 | + | |
| 964 | +/** | |
| 965 | + * damon_commit_ctx() - Commit parameters of a DAMON context to another. | |
| 966 | + * @dst: The commit destination DAMON context. | |
| 967 | + * @src: The commit source DAMON context. | |
| 968 | + * | |
| 969 | + * This function copies user-specified parameters from @src to @dst and update | |
| 970 | + * the internal status and results accordingly. Users should use this function | |
| 971 | + * for context-level parameters update of running context, instead of manual | |
| 972 | + * in-place updates. | |
| 973 | + * | |
| 974 | + * This function should be called from parameters-update safe context, like | |
| 975 | + * DAMON callbacks. | |
| 976 | + */ | |
| 977 | +int damon_commit_ctx(struct damon_ctx *dst, struct damon_ctx *src) | |
| 978 | +{ | |
| 979 | + int err; | |
| 980 | + | |
| 981 | + err = damon_commit_schemes(dst, src); | |
| 982 | + if (err) | |
| 983 | + return err; | |
| 984 | + err = damon_commit_targets(dst, src); | |
| 985 | + if (err) | |
| 986 | + return err; | |
| 987 | + /* | |
| 988 | + * schemes and targets should be updated first, since | |
| 989 | + * 1. damon_set_attrs() updates monitoring results of targets and | |
| 990 | + * next_apply_sis of schemes, and | |
| 991 | + * 2. ops update should be done after pid handling is done (target | |
| 992 | + * committing require putting pids). | |
| 993 | + */ | |
| 994 | + err = damon_set_attrs(dst, &src->attrs); | |
| 995 | + if (err) | |
| 996 | + return err; | |
| 997 | + dst->ops = src->ops; | |
| 998 | + | |
| 999 | + return 0; | |
| 1000 | +} | |
| 1001 | + | |
| 666 | 1002 | /** |
| 667 | 1003 | * damon_nr_running_ctxs() - Return number of currently running contexts. |
| 668 | 1004 | */ |
| @@ -281,7 +281,7 @@ static struct damos **str_to_schemes(const char *str, ssize_t len, · 280 unmodified lines | ||
| 281 | 281 | |
| 282 | 282 | pos += parsed; |
| 283 | 283 | scheme = damon_new_scheme(&pattern, action, 0, "a, |
| 284 | − &wmarks); | |
| 284 | + &wmarks, NUMA_NO_NODE); | |
| 285 | 285 | if (!scheme) |
| 286 | 286 | goto fail; |
| 287 | 287 | |
| @@ -163,7 +163,8 @@ static struct damos *damon_lru_sort_new_scheme( · 162 unmodified lines | ||
| 163 | 163 | /* under the quota. */ |
| 164 | 164 | "a, |
| 165 | 165 | /* (De)activate this according to the watermarks. */ |
| 166 | − &damon_lru_sort_wmarks); | |
| 166 | + &damon_lru_sort_wmarks, | |
| 167 | + NUMA_NO_NODE); | |
| 167 | 168 | } |
| 168 | 169 | |
| 169 | 170 | /* Create a DAMON-based operation scheme for hot memory regions */ |
| @@ -185,61 +186,48 @@ static struct damos *damon_lru_sort_new_cold_scheme(unsigned int cold_thres) · 15 unmodified lines | ||
| 185 | 186 | return damon_lru_sort_new_scheme(&pattern, DAMOS_LRU_DEPRIO); |
| 186 | 187 | } |
| 187 | 188 | |
| 188 | −static void damon_lru_sort_copy_quota_status(struct damos_quota *dst, | |
| 189 | − struct damos_quota *src) | |
| 190 | −{ | |
| 191 | − dst->total_charged_sz = src->total_charged_sz; | |
| 192 | − dst->total_charged_ns = src->total_charged_ns; | |
| 193 | − dst->charged_sz = src->charged_sz; | |
| 194 | − dst->charged_from = src->charged_from; | |
| 195 | − dst->charge_target_from = src->charge_target_from; | |
| 196 | − dst->charge_addr_from = src->charge_addr_from; | |
| 197 | −} | |
| 198 | − | |
| 199 | 189 | static int damon_lru_sort_apply_parameters(void) |
| 200 | 190 | { |
| 201 | − struct damos *scheme, *hot_scheme, *cold_scheme; | |
| 202 | − struct damos *old_hot_scheme = NULL, *old_cold_scheme = NULL; | |
| 191 | + struct damon_ctx *param_ctx; | |
| 192 | + struct damon_target *param_target; | |
| 193 | + struct damos *hot_scheme, *cold_scheme; | |
| 203 | 194 | unsigned int hot_thres, cold_thres; |
| 204 | − int err = 0; | |
| 195 | + int err; | |
| 205 | 196 | |
| 206 | − err = damon_set_attrs(ctx, &damon_lru_sort_mon_attrs); | |
| 197 | + err = damon_modules_new_paddr_ctx_target(¶m_ctx, ¶m_target); | |
| 207 | 198 | if (err) |
| 208 | 199 | return err; |
| 209 | 200 | |
| 210 | − damon_for_each_scheme(scheme, ctx) { | |
| 211 | − if (!old_hot_scheme) { | |
| 212 | − old_hot_scheme = scheme; | |
| 213 | − continue; | |
| 214 | − } | |
| 215 | − old_cold_scheme = scheme; | |
| 216 | − } | |
| 201 | + err = damon_set_attrs(ctx, &damon_lru_sort_mon_attrs); | |
| 202 | + if (err) | |
| 203 | + goto out; | |
| 217 | 204 | |
| 205 | + err = -ENOMEM; | |
| 218 | 206 | hot_thres = damon_max_nr_accesses(&damon_lru_sort_mon_attrs) * |
| 219 | 207 | hot_thres_access_freq / 1000; |
| 220 | 208 | hot_scheme = damon_lru_sort_new_hot_scheme(hot_thres); |
| 221 | 209 | if (!hot_scheme) |
| 222 | − return -ENOMEM; | |
| 223 | − if (old_hot_scheme) | |
| 224 | − damon_lru_sort_copy_quota_status(&hot_scheme->quota, | |
| 225 | − &old_hot_scheme->quota); | |
| 210 | + goto out; | |
| 226 | 211 | |
| 227 | 212 | cold_thres = cold_min_age / damon_lru_sort_mon_attrs.aggr_interval; |
| 228 | 213 | cold_scheme = damon_lru_sort_new_cold_scheme(cold_thres); |
| 229 | 214 | if (!cold_scheme) { |
| 230 | 215 | damon_destroy_scheme(hot_scheme); |
| 231 | − return -ENOMEM; | |
| 216 | + goto out; | |
| 232 | 217 | } |
| 233 | − if (old_cold_scheme) | |
| 234 | − damon_lru_sort_copy_quota_status(&cold_scheme->quota, | |
| 235 | − &old_cold_scheme->quota); | |
| 236 | 218 | |
| 237 | − damon_set_schemes(ctx, &hot_scheme, 1); | |
| 238 | − damon_add_scheme(ctx, cold_scheme); | |
| 219 | + damon_set_schemes(param_ctx, &hot_scheme, 1); | |
| 220 | + damon_add_scheme(param_ctx, cold_scheme); | |
| 239 | 221 | |
| 240 | − return damon_set_region_biggest_system_ram_default(target, | |
| 222 | + err = damon_set_region_biggest_system_ram_default(param_target, | |
| 241 | 223 | &monitor_region_start, |
| 242 | 224 | &monitor_region_end); |
| 225 | + if (err) | |
| 226 | + goto out; | |
| 227 | + err = damon_commit_ctx(ctx, param_ctx); | |
| 228 | +out: | |
| 229 | + damon_destroy_ctx(param_ctx); | |
| 230 | + return err; | |
| 243 | 231 | } |
| 244 | 232 | |
| 245 | 233 | static int damon_lru_sort_turn(bool on) |
| @@ -12,6 +12,9 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <linux/pagemap.h> |
| 13 | 13 | #include <linux/rmap.h> |
| 14 | 14 | #include <linux/swap.h> |
| 15 | +#include <linux/memory-tiers.h> | |
| 16 | +#include <linux/migrate.h> | |
| 17 | +#include <linux/mm_inline.h> | |
| 15 | 18 | |
| 16 | 19 | #include "../internal.h" |
| 17 | 20 | #include "ops-common.h" |
| @@ -325,6 +328,153 @@ static unsigned long damon_pa_deactivate_pages(struct damon_region *r, · 307 unmodified lines | ||
| 325 | 328 | return damon_pa_mark_accessed_or_deactivate(r, s, false); |
| 326 | 329 | } |
| 327 | 330 | |
| 331 | +static unsigned int __damon_pa_migrate_folio_list( | |
| 332 | + struct list_head *migrate_folios, struct pglist_data *pgdat, | |
| 333 | + int target_nid) | |
| 334 | +{ | |
| 335 | + unsigned int nr_succeeded = 0; | |
| 336 | + nodemask_t allowed_mask = NODE_MASK_NONE; | |
| 337 | + struct migration_target_control mtc = { | |
| 338 | + /* | |
| 339 | + * Allocate from 'node', or fail quickly and quietly. | |
| 340 | + * When this happens, 'page' will likely just be discarded | |
| 341 | + * instead of migrated. | |
| 342 | + */ | |
| 343 | + .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | | |
| 344 | + __GFP_NOWARN | __GFP_NOMEMALLOC | GFP_NOWAIT, | |
| 345 | + .nid = target_nid, | |
| 346 | + .nmask = &allowed_mask | |
| 347 | + }; | |
| 348 | + | |
| 349 | + if (pgdat->node_id == target_nid || target_nid == NUMA_NO_NODE) | |
| 350 | + return 0; | |
| 351 | + | |
| 352 | + if (list_empty(migrate_folios)) | |
| 353 | + return 0; | |
| 354 | + | |
| 355 | + /* Migration ignores all cpuset and mempolicy settings */ | |
| 356 | + migrate_pages(migrate_folios, alloc_migrate_folio, NULL, | |
| 357 | + (unsigned long)&mtc, MIGRATE_ASYNC, MR_DAMON, | |
| 358 | + &nr_succeeded); | |
| 359 | + | |
| 360 | + return nr_succeeded; | |
| 361 | +} | |
| 362 | + | |
| 363 | +static unsigned int damon_pa_migrate_folio_list(struct list_head *folio_list, | |
| 364 | + struct pglist_data *pgdat, | |
| 365 | + int target_nid) | |
| 366 | +{ | |
| 367 | + unsigned int nr_migrated = 0; | |
| 368 | + struct folio *folio; | |
| 369 | + LIST_HEAD(ret_folios); | |
| 370 | + LIST_HEAD(migrate_folios); | |
| 371 | + | |
| 372 | + while (!list_empty(folio_list)) { | |
| 373 | + struct folio *folio; | |
| 374 | + | |
| 375 | + cond_resched(); | |
| 376 | + | |
| 377 | + folio = lru_to_folio(folio_list); | |
| 378 | + list_del(&folio->lru); | |
| 379 | + | |
| 380 | + if (!folio_trylock(folio)) | |
| 381 | + goto keep; | |
| 382 | + | |
| 383 | + /* Relocate its contents to another node. */ | |
| 384 | + list_add(&folio->lru, &migrate_folios); | |
| 385 | + folio_unlock(folio); | |
| 386 | + continue; | |
| 387 | +keep: | |
| 388 | + list_add(&folio->lru, &ret_folios); | |
| 389 | + } | |
| 390 | + /* 'folio_list' is always empty here */ | |
| 391 | + | |
| 392 | + /* Migrate folios selected for migration */ | |
| 393 | + nr_migrated += __damon_pa_migrate_folio_list( | |
| 394 | + &migrate_folios, pgdat, target_nid); | |
| 395 | + /* | |
| 396 | + * Folios that could not be migrated are still in @migrate_folios. Add | |
| 397 | + * those back on @folio_list | |
| 398 | + */ | |
| 399 | + if (!list_empty(&migrate_folios)) | |
| 400 | + list_splice_init(&migrate_folios, folio_list); | |
| 401 | + | |
| 402 | + try_to_unmap_flush(); | |
| 403 | + | |
| 404 | + list_splice(&ret_folios, folio_list); | |
| 405 | + | |
| 406 | + while (!list_empty(folio_list)) { | |
| 407 | + folio = lru_to_folio(folio_list); | |
| 408 | + list_del(&folio->lru); | |
| 409 | + folio_putback_lru(folio); | |
| 410 | + } | |
| 411 | + | |
| 412 | + return nr_migrated; | |
| 413 | +} | |
| 414 | + | |
| 415 | +static unsigned long damon_pa_migrate_pages(struct list_head *folio_list, | |
| 416 | + int target_nid) | |
| 417 | +{ | |
| 418 | + int nid; | |
| 419 | + unsigned long nr_migrated = 0; | |
| 420 | + LIST_HEAD(node_folio_list); | |
| 421 | + unsigned int noreclaim_flag; | |
| 422 | + | |
| 423 | + if (list_empty(folio_list)) | |
| 424 | + return nr_migrated; | |
| 425 | + | |
| 426 | + noreclaim_flag = memalloc_noreclaim_save(); | |
| 427 | + | |
| 428 | + nid = folio_nid(lru_to_folio(folio_list)); | |
| 429 | + do { | |
| 430 | + struct folio *folio = lru_to_folio(folio_list); | |
| 431 | + | |
| 432 | + if (nid == folio_nid(folio)) { | |
| 433 | + list_move(&folio->lru, &node_folio_list); | |
| 434 | + continue; | |
| 435 | + } | |
| 436 | + | |
| 437 | + nr_migrated += damon_pa_migrate_folio_list(&node_folio_list, | |
| 438 | + NODE_DATA(nid), | |
| 439 | + target_nid); | |
| 440 | + nid = folio_nid(lru_to_folio(folio_list)); | |
| 441 | + } while (!list_empty(folio_list)); | |
| 442 | + | |
| 443 | + nr_migrated += damon_pa_migrate_folio_list(&node_folio_list, | |
| 444 | + NODE_DATA(nid), | |
| 445 | + target_nid); | |
| 446 | + | |
| 447 | + memalloc_noreclaim_restore(noreclaim_flag); | |
| 448 | + | |
| 449 | + return nr_migrated; | |
| 450 | +} | |
| 451 | + | |
| 452 | +static unsigned long damon_pa_migrate(struct damon_region *r, struct damos *s) | |
| 453 | +{ | |
| 454 | + unsigned long addr, applied; | |
| 455 | + LIST_HEAD(folio_list); | |
| 456 | + | |
| 457 | + for (addr = r->ar.start; addr < r->ar.end; addr += PAGE_SIZE) { | |
| 458 | + struct folio *folio = damon_get_folio(PHYS_PFN(addr)); | |
| 459 | + | |
| 460 | + if (!folio) | |
| 461 | + continue; | |
| 462 | + | |
| 463 | + if (damos_pa_filter_out(s, folio)) | |
| 464 | + goto put_folio; | |
| 465 | + | |
| 466 | + if (!folio_isolate_lru(folio)) | |
| 467 | + goto put_folio; | |
| 468 | + list_add(&folio->lru, &folio_list); | |
| 469 | +put_folio: | |
| 470 | + folio_put(folio); | |
| 471 | + } | |
| 472 | + applied = damon_pa_migrate_pages(&folio_list, s->target_nid); | |
| 473 | + cond_resched(); | |
| 474 | + return applied * PAGE_SIZE; | |
| 475 | +} | |
| 476 | + | |
| 477 | + | |
| 328 | 478 | static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx, |
| 329 | 479 | struct damon_target *t, struct damon_region *r, |
| 330 | 480 | struct damos *scheme) |
| @@ -336,6 +486,9 @@ static unsigned long damon_pa_apply_scheme(struct damon_ctx *ctx, · 5 unmodified lines | ||
| 336 | 486 | return damon_pa_mark_accessed(r, scheme); |
| 337 | 487 | case DAMOS_LRU_DEPRIO: |
| 338 | 488 | return damon_pa_deactivate_pages(r, scheme); |
| 489 | + case DAMOS_MIGRATE_HOT: | |
| 490 | + case DAMOS_MIGRATE_COLD: | |
| 491 | + return damon_pa_migrate(r, scheme); | |
| 339 | 492 | case DAMOS_STAT: |
| 340 | 493 | break; |
| 341 | 494 | default: |
| @@ -356,6 +509,10 @@ static int damon_pa_scheme_score(struct damon_ctx *context, · 14 unmodified lines | ||
| 356 | 509 | return damon_hot_score(context, r, scheme); |
| 357 | 510 | case DAMOS_LRU_DEPRIO: |
| 358 | 511 | return damon_cold_score(context, r, scheme); |
| 512 | + case DAMOS_MIGRATE_HOT: | |
| 513 | + return damon_hot_score(context, r, scheme); | |
| 514 | + case DAMOS_MIGRATE_COLD: | |
| 515 | + return damon_cold_score(context, r, scheme); | |
| 359 | 516 | default: |
| 360 | 517 | break; |
| 361 | 518 | } |
| @@ -177,76 +177,65 @@ static struct damos *damon_reclaim_new_scheme(void) · 176 unmodified lines | ||
| 177 | 177 | /* under the quota. */ |
| 178 | 178 | &damon_reclaim_quota, |
| 179 | 179 | /* (De)activate this according to the watermarks. */ |
| 180 | − &damon_reclaim_wmarks); | |
| 181 | −} | |
| 182 | − | |
| 183 | −static void damon_reclaim_copy_quota_status(struct damos_quota *dst, | |
| 184 | − struct damos_quota *src) | |
| 185 | −{ | |
| 186 | − dst->total_charged_sz = src->total_charged_sz; | |
| 187 | − dst->total_charged_ns = src->total_charged_ns; | |
| 188 | − dst->charged_sz = src->charged_sz; | |
| 189 | − dst->charged_from = src->charged_from; | |
| 190 | − dst->charge_target_from = src->charge_target_from; | |
| 191 | − dst->charge_addr_from = src->charge_addr_from; | |
| 192 | − dst->esz_bp = src->esz_bp; | |
| 180 | + &damon_reclaim_wmarks, | |
| 181 | + NUMA_NO_NODE); | |
| 193 | 182 | } |
| 194 | 183 | |
| 195 | 184 | static int damon_reclaim_apply_parameters(void) |
| 196 | 185 | { |
| 197 | − struct damos *scheme, *old_scheme; | |
| 186 | + struct damon_ctx *param_ctx; | |
| 187 | + struct damon_target *param_target; | |
| 188 | + struct damos *scheme; | |
| 198 | 189 | struct damos_quota_goal *goal; |
| 199 | 190 | struct damos_filter *filter; |
| 200 | − int err = 0; | |
| 191 | + int err; | |
| 201 | 192 | |
| 202 | − err = damon_set_attrs(ctx, &damon_reclaim_mon_attrs); | |
| 193 | + err = damon_modules_new_paddr_ctx_target(¶m_ctx, ¶m_target); | |
| 203 | 194 | if (err) |
| 204 | 195 | return err; |
| 205 | 196 | |
| 206 | − /* Will be freed by next 'damon_set_schemes()' below */ | |
| 197 | + err = damon_set_attrs(ctx, &damon_reclaim_mon_attrs); | |
| 198 | + if (err) | |
| 199 | + goto out; | |
| 200 | + | |
| 201 | + err = -ENOMEM; | |
| 207 | 202 | scheme = damon_reclaim_new_scheme(); |
| 208 | 203 | if (!scheme) |
| 209 | − return -ENOMEM; | |
| 210 | − if (!list_empty(&ctx->schemes)) { | |
| 211 | − damon_for_each_scheme(old_scheme, ctx) | |
| 212 | − damon_reclaim_copy_quota_status(&scheme->quota, | |
| 213 | − &old_scheme->quota); | |
| 214 | − } | |
| 204 | + goto out; | |
| 205 | + damon_set_schemes(ctx, &scheme, 1); | |
| 215 | 206 | |
| 216 | 207 | if (quota_mem_pressure_us) { |
| 217 | 208 | goal = damos_new_quota_goal(DAMOS_QUOTA_SOME_MEM_PSI_US, |
| 218 | 209 | quota_mem_pressure_us); |
| 219 | − if (!goal) { | |
| 220 | − damon_destroy_scheme(scheme); | |
| 221 | − return -ENOMEM; | |
| 222 | − } | |
| 210 | + if (!goal) | |
| 211 | + goto out; | |
| 223 | 212 | damos_add_quota_goal(&scheme->quota, goal); |
| 224 | 213 | } |
| 225 | 214 | |
| 226 | 215 | if (quota_autotune_feedback) { |
| 227 | 216 | goal = damos_new_quota_goal(DAMOS_QUOTA_USER_INPUT, 10000); |
| 228 | − if (!goal) { | |
| 229 | − damon_destroy_scheme(scheme); | |
| 230 | − return -ENOMEM; | |
| 231 | − } | |
| 217 | + if (!goal) | |
| 218 | + goto out; | |
| 232 | 219 | goal->current_value = quota_autotune_feedback; |
| 233 | 220 | damos_add_quota_goal(&scheme->quota, goal); |
| 234 | 221 | } |
| 235 | 222 | |
| 236 | 223 | if (skip_anon) { |
| 237 | 224 | filter = damos_new_filter(DAMOS_FILTER_TYPE_ANON, true); |
| 238 | − if (!filter) { | |
| 239 | − /* Will be freed by next 'damon_set_schemes()' below */ | |
| 240 | − damon_destroy_scheme(scheme); | |
| 241 | − return -ENOMEM; | |
| 242 | − } | |
| 225 | + if (!filter) | |
| 226 | + goto out; | |
| 243 | 227 | damos_add_filter(scheme, filter); |
| 244 | 228 | } |
| 245 | − damon_set_schemes(ctx, &scheme, 1); | |
| 246 | 229 | |
| 247 | − return damon_set_region_biggest_system_ram_default(target, | |
| 230 | + err = damon_set_region_biggest_system_ram_default(param_target, | |
| 248 | 231 | &monitor_region_start, |
| 249 | 232 | &monitor_region_end); |
| 233 | + if (err) | |
| 234 | + goto out; | |
| 235 | + err = damon_commit_ctx(ctx, param_ctx); | |
| 236 | +out: | |
| 237 | + damon_destroy_ctx(param_ctx); | |
| 238 | + return err; | |
| 250 | 239 | } |
| 251 | 240 | |
| 252 | 241 | static int damon_reclaim_turn(bool on) |
| @@ -38,7 +38,7 @@ void damon_sysfs_schemes_rm_dirs(struct damon_sysfs_schemes *schemes); · 37 unmodified lines | ||
| 38 | 38 | |
| 39 | 39 | extern const struct kobj_type damon_sysfs_schemes_ktype; |
| 40 | 40 | |
| 41 | −int damon_sysfs_set_schemes(struct damon_ctx *ctx, | |
| 41 | +int damon_sysfs_add_schemes(struct damon_ctx *ctx, | |
| 42 | 42 | struct damon_sysfs_schemes *sysfs_schemes); |
| 43 | 43 | |
| 44 | 44 | void damon_sysfs_schemes_update_stats( |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | */ |
| 7 | 7 | |
| 8 | 8 | #include <linux/slab.h> |
| 9 | +#include <linux/numa.h> | |
| 9 | 10 | |
| 10 | 11 | #include "sysfs-common.h" |
| 11 | 12 | |
| @@ -1445,6 +1446,7 @@ struct damon_sysfs_scheme { · 1433 unmodified lines | ||
| 1445 | 1446 | struct damon_sysfs_scheme_filters *filters; |
| 1446 | 1447 | struct damon_sysfs_stats *stats; |
| 1447 | 1448 | struct damon_sysfs_scheme_regions *tried_regions; |
| 1449 | + int target_nid; | |
| 1448 | 1450 | }; |
| 1449 | 1451 | |
| 1450 | 1452 | /* This should match with enum damos_action */ |
| @@ -1456,6 +1458,8 @@ static const char * const damon_sysfs_damos_action_strs[] = { · 5 unmodified lines | ||
| 1456 | 1458 | "nohugepage", |
| 1457 | 1459 | "lru_prio", |
| 1458 | 1460 | "lru_deprio", |
| 1461 | + "migrate_hot", | |
| 1462 | + "migrate_cold", | |
| 1459 | 1463 | "stat", |
| 1460 | 1464 | }; |
| 1461 | 1465 | |
| @@ -1470,6 +1474,7 @@ static struct damon_sysfs_scheme *damon_sysfs_scheme_alloc( · 8 unmodified lines | ||
| 1470 | 1474 | scheme->kobj = (struct kobject){}; |
| 1471 | 1475 | scheme->action = action; |
| 1472 | 1476 | scheme->apply_interval_us = apply_interval_us; |
| 1477 | + scheme->target_nid = NUMA_NO_NODE; | |
| 1473 | 1478 | return scheme; |
| 1474 | 1479 | } |
| 1475 | 1480 | |
| @@ -1692,6 +1697,28 @@ static ssize_t apply_interval_us_store(struct kobject *kobj, · 216 unmodified lines | ||
| 1692 | 1697 | return err ? err : count; |
| 1693 | 1698 | } |
| 1694 | 1699 | |
| 1700 | +static ssize_t target_nid_show(struct kobject *kobj, | |
| 1701 | + struct kobj_attribute *attr, char *buf) | |
| 1702 | +{ | |
| 1703 | + struct damon_sysfs_scheme *scheme = container_of(kobj, | |
| 1704 | + struct damon_sysfs_scheme, kobj); | |
| 1705 | + | |
| 1706 | + return sysfs_emit(buf, "%d\n", scheme->target_nid); | |
| 1707 | +} | |
| 1708 | + | |
| 1709 | +static ssize_t target_nid_store(struct kobject *kobj, | |
| 1710 | + struct kobj_attribute *attr, const char *buf, size_t count) | |
| 1711 | +{ | |
| 1712 | + struct damon_sysfs_scheme *scheme = container_of(kobj, | |
| 1713 | + struct damon_sysfs_scheme, kobj); | |
| 1714 | + int err = 0; | |
| 1715 | + | |
| 1716 | + /* TODO: error handling for target_nid range. */ | |
| 1717 | + err = kstrtoint(buf, 0, &scheme->target_nid); | |
| 1718 | + | |
| 1719 | + return err ? err : count; | |
| 1720 | +} | |
| 1721 | + | |
| 1695 | 1722 | static void damon_sysfs_scheme_release(struct kobject *kobj) |
| 1696 | 1723 | { |
| 1697 | 1724 | kfree(container_of(kobj, struct damon_sysfs_scheme, kobj)); |
| @@ -1703,9 +1730,13 @@ static struct kobj_attribute damon_sysfs_scheme_action_attr = · 5 unmodified lines | ||
| 1703 | 1730 | static struct kobj_attribute damon_sysfs_scheme_apply_interval_us_attr = |
| 1704 | 1731 | __ATTR_RW_MODE(apply_interval_us, 0600); |
| 1705 | 1732 | |
| 1733 | +static struct kobj_attribute damon_sysfs_scheme_target_nid_attr = | |
| 1734 | + __ATTR_RW_MODE(target_nid, 0600); | |
| 1735 | + | |
| 1706 | 1736 | static struct attribute *damon_sysfs_scheme_attrs[] = { |
| 1707 | 1737 | &damon_sysfs_scheme_action_attr.attr, |
| 1708 | 1738 | &damon_sysfs_scheme_apply_interval_us_attr.attr, |
| 1739 | + &damon_sysfs_scheme_target_nid_attr.attr, | |
| 1709 | 1740 | NULL, |
| 1710 | 1741 | }; |
| 1711 | 1742 | ATTRIBUTE_GROUPS(damon_sysfs_scheme); |
| @@ -1877,14 +1908,10 @@ static int damon_sysfs_memcg_path_to_id(char *memcg_path, unsigned short *id) · 165 unmodified lines | ||
| 1877 | 1908 | return found ? 0 : -EINVAL; |
| 1878 | 1909 | } |
| 1879 | 1910 | |
| 1880 | −static int damon_sysfs_set_scheme_filters(struct damos *scheme, | |
| 1911 | +static int damon_sysfs_add_scheme_filters(struct damos *scheme, | |
| 1881 | 1912 | struct damon_sysfs_scheme_filters *sysfs_filters) |
| 1882 | 1913 | { |
| 1883 | 1914 | int i; |
| 1884 | − struct damos_filter *filter, *next; | |
| 1885 | − | |
| 1886 | − damos_for_each_filter_safe(filter, next, scheme) | |
| 1887 | − damos_destroy_filter(filter); | |
| 1888 | 1915 | |
| 1889 | 1916 | for (i = 0; i < sysfs_filters->nr; i++) { |
| 1890 | 1917 | struct damon_sysfs_scheme_filter *sysfs_filter = |
| @@ -1920,16 +1947,13 @@ static int damon_sysfs_set_scheme_filters(struct damos *scheme, · 29 unmodified lines | ||
| 1920 | 1947 | return 0; |
| 1921 | 1948 | } |
| 1922 | 1949 | |
| 1923 | −static int damos_sysfs_set_quota_score( | |
| 1950 | +static int damos_sysfs_add_quota_score( | |
| 1924 | 1951 | struct damos_sysfs_quota_goals *sysfs_goals, |
| 1925 | 1952 | struct damos_quota *quota) |
| 1926 | 1953 | { |
| 1927 | − struct damos_quota_goal *goal, *next; | |
| 1954 | + struct damos_quota_goal *goal; | |
| 1928 | 1955 | int i; |
| 1929 | 1956 | |
| 1930 | − damos_for_each_quota_goal_safe(goal, next, quota) | |
| 1931 | − damos_destroy_quota_goal(goal); | |
| 1932 | − | |
| 1933 | 1957 | for (i = 0; i < sysfs_goals->nr; i++) { |
| 1934 | 1958 | struct damos_sysfs_quota_goal *sysfs_goal = |
| 1935 | 1959 | sysfs_goals->goals_arr[i]; |
| @@ -1952,10 +1976,13 @@ int damos_sysfs_set_quota_scores(struct damon_sysfs_schemes *sysfs_schemes, · 16 unmodified lines | ||
| 1952 | 1976 | struct damon_ctx *ctx) |
| 1953 | 1977 | { |
| 1954 | 1978 | struct damos *scheme; |
| 1979 | + struct damos_quota quota = {}; | |
| 1955 | 1980 | int i = 0; |
| 1956 | 1981 | |
| 1982 | + INIT_LIST_HEAD("a.goals); | |
| 1957 | 1983 | damon_for_each_scheme(scheme, ctx) { |
| 1958 | 1984 | struct damon_sysfs_scheme *sysfs_scheme; |
| 1985 | + struct damos_quota_goal *g, *g_next; | |
| 1959 | 1986 | int err; |
| 1960 | 1987 | |
| 1961 | 1988 | /* user could have removed the scheme sysfs dir */ |
| @@ -1963,10 +1990,17 @@ int damos_sysfs_set_quota_scores(struct damon_sysfs_schemes *sysfs_schemes, · 1 unmodified line | ||
| 1963 | 1990 | break; |
| 1964 | 1991 | |
| 1965 | 1992 | sysfs_scheme = sysfs_schemes->schemes_arr[i]; |
| 1966 | − err = damos_sysfs_set_quota_score(sysfs_scheme->quotas->goals, | |
| 1967 | − &scheme->quota); | |
| 1993 | + err = damos_sysfs_add_quota_score(sysfs_scheme->quotas->goals, | |
| 1994 | + "a); | |
| 1995 | + if (err) { | |
| 1996 | + damos_for_each_quota_goal_safe(g, g_next, "a) | |
| 1997 | + damos_destroy_quota_goal(g); | |
| 1998 | + return err; | |
| 1999 | + } | |
| 2000 | + err = damos_commit_quota_goals(&scheme->quota, "a); | |
| 2001 | + damos_for_each_quota_goal_safe(g, g_next, "a) | |
| 2002 | + damos_destroy_quota_goal(g); | |
| 1968 | 2003 | if (err) |
| 1969 | − /* kdamond will clean up schemes and terminated */ | |
| 1970 | 2004 | return err; |
| 1971 | 2005 | i++; |
| 1972 | 2006 | } |
| @@ -2031,17 +2065,18 @@ static struct damos *damon_sysfs_mk_scheme( · 58 unmodified lines | ||
| 2031 | 2065 | }; |
| 2032 | 2066 | |
| 2033 | 2067 | scheme = damon_new_scheme(&pattern, sysfs_scheme->action, |
| 2034 | − sysfs_scheme->apply_interval_us, "a, &wmarks); | |
| 2068 | + sysfs_scheme->apply_interval_us, "a, &wmarks, | |
| 2069 | + sysfs_scheme->target_nid); | |
| 2035 | 2070 | if (!scheme) |
| 2036 | 2071 | return NULL; |
| 2037 | 2072 | |
| 2038 | − err = damos_sysfs_set_quota_score(sysfs_quotas->goals, &scheme->quota); | |
| 2073 | + err = damos_sysfs_add_quota_score(sysfs_quotas->goals, &scheme->quota); | |
| 2039 | 2074 | if (err) { |
| 2040 | 2075 | damon_destroy_scheme(scheme); |
| 2041 | 2076 | return NULL; |
| 2042 | 2077 | } |
| 2043 | 2078 | |
| 2044 | − err = damon_sysfs_set_scheme_filters(scheme, sysfs_filters); | |
| 2079 | + err = damon_sysfs_add_scheme_filters(scheme, sysfs_filters); | |
| 2045 | 2080 | if (err) { |
| 2046 | 2081 | damon_destroy_scheme(scheme); |
| 2047 | 2082 | return NULL; |
| @@ -2049,66 +2084,12 @@ static struct damos *damon_sysfs_mk_scheme( · 1 unmodified line | ||
| 2049 | 2084 | return scheme; |
| 2050 | 2085 | } |
| 2051 | 2086 | |
| 2052 | −static void damon_sysfs_update_scheme(struct damos *scheme, | |
| 2053 | − struct damon_sysfs_scheme *sysfs_scheme) | |
| 2054 | −{ | |
| 2055 | − struct damon_sysfs_access_pattern *access_pattern = | |
| 2056 | − sysfs_scheme->access_pattern; | |
| 2057 | − struct damon_sysfs_quotas *sysfs_quotas = sysfs_scheme->quotas; | |
| 2058 | − struct damon_sysfs_weights *sysfs_weights = sysfs_quotas->weights; | |
| 2059 | − struct damon_sysfs_watermarks *sysfs_wmarks = sysfs_scheme->watermarks; | |
| 2060 | − int err; | |
| 2061 | − | |
| 2062 | − scheme->pattern.min_sz_region = access_pattern->sz->min; | |
| 2063 | − scheme->pattern.max_sz_region = access_pattern->sz->max; | |
| 2064 | − scheme->pattern.min_nr_accesses = access_pattern->nr_accesses->min; | |
| 2065 | − scheme->pattern.max_nr_accesses = access_pattern->nr_accesses->max; | |
| 2066 | − scheme->pattern.min_age_region = access_pattern->age->min; | |
| 2067 | − scheme->pattern.max_age_region = access_pattern->age->max; | |
| 2068 | − | |
| 2069 | − scheme->action = sysfs_scheme->action; | |
| 2070 | − scheme->apply_interval_us = sysfs_scheme->apply_interval_us; | |
| 2071 | − | |
| 2072 | − scheme->quota.ms = sysfs_quotas->ms; | |
| 2073 | − scheme->quota.sz = sysfs_quotas->sz; | |
| 2074 | − scheme->quota.reset_interval = sysfs_quotas->reset_interval_ms; | |
| 2075 | − scheme->quota.weight_sz = sysfs_weights->sz; | |
| 2076 | − scheme->quota.weight_nr_accesses = sysfs_weights->nr_accesses; | |
| 2077 | − scheme->quota.weight_age = sysfs_weights->age; | |
| 2078 | − | |
| 2079 | − err = damos_sysfs_set_quota_score(sysfs_quotas->goals, &scheme->quota); | |
| 2080 | − if (err) { | |
| 2081 | − damon_destroy_scheme(scheme); | |
| 2082 | − return; | |
| 2083 | − } | |
| 2084 | − | |
| 2085 | − scheme->wmarks.metric = sysfs_wmarks->metric; | |
| 2086 | − scheme->wmarks.interval = sysfs_wmarks->interval_us; | |
| 2087 | − scheme->wmarks.high = sysfs_wmarks->high; | |
| 2088 | − scheme->wmarks.mid = sysfs_wmarks->mid; | |
| 2089 | − scheme->wmarks.low = sysfs_wmarks->low; | |
| 2090 | − | |
| 2091 | − err = damon_sysfs_set_scheme_filters(scheme, sysfs_scheme->filters); | |
| 2092 | − if (err) | |
| 2093 | − damon_destroy_scheme(scheme); | |
| 2094 | −} | |
| 2095 | − | |
| 2096 | −int damon_sysfs_set_schemes(struct damon_ctx *ctx, | |
| 2087 | +int damon_sysfs_add_schemes(struct damon_ctx *ctx, | |
| 2097 | 2088 | struct damon_sysfs_schemes *sysfs_schemes) |
| 2098 | 2089 | { |
| 2099 | − struct damos *scheme, *next; | |
| 2100 | − int i = 0; | |
| 2101 | − | |
| 2102 | − damon_for_each_scheme_safe(scheme, next, ctx) { | |
| 2103 | − if (i < sysfs_schemes->nr) | |
| 2104 | − damon_sysfs_update_scheme(scheme, | |
| 2105 | − sysfs_schemes->schemes_arr[i]); | |
| 2106 | − else | |
| 2107 | − damon_destroy_scheme(scheme); | |
| 2108 | − i++; | |
| 2109 | − } | |
| 2090 | + int i; | |
| 2110 | 2091 | |
| 2111 | − for (; i < sysfs_schemes->nr; i++) { | |
| 2092 | + for (i = 0; i < sysfs_schemes->nr; i++) { | |
| 2112 | 2093 | struct damos *scheme, *next; |
| 2113 | 2094 | |
| 2114 | 2095 | scheme = damon_sysfs_mk_scheme(sysfs_schemes->schemes_arr[i]); |
| @@ -38,7 +38,7 @@ static int __damon_sysfs_test_get_any_pid(int min, int max) · 37 unmodified lines | ||
| 38 | 38 | return -1; |
| 39 | 39 | } |
| 40 | 40 | |
| 41 | −static void damon_sysfs_test_set_targets(struct kunit *test) | |
| 41 | +static void damon_sysfs_test_add_targets(struct kunit *test) | |
| 42 | 42 | { |
| 43 | 43 | struct damon_sysfs_targets *sysfs_targets; |
| 44 | 44 | struct damon_sysfs_target *sysfs_target; |
| @@ -56,13 +56,13 @@ static void damon_sysfs_test_set_targets(struct kunit *test) · 11 unmodified lines | ||
| 56 | 56 | |
| 57 | 57 | ctx = damon_new_ctx(); |
| 58 | 58 | |
| 59 | − damon_sysfs_set_targets(ctx, sysfs_targets); | |
| 59 | + damon_sysfs_add_targets(ctx, sysfs_targets); | |
| 60 | 60 | KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx)); |
| 61 | 61 | |
| 62 | 62 | sysfs_target->pid = __damon_sysfs_test_get_any_pid( |
| 63 | 63 | sysfs_target->pid + 1, 200); |
| 64 | − damon_sysfs_set_targets(ctx, sysfs_targets); | |
| 65 | − KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx)); | |
| 64 | + damon_sysfs_add_targets(ctx, sysfs_targets); | |
| 65 | + KUNIT_EXPECT_EQ(test, 2u, nr_damon_targets(ctx)); | |
| 66 | 66 | |
| 67 | 67 | damon_destroy_ctx(ctx); |
| 68 | 68 | kfree(sysfs_targets->targets_arr); |
| @@ -71,7 +71,7 @@ static void damon_sysfs_test_set_targets(struct kunit *test) · 2 unmodified lines | ||
| 71 | 71 | } |
| 72 | 72 | |
| 73 | 73 | static struct kunit_case damon_sysfs_test_cases[] = { |
| 74 | − KUNIT_CASE(damon_sysfs_test_set_targets), | |
| 74 | + KUNIT_CASE(damon_sysfs_test_add_targets), | |
| 75 | 75 | {}, |
| 76 | 76 | }; |
| 77 | 77 | |
| @@ -1162,72 +1162,16 @@ destroy_targets_out: · 1161 unmodified lines | ||
| 1162 | 1162 | return err; |
| 1163 | 1163 | } |
| 1164 | 1164 | |
| 1165 | −static int damon_sysfs_update_target_pid(struct damon_target *target, int pid) | |
| 1166 | −{ | |
| 1167 | − struct pid *pid_new; | |
| 1168 | − | |
| 1169 | − pid_new = find_get_pid(pid); | |
| 1170 | − if (!pid_new) | |
| 1171 | − return -EINVAL; | |
| 1172 | − | |
| 1173 | − if (pid_new == target->pid) { | |
| 1174 | − put_pid(pid_new); | |
| 1175 | − return 0; | |
| 1176 | − } | |
| 1177 | − | |
| 1178 | − put_pid(target->pid); | |
| 1179 | − target->pid = pid_new; | |
| 1180 | − return 0; | |
| 1181 | −} | |
| 1182 | − | |
| 1183 | −static int damon_sysfs_update_target(struct damon_target *target, | |
| 1184 | − struct damon_ctx *ctx, | |
| 1185 | − struct damon_sysfs_target *sys_target) | |
| 1186 | −{ | |
| 1187 | − int err = 0; | |
| 1188 | − | |
| 1189 | − if (damon_target_has_pid(ctx)) { | |
| 1190 | − err = damon_sysfs_update_target_pid(target, sys_target->pid); | |
| 1191 | − if (err) | |
| 1192 | − return err; | |
| 1193 | − } | |
| 1194 | − | |
| 1195 | − /* | |
| 1196 | − * Do monitoring target region boundary update only if one or more | |
| 1197 | − * regions are set by the user. This is for keeping current monitoring | |
| 1198 | − * target results and range easier, especially for dynamic monitoring | |
| 1199 | − * target regions update ops like 'vaddr'. | |
| 1200 | − */ | |
| 1201 | − if (sys_target->regions->nr) | |
| 1202 | − err = damon_sysfs_set_regions(target, sys_target->regions); | |
| 1203 | − return err; | |
| 1204 | −} | |
| 1205 | − | |
| 1206 | −static int damon_sysfs_set_targets(struct damon_ctx *ctx, | |
| 1165 | +static int damon_sysfs_add_targets(struct damon_ctx *ctx, | |
| 1207 | 1166 | struct damon_sysfs_targets *sysfs_targets) |
| 1208 | 1167 | { |
| 1209 | − struct damon_target *t, *next; | |
| 1210 | − int i = 0, err; | |
| 1168 | + int i, err; | |
| 1211 | 1169 | |
| 1212 | 1170 | /* Multiple physical address space monitoring targets makes no sense */ |
| 1213 | 1171 | if (ctx->ops.id == DAMON_OPS_PADDR && sysfs_targets->nr > 1) |
| 1214 | 1172 | return -EINVAL; |
| 1215 | 1173 | |
| 1216 | − damon_for_each_target_safe(t, next, ctx) { | |
| 1217 | − if (i < sysfs_targets->nr) { | |
| 1218 | − err = damon_sysfs_update_target(t, ctx, | |
| 1219 | − sysfs_targets->targets_arr[i]); | |
| 1220 | − if (err) | |
| 1221 | − return err; | |
| 1222 | − } else { | |
| 1223 | − if (damon_target_has_pid(ctx)) | |
| 1224 | − put_pid(t->pid); | |
| 1225 | − damon_destroy_target(t); | |
| 1226 | − } | |
| 1227 | − i++; | |
| 1228 | − } | |
| 1229 | − | |
| 1230 | − for (; i < sysfs_targets->nr; i++) { | |
| 1174 | + for (i = 0; i < sysfs_targets->nr; i++) { | |
| 1231 | 1175 | struct damon_sysfs_target *st = sysfs_targets->targets_arr[i]; |
| 1232 | 1176 | |
| 1233 | 1177 | err = damon_sysfs_add_target(st, ctx); |
| @@ -1339,12 +1283,15 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx, · 105 unmodified lines | ||
| 1339 | 1283 | err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs); |
| 1340 | 1284 | if (err) |
| 1341 | 1285 | return err; |
| 1342 | − err = damon_sysfs_set_targets(ctx, sys_ctx->targets); | |
| 1286 | + err = damon_sysfs_add_targets(ctx, sys_ctx->targets); | |
| 1343 | 1287 | if (err) |
| 1344 | 1288 | return err; |
| 1345 | − return damon_sysfs_set_schemes(ctx, sys_ctx->schemes); | |
| 1289 | + return damon_sysfs_add_schemes(ctx, sys_ctx->schemes); | |
| 1346 | 1290 | } |
| 1347 | 1291 | |
| 1292 | +static struct damon_ctx *damon_sysfs_build_ctx( | |
| 1293 | + struct damon_sysfs_context *sys_ctx); | |
| 1294 | + | |
| 1348 | 1295 | /* |
| 1349 | 1296 | * damon_sysfs_commit_input() - Commit user inputs to a running kdamond. |
| 1350 | 1297 | * @kdamond: The kobject wrapper for the associated kdamond. |
| @@ -1353,14 +1300,22 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx, · 2 unmodified lines | ||
| 1353 | 1300 | */ |
| 1354 | 1301 | static int damon_sysfs_commit_input(struct damon_sysfs_kdamond *kdamond) |
| 1355 | 1302 | { |
| 1303 | + struct damon_ctx *param_ctx; | |
| 1304 | + int err; | |
| 1305 | + | |
| 1356 | 1306 | if (!damon_sysfs_kdamond_running(kdamond)) |
| 1357 | 1307 | return -EINVAL; |
| 1358 | 1308 | /* TODO: Support multiple contexts per kdamond */ |
| 1359 | 1309 | if (kdamond->contexts->nr != 1) |
| 1360 | 1310 | return -EINVAL; |
| 1361 | 1311 | |
| 1362 | − return damon_sysfs_apply_inputs(kdamond->damon_ctx, | |
| 1363 | − kdamond->contexts->contexts_arr[0]); | |
| 1312 | + param_ctx = damon_sysfs_build_ctx(kdamond->contexts->contexts_arr[0]); | |
| 1313 | + if (IS_ERR(param_ctx)) | |
| 1314 | + return PTR_ERR(param_ctx); | |
| 1315 | + err = damon_commit_ctx(kdamond->damon_ctx, param_ctx); | |
| 1316 | + damon_sysfs_destroy_targets(param_ctx); | |
| 1317 | + damon_destroy_ctx(param_ctx); | |
| 1318 | + return err; | |
| 1364 | 1319 | } |
| 1365 | 1320 | |
| 1366 | 1321 | static int damon_sysfs_commit_schemes_quota_goals( |
| @@ -339,7 +339,7 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, · 338 unmodified lines | ||
| 339 | 339 | struct vm_area_struct *vma, unsigned long addr) |
| 340 | 340 | { |
| 341 | 341 | bool referenced = false; |
| 342 | − pte_t entry = huge_ptep_get(pte); | |
| 342 | + pte_t entry = huge_ptep_get(mm, addr, pte); | |
| 343 | 343 | struct folio *folio = pfn_folio(pte_pfn(entry)); |
| 344 | 344 | unsigned long psize = huge_page_size(hstate_vma(vma)); |
| 345 | 345 | |
| @@ -373,7 +373,7 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask, · 27 unmodified lines | ||
| 373 | 373 | pte_t entry; |
| 374 | 374 | |
| 375 | 375 | ptl = huge_pte_lock(h, walk->mm, pte); |
| 376 | − entry = huge_ptep_get(pte); | |
| 376 | + entry = huge_ptep_get(walk->mm, addr, pte); | |
| 377 | 377 | if (!pte_present(entry)) |
| 378 | 378 | goto out; |
| 379 | 379 | |
| @@ -509,7 +509,7 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask, · 129 unmodified lines | ||
| 509 | 509 | pte_t entry; |
| 510 | 510 | |
| 511 | 511 | ptl = huge_pte_lock(h, walk->mm, pte); |
| 512 | − entry = huge_ptep_get(pte); | |
| 512 | + entry = huge_ptep_get(walk->mm, addr, pte); | |
| 513 | 513 | if (!pte_present(entry)) |
| 514 | 514 | goto out; |
| 515 | 515 | |
| @@ -144,4 +144,5 @@ static void dmapool_exit(void) · 143 unmodified lines | ||
| 144 | 144 | |
| 145 | 145 | module_init(dmapool_checks); |
| 146 | 146 | module_exit(dmapool_exit); |
| 147 | +MODULE_DESCRIPTION("dma_pool timing test"); | |
| 147 | 148 | MODULE_LICENSE("GPL"); |
| @@ -1,5 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | #include <linux/fault-inject.h> |
| 3 | +#include <linux/error-injection.h> | |
| 3 | 4 | #include <linux/mm.h> |
| 4 | 5 | |
| 5 | 6 | static struct { |
| @@ -21,7 +22,7 @@ static int __init setup_fail_page_alloc(char *str) · 15 unmodified lines | ||
| 21 | 22 | } |
| 22 | 23 | __setup("fail_page_alloc=", setup_fail_page_alloc); |
| 23 | 24 | |
| 24 | −bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) | |
| 25 | +bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) | |
| 25 | 26 | { |
| 26 | 27 | int flags = 0; |
| 27 | 28 | |
| @@ -41,6 +42,7 @@ bool __should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) · 13 unmodified lines | ||
| 41 | 42 | |
| 42 | 43 | return should_fail_ex(&fail_page_alloc.attr, 1 << order, flags); |
| 43 | 44 | } |
| 45 | +ALLOW_ERROR_INJECTION(should_fail_alloc_page, TRUE); | |
| 44 | 46 | |
| 45 | 47 | #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS |
| 46 | 48 | |
| @@ -1,5 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | #include <linux/fault-inject.h> |
| 3 | +#include <linux/error-injection.h> | |
| 3 | 4 | #include <linux/slab.h> |
| 4 | 5 | #include <linux/mm.h> |
| 5 | 6 | #include "slab.h" |
| @@ -14,23 +15,23 @@ static struct { · 8 unmodified lines | ||
| 14 | 15 | .cache_filter = false, |
| 15 | 16 | }; |
| 16 | 17 | |
| 17 | −bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags) | |
| 18 | +int should_failslab(struct kmem_cache *s, gfp_t gfpflags) | |
| 18 | 19 | { |
| 19 | 20 | int flags = 0; |
| 20 | 21 | |
| 21 | 22 | /* No fault-injection for bootstrap cache */ |
| 22 | 23 | if (unlikely(s == kmem_cache)) |
| 23 | − return false; | |
| 24 | + return 0; | |
| 24 | 25 | |
| 25 | 26 | if (gfpflags & __GFP_NOFAIL) |
| 26 | − return false; | |
| 27 | + return 0; | |
| 27 | 28 | |
| 28 | 29 | if (failslab.ignore_gfp_reclaim && |
| 29 | 30 | (gfpflags & __GFP_DIRECT_RECLAIM)) |
| 30 | − return false; | |
| 31 | + return 0; | |
| 31 | 32 | |
| 32 | 33 | if (failslab.cache_filter && !(s->flags & SLAB_FAILSLAB)) |
| 33 | − return false; | |
| 34 | + return 0; | |
| 34 | 35 | |
| 35 | 36 | /* |
| 36 | 37 | * In some cases, it expects to specify __GFP_NOWARN |
| @@ -41,8 +42,9 @@ bool __should_failslab(struct kmem_cache *s, gfp_t gfpflags) · 4 unmodified lines | ||
| 41 | 42 | if (gfpflags & __GFP_NOWARN) |
| 42 | 43 | flags |= FAULT_NOWARN; |
| 43 | 44 | |
| 44 | − return should_fail_ex(&failslab.attr, s->object_size, flags); | |
| 45 | + return should_fail_ex(&failslab.attr, s->object_size, flags) ? -ENOMEM : 0; | |
| 45 | 46 | } |
| 47 | +ALLOW_ERROR_INJECTION(should_failslab, ERRNO); | |
| 46 | 48 | |
| 47 | 49 | static int __init setup_failslab(char *str) |
| 48 | 50 | { |
| @@ -177,7 +177,7 @@ static void filemap_unaccount_folio(struct address_space *mapping, · 176 unmodified lines | ||
| 177 | 177 | * and we'd rather not leak it: if we're wrong, |
| 178 | 178 | * another bad page check should catch it later. |
| 179 | 179 | */ |
| 180 | − page_mapcount_reset(&folio->page); | |
| 180 | + atomic_set(&folio->_mapcount, -1); | |
| 181 | 181 | folio_ref_sub(folio, mapcount); |
| 182 | 182 | } |
| 183 | 183 | } |
| @@ -1752,12 +1752,12 @@ pgoff_t page_cache_next_miss(struct address_space *mapping, · 1568 unmodified lines | ||
| 1752 | 1752 | while (max_scan--) { |
| 1753 | 1753 | void *entry = xas_next(&xas); |
| 1754 | 1754 | if (!entry || xa_is_value(entry)) |
| 1755 | − break; | |
| 1755 | + return xas.xa_index; | |
| 1756 | 1756 | if (xas.xa_index == 0) |
| 1757 | − break; | |
| 1757 | + return 0; | |
| 1758 | 1758 | } |
| 1759 | 1759 | |
| 1760 | − return xas.xa_index; | |
| 1760 | + return index + max_scan; | |
| 1761 | 1761 | } |
| 1762 | 1762 | EXPORT_SYMBOL(page_cache_next_miss); |
| 1763 | 1763 | |
| @@ -10,12 +10,6 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/swap.h> |
| 11 | 11 | #include "internal.h" |
| 12 | 12 | |
| 13 | −struct address_space *page_mapping(struct page *page) | |
| 14 | −{ | |
| 15 | − return folio_mapping(page_folio(page)); | |
| 16 | −} | |
| 17 | −EXPORT_SYMBOL(page_mapping); | |
| 18 | − | |
| 19 | 13 | void unlock_page(struct page *page) |
| 20 | 14 | { |
| 21 | 15 | return folio_unlock(page_folio(page)); |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <linux/spinlock.h> |
| 6 | 6 | |
| 7 | 7 | #include <linux/mm.h> |
| 8 | +#include <linux/memfd.h> | |
| 8 | 9 | #include <linux/memremap.h> |
| 9 | 10 | #include <linux/pagemap.h> |
| 10 | 11 | #include <linux/rmap.h> |
| @@ -17,6 +18,7 @@ · 6 unmodified lines | ||
| 17 | 18 | #include <linux/hugetlb.h> |
| 18 | 19 | #include <linux/migrate.h> |
| 19 | 20 | #include <linux/mm_inline.h> |
| 21 | +#include <linux/pagevec.h> | |
| 20 | 22 | #include <linux/sched/mm.h> |
| 21 | 23 | #include <linux/shmem_fs.h> |
| 22 | 24 | |
| @@ -188,6 +190,19 @@ void unpin_user_page(struct page *page) · 165 unmodified lines | ||
| 188 | 190 | } |
| 189 | 191 | EXPORT_SYMBOL(unpin_user_page); |
| 190 | 192 | |
| 193 | +/** | |
| 194 | + * unpin_folio() - release a dma-pinned folio | |
| 195 | + * @folio: pointer to folio to be released | |
| 196 | + * | |
| 197 | + * Folios that were pinned via memfd_pin_folios() or other similar routines | |
| 198 | + * must be released either using unpin_folio() or unpin_folios(). | |
| 199 | + */ | |
| 200 | +void unpin_folio(struct folio *folio) | |
| 201 | +{ | |
| 202 | + gup_put_folio(folio, 1, FOLL_PIN); | |
| 203 | +} | |
| 204 | +EXPORT_SYMBOL_GPL(unpin_folio); | |
| 205 | + | |
| 191 | 206 | /** |
| 192 | 207 | * folio_add_pin - Try to get an additional pin on a pinned folio |
| 193 | 208 | * @folio: The folio to be pinned |
| @@ -290,7 +305,7 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, · 96 unmodified lines | ||
| 290 | 305 | * 1) This code sees the page as already dirty, so it |
| 291 | 306 | * skips the call to set_page_dirty(). That could happen |
| 292 | 307 | * because clear_page_dirty_for_io() called |
| 293 | − * page_mkclean(), followed by set_page_dirty(). | |
| 308 | + * folio_mkclean(), followed by set_page_dirty(). | |
| 294 | 309 | * However, now the page is going to get written back, |
| 295 | 310 | * which meets the original intention of setting it |
| 296 | 311 | * dirty, so all is well: clear_page_dirty_for_io() goes |
| @@ -400,6 +415,40 @@ void unpin_user_pages(struct page **pages, unsigned long npages) · 103 unmodified lines | ||
| 400 | 415 | } |
| 401 | 416 | EXPORT_SYMBOL(unpin_user_pages); |
| 402 | 417 | |
| 418 | +/** | |
| 419 | + * unpin_folios() - release an array of gup-pinned folios. | |
| 420 | + * @folios: array of folios to be marked dirty and released. | |
| 421 | + * @nfolios: number of folios in the @folios array. | |
| 422 | + * | |
| 423 | + * For each folio in the @folios array, release the folio using gup_put_folio. | |
| 424 | + * | |
| 425 | + * Please see the unpin_folio() documentation for details. | |
| 426 | + */ | |
| 427 | +void unpin_folios(struct folio **folios, unsigned long nfolios) | |
| 428 | +{ | |
| 429 | + unsigned long i = 0, j; | |
| 430 | + | |
| 431 | + /* | |
| 432 | + * If this WARN_ON() fires, then the system *might* be leaking folios | |
| 433 | + * (by leaving them pinned), but probably not. More likely, gup/pup | |
| 434 | + * returned a hard -ERRNO error to the caller, who erroneously passed | |
| 435 | + * it here. | |
| 436 | + */ | |
| 437 | + if (WARN_ON(IS_ERR_VALUE(nfolios))) | |
| 438 | + return; | |
| 439 | + | |
| 440 | + while (i < nfolios) { | |
| 441 | + for (j = i + 1; j < nfolios; j++) | |
| 442 | + if (folios[i] != folios[j]) | |
| 443 | + break; | |
| 444 | + | |
| 445 | + if (folios[i]) | |
| 446 | + gup_put_folio(folios[i], j - i, FOLL_PIN); | |
| 447 | + i = j; | |
| 448 | + } | |
| 449 | +} | |
| 450 | +EXPORT_SYMBOL_GPL(unpin_folios); | |
| 451 | + | |
| 403 | 452 | /* |
| 404 | 453 | * Set the MMF_HAS_PINNED if not set yet; after set it'll be there for the mm's |
| 405 | 454 | * lifecycle. Avoid setting the bit unless necessary, or it might cause write |
| @@ -413,7 +462,7 @@ static inline void mm_set_has_pinned_flag(unsigned long *mm_flags) · 7 unmodified lines | ||
| 413 | 462 | |
| 414 | 463 | #ifdef CONFIG_MMU |
| 415 | 464 | |
| 416 | −#if defined(CONFIG_ARCH_HAS_HUGEPD) || defined(CONFIG_HAVE_GUP_FAST) | |
| 465 | +#ifdef CONFIG_HAVE_GUP_FAST | |
| 417 | 466 | static int record_subpages(struct page *page, unsigned long sz, |
| 418 | 467 | unsigned long addr, unsigned long end, |
| 419 | 468 | struct page **pages) |
| @@ -523,154 +572,7 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs, · 103 unmodified lines | ||
| 523 | 572 | |
| 524 | 573 | return folio; |
| 525 | 574 | } |
| 526 | −#endif /* CONFIG_ARCH_HAS_HUGEPD || CONFIG_HAVE_GUP_FAST */ | |
| 527 | − | |
| 528 | −#ifdef CONFIG_ARCH_HAS_HUGEPD | |
| 529 | −static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, | |
| 530 | − unsigned long sz) | |
| 531 | −{ | |
| 532 | − unsigned long __boundary = (addr + sz) & ~(sz-1); | |
| 533 | − return (__boundary - 1 < end - 1) ? __boundary : end; | |
| 534 | −} | |
| 535 | − | |
| 536 | −/* | |
| 537 | − * Returns 1 if succeeded, 0 if failed, -EMLINK if unshare needed. | |
| 538 | − * | |
| 539 | − * NOTE: for the same entry, gup-fast and gup-slow can return different | |
| 540 | − * results (0 v.s. -EMLINK) depending on whether vma is available. This is | |
| 541 | − * the expected behavior, where we simply want gup-fast to fallback to | |
| 542 | − * gup-slow to take the vma reference first. | |
| 543 | − */ | |
| 544 | −static int gup_hugepte(struct vm_area_struct *vma, pte_t *ptep, unsigned long sz, | |
| 545 | − unsigned long addr, unsigned long end, unsigned int flags, | |
| 546 | − struct page **pages, int *nr, bool fast) | |
| 547 | −{ | |
| 548 | − unsigned long pte_end; | |
| 549 | − struct page *page; | |
| 550 | − struct folio *folio; | |
| 551 | − pte_t pte; | |
| 552 | − int refs; | |
| 553 | − | |
| 554 | − pte_end = (addr + sz) & ~(sz-1); | |
| 555 | − if (pte_end < end) | |
| 556 | − end = pte_end; | |
| 557 | − | |
| 558 | − pte = huge_ptep_get(ptep); | |
| 559 | − | |
| 560 | − if (!pte_access_permitted(pte, flags & FOLL_WRITE)) | |
| 561 | − return 0; | |
| 562 | − | |
| 563 | − /* hugepages are never "special" */ | |
| 564 | − VM_BUG_ON(!pfn_valid(pte_pfn(pte))); | |
| 565 | − | |
| 566 | − page = pte_page(pte); | |
| 567 | − refs = record_subpages(page, sz, addr, end, pages + *nr); | |
| 568 | − | |
| 569 | − if (fast) { | |
| 570 | − folio = try_grab_folio_fast(page, refs, flags); | |
| 571 | − if (!folio) | |
| 572 | − return 0; | |
| 573 | − } else { | |
| 574 | − folio = page_folio(page); | |
| 575 | − if (try_grab_folio(folio, refs, flags)) | |
| 576 | − return 0; | |
| 577 | − } | |
| 578 | − | |
| 579 | − if (unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) { | |
| 580 | − gup_put_folio(folio, refs, flags); | |
| 581 | − return 0; | |
| 582 | − } | |
| 583 | − | |
| 584 | − if (!pte_write(pte) && gup_must_unshare(vma, flags, &folio->page)) { | |
| 585 | − gup_put_folio(folio, refs, flags); | |
| 586 | − return -EMLINK; | |
| 587 | − } | |
| 588 | − | |
| 589 | − *nr += refs; | |
| 590 | − folio_set_referenced(folio); | |
| 591 | − return 1; | |
| 592 | −} | |
| 593 | − | |
| 594 | −/* | |
| 595 | − * NOTE: currently GUP for a hugepd is only possible on hugetlbfs file | |
| 596 | − * systems on Power, which does not have issue with folio writeback against | |
| 597 | − * GUP updates. When hugepd will be extended to support non-hugetlbfs or | |
| 598 | − * even anonymous memory, we need to do extra check as what we do with most | |
| 599 | − * of the other folios. See writable_file_mapping_allowed() and | |
| 600 | − * gup_fast_folio_allowed() for more information. | |
| 601 | − */ | |
| 602 | −static int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, | |
| 603 | − unsigned long addr, unsigned int pdshift, | |
| 604 | − unsigned long end, unsigned int flags, | |
| 605 | − struct page **pages, int *nr, bool fast) | |
| 606 | −{ | |
| 607 | − pte_t *ptep; | |
| 608 | − unsigned long sz = 1UL << hugepd_shift(hugepd); | |
| 609 | − unsigned long next; | |
| 610 | − int ret; | |
| 611 | − | |
| 612 | − ptep = hugepte_offset(hugepd, addr, pdshift); | |
| 613 | − do { | |
| 614 | − next = hugepte_addr_end(addr, end, sz); | |
| 615 | − ret = gup_hugepte(vma, ptep, sz, addr, end, flags, pages, nr, | |
| 616 | − fast); | |
| 617 | − if (ret != 1) | |
| 618 | − return ret; | |
| 619 | − } while (ptep++, addr = next, addr != end); | |
| 620 | − | |
| 621 | − return 1; | |
| 622 | −} | |
| 623 | − | |
| 624 | −static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, | |
| 625 | − unsigned long addr, unsigned int pdshift, | |
| 626 | − unsigned int flags, | |
| 627 | − struct follow_page_context *ctx) | |
| 628 | −{ | |
| 629 | − struct page *page; | |
| 630 | − struct hstate *h; | |
| 631 | − spinlock_t *ptl; | |
| 632 | − int nr = 0, ret; | |
| 633 | − pte_t *ptep; | |
| 634 | − | |
| 635 | − /* Only hugetlb supports hugepd */ | |
| 636 | − if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma))) | |
| 637 | − return ERR_PTR(-EFAULT); | |
| 638 | − | |
| 639 | − h = hstate_vma(vma); | |
| 640 | − ptep = hugepte_offset(hugepd, addr, pdshift); | |
| 641 | − ptl = huge_pte_lock(h, vma->vm_mm, ptep); | |
| 642 | − ret = gup_hugepd(vma, hugepd, addr, pdshift, addr + PAGE_SIZE, | |
| 643 | − flags, &page, &nr, false); | |
| 644 | − spin_unlock(ptl); | |
| 645 | − | |
| 646 | − if (ret == 1) { | |
| 647 | − /* GUP succeeded */ | |
| 648 | − WARN_ON_ONCE(nr != 1); | |
| 649 | − ctx->page_mask = (1U << huge_page_order(h)) - 1; | |
| 650 | − return page; | |
| 651 | − } | |
| 652 | − | |
| 653 | − /* ret can be either 0 (translates to NULL) or negative */ | |
| 654 | − return ERR_PTR(ret); | |
| 655 | −} | |
| 656 | −#else /* CONFIG_ARCH_HAS_HUGEPD */ | |
| 657 | −static inline int gup_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, | |
| 658 | − unsigned long addr, unsigned int pdshift, | |
| 659 | − unsigned long end, unsigned int flags, | |
| 660 | − struct page **pages, int *nr, bool fast) | |
| 661 | −{ | |
| 662 | − return 0; | |
| 663 | −} | |
| 664 | − | |
| 665 | −static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, | |
| 666 | − unsigned long addr, unsigned int pdshift, | |
| 667 | − unsigned int flags, | |
| 668 | − struct follow_page_context *ctx) | |
| 669 | −{ | |
| 670 | − return NULL; | |
| 671 | −} | |
| 672 | −#endif /* CONFIG_ARCH_HAS_HUGEPD */ | |
| 673 | − | |
| 575 | +#endif /* CONFIG_HAVE_GUP_FAST */ | |
| 674 | 576 | |
| 675 | 577 | static struct page *no_page_table(struct vm_area_struct *vma, |
| 676 | 578 | unsigned int flags, unsigned long address) |
| @@ -786,7 +688,7 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, · 109 unmodified lines | ||
| 786 | 688 | return false; |
| 787 | 689 | |
| 788 | 690 | /* ... and a write-fault isn't required for other reasons. */ |
| 789 | − if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd)) | |
| 691 | + if (pmd_needs_soft_dirty_wp(vma, pmd)) | |
| 790 | 692 | return false; |
| 791 | 693 | return !userfaultfd_huge_pmd_wp(vma, pmd); |
| 792 | 694 | } |
| @@ -907,7 +809,7 @@ static inline bool can_follow_write_pte(pte_t pte, struct page *page, · 114 unmodified lines | ||
| 907 | 809 | return false; |
| 908 | 810 | |
| 909 | 811 | /* ... and a write-fault isn't required for other reasons. */ |
| 910 | − if (vma_soft_dirty_enabled(vma) && !pte_soft_dirty(pte)) | |
| 812 | + if (pte_needs_soft_dirty_wp(vma, pte)) | |
| 911 | 813 | return false; |
| 912 | 814 | return !userfaultfd_pte_wp(vma, pte); |
| 913 | 815 | } |
| @@ -1040,9 +942,6 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, · 126 unmodified lines | ||
| 1040 | 942 | return no_page_table(vma, flags, address); |
| 1041 | 943 | if (!pmd_present(pmdval)) |
| 1042 | 944 | return no_page_table(vma, flags, address); |
| 1043 | − if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval))))) | |
| 1044 | − return follow_hugepd(vma, __hugepd(pmd_val(pmdval)), | |
| 1045 | − address, PMD_SHIFT, flags, ctx); | |
| 1046 | 945 | if (pmd_devmap(pmdval)) { |
| 1047 | 946 | ptl = pmd_lock(mm, pmd); |
| 1048 | 947 | page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); |
| @@ -1093,9 +992,6 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, · 44 unmodified lines | ||
| 1093 | 992 | pud = READ_ONCE(*pudp); |
| 1094 | 993 | if (!pud_present(pud)) |
| 1095 | 994 | return no_page_table(vma, flags, address); |
| 1096 | − if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) | |
| 1097 | − return follow_hugepd(vma, __hugepd(pud_val(pud)), | |
| 1098 | − address, PUD_SHIFT, flags, ctx); | |
| 1099 | 995 | if (pud_leaf(pud)) { |
| 1100 | 996 | ptl = pud_lock(mm, pudp); |
| 1101 | 997 | page = follow_huge_pud(vma, address, pudp, flags, ctx); |
| @@ -1121,10 +1017,6 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, · 19 unmodified lines | ||
| 1121 | 1017 | p4d = READ_ONCE(*p4dp); |
| 1122 | 1018 | BUILD_BUG_ON(p4d_leaf(p4d)); |
| 1123 | 1019 | |
| 1124 | − if (unlikely(is_hugepd(__hugepd(p4d_val(p4d))))) | |
| 1125 | − return follow_hugepd(vma, __hugepd(p4d_val(p4d)), | |
| 1126 | − address, P4D_SHIFT, flags, ctx); | |
| 1127 | − | |
| 1128 | 1020 | if (!p4d_present(p4d) || p4d_bad(p4d)) |
| 1129 | 1021 | return no_page_table(vma, flags, address); |
| 1130 | 1022 | |
| @@ -1168,10 +1060,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, · 37 unmodified lines | ||
| 1168 | 1060 | ctx->page_mask = 0; |
| 1169 | 1061 | pgd = pgd_offset(mm, address); |
| 1170 | 1062 | |
| 1171 | − if (unlikely(is_hugepd(__hugepd(pgd_val(*pgd))))) | |
| 1172 | − page = follow_hugepd(vma, __hugepd(pgd_val(*pgd)), | |
| 1173 | − address, PGDIR_SHIFT, flags, ctx); | |
| 1174 | − else if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) | |
| 1063 | + if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) | |
| 1175 | 1064 | page = no_page_table(vma, flags, address); |
| 1176 | 1065 | else |
| 1177 | 1066 | page = follow_p4d_mask(vma, address, pgd, flags, ctx); |
| @@ -2394,19 +2283,19 @@ struct page *get_dump_page(unsigned long addr) · 1216 unmodified lines | ||
| 2394 | 2283 | |
| 2395 | 2284 | #ifdef CONFIG_MIGRATION |
| 2396 | 2285 | /* |
| 2397 | − * Returns the number of collected pages. Return value is always >= 0. | |
| 2286 | + * Returns the number of collected folios. Return value is always >= 0. | |
| 2398 | 2287 | */ |
| 2399 | −static unsigned long collect_longterm_unpinnable_pages( | |
| 2400 | − struct list_head *movable_page_list, | |
| 2401 | − unsigned long nr_pages, | |
| 2402 | − struct page **pages) | |
| 2288 | +static unsigned long collect_longterm_unpinnable_folios( | |
| 2289 | + struct list_head *movable_folio_list, | |
| 2290 | + unsigned long nr_folios, | |
| 2291 | + struct folio **folios) | |
| 2403 | 2292 | { |
| 2404 | 2293 | unsigned long i, collected = 0; |
| 2405 | 2294 | struct folio *prev_folio = NULL; |
| 2406 | 2295 | bool drain_allow = true; |
| 2407 | 2296 | |
| 2408 | − for (i = 0; i < nr_pages; i++) { | |
| 2409 | − struct folio *folio = page_folio(pages[i]); | |
| 2297 | + for (i = 0; i < nr_folios; i++) { | |
| 2298 | + struct folio *folio = folios[i]; | |
| 2410 | 2299 | |
| 2411 | 2300 | if (folio == prev_folio) |
| 2412 | 2301 | continue; |
| @@ -2421,7 +2310,7 @@ static unsigned long collect_longterm_unpinnable_pages( · 8 unmodified lines | ||
| 2421 | 2310 | continue; |
| 2422 | 2311 | |
| 2423 | 2312 | if (folio_test_hugetlb(folio)) { |
| 2424 | − isolate_hugetlb(folio, movable_page_list); | |
| 2313 | + isolate_hugetlb(folio, movable_folio_list); | |
| 2425 | 2314 | continue; |
| 2426 | 2315 | } |
| 2427 | 2316 | |
| @@ -2433,7 +2322,7 @@ static unsigned long collect_longterm_unpinnable_pages( · 5 unmodified lines | ||
| 2433 | 2322 | if (!folio_isolate_lru(folio)) |
| 2434 | 2323 | continue; |
| 2435 | 2324 | |
| 2436 | − list_add_tail(&folio->lru, movable_page_list); | |
| 2325 | + list_add_tail(&folio->lru, movable_folio_list); | |
| 2437 | 2326 | node_stat_mod_folio(folio, |
| 2438 | 2327 | NR_ISOLATED_ANON + folio_is_file_lru(folio), |
| 2439 | 2328 | folio_nr_pages(folio)); |
| @@ -2443,27 +2332,28 @@ static unsigned long collect_longterm_unpinnable_pages( · 3 unmodified lines | ||
| 2443 | 2332 | } |
| 2444 | 2333 | |
| 2445 | 2334 | /* |
| 2446 | − * Unpins all pages and migrates device coherent pages and movable_page_list. | |
| 2447 | − * Returns -EAGAIN if all pages were successfully migrated or -errno for failure | |
| 2448 | − * (or partial success). | |
| 2335 | + * Unpins all folios and migrates device coherent folios and movable_folio_list. | |
| 2336 | + * Returns -EAGAIN if all folios were successfully migrated or -errno for | |
| 2337 | + * failure (or partial success). | |
| 2449 | 2338 | */ |
| 2450 | −static int migrate_longterm_unpinnable_pages( | |
| 2451 | − struct list_head *movable_page_list, | |
| 2452 | − unsigned long nr_pages, | |
| 2453 | − struct page **pages) | |
| 2339 | +static int migrate_longterm_unpinnable_folios( | |
| 2340 | + struct list_head *movable_folio_list, | |
| 2341 | + unsigned long nr_folios, | |
| 2342 | + struct folio **folios) | |
| 2454 | 2343 | { |
| 2455 | 2344 | int ret; |
| 2456 | 2345 | unsigned long i; |
| 2457 | 2346 | |
| 2458 | − for (i = 0; i < nr_pages; i++) { | |
| 2459 | − struct folio *folio = page_folio(pages[i]); | |
| 2347 | + for (i = 0; i < nr_folios; i++) { | |
| 2348 | + struct folio *folio = folios[i]; | |
| 2460 | 2349 | |
| 2461 | 2350 | if (folio_is_device_coherent(folio)) { |
| 2462 | 2351 | /* |
| 2463 | − * Migration will fail if the page is pinned, so convert | |
| 2464 | − * the pin on the source page to a normal reference. | |
| 2352 | + * Migration will fail if the folio is pinned, so | |
| 2353 | + * convert the pin on the source folio to a normal | |
| 2354 | + * reference. | |
| 2465 | 2355 | */ |
| 2466 | − pages[i] = NULL; | |
| 2356 | + folios[i] = NULL; | |
| 2467 | 2357 | folio_get(folio); |
| 2468 | 2358 | gup_put_folio(folio, 1, FOLL_PIN); |
| 2469 | 2359 | |
| @@ -2476,24 +2366,24 @@ static int migrate_longterm_unpinnable_pages( · 6 unmodified lines | ||
| 2476 | 2366 | } |
| 2477 | 2367 | |
| 2478 | 2368 | /* |
| 2479 | − * We can't migrate pages with unexpected references, so drop | |
| 2369 | + * We can't migrate folios with unexpected references, so drop | |
| 2480 | 2370 | * the reference obtained by __get_user_pages_locked(). |
| 2481 | − * Migrating pages have been added to movable_page_list after | |
| 2371 | + * Migrating folios have been added to movable_folio_list after | |
| 2482 | 2372 | * calling folio_isolate_lru() which takes a reference so the |
| 2483 | − * page won't be freed if it's migrating. | |
| 2373 | + * folio won't be freed if it's migrating. | |
| 2484 | 2374 | */ |
| 2485 | − unpin_user_page(pages[i]); | |
| 2486 | − pages[i] = NULL; | |
| 2375 | + unpin_folio(folios[i]); | |
| 2376 | + folios[i] = NULL; | |
| 2487 | 2377 | } |
| 2488 | 2378 | |
| 2489 | − if (!list_empty(movable_page_list)) { | |
| 2379 | + if (!list_empty(movable_folio_list)) { | |
| 2490 | 2380 | struct migration_target_control mtc = { |
| 2491 | 2381 | .nid = NUMA_NO_NODE, |
| 2492 | 2382 | .gfp_mask = GFP_USER | __GFP_NOWARN, |
| 2493 | 2383 | .reason = MR_LONGTERM_PIN, |
| 2494 | 2384 | }; |
| 2495 | 2385 | |
| 2496 | − if (migrate_pages(movable_page_list, alloc_migration_target, | |
| 2386 | + if (migrate_pages(movable_folio_list, alloc_migration_target, | |
| 2497 | 2387 | NULL, (unsigned long)&mtc, MIGRATE_SYNC, |
| 2498 | 2388 | MR_LONGTERM_PIN, NULL)) { |
| 2499 | 2389 | ret = -ENOMEM; |
| @@ -2501,48 +2391,71 @@ static int migrate_longterm_unpinnable_pages( · 1 unmodified line | ||
| 2501 | 2391 | } |
| 2502 | 2392 | } |
| 2503 | 2393 | |
| 2504 | − putback_movable_pages(movable_page_list); | |
| 2394 | + putback_movable_pages(movable_folio_list); | |
| 2505 | 2395 | |
| 2506 | 2396 | return -EAGAIN; |
| 2507 | 2397 | |
| 2508 | 2398 | err: |
| 2509 | − for (i = 0; i < nr_pages; i++) | |
| 2510 | − if (pages[i]) | |
| 2511 | − unpin_user_page(pages[i]); | |
| 2512 | − putback_movable_pages(movable_page_list); | |
| 2399 | + unpin_folios(folios, nr_folios); | |
| 2400 | + putback_movable_pages(movable_folio_list); | |
| 2513 | 2401 | |
| 2514 | 2402 | return ret; |
| 2515 | 2403 | } |
| 2516 | 2404 | |
| 2517 | 2405 | /* |
| 2518 | − * Check whether all pages are *allowed* to be pinned. Rather confusingly, all | |
| 2519 | − * pages in the range are required to be pinned via FOLL_PIN, before calling | |
| 2520 | − * this routine. | |
| 2406 | + * Check whether all folios are *allowed* to be pinned indefinitely (longterm). | |
| 2407 | + * Rather confusingly, all folios in the range are required to be pinned via | |
| 2408 | + * FOLL_PIN, before calling this routine. | |
| 2521 | 2409 | * |
| 2522 | − * If any pages in the range are not allowed to be pinned, then this routine | |
| 2523 | − * will migrate those pages away, unpin all the pages in the range and return | |
| 2410 | + * If any folios in the range are not allowed to be pinned, then this routine | |
| 2411 | + * will migrate those folios away, unpin all the folios in the range and return | |
| 2524 | 2412 | * -EAGAIN. The caller should re-pin the entire range with FOLL_PIN and then |
| 2525 | 2413 | * call this routine again. |
| 2526 | 2414 | * |
| 2527 | 2415 | * If an error other than -EAGAIN occurs, this indicates a migration failure. |
| 2528 | 2416 | * The caller should give up, and propagate the error back up the call stack. |
| 2529 | 2417 | * |
| 2530 | − * If everything is OK and all pages in the range are allowed to be pinned, then | |
| 2531 | − * this routine leaves all pages pinned and returns zero for success. | |
| 2418 | + * If everything is OK and all folios in the range are allowed to be pinned, | |
| 2419 | + * then this routine leaves all folios pinned and returns zero for success. | |
| 2532 | 2420 | */ |
| 2533 | −static long check_and_migrate_movable_pages(unsigned long nr_pages, | |
| 2534 | − struct page **pages) | |
| 2421 | +static long check_and_migrate_movable_folios(unsigned long nr_folios, | |
| 2422 | + struct folio **folios) | |
| 2535 | 2423 | { |
| 2536 | 2424 | unsigned long collected; |
| 2537 | − LIST_HEAD(movable_page_list); | |
| 2425 | + LIST_HEAD(movable_folio_list); | |
| 2538 | 2426 | |
| 2539 | − collected = collect_longterm_unpinnable_pages(&movable_page_list, | |
| 2540 | − nr_pages, pages); | |
| 2427 | + collected = collect_longterm_unpinnable_folios(&movable_folio_list, | |
| 2428 | + nr_folios, folios); | |
| 2541 | 2429 | if (!collected) |
| 2542 | 2430 | return 0; |
| 2543 | 2431 | |
| 2544 | − return migrate_longterm_unpinnable_pages(&movable_page_list, nr_pages, | |
| 2545 | − pages); | |
| 2432 | + return migrate_longterm_unpinnable_folios(&movable_folio_list, | |
| 2433 | + nr_folios, folios); | |
| 2434 | +} | |
| 2435 | + | |
| 2436 | +/* | |
| 2437 | + * This routine just converts all the pages in the @pages array to folios and | |
| 2438 | + * calls check_and_migrate_movable_folios() to do the heavy lifting. | |
| 2439 | + * | |
| 2440 | + * Please see the check_and_migrate_movable_folios() documentation for details. | |
| 2441 | + */ | |
| 2442 | +static long check_and_migrate_movable_pages(unsigned long nr_pages, | |
| 2443 | + struct page **pages) | |
| 2444 | +{ | |
| 2445 | + struct folio **folios; | |
| 2446 | + long i, ret; | |
| 2447 | + | |
| 2448 | + folios = kmalloc_array(nr_pages, sizeof(*folios), GFP_KERNEL); | |
| 2449 | + if (!folios) | |
| 2450 | + return -ENOMEM; | |
| 2451 | + | |
| 2452 | + for (i = 0; i < nr_pages; i++) | |
| 2453 | + folios[i] = page_folio(pages[i]); | |
| 2454 | + | |
| 2455 | + ret = check_and_migrate_movable_folios(nr_pages, folios); | |
| 2456 | + | |
| 2457 | + kfree(folios); | |
| 2458 | + return ret; | |
| 2546 | 2459 | } |
| 2547 | 2460 | #else |
| 2548 | 2461 | static long check_and_migrate_movable_pages(unsigned long nr_pages, |
| @@ -2550,6 +2463,12 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages, · 1 unmodified line | ||
| 2550 | 2463 | { |
| 2551 | 2464 | return 0; |
| 2552 | 2465 | } |
| 2466 | + | |
| 2467 | +static long check_and_migrate_movable_folios(unsigned long nr_folios, | |
| 2468 | + struct folio **folios) | |
| 2469 | +{ | |
| 2470 | + return 0; | |
| 2471 | +} | |
| 2553 | 2472 | #endif /* CONFIG_MIGRATION */ |
| 2554 | 2473 | |
| 2555 | 2474 | /* |
| @@ -3283,15 +3202,6 @@ static int gup_fast_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, · 727 unmodified lines | ||
| 3283 | 3202 | pages, nr)) |
| 3284 | 3203 | return 0; |
| 3285 | 3204 | |
| 3286 | − } else if (unlikely(is_hugepd(__hugepd(pmd_val(pmd))))) { | |
| 3287 | − /* | |
| 3288 | − * architecture have different format for hugetlbfs | |
| 3289 | − * pmd format and THP pmd format | |
| 3290 | − */ | |
| 3291 | − if (gup_hugepd(NULL, __hugepd(pmd_val(pmd)), addr, | |
| 3292 | − PMD_SHIFT, next, flags, pages, nr, | |
| 3293 | − true) != 1) | |
| 3294 | − return 0; | |
| 3295 | 3205 | } else if (!gup_fast_pte_range(pmd, pmdp, addr, next, flags, |
| 3296 | 3206 | pages, nr)) |
| 3297 | 3207 | return 0; |
| @@ -3318,11 +3228,6 @@ static int gup_fast_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, · 20 unmodified lines | ||
| 3318 | 3228 | if (!gup_fast_pud_leaf(pud, pudp, addr, next, flags, |
| 3319 | 3229 | pages, nr)) |
| 3320 | 3230 | return 0; |
| 3321 | − } else if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) { | |
| 3322 | − if (gup_hugepd(NULL, __hugepd(pud_val(pud)), addr, | |
| 3323 | − PUD_SHIFT, next, flags, pages, nr, | |
| 3324 | − true) != 1) | |
| 3325 | − return 0; | |
| 3326 | 3231 | } else if (!gup_fast_pmd_range(pudp, pud, addr, next, flags, |
| 3327 | 3232 | pages, nr)) |
| 3328 | 3233 | return 0; |
| @@ -3346,13 +3251,8 @@ static int gup_fast_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, · 17 unmodified lines | ||
| 3346 | 3251 | if (!p4d_present(p4d)) |
| 3347 | 3252 | return 0; |
| 3348 | 3253 | BUILD_BUG_ON(p4d_leaf(p4d)); |
| 3349 | − if (unlikely(is_hugepd(__hugepd(p4d_val(p4d))))) { | |
| 3350 | − if (gup_hugepd(NULL, __hugepd(p4d_val(p4d)), addr, | |
| 3351 | − P4D_SHIFT, next, flags, pages, nr, | |
| 3352 | − true) != 1) | |
| 3353 | − return 0; | |
| 3354 | − } else if (!gup_fast_pud_range(p4dp, p4d, addr, next, flags, | |
| 3355 | − pages, nr)) | |
| 3254 | + if (!gup_fast_pud_range(p4dp, p4d, addr, next, flags, | |
| 3255 | + pages, nr)) | |
| 3356 | 3256 | return 0; |
| 3357 | 3257 | } while (p4dp++, addr = next, addr != end); |
| 3358 | 3258 | |
| @@ -3376,11 +3276,6 @@ static void gup_fast_pgd_range(unsigned long addr, unsigned long end, · 17 unmodified lines | ||
| 3376 | 3276 | if (!gup_fast_pgd_leaf(pgd, pgdp, addr, next, flags, |
| 3377 | 3277 | pages, nr)) |
| 3378 | 3278 | return; |
| 3379 | − } else if (unlikely(is_hugepd(__hugepd(pgd_val(pgd))))) { | |
| 3380 | − if (gup_hugepd(NULL, __hugepd(pgd_val(pgd)), addr, | |
| 3381 | − PGDIR_SHIFT, next, flags, pages, nr, | |
| 3382 | − true) != 1) | |
| 3383 | − return; | |
| 3384 | 3279 | } else if (!gup_fast_p4d_range(pgdp, pgd, addr, next, flags, |
| 3385 | 3280 | pages, nr)) |
| 3386 | 3281 | return; |
| @@ -3687,3 +3582,140 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, · 300 unmodified lines | ||
| 3687 | 3582 | &locked, gup_flags); |
| 3688 | 3583 | } |
| 3689 | 3584 | EXPORT_SYMBOL(pin_user_pages_unlocked); |
| 3585 | + | |
| 3586 | +/** | |
| 3587 | + * memfd_pin_folios() - pin folios associated with a memfd | |
| 3588 | + * @memfd: the memfd whose folios are to be pinned | |
| 3589 | + * @start: the first memfd offset | |
| 3590 | + * @end: the last memfd offset (inclusive) | |
| 3591 | + * @folios: array that receives pointers to the folios pinned | |
| 3592 | + * @max_folios: maximum number of entries in @folios | |
| 3593 | + * @offset: the offset into the first folio | |
| 3594 | + * | |
| 3595 | + * Attempt to pin folios associated with a memfd in the contiguous range | |
| 3596 | + * [start, end]. Given that a memfd is either backed by shmem or hugetlb, | |
| 3597 | + * the folios can either be found in the page cache or need to be allocated | |
| 3598 | + * if necessary. Once the folios are located, they are all pinned via | |
| 3599 | + * FOLL_PIN and @offset is populatedwith the offset into the first folio. | |
| 3600 | + * And, eventually, these pinned folios must be released either using | |
| 3601 | + * unpin_folios() or unpin_folio(). | |
| 3602 | + * | |
| 3603 | + * It must be noted that the folios may be pinned for an indefinite amount | |
| 3604 | + * of time. And, in most cases, the duration of time they may stay pinned | |
| 3605 | + * would be controlled by the userspace. This behavior is effectively the | |
| 3606 | + * same as using FOLL_LONGTERM with other GUP APIs. | |
| 3607 | + * | |
| 3608 | + * Returns number of folios pinned, which could be less than @max_folios | |
| 3609 | + * as it depends on the folio sizes that cover the range [start, end]. | |
| 3610 | + * If no folios were pinned, it returns -errno. | |
| 3611 | + */ | |
| 3612 | +long memfd_pin_folios(struct file *memfd, loff_t start, loff_t end, | |
| 3613 | + struct folio **folios, unsigned int max_folios, | |
| 3614 | + pgoff_t *offset) | |
| 3615 | +{ | |
| 3616 | + unsigned int flags, nr_folios, nr_found; | |
| 3617 | + unsigned int i, pgshift = PAGE_SHIFT; | |
| 3618 | + pgoff_t start_idx, end_idx, next_idx; | |
| 3619 | + struct folio *folio = NULL; | |
| 3620 | + struct folio_batch fbatch; | |
| 3621 | + struct hstate *h; | |
| 3622 | + long ret = -EINVAL; | |
| 3623 | + | |
| 3624 | + if (start < 0 || start > end || !max_folios) | |
| 3625 | + return -EINVAL; | |
| 3626 | + | |
| 3627 | + if (!memfd) | |
| 3628 | + return -EINVAL; | |
| 3629 | + | |
| 3630 | + if (!shmem_file(memfd) && !is_file_hugepages(memfd)) | |
| 3631 | + return -EINVAL; | |
| 3632 | + | |
| 3633 | + if (end >= i_size_read(file_inode(memfd))) | |
| 3634 | + return -EINVAL; | |
| 3635 | + | |
| 3636 | + if (is_file_hugepages(memfd)) { | |
| 3637 | + h = hstate_file(memfd); | |
| 3638 | + pgshift = huge_page_shift(h); | |
| 3639 | + } | |
| 3640 | + | |
| 3641 | + flags = memalloc_pin_save(); | |
| 3642 | + do { | |
| 3643 | + nr_folios = 0; | |
| 3644 | + start_idx = start >> pgshift; | |
| 3645 | + end_idx = end >> pgshift; | |
| 3646 | + if (is_file_hugepages(memfd)) { | |
| 3647 | + start_idx <<= huge_page_order(h); | |
| 3648 | + end_idx <<= huge_page_order(h); | |
| 3649 | + } | |
| 3650 | + | |
| 3651 | + folio_batch_init(&fbatch); | |
| 3652 | + while (start_idx <= end_idx && nr_folios < max_folios) { | |
| 3653 | + /* | |
| 3654 | + * In most cases, we should be able to find the folios | |
| 3655 | + * in the page cache. If we cannot find them for some | |
| 3656 | + * reason, we try to allocate them and add them to the | |
| 3657 | + * page cache. | |
| 3658 | + */ | |
| 3659 | + nr_found = filemap_get_folios_contig(memfd->f_mapping, | |
| 3660 | + &start_idx, | |
| 3661 | + end_idx, | |
| 3662 | + &fbatch); | |
| 3663 | + if (folio) { | |
| 3664 | + folio_put(folio); | |
| 3665 | + folio = NULL; | |
| 3666 | + } | |
| 3667 | + | |
| 3668 | + next_idx = 0; | |
| 3669 | + for (i = 0; i < nr_found; i++) { | |
| 3670 | + /* | |
| 3671 | + * As there can be multiple entries for a | |
| 3672 | + * given folio in the batch returned by | |
| 3673 | + * filemap_get_folios_contig(), the below | |
| 3674 | + * check is to ensure that we pin and return a | |
| 3675 | + * unique set of folios between start and end. | |
| 3676 | + */ | |
| 3677 | + if (next_idx && | |
| 3678 | + next_idx != folio_index(fbatch.folios[i])) | |
| 3679 | + continue; | |
| 3680 | + | |
| 3681 | + folio = page_folio(&fbatch.folios[i]->page); | |
| 3682 | + | |
| 3683 | + if (try_grab_folio(folio, 1, FOLL_PIN)) { | |
| 3684 | + folio_batch_release(&fbatch); | |
| 3685 | + ret = -EINVAL; | |
| 3686 | + goto err; | |
| 3687 | + } | |
| 3688 | + | |
| 3689 | + if (nr_folios == 0) | |
| 3690 | + *offset = offset_in_folio(folio, start); | |
| 3691 | + | |
| 3692 | + folios[nr_folios] = folio; | |
| 3693 | + next_idx = folio_next_index(folio); | |
| 3694 | + if (++nr_folios == max_folios) | |
| 3695 | + break; | |
| 3696 | + } | |
| 3697 | + | |
| 3698 | + folio = NULL; | |
| 3699 | + folio_batch_release(&fbatch); | |
| 3700 | + if (!nr_found) { | |
| 3701 | + folio = memfd_alloc_folio(memfd, start_idx); | |
| 3702 | + if (IS_ERR(folio)) { | |
| 3703 | + ret = PTR_ERR(folio); | |
| 3704 | + if (ret != -EEXIST) | |
| 3705 | + goto err; | |
| 3706 | + } | |
| 3707 | + } | |
| 3708 | + } | |
| 3709 | + | |
| 3710 | + ret = check_and_migrate_movable_folios(nr_folios, folios); | |
| 3711 | + } while (ret == -EAGAIN); | |
| 3712 | + | |
| 3713 | + memalloc_pin_restore(flags); | |
| 3714 | + return ret ? ret : nr_folios; | |
| 3715 | +err: | |
| 3716 | + memalloc_pin_restore(flags); | |
| 3717 | + unpin_folios(folios, nr_folios); | |
| 3718 | + | |
| 3719 | + return ret; | |
| 3720 | +} | |
| 3721 | +EXPORT_SYMBOL_GPL(memfd_pin_folios); | |
| @@ -111,13 +111,10 @@ static inline wait_queue_head_t *get_pkmap_wait_queue_head(unsigned int color) · 110 unmodified lines | ||
| 111 | 111 | } |
| 112 | 112 | #endif |
| 113 | 113 | |
| 114 | −atomic_long_t _totalhigh_pages __read_mostly; | |
| 115 | −EXPORT_SYMBOL(_totalhigh_pages); | |
| 116 | − | |
| 117 | −unsigned int __nr_free_highpages(void) | |
| 114 | +unsigned long __nr_free_highpages(void) | |
| 118 | 115 | { |
| 116 | + unsigned long pages = 0; | |
| 119 | 117 | struct zone *zone; |
| 120 | − unsigned int pages = 0; | |
| 121 | 118 | |
| 122 | 119 | for_each_populated_zone(zone) { |
| 123 | 120 | if (is_highmem(zone)) |
| @@ -127,6 +124,20 @@ unsigned int __nr_free_highpages(void) · 3 unmodified lines | ||
| 127 | 124 | return pages; |
| 128 | 125 | } |
| 129 | 126 | |
| 127 | +unsigned long __totalhigh_pages(void) | |
| 128 | +{ | |
| 129 | + unsigned long pages = 0; | |
| 130 | + struct zone *zone; | |
| 131 | + | |
| 132 | + for_each_populated_zone(zone) { | |
| 133 | + if (is_highmem(zone)) | |
| 134 | + pages += zone_managed_pages(zone); | |
| 135 | + } | |
| 136 | + | |
| 137 | + return pages; | |
| 138 | +} | |
| 139 | +EXPORT_SYMBOL(__totalhigh_pages); | |
| 140 | + | |
| 130 | 141 | static int pkmap_count[LAST_PKMAP]; |
| 131 | 142 | static __cacheline_aligned_in_smp DEFINE_SPINLOCK(kmap_lock); |
| 132 | 143 | |
| @@ -480,7 +480,7 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, · 479 unmodified lines | ||
| 480 | 480 | pte_t entry; |
| 481 | 481 | |
| 482 | 482 | ptl = huge_pte_lock(hstate_vma(vma), walk->mm, pte); |
| 483 | − entry = huge_ptep_get(pte); | |
| 483 | + entry = huge_ptep_get(walk->mm, addr, pte); | |
| 484 | 484 | |
| 485 | 485 | i = (start - range->start) >> PAGE_SHIFT; |
| 486 | 486 | pfn_req_flags = range->hmm_pfns[i]; |
| @@ -20,6 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <linux/swapops.h> |
| 21 | 21 | #include <linux/backing-dev.h> |
| 22 | 22 | #include <linux/dax.h> |
| 23 | +#include <linux/mm_types.h> | |
| 23 | 24 | #include <linux/khugepaged.h> |
| 24 | 25 | #include <linux/freezer.h> |
| 25 | 26 | #include <linux/pfn_t.h> |
| @@ -150,10 +151,15 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, · 124 unmodified lines | ||
| 150 | 151 | * Must be done before hugepage flags check since shmem has its |
| 151 | 152 | * own flags. |
| 152 | 153 | */ |
| 153 | − if (!in_pf && shmem_file(vma->vm_file)) | |
| 154 | − return shmem_is_huge(file_inode(vma->vm_file), vma->vm_pgoff, | |
| 155 | − !enforce_sysfs, vma->vm_mm, vm_flags) | |
| 156 | − ? orders : 0; | |
| 154 | + if (!in_pf && shmem_file(vma->vm_file)) { | |
| 155 | + bool global_huge = shmem_is_huge(file_inode(vma->vm_file), vma->vm_pgoff, | |
| 156 | + !enforce_sysfs, vma->vm_mm, vm_flags); | |
| 157 | + | |
| 158 | + if (!vma_is_anon_shmem(vma)) | |
| 159 | + return global_huge ? orders : 0; | |
| 160 | + return shmem_allowable_huge_orders(file_inode(vma->vm_file), | |
| 161 | + vma, vma->vm_pgoff, global_huge); | |
| 162 | + } | |
| 157 | 163 | |
| 158 | 164 | if (!vma_is_anonymous(vma)) { |
| 159 | 165 | /* |
| @@ -449,14 +455,6 @@ static void thpsize_release(struct kobject *kobj); · 289 unmodified lines | ||
| 449 | 455 | static DEFINE_SPINLOCK(huge_anon_orders_lock); |
| 450 | 456 | static LIST_HEAD(thpsize_list); |
| 451 | 457 | |
| 452 | −struct thpsize { | |
| 453 | − struct kobject kobj; | |
| 454 | − struct list_head node; | |
| 455 | − int order; | |
| 456 | −}; | |
| 457 | − | |
| 458 | −#define to_thpsize(kobj) container_of(kobj, struct thpsize, kobj) | |
| 459 | − | |
| 460 | 458 | static ssize_t thpsize_enabled_show(struct kobject *kobj, |
| 461 | 459 | struct kobj_attribute *attr, char *buf) |
| 462 | 460 | { |
| @@ -509,6 +507,13 @@ static ssize_t thpsize_enabled_store(struct kobject *kobj, · 46 unmodified lines | ||
| 509 | 507 | } else |
| 510 | 508 | ret = -EINVAL; |
| 511 | 509 | |
| 510 | + if (ret > 0) { | |
| 511 | + int err; | |
| 512 | + | |
| 513 | + err = start_stop_khugepaged(); | |
| 514 | + if (err) | |
| 515 | + ret = err; | |
| 516 | + } | |
| 512 | 517 | return ret; |
| 513 | 518 | } |
| 514 | 519 | |
| @@ -517,6 +522,9 @@ static struct kobj_attribute thpsize_enabled_attr = · 2 unmodified lines | ||
| 517 | 522 | |
| 518 | 523 | static struct attribute *thpsize_attrs[] = { |
| 519 | 524 | &thpsize_enabled_attr.attr, |
| 525 | +#ifdef CONFIG_SHMEM | |
| 526 | + &thpsize_shmem_enabled_attr.attr, | |
| 527 | +#endif | |
| 520 | 528 | NULL, |
| 521 | 529 | }; |
| 522 | 530 | |
| @@ -560,6 +568,12 @@ DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); · 37 unmodified lines | ||
| 560 | 568 | DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); |
| 561 | 569 | DEFINE_MTHP_STAT_ATTR(swpout, MTHP_STAT_SWPOUT); |
| 562 | 570 | DEFINE_MTHP_STAT_ATTR(swpout_fallback, MTHP_STAT_SWPOUT_FALLBACK); |
| 571 | +DEFINE_MTHP_STAT_ATTR(shmem_alloc, MTHP_STAT_SHMEM_ALLOC); | |
| 572 | +DEFINE_MTHP_STAT_ATTR(shmem_fallback, MTHP_STAT_SHMEM_FALLBACK); | |
| 573 | +DEFINE_MTHP_STAT_ATTR(shmem_fallback_charge, MTHP_STAT_SHMEM_FALLBACK_CHARGE); | |
| 574 | +DEFINE_MTHP_STAT_ATTR(split, MTHP_STAT_SPLIT); | |
| 575 | +DEFINE_MTHP_STAT_ATTR(split_failed, MTHP_STAT_SPLIT_FAILED); | |
| 576 | +DEFINE_MTHP_STAT_ATTR(split_deferred, MTHP_STAT_SPLIT_DEFERRED); | |
| 563 | 577 | |
| 564 | 578 | static struct attribute *stats_attrs[] = { |
| 565 | 579 | &anon_fault_alloc_attr.attr, |
| @@ -567,6 +581,12 @@ static struct attribute *stats_attrs[] = { · 1 unmodified line | ||
| 567 | 581 | &anon_fault_fallback_charge_attr.attr, |
| 568 | 582 | &swpout_attr.attr, |
| 569 | 583 | &swpout_fallback_attr.attr, |
| 584 | + &shmem_alloc_attr.attr, | |
| 585 | + &shmem_fallback_attr.attr, | |
| 586 | + &shmem_fallback_charge_attr.attr, | |
| 587 | + &split_attr.attr, | |
| 588 | + &split_failed_attr.attr, | |
| 589 | + &split_deferred_attr.attr, | |
| 570 | 590 | NULL, |
| 571 | 591 | }; |
| 572 | 592 | |
| @@ -942,10 +962,10 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, · 369 unmodified lines | ||
| 942 | 962 | goto release; |
| 943 | 963 | } |
| 944 | 964 | |
| 945 | − clear_huge_page(page, vmf->address, HPAGE_PMD_NR); | |
| 965 | + folio_zero_user(folio, vmf->address); | |
| 946 | 966 | /* |
| 947 | 967 | * The memory barrier inside __folio_mark_uptodate makes sure that |
| 948 | − * clear_huge_page writes become visible before the set_pmd_at() | |
| 968 | + * folio_zero_user writes become visible before the set_pmd_at() | |
| 949 | 969 | * write. |
| 950 | 970 | */ |
| 951 | 971 | __folio_mark_uptodate(folio); |
| @@ -972,7 +992,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, · 20 unmodified lines | ||
| 972 | 992 | |
| 973 | 993 | entry = mk_huge_pmd(page, vma->vm_page_prot); |
| 974 | 994 | entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); |
| 975 | − folio_add_new_anon_rmap(folio, vma, haddr); | |
| 995 | + folio_add_new_anon_rmap(folio, vma, haddr, RMAP_EXCLUSIVE); | |
| 976 | 996 | folio_add_lru_vma(folio, vma); |
| 977 | 997 | pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable); |
| 978 | 998 | set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry); |
| @@ -1624,7 +1644,7 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, · 645 unmodified lines | ||
| 1624 | 1644 | return false; |
| 1625 | 1645 | |
| 1626 | 1646 | /* Do we need write faults for softdirty tracking? */ |
| 1627 | − if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd)) | |
| 1647 | + if (pmd_needs_soft_dirty_wp(vma, pmd)) | |
| 1628 | 1648 | return false; |
| 1629 | 1649 | |
| 1630 | 1650 | /* Do we need write faults for uffd-wp tracking? */ |
| @@ -1651,7 +1671,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 20 unmodified lines | ||
| 1651 | 1671 | unsigned long haddr = vmf->address & HPAGE_PMD_MASK; |
| 1652 | 1672 | int nid = NUMA_NO_NODE; |
| 1653 | 1673 | int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK); |
| 1654 | − bool migrated = false, writable = false; | |
| 1674 | + bool writable = false; | |
| 1655 | 1675 | int flags = 0; |
| 1656 | 1676 | |
| 1657 | 1677 | vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); |
| @@ -1687,16 +1707,17 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 29 unmodified lines | ||
| 1687 | 1707 | if (node_is_toptier(nid)) |
| 1688 | 1708 | last_cpupid = folio_last_cpupid(folio); |
| 1689 | 1709 | target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags); |
| 1690 | − if (target_nid == NUMA_NO_NODE) { | |
| 1691 | − folio_put(folio); | |
| 1710 | + if (target_nid == NUMA_NO_NODE) | |
| 1711 | + goto out_map; | |
| 1712 | + if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) { | |
| 1713 | + flags |= TNF_MIGRATE_FAIL; | |
| 1692 | 1714 | goto out_map; |
| 1693 | 1715 | } |
| 1694 | − | |
| 1716 | + /* The folio is isolated and isolation code holds a folio reference. */ | |
| 1695 | 1717 | spin_unlock(vmf->ptl); |
| 1696 | 1718 | writable = false; |
| 1697 | 1719 | |
| 1698 | − migrated = migrate_misplaced_folio(folio, vma, target_nid); | |
| 1699 | − if (migrated) { | |
| 1720 | + if (!migrate_misplaced_folio(folio, vma, target_nid)) { | |
| 1700 | 1721 | flags |= TNF_MIGRATED; |
| 1701 | 1722 | nid = target_nid; |
| 1702 | 1723 | } else { |
| @@ -2581,6 +2602,27 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, · 878 unmodified lines | ||
| 2581 | 2602 | pmd_populate(mm, pmd, pgtable); |
| 2582 | 2603 | } |
| 2583 | 2604 | |
| 2605 | +void split_huge_pmd_locked(struct vm_area_struct *vma, unsigned long address, | |
| 2606 | + pmd_t *pmd, bool freeze, struct folio *folio) | |
| 2607 | +{ | |
| 2608 | + VM_WARN_ON_ONCE(folio && !folio_test_pmd_mappable(folio)); | |
| 2609 | + VM_WARN_ON_ONCE(!IS_ALIGNED(address, HPAGE_PMD_SIZE)); | |
| 2610 | + VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); | |
| 2611 | + VM_BUG_ON(freeze && !folio); | |
| 2612 | + | |
| 2613 | + /* | |
| 2614 | + * When the caller requests to set up a migration entry, we | |
| 2615 | + * require a folio to check the PMD against. Otherwise, there | |
| 2616 | + * is a risk of replacing the wrong folio. | |
| 2617 | + */ | |
| 2618 | + if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || | |
| 2619 | + is_pmd_migration_entry(*pmd)) { | |
| 2620 | + if (folio && folio != pmd_folio(*pmd)) | |
| 2621 | + return; | |
| 2622 | + __split_huge_pmd_locked(vma, pmd, address, freeze); | |
| 2623 | + } | |
| 2624 | +} | |
| 2625 | + | |
| 2584 | 2626 | void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, |
| 2585 | 2627 | unsigned long address, bool freeze, struct folio *folio) |
| 2586 | 2628 | { |
| @@ -2592,26 +2634,7 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, · 5 unmodified lines | ||
| 2592 | 2634 | (address & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE); |
| 2593 | 2635 | mmu_notifier_invalidate_range_start(&range); |
| 2594 | 2636 | ptl = pmd_lock(vma->vm_mm, pmd); |
| 2595 | − | |
| 2596 | − /* | |
| 2597 | − * If caller asks to setup a migration entry, we need a folio to check | |
| 2598 | − * pmd against. Otherwise we can end up replacing wrong folio. | |
| 2599 | − */ | |
| 2600 | − VM_BUG_ON(freeze && !folio); | |
| 2601 | − VM_WARN_ON_ONCE(folio && !folio_test_locked(folio)); | |
| 2602 | − | |
| 2603 | − if (pmd_trans_huge(*pmd) || pmd_devmap(*pmd) || | |
| 2604 | − is_pmd_migration_entry(*pmd)) { | |
| 2605 | − /* | |
| 2606 | − * It's safe to call pmd_page when folio is set because it's | |
| 2607 | − * guaranteed that pmd is present. | |
| 2608 | − */ | |
| 2609 | − if (folio && folio != pmd_folio(*pmd)) | |
| 2610 | − goto out; | |
| 2611 | − __split_huge_pmd_locked(vma, pmd, range.start, freeze); | |
| 2612 | − } | |
| 2613 | − | |
| 2614 | −out: | |
| 2637 | + split_huge_pmd_locked(vma, range.start, pmd, freeze, folio); | |
| 2615 | 2638 | spin_unlock(ptl); |
| 2616 | 2639 | mmu_notifier_invalidate_range_end(&range); |
| 2617 | 2640 | } |
| @@ -2685,6 +2708,71 @@ static void unmap_folio(struct folio *folio) · 67 unmodified lines | ||
| 2685 | 2708 | try_to_unmap_flush(); |
| 2686 | 2709 | } |
| 2687 | 2710 | |
| 2711 | +static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, | |
| 2712 | + unsigned long addr, pmd_t *pmdp, | |
| 2713 | + struct folio *folio) | |
| 2714 | +{ | |
| 2715 | + struct mm_struct *mm = vma->vm_mm; | |
| 2716 | + int ref_count, map_count; | |
| 2717 | + pmd_t orig_pmd = *pmdp; | |
| 2718 | + | |
| 2719 | + if (folio_test_dirty(folio) || pmd_dirty(orig_pmd)) | |
| 2720 | + return false; | |
| 2721 | + | |
| 2722 | + orig_pmd = pmdp_huge_clear_flush(vma, addr, pmdp); | |
| 2723 | + | |
| 2724 | + /* | |
| 2725 | + * Syncing against concurrent GUP-fast: | |
| 2726 | + * - clear PMD; barrier; read refcount | |
| 2727 | + * - inc refcount; barrier; read PMD | |
| 2728 | + */ | |
| 2729 | + smp_mb(); | |
| 2730 | + | |
| 2731 | + ref_count = folio_ref_count(folio); | |
| 2732 | + map_count = folio_mapcount(folio); | |
| 2733 | + | |
| 2734 | + /* | |
| 2735 | + * Order reads for folio refcount and dirty flag | |
| 2736 | + * (see comments in __remove_mapping()). | |
| 2737 | + */ | |
| 2738 | + smp_rmb(); | |
| 2739 | + | |
| 2740 | + /* | |
| 2741 | + * If the folio or its PMD is redirtied at this point, or if there | |
| 2742 | + * are unexpected references, we will give up to discard this folio | |
| 2743 | + * and remap it. | |
| 2744 | + * | |
| 2745 | + * The only folio refs must be one from isolation plus the rmap(s). | |
| 2746 | + */ | |
| 2747 | + if (folio_test_dirty(folio) || pmd_dirty(orig_pmd) || | |
| 2748 | + ref_count != map_count + 1) { | |
| 2749 | + set_pmd_at(mm, addr, pmdp, orig_pmd); | |
| 2750 | + return false; | |
| 2751 | + } | |
| 2752 | + | |
| 2753 | + folio_remove_rmap_pmd(folio, pmd_page(orig_pmd), vma); | |
| 2754 | + zap_deposited_table(mm, pmdp); | |
| 2755 | + add_mm_counter(mm, MM_ANONPAGES, -HPAGE_PMD_NR); | |
| 2756 | + if (vma->vm_flags & VM_LOCKED) | |
| 2757 | + mlock_drain_local(); | |
| 2758 | + folio_put(folio); | |
| 2759 | + | |
| 2760 | + return true; | |
| 2761 | +} | |
| 2762 | + | |
| 2763 | +bool unmap_huge_pmd_locked(struct vm_area_struct *vma, unsigned long addr, | |
| 2764 | + pmd_t *pmdp, struct folio *folio) | |
| 2765 | +{ | |
| 2766 | + VM_WARN_ON_FOLIO(!folio_test_pmd_mappable(folio), folio); | |
| 2767 | + VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio); | |
| 2768 | + VM_WARN_ON_ONCE(!IS_ALIGNED(addr, HPAGE_PMD_SIZE)); | |
| 2769 | + | |
| 2770 | + if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) | |
| 2771 | + return __discard_anon_folio_pmd_locked(vma, addr, pmdp, folio); | |
| 2772 | + | |
| 2773 | + return false; | |
| 2774 | +} | |
| 2775 | + | |
| 2688 | 2776 | static void remap_page(struct folio *folio, unsigned long nr) |
| 2689 | 2777 | { |
| 2690 | 2778 | int i = 0; |
| @@ -2838,7 +2926,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, · 147 unmodified lines | ||
| 2838 | 2926 | split_page_memcg(head, order, new_order); |
| 2839 | 2927 | |
| 2840 | 2928 | if (folio_test_anon(folio) && folio_test_swapcache(folio)) { |
| 2841 | − offset = swp_offset(folio->swap); | |
| 2929 | + offset = swap_cache_index(folio->swap); | |
| 2842 | 2930 | swap_cache = swap_address_space(folio->swap); |
| 2843 | 2931 | xa_lock(&swap_cache->i_pages); |
| 2844 | 2932 | } |
| @@ -2998,7 +3086,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, · 153 unmodified lines | ||
| 2998 | 3086 | XA_STATE_ORDER(xas, &folio->mapping->i_pages, folio->index, new_order); |
| 2999 | 3087 | struct anon_vma *anon_vma = NULL; |
| 3000 | 3088 | struct address_space *mapping = NULL; |
| 3001 | − bool is_thp = folio_test_pmd_mappable(folio); | |
| 3089 | + int order = folio_order(folio); | |
| 3002 | 3090 | int extra_pins, ret; |
| 3003 | 3091 | pgoff_t end; |
| 3004 | 3092 | bool is_hzp; |
| @@ -3183,27 +3271,17 @@ out_unlock: · 178 unmodified lines | ||
| 3183 | 3271 | i_mmap_unlock_read(mapping); |
| 3184 | 3272 | out: |
| 3185 | 3273 | xas_destroy(&xas); |
| 3186 | − if (is_thp) | |
| 3274 | + if (order == HPAGE_PMD_ORDER) | |
| 3187 | 3275 | count_vm_event(!ret ? THP_SPLIT_PAGE : THP_SPLIT_PAGE_FAILED); |
| 3276 | + count_mthp_stat(order, !ret ? MTHP_STAT_SPLIT : MTHP_STAT_SPLIT_FAILED); | |
| 3188 | 3277 | return ret; |
| 3189 | 3278 | } |
| 3190 | 3279 | |
| 3191 | −void folio_undo_large_rmappable(struct folio *folio) | |
| 3280 | +void __folio_undo_large_rmappable(struct folio *folio) | |
| 3192 | 3281 | { |
| 3193 | 3282 | struct deferred_split *ds_queue; |
| 3194 | 3283 | unsigned long flags; |
| 3195 | 3284 | |
| 3196 | − if (folio_order(folio) <= 1) | |
| 3197 | − return; | |
| 3198 | − | |
| 3199 | − /* | |
| 3200 | − * At this point, there is no one trying to add the folio to | |
| 3201 | − * deferred_list. If folio is not in deferred_list, it's safe | |
| 3202 | − * to check without acquiring the split_queue_lock. | |
| 3203 | − */ | |
| 3204 | − if (data_race(list_empty(&folio->_deferred_list))) | |
| 3205 | − return; | |
| 3206 | − | |
| 3207 | 3285 | ds_queue = get_deferred_split_queue(folio); |
| 3208 | 3286 | spin_lock_irqsave(&ds_queue->split_queue_lock, flags); |
| 3209 | 3287 | if (!list_empty(&folio->_deferred_list)) { |
| @@ -3248,6 +3326,7 @@ void deferred_split_folio(struct folio *folio) · 38 unmodified lines | ||
| 3248 | 3326 | if (list_empty(&folio->_deferred_list)) { |
| 3249 | 3327 | if (folio_test_pmd_mappable(folio)) |
| 3250 | 3328 | count_vm_event(THP_DEFERRED_SPLIT_PAGE); |
| 3329 | + count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); | |
| 3251 | 3330 | list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); |
| 3252 | 3331 | ds_queue->split_queue_len++; |
| 3253 | 3332 | #ifdef CONFIG_MEMCG |
| @@ -1355,6 +1355,10 @@ static struct folio *dequeue_hugetlb_folio_nodemask(struct hstate *h, gfp_t gfp_ · 1354 unmodified lines | ||
| 1355 | 1355 | struct zoneref *z; |
| 1356 | 1356 | int node = NUMA_NO_NODE; |
| 1357 | 1357 | |
| 1358 | + /* 'nid' should not be NUMA_NO_NODE. Try to catch any misuse of it and rectifiy. */ | |
| 1359 | + if (nid == NUMA_NO_NODE) | |
| 1360 | + nid = numa_node_id(); | |
| 1361 | + | |
| 1358 | 1362 | zonelist = node_zonelist(nid, gfp_mask); |
| 1359 | 1363 | |
| 1360 | 1364 | retry_cpuset: |
| @@ -2257,13 +2261,11 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, · 896 unmodified lines | ||
| 2257 | 2261 | * pages is zero. |
| 2258 | 2262 | */ |
| 2259 | 2263 | static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, |
| 2260 | − gfp_t gfp_mask, int nid, nodemask_t *nmask, | |
| 2261 | − nodemask_t *node_alloc_noretry) | |
| 2264 | + gfp_t gfp_mask, int nid, nodemask_t *nmask) | |
| 2262 | 2265 | { |
| 2263 | 2266 | struct folio *folio; |
| 2264 | 2267 | |
| 2265 | − folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, | |
| 2266 | − node_alloc_noretry); | |
| 2268 | + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); | |
| 2267 | 2269 | if (!folio) |
| 2268 | 2270 | return NULL; |
| 2269 | 2271 | |
| @@ -2481,7 +2483,7 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, · 211 unmodified lines | ||
| 2481 | 2483 | goto out_unlock; |
| 2482 | 2484 | spin_unlock_irq(&hugetlb_lock); |
| 2483 | 2485 | |
| 2484 | − folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); | |
| 2486 | + folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask); | |
| 2485 | 2487 | if (!folio) |
| 2486 | 2488 | return NULL; |
| 2487 | 2489 | |
| @@ -2517,7 +2519,7 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mas · 29 unmodified lines | ||
| 2517 | 2519 | if (hstate_is_gigantic(h)) |
| 2518 | 2520 | return NULL; |
| 2519 | 2521 | |
| 2520 | − folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); | |
| 2522 | + folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask); | |
| 2521 | 2523 | if (!folio) |
| 2522 | 2524 | return NULL; |
| 2523 | 2525 | |
| @@ -2586,6 +2588,23 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, · 62 unmodified lines | ||
| 2586 | 2588 | return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask); |
| 2587 | 2589 | } |
| 2588 | 2590 | |
| 2591 | +static nodemask_t *policy_mbind_nodemask(gfp_t gfp) | |
| 2592 | +{ | |
| 2593 | +#ifdef CONFIG_NUMA | |
| 2594 | + struct mempolicy *mpol = get_task_policy(current); | |
| 2595 | + | |
| 2596 | + /* | |
| 2597 | + * Only enforce MPOL_BIND policy which overlaps with cpuset policy | |
| 2598 | + * (from policy_nodemask) specifically for hugetlb case | |
| 2599 | + */ | |
| 2600 | + if (mpol->mode == MPOL_BIND && | |
| 2601 | + (apply_policy_zone(mpol, gfp_zone(gfp)) && | |
| 2602 | + cpuset_nodemask_valid_mems_allowed(&mpol->nodes))) | |
| 2603 | + return &mpol->nodes; | |
| 2604 | +#endif | |
| 2605 | + return NULL; | |
| 2606 | +} | |
| 2607 | + | |
| 2589 | 2608 | /* |
| 2590 | 2609 | * Increase the hugetlb pool such that it can accommodate a reservation |
| 2591 | 2610 | * of size 'delta'. |
| @@ -2599,6 +2618,8 @@ static int gather_surplus_pages(struct hstate *h, long delta) · 7 unmodified lines | ||
| 2599 | 2618 | long i; |
| 2600 | 2619 | long needed, allocated; |
| 2601 | 2620 | bool alloc_ok = true; |
| 2621 | + int node; | |
| 2622 | + nodemask_t *mbind_nodemask = policy_mbind_nodemask(htlb_alloc_mask(h)); | |
| 2602 | 2623 | |
| 2603 | 2624 | lockdep_assert_held(&hugetlb_lock); |
| 2604 | 2625 | needed = (h->resv_huge_pages + delta) - h->free_huge_pages; |
| @@ -2613,8 +2634,15 @@ static int gather_surplus_pages(struct hstate *h, long delta) · 8 unmodified lines | ||
| 2613 | 2634 | retry: |
| 2614 | 2635 | spin_unlock_irq(&hugetlb_lock); |
| 2615 | 2636 | for (i = 0; i < needed; i++) { |
| 2616 | − folio = alloc_surplus_hugetlb_folio(h, htlb_alloc_mask(h), | |
| 2617 | − NUMA_NO_NODE, NULL); | |
| 2637 | + folio = NULL; | |
| 2638 | + for_each_node_mask(node, cpuset_current_mems_allowed) { | |
| 2639 | + if (!mbind_nodemask || node_isset(node, *mbind_nodemask)) { | |
| 2640 | + folio = alloc_surplus_hugetlb_folio(h, htlb_alloc_mask(h), | |
| 2641 | + node, NULL); | |
| 2642 | + if (folio) | |
| 2643 | + break; | |
| 2644 | + } | |
| 2645 | + } | |
| 2618 | 2646 | if (!folio) { |
| 2619 | 2647 | alloc_ok = false; |
| 2620 | 2648 | break; |
| @@ -3439,7 +3467,7 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) · 818 unmodified lines | ||
| 3439 | 3467 | gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; |
| 3440 | 3468 | |
| 3441 | 3469 | folio = alloc_fresh_hugetlb_folio(h, gfp_mask, nid, |
| 3442 | − &node_states[N_MEMORY], NULL); | |
| 3470 | + &node_states[N_MEMORY]); | |
| 3443 | 3471 | if (!folio) |
| 3444 | 3472 | break; |
| 3445 | 3473 | free_huge_folio(folio); /* free it into the hugepage allocator */ |
| @@ -4617,7 +4645,7 @@ void __init hugetlb_add_hstate(unsigned int order) · 1171 unmodified lines | ||
| 4617 | 4645 | BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE); |
| 4618 | 4646 | BUG_ON(order < order_base_2(__NR_USED_SUBPAGE)); |
| 4619 | 4647 | h = &hstates[hugetlb_max_hstate++]; |
| 4620 | − mutex_init(&h->resize_lock); | |
| 4648 | + __mutex_init(&h->resize_lock, "resize mutex", &h->resize_key); | |
| 4621 | 4649 | h->order = order; |
| 4622 | 4650 | h->mask = ~(huge_page_size(h) - 1); |
| 4623 | 4651 | for (i = 0; i < MAX_NUMNODES; ++i) |
| @@ -4840,23 +4868,6 @@ static int __init default_hugepagesz_setup(char *s) · 216 unmodified lines | ||
| 4840 | 4868 | } |
| 4841 | 4869 | __setup("default_hugepagesz=", default_hugepagesz_setup); |
| 4842 | 4870 | |
| 4843 | −static nodemask_t *policy_mbind_nodemask(gfp_t gfp) | |
| 4844 | −{ | |
| 4845 | −#ifdef CONFIG_NUMA | |
| 4846 | − struct mempolicy *mpol = get_task_policy(current); | |
| 4847 | − | |
| 4848 | − /* | |
| 4849 | − * Only enforce MPOL_BIND policy which overlaps with cpuset policy | |
| 4850 | − * (from policy_nodemask) specifically for hugetlb case | |
| 4851 | − */ | |
| 4852 | − if (mpol->mode == MPOL_BIND && | |
| 4853 | − (apply_policy_zone(mpol, gfp_zone(gfp)) && | |
| 4854 | − cpuset_nodemask_valid_mems_allowed(&mpol->nodes))) | |
| 4855 | − return &mpol->nodes; | |
| 4856 | −#endif | |
| 4857 | − return NULL; | |
| 4858 | −} | |
| 4859 | − | |
| 4860 | 4871 | static unsigned int allowed_mems_nr(struct hstate *h) |
| 4861 | 4872 | { |
| 4862 | 4873 | int node; |
| @@ -4875,7 +4886,7 @@ static unsigned int allowed_mems_nr(struct hstate *h) · 12 unmodified lines | ||
| 4875 | 4886 | } |
| 4876 | 4887 | |
| 4877 | 4888 | #ifdef CONFIG_SYSCTL |
| 4878 | −static int proc_hugetlb_doulongvec_minmax(struct ctl_table *table, int write, | |
| 4889 | +static int proc_hugetlb_doulongvec_minmax(const struct ctl_table *table, int write, | |
| 4879 | 4890 | void *buffer, size_t *length, |
| 4880 | 4891 | loff_t *ppos, unsigned long *out) |
| 4881 | 4892 | { |
| @@ -4892,7 +4903,7 @@ static int proc_hugetlb_doulongvec_minmax(struct ctl_table *table, int write, · 10 unmodified lines | ||
| 4892 | 4903 | } |
| 4893 | 4904 | |
| 4894 | 4905 | static int hugetlb_sysctl_handler_common(bool obey_mempolicy, |
| 4895 | − struct ctl_table *table, int write, | |
| 4906 | + const struct ctl_table *table, int write, | |
| 4896 | 4907 | void *buffer, size_t *length, loff_t *ppos) |
| 4897 | 4908 | { |
| 4898 | 4909 | struct hstate *h = &default_hstate; |
| @@ -5279,7 +5290,7 @@ static void set_huge_ptep_writable(struct vm_area_struct *vma, · 380 unmodified lines | ||
| 5279 | 5290 | { |
| 5280 | 5291 | pte_t entry; |
| 5281 | 5292 | |
| 5282 | − entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep))); | |
| 5293 | + entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(vma->vm_mm, address, ptep))); | |
| 5283 | 5294 | if (huge_ptep_set_access_flags(vma, address, ptep, entry, 1)) |
| 5284 | 5295 | update_mmu_cache(vma, address, ptep); |
| 5285 | 5296 | } |
| @@ -5387,7 +5398,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, · 101 unmodified lines | ||
| 5387 | 5398 | dst_ptl = huge_pte_lock(h, dst, dst_pte); |
| 5388 | 5399 | src_ptl = huge_pte_lockptr(h, src, src_pte); |
| 5389 | 5400 | spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); |
| 5390 | − entry = huge_ptep_get(src_pte); | |
| 5401 | + entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte); | |
| 5391 | 5402 | again: |
| 5392 | 5403 | if (huge_pte_none(entry)) { |
| 5393 | 5404 | /* |
| @@ -5425,7 +5436,7 @@ again: · 31 unmodified lines | ||
| 5425 | 5436 | set_huge_pte_at(dst, addr, dst_pte, |
| 5426 | 5437 | make_pte_marker(marker), sz); |
| 5427 | 5438 | } else { |
| 5428 | − entry = huge_ptep_get(src_pte); | |
| 5439 | + entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte); | |
| 5429 | 5440 | pte_folio = page_folio(pte_page(entry)); |
| 5430 | 5441 | folio_get(pte_folio); |
| 5431 | 5442 | |
| @@ -5454,9 +5465,8 @@ again: · 22 unmodified lines | ||
| 5454 | 5465 | ret = PTR_ERR(new_folio); |
| 5455 | 5466 | break; |
| 5456 | 5467 | } |
| 5457 | − ret = copy_user_large_folio(new_folio, | |
| 5458 | − pte_folio, | |
| 5459 | − addr, dst_vma); | |
| 5468 | + ret = copy_user_large_folio(new_folio, pte_folio, | |
| 5469 | + ALIGN_DOWN(addr, sz), dst_vma); | |
| 5460 | 5470 | folio_put(pte_folio); |
| 5461 | 5471 | if (ret) { |
| 5462 | 5472 | folio_put(new_folio); |
| @@ -5467,7 +5477,7 @@ again: · 4 unmodified lines | ||
| 5467 | 5477 | dst_ptl = huge_pte_lock(h, dst, dst_pte); |
| 5468 | 5478 | src_ptl = huge_pte_lockptr(h, src, src_pte); |
| 5469 | 5479 | spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); |
| 5470 | − entry = huge_ptep_get(src_pte); | |
| 5480 | + entry = huge_ptep_get(src_vma->vm_mm, addr, src_pte); | |
| 5471 | 5481 | if (!pte_same(src_pte_old, entry)) { |
| 5472 | 5482 | restore_reserve_on_error(h, dst_vma, addr, |
| 5473 | 5483 | new_folio); |
| @@ -5577,7 +5587,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, · 103 unmodified lines | ||
| 5577 | 5587 | new_addr |= last_addr_mask; |
| 5578 | 5588 | continue; |
| 5579 | 5589 | } |
| 5580 | − if (huge_pte_none(huge_ptep_get(src_pte))) | |
| 5590 | + if (huge_pte_none(huge_ptep_get(mm, old_addr, src_pte))) | |
| 5581 | 5591 | continue; |
| 5582 | 5592 | |
| 5583 | 5593 | if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) { |
| @@ -5650,7 +5660,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, · 66 unmodified lines | ||
| 5650 | 5660 | continue; |
| 5651 | 5661 | } |
| 5652 | 5662 | |
| 5653 | − pte = huge_ptep_get(ptep); | |
| 5663 | + pte = huge_ptep_get(mm, address, ptep); | |
| 5654 | 5664 | if (huge_pte_none(pte)) { |
| 5655 | 5665 | spin_unlock(ptl); |
| 5656 | 5666 | continue; |
| @@ -5899,7 +5909,7 @@ static vm_fault_t hugetlb_wp(struct folio *pagecache_folio, · 242 unmodified lines | ||
| 5899 | 5909 | struct vm_area_struct *vma = vmf->vma; |
| 5900 | 5910 | struct mm_struct *mm = vma->vm_mm; |
| 5901 | 5911 | const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; |
| 5902 | − pte_t pte = huge_ptep_get(vmf->pte); | |
| 5912 | + pte_t pte = huge_ptep_get(mm, vmf->address, vmf->pte); | |
| 5903 | 5913 | struct hstate *h = hstate_vma(vma); |
| 5904 | 5914 | struct folio *old_folio; |
| 5905 | 5915 | struct folio *new_folio; |
| @@ -6020,7 +6030,7 @@ retry_avoidcopy: · 114 unmodified lines | ||
| 6020 | 6030 | vmf->pte = hugetlb_walk(vma, vmf->address, |
| 6021 | 6031 | huge_page_size(h)); |
| 6022 | 6032 | if (likely(vmf->pte && |
| 6023 | − pte_same(huge_ptep_get(vmf->pte), pte))) | |
| 6033 | + pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), pte))) | |
| 6024 | 6034 | goto retry_avoidcopy; |
| 6025 | 6035 | /* |
| 6026 | 6036 | * race occurs while re-acquiring page table |
| @@ -6058,7 +6068,7 @@ retry_avoidcopy: · 31 unmodified lines | ||
| 6058 | 6068 | */ |
| 6059 | 6069 | spin_lock(vmf->ptl); |
| 6060 | 6070 | vmf->pte = hugetlb_walk(vma, vmf->address, huge_page_size(h)); |
| 6061 | − if (likely(vmf->pte && pte_same(huge_ptep_get(vmf->pte), pte))) { | |
| 6071 | + if (likely(vmf->pte && pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), pte))) { | |
| 6062 | 6072 | pte_t newpte = make_huge_pte(vma, &new_folio->page, !unshare); |
| 6063 | 6073 | |
| 6064 | 6074 | /* Break COW or unshare */ |
| @@ -6159,14 +6169,14 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_fault *vmf, · 94 unmodified lines | ||
| 6159 | 6169 | * Recheck pte with pgtable lock. Returns true if pte didn't change, or |
| 6160 | 6170 | * false if pte changed or is changing. |
| 6161 | 6171 | */ |
| 6162 | −static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm, | |
| 6172 | +static bool hugetlb_pte_stable(struct hstate *h, struct mm_struct *mm, unsigned long addr, | |
| 6163 | 6173 | pte_t *ptep, pte_t old_pte) |
| 6164 | 6174 | { |
| 6165 | 6175 | spinlock_t *ptl; |
| 6166 | 6176 | bool same; |
| 6167 | 6177 | |
| 6168 | 6178 | ptl = huge_pte_lock(h, mm, ptep); |
| 6169 | − same = pte_same(huge_ptep_get(ptep), old_pte); | |
| 6179 | + same = pte_same(huge_ptep_get(mm, addr, ptep), old_pte); | |
| 6170 | 6180 | spin_unlock(ptl); |
| 6171 | 6181 | |
| 6172 | 6182 | return same; |
| @@ -6227,7 +6237,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, · 54 unmodified lines | ||
| 6227 | 6237 | * never happen on the page after UFFDIO_COPY has |
| 6228 | 6238 | * correctly installed the page and returned. |
| 6229 | 6239 | */ |
| 6230 | − if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { | |
| 6240 | + if (!hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte)) { | |
| 6231 | 6241 | ret = 0; |
| 6232 | 6242 | goto out; |
| 6233 | 6243 | } |
| @@ -6256,14 +6266,13 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, · 22 unmodified lines | ||
| 6256 | 6266 | * here. Before returning error, get ptl and make |
| 6257 | 6267 | * sure there really is no pte entry. |
| 6258 | 6268 | */ |
| 6259 | − if (hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) | |
| 6269 | + if (hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte)) | |
| 6260 | 6270 | ret = vmf_error(PTR_ERR(folio)); |
| 6261 | 6271 | else |
| 6262 | 6272 | ret = 0; |
| 6263 | 6273 | goto out; |
| 6264 | 6274 | } |
| 6265 | − clear_huge_page(&folio->page, vmf->real_address, | |
| 6266 | − pages_per_huge_page(h)); | |
| 6275 | + folio_zero_user(folio, vmf->real_address); | |
| 6267 | 6276 | __folio_mark_uptodate(folio); |
| 6268 | 6277 | new_folio = true; |
| 6269 | 6278 | |
| @@ -6306,7 +6315,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, · 36 unmodified lines | ||
| 6306 | 6315 | folio_unlock(folio); |
| 6307 | 6316 | folio_put(folio); |
| 6308 | 6317 | /* See comment in userfaultfd_missing() block above */ |
| 6309 | − if (!hugetlb_pte_stable(h, mm, vmf->pte, vmf->orig_pte)) { | |
| 6318 | + if (!hugetlb_pte_stable(h, mm, vmf->address, vmf->pte, vmf->orig_pte)) { | |
| 6310 | 6319 | ret = 0; |
| 6311 | 6320 | goto out; |
| 6312 | 6321 | } |
| @@ -6333,7 +6342,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, · 20 unmodified lines | ||
| 6333 | 6342 | vmf->ptl = huge_pte_lock(h, mm, vmf->pte); |
| 6334 | 6343 | ret = 0; |
| 6335 | 6344 | /* If pte changed from under us, retry */ |
| 6336 | − if (!pte_same(huge_ptep_get(vmf->pte), vmf->orig_pte)) | |
| 6345 | + if (!pte_same(huge_ptep_get(mm, vmf->address, vmf->pte), vmf->orig_pte)) | |
| 6337 | 6346 | goto backout; |
| 6338 | 6347 | |
| 6339 | 6348 | if (anon_rmap) |
| @@ -6454,7 +6463,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 114 unmodified lines | ||
| 6454 | 6463 | return VM_FAULT_OOM; |
| 6455 | 6464 | } |
| 6456 | 6465 | |
| 6457 | − vmf.orig_pte = huge_ptep_get(vmf.pte); | |
| 6466 | + vmf.orig_pte = huge_ptep_get(mm, vmf.address, vmf.pte); | |
| 6458 | 6467 | if (huge_pte_none_mostly(vmf.orig_pte)) { |
| 6459 | 6468 | if (is_pte_marker(vmf.orig_pte)) { |
| 6460 | 6469 | pte_marker marker = |
| @@ -6495,7 +6504,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 34 unmodified lines | ||
| 6495 | 6504 | * be released there. |
| 6496 | 6505 | */ |
| 6497 | 6506 | mutex_unlock(&hugetlb_fault_mutex_table[hash]); |
| 6498 | − migration_entry_wait_huge(vma, vmf.pte); | |
| 6507 | + migration_entry_wait_huge(vma, vmf.address, vmf.pte); | |
| 6499 | 6508 | return 0; |
| 6500 | 6509 | } else if (unlikely(is_hugetlb_entry_hwpoisoned(vmf.orig_pte))) |
| 6501 | 6510 | ret = VM_FAULT_HWPOISON_LARGE | |
| @@ -6528,11 +6537,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 26 unmodified lines | ||
| 6528 | 6537 | vmf.ptl = huge_pte_lock(h, mm, vmf.pte); |
| 6529 | 6538 | |
| 6530 | 6539 | /* Check for a racing update before calling hugetlb_wp() */ |
| 6531 | − if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(vmf.pte)))) | |
| 6540 | + if (unlikely(!pte_same(vmf.orig_pte, huge_ptep_get(mm, vmf.address, vmf.pte)))) | |
| 6532 | 6541 | goto out_ptl; |
| 6533 | 6542 | |
| 6534 | 6543 | /* Handle userfault-wp first, before trying to lock more pages */ |
| 6535 | − if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(vmf.pte)) && | |
| 6544 | + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(mm, vmf.address, vmf.pte)) && | |
| 6536 | 6545 | (flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) { |
| 6537 | 6546 | if (!userfaultfd_wp_async(vma)) { |
| 6538 | 6547 | spin_unlock(vmf.ptl); |
| @@ -6647,7 +6656,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 108 unmodified lines | ||
| 6647 | 6656 | struct hstate *h = hstate_vma(dst_vma); |
| 6648 | 6657 | struct address_space *mapping = dst_vma->vm_file->f_mapping; |
| 6649 | 6658 | pgoff_t idx = vma_hugecache_offset(h, dst_vma, dst_addr); |
| 6650 | − unsigned long size; | |
| 6659 | + unsigned long size = huge_page_size(h); | |
| 6651 | 6660 | int vm_shared = dst_vma->vm_flags & VM_SHARED; |
| 6652 | 6661 | pte_t _dst_pte; |
| 6653 | 6662 | spinlock_t *ptl; |
| @@ -6660,14 +6669,13 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 6 unmodified lines | ||
| 6660 | 6669 | ptl = huge_pte_lock(h, dst_mm, dst_pte); |
| 6661 | 6670 | |
| 6662 | 6671 | /* Don't overwrite any existing PTEs (even markers) */ |
| 6663 | − if (!huge_pte_none(huge_ptep_get(dst_pte))) { | |
| 6672 | + if (!huge_pte_none(huge_ptep_get(dst_mm, dst_addr, dst_pte))) { | |
| 6664 | 6673 | spin_unlock(ptl); |
| 6665 | 6674 | return -EEXIST; |
| 6666 | 6675 | } |
| 6667 | 6676 | |
| 6668 | 6677 | _dst_pte = make_pte_marker(PTE_MARKER_POISONED); |
| 6669 | − set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, | |
| 6670 | − huge_page_size(h)); | |
| 6678 | + set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, size); | |
| 6671 | 6679 | |
| 6672 | 6680 | /* No need to invalidate - it was non-present before */ |
| 6673 | 6681 | update_mmu_cache(dst_vma, dst_addr, dst_pte); |
| @@ -6741,7 +6749,8 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 67 unmodified lines | ||
| 6741 | 6749 | *foliop = NULL; |
| 6742 | 6750 | goto out; |
| 6743 | 6751 | } |
| 6744 | − ret = copy_user_large_folio(folio, *foliop, dst_addr, dst_vma); | |
| 6752 | + ret = copy_user_large_folio(folio, *foliop, | |
| 6753 | + ALIGN_DOWN(dst_addr, size), dst_vma); | |
| 6745 | 6754 | folio_put(*foliop); |
| 6746 | 6755 | *foliop = NULL; |
| 6747 | 6756 | if (ret) { |
| @@ -6768,9 +6777,8 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 20 unmodified lines | ||
| 6768 | 6777 | |
| 6769 | 6778 | /* Add shared, newly allocated pages to the page cache. */ |
| 6770 | 6779 | if (vm_shared && !is_continue) { |
| 6771 | − size = i_size_read(mapping->host) >> huge_page_shift(h); | |
| 6772 | 6780 | ret = -EFAULT; |
| 6773 | − if (idx >= size) | |
| 6781 | + if (idx >= (i_size_read(mapping->host) >> huge_page_shift(h))) | |
| 6774 | 6782 | goto out_release_nounlock; |
| 6775 | 6783 | |
| 6776 | 6784 | /* |
| @@ -6797,7 +6805,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 20 unmodified lines | ||
| 6797 | 6805 | * page backing it, then access the page. |
| 6798 | 6806 | */ |
| 6799 | 6807 | ret = -EEXIST; |
| 6800 | − if (!huge_pte_none_mostly(huge_ptep_get(dst_pte))) | |
| 6808 | + if (!huge_pte_none_mostly(huge_ptep_get(dst_mm, dst_addr, dst_pte))) | |
| 6801 | 6809 | goto out_release_unlock; |
| 6802 | 6810 | |
| 6803 | 6811 | if (folio_in_pagecache) |
| @@ -6827,7 +6835,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 23 unmodified lines | ||
| 6827 | 6835 | if (wp_enabled) |
| 6828 | 6836 | _dst_pte = huge_pte_mkuffd_wp(_dst_pte); |
| 6829 | 6837 | |
| 6830 | − set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, huge_page_size(h)); | |
| 6838 | + set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, size); | |
| 6831 | 6839 | |
| 6832 | 6840 | hugetlb_count_add(pages_per_huge_page(h), dst_mm); |
| 6833 | 6841 | |
| @@ -6918,7 +6926,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, · 84 unmodified lines | ||
| 6918 | 6926 | address |= last_addr_mask; |
| 6919 | 6927 | continue; |
| 6920 | 6928 | } |
| 6921 | − pte = huge_ptep_get(ptep); | |
| 6929 | + pte = huge_ptep_get(mm, address, ptep); | |
| 6922 | 6930 | if (unlikely(is_hugetlb_entry_hwpoisoned(pte))) { |
| 6923 | 6931 | /* Nothing to do. */ |
| 6924 | 6932 | } else if (unlikely(is_hugetlb_entry_migration(pte))) { |
| @@ -27,7 +27,17 @@ · 26 unmodified lines | ||
| 27 | 27 | #define MEMFILE_IDX(val) (((val) >> 16) & 0xffff) |
| 28 | 28 | #define MEMFILE_ATTR(val) ((val) & 0xffff) |
| 29 | 29 | |
| 30 | +/* Use t->m[0] to encode the offset */ | |
| 31 | +#define MEMFILE_OFFSET(t, m0) (((offsetof(t, m0) << 16) | sizeof_field(t, m0))) | |
| 32 | +#define MEMFILE_OFFSET0(val) (((val) >> 16) & 0xffff) | |
| 33 | +#define MEMFILE_FIELD_SIZE(val) ((val) & 0xffff) | |
| 34 | + | |
| 35 | +#define DFL_TMPL_SIZE ARRAY_SIZE(hugetlb_dfl_tmpl) | |
| 36 | +#define LEGACY_TMPL_SIZE ARRAY_SIZE(hugetlb_legacy_tmpl) | |
| 37 | + | |
| 30 | 38 | static struct hugetlb_cgroup *root_h_cgroup __read_mostly; |
| 39 | +static struct cftype *dfl_files; | |
| 40 | +static struct cftype *legacy_files; | |
| 31 | 41 | |
| 32 | 42 | static inline struct page_counter * |
| 33 | 43 | __hugetlb_cgroup_counter_from_cgroup(struct hugetlb_cgroup *h_cg, int idx, |
| @@ -460,7 +470,7 @@ static int hugetlb_cgroup_read_numa_stat(struct seq_file *seq, void *dummy) · 426 unmodified lines | ||
| 460 | 470 | int nid; |
| 461 | 471 | struct cftype *cft = seq_cft(seq); |
| 462 | 472 | int idx = MEMFILE_IDX(cft->private); |
| 463 | − bool legacy = MEMFILE_ATTR(cft->private); | |
| 473 | + bool legacy = !cgroup_subsys_on_dfl(hugetlb_cgrp_subsys); | |
| 464 | 474 | struct hugetlb_cgroup *h_cg = hugetlb_cgroup_from_css(seq_css(seq)); |
| 465 | 475 | struct cgroup_subsys_state *css; |
| 466 | 476 | unsigned long usage; |
| @@ -702,166 +712,185 @@ static int hugetlb_events_local_show(struct seq_file *seq, void *v) · 235 unmodified lines | ||
| 702 | 712 | return __hugetlb_events_show(seq, true); |
| 703 | 713 | } |
| 704 | 714 | |
| 705 | −static void __init __hugetlb_cgroup_file_dfl_init(int idx) | |
| 715 | +static struct cftype hugetlb_dfl_tmpl[] = { | |
| 716 | + { | |
| 717 | + .name = "max", | |
| 718 | + .private = RES_LIMIT, | |
| 719 | + .seq_show = hugetlb_cgroup_read_u64_max, | |
| 720 | + .write = hugetlb_cgroup_write_dfl, | |
| 721 | + .flags = CFTYPE_NOT_ON_ROOT, | |
| 722 | + }, | |
| 723 | + { | |
| 724 | + .name = "rsvd.max", | |
| 725 | + .private = RES_RSVD_LIMIT, | |
| 726 | + .seq_show = hugetlb_cgroup_read_u64_max, | |
| 727 | + .write = hugetlb_cgroup_write_dfl, | |
| 728 | + .flags = CFTYPE_NOT_ON_ROOT, | |
| 729 | + }, | |
| 730 | + { | |
| 731 | + .name = "current", | |
| 732 | + .private = RES_USAGE, | |
| 733 | + .seq_show = hugetlb_cgroup_read_u64_max, | |
| 734 | + .flags = CFTYPE_NOT_ON_ROOT, | |
| 735 | + }, | |
| 736 | + { | |
| 737 | + .name = "rsvd.current", | |
| 738 | + .private = RES_RSVD_USAGE, | |
| 739 | + .seq_show = hugetlb_cgroup_read_u64_max, | |
| 740 | + .flags = CFTYPE_NOT_ON_ROOT, | |
| 741 | + }, | |
| 742 | + { | |
| 743 | + .name = "events", | |
| 744 | + .seq_show = hugetlb_events_show, | |
| 745 | + .file_offset = MEMFILE_OFFSET(struct hugetlb_cgroup, events_file[0]), | |
| 746 | + .flags = CFTYPE_NOT_ON_ROOT, | |
| 747 | + }, | |
| 748 | + { | |
| 749 | + .name = "events.local", | |
| 750 | + .seq_show = hugetlb_events_local_show, | |
| 751 | + .file_offset = MEMFILE_OFFSET(struct hugetlb_cgroup, events_local_file[0]), | |
| 752 | + .flags = CFTYPE_NOT_ON_ROOT, | |
| 753 | + }, | |
| 754 | + { | |
| 755 | + .name = "numa_stat", | |
| 756 | + .seq_show = hugetlb_cgroup_read_numa_stat, | |
| 757 | + .flags = CFTYPE_NOT_ON_ROOT, | |
| 758 | + }, | |
| 759 | + /* don't need terminator here */ | |
| 760 | +}; | |
| 761 | + | |
| 762 | +static struct cftype hugetlb_legacy_tmpl[] = { | |
| 763 | + { | |
| 764 | + .name = "limit_in_bytes", | |
| 765 | + .private = RES_LIMIT, | |
| 766 | + .read_u64 = hugetlb_cgroup_read_u64, | |
| 767 | + .write = hugetlb_cgroup_write_legacy, | |
| 768 | + }, | |
| 769 | + { | |
| 770 | + .name = "rsvd.limit_in_bytes", | |
| 771 | + .private = RES_RSVD_LIMIT, | |
| 772 | + .read_u64 = hugetlb_cgroup_read_u64, | |
| 773 | + .write = hugetlb_cgroup_write_legacy, | |
| 774 | + }, | |
| 775 | + { | |
| 776 | + .name = "usage_in_bytes", | |
| 777 | + .private = RES_USAGE, | |
| 778 | + .read_u64 = hugetlb_cgroup_read_u64, | |
| 779 | + }, | |
| 780 | + { | |
| 781 | + .name = "rsvd.usage_in_bytes", | |
| 782 | + .private = RES_RSVD_USAGE, | |
| 783 | + .read_u64 = hugetlb_cgroup_read_u64, | |
| 784 | + }, | |
| 785 | + { | |
| 786 | + .name = "max_usage_in_bytes", | |
| 787 | + .private = RES_MAX_USAGE, | |
| 788 | + .write = hugetlb_cgroup_reset, | |
| 789 | + .read_u64 = hugetlb_cgroup_read_u64, | |
| 790 | + }, | |
| 791 | + { | |
| 792 | + .name = "rsvd.max_usage_in_bytes", | |
| 793 | + .private = RES_RSVD_MAX_USAGE, | |
| 794 | + .write = hugetlb_cgroup_reset, | |
| 795 | + .read_u64 = hugetlb_cgroup_read_u64, | |
| 796 | + }, | |
| 797 | + { | |
| 798 | + .name = "failcnt", | |
| 799 | + .private = RES_FAILCNT, | |
| 800 | + .write = hugetlb_cgroup_reset, | |
| 801 | + .read_u64 = hugetlb_cgroup_read_u64, | |
| 802 | + }, | |
| 803 | + { | |
| 804 | + .name = "rsvd.failcnt", | |
| 805 | + .private = RES_RSVD_FAILCNT, | |
| 806 | + .write = hugetlb_cgroup_reset, | |
| 807 | + .read_u64 = hugetlb_cgroup_read_u64, | |
| 808 | + }, | |
| 809 | + { | |
| 810 | + .name = "numa_stat", | |
| 811 | + .seq_show = hugetlb_cgroup_read_numa_stat, | |
| 812 | + }, | |
| 813 | + /* don't need terminator here */ | |
| 814 | +}; | |
| 815 | + | |
| 816 | +static void __init | |
| 817 | +hugetlb_cgroup_cfttypes_init(struct hstate *h, struct cftype *cft, | |
| 818 | + struct cftype *tmpl, int tmpl_size) | |
| 706 | 819 | { |
| 707 | 820 | char buf[32]; |
| 708 | − struct cftype *cft; | |
| 709 | − struct hstate *h = &hstates[idx]; | |
| 821 | + int i, idx = hstate_index(h); | |
| 710 | 822 | |
| 711 | 823 | /* format the size */ |
| 712 | 824 | mem_fmt(buf, sizeof(buf), huge_page_size(h)); |
| 713 | 825 | |
| 714 | − /* Add the limit file */ | |
| 715 | − cft = &h->cgroup_files_dfl[0]; | |
| 716 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.max", buf); | |
| 717 | − cft->private = MEMFILE_PRIVATE(idx, RES_LIMIT); | |
| 718 | − cft->seq_show = hugetlb_cgroup_read_u64_max; | |
| 719 | − cft->write = hugetlb_cgroup_write_dfl; | |
| 720 | − cft->flags = CFTYPE_NOT_ON_ROOT; | |
| 721 | − | |
| 722 | − /* Add the reservation limit file */ | |
| 723 | − cft = &h->cgroup_files_dfl[1]; | |
| 724 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.max", buf); | |
| 725 | − cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_LIMIT); | |
| 726 | − cft->seq_show = hugetlb_cgroup_read_u64_max; | |
| 727 | − cft->write = hugetlb_cgroup_write_dfl; | |
| 728 | − cft->flags = CFTYPE_NOT_ON_ROOT; | |
| 729 | − | |
| 730 | − /* Add the current usage file */ | |
| 731 | − cft = &h->cgroup_files_dfl[2]; | |
| 732 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.current", buf); | |
| 733 | − cft->private = MEMFILE_PRIVATE(idx, RES_USAGE); | |
| 734 | − cft->seq_show = hugetlb_cgroup_read_u64_max; | |
| 735 | − cft->flags = CFTYPE_NOT_ON_ROOT; | |
| 736 | − | |
| 737 | − /* Add the current reservation usage file */ | |
| 738 | − cft = &h->cgroup_files_dfl[3]; | |
| 739 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.current", buf); | |
| 740 | − cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_USAGE); | |
| 741 | − cft->seq_show = hugetlb_cgroup_read_u64_max; | |
| 742 | − cft->flags = CFTYPE_NOT_ON_ROOT; | |
| 743 | − | |
| 744 | − /* Add the events file */ | |
| 745 | − cft = &h->cgroup_files_dfl[4]; | |
| 746 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.events", buf); | |
| 747 | − cft->private = MEMFILE_PRIVATE(idx, 0); | |
| 748 | − cft->seq_show = hugetlb_events_show; | |
| 749 | − cft->file_offset = offsetof(struct hugetlb_cgroup, events_file[idx]); | |
| 750 | − cft->flags = CFTYPE_NOT_ON_ROOT; | |
| 751 | − | |
| 752 | − /* Add the events.local file */ | |
| 753 | − cft = &h->cgroup_files_dfl[5]; | |
| 754 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.events.local", buf); | |
| 755 | − cft->private = MEMFILE_PRIVATE(idx, 0); | |
| 756 | − cft->seq_show = hugetlb_events_local_show; | |
| 757 | − cft->file_offset = offsetof(struct hugetlb_cgroup, | |
| 758 | − events_local_file[idx]); | |
| 759 | − cft->flags = CFTYPE_NOT_ON_ROOT; | |
| 760 | − | |
| 761 | − /* Add the numa stat file */ | |
| 762 | − cft = &h->cgroup_files_dfl[6]; | |
| 763 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.numa_stat", buf); | |
| 764 | − cft->private = MEMFILE_PRIVATE(idx, 0); | |
| 765 | − cft->seq_show = hugetlb_cgroup_read_numa_stat; | |
| 766 | − cft->flags = CFTYPE_NOT_ON_ROOT; | |
| 767 | − | |
| 768 | − /* NULL terminate the last cft */ | |
| 769 | − cft = &h->cgroup_files_dfl[7]; | |
| 770 | − memset(cft, 0, sizeof(*cft)); | |
| 826 | + for (i = 0; i < tmpl_size; cft++, tmpl++, i++) { | |
| 827 | + *cft = *tmpl; | |
| 828 | + /* rebuild the name */ | |
| 829 | + snprintf(cft->name, MAX_CFTYPE_NAME, "%s.%s", buf, tmpl->name); | |
| 830 | + /* rebuild the private */ | |
| 831 | + cft->private = MEMFILE_PRIVATE(idx, tmpl->private); | |
| 832 | + /* rebuild the file_offset */ | |
| 833 | + if (tmpl->file_offset) { | |
| 834 | + unsigned int offset = tmpl->file_offset; | |
| 835 | + | |
| 836 | + cft->file_offset = MEMFILE_OFFSET0(offset) + | |
| 837 | + MEMFILE_FIELD_SIZE(offset) * idx; | |
| 838 | + } | |
| 771 | 839 | |
| 772 | − WARN_ON(cgroup_add_dfl_cftypes(&hugetlb_cgrp_subsys, | |
| 773 | − h->cgroup_files_dfl)); | |
| 840 | + lockdep_register_key(&cft->lockdep_key); | |
| 841 | + } | |
| 774 | 842 | } |
| 775 | 843 | |
| 776 | −static void __init __hugetlb_cgroup_file_legacy_init(int idx) | |
| 844 | +static void __init __hugetlb_cgroup_file_dfl_init(struct hstate *h) | |
| 777 | 845 | { |
| 778 | − char buf[32]; | |
| 779 | − struct cftype *cft; | |
| 780 | − struct hstate *h = &hstates[idx]; | |
| 846 | + int idx = hstate_index(h); | |
| 781 | 847 | |
| 782 | − /* format the size */ | |
| 783 | − mem_fmt(buf, sizeof(buf), huge_page_size(h)); | |
| 848 | + hugetlb_cgroup_cfttypes_init(h, dfl_files + idx * DFL_TMPL_SIZE, | |
| 849 | + hugetlb_dfl_tmpl, DFL_TMPL_SIZE); | |
| 850 | +} | |
| 784 | 851 | |
| 785 | − /* Add the limit file */ | |
| 786 | − cft = &h->cgroup_files_legacy[0]; | |
| 787 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.limit_in_bytes", buf); | |
| 788 | − cft->private = MEMFILE_PRIVATE(idx, RES_LIMIT); | |
| 789 | − cft->read_u64 = hugetlb_cgroup_read_u64; | |
| 790 | − cft->write = hugetlb_cgroup_write_legacy; | |
| 791 | − | |
| 792 | − /* Add the reservation limit file */ | |
| 793 | − cft = &h->cgroup_files_legacy[1]; | |
| 794 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.limit_in_bytes", buf); | |
| 795 | − cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_LIMIT); | |
| 796 | − cft->read_u64 = hugetlb_cgroup_read_u64; | |
| 797 | − cft->write = hugetlb_cgroup_write_legacy; | |
| 798 | − | |
| 799 | − /* Add the usage file */ | |
| 800 | − cft = &h->cgroup_files_legacy[2]; | |
| 801 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.usage_in_bytes", buf); | |
| 802 | − cft->private = MEMFILE_PRIVATE(idx, RES_USAGE); | |
| 803 | − cft->read_u64 = hugetlb_cgroup_read_u64; | |
| 804 | − | |
| 805 | − /* Add the reservation usage file */ | |
| 806 | − cft = &h->cgroup_files_legacy[3]; | |
| 807 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.usage_in_bytes", buf); | |
| 808 | − cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_USAGE); | |
| 809 | − cft->read_u64 = hugetlb_cgroup_read_u64; | |
| 810 | − | |
| 811 | − /* Add the MAX usage file */ | |
| 812 | − cft = &h->cgroup_files_legacy[4]; | |
| 813 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.max_usage_in_bytes", buf); | |
| 814 | − cft->private = MEMFILE_PRIVATE(idx, RES_MAX_USAGE); | |
| 815 | − cft->write = hugetlb_cgroup_reset; | |
| 816 | − cft->read_u64 = hugetlb_cgroup_read_u64; | |
| 817 | − | |
| 818 | − /* Add the MAX reservation usage file */ | |
| 819 | − cft = &h->cgroup_files_legacy[5]; | |
| 820 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.max_usage_in_bytes", buf); | |
| 821 | − cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_MAX_USAGE); | |
| 822 | − cft->write = hugetlb_cgroup_reset; | |
| 823 | − cft->read_u64 = hugetlb_cgroup_read_u64; | |
| 824 | − | |
| 825 | − /* Add the failcntfile */ | |
| 826 | − cft = &h->cgroup_files_legacy[6]; | |
| 827 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.failcnt", buf); | |
| 828 | − cft->private = MEMFILE_PRIVATE(idx, RES_FAILCNT); | |
| 829 | − cft->write = hugetlb_cgroup_reset; | |
| 830 | − cft->read_u64 = hugetlb_cgroup_read_u64; | |
| 831 | − | |
| 832 | − /* Add the reservation failcntfile */ | |
| 833 | − cft = &h->cgroup_files_legacy[7]; | |
| 834 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.rsvd.failcnt", buf); | |
| 835 | − cft->private = MEMFILE_PRIVATE(idx, RES_RSVD_FAILCNT); | |
| 836 | − cft->write = hugetlb_cgroup_reset; | |
| 837 | − cft->read_u64 = hugetlb_cgroup_read_u64; | |
| 838 | − | |
| 839 | − /* Add the numa stat file */ | |
| 840 | − cft = &h->cgroup_files_legacy[8]; | |
| 841 | − snprintf(cft->name, MAX_CFTYPE_NAME, "%s.numa_stat", buf); | |
| 842 | − cft->private = MEMFILE_PRIVATE(idx, 1); | |
| 843 | − cft->seq_show = hugetlb_cgroup_read_numa_stat; | |
| 844 | − | |
| 845 | − /* NULL terminate the last cft */ | |
| 846 | − cft = &h->cgroup_files_legacy[9]; | |
| 847 | − memset(cft, 0, sizeof(*cft)); | |
| 852 | +static void __init __hugetlb_cgroup_file_legacy_init(struct hstate *h) | |
| 853 | +{ | |
| 854 | + int idx = hstate_index(h); | |
| 848 | 855 | |
| 849 | − WARN_ON(cgroup_add_legacy_cftypes(&hugetlb_cgrp_subsys, | |
| 850 | − h->cgroup_files_legacy)); | |
| 856 | + hugetlb_cgroup_cfttypes_init(h, legacy_files + idx * LEGACY_TMPL_SIZE, | |
| 857 | + hugetlb_legacy_tmpl, LEGACY_TMPL_SIZE); | |
| 858 | +} | |
| 859 | + | |
| 860 | +static void __init __hugetlb_cgroup_file_init(struct hstate *h) | |
| 861 | +{ | |
| 862 | + __hugetlb_cgroup_file_dfl_init(h); | |
| 863 | + __hugetlb_cgroup_file_legacy_init(h); | |
| 851 | 864 | } |
| 852 | 865 | |
| 853 | −static void __init __hugetlb_cgroup_file_init(int idx) | |
| 866 | +static void __init __hugetlb_cgroup_file_pre_init(void) | |
| 854 | 867 | { |
| 855 | − __hugetlb_cgroup_file_dfl_init(idx); | |
| 856 | − __hugetlb_cgroup_file_legacy_init(idx); | |
| 868 | + int cft_count; | |
| 869 | + | |
| 870 | + cft_count = hugetlb_max_hstate * DFL_TMPL_SIZE + 1; /* add terminator */ | |
| 871 | + dfl_files = kcalloc(cft_count, sizeof(struct cftype), GFP_KERNEL); | |
| 872 | + BUG_ON(!dfl_files); | |
| 873 | + cft_count = hugetlb_max_hstate * LEGACY_TMPL_SIZE + 1; /* add terminator */ | |
| 874 | + legacy_files = kcalloc(cft_count, sizeof(struct cftype), GFP_KERNEL); | |
| 875 | + BUG_ON(!legacy_files); | |
| 876 | +} | |
| 877 | + | |
| 878 | +static void __init __hugetlb_cgroup_file_post_init(void) | |
| 879 | +{ | |
| 880 | + WARN_ON(cgroup_add_dfl_cftypes(&hugetlb_cgrp_subsys, | |
| 881 | + dfl_files)); | |
| 882 | + WARN_ON(cgroup_add_legacy_cftypes(&hugetlb_cgrp_subsys, | |
| 883 | + legacy_files)); | |
| 857 | 884 | } |
| 858 | 885 | |
| 859 | 886 | void __init hugetlb_cgroup_file_init(void) |
| 860 | 887 | { |
| 861 | 888 | struct hstate *h; |
| 862 | 889 | |
| 890 | + __hugetlb_cgroup_file_pre_init(); | |
| 863 | 891 | for_each_hstate(h) |
| 864 | − __hugetlb_cgroup_file_init(hstate_index(h)); | |
| 892 | + __hugetlb_cgroup_file_init(h); | |
| 893 | + __hugetlb_cgroup_file_post_init(); | |
| 865 | 894 | } |
| 866 | 895 | |
| 867 | 896 | /* |
| @@ -184,10 +184,13 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, · 183 unmodified lines | ||
| 184 | 184 | */ |
| 185 | 185 | static inline void free_vmemmap_page(struct page *page) |
| 186 | 186 | { |
| 187 | − if (PageReserved(page)) | |
| 187 | + if (PageReserved(page)) { | |
| 188 | 188 | free_bootmem_page(page); |
| 189 | − else | |
| 189 | + mod_node_page_state(page_pgdat(page), NR_MEMMAP_BOOT, -1); | |
| 190 | + } else { | |
| 190 | 191 | __free_page(page); |
| 192 | + mod_node_page_state(page_pgdat(page), NR_MEMMAP, -1); | |
| 193 | + } | |
| 191 | 194 | } |
| 192 | 195 | |
| 193 | 196 | /* Free a list of the vmemmap pages */ |
| @@ -338,6 +341,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, · 144 unmodified lines | ||
| 338 | 341 | copy_page(page_to_virt(walk.reuse_page), |
| 339 | 342 | (void *)walk.reuse_addr); |
| 340 | 343 | list_add(&walk.reuse_page->lru, vmemmap_pages); |
| 344 | + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, 1); | |
| 341 | 345 | } |
| 342 | 346 | |
| 343 | 347 | /* |
| @@ -384,14 +388,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, · 40 unmodified lines | ||
| 384 | 388 | unsigned long nr_pages = (end - start) >> PAGE_SHIFT; |
| 385 | 389 | int nid = page_to_nid((struct page *)start); |
| 386 | 390 | struct page *page, *next; |
| 391 | + int i; | |
| 387 | 392 | |
| 388 | − while (nr_pages--) { | |
| 393 | + for (i = 0; i < nr_pages; i++) { | |
| 389 | 394 | page = alloc_pages_node(nid, gfp_mask, 0); |
| 390 | − if (!page) | |
| 395 | + if (!page) { | |
| 396 | + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, i); | |
| 391 | 397 | goto out; |
| 398 | + } | |
| 392 | 399 | list_add(&page->lru, list); |
| 393 | 400 | } |
| 394 | 401 | |
| 402 | + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, nr_pages); | |
| 403 | + | |
| 395 | 404 | return 0; |
| 396 | 405 | out: |
| 397 | 406 | list_for_each_entry_safe(page, next, list, lru) |
| @@ -110,4 +110,5 @@ static int __init pfn_inject_init(void) · 109 unmodified lines | ||
| 110 | 110 | |
| 111 | 111 | module_init(pfn_inject_init); |
| 112 | 112 | module_exit(pfn_inject_exit); |
| 113 | +MODULE_DESCRIPTION("HWPoison pages injector"); | |
| 113 | 114 | MODULE_LICENSE("GPL"); |
| @@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, · 210 unmodified lines | ||
| 211 | 211 | } |
| 212 | 212 | |
| 213 | 213 | /** |
| 214 | − * pte_next_swp_offset - Increment the swap entry offset field of a swap pte. | |
| 214 | + * pte_move_swp_offset - Move the swap entry offset field of a swap pte | |
| 215 | + * forward or backward by delta | |
| 215 | 216 | * @pte: The initial pte state; is_swap_pte(pte) must be true and |
| 216 | 217 | * non_swap_entry() must be false. |
| 218 | + * @delta: The direction and the offset we are moving; forward if delta | |
| 219 | + * is positive; backward if delta is negative | |
| 217 | 220 | * |
| 218 | − * Increments the swap offset, while maintaining all other fields, including | |
| 221 | + * Moves the swap offset, while maintaining all other fields, including | |
| 219 | 222 | * swap type, and any swp pte bits. The resulting pte is returned. |
| 220 | 223 | */ |
| 221 | −static inline pte_t pte_next_swp_offset(pte_t pte) | |
| 224 | +static inline pte_t pte_move_swp_offset(pte_t pte, long delta) | |
| 222 | 225 | { |
| 223 | 226 | swp_entry_t entry = pte_to_swp_entry(pte); |
| 224 | 227 | pte_t new = __swp_entry_to_pte(__swp_entry(swp_type(entry), |
| 225 | − (swp_offset(entry) + 1))); | |
| 228 | + (swp_offset(entry) + delta))); | |
| 226 | 229 | |
| 227 | 230 | if (pte_swp_soft_dirty(pte)) |
| 228 | 231 | new = pte_swp_mksoft_dirty(new); |
| @@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte) · 5 unmodified lines | ||
| 234 | 237 | return new; |
| 235 | 238 | } |
| 236 | 239 | |
| 240 | + | |
| 241 | +/** | |
| 242 | + * pte_next_swp_offset - Increment the swap entry offset field of a swap pte. | |
| 243 | + * @pte: The initial pte state; is_swap_pte(pte) must be true and | |
| 244 | + * non_swap_entry() must be false. | |
| 245 | + * | |
| 246 | + * Increments the swap offset, while maintaining all other fields, including | |
| 247 | + * swap type, and any swp pte bits. The resulting pte is returned. | |
| 248 | + */ | |
| 249 | +static inline pte_t pte_next_swp_offset(pte_t pte) | |
| 250 | +{ | |
| 251 | + return pte_move_swp_offset(pte, 1); | |
| 252 | +} | |
| 253 | + | |
| 237 | 254 | /** |
| 238 | 255 | * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries |
| 239 | 256 | * @start_ptep: Page table pointer for the first entry. |
| @@ -587,7 +604,8 @@ extern void __putback_isolated_page(struct page *page, unsigned int order, · 347 unmodified lines | ||
| 587 | 604 | int mt); |
| 588 | 605 | extern void memblock_free_pages(struct page *page, unsigned long pfn, |
| 589 | 606 | unsigned int order); |
| 590 | −extern void __free_pages_core(struct page *page, unsigned int order); | |
| 607 | +extern void __free_pages_core(struct page *page, unsigned int order, | |
| 608 | + enum meminit_context context); | |
| 591 | 609 | |
| 592 | 610 | /* |
| 593 | 611 | * This will have no effect, other than possibly generating a warning, if the |
| @@ -604,7 +622,22 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) · 10 unmodified lines | ||
| 604 | 622 | #endif |
| 605 | 623 | } |
| 606 | 624 | |
| 607 | −void folio_undo_large_rmappable(struct folio *folio); | |
| 625 | +void __folio_undo_large_rmappable(struct folio *folio); | |
| 626 | +static inline void folio_undo_large_rmappable(struct folio *folio) | |
| 627 | +{ | |
| 628 | + if (folio_order(folio) <= 1 || !folio_test_large_rmappable(folio)) | |
| 629 | + return; | |
| 630 | + | |
| 631 | + /* | |
| 632 | + * At this point, there is no one trying to add the folio to | |
| 633 | + * deferred_list. If folio is not in deferred_list, it's safe | |
| 634 | + * to check without acquiring the split_queue_lock. | |
| 635 | + */ | |
| 636 | + if (data_race(list_empty(&folio->_deferred_list))) | |
| 637 | + return; | |
| 638 | + | |
| 639 | + __folio_undo_large_rmappable(folio); | |
| 640 | +} | |
| 608 | 641 | |
| 609 | 642 | static inline struct folio *page_rmappable_folio(struct page *page) |
| 610 | 643 | { |
| @@ -1045,12 +1078,23 @@ extern u64 hwpoison_filter_flags_mask; · 434 unmodified lines | ||
| 1045 | 1078 | extern u64 hwpoison_filter_flags_value; |
| 1046 | 1079 | extern u64 hwpoison_filter_memcg; |
| 1047 | 1080 | extern u32 hwpoison_filter_enable; |
| 1081 | +#define MAGIC_HWPOISON 0x48575053U /* HWPS */ | |
| 1082 | +void SetPageHWPoisonTakenOff(struct page *page); | |
| 1083 | +void ClearPageHWPoisonTakenOff(struct page *page); | |
| 1084 | +bool take_page_off_buddy(struct page *page); | |
| 1085 | +bool put_page_back_buddy(struct page *page); | |
| 1086 | +struct task_struct *task_early_kill(struct task_struct *tsk, int force_early); | |
| 1087 | +void add_to_kill_ksm(struct task_struct *tsk, struct page *p, | |
| 1088 | + struct vm_area_struct *vma, struct list_head *to_kill, | |
| 1089 | + unsigned long ksm_addr); | |
| 1090 | +unsigned long page_mapped_in_vma(struct page *page, struct vm_area_struct *vma); | |
| 1048 | 1091 | |
| 1049 | 1092 | extern unsigned long __must_check vm_mmap_pgoff(struct file *, unsigned long, |
| 1050 | 1093 | unsigned long, unsigned long, |
| 1051 | 1094 | unsigned long, unsigned long); |
| 1052 | 1095 | |
| 1053 | 1096 | extern void set_pageblock_order(void); |
| 1097 | +struct folio *alloc_migrate_folio(struct folio *src, unsigned long private); | |
| 1054 | 1098 | unsigned long reclaim_pages(struct list_head *folio_list); |
| 1055 | 1099 | unsigned int reclaim_clean_pages_from_list(struct zone *zone, |
| 1056 | 1100 | struct list_head *folio_list); |
| @@ -1316,6 +1360,16 @@ static inline bool vma_soft_dirty_enabled(struct vm_area_struct *vma) · 259 unmodified lines | ||
| 1316 | 1360 | return !(vma->vm_flags & VM_SOFTDIRTY); |
| 1317 | 1361 | } |
| 1318 | 1362 | |
| 1363 | +static inline bool pmd_needs_soft_dirty_wp(struct vm_area_struct *vma, pmd_t pmd) | |
| 1364 | +{ | |
| 1365 | + return vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd); | |
| 1366 | +} | |
| 1367 | + | |
| 1368 | +static inline bool pte_needs_soft_dirty_wp(struct vm_area_struct *vma, pte_t pte) | |
| 1369 | +{ | |
| 1370 | + return vma_soft_dirty_enabled(vma) && !pte_soft_dirty(pte); | |
| 1371 | +} | |
| 1372 | + | |
| 1319 | 1373 | static inline void vma_iter_config(struct vma_iterator *vmi, |
| 1320 | 1374 | unsigned long index, unsigned long last) |
| 1321 | 1375 | { |
| @@ -1515,4 +1569,13 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, · 193 unmodified lines | ||
| 1515 | 1569 | void workingset_update_node(struct xa_node *node); |
| 1516 | 1570 | extern struct list_lru shadow_nodes; |
| 1517 | 1571 | |
| 1572 | +struct unlink_vma_file_batch { | |
| 1573 | + int count; | |
| 1574 | + struct vm_area_struct *vmas[8]; | |
| 1575 | +}; | |
| 1576 | + | |
| 1577 | +void unlink_file_vma_batch_init(struct unlink_vma_file_batch *); | |
| 1578 | +void unlink_file_vma_batch_add(struct unlink_vma_file_batch *, struct vm_area_struct *); | |
| 1579 | +void unlink_file_vma_batch_final(struct unlink_vma_file_batch *); | |
| 1580 | + | |
| 1518 | 1581 | #endif /* __MM_INTERNAL_H */ |
| @@ -305,8 +305,14 @@ metadata_update_state(struct kfence_metadata *meta, enum kfence_object_state nex · 304 unmodified lines | ||
| 305 | 305 | WRITE_ONCE(meta->state, next); |
| 306 | 306 | } |
| 307 | 307 | |
| 308 | +#ifdef CONFIG_KMSAN | |
| 309 | +#define check_canary_attributes noinline __no_kmsan_checks | |
| 310 | +#else | |
| 311 | +#define check_canary_attributes inline | |
| 312 | +#endif | |
| 313 | + | |
| 308 | 314 | /* Check canary byte at @addr. */ |
| 309 | −static inline bool check_canary_byte(u8 *addr) | |
| 315 | +static check_canary_attributes bool check_canary_byte(u8 *addr) | |
| 310 | 316 | { |
| 311 | 317 | struct kfence_metadata *meta; |
| 312 | 318 | unsigned long flags; |
| @@ -341,7 +347,8 @@ static inline void set_canary(const struct kfence_metadata *meta) · 28 unmodified lines | ||
| 341 | 347 | *((u64 *)addr) = KFENCE_CANARY_PATTERN_U64; |
| 342 | 348 | } |
| 343 | 349 | |
| 344 | −static inline void check_canary(const struct kfence_metadata *meta) | |
| 350 | +static check_canary_attributes void | |
| 351 | +check_canary(const struct kfence_metadata *meta) | |
| 345 | 352 | { |
| 346 | 353 | const unsigned long pageaddr = ALIGN_DOWN(meta->addr, PAGE_SIZE); |
| 347 | 354 | unsigned long addr = pageaddr; |
| @@ -595,7 +602,7 @@ static unsigned long kfence_init_pool(void) · 247 unmodified lines | ||
| 595 | 602 | continue; |
| 596 | 603 | |
| 597 | 604 | __folio_set_slab(slab_folio(slab)); |
| 598 | −#ifdef CONFIG_MEMCG_KMEM | |
| 605 | +#ifdef CONFIG_MEMCG | |
| 599 | 606 | slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts | |
| 600 | 607 | MEMCG_DATA_OBJEXTS; |
| 601 | 608 | #endif |
| @@ -645,7 +652,7 @@ reset_slab: · 43 unmodified lines | ||
| 645 | 652 | |
| 646 | 653 | if (!i || (i % 2)) |
| 647 | 654 | continue; |
| 648 | −#ifdef CONFIG_MEMCG_KMEM | |
| 655 | +#ifdef CONFIG_MEMCG | |
| 649 | 656 | slab->obj_exts = 0; |
| 650 | 657 | #endif |
| 651 | 658 | __folio_clear_slab(slab_folio(slab)); |
| @@ -1139,7 +1146,7 @@ void __kfence_free(void *addr) · 487 unmodified lines | ||
| 1139 | 1146 | { |
| 1140 | 1147 | struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr); |
| 1141 | 1148 | |
| 1142 | −#ifdef CONFIG_MEMCG_KMEM | |
| 1149 | +#ifdef CONFIG_MEMCG | |
| 1143 | 1150 | KFENCE_WARN_ON(meta->obj_exts.objcg); |
| 1144 | 1151 | #endif |
| 1145 | 1152 | /* |
| @@ -97,7 +97,7 @@ struct kfence_metadata { · 96 unmodified lines | ||
| 97 | 97 | struct kfence_track free_track; |
| 98 | 98 | /* For updating alloc_covered on frees. */ |
| 99 | 99 | u32 alloc_stack_hash; |
| 100 | −#ifdef CONFIG_MEMCG_KMEM | |
| 100 | +#ifdef CONFIG_MEMCG | |
| 101 | 101 | struct slabobj_ext obj_exts; |
| 102 | 102 | #endif |
| 103 | 103 | }; |
| @@ -852,3 +852,4 @@ kunit_test_suites(&kfence_test_suite); · 851 unmodified lines | ||
| 852 | 852 | |
| 853 | 853 | MODULE_LICENSE("GPL v2"); |
| 854 | 854 | MODULE_AUTHOR("Alexander Potapenko <glider@google.com>, Marco Elver <elver@google.com>"); |
| 855 | +MODULE_DESCRIPTION("kfence unit test suite"); | |
| @@ -385,10 +385,7 @@ int hugepage_madvise(struct vm_area_struct *vma, · 384 unmodified lines | ||
| 385 | 385 | |
| 386 | 386 | int __init khugepaged_init(void) |
| 387 | 387 | { |
| 388 | − mm_slot_cache = kmem_cache_create("khugepaged_mm_slot", | |
| 389 | − sizeof(struct khugepaged_mm_slot), | |
| 390 | − __alignof__(struct khugepaged_mm_slot), | |
| 391 | − 0, NULL); | |
| 388 | + mm_slot_cache = KMEM_CACHE(khugepaged_mm_slot, 0); | |
| 392 | 389 | if (!mm_slot_cache) |
| 393 | 390 | return -ENOMEM; |
| 394 | 391 | |
| @@ -416,6 +413,26 @@ static inline int hpage_collapse_test_exit_or_disable(struct mm_struct *mm) · 21 unmodified lines | ||
| 416 | 413 | test_bit(MMF_DISABLE_THP, &mm->flags); |
| 417 | 414 | } |
| 418 | 415 | |
| 416 | +static bool hugepage_pmd_enabled(void) | |
| 417 | +{ | |
| 418 | + /* | |
| 419 | + * We cover both the anon and the file-backed case here; file-backed | |
| 420 | + * hugepages, when configured in, are determined by the global control. | |
| 421 | + * Anon pmd-sized hugepages are determined by the pmd-size control. | |
| 422 | + */ | |
| 423 | + if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && | |
| 424 | + hugepage_global_enabled()) | |
| 425 | + return true; | |
| 426 | + if (test_bit(PMD_ORDER, &huge_anon_orders_always)) | |
| 427 | + return true; | |
| 428 | + if (test_bit(PMD_ORDER, &huge_anon_orders_madvise)) | |
| 429 | + return true; | |
| 430 | + if (test_bit(PMD_ORDER, &huge_anon_orders_inherit) && | |
| 431 | + hugepage_global_enabled()) | |
| 432 | + return true; | |
| 433 | + return false; | |
| 434 | +} | |
| 435 | + | |
| 419 | 436 | void __khugepaged_enter(struct mm_struct *mm) |
| 420 | 437 | { |
| 421 | 438 | struct khugepaged_mm_slot *mm_slot; |
| @@ -452,7 +469,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, · 30 unmodified lines | ||
| 452 | 469 | unsigned long vm_flags) |
| 453 | 470 | { |
| 454 | 471 | if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && |
| 455 | − hugepage_flags_enabled()) { | |
| 472 | + hugepage_pmd_enabled()) { | |
| 456 | 473 | if (thp_vma_allowable_order(vma, vm_flags, TVA_ENFORCE_SYSFS, |
| 457 | 474 | PMD_ORDER)) |
| 458 | 475 | __khugepaged_enter(vma->vm_mm); |
| @@ -1213,7 +1230,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, · 754 unmodified lines | ||
| 1213 | 1230 | |
| 1214 | 1231 | spin_lock(pmd_ptl); |
| 1215 | 1232 | BUG_ON(!pmd_none(*pmd)); |
| 1216 | − folio_add_new_anon_rmap(folio, vma, address); | |
| 1233 | + folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE); | |
| 1217 | 1234 | folio_add_lru_vma(folio, vma); |
| 1218 | 1235 | pgtable_trans_huge_deposit(mm, pmd, pgtable); |
| 1219 | 1236 | set_pmd_at(mm, address, pmd, _pmd); |
| @@ -2465,8 +2482,7 @@ breakouterloop_mmap_lock: · 1245 unmodified lines | ||
| 2465 | 2482 | |
| 2466 | 2483 | static int khugepaged_has_work(void) |
| 2467 | 2484 | { |
| 2468 | − return !list_empty(&khugepaged_scan.mm_head) && | |
| 2469 | − hugepage_flags_enabled(); | |
| 2485 | + return !list_empty(&khugepaged_scan.mm_head) && hugepage_pmd_enabled(); | |
| 2470 | 2486 | } |
| 2471 | 2487 | |
| 2472 | 2488 | static int khugepaged_wait_event(void) |
| @@ -2539,7 +2555,7 @@ static void khugepaged_wait_work(void) · 66 unmodified lines | ||
| 2539 | 2555 | return; |
| 2540 | 2556 | } |
| 2541 | 2557 | |
| 2542 | − if (hugepage_flags_enabled()) | |
| 2558 | + if (hugepage_pmd_enabled()) | |
| 2543 | 2559 | wait_event_freezable(khugepaged_wait, khugepaged_wait_event()); |
| 2544 | 2560 | } |
| 2545 | 2561 | |
| @@ -2570,7 +2586,7 @@ static void set_recommended_min_free_kbytes(void) · 24 unmodified lines | ||
| 2570 | 2586 | int nr_zones = 0; |
| 2571 | 2587 | unsigned long recommended_min; |
| 2572 | 2588 | |
| 2573 | − if (!hugepage_flags_enabled()) { | |
| 2589 | + if (!hugepage_pmd_enabled()) { | |
| 2574 | 2590 | calculate_min_free_kbytes(); |
| 2575 | 2591 | goto update_wmarks; |
| 2576 | 2592 | } |
| @@ -2620,7 +2636,7 @@ int start_stop_khugepaged(void) · 43 unmodified lines | ||
| 2620 | 2636 | int err = 0; |
| 2621 | 2637 | |
| 2622 | 2638 | mutex_lock(&khugepaged_mutex); |
| 2623 | − if (hugepage_flags_enabled()) { | |
| 2639 | + if (hugepage_pmd_enabled()) { | |
| 2624 | 2640 | if (!khugepaged_thread) |
| 2625 | 2641 | khugepaged_thread = kthread_run(khugepaged, NULL, |
| 2626 | 2642 | "khugepaged"); |
| @@ -2646,7 +2662,7 @@ fail: · 19 unmodified lines | ||
| 2646 | 2662 | void khugepaged_min_free_kbytes_update(void) |
| 2647 | 2663 | { |
| 2648 | 2664 | mutex_lock(&khugepaged_mutex); |
| 2649 | − if (hugepage_flags_enabled() && khugepaged_thread) | |
| 2665 | + if (hugepage_pmd_enabled() && khugepaged_thread) | |
| 2650 | 2666 | set_recommended_min_free_kbytes(); |
| 2651 | 2667 | mutex_unlock(&khugepaged_mutex); |
| 2652 | 2668 | } |
| @@ -657,10 +657,10 @@ static struct kmemleak_object *__alloc_object(gfp_t gfp) · 656 unmodified lines | ||
| 657 | 657 | /* task information */ |
| 658 | 658 | if (in_hardirq()) { |
| 659 | 659 | object->pid = 0; |
| 660 | − strncpy(object->comm, "hardirq", sizeof(object->comm)); | |
| 660 | + strscpy(object->comm, "hardirq"); | |
| 661 | 661 | } else if (in_serving_softirq()) { |
| 662 | 662 | object->pid = 0; |
| 663 | − strncpy(object->comm, "softirq", sizeof(object->comm)); | |
| 663 | + strscpy(object->comm, "softirq"); | |
| 664 | 664 | } else { |
| 665 | 665 | object->pid = current->pid; |
| 666 | 666 | /* |
| @@ -669,7 +669,7 @@ static struct kmemleak_object *__alloc_object(gfp_t gfp) · 2 unmodified lines | ||
| 669 | 669 | * dependency issues with current->alloc_lock. In the worst |
| 670 | 670 | * case, the command line is not correct. |
| 671 | 671 | */ |
| 672 | − strncpy(object->comm, current->comm, sizeof(object->comm)); | |
| 672 | + strscpy(object->comm, current->comm); | |
| 673 | 673 | } |
| 674 | 674 | |
| 675 | 675 | /* kernel backtrace */ |
| @@ -43,7 +43,6 @@ void kmsan_internal_task_create(struct task_struct *task) · 42 unmodified lines | ||
| 43 | 43 | struct thread_info *info = current_thread_info(); |
| 44 | 44 | |
| 45 | 45 | __memset(ctx, 0, sizeof(*ctx)); |
| 46 | − ctx->allow_reporting = true; | |
| 47 | 46 | kmsan_internal_unpoison_memory(info, sizeof(*info), false); |
| 48 | 47 | } |
| 49 | 48 | |
| @@ -250,8 +249,8 @@ struct page *kmsan_vmalloc_to_page_or_null(void *vaddr) · 200 unmodified lines | ||
| 250 | 249 | return NULL; |
| 251 | 250 | } |
| 252 | 251 | |
| 253 | −void kmsan_internal_check_memory(void *addr, size_t size, const void *user_addr, | |
| 254 | − int reason) | |
| 252 | +void kmsan_internal_check_memory(void *addr, size_t size, | |
| 253 | + const void __user *user_addr, int reason) | |
| 255 | 254 | { |
| 256 | 255 | depot_stack_handle_t cur_origin = 0, new_origin = 0; |
| 257 | 256 | unsigned long addr64 = (unsigned long)addr; |
| @@ -39,12 +39,10 @@ void kmsan_task_create(struct task_struct *task) · 38 unmodified lines | ||
| 39 | 39 | |
| 40 | 40 | void kmsan_task_exit(struct task_struct *task) |
| 41 | 41 | { |
| 42 | − struct kmsan_ctx *ctx = &task->kmsan_ctx; | |
| 43 | − | |
| 44 | 42 | if (!kmsan_enabled || kmsan_in_runtime()) |
| 45 | 43 | return; |
| 46 | 44 | |
| 47 | − ctx->allow_reporting = false; | |
| 45 | + kmsan_disable_current(); | |
| 48 | 46 | } |
| 49 | 47 | |
| 50 | 48 | void kmsan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags) |
| @@ -76,7 +74,7 @@ void kmsan_slab_free(struct kmem_cache *s, void *object) · 25 unmodified lines | ||
| 76 | 74 | return; |
| 77 | 75 | |
| 78 | 76 | /* RCU slabs could be legally used after free within the RCU period */ |
| 79 | − if (unlikely(s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON))) | |
| 77 | + if (unlikely(s->flags & SLAB_TYPESAFE_BY_RCU)) | |
| 80 | 78 | return; |
| 81 | 79 | /* |
| 82 | 80 | * If there's a constructor, freed memory must remain in the same state |
| @@ -267,7 +265,8 @@ void kmsan_copy_to_user(void __user *to, const void *from, size_t to_copy, · 184 unmodified lines | ||
| 267 | 265 | return; |
| 268 | 266 | |
| 269 | 267 | ua_flags = user_access_save(); |
| 270 | − if ((u64)to < TASK_SIZE) { | |
| 268 | + if (!IS_ENABLED(CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE) || | |
| 269 | + (u64)to < TASK_SIZE) { | |
| 271 | 270 | /* This is a user memory access, check it. */ |
| 272 | 271 | kmsan_internal_check_memory((void *)from, to_copy - left, to, |
| 273 | 272 | REASON_COPY_TO_USER); |
| @@ -304,7 +303,8 @@ void kmsan_handle_urb(const struct urb *urb, bool is_out) · 30 unmodified lines | ||
| 304 | 303 | if (is_out) |
| 305 | 304 | kmsan_internal_check_memory(urb->transfer_buffer, |
| 306 | 305 | urb->transfer_buffer_length, |
| 307 | − /*user_addr*/ 0, REASON_SUBMIT_URB); | |
| 306 | + /*user_addr*/ NULL, | |
| 307 | + REASON_SUBMIT_URB); | |
| 308 | 308 | else |
| 309 | 309 | kmsan_internal_unpoison_memory(urb->transfer_buffer, |
| 310 | 310 | urb->transfer_buffer_length, |
| @@ -317,14 +317,14 @@ static void kmsan_handle_dma_page(const void *addr, size_t size, · 6 unmodified lines | ||
| 317 | 317 | { |
| 318 | 318 | switch (dir) { |
| 319 | 319 | case DMA_BIDIRECTIONAL: |
| 320 | − kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0, | |
| 321 | − REASON_ANY); | |
| 320 | + kmsan_internal_check_memory((void *)addr, size, | |
| 321 | + /*user_addr*/ NULL, REASON_ANY); | |
| 322 | 322 | kmsan_internal_unpoison_memory((void *)addr, size, |
| 323 | 323 | /*checked*/ false); |
| 324 | 324 | break; |
| 325 | 325 | case DMA_TO_DEVICE: |
| 326 | − kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0, | |
| 327 | − REASON_ANY); | |
| 326 | + kmsan_internal_check_memory((void *)addr, size, | |
| 327 | + /*user_addr*/ NULL, REASON_ANY); | |
| 328 | 328 | break; |
| 329 | 329 | case DMA_FROM_DEVICE: |
| 330 | 330 | kmsan_internal_unpoison_memory((void *)addr, size, |
| @@ -419,7 +419,21 @@ void kmsan_check_memory(const void *addr, size_t size) · 88 unmodified lines | ||
| 419 | 419 | { |
| 420 | 420 | if (!kmsan_enabled) |
| 421 | 421 | return; |
| 422 | − return kmsan_internal_check_memory((void *)addr, size, /*user_addr*/ 0, | |
| 423 | − REASON_ANY); | |
| 422 | + return kmsan_internal_check_memory((void *)addr, size, | |
| 423 | + /*user_addr*/ NULL, REASON_ANY); | |
| 424 | 424 | } |
| 425 | 425 | EXPORT_SYMBOL(kmsan_check_memory); |
| 426 | + | |
| 427 | +void kmsan_enable_current(void) | |
| 428 | +{ | |
| 429 | + KMSAN_WARN_ON(current->kmsan_ctx.depth == 0); | |
| 430 | + current->kmsan_ctx.depth--; | |
| 431 | +} | |
| 432 | +EXPORT_SYMBOL(kmsan_enable_current); | |
| 433 | + | |
| 434 | +void kmsan_disable_current(void) | |
| 435 | +{ | |
| 436 | + current->kmsan_ctx.depth++; | |
| 437 | + KMSAN_WARN_ON(current->kmsan_ctx.depth == 0); | |
| 438 | +} | |
| 439 | +EXPORT_SYMBOL(kmsan_disable_current); | |
| @@ -33,7 +33,10 @@ static void __init kmsan_record_future_shadow_range(void *start, void *end) · 32 unmodified lines | ||
| 33 | 33 | bool merged = false; |
| 34 | 34 | |
| 35 | 35 | KMSAN_WARN_ON(future_index == NUM_FUTURE_RANGES); |
| 36 | − KMSAN_WARN_ON((nstart >= nend) || !nstart || !nend); | |
| 36 | + KMSAN_WARN_ON((nstart >= nend) || | |
| 37 | + /* Virtual address 0 is valid on s390. */ | |
| 38 | + (!IS_ENABLED(CONFIG_S390) && !nstart) || | |
| 39 | + !nend); | |
| 37 | 40 | nstart = ALIGN_DOWN(nstart, PAGE_SIZE); |
| 38 | 41 | nend = ALIGN(nend, PAGE_SIZE); |
| 39 | 42 | |
| @@ -72,7 +75,7 @@ static void __init kmsan_record_future_shadow_range(void *start, void *end) · 32 unmodified lines | ||
| 72 | 75 | */ |
| 73 | 76 | void __init kmsan_init_shadow(void) |
| 74 | 77 | { |
| 75 | − const size_t nd_size = roundup(sizeof(pg_data_t), PAGE_SIZE); | |
| 78 | + const size_t nd_size = sizeof(pg_data_t); | |
| 76 | 79 | phys_addr_t p_start, p_end; |
| 77 | 80 | u64 loop; |
| 78 | 81 | int nid; |
| @@ -172,7 +175,7 @@ static void do_collection(void) · 93 unmodified lines | ||
| 172 | 175 | shadow = smallstack_pop(&collect); |
| 173 | 176 | origin = smallstack_pop(&collect); |
| 174 | 177 | kmsan_setup_meta(page, shadow, origin, collect.order); |
| 175 | − __free_pages_core(page, collect.order); | |
| 178 | + __free_pages_core(page, collect.order, MEMINIT_EARLY); | |
| 176 | 179 | } |
| 177 | 180 | } |
| 178 | 181 | |
| @@ -14,13 +14,15 @@ · 13 unmodified lines | ||
| 14 | 14 | |
| 15 | 15 | #include "kmsan.h" |
| 16 | 16 | #include <linux/gfp.h> |
| 17 | +#include <linux/kmsan.h> | |
| 17 | 18 | #include <linux/kmsan_string.h> |
| 18 | 19 | #include <linux/mm.h> |
| 19 | 20 | #include <linux/uaccess.h> |
| 20 | 21 | |
| 21 | 22 | static inline bool is_bad_asm_addr(void *addr, uintptr_t size, bool is_store) |
| 22 | 23 | { |
| 23 | − if ((u64)addr < TASK_SIZE) | |
| 24 | + if (IS_ENABLED(CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE) && | |
| 25 | + (u64)addr < TASK_SIZE) | |
| 24 | 26 | return true; |
| 25 | 27 | if (!kmsan_get_metadata(addr, KMSAN_META_SHADOW)) |
| 26 | 28 | return true; |
| @@ -110,11 +112,10 @@ void __msan_instrument_asm_store(void *addr, uintptr_t size) · 83 unmodified lines | ||
| 110 | 112 | |
| 111 | 113 | ua_flags = user_access_save(); |
| 112 | 114 | /* |
| 113 | − * Most of the accesses are below 32 bytes. The two exceptions so far | |
| 114 | − * are clwb() (64 bytes) and FPU state (512 bytes). | |
| 115 | − * It's unlikely that the assembly will touch more than 512 bytes. | |
| 115 | + * Most of the accesses are below 32 bytes. The exceptions so far are | |
| 116 | + * clwb() (64 bytes), FPU state (512 bytes) and chsc() (4096 bytes). | |
| 116 | 117 | */ |
| 117 | − if (size > 512) { | |
| 118 | + if (size > 4096) { | |
| 118 | 119 | WARN_ONCE(1, "assembly store size too big: %ld\n", size); |
| 119 | 120 | size = 8; |
| 120 | 121 | } |
| @@ -314,8 +315,8 @@ void __msan_warning(u32 origin) · 193 unmodified lines | ||
| 314 | 315 | if (!kmsan_enabled || kmsan_in_runtime()) |
| 315 | 316 | return; |
| 316 | 317 | kmsan_enter_runtime(); |
| 317 | − kmsan_report(origin, /*address*/ 0, /*size*/ 0, | |
| 318 | − /*off_first*/ 0, /*off_last*/ 0, /*user_addr*/ 0, | |
| 318 | + kmsan_report(origin, /*address*/ NULL, /*size*/ 0, | |
| 319 | + /*off_first*/ 0, /*off_last*/ 0, /*user_addr*/ NULL, | |
| 319 | 320 | REASON_ANY); |
| 320 | 321 | kmsan_leave_runtime(); |
| 321 | 322 | } |
| @@ -10,14 +10,15 @@ · 9 unmodified lines | ||
| 10 | 10 | #ifndef __MM_KMSAN_KMSAN_H |
| 11 | 11 | #define __MM_KMSAN_KMSAN_H |
| 12 | 12 | |
| 13 | −#include <asm/pgtable_64_types.h> | |
| 14 | 13 | #include <linux/irqflags.h> |
| 14 | +#include <linux/kmsan.h> | |
| 15 | +#include <linux/mm.h> | |
| 16 | +#include <linux/nmi.h> | |
| 17 | +#include <linux/pgtable.h> | |
| 18 | +#include <linux/printk.h> | |
| 15 | 19 | #include <linux/sched.h> |
| 16 | 20 | #include <linux/stackdepot.h> |
| 17 | 21 | #include <linux/stacktrace.h> |
| 18 | −#include <linux/nmi.h> | |
| 19 | −#include <linux/mm.h> | |
| 20 | −#include <linux/printk.h> | |
| 21 | 22 | |
| 22 | 23 | #define KMSAN_ALLOCA_MAGIC_ORIGIN 0xabcd0100 |
| 23 | 24 | #define KMSAN_CHAIN_MAGIC_ORIGIN 0xabcd0200 |
| @@ -34,29 +35,6 @@ · 10 unmodified lines | ||
| 34 | 35 | #define KMSAN_META_SHADOW (false) |
| 35 | 36 | #define KMSAN_META_ORIGIN (true) |
| 36 | 37 | |
| 37 | −extern bool kmsan_enabled; | |
| 38 | −extern int panic_on_kmsan; | |
| 39 | − | |
| 40 | −/* | |
| 41 | − * KMSAN performs a lot of consistency checks that are currently enabled by | |
| 42 | − * default. BUG_ON is normally discouraged in the kernel, unless used for | |
| 43 | − * debugging, but KMSAN itself is a debugging tool, so it makes little sense to | |
| 44 | − * recover if something goes wrong. | |
| 45 | − */ | |
| 46 | −#define KMSAN_WARN_ON(cond) \ | |
| 47 | − ({ \ | |
| 48 | − const bool __cond = WARN_ON(cond); \ | |
| 49 | − if (unlikely(__cond)) { \ | |
| 50 | − WRITE_ONCE(kmsan_enabled, false); \ | |
| 51 | − if (panic_on_kmsan) { \ | |
| 52 | − /* Can't call panic() here because */ \ | |
| 53 | − /* of uaccess checks. */ \ | |
| 54 | − BUG(); \ | |
| 55 | − } \ | |
| 56 | − } \ | |
| 57 | − __cond; \ | |
| 58 | − }) | |
| 59 | − | |
| 60 | 38 | /* |
| 61 | 39 | * A pair of metadata pointers to be returned by the instrumentation functions. |
| 62 | 40 | */ |
| @@ -66,7 +44,6 @@ struct shadow_origin_ptr { · 3 unmodified lines | ||
| 66 | 44 | |
| 67 | 45 | struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *addr, u64 size, |
| 68 | 46 | bool store); |
| 69 | −void *kmsan_get_metadata(void *addr, bool is_origin); | |
| 70 | 47 | void __init kmsan_init_alloc_meta_for_range(void *start, void *end); |
| 71 | 48 | |
| 72 | 49 | enum kmsan_bug_reason { |
| @@ -96,7 +73,7 @@ void kmsan_print_origin(depot_stack_handle_t origin); · 23 unmodified lines | ||
| 96 | 73 | * @off_last corresponding to different @origin values. |
| 97 | 74 | */ |
| 98 | 75 | void kmsan_report(depot_stack_handle_t origin, void *address, int size, |
| 99 | − int off_first, int off_last, const void *user_addr, | |
| 76 | + int off_first, int off_last, const void __user *user_addr, | |
| 100 | 77 | enum kmsan_bug_reason reason); |
| 101 | 78 | |
| 102 | 79 | DECLARE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); |
| @@ -186,8 +163,8 @@ depot_stack_handle_t kmsan_internal_chain_origin(depot_stack_handle_t id); · 83 unmodified lines | ||
| 186 | 163 | void kmsan_internal_task_create(struct task_struct *task); |
| 187 | 164 | |
| 188 | 165 | bool kmsan_metadata_is_contiguous(void *addr, size_t size); |
| 189 | −void kmsan_internal_check_memory(void *addr, size_t size, const void *user_addr, | |
| 190 | − int reason); | |
| 166 | +void kmsan_internal_check_memory(void *addr, size_t size, | |
| 167 | + const void __user *user_addr, int reason); | |
| 191 | 168 | |
| 192 | 169 | struct page *kmsan_vmalloc_to_page_or_null(void *vaddr); |
| 193 | 170 | void kmsan_setup_meta(struct page *page, struct page *shadow, |
| @@ -614,6 +614,32 @@ static void test_stackdepot_roundtrip(struct kunit *test) · 613 unmodified lines | ||
| 614 | 614 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); |
| 615 | 615 | } |
| 616 | 616 | |
| 617 | +/* | |
| 618 | + * Test case: ensure that kmsan_unpoison_memory() and the instrumentation work | |
| 619 | + * the same. | |
| 620 | + */ | |
| 621 | +static void test_unpoison_memory(struct kunit *test) | |
| 622 | +{ | |
| 623 | + EXPECTATION_UNINIT_VALUE_FN(expect, "test_unpoison_memory"); | |
| 624 | + volatile char a[4], b[4]; | |
| 625 | + | |
| 626 | + kunit_info( | |
| 627 | + test, | |
| 628 | + "unpoisoning via the instrumentation vs. kmsan_unpoison_memory() (2 UMR reports)\n"); | |
| 629 | + | |
| 630 | + /* Initialize a[0] and check a[1]--a[3]. */ | |
| 631 | + a[0] = 0; | |
| 632 | + kmsan_check_memory((char *)&a[1], 3); | |
| 633 | + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); | |
| 634 | + | |
| 635 | + report_reset(); | |
| 636 | + | |
| 637 | + /* Initialize b[0] and check b[1]--b[3]. */ | |
| 638 | + kmsan_unpoison_memory((char *)&b[0], 1); | |
| 639 | + kmsan_check_memory((char *)&b[1], 3); | |
| 640 | + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); | |
| 641 | +} | |
| 642 | + | |
| 617 | 643 | static struct kunit_case kmsan_test_cases[] = { |
| 618 | 644 | KUNIT_CASE(test_uninit_kmalloc), |
| 619 | 645 | KUNIT_CASE(test_init_kmalloc), |
| @@ -637,6 +663,7 @@ static struct kunit_case kmsan_test_cases[] = { · 17 unmodified lines | ||
| 637 | 663 | KUNIT_CASE(test_memset64), |
| 638 | 664 | KUNIT_CASE(test_long_origin_chain), |
| 639 | 665 | KUNIT_CASE(test_stackdepot_roundtrip), |
| 666 | + KUNIT_CASE(test_unpoison_memory), | |
| 640 | 667 | {}, |
| 641 | 668 | }; |
| 642 | 669 | |
| @@ -659,9 +686,13 @@ static void test_exit(struct kunit *test) · 16 unmodified lines | ||
| 659 | 686 | { |
| 660 | 687 | } |
| 661 | 688 | |
| 689 | +static int orig_panic_on_kmsan; | |
| 690 | + | |
| 662 | 691 | static int kmsan_suite_init(struct kunit_suite *suite) |
| 663 | 692 | { |
| 664 | 693 | register_trace_console(probe_console, NULL); |
| 694 | + orig_panic_on_kmsan = panic_on_kmsan; | |
| 695 | + panic_on_kmsan = 0; | |
| 665 | 696 | return 0; |
| 666 | 697 | } |
| 667 | 698 | |
| @@ -669,6 +700,7 @@ static void kmsan_suite_exit(struct kunit_suite *suite) · 1 unmodified line | ||
| 669 | 700 | { |
| 670 | 701 | unregister_trace_console(probe_console, NULL); |
| 671 | 702 | tracepoint_synchronize_unregister(); |
| 703 | + panic_on_kmsan = orig_panic_on_kmsan; | |
| 672 | 704 | } |
| 673 | 705 | |
| 674 | 706 | static struct kunit_suite kmsan_test_suite = { |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | */ |
| 9 | 9 | |
| 10 | 10 | #include <linux/console.h> |
| 11 | +#include <linux/kmsan.h> | |
| 11 | 12 | #include <linux/moduleparam.h> |
| 12 | 13 | #include <linux/stackdepot.h> |
| 13 | 14 | #include <linux/stacktrace.h> |
| @@ -20,6 +21,7 @@ static DEFINE_RAW_SPINLOCK(kmsan_report_lock); · 6 unmodified lines | ||
| 20 | 21 | /* Protected by kmsan_report_lock */ |
| 21 | 22 | static char report_local_descr[DESCR_SIZE]; |
| 22 | 23 | int panic_on_kmsan __read_mostly; |
| 24 | +EXPORT_SYMBOL_GPL(panic_on_kmsan); | |
| 23 | 25 | |
| 24 | 26 | #ifdef MODULE_PARAM_PREFIX |
| 25 | 27 | #undef MODULE_PARAM_PREFIX |
| @@ -146,7 +148,7 @@ void kmsan_print_origin(depot_stack_handle_t origin) · 120 unmodified lines | ||
| 146 | 148 | } |
| 147 | 149 | |
| 148 | 150 | void kmsan_report(depot_stack_handle_t origin, void *address, int size, |
| 149 | − int off_first, int off_last, const void *user_addr, | |
| 151 | + int off_first, int off_last, const void __user *user_addr, | |
| 150 | 152 | enum kmsan_bug_reason reason) |
| 151 | 153 | { |
| 152 | 154 | unsigned long stack_entries[KMSAN_STACK_DEPTH]; |
| @@ -157,12 +159,12 @@ void kmsan_report(depot_stack_handle_t origin, void *address, int size, · 4 unmodified lines | ||
| 157 | 159 | |
| 158 | 160 | if (!kmsan_enabled) |
| 159 | 161 | return; |
| 160 | − if (!current->kmsan_ctx.allow_reporting) | |
| 162 | + if (current->kmsan_ctx.depth) | |
| 161 | 163 | return; |
| 162 | 164 | if (!origin) |
| 163 | 165 | return; |
| 164 | 166 | |
| 165 | − current->kmsan_ctx.allow_reporting = false; | |
| 167 | + kmsan_disable_current(); | |
| 166 | 168 | ua_flags = user_access_save(); |
| 167 | 169 | raw_spin_lock(&kmsan_report_lock); |
| 168 | 170 | pr_err("=====================================================\n"); |
| @@ -215,5 +217,5 @@ void kmsan_report(depot_stack_handle_t origin, void *address, int size, · 46 unmodified lines | ||
| 215 | 217 | if (panic_on_kmsan) |
| 216 | 218 | panic("kmsan.panic set ...\n"); |
| 217 | 219 | user_access_restore(ua_flags); |
| 218 | − current->kmsan_ctx.allow_reporting = true; | |
| 220 | + kmsan_enable_current(); | |
| 219 | 221 | } |
| @@ -123,14 +123,12 @@ return_dummy: · 122 unmodified lines | ||
| 123 | 123 | */ |
| 124 | 124 | void *kmsan_get_metadata(void *address, bool is_origin) |
| 125 | 125 | { |
| 126 | − u64 addr = (u64)address, pad, off; | |
| 126 | + u64 addr = (u64)address, off; | |
| 127 | 127 | struct page *page; |
| 128 | 128 | void *ret; |
| 129 | 129 | |
| 130 | − if (is_origin && !IS_ALIGNED(addr, KMSAN_ORIGIN_SIZE)) { | |
| 131 | − pad = addr % KMSAN_ORIGIN_SIZE; | |
| 132 | − addr -= pad; | |
| 133 | − } | |
| 130 | + if (is_origin) | |
| 131 | + addr = ALIGN_DOWN(addr, KMSAN_ORIGIN_SIZE); | |
| 134 | 132 | address = (void *)addr; |
| 135 | 133 | if (kmsan_internal_is_vmalloc_addr(address) || |
| 136 | 134 | kmsan_internal_is_module_addr(address)) |
| @@ -243,7 +241,6 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, · 106 unmodified lines | ||
| 243 | 241 | s_pages[i] = shadow_page_for(pages[i]); |
| 244 | 242 | o_pages[i] = origin_page_for(pages[i]); |
| 245 | 243 | } |
| 246 | − prot = __pgprot(pgprot_val(prot) | _PAGE_NX); | |
| 247 | 244 | prot = PAGE_KERNEL; |
| 248 | 245 | |
| 249 | 246 | origin_start = vmalloc_meta((void *)start, KMSAN_META_ORIGIN); |
| @@ -488,21 +488,17 @@ static DECLARE_WAIT_QUEUE_HEAD(ksm_iter_wait); · 487 unmodified lines | ||
| 488 | 488 | static DEFINE_MUTEX(ksm_thread_mutex); |
| 489 | 489 | static DEFINE_SPINLOCK(ksm_mmlist_lock); |
| 490 | 490 | |
| 491 | −#define KSM_KMEM_CACHE(__struct, __flags) kmem_cache_create(#__struct,\ | |
| 492 | − sizeof(struct __struct), __alignof__(struct __struct),\ | |
| 493 | − (__flags), NULL) | |
| 494 | − | |
| 495 | 491 | static int __init ksm_slab_init(void) |
| 496 | 492 | { |
| 497 | − rmap_item_cache = KSM_KMEM_CACHE(ksm_rmap_item, 0); | |
| 493 | + rmap_item_cache = KMEM_CACHE(ksm_rmap_item, 0); | |
| 498 | 494 | if (!rmap_item_cache) |
| 499 | 495 | goto out; |
| 500 | 496 | |
| 501 | − stable_node_cache = KSM_KMEM_CACHE(ksm_stable_node, 0); | |
| 497 | + stable_node_cache = KMEM_CACHE(ksm_stable_node, 0); | |
| 502 | 498 | if (!stable_node_cache) |
| 503 | 499 | goto out_free1; |
| 504 | 500 | |
| 505 | − mm_slot_cache = KSM_KMEM_CACHE(ksm_mm_slot, 0); | |
| 501 | + mm_slot_cache = KMEM_CACHE(ksm_mm_slot, 0); | |
| 506 | 502 | if (!mm_slot_cache) |
| 507 | 503 | goto out_free2; |
| 508 | 504 | |
| @@ -1531,6 +1527,44 @@ out: · 1022 unmodified lines | ||
| 1531 | 1527 | return err; |
| 1532 | 1528 | } |
| 1533 | 1529 | |
| 1530 | +/* | |
| 1531 | + * This function returns 0 if the pages were merged or if they are | |
| 1532 | + * no longer merging candidates (e.g., VMA stale), -EFAULT otherwise. | |
| 1533 | + */ | |
| 1534 | +static int try_to_merge_with_zero_page(struct ksm_rmap_item *rmap_item, | |
| 1535 | + struct page *page) | |
| 1536 | +{ | |
| 1537 | + struct mm_struct *mm = rmap_item->mm; | |
| 1538 | + int err = -EFAULT; | |
| 1539 | + | |
| 1540 | + /* | |
| 1541 | + * Same checksum as an empty page. We attempt to merge it with the | |
| 1542 | + * appropriate zero page if the user enabled this via sysfs. | |
| 1543 | + */ | |
| 1544 | + if (ksm_use_zero_pages && (rmap_item->oldchecksum == zero_checksum)) { | |
| 1545 | + struct vm_area_struct *vma; | |
| 1546 | + | |
| 1547 | + mmap_read_lock(mm); | |
| 1548 | + vma = find_mergeable_vma(mm, rmap_item->address); | |
| 1549 | + if (vma) { | |
| 1550 | + err = try_to_merge_one_page(vma, page, | |
| 1551 | + ZERO_PAGE(rmap_item->address)); | |
| 1552 | + trace_ksm_merge_one_page( | |
| 1553 | + page_to_pfn(ZERO_PAGE(rmap_item->address)), | |
| 1554 | + rmap_item, mm, err); | |
| 1555 | + } else { | |
| 1556 | + /* | |
| 1557 | + * If the vma is out of date, we do not need to | |
| 1558 | + * continue. | |
| 1559 | + */ | |
| 1560 | + err = 0; | |
| 1561 | + } | |
| 1562 | + mmap_read_unlock(mm); | |
| 1563 | + } | |
| 1564 | + | |
| 1565 | + return err; | |
| 1566 | +} | |
| 1567 | + | |
| 1534 | 1568 | /* |
| 1535 | 1569 | * try_to_merge_with_ksm_page - like try_to_merge_two_pages, |
| 1536 | 1570 | * but no new kernel page is allocated: kpage must already be a ksm page. |
| @@ -1625,7 +1659,6 @@ static struct folio *stable_node_dup(struct ksm_stable_node **_stable_node_dup, · 88 unmodified lines | ||
| 1625 | 1659 | struct ksm_stable_node *dup, *found = NULL, *stable_node = *_stable_node; |
| 1626 | 1660 | struct hlist_node *hlist_safe; |
| 1627 | 1661 | struct folio *folio, *tree_folio = NULL; |
| 1628 | − int nr = 0; | |
| 1629 | 1662 | int found_rmap_hlist_len; |
| 1630 | 1663 | |
| 1631 | 1664 | if (!prune_stale_stable_nodes || |
| @@ -1652,33 +1685,26 @@ static struct folio *stable_node_dup(struct ksm_stable_node **_stable_node_dup, · 20 unmodified lines | ||
| 1652 | 1685 | folio = ksm_get_folio(dup, KSM_GET_FOLIO_NOLOCK); |
| 1653 | 1686 | if (!folio) |
| 1654 | 1687 | continue; |
| 1655 | − nr += 1; | |
| 1656 | − if (is_page_sharing_candidate(dup)) { | |
| 1657 | − if (!found || | |
| 1658 | − dup->rmap_hlist_len > found_rmap_hlist_len) { | |
| 1659 | − if (found) | |
| 1660 | − folio_put(tree_folio); | |
| 1661 | − found = dup; | |
| 1662 | − found_rmap_hlist_len = found->rmap_hlist_len; | |
| 1663 | − tree_folio = folio; | |
| 1664 | − | |
| 1665 | − /* skip put_page for found dup */ | |
| 1666 | − if (!prune_stale_stable_nodes) | |
| 1667 | − break; | |
| 1668 | − continue; | |
| 1669 | − } | |
| 1688 | + /* Pick the best candidate if possible. */ | |
| 1689 | + if (!found || (is_page_sharing_candidate(dup) && | |
| 1690 | + (!is_page_sharing_candidate(found) || | |
| 1691 | + dup->rmap_hlist_len > found_rmap_hlist_len))) { | |
| 1692 | + if (found) | |
| 1693 | + folio_put(tree_folio); | |
| 1694 | + found = dup; | |
| 1695 | + found_rmap_hlist_len = found->rmap_hlist_len; | |
| 1696 | + tree_folio = folio; | |
| 1697 | + /* skip put_page for found candidate */ | |
| 1698 | + if (!prune_stale_stable_nodes && | |
| 1699 | + is_page_sharing_candidate(found)) | |
| 1700 | + break; | |
| 1701 | + continue; | |
| 1670 | 1702 | } |
| 1671 | 1703 | folio_put(folio); |
| 1672 | 1704 | } |
| 1673 | 1705 | |
| 1674 | 1706 | if (found) { |
| 1675 | − /* | |
| 1676 | − * nr is counting all dups in the chain only if | |
| 1677 | − * prune_stale_stable_nodes is true, otherwise we may | |
| 1678 | − * break the loop at nr == 1 even if there are | |
| 1679 | − * multiple entries. | |
| 1680 | − */ | |
| 1681 | − if (prune_stale_stable_nodes && nr == 1) { | |
| 1707 | + if (hlist_is_singular_node(&found->hlist_dup, &stable_node->hlist)) { | |
| 1682 | 1708 | /* |
| 1683 | 1709 | * If there's not just one entry it would |
| 1684 | 1710 | * corrupt memory, better BUG_ON. In KSM |
| @@ -1730,25 +1756,15 @@ static struct folio *stable_node_dup(struct ksm_stable_node **_stable_node_dup, · 45 unmodified lines | ||
| 1730 | 1756 | hlist_add_head(&found->hlist_dup, |
| 1731 | 1757 | &stable_node->hlist); |
| 1732 | 1758 | } |
| 1759 | + } else { | |
| 1760 | + /* Its hlist must be empty if no one found. */ | |
| 1761 | + free_stable_node_chain(stable_node, root); | |
| 1733 | 1762 | } |
| 1734 | 1763 | |
| 1735 | 1764 | *_stable_node_dup = found; |
| 1736 | 1765 | return tree_folio; |
| 1737 | 1766 | } |
| 1738 | 1767 | |
| 1739 | −static struct ksm_stable_node *stable_node_dup_any(struct ksm_stable_node *stable_node, | |
| 1740 | − struct rb_root *root) | |
| 1741 | −{ | |
| 1742 | − if (!is_stable_node_chain(stable_node)) | |
| 1743 | − return stable_node; | |
| 1744 | − if (hlist_empty(&stable_node->hlist)) { | |
| 1745 | − free_stable_node_chain(stable_node, root); | |
| 1746 | − return NULL; | |
| 1747 | − } | |
| 1748 | − return hlist_entry(stable_node->hlist.first, | |
| 1749 | − typeof(*stable_node), hlist_dup); | |
| 1750 | −} | |
| 1751 | − | |
| 1752 | 1768 | /* |
| 1753 | 1769 | * Like for ksm_get_folio, this function can free the *_stable_node and |
| 1754 | 1770 | * *_stable_node_dup if the returned tree_page is NULL. |
| @@ -1769,17 +1785,10 @@ static struct folio *__stable_node_chain(struct ksm_stable_node **_stable_node_d · 14 unmodified lines | ||
| 1769 | 1785 | bool prune_stale_stable_nodes) |
| 1770 | 1786 | { |
| 1771 | 1787 | struct ksm_stable_node *stable_node = *_stable_node; |
| 1788 | + | |
| 1772 | 1789 | if (!is_stable_node_chain(stable_node)) { |
| 1773 | − if (is_page_sharing_candidate(stable_node)) { | |
| 1774 | − *_stable_node_dup = stable_node; | |
| 1775 | − return ksm_get_folio(stable_node, KSM_GET_FOLIO_NOLOCK); | |
| 1776 | − } | |
| 1777 | − /* | |
| 1778 | − * _stable_node_dup set to NULL means the stable_node | |
| 1779 | − * reached the ksm_max_page_sharing limit. | |
| 1780 | − */ | |
| 1781 | − *_stable_node_dup = NULL; | |
| 1782 | − return NULL; | |
| 1790 | + *_stable_node_dup = stable_node; | |
| 1791 | + return ksm_get_folio(stable_node, KSM_GET_FOLIO_NOLOCK); | |
| 1783 | 1792 | } |
| 1784 | 1793 | return stable_node_dup(_stable_node_dup, _stable_node, root, |
| 1785 | 1794 | prune_stale_stable_nodes); |
| @@ -1793,16 +1802,10 @@ static __always_inline struct folio *chain_prune(struct ksm_stable_node **s_n_d, · 7 unmodified lines | ||
| 1793 | 1802 | } |
| 1794 | 1803 | |
| 1795 | 1804 | static __always_inline struct folio *chain(struct ksm_stable_node **s_n_d, |
| 1796 | − struct ksm_stable_node *s_n, | |
| 1805 | + struct ksm_stable_node **s_n, | |
| 1797 | 1806 | struct rb_root *root) |
| 1798 | 1807 | { |
| 1799 | − struct ksm_stable_node *old_stable_node = s_n; | |
| 1800 | − struct folio *tree_folio; | |
| 1801 | − | |
| 1802 | − tree_folio = __stable_node_chain(s_n_d, &s_n, root, false); | |
| 1803 | − /* not pruning dups so s_n cannot have changed */ | |
| 1804 | − VM_BUG_ON(s_n != old_stable_node); | |
| 1805 | − return tree_folio; | |
| 1808 | + return __stable_node_chain(s_n_d, s_n, root, false); | |
| 1806 | 1809 | } |
| 1807 | 1810 | |
| 1808 | 1811 | /* |
| @@ -1820,7 +1823,7 @@ static struct page *stable_tree_search(struct page *page) · 11 unmodified lines | ||
| 1820 | 1823 | struct rb_root *root; |
| 1821 | 1824 | struct rb_node **new; |
| 1822 | 1825 | struct rb_node *parent; |
| 1823 | − struct ksm_stable_node *stable_node, *stable_node_dup, *stable_node_any; | |
| 1826 | + struct ksm_stable_node *stable_node, *stable_node_dup; | |
| 1824 | 1827 | struct ksm_stable_node *page_node; |
| 1825 | 1828 | struct folio *folio; |
| 1826 | 1829 | |
| @@ -1844,45 +1847,7 @@ again: · 17 unmodified lines | ||
| 1844 | 1847 | |
| 1845 | 1848 | cond_resched(); |
| 1846 | 1849 | stable_node = rb_entry(*new, struct ksm_stable_node, node); |
| 1847 | − stable_node_any = NULL; | |
| 1848 | 1850 | tree_folio = chain_prune(&stable_node_dup, &stable_node, root); |
| 1849 | − /* | |
| 1850 | − * NOTE: stable_node may have been freed by | |
| 1851 | − * chain_prune() if the returned stable_node_dup is | |
| 1852 | − * not NULL. stable_node_dup may have been inserted in | |
| 1853 | − * the rbtree instead as a regular stable_node (in | |
| 1854 | − * order to collapse the stable_node chain if a single | |
| 1855 | − * stable_node dup was found in it). In such case the | |
| 1856 | − * stable_node is overwritten by the callee to point | |
| 1857 | − * to the stable_node_dup that was collapsed in the | |
| 1858 | − * stable rbtree and stable_node will be equal to | |
| 1859 | − * stable_node_dup like if the chain never existed. | |
| 1860 | − */ | |
| 1861 | − if (!stable_node_dup) { | |
| 1862 | − /* | |
| 1863 | − * Either all stable_node dups were full in | |
| 1864 | − * this stable_node chain, or this chain was | |
| 1865 | − * empty and should be rb_erased. | |
| 1866 | − */ | |
| 1867 | − stable_node_any = stable_node_dup_any(stable_node, | |
| 1868 | − root); | |
| 1869 | − if (!stable_node_any) { | |
| 1870 | − /* rb_erase just run */ | |
| 1871 | − goto again; | |
| 1872 | − } | |
| 1873 | − /* | |
| 1874 | − * Take any of the stable_node dups page of | |
| 1875 | − * this stable_node chain to let the tree walk | |
| 1876 | − * continue. All KSM pages belonging to the | |
| 1877 | − * stable_node dups in a stable_node chain | |
| 1878 | − * have the same content and they're | |
| 1879 | − * write protected at all times. Any will work | |
| 1880 | − * fine to continue the walk. | |
| 1881 | − */ | |
| 1882 | − tree_folio = ksm_get_folio(stable_node_any, | |
| 1883 | − KSM_GET_FOLIO_NOLOCK); | |
| 1884 | − } | |
| 1885 | − VM_BUG_ON(!stable_node_dup ^ !!stable_node_any); | |
| 1886 | 1851 | if (!tree_folio) { |
| 1887 | 1852 | /* |
| 1888 | 1853 | * If we walked over a stale stable_node, |
| @@ -1920,7 +1885,7 @@ again: · 31 unmodified lines | ||
| 1920 | 1885 | goto chain_append; |
| 1921 | 1886 | } |
| 1922 | 1887 | |
| 1923 | − if (!stable_node_dup) { | |
| 1888 | + if (!is_page_sharing_candidate(stable_node_dup)) { | |
| 1924 | 1889 | /* |
| 1925 | 1890 | * If the stable_node is a chain and |
| 1926 | 1891 | * we got a payload match in memcmp |
| @@ -2029,9 +1994,6 @@ replace: · 102 unmodified lines | ||
| 2029 | 1994 | return &folio->page; |
| 2030 | 1995 | |
| 2031 | 1996 | chain_append: |
| 2032 | − /* stable_node_dup could be null if it reached the limit */ | |
| 2033 | − if (!stable_node_dup) | |
| 2034 | − stable_node_dup = stable_node_any; | |
| 2035 | 1997 | /* |
| 2036 | 1998 | * If stable_node was a chain and chain_prune collapsed it, |
| 2037 | 1999 | * stable_node has been updated to be the new regular |
| @@ -2076,7 +2038,7 @@ static struct ksm_stable_node *stable_tree_insert(struct folio *kfolio) · 38 unmodified lines | ||
| 2076 | 2038 | struct rb_root *root; |
| 2077 | 2039 | struct rb_node **new; |
| 2078 | 2040 | struct rb_node *parent; |
| 2079 | − struct ksm_stable_node *stable_node, *stable_node_dup, *stable_node_any; | |
| 2041 | + struct ksm_stable_node *stable_node, *stable_node_dup; | |
| 2080 | 2042 | bool need_chain = false; |
| 2081 | 2043 | |
| 2082 | 2044 | kpfn = folio_pfn(kfolio); |
| @@ -2092,33 +2054,7 @@ again: · 9 unmodified lines | ||
| 2092 | 2054 | |
| 2093 | 2055 | cond_resched(); |
| 2094 | 2056 | stable_node = rb_entry(*new, struct ksm_stable_node, node); |
| 2095 | − stable_node_any = NULL; | |
| 2096 | − tree_folio = chain(&stable_node_dup, stable_node, root); | |
| 2097 | − if (!stable_node_dup) { | |
| 2098 | − /* | |
| 2099 | − * Either all stable_node dups were full in | |
| 2100 | − * this stable_node chain, or this chain was | |
| 2101 | − * empty and should be rb_erased. | |
| 2102 | − */ | |
| 2103 | − stable_node_any = stable_node_dup_any(stable_node, | |
| 2104 | − root); | |
| 2105 | − if (!stable_node_any) { | |
| 2106 | − /* rb_erase just run */ | |
| 2107 | − goto again; | |
| 2108 | − } | |
| 2109 | − /* | |
| 2110 | − * Take any of the stable_node dups page of | |
| 2111 | − * this stable_node chain to let the tree walk | |
| 2112 | − * continue. All KSM pages belonging to the | |
| 2113 | − * stable_node dups in a stable_node chain | |
| 2114 | − * have the same content and they're | |
| 2115 | − * write protected at all times. Any will work | |
| 2116 | − * fine to continue the walk. | |
| 2117 | − */ | |
| 2118 | − tree_folio = ksm_get_folio(stable_node_any, | |
| 2119 | − KSM_GET_FOLIO_NOLOCK); | |
| 2120 | − } | |
| 2121 | − VM_BUG_ON(!stable_node_dup ^ !!stable_node_any); | |
| 2057 | + tree_folio = chain(&stable_node_dup, &stable_node, root); | |
| 2122 | 2058 | if (!tree_folio) { |
| 2123 | 2059 | /* |
| 2124 | 2060 | * If we walked over a stale stable_node, |
| @@ -2306,7 +2242,6 @@ static void stable_tree_append(struct ksm_rmap_item *rmap_item, · 181 unmodified lines | ||
| 2306 | 2242 | */ |
| 2307 | 2243 | static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_item) |
| 2308 | 2244 | { |
| 2309 | − struct mm_struct *mm = rmap_item->mm; | |
| 2310 | 2245 | struct ksm_rmap_item *tree_rmap_item; |
| 2311 | 2246 | struct page *tree_page = NULL; |
| 2312 | 2247 | struct ksm_stable_node *stable_node; |
| @@ -2333,6 +2268,23 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite · 20 unmodified lines | ||
| 2333 | 2268 | */ |
| 2334 | 2269 | if (!is_page_sharing_candidate(stable_node)) |
| 2335 | 2270 | max_page_sharing_bypass = true; |
| 2271 | + } else { | |
| 2272 | + remove_rmap_item_from_tree(rmap_item); | |
| 2273 | + | |
| 2274 | + /* | |
| 2275 | + * If the hash value of the page has changed from the last time | |
| 2276 | + * we calculated it, this page is changing frequently: therefore we | |
| 2277 | + * don't want to insert it in the unstable tree, and we don't want | |
| 2278 | + * to waste our time searching for something identical to it there. | |
| 2279 | + */ | |
| 2280 | + checksum = calc_checksum(page); | |
| 2281 | + if (rmap_item->oldchecksum != checksum) { | |
| 2282 | + rmap_item->oldchecksum = checksum; | |
| 2283 | + return; | |
| 2284 | + } | |
| 2285 | + | |
| 2286 | + if (!try_to_merge_with_zero_page(rmap_item, page)) | |
| 2287 | + return; | |
| 2336 | 2288 | } |
| 2337 | 2289 | |
| 2338 | 2290 | /* We first start with searching the page inside the stable tree */ |
| @@ -2363,48 +2315,6 @@ static void cmp_and_merge_page(struct page *page, struct ksm_rmap_item *rmap_ite · 24 unmodified lines | ||
| 2363 | 2315 | return; |
| 2364 | 2316 | } |
| 2365 | 2317 | |
| 2366 | − /* | |
| 2367 | − * If the hash value of the page has changed from the last time | |
| 2368 | − * we calculated it, this page is changing frequently: therefore we | |
| 2369 | − * don't want to insert it in the unstable tree, and we don't want | |
| 2370 | − * to waste our time searching for something identical to it there. | |
| 2371 | − */ | |
| 2372 | − checksum = calc_checksum(page); | |
| 2373 | − if (rmap_item->oldchecksum != checksum) { | |
| 2374 | − rmap_item->oldchecksum = checksum; | |
| 2375 | − return; | |
| 2376 | − } | |
| 2377 | − | |
| 2378 | − /* | |
| 2379 | − * Same checksum as an empty page. We attempt to merge it with the | |
| 2380 | − * appropriate zero page if the user enabled this via sysfs. | |
| 2381 | − */ | |
| 2382 | − if (ksm_use_zero_pages && (checksum == zero_checksum)) { | |
| 2383 | − struct vm_area_struct *vma; | |
| 2384 | − | |
| 2385 | − mmap_read_lock(mm); | |
| 2386 | − vma = find_mergeable_vma(mm, rmap_item->address); | |
| 2387 | − if (vma) { | |
| 2388 | − err = try_to_merge_one_page(vma, page, | |
| 2389 | − ZERO_PAGE(rmap_item->address)); | |
| 2390 | − trace_ksm_merge_one_page( | |
| 2391 | − page_to_pfn(ZERO_PAGE(rmap_item->address)), | |
| 2392 | − rmap_item, mm, err); | |
| 2393 | − } else { | |
| 2394 | − /* | |
| 2395 | − * If the vma is out of date, we do not need to | |
| 2396 | − * continue. | |
| 2397 | − */ | |
| 2398 | − err = 0; | |
| 2399 | − } | |
| 2400 | − mmap_read_unlock(mm); | |
| 2401 | − /* | |
| 2402 | − * In case of failure, the page was not really empty, so we | |
| 2403 | − * need to continue. Otherwise we're done. | |
| 2404 | − */ | |
| 2405 | − if (!err) | |
| 2406 | − return; | |
| 2407 | − } | |
| 2408 | 2318 | tree_rmap_item = |
| 2409 | 2319 | unstable_tree_search_insert(rmap_item, page, &tree_page); |
| 2410 | 2320 | if (tree_rmap_item) { |
| @@ -3088,7 +2998,6 @@ struct folio *ksm_might_need_to_copy(struct folio *folio, · 677 unmodified lines | ||
| 3088 | 2998 | if (copy_mc_user_highpage(folio_page(new_folio, 0), page, |
| 3089 | 2999 | addr, vma)) { |
| 3090 | 3000 | folio_put(new_folio); |
| 3091 | − memory_failure_queue(folio_pfn(folio), 0); | |
| 3092 | 3001 | return ERR_PTR(-EHWPOISON); |
| 3093 | 3002 | } |
| 3094 | 3003 | folio_set_dirty(new_folio); |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include "slab.h" |
| 16 | 16 | #include "internal.h" |
| 17 | 17 | |
| 18 | −#ifdef CONFIG_MEMCG_KMEM | |
| 18 | +#ifdef CONFIG_MEMCG | |
| 19 | 19 | static LIST_HEAD(memcg_list_lrus); |
| 20 | 20 | static DEFINE_MUTEX(list_lrus_mutex); |
| 21 | 21 | |
| @@ -83,7 +83,7 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx) · 61 unmodified lines | ||
| 83 | 83 | { |
| 84 | 84 | return &lru->node[nid].lru; |
| 85 | 85 | } |
| 86 | −#endif /* CONFIG_MEMCG_KMEM */ | |
| 86 | +#endif /* CONFIG_MEMCG */ | |
| 87 | 87 | |
| 88 | 88 | bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, |
| 89 | 89 | struct mem_cgroup *memcg) |
| @@ -294,7 +294,7 @@ unsigned long list_lru_walk_node(struct list_lru *lru, int nid, · 204 unmodified lines | ||
| 294 | 294 | isolated += list_lru_walk_one(lru, nid, NULL, isolate, cb_arg, |
| 295 | 295 | nr_to_walk); |
| 296 | 296 | |
| 297 | −#ifdef CONFIG_MEMCG_KMEM | |
| 297 | +#ifdef CONFIG_MEMCG | |
| 298 | 298 | if (*nr_to_walk > 0 && list_lru_memcg_aware(lru)) { |
| 299 | 299 | struct list_lru_memcg *mlru; |
| 300 | 300 | unsigned long index; |
| @@ -324,7 +324,7 @@ static void init_one_lru(struct list_lru_one *l) · 23 unmodified lines | ||
| 324 | 324 | l->nr_items = 0; |
| 325 | 325 | } |
| 326 | 326 | |
| 327 | −#ifdef CONFIG_MEMCG_KMEM | |
| 327 | +#ifdef CONFIG_MEMCG | |
| 328 | 328 | static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp) |
| 329 | 329 | { |
| 330 | 330 | int nid; |
| @@ -544,14 +544,14 @@ static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware) · 213 unmodified lines | ||
| 544 | 544 | static void memcg_destroy_list_lru(struct list_lru *lru) |
| 545 | 545 | { |
| 546 | 546 | } |
| 547 | −#endif /* CONFIG_MEMCG_KMEM */ | |
| 547 | +#endif /* CONFIG_MEMCG */ | |
| 548 | 548 | |
| 549 | 549 | int __list_lru_init(struct list_lru *lru, bool memcg_aware, |
| 550 | 550 | struct lock_class_key *key, struct shrinker *shrinker) |
| 551 | 551 | { |
| 552 | 552 | int i; |
| 553 | 553 | |
| 554 | −#ifdef CONFIG_MEMCG_KMEM | |
| 554 | +#ifdef CONFIG_MEMCG | |
| 555 | 555 | if (shrinker) |
| 556 | 556 | lru->shrinker_id = shrinker->id; |
| 557 | 557 | else |
| @@ -591,7 +591,7 @@ void list_lru_destroy(struct list_lru *lru) · 33 unmodified lines | ||
| 591 | 591 | kfree(lru->node); |
| 592 | 592 | lru->node = NULL; |
| 593 | 593 | |
| 594 | −#ifdef CONFIG_MEMCG_KMEM | |
| 594 | +#ifdef CONFIG_MEMCG | |
| 595 | 595 | lru->shrinker_id = -1; |
| 596 | 596 | #endif |
| 597 | 597 | } |
| @@ -1147,7 +1147,7 @@ static int madvise_inject_error(int behavior, · 1146 unmodified lines | ||
| 1147 | 1147 | } else { |
| 1148 | 1148 | pr_info("Injecting memory failure for pfn %#lx at process virtual address %#lx\n", |
| 1149 | 1149 | pfn, start); |
| 1150 | − ret = memory_failure(pfn, MF_COUNT_INCREASED | MF_SW_SIMULATED); | |
| 1150 | + ret = memory_failure(pfn, MF_ACTION_REQUIRED | MF_COUNT_INCREASED | MF_SW_SIMULATED); | |
| 1151 | 1151 | if (ret == -EOPNOTSUPP) |
| 1152 | 1152 | ret = 0; |
| 1153 | 1153 | } |
| @@ -0,0 +1,2969 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-or-later | |
| 2 | + | |
| 3 | +#include <linux/memcontrol.h> | |
| 4 | +#include <linux/swap.h> | |
| 5 | +#include <linux/mm_inline.h> | |
| 6 | +#include <linux/pagewalk.h> | |
| 7 | +#include <linux/backing-dev.h> | |
| 8 | +#include <linux/swap_cgroup.h> | |
| 9 | +#include <linux/eventfd.h> | |
| 10 | +#include <linux/poll.h> | |
| 11 | +#include <linux/sort.h> | |
| 12 | +#include <linux/file.h> | |
| 13 | +#include <linux/seq_buf.h> | |
| 14 | + | |
| 15 | +#include "internal.h" | |
| 16 | +#include "swap.h" | |
| 17 | +#include "memcontrol-v1.h" | |
| 18 | + | |
| 19 | +/* | |
| 20 | + * Cgroups above their limits are maintained in a RB-Tree, independent of | |
| 21 | + * their hierarchy representation | |
| 22 | + */ | |
| 23 | + | |
| 24 | +struct mem_cgroup_tree_per_node { | |
| 25 | + struct rb_root rb_root; | |
| 26 | + struct rb_node *rb_rightmost; | |
| 27 | + spinlock_t lock; | |
| 28 | +}; | |
| 29 | + | |
| 30 | +struct mem_cgroup_tree { | |
| 31 | + struct mem_cgroup_tree_per_node *rb_tree_per_node[MAX_NUMNODES]; | |
| 32 | +}; | |
| 33 | + | |
| 34 | +static struct mem_cgroup_tree soft_limit_tree __read_mostly; | |
| 35 | + | |
| 36 | +/* | |
| 37 | + * Maximum loops in mem_cgroup_soft_reclaim(), used for soft | |
| 38 | + * limit reclaim to prevent infinite loops, if they ever occur. | |
| 39 | + */ | |
| 40 | +#define MEM_CGROUP_MAX_RECLAIM_LOOPS 100 | |
| 41 | +#define MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS 2 | |
| 42 | + | |
| 43 | +/* Stuffs for move charges at task migration. */ | |
| 44 | +/* | |
| 45 | + * Types of charges to be moved. | |
| 46 | + */ | |
| 47 | +#define MOVE_ANON 0x1ULL | |
| 48 | +#define MOVE_FILE 0x2ULL | |
| 49 | +#define MOVE_MASK (MOVE_ANON | MOVE_FILE) | |
| 50 | + | |
| 51 | +/* "mc" and its members are protected by cgroup_mutex */ | |
| 52 | +static struct move_charge_struct { | |
| 53 | + spinlock_t lock; /* for from, to */ | |
| 54 | + struct mm_struct *mm; | |
| 55 | + struct mem_cgroup *from; | |
| 56 | + struct mem_cgroup *to; | |
| 57 | + unsigned long flags; | |
| 58 | + unsigned long precharge; | |
| 59 | + unsigned long moved_charge; | |
| 60 | + unsigned long moved_swap; | |
| 61 | + struct task_struct *moving_task; /* a task moving charges */ | |
| 62 | + wait_queue_head_t waitq; /* a waitq for other context */ | |
| 63 | +} mc = { | |
| 64 | + .lock = __SPIN_LOCK_UNLOCKED(mc.lock), | |
| 65 | + .waitq = __WAIT_QUEUE_HEAD_INITIALIZER(mc.waitq), | |
| 66 | +}; | |
| 67 | + | |
| 68 | +/* for OOM */ | |
| 69 | +struct mem_cgroup_eventfd_list { | |
| 70 | + struct list_head list; | |
| 71 | + struct eventfd_ctx *eventfd; | |
| 72 | +}; | |
| 73 | + | |
| 74 | +/* | |
| 75 | + * cgroup_event represents events which userspace want to receive. | |
| 76 | + */ | |
| 77 | +struct mem_cgroup_event { | |
| 78 | + /* | |
| 79 | + * memcg which the event belongs to. | |
| 80 | + */ | |
| 81 | + struct mem_cgroup *memcg; | |
| 82 | + /* | |
| 83 | + * eventfd to signal userspace about the event. | |
| 84 | + */ | |
| 85 | + struct eventfd_ctx *eventfd; | |
| 86 | + /* | |
| 87 | + * Each of these stored in a list by the cgroup. | |
| 88 | + */ | |
| 89 | + struct list_head list; | |
| 90 | + /* | |
| 91 | + * register_event() callback will be used to add new userspace | |
| 92 | + * waiter for changes related to this event. Use eventfd_signal() | |
| 93 | + * on eventfd to send notification to userspace. | |
| 94 | + */ | |
| 95 | + int (*register_event)(struct mem_cgroup *memcg, | |
| 96 | + struct eventfd_ctx *eventfd, const char *args); | |
| 97 | + /* | |
| 98 | + * unregister_event() callback will be called when userspace closes | |
| 99 | + * the eventfd or on cgroup removing. This callback must be set, | |
| 100 | + * if you want provide notification functionality. | |
| 101 | + */ | |
| 102 | + void (*unregister_event)(struct mem_cgroup *memcg, | |
| 103 | + struct eventfd_ctx *eventfd); | |
| 104 | + /* | |
| 105 | + * All fields below needed to unregister event when | |
| 106 | + * userspace closes eventfd. | |
| 107 | + */ | |
| 108 | + poll_table pt; | |
| 109 | + wait_queue_head_t *wqh; | |
| 110 | + wait_queue_entry_t wait; | |
| 111 | + struct work_struct remove; | |
| 112 | +}; | |
| 113 | + | |
| 114 | +#define MEMFILE_PRIVATE(x, val) ((x) << 16 | (val)) | |
| 115 | +#define MEMFILE_TYPE(val) ((val) >> 16 & 0xffff) | |
| 116 | +#define MEMFILE_ATTR(val) ((val) & 0xffff) | |
| 117 | + | |
| 118 | +enum { | |
| 119 | + RES_USAGE, | |
| 120 | + RES_LIMIT, | |
| 121 | + RES_MAX_USAGE, | |
| 122 | + RES_FAILCNT, | |
| 123 | + RES_SOFT_LIMIT, | |
| 124 | +}; | |
| 125 | + | |
| 126 | +#ifdef CONFIG_LOCKDEP | |
| 127 | +static struct lockdep_map memcg_oom_lock_dep_map = { | |
| 128 | + .name = "memcg_oom_lock", | |
| 129 | +}; | |
| 130 | +#endif | |
| 131 | + | |
| 132 | +DEFINE_SPINLOCK(memcg_oom_lock); | |
| 133 | + | |
| 134 | +static void __mem_cgroup_insert_exceeded(struct mem_cgroup_per_node *mz, | |
| 135 | + struct mem_cgroup_tree_per_node *mctz, | |
| 136 | + unsigned long new_usage_in_excess) | |
| 137 | +{ | |
| 138 | + struct rb_node **p = &mctz->rb_root.rb_node; | |
| 139 | + struct rb_node *parent = NULL; | |
| 140 | + struct mem_cgroup_per_node *mz_node; | |
| 141 | + bool rightmost = true; | |
| 142 | + | |
| 143 | + if (mz->on_tree) | |
| 144 | + return; | |
| 145 | + | |
| 146 | + mz->usage_in_excess = new_usage_in_excess; | |
| 147 | + if (!mz->usage_in_excess) | |
| 148 | + return; | |
| 149 | + while (*p) { | |
| 150 | + parent = *p; | |
| 151 | + mz_node = rb_entry(parent, struct mem_cgroup_per_node, | |
| 152 | + tree_node); | |
| 153 | + if (mz->usage_in_excess < mz_node->usage_in_excess) { | |
| 154 | + p = &(*p)->rb_left; | |
| 155 | + rightmost = false; | |
| 156 | + } else { | |
| 157 | + p = &(*p)->rb_right; | |
| 158 | + } | |
| 159 | + } | |
| 160 | + | |
| 161 | + if (rightmost) | |
| 162 | + mctz->rb_rightmost = &mz->tree_node; | |
| 163 | + | |
| 164 | + rb_link_node(&mz->tree_node, parent, p); | |
| 165 | + rb_insert_color(&mz->tree_node, &mctz->rb_root); | |
| 166 | + mz->on_tree = true; | |
| 167 | +} | |
| 168 | + | |
| 169 | +static void __mem_cgroup_remove_exceeded(struct mem_cgroup_per_node *mz, | |
| 170 | + struct mem_cgroup_tree_per_node *mctz) | |
| 171 | +{ | |
| 172 | + if (!mz->on_tree) | |
| 173 | + return; | |
| 174 | + | |
| 175 | + if (&mz->tree_node == mctz->rb_rightmost) | |
| 176 | + mctz->rb_rightmost = rb_prev(&mz->tree_node); | |
| 177 | + | |
| 178 | + rb_erase(&mz->tree_node, &mctz->rb_root); | |
| 179 | + mz->on_tree = false; | |
| 180 | +} | |
| 181 | + | |
| 182 | +static void mem_cgroup_remove_exceeded(struct mem_cgroup_per_node *mz, | |
| 183 | + struct mem_cgroup_tree_per_node *mctz) | |
| 184 | +{ | |
| 185 | + unsigned long flags; | |
| 186 | + | |
| 187 | + spin_lock_irqsave(&mctz->lock, flags); | |
| 188 | + __mem_cgroup_remove_exceeded(mz, mctz); | |
| 189 | + spin_unlock_irqrestore(&mctz->lock, flags); | |
| 190 | +} | |
| 191 | + | |
| 192 | +static unsigned long soft_limit_excess(struct mem_cgroup *memcg) | |
| 193 | +{ | |
| 194 | + unsigned long nr_pages = page_counter_read(&memcg->memory); | |
| 195 | + unsigned long soft_limit = READ_ONCE(memcg->soft_limit); | |
| 196 | + unsigned long excess = 0; | |
| 197 | + | |
| 198 | + if (nr_pages > soft_limit) | |
| 199 | + excess = nr_pages - soft_limit; | |
| 200 | + | |
| 201 | + return excess; | |
| 202 | +} | |
| 203 | + | |
| 204 | +static void memcg1_update_tree(struct mem_cgroup *memcg, int nid) | |
| 205 | +{ | |
| 206 | + unsigned long excess; | |
| 207 | + struct mem_cgroup_per_node *mz; | |
| 208 | + struct mem_cgroup_tree_per_node *mctz; | |
| 209 | + | |
| 210 | + if (lru_gen_enabled()) { | |
| 211 | + if (soft_limit_excess(memcg)) | |
| 212 | + lru_gen_soft_reclaim(memcg, nid); | |
| 213 | + return; | |
| 214 | + } | |
| 215 | + | |
| 216 | + mctz = soft_limit_tree.rb_tree_per_node[nid]; | |
| 217 | + if (!mctz) | |
| 218 | + return; | |
| 219 | + /* | |
| 220 | + * Necessary to update all ancestors when hierarchy is used. | |
| 221 | + * because their event counter is not touched. | |
| 222 | + */ | |
| 223 | + for (; memcg; memcg = parent_mem_cgroup(memcg)) { | |
| 224 | + mz = memcg->nodeinfo[nid]; | |
| 225 | + excess = soft_limit_excess(memcg); | |
| 226 | + /* | |
| 227 | + * We have to update the tree if mz is on RB-tree or | |
| 228 | + * mem is over its softlimit. | |
| 229 | + */ | |
| 230 | + if (excess || mz->on_tree) { | |
| 231 | + unsigned long flags; | |
| 232 | + | |
| 233 | + spin_lock_irqsave(&mctz->lock, flags); | |
| 234 | + /* if on-tree, remove it */ | |
| 235 | + if (mz->on_tree) | |
| 236 | + __mem_cgroup_remove_exceeded(mz, mctz); | |
| 237 | + /* | |
| 238 | + * Insert again. mz->usage_in_excess will be updated. | |
| 239 | + * If excess is 0, no tree ops. | |
| 240 | + */ | |
| 241 | + __mem_cgroup_insert_exceeded(mz, mctz, excess); | |
| 242 | + spin_unlock_irqrestore(&mctz->lock, flags); | |
| 243 | + } | |
| 244 | + } | |
| 245 | +} | |
| 246 | + | |
| 247 | +void memcg1_remove_from_trees(struct mem_cgroup *memcg) | |
| 248 | +{ | |
| 249 | + struct mem_cgroup_tree_per_node *mctz; | |
| 250 | + struct mem_cgroup_per_node *mz; | |
| 251 | + int nid; | |
| 252 | + | |
| 253 | + for_each_node(nid) { | |
| 254 | + mz = memcg->nodeinfo[nid]; | |
| 255 | + mctz = soft_limit_tree.rb_tree_per_node[nid]; | |
| 256 | + if (mctz) | |
| 257 | + mem_cgroup_remove_exceeded(mz, mctz); | |
| 258 | + } | |
| 259 | +} | |
| 260 | + | |
| 261 | +static struct mem_cgroup_per_node * | |
| 262 | +__mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) | |
| 263 | +{ | |
| 264 | + struct mem_cgroup_per_node *mz; | |
| 265 | + | |
| 266 | +retry: | |
| 267 | + mz = NULL; | |
| 268 | + if (!mctz->rb_rightmost) | |
| 269 | + goto done; /* Nothing to reclaim from */ | |
| 270 | + | |
| 271 | + mz = rb_entry(mctz->rb_rightmost, | |
| 272 | + struct mem_cgroup_per_node, tree_node); | |
| 273 | + /* | |
| 274 | + * Remove the node now but someone else can add it back, | |
| 275 | + * we will to add it back at the end of reclaim to its correct | |
| 276 | + * position in the tree. | |
| 277 | + */ | |
| 278 | + __mem_cgroup_remove_exceeded(mz, mctz); | |
| 279 | + if (!soft_limit_excess(mz->memcg) || | |
| 280 | + !css_tryget(&mz->memcg->css)) | |
| 281 | + goto retry; | |
| 282 | +done: | |
| 283 | + return mz; | |
| 284 | +} | |
| 285 | + | |
| 286 | +static struct mem_cgroup_per_node * | |
| 287 | +mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) | |
| 288 | +{ | |
| 289 | + struct mem_cgroup_per_node *mz; | |
| 290 | + | |
| 291 | + spin_lock_irq(&mctz->lock); | |
| 292 | + mz = __mem_cgroup_largest_soft_limit_node(mctz); | |
| 293 | + spin_unlock_irq(&mctz->lock); | |
| 294 | + return mz; | |
| 295 | +} | |
| 296 | + | |
| 297 | +static int mem_cgroup_soft_reclaim(struct mem_cgroup *root_memcg, | |
| 298 | + pg_data_t *pgdat, | |
| 299 | + gfp_t gfp_mask, | |
| 300 | + unsigned long *total_scanned) | |
| 301 | +{ | |
| 302 | + struct mem_cgroup *victim = NULL; | |
| 303 | + int total = 0; | |
| 304 | + int loop = 0; | |
| 305 | + unsigned long excess; | |
| 306 | + unsigned long nr_scanned; | |
| 307 | + struct mem_cgroup_reclaim_cookie reclaim = { | |
| 308 | + .pgdat = pgdat, | |
| 309 | + }; | |
| 310 | + | |
| 311 | + excess = soft_limit_excess(root_memcg); | |
| 312 | + | |
| 313 | + while (1) { | |
| 314 | + victim = mem_cgroup_iter(root_memcg, victim, &reclaim); | |
| 315 | + if (!victim) { | |
| 316 | + loop++; | |
| 317 | + if (loop >= 2) { | |
| 318 | + /* | |
| 319 | + * If we have not been able to reclaim | |
| 320 | + * anything, it might because there are | |
| 321 | + * no reclaimable pages under this hierarchy | |
| 322 | + */ | |
| 323 | + if (!total) | |
| 324 | + break; | |
| 325 | + /* | |
| 326 | + * We want to do more targeted reclaim. | |
| 327 | + * excess >> 2 is not to excessive so as to | |
| 328 | + * reclaim too much, nor too less that we keep | |
| 329 | + * coming back to reclaim from this cgroup | |
| 330 | + */ | |
| 331 | + if (total >= (excess >> 2) || | |
| 332 | + (loop > MEM_CGROUP_MAX_RECLAIM_LOOPS)) | |
| 333 | + break; | |
| 334 | + } | |
| 335 | + continue; | |
| 336 | + } | |
| 337 | + total += mem_cgroup_shrink_node(victim, gfp_mask, false, | |
| 338 | + pgdat, &nr_scanned); | |
| 339 | + *total_scanned += nr_scanned; | |
| 340 | + if (!soft_limit_excess(root_memcg)) | |
| 341 | + break; | |
| 342 | + } | |
| 343 | + mem_cgroup_iter_break(root_memcg, victim); | |
| 344 | + return total; | |
| 345 | +} | |
| 346 | + | |
| 347 | +unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order, | |
| 348 | + gfp_t gfp_mask, | |
| 349 | + unsigned long *total_scanned) | |
| 350 | +{ | |
| 351 | + unsigned long nr_reclaimed = 0; | |
| 352 | + struct mem_cgroup_per_node *mz, *next_mz = NULL; | |
| 353 | + unsigned long reclaimed; | |
| 354 | + int loop = 0; | |
| 355 | + struct mem_cgroup_tree_per_node *mctz; | |
| 356 | + unsigned long excess; | |
| 357 | + | |
| 358 | + if (lru_gen_enabled()) | |
| 359 | + return 0; | |
| 360 | + | |
| 361 | + if (order > 0) | |
| 362 | + return 0; | |
| 363 | + | |
| 364 | + mctz = soft_limit_tree.rb_tree_per_node[pgdat->node_id]; | |
| 365 | + | |
| 366 | + /* | |
| 367 | + * Do not even bother to check the largest node if the root | |
| 368 | + * is empty. Do it lockless to prevent lock bouncing. Races | |
| 369 | + * are acceptable as soft limit is best effort anyway. | |
| 370 | + */ | |
| 371 | + if (!mctz || RB_EMPTY_ROOT(&mctz->rb_root)) | |
| 372 | + return 0; | |
| 373 | + | |
| 374 | + /* | |
| 375 | + * This loop can run a while, specially if mem_cgroup's continuously | |
| 376 | + * keep exceeding their soft limit and putting the system under | |
| 377 | + * pressure | |
| 378 | + */ | |
| 379 | + do { | |
| 380 | + if (next_mz) | |
| 381 | + mz = next_mz; | |
| 382 | + else | |
| 383 | + mz = mem_cgroup_largest_soft_limit_node(mctz); | |
| 384 | + if (!mz) | |
| 385 | + break; | |
| 386 | + | |
| 387 | + reclaimed = mem_cgroup_soft_reclaim(mz->memcg, pgdat, | |
| 388 | + gfp_mask, total_scanned); | |
| 389 | + nr_reclaimed += reclaimed; | |
| 390 | + spin_lock_irq(&mctz->lock); | |
| 391 | + | |
| 392 | + /* | |
| 393 | + * If we failed to reclaim anything from this memory cgroup | |
| 394 | + * it is time to move on to the next cgroup | |
| 395 | + */ | |
| 396 | + next_mz = NULL; | |
| 397 | + if (!reclaimed) | |
| 398 | + next_mz = __mem_cgroup_largest_soft_limit_node(mctz); | |
| 399 | + | |
| 400 | + excess = soft_limit_excess(mz->memcg); | |
| 401 | + /* | |
| 402 | + * One school of thought says that we should not add | |
| 403 | + * back the node to the tree if reclaim returns 0. | |
| 404 | + * But our reclaim could return 0, simply because due | |
| 405 | + * to priority we are exposing a smaller subset of | |
| 406 | + * memory to reclaim from. Consider this as a longer | |
| 407 | + * term TODO. | |
| 408 | + */ | |
| 409 | + /* If excess == 0, no tree ops */ | |
| 410 | + __mem_cgroup_insert_exceeded(mz, mctz, excess); | |
| 411 | + spin_unlock_irq(&mctz->lock); | |
| 412 | + css_put(&mz->memcg->css); | |
| 413 | + loop++; | |
| 414 | + /* | |
| 415 | + * Could not reclaim anything and there are no more | |
| 416 | + * mem cgroups to try or we seem to be looping without | |
| 417 | + * reclaiming anything. | |
| 418 | + */ | |
| 419 | + if (!nr_reclaimed && | |
| 420 | + (next_mz == NULL || | |
| 421 | + loop > MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS)) | |
| 422 | + break; | |
| 423 | + } while (!nr_reclaimed); | |
| 424 | + if (next_mz) | |
| 425 | + css_put(&next_mz->memcg->css); | |
| 426 | + return nr_reclaimed; | |
| 427 | +} | |
| 428 | + | |
| 429 | +/* | |
| 430 | + * A routine for checking "mem" is under move_account() or not. | |
| 431 | + * | |
| 432 | + * Checking a cgroup is mc.from or mc.to or under hierarchy of | |
| 433 | + * moving cgroups. This is for waiting at high-memory pressure | |
| 434 | + * caused by "move". | |
| 435 | + */ | |
| 436 | +static bool mem_cgroup_under_move(struct mem_cgroup *memcg) | |
| 437 | +{ | |
| 438 | + struct mem_cgroup *from; | |
| 439 | + struct mem_cgroup *to; | |
| 440 | + bool ret = false; | |
| 441 | + /* | |
| 442 | + * Unlike task_move routines, we access mc.to, mc.from not under | |
| 443 | + * mutual exclusion by cgroup_mutex. Here, we take spinlock instead. | |
| 444 | + */ | |
| 445 | + spin_lock(&mc.lock); | |
| 446 | + from = mc.from; | |
| 447 | + to = mc.to; | |
| 448 | + if (!from) | |
| 449 | + goto unlock; | |
| 450 | + | |
| 451 | + ret = mem_cgroup_is_descendant(from, memcg) || | |
| 452 | + mem_cgroup_is_descendant(to, memcg); | |
| 453 | +unlock: | |
| 454 | + spin_unlock(&mc.lock); | |
| 455 | + return ret; | |
| 456 | +} | |
| 457 | + | |
| 458 | +bool memcg1_wait_acct_move(struct mem_cgroup *memcg) | |
| 459 | +{ | |
| 460 | + if (mc.moving_task && current != mc.moving_task) { | |
| 461 | + if (mem_cgroup_under_move(memcg)) { | |
| 462 | + DEFINE_WAIT(wait); | |
| 463 | + prepare_to_wait(&mc.waitq, &wait, TASK_INTERRUPTIBLE); | |
| 464 | + /* moving charge context might have finished. */ | |
| 465 | + if (mc.moving_task) | |
| 466 | + schedule(); | |
| 467 | + finish_wait(&mc.waitq, &wait); | |
| 468 | + return true; | |
| 469 | + } | |
| 470 | + } | |
| 471 | + return false; | |
| 472 | +} | |
| 473 | + | |
| 474 | +/** | |
| 475 | + * folio_memcg_lock - Bind a folio to its memcg. | |
| 476 | + * @folio: The folio. | |
| 477 | + * | |
| 478 | + * This function prevents unlocked LRU folios from being moved to | |
| 479 | + * another cgroup. | |
| 480 | + * | |
| 481 | + * It ensures lifetime of the bound memcg. The caller is responsible | |
| 482 | + * for the lifetime of the folio. | |
| 483 | + */ | |
| 484 | +void folio_memcg_lock(struct folio *folio) | |
| 485 | +{ | |
| 486 | + struct mem_cgroup *memcg; | |
| 487 | + unsigned long flags; | |
| 488 | + | |
| 489 | + /* | |
| 490 | + * The RCU lock is held throughout the transaction. The fast | |
| 491 | + * path can get away without acquiring the memcg->move_lock | |
| 492 | + * because page moving starts with an RCU grace period. | |
| 493 | + */ | |
| 494 | + rcu_read_lock(); | |
| 495 | + | |
| 496 | + if (mem_cgroup_disabled()) | |
| 497 | + return; | |
| 498 | +again: | |
| 499 | + memcg = folio_memcg(folio); | |
| 500 | + if (unlikely(!memcg)) | |
| 501 | + return; | |
| 502 | + | |
| 503 | +#ifdef CONFIG_PROVE_LOCKING | |
| 504 | + local_irq_save(flags); | |
| 505 | + might_lock(&memcg->move_lock); | |
| 506 | + local_irq_restore(flags); | |
| 507 | +#endif | |
| 508 | + | |
| 509 | + if (atomic_read(&memcg->moving_account) <= 0) | |
| 510 | + return; | |
| 511 | + | |
| 512 | + spin_lock_irqsave(&memcg->move_lock, flags); | |
| 513 | + if (memcg != folio_memcg(folio)) { | |
| 514 | + spin_unlock_irqrestore(&memcg->move_lock, flags); | |
| 515 | + goto again; | |
| 516 | + } | |
| 517 | + | |
| 518 | + /* | |
| 519 | + * When charge migration first begins, we can have multiple | |
| 520 | + * critical sections holding the fast-path RCU lock and one | |
| 521 | + * holding the slowpath move_lock. Track the task who has the | |
| 522 | + * move_lock for folio_memcg_unlock(). | |
| 523 | + */ | |
| 524 | + memcg->move_lock_task = current; | |
| 525 | + memcg->move_lock_flags = flags; | |
| 526 | +} | |
| 527 | + | |
| 528 | +static void __folio_memcg_unlock(struct mem_cgroup *memcg) | |
| 529 | +{ | |
| 530 | + if (memcg && memcg->move_lock_task == current) { | |
| 531 | + unsigned long flags = memcg->move_lock_flags; | |
| 532 | + | |
| 533 | + memcg->move_lock_task = NULL; | |
| 534 | + memcg->move_lock_flags = 0; | |
| 535 | + | |
| 536 | + spin_unlock_irqrestore(&memcg->move_lock, flags); | |
| 537 | + } | |
| 538 | + | |
| 539 | + rcu_read_unlock(); | |
| 540 | +} | |
| 541 | + | |
| 542 | +/** | |
| 543 | + * folio_memcg_unlock - Release the binding between a folio and its memcg. | |
| 544 | + * @folio: The folio. | |
| 545 | + * | |
| 546 | + * This releases the binding created by folio_memcg_lock(). This does | |
| 547 | + * not change the accounting of this folio to its memcg, but it does | |
| 548 | + * permit others to change it. | |
| 549 | + */ | |
| 550 | +void folio_memcg_unlock(struct folio *folio) | |
| 551 | +{ | |
| 552 | + __folio_memcg_unlock(folio_memcg(folio)); | |
| 553 | +} | |
| 554 | + | |
| 555 | +#ifdef CONFIG_SWAP | |
| 556 | +/** | |
| 557 | + * mem_cgroup_move_swap_account - move swap charge and swap_cgroup's record. | |
| 558 | + * @entry: swap entry to be moved | |
| 559 | + * @from: mem_cgroup which the entry is moved from | |
| 560 | + * @to: mem_cgroup which the entry is moved to | |
| 561 | + * | |
| 562 | + * It succeeds only when the swap_cgroup's record for this entry is the same | |
| 563 | + * as the mem_cgroup's id of @from. | |
| 564 | + * | |
| 565 | + * Returns 0 on success, -EINVAL on failure. | |
| 566 | + * | |
| 567 | + * The caller must have charged to @to, IOW, called page_counter_charge() about | |
| 568 | + * both res and memsw, and called css_get(). | |
| 569 | + */ | |
| 570 | +static int mem_cgroup_move_swap_account(swp_entry_t entry, | |
| 571 | + struct mem_cgroup *from, struct mem_cgroup *to) | |
| 572 | +{ | |
| 573 | + unsigned short old_id, new_id; | |
| 574 | + | |
| 575 | + old_id = mem_cgroup_id(from); | |
| 576 | + new_id = mem_cgroup_id(to); | |
| 577 | + | |
| 578 | + if (swap_cgroup_cmpxchg(entry, old_id, new_id) == old_id) { | |
| 579 | + mod_memcg_state(from, MEMCG_SWAP, -1); | |
| 580 | + mod_memcg_state(to, MEMCG_SWAP, 1); | |
| 581 | + return 0; | |
| 582 | + } | |
| 583 | + return -EINVAL; | |
| 584 | +} | |
| 585 | +#else | |
| 586 | +static inline int mem_cgroup_move_swap_account(swp_entry_t entry, | |
| 587 | + struct mem_cgroup *from, struct mem_cgroup *to) | |
| 588 | +{ | |
| 589 | + return -EINVAL; | |
| 590 | +} | |
| 591 | +#endif | |
| 592 | + | |
| 593 | +static u64 mem_cgroup_move_charge_read(struct cgroup_subsys_state *css, | |
| 594 | + struct cftype *cft) | |
| 595 | +{ | |
| 596 | + return mem_cgroup_from_css(css)->move_charge_at_immigrate; | |
| 597 | +} | |
| 598 | + | |
| 599 | +#ifdef CONFIG_MMU | |
| 600 | +static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css, | |
| 601 | + struct cftype *cft, u64 val) | |
| 602 | +{ | |
| 603 | + struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 604 | + | |
| 605 | + pr_warn_once("Cgroup memory moving (move_charge_at_immigrate) is deprecated. " | |
| 606 | + "Please report your usecase to linux-mm@kvack.org if you " | |
| 607 | + "depend on this functionality.\n"); | |
| 608 | + | |
| 609 | + if (val & ~MOVE_MASK) | |
| 610 | + return -EINVAL; | |
| 611 | + | |
| 612 | + /* | |
| 613 | + * No kind of locking is needed in here, because ->can_attach() will | |
| 614 | + * check this value once in the beginning of the process, and then carry | |
| 615 | + * on with stale data. This means that changes to this value will only | |
| 616 | + * affect task migrations starting after the change. | |
| 617 | + */ | |
| 618 | + memcg->move_charge_at_immigrate = val; | |
| 619 | + return 0; | |
| 620 | +} | |
| 621 | +#else | |
| 622 | +static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css, | |
| 623 | + struct cftype *cft, u64 val) | |
| 624 | +{ | |
| 625 | + return -ENOSYS; | |
| 626 | +} | |
| 627 | +#endif | |
| 628 | + | |
| 629 | +#ifdef CONFIG_MMU | |
| 630 | +/* Handlers for move charge at task migration. */ | |
| 631 | +static int mem_cgroup_do_precharge(unsigned long count) | |
| 632 | +{ | |
| 633 | + int ret; | |
| 634 | + | |
| 635 | + /* Try a single bulk charge without reclaim first, kswapd may wake */ | |
| 636 | + ret = try_charge(mc.to, GFP_KERNEL & ~__GFP_DIRECT_RECLAIM, count); | |
| 637 | + if (!ret) { | |
| 638 | + mc.precharge += count; | |
| 639 | + return ret; | |
| 640 | + } | |
| 641 | + | |
| 642 | + /* Try charges one by one with reclaim, but do not retry */ | |
| 643 | + while (count--) { | |
| 644 | + ret = try_charge(mc.to, GFP_KERNEL | __GFP_NORETRY, 1); | |
| 645 | + if (ret) | |
| 646 | + return ret; | |
| 647 | + mc.precharge++; | |
| 648 | + cond_resched(); | |
| 649 | + } | |
| 650 | + return 0; | |
| 651 | +} | |
| 652 | + | |
| 653 | +union mc_target { | |
| 654 | + struct folio *folio; | |
| 655 | + swp_entry_t ent; | |
| 656 | +}; | |
| 657 | + | |
| 658 | +enum mc_target_type { | |
| 659 | + MC_TARGET_NONE = 0, | |
| 660 | + MC_TARGET_PAGE, | |
| 661 | + MC_TARGET_SWAP, | |
| 662 | + MC_TARGET_DEVICE, | |
| 663 | +}; | |
| 664 | + | |
| 665 | +static struct page *mc_handle_present_pte(struct vm_area_struct *vma, | |
| 666 | + unsigned long addr, pte_t ptent) | |
| 667 | +{ | |
| 668 | + struct page *page = vm_normal_page(vma, addr, ptent); | |
| 669 | + | |
| 670 | + if (!page) | |
| 671 | + return NULL; | |
| 672 | + if (PageAnon(page)) { | |
| 673 | + if (!(mc.flags & MOVE_ANON)) | |
| 674 | + return NULL; | |
| 675 | + } else { | |
| 676 | + if (!(mc.flags & MOVE_FILE)) | |
| 677 | + return NULL; | |
| 678 | + } | |
| 679 | + get_page(page); | |
| 680 | + | |
| 681 | + return page; | |
| 682 | +} | |
| 683 | + | |
| 684 | +#if defined(CONFIG_SWAP) || defined(CONFIG_DEVICE_PRIVATE) | |
| 685 | +static struct page *mc_handle_swap_pte(struct vm_area_struct *vma, | |
| 686 | + pte_t ptent, swp_entry_t *entry) | |
| 687 | +{ | |
| 688 | + struct page *page = NULL; | |
| 689 | + swp_entry_t ent = pte_to_swp_entry(ptent); | |
| 690 | + | |
| 691 | + if (!(mc.flags & MOVE_ANON)) | |
| 692 | + return NULL; | |
| 693 | + | |
| 694 | + /* | |
| 695 | + * Handle device private pages that are not accessible by the CPU, but | |
| 696 | + * stored as special swap entries in the page table. | |
| 697 | + */ | |
| 698 | + if (is_device_private_entry(ent)) { | |
| 699 | + page = pfn_swap_entry_to_page(ent); | |
| 700 | + if (!get_page_unless_zero(page)) | |
| 701 | + return NULL; | |
| 702 | + return page; | |
| 703 | + } | |
| 704 | + | |
| 705 | + if (non_swap_entry(ent)) | |
| 706 | + return NULL; | |
| 707 | + | |
| 708 | + /* | |
| 709 | + * Because swap_cache_get_folio() updates some statistics counter, | |
| 710 | + * we call find_get_page() with swapper_space directly. | |
| 711 | + */ | |
| 712 | + page = find_get_page(swap_address_space(ent), swap_cache_index(ent)); | |
| 713 | + entry->val = ent.val; | |
| 714 | + | |
| 715 | + return page; | |
| 716 | +} | |
| 717 | +#else | |
| 718 | +static struct page *mc_handle_swap_pte(struct vm_area_struct *vma, | |
| 719 | + pte_t ptent, swp_entry_t *entry) | |
| 720 | +{ | |
| 721 | + return NULL; | |
| 722 | +} | |
| 723 | +#endif | |
| 724 | + | |
| 725 | +static struct page *mc_handle_file_pte(struct vm_area_struct *vma, | |
| 726 | + unsigned long addr, pte_t ptent) | |
| 727 | +{ | |
| 728 | + unsigned long index; | |
| 729 | + struct folio *folio; | |
| 730 | + | |
| 731 | + if (!vma->vm_file) /* anonymous vma */ | |
| 732 | + return NULL; | |
| 733 | + if (!(mc.flags & MOVE_FILE)) | |
| 734 | + return NULL; | |
| 735 | + | |
| 736 | + /* folio is moved even if it's not RSS of this task(page-faulted). */ | |
| 737 | + /* shmem/tmpfs may report page out on swap: account for that too. */ | |
| 738 | + index = linear_page_index(vma, addr); | |
| 739 | + folio = filemap_get_incore_folio(vma->vm_file->f_mapping, index); | |
| 740 | + if (IS_ERR(folio)) | |
| 741 | + return NULL; | |
| 742 | + return folio_file_page(folio, index); | |
| 743 | +} | |
| 744 | + | |
| 745 | +/** | |
| 746 | + * mem_cgroup_move_account - move account of the folio | |
| 747 | + * @folio: The folio. | |
| 748 | + * @compound: charge the page as compound or small page | |
| 749 | + * @from: mem_cgroup which the folio is moved from. | |
| 750 | + * @to: mem_cgroup which the folio is moved to. @from != @to. | |
| 751 | + * | |
| 752 | + * The folio must be locked and not on the LRU. | |
| 753 | + * | |
| 754 | + * This function doesn't do "charge" to new cgroup and doesn't do "uncharge" | |
| 755 | + * from old cgroup. | |
| 756 | + */ | |
| 757 | +static int mem_cgroup_move_account(struct folio *folio, | |
| 758 | + bool compound, | |
| 759 | + struct mem_cgroup *from, | |
| 760 | + struct mem_cgroup *to) | |
| 761 | +{ | |
| 762 | + struct lruvec *from_vec, *to_vec; | |
| 763 | + struct pglist_data *pgdat; | |
| 764 | + unsigned int nr_pages = compound ? folio_nr_pages(folio) : 1; | |
| 765 | + int nid, ret; | |
| 766 | + | |
| 767 | + VM_BUG_ON(from == to); | |
| 768 | + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); | |
| 769 | + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); | |
| 770 | + VM_BUG_ON(compound && !folio_test_large(folio)); | |
| 771 | + | |
| 772 | + ret = -EINVAL; | |
| 773 | + if (folio_memcg(folio) != from) | |
| 774 | + goto out; | |
| 775 | + | |
| 776 | + pgdat = folio_pgdat(folio); | |
| 777 | + from_vec = mem_cgroup_lruvec(from, pgdat); | |
| 778 | + to_vec = mem_cgroup_lruvec(to, pgdat); | |
| 779 | + | |
| 780 | + folio_memcg_lock(folio); | |
| 781 | + | |
| 782 | + if (folio_test_anon(folio)) { | |
| 783 | + if (folio_mapped(folio)) { | |
| 784 | + __mod_lruvec_state(from_vec, NR_ANON_MAPPED, -nr_pages); | |
| 785 | + __mod_lruvec_state(to_vec, NR_ANON_MAPPED, nr_pages); | |
| 786 | + if (folio_test_pmd_mappable(folio)) { | |
| 787 | + __mod_lruvec_state(from_vec, NR_ANON_THPS, | |
| 788 | + -nr_pages); | |
| 789 | + __mod_lruvec_state(to_vec, NR_ANON_THPS, | |
| 790 | + nr_pages); | |
| 791 | + } | |
| 792 | + } | |
| 793 | + } else { | |
| 794 | + __mod_lruvec_state(from_vec, NR_FILE_PAGES, -nr_pages); | |
| 795 | + __mod_lruvec_state(to_vec, NR_FILE_PAGES, nr_pages); | |
| 796 | + | |
| 797 | + if (folio_test_swapbacked(folio)) { | |
| 798 | + __mod_lruvec_state(from_vec, NR_SHMEM, -nr_pages); | |
| 799 | + __mod_lruvec_state(to_vec, NR_SHMEM, nr_pages); | |
| 800 | + } | |
| 801 | + | |
| 802 | + if (folio_mapped(folio)) { | |
| 803 | + __mod_lruvec_state(from_vec, NR_FILE_MAPPED, -nr_pages); | |
| 804 | + __mod_lruvec_state(to_vec, NR_FILE_MAPPED, nr_pages); | |
| 805 | + } | |
| 806 | + | |
| 807 | + if (folio_test_dirty(folio)) { | |
| 808 | + struct address_space *mapping = folio_mapping(folio); | |
| 809 | + | |
| 810 | + if (mapping_can_writeback(mapping)) { | |
| 811 | + __mod_lruvec_state(from_vec, NR_FILE_DIRTY, | |
| 812 | + -nr_pages); | |
| 813 | + __mod_lruvec_state(to_vec, NR_FILE_DIRTY, | |
| 814 | + nr_pages); | |
| 815 | + } | |
| 816 | + } | |
| 817 | + } | |
| 818 | + | |
| 819 | +#ifdef CONFIG_SWAP | |
| 820 | + if (folio_test_swapcache(folio)) { | |
| 821 | + __mod_lruvec_state(from_vec, NR_SWAPCACHE, -nr_pages); | |
| 822 | + __mod_lruvec_state(to_vec, NR_SWAPCACHE, nr_pages); | |
| 823 | + } | |
| 824 | +#endif | |
| 825 | + if (folio_test_writeback(folio)) { | |
| 826 | + __mod_lruvec_state(from_vec, NR_WRITEBACK, -nr_pages); | |
| 827 | + __mod_lruvec_state(to_vec, NR_WRITEBACK, nr_pages); | |
| 828 | + } | |
| 829 | + | |
| 830 | + /* | |
| 831 | + * All state has been migrated, let's switch to the new memcg. | |
| 832 | + * | |
| 833 | + * It is safe to change page's memcg here because the page | |
| 834 | + * is referenced, charged, isolated, and locked: we can't race | |
| 835 | + * with (un)charging, migration, LRU putback, or anything else | |
| 836 | + * that would rely on a stable page's memory cgroup. | |
| 837 | + * | |
| 838 | + * Note that folio_memcg_lock is a memcg lock, not a page lock, | |
| 839 | + * to save space. As soon as we switch page's memory cgroup to a | |
| 840 | + * new memcg that isn't locked, the above state can change | |
| 841 | + * concurrently again. Make sure we're truly done with it. | |
| 842 | + */ | |
| 843 | + smp_mb(); | |
| 844 | + | |
| 845 | + css_get(&to->css); | |
| 846 | + css_put(&from->css); | |
| 847 | + | |
| 848 | + folio->memcg_data = (unsigned long)to; | |
| 849 | + | |
| 850 | + __folio_memcg_unlock(from); | |
| 851 | + | |
| 852 | + ret = 0; | |
| 853 | + nid = folio_nid(folio); | |
| 854 | + | |
| 855 | + local_irq_disable(); | |
| 856 | + mem_cgroup_charge_statistics(to, nr_pages); | |
| 857 | + memcg1_check_events(to, nid); | |
| 858 | + mem_cgroup_charge_statistics(from, -nr_pages); | |
| 859 | + memcg1_check_events(from, nid); | |
| 860 | + local_irq_enable(); | |
| 861 | +out: | |
| 862 | + return ret; | |
| 863 | +} | |
| 864 | + | |
| 865 | +/** | |
| 866 | + * get_mctgt_type - get target type of moving charge | |
| 867 | + * @vma: the vma the pte to be checked belongs | |
| 868 | + * @addr: the address corresponding to the pte to be checked | |
| 869 | + * @ptent: the pte to be checked | |
| 870 | + * @target: the pointer the target page or swap ent will be stored(can be NULL) | |
| 871 | + * | |
| 872 | + * Context: Called with pte lock held. | |
| 873 | + * Return: | |
| 874 | + * * MC_TARGET_NONE - If the pte is not a target for move charge. | |
| 875 | + * * MC_TARGET_PAGE - If the page corresponding to this pte is a target for | |
| 876 | + * move charge. If @target is not NULL, the folio is stored in target->folio | |
| 877 | + * with extra refcnt taken (Caller should release it). | |
| 878 | + * * MC_TARGET_SWAP - If the swap entry corresponding to this pte is a | |
| 879 | + * target for charge migration. If @target is not NULL, the entry is | |
| 880 | + * stored in target->ent. | |
| 881 | + * * MC_TARGET_DEVICE - Like MC_TARGET_PAGE but page is device memory and | |
| 882 | + * thus not on the lru. For now such page is charged like a regular page | |
| 883 | + * would be as it is just special memory taking the place of a regular page. | |
| 884 | + * See Documentations/vm/hmm.txt and include/linux/hmm.h | |
| 885 | + */ | |
| 886 | +static enum mc_target_type get_mctgt_type(struct vm_area_struct *vma, | |
| 887 | + unsigned long addr, pte_t ptent, union mc_target *target) | |
| 888 | +{ | |
| 889 | + struct page *page = NULL; | |
| 890 | + struct folio *folio; | |
| 891 | + enum mc_target_type ret = MC_TARGET_NONE; | |
| 892 | + swp_entry_t ent = { .val = 0 }; | |
| 893 | + | |
| 894 | + if (pte_present(ptent)) | |
| 895 | + page = mc_handle_present_pte(vma, addr, ptent); | |
| 896 | + else if (pte_none_mostly(ptent)) | |
| 897 | + /* | |
| 898 | + * PTE markers should be treated as a none pte here, separated | |
| 899 | + * from other swap handling below. | |
| 900 | + */ | |
| 901 | + page = mc_handle_file_pte(vma, addr, ptent); | |
| 902 | + else if (is_swap_pte(ptent)) | |
| 903 | + page = mc_handle_swap_pte(vma, ptent, &ent); | |
| 904 | + | |
| 905 | + if (page) | |
| 906 | + folio = page_folio(page); | |
| 907 | + if (target && page) { | |
| 908 | + if (!folio_trylock(folio)) { | |
| 909 | + folio_put(folio); | |
| 910 | + return ret; | |
| 911 | + } | |
| 912 | + /* | |
| 913 | + * page_mapped() must be stable during the move. This | |
| 914 | + * pte is locked, so if it's present, the page cannot | |
| 915 | + * become unmapped. If it isn't, we have only partial | |
| 916 | + * control over the mapped state: the page lock will | |
| 917 | + * prevent new faults against pagecache and swapcache, | |
| 918 | + * so an unmapped page cannot become mapped. However, | |
| 919 | + * if the page is already mapped elsewhere, it can | |
| 920 | + * unmap, and there is nothing we can do about it. | |
| 921 | + * Alas, skip moving the page in this case. | |
| 922 | + */ | |
| 923 | + if (!pte_present(ptent) && page_mapped(page)) { | |
| 924 | + folio_unlock(folio); | |
| 925 | + folio_put(folio); | |
| 926 | + return ret; | |
| 927 | + } | |
| 928 | + } | |
| 929 | + | |
| 930 | + if (!page && !ent.val) | |
| 931 | + return ret; | |
| 932 | + if (page) { | |
| 933 | + /* | |
| 934 | + * Do only loose check w/o serialization. | |
| 935 | + * mem_cgroup_move_account() checks the page is valid or | |
| 936 | + * not under LRU exclusion. | |
| 937 | + */ | |
| 938 | + if (folio_memcg(folio) == mc.from) { | |
| 939 | + ret = MC_TARGET_PAGE; | |
| 940 | + if (folio_is_device_private(folio) || | |
| 941 | + folio_is_device_coherent(folio)) | |
| 942 | + ret = MC_TARGET_DEVICE; | |
| 943 | + if (target) | |
| 944 | + target->folio = folio; | |
| 945 | + } | |
| 946 | + if (!ret || !target) { | |
| 947 | + if (target) | |
| 948 | + folio_unlock(folio); | |
| 949 | + folio_put(folio); | |
| 950 | + } | |
| 951 | + } | |
| 952 | + /* | |
| 953 | + * There is a swap entry and a page doesn't exist or isn't charged. | |
| 954 | + * But we cannot move a tail-page in a THP. | |
| 955 | + */ | |
| 956 | + if (ent.val && !ret && (!page || !PageTransCompound(page)) && | |
| 957 | + mem_cgroup_id(mc.from) == lookup_swap_cgroup_id(ent)) { | |
| 958 | + ret = MC_TARGET_SWAP; | |
| 959 | + if (target) | |
| 960 | + target->ent = ent; | |
| 961 | + } | |
| 962 | + return ret; | |
| 963 | +} | |
| 964 | + | |
| 965 | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 966 | +/* | |
| 967 | + * We don't consider PMD mapped swapping or file mapped pages because THP does | |
| 968 | + * not support them for now. | |
| 969 | + * Caller should make sure that pmd_trans_huge(pmd) is true. | |
| 970 | + */ | |
| 971 | +static enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma, | |
| 972 | + unsigned long addr, pmd_t pmd, union mc_target *target) | |
| 973 | +{ | |
| 974 | + struct page *page = NULL; | |
| 975 | + struct folio *folio; | |
| 976 | + enum mc_target_type ret = MC_TARGET_NONE; | |
| 977 | + | |
| 978 | + if (unlikely(is_swap_pmd(pmd))) { | |
| 979 | + VM_BUG_ON(thp_migration_supported() && | |
| 980 | + !is_pmd_migration_entry(pmd)); | |
| 981 | + return ret; | |
| 982 | + } | |
| 983 | + page = pmd_page(pmd); | |
| 984 | + VM_BUG_ON_PAGE(!page || !PageHead(page), page); | |
| 985 | + folio = page_folio(page); | |
| 986 | + if (!(mc.flags & MOVE_ANON)) | |
| 987 | + return ret; | |
| 988 | + if (folio_memcg(folio) == mc.from) { | |
| 989 | + ret = MC_TARGET_PAGE; | |
| 990 | + if (target) { | |
| 991 | + folio_get(folio); | |
| 992 | + if (!folio_trylock(folio)) { | |
| 993 | + folio_put(folio); | |
| 994 | + return MC_TARGET_NONE; | |
| 995 | + } | |
| 996 | + target->folio = folio; | |
| 997 | + } | |
| 998 | + } | |
| 999 | + return ret; | |
| 1000 | +} | |
| 1001 | +#else | |
| 1002 | +static inline enum mc_target_type get_mctgt_type_thp(struct vm_area_struct *vma, | |
| 1003 | + unsigned long addr, pmd_t pmd, union mc_target *target) | |
| 1004 | +{ | |
| 1005 | + return MC_TARGET_NONE; | |
| 1006 | +} | |
| 1007 | +#endif | |
| 1008 | + | |
| 1009 | +static int mem_cgroup_count_precharge_pte_range(pmd_t *pmd, | |
| 1010 | + unsigned long addr, unsigned long end, | |
| 1011 | + struct mm_walk *walk) | |
| 1012 | +{ | |
| 1013 | + struct vm_area_struct *vma = walk->vma; | |
| 1014 | + pte_t *pte; | |
| 1015 | + spinlock_t *ptl; | |
| 1016 | + | |
| 1017 | + ptl = pmd_trans_huge_lock(pmd, vma); | |
| 1018 | + if (ptl) { | |
| 1019 | + /* | |
| 1020 | + * Note their can not be MC_TARGET_DEVICE for now as we do not | |
| 1021 | + * support transparent huge page with MEMORY_DEVICE_PRIVATE but | |
| 1022 | + * this might change. | |
| 1023 | + */ | |
| 1024 | + if (get_mctgt_type_thp(vma, addr, *pmd, NULL) == MC_TARGET_PAGE) | |
| 1025 | + mc.precharge += HPAGE_PMD_NR; | |
| 1026 | + spin_unlock(ptl); | |
| 1027 | + return 0; | |
| 1028 | + } | |
| 1029 | + | |
| 1030 | + pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); | |
| 1031 | + if (!pte) | |
| 1032 | + return 0; | |
| 1033 | + for (; addr != end; pte++, addr += PAGE_SIZE) | |
| 1034 | + if (get_mctgt_type(vma, addr, ptep_get(pte), NULL)) | |
| 1035 | + mc.precharge++; /* increment precharge temporarily */ | |
| 1036 | + pte_unmap_unlock(pte - 1, ptl); | |
| 1037 | + cond_resched(); | |
| 1038 | + | |
| 1039 | + return 0; | |
| 1040 | +} | |
| 1041 | + | |
| 1042 | +static const struct mm_walk_ops precharge_walk_ops = { | |
| 1043 | + .pmd_entry = mem_cgroup_count_precharge_pte_range, | |
| 1044 | + .walk_lock = PGWALK_RDLOCK, | |
| 1045 | +}; | |
| 1046 | + | |
| 1047 | +static unsigned long mem_cgroup_count_precharge(struct mm_struct *mm) | |
| 1048 | +{ | |
| 1049 | + unsigned long precharge; | |
| 1050 | + | |
| 1051 | + mmap_read_lock(mm); | |
| 1052 | + walk_page_range(mm, 0, ULONG_MAX, &precharge_walk_ops, NULL); | |
| 1053 | + mmap_read_unlock(mm); | |
| 1054 | + | |
| 1055 | + precharge = mc.precharge; | |
| 1056 | + mc.precharge = 0; | |
| 1057 | + | |
| 1058 | + return precharge; | |
| 1059 | +} | |
| 1060 | + | |
| 1061 | +static int mem_cgroup_precharge_mc(struct mm_struct *mm) | |
| 1062 | +{ | |
| 1063 | + unsigned long precharge = mem_cgroup_count_precharge(mm); | |
| 1064 | + | |
| 1065 | + VM_BUG_ON(mc.moving_task); | |
| 1066 | + mc.moving_task = current; | |
| 1067 | + return mem_cgroup_do_precharge(precharge); | |
| 1068 | +} | |
| 1069 | + | |
| 1070 | +/* cancels all extra charges on mc.from and mc.to, and wakes up all waiters. */ | |
| 1071 | +static void __mem_cgroup_clear_mc(void) | |
| 1072 | +{ | |
| 1073 | + struct mem_cgroup *from = mc.from; | |
| 1074 | + struct mem_cgroup *to = mc.to; | |
| 1075 | + | |
| 1076 | + /* we must uncharge all the leftover precharges from mc.to */ | |
| 1077 | + if (mc.precharge) { | |
| 1078 | + mem_cgroup_cancel_charge(mc.to, mc.precharge); | |
| 1079 | + mc.precharge = 0; | |
| 1080 | + } | |
| 1081 | + /* | |
| 1082 | + * we didn't uncharge from mc.from at mem_cgroup_move_account(), so | |
| 1083 | + * we must uncharge here. | |
| 1084 | + */ | |
| 1085 | + if (mc.moved_charge) { | |
| 1086 | + mem_cgroup_cancel_charge(mc.from, mc.moved_charge); | |
| 1087 | + mc.moved_charge = 0; | |
| 1088 | + } | |
| 1089 | + /* we must fixup refcnts and charges */ | |
| 1090 | + if (mc.moved_swap) { | |
| 1091 | + /* uncharge swap account from the old cgroup */ | |
| 1092 | + if (!mem_cgroup_is_root(mc.from)) | |
| 1093 | + page_counter_uncharge(&mc.from->memsw, mc.moved_swap); | |
| 1094 | + | |
| 1095 | + mem_cgroup_id_put_many(mc.from, mc.moved_swap); | |
| 1096 | + | |
| 1097 | + /* | |
| 1098 | + * we charged both to->memory and to->memsw, so we | |
| 1099 | + * should uncharge to->memory. | |
| 1100 | + */ | |
| 1101 | + if (!mem_cgroup_is_root(mc.to)) | |
| 1102 | + page_counter_uncharge(&mc.to->memory, mc.moved_swap); | |
| 1103 | + | |
| 1104 | + mc.moved_swap = 0; | |
| 1105 | + } | |
| 1106 | + memcg1_oom_recover(from); | |
| 1107 | + memcg1_oom_recover(to); | |
| 1108 | + wake_up_all(&mc.waitq); | |
| 1109 | +} | |
| 1110 | + | |
| 1111 | +static void mem_cgroup_clear_mc(void) | |
| 1112 | +{ | |
| 1113 | + struct mm_struct *mm = mc.mm; | |
| 1114 | + | |
| 1115 | + /* | |
| 1116 | + * we must clear moving_task before waking up waiters at the end of | |
| 1117 | + * task migration. | |
| 1118 | + */ | |
| 1119 | + mc.moving_task = NULL; | |
| 1120 | + __mem_cgroup_clear_mc(); | |
| 1121 | + spin_lock(&mc.lock); | |
| 1122 | + mc.from = NULL; | |
| 1123 | + mc.to = NULL; | |
| 1124 | + mc.mm = NULL; | |
| 1125 | + spin_unlock(&mc.lock); | |
| 1126 | + | |
| 1127 | + mmput(mm); | |
| 1128 | +} | |
| 1129 | + | |
| 1130 | +int memcg1_can_attach(struct cgroup_taskset *tset) | |
| 1131 | +{ | |
| 1132 | + struct cgroup_subsys_state *css; | |
| 1133 | + struct mem_cgroup *memcg = NULL; /* unneeded init to make gcc happy */ | |
| 1134 | + struct mem_cgroup *from; | |
| 1135 | + struct task_struct *leader, *p; | |
| 1136 | + struct mm_struct *mm; | |
| 1137 | + unsigned long move_flags; | |
| 1138 | + int ret = 0; | |
| 1139 | + | |
| 1140 | + /* charge immigration isn't supported on the default hierarchy */ | |
| 1141 | + if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) | |
| 1142 | + return 0; | |
| 1143 | + | |
| 1144 | + /* | |
| 1145 | + * Multi-process migrations only happen on the default hierarchy | |
| 1146 | + * where charge immigration is not used. Perform charge | |
| 1147 | + * immigration if @tset contains a leader and whine if there are | |
| 1148 | + * multiple. | |
| 1149 | + */ | |
| 1150 | + p = NULL; | |
| 1151 | + cgroup_taskset_for_each_leader(leader, css, tset) { | |
| 1152 | + WARN_ON_ONCE(p); | |
| 1153 | + p = leader; | |
| 1154 | + memcg = mem_cgroup_from_css(css); | |
| 1155 | + } | |
| 1156 | + if (!p) | |
| 1157 | + return 0; | |
| 1158 | + | |
| 1159 | + /* | |
| 1160 | + * We are now committed to this value whatever it is. Changes in this | |
| 1161 | + * tunable will only affect upcoming migrations, not the current one. | |
| 1162 | + * So we need to save it, and keep it going. | |
| 1163 | + */ | |
| 1164 | + move_flags = READ_ONCE(memcg->move_charge_at_immigrate); | |
| 1165 | + if (!move_flags) | |
| 1166 | + return 0; | |
| 1167 | + | |
| 1168 | + from = mem_cgroup_from_task(p); | |
| 1169 | + | |
| 1170 | + VM_BUG_ON(from == memcg); | |
| 1171 | + | |
| 1172 | + mm = get_task_mm(p); | |
| 1173 | + if (!mm) | |
| 1174 | + return 0; | |
| 1175 | + /* We move charges only when we move a owner of the mm */ | |
| 1176 | + if (mm->owner == p) { | |
| 1177 | + VM_BUG_ON(mc.from); | |
| 1178 | + VM_BUG_ON(mc.to); | |
| 1179 | + VM_BUG_ON(mc.precharge); | |
| 1180 | + VM_BUG_ON(mc.moved_charge); | |
| 1181 | + VM_BUG_ON(mc.moved_swap); | |
| 1182 | + | |
| 1183 | + spin_lock(&mc.lock); | |
| 1184 | + mc.mm = mm; | |
| 1185 | + mc.from = from; | |
| 1186 | + mc.to = memcg; | |
| 1187 | + mc.flags = move_flags; | |
| 1188 | + spin_unlock(&mc.lock); | |
| 1189 | + /* We set mc.moving_task later */ | |
| 1190 | + | |
| 1191 | + ret = mem_cgroup_precharge_mc(mm); | |
| 1192 | + if (ret) | |
| 1193 | + mem_cgroup_clear_mc(); | |
| 1194 | + } else { | |
| 1195 | + mmput(mm); | |
| 1196 | + } | |
| 1197 | + return ret; | |
| 1198 | +} | |
| 1199 | + | |
| 1200 | +void memcg1_cancel_attach(struct cgroup_taskset *tset) | |
| 1201 | +{ | |
| 1202 | + if (mc.to) | |
| 1203 | + mem_cgroup_clear_mc(); | |
| 1204 | +} | |
| 1205 | + | |
| 1206 | +static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, | |
| 1207 | + unsigned long addr, unsigned long end, | |
| 1208 | + struct mm_walk *walk) | |
| 1209 | +{ | |
| 1210 | + int ret = 0; | |
| 1211 | + struct vm_area_struct *vma = walk->vma; | |
| 1212 | + pte_t *pte; | |
| 1213 | + spinlock_t *ptl; | |
| 1214 | + enum mc_target_type target_type; | |
| 1215 | + union mc_target target; | |
| 1216 | + struct folio *folio; | |
| 1217 | + | |
| 1218 | + ptl = pmd_trans_huge_lock(pmd, vma); | |
| 1219 | + if (ptl) { | |
| 1220 | + if (mc.precharge < HPAGE_PMD_NR) { | |
| 1221 | + spin_unlock(ptl); | |
| 1222 | + return 0; | |
| 1223 | + } | |
| 1224 | + target_type = get_mctgt_type_thp(vma, addr, *pmd, &target); | |
| 1225 | + if (target_type == MC_TARGET_PAGE) { | |
| 1226 | + folio = target.folio; | |
| 1227 | + if (folio_isolate_lru(folio)) { | |
| 1228 | + if (!mem_cgroup_move_account(folio, true, | |
| 1229 | + mc.from, mc.to)) { | |
| 1230 | + mc.precharge -= HPAGE_PMD_NR; | |
| 1231 | + mc.moved_charge += HPAGE_PMD_NR; | |
| 1232 | + } | |
| 1233 | + folio_putback_lru(folio); | |
| 1234 | + } | |
| 1235 | + folio_unlock(folio); | |
| 1236 | + folio_put(folio); | |
| 1237 | + } else if (target_type == MC_TARGET_DEVICE) { | |
| 1238 | + folio = target.folio; | |
| 1239 | + if (!mem_cgroup_move_account(folio, true, | |
| 1240 | + mc.from, mc.to)) { | |
| 1241 | + mc.precharge -= HPAGE_PMD_NR; | |
| 1242 | + mc.moved_charge += HPAGE_PMD_NR; | |
| 1243 | + } | |
| 1244 | + folio_unlock(folio); | |
| 1245 | + folio_put(folio); | |
| 1246 | + } | |
| 1247 | + spin_unlock(ptl); | |
| 1248 | + return 0; | |
| 1249 | + } | |
| 1250 | + | |
| 1251 | +retry: | |
| 1252 | + pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); | |
| 1253 | + if (!pte) | |
| 1254 | + return 0; | |
| 1255 | + for (; addr != end; addr += PAGE_SIZE) { | |
| 1256 | + pte_t ptent = ptep_get(pte++); | |
| 1257 | + bool device = false; | |
| 1258 | + swp_entry_t ent; | |
| 1259 | + | |
| 1260 | + if (!mc.precharge) | |
| 1261 | + break; | |
| 1262 | + | |
| 1263 | + switch (get_mctgt_type(vma, addr, ptent, &target)) { | |
| 1264 | + case MC_TARGET_DEVICE: | |
| 1265 | + device = true; | |
| 1266 | + fallthrough; | |
| 1267 | + case MC_TARGET_PAGE: | |
| 1268 | + folio = target.folio; | |
| 1269 | + /* | |
| 1270 | + * We can have a part of the split pmd here. Moving it | |
| 1271 | + * can be done but it would be too convoluted so simply | |
| 1272 | + * ignore such a partial THP and keep it in original | |
| 1273 | + * memcg. There should be somebody mapping the head. | |
| 1274 | + */ | |
| 1275 | + if (folio_test_large(folio)) | |
| 1276 | + goto put; | |
| 1277 | + if (!device && !folio_isolate_lru(folio)) | |
| 1278 | + goto put; | |
| 1279 | + if (!mem_cgroup_move_account(folio, false, | |
| 1280 | + mc.from, mc.to)) { | |
| 1281 | + mc.precharge--; | |
| 1282 | + /* we uncharge from mc.from later. */ | |
| 1283 | + mc.moved_charge++; | |
| 1284 | + } | |
| 1285 | + if (!device) | |
| 1286 | + folio_putback_lru(folio); | |
| 1287 | +put: /* get_mctgt_type() gets & locks the page */ | |
| 1288 | + folio_unlock(folio); | |
| 1289 | + folio_put(folio); | |
| 1290 | + break; | |
| 1291 | + case MC_TARGET_SWAP: | |
| 1292 | + ent = target.ent; | |
| 1293 | + if (!mem_cgroup_move_swap_account(ent, mc.from, mc.to)) { | |
| 1294 | + mc.precharge--; | |
| 1295 | + mem_cgroup_id_get_many(mc.to, 1); | |
| 1296 | + /* we fixup other refcnts and charges later. */ | |
| 1297 | + mc.moved_swap++; | |
| 1298 | + } | |
| 1299 | + break; | |
| 1300 | + default: | |
| 1301 | + break; | |
| 1302 | + } | |
| 1303 | + } | |
| 1304 | + pte_unmap_unlock(pte - 1, ptl); | |
| 1305 | + cond_resched(); | |
| 1306 | + | |
| 1307 | + if (addr != end) { | |
| 1308 | + /* | |
| 1309 | + * We have consumed all precharges we got in can_attach(). | |
| 1310 | + * We try charge one by one, but don't do any additional | |
| 1311 | + * charges to mc.to if we have failed in charge once in attach() | |
| 1312 | + * phase. | |
| 1313 | + */ | |
| 1314 | + ret = mem_cgroup_do_precharge(1); | |
| 1315 | + if (!ret) | |
| 1316 | + goto retry; | |
| 1317 | + } | |
| 1318 | + | |
| 1319 | + return ret; | |
| 1320 | +} | |
| 1321 | + | |
| 1322 | +static const struct mm_walk_ops charge_walk_ops = { | |
| 1323 | + .pmd_entry = mem_cgroup_move_charge_pte_range, | |
| 1324 | + .walk_lock = PGWALK_RDLOCK, | |
| 1325 | +}; | |
| 1326 | + | |
| 1327 | +static void mem_cgroup_move_charge(void) | |
| 1328 | +{ | |
| 1329 | + lru_add_drain_all(); | |
| 1330 | + /* | |
| 1331 | + * Signal folio_memcg_lock() to take the memcg's move_lock | |
| 1332 | + * while we're moving its pages to another memcg. Then wait | |
| 1333 | + * for already started RCU-only updates to finish. | |
| 1334 | + */ | |
| 1335 | + atomic_inc(&mc.from->moving_account); | |
| 1336 | + synchronize_rcu(); | |
| 1337 | +retry: | |
| 1338 | + if (unlikely(!mmap_read_trylock(mc.mm))) { | |
| 1339 | + /* | |
| 1340 | + * Someone who are holding the mmap_lock might be waiting in | |
| 1341 | + * waitq. So we cancel all extra charges, wake up all waiters, | |
| 1342 | + * and retry. Because we cancel precharges, we might not be able | |
| 1343 | + * to move enough charges, but moving charge is a best-effort | |
| 1344 | + * feature anyway, so it wouldn't be a big problem. | |
| 1345 | + */ | |
| 1346 | + __mem_cgroup_clear_mc(); | |
| 1347 | + cond_resched(); | |
| 1348 | + goto retry; | |
| 1349 | + } | |
| 1350 | + /* | |
| 1351 | + * When we have consumed all precharges and failed in doing | |
| 1352 | + * additional charge, the page walk just aborts. | |
| 1353 | + */ | |
| 1354 | + walk_page_range(mc.mm, 0, ULONG_MAX, &charge_walk_ops, NULL); | |
| 1355 | + mmap_read_unlock(mc.mm); | |
| 1356 | + atomic_dec(&mc.from->moving_account); | |
| 1357 | +} | |
| 1358 | + | |
| 1359 | +void memcg1_move_task(void) | |
| 1360 | +{ | |
| 1361 | + if (mc.to) { | |
| 1362 | + mem_cgroup_move_charge(); | |
| 1363 | + mem_cgroup_clear_mc(); | |
| 1364 | + } | |
| 1365 | +} | |
| 1366 | + | |
| 1367 | +#else /* !CONFIG_MMU */ | |
| 1368 | +int memcg1_can_attach(struct cgroup_taskset *tset) | |
| 1369 | +{ | |
| 1370 | + return 0; | |
| 1371 | +} | |
| 1372 | +void memcg1_cancel_attach(struct cgroup_taskset *tset) | |
| 1373 | +{ | |
| 1374 | +} | |
| 1375 | +void memcg1_move_task(void) | |
| 1376 | +{ | |
| 1377 | +} | |
| 1378 | +#endif | |
| 1379 | + | |
| 1380 | +static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap) | |
| 1381 | +{ | |
| 1382 | + struct mem_cgroup_threshold_ary *t; | |
| 1383 | + unsigned long usage; | |
| 1384 | + int i; | |
| 1385 | + | |
| 1386 | + rcu_read_lock(); | |
| 1387 | + if (!swap) | |
| 1388 | + t = rcu_dereference(memcg->thresholds.primary); | |
| 1389 | + else | |
| 1390 | + t = rcu_dereference(memcg->memsw_thresholds.primary); | |
| 1391 | + | |
| 1392 | + if (!t) | |
| 1393 | + goto unlock; | |
| 1394 | + | |
| 1395 | + usage = mem_cgroup_usage(memcg, swap); | |
| 1396 | + | |
| 1397 | + /* | |
| 1398 | + * current_threshold points to threshold just below or equal to usage. | |
| 1399 | + * If it's not true, a threshold was crossed after last | |
| 1400 | + * call of __mem_cgroup_threshold(). | |
| 1401 | + */ | |
| 1402 | + i = t->current_threshold; | |
| 1403 | + | |
| 1404 | + /* | |
| 1405 | + * Iterate backward over array of thresholds starting from | |
| 1406 | + * current_threshold and check if a threshold is crossed. | |
| 1407 | + * If none of thresholds below usage is crossed, we read | |
| 1408 | + * only one element of the array here. | |
| 1409 | + */ | |
| 1410 | + for (; i >= 0 && unlikely(t->entries[i].threshold > usage); i--) | |
| 1411 | + eventfd_signal(t->entries[i].eventfd); | |
| 1412 | + | |
| 1413 | + /* i = current_threshold + 1 */ | |
| 1414 | + i++; | |
| 1415 | + | |
| 1416 | + /* | |
| 1417 | + * Iterate forward over array of thresholds starting from | |
| 1418 | + * current_threshold+1 and check if a threshold is crossed. | |
| 1419 | + * If none of thresholds above usage is crossed, we read | |
| 1420 | + * only one element of the array here. | |
| 1421 | + */ | |
| 1422 | + for (; i < t->size && unlikely(t->entries[i].threshold <= usage); i++) | |
| 1423 | + eventfd_signal(t->entries[i].eventfd); | |
| 1424 | + | |
| 1425 | + /* Update current_threshold */ | |
| 1426 | + t->current_threshold = i - 1; | |
| 1427 | +unlock: | |
| 1428 | + rcu_read_unlock(); | |
| 1429 | +} | |
| 1430 | + | |
| 1431 | +static void mem_cgroup_threshold(struct mem_cgroup *memcg) | |
| 1432 | +{ | |
| 1433 | + while (memcg) { | |
| 1434 | + __mem_cgroup_threshold(memcg, false); | |
| 1435 | + if (do_memsw_account()) | |
| 1436 | + __mem_cgroup_threshold(memcg, true); | |
| 1437 | + | |
| 1438 | + memcg = parent_mem_cgroup(memcg); | |
| 1439 | + } | |
| 1440 | +} | |
| 1441 | + | |
| 1442 | +/* | |
| 1443 | + * Check events in order. | |
| 1444 | + * | |
| 1445 | + */ | |
| 1446 | +void memcg1_check_events(struct mem_cgroup *memcg, int nid) | |
| 1447 | +{ | |
| 1448 | + if (IS_ENABLED(CONFIG_PREEMPT_RT)) | |
| 1449 | + return; | |
| 1450 | + | |
| 1451 | + /* threshold event is triggered in finer grain than soft limit */ | |
| 1452 | + if (unlikely(mem_cgroup_event_ratelimit(memcg, | |
| 1453 | + MEM_CGROUP_TARGET_THRESH))) { | |
| 1454 | + bool do_softlimit; | |
| 1455 | + | |
| 1456 | + do_softlimit = mem_cgroup_event_ratelimit(memcg, | |
| 1457 | + MEM_CGROUP_TARGET_SOFTLIMIT); | |
| 1458 | + mem_cgroup_threshold(memcg); | |
| 1459 | + if (unlikely(do_softlimit)) | |
| 1460 | + memcg1_update_tree(memcg, nid); | |
| 1461 | + } | |
| 1462 | +} | |
| 1463 | + | |
| 1464 | +static int compare_thresholds(const void *a, const void *b) | |
| 1465 | +{ | |
| 1466 | + const struct mem_cgroup_threshold *_a = a; | |
| 1467 | + const struct mem_cgroup_threshold *_b = b; | |
| 1468 | + | |
| 1469 | + if (_a->threshold > _b->threshold) | |
| 1470 | + return 1; | |
| 1471 | + | |
| 1472 | + if (_a->threshold < _b->threshold) | |
| 1473 | + return -1; | |
| 1474 | + | |
| 1475 | + return 0; | |
| 1476 | +} | |
| 1477 | + | |
| 1478 | +static int mem_cgroup_oom_notify_cb(struct mem_cgroup *memcg) | |
| 1479 | +{ | |
| 1480 | + struct mem_cgroup_eventfd_list *ev; | |
| 1481 | + | |
| 1482 | + spin_lock(&memcg_oom_lock); | |
| 1483 | + | |
| 1484 | + list_for_each_entry(ev, &memcg->oom_notify, list) | |
| 1485 | + eventfd_signal(ev->eventfd); | |
| 1486 | + | |
| 1487 | + spin_unlock(&memcg_oom_lock); | |
| 1488 | + return 0; | |
| 1489 | +} | |
| 1490 | + | |
| 1491 | +static void mem_cgroup_oom_notify(struct mem_cgroup *memcg) | |
| 1492 | +{ | |
| 1493 | + struct mem_cgroup *iter; | |
| 1494 | + | |
| 1495 | + for_each_mem_cgroup_tree(iter, memcg) | |
| 1496 | + mem_cgroup_oom_notify_cb(iter); | |
| 1497 | +} | |
| 1498 | + | |
| 1499 | +static int __mem_cgroup_usage_register_event(struct mem_cgroup *memcg, | |
| 1500 | + struct eventfd_ctx *eventfd, const char *args, enum res_type type) | |
| 1501 | +{ | |
| 1502 | + struct mem_cgroup_thresholds *thresholds; | |
| 1503 | + struct mem_cgroup_threshold_ary *new; | |
| 1504 | + unsigned long threshold; | |
| 1505 | + unsigned long usage; | |
| 1506 | + int i, size, ret; | |
| 1507 | + | |
| 1508 | + ret = page_counter_memparse(args, "-1", &threshold); | |
| 1509 | + if (ret) | |
| 1510 | + return ret; | |
| 1511 | + | |
| 1512 | + mutex_lock(&memcg->thresholds_lock); | |
| 1513 | + | |
| 1514 | + if (type == _MEM) { | |
| 1515 | + thresholds = &memcg->thresholds; | |
| 1516 | + usage = mem_cgroup_usage(memcg, false); | |
| 1517 | + } else if (type == _MEMSWAP) { | |
| 1518 | + thresholds = &memcg->memsw_thresholds; | |
| 1519 | + usage = mem_cgroup_usage(memcg, true); | |
| 1520 | + } else | |
| 1521 | + BUG(); | |
| 1522 | + | |
| 1523 | + /* Check if a threshold crossed before adding a new one */ | |
| 1524 | + if (thresholds->primary) | |
| 1525 | + __mem_cgroup_threshold(memcg, type == _MEMSWAP); | |
| 1526 | + | |
| 1527 | + size = thresholds->primary ? thresholds->primary->size + 1 : 1; | |
| 1528 | + | |
| 1529 | + /* Allocate memory for new array of thresholds */ | |
| 1530 | + new = kmalloc(struct_size(new, entries, size), GFP_KERNEL); | |
| 1531 | + if (!new) { | |
| 1532 | + ret = -ENOMEM; | |
| 1533 | + goto unlock; | |
| 1534 | + } | |
| 1535 | + new->size = size; | |
| 1536 | + | |
| 1537 | + /* Copy thresholds (if any) to new array */ | |
| 1538 | + if (thresholds->primary) | |
| 1539 | + memcpy(new->entries, thresholds->primary->entries, | |
| 1540 | + flex_array_size(new, entries, size - 1)); | |
| 1541 | + | |
| 1542 | + /* Add new threshold */ | |
| 1543 | + new->entries[size - 1].eventfd = eventfd; | |
| 1544 | + new->entries[size - 1].threshold = threshold; | |
| 1545 | + | |
| 1546 | + /* Sort thresholds. Registering of new threshold isn't time-critical */ | |
| 1547 | + sort(new->entries, size, sizeof(*new->entries), | |
| 1548 | + compare_thresholds, NULL); | |
| 1549 | + | |
| 1550 | + /* Find current threshold */ | |
| 1551 | + new->current_threshold = -1; | |
| 1552 | + for (i = 0; i < size; i++) { | |
| 1553 | + if (new->entries[i].threshold <= usage) { | |
| 1554 | + /* | |
| 1555 | + * new->current_threshold will not be used until | |
| 1556 | + * rcu_assign_pointer(), so it's safe to increment | |
| 1557 | + * it here. | |
| 1558 | + */ | |
| 1559 | + ++new->current_threshold; | |
| 1560 | + } else | |
| 1561 | + break; | |
| 1562 | + } | |
| 1563 | + | |
| 1564 | + /* Free old spare buffer and save old primary buffer as spare */ | |
| 1565 | + kfree(thresholds->spare); | |
| 1566 | + thresholds->spare = thresholds->primary; | |
| 1567 | + | |
| 1568 | + rcu_assign_pointer(thresholds->primary, new); | |
| 1569 | + | |
| 1570 | + /* To be sure that nobody uses thresholds */ | |
| 1571 | + synchronize_rcu(); | |
| 1572 | + | |
| 1573 | +unlock: | |
| 1574 | + mutex_unlock(&memcg->thresholds_lock); | |
| 1575 | + | |
| 1576 | + return ret; | |
| 1577 | +} | |
| 1578 | + | |
| 1579 | +static int mem_cgroup_usage_register_event(struct mem_cgroup *memcg, | |
| 1580 | + struct eventfd_ctx *eventfd, const char *args) | |
| 1581 | +{ | |
| 1582 | + return __mem_cgroup_usage_register_event(memcg, eventfd, args, _MEM); | |
| 1583 | +} | |
| 1584 | + | |
| 1585 | +static int memsw_cgroup_usage_register_event(struct mem_cgroup *memcg, | |
| 1586 | + struct eventfd_ctx *eventfd, const char *args) | |
| 1587 | +{ | |
| 1588 | + return __mem_cgroup_usage_register_event(memcg, eventfd, args, _MEMSWAP); | |
| 1589 | +} | |
| 1590 | + | |
| 1591 | +static void __mem_cgroup_usage_unregister_event(struct mem_cgroup *memcg, | |
| 1592 | + struct eventfd_ctx *eventfd, enum res_type type) | |
| 1593 | +{ | |
| 1594 | + struct mem_cgroup_thresholds *thresholds; | |
| 1595 | + struct mem_cgroup_threshold_ary *new; | |
| 1596 | + unsigned long usage; | |
| 1597 | + int i, j, size, entries; | |
| 1598 | + | |
| 1599 | + mutex_lock(&memcg->thresholds_lock); | |
| 1600 | + | |
| 1601 | + if (type == _MEM) { | |
| 1602 | + thresholds = &memcg->thresholds; | |
| 1603 | + usage = mem_cgroup_usage(memcg, false); | |
| 1604 | + } else if (type == _MEMSWAP) { | |
| 1605 | + thresholds = &memcg->memsw_thresholds; | |
| 1606 | + usage = mem_cgroup_usage(memcg, true); | |
| 1607 | + } else | |
| 1608 | + BUG(); | |
| 1609 | + | |
| 1610 | + if (!thresholds->primary) | |
| 1611 | + goto unlock; | |
| 1612 | + | |
| 1613 | + /* Check if a threshold crossed before removing */ | |
| 1614 | + __mem_cgroup_threshold(memcg, type == _MEMSWAP); | |
| 1615 | + | |
| 1616 | + /* Calculate new number of threshold */ | |
| 1617 | + size = entries = 0; | |
| 1618 | + for (i = 0; i < thresholds->primary->size; i++) { | |
| 1619 | + if (thresholds->primary->entries[i].eventfd != eventfd) | |
| 1620 | + size++; | |
| 1621 | + else | |
| 1622 | + entries++; | |
| 1623 | + } | |
| 1624 | + | |
| 1625 | + new = thresholds->spare; | |
| 1626 | + | |
| 1627 | + /* If no items related to eventfd have been cleared, nothing to do */ | |
| 1628 | + if (!entries) | |
| 1629 | + goto unlock; | |
| 1630 | + | |
| 1631 | + /* Set thresholds array to NULL if we don't have thresholds */ | |
| 1632 | + if (!size) { | |
| 1633 | + kfree(new); | |
| 1634 | + new = NULL; | |
| 1635 | + goto swap_buffers; | |
| 1636 | + } | |
| 1637 | + | |
| 1638 | + new->size = size; | |
| 1639 | + | |
| 1640 | + /* Copy thresholds and find current threshold */ | |
| 1641 | + new->current_threshold = -1; | |
| 1642 | + for (i = 0, j = 0; i < thresholds->primary->size; i++) { | |
| 1643 | + if (thresholds->primary->entries[i].eventfd == eventfd) | |
| 1644 | + continue; | |
| 1645 | + | |
| 1646 | + new->entries[j] = thresholds->primary->entries[i]; | |
| 1647 | + if (new->entries[j].threshold <= usage) { | |
| 1648 | + /* | |
| 1649 | + * new->current_threshold will not be used | |
| 1650 | + * until rcu_assign_pointer(), so it's safe to increment | |
| 1651 | + * it here. | |
| 1652 | + */ | |
| 1653 | + ++new->current_threshold; | |
| 1654 | + } | |
| 1655 | + j++; | |
| 1656 | + } | |
| 1657 | + | |
| 1658 | +swap_buffers: | |
| 1659 | + /* Swap primary and spare array */ | |
| 1660 | + thresholds->spare = thresholds->primary; | |
| 1661 | + | |
| 1662 | + rcu_assign_pointer(thresholds->primary, new); | |
| 1663 | + | |
| 1664 | + /* To be sure that nobody uses thresholds */ | |
| 1665 | + synchronize_rcu(); | |
| 1666 | + | |
| 1667 | + /* If all events are unregistered, free the spare array */ | |
| 1668 | + if (!new) { | |
| 1669 | + kfree(thresholds->spare); | |
| 1670 | + thresholds->spare = NULL; | |
| 1671 | + } | |
| 1672 | +unlock: | |
| 1673 | + mutex_unlock(&memcg->thresholds_lock); | |
| 1674 | +} | |
| 1675 | + | |
| 1676 | +static void mem_cgroup_usage_unregister_event(struct mem_cgroup *memcg, | |
| 1677 | + struct eventfd_ctx *eventfd) | |
| 1678 | +{ | |
| 1679 | + return __mem_cgroup_usage_unregister_event(memcg, eventfd, _MEM); | |
| 1680 | +} | |
| 1681 | + | |
| 1682 | +static void memsw_cgroup_usage_unregister_event(struct mem_cgroup *memcg, | |
| 1683 | + struct eventfd_ctx *eventfd) | |
| 1684 | +{ | |
| 1685 | + return __mem_cgroup_usage_unregister_event(memcg, eventfd, _MEMSWAP); | |
| 1686 | +} | |
| 1687 | + | |
| 1688 | +static int mem_cgroup_oom_register_event(struct mem_cgroup *memcg, | |
| 1689 | + struct eventfd_ctx *eventfd, const char *args) | |
| 1690 | +{ | |
| 1691 | + struct mem_cgroup_eventfd_list *event; | |
| 1692 | + | |
| 1693 | + event = kmalloc(sizeof(*event), GFP_KERNEL); | |
| 1694 | + if (!event) | |
| 1695 | + return -ENOMEM; | |
| 1696 | + | |
| 1697 | + spin_lock(&memcg_oom_lock); | |
| 1698 | + | |
| 1699 | + event->eventfd = eventfd; | |
| 1700 | + list_add(&event->list, &memcg->oom_notify); | |
| 1701 | + | |
| 1702 | + /* already in OOM ? */ | |
| 1703 | + if (memcg->under_oom) | |
| 1704 | + eventfd_signal(eventfd); | |
| 1705 | + spin_unlock(&memcg_oom_lock); | |
| 1706 | + | |
| 1707 | + return 0; | |
| 1708 | +} | |
| 1709 | + | |
| 1710 | +static void mem_cgroup_oom_unregister_event(struct mem_cgroup *memcg, | |
| 1711 | + struct eventfd_ctx *eventfd) | |
| 1712 | +{ | |
| 1713 | + struct mem_cgroup_eventfd_list *ev, *tmp; | |
| 1714 | + | |
| 1715 | + spin_lock(&memcg_oom_lock); | |
| 1716 | + | |
| 1717 | + list_for_each_entry_safe(ev, tmp, &memcg->oom_notify, list) { | |
| 1718 | + if (ev->eventfd == eventfd) { | |
| 1719 | + list_del(&ev->list); | |
| 1720 | + kfree(ev); | |
| 1721 | + } | |
| 1722 | + } | |
| 1723 | + | |
| 1724 | + spin_unlock(&memcg_oom_lock); | |
| 1725 | +} | |
| 1726 | + | |
| 1727 | +/* | |
| 1728 | + * DO NOT USE IN NEW FILES. | |
| 1729 | + * | |
| 1730 | + * "cgroup.event_control" implementation. | |
| 1731 | + * | |
| 1732 | + * This is way over-engineered. It tries to support fully configurable | |
| 1733 | + * events for each user. Such level of flexibility is completely | |
| 1734 | + * unnecessary especially in the light of the planned unified hierarchy. | |
| 1735 | + * | |
| 1736 | + * Please deprecate this and replace with something simpler if at all | |
| 1737 | + * possible. | |
| 1738 | + */ | |
| 1739 | + | |
| 1740 | +/* | |
| 1741 | + * Unregister event and free resources. | |
| 1742 | + * | |
| 1743 | + * Gets called from workqueue. | |
| 1744 | + */ | |
| 1745 | +static void memcg_event_remove(struct work_struct *work) | |
| 1746 | +{ | |
| 1747 | + struct mem_cgroup_event *event = | |
| 1748 | + container_of(work, struct mem_cgroup_event, remove); | |
| 1749 | + struct mem_cgroup *memcg = event->memcg; | |
| 1750 | + | |
| 1751 | + remove_wait_queue(event->wqh, &event->wait); | |
| 1752 | + | |
| 1753 | + event->unregister_event(memcg, event->eventfd); | |
| 1754 | + | |
| 1755 | + /* Notify userspace the event is going away. */ | |
| 1756 | + eventfd_signal(event->eventfd); | |
| 1757 | + | |
| 1758 | + eventfd_ctx_put(event->eventfd); | |
| 1759 | + kfree(event); | |
| 1760 | + css_put(&memcg->css); | |
| 1761 | +} | |
| 1762 | + | |
| 1763 | +/* | |
| 1764 | + * Gets called on EPOLLHUP on eventfd when user closes it. | |
| 1765 | + * | |
| 1766 | + * Called with wqh->lock held and interrupts disabled. | |
| 1767 | + */ | |
| 1768 | +static int memcg_event_wake(wait_queue_entry_t *wait, unsigned mode, | |
| 1769 | + int sync, void *key) | |
| 1770 | +{ | |
| 1771 | + struct mem_cgroup_event *event = | |
| 1772 | + container_of(wait, struct mem_cgroup_event, wait); | |
| 1773 | + struct mem_cgroup *memcg = event->memcg; | |
| 1774 | + __poll_t flags = key_to_poll(key); | |
| 1775 | + | |
| 1776 | + if (flags & EPOLLHUP) { | |
| 1777 | + /* | |
| 1778 | + * If the event has been detached at cgroup removal, we | |
| 1779 | + * can simply return knowing the other side will cleanup | |
| 1780 | + * for us. | |
| 1781 | + * | |
| 1782 | + * We can't race against event freeing since the other | |
| 1783 | + * side will require wqh->lock via remove_wait_queue(), | |
| 1784 | + * which we hold. | |
| 1785 | + */ | |
| 1786 | + spin_lock(&memcg->event_list_lock); | |
| 1787 | + if (!list_empty(&event->list)) { | |
| 1788 | + list_del_init(&event->list); | |
| 1789 | + /* | |
| 1790 | + * We are in atomic context, but cgroup_event_remove() | |
| 1791 | + * may sleep, so we have to call it in workqueue. | |
| 1792 | + */ | |
| 1793 | + schedule_work(&event->remove); | |
| 1794 | + } | |
| 1795 | + spin_unlock(&memcg->event_list_lock); | |
| 1796 | + } | |
| 1797 | + | |
| 1798 | + return 0; | |
| 1799 | +} | |
| 1800 | + | |
| 1801 | +static void memcg_event_ptable_queue_proc(struct file *file, | |
| 1802 | + wait_queue_head_t *wqh, poll_table *pt) | |
| 1803 | +{ | |
| 1804 | + struct mem_cgroup_event *event = | |
| 1805 | + container_of(pt, struct mem_cgroup_event, pt); | |
| 1806 | + | |
| 1807 | + event->wqh = wqh; | |
| 1808 | + add_wait_queue(wqh, &event->wait); | |
| 1809 | +} | |
| 1810 | + | |
| 1811 | +/* | |
| 1812 | + * DO NOT USE IN NEW FILES. | |
| 1813 | + * | |
| 1814 | + * Parse input and register new cgroup event handler. | |
| 1815 | + * | |
| 1816 | + * Input must be in format '<event_fd> <control_fd> <args>'. | |
| 1817 | + * Interpretation of args is defined by control file implementation. | |
| 1818 | + */ | |
| 1819 | +static ssize_t memcg_write_event_control(struct kernfs_open_file *of, | |
| 1820 | + char *buf, size_t nbytes, loff_t off) | |
| 1821 | +{ | |
| 1822 | + struct cgroup_subsys_state *css = of_css(of); | |
| 1823 | + struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 1824 | + struct mem_cgroup_event *event; | |
| 1825 | + struct cgroup_subsys_state *cfile_css; | |
| 1826 | + unsigned int efd, cfd; | |
| 1827 | + struct fd efile; | |
| 1828 | + struct fd cfile; | |
| 1829 | + struct dentry *cdentry; | |
| 1830 | + const char *name; | |
| 1831 | + char *endp; | |
| 1832 | + int ret; | |
| 1833 | + | |
| 1834 | + if (IS_ENABLED(CONFIG_PREEMPT_RT)) | |
| 1835 | + return -EOPNOTSUPP; | |
| 1836 | + | |
| 1837 | + buf = strstrip(buf); | |
| 1838 | + | |
| 1839 | + efd = simple_strtoul(buf, &endp, 10); | |
| 1840 | + if (*endp != ' ') | |
| 1841 | + return -EINVAL; | |
| 1842 | + buf = endp + 1; | |
| 1843 | + | |
| 1844 | + cfd = simple_strtoul(buf, &endp, 10); | |
| 1845 | + if ((*endp != ' ') && (*endp != '\0')) | |
| 1846 | + return -EINVAL; | |
| 1847 | + buf = endp + 1; | |
| 1848 | + | |
| 1849 | + event = kzalloc(sizeof(*event), GFP_KERNEL); | |
| 1850 | + if (!event) | |
| 1851 | + return -ENOMEM; | |
| 1852 | + | |
| 1853 | + event->memcg = memcg; | |
| 1854 | + INIT_LIST_HEAD(&event->list); | |
| 1855 | + init_poll_funcptr(&event->pt, memcg_event_ptable_queue_proc); | |
| 1856 | + init_waitqueue_func_entry(&event->wait, memcg_event_wake); | |
| 1857 | + INIT_WORK(&event->remove, memcg_event_remove); | |
| 1858 | + | |
| 1859 | + efile = fdget(efd); | |
| 1860 | + if (!efile.file) { | |
| 1861 | + ret = -EBADF; | |
| 1862 | + goto out_kfree; | |
| 1863 | + } | |
| 1864 | + | |
| 1865 | + event->eventfd = eventfd_ctx_fileget(efile.file); | |
| 1866 | + if (IS_ERR(event->eventfd)) { | |
| 1867 | + ret = PTR_ERR(event->eventfd); | |
| 1868 | + goto out_put_efile; | |
| 1869 | + } | |
| 1870 | + | |
| 1871 | + cfile = fdget(cfd); | |
| 1872 | + if (!cfile.file) { | |
| 1873 | + ret = -EBADF; | |
| 1874 | + goto out_put_eventfd; | |
| 1875 | + } | |
| 1876 | + | |
| 1877 | + /* the process need read permission on control file */ | |
| 1878 | + /* AV: shouldn't we check that it's been opened for read instead? */ | |
| 1879 | + ret = file_permission(cfile.file, MAY_READ); | |
| 1880 | + if (ret < 0) | |
| 1881 | + goto out_put_cfile; | |
| 1882 | + | |
| 1883 | + /* | |
| 1884 | + * The control file must be a regular cgroup1 file. As a regular cgroup | |
| 1885 | + * file can't be renamed, it's safe to access its name afterwards. | |
| 1886 | + */ | |
| 1887 | + cdentry = cfile.file->f_path.dentry; | |
| 1888 | + if (cdentry->d_sb->s_type != &cgroup_fs_type || !d_is_reg(cdentry)) { | |
| 1889 | + ret = -EINVAL; | |
| 1890 | + goto out_put_cfile; | |
| 1891 | + } | |
| 1892 | + | |
| 1893 | + /* | |
| 1894 | + * Determine the event callbacks and set them in @event. This used | |
| 1895 | + * to be done via struct cftype but cgroup core no longer knows | |
| 1896 | + * about these events. The following is crude but the whole thing | |
| 1897 | + * is for compatibility anyway. | |
| 1898 | + * | |
| 1899 | + * DO NOT ADD NEW FILES. | |
| 1900 | + */ | |
| 1901 | + name = cdentry->d_name.name; | |
| 1902 | + | |
| 1903 | + if (!strcmp(name, "memory.usage_in_bytes")) { | |
| 1904 | + event->register_event = mem_cgroup_usage_register_event; | |
| 1905 | + event->unregister_event = mem_cgroup_usage_unregister_event; | |
| 1906 | + } else if (!strcmp(name, "memory.oom_control")) { | |
| 1907 | + event->register_event = mem_cgroup_oom_register_event; | |
| 1908 | + event->unregister_event = mem_cgroup_oom_unregister_event; | |
| 1909 | + } else if (!strcmp(name, "memory.pressure_level")) { | |
| 1910 | + event->register_event = vmpressure_register_event; | |
| 1911 | + event->unregister_event = vmpressure_unregister_event; | |
| 1912 | + } else if (!strcmp(name, "memory.memsw.usage_in_bytes")) { | |
| 1913 | + event->register_event = memsw_cgroup_usage_register_event; | |
| 1914 | + event->unregister_event = memsw_cgroup_usage_unregister_event; | |
| 1915 | + } else { | |
| 1916 | + ret = -EINVAL; | |
| 1917 | + goto out_put_cfile; | |
| 1918 | + } | |
| 1919 | + | |
| 1920 | + /* | |
| 1921 | + * Verify @cfile should belong to @css. Also, remaining events are | |
| 1922 | + * automatically removed on cgroup destruction but the removal is | |
| 1923 | + * asynchronous, so take an extra ref on @css. | |
| 1924 | + */ | |
| 1925 | + cfile_css = css_tryget_online_from_dir(cdentry->d_parent, | |
| 1926 | + &memory_cgrp_subsys); | |
| 1927 | + ret = -EINVAL; | |
| 1928 | + if (IS_ERR(cfile_css)) | |
| 1929 | + goto out_put_cfile; | |
| 1930 | + if (cfile_css != css) { | |
| 1931 | + css_put(cfile_css); | |
| 1932 | + goto out_put_cfile; | |
| 1933 | + } | |
| 1934 | + | |
| 1935 | + ret = event->register_event(memcg, event->eventfd, buf); | |
| 1936 | + if (ret) | |
| 1937 | + goto out_put_css; | |
| 1938 | + | |
| 1939 | + vfs_poll(efile.file, &event->pt); | |
| 1940 | + | |
| 1941 | + spin_lock_irq(&memcg->event_list_lock); | |
| 1942 | + list_add(&event->list, &memcg->event_list); | |
| 1943 | + spin_unlock_irq(&memcg->event_list_lock); | |
| 1944 | + | |
| 1945 | + fdput(cfile); | |
| 1946 | + fdput(efile); | |
| 1947 | + | |
| 1948 | + return nbytes; | |
| 1949 | + | |
| 1950 | +out_put_css: | |
| 1951 | + css_put(css); | |
| 1952 | +out_put_cfile: | |
| 1953 | + fdput(cfile); | |
| 1954 | +out_put_eventfd: | |
| 1955 | + eventfd_ctx_put(event->eventfd); | |
| 1956 | +out_put_efile: | |
| 1957 | + fdput(efile); | |
| 1958 | +out_kfree: | |
| 1959 | + kfree(event); | |
| 1960 | + | |
| 1961 | + return ret; | |
| 1962 | +} | |
| 1963 | + | |
| 1964 | +void memcg1_memcg_init(struct mem_cgroup *memcg) | |
| 1965 | +{ | |
| 1966 | + INIT_LIST_HEAD(&memcg->oom_notify); | |
| 1967 | + mutex_init(&memcg->thresholds_lock); | |
| 1968 | + spin_lock_init(&memcg->move_lock); | |
| 1969 | + INIT_LIST_HEAD(&memcg->event_list); | |
| 1970 | + spin_lock_init(&memcg->event_list_lock); | |
| 1971 | +} | |
| 1972 | + | |
| 1973 | +void memcg1_css_offline(struct mem_cgroup *memcg) | |
| 1974 | +{ | |
| 1975 | + struct mem_cgroup_event *event, *tmp; | |
| 1976 | + | |
| 1977 | + /* | |
| 1978 | + * Unregister events and notify userspace. | |
| 1979 | + * Notify userspace about cgroup removing only after rmdir of cgroup | |
| 1980 | + * directory to avoid race between userspace and kernelspace. | |
| 1981 | + */ | |
| 1982 | + spin_lock_irq(&memcg->event_list_lock); | |
| 1983 | + list_for_each_entry_safe(event, tmp, &memcg->event_list, list) { | |
| 1984 | + list_del_init(&event->list); | |
| 1985 | + schedule_work(&event->remove); | |
| 1986 | + } | |
| 1987 | + spin_unlock_irq(&memcg->event_list_lock); | |
| 1988 | +} | |
| 1989 | + | |
| 1990 | +/* | |
| 1991 | + * Check OOM-Killer is already running under our hierarchy. | |
| 1992 | + * If someone is running, return false. | |
| 1993 | + */ | |
| 1994 | +static bool mem_cgroup_oom_trylock(struct mem_cgroup *memcg) | |
| 1995 | +{ | |
| 1996 | + struct mem_cgroup *iter, *failed = NULL; | |
| 1997 | + | |
| 1998 | + spin_lock(&memcg_oom_lock); | |
| 1999 | + | |
| 2000 | + for_each_mem_cgroup_tree(iter, memcg) { | |
| 2001 | + if (iter->oom_lock) { | |
| 2002 | + /* | |
| 2003 | + * this subtree of our hierarchy is already locked | |
| 2004 | + * so we cannot give a lock. | |
| 2005 | + */ | |
| 2006 | + failed = iter; | |
| 2007 | + mem_cgroup_iter_break(memcg, iter); | |
| 2008 | + break; | |
| 2009 | + } else | |
| 2010 | + iter->oom_lock = true; | |
| 2011 | + } | |
| 2012 | + | |
| 2013 | + if (failed) { | |
| 2014 | + /* | |
| 2015 | + * OK, we failed to lock the whole subtree so we have | |
| 2016 | + * to clean up what we set up to the failing subtree | |
| 2017 | + */ | |
| 2018 | + for_each_mem_cgroup_tree(iter, memcg) { | |
| 2019 | + if (iter == failed) { | |
| 2020 | + mem_cgroup_iter_break(memcg, iter); | |
| 2021 | + break; | |
| 2022 | + } | |
| 2023 | + iter->oom_lock = false; | |
| 2024 | + } | |
| 2025 | + } else | |
| 2026 | + mutex_acquire(&memcg_oom_lock_dep_map, 0, 1, _RET_IP_); | |
| 2027 | + | |
| 2028 | + spin_unlock(&memcg_oom_lock); | |
| 2029 | + | |
| 2030 | + return !failed; | |
| 2031 | +} | |
| 2032 | + | |
| 2033 | +static void mem_cgroup_oom_unlock(struct mem_cgroup *memcg) | |
| 2034 | +{ | |
| 2035 | + struct mem_cgroup *iter; | |
| 2036 | + | |
| 2037 | + spin_lock(&memcg_oom_lock); | |
| 2038 | + mutex_release(&memcg_oom_lock_dep_map, _RET_IP_); | |
| 2039 | + for_each_mem_cgroup_tree(iter, memcg) | |
| 2040 | + iter->oom_lock = false; | |
| 2041 | + spin_unlock(&memcg_oom_lock); | |
| 2042 | +} | |
| 2043 | + | |
| 2044 | +static void mem_cgroup_mark_under_oom(struct mem_cgroup *memcg) | |
| 2045 | +{ | |
| 2046 | + struct mem_cgroup *iter; | |
| 2047 | + | |
| 2048 | + spin_lock(&memcg_oom_lock); | |
| 2049 | + for_each_mem_cgroup_tree(iter, memcg) | |
| 2050 | + iter->under_oom++; | |
| 2051 | + spin_unlock(&memcg_oom_lock); | |
| 2052 | +} | |
| 2053 | + | |
| 2054 | +static void mem_cgroup_unmark_under_oom(struct mem_cgroup *memcg) | |
| 2055 | +{ | |
| 2056 | + struct mem_cgroup *iter; | |
| 2057 | + | |
| 2058 | + /* | |
| 2059 | + * Be careful about under_oom underflows because a child memcg | |
| 2060 | + * could have been added after mem_cgroup_mark_under_oom. | |
| 2061 | + */ | |
| 2062 | + spin_lock(&memcg_oom_lock); | |
| 2063 | + for_each_mem_cgroup_tree(iter, memcg) | |
| 2064 | + if (iter->under_oom > 0) | |
| 2065 | + iter->under_oom--; | |
| 2066 | + spin_unlock(&memcg_oom_lock); | |
| 2067 | +} | |
| 2068 | + | |
| 2069 | +static DECLARE_WAIT_QUEUE_HEAD(memcg_oom_waitq); | |
| 2070 | + | |
| 2071 | +struct oom_wait_info { | |
| 2072 | + struct mem_cgroup *memcg; | |
| 2073 | + wait_queue_entry_t wait; | |
| 2074 | +}; | |
| 2075 | + | |
| 2076 | +static int memcg_oom_wake_function(wait_queue_entry_t *wait, | |
| 2077 | + unsigned mode, int sync, void *arg) | |
| 2078 | +{ | |
| 2079 | + struct mem_cgroup *wake_memcg = (struct mem_cgroup *)arg; | |
| 2080 | + struct mem_cgroup *oom_wait_memcg; | |
| 2081 | + struct oom_wait_info *oom_wait_info; | |
| 2082 | + | |
| 2083 | + oom_wait_info = container_of(wait, struct oom_wait_info, wait); | |
| 2084 | + oom_wait_memcg = oom_wait_info->memcg; | |
| 2085 | + | |
| 2086 | + if (!mem_cgroup_is_descendant(wake_memcg, oom_wait_memcg) && | |
| 2087 | + !mem_cgroup_is_descendant(oom_wait_memcg, wake_memcg)) | |
| 2088 | + return 0; | |
| 2089 | + return autoremove_wake_function(wait, mode, sync, arg); | |
| 2090 | +} | |
| 2091 | + | |
| 2092 | +void memcg1_oom_recover(struct mem_cgroup *memcg) | |
| 2093 | +{ | |
| 2094 | + /* | |
| 2095 | + * For the following lockless ->under_oom test, the only required | |
| 2096 | + * guarantee is that it must see the state asserted by an OOM when | |
| 2097 | + * this function is called as a result of userland actions | |
| 2098 | + * triggered by the notification of the OOM. This is trivially | |
| 2099 | + * achieved by invoking mem_cgroup_mark_under_oom() before | |
| 2100 | + * triggering notification. | |
| 2101 | + */ | |
| 2102 | + if (memcg && memcg->under_oom) | |
| 2103 | + __wake_up(&memcg_oom_waitq, TASK_NORMAL, 0, memcg); | |
| 2104 | +} | |
| 2105 | + | |
| 2106 | +/** | |
| 2107 | + * mem_cgroup_oom_synchronize - complete memcg OOM handling | |
| 2108 | + * @handle: actually kill/wait or just clean up the OOM state | |
| 2109 | + * | |
| 2110 | + * This has to be called at the end of a page fault if the memcg OOM | |
| 2111 | + * handler was enabled. | |
| 2112 | + * | |
| 2113 | + * Memcg supports userspace OOM handling where failed allocations must | |
| 2114 | + * sleep on a waitqueue until the userspace task resolves the | |
| 2115 | + * situation. Sleeping directly in the charge context with all kinds | |
| 2116 | + * of locks held is not a good idea, instead we remember an OOM state | |
| 2117 | + * in the task and mem_cgroup_oom_synchronize() has to be called at | |
| 2118 | + * the end of the page fault to complete the OOM handling. | |
| 2119 | + * | |
| 2120 | + * Returns %true if an ongoing memcg OOM situation was detected and | |
| 2121 | + * completed, %false otherwise. | |
| 2122 | + */ | |
| 2123 | +bool mem_cgroup_oom_synchronize(bool handle) | |
| 2124 | +{ | |
| 2125 | + struct mem_cgroup *memcg = current->memcg_in_oom; | |
| 2126 | + struct oom_wait_info owait; | |
| 2127 | + bool locked; | |
| 2128 | + | |
| 2129 | + /* OOM is global, do not handle */ | |
| 2130 | + if (!memcg) | |
| 2131 | + return false; | |
| 2132 | + | |
| 2133 | + if (!handle) | |
| 2134 | + goto cleanup; | |
| 2135 | + | |
| 2136 | + owait.memcg = memcg; | |
| 2137 | + owait.wait.flags = 0; | |
| 2138 | + owait.wait.func = memcg_oom_wake_function; | |
| 2139 | + owait.wait.private = current; | |
| 2140 | + INIT_LIST_HEAD(&owait.wait.entry); | |
| 2141 | + | |
| 2142 | + prepare_to_wait(&memcg_oom_waitq, &owait.wait, TASK_KILLABLE); | |
| 2143 | + mem_cgroup_mark_under_oom(memcg); | |
| 2144 | + | |
| 2145 | + locked = mem_cgroup_oom_trylock(memcg); | |
| 2146 | + | |
| 2147 | + if (locked) | |
| 2148 | + mem_cgroup_oom_notify(memcg); | |
| 2149 | + | |
| 2150 | + schedule(); | |
| 2151 | + mem_cgroup_unmark_under_oom(memcg); | |
| 2152 | + finish_wait(&memcg_oom_waitq, &owait.wait); | |
| 2153 | + | |
| 2154 | + if (locked) | |
| 2155 | + mem_cgroup_oom_unlock(memcg); | |
| 2156 | +cleanup: | |
| 2157 | + current->memcg_in_oom = NULL; | |
| 2158 | + css_put(&memcg->css); | |
| 2159 | + return true; | |
| 2160 | +} | |
| 2161 | + | |
| 2162 | + | |
| 2163 | +bool memcg1_oom_prepare(struct mem_cgroup *memcg, bool *locked) | |
| 2164 | +{ | |
| 2165 | + /* | |
| 2166 | + * We are in the middle of the charge context here, so we | |
| 2167 | + * don't want to block when potentially sitting on a callstack | |
| 2168 | + * that holds all kinds of filesystem and mm locks. | |
| 2169 | + * | |
| 2170 | + * cgroup1 allows disabling the OOM killer and waiting for outside | |
| 2171 | + * handling until the charge can succeed; remember the context and put | |
| 2172 | + * the task to sleep at the end of the page fault when all locks are | |
| 2173 | + * released. | |
| 2174 | + * | |
| 2175 | + * On the other hand, in-kernel OOM killer allows for an async victim | |
| 2176 | + * memory reclaim (oom_reaper) and that means that we are not solely | |
| 2177 | + * relying on the oom victim to make a forward progress and we can | |
| 2178 | + * invoke the oom killer here. | |
| 2179 | + * | |
| 2180 | + * Please note that mem_cgroup_out_of_memory might fail to find a | |
| 2181 | + * victim and then we have to bail out from the charge path. | |
| 2182 | + */ | |
| 2183 | + if (READ_ONCE(memcg->oom_kill_disable)) { | |
| 2184 | + if (current->in_user_fault) { | |
| 2185 | + css_get(&memcg->css); | |
| 2186 | + current->memcg_in_oom = memcg; | |
| 2187 | + } | |
| 2188 | + return false; | |
| 2189 | + } | |
| 2190 | + | |
| 2191 | + mem_cgroup_mark_under_oom(memcg); | |
| 2192 | + | |
| 2193 | + *locked = mem_cgroup_oom_trylock(memcg); | |
| 2194 | + | |
| 2195 | + if (*locked) | |
| 2196 | + mem_cgroup_oom_notify(memcg); | |
| 2197 | + | |
| 2198 | + mem_cgroup_unmark_under_oom(memcg); | |
| 2199 | + | |
| 2200 | + return true; | |
| 2201 | +} | |
| 2202 | + | |
| 2203 | +void memcg1_oom_finish(struct mem_cgroup *memcg, bool locked) | |
| 2204 | +{ | |
| 2205 | + if (locked) | |
| 2206 | + mem_cgroup_oom_unlock(memcg); | |
| 2207 | +} | |
| 2208 | + | |
| 2209 | +static DEFINE_MUTEX(memcg_max_mutex); | |
| 2210 | + | |
| 2211 | +static int mem_cgroup_resize_max(struct mem_cgroup *memcg, | |
| 2212 | + unsigned long max, bool memsw) | |
| 2213 | +{ | |
| 2214 | + bool enlarge = false; | |
| 2215 | + bool drained = false; | |
| 2216 | + int ret; | |
| 2217 | + bool limits_invariant; | |
| 2218 | + struct page_counter *counter = memsw ? &memcg->memsw : &memcg->memory; | |
| 2219 | + | |
| 2220 | + do { | |
| 2221 | + if (signal_pending(current)) { | |
| 2222 | + ret = -EINTR; | |
| 2223 | + break; | |
| 2224 | + } | |
| 2225 | + | |
| 2226 | + mutex_lock(&memcg_max_mutex); | |
| 2227 | + /* | |
| 2228 | + * Make sure that the new limit (memsw or memory limit) doesn't | |
| 2229 | + * break our basic invariant rule memory.max <= memsw.max. | |
| 2230 | + */ | |
| 2231 | + limits_invariant = memsw ? max >= READ_ONCE(memcg->memory.max) : | |
| 2232 | + max <= memcg->memsw.max; | |
| 2233 | + if (!limits_invariant) { | |
| 2234 | + mutex_unlock(&memcg_max_mutex); | |
| 2235 | + ret = -EINVAL; | |
| 2236 | + break; | |
| 2237 | + } | |
| 2238 | + if (max > counter->max) | |
| 2239 | + enlarge = true; | |
| 2240 | + ret = page_counter_set_max(counter, max); | |
| 2241 | + mutex_unlock(&memcg_max_mutex); | |
| 2242 | + | |
| 2243 | + if (!ret) | |
| 2244 | + break; | |
| 2245 | + | |
| 2246 | + if (!drained) { | |
| 2247 | + drain_all_stock(memcg); | |
| 2248 | + drained = true; | |
| 2249 | + continue; | |
| 2250 | + } | |
| 2251 | + | |
| 2252 | + if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, | |
| 2253 | + memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP, NULL)) { | |
| 2254 | + ret = -EBUSY; | |
| 2255 | + break; | |
| 2256 | + } | |
| 2257 | + } while (true); | |
| 2258 | + | |
| 2259 | + if (!ret && enlarge) | |
| 2260 | + memcg1_oom_recover(memcg); | |
| 2261 | + | |
| 2262 | + return ret; | |
| 2263 | +} | |
| 2264 | + | |
| 2265 | +/* | |
| 2266 | + * Reclaims as many pages from the given memcg as possible. | |
| 2267 | + * | |
| 2268 | + * Caller is responsible for holding css reference for memcg. | |
| 2269 | + */ | |
| 2270 | +static int mem_cgroup_force_empty(struct mem_cgroup *memcg) | |
| 2271 | +{ | |
| 2272 | + int nr_retries = MAX_RECLAIM_RETRIES; | |
| 2273 | + | |
| 2274 | + /* we call try-to-free pages for make this cgroup empty */ | |
| 2275 | + lru_add_drain_all(); | |
| 2276 | + | |
| 2277 | + drain_all_stock(memcg); | |
| 2278 | + | |
| 2279 | + /* try to free all pages in this cgroup */ | |
| 2280 | + while (nr_retries && page_counter_read(&memcg->memory)) { | |
| 2281 | + if (signal_pending(current)) | |
| 2282 | + return -EINTR; | |
| 2283 | + | |
| 2284 | + if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, | |
| 2285 | + MEMCG_RECLAIM_MAY_SWAP, NULL)) | |
| 2286 | + nr_retries--; | |
| 2287 | + } | |
| 2288 | + | |
| 2289 | + return 0; | |
| 2290 | +} | |
| 2291 | + | |
| 2292 | +static ssize_t mem_cgroup_force_empty_write(struct kernfs_open_file *of, | |
| 2293 | + char *buf, size_t nbytes, | |
| 2294 | + loff_t off) | |
| 2295 | +{ | |
| 2296 | + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); | |
| 2297 | + | |
| 2298 | + if (mem_cgroup_is_root(memcg)) | |
| 2299 | + return -EINVAL; | |
| 2300 | + return mem_cgroup_force_empty(memcg) ?: nbytes; | |
| 2301 | +} | |
| 2302 | + | |
| 2303 | +static u64 mem_cgroup_hierarchy_read(struct cgroup_subsys_state *css, | |
| 2304 | + struct cftype *cft) | |
| 2305 | +{ | |
| 2306 | + return 1; | |
| 2307 | +} | |
| 2308 | + | |
| 2309 | +static int mem_cgroup_hierarchy_write(struct cgroup_subsys_state *css, | |
| 2310 | + struct cftype *cft, u64 val) | |
| 2311 | +{ | |
| 2312 | + if (val == 1) | |
| 2313 | + return 0; | |
| 2314 | + | |
| 2315 | + pr_warn_once("Non-hierarchical mode is deprecated. " | |
| 2316 | + "Please report your usecase to linux-mm@kvack.org if you " | |
| 2317 | + "depend on this functionality.\n"); | |
| 2318 | + | |
| 2319 | + return -EINVAL; | |
| 2320 | +} | |
| 2321 | + | |
| 2322 | +static u64 mem_cgroup_read_u64(struct cgroup_subsys_state *css, | |
| 2323 | + struct cftype *cft) | |
| 2324 | +{ | |
| 2325 | + struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 2326 | + struct page_counter *counter; | |
| 2327 | + | |
| 2328 | + switch (MEMFILE_TYPE(cft->private)) { | |
| 2329 | + case _MEM: | |
| 2330 | + counter = &memcg->memory; | |
| 2331 | + break; | |
| 2332 | + case _MEMSWAP: | |
| 2333 | + counter = &memcg->memsw; | |
| 2334 | + break; | |
| 2335 | + case _KMEM: | |
| 2336 | + counter = &memcg->kmem; | |
| 2337 | + break; | |
| 2338 | + case _TCP: | |
| 2339 | + counter = &memcg->tcpmem; | |
| 2340 | + break; | |
| 2341 | + default: | |
| 2342 | + BUG(); | |
| 2343 | + } | |
| 2344 | + | |
| 2345 | + switch (MEMFILE_ATTR(cft->private)) { | |
| 2346 | + case RES_USAGE: | |
| 2347 | + if (counter == &memcg->memory) | |
| 2348 | + return (u64)mem_cgroup_usage(memcg, false) * PAGE_SIZE; | |
| 2349 | + if (counter == &memcg->memsw) | |
| 2350 | + return (u64)mem_cgroup_usage(memcg, true) * PAGE_SIZE; | |
| 2351 | + return (u64)page_counter_read(counter) * PAGE_SIZE; | |
| 2352 | + case RES_LIMIT: | |
| 2353 | + return (u64)counter->max * PAGE_SIZE; | |
| 2354 | + case RES_MAX_USAGE: | |
| 2355 | + return (u64)counter->watermark * PAGE_SIZE; | |
| 2356 | + case RES_FAILCNT: | |
| 2357 | + return counter->failcnt; | |
| 2358 | + case RES_SOFT_LIMIT: | |
| 2359 | + return (u64)READ_ONCE(memcg->soft_limit) * PAGE_SIZE; | |
| 2360 | + default: | |
| 2361 | + BUG(); | |
| 2362 | + } | |
| 2363 | +} | |
| 2364 | + | |
| 2365 | +/* | |
| 2366 | + * This function doesn't do anything useful. Its only job is to provide a read | |
| 2367 | + * handler for a file so that cgroup_file_mode() will add read permissions. | |
| 2368 | + */ | |
| 2369 | +static int mem_cgroup_dummy_seq_show(__always_unused struct seq_file *m, | |
| 2370 | + __always_unused void *v) | |
| 2371 | +{ | |
| 2372 | + return -EINVAL; | |
| 2373 | +} | |
| 2374 | + | |
| 2375 | +static int memcg_update_tcp_max(struct mem_cgroup *memcg, unsigned long max) | |
| 2376 | +{ | |
| 2377 | + int ret; | |
| 2378 | + | |
| 2379 | + mutex_lock(&memcg_max_mutex); | |
| 2380 | + | |
| 2381 | + ret = page_counter_set_max(&memcg->tcpmem, max); | |
| 2382 | + if (ret) | |
| 2383 | + goto out; | |
| 2384 | + | |
| 2385 | + if (!memcg->tcpmem_active) { | |
| 2386 | + /* | |
| 2387 | + * The active flag needs to be written after the static_key | |
| 2388 | + * update. This is what guarantees that the socket activation | |
| 2389 | + * function is the last one to run. See mem_cgroup_sk_alloc() | |
| 2390 | + * for details, and note that we don't mark any socket as | |
| 2391 | + * belonging to this memcg until that flag is up. | |
| 2392 | + * | |
| 2393 | + * We need to do this, because static_keys will span multiple | |
| 2394 | + * sites, but we can't control their order. If we mark a socket | |
| 2395 | + * as accounted, but the accounting functions are not patched in | |
| 2396 | + * yet, we'll lose accounting. | |
| 2397 | + * | |
| 2398 | + * We never race with the readers in mem_cgroup_sk_alloc(), | |
| 2399 | + * because when this value change, the code to process it is not | |
| 2400 | + * patched in yet. | |
| 2401 | + */ | |
| 2402 | + static_branch_inc(&memcg_sockets_enabled_key); | |
| 2403 | + memcg->tcpmem_active = true; | |
| 2404 | + } | |
| 2405 | +out: | |
| 2406 | + mutex_unlock(&memcg_max_mutex); | |
| 2407 | + return ret; | |
| 2408 | +} | |
| 2409 | + | |
| 2410 | +/* | |
| 2411 | + * The user of this function is... | |
| 2412 | + * RES_LIMIT. | |
| 2413 | + */ | |
| 2414 | +static ssize_t mem_cgroup_write(struct kernfs_open_file *of, | |
| 2415 | + char *buf, size_t nbytes, loff_t off) | |
| 2416 | +{ | |
| 2417 | + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); | |
| 2418 | + unsigned long nr_pages; | |
| 2419 | + int ret; | |
| 2420 | + | |
| 2421 | + buf = strstrip(buf); | |
| 2422 | + ret = page_counter_memparse(buf, "-1", &nr_pages); | |
| 2423 | + if (ret) | |
| 2424 | + return ret; | |
| 2425 | + | |
| 2426 | + switch (MEMFILE_ATTR(of_cft(of)->private)) { | |
| 2427 | + case RES_LIMIT: | |
| 2428 | + if (mem_cgroup_is_root(memcg)) { /* Can't set limit on root */ | |
| 2429 | + ret = -EINVAL; | |
| 2430 | + break; | |
| 2431 | + } | |
| 2432 | + switch (MEMFILE_TYPE(of_cft(of)->private)) { | |
| 2433 | + case _MEM: | |
| 2434 | + ret = mem_cgroup_resize_max(memcg, nr_pages, false); | |
| 2435 | + break; | |
| 2436 | + case _MEMSWAP: | |
| 2437 | + ret = mem_cgroup_resize_max(memcg, nr_pages, true); | |
| 2438 | + break; | |
| 2439 | + case _KMEM: | |
| 2440 | + pr_warn_once("kmem.limit_in_bytes is deprecated and will be removed. " | |
| 2441 | + "Writing any value to this file has no effect. " | |
| 2442 | + "Please report your usecase to linux-mm@kvack.org if you " | |
| 2443 | + "depend on this functionality.\n"); | |
| 2444 | + ret = 0; | |
| 2445 | + break; | |
| 2446 | + case _TCP: | |
| 2447 | + ret = memcg_update_tcp_max(memcg, nr_pages); | |
| 2448 | + break; | |
| 2449 | + } | |
| 2450 | + break; | |
| 2451 | + case RES_SOFT_LIMIT: | |
| 2452 | + if (IS_ENABLED(CONFIG_PREEMPT_RT)) { | |
| 2453 | + ret = -EOPNOTSUPP; | |
| 2454 | + } else { | |
| 2455 | + WRITE_ONCE(memcg->soft_limit, nr_pages); | |
| 2456 | + ret = 0; | |
| 2457 | + } | |
| 2458 | + break; | |
| 2459 | + } | |
| 2460 | + return ret ?: nbytes; | |
| 2461 | +} | |
| 2462 | + | |
| 2463 | +static ssize_t mem_cgroup_reset(struct kernfs_open_file *of, char *buf, | |
| 2464 | + size_t nbytes, loff_t off) | |
| 2465 | +{ | |
| 2466 | + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); | |
| 2467 | + struct page_counter *counter; | |
| 2468 | + | |
| 2469 | + switch (MEMFILE_TYPE(of_cft(of)->private)) { | |
| 2470 | + case _MEM: | |
| 2471 | + counter = &memcg->memory; | |
| 2472 | + break; | |
| 2473 | + case _MEMSWAP: | |
| 2474 | + counter = &memcg->memsw; | |
| 2475 | + break; | |
| 2476 | + case _KMEM: | |
| 2477 | + counter = &memcg->kmem; | |
| 2478 | + break; | |
| 2479 | + case _TCP: | |
| 2480 | + counter = &memcg->tcpmem; | |
| 2481 | + break; | |
| 2482 | + default: | |
| 2483 | + BUG(); | |
| 2484 | + } | |
| 2485 | + | |
| 2486 | + switch (MEMFILE_ATTR(of_cft(of)->private)) { | |
| 2487 | + case RES_MAX_USAGE: | |
| 2488 | + page_counter_reset_watermark(counter); | |
| 2489 | + break; | |
| 2490 | + case RES_FAILCNT: | |
| 2491 | + counter->failcnt = 0; | |
| 2492 | + break; | |
| 2493 | + default: | |
| 2494 | + BUG(); | |
| 2495 | + } | |
| 2496 | + | |
| 2497 | + return nbytes; | |
| 2498 | +} | |
| 2499 | + | |
| 2500 | +#ifdef CONFIG_NUMA | |
| 2501 | + | |
| 2502 | +#define LRU_ALL_FILE (BIT(LRU_INACTIVE_FILE) | BIT(LRU_ACTIVE_FILE)) | |
| 2503 | +#define LRU_ALL_ANON (BIT(LRU_INACTIVE_ANON) | BIT(LRU_ACTIVE_ANON)) | |
| 2504 | +#define LRU_ALL ((1 << NR_LRU_LISTS) - 1) | |
| 2505 | + | |
| 2506 | +static unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg, | |
| 2507 | + int nid, unsigned int lru_mask, bool tree) | |
| 2508 | +{ | |
| 2509 | + struct lruvec *lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); | |
| 2510 | + unsigned long nr = 0; | |
| 2511 | + enum lru_list lru; | |
| 2512 | + | |
| 2513 | + VM_BUG_ON((unsigned)nid >= nr_node_ids); | |
| 2514 | + | |
| 2515 | + for_each_lru(lru) { | |
| 2516 | + if (!(BIT(lru) & lru_mask)) | |
| 2517 | + continue; | |
| 2518 | + if (tree) | |
| 2519 | + nr += lruvec_page_state(lruvec, NR_LRU_BASE + lru); | |
| 2520 | + else | |
| 2521 | + nr += lruvec_page_state_local(lruvec, NR_LRU_BASE + lru); | |
| 2522 | + } | |
| 2523 | + return nr; | |
| 2524 | +} | |
| 2525 | + | |
| 2526 | +static unsigned long mem_cgroup_nr_lru_pages(struct mem_cgroup *memcg, | |
| 2527 | + unsigned int lru_mask, | |
| 2528 | + bool tree) | |
| 2529 | +{ | |
| 2530 | + unsigned long nr = 0; | |
| 2531 | + enum lru_list lru; | |
| 2532 | + | |
| 2533 | + for_each_lru(lru) { | |
| 2534 | + if (!(BIT(lru) & lru_mask)) | |
| 2535 | + continue; | |
| 2536 | + if (tree) | |
| 2537 | + nr += memcg_page_state(memcg, NR_LRU_BASE + lru); | |
| 2538 | + else | |
| 2539 | + nr += memcg_page_state_local(memcg, NR_LRU_BASE + lru); | |
| 2540 | + } | |
| 2541 | + return nr; | |
| 2542 | +} | |
| 2543 | + | |
| 2544 | +static int memcg_numa_stat_show(struct seq_file *m, void *v) | |
| 2545 | +{ | |
| 2546 | + struct numa_stat { | |
| 2547 | + const char *name; | |
| 2548 | + unsigned int lru_mask; | |
| 2549 | + }; | |
| 2550 | + | |
| 2551 | + static const struct numa_stat stats[] = { | |
| 2552 | + { "total", LRU_ALL }, | |
| 2553 | + { "file", LRU_ALL_FILE }, | |
| 2554 | + { "anon", LRU_ALL_ANON }, | |
| 2555 | + { "unevictable", BIT(LRU_UNEVICTABLE) }, | |
| 2556 | + }; | |
| 2557 | + const struct numa_stat *stat; | |
| 2558 | + int nid; | |
| 2559 | + struct mem_cgroup *memcg = mem_cgroup_from_seq(m); | |
| 2560 | + | |
| 2561 | + mem_cgroup_flush_stats(memcg); | |
| 2562 | + | |
| 2563 | + for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { | |
| 2564 | + seq_printf(m, "%s=%lu", stat->name, | |
| 2565 | + mem_cgroup_nr_lru_pages(memcg, stat->lru_mask, | |
| 2566 | + false)); | |
| 2567 | + for_each_node_state(nid, N_MEMORY) | |
| 2568 | + seq_printf(m, " N%d=%lu", nid, | |
| 2569 | + mem_cgroup_node_nr_lru_pages(memcg, nid, | |
| 2570 | + stat->lru_mask, false)); | |
| 2571 | + seq_putc(m, '\n'); | |
| 2572 | + } | |
| 2573 | + | |
| 2574 | + for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { | |
| 2575 | + | |
| 2576 | + seq_printf(m, "hierarchical_%s=%lu", stat->name, | |
| 2577 | + mem_cgroup_nr_lru_pages(memcg, stat->lru_mask, | |
| 2578 | + true)); | |
| 2579 | + for_each_node_state(nid, N_MEMORY) | |
| 2580 | + seq_printf(m, " N%d=%lu", nid, | |
| 2581 | + mem_cgroup_node_nr_lru_pages(memcg, nid, | |
| 2582 | + stat->lru_mask, true)); | |
| 2583 | + seq_putc(m, '\n'); | |
| 2584 | + } | |
| 2585 | + | |
| 2586 | + return 0; | |
| 2587 | +} | |
| 2588 | +#endif /* CONFIG_NUMA */ | |
| 2589 | + | |
| 2590 | +static const unsigned int memcg1_stats[] = { | |
| 2591 | + NR_FILE_PAGES, | |
| 2592 | + NR_ANON_MAPPED, | |
| 2593 | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 2594 | + NR_ANON_THPS, | |
| 2595 | +#endif | |
| 2596 | + NR_SHMEM, | |
| 2597 | + NR_FILE_MAPPED, | |
| 2598 | + NR_FILE_DIRTY, | |
| 2599 | + NR_WRITEBACK, | |
| 2600 | + WORKINGSET_REFAULT_ANON, | |
| 2601 | + WORKINGSET_REFAULT_FILE, | |
| 2602 | +#ifdef CONFIG_SWAP | |
| 2603 | + MEMCG_SWAP, | |
| 2604 | + NR_SWAPCACHE, | |
| 2605 | +#endif | |
| 2606 | +}; | |
| 2607 | + | |
| 2608 | +static const char *const memcg1_stat_names[] = { | |
| 2609 | + "cache", | |
| 2610 | + "rss", | |
| 2611 | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 2612 | + "rss_huge", | |
| 2613 | +#endif | |
| 2614 | + "shmem", | |
| 2615 | + "mapped_file", | |
| 2616 | + "dirty", | |
| 2617 | + "writeback", | |
| 2618 | + "workingset_refault_anon", | |
| 2619 | + "workingset_refault_file", | |
| 2620 | +#ifdef CONFIG_SWAP | |
| 2621 | + "swap", | |
| 2622 | + "swapcached", | |
| 2623 | +#endif | |
| 2624 | +}; | |
| 2625 | + | |
| 2626 | +/* Universal VM events cgroup1 shows, original sort order */ | |
| 2627 | +static const unsigned int memcg1_events[] = { | |
| 2628 | + PGPGIN, | |
| 2629 | + PGPGOUT, | |
| 2630 | + PGFAULT, | |
| 2631 | + PGMAJFAULT, | |
| 2632 | +}; | |
| 2633 | + | |
| 2634 | +void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) | |
| 2635 | +{ | |
| 2636 | + unsigned long memory, memsw; | |
| 2637 | + struct mem_cgroup *mi; | |
| 2638 | + unsigned int i; | |
| 2639 | + | |
| 2640 | + BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); | |
| 2641 | + | |
| 2642 | + mem_cgroup_flush_stats(memcg); | |
| 2643 | + | |
| 2644 | + for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { | |
| 2645 | + unsigned long nr; | |
| 2646 | + | |
| 2647 | + nr = memcg_page_state_local_output(memcg, memcg1_stats[i]); | |
| 2648 | + seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], nr); | |
| 2649 | + } | |
| 2650 | + | |
| 2651 | + for (i = 0; i < ARRAY_SIZE(memcg1_events); i++) | |
| 2652 | + seq_buf_printf(s, "%s %lu\n", vm_event_name(memcg1_events[i]), | |
| 2653 | + memcg_events_local(memcg, memcg1_events[i])); | |
| 2654 | + | |
| 2655 | + for (i = 0; i < NR_LRU_LISTS; i++) | |
| 2656 | + seq_buf_printf(s, "%s %lu\n", lru_list_name(i), | |
| 2657 | + memcg_page_state_local(memcg, NR_LRU_BASE + i) * | |
| 2658 | + PAGE_SIZE); | |
| 2659 | + | |
| 2660 | + /* Hierarchical information */ | |
| 2661 | + memory = memsw = PAGE_COUNTER_MAX; | |
| 2662 | + for (mi = memcg; mi; mi = parent_mem_cgroup(mi)) { | |
| 2663 | + memory = min(memory, READ_ONCE(mi->memory.max)); | |
| 2664 | + memsw = min(memsw, READ_ONCE(mi->memsw.max)); | |
| 2665 | + } | |
| 2666 | + seq_buf_printf(s, "hierarchical_memory_limit %llu\n", | |
| 2667 | + (u64)memory * PAGE_SIZE); | |
| 2668 | + seq_buf_printf(s, "hierarchical_memsw_limit %llu\n", | |
| 2669 | + (u64)memsw * PAGE_SIZE); | |
| 2670 | + | |
| 2671 | + for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { | |
| 2672 | + unsigned long nr; | |
| 2673 | + | |
| 2674 | + nr = memcg_page_state_output(memcg, memcg1_stats[i]); | |
| 2675 | + seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i], | |
| 2676 | + (u64)nr); | |
| 2677 | + } | |
| 2678 | + | |
| 2679 | + for (i = 0; i < ARRAY_SIZE(memcg1_events); i++) | |
| 2680 | + seq_buf_printf(s, "total_%s %llu\n", | |
| 2681 | + vm_event_name(memcg1_events[i]), | |
| 2682 | + (u64)memcg_events(memcg, memcg1_events[i])); | |
| 2683 | + | |
| 2684 | + for (i = 0; i < NR_LRU_LISTS; i++) | |
| 2685 | + seq_buf_printf(s, "total_%s %llu\n", lru_list_name(i), | |
| 2686 | + (u64)memcg_page_state(memcg, NR_LRU_BASE + i) * | |
| 2687 | + PAGE_SIZE); | |
| 2688 | + | |
| 2689 | +#ifdef CONFIG_DEBUG_VM | |
| 2690 | + { | |
| 2691 | + pg_data_t *pgdat; | |
| 2692 | + struct mem_cgroup_per_node *mz; | |
| 2693 | + unsigned long anon_cost = 0; | |
| 2694 | + unsigned long file_cost = 0; | |
| 2695 | + | |
| 2696 | + for_each_online_pgdat(pgdat) { | |
| 2697 | + mz = memcg->nodeinfo[pgdat->node_id]; | |
| 2698 | + | |
| 2699 | + anon_cost += mz->lruvec.anon_cost; | |
| 2700 | + file_cost += mz->lruvec.file_cost; | |
| 2701 | + } | |
| 2702 | + seq_buf_printf(s, "anon_cost %lu\n", anon_cost); | |
| 2703 | + seq_buf_printf(s, "file_cost %lu\n", file_cost); | |
| 2704 | + } | |
| 2705 | +#endif | |
| 2706 | +} | |
| 2707 | + | |
| 2708 | +static u64 mem_cgroup_swappiness_read(struct cgroup_subsys_state *css, | |
| 2709 | + struct cftype *cft) | |
| 2710 | +{ | |
| 2711 | + struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 2712 | + | |
| 2713 | + return mem_cgroup_swappiness(memcg); | |
| 2714 | +} | |
| 2715 | + | |
| 2716 | +static int mem_cgroup_swappiness_write(struct cgroup_subsys_state *css, | |
| 2717 | + struct cftype *cft, u64 val) | |
| 2718 | +{ | |
| 2719 | + struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 2720 | + | |
| 2721 | + if (val > MAX_SWAPPINESS) | |
| 2722 | + return -EINVAL; | |
| 2723 | + | |
| 2724 | + if (!mem_cgroup_is_root(memcg)) | |
| 2725 | + WRITE_ONCE(memcg->swappiness, val); | |
| 2726 | + else | |
| 2727 | + WRITE_ONCE(vm_swappiness, val); | |
| 2728 | + | |
| 2729 | + return 0; | |
| 2730 | +} | |
| 2731 | + | |
| 2732 | +static int mem_cgroup_oom_control_read(struct seq_file *sf, void *v) | |
| 2733 | +{ | |
| 2734 | + struct mem_cgroup *memcg = mem_cgroup_from_seq(sf); | |
| 2735 | + | |
| 2736 | + seq_printf(sf, "oom_kill_disable %d\n", READ_ONCE(memcg->oom_kill_disable)); | |
| 2737 | + seq_printf(sf, "under_oom %d\n", (bool)memcg->under_oom); | |
| 2738 | + seq_printf(sf, "oom_kill %lu\n", | |
| 2739 | + atomic_long_read(&memcg->memory_events[MEMCG_OOM_KILL])); | |
| 2740 | + return 0; | |
| 2741 | +} | |
| 2742 | + | |
| 2743 | +static int mem_cgroup_oom_control_write(struct cgroup_subsys_state *css, | |
| 2744 | + struct cftype *cft, u64 val) | |
| 2745 | +{ | |
| 2746 | + struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 2747 | + | |
| 2748 | + /* cannot set to root cgroup and only 0 and 1 are allowed */ | |
| 2749 | + if (mem_cgroup_is_root(memcg) || !((val == 0) || (val == 1))) | |
| 2750 | + return -EINVAL; | |
| 2751 | + | |
| 2752 | + WRITE_ONCE(memcg->oom_kill_disable, val); | |
| 2753 | + if (!val) | |
| 2754 | + memcg1_oom_recover(memcg); | |
| 2755 | + | |
| 2756 | + return 0; | |
| 2757 | +} | |
| 2758 | + | |
| 2759 | +#ifdef CONFIG_SLUB_DEBUG | |
| 2760 | +static int mem_cgroup_slab_show(struct seq_file *m, void *p) | |
| 2761 | +{ | |
| 2762 | + /* | |
| 2763 | + * Deprecated. | |
| 2764 | + * Please, take a look at tools/cgroup/memcg_slabinfo.py . | |
| 2765 | + */ | |
| 2766 | + return 0; | |
| 2767 | +} | |
| 2768 | +#endif | |
| 2769 | + | |
| 2770 | +struct cftype mem_cgroup_legacy_files[] = { | |
| 2771 | + { | |
| 2772 | + .name = "usage_in_bytes", | |
| 2773 | + .private = MEMFILE_PRIVATE(_MEM, RES_USAGE), | |
| 2774 | + .read_u64 = mem_cgroup_read_u64, | |
| 2775 | + }, | |
| 2776 | + { | |
| 2777 | + .name = "max_usage_in_bytes", | |
| 2778 | + .private = MEMFILE_PRIVATE(_MEM, RES_MAX_USAGE), | |
| 2779 | + .write = mem_cgroup_reset, | |
| 2780 | + .read_u64 = mem_cgroup_read_u64, | |
| 2781 | + }, | |
| 2782 | + { | |
| 2783 | + .name = "limit_in_bytes", | |
| 2784 | + .private = MEMFILE_PRIVATE(_MEM, RES_LIMIT), | |
| 2785 | + .write = mem_cgroup_write, | |
| 2786 | + .read_u64 = mem_cgroup_read_u64, | |
| 2787 | + }, | |
| 2788 | + { | |
| 2789 | + .name = "soft_limit_in_bytes", | |
| 2790 | + .private = MEMFILE_PRIVATE(_MEM, RES_SOFT_LIMIT), | |
| 2791 | + .write = mem_cgroup_write, | |
| 2792 | + .read_u64 = mem_cgroup_read_u64, | |
| 2793 | + }, | |
| 2794 | + { | |
| 2795 | + .name = "failcnt", | |
| 2796 | + .private = MEMFILE_PRIVATE(_MEM, RES_FAILCNT), | |
| 2797 | + .write = mem_cgroup_reset, | |
| 2798 | + .read_u64 = mem_cgroup_read_u64, | |
| 2799 | + }, | |
| 2800 | + { | |
| 2801 | + .name = "stat", | |
| 2802 | + .seq_show = memory_stat_show, | |
| 2803 | + }, | |
| 2804 | + { | |
| 2805 | + .name = "force_empty", | |
| 2806 | + .write = mem_cgroup_force_empty_write, | |
| 2807 | + }, | |
| 2808 | + { | |
| 2809 | + .name = "use_hierarchy", | |
| 2810 | + .write_u64 = mem_cgroup_hierarchy_write, | |
| 2811 | + .read_u64 = mem_cgroup_hierarchy_read, | |
| 2812 | + }, | |
| 2813 | + { | |
| 2814 | + .name = "cgroup.event_control", /* XXX: for compat */ | |
| 2815 | + .write = memcg_write_event_control, | |
| 2816 | + .flags = CFTYPE_NO_PREFIX | CFTYPE_WORLD_WRITABLE, | |
| 2817 | + }, | |
| 2818 | + { | |
| 2819 | + .name = "swappiness", | |
| 2820 | + .read_u64 = mem_cgroup_swappiness_read, | |
| 2821 | + .write_u64 = mem_cgroup_swappiness_write, | |
| 2822 | + }, | |
| 2823 | + { | |
| 2824 | + .name = "move_charge_at_immigrate", | |
| 2825 | + .read_u64 = mem_cgroup_move_charge_read, | |
| 2826 | + .write_u64 = mem_cgroup_move_charge_write, | |
| 2827 | + }, | |
| 2828 | + { | |
| 2829 | + .name = "oom_control", | |
| 2830 | + .seq_show = mem_cgroup_oom_control_read, | |
| 2831 | + .write_u64 = mem_cgroup_oom_control_write, | |
| 2832 | + }, | |
| 2833 | + { | |
| 2834 | + .name = "pressure_level", | |
| 2835 | + .seq_show = mem_cgroup_dummy_seq_show, | |
| 2836 | + }, | |
| 2837 | +#ifdef CONFIG_NUMA | |
| 2838 | + { | |
| 2839 | + .name = "numa_stat", | |
| 2840 | + .seq_show = memcg_numa_stat_show, | |
| 2841 | + }, | |
| 2842 | +#endif | |
| 2843 | + { | |
| 2844 | + .name = "kmem.limit_in_bytes", | |
| 2845 | + .private = MEMFILE_PRIVATE(_KMEM, RES_LIMIT), | |
| 2846 | + .write = mem_cgroup_write, | |
| 2847 | + .read_u64 = mem_cgroup_read_u64, | |
| 2848 | + }, | |
| 2849 | + { | |
| 2850 | + .name = "kmem.usage_in_bytes", | |
| 2851 | + .private = MEMFILE_PRIVATE(_KMEM, RES_USAGE), | |
| 2852 | + .read_u64 = mem_cgroup_read_u64, | |
| 2853 | + }, | |
| 2854 | + { | |
| 2855 | + .name = "kmem.failcnt", | |
| 2856 | + .private = MEMFILE_PRIVATE(_KMEM, RES_FAILCNT), | |
| 2857 | + .write = mem_cgroup_reset, | |
| 2858 | + .read_u64 = mem_cgroup_read_u64, | |
| 2859 | + }, | |
| 2860 | + { | |
| 2861 | + .name = "kmem.max_usage_in_bytes", | |
| 2862 | + .private = MEMFILE_PRIVATE(_KMEM, RES_MAX_USAGE), | |
| 2863 | + .write = mem_cgroup_reset, | |
| 2864 | + .read_u64 = mem_cgroup_read_u64, | |
| 2865 | + }, | |
| 2866 | +#ifdef CONFIG_SLUB_DEBUG | |
| 2867 | + { | |
| 2868 | + .name = "kmem.slabinfo", | |
| 2869 | + .seq_show = mem_cgroup_slab_show, | |
| 2870 | + }, | |
| 2871 | +#endif | |
| 2872 | + { | |
| 2873 | + .name = "kmem.tcp.limit_in_bytes", | |
| 2874 | + .private = MEMFILE_PRIVATE(_TCP, RES_LIMIT), | |
| 2875 | + .write = mem_cgroup_write, | |
| 2876 | + .read_u64 = mem_cgroup_read_u64, | |
| 2877 | + }, | |
| 2878 | + { | |
| 2879 | + .name = "kmem.tcp.usage_in_bytes", | |
| 2880 | + .private = MEMFILE_PRIVATE(_TCP, RES_USAGE), | |
| 2881 | + .read_u64 = mem_cgroup_read_u64, | |
| 2882 | + }, | |
| 2883 | + { | |
| 2884 | + .name = "kmem.tcp.failcnt", | |
| 2885 | + .private = MEMFILE_PRIVATE(_TCP, RES_FAILCNT), | |
| 2886 | + .write = mem_cgroup_reset, | |
| 2887 | + .read_u64 = mem_cgroup_read_u64, | |
| 2888 | + }, | |
| 2889 | + { | |
| 2890 | + .name = "kmem.tcp.max_usage_in_bytes", | |
| 2891 | + .private = MEMFILE_PRIVATE(_TCP, RES_MAX_USAGE), | |
| 2892 | + .write = mem_cgroup_reset, | |
| 2893 | + .read_u64 = mem_cgroup_read_u64, | |
| 2894 | + }, | |
| 2895 | + { }, /* terminate */ | |
| 2896 | +}; | |
| 2897 | + | |
| 2898 | +struct cftype memsw_files[] = { | |
| 2899 | + { | |
| 2900 | + .name = "memsw.usage_in_bytes", | |
| 2901 | + .private = MEMFILE_PRIVATE(_MEMSWAP, RES_USAGE), | |
| 2902 | + .read_u64 = mem_cgroup_read_u64, | |
| 2903 | + }, | |
| 2904 | + { | |
| 2905 | + .name = "memsw.max_usage_in_bytes", | |
| 2906 | + .private = MEMFILE_PRIVATE(_MEMSWAP, RES_MAX_USAGE), | |
| 2907 | + .write = mem_cgroup_reset, | |
| 2908 | + .read_u64 = mem_cgroup_read_u64, | |
| 2909 | + }, | |
| 2910 | + { | |
| 2911 | + .name = "memsw.limit_in_bytes", | |
| 2912 | + .private = MEMFILE_PRIVATE(_MEMSWAP, RES_LIMIT), | |
| 2913 | + .write = mem_cgroup_write, | |
| 2914 | + .read_u64 = mem_cgroup_read_u64, | |
| 2915 | + }, | |
| 2916 | + { | |
| 2917 | + .name = "memsw.failcnt", | |
| 2918 | + .private = MEMFILE_PRIVATE(_MEMSWAP, RES_FAILCNT), | |
| 2919 | + .write = mem_cgroup_reset, | |
| 2920 | + .read_u64 = mem_cgroup_read_u64, | |
| 2921 | + }, | |
| 2922 | + { }, /* terminate */ | |
| 2923 | +}; | |
| 2924 | + | |
| 2925 | +void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages) | |
| 2926 | +{ | |
| 2927 | + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) { | |
| 2928 | + if (nr_pages > 0) | |
| 2929 | + page_counter_charge(&memcg->kmem, nr_pages); | |
| 2930 | + else | |
| 2931 | + page_counter_uncharge(&memcg->kmem, -nr_pages); | |
| 2932 | + } | |
| 2933 | +} | |
| 2934 | + | |
| 2935 | +bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages, | |
| 2936 | + gfp_t gfp_mask) | |
| 2937 | +{ | |
| 2938 | + struct page_counter *fail; | |
| 2939 | + | |
| 2940 | + if (page_counter_try_charge(&memcg->tcpmem, nr_pages, &fail)) { | |
| 2941 | + memcg->tcpmem_pressure = 0; | |
| 2942 | + return true; | |
| 2943 | + } | |
| 2944 | + memcg->tcpmem_pressure = 1; | |
| 2945 | + if (gfp_mask & __GFP_NOFAIL) { | |
| 2946 | + page_counter_charge(&memcg->tcpmem, nr_pages); | |
| 2947 | + return true; | |
| 2948 | + } | |
| 2949 | + return false; | |
| 2950 | +} | |
| 2951 | + | |
| 2952 | +static int __init memcg1_init(void) | |
| 2953 | +{ | |
| 2954 | + int node; | |
| 2955 | + | |
| 2956 | + for_each_node(node) { | |
| 2957 | + struct mem_cgroup_tree_per_node *rtpn; | |
| 2958 | + | |
| 2959 | + rtpn = kzalloc_node(sizeof(*rtpn), GFP_KERNEL, node); | |
| 2960 | + | |
| 2961 | + rtpn->rb_root = RB_ROOT; | |
| 2962 | + rtpn->rb_rightmost = NULL; | |
| 2963 | + spin_lock_init(&rtpn->lock); | |
| 2964 | + soft_limit_tree.rb_tree_per_node[node] = rtpn; | |
| 2965 | + } | |
| 2966 | + | |
| 2967 | + return 0; | |
| 2968 | +} | |
| 2969 | +subsys_initcall(memcg1_init); | |
| @@ -0,0 +1,147 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0-or-later */ | |
| 2 | + | |
| 3 | +#ifndef __MM_MEMCONTROL_V1_H | |
| 4 | +#define __MM_MEMCONTROL_V1_H | |
| 5 | + | |
| 6 | +#include <linux/cgroup-defs.h> | |
| 7 | + | |
| 8 | +/* Cgroup v1 and v2 common declarations */ | |
| 9 | + | |
| 10 | +void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, int nr_pages); | |
| 11 | +int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, | |
| 12 | + unsigned int nr_pages); | |
| 13 | + | |
| 14 | +static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, | |
| 15 | + unsigned int nr_pages) | |
| 16 | +{ | |
| 17 | + if (mem_cgroup_is_root(memcg)) | |
| 18 | + return 0; | |
| 19 | + | |
| 20 | + return try_charge_memcg(memcg, gfp_mask, nr_pages); | |
| 21 | +} | |
| 22 | + | |
| 23 | +void mem_cgroup_id_get_many(struct mem_cgroup *memcg, unsigned int n); | |
| 24 | +void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n); | |
| 25 | + | |
| 26 | +/* | |
| 27 | + * Iteration constructs for visiting all cgroups (under a tree). If | |
| 28 | + * loops are exited prematurely (break), mem_cgroup_iter_break() must | |
| 29 | + * be used for reference counting. | |
| 30 | + */ | |
| 31 | +#define for_each_mem_cgroup_tree(iter, root) \ | |
| 32 | + for (iter = mem_cgroup_iter(root, NULL, NULL); \ | |
| 33 | + iter != NULL; \ | |
| 34 | + iter = mem_cgroup_iter(root, iter, NULL)) | |
| 35 | + | |
| 36 | +#define for_each_mem_cgroup(iter) \ | |
| 37 | + for (iter = mem_cgroup_iter(NULL, NULL, NULL); \ | |
| 38 | + iter != NULL; \ | |
| 39 | + iter = mem_cgroup_iter(NULL, iter, NULL)) | |
| 40 | + | |
| 41 | +/* Whether legacy memory+swap accounting is active */ | |
| 42 | +static bool do_memsw_account(void) | |
| 43 | +{ | |
| 44 | + return !cgroup_subsys_on_dfl(memory_cgrp_subsys); | |
| 45 | +} | |
| 46 | + | |
| 47 | +/* | |
| 48 | + * Per memcg event counter is incremented at every pagein/pageout. With THP, | |
| 49 | + * it will be incremented by the number of pages. This counter is used | |
| 50 | + * to trigger some periodic events. This is straightforward and better | |
| 51 | + * than using jiffies etc. to handle periodic memcg event. | |
| 52 | + */ | |
| 53 | +enum mem_cgroup_events_target { | |
| 54 | + MEM_CGROUP_TARGET_THRESH, | |
| 55 | + MEM_CGROUP_TARGET_SOFTLIMIT, | |
| 56 | + MEM_CGROUP_NTARGETS, | |
| 57 | +}; | |
| 58 | + | |
| 59 | +bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, | |
| 60 | + enum mem_cgroup_events_target target); | |
| 61 | +unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap); | |
| 62 | + | |
| 63 | +void drain_all_stock(struct mem_cgroup *root_memcg); | |
| 64 | + | |
| 65 | +unsigned long memcg_events(struct mem_cgroup *memcg, int event); | |
| 66 | +unsigned long memcg_events_local(struct mem_cgroup *memcg, int event); | |
| 67 | +unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx); | |
| 68 | +unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item); | |
| 69 | +unsigned long memcg_page_state_local_output(struct mem_cgroup *memcg, int item); | |
| 70 | +int memory_stat_show(struct seq_file *m, void *v); | |
| 71 | + | |
| 72 | +/* Cgroup v1-specific declarations */ | |
| 73 | +#ifdef CONFIG_MEMCG_V1 | |
| 74 | +void memcg1_memcg_init(struct mem_cgroup *memcg); | |
| 75 | +void memcg1_remove_from_trees(struct mem_cgroup *memcg); | |
| 76 | + | |
| 77 | +static inline void memcg1_soft_limit_reset(struct mem_cgroup *memcg) | |
| 78 | +{ | |
| 79 | + WRITE_ONCE(memcg->soft_limit, PAGE_COUNTER_MAX); | |
| 80 | +} | |
| 81 | + | |
| 82 | +bool memcg1_wait_acct_move(struct mem_cgroup *memcg); | |
| 83 | + | |
| 84 | +struct cgroup_taskset; | |
| 85 | +int memcg1_can_attach(struct cgroup_taskset *tset); | |
| 86 | +void memcg1_cancel_attach(struct cgroup_taskset *tset); | |
| 87 | +void memcg1_move_task(void); | |
| 88 | +void memcg1_css_offline(struct mem_cgroup *memcg); | |
| 89 | + | |
| 90 | +/* for encoding cft->private value on file */ | |
| 91 | +enum res_type { | |
| 92 | + _MEM, | |
| 93 | + _MEMSWAP, | |
| 94 | + _KMEM, | |
| 95 | + _TCP, | |
| 96 | +}; | |
| 97 | + | |
| 98 | +bool memcg1_oom_prepare(struct mem_cgroup *memcg, bool *locked); | |
| 99 | +void memcg1_oom_finish(struct mem_cgroup *memcg, bool locked); | |
| 100 | +void memcg1_oom_recover(struct mem_cgroup *memcg); | |
| 101 | + | |
| 102 | +void memcg1_check_events(struct mem_cgroup *memcg, int nid); | |
| 103 | + | |
| 104 | +void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s); | |
| 105 | + | |
| 106 | +void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages); | |
| 107 | +static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg) | |
| 108 | +{ | |
| 109 | + return memcg->tcpmem_active; | |
| 110 | +} | |
| 111 | +bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages, | |
| 112 | + gfp_t gfp_mask); | |
| 113 | +static inline void memcg1_uncharge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages) | |
| 114 | +{ | |
| 115 | + page_counter_uncharge(&memcg->tcpmem, nr_pages); | |
| 116 | +} | |
| 117 | + | |
| 118 | +extern struct cftype memsw_files[]; | |
| 119 | +extern struct cftype mem_cgroup_legacy_files[]; | |
| 120 | + | |
| 121 | +#else /* CONFIG_MEMCG_V1 */ | |
| 122 | + | |
| 123 | +static inline void memcg1_memcg_init(struct mem_cgroup *memcg) {} | |
| 124 | +static inline void memcg1_remove_from_trees(struct mem_cgroup *memcg) {} | |
| 125 | +static inline void memcg1_soft_limit_reset(struct mem_cgroup *memcg) {} | |
| 126 | +static inline bool memcg1_wait_acct_move(struct mem_cgroup *memcg) { return false; } | |
| 127 | +static inline void memcg1_css_offline(struct mem_cgroup *memcg) {} | |
| 128 | + | |
| 129 | +static inline bool memcg1_oom_prepare(struct mem_cgroup *memcg, bool *locked) { return true; } | |
| 130 | +static inline void memcg1_oom_finish(struct mem_cgroup *memcg, bool locked) {} | |
| 131 | +static inline void memcg1_oom_recover(struct mem_cgroup *memcg) {} | |
| 132 | + | |
| 133 | +static inline void memcg1_check_events(struct mem_cgroup *memcg, int nid) {} | |
| 134 | + | |
| 135 | +static inline void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) {} | |
| 136 | + | |
| 137 | +static inline void memcg1_account_kmem(struct mem_cgroup *memcg, int nr_pages) {} | |
| 138 | +static inline bool memcg1_tcpmem_active(struct mem_cgroup *memcg) { return false; } | |
| 139 | +static inline bool memcg1_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages, | |
| 140 | + gfp_t gfp_mask) { return true; } | |
| 141 | +static inline void memcg1_uncharge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages) {} | |
| 142 | + | |
| 143 | +extern struct cftype memsw_files[]; | |
| 144 | +extern struct cftype mem_cgroup_legacy_files[]; | |
| 145 | +#endif /* CONFIG_MEMCG_V1 */ | |
| 146 | + | |
| 147 | +#endif /* __MM_MEMCONTROL_V1_H */ | |
| @@ -28,7 +28,6 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <linux/page_counter.h> |
| 29 | 29 | #include <linux/memcontrol.h> |
| 30 | 30 | #include <linux/cgroup.h> |
| 31 | −#include <linux/pagewalk.h> | |
| 32 | 31 | #include <linux/sched/mm.h> |
| 33 | 32 | #include <linux/shmem_fs.h> |
| 34 | 33 | #include <linux/hugetlb.h> |
| @@ -45,14 +44,11 @@ · 10 unmodified lines | ||
| 45 | 44 | #include <linux/mutex.h> |
| 46 | 45 | #include <linux/rbtree.h> |
| 47 | 46 | #include <linux/slab.h> |
| 48 | −#include <linux/swap.h> | |
| 49 | 47 | #include <linux/swapops.h> |
| 50 | 48 | #include <linux/spinlock.h> |
| 51 | −#include <linux/eventfd.h> | |
| 52 | −#include <linux/poll.h> | |
| 53 | −#include <linux/sort.h> | |
| 54 | 49 | #include <linux/fs.h> |
| 55 | 50 | #include <linux/seq_file.h> |
| 51 | +#include <linux/parser.h> | |
| 56 | 52 | #include <linux/vmpressure.h> |
| 57 | 53 | #include <linux/memremap.h> |
| 58 | 54 | #include <linux/mm_inline.h> |
| @@ -60,7 +56,6 @@ · 1 unmodified line | ||
| 60 | 56 | #include <linux/cpu.h> |
| 61 | 57 | #include <linux/oom.h> |
| 62 | 58 | #include <linux/lockdep.h> |
| 63 | −#include <linux/file.h> | |
| 64 | 59 | #include <linux/resume_user_mode.h> |
| 65 | 60 | #include <linux/psi.h> |
| 66 | 61 | #include <linux/seq_buf.h> |
| @@ -70,7 +65,7 @@ · 3 unmodified lines | ||
| 70 | 65 | #include <net/sock.h> |
| 71 | 66 | #include <net/ip.h> |
| 72 | 67 | #include "slab.h" |
| 73 | −#include "swap.h" | |
| 68 | +#include "memcontrol-v1.h" | |
| 74 | 69 | |
| 75 | 70 | #include <linux/uaccess.h> |
| 76 | 71 | |
| @@ -98,140 +93,9 @@ static bool cgroup_memory_nobpf __ro_after_init; · 21 unmodified lines | ||
| 98 | 93 | static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq); |
| 99 | 94 | #endif |
| 100 | 95 | |
| 101 | −/* Whether legacy memory+swap accounting is active */ | |
| 102 | −static bool do_memsw_account(void) | |
| 103 | −{ | |
| 104 | − return !cgroup_subsys_on_dfl(memory_cgrp_subsys); | |
| 105 | −} | |
| 106 | − | |
| 107 | 96 | #define THRESHOLDS_EVENTS_TARGET 128 |
| 108 | 97 | #define SOFTLIMIT_EVENTS_TARGET 1024 |
| 109 | 98 | |
| 110 | −/* | |
| 111 | − * Cgroups above their limits are maintained in a RB-Tree, independent of | |
| 112 | − * their hierarchy representation | |
| 113 | − */ | |
| 114 | − | |
| 115 | −struct mem_cgroup_tree_per_node { | |
| 116 | − struct rb_root rb_root; | |
| 117 | − struct rb_node *rb_rightmost; | |
| 118 | − spinlock_t lock; | |
| 119 | −}; | |
| 120 | − | |
| 121 | −struct mem_cgroup_tree { | |
| 122 | − struct mem_cgroup_tree_per_node *rb_tree_per_node[MAX_NUMNODES]; | |
| 123 | −}; | |
| 124 | − | |
| 125 | −static struct mem_cgroup_tree soft_limit_tree __read_mostly; | |
| 126 | − | |
| 127 | −/* for OOM */ | |
| 128 | −struct mem_cgroup_eventfd_list { | |
| 129 | − struct list_head list; | |
| 130 | − struct eventfd_ctx *eventfd; | |
| 131 | −}; | |
| 132 | − | |
| 133 | −/* | |
| 134 | − * cgroup_event represents events which userspace want to receive. | |
| 135 | − */ | |
| 136 | −struct mem_cgroup_event { | |
| 137 | − /* | |
| 138 | − * memcg which the event belongs to. | |
| 139 | − */ | |
| 140 | − struct mem_cgroup *memcg; | |
| 141 | − /* | |
| 142 | − * eventfd to signal userspace about the event. | |
| 143 | − */ | |
| 144 | − struct eventfd_ctx *eventfd; | |
| 145 | − /* | |
| 146 | − * Each of these stored in a list by the cgroup. | |
| 147 | − */ | |
| 148 | − struct list_head list; | |
| 149 | − /* | |
| 150 | − * register_event() callback will be used to add new userspace | |
| 151 | − * waiter for changes related to this event. Use eventfd_signal() | |
| 152 | − * on eventfd to send notification to userspace. | |
| 153 | − */ | |
| 154 | − int (*register_event)(struct mem_cgroup *memcg, | |
| 155 | − struct eventfd_ctx *eventfd, const char *args); | |
| 156 | − /* | |
| 157 | − * unregister_event() callback will be called when userspace closes | |
| 158 | − * the eventfd or on cgroup removing. This callback must be set, | |
| 159 | − * if you want provide notification functionality. | |
| 160 | − */ | |
| 161 | − void (*unregister_event)(struct mem_cgroup *memcg, | |
| 162 | − struct eventfd_ctx *eventfd); | |
| 163 | − /* | |
| 164 | − * All fields below needed to unregister event when | |
| 165 | − * userspace closes eventfd. | |
| 166 | − */ | |
| 167 | − poll_table pt; | |
| 168 | − wait_queue_head_t *wqh; | |
| 169 | − wait_queue_entry_t wait; | |
| 170 | − struct work_struct remove; | |
| 171 | −}; | |
| 172 | − | |
| 173 | −static void mem_cgroup_threshold(struct mem_cgroup *memcg); | |
| 174 | −static void mem_cgroup_oom_notify(struct mem_cgroup *memcg); | |
| 175 | − | |
| 176 | −/* Stuffs for move charges at task migration. */ | |
| 177 | −/* | |
| 178 | − * Types of charges to be moved. | |
| 179 | − */ | |
| 180 | −#define MOVE_ANON 0x1U | |
| 181 | −#define MOVE_FILE 0x2U | |
| 182 | −#define MOVE_MASK (MOVE_ANON | MOVE_FILE) | |
| 183 | − | |
| 184 | −/* "mc" and its members are protected by cgroup_mutex */ | |
| 185 | −static struct move_charge_struct { | |
| 186 | − spinlock_t lock; /* for from, to */ | |
| 187 | − struct mm_struct *mm; | |
| 188 | − struct mem_cgroup *from; | |
| 189 | − struct mem_cgroup *to; | |
| 190 | − unsigned long flags; | |
| 191 | − unsigned long precharge; | |
| 192 | − unsigned long moved_charge; | |
| 193 | − unsigned long moved_swap; | |
| 194 | − struct task_struct *moving_task; /* a task moving charges */ | |
| 195 | − wait_queue_head_t waitq; /* a waitq for other context */ | |
| 196 | −} mc = { | |
| 197 | − .lock = __SPIN_LOCK_UNLOCKED(mc.lock), | |
| 198 | − .waitq = __WAIT_QUEUE_HEAD_INITIALIZER(mc.waitq), | |
| 199 | −}; | |
| 200 | − | |
| 201 | −/* | |
| 202 | − * Maximum loops in mem_cgroup_soft_reclaim(), used for soft | |
| 203 | − * limit reclaim to prevent infinite loops, if they ever occur. | |
| 204 | − */ | |
| 205 | −#define MEM_CGROUP_MAX_RECLAIM_LOOPS 100 | |
| 206 | −#define MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS 2 | |
| 207 | − | |
| 208 | −/* for encoding cft->private value on file */ | |
| 209 | −enum res_type { | |
| 210 | − _MEM, | |
| 211 | − _MEMSWAP, | |
| 212 | − _KMEM, | |
| 213 | − _TCP, | |
| 214 | −}; | |
| 215 | − | |
| 216 | −#define MEMFILE_PRIVATE(x, val) ((x) << 16 | (val)) | |
| 217 | −#define MEMFILE_TYPE(val) ((val) >> 16 & 0xffff) | |
| 218 | −#define MEMFILE_ATTR(val) ((val) & 0xffff) | |
| 219 | − | |
| 220 | −/* | |
| 221 | − * Iteration constructs for visiting all cgroups (under a tree). If | |
| 222 | − * loops are exited prematurely (break), mem_cgroup_iter_break() must | |
| 223 | − * be used for reference counting. | |
| 224 | − */ | |
| 225 | −#define for_each_mem_cgroup_tree(iter, root) \ | |
| 226 | − for (iter = mem_cgroup_iter(root, NULL, NULL); \ | |
| 227 | − iter != NULL; \ | |
| 228 | − iter = mem_cgroup_iter(root, iter, NULL)) | |
| 229 | − | |
| 230 | −#define for_each_mem_cgroup(iter) \ | |
| 231 | − for (iter = mem_cgroup_iter(NULL, NULL, NULL); \ | |
| 232 | − iter != NULL; \ | |
| 233 | − iter = mem_cgroup_iter(NULL, iter, NULL)) | |
| 234 | − | |
| 235 | 99 | static inline bool task_is_dying(void) |
| 236 | 100 | { |
| 237 | 101 | return tsk_is_oom_victim(current) || fatal_signal_pending(current) || |
| @@ -254,7 +118,6 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressure *vmpr) · 16 unmodified lines | ||
| 254 | 118 | #define CURRENT_OBJCG_UPDATE_BIT 0 |
| 255 | 119 | #define CURRENT_OBJCG_UPDATE_FLAG (1UL << CURRENT_OBJCG_UPDATE_BIT) |
| 256 | 120 | |
| 257 | −#ifdef CONFIG_MEMCG_KMEM | |
| 258 | 121 | static DEFINE_SPINLOCK(objcg_lock); |
| 259 | 122 | |
| 260 | 123 | bool mem_cgroup_kmem_disabled(void) |
| @@ -359,7 +222,6 @@ EXPORT_SYMBOL(memcg_kmem_online_key); · 98 unmodified lines | ||
| 359 | 222 | |
| 360 | 223 | DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key); |
| 361 | 224 | EXPORT_SYMBOL(memcg_bpf_enabled_key); |
| 362 | −#endif | |
| 363 | 225 | |
| 364 | 226 | /** |
| 365 | 227 | * mem_cgroup_css_from_folio - css of the memcg associated with a folio |
| @@ -412,169 +274,6 @@ ino_t page_cgroup_ino(struct page *page) · 46 unmodified lines | ||
| 412 | 274 | return ino; |
| 413 | 275 | } |
| 414 | 276 | |
| 415 | −static void __mem_cgroup_insert_exceeded(struct mem_cgroup_per_node *mz, | |
| 416 | − struct mem_cgroup_tree_per_node *mctz, | |
| 417 | − unsigned long new_usage_in_excess) | |
| 418 | −{ | |
| 419 | − struct rb_node **p = &mctz->rb_root.rb_node; | |
| 420 | − struct rb_node *parent = NULL; | |
| 421 | − struct mem_cgroup_per_node *mz_node; | |
| 422 | − bool rightmost = true; | |
| 423 | − | |
| 424 | − if (mz->on_tree) | |
| 425 | − return; | |
| 426 | − | |
| 427 | − mz->usage_in_excess = new_usage_in_excess; | |
| 428 | − if (!mz->usage_in_excess) | |
| 429 | − return; | |
| 430 | − while (*p) { | |
| 431 | − parent = *p; | |
| 432 | − mz_node = rb_entry(parent, struct mem_cgroup_per_node, | |
| 433 | − tree_node); | |
| 434 | − if (mz->usage_in_excess < mz_node->usage_in_excess) { | |
| 435 | − p = &(*p)->rb_left; | |
| 436 | − rightmost = false; | |
| 437 | − } else { | |
| 438 | − p = &(*p)->rb_right; | |
| 439 | − } | |
| 440 | − } | |
| 441 | − | |
| 442 | − if (rightmost) | |
| 443 | − mctz->rb_rightmost = &mz->tree_node; | |
| 444 | − | |
| 445 | − rb_link_node(&mz->tree_node, parent, p); | |
| 446 | − rb_insert_color(&mz->tree_node, &mctz->rb_root); | |
| 447 | − mz->on_tree = true; | |
| 448 | −} | |
| 449 | − | |
| 450 | −static void __mem_cgroup_remove_exceeded(struct mem_cgroup_per_node *mz, | |
| 451 | − struct mem_cgroup_tree_per_node *mctz) | |
| 452 | −{ | |
| 453 | − if (!mz->on_tree) | |
| 454 | − return; | |
| 455 | − | |
| 456 | − if (&mz->tree_node == mctz->rb_rightmost) | |
| 457 | − mctz->rb_rightmost = rb_prev(&mz->tree_node); | |
| 458 | − | |
| 459 | − rb_erase(&mz->tree_node, &mctz->rb_root); | |
| 460 | − mz->on_tree = false; | |
| 461 | −} | |
| 462 | − | |
| 463 | −static void mem_cgroup_remove_exceeded(struct mem_cgroup_per_node *mz, | |
| 464 | − struct mem_cgroup_tree_per_node *mctz) | |
| 465 | −{ | |
| 466 | − unsigned long flags; | |
| 467 | − | |
| 468 | − spin_lock_irqsave(&mctz->lock, flags); | |
| 469 | − __mem_cgroup_remove_exceeded(mz, mctz); | |
| 470 | − spin_unlock_irqrestore(&mctz->lock, flags); | |
| 471 | −} | |
| 472 | − | |
| 473 | −static unsigned long soft_limit_excess(struct mem_cgroup *memcg) | |
| 474 | −{ | |
| 475 | − unsigned long nr_pages = page_counter_read(&memcg->memory); | |
| 476 | − unsigned long soft_limit = READ_ONCE(memcg->soft_limit); | |
| 477 | − unsigned long excess = 0; | |
| 478 | − | |
| 479 | − if (nr_pages > soft_limit) | |
| 480 | − excess = nr_pages - soft_limit; | |
| 481 | − | |
| 482 | − return excess; | |
| 483 | −} | |
| 484 | − | |
| 485 | −static void mem_cgroup_update_tree(struct mem_cgroup *memcg, int nid) | |
| 486 | −{ | |
| 487 | − unsigned long excess; | |
| 488 | − struct mem_cgroup_per_node *mz; | |
| 489 | − struct mem_cgroup_tree_per_node *mctz; | |
| 490 | − | |
| 491 | − if (lru_gen_enabled()) { | |
| 492 | − if (soft_limit_excess(memcg)) | |
| 493 | − lru_gen_soft_reclaim(memcg, nid); | |
| 494 | − return; | |
| 495 | − } | |
| 496 | − | |
| 497 | − mctz = soft_limit_tree.rb_tree_per_node[nid]; | |
| 498 | − if (!mctz) | |
| 499 | − return; | |
| 500 | − /* | |
| 501 | − * Necessary to update all ancestors when hierarchy is used. | |
| 502 | − * because their event counter is not touched. | |
| 503 | − */ | |
| 504 | − for (; memcg; memcg = parent_mem_cgroup(memcg)) { | |
| 505 | − mz = memcg->nodeinfo[nid]; | |
| 506 | − excess = soft_limit_excess(memcg); | |
| 507 | − /* | |
| 508 | − * We have to update the tree if mz is on RB-tree or | |
| 509 | − * mem is over its softlimit. | |
| 510 | − */ | |
| 511 | − if (excess || mz->on_tree) { | |
| 512 | − unsigned long flags; | |
| 513 | − | |
| 514 | − spin_lock_irqsave(&mctz->lock, flags); | |
| 515 | − /* if on-tree, remove it */ | |
| 516 | − if (mz->on_tree) | |
| 517 | − __mem_cgroup_remove_exceeded(mz, mctz); | |
| 518 | − /* | |
| 519 | − * Insert again. mz->usage_in_excess will be updated. | |
| 520 | − * If excess is 0, no tree ops. | |
| 521 | − */ | |
| 522 | − __mem_cgroup_insert_exceeded(mz, mctz, excess); | |
| 523 | − spin_unlock_irqrestore(&mctz->lock, flags); | |
| 524 | − } | |
| 525 | − } | |
| 526 | −} | |
| 527 | − | |
| 528 | −static void mem_cgroup_remove_from_trees(struct mem_cgroup *memcg) | |
| 529 | −{ | |
| 530 | − struct mem_cgroup_tree_per_node *mctz; | |
| 531 | − struct mem_cgroup_per_node *mz; | |
| 532 | − int nid; | |
| 533 | − | |
| 534 | − for_each_node(nid) { | |
| 535 | − mz = memcg->nodeinfo[nid]; | |
| 536 | − mctz = soft_limit_tree.rb_tree_per_node[nid]; | |
| 537 | − if (mctz) | |
| 538 | − mem_cgroup_remove_exceeded(mz, mctz); | |
| 539 | − } | |
| 540 | −} | |
| 541 | − | |
| 542 | −static struct mem_cgroup_per_node * | |
| 543 | −__mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) | |
| 544 | −{ | |
| 545 | − struct mem_cgroup_per_node *mz; | |
| 546 | − | |
| 547 | −retry: | |
| 548 | − mz = NULL; | |
| 549 | − if (!mctz->rb_rightmost) | |
| 550 | − goto done; /* Nothing to reclaim from */ | |
| 551 | − | |
| 552 | − mz = rb_entry(mctz->rb_rightmost, | |
| 553 | − struct mem_cgroup_per_node, tree_node); | |
| 554 | − /* | |
| 555 | − * Remove the node now but someone else can add it back, | |
| 556 | − * we will to add it back at the end of reclaim to its correct | |
| 557 | − * position in the tree. | |
| 558 | − */ | |
| 559 | − __mem_cgroup_remove_exceeded(mz, mctz); | |
| 560 | − if (!soft_limit_excess(mz->memcg) || | |
| 561 | − !css_tryget(&mz->memcg->css)) | |
| 562 | − goto retry; | |
| 563 | −done: | |
| 564 | − return mz; | |
| 565 | −} | |
| 566 | − | |
| 567 | −static struct mem_cgroup_per_node * | |
| 568 | −mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) | |
| 569 | −{ | |
| 570 | − struct mem_cgroup_per_node *mz; | |
| 571 | − | |
| 572 | − spin_lock_irq(&mctz->lock); | |
| 573 | − mz = __mem_cgroup_largest_soft_limit_node(mctz); | |
| 574 | − spin_unlock_irq(&mctz->lock); | |
| 575 | − return mz; | |
| 576 | −} | |
| 577 | − | |
| 578 | 277 | /* Subset of node_stat_item for memcg stats */ |
| 579 | 278 | static const unsigned int memcg_node_stat_items[] = { |
| 580 | 279 | NR_INACTIVE_ANON, |
| @@ -722,7 +421,7 @@ static const unsigned int memcg_vm_event_stat[] = { · 141 unmodified lines | ||
| 722 | 421 | PGDEACTIVATE, |
| 723 | 422 | PGLAZYFREE, |
| 724 | 423 | PGLAZYFREED, |
| 725 | −#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) | |
| 424 | +#ifdef CONFIG_ZSWAP | |
| 726 | 425 | ZSWPIN, |
| 727 | 426 | ZSWPOUT, |
| 728 | 427 | ZSWPWB, |
| @@ -971,7 +670,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, · 242 unmodified lines | ||
| 971 | 670 | } |
| 972 | 671 | |
| 973 | 672 | /* idx can be of type enum memcg_stat_item or node_stat_item. */ |
| 974 | −static unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) | |
| 673 | +unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) | |
| 975 | 674 | { |
| 976 | 675 | long x; |
| 977 | 676 | int i = memcg_stats_index(idx); |
| @@ -1120,7 +819,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, · 142 unmodified lines | ||
| 1120 | 819 | memcg_stats_unlock(); |
| 1121 | 820 | } |
| 1122 | 821 | |
| 1123 | −static unsigned long memcg_events(struct mem_cgroup *memcg, int event) | |
| 822 | +unsigned long memcg_events(struct mem_cgroup *memcg, int event) | |
| 1124 | 823 | { |
| 1125 | 824 | int i = memcg_events_index(event); |
| 1126 | 825 | |
| @@ -1130,7 +829,7 @@ static unsigned long memcg_events(struct mem_cgroup *memcg, int event) · 3 unmodified lines | ||
| 1130 | 829 | return READ_ONCE(memcg->vmstats->events[i]); |
| 1131 | 830 | } |
| 1132 | 831 | |
| 1133 | −static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) | |
| 832 | +unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) | |
| 1134 | 833 | { |
| 1135 | 834 | int i = memcg_events_index(event); |
| 1136 | 835 | |
| @@ -1140,8 +839,7 @@ static unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) · 3 unmodified lines | ||
| 1140 | 839 | return READ_ONCE(memcg->vmstats->events_local[i]); |
| 1141 | 840 | } |
| 1142 | 841 | |
| 1143 | −static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, | |
| 1144 | − int nr_pages) | |
| 842 | +void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, int nr_pages) | |
| 1145 | 843 | { |
| 1146 | 844 | /* pagein of a big page is an event. So, ignore page size */ |
| 1147 | 845 | if (nr_pages > 0) |
| @@ -1154,8 +852,8 @@ static void mem_cgroup_charge_statistics(struct mem_cgroup *memcg, · 6 unmodified lines | ||
| 1154 | 852 | __this_cpu_add(memcg->vmstats_percpu->nr_page_events, nr_pages); |
| 1155 | 853 | } |
| 1156 | 854 | |
| 1157 | −static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, | |
| 1158 | − enum mem_cgroup_events_target target) | |
| 855 | +bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, | |
| 856 | + enum mem_cgroup_events_target target) | |
| 1159 | 857 | { |
| 1160 | 858 | unsigned long val, next; |
| 1161 | 859 | |
| @@ -1179,28 +877,6 @@ static bool mem_cgroup_event_ratelimit(struct mem_cgroup *memcg, · 17 unmodified lines | ||
| 1179 | 877 | return false; |
| 1180 | 878 | } |
| 1181 | 879 | |
| 1182 | −/* | |
| 1183 | − * Check events in order. | |
| 1184 | − * | |
| 1185 | − */ | |
| 1186 | −static void memcg_check_events(struct mem_cgroup *memcg, int nid) | |
| 1187 | −{ | |
| 1188 | − if (IS_ENABLED(CONFIG_PREEMPT_RT)) | |
| 1189 | − return; | |
| 1190 | − | |
| 1191 | − /* threshold event is triggered in finer grain than soft limit */ | |
| 1192 | − if (unlikely(mem_cgroup_event_ratelimit(memcg, | |
| 1193 | − MEM_CGROUP_TARGET_THRESH))) { | |
| 1194 | − bool do_softlimit; | |
| 1195 | − | |
| 1196 | − do_softlimit = mem_cgroup_event_ratelimit(memcg, | |
| 1197 | − MEM_CGROUP_TARGET_SOFTLIMIT); | |
| 1198 | − mem_cgroup_threshold(memcg); | |
| 1199 | − if (unlikely(do_softlimit)) | |
| 1200 | − mem_cgroup_update_tree(memcg, nid); | |
| 1201 | − } | |
| 1202 | −} | |
| 1203 | − | |
| 1204 | 880 | struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p) |
| 1205 | 881 | { |
| 1206 | 882 | /* |
| @@ -1652,51 +1328,6 @@ static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg) · 445 unmodified lines | ||
| 1652 | 1328 | return margin; |
| 1653 | 1329 | } |
| 1654 | 1330 | |
| 1655 | −/* | |
| 1656 | − * A routine for checking "mem" is under move_account() or not. | |
| 1657 | − * | |
| 1658 | − * Checking a cgroup is mc.from or mc.to or under hierarchy of | |
| 1659 | − * moving cgroups. This is for waiting at high-memory pressure | |
| 1660 | − * caused by "move". | |
| 1661 | − */ | |
| 1662 | −static bool mem_cgroup_under_move(struct mem_cgroup *memcg) | |
| 1663 | −{ | |
| 1664 | − struct mem_cgroup *from; | |
| 1665 | − struct mem_cgroup *to; | |
| 1666 | − bool ret = false; | |
| 1667 | − /* | |
| 1668 | − * Unlike task_move routines, we access mc.to, mc.from not under | |
| 1669 | − * mutual exclusion by cgroup_mutex. Here, we take spinlock instead. | |
| 1670 | − */ | |
| 1671 | − spin_lock(&mc.lock); | |
| 1672 | − from = mc.from; | |
| 1673 | − to = mc.to; | |
| 1674 | − if (!from) | |
| 1675 | − goto unlock; | |
| 1676 | − | |
| 1677 | − ret = mem_cgroup_is_descendant(from, memcg) || | |
| 1678 | − mem_cgroup_is_descendant(to, memcg); | |
| 1679 | −unlock: | |
| 1680 | − spin_unlock(&mc.lock); | |
| 1681 | − return ret; | |
| 1682 | −} | |
| 1683 | − | |
| 1684 | −static bool mem_cgroup_wait_acct_move(struct mem_cgroup *memcg) | |
| 1685 | −{ | |
| 1686 | − if (mc.moving_task && current != mc.moving_task) { | |
| 1687 | − if (mem_cgroup_under_move(memcg)) { | |
| 1688 | − DEFINE_WAIT(wait); | |
| 1689 | − prepare_to_wait(&mc.waitq, &wait, TASK_INTERRUPTIBLE); | |
| 1690 | − /* moving charge context might have finished. */ | |
| 1691 | − if (mc.moving_task) | |
| 1692 | − schedule(); | |
| 1693 | − finish_wait(&mc.waitq, &wait); | |
| 1694 | − return true; | |
| 1695 | − } | |
| 1696 | − } | |
| 1697 | − return false; | |
| 1698 | −} | |
| 1699 | − | |
| 1700 | 1331 | struct memory_stat { |
| 1701 | 1332 | const char *name; |
| 1702 | 1333 | unsigned int idx; |
| @@ -1713,7 +1344,7 @@ static const struct memory_stat memory_stats[] = { · 10 unmodified lines | ||
| 1713 | 1344 | { "sock", MEMCG_SOCK }, |
| 1714 | 1345 | { "vmalloc", MEMCG_VMALLOC }, |
| 1715 | 1346 | { "shmem", NR_SHMEM }, |
| 1716 | −#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) | |
| 1347 | +#ifdef CONFIG_ZSWAP | |
| 1717 | 1348 | { "zswap", MEMCG_ZSWAP_B }, |
| 1718 | 1349 | { "zswapped", MEMCG_ZSWAPPED }, |
| 1719 | 1350 | #endif |
| @@ -1783,15 +1414,13 @@ static int memcg_page_state_output_unit(int item) · 63 unmodified lines | ||
| 1783 | 1414 | } |
| 1784 | 1415 | } |
| 1785 | 1416 | |
| 1786 | −static inline unsigned long memcg_page_state_output(struct mem_cgroup *memcg, | |
| 1787 | − int item) | |
| 1417 | +unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item) | |
| 1788 | 1418 | { |
| 1789 | 1419 | return memcg_page_state(memcg, item) * |
| 1790 | 1420 | memcg_page_state_output_unit(item); |
| 1791 | 1421 | } |
| 1792 | 1422 | |
| 1793 | −static inline unsigned long memcg_page_state_local_output( | |
| 1794 | − struct mem_cgroup *memcg, int item) | |
| 1423 | +unsigned long memcg_page_state_local_output(struct mem_cgroup *memcg, int item) | |
| 1795 | 1424 | { |
| 1796 | 1425 | return memcg_page_state_local(memcg, item) * |
| 1797 | 1426 | memcg_page_state_output_unit(item); |
| @@ -1845,20 +1474,16 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) · 47 unmodified lines | ||
| 1845 | 1474 | vm_event_name(memcg_vm_event_stat[i]), |
| 1846 | 1475 | memcg_events(memcg, memcg_vm_event_stat[i])); |
| 1847 | 1476 | } |
| 1848 | − | |
| 1849 | − /* The above should easily fit into one page */ | |
| 1850 | − WARN_ON_ONCE(seq_buf_has_overflowed(s)); | |
| 1851 | 1477 | } |
| 1852 | 1478 | |
| 1853 | −static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s); | |
| 1854 | − | |
| 1855 | 1479 | static void memory_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) |
| 1856 | 1480 | { |
| 1857 | 1481 | if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) |
| 1858 | 1482 | memcg_stat_format(memcg, s); |
| 1859 | 1483 | else |
| 1860 | 1484 | memcg1_stat_format(memcg, s); |
| 1861 | − WARN_ON_ONCE(seq_buf_has_overflowed(s)); | |
| 1485 | + if (seq_buf_has_overflowed(s)) | |
| 1486 | + pr_warn("%s: Warning, stat buffer overflow, please report\n", __func__); | |
| 1862 | 1487 | } |
| 1863 | 1488 | |
| 1864 | 1489 | /** |
| @@ -1906,6 +1531,7 @@ void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) · 41 unmodified lines | ||
| 1906 | 1531 | pr_info("swap: usage %llukB, limit %llukB, failcnt %lu\n", |
| 1907 | 1532 | K((u64)page_counter_read(&memcg->swap)), |
| 1908 | 1533 | K((u64)READ_ONCE(memcg->swap.max)), memcg->swap.failcnt); |
| 1534 | +#ifdef CONFIG_MEMCG_V1 | |
| 1909 | 1535 | else { |
| 1910 | 1536 | pr_info("memory+swap: usage %llukB, limit %llukB, failcnt %lu\n", |
| 1911 | 1537 | K((u64)page_counter_read(&memcg->memsw)), |
| @@ -1914,6 +1540,7 @@ void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) · 2 unmodified lines | ||
| 1914 | 1540 | K((u64)page_counter_read(&memcg->kmem)), |
| 1915 | 1541 | K((u64)memcg->kmem.max), memcg->kmem.failcnt); |
| 1916 | 1542 | } |
| 1543 | +#endif | |
| 1917 | 1544 | |
| 1918 | 1545 | pr_info("Memory cgroup stats for "); |
| 1919 | 1546 | pr_cont_cgroup_path(memcg->css.cgroup); |
| @@ -1979,180 +1606,6 @@ unlock: · 59 unmodified lines | ||
| 1979 | 1606 | return ret; |
| 1980 | 1607 | } |
| 1981 | 1608 | |
| 1982 | −static int mem_cgroup_soft_reclaim(struct mem_cgroup *root_memcg, | |
| 1983 | − pg_data_t *pgdat, | |
| 1984 | − gfp_t gfp_mask, | |
| 1985 | − unsigned long *total_scanned) | |
| 1986 | −{ | |
| 1987 | − struct mem_cgroup *victim = NULL; | |
| 1988 | − int total = 0; | |
| 1989 | − int loop = 0; | |
| 1990 | − unsigned long excess; | |
| 1991 | − unsigned long nr_scanned; | |
| 1992 | − struct mem_cgroup_reclaim_cookie reclaim = { | |
| 1993 | − .pgdat = pgdat, | |
| 1994 | − }; | |
| 1995 | − | |
| 1996 | − excess = soft_limit_excess(root_memcg); | |
| 1997 | − | |
| 1998 | − while (1) { | |
| 1999 | − victim = mem_cgroup_iter(root_memcg, victim, &reclaim); | |
| 2000 | − if (!victim) { | |
| 2001 | − loop++; | |
| 2002 | − if (loop >= 2) { | |
| 2003 | − /* | |
| 2004 | − * If we have not been able to reclaim | |
| 2005 | − * anything, it might because there are | |
| 2006 | − * no reclaimable pages under this hierarchy | |
| 2007 | − */ | |
| 2008 | − if (!total) | |
| 2009 | − break; | |
| 2010 | − /* | |
| 2011 | − * We want to do more targeted reclaim. | |
| 2012 | − * excess >> 2 is not to excessive so as to | |
| 2013 | − * reclaim too much, nor too less that we keep | |
| 2014 | − * coming back to reclaim from this cgroup | |
| 2015 | − */ | |
| 2016 | − if (total >= (excess >> 2) || | |
| 2017 | − (loop > MEM_CGROUP_MAX_RECLAIM_LOOPS)) | |
| 2018 | − break; | |
| 2019 | − } | |
| 2020 | − continue; | |
| 2021 | − } | |
| 2022 | − total += mem_cgroup_shrink_node(victim, gfp_mask, false, | |
| 2023 | − pgdat, &nr_scanned); | |
| 2024 | − *total_scanned += nr_scanned; | |
| 2025 | − if (!soft_limit_excess(root_memcg)) | |
| 2026 | − break; | |
| 2027 | − } | |
| 2028 | − mem_cgroup_iter_break(root_memcg, victim); | |
| 2029 | − return total; | |
| 2030 | −} | |
| 2031 | − | |
| 2032 | −#ifdef CONFIG_LOCKDEP | |
| 2033 | −static struct lockdep_map memcg_oom_lock_dep_map = { | |
| 2034 | − .name = "memcg_oom_lock", | |
| 2035 | −}; | |
| 2036 | −#endif | |
| 2037 | − | |
| 2038 | −static DEFINE_SPINLOCK(memcg_oom_lock); | |
| 2039 | − | |
| 2040 | −/* | |
| 2041 | − * Check OOM-Killer is already running under our hierarchy. | |
| 2042 | − * If someone is running, return false. | |
| 2043 | − */ | |
| 2044 | −static bool mem_cgroup_oom_trylock(struct mem_cgroup *memcg) | |
| 2045 | −{ | |
| 2046 | − struct mem_cgroup *iter, *failed = NULL; | |
| 2047 | − | |
| 2048 | − spin_lock(&memcg_oom_lock); | |
| 2049 | − | |
| 2050 | − for_each_mem_cgroup_tree(iter, memcg) { | |
| 2051 | − if (iter->oom_lock) { | |
| 2052 | − /* | |
| 2053 | − * this subtree of our hierarchy is already locked | |
| 2054 | − * so we cannot give a lock. | |
| 2055 | − */ | |
| 2056 | − failed = iter; | |
| 2057 | − mem_cgroup_iter_break(memcg, iter); | |
| 2058 | − break; | |
| 2059 | − } else | |
| 2060 | − iter->oom_lock = true; | |
| 2061 | − } | |
| 2062 | − | |
| 2063 | − if (failed) { | |
| 2064 | − /* | |
| 2065 | − * OK, we failed to lock the whole subtree so we have | |
| 2066 | − * to clean up what we set up to the failing subtree | |
| 2067 | − */ | |
| 2068 | − for_each_mem_cgroup_tree(iter, memcg) { | |
| 2069 | − if (iter == failed) { | |
| 2070 | − mem_cgroup_iter_break(memcg, iter); | |
| 2071 | − break; | |
| 2072 | − } | |
| 2073 | − iter->oom_lock = false; | |
| 2074 | − } | |
| 2075 | − } else | |
| 2076 | − mutex_acquire(&memcg_oom_lock_dep_map, 0, 1, _RET_IP_); | |
| 2077 | − | |
| 2078 | − spin_unlock(&memcg_oom_lock); | |
| 2079 | − | |
| 2080 | − return !failed; | |
| 2081 | −} | |
| 2082 | − | |
| 2083 | −static void mem_cgroup_oom_unlock(struct mem_cgroup *memcg) | |
| 2084 | −{ | |
| 2085 | − struct mem_cgroup *iter; | |
| 2086 | − | |
| 2087 | − spin_lock(&memcg_oom_lock); | |
| 2088 | − mutex_release(&memcg_oom_lock_dep_map, _RET_IP_); | |
| 2089 | − for_each_mem_cgroup_tree(iter, memcg) | |
| 2090 | − iter->oom_lock = false; | |
| 2091 | − spin_unlock(&memcg_oom_lock); | |
| 2092 | −} | |
| 2093 | − | |
| 2094 | −static void mem_cgroup_mark_under_oom(struct mem_cgroup *memcg) | |
| 2095 | −{ | |
| 2096 | − struct mem_cgroup *iter; | |
| 2097 | − | |
| 2098 | − spin_lock(&memcg_oom_lock); | |
| 2099 | − for_each_mem_cgroup_tree(iter, memcg) | |
| 2100 | − iter->under_oom++; | |
| 2101 | − spin_unlock(&memcg_oom_lock); | |
| 2102 | −} | |
| 2103 | − | |
| 2104 | −static void mem_cgroup_unmark_under_oom(struct mem_cgroup *memcg) | |
| 2105 | −{ | |
| 2106 | − struct mem_cgroup *iter; | |
| 2107 | − | |
| 2108 | − /* | |
| 2109 | − * Be careful about under_oom underflows because a child memcg | |
| 2110 | − * could have been added after mem_cgroup_mark_under_oom. | |
| 2111 | − */ | |
| 2112 | − spin_lock(&memcg_oom_lock); | |
| 2113 | − for_each_mem_cgroup_tree(iter, memcg) | |
| 2114 | − if (iter->under_oom > 0) | |
| 2115 | − iter->under_oom--; | |
| 2116 | − spin_unlock(&memcg_oom_lock); | |
| 2117 | −} | |
| 2118 | − | |
| 2119 | −static DECLARE_WAIT_QUEUE_HEAD(memcg_oom_waitq); | |
| 2120 | − | |
| 2121 | −struct oom_wait_info { | |
| 2122 | − struct mem_cgroup *memcg; | |
| 2123 | − wait_queue_entry_t wait; | |
| 2124 | −}; | |
| 2125 | − | |
| 2126 | −static int memcg_oom_wake_function(wait_queue_entry_t *wait, | |
| 2127 | − unsigned mode, int sync, void *arg) | |
| 2128 | −{ | |
| 2129 | − struct mem_cgroup *wake_memcg = (struct mem_cgroup *)arg; | |
| 2130 | − struct mem_cgroup *oom_wait_memcg; | |
| 2131 | − struct oom_wait_info *oom_wait_info; | |
| 2132 | − | |
| 2133 | − oom_wait_info = container_of(wait, struct oom_wait_info, wait); | |
| 2134 | − oom_wait_memcg = oom_wait_info->memcg; | |
| 2135 | − | |
| 2136 | − if (!mem_cgroup_is_descendant(wake_memcg, oom_wait_memcg) && | |
| 2137 | − !mem_cgroup_is_descendant(oom_wait_memcg, wake_memcg)) | |
| 2138 | − return 0; | |
| 2139 | − return autoremove_wake_function(wait, mode, sync, arg); | |
| 2140 | −} | |
| 2141 | − | |
| 2142 | −static void memcg_oom_recover(struct mem_cgroup *memcg) | |
| 2143 | −{ | |
| 2144 | − /* | |
| 2145 | − * For the following lockless ->under_oom test, the only required | |
| 2146 | − * guarantee is that it must see the state asserted by an OOM when | |
| 2147 | − * this function is called as a result of userland actions | |
| 2148 | − * triggered by the notification of the OOM. This is trivially | |
| 2149 | − * achieved by invoking mem_cgroup_mark_under_oom() before | |
| 2150 | − * triggering notification. | |
| 2151 | − */ | |
| 2152 | − if (memcg && memcg->under_oom) | |
| 2153 | − __wake_up(&memcg_oom_waitq, TASK_NORMAL, 0, memcg); | |
| 2154 | −} | |
| 2155 | − | |
| 2156 | 1609 | /* |
| 2157 | 1610 | * Returns true if successfully killed one or more processes. Though in some |
| 2158 | 1611 | * corner cases it can return true even without killing any process. |
| @@ -2166,104 +1619,16 @@ static bool mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order) · 7 unmodified lines | ||
| 2166 | 1619 | |
| 2167 | 1620 | memcg_memory_event(memcg, MEMCG_OOM); |
| 2168 | 1621 | |
| 2169 | − /* | |
| 2170 | − * We are in the middle of the charge context here, so we | |
| 2171 | − * don't want to block when potentially sitting on a callstack | |
| 2172 | − * that holds all kinds of filesystem and mm locks. | |
| 2173 | − * | |
| 2174 | − * cgroup1 allows disabling the OOM killer and waiting for outside | |
| 2175 | − * handling until the charge can succeed; remember the context and put | |
| 2176 | − * the task to sleep at the end of the page fault when all locks are | |
| 2177 | − * released. | |
| 2178 | − * | |
| 2179 | − * On the other hand, in-kernel OOM killer allows for an async victim | |
| 2180 | − * memory reclaim (oom_reaper) and that means that we are not solely | |
| 2181 | − * relying on the oom victim to make a forward progress and we can | |
| 2182 | − * invoke the oom killer here. | |
| 2183 | − * | |
| 2184 | − * Please note that mem_cgroup_out_of_memory might fail to find a | |
| 2185 | − * victim and then we have to bail out from the charge path. | |
| 2186 | − */ | |
| 2187 | − if (READ_ONCE(memcg->oom_kill_disable)) { | |
| 2188 | − if (current->in_user_fault) { | |
| 2189 | − css_get(&memcg->css); | |
| 2190 | − current->memcg_in_oom = memcg; | |
| 2191 | − } | |
| 1622 | + if (!memcg1_oom_prepare(memcg, &locked)) | |
| 2192 | 1623 | return false; |
| 2193 | − } | |
| 2194 | − | |
| 2195 | − mem_cgroup_mark_under_oom(memcg); | |
| 2196 | − | |
| 2197 | − locked = mem_cgroup_oom_trylock(memcg); | |
| 2198 | − | |
| 2199 | − if (locked) | |
| 2200 | − mem_cgroup_oom_notify(memcg); | |
| 2201 | 1624 | |
| 2202 | − mem_cgroup_unmark_under_oom(memcg); | |
| 2203 | 1625 | ret = mem_cgroup_out_of_memory(memcg, mask, order); |
| 2204 | 1626 | |
| 2205 | − if (locked) | |
| 2206 | − mem_cgroup_oom_unlock(memcg); | |
| 1627 | + memcg1_oom_finish(memcg, locked); | |
| 2207 | 1628 | |
| 2208 | 1629 | return ret; |
| 2209 | 1630 | } |
| 2210 | 1631 | |
| 2211 | −/** | |
| 2212 | − * mem_cgroup_oom_synchronize - complete memcg OOM handling | |
| 2213 | − * @handle: actually kill/wait or just clean up the OOM state | |
| 2214 | − * | |
| 2215 | − * This has to be called at the end of a page fault if the memcg OOM | |
| 2216 | − * handler was enabled. | |
| 2217 | − * | |
| 2218 | − * Memcg supports userspace OOM handling where failed allocations must | |
| 2219 | − * sleep on a waitqueue until the userspace task resolves the | |
| 2220 | − * situation. Sleeping directly in the charge context with all kinds | |
| 2221 | − * of locks held is not a good idea, instead we remember an OOM state | |
| 2222 | − * in the task and mem_cgroup_oom_synchronize() has to be called at | |
| 2223 | − * the end of the page fault to complete the OOM handling. | |
| 2224 | − * | |
| 2225 | − * Returns %true if an ongoing memcg OOM situation was detected and | |
| 2226 | − * completed, %false otherwise. | |
| 2227 | − */ | |
| 2228 | −bool mem_cgroup_oom_synchronize(bool handle) | |
| 2229 | −{ | |
| 2230 | − struct mem_cgroup *memcg = current->memcg_in_oom; | |
| 2231 | − struct oom_wait_info owait; | |
| 2232 | − bool locked; | |
| 2233 | − | |
| 2234 | − /* OOM is global, do not handle */ | |
| 2235 | − if (!memcg) | |
| 2236 | − return false; | |
| 2237 | − | |
| 2238 | − if (!handle) | |
| 2239 | − goto cleanup; | |
| 2240 | − | |
| 2241 | − owait.memcg = memcg; | |
| 2242 | − owait.wait.flags = 0; | |
| 2243 | − owait.wait.func = memcg_oom_wake_function; | |
| 2244 | − owait.wait.private = current; | |
| 2245 | − INIT_LIST_HEAD(&owait.wait.entry); | |
| 2246 | − | |
| 2247 | − prepare_to_wait(&memcg_oom_waitq, &owait.wait, TASK_KILLABLE); | |
| 2248 | − mem_cgroup_mark_under_oom(memcg); | |
| 2249 | − | |
| 2250 | − locked = mem_cgroup_oom_trylock(memcg); | |
| 2251 | − | |
| 2252 | − if (locked) | |
| 2253 | − mem_cgroup_oom_notify(memcg); | |
| 2254 | − | |
| 2255 | − schedule(); | |
| 2256 | − mem_cgroup_unmark_under_oom(memcg); | |
| 2257 | − finish_wait(&memcg_oom_waitq, &owait.wait); | |
| 2258 | − | |
| 2259 | − if (locked) | |
| 2260 | − mem_cgroup_oom_unlock(memcg); | |
| 2261 | −cleanup: | |
| 2262 | − current->memcg_in_oom = NULL; | |
| 2263 | − css_put(&memcg->css); | |
| 2264 | − return true; | |
| 2265 | −} | |
| 2266 | − | |
| 2267 | 1632 | /** |
| 2268 | 1633 | * mem_cgroup_get_oom_group - get a memory cgroup to clean up after OOM |
| 2269 | 1634 | * @victim: task to be killed by the OOM killer |
| @@ -2328,99 +1693,16 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg) · 58 unmodified lines | ||
| 2328 | 1693 | pr_cont(" are going to be killed due to memory.oom.group set\n"); |
| 2329 | 1694 | } |
| 2330 | 1695 | |
| 2331 | −/** | |
| 2332 | − * folio_memcg_lock - Bind a folio to its memcg. | |
| 2333 | − * @folio: The folio. | |
| 2334 | − * | |
| 2335 | − * This function prevents unlocked LRU folios from being moved to | |
| 2336 | − * another cgroup. | |
| 2337 | − * | |
| 2338 | − * It ensures lifetime of the bound memcg. The caller is responsible | |
| 2339 | − * for the lifetime of the folio. | |
| 2340 | − */ | |
| 2341 | −void folio_memcg_lock(struct folio *folio) | |
| 2342 | −{ | |
| 2343 | − struct mem_cgroup *memcg; | |
| 2344 | − unsigned long flags; | |
| 2345 | − | |
| 2346 | − /* | |
| 2347 | − * The RCU lock is held throughout the transaction. The fast | |
| 2348 | − * path can get away without acquiring the memcg->move_lock | |
| 2349 | − * because page moving starts with an RCU grace period. | |
| 2350 | − */ | |
| 2351 | − rcu_read_lock(); | |
| 2352 | − | |
| 2353 | − if (mem_cgroup_disabled()) | |
| 2354 | − return; | |
| 2355 | −again: | |
| 2356 | − memcg = folio_memcg(folio); | |
| 2357 | − if (unlikely(!memcg)) | |
| 2358 | − return; | |
| 2359 | − | |
| 2360 | −#ifdef CONFIG_PROVE_LOCKING | |
| 2361 | − local_irq_save(flags); | |
| 2362 | − might_lock(&memcg->move_lock); | |
| 2363 | − local_irq_restore(flags); | |
| 2364 | −#endif | |
| 2365 | − | |
| 2366 | − if (atomic_read(&memcg->moving_account) <= 0) | |
| 2367 | − return; | |
| 2368 | − | |
| 2369 | − spin_lock_irqsave(&memcg->move_lock, flags); | |
| 2370 | − if (memcg != folio_memcg(folio)) { | |
| 2371 | − spin_unlock_irqrestore(&memcg->move_lock, flags); | |
| 2372 | − goto again; | |
| 2373 | − } | |
| 2374 | − | |
| 2375 | − /* | |
| 2376 | − * When charge migration first begins, we can have multiple | |
| 2377 | − * critical sections holding the fast-path RCU lock and one | |
| 2378 | − * holding the slowpath move_lock. Track the task who has the | |
| 2379 | − * move_lock for folio_memcg_unlock(). | |
| 2380 | − */ | |
| 2381 | − memcg->move_lock_task = current; | |
| 2382 | − memcg->move_lock_flags = flags; | |
| 2383 | −} | |
| 2384 | − | |
| 2385 | −static void __folio_memcg_unlock(struct mem_cgroup *memcg) | |
| 2386 | −{ | |
| 2387 | − if (memcg && memcg->move_lock_task == current) { | |
| 2388 | − unsigned long flags = memcg->move_lock_flags; | |
| 2389 | − | |
| 2390 | − memcg->move_lock_task = NULL; | |
| 2391 | − memcg->move_lock_flags = 0; | |
| 2392 | − | |
| 2393 | − spin_unlock_irqrestore(&memcg->move_lock, flags); | |
| 2394 | − } | |
| 2395 | − | |
| 2396 | − rcu_read_unlock(); | |
| 2397 | −} | |
| 2398 | − | |
| 2399 | −/** | |
| 2400 | − * folio_memcg_unlock - Release the binding between a folio and its memcg. | |
| 2401 | − * @folio: The folio. | |
| 2402 | − * | |
| 2403 | − * This releases the binding created by folio_memcg_lock(). This does | |
| 2404 | − * not change the accounting of this folio to its memcg, but it does | |
| 2405 | − * permit others to change it. | |
| 2406 | − */ | |
| 2407 | −void folio_memcg_unlock(struct folio *folio) | |
| 2408 | −{ | |
| 2409 | − __folio_memcg_unlock(folio_memcg(folio)); | |
| 2410 | −} | |
| 2411 | − | |
| 2412 | 1696 | struct memcg_stock_pcp { |
| 2413 | 1697 | local_lock_t stock_lock; |
| 2414 | 1698 | struct mem_cgroup *cached; /* this never be root cgroup */ |
| 2415 | 1699 | unsigned int nr_pages; |
| 2416 | 1700 | |
| 2417 | −#ifdef CONFIG_MEMCG_KMEM | |
| 2418 | 1701 | struct obj_cgroup *cached_objcg; |
| 2419 | 1702 | struct pglist_data *cached_pgdat; |
| 2420 | 1703 | unsigned int nr_bytes; |
| 2421 | 1704 | int nr_slab_reclaimable_b; |
| 2422 | 1705 | int nr_slab_unreclaimable_b; |
| 2423 | −#endif | |
| 2424 | 1706 | |
| 2425 | 1707 | struct work_struct work; |
| 2426 | 1708 | unsigned long flags; |
| @@ -2431,26 +1713,9 @@ static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { · 4 unmodified lines | ||
| 2431 | 1713 | }; |
| 2432 | 1714 | static DEFINE_MUTEX(percpu_charge_mutex); |
| 2433 | 1715 | |
| 2434 | −#ifdef CONFIG_MEMCG_KMEM | |
| 2435 | 1716 | static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock); |
| 2436 | 1717 | static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, |
| 2437 | 1718 | struct mem_cgroup *root_memcg); |
| 2438 | −static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages); | |
| 2439 | − | |
| 2440 | −#else | |
| 2441 | −static inline struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock) | |
| 2442 | −{ | |
| 2443 | − return NULL; | |
| 2444 | −} | |
| 2445 | −static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, | |
| 2446 | − struct mem_cgroup *root_memcg) | |
| 2447 | −{ | |
| 2448 | − return false; | |
| 2449 | −} | |
| 2450 | −static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages) | |
| 2451 | −{ | |
| 2452 | −} | |
| 2453 | −#endif | |
| 2454 | 1719 | |
| 2455 | 1720 | /** |
| 2456 | 1721 | * consume_stock: Try to consume stocked charge on this cpu. |
| @@ -2567,7 +1832,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) · 110 unmodified lines | ||
| 2567 | 1832 | * Drains all per-CPU charge caches for given root_memcg resp. subtree |
| 2568 | 1833 | * of the hierarchy under it. |
| 2569 | 1834 | */ |
| 2570 | −static void drain_all_stock(struct mem_cgroup *root_memcg) | |
| 1835 | +void drain_all_stock(struct mem_cgroup *root_memcg) | |
| 2571 | 1836 | { |
| 2572 | 1837 | int cpu, curcpu; |
| 2573 | 1838 | |
| @@ -2636,7 +1901,8 @@ static unsigned long reclaim_high(struct mem_cgroup *memcg, · 62 unmodified lines | ||
| 2636 | 1901 | psi_memstall_enter(&pflags); |
| 2637 | 1902 | nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages, |
| 2638 | 1903 | gfp_mask, |
| 2639 | − MEMCG_RECLAIM_MAY_SWAP); | |
| 1904 | + MEMCG_RECLAIM_MAY_SWAP, | |
| 1905 | + NULL); | |
| 2640 | 1906 | psi_memstall_leave(&pflags); |
| 2641 | 1907 | } while ((memcg = parent_mem_cgroup(memcg)) && |
| 2642 | 1908 | !mem_cgroup_is_root(memcg)); |
| @@ -2887,8 +2153,8 @@ out: · 244 unmodified lines | ||
| 2887 | 2153 | css_put(&memcg->css); |
| 2888 | 2154 | } |
| 2889 | 2155 | |
| 2890 | −static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, | |
| 2891 | − unsigned int nr_pages) | |
| 2156 | +int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, | |
| 2157 | + unsigned int nr_pages) | |
| 2892 | 2158 | { |
| 2893 | 2159 | unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages); |
| 2894 | 2160 | int nr_retries = MAX_RECLAIM_RETRIES; |
| @@ -2942,7 +2208,7 @@ retry: · 47 unmodified lines | ||
| 2942 | 2208 | |
| 2943 | 2209 | psi_memstall_enter(&pflags); |
| 2944 | 2210 | nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages, |
| 2945 | − gfp_mask, reclaim_options); | |
| 2211 | + gfp_mask, reclaim_options, NULL); | |
| 2946 | 2212 | psi_memstall_leave(&pflags); |
| 2947 | 2213 | |
| 2948 | 2214 | if (mem_cgroup_margin(mem_over_limit) >= nr_pages) |
| @@ -2971,7 +2237,7 @@ retry: · 22 unmodified lines | ||
| 2971 | 2237 | * At task move, charge accounts can be doubly counted. So, it's |
| 2972 | 2238 | * better to wait until the end of task_move if something is going on. |
| 2973 | 2239 | */ |
| 2974 | − if (mem_cgroup_wait_acct_move(mem_over_limit)) | |
| 2240 | + if (memcg1_wait_acct_move(mem_over_limit)) | |
| 2975 | 2241 | goto retry; |
| 2976 | 2242 | |
| 2977 | 2243 | if (nr_retries--) |
| @@ -3083,15 +2349,6 @@ done_restock: · 105 unmodified lines | ||
| 3083 | 2349 | return 0; |
| 3084 | 2350 | } |
| 3085 | 2351 | |
| 3086 | −static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, | |
| 3087 | − unsigned int nr_pages) | |
| 3088 | −{ | |
| 3089 | − if (mem_cgroup_is_root(memcg)) | |
| 3090 | − return 0; | |
| 3091 | − | |
| 3092 | − return try_charge_memcg(memcg, gfp_mask, nr_pages); | |
| 3093 | −} | |
| 3094 | − | |
| 3095 | 2352 | /** |
| 3096 | 2353 | * mem_cgroup_cancel_charge() - cancel an uncommitted try_charge() call. |
| 3097 | 2354 | * @memcg: memcg previously charged. |
| @@ -3134,12 +2391,10 @@ void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg) · 36 unmodified lines | ||
| 3134 | 2391 | |
| 3135 | 2392 | local_irq_disable(); |
| 3136 | 2393 | mem_cgroup_charge_statistics(memcg, folio_nr_pages(folio)); |
| 3137 | − memcg_check_events(memcg, folio_nid(folio)); | |
| 2394 | + memcg1_check_events(memcg, folio_nid(folio)); | |
| 3138 | 2395 | local_irq_enable(); |
| 3139 | 2396 | } |
| 3140 | 2397 | |
| 3141 | −#ifdef CONFIG_MEMCG_KMEM | |
| 3142 | − | |
| 3143 | 2398 | static inline void __mod_objcg_mlstate(struct obj_cgroup *objcg, |
| 3144 | 2399 | struct pglist_data *pgdat, |
| 3145 | 2400 | enum node_stat_item idx, int nr) |
| @@ -3367,18 +2622,6 @@ struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio) · 221 unmodified lines | ||
| 3367 | 2622 | return objcg; |
| 3368 | 2623 | } |
| 3369 | 2624 | |
| 3370 | −static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages) | |
| 3371 | −{ | |
| 3372 | − mod_memcg_state(memcg, MEMCG_KMEM, nr_pages); | |
| 3373 | − if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) { | |
| 3374 | − if (nr_pages > 0) | |
| 3375 | − page_counter_charge(&memcg->kmem, nr_pages); | |
| 3376 | − else | |
| 3377 | − page_counter_uncharge(&memcg->kmem, -nr_pages); | |
| 3378 | − } | |
| 3379 | −} | |
| 3380 | − | |
| 3381 | − | |
| 3382 | 2625 | /* |
| 3383 | 2626 | * obj_cgroup_uncharge_pages: uncharge a number of kernel pages from a objcg |
| 3384 | 2627 | * @objcg: object cgroup to uncharge |
| @@ -3391,7 +2634,8 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, · 6 unmodified lines | ||
| 3391 | 2634 | |
| 3392 | 2635 | memcg = get_mem_cgroup_from_objcg(objcg); |
| 3393 | 2636 | |
| 3394 | − memcg_account_kmem(memcg, -nr_pages); | |
| 2637 | + mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages); | |
| 2638 | + memcg1_account_kmem(memcg, -nr_pages); | |
| 3395 | 2639 | refill_stock(memcg, nr_pages); |
| 3396 | 2640 | |
| 3397 | 2641 | css_put(&memcg->css); |
| @@ -3417,7 +2661,8 @@ static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, · 19 unmodified lines | ||
| 3417 | 2661 | if (ret) |
| 3418 | 2662 | goto out; |
| 3419 | 2663 | |
| 3420 | − memcg_account_kmem(memcg, nr_pages); | |
| 2664 | + mod_memcg_state(memcg, MEMCG_KMEM, nr_pages); | |
| 2665 | + memcg1_account_kmem(memcg, nr_pages); | |
| 3421 | 2666 | out: |
| 3422 | 2667 | css_put(&memcg->css); |
| 3423 | 2668 | |
| @@ -3570,7 +2815,8 @@ static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock) · 146 unmodified lines | ||
| 3570 | 2815 | |
| 3571 | 2816 | memcg = get_mem_cgroup_from_objcg(old); |
| 3572 | 2817 | |
| 3573 | − memcg_account_kmem(memcg, -nr_pages); | |
| 2818 | + mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages); | |
| 2819 | + memcg1_account_kmem(memcg, -nr_pages); | |
| 3574 | 2820 | __refill_stock(memcg, nr_pages); |
| 3575 | 2821 | |
| 3576 | 2822 | css_put(&memcg->css); |
| @@ -3804,10 +3050,9 @@ void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, · 227 unmodified lines | ||
| 3804 | 3050 | obj_cgroup_put(objcg); |
| 3805 | 3051 | } |
| 3806 | 3052 | } |
| 3807 | −#endif /* CONFIG_MEMCG_KMEM */ | |
| 3808 | 3053 | |
| 3809 | 3054 | /* |
| 3810 | − * Because page_memcg(head) is not set on tails, set it now. | |
| 3055 | + * Because folio_memcg(head) is not set on tails, set it now. | |
| 3811 | 3056 | */ |
| 3812 | 3057 | void split_page_memcg(struct page *head, int old_order, int new_order) |
| 3813 | 3058 | { |
| @@ -3829,240 +3074,7 @@ void split_page_memcg(struct page *head, int old_order, int new_order) · 15 unmodified lines | ||
| 3829 | 3074 | css_get_many(&memcg->css, old_nr / new_nr - 1); |
| 3830 | 3075 | } |
| 3831 | 3076 | |
| 3832 | −#ifdef CONFIG_SWAP | |
| 3833 | −/** | |
| 3834 | − * mem_cgroup_move_swap_account - move swap charge and swap_cgroup's record. | |
| 3835 | − * @entry: swap entry to be moved | |
| 3836 | − * @from: mem_cgroup which the entry is moved from | |
| 3837 | − * @to: mem_cgroup which the entry is moved to | |
| 3838 | − * | |
| 3839 | − * It succeeds only when the swap_cgroup's record for this entry is the same | |
| 3840 | − * as the mem_cgroup's id of @from. | |
| 3841 | − * | |
| 3842 | − * Returns 0 on success, -EINVAL on failure. | |
| 3843 | − * | |
| 3844 | − * The caller must have charged to @to, IOW, called page_counter_charge() about | |
| 3845 | − * both res and memsw, and called css_get(). | |
| 3846 | − */ | |
| 3847 | −static int mem_cgroup_move_swap_account(swp_entry_t entry, | |
| 3848 | − struct mem_cgroup *from, struct mem_cgroup *to) | |
| 3849 | −{ | |
| 3850 | − unsigned short old_id, new_id; | |
| 3851 | − | |
| 3852 | − old_id = mem_cgroup_id(from); | |
| 3853 | − new_id = mem_cgroup_id(to); | |
| 3854 | − | |
| 3855 | − if (swap_cgroup_cmpxchg(entry, old_id, new_id) == old_id) { | |
| 3856 | − mod_memcg_state(from, MEMCG_SWAP, -1); | |
| 3857 | − mod_memcg_state(to, MEMCG_SWAP, 1); | |
| 3858 | − return 0; | |
| 3859 | − } | |
| 3860 | − return -EINVAL; | |
| 3861 | −} | |
| 3862 | −#else | |
| 3863 | −static inline int mem_cgroup_move_swap_account(swp_entry_t entry, | |
| 3864 | − struct mem_cgroup *from, struct mem_cgroup *to) | |
| 3865 | −{ | |
| 3866 | − return -EINVAL; | |
| 3867 | −} | |
| 3868 | −#endif | |
| 3869 | − | |
| 3870 | −static DEFINE_MUTEX(memcg_max_mutex); | |
| 3871 | − | |
| 3872 | −static int mem_cgroup_resize_max(struct mem_cgroup *memcg, | |
| 3873 | − unsigned long max, bool memsw) | |
| 3874 | −{ | |
| 3875 | − bool enlarge = false; | |
| 3876 | − bool drained = false; | |
| 3877 | − int ret; | |
| 3878 | − bool limits_invariant; | |
| 3879 | − struct page_counter *counter = memsw ? &memcg->memsw : &memcg->memory; | |
| 3880 | − | |
| 3881 | − do { | |
| 3882 | − if (signal_pending(current)) { | |
| 3883 | − ret = -EINTR; | |
| 3884 | − break; | |
| 3885 | − } | |
| 3886 | − | |
| 3887 | − mutex_lock(&memcg_max_mutex); | |
| 3888 | − /* | |
| 3889 | − * Make sure that the new limit (memsw or memory limit) doesn't | |
| 3890 | − * break our basic invariant rule memory.max <= memsw.max. | |
| 3891 | − */ | |
| 3892 | − limits_invariant = memsw ? max >= READ_ONCE(memcg->memory.max) : | |
| 3893 | − max <= memcg->memsw.max; | |
| 3894 | − if (!limits_invariant) { | |
| 3895 | − mutex_unlock(&memcg_max_mutex); | |
| 3896 | − ret = -EINVAL; | |
| 3897 | − break; | |
| 3898 | − } | |
| 3899 | − if (max > counter->max) | |
| 3900 | − enlarge = true; | |
| 3901 | − ret = page_counter_set_max(counter, max); | |
| 3902 | − mutex_unlock(&memcg_max_mutex); | |
| 3903 | − | |
| 3904 | − if (!ret) | |
| 3905 | − break; | |
| 3906 | − | |
| 3907 | − if (!drained) { | |
| 3908 | − drain_all_stock(memcg); | |
| 3909 | − drained = true; | |
| 3910 | − continue; | |
| 3911 | − } | |
| 3912 | − | |
| 3913 | − if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, | |
| 3914 | − memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP)) { | |
| 3915 | − ret = -EBUSY; | |
| 3916 | − break; | |
| 3917 | − } | |
| 3918 | − } while (true); | |
| 3919 | − | |
| 3920 | − if (!ret && enlarge) | |
| 3921 | − memcg_oom_recover(memcg); | |
| 3922 | − | |
| 3923 | − return ret; | |
| 3924 | −} | |
| 3925 | − | |
| 3926 | −unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, | |
| 3927 | − gfp_t gfp_mask, | |
| 3928 | − unsigned long *total_scanned) | |
| 3929 | −{ | |
| 3930 | − unsigned long nr_reclaimed = 0; | |
| 3931 | − struct mem_cgroup_per_node *mz, *next_mz = NULL; | |
| 3932 | − unsigned long reclaimed; | |
| 3933 | − int loop = 0; | |
| 3934 | − struct mem_cgroup_tree_per_node *mctz; | |
| 3935 | − unsigned long excess; | |
| 3936 | − | |
| 3937 | − if (lru_gen_enabled()) | |
| 3938 | − return 0; | |
| 3939 | − | |
| 3940 | − if (order > 0) | |
| 3941 | − return 0; | |
| 3942 | − | |
| 3943 | − mctz = soft_limit_tree.rb_tree_per_node[pgdat->node_id]; | |
| 3944 | − | |
| 3945 | − /* | |
| 3946 | − * Do not even bother to check the largest node if the root | |
| 3947 | − * is empty. Do it lockless to prevent lock bouncing. Races | |
| 3948 | − * are acceptable as soft limit is best effort anyway. | |
| 3949 | − */ | |
| 3950 | − if (!mctz || RB_EMPTY_ROOT(&mctz->rb_root)) | |
| 3951 | − return 0; | |
| 3952 | − | |
| 3953 | − /* | |
| 3954 | − * This loop can run a while, specially if mem_cgroup's continuously | |
| 3955 | − * keep exceeding their soft limit and putting the system under | |
| 3956 | − * pressure | |
| 3957 | − */ | |
| 3958 | − do { | |
| 3959 | − if (next_mz) | |
| 3960 | − mz = next_mz; | |
| 3961 | − else | |
| 3962 | − mz = mem_cgroup_largest_soft_limit_node(mctz); | |
| 3963 | − if (!mz) | |
| 3964 | − break; | |
| 3965 | − | |
| 3966 | − reclaimed = mem_cgroup_soft_reclaim(mz->memcg, pgdat, | |
| 3967 | − gfp_mask, total_scanned); | |
| 3968 | − nr_reclaimed += reclaimed; | |
| 3969 | − spin_lock_irq(&mctz->lock); | |
| 3970 | − | |
| 3971 | − /* | |
| 3972 | − * If we failed to reclaim anything from this memory cgroup | |
| 3973 | − * it is time to move on to the next cgroup | |
| 3974 | − */ | |
| 3975 | − next_mz = NULL; | |
| 3976 | − if (!reclaimed) | |
| 3977 | − next_mz = __mem_cgroup_largest_soft_limit_node(mctz); | |
| 3978 | − | |
| 3979 | − excess = soft_limit_excess(mz->memcg); | |
| 3980 | − /* | |
| 3981 | − * One school of thought says that we should not add | |
| 3982 | − * back the node to the tree if reclaim returns 0. | |
| 3983 | − * But our reclaim could return 0, simply because due | |
| 3984 | − * to priority we are exposing a smaller subset of | |
| 3985 | − * memory to reclaim from. Consider this as a longer | |
| 3986 | − * term TODO. | |
| 3987 | − */ | |
| 3988 | − /* If excess == 0, no tree ops */ | |
| 3989 | − __mem_cgroup_insert_exceeded(mz, mctz, excess); | |
| 3990 | − spin_unlock_irq(&mctz->lock); | |
| 3991 | − css_put(&mz->memcg->css); | |
| 3992 | − loop++; | |
| 3993 | − /* | |
| 3994 | − * Could not reclaim anything and there are no more | |
| 3995 | − * mem cgroups to try or we seem to be looping without | |
| 3996 | − * reclaiming anything. | |
| 3997 | − */ | |
| 3998 | − if (!nr_reclaimed && | |
| 3999 | − (next_mz == NULL || | |
| 4000 | − loop > MEM_CGROUP_MAX_SOFT_LIMIT_RECLAIM_LOOPS)) | |
| 4001 | − break; | |
| 4002 | − } while (!nr_reclaimed); | |
| 4003 | − if (next_mz) | |
| 4004 | − css_put(&next_mz->memcg->css); | |
| 4005 | − return nr_reclaimed; | |
| 4006 | −} | |
| 4007 | − | |
| 4008 | −/* | |
| 4009 | − * Reclaims as many pages from the given memcg as possible. | |
| 4010 | − * | |
| 4011 | − * Caller is responsible for holding css reference for memcg. | |
| 4012 | − */ | |
| 4013 | −static int mem_cgroup_force_empty(struct mem_cgroup *memcg) | |
| 4014 | −{ | |
| 4015 | − int nr_retries = MAX_RECLAIM_RETRIES; | |
| 4016 | − | |
| 4017 | − /* we call try-to-free pages for make this cgroup empty */ | |
| 4018 | − lru_add_drain_all(); | |
| 4019 | − | |
| 4020 | − drain_all_stock(memcg); | |
| 4021 | − | |
| 4022 | − /* try to free all pages in this cgroup */ | |
| 4023 | − while (nr_retries && page_counter_read(&memcg->memory)) { | |
| 4024 | − if (signal_pending(current)) | |
| 4025 | − return -EINTR; | |
| 4026 | − | |
| 4027 | − if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, | |
| 4028 | − MEMCG_RECLAIM_MAY_SWAP)) | |
| 4029 | − nr_retries--; | |
| 4030 | − } | |
| 4031 | − | |
| 4032 | − return 0; | |
| 4033 | −} | |
| 4034 | − | |
| 4035 | −static ssize_t mem_cgroup_force_empty_write(struct kernfs_open_file *of, | |
| 4036 | − char *buf, size_t nbytes, | |
| 4037 | − loff_t off) | |
| 4038 | −{ | |
| 4039 | − struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); | |
| 4040 | − | |
| 4041 | − if (mem_cgroup_is_root(memcg)) | |
| 4042 | − return -EINVAL; | |
| 4043 | − return mem_cgroup_force_empty(memcg) ?: nbytes; | |
| 4044 | −} | |
| 4045 | − | |
| 4046 | −static u64 mem_cgroup_hierarchy_read(struct cgroup_subsys_state *css, | |
| 4047 | − struct cftype *cft) | |
| 4048 | −{ | |
| 4049 | − return 1; | |
| 4050 | −} | |
| 4051 | − | |
| 4052 | −static int mem_cgroup_hierarchy_write(struct cgroup_subsys_state *css, | |
| 4053 | − struct cftype *cft, u64 val) | |
| 4054 | −{ | |
| 4055 | − if (val == 1) | |
| 4056 | − return 0; | |
| 4057 | − | |
| 4058 | − pr_warn_once("Non-hierarchical mode is deprecated. " | |
| 4059 | − "Please report your usecase to linux-mm@kvack.org if you " | |
| 4060 | − "depend on this functionality.\n"); | |
| 4061 | − | |
| 4062 | − return -EINVAL; | |
| 4063 | −} | |
| 4064 | − | |
| 4065 | −static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) | |
| 3077 | +unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) | |
| 4066 | 3078 | { |
| 4067 | 3079 | unsigned long val; |
| 4068 | 3080 | |
| @@ -4084,68 +3096,6 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) · 15 unmodified lines | ||
| 4084 | 3096 | return val; |
| 4085 | 3097 | } |
| 4086 | 3098 | |
| 4087 | −enum { | |
| 4088 | − RES_USAGE, | |
| 4089 | − RES_LIMIT, | |
| 4090 | − RES_MAX_USAGE, | |
| 4091 | − RES_FAILCNT, | |
| 4092 | − RES_SOFT_LIMIT, | |
| 4093 | −}; | |
| 4094 | − | |
| 4095 | −static u64 mem_cgroup_read_u64(struct cgroup_subsys_state *css, | |
| 4096 | − struct cftype *cft) | |
| 4097 | −{ | |
| 4098 | − struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 4099 | − struct page_counter *counter; | |
| 4100 | − | |
| 4101 | − switch (MEMFILE_TYPE(cft->private)) { | |
| 4102 | − case _MEM: | |
| 4103 | − counter = &memcg->memory; | |
| 4104 | − break; | |
| 4105 | − case _MEMSWAP: | |
| 4106 | − counter = &memcg->memsw; | |
| 4107 | − break; | |
| 4108 | − case _KMEM: | |
| 4109 | − counter = &memcg->kmem; | |
| 4110 | − break; | |
| 4111 | − case _TCP: | |
| 4112 | − counter = &memcg->tcpmem; | |
| 4113 | − break; | |
| 4114 | − default: | |
| 4115 | − BUG(); | |
| 4116 | − } | |
| 4117 | − | |
| 4118 | − switch (MEMFILE_ATTR(cft->private)) { | |
| 4119 | − case RES_USAGE: | |
| 4120 | − if (counter == &memcg->memory) | |
| 4121 | − return (u64)mem_cgroup_usage(memcg, false) * PAGE_SIZE; | |
| 4122 | − if (counter == &memcg->memsw) | |
| 4123 | − return (u64)mem_cgroup_usage(memcg, true) * PAGE_SIZE; | |
| 4124 | − return (u64)page_counter_read(counter) * PAGE_SIZE; | |
| 4125 | − case RES_LIMIT: | |
| 4126 | − return (u64)counter->max * PAGE_SIZE; | |
| 4127 | − case RES_MAX_USAGE: | |
| 4128 | − return (u64)counter->watermark * PAGE_SIZE; | |
| 4129 | − case RES_FAILCNT: | |
| 4130 | − return counter->failcnt; | |
| 4131 | − case RES_SOFT_LIMIT: | |
| 4132 | − return (u64)READ_ONCE(memcg->soft_limit) * PAGE_SIZE; | |
| 4133 | − default: | |
| 4134 | − BUG(); | |
| 4135 | − } | |
| 4136 | −} | |
| 4137 | − | |
| 4138 | −/* | |
| 4139 | − * This function doesn't do anything useful. Its only job is to provide a read | |
| 4140 | − * handler for a file so that cgroup_file_mode() will add read permissions. | |
| 4141 | − */ | |
| 4142 | −static int mem_cgroup_dummy_seq_show(__always_unused struct seq_file *m, | |
| 4143 | − __always_unused void *v) | |
| 4144 | −{ | |
| 4145 | − return -EINVAL; | |
| 4146 | −} | |
| 4147 | − | |
| 4148 | −#ifdef CONFIG_MEMCG_KMEM | |
| 4149 | 3099 | static int memcg_online_kmem(struct mem_cgroup *memcg) |
| 4150 | 3100 | { |
| 4151 | 3101 | struct obj_cgroup *objcg; |
| @@ -4196,2625 +3146,725 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg) · 44 unmodified lines | ||
| 4196 | 3146 | */ |
| 4197 | 3147 | memcg_reparent_list_lrus(memcg, parent); |
| 4198 | 3148 | } |
| 4199 | −#else | |
| 4200 | −static int memcg_online_kmem(struct mem_cgroup *memcg) | |
| 3149 | + | |
| 3150 | +#ifdef CONFIG_CGROUP_WRITEBACK | |
| 3151 | + | |
| 3152 | +#include <trace/events/writeback.h> | |
| 3153 | + | |
| 3154 | +static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp) | |
| 4201 | 3155 | { |
| 4202 | − return 0; | |
| 3156 | + return wb_domain_init(&memcg->cgwb_domain, gfp); | |
| 4203 | 3157 | } |
| 4204 | −static void memcg_offline_kmem(struct mem_cgroup *memcg) | |
| 3158 | + | |
| 3159 | +static void memcg_wb_domain_exit(struct mem_cgroup *memcg) | |
| 4205 | 3160 | { |
| 3161 | + wb_domain_exit(&memcg->cgwb_domain); | |
| 4206 | 3162 | } |
| 4207 | −#endif /* CONFIG_MEMCG_KMEM */ | |
| 4208 | 3163 | |
| 4209 | −static int memcg_update_tcp_max(struct mem_cgroup *memcg, unsigned long max) | |
| 3164 | +static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg) | |
| 4210 | 3165 | { |
| 4211 | − int ret; | |
| 4212 | − | |
| 4213 | − mutex_lock(&memcg_max_mutex); | |
| 4214 | − | |
| 4215 | − ret = page_counter_set_max(&memcg->tcpmem, max); | |
| 4216 | − if (ret) | |
| 4217 | − goto out; | |
| 4218 | − | |
| 4219 | − if (!memcg->tcpmem_active) { | |
| 4220 | − /* | |
| 4221 | − * The active flag needs to be written after the static_key | |
| 4222 | − * update. This is what guarantees that the socket activation | |
| 4223 | − * function is the last one to run. See mem_cgroup_sk_alloc() | |
| 4224 | − * for details, and note that we don't mark any socket as | |
| 4225 | − * belonging to this memcg until that flag is up. | |
| 4226 | − * | |
| 4227 | − * We need to do this, because static_keys will span multiple | |
| 4228 | − * sites, but we can't control their order. If we mark a socket | |
| 4229 | − * as accounted, but the accounting functions are not patched in | |
| 4230 | − * yet, we'll lose accounting. | |
| 4231 | − * | |
| 4232 | − * We never race with the readers in mem_cgroup_sk_alloc(), | |
| 4233 | − * because when this value change, the code to process it is not | |
| 4234 | − * patched in yet. | |
| 4235 | − */ | |
| 4236 | − static_branch_inc(&memcg_sockets_enabled_key); | |
| 4237 | − memcg->tcpmem_active = true; | |
| 4238 | − } | |
| 4239 | −out: | |
| 4240 | − mutex_unlock(&memcg_max_mutex); | |
| 4241 | − return ret; | |
| 3166 | + wb_domain_size_changed(&memcg->cgwb_domain); | |
| 4242 | 3167 | } |
| 4243 | 3168 | |
| 4244 | −/* | |
| 4245 | − * The user of this function is... | |
| 4246 | − * RES_LIMIT. | |
| 4247 | − */ | |
| 4248 | −static ssize_t mem_cgroup_write(struct kernfs_open_file *of, | |
| 4249 | − char *buf, size_t nbytes, loff_t off) | |
| 3169 | +struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb) | |
| 4250 | 3170 | { |
| 4251 | − struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); | |
| 4252 | − unsigned long nr_pages; | |
| 4253 | − int ret; | |
| 3171 | + struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); | |
| 4254 | 3172 | |
| 4255 | − buf = strstrip(buf); | |
| 4256 | − ret = page_counter_memparse(buf, "-1", &nr_pages); | |
| 4257 | − if (ret) | |
| 4258 | − return ret; | |
| 3173 | + if (!memcg->css.parent) | |
| 3174 | + return NULL; | |
| 4259 | 3175 | |
| 4260 | − switch (MEMFILE_ATTR(of_cft(of)->private)) { | |
| 4261 | − case RES_LIMIT: | |
| 4262 | − if (mem_cgroup_is_root(memcg)) { /* Can't set limit on root */ | |
| 4263 | − ret = -EINVAL; | |
| 4264 | − break; | |
| 4265 | − } | |
| 4266 | − switch (MEMFILE_TYPE(of_cft(of)->private)) { | |
| 4267 | − case _MEM: | |
| 4268 | − ret = mem_cgroup_resize_max(memcg, nr_pages, false); | |
| 4269 | − break; | |
| 4270 | − case _MEMSWAP: | |
| 4271 | − ret = mem_cgroup_resize_max(memcg, nr_pages, true); | |
| 4272 | − break; | |
| 4273 | − case _KMEM: | |
| 4274 | − pr_warn_once("kmem.limit_in_bytes is deprecated and will be removed. " | |
| 4275 | − "Writing any value to this file has no effect. " | |
| 4276 | − "Please report your usecase to linux-mm@kvack.org if you " | |
| 4277 | − "depend on this functionality.\n"); | |
| 4278 | − ret = 0; | |
| 4279 | − break; | |
| 4280 | − case _TCP: | |
| 4281 | − ret = memcg_update_tcp_max(memcg, nr_pages); | |
| 4282 | − break; | |
| 4283 | − } | |
| 4284 | − break; | |
| 4285 | − case RES_SOFT_LIMIT: | |
| 4286 | − if (IS_ENABLED(CONFIG_PREEMPT_RT)) { | |
| 4287 | − ret = -EOPNOTSUPP; | |
| 4288 | − } else { | |
| 4289 | − WRITE_ONCE(memcg->soft_limit, nr_pages); | |
| 4290 | − ret = 0; | |
| 4291 | − } | |
| 4292 | − break; | |
| 4293 | − } | |
| 4294 | − return ret ?: nbytes; | |
| 3176 | + return &memcg->cgwb_domain; | |
| 4295 | 3177 | } |
| 4296 | 3178 | |
| 4297 | −static ssize_t mem_cgroup_reset(struct kernfs_open_file *of, char *buf, | |
| 4298 | − size_t nbytes, loff_t off) | |
| 3179 | +/** | |
| 3180 | + * mem_cgroup_wb_stats - retrieve writeback related stats from its memcg | |
| 3181 | + * @wb: bdi_writeback in question | |
| 3182 | + * @pfilepages: out parameter for number of file pages | |
| 3183 | + * @pheadroom: out parameter for number of allocatable pages according to memcg | |
| 3184 | + * @pdirty: out parameter for number of dirty pages | |
| 3185 | + * @pwriteback: out parameter for number of pages under writeback | |
| 3186 | + * | |
| 3187 | + * Determine the numbers of file, headroom, dirty, and writeback pages in | |
| 3188 | + * @wb's memcg. File, dirty and writeback are self-explanatory. Headroom | |
| 3189 | + * is a bit more involved. | |
| 3190 | + * | |
| 3191 | + * A memcg's headroom is "min(max, high) - used". In the hierarchy, the | |
| 3192 | + * headroom is calculated as the lowest headroom of itself and the | |
| 3193 | + * ancestors. Note that this doesn't consider the actual amount of | |
| 3194 | + * available memory in the system. The caller should further cap | |
| 3195 | + * *@pheadroom accordingly. | |
| 3196 | + */ | |
| 3197 | +void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, | |
| 3198 | + unsigned long *pheadroom, unsigned long *pdirty, | |
| 3199 | + unsigned long *pwriteback) | |
| 4299 | 3200 | { |
| 4300 | − struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); | |
| 4301 | − struct page_counter *counter; | |
| 3201 | + struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); | |
| 3202 | + struct mem_cgroup *parent; | |
| 4302 | 3203 | |
| 4303 | − switch (MEMFILE_TYPE(of_cft(of)->private)) { | |
| 4304 | − case _MEM: | |
| 4305 | − counter = &memcg->memory; | |
| 4306 | − break; | |
| 4307 | − case _MEMSWAP: | |
| 4308 | − counter = &memcg->memsw; | |
| 4309 | − break; | |
| 4310 | − case _KMEM: | |
| 4311 | − counter = &memcg->kmem; | |
| 4312 | − break; | |
| 4313 | − case _TCP: | |
| 4314 | − counter = &memcg->tcpmem; | |
| 4315 | − break; | |
| 4316 | − default: | |
| 4317 | − BUG(); | |
| 4318 | − } | |
| 3204 | + mem_cgroup_flush_stats_ratelimited(memcg); | |
| 4319 | 3205 | |
| 4320 | − switch (MEMFILE_ATTR(of_cft(of)->private)) { | |
| 4321 | − case RES_MAX_USAGE: | |
| 4322 | − page_counter_reset_watermark(counter); | |
| 4323 | − break; | |
| 4324 | − case RES_FAILCNT: | |
| 4325 | − counter->failcnt = 0; | |
| 4326 | − break; | |
| 4327 | − default: | |
| 4328 | − BUG(); | |
| 4329 | − } | |
| 3206 | + *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); | |
| 3207 | + *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); | |
| 3208 | + *pfilepages = memcg_page_state(memcg, NR_INACTIVE_FILE) + | |
| 3209 | + memcg_page_state(memcg, NR_ACTIVE_FILE); | |
| 4330 | 3210 | |
| 4331 | − return nbytes; | |
| 4332 | −} | |
| 3211 | + *pheadroom = PAGE_COUNTER_MAX; | |
| 3212 | + while ((parent = parent_mem_cgroup(memcg))) { | |
| 3213 | + unsigned long ceiling = min(READ_ONCE(memcg->memory.max), | |
| 3214 | + READ_ONCE(memcg->memory.high)); | |
| 3215 | + unsigned long used = page_counter_read(&memcg->memory); | |
| 4333 | 3216 | |
| 4334 | −static u64 mem_cgroup_move_charge_read(struct cgroup_subsys_state *css, | |
| 4335 | − struct cftype *cft) | |
| 4336 | −{ | |
| 4337 | − return mem_cgroup_from_css(css)->move_charge_at_immigrate; | |
| 3217 | + *pheadroom = min(*pheadroom, ceiling - min(ceiling, used)); | |
| 3218 | + memcg = parent; | |
| 3219 | + } | |
| 4338 | 3220 | } |
| 4339 | 3221 | |
| 4340 | −#ifdef CONFIG_MMU | |
| 4341 | −static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css, | |
| 4342 | − struct cftype *cft, u64 val) | |
| 3222 | +/* | |
| 3223 | + * Foreign dirty flushing | |
| 3224 | + * | |
| 3225 | + * There's an inherent mismatch between memcg and writeback. The former | |
| 3226 | + * tracks ownership per-page while the latter per-inode. This was a | |
| 3227 | + * deliberate design decision because honoring per-page ownership in the | |
| 3228 | + * writeback path is complicated, may lead to higher CPU and IO overheads | |
| 3229 | + * and deemed unnecessary given that write-sharing an inode across | |
| 3230 | + * different cgroups isn't a common use-case. | |
| 3231 | + * | |
| 3232 | + * Combined with inode majority-writer ownership switching, this works well | |
| 3233 | + * enough in most cases but there are some pathological cases. For | |
| 3234 | + * example, let's say there are two cgroups A and B which keep writing to | |
| 3235 | + * different but confined parts of the same inode. B owns the inode and | |
| 3236 | + * A's memory is limited far below B's. A's dirty ratio can rise enough to | |
| 3237 | + * trigger balance_dirty_pages() sleeps but B's can be low enough to avoid | |
| 3238 | + * triggering background writeback. A will be slowed down without a way to | |
| 3239 | + * make writeback of the dirty pages happen. | |
| 3240 | + * | |
| 3241 | + * Conditions like the above can lead to a cgroup getting repeatedly and | |
| 3242 | + * severely throttled after making some progress after each | |
| 3243 | + * dirty_expire_interval while the underlying IO device is almost | |
| 3244 | + * completely idle. | |
| 3245 | + * | |
| 3246 | + * Solving this problem completely requires matching the ownership tracking | |
| 3247 | + * granularities between memcg and writeback in either direction. However, | |
| 3248 | + * the more egregious behaviors can be avoided by simply remembering the | |
| 3249 | + * most recent foreign dirtying events and initiating remote flushes on | |
| 3250 | + * them when local writeback isn't enough to keep the memory clean enough. | |
| 3251 | + * | |
| 3252 | + * The following two functions implement such mechanism. When a foreign | |
| 3253 | + * page - a page whose memcg and writeback ownerships don't match - is | |
| 3254 | + * dirtied, mem_cgroup_track_foreign_dirty() records the inode owning | |
| 3255 | + * bdi_writeback on the page owning memcg. When balance_dirty_pages() | |
| 3256 | + * decides that the memcg needs to sleep due to high dirty ratio, it calls | |
| 3257 | + * mem_cgroup_flush_foreign() which queues writeback on the recorded | |
| 3258 | + * foreign bdi_writebacks which haven't expired. Both the numbers of | |
| 3259 | + * recorded bdi_writebacks and concurrent in-flight foreign writebacks are | |
| 3260 | + * limited to MEMCG_CGWB_FRN_CNT. | |
| 3261 | + * | |
| 3262 | + * The mechanism only remembers IDs and doesn't hold any object references. | |
| 3263 | + * As being wrong occasionally doesn't matter, updates and accesses to the | |
| 3264 | + * records are lockless and racy. | |
| 3265 | + */ | |
| 3266 | +void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio, | |
| 3267 | + struct bdi_writeback *wb) | |
| 4343 | 3268 | { |
| 4344 | − struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 4345 | − | |
| 4346 | − pr_warn_once("Cgroup memory moving (move_charge_at_immigrate) is deprecated. " | |
| 4347 | − "Please report your usecase to linux-mm@kvack.org if you " | |
| 4348 | − "depend on this functionality.\n"); | |
| 3269 | + struct mem_cgroup *memcg = folio_memcg(folio); | |
| 3270 | + struct memcg_cgwb_frn *frn; | |
| 3271 | + u64 now = get_jiffies_64(); | |
| 3272 | + u64 oldest_at = now; | |
| 3273 | + int oldest = -1; | |
| 3274 | + int i; | |
| 4349 | 3275 | |
| 4350 | − if (val & ~MOVE_MASK) | |
| 4351 | − return -EINVAL; | |
| 3276 | + trace_track_foreign_dirty(folio, wb); | |
| 4352 | 3277 | |
| 4353 | 3278 | /* |
| 4354 | − * No kind of locking is needed in here, because ->can_attach() will | |
| 4355 | − * check this value once in the beginning of the process, and then carry | |
| 4356 | − * on with stale data. This means that changes to this value will only | |
| 4357 | − * affect task migrations starting after the change. | |
| 3279 | + * Pick the slot to use. If there is already a slot for @wb, keep | |
| 3280 | + * using it. If not replace the oldest one which isn't being | |
| 3281 | + * written out. | |
| 4358 | 3282 | */ |
| 4359 | − memcg->move_charge_at_immigrate = val; | |
| 4360 | − return 0; | |
| 4361 | −} | |
| 4362 | −#else | |
| 4363 | −static int mem_cgroup_move_charge_write(struct cgroup_subsys_state *css, | |
| 4364 | − struct cftype *cft, u64 val) | |
| 4365 | −{ | |
| 4366 | − return -ENOSYS; | |
| 4367 | −} | |
| 4368 | −#endif | |
| 4369 | − | |
| 4370 | −#ifdef CONFIG_NUMA | |
| 4371 | − | |
| 4372 | −#define LRU_ALL_FILE (BIT(LRU_INACTIVE_FILE) | BIT(LRU_ACTIVE_FILE)) | |
| 4373 | −#define LRU_ALL_ANON (BIT(LRU_INACTIVE_ANON) | BIT(LRU_ACTIVE_ANON)) | |
| 4374 | −#define LRU_ALL ((1 << NR_LRU_LISTS) - 1) | |
| 4375 | − | |
| 4376 | −static unsigned long mem_cgroup_node_nr_lru_pages(struct mem_cgroup *memcg, | |
| 4377 | − int nid, unsigned int lru_mask, bool tree) | |
| 4378 | −{ | |
| 4379 | − struct lruvec *lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); | |
| 4380 | − unsigned long nr = 0; | |
| 4381 | − enum lru_list lru; | |
| 4382 | − | |
| 4383 | − VM_BUG_ON((unsigned)nid >= nr_node_ids); | |
| 4384 | − | |
| 4385 | − for_each_lru(lru) { | |
| 4386 | − if (!(BIT(lru) & lru_mask)) | |
| 4387 | − continue; | |
| 4388 | − if (tree) | |
| 4389 | − nr += lruvec_page_state(lruvec, NR_LRU_BASE + lru); | |
| 4390 | − else | |
| 4391 | − nr += lruvec_page_state_local(lruvec, NR_LRU_BASE + lru); | |
| 3283 | + for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) { | |
| 3284 | + frn = &memcg->cgwb_frn[i]; | |
| 3285 | + if (frn->bdi_id == wb->bdi->id && | |
| 3286 | + frn->memcg_id == wb->memcg_css->id) | |
| 3287 | + break; | |
| 3288 | + if (time_before64(frn->at, oldest_at) && | |
| 3289 | + atomic_read(&frn->done.cnt) == 1) { | |
| 3290 | + oldest = i; | |
| 3291 | + oldest_at = frn->at; | |
| 3292 | + } | |
| 4392 | 3293 | } |
| 4393 | − return nr; | |
| 4394 | −} | |
| 4395 | 3294 | |
| 4396 | −static unsigned long mem_cgroup_nr_lru_pages(struct mem_cgroup *memcg, | |
| 4397 | − unsigned int lru_mask, | |
| 4398 | − bool tree) | |
| 4399 | −{ | |
| 4400 | − unsigned long nr = 0; | |
| 4401 | − enum lru_list lru; | |
| 3295 | + if (i < MEMCG_CGWB_FRN_CNT) { | |
| 3296 | + /* | |
| 3297 | + * Re-using an existing one. Update timestamp lazily to | |
| 3298 | + * avoid making the cacheline hot. We want them to be | |
| 3299 | + * reasonably up-to-date and significantly shorter than | |
| 3300 | + * dirty_expire_interval as that's what expires the record. | |
| 3301 | + * Use the shorter of 1s and dirty_expire_interval / 8. | |
| 3302 | + */ | |
| 3303 | + unsigned long update_intv = | |
| 3304 | + min_t(unsigned long, HZ, | |
| 3305 | + msecs_to_jiffies(dirty_expire_interval * 10) / 8); | |
| 4402 | 3306 | |
| 4403 | − for_each_lru(lru) { | |
| 4404 | − if (!(BIT(lru) & lru_mask)) | |
| 4405 | − continue; | |
| 4406 | − if (tree) | |
| 4407 | − nr += memcg_page_state(memcg, NR_LRU_BASE + lru); | |
| 4408 | − else | |
| 4409 | − nr += memcg_page_state_local(memcg, NR_LRU_BASE + lru); | |
| 3307 | + if (time_before64(frn->at, now - update_intv)) | |
| 3308 | + frn->at = now; | |
| 3309 | + } else if (oldest >= 0) { | |
| 3310 | + /* replace the oldest free one */ | |
| 3311 | + frn = &memcg->cgwb_frn[oldest]; | |
| 3312 | + frn->bdi_id = wb->bdi->id; | |
| 3313 | + frn->memcg_id = wb->memcg_css->id; | |
| 3314 | + frn->at = now; | |
| 4410 | 3315 | } |
| 4411 | − return nr; | |
| 4412 | 3316 | } |
| 4413 | 3317 | |
| 4414 | −static int memcg_numa_stat_show(struct seq_file *m, void *v) | |
| 3318 | +/* issue foreign writeback flushes for recorded foreign dirtying events */ | |
| 3319 | +void mem_cgroup_flush_foreign(struct bdi_writeback *wb) | |
| 4415 | 3320 | { |
| 4416 | − struct numa_stat { | |
| 4417 | − const char *name; | |
| 4418 | − unsigned int lru_mask; | |
| 4419 | − }; | |
| 4420 | − | |
| 4421 | − static const struct numa_stat stats[] = { | |
| 4422 | − { "total", LRU_ALL }, | |
| 4423 | − { "file", LRU_ALL_FILE }, | |
| 4424 | − { "anon", LRU_ALL_ANON }, | |
| 4425 | − { "unevictable", BIT(LRU_UNEVICTABLE) }, | |
| 4426 | − }; | |
| 4427 | − const struct numa_stat *stat; | |
| 4428 | − int nid; | |
| 4429 | − struct mem_cgroup *memcg = mem_cgroup_from_seq(m); | |
| 3321 | + struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); | |
| 3322 | + unsigned long intv = msecs_to_jiffies(dirty_expire_interval * 10); | |
| 3323 | + u64 now = jiffies_64; | |
| 3324 | + int i; | |
| 4430 | 3325 | |
| 4431 | − mem_cgroup_flush_stats(memcg); | |
| 3326 | + for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) { | |
| 3327 | + struct memcg_cgwb_frn *frn = &memcg->cgwb_frn[i]; | |
| 4432 | 3328 | |
| 4433 | − for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { | |
| 4434 | − seq_printf(m, "%s=%lu", stat->name, | |
| 4435 | − mem_cgroup_nr_lru_pages(memcg, stat->lru_mask, | |
| 4436 | − false)); | |
| 4437 | − for_each_node_state(nid, N_MEMORY) | |
| 4438 | − seq_printf(m, " N%d=%lu", nid, | |
| 4439 | − mem_cgroup_node_nr_lru_pages(memcg, nid, | |
| 4440 | − stat->lru_mask, false)); | |
| 4441 | − seq_putc(m, '\n'); | |
| 3329 | + /* | |
| 3330 | + * If the record is older than dirty_expire_interval, | |
| 3331 | + * writeback on it has already started. No need to kick it | |
| 3332 | + * off again. Also, don't start a new one if there's | |
| 3333 | + * already one in flight. | |
| 3334 | + */ | |
| 3335 | + if (time_after64(frn->at, now - intv) && | |
| 3336 | + atomic_read(&frn->done.cnt) == 1) { | |
| 3337 | + frn->at = 0; | |
| 3338 | + trace_flush_foreign(wb, frn->bdi_id, frn->memcg_id); | |
| 3339 | + cgroup_writeback_by_id(frn->bdi_id, frn->memcg_id, | |
| 3340 | + WB_REASON_FOREIGN_FLUSH, | |
| 3341 | + &frn->done); | |
| 3342 | + } | |
| 4442 | 3343 | } |
| 3344 | +} | |
| 4443 | 3345 | |
| 4444 | − for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { | |
| 4445 | − | |
| 4446 | − seq_printf(m, "hierarchical_%s=%lu", stat->name, | |
| 4447 | − mem_cgroup_nr_lru_pages(memcg, stat->lru_mask, | |
| 4448 | − true)); | |
| 4449 | − for_each_node_state(nid, N_MEMORY) | |
| 4450 | − seq_printf(m, " N%d=%lu", nid, | |
| 4451 | − mem_cgroup_node_nr_lru_pages(memcg, nid, | |
| 4452 | − stat->lru_mask, true)); | |
| 4453 | − seq_putc(m, '\n'); | |
| 4454 | − } | |
| 3346 | +#else /* CONFIG_CGROUP_WRITEBACK */ | |
| 4455 | 3347 | |
| 3348 | +static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp) | |
| 3349 | +{ | |
| 4456 | 3350 | return 0; |
| 4457 | 3351 | } |
| 4458 | −#endif /* CONFIG_NUMA */ | |
| 4459 | − | |
| 4460 | −static const unsigned int memcg1_stats[] = { | |
| 4461 | − NR_FILE_PAGES, | |
| 4462 | − NR_ANON_MAPPED, | |
| 4463 | −#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 4464 | − NR_ANON_THPS, | |
| 4465 | −#endif | |
| 4466 | − NR_SHMEM, | |
| 4467 | − NR_FILE_MAPPED, | |
| 4468 | − NR_FILE_DIRTY, | |
| 4469 | − NR_WRITEBACK, | |
| 4470 | − WORKINGSET_REFAULT_ANON, | |
| 4471 | − WORKINGSET_REFAULT_FILE, | |
| 4472 | −#ifdef CONFIG_SWAP | |
| 4473 | − MEMCG_SWAP, | |
| 4474 | − NR_SWAPCACHE, | |
| 4475 | −#endif | |
| 4476 | −}; | |
| 4477 | − | |
| 4478 | −static const char *const memcg1_stat_names[] = { | |
| 4479 | − "cache", | |
| 4480 | − "rss", | |
| 4481 | −#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 4482 | − "rss_huge", | |
| 4483 | −#endif | |
| 4484 | − "shmem", | |
| 4485 | − "mapped_file", | |
| 4486 | − "dirty", | |
| 4487 | − "writeback", | |
| 4488 | − "workingset_refault_anon", | |
| 4489 | − "workingset_refault_file", | |
| 4490 | −#ifdef CONFIG_SWAP | |
| 4491 | − "swap", | |
| 4492 | − "swapcached", | |
| 4493 | −#endif | |
| 4494 | −}; | |
| 4495 | − | |
| 4496 | −/* Universal VM events cgroup1 shows, original sort order */ | |
| 4497 | −static const unsigned int memcg1_events[] = { | |
| 4498 | − PGPGIN, | |
| 4499 | − PGPGOUT, | |
| 4500 | − PGFAULT, | |
| 4501 | − PGMAJFAULT, | |
| 4502 | −}; | |
| 4503 | 3352 | |
| 4504 | −static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) | |
| 3353 | +static void memcg_wb_domain_exit(struct mem_cgroup *memcg) | |
| 4505 | 3354 | { |
| 4506 | − unsigned long memory, memsw; | |
| 4507 | − struct mem_cgroup *mi; | |
| 4508 | − unsigned int i; | |
| 4509 | − | |
| 4510 | − BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); | |
| 4511 | − | |
| 4512 | − mem_cgroup_flush_stats(memcg); | |
| 4513 | − | |
| 4514 | − for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { | |
| 4515 | − unsigned long nr; | |
| 4516 | − | |
| 4517 | − nr = memcg_page_state_local_output(memcg, memcg1_stats[i]); | |
| 4518 | − seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], nr); | |
| 4519 | − } | |
| 4520 | − | |
| 4521 | − for (i = 0; i < ARRAY_SIZE(memcg1_events); i++) | |
| 4522 | − seq_buf_printf(s, "%s %lu\n", vm_event_name(memcg1_events[i]), | |
| 4523 | − memcg_events_local(memcg, memcg1_events[i])); | |
| 4524 | − | |
| 4525 | − for (i = 0; i < NR_LRU_LISTS; i++) | |
| 4526 | − seq_buf_printf(s, "%s %lu\n", lru_list_name(i), | |
| 4527 | − memcg_page_state_local(memcg, NR_LRU_BASE + i) * | |
| 4528 | − PAGE_SIZE); | |
| 4529 | − | |
| 4530 | − /* Hierarchical information */ | |
| 4531 | − memory = memsw = PAGE_COUNTER_MAX; | |
| 4532 | − for (mi = memcg; mi; mi = parent_mem_cgroup(mi)) { | |
| 4533 | − memory = min(memory, READ_ONCE(mi->memory.max)); | |
| 4534 | − memsw = min(memsw, READ_ONCE(mi->memsw.max)); | |
| 4535 | − } | |
| 4536 | − seq_buf_printf(s, "hierarchical_memory_limit %llu\n", | |
| 4537 | − (u64)memory * PAGE_SIZE); | |
| 4538 | − seq_buf_printf(s, "hierarchical_memsw_limit %llu\n", | |
| 4539 | − (u64)memsw * PAGE_SIZE); | |
| 4540 | − | |
| 4541 | − for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { | |
| 4542 | − unsigned long nr; | |
| 4543 | − | |
| 4544 | − nr = memcg_page_state_output(memcg, memcg1_stats[i]); | |
| 4545 | − seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i], | |
| 4546 | − (u64)nr); | |
| 4547 | − } | |
| 4548 | − | |
| 4549 | − for (i = 0; i < ARRAY_SIZE(memcg1_events); i++) | |
| 4550 | − seq_buf_printf(s, "total_%s %llu\n", | |
| 4551 | − vm_event_name(memcg1_events[i]), | |
| 4552 | − (u64)memcg_events(memcg, memcg1_events[i])); | |
| 4553 | − | |
| 4554 | − for (i = 0; i < NR_LRU_LISTS; i++) | |
| 4555 | − seq_buf_printf(s, "total_%s %llu\n", lru_list_name(i), | |
| 4556 | − (u64)memcg_page_state(memcg, NR_LRU_BASE + i) * | |
| 4557 | − PAGE_SIZE); | |
| 4558 | − | |
| 4559 | −#ifdef CONFIG_DEBUG_VM | |
| 4560 | − { | |
| 4561 | − pg_data_t *pgdat; | |
| 4562 | − struct mem_cgroup_per_node *mz; | |
| 4563 | − unsigned long anon_cost = 0; | |
| 4564 | − unsigned long file_cost = 0; | |
| 4565 | − | |
| 4566 | − for_each_online_pgdat(pgdat) { | |
| 4567 | − mz = memcg->nodeinfo[pgdat->node_id]; | |
| 4568 | − | |
| 4569 | − anon_cost += mz->lruvec.anon_cost; | |
| 4570 | − file_cost += mz->lruvec.file_cost; | |
| 4571 | − } | |
| 4572 | − seq_buf_printf(s, "anon_cost %lu\n", anon_cost); | |
| 4573 | − seq_buf_printf(s, "file_cost %lu\n", file_cost); | |
| 4574 | − } | |
| 4575 | −#endif | |
| 4576 | 3355 | } |
| 4577 | 3356 | |
| 4578 | −static u64 mem_cgroup_swappiness_read(struct cgroup_subsys_state *css, | |
| 4579 | − struct cftype *cft) | |
| 3357 | +static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg) | |
| 4580 | 3358 | { |
| 4581 | − struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 4582 | − | |
| 4583 | − return mem_cgroup_swappiness(memcg); | |
| 4584 | 3359 | } |
| 4585 | 3360 | |
| 4586 | −static int mem_cgroup_swappiness_write(struct cgroup_subsys_state *css, | |
| 4587 | − struct cftype *cft, u64 val) | |
| 4588 | −{ | |
| 4589 | − struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 3361 | +#endif /* CONFIG_CGROUP_WRITEBACK */ | |
| 4590 | 3362 | |
| 4591 | − if (val > 200) | |
| 4592 | − return -EINVAL; | |
| 4593 | − | |
| 4594 | − if (!mem_cgroup_is_root(memcg)) | |
| 4595 | − WRITE_ONCE(memcg->swappiness, val); | |
| 4596 | − else | |
| 4597 | − WRITE_ONCE(vm_swappiness, val); | |
| 4598 | − | |
| 4599 | − return 0; | |
| 4600 | −} | |
| 4601 | − | |
| 4602 | −static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap) | |
| 4603 | −{ | |
| 4604 | − struct mem_cgroup_threshold_ary *t; | |
| 4605 | − unsigned long usage; | |
| 4606 | − int i; | |
| 4607 | − | |
| 4608 | − rcu_read_lock(); | |
| 4609 | − if (!swap) | |
| 4610 | − t = rcu_dereference(memcg->thresholds.primary); | |
| 4611 | − else | |
| 4612 | − t = rcu_dereference(memcg->memsw_thresholds.primary); | |
| 4613 | − | |
| 4614 | − if (!t) | |
| 4615 | − goto unlock; | |
| 4616 | − | |
| 4617 | − usage = mem_cgroup_usage(memcg, swap); | |
| 4618 | − | |
| 4619 | − /* | |
| 4620 | − * current_threshold points to threshold just below or equal to usage. | |
| 4621 | − * If it's not true, a threshold was crossed after last | |
| 4622 | − * call of __mem_cgroup_threshold(). | |
| 4623 | − */ | |
| 4624 | − i = t->current_threshold; | |
| 4625 | − | |
| 4626 | − /* | |
| 4627 | − * Iterate backward over array of thresholds starting from | |
| 4628 | − * current_threshold and check if a threshold is crossed. | |
| 4629 | − * If none of thresholds below usage is crossed, we read | |
| 4630 | − * only one element of the array here. | |
| 4631 | − */ | |
| 4632 | − for (; i >= 0 && unlikely(t->entries[i].threshold > usage); i--) | |
| 4633 | − eventfd_signal(t->entries[i].eventfd); | |
| 4634 | − | |
| 4635 | − /* i = current_threshold + 1 */ | |
| 4636 | − i++; | |
| 4637 | − | |
| 4638 | − /* | |
| 4639 | − * Iterate forward over array of thresholds starting from | |
| 4640 | − * current_threshold+1 and check if a threshold is crossed. | |
| 4641 | − * If none of thresholds above usage is crossed, we read | |
| 4642 | − * only one element of the array here. | |
| 4643 | − */ | |
| 4644 | − for (; i < t->size && unlikely(t->entries[i].threshold <= usage); i++) | |
| 4645 | − eventfd_signal(t->entries[i].eventfd); | |
| 4646 | − | |
| 4647 | − /* Update current_threshold */ | |
| 4648 | − t->current_threshold = i - 1; | |
| 4649 | −unlock: | |
| 4650 | − rcu_read_unlock(); | |
| 4651 | −} | |
| 4652 | − | |
| 4653 | −static void mem_cgroup_threshold(struct mem_cgroup *memcg) | |
| 4654 | −{ | |
| 4655 | − while (memcg) { | |
| 4656 | − __mem_cgroup_threshold(memcg, false); | |
| 4657 | − if (do_memsw_account()) | |
| 4658 | − __mem_cgroup_threshold(memcg, true); | |
| 4659 | − | |
| 4660 | − memcg = parent_mem_cgroup(memcg); | |
| 4661 | − } | |
| 4662 | −} | |
| 4663 | − | |
| 4664 | −static int compare_thresholds(const void *a, const void *b) | |
| 4665 | −{ | |
| 4666 | − const struct mem_cgroup_threshold *_a = a; | |
| 4667 | − const struct mem_cgroup_threshold *_b = b; | |
| 4668 | − | |
| 4669 | − if (_a->threshold > _b->threshold) | |
| 4670 | − return 1; | |
| 4671 | − | |
| 4672 | − if (_a->threshold < _b->threshold) | |
| 4673 | − return -1; | |
| 4674 | − | |
| 4675 | − return 0; | |
| 4676 | −} | |
| 4677 | − | |
| 4678 | −static int mem_cgroup_oom_notify_cb(struct mem_cgroup *memcg) | |
| 4679 | −{ | |
| 4680 | − struct mem_cgroup_eventfd_list *ev; | |
| 4681 | − | |
| 4682 | − spin_lock(&memcg_oom_lock); | |
| 4683 | − | |
| 4684 | − list_for_each_entry(ev, &memcg->oom_notify, list) | |
| 4685 | − eventfd_signal(ev->eventfd); | |
| 4686 | − | |
| 4687 | − spin_unlock(&memcg_oom_lock); | |
| 4688 | − return 0; | |
| 4689 | −} | |
| 4690 | − | |
| 4691 | −static void mem_cgroup_oom_notify(struct mem_cgroup *memcg) | |
| 4692 | −{ | |
| 4693 | − struct mem_cgroup *iter; | |
| 4694 | − | |
| 4695 | − for_each_mem_cgroup_tree(iter, memcg) | |
| 4696 | − mem_cgroup_oom_notify_cb(iter); | |
| 4697 | −} | |
| 4698 | − | |
| 4699 | −static int __mem_cgroup_usage_register_event(struct mem_cgroup *memcg, | |
| 4700 | − struct eventfd_ctx *eventfd, const char *args, enum res_type type) | |
| 4701 | −{ | |
| 4702 | − struct mem_cgroup_thresholds *thresholds; | |
| 4703 | − struct mem_cgroup_threshold_ary *new; | |
| 4704 | − unsigned long threshold; | |
| 4705 | − unsigned long usage; | |
| 4706 | − int i, size, ret; | |
| 4707 | − | |
| 4708 | − ret = page_counter_memparse(args, "-1", &threshold); | |
| 4709 | − if (ret) | |
| 4710 | − return ret; | |
| 4711 | − | |
| 4712 | − mutex_lock(&memcg->thresholds_lock); | |
| 4713 | − | |
| 4714 | − if (type == _MEM) { | |
| 4715 | − thresholds = &memcg->thresholds; | |
| 4716 | − usage = mem_cgroup_usage(memcg, false); | |
| 4717 | − } else if (type == _MEMSWAP) { | |
| 4718 | − thresholds = &memcg->memsw_thresholds; | |
| 4719 | − usage = mem_cgroup_usage(memcg, true); | |
| 4720 | − } else | |
| 4721 | − BUG(); | |
| 4722 | − | |
| 4723 | − /* Check if a threshold crossed before adding a new one */ | |
| 4724 | − if (thresholds->primary) | |
| 4725 | − __mem_cgroup_threshold(memcg, type == _MEMSWAP); | |
| 4726 | − | |
| 4727 | − size = thresholds->primary ? thresholds->primary->size + 1 : 1; | |
| 4728 | − | |
| 4729 | − /* Allocate memory for new array of thresholds */ | |
| 4730 | − new = kmalloc(struct_size(new, entries, size), GFP_KERNEL); | |
| 4731 | − if (!new) { | |
| 4732 | − ret = -ENOMEM; | |
| 4733 | − goto unlock; | |
| 4734 | − } | |
| 4735 | − new->size = size; | |
| 4736 | − | |
| 4737 | − /* Copy thresholds (if any) to new array */ | |
| 4738 | − if (thresholds->primary) | |
| 4739 | − memcpy(new->entries, thresholds->primary->entries, | |
| 4740 | − flex_array_size(new, entries, size - 1)); | |
| 4741 | − | |
| 4742 | − /* Add new threshold */ | |
| 4743 | − new->entries[size - 1].eventfd = eventfd; | |
| 4744 | − new->entries[size - 1].threshold = threshold; | |
| 4745 | − | |
| 4746 | − /* Sort thresholds. Registering of new threshold isn't time-critical */ | |
| 4747 | − sort(new->entries, size, sizeof(*new->entries), | |
| 4748 | − compare_thresholds, NULL); | |
| 4749 | − | |
| 4750 | − /* Find current threshold */ | |
| 4751 | − new->current_threshold = -1; | |
| 4752 | − for (i = 0; i < size; i++) { | |
| 4753 | − if (new->entries[i].threshold <= usage) { | |
| 4754 | − /* | |
| 4755 | − * new->current_threshold will not be used until | |
| 4756 | − * rcu_assign_pointer(), so it's safe to increment | |
| 4757 | − * it here. | |
| 4758 | − */ | |
| 4759 | − ++new->current_threshold; | |
| 4760 | − } else | |
| 4761 | − break; | |
| 4762 | − } | |
| 4763 | − | |
| 4764 | − /* Free old spare buffer and save old primary buffer as spare */ | |
| 4765 | − kfree(thresholds->spare); | |
| 4766 | − thresholds->spare = thresholds->primary; | |
| 4767 | − | |
| 4768 | − rcu_assign_pointer(thresholds->primary, new); | |
| 4769 | − | |
| 4770 | − /* To be sure that nobody uses thresholds */ | |
| 4771 | − synchronize_rcu(); | |
| 4772 | − | |
| 4773 | −unlock: | |
| 4774 | − mutex_unlock(&memcg->thresholds_lock); | |
| 4775 | − | |
| 4776 | − return ret; | |
| 4777 | −} | |
| 4778 | − | |
| 4779 | −static int mem_cgroup_usage_register_event(struct mem_cgroup *memcg, | |
| 4780 | − struct eventfd_ctx *eventfd, const char *args) | |
| 4781 | −{ | |
| 4782 | − return __mem_cgroup_usage_register_event(memcg, eventfd, args, _MEM); | |
| 4783 | −} | |
| 4784 | − | |
| 4785 | −static int memsw_cgroup_usage_register_event(struct mem_cgroup *memcg, | |
| 4786 | − struct eventfd_ctx *eventfd, const char *args) | |
| 4787 | −{ | |
| 4788 | − return __mem_cgroup_usage_register_event(memcg, eventfd, args, _MEMSWAP); | |
| 4789 | −} | |
| 4790 | − | |
| 4791 | −static void __mem_cgroup_usage_unregister_event(struct mem_cgroup *memcg, | |
| 4792 | − struct eventfd_ctx *eventfd, enum res_type type) | |
| 4793 | −{ | |
| 4794 | − struct mem_cgroup_thresholds *thresholds; | |
| 4795 | − struct mem_cgroup_threshold_ary *new; | |
| 4796 | − unsigned long usage; | |
| 4797 | − int i, j, size, entries; | |
| 4798 | − | |
| 4799 | − mutex_lock(&memcg->thresholds_lock); | |
| 4800 | − | |
| 4801 | − if (type == _MEM) { | |
| 4802 | − thresholds = &memcg->thresholds; | |
| 4803 | − usage = mem_cgroup_usage(memcg, false); | |
| 4804 | − } else if (type == _MEMSWAP) { | |
| 4805 | − thresholds = &memcg->memsw_thresholds; | |
| 4806 | − usage = mem_cgroup_usage(memcg, true); | |
| 4807 | − } else | |
| 4808 | − BUG(); | |
| 4809 | − | |
| 4810 | − if (!thresholds->primary) | |
| 4811 | − goto unlock; | |
| 4812 | − | |
| 4813 | − /* Check if a threshold crossed before removing */ | |
| 4814 | − __mem_cgroup_threshold(memcg, type == _MEMSWAP); | |
| 4815 | − | |
| 4816 | − /* Calculate new number of threshold */ | |
| 4817 | − size = entries = 0; | |
| 4818 | − for (i = 0; i < thresholds->primary->size; i++) { | |
| 4819 | − if (thresholds->primary->entries[i].eventfd != eventfd) | |
| 4820 | − size++; | |
| 4821 | − else | |
| 4822 | − entries++; | |
| 4823 | − } | |
| 4824 | − | |
| 4825 | − new = thresholds->spare; | |
| 4826 | − | |
| 4827 | − /* If no items related to eventfd have been cleared, nothing to do */ | |
| 4828 | − if (!entries) | |
| 4829 | − goto unlock; | |
| 4830 | − | |
| 4831 | − /* Set thresholds array to NULL if we don't have thresholds */ | |
| 4832 | − if (!size) { | |
| 4833 | − kfree(new); | |
| 4834 | − new = NULL; | |
| 4835 | − goto swap_buffers; | |
| 4836 | − } | |
| 4837 | − | |
| 4838 | − new->size = size; | |
| 4839 | − | |
| 4840 | − /* Copy thresholds and find current threshold */ | |
| 4841 | − new->current_threshold = -1; | |
| 4842 | − for (i = 0, j = 0; i < thresholds->primary->size; i++) { | |
| 4843 | − if (thresholds->primary->entries[i].eventfd == eventfd) | |
| 4844 | − continue; | |
| 4845 | − | |
| 4846 | − new->entries[j] = thresholds->primary->entries[i]; | |
| 4847 | − if (new->entries[j].threshold <= usage) { | |
| 4848 | − /* | |
| 4849 | − * new->current_threshold will not be used | |
| 4850 | − * until rcu_assign_pointer(), so it's safe to increment | |
| 4851 | − * it here. | |
| 4852 | − */ | |
| 4853 | − ++new->current_threshold; | |
| 4854 | − } | |
| 4855 | − j++; | |
| 4856 | − } | |
| 4857 | − | |
| 4858 | −swap_buffers: | |
| 4859 | − /* Swap primary and spare array */ | |
| 4860 | − thresholds->spare = thresholds->primary; | |
| 4861 | − | |
| 4862 | − rcu_assign_pointer(thresholds->primary, new); | |
| 4863 | − | |
| 4864 | − /* To be sure that nobody uses thresholds */ | |
| 4865 | − synchronize_rcu(); | |
| 4866 | − | |
| 4867 | − /* If all events are unregistered, free the spare array */ | |
| 4868 | − if (!new) { | |
| 4869 | − kfree(thresholds->spare); | |
| 4870 | − thresholds->spare = NULL; | |
| 4871 | − } | |
| 4872 | −unlock: | |
| 4873 | − mutex_unlock(&memcg->thresholds_lock); | |
| 4874 | −} | |
| 4875 | − | |
| 4876 | −static void mem_cgroup_usage_unregister_event(struct mem_cgroup *memcg, | |
| 4877 | − struct eventfd_ctx *eventfd) | |
| 4878 | −{ | |
| 4879 | − return __mem_cgroup_usage_unregister_event(memcg, eventfd, _MEM); | |
| 4880 | −} | |
| 4881 | − | |
| 4882 | −static void memsw_cgroup_usage_unregister_event(struct mem_cgroup *memcg, | |
| 4883 | − struct eventfd_ctx *eventfd) | |
| 4884 | −{ | |
| 4885 | − return __mem_cgroup_usage_unregister_event(memcg, eventfd, _MEMSWAP); | |
| 4886 | −} | |
| 4887 | − | |
| 4888 | −static int mem_cgroup_oom_register_event(struct mem_cgroup *memcg, | |
| 4889 | − struct eventfd_ctx *eventfd, const char *args) | |
| 4890 | −{ | |
| 4891 | − struct mem_cgroup_eventfd_list *event; | |
| 4892 | − | |
| 4893 | − event = kmalloc(sizeof(*event), GFP_KERNEL); | |
| 4894 | − if (!event) | |
| 4895 | − return -ENOMEM; | |
| 4896 | − | |
| 4897 | − spin_lock(&memcg_oom_lock); | |
| 4898 | − | |
| 4899 | − event->eventfd = eventfd; | |
| 4900 | − list_add(&event->list, &memcg->oom_notify); | |
| 4901 | − | |
| 4902 | − /* already in OOM ? */ | |
| 4903 | − if (memcg->under_oom) | |
| 4904 | − eventfd_signal(eventfd); | |
| 4905 | − spin_unlock(&memcg_oom_lock); | |
| 4906 | − | |
| 4907 | − return 0; | |
| 4908 | −} | |
| 4909 | − | |
| 4910 | −static void mem_cgroup_oom_unregister_event(struct mem_cgroup *memcg, | |
| 4911 | − struct eventfd_ctx *eventfd) | |
| 4912 | −{ | |
| 4913 | − struct mem_cgroup_eventfd_list *ev, *tmp; | |
| 4914 | − | |
| 4915 | − spin_lock(&memcg_oom_lock); | |
| 4916 | − | |
| 4917 | − list_for_each_entry_safe(ev, tmp, &memcg->oom_notify, list) { | |
| 4918 | − if (ev->eventfd == eventfd) { | |
| 4919 | − list_del(&ev->list); | |
| 4920 | − kfree(ev); | |
| 4921 | − } | |
| 4922 | − } | |
| 4923 | − | |
| 4924 | − spin_unlock(&memcg_oom_lock); | |
| 4925 | −} | |
| 4926 | − | |
| 4927 | −static int mem_cgroup_oom_control_read(struct seq_file *sf, void *v) | |
| 4928 | −{ | |
| 4929 | − struct mem_cgroup *memcg = mem_cgroup_from_seq(sf); | |
| 4930 | − | |
| 4931 | − seq_printf(sf, "oom_kill_disable %d\n", READ_ONCE(memcg->oom_kill_disable)); | |
| 4932 | − seq_printf(sf, "under_oom %d\n", (bool)memcg->under_oom); | |
| 4933 | − seq_printf(sf, "oom_kill %lu\n", | |
| 4934 | − atomic_long_read(&memcg->memory_events[MEMCG_OOM_KILL])); | |
| 4935 | − return 0; | |
| 4936 | −} | |
| 4937 | − | |
| 4938 | −static int mem_cgroup_oom_control_write(struct cgroup_subsys_state *css, | |
| 4939 | − struct cftype *cft, u64 val) | |
| 4940 | −{ | |
| 4941 | − struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 4942 | − | |
| 4943 | − /* cannot set to root cgroup and only 0 and 1 are allowed */ | |
| 4944 | − if (mem_cgroup_is_root(memcg) || !((val == 0) || (val == 1))) | |
| 4945 | − return -EINVAL; | |
| 4946 | − | |
| 4947 | − WRITE_ONCE(memcg->oom_kill_disable, val); | |
| 4948 | − if (!val) | |
| 4949 | − memcg_oom_recover(memcg); | |
| 4950 | − | |
| 4951 | − return 0; | |
| 4952 | −} | |
| 4953 | − | |
| 4954 | −#ifdef CONFIG_CGROUP_WRITEBACK | |
| 4955 | − | |
| 4956 | −#include <trace/events/writeback.h> | |
| 4957 | − | |
| 4958 | −static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp) | |
| 4959 | −{ | |
| 4960 | − return wb_domain_init(&memcg->cgwb_domain, gfp); | |
| 4961 | −} | |
| 4962 | − | |
| 4963 | −static void memcg_wb_domain_exit(struct mem_cgroup *memcg) | |
| 4964 | −{ | |
| 4965 | − wb_domain_exit(&memcg->cgwb_domain); | |
| 4966 | −} | |
| 4967 | − | |
| 4968 | −static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg) | |
| 4969 | −{ | |
| 4970 | − wb_domain_size_changed(&memcg->cgwb_domain); | |
| 4971 | −} | |
| 4972 | − | |
| 4973 | −struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb) | |
| 4974 | −{ | |
| 4975 | − struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); | |
| 4976 | − | |
| 4977 | − if (!memcg->css.parent) | |
| 4978 | − return NULL; | |
| 4979 | − | |
| 4980 | − return &memcg->cgwb_domain; | |
| 4981 | −} | |
| 4982 | − | |
| 4983 | −/** | |
| 4984 | − * mem_cgroup_wb_stats - retrieve writeback related stats from its memcg | |
| 4985 | − * @wb: bdi_writeback in question | |
| 4986 | − * @pfilepages: out parameter for number of file pages | |
| 4987 | − * @pheadroom: out parameter for number of allocatable pages according to memcg | |
| 4988 | − * @pdirty: out parameter for number of dirty pages | |
| 4989 | − * @pwriteback: out parameter for number of pages under writeback | |
| 3363 | +/* | |
| 3364 | + * Private memory cgroup IDR | |
| 4990 | 3365 | * |
| 4991 | − * Determine the numbers of file, headroom, dirty, and writeback pages in | |
| 4992 | − * @wb's memcg. File, dirty and writeback are self-explanatory. Headroom | |
| 4993 | − * is a bit more involved. | |
| 3366 | + * Swap-out records and page cache shadow entries need to store memcg | |
| 3367 | + * references in constrained space, so we maintain an ID space that is | |
| 3368 | + * limited to 16 bit (MEM_CGROUP_ID_MAX), limiting the total number of | |
| 3369 | + * memory-controlled cgroups to 64k. | |
| 4994 | 3370 | * |
| 4995 | − * A memcg's headroom is "min(max, high) - used". In the hierarchy, the | |
| 4996 | − * headroom is calculated as the lowest headroom of itself and the | |
| 4997 | − * ancestors. Note that this doesn't consider the actual amount of | |
| 4998 | − * available memory in the system. The caller should further cap | |
| 4999 | − * *@pheadroom accordingly. | |
| 5000 | − */ | |
| 5001 | −void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, | |
| 5002 | − unsigned long *pheadroom, unsigned long *pdirty, | |
| 5003 | − unsigned long *pwriteback) | |
| 5004 | −{ | |
| 5005 | − struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); | |
| 5006 | − struct mem_cgroup *parent; | |
| 5007 | − | |
| 5008 | − mem_cgroup_flush_stats_ratelimited(memcg); | |
| 5009 | − | |
| 5010 | − *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); | |
| 5011 | − *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); | |
| 5012 | − *pfilepages = memcg_page_state(memcg, NR_INACTIVE_FILE) + | |
| 5013 | − memcg_page_state(memcg, NR_ACTIVE_FILE); | |
| 5014 | − | |
| 5015 | − *pheadroom = PAGE_COUNTER_MAX; | |
| 5016 | − while ((parent = parent_mem_cgroup(memcg))) { | |
| 5017 | − unsigned long ceiling = min(READ_ONCE(memcg->memory.max), | |
| 5018 | − READ_ONCE(memcg->memory.high)); | |
| 5019 | − unsigned long used = page_counter_read(&memcg->memory); | |
| 5020 | − | |
| 5021 | − *pheadroom = min(*pheadroom, ceiling - min(ceiling, used)); | |
| 5022 | − memcg = parent; | |
| 5023 | − } | |
| 5024 | −} | |
| 5025 | − | |
| 5026 | −/* | |
| 5027 | − * Foreign dirty flushing | |
| 3371 | + * However, there usually are many references to the offline CSS after | |
| 3372 | + * the cgroup has been destroyed, such as page cache or reclaimable | |
| 3373 | + * slab objects, that don't need to hang on to the ID. We want to keep | |
| 3374 | + * those dead CSS from occupying IDs, or we might quickly exhaust the | |
| 3375 | + * relatively small ID space and prevent the creation of new cgroups | |
| 3376 | + * even when there are much fewer than 64k cgroups - possibly none. | |
| 5028 | 3377 | * |
| 5029 | − * There's an inherent mismatch between memcg and writeback. The former | |
| 5030 | − * tracks ownership per-page while the latter per-inode. This was a | |
| 5031 | − * deliberate design decision because honoring per-page ownership in the | |
| 5032 | − * writeback path is complicated, may lead to higher CPU and IO overheads | |
| 5033 | − * and deemed unnecessary given that write-sharing an inode across | |
| 5034 | − * different cgroups isn't a common use-case. | |
| 3378 | + * Maintain a private 16-bit ID space for memcg, and allow the ID to | |
| 3379 | + * be freed and recycled when it's no longer needed, which is usually | |
| 3380 | + * when the CSS is offlined. | |
| 5035 | 3381 | * |
| 5036 | − * Combined with inode majority-writer ownership switching, this works well | |
| 5037 | − * enough in most cases but there are some pathological cases. For | |
| 5038 | − * example, let's say there are two cgroups A and B which keep writing to | |
| 5039 | − * different but confined parts of the same inode. B owns the inode and | |
| 5040 | − * A's memory is limited far below B's. A's dirty ratio can rise enough to | |
| 5041 | − * trigger balance_dirty_pages() sleeps but B's can be low enough to avoid | |
| 5042 | − * triggering background writeback. A will be slowed down without a way to | |
| 5043 | − * make writeback of the dirty pages happen. | |
| 5044 | − * | |
| 5045 | − * Conditions like the above can lead to a cgroup getting repeatedly and | |
| 5046 | − * severely throttled after making some progress after each | |
| 5047 | − * dirty_expire_interval while the underlying IO device is almost | |
| 5048 | − * completely idle. | |
| 5049 | − * | |
| 5050 | − * Solving this problem completely requires matching the ownership tracking | |
| 5051 | − * granularities between memcg and writeback in either direction. However, | |
| 5052 | − * the more egregious behaviors can be avoided by simply remembering the | |
| 5053 | − * most recent foreign dirtying events and initiating remote flushes on | |
| 5054 | − * them when local writeback isn't enough to keep the memory clean enough. | |
| 5055 | − * | |
| 5056 | − * The following two functions implement such mechanism. When a foreign | |
| 5057 | − * page - a page whose memcg and writeback ownerships don't match - is | |
| 5058 | − * dirtied, mem_cgroup_track_foreign_dirty() records the inode owning | |
| 5059 | − * bdi_writeback on the page owning memcg. When balance_dirty_pages() | |
| 5060 | − * decides that the memcg needs to sleep due to high dirty ratio, it calls | |
| 5061 | − * mem_cgroup_flush_foreign() which queues writeback on the recorded | |
| 5062 | − * foreign bdi_writebacks which haven't expired. Both the numbers of | |
| 5063 | − * recorded bdi_writebacks and concurrent in-flight foreign writebacks are | |
| 5064 | − * limited to MEMCG_CGWB_FRN_CNT. | |
| 5065 | − * | |
| 5066 | − * The mechanism only remembers IDs and doesn't hold any object references. | |
| 5067 | − * As being wrong occasionally doesn't matter, updates and accesses to the | |
| 5068 | − * records are lockless and racy. | |
| 5069 | − */ | |
| 5070 | −void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio, | |
| 5071 | − struct bdi_writeback *wb) | |
| 5072 | −{ | |
| 5073 | − struct mem_cgroup *memcg = folio_memcg(folio); | |
| 5074 | − struct memcg_cgwb_frn *frn; | |
| 5075 | − u64 now = get_jiffies_64(); | |
| 5076 | − u64 oldest_at = now; | |
| 5077 | − int oldest = -1; | |
| 5078 | − int i; | |
| 5079 | − | |
| 5080 | − trace_track_foreign_dirty(folio, wb); | |
| 5081 | − | |
| 5082 | − /* | |
| 5083 | − * Pick the slot to use. If there is already a slot for @wb, keep | |
| 5084 | − * using it. If not replace the oldest one which isn't being | |
| 5085 | − * written out. | |
| 5086 | − */ | |
| 5087 | − for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) { | |
| 5088 | − frn = &memcg->cgwb_frn[i]; | |
| 5089 | − if (frn->bdi_id == wb->bdi->id && | |
| 5090 | − frn->memcg_id == wb->memcg_css->id) | |
| 5091 | − break; | |
| 5092 | − if (time_before64(frn->at, oldest_at) && | |
| 5093 | − atomic_read(&frn->done.cnt) == 1) { | |
| 5094 | − oldest = i; | |
| 5095 | − oldest_at = frn->at; | |
| 5096 | − } | |
| 5097 | − } | |
| 5098 | − | |
| 5099 | − if (i < MEMCG_CGWB_FRN_CNT) { | |
| 5100 | − /* | |
| 5101 | − * Re-using an existing one. Update timestamp lazily to | |
| 5102 | − * avoid making the cacheline hot. We want them to be | |
| 5103 | − * reasonably up-to-date and significantly shorter than | |
| 5104 | − * dirty_expire_interval as that's what expires the record. | |
| 5105 | − * Use the shorter of 1s and dirty_expire_interval / 8. | |
| 5106 | − */ | |
| 5107 | − unsigned long update_intv = | |
| 5108 | − min_t(unsigned long, HZ, | |
| 5109 | − msecs_to_jiffies(dirty_expire_interval * 10) / 8); | |
| 5110 | − | |
| 5111 | − if (time_before64(frn->at, now - update_intv)) | |
| 5112 | − frn->at = now; | |
| 5113 | − } else if (oldest >= 0) { | |
| 5114 | − /* replace the oldest free one */ | |
| 5115 | − frn = &memcg->cgwb_frn[oldest]; | |
| 5116 | − frn->bdi_id = wb->bdi->id; | |
| 5117 | − frn->memcg_id = wb->memcg_css->id; | |
| 5118 | − frn->at = now; | |
| 5119 | − } | |
| 5120 | −} | |
| 5121 | − | |
| 5122 | −/* issue foreign writeback flushes for recorded foreign dirtying events */ | |
| 5123 | −void mem_cgroup_flush_foreign(struct bdi_writeback *wb) | |
| 5124 | −{ | |
| 5125 | − struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); | |
| 5126 | − unsigned long intv = msecs_to_jiffies(dirty_expire_interval * 10); | |
| 5127 | − u64 now = jiffies_64; | |
| 5128 | − int i; | |
| 5129 | − | |
| 5130 | − for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) { | |
| 5131 | − struct memcg_cgwb_frn *frn = &memcg->cgwb_frn[i]; | |
| 5132 | − | |
| 5133 | − /* | |
| 5134 | − * If the record is older than dirty_expire_interval, | |
| 5135 | − * writeback on it has already started. No need to kick it | |
| 5136 | − * off again. Also, don't start a new one if there's | |
| 5137 | − * already one in flight. | |
| 5138 | − */ | |
| 5139 | − if (time_after64(frn->at, now - intv) && | |
| 5140 | − atomic_read(&frn->done.cnt) == 1) { | |
| 5141 | − frn->at = 0; | |
| 5142 | − trace_flush_foreign(wb, frn->bdi_id, frn->memcg_id); | |
| 5143 | − cgroup_writeback_by_id(frn->bdi_id, frn->memcg_id, | |
| 5144 | − WB_REASON_FOREIGN_FLUSH, | |
| 5145 | − &frn->done); | |
| 5146 | − } | |
| 5147 | − } | |
| 5148 | −} | |
| 5149 | − | |
| 5150 | −#else /* CONFIG_CGROUP_WRITEBACK */ | |
| 5151 | − | |
| 5152 | −static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp) | |
| 5153 | −{ | |
| 5154 | − return 0; | |
| 5155 | −} | |
| 5156 | − | |
| 5157 | −static void memcg_wb_domain_exit(struct mem_cgroup *memcg) | |
| 5158 | −{ | |
| 5159 | −} | |
| 5160 | − | |
| 5161 | −static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg) | |
| 5162 | −{ | |
| 5163 | −} | |
| 5164 | − | |
| 5165 | −#endif /* CONFIG_CGROUP_WRITEBACK */ | |
| 5166 | − | |
| 5167 | −/* | |
| 5168 | − * DO NOT USE IN NEW FILES. | |
| 5169 | − * | |
| 5170 | − * "cgroup.event_control" implementation. | |
| 5171 | − * | |
| 5172 | − * This is way over-engineered. It tries to support fully configurable | |
| 5173 | − * events for each user. Such level of flexibility is completely | |
| 5174 | − * unnecessary especially in the light of the planned unified hierarchy. | |
| 5175 | − * | |
| 5176 | − * Please deprecate this and replace with something simpler if at all | |
| 5177 | − * possible. | |
| 5178 | − */ | |
| 5179 | − | |
| 5180 | −/* | |
| 5181 | − * Unregister event and free resources. | |
| 5182 | − * | |
| 5183 | − * Gets called from workqueue. | |
| 5184 | − */ | |
| 5185 | −static void memcg_event_remove(struct work_struct *work) | |
| 5186 | −{ | |
| 5187 | − struct mem_cgroup_event *event = | |
| 5188 | − container_of(work, struct mem_cgroup_event, remove); | |
| 5189 | − struct mem_cgroup *memcg = event->memcg; | |
| 5190 | − | |
| 5191 | − remove_wait_queue(event->wqh, &event->wait); | |
| 5192 | − | |
| 5193 | − event->unregister_event(memcg, event->eventfd); | |
| 5194 | − | |
| 5195 | − /* Notify userspace the event is going away. */ | |
| 5196 | − eventfd_signal(event->eventfd); | |
| 5197 | − | |
| 5198 | − eventfd_ctx_put(event->eventfd); | |
| 5199 | − kfree(event); | |
| 5200 | − css_put(&memcg->css); | |
| 5201 | −} | |
| 5202 | − | |
| 5203 | −/* | |
| 5204 | − * Gets called on EPOLLHUP on eventfd when user closes it. | |
| 5205 | − * | |
| 5206 | − * Called with wqh->lock held and interrupts disabled. | |
| 5207 | − */ | |
| 5208 | −static int memcg_event_wake(wait_queue_entry_t *wait, unsigned mode, | |
| 5209 | − int sync, void *key) | |
| 5210 | −{ | |
| 5211 | − struct mem_cgroup_event *event = | |
| 5212 | − container_of(wait, struct mem_cgroup_event, wait); | |
| 5213 | − struct mem_cgroup *memcg = event->memcg; | |
| 5214 | − __poll_t flags = key_to_poll(key); | |
| 5215 | − | |
| 5216 | − if (flags & EPOLLHUP) { | |
| 5217 | − /* | |
| 5218 | − * If the event has been detached at cgroup removal, we | |
| 5219 | − * can simply return knowing the other side will cleanup | |
| 5220 | − * for us. | |
| 5221 | − * | |
| 5222 | − * We can't race against event freeing since the other | |
| 5223 | − * side will require wqh->lock via remove_wait_queue(), | |
| 5224 | − * which we hold. | |
| 5225 | − */ | |
| 5226 | − spin_lock(&memcg->event_list_lock); | |
| 5227 | − if (!list_empty(&event->list)) { | |
| 5228 | − list_del_init(&event->list); | |
| 5229 | − /* | |
| 5230 | − * We are in atomic context, but cgroup_event_remove() | |
| 5231 | − * may sleep, so we have to call it in workqueue. | |
| 5232 | − */ | |
| 5233 | − schedule_work(&event->remove); | |
| 5234 | − } | |
| 5235 | − spin_unlock(&memcg->event_list_lock); | |
| 5236 | − } | |
| 5237 | − | |
| 5238 | − return 0; | |
| 5239 | −} | |
| 5240 | − | |
| 5241 | −static void memcg_event_ptable_queue_proc(struct file *file, | |
| 5242 | − wait_queue_head_t *wqh, poll_table *pt) | |
| 5243 | −{ | |
| 5244 | − struct mem_cgroup_event *event = | |
| 5245 | − container_of(pt, struct mem_cgroup_event, pt); | |
| 5246 | − | |
| 5247 | − event->wqh = wqh; | |
| 5248 | − add_wait_queue(wqh, &event->wait); | |
| 5249 | −} | |
| 5250 | − | |
| 5251 | −/* | |
| 5252 | − * DO NOT USE IN NEW FILES. | |
| 5253 | − * | |
| 5254 | − * Parse input and register new cgroup event handler. | |
| 5255 | − * | |
| 5256 | − * Input must be in format '<event_fd> <control_fd> <args>'. | |
| 5257 | − * Interpretation of args is defined by control file implementation. | |
| 5258 | − */ | |
| 5259 | −static ssize_t memcg_write_event_control(struct kernfs_open_file *of, | |
| 5260 | − char *buf, size_t nbytes, loff_t off) | |
| 5261 | −{ | |
| 5262 | − struct cgroup_subsys_state *css = of_css(of); | |
| 5263 | − struct mem_cgroup *memcg = mem_cgroup_from_css(css); | |
| 5264 | − struct mem_cgroup_event *event; | |
| 5265 | − struct cgroup_subsys_state *cfile_css; | |
| 5266 | − unsigned int efd, cfd; | |
| 5267 | − struct fd efile; | |
| 5268 | − struct fd cfile; | |
| 5269 | − struct dentry *cdentry; | |
| 5270 | − const char *name; | |
| 5271 | − char *endp; | |
| 5272 | − int ret; | |
| 5273 | − | |
| 5274 | − if (IS_ENABLED(CONFIG_PREEMPT_RT)) | |
| 5275 | − return -EOPNOTSUPP; | |
| 5276 | − | |
| 5277 | − buf = strstrip(buf); | |
| 5278 | − | |
| 5279 | − efd = simple_strtoul(buf, &endp, 10); | |
| 5280 | − if (*endp != ' ') | |
| 5281 | − return -EINVAL; | |
| 5282 | − buf = endp + 1; | |
| 5283 | − | |
| 5284 | − cfd = simple_strtoul(buf, &endp, 10); | |
| 5285 | − if ((*endp != ' ') && (*endp != '\0')) | |
| 5286 | − return -EINVAL; | |
| 5287 | − buf = endp + 1; | |
| 5288 | − | |
| 5289 | − event = kzalloc(sizeof(*event), GFP_KERNEL); | |
| 5290 | − if (!event) | |
| 5291 | − return -ENOMEM; | |
| 5292 | − | |
| 5293 | − event->memcg = memcg; | |
| 5294 | − INIT_LIST_HEAD(&event->list); | |
| 5295 | − init_poll_funcptr(&event->pt, memcg_event_ptable_queue_proc); | |
| 5296 | − init_waitqueue_func_entry(&event->wait, memcg_event_wake); | |
| 5297 | − INIT_WORK(&event->remove, memcg_event_remove); | |
| 5298 | − | |
| 5299 | − efile = fdget(efd); | |
| 5300 | − if (!efile.file) { | |
| 5301 | − ret = -EBADF; | |
| 5302 | − goto out_kfree; | |
| 5303 | − } | |
| 5304 | − | |
| 5305 | − event->eventfd = eventfd_ctx_fileget(efile.file); | |
| 5306 | − if (IS_ERR(event->eventfd)) { | |
| 5307 | − ret = PTR_ERR(event->eventfd); | |
| 5308 | − goto out_put_efile; | |
| 5309 | − } | |
| 5310 | − | |
| 5311 | − cfile = fdget(cfd); | |
| 5312 | − if (!cfile.file) { | |
| 5313 | − ret = -EBADF; | |
| 5314 | − goto out_put_eventfd; | |
| 5315 | − } | |
| 5316 | − | |
| 5317 | − /* the process need read permission on control file */ | |
| 5318 | − /* AV: shouldn't we check that it's been opened for read instead? */ | |
| 5319 | − ret = file_permission(cfile.file, MAY_READ); | |
| 5320 | − if (ret < 0) | |
| 5321 | − goto out_put_cfile; | |
| 5322 | − | |
| 5323 | − /* | |
| 5324 | − * The control file must be a regular cgroup1 file. As a regular cgroup | |
| 5325 | − * file can't be renamed, it's safe to access its name afterwards. | |
| 5326 | − */ | |
| 5327 | − cdentry = cfile.file->f_path.dentry; | |
| 5328 | − if (cdentry->d_sb->s_type != &cgroup_fs_type || !d_is_reg(cdentry)) { | |
| 5329 | − ret = -EINVAL; | |
| 5330 | − goto out_put_cfile; | |
| 5331 | − } | |
| 5332 | − | |
| 5333 | − /* | |
| 5334 | − * Determine the event callbacks and set them in @event. This used | |
| 5335 | − * to be done via struct cftype but cgroup core no longer knows | |
| 5336 | − * about these events. The following is crude but the whole thing | |
| 5337 | − * is for compatibility anyway. | |
| 5338 | − * | |
| 5339 | − * DO NOT ADD NEW FILES. | |
| 5340 | − */ | |
| 5341 | − name = cdentry->d_name.name; | |
| 5342 | − | |
| 5343 | − if (!strcmp(name, "memory.usage_in_bytes")) { | |
| 5344 | − event->register_event = mem_cgroup_usage_register_event; | |
| 5345 | − event->unregister_event = mem_cgroup_usage_unregister_event; | |
| 5346 | − } else if (!strcmp(name, "memory.oom_control")) { | |
| 5347 | − event->register_event = mem_cgroup_oom_register_event; | |
| 5348 | − event->unregister_event = mem_cgroup_oom_unregister_event; | |
| 5349 | − } else if (!strcmp(name, "memory.pressure_level")) { | |
| 5350 | − event->register_event = vmpressure_register_event; | |
| 5351 | − event->unregister_event = vmpressure_unregister_event; | |
| 5352 | − } else if (!strcmp(name, "memory.memsw.usage_in_bytes")) { | |
| 5353 | − event->register_event = memsw_cgroup_usage_register_event; | |
| 5354 | − event->unregister_event = memsw_cgroup_usage_unregister_event; | |
| 5355 | − } else { | |
| 5356 | − ret = -EINVAL; | |
| 5357 | − goto out_put_cfile; | |
| 5358 | − } | |
| 5359 | − | |
| 5360 | − /* | |
| 5361 | − * Verify @cfile should belong to @css. Also, remaining events are | |
| 5362 | − * automatically removed on cgroup destruction but the removal is | |
| 5363 | − * asynchronous, so take an extra ref on @css. | |
| 5364 | − */ | |
| 5365 | − cfile_css = css_tryget_online_from_dir(cdentry->d_parent, | |
| 5366 | − &memory_cgrp_subsys); | |
| 5367 | − ret = -EINVAL; | |
| 5368 | − if (IS_ERR(cfile_css)) | |
| 5369 | − goto out_put_cfile; | |
| 5370 | − if (cfile_css != css) { | |
| 5371 | − css_put(cfile_css); | |
| 5372 | − goto out_put_cfile; | |
| 5373 | − } | |
| 5374 | − | |
| 5375 | − ret = event->register_event(memcg, event->eventfd, buf); | |
| 5376 | − if (ret) | |
| 5377 | − goto out_put_css; | |
| 5378 | − | |
| 5379 | − vfs_poll(efile.file, &event->pt); | |
| 5380 | − | |
| 5381 | − spin_lock_irq(&memcg->event_list_lock); | |
| 5382 | − list_add(&event->list, &memcg->event_list); | |
| 5383 | − spin_unlock_irq(&memcg->event_list_lock); | |
| 5384 | − | |
| 5385 | − fdput(cfile); | |
| 5386 | − fdput(efile); | |
| 5387 | − | |
| 5388 | − return nbytes; | |
| 5389 | − | |
| 5390 | −out_put_css: | |
| 5391 | − css_put(css); | |
| 5392 | −out_put_cfile: | |
| 5393 | − fdput(cfile); | |
| 5394 | −out_put_eventfd: | |
| 5395 | − eventfd_ctx_put(event->eventfd); | |
| 5396 | −out_put_efile: | |
| 5397 | − fdput(efile); | |
| 5398 | −out_kfree: | |
| 5399 | − kfree(event); | |
| 5400 | − | |
| 5401 | − return ret; | |
| 5402 | −} | |
| 5403 | − | |
| 5404 | −#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_SLUB_DEBUG) | |
| 5405 | −static int mem_cgroup_slab_show(struct seq_file *m, void *p) | |
| 5406 | −{ | |
| 5407 | − /* | |
2281 more lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 688 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 776 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 415 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 220 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 210 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 424 lines aren’t shown here.
This diff is large: its 180 lines aren’t shown here.
This diff is large: its 177 lines aren’t shown here.
This diff is large: its 62 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 92 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 372 lines aren’t shown here.
This diff is large: its 331 lines aren’t shown here.
This diff is large: its 529 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 135 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 264 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 428 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 508 lines aren’t shown here.
This diff is large: its 329 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 553 lines aren’t shown here.
This diff is large: its 332 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 235 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 281 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 229 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 479 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 499 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.