Pull MM updates from Andrew Morton: - Yu Zhao's Multi-Gen LRU patches are here. They've been under test in linux-next for a couple of months without, to my knowledge, any negative reports (or any positive ones, come to that). - Also the Maple Tree from Liam Howlett. An overlapping range-based tree for vmas. It it apparently slightly more efficient in its own right, but is mainly targeted at enabling work to reduce mmap_lock contention. Liam has identified a number of other tree users in the kernel which could be beneficially onverted to mapletrees. Yu Zhao has identified a hard-to-hit but "easy to fix" lockdep splat at [1]. This has yet to be addressed due to Liam's unfortunately timed vacation. He is now back and we'll get this fixed up. - Dmitry Vyukov introduces KMSAN: the Kernel Memory Sanitizer. It uses clang-generated instrumentation to detect used-unintialized bugs down to the single bit level. KMSAN keeps finding bugs. New ones, as well as the legacy ones. - Yang Shi adds a userspace mechanism (madvise) to induce a collapse of memory into THPs. - Zach O'Keefe has expanded Yang Shi's madvise(MADV_COLLAPSE) to support file/shmem-backed pages. - userfaultfd updates from Axel Rasmussen - zsmalloc cleanups from Alexey Romanov - cleanups from Miaohe Lin: vmscan, hugetlb_cgroup, hugetlb and memory-failure - Huang Ying adds enhancements to NUMA balancing memory tiering mode's page promotion, with a new way of detecting hot pages. - memcg updates from Shakeel Butt: charging optimizations and reduced memory consumption. - memcg cleanups from Kairui Song. - memcg fixes and cleanups from Johannes Weiner. - Vishal Moola provides more folio conversions - Zhang Yi removed ll_rw_block() :( - migration enhancements from Peter Xu - migration error-path bugfixes from Huang Ying - Aneesh Kumar added ability for a device driver to alter the memory tiering promotion paths. For optimizations by PMEM drivers, DRM drivers, etc. - vma merging improvements from Jakub Matěn. - NUMA hinting cleanups from David Hildenbrand. - xu xin added aditional userspace visibility into KSM merging activity. - THP & KSM code consolidation from Qi Zheng. - more folio work from Matthew Wilcox. - KASAN updates from Andrey Konovalov. - DAMON cleanups from Kaixu Xia. - DAMON work from SeongJae Park: fixes, cleanups. - hugetlb sysfs cleanups from Muchun Song. - Mike Kravetz fixes locking issues in hugetlbfs and in hugetlb core. Link: https://lkml.kernel.org/r/CAOUHufZabH85CeUN-MEMgL8gJGzJEWUrkiM58JkTbBhh-jew0Q@mail.gmail.com [1] * tag 'mm-stable-2022-10-08' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (555 commits) hugetlb: allocate vma lock for all sharable vmas hugetlb: take hugetlb vma_lock when clearing vma_lock->vma pointer hugetlb: fix vma lock handling during split vma and range unmapping mglru: mm/vmscan.c: fix imprecise comments mm/mglru: don't sync disk for each aging cycle mm: memcontrol: drop dead CONFIG_MEMCG_SWAP config symbol mm: memcontrol: use do_memsw_account() in a few more places mm: memcontrol: deprecate swapaccounting=0 mode mm: memcontrol: don't allocate cgroup swap arrays when memcg is disabled mm/secretmem: remove reduntant return value mm/hugetlb: add available_huge_pages() func mm: remove unused inline functions from include/linux/mm_inline.h selftests/vm: add selftest for MADV_COLLAPSE of uffd-minor memory selftests/vm: add file/shmem MADV_COLLAPSE selftest for cleared pmd selftests/vm: add thp collapse shmem testing selftests/vm: add thp collapse file and tmpfs testing selftests/vm: modularize thp collapse memory operations selftests/vm: dedup THP helpers mm/khugepaged: add tracepoint to hpage_collapse_scan_file() mm/madvise: add file and shmem support to MADV_COLLAPSE ...
Unified · Split
| @@ -0,0 +1,25 @@ | ||
| 1 | +What: /sys/devices/virtual/memory_tiering/ | |
| 2 | +Date: August 2022 | |
| 3 | +Contact: Linux memory management mailing list <linux-mm@kvack.org> | |
| 4 | +Description: A collection of all the memory tiers allocated. | |
| 5 | + | |
| 6 | + Individual memory tier details are contained in subdirectories | |
| 7 | + named by the abstract distance of the memory tier. | |
| 8 | + | |
| 9 | + /sys/devices/virtual/memory_tiering/memory_tierN/ | |
| 10 | + | |
| 11 | + | |
| 12 | +What: /sys/devices/virtual/memory_tiering/memory_tierN/ | |
| 13 | + /sys/devices/virtual/memory_tiering/memory_tierN/nodes | |
| 14 | +Date: August 2022 | |
| 15 | +Contact: Linux memory management mailing list <linux-mm@kvack.org> | |
| 16 | +Description: Directory with details of a specific memory tier | |
| 17 | + | |
| 18 | + This is the directory containing information about a particular | |
| 19 | + memory tier, memtierN, where N is derived based on abstract distance. | |
| 20 | + | |
| 21 | + A smaller value of N implies a higher (faster) memory tier in the | |
| 22 | + hierarchy. | |
| 23 | + | |
| 24 | + nodes: NUMA nodes that are part of this memory tier. | |
| 25 | + | |
| @@ -13,7 +13,7 @@ a) waiting for a CPU (while being runnable) · 12 unmodified lines | ||
| 13 | 13 | b) completion of synchronous block I/O initiated by the task |
| 14 | 14 | c) swapping in pages |
| 15 | 15 | d) memory reclaim |
| 16 | −e) thrashing page cache | |
| 16 | +e) thrashing | |
| 17 | 17 | f) direct compact |
| 18 | 18 | g) write-protect copy |
| 19 | 19 | |
| @@ -299,7 +299,7 @@ Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by · 298 unmodified lines | ||
| 299 | 299 | lruvec->lru_lock; PG_lru bit of page->flags is cleared before |
| 300 | 300 | isolating a page from its LRU under lruvec->lru_lock. |
| 301 | 301 | |
| 302 | −2.7 Kernel Memory Extension (CONFIG_MEMCG_KMEM) | |
| 302 | +2.7 Kernel Memory Extension | |
| 303 | 303 | ----------------------------------------------- |
| 304 | 304 | |
| 305 | 305 | With the Kernel memory extension, the Memory Controller is able to limit |
| @@ -386,8 +386,6 @@ U != 0, K >= U: · 80 unmodified lines | ||
| 386 | 386 | |
| 387 | 387 | a. Enable CONFIG_CGROUPS |
| 388 | 388 | b. Enable CONFIG_MEMCG |
| 389 | −c. Enable CONFIG_MEMCG_SWAP (to use swap extension) | |
| 390 | −d. Enable CONFIG_MEMCG_KMEM (to use kmem extension) | |
| 391 | 389 | |
| 392 | 390 | 3.1. Prepare the cgroups (see cgroups.txt, Why are cgroups needed?) |
| 393 | 391 | ------------------------------------------------------------------- |
| @@ -1469,6 +1469,14 @@ · 1468 unmodified lines | ||
| 1469 | 1469 | Permit 'security.evm' to be updated regardless of |
| 1470 | 1470 | current integrity status. |
| 1471 | 1471 | |
| 1472 | + early_page_ext [KNL] Enforces page_ext initialization to earlier | |
| 1473 | + stages so cover more early boot allocations. | |
| 1474 | + Please note that as side effect some optimizations | |
| 1475 | + might be disabled to achieve that (e.g. parallelized | |
| 1476 | + memory initialization is disabled) so the boot process | |
| 1477 | + might take longer, especially on systems with a lot of | |
| 1478 | + memory. Available with CONFIG_PAGE_EXTENSION=y. | |
| 1479 | + | |
| 1472 | 1480 | failslab= |
| 1473 | 1481 | fail_usercopy= |
| 1474 | 1482 | fail_page_alloc= |
| @@ -6041,12 +6049,6 @@ · 4566 unmodified lines | ||
| 6041 | 6049 | This parameter controls use of the Protected |
| 6042 | 6050 | Execution Facility on pSeries. |
| 6043 | 6051 | |
| 6044 | − swapaccount= [KNL] | |
| 6045 | − Format: [0|1] | |
| 6046 | − Enable accounting of swap in memory resource | |
| 6047 | − controller if no parameter or 1 is given or disable | |
| 6048 | − it if 0 is given (See Documentation/admin-guide/cgroup-v1/memory.rst) | |
| 6049 | − | |
| 6050 | 6052 | swiotlb= [ARM,IA-64,PPC,MIPS,X86] |
| 6051 | 6053 | Format: { <int> [,<int>] | force | noforce } |
| 6052 | 6054 | <int> -- Number of I/O TLB slabs |
| @@ -5,10 +5,10 @@ CMA Debugfs Interface · 4 unmodified lines | ||
| 5 | 5 | The CMA debugfs interface is useful to retrieve basic information out of the |
| 6 | 6 | different CMA areas and to test allocation/release in each of the areas. |
| 7 | 7 | |
| 8 | −Each CMA zone represents a directory under <debugfs>/cma/, indexed by the | |
| 9 | −kernel's CMA index. So the first CMA zone would be: | |
| 8 | +Each CMA area represents a directory under <debugfs>/cma/, represented by | |
| 9 | +its CMA name like below: | |
| 10 | 10 | |
| 11 | − <debugfs>/cma/cma-0 | |
| 11 | + <debugfs>/cma/<cma_name> | |
| 12 | 12 | |
| 13 | 13 | The structure of the files created under that directory is as follows: |
| 14 | 14 | |
| @@ -18,8 +18,8 @@ The structure of the files created under that directory is as follows: · 3 unmodified lines | ||
| 18 | 18 | - [RO] bitmap: The bitmap of page states in the zone. |
| 19 | 19 | - [WO] alloc: Allocate N pages from that CMA area. For example:: |
| 20 | 20 | |
| 21 | − echo 5 > <debugfs>/cma/cma-2/alloc | |
| 21 | + echo 5 > <debugfs>/cma/<cma_name>/alloc | |
| 22 | 22 | |
| 23 | −would try to allocate 5 pages from the cma-2 area. | |
| 23 | +would try to allocate 5 pages from the 'cma_name' area. | |
| 24 | 24 | |
| 25 | 25 | - [WO] free: Free N pages from that CMA area, similar to the above. |
| @@ -1,8 +1,8 @@ | ||
| 1 | 1 | .. SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | |
| 3 | −======================== | |
| 4 | −Monitoring Data Accesses | |
| 5 | −======================== | |
| 3 | +========================== | |
| 4 | +DAMON: Data Access MONitor | |
| 5 | +========================== | |
| 6 | 6 | |
| 7 | 7 | :doc:`DAMON </mm/damon/index>` allows light-weight data access monitoring. |
| 8 | 8 | Using DAMON, users can analyze the memory access patterns of their systems and |
| @@ -29,16 +29,9 @@ called DAMON Operator (DAMO). It is available at · 28 unmodified lines | ||
| 29 | 29 | https://github.com/awslabs/damo. The examples below assume that ``damo`` is on |
| 30 | 30 | your ``$PATH``. It's not mandatory, though. |
| 31 | 31 | |
| 32 | −Because DAMO is using the debugfs interface (refer to :doc:`usage` for the | |
| 33 | −detail) of DAMON, you should ensure debugfs is mounted. Mount it manually as | |
| 34 | −below:: | |
| 35 | − | |
| 36 | − # mount -t debugfs none /sys/kernel/debug/ | |
| 37 | − | |
| 38 | −or append the following line to your ``/etc/fstab`` file so that your system | |
| 39 | −can automatically mount debugfs upon booting:: | |
| 40 | − | |
| 41 | − debugfs /sys/kernel/debug debugfs defaults 0 0 | |
| 32 | +Because DAMO is using the sysfs interface (refer to :doc:`usage` for the | |
| 33 | +detail) of DAMON, you should ensure :doc:`sysfs </filesystems/sysfs>` is | |
| 34 | +mounted. | |
| 42 | 35 | |
| 43 | 36 | |
| 44 | 37 | Recording Data Access Patterns |
| @@ -393,6 +393,11 @@ the files as above. Above is only for an example. · 392 unmodified lines | ||
| 393 | 393 | debugfs Interface |
| 394 | 394 | ================= |
| 395 | 395 | |
| 396 | +.. note:: | |
| 397 | + | |
| 398 | + DAMON debugfs interface will be removed after next LTS kernel is released, so | |
| 399 | + users should move to the :ref:`sysfs interface <sysfs_interface>`. | |
| 400 | + | |
| 396 | 401 | DAMON exports eight files, ``attrs``, ``target_ids``, ``init_regions``, |
| 397 | 402 | ``schemes``, ``monitor_on``, ``kdamond_pid``, ``mk_contexts`` and |
| 398 | 403 | ``rm_contexts`` under its debugfs directory, ``<debugfs>/damon/``. |
| @@ -32,6 +32,7 @@ the Linux memory management. · 31 unmodified lines | ||
| 32 | 32 | idle_page_tracking |
| 33 | 33 | ksm |
| 34 | 34 | memory-hotplug |
| 35 | + multigen_lru | |
| 35 | 36 | nommu-mmap |
| 36 | 37 | numa_memory_policy |
| 37 | 38 | numaperf |
| @@ -184,6 +184,42 @@ The maximum possible ``pages_sharing/pages_shared`` ratio is limited by the · 183 unmodified lines | ||
| 184 | 184 | ``max_page_sharing`` tunable. To increase the ratio ``max_page_sharing`` must |
| 185 | 185 | be increased accordingly. |
| 186 | 186 | |
| 187 | +Monitoring KSM profit | |
| 188 | +===================== | |
| 189 | + | |
| 190 | +KSM can save memory by merging identical pages, but also can consume | |
| 191 | +additional memory, because it needs to generate a number of rmap_items to | |
| 192 | +save each scanned page's brief rmap information. Some of these pages may | |
| 193 | +be merged, but some may not be abled to be merged after being checked | |
| 194 | +several times, which are unprofitable memory consumed. | |
| 195 | + | |
| 196 | +1) How to determine whether KSM save memory or consume memory in system-wide | |
| 197 | + range? Here is a simple approximate calculation for reference:: | |
| 198 | + | |
| 199 | + general_profit =~ pages_sharing * sizeof(page) - (all_rmap_items) * | |
| 200 | + sizeof(rmap_item); | |
| 201 | + | |
| 202 | + where all_rmap_items can be easily obtained by summing ``pages_sharing``, | |
| 203 | + ``pages_shared``, ``pages_unshared`` and ``pages_volatile``. | |
| 204 | + | |
| 205 | +2) The KSM profit inner a single process can be similarly obtained by the | |
| 206 | + following approximate calculation:: | |
| 207 | + | |
| 208 | + process_profit =~ ksm_merging_pages * sizeof(page) - | |
| 209 | + ksm_rmap_items * sizeof(rmap_item). | |
| 210 | + | |
| 211 | + where ksm_merging_pages is shown under the directory ``/proc/<pid>/``, | |
| 212 | + and ksm_rmap_items is shown in ``/proc/<pid>/ksm_stat``. | |
| 213 | + | |
| 214 | +From the perspective of application, a high ratio of ``ksm_rmap_items`` to | |
| 215 | +``ksm_merging_pages`` means a bad madvise-applied policy, so developers or | |
| 216 | +administrators have to rethink how to change madvise policy. Giving an example | |
| 217 | +for reference, a page's size is usually 4K, and the rmap_item's size is | |
| 218 | +separately 32B on 32-bit CPU architecture and 64B on 64-bit CPU architecture. | |
| 219 | +so if the ``ksm_rmap_items/ksm_merging_pages`` ratio exceeds 64 on 64-bit CPU | |
| 220 | +or exceeds 128 on 32-bit CPU, then the app's madvise policy should be dropped, | |
| 221 | +because the ksm profit is approximately zero or negative. | |
| 222 | + | |
| 187 | 223 | Monitoring KSM events |
| 188 | 224 | ===================== |
| 189 | 225 | |
| @@ -0,0 +1,162 @@ | ||
| 1 | +.. SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +============= | |
| 4 | +Multi-Gen LRU | |
| 5 | +============= | |
| 6 | +The multi-gen LRU is an alternative LRU implementation that optimizes | |
| 7 | +page reclaim and improves performance under memory pressure. Page | |
| 8 | +reclaim decides the kernel's caching policy and ability to overcommit | |
| 9 | +memory. It directly impacts the kswapd CPU usage and RAM efficiency. | |
| 10 | + | |
| 11 | +Quick start | |
| 12 | +=========== | |
| 13 | +Build the kernel with the following configurations. | |
| 14 | + | |
| 15 | +* ``CONFIG_LRU_GEN=y`` | |
| 16 | +* ``CONFIG_LRU_GEN_ENABLED=y`` | |
| 17 | + | |
| 18 | +All set! | |
| 19 | + | |
| 20 | +Runtime options | |
| 21 | +=============== | |
| 22 | +``/sys/kernel/mm/lru_gen/`` contains stable ABIs described in the | |
| 23 | +following subsections. | |
| 24 | + | |
| 25 | +Kill switch | |
| 26 | +----------- | |
| 27 | +``enabled`` accepts different values to enable or disable the | |
| 28 | +following components. Its default value depends on | |
| 29 | +``CONFIG_LRU_GEN_ENABLED``. All the components should be enabled | |
| 30 | +unless some of them have unforeseen side effects. Writing to | |
| 31 | +``enabled`` has no effect when a component is not supported by the | |
| 32 | +hardware, and valid values will be accepted even when the main switch | |
| 33 | +is off. | |
| 34 | + | |
| 35 | +====== =============================================================== | |
| 36 | +Values Components | |
| 37 | +====== =============================================================== | |
| 38 | +0x0001 The main switch for the multi-gen LRU. | |
| 39 | +0x0002 Clearing the accessed bit in leaf page table entries in large | |
| 40 | + batches, when MMU sets it (e.g., on x86). This behavior can | |
| 41 | + theoretically worsen lock contention (mmap_lock). If it is | |
| 42 | + disabled, the multi-gen LRU will suffer a minor performance | |
| 43 | + degradation for workloads that contiguously map hot pages, | |
| 44 | + whose accessed bits can be otherwise cleared by fewer larger | |
| 45 | + batches. | |
| 46 | +0x0004 Clearing the accessed bit in non-leaf page table entries as | |
| 47 | + well, when MMU sets it (e.g., on x86). This behavior was not | |
| 48 | + verified on x86 varieties other than Intel and AMD. If it is | |
| 49 | + disabled, the multi-gen LRU will suffer a negligible | |
| 50 | + performance degradation. | |
| 51 | +[yYnN] Apply to all the components above. | |
| 52 | +====== =============================================================== | |
| 53 | + | |
| 54 | +E.g., | |
| 55 | +:: | |
| 56 | + | |
| 57 | + echo y >/sys/kernel/mm/lru_gen/enabled | |
| 58 | + cat /sys/kernel/mm/lru_gen/enabled | |
| 59 | + 0x0007 | |
| 60 | + echo 5 >/sys/kernel/mm/lru_gen/enabled | |
| 61 | + cat /sys/kernel/mm/lru_gen/enabled | |
| 62 | + 0x0005 | |
| 63 | + | |
| 64 | +Thrashing prevention | |
| 65 | +-------------------- | |
| 66 | +Personal computers are more sensitive to thrashing because it can | |
| 67 | +cause janks (lags when rendering UI) and negatively impact user | |
| 68 | +experience. The multi-gen LRU offers thrashing prevention to the | |
| 69 | +majority of laptop and desktop users who do not have ``oomd``. | |
| 70 | + | |
| 71 | +Users can write ``N`` to ``min_ttl_ms`` to prevent the working set of | |
| 72 | +``N`` milliseconds from getting evicted. The OOM killer is triggered | |
| 73 | +if this working set cannot be kept in memory. In other words, this | |
| 74 | +option works as an adjustable pressure relief valve, and when open, it | |
| 75 | +terminates applications that are hopefully not being used. | |
| 76 | + | |
| 77 | +Based on the average human detectable lag (~100ms), ``N=1000`` usually | |
| 78 | +eliminates intolerable janks due to thrashing. Larger values like | |
| 79 | +``N=3000`` make janks less noticeable at the risk of premature OOM | |
| 80 | +kills. | |
| 81 | + | |
| 82 | +The default value ``0`` means disabled. | |
| 83 | + | |
| 84 | +Experimental features | |
| 85 | +===================== | |
| 86 | +``/sys/kernel/debug/lru_gen`` accepts commands described in the | |
| 87 | +following subsections. Multiple command lines are supported, so does | |
| 88 | +concatenation with delimiters ``,`` and ``;``. | |
| 89 | + | |
| 90 | +``/sys/kernel/debug/lru_gen_full`` provides additional stats for | |
| 91 | +debugging. ``CONFIG_LRU_GEN_STATS=y`` keeps historical stats from | |
| 92 | +evicted generations in this file. | |
| 93 | + | |
| 94 | +Working set estimation | |
| 95 | +---------------------- | |
| 96 | +Working set estimation measures how much memory an application needs | |
| 97 | +in a given time interval, and it is usually done with little impact on | |
| 98 | +the performance of the application. E.g., data centers want to | |
| 99 | +optimize job scheduling (bin packing) to improve memory utilizations. | |
| 100 | +When a new job comes in, the job scheduler needs to find out whether | |
| 101 | +each server it manages can allocate a certain amount of memory for | |
| 102 | +this new job before it can pick a candidate. To do so, the job | |
| 103 | +scheduler needs to estimate the working sets of the existing jobs. | |
| 104 | + | |
| 105 | +When it is read, ``lru_gen`` returns a histogram of numbers of pages | |
| 106 | +accessed over different time intervals for each memcg and node. | |
| 107 | +``MAX_NR_GENS`` decides the number of bins for each histogram. The | |
| 108 | +histograms are noncumulative. | |
| 109 | +:: | |
| 110 | + | |
| 111 | + memcg memcg_id memcg_path | |
| 112 | + node node_id | |
| 113 | + min_gen_nr age_in_ms nr_anon_pages nr_file_pages | |
| 114 | + ... | |
| 115 | + max_gen_nr age_in_ms nr_anon_pages nr_file_pages | |
| 116 | + | |
| 117 | +Each bin contains an estimated number of pages that have been accessed | |
| 118 | +within ``age_in_ms``. E.g., ``min_gen_nr`` contains the coldest pages | |
| 119 | +and ``max_gen_nr`` contains the hottest pages, since ``age_in_ms`` of | |
| 120 | +the former is the largest and that of the latter is the smallest. | |
| 121 | + | |
| 122 | +Users can write the following command to ``lru_gen`` to create a new | |
| 123 | +generation ``max_gen_nr+1``: | |
| 124 | + | |
| 125 | + ``+ memcg_id node_id max_gen_nr [can_swap [force_scan]]`` | |
| 126 | + | |
| 127 | +``can_swap`` defaults to the swap setting and, if it is set to ``1``, | |
| 128 | +it forces the scan of anon pages when swap is off, and vice versa. | |
| 129 | +``force_scan`` defaults to ``1`` and, if it is set to ``0``, it | |
| 130 | +employs heuristics to reduce the overhead, which is likely to reduce | |
| 131 | +the coverage as well. | |
| 132 | + | |
| 133 | +A typical use case is that a job scheduler runs this command at a | |
| 134 | +certain time interval to create new generations, and it ranks the | |
| 135 | +servers it manages based on the sizes of their cold pages defined by | |
| 136 | +this time interval. | |
| 137 | + | |
| 138 | +Proactive reclaim | |
| 139 | +----------------- | |
| 140 | +Proactive reclaim induces page reclaim when there is no memory | |
| 141 | +pressure. It usually targets cold pages only. E.g., when a new job | |
| 142 | +comes in, the job scheduler wants to proactively reclaim cold pages on | |
| 143 | +the server it selected, to improve the chance of successfully landing | |
| 144 | +this new job. | |
| 145 | + | |
| 146 | +Users can write the following command to ``lru_gen`` to evict | |
| 147 | +generations less than or equal to ``min_gen_nr``. | |
| 148 | + | |
| 149 | + ``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]`` | |
| 150 | + | |
| 151 | +``min_gen_nr`` should be less than ``max_gen_nr-1``, since | |
| 152 | +``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to | |
| 153 | +the active list) and therefore cannot be evicted. ``swappiness`` | |
| 154 | +overrides the default value in ``/proc/sys/vm/swappiness``. | |
| 155 | +``nr_to_reclaim`` limits the number of pages to evict. | |
| 156 | + | |
| 157 | +A typical use case is that a job scheduler runs this command before it | |
| 158 | +tries to land a new job on a server. If it fails to materialize enough | |
| 159 | +cold pages because of the overestimation, it retries on the next | |
| 160 | +server according to the ranking result obtained from the working set | |
| 161 | +estimation step. This less forceful approach limits the impacts on the | |
| 162 | +existing jobs. | |
| @@ -191,7 +191,14 @@ allocation failure to throttle the next allocation attempt:: · 190 unmodified lines | ||
| 191 | 191 | |
| 192 | 192 | /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs |
| 193 | 193 | |
| 194 | −The khugepaged progress can be seen in the number of pages collapsed:: | |
| 194 | +The khugepaged progress can be seen in the number of pages collapsed (note | |
| 195 | +that this counter may not be an exact count of the number of pages | |
| 196 | +collapsed, since "collapsed" could mean multiple things: (1) A PTE mapping | |
| 197 | +being replaced by a PMD mapping, or (2) All 4K physical pages replaced by | |
| 198 | +one 2M hugepage. Each may happen independently, or together, depending on | |
| 199 | +the type of memory and the failures that occur. As such, this value should | |
| 200 | +be interpreted roughly as a sign of progress, and counters in /proc/vmstat | |
| 201 | +consulted for more accurate accounting):: | |
| 195 | 202 | |
| 196 | 203 | /sys/kernel/mm/transparent_hugepage/khugepaged/pages_collapsed |
| 197 | 204 | |
| @@ -366,10 +373,9 @@ thp_split_pmd · 168 unmodified lines | ||
| 366 | 373 | page table entry. |
| 367 | 374 | |
| 368 | 375 | thp_zero_page_alloc |
| 369 | − is incremented every time a huge zero page is | |
| 370 | − successfully allocated. It includes allocations which where | |
| 371 | − dropped due race with other allocation. Note, it doesn't count | |
| 372 | − every map of the huge zero page, only its allocation. | |
| 376 | + is incremented every time a huge zero page used for thp is | |
| 377 | + successfully allocated. Note, it doesn't count every map of | |
| 378 | + the huge zero page, only its allocation. | |
| 373 | 379 | |
| 374 | 380 | thp_zero_page_alloc_failed |
| 375 | 381 | is incremented if kernel fails to allocate |
| @@ -17,7 +17,10 @@ of the ``PROT_NONE+SIGSEGV`` trick. · 16 unmodified lines | ||
| 17 | 17 | Design |
| 18 | 18 | ====== |
| 19 | 19 | |
| 20 | −Userfaults are delivered and resolved through the ``userfaultfd`` syscall. | |
| 20 | +Userspace creates a new userfaultfd, initializes it, and registers one or more | |
| 21 | +regions of virtual memory with it. Then, any page faults which occur within the | |
| 22 | +region(s) result in a message being delivered to the userfaultfd, notifying | |
| 23 | +userspace of the fault. | |
| 21 | 24 | |
| 22 | 25 | The ``userfaultfd`` (aside from registering and unregistering virtual |
| 23 | 26 | memory ranges) provides two primary functionalities: |
| @@ -34,12 +37,11 @@ The real advantage of userfaults if compared to regular virtual memory · 10 unmodified lines | ||
| 34 | 37 | management of mremap/mprotect is that the userfaults in all their |
| 35 | 38 | operations never involve heavyweight structures like vmas (in fact the |
| 36 | 39 | ``userfaultfd`` runtime load never takes the mmap_lock for writing). |
| 37 | − | |
| 38 | 40 | Vmas are not suitable for page- (or hugepage) granular fault tracking |
| 39 | 41 | when dealing with virtual address spaces that could span |
| 40 | 42 | Terabytes. Too many vmas would be needed for that. |
| 41 | 43 | |
| 42 | −The ``userfaultfd`` once opened by invoking the syscall, can also be | |
| 44 | +The ``userfaultfd``, once created, can also be | |
| 43 | 45 | passed using unix domain sockets to a manager process, so the same |
| 44 | 46 | manager process could handle the userfaults of a multitude of |
| 45 | 47 | different processes without them being aware about what is going on |
| @@ -50,6 +52,39 @@ is a corner case that would currently return ``-EBUSY``). · 4 unmodified lines | ||
| 50 | 52 | API |
| 51 | 53 | === |
| 52 | 54 | |
| 55 | +Creating a userfaultfd | |
| 56 | +---------------------- | |
| 57 | + | |
| 58 | +There are two ways to create a new userfaultfd, each of which provide ways to | |
| 59 | +restrict access to this functionality (since historically userfaultfds which | |
| 60 | +handle kernel page faults have been a useful tool for exploiting the kernel). | |
| 61 | + | |
| 62 | +The first way, supported since userfaultfd was introduced, is the | |
| 63 | +userfaultfd(2) syscall. Access to this is controlled in several ways: | |
| 64 | + | |
| 65 | +- Any user can always create a userfaultfd which traps userspace page faults | |
| 66 | + only. Such a userfaultfd can be created using the userfaultfd(2) syscall | |
| 67 | + with the flag UFFD_USER_MODE_ONLY. | |
| 68 | + | |
| 69 | +- In order to also trap kernel page faults for the address space, either the | |
| 70 | + process needs the CAP_SYS_PTRACE capability, or the system must have | |
| 71 | + vm.unprivileged_userfaultfd set to 1. By default, vm.unprivileged_userfaultfd | |
| 72 | + is set to 0. | |
| 73 | + | |
| 74 | +The second way, added to the kernel more recently, is by opening | |
| 75 | +/dev/userfaultfd and issuing a USERFAULTFD_IOC_NEW ioctl to it. This method | |
| 76 | +yields equivalent userfaultfds to the userfaultfd(2) syscall. | |
| 77 | + | |
| 78 | +Unlike userfaultfd(2), access to /dev/userfaultfd is controlled via normal | |
| 79 | +filesystem permissions (user/group/mode), which gives fine grained access to | |
| 80 | +userfaultfd specifically, without also granting other unrelated privileges at | |
| 81 | +the same time (as e.g. granting CAP_SYS_PTRACE would do). Users who have access | |
| 82 | +to /dev/userfaultfd can always create userfaultfds that trap kernel page faults; | |
| 83 | +vm.unprivileged_userfaultfd is not considered. | |
| 84 | + | |
| 85 | +Initializing a userfaultfd | |
| 86 | +-------------------------- | |
| 87 | + | |
| 53 | 88 | When first opened the ``userfaultfd`` must be enabled invoking the |
| 54 | 89 | ``UFFDIO_API`` ioctl specifying a ``uffdio_api.api`` value set to ``UFFD_API`` (or |
| 55 | 90 | a later API version) which will specify the ``read/POLLIN`` protocol |
| @@ -635,6 +635,17 @@ different types of memory (represented as different NUMA nodes) to · 634 unmodified lines | ||
| 635 | 635 | place the hot pages in the fast memory. This is implemented based on |
| 636 | 636 | unmapping and page fault too. |
| 637 | 637 | |
| 638 | +numa_balancing_promote_rate_limit_MBps | |
| 639 | +====================================== | |
| 640 | + | |
| 641 | +Too high promotion/demotion throughput between different memory types | |
| 642 | +may hurt application latency. This can be used to rate limit the | |
| 643 | +promotion throughput. The per-node max promotion throughput in MB/s | |
| 644 | +will be limited to be no more than the set value. | |
| 645 | + | |
| 646 | +A rule of thumb is to set this to less than 1/10 of the PMEM node | |
| 647 | +write bandwidth. | |
| 648 | + | |
| 638 | 649 | oops_all_cpu_backtrace |
| 639 | 650 | ====================== |
| 640 | 651 | |
| @@ -926,6 +926,9 @@ calls without any restrictions. · 925 unmodified lines | ||
| 926 | 926 | |
| 927 | 927 | The default value is 0. |
| 928 | 928 | |
| 929 | +Another way to control permissions for userfaultfd is to use | |
| 930 | +/dev/userfaultfd instead of userfaultfd(2). See | |
| 931 | +Documentation/admin-guide/mm/userfaultfd.rst. | |
| 929 | 932 | |
| 930 | 933 | user_reserve_kbytes |
| 931 | 934 | =================== |
| @@ -37,6 +37,7 @@ Library functionality that is used throughout the kernel. · 36 unmodified lines | ||
| 37 | 37 | kref |
| 38 | 38 | assoc_array |
| 39 | 39 | xarray |
| 40 | + maple_tree | |
| 40 | 41 | idr |
| 41 | 42 | circular-buffers |
| 42 | 43 | rbtree |
| @@ -0,0 +1,217 @@ | ||
| 1 | +.. SPDX-License-Identifier: GPL-2.0+ | |
| 2 | + | |
| 3 | + | |
| 4 | +========== | |
| 5 | +Maple Tree | |
| 6 | +========== | |
| 7 | + | |
| 8 | +:Author: Liam R. Howlett | |
| 9 | + | |
| 10 | +Overview | |
| 11 | +======== | |
| 12 | + | |
| 13 | +The Maple Tree is a B-Tree data type which is optimized for storing | |
| 14 | +non-overlapping ranges, including ranges of size 1. The tree was designed to | |
| 15 | +be simple to use and does not require a user written search method. It | |
| 16 | +supports iterating over a range of entries and going to the previous or next | |
| 17 | +entry in a cache-efficient manner. The tree can also be put into an RCU-safe | |
| 18 | +mode of operation which allows reading and writing concurrently. Writers must | |
| 19 | +synchronize on a lock, which can be the default spinlock, or the user can set | |
| 20 | +the lock to an external lock of a different type. | |
| 21 | + | |
| 22 | +The Maple Tree maintains a small memory footprint and was designed to use | |
| 23 | +modern processor cache efficiently. The majority of the users will be able to | |
| 24 | +use the normal API. An :ref:`maple-tree-advanced-api` exists for more complex | |
| 25 | +scenarios. The most important usage of the Maple Tree is the tracking of the | |
| 26 | +virtual memory areas. | |
| 27 | + | |
| 28 | +The Maple Tree can store values between ``0`` and ``ULONG_MAX``. The Maple | |
| 29 | +Tree reserves values with the bottom two bits set to '10' which are below 4096 | |
| 30 | +(ie 2, 6, 10 .. 4094) for internal use. If the entries may use reserved | |
| 31 | +entries then the users can convert the entries using xa_mk_value() and convert | |
| 32 | +them back by calling xa_to_value(). If the user needs to use a reserved | |
| 33 | +value, then the user can convert the value when using the | |
| 34 | +:ref:`maple-tree-advanced-api`, but are blocked by the normal API. | |
| 35 | + | |
| 36 | +The Maple Tree can also be configured to support searching for a gap of a given | |
| 37 | +size (or larger). | |
| 38 | + | |
| 39 | +Pre-allocating of nodes is also supported using the | |
| 40 | +:ref:`maple-tree-advanced-api`. This is useful for users who must guarantee a | |
| 41 | +successful store operation within a given | |
| 42 | +code segment when allocating cannot be done. Allocations of nodes are | |
| 43 | +relatively small at around 256 bytes. | |
| 44 | + | |
| 45 | +.. _maple-tree-normal-api: | |
| 46 | + | |
| 47 | +Normal API | |
| 48 | +========== | |
| 49 | + | |
| 50 | +Start by initialising a maple tree, either with DEFINE_MTREE() for statically | |
| 51 | +allocated maple trees or mt_init() for dynamically allocated ones. A | |
| 52 | +freshly-initialised maple tree contains a ``NULL`` pointer for the range ``0`` | |
| 53 | +- ``ULONG_MAX``. There are currently two types of maple trees supported: the | |
| 54 | +allocation tree and the regular tree. The regular tree has a higher branching | |
| 55 | +factor for internal nodes. The allocation tree has a lower branching factor | |
| 56 | +but allows the user to search for a gap of a given size or larger from either | |
| 57 | +``0`` upwards or ``ULONG_MAX`` down. An allocation tree can be used by | |
| 58 | +passing in the ``MT_FLAGS_ALLOC_RANGE`` flag when initialising the tree. | |
| 59 | + | |
| 60 | +You can then set entries using mtree_store() or mtree_store_range(). | |
| 61 | +mtree_store() will overwrite any entry with the new entry and return 0 on | |
| 62 | +success or an error code otherwise. mtree_store_range() works in the same way | |
| 63 | +but takes a range. mtree_load() is used to retrieve the entry stored at a | |
| 64 | +given index. You can use mtree_erase() to erase an entire range by only | |
| 65 | +knowing one value within that range, or mtree_store() call with an entry of | |
| 66 | +NULL may be used to partially erase a range or many ranges at once. | |
| 67 | + | |
| 68 | +If you want to only store a new entry to a range (or index) if that range is | |
| 69 | +currently ``NULL``, you can use mtree_insert_range() or mtree_insert() which | |
| 70 | +return -EEXIST if the range is not empty. | |
| 71 | + | |
| 72 | +You can search for an entry from an index upwards by using mt_find(). | |
| 73 | + | |
| 74 | +You can walk each entry within a range by calling mt_for_each(). You must | |
| 75 | +provide a temporary variable to store a cursor. If you want to walk each | |
| 76 | +element of the tree then ``0`` and ``ULONG_MAX`` may be used as the range. If | |
| 77 | +the caller is going to hold the lock for the duration of the walk then it is | |
| 78 | +worth looking at the mas_for_each() API in the :ref:`maple-tree-advanced-api` | |
| 79 | +section. | |
| 80 | + | |
| 81 | +Sometimes it is necessary to ensure the next call to store to a maple tree does | |
| 82 | +not allocate memory, please see :ref:`maple-tree-advanced-api` for this use case. | |
| 83 | + | |
| 84 | +Finally, you can remove all entries from a maple tree by calling | |
| 85 | +mtree_destroy(). If the maple tree entries are pointers, you may wish to free | |
| 86 | +the entries first. | |
| 87 | + | |
| 88 | +Allocating Nodes | |
| 89 | +---------------- | |
| 90 | + | |
| 91 | +The allocations are handled by the internal tree code. See | |
| 92 | +:ref:`maple-tree-advanced-alloc` for other options. | |
| 93 | + | |
| 94 | +Locking | |
| 95 | +------- | |
| 96 | + | |
| 97 | +You do not have to worry about locking. See :ref:`maple-tree-advanced-locks` | |
| 98 | +for other options. | |
| 99 | + | |
| 100 | +The Maple Tree uses RCU and an internal spinlock to synchronise access: | |
| 101 | + | |
| 102 | +Takes RCU read lock: | |
| 103 | + * mtree_load() | |
| 104 | + * mt_find() | |
| 105 | + * mt_for_each() | |
| 106 | + * mt_next() | |
| 107 | + * mt_prev() | |
| 108 | + | |
| 109 | +Takes ma_lock internally: | |
| 110 | + * mtree_store() | |
| 111 | + * mtree_store_range() | |
| 112 | + * mtree_insert() | |
| 113 | + * mtree_insert_range() | |
| 114 | + * mtree_erase() | |
| 115 | + * mtree_destroy() | |
| 116 | + * mt_set_in_rcu() | |
| 117 | + * mt_clear_in_rcu() | |
| 118 | + | |
| 119 | +If you want to take advantage of the internal lock to protect the data | |
| 120 | +structures that you are storing in the Maple Tree, you can call mtree_lock() | |
| 121 | +before calling mtree_load(), then take a reference count on the object you | |
| 122 | +have found before calling mtree_unlock(). This will prevent stores from | |
| 123 | +removing the object from the tree between looking up the object and | |
| 124 | +incrementing the refcount. You can also use RCU to avoid dereferencing | |
| 125 | +freed memory, but an explanation of that is beyond the scope of this | |
| 126 | +document. | |
| 127 | + | |
| 128 | +.. _maple-tree-advanced-api: | |
| 129 | + | |
| 130 | +Advanced API | |
| 131 | +============ | |
| 132 | + | |
| 133 | +The advanced API offers more flexibility and better performance at the | |
| 134 | +cost of an interface which can be harder to use and has fewer safeguards. | |
| 135 | +You must take care of your own locking while using the advanced API. | |
| 136 | +You can use the ma_lock, RCU or an external lock for protection. | |
| 137 | +You can mix advanced and normal operations on the same array, as long | |
| 138 | +as the locking is compatible. The :ref:`maple-tree-normal-api` is implemented | |
| 139 | +in terms of the advanced API. | |
| 140 | + | |
| 141 | +The advanced API is based around the ma_state, this is where the 'mas' | |
| 142 | +prefix originates. The ma_state struct keeps track of tree operations to make | |
| 143 | +life easier for both internal and external tree users. | |
| 144 | + | |
| 145 | +Initialising the maple tree is the same as in the :ref:`maple-tree-normal-api`. | |
| 146 | +Please see above. | |
| 147 | + | |
| 148 | +The maple state keeps track of the range start and end in mas->index and | |
| 149 | +mas->last, respectively. | |
| 150 | + | |
| 151 | +mas_walk() will walk the tree to the location of mas->index and set the | |
| 152 | +mas->index and mas->last according to the range for the entry. | |
| 153 | + | |
| 154 | +You can set entries using mas_store(). mas_store() will overwrite any entry | |
| 155 | +with the new entry and return the first existing entry that is overwritten. | |
| 156 | +The range is passed in as members of the maple state: index and last. | |
| 157 | + | |
| 158 | +You can use mas_erase() to erase an entire range by setting index and | |
| 159 | +last of the maple state to the desired range to erase. This will erase | |
| 160 | +the first range that is found in that range, set the maple state index | |
| 161 | +and last as the range that was erased and return the entry that existed | |
| 162 | +at that location. | |
| 163 | + | |
| 164 | +You can walk each entry within a range by using mas_for_each(). If you want | |
| 165 | +to walk each element of the tree then ``0`` and ``ULONG_MAX`` may be used as | |
| 166 | +the range. If the lock needs to be periodically dropped, see the locking | |
| 167 | +section mas_pause(). | |
| 168 | + | |
| 169 | +Using a maple state allows mas_next() and mas_prev() to function as if the | |
| 170 | +tree was a linked list. With such a high branching factor the amortized | |
| 171 | +performance penalty is outweighed by cache optimization. mas_next() will | |
| 172 | +return the next entry which occurs after the entry at index. mas_prev() | |
| 173 | +will return the previous entry which occurs before the entry at index. | |
| 174 | + | |
| 175 | +mas_find() will find the first entry which exists at or above index on | |
| 176 | +the first call, and the next entry from every subsequent calls. | |
| 177 | + | |
| 178 | +mas_find_rev() will find the fist entry which exists at or below the last on | |
| 179 | +the first call, and the previous entry from every subsequent calls. | |
| 180 | + | |
| 181 | +If the user needs to yield the lock during an operation, then the maple state | |
| 182 | +must be paused using mas_pause(). | |
| 183 | + | |
| 184 | +There are a few extra interfaces provided when using an allocation tree. | |
| 185 | +If you wish to search for a gap within a range, then mas_empty_area() | |
| 186 | +or mas_empty_area_rev() can be used. mas_empty_area() searches for a gap | |
| 187 | +starting at the lowest index given up to the maximum of the range. | |
| 188 | +mas_empty_area_rev() searches for a gap starting at the highest index given | |
| 189 | +and continues downward to the lower bound of the range. | |
| 190 | + | |
| 191 | +.. _maple-tree-advanced-alloc: | |
| 192 | + | |
| 193 | +Advanced Allocating Nodes | |
| 194 | +------------------------- | |
| 195 | + | |
| 196 | +Allocations are usually handled internally to the tree, however if allocations | |
| 197 | +need to occur before a write occurs then calling mas_expected_entries() will | |
| 198 | +allocate the worst-case number of needed nodes to insert the provided number of | |
| 199 | +ranges. This also causes the tree to enter mass insertion mode. Once | |
| 200 | +insertions are complete calling mas_destroy() on the maple state will free the | |
| 201 | +unused allocations. | |
| 202 | + | |
| 203 | +.. _maple-tree-advanced-locks: | |
| 204 | + | |
| 205 | +Advanced Locking | |
| 206 | +---------------- | |
| 207 | + | |
| 208 | +The maple tree uses a spinlock by default, but external locks can be used for | |
| 209 | +tree updates as well. To use an external lock, the tree must be initialized | |
| 210 | +with the ``MT_FLAGS_LOCK_EXTERN flag``, this is usually done with the | |
| 211 | +MTREE_INIT_EXT() #define, which takes an external lock as an argument. | |
| 212 | + | |
| 213 | +Functions and structures | |
| 214 | +======================== | |
| 215 | + | |
| 216 | +.. kernel-doc:: include/linux/maple_tree.h | |
| 217 | +.. kernel-doc:: lib/maple_tree.c | |
| @@ -19,9 +19,6 @@ User Space Memory Access · 18 unmodified lines | ||
| 19 | 19 | Memory Allocation Controls |
| 20 | 20 | ========================== |
| 21 | 21 | |
| 22 | −.. kernel-doc:: include/linux/gfp.h | |
| 23 | − :internal: | |
| 24 | − | |
| 25 | 22 | .. kernel-doc:: include/linux/gfp_types.h |
| 26 | 23 | :doc: Page mobility and placement hints |
| 27 | 24 | |
| @@ -24,6 +24,7 @@ Documentation/dev-tools/testing-overview.rst · 23 unmodified lines | ||
| 24 | 24 | kcov |
| 25 | 25 | gcov |
| 26 | 26 | kasan |
| 27 | + kmsan | |
| 27 | 28 | ubsan |
| 28 | 29 | kmemleak |
| 29 | 30 | kcsan |
| @@ -111,9 +111,17 @@ parameter can be used to control panic and reporting behaviour: · 110 unmodified lines | ||
| 111 | 111 | report or also panic the kernel (default: ``report``). The panic happens even |
| 112 | 112 | if ``kasan_multi_shot`` is enabled. |
| 113 | 113 | |
| 114 | −Hardware Tag-Based KASAN mode (see the section about various modes below) is | |
| 115 | −intended for use in production as a security mitigation. Therefore, it supports | |
| 116 | −additional boot parameters that allow disabling KASAN or controlling features: | |
| 114 | +Software and Hardware Tag-Based KASAN modes (see the section about various | |
| 115 | +modes below) support altering stack trace collection behavior: | |
| 116 | + | |
| 117 | +- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack | |
| 118 | + traces collection (default: ``on``). | |
| 119 | +- ``kasan.stack_ring_size=<number of entries>`` specifies the number of entries | |
| 120 | + in the stack ring (default: ``32768``). | |
| 121 | + | |
| 122 | +Hardware Tag-Based KASAN mode is intended for use in production as a security | |
| 123 | +mitigation. Therefore, it supports additional boot parameters that allow | |
| 124 | +disabling KASAN altogether or controlling its features: | |
| 117 | 125 | |
| 118 | 126 | - ``kasan=off`` or ``=on`` controls whether KASAN is enabled (default: ``on``). |
| 119 | 127 | |
| @@ -132,9 +140,6 @@ additional boot parameters that allow disabling KASAN or controlling features: · 12 unmodified lines | ||
| 132 | 140 | - ``kasan.vmalloc=off`` or ``=on`` disables or enables tagging of vmalloc |
| 133 | 141 | allocations (default: ``on``). |
| 134 | 142 | |
| 135 | −- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack | |
| 136 | − traces collection (default: ``on``). | |
| 137 | − | |
| 138 | 143 | Error reports |
| 139 | 144 | ~~~~~~~~~~~~~ |
| 140 | 145 | |
| @@ -0,0 +1,427 @@ | ||
| 1 | +.. SPDX-License-Identifier: GPL-2.0 | |
| 2 | +.. Copyright (C) 2022, Google LLC. | |
| 3 | + | |
| 4 | +=================================== | |
| 5 | +The Kernel Memory Sanitizer (KMSAN) | |
| 6 | +=================================== | |
| 7 | + | |
| 8 | +KMSAN is a dynamic error detector aimed at finding uses of uninitialized | |
| 9 | +values. It is based on compiler instrumentation, and is quite similar to the | |
| 10 | +userspace `MemorySanitizer tool`_. | |
| 11 | + | |
| 12 | +An important note is that KMSAN is not intended for production use, because it | |
| 13 | +drastically increases kernel memory footprint and slows the whole system down. | |
| 14 | + | |
| 15 | +Usage | |
| 16 | +===== | |
| 17 | + | |
| 18 | +Building the kernel | |
| 19 | +------------------- | |
| 20 | + | |
| 21 | +In order to build a kernel with KMSAN you will need a fresh Clang (14.0.6+). | |
| 22 | +Please refer to `LLVM documentation`_ for the instructions on how to build Clang. | |
| 23 | + | |
| 24 | +Now configure and build the kernel with CONFIG_KMSAN enabled. | |
| 25 | + | |
| 26 | +Example report | |
| 27 | +-------------- | |
| 28 | + | |
| 29 | +Here is an example of a KMSAN report:: | |
| 30 | + | |
| 31 | + ===================================================== | |
| 32 | + BUG: KMSAN: uninit-value in test_uninit_kmsan_check_memory+0x1be/0x380 [kmsan_test] | |
| 33 | + test_uninit_kmsan_check_memory+0x1be/0x380 mm/kmsan/kmsan_test.c:273 | |
| 34 | + kunit_run_case_internal lib/kunit/test.c:333 | |
| 35 | + kunit_try_run_case+0x206/0x420 lib/kunit/test.c:374 | |
| 36 | + kunit_generic_run_threadfn_adapter+0x6d/0xc0 lib/kunit/try-catch.c:28 | |
| 37 | + kthread+0x721/0x850 kernel/kthread.c:327 | |
| 38 | + ret_from_fork+0x1f/0x30 ??:? | |
| 39 | + | |
| 40 | + Uninit was stored to memory at: | |
| 41 | + do_uninit_local_array+0xfa/0x110 mm/kmsan/kmsan_test.c:260 | |
| 42 | + test_uninit_kmsan_check_memory+0x1a2/0x380 mm/kmsan/kmsan_test.c:271 | |
| 43 | + kunit_run_case_internal lib/kunit/test.c:333 | |
| 44 | + kunit_try_run_case+0x206/0x420 lib/kunit/test.c:374 | |
| 45 | + kunit_generic_run_threadfn_adapter+0x6d/0xc0 lib/kunit/try-catch.c:28 | |
| 46 | + kthread+0x721/0x850 kernel/kthread.c:327 | |
| 47 | + ret_from_fork+0x1f/0x30 ??:? | |
| 48 | + | |
| 49 | + Local variable uninit created at: | |
| 50 | + do_uninit_local_array+0x4a/0x110 mm/kmsan/kmsan_test.c:256 | |
| 51 | + test_uninit_kmsan_check_memory+0x1a2/0x380 mm/kmsan/kmsan_test.c:271 | |
| 52 | + | |
| 53 | + Bytes 4-7 of 8 are uninitialized | |
| 54 | + Memory access of size 8 starts at ffff888083fe3da0 | |
| 55 | + | |
| 56 | + CPU: 0 PID: 6731 Comm: kunit_try_catch Tainted: G B E 5.16.0-rc3+ #104 | |
| 57 | + Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014 | |
| 58 | + ===================================================== | |
| 59 | + | |
| 60 | +The report says that the local variable ``uninit`` was created uninitialized in | |
| 61 | +``do_uninit_local_array()``. The third stack trace corresponds to the place | |
| 62 | +where this variable was created. | |
| 63 | + | |
| 64 | +The first stack trace shows where the uninit value was used (in | |
| 65 | +``test_uninit_kmsan_check_memory()``). The tool shows the bytes which were left | |
| 66 | +uninitialized in the local variable, as well as the stack where the value was | |
| 67 | +copied to another memory location before use. | |
| 68 | + | |
| 69 | +A use of uninitialized value ``v`` is reported by KMSAN in the following cases: | |
| 70 | + - in a condition, e.g. ``if (v) { ... }``; | |
| 71 | + - in an indexing or pointer dereferencing, e.g. ``array[v]`` or ``*v``; | |
| 72 | + - when it is copied to userspace or hardware, e.g. ``copy_to_user(..., &v, ...)``; | |
| 73 | + - when it is passed as an argument to a function, and | |
| 74 | + ``CONFIG_KMSAN_CHECK_PARAM_RETVAL`` is enabled (see below). | |
| 75 | + | |
| 76 | +The mentioned cases (apart from copying data to userspace or hardware, which is | |
| 77 | +a security issue) are considered undefined behavior from the C11 Standard point | |
| 78 | +of view. | |
| 79 | + | |
| 80 | +Disabling the instrumentation | |
| 81 | +----------------------------- | |
| 82 | + | |
| 83 | +A function can be marked with ``__no_kmsan_checks``. Doing so makes KMSAN | |
| 84 | +ignore uninitialized values in that function and mark its output as initialized. | |
| 85 | +As a result, the user will not get KMSAN reports related to that function. | |
| 86 | + | |
| 87 | +Another function attribute supported by KMSAN is ``__no_sanitize_memory``. | |
| 88 | +Applying this attribute to a function will result in KMSAN not instrumenting | |
| 89 | +it, which can be helpful if we do not want the compiler to interfere with some | |
| 90 | +low-level code (e.g. that marked with ``noinstr`` which implicitly adds | |
| 91 | +``__no_sanitize_memory``). | |
| 92 | + | |
| 93 | +This however comes at a cost: stack allocations from such functions will have | |
| 94 | +incorrect shadow/origin values, likely leading to false positives. Functions | |
| 95 | +called from non-instrumented code may also receive incorrect metadata for their | |
| 96 | +parameters. | |
| 97 | + | |
| 98 | +As a rule of thumb, avoid using ``__no_sanitize_memory`` explicitly. | |
| 99 | + | |
| 100 | +It is also possible to disable KMSAN for a single file (e.g. main.o):: | |
| 101 | + | |
| 102 | + KMSAN_SANITIZE_main.o := n | |
| 103 | + | |
| 104 | +or for the whole directory:: | |
| 105 | + | |
| 106 | + KMSAN_SANITIZE := n | |
| 107 | + | |
| 108 | +in the Makefile. Think of this as applying ``__no_sanitize_memory`` to every | |
| 109 | +function in the file or directory. Most users won't need KMSAN_SANITIZE, unless | |
| 110 | +their code gets broken by KMSAN (e.g. runs at early boot time). | |
| 111 | + | |
| 112 | +Support | |
| 113 | +======= | |
| 114 | + | |
| 115 | +In order for KMSAN to work the kernel must be built with Clang, which so far is | |
| 116 | +the only compiler that has KMSAN support. The kernel instrumentation pass is | |
| 117 | +based on the userspace `MemorySanitizer tool`_. | |
| 118 | + | |
| 119 | +The runtime library only supports x86_64 at the moment. | |
| 120 | + | |
| 121 | +How KMSAN works | |
| 122 | +=============== | |
| 123 | + | |
| 124 | +KMSAN shadow memory | |
| 125 | +------------------- | |
| 126 | + | |
| 127 | +KMSAN associates a metadata byte (also called shadow byte) with every byte of | |
| 128 | +kernel memory. A bit in the shadow byte is set iff the corresponding bit of the | |
| 129 | +kernel memory byte is uninitialized. Marking the memory uninitialized (i.e. | |
| 130 | +setting its shadow bytes to ``0xff``) is called poisoning, marking it | |
| 131 | +initialized (setting the shadow bytes to ``0x00``) is called unpoisoning. | |
| 132 | + | |
| 133 | +When a new variable is allocated on the stack, it is poisoned by default by | |
| 134 | +instrumentation code inserted by the compiler (unless it is a stack variable | |
| 135 | +that is immediately initialized). Any new heap allocation done without | |
| 136 | +``__GFP_ZERO`` is also poisoned. | |
| 137 | + | |
| 138 | +Compiler instrumentation also tracks the shadow values as they are used along | |
| 139 | +the code. When needed, instrumentation code invokes the runtime library in | |
| 140 | +``mm/kmsan/`` to persist shadow values. | |
| 141 | + | |
| 142 | +The shadow value of a basic or compound type is an array of bytes of the same | |
| 143 | +length. When a constant value is written into memory, that memory is unpoisoned. | |
| 144 | +When a value is read from memory, its shadow memory is also obtained and | |
| 145 | +propagated into all the operations which use that value. For every instruction | |
| 146 | +that takes one or more values the compiler generates code that calculates the | |
| 147 | +shadow of the result depending on those values and their shadows. | |
| 148 | + | |
| 149 | +Example:: | |
| 150 | + | |
| 151 | + int a = 0xff; // i.e. 0x000000ff | |
| 152 | + int b; | |
| 153 | + int c = a | b; | |
| 154 | + | |
| 155 | +In this case the shadow of ``a`` is ``0``, shadow of ``b`` is ``0xffffffff``, | |
| 156 | +shadow of ``c`` is ``0xffffff00``. This means that the upper three bytes of | |
| 157 | +``c`` are uninitialized, while the lower byte is initialized. | |
| 158 | + | |
| 159 | +Origin tracking | |
| 160 | +--------------- | |
| 161 | + | |
| 162 | +Every four bytes of kernel memory also have a so-called origin mapped to them. | |
| 163 | +This origin describes the point in program execution at which the uninitialized | |
| 164 | +value was created. Every origin is associated with either the full allocation | |
| 165 | +stack (for heap-allocated memory), or the function containing the uninitialized | |
| 166 | +variable (for locals). | |
| 167 | + | |
| 168 | +When an uninitialized variable is allocated on stack or heap, a new origin | |
| 169 | +value is created, and that variable's origin is filled with that value. When a | |
| 170 | +value is read from memory, its origin is also read and kept together with the | |
| 171 | +shadow. For every instruction that takes one or more values, the origin of the | |
| 172 | +result is one of the origins corresponding to any of the uninitialized inputs. | |
| 173 | +If a poisoned value is written into memory, its origin is written to the | |
| 174 | +corresponding storage as well. | |
| 175 | + | |
| 176 | +Example 1:: | |
| 177 | + | |
| 178 | + int a = 42; | |
| 179 | + int b; | |
| 180 | + int c = a + b; | |
| 181 | + | |
| 182 | +In this case the origin of ``b`` is generated upon function entry, and is | |
| 183 | +stored to the origin of ``c`` right before the addition result is written into | |
| 184 | +memory. | |
| 185 | + | |
| 186 | +Several variables may share the same origin address, if they are stored in the | |
| 187 | +same four-byte chunk. In this case every write to either variable updates the | |
| 188 | +origin for all of them. We have to sacrifice precision in this case, because | |
| 189 | +storing origins for individual bits (and even bytes) would be too costly. | |
| 190 | + | |
| 191 | +Example 2:: | |
| 192 | + | |
| 193 | + int combine(short a, short b) { | |
| 194 | + union ret_t { | |
| 195 | + int i; | |
| 196 | + short s[2]; | |
| 197 | + } ret; | |
| 198 | + ret.s[0] = a; | |
| 199 | + ret.s[1] = b; | |
| 200 | + return ret.i; | |
| 201 | + } | |
| 202 | + | |
| 203 | +If ``a`` is initialized and ``b`` is not, the shadow of the result would be | |
| 204 | +0xffff0000, and the origin of the result would be the origin of ``b``. | |
| 205 | +``ret.s[0]`` would have the same origin, but it will never be used, because | |
| 206 | +that variable is initialized. | |
| 207 | + | |
| 208 | +If both function arguments are uninitialized, only the origin of the second | |
| 209 | +argument is preserved. | |
| 210 | + | |
| 211 | +Origin chaining | |
| 212 | +~~~~~~~~~~~~~~~ | |
| 213 | + | |
| 214 | +To ease debugging, KMSAN creates a new origin for every store of an | |
| 215 | +uninitialized value to memory. The new origin references both its creation stack | |
| 216 | +and the previous origin the value had. This may cause increased memory | |
| 217 | +consumption, so we limit the length of origin chains in the runtime. | |
| 218 | + | |
| 219 | +Clang instrumentation API | |
| 220 | +------------------------- | |
| 221 | + | |
| 222 | +Clang instrumentation pass inserts calls to functions defined in | |
| 223 | +``mm/kmsan/nstrumentation.c`` into the kernel code. | |
| 224 | + | |
| 225 | +Shadow manipulation | |
| 226 | +~~~~~~~~~~~~~~~~~~~ | |
| 227 | + | |
| 228 | +For every memory access the compiler emits a call to a function that returns a | |
| 229 | +pair of pointers to the shadow and origin addresses of the given memory:: | |
| 230 | + | |
| 231 | + typedef struct { | |
| 232 | + void *shadow, *origin; | |
| 233 | + } shadow_origin_ptr_t | |
| 234 | + | |
| 235 | + shadow_origin_ptr_t __msan_metadata_ptr_for_load_{1,2,4,8}(void *addr) | |
| 236 | + shadow_origin_ptr_t __msan_metadata_ptr_for_store_{1,2,4,8}(void *addr) | |
| 237 | + shadow_origin_ptr_t __msan_metadata_ptr_for_load_n(void *addr, uintptr_t size) | |
| 238 | + shadow_origin_ptr_t __msan_metadata_ptr_for_store_n(void *addr, uintptr_t size) | |
| 239 | + | |
| 240 | +The function name depends on the memory access size. | |
| 241 | + | |
| 242 | +The compiler makes sure that for every loaded value its shadow and origin | |
| 243 | +values are read from memory. When a value is stored to memory, its shadow and | |
| 244 | +origin are also stored using the metadata pointers. | |
| 245 | + | |
| 246 | +Handling locals | |
| 247 | +~~~~~~~~~~~~~~~ | |
| 248 | + | |
| 249 | +A special function is used to create a new origin value for a local variable and | |
| 250 | +set the origin of that variable to that value:: | |
| 251 | + | |
| 252 | + void __msan_poison_alloca(void *addr, uintptr_t size, char *descr) | |
| 253 | + | |
| 254 | +Access to per-task data | |
| 255 | +~~~~~~~~~~~~~~~~~~~~~~~ | |
| 256 | + | |
| 257 | +At the beginning of every instrumented function KMSAN inserts a call to | |
| 258 | +``__msan_get_context_state()``:: | |
| 259 | + | |
| 260 | + kmsan_context_state *__msan_get_context_state(void) | |
| 261 | + | |
| 262 | +``kmsan_context_state`` is declared in ``include/linux/kmsan.h``:: | |
| 263 | + | |
| 264 | + struct kmsan_context_state { | |
| 265 | + char param_tls[KMSAN_PARAM_SIZE]; | |
| 266 | + char retval_tls[KMSAN_RETVAL_SIZE]; | |
| 267 | + char va_arg_tls[KMSAN_PARAM_SIZE]; | |
| 268 | + char va_arg_origin_tls[KMSAN_PARAM_SIZE]; | |
| 269 | + u64 va_arg_overflow_size_tls; | |
| 270 | + char param_origin_tls[KMSAN_PARAM_SIZE]; | |
| 271 | + depot_stack_handle_t retval_origin_tls; | |
| 272 | + }; | |
| 273 | + | |
| 274 | +This structure is used by KMSAN to pass parameter shadows and origins between | |
| 275 | +instrumented functions (unless the parameters are checked immediately by | |
| 276 | +``CONFIG_KMSAN_CHECK_PARAM_RETVAL``). | |
| 277 | + | |
| 278 | +Passing uninitialized values to functions | |
| 279 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
| 280 | + | |
| 281 | +Clang's MemorySanitizer instrumentation has an option, | |
| 282 | +``-fsanitize-memory-param-retval``, which makes the compiler check function | |
| 283 | +parameters passed by value, as well as function return values. | |
| 284 | + | |
| 285 | +The option is controlled by ``CONFIG_KMSAN_CHECK_PARAM_RETVAL``, which is | |
| 286 | +enabled by default to let KMSAN report uninitialized values earlier. | |
| 287 | +Please refer to the `LKML discussion`_ for more details. | |
| 288 | + | |
| 289 | +Because of the way the checks are implemented in LLVM (they are only applied to | |
| 290 | +parameters marked as ``noundef``), not all parameters are guaranteed to be | |
| 291 | +checked, so we cannot give up the metadata storage in ``kmsan_context_state``. | |
| 292 | + | |
| 293 | +String functions | |
| 294 | +~~~~~~~~~~~~~~~~ | |
| 295 | + | |
| 296 | +The compiler replaces calls to ``memcpy()``/``memmove()``/``memset()`` with the | |
| 297 | +following functions. These functions are also called when data structures are | |
| 298 | +initialized or copied, making sure shadow and origin values are copied alongside | |
| 299 | +with the data:: | |
| 300 | + | |
| 301 | + void *__msan_memcpy(void *dst, void *src, uintptr_t n) | |
| 302 | + void *__msan_memmove(void *dst, void *src, uintptr_t n) | |
| 303 | + void *__msan_memset(void *dst, int c, uintptr_t n) | |
| 304 | + | |
| 305 | +Error reporting | |
| 306 | +~~~~~~~~~~~~~~~ | |
| 307 | + | |
| 308 | +For each use of a value the compiler emits a shadow check that calls | |
| 309 | +``__msan_warning()`` in the case that value is poisoned:: | |
| 310 | + | |
| 311 | + void __msan_warning(u32 origin) | |
| 312 | + | |
| 313 | +``__msan_warning()`` causes KMSAN runtime to print an error report. | |
| 314 | + | |
| 315 | +Inline assembly instrumentation | |
| 316 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
| 317 | + | |
| 318 | +KMSAN instruments every inline assembly output with a call to:: | |
| 319 | + | |
| 320 | + void __msan_instrument_asm_store(void *addr, uintptr_t size) | |
| 321 | + | |
| 322 | +, which unpoisons the memory region. | |
| 323 | + | |
| 324 | +This approach may mask certain errors, but it also helps to avoid a lot of | |
| 325 | +false positives in bitwise operations, atomics etc. | |
| 326 | + | |
| 327 | +Sometimes the pointers passed into inline assembly do not point to valid memory. | |
| 328 | +In such cases they are ignored at runtime. | |
| 329 | + | |
| 330 | + | |
| 331 | +Runtime library | |
| 332 | +--------------- | |
| 333 | + | |
| 334 | +The code is located in ``mm/kmsan/``. | |
| 335 | + | |
| 336 | +Per-task KMSAN state | |
| 337 | +~~~~~~~~~~~~~~~~~~~~ | |
| 338 | + | |
| 339 | +Every task_struct has an associated KMSAN task state that holds the KMSAN | |
| 340 | +context (see above) and a per-task flag disallowing KMSAN reports:: | |
| 341 | + | |
| 342 | + struct kmsan_context { | |
| 343 | + ... | |
| 344 | + bool allow_reporting; | |
| 345 | + struct kmsan_context_state cstate; | |
| 346 | + ... | |
| 347 | + } | |
| 348 | + | |
| 349 | + struct task_struct { | |
| 350 | + ... | |
| 351 | + struct kmsan_context kmsan; | |
| 352 | + ... | |
| 353 | + } | |
| 354 | + | |
| 355 | +KMSAN contexts | |
| 356 | +~~~~~~~~~~~~~~ | |
| 357 | + | |
| 358 | +When running in a kernel task context, KMSAN uses ``current->kmsan.cstate`` to | |
| 359 | +hold the metadata for function parameters and return values. | |
| 360 | + | |
| 361 | +But in the case the kernel is running in the interrupt, softirq or NMI context, | |
| 362 | +where ``current`` is unavailable, KMSAN switches to per-cpu interrupt state:: | |
| 363 | + | |
| 364 | + DEFINE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); | |
| 365 | + | |
| 366 | +Metadata allocation | |
| 367 | +~~~~~~~~~~~~~~~~~~~ | |
| 368 | + | |
| 369 | +There are several places in the kernel for which the metadata is stored. | |
| 370 | + | |
| 371 | +1. Each ``struct page`` instance contains two pointers to its shadow and | |
| 372 | +origin pages:: | |
| 373 | + | |
| 374 | + struct page { | |
| 375 | + ... | |
| 376 | + struct page *shadow, *origin; | |
| 377 | + ... | |
| 378 | + }; | |
| 379 | + | |
| 380 | +At boot-time, the kernel allocates shadow and origin pages for every available | |
| 381 | +kernel page. This is done quite late, when the kernel address space is already | |
| 382 | +fragmented, so normal data pages may arbitrarily interleave with the metadata | |
| 383 | +pages. | |
| 384 | + | |
| 385 | +This means that in general for two contiguous memory pages their shadow/origin | |
| 386 | +pages may not be contiguous. Consequently, if a memory access crosses the | |
| 387 | +boundary of a memory block, accesses to shadow/origin memory may potentially | |
| 388 | +corrupt other pages or read incorrect values from them. | |
| 389 | + | |
| 390 | +In practice, contiguous memory pages returned by the same ``alloc_pages()`` | |
| 391 | +call will have contiguous metadata, whereas if these pages belong to two | |
| 392 | +different allocations their metadata pages can be fragmented. | |
| 393 | + | |
| 394 | +For the kernel data (``.data``, ``.bss`` etc.) and percpu memory regions | |
| 395 | +there also are no guarantees on metadata contiguity. | |
| 396 | + | |
| 397 | +In the case ``__msan_metadata_ptr_for_XXX_YYY()`` hits the border between two | |
| 398 | +pages with non-contiguous metadata, it returns pointers to fake shadow/origin regions:: | |
| 399 | + | |
| 400 | + char dummy_load_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE))); | |
| 401 | + char dummy_store_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE))); | |
| 402 | + | |
| 403 | +``dummy_load_page`` is zero-initialized, so reads from it always yield zeroes. | |
| 404 | +All stores to ``dummy_store_page`` are ignored. | |
| 405 | + | |
| 406 | +2. For vmalloc memory and modules, there is a direct mapping between the memory | |
| 407 | +range, its shadow and origin. KMSAN reduces the vmalloc area by 3/4, making only | |
| 408 | +the first quarter available to ``vmalloc()``. The second quarter of the vmalloc | |
| 409 | +area contains shadow memory for the first quarter, the third one holds the | |
| 410 | +origins. A small part of the fourth quarter contains shadow and origins for the | |
| 411 | +kernel modules. Please refer to ``arch/x86/include/asm/pgtable_64_types.h`` for | |
| 412 | +more details. | |
| 413 | + | |
| 414 | +When an array of pages is mapped into a contiguous virtual memory space, their | |
| 415 | +shadow and origin pages are similarly mapped into contiguous regions. | |
| 416 | + | |
| 417 | +References | |
| 418 | +========== | |
| 419 | + | |
| 420 | +E. Stepanov, K. Serebryany. `MemorySanitizer: fast detector of uninitialized | |
| 421 | +memory use in C++ | |
| 422 | +<https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43308.pdf>`_. | |
| 423 | +In Proceedings of CGO 2015. | |
| 424 | + | |
| 425 | +.. _MemorySanitizer tool: https://clang.llvm.org/docs/MemorySanitizer.html | |
| 426 | +.. _LLVM documentation: https://llvm.org/docs/GettingStarted.html | |
| 427 | +.. _LKML discussion: https://lore.kernel.org/all/20220614144853.3693273-1-glider@google.com/ | |
| @@ -51,6 +51,7 @@ above structured documentation, or deleted if it has served its purpose. · 50 unmodified lines | ||
| 51 | 51 | ksm |
| 52 | 52 | memory-model |
| 53 | 53 | mmu_notifier |
| 54 | + multigen_lru | |
| 54 | 55 | numa |
| 55 | 56 | overcommit-accounting |
| 56 | 57 | page_migration |
| @@ -26,7 +26,7 @@ tree. · 25 unmodified lines | ||
| 26 | 26 | |
| 27 | 27 | If a KSM page is shared between less than ``max_page_sharing`` VMAs, |
| 28 | 28 | the node of the stable tree that represents such KSM page points to a |
| 29 | −list of struct rmap_item and the ``page->mapping`` of the | |
| 29 | +list of struct ksm_rmap_item and the ``page->mapping`` of the | |
| 30 | 30 | KSM page points to the stable tree node. |
| 31 | 31 | |
| 32 | 32 | When the sharing passes this threshold, KSM adds a second dimension to |
| @@ -0,0 +1,159 @@ | ||
| 1 | +.. SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +============= | |
| 4 | +Multi-Gen LRU | |
| 5 | +============= | |
| 6 | +The multi-gen LRU is an alternative LRU implementation that optimizes | |
| 7 | +page reclaim and improves performance under memory pressure. Page | |
| 8 | +reclaim decides the kernel's caching policy and ability to overcommit | |
| 9 | +memory. It directly impacts the kswapd CPU usage and RAM efficiency. | |
| 10 | + | |
| 11 | +Design overview | |
| 12 | +=============== | |
| 13 | +Objectives | |
| 14 | +---------- | |
| 15 | +The design objectives are: | |
| 16 | + | |
| 17 | +* Good representation of access recency | |
| 18 | +* Try to profit from spatial locality | |
| 19 | +* Fast paths to make obvious choices | |
| 20 | +* Simple self-correcting heuristics | |
| 21 | + | |
| 22 | +The representation of access recency is at the core of all LRU | |
| 23 | +implementations. In the multi-gen LRU, each generation represents a | |
| 24 | +group of pages with similar access recency. Generations establish a | |
| 25 | +(time-based) common frame of reference and therefore help make better | |
| 26 | +choices, e.g., between different memcgs on a computer or different | |
| 27 | +computers in a data center (for job scheduling). | |
| 28 | + | |
| 29 | +Exploiting spatial locality improves efficiency when gathering the | |
| 30 | +accessed bit. A rmap walk targets a single page and does not try to | |
| 31 | +profit from discovering a young PTE. A page table walk can sweep all | |
| 32 | +the young PTEs in an address space, but the address space can be too | |
| 33 | +sparse to make a profit. The key is to optimize both methods and use | |
| 34 | +them in combination. | |
| 35 | + | |
| 36 | +Fast paths reduce code complexity and runtime overhead. Unmapped pages | |
| 37 | +do not require TLB flushes; clean pages do not require writeback. | |
| 38 | +These facts are only helpful when other conditions, e.g., access | |
| 39 | +recency, are similar. With generations as a common frame of reference, | |
| 40 | +additional factors stand out. But obvious choices might not be good | |
| 41 | +choices; thus self-correction is necessary. | |
| 42 | + | |
| 43 | +The benefits of simple self-correcting heuristics are self-evident. | |
| 44 | +Again, with generations as a common frame of reference, this becomes | |
| 45 | +attainable. Specifically, pages in the same generation can be | |
| 46 | +categorized based on additional factors, and a feedback loop can | |
| 47 | +statistically compare the refault percentages across those categories | |
| 48 | +and infer which of them are better choices. | |
| 49 | + | |
| 50 | +Assumptions | |
| 51 | +----------- | |
| 52 | +The protection of hot pages and the selection of cold pages are based | |
| 53 | +on page access channels and patterns. There are two access channels: | |
| 54 | + | |
| 55 | +* Accesses through page tables | |
| 56 | +* Accesses through file descriptors | |
| 57 | + | |
| 58 | +The protection of the former channel is by design stronger because: | |
| 59 | + | |
| 60 | +1. The uncertainty in determining the access patterns of the former | |
| 61 | + channel is higher due to the approximation of the accessed bit. | |
| 62 | +2. The cost of evicting the former channel is higher due to the TLB | |
| 63 | + flushes required and the likelihood of encountering the dirty bit. | |
| 64 | +3. The penalty of underprotecting the former channel is higher because | |
| 65 | + applications usually do not prepare themselves for major page | |
| 66 | + faults like they do for blocked I/O. E.g., GUI applications | |
| 67 | + commonly use dedicated I/O threads to avoid blocking rendering | |
| 68 | + threads. | |
| 69 | + | |
| 70 | +There are also two access patterns: | |
| 71 | + | |
| 72 | +* Accesses exhibiting temporal locality | |
| 73 | +* Accesses not exhibiting temporal locality | |
| 74 | + | |
| 75 | +For the reasons listed above, the former channel is assumed to follow | |
| 76 | +the former pattern unless ``VM_SEQ_READ`` or ``VM_RAND_READ`` is | |
| 77 | +present, and the latter channel is assumed to follow the latter | |
| 78 | +pattern unless outlying refaults have been observed. | |
| 79 | + | |
| 80 | +Workflow overview | |
| 81 | +================= | |
| 82 | +Evictable pages are divided into multiple generations for each | |
| 83 | +``lruvec``. The youngest generation number is stored in | |
| 84 | +``lrugen->max_seq`` for both anon and file types as they are aged on | |
| 85 | +an equal footing. The oldest generation numbers are stored in | |
| 86 | +``lrugen->min_seq[]`` separately for anon and file types as clean file | |
| 87 | +pages can be evicted regardless of swap constraints. These three | |
| 88 | +variables are monotonically increasing. | |
| 89 | + | |
| 90 | +Generation numbers are truncated into ``order_base_2(MAX_NR_GENS+1)`` | |
| 91 | +bits in order to fit into the gen counter in ``folio->flags``. Each | |
| 92 | +truncated generation number is an index to ``lrugen->lists[]``. The | |
| 93 | +sliding window technique is used to track at least ``MIN_NR_GENS`` and | |
| 94 | +at most ``MAX_NR_GENS`` generations. The gen counter stores a value | |
| 95 | +within ``[1, MAX_NR_GENS]`` while a page is on one of | |
| 96 | +``lrugen->lists[]``; otherwise it stores zero. | |
| 97 | + | |
| 98 | +Each generation is divided into multiple tiers. A page accessed ``N`` | |
| 99 | +times through file descriptors is in tier ``order_base_2(N)``. Unlike | |
| 100 | +generations, tiers do not have dedicated ``lrugen->lists[]``. In | |
| 101 | +contrast to moving across generations, which requires the LRU lock, | |
| 102 | +moving across tiers only involves atomic operations on | |
| 103 | +``folio->flags`` and therefore has a negligible cost. A feedback loop | |
| 104 | +modeled after the PID controller monitors refaults over all the tiers | |
| 105 | +from anon and file types and decides which tiers from which types to | |
| 106 | +evict or protect. | |
| 107 | + | |
| 108 | +There are two conceptually independent procedures: the aging and the | |
| 109 | +eviction. They form a closed-loop system, i.e., the page reclaim. | |
| 110 | + | |
| 111 | +Aging | |
| 112 | +----- | |
| 113 | +The aging produces young generations. Given an ``lruvec``, it | |
| 114 | +increments ``max_seq`` when ``max_seq-min_seq+1`` approaches | |
| 115 | +``MIN_NR_GENS``. The aging promotes hot pages to the youngest | |
| 116 | +generation when it finds them accessed through page tables; the | |
| 117 | +demotion of cold pages happens consequently when it increments | |
| 118 | +``max_seq``. The aging uses page table walks and rmap walks to find | |
| 119 | +young PTEs. For the former, it iterates ``lruvec_memcg()->mm_list`` | |
| 120 | +and calls ``walk_page_range()`` with each ``mm_struct`` on this list | |
| 121 | +to scan PTEs, and after each iteration, it increments ``max_seq``. For | |
| 122 | +the latter, when the eviction walks the rmap and finds a young PTE, | |
| 123 | +the aging scans the adjacent PTEs. For both, on finding a young PTE, | |
| 124 | +the aging clears the accessed bit and updates the gen counter of the | |
| 125 | +page mapped by this PTE to ``(max_seq%MAX_NR_GENS)+1``. | |
| 126 | + | |
| 127 | +Eviction | |
| 128 | +-------- | |
| 129 | +The eviction consumes old generations. Given an ``lruvec``, it | |
| 130 | +increments ``min_seq`` when ``lrugen->lists[]`` indexed by | |
| 131 | +``min_seq%MAX_NR_GENS`` becomes empty. To select a type and a tier to | |
| 132 | +evict from, it first compares ``min_seq[]`` to select the older type. | |
| 133 | +If both types are equally old, it selects the one whose first tier has | |
| 134 | +a lower refault percentage. The first tier contains single-use | |
| 135 | +unmapped clean pages, which are the best bet. The eviction sorts a | |
| 136 | +page according to its gen counter if the aging has found this page | |
| 137 | +accessed through page tables and updated its gen counter. It also | |
| 138 | +moves a page to the next generation, i.e., ``min_seq+1``, if this page | |
| 139 | +was accessed multiple times through file descriptors and the feedback | |
| 140 | +loop has detected outlying refaults from the tier this page is in. To | |
| 141 | +this end, the feedback loop uses the first tier as the baseline, for | |
| 142 | +the reason stated earlier. | |
| 143 | + | |
| 144 | +Summary | |
| 145 | +------- | |
| 146 | +The multi-gen LRU can be disassembled into the following parts: | |
| 147 | + | |
| 148 | +* Generations | |
| 149 | +* Rmap walks | |
| 150 | +* Page table walks | |
| 151 | +* Bloom filters | |
| 152 | +* PID controller | |
| 153 | + | |
| 154 | +The aging and the eviction form a producer-consumer model; | |
| 155 | +specifically, the latter drives the former by the sliding window over | |
| 156 | +generations. Within the aging, rmap walks drive page table walks by | |
| 157 | +inserting hot densely populated page tables to the Bloom filters. | |
| 158 | +Within the eviction, the PID controller uses refaults as the feedback | |
| 159 | +to select types to evict and tiers to protect. | |
| @@ -94,6 +94,11 @@ Usage · 93 unmodified lines | ||
| 94 | 94 | Page allocated via order XXX, ... |
| 95 | 95 | PFN XXX ... |
| 96 | 96 | // Detailed stack |
| 97 | + By default, it will do full pfn dump, to start with a given pfn, | |
| 98 | + page_owner supports fseek. | |
| 99 | + | |
| 100 | + FILE *fp = fopen("/sys/kernel/debug/page_owner", "r"); | |
| 101 | + fseek(fp, pfn_start, SEEK_SET); | |
| 97 | 102 | |
| 98 | 103 | The ``page_owner_sort`` tool ignores ``PFN`` rows, puts the remaining rows |
| 99 | 104 | in buf, uses regexp to extract the page order value, counts the times |
| @@ -11004,7 +11004,6 @@ F: arch/*/include/asm/*kasan.h · 11003 unmodified lines | ||
| 11004 | 11004 | F: arch/*/mm/kasan_init* |
| 11005 | 11005 | F: include/linux/kasan*.h |
| 11006 | 11006 | F: lib/Kconfig.kasan |
| 11007 | −F: lib/test_kasan*.c | |
| 11008 | 11007 | F: mm/kasan/ |
| 11009 | 11008 | F: scripts/Makefile.kasan |
| 11010 | 11009 | |
| @@ -11438,6 +11437,20 @@ F: kernel/kmod.c · 427 unmodified lines | ||
| 11438 | 11437 | F: lib/test_kmod.c |
| 11439 | 11438 | F: tools/testing/selftests/kmod/ |
| 11440 | 11439 | |
| 11440 | +KMSAN | |
| 11441 | +M: Alexander Potapenko <glider@google.com> | |
| 11442 | +R: Marco Elver <elver@google.com> | |
| 11443 | +R: Dmitry Vyukov <dvyukov@google.com> | |
| 11444 | +L: kasan-dev@googlegroups.com | |
| 11445 | +S: Maintained | |
| 11446 | +F: Documentation/dev-tools/kmsan.rst | |
| 11447 | +F: arch/*/include/asm/kmsan.h | |
| 11448 | +F: arch/*/mm/kmsan_* | |
| 11449 | +F: include/linux/kmsan*.h | |
| 11450 | +F: lib/Kconfig.kmsan | |
| 11451 | +F: mm/kmsan/ | |
| 11452 | +F: scripts/Makefile.kmsan | |
| 11453 | + | |
| 11441 | 11454 | KPROBES |
| 11442 | 11455 | M: Naveen N. Rao <naveen.n.rao@linux.ibm.com> |
| 11443 | 11456 | M: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com> |
| @@ -12168,6 +12181,18 @@ L: linux-man@vger.kernel.org · 724 unmodified lines | ||
| 12168 | 12181 | S: Maintained |
| 12169 | 12182 | W: http://www.kernel.org/doc/man-pages |
| 12170 | 12183 | |
| 12184 | +MAPLE TREE | |
| 12185 | +M: Liam R. Howlett <Liam.Howlett@oracle.com> | |
| 12186 | +L: linux-mm@kvack.org | |
| 12187 | +S: Supported | |
| 12188 | +F: Documentation/core-api/maple_tree.rst | |
| 12189 | +F: include/linux/maple_tree.h | |
| 12190 | +F: include/trace/events/maple_tree.h | |
| 12191 | +F: lib/maple_tree.c | |
| 12192 | +F: lib/test_maple_tree.c | |
| 12193 | +F: tools/testing/radix-tree/linux/maple_tree.h | |
| 12194 | +F: tools/testing/radix-tree/maple.c | |
| 12195 | + | |
| 12171 | 12196 | MARDUK (CREATOR CI40) DEVICE TREE SUPPORT |
| 12172 | 12197 | M: Rahul Bedarkar <rahulbedarkar89@gmail.com> |
| 12173 | 12198 | L: linux-mips@vger.kernel.org |
| @@ -1081,6 +1081,7 @@ include-y := scripts/Makefile.extrawarn · 1080 unmodified lines | ||
| 1081 | 1081 | include-$(CONFIG_DEBUG_INFO) += scripts/Makefile.debug |
| 1082 | 1082 | include-$(CONFIG_KASAN) += scripts/Makefile.kasan |
| 1083 | 1083 | include-$(CONFIG_KCSAN) += scripts/Makefile.kcsan |
| 1084 | +include-$(CONFIG_KMSAN) += scripts/Makefile.kmsan | |
| 1084 | 1085 | include-$(CONFIG_UBSAN) += scripts/Makefile.ubsan |
| 1085 | 1086 | include-$(CONFIG_KCOV) += scripts/Makefile.kcov |
| 1086 | 1087 | include-$(CONFIG_RANDSTRUCT) += scripts/Makefile.randstruct |
| @@ -1416,6 +1416,14 @@ config DYNAMIC_SIGFRAME · 1415 unmodified lines | ||
| 1416 | 1416 | config HAVE_ARCH_NODE_DEV_GROUP |
| 1417 | 1417 | bool |
| 1418 | 1418 | |
| 1419 | +config ARCH_HAS_NONLEAF_PMD_YOUNG | |
| 1420 | + bool | |
| 1421 | + help | |
| 1422 | + Architectures that select this option are capable of setting the | |
| 1423 | + accessed bit in non-leaf PMD entries when using them as part of linear | |
| 1424 | + address translations. Page table walkers that clear the accessed bit | |
| 1425 | + may use this capability to reduce their search space. | |
| 1426 | + | |
| 1419 | 1427 | source "kernel/gcov/Kconfig" |
| 1420 | 1428 | |
| 1421 | 1429 | source "scripts/gcc-plugins/Kconfig" |
| @@ -76,6 +76,8 @@ · 75 unmodified lines | ||
| 76 | 76 | |
| 77 | 77 | #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ |
| 78 | 78 | |
| 79 | +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ | |
| 80 | + | |
| 79 | 81 | /* compatibility flags */ |
| 80 | 82 | #define MAP_FILE 0 |
| 81 | 83 | |
| @@ -554,7 +554,7 @@ config ARC_BUILTIN_DTB_NAME · 553 unmodified lines | ||
| 554 | 554 | |
| 555 | 555 | endmenu # "ARC Architecture Configuration" |
| 556 | 556 | |
| 557 | −config FORCE_MAX_ZONEORDER | |
| 557 | +config ARCH_FORCE_MAX_ORDER | |
| 558 | 558 | int "Maximum zone order" |
| 559 | 559 | default "12" if ARC_HUGEPAGE_16M |
| 560 | 560 | default "11" |
| @@ -1362,7 +1362,7 @@ config ARM_MODULE_PLTS · 1361 unmodified lines | ||
| 1362 | 1362 | Disabling this is usually safe for small single-platform |
| 1363 | 1363 | configurations. If unsure, say y. |
| 1364 | 1364 | |
| 1365 | −config FORCE_MAX_ZONEORDER | |
| 1365 | +config ARCH_FORCE_MAX_ORDER | |
| 1366 | 1366 | int "Maximum zone order" |
| 1367 | 1367 | default "12" if SOC_AM33XX |
| 1368 | 1368 | default "9" if SA1111 |
| @@ -31,7 +31,7 @@ CONFIG_SOC_VF610=y · 30 unmodified lines | ||
| 31 | 31 | CONFIG_SMP=y |
| 32 | 32 | CONFIG_ARM_PSCI=y |
| 33 | 33 | CONFIG_HIGHMEM=y |
| 34 | −CONFIG_FORCE_MAX_ZONEORDER=14 | |
| 34 | +CONFIG_ARCH_FORCE_MAX_ORDER=14 | |
| 35 | 35 | CONFIG_CMDLINE="noinitrd console=ttymxc0,115200" |
| 36 | 36 | CONFIG_KEXEC=y |
| 37 | 37 | CONFIG_CPU_FREQ=y |
| @@ -26,7 +26,7 @@ CONFIG_THUMB2_KERNEL=y · 25 unmodified lines | ||
| 26 | 26 | # CONFIG_THUMB2_AVOID_R_ARM_THM_JUMP11 is not set |
| 27 | 27 | # CONFIG_ARM_PATCH_IDIV is not set |
| 28 | 28 | CONFIG_HIGHMEM=y |
| 29 | −CONFIG_FORCE_MAX_ZONEORDER=12 | |
| 29 | +CONFIG_ARCH_FORCE_MAX_ORDER=12 | |
| 30 | 30 | CONFIG_SECCOMP=y |
| 31 | 31 | CONFIG_KEXEC=y |
| 32 | 32 | CONFIG_EFI=y |
| @@ -12,7 +12,7 @@ CONFIG_ARCH_OXNAS=y · 11 unmodified lines | ||
| 12 | 12 | CONFIG_MACH_OX820=y |
| 13 | 13 | CONFIG_SMP=y |
| 14 | 14 | CONFIG_NR_CPUS=16 |
| 15 | −CONFIG_FORCE_MAX_ZONEORDER=12 | |
| 15 | +CONFIG_ARCH_FORCE_MAX_ORDER=12 | |
| 16 | 16 | CONFIG_SECCOMP=y |
| 17 | 17 | CONFIG_ARM_APPENDED_DTB=y |
| 18 | 18 | CONFIG_ARM_ATAG_DTB_COMPAT=y |
| @@ -21,7 +21,7 @@ CONFIG_MACH_AKITA=y · 20 unmodified lines | ||
| 21 | 21 | CONFIG_MACH_BORZOI=y |
| 22 | 22 | CONFIG_PXA_SYSTEMS_CPLDS=y |
| 23 | 23 | CONFIG_AEABI=y |
| 24 | −CONFIG_FORCE_MAX_ZONEORDER=9 | |
| 24 | +CONFIG_ARCH_FORCE_MAX_ORDER=9 | |
| 25 | 25 | CONFIG_CMDLINE="root=/dev/ram0 ro" |
| 26 | 26 | CONFIG_KEXEC=y |
| 27 | 27 | CONFIG_CPU_FREQ=y |
| @@ -19,7 +19,7 @@ CONFIG_ATMEL_CLOCKSOURCE_TCB=y · 18 unmodified lines | ||
| 19 | 19 | # CONFIG_CACHE_L2X0 is not set |
| 20 | 20 | # CONFIG_ARM_PATCH_IDIV is not set |
| 21 | 21 | # CONFIG_CPU_SW_DOMAIN_PAN is not set |
| 22 | −CONFIG_FORCE_MAX_ZONEORDER=15 | |
| 22 | +CONFIG_ARCH_FORCE_MAX_ORDER=15 | |
| 23 | 23 | CONFIG_UACCESS_WITH_MEMCPY=y |
| 24 | 24 | # CONFIG_ATAGS is not set |
| 25 | 25 | CONFIG_CMDLINE="console=ttyS0,115200 earlyprintk ignore_loglevel" |
| @@ -17,7 +17,7 @@ CONFIG_ARCH_SUNPLUS=y · 16 unmodified lines | ||
| 17 | 17 | # CONFIG_VDSO is not set |
| 18 | 18 | CONFIG_SMP=y |
| 19 | 19 | CONFIG_THUMB2_KERNEL=y |
| 20 | −CONFIG_FORCE_MAX_ZONEORDER=12 | |
| 20 | +CONFIG_ARCH_FORCE_MAX_ORDER=12 | |
| 21 | 21 | CONFIG_VFP=y |
| 22 | 22 | CONFIG_NEON=y |
| 23 | 23 | CONFIG_MODULES=y |
| @@ -1431,7 +1431,7 @@ config XEN · 1430 unmodified lines | ||
| 1431 | 1431 | help |
| 1432 | 1432 | Say Y if you want to run Linux in a Virtual Machine on Xen on ARM64. |
| 1433 | 1433 | |
| 1434 | −config FORCE_MAX_ZONEORDER | |
| 1434 | +config ARCH_FORCE_MAX_ORDER | |
| 1435 | 1435 | int |
| 1436 | 1436 | default "14" if ARM64_64K_PAGES |
| 1437 | 1437 | default "12" if ARM64_16K_PAGES |
| @@ -1082,24 +1082,13 @@ static inline void update_mmu_cache(struct vm_area_struct *vma, · 1081 unmodified lines | ||
| 1082 | 1082 | * page after fork() + CoW for pfn mappings. We don't always have a |
| 1083 | 1083 | * hardware-managed access flag on arm64. |
| 1084 | 1084 | */ |
| 1085 | −static inline bool arch_faults_on_old_pte(void) | |
| 1086 | −{ | |
| 1087 | − /* The register read below requires a stable CPU to make any sense */ | |
| 1088 | − cant_migrate(); | |
| 1089 | − | |
| 1090 | − return !cpu_has_hw_af(); | |
| 1091 | −} | |
| 1092 | −#define arch_faults_on_old_pte arch_faults_on_old_pte | |
| 1085 | +#define arch_has_hw_pte_young cpu_has_hw_af | |
| 1093 | 1086 | |
| 1094 | 1087 | /* |
| 1095 | 1088 | * Experimentally, it's cheap to set the access flag in hardware and we |
| 1096 | 1089 | * benefit from prefaulting mappings as 'old' to start with. |
| 1097 | 1090 | */ |
| 1098 | −static inline bool arch_wants_old_prefaulted_pte(void) | |
| 1099 | −{ | |
| 1100 | − return !arch_faults_on_old_pte(); | |
| 1101 | −} | |
| 1102 | −#define arch_wants_old_prefaulted_pte arch_wants_old_prefaulted_pte | |
| 1091 | +#define arch_wants_old_prefaulted_pte cpu_has_hw_af | |
| 1103 | 1092 | |
| 1104 | 1093 | static inline bool pud_sect_supported(void) |
| 1105 | 1094 | { |
| @@ -8,9 +8,9 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <asm/cpufeature.h> |
| 9 | 9 | #include <asm/mte.h> |
| 10 | 10 | |
| 11 | −#define for_each_mte_vma(tsk, vma) \ | |
| 11 | +#define for_each_mte_vma(vmi, vma) \ | |
| 12 | 12 | if (system_supports_mte()) \ |
| 13 | − for (vma = tsk->mm->mmap; vma; vma = vma->vm_next) \ | |
| 13 | + for_each_vma(vmi, vma) \ | |
| 14 | 14 | if (vma->vm_flags & VM_MTE) |
| 15 | 15 | |
| 16 | 16 | static unsigned long mte_vma_tag_dump_size(struct vm_area_struct *vma) |
| @@ -81,8 +81,9 @@ Elf_Half elf_core_extra_phdrs(void) · 64 unmodified lines | ||
| 81 | 81 | { |
| 82 | 82 | struct vm_area_struct *vma; |
| 83 | 83 | int vma_count = 0; |
| 84 | + VMA_ITERATOR(vmi, current->mm, 0); | |
| 84 | 85 | |
| 85 | − for_each_mte_vma(current, vma) | |
| 86 | + for_each_mte_vma(vmi, vma) | |
| 86 | 87 | vma_count++; |
| 87 | 88 | |
| 88 | 89 | return vma_count; |
| @@ -91,8 +92,9 @@ Elf_Half elf_core_extra_phdrs(void) · 2 unmodified lines | ||
| 91 | 92 | int elf_core_write_extra_phdrs(struct coredump_params *cprm, loff_t offset) |
| 92 | 93 | { |
| 93 | 94 | struct vm_area_struct *vma; |
| 95 | + VMA_ITERATOR(vmi, current->mm, 0); | |
| 94 | 96 | |
| 95 | − for_each_mte_vma(current, vma) { | |
| 97 | + for_each_mte_vma(vmi, vma) { | |
| 96 | 98 | struct elf_phdr phdr; |
| 97 | 99 | |
| 98 | 100 | phdr.p_type = PT_AARCH64_MEMTAG_MTE; |
| @@ -116,8 +118,9 @@ size_t elf_core_extra_data_size(void) · 17 unmodified lines | ||
| 116 | 118 | { |
| 117 | 119 | struct vm_area_struct *vma; |
| 118 | 120 | size_t data_size = 0; |
| 121 | + VMA_ITERATOR(vmi, current->mm, 0); | |
| 119 | 122 | |
| 120 | − for_each_mte_vma(current, vma) | |
| 123 | + for_each_mte_vma(vmi, vma) | |
| 121 | 124 | data_size += mte_vma_tag_dump_size(vma); |
| 122 | 125 | |
| 123 | 126 | return data_size; |
| @@ -126,8 +129,9 @@ size_t elf_core_extra_data_size(void) · 2 unmodified lines | ||
| 126 | 129 | int elf_core_write_extra_data(struct coredump_params *cprm) |
| 127 | 130 | { |
| 128 | 131 | struct vm_area_struct *vma; |
| 132 | + VMA_ITERATOR(vmi, current->mm, 0); | |
| 129 | 133 | |
| 130 | − for_each_mte_vma(current, vma) { | |
| 134 | + for_each_mte_vma(vmi, vma) { | |
| 131 | 135 | if (vma->vm_flags & VM_DONTDUMP) |
| 132 | 136 | continue; |
| 133 | 137 | |
| @@ -133,10 +133,11 @@ int vdso_join_timens(struct task_struct *task, struct time_namespace *ns) · 132 unmodified lines | ||
| 133 | 133 | { |
| 134 | 134 | struct mm_struct *mm = task->mm; |
| 135 | 135 | struct vm_area_struct *vma; |
| 136 | + VMA_ITERATOR(vmi, mm, 0); | |
| 136 | 137 | |
| 137 | 138 | mmap_read_lock(mm); |
| 138 | 139 | |
| 139 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 140 | + for_each_vma(vmi, vma) { | |
| 140 | 141 | unsigned long size = vma->vm_end - vma->vm_start; |
| 141 | 142 | |
| 142 | 143 | if (vma_is_special_mapping(vma, vdso_info[VDSO_ABI_AA64].dm)) |
| @@ -245,7 +245,7 @@ static inline struct folio *hugetlb_swap_entry_to_folio(swp_entry_t entry) · 244 unmodified lines | ||
| 245 | 245 | { |
| 246 | 246 | VM_BUG_ON(!is_migration_entry(entry) && !is_hwpoison_entry(entry)); |
| 247 | 247 | |
| 248 | − return page_folio(pfn_to_page(swp_offset(entry))); | |
| 248 | + return page_folio(pfn_to_page(swp_offset_pfn(entry))); | |
| 249 | 249 | } |
| 250 | 250 | |
| 251 | 251 | void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, |
| @@ -332,7 +332,7 @@ config HIGHMEM · 331 unmodified lines | ||
| 332 | 332 | select KMAP_LOCAL |
| 333 | 333 | default y |
| 334 | 334 | |
| 335 | −config FORCE_MAX_ZONEORDER | |
| 335 | +config ARCH_FORCE_MAX_ORDER | |
| 336 | 336 | int "Maximum zone order" |
| 337 | 337 | default "11" |
| 338 | 338 | |
| @@ -200,7 +200,7 @@ config IA64_CYCLONE · 199 unmodified lines | ||
| 200 | 200 | Say Y here to enable support for IBM EXA Cyclone time source. |
| 201 | 201 | If you're unsure, answer N. |
| 202 | 202 | |
| 203 | −config FORCE_MAX_ZONEORDER | |
| 203 | +config ARCH_FORCE_MAX_ORDER | |
| 204 | 204 | int "MAX_ORDER (11 - 17)" if !HUGETLB_PAGE |
| 205 | 205 | range 11 17 if !HUGETLB_PAGE |
| 206 | 206 | default "17" if HUGETLB_PAGE |
| @@ -11,10 +11,10 @@ · 10 unmodified lines | ||
| 11 | 11 | |
| 12 | 12 | #define SECTION_SIZE_BITS (30) |
| 13 | 13 | #define MAX_PHYSMEM_BITS (50) |
| 14 | −#ifdef CONFIG_FORCE_MAX_ZONEORDER | |
| 15 | −#if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS) | |
| 14 | +#ifdef CONFIG_ARCH_FORCE_MAX_ORDER | |
| 15 | +#if ((CONFIG_ARCH_FORCE_MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS) | |
| 16 | 16 | #undef SECTION_SIZE_BITS |
| 17 | −#define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) | |
| 17 | +#define SECTION_SIZE_BITS (CONFIG_ARCH_FORCE_MAX_ORDER - 1 + PAGE_SHIFT) | |
| 18 | 18 | #endif |
| 19 | 19 | #endif |
| 20 | 20 | |
| @@ -377,7 +377,7 @@ config NODES_SHIFT · 376 unmodified lines | ||
| 377 | 377 | default "6" |
| 378 | 378 | depends on NUMA |
| 379 | 379 | |
| 380 | −config FORCE_MAX_ZONEORDER | |
| 380 | +config ARCH_FORCE_MAX_ORDER | |
| 381 | 381 | int "Maximum zone order" |
| 382 | 382 | range 14 64 if PAGE_SIZE_64KB |
| 383 | 383 | default "14" if PAGE_SIZE_64KB |
| @@ -397,7 +397,7 @@ config SINGLE_MEMORY_CHUNK · 396 unmodified lines | ||
| 397 | 397 | order" to save memory that could be wasted for unused memory map. |
| 398 | 398 | Say N if not sure. |
| 399 | 399 | |
| 400 | −config FORCE_MAX_ZONEORDER | |
| 400 | +config ARCH_FORCE_MAX_ORDER | |
| 401 | 401 | int "Maximum zone order" if ADVANCED |
| 402 | 402 | depends on !SINGLE_MEMORY_CHUNK |
| 403 | 403 | default "11" |
| @@ -2140,7 +2140,7 @@ config PAGE_SIZE_64KB · 2139 unmodified lines | ||
| 2140 | 2140 | |
| 2141 | 2141 | endchoice |
| 2142 | 2142 | |
| 2143 | −config FORCE_MAX_ZONEORDER | |
| 2143 | +config ARCH_FORCE_MAX_ORDER | |
| 2144 | 2144 | int "Maximum zone order" |
| 2145 | 2145 | range 14 64 if MIPS_HUGE_TLB_SUPPORT && PAGE_SIZE_64KB |
| 2146 | 2146 | default "14" if MIPS_HUGE_TLB_SUPPORT && PAGE_SIZE_64KB |
| @@ -9,7 +9,6 @@ CONFIG_HIGH_RES_TIMERS=y · 8 unmodified lines | ||
| 9 | 9 | CONFIG_LOG_BUF_SHIFT=16 |
| 10 | 10 | CONFIG_CGROUPS=y |
| 11 | 11 | CONFIG_MEMCG=y |
| 12 | −CONFIG_MEMCG_SWAP=y | |
| 13 | 12 | CONFIG_BLK_CGROUP=y |
| 14 | 13 | CONFIG_CGROUP_SCHED=y |
| 15 | 14 | CONFIG_CFS_BANDWIDTH=y |
| @@ -3,7 +3,6 @@ CONFIG_NO_HZ_IDLE=y · 2 unmodified lines | ||
| 3 | 3 | CONFIG_IKCONFIG=y |
| 4 | 4 | CONFIG_IKCONFIG_PROC=y |
| 5 | 5 | CONFIG_MEMCG=y |
| 6 | −CONFIG_MEMCG_SWAP=y | |
| 7 | 6 | CONFIG_BLK_CGROUP=y |
| 8 | 7 | CONFIG_CFS_BANDWIDTH=y |
| 9 | 8 | CONFIG_RT_GROUP_SCHED=y |
| @@ -103,6 +103,8 @@ · 102 unmodified lines | ||
| 103 | 103 | |
| 104 | 104 | #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ |
| 105 | 105 | |
| 106 | +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ | |
| 107 | + | |
| 106 | 108 | /* compatibility flags */ |
| 107 | 109 | #define MAP_FILE 0 |
| 108 | 110 | |
| @@ -44,7 +44,7 @@ menu "Kernel features" · 43 unmodified lines | ||
| 44 | 44 | |
| 45 | 45 | source "kernel/Kconfig.hz" |
| 46 | 46 | |
| 47 | −config FORCE_MAX_ZONEORDER | |
| 47 | +config ARCH_FORCE_MAX_ORDER | |
| 48 | 48 | int "Maximum zone order" |
| 49 | 49 | range 9 20 |
| 50 | 50 | default "11" |
| @@ -70,6 +70,8 @@ · 69 unmodified lines | ||
| 70 | 70 | #define MADV_WIPEONFORK 71 /* Zero memory on fork, child only */ |
| 71 | 71 | #define MADV_KEEPONFORK 72 /* Undo MADV_WIPEONFORK */ |
| 72 | 72 | |
| 73 | +#define MADV_COLLAPSE 73 /* Synchronous hugepage collapse */ | |
| 74 | + | |
| 73 | 75 | #define MADV_HWPOISON 100 /* poison a page for testing */ |
| 74 | 76 | #define MADV_SOFT_OFFLINE 101 /* soft offline page for testing */ |
| 75 | 77 | |
| @@ -657,15 +657,20 @@ static inline unsigned long mm_total_size(struct mm_struct *mm) · 656 unmodified lines | ||
| 657 | 657 | { |
| 658 | 658 | struct vm_area_struct *vma; |
| 659 | 659 | unsigned long usize = 0; |
| 660 | + VMA_ITERATOR(vmi, mm, 0); | |
| 660 | 661 | |
| 661 | − for (vma = mm->mmap; vma && usize < parisc_cache_flush_threshold; vma = vma->vm_next) | |
| 662 | + for_each_vma(vmi, vma) { | |
| 663 | + if (usize >= parisc_cache_flush_threshold) | |
| 664 | + break; | |
| 662 | 665 | usize += vma->vm_end - vma->vm_start; |
| 666 | + } | |
| 663 | 667 | return usize; |
| 664 | 668 | } |
| 665 | 669 | |
| 666 | 670 | void flush_cache_mm(struct mm_struct *mm) |
| 667 | 671 | { |
| 668 | 672 | struct vm_area_struct *vma; |
| 673 | + VMA_ITERATOR(vmi, mm, 0); | |
| 669 | 674 | |
| 670 | 675 | /* |
| 671 | 676 | * Flushing the whole cache on each cpu takes forever on |
| @@ -685,7 +690,7 @@ void flush_cache_mm(struct mm_struct *mm) · 13 unmodified lines | ||
| 685 | 690 | } |
| 686 | 691 | |
| 687 | 692 | /* Flush mm */ |
| 688 | − for (vma = mm->mmap; vma; vma = vma->vm_next) | |
| 693 | + for_each_vma(vmi, vma) | |
| 689 | 694 | flush_cache_pages(vma, vma->vm_start, vma->vm_end); |
| 690 | 695 | } |
| 691 | 696 | |
| @@ -846,7 +846,7 @@ config DATA_SHIFT · 845 unmodified lines | ||
| 846 | 846 | in that case. If PIN_TLB is selected, it must be aligned to 8M as |
| 847 | 847 | 8M pages will be pinned. |
| 848 | 848 | |
| 849 | −config FORCE_MAX_ZONEORDER | |
| 849 | +config ARCH_FORCE_MAX_ORDER | |
| 850 | 850 | int "Maximum zone order" |
| 851 | 851 | range 8 9 if PPC64 && PPC_64K_PAGES |
| 852 | 852 | default "9" if PPC64 && PPC_64K_PAGES |
| @@ -30,7 +30,7 @@ CONFIG_PREEMPT=y · 29 unmodified lines | ||
| 30 | 30 | # CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set |
| 31 | 31 | CONFIG_BINFMT_MISC=m |
| 32 | 32 | CONFIG_MATH_EMULATION=y |
| 33 | −CONFIG_FORCE_MAX_ZONEORDER=17 | |
| 33 | +CONFIG_ARCH_FORCE_MAX_ORDER=17 | |
| 34 | 34 | CONFIG_PCI=y |
| 35 | 35 | CONFIG_PCIEPORTBUS=y |
| 36 | 36 | CONFIG_PCI_MSI=y |
| @@ -41,7 +41,7 @@ CONFIG_FIXED_PHY=y · 40 unmodified lines | ||
| 41 | 41 | CONFIG_FONT_8x16=y |
| 42 | 42 | CONFIG_FONT_8x8=y |
| 43 | 43 | CONFIG_FONTS=y |
| 44 | −CONFIG_FORCE_MAX_ZONEORDER=13 | |
| 44 | +CONFIG_ARCH_FORCE_MAX_ORDER=13 | |
| 45 | 45 | CONFIG_FRAMEBUFFER_CONSOLE=y |
| 46 | 46 | CONFIG_FRAME_WARN=1024 |
| 47 | 47 | CONFIG_FTL=y |
| @@ -17,7 +17,6 @@ CONFIG_LOG_CPU_MAX_BUF_SHIFT=13 · 16 unmodified lines | ||
| 17 | 17 | CONFIG_NUMA_BALANCING=y |
| 18 | 18 | CONFIG_CGROUPS=y |
| 19 | 19 | CONFIG_MEMCG=y |
| 20 | −CONFIG_MEMCG_SWAP=y | |
| 21 | 20 | CONFIG_CGROUP_SCHED=y |
| 22 | 21 | CONFIG_CGROUP_FREEZER=y |
| 23 | 22 | CONFIG_CPUSETS=y |
| @@ -18,7 +18,6 @@ CONFIG_LOG_CPU_MAX_BUF_SHIFT=13 · 17 unmodified lines | ||
| 18 | 18 | CONFIG_NUMA_BALANCING=y |
| 19 | 19 | CONFIG_CGROUPS=y |
| 20 | 20 | CONFIG_MEMCG=y |
| 21 | −CONFIG_MEMCG_SWAP=y | |
| 22 | 21 | CONFIG_CGROUP_SCHED=y |
| 23 | 22 | CONFIG_CGROUP_FREEZER=y |
| 24 | 23 | CONFIG_CPUSETS=y |
| @@ -115,18 +115,18 @@ struct vdso_data *arch_get_vdso_data(void *vvar_page) · 114 unmodified lines | ||
| 115 | 115 | int vdso_join_timens(struct task_struct *task, struct time_namespace *ns) |
| 116 | 116 | { |
| 117 | 117 | struct mm_struct *mm = task->mm; |
| 118 | + VMA_ITERATOR(vmi, mm, 0); | |
| 118 | 119 | struct vm_area_struct *vma; |
| 119 | 120 | |
| 120 | 121 | mmap_read_lock(mm); |
| 121 | − | |
| 122 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 122 | + for_each_vma(vmi, vma) { | |
| 123 | 123 | unsigned long size = vma->vm_end - vma->vm_start; |
| 124 | 124 | |
| 125 | 125 | if (vma_is_special_mapping(vma, &vvar_spec)) |
| 126 | 126 | zap_page_range(vma, vma->vm_start, size); |
| 127 | 127 | } |
| 128 | − | |
| 129 | 128 | mmap_read_unlock(mm); |
| 129 | + | |
| 130 | 130 | return 0; |
| 131 | 131 | } |
| 132 | 132 | |
| @@ -81,14 +81,15 @@ EXPORT_SYMBOL(hash__flush_range); · 80 unmodified lines | ||
| 81 | 81 | void hash__flush_tlb_mm(struct mm_struct *mm) |
| 82 | 82 | { |
| 83 | 83 | struct vm_area_struct *mp; |
| 84 | + VMA_ITERATOR(vmi, mm, 0); | |
| 84 | 85 | |
| 85 | 86 | /* |
| 86 | − * It is safe to go down the mm's list of vmas when called | |
| 87 | − * from dup_mmap, holding mmap_lock. It would also be safe from | |
| 88 | − * unmap_region or exit_mmap, but not from vmtruncate on SMP - | |
| 89 | − * but it seems dup_mmap is the only SMP case which gets here. | |
| 87 | + * It is safe to iterate the vmas when called from dup_mmap, | |
| 88 | + * holding mmap_lock. It would also be safe from unmap_region | |
| 89 | + * or exit_mmap, but not from vmtruncate on SMP - but it seems | |
| 90 | + * dup_mmap is the only SMP case which gets here. | |
| 90 | 91 | */ |
| 91 | − for (mp = mm->mmap; mp != NULL; mp = mp->vm_next) | |
| 92 | + for_each_vma(vmi, mp) | |
| 92 | 93 | hash__flush_range(mp->vm_mm, mp->vm_start, mp->vm_end); |
| 93 | 94 | } |
| 94 | 95 | EXPORT_SYMBOL(hash__flush_tlb_mm); |
| @@ -149,24 +149,15 @@ static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr, · 148 unmodified lines | ||
| 149 | 149 | unsigned long len) |
| 150 | 150 | { |
| 151 | 151 | struct vm_area_struct *vma; |
| 152 | + VMA_ITERATOR(vmi, mm, addr); | |
| 152 | 153 | |
| 153 | 154 | /* |
| 154 | 155 | * We don't try too hard, we just mark all the vma in that range |
| 155 | 156 | * VM_NOHUGEPAGE and split them. |
| 156 | 157 | */ |
| 157 | − vma = find_vma(mm, addr); | |
| 158 | − /* | |
| 159 | − * If the range is in unmapped range, just return | |
| 160 | − */ | |
| 161 | − if (vma && ((addr + len) <= vma->vm_start)) | |
| 162 | − return; | |
| 163 | − | |
| 164 | − while (vma) { | |
| 165 | − if (vma->vm_start >= (addr + len)) | |
| 166 | − break; | |
| 158 | + for_each_vma_range(vmi, vma, addr + len) { | |
| 167 | 159 | vma->vm_flags |= VM_NOHUGEPAGE; |
| 168 | 160 | walk_page_vma(vma, &subpage_walk_ops, NULL); |
| 169 | − vma = vma->vm_next; | |
| 170 | 161 | } |
| 171 | 162 | } |
| 172 | 163 | #else |
| @@ -114,11 +114,12 @@ int vdso_join_timens(struct task_struct *task, struct time_namespace *ns) · 113 unmodified lines | ||
| 114 | 114 | { |
| 115 | 115 | struct mm_struct *mm = task->mm; |
| 116 | 116 | struct vm_area_struct *vma; |
| 117 | + VMA_ITERATOR(vmi, mm, 0); | |
| 117 | 118 | struct __vdso_info *vdso_info = mm->context.vdso_info; |
| 118 | 119 | |
| 119 | 120 | mmap_read_lock(mm); |
| 120 | 121 | |
| 121 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 122 | + for_each_vma(vmi, vma) { | |
| 122 | 123 | unsigned long size = vma->vm_end - vma->vm_start; |
| 123 | 124 | |
| 124 | 125 | if (vma_is_special_mapping(vma, vdso_info->dm)) |
| @@ -69,10 +69,11 @@ static struct page *find_timens_vvar_page(struct vm_area_struct *vma) · 68 unmodified lines | ||
| 69 | 69 | int vdso_join_timens(struct task_struct *task, struct time_namespace *ns) |
| 70 | 70 | { |
| 71 | 71 | struct mm_struct *mm = task->mm; |
| 72 | + VMA_ITERATOR(vmi, mm, 0); | |
| 72 | 73 | struct vm_area_struct *vma; |
| 73 | 74 | |
| 74 | 75 | mmap_read_lock(mm); |
| 75 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 76 | + for_each_vma(vmi, vma) { | |
| 76 | 77 | unsigned long size = vma->vm_end - vma->vm_start; |
| 77 | 78 | |
| 78 | 79 | if (!vma_is_special_mapping(vma, &vvar_mapping)) |
| @@ -81,8 +81,9 @@ unsigned long _copy_from_user_key(void *to, const void __user *from, · 80 unmodified lines | ||
| 81 | 81 | |
| 82 | 82 | might_fault(); |
| 83 | 83 | if (!should_fail_usercopy()) { |
| 84 | − instrument_copy_from_user(to, from, n); | |
| 84 | + instrument_copy_from_user_before(to, from, n); | |
| 85 | 85 | res = raw_copy_from_user_key(to, from, n, key); |
| 86 | + instrument_copy_from_user_after(to, from, n, res); | |
| 86 | 87 | } |
| 87 | 88 | if (unlikely(res)) |
| 88 | 89 | memset(to + (n - res), 0, res); |
| @@ -2515,8 +2515,9 @@ static const struct mm_walk_ops thp_split_walk_ops = { · 2514 unmodified lines | ||
| 2515 | 2515 | static inline void thp_split_mm(struct mm_struct *mm) |
| 2516 | 2516 | { |
| 2517 | 2517 | struct vm_area_struct *vma; |
| 2518 | + VMA_ITERATOR(vmi, mm, 0); | |
| 2518 | 2519 | |
| 2519 | − for (vma = mm->mmap; vma != NULL; vma = vma->vm_next) { | |
| 2520 | + for_each_vma(vmi, vma) { | |
| 2520 | 2521 | vma->vm_flags &= ~VM_HUGEPAGE; |
| 2521 | 2522 | vma->vm_flags |= VM_NOHUGEPAGE; |
| 2522 | 2523 | walk_page_vma(vma, &thp_split_walk_ops, NULL); |
| @@ -2584,8 +2585,9 @@ int gmap_mark_unmergeable(void) · 61 unmodified lines | ||
| 2584 | 2585 | struct mm_struct *mm = current->mm; |
| 2585 | 2586 | struct vm_area_struct *vma; |
| 2586 | 2587 | int ret; |
| 2588 | + VMA_ITERATOR(vmi, mm, 0); | |
| 2587 | 2589 | |
| 2588 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 2590 | + for_each_vma(vmi, vma) { | |
| 2589 | 2591 | ret = ksm_madvise(vma, vma->vm_start, vma->vm_end, |
| 2590 | 2592 | MADV_UNMERGEABLE, &vma->vm_flags); |
| 2591 | 2593 | if (ret) |
| @@ -237,16 +237,6 @@ int pud_huge(pud_t pud) · 236 unmodified lines | ||
| 237 | 237 | return pud_large(pud); |
| 238 | 238 | } |
| 239 | 239 | |
| 240 | −struct page * | |
| 241 | −follow_huge_pud(struct mm_struct *mm, unsigned long address, | |
| 242 | − pud_t *pud, int flags) | |
| 243 | −{ | |
| 244 | − if (flags & FOLL_GET) | |
| 245 | − return NULL; | |
| 246 | − | |
| 247 | − return pud_page(*pud) + ((address & ~PUD_MASK) >> PAGE_SHIFT); | |
| 248 | −} | |
| 249 | − | |
| 250 | 240 | bool __init arch_hugetlb_valid_size(unsigned long size) |
| 251 | 241 | { |
| 252 | 242 | if (MACHINE_HAS_EDAT1 && size == PMD_SIZE) |
| @@ -8,7 +8,7 @@ CONFIG_MODULES=y · 7 unmodified lines | ||
| 8 | 8 | CONFIG_MODULE_UNLOAD=y |
| 9 | 9 | # CONFIG_BLK_DEV_BSG is not set |
| 10 | 10 | CONFIG_CPU_SUBTYPE_SH7724=y |
| 11 | −CONFIG_FORCE_MAX_ZONEORDER=12 | |
| 11 | +CONFIG_ARCH_FORCE_MAX_ORDER=12 | |
| 12 | 12 | CONFIG_MEMORY_SIZE=0x10000000 |
| 13 | 13 | CONFIG_FLATMEM_MANUAL=y |
| 14 | 14 | CONFIG_SH_ECOVEC=y |
| @@ -16,7 +16,6 @@ CONFIG_CPUSETS=y · 15 unmodified lines | ||
| 16 | 16 | # CONFIG_PROC_PID_CPUSET is not set |
| 17 | 17 | CONFIG_CGROUP_CPUACCT=y |
| 18 | 18 | CONFIG_CGROUP_MEMCG=y |
| 19 | −CONFIG_CGROUP_MEMCG_SWAP=y | |
| 20 | 19 | CONFIG_CGROUP_SCHED=y |
| 21 | 20 | CONFIG_RT_GROUP_SCHED=y |
| 22 | 21 | CONFIG_BLK_CGROUP=y |
| @@ -14,7 +14,6 @@ CONFIG_CPUSETS=y · 13 unmodified lines | ||
| 14 | 14 | # CONFIG_PROC_PID_CPUSET is not set |
| 15 | 15 | CONFIG_CGROUP_CPUACCT=y |
| 16 | 16 | CONFIG_CGROUP_MEMCG=y |
| 17 | −CONFIG_CGROUP_MEMCG_SWAP=y | |
| 18 | 17 | CONFIG_CGROUP_SCHED=y |
| 19 | 18 | CONFIG_RT_GROUP_SCHED=y |
| 20 | 19 | CONFIG_BLK_DEV_INITRD=y |
| @@ -18,7 +18,7 @@ config PAGE_OFFSET · 17 unmodified lines | ||
| 18 | 18 | default "0x80000000" if MMU |
| 19 | 19 | default "0x00000000" |
| 20 | 20 | |
| 21 | −config FORCE_MAX_ZONEORDER | |
| 21 | +config ARCH_FORCE_MAX_ORDER | |
| 22 | 22 | int "Maximum zone order" |
| 23 | 23 | range 9 64 if PAGE_SIZE_16KB |
| 24 | 24 | default "9" if PAGE_SIZE_16KB |
| @@ -269,7 +269,7 @@ config ARCH_SPARSEMEM_ENABLE · 268 unmodified lines | ||
| 269 | 269 | config ARCH_SPARSEMEM_DEFAULT |
| 270 | 270 | def_bool y if SPARC64 |
| 271 | 271 | |
| 272 | −config FORCE_MAX_ZONEORDER | |
| 272 | +config ARCH_FORCE_MAX_ORDER | |
| 273 | 273 | int "Maximum zone order" |
| 274 | 274 | default "13" |
| 275 | 275 | help |
| @@ -584,21 +584,19 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, · 583 unmodified lines | ||
| 584 | 584 | |
| 585 | 585 | void flush_tlb_mm(struct mm_struct *mm) |
| 586 | 586 | { |
| 587 | − struct vm_area_struct *vma = mm->mmap; | |
| 587 | + struct vm_area_struct *vma; | |
| 588 | + VMA_ITERATOR(vmi, mm, 0); | |
| 588 | 589 | |
| 589 | − while (vma != NULL) { | |
| 590 | + for_each_vma(vmi, vma) | |
| 590 | 591 | fix_range(mm, vma->vm_start, vma->vm_end, 0); |
| 591 | − vma = vma->vm_next; | |
| 592 | − } | |
| 593 | 592 | } |
| 594 | 593 | |
| 595 | 594 | void force_flush_all(void) |
| 596 | 595 | { |
| 597 | 596 | struct mm_struct *mm = current->mm; |
| 598 | − struct vm_area_struct *vma = mm->mmap; | |
| 597 | + struct vm_area_struct *vma; | |
| 598 | + VMA_ITERATOR(vmi, mm, 0); | |
| 599 | 599 | |
| 600 | − while (vma != NULL) { | |
| 600 | + for_each_vma(vmi, vma) | |
| 601 | 601 | fix_range(mm, vma->vm_start, vma->vm_end, 1); |
| 602 | − vma = vma->vm_next; | |
| 603 | − } | |
| 604 | 602 | } |
| @@ -85,6 +85,7 @@ config X86 · 84 unmodified lines | ||
| 85 | 85 | select ARCH_HAS_PMEM_API if X86_64 |
| 86 | 86 | select ARCH_HAS_PTE_DEVMAP if X86_64 |
| 87 | 87 | select ARCH_HAS_PTE_SPECIAL |
| 88 | + select ARCH_HAS_NONLEAF_PMD_YOUNG if PGTABLE_LEVELS > 2 | |
| 88 | 89 | select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 |
| 89 | 90 | select ARCH_HAS_COPY_MC if X86_64 |
| 90 | 91 | select ARCH_HAS_SET_MEMORY |
| @@ -130,7 +131,9 @@ config X86 · 39 unmodified lines | ||
| 130 | 131 | select CLKEVT_I8253 |
| 131 | 132 | select CLOCKSOURCE_VALIDATE_LAST_CYCLE |
| 132 | 133 | select CLOCKSOURCE_WATCHDOG |
| 133 | − select DCACHE_WORD_ACCESS | |
| 134 | + # Word-size accesses may read uninitialized data past the trailing \0 | |
| 135 | + # in strings and cause false KMSAN reports. | |
| 136 | + select DCACHE_WORD_ACCESS if !KMSAN | |
| 134 | 137 | select DYNAMIC_SIGFRAME |
| 135 | 138 | select EDAC_ATOMIC_SCRUB |
| 136 | 139 | select EDAC_SUPPORT |
| @@ -168,6 +171,7 @@ config X86 · 31 unmodified lines | ||
| 168 | 171 | select HAVE_ARCH_KASAN if X86_64 |
| 169 | 172 | select HAVE_ARCH_KASAN_VMALLOC if X86_64 |
| 170 | 173 | select HAVE_ARCH_KFENCE |
| 174 | + select HAVE_ARCH_KMSAN if X86_64 | |
| 171 | 175 | select HAVE_ARCH_KGDB |
| 172 | 176 | select HAVE_ARCH_MMAP_RND_BITS if MMU |
| 173 | 177 | select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT |
| @@ -328,6 +332,10 @@ config GENERIC_ISA_DMA · 154 unmodified lines | ||
| 328 | 332 | def_bool y |
| 329 | 333 | depends on ISA_DMA_API |
| 330 | 334 | |
| 335 | +config GENERIC_CSUM | |
| 336 | + bool | |
| 337 | + default y if KMSAN || KASAN | |
| 338 | + | |
| 331 | 339 | config GENERIC_BUG |
| 332 | 340 | def_bool y |
| 333 | 341 | depends on BUG |
| @@ -12,6 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | # Sanitizer runtimes are unavailable and cannot be linked for early boot code. |
| 13 | 13 | KASAN_SANITIZE := n |
| 14 | 14 | KCSAN_SANITIZE := n |
| 15 | +KMSAN_SANITIZE := n | |
| 15 | 16 | OBJECT_FILES_NON_STANDARD := y |
| 16 | 17 | |
| 17 | 18 | # Kernel does not boot with kcov instrumentation here. |
| @@ -20,6 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | # Sanitizer runtimes are unavailable and cannot be linked for early boot code. |
| 21 | 21 | KASAN_SANITIZE := n |
| 22 | 22 | KCSAN_SANITIZE := n |
| 23 | +KMSAN_SANITIZE := n | |
| 23 | 24 | OBJECT_FILES_NON_STANDARD := y |
| 24 | 25 | |
| 25 | 26 | # Prevents link failures: __sanitizer_cov_trace_pc() is not linked in. |
| @@ -11,6 +11,9 @@ include $(srctree)/lib/vdso/Makefile · 10 unmodified lines | ||
| 11 | 11 | |
| 12 | 12 | # Sanitizer runtimes are unavailable and cannot be linked here. |
| 13 | 13 | KASAN_SANITIZE := n |
| 14 | +KMSAN_SANITIZE_vclock_gettime.o := n | |
| 15 | +KMSAN_SANITIZE_vgetcpu.o := n | |
| 16 | + | |
| 14 | 17 | UBSAN_SANITIZE := n |
| 15 | 18 | KCSAN_SANITIZE := n |
| 16 | 19 | OBJECT_FILES_NON_STANDARD := y |
| @@ -127,17 +127,17 @@ int vdso_join_timens(struct task_struct *task, struct time_namespace *ns) · 126 unmodified lines | ||
| 127 | 127 | { |
| 128 | 128 | struct mm_struct *mm = task->mm; |
| 129 | 129 | struct vm_area_struct *vma; |
| 130 | + VMA_ITERATOR(vmi, mm, 0); | |
| 130 | 131 | |
| 131 | 132 | mmap_read_lock(mm); |
| 132 | − | |
| 133 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 133 | + for_each_vma(vmi, vma) { | |
| 134 | 134 | unsigned long size = vma->vm_end - vma->vm_start; |
| 135 | 135 | |
| 136 | 136 | if (vma_is_special_mapping(vma, &vvar_mapping)) |
| 137 | 137 | zap_page_range(vma, vma->vm_start, size); |
| 138 | 138 | } |
| 139 | − | |
| 140 | 139 | mmap_read_unlock(mm); |
| 140 | + | |
| 141 | 141 | return 0; |
| 142 | 142 | } |
| 143 | 143 | #else |
| @@ -354,6 +354,7 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr) · 210 unmodified lines | ||
| 354 | 354 | { |
| 355 | 355 | struct mm_struct *mm = current->mm; |
| 356 | 356 | struct vm_area_struct *vma; |
| 357 | + VMA_ITERATOR(vmi, mm, 0); | |
| 357 | 358 | |
| 358 | 359 | mmap_write_lock(mm); |
| 359 | 360 | /* |
| @@ -363,7 +364,7 @@ int map_vdso_once(const struct vdso_image *image, unsigned long addr) · 3 unmodified lines | ||
| 363 | 364 | * We could search vma near context.vdso, but it's a slowpath, |
| 364 | 365 | * so let's explicitly check all VMAs to be completely sure. |
| 365 | 366 | */ |
| 366 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 367 | + for_each_vma(vmi, vma) { | |
| 367 | 368 | if (vma_is_special_mapping(vma, &vdso_mapping) || |
| 368 | 369 | vma_is_special_mapping(vma, &vvar_mapping)) { |
| 369 | 370 | mmap_write_unlock(mm); |
| @@ -1,9 +1,13 @@ | ||
| 1 | 1 | /* SPDX-License-Identifier: GPL-2.0 */ |
| 2 | −#define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER 1 | |
| 3 | −#define HAVE_CSUM_COPY_USER | |
| 4 | −#define _HAVE_ARCH_CSUM_AND_COPY | |
| 5 | −#ifdef CONFIG_X86_32 | |
| 6 | −# include <asm/checksum_32.h> | |
| 2 | +#ifdef CONFIG_GENERIC_CSUM | |
| 3 | +# include <asm-generic/checksum.h> | |
| 7 | 4 | #else |
| 8 | −# include <asm/checksum_64.h> | |
| 5 | +# define _HAVE_ARCH_COPY_AND_CSUM_FROM_USER 1 | |
| 6 | +# define HAVE_CSUM_COPY_USER | |
| 7 | +# define _HAVE_ARCH_CSUM_AND_COPY | |
| 8 | +# ifdef CONFIG_X86_32 | |
| 9 | +# include <asm/checksum_32.h> | |
| 10 | +# else | |
| 11 | +# include <asm/checksum_64.h> | |
| 12 | +# endif | |
| 9 | 13 | #endif |
| @@ -0,0 +1,87 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +/* | |
| 3 | + * x86 KMSAN support. | |
| 4 | + * | |
| 5 | + * Copyright (C) 2022, Google LLC | |
| 6 | + * Author: Alexander Potapenko <glider@google.com> | |
| 7 | + */ | |
| 8 | + | |
| 9 | +#ifndef _ASM_X86_KMSAN_H | |
| 10 | +#define _ASM_X86_KMSAN_H | |
| 11 | + | |
| 12 | +#ifndef MODULE | |
| 13 | + | |
| 14 | +#include <asm/cpu_entry_area.h> | |
| 15 | +#include <asm/processor.h> | |
| 16 | +#include <linux/mmzone.h> | |
| 17 | + | |
| 18 | +DECLARE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_shadow); | |
| 19 | +DECLARE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_origin); | |
| 20 | + | |
| 21 | +/* | |
| 22 | + * Functions below are declared in the header to make sure they are inlined. | |
| 23 | + * They all are called from kmsan_get_metadata() for every memory access in | |
| 24 | + * the kernel, so speed is important here. | |
| 25 | + */ | |
| 26 | + | |
| 27 | +/* | |
| 28 | + * Compute metadata addresses for the CPU entry area on x86. | |
| 29 | + */ | |
| 30 | +static inline void *arch_kmsan_get_meta_or_null(void *addr, bool is_origin) | |
| 31 | +{ | |
| 32 | + unsigned long addr64 = (unsigned long)addr; | |
| 33 | + char *metadata_array; | |
| 34 | + unsigned long off; | |
| 35 | + int cpu; | |
| 36 | + | |
| 37 | + if ((addr64 < CPU_ENTRY_AREA_BASE) || | |
| 38 | + (addr64 >= (CPU_ENTRY_AREA_BASE + CPU_ENTRY_AREA_MAP_SIZE))) | |
| 39 | + return NULL; | |
| 40 | + cpu = (addr64 - CPU_ENTRY_AREA_BASE) / CPU_ENTRY_AREA_SIZE; | |
| 41 | + off = addr64 - (unsigned long)get_cpu_entry_area(cpu); | |
| 42 | + if ((off < 0) || (off >= CPU_ENTRY_AREA_SIZE)) | |
| 43 | + return NULL; | |
| 44 | + metadata_array = is_origin ? cpu_entry_area_origin : | |
| 45 | + cpu_entry_area_shadow; | |
| 46 | + return &per_cpu(metadata_array[off], cpu); | |
| 47 | +} | |
| 48 | + | |
| 49 | +/* | |
| 50 | + * Taken from arch/x86/mm/physaddr.h to avoid using an instrumented version. | |
| 51 | + */ | |
| 52 | +static inline bool kmsan_phys_addr_valid(unsigned long addr) | |
| 53 | +{ | |
| 54 | + if (IS_ENABLED(CONFIG_PHYS_ADDR_T_64BIT)) | |
| 55 | + return !(addr >> boot_cpu_data.x86_phys_bits); | |
| 56 | + else | |
| 57 | + return true; | |
| 58 | +} | |
| 59 | + | |
| 60 | +/* | |
| 61 | + * Taken from arch/x86/mm/physaddr.c to avoid using an instrumented version. | |
| 62 | + */ | |
| 63 | +static inline bool kmsan_virt_addr_valid(void *addr) | |
| 64 | +{ | |
| 65 | + unsigned long x = (unsigned long)addr; | |
| 66 | + unsigned long y = x - __START_KERNEL_map; | |
| 67 | + | |
| 68 | + /* use the carry flag to determine if x was < __START_KERNEL_map */ | |
| 69 | + if (unlikely(x > y)) { | |
| 70 | + x = y + phys_base; | |
| 71 | + | |
| 72 | + if (y >= KERNEL_IMAGE_SIZE) | |
| 73 | + return false; | |
| 74 | + } else { | |
| 75 | + x = y + (__START_KERNEL_map - PAGE_OFFSET); | |
| 76 | + | |
| 77 | + /* carry flag will be set if starting x was >= PAGE_OFFSET */ | |
| 78 | + if ((x > y) || !kmsan_phys_addr_valid(x)) | |
| 79 | + return false; | |
| 80 | + } | |
| 81 | + | |
| 82 | + return pfn_valid(x >> PAGE_SHIFT); | |
| 83 | +} | |
| 84 | + | |
| 85 | +#endif /* !MODULE */ | |
| 86 | + | |
| 87 | +#endif /* _ASM_X86_KMSAN_H */ | |
| @@ -8,6 +8,8 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <asm/cpufeatures.h> |
| 9 | 9 | #include <asm/alternative.h> |
| 10 | 10 | |
| 11 | +#include <linux/kmsan-checks.h> | |
| 12 | + | |
| 11 | 13 | /* duplicated to the one in bootmem.h */ |
| 12 | 14 | extern unsigned long max_pfn; |
| 13 | 15 | extern unsigned long phys_base; |
| @@ -47,6 +49,11 @@ void clear_page_erms(void *page); · 33 unmodified lines | ||
| 47 | 49 | |
| 48 | 50 | static inline void clear_page(void *page) |
| 49 | 51 | { |
| 52 | + /* | |
| 53 | + * Clean up KMSAN metadata for the page being cleared. The assembly call | |
| 54 | + * below clobbers @page, so we perform unpoisoning before it. | |
| 55 | + */ | |
| 56 | + kmsan_unpoison_memory(page, PAGE_SIZE); | |
| 50 | 57 | alternative_call_2(clear_page_orig, |
| 51 | 58 | clear_page_rep, X86_FEATURE_REP_GOOD, |
| 52 | 59 | clear_page_erms, X86_FEATURE_ERMS, |
| @@ -256,10 +256,10 @@ static inline pud_t native_pudp_get_and_clear(pud_t *pudp) · 255 unmodified lines | ||
| 256 | 256 | /* We always extract/encode the offset by shifting it all the way up, and then down again */ |
| 257 | 257 | #define SWP_OFFSET_SHIFT (SWP_OFFSET_FIRST_BIT + SWP_TYPE_BITS) |
| 258 | 258 | |
| 259 | −#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > 5) | |
| 260 | −#define __swp_type(x) (((x).val) & 0x1f) | |
| 261 | −#define __swp_offset(x) ((x).val >> 5) | |
| 262 | −#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) << 5}) | |
| 259 | +#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS) | |
| 260 | +#define __swp_type(x) (((x).val) & ((1UL << SWP_TYPE_BITS) - 1)) | |
| 261 | +#define __swp_offset(x) ((x).val >> SWP_TYPE_BITS) | |
| 262 | +#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) << SWP_TYPE_BITS}) | |
| 263 | 263 | |
| 264 | 264 | /* |
| 265 | 265 | * Normally, __swp_entry() converts from arch-independent swp_entry_t to |
| @@ -815,7 +815,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) · 814 unmodified lines | ||
| 815 | 815 | |
| 816 | 816 | static inline int pmd_bad(pmd_t pmd) |
| 817 | 817 | { |
| 818 | − return (pmd_flags(pmd) & ~_PAGE_USER) != _KERNPG_TABLE; | |
| 818 | + return (pmd_flags(pmd) & ~(_PAGE_USER | _PAGE_ACCESSED)) != | |
| 819 | + (_KERNPG_TABLE & ~_PAGE_ACCESSED); | |
| 819 | 820 | } |
| 820 | 821 | |
| 821 | 822 | static inline unsigned long pages_to_mb(unsigned long npg) |
| @@ -1431,10 +1432,10 @@ static inline bool arch_has_pfn_modify_check(void) · 609 unmodified lines | ||
| 1431 | 1432 | return boot_cpu_has_bug(X86_BUG_L1TF); |
| 1432 | 1433 | } |
| 1433 | 1434 | |
| 1434 | −#define arch_faults_on_old_pte arch_faults_on_old_pte | |
| 1435 | −static inline bool arch_faults_on_old_pte(void) | |
| 1435 | +#define arch_has_hw_pte_young arch_has_hw_pte_young | |
| 1436 | +static inline bool arch_has_hw_pte_young(void) | |
| 1436 | 1437 | { |
| 1437 | − return false; | |
| 1438 | + return true; | |
| 1438 | 1439 | } |
| 1439 | 1440 | |
| 1440 | 1441 | #ifdef CONFIG_PAGE_TABLE_CHECK |
| @@ -139,7 +139,52 @@ extern unsigned int ptrs_per_p4d; · 138 unmodified lines | ||
| 139 | 139 | # define VMEMMAP_START __VMEMMAP_BASE_L4 |
| 140 | 140 | #endif /* CONFIG_DYNAMIC_MEMORY_LAYOUT */ |
| 141 | 141 | |
| 142 | −#define VMALLOC_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) | |
| 142 | +/* | |
| 143 | + * End of the region for which vmalloc page tables are pre-allocated. | |
| 144 | + * For non-KMSAN builds, this is the same as VMALLOC_END. | |
| 145 | + * For KMSAN builds, VMALLOC_START..VMEMORY_END is 4 times bigger than | |
| 146 | + * VMALLOC_START..VMALLOC_END (see below). | |
| 147 | + */ | |
| 148 | +#define VMEMORY_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) | |
| 149 | + | |
| 150 | +#ifndef CONFIG_KMSAN | |
| 151 | +#define VMALLOC_END VMEMORY_END | |
| 152 | +#else | |
| 153 | +/* | |
| 154 | + * In KMSAN builds vmalloc area is four times smaller, and the remaining 3/4 | |
| 155 | + * are used to keep the metadata for virtual pages. The memory formerly | |
| 156 | + * belonging to vmalloc area is now laid out as follows: | |
| 157 | + * | |
| 158 | + * 1st quarter: VMALLOC_START to VMALLOC_END - new vmalloc area | |
| 159 | + * 2nd quarter: KMSAN_VMALLOC_SHADOW_START to | |
| 160 | + * VMALLOC_END+KMSAN_VMALLOC_SHADOW_OFFSET - vmalloc area shadow | |
| 161 | + * 3rd quarter: KMSAN_VMALLOC_ORIGIN_START to | |
| 162 | + * VMALLOC_END+KMSAN_VMALLOC_ORIGIN_OFFSET - vmalloc area origins | |
| 163 | + * 4th quarter: KMSAN_MODULES_SHADOW_START to KMSAN_MODULES_ORIGIN_START | |
| 164 | + * - shadow for modules, | |
| 165 | + * KMSAN_MODULES_ORIGIN_START to | |
| 166 | + * KMSAN_MODULES_ORIGIN_START + MODULES_LEN - origins for modules. | |
| 167 | + */ | |
| 168 | +#define VMALLOC_QUARTER_SIZE ((VMALLOC_SIZE_TB << 40) >> 2) | |
| 169 | +#define VMALLOC_END (VMALLOC_START + VMALLOC_QUARTER_SIZE - 1) | |
| 170 | + | |
| 171 | +/* | |
| 172 | + * vmalloc metadata addresses are calculated by adding shadow/origin offsets | |
| 173 | + * to vmalloc address. | |
| 174 | + */ | |
| 175 | +#define KMSAN_VMALLOC_SHADOW_OFFSET VMALLOC_QUARTER_SIZE | |
| 176 | +#define KMSAN_VMALLOC_ORIGIN_OFFSET (VMALLOC_QUARTER_SIZE << 1) | |
| 177 | + | |
| 178 | +#define KMSAN_VMALLOC_SHADOW_START (VMALLOC_START + KMSAN_VMALLOC_SHADOW_OFFSET) | |
| 179 | +#define KMSAN_VMALLOC_ORIGIN_START (VMALLOC_START + KMSAN_VMALLOC_ORIGIN_OFFSET) | |
| 180 | + | |
| 181 | +/* | |
| 182 | + * The shadow/origin for modules are placed one by one in the last 1/4 of | |
| 183 | + * vmalloc space. | |
| 184 | + */ | |
| 185 | +#define KMSAN_MODULES_SHADOW_START (VMALLOC_END + KMSAN_VMALLOC_ORIGIN_OFFSET + 1) | |
| 186 | +#define KMSAN_MODULES_ORIGIN_START (KMSAN_MODULES_SHADOW_START + MODULES_LEN) | |
| 187 | +#endif /* CONFIG_KMSAN */ | |
| 143 | 188 | |
| 144 | 189 | #define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE) |
| 145 | 190 | /* The module sections ends with the start of the fixmap */ |
| @@ -2,6 +2,8 @@ · 1 unmodified line | ||
| 2 | 2 | #ifndef _ASM_X86_SPARSEMEM_H |
| 3 | 3 | #define _ASM_X86_SPARSEMEM_H |
| 4 | 4 | |
| 5 | +#include <linux/types.h> | |
| 6 | + | |
| 5 | 7 | #ifdef CONFIG_SPARSEMEM |
| 6 | 8 | /* |
| 7 | 9 | * generic non-linear memory support: |
| @@ -11,11 +11,23 @@ · 10 unmodified lines | ||
| 11 | 11 | function. */ |
| 12 | 12 | |
| 13 | 13 | #define __HAVE_ARCH_MEMCPY 1 |
| 14 | +#if defined(__SANITIZE_MEMORY__) | |
| 15 | +#undef memcpy | |
| 16 | +void *__msan_memcpy(void *dst, const void *src, size_t size); | |
| 17 | +#define memcpy __msan_memcpy | |
| 18 | +#else | |
| 14 | 19 | extern void *memcpy(void *to, const void *from, size_t len); |
| 20 | +#endif | |
| 15 | 21 | extern void *__memcpy(void *to, const void *from, size_t len); |
| 16 | 22 | |
| 17 | 23 | #define __HAVE_ARCH_MEMSET |
| 24 | +#if defined(__SANITIZE_MEMORY__) | |
| 25 | +extern void *__msan_memset(void *s, int c, size_t n); | |
| 26 | +#undef memset | |
| 27 | +#define memset __msan_memset | |
| 28 | +#else | |
| 18 | 29 | void *memset(void *s, int c, size_t n); |
| 30 | +#endif | |
| 19 | 31 | void *__memset(void *s, int c, size_t n); |
| 20 | 32 | |
| 21 | 33 | #define __HAVE_ARCH_MEMSET16 |
| @@ -55,7 +67,13 @@ static inline void *memset64(uint64_t *s, uint64_t v, size_t n) · 33 unmodified lines | ||
| 55 | 67 | } |
| 56 | 68 | |
| 57 | 69 | #define __HAVE_ARCH_MEMMOVE |
| 70 | +#if defined(__SANITIZE_MEMORY__) | |
| 71 | +#undef memmove | |
| 72 | +void *__msan_memmove(void *dest, const void *src, size_t len); | |
| 73 | +#define memmove __msan_memmove | |
| 74 | +#else | |
| 58 | 75 | void *memmove(void *dest, const void *src, size_t count); |
| 76 | +#endif | |
| 59 | 77 | void *__memmove(void *dest, const void *src, size_t count); |
| 60 | 78 | |
| 61 | 79 | int memcmp(const void *cs, const void *ct, size_t count); |
| @@ -64,8 +82,7 @@ char *strcpy(char *dest, const char *src); · 2 unmodified lines | ||
| 64 | 82 | char *strcat(char *dest, const char *src); |
| 65 | 83 | int strcmp(const char *cs, const char *ct); |
| 66 | 84 | |
| 67 | −#if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__) | |
| 68 | − | |
| 85 | +#if (defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__)) | |
| 69 | 86 | /* |
| 70 | 87 | * For files that not instrumented (e.g. mm/slub.c) we |
| 71 | 88 | * should use not instrumented version of mem* functions. |
| @@ -73,7 +90,9 @@ int strcmp(const char *cs, const char *ct); · 1 unmodified line | ||
| 73 | 90 | |
| 74 | 91 | #undef memcpy |
| 75 | 92 | #define memcpy(dst, src, len) __memcpy(dst, src, len) |
| 93 | +#undef memmove | |
| 76 | 94 | #define memmove(dst, src, len) __memmove(dst, src, len) |
| 95 | +#undef memset | |
| 77 | 96 | #define memset(s, c, n) __memset(s, c, n) |
| 78 | 97 | |
| 79 | 98 | #ifndef __NO_FORTIFY |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | * User space memory access functions |
| 6 | 6 | */ |
| 7 | 7 | #include <linux/compiler.h> |
| 8 | +#include <linux/instrumented.h> | |
| 8 | 9 | #include <linux/kasan-checks.h> |
| 9 | 10 | #include <linux/string.h> |
| 10 | 11 | #include <asm/asm.h> |
| @@ -103,6 +104,7 @@ extern int __get_user_bad(void); · 92 unmodified lines | ||
| 103 | 104 | : "=a" (__ret_gu), "=r" (__val_gu), \ |
| 104 | 105 | ASM_CALL_CONSTRAINT \ |
| 105 | 106 | : "0" (ptr), "i" (sizeof(*(ptr)))); \ |
| 107 | + instrument_get_user(__val_gu); \ | |
| 106 | 108 | (x) = (__force __typeof__(*(ptr))) __val_gu; \ |
| 107 | 109 | __builtin_expect(__ret_gu, 0); \ |
| 108 | 110 | }) |
| @@ -192,9 +194,11 @@ extern void __put_user_nocheck_8(void); · 83 unmodified lines | ||
| 192 | 194 | int __ret_pu; \ |
| 193 | 195 | void __user *__ptr_pu; \ |
| 194 | 196 | register __typeof__(*(ptr)) __val_pu asm("%"_ASM_AX); \ |
| 195 | − __chk_user_ptr(ptr); \ | |
| 196 | − __ptr_pu = (ptr); \ | |
| 197 | − __val_pu = (x); \ | |
| 197 | + __typeof__(*(ptr)) __x = (x); /* eval x once */ \ | |
| 198 | + __typeof__(ptr) __ptr = (ptr); /* eval ptr once */ \ | |
| 199 | + __chk_user_ptr(__ptr); \ | |
| 200 | + __ptr_pu = __ptr; \ | |
| 201 | + __val_pu = __x; \ | |
| 198 | 202 | asm volatile("call __" #fn "_%P[size]" \ |
| 199 | 203 | : "=c" (__ret_pu), \ |
| 200 | 204 | ASM_CALL_CONSTRAINT \ |
| @@ -202,6 +206,7 @@ extern void __put_user_nocheck_8(void); · 1 unmodified line | ||
| 202 | 206 | "r" (__val_pu), \ |
| 203 | 207 | [size] "i" (sizeof(*(ptr))) \ |
| 204 | 208 | :"ebx"); \ |
| 209 | + instrument_put_user(__x, __ptr, sizeof(*(ptr))); \ | |
| 205 | 210 | __builtin_expect(__ret_pu, 0); \ |
| 206 | 211 | }) |
| 207 | 212 | |
| @@ -248,23 +253,25 @@ extern void __put_user_nocheck_8(void); · 40 unmodified lines | ||
| 248 | 253 | |
| 249 | 254 | #define __put_user_size(x, ptr, size, label) \ |
| 250 | 255 | do { \ |
| 256 | + __typeof__(*(ptr)) __x = (x); /* eval x once */ \ | |
| 251 | 257 | __chk_user_ptr(ptr); \ |
| 252 | 258 | switch (size) { \ |
| 253 | 259 | case 1: \ |
| 254 | − __put_user_goto(x, ptr, "b", "iq", label); \ | |
| 260 | + __put_user_goto(__x, ptr, "b", "iq", label); \ | |
| 255 | 261 | break; \ |
| 256 | 262 | case 2: \ |
| 257 | − __put_user_goto(x, ptr, "w", "ir", label); \ | |
| 263 | + __put_user_goto(__x, ptr, "w", "ir", label); \ | |
| 258 | 264 | break; \ |
| 259 | 265 | case 4: \ |
| 260 | − __put_user_goto(x, ptr, "l", "ir", label); \ | |
| 266 | + __put_user_goto(__x, ptr, "l", "ir", label); \ | |
| 261 | 267 | break; \ |
| 262 | 268 | case 8: \ |
| 263 | − __put_user_goto_u64(x, ptr, label); \ | |
| 269 | + __put_user_goto_u64(__x, ptr, label); \ | |
| 264 | 270 | break; \ |
| 265 | 271 | default: \ |
| 266 | 272 | __put_user_bad(); \ |
| 267 | 273 | } \ |
| 274 | + instrument_put_user(__x, ptr, size); \ | |
| 268 | 275 | } while (0) |
| 269 | 276 | |
| 270 | 277 | #ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT |
| @@ -305,6 +312,7 @@ do { \ · 34 unmodified lines | ||
| 305 | 312 | default: \ |
| 306 | 313 | (x) = __get_user_bad(); \ |
| 307 | 314 | } \ |
| 315 | + instrument_get_user(x); \ | |
| 308 | 316 | } while (0) |
| 309 | 317 | |
| 310 | 318 | #define __get_user_asm(x, addr, itype, ltype, label) \ |
| @@ -29,6 +29,8 @@ KASAN_SANITIZE_sev.o := n · 28 unmodified lines | ||
| 29 | 29 | # With some compiler versions the generated code results in boot hangs, caused |
| 30 | 30 | # by several compilation units. To be safe, disable all instrumentation. |
| 31 | 31 | KCSAN_SANITIZE := n |
| 32 | +KMSAN_SANITIZE_head$(BITS).o := n | |
| 33 | +KMSAN_SANITIZE_nmi.o := n | |
| 32 | 34 | |
| 33 | 35 | # If instrumentation of this dir is enabled, boot hangs during first second. |
| 34 | 36 | # Probably could be more selective here, but note that files related to irqs, |
| @@ -12,6 +12,7 @@ endif · 11 unmodified lines | ||
| 12 | 12 | # If these files are instrumented, boot hangs during the first second. |
| 13 | 13 | KCOV_INSTRUMENT_common.o := n |
| 14 | 14 | KCOV_INSTRUMENT_perf_event.o := n |
| 15 | +KMSAN_SANITIZE_common.o := n | |
| 15 | 16 | |
| 16 | 17 | # As above, instrumenting secondary CPU boot code causes boot hangs. |
| 17 | 18 | KCSAN_SANITIZE_common.o := n |
| @@ -177,6 +177,12 @@ static void show_regs_if_on_stack(struct stack_info *info, struct pt_regs *regs, · 176 unmodified lines | ||
| 177 | 177 | } |
| 178 | 178 | } |
| 179 | 179 | |
| 180 | +/* | |
| 181 | + * This function reads pointers from the stack and dereferences them. The | |
| 182 | + * pointers may not have their KMSAN shadow set up properly, which may result | |
| 183 | + * in false positive reports. Disable instrumentation to avoid those. | |
| 184 | + */ | |
| 185 | +__no_kmsan_checks | |
| 180 | 186 | static void show_trace_log_lvl(struct task_struct *task, struct pt_regs *regs, |
| 181 | 187 | unsigned long *stack, const char *log_lvl) |
| 182 | 188 | { |
| @@ -553,6 +553,7 @@ void compat_start_thread(struct pt_regs *regs, u32 new_ip, u32 new_sp, bool x32) · 552 unmodified lines | ||
| 553 | 553 | * Kprobes not supported here. Set the probe on schedule instead. |
| 554 | 554 | * Function graph tracer not supported too. |
| 555 | 555 | */ |
| 556 | +__no_kmsan_checks | |
| 556 | 557 | __visible __notrace_funcgraph struct task_struct * |
| 557 | 558 | __switch_to(struct task_struct *prev_p, struct task_struct *next_p) |
| 558 | 559 | { |
| @@ -95,7 +95,7 @@ void __init tboot_probe(void) · 94 unmodified lines | ||
| 95 | 95 | |
| 96 | 96 | static pgd_t *tboot_pg_dir; |
| 97 | 97 | static struct mm_struct tboot_mm = { |
| 98 | − .mm_rb = RB_ROOT, | |
| 98 | + .mm_mt = MTREE_INIT_EXT(mm_mt, MM_MT_FLAGS, tboot_mm.mmap_lock), | |
| 99 | 99 | .pgd = swapper_pg_dir, |
| 100 | 100 | .mm_users = ATOMIC_INIT(2), |
| 101 | 101 | .mm_count = ATOMIC_INIT(1), |
| @@ -183,6 +183,16 @@ static struct pt_regs *decode_frame_pointer(unsigned long *bp) · 182 unmodified lines | ||
| 183 | 183 | } |
| 184 | 184 | #endif |
| 185 | 185 | |
| 186 | +/* | |
| 187 | + * While walking the stack, KMSAN may stomp on stale locals from other | |
| 188 | + * functions that were marked as uninitialized upon function exit, and | |
| 189 | + * now hold the call frame information for the current function (e.g. the frame | |
| 190 | + * pointer). Because KMSAN does not specifically mark call frames as | |
| 191 | + * initialized, false positive reports are possible. To prevent such reports, | |
| 192 | + * we mark the functions scanning the stack (here and below) with | |
| 193 | + * __no_kmsan_checks. | |
| 194 | + */ | |
| 195 | +__no_kmsan_checks | |
| 186 | 196 | static bool update_stack_state(struct unwind_state *state, |
| 187 | 197 | unsigned long *next_bp) |
| 188 | 198 | { |
| @@ -250,6 +260,7 @@ static bool update_stack_state(struct unwind_state *state, · 61 unmodified lines | ||
| 250 | 260 | return true; |
| 251 | 261 | } |
| 252 | 262 | |
| 263 | +__no_kmsan_checks | |
| 253 | 264 | bool unwind_next_frame(struct unwind_state *state) |
| 254 | 265 | { |
| 255 | 266 | struct pt_regs *regs; |
| @@ -65,7 +65,9 @@ ifneq ($(CONFIG_X86_CMPXCHG64),y) · 64 unmodified lines | ||
| 65 | 65 | endif |
| 66 | 66 | else |
| 67 | 67 | obj-y += iomap_copy_64.o |
| 68 | +ifneq ($(CONFIG_GENERIC_CSUM),y) | |
| 68 | 69 | lib-y += csum-partial_64.o csum-copy_64.o csum-wrappers_64.o |
| 70 | +endif | |
| 69 | 71 | lib-y += clear_page_64.o copy_page_64.o |
| 70 | 72 | lib-y += memmove_64.o memset_64.o |
| 71 | 73 | lib-y += copy_user_64.o |
| @@ -1,6 +1,7 @@ | ||
| 1 | 1 | #include <linux/string.h> |
| 2 | 2 | #include <linux/module.h> |
| 3 | 3 | #include <linux/io.h> |
| 4 | +#include <linux/kmsan-checks.h> | |
| 4 | 5 | |
| 5 | 6 | #define movs(type,to,from) \ |
| 6 | 7 | asm volatile("movs" type:"=&D" (to), "=&S" (from):"0" (to), "1" (from):"memory") |
| @@ -37,6 +38,8 @@ static void string_memcpy_fromio(void *to, const volatile void __iomem *from, si · 30 unmodified lines | ||
| 37 | 38 | n-=2; |
| 38 | 39 | } |
| 39 | 40 | rep_movs(to, (const void *)from, n); |
| 41 | + /* KMSAN must treat values read from devices as initialized. */ | |
| 42 | + kmsan_unpoison_memory(to, n); | |
| 40 | 43 | } |
| 41 | 44 | |
| 42 | 45 | static void string_memcpy_toio(volatile void __iomem *to, const void *from, size_t n) |
| @@ -44,6 +47,8 @@ static void string_memcpy_toio(volatile void __iomem *to, const void *from, size · 1 unmodified line | ||
| 44 | 47 | if (unlikely(!n)) |
| 45 | 48 | return; |
| 46 | 49 | |
| 50 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 51 | + kmsan_check_memory(from, n); | |
| 47 | 52 | /* Align any unaligned destination IO */ |
| 48 | 53 | if (unlikely(1 & (unsigned long)to)) { |
| 49 | 54 | movs("b", to, from); |
| @@ -14,6 +14,8 @@ KASAN_SANITIZE_pgprot.o := n · 13 unmodified lines | ||
| 14 | 14 | # Disable KCSAN entirely, because otherwise we get warnings that some functions |
| 15 | 15 | # reference __initdata sections. |
| 16 | 16 | KCSAN_SANITIZE := n |
| 17 | +# Avoid recursion by not calling KMSAN hooks for CEA code. | |
| 18 | +KMSAN_SANITIZE_cpu_entry_area.o := n | |
| 17 | 19 | |
| 18 | 20 | ifdef CONFIG_FUNCTION_TRACER |
| 19 | 21 | CFLAGS_REMOVE_mem_encrypt.o = -pg |
| @@ -44,6 +46,9 @@ obj-$(CONFIG_HIGHMEM) += highmem_32.o · 24 unmodified lines | ||
| 44 | 46 | KASAN_SANITIZE_kasan_init_$(BITS).o := n |
| 45 | 47 | obj-$(CONFIG_KASAN) += kasan_init_$(BITS).o |
| 46 | 48 | |
| 49 | +KMSAN_SANITIZE_kmsan_shadow.o := n | |
| 50 | +obj-$(CONFIG_KMSAN) += kmsan_shadow.o | |
| 51 | + | |
| 47 | 52 | obj-$(CONFIG_MMIOTRACE) += mmiotrace.o |
| 48 | 53 | mmiotrace-y := kmmio.o pf_in.o mmio-mod.o |
| 49 | 54 | obj-$(CONFIG_MMIOTRACE_TEST) += testmmiotrace.o |
| @@ -260,7 +260,7 @@ static noinline int vmalloc_fault(unsigned long address) · 259 unmodified lines | ||
| 260 | 260 | } |
| 261 | 261 | NOKPROBE_SYMBOL(vmalloc_fault); |
| 262 | 262 | |
| 263 | −void arch_sync_kernel_mappings(unsigned long start, unsigned long end) | |
| 263 | +static void __arch_sync_kernel_mappings(unsigned long start, unsigned long end) | |
| 264 | 264 | { |
| 265 | 265 | unsigned long addr; |
| 266 | 266 | |
| @@ -284,6 +284,27 @@ void arch_sync_kernel_mappings(unsigned long start, unsigned long end) · 17 unmodified lines | ||
| 284 | 284 | } |
| 285 | 285 | } |
| 286 | 286 | |
| 287 | +void arch_sync_kernel_mappings(unsigned long start, unsigned long end) | |
| 288 | +{ | |
| 289 | + __arch_sync_kernel_mappings(start, end); | |
| 290 | +#ifdef CONFIG_KMSAN | |
| 291 | + /* | |
| 292 | + * KMSAN maintains two additional metadata page mappings for the | |
| 293 | + * [VMALLOC_START, VMALLOC_END) range. These mappings start at | |
| 294 | + * KMSAN_VMALLOC_SHADOW_START and KMSAN_VMALLOC_ORIGIN_START and | |
| 295 | + * have to be synced together with the vmalloc memory mapping. | |
| 296 | + */ | |
| 297 | + if (start >= VMALLOC_START && end < VMALLOC_END) { | |
| 298 | + __arch_sync_kernel_mappings( | |
| 299 | + start - VMALLOC_START + KMSAN_VMALLOC_SHADOW_START, | |
| 300 | + end - VMALLOC_START + KMSAN_VMALLOC_SHADOW_START); | |
| 301 | + __arch_sync_kernel_mappings( | |
| 302 | + start - VMALLOC_START + KMSAN_VMALLOC_ORIGIN_START, | |
| 303 | + end - VMALLOC_START + KMSAN_VMALLOC_ORIGIN_START); | |
| 304 | + } | |
| 305 | +#endif | |
| 306 | +} | |
| 307 | + | |
| 287 | 308 | static bool low_pfn(unsigned long pfn) |
| 288 | 309 | { |
| 289 | 310 | return pfn < max_low_pfn; |
| @@ -1054,7 +1054,7 @@ void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache) · 1053 unmodified lines | ||
| 1054 | 1054 | } |
| 1055 | 1055 | |
| 1056 | 1056 | #ifdef CONFIG_SWAP |
| 1057 | −unsigned long max_swapfile_size(void) | |
| 1057 | +unsigned long arch_max_swapfile_size(void) | |
| 1058 | 1058 | { |
| 1059 | 1059 | unsigned long pages; |
| 1060 | 1060 | |
| @@ -1288,7 +1288,7 @@ static void __init preallocate_vmalloc_pages(void) · 1287 unmodified lines | ||
| 1288 | 1288 | unsigned long addr; |
| 1289 | 1289 | const char *lvl; |
| 1290 | 1290 | |
| 1291 | − for (addr = VMALLOC_START; addr <= VMALLOC_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { | |
| 1291 | + for (addr = VMALLOC_START; addr <= VMEMORY_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { | |
| 1292 | 1292 | pgd_t *pgd = pgd_offset_k(addr); |
| 1293 | 1293 | p4d_t *p4d; |
| 1294 | 1294 | pud_t *pud; |
| @@ -17,6 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <linux/cc_platform.h> |
| 18 | 18 | #include <linux/efi.h> |
| 19 | 19 | #include <linux/pgtable.h> |
| 20 | +#include <linux/kmsan.h> | |
| 20 | 21 | |
| 21 | 22 | #include <asm/set_memory.h> |
| 22 | 23 | #include <asm/e820/api.h> |
| @@ -479,6 +480,8 @@ void iounmap(volatile void __iomem *addr) · 456 unmodified lines | ||
| 479 | 480 | return; |
| 480 | 481 | } |
| 481 | 482 | |
| 483 | + kmsan_iounmap_page_range((unsigned long)addr, | |
| 484 | + (unsigned long)addr + get_vm_area_size(p)); | |
| 482 | 485 | memtype_free(p->phys_addr, p->phys_addr + get_vm_area_size(p)); |
| 483 | 486 | |
| 484 | 487 | /* Finally remove it */ |
| @@ -0,0 +1,20 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | +/* | |
| 3 | + * x86-specific bits of KMSAN shadow implementation. | |
| 4 | + * | |
| 5 | + * Copyright (C) 2022 Google LLC | |
| 6 | + * Author: Alexander Potapenko <glider@google.com> | |
| 7 | + */ | |
| 8 | + | |
| 9 | +#include <asm/cpu_entry_area.h> | |
| 10 | +#include <linux/percpu-defs.h> | |
| 11 | + | |
| 12 | +/* | |
| 13 | + * Addresses within the CPU entry area (including e.g. exception stacks) do not | |
| 14 | + * have struct page entries corresponding to them, so they need separate | |
| 15 | + * handling. | |
| 16 | + * arch_kmsan_get_meta_or_null() (declared in the header) maps the addresses in | |
| 17 | + * CPU entry area to addresses in cpu_entry_area_shadow/cpu_entry_area_origin. | |
| 18 | + */ | |
| 19 | +DEFINE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_shadow); | |
| 20 | +DEFINE_PER_CPU(char[CPU_ENTRY_AREA_SIZE], cpu_entry_area_origin); | |
| @@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma, · 549 unmodified lines | ||
| 550 | 550 | return ret; |
| 551 | 551 | } |
| 552 | 552 | |
| 553 | −#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 553 | +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) | |
| 554 | 554 | int pmdp_test_and_clear_young(struct vm_area_struct *vma, |
| 555 | 555 | unsigned long addr, pmd_t *pmdp) |
| 556 | 556 | { |
| @@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma, · 5 unmodified lines | ||
| 562 | 562 | |
| 563 | 563 | return ret; |
| 564 | 564 | } |
| 565 | +#endif | |
| 566 | + | |
| 567 | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 565 | 568 | int pudp_test_and_clear_young(struct vm_area_struct *vma, |
| 566 | 569 | unsigned long addr, pud_t *pudp) |
| 567 | 570 | { |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | # Sanitizer runtimes are unavailable and cannot be linked here. |
| 11 | 11 | KASAN_SANITIZE := n |
| 12 | 12 | KCSAN_SANITIZE := n |
| 13 | +KMSAN_SANITIZE := n | |
| 13 | 14 | OBJECT_FILES_NON_STANDARD := y |
| 14 | 15 | |
| 15 | 16 | # Prevents link failures: __sanitizer_cov_trace_pc() is not linked in. |
| @@ -771,7 +771,7 @@ config HIGHMEM · 770 unmodified lines | ||
| 771 | 771 | |
| 772 | 772 | If unsure, say Y. |
| 773 | 773 | |
| 774 | −config FORCE_MAX_ZONEORDER | |
| 774 | +config ARCH_FORCE_MAX_ORDER | |
| 775 | 775 | int "Maximum zone order" |
| 776 | 776 | default "11" |
| 777 | 777 | help |
| @@ -111,6 +111,8 @@ · 110 unmodified lines | ||
| 111 | 111 | |
| 112 | 112 | #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ |
| 113 | 113 | |
| 114 | +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ | |
| 115 | + | |
| 114 | 116 | /* compatibility flags */ |
| 115 | 117 | #define MAP_FILE 0 |
| 116 | 118 | |
| @@ -58,6 +58,7 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, · 57 unmodified lines | ||
| 58 | 58 | unsigned long len, unsigned long pgoff, unsigned long flags) |
| 59 | 59 | { |
| 60 | 60 | struct vm_area_struct *vmm; |
| 61 | + struct vma_iterator vmi; | |
| 61 | 62 | |
| 62 | 63 | if (flags & MAP_FIXED) { |
| 63 | 64 | /* We do not accept a shared mapping if it would violate |
| @@ -79,15 +80,20 @@ unsigned long arch_get_unmapped_area(struct file *filp, unsigned long addr, · 15 unmodified lines | ||
| 79 | 80 | else |
| 80 | 81 | addr = PAGE_ALIGN(addr); |
| 81 | 82 | |
| 82 | − for (vmm = find_vma(current->mm, addr); ; vmm = vmm->vm_next) { | |
| 83 | − /* At this point: (!vmm || addr < vmm->vm_end). */ | |
| 84 | − if (TASK_SIZE - len < addr) | |
| 85 | − return -ENOMEM; | |
| 86 | − if (!vmm || addr + len <= vm_start_gap(vmm)) | |
| 87 | − return addr; | |
| 83 | + vma_iter_init(&vmi, current->mm, addr); | |
| 84 | + for_each_vma(vmi, vmm) { | |
| 85 | + /* At this point: (addr < vmm->vm_end). */ | |
| 86 | + if (addr + len <= vm_start_gap(vmm)) | |
| 87 | + break; | |
| 88 | + | |
| 88 | 89 | addr = vmm->vm_end; |
| 89 | 90 | if (flags & MAP_SHARED) |
| 90 | 91 | addr = COLOUR_ALIGN(addr, pgoff); |
| 91 | 92 | } |
| 93 | + | |
| 94 | + if (TASK_SIZE - len < addr) | |
| 95 | + return -ENOMEM; | |
| 96 | + | |
| 97 | + return addr; | |
| 92 | 98 | } |
| 93 | 99 | #endif |
| @@ -867,6 +867,8 @@ static inline bool page_is_mergeable(const struct bio_vec *bv, · 866 unmodified lines | ||
| 867 | 867 | *same_page = ((vec_end_addr & PAGE_MASK) == page_addr); |
| 868 | 868 | if (*same_page) |
| 869 | 869 | return true; |
| 870 | + else if (IS_ENABLED(CONFIG_KMSAN)) | |
| 871 | + return false; | |
| 870 | 872 | return (bv->bv_page + bv_end / PAGE_SIZE) == (page + off / PAGE_SIZE); |
| 871 | 873 | } |
| 872 | 874 | |
| @@ -88,6 +88,13 @@ static inline bool biovec_phys_mergeable(struct request_queue *q, · 87 unmodified lines | ||
| 88 | 88 | phys_addr_t addr1 = page_to_phys(vec1->bv_page) + vec1->bv_offset; |
| 89 | 89 | phys_addr_t addr2 = page_to_phys(vec2->bv_page) + vec2->bv_offset; |
| 90 | 90 | |
| 91 | + /* | |
| 92 | + * Merging adjacent physical pages may not work correctly under KMSAN | |
| 93 | + * if their metadata pages aren't adjacent. Just disable merging. | |
| 94 | + */ | |
| 95 | + if (IS_ENABLED(CONFIG_KMSAN)) | |
| 96 | + return false; | |
| 97 | + | |
| 91 | 98 | if (addr1 + vec1->bv_len != addr2) |
| 92 | 99 | return false; |
| 93 | 100 | if (xen_domain() && !xen_biovec_phys_mergeable(vec1, vec2->bv_page)) |
| @@ -1391,6 +1391,7 @@ endmenu · 1390 unmodified lines | ||
| 1391 | 1391 | config CRYPTO_HASH_INFO |
| 1392 | 1392 | bool |
| 1393 | 1393 | |
| 1394 | +if !KMSAN # avoid false positives from assembly | |
| 1394 | 1395 | if ARM |
| 1395 | 1396 | source "arch/arm/crypto/Kconfig" |
| 1396 | 1397 | endif |
| @@ -1412,6 +1413,7 @@ endif · 15 unmodified lines | ||
| 1412 | 1413 | if X86 |
| 1413 | 1414 | source "arch/x86/crypto/Kconfig" |
| 1414 | 1415 | endif |
| 1416 | +endif | |
| 1415 | 1417 | |
| 1416 | 1418 | source "drivers/crypto/Kconfig" |
| 1417 | 1419 | source "crypto/asymmetric_keys/Kconfig" |
| @@ -869,12 +869,6 @@ void remove_memory_block_devices(unsigned long start, unsigned long size) · 868 unmodified lines | ||
| 869 | 869 | } |
| 870 | 870 | } |
| 871 | 871 | |
| 872 | −/* return true if the memory block is offlined, otherwise, return false */ | |
| 873 | −bool is_memblock_offlined(struct memory_block *mem) | |
| 874 | −{ | |
| 875 | − return mem->state == MEM_OFFLINE; | |
| 876 | −} | |
| 877 | − | |
| 878 | 872 | static struct attribute *memory_root_attrs[] = { |
| 879 | 873 | #ifdef CONFIG_ARCH_MEMORY_PROBE |
| 880 | 874 | &dev_attr_probe.attr, |
| @@ -20,6 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <linux/pm_runtime.h> |
| 21 | 21 | #include <linux/swap.h> |
| 22 | 22 | #include <linux/slab.h> |
| 23 | +#include <linux/hugetlb.h> | |
| 23 | 24 | |
| 24 | 25 | static struct bus_type node_subsys = { |
| 25 | 26 | .name = "node", |
| @@ -589,64 +590,9 @@ static const struct attribute_group *node_dev_groups[] = { · 563 unmodified lines | ||
| 589 | 590 | NULL |
| 590 | 591 | }; |
| 591 | 592 | |
| 592 | −#ifdef CONFIG_HUGETLBFS | |
| 593 | −/* | |
| 594 | − * hugetlbfs per node attributes registration interface: | |
| 595 | − * When/if hugetlb[fs] subsystem initializes [sometime after this module], | |
| 596 | − * it will register its per node attributes for all online nodes with | |
| 597 | − * memory. It will also call register_hugetlbfs_with_node(), below, to | |
| 598 | − * register its attribute registration functions with this node driver. | |
| 599 | − * Once these hooks have been initialized, the node driver will call into | |
| 600 | − * the hugetlb module to [un]register attributes for hot-plugged nodes. | |
| 601 | − */ | |
| 602 | −static node_registration_func_t __hugetlb_register_node; | |
| 603 | −static node_registration_func_t __hugetlb_unregister_node; | |
| 604 | − | |
| 605 | −static inline bool hugetlb_register_node(struct node *node) | |
| 606 | −{ | |
| 607 | − if (__hugetlb_register_node && | |
| 608 | − node_state(node->dev.id, N_MEMORY)) { | |
| 609 | − __hugetlb_register_node(node); | |
| 610 | − return true; | |
| 611 | − } | |
| 612 | − return false; | |
| 613 | −} | |
| 614 | − | |
| 615 | −static inline void hugetlb_unregister_node(struct node *node) | |
| 616 | −{ | |
| 617 | − if (__hugetlb_unregister_node) | |
| 618 | − __hugetlb_unregister_node(node); | |
| 619 | −} | |
| 620 | − | |
| 621 | −void register_hugetlbfs_with_node(node_registration_func_t doregister, | |
| 622 | − node_registration_func_t unregister) | |
| 623 | −{ | |
| 624 | − __hugetlb_register_node = doregister; | |
| 625 | − __hugetlb_unregister_node = unregister; | |
| 626 | −} | |
| 627 | −#else | |
| 628 | −static inline void hugetlb_register_node(struct node *node) {} | |
| 629 | − | |
| 630 | −static inline void hugetlb_unregister_node(struct node *node) {} | |
| 631 | −#endif | |
| 632 | − | |
| 633 | 593 | static void node_device_release(struct device *dev) |
| 634 | 594 | { |
| 635 | − struct node *node = to_node(dev); | |
| 636 | − | |
| 637 | −#if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_HUGETLBFS) | |
| 638 | − /* | |
| 639 | − * We schedule the work only when a memory section is | |
| 640 | − * onlined/offlined on this node. When we come here, | |
| 641 | − * all the memory on this node has been offlined, | |
| 642 | − * so we won't enqueue new work to this work. | |
| 643 | − * | |
| 644 | − * The work is using node->node_work, so we should | |
| 645 | − * flush work before freeing the memory. | |
| 646 | − */ | |
| 647 | − flush_work(&node->node_work); | |
| 648 | −#endif | |
| 649 | − kfree(node); | |
| 595 | + kfree(to_node(dev)); | |
| 650 | 596 | } |
| 651 | 597 | |
| 652 | 598 | /* |
| @@ -665,13 +611,13 @@ static int register_node(struct node *node, int num) · 12 unmodified lines | ||
| 665 | 611 | node->dev.groups = node_dev_groups; |
| 666 | 612 | error = device_register(&node->dev); |
| 667 | 613 | |
| 668 | − if (error) | |
| 614 | + if (error) { | |
| 669 | 615 | put_device(&node->dev); |
| 670 | − else { | |
| 616 | + } else { | |
| 671 | 617 | hugetlb_register_node(node); |
| 672 | − | |
| 673 | 618 | compaction_register_node(node); |
| 674 | 619 | } |
| 620 | + | |
| 675 | 621 | return error; |
| 676 | 622 | } |
| 677 | 623 | |
| @@ -684,8 +630,8 @@ static int register_node(struct node *node, int num) · 6 unmodified lines | ||
| 684 | 630 | */ |
| 685 | 631 | void unregister_node(struct node *node) |
| 686 | 632 | { |
| 633 | + hugetlb_unregister_node(node); | |
| 687 | 634 | compaction_unregister_node(node); |
| 688 | − hugetlb_unregister_node(node); /* no-op, if memoryless node */ | |
| 689 | 635 | node_remove_accesses(node); |
| 690 | 636 | node_remove_caches(node); |
| 691 | 637 | device_unregister(&node->dev); |
| @@ -907,74 +853,8 @@ void register_memory_blocks_under_node(int nid, unsigned long start_pfn, · 215 unmodified lines | ||
| 907 | 853 | (void *)&nid, func); |
| 908 | 854 | return; |
| 909 | 855 | } |
| 910 | − | |
| 911 | −#ifdef CONFIG_HUGETLBFS | |
| 912 | −/* | |
| 913 | − * Handle per node hstate attribute [un]registration on transistions | |
| 914 | − * to/from memoryless state. | |
| 915 | − */ | |
| 916 | −static void node_hugetlb_work(struct work_struct *work) | |
| 917 | −{ | |
| 918 | − struct node *node = container_of(work, struct node, node_work); | |
| 919 | − | |
| 920 | − /* | |
| 921 | − * We only get here when a node transitions to/from memoryless state. | |
| 922 | − * We can detect which transition occurred by examining whether the | |
| 923 | − * node has memory now. hugetlb_register_node() already check this | |
| 924 | − * so we try to register the attributes. If that fails, then the | |
| 925 | − * node has transitioned to memoryless, try to unregister the | |
| 926 | − * attributes. | |
| 927 | − */ | |
| 928 | − if (!hugetlb_register_node(node)) | |
| 929 | − hugetlb_unregister_node(node); | |
| 930 | −} | |
| 931 | − | |
| 932 | −static void init_node_hugetlb_work(int nid) | |
| 933 | −{ | |
| 934 | − INIT_WORK(&node_devices[nid]->node_work, node_hugetlb_work); | |
| 935 | −} | |
| 936 | − | |
| 937 | −static int node_memory_callback(struct notifier_block *self, | |
| 938 | − unsigned long action, void *arg) | |
| 939 | −{ | |
| 940 | − struct memory_notify *mnb = arg; | |
| 941 | − int nid = mnb->status_change_nid; | |
| 942 | − | |
| 943 | − switch (action) { | |
| 944 | − case MEM_ONLINE: | |
| 945 | − case MEM_OFFLINE: | |
| 946 | − /* | |
| 947 | − * offload per node hstate [un]registration to a work thread | |
| 948 | − * when transitioning to/from memoryless state. | |
| 949 | − */ | |
| 950 | − if (nid != NUMA_NO_NODE) | |
| 951 | − schedule_work(&node_devices[nid]->node_work); | |
| 952 | − break; | |
| 953 | − | |
| 954 | − case MEM_GOING_ONLINE: | |
| 955 | − case MEM_GOING_OFFLINE: | |
| 956 | − case MEM_CANCEL_ONLINE: | |
| 957 | − case MEM_CANCEL_OFFLINE: | |
| 958 | − default: | |
| 959 | − break; | |
| 960 | − } | |
| 961 | − | |
| 962 | − return NOTIFY_OK; | |
| 963 | −} | |
| 964 | −#endif /* CONFIG_HUGETLBFS */ | |
| 965 | 856 | #endif /* CONFIG_MEMORY_HOTPLUG */ |
| 966 | 857 | |
| 967 | −#if !defined(CONFIG_MEMORY_HOTPLUG) || !defined(CONFIG_HUGETLBFS) | |
| 968 | −static inline int node_memory_callback(struct notifier_block *self, | |
| 969 | − unsigned long action, void *arg) | |
| 970 | −{ | |
| 971 | − return NOTIFY_OK; | |
| 972 | −} | |
| 973 | − | |
| 974 | −static void init_node_hugetlb_work(int nid) { } | |
| 975 | − | |
| 976 | −#endif | |
| 977 | − | |
| 978 | 858 | int __register_one_node(int nid) |
| 979 | 859 | { |
| 980 | 860 | int error; |
| @@ -993,8 +873,6 @@ int __register_one_node(int nid) · 12 unmodified lines | ||
| 993 | 873 | } |
| 994 | 874 | |
| 995 | 875 | INIT_LIST_HEAD(&node_devices[nid]->access_list); |
| 996 | − /* initialize work queue for memory hot plug */ | |
| 997 | − init_node_hugetlb_work(nid); | |
| 998 | 876 | node_init_caches(nid); |
| 999 | 877 | |
| 1000 | 878 | return error; |
| @@ -1065,13 +943,8 @@ static const struct attribute_group *cpu_root_attr_groups[] = { · 64 unmodified lines | ||
| 1065 | 943 | NULL, |
| 1066 | 944 | }; |
| 1067 | 945 | |
| 1068 | −#define NODE_CALLBACK_PRI 2 /* lower than SLAB */ | |
| 1069 | 946 | void __init node_dev_init(void) |
| 1070 | 947 | { |
| 1071 | − static struct notifier_block node_memory_callback_nb = { | |
| 1072 | − .notifier_call = node_memory_callback, | |
| 1073 | − .priority = NODE_CALLBACK_PRI, | |
| 1074 | − }; | |
| 1075 | 948 | int ret, i; |
| 1076 | 949 | |
| 1077 | 950 | BUILD_BUG_ON(ARRAY_SIZE(node_state_attr) != NR_NODE_STATES); |
| @@ -1081,8 +954,6 @@ void __init node_dev_init(void) · 3 unmodified lines | ||
| 1081 | 954 | if (ret) |
| 1082 | 955 | panic("%s() failed to register subsystem: %d\n", __func__, ret); |
| 1083 | 956 | |
| 1084 | − register_hotmemory_notifier(&node_memory_callback_nb); | |
| 1085 | − | |
| 1086 | 957 | /* |
| 1087 | 958 | * Create all node devices, which will properly link the node |
| 1088 | 959 | * to applicable memory block devices and already created cpu devices. |
| @@ -329,8 +329,8 @@ static ssize_t idle_store(struct device *dev, · 328 unmodified lines | ||
| 329 | 329 | |
| 330 | 330 | if (!sysfs_streq(buf, "all")) { |
| 331 | 331 | /* |
| 332 | − * If it did not parse as 'all' try to treat it as an integer when | |
| 333 | − * we have memory tracking enabled. | |
| 332 | + * If it did not parse as 'all' try to treat it as an integer | |
| 333 | + * when we have memory tracking enabled. | |
| 334 | 334 | */ |
| 335 | 335 | u64 age_sec; |
| 336 | 336 | |
| @@ -345,7 +345,10 @@ static ssize_t idle_store(struct device *dev, · 8 unmodified lines | ||
| 345 | 345 | if (!init_done(zram)) |
| 346 | 346 | goto out_unlock; |
| 347 | 347 | |
| 348 | − /* A cutoff_time of 0 marks everything as idle, this is the "all" behavior */ | |
| 348 | + /* | |
| 349 | + * A cutoff_time of 0 marks everything as idle, this is the | |
| 350 | + * "all" behavior. | |
| 351 | + */ | |
| 349 | 352 | mark_idle(zram, cutoff_time); |
| 350 | 353 | rv = len; |
| 351 | 354 | |
| @@ -1410,9 +1413,19 @@ compress_again: · 1058 unmodified lines | ||
| 1410 | 1413 | handle = zs_malloc(zram->mem_pool, comp_len, |
| 1411 | 1414 | GFP_NOIO | __GFP_HIGHMEM | |
| 1412 | 1415 | __GFP_MOVABLE); |
| 1413 | − if (!IS_ERR((void *)handle)) | |
| 1416 | + if (IS_ERR((void *)handle)) | |
| 1417 | + return PTR_ERR((void *)handle); | |
| 1418 | + | |
| 1419 | + if (comp_len != PAGE_SIZE) | |
| 1414 | 1420 | goto compress_again; |
| 1415 | − return PTR_ERR((void *)handle); | |
| 1421 | + /* | |
| 1422 | + * If the page is not compressible, you need to acquire the | |
| 1423 | + * lock and execute the code below. The zcomp_stream_get() | |
| 1424 | + * call is needed to disable the cpu hotplug and grab the | |
| 1425 | + * zstrm buffer back. It is necessary that the dereferencing | |
| 1426 | + * of the zstrm variable below occurs correctly. | |
| 1427 | + */ | |
| 1428 | + zstrm = zcomp_stream_get(zram->comp); | |
| 1416 | 1429 | } |
| 1417 | 1430 | |
| 1418 | 1431 | alloced_pages = zs_get_total_pages(zram->mem_pool); |
| @@ -1710,9 +1723,6 @@ out: · 291 unmodified lines | ||
| 1710 | 1723 | |
| 1711 | 1724 | static void zram_reset_device(struct zram *zram) |
| 1712 | 1725 | { |
| 1713 | − struct zcomp *comp; | |
| 1714 | − u64 disksize; | |
| 1715 | − | |
| 1716 | 1726 | down_write(&zram->init_lock); |
| 1717 | 1727 | |
| 1718 | 1728 | zram->limit_pages = 0; |
| @@ -1722,17 +1732,15 @@ static void zram_reset_device(struct zram *zram) · 3 unmodified lines | ||
| 1722 | 1732 | return; |
| 1723 | 1733 | } |
| 1724 | 1734 | |
| 1725 | − comp = zram->comp; | |
| 1726 | − disksize = zram->disksize; | |
| 1727 | − zram->disksize = 0; | |
| 1728 | − | |
| 1729 | 1735 | set_capacity_and_notify(zram->disk, 0); |
| 1730 | 1736 | part_stat_set_all(zram->disk->part0, 0); |
| 1731 | 1737 | |
| 1732 | 1738 | /* I/O operation under all of CPU are done so let's free */ |
| 1733 | − zram_meta_free(zram, disksize); | |
| 1739 | + zram_meta_free(zram, zram->disksize); | |
| 1740 | + zram->disksize = 0; | |
| 1734 | 1741 | memset(&zram->stats, 0, sizeof(zram->stats)); |
| 1735 | − zcomp_destroy(comp); | |
| 1742 | + zcomp_destroy(zram->comp); | |
| 1743 | + zram->comp = NULL; | |
| 1736 | 1744 | reset_bdev(zram); |
| 1737 | 1745 | |
| 1738 | 1746 | up_write(&zram->init_lock); |
| @@ -2126,6 +2134,8 @@ static int __init zram_init(void) · 387 unmodified lines | ||
| 2126 | 2134 | { |
| 2127 | 2135 | int ret; |
| 2128 | 2136 | |
| 2137 | + BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > BITS_PER_LONG); | |
| 2138 | + | |
| 2129 | 2139 | ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare", |
| 2130 | 2140 | zcomp_cpu_up_prepare, zcomp_cpu_dead); |
| 2131 | 2141 | if (ret < 0) |
| @@ -30,16 +30,15 @@ · 29 unmodified lines | ||
| 30 | 30 | |
| 31 | 31 | |
| 32 | 32 | /* |
| 33 | − * The lower ZRAM_FLAG_SHIFT bits of table.flags is for | |
| 34 | − * object size (excluding header), the higher bits is for | |
| 35 | − * zram_pageflags. | |
| 33 | + * ZRAM is mainly used for memory efficiency so we want to keep memory | |
| 34 | + * footprint small and thus squeeze size and zram pageflags into a flags | |
| 35 | + * member. The lower ZRAM_FLAG_SHIFT bits is for object size (excluding | |
| 36 | + * header), which cannot be larger than PAGE_SIZE (requiring PAGE_SHIFT | |
| 37 | + * bits), the higher bits are for zram_pageflags. | |
| 36 | 38 | * |
| 37 | − * zram is mainly used for memory efficiency so we want to keep memory | |
| 38 | − * footprint small so we can squeeze size and flags into a field. | |
| 39 | − * The lower ZRAM_FLAG_SHIFT bits is for object size (excluding header), | |
| 40 | − * the higher bits is for zram_pageflags. | |
| 39 | + * We use BUILD_BUG_ON() to make sure that zram pageflags don't overflow. | |
| 41 | 40 | */ |
| 42 | −#define ZRAM_FLAG_SHIFT 24 | |
| 41 | +#define ZRAM_FLAG_SHIFT (PAGE_SHIFT + 1) | |
| 43 | 42 | |
| 44 | 43 | /* Flags for zram pages (table[page_no].flags) */ |
| 45 | 44 | enum zram_pageflags { |
| @@ -11,9 +11,17 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/fs.h> |
| 12 | 12 | #include <linux/mm.h> |
| 13 | 13 | #include <linux/mman.h> |
| 14 | +#include <linux/memory-tiers.h> | |
| 14 | 15 | #include "dax-private.h" |
| 15 | 16 | #include "bus.h" |
| 16 | 17 | |
| 18 | +/* | |
| 19 | + * Default abstract distance assigned to the NUMA node onlined | |
| 20 | + * by DAX/kmem if the low level platform driver didn't initialize | |
| 21 | + * one for this NUMA node. | |
| 22 | + */ | |
| 23 | +#define MEMTIER_DEFAULT_DAX_ADISTANCE (MEMTIER_ADISTANCE_DRAM * 5) | |
| 24 | + | |
| 17 | 25 | /* Memory resource name used for add_memory_driver_managed(). */ |
| 18 | 26 | static const char *kmem_name; |
| 19 | 27 | /* Set if any memory will remain added when the driver will be unloaded. */ |
| @@ -41,6 +49,7 @@ struct dax_kmem_data { · 21 unmodified lines | ||
| 41 | 49 | struct resource *res[]; |
| 42 | 50 | }; |
| 43 | 51 | |
| 52 | +static struct memory_dev_type *dax_slowmem_type; | |
| 44 | 53 | static int dev_dax_kmem_probe(struct dev_dax *dev_dax) |
| 45 | 54 | { |
| 46 | 55 | struct device *dev = &dev_dax->dev; |
| @@ -79,11 +88,13 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · 32 unmodified lines | ||
| 79 | 88 | return -EINVAL; |
| 80 | 89 | } |
| 81 | 90 | |
| 91 | + init_node_memory_type(numa_node, dax_slowmem_type); | |
| 92 | + | |
| 93 | + rc = -ENOMEM; | |
| 82 | 94 | data = kzalloc(struct_size(data, res, dev_dax->nr_range), GFP_KERNEL); |
| 83 | 95 | if (!data) |
| 84 | − return -ENOMEM; | |
| 96 | + goto err_dax_kmem_data; | |
| 85 | 97 | |
| 86 | − rc = -ENOMEM; | |
| 87 | 98 | data->res_name = kstrdup(dev_name(dev), GFP_KERNEL); |
| 88 | 99 | if (!data->res_name) |
| 89 | 100 | goto err_res_name; |
| @@ -155,6 +166,8 @@ err_reg_mgid: · 65 unmodified lines | ||
| 155 | 166 | kfree(data->res_name); |
| 156 | 167 | err_res_name: |
| 157 | 168 | kfree(data); |
| 169 | +err_dax_kmem_data: | |
| 170 | + clear_node_memory_type(numa_node, dax_slowmem_type); | |
| 158 | 171 | return rc; |
| 159 | 172 | } |
| 160 | 173 | |
| @@ -162,6 +175,7 @@ err_res_name: · 1 unmodified line | ||
| 162 | 175 | static void dev_dax_kmem_remove(struct dev_dax *dev_dax) |
| 163 | 176 | { |
| 164 | 177 | int i, success = 0; |
| 178 | + int node = dev_dax->target_node; | |
| 165 | 179 | struct device *dev = &dev_dax->dev; |
| 166 | 180 | struct dax_kmem_data *data = dev_get_drvdata(dev); |
| 167 | 181 | |
| @@ -198,6 +212,14 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax) · 30 unmodified lines | ||
| 198 | 212 | kfree(data->res_name); |
| 199 | 213 | kfree(data); |
| 200 | 214 | dev_set_drvdata(dev, NULL); |
| 215 | + /* | |
| 216 | + * Clear the memtype association on successful unplug. | |
| 217 | + * If not, we have memory blocks left which can be | |
| 218 | + * offlined/onlined later. We need to keep memory_dev_type | |
| 219 | + * for that. This implies this reference will be around | |
| 220 | + * till next reboot. | |
| 221 | + */ | |
| 222 | + clear_node_memory_type(node, dax_slowmem_type); | |
| 201 | 223 | } |
| 202 | 224 | } |
| 203 | 225 | #else |
| @@ -228,9 +250,22 @@ static int __init dax_kmem_init(void) · 24 unmodified lines | ||
| 228 | 250 | if (!kmem_name) |
| 229 | 251 | return -ENOMEM; |
| 230 | 252 | |
| 253 | + dax_slowmem_type = alloc_memory_type(MEMTIER_DEFAULT_DAX_ADISTANCE); | |
| 254 | + if (IS_ERR(dax_slowmem_type)) { | |
| 255 | + rc = PTR_ERR(dax_slowmem_type); | |
| 256 | + goto err_dax_slowmem_type; | |
| 257 | + } | |
| 258 | + | |
| 231 | 259 | rc = dax_driver_register(&device_dax_kmem_driver); |
| 232 | 260 | if (rc) |
| 233 | − kfree_const(kmem_name); | |
| 261 | + goto error_dax_driver; | |
| 262 | + | |
| 263 | + return rc; | |
| 264 | + | |
| 265 | +error_dax_driver: | |
| 266 | + destroy_memory_type(dax_slowmem_type); | |
| 267 | +err_dax_slowmem_type: | |
| 268 | + kfree_const(kmem_name); | |
| 234 | 269 | return rc; |
| 235 | 270 | } |
| 236 | 271 | |
| @@ -239,6 +274,7 @@ static void __exit dax_kmem_exit(void) · 2 unmodified lines | ||
| 239 | 274 | dax_driver_unregister(&device_dax_kmem_driver); |
| 240 | 275 | if (!any_hotremove_failed) |
| 241 | 276 | kfree_const(kmem_name); |
| 277 | + destroy_memory_type(dax_slowmem_type); | |
| 242 | 278 | } |
| 243 | 279 | |
| 244 | 280 | MODULE_AUTHOR("Intel Corporation"); |
| @@ -59,7 +59,7 @@ static unsigned long __initdata rt_prop = EFI_INVALID_TABLE_ADDR; · 58 unmodified lines | ||
| 59 | 59 | static unsigned long __initdata initrd = EFI_INVALID_TABLE_ADDR; |
| 60 | 60 | |
| 61 | 61 | struct mm_struct efi_mm = { |
| 62 | − .mm_rb = RB_ROOT, | |
| 62 | + .mm_mt = MTREE_INIT_EXT(mm_mt, MM_MT_FLAGS, efi_mm.mmap_lock), | |
| 63 | 63 | .mm_users = ATOMIC_INIT(2), |
| 64 | 64 | .mm_count = ATOMIC_INIT(1), |
| 65 | 65 | .write_protect_seq = SEQCNT_ZERO(efi_mm.write_protect_seq), |
| @@ -57,6 +57,7 @@ GCOV_PROFILE := n · 56 unmodified lines | ||
| 57 | 57 | # Sanitizer runtimes are unavailable and cannot be linked here. |
| 58 | 58 | KASAN_SANITIZE := n |
| 59 | 59 | KCSAN_SANITIZE := n |
| 60 | +KMSAN_SANITIZE := n | |
| 60 | 61 | UBSAN_SANITIZE := n |
| 61 | 62 | OBJECT_FILES_NON_STANDARD := y |
| 62 | 63 | |
| @@ -426,12 +426,11 @@ static const struct drm_i915_gem_object_ops i915_gem_userptr_ops = { · 425 unmodified lines | ||
| 426 | 426 | static int |
| 427 | 427 | probe_range(struct mm_struct *mm, unsigned long addr, unsigned long len) |
| 428 | 428 | { |
| 429 | − const unsigned long end = addr + len; | |
| 429 | + VMA_ITERATOR(vmi, mm, addr); | |
| 430 | 430 | struct vm_area_struct *vma; |
| 431 | − int ret = -EFAULT; | |
| 432 | 431 | |
| 433 | 432 | mmap_read_lock(mm); |
| 434 | − for (vma = find_vma(mm, addr); vma; vma = vma->vm_next) { | |
| 433 | + for_each_vma_range(vmi, vma, addr + len) { | |
| 435 | 434 | /* Check for holes, note that we also update the addr below */ |
| 436 | 435 | if (vma->vm_start > addr) |
| 437 | 436 | break; |
| @@ -439,16 +438,13 @@ probe_range(struct mm_struct *mm, unsigned long addr, unsigned long len) · 1 unmodified line | ||
| 439 | 438 | if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)) |
| 440 | 439 | break; |
| 441 | 440 | |
| 442 | − if (vma->vm_end >= end) { | |
| 443 | − ret = 0; | |
| 444 | − break; | |
| 445 | − } | |
| 446 | − | |
| 447 | 441 | addr = vma->vm_end; |
| 448 | 442 | } |
| 449 | 443 | mmap_read_unlock(mm); |
| 450 | 444 | |
| 451 | − return ret; | |
| 445 | + if (vma) | |
| 446 | + return -EFAULT; | |
| 447 | + return 0; | |
| 452 | 448 | } |
| 453 | 449 | |
| 454 | 450 | /* |
| @@ -12,6 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <linux/sched.h> |
| 13 | 13 | #include <linux/interrupt.h> |
| 14 | 14 | #include <linux/input.h> |
| 15 | +#include <linux/kmsan-checks.h> | |
| 15 | 16 | #include <linux/serio.h> |
| 16 | 17 | #include <linux/i8042.h> |
| 17 | 18 | #include <linux/libps2.h> |
| @@ -294,9 +295,11 @@ int __ps2_command(struct ps2dev *ps2dev, u8 *param, unsigned int command) · 276 unmodified lines | ||
| 294 | 295 | |
| 295 | 296 | serio_pause_rx(ps2dev->serio); |
| 296 | 297 | |
| 297 | − if (param) | |
| 298 | + if (param) { | |
| 298 | 299 | for (i = 0; i < receive; i++) |
| 299 | 300 | param[i] = ps2dev->cmdbuf[(receive - 1) - i]; |
| 301 | + kmsan_unpoison_memory(param, receive); | |
| 302 | + } | |
| 300 | 303 | |
| 301 | 304 | if (ps2dev->cmdcnt && |
| 302 | 305 | (command != PS2_CMD_RESET_BAT || ps2dev->cmdcnt != 1)) { |
| @@ -280,22 +280,6 @@ void cxl_handle_fault(struct work_struct *fault_work) · 279 unmodified lines | ||
| 280 | 280 | mmput(mm); |
| 281 | 281 | } |
| 282 | 282 | |
| 283 | −static void cxl_prefault_one(struct cxl_context *ctx, u64 ea) | |
| 284 | −{ | |
| 285 | − struct mm_struct *mm; | |
| 286 | − | |
| 287 | − mm = get_mem_context(ctx); | |
| 288 | − if (mm == NULL) { | |
| 289 | − pr_devel("cxl_prefault_one unable to get mm %i\n", | |
| 290 | − pid_nr(ctx->pid)); | |
| 291 | − return; | |
| 292 | − } | |
| 293 | − | |
| 294 | − cxl_fault_segment(ctx, mm, ea); | |
| 295 | − | |
| 296 | − mmput(mm); | |
| 297 | −} | |
| 298 | − | |
| 299 | 283 | static u64 next_segment(u64 ea, u64 vsid) |
| 300 | 284 | { |
| 301 | 285 | if (vsid & SLB_VSID_B_1T) |
| @@ -306,23 +290,16 @@ static u64 next_segment(u64 ea, u64 vsid) · 4 unmodified lines | ||
| 306 | 290 | return ea + 1; |
| 307 | 291 | } |
| 308 | 292 | |
| 309 | −static void cxl_prefault_vma(struct cxl_context *ctx) | |
| 293 | +static void cxl_prefault_vma(struct cxl_context *ctx, struct mm_struct *mm) | |
| 310 | 294 | { |
| 311 | 295 | u64 ea, last_esid = 0; |
| 312 | 296 | struct copro_slb slb; |
| 297 | + VMA_ITERATOR(vmi, mm, 0); | |
| 313 | 298 | struct vm_area_struct *vma; |
| 314 | 299 | int rc; |
| 315 | − struct mm_struct *mm; | |
| 316 | − | |
| 317 | − mm = get_mem_context(ctx); | |
| 318 | − if (mm == NULL) { | |
| 319 | − pr_devel("cxl_prefault_vm unable to get mm %i\n", | |
| 320 | − pid_nr(ctx->pid)); | |
| 321 | − return; | |
| 322 | − } | |
| 323 | 300 | |
| 324 | 301 | mmap_read_lock(mm); |
| 325 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 302 | + for_each_vma(vmi, vma) { | |
| 326 | 303 | for (ea = vma->vm_start; ea < vma->vm_end; |
| 327 | 304 | ea = next_segment(ea, slb.vsid)) { |
| 328 | 305 | rc = copro_calculate_slb(mm, ea, &slb); |
| @@ -337,20 +314,28 @@ static void cxl_prefault_vma(struct cxl_context *ctx) · 8 unmodified lines | ||
| 337 | 314 | } |
| 338 | 315 | } |
| 339 | 316 | mmap_read_unlock(mm); |
| 340 | − | |
| 341 | − mmput(mm); | |
| 342 | 317 | } |
| 343 | 318 | |
| 344 | 319 | void cxl_prefault(struct cxl_context *ctx, u64 wed) |
| 345 | 320 | { |
| 321 | + struct mm_struct *mm = get_mem_context(ctx); | |
| 322 | + | |
| 323 | + if (mm == NULL) { | |
| 324 | + pr_devel("cxl_prefault unable to get mm %i\n", | |
| 325 | + pid_nr(ctx->pid)); | |
| 326 | + return; | |
| 327 | + } | |
| 328 | + | |
| 346 | 329 | switch (ctx->afu->prefault_mode) { |
| 347 | 330 | case CXL_PREFAULT_WED: |
| 348 | − cxl_prefault_one(ctx, wed); | |
| 331 | + cxl_fault_segment(ctx, mm, wed); | |
| 349 | 332 | break; |
| 350 | 333 | case CXL_PREFAULT_ALL: |
| 351 | − cxl_prefault_vma(ctx); | |
| 334 | + cxl_prefault_vma(ctx, mm); | |
| 352 | 335 | break; |
| 353 | 336 | default: |
| 354 | 337 | break; |
| 355 | 338 | } |
| 339 | + | |
| 340 | + mmput(mm); | |
| 356 | 341 | } |
| @@ -76,6 +76,7 @@ config WIREGUARD · 75 unmodified lines | ||
| 76 | 76 | tristate "WireGuard secure network tunnel" |
| 77 | 77 | depends on NET && INET |
| 78 | 78 | depends on IPV6 || !IPV6 |
| 79 | + depends on !KMSAN # KMSAN doesn't support the crypto configs below | |
| 79 | 80 | select NET_UDP_TUNNEL |
| 80 | 81 | select DST_CACHE |
| 81 | 82 | select CRYPTO |
| @@ -652,7 +652,7 @@ void devm_namespace_disable(struct device *dev, · 651 unmodified lines | ||
| 652 | 652 | struct nd_namespace_common *ndns); |
| 653 | 653 | #if IS_ENABLED(CONFIG_ND_CLAIM) |
| 654 | 654 | /* max struct page size independent of kernel config */ |
| 655 | −#define MAX_STRUCT_PAGE_SIZE 64 | |
| 655 | +#define MAX_STRUCT_PAGE_SIZE 128 | |
| 656 | 656 | int nvdimm_setup_pfn(struct nd_pfn *nd_pfn, struct dev_pagemap *pgmap); |
| 657 | 657 | #else |
| 658 | 658 | static inline int nvdimm_setup_pfn(struct nd_pfn *nd_pfn, |
| @@ -787,7 +787,7 @@ static int nd_pfn_init(struct nd_pfn *nd_pfn) · 786 unmodified lines | ||
| 787 | 787 | * when populating the vmemmap. This *should* be equal to |
| 788 | 788 | * PMD_SIZE for most architectures. |
| 789 | 789 | * |
| 790 | − * Also make sure size of struct page is less than 64. We | |
| 790 | + * Also make sure size of struct page is less than 128. We | |
| 791 | 791 | * want to make sure we use large enough size here so that |
| 792 | 792 | * we don't have a dynamic reserve space depending on |
| 793 | 793 | * struct page size. But we also want to make sure we notice |
| @@ -492,15 +492,18 @@ static bool is_normal_memory(pgprot_t p) · 491 unmodified lines | ||
| 492 | 492 | #endif |
| 493 | 493 | } |
| 494 | 494 | |
| 495 | −static int __check_mem_type(struct vm_area_struct *vma, unsigned long end) | |
| 495 | +static int __check_mem_type(struct mm_struct *mm, unsigned long start, | |
| 496 | + unsigned long end) | |
| 496 | 497 | { |
| 497 | − while (vma && is_normal_memory(vma->vm_page_prot)) { | |
| 498 | − if (vma->vm_end >= end) | |
| 499 | − return 0; | |
| 500 | − vma = vma->vm_next; | |
| 498 | + struct vm_area_struct *vma; | |
| 499 | + VMA_ITERATOR(vmi, mm, start); | |
| 500 | + | |
| 501 | + for_each_vma_range(vmi, vma, end) { | |
| 502 | + if (!is_normal_memory(vma->vm_page_prot)) | |
| 503 | + return -EINVAL; | |
| 501 | 504 | } |
| 502 | 505 | |
| 503 | − return -EINVAL; | |
| 506 | + return 0; | |
| 504 | 507 | } |
| 505 | 508 | |
| 506 | 509 | int optee_check_mem_type(unsigned long start, size_t num_pages) |
| @@ -516,8 +519,7 @@ int optee_check_mem_type(unsigned long start, size_t num_pages) · 9 unmodified lines | ||
| 516 | 519 | return 0; |
| 517 | 520 | |
| 518 | 521 | mmap_read_lock(mm); |
| 519 | − rc = __check_mem_type(find_vma(mm, start), | |
| 520 | − start + num_pages * PAGE_SIZE); | |
| 522 | + rc = __check_mem_type(mm, start, start + num_pages * PAGE_SIZE); | |
| 521 | 523 | mmap_read_unlock(mm); |
| 522 | 524 | |
| 523 | 525 | return rc; |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/bitops.h> |
| 9 | 9 | #include <linux/slab.h> |
| 10 | 10 | #include <linux/log2.h> |
| 11 | +#include <linux/kmsan.h> | |
| 11 | 12 | #include <linux/usb.h> |
| 12 | 13 | #include <linux/wait.h> |
| 13 | 14 | #include <linux/usb/hcd.h> |
| @@ -426,6 +427,7 @@ int usb_submit_urb(struct urb *urb, gfp_t mem_flags) · 412 unmodified lines | ||
| 426 | 427 | URB_SETUP_MAP_SINGLE | URB_SETUP_MAP_LOCAL | |
| 427 | 428 | URB_DMA_SG_COMBINED); |
| 428 | 429 | urb->transfer_flags |= (is_out ? URB_DIR_OUT : URB_DIR_IN); |
| 430 | + kmsan_handle_urb(urb, is_out); | |
| 429 | 431 | |
| 430 | 432 | if (xfertype != USB_ENDPOINT_XFER_CONTROL && |
| 431 | 433 | dev->state < USB_STATE_CONFIGURED) |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/module.h> |
| 12 | 12 | #include <linux/hrtimer.h> |
| 13 | 13 | #include <linux/dma-mapping.h> |
| 14 | +#include <linux/kmsan.h> | |
| 14 | 15 | #include <linux/spinlock.h> |
| 15 | 16 | #include <xen/xen.h> |
| 16 | 17 | |
| @@ -352,8 +353,15 @@ static dma_addr_t vring_map_one_sg(const struct vring_virtqueue *vq, · 335 unmodified lines | ||
| 352 | 353 | struct scatterlist *sg, |
| 353 | 354 | enum dma_data_direction direction) |
| 354 | 355 | { |
| 355 | − if (!vq->use_dma_api) | |
| 356 | + if (!vq->use_dma_api) { | |
| 357 | + /* | |
| 358 | + * If DMA is not used, KMSAN doesn't know that the scatterlist | |
| 359 | + * is initialized by the hardware. Explicitly check/unpoison it | |
| 360 | + * depending on the direction. | |
| 361 | + */ | |
| 362 | + kmsan_handle_dma(sg_page(sg), sg->offset, sg->length, direction); | |
| 356 | 363 | return (dma_addr_t)sg_phys(sg); |
| 364 | + } | |
| 357 | 365 | |
| 358 | 366 | /* |
| 359 | 367 | * We can't use dma_map_sg, because we don't use scatterlists in |
| @@ -282,7 +282,7 @@ static long privcmd_ioctl_mmap(struct file *file, void __user *udata) · 281 unmodified lines | ||
| 282 | 282 | struct page, lru); |
| 283 | 283 | struct privcmd_mmap_entry *msg = page_address(page); |
| 284 | 284 | |
| 285 | − vma = find_vma(mm, msg->va); | |
| 285 | + vma = vma_lookup(mm, msg->va); | |
| 286 | 286 | rc = -EINVAL; |
| 287 | 287 | |
| 288 | 288 | if (!vma || (msg->va != vma->vm_start) || vma->vm_private_data) |
| @@ -235,6 +235,7 @@ config ARCH_SUPPORTS_HUGETLBFS · 234 unmodified lines | ||
| 235 | 235 | config HUGETLBFS |
| 236 | 236 | bool "HugeTLB file system support" |
| 237 | 237 | depends on X86 || IA64 || SPARC64 || ARCH_SUPPORTS_HUGETLBFS || BROKEN |
| 238 | + depends on (SYSFS || SYSCTL) | |
| 238 | 239 | help |
| 239 | 240 | hugetlbfs is a filesystem backing for HugeTLB pages, based on |
| 240 | 241 | ramfs. For architectures that support it, say Y here and read |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/file.h> |
| 9 | 9 | #include <linux/fs.h> |
| 10 | 10 | #include <linux/pagemap.h> |
| 11 | +#include <linux/pagevec.h> | |
| 11 | 12 | #include <linux/highmem.h> |
| 12 | 13 | #include <linux/kthread.h> |
| 13 | 14 | #include <linux/time.h> |
| @@ -219,8 +220,7 @@ static noinline void end_compressed_writeback(struct inode *inode, · 205 unmodified lines | ||
| 219 | 220 | struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); |
| 220 | 221 | unsigned long index = cb->start >> PAGE_SHIFT; |
| 221 | 222 | unsigned long end_index = (cb->start + cb->len - 1) >> PAGE_SHIFT; |
| 222 | − struct page *pages[16]; | |
| 223 | − unsigned long nr_pages = end_index - index + 1; | |
| 223 | + struct folio_batch fbatch; | |
| 224 | 224 | const int errno = blk_status_to_errno(cb->status); |
| 225 | 225 | int i; |
| 226 | 226 | int ret; |
| @@ -228,24 +228,23 @@ static noinline void end_compressed_writeback(struct inode *inode, · 1 unmodified line | ||
| 228 | 228 | if (errno) |
| 229 | 229 | mapping_set_error(inode->i_mapping, errno); |
| 230 | 230 | |
| 231 | − while (nr_pages > 0) { | |
| 232 | − ret = find_get_pages_contig(inode->i_mapping, index, | |
| 233 | − min_t(unsigned long, | |
| 234 | − nr_pages, ARRAY_SIZE(pages)), pages); | |
| 235 | − if (ret == 0) { | |
| 236 | − nr_pages -= 1; | |
| 237 | − index += 1; | |
| 238 | − continue; | |
| 239 | − } | |
| 231 | + folio_batch_init(&fbatch); | |
| 232 | + while (index <= end_index) { | |
| 233 | + ret = filemap_get_folios(inode->i_mapping, &index, end_index, | |
| 234 | + &fbatch); | |
| 235 | + | |
| 236 | + if (ret == 0) | |
| 237 | + return; | |
| 238 | + | |
| 240 | 239 | for (i = 0; i < ret; i++) { |
| 240 | + struct folio *folio = fbatch.folios[i]; | |
| 241 | + | |
| 241 | 242 | if (errno) |
| 242 | − SetPageError(pages[i]); | |
| 243 | − btrfs_page_clamp_clear_writeback(fs_info, pages[i], | |
| 243 | + folio_set_error(folio); | |
| 244 | + btrfs_page_clamp_clear_writeback(fs_info, &folio->page, | |
| 244 | 245 | cb->start, cb->len); |
| 245 | − put_page(pages[i]); | |
| 246 | 246 | } |
| 247 | − nr_pages -= ret; | |
| 248 | − index += ret; | |
| 247 | + folio_batch_release(&fbatch); | |
| 249 | 248 | } |
| 250 | 249 | /* the inode may be gone now */ |
| 251 | 250 | } |
| @@ -270,9 +270,8 @@ static int __process_pages_contig(struct address_space *mapping, · 269 unmodified lines | ||
| 270 | 270 | pgoff_t start_index = start >> PAGE_SHIFT; |
| 271 | 271 | pgoff_t end_index = end >> PAGE_SHIFT; |
| 272 | 272 | pgoff_t index = start_index; |
| 273 | − unsigned long nr_pages = end_index - start_index + 1; | |
| 274 | 273 | unsigned long pages_processed = 0; |
| 275 | − struct page *pages[16]; | |
| 274 | + struct folio_batch fbatch; | |
| 276 | 275 | int err = 0; |
| 277 | 276 | int i; |
| 278 | 277 | |
| @@ -281,16 +280,17 @@ static int __process_pages_contig(struct address_space *mapping, · 2 unmodified lines | ||
| 281 | 280 | ASSERT(processed_end && *processed_end == start); |
| 282 | 281 | } |
| 283 | 282 | |
| 284 | − if ((page_ops & PAGE_SET_ERROR) && nr_pages > 0) | |
| 283 | + if ((page_ops & PAGE_SET_ERROR) && start_index <= end_index) | |
| 285 | 284 | mapping_set_error(mapping, -EIO); |
| 286 | 285 | |
| 287 | − while (nr_pages > 0) { | |
| 288 | − int found_pages; | |
| 286 | + folio_batch_init(&fbatch); | |
| 287 | + while (index <= end_index) { | |
| 288 | + int found_folios; | |
| 289 | + | |
| 290 | + found_folios = filemap_get_folios_contig(mapping, &index, | |
| 291 | + end_index, &fbatch); | |
| 289 | 292 | |
| 290 | − found_pages = find_get_pages_contig(mapping, index, | |
| 291 | − min_t(unsigned long, | |
| 292 | − nr_pages, ARRAY_SIZE(pages)), pages); | |
| 293 | − if (found_pages == 0) { | |
| 293 | + if (found_folios == 0) { | |
| 294 | 294 | /* |
| 295 | 295 | * Only if we're going to lock these pages, we can find |
| 296 | 296 | * nothing at @index. |
| @@ -300,23 +300,20 @@ static int __process_pages_contig(struct address_space *mapping, · 3 unmodified lines | ||
| 300 | 300 | goto out; |
| 301 | 301 | } |
| 302 | 302 | |
| 303 | − for (i = 0; i < found_pages; i++) { | |
| 303 | + for (i = 0; i < found_folios; i++) { | |
| 304 | 304 | int process_ret; |
| 305 | − | |
| 305 | + struct folio *folio = fbatch.folios[i]; | |
| 306 | 306 | process_ret = process_one_page(fs_info, mapping, |
| 307 | − pages[i], locked_page, page_ops, | |
| 307 | + &folio->page, locked_page, page_ops, | |
| 308 | 308 | start, end); |
| 309 | 309 | if (process_ret < 0) { |
| 310 | − for (; i < found_pages; i++) | |
| 311 | − put_page(pages[i]); | |
| 312 | 310 | err = -EAGAIN; |
| 311 | + folio_batch_release(&fbatch); | |
| 313 | 312 | goto out; |
| 314 | 313 | } |
| 315 | − put_page(pages[i]); | |
| 316 | − pages_processed++; | |
| 314 | + pages_processed += folio_nr_pages(folio); | |
| 317 | 315 | } |
| 318 | − nr_pages -= found_pages; | |
| 319 | − index += found_pages; | |
| 316 | + folio_batch_release(&fbatch); | |
| 320 | 317 | cond_resched(); |
| 321 | 318 | } |
| 322 | 319 | out: |
| @@ -337,7 +337,7 @@ bool btrfs_subpage_end_and_test_writer(const struct btrfs_fs_info *fs_info, · 336 unmodified lines | ||
| 337 | 337 | * |
| 338 | 338 | * Even with 0 returned, the page still need extra check to make sure |
| 339 | 339 | * it's really the correct page, as the caller is using |
| 340 | − * find_get_pages_contig(), which can race with page invalidating. | |
| 340 | + * filemap_get_folios_contig(), which can race with page invalidating. | |
| 341 | 341 | */ |
| 342 | 342 | int btrfs_page_start_writer_lock(const struct btrfs_fs_info *fs_info, |
| 343 | 343 | struct page *page, u64 start, u32 len) |
| @@ -4,6 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | */ |
| 5 | 5 | |
| 6 | 6 | #include <linux/pagemap.h> |
| 7 | +#include <linux/pagevec.h> | |
| 7 | 8 | #include <linux/sched.h> |
| 8 | 9 | #include <linux/slab.h> |
| 9 | 10 | #include <linux/sizes.h> |
| @@ -20,39 +21,40 @@ static noinline int process_page_range(struct inode *inode, u64 start, u64 end, · 10 unmodified lines | ||
| 20 | 21 | unsigned long flags) |
| 21 | 22 | { |
| 22 | 23 | int ret; |
| 23 | − struct page *pages[16]; | |
| 24 | + struct folio_batch fbatch; | |
| 24 | 25 | unsigned long index = start >> PAGE_SHIFT; |
| 25 | 26 | unsigned long end_index = end >> PAGE_SHIFT; |
| 26 | − unsigned long nr_pages = end_index - index + 1; | |
| 27 | 27 | int i; |
| 28 | 28 | int count = 0; |
| 29 | 29 | int loops = 0; |
| 30 | 30 | |
| 31 | − while (nr_pages > 0) { | |
| 32 | − ret = find_get_pages_contig(inode->i_mapping, index, | |
| 33 | − min_t(unsigned long, nr_pages, | |
| 34 | − ARRAY_SIZE(pages)), pages); | |
| 31 | + folio_batch_init(&fbatch); | |
| 32 | + | |
| 33 | + while (index <= end_index) { | |
| 34 | + ret = filemap_get_folios_contig(inode->i_mapping, &index, | |
| 35 | + end_index, &fbatch); | |
| 35 | 36 | for (i = 0; i < ret; i++) { |
| 37 | + struct folio *folio = fbatch.folios[i]; | |
| 38 | + | |
| 36 | 39 | if (flags & PROCESS_TEST_LOCKED && |
| 37 | − !PageLocked(pages[i])) | |
| 40 | + !folio_test_locked(folio)) | |
| 38 | 41 | count++; |
| 39 | − if (flags & PROCESS_UNLOCK && PageLocked(pages[i])) | |
| 40 | − unlock_page(pages[i]); | |
| 41 | − put_page(pages[i]); | |
| 42 | + if (flags & PROCESS_UNLOCK && folio_test_locked(folio)) | |
| 43 | + folio_unlock(folio); | |
| 42 | 44 | if (flags & PROCESS_RELEASE) |
| 43 | − put_page(pages[i]); | |
| 45 | + folio_put(folio); | |
| 44 | 46 | } |
| 45 | − nr_pages -= ret; | |
| 46 | − index += ret; | |
| 47 | + folio_batch_release(&fbatch); | |
| 47 | 48 | cond_resched(); |
| 48 | 49 | loops++; |
| 49 | 50 | if (loops > 100000) { |
| 50 | 51 | printk(KERN_ERR |
| 51 | − "stuck in a loop, start %llu, end %llu, nr_pages %lu, ret %d\n", | |
| 52 | − start, end, nr_pages, ret); | |
| 52 | + "stuck in a loop, start %llu, end %llu, ret %d\n", | |
| 53 | + start, end, ret); | |
| 53 | 54 | break; |
| 54 | 55 | } |
| 55 | 56 | } |
| 57 | + | |
| 56 | 58 | return count; |
| 57 | 59 | } |
| 58 | 60 | |
| @@ -152,7 +152,7 @@ static void __end_buffer_read_notouch(struct buffer_head *bh, int uptodate) · 151 unmodified lines | ||
| 152 | 152 | |
| 153 | 153 | /* |
| 154 | 154 | * Default synchronous end-of-IO handler.. Just mark it up-to-date and |
| 155 | − * unlock the buffer. This is what ll_rw_block uses too. | |
| 155 | + * unlock the buffer. | |
| 156 | 156 | */ |
| 157 | 157 | void end_buffer_read_sync(struct buffer_head *bh, int uptodate) |
| 158 | 158 | { |
| @@ -491,8 +491,8 @@ int inode_has_buffers(struct inode *inode) · 332 unmodified lines | ||
| 491 | 491 | * all already-submitted IO to complete, but does not queue any new |
| 492 | 492 | * writes to the disk. |
| 493 | 493 | * |
| 494 | − * To do O_SYNC writes, just queue the buffer writes with ll_rw_block as | |
| 495 | − * you dirty the buffers, and then use osync_inode_buffers to wait for | |
| 494 | + * To do O_SYNC writes, just queue the buffer writes with write_dirty_buffer | |
| 495 | + * as you dirty the buffers, and then use osync_inode_buffers to wait for | |
| 496 | 496 | * completion. Any other dirty buffers which are not yet queued for |
| 497 | 497 | * write will not be flushed to disk by the osync. |
| 498 | 498 | */ |
| @@ -562,7 +562,7 @@ void write_boundary_block(struct block_device *bdev, · 63 unmodified lines | ||
| 562 | 562 | struct buffer_head *bh = __find_get_block(bdev, bblock + 1, blocksize); |
| 563 | 563 | if (bh) { |
| 564 | 564 | if (buffer_dirty(bh)) |
| 565 | − ll_rw_block(REQ_OP_WRITE, 1, &bh); | |
| 565 | + write_dirty_buffer(bh, 0); | |
| 566 | 566 | put_bh(bh); |
| 567 | 567 | } |
| 568 | 568 | } |
| @@ -1342,23 +1342,12 @@ void __breadahead(struct block_device *bdev, sector_t block, unsigned size) · 773 unmodified lines | ||
| 1342 | 1342 | { |
| 1343 | 1343 | struct buffer_head *bh = __getblk(bdev, block, size); |
| 1344 | 1344 | if (likely(bh)) { |
| 1345 | − ll_rw_block(REQ_OP_READ | REQ_RAHEAD, 1, &bh); | |
| 1345 | + bh_readahead(bh, REQ_RAHEAD); | |
| 1346 | 1346 | brelse(bh); |
| 1347 | 1347 | } |
| 1348 | 1348 | } |
| 1349 | 1349 | EXPORT_SYMBOL(__breadahead); |
| 1350 | 1350 | |
| 1351 | −void __breadahead_gfp(struct block_device *bdev, sector_t block, unsigned size, | |
| 1352 | − gfp_t gfp) | |
| 1353 | −{ | |
| 1354 | − struct buffer_head *bh = __getblk_gfp(bdev, block, size, gfp); | |
| 1355 | − if (likely(bh)) { | |
| 1356 | − ll_rw_block(REQ_OP_READ | REQ_RAHEAD, 1, &bh); | |
| 1357 | − brelse(bh); | |
| 1358 | − } | |
| 1359 | −} | |
| 1360 | −EXPORT_SYMBOL(__breadahead_gfp); | |
| 1361 | − | |
| 1362 | 1351 | /** |
| 1363 | 1352 | * __bread_gfp() - reads a specified block and returns the bh |
| 1364 | 1353 | * @bdev: the block_device to read from |
| @@ -1817,7 +1806,7 @@ done: · 452 unmodified lines | ||
| 1817 | 1806 | /* |
| 1818 | 1807 | * The page was marked dirty, but the buffers were |
| 1819 | 1808 | * clean. Someone wrote them back by hand with |
| 1820 | − * ll_rw_block/submit_bh. A rare case. | |
| 1809 | + * write_dirty_buffer/submit_bh. A rare case. | |
| 1821 | 1810 | */ |
| 1822 | 1811 | end_page_writeback(page); |
| 1823 | 1812 | |
| @@ -2033,7 +2022,7 @@ int __block_write_begin_int(struct folio *folio, loff_t pos, unsigned len, · 209 unmodified lines | ||
| 2033 | 2022 | if (!buffer_uptodate(bh) && !buffer_delay(bh) && |
| 2034 | 2023 | !buffer_unwritten(bh) && |
| 2035 | 2024 | (block_start < from || block_end > to)) { |
| 2036 | − ll_rw_block(REQ_OP_READ, 1, &bh); | |
| 2025 | + bh_read_nowait(bh, 0); | |
| 2037 | 2026 | *wait_bh++=bh; |
| 2038 | 2027 | } |
| 2039 | 2028 | } |
| @@ -2352,7 +2341,7 @@ int generic_cont_expand_simple(struct inode *inode, loff_t size) · 312 unmodified lines | ||
| 2352 | 2341 | struct address_space *mapping = inode->i_mapping; |
| 2353 | 2342 | const struct address_space_operations *aops = mapping->a_ops; |
| 2354 | 2343 | struct page *page; |
| 2355 | − void *fsdata; | |
| 2344 | + void *fsdata = NULL; | |
| 2356 | 2345 | int err; |
| 2357 | 2346 | |
| 2358 | 2347 | err = inode_newsize_ok(inode, size); |
| @@ -2378,7 +2367,7 @@ static int cont_expand_zero(struct file *file, struct address_space *mapping, · 19 unmodified lines | ||
| 2378 | 2367 | const struct address_space_operations *aops = mapping->a_ops; |
| 2379 | 2368 | unsigned int blocksize = i_blocksize(inode); |
| 2380 | 2369 | struct page *page; |
| 2381 | − void *fsdata; | |
| 2370 | + void *fsdata = NULL; | |
| 2382 | 2371 | pgoff_t index, curidx; |
| 2383 | 2372 | loff_t curpos; |
| 2384 | 2373 | unsigned zerofrom, offset, len; |
| @@ -2593,11 +2582,9 @@ int block_truncate_page(struct address_space *mapping, · 208 unmodified lines | ||
| 2593 | 2582 | set_buffer_uptodate(bh); |
| 2594 | 2583 | |
| 2595 | 2584 | if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh)) { |
| 2596 | − err = -EIO; | |
| 2597 | − ll_rw_block(REQ_OP_READ, 1, &bh); | |
| 2598 | − wait_on_buffer(bh); | |
| 2585 | + err = bh_read(bh, 0); | |
| 2599 | 2586 | /* Uhhuh. Read error. Complain and punt. */ |
| 2600 | − if (!buffer_uptodate(bh)) | |
| 2587 | + if (err < 0) | |
| 2601 | 2588 | goto unlock; |
| 2602 | 2589 | } |
| 2603 | 2590 | |
| @@ -2725,61 +2712,6 @@ void submit_bh(blk_opf_t opf, struct buffer_head *bh) · 121 unmodified lines | ||
| 2725 | 2712 | } |
| 2726 | 2713 | EXPORT_SYMBOL(submit_bh); |
| 2727 | 2714 | |
| 2728 | −/** | |
| 2729 | − * ll_rw_block: low-level access to block devices (DEPRECATED) | |
| 2730 | − * @opf: block layer request operation and flags. | |
| 2731 | − * @nr: number of &struct buffer_heads in the array | |
| 2732 | − * @bhs: array of pointers to &struct buffer_head | |
| 2733 | − * | |
| 2734 | − * ll_rw_block() takes an array of pointers to &struct buffer_heads, and | |
| 2735 | − * requests an I/O operation on them, either a %REQ_OP_READ or a %REQ_OP_WRITE. | |
| 2736 | − * @opf contains flags modifying the detailed I/O behavior, most notably | |
| 2737 | − * %REQ_RAHEAD. | |
| 2738 | − * | |
| 2739 | − * This function drops any buffer that it cannot get a lock on (with the | |
| 2740 | − * BH_Lock state bit), any buffer that appears to be clean when doing a write | |
| 2741 | − * request, and any buffer that appears to be up-to-date when doing read | |
| 2742 | − * request. Further it marks as clean buffers that are processed for | |
| 2743 | − * writing (the buffer cache won't assume that they are actually clean | |
| 2744 | − * until the buffer gets unlocked). | |
| 2745 | − * | |
| 2746 | − * ll_rw_block sets b_end_io to simple completion handler that marks | |
| 2747 | − * the buffer up-to-date (if appropriate), unlocks the buffer and wakes | |
| 2748 | − * any waiters. | |
| 2749 | − * | |
| 2750 | − * All of the buffers must be for the same device, and must also be a | |
| 2751 | − * multiple of the current approved size for the device. | |
| 2752 | − */ | |
| 2753 | −void ll_rw_block(const blk_opf_t opf, int nr, struct buffer_head *bhs[]) | |
| 2754 | −{ | |
| 2755 | − const enum req_op op = opf & REQ_OP_MASK; | |
| 2756 | − int i; | |
| 2757 | − | |
| 2758 | − for (i = 0; i < nr; i++) { | |
| 2759 | − struct buffer_head *bh = bhs[i]; | |
| 2760 | − | |
| 2761 | − if (!trylock_buffer(bh)) | |
| 2762 | − continue; | |
| 2763 | − if (op == REQ_OP_WRITE) { | |
| 2764 | − if (test_clear_buffer_dirty(bh)) { | |
| 2765 | − bh->b_end_io = end_buffer_write_sync; | |
| 2766 | − get_bh(bh); | |
| 2767 | − submit_bh(opf, bh); | |
| 2768 | − continue; | |
| 2769 | − } | |
| 2770 | − } else { | |
| 2771 | − if (!buffer_uptodate(bh)) { | |
| 2772 | − bh->b_end_io = end_buffer_read_sync; | |
| 2773 | − get_bh(bh); | |
| 2774 | − submit_bh(opf, bh); | |
| 2775 | − continue; | |
| 2776 | − } | |
| 2777 | − } | |
| 2778 | − unlock_buffer(bh); | |
| 2779 | − } | |
| 2780 | −} | |
| 2781 | −EXPORT_SYMBOL(ll_rw_block); | |
| 2782 | − | |
| 2783 | 2715 | void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags) |
| 2784 | 2716 | { |
| 2785 | 2717 | lock_buffer(bh); |
| @@ -3026,29 +2958,69 @@ int bh_uptodate_or_lock(struct buffer_head *bh) · 240 unmodified lines | ||
| 3026 | 2958 | EXPORT_SYMBOL(bh_uptodate_or_lock); |
| 3027 | 2959 | |
| 3028 | 2960 | /** |
| 3029 | − * bh_submit_read - Submit a locked buffer for reading | |
| 2961 | + * __bh_read - Submit read for a locked buffer | |
| 3030 | 2962 | * @bh: struct buffer_head |
| 2963 | + * @op_flags: appending REQ_OP_* flags besides REQ_OP_READ | |
| 2964 | + * @wait: wait until reading finish | |
| 3031 | 2965 | * |
| 3032 | − * Returns zero on success and -EIO on error. | |
| 2966 | + * Returns zero on success or don't wait, and -EIO on error. | |
| 3033 | 2967 | */ |
| 3034 | −int bh_submit_read(struct buffer_head *bh) | |
| 2968 | +int __bh_read(struct buffer_head *bh, blk_opf_t op_flags, bool wait) | |
| 3035 | 2969 | { |
| 3036 | − BUG_ON(!buffer_locked(bh)); | |
| 2970 | + int ret = 0; | |
| 3037 | 2971 | |
| 3038 | − if (buffer_uptodate(bh)) { | |
| 3039 | − unlock_buffer(bh); | |
| 3040 | − return 0; | |
| 3041 | − } | |
| 2972 | + BUG_ON(!buffer_locked(bh)); | |
| 3042 | 2973 | |
| 3043 | 2974 | get_bh(bh); |
| 3044 | 2975 | bh->b_end_io = end_buffer_read_sync; |
| 3045 | − submit_bh(REQ_OP_READ, bh); | |
| 3046 | − wait_on_buffer(bh); | |
| 3047 | − if (buffer_uptodate(bh)) | |
| 3048 | − return 0; | |
| 3049 | − return -EIO; | |
| 2976 | + submit_bh(REQ_OP_READ | op_flags, bh); | |
| 2977 | + if (wait) { | |
| 2978 | + wait_on_buffer(bh); | |
| 2979 | + if (!buffer_uptodate(bh)) | |
| 2980 | + ret = -EIO; | |
| 2981 | + } | |
| 2982 | + return ret; | |
| 2983 | +} | |
| 2984 | +EXPORT_SYMBOL(__bh_read); | |
| 2985 | + | |
| 2986 | +/** | |
| 2987 | + * __bh_read_batch - Submit read for a batch of unlocked buffers | |
| 2988 | + * @nr: entry number of the buffer batch | |
| 2989 | + * @bhs: a batch of struct buffer_head | |
| 2990 | + * @op_flags: appending REQ_OP_* flags besides REQ_OP_READ | |
| 2991 | + * @force_lock: force to get a lock on the buffer if set, otherwise drops any | |
| 2992 | + * buffer that cannot lock. | |
| 2993 | + * | |
| 2994 | + * Returns zero on success or don't wait, and -EIO on error. | |
| 2995 | + */ | |
| 2996 | +void __bh_read_batch(int nr, struct buffer_head *bhs[], | |
| 2997 | + blk_opf_t op_flags, bool force_lock) | |
| 2998 | +{ | |
| 2999 | + int i; | |
| 3000 | + | |
| 3001 | + for (i = 0; i < nr; i++) { | |
| 3002 | + struct buffer_head *bh = bhs[i]; | |
| 3003 | + | |
| 3004 | + if (buffer_uptodate(bh)) | |
| 3005 | + continue; | |
| 3006 | + | |
| 3007 | + if (force_lock) | |
| 3008 | + lock_buffer(bh); | |
| 3009 | + else | |
| 3010 | + if (!trylock_buffer(bh)) | |
| 3011 | + continue; | |
| 3012 | + | |
| 3013 | + if (buffer_uptodate(bh)) { | |
| 3014 | + unlock_buffer(bh); | |
| 3015 | + continue; | |
| 3016 | + } | |
| 3017 | + | |
| 3018 | + bh->b_end_io = end_buffer_read_sync; | |
| 3019 | + get_bh(bh); | |
| 3020 | + submit_bh(REQ_OP_READ | op_flags, bh); | |
| 3021 | + } | |
| 3050 | 3022 | } |
| 3051 | −EXPORT_SYMBOL(bh_submit_read); | |
| 3023 | +EXPORT_SYMBOL(__bh_read_batch); | |
| 3052 | 3024 | |
| 3053 | 3025 | void __init buffer_init(void) |
| 3054 | 3026 | { |
| @@ -1100,30 +1100,20 @@ whole: · 1099 unmodified lines | ||
| 1100 | 1100 | return vma->vm_end - vma->vm_start; |
| 1101 | 1101 | } |
| 1102 | 1102 | |
| 1103 | −static struct vm_area_struct *first_vma(struct task_struct *tsk, | |
| 1104 | − struct vm_area_struct *gate_vma) | |
| 1105 | −{ | |
| 1106 | − struct vm_area_struct *ret = tsk->mm->mmap; | |
| 1107 | − | |
| 1108 | − if (ret) | |
| 1109 | − return ret; | |
| 1110 | − return gate_vma; | |
| 1111 | −} | |
| 1112 | − | |
| 1113 | 1103 | /* |
| 1114 | 1104 | * Helper function for iterating across a vma list. It ensures that the caller |
| 1115 | 1105 | * will visit `gate_vma' prior to terminating the search. |
| 1116 | 1106 | */ |
| 1117 | −static struct vm_area_struct *next_vma(struct vm_area_struct *this_vma, | |
| 1107 | +static struct vm_area_struct *coredump_next_vma(struct ma_state *mas, | |
| 1108 | + struct vm_area_struct *vma, | |
| 1118 | 1109 | struct vm_area_struct *gate_vma) |
| 1119 | 1110 | { |
| 1120 | − struct vm_area_struct *ret; | |
| 1121 | − | |
| 1122 | − ret = this_vma->vm_next; | |
| 1123 | − if (ret) | |
| 1124 | − return ret; | |
| 1125 | − if (this_vma == gate_vma) | |
| 1111 | + if (gate_vma && (vma == gate_vma)) | |
| 1126 | 1112 | return NULL; |
| 1113 | + | |
| 1114 | + vma = mas_next(mas, ULONG_MAX); | |
| 1115 | + if (vma) | |
| 1116 | + return vma; | |
| 1127 | 1117 | return gate_vma; |
| 1128 | 1118 | } |
| 1129 | 1119 | |
| @@ -1147,9 +1137,10 @@ static void free_vma_snapshot(struct coredump_params *cprm) · 17 unmodified lines | ||
| 1147 | 1137 | */ |
| 1148 | 1138 | static bool dump_vma_snapshot(struct coredump_params *cprm) |
| 1149 | 1139 | { |
| 1150 | − struct vm_area_struct *vma, *gate_vma; | |
| 1140 | + struct vm_area_struct *gate_vma, *vma = NULL; | |
| 1151 | 1141 | struct mm_struct *mm = current->mm; |
| 1152 | − int i; | |
| 1142 | + MA_STATE(mas, &mm->mm_mt, 0, 0); | |
| 1143 | + int i = 0; | |
| 1153 | 1144 | |
| 1154 | 1145 | /* |
| 1155 | 1146 | * Once the stack expansion code is fixed to not change VMA bounds |
| @@ -1169,8 +1160,7 @@ static bool dump_vma_snapshot(struct coredump_params *cprm) · 13 unmodified lines | ||
| 1169 | 1160 | return false; |
| 1170 | 1161 | } |
| 1171 | 1162 | |
| 1172 | − for (i = 0, vma = first_vma(current, gate_vma); vma != NULL; | |
| 1173 | − vma = next_vma(vma, gate_vma), i++) { | |
| 1163 | + while ((vma = coredump_next_vma(&mas, vma, gate_vma)) != NULL) { | |
| 1174 | 1164 | struct core_vma_metadata *m = cprm->vma_meta + i; |
| 1175 | 1165 | |
| 1176 | 1166 | m->start = vma->vm_start; |
| @@ -1178,10 +1168,10 @@ static bool dump_vma_snapshot(struct coredump_params *cprm) · 1 unmodified line | ||
| 1178 | 1168 | m->flags = vma->vm_flags; |
| 1179 | 1169 | m->dump_size = vma_dump_size(vma, cprm->mm_flags); |
| 1180 | 1170 | m->pgoff = vma->vm_pgoff; |
| 1181 | − | |
| 1182 | 1171 | m->file = vma->vm_file; |
| 1183 | 1172 | if (m->file) |
| 1184 | 1173 | get_file(m->file); |
| 1174 | + i++; | |
| 1185 | 1175 | } |
| 1186 | 1176 | |
| 1187 | 1177 | mmap_write_unlock(mm); |
| @@ -28,7 +28,6 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <linux/file.h> |
| 29 | 29 | #include <linux/fdtable.h> |
| 30 | 30 | #include <linux/mm.h> |
| 31 | −#include <linux/vmacache.h> | |
| 32 | 31 | #include <linux/stat.h> |
| 33 | 32 | #include <linux/fcntl.h> |
| 34 | 33 | #include <linux/swap.h> |
| @@ -683,6 +682,8 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift) · 648 unmodified lines | ||
| 683 | 682 | unsigned long length = old_end - old_start; |
| 684 | 683 | unsigned long new_start = old_start - shift; |
| 685 | 684 | unsigned long new_end = old_end - shift; |
| 685 | + VMA_ITERATOR(vmi, mm, new_start); | |
| 686 | + struct vm_area_struct *next; | |
| 686 | 687 | struct mmu_gather tlb; |
| 687 | 688 | |
| 688 | 689 | BUG_ON(new_start > new_end); |
| @@ -691,7 +692,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift) · 2 unmodified lines | ||
| 691 | 692 | * ensure there are no vmas between where we want to go |
| 692 | 693 | * and where we are |
| 693 | 694 | */ |
| 694 | − if (vma != find_vma(mm, new_start)) | |
| 695 | + if (vma != vma_next(&vmi)) | |
| 695 | 696 | return -EFAULT; |
| 696 | 697 | |
| 697 | 698 | /* |
| @@ -710,12 +711,13 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift) · 12 unmodified lines | ||
| 710 | 711 | |
| 711 | 712 | lru_add_drain(); |
| 712 | 713 | tlb_gather_mmu(&tlb, mm); |
| 714 | + next = vma_next(&vmi); | |
| 713 | 715 | if (new_end > old_start) { |
| 714 | 716 | /* |
| 715 | 717 | * when the old and new regions overlap clear from new_end. |
| 716 | 718 | */ |
| 717 | 719 | free_pgd_range(&tlb, new_end, old_end, new_end, |
| 718 | − vma->vm_next ? vma->vm_next->vm_start : USER_PGTABLES_CEILING); | |
| 720 | + next ? next->vm_start : USER_PGTABLES_CEILING); | |
| 719 | 721 | } else { |
| 720 | 722 | /* |
| 721 | 723 | * otherwise, clean from old_start; this is done to not touch |
| @@ -724,7 +726,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift) · 2 unmodified lines | ||
| 724 | 726 | * for the others its just a little faster. |
| 725 | 727 | */ |
| 726 | 728 | free_pgd_range(&tlb, old_start, old_end, new_end, |
| 727 | − vma->vm_next ? vma->vm_next->vm_start : USER_PGTABLES_CEILING); | |
| 729 | + next ? next->vm_start : USER_PGTABLES_CEILING); | |
| 728 | 730 | } |
| 729 | 731 | tlb_finish_mmu(&tlb); |
| 730 | 732 | |
| @@ -1010,6 +1012,7 @@ static int exec_mmap(struct mm_struct *mm) · 279 unmodified lines | ||
| 1010 | 1012 | active_mm = tsk->active_mm; |
| 1011 | 1013 | tsk->active_mm = mm; |
| 1012 | 1014 | tsk->mm = mm; |
| 1015 | + lru_gen_add_mm(mm); | |
| 1013 | 1016 | /* |
| 1014 | 1017 | * This prevents preemption while active_mm is being loaded and |
| 1015 | 1018 | * it and mm are being updated, which could cause problems for |
| @@ -1022,9 +1025,8 @@ static int exec_mmap(struct mm_struct *mm) · 6 unmodified lines | ||
| 1022 | 1025 | activate_mm(active_mm, mm); |
| 1023 | 1026 | if (IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM)) |
| 1024 | 1027 | local_irq_enable(); |
| 1025 | − tsk->mm->vmacache_seqnum = 0; | |
| 1026 | − vmacache_flush(tsk); | |
| 1027 | 1028 | task_unlock(tsk); |
| 1029 | + lru_gen_use_mm(mm); | |
| 1028 | 1030 | if (old_mm) { |
| 1029 | 1031 | mmap_read_unlock(old_mm); |
| 1030 | 1032 | BUG_ON(active_mm != old_mm); |
| @@ -126,6 +126,7 @@ read_block_bitmap(struct super_block *sb, unsigned int block_group) · 125 unmodified lines | ||
| 126 | 126 | struct ext2_group_desc * desc; |
| 127 | 127 | struct buffer_head * bh = NULL; |
| 128 | 128 | ext2_fsblk_t bitmap_blk; |
| 129 | + int ret; | |
| 129 | 130 | |
| 130 | 131 | desc = ext2_get_group_desc(sb, block_group, NULL); |
| 131 | 132 | if (!desc) |
| @@ -139,10 +140,10 @@ read_block_bitmap(struct super_block *sb, unsigned int block_group) · 7 unmodified lines | ||
| 139 | 140 | block_group, le32_to_cpu(desc->bg_block_bitmap)); |
| 140 | 141 | return NULL; |
| 141 | 142 | } |
| 142 | − if (likely(bh_uptodate_or_lock(bh))) | |
| 143 | + ret = bh_read(bh, 0); | |
| 144 | + if (ret > 0) | |
| 143 | 145 | return bh; |
| 144 | − | |
| 145 | − if (bh_submit_read(bh) < 0) { | |
| 146 | + if (ret < 0) { | |
| 146 | 147 | brelse(bh); |
| 147 | 148 | ext2_error(sb, __func__, |
| 148 | 149 | "Cannot read block bitmap - " |
| @@ -776,7 +776,8 @@ static int fuse_check_page(struct page *page) · 775 unmodified lines | ||
| 776 | 776 | 1 << PG_active | |
| 777 | 777 | 1 << PG_workingset | |
| 778 | 778 | 1 << PG_reclaim | |
| 779 | − 1 << PG_waiters))) { | |
| 779 | + 1 << PG_waiters | | |
| 780 | + LRU_GEN_MASK | LRU_REFS_MASK))) { | |
| 780 | 781 | dump_page(page, "fuse: trying to steal weird page"); |
| 781 | 782 | return 1; |
| 782 | 783 | } |
| @@ -525,8 +525,7 @@ struct buffer_head *gfs2_meta_ra(struct gfs2_glock *gl, u64 dblock, u32 extlen) · 524 unmodified lines | ||
| 525 | 525 | |
| 526 | 526 | if (buffer_uptodate(first_bh)) |
| 527 | 527 | goto out; |
| 528 | − if (!buffer_locked(first_bh)) | |
| 529 | − ll_rw_block(REQ_OP_READ | REQ_META | REQ_PRIO, 1, &first_bh); | |
| 528 | + bh_read_nowait(first_bh, REQ_META | REQ_PRIO); | |
| 530 | 529 | |
| 531 | 530 | dblock++; |
| 532 | 531 | extlen--; |
| @@ -534,9 +533,7 @@ struct buffer_head *gfs2_meta_ra(struct gfs2_glock *gl, u64 dblock, u32 extlen) · 1 unmodified line | ||
| 534 | 533 | while (extlen) { |
| 535 | 534 | bh = gfs2_getbuf(gl, dblock, CREATE); |
| 536 | 535 | |
| 537 | − if (!buffer_uptodate(bh) && !buffer_locked(bh)) | |
| 538 | − ll_rw_block(REQ_OP_READ | REQ_RAHEAD | REQ_META | | |
| 539 | − REQ_PRIO, 1, &bh); | |
| 536 | + bh_readahead(bh, REQ_RAHEAD | REQ_META | REQ_PRIO); | |
| 540 | 537 | brelse(bh); |
| 541 | 538 | dblock++; |
| 542 | 539 | extlen--; |
| @@ -745,12 +745,8 @@ static int gfs2_write_buf_to_page(struct gfs2_inode *ip, unsigned long index, · 744 unmodified lines | ||
| 745 | 745 | } |
| 746 | 746 | if (PageUptodate(page)) |
| 747 | 747 | set_buffer_uptodate(bh); |
| 748 | − if (!buffer_uptodate(bh)) { | |
| 749 | − ll_rw_block(REQ_OP_READ | REQ_META | REQ_PRIO, 1, &bh); | |
| 750 | − wait_on_buffer(bh); | |
| 751 | − if (!buffer_uptodate(bh)) | |
| 752 | − goto unlock_out; | |
| 753 | − } | |
| 748 | + if (bh_read(bh, REQ_META | REQ_PRIO) < 0) | |
| 749 | + goto unlock_out; | |
| 754 | 750 | if (gfs2_is_jdata(ip)) |
| 755 | 751 | gfs2_trans_add_data(ip->i_gl, bh); |
| 756 | 752 | else |
| @@ -364,13 +364,155 @@ static int hugetlbfs_write_end(struct file *file, struct address_space *mapping, · 363 unmodified lines | ||
| 364 | 364 | return -EINVAL; |
| 365 | 365 | } |
| 366 | 366 | |
| 367 | −static void remove_huge_page(struct page *page) | |
| 367 | +static void hugetlb_delete_from_page_cache(struct page *page) | |
| 368 | 368 | { |
| 369 | 369 | ClearPageDirty(page); |
| 370 | 370 | ClearPageUptodate(page); |
| 371 | 371 | delete_from_page_cache(page); |
| 372 | 372 | } |
| 373 | 373 | |
| 374 | +/* | |
| 375 | + * Called with i_mmap_rwsem held for inode based vma maps. This makes | |
| 376 | + * sure vma (and vm_mm) will not go away. We also hold the hugetlb fault | |
| 377 | + * mutex for the page in the mapping. So, we can not race with page being | |
| 378 | + * faulted into the vma. | |
| 379 | + */ | |
| 380 | +static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, | |
| 381 | + unsigned long addr, struct page *page) | |
| 382 | +{ | |
| 383 | + pte_t *ptep, pte; | |
| 384 | + | |
| 385 | + ptep = huge_pte_offset(vma->vm_mm, addr, | |
| 386 | + huge_page_size(hstate_vma(vma))); | |
| 387 | + | |
| 388 | + if (!ptep) | |
| 389 | + return false; | |
| 390 | + | |
| 391 | + pte = huge_ptep_get(ptep); | |
| 392 | + if (huge_pte_none(pte) || !pte_present(pte)) | |
| 393 | + return false; | |
| 394 | + | |
| 395 | + if (pte_page(pte) == page) | |
| 396 | + return true; | |
| 397 | + | |
| 398 | + return false; | |
| 399 | +} | |
| 400 | + | |
| 401 | +/* | |
| 402 | + * Can vma_offset_start/vma_offset_end overflow on 32-bit arches? | |
| 403 | + * No, because the interval tree returns us only those vmas | |
| 404 | + * which overlap the truncated area starting at pgoff, | |
| 405 | + * and no vma on a 32-bit arch can span beyond the 4GB. | |
| 406 | + */ | |
| 407 | +static unsigned long vma_offset_start(struct vm_area_struct *vma, pgoff_t start) | |
| 408 | +{ | |
| 409 | + if (vma->vm_pgoff < start) | |
| 410 | + return (start - vma->vm_pgoff) << PAGE_SHIFT; | |
| 411 | + else | |
| 412 | + return 0; | |
| 413 | +} | |
| 414 | + | |
| 415 | +static unsigned long vma_offset_end(struct vm_area_struct *vma, pgoff_t end) | |
| 416 | +{ | |
| 417 | + unsigned long t_end; | |
| 418 | + | |
| 419 | + if (!end) | |
| 420 | + return vma->vm_end; | |
| 421 | + | |
| 422 | + t_end = ((end - vma->vm_pgoff) << PAGE_SHIFT) + vma->vm_start; | |
| 423 | + if (t_end > vma->vm_end) | |
| 424 | + t_end = vma->vm_end; | |
| 425 | + return t_end; | |
| 426 | +} | |
| 427 | + | |
| 428 | +/* | |
| 429 | + * Called with hugetlb fault mutex held. Therefore, no more mappings to | |
| 430 | + * this folio can be created while executing the routine. | |
| 431 | + */ | |
| 432 | +static void hugetlb_unmap_file_folio(struct hstate *h, | |
| 433 | + struct address_space *mapping, | |
| 434 | + struct folio *folio, pgoff_t index) | |
| 435 | +{ | |
| 436 | + struct rb_root_cached *root = &mapping->i_mmap; | |
| 437 | + struct hugetlb_vma_lock *vma_lock; | |
| 438 | + struct page *page = &folio->page; | |
| 439 | + struct vm_area_struct *vma; | |
| 440 | + unsigned long v_start; | |
| 441 | + unsigned long v_end; | |
| 442 | + pgoff_t start, end; | |
| 443 | + | |
| 444 | + start = index * pages_per_huge_page(h); | |
| 445 | + end = (index + 1) * pages_per_huge_page(h); | |
| 446 | + | |
| 447 | + i_mmap_lock_write(mapping); | |
| 448 | +retry: | |
| 449 | + vma_lock = NULL; | |
| 450 | + vma_interval_tree_foreach(vma, root, start, end - 1) { | |
| 451 | + v_start = vma_offset_start(vma, start); | |
| 452 | + v_end = vma_offset_end(vma, end); | |
| 453 | + | |
| 454 | + if (!hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) | |
| 455 | + continue; | |
| 456 | + | |
| 457 | + if (!hugetlb_vma_trylock_write(vma)) { | |
| 458 | + vma_lock = vma->vm_private_data; | |
| 459 | + /* | |
| 460 | + * If we can not get vma lock, we need to drop | |
| 461 | + * immap_sema and take locks in order. First, | |
| 462 | + * take a ref on the vma_lock structure so that | |
| 463 | + * we can be guaranteed it will not go away when | |
| 464 | + * dropping immap_sema. | |
| 465 | + */ | |
| 466 | + kref_get(&vma_lock->refs); | |
| 467 | + break; | |
| 468 | + } | |
| 469 | + | |
| 470 | + unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, | |
| 471 | + NULL, ZAP_FLAG_DROP_MARKER); | |
| 472 | + hugetlb_vma_unlock_write(vma); | |
| 473 | + } | |
| 474 | + | |
| 475 | + i_mmap_unlock_write(mapping); | |
| 476 | + | |
| 477 | + if (vma_lock) { | |
| 478 | + /* | |
| 479 | + * Wait on vma_lock. We know it is still valid as we have | |
| 480 | + * a reference. We must 'open code' vma locking as we do | |
| 481 | + * not know if vma_lock is still attached to vma. | |
| 482 | + */ | |
| 483 | + down_write(&vma_lock->rw_sema); | |
| 484 | + i_mmap_lock_write(mapping); | |
| 485 | + | |
| 486 | + vma = vma_lock->vma; | |
| 487 | + if (!vma) { | |
| 488 | + /* | |
| 489 | + * If lock is no longer attached to vma, then just | |
| 490 | + * unlock, drop our reference and retry looking for | |
| 491 | + * other vmas. | |
| 492 | + */ | |
| 493 | + up_write(&vma_lock->rw_sema); | |
| 494 | + kref_put(&vma_lock->refs, hugetlb_vma_lock_release); | |
| 495 | + goto retry; | |
| 496 | + } | |
| 497 | + | |
| 498 | + /* | |
| 499 | + * vma_lock is still attached to vma. Check to see if vma | |
| 500 | + * still maps page and if so, unmap. | |
| 501 | + */ | |
| 502 | + v_start = vma_offset_start(vma, start); | |
| 503 | + v_end = vma_offset_end(vma, end); | |
| 504 | + if (hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) | |
| 505 | + unmap_hugepage_range(vma, vma->vm_start + v_start, | |
| 506 | + v_end, NULL, | |
| 507 | + ZAP_FLAG_DROP_MARKER); | |
| 508 | + | |
| 509 | + kref_put(&vma_lock->refs, hugetlb_vma_lock_release); | |
| 510 | + hugetlb_vma_unlock_write(vma); | |
| 511 | + | |
| 512 | + goto retry; | |
| 513 | + } | |
| 514 | +} | |
| 515 | + | |
| 374 | 516 | static void |
| 375 | 517 | hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, |
| 376 | 518 | zap_flags_t zap_flags) |
| @@ -383,32 +525,66 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, · 6 unmodified lines | ||
| 383 | 525 | * an inclusive "last". |
| 384 | 526 | */ |
| 385 | 527 | vma_interval_tree_foreach(vma, root, start, end ? end - 1 : ULONG_MAX) { |
| 386 | − unsigned long v_offset; | |
| 528 | + unsigned long v_start; | |
| 387 | 529 | unsigned long v_end; |
| 388 | 530 | |
| 531 | + if (!hugetlb_vma_trylock_write(vma)) | |
| 532 | + continue; | |
| 533 | + | |
| 534 | + v_start = vma_offset_start(vma, start); | |
| 535 | + v_end = vma_offset_end(vma, end); | |
| 536 | + | |
| 537 | + unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, | |
| 538 | + NULL, zap_flags); | |
| 539 | + | |
| 389 | 540 | /* |
| 390 | − * Can the expression below overflow on 32-bit arches? | |
| 391 | − * No, because the interval tree returns us only those vmas | |
| 392 | − * which overlap the truncated area starting at pgoff, | |
| 393 | − * and no vma on a 32-bit arch can span beyond the 4GB. | |
| 541 | + * Note that vma lock only exists for shared/non-private | |
| 542 | + * vmas. Therefore, lock is not held when calling | |
| 543 | + * unmap_hugepage_range for private vmas. | |
| 394 | 544 | */ |
| 395 | − if (vma->vm_pgoff < start) | |
| 396 | − v_offset = (start - vma->vm_pgoff) << PAGE_SHIFT; | |
| 397 | − else | |
| 398 | − v_offset = 0; | |
| 399 | − | |
| 400 | − if (!end) | |
| 401 | − v_end = vma->vm_end; | |
| 402 | − else { | |
| 403 | − v_end = ((end - vma->vm_pgoff) << PAGE_SHIFT) | |
| 404 | − + vma->vm_start; | |
| 405 | − if (v_end > vma->vm_end) | |
| 406 | − v_end = vma->vm_end; | |
| 407 | − } | |
| 545 | + hugetlb_vma_unlock_write(vma); | |
| 546 | + } | |
| 547 | +} | |
| 408 | 548 | |
| 409 | − unmap_hugepage_range(vma, vma->vm_start + v_offset, v_end, | |
| 410 | − NULL, zap_flags); | |
| 549 | +/* | |
| 550 | + * Called with hugetlb fault mutex held. | |
| 551 | + * Returns true if page was actually removed, false otherwise. | |
| 552 | + */ | |
| 553 | +static bool remove_inode_single_folio(struct hstate *h, struct inode *inode, | |
| 554 | + struct address_space *mapping, | |
| 555 | + struct folio *folio, pgoff_t index, | |
| 556 | + bool truncate_op) | |
| 557 | +{ | |
| 558 | + bool ret = false; | |
| 559 | + | |
| 560 | + /* | |
| 561 | + * If folio is mapped, it was faulted in after being | |
| 562 | + * unmapped in caller. Unmap (again) while holding | |
| 563 | + * the fault mutex. The mutex will prevent faults | |
| 564 | + * until we finish removing the folio. | |
| 565 | + */ | |
| 566 | + if (unlikely(folio_mapped(folio))) | |
| 567 | + hugetlb_unmap_file_folio(h, mapping, folio, index); | |
| 568 | + | |
| 569 | + folio_lock(folio); | |
| 570 | + /* | |
| 571 | + * We must remove the folio from page cache before removing | |
| 572 | + * the region/ reserve map (hugetlb_unreserve_pages). In | |
| 573 | + * rare out of memory conditions, removal of the region/reserve | |
| 574 | + * map could fail. Correspondingly, the subpool and global | |
| 575 | + * reserve usage count can need to be adjusted. | |
| 576 | + */ | |
| 577 | + VM_BUG_ON(HPageRestoreReserve(&folio->page)); | |
| 578 | + hugetlb_delete_from_page_cache(&folio->page); | |
| 579 | + ret = true; | |
| 580 | + if (!truncate_op) { | |
| 581 | + if (unlikely(hugetlb_unreserve_pages(inode, index, | |
| 582 | + index + 1, 1))) | |
| 583 | + hugetlb_fix_reserve_counts(inode); | |
| 411 | 584 | } |
| 585 | + | |
| 586 | + folio_unlock(folio); | |
| 587 | + return ret; | |
| 412 | 588 | } |
| 413 | 589 | |
| 414 | 590 | /* |
| @@ -418,10 +594,10 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, · 3 unmodified lines | ||
| 418 | 594 | * truncation is indicated by end of range being LLONG_MAX |
| 419 | 595 | * In this case, we first scan the range and release found pages. |
| 420 | 596 | * After releasing pages, hugetlb_unreserve_pages cleans up region/reserve |
| 421 | − * maps and global counts. Page faults can not race with truncation | |
| 422 | − * in this routine. hugetlb_no_page() holds i_mmap_rwsem and prevents | |
| 423 | − * page faults in the truncated range by checking i_size. i_size is | |
| 424 | − * modified while holding i_mmap_rwsem. | |
| 597 | + * maps and global counts. Page faults can race with truncation. | |
| 598 | + * During faults, hugetlb_no_page() checks i_size before page allocation, | |
| 599 | + * and again after obtaining page table lock. It will 'back out' | |
| 600 | + * allocations in the truncated range. | |
| 425 | 601 | * hole punch is indicated if end is not LLONG_MAX |
| 426 | 602 | * In the hole punch case we scan the range and release found pages. |
| 427 | 603 | * Only when releasing a page is the associated region/reserve map |
| @@ -451,61 +627,17 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, · 23 unmodified lines | ||
| 451 | 627 | u32 hash = 0; |
| 452 | 628 | |
| 453 | 629 | index = folio->index; |
| 454 | − if (!truncate_op) { | |
| 455 | − /* | |
| 456 | − * Only need to hold the fault mutex in the | |
| 457 | − * hole punch case. This prevents races with | |
| 458 | − * page faults. Races are not possible in the | |
| 459 | − * case of truncation. | |
| 460 | − */ | |
| 461 | − hash = hugetlb_fault_mutex_hash(mapping, index); | |
| 462 | − mutex_lock(&hugetlb_fault_mutex_table[hash]); | |
| 463 | − } | |
| 630 | + hash = hugetlb_fault_mutex_hash(mapping, index); | |
| 631 | + mutex_lock(&hugetlb_fault_mutex_table[hash]); | |
| 464 | 632 | |
| 465 | 633 | /* |
| 466 | − * If folio is mapped, it was faulted in after being | |
| 467 | − * unmapped in caller. Unmap (again) now after taking | |
| 468 | − * the fault mutex. The mutex will prevent faults | |
| 469 | − * until we finish removing the folio. | |
| 470 | − * | |
| 471 | − * This race can only happen in the hole punch case. | |
| 472 | − * Getting here in a truncate operation is a bug. | |
| 634 | + * Remove folio that was part of folio_batch. | |
| 473 | 635 | */ |
| 474 | − if (unlikely(folio_mapped(folio))) { | |
| 475 | − BUG_ON(truncate_op); | |
| 476 | − | |
| 477 | − mutex_unlock(&hugetlb_fault_mutex_table[hash]); | |
| 478 | − i_mmap_lock_write(mapping); | |
| 479 | − mutex_lock(&hugetlb_fault_mutex_table[hash]); | |
| 480 | − hugetlb_vmdelete_list(&mapping->i_mmap, | |
| 481 | − index * pages_per_huge_page(h), | |
| 482 | − (index + 1) * pages_per_huge_page(h), | |
| 483 | − ZAP_FLAG_DROP_MARKER); | |
| 484 | − i_mmap_unlock_write(mapping); | |
| 485 | − } | |
| 486 | − | |
| 487 | − folio_lock(folio); | |
| 488 | − /* | |
| 489 | − * We must free the huge page and remove from page | |
| 490 | − * cache (remove_huge_page) BEFORE removing the | |
| 491 | − * region/reserve map (hugetlb_unreserve_pages). In | |
| 492 | − * rare out of memory conditions, removal of the | |
| 493 | − * region/reserve map could fail. Correspondingly, | |
| 494 | − * the subpool and global reserve usage count can need | |
| 495 | − * to be adjusted. | |
| 496 | − */ | |
| 497 | − VM_BUG_ON(HPageRestoreReserve(&folio->page)); | |
| 498 | − remove_huge_page(&folio->page); | |
| 499 | − freed++; | |
| 500 | − if (!truncate_op) { | |
| 501 | − if (unlikely(hugetlb_unreserve_pages(inode, | |
| 502 | − index, index + 1, 1))) | |
| 503 | − hugetlb_fix_reserve_counts(inode); | |
| 504 | − } | |
| 505 | − | |
| 506 | − folio_unlock(folio); | |
| 507 | − if (!truncate_op) | |
| 508 | − mutex_unlock(&hugetlb_fault_mutex_table[hash]); | |
| 636 | + if (remove_inode_single_folio(h, inode, mapping, folio, | |
| 637 | + index, truncate_op)) | |
| 638 | + freed++; | |
| 639 | + | |
| 640 | + mutex_unlock(&hugetlb_fault_mutex_table[hash]); | |
| 509 | 641 | } |
| 510 | 642 | folio_batch_release(&fbatch); |
| 511 | 643 | cond_resched(); |
| @@ -543,8 +675,8 @@ static void hugetlb_vmtruncate(struct inode *inode, loff_t offset) · 31 unmodified lines | ||
| 543 | 675 | BUG_ON(offset & ~huge_page_mask(h)); |
| 544 | 676 | pgoff = offset >> PAGE_SHIFT; |
| 545 | 677 | |
| 546 | − i_mmap_lock_write(mapping); | |
| 547 | 678 | i_size_write(inode, offset); |
| 679 | + i_mmap_lock_write(mapping); | |
| 548 | 680 | if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) |
| 549 | 681 | hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0, |
| 550 | 682 | ZAP_FLAG_DROP_MARKER); |
| @@ -703,11 +835,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 152 unmodified lines | ||
| 703 | 835 | /* addr is the offset within the file (zero based) */ |
| 704 | 836 | addr = index * hpage_size; |
| 705 | 837 | |
| 706 | − /* | |
| 707 | − * fault mutex taken here, protects against fault path | |
| 708 | − * and hole punch. inode_lock previously taken protects | |
| 709 | − * against truncation. | |
| 710 | − */ | |
| 838 | + /* mutex taken here, fault path and hole punch */ | |
| 711 | 839 | hash = hugetlb_fault_mutex_hash(mapping, index); |
| 712 | 840 | mutex_lock(&hugetlb_fault_mutex_table[hash]); |
| 713 | 841 | |
| @@ -737,7 +865,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 23 unmodified lines | ||
| 737 | 865 | } |
| 738 | 866 | clear_huge_page(page, addr, pages_per_huge_page(h)); |
| 739 | 867 | __SetPageUptodate(page); |
| 740 | − error = huge_add_to_page_cache(page, mapping, index); | |
| 868 | + error = hugetlb_add_to_page_cache(page, mapping, index); | |
| 741 | 869 | if (unlikely(error)) { |
| 742 | 870 | restore_reserve_on_error(h, &pseudo_vma, addr, page); |
| 743 | 871 | put_page(page); |
| @@ -749,7 +877,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 5 unmodified lines | ||
| 749 | 877 | |
| 750 | 878 | SetHPageMigratable(page); |
| 751 | 879 | /* |
| 752 | − * unlock_page because locked by huge_add_to_page_cache() | |
| 880 | + * unlock_page because locked by hugetlb_add_to_page_cache() | |
| 753 | 881 | * put_page() due to reference from alloc_huge_page() |
| 754 | 882 | */ |
| 755 | 883 | unlock_page(page); |
| @@ -994,7 +1122,7 @@ static int hugetlbfs_error_remove_page(struct address_space *mapping, · 238 unmodified lines | ||
| 994 | 1122 | struct inode *inode = mapping->host; |
| 995 | 1123 | pgoff_t index = page->index; |
| 996 | 1124 | |
| 997 | − remove_huge_page(page); | |
| 1125 | + hugetlb_delete_from_page_cache(page); | |
| 998 | 1126 | if (unlikely(hugetlb_unreserve_pages(inode, index, index + 1, 1))) |
| 999 | 1127 | hugetlb_fix_reserve_counts(inode); |
| 1000 | 1128 | |
| @@ -82,7 +82,7 @@ static loff_t zisofs_uncompress_block(struct inode *inode, loff_t block_start, · 81 unmodified lines | ||
| 82 | 82 | return 0; |
| 83 | 83 | } |
| 84 | 84 | haveblocks = isofs_get_blocks(inode, blocknum, bhs, needblocks); |
| 85 | − ll_rw_block(REQ_OP_READ, haveblocks, bhs); | |
| 85 | + bh_read_batch(haveblocks, bhs); | |
| 86 | 86 | |
| 87 | 87 | curbh = 0; |
| 88 | 88 | curpage = 0; |
| @@ -1898,19 +1898,16 @@ static int journal_get_superblock(journal_t *journal) · 1897 unmodified lines | ||
| 1898 | 1898 | { |
| 1899 | 1899 | struct buffer_head *bh; |
| 1900 | 1900 | journal_superblock_t *sb; |
| 1901 | − int err = -EIO; | |
| 1901 | + int err; | |
| 1902 | 1902 | |
| 1903 | 1903 | bh = journal->j_sb_buffer; |
| 1904 | 1904 | |
| 1905 | 1905 | J_ASSERT(bh != NULL); |
| 1906 | − if (!buffer_uptodate(bh)) { | |
| 1907 | − ll_rw_block(REQ_OP_READ, 1, &bh); | |
| 1908 | − wait_on_buffer(bh); | |
| 1909 | − if (!buffer_uptodate(bh)) { | |
| 1910 | − printk(KERN_ERR | |
| 1911 | − "JBD2: IO error reading journal superblock\n"); | |
| 1912 | − goto out; | |
| 1913 | − } | |
| 1906 | + err = bh_read(bh, 0); | |
| 1907 | + if (err < 0) { | |
| 1908 | + printk(KERN_ERR | |
| 1909 | + "JBD2: IO error reading journal superblock\n"); | |
| 1910 | + goto out; | |
| 1914 | 1911 | } |
| 1915 | 1912 | |
| 1916 | 1913 | if (buffer_verified(bh)) |
| @@ -100,7 +100,7 @@ static int do_readahead(journal_t *journal, unsigned int start) · 99 unmodified lines | ||
| 100 | 100 | if (!buffer_uptodate(bh) && !buffer_locked(bh)) { |
| 101 | 101 | bufs[nbufs++] = bh; |
| 102 | 102 | if (nbufs == MAXBUF) { |
| 103 | − ll_rw_block(REQ_OP_READ, nbufs, bufs); | |
| 103 | + bh_readahead_batch(nbufs, bufs, 0); | |
| 104 | 104 | journal_brelse_array(bufs, nbufs); |
| 105 | 105 | nbufs = 0; |
| 106 | 106 | } |
| @@ -109,7 +109,7 @@ static int do_readahead(journal_t *journal, unsigned int start) · 2 unmodified lines | ||
| 109 | 109 | } |
| 110 | 110 | |
| 111 | 111 | if (nbufs) |
| 112 | − ll_rw_block(REQ_OP_READ, nbufs, bufs); | |
| 112 | + bh_readahead_batch(nbufs, bufs, 0); | |
| 113 | 113 | err = 0; |
| 114 | 114 | |
| 115 | 115 | failed: |
| @@ -152,9 +152,14 @@ static int jread(struct buffer_head **bhp, journal_t *journal, · 36 unmodified lines | ||
| 152 | 152 | return -ENOMEM; |
| 153 | 153 | |
| 154 | 154 | if (!buffer_uptodate(bh)) { |
| 155 | − /* If this is a brand new buffer, start readahead. | |
| 156 | − Otherwise, we assume we are already reading it. */ | |
| 157 | − if (!buffer_req(bh)) | |
| 155 | + /* | |
| 156 | + * If this is a brand new buffer, start readahead. | |
| 157 | + * Otherwise, we assume we are already reading it. | |
| 158 | + */ | |
| 159 | + bool need_readahead = !buffer_req(bh); | |
| 160 | + | |
| 161 | + bh_read_nowait(bh, 0); | |
| 162 | + if (need_readahead) | |
| 158 | 163 | do_readahead(journal, offset); |
| 159 | 164 | wait_on_buffer(bh); |
| 160 | 165 | } |
| @@ -688,7 +693,6 @@ static int do_one_pass(journal_t *journal, · 527 unmodified lines | ||
| 688 | 693 | mark_buffer_dirty(nbh); |
| 689 | 694 | BUFFER_TRACE(nbh, "marking uptodate"); |
| 690 | 695 | ++info->nr_replays; |
| 691 | − /* ll_rw_block(WRITE, 1, &nbh); */ | |
| 692 | 696 | unlock_buffer(nbh); |
| 693 | 697 | brelse(obh); |
| 694 | 698 | brelse(nbh); |
| @@ -5088,7 +5088,7 @@ int page_symlink(struct inode *inode, const char *symname, int len) · 5087 unmodified lines | ||
| 5088 | 5088 | const struct address_space_operations *aops = mapping->a_ops; |
| 5089 | 5089 | bool nofs = !mapping_gfp_constraint(mapping, __GFP_FS); |
| 5090 | 5090 | struct page *page; |
| 5091 | − void *fsdata; | |
| 5091 | + void *fsdata = NULL; | |
| 5092 | 5092 | int err; |
| 5093 | 5093 | unsigned int flags; |
| 5094 | 5094 | |
| @@ -480,41 +480,36 @@ unsigned long nilfs_find_uncommitted_extent(struct inode *inode, · 479 unmodified lines | ||
| 480 | 480 | sector_t start_blk, |
| 481 | 481 | sector_t *blkoff) |
| 482 | 482 | { |
| 483 | − unsigned int i; | |
| 483 | + unsigned int i, nr_folios; | |
| 484 | 484 | pgoff_t index; |
| 485 | − unsigned int nblocks_in_page; | |
| 486 | 485 | unsigned long length = 0; |
| 487 | − sector_t b; | |
| 488 | − struct pagevec pvec; | |
| 489 | − struct page *page; | |
| 486 | + struct folio_batch fbatch; | |
| 487 | + struct folio *folio; | |
| 490 | 488 | |
| 491 | 489 | if (inode->i_mapping->nrpages == 0) |
| 492 | 490 | return 0; |
| 493 | 491 | |
| 494 | 492 | index = start_blk >> (PAGE_SHIFT - inode->i_blkbits); |
| 495 | − nblocks_in_page = 1U << (PAGE_SHIFT - inode->i_blkbits); | |
| 496 | 493 | |
| 497 | − pagevec_init(&pvec); | |
| 494 | + folio_batch_init(&fbatch); | |
| 498 | 495 | |
| 499 | 496 | repeat: |
| 500 | − pvec.nr = find_get_pages_contig(inode->i_mapping, index, PAGEVEC_SIZE, | |
| 501 | − pvec.pages); | |
| 502 | − if (pvec.nr == 0) | |
| 497 | + nr_folios = filemap_get_folios_contig(inode->i_mapping, &index, ULONG_MAX, | |
| 498 | + &fbatch); | |
| 499 | + if (nr_folios == 0) | |
| 503 | 500 | return length; |
| 504 | 501 | |
| 505 | − if (length > 0 && pvec.pages[0]->index > index) | |
| 506 | − goto out; | |
| 507 | − | |
| 508 | − b = pvec.pages[0]->index << (PAGE_SHIFT - inode->i_blkbits); | |
| 509 | 502 | i = 0; |
| 510 | 503 | do { |
| 511 | − page = pvec.pages[i]; | |
| 504 | + folio = fbatch.folios[i]; | |
| 512 | 505 | |
| 513 | − lock_page(page); | |
| 514 | − if (page_has_buffers(page)) { | |
| 506 | + folio_lock(folio); | |
| 507 | + if (folio_buffers(folio)) { | |
| 515 | 508 | struct buffer_head *bh, *head; |
| 509 | + sector_t b; | |
| 516 | 510 | |
| 517 | − bh = head = page_buffers(page); | |
| 511 | + b = folio->index << (PAGE_SHIFT - inode->i_blkbits); | |
| 512 | + bh = head = folio_buffers(folio); | |
| 518 | 513 | do { |
| 519 | 514 | if (b < start_blk) |
| 520 | 515 | continue; |
| @@ -529,21 +524,17 @@ repeat: · 8 unmodified lines | ||
| 529 | 524 | } else { |
| 530 | 525 | if (length > 0) |
| 531 | 526 | goto out_locked; |
| 532 | − | |
| 533 | − b += nblocks_in_page; | |
| 534 | 527 | } |
| 535 | − unlock_page(page); | |
| 528 | + folio_unlock(folio); | |
| 536 | 529 | |
| 537 | − } while (++i < pagevec_count(&pvec)); | |
| 530 | + } while (++i < nr_folios); | |
| 538 | 531 | |
| 539 | − index = page->index + 1; | |
| 540 | − pagevec_release(&pvec); | |
| 532 | + folio_batch_release(&fbatch); | |
| 541 | 533 | cond_resched(); |
| 542 | 534 | goto repeat; |
| 543 | 535 | |
| 544 | 536 | out_locked: |
| 545 | − unlock_page(page); | |
| 546 | −out: | |
| 547 | − pagevec_release(&pvec); | |
| 537 | + folio_unlock(folio); | |
| 538 | + folio_batch_release(&fbatch); | |
| 548 | 539 | return length; |
| 549 | 540 | } |
| @@ -630,12 +630,9 @@ static noinline int ntfs_get_block_vbo(struct inode *inode, u64 vbo, · 629 unmodified lines | ||
| 630 | 630 | bh->b_size = block_size; |
| 631 | 631 | off = vbo & (PAGE_SIZE - 1); |
| 632 | 632 | set_bh_page(bh, page, off); |
| 633 | − ll_rw_block(REQ_OP_READ, 1, &bh); | |
| 634 | − wait_on_buffer(bh); | |
| 635 | − if (!buffer_uptodate(bh)) { | |
| 636 | − err = -EIO; | |
| 633 | + err = bh_read(bh, 0); | |
| 634 | + if (err < 0) | |
| 637 | 635 | goto out; |
| 638 | − } | |
| 639 | 636 | zero_user_segment(page, off + voff, off + block_size); |
| 640 | 637 | } |
| 641 | 638 | } |
| @@ -636,7 +636,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 635 unmodified lines | ||
| 636 | 636 | !buffer_new(bh) && |
| 637 | 637 | ocfs2_should_read_blk(inode, page, block_start) && |
| 638 | 638 | (block_start < from || block_end > to)) { |
| 639 | − ll_rw_block(REQ_OP_READ, 1, &bh); | |
| 639 | + bh_read_nowait(bh, 0); | |
| 640 | 640 | *wait_bh++=bh; |
| 641 | 641 | } |
| 642 | 642 | |
| @@ -1764,9 +1764,7 @@ static int ocfs2_get_sector(struct super_block *sb, · 1763 unmodified lines | ||
| 1764 | 1764 | if (!buffer_dirty(*bh)) |
| 1765 | 1765 | clear_buffer_uptodate(*bh); |
| 1766 | 1766 | unlock_buffer(*bh); |
| 1767 | − ll_rw_block(REQ_OP_READ, 1, bh); | |
| 1768 | − wait_on_buffer(*bh); | |
| 1769 | − if (!buffer_uptodate(*bh)) { | |
| 1767 | + if (bh_read(*bh, 0) < 0) { | |
| 1770 | 1768 | mlog_errno(-EIO); |
| 1771 | 1769 | brelse(*bh); |
| 1772 | 1770 | *bh = NULL; |
| @@ -24,6 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <linux/user_namespace.h> |
| 25 | 25 | #include <linux/namei.h> |
| 26 | 26 | #include <linux/mnt_idmapping.h> |
| 27 | +#include <linux/iversion.h> | |
| 27 | 28 | |
| 28 | 29 | static struct posix_acl **acl_by_type(struct inode *inode, int type) |
| 29 | 30 | { |
| @@ -1227,6 +1228,8 @@ int simple_set_acl(struct user_namespace *mnt_userns, struct inode *inode, · 1197 unmodified lines | ||
| 1227 | 1228 | } |
| 1228 | 1229 | |
| 1229 | 1230 | inode->i_ctime = current_time(inode); |
| 1231 | + if (IS_I_VERSION(inode)) | |
| 1232 | + inode_inc_iversion(inode); | |
| 1230 | 1233 | set_cached_acl(inode, type, acl); |
| 1231 | 1234 | return 0; |
| 1232 | 1235 | } |
| @@ -2350,6 +2350,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx) · 2349 unmodified lines | ||
| 2350 | 2350 | GENRADIX(struct map_files_info) fa; |
| 2351 | 2351 | struct map_files_info *p; |
| 2352 | 2352 | int ret; |
| 2353 | + struct vma_iterator vmi; | |
| 2353 | 2354 | |
| 2354 | 2355 | genradix_init(&fa); |
| 2355 | 2356 | |
| @@ -2388,7 +2389,9 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx) · 32 unmodified lines | ||
| 2388 | 2389 | * routine might require mmap_lock taken in might_fault(). |
| 2389 | 2390 | */ |
| 2390 | 2391 | |
| 2391 | − for (vma = mm->mmap, pos = 2; vma; vma = vma->vm_next) { | |
| 2392 | + pos = 2; | |
| 2393 | + vma_iter_init(&vmi, mm, 0); | |
| 2394 | + for_each_vma(vmi, vma) { | |
| 2392 | 2395 | if (!vma->vm_file) |
| 2393 | 2396 | continue; |
| 2394 | 2397 | if (++pos <= ctx->pos) |
| @@ -3196,6 +3199,19 @@ static int proc_pid_ksm_merging_pages(struct seq_file *m, struct pid_namespace * · 801 unmodified lines | ||
| 3196 | 3199 | |
| 3197 | 3200 | return 0; |
| 3198 | 3201 | } |
| 3202 | +static int proc_pid_ksm_stat(struct seq_file *m, struct pid_namespace *ns, | |
| 3203 | + struct pid *pid, struct task_struct *task) | |
| 3204 | +{ | |
| 3205 | + struct mm_struct *mm; | |
| 3206 | + | |
| 3207 | + mm = get_task_mm(task); | |
| 3208 | + if (mm) { | |
| 3209 | + seq_printf(m, "ksm_rmap_items %lu\n", mm->ksm_rmap_items); | |
| 3210 | + mmput(mm); | |
| 3211 | + } | |
| 3212 | + | |
| 3213 | + return 0; | |
| 3214 | +} | |
| 3199 | 3215 | #endif /* CONFIG_KSM */ |
| 3200 | 3216 | |
| 3201 | 3217 | #ifdef CONFIG_STACKLEAK_METRICS |
| @@ -3331,6 +3347,7 @@ static const struct pid_entry tgid_base_stuff[] = { · 129 unmodified lines | ||
| 3331 | 3347 | #endif |
| 3332 | 3348 | #ifdef CONFIG_KSM |
| 3333 | 3349 | ONE("ksm_merging_pages", S_IRUSR, proc_pid_ksm_merging_pages), |
| 3350 | + ONE("ksm_stat", S_IRUSR, proc_pid_ksm_stat), | |
| 3334 | 3351 | #endif |
| 3335 | 3352 | }; |
| 3336 | 3353 | |
| @@ -3668,6 +3685,7 @@ static const struct pid_entry tid_base_stuff[] = { · 331 unmodified lines | ||
| 3668 | 3685 | #endif |
| 3669 | 3686 | #ifdef CONFIG_KSM |
| 3670 | 3687 | ONE("ksm_merging_pages", S_IRUSR, proc_pid_ksm_merging_pages), |
| 3688 | + ONE("ksm_stat", S_IRUSR, proc_pid_ksm_stat), | |
| 3671 | 3689 | #endif |
| 3672 | 3690 | }; |
| 3673 | 3691 | |
| @@ -285,7 +285,7 @@ struct proc_maps_private { · 284 unmodified lines | ||
| 285 | 285 | struct task_struct *task; |
| 286 | 286 | struct mm_struct *mm; |
| 287 | 287 | #ifdef CONFIG_MMU |
| 288 | − struct vm_area_struct *tail_vma; | |
| 288 | + struct vma_iterator iter; | |
| 289 | 289 | #endif |
| 290 | 290 | #ifdef CONFIG_NUMA |
| 291 | 291 | struct mempolicy *task_mempolicy; |
| @@ -1,6 +1,5 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | #include <linux/pagewalk.h> |
| 3 | −#include <linux/vmacache.h> | |
| 4 | 3 | #include <linux/mm_inline.h> |
| 5 | 4 | #include <linux/hugetlb.h> |
| 6 | 5 | #include <linux/huge_mm.h> |
| @@ -124,12 +123,26 @@ static void release_task_mempolicy(struct proc_maps_private *priv) · 117 unmodified lines | ||
| 124 | 123 | } |
| 125 | 124 | #endif |
| 126 | 125 | |
| 126 | +static struct vm_area_struct *proc_get_vma(struct proc_maps_private *priv, | |
| 127 | + loff_t *ppos) | |
| 128 | +{ | |
| 129 | + struct vm_area_struct *vma = vma_next(&priv->iter); | |
| 130 | + | |
| 131 | + if (vma) { | |
| 132 | + *ppos = vma->vm_start; | |
| 133 | + } else { | |
| 134 | + *ppos = -2UL; | |
| 135 | + vma = get_gate_vma(priv->mm); | |
| 136 | + } | |
| 137 | + | |
| 138 | + return vma; | |
| 139 | +} | |
| 140 | + | |
| 127 | 141 | static void *m_start(struct seq_file *m, loff_t *ppos) |
| 128 | 142 | { |
| 129 | 143 | struct proc_maps_private *priv = m->private; |
| 130 | 144 | unsigned long last_addr = *ppos; |
| 131 | 145 | struct mm_struct *mm; |
| 132 | − struct vm_area_struct *vma; | |
| 133 | 146 | |
| 134 | 147 | /* See m_next(). Zero at the start or after lseek. */ |
| 135 | 148 | if (last_addr == -1UL) |
| @@ -153,31 +166,21 @@ static void *m_start(struct seq_file *m, loff_t *ppos) · 17 unmodified lines | ||
| 153 | 166 | return ERR_PTR(-EINTR); |
| 154 | 167 | } |
| 155 | 168 | |
| 169 | + vma_iter_init(&priv->iter, mm, last_addr); | |
| 156 | 170 | hold_task_mempolicy(priv); |
| 157 | − priv->tail_vma = get_gate_vma(mm); | |
| 158 | − | |
| 159 | − vma = find_vma(mm, last_addr); | |
| 160 | − if (vma) | |
| 161 | − return vma; | |
| 171 | + if (last_addr == -2UL) | |
| 172 | + return get_gate_vma(mm); | |
| 162 | 173 | |
| 163 | − return priv->tail_vma; | |
| 174 | + return proc_get_vma(priv, ppos); | |
| 164 | 175 | } |
| 165 | 176 | |
| 166 | 177 | static void *m_next(struct seq_file *m, void *v, loff_t *ppos) |
| 167 | 178 | { |
| 168 | − struct proc_maps_private *priv = m->private; | |
| 169 | − struct vm_area_struct *next, *vma = v; | |
| 170 | − | |
| 171 | − if (vma == priv->tail_vma) | |
| 172 | − next = NULL; | |
| 173 | − else if (vma->vm_next) | |
| 174 | − next = vma->vm_next; | |
| 175 | − else | |
| 176 | − next = priv->tail_vma; | |
| 177 | − | |
| 178 | − *ppos = next ? next->vm_start : -1UL; | |
| 179 | − | |
| 180 | − return next; | |
| 179 | + if (*ppos == -2UL) { | |
| 180 | + *ppos = -1UL; | |
| 181 | + return NULL; | |
| 182 | + } | |
| 183 | + return proc_get_vma(m->private, ppos); | |
| 181 | 184 | } |
| 182 | 185 | |
| 183 | 186 | static void m_stop(struct seq_file *m, void *v) |
| @@ -864,7 +867,7 @@ static int show_smap(struct seq_file *m, void *v) · 680 unmodified lines | ||
| 864 | 867 | __show_smap(m, &mss, false); |
| 865 | 868 | |
| 866 | 869 | seq_printf(m, "THPeligible: %d\n", |
| 867 | − hugepage_vma_check(vma, vma->vm_flags, true, false)); | |
| 870 | + hugepage_vma_check(vma, vma->vm_flags, true, false, true)); | |
| 868 | 871 | |
| 869 | 872 | if (arch_pkeys_enabled()) |
| 870 | 873 | seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma)); |
| @@ -877,16 +880,16 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 6 unmodified lines | ||
| 877 | 880 | { |
| 878 | 881 | struct proc_maps_private *priv = m->private; |
| 879 | 882 | struct mem_size_stats mss; |
| 880 | − struct mm_struct *mm; | |
| 883 | + struct mm_struct *mm = priv->mm; | |
| 881 | 884 | struct vm_area_struct *vma; |
| 882 | − unsigned long last_vma_end = 0; | |
| 885 | + unsigned long vma_start = 0, last_vma_end = 0; | |
| 883 | 886 | int ret = 0; |
| 887 | + MA_STATE(mas, &mm->mm_mt, 0, 0); | |
| 884 | 888 | |
| 885 | 889 | priv->task = get_proc_task(priv->inode); |
| 886 | 890 | if (!priv->task) |
| 887 | 891 | return -ESRCH; |
| 888 | 892 | |
| 889 | − mm = priv->mm; | |
| 890 | 893 | if (!mm || !mmget_not_zero(mm)) { |
| 891 | 894 | ret = -ESRCH; |
| 892 | 895 | goto out_put_task; |
| @@ -899,8 +902,13 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 6 unmodified lines | ||
| 899 | 902 | goto out_put_mm; |
| 900 | 903 | |
| 901 | 904 | hold_task_mempolicy(priv); |
| 905 | + vma = mas_find(&mas, 0); | |
| 906 | + | |
| 907 | + if (unlikely(!vma)) | |
| 908 | + goto empty_set; | |
| 902 | 909 | |
| 903 | − for (vma = priv->mm->mmap; vma;) { | |
| 910 | + vma_start = vma->vm_start; | |
| 911 | + do { | |
| 904 | 912 | smap_gather_stats(vma, &mss, 0); |
| 905 | 913 | last_vma_end = vma->vm_end; |
| 906 | 914 | |
| @@ -909,6 +917,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 2 unmodified lines | ||
| 909 | 917 | * access it for write request. |
| 910 | 918 | */ |
| 911 | 919 | if (mmap_lock_is_contended(mm)) { |
| 920 | + mas_pause(&mas); | |
| 912 | 921 | mmap_read_unlock(mm); |
| 913 | 922 | ret = mmap_read_lock_killable(mm); |
| 914 | 923 | if (ret) { |
| @@ -952,7 +961,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 37 unmodified lines | ||
| 952 | 961 | * contains last_vma_end. |
| 953 | 962 | * Iterate VMA' from last_vma_end. |
| 954 | 963 | */ |
| 955 | − vma = find_vma(mm, last_vma_end - 1); | |
| 964 | + vma = mas_find(&mas, ULONG_MAX); | |
| 956 | 965 | /* Case 3 above */ |
| 957 | 966 | if (!vma) |
| 958 | 967 | break; |
| @@ -966,11 +975,10 @@ static int show_smaps_rollup(struct seq_file *m, void *v) · 7 unmodified lines | ||
| 966 | 975 | smap_gather_stats(vma, &mss, last_vma_end); |
| 967 | 976 | } |
| 968 | 977 | /* Case 2 above */ |
| 969 | − vma = vma->vm_next; | |
| 970 | − } | |
| 978 | + } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL); | |
| 971 | 979 | |
| 972 | − show_vma_header_prefix(m, priv->mm->mmap->vm_start, | |
| 973 | − last_vma_end, 0, 0, 0, 0); | |
| 980 | +empty_set: | |
| 981 | + show_vma_header_prefix(m, vma_start, last_vma_end, 0, 0, 0, 0); | |
| 974 | 982 | seq_pad(m, ' '); |
| 975 | 983 | seq_puts(m, "[rollup]\n"); |
| 976 | 984 | |
| @@ -1263,6 +1271,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, · 286 unmodified lines | ||
| 1263 | 1271 | return -ESRCH; |
| 1264 | 1272 | mm = get_task_mm(task); |
| 1265 | 1273 | if (mm) { |
| 1274 | + MA_STATE(mas, &mm->mm_mt, 0, 0); | |
| 1266 | 1275 | struct mmu_notifier_range range; |
| 1267 | 1276 | struct clear_refs_private cp = { |
| 1268 | 1277 | .type = type, |
| @@ -1282,7 +1291,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, · 13 unmodified lines | ||
| 1282 | 1291 | } |
| 1283 | 1292 | |
| 1284 | 1293 | if (type == CLEAR_REFS_SOFT_DIRTY) { |
| 1285 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 1294 | + mas_for_each(&mas, vma, ULONG_MAX) { | |
| 1286 | 1295 | if (!(vma->vm_flags & VM_SOFTDIRTY)) |
| 1287 | 1296 | continue; |
| 1288 | 1297 | vma->vm_flags &= ~VM_SOFTDIRTY; |
| @@ -1294,8 +1303,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, · 5 unmodified lines | ||
| 1294 | 1303 | 0, NULL, mm, 0, -1UL); |
| 1295 | 1304 | mmu_notifier_invalidate_range_start(&range); |
| 1296 | 1305 | } |
| 1297 | − walk_page_range(mm, 0, mm->highest_vm_end, &clear_refs_walk_ops, | |
| 1298 | − &cp); | |
| 1306 | + walk_page_range(mm, 0, -1, &clear_refs_walk_ops, &cp); | |
| 1299 | 1307 | if (type == CLEAR_REFS_SOFT_DIRTY) { |
| 1300 | 1308 | mmu_notifier_invalidate_range_end(&range); |
| 1301 | 1309 | flush_tlb_mm(mm); |
| @@ -1418,9 +1426,19 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm, · 116 unmodified lines | ||
| 1418 | 1426 | if (pte_swp_uffd_wp(pte)) |
| 1419 | 1427 | flags |= PM_UFFD_WP; |
| 1420 | 1428 | entry = pte_to_swp_entry(pte); |
| 1421 | − if (pm->show_pfn) | |
| 1429 | + if (pm->show_pfn) { | |
| 1430 | + pgoff_t offset; | |
| 1431 | + /* | |
| 1432 | + * For PFN swap offsets, keeping the offset field | |
| 1433 | + * to be PFN only to be compatible with old smaps. | |
| 1434 | + */ | |
| 1435 | + if (is_pfn_swap_entry(entry)) | |
| 1436 | + offset = swp_offset_pfn(entry); | |
| 1437 | + else | |
| 1438 | + offset = swp_offset(entry); | |
| 1422 | 1439 | frame = swp_type(entry) | |
| 1423 | − (swp_offset(entry) << MAX_SWAPFILES_SHIFT); | |
| 1440 | + (offset << MAX_SWAPFILES_SHIFT); | |
| 1441 | + } | |
| 1424 | 1442 | flags |= PM_SWAP; |
| 1425 | 1443 | migration = is_migration_entry(entry); |
| 1426 | 1444 | if (is_pfn_swap_entry(entry)) |
| @@ -1477,7 +1495,11 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, · 50 unmodified lines | ||
| 1477 | 1495 | unsigned long offset; |
| 1478 | 1496 | |
| 1479 | 1497 | if (pm->show_pfn) { |
| 1480 | − offset = swp_offset(entry) + | |
| 1498 | + if (is_pfn_swap_entry(entry)) | |
| 1499 | + offset = swp_offset_pfn(entry); | |
| 1500 | + else | |
| 1501 | + offset = swp_offset(entry); | |
| 1502 | + offset = offset + | |
| 1481 | 1503 | ((addr & ~PMD_MASK) >> PAGE_SHIFT); |
| 1482 | 1504 | frame = swp_type(entry) | |
| 1483 | 1505 | (offset << MAX_SWAPFILES_SHIFT); |
| @@ -20,15 +20,13 @@ · 19 unmodified lines | ||
| 20 | 20 | */ |
| 21 | 21 | void task_mem(struct seq_file *m, struct mm_struct *mm) |
| 22 | 22 | { |
| 23 | + VMA_ITERATOR(vmi, mm, 0); | |
| 23 | 24 | struct vm_area_struct *vma; |
| 24 | 25 | struct vm_region *region; |
| 25 | − struct rb_node *p; | |
| 26 | 26 | unsigned long bytes = 0, sbytes = 0, slack = 0, size; |
| 27 | − | |
| 28 | − mmap_read_lock(mm); | |
| 29 | − for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) { | |
| 30 | − vma = rb_entry(p, struct vm_area_struct, vm_rb); | |
| 31 | 27 | |
| 28 | + mmap_read_lock(mm); | |
| 29 | + for_each_vma(vmi, vma) { | |
| 32 | 30 | bytes += kobjsize(vma); |
| 33 | 31 | |
| 34 | 32 | region = vma->vm_region; |
| @@ -82,15 +80,13 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) · 47 unmodified lines | ||
| 82 | 80 | |
| 83 | 81 | unsigned long task_vsize(struct mm_struct *mm) |
| 84 | 82 | { |
| 83 | + VMA_ITERATOR(vmi, mm, 0); | |
| 85 | 84 | struct vm_area_struct *vma; |
| 86 | − struct rb_node *p; | |
| 87 | 85 | unsigned long vsize = 0; |
| 88 | 86 | |
| 89 | 87 | mmap_read_lock(mm); |
| 90 | − for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) { | |
| 91 | − vma = rb_entry(p, struct vm_area_struct, vm_rb); | |
| 88 | + for_each_vma(vmi, vma) | |
| 92 | 89 | vsize += vma->vm_end - vma->vm_start; |
| 93 | − } | |
| 94 | 90 | mmap_read_unlock(mm); |
| 95 | 91 | return vsize; |
| 96 | 92 | } |
| @@ -99,14 +95,13 @@ unsigned long task_statm(struct mm_struct *mm, · 2 unmodified lines | ||
| 99 | 95 | unsigned long *shared, unsigned long *text, |
| 100 | 96 | unsigned long *data, unsigned long *resident) |
| 101 | 97 | { |
| 98 | + VMA_ITERATOR(vmi, mm, 0); | |
| 102 | 99 | struct vm_area_struct *vma; |
| 103 | 100 | struct vm_region *region; |
| 104 | − struct rb_node *p; | |
| 105 | 101 | unsigned long size = kobjsize(mm); |
| 106 | 102 | |
| 107 | 103 | mmap_read_lock(mm); |
| 108 | − for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) { | |
| 109 | − vma = rb_entry(p, struct vm_area_struct, vm_rb); | |
| 104 | + for_each_vma(vmi, vma) { | |
| 110 | 105 | size += kobjsize(vma); |
| 111 | 106 | region = vma->vm_region; |
| 112 | 107 | if (region) { |
| @@ -190,17 +185,19 @@ static int nommu_vma_show(struct seq_file *m, struct vm_area_struct *vma) · 77 unmodified lines | ||
| 190 | 185 | */ |
| 191 | 186 | static int show_map(struct seq_file *m, void *_p) |
| 192 | 187 | { |
| 193 | − struct rb_node *p = _p; | |
| 194 | − | |
| 195 | − return nommu_vma_show(m, rb_entry(p, struct vm_area_struct, vm_rb)); | |
| 188 | + return nommu_vma_show(m, _p); | |
| 196 | 189 | } |
| 197 | 190 | |
| 198 | 191 | static void *m_start(struct seq_file *m, loff_t *pos) |
| 199 | 192 | { |
| 200 | 193 | struct proc_maps_private *priv = m->private; |
| 201 | 194 | struct mm_struct *mm; |
| 202 | − struct rb_node *p; | |
| 203 | − loff_t n = *pos; | |
| 195 | + struct vm_area_struct *vma; | |
| 196 | + unsigned long addr = *pos; | |
| 197 | + | |
| 198 | + /* See m_next(). Zero at the start or after lseek. */ | |
| 199 | + if (addr == -1UL) | |
| 200 | + return NULL; | |
| 204 | 201 | |
| 205 | 202 | /* pin the task and mm whilst we play with them */ |
| 206 | 203 | priv->task = get_proc_task(priv->inode); |
| @@ -216,10 +213,10 @@ static void *m_start(struct seq_file *m, loff_t *pos) · 9 unmodified lines | ||
| 216 | 213 | return ERR_PTR(-EINTR); |
| 217 | 214 | } |
| 218 | 215 | |
| 219 | − /* start from the Nth VMA */ | |
| 220 | − for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) | |
| 221 | − if (n-- == 0) | |
| 222 | − return p; | |
| 216 | + /* start the next element from addr */ | |
| 217 | + vma = find_vma(mm, addr); | |
| 218 | + if (vma) | |
| 219 | + return vma; | |
| 223 | 220 | |
| 224 | 221 | mmap_read_unlock(mm); |
| 225 | 222 | mmput(mm); |
| @@ -242,10 +239,10 @@ static void m_stop(struct seq_file *m, void *_vml) · 16 unmodified lines | ||
| 242 | 239 | |
| 243 | 240 | static void *m_next(struct seq_file *m, void *_p, loff_t *pos) |
| 244 | 241 | { |
| 245 | − struct rb_node *p = _p; | |
| 242 | + struct vm_area_struct *vma = _p; | |
| 246 | 243 | |
| 247 | − (*pos)++; | |
| 248 | − return p ? rb_next(p) : NULL; | |
| 244 | + *pos = vma->vm_end; | |
| 245 | + return find_vma(vma->vm_mm, vma->vm_end); | |
| 249 | 246 | } |
| 250 | 247 | |
| 251 | 248 | static const struct seq_operations proc_pid_maps_ops = { |
| @@ -203,9 +203,9 @@ static unsigned long ramfs_nommu_get_unmapped_area(struct file *file, · 202 unmodified lines | ||
| 203 | 203 | unsigned long addr, unsigned long len, |
| 204 | 204 | unsigned long pgoff, unsigned long flags) |
| 205 | 205 | { |
| 206 | − unsigned long maxpages, lpages, nr, loop, ret; | |
| 206 | + unsigned long maxpages, lpages, nr_folios, loop, ret, nr_pages, pfn; | |
| 207 | 207 | struct inode *inode = file_inode(file); |
| 208 | − struct page **pages = NULL, **ptr, *page; | |
| 208 | + struct folio_batch fbatch; | |
| 209 | 209 | loff_t isize; |
| 210 | 210 | |
| 211 | 211 | /* the mapping mustn't extend beyond the EOF */ |
| @@ -221,31 +221,39 @@ static unsigned long ramfs_nommu_get_unmapped_area(struct file *file, · 9 unmodified lines | ||
| 221 | 221 | goto out; |
| 222 | 222 | |
| 223 | 223 | /* gang-find the pages */ |
| 224 | − pages = kcalloc(lpages, sizeof(struct page *), GFP_KERNEL); | |
| 225 | − if (!pages) | |
| 226 | − goto out_free; | |
| 227 | − | |
| 228 | − nr = find_get_pages_contig(inode->i_mapping, pgoff, lpages, pages); | |
| 229 | − if (nr != lpages) | |
| 230 | − goto out_free_pages; /* leave if some pages were missing */ | |
| 224 | + folio_batch_init(&fbatch); | |
| 225 | + nr_pages = 0; | |
| 226 | +repeat: | |
| 227 | + nr_folios = filemap_get_folios_contig(inode->i_mapping, &pgoff, | |
| 228 | + ULONG_MAX, &fbatch); | |
| 229 | + if (!nr_folios) { | |
| 230 | + ret = -ENOSYS; | |
| 231 | + return ret; | |
| 232 | + } | |
| 231 | 233 | |
| 234 | + if (ret == -ENOSYS) { | |
| 235 | + ret = (unsigned long) folio_address(fbatch.folios[0]); | |
| 236 | + pfn = folio_pfn(fbatch.folios[0]); | |
| 237 | + } | |
| 232 | 238 | /* check the pages for physical adjacency */ |
| 233 | − ptr = pages; | |
| 234 | − page = *ptr++; | |
| 235 | − page++; | |
| 236 | − for (loop = lpages; loop > 1; loop--) | |
| 237 | − if (*ptr++ != page++) | |
| 238 | − goto out_free_pages; | |
| 239 | + for (loop = 0; loop < nr_folios; loop++) { | |
| 240 | + if (pfn + nr_pages != folio_pfn(fbatch.folios[loop])) { | |
| 241 | + ret = -ENOSYS; | |
| 242 | + goto out_free; /* leave if not physical adjacent */ | |
| 243 | + } | |
| 244 | + nr_pages += folio_nr_pages(fbatch.folios[loop]); | |
| 245 | + if (nr_pages >= lpages) | |
| 246 | + goto out_free; /* successfully found desired pages*/ | |
| 247 | + } | |
| 239 | 248 | |
| 249 | + if (nr_pages < lpages) { | |
| 250 | + folio_batch_release(&fbatch); | |
| 251 | + goto repeat; /* loop if pages are missing */ | |
| 252 | + } | |
| 240 | 253 | /* okay - all conditions fulfilled */ |
| 241 | − ret = (unsigned long) page_address(pages[0]); | |
| 242 | 254 | |
| 243 | −out_free_pages: | |
| 244 | − ptr = pages; | |
| 245 | − for (loop = nr; loop > 0; loop--) | |
| 246 | − put_page(*ptr++); | |
| 247 | 255 | out_free: |
| 248 | − kfree(pages); | |
| 256 | + folio_batch_release(&fbatch); | |
| 249 | 257 | out: |
| 250 | 258 | return ret; |
| 251 | 259 | } |
| @@ -868,7 +868,7 @@ loop_next: · 867 unmodified lines | ||
| 868 | 868 | */ |
| 869 | 869 | if (buffer_dirty(bh) && unlikely(bh->b_page->mapping == NULL)) { |
| 870 | 870 | spin_unlock(lock); |
| 871 | − ll_rw_block(REQ_OP_WRITE, 1, &bh); | |
| 871 | + write_dirty_buffer(bh, 0); | |
| 872 | 872 | spin_lock(lock); |
| 873 | 873 | } |
| 874 | 874 | put_bh(bh); |
| @@ -1054,7 +1054,7 @@ static int flush_commit_list(struct super_block *s, · 179 unmodified lines | ||
| 1054 | 1054 | if (tbh) { |
| 1055 | 1055 | if (buffer_dirty(tbh)) { |
| 1056 | 1056 | depth = reiserfs_write_unlock_nested(s); |
| 1057 | − ll_rw_block(REQ_OP_WRITE, 1, &tbh); | |
| 1057 | + write_dirty_buffer(tbh, 0); | |
| 1058 | 1058 | reiserfs_write_lock_nested(s, depth); |
| 1059 | 1059 | } |
| 1060 | 1060 | put_bh(tbh) ; |
| @@ -2240,7 +2240,7 @@ abort_replay: · 1179 unmodified lines | ||
| 2240 | 2240 | } |
| 2241 | 2241 | } |
| 2242 | 2242 | /* read in the log blocks, memcpy to the corresponding real block */ |
| 2243 | − ll_rw_block(REQ_OP_READ, get_desc_trans_len(desc), log_blocks); | |
| 2243 | + bh_read_batch(get_desc_trans_len(desc), log_blocks); | |
| 2244 | 2244 | for (i = 0; i < get_desc_trans_len(desc); i++) { |
| 2245 | 2245 | |
| 2246 | 2246 | wait_on_buffer(log_blocks[i]); |
| @@ -2342,10 +2342,11 @@ static struct buffer_head *reiserfs_breada(struct block_device *dev, · 95 unmodified lines | ||
| 2342 | 2342 | } else |
| 2343 | 2343 | bhlist[j++] = bh; |
| 2344 | 2344 | } |
| 2345 | − ll_rw_block(REQ_OP_READ, j, bhlist); | |
| 2345 | + bh = bhlist[0]; | |
| 2346 | + bh_read_nowait(bh, 0); | |
| 2347 | + bh_readahead_batch(j - 1, &bhlist[1], 0); | |
| 2346 | 2348 | for (i = 1; i < j; i++) |
| 2347 | 2349 | brelse(bhlist[i]); |
| 2348 | − bh = bhlist[0]; | |
| 2349 | 2350 | wait_on_buffer(bh); |
| 2350 | 2351 | if (buffer_uptodate(bh)) |
| 2351 | 2352 | return bh; |
| @@ -579,7 +579,7 @@ static int search_by_key_reada(struct super_block *s, · 578 unmodified lines | ||
| 579 | 579 | if (!buffer_uptodate(bh[j])) { |
| 580 | 580 | if (depth == -1) |
| 581 | 581 | depth = reiserfs_write_unlock_nested(s); |
| 582 | − ll_rw_block(REQ_OP_READ | REQ_RAHEAD, 1, bh + j); | |
| 582 | + bh_readahead(bh[j], REQ_RAHEAD); | |
| 583 | 583 | } |
| 584 | 584 | brelse(bh[j]); |
| 585 | 585 | } |
| @@ -685,7 +685,7 @@ int search_by_key(struct super_block *sb, const struct cpu_key *key, · 99 unmodified lines | ||
| 685 | 685 | if (!buffer_uptodate(bh) && depth == -1) |
| 686 | 686 | depth = reiserfs_write_unlock_nested(sb); |
| 687 | 687 | |
| 688 | − ll_rw_block(REQ_OP_READ, 1, &bh); | |
| 688 | + bh_read_nowait(bh, 0); | |
| 689 | 689 | wait_on_buffer(bh); |
| 690 | 690 | |
| 691 | 691 | if (depth != -1) |
| @@ -1702,9 +1702,7 @@ static int read_super_block(struct super_block *s, int offset) · 1701 unmodified lines | ||
| 1702 | 1702 | /* after journal replay, reread all bitmap and super blocks */ |
| 1703 | 1703 | static int reread_meta_blocks(struct super_block *s) |
| 1704 | 1704 | { |
| 1705 | − ll_rw_block(REQ_OP_READ, 1, &SB_BUFFER_WITH_SB(s)); | |
| 1706 | − wait_on_buffer(SB_BUFFER_WITH_SB(s)); | |
| 1707 | − if (!buffer_uptodate(SB_BUFFER_WITH_SB(s))) { | |
| 1705 | + if (bh_read(SB_BUFFER_WITH_SB(s), 0) < 0) { | |
| 1708 | 1706 | reiserfs_warning(s, "reiserfs-2504", "error reading the super"); |
| 1709 | 1707 | return 1; |
| 1710 | 1708 | } |
| @@ -130,7 +130,7 @@ static int udf_readdir(struct file *file, struct dir_context *ctx) · 129 unmodified lines | ||
| 130 | 130 | brelse(tmp); |
| 131 | 131 | } |
| 132 | 132 | if (num) { |
| 133 | − ll_rw_block(REQ_OP_READ | REQ_RAHEAD, num, bha); | |
| 133 | + bh_readahead_batch(num, bha, REQ_RAHEAD); | |
| 134 | 134 | for (i = 0; i < num; i++) |
| 135 | 135 | brelse(bha[i]); |
| 136 | 136 | } |
| @@ -89,7 +89,7 @@ struct fileIdentDesc *udf_fileident_read(struct inode *dir, loff_t *nf_pos, · 88 unmodified lines | ||
| 89 | 89 | brelse(tmp); |
| 90 | 90 | } |
| 91 | 91 | if (num) { |
| 92 | − ll_rw_block(REQ_OP_READ | REQ_RAHEAD, num, bha); | |
| 92 | + bh_readahead_batch(num, bha, REQ_RAHEAD); | |
| 93 | 93 | for (i = 0; i < num; i++) |
| 94 | 94 | brelse(bha[i]); |
| 95 | 95 | } |
| @@ -1211,13 +1211,7 @@ struct buffer_head *udf_bread(struct inode *inode, udf_pblk_t block, · 1210 unmodified lines | ||
| 1211 | 1211 | if (!bh) |
| 1212 | 1212 | return NULL; |
| 1213 | 1213 | |
| 1214 | − if (buffer_uptodate(bh)) | |
| 1215 | − return bh; | |
| 1216 | − | |
| 1217 | − ll_rw_block(REQ_OP_READ, 1, &bh); | |
| 1218 | − | |
| 1219 | − wait_on_buffer(bh); | |
| 1220 | − if (buffer_uptodate(bh)) | |
| 1214 | + if (bh_read(bh, 0) >= 0) | |
| 1221 | 1215 | return bh; |
| 1222 | 1216 | |
| 1223 | 1217 | brelse(bh); |
| @@ -295,14 +295,10 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 294 unmodified lines | ||
| 295 | 295 | |
| 296 | 296 | if (!buffer_mapped(bh)) |
| 297 | 297 | map_bh(bh, inode->i_sb, oldb + pos); |
| 298 | − if (!buffer_uptodate(bh)) { | |
| 299 | − ll_rw_block(REQ_OP_READ, 1, &bh); | |
| 300 | − wait_on_buffer(bh); | |
| 301 | − if (!buffer_uptodate(bh)) { | |
| 302 | − ufs_error(inode->i_sb, __func__, | |
| 303 | − "read of block failed\n"); | |
| 304 | − break; | |
| 305 | − } | |
| 298 | + if (bh_read(bh, 0) < 0) { | |
| 299 | + ufs_error(inode->i_sb, __func__, | |
| 300 | + "read of block failed\n"); | |
| 301 | + break; | |
| 306 | 302 | } |
| 307 | 303 | |
| 308 | 304 | UFSD(" change from %llu to %llu, pos %u\n", |
| @@ -30,6 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <linux/security.h> |
| 31 | 31 | #include <linux/hugetlb.h> |
| 32 | 32 | #include <linux/swapops.h> |
| 33 | +#include <linux/miscdevice.h> | |
| 33 | 34 | |
| 34 | 35 | int sysctl_unprivileged_userfaultfd __read_mostly; |
| 35 | 36 | |
| @@ -415,13 +416,8 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) · 379 unmodified lines | ||
| 415 | 416 | |
| 416 | 417 | if (ctx->features & UFFD_FEATURE_SIGBUS) |
| 417 | 418 | goto out; |
| 418 | − if ((vmf->flags & FAULT_FLAG_USER) == 0 && | |
| 419 | − ctx->flags & UFFD_USER_MODE_ONLY) { | |
| 420 | − printk_once(KERN_WARNING "uffd: Set unprivileged_userfaultfd " | |
| 421 | − "sysctl knob to 1 if kernel faults must be handled " | |
| 422 | − "without obtaining CAP_SYS_PTRACE capability\n"); | |
| 419 | + if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY)) | |
| 423 | 420 | goto out; |
| 424 | − } | |
| 425 | 421 | |
| 426 | 422 | /* |
| 427 | 423 | * If it's already released don't get it. This avoids to loop |
| @@ -615,14 +611,16 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, · 187 unmodified lines | ||
| 615 | 611 | if (release_new_ctx) { |
| 616 | 612 | struct vm_area_struct *vma; |
| 617 | 613 | struct mm_struct *mm = release_new_ctx->mm; |
| 614 | + VMA_ITERATOR(vmi, mm, 0); | |
| 618 | 615 | |
| 619 | 616 | /* the various vma->vm_userfaultfd_ctx still points to it */ |
| 620 | 617 | mmap_write_lock(mm); |
| 621 | − for (vma = mm->mmap; vma; vma = vma->vm_next) | |
| 618 | + for_each_vma(vmi, vma) { | |
| 622 | 619 | if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { |
| 623 | 620 | vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; |
| 624 | 621 | vma->vm_flags &= ~__VM_UFFD_FLAGS; |
| 625 | 622 | } |
| 623 | + } | |
| 626 | 624 | mmap_write_unlock(mm); |
| 627 | 625 | |
| 628 | 626 | userfaultfd_ctx_put(release_new_ctx); |
| @@ -803,11 +801,13 @@ static bool has_unmap_ctx(struct userfaultfd_ctx *ctx, struct list_head *unmaps, · 174 unmodified lines | ||
| 803 | 801 | return false; |
| 804 | 802 | } |
| 805 | 803 | |
| 806 | −int userfaultfd_unmap_prep(struct vm_area_struct *vma, | |
| 807 | − unsigned long start, unsigned long end, | |
| 808 | − struct list_head *unmaps) | |
| 804 | +int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start, | |
| 805 | + unsigned long end, struct list_head *unmaps) | |
| 809 | 806 | { |
| 810 | − for ( ; vma && vma->vm_start < end; vma = vma->vm_next) { | |
| 807 | + VMA_ITERATOR(vmi, mm, start); | |
| 808 | + struct vm_area_struct *vma; | |
| 809 | + | |
| 810 | + for_each_vma_range(vmi, vma, end) { | |
| 811 | 811 | struct userfaultfd_unmap_ctx *unmap_ctx; |
| 812 | 812 | struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx; |
| 813 | 813 | |
| @@ -857,6 +857,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) · 43 unmodified lines | ||
| 857 | 857 | /* len == 0 means wake all */ |
| 858 | 858 | struct userfaultfd_wake_range range = { .len = 0, }; |
| 859 | 859 | unsigned long new_flags; |
| 860 | + MA_STATE(mas, &mm->mm_mt, 0, 0); | |
| 860 | 861 | |
| 861 | 862 | WRITE_ONCE(ctx->released, true); |
| 862 | 863 | |
| @@ -873,7 +874,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) · 10 unmodified lines | ||
| 873 | 874 | */ |
| 874 | 875 | mmap_write_lock(mm); |
| 875 | 876 | prev = NULL; |
| 876 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 877 | + mas_for_each(&mas, vma, ULONG_MAX) { | |
| 877 | 878 | cond_resched(); |
| 878 | 879 | BUG_ON(!!vma->vm_userfaultfd_ctx.ctx ^ |
| 879 | 880 | !!(vma->vm_flags & __VM_UFFD_FLAGS)); |
| @@ -887,10 +888,13 @@ static int userfaultfd_release(struct inode *inode, struct file *file) · 7 unmodified lines | ||
| 887 | 888 | vma->vm_file, vma->vm_pgoff, |
| 888 | 889 | vma_policy(vma), |
| 889 | 890 | NULL_VM_UFFD_CTX, anon_vma_name(vma)); |
| 890 | − if (prev) | |
| 891 | + if (prev) { | |
| 892 | + mas_pause(&mas); | |
| 891 | 893 | vma = prev; |
| 892 | − else | |
| 894 | + } else { | |
| 893 | 895 | prev = vma; |
| 896 | + } | |
| 897 | + | |
| 894 | 898 | vma->vm_flags = new_flags; |
| 895 | 899 | vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; |
| 896 | 900 | } |
| @@ -1272,6 +1276,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 375 unmodified lines | ||
| 1272 | 1276 | bool found; |
| 1273 | 1277 | bool basic_ioctls; |
| 1274 | 1278 | unsigned long start, end, vma_end; |
| 1279 | + MA_STATE(mas, &mm->mm_mt, 0, 0); | |
| 1275 | 1280 | |
| 1276 | 1281 | user_uffdio_register = (struct uffdio_register __user *) arg; |
| 1277 | 1282 | |
| @@ -1314,7 +1319,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 36 unmodified lines | ||
| 1314 | 1319 | goto out; |
| 1315 | 1320 | |
| 1316 | 1321 | mmap_write_lock(mm); |
| 1317 | − vma = find_vma_prev(mm, start, &prev); | |
| 1322 | + mas_set(&mas, start); | |
| 1323 | + vma = mas_find(&mas, ULONG_MAX); | |
| 1318 | 1324 | if (!vma) |
| 1319 | 1325 | goto out_unlock; |
| 1320 | 1326 | |
| @@ -1339,7 +1345,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 18 unmodified lines | ||
| 1339 | 1345 | */ |
| 1340 | 1346 | found = false; |
| 1341 | 1347 | basic_ioctls = false; |
| 1342 | − for (cur = vma; cur && cur->vm_start < end; cur = cur->vm_next) { | |
| 1348 | + for (cur = vma; cur; cur = mas_next(&mas, end - 1)) { | |
| 1343 | 1349 | cond_resched(); |
| 1344 | 1350 | |
| 1345 | 1351 | BUG_ON(!!cur->vm_userfaultfd_ctx.ctx ^ |
| @@ -1399,8 +1405,10 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 53 unmodified lines | ||
| 1399 | 1405 | } |
| 1400 | 1406 | BUG_ON(!found); |
| 1401 | 1407 | |
| 1402 | − if (vma->vm_start < start) | |
| 1403 | − prev = vma; | |
| 1408 | + mas_set(&mas, start); | |
| 1409 | + prev = mas_prev(&mas, 0); | |
| 1410 | + if (prev != vma) | |
| 1411 | + mas_next(&mas, ULONG_MAX); | |
| 1404 | 1412 | |
| 1405 | 1413 | ret = 0; |
| 1406 | 1414 | do { |
| @@ -1430,6 +1438,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 23 unmodified lines | ||
| 1430 | 1438 | ((struct vm_userfaultfd_ctx){ ctx }), |
| 1431 | 1439 | anon_vma_name(vma)); |
| 1432 | 1440 | if (prev) { |
| 1441 | + /* vma_merge() invalidated the mas */ | |
| 1442 | + mas_pause(&mas); | |
| 1433 | 1443 | vma = prev; |
| 1434 | 1444 | goto next; |
| 1435 | 1445 | } |
| @@ -1437,11 +1447,15 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 1 unmodified line | ||
| 1437 | 1447 | ret = split_vma(mm, vma, start, 1); |
| 1438 | 1448 | if (ret) |
| 1439 | 1449 | break; |
| 1450 | + /* split_vma() invalidated the mas */ | |
| 1451 | + mas_pause(&mas); | |
| 1440 | 1452 | } |
| 1441 | 1453 | if (vma->vm_end > end) { |
| 1442 | 1454 | ret = split_vma(mm, vma, end, 0); |
| 1443 | 1455 | if (ret) |
| 1444 | 1456 | break; |
| 1457 | + /* split_vma() invalidated the mas */ | |
| 1458 | + mas_pause(&mas); | |
| 1445 | 1459 | } |
| 1446 | 1460 | next: |
| 1447 | 1461 | /* |
| @@ -1458,8 +1472,8 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 10 unmodified lines | ||
| 1458 | 1472 | skip: |
| 1459 | 1473 | prev = vma; |
| 1460 | 1474 | start = vma->vm_end; |
| 1461 | − vma = vma->vm_next; | |
| 1462 | − } while (vma && vma->vm_start < end); | |
| 1475 | + vma = mas_next(&mas, end - 1); | |
| 1476 | + } while (vma); | |
| 1463 | 1477 | out_unlock: |
| 1464 | 1478 | mmap_write_unlock(mm); |
| 1465 | 1479 | mmput(mm); |
| @@ -1503,6 +1517,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 37 unmodified lines | ||
| 1503 | 1517 | bool found; |
| 1504 | 1518 | unsigned long start, end, vma_end; |
| 1505 | 1519 | const void __user *buf = (void __user *)arg; |
| 1520 | + MA_STATE(mas, &mm->mm_mt, 0, 0); | |
| 1506 | 1521 | |
| 1507 | 1522 | ret = -EFAULT; |
| 1508 | 1523 | if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister))) |
| @@ -1521,7 +1536,8 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 12 unmodified lines | ||
| 1521 | 1536 | goto out; |
| 1522 | 1537 | |
| 1523 | 1538 | mmap_write_lock(mm); |
| 1524 | − vma = find_vma_prev(mm, start, &prev); | |
| 1539 | + mas_set(&mas, start); | |
| 1540 | + vma = mas_find(&mas, ULONG_MAX); | |
| 1525 | 1541 | if (!vma) |
| 1526 | 1542 | goto out_unlock; |
| 1527 | 1543 | |
| @@ -1546,7 +1562,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 18 unmodified lines | ||
| 1546 | 1562 | */ |
| 1547 | 1563 | found = false; |
| 1548 | 1564 | ret = -EINVAL; |
| 1549 | − for (cur = vma; cur && cur->vm_start < end; cur = cur->vm_next) { | |
| 1565 | + for (cur = vma; cur; cur = mas_next(&mas, end - 1)) { | |
| 1550 | 1566 | cond_resched(); |
| 1551 | 1567 | |
| 1552 | 1568 | BUG_ON(!!cur->vm_userfaultfd_ctx.ctx ^ |
| @@ -1566,8 +1582,10 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 13 unmodified lines | ||
| 1566 | 1582 | } |
| 1567 | 1583 | BUG_ON(!found); |
| 1568 | 1584 | |
| 1569 | − if (vma->vm_start < start) | |
| 1570 | − prev = vma; | |
| 1585 | + mas_set(&mas, start); | |
| 1586 | + prev = mas_prev(&mas, 0); | |
| 1587 | + if (prev != vma) | |
| 1588 | + mas_next(&mas, ULONG_MAX); | |
| 1571 | 1589 | |
| 1572 | 1590 | ret = 0; |
| 1573 | 1591 | do { |
| @@ -1636,8 +1654,8 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 62 unmodified lines | ||
| 1636 | 1654 | skip: |
| 1637 | 1655 | prev = vma; |
| 1638 | 1656 | start = vma->vm_end; |
| 1639 | − vma = vma->vm_next; | |
| 1640 | − } while (vma && vma->vm_start < end); | |
| 1657 | + vma = mas_next(&mas, end - 1); | |
| 1658 | + } while (vma); | |
| 1641 | 1659 | out_unlock: |
| 1642 | 1660 | mmap_write_unlock(mm); |
| 1643 | 1661 | mmput(mm); |
| @@ -2056,20 +2074,11 @@ static void init_once_userfaultfd_ctx(void *mem) · 412 unmodified lines | ||
| 2056 | 2074 | seqcount_spinlock_init(&ctx->refile_seq, &ctx->fault_pending_wqh.lock); |
| 2057 | 2075 | } |
| 2058 | 2076 | |
| 2059 | −SYSCALL_DEFINE1(userfaultfd, int, flags) | |
| 2077 | +static int new_userfaultfd(int flags) | |
| 2060 | 2078 | { |
| 2061 | 2079 | struct userfaultfd_ctx *ctx; |
| 2062 | 2080 | int fd; |
| 2063 | 2081 | |
| 2064 | − if (!sysctl_unprivileged_userfaultfd && | |
| 2065 | − (flags & UFFD_USER_MODE_ONLY) == 0 && | |
| 2066 | − !capable(CAP_SYS_PTRACE)) { | |
| 2067 | − printk_once(KERN_WARNING "uffd: Set unprivileged_userfaultfd " | |
| 2068 | − "sysctl knob to 1 if kernel faults must be handled " | |
| 2069 | − "without obtaining CAP_SYS_PTRACE capability\n"); | |
| 2070 | − return -EPERM; | |
| 2071 | − } | |
| 2072 | − | |
| 2073 | 2082 | BUG_ON(!current->mm); |
| 2074 | 2083 | |
| 2075 | 2084 | /* Check the UFFD_* constants for consistency. */ |
| @@ -2102,8 +2111,60 @@ SYSCALL_DEFINE1(userfaultfd, int, flags) · 26 unmodified lines | ||
| 2102 | 2111 | return fd; |
| 2103 | 2112 | } |
| 2104 | 2113 | |
| 2114 | +static inline bool userfaultfd_syscall_allowed(int flags) | |
| 2115 | +{ | |
| 2116 | + /* Userspace-only page faults are always allowed */ | |
| 2117 | + if (flags & UFFD_USER_MODE_ONLY) | |
| 2118 | + return true; | |
| 2119 | + | |
| 2120 | + /* | |
| 2121 | + * The user is requesting a userfaultfd which can handle kernel faults. | |
| 2122 | + * Privileged users are always allowed to do this. | |
| 2123 | + */ | |
| 2124 | + if (capable(CAP_SYS_PTRACE)) | |
| 2125 | + return true; | |
| 2126 | + | |
| 2127 | + /* Otherwise, access to kernel fault handling is sysctl controlled. */ | |
| 2128 | + return sysctl_unprivileged_userfaultfd; | |
| 2129 | +} | |
| 2130 | + | |
| 2131 | +SYSCALL_DEFINE1(userfaultfd, int, flags) | |
| 2132 | +{ | |
| 2133 | + if (!userfaultfd_syscall_allowed(flags)) | |
| 2134 | + return -EPERM; | |
| 2135 | + | |
| 2136 | + return new_userfaultfd(flags); | |
| 2137 | +} | |
| 2138 | + | |
| 2139 | +static long userfaultfd_dev_ioctl(struct file *file, unsigned int cmd, unsigned long flags) | |
| 2140 | +{ | |
| 2141 | + if (cmd != USERFAULTFD_IOC_NEW) | |
| 2142 | + return -EINVAL; | |
| 2143 | + | |
| 2144 | + return new_userfaultfd(flags); | |
| 2145 | +} | |
| 2146 | + | |
| 2147 | +static const struct file_operations userfaultfd_dev_fops = { | |
| 2148 | + .unlocked_ioctl = userfaultfd_dev_ioctl, | |
| 2149 | + .compat_ioctl = userfaultfd_dev_ioctl, | |
| 2150 | + .owner = THIS_MODULE, | |
| 2151 | + .llseek = noop_llseek, | |
| 2152 | +}; | |
| 2153 | + | |
| 2154 | +static struct miscdevice userfaultfd_misc = { | |
| 2155 | + .minor = MISC_DYNAMIC_MINOR, | |
| 2156 | + .name = "userfaultfd", | |
| 2157 | + .fops = &userfaultfd_dev_fops | |
| 2158 | +}; | |
| 2159 | + | |
| 2105 | 2160 | static int __init userfaultfd_init(void) |
| 2106 | 2161 | { |
| 2162 | + int ret; | |
| 2163 | + | |
| 2164 | + ret = misc_register(&userfaultfd_misc); | |
| 2165 | + if (ret) | |
| 2166 | + return ret; | |
| 2167 | + | |
| 2107 | 2168 | userfaultfd_ctx_cachep = kmem_cache_create("userfaultfd_ctx_cache", |
| 2108 | 2169 | sizeof(struct userfaultfd_ctx), |
| 2109 | 2170 | 0, |
| @@ -2,6 +2,8 @@ · 1 unmodified line | ||
| 2 | 2 | #ifndef _ASM_GENERIC_CACHEFLUSH_H |
| 3 | 3 | #define _ASM_GENERIC_CACHEFLUSH_H |
| 4 | 4 | |
| 5 | +#include <linux/instrumented.h> | |
| 6 | + | |
| 5 | 7 | struct mm_struct; |
| 6 | 8 | struct vm_area_struct; |
| 7 | 9 | struct page; |
| @@ -105,14 +107,22 @@ static inline void flush_cache_vunmap(unsigned long start, unsigned long end) · 97 unmodified lines | ||
| 105 | 107 | #ifndef copy_to_user_page |
| 106 | 108 | #define copy_to_user_page(vma, page, vaddr, dst, src, len) \ |
| 107 | 109 | do { \ |
| 110 | + instrument_copy_to_user((void __user *)dst, src, len); \ | |
| 108 | 111 | memcpy(dst, src, len); \ |
| 109 | 112 | flush_icache_user_page(vma, page, vaddr, len); \ |
| 110 | 113 | } while (0) |
| 111 | 114 | #endif |
| 112 | 115 | |
| 116 | + | |
| 113 | 117 | #ifndef copy_from_user_page |
| 114 | −#define copy_from_user_page(vma, page, vaddr, dst, src, len) \ | |
| 115 | − memcpy(dst, src, len) | |
| 118 | +#define copy_from_user_page(vma, page, vaddr, dst, src, len) \ | |
| 119 | + do { \ | |
| 120 | + instrument_copy_from_user_before(dst, (void __user *)src, \ | |
| 121 | + len); \ | |
| 122 | + memcpy(dst, src, len); \ | |
| 123 | + instrument_copy_from_user_after(dst, (void __user *)src, len, \ | |
| 124 | + 0); \ | |
| 125 | + } while (0) | |
| 116 | 126 | #endif |
| 117 | 127 | |
| 118 | 128 | #endif /* _ASM_GENERIC_CACHEFLUSH_H */ |
| @@ -225,8 +225,6 @@ struct buffer_head *__getblk_gfp(struct block_device *bdev, sector_t block, · 224 unmodified lines | ||
| 225 | 225 | void __brelse(struct buffer_head *); |
| 226 | 226 | void __bforget(struct buffer_head *); |
| 227 | 227 | void __breadahead(struct block_device *, sector_t block, unsigned int size); |
| 228 | −void __breadahead_gfp(struct block_device *, sector_t block, unsigned int size, | |
| 229 | − gfp_t gfp); | |
| 230 | 228 | struct buffer_head *__bread_gfp(struct block_device *, |
| 231 | 229 | sector_t block, unsigned size, gfp_t gfp); |
| 232 | 230 | void invalidate_bh_lrus(void); |
| @@ -236,7 +234,6 @@ struct buffer_head *alloc_buffer_head(gfp_t gfp_flags); · 3 unmodified lines | ||
| 236 | 234 | void free_buffer_head(struct buffer_head * bh); |
| 237 | 235 | void unlock_buffer(struct buffer_head *bh); |
| 238 | 236 | void __lock_buffer(struct buffer_head *bh); |
| 239 | −void ll_rw_block(blk_opf_t, int, struct buffer_head * bh[]); | |
| 240 | 237 | int sync_dirty_buffer(struct buffer_head *bh); |
| 241 | 238 | int __sync_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags); |
| 242 | 239 | void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags); |
| @@ -244,7 +241,9 @@ void submit_bh(blk_opf_t, struct buffer_head *); · 1 unmodified line | ||
| 244 | 241 | void write_boundary_block(struct block_device *bdev, |
| 245 | 242 | sector_t bblock, unsigned blocksize); |
| 246 | 243 | int bh_uptodate_or_lock(struct buffer_head *bh); |
| 247 | −int bh_submit_read(struct buffer_head *bh); | |
| 244 | +int __bh_read(struct buffer_head *bh, blk_opf_t op_flags, bool wait); | |
| 245 | +void __bh_read_batch(int nr, struct buffer_head *bhs[], | |
| 246 | + blk_opf_t op_flags, bool force_lock); | |
| 248 | 247 | |
| 249 | 248 | extern int buffer_heads_over_limit; |
| 250 | 249 | |
| @@ -351,12 +350,6 @@ sb_breadahead(struct super_block *sb, sector_t block) · 100 unmodified lines | ||
| 351 | 350 | __breadahead(sb->s_bdev, block, sb->s_blocksize); |
| 352 | 351 | } |
| 353 | 352 | |
| 354 | −static inline void | |
| 355 | −sb_breadahead_unmovable(struct super_block *sb, sector_t block) | |
| 356 | −{ | |
| 357 | − __breadahead_gfp(sb->s_bdev, block, sb->s_blocksize, 0); | |
| 358 | −} | |
| 359 | − | |
| 360 | 353 | static inline struct buffer_head * |
| 361 | 354 | sb_getblk(struct super_block *sb, sector_t block) |
| 362 | 355 | { |
| @@ -418,6 +411,41 @@ static inline struct buffer_head *__getblk(struct block_device *bdev, · 55 unmodified lines | ||
| 418 | 411 | return __getblk_gfp(bdev, block, size, __GFP_MOVABLE); |
| 419 | 412 | } |
| 420 | 413 | |
| 414 | +static inline void bh_readahead(struct buffer_head *bh, blk_opf_t op_flags) | |
| 415 | +{ | |
| 416 | + if (!buffer_uptodate(bh) && trylock_buffer(bh)) { | |
| 417 | + if (!buffer_uptodate(bh)) | |
| 418 | + __bh_read(bh, op_flags, false); | |
| 419 | + else | |
| 420 | + unlock_buffer(bh); | |
| 421 | + } | |
| 422 | +} | |
| 423 | + | |
| 424 | +static inline void bh_read_nowait(struct buffer_head *bh, blk_opf_t op_flags) | |
| 425 | +{ | |
| 426 | + if (!bh_uptodate_or_lock(bh)) | |
| 427 | + __bh_read(bh, op_flags, false); | |
| 428 | +} | |
| 429 | + | |
| 430 | +/* Returns 1 if buffer uptodated, 0 on success, and -EIO on error. */ | |
| 431 | +static inline int bh_read(struct buffer_head *bh, blk_opf_t op_flags) | |
| 432 | +{ | |
| 433 | + if (bh_uptodate_or_lock(bh)) | |
| 434 | + return 1; | |
| 435 | + return __bh_read(bh, op_flags, true); | |
| 436 | +} | |
| 437 | + | |
| 438 | +static inline void bh_read_batch(int nr, struct buffer_head *bhs[]) | |
| 439 | +{ | |
| 440 | + __bh_read_batch(nr, bhs, 0, true); | |
| 441 | +} | |
| 442 | + | |
| 443 | +static inline void bh_readahead_batch(int nr, struct buffer_head *bhs[], | |
| 444 | + blk_opf_t op_flags) | |
| 445 | +{ | |
| 446 | + __bh_read_batch(nr, bhs, op_flags, false); | |
| 447 | +} | |
| 448 | + | |
| 421 | 449 | /** |
| 422 | 450 | * __bread() - reads a specified block and returns the bh |
| 423 | 451 | * @bdev: the block_device to read from |
| @@ -85,4 +85,17 @@ · 84 unmodified lines | ||
| 85 | 85 | #define cache_line_size() L1_CACHE_BYTES |
| 86 | 86 | #endif |
| 87 | 87 | |
| 88 | +/* | |
| 89 | + * Helper to add padding within a struct to ensure data fall into separate | |
| 90 | + * cachelines. | |
| 91 | + */ | |
| 92 | +#if defined(CONFIG_SMP) | |
| 93 | +struct cacheline_padding { | |
| 94 | + char x[0]; | |
| 95 | +} ____cacheline_internodealigned_in_smp; | |
| 96 | +#define CACHELINE_PADDING(name) struct cacheline_padding name | |
| 97 | +#else | |
| 98 | +#define CACHELINE_PADDING(name) | |
| 99 | +#endif | |
| 100 | + | |
| 88 | 101 | #endif /* __LINUX_CACHE_H */ |
| @@ -433,6 +433,18 @@ static inline void cgroup_put(struct cgroup *cgrp) · 432 unmodified lines | ||
| 433 | 433 | css_put(&cgrp->self); |
| 434 | 434 | } |
| 435 | 435 | |
| 436 | +extern struct mutex cgroup_mutex; | |
| 437 | + | |
| 438 | +static inline void cgroup_lock(void) | |
| 439 | +{ | |
| 440 | + mutex_lock(&cgroup_mutex); | |
| 441 | +} | |
| 442 | + | |
| 443 | +static inline void cgroup_unlock(void) | |
| 444 | +{ | |
| 445 | + mutex_unlock(&cgroup_mutex); | |
| 446 | +} | |
| 447 | + | |
| 436 | 448 | /** |
| 437 | 449 | * task_css_set_check - obtain a task's css_set with extra access conditions |
| 438 | 450 | * @task: the task to obtain css_set for |
| @@ -447,7 +459,6 @@ static inline void cgroup_put(struct cgroup *cgrp) · 8 unmodified lines | ||
| 447 | 459 | * as locks used during the cgroup_subsys::attach() methods. |
| 448 | 460 | */ |
| 449 | 461 | #ifdef CONFIG_PROVE_RCU |
| 450 | −extern struct mutex cgroup_mutex; | |
| 451 | 462 | extern spinlock_t css_set_lock; |
| 452 | 463 | #define task_css_set_check(task, __c) \ |
| 453 | 464 | rcu_dereference_check((task)->cgroups, \ |
| @@ -707,6 +718,8 @@ struct cgroup; · 253 unmodified lines | ||
| 707 | 718 | static inline u64 cgroup_id(const struct cgroup *cgrp) { return 1; } |
| 708 | 719 | static inline void css_get(struct cgroup_subsys_state *css) {} |
| 709 | 720 | static inline void css_put(struct cgroup_subsys_state *css) {} |
| 721 | +static inline void cgroup_lock(void) {} | |
| 722 | +static inline void cgroup_unlock(void) {} | |
| 710 | 723 | static inline int cgroup_attach_task_all(struct task_struct *from, |
| 711 | 724 | struct task_struct *t) { return 0; } |
| 712 | 725 | static inline int cgroupstats_build(struct cgroupstats *stats, |
| @@ -51,6 +51,29 @@ · 50 unmodified lines | ||
| 51 | 51 | #define __no_sanitize_undefined |
| 52 | 52 | #endif |
| 53 | 53 | |
| 54 | +#if __has_feature(memory_sanitizer) | |
| 55 | +#define __SANITIZE_MEMORY__ | |
| 56 | +/* | |
| 57 | + * Unlike other sanitizers, KMSAN still inserts code into functions marked with | |
| 58 | + * no_sanitize("kernel-memory"). Using disable_sanitizer_instrumentation | |
| 59 | + * provides the behavior consistent with other __no_sanitize_ attributes, | |
| 60 | + * guaranteeing that __no_sanitize_memory functions remain uninstrumented. | |
| 61 | + */ | |
| 62 | +#define __no_sanitize_memory __disable_sanitizer_instrumentation | |
| 63 | + | |
| 64 | +/* | |
| 65 | + * The __no_kmsan_checks attribute ensures that a function does not produce | |
| 66 | + * false positive reports by: | |
| 67 | + * - initializing all local variables and memory stores in this function; | |
| 68 | + * - skipping all shadow checks; | |
| 69 | + * - passing initialized arguments to this function's callees. | |
| 70 | + */ | |
| 71 | +#define __no_kmsan_checks __attribute__((no_sanitize("kernel-memory"))) | |
| 72 | +#else | |
| 73 | +#define __no_sanitize_memory | |
| 74 | +#define __no_kmsan_checks | |
| 75 | +#endif | |
| 76 | + | |
| 54 | 77 | /* |
| 55 | 78 | * Support for __has_feature(coverage_sanitizer) was added in Clang 13 together |
| 56 | 79 | * with no_sanitize("coverage"). Prior versions of Clang support coverage |
| @@ -114,6 +114,12 @@ · 113 unmodified lines | ||
| 114 | 114 | #define __SANITIZE_ADDRESS__ |
| 115 | 115 | #endif |
| 116 | 116 | |
| 117 | +/* | |
| 118 | + * GCC does not support KMSAN. | |
| 119 | + */ | |
| 120 | +#define __no_sanitize_memory | |
| 121 | +#define __no_kmsan_checks | |
| 122 | + | |
| 117 | 123 | /* |
| 118 | 124 | * Turn individual warnings and errors on and off locally, depending |
| 119 | 125 | * on version. |
| @@ -233,7 +233,8 @@ struct ftrace_likely_data { · 232 unmodified lines | ||
| 233 | 233 | /* Section for code which can't be instrumented at all */ |
| 234 | 234 | #define noinstr \ |
| 235 | 235 | noinline notrace __attribute((__section__(".noinstr.text"))) \ |
| 236 | − __no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage | |
| 236 | + __no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage \ | |
| 237 | + __no_sanitize_memory | |
| 237 | 238 | |
| 238 | 239 | #endif /* __KERNEL__ */ |
| 239 | 240 | |
| @@ -216,13 +216,26 @@ struct damos_stat { · 215 unmodified lines | ||
| 216 | 216 | }; |
| 217 | 217 | |
| 218 | 218 | /** |
| 219 | − * struct damos - Represents a Data Access Monitoring-based Operation Scheme. | |
| 219 | + * struct damos_access_pattern - Target access pattern of the given scheme. | |
| 220 | 220 | * @min_sz_region: Minimum size of target regions. |
| 221 | 221 | * @max_sz_region: Maximum size of target regions. |
| 222 | 222 | * @min_nr_accesses: Minimum ``->nr_accesses`` of target regions. |
| 223 | 223 | * @max_nr_accesses: Maximum ``->nr_accesses`` of target regions. |
| 224 | 224 | * @min_age_region: Minimum age of target regions. |
| 225 | 225 | * @max_age_region: Maximum age of target regions. |
| 226 | + */ | |
| 227 | +struct damos_access_pattern { | |
| 228 | + unsigned long min_sz_region; | |
| 229 | + unsigned long max_sz_region; | |
| 230 | + unsigned int min_nr_accesses; | |
| 231 | + unsigned int max_nr_accesses; | |
| 232 | + unsigned int min_age_region; | |
| 233 | + unsigned int max_age_region; | |
| 234 | +}; | |
| 235 | + | |
| 236 | +/** | |
| 237 | + * struct damos - Represents a Data Access Monitoring-based Operation Scheme. | |
| 238 | + * @pattern: Access pattern of target regions. | |
| 226 | 239 | * @action: &damo_action to be applied to the target regions. |
| 227 | 240 | * @quota: Control the aggressiveness of this scheme. |
| 228 | 241 | * @wmarks: Watermarks for automated (in)activation of this scheme. |
| @@ -230,10 +243,8 @@ struct damos_stat { · 1 unmodified line | ||
| 230 | 243 | * @list: List head for siblings. |
| 231 | 244 | * |
| 232 | 245 | * For each aggregation interval, DAMON finds regions which fit in the |
| 233 | − * condition (&min_sz_region, &max_sz_region, &min_nr_accesses, | |
| 234 | − * &max_nr_accesses, &min_age_region, &max_age_region) and applies &action to | |
| 235 | − * those. To avoid consuming too much CPU time or IO resources for the | |
| 236 | − * &action, "a is used. | |
| 246 | + * &pattern and applies &action to those. To avoid consuming too much | |
| 247 | + * CPU time or IO resources for the &action, "a is used. | |
| 237 | 248 | * |
| 238 | 249 | * To do the work only when needed, schemes can be activated for specific |
| 239 | 250 | * system situations using &wmarks. If all schemes that registered to the |
| @@ -248,12 +259,7 @@ struct damos_stat { · 8 unmodified lines | ||
| 248 | 259 | * &action is applied. |
| 249 | 260 | */ |
| 250 | 261 | struct damos { |
| 251 | − unsigned long min_sz_region; | |
| 252 | − unsigned long max_sz_region; | |
| 253 | − unsigned int min_nr_accesses; | |
| 254 | − unsigned int max_nr_accesses; | |
| 255 | − unsigned int min_age_region; | |
| 256 | − unsigned int max_age_region; | |
| 262 | + struct damos_access_pattern pattern; | |
| 257 | 263 | enum damos_action action; |
| 258 | 264 | struct damos_quota quota; |
| 259 | 265 | struct damos_watermarks wmarks; |
| @@ -340,7 +346,7 @@ struct damon_operations { · 80 unmodified lines | ||
| 340 | 346 | unsigned long (*apply_scheme)(struct damon_ctx *context, |
| 341 | 347 | struct damon_target *t, struct damon_region *r, |
| 342 | 348 | struct damos *scheme); |
| 343 | − bool (*target_valid)(void *target); | |
| 349 | + bool (*target_valid)(struct damon_target *t); | |
| 344 | 350 | void (*cleanup)(struct damon_ctx *context); |
| 345 | 351 | }; |
| 346 | 352 | |
| @@ -383,13 +389,15 @@ struct damon_callback { · 36 unmodified lines | ||
| 383 | 389 | }; |
| 384 | 390 | |
| 385 | 391 | /** |
| 386 | − * struct damon_ctx - Represents a context for each monitoring. This is the | |
| 387 | − * main interface that allows users to set the attributes and get the results | |
| 388 | − * of the monitoring. | |
| 392 | + * struct damon_attrs - Monitoring attributes for accuracy/overhead control. | |
| 389 | 393 | * |
| 390 | 394 | * @sample_interval: The time between access samplings. |
| 391 | 395 | * @aggr_interval: The time between monitor results aggregations. |
| 392 | 396 | * @ops_update_interval: The time between monitoring operations updates. |
| 397 | + * @min_nr_regions: The minimum number of adaptive monitoring | |
| 398 | + * regions. | |
| 399 | + * @max_nr_regions: The maximum number of adaptive monitoring | |
| 400 | + * regions. | |
| 393 | 401 | * |
| 394 | 402 | * For each @sample_interval, DAMON checks whether each region is accessed or |
| 395 | 403 | * not. It aggregates and keeps the access information (number of accesses to |
| @@ -399,7 +407,21 @@ struct damon_callback { · 3 unmodified lines | ||
| 399 | 407 | * @ops_update_interval. All time intervals are in micro-seconds. |
| 400 | 408 | * Please refer to &struct damon_operations and &struct damon_callback for more |
| 401 | 409 | * detail. |
| 410 | + */ | |
| 411 | +struct damon_attrs { | |
| 412 | + unsigned long sample_interval; | |
| 413 | + unsigned long aggr_interval; | |
| 414 | + unsigned long ops_update_interval; | |
| 415 | + unsigned long min_nr_regions; | |
| 416 | + unsigned long max_nr_regions; | |
| 417 | +}; | |
| 418 | + | |
| 419 | +/** | |
| 420 | + * struct damon_ctx - Represents a context for each monitoring. This is the | |
| 421 | + * main interface that allows users to set the attributes and get the results | |
| 422 | + * of the monitoring. | |
| 402 | 423 | * |
| 424 | + * @attrs: Monitoring attributes for accuracy/overhead control. | |
| 403 | 425 | * @kdamond: Kernel thread who does the monitoring. |
| 404 | 426 | * @kdamond_lock: Mutex for the synchronizations with @kdamond. |
| 405 | 427 | * |
| @@ -421,15 +443,11 @@ struct damon_callback { · 15 unmodified lines | ||
| 421 | 443 | * @ops: Set of monitoring operations for given use cases. |
| 422 | 444 | * @callback: Set of callbacks for monitoring events notifications. |
| 423 | 445 | * |
| 424 | − * @min_nr_regions: The minimum number of adaptive monitoring regions. | |
| 425 | − * @max_nr_regions: The maximum number of adaptive monitoring regions. | |
| 426 | 446 | * @adaptive_targets: Head of monitoring targets (&damon_target) list. |
| 427 | 447 | * @schemes: Head of schemes (&damos) list. |
| 428 | 448 | */ |
| 429 | 449 | struct damon_ctx { |
| 430 | − unsigned long sample_interval; | |
| 431 | − unsigned long aggr_interval; | |
| 432 | − unsigned long ops_update_interval; | |
| 450 | + struct damon_attrs attrs; | |
| 433 | 451 | |
| 434 | 452 | /* private: internal use only */ |
| 435 | 453 | struct timespec64 last_aggregation; |
| @@ -442,8 +460,6 @@ struct damon_ctx { · 6 unmodified lines | ||
| 442 | 460 | struct damon_operations ops; |
| 443 | 461 | struct damon_callback callback; |
| 444 | 462 | |
| 445 | − unsigned long min_nr_regions; | |
| 446 | − unsigned long max_nr_regions; | |
| 447 | 463 | struct list_head adaptive_targets; |
| 448 | 464 | struct list_head schemes; |
| 449 | 465 | }; |
| @@ -463,9 +479,17 @@ static inline struct damon_region *damon_last_region(struct damon_target *t) · 13 unmodified lines | ||
| 463 | 479 | return list_last_entry(&t->regions_list, struct damon_region, list); |
| 464 | 480 | } |
| 465 | 481 | |
| 482 | +static inline struct damon_region *damon_first_region(struct damon_target *t) | |
| 483 | +{ | |
| 484 | + return list_first_entry(&t->regions_list, struct damon_region, list); | |
| 485 | +} | |
| 486 | + | |
| 466 | 487 | #define damon_for_each_region(r, t) \ |
| 467 | 488 | list_for_each_entry(r, &t->regions_list, list) |
| 468 | 489 | |
| 490 | +#define damon_for_each_region_from(r, t) \ | |
| 491 | + list_for_each_entry_from(r, &t->regions_list, list) | |
| 492 | + | |
| 469 | 493 | #define damon_for_each_region_safe(r, next, t) \ |
| 470 | 494 | list_for_each_entry_safe(r, next, &t->regions_list, list) |
| 471 | 495 | |
| @@ -501,12 +525,9 @@ void damon_destroy_region(struct damon_region *r, struct damon_target *t); · 29 unmodified lines | ||
| 501 | 525 | int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, |
| 502 | 526 | unsigned int nr_ranges); |
| 503 | 527 | |
| 504 | −struct damos *damon_new_scheme( | |
| 505 | − unsigned long min_sz_region, unsigned long max_sz_region, | |
| 506 | − unsigned int min_nr_accesses, unsigned int max_nr_accesses, | |
| 507 | − unsigned int min_age_region, unsigned int max_age_region, | |
| 508 | − enum damos_action action, struct damos_quota *quota, | |
| 509 | − struct damos_watermarks *wmarks); | |
| 528 | +struct damos *damon_new_scheme(struct damos_access_pattern *pattern, | |
| 529 | + enum damos_action action, struct damos_quota *quota, | |
| 530 | + struct damos_watermarks *wmarks); | |
| 510 | 531 | void damon_add_scheme(struct damon_ctx *ctx, struct damos *s); |
| 511 | 532 | void damon_destroy_scheme(struct damos *s); |
| 512 | 533 | |
| @@ -519,10 +540,8 @@ unsigned int damon_nr_regions(struct damon_target *t); · 6 unmodified lines | ||
| 519 | 540 | |
| 520 | 541 | struct damon_ctx *damon_new_ctx(void); |
| 521 | 542 | void damon_destroy_ctx(struct damon_ctx *ctx); |
| 522 | −int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int, | |
| 523 | − unsigned long aggr_int, unsigned long ops_upd_int, | |
| 524 | − unsigned long min_nr_reg, unsigned long max_nr_reg); | |
| 525 | −int damon_set_schemes(struct damon_ctx *ctx, | |
| 543 | +int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs); | |
| 544 | +void damon_set_schemes(struct damon_ctx *ctx, | |
| 526 | 545 | struct damos **schemes, ssize_t nr_schemes); |
| 527 | 546 | int damon_nr_running_ctxs(void); |
| 528 | 547 | bool damon_is_registered_ops(enum damon_ops_id id); |
| @@ -538,6 +557,9 @@ static inline bool damon_target_has_pid(const struct damon_ctx *ctx) · 9 unmodified lines | ||
| 538 | 557 | int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive); |
| 539 | 558 | int damon_stop(struct damon_ctx **ctxs, int nr_ctxs); |
| 540 | 559 | |
| 560 | +int damon_set_region_biggest_system_ram_default(struct damon_target *t, | |
| 561 | + unsigned long *start, unsigned long *end); | |
| 562 | + | |
| 541 | 563 | #endif /* CONFIG_DAMON */ |
| 542 | 564 | |
| 543 | 565 | #endif /* _DAMON_H */ |
| @@ -73,8 +73,8 @@ extern int delayacct_add_tsk(struct taskstats *, struct task_struct *); · 72 unmodified lines | ||
| 73 | 73 | extern __u64 __delayacct_blkio_ticks(struct task_struct *); |
| 74 | 74 | extern void __delayacct_freepages_start(void); |
| 75 | 75 | extern void __delayacct_freepages_end(void); |
| 76 | −extern void __delayacct_thrashing_start(void); | |
| 77 | −extern void __delayacct_thrashing_end(void); | |
| 76 | +extern void __delayacct_thrashing_start(bool *in_thrashing); | |
| 77 | +extern void __delayacct_thrashing_end(bool *in_thrashing); | |
| 78 | 78 | extern void __delayacct_swapin_start(void); |
| 79 | 79 | extern void __delayacct_swapin_end(void); |
| 80 | 80 | extern void __delayacct_compact_start(void); |
| @@ -143,22 +143,22 @@ static inline void delayacct_freepages_end(void) · 62 unmodified lines | ||
| 143 | 143 | __delayacct_freepages_end(); |
| 144 | 144 | } |
| 145 | 145 | |
| 146 | −static inline void delayacct_thrashing_start(void) | |
| 146 | +static inline void delayacct_thrashing_start(bool *in_thrashing) | |
| 147 | 147 | { |
| 148 | 148 | if (!static_branch_unlikely(&delayacct_key)) |
| 149 | 149 | return; |
| 150 | 150 | |
| 151 | 151 | if (current->delays) |
| 152 | − __delayacct_thrashing_start(); | |
| 152 | + __delayacct_thrashing_start(in_thrashing); | |
| 153 | 153 | } |
| 154 | 154 | |
| 155 | −static inline void delayacct_thrashing_end(void) | |
| 155 | +static inline void delayacct_thrashing_end(bool *in_thrashing) | |
| 156 | 156 | { |
| 157 | 157 | if (!static_branch_unlikely(&delayacct_key)) |
| 158 | 158 | return; |
| 159 | 159 | |
| 160 | 160 | if (current->delays) |
| 161 | − __delayacct_thrashing_end(); | |
| 161 | + __delayacct_thrashing_end(in_thrashing); | |
| 162 | 162 | } |
| 163 | 163 | |
| 164 | 164 | static inline void delayacct_swapin_start(void) |
| @@ -237,9 +237,9 @@ static inline void delayacct_freepages_start(void) · 72 unmodified lines | ||
| 237 | 237 | {} |
| 238 | 238 | static inline void delayacct_freepages_end(void) |
| 239 | 239 | {} |
| 240 | −static inline void delayacct_thrashing_start(void) | |
| 240 | +static inline void delayacct_thrashing_start(bool *in_thrashing) | |
| 241 | 241 | {} |
| 242 | −static inline void delayacct_thrashing_end(void) | |
| 242 | +static inline void delayacct_thrashing_end(bool *in_thrashing) | |
| 243 | 243 | {} |
| 244 | 244 | static inline void delayacct_swapin_start(void) |
| 245 | 245 | {} |
| @@ -328,8 +328,10 @@ __FORTIFY_INLINE void fortify_memset_chk(__kernel_size_t size, · 327 unmodified lines | ||
| 328 | 328 | * __struct_size() vs __member_size() must be captured here to avoid |
| 329 | 329 | * evaluating argument side-effects further into the macro layers. |
| 330 | 330 | */ |
| 331 | +#ifndef CONFIG_KMSAN | |
| 331 | 332 | #define memset(p, c, s) __fortify_memset_chk(p, c, s, \ |
| 332 | 333 | __struct_size(p), __member_size(p)) |
| 334 | +#endif | |
| 333 | 335 | |
| 334 | 336 | /* |
| 335 | 337 | * To make sure the compiler can enforce protection against buffer overflows, |
| @@ -18,6 +18,9 @@ static inline int gfp_migratetype(const gfp_t gfp_flags) · 17 unmodified lines | ||
| 18 | 18 | VM_WARN_ON((gfp_flags & GFP_MOVABLE_MASK) == GFP_MOVABLE_MASK); |
| 19 | 19 | BUILD_BUG_ON((1UL << GFP_MOVABLE_SHIFT) != ___GFP_MOVABLE); |
| 20 | 20 | BUILD_BUG_ON((___GFP_MOVABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_MOVABLE); |
| 21 | + BUILD_BUG_ON((___GFP_RECLAIMABLE >> GFP_MOVABLE_SHIFT) != MIGRATE_RECLAIMABLE); | |
| 22 | + BUILD_BUG_ON(((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> | |
| 23 | + GFP_MOVABLE_SHIFT) != MIGRATE_HIGHATOMIC); | |
| 21 | 24 | |
| 22 | 25 | if (unlikely(page_group_by_mobility_disabled)) |
| 23 | 26 | return MIGRATE_UNMOVABLE; |
| @@ -33,29 +36,6 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) · 9 unmodified lines | ||
| 33 | 36 | return !!(gfp_flags & __GFP_DIRECT_RECLAIM); |
| 34 | 37 | } |
| 35 | 38 | |
| 36 | −/** | |
| 37 | − * gfpflags_normal_context - is gfp_flags a normal sleepable context? | |
| 38 | − * @gfp_flags: gfp_flags to test | |
| 39 | − * | |
| 40 | − * Test whether @gfp_flags indicates that the allocation is from the | |
| 41 | − * %current context and allowed to sleep. | |
| 42 | − * | |
| 43 | − * An allocation being allowed to block doesn't mean it owns the %current | |
| 44 | − * context. When direct reclaim path tries to allocate memory, the | |
| 45 | − * allocation context is nested inside whatever %current was doing at the | |
| 46 | − * time of the original allocation. The nested allocation may be allowed | |
| 47 | − * to block but modifying anything %current owns can corrupt the outer | |
| 48 | − * context's expectations. | |
| 49 | − * | |
| 50 | − * %true result from this function indicates that the allocation context | |
| 51 | − * can sleep and use anything that's associated with %current. | |
| 52 | − */ | |
| 53 | −static inline bool gfpflags_normal_context(const gfp_t gfp_flags) | |
| 54 | −{ | |
| 55 | − return (gfp_flags & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC)) == | |
| 56 | − __GFP_DIRECT_RECLAIM; | |
| 57 | −} | |
| 58 | − | |
| 59 | 39 | #ifdef CONFIG_HIGHMEM |
| 60 | 40 | #define OPT_ZONE_HIGHMEM ZONE_HIGHMEM |
| 61 | 41 | #else |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <linux/kernel.h> |
| 7 | 7 | #include <linux/bug.h> |
| 8 | 8 | #include <linux/cacheflush.h> |
| 9 | +#include <linux/kmsan.h> | |
| 9 | 10 | #include <linux/mm.h> |
| 10 | 11 | #include <linux/uaccess.h> |
| 11 | 12 | #include <linux/hardirq.h> |
| @@ -311,6 +312,7 @@ static inline void copy_user_highpage(struct page *to, struct page *from, · 299 unmodified lines | ||
| 311 | 312 | vfrom = kmap_local_page(from); |
| 312 | 313 | vto = kmap_local_page(to); |
| 313 | 314 | copy_user_page(vto, vfrom, vaddr, to); |
| 315 | + kmsan_unpoison_memory(page_address(to), PAGE_SIZE); | |
| 314 | 316 | kunmap_local(vto); |
| 315 | 317 | kunmap_local(vfrom); |
| 316 | 318 | } |
| @@ -326,6 +328,7 @@ static inline void copy_highpage(struct page *to, struct page *from) · 9 unmodified lines | ||
| 326 | 328 | vfrom = kmap_local_page(from); |
| 327 | 329 | vto = kmap_local_page(to); |
| 328 | 330 | copy_page(vto, vfrom); |
| 331 | + kmsan_copy_page_meta(to, from); | |
| 329 | 332 | kunmap_local(vto); |
| 330 | 333 | kunmap_local(vfrom); |
| 331 | 334 | } |
| @@ -168,9 +168,8 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) · 167 unmodified lines | ||
| 168 | 168 | !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode); |
| 169 | 169 | } |
| 170 | 170 | |
| 171 | −bool hugepage_vma_check(struct vm_area_struct *vma, | |
| 172 | − unsigned long vm_flags, | |
| 173 | − bool smaps, bool in_pf); | |
| 171 | +bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, | |
| 172 | + bool smaps, bool in_pf, bool enforce_sysfs); | |
| 174 | 173 | |
| 175 | 174 | #define transparent_hugepage_use_zero_page() \ |
| 176 | 175 | (transparent_hugepage_flags & \ |
| @@ -219,6 +218,9 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, · 42 unmodified lines | ||
| 219 | 218 | |
| 220 | 219 | int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags, |
| 221 | 220 | int advice); |
| 221 | +int madvise_collapse(struct vm_area_struct *vma, | |
| 222 | + struct vm_area_struct **prev, | |
| 223 | + unsigned long start, unsigned long end); | |
| 222 | 224 | void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, |
| 223 | 225 | unsigned long end, long adjust_next); |
| 224 | 226 | spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma); |
| @@ -321,8 +323,8 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, · 96 unmodified lines | ||
| 321 | 323 | } |
| 322 | 324 | |
| 323 | 325 | static inline bool hugepage_vma_check(struct vm_area_struct *vma, |
| 324 | − unsigned long vm_flags, | |
| 325 | − bool smaps, bool in_pf) | |
| 326 | + unsigned long vm_flags, bool smaps, | |
| 327 | + bool in_pf, bool enforce_sysfs) | |
| 326 | 328 | { |
| 327 | 329 | return false; |
| 328 | 330 | } |
| @@ -362,9 +364,16 @@ static inline void split_huge_pmd_address(struct vm_area_struct *vma, · 33 unmodified lines | ||
| 362 | 364 | static inline int hugepage_madvise(struct vm_area_struct *vma, |
| 363 | 365 | unsigned long *vm_flags, int advice) |
| 364 | 366 | { |
| 365 | − BUG(); | |
| 366 | − return 0; | |
| 367 | + return -EINVAL; | |
| 368 | +} | |
| 369 | + | |
| 370 | +static inline int madvise_collapse(struct vm_area_struct *vma, | |
| 371 | + struct vm_area_struct **prev, | |
| 372 | + unsigned long start, unsigned long end) | |
| 373 | +{ | |
| 374 | + return -EINVAL; | |
| 367 | 375 | } |
| 376 | + | |
| 368 | 377 | static inline void vma_adjust_trans_huge(struct vm_area_struct *vma, |
| 369 | 378 | unsigned long start, |
| 370 | 379 | unsigned long end, |
| @@ -435,6 +444,11 @@ static inline int split_folio_to_list(struct folio *folio, · 64 unmodified lines | ||
| 435 | 444 | return split_huge_page_to_list(&folio->page, list); |
| 436 | 445 | } |
| 437 | 446 | |
| 447 | +static inline int split_folio(struct folio *folio) | |
| 448 | +{ | |
| 449 | + return split_folio_to_list(folio, NULL); | |
| 450 | +} | |
| 451 | + | |
| 438 | 452 | /* |
| 439 | 453 | * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to |
| 440 | 454 | * limitations in the implementation like arm64 MTE can override this to |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | struct ctl_table; |
| 17 | 17 | struct user_struct; |
| 18 | 18 | struct mmu_gather; |
| 19 | +struct node; | |
| 19 | 20 | |
| 20 | 21 | #ifndef CONFIG_ARCH_HAS_HUGEPD |
| 21 | 22 | typedef struct { unsigned long pd; } hugepd_t; |
| @@ -114,6 +115,12 @@ struct file_region { · 92 unmodified lines | ||
| 114 | 115 | #endif |
| 115 | 116 | }; |
| 116 | 117 | |
| 118 | +struct hugetlb_vma_lock { | |
| 119 | + struct kref refs; | |
| 120 | + struct rw_semaphore rw_sema; | |
| 121 | + struct vm_area_struct *vma; | |
| 122 | +}; | |
| 123 | + | |
| 117 | 124 | extern struct resv_map *resv_map_alloc(void); |
| 118 | 125 | void resv_map_release(struct kref *ref); |
| 119 | 126 | |
| @@ -126,7 +133,7 @@ struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages, · 6 unmodified lines | ||
| 126 | 133 | long min_hpages); |
| 127 | 134 | void hugepage_put_subpool(struct hugepage_subpool *spool); |
| 128 | 135 | |
| 129 | −void reset_vma_resv_huge_pages(struct vm_area_struct *vma); | |
| 136 | +void hugetlb_dup_vma_private(struct vm_area_struct *vma); | |
| 130 | 137 | void clear_vma_resv_huge_pages(struct vm_area_struct *vma); |
| 131 | 138 | int hugetlb_sysctl_handler(struct ctl_table *, int, void *, size_t *, loff_t *); |
| 132 | 139 | int hugetlb_overcommit_handler(struct ctl_table *, int, void *, size_t *, |
| @@ -214,6 +221,14 @@ struct page *follow_huge_pud(struct mm_struct *mm, unsigned long address, · 81 unmodified lines | ||
| 214 | 221 | struct page *follow_huge_pgd(struct mm_struct *mm, unsigned long address, |
| 215 | 222 | pgd_t *pgd, int flags); |
| 216 | 223 | |
| 224 | +void hugetlb_vma_lock_read(struct vm_area_struct *vma); | |
| 225 | +void hugetlb_vma_unlock_read(struct vm_area_struct *vma); | |
| 226 | +void hugetlb_vma_lock_write(struct vm_area_struct *vma); | |
| 227 | +void hugetlb_vma_unlock_write(struct vm_area_struct *vma); | |
| 228 | +int hugetlb_vma_trylock_write(struct vm_area_struct *vma); | |
| 229 | +void hugetlb_vma_assert_locked(struct vm_area_struct *vma); | |
| 230 | +void hugetlb_vma_lock_release(struct kref *kref); | |
| 231 | + | |
| 217 | 232 | int pmd_huge(pmd_t pmd); |
| 218 | 233 | int pud_huge(pud_t pud); |
| 219 | 234 | unsigned long hugetlb_change_protection(struct vm_area_struct *vma, |
| @@ -225,7 +240,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); · 5 unmodified lines | ||
| 225 | 240 | |
| 226 | 241 | #else /* !CONFIG_HUGETLB_PAGE */ |
| 227 | 242 | |
| 228 | −static inline void reset_vma_resv_huge_pages(struct vm_area_struct *vma) | |
| 243 | +static inline void hugetlb_dup_vma_private(struct vm_area_struct *vma) | |
| 229 | 244 | { |
| 230 | 245 | } |
| 231 | 246 | |
| @@ -336,6 +351,31 @@ static inline int prepare_hugepage_range(struct file *file, · 104 unmodified lines | ||
| 336 | 351 | return -EINVAL; |
| 337 | 352 | } |
| 338 | 353 | |
| 354 | +static inline void hugetlb_vma_lock_read(struct vm_area_struct *vma) | |
| 355 | +{ | |
| 356 | +} | |
| 357 | + | |
| 358 | +static inline void hugetlb_vma_unlock_read(struct vm_area_struct *vma) | |
| 359 | +{ | |
| 360 | +} | |
| 361 | + | |
| 362 | +static inline void hugetlb_vma_lock_write(struct vm_area_struct *vma) | |
| 363 | +{ | |
| 364 | +} | |
| 365 | + | |
| 366 | +static inline void hugetlb_vma_unlock_write(struct vm_area_struct *vma) | |
| 367 | +{ | |
| 368 | +} | |
| 369 | + | |
| 370 | +static inline int hugetlb_vma_trylock_write(struct vm_area_struct *vma) | |
| 371 | +{ | |
| 372 | + return 1; | |
| 373 | +} | |
| 374 | + | |
| 375 | +static inline void hugetlb_vma_assert_locked(struct vm_area_struct *vma) | |
| 376 | +{ | |
| 377 | +} | |
| 378 | + | |
| 339 | 379 | static inline int pmd_huge(pmd_t pmd) |
| 340 | 380 | { |
| 341 | 381 | return 0; |
| @@ -665,7 +705,7 @@ struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_nid, · 323 unmodified lines | ||
| 665 | 705 | nodemask_t *nmask, gfp_t gfp_mask); |
| 666 | 706 | struct page *alloc_huge_page_vma(struct hstate *h, struct vm_area_struct *vma, |
| 667 | 707 | unsigned long address); |
| 668 | −int huge_add_to_page_cache(struct page *page, struct address_space *mapping, | |
| 708 | +int hugetlb_add_to_page_cache(struct page *page, struct address_space *mapping, | |
| 669 | 709 | pgoff_t idx); |
| 670 | 710 | void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, |
| 671 | 711 | unsigned long address, struct page *page); |
| @@ -935,6 +975,11 @@ static inline void huge_ptep_modify_prot_commit(struct vm_area_struct *vma, · 263 unmodified lines | ||
| 935 | 975 | } |
| 936 | 976 | #endif |
| 937 | 977 | |
| 978 | +#ifdef CONFIG_NUMA | |
| 979 | +void hugetlb_register_node(struct node *node); | |
| 980 | +void hugetlb_unregister_node(struct node *node); | |
| 981 | +#endif | |
| 982 | + | |
| 938 | 983 | #else /* CONFIG_HUGETLB_PAGE */ |
| 939 | 984 | struct hstate {}; |
| 940 | 985 | |
| @@ -1109,6 +1154,14 @@ static inline void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, · 168 unmodified lines | ||
| 1109 | 1154 | pte_t *ptep, pte_t pte) |
| 1110 | 1155 | { |
| 1111 | 1156 | } |
| 1157 | + | |
| 1158 | +static inline void hugetlb_register_node(struct node *node) | |
| 1159 | +{ | |
| 1160 | +} | |
| 1161 | + | |
| 1162 | +static inline void hugetlb_unregister_node(struct node *node) | |
| 1163 | +{ | |
| 1164 | +} | |
| 1112 | 1165 | #endif /* CONFIG_HUGETLB_PAGE */ |
| 1113 | 1166 | |
| 1114 | 1167 | static inline spinlock_t *huge_pte_lock(struct hstate *h, |
| @@ -1123,14 +1176,10 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, · 8 unmodified lines | ||
| 1123 | 1176 | |
| 1124 | 1177 | #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA) |
| 1125 | 1178 | extern void __init hugetlb_cma_reserve(int order); |
| 1126 | −extern void __init hugetlb_cma_check(void); | |
| 1127 | 1179 | #else |
| 1128 | 1180 | static inline __init void hugetlb_cma_reserve(int order) |
| 1129 | 1181 | { |
| 1130 | 1182 | } |
| 1131 | −static inline __init void hugetlb_cma_check(void) | |
| 1132 | −{ | |
| 1133 | −} | |
| 1134 | 1183 | #endif |
| 1135 | 1184 | |
| 1136 | 1185 | bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr); |
| @@ -90,32 +90,31 @@ hugetlb_cgroup_from_page_rsvd(struct page *page) · 89 unmodified lines | ||
| 90 | 90 | return __hugetlb_cgroup_from_page(page, true); |
| 91 | 91 | } |
| 92 | 92 | |
| 93 | −static inline int __set_hugetlb_cgroup(struct page *page, | |
| 93 | +static inline void __set_hugetlb_cgroup(struct page *page, | |
| 94 | 94 | struct hugetlb_cgroup *h_cg, bool rsvd) |
| 95 | 95 | { |
| 96 | 96 | VM_BUG_ON_PAGE(!PageHuge(page), page); |
| 97 | 97 | |
| 98 | 98 | if (compound_order(page) < HUGETLB_CGROUP_MIN_ORDER) |
| 99 | − return -1; | |
| 99 | + return; | |
| 100 | 100 | if (rsvd) |
| 101 | 101 | set_page_private(page + SUBPAGE_INDEX_CGROUP_RSVD, |
| 102 | 102 | (unsigned long)h_cg); |
| 103 | 103 | else |
| 104 | 104 | set_page_private(page + SUBPAGE_INDEX_CGROUP, |
| 105 | 105 | (unsigned long)h_cg); |
| 106 | − return 0; | |
| 107 | 106 | } |
| 108 | 107 | |
| 109 | −static inline int set_hugetlb_cgroup(struct page *page, | |
| 108 | +static inline void set_hugetlb_cgroup(struct page *page, | |
| 110 | 109 | struct hugetlb_cgroup *h_cg) |
| 111 | 110 | { |
| 112 | − return __set_hugetlb_cgroup(page, h_cg, false); | |
| 111 | + __set_hugetlb_cgroup(page, h_cg, false); | |
| 113 | 112 | } |
| 114 | 113 | |
| 115 | −static inline int set_hugetlb_cgroup_rsvd(struct page *page, | |
| 114 | +static inline void set_hugetlb_cgroup_rsvd(struct page *page, | |
| 116 | 115 | struct hugetlb_cgroup *h_cg) |
| 117 | 116 | { |
| 118 | − return __set_hugetlb_cgroup(page, h_cg, true); | |
| 117 | + __set_hugetlb_cgroup(page, h_cg, true); | |
| 119 | 118 | } |
| 120 | 119 | |
| 121 | 120 | static inline bool hugetlb_cgroup_disabled(void) |
| @@ -199,16 +198,14 @@ hugetlb_cgroup_from_page_rsvd(struct page *page) · 77 unmodified lines | ||
| 199 | 198 | return NULL; |
| 200 | 199 | } |
| 201 | 200 | |
| 202 | −static inline int set_hugetlb_cgroup(struct page *page, | |
| 201 | +static inline void set_hugetlb_cgroup(struct page *page, | |
| 203 | 202 | struct hugetlb_cgroup *h_cg) |
| 204 | 203 | { |
| 205 | − return 0; | |
| 206 | 204 | } |
| 207 | 205 | |
| 208 | −static inline int set_hugetlb_cgroup_rsvd(struct page *page, | |
| 206 | +static inline void set_hugetlb_cgroup_rsvd(struct page *page, | |
| 209 | 207 | struct hugetlb_cgroup *h_cg) |
| 210 | 208 | { |
| 211 | − return 0; | |
| 212 | 209 | } |
| 213 | 210 | |
| 214 | 211 | static inline bool hugetlb_cgroup_disabled(void) |
| @@ -2,7 +2,7 @@ · 1 unmodified line | ||
| 2 | 2 | |
| 3 | 3 | /* |
| 4 | 4 | * This header provides generic wrappers for memory access instrumentation that |
| 5 | − * the compiler cannot emit for: KASAN, KCSAN. | |
| 5 | + * the compiler cannot emit for: KASAN, KCSAN, KMSAN. | |
| 6 | 6 | */ |
| 7 | 7 | #ifndef _LINUX_INSTRUMENTED_H |
| 8 | 8 | #define _LINUX_INSTRUMENTED_H |
| @@ -10,6 +10,7 @@ · 1 unmodified line | ||
| 10 | 10 | #include <linux/compiler.h> |
| 11 | 11 | #include <linux/kasan-checks.h> |
| 12 | 12 | #include <linux/kcsan-checks.h> |
| 13 | +#include <linux/kmsan-checks.h> | |
| 13 | 14 | #include <linux/types.h> |
| 14 | 15 | |
| 15 | 16 | /** |
| @@ -117,10 +118,11 @@ instrument_copy_to_user(void __user *to, const void *from, unsigned long n) · 101 unmodified lines | ||
| 117 | 118 | { |
| 118 | 119 | kasan_check_read(from, n); |
| 119 | 120 | kcsan_check_read(from, n); |
| 121 | + kmsan_copy_to_user(to, from, n, 0); | |
| 120 | 122 | } |
| 121 | 123 | |
| 122 | 124 | /** |
| 123 | − * instrument_copy_from_user - instrument writes of copy_from_user | |
| 125 | + * instrument_copy_from_user_before - add instrumentation before copy_from_user | |
| 124 | 126 | * |
| 125 | 127 | * Instrument writes to kernel memory, that are due to copy_from_user (and |
| 126 | 128 | * variants). The instrumentation should be inserted before the accesses. |
| @@ -130,10 +132,61 @@ instrument_copy_to_user(void __user *to, const void *from, unsigned long n) · 3 unmodified lines | ||
| 130 | 132 | * @n number of bytes to copy |
| 131 | 133 | */ |
| 132 | 134 | static __always_inline void |
| 133 | −instrument_copy_from_user(const void *to, const void __user *from, unsigned long n) | |
| 135 | +instrument_copy_from_user_before(const void *to, const void __user *from, unsigned long n) | |
| 134 | 136 | { |
| 135 | 137 | kasan_check_write(to, n); |
| 136 | 138 | kcsan_check_write(to, n); |
| 137 | 139 | } |
| 138 | 140 | |
| 141 | +/** | |
| 142 | + * instrument_copy_from_user_after - add instrumentation after copy_from_user | |
| 143 | + * | |
| 144 | + * Instrument writes to kernel memory, that are due to copy_from_user (and | |
| 145 | + * variants). The instrumentation should be inserted after the accesses. | |
| 146 | + * | |
| 147 | + * @to destination address | |
| 148 | + * @from source address | |
| 149 | + * @n number of bytes to copy | |
| 150 | + * @left number of bytes not copied (as returned by copy_from_user) | |
| 151 | + */ | |
| 152 | +static __always_inline void | |
| 153 | +instrument_copy_from_user_after(const void *to, const void __user *from, | |
| 154 | + unsigned long n, unsigned long left) | |
| 155 | +{ | |
| 156 | + kmsan_unpoison_memory(to, n - left); | |
| 157 | +} | |
| 158 | + | |
| 159 | +/** | |
| 160 | + * instrument_get_user() - add instrumentation to get_user()-like macros | |
| 161 | + * | |
| 162 | + * get_user() and friends are fragile, so it may depend on the implementation | |
| 163 | + * whether the instrumentation happens before or after the data is copied from | |
| 164 | + * the userspace. | |
| 165 | + * | |
| 166 | + * @to destination variable, may not be address-taken | |
| 167 | + */ | |
| 168 | +#define instrument_get_user(to) \ | |
| 169 | +({ \ | |
| 170 | + u64 __tmp = (u64)(to); \ | |
| 171 | + kmsan_unpoison_memory(&__tmp, sizeof(__tmp)); \ | |
| 172 | + to = __tmp; \ | |
| 173 | +}) | |
| 174 | + | |
| 175 | + | |
| 176 | +/** | |
| 177 | + * instrument_put_user() - add instrumentation to put_user()-like macros | |
| 178 | + * | |
| 179 | + * put_user() and friends are fragile, so it may depend on the implementation | |
| 180 | + * whether the instrumentation happens before or after the data is copied from | |
| 181 | + * the userspace. | |
| 182 | + * | |
| 183 | + * @from source address | |
| 184 | + * @ptr userspace pointer to copy to | |
| 185 | + * @size number of bytes to copy | |
| 186 | + */ | |
| 187 | +#define instrument_put_user(from, ptr, size) \ | |
| 188 | +({ \ | |
| 189 | + kmsan_copy_to_user(ptr, &from, sizeof(from), 0); \ | |
| 190 | +}) | |
| 191 | + | |
| 139 | 192 | #endif /* _LINUX_INSTRUMENTED_H */ |
| @@ -98,19 +98,13 @@ static inline bool kasan_has_integrated_init(void) · 97 unmodified lines | ||
| 98 | 98 | #ifdef CONFIG_KASAN |
| 99 | 99 | |
| 100 | 100 | struct kasan_cache { |
| 101 | +#ifdef CONFIG_KASAN_GENERIC | |
| 101 | 102 | int alloc_meta_offset; |
| 102 | 103 | int free_meta_offset; |
| 104 | +#endif | |
| 103 | 105 | bool is_kmalloc; |
| 104 | 106 | }; |
| 105 | 107 | |
| 106 | −slab_flags_t __kasan_never_merge(void); | |
| 107 | −static __always_inline slab_flags_t kasan_never_merge(void) | |
| 108 | −{ | |
| 109 | − if (kasan_enabled()) | |
| 110 | − return __kasan_never_merge(); | |
| 111 | − return 0; | |
| 112 | −} | |
| 113 | − | |
| 114 | 108 | void __kasan_unpoison_range(const void *addr, size_t size); |
| 115 | 109 | static __always_inline void kasan_unpoison_range(const void *addr, size_t size) |
| 116 | 110 | { |
| @@ -134,15 +128,6 @@ static __always_inline void kasan_unpoison_pages(struct page *page, · 17 unmodified lines | ||
| 134 | 128 | __kasan_unpoison_pages(page, order, init); |
| 135 | 129 | } |
| 136 | 130 | |
| 137 | −void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size, | |
| 138 | − slab_flags_t *flags); | |
| 139 | −static __always_inline void kasan_cache_create(struct kmem_cache *cache, | |
| 140 | − unsigned int *size, slab_flags_t *flags) | |
| 141 | −{ | |
| 142 | − if (kasan_enabled()) | |
| 143 | − __kasan_cache_create(cache, size, flags); | |
| 144 | −} | |
| 145 | − | |
| 146 | 131 | void __kasan_cache_create_kmalloc(struct kmem_cache *cache); |
| 147 | 132 | static __always_inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) |
| 148 | 133 | { |
| @@ -150,14 +135,6 @@ static __always_inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) · 1 unmodified line | ||
| 150 | 135 | __kasan_cache_create_kmalloc(cache); |
| 151 | 136 | } |
| 152 | 137 | |
| 153 | −size_t __kasan_metadata_size(struct kmem_cache *cache); | |
| 154 | −static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache) | |
| 155 | −{ | |
| 156 | − if (kasan_enabled()) | |
| 157 | − return __kasan_metadata_size(cache); | |
| 158 | − return 0; | |
| 159 | −} | |
| 160 | − | |
| 161 | 138 | void __kasan_poison_slab(struct slab *slab); |
| 162 | 139 | static __always_inline void kasan_poison_slab(struct slab *slab) |
| 163 | 140 | { |
| @@ -269,20 +246,12 @@ static __always_inline bool kasan_check_byte(const void *addr) · 105 unmodified lines | ||
| 269 | 246 | |
| 270 | 247 | #else /* CONFIG_KASAN */ |
| 271 | 248 | |
| 272 | −static inline slab_flags_t kasan_never_merge(void) | |
| 273 | −{ | |
| 274 | − return 0; | |
| 275 | −} | |
| 276 | 249 | static inline void kasan_unpoison_range(const void *address, size_t size) {} |
| 277 | 250 | static inline void kasan_poison_pages(struct page *page, unsigned int order, |
| 278 | 251 | bool init) {} |
| 279 | 252 | static inline void kasan_unpoison_pages(struct page *page, unsigned int order, |
| 280 | 253 | bool init) {} |
| 281 | −static inline void kasan_cache_create(struct kmem_cache *cache, | |
| 282 | − unsigned int *size, | |
| 283 | − slab_flags_t *flags) {} | |
| 284 | 254 | static inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) {} |
| 285 | −static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; } | |
| 286 | 255 | static inline void kasan_poison_slab(struct slab *slab) {} |
| 287 | 256 | static inline void kasan_unpoison_object_data(struct kmem_cache *cache, |
| 288 | 257 | void *object) {} |
| @@ -333,6 +302,11 @@ static inline void kasan_unpoison_task_stack(struct task_struct *task) {} · 44 unmodified lines | ||
| 333 | 302 | |
| 334 | 303 | #ifdef CONFIG_KASAN_GENERIC |
| 335 | 304 | |
| 305 | +size_t kasan_metadata_size(struct kmem_cache *cache); | |
| 306 | +slab_flags_t kasan_never_merge(void); | |
| 307 | +void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, | |
| 308 | + slab_flags_t *flags); | |
| 309 | + | |
| 336 | 310 | void kasan_cache_shrink(struct kmem_cache *cache); |
| 337 | 311 | void kasan_cache_shutdown(struct kmem_cache *cache); |
| 338 | 312 | void kasan_record_aux_stack(void *ptr); |
| @@ -340,6 +314,21 @@ void kasan_record_aux_stack_noalloc(void *ptr); · 1 unmodified line | ||
| 340 | 314 | |
| 341 | 315 | #else /* CONFIG_KASAN_GENERIC */ |
| 342 | 316 | |
| 317 | +/* Tag-based KASAN modes do not use per-object metadata. */ | |
| 318 | +static inline size_t kasan_metadata_size(struct kmem_cache *cache) | |
| 319 | +{ | |
| 320 | + return 0; | |
| 321 | +} | |
| 322 | +/* And thus nothing prevents cache merging. */ | |
| 323 | +static inline slab_flags_t kasan_never_merge(void) | |
| 324 | +{ | |
| 325 | + return 0; | |
| 326 | +} | |
| 327 | +/* And no cache-related metadata initialization is required. */ | |
| 328 | +static inline void kasan_cache_create(struct kmem_cache *cache, | |
| 329 | + unsigned int *size, | |
| 330 | + slab_flags_t *flags) {} | |
| 331 | + | |
| 343 | 332 | static inline void kasan_cache_shrink(struct kmem_cache *cache) {} |
| 344 | 333 | static inline void kasan_cache_shutdown(struct kmem_cache *cache) {} |
| 345 | 334 | static inline void kasan_record_aux_stack(void *ptr) {} |
| @@ -16,11 +16,13 @@ extern void khugepaged_enter_vma(struct vm_area_struct *vma, · 15 unmodified lines | ||
| 16 | 16 | unsigned long vm_flags); |
| 17 | 17 | extern void khugepaged_min_free_kbytes_update(void); |
| 18 | 18 | #ifdef CONFIG_SHMEM |
| 19 | −extern void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr); | |
| 19 | +extern int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, | |
| 20 | + bool install_pmd); | |
| 20 | 21 | #else |
| 21 | −static inline void collapse_pte_mapped_thp(struct mm_struct *mm, | |
| 22 | − unsigned long addr) | |
| 22 | +static inline int collapse_pte_mapped_thp(struct mm_struct *mm, | |
| 23 | + unsigned long addr, bool install_pmd) | |
| 23 | 24 | { |
| 25 | + return 0; | |
| 24 | 26 | } |
| 25 | 27 | #endif |
| 26 | 28 | |
| @@ -46,9 +48,10 @@ static inline void khugepaged_enter_vma(struct vm_area_struct *vma, · 19 unmodified lines | ||
| 46 | 48 | unsigned long vm_flags) |
| 47 | 49 | { |
| 48 | 50 | } |
| 49 | −static inline void collapse_pte_mapped_thp(struct mm_struct *mm, | |
| 50 | − unsigned long addr) | |
| 51 | +static inline int collapse_pte_mapped_thp(struct mm_struct *mm, | |
| 52 | + unsigned long addr, bool install_pmd) | |
| 51 | 53 | { |
| 54 | + return 0; | |
| 52 | 55 | } |
| 53 | 56 | |
| 54 | 57 | static inline void khugepaged_min_free_kbytes_update(void) |
| @@ -0,0 +1,83 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +/* | |
| 3 | + * KMSAN checks to be used for one-off annotations in subsystems. | |
| 4 | + * | |
| 5 | + * Copyright (C) 2017-2022 Google LLC | |
| 6 | + * Author: Alexander Potapenko <glider@google.com> | |
| 7 | + * | |
| 8 | + */ | |
| 9 | + | |
| 10 | +#ifndef _LINUX_KMSAN_CHECKS_H | |
| 11 | +#define _LINUX_KMSAN_CHECKS_H | |
| 12 | + | |
| 13 | +#include <linux/types.h> | |
| 14 | + | |
| 15 | +#ifdef CONFIG_KMSAN | |
| 16 | + | |
| 17 | +/** | |
| 18 | + * kmsan_poison_memory() - Mark the memory range as uninitialized. | |
| 19 | + * @address: address to start with. | |
| 20 | + * @size: size of buffer to poison. | |
| 21 | + * @flags: GFP flags for allocations done by this function. | |
| 22 | + * | |
| 23 | + * Until other data is written to this range, KMSAN will treat it as | |
| 24 | + * uninitialized. Error reports for this memory will reference the call site of | |
| 25 | + * kmsan_poison_memory() as origin. | |
| 26 | + */ | |
| 27 | +void kmsan_poison_memory(const void *address, size_t size, gfp_t flags); | |
| 28 | + | |
| 29 | +/** | |
| 30 | + * kmsan_unpoison_memory() - Mark the memory range as initialized. | |
| 31 | + * @address: address to start with. | |
| 32 | + * @size: size of buffer to unpoison. | |
| 33 | + * | |
| 34 | + * Until other data is written to this range, KMSAN will treat it as | |
| 35 | + * initialized. | |
| 36 | + */ | |
| 37 | +void kmsan_unpoison_memory(const void *address, size_t size); | |
| 38 | + | |
| 39 | +/** | |
| 40 | + * kmsan_check_memory() - Check the memory range for being initialized. | |
| 41 | + * @address: address to start with. | |
| 42 | + * @size: size of buffer to check. | |
| 43 | + * | |
| 44 | + * If any piece of the given range is marked as uninitialized, KMSAN will report | |
| 45 | + * an error. | |
| 46 | + */ | |
| 47 | +void kmsan_check_memory(const void *address, size_t size); | |
| 48 | + | |
| 49 | +/** | |
| 50 | + * kmsan_copy_to_user() - Notify KMSAN about a data transfer to userspace. | |
| 51 | + * @to: destination address in the userspace. | |
| 52 | + * @from: source address in the kernel. | |
| 53 | + * @to_copy: number of bytes to copy. | |
| 54 | + * @left: number of bytes not copied. | |
| 55 | + * | |
| 56 | + * If this is a real userspace data transfer, KMSAN checks the bytes that were | |
| 57 | + * actually copied to ensure there was no information leak. If @to belongs to | |
| 58 | + * the kernel space (which is possible for compat syscalls), KMSAN just copies | |
| 59 | + * the metadata. | |
| 60 | + */ | |
| 61 | +void kmsan_copy_to_user(void __user *to, const void *from, size_t to_copy, | |
| 62 | + size_t left); | |
| 63 | + | |
| 64 | +#else | |
| 65 | + | |
| 66 | +static inline void kmsan_poison_memory(const void *address, size_t size, | |
| 67 | + gfp_t flags) | |
| 68 | +{ | |
| 69 | +} | |
| 70 | +static inline void kmsan_unpoison_memory(const void *address, size_t size) | |
| 71 | +{ | |
| 72 | +} | |
| 73 | +static inline void kmsan_check_memory(const void *address, size_t size) | |
| 74 | +{ | |
| 75 | +} | |
| 76 | +static inline void kmsan_copy_to_user(void __user *to, const void *from, | |
| 77 | + size_t to_copy, size_t left) | |
| 78 | +{ | |
| 79 | +} | |
| 80 | + | |
| 81 | +#endif | |
| 82 | + | |
| 83 | +#endif /* _LINUX_KMSAN_CHECKS_H */ | |
| @@ -0,0 +1,330 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +/* | |
| 3 | + * KMSAN API for subsystems. | |
| 4 | + * | |
| 5 | + * Copyright (C) 2017-2022 Google LLC | |
| 6 | + * Author: Alexander Potapenko <glider@google.com> | |
| 7 | + * | |
| 8 | + */ | |
| 9 | +#ifndef _LINUX_KMSAN_H | |
| 10 | +#define _LINUX_KMSAN_H | |
| 11 | + | |
| 12 | +#include <linux/dma-direction.h> | |
| 13 | +#include <linux/gfp.h> | |
| 14 | +#include <linux/kmsan-checks.h> | |
| 15 | +#include <linux/types.h> | |
| 16 | + | |
| 17 | +struct page; | |
| 18 | +struct kmem_cache; | |
| 19 | +struct task_struct; | |
| 20 | +struct scatterlist; | |
| 21 | +struct urb; | |
| 22 | + | |
| 23 | +#ifdef CONFIG_KMSAN | |
| 24 | + | |
| 25 | +/** | |
| 26 | + * kmsan_task_create() - Initialize KMSAN state for the task. | |
| 27 | + * @task: task to initialize. | |
| 28 | + */ | |
| 29 | +void kmsan_task_create(struct task_struct *task); | |
| 30 | + | |
| 31 | +/** | |
| 32 | + * kmsan_task_exit() - Notify KMSAN that a task has exited. | |
| 33 | + * @task: task about to finish. | |
| 34 | + */ | |
| 35 | +void kmsan_task_exit(struct task_struct *task); | |
| 36 | + | |
| 37 | +/** | |
| 38 | + * kmsan_init_shadow() - Initialize KMSAN shadow at boot time. | |
| 39 | + * | |
| 40 | + * Allocate and initialize KMSAN metadata for early allocations. | |
| 41 | + */ | |
| 42 | +void __init kmsan_init_shadow(void); | |
| 43 | + | |
| 44 | +/** | |
| 45 | + * kmsan_init_runtime() - Initialize KMSAN state and enable KMSAN. | |
| 46 | + */ | |
| 47 | +void __init kmsan_init_runtime(void); | |
| 48 | + | |
| 49 | +/** | |
| 50 | + * kmsan_memblock_free_pages() - handle freeing of memblock pages. | |
| 51 | + * @page: struct page to free. | |
| 52 | + * @order: order of @page. | |
| 53 | + * | |
| 54 | + * Freed pages are either returned to buddy allocator or held back to be used | |
| 55 | + * as metadata pages. | |
| 56 | + */ | |
| 57 | +bool __init kmsan_memblock_free_pages(struct page *page, unsigned int order); | |
| 58 | + | |
| 59 | +/** | |
| 60 | + * kmsan_alloc_page() - Notify KMSAN about an alloc_pages() call. | |
| 61 | + * @page: struct page pointer returned by alloc_pages(). | |
| 62 | + * @order: order of allocated struct page. | |
| 63 | + * @flags: GFP flags used by alloc_pages() | |
| 64 | + * | |
| 65 | + * KMSAN marks 1<<@order pages starting at @page as uninitialized, unless | |
| 66 | + * @flags contain __GFP_ZERO. | |
| 67 | + */ | |
| 68 | +void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags); | |
| 69 | + | |
| 70 | +/** | |
| 71 | + * kmsan_free_page() - Notify KMSAN about a free_pages() call. | |
| 72 | + * @page: struct page pointer passed to free_pages(). | |
| 73 | + * @order: order of deallocated struct page. | |
| 74 | + * | |
| 75 | + * KMSAN marks freed memory as uninitialized. | |
| 76 | + */ | |
| 77 | +void kmsan_free_page(struct page *page, unsigned int order); | |
| 78 | + | |
| 79 | +/** | |
| 80 | + * kmsan_copy_page_meta() - Copy KMSAN metadata between two pages. | |
| 81 | + * @dst: destination page. | |
| 82 | + * @src: source page. | |
| 83 | + * | |
| 84 | + * KMSAN copies the contents of metadata pages for @src into the metadata pages | |
| 85 | + * for @dst. If @dst has no associated metadata pages, nothing happens. | |
| 86 | + * If @src has no associated metadata pages, @dst metadata pages are unpoisoned. | |
| 87 | + */ | |
| 88 | +void kmsan_copy_page_meta(struct page *dst, struct page *src); | |
| 89 | + | |
| 90 | +/** | |
| 91 | + * kmsan_slab_alloc() - Notify KMSAN about a slab allocation. | |
| 92 | + * @s: slab cache the object belongs to. | |
| 93 | + * @object: object pointer. | |
| 94 | + * @flags: GFP flags passed to the allocator. | |
| 95 | + * | |
| 96 | + * Depending on cache flags and GFP flags, KMSAN sets up the metadata of the | |
| 97 | + * newly created object, marking it as initialized or uninitialized. | |
| 98 | + */ | |
| 99 | +void kmsan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags); | |
| 100 | + | |
| 101 | +/** | |
| 102 | + * kmsan_slab_free() - Notify KMSAN about a slab deallocation. | |
| 103 | + * @s: slab cache the object belongs to. | |
| 104 | + * @object: object pointer. | |
| 105 | + * | |
| 106 | + * KMSAN marks the freed object as uninitialized. | |
| 107 | + */ | |
| 108 | +void kmsan_slab_free(struct kmem_cache *s, void *object); | |
| 109 | + | |
| 110 | +/** | |
| 111 | + * kmsan_kmalloc_large() - Notify KMSAN about a large slab allocation. | |
| 112 | + * @ptr: object pointer. | |
| 113 | + * @size: object size. | |
| 114 | + * @flags: GFP flags passed to the allocator. | |
| 115 | + * | |
| 116 | + * Similar to kmsan_slab_alloc(), but for large allocations. | |
| 117 | + */ | |
| 118 | +void kmsan_kmalloc_large(const void *ptr, size_t size, gfp_t flags); | |
| 119 | + | |
| 120 | +/** | |
| 121 | + * kmsan_kfree_large() - Notify KMSAN about a large slab deallocation. | |
| 122 | + * @ptr: object pointer. | |
| 123 | + * | |
| 124 | + * Similar to kmsan_slab_free(), but for large allocations. | |
| 125 | + */ | |
| 126 | +void kmsan_kfree_large(const void *ptr); | |
| 127 | + | |
| 128 | +/** | |
| 129 | + * kmsan_map_kernel_range_noflush() - Notify KMSAN about a vmap. | |
| 130 | + * @start: start of vmapped range. | |
| 131 | + * @end: end of vmapped range. | |
| 132 | + * @prot: page protection flags used for vmap. | |
| 133 | + * @pages: array of pages. | |
| 134 | + * @page_shift: page_shift passed to vmap_range_noflush(). | |
| 135 | + * | |
| 136 | + * KMSAN maps shadow and origin pages of @pages into contiguous ranges in | |
| 137 | + * vmalloc metadata address range. | |
| 138 | + */ | |
| 139 | +void kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end, | |
| 140 | + pgprot_t prot, struct page **pages, | |
| 141 | + unsigned int page_shift); | |
| 142 | + | |
| 143 | +/** | |
| 144 | + * kmsan_vunmap_kernel_range_noflush() - Notify KMSAN about a vunmap. | |
| 145 | + * @start: start of vunmapped range. | |
| 146 | + * @end: end of vunmapped range. | |
| 147 | + * | |
| 148 | + * KMSAN unmaps the contiguous metadata ranges created by | |
| 149 | + * kmsan_map_kernel_range_noflush(). | |
| 150 | + */ | |
| 151 | +void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end); | |
| 152 | + | |
| 153 | +/** | |
| 154 | + * kmsan_ioremap_page_range() - Notify KMSAN about a ioremap_page_range() call. | |
| 155 | + * @addr: range start. | |
| 156 | + * @end: range end. | |
| 157 | + * @phys_addr: physical range start. | |
| 158 | + * @prot: page protection flags used for ioremap_page_range(). | |
| 159 | + * @page_shift: page_shift argument passed to vmap_range_noflush(). | |
| 160 | + * | |
| 161 | + * KMSAN creates new metadata pages for the physical pages mapped into the | |
| 162 | + * virtual memory. | |
| 163 | + */ | |
| 164 | +void kmsan_ioremap_page_range(unsigned long addr, unsigned long end, | |
| 165 | + phys_addr_t phys_addr, pgprot_t prot, | |
| 166 | + unsigned int page_shift); | |
| 167 | + | |
| 168 | +/** | |
| 169 | + * kmsan_iounmap_page_range() - Notify KMSAN about a iounmap_page_range() call. | |
| 170 | + * @start: range start. | |
| 171 | + * @end: range end. | |
| 172 | + * | |
| 173 | + * KMSAN unmaps the metadata pages for the given range and, unlike for | |
| 174 | + * vunmap_page_range(), also deallocates them. | |
| 175 | + */ | |
| 176 | +void kmsan_iounmap_page_range(unsigned long start, unsigned long end); | |
| 177 | + | |
| 178 | +/** | |
| 179 | + * kmsan_handle_dma() - Handle a DMA data transfer. | |
| 180 | + * @page: first page of the buffer. | |
| 181 | + * @offset: offset of the buffer within the first page. | |
| 182 | + * @size: buffer size. | |
| 183 | + * @dir: one of possible dma_data_direction values. | |
| 184 | + * | |
| 185 | + * Depending on @direction, KMSAN: | |
| 186 | + * * checks the buffer, if it is copied to device; | |
| 187 | + * * initializes the buffer, if it is copied from device; | |
| 188 | + * * does both, if this is a DMA_BIDIRECTIONAL transfer. | |
| 189 | + */ | |
| 190 | +void kmsan_handle_dma(struct page *page, size_t offset, size_t size, | |
| 191 | + enum dma_data_direction dir); | |
| 192 | + | |
| 193 | +/** | |
| 194 | + * kmsan_handle_dma_sg() - Handle a DMA transfer using scatterlist. | |
| 195 | + * @sg: scatterlist holding DMA buffers. | |
| 196 | + * @nents: number of scatterlist entries. | |
| 197 | + * @dir: one of possible dma_data_direction values. | |
| 198 | + * | |
| 199 | + * Depending on @direction, KMSAN: | |
| 200 | + * * checks the buffers in the scatterlist, if they are copied to device; | |
| 201 | + * * initializes the buffers, if they are copied from device; | |
| 202 | + * * does both, if this is a DMA_BIDIRECTIONAL transfer. | |
| 203 | + */ | |
| 204 | +void kmsan_handle_dma_sg(struct scatterlist *sg, int nents, | |
| 205 | + enum dma_data_direction dir); | |
| 206 | + | |
| 207 | +/** | |
| 208 | + * kmsan_handle_urb() - Handle a USB data transfer. | |
| 209 | + * @urb: struct urb pointer. | |
| 210 | + * @is_out: data transfer direction (true means output to hardware). | |
| 211 | + * | |
| 212 | + * If @is_out is true, KMSAN checks the transfer buffer of @urb. Otherwise, | |
| 213 | + * KMSAN initializes the transfer buffer. | |
| 214 | + */ | |
| 215 | +void kmsan_handle_urb(const struct urb *urb, bool is_out); | |
| 216 | + | |
| 217 | +/** | |
| 218 | + * kmsan_unpoison_entry_regs() - Handle pt_regs in low-level entry code. | |
| 219 | + * @regs: struct pt_regs pointer received from assembly code. | |
| 220 | + * | |
| 221 | + * KMSAN unpoisons the contents of the passed pt_regs, preventing potential | |
| 222 | + * false positive reports. Unlike kmsan_unpoison_memory(), | |
| 223 | + * kmsan_unpoison_entry_regs() can be called from the regions where | |
| 224 | + * kmsan_in_runtime() returns true, which is the case in early entry code. | |
| 225 | + */ | |
| 226 | +void kmsan_unpoison_entry_regs(const struct pt_regs *regs); | |
| 227 | + | |
| 228 | +#else | |
| 229 | + | |
| 230 | +static inline void kmsan_init_shadow(void) | |
| 231 | +{ | |
| 232 | +} | |
| 233 | + | |
| 234 | +static inline void kmsan_init_runtime(void) | |
| 235 | +{ | |
| 236 | +} | |
| 237 | + | |
| 238 | +static inline bool kmsan_memblock_free_pages(struct page *page, | |
| 239 | + unsigned int order) | |
| 240 | +{ | |
| 241 | + return true; | |
| 242 | +} | |
| 243 | + | |
| 244 | +static inline void kmsan_task_create(struct task_struct *task) | |
| 245 | +{ | |
| 246 | +} | |
| 247 | + | |
| 248 | +static inline void kmsan_task_exit(struct task_struct *task) | |
| 249 | +{ | |
| 250 | +} | |
| 251 | + | |
| 252 | +static inline int kmsan_alloc_page(struct page *page, unsigned int order, | |
| 253 | + gfp_t flags) | |
| 254 | +{ | |
| 255 | + return 0; | |
| 256 | +} | |
| 257 | + | |
| 258 | +static inline void kmsan_free_page(struct page *page, unsigned int order) | |
| 259 | +{ | |
| 260 | +} | |
| 261 | + | |
| 262 | +static inline void kmsan_copy_page_meta(struct page *dst, struct page *src) | |
| 263 | +{ | |
| 264 | +} | |
| 265 | + | |
| 266 | +static inline void kmsan_slab_alloc(struct kmem_cache *s, void *object, | |
| 267 | + gfp_t flags) | |
| 268 | +{ | |
| 269 | +} | |
| 270 | + | |
| 271 | +static inline void kmsan_slab_free(struct kmem_cache *s, void *object) | |
| 272 | +{ | |
| 273 | +} | |
| 274 | + | |
| 275 | +static inline void kmsan_kmalloc_large(const void *ptr, size_t size, | |
| 276 | + gfp_t flags) | |
| 277 | +{ | |
| 278 | +} | |
| 279 | + | |
| 280 | +static inline void kmsan_kfree_large(const void *ptr) | |
| 281 | +{ | |
| 282 | +} | |
| 283 | + | |
| 284 | +static inline void kmsan_vmap_pages_range_noflush(unsigned long start, | |
| 285 | + unsigned long end, | |
| 286 | + pgprot_t prot, | |
| 287 | + struct page **pages, | |
| 288 | + unsigned int page_shift) | |
| 289 | +{ | |
| 290 | +} | |
| 291 | + | |
| 292 | +static inline void kmsan_vunmap_range_noflush(unsigned long start, | |
| 293 | + unsigned long end) | |
| 294 | +{ | |
| 295 | +} | |
| 296 | + | |
| 297 | +static inline void kmsan_ioremap_page_range(unsigned long start, | |
| 298 | + unsigned long end, | |
| 299 | + phys_addr_t phys_addr, | |
| 300 | + pgprot_t prot, | |
| 301 | + unsigned int page_shift) | |
| 302 | +{ | |
| 303 | +} | |
| 304 | + | |
| 305 | +static inline void kmsan_iounmap_page_range(unsigned long start, | |
| 306 | + unsigned long end) | |
| 307 | +{ | |
| 308 | +} | |
| 309 | + | |
| 310 | +static inline void kmsan_handle_dma(struct page *page, size_t offset, | |
| 311 | + size_t size, enum dma_data_direction dir) | |
| 312 | +{ | |
| 313 | +} | |
| 314 | + | |
| 315 | +static inline void kmsan_handle_dma_sg(struct scatterlist *sg, int nents, | |
| 316 | + enum dma_data_direction dir) | |
| 317 | +{ | |
| 318 | +} | |
| 319 | + | |
| 320 | +static inline void kmsan_handle_urb(const struct urb *urb, bool is_out) | |
| 321 | +{ | |
| 322 | +} | |
| 323 | + | |
| 324 | +static inline void kmsan_unpoison_entry_regs(const struct pt_regs *regs) | |
| 325 | +{ | |
| 326 | +} | |
| 327 | + | |
| 328 | +#endif | |
| 329 | + | |
| 330 | +#endif /* _LINUX_KMSAN_H */ | |
| @@ -0,0 +1,35 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +/* | |
| 3 | + * A minimal header declaring types added by KMSAN to existing kernel structs. | |
| 4 | + * | |
| 5 | + * Copyright (C) 2017-2022 Google LLC | |
| 6 | + * Author: Alexander Potapenko <glider@google.com> | |
| 7 | + * | |
| 8 | + */ | |
| 9 | +#ifndef _LINUX_KMSAN_TYPES_H | |
| 10 | +#define _LINUX_KMSAN_TYPES_H | |
| 11 | + | |
| 12 | +/* These constants are defined in the MSan LLVM instrumentation pass. */ | |
| 13 | +#define KMSAN_RETVAL_SIZE 800 | |
| 14 | +#define KMSAN_PARAM_SIZE 800 | |
| 15 | + | |
| 16 | +struct kmsan_context_state { | |
| 17 | + char param_tls[KMSAN_PARAM_SIZE]; | |
| 18 | + char retval_tls[KMSAN_RETVAL_SIZE]; | |
| 19 | + char va_arg_tls[KMSAN_PARAM_SIZE]; | |
| 20 | + char va_arg_origin_tls[KMSAN_PARAM_SIZE]; | |
| 21 | + u64 va_arg_overflow_size_tls; | |
| 22 | + char param_origin_tls[KMSAN_PARAM_SIZE]; | |
| 23 | + u32 retval_origin_tls; | |
| 24 | +}; | |
| 25 | + | |
| 26 | +#undef KMSAN_PARAM_SIZE | |
| 27 | +#undef KMSAN_RETVAL_SIZE | |
| 28 | + | |
| 29 | +struct kmsan_ctx { | |
| 30 | + struct kmsan_context_state cstate; | |
| 31 | + int kmsan_in_runtime; | |
| 32 | + bool allow_reporting; | |
| 33 | +}; | |
| 34 | + | |
| 35 | +#endif /* _LINUX_KMSAN_TYPES_H */ | |
| @@ -15,9 +15,6 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/sched.h> |
| 16 | 16 | #include <linux/sched/coredump.h> |
| 17 | 17 | |
| 18 | −struct stable_node; | |
| 19 | −struct mem_cgroup; | |
| 20 | − | |
| 21 | 18 | #ifdef CONFIG_KSM |
| 22 | 19 | int ksm_madvise(struct vm_area_struct *vma, unsigned long start, |
| 23 | 20 | unsigned long end, int advice, unsigned long *vm_flags); |
| @@ -0,0 +1,685 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0+ */ | |
| 2 | +#ifndef _LINUX_MAPLE_TREE_H | |
| 3 | +#define _LINUX_MAPLE_TREE_H | |
| 4 | +/* | |
| 5 | + * Maple Tree - An RCU-safe adaptive tree for storing ranges | |
| 6 | + * Copyright (c) 2018-2022 Oracle | |
| 7 | + * Authors: Liam R. Howlett <Liam.Howlett@Oracle.com> | |
| 8 | + * Matthew Wilcox <willy@infradead.org> | |
| 9 | + */ | |
| 10 | + | |
| 11 | +#include <linux/kernel.h> | |
| 12 | +#include <linux/rcupdate.h> | |
| 13 | +#include <linux/spinlock.h> | |
| 14 | +/* #define CONFIG_MAPLE_RCU_DISABLED */ | |
| 15 | +/* #define CONFIG_DEBUG_MAPLE_TREE_VERBOSE */ | |
| 16 | + | |
| 17 | +/* | |
| 18 | + * Allocated nodes are mutable until they have been inserted into the tree, | |
| 19 | + * at which time they cannot change their type until they have been removed | |
| 20 | + * from the tree and an RCU grace period has passed. | |
| 21 | + * | |
| 22 | + * Removed nodes have their ->parent set to point to themselves. RCU readers | |
| 23 | + * check ->parent before relying on the value that they loaded from the | |
| 24 | + * slots array. This lets us reuse the slots array for the RCU head. | |
| 25 | + * | |
| 26 | + * Nodes in the tree point to their parent unless bit 0 is set. | |
| 27 | + */ | |
| 28 | +#if defined(CONFIG_64BIT) || defined(BUILD_VDSO32_64) | |
| 29 | +/* 64bit sizes */ | |
| 30 | +#define MAPLE_NODE_SLOTS 31 /* 256 bytes including ->parent */ | |
| 31 | +#define MAPLE_RANGE64_SLOTS 16 /* 256 bytes */ | |
| 32 | +#define MAPLE_ARANGE64_SLOTS 10 /* 240 bytes */ | |
| 33 | +#define MAPLE_ARANGE64_META_MAX 15 /* Out of range for metadata */ | |
| 34 | +#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 1) | |
| 35 | +#else | |
| 36 | +/* 32bit sizes */ | |
| 37 | +#define MAPLE_NODE_SLOTS 63 /* 256 bytes including ->parent */ | |
| 38 | +#define MAPLE_RANGE64_SLOTS 32 /* 256 bytes */ | |
| 39 | +#define MAPLE_ARANGE64_SLOTS 21 /* 240 bytes */ | |
| 40 | +#define MAPLE_ARANGE64_META_MAX 31 /* Out of range for metadata */ | |
| 41 | +#define MAPLE_ALLOC_SLOTS (MAPLE_NODE_SLOTS - 2) | |
| 42 | +#endif /* defined(CONFIG_64BIT) || defined(BUILD_VDSO32_64) */ | |
| 43 | + | |
| 44 | +#define MAPLE_NODE_MASK 255UL | |
| 45 | + | |
| 46 | +/* | |
| 47 | + * The node->parent of the root node has bit 0 set and the rest of the pointer | |
| 48 | + * is a pointer to the tree itself. No more bits are available in this pointer | |
| 49 | + * (on m68k, the data structure may only be 2-byte aligned). | |
| 50 | + * | |
| 51 | + * Internal non-root nodes can only have maple_range_* nodes as parents. The | |
| 52 | + * parent pointer is 256B aligned like all other tree nodes. When storing a 32 | |
| 53 | + * or 64 bit values, the offset can fit into 4 bits. The 16 bit values need an | |
| 54 | + * extra bit to store the offset. This extra bit comes from a reuse of the last | |
| 55 | + * bit in the node type. This is possible by using bit 1 to indicate if bit 2 | |
| 56 | + * is part of the type or the slot. | |
| 57 | + * | |
| 58 | + * Once the type is decided, the decision of an allocation range type or a range | |
| 59 | + * type is done by examining the immutable tree flag for the MAPLE_ALLOC_RANGE | |
| 60 | + * flag. | |
| 61 | + * | |
| 62 | + * Node types: | |
| 63 | + * 0x??1 = Root | |
| 64 | + * 0x?00 = 16 bit nodes | |
| 65 | + * 0x010 = 32 bit nodes | |
| 66 | + * 0x110 = 64 bit nodes | |
| 67 | + * | |
| 68 | + * Slot size and location in the parent pointer: | |
| 69 | + * type : slot location | |
| 70 | + * 0x??1 : Root | |
| 71 | + * 0x?00 : 16 bit values, type in 0-1, slot in 2-6 | |
| 72 | + * 0x010 : 32 bit values, type in 0-2, slot in 3-6 | |
| 73 | + * 0x110 : 64 bit values, type in 0-2, slot in 3-6 | |
| 74 | + */ | |
| 75 | + | |
| 76 | +/* | |
| 77 | + * This metadata is used to optimize the gap updating code and in reverse | |
| 78 | + * searching for gaps or any other code that needs to find the end of the data. | |
| 79 | + */ | |
| 80 | +struct maple_metadata { | |
| 81 | + unsigned char end; | |
| 82 | + unsigned char gap; | |
| 83 | +}; | |
| 84 | + | |
| 85 | +/* | |
| 86 | + * Leaf nodes do not store pointers to nodes, they store user data. Users may | |
| 87 | + * store almost any bit pattern. As noted above, the optimisation of storing an | |
| 88 | + * entry at 0 in the root pointer cannot be done for data which have the bottom | |
| 89 | + * two bits set to '10'. We also reserve values with the bottom two bits set to | |
| 90 | + * '10' which are below 4096 (ie 2, 6, 10 .. 4094) for internal use. Some APIs | |
| 91 | + * return errnos as a negative errno shifted right by two bits and the bottom | |
| 92 | + * two bits set to '10', and while choosing to store these values in the array | |
| 93 | + * is not an error, it may lead to confusion if you're testing for an error with | |
| 94 | + * mas_is_err(). | |
| 95 | + * | |
| 96 | + * Non-leaf nodes store the type of the node pointed to (enum maple_type in bits | |
| 97 | + * 3-6), bit 2 is reserved. That leaves bits 0-1 unused for now. | |
| 98 | + * | |
| 99 | + * In regular B-Tree terms, pivots are called keys. The term pivot is used to | |
| 100 | + * indicate that the tree is specifying ranges, Pivots may appear in the | |
| 101 | + * subtree with an entry attached to the value whereas keys are unique to a | |
| 102 | + * specific position of a B-tree. Pivot values are inclusive of the slot with | |
| 103 | + * the same index. | |
| 104 | + */ | |
| 105 | + | |
| 106 | +struct maple_range_64 { | |
| 107 | + struct maple_pnode *parent; | |
| 108 | + unsigned long pivot[MAPLE_RANGE64_SLOTS - 1]; | |
| 109 | + union { | |
| 110 | + void __rcu *slot[MAPLE_RANGE64_SLOTS]; | |
| 111 | + struct { | |
| 112 | + void __rcu *pad[MAPLE_RANGE64_SLOTS - 1]; | |
| 113 | + struct maple_metadata meta; | |
| 114 | + }; | |
| 115 | + }; | |
| 116 | +}; | |
| 117 | + | |
| 118 | +/* | |
| 119 | + * At tree creation time, the user can specify that they're willing to trade off | |
| 120 | + * storing fewer entries in a tree in return for storing more information in | |
| 121 | + * each node. | |
| 122 | + * | |
| 123 | + * The maple tree supports recording the largest range of NULL entries available | |
| 124 | + * in this node, also called gaps. This optimises the tree for allocating a | |
| 125 | + * range. | |
| 126 | + */ | |
| 127 | +struct maple_arange_64 { | |
| 128 | + struct maple_pnode *parent; | |
| 129 | + unsigned long pivot[MAPLE_ARANGE64_SLOTS - 1]; | |
| 130 | + void __rcu *slot[MAPLE_ARANGE64_SLOTS]; | |
| 131 | + unsigned long gap[MAPLE_ARANGE64_SLOTS]; | |
| 132 | + struct maple_metadata meta; | |
| 133 | +}; | |
| 134 | + | |
| 135 | +struct maple_alloc { | |
| 136 | + unsigned long total; | |
| 137 | + unsigned char node_count; | |
| 138 | + unsigned int request_count; | |
| 139 | + struct maple_alloc *slot[MAPLE_ALLOC_SLOTS]; | |
| 140 | +}; | |
| 141 | + | |
| 142 | +struct maple_topiary { | |
| 143 | + struct maple_pnode *parent; | |
| 144 | + struct maple_enode *next; /* Overlaps the pivot */ | |
| 145 | +}; | |
| 146 | + | |
| 147 | +enum maple_type { | |
| 148 | + maple_dense, | |
| 149 | + maple_leaf_64, | |
| 150 | + maple_range_64, | |
| 151 | + maple_arange_64, | |
| 152 | +}; | |
| 153 | + | |
| 154 | + | |
| 155 | +/** | |
| 156 | + * DOC: Maple tree flags | |
| 157 | + * | |
| 158 | + * * MT_FLAGS_ALLOC_RANGE - Track gaps in this tree | |
| 159 | + * * MT_FLAGS_USE_RCU - Operate in RCU mode | |
| 160 | + * * MT_FLAGS_HEIGHT_OFFSET - The position of the tree height in the flags | |
| 161 | + * * MT_FLAGS_HEIGHT_MASK - The mask for the maple tree height value | |
| 162 | + * * MT_FLAGS_LOCK_MASK - How the mt_lock is used | |
| 163 | + * * MT_FLAGS_LOCK_IRQ - Acquired irq-safe | |
| 164 | + * * MT_FLAGS_LOCK_BH - Acquired bh-safe | |
| 165 | + * * MT_FLAGS_LOCK_EXTERN - mt_lock is not used | |
| 166 | + * | |
| 167 | + * MAPLE_HEIGHT_MAX The largest height that can be stored | |
| 168 | + */ | |
| 169 | +#define MT_FLAGS_ALLOC_RANGE 0x01 | |
| 170 | +#define MT_FLAGS_USE_RCU 0x02 | |
| 171 | +#define MT_FLAGS_HEIGHT_OFFSET 0x02 | |
| 172 | +#define MT_FLAGS_HEIGHT_MASK 0x7C | |
| 173 | +#define MT_FLAGS_LOCK_MASK 0x300 | |
| 174 | +#define MT_FLAGS_LOCK_IRQ 0x100 | |
| 175 | +#define MT_FLAGS_LOCK_BH 0x200 | |
| 176 | +#define MT_FLAGS_LOCK_EXTERN 0x300 | |
| 177 | + | |
| 178 | +#define MAPLE_HEIGHT_MAX 31 | |
| 179 | + | |
| 180 | + | |
| 181 | +#define MAPLE_NODE_TYPE_MASK 0x0F | |
| 182 | +#define MAPLE_NODE_TYPE_SHIFT 0x03 | |
| 183 | + | |
| 184 | +#define MAPLE_RESERVED_RANGE 4096 | |
| 185 | + | |
| 186 | +#ifdef CONFIG_LOCKDEP | |
| 187 | +typedef struct lockdep_map *lockdep_map_p; | |
| 188 | +#define mt_lock_is_held(mt) lock_is_held(mt->ma_external_lock) | |
| 189 | +#define mt_set_external_lock(mt, lock) \ | |
| 190 | + (mt)->ma_external_lock = &(lock)->dep_map | |
| 191 | +#else | |
| 192 | +typedef struct { /* nothing */ } lockdep_map_p; | |
| 193 | +#define mt_lock_is_held(mt) 1 | |
| 194 | +#define mt_set_external_lock(mt, lock) do { } while (0) | |
| 195 | +#endif | |
| 196 | + | |
| 197 | +/* | |
| 198 | + * If the tree contains a single entry at index 0, it is usually stored in | |
| 199 | + * tree->ma_root. To optimise for the page cache, an entry which ends in '00', | |
| 200 | + * '01' or '11' is stored in the root, but an entry which ends in '10' will be | |
| 201 | + * stored in a node. Bits 3-6 are used to store enum maple_type. | |
| 202 | + * | |
| 203 | + * The flags are used both to store some immutable information about this tree | |
| 204 | + * (set at tree creation time) and dynamic information set under the spinlock. | |
| 205 | + * | |
| 206 | + * Another use of flags are to indicate global states of the tree. This is the | |
| 207 | + * case with the MAPLE_USE_RCU flag, which indicates the tree is currently in | |
| 208 | + * RCU mode. This mode was added to allow the tree to reuse nodes instead of | |
| 209 | + * re-allocating and RCU freeing nodes when there is a single user. | |
| 210 | + */ | |
| 211 | +struct maple_tree { | |
| 212 | + union { | |
| 213 | + spinlock_t ma_lock; | |
| 214 | + lockdep_map_p ma_external_lock; | |
| 215 | + }; | |
| 216 | + void __rcu *ma_root; | |
| 217 | + unsigned int ma_flags; | |
| 218 | +}; | |
| 219 | + | |
| 220 | +/** | |
| 221 | + * MTREE_INIT() - Initialize a maple tree | |
| 222 | + * @name: The maple tree name | |
| 223 | + * @__flags: The maple tree flags | |
| 224 | + * | |
| 225 | + */ | |
| 226 | +#define MTREE_INIT(name, __flags) { \ | |
| 227 | + .ma_lock = __SPIN_LOCK_UNLOCKED((name).ma_lock), \ | |
| 228 | + .ma_flags = __flags, \ | |
| 229 | + .ma_root = NULL, \ | |
| 230 | +} | |
| 231 | + | |
| 232 | +/** | |
| 233 | + * MTREE_INIT_EXT() - Initialize a maple tree with an external lock. | |
| 234 | + * @name: The tree name | |
| 235 | + * @__flags: The maple tree flags | |
| 236 | + * @__lock: The external lock | |
| 237 | + */ | |
| 238 | +#ifdef CONFIG_LOCKDEP | |
| 239 | +#define MTREE_INIT_EXT(name, __flags, __lock) { \ | |
| 240 | + .ma_external_lock = &(__lock).dep_map, \ | |
| 241 | + .ma_flags = (__flags), \ | |
| 242 | + .ma_root = NULL, \ | |
| 243 | +} | |
| 244 | +#else | |
| 245 | +#define MTREE_INIT_EXT(name, __flags, __lock) MTREE_INIT(name, __flags) | |
| 246 | +#endif | |
| 247 | + | |
| 248 | +#define DEFINE_MTREE(name) \ | |
| 249 | + struct maple_tree name = MTREE_INIT(name, 0) | |
| 250 | + | |
| 251 | +#define mtree_lock(mt) spin_lock((&(mt)->ma_lock)) | |
| 252 | +#define mtree_unlock(mt) spin_unlock((&(mt)->ma_lock)) | |
| 253 | + | |
| 254 | +/* | |
| 255 | + * The Maple Tree squeezes various bits in at various points which aren't | |
| 256 | + * necessarily obvious. Usually, this is done by observing that pointers are | |
| 257 | + * N-byte aligned and thus the bottom log_2(N) bits are available for use. We | |
| 258 | + * don't use the high bits of pointers to store additional information because | |
| 259 | + * we don't know what bits are unused on any given architecture. | |
| 260 | + * | |
| 261 | + * Nodes are 256 bytes in size and are also aligned to 256 bytes, giving us 8 | |
| 262 | + * low bits for our own purposes. Nodes are currently of 4 types: | |
| 263 | + * 1. Single pointer (Range is 0-0) | |
| 264 | + * 2. Non-leaf Allocation Range nodes | |
| 265 | + * 3. Non-leaf Range nodes | |
| 266 | + * 4. Leaf Range nodes All nodes consist of a number of node slots, | |
| 267 | + * pivots, and a parent pointer. | |
| 268 | + */ | |
| 269 | + | |
| 270 | +struct maple_node { | |
| 271 | + union { | |
| 272 | + struct { | |
| 273 | + struct maple_pnode *parent; | |
| 274 | + void __rcu *slot[MAPLE_NODE_SLOTS]; | |
| 275 | + }; | |
| 276 | + struct { | |
| 277 | + void *pad; | |
| 278 | + struct rcu_head rcu; | |
| 279 | + struct maple_enode *piv_parent; | |
| 280 | + unsigned char parent_slot; | |
| 281 | + enum maple_type type; | |
| 282 | + unsigned char slot_len; | |
| 283 | + unsigned int ma_flags; | |
| 284 | + }; | |
| 285 | + struct maple_range_64 mr64; | |
| 286 | + struct maple_arange_64 ma64; | |
| 287 | + struct maple_alloc alloc; | |
| 288 | + }; | |
| 289 | +}; | |
| 290 | + | |
| 291 | +/* | |
| 292 | + * More complicated stores can cause two nodes to become one or three and | |
| 293 | + * potentially alter the height of the tree. Either half of the tree may need | |
| 294 | + * to be rebalanced against the other. The ma_topiary struct is used to track | |
| 295 | + * which nodes have been 'cut' from the tree so that the change can be done | |
| 296 | + * safely at a later date. This is done to support RCU. | |
| 297 | + */ | |
| 298 | +struct ma_topiary { | |
| 299 | + struct maple_enode *head; | |
| 300 | + struct maple_enode *tail; | |
| 301 | + struct maple_tree *mtree; | |
| 302 | +}; | |
| 303 | + | |
| 304 | +void *mtree_load(struct maple_tree *mt, unsigned long index); | |
| 305 | + | |
| 306 | +int mtree_insert(struct maple_tree *mt, unsigned long index, | |
| 307 | + void *entry, gfp_t gfp); | |
| 308 | +int mtree_insert_range(struct maple_tree *mt, unsigned long first, | |
| 309 | + unsigned long last, void *entry, gfp_t gfp); | |
| 310 | +int mtree_alloc_range(struct maple_tree *mt, unsigned long *startp, | |
| 311 | + void *entry, unsigned long size, unsigned long min, | |
| 312 | + unsigned long max, gfp_t gfp); | |
| 313 | +int mtree_alloc_rrange(struct maple_tree *mt, unsigned long *startp, | |
| 314 | + void *entry, unsigned long size, unsigned long min, | |
| 315 | + unsigned long max, gfp_t gfp); | |
| 316 | + | |
| 317 | +int mtree_store_range(struct maple_tree *mt, unsigned long first, | |
| 318 | + unsigned long last, void *entry, gfp_t gfp); | |
| 319 | +int mtree_store(struct maple_tree *mt, unsigned long index, | |
| 320 | + void *entry, gfp_t gfp); | |
| 321 | +void *mtree_erase(struct maple_tree *mt, unsigned long index); | |
| 322 | + | |
| 323 | +void mtree_destroy(struct maple_tree *mt); | |
| 324 | +void __mt_destroy(struct maple_tree *mt); | |
| 325 | + | |
| 326 | +/** | |
| 327 | + * mtree_empty() - Determine if a tree has any present entries. | |
| 328 | + * @mt: Maple Tree. | |
| 329 | + * | |
| 330 | + * Context: Any context. | |
| 331 | + * Return: %true if the tree contains only NULL pointers. | |
| 332 | + */ | |
| 333 | +static inline bool mtree_empty(const struct maple_tree *mt) | |
| 334 | +{ | |
| 335 | + return mt->ma_root == NULL; | |
| 336 | +} | |
| 337 | + | |
| 338 | +/* Advanced API */ | |
| 339 | + | |
| 340 | +/* | |
| 341 | + * The maple state is defined in the struct ma_state and is used to keep track | |
| 342 | + * of information during operations, and even between operations when using the | |
| 343 | + * advanced API. | |
| 344 | + * | |
| 345 | + * If state->node has bit 0 set then it references a tree location which is not | |
| 346 | + * a node (eg the root). If bit 1 is set, the rest of the bits are a negative | |
| 347 | + * errno. Bit 2 (the 'unallocated slots' bit) is clear. Bits 3-6 indicate the | |
| 348 | + * node type. | |
| 349 | + * | |
| 350 | + * state->alloc either has a request number of nodes or an allocated node. If | |
| 351 | + * stat->alloc has a requested number of nodes, the first bit will be set (0x1) | |
| 352 | + * and the remaining bits are the value. If state->alloc is a node, then the | |
| 353 | + * node will be of type maple_alloc. maple_alloc has MAPLE_NODE_SLOTS - 1 for | |
| 354 | + * storing more allocated nodes, a total number of nodes allocated, and the | |
| 355 | + * node_count in this node. node_count is the number of allocated nodes in this | |
| 356 | + * node. The scaling beyond MAPLE_NODE_SLOTS - 1 is handled by storing further | |
| 357 | + * nodes into state->alloc->slot[0]'s node. Nodes are taken from state->alloc | |
| 358 | + * by removing a node from the state->alloc node until state->alloc->node_count | |
| 359 | + * is 1, when state->alloc is returned and the state->alloc->slot[0] is promoted | |
| 360 | + * to state->alloc. Nodes are pushed onto state->alloc by putting the current | |
| 361 | + * state->alloc into the pushed node's slot[0]. | |
| 362 | + * | |
| 363 | + * The state also contains the implied min/max of the state->node, the depth of | |
| 364 | + * this search, and the offset. The implied min/max are either from the parent | |
| 365 | + * node or are 0-oo for the root node. The depth is incremented or decremented | |
| 366 | + * every time a node is walked down or up. The offset is the slot/pivot of | |
| 367 | + * interest in the node - either for reading or writing. | |
| 368 | + * | |
| 369 | + * When returning a value the maple state index and last respectively contain | |
| 370 | + * the start and end of the range for the entry. Ranges are inclusive in the | |
| 371 | + * Maple Tree. | |
| 372 | + */ | |
| 373 | +struct ma_state { | |
| 374 | + struct maple_tree *tree; /* The tree we're operating in */ | |
| 375 | + unsigned long index; /* The index we're operating on - range start */ | |
| 376 | + unsigned long last; /* The last index we're operating on - range end */ | |
| 377 | + struct maple_enode *node; /* The node containing this entry */ | |
| 378 | + unsigned long min; /* The minimum index of this node - implied pivot min */ | |
| 379 | + unsigned long max; /* The maximum index of this node - implied pivot max */ | |
| 380 | + struct maple_alloc *alloc; /* Allocated nodes for this operation */ | |
| 381 | + unsigned char depth; /* depth of tree descent during write */ | |
| 382 | + unsigned char offset; | |
| 383 | + unsigned char mas_flags; | |
| 384 | +}; | |
| 385 | + | |
| 386 | +struct ma_wr_state { | |
| 387 | + struct ma_state *mas; | |
| 388 | + struct maple_node *node; /* Decoded mas->node */ | |
| 389 | + unsigned long r_min; /* range min */ | |
| 390 | + unsigned long r_max; /* range max */ | |
| 391 | + enum maple_type type; /* mas->node type */ | |
| 392 | + unsigned char offset_end; /* The offset where the write ends */ | |
| 393 | + unsigned char node_end; /* mas->node end */ | |
| 394 | + unsigned long *pivots; /* mas->node->pivots pointer */ | |
| 395 | + unsigned long end_piv; /* The pivot at the offset end */ | |
| 396 | + void __rcu **slots; /* mas->node->slots pointer */ | |
| 397 | + void *entry; /* The entry to write */ | |
| 398 | + void *content; /* The existing entry that is being overwritten */ | |
| 399 | +}; | |
| 400 | + | |
| 401 | +#define mas_lock(mas) spin_lock(&((mas)->tree->ma_lock)) | |
| 402 | +#define mas_unlock(mas) spin_unlock(&((mas)->tree->ma_lock)) | |
| 403 | + | |
| 404 | + | |
| 405 | +/* | |
| 406 | + * Special values for ma_state.node. | |
| 407 | + * MAS_START means we have not searched the tree. | |
| 408 | + * MAS_ROOT means we have searched the tree and the entry we found lives in | |
| 409 | + * the root of the tree (ie it has index 0, length 1 and is the only entry in | |
| 410 | + * the tree). | |
| 411 | + * MAS_NONE means we have searched the tree and there is no node in the | |
| 412 | + * tree for this entry. For example, we searched for index 1 in an empty | |
| 413 | + * tree. Or we have a tree which points to a full leaf node and we | |
| 414 | + * searched for an entry which is larger than can be contained in that | |
| 415 | + * leaf node. | |
| 416 | + * MA_ERROR represents an errno. After dropping the lock and attempting | |
| 417 | + * to resolve the error, the walk would have to be restarted from the | |
| 418 | + * top of the tree as the tree may have been modified. | |
| 419 | + */ | |
| 420 | +#define MAS_START ((struct maple_enode *)1UL) | |
| 421 | +#define MAS_ROOT ((struct maple_enode *)5UL) | |
| 422 | +#define MAS_NONE ((struct maple_enode *)9UL) | |
| 423 | +#define MAS_PAUSE ((struct maple_enode *)17UL) | |
| 424 | +#define MA_ERROR(err) \ | |
| 425 | + ((struct maple_enode *)(((unsigned long)err << 2) | 2UL)) | |
| 426 | + | |
| 427 | +#define MA_STATE(name, mt, first, end) \ | |
| 428 | + struct ma_state name = { \ | |
| 429 | + .tree = mt, \ | |
| 430 | + .index = first, \ | |
| 431 | + .last = end, \ | |
| 432 | + .node = MAS_START, \ | |
| 433 | + .min = 0, \ | |
| 434 | + .max = ULONG_MAX, \ | |
| 435 | + .alloc = NULL, \ | |
| 436 | + } | |
| 437 | + | |
| 438 | +#define MA_WR_STATE(name, ma_state, wr_entry) \ | |
| 439 | + struct ma_wr_state name = { \ | |
| 440 | + .mas = ma_state, \ | |
| 441 | + .content = NULL, \ | |
| 442 | + .entry = wr_entry, \ | |
| 443 | + } | |
| 444 | + | |
| 445 | +#define MA_TOPIARY(name, tree) \ | |
| 446 | + struct ma_topiary name = { \ | |
| 447 | + .head = NULL, \ | |
| 448 | + .tail = NULL, \ | |
| 449 | + .mtree = tree, \ | |
| 450 | + } | |
| 451 | + | |
| 452 | +void *mas_walk(struct ma_state *mas); | |
| 453 | +void *mas_store(struct ma_state *mas, void *entry); | |
| 454 | +void *mas_erase(struct ma_state *mas); | |
| 455 | +int mas_store_gfp(struct ma_state *mas, void *entry, gfp_t gfp); | |
| 456 | +void mas_store_prealloc(struct ma_state *mas, void *entry); | |
| 457 | +void *mas_find(struct ma_state *mas, unsigned long max); | |
| 458 | +void *mas_find_rev(struct ma_state *mas, unsigned long min); | |
| 459 | +int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp); | |
| 460 | +bool mas_is_err(struct ma_state *mas); | |
| 461 | + | |
| 462 | +bool mas_nomem(struct ma_state *mas, gfp_t gfp); | |
| 463 | +void mas_pause(struct ma_state *mas); | |
| 464 | +void maple_tree_init(void); | |
| 465 | +void mas_destroy(struct ma_state *mas); | |
| 466 | +int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries); | |
| 467 | + | |
| 468 | +void *mas_prev(struct ma_state *mas, unsigned long min); | |
| 469 | +void *mas_next(struct ma_state *mas, unsigned long max); | |
| 470 | + | |
| 471 | +int mas_empty_area(struct ma_state *mas, unsigned long min, unsigned long max, | |
| 472 | + unsigned long size); | |
| 473 | + | |
| 474 | +/* Checks if a mas has not found anything */ | |
| 475 | +static inline bool mas_is_none(struct ma_state *mas) | |
| 476 | +{ | |
| 477 | + return mas->node == MAS_NONE; | |
| 478 | +} | |
| 479 | + | |
| 480 | +/* Checks if a mas has been paused */ | |
| 481 | +static inline bool mas_is_paused(struct ma_state *mas) | |
| 482 | +{ | |
| 483 | + return mas->node == MAS_PAUSE; | |
| 484 | +} | |
| 485 | + | |
| 486 | +void mas_dup_tree(struct ma_state *oldmas, struct ma_state *mas); | |
| 487 | +void mas_dup_store(struct ma_state *mas, void *entry); | |
| 488 | + | |
| 489 | +/* | |
| 490 | + * This finds an empty area from the highest address to the lowest. | |
| 491 | + * AKA "Topdown" version, | |
| 492 | + */ | |
| 493 | +int mas_empty_area_rev(struct ma_state *mas, unsigned long min, | |
| 494 | + unsigned long max, unsigned long size); | |
| 495 | +/** | |
| 496 | + * mas_reset() - Reset a Maple Tree operation state. | |
| 497 | + * @mas: Maple Tree operation state. | |
| 498 | + * | |
| 499 | + * Resets the error or walk state of the @mas so future walks of the | |
| 500 | + * array will start from the root. Use this if you have dropped the | |
| 501 | + * lock and want to reuse the ma_state. | |
| 502 | + * | |
| 503 | + * Context: Any context. | |
| 504 | + */ | |
| 505 | +static inline void mas_reset(struct ma_state *mas) | |
| 506 | +{ | |
| 507 | + mas->node = MAS_START; | |
| 508 | +} | |
| 509 | + | |
| 510 | +/** | |
| 511 | + * mas_for_each() - Iterate over a range of the maple tree. | |
| 512 | + * @__mas: Maple Tree operation state (maple_state) | |
| 513 | + * @__entry: Entry retrieved from the tree | |
| 514 | + * @__max: maximum index to retrieve from the tree | |
| 515 | + * | |
| 516 | + * When returned, mas->index and mas->last will hold the entire range for the | |
| 517 | + * entry. | |
| 518 | + * | |
| 519 | + * Note: may return the zero entry. | |
| 520 | + * | |
| 521 | + */ | |
| 522 | +#define mas_for_each(__mas, __entry, __max) \ | |
| 523 | + while (((__entry) = mas_find((__mas), (__max))) != NULL) | |
| 524 | + | |
| 525 | + | |
| 526 | +/** | |
| 527 | + * mas_set_range() - Set up Maple Tree operation state for a different index. | |
| 528 | + * @mas: Maple Tree operation state. | |
| 529 | + * @start: New start of range in the Maple Tree. | |
| 530 | + * @last: New end of range in the Maple Tree. | |
| 531 | + * | |
| 532 | + * Move the operation state to refer to a different range. This will | |
| 533 | + * have the effect of starting a walk from the top; see mas_next() | |
| 534 | + * to move to an adjacent index. | |
| 535 | + */ | |
| 536 | +static inline | |
| 537 | +void mas_set_range(struct ma_state *mas, unsigned long start, unsigned long last) | |
| 538 | +{ | |
| 539 | + mas->index = start; | |
| 540 | + mas->last = last; | |
| 541 | + mas->node = MAS_START; | |
| 542 | +} | |
| 543 | + | |
| 544 | +/** | |
| 545 | + * mas_set() - Set up Maple Tree operation state for a different index. | |
| 546 | + * @mas: Maple Tree operation state. | |
| 547 | + * @index: New index into the Maple Tree. | |
| 548 | + * | |
| 549 | + * Move the operation state to refer to a different index. This will | |
| 550 | + * have the effect of starting a walk from the top; see mas_next() | |
| 551 | + * to move to an adjacent index. | |
| 552 | + */ | |
| 553 | +static inline void mas_set(struct ma_state *mas, unsigned long index) | |
| 554 | +{ | |
| 555 | + | |
| 556 | + mas_set_range(mas, index, index); | |
| 557 | +} | |
| 558 | + | |
| 559 | +static inline bool mt_external_lock(const struct maple_tree *mt) | |
| 560 | +{ | |
| 561 | + return (mt->ma_flags & MT_FLAGS_LOCK_MASK) == MT_FLAGS_LOCK_EXTERN; | |
| 562 | +} | |
| 563 | + | |
| 564 | +/** | |
| 565 | + * mt_init_flags() - Initialise an empty maple tree with flags. | |
| 566 | + * @mt: Maple Tree | |
| 567 | + * @flags: maple tree flags. | |
| 568 | + * | |
| 569 | + * If you need to initialise a Maple Tree with special flags (eg, an | |
| 570 | + * allocation tree), use this function. | |
| 571 | + * | |
| 572 | + * Context: Any context. | |
| 573 | + */ | |
| 574 | +static inline void mt_init_flags(struct maple_tree *mt, unsigned int flags) | |
| 575 | +{ | |
| 576 | + mt->ma_flags = flags; | |
| 577 | + if (!mt_external_lock(mt)) | |
| 578 | + spin_lock_init(&mt->ma_lock); | |
| 579 | + rcu_assign_pointer(mt->ma_root, NULL); | |
| 580 | +} | |
| 581 | + | |
| 582 | +/** | |
| 583 | + * mt_init() - Initialise an empty maple tree. | |
| 584 | + * @mt: Maple Tree | |
| 585 | + * | |
| 586 | + * An empty Maple Tree. | |
| 587 | + * | |
| 588 | + * Context: Any context. | |
| 589 | + */ | |
| 590 | +static inline void mt_init(struct maple_tree *mt) | |
| 591 | +{ | |
| 592 | + mt_init_flags(mt, 0); | |
| 593 | +} | |
| 594 | + | |
| 595 | +static inline bool mt_in_rcu(struct maple_tree *mt) | |
| 596 | +{ | |
| 597 | +#ifdef CONFIG_MAPLE_RCU_DISABLED | |
| 598 | + return false; | |
| 599 | +#endif | |
| 600 | + return mt->ma_flags & MT_FLAGS_USE_RCU; | |
| 601 | +} | |
| 602 | + | |
| 603 | +/** | |
| 604 | + * mt_clear_in_rcu() - Switch the tree to non-RCU mode. | |
| 605 | + * @mt: The Maple Tree | |
| 606 | + */ | |
| 607 | +static inline void mt_clear_in_rcu(struct maple_tree *mt) | |
| 608 | +{ | |
| 609 | + if (!mt_in_rcu(mt)) | |
| 610 | + return; | |
| 611 | + | |
| 612 | + if (mt_external_lock(mt)) { | |
| 613 | + BUG_ON(!mt_lock_is_held(mt)); | |
| 614 | + mt->ma_flags &= ~MT_FLAGS_USE_RCU; | |
| 615 | + } else { | |
| 616 | + mtree_lock(mt); | |
| 617 | + mt->ma_flags &= ~MT_FLAGS_USE_RCU; | |
| 618 | + mtree_unlock(mt); | |
| 619 | + } | |
| 620 | +} | |
| 621 | + | |
| 622 | +/** | |
| 623 | + * mt_set_in_rcu() - Switch the tree to RCU safe mode. | |
| 624 | + * @mt: The Maple Tree | |
| 625 | + */ | |
| 626 | +static inline void mt_set_in_rcu(struct maple_tree *mt) | |
| 627 | +{ | |
| 628 | + if (mt_in_rcu(mt)) | |
| 629 | + return; | |
| 630 | + | |
| 631 | + if (mt_external_lock(mt)) { | |
| 632 | + BUG_ON(!mt_lock_is_held(mt)); | |
| 633 | + mt->ma_flags |= MT_FLAGS_USE_RCU; | |
| 634 | + } else { | |
| 635 | + mtree_lock(mt); | |
| 636 | + mt->ma_flags |= MT_FLAGS_USE_RCU; | |
| 637 | + mtree_unlock(mt); | |
| 638 | + } | |
| 639 | +} | |
| 640 | + | |
| 641 | +void *mt_find(struct maple_tree *mt, unsigned long *index, unsigned long max); | |
| 642 | +void *mt_find_after(struct maple_tree *mt, unsigned long *index, | |
| 643 | + unsigned long max); | |
| 644 | +void *mt_prev(struct maple_tree *mt, unsigned long index, unsigned long min); | |
| 645 | +void *mt_next(struct maple_tree *mt, unsigned long index, unsigned long max); | |
| 646 | + | |
| 647 | +/** | |
| 648 | + * mt_for_each - Iterate over each entry starting at index until max. | |
| 649 | + * @__tree: The Maple Tree | |
| 650 | + * @__entry: The current entry | |
| 651 | + * @__index: The index to update to track the location in the tree | |
| 652 | + * @__max: The maximum limit for @index | |
| 653 | + * | |
| 654 | + * Note: Will not return the zero entry. | |
| 655 | + */ | |
| 656 | +#define mt_for_each(__tree, __entry, __index, __max) \ | |
| 657 | + for (__entry = mt_find(__tree, &(__index), __max); \ | |
| 658 | + __entry; __entry = mt_find_after(__tree, &(__index), __max)) | |
| 659 | + | |
| 660 | + | |
| 661 | +#ifdef CONFIG_DEBUG_MAPLE_TREE | |
| 662 | +extern atomic_t maple_tree_tests_run; | |
| 663 | +extern atomic_t maple_tree_tests_passed; | |
| 664 | + | |
| 665 | +void mt_dump(const struct maple_tree *mt); | |
| 666 | +void mt_validate(struct maple_tree *mt); | |
| 667 | +#define MT_BUG_ON(__tree, __x) do { \ | |
| 668 | + atomic_inc(&maple_tree_tests_run); \ | |
| 669 | + if (__x) { \ | |
| 670 | + pr_info("BUG at %s:%d (%u)\n", \ | |
| 671 | + __func__, __LINE__, __x); \ | |
| 672 | + mt_dump(__tree); \ | |
| 673 | + pr_info("Pass: %u Run:%u\n", \ | |
| 674 | + atomic_read(&maple_tree_tests_passed), \ | |
| 675 | + atomic_read(&maple_tree_tests_run)); \ | |
| 676 | + dump_stack(); \ | |
| 677 | + } else { \ | |
| 678 | + atomic_inc(&maple_tree_tests_passed); \ | |
| 679 | + } \ | |
| 680 | +} while (0) | |
| 681 | +#else | |
| 682 | +#define MT_BUG_ON(__tree, __x) BUG_ON(__x) | |
| 683 | +#endif /* CONFIG_DEBUG_MAPLE_TREE */ | |
| 684 | + | |
| 685 | +#endif /*_LINUX_MAPLE_TREE_H */ | |
| @@ -80,29 +80,8 @@ enum mem_cgroup_events_target { · 79 unmodified lines | ||
| 80 | 80 | MEM_CGROUP_NTARGETS, |
| 81 | 81 | }; |
| 82 | 82 | |
| 83 | −struct memcg_vmstats_percpu { | |
| 84 | − /* Local (CPU and cgroup) page state & events */ | |
| 85 | − long state[MEMCG_NR_STAT]; | |
| 86 | − unsigned long events[NR_VM_EVENT_ITEMS]; | |
| 87 | − | |
| 88 | − /* Delta calculation for lockless upward propagation */ | |
| 89 | − long state_prev[MEMCG_NR_STAT]; | |
| 90 | − unsigned long events_prev[NR_VM_EVENT_ITEMS]; | |
| 91 | − | |
| 92 | − /* Cgroup1: threshold notifications & softlimit tree updates */ | |
| 93 | − unsigned long nr_page_events; | |
| 94 | − unsigned long targets[MEM_CGROUP_NTARGETS]; | |
| 95 | −}; | |
| 96 | − | |
| 97 | −struct memcg_vmstats { | |
| 98 | − /* Aggregated (CPU and subtree) page state & events */ | |
| 99 | − long state[MEMCG_NR_STAT]; | |
| 100 | − unsigned long events[NR_VM_EVENT_ITEMS]; | |
| 101 | − | |
| 102 | − /* Pending child counts during tree propagation */ | |
| 103 | − long state_pending[MEMCG_NR_STAT]; | |
| 104 | − unsigned long events_pending[NR_VM_EVENT_ITEMS]; | |
| 105 | −}; | |
| 83 | +struct memcg_vmstats_percpu; | |
| 84 | +struct memcg_vmstats; | |
| 106 | 85 | |
| 107 | 86 | struct mem_cgroup_reclaim_iter { |
| 108 | 87 | struct mem_cgroup *position; |
| @@ -185,15 +164,6 @@ struct mem_cgroup_thresholds { · 76 unmodified lines | ||
| 185 | 164 | struct mem_cgroup_threshold_ary *spare; |
| 186 | 165 | }; |
| 187 | 166 | |
| 188 | −#if defined(CONFIG_SMP) | |
| 189 | −struct memcg_padding { | |
| 190 | − char x[0]; | |
| 191 | −} ____cacheline_internodealigned_in_smp; | |
| 192 | −#define MEMCG_PADDING(name) struct memcg_padding name | |
| 193 | −#else | |
| 194 | −#define MEMCG_PADDING(name) | |
| 195 | −#endif | |
| 196 | − | |
| 197 | 167 | /* |
| 198 | 168 | * Remember four most recent foreign writebacks with dirty pages in this |
| 199 | 169 | * cgroup. Inode sharing is expected to be uncommon and, even if we miss |
| @@ -304,10 +274,10 @@ struct mem_cgroup { · 104 unmodified lines | ||
| 304 | 274 | spinlock_t move_lock; |
| 305 | 275 | unsigned long move_lock_flags; |
| 306 | 276 | |
| 307 | − MEMCG_PADDING(_pad1_); | |
| 277 | + CACHELINE_PADDING(_pad1_); | |
| 308 | 278 | |
| 309 | 279 | /* memory.stat */ |
| 310 | − struct memcg_vmstats vmstats; | |
| 280 | + struct memcg_vmstats *vmstats; | |
| 311 | 281 | |
| 312 | 282 | /* memory.events */ |
| 313 | 283 | atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS]; |
| @@ -326,7 +296,7 @@ struct mem_cgroup { · 12 unmodified lines | ||
| 326 | 296 | struct list_head objcg_list; |
| 327 | 297 | #endif |
| 328 | 298 | |
| 329 | − MEMCG_PADDING(_pad2_); | |
| 299 | + CACHELINE_PADDING(_pad2_); | |
| 330 | 300 | |
| 331 | 301 | /* |
| 332 | 302 | * set > 0 if pages under this cgroup are moving to other cgroup. |
| @@ -350,14 +320,20 @@ struct mem_cgroup { · 17 unmodified lines | ||
| 350 | 320 | struct deferred_split deferred_split_queue; |
| 351 | 321 | #endif |
| 352 | 322 | |
| 323 | +#ifdef CONFIG_LRU_GEN | |
| 324 | + /* per-memcg mm_struct list */ | |
| 325 | + struct lru_gen_mm_list mm_list; | |
| 326 | +#endif | |
| 327 | + | |
| 353 | 328 | struct mem_cgroup_per_node *nodeinfo[]; |
| 354 | 329 | }; |
| 355 | 330 | |
| 356 | 331 | /* |
| 357 | − * size of first charge trial. "32" comes from vmscan.c's magic value. | |
| 358 | − * TODO: maybe necessary to use big numbers in big irons. | |
| 332 | + * size of first charge trial. | |
| 333 | + * TODO: maybe necessary to use big numbers in big irons or dynamic based of the | |
| 334 | + * workload. | |
| 359 | 335 | */ |
| 360 | −#define MEMCG_CHARGE_BATCH 32U | |
| 336 | +#define MEMCG_CHARGE_BATCH 64U | |
| 361 | 337 | |
| 362 | 338 | extern struct mem_cgroup *root_mem_cgroup; |
| 363 | 339 | |
| @@ -444,6 +420,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) · 80 unmodified lines | ||
| 444 | 420 | * - LRU isolation |
| 445 | 421 | * - lock_page_memcg() |
| 446 | 422 | * - exclusive reference |
| 423 | + * - mem_cgroup_trylock_pages() | |
| 447 | 424 | * |
| 448 | 425 | * For a kmem folio a caller should hold an rcu read lock to protect memcg |
| 449 | 426 | * associated with a kmem folio from being released. |
| @@ -505,6 +482,7 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) · 55 unmodified lines | ||
| 505 | 482 | * - LRU isolation |
| 506 | 483 | * - lock_page_memcg() |
| 507 | 484 | * - exclusive reference |
| 485 | + * - mem_cgroup_trylock_pages() | |
| 508 | 486 | * |
| 509 | 487 | * For a kmem page a caller should hold an rcu read lock to protect memcg |
| 510 | 488 | * associated with a kmem page from being released. |
| @@ -689,7 +667,7 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, · 178 unmodified lines | ||
| 689 | 667 | return __mem_cgroup_charge(folio, mm, gfp); |
| 690 | 668 | } |
| 691 | 669 | |
| 692 | −int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, | |
| 670 | +int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, | |
| 693 | 671 | gfp_t gfp, swp_entry_t entry); |
| 694 | 672 | void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry); |
| 695 | 673 | |
| @@ -959,6 +937,23 @@ void unlock_page_memcg(struct page *page); · 263 unmodified lines | ||
| 959 | 937 | |
| 960 | 938 | void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val); |
| 961 | 939 | |
| 940 | +/* try to stablize folio_memcg() for all the pages in a memcg */ | |
| 941 | +static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg) | |
| 942 | +{ | |
| 943 | + rcu_read_lock(); | |
| 944 | + | |
| 945 | + if (mem_cgroup_disabled() || !atomic_read(&memcg->moving_account)) | |
| 946 | + return true; | |
| 947 | + | |
| 948 | + rcu_read_unlock(); | |
| 949 | + return false; | |
| 950 | +} | |
| 951 | + | |
| 952 | +static inline void mem_cgroup_unlock_pages(void) | |
| 953 | +{ | |
| 954 | + rcu_read_unlock(); | |
| 955 | +} | |
| 956 | + | |
| 962 | 957 | /* idx can be of type enum memcg_stat_item or node_stat_item */ |
| 963 | 958 | static inline void mod_memcg_state(struct mem_cgroup *memcg, |
| 964 | 959 | int idx, int val) |
| @@ -985,15 +980,7 @@ static inline void mod_memcg_page_state(struct page *page, · 20 unmodified lines | ||
| 985 | 980 | rcu_read_unlock(); |
| 986 | 981 | } |
| 987 | 982 | |
| 988 | −static inline unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) | |
| 989 | −{ | |
| 990 | − long x = READ_ONCE(memcg->vmstats.state[idx]); | |
| 991 | −#ifdef CONFIG_SMP | |
| 992 | − if (x < 0) | |
| 993 | − x = 0; | |
| 994 | −#endif | |
| 995 | − return x; | |
| 996 | −} | |
| 983 | +unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx); | |
| 997 | 984 | |
| 998 | 985 | static inline unsigned long lruvec_page_state(struct lruvec *lruvec, |
| 999 | 986 | enum node_stat_item idx) |
| @@ -1238,7 +1225,7 @@ static inline int mem_cgroup_charge(struct folio *folio, · 238 unmodified lines | ||
| 1238 | 1225 | return 0; |
| 1239 | 1226 | } |
| 1240 | 1227 | |
| 1241 | −static inline int mem_cgroup_swapin_charge_page(struct page *page, | |
| 1228 | +static inline int mem_cgroup_swapin_charge_folio(struct folio *folio, | |
| 1242 | 1229 | struct mm_struct *mm, gfp_t gfp, swp_entry_t entry) |
| 1243 | 1230 | { |
| 1244 | 1231 | return 0; |
| @@ -1433,6 +1420,18 @@ static inline void folio_memcg_unlock(struct folio *folio) · 188 unmodified lines | ||
| 1433 | 1420 | { |
| 1434 | 1421 | } |
| 1435 | 1422 | |
| 1423 | +static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg) | |
| 1424 | +{ | |
| 1425 | + /* to match folio_memcg_rcu() */ | |
| 1426 | + rcu_read_lock(); | |
| 1427 | + return true; | |
| 1428 | +} | |
| 1429 | + | |
| 1430 | +static inline void mem_cgroup_unlock_pages(void) | |
| 1431 | +{ | |
| 1432 | + rcu_read_unlock(); | |
| 1433 | +} | |
| 1434 | + | |
| 1436 | 1435 | static inline void mem_cgroup_handle_over_high(void) |
| 1437 | 1436 | { |
| 1438 | 1437 | } |
| @@ -1779,7 +1778,7 @@ static inline void count_objcg_event(struct obj_cgroup *objcg, · 340 unmodified lines | ||
| 1779 | 1778 | { |
| 1780 | 1779 | struct mem_cgroup *memcg; |
| 1781 | 1780 | |
| 1782 | − if (mem_cgroup_kmem_disabled()) | |
| 1781 | + if (!memcg_kmem_enabled()) | |
| 1783 | 1782 | return; |
| 1784 | 1783 | |
| 1785 | 1784 | rcu_read_lock(); |
| @@ -0,0 +1,102 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +#ifndef _LINUX_MEMORY_TIERS_H | |
| 3 | +#define _LINUX_MEMORY_TIERS_H | |
| 4 | + | |
| 5 | +#include <linux/types.h> | |
| 6 | +#include <linux/nodemask.h> | |
| 7 | +#include <linux/kref.h> | |
| 8 | +#include <linux/mmzone.h> | |
| 9 | +/* | |
| 10 | + * Each tier cover a abstrace distance chunk size of 128 | |
| 11 | + */ | |
| 12 | +#define MEMTIER_CHUNK_BITS 7 | |
| 13 | +#define MEMTIER_CHUNK_SIZE (1 << MEMTIER_CHUNK_BITS) | |
| 14 | +/* | |
| 15 | + * Smaller abstract distance values imply faster (higher) memory tiers. Offset | |
| 16 | + * the DRAM adistance so that we can accommodate devices with a slightly lower | |
| 17 | + * adistance value (slightly faster) than default DRAM adistance to be part of | |
| 18 | + * the same memory tier. | |
| 19 | + */ | |
| 20 | +#define MEMTIER_ADISTANCE_DRAM ((4 * MEMTIER_CHUNK_SIZE) + (MEMTIER_CHUNK_SIZE >> 1)) | |
| 21 | +#define MEMTIER_HOTPLUG_PRIO 100 | |
| 22 | + | |
| 23 | +struct memory_tier; | |
| 24 | +struct memory_dev_type { | |
| 25 | + /* list of memory types that are part of same tier as this type */ | |
| 26 | + struct list_head tier_sibiling; | |
| 27 | + /* abstract distance for this specific memory type */ | |
| 28 | + int adistance; | |
| 29 | + /* Nodes of same abstract distance */ | |
| 30 | + nodemask_t nodes; | |
| 31 | + struct kref kref; | |
| 32 | +}; | |
| 33 | + | |
| 34 | +#ifdef CONFIG_NUMA | |
| 35 | +extern bool numa_demotion_enabled; | |
| 36 | +struct memory_dev_type *alloc_memory_type(int adistance); | |
| 37 | +void destroy_memory_type(struct memory_dev_type *memtype); | |
| 38 | +void init_node_memory_type(int node, struct memory_dev_type *default_type); | |
| 39 | +void clear_node_memory_type(int node, struct memory_dev_type *memtype); | |
| 40 | +#ifdef CONFIG_MIGRATION | |
| 41 | +int next_demotion_node(int node); | |
| 42 | +void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets); | |
| 43 | +bool node_is_toptier(int node); | |
| 44 | +#else | |
| 45 | +static inline int next_demotion_node(int node) | |
| 46 | +{ | |
| 47 | + return NUMA_NO_NODE; | |
| 48 | +} | |
| 49 | + | |
| 50 | +static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets) | |
| 51 | +{ | |
| 52 | + *targets = NODE_MASK_NONE; | |
| 53 | +} | |
| 54 | + | |
| 55 | +static inline bool node_is_toptier(int node) | |
| 56 | +{ | |
| 57 | + return true; | |
| 58 | +} | |
| 59 | +#endif | |
| 60 | + | |
| 61 | +#else | |
| 62 | + | |
| 63 | +#define numa_demotion_enabled false | |
| 64 | +/* | |
| 65 | + * CONFIG_NUMA implementation returns non NULL error. | |
| 66 | + */ | |
| 67 | +static inline struct memory_dev_type *alloc_memory_type(int adistance) | |
| 68 | +{ | |
| 69 | + return NULL; | |
| 70 | +} | |
| 71 | + | |
| 72 | +static inline void destroy_memory_type(struct memory_dev_type *memtype) | |
| 73 | +{ | |
| 74 | + | |
| 75 | +} | |
| 76 | + | |
| 77 | +static inline void init_node_memory_type(int node, struct memory_dev_type *default_type) | |
| 78 | +{ | |
| 79 | + | |
| 80 | +} | |
| 81 | + | |
| 82 | +static inline void clear_node_memory_type(int node, struct memory_dev_type *memtype) | |
| 83 | +{ | |
| 84 | + | |
| 85 | +} | |
| 86 | + | |
| 87 | +static inline int next_demotion_node(int node) | |
| 88 | +{ | |
| 89 | + return NUMA_NO_NODE; | |
| 90 | +} | |
| 91 | + | |
| 92 | +static inline void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets) | |
| 93 | +{ | |
| 94 | + *targets = NODE_MASK_NONE; | |
| 95 | +} | |
| 96 | + | |
| 97 | +static inline bool node_is_toptier(int node) | |
| 98 | +{ | |
| 99 | + return true; | |
| 100 | +} | |
| 101 | +#endif /* CONFIG_NUMA */ | |
| 102 | +#endif /* _LINUX_MEMORY_TIERS_H */ | |
| @@ -11,7 +11,6 @@ struct page; · 10 unmodified lines | ||
| 11 | 11 | struct zone; |
| 12 | 12 | struct pglist_data; |
| 13 | 13 | struct mem_section; |
| 14 | −struct memory_block; | |
| 15 | 14 | struct memory_group; |
| 16 | 15 | struct resource; |
| 17 | 16 | struct vmem_altmap; |
| @@ -44,11 +43,6 @@ extern void arch_refresh_nodedata(int nid, pg_data_t *pgdat); · 26 unmodified lines | ||
| 44 | 43 | ({ \ |
| 45 | 44 | memblock_alloc(sizeof(*pgdat), SMP_CACHE_BYTES); \ |
| 46 | 45 | }) |
| 47 | −/* | |
| 48 | − * This definition is just for error path in node hotadd. | |
| 49 | − * For node hotremove, we have to replace this. | |
| 50 | − */ | |
| 51 | −#define generic_free_nodedata(pgdat) kfree(pgdat) | |
| 52 | 46 | |
| 53 | 47 | extern pg_data_t *node_data[]; |
| 54 | 48 | static inline void arch_refresh_nodedata(int nid, pg_data_t *pgdat) |
| @@ -64,9 +58,6 @@ static inline pg_data_t *generic_alloc_nodedata(int nid) · 9 unmodified lines | ||
| 64 | 58 | BUG(); |
| 65 | 59 | return NULL; |
| 66 | 60 | } |
| 67 | −static inline void generic_free_nodedata(pg_data_t *pgdat) | |
| 68 | −{ | |
| 69 | −} | |
| 70 | 61 | static inline void arch_refresh_nodedata(int nid, pg_data_t *pgdat) |
| 71 | 62 | { |
| 72 | 63 | } |
| @@ -216,6 +207,22 @@ void put_online_mems(void); · 143 unmodified lines | ||
| 216 | 207 | void mem_hotplug_begin(void); |
| 217 | 208 | void mem_hotplug_done(void); |
| 218 | 209 | |
| 210 | +/* See kswapd_is_running() */ | |
| 211 | +static inline void pgdat_kswapd_lock(pg_data_t *pgdat) | |
| 212 | +{ | |
| 213 | + mutex_lock(&pgdat->kswapd_lock); | |
| 214 | +} | |
| 215 | + | |
| 216 | +static inline void pgdat_kswapd_unlock(pg_data_t *pgdat) | |
| 217 | +{ | |
| 218 | + mutex_unlock(&pgdat->kswapd_lock); | |
| 219 | +} | |
| 220 | + | |
| 221 | +static inline void pgdat_kswapd_lock_init(pg_data_t *pgdat) | |
| 222 | +{ | |
| 223 | + mutex_init(&pgdat->kswapd_lock); | |
| 224 | +} | |
| 225 | + | |
| 219 | 226 | #else /* ! CONFIG_MEMORY_HOTPLUG */ |
| 220 | 227 | #define pfn_to_online_page(pfn) \ |
| 221 | 228 | ({ \ |
| @@ -252,6 +259,10 @@ static inline bool movable_node_is_enabled(void) · 30 unmodified lines | ||
| 252 | 259 | { |
| 253 | 260 | return false; |
| 254 | 261 | } |
| 262 | + | |
| 263 | +static inline void pgdat_kswapd_lock(pg_data_t *pgdat) {} | |
| 264 | +static inline void pgdat_kswapd_unlock(pg_data_t *pgdat) {} | |
| 265 | +static inline void pgdat_kswapd_lock_init(pg_data_t *pgdat) {} | |
| 255 | 266 | #endif /* ! CONFIG_MEMORY_HOTPLUG */ |
| 256 | 267 | |
| 257 | 268 | /* |
| @@ -333,7 +344,6 @@ extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn, · 75 unmodified lines | ||
| 333 | 344 | extern void remove_pfn_range_from_zone(struct zone *zone, |
| 334 | 345 | unsigned long start_pfn, |
| 335 | 346 | unsigned long nr_pages); |
| 336 | −extern bool is_memblock_offlined(struct memory_block *mem); | |
| 337 | 347 | extern int sparse_add_section(int nid, unsigned long pfn, |
| 338 | 348 | unsigned long nr_pages, struct vmem_altmap *altmap, |
| 339 | 349 | struct dev_pagemap *pgmap); |
| @@ -151,13 +151,6 @@ extern bool mempolicy_in_oom_domain(struct task_struct *tsk, · 150 unmodified lines | ||
| 151 | 151 | const nodemask_t *mask); |
| 152 | 152 | extern nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy); |
| 153 | 153 | |
| 154 | −static inline nodemask_t *policy_nodemask_current(gfp_t gfp) | |
| 155 | −{ | |
| 156 | − struct mempolicy *mpol = get_task_policy(current); | |
| 157 | − | |
| 158 | − return policy_nodemask(gfp, mpol); | |
| 159 | −} | |
| 160 | − | |
| 161 | 154 | extern unsigned int mempolicy_slab_node(void); |
| 162 | 155 | |
| 163 | 156 | extern enum zone_type policy_zone; |
| @@ -189,6 +182,7 @@ static inline bool mpol_is_preferred_many(struct mempolicy *pol) · 25 unmodified lines | ||
| 189 | 182 | return (pol->mode == MPOL_PREFERRED_MANY); |
| 190 | 183 | } |
| 191 | 184 | |
| 185 | +extern bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone); | |
| 192 | 186 | |
| 193 | 187 | #else |
| 194 | 188 | |
| @@ -294,11 +288,6 @@ static inline void mpol_put_task_policy(struct task_struct *task) · 99 unmodified lines | ||
| 294 | 288 | { |
| 295 | 289 | } |
| 296 | 290 | |
| 297 | −static inline nodemask_t *policy_nodemask_current(gfp_t gfp) | |
| 298 | −{ | |
| 299 | − return NULL; | |
| 300 | −} | |
| 301 | − | |
| 302 | 291 | static inline bool mpol_is_preferred_many(struct mempolicy *pol) |
| 303 | 292 | { |
| 304 | 293 | return false; |
| @@ -100,21 +100,6 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping, · 99 unmodified lines | ||
| 100 | 100 | |
| 101 | 101 | #endif /* CONFIG_MIGRATION */ |
| 102 | 102 | |
| 103 | −#if defined(CONFIG_MIGRATION) && defined(CONFIG_NUMA) | |
| 104 | −extern void set_migration_target_nodes(void); | |
| 105 | −extern void migrate_on_reclaim_init(void); | |
| 106 | −extern bool numa_demotion_enabled; | |
| 107 | −extern int next_demotion_node(int node); | |
| 108 | −#else | |
| 109 | −static inline void set_migration_target_nodes(void) {} | |
| 110 | −static inline void migrate_on_reclaim_init(void) {} | |
| 111 | −static inline int next_demotion_node(int node) | |
| 112 | −{ | |
| 113 | − return NUMA_NO_NODE; | |
| 114 | −} | |
| 115 | −#define numa_demotion_enabled false | |
| 116 | −#endif | |
| 117 | − | |
| 118 | 103 | #ifdef CONFIG_COMPACTION |
| 119 | 104 | bool PageMovable(struct page *page); |
| 120 | 105 | void __SetPageMovable(struct page *page, const struct movable_operations *ops); |
| @@ -661,6 +661,38 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma) · 660 unmodified lines | ||
| 661 | 661 | return vma->vm_flags & VM_ACCESS_FLAGS; |
| 662 | 662 | } |
| 663 | 663 | |
| 664 | +static inline | |
| 665 | +struct vm_area_struct *vma_find(struct vma_iterator *vmi, unsigned long max) | |
| 666 | +{ | |
| 667 | + return mas_find(&vmi->mas, max); | |
| 668 | +} | |
| 669 | + | |
| 670 | +static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) | |
| 671 | +{ | |
| 672 | + /* | |
| 673 | + * Uses vma_find() to get the first VMA when the iterator starts. | |
| 674 | + * Calling mas_next() could skip the first entry. | |
| 675 | + */ | |
| 676 | + return vma_find(vmi, ULONG_MAX); | |
| 677 | +} | |
| 678 | + | |
| 679 | +static inline struct vm_area_struct *vma_prev(struct vma_iterator *vmi) | |
| 680 | +{ | |
| 681 | + return mas_prev(&vmi->mas, 0); | |
| 682 | +} | |
| 683 | + | |
| 684 | +static inline unsigned long vma_iter_addr(struct vma_iterator *vmi) | |
| 685 | +{ | |
| 686 | + return vmi->mas.index; | |
| 687 | +} | |
| 688 | + | |
| 689 | +#define for_each_vma(__vmi, __vma) \ | |
| 690 | + while (((__vma) = vma_next(&(__vmi))) != NULL) | |
| 691 | + | |
| 692 | +/* The MM code likes to work with exclusive end addresses */ | |
| 693 | +#define for_each_vma_range(__vmi, __vma, __end) \ | |
| 694 | + while (((__vma) = vma_find(&(__vmi), (__end) - 1)) != NULL) | |
| 695 | + | |
| 664 | 696 | #ifdef CONFIG_SHMEM |
| 665 | 697 | /* |
| 666 | 698 | * The vma_is_shmem is not inline because it is used only by slow |
| @@ -697,7 +729,9 @@ static inline unsigned int compound_order(struct page *page) · 30 unmodified lines | ||
| 697 | 729 | */ |
| 698 | 730 | static inline unsigned int folio_order(struct folio *folio) |
| 699 | 731 | { |
| 700 | − return compound_order(&folio->page); | |
| 732 | + if (!folio_test_large(folio)) | |
| 733 | + return 0; | |
| 734 | + return folio->_folio_order; | |
| 701 | 735 | } |
| 702 | 736 | |
| 703 | 737 | #include <linux/huge_mm.h> |
| @@ -1255,6 +1289,18 @@ static inline int folio_nid(const struct folio *folio) · 551 unmodified lines | ||
| 1255 | 1289 | } |
| 1256 | 1290 | |
| 1257 | 1291 | #ifdef CONFIG_NUMA_BALANCING |
| 1292 | +/* page access time bits needs to hold at least 4 seconds */ | |
| 1293 | +#define PAGE_ACCESS_TIME_MIN_BITS 12 | |
| 1294 | +#if LAST_CPUPID_SHIFT < PAGE_ACCESS_TIME_MIN_BITS | |
| 1295 | +#define PAGE_ACCESS_TIME_BUCKETS \ | |
| 1296 | + (PAGE_ACCESS_TIME_MIN_BITS - LAST_CPUPID_SHIFT) | |
| 1297 | +#else | |
| 1298 | +#define PAGE_ACCESS_TIME_BUCKETS 0 | |
| 1299 | +#endif | |
| 1300 | + | |
| 1301 | +#define PAGE_ACCESS_TIME_MASK \ | |
| 1302 | + (LAST_CPUPID_MASK << PAGE_ACCESS_TIME_BUCKETS) | |
| 1303 | + | |
| 1258 | 1304 | static inline int cpu_pid_to_cpupid(int cpu, int pid) |
| 1259 | 1305 | { |
| 1260 | 1306 | return ((cpu & LAST__CPU_MASK) << LAST__PID_SHIFT) | (pid & LAST__PID_MASK); |
| @@ -1318,12 +1364,25 @@ static inline void page_cpupid_reset_last(struct page *page) · 57 unmodified lines | ||
| 1318 | 1364 | page->flags |= LAST_CPUPID_MASK << LAST_CPUPID_PGSHIFT; |
| 1319 | 1365 | } |
| 1320 | 1366 | #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */ |
| 1367 | + | |
| 1368 | +static inline int xchg_page_access_time(struct page *page, int time) | |
| 1369 | +{ | |
| 1370 | + int last_time; | |
| 1371 | + | |
| 1372 | + last_time = page_cpupid_xchg_last(page, time >> PAGE_ACCESS_TIME_BUCKETS); | |
| 1373 | + return last_time << PAGE_ACCESS_TIME_BUCKETS; | |
| 1374 | +} | |
| 1321 | 1375 | #else /* !CONFIG_NUMA_BALANCING */ |
| 1322 | 1376 | static inline int page_cpupid_xchg_last(struct page *page, int cpupid) |
| 1323 | 1377 | { |
| 1324 | 1378 | return page_to_nid(page); /* XXX */ |
| 1325 | 1379 | } |
| 1326 | 1380 | |
| 1381 | +static inline int xchg_page_access_time(struct page *page, int time) | |
| 1382 | +{ | |
| 1383 | + return 0; | |
| 1384 | +} | |
| 1385 | + | |
| 1327 | 1386 | static inline int page_cpupid_last(struct page *page) |
| 1328 | 1387 | { |
| 1329 | 1388 | return page_to_nid(page); /* XXX */ |
| @@ -1465,6 +1524,11 @@ static inline unsigned long folio_pfn(struct folio *folio) · 135 unmodified lines | ||
| 1465 | 1524 | return page_to_pfn(&folio->page); |
| 1466 | 1525 | } |
| 1467 | 1526 | |
| 1527 | +static inline struct folio *pfn_folio(unsigned long pfn) | |
| 1528 | +{ | |
| 1529 | + return page_folio(pfn_to_page(pfn)); | |
| 1530 | +} | |
| 1531 | + | |
| 1468 | 1532 | static inline atomic_t *folio_pincount_ptr(struct folio *folio) |
| 1469 | 1533 | { |
| 1470 | 1534 | return &folio_page(folio, 1)->compound_pincount; |
| @@ -1597,7 +1661,13 @@ static inline void set_page_links(struct page *page, enum zone_type zone, · 126 unmodified lines | ||
| 1597 | 1661 | */ |
| 1598 | 1662 | static inline long folio_nr_pages(struct folio *folio) |
| 1599 | 1663 | { |
| 1600 | − return compound_nr(&folio->page); | |
| 1664 | + if (!folio_test_large(folio)) | |
| 1665 | + return 1; | |
| 1666 | +#ifdef CONFIG_64BIT | |
| 1667 | + return folio->_folio_nr_pages; | |
| 1668 | +#else | |
| 1669 | + return 1L << folio->_folio_order; | |
| 1670 | +#endif | |
| 1601 | 1671 | } |
| 1602 | 1672 | |
| 1603 | 1673 | /** |
| @@ -1776,7 +1846,11 @@ extern void pagefault_out_of_memory(void); · 172 unmodified lines | ||
| 1776 | 1846 | */ |
| 1777 | 1847 | #define SHOW_MEM_FILTER_NODES (0x0001u) /* disallowed nodes */ |
| 1778 | 1848 | |
| 1779 | −extern void show_free_areas(unsigned int flags, nodemask_t *nodemask); | |
| 1849 | +extern void __show_free_areas(unsigned int flags, nodemask_t *nodemask, int max_zone_idx); | |
| 1850 | +static void __maybe_unused show_free_areas(unsigned int flags, nodemask_t *nodemask) | |
| 1851 | +{ | |
| 1852 | + __show_free_areas(flags, nodemask, MAX_NR_ZONES - 1); | |
| 1853 | +} | |
| 1780 | 1854 | |
| 1781 | 1855 | #ifdef CONFIG_MMU |
| 1782 | 1856 | extern bool can_do_mlock(void); |
| @@ -1795,8 +1869,9 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, · 12 unmodified lines | ||
| 1795 | 1869 | unsigned long size); |
| 1796 | 1870 | void zap_page_range(struct vm_area_struct *vma, unsigned long address, |
| 1797 | 1871 | unsigned long size); |
| 1798 | −void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma, | |
| 1799 | − unsigned long start, unsigned long end); | |
| 1872 | +void unmap_vmas(struct mmu_gather *tlb, struct maple_tree *mt, | |
| 1873 | + struct vm_area_struct *start_vma, unsigned long start, | |
| 1874 | + unsigned long end); | |
| 1800 | 1875 | |
| 1801 | 1876 | struct mmu_notifier_range; |
| 1802 | 1877 | |
| @@ -2495,7 +2570,6 @@ extern unsigned long absent_pages_in_range(unsigned long start_pfn, · 692 unmodified lines | ||
| 2495 | 2570 | unsigned long end_pfn); |
| 2496 | 2571 | extern void get_pfn_range_for_nid(unsigned int nid, |
| 2497 | 2572 | unsigned long *start_pfn, unsigned long *end_pfn); |
| 2498 | −extern unsigned long find_min_pfn_with_active_regions(void); | |
| 2499 | 2573 | |
| 2500 | 2574 | #ifndef CONFIG_NUMA |
| 2501 | 2575 | static inline int early_pfn_to_nid(unsigned long pfn) |
| @@ -2516,7 +2590,12 @@ extern void calculate_min_free_kbytes(void); · 14 unmodified lines | ||
| 2516 | 2590 | extern int __meminit init_per_zone_wmark_min(void); |
| 2517 | 2591 | extern void mem_init(void); |
| 2518 | 2592 | extern void __init mmap_init(void); |
| 2519 | −extern void show_mem(unsigned int flags, nodemask_t *nodemask); | |
| 2593 | + | |
| 2594 | +extern void __show_mem(unsigned int flags, nodemask_t *nodemask, int max_zone_idx); | |
| 2595 | +static inline void show_mem(unsigned int flags, nodemask_t *nodemask) | |
| 2596 | +{ | |
| 2597 | + __show_mem(flags, nodemask, MAX_NR_ZONES - 1); | |
| 2598 | +} | |
| 2520 | 2599 | extern long si_mem_available(void); |
| 2521 | 2600 | extern void si_meminfo(struct sysinfo * val); |
| 2522 | 2601 | extern void si_meminfo_node(struct sysinfo *val, int nid); |
| @@ -2593,14 +2672,15 @@ extern int __split_vma(struct mm_struct *, struct vm_area_struct *, · 70 unmodified lines | ||
| 2593 | 2672 | extern int split_vma(struct mm_struct *, struct vm_area_struct *, |
| 2594 | 2673 | unsigned long addr, int new_below); |
| 2595 | 2674 | extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *); |
| 2596 | −extern void __vma_link_rb(struct mm_struct *, struct vm_area_struct *, | |
| 2597 | − struct rb_node **, struct rb_node *); | |
| 2598 | 2675 | extern void unlink_file_vma(struct vm_area_struct *); |
| 2599 | 2676 | extern struct vm_area_struct *copy_vma(struct vm_area_struct **, |
| 2600 | 2677 | unsigned long addr, unsigned long len, pgoff_t pgoff, |
| 2601 | 2678 | bool *need_rmap_locks); |
| 2602 | 2679 | extern void exit_mmap(struct mm_struct *); |
| 2603 | 2680 | |
| 2681 | +void vma_mas_store(struct vm_area_struct *vma, struct ma_state *mas); | |
| 2682 | +void vma_mas_remove(struct vm_area_struct *vma, struct ma_state *mas); | |
| 2683 | + | |
| 2604 | 2684 | static inline int check_data_rlimit(unsigned long rlim, |
| 2605 | 2685 | unsigned long new, |
| 2606 | 2686 | unsigned long start, |
| @@ -2648,8 +2728,9 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr, · 41 unmodified lines | ||
| 2648 | 2728 | extern unsigned long do_mmap(struct file *file, unsigned long addr, |
| 2649 | 2729 | unsigned long len, unsigned long prot, unsigned long flags, |
| 2650 | 2730 | unsigned long pgoff, unsigned long *populate, struct list_head *uf); |
| 2651 | −extern int __do_munmap(struct mm_struct *, unsigned long, size_t, | |
| 2652 | − struct list_head *uf, bool downgrade); | |
| 2731 | +extern int do_mas_munmap(struct ma_state *mas, struct mm_struct *mm, | |
| 2732 | + unsigned long start, size_t len, struct list_head *uf, | |
| 2733 | + bool downgrade); | |
| 2653 | 2734 | extern int do_munmap(struct mm_struct *, unsigned long, size_t, |
| 2654 | 2735 | struct list_head *uf); |
| 2655 | 2736 | extern int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior); |
| @@ -2716,26 +2797,12 @@ extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long add · 60 unmodified lines | ||
| 2716 | 2797 | extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr, |
| 2717 | 2798 | struct vm_area_struct **pprev); |
| 2718 | 2799 | |
| 2719 | −/** | |
| 2720 | − * find_vma_intersection() - Look up the first VMA which intersects the interval | |
| 2721 | − * @mm: The process address space. | |
| 2722 | − * @start_addr: The inclusive start user address. | |
| 2723 | − * @end_addr: The exclusive end user address. | |
| 2724 | − * | |
| 2725 | − * Returns: The first VMA within the provided range, %NULL otherwise. Assumes | |
| 2726 | − * start_addr < end_addr. | |
| 2800 | +/* | |
| 2801 | + * Look up the first VMA which intersects the interval [start_addr, end_addr) | |
| 2802 | + * NULL if none. Assume start_addr < end_addr. | |
| 2727 | 2803 | */ |
| 2728 | −static inline | |
| 2729 | 2804 | struct vm_area_struct *find_vma_intersection(struct mm_struct *mm, |
| 2730 | − unsigned long start_addr, | |
| 2731 | − unsigned long end_addr) | |
| 2732 | −{ | |
| 2733 | − struct vm_area_struct *vma = find_vma(mm, start_addr); | |
| 2734 | − | |
| 2735 | − if (vma && end_addr <= vma->vm_start) | |
| 2736 | − vma = NULL; | |
| 2737 | − return vma; | |
| 2738 | −} | |
| 2805 | + unsigned long start_addr, unsigned long end_addr); | |
| 2739 | 2806 | |
| 2740 | 2807 | /** |
| 2741 | 2808 | * vma_lookup() - Find a VMA at a specific address |
| @@ -2747,12 +2814,7 @@ struct vm_area_struct *find_vma_intersection(struct mm_struct *mm, · 5 unmodified lines | ||
| 2747 | 2814 | static inline |
| 2748 | 2815 | struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr) |
| 2749 | 2816 | { |
| 2750 | − struct vm_area_struct *vma = find_vma(mm, addr); | |
| 2751 | − | |
| 2752 | − if (vma && addr < vma->vm_start) | |
| 2753 | − vma = NULL; | |
| 2754 | − | |
| 2755 | − return vma; | |
| 2817 | + return mtree_load(&mm->mm_mt, addr); | |
| 2756 | 2818 | } |
| 2757 | 2819 | |
| 2758 | 2820 | static inline unsigned long vm_start_gap(struct vm_area_struct *vma) |
| @@ -2788,7 +2850,7 @@ static inline unsigned long vma_pages(struct vm_area_struct *vma) · 29 unmodified lines | ||
| 2788 | 2850 | static inline struct vm_area_struct *find_exact_vma(struct mm_struct *mm, |
| 2789 | 2851 | unsigned long vm_start, unsigned long vm_end) |
| 2790 | 2852 | { |
| 2791 | − struct vm_area_struct *vma = find_vma(mm, vm_start); | |
| 2853 | + struct vm_area_struct *vma = vma_lookup(mm, vm_start); | |
| 2792 | 2854 | |
| 2793 | 2855 | if (vma && (vma->vm_start != vm_start || vma->vm_end != vm_end)) |
| 2794 | 2856 | vma = NULL; |
| @@ -2888,7 +2950,6 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, · 93 unmodified lines | ||
| 2888 | 2950 | * and return without waiting upon it */ |
| 2889 | 2951 | #define FOLL_NOFAULT 0x80 /* do not fault in pages */ |
| 2890 | 2952 | #define FOLL_HWPOISON 0x100 /* check page is hwpoisoned */ |
| 2891 | −#define FOLL_NUMA 0x200 /* force NUMA hinting page fault */ | |
| 2892 | 2953 | #define FOLL_MIGRATION 0x400 /* wait for page to replace migration entry */ |
| 2893 | 2954 | #define FOLL_TRIED 0x800 /* a retry, previous pass started an IO */ |
| 2894 | 2955 | #define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */ |
| @@ -2975,8 +3036,8 @@ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags) · 80 unmodified lines | ||
| 2975 | 3036 | * PageAnonExclusive() has to protect against concurrent GUP: |
| 2976 | 3037 | * * Ordinary GUP: Using the PT lock |
| 2977 | 3038 | * * GUP-fast and fork(): mm->write_protect_seq |
| 2978 | − * * GUP-fast and KSM or temporary unmapping (swap, migration): | |
| 2979 | − * clear/invalidate+flush of the page table entry | |
| 3039 | + * * GUP-fast and KSM or temporary unmapping (swap, migration): see | |
| 3040 | + * page_try_share_anon_rmap() | |
| 2980 | 3041 | * |
| 2981 | 3042 | * Must be called with the (sub)page that's actually referenced via the |
| 2982 | 3043 | * page table entry, which might not necessarily be the head page for a |
| @@ -2997,6 +3058,11 @@ static inline bool gup_must_unshare(unsigned int flags, struct page *page) · 14 unmodified lines | ||
| 2997 | 3058 | */ |
| 2998 | 3059 | if (!PageAnon(page)) |
| 2999 | 3060 | return false; |
| 3061 | + | |
| 3062 | + /* Paired with a memory barrier in page_try_share_anon_rmap(). */ | |
| 3063 | + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) | |
| 3064 | + smp_rmb(); | |
| 3065 | + | |
| 3000 | 3066 | /* |
| 3001 | 3067 | * Note that PageKsm() pages cannot be exclusive, and consequently, |
| 3002 | 3068 | * cannot get pinned. |
| @@ -3004,6 +3070,21 @@ static inline bool gup_must_unshare(unsigned int flags, struct page *page) · 1 unmodified line | ||
| 3004 | 3070 | return !PageAnonExclusive(page); |
| 3005 | 3071 | } |
| 3006 | 3072 | |
| 3073 | +/* | |
| 3074 | + * Indicates whether GUP can follow a PROT_NONE mapped page, or whether | |
| 3075 | + * a (NUMA hinting) fault is required. | |
| 3076 | + */ | |
| 3077 | +static inline bool gup_can_follow_protnone(unsigned int flags) | |
| 3078 | +{ | |
| 3079 | + /* | |
| 3080 | + * FOLL_FORCE has to be able to make progress even if the VMA is | |
| 3081 | + * inaccessible. Further, FOLL_FORCE access usually does not represent | |
| 3082 | + * application behaviour and we should avoid triggering NUMA hinting | |
| 3083 | + * faults. | |
| 3084 | + */ | |
| 3085 | + return flags & FOLL_FORCE; | |
| 3086 | +} | |
| 3087 | + | |
| 3007 | 3088 | typedef int (*pte_fn_t)(pte_t *pte, unsigned long addr, void *data); |
| 3008 | 3089 | extern int apply_to_page_range(struct mm_struct *mm, unsigned long address, |
| 3009 | 3090 | unsigned long size, pte_fn_t fn, void *data); |
| @@ -3011,7 +3092,7 @@ extern int apply_to_existing_page_range(struct mm_struct *mm, · 1 unmodified line | ||
| 3011 | 3092 | unsigned long address, unsigned long size, |
| 3012 | 3093 | pte_fn_t fn, void *data); |
| 3013 | 3094 | |
| 3014 | −extern void init_mem_debugging_and_hardening(void); | |
| 3095 | +extern void __init init_mem_debugging_and_hardening(void); | |
| 3015 | 3096 | #ifdef CONFIG_PAGE_POISONING |
| 3016 | 3097 | extern void __kernel_poison_pages(struct page *page, int numpages); |
| 3017 | 3098 | extern void __kernel_unpoison_pages(struct page *page, int numpages); |
| @@ -34,15 +34,25 @@ static inline int page_is_file_lru(struct page *page) · 33 unmodified lines | ||
| 34 | 34 | return folio_is_file_lru(page_folio(page)); |
| 35 | 35 | } |
| 36 | 36 | |
| 37 | −static __always_inline void update_lru_size(struct lruvec *lruvec, | |
| 37 | +static __always_inline void __update_lru_size(struct lruvec *lruvec, | |
| 38 | 38 | enum lru_list lru, enum zone_type zid, |
| 39 | 39 | long nr_pages) |
| 40 | 40 | { |
| 41 | 41 | struct pglist_data *pgdat = lruvec_pgdat(lruvec); |
| 42 | 42 | |
| 43 | + lockdep_assert_held(&lruvec->lru_lock); | |
| 44 | + WARN_ON_ONCE(nr_pages != (int)nr_pages); | |
| 45 | + | |
| 43 | 46 | __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages); |
| 44 | 47 | __mod_zone_page_state(&pgdat->node_zones[zid], |
| 45 | 48 | NR_ZONE_LRU_BASE + lru, nr_pages); |
| 49 | +} | |
| 50 | + | |
| 51 | +static __always_inline void update_lru_size(struct lruvec *lruvec, | |
| 52 | + enum lru_list lru, enum zone_type zid, | |
| 53 | + long nr_pages) | |
| 54 | +{ | |
| 55 | + __update_lru_size(lruvec, lru, zid, nr_pages); | |
| 46 | 56 | #ifdef CONFIG_MEMCG |
| 47 | 57 | mem_cgroup_update_lru_size(lruvec, lru, zid, nr_pages); |
| 48 | 58 | #endif |
| @@ -66,11 +76,6 @@ static __always_inline void __folio_clear_lru_flags(struct folio *folio) · 17 unmodified lines | ||
| 66 | 76 | __folio_clear_unevictable(folio); |
| 67 | 77 | } |
| 68 | 78 | |
| 69 | −static __always_inline void __clear_page_lru_flags(struct page *page) | |
| 70 | −{ | |
| 71 | − __folio_clear_lru_flags(page_folio(page)); | |
| 72 | −} | |
| 73 | − | |
| 74 | 79 | /** |
| 75 | 80 | * folio_lru_list - Which LRU list should a folio be on? |
| 76 | 81 | * @folio: The folio to test. |
| @@ -94,11 +99,224 @@ static __always_inline enum lru_list folio_lru_list(struct folio *folio) · 17 unmodified lines | ||
| 94 | 99 | return lru; |
| 95 | 100 | } |
| 96 | 101 | |
| 102 | +#ifdef CONFIG_LRU_GEN | |
| 103 | + | |
| 104 | +#ifdef CONFIG_LRU_GEN_ENABLED | |
| 105 | +static inline bool lru_gen_enabled(void) | |
| 106 | +{ | |
| 107 | + DECLARE_STATIC_KEY_TRUE(lru_gen_caps[NR_LRU_GEN_CAPS]); | |
| 108 | + | |
| 109 | + return static_branch_likely(&lru_gen_caps[LRU_GEN_CORE]); | |
| 110 | +} | |
| 111 | +#else | |
| 112 | +static inline bool lru_gen_enabled(void) | |
| 113 | +{ | |
| 114 | + DECLARE_STATIC_KEY_FALSE(lru_gen_caps[NR_LRU_GEN_CAPS]); | |
| 115 | + | |
| 116 | + return static_branch_unlikely(&lru_gen_caps[LRU_GEN_CORE]); | |
| 117 | +} | |
| 118 | +#endif | |
| 119 | + | |
| 120 | +static inline bool lru_gen_in_fault(void) | |
| 121 | +{ | |
| 122 | + return current->in_lru_fault; | |
| 123 | +} | |
| 124 | + | |
| 125 | +static inline int lru_gen_from_seq(unsigned long seq) | |
| 126 | +{ | |
| 127 | + return seq % MAX_NR_GENS; | |
| 128 | +} | |
| 129 | + | |
| 130 | +static inline int lru_hist_from_seq(unsigned long seq) | |
| 131 | +{ | |
| 132 | + return seq % NR_HIST_GENS; | |
| 133 | +} | |
| 134 | + | |
| 135 | +static inline int lru_tier_from_refs(int refs) | |
| 136 | +{ | |
| 137 | + VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH)); | |
| 138 | + | |
| 139 | + /* see the comment in folio_lru_refs() */ | |
| 140 | + return order_base_2(refs + 1); | |
| 141 | +} | |
| 142 | + | |
| 143 | +static inline int folio_lru_refs(struct folio *folio) | |
| 144 | +{ | |
| 145 | + unsigned long flags = READ_ONCE(folio->flags); | |
| 146 | + bool workingset = flags & BIT(PG_workingset); | |
| 147 | + | |
| 148 | + /* | |
| 149 | + * Return the number of accesses beyond PG_referenced, i.e., N-1 if the | |
| 150 | + * total number of accesses is N>1, since N=0,1 both map to the first | |
| 151 | + * tier. lru_tier_from_refs() will account for this off-by-one. Also see | |
| 152 | + * the comment on MAX_NR_TIERS. | |
| 153 | + */ | |
| 154 | + return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + workingset; | |
| 155 | +} | |
| 156 | + | |
| 157 | +static inline int folio_lru_gen(struct folio *folio) | |
| 158 | +{ | |
| 159 | + unsigned long flags = READ_ONCE(folio->flags); | |
| 160 | + | |
| 161 | + return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; | |
| 162 | +} | |
| 163 | + | |
| 164 | +static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) | |
| 165 | +{ | |
| 166 | + unsigned long max_seq = lruvec->lrugen.max_seq; | |
| 167 | + | |
| 168 | + VM_WARN_ON_ONCE(gen >= MAX_NR_GENS); | |
| 169 | + | |
| 170 | + /* see the comment on MIN_NR_GENS */ | |
| 171 | + return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1); | |
| 172 | +} | |
| 173 | + | |
| 174 | +static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *folio, | |
| 175 | + int old_gen, int new_gen) | |
| 176 | +{ | |
| 177 | + int type = folio_is_file_lru(folio); | |
| 178 | + int zone = folio_zonenum(folio); | |
| 179 | + int delta = folio_nr_pages(folio); | |
| 180 | + enum lru_list lru = type * LRU_INACTIVE_FILE; | |
| 181 | + struct lru_gen_struct *lrugen = &lruvec->lrugen; | |
| 182 | + | |
| 183 | + VM_WARN_ON_ONCE(old_gen != -1 && old_gen >= MAX_NR_GENS); | |
| 184 | + VM_WARN_ON_ONCE(new_gen != -1 && new_gen >= MAX_NR_GENS); | |
| 185 | + VM_WARN_ON_ONCE(old_gen == -1 && new_gen == -1); | |
| 186 | + | |
| 187 | + if (old_gen >= 0) | |
| 188 | + WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone], | |
| 189 | + lrugen->nr_pages[old_gen][type][zone] - delta); | |
| 190 | + if (new_gen >= 0) | |
| 191 | + WRITE_ONCE(lrugen->nr_pages[new_gen][type][zone], | |
| 192 | + lrugen->nr_pages[new_gen][type][zone] + delta); | |
| 193 | + | |
| 194 | + /* addition */ | |
| 195 | + if (old_gen < 0) { | |
| 196 | + if (lru_gen_is_active(lruvec, new_gen)) | |
| 197 | + lru += LRU_ACTIVE; | |
| 198 | + __update_lru_size(lruvec, lru, zone, delta); | |
| 199 | + return; | |
| 200 | + } | |
| 201 | + | |
| 202 | + /* deletion */ | |
| 203 | + if (new_gen < 0) { | |
| 204 | + if (lru_gen_is_active(lruvec, old_gen)) | |
| 205 | + lru += LRU_ACTIVE; | |
| 206 | + __update_lru_size(lruvec, lru, zone, -delta); | |
| 207 | + return; | |
| 208 | + } | |
| 209 | + | |
| 210 | + /* promotion */ | |
| 211 | + if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) { | |
| 212 | + __update_lru_size(lruvec, lru, zone, -delta); | |
| 213 | + __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta); | |
| 214 | + } | |
| 215 | + | |
| 216 | + /* demotion requires isolation, e.g., lru_deactivate_fn() */ | |
| 217 | + VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen)); | |
| 218 | +} | |
| 219 | + | |
| 220 | +static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) | |
| 221 | +{ | |
| 222 | + unsigned long seq; | |
| 223 | + unsigned long flags; | |
| 224 | + int gen = folio_lru_gen(folio); | |
| 225 | + int type = folio_is_file_lru(folio); | |
| 226 | + int zone = folio_zonenum(folio); | |
| 227 | + struct lru_gen_struct *lrugen = &lruvec->lrugen; | |
| 228 | + | |
| 229 | + VM_WARN_ON_ONCE_FOLIO(gen != -1, folio); | |
| 230 | + | |
| 231 | + if (folio_test_unevictable(folio) || !lrugen->enabled) | |
| 232 | + return false; | |
| 233 | + /* | |
| 234 | + * There are three common cases for this page: | |
| 235 | + * 1. If it's hot, e.g., freshly faulted in or previously hot and | |
| 236 | + * migrated, add it to the youngest generation. | |
| 237 | + * 2. If it's cold but can't be evicted immediately, i.e., an anon page | |
| 238 | + * not in swapcache or a dirty page pending writeback, add it to the | |
| 239 | + * second oldest generation. | |
| 240 | + * 3. Everything else (clean, cold) is added to the oldest generation. | |
| 241 | + */ | |
| 242 | + if (folio_test_active(folio)) | |
| 243 | + seq = lrugen->max_seq; | |
| 244 | + else if ((type == LRU_GEN_ANON && !folio_test_swapcache(folio)) || | |
| 245 | + (folio_test_reclaim(folio) && | |
| 246 | + (folio_test_dirty(folio) || folio_test_writeback(folio)))) | |
| 247 | + seq = lrugen->min_seq[type] + 1; | |
| 248 | + else | |
| 249 | + seq = lrugen->min_seq[type]; | |
| 250 | + | |
| 251 | + gen = lru_gen_from_seq(seq); | |
| 252 | + flags = (gen + 1UL) << LRU_GEN_PGOFF; | |
| 253 | + /* see the comment on MIN_NR_GENS about PG_active */ | |
| 254 | + set_mask_bits(&folio->flags, LRU_GEN_MASK | BIT(PG_active), flags); | |
| 255 | + | |
| 256 | + lru_gen_update_size(lruvec, folio, -1, gen); | |
| 257 | + /* for folio_rotate_reclaimable() */ | |
| 258 | + if (reclaiming) | |
| 259 | + list_add_tail(&folio->lru, &lrugen->lists[gen][type][zone]); | |
| 260 | + else | |
| 261 | + list_add(&folio->lru, &lrugen->lists[gen][type][zone]); | |
| 262 | + | |
| 263 | + return true; | |
| 264 | +} | |
| 265 | + | |
| 266 | +static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) | |
| 267 | +{ | |
| 268 | + unsigned long flags; | |
| 269 | + int gen = folio_lru_gen(folio); | |
| 270 | + | |
| 271 | + if (gen < 0) | |
| 272 | + return false; | |
| 273 | + | |
| 274 | + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); | |
| 275 | + VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); | |
| 276 | + | |
| 277 | + /* for folio_migrate_flags() */ | |
| 278 | + flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0; | |
| 279 | + flags = set_mask_bits(&folio->flags, LRU_GEN_MASK, flags); | |
| 280 | + gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; | |
| 281 | + | |
| 282 | + lru_gen_update_size(lruvec, folio, gen, -1); | |
| 283 | + list_del(&folio->lru); | |
| 284 | + | |
| 285 | + return true; | |
| 286 | +} | |
| 287 | + | |
| 288 | +#else /* !CONFIG_LRU_GEN */ | |
| 289 | + | |
| 290 | +static inline bool lru_gen_enabled(void) | |
| 291 | +{ | |
| 292 | + return false; | |
| 293 | +} | |
| 294 | + | |
| 295 | +static inline bool lru_gen_in_fault(void) | |
| 296 | +{ | |
| 297 | + return false; | |
| 298 | +} | |
| 299 | + | |
| 300 | +static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) | |
| 301 | +{ | |
| 302 | + return false; | |
| 303 | +} | |
| 304 | + | |
| 305 | +static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming) | |
| 306 | +{ | |
| 307 | + return false; | |
| 308 | +} | |
| 309 | + | |
| 310 | +#endif /* CONFIG_LRU_GEN */ | |
| 311 | + | |
| 97 | 312 | static __always_inline |
| 98 | 313 | void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) |
| 99 | 314 | { |
| 100 | 315 | enum lru_list lru = folio_lru_list(folio); |
| 101 | 316 | |
| 317 | + if (lru_gen_add_folio(lruvec, folio, false)) | |
| 318 | + return; | |
| 319 | + | |
| 102 | 320 | update_lru_size(lruvec, lru, folio_zonenum(folio), |
| 103 | 321 | folio_nr_pages(folio)); |
| 104 | 322 | if (lru != LRU_UNEVICTABLE) |
| @@ -116,23 +334,23 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) · 11 unmodified lines | ||
| 116 | 334 | { |
| 117 | 335 | enum lru_list lru = folio_lru_list(folio); |
| 118 | 336 | |
| 337 | + if (lru_gen_add_folio(lruvec, folio, true)) | |
| 338 | + return; | |
| 339 | + | |
| 119 | 340 | update_lru_size(lruvec, lru, folio_zonenum(folio), |
| 120 | 341 | folio_nr_pages(folio)); |
| 121 | 342 | /* This is not expected to be used on LRU_UNEVICTABLE */ |
| 122 | 343 | list_add_tail(&folio->lru, &lruvec->lists[lru]); |
| 123 | 344 | } |
| 124 | 345 | |
| 125 | −static __always_inline void add_page_to_lru_list_tail(struct page *page, | |
| 126 | − struct lruvec *lruvec) | |
| 127 | −{ | |
| 128 | − lruvec_add_folio_tail(lruvec, page_folio(page)); | |
| 129 | −} | |
| 130 | − | |
| 131 | 346 | static __always_inline |
| 132 | 347 | void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) |
| 133 | 348 | { |
| 134 | 349 | enum lru_list lru = folio_lru_list(folio); |
| 135 | 350 | |
| 351 | + if (lru_gen_del_folio(lruvec, folio, false)) | |
| 352 | + return; | |
| 353 | + | |
| 136 | 354 | if (lru != LRU_UNEVICTABLE) |
| 137 | 355 | list_del(&folio->lru); |
| 138 | 356 | update_lru_size(lruvec, lru, folio_zonenum(folio), |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <linux/list.h> |
| 10 | 10 | #include <linux/spinlock.h> |
| 11 | 11 | #include <linux/rbtree.h> |
| 12 | +#include <linux/maple_tree.h> | |
| 12 | 13 | #include <linux/rwsem.h> |
| 13 | 14 | #include <linux/completion.h> |
| 14 | 15 | #include <linux/cpumask.h> |
| @@ -223,6 +224,18 @@ struct page { · 208 unmodified lines | ||
| 223 | 224 | not kmapped, ie. highmem) */ |
| 224 | 225 | #endif /* WANT_PAGE_VIRTUAL */ |
| 225 | 226 | |
| 227 | +#ifdef CONFIG_KMSAN | |
| 228 | + /* | |
| 229 | + * KMSAN metadata for this page: | |
| 230 | + * - shadow page: every bit indicates whether the corresponding | |
| 231 | + * bit of the original page is initialized (0) or not (1); | |
| 232 | + * - origin page: every 4 bytes contain an id of the stack trace | |
| 233 | + * where the uninitialized value was created. | |
| 234 | + */ | |
| 235 | + struct page *kmsan_shadow; | |
| 236 | + struct page *kmsan_origin; | |
| 237 | +#endif | |
| 238 | + | |
| 226 | 239 | #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS |
| 227 | 240 | int _last_cpupid; |
| 228 | 241 | #endif |
| @@ -244,6 +257,13 @@ struct page { · 15 unmodified lines | ||
| 244 | 257 | * @_refcount: Do not access this member directly. Use folio_ref_count() |
| 245 | 258 | * to find how many references there are to this folio. |
| 246 | 259 | * @memcg_data: Memory Control Group data. |
| 260 | + * @_flags_1: For large folios, additional page flags. | |
| 261 | + * @__head: Points to the folio. Do not use. | |
| 262 | + * @_folio_dtor: Which destructor to use for this folio. | |
| 263 | + * @_folio_order: Do not use directly, call folio_order(). | |
| 264 | + * @_total_mapcount: Do not use directly, call folio_entire_mapcount(). | |
| 265 | + * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). | |
| 266 | + * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). | |
| 247 | 267 | * |
| 248 | 268 | * A folio is a physically, virtually and logically contiguous set |
| 249 | 269 | * of bytes. It is a power-of-two in size, and it is aligned to that |
| @@ -282,9 +302,17 @@ struct folio { · 32 unmodified lines | ||
| 282 | 302 | }; |
| 283 | 303 | struct page page; |
| 284 | 304 | }; |
| 305 | + unsigned long _flags_1; | |
| 306 | + unsigned long __head; | |
| 307 | + unsigned char _folio_dtor; | |
| 308 | + unsigned char _folio_order; | |
| 309 | + atomic_t _total_mapcount; | |
| 310 | + atomic_t _pincount; | |
| 311 | +#ifdef CONFIG_64BIT | |
| 312 | + unsigned int _folio_nr_pages; | |
| 313 | +#endif | |
| 285 | 314 | }; |
| 286 | 315 | |
| 287 | −static_assert(sizeof(struct page) == sizeof(struct folio)); | |
| 288 | 316 | #define FOLIO_MATCH(pg, fl) \ |
| 289 | 317 | static_assert(offsetof(struct page, pg) == offsetof(struct folio, fl)) |
| 290 | 318 | FOLIO_MATCH(flags, flags); |
| @@ -299,6 +327,19 @@ FOLIO_MATCH(_refcount, _refcount); · 8 unmodified lines | ||
| 299 | 327 | FOLIO_MATCH(memcg_data, memcg_data); |
| 300 | 328 | #endif |
| 301 | 329 | #undef FOLIO_MATCH |
| 330 | +#define FOLIO_MATCH(pg, fl) \ | |
| 331 | + static_assert(offsetof(struct folio, fl) == \ | |
| 332 | + offsetof(struct page, pg) + sizeof(struct page)) | |
| 333 | +FOLIO_MATCH(flags, _flags_1); | |
| 334 | +FOLIO_MATCH(compound_head, __head); | |
| 335 | +FOLIO_MATCH(compound_dtor, _folio_dtor); | |
| 336 | +FOLIO_MATCH(compound_order, _folio_order); | |
| 337 | +FOLIO_MATCH(compound_mapcount, _total_mapcount); | |
| 338 | +FOLIO_MATCH(compound_pincount, _pincount); | |
| 339 | +#ifdef CONFIG_64BIT | |
| 340 | +FOLIO_MATCH(compound_nr, _folio_nr_pages); | |
| 341 | +#endif | |
| 342 | +#undef FOLIO_MATCH | |
| 302 | 343 | |
| 303 | 344 | static inline atomic_t *folio_mapcount_ptr(struct folio *folio) |
| 304 | 345 | { |
| @@ -407,21 +448,6 @@ struct vm_area_struct { · 102 unmodified lines | ||
| 407 | 448 | unsigned long vm_end; /* The first byte after our end address |
| 408 | 449 | within vm_mm. */ |
| 409 | 450 | |
| 410 | − /* linked list of VM areas per task, sorted by address */ | |
| 411 | − struct vm_area_struct *vm_next, *vm_prev; | |
| 412 | − | |
| 413 | − struct rb_node vm_rb; | |
| 414 | − | |
| 415 | − /* | |
| 416 | − * Largest free memory gap in bytes to the left of this VMA. | |
| 417 | − * Either between this VMA and vma->vm_prev, or between one of the | |
| 418 | − * VMAs below us in the VMA rbtree and its ->vm_prev. This helps | |
| 419 | − * get_unmapped_area find a free area of the right size. | |
| 420 | − */ | |
| 421 | − unsigned long rb_subtree_gap; | |
| 422 | − | |
| 423 | − /* Second cache line starts here. */ | |
| 424 | − | |
| 425 | 451 | struct mm_struct *vm_mm; /* The address space we belong to. */ |
| 426 | 452 | |
| 427 | 453 | /* |
| @@ -485,9 +511,7 @@ struct vm_area_struct { · 57 unmodified lines | ||
| 485 | 511 | struct kioctx_table; |
| 486 | 512 | struct mm_struct { |
| 487 | 513 | struct { |
| 488 | − struct vm_area_struct *mmap; /* list of VMAs */ | |
| 489 | − struct rb_root mm_rb; | |
| 490 | − u64 vmacache_seqnum; /* per-thread vmacache */ | |
| 514 | + struct maple_tree mm_mt; | |
| 491 | 515 | #ifdef CONFIG_MMU |
| 492 | 516 | unsigned long (*get_unmapped_area) (struct file *filp, |
| 493 | 517 | unsigned long addr, unsigned long len, |
| @@ -501,7 +525,6 @@ struct mm_struct { · 7 unmodified lines | ||
| 501 | 525 | unsigned long mmap_compat_legacy_base; |
| 502 | 526 | #endif |
| 503 | 527 | unsigned long task_size; /* size of task vm space */ |
| 504 | − unsigned long highest_vm_end; /* highest vma end address */ | |
| 505 | 528 | pgd_t * pgd; |
| 506 | 529 | |
| 507 | 530 | #ifdef CONFIG_MEMBARRIER |
| @@ -631,22 +654,22 @@ struct mm_struct { · 123 unmodified lines | ||
| 631 | 654 | #endif |
| 632 | 655 | #ifdef CONFIG_NUMA_BALANCING |
| 633 | 656 | /* |
| 634 | − * numa_next_scan is the next time that the PTEs will be marked | |
| 635 | − * pte_numa. NUMA hinting faults will gather statistics and | |
| 636 | − * migrate pages to new nodes if necessary. | |
| 657 | + * numa_next_scan is the next time that PTEs will be remapped | |
| 658 | + * PROT_NONE to trigger NUMA hinting faults; such faults gather | |
| 659 | + * statistics and migrate pages to new nodes if necessary. | |
| 637 | 660 | */ |
| 638 | 661 | unsigned long numa_next_scan; |
| 639 | 662 | |
| 640 | − /* Restart point for scanning and setting pte_numa */ | |
| 663 | + /* Restart point for scanning and remapping PTEs. */ | |
| 641 | 664 | unsigned long numa_scan_offset; |
| 642 | 665 | |
| 643 | − /* numa_scan_seq prevents two threads setting pte_numa */ | |
| 666 | + /* numa_scan_seq prevents two threads remapping PTEs. */ | |
| 644 | 667 | int numa_scan_seq; |
| 645 | 668 | #endif |
| 646 | 669 | /* |
| 647 | 670 | * An operation with batched TLB flushing is going on. Anything |
| 648 | 671 | * that can move process memory needs to flush the TLB when |
| 649 | − * moving a PROT_NONE or PROT_NUMA mapped page. | |
| 672 | + * moving a PROT_NONE mapped page. | |
| 650 | 673 | */ |
| 651 | 674 | atomic_t tlb_flush_pending; |
| 652 | 675 | #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH |
| @@ -671,7 +694,28 @@ struct mm_struct { · 18 unmodified lines | ||
| 671 | 694 | * merging. |
| 672 | 695 | */ |
| 673 | 696 | unsigned long ksm_merging_pages; |
| 697 | + /* | |
| 698 | + * Represent how many pages are checked for ksm merging | |
| 699 | + * including merged and not merged. | |
| 700 | + */ | |
| 701 | + unsigned long ksm_rmap_items; | |
| 702 | +#endif | |
| 703 | +#ifdef CONFIG_LRU_GEN | |
| 704 | + struct { | |
| 705 | + /* this mm_struct is on lru_gen_mm_list */ | |
| 706 | + struct list_head list; | |
| 707 | + /* | |
| 708 | + * Set when switching to this mm_struct, as a hint of | |
| 709 | + * whether it has been used since the last time per-node | |
| 710 | + * page table walkers cleared the corresponding bits. | |
| 711 | + */ | |
| 712 | + unsigned long bitmap; | |
| 713 | +#ifdef CONFIG_MEMCG | |
| 714 | + /* points to the memcg of "owner" above */ | |
| 715 | + struct mem_cgroup *memcg; | |
| 674 | 716 | #endif |
| 717 | + } lru_gen; | |
| 718 | +#endif /* CONFIG_LRU_GEN */ | |
| 675 | 719 | } __randomize_layout; |
| 676 | 720 | |
| 677 | 721 | /* |
| @@ -681,6 +725,7 @@ struct mm_struct { · 3 unmodified lines | ||
| 681 | 725 | unsigned long cpu_bitmap[]; |
| 682 | 726 | }; |
| 683 | 727 | |
| 728 | +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) | |
| 684 | 729 | extern struct mm_struct init_mm; |
| 685 | 730 | |
| 686 | 731 | /* Pointer magic because the dynamic array size confuses some compilers. */ |
| @@ -698,6 +743,87 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm) · 11 unmodified lines | ||
| 698 | 743 | return (struct cpumask *)&mm->cpu_bitmap; |
| 699 | 744 | } |
| 700 | 745 | |
| 746 | +#ifdef CONFIG_LRU_GEN | |
| 747 | + | |
| 748 | +struct lru_gen_mm_list { | |
| 749 | + /* mm_struct list for page table walkers */ | |
| 750 | + struct list_head fifo; | |
| 751 | + /* protects the list above */ | |
| 752 | + spinlock_t lock; | |
| 753 | +}; | |
| 754 | + | |
| 755 | +void lru_gen_add_mm(struct mm_struct *mm); | |
| 756 | +void lru_gen_del_mm(struct mm_struct *mm); | |
| 757 | +#ifdef CONFIG_MEMCG | |
| 758 | +void lru_gen_migrate_mm(struct mm_struct *mm); | |
| 759 | +#endif | |
| 760 | + | |
| 761 | +static inline void lru_gen_init_mm(struct mm_struct *mm) | |
| 762 | +{ | |
| 763 | + INIT_LIST_HEAD(&mm->lru_gen.list); | |
| 764 | + mm->lru_gen.bitmap = 0; | |
| 765 | +#ifdef CONFIG_MEMCG | |
| 766 | + mm->lru_gen.memcg = NULL; | |
| 767 | +#endif | |
| 768 | +} | |
| 769 | + | |
| 770 | +static inline void lru_gen_use_mm(struct mm_struct *mm) | |
| 771 | +{ | |
| 772 | + /* | |
| 773 | + * When the bitmap is set, page reclaim knows this mm_struct has been | |
| 774 | + * used since the last time it cleared the bitmap. So it might be worth | |
| 775 | + * walking the page tables of this mm_struct to clear the accessed bit. | |
| 776 | + */ | |
| 777 | + WRITE_ONCE(mm->lru_gen.bitmap, -1); | |
| 778 | +} | |
| 779 | + | |
| 780 | +#else /* !CONFIG_LRU_GEN */ | |
| 781 | + | |
| 782 | +static inline void lru_gen_add_mm(struct mm_struct *mm) | |
| 783 | +{ | |
| 784 | +} | |
| 785 | + | |
| 786 | +static inline void lru_gen_del_mm(struct mm_struct *mm) | |
| 787 | +{ | |
| 788 | +} | |
| 789 | + | |
| 790 | +#ifdef CONFIG_MEMCG | |
| 791 | +static inline void lru_gen_migrate_mm(struct mm_struct *mm) | |
| 792 | +{ | |
| 793 | +} | |
| 794 | +#endif | |
| 795 | + | |
| 796 | +static inline void lru_gen_init_mm(struct mm_struct *mm) | |
| 797 | +{ | |
| 798 | +} | |
| 799 | + | |
| 800 | +static inline void lru_gen_use_mm(struct mm_struct *mm) | |
| 801 | +{ | |
| 802 | +} | |
| 803 | + | |
| 804 | +#endif /* CONFIG_LRU_GEN */ | |
| 805 | + | |
| 806 | +struct vma_iterator { | |
| 807 | + struct ma_state mas; | |
| 808 | +}; | |
| 809 | + | |
| 810 | +#define VMA_ITERATOR(name, __mm, __addr) \ | |
| 811 | + struct vma_iterator name = { \ | |
| 812 | + .mas = { \ | |
| 813 | + .tree = &(__mm)->mm_mt, \ | |
| 814 | + .index = __addr, \ | |
| 815 | + .node = MAS_START, \ | |
| 816 | + }, \ | |
| 817 | + } | |
| 818 | + | |
| 819 | +static inline void vma_iter_init(struct vma_iterator *vmi, | |
| 820 | + struct mm_struct *mm, unsigned long addr) | |
| 821 | +{ | |
| 822 | + vmi->mas.tree = &mm->mm_mt; | |
| 823 | + vmi->mas.index = addr; | |
| 824 | + vmi->mas.node = MAS_START; | |
| 825 | +} | |
| 826 | + | |
| 701 | 827 | struct mmu_gather; |
| 702 | 828 | extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm); |
| 703 | 829 | extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm); |
| @@ -24,18 +24,6 @@ · 23 unmodified lines | ||
| 24 | 24 | IS_ENABLED(CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK)) |
| 25 | 25 | #define ALLOC_SPLIT_PTLOCKS (SPINLOCK_SIZE > BITS_PER_LONG/8) |
| 26 | 26 | |
| 27 | −/* | |
| 28 | − * The per task VMA cache array: | |
| 29 | − */ | |
| 30 | −#define VMACACHE_BITS 2 | |
| 31 | −#define VMACACHE_SIZE (1U << VMACACHE_BITS) | |
| 32 | −#define VMACACHE_MASK (VMACACHE_SIZE - 1) | |
| 33 | − | |
| 34 | −struct vmacache { | |
| 35 | − u64 seqnum; | |
| 36 | − struct vm_area_struct *vmas[VMACACHE_SIZE]; | |
| 37 | −}; | |
| 38 | − | |
| 39 | 27 | /* |
| 40 | 28 | * When updating this, please also update struct resident_page_types[] in |
| 41 | 29 | * kernel/fork.c |
| @@ -24,10 +24,10 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <asm/page.h> |
| 25 | 25 | |
| 26 | 26 | /* Free memory management - zoned buddy allocator. */ |
| 27 | −#ifndef CONFIG_FORCE_MAX_ZONEORDER | |
| 27 | +#ifndef CONFIG_ARCH_FORCE_MAX_ORDER | |
| 28 | 28 | #define MAX_ORDER 11 |
| 29 | 29 | #else |
| 30 | −#define MAX_ORDER CONFIG_FORCE_MAX_ZONEORDER | |
| 30 | +#define MAX_ORDER CONFIG_ARCH_FORCE_MAX_ORDER | |
| 31 | 31 | #endif |
| 32 | 32 | #define MAX_ORDER_NR_PAGES (1 << (MAX_ORDER - 1)) |
| 33 | 33 | |
| @@ -121,20 +121,6 @@ static inline bool free_area_empty(struct free_area *area, int migratetype) · 87 unmodified lines | ||
| 121 | 121 | |
| 122 | 122 | struct pglist_data; |
| 123 | 123 | |
| 124 | −/* | |
| 125 | − * Add a wild amount of padding here to ensure data fall into separate | |
| 126 | − * cachelines. There are very few zone structures in the machine, so space | |
| 127 | − * consumption is not a concern here. | |
| 128 | − */ | |
| 129 | −#if defined(CONFIG_SMP) | |
| 130 | −struct zone_padding { | |
| 131 | − char x[0]; | |
| 132 | −} ____cacheline_internodealigned_in_smp; | |
| 133 | −#define ZONE_PADDING(name) struct zone_padding name; | |
| 134 | −#else | |
| 135 | −#define ZONE_PADDING(name) | |
| 136 | −#endif | |
| 137 | − | |
| 138 | 124 | #ifdef CONFIG_NUMA |
| 139 | 125 | enum numa_stat_item { |
| 140 | 126 | NUMA_HIT, /* allocated in intended node */ |
| @@ -222,6 +208,7 @@ enum node_stat_item { · 81 unmodified lines | ||
| 222 | 208 | #endif |
| 223 | 209 | #ifdef CONFIG_NUMA_BALANCING |
| 224 | 210 | PGPROMOTE_SUCCESS, /* promote successfully */ |
| 211 | + PGPROMOTE_CANDIDATE, /* candidate pages to promote */ | |
| 225 | 212 | #endif |
| 226 | 213 | NR_VM_NODE_STAT_ITEMS |
| 227 | 214 | }; |
| @@ -307,6 +294,8 @@ static inline bool is_active_lru(enum lru_list lru) · 79 unmodified lines | ||
| 307 | 294 | return (lru == LRU_ACTIVE_ANON || lru == LRU_ACTIVE_FILE); |
| 308 | 295 | } |
| 309 | 296 | |
| 297 | +#define WORKINGSET_ANON 0 | |
| 298 | +#define WORKINGSET_FILE 1 | |
| 310 | 299 | #define ANON_AND_FILE 2 |
| 311 | 300 | |
| 312 | 301 | enum lruvec_flags { |
| @@ -315,6 +304,207 @@ enum lruvec_flags { · 2 unmodified lines | ||
| 315 | 304 | */ |
| 316 | 305 | }; |
| 317 | 306 | |
| 307 | +#endif /* !__GENERATING_BOUNDS_H */ | |
| 308 | + | |
| 309 | +/* | |
| 310 | + * Evictable pages are divided into multiple generations. The youngest and the | |
| 311 | + * oldest generation numbers, max_seq and min_seq, are monotonically increasing. | |
| 312 | + * They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS]. An | |
| 313 | + * offset within MAX_NR_GENS, i.e., gen, indexes the LRU list of the | |
| 314 | + * corresponding generation. The gen counter in folio->flags stores gen+1 while | |
| 315 | + * a page is on one of lrugen->lists[]. Otherwise it stores 0. | |
| 316 | + * | |
| 317 | + * A page is added to the youngest generation on faulting. The aging needs to | |
| 318 | + * check the accessed bit at least twice before handing this page over to the | |
| 319 | + * eviction. The first check takes care of the accessed bit set on the initial | |
| 320 | + * fault; the second check makes sure this page hasn't been used since then. | |
| 321 | + * This process, AKA second chance, requires a minimum of two generations, | |
| 322 | + * hence MIN_NR_GENS. And to maintain ABI compatibility with the active/inactive | |
| 323 | + * LRU, e.g., /proc/vmstat, these two generations are considered active; the | |
| 324 | + * rest of generations, if they exist, are considered inactive. See | |
| 325 | + * lru_gen_is_active(). | |
| 326 | + * | |
| 327 | + * PG_active is always cleared while a page is on one of lrugen->lists[] so that | |
| 328 | + * the aging needs not to worry about it. And it's set again when a page | |
| 329 | + * considered active is isolated for non-reclaiming purposes, e.g., migration. | |
| 330 | + * See lru_gen_add_folio() and lru_gen_del_folio(). | |
| 331 | + * | |
| 332 | + * MAX_NR_GENS is set to 4 so that the multi-gen LRU can support twice the | |
| 333 | + * number of categories of the active/inactive LRU when keeping track of | |
| 334 | + * accesses through page tables. This requires order_base_2(MAX_NR_GENS+1) bits | |
| 335 | + * in folio->flags. | |
| 336 | + */ | |
| 337 | +#define MIN_NR_GENS 2U | |
| 338 | +#define MAX_NR_GENS 4U | |
| 339 | + | |
| 340 | +/* | |
| 341 | + * Each generation is divided into multiple tiers. A page accessed N times | |
| 342 | + * through file descriptors is in tier order_base_2(N). A page in the first tier | |
| 343 | + * (N=0,1) is marked by PG_referenced unless it was faulted in through page | |
| 344 | + * tables or read ahead. A page in any other tier (N>1) is marked by | |
| 345 | + * PG_referenced and PG_workingset. This implies a minimum of two tiers is | |
| 346 | + * supported without using additional bits in folio->flags. | |
| 347 | + * | |
| 348 | + * In contrast to moving across generations which requires the LRU lock, moving | |
| 349 | + * across tiers only involves atomic operations on folio->flags and therefore | |
| 350 | + * has a negligible cost in the buffered access path. In the eviction path, | |
| 351 | + * comparisons of refaulted/(evicted+protected) from the first tier and the | |
| 352 | + * rest infer whether pages accessed multiple times through file descriptors | |
| 353 | + * are statistically hot and thus worth protecting. | |
| 354 | + * | |
| 355 | + * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the | |
| 356 | + * number of categories of the active/inactive LRU when keeping track of | |
| 357 | + * accesses through file descriptors. This uses MAX_NR_TIERS-2 spare bits in | |
| 358 | + * folio->flags. | |
| 359 | + */ | |
| 360 | +#define MAX_NR_TIERS 4U | |
| 361 | + | |
| 362 | +#ifndef __GENERATING_BOUNDS_H | |
| 363 | + | |
| 364 | +struct lruvec; | |
| 365 | +struct page_vma_mapped_walk; | |
| 366 | + | |
| 367 | +#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) | |
| 368 | +#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) | |
| 369 | + | |
| 370 | +#ifdef CONFIG_LRU_GEN | |
| 371 | + | |
| 372 | +enum { | |
| 373 | + LRU_GEN_ANON, | |
| 374 | + LRU_GEN_FILE, | |
| 375 | +}; | |
| 376 | + | |
| 377 | +enum { | |
| 378 | + LRU_GEN_CORE, | |
| 379 | + LRU_GEN_MM_WALK, | |
| 380 | + LRU_GEN_NONLEAF_YOUNG, | |
| 381 | + NR_LRU_GEN_CAPS | |
| 382 | +}; | |
| 383 | + | |
| 384 | +#define MIN_LRU_BATCH BITS_PER_LONG | |
| 385 | +#define MAX_LRU_BATCH (MIN_LRU_BATCH * 64) | |
| 386 | + | |
| 387 | +/* whether to keep historical stats from evicted generations */ | |
| 388 | +#ifdef CONFIG_LRU_GEN_STATS | |
| 389 | +#define NR_HIST_GENS MAX_NR_GENS | |
| 390 | +#else | |
| 391 | +#define NR_HIST_GENS 1U | |
| 392 | +#endif | |
| 393 | + | |
| 394 | +/* | |
| 395 | + * The youngest generation number is stored in max_seq for both anon and file | |
| 396 | + * types as they are aged on an equal footing. The oldest generation numbers are | |
| 397 | + * stored in min_seq[] separately for anon and file types as clean file pages | |
| 398 | + * can be evicted regardless of swap constraints. | |
| 399 | + * | |
| 400 | + * Normally anon and file min_seq are in sync. But if swapping is constrained, | |
| 401 | + * e.g., out of swap space, file min_seq is allowed to advance and leave anon | |
| 402 | + * min_seq behind. | |
| 403 | + * | |
| 404 | + * The number of pages in each generation is eventually consistent and therefore | |
| 405 | + * can be transiently negative when reset_batch_size() is pending. | |
| 406 | + */ | |
| 407 | +struct lru_gen_struct { | |
| 408 | + /* the aging increments the youngest generation number */ | |
| 409 | + unsigned long max_seq; | |
| 410 | + /* the eviction increments the oldest generation numbers */ | |
| 411 | + unsigned long min_seq[ANON_AND_FILE]; | |
| 412 | + /* the birth time of each generation in jiffies */ | |
| 413 | + unsigned long timestamps[MAX_NR_GENS]; | |
| 414 | + /* the multi-gen LRU lists, lazily sorted on eviction */ | |
| 415 | + struct list_head lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; | |
| 416 | + /* the multi-gen LRU sizes, eventually consistent */ | |
| 417 | + long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; | |
| 418 | + /* the exponential moving average of refaulted */ | |
| 419 | + unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS]; | |
| 420 | + /* the exponential moving average of evicted+protected */ | |
| 421 | + unsigned long avg_total[ANON_AND_FILE][MAX_NR_TIERS]; | |
| 422 | + /* the first tier doesn't need protection, hence the minus one */ | |
| 423 | + unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS - 1]; | |
| 424 | + /* can be modified without holding the LRU lock */ | |
| 425 | + atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; | |
| 426 | + atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; | |
| 427 | + /* whether the multi-gen LRU is enabled */ | |
| 428 | + bool enabled; | |
| 429 | +}; | |
| 430 | + | |
| 431 | +enum { | |
| 432 | + MM_LEAF_TOTAL, /* total leaf entries */ | |
| 433 | + MM_LEAF_OLD, /* old leaf entries */ | |
| 434 | + MM_LEAF_YOUNG, /* young leaf entries */ | |
| 435 | + MM_NONLEAF_TOTAL, /* total non-leaf entries */ | |
| 436 | + MM_NONLEAF_FOUND, /* non-leaf entries found in Bloom filters */ | |
| 437 | + MM_NONLEAF_ADDED, /* non-leaf entries added to Bloom filters */ | |
| 438 | + NR_MM_STATS | |
| 439 | +}; | |
| 440 | + | |
| 441 | +/* double-buffering Bloom filters */ | |
| 442 | +#define NR_BLOOM_FILTERS 2 | |
| 443 | + | |
| 444 | +struct lru_gen_mm_state { | |
| 445 | + /* set to max_seq after each iteration */ | |
| 446 | + unsigned long seq; | |
| 447 | + /* where the current iteration continues (inclusive) */ | |
| 448 | + struct list_head *head; | |
| 449 | + /* where the last iteration ended (exclusive) */ | |
| 450 | + struct list_head *tail; | |
| 451 | + /* to wait for the last page table walker to finish */ | |
| 452 | + struct wait_queue_head wait; | |
| 453 | + /* Bloom filters flip after each iteration */ | |
| 454 | + unsigned long *filters[NR_BLOOM_FILTERS]; | |
| 455 | + /* the mm stats for debugging */ | |
| 456 | + unsigned long stats[NR_HIST_GENS][NR_MM_STATS]; | |
| 457 | + /* the number of concurrent page table walkers */ | |
| 458 | + int nr_walkers; | |
| 459 | +}; | |
| 460 | + | |
| 461 | +struct lru_gen_mm_walk { | |
| 462 | + /* the lruvec under reclaim */ | |
| 463 | + struct lruvec *lruvec; | |
| 464 | + /* unstable max_seq from lru_gen_struct */ | |
| 465 | + unsigned long max_seq; | |
| 466 | + /* the next address within an mm to scan */ | |
| 467 | + unsigned long next_addr; | |
| 468 | + /* to batch promoted pages */ | |
| 469 | + int nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; | |
| 470 | + /* to batch the mm stats */ | |
| 471 | + int mm_stats[NR_MM_STATS]; | |
| 472 | + /* total batched items */ | |
| 473 | + int batched; | |
| 474 | + bool can_swap; | |
| 475 | + bool force_scan; | |
| 476 | +}; | |
| 477 | + | |
| 478 | +void lru_gen_init_lruvec(struct lruvec *lruvec); | |
| 479 | +void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); | |
| 480 | + | |
| 481 | +#ifdef CONFIG_MEMCG | |
| 482 | +void lru_gen_init_memcg(struct mem_cgroup *memcg); | |
| 483 | +void lru_gen_exit_memcg(struct mem_cgroup *memcg); | |
| 484 | +#endif | |
| 485 | + | |
| 486 | +#else /* !CONFIG_LRU_GEN */ | |
| 487 | + | |
| 488 | +static inline void lru_gen_init_lruvec(struct lruvec *lruvec) | |
| 489 | +{ | |
| 490 | +} | |
| 491 | + | |
| 492 | +static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) | |
| 493 | +{ | |
| 494 | +} | |
| 495 | + | |
| 496 | +#ifdef CONFIG_MEMCG | |
| 497 | +static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) | |
| 498 | +{ | |
| 499 | +} | |
| 500 | + | |
| 501 | +static inline void lru_gen_exit_memcg(struct mem_cgroup *memcg) | |
| 502 | +{ | |
| 503 | +} | |
| 504 | +#endif | |
| 505 | + | |
| 506 | +#endif /* CONFIG_LRU_GEN */ | |
| 507 | + | |
| 318 | 508 | struct lruvec { |
| 319 | 509 | struct list_head lists[NR_LRU_LISTS]; |
| 320 | 510 | /* per lruvec lru_lock for memcg */ |
| @@ -332,6 +522,12 @@ struct lruvec { · 11 unmodified lines | ||
| 332 | 522 | unsigned long refaults[ANON_AND_FILE]; |
| 333 | 523 | /* Various lruvec state flags (enum lruvec_flags) */ |
| 334 | 524 | unsigned long flags; |
| 525 | +#ifdef CONFIG_LRU_GEN | |
| 526 | + /* evictable pages divided into generations */ | |
| 527 | + struct lru_gen_struct lrugen; | |
| 528 | + /* to concurrently iterate lru_gen_mm_list */ | |
| 529 | + struct lru_gen_mm_state mm_state; | |
| 530 | +#endif | |
| 335 | 531 | #ifdef CONFIG_MEMCG |
| 336 | 532 | struct pglist_data *pgdat; |
| 337 | 533 | #endif |
| @@ -369,13 +565,6 @@ enum zone_watermarks { · 31 unmodified lines | ||
| 369 | 565 | #define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER + 1)) |
| 370 | 566 | #define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) |
| 371 | 567 | |
| 372 | −/* | |
| 373 | − * Shift to encode migratetype and order in the same integer, with order | |
| 374 | − * in the least significant bits. | |
| 375 | − */ | |
| 376 | −#define NR_PCP_ORDER_WIDTH 8 | |
| 377 | −#define NR_PCP_ORDER_MASK ((1<<NR_PCP_ORDER_WIDTH) - 1) | |
| 378 | − | |
| 379 | 568 | #define min_wmark_pages(z) (z->_watermark[WMARK_MIN] + z->watermark_boost) |
| 380 | 569 | #define low_wmark_pages(z) (z->_watermark[WMARK_LOW] + z->watermark_boost) |
| 381 | 570 | #define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost) |
| @@ -628,7 +817,7 @@ struct zone { · 246 unmodified lines | ||
| 628 | 817 | int initialized; |
| 629 | 818 | |
| 630 | 819 | /* Write-intensive fields used from the page allocator */ |
| 631 | − ZONE_PADDING(_pad1_) | |
| 820 | + CACHELINE_PADDING(_pad1_); | |
| 632 | 821 | |
| 633 | 822 | /* free areas of different sizes */ |
| 634 | 823 | struct free_area free_area[MAX_ORDER]; |
| @@ -640,7 +829,7 @@ struct zone { · 5 unmodified lines | ||
| 640 | 829 | spinlock_t lock; |
| 641 | 830 | |
| 642 | 831 | /* Write-intensive fields used by compaction and vmstats. */ |
| 643 | − ZONE_PADDING(_pad2_) | |
| 832 | + CACHELINE_PADDING(_pad2_); | |
| 644 | 833 | |
| 645 | 834 | /* |
| 646 | 835 | * When free pages are below this point, additional steps are taken |
| @@ -677,7 +866,7 @@ struct zone { · 30 unmodified lines | ||
| 677 | 866 | |
| 678 | 867 | bool contiguous; |
| 679 | 868 | |
| 680 | − ZONE_PADDING(_pad3_) | |
| 869 | + CACHELINE_PADDING(_pad3_); | |
| 681 | 870 | /* Zone statistics */ |
| 682 | 871 | atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; |
| 683 | 872 | atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; |
| @@ -747,6 +936,8 @@ static inline bool zone_is_empty(struct zone *zone) · 63 unmodified lines | ||
| 747 | 936 | #define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH) |
| 748 | 937 | #define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH) |
| 749 | 938 | #define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH) |
| 939 | +#define LRU_GEN_PGOFF (KASAN_TAG_PGOFF - LRU_GEN_WIDTH) | |
| 940 | +#define LRU_REFS_PGOFF (LRU_GEN_PGOFF - LRU_REFS_WIDTH) | |
| 750 | 941 | |
| 751 | 942 | /* |
| 752 | 943 | * Define the bit shifts to access each section. For non-existent |
| @@ -954,8 +1145,10 @@ typedef struct pglist_data { · 201 unmodified lines | ||
| 954 | 1145 | atomic_t nr_writeback_throttled;/* nr of writeback-throttled tasks */ |
| 955 | 1146 | unsigned long nr_reclaim_start; /* nr pages written while throttled |
| 956 | 1147 | * when throttling started. */ |
| 957 | − struct task_struct *kswapd; /* Protected by | |
| 958 | − mem_hotplug_begin/done() */ | |
| 1148 | +#ifdef CONFIG_MEMORY_HOTPLUG | |
| 1149 | + struct mutex kswapd_lock; | |
| 1150 | +#endif | |
| 1151 | + struct task_struct *kswapd; /* Protected by kswapd_lock */ | |
| 959 | 1152 | int kswapd_order; |
| 960 | 1153 | enum zone_type kswapd_highest_zoneidx; |
| 961 | 1154 | |
| @@ -983,7 +1176,7 @@ typedef struct pglist_data { · 21 unmodified lines | ||
| 983 | 1176 | #endif /* CONFIG_NUMA */ |
| 984 | 1177 | |
| 985 | 1178 | /* Write-intensive fields used by page reclaim */ |
| 986 | − ZONE_PADDING(_pad1_) | |
| 1179 | + CACHELINE_PADDING(_pad1_); | |
| 987 | 1180 | |
| 988 | 1181 | #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT |
| 989 | 1182 | /* |
| @@ -997,6 +1190,21 @@ typedef struct pglist_data { · 7 unmodified lines | ||
| 997 | 1190 | struct deferred_split deferred_split_queue; |
| 998 | 1191 | #endif |
| 999 | 1192 | |
| 1193 | +#ifdef CONFIG_NUMA_BALANCING | |
| 1194 | + /* start time in ms of current promote rate limit period */ | |
| 1195 | + unsigned int nbp_rl_start; | |
| 1196 | + /* number of promote candidate pages at start time of current rate limit period */ | |
| 1197 | + unsigned long nbp_rl_nr_cand; | |
| 1198 | + /* promote threshold in ms */ | |
| 1199 | + unsigned int nbp_threshold; | |
| 1200 | + /* start time in ms of current promote threshold adjustment period */ | |
| 1201 | + unsigned int nbp_th_start; | |
| 1202 | + /* | |
| 1203 | + * number of promote candidate pages at stat time of current promote | |
| 1204 | + * threshold adjustment period | |
| 1205 | + */ | |
| 1206 | + unsigned long nbp_th_nr_cand; | |
| 1207 | +#endif | |
| 1000 | 1208 | /* Fields commonly accessed by the page reclaim scanner */ |
| 1001 | 1209 | |
| 1002 | 1210 | /* |
| @@ -1008,11 +1216,19 @@ typedef struct pglist_data { · 5 unmodified lines | ||
| 1008 | 1216 | |
| 1009 | 1217 | unsigned long flags; |
| 1010 | 1218 | |
| 1011 | − ZONE_PADDING(_pad2_) | |
| 1219 | +#ifdef CONFIG_LRU_GEN | |
| 1220 | + /* kswap mm walk data */ | |
| 1221 | + struct lru_gen_mm_walk mm_walk; | |
| 1222 | +#endif | |
| 1223 | + | |
| 1224 | + CACHELINE_PADDING(_pad2_); | |
| 1012 | 1225 | |
| 1013 | 1226 | /* Per-node vmstats */ |
| 1014 | 1227 | struct per_cpu_nodestat __percpu *per_cpu_nodestats; |
| 1015 | 1228 | atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS]; |
| 1229 | +#ifdef CONFIG_NUMA | |
| 1230 | + struct memory_tier __rcu *memtier; | |
| 1231 | +#endif | |
| 1016 | 1232 | } pg_data_t; |
| 1017 | 1233 | |
| 1018 | 1234 | #define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages) |
| @@ -1026,11 +1242,6 @@ static inline unsigned long pgdat_end_pfn(pg_data_t *pgdat) · 7 unmodified lines | ||
| 1026 | 1242 | return pgdat->node_start_pfn + pgdat->node_spanned_pages; |
| 1027 | 1243 | } |
| 1028 | 1244 | |
| 1029 | −static inline bool pgdat_is_empty(pg_data_t *pgdat) | |
| 1030 | −{ | |
| 1031 | − return !pgdat->node_start_pfn && !pgdat->node_spanned_pages; | |
| 1032 | −} | |
| 1033 | − | |
| 1034 | 1245 | #include <linux/memory_hotplug.h> |
| 1035 | 1246 | |
| 1036 | 1247 | void build_all_zonelists(pg_data_t *pgdat); |
| @@ -2,15 +2,15 @@ · 1 unmodified line | ||
| 2 | 2 | /* |
| 3 | 3 | * include/linux/node.h - generic node definition |
| 4 | 4 | * |
| 5 | − * This is mainly for topological representation. We define the | |
| 6 | − * basic 'struct node' here, which can be embedded in per-arch | |
| 5 | + * This is mainly for topological representation. We define the | |
| 6 | + * basic 'struct node' here, which can be embedded in per-arch | |
| 7 | 7 | * definitions of processors. |
| 8 | 8 | * |
| 9 | 9 | * Basic handling of the devices is done in drivers/base/node.c |
| 10 | − * and system devices are handled in drivers/base/sys.c. | |
| 10 | + * and system devices are handled in drivers/base/sys.c. | |
| 11 | 11 | * |
| 12 | 12 | * Nodes are exported via driverfs in the class/node/devices/ |
| 13 | − * directory. | |
| 13 | + * directory. | |
| 14 | 14 | */ |
| 15 | 15 | #ifndef _LINUX_NODE_H_ |
| 16 | 16 | #define _LINUX_NODE_H_ |
| @@ -18,7 +18,6 @@ · 1 unmodified line | ||
| 18 | 18 | #include <linux/device.h> |
| 19 | 19 | #include <linux/cpumask.h> |
| 20 | 20 | #include <linux/list.h> |
| 21 | −#include <linux/workqueue.h> | |
| 22 | 21 | |
| 23 | 22 | /** |
| 24 | 23 | * struct node_hmem_attrs - heterogeneous memory performance attributes |
| @@ -84,10 +83,6 @@ static inline void node_set_perf_attrs(unsigned int nid, · 59 unmodified lines | ||
| 84 | 83 | struct node { |
| 85 | 84 | struct device dev; |
| 86 | 85 | struct list_head access_list; |
| 87 | − | |
| 88 | −#if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_HUGETLBFS) | |
| 89 | − struct work_struct node_work; | |
| 90 | −#endif | |
| 91 | 86 | #ifdef CONFIG_HMEM_REPORTING |
| 92 | 87 | struct list_head cache_attrs; |
| 93 | 88 | struct device *cache_dev; |
| @@ -96,7 +91,6 @@ struct node { · 2 unmodified lines | ||
| 96 | 91 | |
| 97 | 92 | struct memory_block; |
| 98 | 93 | extern struct node *node_devices[]; |
| 99 | −typedef void (*node_registration_func_t)(struct node *); | |
| 100 | 94 | |
| 101 | 95 | #if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_NUMA) |
| 102 | 96 | void register_memory_blocks_under_node(int nid, unsigned long start_pfn, |
| @@ -144,11 +138,6 @@ extern void unregister_memory_block_under_nodes(struct memory_block *mem_blk); · 41 unmodified lines | ||
| 144 | 138 | extern int register_memory_node_under_compute_node(unsigned int mem_nid, |
| 145 | 139 | unsigned int cpu_nid, |
| 146 | 140 | unsigned access); |
| 147 | − | |
| 148 | −#ifdef CONFIG_HUGETLBFS | |
| 149 | −extern void register_hugetlbfs_with_node(node_registration_func_t doregister, | |
| 150 | − node_registration_func_t unregister); | |
| 151 | −#endif | |
| 152 | 141 | #else |
| 153 | 142 | static inline void node_dev_init(void) |
| 154 | 143 | { |
| @@ -176,18 +165,8 @@ static inline int unregister_cpu_under_node(unsigned int cpu, unsigned int nid) · 21 unmodified lines | ||
| 176 | 165 | static inline void unregister_memory_block_under_nodes(struct memory_block *mem_blk) |
| 177 | 166 | { |
| 178 | 167 | } |
| 179 | − | |
| 180 | −static inline void register_hugetlbfs_with_node(node_registration_func_t reg, | |
| 181 | − node_registration_func_t unreg) | |
| 182 | −{ | |
| 183 | −} | |
| 184 | 168 | #endif |
| 185 | 169 | |
| 186 | 170 | #define to_node(device) container_of(device, struct node, dev) |
| 187 | 171 | |
| 188 | −static inline bool node_is_toptier(int node) | |
| 189 | −{ | |
| 190 | − return node_state(node, N_CPU); | |
| 191 | −} | |
| 192 | − | |
| 193 | 172 | #endif /* _LINUX_NODE_H_ */ |
| @@ -493,6 +493,7 @@ static inline int num_node_state(enum node_states state) · 492 unmodified lines | ||
| 493 | 493 | #define first_online_node 0 |
| 494 | 494 | #define first_memory_node 0 |
| 495 | 495 | #define next_online_node(nid) (MAX_NUMNODES) |
| 496 | +#define next_memory_node(nid) (MAX_NUMNODES) | |
| 496 | 497 | #define nr_node_ids 1U |
| 497 | 498 | #define nr_online_nodes 1U |
| 498 | 499 | |
| @@ -504,11 +505,20 @@ static inline int num_node_state(enum node_states state) · 5 unmodified lines | ||
| 504 | 505 | static inline int node_random(const nodemask_t *maskp) |
| 505 | 506 | { |
| 506 | 507 | #if defined(CONFIG_NUMA) && (MAX_NUMNODES > 1) |
| 507 | − int w, bit = NUMA_NO_NODE; | |
| 508 | + int w, bit; | |
| 508 | 509 | |
| 509 | 510 | w = nodes_weight(*maskp); |
| 510 | − if (w) | |
| 511 | + switch (w) { | |
| 512 | + case 0: | |
| 513 | + bit = NUMA_NO_NODE; | |
| 514 | + break; | |
| 515 | + case 1: | |
| 516 | + bit = first_node(*maskp); | |
| 517 | + break; | |
| 518 | + default: | |
| 511 | 519 | bit = find_nth_bit(maskp->bits, MAX_NUMNODES, get_random_int() % w); |
| 520 | + break; | |
| 521 | + } | |
| 512 | 522 | return bit; |
| 513 | 523 | #else |
| 514 | 524 | return 0; |
| @@ -77,15 +77,6 @@ static inline bool tsk_is_oom_victim(struct task_struct * tsk) · 76 unmodified lines | ||
| 77 | 77 | return tsk->signal->oom_mm; |
| 78 | 78 | } |
| 79 | 79 | |
| 80 | −/* | |
| 81 | − * Use this helper if tsk->mm != mm and the victim mm needs a special | |
| 82 | − * handling. This is guaranteed to stay true after once set. | |
| 83 | − */ | |
| 84 | −static inline bool mm_is_oom_victim(struct mm_struct *mm) | |
| 85 | −{ | |
| 86 | − return test_bit(MMF_OOM_VICTIM, &mm->flags); | |
| 87 | −} | |
| 88 | − | |
| 89 | 80 | /* |
| 90 | 81 | * Checks whether a page fault on the given mm is still reliable. |
| 91 | 82 | * This is no longer true if the oom reaper started to reap the |
| @@ -106,8 +97,6 @@ static inline vm_fault_t check_stable_address_space(struct mm_struct *mm) · 14 unmodified lines | ||
| 106 | 97 | return 0; |
| 107 | 98 | } |
| 108 | 99 | |
| 109 | −bool __oom_reap_task_mm(struct mm_struct *mm); | |
| 110 | − | |
| 111 | 100 | long oom_badness(struct task_struct *p, |
| 112 | 101 | unsigned long totalpages); |
| 113 | 102 | |
| @@ -55,7 +55,8 @@ · 54 unmodified lines | ||
| 55 | 55 | #define SECTIONS_WIDTH 0 |
| 56 | 56 | #endif |
| 57 | 57 | |
| 58 | −#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS | |
| 58 | +#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_SHIFT \ | |
| 59 | + <= BITS_PER_LONG - NR_PAGEFLAGS | |
| 59 | 60 | #define NODES_WIDTH NODES_SHIFT |
| 60 | 61 | #elif defined(CONFIG_SPARSEMEM_VMEMMAP) |
| 61 | 62 | #error "Vmemmap: No space for nodes field in page flags" |
| @@ -89,8 +90,8 @@ · 27 unmodified lines | ||
| 89 | 90 | #define LAST_CPUPID_SHIFT 0 |
| 90 | 91 | #endif |
| 91 | 92 | |
| 92 | −#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + KASAN_TAG_WIDTH + LAST_CPUPID_SHIFT \ | |
| 93 | − <= BITS_PER_LONG - NR_PAGEFLAGS | |
| 93 | +#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + \ | |
| 94 | + KASAN_TAG_WIDTH + LAST_CPUPID_SHIFT <= BITS_PER_LONG - NR_PAGEFLAGS | |
| 94 | 95 | #define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT |
| 95 | 96 | #else |
| 96 | 97 | #define LAST_CPUPID_WIDTH 0 |
| @@ -100,10 +101,15 @@ · 3 unmodified lines | ||
| 100 | 101 | #define LAST_CPUPID_NOT_IN_PAGE_FLAGS |
| 101 | 102 | #endif |
| 102 | 103 | |
| 103 | −#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + KASAN_TAG_WIDTH + LAST_CPUPID_WIDTH \ | |
| 104 | − > BITS_PER_LONG - NR_PAGEFLAGS | |
| 104 | +#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + \ | |
| 105 | + KASAN_TAG_WIDTH + LAST_CPUPID_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS | |
| 105 | 106 | #error "Not enough bits in page flags" |
| 106 | 107 | #endif |
| 107 | 108 | |
| 109 | +/* see the comment on MAX_NR_TIERS */ | |
| 110 | +#define LRU_REFS_WIDTH min(__LRU_REFS_WIDTH, BITS_PER_LONG - NR_PAGEFLAGS - \ | |
| 111 | + ZONES_WIDTH - LRU_GEN_WIDTH - SECTIONS_WIDTH - \ | |
| 112 | + NODES_WIDTH - KASAN_TAG_WIDTH - LAST_CPUPID_WIDTH) | |
| 113 | + | |
| 108 | 114 | #endif |
| 109 | 115 | #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ |
| @@ -1058,7 +1058,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page) · 1057 unmodified lines | ||
| 1058 | 1058 | 1UL << PG_private | 1UL << PG_private_2 | \ |
| 1059 | 1059 | 1UL << PG_writeback | 1UL << PG_reserved | \ |
| 1060 | 1060 | 1UL << PG_slab | 1UL << PG_active | \ |
| 1061 | − 1UL << PG_unevictable | __PG_MLOCKED) | |
| 1061 | + 1UL << PG_unevictable | __PG_MLOCKED | LRU_GEN_MASK) | |
| 1062 | 1062 | |
| 1063 | 1063 | /* |
| 1064 | 1064 | * Flags checked when a page is prepped for return by the page allocator. |
| @@ -1069,7 +1069,7 @@ static __always_inline void __ClearPageAnonExclusive(struct page *page) · 4 unmodified lines | ||
| 1069 | 1069 | * alloc-free cycle to prevent from reusing the page. |
| 1070 | 1070 | */ |
| 1071 | 1071 | #define PAGE_FLAGS_CHECK_AT_PREP \ |
| 1072 | − (PAGEFLAGS_MASK & ~__PG_HWPOISON) | |
| 1072 | + ((PAGEFLAGS_MASK & ~__PG_HWPOISON) | LRU_GEN_MASK | LRU_REFS_MASK) | |
| 1073 | 1073 | |
| 1074 | 1074 | #define PAGE_FLAGS_PRIVATE \ |
| 1075 | 1075 | (1UL << PG_private | 1UL << PG_private_2) |
| @@ -3,15 +3,17 @@ · 2 unmodified lines | ||
| 3 | 3 | #define _LINUX_PAGE_COUNTER_H |
| 4 | 4 | |
| 5 | 5 | #include <linux/atomic.h> |
| 6 | +#include <linux/cache.h> | |
| 6 | 7 | #include <linux/kernel.h> |
| 7 | 8 | #include <asm/page.h> |
| 8 | 9 | |
| 9 | 10 | struct page_counter { |
| 11 | + /* | |
| 12 | + * Make sure 'usage' does not share cacheline with any other field. The | |
| 13 | + * memcg->memory.usage is a hot member of struct mem_cgroup. | |
| 14 | + */ | |
| 10 | 15 | atomic_long_t usage; |
| 11 | − unsigned long min; | |
| 12 | − unsigned long low; | |
| 13 | − unsigned long high; | |
| 14 | − unsigned long max; | |
| 16 | + CACHELINE_PADDING(_pad1_); | |
| 15 | 17 | |
| 16 | 18 | /* effective memory.min and memory.min usage tracking */ |
| 17 | 19 | unsigned long emin; |
| @@ -23,18 +25,18 @@ struct page_counter { · 5 unmodified lines | ||
| 23 | 25 | atomic_long_t low_usage; |
| 24 | 26 | atomic_long_t children_low_usage; |
| 25 | 27 | |
| 26 | − /* legacy */ | |
| 27 | 28 | unsigned long watermark; |
| 28 | 29 | unsigned long failcnt; |
| 29 | 30 | |
| 30 | − /* | |
| 31 | − * 'parent' is placed here to be far from 'usage' to reduce | |
| 32 | − * cache false sharing, as 'usage' is written mostly while | |
| 33 | − * parent is frequently read for cgroup's hierarchical | |
| 34 | − * counting nature. | |
| 35 | − */ | |
| 31 | + /* Keep all the read most fields in a separete cacheline. */ | |
| 32 | + CACHELINE_PADDING(_pad2_); | |
| 33 | + | |
| 34 | + unsigned long min; | |
| 35 | + unsigned long low; | |
| 36 | + unsigned long high; | |
| 37 | + unsigned long max; | |
| 36 | 38 | struct page_counter *parent; |
| 37 | −}; | |
| 39 | +} ____cacheline_internodealigned_in_smp; | |
| 38 | 40 | |
| 39 | 41 | #if BITS_PER_LONG == 32 |
| 40 | 42 | #define PAGE_COUNTER_MAX LONG_MAX |
| @@ -36,9 +36,15 @@ struct page_ext { · 35 unmodified lines | ||
| 36 | 36 | unsigned long flags; |
| 37 | 37 | }; |
| 38 | 38 | |
| 39 | +extern bool early_page_ext; | |
| 39 | 40 | extern unsigned long page_ext_size; |
| 40 | 41 | extern void pgdat_page_ext_init(struct pglist_data *pgdat); |
| 41 | 42 | |
| 43 | +static inline bool early_page_ext_enabled(void) | |
| 44 | +{ | |
| 45 | + return early_page_ext; | |
| 46 | +} | |
| 47 | + | |
| 42 | 48 | #ifdef CONFIG_SPARSEMEM |
| 43 | 49 | static inline void page_ext_init_flatmem(void) |
| 44 | 50 | { |
| @@ -55,7 +61,8 @@ static inline void page_ext_init(void) · 10 unmodified lines | ||
| 55 | 61 | } |
| 56 | 62 | #endif |
| 57 | 63 | |
| 58 | −struct page_ext *lookup_page_ext(const struct page *page); | |
| 64 | +extern struct page_ext *page_ext_get(struct page *page); | |
| 65 | +extern void page_ext_put(struct page_ext *page_ext); | |
| 59 | 66 | |
| 60 | 67 | static inline struct page_ext *page_ext_next(struct page_ext *curr) |
| 61 | 68 | { |
| @@ -67,13 +74,13 @@ static inline struct page_ext *page_ext_next(struct page_ext *curr) · 5 unmodified lines | ||
| 67 | 74 | #else /* !CONFIG_PAGE_EXTENSION */ |
| 68 | 75 | struct page_ext; |
| 69 | 76 | |
| 70 | −static inline void pgdat_page_ext_init(struct pglist_data *pgdat) | |
| 77 | +static inline bool early_page_ext_enabled(void) | |
| 71 | 78 | { |
| 79 | + return false; | |
| 72 | 80 | } |
| 73 | 81 | |
| 74 | −static inline struct page_ext *lookup_page_ext(const struct page *page) | |
| 82 | +static inline void pgdat_page_ext_init(struct pglist_data *pgdat) | |
| 75 | 83 | { |
| 76 | − return NULL; | |
| 77 | 84 | } |
| 78 | 85 | |
| 79 | 86 | static inline void page_ext_init(void) |
| @@ -87,5 +94,14 @@ static inline void page_ext_init_flatmem_late(void) · 7 unmodified lines | ||
| 87 | 94 | static inline void page_ext_init_flatmem(void) |
| 88 | 95 | { |
| 89 | 96 | } |
| 97 | + | |
| 98 | +static inline struct page_ext *page_ext_get(struct page *page) | |
| 99 | +{ | |
| 100 | + return NULL; | |
| 101 | +} | |
| 102 | + | |
| 103 | +static inline void page_ext_put(struct page_ext *page_ext) | |
| 104 | +{ | |
| 105 | +} | |
| 90 | 106 | #endif /* CONFIG_PAGE_EXTENSION */ |
| 91 | 107 | #endif /* __LINUX_PAGE_EXT_H */ |
| @@ -13,65 +13,79 @@ · 12 unmodified lines | ||
| 13 | 13 | * If there is not enough space to store Idle and Young bits in page flags, use |
| 14 | 14 | * page ext flags instead. |
| 15 | 15 | */ |
| 16 | − | |
| 17 | 16 | static inline bool folio_test_young(struct folio *folio) |
| 18 | 17 | { |
| 19 | − struct page_ext *page_ext = lookup_page_ext(&folio->page); | |
| 18 | + struct page_ext *page_ext = page_ext_get(&folio->page); | |
| 19 | + bool page_young; | |
| 20 | 20 | |
| 21 | 21 | if (unlikely(!page_ext)) |
| 22 | 22 | return false; |
| 23 | 23 | |
| 24 | − return test_bit(PAGE_EXT_YOUNG, &page_ext->flags); | |
| 24 | + page_young = test_bit(PAGE_EXT_YOUNG, &page_ext->flags); | |
| 25 | + page_ext_put(page_ext); | |
| 26 | + | |
| 27 | + return page_young; | |
| 25 | 28 | } |
| 26 | 29 | |
| 27 | 30 | static inline void folio_set_young(struct folio *folio) |
| 28 | 31 | { |
| 29 | − struct page_ext *page_ext = lookup_page_ext(&folio->page); | |
| 32 | + struct page_ext *page_ext = page_ext_get(&folio->page); | |
| 30 | 33 | |
| 31 | 34 | if (unlikely(!page_ext)) |
| 32 | 35 | return; |
| 33 | 36 | |
| 34 | 37 | set_bit(PAGE_EXT_YOUNG, &page_ext->flags); |
| 38 | + page_ext_put(page_ext); | |
| 35 | 39 | } |
| 36 | 40 | |
| 37 | 41 | static inline bool folio_test_clear_young(struct folio *folio) |
| 38 | 42 | { |
| 39 | − struct page_ext *page_ext = lookup_page_ext(&folio->page); | |
| 43 | + struct page_ext *page_ext = page_ext_get(&folio->page); | |
| 44 | + bool page_young; | |
| 40 | 45 | |
| 41 | 46 | if (unlikely(!page_ext)) |
| 42 | 47 | return false; |
| 43 | 48 | |
| 44 | − return test_and_clear_bit(PAGE_EXT_YOUNG, &page_ext->flags); | |
| 49 | + page_young = test_and_clear_bit(PAGE_EXT_YOUNG, &page_ext->flags); | |
| 50 | + page_ext_put(page_ext); | |
| 51 | + | |
| 52 | + return page_young; | |
| 45 | 53 | } |
| 46 | 54 | |
| 47 | 55 | static inline bool folio_test_idle(struct folio *folio) |
| 48 | 56 | { |
| 49 | − struct page_ext *page_ext = lookup_page_ext(&folio->page); | |
| 57 | + struct page_ext *page_ext = page_ext_get(&folio->page); | |
| 58 | + bool page_idle; | |
| 50 | 59 | |
| 51 | 60 | if (unlikely(!page_ext)) |
| 52 | 61 | return false; |
| 53 | 62 | |
| 54 | − return test_bit(PAGE_EXT_IDLE, &page_ext->flags); | |
| 63 | + page_idle = test_bit(PAGE_EXT_IDLE, &page_ext->flags); | |
| 64 | + page_ext_put(page_ext); | |
| 65 | + | |
| 66 | + return page_idle; | |
| 55 | 67 | } |
| 56 | 68 | |
| 57 | 69 | static inline void folio_set_idle(struct folio *folio) |
| 58 | 70 | { |
| 59 | − struct page_ext *page_ext = lookup_page_ext(&folio->page); | |
| 71 | + struct page_ext *page_ext = page_ext_get(&folio->page); | |
| 60 | 72 | |
| 61 | 73 | if (unlikely(!page_ext)) |
| 62 | 74 | return; |
| 63 | 75 | |
| 64 | 76 | set_bit(PAGE_EXT_IDLE, &page_ext->flags); |
| 77 | + page_ext_put(page_ext); | |
| 65 | 78 | } |
| 66 | 79 | |
| 67 | 80 | static inline void folio_clear_idle(struct folio *folio) |
| 68 | 81 | { |
| 69 | − struct page_ext *page_ext = lookup_page_ext(&folio->page); | |
| 82 | + struct page_ext *page_ext = page_ext_get(&folio->page); | |
| 70 | 83 | |
| 71 | 84 | if (unlikely(!page_ext)) |
| 72 | 85 | return; |
| 73 | 86 | |
| 74 | 87 | clear_bit(PAGE_EXT_IDLE, &page_ext->flags); |
| 88 | + page_ext_put(page_ext); | |
| 75 | 89 | } |
| 76 | 90 | #endif /* !CONFIG_64BIT */ |
| 77 | 91 | |
| @@ -53,6 +53,10 @@ extern unsigned int pageblock_order; · 52 unmodified lines | ||
| 53 | 53 | #endif /* CONFIG_HUGETLB_PAGE */ |
| 54 | 54 | |
| 55 | 55 | #define pageblock_nr_pages (1UL << pageblock_order) |
| 56 | +#define pageblock_align(pfn) ALIGN((pfn), pageblock_nr_pages) | |
| 57 | +#define pageblock_aligned(pfn) IS_ALIGNED((pfn), pageblock_nr_pages) | |
| 58 | +#define pageblock_start_pfn(pfn) ALIGN_DOWN((pfn), pageblock_nr_pages) | |
| 59 | +#define pageblock_end_pfn(pfn) ALIGN((pfn) + 1, pageblock_nr_pages) | |
| 56 | 60 | |
| 57 | 61 | /* Forward declaration */ |
| 58 | 62 | struct page; |
| @@ -718,8 +718,8 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index) · 717 unmodified lines | ||
| 718 | 718 | |
| 719 | 719 | unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start, |
| 720 | 720 | pgoff_t end, struct folio_batch *fbatch); |
| 721 | −unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t start, | |
| 722 | − unsigned int nr_pages, struct page **pages); | |
| 721 | +unsigned filemap_get_folios_contig(struct address_space *mapping, | |
| 722 | + pgoff_t *start, pgoff_t end, struct folio_batch *fbatch); | |
| 723 | 723 | unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index, |
| 724 | 724 | pgoff_t end, xa_mark_t tag, unsigned int nr_pages, |
| 725 | 725 | struct page **pages); |
| @@ -989,19 +989,16 @@ static inline int lock_page_killable(struct page *page) · 263 unmodified lines | ||
| 989 | 989 | } |
| 990 | 990 | |
| 991 | 991 | /* |
| 992 | − * lock_page_or_retry - Lock the page, unless this would block and the | |
| 992 | + * folio_lock_or_retry - Lock the folio, unless this would block and the | |
| 993 | 993 | * caller indicated that it can handle a retry. |
| 994 | 994 | * |
| 995 | 995 | * Return value and mmap_lock implications depend on flags; see |
| 996 | 996 | * __folio_lock_or_retry(). |
| 997 | 997 | */ |
| 998 | −static inline bool lock_page_or_retry(struct page *page, struct mm_struct *mm, | |
| 999 | − unsigned int flags) | |
| 998 | +static inline bool folio_lock_or_retry(struct folio *folio, | |
| 999 | + struct mm_struct *mm, unsigned int flags) | |
| 1000 | 1000 | { |
| 1001 | − struct folio *folio; | |
| 1002 | 1001 | might_sleep(); |
| 1003 | − | |
| 1004 | − folio = page_folio(page); | |
| 1005 | 1002 | return folio_trylock(folio) || __folio_lock_or_retry(folio, mm, flags); |
| 1006 | 1003 | } |
| 1007 | 1004 | |
| @@ -1042,7 +1039,6 @@ static inline int wait_on_page_locked_killable(struct page *page) · 34 unmodified lines | ||
| 1042 | 1039 | return folio_wait_locked_killable(page_folio(page)); |
| 1043 | 1040 | } |
| 1044 | 1041 | |
| 1045 | −int folio_put_wait_locked(struct folio *folio, int state); | |
| 1046 | 1042 | void wait_on_page_writeback(struct page *page); |
| 1047 | 1043 | void folio_wait_writeback(struct folio *folio); |
| 1048 | 1044 | int folio_wait_writeback_killable(struct folio *folio); |
| @@ -15,12 +15,12 @@ struct mm_walk; · 14 unmodified lines | ||
| 15 | 15 | * this handler is required to be able to handle |
| 16 | 16 | * pmd_trans_huge() pmds. They may simply choose to |
| 17 | 17 | * split_huge_page() instead of handling it explicitly. |
| 18 | − * @pte_entry: if set, called for each non-empty PTE (lowest-level) | |
| 19 | − * entry | |
| 18 | + * @pte_entry: if set, called for each PTE (lowest-level) entry, | |
| 19 | + * including empty ones | |
| 20 | 20 | * @pte_hole: if set, called for each hole at all levels, |
| 21 | − * depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD | |
| 22 | − * 4:PTE. Any folded depths (where PTRS_PER_P?D is equal | |
| 23 | − * to 1) are skipped. | |
| 21 | + * depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD. | |
| 22 | + * Any folded depths (where PTRS_PER_P?D is equal to 1) | |
| 23 | + * are skipped. | |
| 24 | 24 | * @hugetlb_entry: if set, called for each hugetlb entry |
| 25 | 25 | * @test_walk: caller specific callback function to determine whether |
| 26 | 26 | * we walk over the current vma or not. Returning 0 means |
| @@ -213,7 +213,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, · 212 unmodified lines | ||
| 213 | 213 | #endif |
| 214 | 214 | |
| 215 | 215 | #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG |
| 216 | −#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 216 | +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) | |
| 217 | 217 | static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, |
| 218 | 218 | unsigned long address, |
| 219 | 219 | pmd_t *pmdp) |
| @@ -234,7 +234,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, · 14 unmodified lines | ||
| 234 | 234 | BUILD_BUG(); |
| 235 | 235 | return 0; |
| 236 | 236 | } |
| 237 | −#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ | |
| 237 | +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ | |
| 238 | 238 | #endif |
| 239 | 239 | |
| 240 | 240 | #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH |
| @@ -260,6 +260,19 @@ static inline int pmdp_clear_flush_young(struct vm_area_struct *vma, · 19 unmodified lines | ||
| 260 | 260 | #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ |
| 261 | 261 | #endif |
| 262 | 262 | |
| 263 | +#ifndef arch_has_hw_pte_young | |
| 264 | +/* | |
| 265 | + * Return whether the accessed bit is supported on the local CPU. | |
| 266 | + * | |
| 267 | + * This stub assumes accessing through an old PTE triggers a page fault. | |
| 268 | + * Architectures that automatically set the access bit should overwrite it. | |
| 269 | + */ | |
| 270 | +static inline bool arch_has_hw_pte_young(void) | |
| 271 | +{ | |
| 272 | + return false; | |
| 273 | +} | |
| 274 | +#endif | |
| 275 | + | |
| 263 | 276 | #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR |
| 264 | 277 | static inline pte_t ptep_get_and_clear(struct mm_struct *mm, |
| 265 | 278 | unsigned long address, |
| @@ -1276,8 +1289,7 @@ static inline int pgd_devmap(pgd_t pgd) · 1010 unmodified lines | ||
| 1276 | 1289 | #endif |
| 1277 | 1290 | |
| 1278 | 1291 | #if !defined(CONFIG_TRANSPARENT_HUGEPAGE) || \ |
| 1279 | − (defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ | |
| 1280 | − !defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)) | |
| 1292 | + !defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) | |
| 1281 | 1293 | static inline int pud_trans_huge(pud_t pud) |
| 1282 | 1294 | { |
| 1283 | 1295 | return 0; |
| @@ -1598,11 +1610,7 @@ typedef unsigned int pgtbl_mod_mask; · 314 unmodified lines | ||
| 1598 | 1610 | #endif |
| 1599 | 1611 | |
| 1600 | 1612 | #ifndef has_transparent_hugepage |
| 1601 | −#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 1602 | −#define has_transparent_hugepage() 1 | |
| 1603 | −#else | |
| 1604 | −#define has_transparent_hugepage() 0 | |
| 1605 | −#endif | |
| 1613 | +#define has_transparent_hugepage() IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) | |
| 1606 | 1614 | #endif |
| 1607 | 1615 | |
| 1608 | 1616 | /* |
| @@ -166,7 +166,7 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, · 165 unmodified lines | ||
| 166 | 166 | unlink_anon_vmas(next); |
| 167 | 167 | } |
| 168 | 168 | |
| 169 | −struct anon_vma *page_get_anon_vma(struct page *page); | |
| 169 | +struct anon_vma *folio_get_anon_vma(struct folio *folio); | |
| 170 | 170 | |
| 171 | 171 | /* RMAP flags, currently only relevant for some anon rmap operations. */ |
| 172 | 172 | typedef int __bitwise rmap_t; |
| @@ -270,7 +270,7 @@ dup: · 97 unmodified lines | ||
| 270 | 270 | * @page: the exclusive anonymous page to try marking possibly shared |
| 271 | 271 | * |
| 272 | 272 | * The caller needs to hold the PT lock and has to have the page table entry |
| 273 | − * cleared/invalidated+flushed, to properly sync against GUP-fast. | |
| 273 | + * cleared/invalidated. | |
| 274 | 274 | * |
| 275 | 275 | * This is similar to page_try_dup_anon_rmap(), however, not used during fork() |
| 276 | 276 | * to duplicate a mapping, but instead to prepare for KSM or temporarily |
| @@ -286,12 +286,68 @@ static inline int page_try_share_anon_rmap(struct page *page) · 9 unmodified lines | ||
| 286 | 286 | { |
| 287 | 287 | VM_BUG_ON_PAGE(!PageAnon(page) || !PageAnonExclusive(page), page); |
| 288 | 288 | |
| 289 | − /* See page_try_dup_anon_rmap(). */ | |
| 290 | − if (likely(!is_device_private_page(page) && | |
| 291 | − unlikely(page_maybe_dma_pinned(page)))) | |
| 292 | − return -EBUSY; | |
| 289 | + /* device private pages cannot get pinned via GUP. */ | |
| 290 | + if (unlikely(is_device_private_page(page))) { | |
| 291 | + ClearPageAnonExclusive(page); | |
| 292 | + return 0; | |
| 293 | + } | |
| 294 | + | |
| 295 | + /* | |
| 296 | + * We have to make sure that when we clear PageAnonExclusive, that | |
| 297 | + * the page is not pinned and that concurrent GUP-fast won't succeed in | |
| 298 | + * concurrently pinning the page. | |
| 299 | + * | |
| 300 | + * Conceptually, PageAnonExclusive clearing consists of: | |
| 301 | + * (A1) Clear PTE | |
| 302 | + * (A2) Check if the page is pinned; back off if so. | |
| 303 | + * (A3) Clear PageAnonExclusive | |
| 304 | + * (A4) Restore PTE (optional, but certainly not writable) | |
| 305 | + * | |
| 306 | + * When clearing PageAnonExclusive, we cannot possibly map the page | |
| 307 | + * writable again, because anon pages that may be shared must never | |
| 308 | + * be writable. So in any case, if the PTE was writable it cannot | |
| 309 | + * be writable anymore afterwards and there would be a PTE change. Only | |
| 310 | + * if the PTE wasn't writable, there might not be a PTE change. | |
| 311 | + * | |
| 312 | + * Conceptually, GUP-fast pinning of an anon page consists of: | |
| 313 | + * (B1) Read the PTE | |
| 314 | + * (B2) FOLL_WRITE: check if the PTE is not writable; back off if so. | |
| 315 | + * (B3) Pin the mapped page | |
| 316 | + * (B4) Check if the PTE changed by re-reading it; back off if so. | |
| 317 | + * (B5) If the original PTE is not writable, check if | |
| 318 | + * PageAnonExclusive is not set; back off if so. | |
| 319 | + * | |
| 320 | + * If the PTE was writable, we only have to make sure that GUP-fast | |
| 321 | + * observes a PTE change and properly backs off. | |
| 322 | + * | |
| 323 | + * If the PTE was not writable, we have to make sure that GUP-fast either | |
| 324 | + * detects a (temporary) PTE change or that PageAnonExclusive is cleared | |
| 325 | + * and properly backs off. | |
| 326 | + * | |
| 327 | + * Consequently, when clearing PageAnonExclusive(), we have to make | |
| 328 | + * sure that (A1), (A2)/(A3) and (A4) happen in the right memory | |
| 329 | + * order. In GUP-fast pinning code, we have to make sure that (B3),(B4) | |
| 330 | + * and (B5) happen in the right memory order. | |
| 331 | + * | |
| 332 | + * We assume that there might not be a memory barrier after | |
| 333 | + * clearing/invalidating the PTE (A1) and before restoring the PTE (A4), | |
| 334 | + * so we use explicit ones here. | |
| 335 | + */ | |
| 293 | 336 | |
| 337 | + /* Paired with the memory barrier in try_grab_folio(). */ | |
| 338 | + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) | |
| 339 | + smp_mb(); | |
| 340 | + | |
| 341 | + if (unlikely(page_maybe_dma_pinned(page))) | |
| 342 | + return -EBUSY; | |
| 294 | 343 | ClearPageAnonExclusive(page); |
| 344 | + | |
| 345 | + /* | |
| 346 | + * This is conceptually a smp_wmb() paired with the smp_rmb() in | |
| 347 | + * gup_must_unshare(). | |
| 348 | + */ | |
| 349 | + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) | |
| 350 | + smp_mb__after_atomic(); | |
| 295 | 351 | return 0; |
| 296 | 352 | } |
| 297 | 353 | |
| @@ -405,13 +461,8 @@ struct rmap_walk_control { · 107 unmodified lines | ||
| 405 | 461 | |
| 406 | 462 | void rmap_walk(struct folio *folio, struct rmap_walk_control *rwc); |
| 407 | 463 | void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc); |
| 408 | − | |
| 409 | −/* | |
| 410 | − * Called by memory-failure.c to kill processes. | |
| 411 | − */ | |
| 412 | 464 | struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, |
| 413 | 465 | struct rmap_walk_control *rwc); |
| 414 | −void page_unlock_anon_vma_read(struct anon_vma *anon_vma); | |
| 415 | 466 | |
| 416 | 467 | #else /* !CONFIG_MMU */ |
| 417 | 468 | |
| @@ -14,6 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <linux/pid.h> |
| 15 | 15 | #include <linux/sem.h> |
| 16 | 16 | #include <linux/shm.h> |
| 17 | +#include <linux/kmsan_types.h> | |
| 17 | 18 | #include <linux/mutex.h> |
| 18 | 19 | #include <linux/plist.h> |
| 19 | 20 | #include <linux/hrtimer.h> |
| @@ -870,7 +871,6 @@ struct task_struct { · 850 unmodified lines | ||
| 870 | 871 | struct mm_struct *active_mm; |
| 871 | 872 | |
| 872 | 873 | /* Per-thread vma caching: */ |
| 873 | − struct vmacache vmacache; | |
| 874 | 874 | |
| 875 | 875 | #ifdef SPLIT_RSS_COUNTING |
| 876 | 876 | struct task_rss_stat rss_stat; |
| @@ -923,6 +923,10 @@ struct task_struct { · 46 unmodified lines | ||
| 923 | 923 | #ifdef CONFIG_MEMCG |
| 924 | 924 | unsigned in_user_fault:1; |
| 925 | 925 | #endif |
| 926 | +#ifdef CONFIG_LRU_GEN | |
| 927 | + /* whether the LRU algorithm may apply to this access */ | |
| 928 | + unsigned in_lru_fault:1; | |
| 929 | +#endif | |
| 926 | 930 | #ifdef CONFIG_COMPAT_BRK |
| 927 | 931 | unsigned brk_randomized:1; |
| 928 | 932 | #endif |
| @@ -953,6 +957,10 @@ struct task_struct { · 24 unmodified lines | ||
| 953 | 957 | #ifdef CONFIG_CPU_SUP_INTEL |
| 954 | 958 | unsigned reported_split_lock:1; |
| 955 | 959 | #endif |
| 960 | +#ifdef CONFIG_TASK_DELAY_ACCT | |
| 961 | + /* delay due to memory thrashing */ | |
| 962 | + unsigned in_thrashing:1; | |
| 963 | +#endif | |
| 956 | 964 | |
| 957 | 965 | unsigned long atomic_flags; /* Flags requiring atomic access. */ |
| 958 | 966 | |
| @@ -1364,6 +1372,10 @@ struct task_struct { · 405 unmodified lines | ||
| 1364 | 1372 | #endif |
| 1365 | 1373 | #endif |
| 1366 | 1374 | |
| 1375 | +#ifdef CONFIG_KMSAN | |
| 1376 | + struct kmsan_ctx kmsan_ctx; | |
| 1377 | +#endif | |
| 1378 | + | |
| 1367 | 1379 | #if IS_ENABLED(CONFIG_KUNIT) |
| 1368 | 1380 | struct kunit *kunit_test; |
| 1369 | 1381 | #endif |
| @@ -71,9 +71,8 @@ static inline int get_dumpable(struct mm_struct *mm) · 70 unmodified lines | ||
| 71 | 71 | #define MMF_UNSTABLE 22 /* mm is unstable for copy_from_user */ |
| 72 | 72 | #define MMF_HUGE_ZERO_PAGE 23 /* mm has ever used the global huge zero page */ |
| 73 | 73 | #define MMF_DISABLE_THP 24 /* disable THP for all VMAs */ |
| 74 | −#define MMF_OOM_VICTIM 25 /* mm is the oom victim */ | |
| 75 | −#define MMF_OOM_REAP_QUEUED 26 /* mm was queued for oom_reaper */ | |
| 76 | −#define MMF_MULTIPROCESS 27 /* mm is shared between processes */ | |
| 74 | +#define MMF_OOM_REAP_QUEUED 25 /* mm was queued for oom_reaper */ | |
| 75 | +#define MMF_MULTIPROCESS 26 /* mm is shared between processes */ | |
| 77 | 76 | /* |
| 78 | 77 | * MMF_HAS_PINNED: Whether this mm has pinned any pages. This can be either |
| 79 | 78 | * replaced in the future by mm.pinned_vm when it becomes stable, or grow into |
| @@ -81,7 +80,7 @@ static inline int get_dumpable(struct mm_struct *mm) · 1 unmodified line | ||
| 81 | 80 | * pinned pages were unpinned later on, we'll still keep this bit set for the |
| 82 | 81 | * lifecycle of this mm, just for simplicity. |
| 83 | 82 | */ |
| 84 | −#define MMF_HAS_PINNED 28 /* FOLL_PIN has run, never cleared */ | |
| 83 | +#define MMF_HAS_PINNED 27 /* FOLL_PIN has run, never cleared */ | |
| 85 | 84 | #define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP) |
| 86 | 85 | |
| 87 | 86 | #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ |
| @@ -27,6 +27,7 @@ enum sched_tunable_scaling { · 26 unmodified lines | ||
| 27 | 27 | |
| 28 | 28 | #ifdef CONFIG_NUMA_BALANCING |
| 29 | 29 | extern int sysctl_numa_balancing_mode; |
| 30 | +extern unsigned int sysctl_numa_balancing_promote_rate_limit; | |
| 30 | 31 | #else |
| 31 | 32 | #define sysctl_numa_balancing_mode 0 |
| 32 | 33 | #endif |
| @@ -92,17 +92,19 @@ extern struct page *shmem_read_mapping_page_gfp(struct address_space *mapping, · 91 unmodified lines | ||
| 92 | 92 | extern void shmem_truncate_range(struct inode *inode, loff_t start, loff_t end); |
| 93 | 93 | int shmem_unuse(unsigned int type); |
| 94 | 94 | |
| 95 | −extern bool shmem_is_huge(struct vm_area_struct *vma, | |
| 96 | − struct inode *inode, pgoff_t index); | |
| 97 | −static inline bool shmem_huge_enabled(struct vm_area_struct *vma) | |
| 95 | +extern bool shmem_is_huge(struct vm_area_struct *vma, struct inode *inode, | |
| 96 | + pgoff_t index, bool shmem_huge_force); | |
| 97 | +static inline bool shmem_huge_enabled(struct vm_area_struct *vma, | |
| 98 | + bool shmem_huge_force) | |
| 98 | 99 | { |
| 99 | − return shmem_is_huge(vma, file_inode(vma->vm_file), vma->vm_pgoff); | |
| 100 | + return shmem_is_huge(vma, file_inode(vma->vm_file), vma->vm_pgoff, | |
| 101 | + shmem_huge_force); | |
| 100 | 102 | } |
| 101 | 103 | extern unsigned long shmem_swap_usage(struct vm_area_struct *vma); |
| 102 | 104 | extern unsigned long shmem_partial_swap_usage(struct address_space *mapping, |
| 103 | 105 | pgoff_t start, pgoff_t end); |
| 104 | 106 | |
| 105 | −/* Flag allocation requirements to shmem_getpage */ | |
| 107 | +/* Flag allocation requirements to shmem_get_folio */ | |
| 106 | 108 | enum sgp_type { |
| 107 | 109 | SGP_READ, /* don't exceed i_size, don't allocate page */ |
| 108 | 110 | SGP_NOALLOC, /* similar, but fail on hole or use fallocated page */ |
| @@ -111,8 +113,8 @@ enum sgp_type { · 2 unmodified lines | ||
| 111 | 113 | SGP_FALLOC, /* like SGP_WRITE, but make existing page Uptodate */ |
| 112 | 114 | }; |
| 113 | 115 | |
| 114 | −extern int shmem_getpage(struct inode *inode, pgoff_t index, | |
| 115 | − struct page **pagep, enum sgp_type sgp); | |
| 116 | +int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, | |
| 117 | + enum sgp_type sgp); | |
| 116 | 118 | |
| 117 | 119 | static inline struct page *shmem_read_mapping_page( |
| 118 | 120 | struct address_space *mapping, pgoff_t index) |
| @@ -108,7 +108,7 @@ · 107 unmodified lines | ||
| 108 | 108 | # define SLAB_ACCOUNT 0 |
| 109 | 109 | #endif |
| 110 | 110 | |
| 111 | −#ifdef CONFIG_KASAN | |
| 111 | +#ifdef CONFIG_KASAN_GENERIC | |
| 112 | 112 | #define SLAB_KASAN ((slab_flags_t __force)0x08000000U) |
| 113 | 113 | #else |
| 114 | 114 | #define SLAB_KASAN 0 |
| @@ -121,6 +121,12 @@ · 6 unmodified lines | ||
| 121 | 121 | */ |
| 122 | 122 | #define SLAB_NO_USER_FLAGS ((slab_flags_t __force)0x10000000U) |
| 123 | 123 | |
| 124 | +#ifdef CONFIG_KFENCE | |
| 125 | +#define SLAB_SKIP_KFENCE ((slab_flags_t __force)0x20000000U) | |
| 126 | +#else | |
| 127 | +#define SLAB_SKIP_KFENCE 0 | |
| 128 | +#endif | |
| 129 | + | |
| 124 | 130 | /* The following flags affect the page allocator grouping pages by mobility */ |
| 125 | 131 | /* Objects are reclaimable */ |
| 126 | 132 | #define SLAB_RECLAIM_ACCOUNT ((slab_flags_t __force)0x00020000U) |
| @@ -14,9 +14,15 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <linux/gfp.h> |
| 15 | 15 | |
| 16 | 16 | typedef u32 depot_stack_handle_t; |
| 17 | +/* | |
| 18 | + * Number of bits in the handle that stack depot doesn't use. Users may store | |
| 19 | + * information in them. | |
| 20 | + */ | |
| 21 | +#define STACK_DEPOT_EXTRA_BITS 5 | |
| 17 | 22 | |
| 18 | 23 | depot_stack_handle_t __stack_depot_save(unsigned long *entries, |
| 19 | 24 | unsigned int nr_entries, |
| 25 | + unsigned int extra_bits, | |
| 20 | 26 | gfp_t gfp_flags, bool can_alloc); |
| 21 | 27 | |
| 22 | 28 | /* |
| @@ -59,6 +65,8 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries, · 36 unmodified lines | ||
| 59 | 65 | unsigned int stack_depot_fetch(depot_stack_handle_t handle, |
| 60 | 66 | unsigned long **entries); |
| 61 | 67 | |
| 68 | +unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle); | |
| 69 | + | |
| 62 | 70 | int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size, |
| 63 | 71 | int spaces); |
| 64 | 72 | |
| @@ -162,6 +162,10 @@ union swap_header { · 161 unmodified lines | ||
| 162 | 162 | */ |
| 163 | 163 | struct reclaim_state { |
| 164 | 164 | unsigned long reclaimed_slab; |
| 165 | +#ifdef CONFIG_LRU_GEN | |
| 166 | + /* per-thread mm walk data */ | |
| 167 | + struct lru_gen_mm_walk *mm_walk; | |
| 168 | +#endif | |
| 165 | 169 | }; |
| 166 | 170 | |
| 167 | 171 | #ifdef __KERNEL__ |
| @@ -351,6 +355,11 @@ static inline swp_entry_t folio_swap_entry(struct folio *folio) · 183 unmodified lines | ||
| 351 | 355 | return entry; |
| 352 | 356 | } |
| 353 | 357 | |
| 358 | +static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry) | |
| 359 | +{ | |
| 360 | + folio->private = (void *)entry.val; | |
| 361 | +} | |
| 362 | + | |
| 354 | 363 | /* linux/mm/workingset.c */ |
| 355 | 364 | void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); |
| 356 | 365 | void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg); |
| @@ -375,11 +384,11 @@ extern unsigned long totalreserve_pages; · 18 unmodified lines | ||
| 375 | 384 | |
| 376 | 385 | |
| 377 | 386 | /* linux/mm/swap.c */ |
| 378 | −extern void lru_note_cost(struct lruvec *lruvec, bool file, | |
| 379 | − unsigned int nr_pages); | |
| 380 | −extern void lru_note_cost_folio(struct folio *); | |
| 381 | −extern void folio_add_lru(struct folio *); | |
| 382 | −extern void lru_cache_add(struct page *); | |
| 387 | +void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages); | |
| 388 | +void lru_note_cost_folio(struct folio *); | |
| 389 | +void folio_add_lru(struct folio *); | |
| 390 | +void folio_add_lru_vma(struct folio *, struct vm_area_struct *); | |
| 391 | +void lru_cache_add(struct page *); | |
| 383 | 392 | void mark_page_accessed(struct page *); |
| 384 | 393 | void folio_mark_accessed(struct folio *); |
| 385 | 394 | |
| @@ -481,7 +490,8 @@ static inline long get_nr_swap_pages(void) · 95 unmodified lines | ||
| 481 | 490 | |
| 482 | 491 | extern void si_swapinfo(struct sysinfo *); |
| 483 | 492 | swp_entry_t folio_alloc_swap(struct folio *folio); |
| 484 | −extern void put_swap_page(struct page *page, swp_entry_t entry); | |
| 493 | +bool folio_free_swap(struct folio *folio); | |
| 494 | +void put_swap_folio(struct folio *folio, swp_entry_t entry); | |
| 485 | 495 | extern swp_entry_t get_swap_page_of_type(int); |
| 486 | 496 | extern int get_swap_pages(int n, swp_entry_t swp_entries[], int entry_size); |
| 487 | 497 | extern int add_swap_count_continuation(swp_entry_t, gfp_t); |
| @@ -500,7 +510,6 @@ extern int __swp_swapcount(swp_entry_t entry); · 12 unmodified lines | ||
| 500 | 510 | extern int swp_swapcount(swp_entry_t entry); |
| 501 | 511 | extern struct swap_info_struct *page_swap_info(struct page *); |
| 502 | 512 | extern struct swap_info_struct *swp_swap_info(swp_entry_t entry); |
| 503 | −extern int try_to_free_swap(struct page *); | |
| 504 | 513 | struct backing_dev_info; |
| 505 | 514 | extern int init_swap_address_space(unsigned int type, unsigned long nr_pages); |
| 506 | 515 | extern void exit_swap_address_space(unsigned int type); |
| @@ -566,7 +575,7 @@ static inline void swap_free(swp_entry_t swp) · 59 unmodified lines | ||
| 566 | 575 | { |
| 567 | 576 | } |
| 568 | 577 | |
| 569 | −static inline void put_swap_page(struct page *page, swp_entry_t swp) | |
| 578 | +static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) | |
| 570 | 579 | { |
| 571 | 580 | } |
| 572 | 581 | |
| @@ -585,11 +594,6 @@ static inline int swp_swapcount(swp_entry_t entry) · 12 unmodified lines | ||
| 585 | 594 | return 0; |
| 586 | 595 | } |
| 587 | 596 | |
| 588 | −static inline int try_to_free_swap(struct page *page) | |
| 589 | −{ | |
| 590 | − return 0; | |
| 591 | −} | |
| 592 | − | |
| 593 | 597 | static inline swp_entry_t folio_alloc_swap(struct folio *folio) |
| 594 | 598 | { |
| 595 | 599 | swp_entry_t entry; |
| @@ -597,6 +601,11 @@ static inline swp_entry_t folio_alloc_swap(struct folio *folio) · 1 unmodified line | ||
| 597 | 601 | return entry; |
| 598 | 602 | } |
| 599 | 603 | |
| 604 | +static inline bool folio_free_swap(struct folio *folio) | |
| 605 | +{ | |
| 606 | + return false; | |
| 607 | +} | |
| 608 | + | |
| 600 | 609 | static inline int add_swap_extent(struct swap_info_struct *sis, |
| 601 | 610 | unsigned long start_page, |
| 602 | 611 | unsigned long nr_pages, sector_t start_block) |
| @@ -657,7 +666,7 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp) · 54 unmodified lines | ||
| 657 | 666 | cgroup_throttle_swaprate(&folio->page, gfp); |
| 658 | 667 | } |
| 659 | 668 | |
| 660 | −#ifdef CONFIG_MEMCG_SWAP | |
| 669 | +#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP) | |
| 661 | 670 | void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry); |
| 662 | 671 | int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry); |
| 663 | 672 | static inline int mem_cgroup_try_charge_swap(struct folio *folio, |
| @@ -677,7 +686,7 @@ static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_p · 13 unmodified lines | ||
| 677 | 686 | } |
| 678 | 687 | |
| 679 | 688 | extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg); |
| 680 | −extern bool mem_cgroup_swap_full(struct page *page); | |
| 689 | +extern bool mem_cgroup_swap_full(struct folio *folio); | |
| 681 | 690 | #else |
| 682 | 691 | static inline void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) |
| 683 | 692 | { |
| @@ -699,7 +708,7 @@ static inline long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) · 15 unmodified lines | ||
| 699 | 708 | return get_nr_swap_pages(); |
| 700 | 709 | } |
| 701 | 710 | |
| 702 | −static inline bool mem_cgroup_swap_full(struct page *page) | |
| 711 | +static inline bool mem_cgroup_swap_full(struct folio *folio) | |
| 703 | 712 | { |
| 704 | 713 | return vm_swap_full(); |
| 705 | 714 | } |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | |
| 5 | 5 | #include <linux/swap.h> |
| 6 | 6 | |
| 7 | −#ifdef CONFIG_MEMCG_SWAP | |
| 7 | +#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP) | |
| 8 | 8 | |
| 9 | 9 | extern unsigned short swap_cgroup_cmpxchg(swp_entry_t ent, |
| 10 | 10 | unsigned short old, unsigned short new); |
| @@ -40,6 +40,6 @@ static inline void swap_cgroup_swapoff(int type) · 29 unmodified lines | ||
| 40 | 40 | return; |
| 41 | 41 | } |
| 42 | 42 | |
| 43 | −#endif /* CONFIG_MEMCG_SWAP */ | |
| 43 | +#endif | |
| 44 | 44 | |
| 45 | 45 | #endif /* __LINUX_SWAP_CGROUP_H */ |
| @@ -8,6 +8,11 @@ · 7 unmodified lines | ||
| 8 | 8 | */ |
| 9 | 9 | extern struct swap_info_struct *swap_info[]; |
| 10 | 10 | extern unsigned long generic_max_swapfile_size(void); |
| 11 | −extern unsigned long max_swapfile_size(void); | |
| 11 | +unsigned long arch_max_swapfile_size(void); | |
| 12 | + | |
| 13 | +/* Maximum swapfile size supported for the arch (not inclusive). */ | |
| 14 | +extern unsigned long swapfile_maximum_size; | |
| 15 | +/* Whether swap migration entry supports storing A/D bits for the arch */ | |
| 16 | +extern bool swap_migration_ad_supported; | |
| 12 | 17 | |
| 13 | 18 | #endif /* _LINUX_SWAPFILE_H */ |
| @@ -8,6 +8,10 @@ · 7 unmodified lines | ||
| 8 | 8 | |
| 9 | 9 | #ifdef CONFIG_MMU |
| 10 | 10 | |
| 11 | +#ifdef CONFIG_SWAP | |
| 12 | +#include <linux/swapfile.h> | |
| 13 | +#endif /* CONFIG_SWAP */ | |
| 14 | + | |
| 11 | 15 | /* |
| 12 | 16 | * swapcache pages are stored in the swapper_space radix tree. We want to |
| 13 | 17 | * get good packing density in that tree, so the index should be dense in |
| @@ -23,6 +27,45 @@ · 9 unmodified lines | ||
| 23 | 27 | #define SWP_TYPE_SHIFT (BITS_PER_XA_VALUE - MAX_SWAPFILES_SHIFT) |
| 24 | 28 | #define SWP_OFFSET_MASK ((1UL << SWP_TYPE_SHIFT) - 1) |
| 25 | 29 | |
| 30 | +/* | |
| 31 | + * Definitions only for PFN swap entries (see is_pfn_swap_entry()). To | |
| 32 | + * store PFN, we only need SWP_PFN_BITS bits. Each of the pfn swap entries | |
| 33 | + * can use the extra bits to store other information besides PFN. | |
| 34 | + */ | |
| 35 | +#ifdef MAX_PHYSMEM_BITS | |
| 36 | +#define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) | |
| 37 | +#else /* MAX_PHYSMEM_BITS */ | |
| 38 | +#define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT) | |
| 39 | +#endif /* MAX_PHYSMEM_BITS */ | |
| 40 | +#define SWP_PFN_MASK (BIT(SWP_PFN_BITS) - 1) | |
| 41 | + | |
| 42 | +/** | |
| 43 | + * Migration swap entry specific bitfield definitions. Layout: | |
| 44 | + * | |
| 45 | + * |----------+--------------------| | |
| 46 | + * | swp_type | swp_offset | | |
| 47 | + * |----------+--------+-+-+-------| | |
| 48 | + * | | resv |D|A| PFN | | |
| 49 | + * |----------+--------+-+-+-------| | |
| 50 | + * | |
| 51 | + * @SWP_MIG_YOUNG_BIT: Whether the page used to have young bit set (bit A) | |
| 52 | + * @SWP_MIG_DIRTY_BIT: Whether the page used to have dirty bit set (bit D) | |
| 53 | + * | |
| 54 | + * Note: A/D bits will be stored in migration entries iff there're enough | |
| 55 | + * free bits in arch specific swp offset. By default we'll ignore A/D bits | |
| 56 | + * when migrating a page. Please refer to migration_entry_supports_ad() | |
| 57 | + * for more information. If there're more bits besides PFN and A/D bits, | |
| 58 | + * they should be reserved and always be zeros. | |
| 59 | + */ | |
| 60 | +#define SWP_MIG_YOUNG_BIT (SWP_PFN_BITS) | |
| 61 | +#define SWP_MIG_DIRTY_BIT (SWP_PFN_BITS + 1) | |
| 62 | +#define SWP_MIG_TOTAL_BITS (SWP_PFN_BITS + 2) | |
| 63 | + | |
| 64 | +#define SWP_MIG_YOUNG BIT(SWP_MIG_YOUNG_BIT) | |
| 65 | +#define SWP_MIG_DIRTY BIT(SWP_MIG_DIRTY_BIT) | |
| 66 | + | |
| 67 | +static inline bool is_pfn_swap_entry(swp_entry_t entry); | |
| 68 | + | |
| 26 | 69 | /* Clear all flags but only keep swp_entry_t related information */ |
| 27 | 70 | static inline pte_t pte_swp_clear_flags(pte_t pte) |
| 28 | 71 | { |
| @@ -64,6 +107,17 @@ static inline pgoff_t swp_offset(swp_entry_t entry) · 35 unmodified lines | ||
| 64 | 107 | return entry.val & SWP_OFFSET_MASK; |
| 65 | 108 | } |
| 66 | 109 | |
| 110 | +/* | |
| 111 | + * This should only be called upon a pfn swap entry to get the PFN stored | |
| 112 | + * in the swap entry. Please refers to is_pfn_swap_entry() for definition | |
| 113 | + * of pfn swap entry. | |
| 114 | + */ | |
| 115 | +static inline unsigned long swp_offset_pfn(swp_entry_t entry) | |
| 116 | +{ | |
| 117 | + VM_BUG_ON(!is_pfn_swap_entry(entry)); | |
| 118 | + return swp_offset(entry) & SWP_PFN_MASK; | |
| 119 | +} | |
| 120 | + | |
| 67 | 121 | /* check whether a pte points to a swap entry */ |
| 68 | 122 | static inline int is_swap_pte(pte_t pte) |
| 69 | 123 | { |
| @@ -240,6 +294,52 @@ static inline swp_entry_t make_writable_migration_entry(pgoff_t offset) · 170 unmodified lines | ||
| 240 | 294 | return swp_entry(SWP_MIGRATION_WRITE, offset); |
| 241 | 295 | } |
| 242 | 296 | |
| 297 | +/* | |
| 298 | + * Returns whether the host has large enough swap offset field to support | |
| 299 | + * carrying over pgtable A/D bits for page migrations. The result is | |
| 300 | + * pretty much arch specific. | |
| 301 | + */ | |
| 302 | +static inline bool migration_entry_supports_ad(void) | |
| 303 | +{ | |
| 304 | +#ifdef CONFIG_SWAP | |
| 305 | + return swap_migration_ad_supported; | |
| 306 | +#else /* CONFIG_SWAP */ | |
| 307 | + return false; | |
| 308 | +#endif /* CONFIG_SWAP */ | |
| 309 | +} | |
| 310 | + | |
| 311 | +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) | |
| 312 | +{ | |
| 313 | + if (migration_entry_supports_ad()) | |
| 314 | + return swp_entry(swp_type(entry), | |
| 315 | + swp_offset(entry) | SWP_MIG_YOUNG); | |
| 316 | + return entry; | |
| 317 | +} | |
| 318 | + | |
| 319 | +static inline bool is_migration_entry_young(swp_entry_t entry) | |
| 320 | +{ | |
| 321 | + if (migration_entry_supports_ad()) | |
| 322 | + return swp_offset(entry) & SWP_MIG_YOUNG; | |
| 323 | + /* Keep the old behavior of aging page after migration */ | |
| 324 | + return false; | |
| 325 | +} | |
| 326 | + | |
| 327 | +static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry) | |
| 328 | +{ | |
| 329 | + if (migration_entry_supports_ad()) | |
| 330 | + return swp_entry(swp_type(entry), | |
| 331 | + swp_offset(entry) | SWP_MIG_DIRTY); | |
| 332 | + return entry; | |
| 333 | +} | |
| 334 | + | |
| 335 | +static inline bool is_migration_entry_dirty(swp_entry_t entry) | |
| 336 | +{ | |
| 337 | + if (migration_entry_supports_ad()) | |
| 338 | + return swp_offset(entry) & SWP_MIG_DIRTY; | |
| 339 | + /* Keep the old behavior of clean page after migration */ | |
| 340 | + return false; | |
| 341 | +} | |
| 342 | + | |
| 243 | 343 | extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, |
| 244 | 344 | spinlock_t *ptl); |
| 245 | 345 | extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, |
| @@ -247,8 +347,8 @@ extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, · 1 unmodified line | ||
| 247 | 347 | #ifdef CONFIG_HUGETLB_PAGE |
| 248 | 348 | extern void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl); |
| 249 | 349 | extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); |
| 250 | −#endif | |
| 251 | −#else | |
| 350 | +#endif /* CONFIG_HUGETLB_PAGE */ | |
| 351 | +#else /* CONFIG_MIGRATION */ | |
| 252 | 352 | static inline swp_entry_t make_readable_migration_entry(pgoff_t offset) |
| 253 | 353 | { |
| 254 | 354 | return swp_entry(0, 0); |
| @@ -276,7 +376,7 @@ static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, · 21 unmodified lines | ||
| 276 | 376 | #ifdef CONFIG_HUGETLB_PAGE |
| 277 | 377 | static inline void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl) { } |
| 278 | 378 | static inline void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { } |
| 279 | −#endif | |
| 379 | +#endif /* CONFIG_HUGETLB_PAGE */ | |
| 280 | 380 | static inline int is_writable_migration_entry(swp_entry_t entry) |
| 281 | 381 | { |
| 282 | 382 | return 0; |
| @@ -286,7 +386,26 @@ static inline int is_readable_migration_entry(swp_entry_t entry) · 3 unmodified lines | ||
| 286 | 386 | return 0; |
| 287 | 387 | } |
| 288 | 388 | |
| 289 | −#endif | |
| 389 | +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) | |
| 390 | +{ | |
| 391 | + return entry; | |
| 392 | +} | |
| 393 | + | |
| 394 | +static inline bool is_migration_entry_young(swp_entry_t entry) | |
| 395 | +{ | |
| 396 | + return false; | |
| 397 | +} | |
| 398 | + | |
| 399 | +static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry) | |
| 400 | +{ | |
| 401 | + return entry; | |
| 402 | +} | |
| 403 | + | |
| 404 | +static inline bool is_migration_entry_dirty(swp_entry_t entry) | |
| 405 | +{ | |
| 406 | + return false; | |
| 407 | +} | |
| 408 | +#endif /* CONFIG_MIGRATION */ | |
| 290 | 409 | |
| 291 | 410 | typedef unsigned long pte_marker; |
| 292 | 411 | |
| @@ -369,7 +488,7 @@ static inline int pte_none_mostly(pte_t pte) · 76 unmodified lines | ||
| 369 | 488 | |
| 370 | 489 | static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry) |
| 371 | 490 | { |
| 372 | − struct page *p = pfn_to_page(swp_offset(entry)); | |
| 491 | + struct page *p = pfn_to_page(swp_offset_pfn(entry)); | |
| 373 | 492 | |
| 374 | 493 | /* |
| 375 | 494 | * Any use of migration entries may only occur while the |
| @@ -387,6 +506,9 @@ static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry) · 11 unmodified lines | ||
| 387 | 506 | */ |
| 388 | 507 | static inline bool is_pfn_swap_entry(swp_entry_t entry) |
| 389 | 508 | { |
| 509 | + /* Make sure the swp offset can always store the needed fields */ | |
| 510 | + BUILD_BUG_ON(SWP_TYPE_SHIFT < SWP_PFN_BITS); | |
| 511 | + | |
| 390 | 512 | return is_migration_entry(entry) || is_device_private_entry(entry) || |
| 391 | 513 | is_device_exclusive_entry(entry); |
| 392 | 514 | } |
| @@ -426,7 +548,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) · 33 unmodified lines | ||
| 426 | 548 | { |
| 427 | 549 | return is_swap_pmd(pmd) && is_migration_entry(pmd_to_swp_entry(pmd)); |
| 428 | 550 | } |
| 429 | −#else | |
| 551 | +#else /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ | |
| 430 | 552 | static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, |
| 431 | 553 | struct page *page) |
| 432 | 554 | { |
| @@ -455,7 +577,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) · 22 unmodified lines | ||
| 455 | 577 | { |
| 456 | 578 | return 0; |
| 457 | 579 | } |
| 458 | −#endif | |
| 580 | +#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ | |
| 459 | 581 | |
| 460 | 582 | #ifdef CONFIG_MEMORY_FAILURE |
| 461 | 583 | |
| @@ -475,27 +597,17 @@ static inline int is_hwpoison_entry(swp_entry_t entry) · 13 unmodified lines | ||
| 475 | 597 | return swp_type(entry) == SWP_HWPOISON; |
| 476 | 598 | } |
| 477 | 599 | |
| 478 | −static inline unsigned long hwpoison_entry_to_pfn(swp_entry_t entry) | |
| 479 | −{ | |
| 480 | − return swp_offset(entry); | |
| 481 | −} | |
| 482 | − | |
| 483 | 600 | static inline void num_poisoned_pages_inc(void) |
| 484 | 601 | { |
| 485 | 602 | atomic_long_inc(&num_poisoned_pages); |
| 486 | 603 | } |
| 487 | 604 | |
| 488 | −static inline void num_poisoned_pages_dec(void) | |
| 489 | −{ | |
| 490 | − atomic_long_dec(&num_poisoned_pages); | |
| 491 | −} | |
| 492 | − | |
| 493 | 605 | static inline void num_poisoned_pages_sub(long i) |
| 494 | 606 | { |
| 495 | 607 | atomic_long_sub(i, &num_poisoned_pages); |
| 496 | 608 | } |
| 497 | 609 | |
| 498 | −#else | |
| 610 | +#else /* CONFIG_MEMORY_FAILURE */ | |
| 499 | 611 | |
| 500 | 612 | static inline swp_entry_t make_hwpoison_entry(struct page *page) |
| 501 | 613 | { |
| @@ -514,7 +626,7 @@ static inline void num_poisoned_pages_inc(void) · 12 unmodified lines | ||
| 514 | 626 | static inline void num_poisoned_pages_sub(long i) |
| 515 | 627 | { |
| 516 | 628 | } |
| 517 | −#endif | |
| 629 | +#endif /* CONFIG_MEMORY_FAILURE */ | |
| 518 | 630 | |
| 519 | 631 | static inline int non_swap_entry(swp_entry_t entry) |
| 520 | 632 | { |
| @@ -58,20 +58,28 @@ · 57 unmodified lines | ||
| 58 | 58 | static __always_inline __must_check unsigned long |
| 59 | 59 | __copy_from_user_inatomic(void *to, const void __user *from, unsigned long n) |
| 60 | 60 | { |
| 61 | − instrument_copy_from_user(to, from, n); | |
| 61 | + unsigned long res; | |
| 62 | + | |
| 63 | + instrument_copy_from_user_before(to, from, n); | |
| 62 | 64 | check_object_size(to, n, false); |
| 63 | − return raw_copy_from_user(to, from, n); | |
| 65 | + res = raw_copy_from_user(to, from, n); | |
| 66 | + instrument_copy_from_user_after(to, from, n, res); | |
| 67 | + return res; | |
| 64 | 68 | } |
| 65 | 69 | |
| 66 | 70 | static __always_inline __must_check unsigned long |
| 67 | 71 | __copy_from_user(void *to, const void __user *from, unsigned long n) |
| 68 | 72 | { |
| 73 | + unsigned long res; | |
| 74 | + | |
| 69 | 75 | might_fault(); |
| 76 | + instrument_copy_from_user_before(to, from, n); | |
| 70 | 77 | if (should_fail_usercopy()) |
| 71 | 78 | return n; |
| 72 | − instrument_copy_from_user(to, from, n); | |
| 73 | 79 | check_object_size(to, n, false); |
| 74 | − return raw_copy_from_user(to, from, n); | |
| 80 | + res = raw_copy_from_user(to, from, n); | |
| 81 | + instrument_copy_from_user_after(to, from, n, res); | |
| 82 | + return res; | |
| 75 | 83 | } |
| 76 | 84 | |
| 77 | 85 | /** |
| @@ -115,8 +123,9 @@ _copy_from_user(void *to, const void __user *from, unsigned long n) · 37 unmodified lines | ||
| 115 | 123 | unsigned long res = n; |
| 116 | 124 | might_fault(); |
| 117 | 125 | if (!should_fail_usercopy() && likely(access_ok(from, n))) { |
| 118 | − instrument_copy_from_user(to, from, n); | |
| 126 | + instrument_copy_from_user_before(to, from, n); | |
| 119 | 127 | res = raw_copy_from_user(to, from, n); |
| 128 | + instrument_copy_from_user_after(to, from, n, res); | |
| 120 | 129 | } |
| 121 | 130 | if (unlikely(res)) |
| 122 | 131 | memset(to + (n - res), 0, res); |
| @@ -175,9 +175,8 @@ extern bool userfaultfd_remove(struct vm_area_struct *vma, · 174 unmodified lines | ||
| 175 | 175 | unsigned long start, |
| 176 | 176 | unsigned long end); |
| 177 | 177 | |
| 178 | −extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, | |
| 179 | − unsigned long start, unsigned long end, | |
| 180 | − struct list_head *uf); | |
| 178 | +extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start, | |
| 179 | + unsigned long end, struct list_head *uf); | |
| 181 | 180 | extern void userfaultfd_unmap_complete(struct mm_struct *mm, |
| 182 | 181 | struct list_head *uf); |
| 183 | 182 | |
| @@ -258,7 +257,7 @@ static inline bool userfaultfd_remove(struct vm_area_struct *vma, · 74 unmodified lines | ||
| 258 | 257 | return true; |
| 259 | 258 | } |
| 260 | 259 | |
| 261 | −static inline int userfaultfd_unmap_prep(struct vm_area_struct *vma, | |
| 260 | +static inline int userfaultfd_unmap_prep(struct mm_struct *mm, | |
| 262 | 261 | unsigned long start, unsigned long end, |
| 263 | 262 | struct list_head *uf) |
| 264 | 263 | { |
| @@ -129,10 +129,6 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, · 128 unmodified lines | ||
| 129 | 129 | NR_TLB_LOCAL_FLUSH_ALL, |
| 130 | 130 | NR_TLB_LOCAL_FLUSH_ONE, |
| 131 | 131 | #endif /* CONFIG_DEBUG_TLBFLUSH */ |
| 132 | −#ifdef CONFIG_DEBUG_VM_VMACACHE | |
| 133 | − VMACACHE_FIND_CALLS, | |
| 134 | − VMACACHE_FIND_HITS, | |
| 135 | −#endif | |
| 136 | 132 | #ifdef CONFIG_SWAP |
| 137 | 133 | SWAP_RA, |
| 138 | 134 | SWAP_RA_HIT, |
| @@ -1,28 +0,0 @@ | ||
| 1 | −/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | −#ifndef __LINUX_VMACACHE_H | |
| 3 | −#define __LINUX_VMACACHE_H | |
| 4 | − | |
| 5 | −#include <linux/sched.h> | |
| 6 | −#include <linux/mm.h> | |
| 7 | − | |
| 8 | −static inline void vmacache_flush(struct task_struct *tsk) | |
| 9 | −{ | |
| 10 | − memset(tsk->vmacache.vmas, 0, sizeof(tsk->vmacache.vmas)); | |
| 11 | −} | |
| 12 | − | |
| 13 | −extern void vmacache_update(unsigned long addr, struct vm_area_struct *newvma); | |
| 14 | −extern struct vm_area_struct *vmacache_find(struct mm_struct *mm, | |
| 15 | − unsigned long addr); | |
| 16 | − | |
| 17 | −#ifndef CONFIG_MMU | |
| 18 | −extern struct vm_area_struct *vmacache_find_exact(struct mm_struct *mm, | |
| 19 | − unsigned long start, | |
| 20 | − unsigned long end); | |
| 21 | −#endif | |
| 22 | − | |
| 23 | −static inline void vmacache_invalidate(struct mm_struct *mm) | |
| 24 | −{ | |
| 25 | − mm->vmacache_seqnum++; | |
| 26 | −} | |
| 27 | − | |
| 28 | −#endif /* __LINUX_VMACACHE_H */ | |
| @@ -125,12 +125,6 @@ static inline void vm_events_fold_cpu(int cpu) · 124 unmodified lines | ||
| 125 | 125 | #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) |
| 126 | 126 | #endif |
| 127 | 127 | |
| 128 | −#ifdef CONFIG_DEBUG_VM_VMACACHE | |
| 129 | −#define count_vm_vmacache_event(x) count_vm_event(x) | |
| 130 | −#else | |
| 131 | −#define count_vm_vmacache_event(x) do {} while (0) | |
| 132 | −#endif | |
| 133 | − | |
| 134 | 128 | #define __count_zid_vm_events(item, zid, delta) \ |
| 135 | 129 | __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) |
| 136 | 130 | |
| @@ -17,20 +17,12 @@ struct bio; · 16 unmodified lines | ||
| 17 | 17 | DECLARE_PER_CPU(int, dirty_throttle_leaks); |
| 18 | 18 | |
| 19 | 19 | /* |
| 20 | − * The 1/4 region under the global dirty thresh is for smooth dirty throttling: | |
| 21 | − * | |
| 22 | − * (thresh - thresh/DIRTY_FULL_SCOPE, thresh) | |
| 23 | − * | |
| 24 | − * Further beyond, all dirtier tasks will enter a loop waiting (possibly long | |
| 25 | − * time) for the dirty pages to drop, unless written enough pages. | |
| 26 | − * | |
| 27 | 20 | * The global dirty threshold is normally equal to the global dirty limit, |
| 28 | 21 | * except when the system suddenly allocates a lot of anonymous memory and |
| 29 | 22 | * knocks down the global dirty threshold quickly, in which case the global |
| 30 | 23 | * dirty limit will follow down slowly to prevent livelocking all dirtier tasks. |
| 31 | 24 | */ |
| 32 | 25 | #define DIRTY_SCOPE 8 |
| 33 | −#define DIRTY_FULL_SCOPE (DIRTY_SCOPE / 2) | |
| 34 | 26 | |
| 35 | 27 | struct backing_dev_info; |
| 36 | 28 | |
| @@ -11,11 +11,14 @@ · 10 unmodified lines | ||
| 11 | 11 | EM( SCAN_FAIL, "failed") \ |
| 12 | 12 | EM( SCAN_SUCCEED, "succeeded") \ |
| 13 | 13 | EM( SCAN_PMD_NULL, "pmd_null") \ |
| 14 | + EM( SCAN_PMD_NONE, "pmd_none") \ | |
| 15 | + EM( SCAN_PMD_MAPPED, "page_pmd_mapped") \ | |
| 14 | 16 | EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \ |
| 15 | 17 | EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ |
| 16 | 18 | EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \ |
| 17 | 19 | EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \ |
| 18 | 20 | EM( SCAN_PTE_UFFD_WP, "pte_uffd_wp") \ |
| 21 | + EM( SCAN_PTE_MAPPED_HUGEPAGE, "pte_mapped_hugepage") \ | |
| 19 | 22 | EM( SCAN_PAGE_RO, "no_writable_page") \ |
| 20 | 23 | EM( SCAN_LACK_REFERENCED_PAGE, "lack_referenced_page") \ |
| 21 | 24 | EM( SCAN_PAGE_NULL, "page_null") \ |
| @@ -166,5 +169,39 @@ TRACE_EVENT(mm_collapse_huge_page_swapin, · 144 unmodified lines | ||
| 166 | 169 | __entry->ret) |
| 167 | 170 | ); |
| 168 | 171 | |
| 172 | +TRACE_EVENT(mm_khugepaged_scan_file, | |
| 173 | + | |
| 174 | + TP_PROTO(struct mm_struct *mm, struct page *page, const char *filename, | |
| 175 | + int present, int swap, int result), | |
| 176 | + | |
| 177 | + TP_ARGS(mm, page, filename, present, swap, result), | |
| 178 | + | |
| 179 | + TP_STRUCT__entry( | |
| 180 | + __field(struct mm_struct *, mm) | |
| 181 | + __field(unsigned long, pfn) | |
| 182 | + __string(filename, filename) | |
| 183 | + __field(int, present) | |
| 184 | + __field(int, swap) | |
| 185 | + __field(int, result) | |
| 186 | + ), | |
| 187 | + | |
| 188 | + TP_fast_assign( | |
| 189 | + __entry->mm = mm; | |
| 190 | + __entry->pfn = page ? page_to_pfn(page) : -1; | |
| 191 | + __assign_str(filename, filename); | |
| 192 | + __entry->present = present; | |
| 193 | + __entry->swap = swap; | |
| 194 | + __entry->result = result; | |
| 195 | + ), | |
| 196 | + | |
| 197 | + TP_printk("mm=%p, scan_pfn=0x%lx, filename=%s, present=%d, swap=%d, result=%s", | |
| 198 | + __entry->mm, | |
| 199 | + __entry->pfn, | |
| 200 | + __get_str(filename), | |
| 201 | + __entry->present, | |
| 202 | + __entry->swap, | |
| 203 | + __print_symbolic(__entry->result, SCAN_STATUS)) | |
| 204 | +); | |
| 205 | + | |
| 169 | 206 | #endif /* __HUGE_MEMORY_H */ |
| 170 | 207 | #include <trace/define_trace.h> |
| @@ -0,0 +1,123 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +#undef TRACE_SYSTEM | |
| 3 | +#define TRACE_SYSTEM maple_tree | |
| 4 | + | |
| 5 | +#if !defined(_TRACE_MM_H) || defined(TRACE_HEADER_MULTI_READ) | |
| 6 | +#define _TRACE_MM_H | |
| 7 | + | |
| 8 | + | |
| 9 | +#include <linux/tracepoint.h> | |
| 10 | + | |
| 11 | +struct ma_state; | |
| 12 | + | |
| 13 | +TRACE_EVENT(ma_op, | |
| 14 | + | |
| 15 | + TP_PROTO(const char *fn, struct ma_state *mas), | |
| 16 | + | |
| 17 | + TP_ARGS(fn, mas), | |
| 18 | + | |
| 19 | + TP_STRUCT__entry( | |
| 20 | + __field(const char *, fn) | |
| 21 | + __field(unsigned long, min) | |
| 22 | + __field(unsigned long, max) | |
| 23 | + __field(unsigned long, index) | |
| 24 | + __field(unsigned long, last) | |
| 25 | + __field(void *, node) | |
| 26 | + ), | |
| 27 | + | |
| 28 | + TP_fast_assign( | |
| 29 | + __entry->fn = fn; | |
| 30 | + __entry->min = mas->min; | |
| 31 | + __entry->max = mas->max; | |
| 32 | + __entry->index = mas->index; | |
| 33 | + __entry->last = mas->last; | |
| 34 | + __entry->node = mas->node; | |
| 35 | + ), | |
| 36 | + | |
| 37 | + TP_printk("%s\tNode: %p (%lu %lu) range: %lu-%lu", | |
| 38 | + __entry->fn, | |
| 39 | + (void *) __entry->node, | |
| 40 | + (unsigned long) __entry->min, | |
| 41 | + (unsigned long) __entry->max, | |
| 42 | + (unsigned long) __entry->index, | |
| 43 | + (unsigned long) __entry->last | |
| 44 | + ) | |
| 45 | +) | |
| 46 | +TRACE_EVENT(ma_read, | |
| 47 | + | |
| 48 | + TP_PROTO(const char *fn, struct ma_state *mas), | |
| 49 | + | |
| 50 | + TP_ARGS(fn, mas), | |
| 51 | + | |
| 52 | + TP_STRUCT__entry( | |
| 53 | + __field(const char *, fn) | |
| 54 | + __field(unsigned long, min) | |
| 55 | + __field(unsigned long, max) | |
| 56 | + __field(unsigned long, index) | |
| 57 | + __field(unsigned long, last) | |
| 58 | + __field(void *, node) | |
| 59 | + ), | |
| 60 | + | |
| 61 | + TP_fast_assign( | |
| 62 | + __entry->fn = fn; | |
| 63 | + __entry->min = mas->min; | |
| 64 | + __entry->max = mas->max; | |
| 65 | + __entry->index = mas->index; | |
| 66 | + __entry->last = mas->last; | |
| 67 | + __entry->node = mas->node; | |
| 68 | + ), | |
| 69 | + | |
| 70 | + TP_printk("%s\tNode: %p (%lu %lu) range: %lu-%lu", | |
| 71 | + __entry->fn, | |
| 72 | + (void *) __entry->node, | |
| 73 | + (unsigned long) __entry->min, | |
| 74 | + (unsigned long) __entry->max, | |
| 75 | + (unsigned long) __entry->index, | |
| 76 | + (unsigned long) __entry->last | |
| 77 | + ) | |
| 78 | +) | |
| 79 | + | |
| 80 | +TRACE_EVENT(ma_write, | |
| 81 | + | |
| 82 | + TP_PROTO(const char *fn, struct ma_state *mas, unsigned long piv, | |
| 83 | + void *val), | |
| 84 | + | |
| 85 | + TP_ARGS(fn, mas, piv, val), | |
| 86 | + | |
| 87 | + TP_STRUCT__entry( | |
| 88 | + __field(const char *, fn) | |
| 89 | + __field(unsigned long, min) | |
| 90 | + __field(unsigned long, max) | |
| 91 | + __field(unsigned long, index) | |
| 92 | + __field(unsigned long, last) | |
| 93 | + __field(unsigned long, piv) | |
| 94 | + __field(void *, val) | |
| 95 | + __field(void *, node) | |
| 96 | + ), | |
| 97 | + | |
| 98 | + TP_fast_assign( | |
| 99 | + __entry->fn = fn; | |
| 100 | + __entry->min = mas->min; | |
| 101 | + __entry->max = mas->max; | |
| 102 | + __entry->index = mas->index; | |
| 103 | + __entry->last = mas->last; | |
| 104 | + __entry->piv = piv; | |
| 105 | + __entry->val = val; | |
| 106 | + __entry->node = mas->node; | |
| 107 | + ), | |
| 108 | + | |
| 109 | + TP_printk("%s\tNode %p (%lu %lu) range:%lu-%lu piv (%lu) val %p", | |
| 110 | + __entry->fn, | |
| 111 | + (void *) __entry->node, | |
| 112 | + (unsigned long) __entry->min, | |
| 113 | + (unsigned long) __entry->max, | |
| 114 | + (unsigned long) __entry->index, | |
| 115 | + (unsigned long) __entry->last, | |
| 116 | + (unsigned long) __entry->piv, | |
| 117 | + (void *) __entry->val | |
| 118 | + ) | |
| 119 | +) | |
| 120 | +#endif /* _TRACE_MM_H */ | |
| 121 | + | |
| 122 | +/* This part must be outside protection */ | |
| 123 | +#include <trace/define_trace.h> | |
| @@ -42,6 +42,79 @@ TRACE_EVENT(vm_unmapped_area, · 41 unmodified lines | ||
| 42 | 42 | __entry->low_limit, __entry->high_limit, __entry->align_mask, |
| 43 | 43 | __entry->align_offset) |
| 44 | 44 | ); |
| 45 | + | |
| 46 | +TRACE_EVENT(vma_mas_szero, | |
| 47 | + TP_PROTO(struct maple_tree *mt, unsigned long start, | |
| 48 | + unsigned long end), | |
| 49 | + | |
| 50 | + TP_ARGS(mt, start, end), | |
| 51 | + | |
| 52 | + TP_STRUCT__entry( | |
| 53 | + __field(struct maple_tree *, mt) | |
| 54 | + __field(unsigned long, start) | |
| 55 | + __field(unsigned long, end) | |
| 56 | + ), | |
| 57 | + | |
| 58 | + TP_fast_assign( | |
| 59 | + __entry->mt = mt; | |
| 60 | + __entry->start = start; | |
| 61 | + __entry->end = end; | |
| 62 | + ), | |
| 63 | + | |
| 64 | + TP_printk("mt_mod %p, (NULL), SNULL, %lu, %lu,", | |
| 65 | + __entry->mt, | |
| 66 | + (unsigned long) __entry->start, | |
| 67 | + (unsigned long) __entry->end | |
| 68 | + ) | |
| 69 | +); | |
| 70 | + | |
| 71 | +TRACE_EVENT(vma_store, | |
| 72 | + TP_PROTO(struct maple_tree *mt, struct vm_area_struct *vma), | |
| 73 | + | |
| 74 | + TP_ARGS(mt, vma), | |
| 75 | + | |
| 76 | + TP_STRUCT__entry( | |
| 77 | + __field(struct maple_tree *, mt) | |
| 78 | + __field(struct vm_area_struct *, vma) | |
| 79 | + __field(unsigned long, vm_start) | |
| 80 | + __field(unsigned long, vm_end) | |
| 81 | + ), | |
| 82 | + | |
| 83 | + TP_fast_assign( | |
| 84 | + __entry->mt = mt; | |
| 85 | + __entry->vma = vma; | |
| 86 | + __entry->vm_start = vma->vm_start; | |
| 87 | + __entry->vm_end = vma->vm_end - 1; | |
| 88 | + ), | |
| 89 | + | |
| 90 | + TP_printk("mt_mod %p, (%p), STORE, %lu, %lu,", | |
| 91 | + __entry->mt, __entry->vma, | |
| 92 | + (unsigned long) __entry->vm_start, | |
| 93 | + (unsigned long) __entry->vm_end | |
| 94 | + ) | |
| 95 | +); | |
| 96 | + | |
| 97 | + | |
| 98 | +TRACE_EVENT(exit_mmap, | |
| 99 | + TP_PROTO(struct mm_struct *mm), | |
| 100 | + | |
| 101 | + TP_ARGS(mm), | |
| 102 | + | |
| 103 | + TP_STRUCT__entry( | |
| 104 | + __field(struct mm_struct *, mm) | |
| 105 | + __field(struct maple_tree *, mt) | |
| 106 | + ), | |
| 107 | + | |
| 108 | + TP_fast_assign( | |
| 109 | + __entry->mm = mm; | |
| 110 | + __entry->mt = &mm->mm_mt; | |
| 111 | + ), | |
| 112 | + | |
| 113 | + TP_printk("mt_mod %p, DESTROY\n", | |
| 114 | + __entry->mt | |
| 115 | + ) | |
| 116 | +); | |
| 117 | + | |
| 45 | 118 | #endif |
| 46 | 119 | |
| 47 | 120 | /* This part must be outside protection */ |
| @@ -20,18 +20,18 @@ · 19 unmodified lines | ||
| 20 | 20 | #define HUGETLB_FLAG_ENCODE_SHIFT 26 |
| 21 | 21 | #define HUGETLB_FLAG_ENCODE_MASK 0x3f |
| 22 | 22 | |
| 23 | −#define HUGETLB_FLAG_ENCODE_16KB (14 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 24 | −#define HUGETLB_FLAG_ENCODE_64KB (16 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 25 | −#define HUGETLB_FLAG_ENCODE_512KB (19 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 26 | −#define HUGETLB_FLAG_ENCODE_1MB (20 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 27 | −#define HUGETLB_FLAG_ENCODE_2MB (21 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 28 | −#define HUGETLB_FLAG_ENCODE_8MB (23 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 29 | −#define HUGETLB_FLAG_ENCODE_16MB (24 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 30 | −#define HUGETLB_FLAG_ENCODE_32MB (25 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 31 | −#define HUGETLB_FLAG_ENCODE_256MB (28 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 32 | −#define HUGETLB_FLAG_ENCODE_512MB (29 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 33 | −#define HUGETLB_FLAG_ENCODE_1GB (30 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 34 | −#define HUGETLB_FLAG_ENCODE_2GB (31 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 35 | −#define HUGETLB_FLAG_ENCODE_16GB (34 << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 23 | +#define HUGETLB_FLAG_ENCODE_16KB (14U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 24 | +#define HUGETLB_FLAG_ENCODE_64KB (16U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 25 | +#define HUGETLB_FLAG_ENCODE_512KB (19U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 26 | +#define HUGETLB_FLAG_ENCODE_1MB (20U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 27 | +#define HUGETLB_FLAG_ENCODE_2MB (21U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 28 | +#define HUGETLB_FLAG_ENCODE_8MB (23U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 29 | +#define HUGETLB_FLAG_ENCODE_16MB (24U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 30 | +#define HUGETLB_FLAG_ENCODE_32MB (25U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 31 | +#define HUGETLB_FLAG_ENCODE_256MB (28U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 32 | +#define HUGETLB_FLAG_ENCODE_512MB (29U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 33 | +#define HUGETLB_FLAG_ENCODE_1GB (30U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 34 | +#define HUGETLB_FLAG_ENCODE_2GB (31U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 35 | +#define HUGETLB_FLAG_ENCODE_16GB (34U << HUGETLB_FLAG_ENCODE_SHIFT) | |
| 36 | 36 | |
| 37 | 37 | #endif /* _ASM_GENERIC_HUGETLB_ENCODE_H_ */ |
| @@ -77,6 +77,8 @@ · 76 unmodified lines | ||
| 77 | 77 | |
| 78 | 78 | #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ |
| 79 | 79 | |
| 80 | +#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */ | |
| 81 | + | |
| 80 | 82 | /* compatibility flags */ |
| 81 | 83 | #define MAP_FILE 0 |
| 82 | 84 | |
| @@ -12,6 +12,10 @@ · 11 unmodified lines | ||
| 12 | 12 | |
| 13 | 13 | #include <linux/types.h> |
| 14 | 14 | |
| 15 | +/* ioctls for /dev/userfaultfd */ | |
| 16 | +#define USERFAULTFD_IOC 0xAA | |
| 17 | +#define USERFAULTFD_IOC_NEW _IO(USERFAULTFD_IOC, 0x00) | |
| 18 | + | |
| 15 | 19 | /* |
| 16 | 20 | * If the UFFDIO_API is upgraded someday, the UFFDIO_UNREGISTER and |
| 17 | 21 | * UFFDIO_WAKE ioctls should be defined as _IOW and not as _IOR. In |
| @@ -969,11 +969,6 @@ config MEMCG · 968 unmodified lines | ||
| 969 | 969 | help |
| 970 | 970 | Provides control over the memory footprint of tasks in a cgroup. |
| 971 | 971 | |
| 972 | −config MEMCG_SWAP | |
| 973 | − bool | |
| 974 | − depends on MEMCG && SWAP | |
| 975 | − default y | |
| 976 | − | |
| 977 | 972 | config MEMCG_KMEM |
| 978 | 973 | bool |
| 979 | 974 | depends on MEMCG && !SLOB |
| @@ -34,6 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <linux/percpu.h> |
| 35 | 35 | #include <linux/kmod.h> |
| 36 | 36 | #include <linux/kprobes.h> |
| 37 | +#include <linux/kmsan.h> | |
| 37 | 38 | #include <linux/vmalloc.h> |
| 38 | 39 | #include <linux/kernel_stat.h> |
| 39 | 40 | #include <linux/start_kernel.h> |
| @@ -117,6 +118,7 @@ static int kernel_init(void *); · 77 unmodified lines | ||
| 117 | 118 | |
| 118 | 119 | extern void init_IRQ(void); |
| 119 | 120 | extern void radix_tree_init(void); |
| 121 | +extern void maple_tree_init(void); | |
| 120 | 122 | |
| 121 | 123 | /* |
| 122 | 124 | * Debug helper: via this flag we know that we are in 'early bootup code' |
| @@ -836,6 +838,7 @@ static void __init mm_init(void) · 713 unmodified lines | ||
| 836 | 838 | init_mem_debugging_and_hardening(); |
| 837 | 839 | kfence_alloc_pool(); |
| 838 | 840 | report_meminit(); |
| 841 | + kmsan_init_shadow(); | |
| 839 | 842 | stack_depot_early_init(); |
| 840 | 843 | mem_init(); |
| 841 | 844 | mem_init_print_info(); |
| @@ -849,10 +852,14 @@ static void __init mm_init(void) · 7 unmodified lines | ||
| 849 | 852 | pgtable_init(); |
| 850 | 853 | debug_objects_mem_init(); |
| 851 | 854 | vmalloc_init(); |
| 855 | + /* Should be run after vmap initialization */ | |
| 856 | + if (early_page_ext_enabled()) | |
| 857 | + page_ext_init(); | |
| 852 | 858 | /* Should be run before the first non-init thread is created */ |
| 853 | 859 | init_espfix_bsp(); |
| 854 | 860 | /* Should be run after espfix64 is set up. */ |
| 855 | 861 | pti_init(); |
| 862 | + kmsan_init_runtime(); | |
| 856 | 863 | } |
| 857 | 864 | |
| 858 | 865 | #ifdef CONFIG_RANDOMIZE_KSTACK_OFFSET |
| @@ -1005,6 +1012,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void) · 146 unmodified lines | ||
| 1005 | 1012 | "Interrupts were enabled *very* early, fixing it\n")) |
| 1006 | 1013 | local_irq_disable(); |
| 1007 | 1014 | radix_tree_init(); |
| 1015 | + maple_tree_init(); | |
| 1008 | 1016 | |
| 1009 | 1017 | /* |
| 1010 | 1018 | * Set up housekeeping before setting up workqueues to allow the unbound |
| @@ -1617,7 +1625,8 @@ static noinline void __init kernel_init_freeable(void) · 606 unmodified lines | ||
| 1617 | 1625 | padata_init(); |
| 1618 | 1626 | page_alloc_init_late(); |
| 1619 | 1627 | /* Initialize page ext after all struct pages are initialized. */ |
| 1620 | − page_ext_init(); | |
| 1628 | + if (!early_page_ext_enabled()) | |
| 1629 | + page_ext_init(); | |
| 1621 | 1630 | |
| 1622 | 1631 | do_basic_setup(); |
| 1623 | 1632 | |
| @@ -1721,7 +1721,7 @@ long ksys_shmdt(char __user *shmaddr) · 1720 unmodified lines | ||
| 1721 | 1721 | #ifdef CONFIG_MMU |
| 1722 | 1722 | loff_t size = 0; |
| 1723 | 1723 | struct file *file; |
| 1724 | − struct vm_area_struct *next; | |
| 1724 | + VMA_ITERATOR(vmi, mm, addr); | |
| 1725 | 1725 | #endif |
| 1726 | 1726 | |
| 1727 | 1727 | if (addr & ~PAGE_MASK) |
| @@ -1751,12 +1751,9 @@ long ksys_shmdt(char __user *shmaddr) · 23 unmodified lines | ||
| 1751 | 1751 | * match the usual checks anyway. So assume all vma's are |
| 1752 | 1752 | * above the starting address given. |
| 1753 | 1753 | */ |
| 1754 | − vma = find_vma(mm, addr); | |
| 1755 | 1754 | |
| 1756 | 1755 | #ifdef CONFIG_MMU |
| 1757 | − while (vma) { | |
| 1758 | − next = vma->vm_next; | |
| 1759 | − | |
| 1756 | + for_each_vma(vmi, vma) { | |
| 1760 | 1757 | /* |
| 1761 | 1758 | * Check if the starting address would match, i.e. it's |
| 1762 | 1759 | * a fragment created by mprotect() and/or munmap(), or it |
| @@ -1774,6 +1771,7 @@ long ksys_shmdt(char __user *shmaddr) · 11 unmodified lines | ||
| 1774 | 1771 | file = vma->vm_file; |
| 1775 | 1772 | size = i_size_read(file_inode(vma->vm_file)); |
| 1776 | 1773 | do_munmap(mm, vma->vm_start, vma->vm_end - vma->vm_start, NULL); |
| 1774 | + mas_pause(&vmi.mas); | |
| 1777 | 1775 | /* |
| 1778 | 1776 | * We discovered the size of the shm segment, so |
| 1779 | 1777 | * break out of here and fall through to the next |
| @@ -1781,10 +1779,9 @@ long ksys_shmdt(char __user *shmaddr) · 1 unmodified line | ||
| 1781 | 1779 | * searching for matching vma's. |
| 1782 | 1780 | */ |
| 1783 | 1781 | retval = 0; |
| 1784 | − vma = next; | |
| 1782 | + vma = vma_next(&vmi); | |
| 1785 | 1783 | break; |
| 1786 | 1784 | } |
| 1787 | − vma = next; | |
| 1788 | 1785 | } |
| 1789 | 1786 | |
| 1790 | 1787 | /* |
| @@ -1794,17 +1791,19 @@ long ksys_shmdt(char __user *shmaddr) · 3 unmodified lines | ||
| 1794 | 1791 | */ |
| 1795 | 1792 | size = PAGE_ALIGN(size); |
| 1796 | 1793 | while (vma && (loff_t)(vma->vm_end - addr) <= size) { |
| 1797 | − next = vma->vm_next; | |
| 1798 | − | |
| 1799 | 1794 | /* finding a matching vma now does not alter retval */ |
| 1800 | 1795 | if ((vma->vm_ops == &shm_vm_ops) && |
| 1801 | 1796 | ((vma->vm_start - addr)/PAGE_SIZE == vma->vm_pgoff) && |
| 1802 | − (vma->vm_file == file)) | |
| 1797 | + (vma->vm_file == file)) { | |
| 1803 | 1798 | do_munmap(mm, vma->vm_start, vma->vm_end - vma->vm_start, NULL); |
| 1804 | − vma = next; | |
| 1799 | + mas_pause(&vmi.mas); | |
| 1800 | + } | |
| 1801 | + | |
| 1802 | + vma = vma_next(&vmi); | |
| 1805 | 1803 | } |
| 1806 | 1804 | |
| 1807 | 1805 | #else /* CONFIG_MMU */ |
| 1806 | + vma = vma_lookup(mm, addr); | |
| 1808 | 1807 | /* under NOMMU conditions, the exact address to be destroyed must be |
| 1809 | 1808 | * given |
| 1810 | 1809 | */ |
| @@ -38,6 +38,7 @@ KCOV_INSTRUMENT_kcov.o := n · 37 unmodified lines | ||
| 38 | 38 | KASAN_SANITIZE_kcov.o := n |
| 39 | 39 | KCSAN_SANITIZE_kcov.o := n |
| 40 | 40 | UBSAN_SANITIZE_kcov.o := n |
| 41 | +KMSAN_SANITIZE_kcov.o := n | |
| 41 | 42 | CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack) -fno-stack-protector |
| 42 | 43 | |
| 43 | 44 | # Don't instrument error handlers |
| @@ -555,15 +555,14 @@ void acct_collect(long exitcode, int group_dead) · 554 unmodified lines | ||
| 555 | 555 | unsigned long vsize = 0; |
| 556 | 556 | |
| 557 | 557 | if (group_dead && current->mm) { |
| 558 | + struct mm_struct *mm = current->mm; | |
| 559 | + VMA_ITERATOR(vmi, mm, 0); | |
| 558 | 560 | struct vm_area_struct *vma; |
| 559 | 561 | |
| 560 | − mmap_read_lock(current->mm); | |
| 561 | − vma = current->mm->mmap; | |
| 562 | − while (vma) { | |
| 562 | + mmap_read_lock(mm); | |
| 563 | + for_each_vma(vmi, vma) | |
| 563 | 564 | vsize += vma->vm_end - vma->vm_start; |
| 564 | − vma = vma->vm_next; | |
| 565 | − } | |
| 566 | − mmap_read_unlock(current->mm); | |
| 565 | + mmap_read_unlock(mm); | |
| 567 | 566 | } |
| 568 | 567 | |
| 569 | 568 | spin_lock_irq(¤t->sighand->siglock); |
| @@ -22,6 +22,13 @@ int main(void) · 21 unmodified lines | ||
| 22 | 22 | DEFINE(NR_CPUS_BITS, ilog2(CONFIG_NR_CPUS)); |
| 23 | 23 | #endif |
| 24 | 24 | DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t)); |
| 25 | +#ifdef CONFIG_LRU_GEN | |
| 26 | + DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1)); | |
| 27 | + DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 2); | |
| 28 | +#else | |
| 29 | + DEFINE(LRU_GEN_WIDTH, 0); | |
| 30 | + DEFINE(__LRU_REFS_WIDTH, 0); | |
| 31 | +#endif | |
| 25 | 32 | /* End of constants */ |
| 26 | 33 | |
| 27 | 34 | return 0; |
| @@ -2007,7 +2007,7 @@ out: · 2006 unmodified lines | ||
| 2007 | 2007 | static unsigned int PROG_NAME(stack_size)(const void *ctx, const struct bpf_insn *insn) \ |
| 2008 | 2008 | { \ |
| 2009 | 2009 | u64 stack[stack_size / sizeof(u64)]; \ |
| 2010 | − u64 regs[MAX_BPF_EXT_REG]; \ | |
| 2010 | + u64 regs[MAX_BPF_EXT_REG] = {}; \ | |
| 2011 | 2011 | \ |
| 2012 | 2012 | FP = (u64) (unsigned long) &stack[ARRAY_SIZE(stack)]; \ |
| 2013 | 2013 | ARG1 = (u64) (unsigned long) ctx; \ |
| @@ -445,8 +445,8 @@ struct bpf_iter_seq_task_vma_info { · 444 unmodified lines | ||
| 445 | 445 | }; |
| 446 | 446 | |
| 447 | 447 | enum bpf_task_vma_iter_find_op { |
| 448 | − task_vma_iter_first_vma, /* use mm->mmap */ | |
| 449 | − task_vma_iter_next_vma, /* use curr_vma->vm_next */ | |
| 448 | + task_vma_iter_first_vma, /* use find_vma() with addr 0 */ | |
| 449 | + task_vma_iter_next_vma, /* use vma_next() with curr_vma */ | |
| 450 | 450 | task_vma_iter_find_vma, /* use find_vma() to find next vma */ |
| 451 | 451 | }; |
| 452 | 452 | |
| @@ -544,10 +544,10 @@ again: · 91 unmodified lines | ||
| 544 | 544 | |
| 545 | 545 | switch (op) { |
| 546 | 546 | case task_vma_iter_first_vma: |
| 547 | − curr_vma = curr_task->mm->mmap; | |
| 547 | + curr_vma = find_vma(curr_task->mm, 0); | |
| 548 | 548 | break; |
| 549 | 549 | case task_vma_iter_next_vma: |
| 550 | − curr_vma = curr_vma->vm_next; | |
| 550 | + curr_vma = find_vma(curr_task->mm, curr_vma->vm_end); | |
| 551 | 551 | break; |
| 552 | 552 | case task_vma_iter_find_vma: |
| 553 | 553 | /* We dropped mmap_lock so it is necessary to use find_vma |
| @@ -561,7 +561,7 @@ again: · 7 unmodified lines | ||
| 561 | 561 | if (curr_vma && |
| 562 | 562 | curr_vma->vm_start == info->prev_vm_start && |
| 563 | 563 | curr_vma->vm_end == info->prev_vm_end) |
| 564 | − curr_vma = curr_vma->vm_next; | |
| 564 | + curr_vma = find_vma(curr_task->mm, curr_vma->vm_end); | |
| 565 | 565 | break; |
| 566 | 566 | } |
| 567 | 567 | if (!curr_vma) { |
| @@ -164,7 +164,6 @@ struct cgroup_mgctx { · 163 unmodified lines | ||
| 164 | 164 | #define DEFINE_CGROUP_MGCTX(name) \ |
| 165 | 165 | struct cgroup_mgctx name = CGROUP_MGCTX_INIT(name) |
| 166 | 166 | |
| 167 | −extern struct mutex cgroup_mutex; | |
| 168 | 167 | extern spinlock_t css_set_lock; |
| 169 | 168 | extern struct cgroup_subsys *cgroup_subsys[]; |
| 170 | 169 | extern struct list_head cgroup_roots; |
| @@ -50,7 +50,6 @@ · 49 unmodified lines | ||
| 50 | 50 | #include <linux/pid.h> |
| 51 | 51 | #include <linux/smp.h> |
| 52 | 52 | #include <linux/mm.h> |
| 53 | −#include <linux/vmacache.h> | |
| 54 | 53 | #include <linux/rcupdate.h> |
| 55 | 54 | #include <linux/irq.h> |
| 56 | 55 | #include <linux/security.h> |
| @@ -283,17 +282,6 @@ static void kgdb_flush_swbreak_addr(unsigned long addr) · 226 unmodified lines | ||
| 283 | 282 | if (!CACHE_FLUSH_IS_SAFE) |
| 284 | 283 | return; |
| 285 | 284 | |
| 286 | − if (current->mm) { | |
| 287 | − int i; | |
| 288 | − | |
| 289 | − for (i = 0; i < VMACACHE_SIZE; i++) { | |
| 290 | − if (!current->vmacache.vmas[i]) | |
| 291 | − continue; | |
| 292 | − flush_cache_range(current->vmacache.vmas[i], | |
| 293 | − addr, addr + BREAK_INSTR_SIZE); | |
| 294 | − } | |
| 295 | − } | |
| 296 | − | |
| 297 | 285 | /* Force flush instruction cache if it was outside the mm */ |
| 298 | 286 | flush_icache_range(addr, addr + BREAK_INSTR_SIZE); |
| 299 | 287 | } |
| @@ -214,13 +214,22 @@ void __delayacct_freepages_end(void) · 213 unmodified lines | ||
| 214 | 214 | ¤t->delays->freepages_count); |
| 215 | 215 | } |
| 216 | 216 | |
| 217 | −void __delayacct_thrashing_start(void) | |
| 217 | +void __delayacct_thrashing_start(bool *in_thrashing) | |
| 218 | 218 | { |
| 219 | + *in_thrashing = !!current->in_thrashing; | |
| 220 | + if (*in_thrashing) | |
| 221 | + return; | |
| 222 | + | |
| 223 | + current->in_thrashing = 1; | |
| 219 | 224 | current->delays->thrashing_start = local_clock(); |
| 220 | 225 | } |
| 221 | 226 | |
| 222 | −void __delayacct_thrashing_end(void) | |
| 227 | +void __delayacct_thrashing_end(bool *in_thrashing) | |
| 223 | 228 | { |
| 229 | + if (*in_thrashing) | |
| 230 | + return; | |
| 231 | + | |
| 232 | + current->in_thrashing = 0; | |
| 224 | 233 | delayacct_end(¤t->delays->lock, |
| 225 | 234 | ¤t->delays->thrashing_start, |
| 226 | 235 | ¤t->delays->thrashing_delay, |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/dma-map-ops.h> |
| 11 | 11 | #include <linux/export.h> |
| 12 | 12 | #include <linux/gfp.h> |
| 13 | +#include <linux/kmsan.h> | |
| 13 | 14 | #include <linux/of_device.h> |
| 14 | 15 | #include <linux/slab.h> |
| 15 | 16 | #include <linux/vmalloc.h> |
| @@ -156,6 +157,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, · 140 unmodified lines | ||
| 156 | 157 | addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); |
| 157 | 158 | else |
| 158 | 159 | addr = ops->map_page(dev, page, offset, size, dir, attrs); |
| 160 | + kmsan_handle_dma(page, offset, size, dir); | |
| 159 | 161 | debug_dma_map_page(dev, page, offset, size, dir, addr, attrs); |
| 160 | 162 | |
| 161 | 163 | return addr; |
| @@ -194,11 +196,13 @@ static int __dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, · 32 unmodified lines | ||
| 194 | 196 | else |
| 195 | 197 | ents = ops->map_sg(dev, sg, nents, dir, attrs); |
| 196 | 198 | |
| 197 | − if (ents > 0) | |
| 199 | + if (ents > 0) { | |
| 200 | + kmsan_handle_dma_sg(sg, nents, dir); | |
| 198 | 201 | debug_dma_map_sg(dev, sg, nents, ents, dir, attrs); |
| 199 | − else if (WARN_ON_ONCE(ents != -EINVAL && ents != -ENOMEM && | |
| 200 | − ents != -EIO && ents != -EREMOTEIO)) | |
| 202 | + } else if (WARN_ON_ONCE(ents != -EINVAL && ents != -ENOMEM && | |
| 203 | + ents != -EIO && ents != -EREMOTEIO)) { | |
| 201 | 204 | return -EIO; |
| 205 | + } | |
| 202 | 206 | |
| 203 | 207 | return ents; |
| 204 | 208 | } |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <linux/resume_user_mode.h> |
| 6 | 6 | #include <linux/highmem.h> |
| 7 | 7 | #include <linux/jump_label.h> |
| 8 | +#include <linux/kmsan.h> | |
| 8 | 9 | #include <linux/livepatch.h> |
| 9 | 10 | #include <linux/audit.h> |
| 10 | 11 | #include <linux/tick.h> |
| @@ -24,6 +25,7 @@ static __always_inline void __enter_from_user_mode(struct pt_regs *regs) · 13 unmodified lines | ||
| 24 | 25 | user_exit_irqoff(); |
| 25 | 26 | |
| 26 | 27 | instrumentation_begin(); |
| 28 | + kmsan_unpoison_entry_regs(regs); | |
| 27 | 29 | trace_hardirqs_off_finish(); |
| 28 | 30 | instrumentation_end(); |
| 29 | 31 | } |
| @@ -352,6 +354,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs) · 322 unmodified lines | ||
| 352 | 354 | lockdep_hardirqs_off(CALLER_ADDR0); |
| 353 | 355 | ct_irq_enter(); |
| 354 | 356 | instrumentation_begin(); |
| 357 | + kmsan_unpoison_entry_regs(regs); | |
| 355 | 358 | trace_hardirqs_off_finish(); |
| 356 | 359 | instrumentation_end(); |
| 357 | 360 | |
| @@ -367,6 +370,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs) · 9 unmodified lines | ||
| 367 | 370 | */ |
| 368 | 371 | lockdep_hardirqs_off(CALLER_ADDR0); |
| 369 | 372 | instrumentation_begin(); |
| 373 | + kmsan_unpoison_entry_regs(regs); | |
| 370 | 374 | rcu_irq_enter_check_tick(); |
| 371 | 375 | trace_hardirqs_off_finish(); |
| 372 | 376 | instrumentation_end(); |
| @@ -452,6 +456,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs) · 79 unmodified lines | ||
| 452 | 456 | ct_nmi_enter(); |
| 453 | 457 | |
| 454 | 458 | instrumentation_begin(); |
| 459 | + kmsan_unpoison_entry_regs(regs); | |
| 455 | 460 | trace_hardirqs_off_finish(); |
| 456 | 461 | ftrace_nmi_enter(); |
| 457 | 462 | instrumentation_end(); |
| @@ -10270,8 +10270,9 @@ static void perf_addr_filter_apply(struct perf_addr_filter *filter, · 10269 unmodified lines | ||
| 10270 | 10270 | struct perf_addr_filter_range *fr) |
| 10271 | 10271 | { |
| 10272 | 10272 | struct vm_area_struct *vma; |
| 10273 | + VMA_ITERATOR(vmi, mm, 0); | |
| 10273 | 10274 | |
| 10274 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 10275 | + for_each_vma(vmi, vma) { | |
| 10275 | 10276 | if (!vma->vm_file) |
| 10276 | 10277 | continue; |
| 10277 | 10278 | |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <linux/export.h> |
| 20 | 20 | #include <linux/rmap.h> /* anon_vma_prepare */ |
| 21 | 21 | #include <linux/mmu_notifier.h> /* set_pte_at_notify */ |
| 22 | −#include <linux/swap.h> /* try_to_free_swap */ | |
| 22 | +#include <linux/swap.h> /* folio_free_swap */ | |
| 23 | 23 | #include <linux/ptrace.h> /* user_enable_single_step */ |
| 24 | 24 | #include <linux/kdebug.h> /* notifier mechanism */ |
| 25 | 25 | #include "../../mm/internal.h" /* munlock_vma_page */ |
| @@ -154,8 +154,10 @@ static loff_t vaddr_to_offset(struct vm_area_struct *vma, unsigned long vaddr) · 128 unmodified lines | ||
| 154 | 154 | static int __replace_page(struct vm_area_struct *vma, unsigned long addr, |
| 155 | 155 | struct page *old_page, struct page *new_page) |
| 156 | 156 | { |
| 157 | + struct folio *old_folio = page_folio(old_page); | |
| 158 | + struct folio *new_folio; | |
| 157 | 159 | struct mm_struct *mm = vma->vm_mm; |
| 158 | − DEFINE_FOLIO_VMA_WALK(pvmw, page_folio(old_page), vma, addr, 0); | |
| 160 | + DEFINE_FOLIO_VMA_WALK(pvmw, old_folio, vma, addr, 0); | |
| 159 | 161 | int err; |
| 160 | 162 | struct mmu_notifier_range range; |
| 161 | 163 | |
| @@ -163,14 +165,14 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, · 1 unmodified line | ||
| 163 | 165 | addr + PAGE_SIZE); |
| 164 | 166 | |
| 165 | 167 | if (new_page) { |
| 166 | − err = mem_cgroup_charge(page_folio(new_page), vma->vm_mm, | |
| 167 | − GFP_KERNEL); | |
| 168 | + new_folio = page_folio(new_page); | |
| 169 | + err = mem_cgroup_charge(new_folio, vma->vm_mm, GFP_KERNEL); | |
| 168 | 170 | if (err) |
| 169 | 171 | return err; |
| 170 | 172 | } |
| 171 | 173 | |
| 172 | − /* For try_to_free_swap() below */ | |
| 173 | − lock_page(old_page); | |
| 174 | + /* For folio_free_swap() below */ | |
| 175 | + folio_lock(old_folio); | |
| 174 | 176 | |
| 175 | 177 | mmu_notifier_invalidate_range_start(&range); |
| 176 | 178 | err = -EAGAIN; |
| @@ -179,14 +181,14 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, · 2 unmodified lines | ||
| 179 | 181 | VM_BUG_ON_PAGE(addr != pvmw.address, old_page); |
| 180 | 182 | |
| 181 | 183 | if (new_page) { |
| 182 | − get_page(new_page); | |
| 184 | + folio_get(new_folio); | |
| 183 | 185 | page_add_new_anon_rmap(new_page, vma, addr); |
| 184 | − lru_cache_add_inactive_or_unevictable(new_page, vma); | |
| 186 | + folio_add_lru_vma(new_folio, vma); | |
| 185 | 187 | } else |
| 186 | 188 | /* no new page, just dec_mm_counter for old_page */ |
| 187 | 189 | dec_mm_counter(mm, MM_ANONPAGES); |
| 188 | 190 | |
| 189 | − if (!PageAnon(old_page)) { | |
| 191 | + if (!folio_test_anon(old_folio)) { | |
| 190 | 192 | dec_mm_counter(mm, mm_counter_file(old_page)); |
| 191 | 193 | inc_mm_counter(mm, MM_ANONPAGES); |
| 192 | 194 | } |
| @@ -198,15 +200,15 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, · 5 unmodified lines | ||
| 198 | 200 | mk_pte(new_page, vma->vm_page_prot)); |
| 199 | 201 | |
| 200 | 202 | page_remove_rmap(old_page, vma, false); |
| 201 | − if (!page_mapped(old_page)) | |
| 202 | − try_to_free_swap(old_page); | |
| 203 | + if (!folio_mapped(old_folio)) | |
| 204 | + folio_free_swap(old_folio); | |
| 203 | 205 | page_vma_mapped_walk_done(&pvmw); |
| 204 | − put_page(old_page); | |
| 206 | + folio_put(old_folio); | |
| 205 | 207 | |
| 206 | 208 | err = 0; |
| 207 | 209 | unlock: |
| 208 | 210 | mmu_notifier_invalidate_range_end(&range); |
| 209 | − unlock_page(old_page); | |
| 211 | + folio_unlock(old_folio); | |
| 210 | 212 | return err; |
| 211 | 213 | } |
| 212 | 214 | |
| @@ -349,9 +351,10 @@ static bool valid_ref_ctr_vma(struct uprobe *uprobe, · 136 unmodified lines | ||
| 349 | 351 | static struct vm_area_struct * |
| 350 | 352 | find_ref_ctr_vma(struct uprobe *uprobe, struct mm_struct *mm) |
| 351 | 353 | { |
| 354 | + VMA_ITERATOR(vmi, mm, 0); | |
| 352 | 355 | struct vm_area_struct *tmp; |
| 353 | 356 | |
| 354 | − for (tmp = mm->mmap; tmp; tmp = tmp->vm_next) | |
| 357 | + for_each_vma(vmi, tmp) | |
| 355 | 358 | if (valid_ref_ctr_vma(uprobe, tmp)) |
| 356 | 359 | return tmp; |
| 357 | 360 | |
| @@ -552,7 +555,7 @@ put_old: · 194 unmodified lines | ||
| 552 | 555 | |
| 553 | 556 | /* try collapse pmd for compound page */ |
| 554 | 557 | if (!ret && orig_page_huge) |
| 555 | − collapse_pte_mapped_thp(mm, vaddr); | |
| 558 | + collapse_pte_mapped_thp(mm, vaddr, false); | |
| 556 | 559 | |
| 557 | 560 | return ret; |
| 558 | 561 | } |
| @@ -1231,11 +1234,12 @@ int uprobe_apply(struct inode *inode, loff_t offset, · 672 unmodified lines | ||
| 1231 | 1234 | |
| 1232 | 1235 | static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm) |
| 1233 | 1236 | { |
| 1237 | + VMA_ITERATOR(vmi, mm, 0); | |
| 1234 | 1238 | struct vm_area_struct *vma; |
| 1235 | 1239 | int err = 0; |
| 1236 | 1240 | |
| 1237 | 1241 | mmap_read_lock(mm); |
| 1238 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 1242 | + for_each_vma(vmi, vma) { | |
| 1239 | 1243 | unsigned long vaddr; |
| 1240 | 1244 | loff_t offset; |
| 1241 | 1245 | |
| @@ -1983,9 +1987,10 @@ bool uprobe_deny_signal(void) · 741 unmodified lines | ||
| 1983 | 1987 | |
| 1984 | 1988 | static void mmf_recalc_uprobes(struct mm_struct *mm) |
| 1985 | 1989 | { |
| 1990 | + VMA_ITERATOR(vmi, mm, 0); | |
| 1986 | 1991 | struct vm_area_struct *vma; |
| 1987 | 1992 | |
| 1988 | − for (vma = mm->mmap; vma; vma = vma->vm_next) { | |
| 1993 | + for_each_vma(vmi, vma) { | |
| 1989 | 1994 | if (!valid_vma(vma, false)) |
| 1990 | 1995 | continue; |
| 1991 | 1996 | /* |
| @@ -60,6 +60,7 @@ · 59 unmodified lines | ||
| 60 | 60 | #include <linux/writeback.h> |
| 61 | 61 | #include <linux/shm.h> |
| 62 | 62 | #include <linux/kcov.h> |
| 63 | +#include <linux/kmsan.h> | |
| 63 | 64 | #include <linux/random.h> |
| 64 | 65 | #include <linux/rcuwait.h> |
| 65 | 66 | #include <linux/compat.h> |
| @@ -466,6 +467,7 @@ assign_new_owner: · 400 unmodified lines | ||
| 466 | 467 | goto retry; |
| 467 | 468 | } |
| 468 | 469 | WRITE_ONCE(mm->owner, c); |
| 470 | + lru_gen_migrate_mm(mm); | |
| 469 | 471 | task_unlock(c); |
| 470 | 472 | put_task_struct(c); |
| 471 | 473 | } |
| @@ -759,6 +761,7 @@ void __noreturn do_exit(long code) · 287 unmodified lines | ||
| 759 | 761 | WARN_ON(tsk->plug); |
| 760 | 762 | |
| 761 | 763 | kcov_task_exit(tsk); |
| 764 | + kmsan_task_exit(tsk); | |
| 762 | 765 | |
| 763 | 766 | coredump_task_exit(tsk); |
| 764 | 767 | ptrace_event(PTRACE_EVENT_EXIT, code); |
| @@ -37,13 +37,13 @@ · 36 unmodified lines | ||
| 37 | 37 | #include <linux/fdtable.h> |
| 38 | 38 | #include <linux/iocontext.h> |
| 39 | 39 | #include <linux/key.h> |
| 40 | +#include <linux/kmsan.h> | |
| 40 | 41 | #include <linux/binfmts.h> |
| 41 | 42 | #include <linux/mman.h> |
| 42 | 43 | #include <linux/mmu_notifier.h> |
| 43 | 44 | #include <linux/fs.h> |
| 44 | 45 | #include <linux/mm.h> |
| 45 | 46 | #include <linux/mm_inline.h> |
| 46 | −#include <linux/vmacache.h> | |
| 47 | 47 | #include <linux/nsproxy.h> |
| 48 | 48 | #include <linux/capability.h> |
| 49 | 49 | #include <linux/cpu.h> |
| @@ -475,7 +475,6 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) · 425 unmodified lines | ||
| 475 | 475 | */ |
| 476 | 476 | *new = data_race(*orig); |
| 477 | 477 | INIT_LIST_HEAD(&new->anon_vma_chain); |
| 478 | − new->vm_next = new->vm_prev = NULL; | |
| 479 | 478 | dup_anon_vma_name(orig, new); |
| 480 | 479 | } |
| 481 | 480 | return new; |
| @@ -580,11 +579,12 @@ static void dup_mm_exe_file(struct mm_struct *mm, struct mm_struct *oldmm) · 98 unmodified lines | ||
| 580 | 579 | static __latent_entropy int dup_mmap(struct mm_struct *mm, |
| 581 | 580 | struct mm_struct *oldmm) |
| 582 | 581 | { |
| 583 | − struct vm_area_struct *mpnt, *tmp, *prev, **pprev; | |
| 584 | − struct rb_node **rb_link, *rb_parent; | |
| 582 | + struct vm_area_struct *mpnt, *tmp; | |
| 585 | 583 | int retval; |
| 586 | − unsigned long charge; | |
| 584 | + unsigned long charge = 0; | |
| 587 | 585 | LIST_HEAD(uf); |
| 586 | + MA_STATE(old_mas, &oldmm->mm_mt, 0, 0); | |
| 587 | + MA_STATE(mas, &mm->mm_mt, 0, 0); | |
| 588 | 588 | |
| 589 | 589 | uprobe_start_dup_mmap(); |
| 590 | 590 | if (mmap_write_lock_killable(oldmm)) { |
| @@ -606,16 +606,16 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, · 15 unmodified lines | ||
| 606 | 606 | mm->exec_vm = oldmm->exec_vm; |
| 607 | 607 | mm->stack_vm = oldmm->stack_vm; |
| 608 | 608 | |
| 609 | − rb_link = &mm->mm_rb.rb_node; | |
| 610 | − rb_parent = NULL; | |
| 611 | − pprev = &mm->mmap; | |
| 612 | 609 | retval = ksm_fork(mm, oldmm); |
| 613 | 610 | if (retval) |
| 614 | 611 | goto out; |
| 615 | 612 | khugepaged_fork(mm, oldmm); |
| 616 | 613 | |
| 617 | − prev = NULL; | |
| 618 | − for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) { | |
| 614 | + retval = mas_expected_entries(&mas, oldmm->map_count); | |
| 615 | + if (retval) | |
| 616 | + goto out; | |
| 617 | + | |
| 618 | + mas_for_each(&old_mas, mpnt, ULONG_MAX) { | |
| 619 | 619 | struct file *file; |
| 620 | 620 | |
| 621 | 621 | if (mpnt->vm_flags & VM_DONTCOPY) { |
| @@ -629,7 +629,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, · 7 unmodified lines | ||
| 629 | 629 | */ |
| 630 | 630 | if (fatal_signal_pending(current)) { |
| 631 | 631 | retval = -EINTR; |
| 632 | − goto out; | |
| 632 | + goto loop_out; | |
| 633 | 633 | } |
| 634 | 634 | if (mpnt->vm_flags & VM_ACCOUNT) { |
| 635 | 635 | unsigned long len = vma_pages(mpnt); |
| @@ -675,24 +675,17 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, · 39 unmodified lines | ||
| 675 | 675 | } |
| 676 | 676 | |
| 677 | 677 | /* |
| 678 | − * Clear hugetlb-related page reserves for children. This only | |
| 679 | − * affects MAP_PRIVATE mappings. Faults generated by the child | |
| 680 | − * are not guaranteed to succeed, even if read-only | |
| 678 | + * Copy/update hugetlb private vma information. | |
| 681 | 679 | */ |
| 682 | 680 | if (is_vm_hugetlb_page(tmp)) |
| 683 | − reset_vma_resv_huge_pages(tmp); | |
| 684 | − | |
| 685 | − /* | |
| 686 | − * Link in the new vma and copy the page table entries. | |
| 687 | − */ | |
| 688 | − *pprev = tmp; | |
| 689 | − pprev = &tmp->vm_next; | |
| 690 | − tmp->vm_prev = prev; | |
| 691 | − prev = tmp; | |
| 681 | + hugetlb_dup_vma_private(tmp); | |
| 692 | 682 | |
| 693 | − __vma_link_rb(mm, tmp, rb_link, rb_parent); | |
| 694 | − rb_link = &tmp->vm_rb.rb_right; | |
| 695 | − rb_parent = &tmp->vm_rb; | |
| 683 | + /* Link the vma into the MT */ | |
| 684 | + mas.index = tmp->vm_start; | |
| 685 | + mas.last = tmp->vm_end - 1; | |
| 686 | + mas_store(&mas, tmp); | |
| 687 | + if (mas_is_err(&mas)) | |
| 688 | + goto fail_nomem_mas_store; | |
| 696 | 689 | |
| 697 | 690 | mm->map_count++; |
| 698 | 691 | if (!(tmp->vm_flags & VM_WIPEONFORK)) |
| @@ -702,10 +695,12 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, · 3 unmodified lines | ||
| 702 | 695 | tmp->vm_ops->open(tmp); |
| 703 | 696 | |
| 704 | 697 | if (retval) |
| 705 | − goto out; | |
| 698 | + goto loop_out; | |
| 706 | 699 | } |
| 707 | 700 | /* a new mm has just been created */ |
| 708 | 701 | retval = arch_dup_mmap(oldmm, mm); |
| 702 | +loop_out: | |
| 703 | + mas_destroy(&mas); | |
| 709 | 704 | out: |
| 710 | 705 | mmap_write_unlock(mm); |
| 711 | 706 | flush_tlb_mm(oldmm); |
| @@ -714,6 +709,9 @@ out: · 2 unmodified lines | ||
| 714 | 709 | fail_uprobe_end: |
| 715 | 710 | uprobe_end_dup_mmap(); |
| 716 | 711 | return retval; |
| 712 | + | |
| 713 | +fail_nomem_mas_store: | |
| 714 | + unlink_anon_vmas(tmp); | |
| 717 | 715 | fail_nomem_anon_vma_fork: |
| 718 | 716 | mpol_put(vma_policy(tmp)); |
| 719 | 717 | fail_nomem_policy: |
| @@ -721,7 +719,7 @@ fail_nomem_policy: · 1 unmodified line | ||
| 721 | 719 | fail_nomem: |
| 722 | 720 | retval = -ENOMEM; |
| 723 | 721 | vm_unacct_memory(charge); |
| 724 | − goto out; | |
| 722 | + goto loop_out; | |
| 725 | 723 | } |
| 726 | 724 | |
| 727 | 725 | static inline int mm_alloc_pgd(struct mm_struct *mm) |
| @@ -1026,6 +1024,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) · 298 unmodified lines | ||
| 1026 | 1024 | tsk->worker_private = NULL; |
| 1027 | 1025 | |
| 1028 | 1026 | kcov_task_init(tsk); |
| 1027 | + kmsan_task_create(tsk); | |
| 1029 | 1028 | kmap_local_fork(tsk); |
| 1030 | 1029 | |
| 1031 | 1030 | #ifdef CONFIG_FAULT_INJECTION |
| @@ -1109,9 +1108,8 @@ static void mm_init_uprobes_state(struct mm_struct *mm) · 77 unmodified lines | ||
| 1109 | 1108 | static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, |
| 1110 | 1109 | struct user_namespace *user_ns) |
| 1111 | 1110 | { |
| 1112 | − mm->mmap = NULL; | |
| 1113 | − mm->mm_rb = RB_ROOT; | |
| 1114 | − mm->vmacache_seqnum = 0; | |
| 1111 | + mt_init_flags(&mm->mm_mt, MM_MT_FLAGS); | |
| 1112 | + mt_set_external_lock(&mm->mm_mt, &mm->mmap_lock); | |
| 1115 | 1113 | atomic_set(&mm->mm_users, 1); |
| 1116 | 1114 | atomic_set(&mm->mm_count, 1); |
| 1117 | 1115 | seqcount_init(&mm->write_protect_seq); |
| @@ -1152,6 +1150,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, · 34 unmodified lines | ||
| 1152 | 1150 | goto fail_nocontext; |
| 1153 | 1151 | |
| 1154 | 1152 | mm->user_ns = get_user_ns(user_ns); |
| 1153 | + lru_gen_init_mm(mm); | |
| 1155 | 1154 | return mm; |
| 1156 | 1155 | |
| 1157 | 1156 | fail_nocontext: |
| @@ -1194,6 +1193,7 @@ static inline void __mmput(struct mm_struct *mm) · 36 unmodified lines | ||
| 1194 | 1193 | } |
| 1195 | 1194 | if (mm->binfmt) |
| 1196 | 1195 | module_put(mm->binfmt->module); |
| 1196 | + lru_gen_del_mm(mm); | |
| 1197 | 1197 | mmdrop(mm); |
| 1198 | 1198 | } |
| 1199 | 1199 | |
| @@ -1285,13 +1285,16 @@ int replace_mm_exe_file(struct mm_struct *mm, struct file *new_exe_file) · 85 unmodified lines | ||
| 1285 | 1285 | /* Forbid mm->exe_file change if old file still mapped. */ |
| 1286 | 1286 | old_exe_file = get_mm_exe_file(mm); |
| 1287 | 1287 | if (old_exe_file) { |
| 1288 | + VMA_ITERATOR(vmi, mm, 0); | |
| 1288 | 1289 | mmap_read_lock(mm); |
| 1289 | − for (vma = mm->mmap; vma && !ret; vma = vma->vm_next) { | |
| 1290 | + for_each_vma(vmi, vma) { | |
| 1290 | 1291 | if (!vma->vm_file) |
| 1291 | 1292 | continue; |
| 1292 | 1293 | if (path_equal(&vma->vm_file->f_path, |
| 1293 | − &old_exe_file->f_path)) | |
| 1294 | + &old_exe_file->f_path)) { | |
| 1294 | 1295 | ret = -EBUSY; |
| 1296 | + break; | |
| 1297 | + } | |
| 1295 | 1298 | } |
| 1296 | 1299 | mmap_read_unlock(mm); |
| 1297 | 1300 | fput(old_exe_file); |
| @@ -1566,9 +1569,6 @@ static int copy_mm(unsigned long clone_flags, struct task_struct *tsk) · 268 unmodified lines | ||
| 1566 | 1569 | if (!oldmm) |
| 1567 | 1570 | return 0; |
| 1568 | 1571 | |
| 1569 | − /* initialize the new vmacache entries */ | |
| 1570 | − vmacache_flush(tsk); | |
| 1571 | − | |
| 1572 | 1572 | if (clone_flags & CLONE_VM) { |
| 1573 | 1573 | mmget(oldmm); |
| 1574 | 1574 | mm = oldmm; |
| @@ -2693,6 +2693,13 @@ pid_t kernel_clone(struct kernel_clone_args *args) · 1118 unmodified lines | ||
| 2693 | 2693 | get_task_struct(p); |
| 2694 | 2694 | } |
| 2695 | 2695 | |
| 2696 | + if (IS_ENABLED(CONFIG_LRU_GEN) && !(clone_flags & CLONE_VM)) { | |
| 2697 | + /* lock the task to synchronize with memcg migration */ | |
| 2698 | + task_lock(p); | |
| 2699 | + lru_gen_add_mm(p->mm); | |
| 2700 | + task_unlock(p); | |
| 2701 | + } | |
| 2702 | + | |
| 2696 | 2703 | wake_up_new_task(p); |
| 2697 | 2704 | |
| 2698 | 2705 | /* forking complete and child started to run, tell ptracer */ |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/fs.h> |
| 12 | 12 | #include <linux/hashtable.h> |
| 13 | 13 | #include <linux/init.h> |
| 14 | +#include <linux/kmsan-checks.h> | |
| 14 | 15 | #include <linux/mm.h> |
| 15 | 16 | #include <linux/preempt.h> |
| 16 | 17 | #include <linux/printk.h> |
| @@ -152,6 +153,12 @@ static void kcov_remote_area_put(struct kcov_remote_area *area, · 135 unmodified lines | ||
| 152 | 153 | INIT_LIST_HEAD(&area->list); |
| 153 | 154 | area->size = size; |
| 154 | 155 | list_add(&area->list, &kcov_remote_areas); |
| 156 | + /* | |
| 157 | + * KMSAN doesn't instrument this file, so it may not know area->list | |
| 158 | + * is initialized. Unpoison it explicitly to avoid reports in | |
| 159 | + * kcov_remote_area_get(). | |
| 160 | + */ | |
| 161 | + kmsan_unpoison_memory(&area->list, sizeof(area->list)); | |
| 155 | 162 | } |
| 156 | 163 | |
| 157 | 164 | static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_struct *t) |
| @@ -5,8 +5,9 @@ KCOV_INSTRUMENT := n · 4 unmodified lines | ||
| 5 | 5 | |
| 6 | 6 | obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o |
| 7 | 7 | |
| 8 | −# Avoid recursion lockdep -> KCSAN -> ... -> lockdep. | |
| 8 | +# Avoid recursion lockdep -> sanitizer -> ... -> lockdep. | |
| 9 | 9 | KCSAN_SANITIZE_lockdep.o := n |
| 10 | +KMSAN_SANITIZE_lockdep.o := n | |
| 10 | 11 | |
| 11 | 12 | ifdef CONFIG_FUNCTION_TRACER |
| 12 | 13 | CFLAGS_REMOVE_lockdep.o = $(CC_FLAGS_FTRACE) |
| @@ -4389,6 +4389,17 @@ void set_numabalancing_state(bool enabled) · 4388 unmodified lines | ||
| 4389 | 4389 | } |
| 4390 | 4390 | |
| 4391 | 4391 | #ifdef CONFIG_PROC_SYSCTL |
| 4392 | +static void reset_memory_tiering(void) | |
| 4393 | +{ | |
| 4394 | + struct pglist_data *pgdat; | |
| 4395 | + | |
| 4396 | + for_each_online_pgdat(pgdat) { | |
| 4397 | + pgdat->nbp_threshold = 0; | |
| 4398 | + pgdat->nbp_th_nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); | |
| 4399 | + pgdat->nbp_th_start = jiffies_to_msecs(jiffies); | |
| 4400 | + } | |
| 4401 | +} | |
| 4402 | + | |
| 4392 | 4403 | int sysctl_numa_balancing(struct ctl_table *table, int write, |
| 4393 | 4404 | void *buffer, size_t *lenp, loff_t *ppos) |
| 4394 | 4405 | { |
| @@ -4405,6 +4416,9 @@ int sysctl_numa_balancing(struct ctl_table *table, int write, · 10 unmodified lines | ||
| 4405 | 4416 | if (err < 0) |
| 4406 | 4417 | return err; |
| 4407 | 4418 | if (write) { |
| 4419 | + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && | |
| 4420 | + (state & NUMA_BALANCING_MEMORY_TIERING)) | |
| 4421 | + reset_memory_tiering(); | |
| 4408 | 4422 | sysctl_numa_balancing_mode = state; |
| 4409 | 4423 | __set_numabalancing_state(state); |
| 4410 | 4424 | } |
| @@ -5159,6 +5173,7 @@ context_switch(struct rq *rq, struct task_struct *prev, · 748 unmodified lines | ||
| 5159 | 5173 | * finish_task_switch()'s mmdrop(). |
| 5160 | 5174 | */ |
| 5161 | 5175 | switch_mm_irqs_off(prev->active_mm, next->mm, next); |
| 5176 | + lru_gen_use_mm(next->mm); | |
| 5162 | 5177 | |
| 5163 | 5178 | if (!prev->mm) { // from kernel |
| 5164 | 5179 | /* will mmdrop() in finish_task_switch(). */ |
| @@ -333,6 +333,7 @@ static __init int sched_init_debug(void) · 332 unmodified lines | ||
| 333 | 333 | debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min); |
| 334 | 334 | debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max); |
| 335 | 335 | debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size); |
| 336 | + debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold); | |
| 336 | 337 | #endif |
| 337 | 338 | |
| 338 | 339 | debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops); |
| @@ -40,6 +40,7 @@ · 39 unmodified lines | ||
| 40 | 40 | |
| 41 | 41 | #include <linux/cpuidle.h> |
| 42 | 42 | #include <linux/interrupt.h> |
| 43 | +#include <linux/memory-tiers.h> | |
| 43 | 44 | #include <linux/mempolicy.h> |
| 44 | 45 | #include <linux/mutex_api.h> |
| 45 | 46 | #include <linux/profile.h> |
| @@ -1090,6 +1091,12 @@ unsigned int sysctl_numa_balancing_scan_size = 256; · 1044 unmodified lines | ||
| 1090 | 1091 | /* Scan @scan_size MB every @scan_period after an initial @scan_delay in ms */ |
| 1091 | 1092 | unsigned int sysctl_numa_balancing_scan_delay = 1000; |
| 1092 | 1093 | |
| 1094 | +/* The page with hint page fault latency < threshold in ms is considered hot */ | |
| 1095 | +unsigned int sysctl_numa_balancing_hot_threshold = MSEC_PER_SEC; | |
| 1096 | + | |
| 1097 | +/* Restrict the NUMA promotion throughput (MB/s) for each target node. */ | |
| 1098 | +unsigned int sysctl_numa_balancing_promote_rate_limit = 65536; | |
| 1099 | + | |
| 1093 | 1100 | struct numa_group { |
| 1094 | 1101 | refcount_t refcount; |
| 1095 | 1102 | |
| @@ -1432,6 +1439,120 @@ static inline unsigned long group_weight(struct task_struct *p, int nid, · 336 unmodified lines | ||
| 1432 | 1439 | return 1000 * faults / total_faults; |
| 1433 | 1440 | } |
| 1434 | 1441 | |
| 1442 | +/* | |
| 1443 | + * If memory tiering mode is enabled, cpupid of slow memory page is | |
| 1444 | + * used to record scan time instead of CPU and PID. When tiering mode | |
| 1445 | + * is disabled at run time, the scan time (in cpupid) will be | |
| 1446 | + * interpreted as CPU and PID. So CPU needs to be checked to avoid to | |
| 1447 | + * access out of array bound. | |
| 1448 | + */ | |
| 1449 | +static inline bool cpupid_valid(int cpupid) | |
| 1450 | +{ | |
| 1451 | + return cpupid_to_cpu(cpupid) < nr_cpu_ids; | |
| 1452 | +} | |
| 1453 | + | |
| 1454 | +/* | |
| 1455 | + * For memory tiering mode, if there are enough free pages (more than | |
| 1456 | + * enough watermark defined here) in fast memory node, to take full | |
| 1457 | + * advantage of fast memory capacity, all recently accessed slow | |
| 1458 | + * memory pages will be migrated to fast memory node without | |
| 1459 | + * considering hot threshold. | |
| 1460 | + */ | |
| 1461 | +static bool pgdat_free_space_enough(struct pglist_data *pgdat) | |
| 1462 | +{ | |
| 1463 | + int z; | |
| 1464 | + unsigned long enough_wmark; | |
| 1465 | + | |
| 1466 | + enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT, | |
| 1467 | + pgdat->node_present_pages >> 4); | |
| 1468 | + for (z = pgdat->nr_zones - 1; z >= 0; z--) { | |
| 1469 | + struct zone *zone = pgdat->node_zones + z; | |
| 1470 | + | |
| 1471 | + if (!populated_zone(zone)) | |
| 1472 | + continue; | |
| 1473 | + | |
| 1474 | + if (zone_watermark_ok(zone, 0, | |
| 1475 | + wmark_pages(zone, WMARK_PROMO) + enough_wmark, | |
| 1476 | + ZONE_MOVABLE, 0)) | |
| 1477 | + return true; | |
| 1478 | + } | |
| 1479 | + return false; | |
| 1480 | +} | |
| 1481 | + | |
| 1482 | +/* | |
| 1483 | + * For memory tiering mode, when page tables are scanned, the scan | |
| 1484 | + * time will be recorded in struct page in addition to make page | |
| 1485 | + * PROT_NONE for slow memory page. So when the page is accessed, in | |
| 1486 | + * hint page fault handler, the hint page fault latency is calculated | |
| 1487 | + * via, | |
| 1488 | + * | |
| 1489 | + * hint page fault latency = hint page fault time - scan time | |
| 1490 | + * | |
| 1491 | + * The smaller the hint page fault latency, the higher the possibility | |
| 1492 | + * for the page to be hot. | |
| 1493 | + */ | |
| 1494 | +static int numa_hint_fault_latency(struct page *page) | |
| 1495 | +{ | |
| 1496 | + int last_time, time; | |
| 1497 | + | |
| 1498 | + time = jiffies_to_msecs(jiffies); | |
| 1499 | + last_time = xchg_page_access_time(page, time); | |
| 1500 | + | |
| 1501 | + return (time - last_time) & PAGE_ACCESS_TIME_MASK; | |
| 1502 | +} | |
| 1503 | + | |
| 1504 | +/* | |
| 1505 | + * For memory tiering mode, too high promotion/demotion throughput may | |
| 1506 | + * hurt application latency. So we provide a mechanism to rate limit | |
| 1507 | + * the number of pages that are tried to be promoted. | |
| 1508 | + */ | |
| 1509 | +static bool numa_promotion_rate_limit(struct pglist_data *pgdat, | |
| 1510 | + unsigned long rate_limit, int nr) | |
| 1511 | +{ | |
| 1512 | + unsigned long nr_cand; | |
| 1513 | + unsigned int now, start; | |
| 1514 | + | |
| 1515 | + now = jiffies_to_msecs(jiffies); | |
| 1516 | + mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE, nr); | |
| 1517 | + nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); | |
| 1518 | + start = pgdat->nbp_rl_start; | |
| 1519 | + if (now - start > MSEC_PER_SEC && | |
| 1520 | + cmpxchg(&pgdat->nbp_rl_start, start, now) == start) | |
| 1521 | + pgdat->nbp_rl_nr_cand = nr_cand; | |
| 1522 | + if (nr_cand - pgdat->nbp_rl_nr_cand >= rate_limit) | |
| 1523 | + return true; | |
| 1524 | + return false; | |
| 1525 | +} | |
| 1526 | + | |
| 1527 | +#define NUMA_MIGRATION_ADJUST_STEPS 16 | |
| 1528 | + | |
| 1529 | +static void numa_promotion_adjust_threshold(struct pglist_data *pgdat, | |
| 1530 | + unsigned long rate_limit, | |
| 1531 | + unsigned int ref_th) | |
| 1532 | +{ | |
| 1533 | + unsigned int now, start, th_period, unit_th, th; | |
| 1534 | + unsigned long nr_cand, ref_cand, diff_cand; | |
| 1535 | + | |
| 1536 | + now = jiffies_to_msecs(jiffies); | |
| 1537 | + th_period = sysctl_numa_balancing_scan_period_max; | |
| 1538 | + start = pgdat->nbp_th_start; | |
| 1539 | + if (now - start > th_period && | |
| 1540 | + cmpxchg(&pgdat->nbp_th_start, start, now) == start) { | |
| 1541 | + ref_cand = rate_limit * | |
| 1542 | + sysctl_numa_balancing_scan_period_max / MSEC_PER_SEC; | |
| 1543 | + nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE); | |
| 1544 | + diff_cand = nr_cand - pgdat->nbp_th_nr_cand; | |
| 1545 | + unit_th = ref_th * 2 / NUMA_MIGRATION_ADJUST_STEPS; | |
| 1546 | + th = pgdat->nbp_threshold ? : ref_th; | |
| 1547 | + if (diff_cand > ref_cand * 11 / 10) | |
| 1548 | + th = max(th - unit_th, unit_th); | |
| 1549 | + else if (diff_cand < ref_cand * 9 / 10) | |
| 1550 | + th = min(th + unit_th, ref_th * 2); | |
| 1551 | + pgdat->nbp_th_nr_cand = nr_cand; | |
| 1552 | + pgdat->nbp_threshold = th; | |
| 1553 | + } | |
| 1554 | +} | |
| 1555 | + | |
| 1435 | 1556 | bool should_numa_migrate_memory(struct task_struct *p, struct page * page, |
| 1436 | 1557 | int src_nid, int dst_cpu) |
| 1437 | 1558 | { |
| @@ -1439,9 +1560,44 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, · 1 unmodified line | ||
| 1439 | 1560 | int dst_nid = cpu_to_node(dst_cpu); |
| 1440 | 1561 | int last_cpupid, this_cpupid; |
| 1441 | 1562 | |
| 1563 | + /* | |
| 1564 | + * The pages in slow memory node should be migrated according | |
| 1565 | + * to hot/cold instead of private/shared. | |
| 1566 | + */ | |
| 1567 | + if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && | |
| 1568 | + !node_is_toptier(src_nid)) { | |
| 1569 | + struct pglist_data *pgdat; | |
| 1570 | + unsigned long rate_limit; | |
| 1571 | + unsigned int latency, th, def_th; | |
| 1572 | + | |
| 1573 | + pgdat = NODE_DATA(dst_nid); | |
| 1574 | + if (pgdat_free_space_enough(pgdat)) { | |
| 1575 | + /* workload changed, reset hot threshold */ | |
| 1576 | + pgdat->nbp_threshold = 0; | |
| 1577 | + return true; | |
| 1578 | + } | |
| 1579 | + | |
| 1580 | + def_th = sysctl_numa_balancing_hot_threshold; | |
| 1581 | + rate_limit = sysctl_numa_balancing_promote_rate_limit << \ | |
| 1582 | + (20 - PAGE_SHIFT); | |
| 1583 | + numa_promotion_adjust_threshold(pgdat, rate_limit, def_th); | |
| 1584 | + | |
| 1585 | + th = pgdat->nbp_threshold ? : def_th; | |
| 1586 | + latency = numa_hint_fault_latency(page); | |
| 1587 | + if (latency >= th) | |
| 1588 | + return false; | |
| 1589 | + | |
| 1590 | + return !numa_promotion_rate_limit(pgdat, rate_limit, | |
| 1591 | + thp_nr_pages(page)); | |
| 1592 | + } | |
| 1593 | + | |
| 1442 | 1594 | this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid); |
| 1443 | 1595 | last_cpupid = page_cpupid_xchg_last(page, this_cpupid); |
| 1444 | 1596 | |
| 1597 | + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && | |
| 1598 | + !node_is_toptier(src_nid) && !cpupid_valid(last_cpupid)) | |
| 1599 | + return false; | |
| 1600 | + | |
| 1445 | 1601 | /* |
| 1446 | 1602 | * Allow first faults or private faults to migrate immediately early in |
| 1447 | 1603 | * the lifetime of a task. The magic number 4 is based on waiting for |
| @@ -2681,6 +2837,15 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags) · 1233 unmodified lines | ||
| 2681 | 2837 | if (!p->mm) |
| 2682 | 2838 | return; |
| 2683 | 2839 | |
| 2840 | + /* | |
| 2841 | + * NUMA faults statistics are unnecessary for the slow memory | |
| 2842 | + * node for memory tiering mode. | |
| 2843 | + */ | |
| 2844 | + if (!node_is_toptier(mem_node) && | |
| 2845 | + (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING || | |
| 2846 | + !cpupid_valid(last_cpupid))) | |
| 2847 | + return; | |
| 2848 | + | |
| 2684 | 2849 | /* Allocate buffer to track faults on a per-node basis */ |
| 2685 | 2850 | if (unlikely(!p->numa_faults)) { |
| 2686 | 2851 | int size = sizeof(*p->numa_faults) * |
| @@ -2761,6 +2926,7 @@ static void task_numa_work(struct callback_head *work) · 74 unmodified lines | ||
| 2761 | 2926 | struct task_struct *p = current; |
| 2762 | 2927 | struct mm_struct *mm = p->mm; |
| 2763 | 2928 | u64 runtime = p->se.sum_exec_runtime; |
| 2929 | + MA_STATE(mas, &mm->mm_mt, 0, 0); | |
| 2764 | 2930 | struct vm_area_struct *vma; |
| 2765 | 2931 | unsigned long start, end; |
| 2766 | 2932 | unsigned long nr_pte_updates = 0; |
| @@ -2817,13 +2983,16 @@ static void task_numa_work(struct callback_head *work) · 50 unmodified lines | ||
| 2817 | 2983 | |
| 2818 | 2984 | if (!mmap_read_trylock(mm)) |
| 2819 | 2985 | return; |
| 2820 | − vma = find_vma(mm, start); | |
| 2986 | + mas_set(&mas, start); | |
| 2987 | + vma = mas_find(&mas, ULONG_MAX); | |
| 2821 | 2988 | if (!vma) { |
| 2822 | 2989 | reset_ptenuma_scan(p); |
| 2823 | 2990 | start = 0; |
| 2824 | − vma = mm->mmap; | |
| 2991 | + mas_set(&mas, start); | |
| 2992 | + vma = mas_find(&mas, ULONG_MAX); | |
| 2825 | 2993 | } |
| 2826 | − for (; vma; vma = vma->vm_next) { | |
| 2994 | + | |
| 2995 | + for (; vma; vma = mas_find(&mas, ULONG_MAX)) { | |
| 2827 | 2996 | if (!vma_migratable(vma) || !vma_policy_mof(vma) || |
| 2828 | 2997 | is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { |
| 2829 | 2998 | continue; |
| @@ -2446,6 +2446,7 @@ extern unsigned int sysctl_numa_balancing_scan_delay; · 2445 unmodified lines | ||
| 2446 | 2446 | extern unsigned int sysctl_numa_balancing_scan_period_min; |
| 2447 | 2447 | extern unsigned int sysctl_numa_balancing_scan_period_max; |
| 2448 | 2448 | extern unsigned int sysctl_numa_balancing_scan_size; |
| 2449 | +extern unsigned int sysctl_numa_balancing_hot_threshold; | |
| 2449 | 2450 | #endif |
| 2450 | 2451 | |
| 2451 | 2452 | #ifdef CONFIG_SCHED_HRTICK |
| @@ -1643,6 +1643,14 @@ static struct ctl_table kern_table[] = { · 1642 unmodified lines | ||
| 1643 | 1643 | .extra1 = SYSCTL_ZERO, |
| 1644 | 1644 | .extra2 = SYSCTL_FOUR, |
| 1645 | 1645 | }, |
| 1646 | + { | |
| 1647 | + .procname = "numa_balancing_promote_rate_limit_MBps", | |
| 1648 | + .data = &sysctl_numa_balancing_promote_rate_limit, | |
| 1649 | + .maxlen = sizeof(unsigned int), | |
| 1650 | + .mode = 0644, | |
| 1651 | + .proc_handler = proc_dointvec_minmax, | |
| 1652 | + .extra1 = SYSCTL_ZERO, | |
| 1653 | + }, | |
| 1646 | 1654 | #endif /* CONFIG_NUMA_BALANCING */ |
| 1647 | 1655 | { |
| 1648 | 1656 | .procname = "panic", |
| @@ -817,13 +817,12 @@ config DEBUG_VM · 816 unmodified lines | ||
| 817 | 817 | |
| 818 | 818 | If unsure, say N. |
| 819 | 819 | |
| 820 | −config DEBUG_VM_VMACACHE | |
| 821 | − bool "Debug VMA caching" | |
| 820 | +config DEBUG_VM_MAPLE_TREE | |
| 821 | + bool "Debug VM maple trees" | |
| 822 | 822 | depends on DEBUG_VM |
| 823 | + select DEBUG_MAPLE_TREE | |
| 823 | 824 | help |
| 824 | − Enable this to turn on VMA caching debug information. Doing so | |
| 825 | − can cause significant overhead, so only enable it in non-production | |
| 826 | − environments. | |
| 825 | + Enable VM maple tree debugging information and extra validations. | |
| 827 | 826 | |
| 828 | 827 | If unsure, say N. |
| 829 | 828 | |
| @@ -976,6 +975,7 @@ config DEBUG_STACKOVERFLOW · 146 unmodified lines | ||
| 976 | 975 | |
| 977 | 976 | source "lib/Kconfig.kasan" |
| 978 | 977 | source "lib/Kconfig.kfence" |
| 978 | +source "lib/Kconfig.kmsan" | |
| 979 | 979 | |
| 980 | 980 | endmenu # "Memory Debugging" |
| 981 | 981 | |
| @@ -1640,6 +1640,14 @@ config BUG_ON_DATA_CORRUPTION · 658 unmodified lines | ||
| 1640 | 1640 | |
| 1641 | 1641 | If unsure, say N. |
| 1642 | 1642 | |
| 1643 | +config DEBUG_MAPLE_TREE | |
| 1644 | + bool "Debug maple trees" | |
| 1645 | + depends on DEBUG_KERNEL | |
| 1646 | + help | |
| 1647 | + Enable maple tree debugging information and extra validations. | |
| 1648 | + | |
| 1649 | + If unsure, say N. | |
| 1650 | + | |
| 1643 | 1651 | endmenu |
| 1644 | 1652 | |
| 1645 | 1653 | config DEBUG_CREDENTIALS |
| @@ -167,14 +167,6 @@ config KASAN_STACK · 166 unmodified lines | ||
| 167 | 167 | as well, as it adds inline-style instrumentation that is run |
| 168 | 168 | unconditionally. |
| 169 | 169 | |
| 170 | −config KASAN_TAGS_IDENTIFY | |
| 171 | − bool "Memory corruption type identification" | |
| 172 | − depends on KASAN_SW_TAGS || KASAN_HW_TAGS | |
| 173 | − help | |
| 174 | − Enables best-effort identification of the bug types (use-after-free | |
| 175 | − or out-of-bounds) at the cost of increased memory consumption. | |
| 176 | − Only applicable for the tag-based KASAN modes. | |
| 177 | − | |
| 178 | 170 | config KASAN_VMALLOC |
| 179 | 171 | bool "Check accesses to vmalloc allocations" |
| 180 | 172 | depends on HAVE_ARCH_KASAN_VMALLOC |
| @@ -0,0 +1,62 @@ | ||
| 1 | +# SPDX-License-Identifier: GPL-2.0-only | |
| 2 | +config HAVE_ARCH_KMSAN | |
| 3 | + bool | |
| 4 | + | |
| 5 | +config HAVE_KMSAN_COMPILER | |
| 6 | + # Clang versions <14.0.0 also support -fsanitize=kernel-memory, but not | |
| 7 | + # all the features necessary to build the kernel with KMSAN. | |
| 8 | + depends on CC_IS_CLANG && CLANG_VERSION >= 140000 | |
| 9 | + def_bool $(cc-option,-fsanitize=kernel-memory -mllvm -msan-disable-checks=1) | |
| 10 | + | |
| 11 | +config KMSAN | |
| 12 | + bool "KMSAN: detector of uninitialized values use" | |
| 13 | + depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER | |
| 14 | + depends on SLUB && DEBUG_KERNEL && !KASAN && !KCSAN | |
| 15 | + select STACKDEPOT | |
| 16 | + select STACKDEPOT_ALWAYS_INIT | |
| 17 | + help | |
| 18 | + KernelMemorySanitizer (KMSAN) is a dynamic detector of uses of | |
| 19 | + uninitialized values in the kernel. It is based on compiler | |
| 20 | + instrumentation provided by Clang and thus requires Clang to build. | |
| 21 | + | |
| 22 | + An important note is that KMSAN is not intended for production use, | |
| 23 | + because it drastically increases kernel memory footprint and slows | |
| 24 | + the whole system down. | |
| 25 | + | |
| 26 | + See <file:Documentation/dev-tools/kmsan.rst> for more details. | |
| 27 | + | |
| 28 | +if KMSAN | |
| 29 | + | |
| 30 | +config HAVE_KMSAN_PARAM_RETVAL | |
| 31 | + # -fsanitize-memory-param-retval is supported only by Clang >= 14. | |
| 32 | + depends on HAVE_KMSAN_COMPILER | |
| 33 | + def_bool $(cc-option,-fsanitize=kernel-memory -fsanitize-memory-param-retval) | |
| 34 | + | |
| 35 | +config KMSAN_CHECK_PARAM_RETVAL | |
| 36 | + bool "Check for uninitialized values passed to and returned from functions" | |
| 37 | + default y | |
| 38 | + depends on HAVE_KMSAN_PARAM_RETVAL | |
| 39 | + help | |
| 40 | + If the compiler supports -fsanitize-memory-param-retval, KMSAN will | |
| 41 | + eagerly check every function parameter passed by value and every | |
| 42 | + function return value. | |
| 43 | + | |
| 44 | + Disabling KMSAN_CHECK_PARAM_RETVAL will result in tracking shadow for | |
| 45 | + function parameters and return values across function borders. This | |
| 46 | + is a more relaxed mode, but it generates more instrumentation code and | |
| 47 | + may potentially report errors in corner cases when non-instrumented | |
| 48 | + functions call instrumented ones. | |
| 49 | + | |
| 50 | +config KMSAN_KUNIT_TEST | |
| 51 | + tristate "KMSAN integration test suite" if !KUNIT_ALL_TESTS | |
| 52 | + default KUNIT_ALL_TESTS | |
| 53 | + depends on TRACEPOINTS && KUNIT | |
| 54 | + help | |
| 55 | + Test suite for KMSAN, testing various error detection scenarios, | |
| 56 | + and checking that reports are correctly output to console. | |
| 57 | + | |
| 58 | + Say Y here if you want the test to be built into the kernel and run | |
| 59 | + during boot; say M if you want the test to build as a module; say N | |
| 60 | + if you are unsure. | |
| 61 | + | |
| 62 | +endif | |
| @@ -29,7 +29,7 @@ endif · 28 unmodified lines | ||
| 29 | 29 | |
| 30 | 30 | lib-y := ctype.o string.o vsprintf.o cmdline.o \ |
| 31 | 31 | rbtree.o radix-tree.o timerqueue.o xarray.o \ |
| 32 | − idr.o extable.o irq_regs.o argv_split.o \ | |
| 32 | + maple_tree.o idr.o extable.o irq_regs.o argv_split.o \ | |
| 33 | 33 | flex_proportions.o ratelimit.o show_mem.o \ |
| 34 | 34 | is_single_threaded.o plist.o decompress.o kobject_uevent.o \ |
| 35 | 35 | earlycpio.o seq_buf.o siphash.o dec_and_lock.o \ |
| @@ -65,11 +65,6 @@ obj-$(CONFIG_TEST_SYSCTL) += test_sysctl.o · 29 unmodified lines | ||
| 65 | 65 | obj-$(CONFIG_TEST_SIPHASH) += test_siphash.o |
| 66 | 66 | obj-$(CONFIG_HASH_KUNIT_TEST) += test_hash.o |
| 67 | 67 | obj-$(CONFIG_TEST_IDA) += test_ida.o |
| 68 | −obj-$(CONFIG_KASAN_KUNIT_TEST) += test_kasan.o | |
| 69 | −CFLAGS_test_kasan.o += -fno-builtin | |
| 70 | −CFLAGS_test_kasan.o += $(call cc-disable-warning, vla) | |
| 71 | −obj-$(CONFIG_KASAN_MODULE_TEST) += test_kasan_module.o | |
| 72 | −CFLAGS_test_kasan_module.o += -fno-builtin | |
| 73 | 68 | obj-$(CONFIG_TEST_UBSAN) += test_ubsan.o |
| 74 | 69 | CFLAGS_test_ubsan.o += $(call cc-disable-warning, vla) |
| 75 | 70 | UBSAN_SANITIZE_test_ubsan.o := y |
| @@ -275,6 +270,9 @@ obj-$(CONFIG_POLYNOMIAL) += polynomial.o · 199 unmodified lines | ||
| 275 | 270 | CFLAGS_stackdepot.o += -fno-builtin |
| 276 | 271 | obj-$(CONFIG_STACKDEPOT) += stackdepot.o |
| 277 | 272 | KASAN_SANITIZE_stackdepot.o := n |
| 273 | +# In particular, instrumenting stackdepot.c with KMSAN will result in infinite | |
| 274 | +# recursion. | |
| 275 | +KMSAN_SANITIZE_stackdepot.o := n | |
| 278 | 276 | KCOV_INSTRUMENT_stackdepot.o := n |
| 279 | 277 | |
| 280 | 278 | obj-$(CONFIG_REF_TRACKER) += ref_tracker.o |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | */ |
| 7 | 7 | #include <linux/pci.h> |
| 8 | 8 | #include <linux/io.h> |
| 9 | +#include <linux/kmsan-checks.h> | |
| 9 | 10 | |
| 10 | 11 | #include <linux/export.h> |
| 11 | 12 | |
| @@ -70,26 +71,35 @@ static void bad_io_access(unsigned long port, const char *access) · 58 unmodified lines | ||
| 70 | 71 | #define mmio_read64be(addr) swab64(readq(addr)) |
| 71 | 72 | #endif |
| 72 | 73 | |
| 74 | +/* | |
| 75 | + * Here and below, we apply __no_kmsan_checks to functions reading data from | |
| 76 | + * hardware, to ensure that KMSAN marks their return values as initialized. | |
| 77 | + */ | |
| 78 | +__no_kmsan_checks | |
| 73 | 79 | unsigned int ioread8(const void __iomem *addr) |
| 74 | 80 | { |
| 75 | 81 | IO_COND(addr, return inb(port), return readb(addr)); |
| 76 | 82 | return 0xff; |
| 77 | 83 | } |
| 84 | +__no_kmsan_checks | |
| 78 | 85 | unsigned int ioread16(const void __iomem *addr) |
| 79 | 86 | { |
| 80 | 87 | IO_COND(addr, return inw(port), return readw(addr)); |
| 81 | 88 | return 0xffff; |
| 82 | 89 | } |
| 90 | +__no_kmsan_checks | |
| 83 | 91 | unsigned int ioread16be(const void __iomem *addr) |
| 84 | 92 | { |
| 85 | 93 | IO_COND(addr, return pio_read16be(port), return mmio_read16be(addr)); |
| 86 | 94 | return 0xffff; |
| 87 | 95 | } |
| 96 | +__no_kmsan_checks | |
| 88 | 97 | unsigned int ioread32(const void __iomem *addr) |
| 89 | 98 | { |
| 90 | 99 | IO_COND(addr, return inl(port), return readl(addr)); |
| 91 | 100 | return 0xffffffff; |
| 92 | 101 | } |
| 102 | +__no_kmsan_checks | |
| 93 | 103 | unsigned int ioread32be(const void __iomem *addr) |
| 94 | 104 | { |
| 95 | 105 | IO_COND(addr, return pio_read32be(port), return mmio_read32be(addr)); |
| @@ -142,18 +152,21 @@ static u64 pio_read64be_hi_lo(unsigned long port) · 46 unmodified lines | ||
| 142 | 152 | return lo | (hi << 32); |
| 143 | 153 | } |
| 144 | 154 | |
| 155 | +__no_kmsan_checks | |
| 145 | 156 | u64 ioread64_lo_hi(const void __iomem *addr) |
| 146 | 157 | { |
| 147 | 158 | IO_COND(addr, return pio_read64_lo_hi(port), return readq(addr)); |
| 148 | 159 | return 0xffffffffffffffffULL; |
| 149 | 160 | } |
| 150 | 161 | |
| 162 | +__no_kmsan_checks | |
| 151 | 163 | u64 ioread64_hi_lo(const void __iomem *addr) |
| 152 | 164 | { |
| 153 | 165 | IO_COND(addr, return pio_read64_hi_lo(port), return readq(addr)); |
| 154 | 166 | return 0xffffffffffffffffULL; |
| 155 | 167 | } |
| 156 | 168 | |
| 169 | +__no_kmsan_checks | |
| 157 | 170 | u64 ioread64be_lo_hi(const void __iomem *addr) |
| 158 | 171 | { |
| 159 | 172 | IO_COND(addr, return pio_read64be_lo_hi(port), |
| @@ -161,6 +174,7 @@ u64 ioread64be_lo_hi(const void __iomem *addr) · 1 unmodified line | ||
| 161 | 174 | return 0xffffffffffffffffULL; |
| 162 | 175 | } |
| 163 | 176 | |
| 177 | +__no_kmsan_checks | |
| 164 | 178 | u64 ioread64be_hi_lo(const void __iomem *addr) |
| 165 | 179 | { |
| 166 | 180 | IO_COND(addr, return pio_read64be_hi_lo(port), |
| @@ -188,22 +202,32 @@ EXPORT_SYMBOL(ioread64be_hi_lo); · 21 unmodified lines | ||
| 188 | 202 | |
| 189 | 203 | void iowrite8(u8 val, void __iomem *addr) |
| 190 | 204 | { |
| 205 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 206 | + kmsan_check_memory(&val, sizeof(val)); | |
| 191 | 207 | IO_COND(addr, outb(val,port), writeb(val, addr)); |
| 192 | 208 | } |
| 193 | 209 | void iowrite16(u16 val, void __iomem *addr) |
| 194 | 210 | { |
| 211 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 212 | + kmsan_check_memory(&val, sizeof(val)); | |
| 195 | 213 | IO_COND(addr, outw(val,port), writew(val, addr)); |
| 196 | 214 | } |
| 197 | 215 | void iowrite16be(u16 val, void __iomem *addr) |
| 198 | 216 | { |
| 217 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 218 | + kmsan_check_memory(&val, sizeof(val)); | |
| 199 | 219 | IO_COND(addr, pio_write16be(val,port), mmio_write16be(val, addr)); |
| 200 | 220 | } |
| 201 | 221 | void iowrite32(u32 val, void __iomem *addr) |
| 202 | 222 | { |
| 223 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 224 | + kmsan_check_memory(&val, sizeof(val)); | |
| 203 | 225 | IO_COND(addr, outl(val,port), writel(val, addr)); |
| 204 | 226 | } |
| 205 | 227 | void iowrite32be(u32 val, void __iomem *addr) |
| 206 | 228 | { |
| 229 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 230 | + kmsan_check_memory(&val, sizeof(val)); | |
| 207 | 231 | IO_COND(addr, pio_write32be(val,port), mmio_write32be(val, addr)); |
| 208 | 232 | } |
| 209 | 233 | EXPORT_SYMBOL(iowrite8); |
| @@ -239,24 +263,32 @@ static void pio_write64be_hi_lo(u64 val, unsigned long port) · 29 unmodified lines | ||
| 239 | 263 | |
| 240 | 264 | void iowrite64_lo_hi(u64 val, void __iomem *addr) |
| 241 | 265 | { |
| 266 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 267 | + kmsan_check_memory(&val, sizeof(val)); | |
| 242 | 268 | IO_COND(addr, pio_write64_lo_hi(val, port), |
| 243 | 269 | writeq(val, addr)); |
| 244 | 270 | } |
| 245 | 271 | |
| 246 | 272 | void iowrite64_hi_lo(u64 val, void __iomem *addr) |
| 247 | 273 | { |
| 274 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 275 | + kmsan_check_memory(&val, sizeof(val)); | |
| 248 | 276 | IO_COND(addr, pio_write64_hi_lo(val, port), |
| 249 | 277 | writeq(val, addr)); |
| 250 | 278 | } |
| 251 | 279 | |
| 252 | 280 | void iowrite64be_lo_hi(u64 val, void __iomem *addr) |
| 253 | 281 | { |
| 282 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 283 | + kmsan_check_memory(&val, sizeof(val)); | |
| 254 | 284 | IO_COND(addr, pio_write64be_lo_hi(val, port), |
| 255 | 285 | mmio_write64be(val, addr)); |
| 256 | 286 | } |
| 257 | 287 | |
| 258 | 288 | void iowrite64be_hi_lo(u64 val, void __iomem *addr) |
| 259 | 289 | { |
| 290 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 291 | + kmsan_check_memory(&val, sizeof(val)); | |
| 260 | 292 | IO_COND(addr, pio_write64be_hi_lo(val, port), |
| 261 | 293 | mmio_write64be(val, addr)); |
| 262 | 294 | } |
| @@ -328,14 +360,20 @@ static inline void mmio_outsl(void __iomem *addr, const u32 *src, int count) · 65 unmodified lines | ||
| 328 | 360 | void ioread8_rep(const void __iomem *addr, void *dst, unsigned long count) |
| 329 | 361 | { |
| 330 | 362 | IO_COND(addr, insb(port,dst,count), mmio_insb(addr, dst, count)); |
| 363 | + /* KMSAN must treat values read from devices as initialized. */ | |
| 364 | + kmsan_unpoison_memory(dst, count); | |
| 331 | 365 | } |
| 332 | 366 | void ioread16_rep(const void __iomem *addr, void *dst, unsigned long count) |
| 333 | 367 | { |
| 334 | 368 | IO_COND(addr, insw(port,dst,count), mmio_insw(addr, dst, count)); |
| 369 | + /* KMSAN must treat values read from devices as initialized. */ | |
| 370 | + kmsan_unpoison_memory(dst, count * 2); | |
| 335 | 371 | } |
| 336 | 372 | void ioread32_rep(const void __iomem *addr, void *dst, unsigned long count) |
| 337 | 373 | { |
| 338 | 374 | IO_COND(addr, insl(port,dst,count), mmio_insl(addr, dst, count)); |
| 375 | + /* KMSAN must treat values read from devices as initialized. */ | |
| 376 | + kmsan_unpoison_memory(dst, count * 4); | |
| 339 | 377 | } |
| 340 | 378 | EXPORT_SYMBOL(ioread8_rep); |
| 341 | 379 | EXPORT_SYMBOL(ioread16_rep); |
| @@ -343,14 +381,20 @@ EXPORT_SYMBOL(ioread32_rep); · 1 unmodified line | ||
| 343 | 381 | |
| 344 | 382 | void iowrite8_rep(void __iomem *addr, const void *src, unsigned long count) |
| 345 | 383 | { |
| 384 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 385 | + kmsan_check_memory(src, count); | |
| 346 | 386 | IO_COND(addr, outsb(port, src, count), mmio_outsb(addr, src, count)); |
| 347 | 387 | } |
| 348 | 388 | void iowrite16_rep(void __iomem *addr, const void *src, unsigned long count) |
| 349 | 389 | { |
| 390 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 391 | + kmsan_check_memory(src, count * 2); | |
| 350 | 392 | IO_COND(addr, outsw(port, src, count), mmio_outsw(addr, src, count)); |
| 351 | 393 | } |
| 352 | 394 | void iowrite32_rep(void __iomem *addr, const void *src, unsigned long count) |
| 353 | 395 | { |
| 396 | + /* Make sure uninitialized memory isn't copied to devices. */ | |
| 397 | + kmsan_check_memory(src, count * 4); | |
| 354 | 398 | IO_COND(addr, outsl(port, src,count), mmio_outsl(addr, src, count)); |
| 355 | 399 | } |
| 356 | 400 | EXPORT_SYMBOL(iowrite8_rep); |
| @@ -174,13 +174,16 @@ static int copyout(void __user *to, const void *from, size_t n) · 173 unmodified lines | ||
| 174 | 174 | |
| 175 | 175 | static int copyin(void *to, const void __user *from, size_t n) |
| 176 | 176 | { |
| 177 | + size_t res = n; | |
| 178 | + | |
| 177 | 179 | if (should_fail_usercopy()) |
| 178 | 180 | return n; |
| 179 | 181 | if (access_ok(from, n)) { |
| 180 | − instrument_copy_from_user(to, from, n); | |
| 181 | − n = raw_copy_from_user(to, from, n); | |
| 182 | + instrument_copy_from_user_before(to, from, n); | |
| 183 | + res = raw_copy_from_user(to, from, n); | |
| 184 | + instrument_copy_from_user_after(to, from, n, res); | |
| 182 | 185 | } |
| 183 | − return n; | |
| 186 | + return res; | |
| 184 | 187 | } |
| 185 | 188 | |
| 186 | 189 | static inline struct pipe_buffer *pipe_buf(const struct pipe_inode_info *pipe, |
| @@ -0,0 +1,7130 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0+ | |
| 2 | +/* | |
| 3 | + * Maple Tree implementation | |
| 4 | + * Copyright (c) 2018-2022 Oracle Corporation | |
| 5 | + * Authors: Liam R. Howlett <Liam.Howlett@oracle.com> | |
| 6 | + * Matthew Wilcox <willy@infradead.org> | |
| 7 | + */ | |
| 8 | + | |
| 9 | +/* | |
| 10 | + * DOC: Interesting implementation details of the Maple Tree | |
| 11 | + * | |
| 12 | + * Each node type has a number of slots for entries and a number of slots for | |
| 13 | + * pivots. In the case of dense nodes, the pivots are implied by the position | |
| 14 | + * and are simply the slot index + the minimum of the node. | |
| 15 | + * | |
| 16 | + * In regular B-Tree terms, pivots are called keys. The term pivot is used to | |
| 17 | + * indicate that the tree is specifying ranges, Pivots may appear in the | |
| 18 | + * subtree with an entry attached to the value where as keys are unique to a | |
| 19 | + * specific position of a B-tree. Pivot values are inclusive of the slot with | |
| 20 | + * the same index. | |
| 21 | + * | |
| 22 | + * | |
| 23 | + * The following illustrates the layout of a range64 nodes slots and pivots. | |
| 24 | + * | |
| 25 | + * | |
| 26 | + * Slots -> | 0 | 1 | 2 | ... | 12 | 13 | 14 | 15 | | |
| 27 | + * ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ ┬ | |
| 28 | + * │ │ │ │ │ │ │ │ └─ Implied maximum | |
| 29 | + * │ │ │ │ │ │ │ └─ Pivot 14 | |
| 30 | + * │ │ │ │ │ │ └─ Pivot 13 | |
| 31 | + * │ │ │ │ │ └─ Pivot 12 | |
| 32 | + * │ │ │ │ └─ Pivot 11 | |
| 33 | + * │ │ │ └─ Pivot 2 | |
| 34 | + * │ │ └─ Pivot 1 | |
| 35 | + * │ └─ Pivot 0 | |
| 36 | + * └─ Implied minimum | |
| 37 | + * | |
| 38 | + * Slot contents: | |
| 39 | + * Internal (non-leaf) nodes contain pointers to other nodes. | |
| 40 | + * Leaf nodes contain entries. | |
| 41 | + * | |
| 42 | + * The location of interest is often referred to as an offset. All offsets have | |
| 43 | + * a slot, but the last offset has an implied pivot from the node above (or | |
| 44 | + * UINT_MAX for the root node. | |
| 45 | + * | |
| 46 | + * Ranges complicate certain write activities. When modifying any of | |
| 47 | + * the B-tree variants, it is known that one entry will either be added or | |
| 48 | + * deleted. When modifying the Maple Tree, one store operation may overwrite | |
| 49 | + * the entire data set, or one half of the tree, or the middle half of the tree. | |
| 50 | + * | |
| 51 | + */ | |
| 52 | + | |
| 53 | + | |
| 54 | +#include <linux/maple_tree.h> | |
| 55 | +#include <linux/xarray.h> | |
| 56 | +#include <linux/types.h> | |
| 57 | +#include <linux/export.h> | |
| 58 | +#include <linux/slab.h> | |
| 59 | +#include <linux/limits.h> | |
| 60 | +#include <asm/barrier.h> | |
| 61 | + | |
| 62 | +#define CREATE_TRACE_POINTS | |
| 63 | +#include <trace/events/maple_tree.h> | |
| 64 | + | |
| 65 | +#define MA_ROOT_PARENT 1 | |
| 66 | + | |
| 67 | +/* | |
| 68 | + * Maple state flags | |
| 69 | + * * MA_STATE_BULK - Bulk insert mode | |
| 70 | + * * MA_STATE_REBALANCE - Indicate a rebalance during bulk insert | |
| 71 | + * * MA_STATE_PREALLOC - Preallocated nodes, WARN_ON allocation | |
| 72 | + */ | |
| 73 | +#define MA_STATE_BULK 1 | |
| 74 | +#define MA_STATE_REBALANCE 2 | |
| 75 | +#define MA_STATE_PREALLOC 4 | |
| 76 | + | |
| 77 | +#define ma_parent_ptr(x) ((struct maple_pnode *)(x)) | |
| 78 | +#define ma_mnode_ptr(x) ((struct maple_node *)(x)) | |
| 79 | +#define ma_enode_ptr(x) ((struct maple_enode *)(x)) | |
| 80 | +static struct kmem_cache *maple_node_cache; | |
| 81 | + | |
| 82 | +#ifdef CONFIG_DEBUG_MAPLE_TREE | |
| 83 | +static const unsigned long mt_max[] = { | |
| 84 | + [maple_dense] = MAPLE_NODE_SLOTS, | |
| 85 | + [maple_leaf_64] = ULONG_MAX, | |
| 86 | + [maple_range_64] = ULONG_MAX, | |
| 87 | + [maple_arange_64] = ULONG_MAX, | |
| 88 | +}; | |
| 89 | +#define mt_node_max(x) mt_max[mte_node_type(x)] | |
| 90 | +#endif | |
| 91 | + | |
| 92 | +static const unsigned char mt_slots[] = { | |
| 93 | + [maple_dense] = MAPLE_NODE_SLOTS, | |
| 94 | + [maple_leaf_64] = MAPLE_RANGE64_SLOTS, | |
| 95 | + [maple_range_64] = MAPLE_RANGE64_SLOTS, | |
| 96 | + [maple_arange_64] = MAPLE_ARANGE64_SLOTS, | |
| 97 | +}; | |
| 98 | +#define mt_slot_count(x) mt_slots[mte_node_type(x)] | |
| 99 | + | |
| 100 | +static const unsigned char mt_pivots[] = { | |
| 101 | + [maple_dense] = 0, | |
| 102 | + [maple_leaf_64] = MAPLE_RANGE64_SLOTS - 1, | |
| 103 | + [maple_range_64] = MAPLE_RANGE64_SLOTS - 1, | |
| 104 | + [maple_arange_64] = MAPLE_ARANGE64_SLOTS - 1, | |
| 105 | +}; | |
| 106 | +#define mt_pivot_count(x) mt_pivots[mte_node_type(x)] | |
| 107 | + | |
| 108 | +static const unsigned char mt_min_slots[] = { | |
| 109 | + [maple_dense] = MAPLE_NODE_SLOTS / 2, | |
| 110 | + [maple_leaf_64] = (MAPLE_RANGE64_SLOTS / 2) - 2, | |
| 111 | + [maple_range_64] = (MAPLE_RANGE64_SLOTS / 2) - 2, | |
| 112 | + [maple_arange_64] = (MAPLE_ARANGE64_SLOTS / 2) - 1, | |
| 113 | +}; | |
| 114 | +#define mt_min_slot_count(x) mt_min_slots[mte_node_type(x)] | |
| 115 | + | |
| 116 | +#define MAPLE_BIG_NODE_SLOTS (MAPLE_RANGE64_SLOTS * 2 + 2) | |
| 117 | +#define MAPLE_BIG_NODE_GAPS (MAPLE_ARANGE64_SLOTS * 2 + 1) | |
| 118 | + | |
| 119 | +struct maple_big_node { | |
| 120 | + struct maple_pnode *parent; | |
| 121 | + unsigned long pivot[MAPLE_BIG_NODE_SLOTS - 1]; | |
| 122 | + union { | |
| 123 | + struct maple_enode *slot[MAPLE_BIG_NODE_SLOTS]; | |
| 124 | + struct { | |
| 125 | + unsigned long padding[MAPLE_BIG_NODE_GAPS]; | |
| 126 | + unsigned long gap[MAPLE_BIG_NODE_GAPS]; | |
| 127 | + }; | |
| 128 | + }; | |
| 129 | + unsigned char b_end; | |
| 130 | + enum maple_type type; | |
| 131 | +}; | |
| 132 | + | |
| 133 | +/* | |
| 134 | + * The maple_subtree_state is used to build a tree to replace a segment of an | |
| 135 | + * existing tree in a more atomic way. Any walkers of the older tree will hit a | |
| 136 | + * dead node and restart on updates. | |
| 137 | + */ | |
| 138 | +struct maple_subtree_state { | |
| 139 | + struct ma_state *orig_l; /* Original left side of subtree */ | |
| 140 | + struct ma_state *orig_r; /* Original right side of subtree */ | |
| 141 | + struct ma_state *l; /* New left side of subtree */ | |
| 142 | + struct ma_state *m; /* New middle of subtree (rare) */ | |
| 143 | + struct ma_state *r; /* New right side of subtree */ | |
| 144 | + struct ma_topiary *free; /* nodes to be freed */ | |
| 145 | + struct ma_topiary *destroy; /* Nodes to be destroyed (walked and freed) */ | |
| 146 | + struct maple_big_node *bn; | |
| 147 | +}; | |
| 148 | + | |
| 149 | +/* Functions */ | |
| 150 | +static inline struct maple_node *mt_alloc_one(gfp_t gfp) | |
| 151 | +{ | |
| 152 | + return kmem_cache_alloc(maple_node_cache, gfp | __GFP_ZERO); | |
| 153 | +} | |
| 154 | + | |
| 155 | +static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes) | |
| 156 | +{ | |
| 157 | + return kmem_cache_alloc_bulk(maple_node_cache, gfp | __GFP_ZERO, size, | |
| 158 | + nodes); | |
| 159 | +} | |
| 160 | + | |
| 161 | +static inline void mt_free_bulk(size_t size, void __rcu **nodes) | |
| 162 | +{ | |
| 163 | + kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes); | |
| 164 | +} | |
| 165 | + | |
| 166 | +static void mt_free_rcu(struct rcu_head *head) | |
| 167 | +{ | |
| 168 | + struct maple_node *node = container_of(head, struct maple_node, rcu); | |
| 169 | + | |
| 170 | + kmem_cache_free(maple_node_cache, node); | |
| 171 | +} | |
| 172 | + | |
| 173 | +/* | |
| 174 | + * ma_free_rcu() - Use rcu callback to free a maple node | |
| 175 | + * @node: The node to free | |
| 176 | + * | |
| 177 | + * The maple tree uses the parent pointer to indicate this node is no longer in | |
| 178 | + * use and will be freed. | |
| 179 | + */ | |
| 180 | +static void ma_free_rcu(struct maple_node *node) | |
| 181 | +{ | |
| 182 | + node->parent = ma_parent_ptr(node); | |
| 183 | + call_rcu(&node->rcu, mt_free_rcu); | |
| 184 | +} | |
| 185 | + | |
| 186 | +static unsigned int mt_height(const struct maple_tree *mt) | |
| 187 | +{ | |
| 188 | + return (mt->ma_flags & MT_FLAGS_HEIGHT_MASK) >> MT_FLAGS_HEIGHT_OFFSET; | |
| 189 | +} | |
| 190 | + | |
| 191 | +static void mas_set_height(struct ma_state *mas) | |
| 192 | +{ | |
| 193 | + unsigned int new_flags = mas->tree->ma_flags; | |
| 194 | + | |
| 195 | + new_flags &= ~MT_FLAGS_HEIGHT_MASK; | |
| 196 | + BUG_ON(mas->depth > MAPLE_HEIGHT_MAX); | |
| 197 | + new_flags |= mas->depth << MT_FLAGS_HEIGHT_OFFSET; | |
| 198 | + mas->tree->ma_flags = new_flags; | |
| 199 | +} | |
| 200 | + | |
| 201 | +static unsigned int mas_mt_height(struct ma_state *mas) | |
| 202 | +{ | |
| 203 | + return mt_height(mas->tree); | |
| 204 | +} | |
| 205 | + | |
| 206 | +static inline enum maple_type mte_node_type(const struct maple_enode *entry) | |
| 207 | +{ | |
| 208 | + return ((unsigned long)entry >> MAPLE_NODE_TYPE_SHIFT) & | |
| 209 | + MAPLE_NODE_TYPE_MASK; | |
| 210 | +} | |
| 211 | + | |
| 212 | +static inline bool ma_is_dense(const enum maple_type type) | |
| 213 | +{ | |
| 214 | + return type < maple_leaf_64; | |
| 215 | +} | |
| 216 | + | |
| 217 | +static inline bool ma_is_leaf(const enum maple_type type) | |
| 218 | +{ | |
| 219 | + return type < maple_range_64; | |
| 220 | +} | |
| 221 | + | |
| 222 | +static inline bool mte_is_leaf(const struct maple_enode *entry) | |
| 223 | +{ | |
| 224 | + return ma_is_leaf(mte_node_type(entry)); | |
| 225 | +} | |
| 226 | + | |
| 227 | +/* | |
| 228 | + * We also reserve values with the bottom two bits set to '10' which are | |
| 229 | + * below 4096 | |
| 230 | + */ | |
| 231 | +static inline bool mt_is_reserved(const void *entry) | |
| 232 | +{ | |
| 233 | + return ((unsigned long)entry < MAPLE_RESERVED_RANGE) && | |
| 234 | + xa_is_internal(entry); | |
| 235 | +} | |
| 236 | + | |
| 237 | +static inline void mas_set_err(struct ma_state *mas, long err) | |
| 238 | +{ | |
| 239 | + mas->node = MA_ERROR(err); | |
| 240 | +} | |
| 241 | + | |
| 242 | +static inline bool mas_is_ptr(struct ma_state *mas) | |
| 243 | +{ | |
| 244 | + return mas->node == MAS_ROOT; | |
| 245 | +} | |
| 246 | + | |
| 247 | +static inline bool mas_is_start(struct ma_state *mas) | |
| 248 | +{ | |
| 249 | + return mas->node == MAS_START; | |
| 250 | +} | |
| 251 | + | |
| 252 | +bool mas_is_err(struct ma_state *mas) | |
| 253 | +{ | |
| 254 | + return xa_is_err(mas->node); | |
| 255 | +} | |
| 256 | + | |
| 257 | +static inline bool mas_searchable(struct ma_state *mas) | |
| 258 | +{ | |
| 259 | + if (mas_is_none(mas)) | |
| 260 | + return false; | |
| 261 | + | |
| 262 | + if (mas_is_ptr(mas)) | |
| 263 | + return false; | |
| 264 | + | |
| 265 | + return true; | |
| 266 | +} | |
| 267 | + | |
| 268 | +static inline struct maple_node *mte_to_node(const struct maple_enode *entry) | |
| 269 | +{ | |
| 270 | + return (struct maple_node *)((unsigned long)entry & ~MAPLE_NODE_MASK); | |
| 271 | +} | |
| 272 | + | |
| 273 | +/* | |
| 274 | + * mte_to_mat() - Convert a maple encoded node to a maple topiary node. | |
| 275 | + * @entry: The maple encoded node | |
| 276 | + * | |
| 277 | + * Return: a maple topiary pointer | |
| 278 | + */ | |
| 279 | +static inline struct maple_topiary *mte_to_mat(const struct maple_enode *entry) | |
| 280 | +{ | |
| 281 | + return (struct maple_topiary *) | |
| 282 | + ((unsigned long)entry & ~MAPLE_NODE_MASK); | |
| 283 | +} | |
| 284 | + | |
| 285 | +/* | |
| 286 | + * mas_mn() - Get the maple state node. | |
| 287 | + * @mas: The maple state | |
| 288 | + * | |
| 289 | + * Return: the maple node (not encoded - bare pointer). | |
| 290 | + */ | |
| 291 | +static inline struct maple_node *mas_mn(const struct ma_state *mas) | |
| 292 | +{ | |
| 293 | + return mte_to_node(mas->node); | |
| 294 | +} | |
| 295 | + | |
| 296 | +/* | |
| 297 | + * mte_set_node_dead() - Set a maple encoded node as dead. | |
| 298 | + * @mn: The maple encoded node. | |
| 299 | + */ | |
| 300 | +static inline void mte_set_node_dead(struct maple_enode *mn) | |
| 301 | +{ | |
| 302 | + mte_to_node(mn)->parent = ma_parent_ptr(mte_to_node(mn)); | |
| 303 | + smp_wmb(); /* Needed for RCU */ | |
| 304 | +} | |
| 305 | + | |
| 306 | +/* Bit 1 indicates the root is a node */ | |
| 307 | +#define MAPLE_ROOT_NODE 0x02 | |
| 308 | +/* maple_type stored bit 3-6 */ | |
| 309 | +#define MAPLE_ENODE_TYPE_SHIFT 0x03 | |
| 310 | +/* Bit 2 means a NULL somewhere below */ | |
| 311 | +#define MAPLE_ENODE_NULL 0x04 | |
| 312 | + | |
| 313 | +static inline struct maple_enode *mt_mk_node(const struct maple_node *node, | |
| 314 | + enum maple_type type) | |
| 315 | +{ | |
| 316 | + return (void *)((unsigned long)node | | |
| 317 | + (type << MAPLE_ENODE_TYPE_SHIFT) | MAPLE_ENODE_NULL); | |
| 318 | +} | |
| 319 | + | |
| 320 | +static inline void *mte_mk_root(const struct maple_enode *node) | |
| 321 | +{ | |
| 322 | + return (void *)((unsigned long)node | MAPLE_ROOT_NODE); | |
| 323 | +} | |
| 324 | + | |
| 325 | +static inline void *mte_safe_root(const struct maple_enode *node) | |
| 326 | +{ | |
| 327 | + return (void *)((unsigned long)node & ~MAPLE_ROOT_NODE); | |
| 328 | +} | |
| 329 | + | |
| 330 | +static inline void mte_set_full(const struct maple_enode *node) | |
| 331 | +{ | |
| 332 | + node = (void *)((unsigned long)node & ~MAPLE_ENODE_NULL); | |
| 333 | +} | |
| 334 | + | |
| 335 | +static inline void mte_clear_full(const struct maple_enode *node) | |
| 336 | +{ | |
| 337 | + node = (void *)((unsigned long)node | MAPLE_ENODE_NULL); | |
| 338 | +} | |
| 339 | + | |
| 340 | +static inline bool ma_is_root(struct maple_node *node) | |
| 341 | +{ | |
| 342 | + return ((unsigned long)node->parent & MA_ROOT_PARENT); | |
| 343 | +} | |
| 344 | + | |
| 345 | +static inline bool mte_is_root(const struct maple_enode *node) | |
| 346 | +{ | |
| 347 | + return ma_is_root(mte_to_node(node)); | |
| 348 | +} | |
| 349 | + | |
| 350 | +static inline bool mas_is_root_limits(const struct ma_state *mas) | |
| 351 | +{ | |
| 352 | + return !mas->min && mas->max == ULONG_MAX; | |
| 353 | +} | |
| 354 | + | |
| 355 | +static inline bool mt_is_alloc(struct maple_tree *mt) | |
| 356 | +{ | |
| 357 | + return (mt->ma_flags & MT_FLAGS_ALLOC_RANGE); | |
| 358 | +} | |
| 359 | + | |
| 360 | +/* | |
| 361 | + * The Parent Pointer | |
| 362 | + * Excluding root, the parent pointer is 256B aligned like all other tree nodes. | |
| 363 | + * When storing a 32 or 64 bit values, the offset can fit into 5 bits. The 16 | |
| 364 | + * bit values need an extra bit to store the offset. This extra bit comes from | |
| 365 | + * a reuse of the last bit in the node type. This is possible by using bit 1 to | |
| 366 | + * indicate if bit 2 is part of the type or the slot. | |
| 367 | + * | |
| 368 | + * Note types: | |
| 369 | + * 0x??1 = Root | |
| 370 | + * 0x?00 = 16 bit nodes | |
| 371 | + * 0x010 = 32 bit nodes | |
| 372 | + * 0x110 = 64 bit nodes | |
| 373 | + * | |
| 374 | + * Slot size and alignment | |
| 375 | + * 0b??1 : Root | |
| 376 | + * 0b?00 : 16 bit values, type in 0-1, slot in 2-7 | |
| 377 | + * 0b010 : 32 bit values, type in 0-2, slot in 3-7 | |
| 378 | + * 0b110 : 64 bit values, type in 0-2, slot in 3-7 | |
| 379 | + */ | |
| 380 | + | |
| 381 | +#define MAPLE_PARENT_ROOT 0x01 | |
| 382 | + | |
| 383 | +#define MAPLE_PARENT_SLOT_SHIFT 0x03 | |
| 384 | +#define MAPLE_PARENT_SLOT_MASK 0xF8 | |
| 385 | + | |
| 386 | +#define MAPLE_PARENT_16B_SLOT_SHIFT 0x02 | |
| 387 | +#define MAPLE_PARENT_16B_SLOT_MASK 0xFC | |
| 388 | + | |
| 389 | +#define MAPLE_PARENT_RANGE64 0x06 | |
| 390 | +#define MAPLE_PARENT_RANGE32 0x04 | |
| 391 | +#define MAPLE_PARENT_NOT_RANGE16 0x02 | |
| 392 | + | |
| 393 | +/* | |
| 394 | + * mte_parent_shift() - Get the parent shift for the slot storage. | |
| 395 | + * @parent: The parent pointer cast as an unsigned long | |
| 396 | + * Return: The shift into that pointer to the star to of the slot | |
| 397 | + */ | |
| 398 | +static inline unsigned long mte_parent_shift(unsigned long parent) | |
| 399 | +{ | |
| 400 | + /* Note bit 1 == 0 means 16B */ | |
| 401 | + if (likely(parent & MAPLE_PARENT_NOT_RANGE16)) | |
| 402 | + return MAPLE_PARENT_SLOT_SHIFT; | |
| 403 | + | |
| 404 | + return MAPLE_PARENT_16B_SLOT_SHIFT; | |
| 405 | +} | |
| 406 | + | |
| 407 | +/* | |
| 408 | + * mte_parent_slot_mask() - Get the slot mask for the parent. | |
| 409 | + * @parent: The parent pointer cast as an unsigned long. | |
| 410 | + * Return: The slot mask for that parent. | |
| 411 | + */ | |
| 412 | +static inline unsigned long mte_parent_slot_mask(unsigned long parent) | |
| 413 | +{ | |
| 414 | + /* Note bit 1 == 0 means 16B */ | |
| 415 | + if (likely(parent & MAPLE_PARENT_NOT_RANGE16)) | |
| 416 | + return MAPLE_PARENT_SLOT_MASK; | |
| 417 | + | |
| 418 | + return MAPLE_PARENT_16B_SLOT_MASK; | |
| 419 | +} | |
| 420 | + | |
| 421 | +/* | |
| 422 | + * mas_parent_enum() - Return the maple_type of the parent from the stored | |
| 423 | + * parent type. | |
| 424 | + * @mas: The maple state | |
| 425 | + * @node: The maple_enode to extract the parent's enum | |
| 426 | + * Return: The node->parent maple_type | |
| 427 | + */ | |
| 428 | +static inline | |
| 429 | +enum maple_type mte_parent_enum(struct maple_enode *p_enode, | |
| 430 | + struct maple_tree *mt) | |
| 431 | +{ | |
| 432 | + unsigned long p_type; | |
| 433 | + | |
| 434 | + p_type = (unsigned long)p_enode; | |
| 435 | + if (p_type & MAPLE_PARENT_ROOT) | |
| 436 | + return 0; /* Validated in the caller. */ | |
| 437 | + | |
| 438 | + p_type &= MAPLE_NODE_MASK; | |
| 439 | + p_type = p_type & ~(MAPLE_PARENT_ROOT | mte_parent_slot_mask(p_type)); | |
| 440 | + | |
| 441 | + switch (p_type) { | |
| 442 | + case MAPLE_PARENT_RANGE64: /* or MAPLE_PARENT_ARANGE64 */ | |
| 443 | + if (mt_is_alloc(mt)) | |
| 444 | + return maple_arange_64; | |
| 445 | + return maple_range_64; | |
| 446 | + } | |
| 447 | + | |
| 448 | + return 0; | |
| 449 | +} | |
| 450 | + | |
| 451 | +static inline | |
| 452 | +enum maple_type mas_parent_enum(struct ma_state *mas, struct maple_enode *enode) | |
| 453 | +{ | |
| 454 | + return mte_parent_enum(ma_enode_ptr(mte_to_node(enode)->parent), mas->tree); | |
| 455 | +} | |
| 456 | + | |
| 457 | +/* | |
| 458 | + * mte_set_parent() - Set the parent node and encode the slot | |
| 459 | + * @enode: The encoded maple node. | |
| 460 | + * @parent: The encoded maple node that is the parent of @enode. | |
| 461 | + * @slot: The slot that @enode resides in @parent. | |
| 462 | + * | |
| 463 | + * Slot number is encoded in the enode->parent bit 3-6 or 2-6, depending on the | |
| 464 | + * parent type. | |
| 465 | + */ | |
| 466 | +static inline | |
| 467 | +void mte_set_parent(struct maple_enode *enode, const struct maple_enode *parent, | |
| 468 | + unsigned char slot) | |
| 469 | +{ | |
| 470 | + unsigned long val = (unsigned long) parent; | |
| 471 | + unsigned long shift; | |
| 472 | + unsigned long type; | |
| 473 | + enum maple_type p_type = mte_node_type(parent); | |
| 474 | + | |
| 475 | + BUG_ON(p_type == maple_dense); | |
| 476 | + BUG_ON(p_type == maple_leaf_64); | |
| 477 | + | |
| 478 | + switch (p_type) { | |
| 479 | + case maple_range_64: | |
| 480 | + case maple_arange_64: | |
| 481 | + shift = MAPLE_PARENT_SLOT_SHIFT; | |
| 482 | + type = MAPLE_PARENT_RANGE64; | |
| 483 | + break; | |
| 484 | + default: | |
| 485 | + case maple_dense: | |
| 486 | + case maple_leaf_64: | |
| 487 | + shift = type = 0; | |
| 488 | + break; | |
| 489 | + } | |
| 490 | + | |
| 491 | + val &= ~MAPLE_NODE_MASK; /* Clear all node metadata in parent */ | |
| 492 | + val |= (slot << shift) | type; | |
| 493 | + mte_to_node(enode)->parent = ma_parent_ptr(val); | |
| 494 | +} | |
| 495 | + | |
| 496 | +/* | |
| 497 | + * mte_parent_slot() - get the parent slot of @enode. | |
| 498 | + * @enode: The encoded maple node. | |
| 499 | + * | |
| 500 | + * Return: The slot in the parent node where @enode resides. | |
| 501 | + */ | |
| 502 | +static inline unsigned int mte_parent_slot(const struct maple_enode *enode) | |
| 503 | +{ | |
| 504 | + unsigned long val = (unsigned long) mte_to_node(enode)->parent; | |
| 505 | + | |
| 506 | + /* Root. */ | |
| 507 | + if (val & 1) | |
| 508 | + return 0; | |
| 509 | + | |
| 510 | + /* | |
| 511 | + * Okay to use MAPLE_PARENT_16B_SLOT_MASK as the last bit will be lost | |
| 512 | + * by shift if the parent shift is MAPLE_PARENT_SLOT_SHIFT | |
| 513 | + */ | |
| 514 | + return (val & MAPLE_PARENT_16B_SLOT_MASK) >> mte_parent_shift(val); | |
| 515 | +} | |
| 516 | + | |
| 517 | +/* | |
| 518 | + * mte_parent() - Get the parent of @node. | |
| 519 | + * @node: The encoded maple node. | |
| 520 | + * | |
| 521 | + * Return: The parent maple node. | |
| 522 | + */ | |
| 523 | +static inline struct maple_node *mte_parent(const struct maple_enode *enode) | |
| 524 | +{ | |
| 525 | + return (void *)((unsigned long) | |
| 526 | + (mte_to_node(enode)->parent) & ~MAPLE_NODE_MASK); | |
| 527 | +} | |
| 528 | + | |
| 529 | +/* | |
| 530 | + * ma_dead_node() - check if the @enode is dead. | |
| 531 | + * @enode: The encoded maple node | |
| 532 | + * | |
| 533 | + * Return: true if dead, false otherwise. | |
| 534 | + */ | |
| 535 | +static inline bool ma_dead_node(const struct maple_node *node) | |
| 536 | +{ | |
| 537 | + struct maple_node *parent = (void *)((unsigned long) | |
| 538 | + node->parent & ~MAPLE_NODE_MASK); | |
| 539 | + | |
| 540 | + return (parent == node); | |
| 541 | +} | |
| 542 | +/* | |
| 543 | + * mte_dead_node() - check if the @enode is dead. | |
| 544 | + * @enode: The encoded maple node | |
| 545 | + * | |
| 546 | + * Return: true if dead, false otherwise. | |
| 547 | + */ | |
| 548 | +static inline bool mte_dead_node(const struct maple_enode *enode) | |
| 549 | +{ | |
| 550 | + struct maple_node *parent, *node; | |
| 551 | + | |
| 552 | + node = mte_to_node(enode); | |
| 553 | + parent = mte_parent(enode); | |
| 554 | + return (parent == node); | |
| 555 | +} | |
| 556 | + | |
| 557 | +/* | |
| 558 | + * mas_allocated() - Get the number of nodes allocated in a maple state. | |
| 559 | + * @mas: The maple state | |
| 560 | + * | |
| 561 | + * The ma_state alloc member is overloaded to hold a pointer to the first | |
| 562 | + * allocated node or to the number of requested nodes to allocate. If bit 0 is | |
| 563 | + * set, then the alloc contains the number of requested nodes. If there is an | |
| 564 | + * allocated node, then the total allocated nodes is in that node. | |
| 565 | + * | |
| 566 | + * Return: The total number of nodes allocated | |
| 567 | + */ | |
| 568 | +static inline unsigned long mas_allocated(const struct ma_state *mas) | |
| 569 | +{ | |
| 570 | + if (!mas->alloc || ((unsigned long)mas->alloc & 0x1)) | |
| 571 | + return 0; | |
| 572 | + | |
| 573 | + return mas->alloc->total; | |
| 574 | +} | |
| 575 | + | |
| 576 | +/* | |
| 577 | + * mas_set_alloc_req() - Set the requested number of allocations. | |
| 578 | + * @mas: the maple state | |
| 579 | + * @count: the number of allocations. | |
| 580 | + * | |
| 581 | + * The requested number of allocations is either in the first allocated node, | |
| 582 | + * located in @mas->alloc->request_count, or directly in @mas->alloc if there is | |
| 583 | + * no allocated node. Set the request either in the node or do the necessary | |
| 584 | + * encoding to store in @mas->alloc directly. | |
| 585 | + */ | |
| 586 | +static inline void mas_set_alloc_req(struct ma_state *mas, unsigned long count) | |
| 587 | +{ | |
| 588 | + if (!mas->alloc || ((unsigned long)mas->alloc & 0x1)) { | |
| 589 | + if (!count) | |
| 590 | + mas->alloc = NULL; | |
| 591 | + else | |
| 592 | + mas->alloc = (struct maple_alloc *)(((count) << 1U) | 1U); | |
| 593 | + return; | |
| 594 | + } | |
| 595 | + | |
| 596 | + mas->alloc->request_count = count; | |
| 597 | +} | |
| 598 | + | |
| 599 | +/* | |
| 600 | + * mas_alloc_req() - get the requested number of allocations. | |
| 601 | + * @mas: The maple state | |
| 602 | + * | |
| 603 | + * The alloc count is either stored directly in @mas, or in | |
| 604 | + * @mas->alloc->request_count if there is at least one node allocated. Decode | |
| 605 | + * the request count if it's stored directly in @mas->alloc. | |
| 606 | + * | |
| 607 | + * Return: The allocation request count. | |
| 608 | + */ | |
| 609 | +static inline unsigned int mas_alloc_req(const struct ma_state *mas) | |
| 610 | +{ | |
| 611 | + if ((unsigned long)mas->alloc & 0x1) | |
| 612 | + return (unsigned long)(mas->alloc) >> 1; | |
| 613 | + else if (mas->alloc) | |
| 614 | + return mas->alloc->request_count; | |
| 615 | + return 0; | |
| 616 | +} | |
| 617 | + | |
| 618 | +/* | |
| 619 | + * ma_pivots() - Get a pointer to the maple node pivots. | |
| 620 | + * @node - the maple node | |
| 621 | + * @type - the node type | |
| 622 | + * | |
| 623 | + * Return: A pointer to the maple node pivots | |
| 624 | + */ | |
| 625 | +static inline unsigned long *ma_pivots(struct maple_node *node, | |
| 626 | + enum maple_type type) | |
| 627 | +{ | |
| 628 | + switch (type) { | |
| 629 | + case maple_arange_64: | |
| 630 | + return node->ma64.pivot; | |
| 631 | + case maple_range_64: | |
| 632 | + case maple_leaf_64: | |
| 633 | + return node->mr64.pivot; | |
| 634 | + case maple_dense: | |
| 635 | + return NULL; | |
| 636 | + } | |
| 637 | + return NULL; | |
| 638 | +} | |
| 639 | + | |
| 640 | +/* | |
| 641 | + * ma_gaps() - Get a pointer to the maple node gaps. | |
| 642 | + * @node - the maple node | |
| 643 | + * @type - the node type | |
| 644 | + * | |
| 645 | + * Return: A pointer to the maple node gaps | |
| 646 | + */ | |
| 647 | +static inline unsigned long *ma_gaps(struct maple_node *node, | |
| 648 | + enum maple_type type) | |
| 649 | +{ | |
| 650 | + switch (type) { | |
| 651 | + case maple_arange_64: | |
| 652 | + return node->ma64.gap; | |
| 653 | + case maple_range_64: | |
| 654 | + case maple_leaf_64: | |
| 655 | + case maple_dense: | |
| 656 | + return NULL; | |
| 657 | + } | |
| 658 | + return NULL; | |
| 659 | +} | |
| 660 | + | |
| 661 | +/* | |
| 662 | + * mte_pivot() - Get the pivot at @piv of the maple encoded node. | |
| 663 | + * @mn: The maple encoded node. | |
| 664 | + * @piv: The pivot. | |
| 665 | + * | |
| 666 | + * Return: the pivot at @piv of @mn. | |
| 667 | + */ | |
| 668 | +static inline unsigned long mte_pivot(const struct maple_enode *mn, | |
| 669 | + unsigned char piv) | |
| 670 | +{ | |
| 671 | + struct maple_node *node = mte_to_node(mn); | |
| 672 | + | |
| 673 | + if (piv >= mt_pivots[piv]) { | |
| 674 | + WARN_ON(1); | |
| 675 | + return 0; | |
| 676 | + } | |
| 677 | + switch (mte_node_type(mn)) { | |
| 678 | + case maple_arange_64: | |
| 679 | + return node->ma64.pivot[piv]; | |
| 680 | + case maple_range_64: | |
| 681 | + case maple_leaf_64: | |
| 682 | + return node->mr64.pivot[piv]; | |
| 683 | + case maple_dense: | |
| 684 | + return 0; | |
| 685 | + } | |
| 686 | + return 0; | |
| 687 | +} | |
| 688 | + | |
| 689 | +/* | |
| 690 | + * mas_safe_pivot() - get the pivot at @piv or mas->max. | |
| 691 | + * @mas: The maple state | |
| 692 | + * @pivots: The pointer to the maple node pivots | |
| 693 | + * @piv: The pivot to fetch | |
| 694 | + * @type: The maple node type | |
| 695 | + * | |
| 696 | + * Return: The pivot at @piv within the limit of the @pivots array, @mas->max | |
| 697 | + * otherwise. | |
| 698 | + */ | |
| 699 | +static inline unsigned long | |
| 700 | +mas_safe_pivot(const struct ma_state *mas, unsigned long *pivots, | |
| 701 | + unsigned char piv, enum maple_type type) | |
| 702 | +{ | |
| 703 | + if (piv >= mt_pivots[type]) | |
| 704 | + return mas->max; | |
| 705 | + | |
| 706 | + return pivots[piv]; | |
| 707 | +} | |
| 708 | + | |
| 709 | +/* | |
| 710 | + * mas_safe_min() - Return the minimum for a given offset. | |
| 711 | + * @mas: The maple state | |
| 712 | + * @pivots: The pointer to the maple node pivots | |
| 713 | + * @offset: The offset into the pivot array | |
| 714 | + * | |
| 715 | + * Return: The minimum range value that is contained in @offset. | |
| 716 | + */ | |
| 717 | +static inline unsigned long | |
| 718 | +mas_safe_min(struct ma_state *mas, unsigned long *pivots, unsigned char offset) | |
| 719 | +{ | |
| 720 | + if (likely(offset)) | |
| 721 | + return pivots[offset - 1] + 1; | |
| 722 | + | |
| 723 | + return mas->min; | |
| 724 | +} | |
| 725 | + | |
| 726 | +/* | |
| 727 | + * mas_logical_pivot() - Get the logical pivot of a given offset. | |
| 728 | + * @mas: The maple state | |
| 729 | + * @pivots: The pointer to the maple node pivots | |
| 730 | + * @offset: The offset into the pivot array | |
| 731 | + * @type: The maple node type | |
| 732 | + * | |
| 733 | + * When there is no value at a pivot (beyond the end of the data), then the | |
| 734 | + * pivot is actually @mas->max. | |
| 735 | + * | |
| 736 | + * Return: the logical pivot of a given @offset. | |
| 737 | + */ | |
| 738 | +static inline unsigned long | |
| 739 | +mas_logical_pivot(struct ma_state *mas, unsigned long *pivots, | |
| 740 | + unsigned char offset, enum maple_type type) | |
| 741 | +{ | |
| 742 | + unsigned long lpiv = mas_safe_pivot(mas, pivots, offset, type); | |
| 743 | + | |
| 744 | + if (likely(lpiv)) | |
| 745 | + return lpiv; | |
| 746 | + | |
| 747 | + if (likely(offset)) | |
| 748 | + return mas->max; | |
| 749 | + | |
| 750 | + return lpiv; | |
| 751 | +} | |
| 752 | + | |
| 753 | +/* | |
| 754 | + * mte_set_pivot() - Set a pivot to a value in an encoded maple node. | |
| 755 | + * @mn: The encoded maple node | |
| 756 | + * @piv: The pivot offset | |
| 757 | + * @val: The value of the pivot | |
| 758 | + */ | |
| 759 | +static inline void mte_set_pivot(struct maple_enode *mn, unsigned char piv, | |
| 760 | + unsigned long val) | |
| 761 | +{ | |
| 762 | + struct maple_node *node = mte_to_node(mn); | |
| 763 | + enum maple_type type = mte_node_type(mn); | |
| 764 | + | |
| 765 | + BUG_ON(piv >= mt_pivots[type]); | |
| 766 | + switch (type) { | |
| 767 | + default: | |
| 768 | + case maple_range_64: | |
| 769 | + case maple_leaf_64: | |
| 770 | + node->mr64.pivot[piv] = val; | |
| 771 | + break; | |
| 772 | + case maple_arange_64: | |
| 773 | + node->ma64.pivot[piv] = val; | |
| 774 | + break; | |
| 775 | + case maple_dense: | |
| 776 | + break; | |
| 777 | + } | |
| 778 | + | |
| 779 | +} | |
| 780 | + | |
| 781 | +/* | |
| 782 | + * ma_slots() - Get a pointer to the maple node slots. | |
| 783 | + * @mn: The maple node | |
| 784 | + * @mt: The maple node type | |
| 785 | + * | |
| 786 | + * Return: A pointer to the maple node slots | |
| 787 | + */ | |
| 788 | +static inline void __rcu **ma_slots(struct maple_node *mn, enum maple_type mt) | |
| 789 | +{ | |
| 790 | + switch (mt) { | |
| 791 | + default: | |
| 792 | + case maple_arange_64: | |
| 793 | + return mn->ma64.slot; | |
| 794 | + case maple_range_64: | |
| 795 | + case maple_leaf_64: | |
| 796 | + return mn->mr64.slot; | |
| 797 | + case maple_dense: | |
| 798 | + return mn->slot; | |
| 799 | + } | |
| 800 | +} | |
| 801 | + | |
| 802 | +static inline bool mt_locked(const struct maple_tree *mt) | |
| 803 | +{ | |
| 804 | + return mt_external_lock(mt) ? mt_lock_is_held(mt) : | |
| 805 | + lockdep_is_held(&mt->ma_lock); | |
| 806 | +} | |
| 807 | + | |
| 808 | +static inline void *mt_slot(const struct maple_tree *mt, | |
| 809 | + void __rcu **slots, unsigned char offset) | |
| 810 | +{ | |
| 811 | + return rcu_dereference_check(slots[offset], mt_locked(mt)); | |
| 812 | +} | |
| 813 | + | |
| 814 | +/* | |
| 815 | + * mas_slot_locked() - Get the slot value when holding the maple tree lock. | |
| 816 | + * @mas: The maple state | |
| 817 | + * @slots: The pointer to the slots | |
| 818 | + * @offset: The offset into the slots array to fetch | |
| 819 | + * | |
| 820 | + * Return: The entry stored in @slots at the @offset. | |
| 821 | + */ | |
| 822 | +static inline void *mas_slot_locked(struct ma_state *mas, void __rcu **slots, | |
| 823 | + unsigned char offset) | |
| 824 | +{ | |
| 825 | + return rcu_dereference_protected(slots[offset], mt_locked(mas->tree)); | |
| 826 | +} | |
| 827 | + | |
| 828 | +/* | |
| 829 | + * mas_slot() - Get the slot value when not holding the maple tree lock. | |
| 830 | + * @mas: The maple state | |
| 831 | + * @slots: The pointer to the slots | |
| 832 | + * @offset: The offset into the slots array to fetch | |
| 833 | + * | |
| 834 | + * Return: The entry stored in @slots at the @offset | |
| 835 | + */ | |
| 836 | +static inline void *mas_slot(struct ma_state *mas, void __rcu **slots, | |
| 837 | + unsigned char offset) | |
| 838 | +{ | |
| 839 | + return mt_slot(mas->tree, slots, offset); | |
| 840 | +} | |
| 841 | + | |
| 842 | +/* | |
| 843 | + * mas_root() - Get the maple tree root. | |
| 844 | + * @mas: The maple state. | |
| 845 | + * | |
| 846 | + * Return: The pointer to the root of the tree | |
| 847 | + */ | |
| 848 | +static inline void *mas_root(struct ma_state *mas) | |
| 849 | +{ | |
| 850 | + return rcu_dereference_check(mas->tree->ma_root, mt_locked(mas->tree)); | |
| 851 | +} | |
| 852 | + | |
| 853 | +static inline void *mt_root_locked(struct maple_tree *mt) | |
| 854 | +{ | |
| 855 | + return rcu_dereference_protected(mt->ma_root, mt_locked(mt)); | |
| 856 | +} | |
| 857 | + | |
| 858 | +/* | |
| 859 | + * mas_root_locked() - Get the maple tree root when holding the maple tree lock. | |
| 860 | + * @mas: The maple state. | |
| 861 | + * | |
| 862 | + * Return: The pointer to the root of the tree | |
| 863 | + */ | |
| 864 | +static inline void *mas_root_locked(struct ma_state *mas) | |
| 865 | +{ | |
| 866 | + return mt_root_locked(mas->tree); | |
| 867 | +} | |
| 868 | + | |
| 869 | +static inline struct maple_metadata *ma_meta(struct maple_node *mn, | |
| 870 | + enum maple_type mt) | |
| 871 | +{ | |
| 872 | + switch (mt) { | |
| 873 | + case maple_arange_64: | |
| 874 | + return &mn->ma64.meta; | |
| 875 | + default: | |
| 876 | + return &mn->mr64.meta; | |
| 877 | + } | |
| 878 | +} | |
| 879 | + | |
| 880 | +/* | |
| 881 | + * ma_set_meta() - Set the metadata information of a node. | |
| 882 | + * @mn: The maple node | |
| 883 | + * @mt: The maple node type | |
| 884 | + * @offset: The offset of the highest sub-gap in this node. | |
| 885 | + * @end: The end of the data in this node. | |
| 886 | + */ | |
| 887 | +static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, | |
| 888 | + unsigned char offset, unsigned char end) | |
| 889 | +{ | |
| 890 | + struct maple_metadata *meta = ma_meta(mn, mt); | |
| 891 | + | |
| 892 | + meta->gap = offset; | |
| 893 | + meta->end = end; | |
| 894 | +} | |
| 895 | + | |
| 896 | +/* | |
| 897 | + * ma_meta_end() - Get the data end of a node from the metadata | |
| 898 | + * @mn: The maple node | |
| 899 | + * @mt: The maple node type | |
| 900 | + */ | |
| 901 | +static inline unsigned char ma_meta_end(struct maple_node *mn, | |
| 902 | + enum maple_type mt) | |
| 903 | +{ | |
| 904 | + struct maple_metadata *meta = ma_meta(mn, mt); | |
| 905 | + | |
| 906 | + return meta->end; | |
| 907 | +} | |
| 908 | + | |
| 909 | +/* | |
| 910 | + * ma_meta_gap() - Get the largest gap location of a node from the metadata | |
| 911 | + * @mn: The maple node | |
| 912 | + * @mt: The maple node type | |
| 913 | + */ | |
| 914 | +static inline unsigned char ma_meta_gap(struct maple_node *mn, | |
| 915 | + enum maple_type mt) | |
| 916 | +{ | |
| 917 | + BUG_ON(mt != maple_arange_64); | |
| 918 | + | |
| 919 | + return mn->ma64.meta.gap; | |
| 920 | +} | |
| 921 | + | |
| 922 | +/* | |
| 923 | + * ma_set_meta_gap() - Set the largest gap location in a nodes metadata | |
| 924 | + * @mn: The maple node | |
| 925 | + * @mn: The maple node type | |
| 926 | + * @offset: The location of the largest gap. | |
| 927 | + */ | |
| 928 | +static inline void ma_set_meta_gap(struct maple_node *mn, enum maple_type mt, | |
| 929 | + unsigned char offset) | |
| 930 | +{ | |
| 931 | + | |
| 932 | + struct maple_metadata *meta = ma_meta(mn, mt); | |
| 933 | + | |
| 934 | + meta->gap = offset; | |
| 935 | +} | |
| 936 | + | |
| 937 | +/* | |
| 938 | + * mat_add() - Add a @dead_enode to the ma_topiary of a list of dead nodes. | |
| 939 | + * @mat - the ma_topiary, a linked list of dead nodes. | |
| 940 | + * @dead_enode - the node to be marked as dead and added to the tail of the list | |
| 941 | + * | |
| 942 | + * Add the @dead_enode to the linked list in @mat. | |
| 943 | + */ | |
| 944 | +static inline void mat_add(struct ma_topiary *mat, | |
| 945 | + struct maple_enode *dead_enode) | |
| 946 | +{ | |
| 947 | + mte_set_node_dead(dead_enode); | |
| 948 | + mte_to_mat(dead_enode)->next = NULL; | |
| 949 | + if (!mat->tail) { | |
| 950 | + mat->tail = mat->head = dead_enode; | |
| 951 | + return; | |
| 952 | + } | |
| 953 | + | |
| 954 | + mte_to_mat(mat->tail)->next = dead_enode; | |
| 955 | + mat->tail = dead_enode; | |
| 956 | +} | |
| 957 | + | |
| 958 | +static void mte_destroy_walk(struct maple_enode *, struct maple_tree *); | |
| 959 | +static inline void mas_free(struct ma_state *mas, struct maple_enode *used); | |
| 960 | + | |
| 961 | +/* | |
| 962 | + * mas_mat_free() - Free all nodes in a dead list. | |
| 963 | + * @mas - the maple state | |
| 964 | + * @mat - the ma_topiary linked list of dead nodes to free. | |
| 965 | + * | |
| 966 | + * Free walk a dead list. | |
| 967 | + */ | |
| 968 | +static void mas_mat_free(struct ma_state *mas, struct ma_topiary *mat) | |
| 969 | +{ | |
| 970 | + struct maple_enode *next; | |
| 971 | + | |
| 972 | + while (mat->head) { | |
| 973 | + next = mte_to_mat(mat->head)->next; | |
| 974 | + mas_free(mas, mat->head); | |
| 975 | + mat->head = next; | |
| 976 | + } | |
| 977 | +} | |
| 978 | + | |
| 979 | +/* | |
| 980 | + * mas_mat_destroy() - Free all nodes and subtrees in a dead list. | |
| 981 | + * @mas - the maple state | |
| 982 | + * @mat - the ma_topiary linked list of dead nodes to free. | |
| 983 | + * | |
| 984 | + * Destroy walk a dead list. | |
| 985 | + */ | |
| 986 | +static void mas_mat_destroy(struct ma_state *mas, struct ma_topiary *mat) | |
| 987 | +{ | |
| 988 | + struct maple_enode *next; | |
| 989 | + | |
| 990 | + while (mat->head) { | |
| 991 | + next = mte_to_mat(mat->head)->next; | |
| 992 | + mte_destroy_walk(mat->head, mat->mtree); | |
| 993 | + mat->head = next; | |
| 994 | + } | |
| 995 | +} | |
| 996 | +/* | |
| 997 | + * mas_descend() - Descend into the slot stored in the ma_state. | |
| 998 | + * @mas - the maple state. | |
| 999 | + * | |
| 1000 | + * Note: Not RCU safe, only use in write side or debug code. | |
| 1001 | + */ | |
| 1002 | +static inline void mas_descend(struct ma_state *mas) | |
| 1003 | +{ | |
| 1004 | + enum maple_type type; | |
| 1005 | + unsigned long *pivots; | |
| 1006 | + struct maple_node *node; | |
| 1007 | + void __rcu **slots; | |
| 1008 | + | |
| 1009 | + node = mas_mn(mas); | |
| 1010 | + type = mte_node_type(mas->node); | |
| 1011 | + pivots = ma_pivots(node, type); | |
| 1012 | + slots = ma_slots(node, type); | |
| 1013 | + | |
| 1014 | + if (mas->offset) | |
| 1015 | + mas->min = pivots[mas->offset - 1] + 1; | |
| 1016 | + mas->max = mas_safe_pivot(mas, pivots, mas->offset, type); | |
| 1017 | + mas->node = mas_slot(mas, slots, mas->offset); | |
| 1018 | +} | |
| 1019 | + | |
| 1020 | +/* | |
| 1021 | + * mte_set_gap() - Set a maple node gap. | |
| 1022 | + * @mn: The encoded maple node | |
| 1023 | + * @gap: The offset of the gap to set | |
| 1024 | + * @val: The gap value | |
| 1025 | + */ | |
| 1026 | +static inline void mte_set_gap(const struct maple_enode *mn, | |
| 1027 | + unsigned char gap, unsigned long val) | |
| 1028 | +{ | |
| 1029 | + switch (mte_node_type(mn)) { | |
| 1030 | + default: | |
| 1031 | + break; | |
| 1032 | + case maple_arange_64: | |
| 1033 | + mte_to_node(mn)->ma64.gap[gap] = val; | |
| 1034 | + break; | |
| 1035 | + } | |
| 1036 | +} | |
| 1037 | + | |
| 1038 | +/* | |
| 1039 | + * mas_ascend() - Walk up a level of the tree. | |
| 1040 | + * @mas: The maple state | |
| 1041 | + * | |
| 1042 | + * Sets the @mas->max and @mas->min to the correct values when walking up. This | |
| 1043 | + * may cause several levels of walking up to find the correct min and max. | |
| 1044 | + * May find a dead node which will cause a premature return. | |
| 1045 | + * Return: 1 on dead node, 0 otherwise | |
| 1046 | + */ | |
| 1047 | +static int mas_ascend(struct ma_state *mas) | |
| 1048 | +{ | |
| 1049 | + struct maple_enode *p_enode; /* parent enode. */ | |
| 1050 | + struct maple_enode *a_enode; /* ancestor enode. */ | |
| 1051 | + struct maple_node *a_node; /* ancestor node. */ | |
| 1052 | + struct maple_node *p_node; /* parent node. */ | |
| 1053 | + unsigned char a_slot; | |
| 1054 | + enum maple_type a_type; | |
| 1055 | + unsigned long min, max; | |
| 1056 | + unsigned long *pivots; | |
| 1057 | + unsigned char offset; | |
| 1058 | + bool set_max = false, set_min = false; | |
| 1059 | + | |
| 1060 | + a_node = mas_mn(mas); | |
| 1061 | + if (ma_is_root(a_node)) { | |
| 1062 | + mas->offset = 0; | |
| 1063 | + return 0; | |
| 1064 | + } | |
| 1065 | + | |
| 1066 | + p_node = mte_parent(mas->node); | |
| 1067 | + if (unlikely(a_node == p_node)) | |
| 1068 | + return 1; | |
| 1069 | + a_type = mas_parent_enum(mas, mas->node); | |
| 1070 | + offset = mte_parent_slot(mas->node); | |
| 1071 | + a_enode = mt_mk_node(p_node, a_type); | |
| 1072 | + | |
| 1073 | + /* Check to make sure all parent information is still accurate */ | |
| 1074 | + if (p_node != mte_parent(mas->node)) | |
| 1075 | + return 1; | |
| 1076 | + | |
| 1077 | + mas->node = a_enode; | |
| 1078 | + mas->offset = offset; | |
| 1079 | + | |
| 1080 | + if (mte_is_root(a_enode)) { | |
| 1081 | + mas->max = ULONG_MAX; | |
| 1082 | + mas->min = 0; | |
| 1083 | + return 0; | |
| 1084 | + } | |
| 1085 | + | |
| 1086 | + min = 0; | |
| 1087 | + max = ULONG_MAX; | |
| 1088 | + do { | |
| 1089 | + p_enode = a_enode; | |
| 1090 | + a_type = mas_parent_enum(mas, p_enode); | |
| 1091 | + a_node = mte_parent(p_enode); | |
| 1092 | + a_slot = mte_parent_slot(p_enode); | |
| 1093 | + pivots = ma_pivots(a_node, a_type); | |
| 1094 | + a_enode = mt_mk_node(a_node, a_type); | |
| 1095 | + | |
| 1096 | + if (!set_min && a_slot) { | |
| 1097 | + set_min = true; | |
| 1098 | + min = pivots[a_slot - 1] + 1; | |
| 1099 | + } | |
| 1100 | + | |
| 1101 | + if (!set_max && a_slot < mt_pivots[a_type]) { | |
| 1102 | + set_max = true; | |
| 1103 | + max = pivots[a_slot]; | |
| 1104 | + } | |
| 1105 | + | |
| 1106 | + if (unlikely(ma_dead_node(a_node))) | |
| 1107 | + return 1; | |
| 1108 | + | |
| 1109 | + if (unlikely(ma_is_root(a_node))) | |
| 1110 | + break; | |
| 1111 | + | |
| 1112 | + } while (!set_min || !set_max); | |
| 1113 | + | |
| 1114 | + mas->max = max; | |
| 1115 | + mas->min = min; | |
| 1116 | + return 0; | |
| 1117 | +} | |
| 1118 | + | |
| 1119 | +/* | |
| 1120 | + * mas_pop_node() - Get a previously allocated maple node from the maple state. | |
| 1121 | + * @mas: The maple state | |
| 1122 | + * | |
| 1123 | + * Return: A pointer to a maple node. | |
| 1124 | + */ | |
| 1125 | +static inline struct maple_node *mas_pop_node(struct ma_state *mas) | |
| 1126 | +{ | |
| 1127 | + struct maple_alloc *ret, *node = mas->alloc; | |
| 1128 | + unsigned long total = mas_allocated(mas); | |
| 1129 | + | |
| 1130 | + /* nothing or a request pending. */ | |
| 1131 | + if (unlikely(!total)) | |
| 1132 | + return NULL; | |
| 1133 | + | |
| 1134 | + if (total == 1) { | |
| 1135 | + /* single allocation in this ma_state */ | |
| 1136 | + mas->alloc = NULL; | |
| 1137 | + ret = node; | |
| 1138 | + goto single_node; | |
| 1139 | + } | |
| 1140 | + | |
| 1141 | + if (!node->node_count) { | |
| 1142 | + /* Single allocation in this node. */ | |
| 1143 | + mas->alloc = node->slot[0]; | |
| 1144 | + node->slot[0] = NULL; | |
| 1145 | + mas->alloc->total = node->total - 1; | |
| 1146 | + ret = node; | |
| 1147 | + goto new_head; | |
| 1148 | + } | |
| 1149 | + | |
| 1150 | + node->total--; | |
| 1151 | + ret = node->slot[node->node_count]; | |
| 1152 | + node->slot[node->node_count--] = NULL; | |
| 1153 | + | |
| 1154 | +single_node: | |
| 1155 | +new_head: | |
| 1156 | + ret->total = 0; | |
| 1157 | + ret->node_count = 0; | |
| 1158 | + if (ret->request_count) { | |
| 1159 | + mas_set_alloc_req(mas, ret->request_count + 1); | |
| 1160 | + ret->request_count = 0; | |
| 1161 | + } | |
| 1162 | + return (struct maple_node *)ret; | |
| 1163 | +} | |
| 1164 | + | |
| 1165 | +/* | |
| 1166 | + * mas_push_node() - Push a node back on the maple state allocation. | |
| 1167 | + * @mas: The maple state | |
| 1168 | + * @used: The used maple node | |
| 1169 | + * | |
| 1170 | + * Stores the maple node back into @mas->alloc for reuse. Updates allocated and | |
| 1171 | + * requested node count as necessary. | |
| 1172 | + */ | |
| 1173 | +static inline void mas_push_node(struct ma_state *mas, struct maple_node *used) | |
| 1174 | +{ | |
| 1175 | + struct maple_alloc *reuse = (struct maple_alloc *)used; | |
| 1176 | + struct maple_alloc *head = mas->alloc; | |
| 1177 | + unsigned long count; | |
| 1178 | + unsigned int requested = mas_alloc_req(mas); | |
| 1179 | + | |
| 1180 | + memset(reuse, 0, sizeof(*reuse)); | |
| 1181 | + count = mas_allocated(mas); | |
| 1182 | + | |
| 1183 | + if (count && (head->node_count < MAPLE_ALLOC_SLOTS - 1)) { | |
| 1184 | + if (head->slot[0]) | |
| 1185 | + head->node_count++; | |
| 1186 | + head->slot[head->node_count] = reuse; | |
| 1187 | + head->total++; | |
| 1188 | + goto done; | |
| 1189 | + } | |
| 1190 | + | |
| 1191 | + reuse->total = 1; | |
| 1192 | + if ((head) && !((unsigned long)head & 0x1)) { | |
| 1193 | + head->request_count = 0; | |
| 1194 | + reuse->slot[0] = head; | |
| 1195 | + reuse->total += head->total; | |
| 1196 | + } | |
| 1197 | + | |
| 1198 | + mas->alloc = reuse; | |
| 1199 | +done: | |
| 1200 | + if (requested > 1) | |
| 1201 | + mas_set_alloc_req(mas, requested - 1); | |
| 1202 | +} | |
| 1203 | + | |
| 1204 | +/* | |
| 1205 | + * mas_alloc_nodes() - Allocate nodes into a maple state | |
| 1206 | + * @mas: The maple state | |
| 1207 | + * @gfp: The GFP Flags | |
| 1208 | + */ | |
| 1209 | +static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp) | |
| 1210 | +{ | |
| 1211 | + struct maple_alloc *node; | |
| 1212 | + struct maple_alloc **nodep = &mas->alloc; | |
| 1213 | + unsigned long allocated = mas_allocated(mas); | |
| 1214 | + unsigned long success = allocated; | |
| 1215 | + unsigned int requested = mas_alloc_req(mas); | |
| 1216 | + unsigned int count; | |
| 1217 | + void **slots = NULL; | |
| 1218 | + unsigned int max_req = 0; | |
| 1219 | + | |
| 1220 | + if (!requested) | |
| 1221 | + return; | |
| 1222 | + | |
| 1223 | + mas_set_alloc_req(mas, 0); | |
| 1224 | + if (mas->mas_flags & MA_STATE_PREALLOC) { | |
| 1225 | + if (allocated) | |
| 1226 | + return; | |
| 1227 | + WARN_ON(!allocated); | |
| 1228 | + } | |
| 1229 | + | |
| 1230 | + if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS - 1) { | |
| 1231 | + node = (struct maple_alloc *)mt_alloc_one(gfp); | |
| 1232 | + if (!node) | |
| 1233 | + goto nomem_one; | |
| 1234 | + | |
| 1235 | + if (allocated) | |
| 1236 | + node->slot[0] = mas->alloc; | |
| 1237 | + | |
| 1238 | + success++; | |
| 1239 | + mas->alloc = node; | |
| 1240 | + requested--; | |
| 1241 | + } | |
| 1242 | + | |
| 1243 | + node = mas->alloc; | |
| 1244 | + while (requested) { | |
| 1245 | + max_req = MAPLE_ALLOC_SLOTS; | |
| 1246 | + if (node->slot[0]) { | |
| 1247 | + unsigned int offset = node->node_count + 1; | |
| 1248 | + | |
| 1249 | + slots = (void **)&node->slot[offset]; | |
| 1250 | + max_req -= offset; | |
| 1251 | + } else { | |
| 1252 | + slots = (void **)&node->slot; | |
| 1253 | + } | |
| 1254 | + | |
| 1255 | + max_req = min(requested, max_req); | |
| 1256 | + count = mt_alloc_bulk(gfp, max_req, slots); | |
| 1257 | + if (!count) | |
| 1258 | + goto nomem_bulk; | |
| 1259 | + | |
| 1260 | + node->node_count += count; | |
| 1261 | + /* zero indexed. */ | |
| 1262 | + if (slots == (void **)&node->slot) | |
| 1263 | + node->node_count--; | |
| 1264 | + | |
| 1265 | + success += count; | |
| 1266 | + nodep = &node->slot[0]; | |
| 1267 | + node = *nodep; | |
| 1268 | + requested -= count; | |
| 1269 | + } | |
| 1270 | + mas->alloc->total = success; | |
| 1271 | + return; | |
| 1272 | + | |
| 1273 | +nomem_bulk: | |
| 1274 | + /* Clean up potential freed allocations on bulk failure */ | |
| 1275 | + memset(slots, 0, max_req * sizeof(unsigned long)); | |
| 1276 | +nomem_one: | |
| 1277 | + mas_set_alloc_req(mas, requested); | |
| 1278 | + if (mas->alloc && !(((unsigned long)mas->alloc & 0x1))) | |
| 1279 | + mas->alloc->total = success; | |
| 1280 | + mas_set_err(mas, -ENOMEM); | |
| 1281 | + return; | |
| 1282 | + | |
| 1283 | +} | |
| 1284 | + | |
| 1285 | +/* | |
| 1286 | + * mas_free() - Free an encoded maple node | |
| 1287 | + * @mas: The maple state | |
| 1288 | + * @used: The encoded maple node to free. | |
| 1289 | + * | |
| 1290 | + * Uses rcu free if necessary, pushes @used back on the maple state allocations | |
| 1291 | + * otherwise. | |
| 1292 | + */ | |
| 1293 | +static inline void mas_free(struct ma_state *mas, struct maple_enode *used) | |
| 1294 | +{ | |
| 1295 | + struct maple_node *tmp = mte_to_node(used); | |
| 1296 | + | |
| 1297 | + if (mt_in_rcu(mas->tree)) | |
| 1298 | + ma_free_rcu(tmp); | |
| 1299 | + else | |
| 1300 | + mas_push_node(mas, tmp); | |
| 1301 | +} | |
| 1302 | + | |
| 1303 | +/* | |
| 1304 | + * mas_node_count() - Check if enough nodes are allocated and request more if | |
| 1305 | + * there is not enough nodes. | |
| 1306 | + * @mas: The maple state | |
| 1307 | + * @count: The number of nodes needed | |
| 1308 | + * @gfp: the gfp flags | |
| 1309 | + */ | |
| 1310 | +static void mas_node_count_gfp(struct ma_state *mas, int count, gfp_t gfp) | |
| 1311 | +{ | |
| 1312 | + unsigned long allocated = mas_allocated(mas); | |
| 1313 | + | |
| 1314 | + if (allocated < count) { | |
| 1315 | + mas_set_alloc_req(mas, count - allocated); | |
| 1316 | + mas_alloc_nodes(mas, gfp); | |
| 1317 | + } | |
| 1318 | +} | |
| 1319 | + | |
| 1320 | +/* | |
| 1321 | + * mas_node_count() - Check if enough nodes are allocated and request more if | |
| 1322 | + * there is not enough nodes. | |
| 1323 | + * @mas: The maple state | |
| 1324 | + * @count: The number of nodes needed | |
| 1325 | + * | |
| 1326 | + * Note: Uses GFP_NOWAIT | __GFP_NOWARN for gfp flags. | |
| 1327 | + */ | |
| 1328 | +static void mas_node_count(struct ma_state *mas, int count) | |
| 1329 | +{ | |
| 1330 | + return mas_node_count_gfp(mas, count, GFP_NOWAIT | __GFP_NOWARN); | |
| 1331 | +} | |
| 1332 | + | |
| 1333 | +/* | |
| 1334 | + * mas_start() - Sets up maple state for operations. | |
| 1335 | + * @mas: The maple state. | |
| 1336 | + * | |
| 1337 | + * If mas->node == MAS_START, then set the min, max, depth, and offset to | |
| 1338 | + * defaults. | |
| 1339 | + * | |
| 1340 | + * Return: | |
| 1341 | + * - If mas->node is an error or not MAS_START, return NULL. | |
| 1342 | + * - If it's an empty tree: NULL & mas->node == MAS_NONE | |
| 1343 | + * - If it's a single entry: The entry & mas->node == MAS_ROOT | |
| 1344 | + * - If it's a tree: NULL & mas->node == safe root node. | |
| 1345 | + */ | |
| 1346 | +static inline struct maple_enode *mas_start(struct ma_state *mas) | |
| 1347 | +{ | |
| 1348 | + if (likely(mas_is_start(mas))) { | |
| 1349 | + struct maple_enode *root; | |
| 1350 | + | |
| 1351 | + mas->node = MAS_NONE; | |
| 1352 | + mas->min = 0; | |
| 1353 | + mas->max = ULONG_MAX; | |
| 1354 | + mas->depth = 0; | |
| 1355 | + mas->offset = 0; | |
| 1356 | + | |
| 1357 | + root = mas_root(mas); | |
| 1358 | + /* Tree with nodes */ | |
| 1359 | + if (likely(xa_is_node(root))) { | |
| 1360 | + mas->node = mte_safe_root(root); | |
| 1361 | + return NULL; | |
| 1362 | + } | |
| 1363 | + | |
| 1364 | + /* empty tree */ | |
| 1365 | + if (unlikely(!root)) { | |
| 1366 | + mas->offset = MAPLE_NODE_SLOTS; | |
| 1367 | + return NULL; | |
| 1368 | + } | |
| 1369 | + | |
| 1370 | + /* Single entry tree */ | |
| 1371 | + mas->node = MAS_ROOT; | |
| 1372 | + mas->offset = MAPLE_NODE_SLOTS; | |
| 1373 | + | |
| 1374 | + /* Single entry tree. */ | |
| 1375 | + if (mas->index > 0) | |
| 1376 | + return NULL; | |
| 1377 | + | |
| 1378 | + return root; | |
| 1379 | + } | |
| 1380 | + | |
| 1381 | + return NULL; | |
| 1382 | +} | |
| 1383 | + | |
| 1384 | +/* | |
| 1385 | + * ma_data_end() - Find the end of the data in a node. | |
| 1386 | + * @node: The maple node | |
| 1387 | + * @type: The maple node type | |
| 1388 | + * @pivots: The array of pivots in the node | |
| 1389 | + * @max: The maximum value in the node | |
| 1390 | + * | |
| 1391 | + * Uses metadata to find the end of the data when possible. | |
| 1392 | + * Return: The zero indexed last slot with data (may be null). | |
| 1393 | + */ | |
| 1394 | +static inline unsigned char ma_data_end(struct maple_node *node, | |
| 1395 | + enum maple_type type, | |
| 1396 | + unsigned long *pivots, | |
| 1397 | + unsigned long max) | |
| 1398 | +{ | |
| 1399 | + unsigned char offset; | |
| 1400 | + | |
| 1401 | + if (type == maple_arange_64) | |
| 1402 | + return ma_meta_end(node, type); | |
| 1403 | + | |
| 1404 | + offset = mt_pivots[type] - 1; | |
| 1405 | + if (likely(!pivots[offset])) | |
| 1406 | + return ma_meta_end(node, type); | |
| 1407 | + | |
| 1408 | + if (likely(pivots[offset] == max)) | |
| 1409 | + return offset; | |
| 1410 | + | |
| 1411 | + return mt_pivots[type]; | |
| 1412 | +} | |
| 1413 | + | |
| 1414 | +/* | |
| 1415 | + * mas_data_end() - Find the end of the data (slot). | |
| 1416 | + * @mas: the maple state | |
| 1417 | + * | |
| 1418 | + * This method is optimized to check the metadata of a node if the node type | |
| 1419 | + * supports data end metadata. | |
| 1420 | + * | |
| 1421 | + * Return: The zero indexed last slot with data (may be null). | |
| 1422 | + */ | |
| 1423 | +static inline unsigned char mas_data_end(struct ma_state *mas) | |
| 1424 | +{ | |
| 1425 | + enum maple_type type; | |
| 1426 | + struct maple_node *node; | |
| 1427 | + unsigned char offset; | |
| 1428 | + unsigned long *pivots; | |
| 1429 | + | |
| 1430 | + type = mte_node_type(mas->node); | |
| 1431 | + node = mas_mn(mas); | |
| 1432 | + if (type == maple_arange_64) | |
| 1433 | + return ma_meta_end(node, type); | |
| 1434 | + | |
| 1435 | + pivots = ma_pivots(node, type); | |
| 1436 | + offset = mt_pivots[type] - 1; | |
| 1437 | + if (likely(!pivots[offset])) | |
| 1438 | + return ma_meta_end(node, type); | |
| 1439 | + | |
| 1440 | + if (likely(pivots[offset] == mas->max)) | |
| 1441 | + return offset; | |
| 1442 | + | |
| 1443 | + return mt_pivots[type]; | |
| 1444 | +} | |
| 1445 | + | |
| 1446 | +/* | |
| 1447 | + * mas_leaf_max_gap() - Returns the largest gap in a leaf node | |
| 1448 | + * @mas - the maple state | |
| 1449 | + * | |
| 1450 | + * Return: The maximum gap in the leaf. | |
| 1451 | + */ | |
| 1452 | +static unsigned long mas_leaf_max_gap(struct ma_state *mas) | |
| 1453 | +{ | |
| 1454 | + enum maple_type mt; | |
| 1455 | + unsigned long pstart, gap, max_gap; | |
| 1456 | + struct maple_node *mn; | |
| 1457 | + unsigned long *pivots; | |
| 1458 | + void __rcu **slots; | |
| 1459 | + unsigned char i; | |
| 1460 | + unsigned char max_piv; | |
| 1461 | + | |
| 1462 | + mt = mte_node_type(mas->node); | |
| 1463 | + mn = mas_mn(mas); | |
| 1464 | + slots = ma_slots(mn, mt); | |
| 1465 | + max_gap = 0; | |
| 1466 | + if (unlikely(ma_is_dense(mt))) { | |
| 1467 | + gap = 0; | |
| 1468 | + for (i = 0; i < mt_slots[mt]; i++) { | |
| 1469 | + if (slots[i]) { | |
| 1470 | + if (gap > max_gap) | |
| 1471 | + max_gap = gap; | |
| 1472 | + gap = 0; | |
| 1473 | + } else { | |
| 1474 | + gap++; | |
| 1475 | + } | |
| 1476 | + } | |
| 1477 | + if (gap > max_gap) | |
| 1478 | + max_gap = gap; | |
| 1479 | + return max_gap; | |
| 1480 | + } | |
| 1481 | + | |
| 1482 | + /* | |
| 1483 | + * Check the first implied pivot optimizes the loop below and slot 1 may | |
| 1484 | + * be skipped if there is a gap in slot 0. | |
| 1485 | + */ | |
| 1486 | + pivots = ma_pivots(mn, mt); | |
| 1487 | + if (likely(!slots[0])) { | |
| 1488 | + max_gap = pivots[0] - mas->min + 1; | |
| 1489 | + i = 2; | |
| 1490 | + } else { | |
| 1491 | + i = 1; | |
| 1492 | + } | |
| 1493 | + | |
| 1494 | + /* reduce max_piv as the special case is checked before the loop */ | |
| 1495 | + max_piv = ma_data_end(mn, mt, pivots, mas->max) - 1; | |
| 1496 | + /* | |
| 1497 | + * Check end implied pivot which can only be a gap on the right most | |
| 1498 | + * node. | |
| 1499 | + */ | |
| 1500 | + if (unlikely(mas->max == ULONG_MAX) && !slots[max_piv + 1]) { | |
| 1501 | + gap = ULONG_MAX - pivots[max_piv]; | |
| 1502 | + if (gap > max_gap) | |
| 1503 | + max_gap = gap; | |
| 1504 | + } | |
| 1505 | + | |
| 1506 | + for (; i <= max_piv; i++) { | |
| 1507 | + /* data == no gap. */ | |
| 1508 | + if (likely(slots[i])) | |
| 1509 | + continue; | |
| 1510 | + | |
| 1511 | + pstart = pivots[i - 1]; | |
| 1512 | + gap = pivots[i] - pstart; | |
| 1513 | + if (gap > max_gap) | |
| 1514 | + max_gap = gap; | |
| 1515 | + | |
| 1516 | + /* There cannot be two gaps in a row. */ | |
| 1517 | + i++; | |
| 1518 | + } | |
| 1519 | + return max_gap; | |
| 1520 | +} | |
| 1521 | + | |
| 1522 | +/* | |
| 1523 | + * ma_max_gap() - Get the maximum gap in a maple node (non-leaf) | |
| 1524 | + * @node: The maple node | |
| 1525 | + * @gaps: The pointer to the gaps | |
| 1526 | + * @mt: The maple node type | |
| 1527 | + * @*off: Pointer to store the offset location of the gap. | |
| 1528 | + * | |
| 1529 | + * Uses the metadata data end to scan backwards across set gaps. | |
| 1530 | + * | |
| 1531 | + * Return: The maximum gap value | |
| 1532 | + */ | |
| 1533 | +static inline unsigned long | |
| 1534 | +ma_max_gap(struct maple_node *node, unsigned long *gaps, enum maple_type mt, | |
| 1535 | + unsigned char *off) | |
| 1536 | +{ | |
| 1537 | + unsigned char offset, i; | |
| 1538 | + unsigned long max_gap = 0; | |
| 1539 | + | |
| 1540 | + i = offset = ma_meta_end(node, mt); | |
| 1541 | + do { | |
| 1542 | + if (gaps[i] > max_gap) { | |
| 1543 | + max_gap = gaps[i]; | |
| 1544 | + offset = i; | |
| 1545 | + } | |
| 1546 | + } while (i--); | |
| 1547 | + | |
| 1548 | + *off = offset; | |
| 1549 | + return max_gap; | |
| 1550 | +} | |
| 1551 | + | |
| 1552 | +/* | |
| 1553 | + * mas_max_gap() - find the largest gap in a non-leaf node and set the slot. | |
| 1554 | + * @mas: The maple state. | |
| 1555 | + * | |
| 1556 | + * If the metadata gap is set to MAPLE_ARANGE64_META_MAX, there is no gap. | |
| 1557 | + * | |
| 1558 | + * Return: The gap value. | |
| 1559 | + */ | |
| 1560 | +static inline unsigned long mas_max_gap(struct ma_state *mas) | |
| 1561 | +{ | |
| 1562 | + unsigned long *gaps; | |
| 1563 | + unsigned char offset; | |
| 1564 | + enum maple_type mt; | |
| 1565 | + struct maple_node *node; | |
| 1566 | + | |
| 1567 | + mt = mte_node_type(mas->node); | |
| 1568 | + if (ma_is_leaf(mt)) | |
| 1569 | + return mas_leaf_max_gap(mas); | |
| 1570 | + | |
| 1571 | + node = mas_mn(mas); | |
| 1572 | + offset = ma_meta_gap(node, mt); | |
| 1573 | + if (offset == MAPLE_ARANGE64_META_MAX) | |
| 1574 | + return 0; | |
| 1575 | + | |
| 1576 | + gaps = ma_gaps(node, mt); | |
| 1577 | + return gaps[offset]; | |
| 1578 | +} | |
| 1579 | + | |
| 1580 | +/* | |
| 1581 | + * mas_parent_gap() - Set the parent gap and any gaps above, as needed | |
| 1582 | + * @mas: The maple state | |
| 1583 | + * @offset: The gap offset in the parent to set | |
| 1584 | + * @new: The new gap value. | |
| 1585 | + * | |
| 1586 | + * Set the parent gap then continue to set the gap upwards, using the metadata | |
| 1587 | + * of the parent to see if it is necessary to check the node above. | |
| 1588 | + */ | |
| 1589 | +static inline void mas_parent_gap(struct ma_state *mas, unsigned char offset, | |
| 1590 | + unsigned long new) | |
| 1591 | +{ | |
| 1592 | + unsigned long meta_gap = 0; | |
| 1593 | + struct maple_node *pnode; | |
| 1594 | + struct maple_enode *penode; | |
| 1595 | + unsigned long *pgaps; | |
| 1596 | + unsigned char meta_offset; | |
| 1597 | + enum maple_type pmt; | |
| 1598 | + | |
| 1599 | + pnode = mte_parent(mas->node); | |
| 1600 | + pmt = mas_parent_enum(mas, mas->node); | |
| 1601 | + penode = mt_mk_node(pnode, pmt); | |
| 1602 | + pgaps = ma_gaps(pnode, pmt); | |
| 1603 | + | |
| 1604 | +ascend: | |
| 1605 | + meta_offset = ma_meta_gap(pnode, pmt); | |
| 1606 | + if (meta_offset == MAPLE_ARANGE64_META_MAX) | |
| 1607 | + meta_gap = 0; | |
| 1608 | + else | |
| 1609 | + meta_gap = pgaps[meta_offset]; | |
| 1610 | + | |
| 1611 | + pgaps[offset] = new; | |
| 1612 | + | |
| 1613 | + if (meta_gap == new) | |
| 1614 | + return; | |
| 1615 | + | |
| 1616 | + if (offset != meta_offset) { | |
| 1617 | + if (meta_gap > new) | |
| 1618 | + return; | |
| 1619 | + | |
| 1620 | + ma_set_meta_gap(pnode, pmt, offset); | |
| 1621 | + } else if (new < meta_gap) { | |
| 1622 | + meta_offset = 15; | |
| 1623 | + new = ma_max_gap(pnode, pgaps, pmt, &meta_offset); | |
| 1624 | + ma_set_meta_gap(pnode, pmt, meta_offset); | |
| 1625 | + } | |
| 1626 | + | |
| 1627 | + if (ma_is_root(pnode)) | |
| 1628 | + return; | |
| 1629 | + | |
| 1630 | + /* Go to the parent node. */ | |
| 1631 | + pnode = mte_parent(penode); | |
| 1632 | + pmt = mas_parent_enum(mas, penode); | |
| 1633 | + pgaps = ma_gaps(pnode, pmt); | |
| 1634 | + offset = mte_parent_slot(penode); | |
| 1635 | + penode = mt_mk_node(pnode, pmt); | |
| 1636 | + goto ascend; | |
| 1637 | +} | |
| 1638 | + | |
| 1639 | +/* | |
| 1640 | + * mas_update_gap() - Update a nodes gaps and propagate up if necessary. | |
| 1641 | + * @mas - the maple state. | |
| 1642 | + */ | |
| 1643 | +static inline void mas_update_gap(struct ma_state *mas) | |
| 1644 | +{ | |
| 1645 | + unsigned char pslot; | |
| 1646 | + unsigned long p_gap; | |
| 1647 | + unsigned long max_gap; | |
| 1648 | + | |
| 1649 | + if (!mt_is_alloc(mas->tree)) | |
| 1650 | + return; | |
| 1651 | + | |
| 1652 | + if (mte_is_root(mas->node)) | |
| 1653 | + return; | |
| 1654 | + | |
| 1655 | + max_gap = mas_max_gap(mas); | |
| 1656 | + | |
| 1657 | + pslot = mte_parent_slot(mas->node); | |
| 1658 | + p_gap = ma_gaps(mte_parent(mas->node), | |
| 1659 | + mas_parent_enum(mas, mas->node))[pslot]; | |
| 1660 | + | |
| 1661 | + if (p_gap != max_gap) | |
| 1662 | + mas_parent_gap(mas, pslot, max_gap); | |
| 1663 | +} | |
| 1664 | + | |
| 1665 | +/* | |
| 1666 | + * mas_adopt_children() - Set the parent pointer of all nodes in @parent to | |
| 1667 | + * @parent with the slot encoded. | |
| 1668 | + * @mas - the maple state (for the tree) | |
| 1669 | + * @parent - the maple encoded node containing the children. | |
| 1670 | + */ | |
| 1671 | +static inline void mas_adopt_children(struct ma_state *mas, | |
| 1672 | + struct maple_enode *parent) | |
| 1673 | +{ | |
| 1674 | + enum maple_type type = mte_node_type(parent); | |
| 1675 | + struct maple_node *node = mas_mn(mas); | |
| 1676 | + void __rcu **slots = ma_slots(node, type); | |
| 1677 | + unsigned long *pivots = ma_pivots(node, type); | |
| 1678 | + struct maple_enode *child; | |
| 1679 | + unsigned char offset; | |
| 1680 | + | |
| 1681 | + offset = ma_data_end(node, type, pivots, mas->max); | |
| 1682 | + do { | |
| 1683 | + child = mas_slot_locked(mas, slots, offset); | |
| 1684 | + mte_set_parent(child, parent, offset); | |
| 1685 | + } while (offset--); | |
| 1686 | +} | |
| 1687 | + | |
| 1688 | +/* | |
| 1689 | + * mas_replace() - Replace a maple node in the tree with mas->node. Uses the | |
| 1690 | + * parent encoding to locate the maple node in the tree. | |
| 1691 | + * @mas - the ma_state to use for operations. | |
| 1692 | + * @advanced - boolean to adopt the child nodes and free the old node (false) or | |
| 1693 | + * leave the node (true) and handle the adoption and free elsewhere. | |
| 1694 | + */ | |
| 1695 | +static inline void mas_replace(struct ma_state *mas, bool advanced) | |
| 1696 | + __must_hold(mas->tree->lock) | |
| 1697 | +{ | |
| 1698 | + struct maple_node *mn = mas_mn(mas); | |
| 1699 | + struct maple_enode *old_enode; | |
| 1700 | + unsigned char offset = 0; | |
| 1701 | + void __rcu **slots = NULL; | |
| 1702 | + | |
| 1703 | + if (ma_is_root(mn)) { | |
| 1704 | + old_enode = mas_root_locked(mas); | |
| 1705 | + } else { | |
| 1706 | + offset = mte_parent_slot(mas->node); | |
| 1707 | + slots = ma_slots(mte_parent(mas->node), | |
| 1708 | + mas_parent_enum(mas, mas->node)); | |
| 1709 | + old_enode = mas_slot_locked(mas, slots, offset); | |
| 1710 | + } | |
| 1711 | + | |
| 1712 | + if (!advanced && !mte_is_leaf(mas->node)) | |
| 1713 | + mas_adopt_children(mas, mas->node); | |
| 1714 | + | |
| 1715 | + if (mte_is_root(mas->node)) { | |
| 1716 | + mn->parent = ma_parent_ptr( | |
| 1717 | + ((unsigned long)mas->tree | MA_ROOT_PARENT)); | |
| 1718 | + rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node)); | |
| 1719 | + mas_set_height(mas); | |
| 1720 | + } else { | |
| 1721 | + rcu_assign_pointer(slots[offset], mas->node); | |
| 1722 | + } | |
| 1723 | + | |
| 1724 | + if (!advanced) | |
| 1725 | + mas_free(mas, old_enode); | |
| 1726 | +} | |
| 1727 | + | |
| 1728 | +/* | |
| 1729 | + * mas_new_child() - Find the new child of a node. | |
| 1730 | + * @mas: the maple state | |
| 1731 | + * @child: the maple state to store the child. | |
| 1732 | + */ | |
| 1733 | +static inline bool mas_new_child(struct ma_state *mas, struct ma_state *child) | |
| 1734 | + __must_hold(mas->tree->lock) | |
| 1735 | +{ | |
| 1736 | + enum maple_type mt; | |
| 1737 | + unsigned char offset; | |
| 1738 | + unsigned char end; | |
| 1739 | + unsigned long *pivots; | |
| 1740 | + struct maple_enode *entry; | |
| 1741 | + struct maple_node *node; | |
| 1742 | + void __rcu **slots; | |
| 1743 | + | |
| 1744 | + mt = mte_node_type(mas->node); | |
| 1745 | + node = mas_mn(mas); | |
| 1746 | + slots = ma_slots(node, mt); | |
| 1747 | + pivots = ma_pivots(node, mt); | |
| 1748 | + end = ma_data_end(node, mt, pivots, mas->max); | |
| 1749 | + for (offset = mas->offset; offset <= end; offset++) { | |
| 1750 | + entry = mas_slot_locked(mas, slots, offset); | |
| 1751 | + if (mte_parent(entry) == node) { | |
| 1752 | + *child = *mas; | |
| 1753 | + mas->offset = offset + 1; | |
| 1754 | + child->offset = offset; | |
| 1755 | + mas_descend(child); | |
| 1756 | + child->offset = 0; | |
| 1757 | + return true; | |
| 1758 | + } | |
| 1759 | + } | |
| 1760 | + return false; | |
| 1761 | +} | |
| 1762 | + | |
| 1763 | +/* | |
| 1764 | + * mab_shift_right() - Shift the data in mab right. Note, does not clean out the | |
| 1765 | + * old data or set b_node->b_end. | |
| 1766 | + * @b_node: the maple_big_node | |
| 1767 | + * @shift: the shift count | |
| 1768 | + */ | |
| 1769 | +static inline void mab_shift_right(struct maple_big_node *b_node, | |
| 1770 | + unsigned char shift) | |
| 1771 | +{ | |
| 1772 | + unsigned long size = b_node->b_end * sizeof(unsigned long); | |
| 1773 | + | |
| 1774 | + memmove(b_node->pivot + shift, b_node->pivot, size); | |
| 1775 | + memmove(b_node->slot + shift, b_node->slot, size); | |
| 1776 | + if (b_node->type == maple_arange_64) | |
| 1777 | + memmove(b_node->gap + shift, b_node->gap, size); | |
| 1778 | +} | |
| 1779 | + | |
| 1780 | +/* | |
| 1781 | + * mab_middle_node() - Check if a middle node is needed (unlikely) | |
| 1782 | + * @b_node: the maple_big_node that contains the data. | |
| 1783 | + * @size: the amount of data in the b_node | |
| 1784 | + * @split: the potential split location | |
| 1785 | + * @slot_count: the size that can be stored in a single node being considered. | |
| 1786 | + * | |
| 1787 | + * Return: true if a middle node is required. | |
| 1788 | + */ | |
| 1789 | +static inline bool mab_middle_node(struct maple_big_node *b_node, int split, | |
| 1790 | + unsigned char slot_count) | |
| 1791 | +{ | |
| 1792 | + unsigned char size = b_node->b_end; | |
| 1793 | + | |
| 1794 | + if (size >= 2 * slot_count) | |
| 1795 | + return true; | |
| 1796 | + | |
| 1797 | + if (!b_node->slot[split] && (size >= 2 * slot_count - 1)) | |
| 1798 | + return true; | |
| 1799 | + | |
| 1800 | + return false; | |
| 1801 | +} | |
| 1802 | + | |
| 1803 | +/* | |
| 1804 | + * mab_no_null_split() - ensure the split doesn't fall on a NULL | |
| 1805 | + * @b_node: the maple_big_node with the data | |
| 1806 | + * @split: the suggested split location | |
| 1807 | + * @slot_count: the number of slots in the node being considered. | |
| 1808 | + * | |
| 1809 | + * Return: the split location. | |
| 1810 | + */ | |
| 1811 | +static inline int mab_no_null_split(struct maple_big_node *b_node, | |
| 1812 | + unsigned char split, unsigned char slot_count) | |
| 1813 | +{ | |
| 1814 | + if (!b_node->slot[split]) { | |
| 1815 | + /* | |
| 1816 | + * If the split is less than the max slot && the right side will | |
| 1817 | + * still be sufficient, then increment the split on NULL. | |
| 1818 | + */ | |
| 1819 | + if ((split < slot_count - 1) && | |
| 1820 | + (b_node->b_end - split) > (mt_min_slots[b_node->type])) | |
| 1821 | + split++; | |
| 1822 | + else | |
| 1823 | + split--; | |
| 1824 | + } | |
| 1825 | + return split; | |
| 1826 | +} | |
| 1827 | + | |
| 1828 | +/* | |
| 1829 | + * mab_calc_split() - Calculate the split location and if there needs to be two | |
| 1830 | + * splits. | |
| 1831 | + * @bn: The maple_big_node with the data | |
| 1832 | + * @mid_split: The second split, if required. 0 otherwise. | |
| 1833 | + * | |
| 1834 | + * Return: The first split location. The middle split is set in @mid_split. | |
| 1835 | + */ | |
| 1836 | +static inline int mab_calc_split(struct ma_state *mas, | |
| 1837 | + struct maple_big_node *bn, unsigned char *mid_split, unsigned long min) | |
| 1838 | +{ | |
| 1839 | + unsigned char b_end = bn->b_end; | |
| 1840 | + int split = b_end / 2; /* Assume equal split. */ | |
| 1841 | + unsigned char slot_min, slot_count = mt_slots[bn->type]; | |
| 1842 | + | |
| 1843 | + /* | |
| 1844 | + * To support gap tracking, all NULL entries are kept together and a node cannot | |
| 1845 | + * end on a NULL entry, with the exception of the left-most leaf. The | |
| 1846 | + * limitation means that the split of a node must be checked for this condition | |
| 1847 | + * and be able to put more data in one direction or the other. | |
| 1848 | + */ | |
| 1849 | + if (unlikely((mas->mas_flags & MA_STATE_BULK))) { | |
| 1850 | + *mid_split = 0; | |
| 1851 | + split = b_end - mt_min_slots[bn->type]; | |
| 1852 | + | |
| 1853 | + if (!ma_is_leaf(bn->type)) | |
| 1854 | + return split; | |
| 1855 | + | |
| 1856 | + mas->mas_flags |= MA_STATE_REBALANCE; | |
| 1857 | + if (!bn->slot[split]) | |
| 1858 | + split--; | |
| 1859 | + return split; | |
| 1860 | + } | |
| 1861 | + | |
| 1862 | + /* | |
| 1863 | + * Although extremely rare, it is possible to enter what is known as the 3-way | |
| 1864 | + * split scenario. The 3-way split comes about by means of a store of a range | |
| 1865 | + * that overwrites the end and beginning of two full nodes. The result is a set | |
| 1866 | + * of entries that cannot be stored in 2 nodes. Sometimes, these two nodes can | |
| 1867 | + * also be located in different parent nodes which are also full. This can | |
| 1868 | + * carry upwards all the way to the root in the worst case. | |
| 1869 | + */ | |
| 1870 | + if (unlikely(mab_middle_node(bn, split, slot_count))) { | |
| 1871 | + split = b_end / 3; | |
| 1872 | + *mid_split = split * 2; | |
| 1873 | + } else { | |
| 1874 | + slot_min = mt_min_slots[bn->type]; | |
| 1875 | + | |
| 1876 | + *mid_split = 0; | |
| 1877 | + /* | |
| 1878 | + * Avoid having a range less than the slot count unless it | |
| 1879 | + * causes one node to be deficient. | |
| 1880 | + * NOTE: mt_min_slots is 1 based, b_end and split are zero. | |
| 1881 | + */ | |
| 1882 | + while (((bn->pivot[split] - min) < slot_count - 1) && | |
| 1883 | + (split < slot_count - 1) && (b_end - split > slot_min)) | |
| 1884 | + split++; | |
| 1885 | + } | |
| 1886 | + | |
| 1887 | + /* Avoid ending a node on a NULL entry */ | |
| 1888 | + split = mab_no_null_split(bn, split, slot_count); | |
| 1889 | + if (!(*mid_split)) | |
| 1890 | + return split; | |
| 1891 | + | |
| 1892 | + *mid_split = mab_no_null_split(bn, *mid_split, slot_count); | |
| 1893 | + | |
| 1894 | + return split; | |
| 1895 | +} | |
| 1896 | + | |
| 1897 | +/* | |
| 1898 | + * mas_mab_cp() - Copy data from a maple state inclusively to a maple_big_node | |
| 1899 | + * and set @b_node->b_end to the next free slot. | |
| 1900 | + * @mas: The maple state | |
| 1901 | + * @mas_start: The starting slot to copy | |
| 1902 | + * @mas_end: The end slot to copy (inclusively) | |
| 1903 | + * @b_node: The maple_big_node to place the data | |
| 1904 | + * @mab_start: The starting location in maple_big_node to store the data. | |
| 1905 | + */ | |
| 1906 | +static inline void mas_mab_cp(struct ma_state *mas, unsigned char mas_start, | |
| 1907 | + unsigned char mas_end, struct maple_big_node *b_node, | |
| 1908 | + unsigned char mab_start) | |
| 1909 | +{ | |
| 1910 | + enum maple_type mt; | |
| 1911 | + struct maple_node *node; | |
| 1912 | + void __rcu **slots; | |
| 1913 | + unsigned long *pivots, *gaps; | |
| 1914 | + int i = mas_start, j = mab_start; | |
| 1915 | + unsigned char piv_end; | |
| 1916 | + | |
| 1917 | + node = mas_mn(mas); | |
| 1918 | + mt = mte_node_type(mas->node); | |
| 1919 | + pivots = ma_pivots(node, mt); | |
| 1920 | + if (!i) { | |
| 1921 | + b_node->pivot[j] = pivots[i++]; | |
| 1922 | + if (unlikely(i > mas_end)) | |
| 1923 | + goto complete; | |
| 1924 | + j++; | |
| 1925 | + } | |
| 1926 | + | |
| 1927 | + piv_end = min(mas_end, mt_pivots[mt]); | |
| 1928 | + for (; i < piv_end; i++, j++) { | |
| 1929 | + b_node->pivot[j] = pivots[i]; | |
| 1930 | + if (unlikely(!b_node->pivot[j])) | |
| 1931 | + break; | |
| 1932 | + | |
| 1933 | + if (unlikely(mas->max == b_node->pivot[j])) | |
| 1934 | + goto complete; | |
| 1935 | + } | |
| 1936 | + | |
| 1937 | + if (likely(i <= mas_end)) | |
| 1938 | + b_node->pivot[j] = mas_safe_pivot(mas, pivots, i, mt); | |
| 1939 | + | |
| 1940 | +complete: | |
| 1941 | + b_node->b_end = ++j; | |
| 1942 | + j -= mab_start; | |
| 1943 | + slots = ma_slots(node, mt); | |
| 1944 | + memcpy(b_node->slot + mab_start, slots + mas_start, sizeof(void *) * j); | |
| 1945 | + if (!ma_is_leaf(mt) && mt_is_alloc(mas->tree)) { | |
| 1946 | + gaps = ma_gaps(node, mt); | |
| 1947 | + memcpy(b_node->gap + mab_start, gaps + mas_start, | |
| 1948 | + sizeof(unsigned long) * j); | |
| 1949 | + } | |
| 1950 | +} | |
| 1951 | + | |
| 1952 | +/* | |
| 1953 | + * mas_leaf_set_meta() - Set the metadata of a leaf if possible. | |
| 1954 | + * @mas: The maple state | |
| 1955 | + * @node: The maple node | |
| 1956 | + * @pivots: pointer to the maple node pivots | |
| 1957 | + * @mt: The maple type | |
| 1958 | + * @end: The assumed end | |
| 1959 | + * | |
| 1960 | + * Note, end may be incremented within this function but not modified at the | |
| 1961 | + * source. This is fine since the metadata is the last thing to be stored in a | |
| 1962 | + * node during a write. | |
| 1963 | + */ | |
| 1964 | +static inline void mas_leaf_set_meta(struct ma_state *mas, | |
| 1965 | + struct maple_node *node, unsigned long *pivots, | |
| 1966 | + enum maple_type mt, unsigned char end) | |
| 1967 | +{ | |
| 1968 | + /* There is no room for metadata already */ | |
| 1969 | + if (mt_pivots[mt] <= end) | |
| 1970 | + return; | |
| 1971 | + | |
| 1972 | + if (pivots[end] && pivots[end] < mas->max) | |
| 1973 | + end++; | |
| 1974 | + | |
| 1975 | + if (end < mt_slots[mt] - 1) | |
| 1976 | + ma_set_meta(node, mt, 0, end); | |
| 1977 | +} | |
| 1978 | + | |
| 1979 | +/* | |
| 1980 | + * mab_mas_cp() - Copy data from maple_big_node to a maple encoded node. | |
| 1981 | + * @b_node: the maple_big_node that has the data | |
| 1982 | + * @mab_start: the start location in @b_node. | |
| 1983 | + * @mab_end: The end location in @b_node (inclusively) | |
| 1984 | + * @mas: The maple state with the maple encoded node. | |
| 1985 | + */ | |
| 1986 | +static inline void mab_mas_cp(struct maple_big_node *b_node, | |
| 1987 | + unsigned char mab_start, unsigned char mab_end, | |
| 1988 | + struct ma_state *mas, bool new_max) | |
| 1989 | +{ | |
| 1990 | + int i, j = 0; | |
| 1991 | + enum maple_type mt = mte_node_type(mas->node); | |
| 1992 | + struct maple_node *node = mte_to_node(mas->node); | |
| 1993 | + void __rcu **slots = ma_slots(node, mt); | |
| 1994 | + unsigned long *pivots = ma_pivots(node, mt); | |
| 1995 | + unsigned long *gaps = NULL; | |
| 1996 | + unsigned char end; | |
| 1997 | + | |
| 1998 | + if (mab_end - mab_start > mt_pivots[mt]) | |
| 1999 | + mab_end--; | |
| 2000 | + | |
| 2001 | + if (!pivots[mt_pivots[mt] - 1]) | |
| 2002 | + slots[mt_pivots[mt]] = NULL; | |
| 2003 | + | |
| 2004 | + i = mab_start; | |
| 2005 | + do { | |
| 2006 | + pivots[j++] = b_node->pivot[i++]; | |
| 2007 | + } while (i <= mab_end && likely(b_node->pivot[i])); | |
| 2008 | + | |
| 2009 | + memcpy(slots, b_node->slot + mab_start, | |
| 2010 | + sizeof(void *) * (i - mab_start)); | |
| 2011 | + | |
| 2012 | + if (new_max) | |
| 2013 | + mas->max = b_node->pivot[i - 1]; | |
| 2014 | + | |
| 2015 | + end = j - 1; | |
| 2016 | + if (likely(!ma_is_leaf(mt) && mt_is_alloc(mas->tree))) { | |
| 2017 | + unsigned long max_gap = 0; | |
| 2018 | + unsigned char offset = 15; | |
| 2019 | + | |
| 2020 | + gaps = ma_gaps(node, mt); | |
| 2021 | + do { | |
| 2022 | + gaps[--j] = b_node->gap[--i]; | |
| 2023 | + if (gaps[j] > max_gap) { | |
| 2024 | + offset = j; | |
| 2025 | + max_gap = gaps[j]; | |
| 2026 | + } | |
| 2027 | + } while (j); | |
| 2028 | + | |
| 2029 | + ma_set_meta(node, mt, offset, end); | |
| 2030 | + } else { | |
| 2031 | + mas_leaf_set_meta(mas, node, pivots, mt, end); | |
| 2032 | + } | |
| 2033 | +} | |
| 2034 | + | |
| 2035 | +/* | |
| 2036 | + * mas_descend_adopt() - Descend through a sub-tree and adopt children. | |
| 2037 | + * @mas: the maple state with the maple encoded node of the sub-tree. | |
| 2038 | + * | |
| 2039 | + * Descend through a sub-tree and adopt children who do not have the correct | |
| 2040 | + * parents set. Follow the parents which have the correct parents as they are | |
| 2041 | + * the new entries which need to be followed to find other incorrectly set | |
| 2042 | + * parents. | |
| 2043 | + */ | |
| 2044 | +static inline void mas_descend_adopt(struct ma_state *mas) | |
| 2045 | +{ | |
| 2046 | + struct ma_state list[3], next[3]; | |
| 2047 | + int i, n; | |
| 2048 | + | |
| 2049 | + /* | |
| 2050 | + * At each level there may be up to 3 correct parent pointers which indicates | |
| 2051 | + * the new nodes which need to be walked to find any new nodes at a lower level. | |
| 2052 | + */ | |
| 2053 | + | |
| 2054 | + for (i = 0; i < 3; i++) { | |
| 2055 | + list[i] = *mas; | |
| 2056 | + list[i].offset = 0; | |
| 2057 | + next[i].offset = 0; | |
| 2058 | + } | |
| 2059 | + next[0] = *mas; | |
| 2060 | + | |
| 2061 | + while (!mte_is_leaf(list[0].node)) { | |
| 2062 | + n = 0; | |
| 2063 | + for (i = 0; i < 3; i++) { | |
| 2064 | + if (mas_is_none(&list[i])) | |
| 2065 | + continue; | |
| 2066 | + | |
| 2067 | + if (i && list[i-1].node == list[i].node) | |
| 2068 | + continue; | |
| 2069 | + | |
| 2070 | + while ((n < 3) && (mas_new_child(&list[i], &next[n]))) | |
| 2071 | + n++; | |
| 2072 | + | |
| 2073 | + mas_adopt_children(&list[i], list[i].node); | |
| 2074 | + } | |
| 2075 | + | |
| 2076 | + while (n < 3) | |
| 2077 | + next[n++].node = MAS_NONE; | |
| 2078 | + | |
| 2079 | + /* descend by setting the list to the children */ | |
| 2080 | + for (i = 0; i < 3; i++) | |
| 2081 | + list[i] = next[i]; | |
| 2082 | + } | |
| 2083 | +} | |
| 2084 | + | |
| 2085 | +/* | |
| 2086 | + * mas_bulk_rebalance() - Rebalance the end of a tree after a bulk insert. | |
| 2087 | + * @mas: The maple state | |
| 2088 | + * @end: The maple node end | |
| 2089 | + * @mt: The maple node type | |
| 2090 | + */ | |
| 2091 | +static inline void mas_bulk_rebalance(struct ma_state *mas, unsigned char end, | |
| 2092 | + enum maple_type mt) | |
| 2093 | +{ | |
| 2094 | + if (!(mas->mas_flags & MA_STATE_BULK)) | |
| 2095 | + return; | |
| 2096 | + | |
| 2097 | + if (mte_is_root(mas->node)) | |
| 2098 | + return; | |
| 2099 | + | |
| 2100 | + if (end > mt_min_slots[mt]) { | |
| 2101 | + mas->mas_flags &= ~MA_STATE_REBALANCE; | |
| 2102 | + return; | |
| 2103 | + } | |
| 2104 | +} | |
| 2105 | + | |
| 2106 | +/* | |
| 2107 | + * mas_store_b_node() - Store an @entry into the b_node while also copying the | |
| 2108 | + * data from a maple encoded node. | |
| 2109 | + * @wr_mas: the maple write state | |
| 2110 | + * @b_node: the maple_big_node to fill with data | |
| 2111 | + * @offset_end: the offset to end copying | |
| 2112 | + * | |
| 2113 | + * Return: The actual end of the data stored in @b_node | |
| 2114 | + */ | |
| 2115 | +static inline void mas_store_b_node(struct ma_wr_state *wr_mas, | |
| 2116 | + struct maple_big_node *b_node, unsigned char offset_end) | |
| 2117 | +{ | |
| 2118 | + unsigned char slot; | |
| 2119 | + unsigned char b_end; | |
| 2120 | + /* Possible underflow of piv will wrap back to 0 before use. */ | |
| 2121 | + unsigned long piv; | |
| 2122 | + struct ma_state *mas = wr_mas->mas; | |
| 2123 | + | |
| 2124 | + b_node->type = wr_mas->type; | |
| 2125 | + b_end = 0; | |
| 2126 | + slot = mas->offset; | |
| 2127 | + if (slot) { | |
| 2128 | + /* Copy start data up to insert. */ | |
| 2129 | + mas_mab_cp(mas, 0, slot - 1, b_node, 0); | |
| 2130 | + b_end = b_node->b_end; | |
| 2131 | + piv = b_node->pivot[b_end - 1]; | |
| 2132 | + } else | |
| 2133 | + piv = mas->min - 1; | |
| 2134 | + | |
| 2135 | + if (piv + 1 < mas->index) { | |
| 2136 | + /* Handle range starting after old range */ | |
| 2137 | + b_node->slot[b_end] = wr_mas->content; | |
| 2138 | + if (!wr_mas->content) | |
| 2139 | + b_node->gap[b_end] = mas->index - 1 - piv; | |
| 2140 | + b_node->pivot[b_end++] = mas->index - 1; | |
| 2141 | + } | |
| 2142 | + | |
| 2143 | + /* Store the new entry. */ | |
| 2144 | + mas->offset = b_end; | |
| 2145 | + b_node->slot[b_end] = wr_mas->entry; | |
| 2146 | + b_node->pivot[b_end] = mas->last; | |
| 2147 | + | |
| 2148 | + /* Appended. */ | |
| 2149 | + if (mas->last >= mas->max) | |
| 2150 | + goto b_end; | |
| 2151 | + | |
| 2152 | + /* Handle new range ending before old range ends */ | |
| 2153 | + piv = mas_logical_pivot(mas, wr_mas->pivots, offset_end, wr_mas->type); | |
| 2154 | + if (piv > mas->last) { | |
| 2155 | + if (piv == ULONG_MAX) | |
| 2156 | + mas_bulk_rebalance(mas, b_node->b_end, wr_mas->type); | |
| 2157 | + | |
| 2158 | + if (offset_end != slot) | |
| 2159 | + wr_mas->content = mas_slot_locked(mas, wr_mas->slots, | |
| 2160 | + offset_end); | |
| 2161 | + | |
| 2162 | + b_node->slot[++b_end] = wr_mas->content; | |
| 2163 | + if (!wr_mas->content) | |
| 2164 | + b_node->gap[b_end] = piv - mas->last + 1; | |
| 2165 | + b_node->pivot[b_end] = piv; | |
| 2166 | + } | |
| 2167 | + | |
| 2168 | + slot = offset_end + 1; | |
| 2169 | + if (slot > wr_mas->node_end) | |
| 2170 | + goto b_end; | |
| 2171 | + | |
| 2172 | + /* Copy end data to the end of the node. */ | |
| 2173 | + mas_mab_cp(mas, slot, wr_mas->node_end + 1, b_node, ++b_end); | |
| 2174 | + b_node->b_end--; | |
| 2175 | + return; | |
| 2176 | + | |
| 2177 | +b_end: | |
| 2178 | + b_node->b_end = b_end; | |
| 2179 | +} | |
| 2180 | + | |
| 2181 | +/* | |
| 2182 | + * mas_prev_sibling() - Find the previous node with the same parent. | |
| 2183 | + * @mas: the maple state | |
| 2184 | + * | |
| 2185 | + * Return: True if there is a previous sibling, false otherwise. | |
| 2186 | + */ | |
| 2187 | +static inline bool mas_prev_sibling(struct ma_state *mas) | |
| 2188 | +{ | |
| 2189 | + unsigned int p_slot = mte_parent_slot(mas->node); | |
| 2190 | + | |
| 2191 | + if (mte_is_root(mas->node)) | |
| 2192 | + return false; | |
| 2193 | + | |
| 2194 | + if (!p_slot) | |
| 2195 | + return false; | |
| 2196 | + | |
| 2197 | + mas_ascend(mas); | |
| 2198 | + mas->offset = p_slot - 1; | |
| 2199 | + mas_descend(mas); | |
| 2200 | + return true; | |
| 2201 | +} | |
| 2202 | + | |
| 2203 | +/* | |
| 2204 | + * mas_next_sibling() - Find the next node with the same parent. | |
| 2205 | + * @mas: the maple state | |
| 2206 | + * | |
| 2207 | + * Return: true if there is a next sibling, false otherwise. | |
| 2208 | + */ | |
| 2209 | +static inline bool mas_next_sibling(struct ma_state *mas) | |
| 2210 | +{ | |
| 2211 | + MA_STATE(parent, mas->tree, mas->index, mas->last); | |
| 2212 | + | |
| 2213 | + if (mte_is_root(mas->node)) | |
| 2214 | + return false; | |
| 2215 | + | |
| 2216 | + parent = *mas; | |
| 2217 | + mas_ascend(&parent); | |
| 2218 | + parent.offset = mte_parent_slot(mas->node) + 1; | |
| 2219 | + if (parent.offset > mas_data_end(&parent)) | |
| 2220 | + return false; | |
| 2221 | + | |
| 2222 | + *mas = parent; | |
| 2223 | + mas_descend(mas); | |
| 2224 | + return true; | |
| 2225 | +} | |
| 2226 | + | |
| 2227 | +/* | |
| 2228 | + * mte_node_or_node() - Return the encoded node or MAS_NONE. | |
| 2229 | + * @enode: The encoded maple node. | |
| 2230 | + * | |
| 2231 | + * Shorthand to avoid setting %NULLs in the tree or maple_subtree_state. | |
| 2232 | + * | |
| 2233 | + * Return: @enode or MAS_NONE | |
| 2234 | + */ | |
| 2235 | +static inline struct maple_enode *mte_node_or_none(struct maple_enode *enode) | |
| 2236 | +{ | |
| 2237 | + if (enode) | |
| 2238 | + return enode; | |
| 2239 | + | |
| 2240 | + return ma_enode_ptr(MAS_NONE); | |
| 2241 | +} | |
| 2242 | + | |
| 2243 | +/* | |
| 2244 | + * mas_wr_node_walk() - Find the correct offset for the index in the @mas. | |
| 2245 | + * @wr_mas: The maple write state | |
| 2246 | + * | |
| 2247 | + * Uses mas_slot_locked() and does not need to worry about dead nodes. | |
| 2248 | + */ | |
| 2249 | +static inline void mas_wr_node_walk(struct ma_wr_state *wr_mas) | |
| 2250 | +{ | |
| 2251 | + struct ma_state *mas = wr_mas->mas; | |
| 2252 | + unsigned char count; | |
| 2253 | + unsigned char offset; | |
| 2254 | + unsigned long index, min, max; | |
| 2255 | + | |
| 2256 | + if (unlikely(ma_is_dense(wr_mas->type))) { | |
| 2257 | + wr_mas->r_max = wr_mas->r_min = mas->index; | |
| 2258 | + mas->offset = mas->index = mas->min; | |
| 2259 | + return; | |
| 2260 | + } | |
| 2261 | + | |
| 2262 | + wr_mas->node = mas_mn(wr_mas->mas); | |
| 2263 | + wr_mas->pivots = ma_pivots(wr_mas->node, wr_mas->type); | |
| 2264 | + count = wr_mas->node_end = ma_data_end(wr_mas->node, wr_mas->type, | |
| 2265 | + wr_mas->pivots, mas->max); | |
| 2266 | + offset = mas->offset; | |
| 2267 | + min = mas_safe_min(mas, wr_mas->pivots, offset); | |
| 2268 | + if (unlikely(offset == count)) | |
| 2269 | + goto max; | |
| 2270 | + | |
| 2271 | + max = wr_mas->pivots[offset]; | |
| 2272 | + index = mas->index; | |
| 2273 | + if (unlikely(index <= max)) | |
| 2274 | + goto done; | |
| 2275 | + | |
| 2276 | + if (unlikely(!max && offset)) | |
| 2277 | + goto max; | |
| 2278 | + | |
| 2279 | + min = max + 1; | |
| 2280 | + while (++offset < count) { | |
| 2281 | + max = wr_mas->pivots[offset]; | |
| 2282 | + if (index <= max) | |
| 2283 | + goto done; | |
| 2284 | + else if (unlikely(!max)) | |
| 2285 | + break; | |
| 2286 | + | |
| 2287 | + min = max + 1; | |
| 2288 | + } | |
| 2289 | + | |
| 2290 | +max: | |
| 2291 | + max = mas->max; | |
| 2292 | +done: | |
| 2293 | + wr_mas->r_max = max; | |
| 2294 | + wr_mas->r_min = min; | |
| 2295 | + wr_mas->offset_end = mas->offset = offset; | |
| 2296 | +} | |
| 2297 | + | |
| 2298 | +/* | |
| 2299 | + * mas_topiary_range() - Add a range of slots to the topiary. | |
| 2300 | + * @mas: The maple state | |
| 2301 | + * @destroy: The topiary to add the slots (usually destroy) | |
| 2302 | + * @start: The starting slot inclusively | |
| 2303 | + * @end: The end slot inclusively | |
| 2304 | + */ | |
| 2305 | +static inline void mas_topiary_range(struct ma_state *mas, | |
| 2306 | + struct ma_topiary *destroy, unsigned char start, unsigned char end) | |
| 2307 | +{ | |
| 2308 | + void __rcu **slots; | |
| 2309 | + unsigned char offset; | |
| 2310 | + | |
| 2311 | + MT_BUG_ON(mas->tree, mte_is_leaf(mas->node)); | |
| 2312 | + slots = ma_slots(mas_mn(mas), mte_node_type(mas->node)); | |
| 2313 | + for (offset = start; offset <= end; offset++) { | |
| 2314 | + struct maple_enode *enode = mas_slot_locked(mas, slots, offset); | |
| 2315 | + | |
| 2316 | + if (mte_dead_node(enode)) | |
| 2317 | + continue; | |
| 2318 | + | |
| 2319 | + mat_add(destroy, enode); | |
| 2320 | + } | |
| 2321 | +} | |
| 2322 | + | |
| 2323 | +/* | |
| 2324 | + * mast_topiary() - Add the portions of the tree to the removal list; either to | |
| 2325 | + * be freed or discarded (destroy walk). | |
| 2326 | + * @mast: The maple_subtree_state. | |
| 2327 | + */ | |
| 2328 | +static inline void mast_topiary(struct maple_subtree_state *mast) | |
| 2329 | +{ | |
| 2330 | + MA_WR_STATE(wr_mas, mast->orig_l, NULL); | |
| 2331 | + unsigned char r_start, r_end; | |
| 2332 | + unsigned char l_start, l_end; | |
| 2333 | + void __rcu **l_slots, **r_slots; | |
| 2334 | + | |
| 2335 | + wr_mas.type = mte_node_type(mast->orig_l->node); | |
| 2336 | + mast->orig_l->index = mast->orig_l->last; | |
| 2337 | + mas_wr_node_walk(&wr_mas); | |
| 2338 | + l_start = mast->orig_l->offset + 1; | |
| 2339 | + l_end = mas_data_end(mast->orig_l); | |
| 2340 | + r_start = 0; | |
| 2341 | + r_end = mast->orig_r->offset; | |
| 2342 | + | |
| 2343 | + if (r_end) | |
| 2344 | + r_end--; | |
| 2345 | + | |
| 2346 | + l_slots = ma_slots(mas_mn(mast->orig_l), | |
| 2347 | + mte_node_type(mast->orig_l->node)); | |
| 2348 | + | |
| 2349 | + r_slots = ma_slots(mas_mn(mast->orig_r), | |
| 2350 | + mte_node_type(mast->orig_r->node)); | |
| 2351 | + | |
| 2352 | + if ((l_start < l_end) && | |
| 2353 | + mte_dead_node(mas_slot_locked(mast->orig_l, l_slots, l_start))) { | |
| 2354 | + l_start++; | |
| 2355 | + } | |
| 2356 | + | |
| 2357 | + if (mte_dead_node(mas_slot_locked(mast->orig_r, r_slots, r_end))) { | |
| 2358 | + if (r_end) | |
| 2359 | + r_end--; | |
| 2360 | + } | |
| 2361 | + | |
| 2362 | + if ((l_start > r_end) && (mast->orig_l->node == mast->orig_r->node)) | |
| 2363 | + return; | |
| 2364 | + | |
| 2365 | + /* At the node where left and right sides meet, add the parts between */ | |
| 2366 | + if (mast->orig_l->node == mast->orig_r->node) { | |
| 2367 | + return mas_topiary_range(mast->orig_l, mast->destroy, | |
| 2368 | + l_start, r_end); | |
| 2369 | + } | |
| 2370 | + | |
| 2371 | + /* mast->orig_r is different and consumed. */ | |
| 2372 | + if (mte_is_leaf(mast->orig_r->node)) | |
| 2373 | + return; | |
| 2374 | + | |
| 2375 | + if (mte_dead_node(mas_slot_locked(mast->orig_l, l_slots, l_end))) | |
| 2376 | + l_end--; | |
| 2377 | + | |
| 2378 | + | |
| 2379 | + if (l_start <= l_end) | |
| 2380 | + mas_topiary_range(mast->orig_l, mast->destroy, l_start, l_end); | |
| 2381 | + | |
| 2382 | + if (mte_dead_node(mas_slot_locked(mast->orig_r, r_slots, r_start))) | |
| 2383 | + r_start++; | |
| 2384 | + | |
| 2385 | + if (r_start <= r_end) | |
| 2386 | + mas_topiary_range(mast->orig_r, mast->destroy, 0, r_end); | |
| 2387 | +} | |
| 2388 | + | |
| 2389 | +/* | |
| 2390 | + * mast_rebalance_next() - Rebalance against the next node | |
| 2391 | + * @mast: The maple subtree state | |
| 2392 | + * @old_r: The encoded maple node to the right (next node). | |
| 2393 | + */ | |
| 2394 | +static inline void mast_rebalance_next(struct maple_subtree_state *mast) | |
| 2395 | +{ | |
| 2396 | + unsigned char b_end = mast->bn->b_end; | |
| 2397 | + | |
| 2398 | + mas_mab_cp(mast->orig_r, 0, mt_slot_count(mast->orig_r->node), | |
| 2399 | + mast->bn, b_end); | |
| 2400 | + mast->orig_r->last = mast->orig_r->max; | |
| 2401 | +} | |
| 2402 | + | |
| 2403 | +/* | |
| 2404 | + * mast_rebalance_prev() - Rebalance against the previous node | |
| 2405 | + * @mast: The maple subtree state | |
| 2406 | + * @old_l: The encoded maple node to the left (previous node) | |
| 2407 | + */ | |
| 2408 | +static inline void mast_rebalance_prev(struct maple_subtree_state *mast) | |
| 2409 | +{ | |
| 2410 | + unsigned char end = mas_data_end(mast->orig_l) + 1; | |
| 2411 | + unsigned char b_end = mast->bn->b_end; | |
| 2412 | + | |
| 2413 | + mab_shift_right(mast->bn, end); | |
| 2414 | + mas_mab_cp(mast->orig_l, 0, end - 1, mast->bn, 0); | |
| 2415 | + mast->l->min = mast->orig_l->min; | |
| 2416 | + mast->orig_l->index = mast->orig_l->min; | |
| 2417 | + mast->bn->b_end = end + b_end; | |
| 2418 | + mast->l->offset += end; | |
| 2419 | +} | |
| 2420 | + | |
| 2421 | +/* | |
| 2422 | + * mast_spanning_rebalance() - Rebalance nodes with nearest neighbour favouring | |
| 2423 | + * the node to the right. Checking the nodes to the right then the left at each | |
| 2424 | + * level upwards until root is reached. Free and destroy as needed. | |
| 2425 | + * Data is copied into the @mast->bn. | |
| 2426 | + * @mast: The maple_subtree_state. | |
| 2427 | + */ | |
| 2428 | +static inline | |
| 2429 | +bool mast_spanning_rebalance(struct maple_subtree_state *mast) | |
| 2430 | +{ | |
| 2431 | + struct ma_state r_tmp = *mast->orig_r; | |
| 2432 | + struct ma_state l_tmp = *mast->orig_l; | |
| 2433 | + struct maple_enode *ancestor = NULL; | |
| 2434 | + unsigned char start, end; | |
| 2435 | + unsigned char depth = 0; | |
| 2436 | + | |
| 2437 | + r_tmp = *mast->orig_r; | |
| 2438 | + l_tmp = *mast->orig_l; | |
| 2439 | + do { | |
| 2440 | + mas_ascend(mast->orig_r); | |
| 2441 | + mas_ascend(mast->orig_l); | |
| 2442 | + depth++; | |
| 2443 | + if (!ancestor && | |
| 2444 | + (mast->orig_r->node == mast->orig_l->node)) { | |
| 2445 | + ancestor = mast->orig_r->node; | |
| 2446 | + end = mast->orig_r->offset - 1; | |
| 2447 | + start = mast->orig_l->offset + 1; | |
| 2448 | + } | |
| 2449 | + | |
| 2450 | + if (mast->orig_r->offset < mas_data_end(mast->orig_r)) { | |
| 2451 | + if (!ancestor) { | |
| 2452 | + ancestor = mast->orig_r->node; | |
| 2453 | + start = 0; | |
| 2454 | + } | |
| 2455 | + | |
| 2456 | + mast->orig_r->offset++; | |
| 2457 | + do { | |
| 2458 | + mas_descend(mast->orig_r); | |
| 2459 | + mast->orig_r->offset = 0; | |
| 2460 | + depth--; | |
| 2461 | + } while (depth); | |
| 2462 | + | |
| 2463 | + mast_rebalance_next(mast); | |
| 2464 | + do { | |
| 2465 | + unsigned char l_off = 0; | |
| 2466 | + struct maple_enode *child = r_tmp.node; | |
| 2467 | + | |
| 2468 | + mas_ascend(&r_tmp); | |
| 2469 | + if (ancestor == r_tmp.node) | |
| 2470 | + l_off = start; | |
| 2471 | + | |
| 2472 | + if (r_tmp.offset) | |
| 2473 | + r_tmp.offset--; | |
| 2474 | + | |
| 2475 | + if (l_off < r_tmp.offset) | |
| 2476 | + mas_topiary_range(&r_tmp, mast->destroy, | |
| 2477 | + l_off, r_tmp.offset); | |
| 2478 | + | |
| 2479 | + if (l_tmp.node != child) | |
| 2480 | + mat_add(mast->free, child); | |
| 2481 | + | |
| 2482 | + } while (r_tmp.node != ancestor); | |
| 2483 | + | |
| 2484 | + *mast->orig_l = l_tmp; | |
| 2485 | + return true; | |
| 2486 | + | |
| 2487 | + } else if (mast->orig_l->offset != 0) { | |
| 2488 | + if (!ancestor) { | |
| 2489 | + ancestor = mast->orig_l->node; | |
| 2490 | + end = mas_data_end(mast->orig_l); | |
| 2491 | + } | |
| 2492 | + | |
| 2493 | + mast->orig_l->offset--; | |
| 2494 | + do { | |
| 2495 | + mas_descend(mast->orig_l); | |
| 2496 | + mast->orig_l->offset = | |
| 2497 | + mas_data_end(mast->orig_l); | |
| 2498 | + depth--; | |
| 2499 | + } while (depth); | |
| 2500 | + | |
| 2501 | + mast_rebalance_prev(mast); | |
| 2502 | + do { | |
| 2503 | + unsigned char r_off; | |
| 2504 | + struct maple_enode *child = l_tmp.node; | |
| 2505 | + | |
| 2506 | + mas_ascend(&l_tmp); | |
| 2507 | + if (ancestor == l_tmp.node) | |
| 2508 | + r_off = end; | |
| 2509 | + else | |
| 2510 | + r_off = mas_data_end(&l_tmp); | |
| 2511 | + | |
| 2512 | + if (l_tmp.offset < r_off) | |
| 2513 | + l_tmp.offset++; | |
| 2514 | + | |
| 2515 | + if (l_tmp.offset < r_off) | |
| 2516 | + mas_topiary_range(&l_tmp, mast->destroy, | |
| 2517 | + l_tmp.offset, r_off); | |
| 2518 | + | |
| 2519 | + if (r_tmp.node != child) | |
| 2520 | + mat_add(mast->free, child); | |
| 2521 | + | |
| 2522 | + } while (l_tmp.node != ancestor); | |
| 2523 | + | |
| 2524 | + *mast->orig_r = r_tmp; | |
| 2525 | + return true; | |
| 2526 | + } | |
| 2527 | + } while (!mte_is_root(mast->orig_r->node)); | |
| 2528 | + | |
| 2529 | + *mast->orig_r = r_tmp; | |
| 2530 | + *mast->orig_l = l_tmp; | |
| 2531 | + return false; | |
| 2532 | +} | |
| 2533 | + | |
| 2534 | +/* | |
| 2535 | + * mast_ascend_free() - Add current original maple state nodes to the free list | |
| 2536 | + * and ascend. | |
| 2537 | + * @mast: the maple subtree state. | |
| 2538 | + * | |
| 2539 | + * Ascend the original left and right sides and add the previous nodes to the | |
| 2540 | + * free list. Set the slots to point to the correct location in the new nodes. | |
| 2541 | + */ | |
| 2542 | +static inline void | |
| 2543 | +mast_ascend_free(struct maple_subtree_state *mast) | |
| 2544 | +{ | |
| 2545 | + MA_WR_STATE(wr_mas, mast->orig_r, NULL); | |
| 2546 | + struct maple_enode *left = mast->orig_l->node; | |
| 2547 | + struct maple_enode *right = mast->orig_r->node; | |
| 2548 | + | |
| 2549 | + mas_ascend(mast->orig_l); | |
| 2550 | + mas_ascend(mast->orig_r); | |
| 2551 | + mat_add(mast->free, left); | |
| 2552 | + | |
| 2553 | + if (left != right) | |
| 2554 | + mat_add(mast->free, right); | |
| 2555 | + | |
| 2556 | + mast->orig_r->offset = 0; | |
| 2557 | + mast->orig_r->index = mast->r->max; | |
| 2558 | + /* last should be larger than or equal to index */ | |
| 2559 | + if (mast->orig_r->last < mast->orig_r->index) | |
| 2560 | + mast->orig_r->last = mast->orig_r->index; | |
| 2561 | + /* | |
| 2562 | + * The node may not contain the value so set slot to ensure all | |
| 2563 | + * of the nodes contents are freed or destroyed. | |
| 2564 | + */ | |
| 2565 | + wr_mas.type = mte_node_type(mast->orig_r->node); | |
| 2566 | + mas_wr_node_walk(&wr_mas); | |
| 2567 | + /* Set up the left side of things */ | |
| 2568 | + mast->orig_l->offset = 0; | |
| 2569 | + mast->orig_l->index = mast->l->min; | |
| 2570 | + wr_mas.mas = mast->orig_l; | |
| 2571 | + wr_mas.type = mte_node_type(mast->orig_l->node); | |
| 2572 | + mas_wr_node_walk(&wr_mas); | |
| 2573 | + | |
| 2574 | + mast->bn->type = wr_mas.type; | |
| 2575 | +} | |
| 2576 | + | |
| 2577 | +/* | |
| 2578 | + * mas_new_ma_node() - Create and return a new maple node. Helper function. | |
| 2579 | + * @mas: the maple state with the allocations. | |
| 2580 | + * @b_node: the maple_big_node with the type encoding. | |
| 2581 | + * | |
| 2582 | + * Use the node type from the maple_big_node to allocate a new node from the | |
| 2583 | + * ma_state. This function exists mainly for code readability. | |
| 2584 | + * | |
| 2585 | + * Return: A new maple encoded node | |
| 2586 | + */ | |
| 2587 | +static inline struct maple_enode | |
| 2588 | +*mas_new_ma_node(struct ma_state *mas, struct maple_big_node *b_node) | |
| 2589 | +{ | |
| 2590 | + return mt_mk_node(ma_mnode_ptr(mas_pop_node(mas)), b_node->type); | |
| 2591 | +} | |
| 2592 | + | |
| 2593 | +/* | |
| 2594 | + * mas_mab_to_node() - Set up right and middle nodes | |
| 2595 | + * | |
| 2596 | + * @mas: the maple state that contains the allocations. | |
| 2597 | + * @b_node: the node which contains the data. | |
| 2598 | + * @left: The pointer which will have the left node | |
| 2599 | + * @right: The pointer which may have the right node | |
| 2600 | + * @middle: the pointer which may have the middle node (rare) | |
| 2601 | + * @mid_split: the split location for the middle node | |
| 2602 | + * | |
| 2603 | + * Return: the split of left. | |
| 2604 | + */ | |
| 2605 | +static inline unsigned char mas_mab_to_node(struct ma_state *mas, | |
| 2606 | + struct maple_big_node *b_node, struct maple_enode **left, | |
| 2607 | + struct maple_enode **right, struct maple_enode **middle, | |
| 2608 | + unsigned char *mid_split, unsigned long min) | |
| 2609 | +{ | |
| 2610 | + unsigned char split = 0; | |
| 2611 | + unsigned char slot_count = mt_slots[b_node->type]; | |
| 2612 | + | |
| 2613 | + *left = mas_new_ma_node(mas, b_node); | |
| 2614 | + *right = NULL; | |
| 2615 | + *middle = NULL; | |
| 2616 | + *mid_split = 0; | |
| 2617 | + | |
| 2618 | + if (b_node->b_end < slot_count) { | |
| 2619 | + split = b_node->b_end; | |
| 2620 | + } else { | |
| 2621 | + split = mab_calc_split(mas, b_node, mid_split, min); | |
| 2622 | + *right = mas_new_ma_node(mas, b_node); | |
| 2623 | + } | |
| 2624 | + | |
| 2625 | + if (*mid_split) | |
| 2626 | + *middle = mas_new_ma_node(mas, b_node); | |
| 2627 | + | |
| 2628 | + return split; | |
| 2629 | + | |
| 2630 | +} | |
| 2631 | + | |
| 2632 | +/* | |
| 2633 | + * mab_set_b_end() - Add entry to b_node at b_node->b_end and increment the end | |
| 2634 | + * pointer. | |
| 2635 | + * @b_node - the big node to add the entry | |
| 2636 | + * @mas - the maple state to get the pivot (mas->max) | |
| 2637 | + * @entry - the entry to add, if NULL nothing happens. | |
| 2638 | + */ | |
| 2639 | +static inline void mab_set_b_end(struct maple_big_node *b_node, | |
| 2640 | + struct ma_state *mas, | |
| 2641 | + void *entry) | |
| 2642 | +{ | |
| 2643 | + if (!entry) | |
| 2644 | + return; | |
| 2645 | + | |
| 2646 | + b_node->slot[b_node->b_end] = entry; | |
| 2647 | + if (mt_is_alloc(mas->tree)) | |
| 2648 | + b_node->gap[b_node->b_end] = mas_max_gap(mas); | |
| 2649 | + b_node->pivot[b_node->b_end++] = mas->max; | |
| 2650 | +} | |
| 2651 | + | |
| 2652 | +/* | |
| 2653 | + * mas_set_split_parent() - combine_then_separate helper function. Sets the parent | |
| 2654 | + * of @mas->node to either @left or @right, depending on @slot and @split | |
| 2655 | + * | |
| 2656 | + * @mas - the maple state with the node that needs a parent | |
| 2657 | + * @left - possible parent 1 | |
| 2658 | + * @right - possible parent 2 | |
| 2659 | + * @slot - the slot the mas->node was placed | |
| 2660 | + * @split - the split location between @left and @right | |
| 2661 | + */ | |
| 2662 | +static inline void mas_set_split_parent(struct ma_state *mas, | |
| 2663 | + struct maple_enode *left, | |
| 2664 | + struct maple_enode *right, | |
| 2665 | + unsigned char *slot, unsigned char split) | |
| 2666 | +{ | |
| 2667 | + if (mas_is_none(mas)) | |
| 2668 | + return; | |
| 2669 | + | |
| 2670 | + if ((*slot) <= split) | |
| 2671 | + mte_set_parent(mas->node, left, *slot); | |
| 2672 | + else if (right) | |
| 2673 | + mte_set_parent(mas->node, right, (*slot) - split - 1); | |
| 2674 | + | |
| 2675 | + (*slot)++; | |
| 2676 | +} | |
| 2677 | + | |
| 2678 | +/* | |
| 2679 | + * mte_mid_split_check() - Check if the next node passes the mid-split | |
| 2680 | + * @**l: Pointer to left encoded maple node. | |
| 2681 | + * @**m: Pointer to middle encoded maple node. | |
| 2682 | + * @**r: Pointer to right encoded maple node. | |
| 2683 | + * @slot: The offset | |
| 2684 | + * @*split: The split location. | |
| 2685 | + * @mid_split: The middle split. | |
| 2686 | + */ | |
| 2687 | +static inline void mte_mid_split_check(struct maple_enode **l, | |
| 2688 | + struct maple_enode **r, | |
| 2689 | + struct maple_enode *right, | |
| 2690 | + unsigned char slot, | |
| 2691 | + unsigned char *split, | |
| 2692 | + unsigned char mid_split) | |
| 2693 | +{ | |
| 2694 | + if (*r == right) | |
| 2695 | + return; | |
| 2696 | + | |
| 2697 | + if (slot < mid_split) | |
| 2698 | + return; | |
| 2699 | + | |
| 2700 | + *l = *r; | |
| 2701 | + *r = right; | |
| 2702 | + *split = mid_split; | |
| 2703 | +} | |
| 2704 | + | |
| 2705 | +/* | |
| 2706 | + * mast_set_split_parents() - Helper function to set three nodes parents. Slot | |
| 2707 | + * is taken from @mast->l. | |
| 2708 | + * @mast - the maple subtree state | |
| 2709 | + * @left - the left node | |
| 2710 | + * @right - the right node | |
| 2711 | + * @split - the split location. | |
| 2712 | + */ | |
| 2713 | +static inline void mast_set_split_parents(struct maple_subtree_state *mast, | |
| 2714 | + struct maple_enode *left, | |
| 2715 | + struct maple_enode *middle, | |
| 2716 | + struct maple_enode *right, | |
| 2717 | + unsigned char split, | |
| 2718 | + unsigned char mid_split) | |
| 2719 | +{ | |
| 2720 | + unsigned char slot; | |
| 2721 | + struct maple_enode *l = left; | |
| 2722 | + struct maple_enode *r = right; | |
| 2723 | + | |
| 2724 | + if (mas_is_none(mast->l)) | |
| 2725 | + return; | |
| 2726 | + | |
| 2727 | + if (middle) | |
| 2728 | + r = middle; | |
| 2729 | + | |
| 2730 | + slot = mast->l->offset; | |
| 2731 | + | |
| 2732 | + mte_mid_split_check(&l, &r, right, slot, &split, mid_split); | |
| 2733 | + mas_set_split_parent(mast->l, l, r, &slot, split); | |
| 2734 | + | |
| 2735 | + mte_mid_split_check(&l, &r, right, slot, &split, mid_split); | |
| 2736 | + mas_set_split_parent(mast->m, l, r, &slot, split); | |
| 2737 | + | |
| 2738 | + mte_mid_split_check(&l, &r, right, slot, &split, mid_split); | |
| 2739 | + mas_set_split_parent(mast->r, l, r, &slot, split); | |
| 2740 | +} | |
| 2741 | + | |
| 2742 | +/* | |
| 2743 | + * mas_wmb_replace() - Write memory barrier and replace | |
| 2744 | + * @mas: The maple state | |
| 2745 | + * @free: the maple topiary list of nodes to free | |
| 2746 | + * @destroy: The maple topiary list of nodes to destroy (walk and free) | |
| 2747 | + * | |
| 2748 | + * Updates gap as necessary. | |
| 2749 | + */ | |
| 2750 | +static inline void mas_wmb_replace(struct ma_state *mas, | |
| 2751 | + struct ma_topiary *free, | |
| 2752 | + struct ma_topiary *destroy) | |
| 2753 | +{ | |
| 2754 | + /* All nodes must see old data as dead prior to replacing that data */ | |
| 2755 | + smp_wmb(); /* Needed for RCU */ | |
| 2756 | + | |
| 2757 | + /* Insert the new data in the tree */ | |
| 2758 | + mas_replace(mas, true); | |
| 2759 | + | |
| 2760 | + if (!mte_is_leaf(mas->node)) | |
| 2761 | + mas_descend_adopt(mas); | |
| 2762 | + | |
| 2763 | + mas_mat_free(mas, free); | |
| 2764 | + | |
| 2765 | + if (destroy) | |
| 2766 | + mas_mat_destroy(mas, destroy); | |
| 2767 | + | |
| 2768 | + if (mte_is_leaf(mas->node)) | |
| 2769 | + return; | |
| 2770 | + | |
| 2771 | + mas_update_gap(mas); | |
| 2772 | +} | |
| 2773 | + | |
| 2774 | +/* | |
| 2775 | + * mast_new_root() - Set a new tree root during subtree creation | |
| 2776 | + * @mast: The maple subtree state | |
| 2777 | + * @mas: The maple state | |
| 2778 | + */ | |
| 2779 | +static inline void mast_new_root(struct maple_subtree_state *mast, | |
| 2780 | + struct ma_state *mas) | |
| 2781 | +{ | |
| 2782 | + mas_mn(mast->l)->parent = | |
| 2783 | + ma_parent_ptr(((unsigned long)mas->tree | MA_ROOT_PARENT)); | |
| 2784 | + if (!mte_dead_node(mast->orig_l->node) && | |
| 2785 | + !mte_is_root(mast->orig_l->node)) { | |
| 2786 | + do { | |
| 2787 | + mast_ascend_free(mast); | |
| 2788 | + mast_topiary(mast); | |
| 2789 | + } while (!mte_is_root(mast->orig_l->node)); | |
| 2790 | + } | |
| 2791 | + if ((mast->orig_l->node != mas->node) && | |
| 2792 | + (mast->l->depth > mas_mt_height(mas))) { | |
| 2793 | + mat_add(mast->free, mas->node); | |
| 2794 | + } | |
| 2795 | +} | |
| 2796 | + | |
| 2797 | +/* | |
| 2798 | + * mast_cp_to_nodes() - Copy data out to nodes. | |
| 2799 | + * @mast: The maple subtree state | |
| 2800 | + * @left: The left encoded maple node | |
| 2801 | + * @middle: The middle encoded maple node | |
| 2802 | + * @right: The right encoded maple node | |
| 2803 | + * @split: The location to split between left and (middle ? middle : right) | |
| 2804 | + * @mid_split: The location to split between middle and right. | |
| 2805 | + */ | |
| 2806 | +static inline void mast_cp_to_nodes(struct maple_subtree_state *mast, | |
| 2807 | + struct maple_enode *left, struct maple_enode *middle, | |
| 2808 | + struct maple_enode *right, unsigned char split, unsigned char mid_split) | |
| 2809 | +{ | |
| 2810 | + bool new_lmax = true; | |
| 2811 | + | |
| 2812 | + mast->l->node = mte_node_or_none(left); | |
| 2813 | + mast->m->node = mte_node_or_none(middle); | |
| 2814 | + mast->r->node = mte_node_or_none(right); | |
| 2815 | + | |
| 2816 | + mast->l->min = mast->orig_l->min; | |
| 2817 | + if (split == mast->bn->b_end) { | |
| 2818 | + mast->l->max = mast->orig_r->max; | |
| 2819 | + new_lmax = false; | |
| 2820 | + } | |
| 2821 | + | |
| 2822 | + mab_mas_cp(mast->bn, 0, split, mast->l, new_lmax); | |
| 2823 | + | |
| 2824 | + if (middle) { | |
| 2825 | + mab_mas_cp(mast->bn, 1 + split, mid_split, mast->m, true); | |
| 2826 | + mast->m->min = mast->bn->pivot[split] + 1; | |
| 2827 | + split = mid_split; | |
| 2828 | + } | |
| 2829 | + | |
| 2830 | + mast->r->max = mast->orig_r->max; | |
| 2831 | + if (right) { | |
| 2832 | + mab_mas_cp(mast->bn, 1 + split, mast->bn->b_end, mast->r, false); | |
| 2833 | + mast->r->min = mast->bn->pivot[split] + 1; | |
| 2834 | + } | |
| 2835 | +} | |
| 2836 | + | |
| 2837 | +/* | |
| 2838 | + * mast_combine_cp_left - Copy in the original left side of the tree into the | |
| 2839 | + * combined data set in the maple subtree state big node. | |
| 2840 | + * @mast: The maple subtree state | |
| 2841 | + */ | |
| 2842 | +static inline void mast_combine_cp_left(struct maple_subtree_state *mast) | |
| 2843 | +{ | |
| 2844 | + unsigned char l_slot = mast->orig_l->offset; | |
| 2845 | + | |
| 2846 | + if (!l_slot) | |
| 2847 | + return; | |
| 2848 | + | |
| 2849 | + mas_mab_cp(mast->orig_l, 0, l_slot - 1, mast->bn, 0); | |
| 2850 | +} | |
| 2851 | + | |
| 2852 | +/* | |
| 2853 | + * mast_combine_cp_right: Copy in the original right side of the tree into the | |
| 2854 | + * combined data set in the maple subtree state big node. | |
| 2855 | + * @mast: The maple subtree state | |
| 2856 | + */ | |
| 2857 | +static inline void mast_combine_cp_right(struct maple_subtree_state *mast) | |
| 2858 | +{ | |
| 2859 | + if (mast->bn->pivot[mast->bn->b_end - 1] >= mast->orig_r->max) | |
| 2860 | + return; | |
| 2861 | + | |
| 2862 | + mas_mab_cp(mast->orig_r, mast->orig_r->offset + 1, | |
| 2863 | + mt_slot_count(mast->orig_r->node), mast->bn, | |
| 2864 | + mast->bn->b_end); | |
| 2865 | + mast->orig_r->last = mast->orig_r->max; | |
| 2866 | +} | |
| 2867 | + | |
| 2868 | +/* | |
| 2869 | + * mast_sufficient: Check if the maple subtree state has enough data in the big | |
| 2870 | + * node to create at least one sufficient node | |
| 2871 | + * @mast: the maple subtree state | |
| 2872 | + */ | |
| 2873 | +static inline bool mast_sufficient(struct maple_subtree_state *mast) | |
| 2874 | +{ | |
| 2875 | + if (mast->bn->b_end > mt_min_slot_count(mast->orig_l->node)) | |
| 2876 | + return true; | |
| 2877 | + | |
| 2878 | + return false; | |
| 2879 | +} | |
| 2880 | + | |
| 2881 | +/* | |
| 2882 | + * mast_overflow: Check if there is too much data in the subtree state for a | |
| 2883 | + * single node. | |
| 2884 | + * @mast: The maple subtree state | |
| 2885 | + */ | |
| 2886 | +static inline bool mast_overflow(struct maple_subtree_state *mast) | |
| 2887 | +{ | |
| 2888 | + if (mast->bn->b_end >= mt_slot_count(mast->orig_l->node)) | |
| 2889 | + return true; | |
| 2890 | + | |
| 2891 | + return false; | |
| 2892 | +} | |
| 2893 | + | |
| 2894 | +static inline void *mtree_range_walk(struct ma_state *mas) | |
| 2895 | +{ | |
| 2896 | + unsigned long *pivots; | |
| 2897 | + unsigned char offset; | |
| 2898 | + struct maple_node *node; | |
| 2899 | + struct maple_enode *next, *last; | |
| 2900 | + enum maple_type type; | |
| 2901 | + void __rcu **slots; | |
| 2902 | + unsigned char end; | |
| 2903 | + unsigned long max, min; | |
| 2904 | + unsigned long prev_max, prev_min; | |
| 2905 | + | |
| 2906 | + last = next = mas->node; | |
| 2907 | + prev_min = min = mas->min; | |
| 2908 | + max = mas->max; | |
| 2909 | + do { | |
| 2910 | + offset = 0; | |
| 2911 | + last = next; | |
| 2912 | + node = mte_to_node(next); | |
| 2913 | + type = mte_node_type(next); | |
| 2914 | + pivots = ma_pivots(node, type); | |
| 2915 | + end = ma_data_end(node, type, pivots, max); | |
| 2916 | + if (unlikely(ma_dead_node(node))) | |
| 2917 | + goto dead_node; | |
| 2918 | + | |
| 2919 | + if (pivots[offset] >= mas->index) { | |
| 2920 | + prev_max = max; | |
| 2921 | + prev_min = min; | |
| 2922 | + max = pivots[offset]; | |
| 2923 | + goto next; | |
| 2924 | + } | |
| 2925 | + | |
| 2926 | + do { | |
| 2927 | + offset++; | |
| 2928 | + } while ((offset < end) && (pivots[offset] < mas->index)); | |
| 2929 | + | |
| 2930 | + prev_min = min; | |
| 2931 | + min = pivots[offset - 1] + 1; | |
| 2932 | + prev_max = max; | |
| 2933 | + if (likely(offset < end && pivots[offset])) | |
| 2934 | + max = pivots[offset]; | |
| 2935 | + | |
| 2936 | +next: | |
| 2937 | + slots = ma_slots(node, type); | |
| 2938 | + next = mt_slot(mas->tree, slots, offset); | |
| 2939 | + if (unlikely(ma_dead_node(node))) | |
| 2940 | + goto dead_node; | |
| 2941 | + } while (!ma_is_leaf(type)); | |
| 2942 | + | |
| 2943 | + mas->offset = offset; | |
| 2944 | + mas->index = min; | |
| 2945 | + mas->last = max; | |
| 2946 | + mas->min = prev_min; | |
| 2947 | + mas->max = prev_max; | |
| 2948 | + mas->node = last; | |
| 2949 | + return (void *) next; | |
| 2950 | + | |
| 2951 | +dead_node: | |
| 2952 | + mas_reset(mas); | |
| 2953 | + return NULL; | |
| 2954 | +} | |
| 2955 | + | |
| 2956 | +/* | |
| 2957 | + * mas_spanning_rebalance() - Rebalance across two nodes which may not be peers. | |
| 2958 | + * @mas: The starting maple state | |
| 2959 | + * @mast: The maple_subtree_state, keeps track of 4 maple states. | |
| 2960 | + * @count: The estimated count of iterations needed. | |
| 2961 | + * | |
| 2962 | + * Follow the tree upwards from @l_mas and @r_mas for @count, or until the root | |
| 2963 | + * is hit. First @b_node is split into two entries which are inserted into the | |
| 2964 | + * next iteration of the loop. @b_node is returned populated with the final | |
| 2965 | + * iteration. @mas is used to obtain allocations. orig_l_mas keeps track of the | |
| 2966 | + * nodes that will remain active by using orig_l_mas->index and orig_l_mas->last | |
| 2967 | + * to account of what has been copied into the new sub-tree. The update of | |
| 2968 | + * orig_l_mas->last is used in mas_consume to find the slots that will need to | |
| 2969 | + * be either freed or destroyed. orig_l_mas->depth keeps track of the height of | |
| 2970 | + * the new sub-tree in case the sub-tree becomes the full tree. | |
| 2971 | + * | |
| 2972 | + * Return: the number of elements in b_node during the last loop. | |
| 2973 | + */ | |
| 2974 | +static int mas_spanning_rebalance(struct ma_state *mas, | |
| 2975 | + struct maple_subtree_state *mast, unsigned char count) | |
| 2976 | +{ | |
| 2977 | + unsigned char split, mid_split; | |
| 2978 | + unsigned char slot = 0; | |
| 2979 | + struct maple_enode *left = NULL, *middle = NULL, *right = NULL; | |
| 2980 | + | |
| 2981 | + MA_STATE(l_mas, mas->tree, mas->index, mas->index); | |
| 2982 | + MA_STATE(r_mas, mas->tree, mas->index, mas->last); | |
| 2983 | + MA_STATE(m_mas, mas->tree, mas->index, mas->index); | |
| 2984 | + MA_TOPIARY(free, mas->tree); | |
| 2985 | + MA_TOPIARY(destroy, mas->tree); | |
| 2986 | + | |
| 2987 | + /* | |
| 2988 | + * The tree needs to be rebalanced and leaves need to be kept at the same level. | |
| 2989 | + * Rebalancing is done by use of the ``struct maple_topiary``. | |
| 2990 | + */ | |
| 2991 | + mast->l = &l_mas; | |
| 2992 | + mast->m = &m_mas; | |
| 2993 | + mast->r = &r_mas; | |
| 2994 | + mast->free = &free; | |
| 2995 | + mast->destroy = &destroy; | |
| 2996 | + l_mas.node = r_mas.node = m_mas.node = MAS_NONE; | |
| 2997 | + if (!(mast->orig_l->min && mast->orig_r->max == ULONG_MAX) && | |
| 2998 | + unlikely(mast->bn->b_end <= mt_min_slots[mast->bn->type])) | |
| 2999 | + mast_spanning_rebalance(mast); | |
| 3000 | + | |
| 3001 | + mast->orig_l->depth = 0; | |
| 3002 | + | |
| 3003 | + /* | |
| 3004 | + * Each level of the tree is examined and balanced, pushing data to the left or | |
| 3005 | + * right, or rebalancing against left or right nodes is employed to avoid | |
| 3006 | + * rippling up the tree to limit the amount of churn. Once a new sub-section of | |
| 3007 | + * the tree is created, there may be a mix of new and old nodes. The old nodes | |
| 3008 | + * will have the incorrect parent pointers and currently be in two trees: the | |
| 3009 | + * original tree and the partially new tree. To remedy the parent pointers in | |
| 3010 | + * the old tree, the new data is swapped into the active tree and a walk down | |
| 3011 | + * the tree is performed and the parent pointers are updated. | |
| 3012 | + * See mas_descend_adopt() for more information.. | |
| 3013 | + */ | |
| 3014 | + while (count--) { | |
| 3015 | + mast->bn->b_end--; | |
| 3016 | + mast->bn->type = mte_node_type(mast->orig_l->node); | |
| 3017 | + split = mas_mab_to_node(mas, mast->bn, &left, &right, &middle, | |
| 3018 | + &mid_split, mast->orig_l->min); | |
| 3019 | + mast_set_split_parents(mast, left, middle, right, split, | |
| 3020 | + mid_split); | |
| 3021 | + mast_cp_to_nodes(mast, left, middle, right, split, mid_split); | |
| 3022 | + | |
| 3023 | + /* | |
| 3024 | + * Copy data from next level in the tree to mast->bn from next | |
| 3025 | + * iteration | |
| 3026 | + */ | |
| 3027 | + memset(mast->bn, 0, sizeof(struct maple_big_node)); | |
| 3028 | + mast->bn->type = mte_node_type(left); | |
| 3029 | + mast->orig_l->depth++; | |
| 3030 | + | |
| 3031 | + /* Root already stored in l->node. */ | |
| 3032 | + if (mas_is_root_limits(mast->l)) | |
| 3033 | + goto new_root; | |
| 3034 | + | |
| 3035 | + mast_ascend_free(mast); | |
| 3036 | + mast_combine_cp_left(mast); | |
| 3037 | + l_mas.offset = mast->bn->b_end; | |
| 3038 | + mab_set_b_end(mast->bn, &l_mas, left); | |
| 3039 | + mab_set_b_end(mast->bn, &m_mas, middle); | |
| 3040 | + mab_set_b_end(mast->bn, &r_mas, right); | |
| 3041 | + | |
| 3042 | + /* Copy anything necessary out of the right node. */ | |
| 3043 | + mast_combine_cp_right(mast); | |
| 3044 | + mast_topiary(mast); | |
| 3045 | + mast->orig_l->last = mast->orig_l->max; | |
| 3046 | + | |
| 3047 | + if (mast_sufficient(mast)) | |
| 3048 | + continue; | |
| 3049 | + | |
| 3050 | + if (mast_overflow(mast)) | |
| 3051 | + continue; | |
| 3052 | + | |
| 3053 | + /* May be a new root stored in mast->bn */ | |
| 3054 | + if (mas_is_root_limits(mast->orig_l)) | |
| 3055 | + break; | |
| 3056 | + | |
| 3057 | + mast_spanning_rebalance(mast); | |
| 3058 | + | |
| 3059 | + /* rebalancing from other nodes may require another loop. */ | |
| 3060 | + if (!count) | |
| 3061 | + count++; | |
| 3062 | + } | |
| 3063 | + | |
| 3064 | + l_mas.node = mt_mk_node(ma_mnode_ptr(mas_pop_node(mas)), | |
| 3065 | + mte_node_type(mast->orig_l->node)); | |
| 3066 | + mast->orig_l->depth++; | |
| 3067 | + mab_mas_cp(mast->bn, 0, mt_slots[mast->bn->type] - 1, &l_mas, true); | |
| 3068 | + mte_set_parent(left, l_mas.node, slot); | |
| 3069 | + if (middle) | |
| 3070 | + mte_set_parent(middle, l_mas.node, ++slot); | |
| 3071 | + | |
| 3072 | + if (right) | |
| 3073 | + mte_set_parent(right, l_mas.node, ++slot); | |
| 3074 | + | |
| 3075 | + if (mas_is_root_limits(mast->l)) { | |
| 3076 | +new_root: | |
| 3077 | + mast_new_root(mast, mas); | |
| 3078 | + } else { | |
| 3079 | + mas_mn(&l_mas)->parent = mas_mn(mast->orig_l)->parent; | |
| 3080 | + } | |
| 3081 | + | |
| 3082 | + if (!mte_dead_node(mast->orig_l->node)) | |
| 3083 | + mat_add(&free, mast->orig_l->node); | |
| 3084 | + | |
| 3085 | + mas->depth = mast->orig_l->depth; | |
| 3086 | + *mast->orig_l = l_mas; | |
| 3087 | + mte_set_node_dead(mas->node); | |
| 3088 | + | |
| 3089 | + /* Set up mas for insertion. */ | |
| 3090 | + mast->orig_l->depth = mas->depth; | |
| 3091 | + mast->orig_l->alloc = mas->alloc; | |
| 3092 | + *mas = *mast->orig_l; | |
| 3093 | + mas_wmb_replace(mas, &free, &destroy); | |
| 3094 | + mtree_range_walk(mas); | |
| 3095 | + return mast->bn->b_end; | |
| 3096 | +} | |
| 3097 | + | |
| 3098 | +/* | |
| 3099 | + * mas_rebalance() - Rebalance a given node. | |
| 3100 | + * @mas: The maple state | |
| 3101 | + * @b_node: The big maple node. | |
| 3102 | + * | |
| 3103 | + * Rebalance two nodes into a single node or two new nodes that are sufficient. | |
| 3104 | + * Continue upwards until tree is sufficient. | |
| 3105 | + * | |
| 3106 | + * Return: the number of elements in b_node during the last loop. | |
| 3107 | + */ | |
| 3108 | +static inline int mas_rebalance(struct ma_state *mas, | |
| 3109 | + struct maple_big_node *b_node) | |
| 3110 | +{ | |
| 3111 | + char empty_count = mas_mt_height(mas); | |
| 3112 | + struct maple_subtree_state mast; | |
| 3113 | + unsigned char shift, b_end = ++b_node->b_end; | |
| 3114 | + | |
| 3115 | + MA_STATE(l_mas, mas->tree, mas->index, mas->last); | |
| 3116 | + MA_STATE(r_mas, mas->tree, mas->index, mas->last); | |
| 3117 | + | |
| 3118 | + trace_ma_op(__func__, mas); | |
| 3119 | + | |
| 3120 | + /* | |
| 3121 | + * Rebalancing occurs if a node is insufficient. Data is rebalanced | |
| 3122 | + * against the node to the right if it exists, otherwise the node to the | |
| 3123 | + * left of this node is rebalanced against this node. If rebalancing | |
| 3124 | + * causes just one node to be produced instead of two, then the parent | |
| 3125 | + * is also examined and rebalanced if it is insufficient. Every level | |
| 3126 | + * tries to combine the data in the same way. If one node contains the | |
| 3127 | + * entire range of the tree, then that node is used as a new root node. | |
| 3128 | + */ | |
| 3129 | + mas_node_count(mas, 1 + empty_count * 3); | |
| 3130 | + if (mas_is_err(mas)) | |
| 3131 | + return 0; | |
| 3132 | + | |
| 3133 | + mast.orig_l = &l_mas; | |
| 3134 | + mast.orig_r = &r_mas; | |
| 3135 | + mast.bn = b_node; | |
| 3136 | + mast.bn->type = mte_node_type(mas->node); | |
| 3137 | + | |
| 3138 | + l_mas = r_mas = *mas; | |
| 3139 | + | |
| 3140 | + if (mas_next_sibling(&r_mas)) { | |
| 3141 | + mas_mab_cp(&r_mas, 0, mt_slot_count(r_mas.node), b_node, b_end); | |
| 3142 | + r_mas.last = r_mas.index = r_mas.max; | |
| 3143 | + } else { | |
| 3144 | + mas_prev_sibling(&l_mas); | |
| 3145 | + shift = mas_data_end(&l_mas) + 1; | |
| 3146 | + mab_shift_right(b_node, shift); | |
| 3147 | + mas->offset += shift; | |
| 3148 | + mas_mab_cp(&l_mas, 0, shift - 1, b_node, 0); | |
| 3149 | + b_node->b_end = shift + b_end; | |
| 3150 | + l_mas.index = l_mas.last = l_mas.min; | |
| 3151 | + } | |
| 3152 | + | |
| 3153 | + return mas_spanning_rebalance(mas, &mast, empty_count); | |
| 3154 | +} | |
| 3155 | + | |
| 3156 | +/* | |
| 3157 | + * mas_destroy_rebalance() - Rebalance left-most node while destroying the maple | |
| 3158 | + * state. | |
| 3159 | + * @mas: The maple state | |
| 3160 | + * @end: The end of the left-most node. | |
| 3161 | + * | |
| 3162 | + * During a mass-insert event (such as forking), it may be necessary to | |
| 3163 | + * rebalance the left-most node when it is not sufficient. | |
| 3164 | + */ | |
| 3165 | +static inline void mas_destroy_rebalance(struct ma_state *mas, unsigned char end) | |
| 3166 | +{ | |
| 3167 | + enum maple_type mt = mte_node_type(mas->node); | |
| 3168 | + struct maple_node reuse, *newnode, *parent, *new_left, *left, *node; | |
| 3169 | + struct maple_enode *eparent; | |
| 3170 | + unsigned char offset, tmp, split = mt_slots[mt] / 2; | |
| 3171 | + void __rcu **l_slots, **slots; | |
| 3172 | + unsigned long *l_pivs, *pivs, gap; | |
| 3173 | + bool in_rcu = mt_in_rcu(mas->tree); | |
| 3174 | + | |
| 3175 | + MA_STATE(l_mas, mas->tree, mas->index, mas->last); | |
| 3176 | + | |
| 3177 | + l_mas = *mas; | |
| 3178 | + mas_prev_sibling(&l_mas); | |
| 3179 | + | |
| 3180 | + /* set up node. */ | |
| 3181 | + if (in_rcu) { | |
| 3182 | + /* Allocate for both left and right as well as parent. */ | |
| 3183 | + mas_node_count(mas, 3); | |
| 3184 | + if (mas_is_err(mas)) | |
| 3185 | + return; | |
| 3186 | + | |
| 3187 | + newnode = mas_pop_node(mas); | |
| 3188 | + } else { | |
| 3189 | + newnode = &reuse; | |
| 3190 | + } | |
| 3191 | + | |
| 3192 | + node = mas_mn(mas); | |
| 3193 | + newnode->parent = node->parent; | |
| 3194 | + slots = ma_slots(newnode, mt); | |
| 3195 | + pivs = ma_pivots(newnode, mt); | |
| 3196 | + left = mas_mn(&l_mas); | |
| 3197 | + l_slots = ma_slots(left, mt); | |
| 3198 | + l_pivs = ma_pivots(left, mt); | |
| 3199 | + if (!l_slots[split]) | |
| 3200 | + split++; | |
| 3201 | + tmp = mas_data_end(&l_mas) - split; | |
| 3202 | + | |
| 3203 | + memcpy(slots, l_slots + split + 1, sizeof(void *) * tmp); | |
| 3204 | + memcpy(pivs, l_pivs + split + 1, sizeof(unsigned long) * tmp); | |
| 3205 | + pivs[tmp] = l_mas.max; | |
| 3206 | + memcpy(slots + tmp, ma_slots(node, mt), sizeof(void *) * end); | |
| 3207 | + memcpy(pivs + tmp, ma_pivots(node, mt), sizeof(unsigned long) * end); | |
| 3208 | + | |
| 3209 | + l_mas.max = l_pivs[split]; | |
| 3210 | + mas->min = l_mas.max + 1; | |
| 3211 | + eparent = mt_mk_node(mte_parent(l_mas.node), | |
| 3212 | + mas_parent_enum(&l_mas, l_mas.node)); | |
| 3213 | + tmp += end; | |
| 3214 | + if (!in_rcu) { | |
| 3215 | + unsigned char max_p = mt_pivots[mt]; | |
| 3216 | + unsigned char max_s = mt_slots[mt]; | |
| 3217 | + | |
| 3218 | + if (tmp < max_p) | |
| 3219 | + memset(pivs + tmp, 0, | |
| 3220 | + sizeof(unsigned long *) * (max_p - tmp)); | |
| 3221 | + | |
| 3222 | + if (tmp < mt_slots[mt]) | |
| 3223 | + memset(slots + tmp, 0, sizeof(void *) * (max_s - tmp)); | |
| 3224 | + | |
| 3225 | + memcpy(node, newnode, sizeof(struct maple_node)); | |
| 3226 | + ma_set_meta(node, mt, 0, tmp - 1); | |
| 3227 | + mte_set_pivot(eparent, mte_parent_slot(l_mas.node), | |
| 3228 | + l_pivs[split]); | |
| 3229 | + | |
| 3230 | + /* Remove data from l_pivs. */ | |
| 3231 | + tmp = split + 1; | |
| 3232 | + memset(l_pivs + tmp, 0, sizeof(unsigned long) * (max_p - tmp)); | |
| 3233 | + memset(l_slots + tmp, 0, sizeof(void *) * (max_s - tmp)); | |
| 3234 | + ma_set_meta(left, mt, 0, split); | |
| 3235 | + | |
| 3236 | + goto done; | |
| 3237 | + } | |
| 3238 | + | |
| 3239 | + /* RCU requires replacing both l_mas, mas, and parent. */ | |
| 3240 | + mas->node = mt_mk_node(newnode, mt); | |
| 3241 | + ma_set_meta(newnode, mt, 0, tmp); | |
| 3242 | + | |
| 3243 | + new_left = mas_pop_node(mas); | |
| 3244 | + new_left->parent = left->parent; | |
| 3245 | + mt = mte_node_type(l_mas.node); | |
| 3246 | + slots = ma_slots(new_left, mt); | |
| 3247 | + pivs = ma_pivots(new_left, mt); | |
| 3248 | + memcpy(slots, l_slots, sizeof(void *) * split); | |
| 3249 | + memcpy(pivs, l_pivs, sizeof(unsigned long) * split); | |
| 3250 | + ma_set_meta(new_left, mt, 0, split); | |
| 3251 | + l_mas.node = mt_mk_node(new_left, mt); | |
| 3252 | + | |
| 3253 | + /* replace parent. */ | |
| 3254 | + offset = mte_parent_slot(mas->node); | |
| 3255 | + mt = mas_parent_enum(&l_mas, l_mas.node); | |
| 3256 | + parent = mas_pop_node(mas); | |
| 3257 | + slots = ma_slots(parent, mt); | |
| 3258 | + pivs = ma_pivots(parent, mt); | |
| 3259 | + memcpy(parent, mte_to_node(eparent), sizeof(struct maple_node)); | |
| 3260 | + rcu_assign_pointer(slots[offset], mas->node); | |
| 3261 | + rcu_assign_pointer(slots[offset - 1], l_mas.node); | |
| 3262 | + pivs[offset - 1] = l_mas.max; | |
| 3263 | + eparent = mt_mk_node(parent, mt); | |
| 3264 | +done: | |
| 3265 | + gap = mas_leaf_max_gap(mas); | |
| 3266 | + mte_set_gap(eparent, mte_parent_slot(mas->node), gap); | |
| 3267 | + gap = mas_leaf_max_gap(&l_mas); | |
| 3268 | + mte_set_gap(eparent, mte_parent_slot(l_mas.node), gap); | |
| 3269 | + mas_ascend(mas); | |
| 3270 | + | |
| 3271 | + if (in_rcu) | |
| 3272 | + mas_replace(mas, false); | |
| 3273 | + | |
| 3274 | + mas_update_gap(mas); | |
| 3275 | +} | |
| 3276 | + | |
| 3277 | +/* | |
| 3278 | + * mas_split_final_node() - Split the final node in a subtree operation. | |
| 3279 | + * @mast: the maple subtree state | |
| 3280 | + * @mas: The maple state | |
| 3281 | + * @height: The height of the tree in case it's a new root. | |
| 3282 | + */ | |
| 3283 | +static inline bool mas_split_final_node(struct maple_subtree_state *mast, | |
| 3284 | + struct ma_state *mas, int height) | |
| 3285 | +{ | |
| 3286 | + struct maple_enode *ancestor; | |
| 3287 | + | |
| 3288 | + if (mte_is_root(mas->node)) { | |
| 3289 | + if (mt_is_alloc(mas->tree)) | |
| 3290 | + mast->bn->type = maple_arange_64; | |
| 3291 | + else | |
| 3292 | + mast->bn->type = maple_range_64; | |
| 3293 | + mas->depth = height; | |
| 3294 | + } | |
| 3295 | + /* | |
| 3296 | + * Only a single node is used here, could be root. | |
| 3297 | + * The Big_node data should just fit in a single node. | |
| 3298 | + */ | |
| 3299 | + ancestor = mas_new_ma_node(mas, mast->bn); | |
| 3300 | + mte_set_parent(mast->l->node, ancestor, mast->l->offset); | |
| 3301 | + mte_set_parent(mast->r->node, ancestor, mast->r->offset); | |
| 3302 | + mte_to_node(ancestor)->parent = mas_mn(mas)->parent; | |
| 3303 | + | |
| 3304 | + mast->l->node = ancestor; | |
| 3305 | + mab_mas_cp(mast->bn, 0, mt_slots[mast->bn->type] - 1, mast->l, true); | |
| 3306 | + mas->offset = mast->bn->b_end - 1; | |
| 3307 | + return true; | |
| 3308 | +} | |
| 3309 | + | |
| 3310 | +/* | |
| 3311 | + * mast_fill_bnode() - Copy data into the big node in the subtree state | |
| 3312 | + * @mast: The maple subtree state | |
| 3313 | + * @mas: the maple state | |
| 3314 | + * @skip: The number of entries to skip for new nodes insertion. | |
| 3315 | + */ | |
| 3316 | +static inline void mast_fill_bnode(struct maple_subtree_state *mast, | |
| 3317 | + struct ma_state *mas, | |
| 3318 | + unsigned char skip) | |
| 3319 | +{ | |
| 3320 | + bool cp = true; | |
| 3321 | + struct maple_enode *old = mas->node; | |
| 3322 | + unsigned char split; | |
| 3323 | + | |
| 3324 | + memset(mast->bn->gap, 0, sizeof(unsigned long) * ARRAY_SIZE(mast->bn->gap)); | |
| 3325 | + memset(mast->bn->slot, 0, sizeof(unsigned long) * ARRAY_SIZE(mast->bn->slot)); | |
| 3326 | + memset(mast->bn->pivot, 0, sizeof(unsigned long) * ARRAY_SIZE(mast->bn->pivot)); | |
| 3327 | + mast->bn->b_end = 0; | |
| 3328 | + | |
| 3329 | + if (mte_is_root(mas->node)) { | |
| 3330 | + cp = false; | |
| 3331 | + } else { | |
| 3332 | + mas_ascend(mas); | |
| 3333 | + mat_add(mast->free, old); | |
| 3334 | + mas->offset = mte_parent_slot(mas->node); | |
| 3335 | + } | |
| 3336 | + | |
| 3337 | + if (cp && mast->l->offset) | |
| 3338 | + mas_mab_cp(mas, 0, mast->l->offset - 1, mast->bn, 0); | |
| 3339 | + | |
| 3340 | + split = mast->bn->b_end; | |
| 3341 | + mab_set_b_end(mast->bn, mast->l, mast->l->node); | |
| 3342 | + mast->r->offset = mast->bn->b_end; | |
| 3343 | + mab_set_b_end(mast->bn, mast->r, mast->r->node); | |
| 3344 | + if (mast->bn->pivot[mast->bn->b_end - 1] == mas->max) | |
| 3345 | + cp = false; | |
| 3346 | + | |
| 3347 | + if (cp) | |
| 3348 | + mas_mab_cp(mas, split + skip, mt_slot_count(mas->node) - 1, | |
| 3349 | + mast->bn, mast->bn->b_end); | |
| 3350 | + | |
| 3351 | + mast->bn->b_end--; | |
| 3352 | + mast->bn->type = mte_node_type(mas->node); | |
| 3353 | +} | |
| 3354 | + | |
| 3355 | +/* | |
| 3356 | + * mast_split_data() - Split the data in the subtree state big node into regular | |
| 3357 | + * nodes. | |
| 3358 | + * @mast: The maple subtree state | |
| 3359 | + * @mas: The maple state | |
| 3360 | + * @split: The location to split the big node | |
| 3361 | + */ | |
| 3362 | +static inline void mast_split_data(struct maple_subtree_state *mast, | |
| 3363 | + struct ma_state *mas, unsigned char split) | |
| 3364 | +{ | |
| 3365 | + unsigned char p_slot; | |
| 3366 | + | |
| 3367 | + mab_mas_cp(mast->bn, 0, split, mast->l, true); | |
| 3368 | + mte_set_pivot(mast->r->node, 0, mast->r->max); | |
| 3369 | + mab_mas_cp(mast->bn, split + 1, mast->bn->b_end, mast->r, false); | |
| 3370 | + mast->l->offset = mte_parent_slot(mas->node); | |
| 3371 | + mast->l->max = mast->bn->pivot[split]; | |
| 3372 | + mast->r->min = mast->l->max + 1; | |
| 3373 | + if (mte_is_leaf(mas->node)) | |
| 3374 | + return; | |
| 3375 | + | |
| 3376 | + p_slot = mast->orig_l->offset; | |
| 3377 | + mas_set_split_parent(mast->orig_l, mast->l->node, mast->r->node, | |
| 3378 | + &p_slot, split); | |
| 3379 | + mas_set_split_parent(mast->orig_r, mast->l->node, mast->r->node, | |
| 3380 | + &p_slot, split); | |
| 3381 | +} | |
| 3382 | + | |
| 3383 | +/* | |
| 3384 | + * mas_push_data() - Instead of splitting a node, it is beneficial to push the | |
| 3385 | + * data to the right or left node if there is room. | |
| 3386 | + * @mas: The maple state | |
| 3387 | + * @height: The current height of the maple state | |
| 3388 | + * @mast: The maple subtree state | |
| 3389 | + * @left: Push left or not. | |
| 3390 | + * | |
| 3391 | + * Keeping the height of the tree low means faster lookups. | |
| 3392 | + * | |
| 3393 | + * Return: True if pushed, false otherwise. | |
| 3394 | + */ | |
| 3395 | +static inline bool mas_push_data(struct ma_state *mas, int height, | |
| 3396 | + struct maple_subtree_state *mast, bool left) | |
| 3397 | +{ | |
| 3398 | + unsigned char slot_total = mast->bn->b_end; | |
| 3399 | + unsigned char end, space, split; | |
| 3400 | + | |
| 3401 | + MA_STATE(tmp_mas, mas->tree, mas->index, mas->last); | |
| 3402 | + tmp_mas = *mas; | |
| 3403 | + tmp_mas.depth = mast->l->depth; | |
| 3404 | + | |
| 3405 | + if (left && !mas_prev_sibling(&tmp_mas)) | |
| 3406 | + return false; | |
| 3407 | + else if (!left && !mas_next_sibling(&tmp_mas)) | |
| 3408 | + return false; | |
| 3409 | + | |
| 3410 | + end = mas_data_end(&tmp_mas); | |
| 3411 | + slot_total += end; | |
| 3412 | + space = 2 * mt_slot_count(mas->node) - 2; | |
| 3413 | + /* -2 instead of -1 to ensure there isn't a triple split */ | |
| 3414 | + if (ma_is_leaf(mast->bn->type)) | |
| 3415 | + space--; | |
| 3416 | + | |
| 3417 | + if (mas->max == ULONG_MAX) | |
| 3418 | + space--; | |
| 3419 | + | |
| 3420 | + if (slot_total >= space) | |
| 3421 | + return false; | |
| 3422 | + | |
| 3423 | + /* Get the data; Fill mast->bn */ | |
| 3424 | + mast->bn->b_end++; | |
| 3425 | + if (left) { | |
| 3426 | + mab_shift_right(mast->bn, end + 1); | |
| 3427 | + mas_mab_cp(&tmp_mas, 0, end, mast->bn, 0); | |
| 3428 | + mast->bn->b_end = slot_total + 1; | |
| 3429 | + } else { | |
| 3430 | + mas_mab_cp(&tmp_mas, 0, end, mast->bn, mast->bn->b_end); | |
| 3431 | + } | |
| 3432 | + | |
| 3433 | + /* Configure mast for splitting of mast->bn */ | |
| 3434 | + split = mt_slots[mast->bn->type] - 2; | |
| 3435 | + if (left) { | |
| 3436 | + /* Switch mas to prev node */ | |
| 3437 | + mat_add(mast->free, mas->node); | |
| 3438 | + *mas = tmp_mas; | |
| 3439 | + /* Start using mast->l for the left side. */ | |
| 3440 | + tmp_mas.node = mast->l->node; | |
| 3441 | + *mast->l = tmp_mas; | |
| 3442 | + } else { | |
| 3443 | + mat_add(mast->free, tmp_mas.node); | |
| 3444 | + tmp_mas.node = mast->r->node; | |
| 3445 | + *mast->r = tmp_mas; | |
| 3446 | + split = slot_total - split; | |
| 3447 | + } | |
| 3448 | + split = mab_no_null_split(mast->bn, split, mt_slots[mast->bn->type]); | |
| 3449 | + /* Update parent slot for split calculation. */ | |
| 3450 | + if (left) | |
| 3451 | + mast->orig_l->offset += end + 1; | |
| 3452 | + | |
| 3453 | + mast_split_data(mast, mas, split); | |
| 3454 | + mast_fill_bnode(mast, mas, 2); | |
| 3455 | + mas_split_final_node(mast, mas, height + 1); | |
| 3456 | + return true; | |
| 3457 | +} | |
| 3458 | + | |
| 3459 | +/* | |
| 3460 | + * mas_split() - Split data that is too big for one node into two. | |
| 3461 | + * @mas: The maple state | |
| 3462 | + * @b_node: The maple big node | |
| 3463 | + * Return: 1 on success, 0 on failure. | |
| 3464 | + */ | |
| 3465 | +static int mas_split(struct ma_state *mas, struct maple_big_node *b_node) | |
| 3466 | +{ | |
| 3467 | + | |
| 3468 | + struct maple_subtree_state mast; | |
| 3469 | + int height = 0; | |
| 3470 | + unsigned char mid_split, split = 0; | |
| 3471 | + | |
| 3472 | + /* | |
| 3473 | + * Splitting is handled differently from any other B-tree; the Maple | |
| 3474 | + * Tree splits upwards. Splitting up means that the split operation | |
| 3475 | + * occurs when the walk of the tree hits the leaves and not on the way | |
| 3476 | + * down. The reason for splitting up is that it is impossible to know | |
| 3477 | + * how much space will be needed until the leaf is (or leaves are) | |
| 3478 | + * reached. Since overwriting data is allowed and a range could | |
| 3479 | + * overwrite more than one range or result in changing one entry into 3 | |
| 3480 | + * entries, it is impossible to know if a split is required until the | |
| 3481 | + * data is examined. | |
| 3482 | + * | |
| 3483 | + * Splitting is a balancing act between keeping allocations to a minimum | |
| 3484 | + * and avoiding a 'jitter' event where a tree is expanded to make room | |
| 3485 | + * for an entry followed by a contraction when the entry is removed. To | |
| 3486 | + * accomplish the balance, there are empty slots remaining in both left | |
| 3487 | + * and right nodes after a split. | |
| 3488 | + */ | |
| 3489 | + MA_STATE(l_mas, mas->tree, mas->index, mas->last); | |
| 3490 | + MA_STATE(r_mas, mas->tree, mas->index, mas->last); | |
| 3491 | + MA_STATE(prev_l_mas, mas->tree, mas->index, mas->last); | |
| 3492 | + MA_STATE(prev_r_mas, mas->tree, mas->index, mas->last); | |
| 3493 | + MA_TOPIARY(mat, mas->tree); | |
| 3494 | + | |
| 3495 | + trace_ma_op(__func__, mas); | |
| 3496 | + mas->depth = mas_mt_height(mas); | |
| 3497 | + /* Allocation failures will happen early. */ | |
| 3498 | + mas_node_count(mas, 1 + mas->depth * 2); | |
| 3499 | + if (mas_is_err(mas)) | |
| 3500 | + return 0; | |
| 3501 | + | |
| 3502 | + mast.l = &l_mas; | |
| 3503 | + mast.r = &r_mas; | |
| 3504 | + mast.orig_l = &prev_l_mas; | |
| 3505 | + mast.orig_r = &prev_r_mas; | |
| 3506 | + mast.free = &mat; | |
| 3507 | + mast.bn = b_node; | |
| 3508 | + | |
| 3509 | + while (height++ <= mas->depth) { | |
| 3510 | + if (mt_slots[b_node->type] > b_node->b_end) { | |
| 3511 | + mas_split_final_node(&mast, mas, height); | |
| 3512 | + break; | |
| 3513 | + } | |
| 3514 | + | |
| 3515 | + l_mas = r_mas = *mas; | |
| 3516 | + l_mas.node = mas_new_ma_node(mas, b_node); | |
| 3517 | + r_mas.node = mas_new_ma_node(mas, b_node); | |
| 3518 | + /* | |
| 3519 | + * Another way that 'jitter' is avoided is to terminate a split up early if the | |
| 3520 | + * left or right node has space to spare. This is referred to as "pushing left" | |
| 3521 | + * or "pushing right" and is similar to the B* tree, except the nodes left or | |
| 3522 | + * right can rarely be reused due to RCU, but the ripple upwards is halted which | |
| 3523 | + * is a significant savings. | |
| 3524 | + */ | |
| 3525 | + /* Try to push left. */ | |
| 3526 | + if (mas_push_data(mas, height, &mast, true)) | |
| 3527 | + break; | |
| 3528 | + | |
| 3529 | + /* Try to push right. */ | |
| 3530 | + if (mas_push_data(mas, height, &mast, false)) | |
| 3531 | + break; | |
| 3532 | + | |
| 3533 | + split = mab_calc_split(mas, b_node, &mid_split, prev_l_mas.min); | |
| 3534 | + mast_split_data(&mast, mas, split); | |
| 3535 | + /* | |
| 3536 | + * Usually correct, mab_mas_cp in the above call overwrites | |
| 3537 | + * r->max. | |
| 3538 | + */ | |
| 3539 | + mast.r->max = mas->max; | |
| 3540 | + mast_fill_bnode(&mast, mas, 1); | |
| 3541 | + prev_l_mas = *mast.l; | |
| 3542 | + prev_r_mas = *mast.r; | |
| 3543 | + } | |
| 3544 | + | |
| 3545 | + /* Set the original node as dead */ | |
| 3546 | + mat_add(mast.free, mas->node); | |
| 3547 | + mas->node = l_mas.node; | |
| 3548 | + mas_wmb_replace(mas, mast.free, NULL); | |
| 3549 | + mtree_range_walk(mas); | |
| 3550 | + return 1; | |
| 3551 | +} | |
| 3552 | + | |
| 3553 | +/* | |
| 3554 | + * mas_reuse_node() - Reuse the node to store the data. | |
| 3555 | + * @wr_mas: The maple write state | |
| 3556 | + * @bn: The maple big node | |
| 3557 | + * @end: The end of the data. | |
| 3558 | + * | |
| 3559 | + * Will always return false in RCU mode. | |
| 3560 | + * | |
| 3561 | + * Return: True if node was reused, false otherwise. | |
| 3562 | + */ | |
| 3563 | +static inline bool mas_reuse_node(struct ma_wr_state *wr_mas, | |
| 3564 | + struct maple_big_node *bn, unsigned char end) | |
| 3565 | +{ | |
| 3566 | + /* Need to be rcu safe. */ | |
| 3567 | + if (mt_in_rcu(wr_mas->mas->tree)) | |
| 3568 | + return false; | |
| 3569 | + | |
| 3570 | + if (end > bn->b_end) { | |
| 3571 | + int clear = mt_slots[wr_mas->type] - bn->b_end; | |
| 3572 | + | |
| 3573 | + memset(wr_mas->slots + bn->b_end, 0, sizeof(void *) * clear--); | |
| 3574 | + memset(wr_mas->pivots + bn->b_end, 0, sizeof(void *) * clear); | |
| 3575 | + } | |
| 3576 | + mab_mas_cp(bn, 0, bn->b_end, wr_mas->mas, false); | |
| 3577 | + return true; | |
| 3578 | +} | |
| 3579 | + | |
| 3580 | +/* | |
| 3581 | + * mas_commit_b_node() - Commit the big node into the tree. | |
| 3582 | + * @wr_mas: The maple write state | |
| 3583 | + * @b_node: The maple big node | |
| 3584 | + * @end: The end of the data. | |
| 3585 | + */ | |
| 3586 | +static inline int mas_commit_b_node(struct ma_wr_state *wr_mas, | |
| 3587 | + struct maple_big_node *b_node, unsigned char end) | |
| 3588 | +{ | |
| 3589 | + struct maple_node *node; | |
| 3590 | + unsigned char b_end = b_node->b_end; | |
| 3591 | + enum maple_type b_type = b_node->type; | |
| 3592 | + | |
| 3593 | + if ((b_end < mt_min_slots[b_type]) && | |
| 3594 | + (!mte_is_root(wr_mas->mas->node)) && | |
| 3595 | + (mas_mt_height(wr_mas->mas) > 1)) | |
| 3596 | + return mas_rebalance(wr_mas->mas, b_node); | |
| 3597 | + | |
| 3598 | + if (b_end >= mt_slots[b_type]) | |
| 3599 | + return mas_split(wr_mas->mas, b_node); | |
| 3600 | + | |
| 3601 | + if (mas_reuse_node(wr_mas, b_node, end)) | |
| 3602 | + goto reuse_node; | |
| 3603 | + | |
| 3604 | + mas_node_count(wr_mas->mas, 1); | |
| 3605 | + if (mas_is_err(wr_mas->mas)) | |
| 3606 | + return 0; | |
| 3607 | + | |
| 3608 | + node = mas_pop_node(wr_mas->mas); | |
| 3609 | + node->parent = mas_mn(wr_mas->mas)->parent; | |
| 3610 | + wr_mas->mas->node = mt_mk_node(node, b_type); | |
| 3611 | + mab_mas_cp(b_node, 0, b_end, wr_mas->mas, true); | |
| 3612 | + | |
| 3613 | + mas_replace(wr_mas->mas, false); | |
| 3614 | +reuse_node: | |
| 3615 | + mas_update_gap(wr_mas->mas); | |
| 3616 | + return 1; | |
| 3617 | +} | |
| 3618 | + | |
| 3619 | +/* | |
| 3620 | + * mas_root_expand() - Expand a root to a node | |
| 3621 | + * @mas: The maple state | |
| 3622 | + * @entry: The entry to store into the tree | |
| 3623 | + */ | |
| 3624 | +static inline int mas_root_expand(struct ma_state *mas, void *entry) | |
| 3625 | +{ | |
| 3626 | + void *contents = mas_root_locked(mas); | |
| 3627 | + enum maple_type type = maple_leaf_64; | |
| 3628 | + struct maple_node *node; | |
| 3629 | + void __rcu **slots; | |
| 3630 | + unsigned long *pivots; | |
| 3631 | + int slot = 0; | |
| 3632 | + | |
| 3633 | + mas_node_count(mas, 1); | |
| 3634 | + if (unlikely(mas_is_err(mas))) | |
| 3635 | + return 0; | |
| 3636 | + | |
| 3637 | + node = mas_pop_node(mas); | |
| 3638 | + pivots = ma_pivots(node, type); | |
| 3639 | + slots = ma_slots(node, type); | |
| 3640 | + node->parent = ma_parent_ptr( | |
| 3641 | + ((unsigned long)mas->tree | MA_ROOT_PARENT)); | |
| 3642 | + mas->node = mt_mk_node(node, type); | |
| 3643 | + | |
| 3644 | + if (mas->index) { | |
| 3645 | + if (contents) { | |
| 3646 | + rcu_assign_pointer(slots[slot], contents); | |
| 3647 | + if (likely(mas->index > 1)) | |
| 3648 | + slot++; | |
| 3649 | + } | |
| 3650 | + pivots[slot++] = mas->index - 1; | |
| 3651 | + } | |
| 3652 | + | |
| 3653 | + rcu_assign_pointer(slots[slot], entry); | |
| 3654 | + mas->offset = slot; | |
| 3655 | + pivots[slot] = mas->last; | |
| 3656 | + if (mas->last != ULONG_MAX) | |
| 3657 | + slot++; | |
| 3658 | + mas->depth = 1; | |
| 3659 | + mas_set_height(mas); | |
| 3660 | + | |
| 3661 | + /* swap the new root into the tree */ | |
| 3662 | + rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node)); | |
| 3663 | + ma_set_meta(node, maple_leaf_64, 0, slot); | |
| 3664 | + return slot; | |
| 3665 | +} | |
| 3666 | + | |
| 3667 | +static inline void mas_store_root(struct ma_state *mas, void *entry) | |
| 3668 | +{ | |
| 3669 | + if (likely((mas->last != 0) || (mas->index != 0))) | |
| 3670 | + mas_root_expand(mas, entry); | |
| 3671 | + else if (((unsigned long) (entry) & 3) == 2) | |
| 3672 | + mas_root_expand(mas, entry); | |
| 3673 | + else { | |
| 3674 | + rcu_assign_pointer(mas->tree->ma_root, entry); | |
| 3675 | + mas->node = MAS_START; | |
| 3676 | + } | |
| 3677 | +} | |
| 3678 | + | |
| 3679 | +/* | |
| 3680 | + * mas_is_span_wr() - Check if the write needs to be treated as a write that | |
| 3681 | + * spans the node. | |
| 3682 | + * @mas: The maple state | |
| 3683 | + * @piv: The pivot value being written | |
| 3684 | + * @type: The maple node type | |
| 3685 | + * @entry: The data to write | |
| 3686 | + * | |
| 3687 | + * Spanning writes are writes that start in one node and end in another OR if | |
| 3688 | + * the write of a %NULL will cause the node to end with a %NULL. | |
| 3689 | + * | |
| 3690 | + * Return: True if this is a spanning write, false otherwise. | |
| 3691 | + */ | |
| 3692 | +static bool mas_is_span_wr(struct ma_wr_state *wr_mas) | |
| 3693 | +{ | |
| 3694 | + unsigned long max; | |
| 3695 | + unsigned long last = wr_mas->mas->last; | |
| 3696 | + unsigned long piv = wr_mas->r_max; | |
| 3697 | + enum maple_type type = wr_mas->type; | |
| 3698 | + void *entry = wr_mas->entry; | |
| 3699 | + | |
| 3700 | + /* Contained in this pivot */ | |
| 3701 | + if (piv > last) | |
| 3702 | + return false; | |
| 3703 | + | |
| 3704 | + max = wr_mas->mas->max; | |
| 3705 | + if (unlikely(ma_is_leaf(type))) { | |
| 3706 | + /* Fits in the node, but may span slots. */ | |
| 3707 | + if (last < max) | |
| 3708 | + return false; | |
| 3709 | + | |
| 3710 | + /* Writes to the end of the node but not null. */ | |
| 3711 | + if ((last == max) && entry) | |
| 3712 | + return false; | |
| 3713 | + | |
| 3714 | + /* | |
| 3715 | + * Writing ULONG_MAX is not a spanning write regardless of the | |
| 3716 | + * value being written as long as the range fits in the node. | |
| 3717 | + */ | |
| 3718 | + if ((last == ULONG_MAX) && (last == max)) | |
| 3719 | + return false; | |
| 3720 | + } else if (piv == last) { | |
| 3721 | + if (entry) | |
| 3722 | + return false; | |
| 3723 | + | |
| 3724 | + /* Detect spanning store wr walk */ | |
| 3725 | + if (last == ULONG_MAX) | |
| 3726 | + return false; | |
| 3727 | + } | |
| 3728 | + | |
| 3729 | + trace_ma_write(__func__, wr_mas->mas, piv, entry); | |
| 3730 | + | |
| 3731 | + return true; | |
| 3732 | +} | |
| 3733 | + | |
| 3734 | +static inline void mas_wr_walk_descend(struct ma_wr_state *wr_mas) | |
| 3735 | +{ | |
| 3736 | + wr_mas->mas->depth++; | |
| 3737 | + wr_mas->type = mte_node_type(wr_mas->mas->node); | |
| 3738 | + mas_wr_node_walk(wr_mas); | |
| 3739 | + wr_mas->slots = ma_slots(wr_mas->node, wr_mas->type); | |
| 3740 | +} | |
| 3741 | + | |
| 3742 | +static inline void mas_wr_walk_traverse(struct ma_wr_state *wr_mas) | |
| 3743 | +{ | |
| 3744 | + wr_mas->mas->max = wr_mas->r_max; | |
| 3745 | + wr_mas->mas->min = wr_mas->r_min; | |
| 3746 | + wr_mas->mas->node = wr_mas->content; | |
| 3747 | + wr_mas->mas->offset = 0; | |
| 3748 | +} | |
| 3749 | +/* | |
| 3750 | + * mas_wr_walk() - Walk the tree for a write. | |
| 3751 | + * @wr_mas: The maple write state | |
| 3752 | + * | |
| 3753 | + * Uses mas_slot_locked() and does not need to worry about dead nodes. | |
| 3754 | + * | |
| 3755 | + * Return: True if it's contained in a node, false on spanning write. | |
| 3756 | + */ | |
| 3757 | +static bool mas_wr_walk(struct ma_wr_state *wr_mas) | |
| 3758 | +{ | |
| 3759 | + struct ma_state *mas = wr_mas->mas; | |
| 3760 | + | |
| 3761 | + while (true) { | |
| 3762 | + mas_wr_walk_descend(wr_mas); | |
| 3763 | + if (unlikely(mas_is_span_wr(wr_mas))) | |
| 3764 | + return false; | |
| 3765 | + | |
| 3766 | + wr_mas->content = mas_slot_locked(mas, wr_mas->slots, | |
| 3767 | + mas->offset); | |
| 3768 | + if (ma_is_leaf(wr_mas->type)) | |
| 3769 | + return true; | |
| 3770 | + | |
| 3771 | + mas_wr_walk_traverse(wr_mas); | |
| 3772 | + } | |
| 3773 | + | |
| 3774 | + return true; | |
| 3775 | +} | |
| 3776 | + | |
| 3777 | +static bool mas_wr_walk_index(struct ma_wr_state *wr_mas) | |
| 3778 | +{ | |
| 3779 | + struct ma_state *mas = wr_mas->mas; | |
| 3780 | + | |
| 3781 | + while (true) { | |
| 3782 | + mas_wr_walk_descend(wr_mas); | |
| 3783 | + wr_mas->content = mas_slot_locked(mas, wr_mas->slots, | |
| 3784 | + mas->offset); | |
| 3785 | + if (ma_is_leaf(wr_mas->type)) | |
| 3786 | + return true; | |
| 3787 | + mas_wr_walk_traverse(wr_mas); | |
| 3788 | + | |
| 3789 | + } | |
| 3790 | + return true; | |
| 3791 | +} | |
| 3792 | +/* | |
| 3793 | + * mas_extend_spanning_null() - Extend a store of a %NULL to include surrounding %NULLs. | |
| 3794 | + * @l_wr_mas: The left maple write state | |
| 3795 | + * @r_wr_mas: The right maple write state | |
| 3796 | + */ | |
| 3797 | +static inline void mas_extend_spanning_null(struct ma_wr_state *l_wr_mas, | |
| 3798 | + struct ma_wr_state *r_wr_mas) | |
| 3799 | +{ | |
| 3800 | + struct ma_state *r_mas = r_wr_mas->mas; | |
| 3801 | + struct ma_state *l_mas = l_wr_mas->mas; | |
| 3802 | + unsigned char l_slot; | |
| 3803 | + | |
| 3804 | + l_slot = l_mas->offset; | |
| 3805 | + if (!l_wr_mas->content) | |
| 3806 | + l_mas->index = l_wr_mas->r_min; | |
| 3807 | + | |
| 3808 | + if ((l_mas->index == l_wr_mas->r_min) && | |
| 3809 | + (l_slot && | |
| 3810 | + !mas_slot_locked(l_mas, l_wr_mas->slots, l_slot - 1))) { | |
| 3811 | + if (l_slot > 1) | |
| 3812 | + l_mas->index = l_wr_mas->pivots[l_slot - 2] + 1; | |
| 3813 | + else | |
| 3814 | + l_mas->index = l_mas->min; | |
| 3815 | + | |
| 3816 | + l_mas->offset = l_slot - 1; | |
| 3817 | + } | |
| 3818 | + | |
| 3819 | + if (!r_wr_mas->content) { | |
| 3820 | + if (r_mas->last < r_wr_mas->r_max) | |
| 3821 | + r_mas->last = r_wr_mas->r_max; | |
| 3822 | + r_mas->offset++; | |
| 3823 | + } else if ((r_mas->last == r_wr_mas->r_max) && | |
| 3824 | + (r_mas->last < r_mas->max) && | |
| 3825 | + !mas_slot_locked(r_mas, r_wr_mas->slots, r_mas->offset + 1)) { | |
| 3826 | + r_mas->last = mas_safe_pivot(r_mas, r_wr_mas->pivots, | |
| 3827 | + r_wr_mas->type, r_mas->offset + 1); | |
| 3828 | + r_mas->offset++; | |
| 3829 | + } | |
| 3830 | +} | |
| 3831 | + | |
| 3832 | +static inline void *mas_state_walk(struct ma_state *mas) | |
| 3833 | +{ | |
| 3834 | + void *entry; | |
| 3835 | + | |
| 3836 | + entry = mas_start(mas); | |
| 3837 | + if (mas_is_none(mas)) | |
| 3838 | + return NULL; | |
| 3839 | + | |
| 3840 | + if (mas_is_ptr(mas)) | |
| 3841 | + return entry; | |
| 3842 | + | |
| 3843 | + return mtree_range_walk(mas); | |
| 3844 | +} | |
| 3845 | + | |
| 3846 | +/* | |
| 3847 | + * mtree_lookup_walk() - Internal quick lookup that does not keep maple state up | |
| 3848 | + * to date. | |
| 3849 | + * | |
| 3850 | + * @mas: The maple state. | |
| 3851 | + * | |
| 3852 | + * Note: Leaves mas in undesirable state. | |
| 3853 | + * Return: The entry for @mas->index or %NULL on dead node. | |
| 3854 | + */ | |
| 3855 | +static inline void *mtree_lookup_walk(struct ma_state *mas) | |
| 3856 | +{ | |
| 3857 | + unsigned long *pivots; | |
| 3858 | + unsigned char offset; | |
| 3859 | + struct maple_node *node; | |
| 3860 | + struct maple_enode *next; | |
| 3861 | + enum maple_type type; | |
| 3862 | + void __rcu **slots; | |
| 3863 | + unsigned char end; | |
| 3864 | + unsigned long max; | |
| 3865 | + | |
| 3866 | + next = mas->node; | |
| 3867 | + max = ULONG_MAX; | |
| 3868 | + do { | |
| 3869 | + offset = 0; | |
| 3870 | + node = mte_to_node(next); | |
| 3871 | + type = mte_node_type(next); | |
| 3872 | + pivots = ma_pivots(node, type); | |
| 3873 | + end = ma_data_end(node, type, pivots, max); | |
| 3874 | + if (unlikely(ma_dead_node(node))) | |
| 3875 | + goto dead_node; | |
| 3876 | + | |
| 3877 | + if (pivots[offset] >= mas->index) | |
| 3878 | + goto next; | |
| 3879 | + | |
| 3880 | + do { | |
| 3881 | + offset++; | |
| 3882 | + } while ((offset < end) && (pivots[offset] < mas->index)); | |
| 3883 | + | |
| 3884 | + if (likely(offset > end)) | |
| 3885 | + max = pivots[offset]; | |
| 3886 | + | |
| 3887 | +next: | |
| 3888 | + slots = ma_slots(node, type); | |
| 3889 | + next = mt_slot(mas->tree, slots, offset); | |
| 3890 | + if (unlikely(ma_dead_node(node))) | |
| 3891 | + goto dead_node; | |
| 3892 | + } while (!ma_is_leaf(type)); | |
| 3893 | + | |
| 3894 | + return (void *) next; | |
| 3895 | + | |
| 3896 | +dead_node: | |
| 3897 | + mas_reset(mas); | |
| 3898 | + return NULL; | |
| 3899 | +} | |
| 3900 | + | |
| 3901 | +/* | |
| 3902 | + * mas_new_root() - Create a new root node that only contains the entry passed | |
| 3903 | + * in. | |
| 3904 | + * @mas: The maple state | |
| 3905 | + * @entry: The entry to store. | |
| 3906 | + * | |
| 3907 | + * Only valid when the index == 0 and the last == ULONG_MAX | |
| 3908 | + * | |
| 3909 | + * Return 0 on error, 1 on success. | |
| 3910 | + */ | |
| 3911 | +static inline int mas_new_root(struct ma_state *mas, void *entry) | |
| 3912 | +{ | |
| 3913 | + struct maple_enode *root = mas_root_locked(mas); | |
| 3914 | + enum maple_type type = maple_leaf_64; | |
| 3915 | + struct maple_node *node; | |
| 3916 | + void __rcu **slots; | |
| 3917 | + unsigned long *pivots; | |
| 3918 | + | |
| 3919 | + if (!entry && !mas->index && mas->last == ULONG_MAX) { | |
| 3920 | + mas->depth = 0; | |
| 3921 | + mas_set_height(mas); | |
| 3922 | + rcu_assign_pointer(mas->tree->ma_root, entry); | |
| 3923 | + mas->node = MAS_START; | |
| 3924 | + goto done; | |
| 3925 | + } | |
| 3926 | + | |
| 3927 | + mas_node_count(mas, 1); | |
| 3928 | + if (mas_is_err(mas)) | |
| 3929 | + return 0; | |
| 3930 | + | |
| 3931 | + node = mas_pop_node(mas); | |
| 3932 | + pivots = ma_pivots(node, type); | |
| 3933 | + slots = ma_slots(node, type); | |
| 3934 | + node->parent = ma_parent_ptr( | |
| 3935 | + ((unsigned long)mas->tree | MA_ROOT_PARENT)); | |
| 3936 | + mas->node = mt_mk_node(node, type); | |
| 3937 | + rcu_assign_pointer(slots[0], entry); | |
| 3938 | + pivots[0] = mas->last; | |
| 3939 | + mas->depth = 1; | |
| 3940 | + mas_set_height(mas); | |
| 3941 | + rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node)); | |
| 3942 | + | |
| 3943 | +done: | |
| 3944 | + if (xa_is_node(root)) | |
| 3945 | + mte_destroy_walk(root, mas->tree); | |
| 3946 | + | |
| 3947 | + return 1; | |
| 3948 | +} | |
| 3949 | +/* | |
| 3950 | + * mas_wr_spanning_store() - Create a subtree with the store operation completed | |
| 3951 | + * and new nodes where necessary, then place the sub-tree in the actual tree. | |
| 3952 | + * Note that mas is expected to point to the node which caused the store to | |
| 3953 | + * span. | |
| 3954 | + * @wr_mas: The maple write state | |
| 3955 | + * | |
| 3956 | + * Return: 0 on error, positive on success. | |
| 3957 | + */ | |
| 3958 | +static inline int mas_wr_spanning_store(struct ma_wr_state *wr_mas) | |
| 3959 | +{ | |
| 3960 | + struct maple_subtree_state mast; | |
| 3961 | + struct maple_big_node b_node; | |
| 3962 | + struct ma_state *mas; | |
| 3963 | + unsigned char height; | |
| 3964 | + | |
| 3965 | + /* Left and Right side of spanning store */ | |
| 3966 | + MA_STATE(l_mas, NULL, 0, 0); | |
| 3967 | + MA_STATE(r_mas, NULL, 0, 0); | |
| 3968 | + | |
| 3969 | + MA_WR_STATE(r_wr_mas, &r_mas, wr_mas->entry); | |
| 3970 | + MA_WR_STATE(l_wr_mas, &l_mas, wr_mas->entry); | |
| 3971 | + | |
| 3972 | + /* | |
| 3973 | + * A store operation that spans multiple nodes is called a spanning | |
| 3974 | + * store and is handled early in the store call stack by the function | |
| 3975 | + * mas_is_span_wr(). When a spanning store is identified, the maple | |
| 3976 | + * state is duplicated. The first maple state walks the left tree path | |
| 3977 | + * to ``index``, the duplicate walks the right tree path to ``last``. | |
| 3978 | + * The data in the two nodes are combined into a single node, two nodes, | |
| 3979 | + * or possibly three nodes (see the 3-way split above). A ``NULL`` | |
| 3980 | + * written to the last entry of a node is considered a spanning store as | |
| 3981 | + * a rebalance is required for the operation to complete and an overflow | |
| 3982 | + * of data may happen. | |
| 3983 | + */ | |
| 3984 | + mas = wr_mas->mas; | |
| 3985 | + trace_ma_op(__func__, mas); | |
| 3986 | + | |
| 3987 | + if (unlikely(!mas->index && mas->last == ULONG_MAX)) | |
| 3988 | + return mas_new_root(mas, wr_mas->entry); | |
| 3989 | + /* | |
| 3990 | + * Node rebalancing may occur due to this store, so there may be three new | |
| 3991 | + * entries per level plus a new root. | |
| 3992 | + */ | |
| 3993 | + height = mas_mt_height(mas); | |
| 3994 | + mas_node_count(mas, 1 + height * 3); | |
| 3995 | + if (mas_is_err(mas)) | |
| 3996 | + return 0; | |
| 3997 | + | |
| 3998 | + /* | |
| 3999 | + * Set up right side. Need to get to the next offset after the spanning | |
| 4000 | + * store to ensure it's not NULL and to combine both the next node and | |
| 4001 | + * the node with the start together. | |
| 4002 | + */ | |
| 4003 | + r_mas = *mas; | |
| 4004 | + /* Avoid overflow, walk to next slot in the tree. */ | |
| 4005 | + if (r_mas.last + 1) | |
| 4006 | + r_mas.last++; | |
| 4007 | + | |
| 4008 | + r_mas.index = r_mas.last; | |
| 4009 | + mas_wr_walk_index(&r_wr_mas); | |
| 4010 | + r_mas.last = r_mas.index = mas->last; | |
| 4011 | + | |
| 4012 | + /* Set up left side. */ | |
| 4013 | + l_mas = *mas; | |
| 4014 | + mas_wr_walk_index(&l_wr_mas); | |
| 4015 | + | |
| 4016 | + if (!wr_mas->entry) { | |
| 4017 | + mas_extend_spanning_null(&l_wr_mas, &r_wr_mas); | |
| 4018 | + mas->offset = l_mas.offset; | |
| 4019 | + mas->index = l_mas.index; | |
| 4020 | + mas->last = l_mas.last = r_mas.last; | |
| 4021 | + } | |
| 4022 | + | |
| 4023 | + /* expanding NULLs may make this cover the entire range */ | |
| 4024 | + if (!l_mas.index && r_mas.last == ULONG_MAX) { | |
| 4025 | + mas_set_range(mas, 0, ULONG_MAX); | |
| 4026 | + return mas_new_root(mas, wr_mas->entry); | |
| 4027 | + } | |
| 4028 | + | |
| 4029 | + memset(&b_node, 0, sizeof(struct maple_big_node)); | |
| 4030 | + /* Copy l_mas and store the value in b_node. */ | |
| 4031 | + mas_store_b_node(&l_wr_mas, &b_node, l_wr_mas.node_end); | |
| 4032 | + /* Copy r_mas into b_node. */ | |
| 4033 | + if (r_mas.offset <= r_wr_mas.node_end) | |
| 4034 | + mas_mab_cp(&r_mas, r_mas.offset, r_wr_mas.node_end, | |
| 4035 | + &b_node, b_node.b_end + 1); | |
| 4036 | + else | |
| 4037 | + b_node.b_end++; | |
| 4038 | + | |
| 4039 | + /* Stop spanning searches by searching for just index. */ | |
| 4040 | + l_mas.index = l_mas.last = mas->index; | |
| 4041 | + | |
| 4042 | + mast.bn = &b_node; | |
| 4043 | + mast.orig_l = &l_mas; | |
| 4044 | + mast.orig_r = &r_mas; | |
| 4045 | + /* Combine l_mas and r_mas and split them up evenly again. */ | |
| 4046 | + return mas_spanning_rebalance(mas, &mast, height + 1); | |
| 4047 | +} | |
| 4048 | + | |
| 4049 | +/* | |
| 4050 | + * mas_wr_node_store() - Attempt to store the value in a node | |
| 4051 | + * @wr_mas: The maple write state | |
| 4052 | + * | |
| 4053 | + * Attempts to reuse the node, but may allocate. | |
| 4054 | + * | |
| 4055 | + * Return: True if stored, false otherwise | |
| 4056 | + */ | |
| 4057 | +static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) | |
| 4058 | +{ | |
| 4059 | + struct ma_state *mas = wr_mas->mas; | |
| 4060 | + void __rcu **dst_slots; | |
| 4061 | + unsigned long *dst_pivots; | |
| 4062 | + unsigned char dst_offset; | |
| 4063 | + unsigned char new_end = wr_mas->node_end; | |
| 4064 | + unsigned char offset; | |
| 4065 | + unsigned char node_slots = mt_slots[wr_mas->type]; | |
| 4066 | + struct maple_node reuse, *newnode; | |
| 4067 | + unsigned char copy_size, max_piv = mt_pivots[wr_mas->type]; | |
| 4068 | + bool in_rcu = mt_in_rcu(mas->tree); | |
| 4069 | + | |
| 4070 | + offset = mas->offset; | |
| 4071 | + if (mas->last == wr_mas->r_max) { | |
| 4072 | + /* runs right to the end of the node */ | |
| 4073 | + if (mas->last == mas->max) | |
| 4074 | + new_end = offset; | |
| 4075 | + /* don't copy this offset */ | |
| 4076 | + wr_mas->offset_end++; | |
| 4077 | + } else if (mas->last < wr_mas->r_max) { | |
| 4078 | + /* new range ends in this range */ | |
| 4079 | + if (unlikely(wr_mas->r_max == ULONG_MAX)) | |
| 4080 | + mas_bulk_rebalance(mas, wr_mas->node_end, wr_mas->type); | |
| 4081 | + | |
| 4082 | + new_end++; | |
| 4083 | + } else { | |
| 4084 | + if (wr_mas->end_piv == mas->last) | |
| 4085 | + wr_mas->offset_end++; | |
| 4086 | + | |
| 4087 | + new_end -= wr_mas->offset_end - offset - 1; | |
| 4088 | + } | |
| 4089 | + | |
| 4090 | + /* new range starts within a range */ | |
| 4091 | + if (wr_mas->r_min < mas->index) | |
| 4092 | + new_end++; | |
| 4093 | + | |
| 4094 | + /* Not enough room */ | |
| 4095 | + if (new_end >= node_slots) | |
| 4096 | + return false; | |
| 4097 | + | |
| 4098 | + /* Not enough data. */ | |
| 4099 | + if (!mte_is_root(mas->node) && (new_end <= mt_min_slots[wr_mas->type]) && | |
| 4100 | + !(mas->mas_flags & MA_STATE_BULK)) | |
| 4101 | + return false; | |
| 4102 | + | |
| 4103 | + /* set up node. */ | |
| 4104 | + if (in_rcu) { | |
| 4105 | + mas_node_count(mas, 1); | |
| 4106 | + if (mas_is_err(mas)) | |
| 4107 | + return false; | |
| 4108 | + | |
| 4109 | + newnode = mas_pop_node(mas); | |
| 4110 | + } else { | |
| 4111 | + memset(&reuse, 0, sizeof(struct maple_node)); | |
| 4112 | + newnode = &reuse; | |
| 4113 | + } | |
| 4114 | + | |
| 4115 | + newnode->parent = mas_mn(mas)->parent; | |
| 4116 | + dst_pivots = ma_pivots(newnode, wr_mas->type); | |
| 4117 | + dst_slots = ma_slots(newnode, wr_mas->type); | |
| 4118 | + /* Copy from start to insert point */ | |
| 4119 | + memcpy(dst_pivots, wr_mas->pivots, sizeof(unsigned long) * (offset + 1)); | |
| 4120 | + memcpy(dst_slots, wr_mas->slots, sizeof(void *) * (offset + 1)); | |
| 4121 | + dst_offset = offset; | |
| 4122 | + | |
| 4123 | + /* Handle insert of new range starting after old range */ | |
| 4124 | + if (wr_mas->r_min < mas->index) { | |
| 4125 | + mas->offset++; | |
| 4126 | + rcu_assign_pointer(dst_slots[dst_offset], wr_mas->content); | |
| 4127 | + dst_pivots[dst_offset++] = mas->index - 1; | |
| 4128 | + } | |
| 4129 | + | |
| 4130 | + /* Store the new entry and range end. */ | |
| 4131 | + if (dst_offset < max_piv) | |
| 4132 | + dst_pivots[dst_offset] = mas->last; | |
| 4133 | + mas->offset = dst_offset; | |
| 4134 | + rcu_assign_pointer(dst_slots[dst_offset], wr_mas->entry); | |
| 4135 | + | |
| 4136 | + /* | |
| 4137 | + * this range wrote to the end of the node or it overwrote the rest of | |
| 4138 | + * the data | |
| 4139 | + */ | |
| 4140 | + if (wr_mas->offset_end > wr_mas->node_end || mas->last >= mas->max) { | |
| 4141 | + new_end = dst_offset; | |
| 4142 | + goto done; | |
| 4143 | + } | |
| 4144 | + | |
| 4145 | + dst_offset++; | |
| 4146 | + /* Copy to the end of node if necessary. */ | |
| 4147 | + copy_size = wr_mas->node_end - wr_mas->offset_end + 1; | |
| 4148 | + memcpy(dst_slots + dst_offset, wr_mas->slots + wr_mas->offset_end, | |
| 4149 | + sizeof(void *) * copy_size); | |
| 4150 | + if (dst_offset < max_piv) { | |
| 4151 | + if (copy_size > max_piv - dst_offset) | |
| 4152 | + copy_size = max_piv - dst_offset; | |
| 4153 | + | |
| 4154 | + memcpy(dst_pivots + dst_offset, | |
| 4155 | + wr_mas->pivots + wr_mas->offset_end, | |
| 4156 | + sizeof(unsigned long) * copy_size); | |
| 4157 | + } | |
| 4158 | + | |
| 4159 | + if ((wr_mas->node_end == node_slots - 1) && (new_end < node_slots - 1)) | |
| 4160 | + dst_pivots[new_end] = mas->max; | |
| 4161 | + | |
| 4162 | +done: | |
| 4163 | + mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); | |
| 4164 | + if (in_rcu) { | |
| 4165 | + mas->node = mt_mk_node(newnode, wr_mas->type); | |
| 4166 | + mas_replace(mas, false); | |
| 4167 | + } else { | |
| 4168 | + memcpy(wr_mas->node, newnode, sizeof(struct maple_node)); | |
| 4169 | + } | |
| 4170 | + trace_ma_write(__func__, mas, 0, wr_mas->entry); | |
| 4171 | + mas_update_gap(mas); | |
| 4172 | + return true; | |
| 4173 | +} | |
| 4174 | + | |
| 4175 | +/* | |
| 4176 | + * mas_wr_slot_store: Attempt to store a value in a slot. | |
| 4177 | + * @wr_mas: the maple write state | |
| 4178 | + * | |
| 4179 | + * Return: True if stored, false otherwise | |
| 4180 | + */ | |
| 4181 | +static inline bool mas_wr_slot_store(struct ma_wr_state *wr_mas) | |
| 4182 | +{ | |
| 4183 | + struct ma_state *mas = wr_mas->mas; | |
| 4184 | + unsigned long lmax; /* Logical max. */ | |
| 4185 | + unsigned char offset = mas->offset; | |
| 4186 | + | |
| 4187 | + if ((wr_mas->r_max > mas->last) && ((wr_mas->r_min != mas->index) || | |
| 4188 | + (offset != wr_mas->node_end))) | |
| 4189 | + return false; | |
| 4190 | + | |
| 4191 | + if (offset == wr_mas->node_end - 1) | |
| 4192 | + lmax = mas->max; | |
| 4193 | + else | |
| 4194 | + lmax = wr_mas->pivots[offset + 1]; | |
| 4195 | + | |
| 4196 | + /* going to overwrite too many slots. */ | |
| 4197 | + if (lmax < mas->last) | |
| 4198 | + return false; | |
| 4199 | + | |
| 4200 | + if (wr_mas->r_min == mas->index) { | |
| 4201 | + /* overwriting two or more ranges with one. */ | |
| 4202 | + if (lmax == mas->last) | |
| 4203 | + return false; | |
| 4204 | + | |
| 4205 | + /* Overwriting all of offset and a portion of offset + 1. */ | |
| 4206 | + rcu_assign_pointer(wr_mas->slots[offset], wr_mas->entry); | |
| 4207 | + wr_mas->pivots[offset] = mas->last; | |
| 4208 | + goto done; | |
| 4209 | + } | |
| 4210 | + | |
| 4211 | + /* Doesn't end on the next range end. */ | |
| 4212 | + if (lmax != mas->last) | |
| 4213 | + return false; | |
| 4214 | + | |
| 4215 | + /* Overwriting a portion of offset and all of offset + 1 */ | |
| 4216 | + if ((offset + 1 < mt_pivots[wr_mas->type]) && | |
| 4217 | + (wr_mas->entry || wr_mas->pivots[offset + 1])) | |
| 4218 | + wr_mas->pivots[offset + 1] = mas->last; | |
| 4219 | + | |
| 4220 | + rcu_assign_pointer(wr_mas->slots[offset + 1], wr_mas->entry); | |
| 4221 | + wr_mas->pivots[offset] = mas->index - 1; | |
| 4222 | + mas->offset++; /* Keep mas accurate. */ | |
| 4223 | + | |
| 4224 | +done: | |
| 4225 | + trace_ma_write(__func__, mas, 0, wr_mas->entry); | |
| 4226 | + mas_update_gap(mas); | |
| 4227 | + return true; | |
| 4228 | +} | |
| 4229 | + | |
| 4230 | +static inline void mas_wr_end_piv(struct ma_wr_state *wr_mas) | |
| 4231 | +{ | |
| 4232 | + while ((wr_mas->mas->last > wr_mas->end_piv) && | |
| 4233 | + (wr_mas->offset_end < wr_mas->node_end)) | |
| 4234 | + wr_mas->end_piv = wr_mas->pivots[++wr_mas->offset_end]; | |
| 4235 | + | |
| 4236 | + if (wr_mas->mas->last > wr_mas->end_piv) | |
| 4237 | + wr_mas->end_piv = wr_mas->mas->max; | |
| 4238 | +} | |
| 4239 | + | |
| 4240 | +static inline void mas_wr_extend_null(struct ma_wr_state *wr_mas) | |
| 4241 | +{ | |
| 4242 | + struct ma_state *mas = wr_mas->mas; | |
| 4243 | + | |
| 4244 | + if (mas->last < wr_mas->end_piv && !wr_mas->slots[wr_mas->offset_end]) | |
| 4245 | + mas->last = wr_mas->end_piv; | |
| 4246 | + | |
| 4247 | + /* Check next slot(s) if we are overwriting the end */ | |
| 4248 | + if ((mas->last == wr_mas->end_piv) && | |
| 4249 | + (wr_mas->node_end != wr_mas->offset_end) && | |
| 4250 | + !wr_mas->slots[wr_mas->offset_end + 1]) { | |
| 4251 | + wr_mas->offset_end++; | |
| 4252 | + if (wr_mas->offset_end == wr_mas->node_end) | |
| 4253 | + mas->last = mas->max; | |
| 4254 | + else | |
| 4255 | + mas->last = wr_mas->pivots[wr_mas->offset_end]; | |
| 4256 | + wr_mas->end_piv = mas->last; | |
| 4257 | + } | |
| 4258 | + | |
| 4259 | + if (!wr_mas->content) { | |
| 4260 | + /* If this one is null, the next and prev are not */ | |
| 4261 | + mas->index = wr_mas->r_min; | |
| 4262 | + } else { | |
| 4263 | + /* Check prev slot if we are overwriting the start */ | |
| 4264 | + if (mas->index == wr_mas->r_min && mas->offset && | |
| 4265 | + !wr_mas->slots[mas->offset - 1]) { | |
| 4266 | + mas->offset--; | |
| 4267 | + wr_mas->r_min = mas->index = | |
| 4268 | + mas_safe_min(mas, wr_mas->pivots, mas->offset); | |
| 4269 | + wr_mas->r_max = wr_mas->pivots[mas->offset]; | |
| 4270 | + } | |
| 4271 | + } | |
| 4272 | +} | |
| 4273 | + | |
| 4274 | +static inline bool mas_wr_append(struct ma_wr_state *wr_mas) | |
| 4275 | +{ | |
| 4276 | + unsigned char end = wr_mas->node_end; | |
| 4277 | + unsigned char new_end = end + 1; | |
| 4278 | + struct ma_state *mas = wr_mas->mas; | |
| 4279 | + unsigned char node_pivots = mt_pivots[wr_mas->type]; | |
| 4280 | + | |
| 4281 | + if ((mas->index != wr_mas->r_min) && (mas->last == wr_mas->r_max)) { | |
| 4282 | + if (new_end < node_pivots) | |
| 4283 | + wr_mas->pivots[new_end] = wr_mas->pivots[end]; | |
| 4284 | + | |
| 4285 | + if (new_end < node_pivots) | |
| 4286 | + ma_set_meta(wr_mas->node, maple_leaf_64, 0, new_end); | |
| 4287 | + | |
| 4288 | + rcu_assign_pointer(wr_mas->slots[new_end], wr_mas->entry); | |
| 4289 | + mas->offset = new_end; | |
| 4290 | + wr_mas->pivots[end] = mas->index - 1; | |
| 4291 | + | |
| 4292 | + return true; | |
| 4293 | + } | |
| 4294 | + | |
| 4295 | + if ((mas->index == wr_mas->r_min) && (mas->last < wr_mas->r_max)) { | |
| 4296 | + if (new_end < node_pivots) | |
| 4297 | + wr_mas->pivots[new_end] = wr_mas->pivots[end]; | |
| 4298 | + | |
| 4299 | + rcu_assign_pointer(wr_mas->slots[new_end], wr_mas->content); | |
| 4300 | + if (new_end < node_pivots) | |
| 4301 | + ma_set_meta(wr_mas->node, maple_leaf_64, 0, new_end); | |
| 4302 | + | |
| 4303 | + wr_mas->pivots[end] = mas->last; | |
| 4304 | + rcu_assign_pointer(wr_mas->slots[end], wr_mas->entry); | |
| 4305 | + return true; | |
| 4306 | + } | |
| 4307 | + | |
| 4308 | + return false; | |
| 4309 | +} | |
| 4310 | + | |
| 4311 | +/* | |
| 4312 | + * mas_wr_bnode() - Slow path for a modification. | |
| 4313 | + * @wr_mas: The write maple state | |
| 4314 | + * | |
| 4315 | + * This is where split, rebalance end up. | |
| 4316 | + */ | |
| 4317 | +static void mas_wr_bnode(struct ma_wr_state *wr_mas) | |
| 4318 | +{ | |
| 4319 | + struct maple_big_node b_node; | |
| 4320 | + | |
| 4321 | + trace_ma_write(__func__, wr_mas->mas, 0, wr_mas->entry); | |
| 4322 | + memset(&b_node, 0, sizeof(struct maple_big_node)); | |
| 4323 | + mas_store_b_node(wr_mas, &b_node, wr_mas->offset_end); | |
| 4324 | + mas_commit_b_node(wr_mas, &b_node, wr_mas->node_end); | |
| 4325 | +} | |
| 4326 | + | |
| 4327 | +static inline void mas_wr_modify(struct ma_wr_state *wr_mas) | |
| 4328 | +{ | |
| 4329 | + unsigned char node_slots; | |
| 4330 | + unsigned char node_size; | |
| 4331 | + struct ma_state *mas = wr_mas->mas; | |
| 4332 | + | |
| 4333 | + /* Direct replacement */ | |
| 4334 | + if (wr_mas->r_min == mas->index && wr_mas->r_max == mas->last) { | |
| 4335 | + rcu_assign_pointer(wr_mas->slots[mas->offset], wr_mas->entry); | |
| 4336 | + if (!!wr_mas->entry ^ !!wr_mas->content) | |
| 4337 | + mas_update_gap(mas); | |
| 4338 | + return; | |
| 4339 | + } | |
| 4340 | + | |
| 4341 | + /* Attempt to append */ | |
| 4342 | + node_slots = mt_slots[wr_mas->type]; | |
| 4343 | + node_size = wr_mas->node_end - wr_mas->offset_end + mas->offset + 2; | |
| 4344 | + if (mas->max == ULONG_MAX) | |
| 4345 | + node_size++; | |
| 4346 | + | |
| 4347 | + /* slot and node store will not fit, go to the slow path */ | |
| 4348 | + if (unlikely(node_size >= node_slots)) | |
| 4349 | + goto slow_path; | |
| 4350 | + | |
| 4351 | + if (wr_mas->entry && (wr_mas->node_end < node_slots - 1) && | |
| 4352 | + (mas->offset == wr_mas->node_end) && mas_wr_append(wr_mas)) { | |
| 4353 | + if (!wr_mas->content || !wr_mas->entry) | |
| 4354 | + mas_update_gap(mas); | |
| 4355 | + return; | |
| 4356 | + } | |
| 4357 | + | |
| 4358 | + if ((wr_mas->offset_end - mas->offset <= 1) && mas_wr_slot_store(wr_mas)) | |
| 4359 | + return; | |
| 4360 | + else if (mas_wr_node_store(wr_mas)) | |
| 4361 | + return; | |
| 4362 | + | |
| 4363 | + if (mas_is_err(mas)) | |
| 4364 | + return; | |
| 4365 | + | |
| 4366 | +slow_path: | |
| 4367 | + mas_wr_bnode(wr_mas); | |
| 4368 | +} | |
| 4369 | + | |
| 4370 | +/* | |
| 4371 | + * mas_wr_store_entry() - Internal call to store a value | |
| 4372 | + * @mas: The maple state | |
| 4373 | + * @entry: The entry to store. | |
| 4374 | + * | |
| 4375 | + * Return: The contents that was stored at the index. | |
| 4376 | + */ | |
| 4377 | +static inline void *mas_wr_store_entry(struct ma_wr_state *wr_mas) | |
| 4378 | +{ | |
| 4379 | + struct ma_state *mas = wr_mas->mas; | |
| 4380 | + | |
| 4381 | + wr_mas->content = mas_start(mas); | |
| 4382 | + if (mas_is_none(mas) || mas_is_ptr(mas)) { | |
| 4383 | + mas_store_root(mas, wr_mas->entry); | |
| 4384 | + return wr_mas->content; | |
| 4385 | + } | |
| 4386 | + | |
| 4387 | + if (unlikely(!mas_wr_walk(wr_mas))) { | |
| 4388 | + mas_wr_spanning_store(wr_mas); | |
| 4389 | + return wr_mas->content; | |
| 4390 | + } | |
| 4391 | + | |
| 4392 | + /* At this point, we are at the leaf node that needs to be altered. */ | |
| 4393 | + wr_mas->end_piv = wr_mas->r_max; | |
| 4394 | + mas_wr_end_piv(wr_mas); | |
| 4395 | + | |
| 4396 | + if (!wr_mas->entry) | |
| 4397 | + mas_wr_extend_null(wr_mas); | |
| 4398 | + | |
| 4399 | + /* New root for a single pointer */ | |
| 4400 | + if (unlikely(!mas->index && mas->last == ULONG_MAX)) { | |
| 4401 | + mas_new_root(mas, wr_mas->entry); | |
| 4402 | + return wr_mas->content; | |
| 4403 | + } | |
| 4404 | + | |
| 4405 | + mas_wr_modify(wr_mas); | |
| 4406 | + return wr_mas->content; | |
| 4407 | +} | |
| 4408 | + | |
| 4409 | +/** | |
| 4410 | + * mas_insert() - Internal call to insert a value | |
| 4411 | + * @mas: The maple state | |
| 4412 | + * @entry: The entry to store | |
| 4413 | + * | |
| 4414 | + * Return: %NULL or the contents that already exists at the requested index | |
| 4415 | + * otherwise. The maple state needs to be checked for error conditions. | |
| 4416 | + */ | |
| 4417 | +static inline void *mas_insert(struct ma_state *mas, void *entry) | |
| 4418 | +{ | |
| 4419 | + MA_WR_STATE(wr_mas, mas, entry); | |
| 4420 | + | |
| 4421 | + /* | |
| 4422 | + * Inserting a new range inserts either 0, 1, or 2 pivots within the | |
| 4423 | + * tree. If the insert fits exactly into an existing gap with a value | |
| 4424 | + * of NULL, then the slot only needs to be written with the new value. | |
| 4425 | + * If the range being inserted is adjacent to another range, then only a | |
| 4426 | + * single pivot needs to be inserted (as well as writing the entry). If | |
| 4427 | + * the new range is within a gap but does not touch any other ranges, | |
| 4428 | + * then two pivots need to be inserted: the start - 1, and the end. As | |
| 4429 | + * usual, the entry must be written. Most operations require a new node | |
| 4430 | + * to be allocated and replace an existing node to ensure RCU safety, | |
| 4431 | + * when in RCU mode. The exception to requiring a newly allocated node | |
| 4432 | + * is when inserting at the end of a node (appending). When done | |
| 4433 | + * carefully, appending can reuse the node in place. | |
| 4434 | + */ | |
| 4435 | + wr_mas.content = mas_start(mas); | |
| 4436 | + if (wr_mas.content) | |
| 4437 | + goto exists; | |
| 4438 | + | |
| 4439 | + if (mas_is_none(mas) || mas_is_ptr(mas)) { | |
| 4440 | + mas_store_root(mas, entry); | |
| 4441 | + return NULL; | |
| 4442 | + } | |
| 4443 | + | |
| 4444 | + /* spanning writes always overwrite something */ | |
| 4445 | + if (!mas_wr_walk(&wr_mas)) | |
| 4446 | + goto exists; | |
| 4447 | + | |
| 4448 | + /* At this point, we are at the leaf node that needs to be altered. */ | |
| 4449 | + wr_mas.offset_end = mas->offset; | |
| 4450 | + wr_mas.end_piv = wr_mas.r_max; | |
| 4451 | + | |
| 4452 | + if (wr_mas.content || (mas->last > wr_mas.r_max)) | |
| 4453 | + goto exists; | |
| 4454 | + | |
| 4455 | + if (!entry) | |
| 4456 | + return NULL; | |
| 4457 | + | |
| 4458 | + mas_wr_modify(&wr_mas); | |
| 4459 | + return wr_mas.content; | |
| 4460 | + | |
| 4461 | +exists: | |
| 4462 | + mas_set_err(mas, -EEXIST); | |
| 4463 | + return wr_mas.content; | |
| 4464 | + | |
| 4465 | +} | |
| 4466 | + | |
| 4467 | +/* | |
| 4468 | + * mas_prev_node() - Find the prev non-null entry at the same level in the | |
| 4469 | + * tree. The prev value will be mas->node[mas->offset] or MAS_NONE. | |
| 4470 | + * @mas: The maple state | |
| 4471 | + * @min: The lower limit to search | |
| 4472 | + * | |
| 4473 | + * The prev node value will be mas->node[mas->offset] or MAS_NONE. | |
| 4474 | + * Return: 1 if the node is dead, 0 otherwise. | |
| 4475 | + */ | |
| 4476 | +static inline int mas_prev_node(struct ma_state *mas, unsigned long min) | |
| 4477 | +{ | |
| 4478 | + enum maple_type mt; | |
| 4479 | + int offset, level; | |
| 4480 | + void __rcu **slots; | |
| 4481 | + struct maple_node *node; | |
| 4482 | + struct maple_enode *enode; | |
| 4483 | + unsigned long *pivots; | |
| 4484 | + | |
| 4485 | + if (mas_is_none(mas)) | |
| 4486 | + return 0; | |
| 4487 | + | |
| 4488 | + level = 0; | |
| 4489 | + do { | |
| 4490 | + node = mas_mn(mas); | |
| 4491 | + if (ma_is_root(node)) | |
| 4492 | + goto no_entry; | |
| 4493 | + | |
| 4494 | + /* Walk up. */ | |
| 4495 | + if (unlikely(mas_ascend(mas))) | |
| 4496 | + return 1; | |
| 4497 | + offset = mas->offset; | |
| 4498 | + level++; | |
| 4499 | + } while (!offset); | |
| 4500 | + | |
| 4501 | + offset--; | |
| 4502 | + mt = mte_node_type(mas->node); | |
| 4503 | + node = mas_mn(mas); | |
| 4504 | + slots = ma_slots(node, mt); | |
| 4505 | + pivots = ma_pivots(node, mt); | |
| 4506 | + mas->max = pivots[offset]; | |
| 4507 | + if (offset) | |
| 4508 | + mas->min = pivots[offset - 1] + 1; | |
| 4509 | + if (unlikely(ma_dead_node(node))) | |
| 4510 | + return 1; | |
| 4511 | + | |
| 4512 | + if (mas->max < min) | |
| 4513 | + goto no_entry_min; | |
| 4514 | + | |
| 4515 | + while (level > 1) { | |
| 4516 | + level--; | |
| 4517 | + enode = mas_slot(mas, slots, offset); | |
| 4518 | + if (unlikely(ma_dead_node(node))) | |
| 4519 | + return 1; | |
| 4520 | + | |
| 4521 | + mas->node = enode; | |
| 4522 | + mt = mte_node_type(mas->node); | |
| 4523 | + node = mas_mn(mas); | |
| 4524 | + slots = ma_slots(node, mt); | |
| 4525 | + pivots = ma_pivots(node, mt); | |
| 4526 | + offset = ma_data_end(node, mt, pivots, mas->max); | |
| 4527 | + if (offset) | |
| 4528 | + mas->min = pivots[offset - 1] + 1; | |
| 4529 | + | |
| 4530 | + if (offset < mt_pivots[mt]) | |
| 4531 | + mas->max = pivots[offset]; | |
| 4532 | + | |
| 4533 | + if (mas->max < min) | |
| 4534 | + goto no_entry; | |
| 4535 | + } | |
| 4536 | + | |
| 4537 | + mas->node = mas_slot(mas, slots, offset); | |
| 4538 | + if (unlikely(ma_dead_node(node))) | |
| 4539 | + return 1; | |
| 4540 | + | |
| 4541 | + mas->offset = mas_data_end(mas); | |
| 4542 | + if (unlikely(mte_dead_node(mas->node))) | |
| 4543 | + return 1; | |
| 4544 | + | |
| 4545 | + return 0; | |
| 4546 | + | |
| 4547 | +no_entry_min: | |
| 4548 | + mas->offset = offset; | |
| 4549 | + if (offset) | |
| 4550 | + mas->min = pivots[offset - 1] + 1; | |
| 4551 | +no_entry: | |
| 4552 | + if (unlikely(ma_dead_node(node))) | |
| 4553 | + return 1; | |
| 4554 | + | |
| 4555 | + mas->node = MAS_NONE; | |
| 4556 | + return 0; | |
| 4557 | +} | |
| 4558 | + | |
| 4559 | +/* | |
| 4560 | + * mas_next_node() - Get the next node at the same level in the tree. | |
| 4561 | + * @mas: The maple state | |
| 4562 | + * @max: The maximum pivot value to check. | |
| 4563 | + * | |
| 4564 | + * The next value will be mas->node[mas->offset] or MAS_NONE. | |
| 4565 | + * Return: 1 on dead node, 0 otherwise. | |
| 4566 | + */ | |
| 4567 | +static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, | |
| 4568 | + unsigned long max) | |
| 4569 | +{ | |
| 4570 | + unsigned long min, pivot; | |
| 4571 | + unsigned long *pivots; | |
| 4572 | + struct maple_enode *enode; | |
| 4573 | + int level = 0; | |
| 4574 | + unsigned char offset; | |
| 4575 | + enum maple_type mt; | |
| 4576 | + void __rcu **slots; | |
| 4577 | + | |
| 4578 | + if (mas->max >= max) | |
| 4579 | + goto no_entry; | |
| 4580 | + | |
| 4581 | + level = 0; | |
| 4582 | + do { | |
| 4583 | + if (ma_is_root(node)) | |
| 4584 | + goto no_entry; | |
| 4585 | + | |
| 4586 | + min = mas->max + 1; | |
| 4587 | + if (min > max) | |
| 4588 | + goto no_entry; | |
| 4589 | + | |
| 4590 | + if (unlikely(mas_ascend(mas))) | |
| 4591 | + return 1; | |
| 4592 | + | |
| 4593 | + offset = mas->offset; | |
| 4594 | + level++; | |
| 4595 | + node = mas_mn(mas); | |
| 4596 | + mt = mte_node_type(mas->node); | |
| 4597 | + pivots = ma_pivots(node, mt); | |
| 4598 | + } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); | |
| 4599 | + | |
| 4600 | + slots = ma_slots(node, mt); | |
| 4601 | + pivot = mas_safe_pivot(mas, pivots, ++offset, mt); | |
| 4602 | + while (unlikely(level > 1)) { | |
| 4603 | + /* Descend, if necessary */ | |
| 4604 | + enode = mas_slot(mas, slots, offset); | |
| 4605 | + if (unlikely(ma_dead_node(node))) | |
| 4606 | + return 1; | |
| 4607 | + | |
| 4608 | + mas->node = enode; | |
| 4609 | + level--; | |
| 4610 | + node = mas_mn(mas); | |
| 4611 | + mt = mte_node_type(mas->node); | |
| 4612 | + slots = ma_slots(node, mt); | |
| 4613 | + pivots = ma_pivots(node, mt); | |
| 4614 | + offset = 0; | |
| 4615 | + pivot = pivots[0]; | |
| 4616 | + } | |
| 4617 | + | |
| 4618 | + enode = mas_slot(mas, slots, offset); | |
| 4619 | + if (unlikely(ma_dead_node(node))) | |
| 4620 | + return 1; | |
| 4621 | + | |
| 4622 | + mas->node = enode; | |
| 4623 | + mas->min = min; | |
| 4624 | + mas->max = pivot; | |
| 4625 | + return 0; | |
| 4626 | + | |
| 4627 | +no_entry: | |
| 4628 | + if (unlikely(ma_dead_node(node))) | |
| 4629 | + return 1; | |
| 4630 | + | |
| 4631 | + mas->node = MAS_NONE; | |
| 4632 | + return 0; | |
| 4633 | +} | |
| 4634 | + | |
| 4635 | +/* | |
| 4636 | + * mas_next_nentry() - Get the next node entry | |
| 4637 | + * @mas: The maple state | |
| 4638 | + * @max: The maximum value to check | |
| 4639 | + * @*range_start: Pointer to store the start of the range. | |
| 4640 | + * | |
| 4641 | + * Sets @mas->offset to the offset of the next node entry, @mas->last to the | |
| 4642 | + * pivot of the entry. | |
| 4643 | + * | |
| 4644 | + * Return: The next entry, %NULL otherwise | |
| 4645 | + */ | |
| 4646 | +static inline void *mas_next_nentry(struct ma_state *mas, | |
| 4647 | + struct maple_node *node, unsigned long max, enum maple_type type) | |
| 4648 | +{ | |
| 4649 | + unsigned char count; | |
| 4650 | + unsigned long pivot; | |
| 4651 | + unsigned long *pivots; | |
| 4652 | + void __rcu **slots; | |
| 4653 | + void *entry; | |
| 4654 | + | |
| 4655 | + if (mas->last == mas->max) { | |
| 4656 | + mas->index = mas->max; | |
| 4657 | + return NULL; | |
| 4658 | + } | |
| 4659 | + | |
| 4660 | + pivots = ma_pivots(node, type); | |
| 4661 | + slots = ma_slots(node, type); | |
| 4662 | + mas->index = mas_safe_min(mas, pivots, mas->offset); | |
| 4663 | + if (ma_dead_node(node)) | |
| 4664 | + return NULL; | |
| 4665 | + | |
| 4666 | + if (mas->index > max) | |
| 4667 | + return NULL; | |
| 4668 | + | |
| 4669 | + count = ma_data_end(node, type, pivots, mas->max); | |
| 4670 | + if (mas->offset > count) | |
| 4671 | + return NULL; | |
| 4672 | + | |
| 4673 | + while (mas->offset < count) { | |
| 4674 | + pivot = pivots[mas->offset]; | |
| 4675 | + entry = mas_slot(mas, slots, mas->offset); | |
| 4676 | + if (ma_dead_node(node)) | |
| 4677 | + return NULL; | |
| 4678 | + | |
| 4679 | + if (entry) | |
| 4680 | + goto found; | |
| 4681 | + | |
| 4682 | + if (pivot >= max) | |
| 4683 | + return NULL; | |
| 4684 | + | |
| 4685 | + mas->index = pivot + 1; | |
| 4686 | + mas->offset++; | |
| 4687 | + } | |
| 4688 | + | |
| 4689 | + if (mas->index > mas->max) { | |
| 4690 | + mas->index = mas->last; | |
| 4691 | + return NULL; | |
| 4692 | + } | |
| 4693 | + | |
| 4694 | + pivot = mas_safe_pivot(mas, pivots, mas->offset, type); | |
| 4695 | + entry = mas_slot(mas, slots, mas->offset); | |
| 4696 | + if (ma_dead_node(node)) | |
| 4697 | + return NULL; | |
| 4698 | + | |
| 4699 | + if (!pivot) | |
| 4700 | + return NULL; | |
| 4701 | + | |
| 4702 | + if (!entry) | |
| 4703 | + return NULL; | |
| 4704 | + | |
| 4705 | +found: | |
| 4706 | + mas->last = pivot; | |
| 4707 | + return entry; | |
| 4708 | +} | |
| 4709 | + | |
| 4710 | +static inline void mas_rewalk(struct ma_state *mas, unsigned long index) | |
| 4711 | +{ | |
| 4712 | + | |
| 4713 | +retry: | |
| 4714 | + mas_set(mas, index); | |
| 4715 | + mas_state_walk(mas); | |
| 4716 | + if (mas_is_start(mas)) | |
| 4717 | + goto retry; | |
| 4718 | + | |
| 4719 | + return; | |
| 4720 | + | |
| 4721 | +} | |
| 4722 | + | |
| 4723 | +/* | |
| 4724 | + * mas_next_entry() - Internal function to get the next entry. | |
| 4725 | + * @mas: The maple state | |
| 4726 | + * @limit: The maximum range start. | |
| 4727 | + * | |
| 4728 | + * Set the @mas->node to the next entry and the range_start to | |
| 4729 | + * the beginning value for the entry. Does not check beyond @limit. | |
| 4730 | + * Sets @mas->index and @mas->last to the limit if it is hit. | |
| 4731 | + * Restarts on dead nodes. | |
| 4732 | + * | |
| 4733 | + * Return: the next entry or %NULL. | |
| 4734 | + */ | |
| 4735 | +static inline void *mas_next_entry(struct ma_state *mas, unsigned long limit) | |
| 4736 | +{ | |
| 4737 | + void *entry = NULL; | |
| 4738 | + struct maple_enode *prev_node; | |
| 4739 | + struct maple_node *node; | |
| 4740 | + unsigned char offset; | |
| 4741 | + unsigned long last; | |
| 4742 | + enum maple_type mt; | |
| 4743 | + | |
| 4744 | + last = mas->last; | |
| 4745 | +retry: | |
| 4746 | + offset = mas->offset; | |
| 4747 | + prev_node = mas->node; | |
| 4748 | + node = mas_mn(mas); | |
| 4749 | + mt = mte_node_type(mas->node); | |
| 4750 | + mas->offset++; | |
| 4751 | + if (unlikely(mas->offset >= mt_slots[mt])) { | |
| 4752 | + mas->offset = mt_slots[mt] - 1; | |
| 4753 | + goto next_node; | |
| 4754 | + } | |
| 4755 | + | |
| 4756 | + while (!mas_is_none(mas)) { | |
| 4757 | + entry = mas_next_nentry(mas, node, limit, mt); | |
| 4758 | + if (unlikely(ma_dead_node(node))) { | |
| 4759 | + mas_rewalk(mas, last); | |
| 4760 | + goto retry; | |
| 4761 | + } | |
| 4762 | + | |
| 4763 | + if (likely(entry)) | |
| 4764 | + return entry; | |
| 4765 | + | |
| 4766 | + if (unlikely((mas->index > limit))) | |
| 4767 | + break; | |
| 4768 | + | |
| 4769 | +next_node: | |
| 4770 | + prev_node = mas->node; | |
| 4771 | + offset = mas->offset; | |
| 4772 | + if (unlikely(mas_next_node(mas, node, limit))) { | |
| 4773 | + mas_rewalk(mas, last); | |
| 4774 | + goto retry; | |
| 4775 | + } | |
| 4776 | + mas->offset = 0; | |
| 4777 | + node = mas_mn(mas); | |
| 4778 | + mt = mte_node_type(mas->node); | |
| 4779 | + } | |
| 4780 | + | |
| 4781 | + mas->index = mas->last = limit; | |
| 4782 | + mas->offset = offset; | |
| 4783 | + mas->node = prev_node; | |
| 4784 | + return NULL; | |
| 4785 | +} | |
| 4786 | + | |
| 4787 | +/* | |
| 4788 | + * mas_prev_nentry() - Get the previous node entry. | |
| 4789 | + * @mas: The maple state. | |
| 4790 | + * @limit: The lower limit to check for a value. | |
| 4791 | + * | |
| 4792 | + * Return: the entry, %NULL otherwise. | |
| 4793 | + */ | |
| 4794 | +static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, | |
| 4795 | + unsigned long index) | |
| 4796 | +{ | |
| 4797 | + unsigned long pivot, min; | |
| 4798 | + unsigned char offset; | |
| 4799 | + struct maple_node *mn; | |
| 4800 | + enum maple_type mt; | |
| 4801 | + unsigned long *pivots; | |
| 4802 | + void __rcu **slots; | |
| 4803 | + void *entry; | |
| 4804 | + | |
| 4805 | +retry: | |
| 4806 | + if (!mas->offset) | |
| 4807 | + return NULL; | |
| 4808 | + | |
| 4809 | + mn = mas_mn(mas); | |
| 4810 | + mt = mte_node_type(mas->node); | |
| 4811 | + offset = mas->offset - 1; | |
| 4812 | + if (offset >= mt_slots[mt]) | |
| 4813 | + offset = mt_slots[mt] - 1; | |
| 4814 | + | |
| 4815 | + slots = ma_slots(mn, mt); | |
| 4816 | + pivots = ma_pivots(mn, mt); | |
| 4817 | + if (offset == mt_pivots[mt]) | |
| 4818 | + pivot = mas->max; | |
| 4819 | + else | |
| 4820 | + pivot = pivots[offset]; | |
| 4821 | + | |
| 4822 | + if (unlikely(ma_dead_node(mn))) { | |
| 4823 | + mas_rewalk(mas, index); | |
| 4824 | + goto retry; | |
| 4825 | + } | |
| 4826 | + | |
| 4827 | + while (offset && ((!mas_slot(mas, slots, offset) && pivot >= limit) || | |
| 4828 | + !pivot)) | |
| 4829 | + pivot = pivots[--offset]; | |
| 4830 | + | |
| 4831 | + min = mas_safe_min(mas, pivots, offset); | |
| 4832 | + entry = mas_slot(mas, slots, offset); | |
| 4833 | + if (unlikely(ma_dead_node(mn))) { | |
| 4834 | + mas_rewalk(mas, index); | |
| 4835 | + goto retry; | |
| 4836 | + } | |
| 4837 | + | |
| 4838 | + if (likely(entry)) { | |
| 4839 | + mas->offset = offset; | |
| 4840 | + mas->last = pivot; | |
| 4841 | + mas->index = min; | |
| 4842 | + } | |
| 4843 | + return entry; | |
| 4844 | +} | |
| 4845 | + | |
| 4846 | +static inline void *mas_prev_entry(struct ma_state *mas, unsigned long min) | |
| 4847 | +{ | |
| 4848 | + void *entry; | |
| 4849 | + | |
| 4850 | +retry: | |
| 4851 | + while (likely(!mas_is_none(mas))) { | |
| 4852 | + entry = mas_prev_nentry(mas, min, mas->index); | |
| 4853 | + if (unlikely(mas->last < min)) | |
| 4854 | + goto not_found; | |
| 4855 | + | |
| 4856 | + if (likely(entry)) | |
| 4857 | + return entry; | |
| 4858 | + | |
| 4859 | + if (unlikely(mas_prev_node(mas, min))) { | |
| 4860 | + mas_rewalk(mas, mas->index); | |
| 4861 | + goto retry; | |
| 4862 | + } | |
| 4863 | + | |
| 4864 | + mas->offset++; | |
| 4865 | + } | |
| 4866 | + | |
| 4867 | + mas->offset--; | |
| 4868 | +not_found: | |
| 4869 | + mas->index = mas->last = min; | |
| 4870 | + return NULL; | |
| 4871 | +} | |
| 4872 | + | |
| 4873 | +/* | |
| 4874 | + * mas_rev_awalk() - Internal function. Reverse allocation walk. Find the | |
| 4875 | + * highest gap address of a given size in a given node and descend. | |
| 4876 | + * @mas: The maple state | |
| 4877 | + * @size: The needed size. | |
| 4878 | + * | |
| 4879 | + * Return: True if found in a leaf, false otherwise. | |
| 4880 | + * | |
| 4881 | + */ | |
| 4882 | +static bool mas_rev_awalk(struct ma_state *mas, unsigned long size) | |
| 4883 | +{ | |
| 4884 | + enum maple_type type = mte_node_type(mas->node); | |
| 4885 | + struct maple_node *node = mas_mn(mas); | |
| 4886 | + unsigned long *pivots, *gaps; | |
| 4887 | + void __rcu **slots; | |
| 4888 | + unsigned long gap = 0; | |
| 4889 | + unsigned long max, min, index; | |
| 4890 | + unsigned char offset; | |
| 4891 | + | |
| 4892 | + if (unlikely(mas_is_err(mas))) | |
| 4893 | + return true; | |
| 4894 | + | |
| 4895 | + if (ma_is_dense(type)) { | |
| 4896 | + /* dense nodes. */ | |
| 4897 | + mas->offset = (unsigned char)(mas->index - mas->min); | |
| 4898 | + return true; | |
| 4899 | + } | |
| 4900 | + | |
| 4901 | + pivots = ma_pivots(node, type); | |
| 4902 | + slots = ma_slots(node, type); | |
| 4903 | + gaps = ma_gaps(node, type); | |
| 4904 | + offset = mas->offset; | |
| 4905 | + min = mas_safe_min(mas, pivots, offset); | |
| 4906 | + /* Skip out of bounds. */ | |
| 4907 | + while (mas->last < min) | |
| 4908 | + min = mas_safe_min(mas, pivots, --offset); | |
| 4909 | + | |
| 4910 | + max = mas_safe_pivot(mas, pivots, offset, type); | |
| 4911 | + index = mas->index; | |
| 4912 | + while (index <= max) { | |
| 4913 | + gap = 0; | |
| 4914 | + if (gaps) | |
| 4915 | + gap = gaps[offset]; | |
| 4916 | + else if (!mas_slot(mas, slots, offset)) | |
| 4917 | + gap = max - min + 1; | |
| 4918 | + | |
| 4919 | + if (gap) { | |
| 4920 | + if ((size <= gap) && (size <= mas->last - min + 1)) | |
| 4921 | + break; | |
| 4922 | + | |
| 4923 | + if (!gaps) { | |
| 4924 | + /* Skip the next slot, it cannot be a gap. */ | |
| 4925 | + if (offset < 2) | |
| 4926 | + goto ascend; | |
| 4927 | + | |
| 4928 | + offset -= 2; | |
| 4929 | + max = pivots[offset]; | |
| 4930 | + min = mas_safe_min(mas, pivots, offset); | |
| 4931 | + continue; | |
| 4932 | + } | |
| 4933 | + } | |
| 4934 | + | |
| 4935 | + if (!offset) | |
| 4936 | + goto ascend; | |
| 4937 | + | |
| 4938 | + offset--; | |
| 4939 | + max = min - 1; | |
| 4940 | + min = mas_safe_min(mas, pivots, offset); | |
| 4941 | + } | |
| 4942 | + | |
| 4943 | + if (unlikely(index > max)) { | |
| 4944 | + mas_set_err(mas, -EBUSY); | |
| 4945 | + return false; | |
| 4946 | + } | |
| 4947 | + | |
| 4948 | + if (unlikely(ma_is_leaf(type))) { | |
| 4949 | + mas->offset = offset; | |
| 4950 | + mas->min = min; | |
| 4951 | + mas->max = min + gap - 1; | |
| 4952 | + return true; | |
| 4953 | + } | |
| 4954 | + | |
| 4955 | + /* descend, only happens under lock. */ | |
| 4956 | + mas->node = mas_slot(mas, slots, offset); | |
| 4957 | + mas->min = min; | |
| 4958 | + mas->max = max; | |
| 4959 | + mas->offset = mas_data_end(mas); | |
| 4960 | + return false; | |
| 4961 | + | |
| 4962 | +ascend: | |
| 4963 | + if (mte_is_root(mas->node)) | |
| 4964 | + mas_set_err(mas, -EBUSY); | |
| 4965 | + | |
| 4966 | + return false; | |
| 4967 | +} | |
| 4968 | + | |
| 4969 | +static inline bool mas_anode_descend(struct ma_state *mas, unsigned long size) | |
| 4970 | +{ | |
| 4971 | + enum maple_type type = mte_node_type(mas->node); | |
| 4972 | + unsigned long pivot, min, gap = 0; | |
| 4973 | + unsigned char count, offset; | |
| 4974 | + unsigned long *gaps = NULL, *pivots = ma_pivots(mas_mn(mas), type); | |
| 4975 | + void __rcu **slots = ma_slots(mas_mn(mas), type); | |
| 4976 | + bool found = false; | |
| 4977 | + | |
| 4978 | + if (ma_is_dense(type)) { | |
| 4979 | + mas->offset = (unsigned char)(mas->index - mas->min); | |
| 4980 | + return true; | |
| 4981 | + } | |
| 4982 | + | |
| 4983 | + gaps = ma_gaps(mte_to_node(mas->node), type); | |
| 4984 | + offset = mas->offset; | |
| 4985 | + count = mt_slots[type]; | |
| 4986 | + min = mas_safe_min(mas, pivots, offset); | |
| 4987 | + for (; offset < count; offset++) { | |
| 4988 | + pivot = mas_safe_pivot(mas, pivots, offset, type); | |
| 4989 | + if (offset && !pivot) | |
| 4990 | + break; | |
| 4991 | + | |
| 4992 | + /* Not within lower bounds */ | |
| 4993 | + if (mas->index > pivot) | |
| 4994 | + goto next_slot; | |
| 4995 | + | |
| 4996 | + if (gaps) | |
| 4997 | + gap = gaps[offset]; | |
| 4998 | + else if (!mas_slot(mas, slots, offset)) | |
| 4999 | + gap = min(pivot, mas->last) - max(mas->index, min) + 1; | |
2131 more lines aren’t shown here.
| @@ -8,13 +8,13 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/mm.h> |
| 9 | 9 | #include <linux/cma.h> |
| 10 | 10 | |
| 11 | −void show_mem(unsigned int filter, nodemask_t *nodemask) | |
| 11 | +void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) | |
| 12 | 12 | { |
| 13 | 13 | pg_data_t *pgdat; |
| 14 | 14 | unsigned long total = 0, reserved = 0, highmem = 0; |
| 15 | 15 | |
| 16 | 16 | printk("Mem-Info:\n"); |
| 17 | − show_free_areas(filter, nodemask); | |
| 17 | + __show_free_areas(filter, nodemask, max_zone_idx); | |
| 18 | 18 | |
| 19 | 19 | for_each_online_pgdat(pgdat) { |
| 20 | 20 | int zoneid; |
| @@ -43,7 +43,8 @@ · 42 unmodified lines | ||
| 43 | 43 | #define STACK_ALLOC_OFFSET_BITS (STACK_ALLOC_ORDER + PAGE_SHIFT - \ |
| 44 | 44 | STACK_ALLOC_ALIGN) |
| 45 | 45 | #define STACK_ALLOC_INDEX_BITS (DEPOT_STACK_BITS - \ |
| 46 | − STACK_ALLOC_NULL_PROTECTION_BITS - STACK_ALLOC_OFFSET_BITS) | |
| 46 | + STACK_ALLOC_NULL_PROTECTION_BITS - \ | |
| 47 | + STACK_ALLOC_OFFSET_BITS - STACK_DEPOT_EXTRA_BITS) | |
| 47 | 48 | #define STACK_ALLOC_SLABS_CAP 8192 |
| 48 | 49 | #define STACK_ALLOC_MAX_SLABS \ |
| 49 | 50 | (((1LL << (STACK_ALLOC_INDEX_BITS)) < STACK_ALLOC_SLABS_CAP) ? \ |
| @@ -56,6 +57,7 @@ union handle_parts { · 6 unmodified lines | ||
| 56 | 57 | u32 slabindex : STACK_ALLOC_INDEX_BITS; |
| 57 | 58 | u32 offset : STACK_ALLOC_OFFSET_BITS; |
| 58 | 59 | u32 valid : STACK_ALLOC_NULL_PROTECTION_BITS; |
| 60 | + u32 extra : STACK_DEPOT_EXTRA_BITS; | |
| 59 | 61 | }; |
| 60 | 62 | }; |
| 61 | 63 | |
| @@ -77,6 +79,14 @@ static int next_slab_inited; · 15 unmodified lines | ||
| 77 | 79 | static size_t depot_offset; |
| 78 | 80 | static DEFINE_RAW_SPINLOCK(depot_lock); |
| 79 | 81 | |
| 82 | +unsigned int stack_depot_get_extra_bits(depot_stack_handle_t handle) | |
| 83 | +{ | |
| 84 | + union handle_parts parts = { .handle = handle }; | |
| 85 | + | |
| 86 | + return parts.extra; | |
| 87 | +} | |
| 88 | +EXPORT_SYMBOL(stack_depot_get_extra_bits); | |
| 89 | + | |
| 80 | 90 | static bool init_stack_slab(void **prealloc) |
| 81 | 91 | { |
| 82 | 92 | if (!*prealloc) |
| @@ -140,6 +150,7 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) · 57 unmodified lines | ||
| 140 | 150 | stack->handle.slabindex = depot_index; |
| 141 | 151 | stack->handle.offset = depot_offset >> STACK_ALLOC_ALIGN; |
| 142 | 152 | stack->handle.valid = 1; |
| 153 | + stack->handle.extra = 0; | |
| 143 | 154 | memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); |
| 144 | 155 | depot_offset += required_size; |
| 145 | 156 | |
| @@ -382,6 +393,7 @@ EXPORT_SYMBOL_GPL(stack_depot_fetch); · 236 unmodified lines | ||
| 382 | 393 | * |
| 383 | 394 | * @entries: Pointer to storage array |
| 384 | 395 | * @nr_entries: Size of the storage array |
| 396 | + * @extra_bits: Flags to store in unused bits of depot_stack_handle_t | |
| 385 | 397 | * @alloc_flags: Allocation gfp flags |
| 386 | 398 | * @can_alloc: Allocate stack slabs (increased chance of failure if false) |
| 387 | 399 | * |
| @@ -393,6 +405,10 @@ EXPORT_SYMBOL_GPL(stack_depot_fetch); · 5 unmodified lines | ||
| 393 | 405 | * If the stack trace in @entries is from an interrupt, only the portion up to |
| 394 | 406 | * interrupt entry is saved. |
| 395 | 407 | * |
| 408 | + * Additional opaque flags can be passed in @extra_bits, stored in the unused | |
| 409 | + * bits of the stack handle, and retrieved using stack_depot_get_extra_bits() | |
| 410 | + * without calling stack_depot_fetch(). | |
| 411 | + * | |
| 396 | 412 | * Context: Any context, but setting @can_alloc to %false is required if |
| 397 | 413 | * alloc_pages() cannot be used from the current context. Currently |
| 398 | 414 | * this is the case from contexts where neither %GFP_ATOMIC nor |
| @@ -402,10 +418,11 @@ EXPORT_SYMBOL_GPL(stack_depot_fetch); · 3 unmodified lines | ||
| 402 | 418 | */ |
| 403 | 419 | depot_stack_handle_t __stack_depot_save(unsigned long *entries, |
| 404 | 420 | unsigned int nr_entries, |
| 421 | + unsigned int extra_bits, | |
| 405 | 422 | gfp_t alloc_flags, bool can_alloc) |
| 406 | 423 | { |
| 407 | 424 | struct stack_record *found = NULL, **bucket; |
| 408 | − depot_stack_handle_t retval = 0; | |
| 425 | + union handle_parts retval = { .handle = 0 }; | |
| 409 | 426 | struct page *page = NULL; |
| 410 | 427 | void *prealloc = NULL; |
| 411 | 428 | unsigned long flags; |
| @@ -489,9 +506,11 @@ exit: · 77 unmodified lines | ||
| 489 | 506 | free_pages((unsigned long)prealloc, STACK_ALLOC_ORDER); |
| 490 | 507 | } |
| 491 | 508 | if (found) |
| 492 | − retval = found->handle.handle; | |
| 509 | + retval.handle = found->handle.handle; | |
| 493 | 510 | fast_exit: |
| 494 | − return retval; | |
| 511 | + retval.extra = extra_bits; | |
| 512 | + | |
| 513 | + return retval.handle; | |
| 495 | 514 | } |
| 496 | 515 | EXPORT_SYMBOL_GPL(__stack_depot_save); |
| 497 | 516 | |
| @@ -511,6 +530,6 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries, · 13 unmodified lines | ||
| 511 | 530 | unsigned int nr_entries, |
| 512 | 531 | gfp_t alloc_flags) |
| 513 | 532 | { |
| 514 | − return __stack_depot_save(entries, nr_entries, alloc_flags, true); | |
| 533 | + return __stack_depot_save(entries, nr_entries, 0, alloc_flags, true); | |
| 515 | 534 | } |
| 516 | 535 | EXPORT_SYMBOL_GPL(stack_depot_save); |
| @@ -197,6 +197,14 @@ ssize_t strscpy(char *dest, const char *src, size_t count) · 196 unmodified lines | ||
| 197 | 197 | max = 0; |
| 198 | 198 | #endif |
| 199 | 199 | |
| 200 | + /* | |
| 201 | + * read_word_at_a_time() below may read uninitialized bytes after the | |
| 202 | + * trailing zero and use them in comparisons. Disable this optimization | |
| 203 | + * under KMSAN to prevent false positive reports. | |
| 204 | + */ | |
| 205 | + if (IS_ENABLED(CONFIG_KMSAN)) | |
| 206 | + max = 0; | |
| 207 | + | |
| 200 | 208 | while (max >= sizeof(unsigned long)) { |
| 201 | 209 | unsigned long c, data; |
| 202 | 210 | |
| @@ -107,8 +107,8 @@ struct dmirror_chunk { · 106 unmodified lines | ||
| 107 | 107 | */ |
| 108 | 108 | struct dmirror_device { |
| 109 | 109 | struct cdev cdevice; |
| 110 | − struct hmm_devmem *devmem; | |
| 111 | 110 | unsigned int zone_device_type; |
| 111 | + struct device device; | |
| 112 | 112 | |
| 113 | 113 | unsigned int devmem_capacity; |
| 114 | 114 | unsigned int devmem_count; |
| @@ -1390,7 +1390,14 @@ static int dmirror_device_init(struct dmirror_device *mdevice, int id) · 1275 unmodified lines | ||
| 1390 | 1390 | |
| 1391 | 1391 | cdev_init(&mdevice->cdevice, &dmirror_fops); |
| 1392 | 1392 | mdevice->cdevice.owner = THIS_MODULE; |
| 1393 | − ret = cdev_add(&mdevice->cdevice, dev, 1); | |
| 1393 | + device_initialize(&mdevice->device); | |
| 1394 | + mdevice->device.devt = dev; | |
| 1395 | + | |
| 1396 | + ret = dev_set_name(&mdevice->device, "hmm_dmirror%u", id); | |
| 1397 | + if (ret) | |
| 1398 | + return ret; | |
| 1399 | + | |
| 1400 | + ret = cdev_device_add(&mdevice->cdevice, &mdevice->device); | |
| 1394 | 1401 | if (ret) |
| 1395 | 1402 | return ret; |
| 1396 | 1403 | |
| @@ -1416,7 +1423,7 @@ static void dmirror_device_remove(struct dmirror_device *mdevice) · 19 unmodified lines | ||
| 1416 | 1423 | kfree(mdevice->devmem_chunks); |
| 1417 | 1424 | } |
| 1418 | 1425 | |
| 1419 | − cdev_del(&mdevice->cdevice); | |
| 1426 | + cdev_device_del(&mdevice->cdevice, &mdevice->device); | |
| 1420 | 1427 | } |
| 1421 | 1428 | |
| 1422 | 1429 | static int __init hmm_dmirror_init(void) |
| @@ -0,0 +1,38307 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0+ | |
| 2 | +/* | |
| 3 | + * test_maple_tree.c: Test the maple tree API | |
| 4 | + * Copyright (c) 2018 Liam R. Howlett | |
| 5 | + * Author: Liam R. Howlett <Liam.Howlett@Oracle.com> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +#include <linux/maple_tree.h> | |
| 9 | +#include <linux/module.h> | |
| 10 | +#include <stdlib.h> | |
| 11 | +#include <time.h> | |
| 12 | + | |
| 13 | +#define MTREE_ALLOC_MAX 0x2000000000000Ul | |
| 14 | +#define CONFIG_DEBUG_MAPLE_TREE | |
| 15 | +#define CONFIG_MAPLE_SEARCH | |
| 16 | +/* #define BENCH_SLOT_STORE */ | |
| 17 | +/* #define BENCH_NODE_STORE */ | |
| 18 | +/* #define BENCH_AWALK */ | |
| 19 | +/* #define BENCH_WALK */ | |
| 20 | +/* #define BENCH_MT_FOR_EACH */ | |
| 21 | +/* #define BENCH_FORK */ | |
| 22 | +static | |
| 23 | +int mtree_insert_index(struct maple_tree *mt, unsigned long index, gfp_t gfp) | |
| 24 | +{ | |
| 25 | + return mtree_insert(mt, index, xa_mk_value(index & LONG_MAX), gfp); | |
| 26 | +} | |
| 27 | + | |
| 28 | +static void mtree_erase_index(struct maple_tree *mt, unsigned long index) | |
| 29 | +{ | |
| 30 | + MT_BUG_ON(mt, mtree_erase(mt, index) != xa_mk_value(index & LONG_MAX)); | |
| 31 | + MT_BUG_ON(mt, mtree_load(mt, index) != NULL); | |
| 32 | +} | |
| 33 | + | |
| 34 | +static int mtree_test_insert(struct maple_tree *mt, unsigned long index, | |
| 35 | + void *ptr) | |
| 36 | +{ | |
| 37 | + return mtree_insert(mt, index, ptr, GFP_KERNEL); | |
| 38 | +} | |
| 39 | + | |
| 40 | +static int mtree_test_store_range(struct maple_tree *mt, unsigned long start, | |
| 41 | + unsigned long end, void *ptr) | |
| 42 | +{ | |
| 43 | + return mtree_store_range(mt, start, end, ptr, GFP_KERNEL); | |
| 44 | +} | |
| 45 | + | |
| 46 | +static int mtree_test_store(struct maple_tree *mt, unsigned long start, | |
| 47 | + void *ptr) | |
| 48 | +{ | |
| 49 | + return mtree_test_store_range(mt, start, start, ptr); | |
| 50 | +} | |
| 51 | + | |
| 52 | +static int mtree_test_insert_range(struct maple_tree *mt, unsigned long start, | |
| 53 | + unsigned long end, void *ptr) | |
| 54 | +{ | |
| 55 | + return mtree_insert_range(mt, start, end, ptr, GFP_KERNEL); | |
| 56 | +} | |
| 57 | + | |
| 58 | +static void *mtree_test_load(struct maple_tree *mt, unsigned long index) | |
| 59 | +{ | |
| 60 | + return mtree_load(mt, index); | |
| 61 | +} | |
| 62 | + | |
| 63 | +static void *mtree_test_erase(struct maple_tree *mt, unsigned long index) | |
| 64 | +{ | |
| 65 | + return mtree_erase(mt, index); | |
| 66 | +} | |
| 67 | + | |
| 68 | +static noinline void check_mtree_alloc_range(struct maple_tree *mt, | |
| 69 | + unsigned long start, unsigned long end, unsigned long size, | |
| 70 | + unsigned long expected, int eret, void *ptr) | |
| 71 | +{ | |
| 72 | + | |
| 73 | + unsigned long result = expected + 1; | |
| 74 | + int ret; | |
| 75 | + | |
| 76 | + ret = mtree_alloc_range(mt, &result, ptr, size, start, end, | |
| 77 | + GFP_KERNEL); | |
| 78 | + MT_BUG_ON(mt, ret != eret); | |
| 79 | + if (ret) | |
| 80 | + return; | |
| 81 | + | |
| 82 | + MT_BUG_ON(mt, result != expected); | |
| 83 | +} | |
| 84 | + | |
| 85 | +static noinline void check_mtree_alloc_rrange(struct maple_tree *mt, | |
| 86 | + unsigned long start, unsigned long end, unsigned long size, | |
| 87 | + unsigned long expected, int eret, void *ptr) | |
| 88 | +{ | |
| 89 | + | |
| 90 | + unsigned long result = expected + 1; | |
| 91 | + int ret; | |
| 92 | + | |
| 93 | + ret = mtree_alloc_rrange(mt, &result, ptr, size, start, end - 1, | |
| 94 | + GFP_KERNEL); | |
| 95 | + MT_BUG_ON(mt, ret != eret); | |
| 96 | + if (ret) | |
| 97 | + return; | |
| 98 | + | |
| 99 | + MT_BUG_ON(mt, result != expected); | |
| 100 | +} | |
| 101 | + | |
| 102 | +static noinline void check_load(struct maple_tree *mt, unsigned long index, | |
| 103 | + void *ptr) | |
| 104 | +{ | |
| 105 | + void *ret = mtree_test_load(mt, index); | |
| 106 | + | |
| 107 | + if (ret != ptr) | |
| 108 | + pr_err("Load %lu returned %p expect %p\n", index, ret, ptr); | |
| 109 | + MT_BUG_ON(mt, ret != ptr); | |
| 110 | +} | |
| 111 | + | |
| 112 | +static noinline void check_store_range(struct maple_tree *mt, | |
| 113 | + unsigned long start, unsigned long end, void *ptr, int expected) | |
| 114 | +{ | |
| 115 | + int ret = -EINVAL; | |
| 116 | + unsigned long i; | |
| 117 | + | |
| 118 | + ret = mtree_test_store_range(mt, start, end, ptr); | |
| 119 | + MT_BUG_ON(mt, ret != expected); | |
| 120 | + | |
| 121 | + if (ret) | |
| 122 | + return; | |
| 123 | + | |
| 124 | + for (i = start; i <= end; i++) | |
| 125 | + check_load(mt, i, ptr); | |
| 126 | +} | |
| 127 | + | |
| 128 | +static noinline void check_insert_range(struct maple_tree *mt, | |
| 129 | + unsigned long start, unsigned long end, void *ptr, int expected) | |
| 130 | +{ | |
| 131 | + int ret = -EINVAL; | |
| 132 | + unsigned long i; | |
| 133 | + | |
| 134 | + ret = mtree_test_insert_range(mt, start, end, ptr); | |
| 135 | + MT_BUG_ON(mt, ret != expected); | |
| 136 | + | |
| 137 | + if (ret) | |
| 138 | + return; | |
| 139 | + | |
| 140 | + for (i = start; i <= end; i++) | |
| 141 | + check_load(mt, i, ptr); | |
| 142 | +} | |
| 143 | + | |
| 144 | +static noinline void check_insert(struct maple_tree *mt, unsigned long index, | |
| 145 | + void *ptr) | |
| 146 | +{ | |
| 147 | + int ret = -EINVAL; | |
| 148 | + | |
| 149 | + ret = mtree_test_insert(mt, index, ptr); | |
| 150 | + MT_BUG_ON(mt, ret != 0); | |
| 151 | +} | |
| 152 | + | |
| 153 | +static noinline void check_erase(struct maple_tree *mt, unsigned long index, | |
| 154 | + void *ptr) | |
| 155 | +{ | |
| 156 | + MT_BUG_ON(mt, mtree_test_erase(mt, index) != ptr); | |
| 157 | +} | |
| 158 | + | |
| 159 | +static noinline void check_dup_insert(struct maple_tree *mt, | |
| 160 | + unsigned long index, void *ptr) | |
| 161 | +{ | |
| 162 | + int ret = -EINVAL; | |
| 163 | + | |
| 164 | + ret = mtree_test_insert(mt, index, ptr); | |
| 165 | + MT_BUG_ON(mt, ret != -EEXIST); | |
| 166 | +} | |
| 167 | + | |
| 168 | + | |
| 169 | +static noinline | |
| 170 | +void check_index_load(struct maple_tree *mt, unsigned long index) | |
| 171 | +{ | |
| 172 | + return check_load(mt, index, xa_mk_value(index & LONG_MAX)); | |
| 173 | +} | |
| 174 | + | |
| 175 | +static noinline void check_nomem(struct maple_tree *mt) | |
| 176 | +{ | |
| 177 | + MA_STATE(ms, mt, 1, 1); | |
| 178 | + | |
| 179 | + MT_BUG_ON(mt, !mtree_empty(mt)); | |
| 180 | + /* Ensure no bypassing of allocation failures */ | |
| 181 | + mt_set_non_kernel(0); | |
| 182 | + | |
| 183 | + /* Storing something at 1 requires memory allocation */ | |
| 184 | + MT_BUG_ON(mt, mtree_insert(mt, 1, &ms, GFP_ATOMIC) != -ENOMEM); | |
| 185 | + /* Storing something at 0 does not */ | |
| 186 | + MT_BUG_ON(mt, mtree_insert(mt, 0, &ms, GFP_ATOMIC) != 0); | |
| 187 | + | |
| 188 | + /* | |
| 189 | + * Simulate two threads racing; the first one fails to allocate | |
| 190 | + * memory to insert an entry at 1, then the second one succeeds | |
| 191 | + * in allocating memory to insert an entry at 2. The first one | |
| 192 | + * then needs to free the node it allocated. LeakSanitizer will | |
| 193 | + * notice this, as will the 'nr_allocated' debugging aid in the | |
| 194 | + * userspace test suite. | |
| 195 | + */ | |
| 196 | + mtree_lock(mt); | |
| 197 | + mas_store(&ms, &ms); /* insert 1 -> &ms, fails. */ | |
| 198 | + MT_BUG_ON(mt, ms.node != MA_ERROR(-ENOMEM)); | |
| 199 | + mas_nomem(&ms, GFP_KERNEL); /* Node allocated in here. */ | |
| 200 | + MT_BUG_ON(mt, ms.node != MAS_START); | |
| 201 | + mtree_unlock(mt); | |
| 202 | + MT_BUG_ON(mt, mtree_insert(mt, 2, mt, GFP_KERNEL) != 0); | |
| 203 | + mtree_lock(mt); | |
| 204 | + mas_store(&ms, &ms); /* insert 1 -> &ms */ | |
| 205 | + mas_nomem(&ms, GFP_KERNEL); /* Node allocated in here. */ | |
| 206 | + mtree_unlock(mt); | |
| 207 | + mtree_destroy(mt); | |
| 208 | +} | |
| 209 | + | |
| 210 | +static inline int not_empty(struct maple_node *node) | |
| 211 | +{ | |
| 212 | + int i; | |
| 213 | + | |
| 214 | + if (node->parent) | |
| 215 | + return 1; | |
| 216 | + | |
| 217 | + for (i = 0; i < ARRAY_SIZE(node->slot); i++) | |
| 218 | + if (node->slot[i]) | |
| 219 | + return 1; | |
| 220 | + | |
| 221 | + return 0; | |
| 222 | +} | |
| 223 | + | |
| 224 | +static noinline void check_new_node(struct maple_tree *mt) | |
| 225 | +{ | |
| 226 | + | |
| 227 | + struct maple_node *mn, *mn2, *mn3; | |
| 228 | + struct maple_alloc *smn; | |
| 229 | + struct maple_node *nodes[100]; | |
| 230 | + int i, j, total; | |
| 231 | + | |
| 232 | + MA_STATE(mas, mt, 0, 0); | |
| 233 | + | |
| 234 | + /* Try allocating 3 nodes */ | |
| 235 | + mtree_lock(mt); | |
| 236 | + /* request 3 nodes to be allocated. */ | |
| 237 | + mas_node_count(&mas, 3); | |
| 238 | + /* Allocation request of 3. */ | |
| 239 | + MT_BUG_ON(mt, mas_alloc_req(&mas) != 3); | |
| 240 | + /* Allocate failed. */ | |
| 241 | + MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM)); | |
| 242 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 243 | + | |
| 244 | + MT_BUG_ON(mt, mas_allocated(&mas) != 3); | |
| 245 | + mn = mas_pop_node(&mas); | |
| 246 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 247 | + MT_BUG_ON(mt, mn == NULL); | |
| 248 | + MT_BUG_ON(mt, mas.alloc == NULL); | |
| 249 | + MT_BUG_ON(mt, mas.alloc->slot[0] == NULL); | |
| 250 | + mas_push_node(&mas, mn); | |
| 251 | + mas_nomem(&mas, GFP_KERNEL); /* free */ | |
| 252 | + mtree_unlock(mt); | |
| 253 | + | |
| 254 | + | |
| 255 | + /* Try allocating 1 node, then 2 more */ | |
| 256 | + mtree_lock(mt); | |
| 257 | + /* Set allocation request to 1. */ | |
| 258 | + mas_set_alloc_req(&mas, 1); | |
| 259 | + /* Check Allocation request of 1. */ | |
| 260 | + MT_BUG_ON(mt, mas_alloc_req(&mas) != 1); | |
| 261 | + mas_set_err(&mas, -ENOMEM); | |
| 262 | + /* Validate allocation request. */ | |
| 263 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 264 | + /* Eat the requested node. */ | |
| 265 | + mn = mas_pop_node(&mas); | |
| 266 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 267 | + MT_BUG_ON(mt, mn == NULL); | |
| 268 | + MT_BUG_ON(mt, mn->slot[0] != NULL); | |
| 269 | + MT_BUG_ON(mt, mn->slot[1] != NULL); | |
| 270 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 271 | + | |
| 272 | + ma_free_rcu(mn); | |
| 273 | + mas.node = MAS_START; | |
| 274 | + mas_nomem(&mas, GFP_KERNEL); | |
| 275 | + /* Allocate 3 nodes, will fail. */ | |
| 276 | + mas_node_count(&mas, 3); | |
| 277 | + /* Drop the lock and allocate 3 nodes. */ | |
| 278 | + mas_nomem(&mas, GFP_KERNEL); | |
| 279 | + /* Ensure 3 are allocated. */ | |
| 280 | + MT_BUG_ON(mt, mas_allocated(&mas) != 3); | |
| 281 | + /* Allocation request of 0. */ | |
| 282 | + MT_BUG_ON(mt, mas_alloc_req(&mas) != 0); | |
| 283 | + | |
| 284 | + MT_BUG_ON(mt, mas.alloc == NULL); | |
| 285 | + MT_BUG_ON(mt, mas.alloc->slot[0] == NULL); | |
| 286 | + MT_BUG_ON(mt, mas.alloc->slot[1] == NULL); | |
| 287 | + /* Ensure we counted 3. */ | |
| 288 | + MT_BUG_ON(mt, mas_allocated(&mas) != 3); | |
| 289 | + /* Free. */ | |
| 290 | + mas_nomem(&mas, GFP_KERNEL); | |
| 291 | + | |
| 292 | + /* Set allocation request to 1. */ | |
| 293 | + mas_set_alloc_req(&mas, 1); | |
| 294 | + MT_BUG_ON(mt, mas_alloc_req(&mas) != 1); | |
| 295 | + mas_set_err(&mas, -ENOMEM); | |
| 296 | + /* Validate allocation request. */ | |
| 297 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 298 | + MT_BUG_ON(mt, mas_allocated(&mas) != 1); | |
| 299 | + /* Check the node is only one node. */ | |
| 300 | + mn = mas_pop_node(&mas); | |
| 301 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 302 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 303 | + MT_BUG_ON(mt, mn == NULL); | |
| 304 | + MT_BUG_ON(mt, mn->slot[0] != NULL); | |
| 305 | + MT_BUG_ON(mt, mn->slot[1] != NULL); | |
| 306 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 307 | + mas_push_node(&mas, mn); | |
| 308 | + MT_BUG_ON(mt, mas_allocated(&mas) != 1); | |
| 309 | + MT_BUG_ON(mt, mas.alloc->node_count); | |
| 310 | + | |
| 311 | + mas_set_alloc_req(&mas, 2); /* request 2 more. */ | |
| 312 | + MT_BUG_ON(mt, mas_alloc_req(&mas) != 2); | |
| 313 | + mas_set_err(&mas, -ENOMEM); | |
| 314 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 315 | + MT_BUG_ON(mt, mas_allocated(&mas) != 3); | |
| 316 | + MT_BUG_ON(mt, mas.alloc == NULL); | |
| 317 | + MT_BUG_ON(mt, mas.alloc->slot[0] == NULL); | |
| 318 | + MT_BUG_ON(mt, mas.alloc->slot[1] == NULL); | |
| 319 | + for (i = 2; i >= 0; i--) { | |
| 320 | + mn = mas_pop_node(&mas); | |
| 321 | + MT_BUG_ON(mt, mas_allocated(&mas) != i); | |
| 322 | + MT_BUG_ON(mt, !mn); | |
| 323 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 324 | + ma_free_rcu(mn); | |
| 325 | + } | |
| 326 | + | |
| 327 | + total = 64; | |
| 328 | + mas_set_alloc_req(&mas, total); /* request 2 more. */ | |
| 329 | + MT_BUG_ON(mt, mas_alloc_req(&mas) != total); | |
| 330 | + mas_set_err(&mas, -ENOMEM); | |
| 331 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 332 | + for (i = total; i > 0; i--) { | |
| 333 | + unsigned int e = 0; /* expected node_count */ | |
| 334 | + | |
| 335 | + if (i >= 35) | |
| 336 | + e = i - 35; | |
| 337 | + else if (i >= 5) | |
| 338 | + e = i - 5; | |
| 339 | + else if (i >= 2) | |
| 340 | + e = i - 2; | |
| 341 | + MT_BUG_ON(mt, mas.alloc->node_count != e); | |
| 342 | + mn = mas_pop_node(&mas); | |
| 343 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 344 | + MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); | |
| 345 | + MT_BUG_ON(mt, !mn); | |
| 346 | + ma_free_rcu(mn); | |
| 347 | + } | |
| 348 | + | |
| 349 | + total = 100; | |
| 350 | + for (i = 1; i < total; i++) { | |
| 351 | + mas_set_alloc_req(&mas, i); | |
| 352 | + mas_set_err(&mas, -ENOMEM); | |
| 353 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 354 | + for (j = i; j > 0; j--) { | |
| 355 | + mn = mas_pop_node(&mas); | |
| 356 | + MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); | |
| 357 | + MT_BUG_ON(mt, !mn); | |
| 358 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 359 | + mas_push_node(&mas, mn); | |
| 360 | + MT_BUG_ON(mt, mas_allocated(&mas) != j); | |
| 361 | + mn = mas_pop_node(&mas); | |
| 362 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 363 | + MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); | |
| 364 | + ma_free_rcu(mn); | |
| 365 | + } | |
| 366 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 367 | + | |
| 368 | + mas_set_alloc_req(&mas, i); | |
| 369 | + mas_set_err(&mas, -ENOMEM); | |
| 370 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 371 | + for (j = 0; j <= i/2; j++) { | |
| 372 | + MT_BUG_ON(mt, mas_allocated(&mas) != i - j); | |
| 373 | + nodes[j] = mas_pop_node(&mas); | |
| 374 | + MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); | |
| 375 | + } | |
| 376 | + | |
| 377 | + while (j) { | |
| 378 | + j--; | |
| 379 | + mas_push_node(&mas, nodes[j]); | |
| 380 | + MT_BUG_ON(mt, mas_allocated(&mas) != i - j); | |
| 381 | + } | |
| 382 | + MT_BUG_ON(mt, mas_allocated(&mas) != i); | |
| 383 | + for (j = 0; j <= i/2; j++) { | |
| 384 | + MT_BUG_ON(mt, mas_allocated(&mas) != i - j); | |
| 385 | + mn = mas_pop_node(&mas); | |
| 386 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 387 | + ma_free_rcu(mn); | |
| 388 | + MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); | |
| 389 | + } | |
| 390 | + MT_BUG_ON(mt, mas_nomem(&mas, GFP_KERNEL)); | |
| 391 | + | |
| 392 | + } | |
| 393 | + | |
| 394 | + /* Set allocation request. */ | |
| 395 | + total = 500; | |
| 396 | + mas_node_count(&mas, total); | |
| 397 | + /* Drop the lock and allocate the nodes. */ | |
| 398 | + mas_nomem(&mas, GFP_KERNEL); | |
| 399 | + MT_BUG_ON(mt, !mas.alloc); | |
| 400 | + i = 1; | |
| 401 | + smn = mas.alloc; | |
| 402 | + while (i < total) { | |
| 403 | + for (j = 0; j < MAPLE_ALLOC_SLOTS; j++) { | |
| 404 | + i++; | |
| 405 | + MT_BUG_ON(mt, !smn->slot[j]); | |
| 406 | + if (i == total) | |
| 407 | + break; | |
| 408 | + } | |
| 409 | + smn = smn->slot[0]; /* next. */ | |
| 410 | + } | |
| 411 | + MT_BUG_ON(mt, mas_allocated(&mas) != total); | |
| 412 | + mas_nomem(&mas, GFP_KERNEL); /* Free. */ | |
| 413 | + | |
| 414 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 415 | + for (i = 1; i < 128; i++) { | |
| 416 | + mas_node_count(&mas, i); /* Request */ | |
| 417 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 418 | + MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */ | |
| 419 | + for (j = i; j > 0; j--) { /*Free the requests */ | |
| 420 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 421 | + MT_BUG_ON(mt, mn == NULL); | |
| 422 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 423 | + ma_free_rcu(mn); | |
| 424 | + } | |
| 425 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 426 | + } | |
| 427 | + | |
| 428 | + for (i = 1; i < MAPLE_NODE_MASK + 1; i++) { | |
| 429 | + MA_STATE(mas2, mt, 0, 0); | |
| 430 | + mas_node_count(&mas, i); /* Request */ | |
| 431 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 432 | + MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */ | |
| 433 | + for (j = 1; j <= i; j++) { /* Move the allocations to mas2 */ | |
| 434 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 435 | + MT_BUG_ON(mt, mn == NULL); | |
| 436 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 437 | + mas_push_node(&mas2, mn); | |
| 438 | + MT_BUG_ON(mt, mas_allocated(&mas2) != j); | |
| 439 | + } | |
| 440 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 441 | + MT_BUG_ON(mt, mas_allocated(&mas2) != i); | |
| 442 | + | |
| 443 | + for (j = i; j > 0; j--) { /*Free the requests */ | |
| 444 | + MT_BUG_ON(mt, mas_allocated(&mas2) != j); | |
| 445 | + mn = mas_pop_node(&mas2); /* get the next node. */ | |
| 446 | + MT_BUG_ON(mt, mn == NULL); | |
| 447 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 448 | + ma_free_rcu(mn); | |
| 449 | + } | |
| 450 | + MT_BUG_ON(mt, mas_allocated(&mas2) != 0); | |
| 451 | + } | |
| 452 | + | |
| 453 | + | |
| 454 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 455 | + mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */ | |
| 456 | + MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM)); | |
| 457 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 458 | + MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1); | |
| 459 | + MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1); | |
| 460 | + | |
| 461 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 462 | + MT_BUG_ON(mt, mn == NULL); | |
| 463 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 464 | + MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS); | |
| 465 | + MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 2); | |
| 466 | + | |
| 467 | + mas_push_node(&mas, mn); | |
| 468 | + MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1); | |
| 469 | + MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1); | |
| 470 | + | |
| 471 | + /* Check the limit of pop/push/pop */ | |
| 472 | + mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 2); /* Request */ | |
| 473 | + MT_BUG_ON(mt, mas_alloc_req(&mas) != 1); | |
| 474 | + MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM)); | |
| 475 | + MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL)); | |
| 476 | + MT_BUG_ON(mt, mas_alloc_req(&mas)); | |
| 477 | + MT_BUG_ON(mt, mas.alloc->node_count); | |
| 478 | + MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); | |
| 479 | + mn = mas_pop_node(&mas); | |
| 480 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 481 | + MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1); | |
| 482 | + MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1); | |
| 483 | + mas_push_node(&mas, mn); | |
| 484 | + MT_BUG_ON(mt, mas.alloc->node_count); | |
| 485 | + MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); | |
| 486 | + mn = mas_pop_node(&mas); | |
| 487 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 488 | + ma_free_rcu(mn); | |
| 489 | + for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { | |
| 490 | + mn = mas_pop_node(&mas); | |
| 491 | + MT_BUG_ON(mt, not_empty(mn)); | |
| 492 | + ma_free_rcu(mn); | |
| 493 | + } | |
| 494 | + MT_BUG_ON(mt, mas_allocated(&mas) != 0); | |
| 495 | + | |
| 496 | + | |
| 497 | + for (i = 3; i < MAPLE_NODE_MASK * 3; i++) { | |
| 498 | + mas.node = MA_ERROR(-ENOMEM); | |
| 499 | + mas_node_count(&mas, i); /* Request */ | |
| 500 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 501 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 502 | + mas_push_node(&mas, mn); /* put it back */ | |
| 503 | + mas_destroy(&mas); | |
| 504 | + | |
| 505 | + mas.node = MA_ERROR(-ENOMEM); | |
| 506 | + mas_node_count(&mas, i); /* Request */ | |
| 507 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 508 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 509 | + mn2 = mas_pop_node(&mas); /* get the next node. */ | |
| 510 | + mas_push_node(&mas, mn); /* put them back */ | |
| 511 | + mas_push_node(&mas, mn2); | |
| 512 | + mas_destroy(&mas); | |
| 513 | + | |
| 514 | + mas.node = MA_ERROR(-ENOMEM); | |
| 515 | + mas_node_count(&mas, i); /* Request */ | |
| 516 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 517 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 518 | + mn2 = mas_pop_node(&mas); /* get the next node. */ | |
| 519 | + mn3 = mas_pop_node(&mas); /* get the next node. */ | |
| 520 | + mas_push_node(&mas, mn); /* put them back */ | |
| 521 | + mas_push_node(&mas, mn2); | |
| 522 | + mas_push_node(&mas, mn3); | |
| 523 | + mas_destroy(&mas); | |
| 524 | + | |
| 525 | + mas.node = MA_ERROR(-ENOMEM); | |
| 526 | + mas_node_count(&mas, i); /* Request */ | |
| 527 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 528 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 529 | + ma_free_rcu(mn); | |
| 530 | + mas_destroy(&mas); | |
| 531 | + | |
| 532 | + mas.node = MA_ERROR(-ENOMEM); | |
| 533 | + mas_node_count(&mas, i); /* Request */ | |
| 534 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 535 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 536 | + ma_free_rcu(mn); | |
| 537 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 538 | + ma_free_rcu(mn); | |
| 539 | + mn = mas_pop_node(&mas); /* get the next node. */ | |
| 540 | + ma_free_rcu(mn); | |
| 541 | + mas_destroy(&mas); | |
| 542 | + } | |
| 543 | + | |
| 544 | + mas.node = MA_ERROR(-ENOMEM); | |
| 545 | + mas_node_count(&mas, 5); /* Request */ | |
| 546 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 547 | + MT_BUG_ON(mt, mas_allocated(&mas) != 5); | |
| 548 | + mas.node = MA_ERROR(-ENOMEM); | |
| 549 | + mas_node_count(&mas, 10); /* Request */ | |
| 550 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 551 | + mas.node = MAS_START; | |
| 552 | + MT_BUG_ON(mt, mas_allocated(&mas) != 10); | |
| 553 | + mas_destroy(&mas); | |
| 554 | + | |
| 555 | + mas.node = MA_ERROR(-ENOMEM); | |
| 556 | + mas_node_count(&mas, MAPLE_ALLOC_SLOTS - 1); /* Request */ | |
| 557 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 558 | + MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS - 1); | |
| 559 | + mas.node = MA_ERROR(-ENOMEM); | |
| 560 | + mas_node_count(&mas, 10 + MAPLE_ALLOC_SLOTS - 1); /* Request */ | |
| 561 | + mas_nomem(&mas, GFP_KERNEL); /* Fill request */ | |
| 562 | + mas.node = MAS_START; | |
| 563 | + MT_BUG_ON(mt, mas_allocated(&mas) != 10 + MAPLE_ALLOC_SLOTS - 1); | |
| 564 | + mas_destroy(&mas); | |
| 565 | + | |
| 566 | + mtree_unlock(mt); | |
| 567 | +} | |
| 568 | + | |
| 569 | +static noinline void check_rev_seq(struct maple_tree *mt, unsigned long max, | |
| 570 | + bool verbose) | |
| 571 | +{ | |
| 572 | + unsigned long i = max, j; | |
| 573 | + | |
| 574 | + MT_BUG_ON(mt, !mtree_empty(mt)); | |
| 575 | + | |
| 576 | + mt_zero_nr_tallocated(); | |
| 577 | + while (i) { | |
| 578 | + MT_BUG_ON(mt, mtree_insert_index(mt, i, GFP_KERNEL)); | |
| 579 | + for (j = i; j <= max; j++) | |
| 580 | + check_index_load(mt, j); | |
| 581 | + | |
| 582 | + check_load(mt, i - 1, NULL); | |
| 583 | + mt_set_in_rcu(mt); | |
| 584 | + MT_BUG_ON(mt, !mt_height(mt)); | |
| 585 | + mt_clear_in_rcu(mt); | |
| 586 | + MT_BUG_ON(mt, !mt_height(mt)); | |
| 587 | + i--; | |
| 588 | + } | |
| 589 | + check_load(mt, max + 1, NULL); | |
| 590 | + | |
| 591 | + if (verbose) { | |
| 592 | + rcu_barrier(); | |
| 593 | + mt_dump(mt); | |
| 594 | + pr_info(" %s test of 0-%lu %luK in %d active (%d total)\n", | |
| 595 | + __func__, max, mt_get_alloc_size()/1024, mt_nr_allocated(), | |
| 596 | + mt_nr_tallocated()); | |
| 597 | + } | |
| 598 | +} | |
| 599 | + | |
| 600 | +static noinline void check_seq(struct maple_tree *mt, unsigned long max, | |
| 601 | + bool verbose) | |
| 602 | +{ | |
| 603 | + unsigned long i, j; | |
| 604 | + | |
| 605 | + MT_BUG_ON(mt, !mtree_empty(mt)); | |
| 606 | + | |
| 607 | + mt_zero_nr_tallocated(); | |
| 608 | + for (i = 0; i <= max; i++) { | |
| 609 | + MT_BUG_ON(mt, mtree_insert_index(mt, i, GFP_KERNEL)); | |
| 610 | + for (j = 0; j <= i; j++) | |
| 611 | + check_index_load(mt, j); | |
| 612 | + | |
| 613 | + if (i) | |
| 614 | + MT_BUG_ON(mt, !mt_height(mt)); | |
| 615 | + check_load(mt, i + 1, NULL); | |
| 616 | + } | |
| 617 | + if (verbose) { | |
| 618 | + rcu_barrier(); | |
| 619 | + mt_dump(mt); | |
| 620 | + pr_info(" seq test of 0-%lu %luK in %d active (%d total)\n", | |
| 621 | + max, mt_get_alloc_size()/1024, mt_nr_allocated(), | |
| 622 | + mt_nr_tallocated()); | |
| 623 | + } | |
| 624 | +} | |
| 625 | + | |
| 626 | +static noinline void check_lb_not_empty(struct maple_tree *mt) | |
| 627 | +{ | |
| 628 | + unsigned long i, j; | |
| 629 | + unsigned long huge = 4000UL * 1000 * 1000; | |
| 630 | + | |
| 631 | + | |
| 632 | + i = huge; | |
| 633 | + while (i > 4096) { | |
| 634 | + check_insert(mt, i, (void *) i); | |
| 635 | + for (j = huge; j >= i; j /= 2) { | |
| 636 | + check_load(mt, j-1, NULL); | |
| 637 | + check_load(mt, j, (void *) j); | |
| 638 | + check_load(mt, j+1, NULL); | |
| 639 | + } | |
| 640 | + i /= 2; | |
| 641 | + } | |
| 642 | + mtree_destroy(mt); | |
| 643 | +} | |
| 644 | + | |
| 645 | +static noinline void check_lower_bound_split(struct maple_tree *mt) | |
| 646 | +{ | |
| 647 | + MT_BUG_ON(mt, !mtree_empty(mt)); | |
| 648 | + check_lb_not_empty(mt); | |
| 649 | +} | |
| 650 | + | |
| 651 | +static noinline void check_upper_bound_split(struct maple_tree *mt) | |
| 652 | +{ | |
| 653 | + unsigned long i, j; | |
| 654 | + unsigned long huge = 4000UL * 1000 * 1000; | |
| 655 | + | |
| 656 | + MT_BUG_ON(mt, !mtree_empty(mt)); | |
| 657 | + | |
| 658 | + i = 4096; | |
| 659 | + while (i < huge) { | |
| 660 | + check_insert(mt, i, (void *) i); | |
| 661 | + for (j = i; j >= huge; j *= 2) { | |
| 662 | + check_load(mt, j-1, NULL); | |
| 663 | + check_load(mt, j, (void *) j); | |
| 664 | + check_load(mt, j+1, NULL); | |
| 665 | + } | |
| 666 | + i *= 2; | |
| 667 | + } | |
| 668 | + mtree_destroy(mt); | |
| 669 | +} | |
| 670 | + | |
| 671 | +static noinline void check_mid_split(struct maple_tree *mt) | |
| 672 | +{ | |
| 673 | + unsigned long huge = 8000UL * 1000 * 1000; | |
| 674 | + | |
| 675 | + check_insert(mt, huge, (void *) huge); | |
| 676 | + check_insert(mt, 0, xa_mk_value(0)); | |
| 677 | + check_lb_not_empty(mt); | |
| 678 | +} | |
| 679 | + | |
| 680 | +static noinline void check_rev_find(struct maple_tree *mt) | |
| 681 | +{ | |
| 682 | + int i, nr_entries = 200; | |
| 683 | + void *val; | |
| 684 | + MA_STATE(mas, mt, 0, 0); | |
| 685 | + | |
| 686 | + for (i = 0; i <= nr_entries; i++) | |
| 687 | + mtree_store_range(mt, i*10, i*10 + 5, | |
| 688 | + xa_mk_value(i), GFP_KERNEL); | |
| 689 | + | |
| 690 | + mas_set(&mas, 1000); | |
| 691 | + val = mas_find_rev(&mas, 1000); | |
| 692 | + MT_BUG_ON(mt, val != xa_mk_value(100)); | |
| 693 | + val = mas_find_rev(&mas, 1000); | |
| 694 | + MT_BUG_ON(mt, val != NULL); | |
| 695 | + | |
| 696 | + mas_set(&mas, 999); | |
| 697 | + val = mas_find_rev(&mas, 997); | |
| 698 | + MT_BUG_ON(mt, val != NULL); | |
| 699 | + | |
| 700 | + mas_set(&mas, 1000); | |
| 701 | + val = mas_find_rev(&mas, 900); | |
| 702 | + MT_BUG_ON(mt, val != xa_mk_value(100)); | |
| 703 | + val = mas_find_rev(&mas, 900); | |
| 704 | + MT_BUG_ON(mt, val != xa_mk_value(99)); | |
| 705 | + | |
| 706 | + mas_set(&mas, 20); | |
| 707 | + val = mas_find_rev(&mas, 0); | |
| 708 | + MT_BUG_ON(mt, val != xa_mk_value(2)); | |
| 709 | + val = mas_find_rev(&mas, 0); | |
| 710 | + MT_BUG_ON(mt, val != xa_mk_value(1)); | |
| 711 | + val = mas_find_rev(&mas, 0); | |
| 712 | + MT_BUG_ON(mt, val != xa_mk_value(0)); | |
| 713 | + val = mas_find_rev(&mas, 0); | |
| 714 | + MT_BUG_ON(mt, val != NULL); | |
| 715 | +} | |
| 716 | + | |
| 717 | +static noinline void check_find(struct maple_tree *mt) | |
| 718 | +{ | |
| 719 | + unsigned long val = 0; | |
| 720 | + unsigned long count = 20; | |
| 721 | + unsigned long max; | |
| 722 | + unsigned long last = 0, index = 0; | |
| 723 | + void *entry, *entry2; | |
| 724 | + | |
| 725 | + MA_STATE(mas, mt, 0, 0); | |
| 726 | + | |
| 727 | + /* Insert 0. */ | |
| 728 | + MT_BUG_ON(mt, mtree_insert_index(mt, val++, GFP_KERNEL)); | |
| 729 | + | |
| 730 | + for (int i = 0; i <= count; i++) { | |
| 731 | + if (val != 64) | |
| 732 | + MT_BUG_ON(mt, mtree_insert_index(mt, val, GFP_KERNEL)); | |
| 733 | + else | |
| 734 | + MT_BUG_ON(mt, mtree_insert(mt, val, | |
| 735 | + XA_ZERO_ENTRY, GFP_KERNEL)); | |
| 736 | + | |
| 737 | + val <<= 2; | |
| 738 | + } | |
| 739 | + | |
| 740 | + val = 0; | |
| 741 | + mas_set(&mas, val); | |
| 742 | + mas_lock(&mas); | |
| 743 | + while ((entry = mas_find(&mas, 268435456)) != NULL) { | |
| 744 | + if (val != 64) | |
| 745 | + MT_BUG_ON(mt, xa_mk_value(val) != entry); | |
| 746 | + else | |
| 747 | + MT_BUG_ON(mt, entry != XA_ZERO_ENTRY); | |
| 748 | + | |
| 749 | + val <<= 2; | |
| 750 | + /* For zero check. */ | |
| 751 | + if (!val) | |
| 752 | + val = 1; | |
| 753 | + } | |
| 754 | + mas_unlock(&mas); | |
| 755 | + | |
| 756 | + val = 0; | |
| 757 | + mas_set(&mas, val); | |
| 758 | + mas_lock(&mas); | |
| 759 | + mas_for_each(&mas, entry, ULONG_MAX) { | |
| 760 | + if (val != 64) | |
| 761 | + MT_BUG_ON(mt, xa_mk_value(val) != entry); | |
| 762 | + else | |
| 763 | + MT_BUG_ON(mt, entry != XA_ZERO_ENTRY); | |
| 764 | + val <<= 2; | |
| 765 | + /* For zero check. */ | |
| 766 | + if (!val) | |
| 767 | + val = 1; | |
| 768 | + } | |
| 769 | + mas_unlock(&mas); | |
| 770 | + | |
| 771 | + /* Test mas_pause */ | |
| 772 | + val = 0; | |
| 773 | + mas_set(&mas, val); | |
| 774 | + mas_lock(&mas); | |
| 775 | + mas_for_each(&mas, entry, ULONG_MAX) { | |
| 776 | + if (val != 64) | |
| 777 | + MT_BUG_ON(mt, xa_mk_value(val) != entry); | |
| 778 | + else | |
| 779 | + MT_BUG_ON(mt, entry != XA_ZERO_ENTRY); | |
| 780 | + val <<= 2; | |
| 781 | + /* For zero check. */ | |
| 782 | + if (!val) | |
| 783 | + val = 1; | |
| 784 | + | |
| 785 | + mas_pause(&mas); | |
| 786 | + mas_unlock(&mas); | |
| 787 | + mas_lock(&mas); | |
| 788 | + } | |
| 789 | + mas_unlock(&mas); | |
| 790 | + | |
| 791 | + val = 0; | |
| 792 | + max = 300; /* A value big enough to include XA_ZERO_ENTRY at 64. */ | |
| 793 | + mt_for_each(mt, entry, index, max) { | |
| 794 | + MT_BUG_ON(mt, xa_mk_value(val) != entry); | |
| 795 | + val <<= 2; | |
| 796 | + if (val == 64) /* Skip zero entry. */ | |
| 797 | + val <<= 2; | |
| 798 | + /* For zero check. */ | |
| 799 | + if (!val) | |
| 800 | + val = 1; | |
| 801 | + } | |
| 802 | + | |
| 803 | + val = 0; | |
| 804 | + max = 0; | |
| 805 | + index = 0; | |
| 806 | + MT_BUG_ON(mt, mtree_insert_index(mt, ULONG_MAX, GFP_KERNEL)); | |
| 807 | + mt_for_each(mt, entry, index, ULONG_MAX) { | |
| 808 | + if (val == 4398046511104) | |
| 809 | + MT_BUG_ON(mt, entry != | |
| 810 | + xa_mk_value(ULONG_MAX & LONG_MAX)); | |
| 811 | + else | |
| 812 | + MT_BUG_ON(mt, xa_mk_value(val) != entry); | |
| 813 | + val <<= 2; | |
| 814 | + if (val == 64) /* Skip zero entry. */ | |
| 815 | + val <<= 2; | |
| 816 | + /* For zero check. */ | |
| 817 | + if (!val) | |
| 818 | + val = 1; | |
| 819 | + max++; | |
| 820 | + MT_BUG_ON(mt, max > 25); | |
| 821 | + } | |
| 822 | + mtree_erase_index(mt, ULONG_MAX); | |
| 823 | + | |
| 824 | + mas_reset(&mas); | |
| 825 | + index = 17; | |
| 826 | + entry = mt_find(mt, &index, 512); | |
| 827 | + MT_BUG_ON(mt, xa_mk_value(256) != entry); | |
| 828 | + | |
| 829 | + mas_reset(&mas); | |
| 830 | + index = 17; | |
| 831 | + entry = mt_find(mt, &index, 20); | |
| 832 | + MT_BUG_ON(mt, entry != NULL); | |
| 833 | + | |
| 834 | + | |
| 835 | + /* Range check.. */ | |
| 836 | + /* Insert ULONG_MAX */ | |
| 837 | + MT_BUG_ON(mt, mtree_insert_index(mt, ULONG_MAX, GFP_KERNEL)); | |
| 838 | + | |
| 839 | + val = 0; | |
| 840 | + mas_set(&mas, 0); | |
| 841 | + mas_lock(&mas); | |
| 842 | + mas_for_each(&mas, entry, ULONG_MAX) { | |
| 843 | + if (val == 64) | |
| 844 | + MT_BUG_ON(mt, entry != XA_ZERO_ENTRY); | |
| 845 | + else if (val == 4398046511104) | |
| 846 | + MT_BUG_ON(mt, entry != xa_mk_value(ULONG_MAX & LONG_MAX)); | |
| 847 | + else | |
| 848 | + MT_BUG_ON(mt, xa_mk_value(val) != entry); | |
| 849 | + val <<= 2; | |
| 850 | + | |
| 851 | + /* For zero check. */ | |
| 852 | + if (!val) | |
| 853 | + val = 1; | |
| 854 | + mas_pause(&mas); | |
| 855 | + mas_unlock(&mas); | |
| 856 | + mas_lock(&mas); | |
| 857 | + } | |
| 858 | + mas_unlock(&mas); | |
| 859 | + | |
| 860 | + mas_set(&mas, 1048576); | |
| 861 | + mas_lock(&mas); | |
| 862 | + entry = mas_find(&mas, 1048576); | |
| 863 | + mas_unlock(&mas); | |
| 864 | + MT_BUG_ON(mas.tree, entry == NULL); | |
| 865 | + | |
| 866 | + /* | |
| 867 | + * Find last value. | |
| 868 | + * 1. get the expected value, leveraging the existence of an end entry | |
| 869 | + * 2. delete end entry | |
| 870 | + * 3. find the last value but searching for ULONG_MAX and then using | |
| 871 | + * prev | |
| 872 | + */ | |
| 873 | + /* First, get the expected result. */ | |
| 874 | + mas_lock(&mas); | |
| 875 | + mas_reset(&mas); | |
| 876 | + mas.index = ULONG_MAX; /* start at max.. */ | |
| 877 | + entry = mas_find(&mas, ULONG_MAX); | |
| 878 | + entry = mas_prev(&mas, 0); | |
| 879 | + index = mas.index; | |
| 880 | + last = mas.last; | |
| 881 | + | |
| 882 | + /* Erase the last entry. */ | |
| 883 | + mas_reset(&mas); | |
| 884 | + mas.index = ULONG_MAX; | |
| 885 | + mas.last = ULONG_MAX; | |
| 886 | + mas_erase(&mas); | |
| 887 | + | |
| 888 | + /* Get the previous value from MAS_START */ | |
| 889 | + mas_reset(&mas); | |
| 890 | + entry2 = mas_prev(&mas, 0); | |
| 891 | + | |
| 892 | + /* Check results. */ | |
| 893 | + MT_BUG_ON(mt, entry != entry2); | |
| 894 | + MT_BUG_ON(mt, index != mas.index); | |
| 895 | + MT_BUG_ON(mt, last != mas.last); | |
| 896 | + | |
| 897 | + | |
| 898 | + mas.node = MAS_NONE; | |
| 899 | + mas.index = ULONG_MAX; | |
| 900 | + mas.last = ULONG_MAX; | |
| 901 | + entry2 = mas_prev(&mas, 0); | |
| 902 | + MT_BUG_ON(mt, entry != entry2); | |
| 903 | + | |
| 904 | + mas_set(&mas, 0); | |
| 905 | + MT_BUG_ON(mt, mas_prev(&mas, 0) != NULL); | |
| 906 | + | |
| 907 | + mas_unlock(&mas); | |
| 908 | + mtree_destroy(mt); | |
| 909 | +} | |
| 910 | + | |
| 911 | +static noinline void check_find_2(struct maple_tree *mt) | |
| 912 | +{ | |
| 913 | + unsigned long i, j; | |
| 914 | + void *entry; | |
| 915 | + | |
| 916 | + MA_STATE(mas, mt, 0, 0); | |
| 917 | + rcu_read_lock(); | |
| 918 | + mas_for_each(&mas, entry, ULONG_MAX) | |
| 919 | + MT_BUG_ON(mt, true); | |
| 920 | + rcu_read_unlock(); | |
| 921 | + | |
| 922 | + for (i = 0; i < 256; i++) { | |
| 923 | + mtree_insert_index(mt, i, GFP_KERNEL); | |
| 924 | + j = 0; | |
| 925 | + mas_set(&mas, 0); | |
| 926 | + rcu_read_lock(); | |
| 927 | + mas_for_each(&mas, entry, ULONG_MAX) { | |
| 928 | + MT_BUG_ON(mt, entry != xa_mk_value(j)); | |
| 929 | + j++; | |
| 930 | + } | |
| 931 | + rcu_read_unlock(); | |
| 932 | + MT_BUG_ON(mt, j != i + 1); | |
| 933 | + } | |
| 934 | + | |
| 935 | + for (i = 0; i < 256; i++) { | |
| 936 | + mtree_erase_index(mt, i); | |
| 937 | + j = i + 1; | |
| 938 | + mas_set(&mas, 0); | |
| 939 | + rcu_read_lock(); | |
| 940 | + mas_for_each(&mas, entry, ULONG_MAX) { | |
| 941 | + if (xa_is_zero(entry)) | |
| 942 | + continue; | |
| 943 | + | |
| 944 | + MT_BUG_ON(mt, entry != xa_mk_value(j)); | |
| 945 | + j++; | |
| 946 | + } | |
| 947 | + rcu_read_unlock(); | |
| 948 | + MT_BUG_ON(mt, j != 256); | |
| 949 | + } | |
| 950 | + | |
| 951 | + /*MT_BUG_ON(mt, !mtree_empty(mt)); */ | |
| 952 | +} | |
| 953 | + | |
| 954 | +#define erase_ptr(i) entry[i%2] | |
| 955 | +#define erase_check_load(mt, i) check_load(mt, set[i], entry[i%2]) | |
| 956 | +#define erase_check_insert(mt, i) check_insert(mt, set[i], entry[i%2]) | |
| 957 | +#define erase_check_erase(mt, i) check_erase(mt, set[i], entry[i%2]) | |
| 958 | + | |
| 959 | +static noinline void check_erase_testset(struct maple_tree *mt) | |
| 960 | +{ | |
| 961 | + unsigned long set[] = { 5015, 5014, 5017, 25, 1000, | |
| 962 | + 1001, 1002, 1003, 1005, 0, | |
| 963 | + 6003, 6002, 6008, 6012, 6015, | |
| 964 | + 7003, 7002, 7008, 7012, 7015, | |
| 965 | + 8003, 8002, 8008, 8012, 8015, | |
| 966 | + 9003, 9002, 9008, 9012, 9015, | |
| 967 | + 10003, 10002, 10008, 10012, 10015, | |
| 968 | + 11003, 11002, 11008, 11012, 11015, | |
| 969 | + 12003, 12002, 12008, 12012, 12015, | |
| 970 | + 13003, 13002, 13008, 13012, 13015, | |
| 971 | + 14003, 14002, 14008, 14012, 14015, | |
| 972 | + 15003, 15002, 15008, 15012, 15015, | |
| 973 | + }; | |
| 974 | + | |
| 975 | + | |
| 976 | + void *ptr = &set; | |
| 977 | + void *entry[2] = { ptr, mt }; | |
| 978 | + void *root_node; | |
| 979 | + | |
| 980 | + | |
| 981 | + rcu_register_thread(); | |
| 982 | + mt_set_in_rcu(mt); | |
| 983 | + for (int i = 0; i < 4; i++) | |
| 984 | + erase_check_insert(mt, i); | |
| 985 | + for (int i = 0; i < 4; i++) | |
| 986 | + erase_check_load(mt, i); | |
| 987 | + | |
| 988 | + mt_set_non_kernel(2); | |
| 989 | + erase_check_erase(mt, 1); | |
| 990 | + erase_check_load(mt, 0); | |
| 991 | + check_load(mt, set[1], NULL); | |
| 992 | + for (int i = 2; i < 4; i++) | |
| 993 | + erase_check_load(mt, i); | |
| 994 | + | |
| 995 | + | |
| 996 | + erase_check_erase(mt, 2); | |
| 997 | + erase_check_load(mt, 0); | |
| 998 | + check_load(mt, set[1], NULL); | |
| 999 | + check_load(mt, set[2], NULL); | |
| 1000 | + | |
| 1001 | + erase_check_insert(mt, 1); | |
| 1002 | + erase_check_insert(mt, 2); | |
| 1003 | + | |
| 1004 | + for (int i = 0; i < 4; i++) | |
| 1005 | + erase_check_load(mt, i); | |
| 1006 | + | |
| 1007 | + /* Check erase and load without an allocation. */ | |
| 1008 | + erase_check_load(mt, 3); | |
| 1009 | + erase_check_erase(mt, 1); | |
| 1010 | + erase_check_load(mt, 0); | |
| 1011 | + check_load(mt, set[1], NULL); | |
| 1012 | + for (int i = 2; i < 4; i++) | |
| 1013 | + erase_check_load(mt, i); | |
| 1014 | + | |
| 1015 | + /* | |
| 1016 | + * Set the newly erased node. This will produce a different allocated | |
| 1017 | + * node to avoid busy slots. | |
| 1018 | + */ | |
| 1019 | + root_node = mt->ma_root; | |
| 1020 | + erase_check_insert(mt, 1); | |
| 1021 | + | |
| 1022 | + erase_check_load(mt, 0); | |
| 1023 | + check_load(mt, 5016, NULL); | |
| 1024 | + erase_check_load(mt, 1); | |
| 1025 | + check_load(mt, 5013, NULL); | |
| 1026 | + erase_check_load(mt, 2); | |
| 1027 | + check_load(mt, 5018, NULL); | |
| 1028 | + erase_check_load(mt, 3); | |
| 1029 | + | |
| 1030 | + erase_check_erase(mt, 2); /* erase 5017 to check append */ | |
| 1031 | + erase_check_load(mt, 0); | |
| 1032 | + check_load(mt, 5016, NULL); | |
| 1033 | + erase_check_load(mt, 1); | |
| 1034 | + check_load(mt, 5013, NULL); | |
| 1035 | + check_load(mt, set[2], NULL); | |
| 1036 | + check_load(mt, 5018, NULL); | |
| 1037 | + | |
| 1038 | + erase_check_load(mt, 3); | |
| 1039 | + | |
| 1040 | + root_node = mt->ma_root; | |
| 1041 | + erase_check_insert(mt, 2); | |
| 1042 | + | |
| 1043 | + erase_check_load(mt, 0); | |
| 1044 | + check_load(mt, 5016, NULL); | |
| 1045 | + erase_check_load(mt, 1); | |
| 1046 | + check_load(mt, 5013, NULL); | |
| 1047 | + erase_check_load(mt, 2); | |
| 1048 | + check_load(mt, 5018, NULL); | |
| 1049 | + erase_check_load(mt, 3); | |
| 1050 | + | |
| 1051 | + mt_set_non_kernel(1); | |
| 1052 | + erase_check_erase(mt, 2); /* erase 5017 to check append */ | |
| 1053 | + erase_check_load(mt, 0); | |
| 1054 | + check_load(mt, 5016, NULL); | |
| 1055 | + check_load(mt, set[2], NULL); | |
| 1056 | + erase_check_erase(mt, 0); /* erase 5015 to check append */ | |
| 1057 | + check_load(mt, set[0], NULL); | |
| 1058 | + check_load(mt, 5016, NULL); | |
| 1059 | + erase_check_insert(mt, 4); /* 1000 < Should not split. */ | |
| 1060 | + check_load(mt, set[0], NULL); | |
| 1061 | + check_load(mt, 5016, NULL); | |
| 1062 | + erase_check_load(mt, 1); | |
| 1063 | + check_load(mt, 5013, NULL); | |
| 1064 | + check_load(mt, set[2], NULL); | |
| 1065 | + check_load(mt, 5018, NULL); | |
| 1066 | + erase_check_load(mt, 4); | |
| 1067 | + check_load(mt, 999, NULL); | |
| 1068 | + check_load(mt, 1001, NULL); | |
| 1069 | + erase_check_load(mt, 4); | |
| 1070 | + if (mt_in_rcu(mt)) | |
| 1071 | + MT_BUG_ON(mt, root_node == mt->ma_root); | |
| 1072 | + else | |
| 1073 | + MT_BUG_ON(mt, root_node != mt->ma_root); | |
| 1074 | + | |
| 1075 | + /* Should not have split. */ | |
| 1076 | + MT_BUG_ON(mt, !mte_is_leaf(mt->ma_root)); | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + /* Coalesce testing */ | |
| 1080 | + erase_check_insert(mt, 0); | |
| 1081 | + erase_check_insert(mt, 2); | |
| 1082 | + | |
| 1083 | + for (int i = 5; i < 25; i++) { | |
| 1084 | + erase_check_insert(mt, i); | |
| 1085 | + for (int j = i; j >= 0; j--) | |
| 1086 | + erase_check_load(mt, j); | |
| 1087 | + } | |
| 1088 | + | |
| 1089 | + erase_check_erase(mt, 14); /*6015 */ | |
| 1090 | + for (int i = 0; i < 25; i++) { | |
| 1091 | + if (i == 14) | |
| 1092 | + check_load(mt, set[i], NULL); | |
| 1093 | + else | |
| 1094 | + erase_check_load(mt, i); | |
| 1095 | + } | |
| 1096 | + erase_check_erase(mt, 16); /*7002 */ | |
| 1097 | + for (int i = 0; i < 25; i++) { | |
| 1098 | + if (i == 16 || i == 14) | |
| 1099 | + check_load(mt, set[i], NULL); | |
| 1100 | + else | |
| 1101 | + erase_check_load(mt, i); | |
| 1102 | + } | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + mt_set_non_kernel(1); | |
| 1106 | + erase_check_erase(mt, 13); /*6012 */ | |
| 1107 | + for (int i = 0; i < 25; i++) { | |
| 1108 | + if (i == 16 || i == 14 || i == 13) | |
| 1109 | + check_load(mt, set[i], NULL); | |
| 1110 | + else | |
| 1111 | + erase_check_load(mt, i); | |
| 1112 | + } | |
| 1113 | + | |
| 1114 | + erase_check_erase(mt, 15); /*7003 */ | |
| 1115 | + for (int i = 0; i < 25; i++) { | |
| 1116 | + if (i <= 16 && i >= 13) | |
| 1117 | + check_load(mt, set[i], NULL); | |
| 1118 | + else | |
| 1119 | + erase_check_load(mt, i); | |
| 1120 | + } | |
| 1121 | + | |
| 1122 | + mt_set_non_kernel(2); | |
| 1123 | + erase_check_erase(mt, 17); /*7008 *should* cause coalesce. */ | |
| 1124 | + for (int i = 0; i < 25; i++) { | |
| 1125 | + if (i <= 17 && i >= 13) | |
| 1126 | + check_load(mt, set[i], NULL); | |
| 1127 | + else | |
| 1128 | + erase_check_load(mt, i); | |
| 1129 | + } | |
| 1130 | + | |
| 1131 | + erase_check_erase(mt, 18); /*7012 */ | |
| 1132 | + for (int i = 0; i < 25; i++) { | |
| 1133 | + if (i <= 18 && i >= 13) | |
| 1134 | + check_load(mt, set[i], NULL); | |
| 1135 | + else | |
| 1136 | + erase_check_load(mt, i); | |
| 1137 | + } | |
| 1138 | + | |
| 1139 | + mt_set_non_kernel(2); | |
| 1140 | + erase_check_erase(mt, 19); /*7015 */ | |
| 1141 | + for (int i = 0; i < 25; i++) { | |
| 1142 | + if (i <= 19 && i >= 13) | |
| 1143 | + check_load(mt, set[i], NULL); | |
| 1144 | + else | |
| 1145 | + erase_check_load(mt, i); | |
| 1146 | + } | |
| 1147 | + | |
| 1148 | + erase_check_erase(mt, 20); /*8003 */ | |
| 1149 | + for (int i = 0; i < 25; i++) { | |
| 1150 | + if (i <= 20 && i >= 13) | |
| 1151 | + check_load(mt, set[i], NULL); | |
| 1152 | + else | |
| 1153 | + erase_check_load(mt, i); | |
| 1154 | + } | |
| 1155 | + | |
| 1156 | + erase_check_erase(mt, 21); /*8002 */ | |
| 1157 | + for (int i = 0; i < 25; i++) { | |
| 1158 | + if (i <= 21 && i >= 13) | |
| 1159 | + check_load(mt, set[i], NULL); | |
| 1160 | + else | |
| 1161 | + erase_check_load(mt, i); | |
| 1162 | + } | |
| 1163 | + | |
| 1164 | + mt_set_non_kernel(2); | |
| 1165 | + erase_check_erase(mt, 22); /*8008 */ | |
| 1166 | + for (int i = 0; i < 25; i++) { | |
| 1167 | + if (i <= 22 && i >= 13) | |
| 1168 | + check_load(mt, set[i], NULL); | |
| 1169 | + else | |
| 1170 | + erase_check_load(mt, i); | |
| 1171 | + } | |
| 1172 | + for (int i = 23; i < 25; i++) | |
| 1173 | + erase_check_erase(mt, i); | |
| 1174 | + | |
| 1175 | + for (int i = 0; i < 25; i++) { | |
| 1176 | + if (i <= 25 && i >= 13) | |
| 1177 | + check_load(mt, set[i], NULL); | |
| 1178 | + else | |
| 1179 | + erase_check_load(mt, i); | |
| 1180 | + } | |
| 1181 | + | |
| 1182 | + /* Shrinking tree test. */ | |
| 1183 | + | |
| 1184 | + for (int i = 13; i < ARRAY_SIZE(set); i++) | |
| 1185 | + erase_check_insert(mt, i); | |
| 1186 | + | |
| 1187 | + mt_set_non_kernel(99); | |
| 1188 | + for (int i = 18; i < ARRAY_SIZE(set); i++) { | |
| 1189 | + erase_check_erase(mt, i); | |
| 1190 | + for (int j = 0; j < ARRAY_SIZE(set); j++) { | |
| 1191 | + if (j < 18 || j > i) | |
| 1192 | + erase_check_load(mt, j); | |
| 1193 | + else | |
| 1194 | + check_load(mt, set[j], NULL); | |
| 1195 | + } | |
| 1196 | + } | |
| 1197 | + mt_set_non_kernel(35); | |
| 1198 | + for (int i = 0; i < 18; i++) { | |
| 1199 | + erase_check_erase(mt, i); | |
| 1200 | + for (int j = 0; j < ARRAY_SIZE(set); j++) { | |
| 1201 | + if (j < 18 && j > i) | |
| 1202 | + erase_check_load(mt, j); | |
| 1203 | + else | |
| 1204 | + check_load(mt, set[j], NULL); | |
| 1205 | + } | |
| 1206 | + } | |
| 1207 | + erase_check_insert(mt, 8); | |
| 1208 | + erase_check_insert(mt, 9); | |
| 1209 | + erase_check_erase(mt, 8); | |
| 1210 | + rcu_unregister_thread(); | |
| 1211 | +} | |
| 1212 | + | |
| 1213 | +#define erase_check_store_range(mt, a, i, ptr) mtree_test_store_range(mt, \ | |
| 1214 | + a[(i)], a[(i + 1)], ptr) | |
| 1215 | +#define STORE 1 | |
| 1216 | +#define SNULL 2 | |
| 1217 | +#define ERASE 3 | |
| 1218 | +#define ec_type_str(x) \ | |
| 1219 | + (((x) == STORE) ? \ | |
| 1220 | + "STORE" : \ | |
| 1221 | + (((x) == SNULL) ? \ | |
| 1222 | + "SNULL" : "ERASE") \ | |
| 1223 | + ) | |
| 1224 | +#define check_erase2_debug 0 | |
| 1225 | +void *mas_next(struct ma_state *mas, unsigned long max); | |
| 1226 | + | |
| 1227 | +/* Calculate the overwritten entries. */ | |
| 1228 | +int mas_ce2_over_count(struct ma_state *mas_start, struct ma_state *mas_end, | |
| 1229 | + void *s_entry, unsigned long s_min, | |
| 1230 | + void *e_entry, unsigned long e_max, | |
| 1231 | + unsigned long *set, int i, bool null_entry) | |
| 1232 | +{ | |
| 1233 | + int count = 0, span = 0; | |
| 1234 | + unsigned long retry = 0; | |
| 1235 | + void *entry; | |
| 1236 | + struct ma_state tmp; | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + /* count slots */ | |
| 1240 | + memcpy(&tmp, mas_start, sizeof(tmp)); | |
| 1241 | + entry = mas_next(&tmp, mas_end->last); | |
| 1242 | + while (entry) { | |
| 1243 | + BUG_ON(retry > 50); /* stop infinite retry on testing. */ | |
| 1244 | + if (xa_is_zero(s_entry)) { | |
| 1245 | + retry++; | |
| 1246 | + continue; | |
| 1247 | + } | |
| 1248 | + count++; | |
| 1249 | + span++; | |
| 1250 | + entry = mas_next(&tmp, mas_end->last); | |
| 1251 | + } | |
| 1252 | + | |
| 1253 | + if (null_entry) { | |
| 1254 | + /* Check splitting end. */ | |
| 1255 | + if (e_entry && (e_max > mas_end->last)) | |
| 1256 | + count--; | |
| 1257 | + | |
| 1258 | + /* check overwrite of entire start */ | |
| 1259 | + if (s_entry && (s_min == mas_start->index)) | |
| 1260 | + count++; | |
| 1261 | + } else { /* !null_entry (store) */ | |
| 1262 | + bool esplit = e_max > mas_end->last; | |
| 1263 | + bool ssplit = s_min != mas_start->index; | |
| 1264 | + | |
| 1265 | + if (s_entry && e_entry) { | |
| 1266 | + if (esplit && ssplit) | |
| 1267 | + count--; | |
| 1268 | + else if (ssplit) | |
| 1269 | + count--; | |
| 1270 | + else if (esplit) { | |
| 1271 | + if (span) | |
| 1272 | + count--; | |
| 1273 | + } | |
| 1274 | + } else if (s_entry && !e_entry) { | |
| 1275 | + if (ssplit) | |
| 1276 | + count--; | |
| 1277 | + } else if (!s_entry && e_entry) { | |
| 1278 | + if (esplit) | |
| 1279 | + count--; | |
| 1280 | + count--; | |
| 1281 | + } else { | |
| 1282 | + count--; | |
| 1283 | + } | |
| 1284 | + } | |
| 1285 | + return count; | |
| 1286 | +} | |
| 1287 | + | |
| 1288 | +/* | |
| 1289 | + * mas_node_walk() - Walk a maple node to offset of the index. | |
| 1290 | + * @mas: The maple state | |
| 1291 | + * @type: The maple node type | |
| 1292 | + * @*range_min: Pointer to store the minimum range of the offset | |
| 1293 | + * @*range_max: Pointer to store the maximum range of the offset | |
| 1294 | + * | |
| 1295 | + * The offset will be stored in the maple state. | |
| 1296 | + * | |
| 1297 | + */ | |
| 1298 | +static inline void mas_node_walk(struct ma_state *mas, struct maple_node *node, | |
| 1299 | + enum maple_type type, unsigned long *range_min, | |
| 1300 | + unsigned long *range_max) | |
| 1301 | + | |
| 1302 | +{ | |
| 1303 | + unsigned long *pivots; | |
| 1304 | + unsigned char count; | |
| 1305 | + unsigned long prev, max; | |
| 1306 | + unsigned char offset; | |
| 1307 | + unsigned long index; | |
| 1308 | + | |
| 1309 | + if (unlikely(ma_is_dense(type))) { | |
| 1310 | + (*range_max) = (*range_min) = mas->index; | |
| 1311 | + if (unlikely(ma_dead_node(node))) | |
| 1312 | + return; | |
| 1313 | + | |
| 1314 | + mas->offset = mas->index = mas->min; | |
| 1315 | + return; | |
| 1316 | + } | |
| 1317 | + | |
| 1318 | + pivots = ma_pivots(node, type); | |
| 1319 | + max = pivots[0]; | |
| 1320 | + if (unlikely(ma_dead_node(node))) | |
| 1321 | + return; | |
| 1322 | + | |
| 1323 | + offset = 0; | |
| 1324 | + prev = mas->min; | |
| 1325 | + index = mas->index; | |
| 1326 | + if (unlikely(index <= max)) | |
| 1327 | + goto offset_zero; | |
| 1328 | + | |
| 1329 | + count = mt_pivots[type]; | |
| 1330 | + while (++offset < count) { | |
| 1331 | + prev = max; | |
| 1332 | + max = pivots[offset]; | |
| 1333 | + if (unlikely(ma_dead_node(node))) | |
| 1334 | + return; | |
| 1335 | + | |
| 1336 | + if (index <= max) | |
| 1337 | + goto offset_found; | |
| 1338 | + else if (unlikely(!max)) | |
| 1339 | + goto mas_max; | |
| 1340 | + } | |
| 1341 | + | |
| 1342 | + prev = max; | |
| 1343 | +mas_max: | |
| 1344 | + max = mas->max; | |
| 1345 | +offset_found: | |
| 1346 | + prev++; | |
| 1347 | +offset_zero: | |
| 1348 | + mas->offset = offset; | |
| 1349 | + if (ma_is_leaf(type)) { | |
| 1350 | + *range_max = max; | |
| 1351 | + *range_min = prev; | |
| 1352 | + } else { | |
| 1353 | + mas->max = max; | |
| 1354 | + mas->min = prev; | |
| 1355 | + } | |
| 1356 | +} | |
| 1357 | + | |
| 1358 | +/* | |
| 1359 | + * mas_descend_walk(): Locates a value and sets the mas->node and slot | |
| 1360 | + * accordingly. range_min and range_max are set to the range which the entry is | |
| 1361 | + * valid. | |
| 1362 | + * @mas: The maple state | |
| 1363 | + * @*range_min: A pointer to store the minimum of the range | |
| 1364 | + * @*range_max: A pointer to store the maximum of the range | |
| 1365 | + * | |
| 1366 | + * Check mas->node is still valid on return of any value. | |
| 1367 | + * | |
| 1368 | + * Return: true if pointing to a valid node and offset. False otherwise. | |
| 1369 | + */ | |
| 1370 | +static inline bool mas_descend_walk(struct ma_state *mas, | |
| 1371 | + unsigned long *range_min, unsigned long *range_max) | |
| 1372 | +{ | |
| 1373 | + struct maple_enode *next; | |
| 1374 | + struct maple_node *node; | |
| 1375 | + enum maple_type type; | |
| 1376 | + | |
| 1377 | + next = mas->node; | |
| 1378 | + while (true) { | |
| 1379 | + node = mte_to_node(next); | |
| 1380 | + type = mte_node_type(next); | |
| 1381 | + mas_node_walk(mas, node, type, range_min, range_max); | |
| 1382 | + next = mas_slot(mas, ma_slots(node, type), mas->offset); | |
| 1383 | + if (unlikely(ma_dead_node(node))) | |
| 1384 | + return false; | |
| 1385 | + | |
| 1386 | + if (unlikely(ma_is_leaf(type))) | |
| 1387 | + return true; | |
| 1388 | + | |
| 1389 | + /* Descend. */ | |
| 1390 | + mas->node = next; | |
| 1391 | + } | |
| 1392 | + return false; | |
| 1393 | +} | |
| 1394 | + | |
| 1395 | +/* | |
| 1396 | + * mas_tree_walk() - Walk to @mas->index and set the range values. | |
| 1397 | + * @mas: The maple state. | |
| 1398 | + * @*range_min: The minimum range to be set. | |
| 1399 | + * @*range_max: The maximum range to be set. | |
| 1400 | + * | |
| 1401 | + * Ranges are only valid if there is a valid entry at @mas->index. | |
| 1402 | + * | |
| 1403 | + * Return: True if a value exists, false otherwise. | |
| 1404 | + */ | |
| 1405 | +static inline bool mas_tree_walk(struct ma_state *mas, unsigned long *range_min, | |
| 1406 | + unsigned long *range_max) | |
| 1407 | +{ | |
| 1408 | + bool ret; | |
| 1409 | + | |
| 1410 | +retry: | |
| 1411 | + ret = false; | |
| 1412 | + mas_start(mas); | |
| 1413 | + if (mas_is_none(mas)) | |
| 1414 | + goto not_found; | |
| 1415 | + | |
| 1416 | + if (mas_is_ptr(mas)) { | |
| 1417 | + *range_min = *range_max = 0; | |
| 1418 | + if (!mas->index) | |
| 1419 | + return true; | |
| 1420 | + | |
| 1421 | + goto not_found; | |
| 1422 | + } | |
| 1423 | + | |
| 1424 | + ret = mas_descend_walk(mas, range_min, range_max); | |
| 1425 | + if (unlikely(mte_dead_node(mas->node))) { | |
| 1426 | + mas->node = MAS_START; | |
| 1427 | + goto retry; | |
| 1428 | + } | |
| 1429 | + | |
| 1430 | + return ret; | |
| 1431 | + | |
| 1432 | +not_found: | |
| 1433 | + mas->offset = MAPLE_NODE_SLOTS; | |
| 1434 | + return false; | |
| 1435 | +} | |
| 1436 | + | |
| 1437 | +static inline void *mas_range_load(struct ma_state *mas, | |
| 1438 | + unsigned long *range_min, unsigned long *range_max) | |
| 1439 | + | |
| 1440 | +{ | |
| 1441 | + void *entry = NULL; | |
| 1442 | + unsigned long index = mas->index; | |
| 1443 | + | |
| 1444 | + if (mas_is_none(mas) || mas_is_paused(mas)) | |
| 1445 | + mas->node = MAS_START; | |
| 1446 | +retry: | |
| 1447 | + if (mas_tree_walk(mas, range_min, range_max)) | |
| 1448 | + if (unlikely(mas->node == MAS_ROOT)) | |
| 1449 | + return mas_root(mas); | |
| 1450 | + | |
| 1451 | + if (likely(mas->offset != MAPLE_NODE_SLOTS)) | |
| 1452 | + entry = mas_get_slot(mas, mas->offset); | |
| 1453 | + | |
| 1454 | + if (mas_dead_node(mas, index)) | |
| 1455 | + goto retry; | |
| 1456 | + | |
| 1457 | + return entry; | |
| 1458 | +} | |
| 1459 | +static noinline void check_erase2_testset(struct maple_tree *mt, | |
| 1460 | + unsigned long *set, unsigned long size) | |
| 1461 | +{ | |
| 1462 | + int entry_count = 0; | |
| 1463 | + int check = 0; | |
| 1464 | + void *foo; | |
| 1465 | + unsigned long addr = 0; | |
| 1466 | + void *s_entry = NULL, *e_entry = NULL; | |
| 1467 | + | |
| 1468 | + MA_STATE(mas, mt, 0, 0); | |
| 1469 | + | |
| 1470 | + for (int i = 0; i < size; i += 3) { | |
| 1471 | + unsigned long s_min, s_max; | |
| 1472 | + unsigned long e_min, e_max; | |
| 1473 | + void *value = NULL; | |
| 1474 | + | |
| 1475 | + MA_STATE(mas_start, mt, set[i+1], set[i+1]); | |
| 1476 | + MA_STATE(mas_end, mt, set[i+2], set[i+2]); | |
| 1477 | + mt_set_non_kernel(127); | |
| 1478 | +#if check_erase2_debug | |
| 1479 | + pr_err("%s: %d %s %lu - %lu\n", __func__, i, | |
| 1480 | + ec_type_str(set[i]), | |
| 1481 | + set[i+1], set[i+2]); | |
| 1482 | +#endif | |
| 1483 | + s_entry = mas_range_load(&mas_start, &s_min, &s_max); | |
| 1484 | + e_entry = mas_range_load(&mas_end, &e_min, &e_max); | |
| 1485 | + | |
| 1486 | + switch (set[i]) { | |
| 1487 | + case SNULL: | |
| 1488 | + if ((s_min == set[i+1]) && (s_max == set[i+2])) { | |
| 1489 | + if (s_entry) | |
| 1490 | + entry_count--; | |
| 1491 | + } else if ((s_min != set[i+1]) && (s_max != set[i+2])) { | |
| 1492 | + entry_count++; | |
| 1493 | + } else if ((mas_start.node != mas_end.node) || | |
| 1494 | + (mas_start.offset != mas_end.offset)) { | |
| 1495 | + entry_count -= | |
| 1496 | + mas_ce2_over_count(&mas_start, &mas_end, | |
| 1497 | + s_entry, s_min, | |
| 1498 | + e_entry, e_max, set, i, | |
| 1499 | + true); | |
| 1500 | + } | |
| 1501 | + | |
| 1502 | + | |
| 1503 | + erase_check_store_range(mt, set, i + 1, value); | |
| 1504 | + break; | |
| 1505 | + case STORE: | |
| 1506 | + value = xa_mk_value(set[i + 1]); | |
| 1507 | + if (mas_start.offset > mt_slot_count(mas_start.node)) { | |
| 1508 | + entry_count++; /* appending an entry. */ | |
| 1509 | + } else if ((s_min == e_min) && (s_max == e_max)) { | |
| 1510 | + if (!entry_count) | |
| 1511 | + entry_count++; | |
| 1512 | + | |
| 1513 | + else if (s_entry) { | |
| 1514 | + if (e_max > mas_end.last) | |
| 1515 | + entry_count++; | |
| 1516 | + | |
| 1517 | + if (s_min < mas_start.index) | |
| 1518 | + entry_count++; | |
| 1519 | + | |
| 1520 | + } else { | |
| 1521 | + entry_count++; | |
| 1522 | + } | |
| 1523 | + } else { | |
| 1524 | + entry_count -= | |
| 1525 | + mas_ce2_over_count(&mas_start, &mas_end, | |
| 1526 | + s_entry, s_min, | |
| 1527 | + e_entry, e_max, set, i, | |
| 1528 | + false); | |
| 1529 | + } | |
| 1530 | + | |
| 1531 | + erase_check_store_range(mt, set, i + 1, value); | |
| 1532 | + break; | |
| 1533 | + case ERASE: | |
| 1534 | + if (!s_entry) | |
| 1535 | + break; | |
| 1536 | + check_erase(mt, set[i+1], xa_mk_value(set[i+1])); | |
| 1537 | + entry_count--; | |
| 1538 | + break; | |
| 1539 | + } | |
| 1540 | + mt_validate(mt); | |
| 1541 | + if (entry_count) | |
| 1542 | + MT_BUG_ON(mt, !mt_height(mt)); | |
| 1543 | +#if check_erase2_debug > 1 | |
| 1544 | + mt_dump(mt); | |
| 1545 | +#endif | |
| 1546 | +#if check_erase2_debug | |
| 1547 | + pr_err("Done\n"); | |
| 1548 | +#endif | |
| 1549 | + | |
| 1550 | + check = 0; | |
| 1551 | + addr = 0; | |
| 1552 | + mt_for_each(mt, foo, addr, ULONG_MAX) { | |
| 1553 | + check++; | |
| 1554 | +#if check_erase2_debug > 2 | |
| 1555 | + pr_err("mt: %lu -> %p (%d)\n", addr+1, foo, check); | |
| 1556 | +#endif | |
| 1557 | + if (check > entry_count) | |
| 1558 | + break; | |
| 1559 | + } | |
| 1560 | + | |
| 1561 | +#if check_erase2_debug > 2 | |
| 1562 | + pr_err("mt_for_each %d and count %d\n", check, entry_count); | |
| 1563 | +#endif | |
| 1564 | + | |
| 1565 | + MT_BUG_ON(mt, check != entry_count); | |
| 1566 | + | |
| 1567 | + check = 0; | |
| 1568 | + addr = 0; | |
| 1569 | + mas_reset(&mas); | |
| 1570 | + mas.index = 0; | |
| 1571 | + rcu_read_lock(); | |
| 1572 | + mas_for_each(&mas, foo, ULONG_MAX) { | |
| 1573 | + if (xa_is_zero(foo)) { | |
| 1574 | + if (addr == mas.index) { | |
| 1575 | + mt_dump(mas.tree); | |
| 1576 | + pr_err("retry failed %lu - %lu\n", | |
| 1577 | + mas.index, mas.last); | |
| 1578 | + MT_BUG_ON(mt, 1); | |
| 1579 | + } | |
| 1580 | + addr = mas.index; | |
| 1581 | + continue; | |
| 1582 | + } | |
| 1583 | +#if check_erase2_debug > 2 | |
| 1584 | + pr_err("mas: %lu -> %p\n", mas.index, foo); | |
| 1585 | +#endif | |
| 1586 | + check++; | |
| 1587 | + if (check > entry_count) | |
| 1588 | + break; | |
| 1589 | + } | |
| 1590 | + rcu_read_unlock(); | |
| 1591 | +#if check_erase2_debug > 2 | |
| 1592 | + pr_err("mas_for_each %d and count %d\n", check, entry_count); | |
| 1593 | + mt_validate(mt); | |
| 1594 | +#endif | |
| 1595 | + | |
| 1596 | + MT_BUG_ON(mt, check != entry_count); | |
| 1597 | + | |
| 1598 | + MT_BUG_ON(mt, mtree_load(mas.tree, 0) != NULL); | |
| 1599 | + } | |
| 1600 | +} | |
| 1601 | + | |
| 1602 | + | |
| 1603 | +/* These tests were pulled from kvm tests. */ | |
| 1604 | +static noinline void check_erase2_sets(struct maple_tree *mt) | |
| 1605 | +{ | |
| 1606 | + void *entry; | |
| 1607 | + unsigned long start = 0; | |
| 1608 | + unsigned long set[] = { | |
| 1609 | +STORE, 140737488347136, 140737488351231, | |
| 1610 | +STORE, 140721266458624, 140737488351231, | |
| 1611 | +ERASE, 140721266458624, 140737488351231, | |
| 1612 | +STORE, 140721266458624, 140721266462719, | |
| 1613 | +STORE, 94735788949504, 94735789121535, | |
| 1614 | +ERASE, 94735788949504, 94735789121535, | |
| 1615 | +STORE, 94735788949504, 94735788965887, | |
| 1616 | +STORE, 94735788965888, 94735789121535, | |
| 1617 | +ERASE, 94735788965888, 94735789121535, | |
| 1618 | +STORE, 94735788965888, 94735789068287, | |
| 1619 | +STORE, 94735789068288, 94735789109247, | |
| 1620 | +STORE, 94735789109248, 94735789121535, | |
| 1621 | +STORE, 140253902692352, 140253902864383, | |
| 1622 | +ERASE, 140253902692352, 140253902864383, | |
| 1623 | +STORE, 140253902692352, 140253902696447, | |
| 1624 | +STORE, 140253902696448, 140253902864383, | |
| 1625 | + }; | |
| 1626 | + unsigned long set2[] = { | |
| 1627 | +STORE, 140737488347136, 140737488351231, | |
| 1628 | +STORE, 140735933583360, 140737488351231, | |
| 1629 | +ERASE, 140735933583360, 140737488351231, | |
| 1630 | +STORE, 140735933583360, 140735933587455, | |
| 1631 | +STORE, 94811003260928, 94811003432959, | |
| 1632 | +ERASE, 94811003260928, 94811003432959, | |
| 1633 | +STORE, 94811003260928, 94811003277311, | |
| 1634 | +STORE, 94811003277312, 94811003432959, | |
| 1635 | +ERASE, 94811003277312, 94811003432959, | |
| 1636 | +STORE, 94811003277312, 94811003379711, | |
| 1637 | +STORE, 94811003379712, 94811003420671, | |
| 1638 | +STORE, 94811003420672, 94811003432959, | |
| 1639 | +STORE, 140277094653952, 140277094825983, | |
| 1640 | +ERASE, 140277094653952, 140277094825983, | |
| 1641 | +STORE, 140277094653952, 140277094658047, | |
| 1642 | +STORE, 140277094658048, 140277094825983, | |
| 1643 | +ERASE, 140277094658048, 140277094825983, | |
| 1644 | +STORE, 140277094658048, 140277094780927, | |
| 1645 | +STORE, 140277094780928, 140277094813695, | |
| 1646 | +STORE, 140277094813696, 140277094821887, | |
| 1647 | +STORE, 140277094821888, 140277094825983, | |
| 1648 | +STORE, 140735933906944, 140735933911039, | |
| 1649 | + }; | |
| 1650 | + unsigned long set3[] = { | |
| 1651 | +STORE, 140737488347136, 140737488351231, | |
| 1652 | +STORE, 140735790264320, 140737488351231, | |
| 1653 | +ERASE, 140735790264320, 140737488351231, | |
| 1654 | +STORE, 140735790264320, 140735790268415, | |
| 1655 | +STORE, 94016597282816, 94016597454847, | |
| 1656 | +ERASE, 94016597282816, 94016597454847, | |
| 1657 | +STORE, 94016597282816, 94016597299199, | |
| 1658 | +STORE, 94016597299200, 94016597454847, | |
| 1659 | +ERASE, 94016597299200, 94016597454847, | |
| 1660 | +STORE, 94016597299200, 94016597401599, | |
| 1661 | +STORE, 94016597401600, 94016597442559, | |
| 1662 | +STORE, 94016597442560, 94016597454847, | |
| 1663 | +STORE, 140496959283200, 140496959455231, | |
| 1664 | +ERASE, 140496959283200, 140496959455231, | |
| 1665 | +STORE, 140496959283200, 140496959287295, | |
| 1666 | +STORE, 140496959287296, 140496959455231, | |
| 1667 | +ERASE, 140496959287296, 140496959455231, | |
| 1668 | +STORE, 140496959287296, 140496959410175, | |
| 1669 | +STORE, 140496959410176, 140496959442943, | |
| 1670 | +STORE, 140496959442944, 140496959451135, | |
| 1671 | +STORE, 140496959451136, 140496959455231, | |
| 1672 | +STORE, 140735791718400, 140735791722495, | |
| 1673 | +STORE, 140735791706112, 140735791718399, | |
| 1674 | +STORE, 47135835713536, 47135835721727, | |
| 1675 | +STORE, 47135835721728, 47135835729919, | |
| 1676 | +STORE, 47135835729920, 47135835893759, | |
| 1677 | +ERASE, 47135835729920, 47135835893759, | |
| 1678 | +STORE, 47135835729920, 47135835742207, | |
| 1679 | +STORE, 47135835742208, 47135835893759, | |
| 1680 | +STORE, 47135835840512, 47135835893759, | |
| 1681 | +STORE, 47135835742208, 47135835840511, | |
| 1682 | +ERASE, 47135835742208, 47135835840511, | |
| 1683 | +STORE, 47135835742208, 47135835840511, | |
| 1684 | +STORE, 47135835885568, 47135835893759, | |
| 1685 | +STORE, 47135835840512, 47135835885567, | |
| 1686 | +ERASE, 47135835840512, 47135835885567, | |
| 1687 | +STORE, 47135835840512, 47135835893759, | |
| 1688 | +ERASE, 47135835840512, 47135835893759, | |
| 1689 | +STORE, 47135835840512, 47135835885567, | |
| 1690 | +STORE, 47135835885568, 47135835893759, | |
| 1691 | + }; | |
| 1692 | + | |
| 1693 | + unsigned long set4[] = { | |
| 1694 | +STORE, 140737488347136, 140737488351231, | |
| 1695 | +STORE, 140728251703296, 140737488351231, | |
| 1696 | +ERASE, 140728251703296, 140737488351231, | |
| 1697 | +STORE, 140728251703296, 140728251707391, | |
| 1698 | +STORE, 94668429205504, 94668429377535, | |
| 1699 | +ERASE, 94668429205504, 94668429377535, | |
| 1700 | +STORE, 94668429205504, 94668429221887, | |
| 1701 | +STORE, 94668429221888, 94668429377535, | |
| 1702 | +ERASE, 94668429221888, 94668429377535, | |
| 1703 | +STORE, 94668429221888, 94668429324287, | |
| 1704 | +STORE, 94668429324288, 94668429365247, | |
| 1705 | +STORE, 94668429365248, 94668429377535, | |
| 1706 | +STORE, 47646523273216, 47646523445247, | |
| 1707 | +ERASE, 47646523273216, 47646523445247, | |
| 1708 | +STORE, 47646523273216, 47646523277311, | |
| 1709 | +STORE, 47646523277312, 47646523445247, | |
| 1710 | +ERASE, 47646523277312, 47646523445247, | |
| 1711 | +STORE, 47646523277312, 47646523400191, | |
| 1712 | + }; | |
| 1713 | + | |
| 1714 | + unsigned long set5[] = { | |
| 1715 | +STORE, 140737488347136, 140737488351231, | |
| 1716 | +STORE, 140726874062848, 140737488351231, | |
| 1717 | +ERASE, 140726874062848, 140737488351231, | |
| 1718 | +STORE, 140726874062848, 140726874066943, | |
| 1719 | +STORE, 94248892870656, 94248893042687, | |
| 1720 | +ERASE, 94248892870656, 94248893042687, | |
| 1721 | +STORE, 94248892870656, 94248892887039, | |
| 1722 | +STORE, 94248892887040, 94248893042687, | |
| 1723 | +ERASE, 94248892887040, 94248893042687, | |
| 1724 | +STORE, 94248892887040, 94248892989439, | |
| 1725 | +STORE, 94248892989440, 94248893030399, | |
| 1726 | +STORE, 94248893030400, 94248893042687, | |
| 1727 | +STORE, 47884786266112, 47884786438143, | |
| 1728 | +ERASE, 47884786266112, 47884786438143, | |
| 1729 | +STORE, 47884786266112, 47884786270207, | |
| 1730 | +STORE, 47884786270208, 47884786438143, | |
| 1731 | +ERASE, 47884786270208, 47884786438143, | |
| 1732 | +STORE, 47884786270208, 47884786393087, | |
| 1733 | +STORE, 47884786393088, 47884786425855, | |
| 1734 | +STORE, 47884786425856, 47884786434047, | |
| 1735 | +STORE, 47884786434048, 47884786438143, | |
| 1736 | +STORE, 140726874513408, 140726874517503, | |
| 1737 | +STORE, 140726874501120, 140726874513407, | |
| 1738 | +STORE, 47884786438144, 47884786446335, | |
| 1739 | +STORE, 47884786446336, 47884786454527, | |
| 1740 | +STORE, 47884786454528, 47884786618367, | |
| 1741 | +ERASE, 47884786454528, 47884786618367, | |
| 1742 | +STORE, 47884786454528, 47884786466815, | |
| 1743 | +STORE, 47884786466816, 47884786618367, | |
| 1744 | +STORE, 47884786565120, 47884786618367, | |
| 1745 | +STORE, 47884786466816, 47884786565119, | |
| 1746 | +ERASE, 47884786466816, 47884786565119, | |
| 1747 | +STORE, 47884786466816, 47884786565119, | |
| 1748 | +STORE, 47884786610176, 47884786618367, | |
| 1749 | +STORE, 47884786565120, 47884786610175, | |
| 1750 | +ERASE, 47884786565120, 47884786610175, | |
| 1751 | +STORE, 47884786565120, 47884786618367, | |
| 1752 | +ERASE, 47884786565120, 47884786618367, | |
| 1753 | +STORE, 47884786565120, 47884786610175, | |
| 1754 | +STORE, 47884786610176, 47884786618367, | |
| 1755 | +ERASE, 47884786610176, 47884786618367, | |
| 1756 | +STORE, 47884786610176, 47884786618367, | |
| 1757 | +STORE, 47884786618368, 47884789669887, | |
| 1758 | +STORE, 47884787163136, 47884789669887, | |
| 1759 | +STORE, 47884786618368, 47884787163135, | |
| 1760 | +ERASE, 47884787163136, 47884789669887, | |
| 1761 | +STORE, 47884787163136, 47884789448703, | |
| 1762 | +STORE, 47884789448704, 47884789669887, | |
| 1763 | +STORE, 47884788858880, 47884789448703, | |
| 1764 | +STORE, 47884787163136, 47884788858879, | |
| 1765 | +ERASE, 47884787163136, 47884788858879, | |
| 1766 | +STORE, 47884787163136, 47884788858879, | |
| 1767 | +STORE, 47884789444608, 47884789448703, | |
| 1768 | +STORE, 47884788858880, 47884789444607, | |
| 1769 | +ERASE, 47884788858880, 47884789444607, | |
| 1770 | +STORE, 47884788858880, 47884789444607, | |
| 1771 | +STORE, 47884789653504, 47884789669887, | |
| 1772 | +STORE, 47884789448704, 47884789653503, | |
| 1773 | +ERASE, 47884789448704, 47884789653503, | |
| 1774 | +STORE, 47884789448704, 47884789653503, | |
| 1775 | +ERASE, 47884789653504, 47884789669887, | |
| 1776 | +STORE, 47884789653504, 47884789669887, | |
| 1777 | +STORE, 47884789669888, 47884791508991, | |
| 1778 | +STORE, 47884789809152, 47884791508991, | |
| 1779 | +STORE, 47884789669888, 47884789809151, | |
| 1780 | +ERASE, 47884789809152, 47884791508991, | |
| 1781 | +STORE, 47884789809152, 47884791468031, | |
| 1782 | +STORE, 47884791468032, 47884791508991, | |
| 1783 | +STORE, 47884791152640, 47884791468031, | |
| 1784 | +STORE, 47884789809152, 47884791152639, | |
| 1785 | +ERASE, 47884789809152, 47884791152639, | |
| 1786 | +STORE, 47884789809152, 47884791152639, | |
| 1787 | +STORE, 47884791463936, 47884791468031, | |
| 1788 | +STORE, 47884791152640, 47884791463935, | |
| 1789 | +ERASE, 47884791152640, 47884791463935, | |
| 1790 | +STORE, 47884791152640, 47884791463935, | |
| 1791 | +STORE, 47884791492608, 47884791508991, | |
| 1792 | +STORE, 47884791468032, 47884791492607, | |
| 1793 | +ERASE, 47884791468032, 47884791492607, | |
| 1794 | +STORE, 47884791468032, 47884791492607, | |
| 1795 | +ERASE, 47884791492608, 47884791508991, | |
| 1796 | +STORE, 47884791492608, 47884791508991, | |
| 1797 | +STORE, 47884791508992, 47884791644159, | |
| 1798 | +ERASE, 47884791508992, 47884791644159, | |
| 1799 | +STORE, 47884791508992, 47884791533567, | |
| 1800 | +STORE, 47884791533568, 47884791644159, | |
| 1801 | +STORE, 47884791595008, 47884791644159, | |
| 1802 | +STORE, 47884791533568, 47884791595007, | |
| 1803 | +ERASE, 47884791533568, 47884791595007, | |
| 1804 | +STORE, 47884791533568, 47884791595007, | |
| 1805 | +STORE, 47884791619584, 47884791644159, | |
| 1806 | +STORE, 47884791595008, 47884791619583, | |
| 1807 | +ERASE, 47884791595008, 47884791619583, | |
| 1808 | +STORE, 47884791595008, 47884791644159, | |
| 1809 | +ERASE, 47884791595008, 47884791644159, | |
| 1810 | +STORE, 47884791595008, 47884791619583, | |
| 1811 | +STORE, 47884791619584, 47884791644159, | |
| 1812 | +STORE, 47884791627776, 47884791644159, | |
| 1813 | +STORE, 47884791619584, 47884791627775, | |
| 1814 | +ERASE, 47884791619584, 47884791627775, | |
| 1815 | +STORE, 47884791619584, 47884791627775, | |
| 1816 | +ERASE, 47884791627776, 47884791644159, | |
| 1817 | +STORE, 47884791627776, 47884791644159, | |
| 1818 | +STORE, 47884791644160, 47884791664639, | |
| 1819 | +ERASE, 47884791644160, 47884791664639, | |
| 1820 | +STORE, 47884791644160, 47884791648255, | |
| 1821 | +STORE, 47884791648256, 47884791664639, | |
| 1822 | +STORE, 47884791652352, 47884791664639, | |
| 1823 | +STORE, 47884791648256, 47884791652351, | |
| 1824 | +ERASE, 47884791648256, 47884791652351, | |
| 1825 | +STORE, 47884791648256, 47884791652351, | |
| 1826 | +STORE, 47884791656448, 47884791664639, | |
| 1827 | +STORE, 47884791652352, 47884791656447, | |
| 1828 | +ERASE, 47884791652352, 47884791656447, | |
| 1829 | +STORE, 47884791652352, 47884791664639, | |
| 1830 | +ERASE, 47884791652352, 47884791664639, | |
| 1831 | +STORE, 47884791652352, 47884791656447, | |
| 1832 | +STORE, 47884791656448, 47884791664639, | |
| 1833 | +ERASE, 47884791656448, 47884791664639, | |
| 1834 | +STORE, 47884791656448, 47884791664639, | |
| 1835 | +STORE, 47884791664640, 47884791672831, | |
| 1836 | +ERASE, 47884791468032, 47884791492607, | |
| 1837 | +STORE, 47884791468032, 47884791484415, | |
| 1838 | +STORE, 47884791484416, 47884791492607, | |
| 1839 | +ERASE, 47884791656448, 47884791664639, | |
| 1840 | +STORE, 47884791656448, 47884791660543, | |
| 1841 | +STORE, 47884791660544, 47884791664639, | |
| 1842 | +ERASE, 47884791619584, 47884791627775, | |
| 1843 | +STORE, 47884791619584, 47884791623679, | |
| 1844 | +STORE, 47884791623680, 47884791627775, | |
| 1845 | + }; | |
| 1846 | + | |
| 1847 | + unsigned long set6[] = { | |
| 1848 | +STORE, 140737488347136, 140737488351231, | |
| 1849 | +STORE, 140722999021568, 140737488351231, | |
| 1850 | +ERASE, 140722999021568, 140737488351231, | |
| 1851 | +STORE, 140722999021568, 140722999025663, | |
| 1852 | +STORE, 94901500268544, 94901500440575, | |
| 1853 | +ERASE, 94901500268544, 94901500440575, | |
| 1854 | +STORE, 94901500268544, 94901500284927, | |
| 1855 | +STORE, 94901500284928, 94901500440575, | |
| 1856 | +ERASE, 94901500284928, 94901500440575, | |
| 1857 | +STORE, 94901500284928, 94901500387327, | |
| 1858 | +STORE, 94901500387328, 94901500428287, | |
| 1859 | +STORE, 94901500428288, 94901500440575, | |
| 1860 | +STORE, 47430426660864, 47430426832895, | |
| 1861 | +ERASE, 47430426660864, 47430426832895, | |
| 1862 | +STORE, 47430426660864, 47430426664959, | |
| 1863 | +STORE, 47430426664960, 47430426832895, | |
| 1864 | +ERASE, 47430426664960, 47430426832895, | |
| 1865 | +STORE, 47430426664960, 47430426787839, | |
| 1866 | +STORE, 47430426787840, 47430426820607, | |
| 1867 | +STORE, 47430426820608, 47430426828799, | |
| 1868 | +STORE, 47430426828800, 47430426832895, | |
| 1869 | +STORE, 140722999115776, 140722999119871, | |
| 1870 | +STORE, 140722999103488, 140722999115775, | |
| 1871 | +STORE, 47430426832896, 47430426841087, | |
| 1872 | +STORE, 47430426841088, 47430426849279, | |
| 1873 | +STORE, 47430426849280, 47430427013119, | |
| 1874 | +ERASE, 47430426849280, 47430427013119, | |
| 1875 | +STORE, 47430426849280, 47430426861567, | |
| 1876 | +STORE, 47430426861568, 47430427013119, | |
| 1877 | +STORE, 47430426959872, 47430427013119, | |
| 1878 | +STORE, 47430426861568, 47430426959871, | |
| 1879 | +ERASE, 47430426861568, 47430426959871, | |
| 1880 | +STORE, 47430426861568, 47430426959871, | |
| 1881 | +STORE, 47430427004928, 47430427013119, | |
| 1882 | +STORE, 47430426959872, 47430427004927, | |
| 1883 | +ERASE, 47430426959872, 47430427004927, | |
| 1884 | +STORE, 47430426959872, 47430427013119, | |
| 1885 | +ERASE, 47430426959872, 47430427013119, | |
| 1886 | +STORE, 47430426959872, 47430427004927, | |
| 1887 | +STORE, 47430427004928, 47430427013119, | |
| 1888 | +ERASE, 47430427004928, 47430427013119, | |
| 1889 | +STORE, 47430427004928, 47430427013119, | |
| 1890 | +STORE, 47430427013120, 47430430064639, | |
| 1891 | +STORE, 47430427557888, 47430430064639, | |
| 1892 | +STORE, 47430427013120, 47430427557887, | |
| 1893 | +ERASE, 47430427557888, 47430430064639, | |
| 1894 | +STORE, 47430427557888, 47430429843455, | |
| 1895 | +STORE, 47430429843456, 47430430064639, | |
| 1896 | +STORE, 47430429253632, 47430429843455, | |
| 1897 | +STORE, 47430427557888, 47430429253631, | |
| 1898 | +ERASE, 47430427557888, 47430429253631, | |
| 1899 | +STORE, 47430427557888, 47430429253631, | |
| 1900 | +STORE, 47430429839360, 47430429843455, | |
| 1901 | +STORE, 47430429253632, 47430429839359, | |
| 1902 | +ERASE, 47430429253632, 47430429839359, | |
| 1903 | +STORE, 47430429253632, 47430429839359, | |
| 1904 | +STORE, 47430430048256, 47430430064639, | |
| 1905 | +STORE, 47430429843456, 47430430048255, | |
| 1906 | +ERASE, 47430429843456, 47430430048255, | |
| 1907 | +STORE, 47430429843456, 47430430048255, | |
| 1908 | +ERASE, 47430430048256, 47430430064639, | |
| 1909 | +STORE, 47430430048256, 47430430064639, | |
| 1910 | +STORE, 47430430064640, 47430431903743, | |
| 1911 | +STORE, 47430430203904, 47430431903743, | |
| 1912 | +STORE, 47430430064640, 47430430203903, | |
| 1913 | +ERASE, 47430430203904, 47430431903743, | |
| 1914 | +STORE, 47430430203904, 47430431862783, | |
| 1915 | +STORE, 47430431862784, 47430431903743, | |
| 1916 | +STORE, 47430431547392, 47430431862783, | |
| 1917 | +STORE, 47430430203904, 47430431547391, | |
| 1918 | +ERASE, 47430430203904, 47430431547391, | |
| 1919 | +STORE, 47430430203904, 47430431547391, | |
| 1920 | +STORE, 47430431858688, 47430431862783, | |
| 1921 | +STORE, 47430431547392, 47430431858687, | |
| 1922 | +ERASE, 47430431547392, 47430431858687, | |
| 1923 | +STORE, 47430431547392, 47430431858687, | |
| 1924 | +STORE, 47430431887360, 47430431903743, | |
| 1925 | +STORE, 47430431862784, 47430431887359, | |
| 1926 | +ERASE, 47430431862784, 47430431887359, | |
| 1927 | +STORE, 47430431862784, 47430431887359, | |
| 1928 | +ERASE, 47430431887360, 47430431903743, | |
| 1929 | +STORE, 47430431887360, 47430431903743, | |
| 1930 | +STORE, 47430431903744, 47430432038911, | |
| 1931 | +ERASE, 47430431903744, 47430432038911, | |
| 1932 | +STORE, 47430431903744, 47430431928319, | |
| 1933 | +STORE, 47430431928320, 47430432038911, | |
| 1934 | +STORE, 47430431989760, 47430432038911, | |
| 1935 | +STORE, 47430431928320, 47430431989759, | |
| 1936 | +ERASE, 47430431928320, 47430431989759, | |
| 1937 | +STORE, 47430431928320, 47430431989759, | |
| 1938 | +STORE, 47430432014336, 47430432038911, | |
| 1939 | +STORE, 47430431989760, 47430432014335, | |
| 1940 | +ERASE, 47430431989760, 47430432014335, | |
| 1941 | +STORE, 47430431989760, 47430432038911, | |
| 1942 | +ERASE, 47430431989760, 47430432038911, | |
| 1943 | +STORE, 47430431989760, 47430432014335, | |
| 1944 | +STORE, 47430432014336, 47430432038911, | |
| 1945 | +STORE, 47430432022528, 47430432038911, | |
| 1946 | +STORE, 47430432014336, 47430432022527, | |
| 1947 | +ERASE, 47430432014336, 47430432022527, | |
| 1948 | +STORE, 47430432014336, 47430432022527, | |
| 1949 | +ERASE, 47430432022528, 47430432038911, | |
| 1950 | +STORE, 47430432022528, 47430432038911, | |
| 1951 | +STORE, 47430432038912, 47430432059391, | |
| 1952 | +ERASE, 47430432038912, 47430432059391, | |
| 1953 | +STORE, 47430432038912, 47430432043007, | |
| 1954 | +STORE, 47430432043008, 47430432059391, | |
| 1955 | +STORE, 47430432047104, 47430432059391, | |
| 1956 | +STORE, 47430432043008, 47430432047103, | |
| 1957 | +ERASE, 47430432043008, 47430432047103, | |
| 1958 | +STORE, 47430432043008, 47430432047103, | |
| 1959 | +STORE, 47430432051200, 47430432059391, | |
| 1960 | +STORE, 47430432047104, 47430432051199, | |
| 1961 | +ERASE, 47430432047104, 47430432051199, | |
| 1962 | +STORE, 47430432047104, 47430432059391, | |
| 1963 | +ERASE, 47430432047104, 47430432059391, | |
| 1964 | +STORE, 47430432047104, 47430432051199, | |
| 1965 | +STORE, 47430432051200, 47430432059391, | |
| 1966 | +ERASE, 47430432051200, 47430432059391, | |
| 1967 | +STORE, 47430432051200, 47430432059391, | |
| 1968 | +STORE, 47430432059392, 47430432067583, | |
| 1969 | +ERASE, 47430431862784, 47430431887359, | |
| 1970 | +STORE, 47430431862784, 47430431879167, | |
| 1971 | +STORE, 47430431879168, 47430431887359, | |
| 1972 | +ERASE, 47430432051200, 47430432059391, | |
| 1973 | +STORE, 47430432051200, 47430432055295, | |
| 1974 | +STORE, 47430432055296, 47430432059391, | |
| 1975 | +ERASE, 47430432014336, 47430432022527, | |
| 1976 | +STORE, 47430432014336, 47430432018431, | |
| 1977 | +STORE, 47430432018432, 47430432022527, | |
| 1978 | + }; | |
| 1979 | + unsigned long set7[] = { | |
| 1980 | +STORE, 140737488347136, 140737488351231, | |
| 1981 | +STORE, 140729808330752, 140737488351231, | |
| 1982 | +ERASE, 140729808330752, 140737488351231, | |
| 1983 | +STORE, 140729808330752, 140729808334847, | |
| 1984 | +STORE, 94629632020480, 94629632192511, | |
| 1985 | +ERASE, 94629632020480, 94629632192511, | |
| 1986 | +STORE, 94629632020480, 94629632036863, | |
| 1987 | +STORE, 94629632036864, 94629632192511, | |
| 1988 | +ERASE, 94629632036864, 94629632192511, | |
| 1989 | +STORE, 94629632036864, 94629632139263, | |
| 1990 | +STORE, 94629632139264, 94629632180223, | |
| 1991 | +STORE, 94629632180224, 94629632192511, | |
| 1992 | +STORE, 47439981776896, 47439981948927, | |
| 1993 | +ERASE, 47439981776896, 47439981948927, | |
| 1994 | +STORE, 47439981776896, 47439981780991, | |
| 1995 | +STORE, 47439981780992, 47439981948927, | |
| 1996 | +ERASE, 47439981780992, 47439981948927, | |
| 1997 | +STORE, 47439981780992, 47439981903871, | |
| 1998 | +STORE, 47439981903872, 47439981936639, | |
| 1999 | +STORE, 47439981936640, 47439981944831, | |
| 2000 | +STORE, 47439981944832, 47439981948927, | |
| 2001 | +STORE, 140729808474112, 140729808478207, | |
| 2002 | +STORE, 140729808461824, 140729808474111, | |
| 2003 | +STORE, 47439981948928, 47439981957119, | |
| 2004 | +STORE, 47439981957120, 47439981965311, | |
| 2005 | +STORE, 47439981965312, 47439982129151, | |
| 2006 | +ERASE, 47439981965312, 47439982129151, | |
| 2007 | +STORE, 47439981965312, 47439981977599, | |
| 2008 | +STORE, 47439981977600, 47439982129151, | |
| 2009 | +STORE, 47439982075904, 47439982129151, | |
| 2010 | +STORE, 47439981977600, 47439982075903, | |
| 2011 | +ERASE, 47439981977600, 47439982075903, | |
| 2012 | +STORE, 47439981977600, 47439982075903, | |
| 2013 | +STORE, 47439982120960, 47439982129151, | |
| 2014 | +STORE, 47439982075904, 47439982120959, | |
| 2015 | +ERASE, 47439982075904, 47439982120959, | |
| 2016 | +STORE, 47439982075904, 47439982129151, | |
| 2017 | +ERASE, 47439982075904, 47439982129151, | |
| 2018 | +STORE, 47439982075904, 47439982120959, | |
| 2019 | +STORE, 47439982120960, 47439982129151, | |
| 2020 | +ERASE, 47439982120960, 47439982129151, | |
| 2021 | +STORE, 47439982120960, 47439982129151, | |
| 2022 | +STORE, 47439982129152, 47439985180671, | |
| 2023 | +STORE, 47439982673920, 47439985180671, | |
| 2024 | +STORE, 47439982129152, 47439982673919, | |
| 2025 | +ERASE, 47439982673920, 47439985180671, | |
| 2026 | +STORE, 47439982673920, 47439984959487, | |
| 2027 | +STORE, 47439984959488, 47439985180671, | |
| 2028 | +STORE, 47439984369664, 47439984959487, | |
| 2029 | +STORE, 47439982673920, 47439984369663, | |
| 2030 | +ERASE, 47439982673920, 47439984369663, | |
| 2031 | +STORE, 47439982673920, 47439984369663, | |
| 2032 | +STORE, 47439984955392, 47439984959487, | |
| 2033 | +STORE, 47439984369664, 47439984955391, | |
| 2034 | +ERASE, 47439984369664, 47439984955391, | |
| 2035 | +STORE, 47439984369664, 47439984955391, | |
| 2036 | +STORE, 47439985164288, 47439985180671, | |
| 2037 | +STORE, 47439984959488, 47439985164287, | |
| 2038 | +ERASE, 47439984959488, 47439985164287, | |
| 2039 | +STORE, 47439984959488, 47439985164287, | |
| 2040 | +ERASE, 47439985164288, 47439985180671, | |
| 2041 | +STORE, 47439985164288, 47439985180671, | |
| 2042 | +STORE, 47439985180672, 47439987019775, | |
| 2043 | +STORE, 47439985319936, 47439987019775, | |
| 2044 | +STORE, 47439985180672, 47439985319935, | |
| 2045 | +ERASE, 47439985319936, 47439987019775, | |
| 2046 | +STORE, 47439985319936, 47439986978815, | |
| 2047 | +STORE, 47439986978816, 47439987019775, | |
| 2048 | +STORE, 47439986663424, 47439986978815, | |
| 2049 | +STORE, 47439985319936, 47439986663423, | |
| 2050 | +ERASE, 47439985319936, 47439986663423, | |
| 2051 | +STORE, 47439985319936, 47439986663423, | |
| 2052 | +STORE, 47439986974720, 47439986978815, | |
| 2053 | +STORE, 47439986663424, 47439986974719, | |
| 2054 | +ERASE, 47439986663424, 47439986974719, | |
| 2055 | +STORE, 47439986663424, 47439986974719, | |
| 2056 | +STORE, 47439987003392, 47439987019775, | |
| 2057 | +STORE, 47439986978816, 47439987003391, | |
| 2058 | +ERASE, 47439986978816, 47439987003391, | |
| 2059 | +STORE, 47439986978816, 47439987003391, | |
| 2060 | +ERASE, 47439987003392, 47439987019775, | |
| 2061 | +STORE, 47439987003392, 47439987019775, | |
| 2062 | +STORE, 47439987019776, 47439987154943, | |
| 2063 | +ERASE, 47439987019776, 47439987154943, | |
| 2064 | +STORE, 47439987019776, 47439987044351, | |
| 2065 | +STORE, 47439987044352, 47439987154943, | |
| 2066 | +STORE, 47439987105792, 47439987154943, | |
| 2067 | +STORE, 47439987044352, 47439987105791, | |
| 2068 | +ERASE, 47439987044352, 47439987105791, | |
| 2069 | +STORE, 47439987044352, 47439987105791, | |
| 2070 | +STORE, 47439987130368, 47439987154943, | |
| 2071 | +STORE, 47439987105792, 47439987130367, | |
| 2072 | +ERASE, 47439987105792, 47439987130367, | |
| 2073 | +STORE, 47439987105792, 47439987154943, | |
| 2074 | +ERASE, 47439987105792, 47439987154943, | |
| 2075 | +STORE, 47439987105792, 47439987130367, | |
| 2076 | +STORE, 47439987130368, 47439987154943, | |
| 2077 | +STORE, 47439987138560, 47439987154943, | |
| 2078 | +STORE, 47439987130368, 47439987138559, | |
| 2079 | +ERASE, 47439987130368, 47439987138559, | |
| 2080 | +STORE, 47439987130368, 47439987138559, | |
| 2081 | +ERASE, 47439987138560, 47439987154943, | |
| 2082 | +STORE, 47439987138560, 47439987154943, | |
| 2083 | +STORE, 47439987154944, 47439987175423, | |
| 2084 | +ERASE, 47439987154944, 47439987175423, | |
| 2085 | +STORE, 47439987154944, 47439987159039, | |
| 2086 | +STORE, 47439987159040, 47439987175423, | |
| 2087 | +STORE, 47439987163136, 47439987175423, | |
| 2088 | +STORE, 47439987159040, 47439987163135, | |
| 2089 | +ERASE, 47439987159040, 47439987163135, | |
| 2090 | +STORE, 47439987159040, 47439987163135, | |
| 2091 | +STORE, 47439987167232, 47439987175423, | |
| 2092 | +STORE, 47439987163136, 47439987167231, | |
| 2093 | +ERASE, 47439987163136, 47439987167231, | |
| 2094 | +STORE, 47439987163136, 47439987175423, | |
| 2095 | +ERASE, 47439987163136, 47439987175423, | |
| 2096 | +STORE, 47439987163136, 47439987167231, | |
| 2097 | +STORE, 47439987167232, 47439987175423, | |
| 2098 | +ERASE, 47439987167232, 47439987175423, | |
| 2099 | +STORE, 47439987167232, 47439987175423, | |
| 2100 | +STORE, 47439987175424, 47439987183615, | |
| 2101 | +ERASE, 47439986978816, 47439987003391, | |
| 2102 | +STORE, 47439986978816, 47439986995199, | |
| 2103 | +STORE, 47439986995200, 47439987003391, | |
| 2104 | +ERASE, 47439987167232, 47439987175423, | |
| 2105 | +STORE, 47439987167232, 47439987171327, | |
| 2106 | +STORE, 47439987171328, 47439987175423, | |
| 2107 | +ERASE, 47439987130368, 47439987138559, | |
| 2108 | +STORE, 47439987130368, 47439987134463, | |
| 2109 | +STORE, 47439987134464, 47439987138559, | |
| 2110 | + }; | |
| 2111 | + unsigned long set8[] = { | |
| 2112 | +STORE, 140737488347136, 140737488351231, | |
| 2113 | +STORE, 140722482974720, 140737488351231, | |
| 2114 | +ERASE, 140722482974720, 140737488351231, | |
| 2115 | +STORE, 140722482974720, 140722482978815, | |
| 2116 | +STORE, 94121505034240, 94121505206271, | |
| 2117 | +ERASE, 94121505034240, 94121505206271, | |
| 2118 | +STORE, 94121505034240, 94121505050623, | |
| 2119 | +STORE, 94121505050624, 94121505206271, | |
| 2120 | +ERASE, 94121505050624, 94121505206271, | |
| 2121 | +STORE, 94121505050624, 94121505153023, | |
| 2122 | +STORE, 94121505153024, 94121505193983, | |
| 2123 | +STORE, 94121505193984, 94121505206271, | |
| 2124 | +STORE, 47708483284992, 47708483457023, | |
| 2125 | +ERASE, 47708483284992, 47708483457023, | |
| 2126 | +STORE, 47708483284992, 47708483289087, | |
| 2127 | +STORE, 47708483289088, 47708483457023, | |
| 2128 | +ERASE, 47708483289088, 47708483457023, | |
| 2129 | +STORE, 47708483289088, 47708483411967, | |
| 2130 | +STORE, 47708483411968, 47708483444735, | |
| 2131 | +STORE, 47708483444736, 47708483452927, | |
| 2132 | +STORE, 47708483452928, 47708483457023, | |
| 2133 | +STORE, 140722483142656, 140722483146751, | |
| 2134 | +STORE, 140722483130368, 140722483142655, | |
| 2135 | +STORE, 47708483457024, 47708483465215, | |
| 2136 | +STORE, 47708483465216, 47708483473407, | |
| 2137 | +STORE, 47708483473408, 47708483637247, | |
| 2138 | +ERASE, 47708483473408, 47708483637247, | |
| 2139 | +STORE, 47708483473408, 47708483485695, | |
| 2140 | +STORE, 47708483485696, 47708483637247, | |
| 2141 | +STORE, 47708483584000, 47708483637247, | |
| 2142 | +STORE, 47708483485696, 47708483583999, | |
| 2143 | +ERASE, 47708483485696, 47708483583999, | |
| 2144 | +STORE, 47708483485696, 47708483583999, | |
| 2145 | +STORE, 47708483629056, 47708483637247, | |
| 2146 | +STORE, 47708483584000, 47708483629055, | |
| 2147 | +ERASE, 47708483584000, 47708483629055, | |
| 2148 | +STORE, 47708483584000, 47708483637247, | |
| 2149 | +ERASE, 47708483584000, 47708483637247, | |
| 2150 | +STORE, 47708483584000, 47708483629055, | |
| 2151 | +STORE, 47708483629056, 47708483637247, | |
| 2152 | +ERASE, 47708483629056, 47708483637247, | |
| 2153 | +STORE, 47708483629056, 47708483637247, | |
| 2154 | +STORE, 47708483637248, 47708486688767, | |
| 2155 | +STORE, 47708484182016, 47708486688767, | |
| 2156 | +STORE, 47708483637248, 47708484182015, | |
| 2157 | +ERASE, 47708484182016, 47708486688767, | |
| 2158 | +STORE, 47708484182016, 47708486467583, | |
| 2159 | +STORE, 47708486467584, 47708486688767, | |
| 2160 | +STORE, 47708485877760, 47708486467583, | |
| 2161 | +STORE, 47708484182016, 47708485877759, | |
| 2162 | +ERASE, 47708484182016, 47708485877759, | |
| 2163 | +STORE, 47708484182016, 47708485877759, | |
| 2164 | +STORE, 47708486463488, 47708486467583, | |
| 2165 | +STORE, 47708485877760, 47708486463487, | |
| 2166 | +ERASE, 47708485877760, 47708486463487, | |
| 2167 | +STORE, 47708485877760, 47708486463487, | |
| 2168 | +STORE, 47708486672384, 47708486688767, | |
| 2169 | +STORE, 47708486467584, 47708486672383, | |
| 2170 | +ERASE, 47708486467584, 47708486672383, | |
| 2171 | +STORE, 47708486467584, 47708486672383, | |
| 2172 | +ERASE, 47708486672384, 47708486688767, | |
| 2173 | +STORE, 47708486672384, 47708486688767, | |
| 2174 | +STORE, 47708486688768, 47708488527871, | |
| 2175 | +STORE, 47708486828032, 47708488527871, | |
| 2176 | +STORE, 47708486688768, 47708486828031, | |
| 2177 | +ERASE, 47708486828032, 47708488527871, | |
| 2178 | +STORE, 47708486828032, 47708488486911, | |
| 2179 | +STORE, 47708488486912, 47708488527871, | |
| 2180 | +STORE, 47708488171520, 47708488486911, | |
| 2181 | +STORE, 47708486828032, 47708488171519, | |
| 2182 | +ERASE, 47708486828032, 47708488171519, | |
| 2183 | +STORE, 47708486828032, 47708488171519, | |
| 2184 | +STORE, 47708488482816, 47708488486911, | |
| 2185 | +STORE, 47708488171520, 47708488482815, | |
| 2186 | +ERASE, 47708488171520, 47708488482815, | |
| 2187 | +STORE, 47708488171520, 47708488482815, | |
| 2188 | +STORE, 47708488511488, 47708488527871, | |
| 2189 | +STORE, 47708488486912, 47708488511487, | |
| 2190 | +ERASE, 47708488486912, 47708488511487, | |
| 2191 | +STORE, 47708488486912, 47708488511487, | |
| 2192 | +ERASE, 47708488511488, 47708488527871, | |
| 2193 | +STORE, 47708488511488, 47708488527871, | |
36114 more lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 520 lines aren’t shown here.
This diff is large: its 134 lines aren’t shown here.
This diff is large: its 447 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 111 lines aren’t shown here.
This diff is large: its 343 lines aren’t shown here.
This diff is large: its 392 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 223 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 247 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 369 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 541 lines aren’t shown here.
This diff is large: its 1626 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 256 lines aren’t shown here.
This diff is large: its 172 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 292 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 239 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 137 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 169 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 2018 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 451 lines aren’t shown here.
This diff is large: its 385 lines aren’t shown here.
This diff is large: its 236 lines aren’t shown here.
This diff is large: its 308 lines aren’t shown here.
This diff is large: its 210 lines aren’t shown here.
This diff is large: its 582 lines aren’t shown here.
This diff is large: its 220 lines aren’t shown here.
This diff is large: its 299 lines aren’t shown here.
This diff is large: its 1066 lines aren’t shown here.
This diff is large: its 155 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 607 lines aren’t shown here.
This diff is large: its 339 lines aren’t shown here.
This diff is large: its 733 lines aren’t shown here.
This diff is large: its 672 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 205 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 994 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 3057 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 462 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 613 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 206 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 303 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 326 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 1082 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 135 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 267 lines aren’t shown here.
This diff is large: its 371 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 4138 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 3 lines aren’t shown here.
This diff is large: its 210 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 185 lines aren’t shown here.
This diff is large: its 1625 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 477 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.