Pull MM updates from Andrew Morton: "Many singleton patches against the MM code. The patch series which are included in this merge do the following: - Kemeng Shi has contributed some compation maintenance work in the series 'Fixes and cleanups to compaction' - Joel Fernandes has a patchset ('Optimize mremap during mutual alignment within PMD') which fixes an obscure issue with mremap()'s pagetable handling during a subsequent exec(), based upon an implementation which Linus suggested - More DAMON/DAMOS maintenance and feature work from SeongJae Park i the following patch series: mm/damon: misc fixups for documents, comments and its tracepoint mm/damon: add a tracepoint for damos apply target regions mm/damon: provide pseudo-moving sum based access rate mm/damon: implement DAMOS apply intervals mm/damon/core-test: Fix memory leaks in core-test mm/damon/sysfs-schemes: Do DAMOS tried regions update for only one apply interval - In the series 'Do not try to access unaccepted memory' Adrian Hunter provides some fixups for the recently-added 'unaccepted memory' feature. To increase the feature's checking coverage. 'Plug a few gaps where RAM is exposed without checking if it is unaccepted memory' - In the series 'cleanups for lockless slab shrink' Qi Zheng has done some maintenance work which is preparation for the lockless slab shrinking code - Qi Zheng has redone the earlier (and reverted) attempt to make slab shrinking lockless in the series 'use refcount+RCU method to implement lockless slab shrink' - David Hildenbrand contributes some maintenance work for the rmap code in the series 'Anon rmap cleanups' - Kefeng Wang does more folio conversions and some maintenance work in the migration code. Series 'mm: migrate: more folio conversion and unification' - Matthew Wilcox has fixed an issue in the buffer_head code which was causing long stalls under some heavy memory/IO loads. Some cleanups were added on the way. Series 'Add and use bdev_getblk()' - In the series 'Use nth_page() in place of direct struct page manipulation' Zi Yan has fixed a potential issue with the direct manipulation of hugetlb page frames - In the series 'mm: hugetlb: Skip initialization of gigantic tail struct pages if freed by HVO' has improved our handling of gigantic pages in the hugetlb vmmemmep optimizaton code. This provides significant boot time improvements when significant amounts of gigantic pages are in use - Matthew Wilcox has sent the series 'Small hugetlb cleanups' - code rationalization and folio conversions in the hugetlb code - Yin Fengwei has improved mlock()'s handling of large folios in the series 'support large folio for mlock' - In the series 'Expose swapcache stat for memcg v1' Liu Shixin has added statistics for memcg v1 users which are available (and useful) under memcg v2 - Florent Revest has enhanced the MDWE (Memory-Deny-Write-Executable) prctl so that userspace may direct the kernel to not automatically propagate the denial to child processes. The series is named 'MDWE without inheritance' - Kefeng Wang has provided the series 'mm: convert numa balancing functions to use a folio' which does what it says - In the series 'mm/ksm: add fork-exec support for prctl' Stefan Roesch makes is possible for a process to propagate KSM treatment across exec() - Huang Ying has enhanced memory tiering's calculation of memory distances. This is used to permit the dax/kmem driver to use 'high bandwidth memory' in addition to Optane Data Center Persistent Memory Modules (DCPMM). The series is named 'memory tiering: calculate abstract distance based on ACPI HMAT' - In the series 'Smart scanning mode for KSM' Stefan Roesch has optimized KSM by teaching it to retain and use some historical information from previous scans - Yosry Ahmed has fixed some inconsistencies in memcg statistics in the series 'mm: memcg: fix tracking of pending stats updates values' - In the series 'Implement IOCTL to get and optionally clear info about PTEs' Peter Xu has added an ioctl to /proc/<pid>/pagemap which permits us to atomically read-then-clear page softdirty state. This is mainly used by CRIU - Hugh Dickins contributed the series 'shmem,tmpfs: general maintenance', a bunch of relatively minor maintenance tweaks to this code - Matthew Wilcox has increased the use of the VMA lock over file-backed page faults in the series 'Handle more faults under the VMA lock'. Some rationalizations of the fault path became possible as a result - In the series 'mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap()' David Hildenbrand has implemented some cleanups and folio conversions - In the series 'various improvements to the GUP interface' Lorenzo Stoakes has simplified and improved the GUP interface with an eye to providing groundwork for future improvements - Andrey Konovalov has sent along the series 'kasan: assorted fixes and improvements' which does those things - Some page allocator maintenance work from Kemeng Shi in the series 'Two minor cleanups to break_down_buddy_pages' - In thes series 'New selftest for mm' Breno Leitao has developed another MM self test which tickles a race we had between madvise() and page faults - In the series 'Add folio_end_read' Matthew Wilcox provides cleanups and an optimization to the core pagecache code - Nhat Pham has added memcg accounting for hugetlb memory in the series 'hugetlb memcg accounting' - Cleanups and rationalizations to the pagemap code from Lorenzo Stoakes, in the series 'Abstract vma_merge() and split_vma()' - Audra Mitchell has fixed issues in the procfs page_owner code's new timestamping feature which was causing some misbehaviours. In the series 'Fix page_owner's use of free timestamps' - Lorenzo Stoakes has fixed the handling of new mappings of sealed files in the series 'permit write-sealed memfd read-only shared mappings' - Mike Kravetz has optimized the hugetlb vmemmap optimization in the series 'Batch hugetlb vmemmap modification operations' - Some buffer_head folio conversions and cleanups from Matthew Wilcox in the series 'Finish the create_empty_buffers() transition' - As a page allocator performance optimization Huang Ying has added automatic tuning to the allocator's per-cpu-pages feature, in the series 'mm: PCP high auto-tuning' - Roman Gushchin has contributed the patchset 'mm: improve performance of accounted kernel memory allocations' which improves their performance by ~30% as measured by a micro-benchmark - folio conversions from Kefeng Wang in the series 'mm: convert page cpupid functions to folios' - Some kmemleak fixups in Liu Shixin's series 'Some bugfix about kmemleak' - Qi Zheng has improved our handling of memoryless nodes by keeping them off the allocation fallback list. This is done in the series 'handle memoryless nodes more appropriately' - khugepaged conversions from Vishal Moola in the series 'Some khugepaged folio conversions'" [ bcachefs conflicts with the dynamically allocated shrinkers have been resolved as per Stephen Rothwell in https://lore.kernel.org/all/20230913093553.4290421e@canb.auug.org.au/ with help from Qi Zheng. The clone3 test filtering conflict was half-arsed by yours truly ] * tag 'mm-stable-2023-11-01-14-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (406 commits) mm/damon/sysfs: update monitoring target regions for online input commit mm/damon/sysfs: remove requested targets when online-commit inputs selftests: add a sanity check for zswap Documentation: maple_tree: fix word spelling error mm/vmalloc: fix the unchecked dereference warning in vread_iter() zswap: export compression failure stats Documentation: ubsan: drop "the" from article title mempolicy: migration attempt to match interleave nodes mempolicy: mmap_lock is not needed while migrating folios mempolicy: alloc_pages_mpol() for NUMA policy without vma mm: add page_rmappable_folio() wrapper mempolicy: remove confusing MPOL_MF_LAZY dead code mempolicy: mpol_shared_policy_init() without pseudo-vma mempolicy trivia: use pgoff_t in shared mempolicy tree mempolicy trivia: slightly more consistent naming mempolicy trivia: delete those ancient pr_debug()s mempolicy: fix migrate_pages(2) syscall return nr_failed kernfs: drop shared NUMA mempolicy hooks hugetlbfs: drop shared NUMA mempolicy pretence mm/damon/sysfs-test: add a unit test for damon_sysfs_set_targets() ...
Unified · Split
| @@ -151,6 +151,13 @@ Contact: SeongJae Park <sj@kernel.org> · 150 unmodified lines | ||
| 151 | 151 | Description: Writing to and reading from this file sets and gets the action |
| 152 | 152 | of the scheme. |
| 153 | 153 | |
| 154 | +What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/apply_interval_us | |
| 155 | +Date: Sep 2023 | |
| 156 | +Contact: SeongJae Park <sj@kernel.org> | |
| 157 | +Description: Writing a value to this file sets the action apply interval of | |
| 158 | + the scheme in microseconds. Reading this file returns the | |
| 159 | + value. | |
| 160 | + | |
| 154 | 161 | What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/access_pattern/sz/min |
| 155 | 162 | Date: Mar 2022 |
| 156 | 163 | Contact: SeongJae Park <sj@kernel.org> |
| @@ -551,6 +551,7 @@ memory.stat file includes following statistics: · 550 unmodified lines | ||
| 551 | 551 | event happens each time a page is unaccounted from the |
| 552 | 552 | cgroup. |
| 553 | 553 | swap # of bytes of swap usage |
| 554 | + swapcached # of bytes of swap cached in memory | |
| 554 | 555 | dirty # of bytes that are waiting to get written back to the disk. |
| 555 | 556 | writeback # of bytes of file/anon cache that are queued for syncing to |
| 556 | 557 | disk. |
| @@ -210,6 +210,35 @@ cgroup v2 currently supports the following mount options. · 209 unmodified lines | ||
| 210 | 210 | relying on the original semantics (e.g. specifying bogusly |
| 211 | 211 | high 'bypass' protection values at higher tree levels). |
| 212 | 212 | |
| 213 | + memory_hugetlb_accounting | |
| 214 | + Count HugeTLB memory usage towards the cgroup's overall | |
| 215 | + memory usage for the memory controller (for the purpose of | |
| 216 | + statistics reporting and memory protetion). This is a new | |
| 217 | + behavior that could regress existing setups, so it must be | |
| 218 | + explicitly opted in with this mount option. | |
| 219 | + | |
| 220 | + A few caveats to keep in mind: | |
| 221 | + | |
| 222 | + * There is no HugeTLB pool management involved in the memory | |
| 223 | + controller. The pre-allocated pool does not belong to anyone. | |
| 224 | + Specifically, when a new HugeTLB folio is allocated to | |
| 225 | + the pool, it is not accounted for from the perspective of the | |
| 226 | + memory controller. It is only charged to a cgroup when it is | |
| 227 | + actually used (for e.g at page fault time). Host memory | |
| 228 | + overcommit management has to consider this when configuring | |
| 229 | + hard limits. In general, HugeTLB pool management should be | |
| 230 | + done via other mechanisms (such as the HugeTLB controller). | |
| 231 | + * Failure to charge a HugeTLB folio to the memory controller | |
| 232 | + results in SIGBUS. This could happen even if the HugeTLB pool | |
| 233 | + still has pages available (but the cgroup limit is hit and | |
| 234 | + reclaim attempt fails). | |
| 235 | + * Charging HugeTLB memory towards the memory controller affects | |
| 236 | + memory protection and reclaim dynamics. Any userspace tuning | |
| 237 | + (of low, min limits for e.g) needs to take this into account. | |
| 238 | + * HugeTLB pages utilized while this option is not selected | |
| 239 | + will not be tracked by the memory controller (even if cgroup | |
| 240 | + v2 is remounted later on). | |
| 241 | + | |
| 213 | 242 | |
| 214 | 243 | Organizing Processes and Threads |
| 215 | 244 | -------------------------------- |
| @@ -1539,6 +1568,15 @@ PAGE_SIZE multiple when read back. · 1323 unmodified lines | ||
| 1539 | 1568 | collapsing an existing range of pages. This counter is not |
| 1540 | 1569 | present when CONFIG_TRANSPARENT_HUGEPAGE is not set. |
| 1541 | 1570 | |
| 1571 | + thp_swpout (npn) | |
| 1572 | + Number of transparent hugepages which are swapout in one piece | |
| 1573 | + without splitting. | |
| 1574 | + | |
| 1575 | + thp_swpout_fallback (npn) | |
| 1576 | + Number of transparent hugepages which were split before swapout. | |
| 1577 | + Usually because failed to allocate some continuous swap space | |
| 1578 | + for the huge page. | |
| 1579 | + | |
| 1542 | 1580 | memory.numa_stat |
| 1543 | 1581 | A read-only nested-keyed file which exists on non-root cgroups. |
| 1544 | 1582 | |
| @@ -20,18 +20,18 @@ DAMON provides below interfaces for different users. · 19 unmodified lines | ||
| 20 | 20 | you can write and use your personalized DAMON sysfs wrapper programs that |
| 21 | 21 | reads/writes the sysfs files instead of you. The `DAMON user space tool |
| 22 | 22 | <https://github.com/awslabs/damo>`_ is one example of such programs. |
| 23 | −- *debugfs interface. (DEPRECATED!)* | |
| 24 | − :ref:`This <debugfs_interface>` is almost identical to :ref:`sysfs interface | |
| 25 | − <sysfs_interface>`. This is deprecated, so users should move to the | |
| 26 | − :ref:`sysfs interface <sysfs_interface>`. If you depend on this and cannot | |
| 27 | − move, please report your usecase to damon@lists.linux.dev and | |
| 28 | − linux-mm@kvack.org. | |
| 29 | 23 | - *Kernel Space Programming Interface.* |
| 30 | 24 | :doc:`This </mm/damon/api>` is for kernel space programmers. Using this, |
| 31 | 25 | users can utilize every feature of DAMON most flexibly and efficiently by |
| 32 | 26 | writing kernel space DAMON application programs for you. You can even extend |
| 33 | 27 | DAMON for various address spaces. For detail, please refer to the interface |
| 34 | 28 | :doc:`document </mm/damon/api>`. |
| 29 | +- *debugfs interface. (DEPRECATED!)* | |
| 30 | + :ref:`This <debugfs_interface>` is almost identical to :ref:`sysfs interface | |
| 31 | + <sysfs_interface>`. This is deprecated, so users should move to the | |
| 32 | + :ref:`sysfs interface <sysfs_interface>`. If you depend on this and cannot | |
| 33 | + move, please report your usecase to damon@lists.linux.dev and | |
| 34 | + linux-mm@kvack.org. | |
| 35 | 35 | |
| 36 | 36 | .. _sysfs_interface: |
| 37 | 37 | |
| @@ -76,7 +76,7 @@ comma (","). :: · 38 unmodified lines | ||
| 76 | 76 | │ │ │ │ │ │ │ │ ... |
| 77 | 77 | │ │ │ │ │ │ ... |
| 78 | 78 | │ │ │ │ │ schemes/nr_schemes |
| 79 | − │ │ │ │ │ │ 0/action | |
| 79 | + │ │ │ │ │ │ 0/action,apply_interval_us | |
| 80 | 80 | │ │ │ │ │ │ │ access_pattern/ |
| 81 | 81 | │ │ │ │ │ │ │ │ sz/min,max |
| 82 | 82 | │ │ │ │ │ │ │ │ nr_accesses/min,max |
| @@ -105,14 +105,12 @@ having the root permission could use this directory. · 22 unmodified lines | ||
| 105 | 105 | kdamonds/ |
| 106 | 106 | --------- |
| 107 | 107 | |
| 108 | −The monitoring-related information including request specifications and results | |
| 109 | −are called DAMON context. DAMON executes each context with a kernel thread | |
| 110 | −called kdamond, and multiple kdamonds could run in parallel. | |
| 111 | − | |
| 112 | 108 | Under the ``admin`` directory, one directory, ``kdamonds``, which has files for |
| 113 | −controlling the kdamonds exist. In the beginning, this directory has only one | |
| 114 | −file, ``nr_kdamonds``. Writing a number (``N``) to the file creates the number | |
| 115 | −of child directories named ``0`` to ``N-1``. Each directory represents each | |
| 109 | +controlling the kdamonds (refer to | |
| 110 | +:ref:`design <damon_design_execution_model_and_data_structures>` for more | |
| 111 | +details) exists. In the beginning, this directory has only one file, | |
| 112 | +``nr_kdamonds``. Writing a number (``N``) to the file creates the number of | |
| 113 | +child directories named ``0`` to ``N-1``. Each directory represents each | |
| 116 | 114 | kdamond. |
| 117 | 115 | |
| 118 | 116 | kdamonds/<N>/ |
| @@ -150,9 +148,10 @@ kdamonds/<N>/contexts/ · 31 unmodified lines | ||
| 150 | 148 | |
| 151 | 149 | In the beginning, this directory has only one file, ``nr_contexts``. Writing a |
| 152 | 150 | number (``N``) to the file creates the number of child directories named as |
| 153 | −``0`` to ``N-1``. Each directory represents each monitoring context. At the | |
| 154 | −moment, only one context per kdamond is supported, so only ``0`` or ``1`` can | |
| 155 | −be written to the file. | |
| 151 | +``0`` to ``N-1``. Each directory represents each monitoring context (refer to | |
| 152 | +:ref:`design <damon_design_execution_model_and_data_structures>` for more | |
| 153 | +details). At the moment, only one context per kdamond is supported, so only | |
| 154 | +``0`` or ``1`` can be written to the file. | |
| 156 | 155 | |
| 157 | 156 | .. _sysfs_contexts: |
| 158 | 157 | |
| @@ -270,8 +269,8 @@ schemes/<N>/ · 111 unmodified lines | ||
| 270 | 269 | ------------ |
| 271 | 270 | |
| 272 | 271 | In each scheme directory, five directories (``access_pattern``, ``quotas``, |
| 273 | −``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and one file | |
| 274 | −(``action``) exist. | |
| 272 | +``watermarks``, ``filters``, ``stats``, and ``tried_regions``) and two files | |
| 273 | +(``action`` and ``apply_interval``) exist. | |
| 275 | 274 | |
| 276 | 275 | The ``action`` file is for setting and getting the scheme's :ref:`action |
| 277 | 276 | <damon_design_damos_action>`. The keywords that can be written to and read |
| @@ -297,6 +296,9 @@ Note that support of each action depends on the running DAMON operations set · 19 unmodified lines | ||
| 297 | 296 | - ``stat``: Do nothing but count the statistics. |
| 298 | 297 | Supported by all operations sets. |
| 299 | 298 | |
| 299 | +The ``apply_interval_us`` file is for setting and getting the scheme's | |
| 300 | +:ref:`apply_interval <damon_design_damos>` in microseconds. | |
| 301 | + | |
| 300 | 302 | schemes/<N>/access_pattern/ |
| 301 | 303 | --------------------------- |
| 302 | 304 | |
| @@ -392,7 +394,7 @@ pages of all memory cgroups except ``/having_care_already``.:: · 89 unmodified lines | ||
| 392 | 394 | echo N > 1/matching |
| 393 | 395 | |
| 394 | 396 | Note that ``anon`` and ``memcg`` filters are currently supported only when |
| 395 | −``paddr`` `implementation <sysfs_contexts>` is being used. | |
| 397 | +``paddr`` :ref:`implementation <sysfs_contexts>` is being used. | |
| 396 | 398 | |
| 397 | 399 | Also, memory regions that are filtered out by ``addr`` or ``target`` filters |
| 398 | 400 | are not counted as the scheme has tried to those, while regions that filtered |
| @@ -430,9 +432,9 @@ that reading it returns the total size of the scheme tried regions, and creates · 31 unmodified lines | ||
| 430 | 432 | directories named integer starting from ``0`` under this directory. Each |
| 431 | 433 | directory contains files exposing detailed information about each of the memory |
| 432 | 434 | region that the corresponding scheme's ``action`` has tried to be applied under |
| 433 | −this directory, during next :ref:`aggregation interval | |
| 434 | −<sysfs_monitoring_attrs>`. The information includes address range, | |
| 435 | −``nr_accesses``, and ``age`` of the region. | |
| 435 | +this directory, during next :ref:`apply interval <damon_design_damos>` of the | |
| 436 | +corresponding scheme. The information includes address range, ``nr_accesses``, | |
| 437 | +and ``age`` of the region. | |
| 436 | 438 | |
| 437 | 439 | Writing ``update_schemes_tried_bytes`` to the relevant ``kdamonds/<N>/state`` |
| 438 | 440 | file will only update the ``total_bytes`` file, and will not create the |
| @@ -495,6 +497,62 @@ Please note that it's highly recommended to use user space tools like `damo · 56 unmodified lines | ||
| 495 | 497 | <https://github.com/awslabs/damo>`_ rather than manually reading and writing |
| 496 | 498 | the files as above. Above is only for an example. |
| 497 | 499 | |
| 500 | +.. _tracepoint: | |
| 501 | + | |
| 502 | +Tracepoints for Monitoring Results | |
| 503 | +================================== | |
| 504 | + | |
| 505 | +Users can get the monitoring results via the :ref:`tried_regions | |
| 506 | +<sysfs_schemes_tried_regions>`. The interface is useful for getting a | |
| 507 | +snapshot, but it could be inefficient for fully recording all the monitoring | |
| 508 | +results. For the purpose, two trace points, namely ``damon:damon_aggregated`` | |
| 509 | +and ``damon:damos_before_apply``, are provided. ``damon:damon_aggregated`` | |
| 510 | +provides the whole monitoring results, while ``damon:damos_before_apply`` | |
| 511 | +provides the monitoring results for regions that each DAMON-based Operation | |
| 512 | +Scheme (:ref:`DAMOS <damon_design_damos>`) is gonna be applied. Hence, | |
| 513 | +``damon:damos_before_apply`` is more useful for recording internal behavior of | |
| 514 | +DAMOS, or DAMOS target access | |
| 515 | +:ref:`pattern <damon_design_damos_access_pattern>` based query-like efficient | |
| 516 | +monitoring results recording. | |
| 517 | + | |
| 518 | +While the monitoring is turned on, you could record the tracepoint events and | |
| 519 | +show results using tracepoint supporting tools like ``perf``. For example:: | |
| 520 | + | |
| 521 | + # echo on > monitor_on | |
| 522 | + # perf record -e damon:damon_aggregated & | |
| 523 | + # sleep 5 | |
| 524 | + # kill 9 $(pidof perf) | |
| 525 | + # echo off > monitor_on | |
| 526 | + # perf script | |
| 527 | + kdamond.0 46568 [027] 79357.842179: damon:damon_aggregated: target_id=0 nr_regions=11 122509119488-135708762112: 0 864 | |
| 528 | + [...] | |
| 529 | + | |
| 530 | +Each line of the perf script output represents each monitoring region. The | |
| 531 | +first five fields are as usual other tracepoint outputs. The sixth field | |
| 532 | +(``target_id=X``) shows the ide of the monitoring target of the region. The | |
| 533 | +seventh field (``nr_regions=X``) shows the total number of monitoring regions | |
| 534 | +for the target. The eighth field (``X-Y:``) shows the start (``X``) and end | |
| 535 | +(``Y``) addresses of the region in bytes. The ninth field (``X``) shows the | |
| 536 | +``nr_accesses`` of the region (refer to | |
| 537 | +:ref:`design <damon_design_region_based_sampling>` for more details of the | |
| 538 | +counter). Finally the tenth field (``X``) shows the ``age`` of the region | |
| 539 | +(refer to :ref:`design <damon_design_age_tracking>` for more details of the | |
| 540 | +counter). | |
| 541 | + | |
| 542 | +If the event was ``damon:damos_beofre_apply``, the ``perf script`` output would | |
| 543 | +be somewhat like below:: | |
| 544 | + | |
| 545 | + kdamond.0 47293 [000] 80801.060214: damon:damos_before_apply: ctx_idx=0 scheme_idx=0 target_idx=0 nr_regions=11 121932607488-135128711168: 0 136 | |
| 546 | + [...] | |
| 547 | + | |
| 548 | +Each line of the output represents each monitoring region that each DAMON-based | |
| 549 | +Operation Scheme was about to be applied at the traced time. The first five | |
| 550 | +fields are as usual. It shows the index of the DAMON context (``ctx_idx=X``) | |
| 551 | +of the scheme in the list of the contexts of the context's kdamond, the index | |
| 552 | +of the scheme (``scheme_idx=X``) in the list of the schemes of the context, in | |
| 553 | +addition to the output of ``damon_aggregated`` tracepoint. | |
| 554 | + | |
| 555 | + | |
| 498 | 556 | .. _debugfs_interface: |
| 499 | 557 | |
| 500 | 558 | debugfs Interface (DEPRECATED!) |
| @@ -790,23 +848,3 @@ directory by putting the name of the context to the ``rm_contexts`` file. :: · 289 unmodified lines | ||
| 790 | 848 | |
| 791 | 849 | Note that ``mk_contexts``, ``rm_contexts``, and ``monitor_on`` files are in the |
| 792 | 850 | root directory only. |
| 793 | − | |
| 794 | − | |
| 795 | −.. _tracepoint: | |
| 796 | − | |
| 797 | −Tracepoint for Monitoring Results | |
| 798 | −================================= | |
| 799 | − | |
| 800 | −Users can get the monitoring results via the :ref:`tried_regions | |
| 801 | −<sysfs_schemes_tried_regions>` or a tracepoint, ``damon:damon_aggregated``. | |
| 802 | −While the tried regions directory is useful for getting a snapshot, the | |
| 803 | −tracepoint is useful for getting a full record of the results. While the | |
| 804 | −monitoring is turned on, you could record the tracepoint events and show | |
| 805 | −results using tracepoint supporting tools like ``perf``. For example:: | |
| 806 | − | |
| 807 | − # echo on > monitor_on | |
| 808 | − # perf record -e damon:damon_aggregated & | |
| 809 | − # sleep 5 | |
| 810 | − # kill 9 $(pidof perf) | |
| 811 | − # echo off > monitor_on | |
| 812 | − # perf script | |
| @@ -155,6 +155,15 @@ stable_node_chains_prune_millisecs · 154 unmodified lines | ||
| 155 | 155 | scan. It's a noop if not a single KSM page hit the |
| 156 | 156 | ``max_page_sharing`` yet. |
| 157 | 157 | |
| 158 | +smart_scan | |
| 159 | + Historically KSM checked every candidate page for each scan. It did | |
| 160 | + not take into account historic information. When smart scan is | |
| 161 | + enabled, pages that have previously not been de-duplicated get | |
| 162 | + skipped. How often these pages are skipped depends on how often | |
| 163 | + de-duplication has already been tried and failed. By default this | |
| 164 | + optimization is enabled. The ``pages_skipped`` metric shows how | |
| 165 | + effective the setting is. | |
| 166 | + | |
| 158 | 167 | The effectiveness of KSM and MADV_MERGEABLE is shown in ``/sys/kernel/mm/ksm/``: |
| 159 | 168 | |
| 160 | 169 | general_profit |
| @@ -169,6 +178,8 @@ pages_unshared · 8 unmodified lines | ||
| 169 | 178 | how many pages unique but repeatedly checked for merging |
| 170 | 179 | pages_volatile |
| 171 | 180 | how many pages changing too fast to be placed in a tree |
| 181 | +pages_skipped | |
| 182 | + how many pages did the "smart" page scanning algorithm skip | |
| 172 | 183 | full_scans |
| 173 | 184 | how many times all mergeable areas have been scanned |
| 174 | 185 | stable_node_chains |
| @@ -227,3 +227,92 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is · 226 unmodified lines | ||
| 227 | 227 | always 12 at most architectures). Since Linux 3.11 their meaning changes |
| 228 | 228 | after first clear of soft-dirty bits. Since Linux 4.2 they are used for |
| 229 | 229 | flags unconditionally. |
| 230 | + | |
| 231 | +Pagemap Scan IOCTL | |
| 232 | +================== | |
| 233 | + | |
| 234 | +The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally | |
| 235 | +clear the info about page table entries. The following operations are supported | |
| 236 | +in this IOCTL: | |
| 237 | + | |
| 238 | +- Scan the address range and get the memory ranges matching the provided criteria. | |
| 239 | + This is performed when the output buffer is specified. | |
| 240 | +- Write-protect the pages. The ``PM_SCAN_WP_MATCHING`` is used to write-protect | |
| 241 | + the pages of interest. The ``PM_SCAN_CHECK_WPASYNC`` aborts the operation if | |
| 242 | + non-Async Write Protected pages are found. The ``PM_SCAN_WP_MATCHING`` can be | |
| 243 | + used with or without ``PM_SCAN_CHECK_WPASYNC``. | |
| 244 | +- Both of those operations can be combined into one atomic operation where we can | |
| 245 | + get and write protect the pages as well. | |
| 246 | + | |
| 247 | +Following flags about pages are currently supported: | |
| 248 | + | |
| 249 | +- ``PAGE_IS_WPALLOWED`` - Page has async-write-protection enabled | |
| 250 | +- ``PAGE_IS_WRITTEN`` - Page has been written to from the time it was write protected | |
| 251 | +- ``PAGE_IS_FILE`` - Page is file backed | |
| 252 | +- ``PAGE_IS_PRESENT`` - Page is present in the memory | |
| 253 | +- ``PAGE_IS_SWAPPED`` - Page is in swapped | |
| 254 | +- ``PAGE_IS_PFNZERO`` - Page has zero PFN | |
| 255 | +- ``PAGE_IS_HUGE`` - Page is THP or Hugetlb backed | |
| 256 | + | |
| 257 | +The ``struct pm_scan_arg`` is used as the argument of the IOCTL. | |
| 258 | + | |
| 259 | + 1. The size of the ``struct pm_scan_arg`` must be specified in the ``size`` | |
| 260 | + field. This field will be helpful in recognizing the structure if extensions | |
| 261 | + are done later. | |
| 262 | + 2. The flags can be specified in the ``flags`` field. The ``PM_SCAN_WP_MATCHING`` | |
| 263 | + and ``PM_SCAN_CHECK_WPASYNC`` are the only added flags at this time. The get | |
| 264 | + operation is optionally performed depending upon if the output buffer is | |
| 265 | + provided or not. | |
| 266 | + 3. The range is specified through ``start`` and ``end``. | |
| 267 | + 4. The walk can abort before visiting the complete range such as the user buffer | |
| 268 | + can get full etc. The walk ending address is specified in``end_walk``. | |
| 269 | + 5. The output buffer of ``struct page_region`` array and size is specified in | |
| 270 | + ``vec`` and ``vec_len``. | |
| 271 | + 6. The optional maximum requested pages are specified in the ``max_pages``. | |
| 272 | + 7. The masks are specified in ``category_mask``, ``category_anyof_mask``, | |
| 273 | + ``category_inverted`` and ``return_mask``. | |
| 274 | + | |
| 275 | +Find pages which have been written and WP them as well:: | |
| 276 | + | |
| 277 | + struct pm_scan_arg arg = { | |
| 278 | + .size = sizeof(arg), | |
| 279 | + .flags = PM_SCAN_CHECK_WPASYNC | PM_SCAN_CHECK_WPASYNC, | |
| 280 | + .. | |
| 281 | + .category_mask = PAGE_IS_WRITTEN, | |
| 282 | + .return_mask = PAGE_IS_WRITTEN, | |
| 283 | + }; | |
| 284 | + | |
| 285 | +Find pages which have been written, are file backed, not swapped and either | |
| 286 | +present or huge:: | |
| 287 | + | |
| 288 | + struct pm_scan_arg arg = { | |
| 289 | + .size = sizeof(arg), | |
| 290 | + .flags = 0, | |
| 291 | + .. | |
| 292 | + .category_mask = PAGE_IS_WRITTEN | PAGE_IS_SWAPPED, | |
| 293 | + .category_inverted = PAGE_IS_SWAPPED, | |
| 294 | + .category_anyof_mask = PAGE_IS_PRESENT | PAGE_IS_HUGE, | |
| 295 | + .return_mask = PAGE_IS_WRITTEN | PAGE_IS_SWAPPED | | |
| 296 | + PAGE_IS_PRESENT | PAGE_IS_HUGE, | |
| 297 | + }; | |
| 298 | + | |
| 299 | +The ``PAGE_IS_WRITTEN`` flag can be considered as a better-performing alternative | |
| 300 | +of soft-dirty flag. It doesn't get affected by VMA merging of the kernel and hence | |
| 301 | +the user can find the true soft-dirty pages in case of normal pages. (There may | |
| 302 | +still be extra dirty pages reported for THP or Hugetlb pages.) | |
| 303 | + | |
| 304 | +"PAGE_IS_WRITTEN" category is used with uffd write protect-enabled ranges to | |
| 305 | +implement memory dirty tracking in userspace: | |
| 306 | + | |
| 307 | + 1. The userfaultfd file descriptor is created with ``userfaultfd`` syscall. | |
| 308 | + 2. The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features | |
| 309 | + are set by ``UFFDIO_API`` IOCTL. | |
| 310 | + 3. The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode | |
| 311 | + through ``UFFDIO_REGISTER`` IOCTL. | |
| 312 | + 4. Then any part of the registered memory or the whole memory region must | |
| 313 | + be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_WP_MATCHING`` | |
| 314 | + or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the | |
| 315 | + same operation. The former is better in terms of performance. | |
| 316 | + 5. Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which | |
| 317 | + have been written to since they were last marked and/or optionally write protect | |
| 318 | + the pages as well. | |
| @@ -244,6 +244,41 @@ write-protected (so future writes will also result in a WP fault). These ioctls · 243 unmodified lines | ||
| 244 | 244 | support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP`` |
| 245 | 245 | respectively) to configure the mapping this way. |
| 246 | 246 | |
| 247 | +If the userfaultfd context has ``UFFD_FEATURE_WP_ASYNC`` feature bit set, | |
| 248 | +any vma registered with write-protection will work in async mode rather | |
| 249 | +than the default sync mode. | |
| 250 | + | |
| 251 | +In async mode, there will be no message generated when a write operation | |
| 252 | +happens, meanwhile the write-protection will be resolved automatically by | |
| 253 | +the kernel. It can be seen as a more accurate version of soft-dirty | |
| 254 | +tracking and it can be different in a few ways: | |
| 255 | + | |
| 256 | + - The dirty result will not be affected by vma changes (e.g. vma | |
| 257 | + merging) because the dirty is only tracked by the pte. | |
| 258 | + | |
| 259 | + - It supports range operations by default, so one can enable tracking on | |
| 260 | + any range of memory as long as page aligned. | |
| 261 | + | |
| 262 | + - Dirty information will not get lost if the pte was zapped due to | |
| 263 | + various reasons (e.g. during split of a shmem transparent huge page). | |
| 264 | + | |
| 265 | + - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit | |
| 266 | + set; dirty when uffd-wp bit cleared), it has different semantics on | |
| 267 | + some of the memory operations. For example: ``MADV_DONTNEED`` on | |
| 268 | + anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as | |
| 269 | + dirtying of memory by dropping uffd-wp bit during the procedure. | |
| 270 | + | |
| 271 | +The user app can collect the "written/dirty" status by looking up the | |
| 272 | +uffd-wp bit for the pages being interested in /proc/pagemap. | |
| 273 | + | |
| 274 | +The page will not be under track of uffd-wp async mode until the page is | |
| 275 | +explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode | |
| 276 | +flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault | |
| 277 | +that was tracked by async mode userfaultfd-wp is invalid. | |
| 278 | + | |
| 279 | +When userfaultfd-wp async mode is used alone, it can be applied to all | |
| 280 | +kinds of memory. | |
| 281 | + | |
| 247 | 282 | Memory Poisioning Emulation |
| 248 | 283 | --------------------------- |
| 249 | 284 | |
| @@ -175,7 +175,7 @@ will return the previous entry which occurs before the entry at index. · 174 unmodified lines | ||
| 175 | 175 | mas_find() will find the first entry which exists at or above index on |
| 176 | 176 | the first call, and the next entry from every subsequent calls. |
| 177 | 177 | |
| 178 | −mas_find_rev() will find the fist entry which exists at or below the last on | |
| 178 | +mas_find_rev() will find the first entry which exists at or below the last on | |
| 179 | 179 | the first call, and the previous entry from every subsequent calls. |
| 180 | 180 | |
| 181 | 181 | If the user needs to yield the lock during an operation, then the maple state |
| @@ -1,5 +1,8 @@ | ||
| 1 | −The Kernel Address Sanitizer (KASAN) | |
| 2 | −==================================== | |
| 1 | +.. SPDX-License-Identifier: GPL-2.0 | |
| 2 | +.. Copyright (C) 2023, Google LLC. | |
| 3 | + | |
| 4 | +Kernel Address Sanitizer (KASAN) | |
| 5 | +================================ | |
| 3 | 6 | |
| 4 | 7 | Overview |
| 5 | 8 | -------- |
| @@ -1,8 +1,8 @@ | ||
| 1 | 1 | .. SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | .. Copyright (C) 2019, Google LLC. |
| 3 | 3 | |
| 4 | −The Kernel Concurrency Sanitizer (KCSAN) | |
| 5 | −======================================== | |
| 4 | +Kernel Concurrency Sanitizer (KCSAN) | |
| 5 | +==================================== | |
| 6 | 6 | |
| 7 | 7 | The Kernel Concurrency Sanitizer (KCSAN) is a dynamic race detector, which |
| 8 | 8 | relies on compile-time instrumentation, and uses a watchpoint-based sampling |
| @@ -1,9 +1,9 @@ | ||
| 1 | 1 | .. SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | .. Copyright (C) 2022, Google LLC. |
| 3 | 3 | |
| 4 | −=================================== | |
| 5 | −The Kernel Memory Sanitizer (KMSAN) | |
| 6 | −=================================== | |
| 4 | +=============================== | |
| 5 | +Kernel Memory Sanitizer (KMSAN) | |
| 6 | +=============================== | |
| 7 | 7 | |
| 8 | 8 | KMSAN is a dynamic error detector aimed at finding uses of uninitialized |
| 9 | 9 | values. It is based on compiler instrumentation, and is quite similar to the |
| @@ -1,5 +1,7 @@ | ||
| 1 | −The Undefined Behavior Sanitizer - UBSAN | |
| 2 | −======================================== | |
| 1 | +.. SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +Undefined Behavior Sanitizer - UBSAN | |
| 4 | +==================================== | |
| 3 | 5 | |
| 4 | 6 | UBSAN is a runtime undefined behaviour checker. |
| 5 | 7 | |
| @@ -154,6 +154,8 @@ The monitoring overhead of this mechanism will arbitrarily increase as the · 153 unmodified lines | ||
| 154 | 154 | size of the target workload grows. |
| 155 | 155 | |
| 156 | 156 | |
| 157 | +.. _damon_design_region_based_sampling: | |
| 158 | + | |
| 157 | 159 | Region Based Sampling |
| 158 | 160 | ~~~~~~~~~~~~~~~~~~~~~ |
| 159 | 161 | |
| @@ -163,9 +165,10 @@ assumption (pages in a region have the same access frequencies) is kept, only · 3 unmodified lines | ||
| 163 | 165 | one page in the region is required to be checked. Thus, for each ``sampling |
| 164 | 166 | interval``, DAMON randomly picks one page in each region, waits for one |
| 165 | 167 | ``sampling interval``, checks whether the page is accessed meanwhile, and |
| 166 | −increases the access frequency of the region if so. Therefore, the monitoring | |
| 167 | −overhead is controllable by setting the number of regions. DAMON allows users | |
| 168 | −to set the minimum and the maximum number of regions for the trade-off. | |
| 168 | +increases the access frequency counter of the region if so. The counter is | |
| 169 | +called ``nr_regions`` of the region. Therefore, the monitoring overhead is | |
| 170 | +controllable by setting the number of regions. DAMON allows users to set the | |
| 171 | +minimum and the maximum number of regions for the trade-off. | |
| 169 | 172 | |
| 170 | 173 | This scheme, however, cannot preserve the quality of the output if the |
| 171 | 174 | assumption is not guaranteed. |
| @@ -190,6 +193,8 @@ In this way, DAMON provides its best-effort quality and minimal overhead while · 18 unmodified lines | ||
| 190 | 193 | keeping the bounds users set for their trade-off. |
| 191 | 194 | |
| 192 | 195 | |
| 196 | +.. _damon_design_age_tracking: | |
| 197 | + | |
| 193 | 198 | Age Tracking |
| 194 | 199 | ~~~~~~~~~~~~ |
| 195 | 200 | |
| @@ -254,7 +259,8 @@ works, DAMON provides a feature called Data Access Monitoring-based Operation · 58 unmodified lines | ||
| 254 | 259 | Schemes (DAMOS). It lets users specify their desired schemes at a high |
| 255 | 260 | level. For such specifications, DAMON starts monitoring, finds regions having |
| 256 | 261 | the access pattern of interest, and applies the user-desired operation actions |
| 257 | −to the regions as soon as found. | |
| 262 | +to the regions, for every user-specified time interval called | |
| 263 | +``apply_interval``. | |
| 258 | 264 | |
| 259 | 265 | |
| 260 | 266 | .. _damon_design_damos_action: |
| @@ -471,3 +477,15 @@ modules for proactive reclamation and LRU lists manipulation are provided. For · 210 unmodified lines | ||
| 471 | 477 | more detail, please read the usage documents for those |
| 472 | 478 | (:doc:`/admin-guide/mm/damon/reclaim` and |
| 473 | 479 | :doc:`/admin-guide/mm/damon/lru_sort`). |
| 480 | + | |
| 481 | + | |
| 482 | +.. _damon_design_execution_model_and_data_structures: | |
| 483 | + | |
| 484 | +Execution Model and Data Structures | |
| 485 | +=================================== | |
| 486 | + | |
| 487 | +The monitoring-related information including the monitoring request | |
| 488 | +specification and DAMON-based operation schemes are stored in a data structure | |
| 489 | +called DAMON ``context``. DAMON executes each context with a kernel thread | |
| 490 | +called ``kdamond``. Multiple kdamonds could run in parallel, for different | |
| 491 | +types of monitoring. | |
| @@ -107,14 +107,14 @@ GetOptions( · 106 unmodified lines | ||
| 107 | 107 | ); |
| 108 | 108 | |
| 109 | 109 | # Defaults for dynamically discovered regex's |
| 110 | −my $regex_direct_begin_default = 'order=([0-9]*) may_writepage=([0-9]*) gfp_flags=([A-Z_|]*)'; | |
| 110 | +my $regex_direct_begin_default = 'order=([0-9]*) gfp_flags=([A-Z_|]*)'; | |
| 111 | 111 | my $regex_direct_end_default = 'nr_reclaimed=([0-9]*)'; |
| 112 | 112 | my $regex_kswapd_wake_default = 'nid=([0-9]*) order=([0-9]*)'; |
| 113 | 113 | my $regex_kswapd_sleep_default = 'nid=([0-9]*)'; |
| 114 | −my $regex_wakeup_kswapd_default = 'nid=([0-9]*) zid=([0-9]*) order=([0-9]*) gfp_flags=([A-Z_|]*)'; | |
| 115 | −my $regex_lru_isolate_default = 'isolate_mode=([0-9]*) classzone_idx=([0-9]*) order=([0-9]*) nr_requested=([0-9]*) nr_scanned=([0-9]*) nr_skipped=([0-9]*) nr_taken=([0-9]*) lru=([a-z_]*)'; | |
| 114 | +my $regex_wakeup_kswapd_default = 'nid=([0-9]*) order=([0-9]*) gfp_flags=([A-Z_|]*)'; | |
| 115 | +my $regex_lru_isolate_default = 'classzone=([0-9]*) order=([0-9]*) nr_requested=([0-9]*) nr_scanned=([0-9]*) nr_skipped=([0-9]*) nr_taken=([0-9]*) lru=([a-z_]*)'; | |
| 116 | 116 | my $regex_lru_shrink_inactive_default = 'nid=([0-9]*) nr_scanned=([0-9]*) nr_reclaimed=([0-9]*) nr_dirty=([0-9]*) nr_writeback=([0-9]*) nr_congested=([0-9]*) nr_immediate=([0-9]*) nr_activate_anon=([0-9]*) nr_activate_file=([0-9]*) nr_ref_keep=([0-9]*) nr_unmap_fail=([0-9]*) priority=([0-9]*) flags=([A-Z_|]*)'; |
| 117 | −my $regex_lru_shrink_active_default = 'lru=([A-Z_]*) nr_scanned=([0-9]*) nr_rotated=([0-9]*) priority=([0-9]*)'; | |
| 117 | +my $regex_lru_shrink_active_default = 'lru=([A-Z_]*) nr_taken=([0-9]*) nr_active=([0-9]*) nr_deactivated=([0-9]*) nr_referenced=([0-9]*) priority=([0-9]*) flags=([A-Z_|]*)' ; | |
| 118 | 118 | my $regex_writepage_default = 'page=([0-9a-f]*) pfn=([0-9]*) flags=([A-Z_|]*)'; |
| 119 | 119 | |
| 120 | 120 | # Dyanically discovered regex |
| @@ -184,8 +184,7 @@ sub generate_traceevent_regex { · 63 unmodified lines | ||
| 184 | 184 | $regex_direct_begin = generate_traceevent_regex( |
| 185 | 185 | "vmscan/mm_vmscan_direct_reclaim_begin", |
| 186 | 186 | $regex_direct_begin_default, |
| 187 | − "order", "may_writepage", | |
| 188 | − "gfp_flags"); | |
| 187 | + "order", "gfp_flags"); | |
| 189 | 188 | $regex_direct_end = generate_traceevent_regex( |
| 190 | 189 | "vmscan/mm_vmscan_direct_reclaim_end", |
| 191 | 190 | $regex_direct_end_default, |
| @@ -201,11 +200,11 @@ $regex_kswapd_sleep = generate_traceevent_regex( · 9 unmodified lines | ||
| 201 | 200 | $regex_wakeup_kswapd = generate_traceevent_regex( |
| 202 | 201 | "vmscan/mm_vmscan_wakeup_kswapd", |
| 203 | 202 | $regex_wakeup_kswapd_default, |
| 204 | − "nid", "zid", "order", "gfp_flags"); | |
| 203 | + "nid", "order", "gfp_flags"); | |
| 205 | 204 | $regex_lru_isolate = generate_traceevent_regex( |
| 206 | 205 | "vmscan/mm_vmscan_lru_isolate", |
| 207 | 206 | $regex_lru_isolate_default, |
| 208 | − "isolate_mode", "classzone_idx", "order", | |
| 207 | + "classzone", "order", | |
| 209 | 208 | "nr_requested", "nr_scanned", "nr_skipped", "nr_taken", |
| 210 | 209 | "lru"); |
| 211 | 210 | $regex_lru_shrink_inactive = generate_traceevent_regex( |
| @@ -218,11 +217,10 @@ $regex_lru_shrink_inactive = generate_traceevent_regex( · 6 unmodified lines | ||
| 218 | 217 | $regex_lru_shrink_active = generate_traceevent_regex( |
| 219 | 218 | "vmscan/mm_vmscan_lru_shrink_active", |
| 220 | 219 | $regex_lru_shrink_active_default, |
| 221 | − "nid", "zid", | |
| 222 | − "lru", | |
| 223 | − "nr_scanned", "nr_rotated", "priority"); | |
| 220 | + "nid", "nr_taken", "nr_active", "nr_deactivated", "nr_referenced", | |
| 221 | + "priority", "flags"); | |
| 224 | 222 | $regex_writepage = generate_traceevent_regex( |
| 225 | − "vmscan/mm_vmscan_writepage", | |
| 223 | + "vmscan/mm_vmscan_write_folio", | |
| 226 | 224 | $regex_writepage_default, |
| 227 | 225 | "page", "pfn", "flags"); |
| 228 | 226 | |
| @@ -371,7 +369,7 @@ EVENT_PROCESS: · 142 unmodified lines | ||
| 371 | 369 | print " $regex_wakeup_kswapd\n"; |
| 372 | 370 | next; |
| 373 | 371 | } |
| 374 | − my $order = $3; | |
| 372 | + my $order = $2; | |
| 375 | 373 | $perprocesspid{$process_pid}->{MM_VMSCAN_WAKEUP_KSWAPD_PERORDER}[$order]++; |
| 376 | 374 | } elsif ($tracepoint eq "mm_vmscan_lru_isolate") { |
| 377 | 375 | $details = $6; |
| @@ -381,18 +379,14 @@ EVENT_PROCESS: · 3 unmodified lines | ||
| 381 | 379 | print " $regex_lru_isolate/o\n"; |
| 382 | 380 | next; |
| 383 | 381 | } |
| 384 | − my $isolate_mode = $1; | |
| 385 | − my $nr_scanned = $5; | |
| 386 | − my $file = $8; | |
| 387 | − | |
| 388 | − # To closer match vmstat scanning statistics, only count isolate_both | |
| 389 | − # and isolate_inactive as scanning. isolate_active is rotation | |
| 390 | − # isolate_inactive == 1 | |
| 391 | − # isolate_active == 2 | |
| 392 | − # isolate_both == 3 | |
| 393 | − if ($isolate_mode != 2) { | |
| 382 | + my $nr_scanned = $4; | |
| 383 | + my $lru = $7; | |
| 384 | + | |
| 385 | + # To closer match vmstat scanning statistics, only count | |
| 386 | + # inactive lru as scanning | |
| 387 | + if ($lru =~ /inactive_/) { | |
| 394 | 388 | $perprocesspid{$process_pid}->{HIGH_NR_SCANNED} += $nr_scanned; |
| 395 | − if ($file =~ /_file/) { | |
| 389 | + if ($lru =~ /_file/) { | |
| 396 | 390 | $perprocesspid{$process_pid}->{HIGH_NR_FILE_SCANNED} += $nr_scanned; |
| 397 | 391 | } else { |
| 398 | 392 | $perprocesspid{$process_pid}->{HIGH_NR_ANON_SCANNED} += $nr_scanned; |
| @@ -5332,6 +5332,7 @@ S: Maintained · 5331 unmodified lines | ||
| 5332 | 5332 | F: mm/memcontrol.c |
| 5333 | 5333 | F: mm/swap_cgroup.c |
| 5334 | 5334 | F: tools/testing/selftests/cgroup/memcg_protection.m |
| 5335 | +F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c | |
| 5335 | 5336 | F: tools/testing/selftests/cgroup/test_kmem.c |
| 5336 | 5337 | F: tools/testing/selftests/cgroup/test_memcontrol.c |
| 5337 | 5338 | |
| @@ -9754,6 +9755,7 @@ F: include/linux/hugetlb.h · 4416 unmodified lines | ||
| 9754 | 9755 | F: mm/hugetlb.c |
| 9755 | 9756 | F: mm/hugetlb_vmemmap.c |
| 9756 | 9757 | F: mm/hugetlb_vmemmap.h |
| 9758 | +F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c | |
| 9757 | 9759 | |
| 9758 | 9760 | HVA ST MEDIA DRIVER |
| 9759 | 9761 | M: Jean-Christophe Trotin <jean-christophe.trotin@foss.st.com> |
| @@ -286,6 +286,26 @@ arch___test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 285 unmodified lines | ||
| 286 | 286 | #define arch_test_bit generic_test_bit |
| 287 | 287 | #define arch_test_bit_acquire generic_test_bit_acquire |
| 288 | 288 | |
| 289 | +static inline bool xor_unlock_is_negative_byte(unsigned long mask, | |
| 290 | + volatile unsigned long *p) | |
| 291 | +{ | |
| 292 | + unsigned long temp, old; | |
| 293 | + | |
| 294 | + __asm__ __volatile__( | |
| 295 | + "1: ldl_l %0,%4\n" | |
| 296 | + " mov %0,%2\n" | |
| 297 | + " xor %0,%3,%0\n" | |
| 298 | + " stl_c %0,%1\n" | |
| 299 | + " beq %0,2f\n" | |
| 300 | + ".subsection 2\n" | |
| 301 | + "2: br 1b\n" | |
| 302 | + ".previous" | |
| 303 | + :"=&r" (temp), "=m" (*p), "=&r" (old) | |
| 304 | + :"Ir" (mask), "m" (*p)); | |
| 305 | + | |
| 306 | + return (old & BIT(7)) != 0; | |
| 307 | +} | |
| 308 | + | |
| 289 | 309 | /* |
| 290 | 310 | * ffz = Find First Zero in word. Undefined if no zero exists, |
| 291 | 311 | * so code should check against ~0UL first.. |
| @@ -96,7 +96,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, · 95 unmodified lines | ||
| 96 | 96 | static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, |
| 97 | 97 | unsigned long addr) |
| 98 | 98 | { |
| 99 | − tlb_remove_ptdesc(tlb, virt_to_ptdesc(pudp)); | |
| 99 | + struct ptdesc *ptdesc = virt_to_ptdesc(pudp); | |
| 100 | + | |
| 101 | + pagetable_pud_dtor(ptdesc); | |
| 102 | + tlb_remove_ptdesc(tlb, ptdesc); | |
| 100 | 103 | } |
| 101 | 104 | #endif |
| 102 | 105 | |
| @@ -411,8 +411,8 @@ static int __access_remote_tags(struct mm_struct *mm, unsigned long addr, · 410 unmodified lines | ||
| 411 | 411 | struct page *page = get_user_page_vma_remote(mm, addr, |
| 412 | 412 | gup_flags, &vma); |
| 413 | 413 | |
| 414 | − if (IS_ERR_OR_NULL(page)) { | |
| 415 | − err = page == NULL ? -EIO : PTR_ERR(page); | |
| 414 | + if (IS_ERR(page)) { | |
| 415 | + err = PTR_ERR(page); | |
| 416 | 416 | break; |
| 417 | 417 | } |
| 418 | 418 | |
| @@ -300,7 +300,11 @@ void __init kasan_init(void) · 299 unmodified lines | ||
| 300 | 300 | kasan_init_shadow(); |
| 301 | 301 | kasan_init_depth(); |
| 302 | 302 | #if defined(CONFIG_KASAN_GENERIC) |
| 303 | − /* CONFIG_KASAN_SW_TAGS also requires kasan_init_sw_tags(). */ | |
| 303 | + /* | |
| 304 | + * Generic KASAN is now fully initialized. | |
| 305 | + * Software and Hardware Tag-Based modes still require | |
| 306 | + * kasan_init_sw_tags() and kasan_init_hw_tags() correspondingly. | |
| 307 | + */ | |
| 304 | 308 | pr_info("KernelAddressSanitizer initialized (generic)\n"); |
| 305 | 309 | #endif |
| 306 | 310 | } |
| @@ -84,6 +84,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) · 83 unmodified lines | ||
| 84 | 84 | |
| 85 | 85 | if (!ptdesc) |
| 86 | 86 | return NULL; |
| 87 | + pagetable_pud_ctor(ptdesc); | |
| 87 | 88 | pud = ptdesc_address(ptdesc); |
| 88 | 89 | |
| 89 | 90 | pud_init(pud); |
| @@ -319,6 +319,27 @@ arch___test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 318 unmodified lines | ||
| 319 | 319 | return test_and_change_bit(nr, addr); |
| 320 | 320 | } |
| 321 | 321 | |
| 322 | +static inline bool xor_unlock_is_negative_byte(unsigned long mask, | |
| 323 | + volatile unsigned long *p) | |
| 324 | +{ | |
| 325 | +#ifdef CONFIG_COLDFIRE | |
| 326 | + __asm__ __volatile__ ("eorl %1, %0" | |
| 327 | + : "+m" (*p) | |
| 328 | + : "d" (mask) | |
| 329 | + : "memory"); | |
| 330 | + return *p & (1 << 7); | |
| 331 | +#else | |
| 332 | + char result; | |
| 333 | + char *cp = (char *)p + 3; /* m68k is big-endian */ | |
| 334 | + | |
| 335 | + __asm__ __volatile__ ("eor.b %1, %2; smi %0" | |
| 336 | + : "=d" (result) | |
| 337 | + : "di" (mask), "o" (*cp) | |
| 338 | + : "memory"); | |
| 339 | + return result; | |
| 340 | +#endif | |
| 341 | +} | |
| 342 | + | |
| 322 | 343 | /* |
| 323 | 344 | * The true 68020 and more advanced processors support the "bfffo" |
| 324 | 345 | * instruction for finding bits. ColdFire and simple 68000 parts |
| @@ -73,7 +73,8 @@ int __mips_test_and_clear_bit(unsigned long nr, · 72 unmodified lines | ||
| 73 | 73 | volatile unsigned long *addr); |
| 74 | 74 | int __mips_test_and_change_bit(unsigned long nr, |
| 75 | 75 | volatile unsigned long *addr); |
| 76 | − | |
| 76 | +bool __mips_xor_is_negative_byte(unsigned long mask, | |
| 77 | + volatile unsigned long *addr); | |
| 77 | 78 | |
| 78 | 79 | /* |
| 79 | 80 | * set_bit - Atomically set a bit in memory |
| @@ -279,6 +280,28 @@ static inline int test_and_change_bit(unsigned long nr, · 199 unmodified lines | ||
| 279 | 280 | return res; |
| 280 | 281 | } |
| 281 | 282 | |
| 283 | +static inline bool xor_unlock_is_negative_byte(unsigned long mask, | |
| 284 | + volatile unsigned long *p) | |
| 285 | +{ | |
| 286 | + unsigned long orig; | |
| 287 | + bool res; | |
| 288 | + | |
| 289 | + smp_mb__before_atomic(); | |
| 290 | + | |
| 291 | + if (!kernel_uses_llsc) { | |
| 292 | + res = __mips_xor_is_negative_byte(mask, p); | |
| 293 | + } else { | |
| 294 | + orig = __test_bit_op(*p, "%0", | |
| 295 | + "xor\t%1, %0, %3", | |
| 296 | + "ir"(mask)); | |
| 297 | + res = (orig & BIT(7)) != 0; | |
| 298 | + } | |
| 299 | + | |
| 300 | + smp_llsc_mb(); | |
| 301 | + | |
| 302 | + return res; | |
| 303 | +} | |
| 304 | + | |
| 282 | 305 | #undef __bit_op |
| 283 | 306 | #undef __test_bit_op |
| 284 | 307 | |
| @@ -95,6 +95,7 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) · 94 unmodified lines | ||
| 95 | 95 | |
| 96 | 96 | if (!ptdesc) |
| 97 | 97 | return NULL; |
| 98 | + pagetable_pud_ctor(ptdesc); | |
| 98 | 99 | pud = ptdesc_address(ptdesc); |
| 99 | 100 | |
| 100 | 101 | pud_init(pud); |
| @@ -146,3 +146,17 @@ int __mips_test_and_change_bit(unsigned long nr, volatile unsigned long *addr) · 145 unmodified lines | ||
| 146 | 146 | return res; |
| 147 | 147 | } |
| 148 | 148 | EXPORT_SYMBOL(__mips_test_and_change_bit); |
| 149 | + | |
| 150 | +bool __mips_xor_is_negative_byte(unsigned long mask, | |
| 151 | + volatile unsigned long *addr) | |
| 152 | +{ | |
| 153 | + unsigned long flags; | |
| 154 | + unsigned long data; | |
| 155 | + | |
| 156 | + raw_local_irq_save(flags); | |
| 157 | + data = *addr; | |
| 158 | + *addr = data ^ mask; | |
| 159 | + raw_local_irq_restore(flags); | |
| 160 | + | |
| 161 | + return (data & BIT(7)) != 0; | |
| 162 | +} | |
| @@ -117,7 +117,7 @@ void __flush_dcache_pages(struct page *page, unsigned int nr) · 116 unmodified lines | ||
| 117 | 117 | * get faulted into the tlb (and thus flushed) anyways. |
| 118 | 118 | */ |
| 119 | 119 | for (i = 0; i < nr; i++) { |
| 120 | − addr = (unsigned long)kmap_local_page(page + i); | |
| 120 | + addr = (unsigned long)kmap_local_page(nth_page(page, i)); | |
| 121 | 121 | flush_data_cache_page(addr); |
| 122 | 122 | kunmap_local((void *)addr); |
| 123 | 123 | } |
| @@ -233,35 +233,24 @@ static inline int arch_test_and_change_bit(unsigned long nr, · 232 unmodified lines | ||
| 233 | 233 | return test_and_change_bits(BIT_MASK(nr), addr + BIT_WORD(nr)) != 0; |
| 234 | 234 | } |
| 235 | 235 | |
| 236 | −#ifdef CONFIG_PPC64 | |
| 237 | −static inline unsigned long | |
| 238 | −clear_bit_unlock_return_word(int nr, volatile unsigned long *addr) | |
| 236 | +static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, | |
| 237 | + volatile unsigned long *p) | |
| 239 | 238 | { |
| 240 | 239 | unsigned long old, t; |
| 241 | − unsigned long *p = (unsigned long *)addr + BIT_WORD(nr); | |
| 242 | − unsigned long mask = BIT_MASK(nr); | |
| 243 | 240 | |
| 244 | 241 | __asm__ __volatile__ ( |
| 245 | 242 | PPC_RELEASE_BARRIER |
| 246 | 243 | "1:" PPC_LLARX "%0,0,%3,0\n" |
| 247 | − "andc %1,%0,%2\n" | |
| 244 | + "xor %1,%0,%2\n" | |
| 248 | 245 | PPC_STLCX "%1,0,%3\n" |
| 249 | 246 | "bne- 1b\n" |
| 250 | 247 | : "=&r" (old), "=&r" (t) |
| 251 | 248 | : "r" (mask), "r" (p) |
| 252 | 249 | : "cc", "memory"); |
| 253 | 250 | |
| 254 | − return old; | |
| 251 | + return (old & BIT_MASK(7)) != 0; | |
| 255 | 252 | } |
| 256 | − | |
| 257 | −/* | |
| 258 | − * This is a special function for mm/filemap.c | |
| 259 | − * Bit 7 corresponds to PG_waiters. | |
| 260 | − */ | |
| 261 | −#define arch_clear_bit_unlock_is_negative_byte(nr, addr) \ | |
| 262 | − (clear_bit_unlock_return_word(nr, addr) & BIT_MASK(7)) | |
| 263 | − | |
| 264 | −#endif /* CONFIG_PPC64 */ | |
| 253 | +#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte | |
| 265 | 254 | |
| 266 | 255 | #include <asm-generic/bitops/non-atomic.h> |
| 267 | 256 | |
| @@ -191,6 +191,18 @@ static inline void __clear_bit_unlock( · 190 unmodified lines | ||
| 191 | 191 | clear_bit_unlock(nr, addr); |
| 192 | 192 | } |
| 193 | 193 | |
| 194 | +static inline bool xor_unlock_is_negative_byte(unsigned long mask, | |
| 195 | + volatile unsigned long *addr) | |
| 196 | +{ | |
| 197 | + unsigned long res; | |
| 198 | + __asm__ __volatile__ ( | |
| 199 | + __AMO(xor) ".rl %0, %2, %1" | |
| 200 | + : "=r" (res), "+A" (*addr) | |
| 201 | + : "r" (__NOP(mask)) | |
| 202 | + : "memory"); | |
| 203 | + return (res & BIT(7)) != 0; | |
| 204 | +} | |
| 205 | + | |
| 194 | 206 | #undef __test_and_op_bit |
| 195 | 207 | #undef __op_bit |
| 196 | 208 | #undef __NOP |
| @@ -201,6 +201,16 @@ static inline void arch___clear_bit_unlock(unsigned long nr, · 200 unmodified lines | ||
| 201 | 201 | arch___clear_bit(nr, ptr); |
| 202 | 202 | } |
| 203 | 203 | |
| 204 | +static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, | |
| 205 | + volatile unsigned long *ptr) | |
| 206 | +{ | |
| 207 | + unsigned long old; | |
| 208 | + | |
| 209 | + old = __atomic64_xor_barrier(mask, (long *)ptr); | |
| 210 | + return old & BIT(7); | |
| 211 | +} | |
| 212 | +#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte | |
| 213 | + | |
| 204 | 214 | #include <asm-generic/bitops/instrumented-atomic.h> |
| 205 | 215 | #include <asm-generic/bitops/instrumented-non-atomic.h> |
| 206 | 216 | #include <asm-generic/bitops/instrumented-lock.h> |
| @@ -94,18 +94,17 @@ arch___clear_bit(unsigned long nr, volatile unsigned long *addr) · 93 unmodified lines | ||
| 94 | 94 | asm volatile(__ASM_SIZE(btr) " %1,%0" : : ADDR, "Ir" (nr) : "memory"); |
| 95 | 95 | } |
| 96 | 96 | |
| 97 | −static __always_inline bool | |
| 98 | −arch_clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr) | |
| 97 | +static __always_inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, | |
| 98 | + volatile unsigned long *addr) | |
| 99 | 99 | { |
| 100 | 100 | bool negative; |
| 101 | − asm volatile(LOCK_PREFIX "andb %2,%1" | |
| 101 | + asm volatile(LOCK_PREFIX "xorb %2,%1" | |
| 102 | 102 | CC_SET(s) |
| 103 | 103 | : CC_OUT(s) (negative), WBYTE_ADDR(addr) |
| 104 | − : "ir" ((char) ~(1 << nr)) : "memory"); | |
| 104 | + : "iq" ((char)mask) : "memory"); | |
| 105 | 105 | return negative; |
| 106 | 106 | } |
| 107 | −#define arch_clear_bit_unlock_is_negative_byte \ | |
| 108 | − arch_clear_bit_unlock_is_negative_byte | |
| 107 | +#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte | |
| 109 | 108 | |
| 110 | 109 | static __always_inline void |
| 111 | 110 | arch___clear_bit_unlock(long nr, volatile unsigned long *addr) |
| @@ -6800,11 +6800,7 @@ static unsigned long mmu_shrink_count(struct shrinker *shrink, · 6799 unmodified lines | ||
| 6800 | 6800 | return percpu_counter_read_positive(&kvm_total_used_mmu_pages); |
| 6801 | 6801 | } |
| 6802 | 6802 | |
| 6803 | −static struct shrinker mmu_shrinker = { | |
| 6804 | − .count_objects = mmu_shrink_count, | |
| 6805 | − .scan_objects = mmu_shrink_scan, | |
| 6806 | − .seeks = DEFAULT_SEEKS * 10, | |
| 6807 | −}; | |
| 6803 | +static struct shrinker *mmu_shrinker; | |
| 6808 | 6804 | |
| 6809 | 6805 | static void mmu_destroy_caches(void) |
| 6810 | 6806 | { |
| @@ -6937,10 +6933,16 @@ int kvm_mmu_vendor_module_init(void) · 126 unmodified lines | ||
| 6937 | 6933 | if (percpu_counter_init(&kvm_total_used_mmu_pages, 0, GFP_KERNEL)) |
| 6938 | 6934 | goto out; |
| 6939 | 6935 | |
| 6940 | − ret = register_shrinker(&mmu_shrinker, "x86-mmu"); | |
| 6941 | − if (ret) | |
| 6936 | + mmu_shrinker = shrinker_alloc(0, "x86-mmu"); | |
| 6937 | + if (!mmu_shrinker) | |
| 6942 | 6938 | goto out_shrinker; |
| 6943 | 6939 | |
| 6940 | + mmu_shrinker->count_objects = mmu_shrink_count; | |
| 6941 | + mmu_shrinker->scan_objects = mmu_shrink_scan; | |
| 6942 | + mmu_shrinker->seeks = DEFAULT_SEEKS * 10; | |
| 6943 | + | |
| 6944 | + shrinker_register(mmu_shrinker); | |
| 6945 | + | |
| 6944 | 6946 | return 0; |
| 6945 | 6947 | |
| 6946 | 6948 | out_shrinker: |
| @@ -6962,7 +6964,7 @@ void kvm_mmu_vendor_module_exit(void) · 15 unmodified lines | ||
| 6962 | 6964 | { |
| 6963 | 6965 | mmu_destroy_caches(); |
| 6964 | 6966 | percpu_counter_destroy(&kvm_total_used_mmu_pages); |
| 6965 | − unregister_shrinker(&mmu_shrinker); | |
| 6967 | + shrinker_free(mmu_shrinker); | |
| 6966 | 6968 | } |
| 6967 | 6969 | |
| 6968 | 6970 | /* |
| @@ -76,6 +76,9 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) · 75 unmodified lines | ||
| 76 | 76 | #if CONFIG_PGTABLE_LEVELS > 3 |
| 77 | 77 | void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) |
| 78 | 78 | { |
| 79 | + struct ptdesc *ptdesc = virt_to_ptdesc(pud); | |
| 80 | + | |
| 81 | + pagetable_pud_dtor(ptdesc); | |
| 79 | 82 | paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); |
| 80 | 83 | paravirt_tlb_remove_table(tlb, virt_to_page(pud)); |
| 81 | 84 | } |
| @@ -101,7 +101,7 @@ static void round_robin_cpu(unsigned int tsk_index) · 100 unmodified lines | ||
| 101 | 101 | for_each_cpu(cpu, pad_busy_cpus) |
| 102 | 102 | cpumask_or(tmp, tmp, topology_sibling_cpumask(cpu)); |
| 103 | 103 | cpumask_andnot(tmp, cpu_online_mask, tmp); |
| 104 | − /* avoid HT sibilings if possible */ | |
| 104 | + /* avoid HT siblings if possible */ | |
| 105 | 105 | if (cpumask_empty(tmp)) |
| 106 | 106 | cpumask_andnot(tmp, cpu_online_mask, pad_busy_cpus); |
| 107 | 107 | if (cpumask_empty(tmp)) { |
| @@ -24,6 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <linux/node.h> |
| 25 | 25 | #include <linux/sysfs.h> |
| 26 | 26 | #include <linux/dax.h> |
| 27 | +#include <linux/memory-tiers.h> | |
| 27 | 28 | |
| 28 | 29 | static u8 hmat_revision; |
| 29 | 30 | static int hmat_disable __initdata; |
| @@ -582,28 +583,25 @@ static int initiators_to_nodemask(unsigned long *p_nodes) · 552 unmodified lines | ||
| 582 | 583 | return 0; |
| 583 | 584 | } |
| 584 | 585 | |
| 585 | −static void hmat_register_target_initiators(struct memory_target *target) | |
| 586 | +static void hmat_update_target_attrs(struct memory_target *target, | |
| 587 | + unsigned long *p_nodes, int access) | |
| 586 | 588 | { |
| 587 | − static DECLARE_BITMAP(p_nodes, MAX_NUMNODES); | |
| 588 | 589 | struct memory_initiator *initiator; |
| 589 | − unsigned int mem_nid, cpu_nid; | |
| 590 | + unsigned int cpu_nid; | |
| 590 | 591 | struct memory_locality *loc = NULL; |
| 591 | 592 | u32 best = 0; |
| 592 | − bool access0done = false; | |
| 593 | 593 | int i; |
| 594 | 594 | |
| 595 | − mem_nid = pxm_to_node(target->memory_pxm); | |
| 595 | + bitmap_zero(p_nodes, MAX_NUMNODES); | |
| 596 | 596 | /* |
| 597 | − * If the Address Range Structure provides a local processor pxm, link | |
| 597 | + * If the Address Range Structure provides a local processor pxm, set | |
| 598 | 598 | * only that one. Otherwise, find the best performance attributes and |
| 599 | − * register all initiators that match. | |
| 599 | + * collect all initiators that match. | |
| 600 | 600 | */ |
| 601 | 601 | if (target->processor_pxm != PXM_INVAL) { |
| 602 | 602 | cpu_nid = pxm_to_node(target->processor_pxm); |
| 603 | − register_memory_node_under_compute_node(mem_nid, cpu_nid, 0); | |
| 604 | − access0done = true; | |
| 605 | − if (node_state(cpu_nid, N_CPU)) { | |
| 606 | − register_memory_node_under_compute_node(mem_nid, cpu_nid, 1); | |
| 603 | + if (access == 0 || node_state(cpu_nid, N_CPU)) { | |
| 604 | + set_bit(target->processor_pxm, p_nodes); | |
| 607 | 605 | return; |
| 608 | 606 | } |
| 609 | 607 | } |
| @@ -617,47 +615,10 @@ static void hmat_register_target_initiators(struct memory_target *target) · 7 unmodified lines | ||
| 617 | 615 | * We'll also use the sorting to prime the candidate nodes with known |
| 618 | 616 | * initiators. |
| 619 | 617 | */ |
| 620 | − bitmap_zero(p_nodes, MAX_NUMNODES); | |
| 621 | 618 | list_sort(NULL, &initiators, initiator_cmp); |
| 622 | 619 | if (initiators_to_nodemask(p_nodes) < 0) |
| 623 | 620 | return; |
| 624 | 621 | |
| 625 | − if (!access0done) { | |
| 626 | − for (i = WRITE_LATENCY; i <= READ_BANDWIDTH; i++) { | |
| 627 | − loc = localities_types[i]; | |
| 628 | − if (!loc) | |
| 629 | − continue; | |
| 630 | − | |
| 631 | − best = 0; | |
| 632 | − list_for_each_entry(initiator, &initiators, node) { | |
| 633 | − u32 value; | |
| 634 | − | |
| 635 | − if (!test_bit(initiator->processor_pxm, p_nodes)) | |
| 636 | − continue; | |
| 637 | − | |
| 638 | − value = hmat_initiator_perf(target, initiator, | |
| 639 | − loc->hmat_loc); | |
| 640 | − if (hmat_update_best(loc->hmat_loc->data_type, value, &best)) | |
| 641 | − bitmap_clear(p_nodes, 0, initiator->processor_pxm); | |
| 642 | − if (value != best) | |
| 643 | − clear_bit(initiator->processor_pxm, p_nodes); | |
| 644 | − } | |
| 645 | − if (best) | |
| 646 | − hmat_update_target_access(target, loc->hmat_loc->data_type, | |
| 647 | − best, 0); | |
| 648 | − } | |
| 649 | − | |
| 650 | − for_each_set_bit(i, p_nodes, MAX_NUMNODES) { | |
| 651 | − cpu_nid = pxm_to_node(i); | |
| 652 | − register_memory_node_under_compute_node(mem_nid, cpu_nid, 0); | |
| 653 | − } | |
| 654 | − } | |
| 655 | − | |
| 656 | − /* Access 1 ignores Generic Initiators */ | |
| 657 | − bitmap_zero(p_nodes, MAX_NUMNODES); | |
| 658 | − if (initiators_to_nodemask(p_nodes) < 0) | |
| 659 | − return; | |
| 660 | − | |
| 661 | 622 | for (i = WRITE_LATENCY; i <= READ_BANDWIDTH; i++) { |
| 662 | 623 | loc = localities_types[i]; |
| 663 | 624 | if (!loc) |
| @@ -667,7 +628,7 @@ static void hmat_register_target_initiators(struct memory_target *target) · 3 unmodified lines | ||
| 667 | 628 | list_for_each_entry(initiator, &initiators, node) { |
| 668 | 629 | u32 value; |
| 669 | 630 | |
| 670 | − if (!initiator->has_cpu) { | |
| 631 | + if (access == 1 && !initiator->has_cpu) { | |
| 671 | 632 | clear_bit(initiator->processor_pxm, p_nodes); |
| 672 | 633 | continue; |
| 673 | 634 | } |
| @@ -681,14 +642,33 @@ static void hmat_register_target_initiators(struct memory_target *target) · 7 unmodified lines | ||
| 681 | 642 | clear_bit(initiator->processor_pxm, p_nodes); |
| 682 | 643 | } |
| 683 | 644 | if (best) |
| 684 | − hmat_update_target_access(target, loc->hmat_loc->data_type, best, 1); | |
| 645 | + hmat_update_target_access(target, loc->hmat_loc->data_type, best, access); | |
| 685 | 646 | } |
| 647 | +} | |
| 648 | + | |
| 649 | +static void __hmat_register_target_initiators(struct memory_target *target, | |
| 650 | + unsigned long *p_nodes, | |
| 651 | + int access) | |
| 652 | +{ | |
| 653 | + unsigned int mem_nid, cpu_nid; | |
| 654 | + int i; | |
| 655 | + | |
| 656 | + mem_nid = pxm_to_node(target->memory_pxm); | |
| 657 | + hmat_update_target_attrs(target, p_nodes, access); | |
| 686 | 658 | for_each_set_bit(i, p_nodes, MAX_NUMNODES) { |
| 687 | 659 | cpu_nid = pxm_to_node(i); |
| 688 | − register_memory_node_under_compute_node(mem_nid, cpu_nid, 1); | |
| 660 | + register_memory_node_under_compute_node(mem_nid, cpu_nid, access); | |
| 689 | 661 | } |
| 690 | 662 | } |
| 691 | 663 | |
| 664 | +static void hmat_register_target_initiators(struct memory_target *target) | |
| 665 | +{ | |
| 666 | + static DECLARE_BITMAP(p_nodes, MAX_NUMNODES); | |
| 667 | + | |
| 668 | + __hmat_register_target_initiators(target, p_nodes, 0); | |
| 669 | + __hmat_register_target_initiators(target, p_nodes, 1); | |
| 670 | +} | |
| 671 | + | |
| 692 | 672 | static void hmat_register_target_cache(struct memory_target *target) |
| 693 | 673 | { |
| 694 | 674 | unsigned mem_nid = pxm_to_node(target->memory_pxm); |
| @@ -780,6 +760,61 @@ static int hmat_callback(struct notifier_block *self, · 85 unmodified lines | ||
| 780 | 760 | return NOTIFY_OK; |
| 781 | 761 | } |
| 782 | 762 | |
| 763 | +static int hmat_set_default_dram_perf(void) | |
| 764 | +{ | |
| 765 | + int rc; | |
| 766 | + int nid, pxm; | |
| 767 | + struct memory_target *target; | |
| 768 | + struct node_hmem_attrs *attrs; | |
| 769 | + | |
| 770 | + if (!default_dram_type) | |
| 771 | + return -EIO; | |
| 772 | + | |
| 773 | + for_each_node_mask(nid, default_dram_type->nodes) { | |
| 774 | + pxm = node_to_pxm(nid); | |
| 775 | + target = find_mem_target(pxm); | |
| 776 | + if (!target) | |
| 777 | + continue; | |
| 778 | + attrs = &target->hmem_attrs[1]; | |
| 779 | + rc = mt_set_default_dram_perf(nid, attrs, "ACPI HMAT"); | |
| 780 | + if (rc) | |
| 781 | + return rc; | |
| 782 | + } | |
| 783 | + | |
| 784 | + return 0; | |
| 785 | +} | |
| 786 | + | |
| 787 | +static int hmat_calculate_adistance(struct notifier_block *self, | |
| 788 | + unsigned long nid, void *data) | |
| 789 | +{ | |
| 790 | + static DECLARE_BITMAP(p_nodes, MAX_NUMNODES); | |
| 791 | + struct memory_target *target; | |
| 792 | + struct node_hmem_attrs *perf; | |
| 793 | + int *adist = data; | |
| 794 | + int pxm; | |
| 795 | + | |
| 796 | + pxm = node_to_pxm(nid); | |
| 797 | + target = find_mem_target(pxm); | |
| 798 | + if (!target) | |
| 799 | + return NOTIFY_OK; | |
| 800 | + | |
| 801 | + mutex_lock(&target_lock); | |
| 802 | + hmat_update_target_attrs(target, p_nodes, 1); | |
| 803 | + mutex_unlock(&target_lock); | |
| 804 | + | |
| 805 | + perf = &target->hmem_attrs[1]; | |
| 806 | + | |
| 807 | + if (mt_perf_to_adistance(perf, adist)) | |
| 808 | + return NOTIFY_OK; | |
| 809 | + | |
| 810 | + return NOTIFY_STOP; | |
| 811 | +} | |
| 812 | + | |
| 813 | +static struct notifier_block hmat_adist_nb __meminitdata = { | |
| 814 | + .notifier_call = hmat_calculate_adistance, | |
| 815 | + .priority = 100, | |
| 816 | +}; | |
| 817 | + | |
| 783 | 818 | static __init void hmat_free_structures(void) |
| 784 | 819 | { |
| 785 | 820 | struct memory_target *target, *tnext; |
| @@ -862,8 +897,13 @@ static __init int hmat_init(void) · 76 unmodified lines | ||
| 862 | 897 | hmat_register_targets(); |
| 863 | 898 | |
| 864 | 899 | /* Keep the table and structures if the notifier may use them */ |
| 865 | − if (!hotplug_memory_notifier(hmat_callback, HMAT_CALLBACK_PRI)) | |
| 866 | − return 0; | |
| 900 | + if (hotplug_memory_notifier(hmat_callback, HMAT_CALLBACK_PRI)) | |
| 901 | + goto out_put; | |
| 902 | + | |
| 903 | + if (!hmat_set_default_dram_perf()) | |
| 904 | + register_mt_adistance_algorithm(&hmat_adist_nb); | |
| 905 | + | |
| 906 | + return 0; | |
| 867 | 907 | out_put: |
| 868 | 908 | hmat_free_structures(); |
| 869 | 909 | acpi_put_table(tbl); |
| @@ -1053,11 +1053,7 @@ binder_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 1052 unmodified lines | ||
| 1053 | 1053 | NULL, sc->nr_to_scan); |
| 1054 | 1054 | } |
| 1055 | 1055 | |
| 1056 | −static struct shrinker binder_shrinker = { | |
| 1057 | − .count_objects = binder_shrink_count, | |
| 1058 | − .scan_objects = binder_shrink_scan, | |
| 1059 | − .seeks = DEFAULT_SEEKS, | |
| 1060 | −}; | |
| 1056 | +static struct shrinker *binder_shrinker; | |
| 1061 | 1057 | |
| 1062 | 1058 | /** |
| 1063 | 1059 | * binder_alloc_init() - called by binder_open() for per-proc initialization |
| @@ -1077,19 +1073,29 @@ void binder_alloc_init(struct binder_alloc *alloc) · 13 unmodified lines | ||
| 1077 | 1073 | |
| 1078 | 1074 | int binder_alloc_shrinker_init(void) |
| 1079 | 1075 | { |
| 1080 | − int ret = list_lru_init(&binder_alloc_lru); | |
| 1076 | + int ret; | |
| 1081 | 1077 | |
| 1082 | − if (ret == 0) { | |
| 1083 | − ret = register_shrinker(&binder_shrinker, "android-binder"); | |
| 1084 | − if (ret) | |
| 1085 | − list_lru_destroy(&binder_alloc_lru); | |
| 1078 | + ret = list_lru_init(&binder_alloc_lru); | |
| 1079 | + if (ret) | |
| 1080 | + return ret; | |
| 1081 | + | |
| 1082 | + binder_shrinker = shrinker_alloc(0, "android-binder"); | |
| 1083 | + if (!binder_shrinker) { | |
| 1084 | + list_lru_destroy(&binder_alloc_lru); | |
| 1085 | + return -ENOMEM; | |
| 1086 | 1086 | } |
| 1087 | − return ret; | |
| 1087 | + | |
| 1088 | + binder_shrinker->count_objects = binder_shrink_count; | |
| 1089 | + binder_shrinker->scan_objects = binder_shrink_scan; | |
| 1090 | + | |
| 1091 | + shrinker_register(binder_shrinker); | |
| 1092 | + | |
| 1093 | + return 0; | |
| 1088 | 1094 | } |
| 1089 | 1095 | |
| 1090 | 1096 | void binder_alloc_shrinker_exit(void) |
| 1091 | 1097 | { |
| 1092 | − unregister_shrinker(&binder_shrinker); | |
| 1098 | + shrinker_free(binder_shrinker); | |
| 1093 | 1099 | list_lru_destroy(&binder_alloc_lru); |
| 1094 | 1100 | } |
| 1095 | 1101 | |
| @@ -898,6 +898,48 @@ err: · 897 unmodified lines | ||
| 898 | 898 | return rc; |
| 899 | 899 | } |
| 900 | 900 | |
| 901 | +/* | |
| 902 | + * Calculate the size of the per-CPU data cache slice. This can be | |
| 903 | + * used to estimate the size of the data cache slice that can be used | |
| 904 | + * by one CPU under ideal circumstances. UNIFIED caches are counted | |
| 905 | + * in addition to DATA caches. So, please consider code cache usage | |
| 906 | + * when use the result. | |
| 907 | + * | |
| 908 | + * Because the cache inclusive/non-inclusive information isn't | |
| 909 | + * available, we just use the size of the per-CPU slice of LLC to make | |
| 910 | + * the result more predictable across architectures. | |
| 911 | + */ | |
| 912 | +static void update_per_cpu_data_slice_size_cpu(unsigned int cpu) | |
| 913 | +{ | |
| 914 | + struct cpu_cacheinfo *ci; | |
| 915 | + struct cacheinfo *llc; | |
| 916 | + unsigned int nr_shared; | |
| 917 | + | |
| 918 | + if (!last_level_cache_is_valid(cpu)) | |
| 919 | + return; | |
| 920 | + | |
| 921 | + ci = ci_cacheinfo(cpu); | |
| 922 | + llc = per_cpu_cacheinfo_idx(cpu, cache_leaves(cpu) - 1); | |
| 923 | + | |
| 924 | + if (llc->type != CACHE_TYPE_DATA && llc->type != CACHE_TYPE_UNIFIED) | |
| 925 | + return; | |
| 926 | + | |
| 927 | + nr_shared = cpumask_weight(&llc->shared_cpu_map); | |
| 928 | + if (nr_shared) | |
| 929 | + ci->per_cpu_data_slice_size = llc->size / nr_shared; | |
| 930 | +} | |
| 931 | + | |
| 932 | +static void update_per_cpu_data_slice_size(bool cpu_online, unsigned int cpu) | |
| 933 | +{ | |
| 934 | + unsigned int icpu; | |
| 935 | + | |
| 936 | + for_each_online_cpu(icpu) { | |
| 937 | + if (!cpu_online && icpu == cpu) | |
| 938 | + continue; | |
| 939 | + update_per_cpu_data_slice_size_cpu(icpu); | |
| 940 | + } | |
| 941 | +} | |
| 942 | + | |
| 901 | 943 | static int cacheinfo_cpu_online(unsigned int cpu) |
| 902 | 944 | { |
| 903 | 945 | int rc = detect_cache_attributes(cpu); |
| @@ -906,7 +948,12 @@ static int cacheinfo_cpu_online(unsigned int cpu) · 2 unmodified lines | ||
| 906 | 948 | return rc; |
| 907 | 949 | rc = cache_add_dev(cpu); |
| 908 | 950 | if (rc) |
| 909 | − free_cache_attributes(cpu); | |
| 951 | + goto err; | |
| 952 | + update_per_cpu_data_slice_size(true, cpu); | |
| 953 | + setup_pcp_cacheinfo(); | |
| 954 | + return 0; | |
| 955 | +err: | |
| 956 | + free_cache_attributes(cpu); | |
| 910 | 957 | return rc; |
| 911 | 958 | } |
| 912 | 959 | |
| @@ -916,6 +963,8 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu) · 3 unmodified lines | ||
| 916 | 963 | cpu_cache_sysfs_exit(cpu); |
| 917 | 964 | |
| 918 | 965 | free_cache_attributes(cpu); |
| 966 | + update_per_cpu_data_slice_size(false, cpu); | |
| 967 | + setup_pcp_cacheinfo(); | |
| 919 | 968 | return 0; |
| 920 | 969 | } |
| 921 | 970 | |
| @@ -49,14 +49,52 @@ struct dax_kmem_data { · 48 unmodified lines | ||
| 49 | 49 | struct resource *res[]; |
| 50 | 50 | }; |
| 51 | 51 | |
| 52 | −static struct memory_dev_type *dax_slowmem_type; | |
| 52 | +static DEFINE_MUTEX(kmem_memory_type_lock); | |
| 53 | +static LIST_HEAD(kmem_memory_types); | |
| 54 | + | |
| 55 | +static struct memory_dev_type *kmem_find_alloc_memory_type(int adist) | |
| 56 | +{ | |
| 57 | + bool found = false; | |
| 58 | + struct memory_dev_type *mtype; | |
| 59 | + | |
| 60 | + mutex_lock(&kmem_memory_type_lock); | |
| 61 | + list_for_each_entry(mtype, &kmem_memory_types, list) { | |
| 62 | + if (mtype->adistance == adist) { | |
| 63 | + found = true; | |
| 64 | + break; | |
| 65 | + } | |
| 66 | + } | |
| 67 | + if (!found) { | |
| 68 | + mtype = alloc_memory_type(adist); | |
| 69 | + if (!IS_ERR(mtype)) | |
| 70 | + list_add(&mtype->list, &kmem_memory_types); | |
| 71 | + } | |
| 72 | + mutex_unlock(&kmem_memory_type_lock); | |
| 73 | + | |
| 74 | + return mtype; | |
| 75 | +} | |
| 76 | + | |
| 77 | +static void kmem_put_memory_types(void) | |
| 78 | +{ | |
| 79 | + struct memory_dev_type *mtype, *mtn; | |
| 80 | + | |
| 81 | + mutex_lock(&kmem_memory_type_lock); | |
| 82 | + list_for_each_entry_safe(mtype, mtn, &kmem_memory_types, list) { | |
| 83 | + list_del(&mtype->list); | |
| 84 | + put_memory_type(mtype); | |
| 85 | + } | |
| 86 | + mutex_unlock(&kmem_memory_type_lock); | |
| 87 | +} | |
| 88 | + | |
| 53 | 89 | static int dev_dax_kmem_probe(struct dev_dax *dev_dax) |
| 54 | 90 | { |
| 55 | 91 | struct device *dev = &dev_dax->dev; |
| 56 | 92 | unsigned long total_len = 0; |
| 57 | 93 | struct dax_kmem_data *data; |
| 94 | + struct memory_dev_type *mtype; | |
| 58 | 95 | int i, rc, mapped = 0; |
| 59 | 96 | int numa_node; |
| 97 | + int adist = MEMTIER_DEFAULT_DAX_ADISTANCE; | |
| 60 | 98 | |
| 61 | 99 | /* |
| 62 | 100 | * Ensure good NUMA information for the persistent memory. |
| @@ -71,6 +109,11 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · 8 unmodified lines | ||
| 71 | 109 | return -EINVAL; |
| 72 | 110 | } |
| 73 | 111 | |
| 112 | + mt_calc_adistance(numa_node, &adist); | |
| 113 | + mtype = kmem_find_alloc_memory_type(adist); | |
| 114 | + if (IS_ERR(mtype)) | |
| 115 | + return PTR_ERR(mtype); | |
| 116 | + | |
| 74 | 117 | for (i = 0; i < dev_dax->nr_range; i++) { |
| 75 | 118 | struct range range; |
| 76 | 119 | |
| @@ -88,7 +131,7 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) · 11 unmodified lines | ||
| 88 | 131 | return -EINVAL; |
| 89 | 132 | } |
| 90 | 133 | |
| 91 | − init_node_memory_type(numa_node, dax_slowmem_type); | |
| 134 | + init_node_memory_type(numa_node, mtype); | |
| 92 | 135 | |
| 93 | 136 | rc = -ENOMEM; |
| 94 | 137 | data = kzalloc(struct_size(data, res, dev_dax->nr_range), GFP_KERNEL); |
| @@ -167,7 +210,7 @@ err_reg_mgid: · 72 unmodified lines | ||
| 167 | 210 | err_res_name: |
| 168 | 211 | kfree(data); |
| 169 | 212 | err_dax_kmem_data: |
| 170 | − clear_node_memory_type(numa_node, dax_slowmem_type); | |
| 213 | + clear_node_memory_type(numa_node, mtype); | |
| 171 | 214 | return rc; |
| 172 | 215 | } |
| 173 | 216 | |
| @@ -219,7 +262,7 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax) · 45 unmodified lines | ||
| 219 | 262 | * for that. This implies this reference will be around |
| 220 | 263 | * till next reboot. |
| 221 | 264 | */ |
| 222 | − clear_node_memory_type(node, dax_slowmem_type); | |
| 265 | + clear_node_memory_type(node, NULL); | |
| 223 | 266 | } |
| 224 | 267 | } |
| 225 | 268 | #else |
| @@ -251,12 +294,6 @@ static int __init dax_kmem_init(void) · 25 unmodified lines | ||
| 251 | 294 | if (!kmem_name) |
| 252 | 295 | return -ENOMEM; |
| 253 | 296 | |
| 254 | − dax_slowmem_type = alloc_memory_type(MEMTIER_DEFAULT_DAX_ADISTANCE); | |
| 255 | − if (IS_ERR(dax_slowmem_type)) { | |
| 256 | − rc = PTR_ERR(dax_slowmem_type); | |
| 257 | − goto err_dax_slowmem_type; | |
| 258 | − } | |
| 259 | − | |
| 260 | 297 | rc = dax_driver_register(&device_dax_kmem_driver); |
| 261 | 298 | if (rc) |
| 262 | 299 | goto error_dax_driver; |
| @@ -264,8 +301,7 @@ static int __init dax_kmem_init(void) · 1 unmodified line | ||
| 264 | 301 | return rc; |
| 265 | 302 | |
| 266 | 303 | error_dax_driver: |
| 267 | − put_memory_type(dax_slowmem_type); | |
| 268 | −err_dax_slowmem_type: | |
| 304 | + kmem_put_memory_types(); | |
| 269 | 305 | kfree_const(kmem_name); |
| 270 | 306 | return rc; |
| 271 | 307 | } |
| @@ -275,7 +311,7 @@ static void __exit dax_kmem_exit(void) · 3 unmodified lines | ||
| 275 | 311 | dax_driver_unregister(&device_dax_kmem_driver); |
| 276 | 312 | if (!any_hotremove_failed) |
| 277 | 313 | kfree_const(kmem_name); |
| 278 | − put_memory_type(dax_slowmem_type); | |
| 314 | + kmem_put_memory_types(); | |
| 279 | 315 | } |
| 280 | 316 | |
| 281 | 317 | MODULE_AUTHOR("Intel Corporation"); |
| @@ -3,6 +3,7 @@ · 2 unmodified lines | ||
| 3 | 3 | #include <linux/efi.h> |
| 4 | 4 | #include <linux/memblock.h> |
| 5 | 5 | #include <linux/spinlock.h> |
| 6 | +#include <linux/crash_dump.h> | |
| 6 | 7 | #include <asm/unaccepted_memory.h> |
| 7 | 8 | |
| 8 | 9 | /* Protects unaccepted memory bitmap and accepting_list */ |
| @@ -201,3 +202,22 @@ bool range_contains_unaccepted_memory(phys_addr_t start, phys_addr_t end) · 192 unmodified lines | ||
| 201 | 202 | |
| 202 | 203 | return ret; |
| 203 | 204 | } |
| 205 | + | |
| 206 | +#ifdef CONFIG_PROC_VMCORE | |
| 207 | +static bool unaccepted_memory_vmcore_pfn_is_ram(struct vmcore_cb *cb, | |
| 208 | + unsigned long pfn) | |
| 209 | +{ | |
| 210 | + return !pfn_is_unaccepted_memory(pfn); | |
| 211 | +} | |
| 212 | + | |
| 213 | +static struct vmcore_cb vmcore_cb = { | |
| 214 | + .pfn_is_ram = unaccepted_memory_vmcore_pfn_is_ram, | |
| 215 | +}; | |
| 216 | + | |
| 217 | +static int __init unaccepted_memory_init_kdump(void) | |
| 218 | +{ | |
| 219 | + register_vmcore_cb(&vmcore_cb); | |
| 220 | + return 0; | |
| 221 | +} | |
| 222 | +core_initcall(unaccepted_memory_init_kdump); | |
| 223 | +#endif /* CONFIG_PROC_VMCORE */ | |
| @@ -288,8 +288,7 @@ unsigned long i915_gem_shrink_all(struct drm_i915_private *i915) · 287 unmodified lines | ||
| 288 | 288 | static unsigned long |
| 289 | 289 | i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) |
| 290 | 290 | { |
| 291 | − struct drm_i915_private *i915 = | |
| 292 | − container_of(shrinker, struct drm_i915_private, mm.shrinker); | |
| 291 | + struct drm_i915_private *i915 = shrinker->private_data; | |
| 293 | 292 | unsigned long num_objects; |
| 294 | 293 | unsigned long count; |
| 295 | 294 | |
| @@ -306,8 +305,8 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) · 10 unmodified lines | ||
| 306 | 305 | if (num_objects) { |
| 307 | 306 | unsigned long avg = 2 * count / num_objects; |
| 308 | 307 | |
| 309 | − i915->mm.shrinker.batch = | |
| 310 | − max((i915->mm.shrinker.batch + avg) >> 1, | |
| 308 | + i915->mm.shrinker->batch = | |
| 309 | + max((i915->mm.shrinker->batch + avg) >> 1, | |
| 311 | 310 | 128ul /* default SHRINK_BATCH */); |
| 312 | 311 | } |
| 313 | 312 | |
| @@ -317,8 +316,7 @@ i915_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) · 3 unmodified lines | ||
| 317 | 316 | static unsigned long |
| 318 | 317 | i915_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) |
| 319 | 318 | { |
| 320 | − struct drm_i915_private *i915 = | |
| 321 | − container_of(shrinker, struct drm_i915_private, mm.shrinker); | |
| 319 | + struct drm_i915_private *i915 = shrinker->private_data; | |
| 322 | 320 | unsigned long freed; |
| 323 | 321 | |
| 324 | 322 | sc->nr_scanned = 0; |
| @@ -430,12 +428,17 @@ i915_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr · 105 unmodified lines | ||
| 430 | 428 | |
| 431 | 429 | void i915_gem_driver_register__shrinker(struct drm_i915_private *i915) |
| 432 | 430 | { |
| 433 | − i915->mm.shrinker.scan_objects = i915_gem_shrinker_scan; | |
| 434 | − i915->mm.shrinker.count_objects = i915_gem_shrinker_count; | |
| 435 | − i915->mm.shrinker.seeks = DEFAULT_SEEKS; | |
| 436 | − i915->mm.shrinker.batch = 4096; | |
| 437 | − drm_WARN_ON(&i915->drm, register_shrinker(&i915->mm.shrinker, | |
| 438 | − "drm-i915_gem")); | |
| 431 | + i915->mm.shrinker = shrinker_alloc(0, "drm-i915_gem"); | |
| 432 | + if (!i915->mm.shrinker) { | |
| 433 | + drm_WARN_ON(&i915->drm, 1); | |
| 434 | + } else { | |
| 435 | + i915->mm.shrinker->scan_objects = i915_gem_shrinker_scan; | |
| 436 | + i915->mm.shrinker->count_objects = i915_gem_shrinker_count; | |
| 437 | + i915->mm.shrinker->batch = 4096; | |
| 438 | + i915->mm.shrinker->private_data = i915; | |
| 439 | + | |
| 440 | + shrinker_register(i915->mm.shrinker); | |
| 441 | + } | |
| 439 | 442 | |
| 440 | 443 | i915->mm.oom_notifier.notifier_call = i915_gem_shrinker_oom; |
| 441 | 444 | drm_WARN_ON(&i915->drm, register_oom_notifier(&i915->mm.oom_notifier)); |
| @@ -451,7 +454,7 @@ void i915_gem_driver_unregister__shrinker(struct drm_i915_private *i915) · 9 unmodified lines | ||
| 451 | 454 | unregister_vmap_purge_notifier(&i915->mm.vmap_notifier)); |
| 452 | 455 | drm_WARN_ON(&i915->drm, |
| 453 | 456 | unregister_oom_notifier(&i915->mm.oom_notifier)); |
| 454 | − unregister_shrinker(&i915->mm.shrinker); | |
| 457 | + shrinker_free(i915->mm.shrinker); | |
| 455 | 458 | } |
| 456 | 459 | |
| 457 | 460 | void i915_gem_shrinker_taints_mutex(struct drm_i915_private *i915, |
| @@ -163,7 +163,7 @@ struct i915_gem_mm { · 162 unmodified lines | ||
| 163 | 163 | |
| 164 | 164 | struct notifier_block oom_notifier; |
| 165 | 165 | struct notifier_block vmap_notifier; |
| 166 | − struct shrinker shrinker; | |
| 166 | + struct shrinker *shrinker; | |
| 167 | 167 | |
| 168 | 168 | #ifdef CONFIG_MMU_NOTIFIER |
| 169 | 169 | /** |
| @@ -265,7 +265,9 @@ static int msm_drm_init(struct device *dev, const struct drm_driver *drv) · 264 unmodified lines | ||
| 265 | 265 | if (ret) |
| 266 | 266 | goto err_deinit_vram; |
| 267 | 267 | |
| 268 | − msm_gem_shrinker_init(ddev); | |
| 268 | + ret = msm_gem_shrinker_init(ddev); | |
| 269 | + if (ret) | |
| 270 | + goto err_msm_uninit; | |
| 269 | 271 | |
| 270 | 272 | if (priv->kms_init) { |
| 271 | 273 | ret = msm_drm_kms_init(dev, drv); |
| @@ -218,7 +218,7 @@ struct msm_drm_private { · 217 unmodified lines | ||
| 218 | 218 | } vram; |
| 219 | 219 | |
| 220 | 220 | struct notifier_block vmap_notifier; |
| 221 | − struct shrinker shrinker; | |
| 221 | + struct shrinker *shrinker; | |
| 222 | 222 | |
| 223 | 223 | struct drm_atomic_state *pm_state; |
| 224 | 224 | |
| @@ -280,7 +280,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data, · 55 unmodified lines | ||
| 280 | 280 | unsigned long msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan); |
| 281 | 281 | #endif |
| 282 | 282 | |
| 283 | −void msm_gem_shrinker_init(struct drm_device *dev); | |
| 283 | +int msm_gem_shrinker_init(struct drm_device *dev); | |
| 284 | 284 | void msm_gem_shrinker_cleanup(struct drm_device *dev); |
| 285 | 285 | |
| 286 | 286 | struct sg_table *msm_gem_prime_get_sg_table(struct drm_gem_object *obj); |
| @@ -34,8 +34,7 @@ static bool can_block(struct shrink_control *sc) · 33 unmodified lines | ||
| 34 | 34 | static unsigned long |
| 35 | 35 | msm_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) |
| 36 | 36 | { |
| 37 | − struct msm_drm_private *priv = | |
| 38 | − container_of(shrinker, struct msm_drm_private, shrinker); | |
| 37 | + struct msm_drm_private *priv = shrinker->private_data; | |
| 39 | 38 | unsigned count = priv->lru.dontneed.count; |
| 40 | 39 | |
| 41 | 40 | if (can_swap()) |
| @@ -100,8 +99,7 @@ active_evict(struct drm_gem_object *obj) · 58 unmodified lines | ||
| 100 | 99 | static unsigned long |
| 101 | 100 | msm_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) |
| 102 | 101 | { |
| 103 | − struct msm_drm_private *priv = | |
| 104 | − container_of(shrinker, struct msm_drm_private, shrinker); | |
| 102 | + struct msm_drm_private *priv = shrinker->private_data; | |
| 105 | 103 | struct { |
| 106 | 104 | struct drm_gem_lru *lru; |
| 107 | 105 | bool (*shrink)(struct drm_gem_object *obj); |
| @@ -148,10 +146,11 @@ msm_gem_shrinker_shrink(struct drm_device *dev, unsigned long nr_to_scan) · 40 unmodified lines | ||
| 148 | 146 | struct shrink_control sc = { |
| 149 | 147 | .nr_to_scan = nr_to_scan, |
| 150 | 148 | }; |
| 151 | − int ret; | |
| 149 | + unsigned long ret = SHRINK_STOP; | |
| 152 | 150 | |
| 153 | 151 | fs_reclaim_acquire(GFP_KERNEL); |
| 154 | − ret = msm_gem_shrinker_scan(&priv->shrinker, &sc); | |
| 152 | + if (priv->shrinker) | |
| 153 | + ret = msm_gem_shrinker_scan(priv->shrinker, &sc); | |
| 155 | 154 | fs_reclaim_release(GFP_KERNEL); |
| 156 | 155 | |
| 157 | 156 | return ret; |
| @@ -210,16 +209,24 @@ msm_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr) · 52 unmodified lines | ||
| 210 | 209 | * |
| 211 | 210 | * This function registers and sets up the msm shrinker. |
| 212 | 211 | */ |
| 213 | −void msm_gem_shrinker_init(struct drm_device *dev) | |
| 212 | +int msm_gem_shrinker_init(struct drm_device *dev) | |
| 214 | 213 | { |
| 215 | 214 | struct msm_drm_private *priv = dev->dev_private; |
| 216 | − priv->shrinker.count_objects = msm_gem_shrinker_count; | |
| 217 | − priv->shrinker.scan_objects = msm_gem_shrinker_scan; | |
| 218 | − priv->shrinker.seeks = DEFAULT_SEEKS; | |
| 219 | − WARN_ON(register_shrinker(&priv->shrinker, "drm-msm_gem")); | |
| 215 | + | |
| 216 | + priv->shrinker = shrinker_alloc(0, "drm-msm_gem"); | |
| 217 | + if (!priv->shrinker) | |
| 218 | + return -ENOMEM; | |
| 219 | + | |
| 220 | + priv->shrinker->count_objects = msm_gem_shrinker_count; | |
| 221 | + priv->shrinker->scan_objects = msm_gem_shrinker_scan; | |
| 222 | + priv->shrinker->private_data = priv; | |
| 223 | + | |
| 224 | + shrinker_register(priv->shrinker); | |
| 220 | 225 | |
| 221 | 226 | priv->vmap_notifier.notifier_call = msm_gem_shrinker_vmap; |
| 222 | 227 | WARN_ON(register_vmap_purge_notifier(&priv->vmap_notifier)); |
| 228 | + | |
| 229 | + return 0; | |
| 223 | 230 | } |
| 224 | 231 | |
| 225 | 232 | /** |
| @@ -232,8 +239,8 @@ void msm_gem_shrinker_cleanup(struct drm_device *dev) · 6 unmodified lines | ||
| 232 | 239 | { |
| 233 | 240 | struct msm_drm_private *priv = dev->dev_private; |
| 234 | 241 | |
| 235 | − if (priv->shrinker.nr_deferred) { | |
| 242 | + if (priv->shrinker) { | |
| 236 | 243 | WARN_ON(unregister_vmap_purge_notifier(&priv->vmap_notifier)); |
| 237 | − unregister_shrinker(&priv->shrinker); | |
| 244 | + shrinker_free(priv->shrinker); | |
| 238 | 245 | } |
| 239 | 246 | } |
| @@ -119,7 +119,7 @@ struct panfrost_device { · 118 unmodified lines | ||
| 119 | 119 | |
| 120 | 120 | struct mutex shrinker_lock; |
| 121 | 121 | struct list_head shrinker_list; |
| 122 | − struct shrinker shrinker; | |
| 122 | + struct shrinker *shrinker; | |
| 123 | 123 | |
| 124 | 124 | struct panfrost_devfreq pfdevfreq; |
| 125 | 125 | |
| @@ -659,10 +659,14 @@ static int panfrost_probe(struct platform_device *pdev) · 658 unmodified lines | ||
| 659 | 659 | if (err < 0) |
| 660 | 660 | goto err_out1; |
| 661 | 661 | |
| 662 | − panfrost_gem_shrinker_init(ddev); | |
| 662 | + err = panfrost_gem_shrinker_init(ddev); | |
| 663 | + if (err) | |
| 664 | + goto err_out2; | |
| 663 | 665 | |
| 664 | 666 | return 0; |
| 665 | 667 | |
| 668 | +err_out2: | |
| 669 | + drm_dev_unregister(ddev); | |
| 666 | 670 | err_out1: |
| 667 | 671 | pm_runtime_disable(pfdev->dev); |
| 668 | 672 | panfrost_device_fini(pfdev); |
| @@ -86,7 +86,7 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo, · 85 unmodified lines | ||
| 86 | 86 | void panfrost_gem_mapping_put(struct panfrost_gem_mapping *mapping); |
| 87 | 87 | void panfrost_gem_teardown_mappings_locked(struct panfrost_gem_object *bo); |
| 88 | 88 | |
| 89 | −void panfrost_gem_shrinker_init(struct drm_device *dev); | |
| 89 | +int panfrost_gem_shrinker_init(struct drm_device *dev); | |
| 90 | 90 | void panfrost_gem_shrinker_cleanup(struct drm_device *dev); |
| 91 | 91 | |
| 92 | 92 | #endif /* __PANFROST_GEM_H__ */ |
| @@ -18,8 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | static unsigned long |
| 19 | 19 | panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *sc) |
| 20 | 20 | { |
| 21 | − struct panfrost_device *pfdev = | |
| 22 | − container_of(shrinker, struct panfrost_device, shrinker); | |
| 21 | + struct panfrost_device *pfdev = shrinker->private_data; | |
| 23 | 22 | struct drm_gem_shmem_object *shmem; |
| 24 | 23 | unsigned long count = 0; |
| 25 | 24 | |
| @@ -65,8 +64,7 @@ unlock_mappings: · 39 unmodified lines | ||
| 65 | 64 | static unsigned long |
| 66 | 65 | panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) |
| 67 | 66 | { |
| 68 | − struct panfrost_device *pfdev = | |
| 69 | − container_of(shrinker, struct panfrost_device, shrinker); | |
| 67 | + struct panfrost_device *pfdev = shrinker->private_data; | |
| 70 | 68 | struct drm_gem_shmem_object *shmem, *tmp; |
| 71 | 69 | unsigned long freed = 0; |
| 72 | 70 | |
| @@ -97,13 +95,21 @@ panfrost_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) · 24 unmodified lines | ||
| 97 | 95 | * |
| 98 | 96 | * This function registers and sets up the panfrost shrinker. |
| 99 | 97 | */ |
| 100 | −void panfrost_gem_shrinker_init(struct drm_device *dev) | |
| 98 | +int panfrost_gem_shrinker_init(struct drm_device *dev) | |
| 101 | 99 | { |
| 102 | 100 | struct panfrost_device *pfdev = dev->dev_private; |
| 103 | − pfdev->shrinker.count_objects = panfrost_gem_shrinker_count; | |
| 104 | − pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan; | |
| 105 | − pfdev->shrinker.seeks = DEFAULT_SEEKS; | |
| 106 | − WARN_ON(register_shrinker(&pfdev->shrinker, "drm-panfrost")); | |
| 101 | + | |
| 102 | + pfdev->shrinker = shrinker_alloc(0, "drm-panfrost"); | |
| 103 | + if (!pfdev->shrinker) | |
| 104 | + return -ENOMEM; | |
| 105 | + | |
| 106 | + pfdev->shrinker->count_objects = panfrost_gem_shrinker_count; | |
| 107 | + pfdev->shrinker->scan_objects = panfrost_gem_shrinker_scan; | |
| 108 | + pfdev->shrinker->private_data = pfdev; | |
| 109 | + | |
| 110 | + shrinker_register(pfdev->shrinker); | |
| 111 | + | |
| 112 | + return 0; | |
| 107 | 113 | } |
| 108 | 114 | |
| 109 | 115 | /** |
| @@ -116,7 +122,6 @@ void panfrost_gem_shrinker_cleanup(struct drm_device *dev) · 6 unmodified lines | ||
| 116 | 122 | { |
| 117 | 123 | struct panfrost_device *pfdev = dev->dev_private; |
| 118 | 124 | |
| 119 | − if (pfdev->shrinker.nr_deferred) { | |
| 120 | − unregister_shrinker(&pfdev->shrinker); | |
| 121 | − } | |
| 125 | + if (pfdev->shrinker) | |
| 126 | + shrinker_free(pfdev->shrinker); | |
| 122 | 127 | } |
| @@ -73,7 +73,8 @@ static struct ttm_pool_type global_dma32_uncached[MAX_ORDER + 1]; · 72 unmodified lines | ||
| 73 | 73 | |
| 74 | 74 | static spinlock_t shrinker_lock; |
| 75 | 75 | static struct list_head shrinker_list; |
| 76 | −static struct shrinker mm_shrinker; | |
| 76 | +static struct shrinker *mm_shrinker; | |
| 77 | +static DECLARE_RWSEM(pool_shrink_rwsem); | |
| 77 | 78 | |
| 78 | 79 | /* Allocate pages of size 1 << order with the given gfp_flags */ |
| 79 | 80 | static struct page *ttm_pool_alloc_page(struct ttm_pool *pool, gfp_t gfp_flags, |
| @@ -317,6 +318,7 @@ static unsigned int ttm_pool_shrink(void) · 237 unmodified lines | ||
| 317 | 318 | unsigned int num_pages; |
| 318 | 319 | struct page *p; |
| 319 | 320 | |
| 321 | + down_read(&pool_shrink_rwsem); | |
| 320 | 322 | spin_lock(&shrinker_lock); |
| 321 | 323 | pt = list_first_entry(&shrinker_list, typeof(*pt), shrinker_list); |
| 322 | 324 | list_move_tail(&pt->shrinker_list, &shrinker_list); |
| @@ -329,6 +331,7 @@ static unsigned int ttm_pool_shrink(void) · 6 unmodified lines | ||
| 329 | 331 | } else { |
| 330 | 332 | num_pages = 0; |
| 331 | 333 | } |
| 334 | + up_read(&pool_shrink_rwsem); | |
| 332 | 335 | |
| 333 | 336 | return num_pages; |
| 334 | 337 | } |
| @@ -572,6 +575,18 @@ void ttm_pool_init(struct ttm_pool *pool, struct device *dev, · 237 unmodified lines | ||
| 572 | 575 | } |
| 573 | 576 | EXPORT_SYMBOL(ttm_pool_init); |
| 574 | 577 | |
| 578 | +/** | |
| 579 | + * ttm_pool_synchronize_shrinkers - Wait for all running shrinkers to complete. | |
| 580 | + * | |
| 581 | + * This is useful to guarantee that all shrinker invocations have seen an | |
| 582 | + * update, before freeing memory, similar to rcu. | |
| 583 | + */ | |
| 584 | +static void ttm_pool_synchronize_shrinkers(void) | |
| 585 | +{ | |
| 586 | + down_write(&pool_shrink_rwsem); | |
| 587 | + up_write(&pool_shrink_rwsem); | |
| 588 | +} | |
| 589 | + | |
| 575 | 590 | /** |
| 576 | 591 | * ttm_pool_fini - Cleanup a pool |
| 577 | 592 | * |
| @@ -593,7 +608,7 @@ void ttm_pool_fini(struct ttm_pool *pool) · 15 unmodified lines | ||
| 593 | 608 | /* We removed the pool types from the LRU, but we need to also make sure |
| 594 | 609 | * that no shrinker is concurrently freeing pages from the pool. |
| 595 | 610 | */ |
| 596 | − synchronize_shrinkers(); | |
| 611 | + ttm_pool_synchronize_shrinkers(); | |
| 597 | 612 | } |
| 598 | 613 | EXPORT_SYMBOL(ttm_pool_fini); |
| 599 | 614 | |
| @@ -734,8 +749,8 @@ static int ttm_pool_debugfs_shrink_show(struct seq_file *m, void *data) · 134 unmodified lines | ||
| 734 | 749 | struct shrink_control sc = { .gfp_mask = GFP_NOFS }; |
| 735 | 750 | |
| 736 | 751 | fs_reclaim_acquire(GFP_KERNEL); |
| 737 | − seq_printf(m, "%lu/%lu\n", ttm_pool_shrinker_count(&mm_shrinker, &sc), | |
| 738 | − ttm_pool_shrinker_scan(&mm_shrinker, &sc)); | |
| 752 | + seq_printf(m, "%lu/%lu\n", ttm_pool_shrinker_count(mm_shrinker, &sc), | |
| 753 | + ttm_pool_shrinker_scan(mm_shrinker, &sc)); | |
| 739 | 754 | fs_reclaim_release(GFP_KERNEL); |
| 740 | 755 | |
| 741 | 756 | return 0; |
| @@ -779,10 +794,17 @@ int ttm_pool_mgr_init(unsigned long num_pages) · 37 unmodified lines | ||
| 779 | 794 | &ttm_pool_debugfs_shrink_fops); |
| 780 | 795 | #endif |
| 781 | 796 | |
| 782 | − mm_shrinker.count_objects = ttm_pool_shrinker_count; | |
| 783 | − mm_shrinker.scan_objects = ttm_pool_shrinker_scan; | |
| 784 | − mm_shrinker.seeks = 1; | |
| 785 | − return register_shrinker(&mm_shrinker, "drm-ttm_pool"); | |
| 797 | + mm_shrinker = shrinker_alloc(0, "drm-ttm_pool"); | |
| 798 | + if (!mm_shrinker) | |
| 799 | + return -ENOMEM; | |
| 800 | + | |
| 801 | + mm_shrinker->count_objects = ttm_pool_shrinker_count; | |
| 802 | + mm_shrinker->scan_objects = ttm_pool_shrinker_scan; | |
| 803 | + mm_shrinker->seeks = 1; | |
| 804 | + | |
| 805 | + shrinker_register(mm_shrinker); | |
| 806 | + | |
| 807 | + return 0; | |
| 786 | 808 | } |
| 787 | 809 | |
| 788 | 810 | /** |
| @@ -802,6 +824,6 @@ void ttm_pool_mgr_fini(void) · 13 unmodified lines | ||
| 802 | 824 | ttm_pool_type_fini(&global_dma32_uncached[i]); |
| 803 | 825 | } |
| 804 | 826 | |
| 805 | − unregister_shrinker(&mm_shrinker); | |
| 827 | + shrinker_free(mm_shrinker); | |
| 806 | 828 | WARN_ON(!list_empty(&shrinker_list)); |
| 807 | 829 | } |
| @@ -543,7 +543,7 @@ struct cache_set { · 542 unmodified lines | ||
| 543 | 543 | struct bio_set bio_split; |
| 544 | 544 | |
| 545 | 545 | /* For the btree cache */ |
| 546 | − struct shrinker shrink; | |
| 546 | + struct shrinker *shrink; | |
| 547 | 547 | |
| 548 | 548 | /* For the btree cache and anything allocation related */ |
| 549 | 549 | struct mutex bucket_lock; |
| @@ -667,7 +667,7 @@ out_unlock: · 666 unmodified lines | ||
| 667 | 667 | static unsigned long bch_mca_scan(struct shrinker *shrink, |
| 668 | 668 | struct shrink_control *sc) |
| 669 | 669 | { |
| 670 | − struct cache_set *c = container_of(shrink, struct cache_set, shrink); | |
| 670 | + struct cache_set *c = shrink->private_data; | |
| 671 | 671 | struct btree *b, *t; |
| 672 | 672 | unsigned long i, nr = sc->nr_to_scan; |
| 673 | 673 | unsigned long freed = 0; |
| @@ -734,7 +734,7 @@ out: · 60 unmodified lines | ||
| 734 | 734 | static unsigned long bch_mca_count(struct shrinker *shrink, |
| 735 | 735 | struct shrink_control *sc) |
| 736 | 736 | { |
| 737 | − struct cache_set *c = container_of(shrink, struct cache_set, shrink); | |
| 737 | + struct cache_set *c = shrink->private_data; | |
| 738 | 738 | |
| 739 | 739 | if (c->shrinker_disabled) |
| 740 | 740 | return 0; |
| @@ -752,8 +752,8 @@ void bch_btree_cache_free(struct cache_set *c) · 11 unmodified lines | ||
| 752 | 752 | |
| 753 | 753 | closure_init_stack(&cl); |
| 754 | 754 | |
| 755 | − if (c->shrink.list.next) | |
| 756 | − unregister_shrinker(&c->shrink); | |
| 755 | + if (c->shrink) | |
| 756 | + shrinker_free(c->shrink); | |
| 757 | 757 | |
| 758 | 758 | mutex_lock(&c->bucket_lock); |
| 759 | 759 | |
| @@ -828,14 +828,19 @@ int bch_btree_cache_alloc(struct cache_set *c) · 68 unmodified lines | ||
| 828 | 828 | c->verify_data = NULL; |
| 829 | 829 | #endif |
| 830 | 830 | |
| 831 | − c->shrink.count_objects = bch_mca_count; | |
| 832 | − c->shrink.scan_objects = bch_mca_scan; | |
| 833 | − c->shrink.seeks = 4; | |
| 834 | − c->shrink.batch = c->btree_pages * 2; | |
| 831 | + c->shrink = shrinker_alloc(0, "md-bcache:%pU", c->set_uuid); | |
| 832 | + if (!c->shrink) { | |
| 833 | + pr_warn("bcache: %s: could not allocate shrinker\n", __func__); | |
| 834 | + return 0; | |
| 835 | + } | |
| 836 | + | |
| 837 | + c->shrink->count_objects = bch_mca_count; | |
| 838 | + c->shrink->scan_objects = bch_mca_scan; | |
| 839 | + c->shrink->seeks = 4; | |
| 840 | + c->shrink->batch = c->btree_pages * 2; | |
| 841 | + c->shrink->private_data = c; | |
| 835 | 842 | |
| 836 | − if (register_shrinker(&c->shrink, "md-bcache:%pU", c->set_uuid)) | |
| 837 | − pr_warn("bcache: %s: could not register shrinker\n", | |
| 838 | − __func__); | |
| 843 | + shrinker_register(c->shrink); | |
| 839 | 844 | |
| 840 | 845 | return 0; |
| 841 | 846 | } |
| @@ -866,7 +866,8 @@ STORE(__bch_cache_set) · 865 unmodified lines | ||
| 866 | 866 | |
| 867 | 867 | sc.gfp_mask = GFP_KERNEL; |
| 868 | 868 | sc.nr_to_scan = strtoul_or_return(buf); |
| 869 | − c->shrink.scan_objects(&c->shrink, &sc); | |
| 869 | + if (c->shrink) | |
| 870 | + c->shrink->scan_objects(c->shrink, &sc); | |
| 870 | 871 | } |
| 871 | 872 | |
| 872 | 873 | sysfs_strtoul_clamp(congested_read_threshold_us, |
| @@ -963,7 +963,7 @@ struct dm_bufio_client { · 962 unmodified lines | ||
| 963 | 963 | |
| 964 | 964 | sector_t start; |
| 965 | 965 | |
| 966 | − struct shrinker shrinker; | |
| 966 | + struct shrinker *shrinker; | |
| 967 | 967 | struct work_struct shrink_work; |
| 968 | 968 | atomic_long_t need_shrink; |
| 969 | 969 | |
| @@ -2368,7 +2368,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink · 1398 unmodified lines | ||
| 2368 | 2368 | { |
| 2369 | 2369 | struct dm_bufio_client *c; |
| 2370 | 2370 | |
| 2371 | − c = container_of(shrink, struct dm_bufio_client, shrinker); | |
| 2371 | + c = shrink->private_data; | |
| 2372 | 2372 | atomic_long_add(sc->nr_to_scan, &c->need_shrink); |
| 2373 | 2373 | queue_work(dm_bufio_wq, &c->shrink_work); |
| 2374 | 2374 | |
| @@ -2377,7 +2377,7 @@ static unsigned long dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink · 2 unmodified lines | ||
| 2377 | 2377 | |
| 2378 | 2378 | static unsigned long dm_bufio_shrink_count(struct shrinker *shrink, struct shrink_control *sc) |
| 2379 | 2379 | { |
| 2380 | − struct dm_bufio_client *c = container_of(shrink, struct dm_bufio_client, shrinker); | |
| 2380 | + struct dm_bufio_client *c = shrink->private_data; | |
| 2381 | 2381 | unsigned long count = cache_total(&c->cache); |
| 2382 | 2382 | unsigned long retain_target = get_retain_buffers(c); |
| 2383 | 2383 | unsigned long queued_for_cleanup = atomic_long_read(&c->need_shrink); |
| @@ -2490,14 +2490,20 @@ struct dm_bufio_client *dm_bufio_client_create(struct block_device *bdev, unsign · 106 unmodified lines | ||
| 2490 | 2490 | INIT_WORK(&c->shrink_work, shrink_work); |
| 2491 | 2491 | atomic_long_set(&c->need_shrink, 0); |
| 2492 | 2492 | |
| 2493 | − c->shrinker.count_objects = dm_bufio_shrink_count; | |
| 2494 | − c->shrinker.scan_objects = dm_bufio_shrink_scan; | |
| 2495 | − c->shrinker.seeks = 1; | |
| 2496 | − c->shrinker.batch = 0; | |
| 2497 | − r = register_shrinker(&c->shrinker, "dm-bufio:(%u:%u)", | |
| 2498 | − MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev)); | |
| 2499 | − if (r) | |
| 2493 | + c->shrinker = shrinker_alloc(0, "dm-bufio:(%u:%u)", | |
| 2494 | + MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev)); | |
| 2495 | + if (!c->shrinker) { | |
| 2496 | + r = -ENOMEM; | |
| 2500 | 2497 | goto bad; |
| 2498 | + } | |
| 2499 | + | |
| 2500 | + c->shrinker->count_objects = dm_bufio_shrink_count; | |
| 2501 | + c->shrinker->scan_objects = dm_bufio_shrink_scan; | |
| 2502 | + c->shrinker->seeks = 1; | |
| 2503 | + c->shrinker->batch = 0; | |
| 2504 | + c->shrinker->private_data = c; | |
| 2505 | + | |
| 2506 | + shrinker_register(c->shrinker); | |
| 2501 | 2507 | |
| 2502 | 2508 | mutex_lock(&dm_bufio_clients_lock); |
| 2503 | 2509 | dm_bufio_client_count++; |
| @@ -2537,7 +2543,7 @@ void dm_bufio_client_destroy(struct dm_bufio_client *c) · 33 unmodified lines | ||
| 2537 | 2543 | |
| 2538 | 2544 | drop_buffers(c); |
| 2539 | 2545 | |
| 2540 | − unregister_shrinker(&c->shrinker); | |
| 2546 | + shrinker_free(c->shrinker); | |
| 2541 | 2547 | flush_work(&c->shrink_work); |
| 2542 | 2548 | |
| 2543 | 2549 | mutex_lock(&dm_bufio_clients_lock); |
| @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) · 1827 unmodified lines | ||
| 1828 | 1828 | * Replacement block manager (new_bm) is created and old_bm destroyed outside of |
| 1829 | 1829 | * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of |
| 1830 | 1830 | * shrinker associated with the block manager's bufio client vs cmd root_lock). |
| 1831 | − * - must take shrinker_rwsem without holding cmd->root_lock | |
| 1831 | + * - must take shrinker_mutex without holding cmd->root_lock | |
| 1832 | 1832 | */ |
| 1833 | 1833 | new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, |
| 1834 | 1834 | CACHE_MAX_CONCURRENT_LOCKS); |
| @@ -187,7 +187,7 @@ struct dmz_metadata { · 186 unmodified lines | ||
| 187 | 187 | struct rb_root mblk_rbtree; |
| 188 | 188 | struct list_head mblk_lru_list; |
| 189 | 189 | struct list_head mblk_dirty_list; |
| 190 | − struct shrinker mblk_shrinker; | |
| 190 | + struct shrinker *mblk_shrinker; | |
| 191 | 191 | |
| 192 | 192 | /* Zone allocation management */ |
| 193 | 193 | struct mutex map_lock; |
| @@ -615,7 +615,7 @@ static unsigned long dmz_shrink_mblock_cache(struct dmz_metadata *zmd, · 421 unmodified lines | ||
| 615 | 615 | static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink, |
| 616 | 616 | struct shrink_control *sc) |
| 617 | 617 | { |
| 618 | − struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker); | |
| 618 | + struct dmz_metadata *zmd = shrink->private_data; | |
| 619 | 619 | |
| 620 | 620 | return atomic_read(&zmd->nr_mblks); |
| 621 | 621 | } |
| @@ -626,7 +626,7 @@ static unsigned long dmz_mblock_shrinker_count(struct shrinker *shrink, · 4 unmodified lines | ||
| 626 | 626 | static unsigned long dmz_mblock_shrinker_scan(struct shrinker *shrink, |
| 627 | 627 | struct shrink_control *sc) |
| 628 | 628 | { |
| 629 | − struct dmz_metadata *zmd = container_of(shrink, struct dmz_metadata, mblk_shrinker); | |
| 629 | + struct dmz_metadata *zmd = shrink->private_data; | |
| 630 | 630 | unsigned long count; |
| 631 | 631 | |
| 632 | 632 | spin_lock(&zmd->mblk_lock); |
| @@ -2936,19 +2936,23 @@ int dmz_ctr_metadata(struct dmz_dev *dev, int num_dev, · 2303 unmodified lines | ||
| 2936 | 2936 | */ |
| 2937 | 2937 | zmd->min_nr_mblks = 2 + zmd->nr_map_blocks + zmd->zone_nr_bitmap_blocks * 16; |
| 2938 | 2938 | zmd->max_nr_mblks = zmd->min_nr_mblks + 512; |
| 2939 | − zmd->mblk_shrinker.count_objects = dmz_mblock_shrinker_count; | |
| 2940 | − zmd->mblk_shrinker.scan_objects = dmz_mblock_shrinker_scan; | |
| 2941 | − zmd->mblk_shrinker.seeks = DEFAULT_SEEKS; | |
| 2942 | 2939 | |
| 2943 | 2940 | /* Metadata cache shrinker */ |
| 2944 | − ret = register_shrinker(&zmd->mblk_shrinker, "dm-zoned-meta:(%u:%u)", | |
| 2945 | − MAJOR(dev->bdev->bd_dev), | |
| 2946 | − MINOR(dev->bdev->bd_dev)); | |
| 2947 | − if (ret) { | |
| 2948 | − dmz_zmd_err(zmd, "Register metadata cache shrinker failed"); | |
| 2941 | + zmd->mblk_shrinker = shrinker_alloc(0, "dm-zoned-meta:(%u:%u)", | |
| 2942 | + MAJOR(dev->bdev->bd_dev), | |
| 2943 | + MINOR(dev->bdev->bd_dev)); | |
| 2944 | + if (!zmd->mblk_shrinker) { | |
| 2945 | + ret = -ENOMEM; | |
| 2946 | + dmz_zmd_err(zmd, "Allocate metadata cache shrinker failed"); | |
| 2949 | 2947 | goto err; |
| 2950 | 2948 | } |
| 2951 | 2949 | |
| 2950 | + zmd->mblk_shrinker->count_objects = dmz_mblock_shrinker_count; | |
| 2951 | + zmd->mblk_shrinker->scan_objects = dmz_mblock_shrinker_scan; | |
| 2952 | + zmd->mblk_shrinker->private_data = zmd; | |
| 2953 | + | |
| 2954 | + shrinker_register(zmd->mblk_shrinker); | |
| 2955 | + | |
| 2952 | 2956 | dmz_zmd_info(zmd, "DM-Zoned metadata version %d", zmd->sb_version); |
| 2953 | 2957 | for (i = 0; i < zmd->nr_devs; i++) |
| 2954 | 2958 | dmz_print_dev(zmd, i); |
| @@ -2995,7 +2999,7 @@ err: · 40 unmodified lines | ||
| 2995 | 2999 | */ |
| 2996 | 3000 | void dmz_dtr_metadata(struct dmz_metadata *zmd) |
| 2997 | 3001 | { |
| 2998 | − unregister_shrinker(&zmd->mblk_shrinker); | |
| 3002 | + shrinker_free(zmd->mblk_shrinker); | |
| 2999 | 3003 | dmz_cleanup_metadata(zmd); |
| 3000 | 3004 | kfree(zmd); |
| 3001 | 3005 | } |
| @@ -7378,7 +7378,7 @@ static void free_conf(struct r5conf *conf) · 7377 unmodified lines | ||
| 7378 | 7378 | |
| 7379 | 7379 | log_exit(conf); |
| 7380 | 7380 | |
| 7381 | − unregister_shrinker(&conf->shrinker); | |
| 7381 | + shrinker_free(conf->shrinker); | |
| 7382 | 7382 | free_thread_groups(conf); |
| 7383 | 7383 | shrink_stripes(conf); |
| 7384 | 7384 | raid5_free_percpu(conf); |
| @@ -7426,7 +7426,7 @@ static int raid5_alloc_percpu(struct r5conf *conf) · 41 unmodified lines | ||
| 7426 | 7426 | static unsigned long raid5_cache_scan(struct shrinker *shrink, |
| 7427 | 7427 | struct shrink_control *sc) |
| 7428 | 7428 | { |
| 7429 | − struct r5conf *conf = container_of(shrink, struct r5conf, shrinker); | |
| 7429 | + struct r5conf *conf = shrink->private_data; | |
| 7430 | 7430 | unsigned long ret = SHRINK_STOP; |
| 7431 | 7431 | |
| 7432 | 7432 | if (mutex_trylock(&conf->cache_size_mutex)) { |
| @@ -7447,7 +7447,7 @@ static unsigned long raid5_cache_scan(struct shrinker *shrink, · 14 unmodified lines | ||
| 7447 | 7447 | static unsigned long raid5_cache_count(struct shrinker *shrink, |
| 7448 | 7448 | struct shrink_control *sc) |
| 7449 | 7449 | { |
| 7450 | − struct r5conf *conf = container_of(shrink, struct r5conf, shrinker); | |
| 7450 | + struct r5conf *conf = shrink->private_data; | |
| 7451 | 7451 | |
| 7452 | 7452 | if (conf->max_nr_stripes < conf->min_nr_stripes) |
| 7453 | 7453 | /* unlikely, but not impossible */ |
| @@ -7682,18 +7682,22 @@ static struct r5conf *setup_conf(struct mddev *mddev) · 228 unmodified lines | ||
| 7682 | 7682 | * it reduces the queue depth and so can hurt throughput. |
| 7683 | 7683 | * So set it rather large, scaled by number of devices. |
| 7684 | 7684 | */ |
| 7685 | − conf->shrinker.seeks = DEFAULT_SEEKS * conf->raid_disks * 4; | |
| 7686 | − conf->shrinker.scan_objects = raid5_cache_scan; | |
| 7687 | − conf->shrinker.count_objects = raid5_cache_count; | |
| 7688 | − conf->shrinker.batch = 128; | |
| 7689 | − conf->shrinker.flags = 0; | |
| 7690 | − ret = register_shrinker(&conf->shrinker, "md-raid5:%s", mdname(mddev)); | |
| 7691 | − if (ret) { | |
| 7692 | − pr_warn("md/raid:%s: couldn't register shrinker.\n", | |
| 7685 | + conf->shrinker = shrinker_alloc(0, "md-raid5:%s", mdname(mddev)); | |
| 7686 | + if (!conf->shrinker) { | |
| 7687 | + ret = -ENOMEM; | |
| 7688 | + pr_warn("md/raid:%s: couldn't allocate shrinker.\n", | |
| 7693 | 7689 | mdname(mddev)); |
| 7694 | 7690 | goto abort; |
| 7695 | 7691 | } |
| 7696 | 7692 | |
| 7693 | + conf->shrinker->seeks = DEFAULT_SEEKS * conf->raid_disks * 4; | |
| 7694 | + conf->shrinker->scan_objects = raid5_cache_scan; | |
| 7695 | + conf->shrinker->count_objects = raid5_cache_count; | |
| 7696 | + conf->shrinker->batch = 128; | |
| 7697 | + conf->shrinker->private_data = conf; | |
| 7698 | + | |
| 7699 | + shrinker_register(conf->shrinker); | |
| 7700 | + | |
| 7697 | 7701 | sprintf(pers_name, "raid%d", mddev->new_level); |
| 7698 | 7702 | rcu_assign_pointer(conf->thread, |
| 7699 | 7703 | md_register_thread(raid5d, mddev, pers_name)); |
| @@ -670,7 +670,7 @@ struct r5conf { · 669 unmodified lines | ||
| 670 | 670 | wait_queue_head_t wait_for_stripe; |
| 671 | 671 | wait_queue_head_t wait_for_overlap; |
| 672 | 672 | unsigned long cache_state; |
| 673 | − struct shrinker shrinker; | |
| 673 | + struct shrinker *shrinker; | |
| 674 | 674 | int pool_size; /* number of disks in stripeheads in pool */ |
| 675 | 675 | spinlock_t device_lock; |
| 676 | 676 | struct disk_info *disks; |
| @@ -380,16 +380,7 @@ struct vmballoon { · 379 unmodified lines | ||
| 380 | 380 | /** |
| 381 | 381 | * @shrinker: shrinker interface that is used to avoid over-inflation. |
| 382 | 382 | */ |
| 383 | − struct shrinker shrinker; | |
| 384 | − | |
| 385 | − /** | |
| 386 | − * @shrinker_registered: whether the shrinker was registered. | |
| 387 | − * | |
| 388 | − * The shrinker interface does not handle gracefully the removal of | |
| 389 | − * shrinker that was not registered before. This indication allows to | |
| 390 | − * simplify the unregistration process. | |
| 391 | − */ | |
| 392 | − bool shrinker_registered; | |
| 383 | + struct shrinker *shrinker; | |
| 393 | 384 | }; |
| 394 | 385 | |
| 395 | 386 | static struct vmballoon balloon; |
| @@ -1568,29 +1559,27 @@ static unsigned long vmballoon_shrinker_count(struct shrinker *shrinker, · 1172 unmodified lines | ||
| 1568 | 1559 | |
| 1569 | 1560 | static void vmballoon_unregister_shrinker(struct vmballoon *b) |
| 1570 | 1561 | { |
| 1571 | − if (b->shrinker_registered) | |
| 1572 | − unregister_shrinker(&b->shrinker); | |
| 1573 | − b->shrinker_registered = false; | |
| 1562 | + shrinker_free(b->shrinker); | |
| 1563 | + b->shrinker = NULL; | |
| 1574 | 1564 | } |
| 1575 | 1565 | |
| 1576 | 1566 | static int vmballoon_register_shrinker(struct vmballoon *b) |
| 1577 | 1567 | { |
| 1578 | − int r; | |
| 1579 | − | |
| 1580 | 1568 | /* Do nothing if the shrinker is not enabled */ |
| 1581 | 1569 | if (!vmwballoon_shrinker_enable) |
| 1582 | 1570 | return 0; |
| 1583 | 1571 | |
| 1584 | − b->shrinker.scan_objects = vmballoon_shrinker_scan; | |
| 1585 | − b->shrinker.count_objects = vmballoon_shrinker_count; | |
| 1586 | − b->shrinker.seeks = DEFAULT_SEEKS; | |
| 1572 | + b->shrinker = shrinker_alloc(0, "vmw-balloon"); | |
| 1573 | + if (!b->shrinker) | |
| 1574 | + return -ENOMEM; | |
| 1587 | 1575 | |
| 1588 | − r = register_shrinker(&b->shrinker, "vmw-balloon"); | |
| 1576 | + b->shrinker->scan_objects = vmballoon_shrinker_scan; | |
| 1577 | + b->shrinker->count_objects = vmballoon_shrinker_count; | |
| 1578 | + b->shrinker->private_data = b; | |
| 1589 | 1579 | |
| 1590 | − if (r == 0) | |
| 1591 | − b->shrinker_registered = true; | |
| 1580 | + shrinker_register(b->shrinker); | |
| 1592 | 1581 | |
| 1593 | − return r; | |
| 1582 | + return 0; | |
| 1594 | 1583 | } |
| 1595 | 1584 | |
| 1596 | 1585 | /* |
| @@ -1883,7 +1872,7 @@ static int __init vmballoon_init(void) · 286 unmodified lines | ||
| 1883 | 1872 | |
| 1884 | 1873 | error = vmballoon_register_shrinker(&balloon); |
| 1885 | 1874 | if (error) |
| 1886 | − goto fail; | |
| 1875 | + return error; | |
| 1887 | 1876 | |
| 1888 | 1877 | /* |
| 1889 | 1878 | * Initialization of compaction must be done after the call to |
| @@ -1905,9 +1894,6 @@ static int __init vmballoon_init(void) · 15 unmodified lines | ||
| 1905 | 1894 | vmballoon_debugfs_init(&balloon); |
| 1906 | 1895 | |
| 1907 | 1896 | return 0; |
| 1908 | −fail: | |
| 1909 | − vmballoon_unregister_shrinker(&balloon); | |
| 1910 | − return error; | |
| 1911 | 1897 | } |
| 1912 | 1898 | |
| 1913 | 1899 | /* |
| @@ -111,7 +111,7 @@ struct virtio_balloon { · 110 unmodified lines | ||
| 111 | 111 | struct virtio_balloon_stat stats[VIRTIO_BALLOON_S_NR]; |
| 112 | 112 | |
| 113 | 113 | /* Shrinker to return free pages - VIRTIO_BALLOON_F_FREE_PAGE_HINT */ |
| 114 | − struct shrinker shrinker; | |
| 114 | + struct shrinker *shrinker; | |
| 115 | 115 | |
| 116 | 116 | /* OOM notifier to deflate on OOM - VIRTIO_BALLOON_F_DEFLATE_ON_OOM */ |
| 117 | 117 | struct notifier_block oom_nb; |
| @@ -820,8 +820,7 @@ static unsigned long shrink_free_pages(struct virtio_balloon *vb, · 702 unmodified lines | ||
| 820 | 820 | static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker, |
| 821 | 821 | struct shrink_control *sc) |
| 822 | 822 | { |
| 823 | − struct virtio_balloon *vb = container_of(shrinker, | |
| 824 | − struct virtio_balloon, shrinker); | |
| 823 | + struct virtio_balloon *vb = shrinker->private_data; | |
| 825 | 824 | |
| 826 | 825 | return shrink_free_pages(vb, sc->nr_to_scan); |
| 827 | 826 | } |
| @@ -829,8 +828,7 @@ static unsigned long virtio_balloon_shrinker_scan(struct shrinker *shrinker, · 1 unmodified line | ||
| 829 | 828 | static unsigned long virtio_balloon_shrinker_count(struct shrinker *shrinker, |
| 830 | 829 | struct shrink_control *sc) |
| 831 | 830 | { |
| 832 | − struct virtio_balloon *vb = container_of(shrinker, | |
| 833 | − struct virtio_balloon, shrinker); | |
| 831 | + struct virtio_balloon *vb = shrinker->private_data; | |
| 834 | 832 | |
| 835 | 833 | return vb->num_free_page_blocks * VIRTIO_BALLOON_HINT_BLOCK_PAGES; |
| 836 | 834 | } |
| @@ -851,16 +849,22 @@ static int virtio_balloon_oom_notify(struct notifier_block *nb, · 14 unmodified lines | ||
| 851 | 849 | |
| 852 | 850 | static void virtio_balloon_unregister_shrinker(struct virtio_balloon *vb) |
| 853 | 851 | { |
| 854 | − unregister_shrinker(&vb->shrinker); | |
| 852 | + shrinker_free(vb->shrinker); | |
| 855 | 853 | } |
| 856 | 854 | |
| 857 | 855 | static int virtio_balloon_register_shrinker(struct virtio_balloon *vb) |
| 858 | 856 | { |
| 859 | − vb->shrinker.scan_objects = virtio_balloon_shrinker_scan; | |
| 860 | − vb->shrinker.count_objects = virtio_balloon_shrinker_count; | |
| 861 | − vb->shrinker.seeks = DEFAULT_SEEKS; | |
| 857 | + vb->shrinker = shrinker_alloc(0, "virtio-balloon"); | |
| 858 | + if (!vb->shrinker) | |
| 859 | + return -ENOMEM; | |
| 862 | 860 | |
| 863 | − return register_shrinker(&vb->shrinker, "virtio-balloon"); | |
| 861 | + vb->shrinker->scan_objects = virtio_balloon_shrinker_scan; | |
| 862 | + vb->shrinker->count_objects = virtio_balloon_shrinker_count; | |
| 863 | + vb->shrinker->private_data = vb; | |
| 864 | + | |
| 865 | + shrinker_register(vb->shrinker); | |
| 866 | + | |
| 867 | + return 0; | |
| 864 | 868 | } |
| 865 | 869 | |
| 866 | 870 | static int virtballoon_probe(struct virtio_device *vdev) |
| @@ -284,13 +284,9 @@ static unsigned long backend_shrink_memory_count(struct shrinker *shrinker, · 283 unmodified lines | ||
| 284 | 284 | return 0; |
| 285 | 285 | } |
| 286 | 286 | |
| 287 | −static struct shrinker backend_memory_shrinker = { | |
| 288 | − .count_objects = backend_shrink_memory_count, | |
| 289 | − .seeks = DEFAULT_SEEKS, | |
| 290 | −}; | |
| 291 | − | |
| 292 | 287 | static int __init xenbus_probe_backend_init(void) |
| 293 | 288 | { |
| 289 | + struct shrinker *backend_memory_shrinker; | |
| 294 | 290 | static struct notifier_block xenstore_notifier = { |
| 295 | 291 | .notifier_call = backend_probe_and_watch |
| 296 | 292 | }; |
| @@ -305,8 +301,15 @@ static int __init xenbus_probe_backend_init(void) · 8 unmodified lines | ||
| 305 | 301 | |
| 306 | 302 | register_xenstore_notifier(&xenstore_notifier); |
| 307 | 303 | |
| 308 | − if (register_shrinker(&backend_memory_shrinker, "xen-backend")) | |
| 309 | − pr_warn("shrinker registration failed\n"); | |
| 304 | + backend_memory_shrinker = shrinker_alloc(0, "xen-backend"); | |
| 305 | + if (!backend_memory_shrinker) { | |
| 306 | + pr_warn("shrinker allocation failed\n"); | |
| 307 | + return 0; | |
| 308 | + } | |
| 309 | + | |
| 310 | + backend_memory_shrinker->count_objects = backend_shrink_memory_count; | |
| 311 | + | |
| 312 | + shrinker_register(backend_memory_shrinker); | |
| 310 | 313 | |
| 311 | 314 | return 0; |
| 312 | 315 | } |
| @@ -285,8 +285,7 @@ static int btree_node_write_and_reclaim(struct bch_fs *c, struct btree *b) · 284 unmodified lines | ||
| 285 | 285 | static unsigned long bch2_btree_cache_scan(struct shrinker *shrink, |
| 286 | 286 | struct shrink_control *sc) |
| 287 | 287 | { |
| 288 | − struct bch_fs *c = container_of(shrink, struct bch_fs, | |
| 289 | − btree_cache.shrink); | |
| 288 | + struct bch_fs *c = shrink->private_data; | |
| 290 | 289 | struct btree_cache *bc = &c->btree_cache; |
| 291 | 290 | struct btree *b, *t; |
| 292 | 291 | unsigned long nr = sc->nr_to_scan; |
| @@ -384,8 +383,7 @@ out_nounlock: · 91 unmodified lines | ||
| 384 | 383 | static unsigned long bch2_btree_cache_count(struct shrinker *shrink, |
| 385 | 384 | struct shrink_control *sc) |
| 386 | 385 | { |
| 387 | − struct bch_fs *c = container_of(shrink, struct bch_fs, | |
| 388 | − btree_cache.shrink); | |
| 386 | + struct bch_fs *c = shrink->private_data; | |
| 389 | 387 | struct btree_cache *bc = &c->btree_cache; |
| 390 | 388 | |
| 391 | 389 | if (bch2_btree_shrinker_disabled) |
| @@ -400,7 +398,7 @@ void bch2_fs_btree_cache_exit(struct bch_fs *c) · 8 unmodified lines | ||
| 400 | 398 | struct btree *b; |
| 401 | 399 | unsigned i, flags; |
| 402 | 400 | |
| 403 | − unregister_shrinker(&bc->shrink); | |
| 401 | + shrinker_free(bc->shrink); | |
| 404 | 402 | |
| 405 | 403 | /* vfree() can allocate memory: */ |
| 406 | 404 | flags = memalloc_nofs_save(); |
| @@ -454,6 +452,7 @@ void bch2_fs_btree_cache_exit(struct bch_fs *c) · 47 unmodified lines | ||
| 454 | 452 | int bch2_fs_btree_cache_init(struct bch_fs *c) |
| 455 | 453 | { |
| 456 | 454 | struct btree_cache *bc = &c->btree_cache; |
| 455 | + struct shrinker *shrink; | |
| 457 | 456 | unsigned i; |
| 458 | 457 | int ret = 0; |
| 459 | 458 | |
| @@ -473,12 +472,15 @@ int bch2_fs_btree_cache_init(struct bch_fs *c) · 13 unmodified lines | ||
| 473 | 472 | |
| 474 | 473 | mutex_init(&c->verify_lock); |
| 475 | 474 | |
| 476 | − bc->shrink.count_objects = bch2_btree_cache_count; | |
| 477 | − bc->shrink.scan_objects = bch2_btree_cache_scan; | |
| 478 | − bc->shrink.seeks = 4; | |
| 479 | − ret = register_shrinker(&bc->shrink, "%s/btree_cache", c->name); | |
| 480 | − if (ret) | |
| 475 | + shrink = shrinker_alloc(0, "%s/btree_cache", c->name); | |
| 476 | + if (!shrink) | |
| 481 | 477 | goto err; |
| 478 | + bc->shrink = shrink; | |
| 479 | + shrink->count_objects = bch2_btree_cache_count; | |
| 480 | + shrink->scan_objects = bch2_btree_cache_scan; | |
| 481 | + shrink->seeks = 4; | |
| 482 | + shrink->private_data = c; | |
| 483 | + shrinker_register(shrink); | |
| 482 | 484 | |
| 483 | 485 | return 0; |
| 484 | 486 | err: |
| @@ -834,8 +834,7 @@ void bch2_btree_key_cache_drop(struct btree_trans *trans, · 833 unmodified lines | ||
| 834 | 834 | static unsigned long bch2_btree_key_cache_scan(struct shrinker *shrink, |
| 835 | 835 | struct shrink_control *sc) |
| 836 | 836 | { |
| 837 | − struct bch_fs *c = container_of(shrink, struct bch_fs, | |
| 838 | − btree_key_cache.shrink); | |
| 837 | + struct bch_fs *c = shrink->private_data; | |
| 839 | 838 | struct btree_key_cache *bc = &c->btree_key_cache; |
| 840 | 839 | struct bucket_table *tbl; |
| 841 | 840 | struct bkey_cached *ck, *t; |
| @@ -932,8 +931,7 @@ out: · 90 unmodified lines | ||
| 932 | 931 | static unsigned long bch2_btree_key_cache_count(struct shrinker *shrink, |
| 933 | 932 | struct shrink_control *sc) |
| 934 | 933 | { |
| 935 | − struct bch_fs *c = container_of(shrink, struct bch_fs, | |
| 936 | − btree_key_cache.shrink); | |
| 934 | + struct bch_fs *c = shrink->private_data; | |
| 937 | 935 | struct btree_key_cache *bc = &c->btree_key_cache; |
| 938 | 936 | long nr = atomic_long_read(&bc->nr_keys) - |
| 939 | 937 | atomic_long_read(&bc->nr_dirty); |
| @@ -953,7 +951,7 @@ void bch2_fs_btree_key_cache_exit(struct btree_key_cache *bc) · 13 unmodified lines | ||
| 953 | 951 | int cpu; |
| 954 | 952 | #endif |
| 955 | 953 | |
| 956 | − unregister_shrinker(&bc->shrink); | |
| 954 | + shrinker_free(bc->shrink); | |
| 957 | 955 | |
| 958 | 956 | mutex_lock(&bc->lock); |
| 959 | 957 | |
| @@ -1027,6 +1025,7 @@ void bch2_fs_btree_key_cache_init_early(struct btree_key_cache *c) · 67 unmodified lines | ||
| 1027 | 1025 | int bch2_fs_btree_key_cache_init(struct btree_key_cache *bc) |
| 1028 | 1026 | { |
| 1029 | 1027 | struct bch_fs *c = container_of(bc, struct bch_fs, btree_key_cache); |
| 1028 | + struct shrinker *shrink; | |
| 1030 | 1029 | |
| 1031 | 1030 | #ifdef __KERNEL__ |
| 1032 | 1031 | bc->pcpu_freed = alloc_percpu(struct btree_key_cache_freelist); |
| @@ -1039,11 +1038,15 @@ int bch2_fs_btree_key_cache_init(struct btree_key_cache *bc) · 6 unmodified lines | ||
| 1039 | 1038 | |
| 1040 | 1039 | bc->table_init_done = true; |
| 1041 | 1040 | |
| 1042 | − bc->shrink.seeks = 0; | |
| 1043 | − bc->shrink.count_objects = bch2_btree_key_cache_count; | |
| 1044 | − bc->shrink.scan_objects = bch2_btree_key_cache_scan; | |
| 1045 | − if (register_shrinker(&bc->shrink, "%s/btree_key_cache", c->name)) | |
| 1041 | + shrink = shrinker_alloc(0, "%s/btree_key_cache", c->name); | |
| 1042 | + if (!shrink) | |
| 1046 | 1043 | return -BCH_ERR_ENOMEM_fs_btree_cache_init; |
| 1044 | + bc->shrink = shrink; | |
| 1045 | + shrink->seeks = 0; | |
| 1046 | + shrink->count_objects = bch2_btree_key_cache_count; | |
| 1047 | + shrink->scan_objects = bch2_btree_key_cache_scan; | |
| 1048 | + shrink->private_data = c; | |
| 1049 | + shrinker_register(shrink); | |
| 1047 | 1050 | return 0; |
| 1048 | 1051 | } |
| 1049 | 1052 | |
| @@ -163,7 +163,7 @@ struct btree_cache { · 162 unmodified lines | ||
| 163 | 163 | unsigned used; |
| 164 | 164 | unsigned reserve; |
| 165 | 165 | atomic_t dirty; |
| 166 | − struct shrinker shrink; | |
| 166 | + struct shrinker *shrink; | |
| 167 | 167 | |
| 168 | 168 | /* |
| 169 | 169 | * If we need to allocate memory for a new btree node and that |
| @@ -321,7 +321,7 @@ struct btree_key_cache { · 151 unmodified lines | ||
| 321 | 321 | bool table_init_done; |
| 322 | 322 | struct list_head freed_pcpu; |
| 323 | 323 | struct list_head freed_nonpcpu; |
| 324 | − struct shrinker shrink; | |
| 324 | + struct shrinker *shrink; | |
| 325 | 325 | unsigned shrink_iter; |
| 326 | 326 | struct btree_key_cache_freelist __percpu *pcpu_freed; |
| 327 | 327 | |
| @@ -1904,7 +1904,7 @@ got_sb: · 1903 unmodified lines | ||
| 1904 | 1904 | sb->s_flags |= SB_POSIXACL; |
| 1905 | 1905 | #endif |
| 1906 | 1906 | |
| 1907 | − sb->s_shrink.seeks = 0; | |
| 1907 | + sb->s_shrink->seeks = 0; | |
| 1908 | 1908 | |
| 1909 | 1909 | vinode = bch2_vfs_inode_get(c, BCACHEFS_ROOT_SUBVOL_INUM); |
| 1910 | 1910 | ret = PTR_ERR_OR_ZERO(vinode); |
| @@ -494,7 +494,7 @@ STORE(bch2_fs) · 493 unmodified lines | ||
| 494 | 494 | |
| 495 | 495 | sc.gfp_mask = GFP_KERNEL; |
| 496 | 496 | sc.nr_to_scan = strtoul_or_return(buf); |
| 497 | − c->btree_cache.shrink.scan_objects(&c->btree_cache.shrink, &sc); | |
| 497 | + c->btree_cache.shrink->scan_objects(c->btree_cache.shrink, &sc); | |
| 498 | 498 | } |
| 499 | 499 | |
| 500 | 500 | if (attr == &sysfs_btree_wakeup) |
| @@ -1472,7 +1472,7 @@ static struct dentry *btrfs_mount_root(struct file_system_type *fs_type, · 1471 unmodified lines | ||
| 1472 | 1472 | error = -EBUSY; |
| 1473 | 1473 | } else { |
| 1474 | 1474 | snprintf(s->s_id, sizeof(s->s_id), "%pg", bdev); |
| 1475 | − shrinker_debugfs_rename(&s->s_shrink, "sb-%s:%s", fs_type->name, | |
| 1475 | + shrinker_debugfs_rename(s->s_shrink, "sb-%s:%s", fs_type->name, | |
| 1476 | 1476 | s->s_id); |
| 1477 | 1477 | btrfs_sb(s)->bdev_holder = fs_type; |
| 1478 | 1478 | error = btrfs_fill_super(s, fs_devices, data); |
| @@ -282,13 +282,7 @@ static void end_buffer_async_read(struct buffer_head *bh, int uptodate) · 281 unmodified lines | ||
| 282 | 282 | } while (tmp != bh); |
| 283 | 283 | spin_unlock_irqrestore(&first->b_uptodate_lock, flags); |
| 284 | 284 | |
| 285 | − /* | |
| 286 | − * If all of the buffers are uptodate then we can set the page | |
| 287 | − * uptodate. | |
| 288 | − */ | |
| 289 | − if (folio_uptodate) | |
| 290 | − folio_mark_uptodate(folio); | |
| 291 | − folio_unlock(folio); | |
| 285 | + folio_end_read(folio, folio_uptodate); | |
| 292 | 286 | return; |
| 293 | 287 | |
| 294 | 288 | still_busy: |
| @@ -915,16 +909,12 @@ int remove_inode_buffers(struct inode *inode) · 620 unmodified lines | ||
| 915 | 909 | * which may not fail from ordinary buffer allocations. |
| 916 | 910 | */ |
| 917 | 911 | struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size, |
| 918 | − bool retry) | |
| 912 | + gfp_t gfp) | |
| 919 | 913 | { |
| 920 | 914 | struct buffer_head *bh, *head; |
| 921 | − gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT; | |
| 922 | 915 | long offset; |
| 923 | 916 | struct mem_cgroup *memcg, *old_memcg; |
| 924 | 917 | |
| 925 | − if (retry) | |
| 926 | − gfp |= __GFP_NOFAIL; | |
| 927 | − | |
| 928 | 918 | /* The folio lock pins the memcg */ |
| 929 | 919 | memcg = folio_memcg(folio); |
| 930 | 920 | old_memcg = set_active_memcg(memcg); |
| @@ -967,7 +957,11 @@ EXPORT_SYMBOL_GPL(folio_alloc_buffers); · 36 unmodified lines | ||
| 967 | 957 | struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size, |
| 968 | 958 | bool retry) |
| 969 | 959 | { |
| 970 | − return folio_alloc_buffers(page_folio(page), size, retry); | |
| 960 | + gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT; | |
| 961 | + if (retry) | |
| 962 | + gfp |= __GFP_NOFAIL; | |
| 963 | + | |
| 964 | + return folio_alloc_buffers(page_folio(page), size, gfp); | |
| 971 | 965 | } |
| 972 | 966 | EXPORT_SYMBOL_GPL(alloc_page_buffers); |
| 973 | 967 | |
| @@ -1043,20 +1037,11 @@ grow_dev_page(struct block_device *bdev, sector_t block, · 69 unmodified lines | ||
| 1043 | 1037 | struct buffer_head *bh; |
| 1044 | 1038 | sector_t end_block; |
| 1045 | 1039 | int ret = 0; |
| 1046 | − gfp_t gfp_mask; | |
| 1047 | − | |
| 1048 | − gfp_mask = mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS) | gfp; | |
| 1049 | − | |
| 1050 | − /* | |
| 1051 | − * XXX: __getblk_slow() can not really deal with failure and | |
| 1052 | − * will endlessly loop on improvised global reclaim. Prefer | |
| 1053 | − * looping in the allocator rather than here, at least that | |
| 1054 | − * code knows what it's doing. | |
| 1055 | − */ | |
| 1056 | − gfp_mask |= __GFP_NOFAIL; | |
| 1057 | 1040 | |
| 1058 | 1041 | folio = __filemap_get_folio(inode->i_mapping, index, |
| 1059 | − FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp_mask); | |
| 1042 | + FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp); | |
| 1043 | + if (IS_ERR(folio)) | |
| 1044 | + return PTR_ERR(folio); | |
| 1060 | 1045 | |
| 1061 | 1046 | bh = folio_buffers(folio); |
| 1062 | 1047 | if (bh) { |
| @@ -1069,7 +1054,10 @@ grow_dev_page(struct block_device *bdev, sector_t block, · 6 unmodified lines | ||
| 1069 | 1054 | goto failed; |
| 1070 | 1055 | } |
| 1071 | 1056 | |
| 1072 | − bh = folio_alloc_buffers(folio, size, true); | |
| 1057 | + ret = -ENOMEM; | |
| 1058 | + bh = folio_alloc_buffers(folio, size, gfp | __GFP_ACCOUNT); | |
| 1059 | + if (!bh) | |
| 1060 | + goto failed; | |
| 1073 | 1061 | |
| 1074 | 1062 | /* |
| 1075 | 1063 | * Link the folio to the buffers and initialise them. Take the |
| @@ -1420,33 +1408,36 @@ __find_get_block(struct block_device *bdev, sector_t block, unsigned size) · 344 unmodified lines | ||
| 1420 | 1408 | } |
| 1421 | 1409 | EXPORT_SYMBOL(__find_get_block); |
| 1422 | 1410 | |
| 1423 | −/* | |
| 1424 | − * __getblk_gfp() will locate (and, if necessary, create) the buffer_head | |
| 1425 | − * which corresponds to the passed block_device, block and size. The | |
| 1426 | − * returned buffer has its reference count incremented. | |
| 1411 | +/** | |
| 1412 | + * bdev_getblk - Get a buffer_head in a block device's buffer cache. | |
| 1413 | + * @bdev: The block device. | |
| 1414 | + * @block: The block number. | |
| 1415 | + * @size: The size of buffer_heads for this @bdev. | |
| 1416 | + * @gfp: The memory allocation flags to use. | |
| 1427 | 1417 | * |
| 1428 | − * __getblk_gfp() will lock up the machine if grow_dev_page's | |
| 1429 | − * try_to_free_buffers() attempt is failing. FIXME, perhaps? | |
| 1418 | + * Return: The buffer head, or NULL if memory could not be allocated. | |
| 1430 | 1419 | */ |
| 1431 | −struct buffer_head * | |
| 1432 | −__getblk_gfp(struct block_device *bdev, sector_t block, | |
| 1433 | − unsigned size, gfp_t gfp) | |
| 1420 | +struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block, | |
| 1421 | + unsigned size, gfp_t gfp) | |
| 1434 | 1422 | { |
| 1435 | 1423 | struct buffer_head *bh = __find_get_block(bdev, block, size); |
| 1436 | 1424 | |
| 1437 | − might_sleep(); | |
| 1438 | − if (bh == NULL) | |
| 1439 | − bh = __getblk_slow(bdev, block, size, gfp); | |
| 1440 | − return bh; | |
| 1425 | + might_alloc(gfp); | |
| 1426 | + if (bh) | |
| 1427 | + return bh; | |
| 1428 | + | |
| 1429 | + return __getblk_slow(bdev, block, size, gfp); | |
| 1441 | 1430 | } |
| 1442 | −EXPORT_SYMBOL(__getblk_gfp); | |
| 1431 | +EXPORT_SYMBOL(bdev_getblk); | |
| 1443 | 1432 | |
| 1444 | 1433 | /* |
| 1445 | 1434 | * Do async read-ahead on a buffer.. |
| 1446 | 1435 | */ |
| 1447 | 1436 | void __breadahead(struct block_device *bdev, sector_t block, unsigned size) |
| 1448 | 1437 | { |
| 1449 | − struct buffer_head *bh = __getblk(bdev, block, size); | |
| 1438 | + struct buffer_head *bh = bdev_getblk(bdev, block, size, | |
| 1439 | + GFP_NOWAIT | __GFP_MOVABLE); | |
| 1440 | + | |
| 1450 | 1441 | if (likely(bh)) { |
| 1451 | 1442 | bh_readahead(bh, REQ_RAHEAD); |
| 1452 | 1443 | brelse(bh); |
| @@ -1470,7 +1461,17 @@ struct buffer_head * · 17 unmodified lines | ||
| 1470 | 1461 | __bread_gfp(struct block_device *bdev, sector_t block, |
| 1471 | 1462 | unsigned size, gfp_t gfp) |
| 1472 | 1463 | { |
| 1473 | − struct buffer_head *bh = __getblk_gfp(bdev, block, size, gfp); | |
| 1464 | + struct buffer_head *bh; | |
| 1465 | + | |
| 1466 | + gfp |= mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS); | |
| 1467 | + | |
| 1468 | + /* | |
| 1469 | + * Prefer looping in the allocator rather than here, at least that | |
| 1470 | + * code knows what it's doing. | |
| 1471 | + */ | |
| 1472 | + gfp |= __GFP_NOFAIL; | |
| 1473 | + | |
| 1474 | + bh = bdev_getblk(bdev, block, size, gfp); | |
| 1474 | 1475 | |
| 1475 | 1476 | if (likely(bh) && !buffer_uptodate(bh)) |
| 1476 | 1477 | bh = __bread_slow(bh); |
| @@ -1640,12 +1641,13 @@ EXPORT_SYMBOL(block_invalidate_folio); · 163 unmodified lines | ||
| 1640 | 1641 | * block_dirty_folio() via private_lock. try_to_free_buffers |
| 1641 | 1642 | * is already excluded via the folio lock. |
| 1642 | 1643 | */ |
| 1643 | −void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize, | |
| 1644 | − unsigned long b_state) | |
| 1644 | +struct buffer_head *create_empty_buffers(struct folio *folio, | |
| 1645 | + unsigned long blocksize, unsigned long b_state) | |
| 1645 | 1646 | { |
| 1646 | 1647 | struct buffer_head *bh, *head, *tail; |
| 1648 | + gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT | __GFP_NOFAIL; | |
| 1647 | 1649 | |
| 1648 | − head = folio_alloc_buffers(folio, blocksize, true); | |
| 1650 | + head = folio_alloc_buffers(folio, blocksize, gfp); | |
| 1649 | 1651 | bh = head; |
| 1650 | 1652 | do { |
| 1651 | 1653 | bh->b_state |= b_state; |
| @@ -1667,13 +1669,8 @@ void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize, · 15 unmodified lines | ||
| 1667 | 1669 | } |
| 1668 | 1670 | folio_attach_private(folio, head); |
| 1669 | 1671 | spin_unlock(&folio->mapping->private_lock); |
| 1670 | −} | |
| 1671 | −EXPORT_SYMBOL(folio_create_empty_buffers); | |
| 1672 | 1672 | |
| 1673 | −void create_empty_buffers(struct page *page, | |
| 1674 | − unsigned long blocksize, unsigned long b_state) | |
| 1675 | −{ | |
| 1676 | − folio_create_empty_buffers(page_folio(page), blocksize, b_state); | |
| 1673 | + return head; | |
| 1677 | 1674 | } |
| 1678 | 1675 | EXPORT_SYMBOL(create_empty_buffers); |
| 1679 | 1676 | |
| @@ -1768,13 +1765,15 @@ static struct buffer_head *folio_create_buffers(struct folio *folio, · 88 unmodified lines | ||
| 1768 | 1765 | struct inode *inode, |
| 1769 | 1766 | unsigned int b_state) |
| 1770 | 1767 | { |
| 1768 | + struct buffer_head *bh; | |
| 1769 | + | |
| 1771 | 1770 | BUG_ON(!folio_test_locked(folio)); |
| 1772 | 1771 | |
| 1773 | − if (!folio_buffers(folio)) | |
| 1774 | − folio_create_empty_buffers(folio, | |
| 1775 | − 1 << READ_ONCE(inode->i_blkbits), | |
| 1776 | − b_state); | |
| 1777 | − return folio_buffers(folio); | |
| 1772 | + bh = folio_buffers(folio); | |
| 1773 | + if (!bh) | |
| 1774 | + bh = create_empty_buffers(folio, | |
| 1775 | + 1 << READ_ONCE(inode->i_blkbits), b_state); | |
| 1776 | + return bh; | |
| 1778 | 1777 | } |
| 1779 | 1778 | |
| 1780 | 1779 | /* |
| @@ -2425,12 +2424,10 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) · 644 unmodified lines | ||
| 2425 | 2424 | |
| 2426 | 2425 | if (!nr) { |
| 2427 | 2426 | /* |
| 2428 | − * All buffers are uptodate - we can set the folio uptodate | |
| 2429 | − * as well. But not if get_block() returned an error. | |
| 2427 | + * All buffers are uptodate or get_block() returned an | |
| 2428 | + * error when trying to map them - we can finish the read. | |
| 2430 | 2429 | */ |
| 2431 | − if (!page_error) | |
| 2432 | − folio_mark_uptodate(folio); | |
| 2433 | − folio_unlock(folio); | |
| 2430 | + folio_end_read(folio, !page_error); | |
| 2434 | 2431 | return 0; |
| 2435 | 2432 | } |
| 2436 | 2433 | |
| @@ -2676,10 +2673,8 @@ int block_truncate_page(struct address_space *mapping, · 239 unmodified lines | ||
| 2676 | 2673 | return PTR_ERR(folio); |
| 2677 | 2674 | |
| 2678 | 2675 | bh = folio_buffers(folio); |
| 2679 | − if (!bh) { | |
| 2680 | − folio_create_empty_buffers(folio, blocksize, 0); | |
| 2681 | − bh = folio_buffers(folio); | |
| 2682 | − } | |
| 2676 | + if (!bh) | |
| 2677 | + bh = create_empty_buffers(folio, blocksize, 0); | |
| 2683 | 2678 | |
| 2684 | 2679 | /* Find the buffer that contains "offset" */ |
| 2685 | 2680 | offset = offset_in_folio(folio, from); |
| @@ -412,23 +412,23 @@ static struct page *dax_busy_page(void *entry) · 411 unmodified lines | ||
| 412 | 412 | return NULL; |
| 413 | 413 | } |
| 414 | 414 | |
| 415 | −/* | |
| 416 | − * dax_lock_page - Lock the DAX entry corresponding to a page | |
| 417 | − * @page: The page whose entry we want to lock | |
| 415 | +/** | |
| 416 | + * dax_lock_folio - Lock the DAX entry corresponding to a folio | |
| 417 | + * @folio: The folio whose entry we want to lock | |
| 418 | 418 | * |
| 419 | 419 | * Context: Process context. |
| 420 | − * Return: A cookie to pass to dax_unlock_page() or 0 if the entry could | |
| 420 | + * Return: A cookie to pass to dax_unlock_folio() or 0 if the entry could | |
| 421 | 421 | * not be locked. |
| 422 | 422 | */ |
| 423 | −dax_entry_t dax_lock_page(struct page *page) | |
| 423 | +dax_entry_t dax_lock_folio(struct folio *folio) | |
| 424 | 424 | { |
| 425 | 425 | XA_STATE(xas, NULL, 0); |
| 426 | 426 | void *entry; |
| 427 | 427 | |
| 428 | − /* Ensure page->mapping isn't freed while we look at it */ | |
| 428 | + /* Ensure folio->mapping isn't freed while we look at it */ | |
| 429 | 429 | rcu_read_lock(); |
| 430 | 430 | for (;;) { |
| 431 | − struct address_space *mapping = READ_ONCE(page->mapping); | |
| 431 | + struct address_space *mapping = READ_ONCE(folio->mapping); | |
| 432 | 432 | |
| 433 | 433 | entry = NULL; |
| 434 | 434 | if (!mapping || !dax_mapping(mapping)) |
| @@ -447,11 +447,11 @@ dax_entry_t dax_lock_page(struct page *page) · 12 unmodified lines | ||
| 447 | 447 | |
| 448 | 448 | xas.xa = &mapping->i_pages; |
| 449 | 449 | xas_lock_irq(&xas); |
| 450 | − if (mapping != page->mapping) { | |
| 450 | + if (mapping != folio->mapping) { | |
| 451 | 451 | xas_unlock_irq(&xas); |
| 452 | 452 | continue; |
| 453 | 453 | } |
| 454 | − xas_set(&xas, page->index); | |
| 454 | + xas_set(&xas, folio->index); | |
| 455 | 455 | entry = xas_load(&xas); |
| 456 | 456 | if (dax_is_locked(entry)) { |
| 457 | 457 | rcu_read_unlock(); |
| @@ -467,10 +467,10 @@ dax_entry_t dax_lock_page(struct page *page) · 9 unmodified lines | ||
| 467 | 467 | return (dax_entry_t)entry; |
| 468 | 468 | } |
| 469 | 469 | |
| 470 | −void dax_unlock_page(struct page *page, dax_entry_t cookie) | |
| 470 | +void dax_unlock_folio(struct folio *folio, dax_entry_t cookie) | |
| 471 | 471 | { |
| 472 | − struct address_space *mapping = page->mapping; | |
| 473 | − XA_STATE(xas, &mapping->i_pages, page->index); | |
| 472 | + struct address_space *mapping = folio->mapping; | |
| 473 | + XA_STATE(xas, &mapping->i_pages, folio->index); | |
| 474 | 474 | |
| 475 | 475 | if (S_ISCHR(mapping->host->i_mode)) |
| 476 | 476 | return; |
| @@ -264,19 +264,24 @@ static unsigned long erofs_shrink_scan(struct shrinker *shrink, · 263 unmodified lines | ||
| 264 | 264 | return freed; |
| 265 | 265 | } |
| 266 | 266 | |
| 267 | −static struct shrinker erofs_shrinker_info = { | |
| 268 | − .scan_objects = erofs_shrink_scan, | |
| 269 | − .count_objects = erofs_shrink_count, | |
| 270 | − .seeks = DEFAULT_SEEKS, | |
| 271 | −}; | |
| 267 | +static struct shrinker *erofs_shrinker_info; | |
| 272 | 268 | |
| 273 | 269 | int __init erofs_init_shrinker(void) |
| 274 | 270 | { |
| 275 | − return register_shrinker(&erofs_shrinker_info, "erofs-shrinker"); | |
| 271 | + erofs_shrinker_info = shrinker_alloc(0, "erofs-shrinker"); | |
| 272 | + if (!erofs_shrinker_info) | |
| 273 | + return -ENOMEM; | |
| 274 | + | |
| 275 | + erofs_shrinker_info->count_objects = erofs_shrink_count; | |
| 276 | + erofs_shrinker_info->scan_objects = erofs_shrink_scan; | |
| 277 | + | |
| 278 | + shrinker_register(erofs_shrinker_info); | |
| 279 | + | |
| 280 | + return 0; | |
| 276 | 281 | } |
| 277 | 282 | |
| 278 | 283 | void erofs_exit_shrinker(void) |
| 279 | 284 | { |
| 280 | − unregister_shrinker(&erofs_shrinker_info); | |
| 285 | + shrinker_free(erofs_shrinker_info); | |
| 281 | 286 | } |
| 282 | 287 | #endif /* !CONFIG_EROFS_FS_ZIP */ |
| @@ -713,7 +713,7 @@ static int shift_arg_pages(struct vm_area_struct *vma, unsigned long shift) · 712 unmodified lines | ||
| 713 | 713 | * process cleanup to remove whatever mess we made. |
| 714 | 714 | */ |
| 715 | 715 | if (length != move_page_tables(vma, old_start, |
| 716 | − vma, new_start, length, false)) | |
| 716 | + vma, new_start, length, false, true)) | |
| 717 | 717 | return -ENOMEM; |
| 718 | 718 | |
| 719 | 719 | lru_add_drain(); |
| @@ -986,8 +986,6 @@ static int exec_mmap(struct mm_struct *mm) · 266 unmodified lines | ||
| 986 | 986 | tsk = current; |
| 987 | 987 | old_mm = current->mm; |
| 988 | 988 | exec_mm_release(tsk, old_mm); |
| 989 | − if (old_mm) | |
| 990 | − sync_mm_rss(old_mm); | |
| 991 | 989 | |
| 992 | 990 | ret = down_write_killable(&tsk->signal->exec_update_lock); |
| 993 | 991 | if (ret) |
| @@ -1664,7 +1664,7 @@ struct ext4_sb_info { · 1663 unmodified lines | ||
| 1664 | 1664 | __u32 s_csum_seed; |
| 1665 | 1665 | |
| 1666 | 1666 | /* Reclaim extents from extent status tree */ |
| 1667 | − struct shrinker s_es_shrinker; | |
| 1667 | + struct shrinker *s_es_shrinker; | |
| 1668 | 1668 | struct list_head s_es_list; /* List of inodes with reclaimable extents */ |
| 1669 | 1669 | long s_es_nr_inode; |
| 1670 | 1670 | struct ext4_es_stats s_es_stats; |
| @@ -1632,7 +1632,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink, · 1631 unmodified lines | ||
| 1632 | 1632 | unsigned long nr; |
| 1633 | 1633 | struct ext4_sb_info *sbi; |
| 1634 | 1634 | |
| 1635 | − sbi = container_of(shrink, struct ext4_sb_info, s_es_shrinker); | |
| 1635 | + sbi = shrink->private_data; | |
| 1636 | 1636 | nr = percpu_counter_read_positive(&sbi->s_es_stats.es_stats_shk_cnt); |
| 1637 | 1637 | trace_ext4_es_shrink_count(sbi->s_sb, sc->nr_to_scan, nr); |
| 1638 | 1638 | return nr; |
| @@ -1641,8 +1641,7 @@ static unsigned long ext4_es_count(struct shrinker *shrink, · 2 unmodified lines | ||
| 1641 | 1641 | static unsigned long ext4_es_scan(struct shrinker *shrink, |
| 1642 | 1642 | struct shrink_control *sc) |
| 1643 | 1643 | { |
| 1644 | − struct ext4_sb_info *sbi = container_of(shrink, | |
| 1645 | − struct ext4_sb_info, s_es_shrinker); | |
| 1644 | + struct ext4_sb_info *sbi = shrink->private_data; | |
| 1646 | 1645 | int nr_to_scan = sc->nr_to_scan; |
| 1647 | 1646 | int ret, nr_shrunk; |
| 1648 | 1647 | |
| @@ -1726,13 +1725,17 @@ int ext4_es_register_shrinker(struct ext4_sb_info *sbi) · 77 unmodified lines | ||
| 1726 | 1725 | if (err) |
| 1727 | 1726 | goto err3; |
| 1728 | 1727 | |
| 1729 | − sbi->s_es_shrinker.scan_objects = ext4_es_scan; | |
| 1730 | − sbi->s_es_shrinker.count_objects = ext4_es_count; | |
| 1731 | − sbi->s_es_shrinker.seeks = DEFAULT_SEEKS; | |
| 1732 | − err = register_shrinker(&sbi->s_es_shrinker, "ext4-es:%s", | |
| 1733 | − sbi->s_sb->s_id); | |
| 1734 | − if (err) | |
| 1728 | + sbi->s_es_shrinker = shrinker_alloc(0, "ext4-es:%s", sbi->s_sb->s_id); | |
| 1729 | + if (!sbi->s_es_shrinker) { | |
| 1730 | + err = -ENOMEM; | |
| 1735 | 1731 | goto err4; |
| 1732 | + } | |
| 1733 | + | |
| 1734 | + sbi->s_es_shrinker->scan_objects = ext4_es_scan; | |
| 1735 | + sbi->s_es_shrinker->count_objects = ext4_es_count; | |
| 1736 | + sbi->s_es_shrinker->private_data = sbi; | |
| 1737 | + | |
| 1738 | + shrinker_register(sbi->s_es_shrinker); | |
| 1736 | 1739 | |
| 1737 | 1740 | return 0; |
| 1738 | 1741 | err4: |
| @@ -1752,7 +1755,7 @@ void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi) · 13 unmodified lines | ||
| 1752 | 1755 | percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_misses); |
| 1753 | 1756 | percpu_counter_destroy(&sbi->s_es_stats.es_stats_all_cnt); |
| 1754 | 1757 | percpu_counter_destroy(&sbi->s_es_stats.es_stats_shk_cnt); |
| 1755 | − unregister_shrinker(&sbi->s_es_shrinker); | |
| 1758 | + shrinker_free(sbi->s_es_shrinker); | |
| 1756 | 1759 | } |
| 1757 | 1760 | |
| 1758 | 1761 | /* |
| @@ -1032,10 +1032,8 @@ static int ext4_block_write_begin(struct folio *folio, loff_t pos, unsigned len, · 1031 unmodified lines | ||
| 1032 | 1032 | BUG_ON(from > to); |
| 1033 | 1033 | |
| 1034 | 1034 | head = folio_buffers(folio); |
| 1035 | − if (!head) { | |
| 1036 | − create_empty_buffers(&folio->page, blocksize, 0); | |
| 1037 | − head = folio_buffers(folio); | |
| 1038 | − } | |
| 1035 | + if (!head) | |
| 1036 | + head = create_empty_buffers(folio, blocksize, 0); | |
| 1039 | 1037 | bbits = ilog2(blocksize); |
| 1040 | 1038 | block = (sector_t)folio->index << (PAGE_SHIFT - bbits); |
| 1041 | 1039 | |
| @@ -1165,7 +1163,7 @@ retry_grab: · 123 unmodified lines | ||
| 1165 | 1163 | * starting the handle. |
| 1166 | 1164 | */ |
| 1167 | 1165 | if (!folio_buffers(folio)) |
| 1168 | − create_empty_buffers(&folio->page, inode->i_sb->s_blocksize, 0); | |
| 1166 | + create_empty_buffers(folio, inode->i_sb->s_blocksize, 0); | |
| 1169 | 1167 | |
| 1170 | 1168 | folio_unlock(folio); |
| 1171 | 1169 | |
| @@ -3655,10 +3653,8 @@ static int __ext4_block_zero_page_range(handle_t *handle, · 2483 unmodified lines | ||
| 3655 | 3653 | iblock = index << (PAGE_SHIFT - inode->i_sb->s_blocksize_bits); |
| 3656 | 3654 | |
| 3657 | 3655 | bh = folio_buffers(folio); |
| 3658 | − if (!bh) { | |
| 3659 | − create_empty_buffers(&folio->page, blocksize, 0); | |
| 3660 | − bh = folio_buffers(folio); | |
| 3661 | − } | |
| 3656 | + if (!bh) | |
| 3657 | + bh = create_empty_buffers(folio, blocksize, 0); | |
| 3662 | 3658 | |
| 3663 | 3659 | /* Find the buffer that contains "offset" */ |
| 3664 | 3660 | pos = blocksize; |
| @@ -183,10 +183,8 @@ mext_page_mkuptodate(struct folio *folio, unsigned from, unsigned to) · 182 unmodified lines | ||
| 183 | 183 | |
| 184 | 184 | blocksize = i_blocksize(inode); |
| 185 | 185 | head = folio_buffers(folio); |
| 186 | − if (!head) { | |
| 187 | − create_empty_buffers(&folio->page, blocksize, 0); | |
| 188 | − head = folio_buffers(folio); | |
| 189 | − } | |
| 186 | + if (!head) | |
| 187 | + head = create_empty_buffers(folio, blocksize, 0); | |
| 190 | 188 | |
| 191 | 189 | block = (sector_t)folio->index << (PAGE_SHIFT - inode->i_blkbits); |
| 192 | 190 | for (bh = head, block_start = 0; bh != head || !block_start; |
| @@ -380,9 +378,10 @@ data_copy: · 187 unmodified lines | ||
| 380 | 378 | } |
| 381 | 379 | /* Perform all necessary steps similar write_begin()/write_end() |
| 382 | 380 | * but keeping in mind that i_size will not change */ |
| 383 | − if (!folio_buffers(folio[0])) | |
| 384 | − create_empty_buffers(&folio[0]->page, 1 << orig_inode->i_blkbits, 0); | |
| 385 | 381 | bh = folio_buffers(folio[0]); |
| 382 | + if (!bh) | |
| 383 | + bh = create_empty_buffers(folio[0], | |
| 384 | + 1 << orig_inode->i_blkbits, 0); | |
| 386 | 385 | for (i = 0; i < data_offset_in_page; i++) |
| 387 | 386 | bh = bh->b_this_page; |
| 388 | 387 | for (i = 0; i < block_len_in_page; i++) { |
| @@ -70,15 +70,8 @@ static void __read_end_io(struct bio *bio) · 69 unmodified lines | ||
| 70 | 70 | { |
| 71 | 71 | struct folio_iter fi; |
| 72 | 72 | |
| 73 | − bio_for_each_folio_all(fi, bio) { | |
| 74 | − struct folio *folio = fi.folio; | |
| 75 | − | |
| 76 | − if (bio->bi_status) | |
| 77 | − folio_clear_uptodate(folio); | |
| 78 | − else | |
| 79 | − folio_mark_uptodate(folio); | |
| 80 | − folio_unlock(folio); | |
| 81 | − } | |
| 73 | + bio_for_each_folio_all(fi, bio) | |
| 74 | + folio_end_read(fi.folio, bio->bi_status == 0); | |
| 82 | 75 | if (bio->bi_private) |
| 83 | 76 | mempool_free(bio->bi_private, bio_post_read_ctx_pool); |
| 84 | 77 | bio_put(bio); |
| @@ -336,8 +329,7 @@ int ext4_mpage_readpages(struct inode *inode, · 251 unmodified lines | ||
| 336 | 329 | if (ext4_need_verity(inode, folio->index) && |
| 337 | 330 | !fsverity_verify_folio(folio)) |
| 338 | 331 | goto set_error_page; |
| 339 | − folio_mark_uptodate(folio); | |
| 340 | − folio_unlock(folio); | |
| 332 | + folio_end_read(folio, true); | |
| 341 | 333 | continue; |
| 342 | 334 | } |
| 343 | 335 | } else if (fully_mapped) { |
| @@ -244,18 +244,25 @@ static struct buffer_head *__ext4_sb_bread_gfp(struct super_block *sb, · 243 unmodified lines | ||
| 244 | 244 | struct buffer_head *ext4_sb_bread(struct super_block *sb, sector_t block, |
| 245 | 245 | blk_opf_t op_flags) |
| 246 | 246 | { |
| 247 | − return __ext4_sb_bread_gfp(sb, block, op_flags, __GFP_MOVABLE); | |
| 247 | + gfp_t gfp = mapping_gfp_constraint(sb->s_bdev->bd_inode->i_mapping, | |
| 248 | + ~__GFP_FS) | __GFP_MOVABLE; | |
| 249 | + | |
| 250 | + return __ext4_sb_bread_gfp(sb, block, op_flags, gfp); | |
| 248 | 251 | } |
| 249 | 252 | |
| 250 | 253 | struct buffer_head *ext4_sb_bread_unmovable(struct super_block *sb, |
| 251 | 254 | sector_t block) |
| 252 | 255 | { |
| 253 | − return __ext4_sb_bread_gfp(sb, block, 0, 0); | |
| 256 | + gfp_t gfp = mapping_gfp_constraint(sb->s_bdev->bd_inode->i_mapping, | |
| 257 | + ~__GFP_FS); | |
| 258 | + | |
| 259 | + return __ext4_sb_bread_gfp(sb, block, 0, gfp); | |
| 254 | 260 | } |
| 255 | 261 | |
| 256 | 262 | void ext4_sb_breadahead_unmovable(struct super_block *sb, sector_t block) |
| 257 | 263 | { |
| 258 | − struct buffer_head *bh = sb_getblk_gfp(sb, block, 0); | |
| 264 | + struct buffer_head *bh = bdev_getblk(sb->s_bdev, block, | |
| 265 | + sb->s_blocksize, GFP_NOWAIT | __GFP_NOWARN); | |
| 259 | 266 | |
| 260 | 267 | if (likely(bh)) { |
| 261 | 268 | if (trylock_buffer(bh)) |
| @@ -83,11 +83,26 @@ void f2fs_build_fault_attr(struct f2fs_sb_info *sbi, unsigned int rate, · 82 unmodified lines | ||
| 83 | 83 | #endif |
| 84 | 84 | |
| 85 | 85 | /* f2fs-wide shrinker description */ |
| 86 | −static struct shrinker f2fs_shrinker_info = { | |
| 87 | − .scan_objects = f2fs_shrink_scan, | |
| 88 | − .count_objects = f2fs_shrink_count, | |
| 89 | − .seeks = DEFAULT_SEEKS, | |
| 90 | −}; | |
| 86 | +static struct shrinker *f2fs_shrinker_info; | |
| 87 | + | |
| 88 | +static int __init f2fs_init_shrinker(void) | |
| 89 | +{ | |
| 90 | + f2fs_shrinker_info = shrinker_alloc(0, "f2fs-shrinker"); | |
| 91 | + if (!f2fs_shrinker_info) | |
| 92 | + return -ENOMEM; | |
| 93 | + | |
| 94 | + f2fs_shrinker_info->count_objects = f2fs_shrink_count; | |
| 95 | + f2fs_shrinker_info->scan_objects = f2fs_shrink_scan; | |
| 96 | + | |
| 97 | + shrinker_register(f2fs_shrinker_info); | |
| 98 | + | |
| 99 | + return 0; | |
| 100 | +} | |
| 101 | + | |
| 102 | +static void f2fs_exit_shrinker(void) | |
| 103 | +{ | |
| 104 | + shrinker_free(f2fs_shrinker_info); | |
| 105 | +} | |
| 91 | 106 | |
| 92 | 107 | enum { |
| 93 | 108 | Opt_gc_background, |
| @@ -4940,7 +4955,7 @@ static int __init init_f2fs_fs(void) · 4846 unmodified lines | ||
| 4940 | 4955 | err = f2fs_init_sysfs(); |
| 4941 | 4956 | if (err) |
| 4942 | 4957 | goto free_garbage_collection_cache; |
| 4943 | − err = register_shrinker(&f2fs_shrinker_info, "f2fs-shrinker"); | |
| 4958 | + err = f2fs_init_shrinker(); | |
| 4944 | 4959 | if (err) |
| 4945 | 4960 | goto free_sysfs; |
| 4946 | 4961 | err = register_filesystem(&f2fs_fs_type); |
| @@ -4985,7 +5000,7 @@ free_root_stats: · 38 unmodified lines | ||
| 4985 | 5000 | f2fs_destroy_root_stats(); |
| 4986 | 5001 | unregister_filesystem(&f2fs_fs_type); |
| 4987 | 5002 | free_shrinker: |
| 4988 | − unregister_shrinker(&f2fs_shrinker_info); | |
| 5003 | + f2fs_exit_shrinker(); | |
| 4989 | 5004 | free_sysfs: |
| 4990 | 5005 | f2fs_exit_sysfs(); |
| 4991 | 5006 | free_garbage_collection_cache: |
| @@ -5017,7 +5032,7 @@ static void __exit exit_f2fs_fs(void) · 25 unmodified lines | ||
| 5017 | 5032 | f2fs_destroy_post_read_processing(); |
| 5018 | 5033 | f2fs_destroy_root_stats(); |
| 5019 | 5034 | unregister_filesystem(&f2fs_fs_type); |
| 5020 | − unregister_shrinker(&f2fs_shrinker_info); | |
| 5035 | + f2fs_exit_shrinker(); | |
| 5021 | 5036 | f2fs_exit_sysfs(); |
| 5022 | 5037 | f2fs_destroy_garbage_collection_cache(); |
| 5023 | 5038 | f2fs_destroy_extent_cache(); |
| @@ -130,7 +130,7 @@ static int __gfs2_jdata_write_folio(struct folio *folio, · 129 unmodified lines | ||
| 130 | 130 | if (folio_test_checked(folio)) { |
| 131 | 131 | folio_clear_checked(folio); |
| 132 | 132 | if (!folio_buffers(folio)) { |
| 133 | − folio_create_empty_buffers(folio, | |
| 133 | + create_empty_buffers(folio, | |
| 134 | 134 | inode->i_sb->s_blocksize, |
| 135 | 135 | BIT(BH_Dirty)|BIT(BH_Uptodate)); |
| 136 | 136 | } |
| @@ -43,53 +43,51 @@ struct metapath { · 42 unmodified lines | ||
| 43 | 43 | static int punch_hole(struct gfs2_inode *ip, u64 offset, u64 length); |
| 44 | 44 | |
| 45 | 45 | /** |
| 46 | − * gfs2_unstuffer_page - unstuff a stuffed inode into a block cached by a page | |
| 46 | + * gfs2_unstuffer_folio - unstuff a stuffed inode into a block cached by a folio | |
| 47 | 47 | * @ip: the inode |
| 48 | 48 | * @dibh: the dinode buffer |
| 49 | 49 | * @block: the block number that was allocated |
| 50 | − * @page: The (optional) page. This is looked up if @page is NULL | |
| 50 | + * @folio: The folio. | |
| 51 | 51 | * |
| 52 | 52 | * Returns: errno |
| 53 | 53 | */ |
| 54 | − | |
| 55 | −static int gfs2_unstuffer_page(struct gfs2_inode *ip, struct buffer_head *dibh, | |
| 56 | − u64 block, struct page *page) | |
| 54 | +static int gfs2_unstuffer_folio(struct gfs2_inode *ip, struct buffer_head *dibh, | |
| 55 | + u64 block, struct folio *folio) | |
| 57 | 56 | { |
| 58 | 57 | struct inode *inode = &ip->i_inode; |
| 59 | 58 | |
| 60 | − if (!PageUptodate(page)) { | |
| 61 | − void *kaddr = kmap(page); | |
| 59 | + if (!folio_test_uptodate(folio)) { | |
| 60 | + void *kaddr = kmap_local_folio(folio, 0); | |
| 62 | 61 | u64 dsize = i_size_read(inode); |
| 63 | 62 | |
| 64 | 63 | memcpy(kaddr, dibh->b_data + sizeof(struct gfs2_dinode), dsize); |
| 65 | − memset(kaddr + dsize, 0, PAGE_SIZE - dsize); | |
| 66 | − kunmap(page); | |
| 64 | + memset(kaddr + dsize, 0, folio_size(folio) - dsize); | |
| 65 | + kunmap_local(kaddr); | |
| 67 | 66 | |
| 68 | − SetPageUptodate(page); | |
| 67 | + folio_mark_uptodate(folio); | |
| 69 | 68 | } |
| 70 | 69 | |
| 71 | 70 | if (gfs2_is_jdata(ip)) { |
| 72 | − struct buffer_head *bh; | |
| 71 | + struct buffer_head *bh = folio_buffers(folio); | |
| 73 | 72 | |
| 74 | − if (!page_has_buffers(page)) | |
| 75 | − create_empty_buffers(page, BIT(inode->i_blkbits), | |
| 76 | − BIT(BH_Uptodate)); | |
| 73 | + if (!bh) | |
| 74 | + bh = create_empty_buffers(folio, | |
| 75 | + BIT(inode->i_blkbits), BIT(BH_Uptodate)); | |
| 77 | 76 | |
| 78 | − bh = page_buffers(page); | |
| 79 | 77 | if (!buffer_mapped(bh)) |
| 80 | 78 | map_bh(bh, inode->i_sb, block); |
| 81 | 79 | |
| 82 | 80 | set_buffer_uptodate(bh); |
| 83 | 81 | gfs2_trans_add_data(ip->i_gl, bh); |
| 84 | 82 | } else { |
| 85 | − set_page_dirty(page); | |
| 83 | + folio_mark_dirty(folio); | |
| 86 | 84 | gfs2_ordered_add_inode(ip); |
| 87 | 85 | } |
| 88 | 86 | |
| 89 | 87 | return 0; |
| 90 | 88 | } |
| 91 | 89 | |
| 92 | −static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct page *page) | |
| 90 | +static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct folio *folio) | |
| 93 | 91 | { |
| 94 | 92 | struct buffer_head *bh, *dibh; |
| 95 | 93 | struct gfs2_dinode *di; |
| @@ -118,7 +116,7 @@ static int __gfs2_unstuff_inode(struct gfs2_inode *ip, struct page *page) · 22 unmodified lines | ||
| 118 | 116 | dibh, sizeof(struct gfs2_dinode)); |
| 119 | 117 | brelse(bh); |
| 120 | 118 | } else { |
| 121 | − error = gfs2_unstuffer_page(ip, dibh, block, page); | |
| 119 | + error = gfs2_unstuffer_folio(ip, dibh, block, folio); | |
| 122 | 120 | if (error) |
| 123 | 121 | goto out_brelse; |
| 124 | 122 | } |
| @@ -157,17 +155,17 @@ out_brelse: · 32 unmodified lines | ||
| 157 | 155 | int gfs2_unstuff_dinode(struct gfs2_inode *ip) |
| 158 | 156 | { |
| 159 | 157 | struct inode *inode = &ip->i_inode; |
| 160 | − struct page *page; | |
| 158 | + struct folio *folio; | |
| 161 | 159 | int error; |
| 162 | 160 | |
| 163 | 161 | down_write(&ip->i_rw_mutex); |
| 164 | − page = grab_cache_page(inode->i_mapping, 0); | |
| 165 | − error = -ENOMEM; | |
| 166 | − if (!page) | |
| 162 | + folio = filemap_grab_folio(inode->i_mapping, 0); | |
| 163 | + error = PTR_ERR(folio); | |
| 164 | + if (IS_ERR(folio)) | |
| 167 | 165 | goto out; |
| 168 | − error = __gfs2_unstuff_inode(ip, page); | |
| 169 | − unlock_page(page); | |
| 170 | − put_page(page); | |
| 166 | + error = __gfs2_unstuff_inode(ip, folio); | |
| 167 | + folio_unlock(folio); | |
| 168 | + folio_put(folio); | |
| 171 | 169 | out: |
| 172 | 170 | up_write(&ip->i_rw_mutex); |
| 173 | 171 | return error; |
| @@ -2041,11 +2041,7 @@ static unsigned long gfs2_glock_shrink_count(struct shrinker *shrink, · 2040 unmodified lines | ||
| 2041 | 2041 | return vfs_pressure_ratio(atomic_read(&lru_count)); |
| 2042 | 2042 | } |
| 2043 | 2043 | |
| 2044 | −static struct shrinker glock_shrinker = { | |
| 2045 | − .seeks = DEFAULT_SEEKS, | |
| 2046 | − .count_objects = gfs2_glock_shrink_count, | |
| 2047 | − .scan_objects = gfs2_glock_shrink_scan, | |
| 2048 | −}; | |
| 2044 | +static struct shrinker *glock_shrinker; | |
| 2049 | 2045 | |
| 2050 | 2046 | /** |
| 2051 | 2047 | * glock_hash_walk - Call a function for glock in a hash bucket |
| @@ -2465,13 +2461,18 @@ int __init gfs2_glock_init(void) · 413 unmodified lines | ||
| 2465 | 2461 | return -ENOMEM; |
| 2466 | 2462 | } |
| 2467 | 2463 | |
| 2468 | − ret = register_shrinker(&glock_shrinker, "gfs2-glock"); | |
| 2469 | − if (ret) { | |
| 2464 | + glock_shrinker = shrinker_alloc(0, "gfs2-glock"); | |
| 2465 | + if (!glock_shrinker) { | |
| 2470 | 2466 | destroy_workqueue(glock_workqueue); |
| 2471 | 2467 | rhashtable_destroy(&gl_hash_table); |
| 2472 | − return ret; | |
| 2468 | + return -ENOMEM; | |
| 2473 | 2469 | } |
| 2474 | 2470 | |
| 2471 | + glock_shrinker->count_objects = gfs2_glock_shrink_count; | |
| 2472 | + glock_shrinker->scan_objects = gfs2_glock_shrink_scan; | |
| 2473 | + | |
| 2474 | + shrinker_register(glock_shrinker); | |
| 2475 | + | |
| 2475 | 2476 | for (i = 0; i < GLOCK_WAIT_TABLE_SIZE; i++) |
| 2476 | 2477 | init_waitqueue_head(glock_wait_table + i); |
| 2477 | 2478 | |
| @@ -2480,7 +2481,7 @@ int __init gfs2_glock_init(void) · 2 unmodified lines | ||
| 2480 | 2481 | |
| 2481 | 2482 | void gfs2_glock_exit(void) |
| 2482 | 2483 | { |
| 2483 | − unregister_shrinker(&glock_shrinker); | |
| 2484 | + shrinker_free(glock_shrinker); | |
| 2484 | 2485 | rhashtable_destroy(&gl_hash_table); |
| 2485 | 2486 | destroy_workqueue(glock_workqueue); |
| 2486 | 2487 | } |
| @@ -147,7 +147,7 @@ static int __init init_gfs2_fs(void) · 146 unmodified lines | ||
| 147 | 147 | if (!gfs2_trans_cachep) |
| 148 | 148 | goto fail_cachep8; |
| 149 | 149 | |
| 150 | − error = register_shrinker(&gfs2_qd_shrinker, "gfs2-qd"); | |
| 150 | + error = gfs2_qd_shrinker_init(); | |
| 151 | 151 | if (error) |
| 152 | 152 | goto fail_shrinker; |
| 153 | 153 | |
| @@ -196,7 +196,7 @@ fail_wq3: · 42 unmodified lines | ||
| 196 | 196 | fail_wq2: |
| 197 | 197 | destroy_workqueue(gfs2_recovery_wq); |
| 198 | 198 | fail_wq1: |
| 199 | − unregister_shrinker(&gfs2_qd_shrinker); | |
| 199 | + gfs2_qd_shrinker_exit(); | |
| 200 | 200 | fail_shrinker: |
| 201 | 201 | kmem_cache_destroy(gfs2_trans_cachep); |
| 202 | 202 | fail_cachep8: |
| @@ -229,7 +229,7 @@ fail_lru: · 26 unmodified lines | ||
| 229 | 229 | |
| 230 | 230 | static void __exit exit_gfs2_fs(void) |
| 231 | 231 | { |
| 232 | − unregister_shrinker(&gfs2_qd_shrinker); | |
| 232 | + gfs2_qd_shrinker_exit(); | |
| 233 | 233 | gfs2_glock_exit(); |
| 234 | 234 | gfs2_unregister_debugfs(); |
| 235 | 235 | unregister_filesystem(&gfs2_fs_type); |
| @@ -115,7 +115,7 @@ struct buffer_head *gfs2_getbuf(struct gfs2_glock *gl, u64 blkno, int create) · 114 unmodified lines | ||
| 115 | 115 | { |
| 116 | 116 | struct address_space *mapping = gfs2_glock2aspace(gl); |
| 117 | 117 | struct gfs2_sbd *sdp = gl->gl_name.ln_sbd; |
| 118 | − struct page *page; | |
| 118 | + struct folio *folio; | |
| 119 | 119 | struct buffer_head *bh; |
| 120 | 120 | unsigned int shift; |
| 121 | 121 | unsigned long index; |
| @@ -129,36 +129,31 @@ struct buffer_head *gfs2_getbuf(struct gfs2_glock *gl, u64 blkno, int create) · 7 unmodified lines | ||
| 129 | 129 | bufnum = blkno - (index << shift); /* block buf index within page */ |
| 130 | 130 | |
| 131 | 131 | if (create) { |
| 132 | − for (;;) { | |
| 133 | − page = grab_cache_page(mapping, index); | |
| 134 | − if (page) | |
| 135 | − break; | |
| 136 | − yield(); | |
| 137 | − } | |
| 138 | − if (!page_has_buffers(page)) | |
| 139 | − create_empty_buffers(page, sdp->sd_sb.sb_bsize, 0); | |
| 132 | + folio = __filemap_get_folio(mapping, index, | |
| 133 | + FGP_LOCK | FGP_ACCESSED | FGP_CREAT, | |
| 134 | + mapping_gfp_mask(mapping) | __GFP_NOFAIL); | |
| 135 | + bh = folio_buffers(folio); | |
| 136 | + if (!bh) | |
| 137 | + bh = create_empty_buffers(folio, | |
| 138 | + sdp->sd_sb.sb_bsize, 0); | |
| 140 | 139 | } else { |
| 141 | − page = find_get_page_flags(mapping, index, | |
| 142 | − FGP_LOCK|FGP_ACCESSED); | |
| 143 | − if (!page) | |
| 140 | + folio = __filemap_get_folio(mapping, index, | |
| 141 | + FGP_LOCK | FGP_ACCESSED, 0); | |
| 142 | + if (IS_ERR(folio)) | |
| 144 | 143 | return NULL; |
| 145 | − if (!page_has_buffers(page)) { | |
| 146 | − bh = NULL; | |
| 147 | − goto out_unlock; | |
| 148 | − } | |
| 144 | + bh = folio_buffers(folio); | |
| 149 | 145 | } |
| 150 | 146 | |
| 151 | − /* Locate header for our buffer within our page */ | |
| 152 | − for (bh = page_buffers(page); bufnum--; bh = bh->b_this_page) | |
| 153 | − /* Do nothing */; | |
| 154 | − get_bh(bh); | |
| 147 | + if (!bh) | |
| 148 | + goto out_unlock; | |
| 155 | 149 | |
| 150 | + bh = get_nth_bh(bh, bufnum); | |
| 156 | 151 | if (!buffer_mapped(bh)) |
| 157 | 152 | map_bh(bh, sdp->sd_vfs, blkno); |
| 158 | 153 | |
| 159 | 154 | out_unlock: |
| 160 | − unlock_page(page); | |
| 161 | − put_page(page); | |
| 155 | + folio_unlock(folio); | |
| 156 | + folio_put(folio); | |
| 162 | 157 | |
| 163 | 158 | return bh; |
| 164 | 159 | } |
| @@ -405,26 +400,20 @@ static struct buffer_head *gfs2_getjdatabuf(struct gfs2_inode *ip, u64 blkno) · 240 unmodified lines | ||
| 405 | 400 | { |
| 406 | 401 | struct address_space *mapping = ip->i_inode.i_mapping; |
| 407 | 402 | struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode); |
| 408 | − struct page *page; | |
| 403 | + struct folio *folio; | |
| 409 | 404 | struct buffer_head *bh; |
| 410 | 405 | unsigned int shift = PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift; |
| 411 | 406 | unsigned long index = blkno >> shift; /* convert block to page */ |
| 412 | 407 | unsigned int bufnum = blkno - (index << shift); |
| 413 | 408 | |
| 414 | − page = find_get_page_flags(mapping, index, FGP_LOCK|FGP_ACCESSED); | |
| 415 | − if (!page) | |
| 409 | + folio = __filemap_get_folio(mapping, index, FGP_LOCK | FGP_ACCESSED, 0); | |
| 410 | + if (IS_ERR(folio)) | |
| 416 | 411 | return NULL; |
| 417 | − if (!page_has_buffers(page)) { | |
| 418 | − unlock_page(page); | |
| 419 | − put_page(page); | |
| 420 | − return NULL; | |
| 421 | − } | |
| 422 | − /* Locate header for our buffer within our page */ | |
| 423 | − for (bh = page_buffers(page); bufnum--; bh = bh->b_this_page) | |
| 424 | − /* Do nothing */; | |
| 425 | − get_bh(bh); | |
| 426 | − unlock_page(page); | |
| 427 | − put_page(page); | |
| 412 | + bh = folio_buffers(folio); | |
| 413 | + if (bh) | |
| 414 | + bh = get_nth_bh(bh, bufnum); | |
| 415 | + folio_unlock(folio); | |
| 416 | + folio_put(folio); | |
| 428 | 417 | return bh; |
| 429 | 418 | } |
| 430 | 419 | |
| @@ -196,13 +196,26 @@ static unsigned long gfs2_qd_shrink_count(struct shrinker *shrink, · 195 unmodified lines | ||
| 196 | 196 | return vfs_pressure_ratio(list_lru_shrink_count(&gfs2_qd_lru, sc)); |
| 197 | 197 | } |
| 198 | 198 | |
| 199 | −struct shrinker gfs2_qd_shrinker = { | |
| 200 | − .count_objects = gfs2_qd_shrink_count, | |
| 201 | − .scan_objects = gfs2_qd_shrink_scan, | |
| 202 | − .seeks = DEFAULT_SEEKS, | |
| 203 | − .flags = SHRINKER_NUMA_AWARE, | |
| 204 | −}; | |
| 199 | +static struct shrinker *gfs2_qd_shrinker; | |
| 200 | + | |
| 201 | +int __init gfs2_qd_shrinker_init(void) | |
| 202 | +{ | |
| 203 | + gfs2_qd_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "gfs2-qd"); | |
| 204 | + if (!gfs2_qd_shrinker) | |
| 205 | + return -ENOMEM; | |
| 206 | + | |
| 207 | + gfs2_qd_shrinker->count_objects = gfs2_qd_shrink_count; | |
| 208 | + gfs2_qd_shrinker->scan_objects = gfs2_qd_shrink_scan; | |
| 209 | + | |
| 210 | + shrinker_register(gfs2_qd_shrinker); | |
| 205 | 211 | |
| 212 | + return 0; | |
| 213 | +} | |
| 214 | + | |
| 215 | +void gfs2_qd_shrinker_exit(void) | |
| 216 | +{ | |
| 217 | + shrinker_free(gfs2_qd_shrinker); | |
| 218 | +} | |
| 206 | 219 | |
| 207 | 220 | static u64 qd2index(struct gfs2_quota_data *qd) |
| 208 | 221 | { |
| @@ -736,7 +749,7 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 527 unmodified lines | ||
| 736 | 749 | struct gfs2_inode *ip = GFS2_I(sdp->sd_quota_inode); |
| 737 | 750 | struct inode *inode = &ip->i_inode; |
| 738 | 751 | struct address_space *mapping = inode->i_mapping; |
| 739 | − struct page *page; | |
| 752 | + struct folio *folio; | |
| 740 | 753 | struct buffer_head *bh; |
| 741 | 754 | u64 blk; |
| 742 | 755 | unsigned bsize = sdp->sd_sb.sb_bsize, bnum = 0, boff = 0; |
| @@ -745,15 +758,15 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 2 unmodified lines | ||
| 745 | 758 | blk = index << (PAGE_SHIFT - sdp->sd_sb.sb_bsize_shift); |
| 746 | 759 | boff = off % bsize; |
| 747 | 760 | |
| 748 | − page = grab_cache_page(mapping, index); | |
| 749 | − if (!page) | |
| 750 | − return -ENOMEM; | |
| 751 | − if (!page_has_buffers(page)) | |
| 752 | − create_empty_buffers(page, bsize, 0); | |
| 761 | + folio = filemap_grab_folio(mapping, index); | |
| 762 | + if (IS_ERR(folio)) | |
| 763 | + return PTR_ERR(folio); | |
| 764 | + bh = folio_buffers(folio); | |
| 765 | + if (!bh) | |
| 766 | + bh = create_empty_buffers(folio, bsize, 0); | |
| 753 | 767 | |
| 754 | − bh = page_buffers(page); | |
| 755 | − for(;;) { | |
| 756 | − /* Find the beginning block within the page */ | |
| 768 | + for (;;) { | |
| 769 | + /* Find the beginning block within the folio */ | |
| 757 | 770 | if (pg_off >= ((bnum * bsize) + bsize)) { |
| 758 | 771 | bh = bh->b_this_page; |
| 759 | 772 | bnum++; |
| @@ -766,9 +779,10 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 6 unmodified lines | ||
| 766 | 779 | goto unlock_out; |
| 767 | 780 | /* If it's a newly allocated disk block, zero it */ |
| 768 | 781 | if (buffer_new(bh)) |
| 769 | − zero_user(page, bnum * bsize, bh->b_size); | |
| 782 | + folio_zero_range(folio, bnum * bsize, | |
| 783 | + bh->b_size); | |
| 770 | 784 | } |
| 771 | − if (PageUptodate(page)) | |
| 785 | + if (folio_test_uptodate(folio)) | |
| 772 | 786 | set_buffer_uptodate(bh); |
| 773 | 787 | if (bh_read(bh, REQ_META | REQ_PRIO) < 0) |
| 774 | 788 | goto unlock_out; |
| @@ -784,17 +798,17 @@ static int gfs2_write_buf_to_page(struct gfs2_sbd *sdp, unsigned long index, · 9 unmodified lines | ||
| 784 | 798 | break; |
| 785 | 799 | } |
| 786 | 800 | |
| 787 | − /* Write to the page, now that we have setup the buffer(s) */ | |
| 788 | − memcpy_to_page(page, off, buf, bytes); | |
| 789 | − flush_dcache_page(page); | |
| 790 | − unlock_page(page); | |
| 791 | − put_page(page); | |
| 801 | + /* Write to the folio, now that we have setup the buffer(s) */ | |
| 802 | + memcpy_to_folio(folio, off, buf, bytes); | |
| 803 | + flush_dcache_folio(folio); | |
| 804 | + folio_unlock(folio); | |
| 805 | + folio_put(folio); | |
| 792 | 806 | |
| 793 | 807 | return 0; |
| 794 | 808 | |
| 795 | 809 | unlock_out: |
| 796 | − unlock_page(page); | |
| 797 | − put_page(page); | |
| 810 | + folio_unlock(folio); | |
| 811 | + folio_put(folio); | |
| 798 | 812 | return -EIO; |
| 799 | 813 | } |
| 800 | 814 | |
| @@ -60,7 +60,8 @@ static inline int gfs2_quota_lock_check(struct gfs2_inode *ip, · 59 unmodified lines | ||
| 60 | 60 | } |
| 61 | 61 | |
| 62 | 62 | extern const struct quotactl_ops gfs2_quotactl_ops; |
| 63 | −extern struct shrinker gfs2_qd_shrinker; | |
| 63 | +int __init gfs2_qd_shrinker_init(void); | |
| 64 | +void gfs2_qd_shrinker_exit(void); | |
| 64 | 65 | extern struct list_lru gfs2_qd_lru; |
| 65 | 66 | extern void __init gfs2_quota_hash_init(void); |
| 66 | 67 | |
| @@ -83,29 +83,6 @@ static const struct fs_parameter_spec hugetlb_fs_parameters[] = { · 82 unmodified lines | ||
| 83 | 83 | {} |
| 84 | 84 | }; |
| 85 | 85 | |
| 86 | −#ifdef CONFIG_NUMA | |
| 87 | −static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma, | |
| 88 | − struct inode *inode, pgoff_t index) | |
| 89 | −{ | |
| 90 | − vma->vm_policy = mpol_shared_policy_lookup(&HUGETLBFS_I(inode)->policy, | |
| 91 | − index); | |
| 92 | −} | |
| 93 | − | |
| 94 | −static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma) | |
| 95 | −{ | |
| 96 | − mpol_cond_put(vma->vm_policy); | |
| 97 | −} | |
| 98 | −#else | |
| 99 | −static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma, | |
| 100 | − struct inode *inode, pgoff_t index) | |
| 101 | −{ | |
| 102 | −} | |
| 103 | − | |
| 104 | −static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma) | |
| 105 | −{ | |
| 106 | −} | |
| 107 | −#endif | |
| 108 | − | |
| 109 | 86 | /* |
| 110 | 87 | * Mask used when checking the page offset value passed in via system |
| 111 | 88 | * calls. This value will be converted to a loff_t which is signed. |
| @@ -135,7 +112,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) · 23 unmodified lines | ||
| 135 | 112 | vm_flags_set(vma, VM_HUGETLB | VM_DONTEXPAND); |
| 136 | 113 | vma->vm_ops = &hugetlb_vm_ops; |
| 137 | 114 | |
| 138 | − ret = seal_check_future_write(info->seals, vma); | |
| 115 | + ret = seal_check_write(info->seals, vma); | |
| 139 | 116 | if (ret) |
| 140 | 117 | return ret; |
| 141 | 118 | |
| @@ -295,7 +272,7 @@ static size_t adjust_range_hwpoison(struct page *page, size_t offset, size_t byt · 153 unmodified lines | ||
| 295 | 272 | size_t res = 0; |
| 296 | 273 | |
| 297 | 274 | /* First subpage to start the loop. */ |
| 298 | − page += offset / PAGE_SIZE; | |
| 275 | + page = nth_page(page, offset / PAGE_SIZE); | |
| 299 | 276 | offset %= PAGE_SIZE; |
| 300 | 277 | while (1) { |
| 301 | 278 | if (is_raw_hwpoison_page_in_hugepage(page)) |
| @@ -309,7 +286,7 @@ static size_t adjust_range_hwpoison(struct page *page, size_t offset, size_t byt · 7 unmodified lines | ||
| 309 | 286 | break; |
| 310 | 287 | offset += n; |
| 311 | 288 | if (offset == PAGE_SIZE) { |
| 312 | − page++; | |
| 289 | + page = nth_page(page, 1); | |
| 313 | 290 | offset = 0; |
| 314 | 291 | } |
| 315 | 292 | } |
| @@ -334,7 +311,7 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 18 unmodified lines | ||
| 334 | 311 | ssize_t retval = 0; |
| 335 | 312 | |
| 336 | 313 | while (iov_iter_count(to)) { |
| 337 | − struct page *page; | |
| 314 | + struct folio *folio; | |
| 338 | 315 | size_t nr, copied, want; |
| 339 | 316 | |
| 340 | 317 | /* nr is the maximum number of bytes to copy from this page */ |
| @@ -352,18 +329,18 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 11 unmodified lines | ||
| 352 | 329 | } |
| 353 | 330 | nr = nr - offset; |
| 354 | 331 | |
| 355 | − /* Find the page */ | |
| 356 | − page = find_lock_page(mapping, index); | |
| 357 | − if (unlikely(page == NULL)) { | |
| 332 | + /* Find the folio */ | |
| 333 | + folio = filemap_lock_hugetlb_folio(h, mapping, index); | |
| 334 | + if (IS_ERR(folio)) { | |
| 358 | 335 | /* |
| 359 | 336 | * We have a HOLE, zero out the user-buffer for the |
| 360 | 337 | * length of the hole or request. |
| 361 | 338 | */ |
| 362 | 339 | copied = iov_iter_zero(nr, to); |
| 363 | 340 | } else { |
| 364 | − unlock_page(page); | |
| 341 | + folio_unlock(folio); | |
| 365 | 342 | |
| 366 | − if (!PageHWPoison(page)) | |
| 343 | + if (!folio_test_has_hwpoisoned(folio)) | |
| 367 | 344 | want = nr; |
| 368 | 345 | else { |
| 369 | 346 | /* |
| @@ -371,19 +348,19 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) · 1 unmodified line | ||
| 371 | 348 | * touching the 1st raw HWPOISON subpage after |
| 372 | 349 | * offset. |
| 373 | 350 | */ |
| 374 | − want = adjust_range_hwpoison(page, offset, nr); | |
| 351 | + want = adjust_range_hwpoison(&folio->page, offset, nr); | |
| 375 | 352 | if (want == 0) { |
| 376 | − put_page(page); | |
| 353 | + folio_put(folio); | |
| 377 | 354 | retval = -EIO; |
| 378 | 355 | break; |
| 379 | 356 | } |
| 380 | 357 | } |
| 381 | 358 | |
| 382 | 359 | /* |
| 383 | − * We have the page, copy it to user space buffer. | |
| 360 | + * We have the folio, copy it to user space buffer. | |
| 384 | 361 | */ |
| 385 | − copied = copy_page_to_iter(page, offset, want, to); | |
| 386 | − put_page(page); | |
| 362 | + copied = copy_folio_to_iter(folio, offset, want, to); | |
| 363 | + folio_put(folio); | |
| 387 | 364 | } |
| 388 | 365 | offset += copied; |
| 389 | 366 | retval += copied; |
| @@ -661,21 +638,20 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, · 271 unmodified lines | ||
| 661 | 638 | { |
| 662 | 639 | struct hstate *h = hstate_inode(inode); |
| 663 | 640 | struct address_space *mapping = &inode->i_data; |
| 664 | − const pgoff_t start = lstart >> huge_page_shift(h); | |
| 665 | − const pgoff_t end = lend >> huge_page_shift(h); | |
| 641 | + const pgoff_t end = lend >> PAGE_SHIFT; | |
| 666 | 642 | struct folio_batch fbatch; |
| 667 | 643 | pgoff_t next, index; |
| 668 | 644 | int i, freed = 0; |
| 669 | 645 | bool truncate_op = (lend == LLONG_MAX); |
| 670 | 646 | |
| 671 | 647 | folio_batch_init(&fbatch); |
| 672 | − next = start; | |
| 648 | + next = lstart >> PAGE_SHIFT; | |
| 673 | 649 | while (filemap_get_folios(mapping, &next, end - 1, &fbatch)) { |
| 674 | 650 | for (i = 0; i < folio_batch_count(&fbatch); ++i) { |
| 675 | 651 | struct folio *folio = fbatch.folios[i]; |
| 676 | 652 | u32 hash = 0; |
| 677 | 653 | |
| 678 | − index = folio->index; | |
| 654 | + index = folio->index >> huge_page_order(h); | |
| 679 | 655 | hash = hugetlb_fault_mutex_hash(mapping, index); |
| 680 | 656 | mutex_lock(&hugetlb_fault_mutex_table[hash]); |
| 681 | 657 | |
| @@ -693,7 +669,9 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, · 11 unmodified lines | ||
| 693 | 669 | } |
| 694 | 670 | |
| 695 | 671 | if (truncate_op) |
| 696 | − (void)hugetlb_unreserve_pages(inode, start, LONG_MAX, freed); | |
| 672 | + (void)hugetlb_unreserve_pages(inode, | |
| 673 | + lstart >> huge_page_shift(h), | |
| 674 | + LONG_MAX, freed); | |
| 697 | 675 | } |
| 698 | 676 | |
| 699 | 677 | static void hugetlbfs_evict_inode(struct inode *inode) |
| @@ -741,7 +719,7 @@ static void hugetlbfs_zero_partial_page(struct hstate *h, · 41 unmodified lines | ||
| 741 | 719 | pgoff_t idx = start >> huge_page_shift(h); |
| 742 | 720 | struct folio *folio; |
| 743 | 721 | |
| 744 | − folio = filemap_lock_folio(mapping, idx); | |
| 722 | + folio = filemap_lock_hugetlb_folio(h, mapping, idx); | |
| 745 | 723 | if (IS_ERR(folio)) |
| 746 | 724 | return; |
| 747 | 725 | |
| @@ -852,8 +830,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 104 unmodified lines | ||
| 852 | 830 | |
| 853 | 831 | /* |
| 854 | 832 | * Initialize a pseudo vma as this is required by the huge page |
| 855 | − * allocation routines. If NUMA is configured, use page index | |
| 856 | − * as input to create an allocation policy. | |
| 833 | + * allocation routines. | |
| 857 | 834 | */ |
| 858 | 835 | vma_init(&pseudo_vma, mm); |
| 859 | 836 | vm_flags_init(&pseudo_vma, VM_HUGETLB | VM_MAYSHARE | VM_SHARED); |
| @@ -886,7 +863,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 26 unmodified lines | ||
| 886 | 863 | mutex_lock(&hugetlb_fault_mutex_table[hash]); |
| 887 | 864 | |
| 888 | 865 | /* See if already present in mapping to avoid alloc/free */ |
| 889 | − folio = filemap_get_folio(mapping, index); | |
| 866 | + folio = filemap_get_folio(mapping, index << huge_page_order(h)); | |
| 890 | 867 | if (!IS_ERR(folio)) { |
| 891 | 868 | folio_put(folio); |
| 892 | 869 | mutex_unlock(&hugetlb_fault_mutex_table[hash]); |
| @@ -901,9 +878,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, · 8 unmodified lines | ||
| 901 | 878 | * folios in these areas, we need to consume the reserves |
| 902 | 879 | * to keep reservation accounting consistent. |
| 903 | 880 | */ |
| 904 | − hugetlb_set_vma_policy(&pseudo_vma, inode, index); | |
| 905 | 881 | folio = alloc_hugetlb_folio(&pseudo_vma, addr, 0); |
| 906 | − hugetlb_drop_vma_policy(&pseudo_vma); | |
| 907 | 882 | if (IS_ERR(folio)) { |
| 908 | 883 | mutex_unlock(&hugetlb_fault_mutex_table[hash]); |
| 909 | 884 | error = PTR_ERR(folio); |
| @@ -1282,18 +1257,6 @@ static struct inode *hugetlbfs_alloc_inode(struct super_block *sb) · 372 unmodified lines | ||
| 1282 | 1257 | hugetlbfs_inc_free_inodes(sbinfo); |
| 1283 | 1258 | return NULL; |
| 1284 | 1259 | } |
| 1285 | − | |
| 1286 | − /* | |
| 1287 | − * Any time after allocation, hugetlbfs_destroy_inode can be called | |
| 1288 | − * for the inode. mpol_free_shared_policy is unconditionally called | |
| 1289 | − * as part of hugetlbfs_destroy_inode. So, initialize policy here | |
| 1290 | − * in case of a quick call to destroy. | |
| 1291 | − * | |
| 1292 | − * Note that the policy is initialized even if we are creating a | |
| 1293 | − * private inode. This simplifies hugetlbfs_destroy_inode. | |
| 1294 | − */ | |
| 1295 | − mpol_shared_policy_init(&p->policy, NULL); | |
| 1296 | − | |
| 1297 | 1260 | return &p->vfs_inode; |
| 1298 | 1261 | } |
| 1299 | 1262 | |
| @@ -1305,7 +1268,6 @@ static void hugetlbfs_free_inode(struct inode *inode) · 5 unmodified lines | ||
| 1305 | 1268 | static void hugetlbfs_destroy_inode(struct inode *inode) |
| 1306 | 1269 | { |
| 1307 | 1270 | hugetlbfs_inc_free_inodes(HUGETLBFS_SB(inode->i_sb)); |
| 1308 | − mpol_free_shared_policy(&HUGETLBFS_I(inode)->policy); | |
| 1309 | 1271 | } |
| 1310 | 1272 | |
| 1311 | 1273 | static const struct address_space_operations hugetlbfs_aops = { |
| @@ -29,9 +29,9 @@ typedef int (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length); · 28 unmodified lines | ||
| 29 | 29 | * and I/O completions. |
| 30 | 30 | */ |
| 31 | 31 | struct iomap_folio_state { |
| 32 | − atomic_t read_bytes_pending; | |
| 33 | − atomic_t write_bytes_pending; | |
| 34 | 32 | spinlock_t state_lock; |
| 33 | + unsigned int read_bytes_pending; | |
| 34 | + atomic_t write_bytes_pending; | |
| 35 | 35 | |
| 36 | 36 | /* |
| 37 | 37 | * Each block has two bits in this bitmap: |
| @@ -57,30 +57,32 @@ static inline bool ifs_block_is_uptodate(struct iomap_folio_state *ifs, · 19 unmodified lines | ||
| 57 | 57 | return test_bit(block, ifs->state); |
| 58 | 58 | } |
| 59 | 59 | |
| 60 | −static void ifs_set_range_uptodate(struct folio *folio, | |
| 60 | +static bool ifs_set_range_uptodate(struct folio *folio, | |
| 61 | 61 | struct iomap_folio_state *ifs, size_t off, size_t len) |
| 62 | 62 | { |
| 63 | 63 | struct inode *inode = folio->mapping->host; |
| 64 | 64 | unsigned int first_blk = off >> inode->i_blkbits; |
| 65 | 65 | unsigned int last_blk = (off + len - 1) >> inode->i_blkbits; |
| 66 | 66 | unsigned int nr_blks = last_blk - first_blk + 1; |
| 67 | − unsigned long flags; | |
| 68 | 67 | |
| 69 | − spin_lock_irqsave(&ifs->state_lock, flags); | |
| 70 | 68 | bitmap_set(ifs->state, first_blk, nr_blks); |
| 71 | − if (ifs_is_fully_uptodate(folio, ifs)) | |
| 72 | − folio_mark_uptodate(folio); | |
| 73 | − spin_unlock_irqrestore(&ifs->state_lock, flags); | |
| 69 | + return ifs_is_fully_uptodate(folio, ifs); | |
| 74 | 70 | } |
| 75 | 71 | |
| 76 | 72 | static void iomap_set_range_uptodate(struct folio *folio, size_t off, |
| 77 | 73 | size_t len) |
| 78 | 74 | { |
| 79 | 75 | struct iomap_folio_state *ifs = folio->private; |
| 76 | + unsigned long flags; | |
| 77 | + bool uptodate = true; | |
| 80 | 78 | |
| 81 | − if (ifs) | |
| 82 | − ifs_set_range_uptodate(folio, ifs, off, len); | |
| 83 | − else | |
| 79 | + if (ifs) { | |
| 80 | + spin_lock_irqsave(&ifs->state_lock, flags); | |
| 81 | + uptodate = ifs_set_range_uptodate(folio, ifs, off, len); | |
| 82 | + spin_unlock_irqrestore(&ifs->state_lock, flags); | |
| 83 | + } | |
| 84 | + | |
| 85 | + if (uptodate) | |
| 84 | 86 | folio_mark_uptodate(folio); |
| 85 | 87 | } |
| 86 | 88 | |
| @@ -181,7 +183,7 @@ static void ifs_free(struct folio *folio) · 94 unmodified lines | ||
| 181 | 183 | |
| 182 | 184 | if (!ifs) |
| 183 | 185 | return; |
| 184 | − WARN_ON_ONCE(atomic_read(&ifs->read_bytes_pending)); | |
| 186 | + WARN_ON_ONCE(ifs->read_bytes_pending != 0); | |
| 185 | 187 | WARN_ON_ONCE(atomic_read(&ifs->write_bytes_pending)); |
| 186 | 188 | WARN_ON_ONCE(ifs_is_fully_uptodate(folio, ifs) != |
| 187 | 189 | folio_test_uptodate(folio)); |
| @@ -248,20 +250,28 @@ static void iomap_adjust_read_range(struct inode *inode, struct folio *folio, · 60 unmodified lines | ||
| 248 | 250 | *lenp = plen; |
| 249 | 251 | } |
| 250 | 252 | |
| 251 | −static void iomap_finish_folio_read(struct folio *folio, size_t offset, | |
| 253 | +static void iomap_finish_folio_read(struct folio *folio, size_t off, | |
| 252 | 254 | size_t len, int error) |
| 253 | 255 | { |
| 254 | 256 | struct iomap_folio_state *ifs = folio->private; |
| 257 | + bool uptodate = !error; | |
| 258 | + bool finished = true; | |
| 255 | 259 | |
| 256 | − if (unlikely(error)) { | |
| 257 | − folio_clear_uptodate(folio); | |
| 258 | − folio_set_error(folio); | |
| 259 | − } else { | |
| 260 | − iomap_set_range_uptodate(folio, offset, len); | |
| 260 | + if (ifs) { | |
| 261 | + unsigned long flags; | |
| 262 | + | |
| 263 | + spin_lock_irqsave(&ifs->state_lock, flags); | |
| 264 | + if (!error) | |
| 265 | + uptodate = ifs_set_range_uptodate(folio, ifs, off, len); | |
| 266 | + ifs->read_bytes_pending -= len; | |
| 267 | + finished = !ifs->read_bytes_pending; | |
| 268 | + spin_unlock_irqrestore(&ifs->state_lock, flags); | |
| 261 | 269 | } |
| 262 | 270 | |
| 263 | − if (!ifs || atomic_sub_and_test(len, &ifs->read_bytes_pending)) | |
| 264 | − folio_unlock(folio); | |
| 271 | + if (error) | |
| 272 | + folio_set_error(folio); | |
| 273 | + if (finished) | |
| 274 | + folio_end_read(folio, uptodate); | |
| 265 | 275 | } |
| 266 | 276 | |
| 267 | 277 | static void iomap_read_end_io(struct bio *bio) |
| @@ -358,8 +368,11 @@ static loff_t iomap_readpage_iter(const struct iomap_iter *iter, · 90 unmodified lines | ||
| 358 | 368 | } |
| 359 | 369 | |
| 360 | 370 | ctx->cur_folio_in_bio = true; |
| 361 | − if (ifs) | |
| 362 | − atomic_add(plen, &ifs->read_bytes_pending); | |
| 371 | + if (ifs) { | |
| 372 | + spin_lock_irq(&ifs->state_lock); | |
| 373 | + ifs->read_bytes_pending += plen; | |
| 374 | + spin_unlock_irq(&ifs->state_lock); | |
| 375 | + } | |
| 363 | 376 | |
| 364 | 377 | sector = iomap_sector(iomap, pos); |
| 365 | 378 | if (!ctx->bio || |
| @@ -1290,7 +1290,7 @@ static int jbd2_min_tag_size(void) · 1289 unmodified lines | ||
| 1290 | 1290 | static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink, |
| 1291 | 1291 | struct shrink_control *sc) |
| 1292 | 1292 | { |
| 1293 | − journal_t *journal = container_of(shrink, journal_t, j_shrinker); | |
| 1293 | + journal_t *journal = shrink->private_data; | |
| 1294 | 1294 | unsigned long nr_to_scan = sc->nr_to_scan; |
| 1295 | 1295 | unsigned long nr_shrunk; |
| 1296 | 1296 | unsigned long count; |
| @@ -1316,7 +1316,7 @@ static unsigned long jbd2_journal_shrink_scan(struct shrinker *shrink, · 19 unmodified lines | ||
| 1316 | 1316 | static unsigned long jbd2_journal_shrink_count(struct shrinker *shrink, |
| 1317 | 1317 | struct shrink_control *sc) |
| 1318 | 1318 | { |
| 1319 | − journal_t *journal = container_of(shrink, journal_t, j_shrinker); | |
| 1319 | + journal_t *journal = shrink->private_data; | |
| 1320 | 1320 | unsigned long count; |
| 1321 | 1321 | |
| 1322 | 1322 | count = percpu_counter_read_positive(&journal->j_checkpoint_jh_count); |
| @@ -1588,14 +1588,21 @@ static journal_t *journal_init_common(struct block_device *bdev, · 265 unmodified lines | ||
| 1588 | 1588 | goto err_cleanup; |
| 1589 | 1589 | |
| 1590 | 1590 | journal->j_shrink_transaction = NULL; |
| 1591 | − journal->j_shrinker.scan_objects = jbd2_journal_shrink_scan; | |
| 1592 | − journal->j_shrinker.count_objects = jbd2_journal_shrink_count; | |
| 1593 | − journal->j_shrinker.seeks = DEFAULT_SEEKS; | |
| 1594 | − journal->j_shrinker.batch = journal->j_max_transaction_buffers; | |
| 1595 | − err = register_shrinker(&journal->j_shrinker, "jbd2-journal:(%u:%u)", | |
| 1596 | − MAJOR(bdev->bd_dev), MINOR(bdev->bd_dev)); | |
| 1597 | − if (err) | |
| 1591 | + | |
| 1592 | + journal->j_shrinker = shrinker_alloc(0, "jbd2-journal:(%u:%u)", | |
| 1593 | + MAJOR(bdev->bd_dev), | |
| 1594 | + MINOR(bdev->bd_dev)); | |
| 1595 | + if (!journal->j_shrinker) { | |
| 1596 | + err = -ENOMEM; | |
| 1598 | 1597 | goto err_cleanup; |
| 1598 | + } | |
| 1599 | + | |
| 1600 | + journal->j_shrinker->scan_objects = jbd2_journal_shrink_scan; | |
| 1601 | + journal->j_shrinker->count_objects = jbd2_journal_shrink_count; | |
| 1602 | + journal->j_shrinker->batch = journal->j_max_transaction_buffers; | |
| 1603 | + journal->j_shrinker->private_data = journal; | |
| 1604 | + | |
| 1605 | + shrinker_register(journal->j_shrinker); | |
| 1599 | 1606 | |
| 1600 | 1607 | return journal; |
| 1601 | 1608 | |
| @@ -2172,9 +2179,9 @@ int jbd2_journal_destroy(journal_t *journal) · 570 unmodified lines | ||
| 2172 | 2179 | brelse(journal->j_sb_buffer); |
| 2173 | 2180 | } |
| 2174 | 2181 | |
| 2175 | − if (journal->j_shrinker.flags & SHRINKER_REGISTERED) { | |
| 2182 | + if (journal->j_shrinker) { | |
| 2176 | 2183 | percpu_counter_destroy(&journal->j_checkpoint_jh_count); |
| 2177 | − unregister_shrinker(&journal->j_shrinker); | |
| 2184 | + shrinker_free(journal->j_shrinker); | |
| 2178 | 2185 | } |
| 2179 | 2186 | if (journal->j_proc_entry) |
| 2180 | 2187 | jbd2_stats_proc_exit(journal); |
| @@ -429,60 +429,11 @@ static int kernfs_vma_access(struct vm_area_struct *vma, unsigned long addr, · 428 unmodified lines | ||
| 429 | 429 | return ret; |
| 430 | 430 | } |
| 431 | 431 | |
| 432 | −#ifdef CONFIG_NUMA | |
| 433 | −static int kernfs_vma_set_policy(struct vm_area_struct *vma, | |
| 434 | − struct mempolicy *new) | |
| 435 | −{ | |
| 436 | − struct file *file = vma->vm_file; | |
| 437 | − struct kernfs_open_file *of = kernfs_of(file); | |
| 438 | − int ret; | |
| 439 | − | |
| 440 | − if (!of->vm_ops) | |
| 441 | − return 0; | |
| 442 | − | |
| 443 | − if (!kernfs_get_active(of->kn)) | |
| 444 | − return -EINVAL; | |
| 445 | − | |
| 446 | − ret = 0; | |
| 447 | − if (of->vm_ops->set_policy) | |
| 448 | − ret = of->vm_ops->set_policy(vma, new); | |
| 449 | − | |
| 450 | − kernfs_put_active(of->kn); | |
| 451 | − return ret; | |
| 452 | −} | |
| 453 | − | |
| 454 | −static struct mempolicy *kernfs_vma_get_policy(struct vm_area_struct *vma, | |
| 455 | − unsigned long addr) | |
| 456 | −{ | |
| 457 | − struct file *file = vma->vm_file; | |
| 458 | − struct kernfs_open_file *of = kernfs_of(file); | |
| 459 | − struct mempolicy *pol; | |
| 460 | − | |
| 461 | − if (!of->vm_ops) | |
| 462 | − return vma->vm_policy; | |
| 463 | − | |
| 464 | − if (!kernfs_get_active(of->kn)) | |
| 465 | − return vma->vm_policy; | |
| 466 | − | |
| 467 | − pol = vma->vm_policy; | |
| 468 | − if (of->vm_ops->get_policy) | |
| 469 | − pol = of->vm_ops->get_policy(vma, addr); | |
| 470 | − | |
| 471 | − kernfs_put_active(of->kn); | |
| 472 | − return pol; | |
| 473 | −} | |
| 474 | − | |
| 475 | −#endif | |
| 476 | − | |
| 477 | 432 | static const struct vm_operations_struct kernfs_vm_ops = { |
| 478 | 433 | .open = kernfs_vma_open, |
| 479 | 434 | .fault = kernfs_vma_fault, |
| 480 | 435 | .page_mkwrite = kernfs_vma_page_mkwrite, |
| 481 | 436 | .access = kernfs_vma_access, |
| 482 | −#ifdef CONFIG_NUMA | |
| 483 | − .set_policy = kernfs_vma_set_policy, | |
| 484 | − .get_policy = kernfs_vma_get_policy, | |
| 485 | −#endif | |
| 486 | 437 | }; |
| 487 | 438 | |
| 488 | 439 | static int kernfs_fop_mmap(struct file *file, struct vm_area_struct *vma) |
| @@ -265,7 +265,7 @@ static int kernfs_fill_super(struct super_block *sb, struct kernfs_fs_context *k · 264 unmodified lines | ||
| 265 | 265 | sb->s_time_gran = 1; |
| 266 | 266 | |
| 267 | 267 | /* sysfs dentries and inodes don't require IO to create */ |
| 268 | − sb->s_shrink.seeks = 0; | |
| 268 | + sb->s_shrink->seeks = 0; | |
| 269 | 269 | |
| 270 | 270 | /* get root inode, initialize and unlock it */ |
| 271 | 271 | down_read(&kf_root->kernfs_rwsem); |
| @@ -37,7 +37,7 @@ struct mb_cache { · 36 unmodified lines | ||
| 37 | 37 | struct list_head c_list; |
| 38 | 38 | /* Number of entries in cache */ |
| 39 | 39 | unsigned long c_entry_count; |
| 40 | − struct shrinker c_shrink; | |
| 40 | + struct shrinker *c_shrink; | |
| 41 | 41 | /* Work for shrinking when the cache has too many entries */ |
| 42 | 42 | struct work_struct c_shrink_work; |
| 43 | 43 | }; |
| @@ -293,8 +293,7 @@ EXPORT_SYMBOL(mb_cache_entry_touch); · 249 unmodified lines | ||
| 293 | 293 | static unsigned long mb_cache_count(struct shrinker *shrink, |
| 294 | 294 | struct shrink_control *sc) |
| 295 | 295 | { |
| 296 | − struct mb_cache *cache = container_of(shrink, struct mb_cache, | |
| 297 | − c_shrink); | |
| 296 | + struct mb_cache *cache = shrink->private_data; | |
| 298 | 297 | |
| 299 | 298 | return cache->c_entry_count; |
| 300 | 299 | } |
| @@ -333,8 +332,7 @@ static unsigned long mb_cache_shrink(struct mb_cache *cache, · 32 unmodified lines | ||
| 333 | 332 | static unsigned long mb_cache_scan(struct shrinker *shrink, |
| 334 | 333 | struct shrink_control *sc) |
| 335 | 334 | { |
| 336 | − struct mb_cache *cache = container_of(shrink, struct mb_cache, | |
| 337 | − c_shrink); | |
| 335 | + struct mb_cache *cache = shrink->private_data; | |
| 338 | 336 | return mb_cache_shrink(cache, sc->nr_to_scan); |
| 339 | 337 | } |
| 340 | 338 | |
| @@ -377,15 +375,19 @@ struct mb_cache *mb_cache_create(int bucket_bits) · 36 unmodified lines | ||
| 377 | 375 | for (i = 0; i < bucket_count; i++) |
| 378 | 376 | INIT_HLIST_BL_HEAD(&cache->c_hash[i]); |
| 379 | 377 | |
| 380 | − cache->c_shrink.count_objects = mb_cache_count; | |
| 381 | − cache->c_shrink.scan_objects = mb_cache_scan; | |
| 382 | − cache->c_shrink.seeks = DEFAULT_SEEKS; | |
| 383 | − if (register_shrinker(&cache->c_shrink, "mbcache-shrinker")) { | |
| 378 | + cache->c_shrink = shrinker_alloc(0, "mbcache-shrinker"); | |
| 379 | + if (!cache->c_shrink) { | |
| 384 | 380 | kfree(cache->c_hash); |
| 385 | 381 | kfree(cache); |
| 386 | 382 | goto err_out; |
| 387 | 383 | } |
| 388 | 384 | |
| 385 | + cache->c_shrink->count_objects = mb_cache_count; | |
| 386 | + cache->c_shrink->scan_objects = mb_cache_scan; | |
| 387 | + cache->c_shrink->private_data = cache; | |
| 388 | + | |
| 389 | + shrinker_register(cache->c_shrink); | |
| 390 | + | |
| 389 | 391 | INIT_WORK(&cache->c_shrink_work, mb_cache_shrink_worker); |
| 390 | 392 | |
| 391 | 393 | return cache; |
| @@ -406,7 +408,7 @@ void mb_cache_destroy(struct mb_cache *cache) · 14 unmodified lines | ||
| 406 | 408 | { |
| 407 | 409 | struct mb_cache_entry *entry, *next; |
| 408 | 410 | |
| 409 | − unregister_shrinker(&cache->c_shrink); | |
| 411 | + shrinker_free(cache->c_shrink); | |
| 410 | 412 | |
| 411 | 413 | /* |
| 412 | 414 | * We don't bother with any locking. Cache must not be used at this |
| @@ -119,8 +119,7 @@ static void map_buffer_to_folio(struct folio *folio, struct buffer_head *bh, · 118 unmodified lines | ||
| 119 | 119 | folio_mark_uptodate(folio); |
| 120 | 120 | return; |
| 121 | 121 | } |
| 122 | − create_empty_buffers(&folio->page, i_blocksize(inode), 0); | |
| 123 | − head = folio_buffers(folio); | |
| 122 | + head = create_empty_buffers(folio, i_blocksize(inode), 0); | |
| 124 | 123 | } |
| 125 | 124 | |
| 126 | 125 | page_bh = head; |
| @@ -796,28 +796,9 @@ static unsigned long nfs4_xattr_cache_scan(struct shrinker *shrink, · 795 unmodified lines | ||
| 796 | 796 | static unsigned long nfs4_xattr_entry_scan(struct shrinker *shrink, |
| 797 | 797 | struct shrink_control *sc); |
| 798 | 798 | |
| 799 | −static struct shrinker nfs4_xattr_cache_shrinker = { | |
| 800 | − .count_objects = nfs4_xattr_cache_count, | |
| 801 | − .scan_objects = nfs4_xattr_cache_scan, | |
| 802 | − .seeks = DEFAULT_SEEKS, | |
| 803 | − .flags = SHRINKER_MEMCG_AWARE, | |
| 804 | −}; | |
| 805 | − | |
| 806 | −static struct shrinker nfs4_xattr_entry_shrinker = { | |
| 807 | − .count_objects = nfs4_xattr_entry_count, | |
| 808 | − .scan_objects = nfs4_xattr_entry_scan, | |
| 809 | − .seeks = DEFAULT_SEEKS, | |
| 810 | − .batch = 512, | |
| 811 | − .flags = SHRINKER_MEMCG_AWARE, | |
| 812 | −}; | |
| 813 | − | |
| 814 | −static struct shrinker nfs4_xattr_large_entry_shrinker = { | |
| 815 | − .count_objects = nfs4_xattr_entry_count, | |
| 816 | − .scan_objects = nfs4_xattr_entry_scan, | |
| 817 | − .seeks = 1, | |
| 818 | − .batch = 512, | |
| 819 | − .flags = SHRINKER_MEMCG_AWARE, | |
| 820 | −}; | |
| 799 | +static struct shrinker *nfs4_xattr_cache_shrinker; | |
| 800 | +static struct shrinker *nfs4_xattr_entry_shrinker; | |
| 801 | +static struct shrinker *nfs4_xattr_large_entry_shrinker; | |
| 821 | 802 | |
| 822 | 803 | static enum lru_status |
| 823 | 804 | cache_lru_isolate(struct list_head *item, |
| @@ -943,7 +924,7 @@ nfs4_xattr_entry_scan(struct shrinker *shrink, struct shrink_control *sc) · 119 unmodified lines | ||
| 943 | 924 | struct nfs4_xattr_entry *entry; |
| 944 | 925 | struct list_lru *lru; |
| 945 | 926 | |
| 946 | − lru = (shrink == &nfs4_xattr_large_entry_shrinker) ? | |
| 927 | + lru = (shrink == nfs4_xattr_large_entry_shrinker) ? | |
| 947 | 928 | &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru; |
| 948 | 929 | |
| 949 | 930 | freed = list_lru_shrink_walk(lru, sc, entry_lru_isolate, &dispose); |
| @@ -971,7 +952,7 @@ nfs4_xattr_entry_count(struct shrinker *shrink, struct shrink_control *sc) · 21 unmodified lines | ||
| 971 | 952 | unsigned long count; |
| 972 | 953 | struct list_lru *lru; |
| 973 | 954 | |
| 974 | − lru = (shrink == &nfs4_xattr_large_entry_shrinker) ? | |
| 955 | + lru = (shrink == nfs4_xattr_large_entry_shrinker) ? | |
| 975 | 956 | &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru; |
| 976 | 957 | |
| 977 | 958 | count = list_lru_shrink_count(lru, sc); |
| @@ -991,18 +972,34 @@ static void nfs4_xattr_cache_init_once(void *p) · 13 unmodified lines | ||
| 991 | 972 | INIT_LIST_HEAD(&cache->dispose); |
| 992 | 973 | } |
| 993 | 974 | |
| 994 | −static int nfs4_xattr_shrinker_init(struct shrinker *shrinker, | |
| 995 | − struct list_lru *lru, const char *name) | |
| 975 | +typedef unsigned long (*count_objects_cb)(struct shrinker *s, | |
| 976 | + struct shrink_control *sc); | |
| 977 | +typedef unsigned long (*scan_objects_cb)(struct shrinker *s, | |
| 978 | + struct shrink_control *sc); | |
| 979 | + | |
| 980 | +static int __init nfs4_xattr_shrinker_init(struct shrinker **shrinker, | |
| 981 | + struct list_lru *lru, const char *name, | |
| 982 | + count_objects_cb count, | |
| 983 | + scan_objects_cb scan, long batch, int seeks) | |
| 996 | 984 | { |
| 997 | − int ret = 0; | |
| 985 | + int ret; | |
| 998 | 986 | |
| 999 | − ret = register_shrinker(shrinker, name); | |
| 1000 | − if (ret) | |
| 987 | + *shrinker = shrinker_alloc(SHRINKER_MEMCG_AWARE, name); | |
| 988 | + if (!*shrinker) | |
| 989 | + return -ENOMEM; | |
| 990 | + | |
| 991 | + ret = list_lru_init_memcg(lru, *shrinker); | |
| 992 | + if (ret) { | |
| 993 | + shrinker_free(*shrinker); | |
| 1001 | 994 | return ret; |
| 995 | + } | |
| 1002 | 996 | |
| 1003 | − ret = list_lru_init_memcg(lru, shrinker); | |
| 1004 | − if (ret) | |
| 1005 | − unregister_shrinker(shrinker); | |
| 997 | + (*shrinker)->count_objects = count; | |
| 998 | + (*shrinker)->scan_objects = scan; | |
| 999 | + (*shrinker)->batch = batch; | |
| 1000 | + (*shrinker)->seeks = seeks; | |
| 1001 | + | |
| 1002 | + shrinker_register(*shrinker); | |
| 1006 | 1003 | |
| 1007 | 1004 | return ret; |
| 1008 | 1005 | } |
| @@ -1010,7 +1007,7 @@ static int nfs4_xattr_shrinker_init(struct shrinker *shrinker, · 1 unmodified line | ||
| 1010 | 1007 | static void nfs4_xattr_shrinker_destroy(struct shrinker *shrinker, |
| 1011 | 1008 | struct list_lru *lru) |
| 1012 | 1009 | { |
| 1013 | − unregister_shrinker(shrinker); | |
| 1010 | + shrinker_free(shrinker); | |
| 1014 | 1011 | list_lru_destroy(lru); |
| 1015 | 1012 | } |
| 1016 | 1013 | |
| @@ -1026,27 +1023,31 @@ int __init nfs4_xattr_cache_init(void) · 9 unmodified lines | ||
| 1026 | 1023 | return -ENOMEM; |
| 1027 | 1024 | |
| 1028 | 1025 | ret = nfs4_xattr_shrinker_init(&nfs4_xattr_cache_shrinker, |
| 1029 | − &nfs4_xattr_cache_lru, | |
| 1030 | − "nfs-xattr_cache"); | |
| 1026 | + &nfs4_xattr_cache_lru, "nfs-xattr_cache", | |
| 1027 | + nfs4_xattr_cache_count, | |
| 1028 | + nfs4_xattr_cache_scan, 0, DEFAULT_SEEKS); | |
| 1031 | 1029 | if (ret) |
| 1032 | 1030 | goto out1; |
| 1033 | 1031 | |
| 1034 | 1032 | ret = nfs4_xattr_shrinker_init(&nfs4_xattr_entry_shrinker, |
| 1035 | − &nfs4_xattr_entry_lru, | |
| 1036 | − "nfs-xattr_entry"); | |
| 1033 | + &nfs4_xattr_entry_lru, "nfs-xattr_entry", | |
| 1034 | + nfs4_xattr_entry_count, | |
| 1035 | + nfs4_xattr_entry_scan, 512, DEFAULT_SEEKS); | |
| 1037 | 1036 | if (ret) |
| 1038 | 1037 | goto out2; |
| 1039 | 1038 | |
| 1040 | 1039 | ret = nfs4_xattr_shrinker_init(&nfs4_xattr_large_entry_shrinker, |
| 1041 | 1040 | &nfs4_xattr_large_entry_lru, |
| 1042 | − "nfs-xattr_large_entry"); | |
| 1041 | + "nfs-xattr_large_entry", | |
| 1042 | + nfs4_xattr_entry_count, | |
| 1043 | + nfs4_xattr_entry_scan, 512, 1); | |
| 1043 | 1044 | if (!ret) |
| 1044 | 1045 | return 0; |
| 1045 | 1046 | |
| 1046 | − nfs4_xattr_shrinker_destroy(&nfs4_xattr_entry_shrinker, | |
| 1047 | + nfs4_xattr_shrinker_destroy(nfs4_xattr_entry_shrinker, | |
| 1047 | 1048 | &nfs4_xattr_entry_lru); |
| 1048 | 1049 | out2: |
| 1049 | − nfs4_xattr_shrinker_destroy(&nfs4_xattr_cache_shrinker, | |
| 1050 | + nfs4_xattr_shrinker_destroy(nfs4_xattr_cache_shrinker, | |
| 1050 | 1051 | &nfs4_xattr_cache_lru); |
| 1051 | 1052 | out1: |
| 1052 | 1053 | kmem_cache_destroy(nfs4_xattr_cache_cachep); |
| @@ -1056,11 +1057,11 @@ out1: · 3 unmodified lines | ||
| 1056 | 1057 | |
| 1057 | 1058 | void nfs4_xattr_cache_exit(void) |
| 1058 | 1059 | { |
| 1059 | − nfs4_xattr_shrinker_destroy(&nfs4_xattr_large_entry_shrinker, | |
| 1060 | + nfs4_xattr_shrinker_destroy(nfs4_xattr_large_entry_shrinker, | |
| 1060 | 1061 | &nfs4_xattr_large_entry_lru); |
| 1061 | − nfs4_xattr_shrinker_destroy(&nfs4_xattr_entry_shrinker, | |
| 1062 | + nfs4_xattr_shrinker_destroy(nfs4_xattr_entry_shrinker, | |
| 1062 | 1063 | &nfs4_xattr_entry_lru); |
| 1063 | − nfs4_xattr_shrinker_destroy(&nfs4_xattr_cache_shrinker, | |
| 1064 | + nfs4_xattr_shrinker_destroy(nfs4_xattr_cache_shrinker, | |
| 1064 | 1065 | &nfs4_xattr_cache_lru); |
| 1065 | 1066 | kmem_cache_destroy(nfs4_xattr_cache_cachep); |
| 1066 | 1067 | } |
| @@ -129,11 +129,7 @@ static void nfs_ssc_unregister_ops(void) · 128 unmodified lines | ||
| 129 | 129 | } |
| 130 | 130 | #endif /* CONFIG_NFS_V4_2 */ |
| 131 | 131 | |
| 132 | −static struct shrinker acl_shrinker = { | |
| 133 | − .count_objects = nfs_access_cache_count, | |
| 134 | − .scan_objects = nfs_access_cache_scan, | |
| 135 | − .seeks = DEFAULT_SEEKS, | |
| 136 | −}; | |
| 132 | +static struct shrinker *acl_shrinker; | |
| 137 | 133 | |
| 138 | 134 | /* |
| 139 | 135 | * Register the NFS filesystems |
| @@ -153,9 +149,18 @@ int __init register_nfs_fs(void) · 13 unmodified lines | ||
| 153 | 149 | ret = nfs_register_sysctl(); |
| 154 | 150 | if (ret < 0) |
| 155 | 151 | goto error_2; |
| 156 | − ret = register_shrinker(&acl_shrinker, "nfs-acl"); | |
| 157 | − if (ret < 0) | |
| 152 | + | |
| 153 | + acl_shrinker = shrinker_alloc(0, "nfs-acl"); | |
| 154 | + if (!acl_shrinker) { | |
| 155 | + ret = -ENOMEM; | |
| 158 | 156 | goto error_3; |
| 157 | + } | |
| 158 | + | |
| 159 | + acl_shrinker->count_objects = nfs_access_cache_count; | |
| 160 | + acl_shrinker->scan_objects = nfs_access_cache_scan; | |
| 161 | + | |
| 162 | + shrinker_register(acl_shrinker); | |
| 163 | + | |
| 159 | 164 | #ifdef CONFIG_NFS_V4_2 |
| 160 | 165 | nfs_ssc_register_ops(); |
| 161 | 166 | #endif |
| @@ -175,7 +180,7 @@ error_0: · 13 unmodified lines | ||
| 175 | 180 | */ |
| 176 | 181 | void __exit unregister_nfs_fs(void) |
| 177 | 182 | { |
| 178 | − unregister_shrinker(&acl_shrinker); | |
| 183 | + shrinker_free(acl_shrinker); | |
| 179 | 184 | nfs_unregister_sysctl(); |
| 180 | 185 | unregister_nfs4_fs(); |
| 181 | 186 | #ifdef CONFIG_NFS_V4_2 |
| @@ -521,11 +521,7 @@ nfsd_file_lru_scan(struct shrinker *s, struct shrink_control *sc) · 520 unmodified lines | ||
| 521 | 521 | return ret; |
| 522 | 522 | } |
| 523 | 523 | |
| 524 | −static struct shrinker nfsd_file_shrinker = { | |
| 525 | − .scan_objects = nfsd_file_lru_scan, | |
| 526 | − .count_objects = nfsd_file_lru_count, | |
| 527 | − .seeks = 1, | |
| 528 | −}; | |
| 524 | +static struct shrinker *nfsd_file_shrinker; | |
| 529 | 525 | |
| 530 | 526 | /** |
| 531 | 527 | * nfsd_file_cond_queue - conditionally unhash and queue a nfsd_file |
| @@ -746,12 +742,19 @@ nfsd_file_cache_init(void) · 214 unmodified lines | ||
| 746 | 742 | goto out_err; |
| 747 | 743 | } |
| 748 | 744 | |
| 749 | − ret = register_shrinker(&nfsd_file_shrinker, "nfsd-filecache"); | |
| 750 | − if (ret) { | |
| 751 | − pr_err("nfsd: failed to register nfsd_file_shrinker: %d\n", ret); | |
| 745 | + nfsd_file_shrinker = shrinker_alloc(0, "nfsd-filecache"); | |
| 746 | + if (!nfsd_file_shrinker) { | |
| 747 | + ret = -ENOMEM; | |
| 748 | + pr_err("nfsd: failed to allocate nfsd_file_shrinker\n"); | |
| 752 | 749 | goto out_lru; |
| 753 | 750 | } |
| 754 | 751 | |
| 752 | + nfsd_file_shrinker->count_objects = nfsd_file_lru_count; | |
| 753 | + nfsd_file_shrinker->scan_objects = nfsd_file_lru_scan; | |
| 754 | + nfsd_file_shrinker->seeks = 1; | |
| 755 | + | |
| 756 | + shrinker_register(nfsd_file_shrinker); | |
| 757 | + | |
| 755 | 758 | ret = lease_register_notifier(&nfsd_file_lease_notifier); |
| 756 | 759 | if (ret) { |
| 757 | 760 | pr_err("nfsd: unable to register lease notifier: %d\n", ret); |
| @@ -774,7 +777,7 @@ out: · 16 unmodified lines | ||
| 774 | 777 | out_notifier: |
| 775 | 778 | lease_unregister_notifier(&nfsd_file_lease_notifier); |
| 776 | 779 | out_shrinker: |
| 777 | − unregister_shrinker(&nfsd_file_shrinker); | |
| 780 | + shrinker_free(nfsd_file_shrinker); | |
| 778 | 781 | out_lru: |
| 779 | 782 | list_lru_destroy(&nfsd_file_lru); |
| 780 | 783 | out_err: |
| @@ -891,7 +894,7 @@ nfsd_file_cache_shutdown(void) · 110 unmodified lines | ||
| 891 | 894 | return; |
| 892 | 895 | |
| 893 | 896 | lease_unregister_notifier(&nfsd_file_lease_notifier); |
| 894 | − unregister_shrinker(&nfsd_file_shrinker); | |
| 897 | + shrinker_free(nfsd_file_shrinker); | |
| 895 | 898 | /* |
| 896 | 899 | * make sure all callers of nfsd_file_lru_cb are done before |
| 897 | 900 | * calling nfsd_file_cache_purge |
| @@ -177,7 +177,7 @@ struct nfsd_net { · 176 unmodified lines | ||
| 177 | 177 | /* size of cache when we saw the longest hash chain */ |
| 178 | 178 | unsigned int longest_chain_cachesize; |
| 179 | 179 | |
| 180 | − struct shrinker nfsd_reply_cache_shrinker; | |
| 180 | + struct shrinker *nfsd_reply_cache_shrinker; | |
| 181 | 181 | |
| 182 | 182 | /* tracking server-to-server copy mounts */ |
| 183 | 183 | spinlock_t nfsd_ssc_lock; |
| @@ -195,7 +195,7 @@ struct nfsd_net { · 11 unmodified lines | ||
| 195 | 195 | int nfs4_max_clients; |
| 196 | 196 | |
| 197 | 197 | atomic_t nfsd_courtesy_clients; |
| 198 | − struct shrinker nfsd_client_shrinker; | |
| 198 | + struct shrinker *nfsd_client_shrinker; | |
| 199 | 199 | struct work_struct nfsd_shrinker_work; |
| 200 | 200 | }; |
| 201 | 201 | |
| @@ -4452,8 +4452,7 @@ static unsigned long · 4451 unmodified lines | ||
| 4452 | 4452 | nfsd4_state_shrinker_count(struct shrinker *shrink, struct shrink_control *sc) |
| 4453 | 4453 | { |
| 4454 | 4454 | int count; |
| 4455 | − struct nfsd_net *nn = container_of(shrink, | |
| 4456 | − struct nfsd_net, nfsd_client_shrinker); | |
| 4455 | + struct nfsd_net *nn = shrink->private_data; | |
| 4457 | 4456 | |
| 4458 | 4457 | count = atomic_read(&nn->nfsd_courtesy_clients); |
| 4459 | 4458 | if (!count) |
| @@ -8235,12 +8234,16 @@ static int nfs4_state_create_net(struct net *net) · 3775 unmodified lines | ||
| 8235 | 8234 | INIT_WORK(&nn->nfsd_shrinker_work, nfsd4_state_shrinker_worker); |
| 8236 | 8235 | get_net(net); |
| 8237 | 8236 | |
| 8238 | − nn->nfsd_client_shrinker.scan_objects = nfsd4_state_shrinker_scan; | |
| 8239 | − nn->nfsd_client_shrinker.count_objects = nfsd4_state_shrinker_count; | |
| 8240 | − nn->nfsd_client_shrinker.seeks = DEFAULT_SEEKS; | |
| 8241 | − | |
| 8242 | − if (register_shrinker(&nn->nfsd_client_shrinker, "nfsd-client")) | |
| 8237 | + nn->nfsd_client_shrinker = shrinker_alloc(0, "nfsd-client"); | |
| 8238 | + if (!nn->nfsd_client_shrinker) | |
| 8243 | 8239 | goto err_shrinker; |
| 8240 | + | |
| 8241 | + nn->nfsd_client_shrinker->scan_objects = nfsd4_state_shrinker_scan; | |
| 8242 | + nn->nfsd_client_shrinker->count_objects = nfsd4_state_shrinker_count; | |
| 8243 | + nn->nfsd_client_shrinker->private_data = nn; | |
| 8244 | + | |
| 8245 | + shrinker_register(nn->nfsd_client_shrinker); | |
| 8246 | + | |
| 8244 | 8247 | return 0; |
| 8245 | 8248 | |
| 8246 | 8249 | err_shrinker: |
| @@ -8338,7 +8341,7 @@ nfs4_state_shutdown_net(struct net *net) · 91 unmodified lines | ||
| 8338 | 8341 | struct list_head *pos, *next, reaplist; |
| 8339 | 8342 | struct nfsd_net *nn = net_generic(net, nfsd_net_id); |
| 8340 | 8343 | |
| 8341 | − unregister_shrinker(&nn->nfsd_client_shrinker); | |
| 8344 | + shrinker_free(nn->nfsd_client_shrinker); | |
| 8342 | 8345 | cancel_work(&nn->nfsd_shrinker_work); |
| 8343 | 8346 | cancel_delayed_work_sync(&nn->laundromat_work); |
| 8344 | 8347 | locks_end_grace(&nn->nfsd4_manager); |
| @@ -201,26 +201,29 @@ int nfsd_reply_cache_init(struct nfsd_net *nn) · 200 unmodified lines | ||
| 201 | 201 | { |
| 202 | 202 | unsigned int hashsize; |
| 203 | 203 | unsigned int i; |
| 204 | − int status = 0; | |
| 205 | 204 | |
| 206 | 205 | nn->max_drc_entries = nfsd_cache_size_limit(); |
| 207 | 206 | atomic_set(&nn->num_drc_entries, 0); |
| 208 | 207 | hashsize = nfsd_hashsize(nn->max_drc_entries); |
| 209 | 208 | nn->maskbits = ilog2(hashsize); |
| 210 | 209 | |
| 211 | − nn->nfsd_reply_cache_shrinker.scan_objects = nfsd_reply_cache_scan; | |
| 212 | − nn->nfsd_reply_cache_shrinker.count_objects = nfsd_reply_cache_count; | |
| 213 | − nn->nfsd_reply_cache_shrinker.seeks = 1; | |
| 214 | − status = register_shrinker(&nn->nfsd_reply_cache_shrinker, | |
| 215 | − "nfsd-reply:%s", nn->nfsd_name); | |
| 216 | − if (status) | |
| 217 | − return status; | |
| 218 | − | |
| 219 | 210 | nn->drc_hashtbl = kvzalloc(array_size(hashsize, |
| 220 | 211 | sizeof(*nn->drc_hashtbl)), GFP_KERNEL); |
| 221 | 212 | if (!nn->drc_hashtbl) |
| 213 | + return -ENOMEM; | |
| 214 | + | |
| 215 | + nn->nfsd_reply_cache_shrinker = shrinker_alloc(0, "nfsd-reply:%s", | |
| 216 | + nn->nfsd_name); | |
| 217 | + if (!nn->nfsd_reply_cache_shrinker) | |
| 222 | 218 | goto out_shrinker; |
| 223 | 219 | |
| 220 | + nn->nfsd_reply_cache_shrinker->scan_objects = nfsd_reply_cache_scan; | |
| 221 | + nn->nfsd_reply_cache_shrinker->count_objects = nfsd_reply_cache_count; | |
| 222 | + nn->nfsd_reply_cache_shrinker->seeks = 1; | |
| 223 | + nn->nfsd_reply_cache_shrinker->private_data = nn; | |
| 224 | + | |
| 225 | + shrinker_register(nn->nfsd_reply_cache_shrinker); | |
| 226 | + | |
| 224 | 227 | for (i = 0; i < hashsize; i++) { |
| 225 | 228 | INIT_LIST_HEAD(&nn->drc_hashtbl[i].lru_head); |
| 226 | 229 | spin_lock_init(&nn->drc_hashtbl[i].cache_lock); |
| @@ -229,7 +232,7 @@ int nfsd_reply_cache_init(struct nfsd_net *nn) · 2 unmodified lines | ||
| 229 | 232 | |
| 230 | 233 | return 0; |
| 231 | 234 | out_shrinker: |
| 232 | − unregister_shrinker(&nn->nfsd_reply_cache_shrinker); | |
| 235 | + kvfree(nn->drc_hashtbl); | |
| 233 | 236 | printk(KERN_ERR "nfsd: failed to allocate reply cache\n"); |
| 234 | 237 | return -ENOMEM; |
| 235 | 238 | } |
| @@ -239,7 +242,7 @@ void nfsd_reply_cache_shutdown(struct nfsd_net *nn) · 3 unmodified lines | ||
| 239 | 242 | struct nfsd_cacherep *rp; |
| 240 | 243 | unsigned int i; |
| 241 | 244 | |
| 242 | − unregister_shrinker(&nn->nfsd_reply_cache_shrinker); | |
| 245 | + shrinker_free(nn->nfsd_reply_cache_shrinker); | |
| 243 | 246 | |
| 244 | 247 | for (i = 0; i < nn->drc_hashsize; i++) { |
| 245 | 248 | struct list_head *head = &nn->drc_hashtbl[i].lru_head; |
| @@ -323,8 +326,7 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b, · 77 unmodified lines | ||
| 323 | 326 | static unsigned long |
| 324 | 327 | nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc) |
| 325 | 328 | { |
| 326 | − struct nfsd_net *nn = container_of(shrink, | |
| 327 | − struct nfsd_net, nfsd_reply_cache_shrinker); | |
| 329 | + struct nfsd_net *nn = shrink->private_data; | |
| 328 | 330 | |
| 329 | 331 | return atomic_read(&nn->num_drc_entries); |
| 330 | 332 | } |
| @@ -343,8 +345,7 @@ nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc) · 12 unmodified lines | ||
| 343 | 345 | static unsigned long |
| 344 | 346 | nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc) |
| 345 | 347 | { |
| 346 | − struct nfsd_net *nn = container_of(shrink, | |
| 347 | − struct nfsd_net, nfsd_reply_cache_shrinker); | |
| 348 | + struct nfsd_net *nn = shrink->private_data; | |
| 348 | 349 | unsigned long freed = 0; |
| 349 | 350 | LIST_HEAD(dispose); |
| 350 | 351 | unsigned int i; |
| @@ -356,30 +356,28 @@ int nilfs_mdt_delete_block(struct inode *inode, unsigned long block) · 355 unmodified lines | ||
| 356 | 356 | */ |
| 357 | 357 | int nilfs_mdt_forget_block(struct inode *inode, unsigned long block) |
| 358 | 358 | { |
| 359 | − pgoff_t index = (pgoff_t)block >> | |
| 360 | − (PAGE_SHIFT - inode->i_blkbits); | |
| 361 | − struct page *page; | |
| 362 | − unsigned long first_block; | |
| 359 | + pgoff_t index = block >> (PAGE_SHIFT - inode->i_blkbits); | |
| 360 | + struct folio *folio; | |
| 361 | + struct buffer_head *bh; | |
| 363 | 362 | int ret = 0; |
| 364 | 363 | int still_dirty; |
| 365 | 364 | |
| 366 | − page = find_lock_page(inode->i_mapping, index); | |
| 367 | − if (!page) | |
| 365 | + folio = filemap_lock_folio(inode->i_mapping, index); | |
| 366 | + if (IS_ERR(folio)) | |
| 368 | 367 | return -ENOENT; |
| 369 | 368 | |
| 370 | − wait_on_page_writeback(page); | |
| 371 | − | |
| 372 | − first_block = (unsigned long)index << | |
| 373 | − (PAGE_SHIFT - inode->i_blkbits); | |
| 374 | − if (page_has_buffers(page)) { | |
| 375 | − struct buffer_head *bh; | |
| 369 | + folio_wait_writeback(folio); | |
| 376 | 370 | |
| 377 | − bh = nilfs_page_get_nth_block(page, block - first_block); | |
| 371 | + bh = folio_buffers(folio); | |
| 372 | + if (bh) { | |
| 373 | + unsigned long first_block = index << | |
| 374 | + (PAGE_SHIFT - inode->i_blkbits); | |
| 375 | + bh = get_nth_bh(bh, block - first_block); | |
| 378 | 376 | nilfs_forget_buffer(bh); |
| 379 | 377 | } |
| 380 | − still_dirty = PageDirty(page); | |
| 381 | − unlock_page(page); | |
| 382 | − put_page(page); | |
| 378 | + still_dirty = folio_test_dirty(folio); | |
| 379 | + folio_unlock(folio); | |
| 380 | + folio_put(folio); | |
| 383 | 381 | |
| 384 | 382 | if (still_dirty || |
| 385 | 383 | invalidate_inode_pages2_range(inode->i_mapping, index, index) != 0) |
| @@ -560,17 +558,19 @@ int nilfs_mdt_freeze_buffer(struct inode *inode, struct buffer_head *bh) · 174 unmodified lines | ||
| 560 | 558 | { |
| 561 | 559 | struct nilfs_shadow_map *shadow = NILFS_MDT(inode)->mi_shadow; |
| 562 | 560 | struct buffer_head *bh_frozen; |
| 563 | − struct page *page; | |
| 561 | + struct folio *folio; | |
| 564 | 562 | int blkbits = inode->i_blkbits; |
| 565 | 563 | |
| 566 | − page = grab_cache_page(shadow->inode->i_mapping, bh->b_folio->index); | |
| 567 | − if (!page) | |
| 568 | − return -ENOMEM; | |
| 564 | + folio = filemap_grab_folio(shadow->inode->i_mapping, | |
| 565 | + bh->b_folio->index); | |
| 566 | + if (IS_ERR(folio)) | |
| 567 | + return PTR_ERR(folio); | |
| 569 | 568 | |
| 570 | − if (!page_has_buffers(page)) | |
| 571 | − create_empty_buffers(page, 1 << blkbits, 0); | |
| 569 | + bh_frozen = folio_buffers(folio); | |
| 570 | + if (!bh_frozen) | |
| 571 | + bh_frozen = create_empty_buffers(folio, 1 << blkbits, 0); | |
| 572 | 572 | |
| 573 | − bh_frozen = nilfs_page_get_nth_block(page, bh_offset(bh) >> blkbits); | |
| 573 | + bh_frozen = get_nth_bh(bh_frozen, bh_offset(bh) >> blkbits); | |
| 574 | 574 | |
| 575 | 575 | if (!buffer_uptodate(bh_frozen)) |
| 576 | 576 | nilfs_copy_buffer(bh_frozen, bh); |
| @@ -582,8 +582,8 @@ int nilfs_mdt_freeze_buffer(struct inode *inode, struct buffer_head *bh) · 5 unmodified lines | ||
| 582 | 582 | brelse(bh_frozen); /* already frozen */ |
| 583 | 583 | } |
| 584 | 584 | |
| 585 | − unlock_page(page); | |
| 586 | − put_page(page); | |
| 585 | + folio_unlock(folio); | |
| 586 | + folio_put(folio); | |
| 587 | 587 | return 0; |
| 588 | 588 | } |
| 589 | 589 | |
| @@ -592,17 +592,19 @@ nilfs_mdt_get_frozen_buffer(struct inode *inode, struct buffer_head *bh) · 2 unmodified lines | ||
| 592 | 592 | { |
| 593 | 593 | struct nilfs_shadow_map *shadow = NILFS_MDT(inode)->mi_shadow; |
| 594 | 594 | struct buffer_head *bh_frozen = NULL; |
| 595 | − struct page *page; | |
| 595 | + struct folio *folio; | |
| 596 | 596 | int n; |
| 597 | 597 | |
| 598 | − page = find_lock_page(shadow->inode->i_mapping, bh->b_folio->index); | |
| 599 | − if (page) { | |
| 600 | − if (page_has_buffers(page)) { | |
| 598 | + folio = filemap_lock_folio(shadow->inode->i_mapping, | |
| 599 | + bh->b_folio->index); | |
| 600 | + if (!IS_ERR(folio)) { | |
| 601 | + bh_frozen = folio_buffers(folio); | |
| 602 | + if (bh_frozen) { | |
| 601 | 603 | n = bh_offset(bh) >> inode->i_blkbits; |
| 602 | − bh_frozen = nilfs_page_get_nth_block(page, n); | |
| 604 | + bh_frozen = get_nth_bh(bh_frozen, n); | |
| 603 | 605 | } |
| 604 | − unlock_page(page); | |
| 605 | − put_page(page); | |
| 606 | + folio_unlock(folio); | |
| 607 | + folio_put(folio); | |
| 606 | 608 | } |
| 607 | 609 | return bh_frozen; |
| 608 | 610 | } |
| @@ -25,19 +25,19 @@ · 24 unmodified lines | ||
| 25 | 25 | (BIT(BH_Uptodate) | BIT(BH_Mapped) | BIT(BH_NILFS_Node) | \ |
| 26 | 26 | BIT(BH_NILFS_Volatile) | BIT(BH_NILFS_Checked)) |
| 27 | 27 | |
| 28 | −static struct buffer_head * | |
| 29 | −__nilfs_get_page_block(struct page *page, unsigned long block, pgoff_t index, | |
| 30 | − int blkbits, unsigned long b_state) | |
| 28 | +static struct buffer_head *__nilfs_get_folio_block(struct folio *folio, | |
| 29 | + unsigned long block, pgoff_t index, int blkbits, | |
| 30 | + unsigned long b_state) | |
| 31 | 31 | |
| 32 | 32 | { |
| 33 | 33 | unsigned long first_block; |
| 34 | − struct buffer_head *bh; | |
| 34 | + struct buffer_head *bh = folio_buffers(folio); | |
| 35 | 35 | |
| 36 | − if (!page_has_buffers(page)) | |
| 37 | − create_empty_buffers(page, 1 << blkbits, b_state); | |
| 36 | + if (!bh) | |
| 37 | + bh = create_empty_buffers(folio, 1 << blkbits, b_state); | |
| 38 | 38 | |
| 39 | 39 | first_block = (unsigned long)index << (PAGE_SHIFT - blkbits); |
| 40 | − bh = nilfs_page_get_nth_block(page, block - first_block); | |
| 40 | + bh = get_nth_bh(bh, block - first_block); | |
| 41 | 41 | |
| 42 | 42 | touch_buffer(bh); |
| 43 | 43 | wait_on_buffer(bh); |
| @@ -51,17 +51,17 @@ struct buffer_head *nilfs_grab_buffer(struct inode *inode, · 7 unmodified lines | ||
| 51 | 51 | { |
| 52 | 52 | int blkbits = inode->i_blkbits; |
| 53 | 53 | pgoff_t index = blkoff >> (PAGE_SHIFT - blkbits); |
| 54 | − struct page *page; | |
| 54 | + struct folio *folio; | |
| 55 | 55 | struct buffer_head *bh; |
| 56 | 56 | |
| 57 | − page = grab_cache_page(mapping, index); | |
| 58 | − if (unlikely(!page)) | |
| 57 | + folio = filemap_grab_folio(mapping, index); | |
| 58 | + if (IS_ERR(folio)) | |
| 59 | 59 | return NULL; |
| 60 | 60 | |
| 61 | − bh = __nilfs_get_page_block(page, blkoff, index, blkbits, b_state); | |
| 61 | + bh = __nilfs_get_folio_block(folio, blkoff, index, blkbits, b_state); | |
| 62 | 62 | if (unlikely(!bh)) { |
| 63 | − unlock_page(page); | |
| 64 | − put_page(page); | |
| 63 | + folio_unlock(folio); | |
| 64 | + folio_put(folio); | |
| 65 | 65 | return NULL; |
| 66 | 66 | } |
| 67 | 67 | return bh; |
| @@ -184,30 +184,32 @@ void nilfs_page_bug(struct page *page) · 116 unmodified lines | ||
| 184 | 184 | } |
| 185 | 185 | |
| 186 | 186 | /** |
| 187 | − * nilfs_copy_page -- copy the page with buffers | |
| 188 | − * @dst: destination page | |
| 189 | − * @src: source page | |
| 190 | − * @copy_dirty: flag whether to copy dirty states on the page's buffer heads. | |
| 187 | + * nilfs_copy_folio -- copy the folio with buffers | |
| 188 | + * @dst: destination folio | |
| 189 | + * @src: source folio | |
| 190 | + * @copy_dirty: flag whether to copy dirty states on the folio's buffer heads. | |
| 191 | 191 | * |
| 192 | − * This function is for both data pages and btnode pages. The dirty flag | |
| 193 | − * should be treated by caller. The page must not be under i/o. | |
| 194 | − * Both src and dst page must be locked | |
| 192 | + * This function is for both data folios and btnode folios. The dirty flag | |
| 193 | + * should be treated by caller. The folio must not be under i/o. | |
| 194 | + * Both src and dst folio must be locked | |
| 195 | 195 | */ |
| 196 | −static void nilfs_copy_page(struct page *dst, struct page *src, int copy_dirty) | |
| 196 | +static void nilfs_copy_folio(struct folio *dst, struct folio *src, | |
| 197 | + bool copy_dirty) | |
| 197 | 198 | { |
| 198 | 199 | struct buffer_head *dbh, *dbufs, *sbh; |
| 199 | 200 | unsigned long mask = NILFS_BUFFER_INHERENT_BITS; |
| 200 | 201 | |
| 201 | − BUG_ON(PageWriteback(dst)); | |
| 202 | + BUG_ON(folio_test_writeback(dst)); | |
| 202 | 203 | |
| 203 | − sbh = page_buffers(src); | |
| 204 | − if (!page_has_buffers(dst)) | |
| 205 | − create_empty_buffers(dst, sbh->b_size, 0); | |
| 204 | + sbh = folio_buffers(src); | |
| 205 | + dbh = folio_buffers(dst); | |
| 206 | + if (!dbh) | |
| 207 | + dbh = create_empty_buffers(dst, sbh->b_size, 0); | |
| 206 | 208 | |
| 207 | 209 | if (copy_dirty) |
| 208 | 210 | mask |= BIT(BH_Dirty); |
| 209 | 211 | |
| 210 | − dbh = dbufs = page_buffers(dst); | |
| 212 | + dbufs = dbh; | |
| 211 | 213 | do { |
| 212 | 214 | lock_buffer(sbh); |
| 213 | 215 | lock_buffer(dbh); |
| @@ -218,16 +220,16 @@ static void nilfs_copy_page(struct page *dst, struct page *src, int copy_dirty) · 4 unmodified lines | ||
| 218 | 220 | dbh = dbh->b_this_page; |
| 219 | 221 | } while (dbh != dbufs); |
| 220 | 222 | |
| 221 | − copy_highpage(dst, src); | |
| 223 | + folio_copy(dst, src); | |
| 222 | 224 | |
| 223 | − if (PageUptodate(src) && !PageUptodate(dst)) | |
| 224 | − SetPageUptodate(dst); | |
| 225 | − else if (!PageUptodate(src) && PageUptodate(dst)) | |
| 226 | − ClearPageUptodate(dst); | |
| 227 | − if (PageMappedToDisk(src) && !PageMappedToDisk(dst)) | |
| 228 | − SetPageMappedToDisk(dst); | |
| 229 | − else if (!PageMappedToDisk(src) && PageMappedToDisk(dst)) | |
| 230 | − ClearPageMappedToDisk(dst); | |
| 225 | + if (folio_test_uptodate(src) && !folio_test_uptodate(dst)) | |
| 226 | + folio_mark_uptodate(dst); | |
| 227 | + else if (!folio_test_uptodate(src) && folio_test_uptodate(dst)) | |
| 228 | + folio_clear_uptodate(dst); | |
| 229 | + if (folio_test_mappedtodisk(src) && !folio_test_mappedtodisk(dst)) | |
| 230 | + folio_set_mappedtodisk(dst); | |
| 231 | + else if (!folio_test_mappedtodisk(src) && folio_test_mappedtodisk(dst)) | |
| 232 | + folio_clear_mappedtodisk(dst); | |
| 231 | 233 | |
| 232 | 234 | do { |
| 233 | 235 | unlock_buffer(sbh); |
| @@ -269,7 +271,7 @@ repeat: · 35 unmodified lines | ||
| 269 | 271 | NILFS_PAGE_BUG(&folio->page, |
| 270 | 272 | "found empty page in dat page cache"); |
| 271 | 273 | |
| 272 | − nilfs_copy_page(&dfolio->page, &folio->page, 1); | |
| 274 | + nilfs_copy_folio(dfolio, folio, true); | |
| 273 | 275 | filemap_dirty_folio(folio_mapping(dfolio), dfolio); |
| 274 | 276 | |
| 275 | 277 | folio_unlock(dfolio); |
| @@ -314,7 +316,7 @@ repeat: · 38 unmodified lines | ||
| 314 | 316 | if (!IS_ERR(dfolio)) { |
| 315 | 317 | /* overwrite existing folio in the destination cache */ |
| 316 | 318 | WARN_ON(folio_test_dirty(dfolio)); |
| 317 | − nilfs_copy_page(&dfolio->page, &folio->page, 0); | |
| 319 | + nilfs_copy_folio(dfolio, folio, false); | |
| 318 | 320 | folio_unlock(dfolio); |
| 319 | 321 | folio_put(dfolio); |
| 320 | 322 | /* Do we not need to remove folio from smap here? */ |
| @@ -52,15 +52,4 @@ unsigned long nilfs_find_uncommitted_extent(struct inode *inode, · 51 unmodified lines | ||
| 52 | 52 | #define NILFS_PAGE_BUG(page, m, a...) \ |
| 53 | 53 | do { nilfs_page_bug(page); BUG(); } while (0) |
| 54 | 54 | |
| 55 | −static inline struct buffer_head * | |
| 56 | −nilfs_page_get_nth_block(struct page *page, unsigned int count) | |
| 57 | −{ | |
| 58 | − struct buffer_head *bh = page_buffers(page); | |
| 59 | − | |
| 60 | − while (count-- > 0) | |
| 61 | − bh = bh->b_this_page; | |
| 62 | − get_bh(bh); | |
| 63 | − return bh; | |
| 64 | −} | |
| 65 | − | |
| 66 | 55 | #endif /* _NILFS_PAGE_H */ |
| @@ -731,10 +731,9 @@ static size_t nilfs_lookup_dirty_data_buffers(struct inode *inode, · 730 unmodified lines | ||
| 731 | 731 | continue; |
| 732 | 732 | } |
| 733 | 733 | head = folio_buffers(folio); |
| 734 | − if (!head) { | |
| 735 | − create_empty_buffers(&folio->page, i_blocksize(inode), 0); | |
| 736 | − head = folio_buffers(folio); | |
| 737 | − } | |
| 734 | + if (!head) | |
| 735 | + head = create_empty_buffers(folio, | |
| 736 | + i_blocksize(inode), 0); | |
| 738 | 737 | folio_unlock(folio); |
| 739 | 738 | |
| 740 | 739 | bh = head; |
| @@ -145,13 +145,12 @@ still_busy: · 144 unmodified lines | ||
| 145 | 145 | } |
| 146 | 146 | |
| 147 | 147 | /** |
| 148 | − * ntfs_read_block - fill a @page of an address space with data | |
| 149 | − * @page: page cache page to fill with data | |
| 148 | + * ntfs_read_block - fill a @folio of an address space with data | |
| 149 | + * @folio: page cache folio to fill with data | |
| 150 | 150 | * |
| 151 | − * Fill the page @page of the address space belonging to the @page->host inode. | |
| 152 | 151 | * We read each buffer asynchronously and when all buffers are read in, our io |
| 153 | 152 | * completion handler ntfs_end_buffer_read_async(), if required, automatically |
| 154 | − * applies the mst fixups to the page before finally marking it uptodate and | |
| 153 | + * applies the mst fixups to the folio before finally marking it uptodate and | |
| 155 | 154 | * unlocking it. |
| 156 | 155 | * |
| 157 | 156 | * We only enforce allocated_size limit because i_size is checked for in |
| @@ -161,7 +160,7 @@ still_busy: · 3 unmodified lines | ||
| 161 | 160 | * |
| 162 | 161 | * Contains an adapted version of fs/buffer.c::block_read_full_folio(). |
| 163 | 162 | */ |
| 164 | −static int ntfs_read_block(struct page *page) | |
| 163 | +static int ntfs_read_block(struct folio *folio) | |
| 165 | 164 | { |
| 166 | 165 | loff_t i_size; |
| 167 | 166 | VCN vcn; |
| @@ -178,7 +177,7 @@ static int ntfs_read_block(struct page *page) · 10 unmodified lines | ||
| 178 | 177 | int i, nr; |
| 179 | 178 | unsigned char blocksize_bits; |
| 180 | 179 | |
| 181 | − vi = page->mapping->host; | |
| 180 | + vi = folio->mapping->host; | |
| 182 | 181 | ni = NTFS_I(vi); |
| 183 | 182 | vol = ni->vol; |
| 184 | 183 | |
| @@ -188,15 +187,10 @@ static int ntfs_read_block(struct page *page) · 3 unmodified lines | ||
| 188 | 187 | blocksize = vol->sb->s_blocksize; |
| 189 | 188 | blocksize_bits = vol->sb->s_blocksize_bits; |
| 190 | 189 | |
| 191 | − if (!page_has_buffers(page)) { | |
| 192 | − create_empty_buffers(page, blocksize, 0); | |
| 193 | − if (unlikely(!page_has_buffers(page))) { | |
| 194 | − unlock_page(page); | |
| 195 | − return -ENOMEM; | |
| 196 | − } | |
| 197 | − } | |
| 198 | − bh = head = page_buffers(page); | |
| 199 | − BUG_ON(!bh); | |
| 190 | + head = folio_buffers(folio); | |
| 191 | + if (!head) | |
| 192 | + head = create_empty_buffers(folio, blocksize, 0); | |
| 193 | + bh = head; | |
| 200 | 194 | |
| 201 | 195 | /* |
| 202 | 196 | * We may be racing with truncate. To avoid some of the problems we |
| @@ -205,11 +199,11 @@ static int ntfs_read_block(struct page *page) · 2 unmodified lines | ||
| 205 | 199 | * may leave some buffers unmapped which are now allocated. This is |
| 206 | 200 | * not a problem since these buffers will just get mapped when a write |
| 207 | 201 | * occurs. In case of a shrinking truncate, we will detect this later |
| 208 | − * on due to the runlist being incomplete and if the page is being | |
| 202 | + * on due to the runlist being incomplete and if the folio is being | |
| 209 | 203 | * fully truncated, truncate will throw it away as soon as we unlock |
| 210 | 204 | * it so no need to worry what we do with it. |
| 211 | 205 | */ |
| 212 | − iblock = (s64)page->index << (PAGE_SHIFT - blocksize_bits); | |
| 206 | + iblock = (s64)folio->index << (PAGE_SHIFT - blocksize_bits); | |
| 213 | 207 | read_lock_irqsave(&ni->size_lock, flags); |
| 214 | 208 | lblock = (ni->allocated_size + blocksize - 1) >> blocksize_bits; |
| 215 | 209 | init_size = ni->initialized_size; |
| @@ -221,7 +215,7 @@ static int ntfs_read_block(struct page *page) · 5 unmodified lines | ||
| 221 | 215 | } |
| 222 | 216 | zblock = (init_size + blocksize - 1) >> blocksize_bits; |
| 223 | 217 | |
| 224 | − /* Loop through all the buffers in the page. */ | |
| 218 | + /* Loop through all the buffers in the folio. */ | |
| 225 | 219 | rl = NULL; |
| 226 | 220 | nr = i = 0; |
| 227 | 221 | do { |
| @@ -299,7 +293,7 @@ lock_retry_remap: · 71 unmodified lines | ||
| 299 | 293 | if (!err) |
| 300 | 294 | err = -EIO; |
| 301 | 295 | bh->b_blocknr = -1; |
| 302 | − SetPageError(page); | |
| 296 | + folio_set_error(folio); | |
| 303 | 297 | ntfs_error(vol->sb, "Failed to read from inode 0x%lx, " |
| 304 | 298 | "attribute type 0x%x, vcn 0x%llx, " |
| 305 | 299 | "offset 0x%x because its location on " |
| @@ -312,13 +306,13 @@ lock_retry_remap: · 6 unmodified lines | ||
| 312 | 306 | /* |
| 313 | 307 | * Either iblock was outside lblock limits or |
| 314 | 308 | * ntfs_rl_vcn_to_lcn() returned error. Just zero that portion |
| 315 | − * of the page and set the buffer uptodate. | |
| 309 | + * of the folio and set the buffer uptodate. | |
| 316 | 310 | */ |
| 317 | 311 | handle_hole: |
| 318 | 312 | bh->b_blocknr = -1UL; |
| 319 | 313 | clear_buffer_mapped(bh); |
| 320 | 314 | handle_zblock: |
| 321 | − zero_user(page, i * blocksize, blocksize); | |
| 315 | + folio_zero_range(folio, i * blocksize, blocksize); | |
| 322 | 316 | if (likely(!err)) |
| 323 | 317 | set_buffer_uptodate(bh); |
| 324 | 318 | } while (i++, iblock++, (bh = bh->b_this_page) != head); |
| @@ -349,11 +343,11 @@ handle_zblock: · 24 unmodified lines | ||
| 349 | 343 | return 0; |
| 350 | 344 | } |
| 351 | 345 | /* No i/o was scheduled on any of the buffers. */ |
| 352 | − if (likely(!PageError(page))) | |
| 353 | − SetPageUptodate(page); | |
| 346 | + if (likely(!folio_test_error(folio))) | |
| 347 | + folio_mark_uptodate(folio); | |
| 354 | 348 | else /* Signal synchronous i/o error. */ |
| 355 | 349 | nr = -EIO; |
| 356 | − unlock_page(page); | |
| 350 | + folio_unlock(folio); | |
| 357 | 351 | return nr; |
| 358 | 352 | } |
| 359 | 353 | |
| @@ -433,7 +427,7 @@ retry_readpage: · 73 unmodified lines | ||
| 433 | 427 | /* NInoNonResident() == NInoIndexAllocPresent() */ |
| 434 | 428 | if (NInoNonResident(ni)) { |
| 435 | 429 | /* Normal, non-resident data stream. */ |
| 436 | − return ntfs_read_block(page); | |
| 430 | + return ntfs_read_block(folio); | |
| 437 | 431 | } |
| 438 | 432 | /* |
| 439 | 433 | * Attribute is resident, implying it is not compressed or encrypted. |
| @@ -507,28 +501,29 @@ err_out: · 67 unmodified lines | ||
| 507 | 501 | #ifdef NTFS_RW |
| 508 | 502 | |
| 509 | 503 | /** |
| 510 | − * ntfs_write_block - write a @page to the backing store | |
| 511 | − * @page: page cache page to write out | |
| 504 | + * ntfs_write_block - write a @folio to the backing store | |
| 505 | + * @folio: page cache folio to write out | |
| 512 | 506 | * @wbc: writeback control structure |
| 513 | 507 | * |
| 514 | − * This function is for writing pages belonging to non-resident, non-mst | |
| 508 | + * This function is for writing folios belonging to non-resident, non-mst | |
| 515 | 509 | * protected attributes to their backing store. |
| 516 | 510 | * |
| 517 | − * For a page with buffers, map and write the dirty buffers asynchronously | |
| 518 | − * under page writeback. For a page without buffers, create buffers for the | |
| 519 | − * page, then proceed as above. | |
| 511 | + * For a folio with buffers, map and write the dirty buffers asynchronously | |
| 512 | + * under folio writeback. For a folio without buffers, create buffers for the | |
| 513 | + * folio, then proceed as above. | |
| 520 | 514 | * |
| 521 | − * If a page doesn't have buffers the page dirty state is definitive. If a page | |
| 522 | − * does have buffers, the page dirty state is just a hint, and the buffer dirty | |
| 523 | − * state is definitive. (A hint which has rules: dirty buffers against a clean | |
| 524 | − * page is illegal. Other combinations are legal and need to be handled. In | |
| 525 | − * particular a dirty page containing clean buffers for example.) | |
| 515 | + * If a folio doesn't have buffers the folio dirty state is definitive. If | |
| 516 | + * a folio does have buffers, the folio dirty state is just a hint, | |
| 517 | + * and the buffer dirty state is definitive. (A hint which has rules: | |
| 518 | + * dirty buffers against a clean folio is illegal. Other combinations are | |
| 519 | + * legal and need to be handled. In particular a dirty folio containing | |
| 520 | + * clean buffers for example.) | |
| 526 | 521 | * |
| 527 | 522 | * Return 0 on success and -errno on error. |
| 528 | 523 | * |
| 529 | 524 | * Based on ntfs_read_block() and __block_write_full_folio(). |
| 530 | 525 | */ |
| 531 | −static int ntfs_write_block(struct page *page, struct writeback_control *wbc) | |
| 526 | +static int ntfs_write_block(struct folio *folio, struct writeback_control *wbc) | |
| 532 | 527 | { |
| 533 | 528 | VCN vcn; |
| 534 | 529 | LCN lcn; |
| @@ -546,41 +541,29 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 11 unmodified lines | ||
| 546 | 541 | bool need_end_writeback; |
| 547 | 542 | unsigned char blocksize_bits; |
| 548 | 543 | |
| 549 | − vi = page->mapping->host; | |
| 544 | + vi = folio->mapping->host; | |
| 550 | 545 | ni = NTFS_I(vi); |
| 551 | 546 | vol = ni->vol; |
| 552 | 547 | |
| 553 | 548 | ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, page index " |
| 554 | − "0x%lx.", ni->mft_no, ni->type, page->index); | |
| 549 | + "0x%lx.", ni->mft_no, ni->type, folio->index); | |
| 555 | 550 | |
| 556 | 551 | BUG_ON(!NInoNonResident(ni)); |
| 557 | 552 | BUG_ON(NInoMstProtected(ni)); |
| 558 | 553 | blocksize = vol->sb->s_blocksize; |
| 559 | 554 | blocksize_bits = vol->sb->s_blocksize_bits; |
| 560 | − if (!page_has_buffers(page)) { | |
| 561 | − BUG_ON(!PageUptodate(page)); | |
| 562 | − create_empty_buffers(page, blocksize, | |
| 555 | + head = folio_buffers(folio); | |
| 556 | + if (!head) { | |
| 557 | + BUG_ON(!folio_test_uptodate(folio)); | |
| 558 | + head = create_empty_buffers(folio, blocksize, | |
| 563 | 559 | (1 << BH_Uptodate) | (1 << BH_Dirty)); |
| 564 | − if (unlikely(!page_has_buffers(page))) { | |
| 565 | − ntfs_warning(vol->sb, "Error allocating page " | |
| 566 | − "buffers. Redirtying page so we try " | |
| 567 | − "again later."); | |
| 568 | − /* | |
| 569 | − * Put the page back on mapping->dirty_pages, but leave | |
| 570 | − * its buffers' dirty state as-is. | |
| 571 | − */ | |
| 572 | − redirty_page_for_writepage(wbc, page); | |
| 573 | − unlock_page(page); | |
| 574 | − return 0; | |
| 575 | − } | |
| 576 | 560 | } |
| 577 | − bh = head = page_buffers(page); | |
| 578 | − BUG_ON(!bh); | |
| 561 | + bh = head; | |
| 579 | 562 | |
| 580 | 563 | /* NOTE: Different naming scheme to ntfs_read_block()! */ |
| 581 | 564 | |
| 582 | − /* The first block in the page. */ | |
| 583 | − block = (s64)page->index << (PAGE_SHIFT - blocksize_bits); | |
| 565 | + /* The first block in the folio. */ | |
| 566 | + block = (s64)folio->index << (PAGE_SHIFT - blocksize_bits); | |
| 584 | 567 | |
| 585 | 568 | read_lock_irqsave(&ni->size_lock, flags); |
| 586 | 569 | i_size = i_size_read(vi); |
| @@ -597,14 +580,14 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 10 unmodified lines | ||
| 597 | 580 | * Be very careful. We have no exclusion from block_dirty_folio |
| 598 | 581 | * here, and the (potentially unmapped) buffers may become dirty at |
| 599 | 582 | * any time. If a buffer becomes dirty here after we've inspected it |
| 600 | − * then we just miss that fact, and the page stays dirty. | |
| 583 | + * then we just miss that fact, and the folio stays dirty. | |
| 601 | 584 | * |
| 602 | 585 | * Buffers outside i_size may be dirtied by block_dirty_folio; |
| 603 | 586 | * handle that here by just cleaning them. |
| 604 | 587 | */ |
| 605 | 588 | |
| 606 | 589 | /* |
| 607 | − * Loop through all the buffers in the page, mapping all the dirty | |
| 590 | + * Loop through all the buffers in the folio, mapping all the dirty | |
| 608 | 591 | * buffers to disk addresses and handling any aliases from the |
| 609 | 592 | * underlying block device's mapping. |
| 610 | 593 | */ |
| @@ -616,13 +599,13 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 5 unmodified lines | ||
| 616 | 599 | if (unlikely(block >= dblock)) { |
| 617 | 600 | /* |
| 618 | 601 | * Mapped buffers outside i_size will occur, because |
| 619 | − * this page can be outside i_size when there is a | |
| 602 | + * this folio can be outside i_size when there is a | |
| 620 | 603 | * truncate in progress. The contents of such buffers |
| 621 | 604 | * were zeroed by ntfs_writepage(). |
| 622 | 605 | * |
| 623 | 606 | * FIXME: What about the small race window where |
| 624 | 607 | * ntfs_writepage() has not done any clearing because |
| 625 | − * the page was within i_size but before we get here, | |
| 608 | + * the folio was within i_size but before we get here, | |
| 626 | 609 | * vmtruncate() modifies i_size? |
| 627 | 610 | */ |
| 628 | 611 | clear_buffer_dirty(bh); |
| @@ -638,38 +621,38 @@ static int ntfs_write_block(struct page *page, struct writeback_control *wbc) · 9 unmodified lines | ||
| 638 | 621 | if (unlikely((block >= iblock) && |
| 639 | 622 | (initialized_size < i_size))) { |
| 640 | 623 | /* |
| 641 | − * If this page is fully outside initialized | |
| 642 | − * size, zero out all pages between the current | |
| 643 | − * initialized size and the current page. Just | |
| 624 | + * If this folio is fully outside initialized | |
| 625 | + * size, zero out all folios between the current | |
| 626 | + * initialized size and the current folio. Just | |
| 644 | 627 | * use ntfs_read_folio() to do the zeroing |
| 645 | 628 | * transparently. |
| 646 | 629 | */ |
| 647 | 630 | if (block > iblock) { |
| 648 | 631 | // TODO: |
| 649 | − // For each page do: | |
| 650 | − // - read_cache_page() | |
| 651 | − // Again for each page do: | |
| 652 | − // - wait_on_page_locked() | |
| 653 | − // - Check (PageUptodate(page) && | |
| 654 | − // !PageError(page)) | |
| 632 | + // For each folio do: | |
| 633 | + // - read_cache_folio() | |
| 634 | + // Again for each folio do: | |
| 635 | + // - wait_on_folio_locked() | |
| 636 | + // - Check (folio_test_uptodate(folio) && | |
| 637 | + // !folio_test_error(folio)) | |
| 655 | 638 | // Update initialized size in the attribute and |
| 656 | 639 | // in the inode. |
| 657 | − // Again, for each page do: | |
| 640 | + // Again, for each folio do: | |
| 658 | 641 | // block_dirty_folio(); |
| 659 | − // put_page() | |
| 642 | + // folio_put() | |
| 660 | 643 | // We don't need to wait on the writes. |
| 661 | 644 | // Update iblock. |
| 662 | 645 | } |
| 663 | 646 | /* |
| 664 | − * The current page straddles initialized size. Zero | |
| 647 | + * The current folio straddles initialized size. Zero | |
| 665 | 648 | * all non-uptodate buffers and set them uptodate (and |
| 666 | 649 | * dirty?). Note, there aren't any non-uptodate buffers |
| 667 | − * if the page is uptodate. | |
| 668 | − * FIXME: For an uptodate page, the buffers may need to | |
| 650 | + * if the folio is uptodate. | |
| 651 | + * FIXME: For an uptodate folio, the buffers may need to | |
| 669 | 652 | * be written out because they were not initialized on |
| 670 | 653 | * disk before. |
| 671 | 654 | */ |
| 672 | − if (!PageUptodate(page)) { | |
| 655 | + if (!folio_test_uptodate(folio)) { | |
| 673 | 656 | // TODO: |
| 674 | 657 | // Zero any non-uptodate buffers up to i_size. |
| 675 | 658 | // Set them uptodate and dirty. |
| @@ -727,14 +710,14 @@ lock_retry_remap: · 51 unmodified lines | ||
| 727 | 710 | unsigned long *bpos, *bend; |
| 728 | 711 | |
| 729 | 712 | /* Check if the buffer is zero. */ |
| 730 | − kaddr = kmap_atomic(page); | |
| 731 | − bpos = (unsigned long *)(kaddr + bh_offset(bh)); | |
| 732 | − bend = (unsigned long *)((u8*)bpos + blocksize); | |
| 713 | + kaddr = kmap_local_folio(folio, bh_offset(bh)); | |
| 714 | + bpos = (unsigned long *)kaddr; | |
| 715 | + bend = (unsigned long *)(kaddr + blocksize); | |
| 733 | 716 | do { |
| 734 | 717 | if (unlikely(*bpos)) |
| 735 | 718 | break; |
| 736 | 719 | } while (likely(++bpos < bend)); |
| 737 | − kunmap_atomic(kaddr); | |
| 720 | + kunmap_local(kaddr); | |
| 738 | 721 | if (bpos == bend) { |
| 739 | 722 | /* |
| 740 | 723 | * Buffer is zero and sparse, no need to write |
| @@ -774,7 +757,7 @@ lock_retry_remap: · 33 unmodified lines | ||
| 774 | 757 | if (err == -ENOENT || lcn == LCN_ENOENT) { |
| 775 | 758 | bh->b_blocknr = -1; |
| 776 | 759 | clear_buffer_dirty(bh); |
| 777 | − zero_user(page, bh_offset(bh), blocksize); | |
| 760 | + folio_zero_range(folio, bh_offset(bh), blocksize); | |
| 778 | 761 | set_buffer_uptodate(bh); |
| 779 | 762 | err = 0; |
| 780 | 763 | continue; |
| @@ -801,7 +784,7 @@ lock_retry_remap: · 20 unmodified lines | ||
| 801 | 784 | bh = head; |
| 802 | 785 | |
| 803 | 786 | /* Just an optimization, so ->read_folio() is not called later. */ |
| 804 | − if (unlikely(!PageUptodate(page))) { | |
| 787 | + if (unlikely(!folio_test_uptodate(folio))) { | |
| 805 | 788 | int uptodate = 1; |
| 806 | 789 | do { |
| 807 | 790 | if (!buffer_uptodate(bh)) { |
| @@ -811,7 +794,7 @@ lock_retry_remap: · 3 unmodified lines | ||
| 811 | 794 | } |
| 812 | 795 | } while ((bh = bh->b_this_page) != head); |
| 813 | 796 | if (uptodate) |
| 814 | − SetPageUptodate(page); | |
| 797 | + folio_mark_uptodate(folio); | |
| 815 | 798 | } |
| 816 | 799 | |
| 817 | 800 | /* Setup all mapped, dirty buffers for async write i/o. */ |
| @@ -826,7 +809,7 @@ lock_retry_remap: · 8 unmodified lines | ||
| 826 | 809 | } else if (unlikely(err)) { |
| 827 | 810 | /* |
| 828 | 811 | * For the error case. The buffer may have been set |
| 829 | − * dirty during attachment to a dirty page. | |
| 812 | + * dirty during attachment to a dirty folio. | |
| 830 | 813 | */ |
| 831 | 814 | if (err != -ENOMEM) |
| 832 | 815 | clear_buffer_dirty(bh); |
| @@ -839,20 +822,20 @@ lock_retry_remap: · 6 unmodified lines | ||
| 839 | 822 | err = 0; |
| 840 | 823 | else if (err == -ENOMEM) { |
| 841 | 824 | ntfs_warning(vol->sb, "Error allocating memory. " |
| 842 | − "Redirtying page so we try again " | |
| 825 | + "Redirtying folio so we try again " | |
| 843 | 826 | "later."); |
| 844 | 827 | /* |
| 845 | − * Put the page back on mapping->dirty_pages, but | |
| 828 | + * Put the folio back on mapping->dirty_pages, but | |
| 846 | 829 | * leave its buffer's dirty state as-is. |
| 847 | 830 | */ |
| 848 | − redirty_page_for_writepage(wbc, page); | |
| 831 | + folio_redirty_for_writepage(wbc, folio); | |
| 849 | 832 | err = 0; |
| 850 | 833 | } else |
| 851 | − SetPageError(page); | |
| 834 | + folio_set_error(folio); | |
| 852 | 835 | } |
| 853 | 836 | |
| 854 | − BUG_ON(PageWriteback(page)); | |
| 855 | − set_page_writeback(page); /* Keeps try_to_free_buffers() away. */ | |
| 837 | + BUG_ON(folio_test_writeback(folio)); | |
| 838 | + folio_start_writeback(folio); /* Keeps try_to_free_buffers() away. */ | |
| 856 | 839 | |
| 857 | 840 | /* Submit the prepared buffers for i/o. */ |
| 858 | 841 | need_end_writeback = true; |
| @@ -864,11 +847,11 @@ lock_retry_remap: · 5 unmodified lines | ||
| 864 | 847 | } |
| 865 | 848 | bh = next; |
| 866 | 849 | } while (bh != head); |
| 867 | − unlock_page(page); | |
| 850 | + folio_unlock(folio); | |
| 868 | 851 | |
| 869 | − /* If no i/o was started, need to end_page_writeback(). */ | |
| 852 | + /* If no i/o was started, need to end writeback here. */ | |
| 870 | 853 | if (unlikely(need_end_writeback)) |
| 871 | − end_page_writeback(page); | |
| 854 | + folio_end_writeback(folio); | |
| 872 | 855 | |
| 873 | 856 | ntfs_debug("Done."); |
| 874 | 857 | return err; |
| @@ -1337,8 +1320,9 @@ done: · 462 unmodified lines | ||
| 1337 | 1320 | */ |
| 1338 | 1321 | static int ntfs_writepage(struct page *page, struct writeback_control *wbc) |
| 1339 | 1322 | { |
| 1323 | + struct folio *folio = page_folio(page); | |
| 1340 | 1324 | loff_t i_size; |
| 1341 | − struct inode *vi = page->mapping->host; | |
| 1325 | + struct inode *vi = folio->mapping->host; | |
| 1342 | 1326 | ntfs_inode *base_ni = NULL, *ni = NTFS_I(vi); |
| 1343 | 1327 | char *addr; |
| 1344 | 1328 | ntfs_attr_search_ctx *ctx = NULL; |
| @@ -1347,14 +1331,13 @@ static int ntfs_writepage(struct page *page, struct writeback_control *wbc) · 2 unmodified lines | ||
| 1347 | 1331 | int err; |
| 1348 | 1332 | |
| 1349 | 1333 | retry_writepage: |
| 1350 | − BUG_ON(!PageLocked(page)); | |
| 1334 | + BUG_ON(!folio_test_locked(folio)); | |
| 1351 | 1335 | i_size = i_size_read(vi); |
| 1352 | − /* Is the page fully outside i_size? (truncate in progress) */ | |
| 1353 | − if (unlikely(page->index >= (i_size + PAGE_SIZE - 1) >> | |
| 1336 | + /* Is the folio fully outside i_size? (truncate in progress) */ | |
| 1337 | + if (unlikely(folio->index >= (i_size + PAGE_SIZE - 1) >> | |
| 1354 | 1338 | PAGE_SHIFT)) { |
| 1355 | − struct folio *folio = page_folio(page); | |
| 1356 | 1339 | /* |
| 1357 | − * The page may have dirty, unmapped buffers. Make them | |
| 1340 | + * The folio may have dirty, unmapped buffers. Make them | |
| 1358 | 1341 | * freeable here, so the page does not leak. |
| 1359 | 1342 | */ |
| 1360 | 1343 | block_invalidate_folio(folio, 0, folio_size(folio)); |
| @@ -1373,7 +1356,7 @@ retry_writepage: · 12 unmodified lines | ||
| 1373 | 1356 | if (ni->type != AT_INDEX_ALLOCATION) { |
| 1374 | 1357 | /* If file is encrypted, deny access, just like NT4. */ |
| 1375 | 1358 | if (NInoEncrypted(ni)) { |
| 1376 | − unlock_page(page); | |
| 1359 | + folio_unlock(folio); | |
| 1377 | 1360 | BUG_ON(ni->type != AT_DATA); |
| 1378 | 1361 | ntfs_debug("Denying write access to encrypted file."); |
| 1379 | 1362 | return -EACCES; |
| @@ -1384,14 +1367,14 @@ retry_writepage: · 4 unmodified lines | ||
| 1384 | 1367 | BUG_ON(ni->name_len); |
| 1385 | 1368 | // TODO: Implement and replace this with |
| 1386 | 1369 | // return ntfs_write_compressed_block(page); |
| 1387 | − unlock_page(page); | |
| 1370 | + folio_unlock(folio); | |
| 1388 | 1371 | ntfs_error(vi->i_sb, "Writing to compressed files is " |
| 1389 | 1372 | "not supported yet. Sorry."); |
| 1390 | 1373 | return -EOPNOTSUPP; |
| 1391 | 1374 | } |
| 1392 | 1375 | // TODO: Implement and remove this check. |
| 1393 | 1376 | if (NInoNonResident(ni) && NInoSparse(ni)) { |
| 1394 | − unlock_page(page); | |
| 1377 | + folio_unlock(folio); | |
| 1395 | 1378 | ntfs_error(vi->i_sb, "Writing to sparse files is not " |
| 1396 | 1379 | "supported yet. Sorry."); |
| 1397 | 1380 | return -EOPNOTSUPP; |
| @@ -1400,34 +1383,34 @@ retry_writepage: · 2 unmodified lines | ||
| 1400 | 1383 | /* NInoNonResident() == NInoIndexAllocPresent() */ |
| 1401 | 1384 | if (NInoNonResident(ni)) { |
| 1402 | 1385 | /* We have to zero every time due to mmap-at-end-of-file. */ |
| 1403 | − if (page->index >= (i_size >> PAGE_SHIFT)) { | |
| 1404 | − /* The page straddles i_size. */ | |
| 1405 | − unsigned int ofs = i_size & ~PAGE_MASK; | |
| 1406 | − zero_user_segment(page, ofs, PAGE_SIZE); | |
| 1386 | + if (folio->index >= (i_size >> PAGE_SHIFT)) { | |
| 1387 | + /* The folio straddles i_size. */ | |
| 1388 | + unsigned int ofs = i_size & (folio_size(folio) - 1); | |
| 1389 | + folio_zero_segment(folio, ofs, folio_size(folio)); | |
| 1407 | 1390 | } |
| 1408 | 1391 | /* Handle mst protected attributes. */ |
| 1409 | 1392 | if (NInoMstProtected(ni)) |
| 1410 | 1393 | return ntfs_write_mst_block(page, wbc); |
| 1411 | 1394 | /* Normal, non-resident data stream. */ |
| 1412 | − return ntfs_write_block(page, wbc); | |
| 1395 | + return ntfs_write_block(folio, wbc); | |
| 1413 | 1396 | } |
| 1414 | 1397 | /* |
| 1415 | 1398 | * Attribute is resident, implying it is not compressed, encrypted, or |
| 1416 | 1399 | * mst protected. This also means the attribute is smaller than an mft |
| 1417 | − * record and hence smaller than a page, so can simply return error on | |
| 1418 | − * any pages with index above 0. Note the attribute can actually be | |
| 1400 | + * record and hence smaller than a folio, so can simply return error on | |
| 1401 | + * any folios with index above 0. Note the attribute can actually be | |
| 1419 | 1402 | * marked compressed but if it is resident the actual data is not |
| 1420 | 1403 | * compressed so we are ok to ignore the compressed flag here. |
| 1421 | 1404 | */ |
| 1422 | − BUG_ON(page_has_buffers(page)); | |
| 1423 | − BUG_ON(!PageUptodate(page)); | |
| 1424 | − if (unlikely(page->index > 0)) { | |
| 1425 | − ntfs_error(vi->i_sb, "BUG()! page->index (0x%lx) > 0. " | |
| 1426 | − "Aborting write.", page->index); | |
| 1427 | − BUG_ON(PageWriteback(page)); | |
| 1428 | − set_page_writeback(page); | |
| 1429 | − unlock_page(page); | |
| 1430 | − end_page_writeback(page); | |
| 1405 | + BUG_ON(folio_buffers(folio)); | |
| 1406 | + BUG_ON(!folio_test_uptodate(folio)); | |
| 1407 | + if (unlikely(folio->index > 0)) { | |
| 1408 | + ntfs_error(vi->i_sb, "BUG()! folio->index (0x%lx) > 0. " | |
| 1409 | + "Aborting write.", folio->index); | |
| 1410 | + BUG_ON(folio_test_writeback(folio)); | |
| 1411 | + folio_start_writeback(folio); | |
| 1412 | + folio_unlock(folio); | |
| 1413 | + folio_end_writeback(folio); | |
| 1431 | 1414 | return -EIO; |
| 1432 | 1415 | } |
| 1433 | 1416 | if (!NInoAttr(ni)) |
| @@ -1460,12 +1443,12 @@ retry_writepage: · 26 unmodified lines | ||
| 1460 | 1443 | if (unlikely(err)) |
| 1461 | 1444 | goto err_out; |
| 1462 | 1445 | /* |
| 1463 | − * Keep the VM happy. This must be done otherwise the radix-tree tag | |
| 1464 | − * PAGECACHE_TAG_DIRTY remains set even though the page is clean. | |
| 1446 | + * Keep the VM happy. This must be done otherwise | |
| 1447 | + * PAGECACHE_TAG_DIRTY remains set even though the folio is clean. | |
| 1465 | 1448 | */ |
| 1466 | − BUG_ON(PageWriteback(page)); | |
| 1467 | − set_page_writeback(page); | |
| 1468 | − unlock_page(page); | |
| 1449 | + BUG_ON(folio_test_writeback(folio)); | |
| 1450 | + folio_start_writeback(folio); | |
| 1451 | + folio_unlock(folio); | |
| 1469 | 1452 | attr_len = le32_to_cpu(ctx->attr->data.resident.value_length); |
| 1470 | 1453 | i_size = i_size_read(vi); |
| 1471 | 1454 | if (unlikely(attr_len > i_size)) { |
| @@ -1480,18 +1463,18 @@ retry_writepage: · 8 unmodified lines | ||
| 1480 | 1463 | /* Shrinking cannot fail. */ |
| 1481 | 1464 | BUG_ON(err); |
| 1482 | 1465 | } |
| 1483 | − addr = kmap_atomic(page); | |
| 1484 | − /* Copy the data from the page to the mft record. */ | |
| 1466 | + addr = kmap_local_folio(folio, 0); | |
| 1467 | + /* Copy the data from the folio to the mft record. */ | |
| 1485 | 1468 | memcpy((u8*)ctx->attr + |
| 1486 | 1469 | le16_to_cpu(ctx->attr->data.resident.value_offset), |
| 1487 | 1470 | addr, attr_len); |
| 1488 | − /* Zero out of bounds area in the page cache page. */ | |
| 1489 | − memset(addr + attr_len, 0, PAGE_SIZE - attr_len); | |
| 1490 | − kunmap_atomic(addr); | |
| 1491 | − flush_dcache_page(page); | |
| 1471 | + /* Zero out of bounds area in the page cache folio. */ | |
| 1472 | + memset(addr + attr_len, 0, folio_size(folio) - attr_len); | |
| 1473 | + kunmap_local(addr); | |
| 1474 | + flush_dcache_folio(folio); | |
| 1492 | 1475 | flush_dcache_mft_record_page(ctx->ntfs_ino); |
| 1493 | − /* We are done with the page. */ | |
| 1494 | − end_page_writeback(page); | |
| 1476 | + /* We are done with the folio. */ | |
| 1477 | + folio_end_writeback(folio); | |
| 1495 | 1478 | /* Finally, mark the mft record dirty, so it gets written back. */ |
| 1496 | 1479 | mark_mft_record_dirty(ctx->ntfs_ino); |
| 1497 | 1480 | ntfs_attr_put_search_ctx(ctx); |
| @@ -1502,18 +1485,18 @@ err_out: · 4 unmodified lines | ||
| 1502 | 1485 | ntfs_warning(vi->i_sb, "Error allocating memory. Redirtying " |
| 1503 | 1486 | "page so we try again later."); |
| 1504 | 1487 | /* |
| 1505 | − * Put the page back on mapping->dirty_pages, but leave its | |
| 1488 | + * Put the folio back on mapping->dirty_pages, but leave its | |
| 1506 | 1489 | * buffers' dirty state as-is. |
| 1507 | 1490 | */ |
| 1508 | − redirty_page_for_writepage(wbc, page); | |
| 1491 | + folio_redirty_for_writepage(wbc, folio); | |
| 1509 | 1492 | err = 0; |
| 1510 | 1493 | } else { |
| 1511 | 1494 | ntfs_error(vi->i_sb, "Resident attribute write failed with " |
| 1512 | 1495 | "error %i.", err); |
| 1513 | − SetPageError(page); | |
| 1496 | + folio_set_error(folio); | |
| 1514 | 1497 | NVolSetErrors(ni->vol); |
| 1515 | 1498 | } |
| 1516 | − unlock_page(page); | |
| 1499 | + folio_unlock(folio); | |
| 1517 | 1500 | if (ctx) |
| 1518 | 1501 | ntfs_attr_put_search_ctx(ctx); |
| 1519 | 1502 | if (m) |
| @@ -567,7 +567,7 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 566 unmodified lines | ||
| 567 | 567 | LCN lcn; |
| 568 | 568 | s64 bh_pos, vcn_len, end, initialized_size; |
| 569 | 569 | sector_t lcn_block; |
| 570 | − struct page *page; | |
| 570 | + struct folio *folio; | |
| 571 | 571 | struct inode *vi; |
| 572 | 572 | ntfs_inode *ni, *base_ni = NULL; |
| 573 | 573 | ntfs_volume *vol; |
| @@ -601,20 +601,6 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 27 unmodified lines | ||
| 601 | 601 | (long long)pos, bytes); |
| 602 | 602 | blocksize = vol->sb->s_blocksize; |
| 603 | 603 | blocksize_bits = vol->sb->s_blocksize_bits; |
| 604 | − u = 0; | |
| 605 | − do { | |
| 606 | − page = pages[u]; | |
| 607 | − BUG_ON(!page); | |
| 608 | − /* | |
| 609 | − * create_empty_buffers() will create uptodate/dirty buffers if | |
| 610 | − * the page is uptodate/dirty. | |
| 611 | − */ | |
| 612 | − if (!page_has_buffers(page)) { | |
| 613 | − create_empty_buffers(page, blocksize, 0); | |
| 614 | − if (unlikely(!page_has_buffers(page))) | |
| 615 | − return -ENOMEM; | |
| 616 | − } | |
| 617 | − } while (++u < nr_pages); | |
| 618 | 604 | rl_write_locked = false; |
| 619 | 605 | rl = NULL; |
| 620 | 606 | err = 0; |
| @@ -626,14 +612,21 @@ static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, · 5 unmodified lines | ||
| 626 | 612 | end = pos + bytes; |
| 627 | 613 | cend = (end + vol->cluster_size - 1) >> vol->cluster_size_bits; |
| 628 | 614 | /* |
| 629 | − * Loop over each page and for each page over each buffer. Use goto to | |
| 615 | + * Loop over each buffer in each folio. Use goto to | |
| 630 | 616 | * reduce indentation. |
| 631 | 617 | */ |
| 632 | 618 | u = 0; |
| 633 | −do_next_page: | |
| 634 | − page = pages[u]; | |
| 635 | − bh_pos = (s64)page->index << PAGE_SHIFT; | |
| 636 | − bh = head = page_buffers(page); | |
| 619 | +do_next_folio: | |
| 620 | + folio = page_folio(pages[u]); | |
| 621 | + bh_pos = folio_pos(folio); | |
| 622 | + head = folio_buffers(folio); | |
| 623 | + if (!head) | |
| 624 | + /* | |
| 625 | + * create_empty_buffers() will create uptodate/dirty | |
| 626 | + * buffers if the folio is uptodate/dirty. | |
| 627 | + */ | |
| 628 | + head = create_empty_buffers(folio, blocksize, 0); | |
| 629 | + bh = head; | |
| 637 | 630 | do { |
| 638 | 631 | VCN cdelta; |
| 639 | 632 | s64 bh_end; |
| @@ -653,15 +646,15 @@ do_next_page: · 13 unmodified lines | ||
| 653 | 646 | if (buffer_uptodate(bh)) |
| 654 | 647 | continue; |
| 655 | 648 | /* |
| 656 | − * The buffer is not uptodate. If the page is uptodate | |
| 649 | + * The buffer is not uptodate. If the folio is uptodate | |
| 657 | 650 | * set the buffer uptodate and otherwise ignore it. |
| 658 | 651 | */ |
| 659 | − if (PageUptodate(page)) { | |
| 652 | + if (folio_test_uptodate(folio)) { | |
| 660 | 653 | set_buffer_uptodate(bh); |
| 661 | 654 | continue; |
| 662 | 655 | } |
| 663 | 656 | /* |
| 664 | − * Neither the page nor the buffer are uptodate. If | |
| 657 | + * Neither the folio nor the buffer are uptodate. If | |
| 665 | 658 | * the buffer is only partially being written to, we |
| 666 | 659 | * need to read it in before the write, i.e. now. |
| 667 | 660 | */ |
| @@ -679,7 +672,7 @@ do_next_page: · 11 unmodified lines | ||
| 679 | 672 | ntfs_submit_bh_for_read(bh); |
| 680 | 673 | *wait_bh++ = bh; |
| 681 | 674 | } else { |
| 682 | − zero_user(page, bh_offset(bh), | |
| 675 | + folio_zero_range(folio, bh_offset(bh), | |
| 683 | 676 | blocksize); |
| 684 | 677 | set_buffer_uptodate(bh); |
| 685 | 678 | } |
| @@ -706,7 +699,7 @@ map_buffer_cached: · 20 unmodified lines | ||
| 706 | 699 | (bh_cofs >> blocksize_bits); |
| 707 | 700 | set_buffer_mapped(bh); |
| 708 | 701 | /* |
| 709 | − * If the page is uptodate so is the buffer. If the | |
| 702 | + * If the folio is uptodate so is the buffer. If the | |
| 710 | 703 | * buffer is fully outside the write, we ignore it if |
| 711 | 704 | * it was already allocated and we mark it dirty so it |
| 712 | 705 | * gets written out if we allocated it. On the other |
| @@ -714,7 +707,7 @@ map_buffer_cached: · 1 unmodified line | ||
| 714 | 707 | * marking it dirty we set buffer_new so we can do |
| 715 | 708 | * error recovery. |
| 716 | 709 | */ |
| 717 | − if (PageUptodate(page)) { | |
| 710 | + if (folio_test_uptodate(folio)) { | |
| 718 | 711 | if (!buffer_uptodate(bh)) |
| 719 | 712 | set_buffer_uptodate(bh); |
| 720 | 713 | if (unlikely(was_hole)) { |
| @@ -754,7 +747,8 @@ map_buffer_cached: · 33 unmodified lines | ||
| 754 | 747 | ntfs_submit_bh_for_read(bh); |
| 755 | 748 | *wait_bh++ = bh; |
| 756 | 749 | } else { |
| 757 | − zero_user(page, bh_offset(bh), | |
| 750 | + folio_zero_range(folio, | |
| 751 | + bh_offset(bh), | |
| 758 | 752 | blocksize); |
| 759 | 753 | set_buffer_uptodate(bh); |
| 760 | 754 | } |
| @@ -773,7 +767,7 @@ map_buffer_cached: · 12 unmodified lines | ||
| 773 | 767 | */ |
| 774 | 768 | if (bh_end <= pos || bh_pos >= end) { |
| 775 | 769 | if (!buffer_uptodate(bh)) { |
| 776 | − zero_user(page, bh_offset(bh), | |
| 770 | + folio_zero_range(folio, bh_offset(bh), | |
| 777 | 771 | blocksize); |
| 778 | 772 | set_buffer_uptodate(bh); |
| 779 | 773 | } |
| @@ -786,7 +780,7 @@ map_buffer_cached: · 6 unmodified lines | ||
| 786 | 780 | u8 *kaddr; |
| 787 | 781 | unsigned pofs; |
| 788 | 782 | |
| 789 | − kaddr = kmap_atomic(page); | |
| 783 | + kaddr = kmap_local_folio(folio, 0); | |
| 790 | 784 | if (bh_pos < pos) { |
| 791 | 785 | pofs = bh_pos & ~PAGE_MASK; |
| 792 | 786 | memset(kaddr + pofs, 0, pos - bh_pos); |
| @@ -795,8 +789,8 @@ map_buffer_cached: · 2 unmodified lines | ||
| 795 | 789 | pofs = end & ~PAGE_MASK; |
| 796 | 790 | memset(kaddr + pofs, 0, bh_end - end); |
| 797 | 791 | } |
| 798 | − kunmap_atomic(kaddr); | |
| 799 | − flush_dcache_page(page); | |
| 792 | + kunmap_local(kaddr); | |
| 793 | + flush_dcache_folio(folio); | |
| 800 | 794 | } |
| 801 | 795 | continue; |
| 802 | 796 | } |
| @@ -809,11 +803,12 @@ map_buffer_cached: · 6 unmodified lines | ||
| 809 | 803 | initialized_size = ni->allocated_size; |
| 810 | 804 | read_unlock_irqrestore(&ni->size_lock, flags); |
| 811 | 805 | if (bh_pos > initialized_size) { |
| 812 | − if (PageUptodate(page)) { | |
| 806 | + if (folio_test_uptodate(folio)) { | |
| 813 | 807 | if (!buffer_uptodate(bh)) |
| 814 | 808 | set_buffer_uptodate(bh); |
| 815 | 809 | } else if (!buffer_uptodate(bh)) { |
| 816 | − zero_user(page, bh_offset(bh), blocksize); | |
| 810 | + folio_zero_range(folio, bh_offset(bh), | |
| 811 | + blocksize); | |
| 817 | 812 | set_buffer_uptodate(bh); |
| 818 | 813 | } |
| 819 | 814 | continue; |
| @@ -927,17 +922,17 @@ rl_not_mapped_enoent: · 107 unmodified lines | ||
| 927 | 922 | bh->b_blocknr = -1; |
| 928 | 923 | /* |
| 929 | 924 | * If the buffer is uptodate we skip it. If it |
| 930 | − * is not but the page is uptodate, we can set | |
| 931 | − * the buffer uptodate. If the page is not | |
| 925 | + * is not but the folio is uptodate, we can set | |
| 926 | + * the buffer uptodate. If the folio is not | |
| 932 | 927 | * uptodate, we can clear the buffer and set it |
| 933 | 928 | * uptodate. Whether this is worthwhile is |
| 934 | 929 | * debatable and this could be removed. |
| 935 | 930 | */ |
| 936 | − if (PageUptodate(page)) { | |
| 931 | + if (folio_test_uptodate(folio)) { | |
| 937 | 932 | if (!buffer_uptodate(bh)) |
| 938 | 933 | set_buffer_uptodate(bh); |
| 939 | 934 | } else if (!buffer_uptodate(bh)) { |
| 940 | − zero_user(page, bh_offset(bh), | |
| 935 | + folio_zero_range(folio, bh_offset(bh), | |
| 941 | 936 | blocksize); |
| 942 | 937 | set_buffer_uptodate(bh); |
| 943 | 938 | } |
| @@ -1167,7 +1162,7 @@ rl_not_mapped_enoent: · 223 unmodified lines | ||
| 1167 | 1162 | } while (bh_pos += blocksize, (bh = bh->b_this_page) != head); |
| 1168 | 1163 | /* If there are no errors, do the next page. */ |
| 1169 | 1164 | if (likely(!err && ++u < nr_pages)) |
| 1170 | − goto do_next_page; | |
| 1165 | + goto do_next_folio; | |
| 1171 | 1166 | /* If there are no errors, release the runlist lock if we took it. */ |
| 1172 | 1167 | if (likely(!err)) { |
| 1173 | 1168 | if (unlikely(rl_write_locked)) { |
| @@ -1185,9 +1180,8 @@ rl_not_mapped_enoent: · 11 unmodified lines | ||
| 1185 | 1180 | bh = *--wait_bh; |
| 1186 | 1181 | wait_on_buffer(bh); |
| 1187 | 1182 | if (likely(buffer_uptodate(bh))) { |
| 1188 | − page = bh->b_page; | |
| 1189 | − bh_pos = ((s64)page->index << PAGE_SHIFT) + | |
| 1190 | − bh_offset(bh); | |
| 1183 | + folio = bh->b_folio; | |
| 1184 | + bh_pos = folio_pos(folio) + bh_offset(bh); | |
| 1191 | 1185 | /* |
| 1192 | 1186 | * If the buffer overflows the initialized size, need |
| 1193 | 1187 | * to zero the overflowing region. |
| @@ -1197,7 +1191,7 @@ rl_not_mapped_enoent: · 3 unmodified lines | ||
| 1197 | 1191 | |
| 1198 | 1192 | if (likely(bh_pos < initialized_size)) |
| 1199 | 1193 | ofs = initialized_size - bh_pos; |
| 1200 | − zero_user_segment(page, bh_offset(bh) + ofs, | |
| 1194 | + folio_zero_segment(folio, bh_offset(bh) + ofs, | |
| 1201 | 1195 | blocksize); |
| 1202 | 1196 | } |
| 1203 | 1197 | } else /* if (unlikely(!buffer_uptodate(bh))) */ |
| @@ -1324,21 +1318,20 @@ rl_not_mapped_enoent: · 120 unmodified lines | ||
| 1324 | 1318 | u = 0; |
| 1325 | 1319 | end = bh_cpos << vol->cluster_size_bits; |
| 1326 | 1320 | do { |
| 1327 | − page = pages[u]; | |
| 1328 | − bh = head = page_buffers(page); | |
| 1321 | + folio = page_folio(pages[u]); | |
| 1322 | + bh = head = folio_buffers(folio); | |
| 1329 | 1323 | do { |
| 1330 | 1324 | if (u == nr_pages && |
| 1331 | − ((s64)page->index << PAGE_SHIFT) + | |
| 1332 | − bh_offset(bh) >= end) | |
| 1325 | + folio_pos(folio) + bh_offset(bh) >= end) | |
| 1333 | 1326 | break; |
| 1334 | 1327 | if (!buffer_new(bh)) |
| 1335 | 1328 | continue; |
| 1336 | 1329 | clear_buffer_new(bh); |
| 1337 | 1330 | if (!buffer_uptodate(bh)) { |
| 1338 | − if (PageUptodate(page)) | |
| 1331 | + if (folio_test_uptodate(folio)) | |
| 1339 | 1332 | set_buffer_uptodate(bh); |
| 1340 | 1333 | else { |
| 1341 | − zero_user(page, bh_offset(bh), | |
| 1334 | + folio_zero_range(folio, bh_offset(bh), | |
| 1342 | 1335 | blocksize); |
| 1343 | 1336 | set_buffer_uptodate(bh); |
| 1344 | 1337 | } |
| @@ -187,7 +187,7 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 186 unmodified lines | ||
| 187 | 187 | struct buffer_head *head, *bh; |
| 188 | 188 | u32 bh_next, bh_off, to; |
| 189 | 189 | sector_t iblock; |
| 190 | − struct page *page; | |
| 190 | + struct folio *folio; | |
| 191 | 191 | |
| 192 | 192 | for (; idx < idx_end; idx += 1, from = 0) { |
| 193 | 193 | page_off = (loff_t)idx << PAGE_SHIFT; |
| @@ -195,16 +195,17 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 1 unmodified line | ||
| 195 | 195 | PAGE_SIZE; |
| 196 | 196 | iblock = page_off >> inode->i_blkbits; |
| 197 | 197 | |
| 198 | − page = find_or_create_page(mapping, idx, | |
| 199 | − mapping_gfp_constraint(mapping, | |
| 200 | − ~__GFP_FS)); | |
| 201 | − if (!page) | |
| 202 | − return -ENOMEM; | |
| 198 | + folio = __filemap_get_folio(mapping, idx, | |
| 199 | + FGP_LOCK | FGP_ACCESSED | FGP_CREAT, | |
| 200 | + mapping_gfp_constraint(mapping, ~__GFP_FS)); | |
| 201 | + if (IS_ERR(folio)) | |
| 202 | + return PTR_ERR(folio); | |
| 203 | 203 | |
| 204 | − if (!page_has_buffers(page)) | |
| 205 | − create_empty_buffers(page, blocksize, 0); | |
| 204 | + head = folio_buffers(folio); | |
| 205 | + if (!head) | |
| 206 | + head = create_empty_buffers(folio, blocksize, 0); | |
| 206 | 207 | |
| 207 | − bh = head = page_buffers(page); | |
| 208 | + bh = head; | |
| 208 | 209 | bh_off = 0; |
| 209 | 210 | do { |
| 210 | 211 | bh_next = bh_off + blocksize; |
| @@ -220,14 +221,14 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 9 unmodified lines | ||
| 220 | 221 | } |
| 221 | 222 | |
| 222 | 223 | /* Ok, it's mapped. Make sure it's up-to-date. */ |
| 223 | − if (PageUptodate(page)) | |
| 224 | + if (folio_test_uptodate(folio)) | |
| 224 | 225 | set_buffer_uptodate(bh); |
| 225 | 226 | |
| 226 | 227 | if (!buffer_uptodate(bh)) { |
| 227 | 228 | err = bh_read(bh, 0); |
| 228 | 229 | if (err < 0) { |
| 229 | − unlock_page(page); | |
| 230 | − put_page(page); | |
| 230 | + folio_unlock(folio); | |
| 231 | + folio_put(folio); | |
| 231 | 232 | goto out; |
| 232 | 233 | } |
| 233 | 234 | } |
| @@ -237,10 +238,10 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) · 3 unmodified lines | ||
| 237 | 238 | } while (bh_off = bh_next, iblock += 1, |
| 238 | 239 | head != (bh = bh->b_this_page)); |
| 239 | 240 | |
| 240 | − zero_user_segment(page, from, to); | |
| 241 | + folio_zero_segment(folio, from, to); | |
| 241 | 242 | |
| 242 | − unlock_page(page); | |
| 243 | − put_page(page); | |
| 243 | + folio_unlock(folio); | |
| 244 | + folio_put(folio); | |
| 244 | 245 | cond_resched(); |
| 245 | 246 | } |
| 246 | 247 | out: |
| @@ -568,10 +568,10 @@ static void ocfs2_clear_page_regions(struct page *page, · 567 unmodified lines | ||
| 568 | 568 | * read-in the blocks at the tail of our file. Avoid reading them by |
| 569 | 569 | * testing i_size against each block offset. |
| 570 | 570 | */ |
| 571 | −static int ocfs2_should_read_blk(struct inode *inode, struct page *page, | |
| 571 | +static int ocfs2_should_read_blk(struct inode *inode, struct folio *folio, | |
| 572 | 572 | unsigned int block_start) |
| 573 | 573 | { |
| 574 | − u64 offset = page_offset(page) + block_start; | |
| 574 | + u64 offset = folio_pos(folio) + block_start; | |
| 575 | 575 | |
| 576 | 576 | if (ocfs2_sparse_alloc(OCFS2_SB(inode->i_sb))) |
| 577 | 577 | return 1; |
| @@ -593,15 +593,16 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 15 unmodified lines | ||
| 593 | 593 | struct inode *inode, unsigned int from, |
| 594 | 594 | unsigned int to, int new) |
| 595 | 595 | { |
| 596 | + struct folio *folio = page_folio(page); | |
| 596 | 597 | int ret = 0; |
| 597 | 598 | struct buffer_head *head, *bh, *wait[2], **wait_bh = wait; |
| 598 | 599 | unsigned int block_end, block_start; |
| 599 | 600 | unsigned int bsize = i_blocksize(inode); |
| 600 | 601 | |
| 601 | − if (!page_has_buffers(page)) | |
| 602 | − create_empty_buffers(page, bsize, 0); | |
| 602 | + head = folio_buffers(folio); | |
| 603 | + if (!head) | |
| 604 | + head = create_empty_buffers(folio, bsize, 0); | |
| 603 | 605 | |
| 604 | − head = page_buffers(page); | |
| 605 | 606 | for (bh = head, block_start = 0; bh != head || !block_start; |
| 606 | 607 | bh = bh->b_this_page, block_start += bsize) { |
| 607 | 608 | block_end = block_start + bsize; |
| @@ -613,7 +614,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 5 unmodified lines | ||
| 613 | 614 | * they may belong to unallocated clusters. |
| 614 | 615 | */ |
| 615 | 616 | if (block_start >= to || block_end <= from) { |
| 616 | − if (PageUptodate(page)) | |
| 617 | + if (folio_test_uptodate(folio)) | |
| 617 | 618 | set_buffer_uptodate(bh); |
| 618 | 619 | continue; |
| 619 | 620 | } |
| @@ -630,11 +631,11 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 10 unmodified lines | ||
| 630 | 631 | clean_bdev_bh_alias(bh); |
| 631 | 632 | } |
| 632 | 633 | |
| 633 | − if (PageUptodate(page)) { | |
| 634 | + if (folio_test_uptodate(folio)) { | |
| 634 | 635 | set_buffer_uptodate(bh); |
| 635 | 636 | } else if (!buffer_uptodate(bh) && !buffer_delay(bh) && |
| 636 | 637 | !buffer_new(bh) && |
| 637 | − ocfs2_should_read_blk(inode, page, block_start) && | |
| 638 | + ocfs2_should_read_blk(inode, folio, block_start) && | |
| 638 | 639 | (block_start < from || block_end > to)) { |
| 639 | 640 | bh_read_nowait(bh, 0); |
| 640 | 641 | *wait_bh++=bh; |
| @@ -668,7 +669,7 @@ int ocfs2_map_page_blocks(struct page *page, u64 *p_blkno, · 27 unmodified lines | ||
| 668 | 669 | if (block_start >= to) |
| 669 | 670 | break; |
| 670 | 671 | |
| 671 | − zero_user(page, block_start, bh->b_size); | |
| 672 | + folio_zero_range(folio, block_start, bh->b_size); | |
| 672 | 673 | set_buffer_uptodate(bh); |
| 673 | 674 | mark_buffer_dirty(bh); |
| 674 | 675 | |
| @@ -546,7 +546,8 @@ static ssize_t read_kcore_iter(struct kiocb *iocb, struct iov_iter *iter) · 545 unmodified lines | ||
| 546 | 546 | * and explicitly excluded physical ranges. |
| 547 | 547 | */ |
| 548 | 548 | if (!page || PageOffline(page) || |
| 549 | − is_page_hwpoison(page) || !pfn_is_ram(pfn)) { | |
| 549 | + is_page_hwpoison(page) || !pfn_is_ram(pfn) || | |
| 550 | + pfn_is_unaccepted_memory(pfn)) { | |
| 550 | 551 | if (iov_iter_zero(tsz, iter) != tsz) { |
| 551 | 552 | ret = -EFAULT; |
| 552 | 553 | goto out; |
| @@ -188,7 +188,7 @@ static int proc_fill_super(struct super_block *s, struct fs_context *fc) · 187 unmodified lines | ||
| 188 | 188 | s->s_stack_depth = FILESYSTEM_MAX_STACK_DEPTH; |
| 189 | 189 | |
| 190 | 190 | /* procfs dentries and inodes don't require IO to create */ |
| 191 | − s->s_shrink.seeks = 0; | |
| 191 | + s->s_shrink->seeks = 0; | |
| 192 | 192 | |
| 193 | 193 | pde_get(&proc_root); |
| 194 | 194 | root_inode = proc_get_inode(s, &proc_root); |
| @@ -20,6 +20,8 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <linux/shmem_fs.h> |
| 21 | 21 | #include <linux/uaccess.h> |
| 22 | 22 | #include <linux/pkeys.h> |
| 23 | +#include <linux/minmax.h> | |
| 24 | +#include <linux/overflow.h> | |
| 23 | 25 | |
| 24 | 26 | #include <asm/elf.h> |
| 25 | 27 | #include <asm/tlb.h> |
| @@ -1761,11 +1763,737 @@ static int pagemap_release(struct inode *inode, struct file *file) · 1735 unmodified lines | ||
| 1761 | 1763 | return 0; |
| 1762 | 1764 | } |
| 1763 | 1765 | |
| 1766 | +#define PM_SCAN_CATEGORIES (PAGE_IS_WPALLOWED | PAGE_IS_WRITTEN | \ | |
| 1767 | + PAGE_IS_FILE | PAGE_IS_PRESENT | \ | |
| 1768 | + PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \ | |
| 1769 | + PAGE_IS_HUGE) | |
| 1770 | +#define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC) | |
| 1771 | + | |
| 1772 | +struct pagemap_scan_private { | |
| 1773 | + struct pm_scan_arg arg; | |
| 1774 | + unsigned long masks_of_interest, cur_vma_category; | |
| 1775 | + struct page_region *vec_buf; | |
| 1776 | + unsigned long vec_buf_len, vec_buf_index, found_pages; | |
| 1777 | + struct page_region __user *vec_out; | |
| 1778 | +}; | |
| 1779 | + | |
| 1780 | +static unsigned long pagemap_page_category(struct pagemap_scan_private *p, | |
| 1781 | + struct vm_area_struct *vma, | |
| 1782 | + unsigned long addr, pte_t pte) | |
| 1783 | +{ | |
| 1784 | + unsigned long categories = 0; | |
| 1785 | + | |
| 1786 | + if (pte_present(pte)) { | |
| 1787 | + struct page *page; | |
| 1788 | + | |
| 1789 | + categories |= PAGE_IS_PRESENT; | |
| 1790 | + if (!pte_uffd_wp(pte)) | |
| 1791 | + categories |= PAGE_IS_WRITTEN; | |
| 1792 | + | |
| 1793 | + if (p->masks_of_interest & PAGE_IS_FILE) { | |
| 1794 | + page = vm_normal_page(vma, addr, pte); | |
| 1795 | + if (page && !PageAnon(page)) | |
| 1796 | + categories |= PAGE_IS_FILE; | |
| 1797 | + } | |
| 1798 | + | |
| 1799 | + if (is_zero_pfn(pte_pfn(pte))) | |
| 1800 | + categories |= PAGE_IS_PFNZERO; | |
| 1801 | + } else if (is_swap_pte(pte)) { | |
| 1802 | + swp_entry_t swp; | |
| 1803 | + | |
| 1804 | + categories |= PAGE_IS_SWAPPED; | |
| 1805 | + if (!pte_swp_uffd_wp_any(pte)) | |
| 1806 | + categories |= PAGE_IS_WRITTEN; | |
| 1807 | + | |
| 1808 | + if (p->masks_of_interest & PAGE_IS_FILE) { | |
| 1809 | + swp = pte_to_swp_entry(pte); | |
| 1810 | + if (is_pfn_swap_entry(swp) && | |
| 1811 | + !PageAnon(pfn_swap_entry_to_page(swp))) | |
| 1812 | + categories |= PAGE_IS_FILE; | |
| 1813 | + } | |
| 1814 | + } | |
| 1815 | + | |
| 1816 | + return categories; | |
| 1817 | +} | |
| 1818 | + | |
| 1819 | +static void make_uffd_wp_pte(struct vm_area_struct *vma, | |
| 1820 | + unsigned long addr, pte_t *pte) | |
| 1821 | +{ | |
| 1822 | + pte_t ptent = ptep_get(pte); | |
| 1823 | + | |
| 1824 | + if (pte_present(ptent)) { | |
| 1825 | + pte_t old_pte; | |
| 1826 | + | |
| 1827 | + old_pte = ptep_modify_prot_start(vma, addr, pte); | |
| 1828 | + ptent = pte_mkuffd_wp(ptent); | |
| 1829 | + ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent); | |
| 1830 | + } else if (is_swap_pte(ptent)) { | |
| 1831 | + ptent = pte_swp_mkuffd_wp(ptent); | |
| 1832 | + set_pte_at(vma->vm_mm, addr, pte, ptent); | |
| 1833 | + } else { | |
| 1834 | + set_pte_at(vma->vm_mm, addr, pte, | |
| 1835 | + make_pte_marker(PTE_MARKER_UFFD_WP)); | |
| 1836 | + } | |
| 1837 | +} | |
| 1838 | + | |
| 1839 | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 1840 | +static unsigned long pagemap_thp_category(struct pagemap_scan_private *p, | |
| 1841 | + struct vm_area_struct *vma, | |
| 1842 | + unsigned long addr, pmd_t pmd) | |
| 1843 | +{ | |
| 1844 | + unsigned long categories = PAGE_IS_HUGE; | |
| 1845 | + | |
| 1846 | + if (pmd_present(pmd)) { | |
| 1847 | + struct page *page; | |
| 1848 | + | |
| 1849 | + categories |= PAGE_IS_PRESENT; | |
| 1850 | + if (!pmd_uffd_wp(pmd)) | |
| 1851 | + categories |= PAGE_IS_WRITTEN; | |
| 1852 | + | |
| 1853 | + if (p->masks_of_interest & PAGE_IS_FILE) { | |
| 1854 | + page = vm_normal_page_pmd(vma, addr, pmd); | |
| 1855 | + if (page && !PageAnon(page)) | |
| 1856 | + categories |= PAGE_IS_FILE; | |
| 1857 | + } | |
| 1858 | + | |
| 1859 | + if (is_zero_pfn(pmd_pfn(pmd))) | |
| 1860 | + categories |= PAGE_IS_PFNZERO; | |
| 1861 | + } else if (is_swap_pmd(pmd)) { | |
| 1862 | + swp_entry_t swp; | |
| 1863 | + | |
| 1864 | + categories |= PAGE_IS_SWAPPED; | |
| 1865 | + if (!pmd_swp_uffd_wp(pmd)) | |
| 1866 | + categories |= PAGE_IS_WRITTEN; | |
| 1867 | + | |
| 1868 | + if (p->masks_of_interest & PAGE_IS_FILE) { | |
| 1869 | + swp = pmd_to_swp_entry(pmd); | |
| 1870 | + if (is_pfn_swap_entry(swp) && | |
| 1871 | + !PageAnon(pfn_swap_entry_to_page(swp))) | |
| 1872 | + categories |= PAGE_IS_FILE; | |
| 1873 | + } | |
| 1874 | + } | |
| 1875 | + | |
| 1876 | + return categories; | |
| 1877 | +} | |
| 1878 | + | |
| 1879 | +static void make_uffd_wp_pmd(struct vm_area_struct *vma, | |
| 1880 | + unsigned long addr, pmd_t *pmdp) | |
| 1881 | +{ | |
| 1882 | + pmd_t old, pmd = *pmdp; | |
| 1883 | + | |
| 1884 | + if (pmd_present(pmd)) { | |
| 1885 | + old = pmdp_invalidate_ad(vma, addr, pmdp); | |
| 1886 | + pmd = pmd_mkuffd_wp(old); | |
| 1887 | + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); | |
| 1888 | + } else if (is_migration_entry(pmd_to_swp_entry(pmd))) { | |
| 1889 | + pmd = pmd_swp_mkuffd_wp(pmd); | |
| 1890 | + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); | |
| 1891 | + } | |
| 1892 | +} | |
| 1893 | +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ | |
| 1894 | + | |
| 1895 | +#ifdef CONFIG_HUGETLB_PAGE | |
| 1896 | +static unsigned long pagemap_hugetlb_category(pte_t pte) | |
| 1897 | +{ | |
| 1898 | + unsigned long categories = PAGE_IS_HUGE; | |
| 1899 | + | |
| 1900 | + /* | |
| 1901 | + * According to pagemap_hugetlb_range(), file-backed HugeTLB | |
| 1902 | + * page cannot be swapped. So PAGE_IS_FILE is not checked for | |
| 1903 | + * swapped pages. | |
| 1904 | + */ | |
| 1905 | + if (pte_present(pte)) { | |
| 1906 | + categories |= PAGE_IS_PRESENT; | |
| 1907 | + if (!huge_pte_uffd_wp(pte)) | |
| 1908 | + categories |= PAGE_IS_WRITTEN; | |
| 1909 | + if (!PageAnon(pte_page(pte))) | |
| 1910 | + categories |= PAGE_IS_FILE; | |
| 1911 | + if (is_zero_pfn(pte_pfn(pte))) | |
| 1912 | + categories |= PAGE_IS_PFNZERO; | |
| 1913 | + } else if (is_swap_pte(pte)) { | |
| 1914 | + categories |= PAGE_IS_SWAPPED; | |
| 1915 | + if (!pte_swp_uffd_wp_any(pte)) | |
| 1916 | + categories |= PAGE_IS_WRITTEN; | |
| 1917 | + } | |
| 1918 | + | |
| 1919 | + return categories; | |
| 1920 | +} | |
| 1921 | + | |
| 1922 | +static void make_uffd_wp_huge_pte(struct vm_area_struct *vma, | |
| 1923 | + unsigned long addr, pte_t *ptep, | |
| 1924 | + pte_t ptent) | |
| 1925 | +{ | |
| 1926 | + unsigned long psize; | |
| 1927 | + | |
| 1928 | + if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent)) | |
| 1929 | + return; | |
| 1930 | + | |
| 1931 | + psize = huge_page_size(hstate_vma(vma)); | |
| 1932 | + | |
| 1933 | + if (is_hugetlb_entry_migration(ptent)) | |
| 1934 | + set_huge_pte_at(vma->vm_mm, addr, ptep, | |
| 1935 | + pte_swp_mkuffd_wp(ptent), psize); | |
| 1936 | + else if (!huge_pte_none(ptent)) | |
| 1937 | + huge_ptep_modify_prot_commit(vma, addr, ptep, ptent, | |
| 1938 | + huge_pte_mkuffd_wp(ptent)); | |
| 1939 | + else | |
| 1940 | + set_huge_pte_at(vma->vm_mm, addr, ptep, | |
| 1941 | + make_pte_marker(PTE_MARKER_UFFD_WP), psize); | |
| 1942 | +} | |
| 1943 | +#endif /* CONFIG_HUGETLB_PAGE */ | |
| 1944 | + | |
| 1945 | +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLB_PAGE) | |
| 1946 | +static void pagemap_scan_backout_range(struct pagemap_scan_private *p, | |
| 1947 | + unsigned long addr, unsigned long end) | |
| 1948 | +{ | |
| 1949 | + struct page_region *cur_buf = &p->vec_buf[p->vec_buf_index]; | |
| 1950 | + | |
| 1951 | + if (cur_buf->start != addr) | |
| 1952 | + cur_buf->end = addr; | |
| 1953 | + else | |
| 1954 | + cur_buf->start = cur_buf->end = 0; | |
| 1955 | + | |
| 1956 | + p->found_pages -= (end - addr) / PAGE_SIZE; | |
| 1957 | +} | |
| 1958 | +#endif | |
| 1959 | + | |
| 1960 | +static bool pagemap_scan_is_interesting_page(unsigned long categories, | |
| 1961 | + const struct pagemap_scan_private *p) | |
| 1962 | +{ | |
| 1963 | + categories ^= p->arg.category_inverted; | |
| 1964 | + if ((categories & p->arg.category_mask) != p->arg.category_mask) | |
| 1965 | + return false; | |
| 1966 | + if (p->arg.category_anyof_mask && !(categories & p->arg.category_anyof_mask)) | |
| 1967 | + return false; | |
| 1968 | + | |
| 1969 | + return true; | |
| 1970 | +} | |
| 1971 | + | |
| 1972 | +static bool pagemap_scan_is_interesting_vma(unsigned long categories, | |
| 1973 | + const struct pagemap_scan_private *p) | |
| 1974 | +{ | |
| 1975 | + unsigned long required = p->arg.category_mask & PAGE_IS_WPALLOWED; | |
| 1976 | + | |
| 1977 | + categories ^= p->arg.category_inverted; | |
| 1978 | + if ((categories & required) != required) | |
| 1979 | + return false; | |
| 1980 | + | |
| 1981 | + return true; | |
| 1982 | +} | |
| 1983 | + | |
| 1984 | +static int pagemap_scan_test_walk(unsigned long start, unsigned long end, | |
| 1985 | + struct mm_walk *walk) | |
| 1986 | +{ | |
| 1987 | + struct pagemap_scan_private *p = walk->private; | |
| 1988 | + struct vm_area_struct *vma = walk->vma; | |
| 1989 | + unsigned long vma_category = 0; | |
| 1990 | + | |
| 1991 | + if (userfaultfd_wp_async(vma) && userfaultfd_wp_use_markers(vma)) | |
| 1992 | + vma_category |= PAGE_IS_WPALLOWED; | |
| 1993 | + else if (p->arg.flags & PM_SCAN_CHECK_WPASYNC) | |
| 1994 | + return -EPERM; | |
| 1995 | + | |
| 1996 | + if (vma->vm_flags & VM_PFNMAP) | |
| 1997 | + return 1; | |
| 1998 | + | |
| 1999 | + if (!pagemap_scan_is_interesting_vma(vma_category, p)) | |
| 2000 | + return 1; | |
| 2001 | + | |
| 2002 | + p->cur_vma_category = vma_category; | |
| 2003 | + | |
| 2004 | + return 0; | |
| 2005 | +} | |
| 2006 | + | |
| 2007 | +static bool pagemap_scan_push_range(unsigned long categories, | |
| 2008 | + struct pagemap_scan_private *p, | |
| 2009 | + unsigned long addr, unsigned long end) | |
| 2010 | +{ | |
| 2011 | + struct page_region *cur_buf = &p->vec_buf[p->vec_buf_index]; | |
| 2012 | + | |
| 2013 | + /* | |
| 2014 | + * When there is no output buffer provided at all, the sentinel values | |
| 2015 | + * won't match here. There is no other way for `cur_buf->end` to be | |
| 2016 | + * non-zero other than it being non-empty. | |
| 2017 | + */ | |
| 2018 | + if (addr == cur_buf->end && categories == cur_buf->categories) { | |
| 2019 | + cur_buf->end = end; | |
| 2020 | + return true; | |
| 2021 | + } | |
| 2022 | + | |
| 2023 | + if (cur_buf->end) { | |
| 2024 | + if (p->vec_buf_index >= p->vec_buf_len - 1) | |
| 2025 | + return false; | |
| 2026 | + | |
| 2027 | + cur_buf = &p->vec_buf[++p->vec_buf_index]; | |
| 2028 | + } | |
| 2029 | + | |
| 2030 | + cur_buf->start = addr; | |
| 2031 | + cur_buf->end = end; | |
| 2032 | + cur_buf->categories = categories; | |
| 2033 | + | |
| 2034 | + return true; | |
| 2035 | +} | |
| 2036 | + | |
| 2037 | +static int pagemap_scan_output(unsigned long categories, | |
| 2038 | + struct pagemap_scan_private *p, | |
| 2039 | + unsigned long addr, unsigned long *end) | |
| 2040 | +{ | |
| 2041 | + unsigned long n_pages, total_pages; | |
| 2042 | + int ret = 0; | |
| 2043 | + | |
| 2044 | + if (!p->vec_buf) | |
| 2045 | + return 0; | |
| 2046 | + | |
| 2047 | + categories &= p->arg.return_mask; | |
| 2048 | + | |
| 2049 | + n_pages = (*end - addr) / PAGE_SIZE; | |
| 2050 | + if (check_add_overflow(p->found_pages, n_pages, &total_pages) || | |
| 2051 | + total_pages > p->arg.max_pages) { | |
| 2052 | + size_t n_too_much = total_pages - p->arg.max_pages; | |
| 2053 | + *end -= n_too_much * PAGE_SIZE; | |
| 2054 | + n_pages -= n_too_much; | |
| 2055 | + ret = -ENOSPC; | |
| 2056 | + } | |
| 2057 | + | |
| 2058 | + if (!pagemap_scan_push_range(categories, p, addr, *end)) { | |
| 2059 | + *end = addr; | |
| 2060 | + n_pages = 0; | |
| 2061 | + ret = -ENOSPC; | |
| 2062 | + } | |
| 2063 | + | |
| 2064 | + p->found_pages += n_pages; | |
| 2065 | + if (ret) | |
| 2066 | + p->arg.walk_end = *end; | |
| 2067 | + | |
| 2068 | + return ret; | |
| 2069 | +} | |
| 2070 | + | |
| 2071 | +static int pagemap_scan_thp_entry(pmd_t *pmd, unsigned long start, | |
| 2072 | + unsigned long end, struct mm_walk *walk) | |
| 2073 | +{ | |
| 2074 | +#ifdef CONFIG_TRANSPARENT_HUGEPAGE | |
| 2075 | + struct pagemap_scan_private *p = walk->private; | |
| 2076 | + struct vm_area_struct *vma = walk->vma; | |
| 2077 | + unsigned long categories; | |
| 2078 | + spinlock_t *ptl; | |
| 2079 | + int ret = 0; | |
| 2080 | + | |
| 2081 | + ptl = pmd_trans_huge_lock(pmd, vma); | |
| 2082 | + if (!ptl) | |
| 2083 | + return -ENOENT; | |
| 2084 | + | |
| 2085 | + categories = p->cur_vma_category | | |
| 2086 | + pagemap_thp_category(p, vma, start, *pmd); | |
| 2087 | + | |
| 2088 | + if (!pagemap_scan_is_interesting_page(categories, p)) | |
| 2089 | + goto out_unlock; | |
| 2090 | + | |
| 2091 | + ret = pagemap_scan_output(categories, p, start, &end); | |
| 2092 | + if (start == end) | |
| 2093 | + goto out_unlock; | |
| 2094 | + | |
| 2095 | + if (~p->arg.flags & PM_SCAN_WP_MATCHING) | |
| 2096 | + goto out_unlock; | |
| 2097 | + if (~categories & PAGE_IS_WRITTEN) | |
| 2098 | + goto out_unlock; | |
| 2099 | + | |
| 2100 | + /* | |
| 2101 | + * Break huge page into small pages if the WP operation | |
| 2102 | + * needs to be performed on a portion of the huge page. | |
| 2103 | + */ | |
| 2104 | + if (end != start + HPAGE_SIZE) { | |
| 2105 | + spin_unlock(ptl); | |
| 2106 | + split_huge_pmd(vma, pmd, start); | |
| 2107 | + pagemap_scan_backout_range(p, start, end); | |
| 2108 | + /* Report as if there was no THP */ | |
| 2109 | + return -ENOENT; | |
| 2110 | + } | |
| 2111 | + | |
| 2112 | + make_uffd_wp_pmd(vma, start, pmd); | |
| 2113 | + flush_tlb_range(vma, start, end); | |
| 2114 | +out_unlock: | |
| 2115 | + spin_unlock(ptl); | |
| 2116 | + return ret; | |
| 2117 | +#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ | |
| 2118 | + return -ENOENT; | |
| 2119 | +#endif | |
| 2120 | +} | |
| 2121 | + | |
| 2122 | +static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, | |
| 2123 | + unsigned long end, struct mm_walk *walk) | |
| 2124 | +{ | |
| 2125 | + struct pagemap_scan_private *p = walk->private; | |
| 2126 | + struct vm_area_struct *vma = walk->vma; | |
| 2127 | + unsigned long addr, flush_end = 0; | |
| 2128 | + pte_t *pte, *start_pte; | |
| 2129 | + spinlock_t *ptl; | |
| 2130 | + int ret; | |
| 2131 | + | |
| 2132 | + arch_enter_lazy_mmu_mode(); | |
| 2133 | + | |
| 2134 | + ret = pagemap_scan_thp_entry(pmd, start, end, walk); | |
| 2135 | + if (ret != -ENOENT) { | |
| 2136 | + arch_leave_lazy_mmu_mode(); | |
| 2137 | + return ret; | |
| 2138 | + } | |
| 2139 | + | |
| 2140 | + ret = 0; | |
| 2141 | + start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl); | |
| 2142 | + if (!pte) { | |
| 2143 | + arch_leave_lazy_mmu_mode(); | |
| 2144 | + walk->action = ACTION_AGAIN; | |
| 2145 | + return 0; | |
| 2146 | + } | |
| 2147 | + | |
| 2148 | + if (!p->vec_out) { | |
| 2149 | + /* Fast path for performing exclusive WP */ | |
| 2150 | + for (addr = start; addr != end; pte++, addr += PAGE_SIZE) { | |
| 2151 | + if (pte_uffd_wp(ptep_get(pte))) | |
| 2152 | + continue; | |
| 2153 | + make_uffd_wp_pte(vma, addr, pte); | |
| 2154 | + if (!flush_end) | |
| 2155 | + start = addr; | |
| 2156 | + flush_end = addr + PAGE_SIZE; | |
| 2157 | + } | |
| 2158 | + goto flush_and_return; | |
| 2159 | + } | |
| 2160 | + | |
| 2161 | + if (!p->arg.category_anyof_mask && !p->arg.category_inverted && | |
| 2162 | + p->arg.category_mask == PAGE_IS_WRITTEN && | |
| 2163 | + p->arg.return_mask == PAGE_IS_WRITTEN) { | |
| 2164 | + for (addr = start; addr < end; pte++, addr += PAGE_SIZE) { | |
| 2165 | + unsigned long next = addr + PAGE_SIZE; | |
| 2166 | + | |
| 2167 | + if (pte_uffd_wp(ptep_get(pte))) | |
| 2168 | + continue; | |
| 2169 | + ret = pagemap_scan_output(p->cur_vma_category | PAGE_IS_WRITTEN, | |
| 2170 | + p, addr, &next); | |
| 2171 | + if (next == addr) | |
| 2172 | + break; | |
| 2173 | + if (~p->arg.flags & PM_SCAN_WP_MATCHING) | |
| 2174 | + continue; | |
| 2175 | + make_uffd_wp_pte(vma, addr, pte); | |
| 2176 | + if (!flush_end) | |
| 2177 | + start = addr; | |
| 2178 | + flush_end = next; | |
| 2179 | + } | |
| 2180 | + goto flush_and_return; | |
| 2181 | + } | |
| 2182 | + | |
| 2183 | + for (addr = start; addr != end; pte++, addr += PAGE_SIZE) { | |
| 2184 | + unsigned long categories = p->cur_vma_category | | |
| 2185 | + pagemap_page_category(p, vma, addr, ptep_get(pte)); | |
| 2186 | + unsigned long next = addr + PAGE_SIZE; | |
| 2187 | + | |
| 2188 | + if (!pagemap_scan_is_interesting_page(categories, p)) | |
| 2189 | + continue; | |
| 2190 | + | |
| 2191 | + ret = pagemap_scan_output(categories, p, addr, &next); | |
| 2192 | + if (next == addr) | |
| 2193 | + break; | |
| 2194 | + | |
| 2195 | + if (~p->arg.flags & PM_SCAN_WP_MATCHING) | |
| 2196 | + continue; | |
| 2197 | + if (~categories & PAGE_IS_WRITTEN) | |
| 2198 | + continue; | |
| 2199 | + | |
| 2200 | + make_uffd_wp_pte(vma, addr, pte); | |
| 2201 | + if (!flush_end) | |
| 2202 | + start = addr; | |
| 2203 | + flush_end = next; | |
| 2204 | + } | |
| 2205 | + | |
| 2206 | +flush_and_return: | |
| 2207 | + if (flush_end) | |
| 2208 | + flush_tlb_range(vma, start, addr); | |
| 2209 | + | |
| 2210 | + pte_unmap_unlock(start_pte, ptl); | |
| 2211 | + arch_leave_lazy_mmu_mode(); | |
| 2212 | + | |
| 2213 | + cond_resched(); | |
| 2214 | + return ret; | |
| 2215 | +} | |
| 2216 | + | |
| 2217 | +#ifdef CONFIG_HUGETLB_PAGE | |
| 2218 | +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, | |
| 2219 | + unsigned long start, unsigned long end, | |
| 2220 | + struct mm_walk *walk) | |
| 2221 | +{ | |
| 2222 | + struct pagemap_scan_private *p = walk->private; | |
| 2223 | + struct vm_area_struct *vma = walk->vma; | |
| 2224 | + unsigned long categories; | |
| 2225 | + spinlock_t *ptl; | |
| 2226 | + int ret = 0; | |
| 2227 | + pte_t pte; | |
| 2228 | + | |
| 2229 | + if (~p->arg.flags & PM_SCAN_WP_MATCHING) { | |
| 2230 | + /* Go the short route when not write-protecting pages. */ | |
| 2231 | + | |
| 2232 | + pte = huge_ptep_get(ptep); | |
| 2233 | + categories = p->cur_vma_category | pagemap_hugetlb_category(pte); | |
| 2234 | + | |
| 2235 | + if (!pagemap_scan_is_interesting_page(categories, p)) | |
| 2236 | + return 0; | |
| 2237 | + | |
| 2238 | + return pagemap_scan_output(categories, p, start, &end); | |
| 2239 | + } | |
| 2240 | + | |
| 2241 | + i_mmap_lock_write(vma->vm_file->f_mapping); | |
| 2242 | + ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep); | |
| 2243 | + | |
| 2244 | + pte = huge_ptep_get(ptep); | |
| 2245 | + categories = p->cur_vma_category | pagemap_hugetlb_category(pte); | |
| 2246 | + | |
| 2247 | + if (!pagemap_scan_is_interesting_page(categories, p)) | |
| 2248 | + goto out_unlock; | |
| 2249 | + | |
| 2250 | + ret = pagemap_scan_output(categories, p, start, &end); | |
| 2251 | + if (start == end) | |
| 2252 | + goto out_unlock; | |
| 2253 | + | |
| 2254 | + if (~categories & PAGE_IS_WRITTEN) | |
| 2255 | + goto out_unlock; | |
| 2256 | + | |
| 2257 | + if (end != start + HPAGE_SIZE) { | |
| 2258 | + /* Partial HugeTLB page WP isn't possible. */ | |
| 2259 | + pagemap_scan_backout_range(p, start, end); | |
| 2260 | + p->arg.walk_end = start; | |
| 2261 | + ret = 0; | |
| 2262 | + goto out_unlock; | |
| 2263 | + } | |
| 2264 | + | |
| 2265 | + make_uffd_wp_huge_pte(vma, start, ptep, pte); | |
| 2266 | + flush_hugetlb_tlb_range(vma, start, end); | |
| 2267 | + | |
| 2268 | +out_unlock: | |
| 2269 | + spin_unlock(ptl); | |
| 2270 | + i_mmap_unlock_write(vma->vm_file->f_mapping); | |
| 2271 | + | |
| 2272 | + return ret; | |
| 2273 | +} | |
| 2274 | +#else | |
| 2275 | +#define pagemap_scan_hugetlb_entry NULL | |
| 2276 | +#endif | |
| 2277 | + | |
| 2278 | +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, | |
| 2279 | + int depth, struct mm_walk *walk) | |
| 2280 | +{ | |
| 2281 | + struct pagemap_scan_private *p = walk->private; | |
| 2282 | + struct vm_area_struct *vma = walk->vma; | |
| 2283 | + int ret, err; | |
| 2284 | + | |
| 2285 | + if (!vma || !pagemap_scan_is_interesting_page(p->cur_vma_category, p)) | |
| 2286 | + return 0; | |
| 2287 | + | |
| 2288 | + ret = pagemap_scan_output(p->cur_vma_category, p, addr, &end); | |
| 2289 | + if (addr == end) | |
| 2290 | + return ret; | |
| 2291 | + | |
| 2292 | + if (~p->arg.flags & PM_SCAN_WP_MATCHING) | |
| 2293 | + return ret; | |
| 2294 | + | |
| 2295 | + err = uffd_wp_range(vma, addr, end - addr, true); | |
| 2296 | + if (err < 0) | |
| 2297 | + ret = err; | |
| 2298 | + | |
| 2299 | + return ret; | |
| 2300 | +} | |
| 2301 | + | |
| 2302 | +static const struct mm_walk_ops pagemap_scan_ops = { | |
| 2303 | + .test_walk = pagemap_scan_test_walk, | |
| 2304 | + .pmd_entry = pagemap_scan_pmd_entry, | |
| 2305 | + .pte_hole = pagemap_scan_pte_hole, | |
| 2306 | + .hugetlb_entry = pagemap_scan_hugetlb_entry, | |
| 2307 | +}; | |
| 2308 | + | |
| 2309 | +static int pagemap_scan_get_args(struct pm_scan_arg *arg, | |
| 2310 | + unsigned long uarg) | |
| 2311 | +{ | |
| 2312 | + if (copy_from_user(arg, (void __user *)uarg, sizeof(*arg))) | |
| 2313 | + return -EFAULT; | |
| 2314 | + | |
| 2315 | + if (arg->size != sizeof(struct pm_scan_arg)) | |
| 2316 | + return -EINVAL; | |
| 2317 | + | |
| 2318 | + /* Validate requested features */ | |
| 2319 | + if (arg->flags & ~PM_SCAN_FLAGS) | |
| 2320 | + return -EINVAL; | |
| 2321 | + if ((arg->category_inverted | arg->category_mask | | |
| 2322 | + arg->category_anyof_mask | arg->return_mask) & ~PM_SCAN_CATEGORIES) | |
| 2323 | + return -EINVAL; | |
| 2324 | + | |
| 2325 | + arg->start = untagged_addr((unsigned long)arg->start); | |
| 2326 | + arg->end = untagged_addr((unsigned long)arg->end); | |
| 2327 | + arg->vec = untagged_addr((unsigned long)arg->vec); | |
| 2328 | + | |
| 2329 | + /* Validate memory pointers */ | |
| 2330 | + if (!IS_ALIGNED(arg->start, PAGE_SIZE)) | |
| 2331 | + return -EINVAL; | |
| 2332 | + if (!access_ok((void __user *)(long)arg->start, arg->end - arg->start)) | |
| 2333 | + return -EFAULT; | |
| 2334 | + if (!arg->vec && arg->vec_len) | |
| 2335 | + return -EINVAL; | |
| 2336 | + if (arg->vec && !access_ok((void __user *)(long)arg->vec, | |
| 2337 | + arg->vec_len * sizeof(struct page_region))) | |
| 2338 | + return -EFAULT; | |
| 2339 | + | |
| 2340 | + /* Fixup default values */ | |
| 2341 | + arg->end = ALIGN(arg->end, PAGE_SIZE); | |
| 2342 | + arg->walk_end = 0; | |
| 2343 | + if (!arg->max_pages) | |
| 2344 | + arg->max_pages = ULONG_MAX; | |
| 2345 | + | |
| 2346 | + return 0; | |
| 2347 | +} | |
| 2348 | + | |
| 2349 | +static int pagemap_scan_writeback_args(struct pm_scan_arg *arg, | |
| 2350 | + unsigned long uargl) | |
| 2351 | +{ | |
| 2352 | + struct pm_scan_arg __user *uarg = (void __user *)uargl; | |
| 2353 | + | |
| 2354 | + if (copy_to_user(&uarg->walk_end, &arg->walk_end, sizeof(arg->walk_end))) | |
| 2355 | + return -EFAULT; | |
| 2356 | + | |
| 2357 | + return 0; | |
| 2358 | +} | |
| 2359 | + | |
| 2360 | +static int pagemap_scan_init_bounce_buffer(struct pagemap_scan_private *p) | |
| 2361 | +{ | |
| 2362 | + if (!p->arg.vec_len) | |
| 2363 | + return 0; | |
| 2364 | + | |
| 2365 | + p->vec_buf_len = min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT, | |
| 2366 | + p->arg.vec_len); | |
| 2367 | + p->vec_buf = kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf), | |
| 2368 | + GFP_KERNEL); | |
| 2369 | + if (!p->vec_buf) | |
| 2370 | + return -ENOMEM; | |
| 2371 | + | |
| 2372 | + p->vec_buf->start = p->vec_buf->end = 0; | |
| 2373 | + p->vec_out = (struct page_region __user *)(long)p->arg.vec; | |
| 2374 | + | |
| 2375 | + return 0; | |
| 2376 | +} | |
| 2377 | + | |
| 2378 | +static long pagemap_scan_flush_buffer(struct pagemap_scan_private *p) | |
| 2379 | +{ | |
| 2380 | + const struct page_region *buf = p->vec_buf; | |
| 2381 | + long n = p->vec_buf_index; | |
| 2382 | + | |
| 2383 | + if (!p->vec_buf) | |
| 2384 | + return 0; | |
| 2385 | + | |
| 2386 | + if (buf[n].end != buf[n].start) | |
| 2387 | + n++; | |
| 2388 | + | |
| 2389 | + if (!n) | |
| 2390 | + return 0; | |
| 2391 | + | |
| 2392 | + if (copy_to_user(p->vec_out, buf, n * sizeof(*buf))) | |
| 2393 | + return -EFAULT; | |
| 2394 | + | |
| 2395 | + p->arg.vec_len -= n; | |
| 2396 | + p->vec_out += n; | |
| 2397 | + | |
| 2398 | + p->vec_buf_index = 0; | |
| 2399 | + p->vec_buf_len = min_t(size_t, p->vec_buf_len, p->arg.vec_len); | |
| 2400 | + p->vec_buf->start = p->vec_buf->end = 0; | |
| 2401 | + | |
| 2402 | + return n; | |
| 2403 | +} | |
| 2404 | + | |
| 2405 | +static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) | |
| 2406 | +{ | |
| 2407 | + struct mmu_notifier_range range; | |
| 2408 | + struct pagemap_scan_private p = {0}; | |
| 2409 | + unsigned long walk_start; | |
| 2410 | + size_t n_ranges_out = 0; | |
| 2411 | + int ret; | |
| 2412 | + | |
| 2413 | + ret = pagemap_scan_get_args(&p.arg, uarg); | |
| 2414 | + if (ret) | |
| 2415 | + return ret; | |
| 2416 | + | |
| 2417 | + p.masks_of_interest = p.arg.category_mask | p.arg.category_anyof_mask | | |
| 2418 | + p.arg.return_mask; | |
| 2419 | + ret = pagemap_scan_init_bounce_buffer(&p); | |
| 2420 | + if (ret) | |
| 2421 | + return ret; | |
| 2422 | + | |
| 2423 | + /* Protection change for the range is going to happen. */ | |
| 2424 | + if (p.arg.flags & PM_SCAN_WP_MATCHING) { | |
| 2425 | + mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, | |
| 2426 | + mm, p.arg.start, p.arg.end); | |
| 2427 | + mmu_notifier_invalidate_range_start(&range); | |
| 2428 | + } | |
| 2429 | + | |
| 2430 | + for (walk_start = p.arg.start; walk_start < p.arg.end; | |
| 2431 | + walk_start = p.arg.walk_end) { | |
| 2432 | + long n_out; | |
| 2433 | + | |
| 2434 | + if (fatal_signal_pending(current)) { | |
| 2435 | + ret = -EINTR; | |
| 2436 | + break; | |
| 2437 | + } | |
| 2438 | + | |
| 2439 | + ret = mmap_read_lock_killable(mm); | |
| 2440 | + if (ret) | |
| 2441 | + break; | |
| 2442 | + ret = walk_page_range(mm, walk_start, p.arg.end, | |
| 2443 | + &pagemap_scan_ops, &p); | |
| 2444 | + mmap_read_unlock(mm); | |
| 2445 | + | |
| 2446 | + n_out = pagemap_scan_flush_buffer(&p); | |
| 2447 | + if (n_out < 0) | |
| 2448 | + ret = n_out; | |
| 2449 | + else | |
| 2450 | + n_ranges_out += n_out; | |
| 2451 | + | |
| 2452 | + if (ret != -ENOSPC) | |
| 2453 | + break; | |
| 2454 | + | |
| 2455 | + if (p.arg.vec_len == 0 || p.found_pages == p.arg.max_pages) | |
| 2456 | + break; | |
| 2457 | + } | |
| 2458 | + | |
| 2459 | + /* ENOSPC signifies early stop (buffer full) from the walk. */ | |
| 2460 | + if (!ret || ret == -ENOSPC) | |
| 2461 | + ret = n_ranges_out; | |
| 2462 | + | |
| 2463 | + /* The walk_end isn't set when ret is zero */ | |
| 2464 | + if (!p.arg.walk_end) | |
| 2465 | + p.arg.walk_end = p.arg.end; | |
| 2466 | + if (pagemap_scan_writeback_args(&p.arg, uarg)) | |
| 2467 | + ret = -EFAULT; | |
| 2468 | + | |
| 2469 | + if (p.arg.flags & PM_SCAN_WP_MATCHING) | |
| 2470 | + mmu_notifier_invalidate_range_end(&range); | |
| 2471 | + | |
| 2472 | + kfree(p.vec_buf); | |
| 2473 | + return ret; | |
| 2474 | +} | |
| 2475 | + | |
| 2476 | +static long do_pagemap_cmd(struct file *file, unsigned int cmd, | |
| 2477 | + unsigned long arg) | |
| 2478 | +{ | |
| 2479 | + struct mm_struct *mm = file->private_data; | |
| 2480 | + | |
| 2481 | + switch (cmd) { | |
| 2482 | + case PAGEMAP_SCAN: | |
| 2483 | + return do_pagemap_scan(mm, arg); | |
| 2484 | + | |
| 2485 | + default: | |
| 2486 | + return -EINVAL; | |
| 2487 | + } | |
| 2488 | +} | |
| 2489 | + | |
| 1764 | 2490 | const struct file_operations proc_pagemap_operations = { |
| 1765 | 2491 | .llseek = mem_lseek, /* borrow this */ |
| 1766 | 2492 | .read = pagemap_read, |
| 1767 | 2493 | .open = pagemap_open, |
| 1768 | 2494 | .release = pagemap_release, |
| 2495 | + .unlocked_ioctl = do_pagemap_cmd, | |
| 2496 | + .compat_ioctl = do_pagemap_cmd, | |
| 1769 | 2497 | }; |
| 1770 | 2498 | #endif /* CONFIG_PROC_PAGE_MONITOR */ |
| 1771 | 2499 | |
| @@ -1945,8 +2673,9 @@ static int show_numa_map(struct seq_file *m, void *v) · 173 unmodified lines | ||
| 1945 | 2673 | struct numa_maps *md = &numa_priv->md; |
| 1946 | 2674 | struct file *file = vma->vm_file; |
| 1947 | 2675 | struct mm_struct *mm = vma->vm_mm; |
| 1948 | − struct mempolicy *pol; | |
| 1949 | 2676 | char buffer[64]; |
| 2677 | + struct mempolicy *pol; | |
| 2678 | + pgoff_t ilx; | |
| 1950 | 2679 | int nid; |
| 1951 | 2680 | |
| 1952 | 2681 | if (!mm) |
| @@ -1955,7 +2684,7 @@ static int show_numa_map(struct seq_file *m, void *v) · 2 unmodified lines | ||
| 1955 | 2684 | /* Ensure we start with an empty set of numa_maps statistics. */ |
| 1956 | 2685 | memset(md, 0, sizeof(*md)); |
| 1957 | 2686 | |
| 1958 | − pol = __get_vma_policy(vma, vma->vm_start); | |
| 2687 | + pol = __get_vma_policy(vma, vma->vm_start, &ilx); | |
| 1959 | 2688 | if (pol) { |
| 1960 | 2689 | mpol_to_str(buffer, sizeof(buffer), pol); |
| 1961 | 2690 | mpol_cond_put(pol); |
| @@ -803,12 +803,6 @@ dqcache_shrink_count(struct shrinker *shrink, struct shrink_control *sc) · 802 unmodified lines | ||
| 803 | 803 | percpu_counter_read_positive(&dqstats.counter[DQST_FREE_DQUOTS])); |
| 804 | 804 | } |
| 805 | 805 | |
| 806 | −static struct shrinker dqcache_shrinker = { | |
| 807 | − .count_objects = dqcache_shrink_count, | |
| 808 | − .scan_objects = dqcache_shrink_scan, | |
| 809 | − .seeks = DEFAULT_SEEKS, | |
| 810 | −}; | |
| 811 | − | |
| 812 | 806 | /* |
| 813 | 807 | * Safely release dquot and put reference to dquot. |
| 814 | 808 | */ |
| @@ -2982,6 +2976,7 @@ static int __init dquot_init(void) · 2167 unmodified lines | ||
| 2982 | 2976 | { |
| 2983 | 2977 | int i, ret; |
| 2984 | 2978 | unsigned long nr_hash, order; |
| 2979 | + struct shrinker *dqcache_shrinker; | |
| 2985 | 2980 | |
| 2986 | 2981 | printk(KERN_NOTICE "VFS: Disk quotas %s\n", __DQUOT_VERSION__); |
| 2987 | 2982 | |
| @@ -3016,8 +3011,14 @@ static int __init dquot_init(void) · 28 unmodified lines | ||
| 3016 | 3011 | pr_info("VFS: Dquot-cache hash table entries: %ld (order %ld," |
| 3017 | 3012 | " %ld bytes)\n", nr_hash, order, (PAGE_SIZE << order)); |
| 3018 | 3013 | |
| 3019 | − if (register_shrinker(&dqcache_shrinker, "dquota-cache")) | |
| 3020 | − panic("Cannot register dquot shrinker"); | |
| 3014 | + dqcache_shrinker = shrinker_alloc(0, "dquota-cache"); | |
| 3015 | + if (!dqcache_shrinker) | |
| 3016 | + panic("Cannot allocate dquot shrinker"); | |
| 3017 | + | |
| 3018 | + dqcache_shrinker->count_objects = dqcache_shrink_count; | |
| 3019 | + dqcache_shrinker->scan_objects = dqcache_shrink_scan; | |
| 3020 | + | |
| 3021 | + shrinker_register(dqcache_shrinker); | |
| 3021 | 3022 | |
| 3022 | 3023 | return 0; |
| 3023 | 3024 | } |
| @@ -2503,10 +2503,10 @@ out: · 2502 unmodified lines | ||
| 2503 | 2503 | * start/recovery path as __block_write_full_folio, along with special |
| 2504 | 2504 | * code to handle reiserfs tails. |
| 2505 | 2505 | */ |
| 2506 | −static int reiserfs_write_full_page(struct page *page, | |
| 2506 | +static int reiserfs_write_full_folio(struct folio *folio, | |
| 2507 | 2507 | struct writeback_control *wbc) |
| 2508 | 2508 | { |
| 2509 | − struct inode *inode = page->mapping->host; | |
| 2509 | + struct inode *inode = folio->mapping->host; | |
| 2510 | 2510 | unsigned long end_index = inode->i_size >> PAGE_SHIFT; |
| 2511 | 2511 | int error = 0; |
| 2512 | 2512 | unsigned long block; |
| @@ -2514,7 +2514,7 @@ static int reiserfs_write_full_page(struct page *page, · 1 unmodified line | ||
| 2514 | 2514 | struct buffer_head *head, *bh; |
| 2515 | 2515 | int partial = 0; |
| 2516 | 2516 | int nr = 0; |
| 2517 | − int checked = PageChecked(page); | |
| 2517 | + int checked = folio_test_checked(folio); | |
| 2518 | 2518 | struct reiserfs_transaction_handle th; |
| 2519 | 2519 | struct super_block *s = inode->i_sb; |
| 2520 | 2520 | int bh_per_page = PAGE_SIZE / s->s_blocksize; |
| @@ -2522,47 +2522,46 @@ static int reiserfs_write_full_page(struct page *page, · 1 unmodified line | ||
| 2522 | 2522 | |
| 2523 | 2523 | /* no logging allowed when nonblocking or from PF_MEMALLOC */ |
| 2524 | 2524 | if (checked && (current->flags & PF_MEMALLOC)) { |
| 2525 | − redirty_page_for_writepage(wbc, page); | |
| 2526 | − unlock_page(page); | |
| 2525 | + folio_redirty_for_writepage(wbc, folio); | |
| 2526 | + folio_unlock(folio); | |
| 2527 | 2527 | return 0; |
| 2528 | 2528 | } |
| 2529 | 2529 | |
| 2530 | 2530 | /* |
| 2531 | − * The page dirty bit is cleared before writepage is called, which | |
| 2531 | + * The folio dirty bit is cleared before writepage is called, which | |
| 2532 | 2532 | * means we have to tell create_empty_buffers to make dirty buffers |
| 2533 | − * The page really should be up to date at this point, so tossing | |
| 2533 | + * The folio really should be up to date at this point, so tossing | |
| 2534 | 2534 | * in the BH_Uptodate is just a sanity check. |
| 2535 | 2535 | */ |
| 2536 | − if (!page_has_buffers(page)) { | |
| 2537 | − create_empty_buffers(page, s->s_blocksize, | |
| 2536 | + head = folio_buffers(folio); | |
| 2537 | + if (!head) | |
| 2538 | + head = create_empty_buffers(folio, s->s_blocksize, | |
| 2538 | 2539 | (1 << BH_Dirty) | (1 << BH_Uptodate)); |
| 2539 | − } | |
| 2540 | − head = page_buffers(page); | |
| 2541 | 2540 | |
| 2542 | 2541 | /* |
| 2543 | − * last page in the file, zero out any contents past the | |
| 2542 | + * last folio in the file, zero out any contents past the | |
| 2544 | 2543 | * last byte in the file |
| 2545 | 2544 | */ |
| 2546 | − if (page->index >= end_index) { | |
| 2545 | + if (folio->index >= end_index) { | |
| 2547 | 2546 | unsigned last_offset; |
| 2548 | 2547 | |
| 2549 | 2548 | last_offset = inode->i_size & (PAGE_SIZE - 1); |
| 2550 | − /* no file contents in this page */ | |
| 2551 | − if (page->index >= end_index + 1 || !last_offset) { | |
| 2552 | − unlock_page(page); | |
| 2549 | + /* no file contents in this folio */ | |
| 2550 | + if (folio->index >= end_index + 1 || !last_offset) { | |
| 2551 | + folio_unlock(folio); | |
| 2553 | 2552 | return 0; |
| 2554 | 2553 | } |
| 2555 | − zero_user_segment(page, last_offset, PAGE_SIZE); | |
| 2554 | + folio_zero_segment(folio, last_offset, folio_size(folio)); | |
| 2556 | 2555 | } |
| 2557 | 2556 | bh = head; |
| 2558 | − block = page->index << (PAGE_SHIFT - s->s_blocksize_bits); | |
| 2557 | + block = folio->index << (PAGE_SHIFT - s->s_blocksize_bits); | |
| 2559 | 2558 | last_block = (i_size_read(inode) - 1) >> inode->i_blkbits; |
| 2560 | 2559 | /* first map all the buffers, logging any direct items we find */ |
| 2561 | 2560 | do { |
| 2562 | 2561 | if (block > last_block) { |
| 2563 | 2562 | /* |
| 2564 | 2563 | * This can happen when the block size is less than |
| 2565 | − * the page size. The corresponding bytes in the page | |
| 2564 | + * the folio size. The corresponding bytes in the folio | |
| 2566 | 2565 | * were zero filled above |
| 2567 | 2566 | */ |
| 2568 | 2567 | clear_buffer_dirty(bh); |
| @@ -2589,7 +2588,7 @@ static int reiserfs_write_full_page(struct page *page, · 20 unmodified lines | ||
| 2589 | 2588 | * blocks we're going to log |
| 2590 | 2589 | */ |
| 2591 | 2590 | if (checked) { |
| 2592 | − ClearPageChecked(page); | |
| 2591 | + folio_clear_checked(folio); | |
| 2593 | 2592 | reiserfs_write_lock(s); |
| 2594 | 2593 | error = journal_begin(&th, s, bh_per_page + 1); |
| 2595 | 2594 | if (error) { |
| @@ -2598,7 +2597,7 @@ static int reiserfs_write_full_page(struct page *page, · 2 unmodified lines | ||
| 2598 | 2597 | } |
| 2599 | 2598 | reiserfs_update_inode_transaction(inode); |
| 2600 | 2599 | } |
| 2601 | − /* now go through and lock any dirty buffers on the page */ | |
| 2600 | + /* now go through and lock any dirty buffers on the folio */ | |
| 2602 | 2601 | do { |
| 2603 | 2602 | get_bh(bh); |
| 2604 | 2603 | if (!buffer_mapped(bh)) |
| @@ -2619,7 +2618,7 @@ static int reiserfs_write_full_page(struct page *page, · 14 unmodified lines | ||
| 2619 | 2618 | lock_buffer(bh); |
| 2620 | 2619 | } else { |
| 2621 | 2620 | if (!trylock_buffer(bh)) { |
| 2622 | − redirty_page_for_writepage(wbc, page); | |
| 2621 | + folio_redirty_for_writepage(wbc, folio); | |
| 2623 | 2622 | continue; |
| 2624 | 2623 | } |
| 2625 | 2624 | } |
| @@ -2636,13 +2635,13 @@ static int reiserfs_write_full_page(struct page *page, · 10 unmodified lines | ||
| 2636 | 2635 | if (error) |
| 2637 | 2636 | goto fail; |
| 2638 | 2637 | } |
| 2639 | − BUG_ON(PageWriteback(page)); | |
| 2640 | − set_page_writeback(page); | |
| 2641 | − unlock_page(page); | |
| 2638 | + BUG_ON(folio_test_writeback(folio)); | |
| 2639 | + folio_start_writeback(folio); | |
| 2640 | + folio_unlock(folio); | |
| 2642 | 2641 | |
| 2643 | 2642 | /* |
| 2644 | − * since any buffer might be the only dirty buffer on the page, | |
| 2645 | − * the first submit_bh can bring the page out of writeback. | |
| 2643 | + * since any buffer might be the only dirty buffer on the folio, | |
| 2644 | + * the first submit_bh can bring the folio out of writeback. | |
| 2646 | 2645 | * be careful with the buffers. |
| 2647 | 2646 | */ |
| 2648 | 2647 | do { |
| @@ -2659,10 +2658,10 @@ static int reiserfs_write_full_page(struct page *page, · 10 unmodified lines | ||
| 2659 | 2658 | done: |
| 2660 | 2659 | if (nr == 0) { |
| 2661 | 2660 | /* |
| 2662 | − * if this page only had a direct item, it is very possible for | |
| 2661 | + * if this folio only had a direct item, it is very possible for | |
| 2663 | 2662 | * no io to be required without there being an error. Or, |
| 2664 | 2663 | * someone else could have locked them and sent them down the |
| 2665 | − * pipe without locking the page | |
| 2664 | + * pipe without locking the folio | |
| 2666 | 2665 | */ |
| 2667 | 2666 | bh = head; |
| 2668 | 2667 | do { |
| @@ -2673,18 +2672,18 @@ done: · 4 unmodified lines | ||
| 2673 | 2672 | bh = bh->b_this_page; |
| 2674 | 2673 | } while (bh != head); |
| 2675 | 2674 | if (!partial) |
| 2676 | − SetPageUptodate(page); | |
| 2677 | − end_page_writeback(page); | |
| 2675 | + folio_mark_uptodate(folio); | |
| 2676 | + folio_end_writeback(folio); | |
| 2678 | 2677 | } |
| 2679 | 2678 | return error; |
| 2680 | 2679 | |
| 2681 | 2680 | fail: |
| 2682 | 2681 | /* |
| 2683 | 2682 | * catches various errors, we need to make sure any valid dirty blocks |
| 2684 | − * get to the media. The page is currently locked and not marked for | |
| 2683 | + * get to the media. The folio is currently locked and not marked for | |
| 2685 | 2684 | * writeback |
| 2686 | 2685 | */ |
| 2687 | − ClearPageUptodate(page); | |
| 2686 | + folio_clear_uptodate(folio); | |
| 2688 | 2687 | bh = head; |
| 2689 | 2688 | do { |
| 2690 | 2689 | get_bh(bh); |
| @@ -2694,16 +2693,16 @@ fail: · 3 unmodified lines | ||
| 2694 | 2693 | } else { |
| 2695 | 2694 | /* |
| 2696 | 2695 | * clear any dirty bits that might have come from |
| 2697 | − * getting attached to a dirty page | |
| 2696 | + * getting attached to a dirty folio | |
| 2698 | 2697 | */ |
| 2699 | 2698 | clear_buffer_dirty(bh); |
| 2700 | 2699 | } |
| 2701 | 2700 | bh = bh->b_this_page; |
| 2702 | 2701 | } while (bh != head); |
| 2703 | − SetPageError(page); | |
| 2704 | − BUG_ON(PageWriteback(page)); | |
| 2705 | − set_page_writeback(page); | |
| 2706 | − unlock_page(page); | |
| 2702 | + folio_set_error(folio); | |
| 2703 | + BUG_ON(folio_test_writeback(folio)); | |
| 2704 | + folio_start_writeback(folio); | |
| 2705 | + folio_unlock(folio); | |
| 2707 | 2706 | do { |
| 2708 | 2707 | struct buffer_head *next = bh->b_this_page; |
| 2709 | 2708 | if (buffer_async_write(bh)) { |
| @@ -2724,9 +2723,10 @@ static int reiserfs_read_folio(struct file *f, struct folio *folio) · 14 unmodified lines | ||
| 2724 | 2723 | |
| 2725 | 2724 | static int reiserfs_writepage(struct page *page, struct writeback_control *wbc) |
| 2726 | 2725 | { |
| 2727 | − struct inode *inode = page->mapping->host; | |
| 2726 | + struct folio *folio = page_folio(page); | |
| 2727 | + struct inode *inode = folio->mapping->host; | |
| 2728 | 2728 | reiserfs_wait_on_write_block(inode->i_sb); |
| 2729 | − return reiserfs_write_full_page(page, wbc); | |
| 2729 | + return reiserfs_write_full_folio(folio, wbc); | |
| 2730 | 2730 | } |
| 2731 | 2731 | |
| 2732 | 2732 | static void reiserfs_truncate_failed_write(struct inode *inode) |
| @@ -178,7 +178,7 @@ static void super_wake(struct super_block *sb, unsigned int flag) · 177 unmodified lines | ||
| 178 | 178 | * One thing we have to be careful of with a per-sb shrinker is that we don't |
| 179 | 179 | * drop the last active reference to the superblock from within the shrinker. |
| 180 | 180 | * If that happens we could trigger unregistering the shrinker from within the |
| 181 | − * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we | |
| 181 | + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we | |
| 182 | 182 | * take a passive reference to the superblock to avoid this from occurring. |
| 183 | 183 | */ |
| 184 | 184 | static unsigned long super_cache_scan(struct shrinker *shrink, |
| @@ -191,7 +191,7 @@ static unsigned long super_cache_scan(struct shrinker *shrink, · 6 unmodified lines | ||
| 191 | 191 | long dentries; |
| 192 | 192 | long inodes; |
| 193 | 193 | |
| 194 | − sb = container_of(shrink, struct super_block, s_shrink); | |
| 194 | + sb = shrink->private_data; | |
| 195 | 195 | |
| 196 | 196 | /* |
| 197 | 197 | * Deadlock avoidance. We may hold various FS locks, and we don't want |
| @@ -244,7 +244,7 @@ static unsigned long super_cache_count(struct shrinker *shrink, · 46 unmodified lines | ||
| 244 | 244 | struct super_block *sb; |
| 245 | 245 | long total_objects = 0; |
| 246 | 246 | |
| 247 | − sb = container_of(shrink, struct super_block, s_shrink); | |
| 247 | + sb = shrink->private_data; | |
| 248 | 248 | |
| 249 | 249 | /* |
| 250 | 250 | * We don't call super_trylock_shared() here as it is a scalability |
| @@ -306,7 +306,7 @@ static void destroy_unused_super(struct super_block *s) · 55 unmodified lines | ||
| 306 | 306 | security_sb_free(s); |
| 307 | 307 | put_user_ns(s->s_user_ns); |
| 308 | 308 | kfree(s->s_subtype); |
| 309 | − free_prealloced_shrinker(&s->s_shrink); | |
| 309 | + shrinker_free(s->s_shrink); | |
| 310 | 310 | /* no delays needed */ |
| 311 | 311 | destroy_super_work(&s->destroy_work); |
| 312 | 312 | } |
| @@ -383,16 +383,19 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags, · 70 unmodified lines | ||
| 383 | 383 | s->s_time_min = TIME64_MIN; |
| 384 | 384 | s->s_time_max = TIME64_MAX; |
| 385 | 385 | |
| 386 | − s->s_shrink.seeks = DEFAULT_SEEKS; | |
| 387 | − s->s_shrink.scan_objects = super_cache_scan; | |
| 388 | − s->s_shrink.count_objects = super_cache_count; | |
| 389 | − s->s_shrink.batch = 1024; | |
| 390 | − s->s_shrink.flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE; | |
| 391 | − if (prealloc_shrinker(&s->s_shrink, "sb-%s", type->name)) | |
| 386 | + s->s_shrink = shrinker_alloc(SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE, | |
| 387 | + "sb-%s", type->name); | |
| 388 | + if (!s->s_shrink) | |
| 392 | 389 | goto fail; |
| 393 | − if (list_lru_init_memcg(&s->s_dentry_lru, &s->s_shrink)) | |
| 390 | + | |
| 391 | + s->s_shrink->scan_objects = super_cache_scan; | |
| 392 | + s->s_shrink->count_objects = super_cache_count; | |
| 393 | + s->s_shrink->batch = 1024; | |
| 394 | + s->s_shrink->private_data = s; | |
| 395 | + | |
| 396 | + if (list_lru_init_memcg(&s->s_dentry_lru, s->s_shrink)) | |
| 394 | 397 | goto fail; |
| 395 | − if (list_lru_init_memcg(&s->s_inode_lru, &s->s_shrink)) | |
| 398 | + if (list_lru_init_memcg(&s->s_inode_lru, s->s_shrink)) | |
| 396 | 399 | goto fail; |
| 397 | 400 | return s; |
| 398 | 401 | |
| @@ -477,7 +480,7 @@ void deactivate_locked_super(struct super_block *s) · 78 unmodified lines | ||
| 477 | 480 | { |
| 478 | 481 | struct file_system_type *fs = s->s_type; |
| 479 | 482 | if (atomic_dec_and_test(&s->s_active)) { |
| 480 | − unregister_shrinker(&s->s_shrink); | |
| 483 | + shrinker_free(s->s_shrink); | |
| 481 | 484 | fs->kill_sb(s); |
| 482 | 485 | |
| 483 | 486 | kill_super_notify(s); |
| @@ -818,7 +821,7 @@ retry: · 334 unmodified lines | ||
| 818 | 821 | hlist_add_head(&s->s_instances, &s->s_type->fs_supers); |
| 819 | 822 | spin_unlock(&sb_lock); |
| 820 | 823 | get_filesystem(s->s_type); |
| 821 | − register_shrinker_prepared(&s->s_shrink); | |
| 824 | + shrinker_register(s->s_shrink); | |
| 822 | 825 | return s; |
| 823 | 826 | |
| 824 | 827 | share_extant_sb: |
| @@ -901,7 +904,7 @@ retry: · 76 unmodified lines | ||
| 901 | 904 | hlist_add_head(&s->s_instances, &type->fs_supers); |
| 902 | 905 | spin_unlock(&sb_lock); |
| 903 | 906 | get_filesystem(type); |
| 904 | − register_shrinker_prepared(&s->s_shrink); | |
| 907 | + shrinker_register(s->s_shrink); | |
| 905 | 908 | return s; |
| 906 | 909 | } |
| 907 | 910 | EXPORT_SYMBOL(sget); |
| @@ -1540,7 +1543,7 @@ int setup_bdev_super(struct super_block *sb, int sb_flags, · 632 unmodified lines | ||
| 1540 | 1543 | mutex_unlock(&bdev->bd_fsfreeze_mutex); |
| 1541 | 1544 | |
| 1542 | 1545 | snprintf(sb->s_id, sizeof(sb->s_id), "%pg", bdev); |
| 1543 | − shrinker_debugfs_rename(&sb->s_shrink, "sb-%s:%s", sb->s_type->name, | |
| 1546 | + shrinker_debugfs_rename(sb->s_shrink, "sb-%s:%s", sb->s_type->name, | |
| 1544 | 1547 | sb->s_id); |
| 1545 | 1548 | sb_set_blocksize(sb, block_size(bdev)); |
| 1546 | 1549 | return 0; |
| @@ -54,11 +54,7 @@ module_param_cb(default_version, &ubifs_default_version_ops, &ubifs_default_vers · 53 unmodified lines | ||
| 54 | 54 | static struct kmem_cache *ubifs_inode_slab; |
| 55 | 55 | |
| 56 | 56 | /* UBIFS TNC shrinker description */ |
| 57 | −static struct shrinker ubifs_shrinker_info = { | |
| 58 | − .scan_objects = ubifs_shrink_scan, | |
| 59 | − .count_objects = ubifs_shrink_count, | |
| 60 | − .seeks = DEFAULT_SEEKS, | |
| 61 | −}; | |
| 57 | +static struct shrinker *ubifs_shrinker_info; | |
| 62 | 58 | |
| 63 | 59 | /** |
| 64 | 60 | * validate_inode - validate inode. |
| @@ -2373,7 +2369,7 @@ static void inode_slab_ctor(void *obj) · 2308 unmodified lines | ||
| 2373 | 2369 | |
| 2374 | 2370 | static int __init ubifs_init(void) |
| 2375 | 2371 | { |
| 2376 | − int err; | |
| 2372 | + int err = -ENOMEM; | |
| 2377 | 2373 | |
| 2378 | 2374 | BUILD_BUG_ON(sizeof(struct ubifs_ch) != 24); |
| 2379 | 2375 | |
| @@ -2439,10 +2435,15 @@ static int __init ubifs_init(void) · 59 unmodified lines | ||
| 2439 | 2435 | if (!ubifs_inode_slab) |
| 2440 | 2436 | return -ENOMEM; |
| 2441 | 2437 | |
| 2442 | − err = register_shrinker(&ubifs_shrinker_info, "ubifs-slab"); | |
| 2443 | − if (err) | |
| 2438 | + ubifs_shrinker_info = shrinker_alloc(0, "ubifs-slab"); | |
| 2439 | + if (!ubifs_shrinker_info) | |
| 2444 | 2440 | goto out_slab; |
| 2445 | 2441 | |
| 2442 | + ubifs_shrinker_info->count_objects = ubifs_shrink_count; | |
| 2443 | + ubifs_shrinker_info->scan_objects = ubifs_shrink_scan; | |
| 2444 | + | |
| 2445 | + shrinker_register(ubifs_shrinker_info); | |
| 2446 | + | |
| 2446 | 2447 | err = ubifs_compressors_init(); |
| 2447 | 2448 | if (err) |
| 2448 | 2449 | goto out_shrinker; |
| @@ -2467,7 +2468,7 @@ out_dbg: · 18 unmodified lines | ||
| 2467 | 2468 | dbg_debugfs_exit(); |
| 2468 | 2469 | ubifs_compressors_exit(); |
| 2469 | 2470 | out_shrinker: |
| 2470 | − unregister_shrinker(&ubifs_shrinker_info); | |
| 2471 | + shrinker_free(ubifs_shrinker_info); | |
| 2471 | 2472 | out_slab: |
| 2472 | 2473 | kmem_cache_destroy(ubifs_inode_slab); |
| 2473 | 2474 | return err; |
| @@ -2483,7 +2484,7 @@ static void __exit ubifs_exit(void) · 9 unmodified lines | ||
| 2483 | 2484 | dbg_debugfs_exit(); |
| 2484 | 2485 | ubifs_sysfs_exit(); |
| 2485 | 2486 | ubifs_compressors_exit(); |
| 2486 | − unregister_shrinker(&ubifs_shrinker_info); | |
| 2487 | + shrinker_free(ubifs_shrinker_info); | |
| 2487 | 2488 | |
| 2488 | 2489 | /* |
| 2489 | 2490 | * Make sure all delayed rcu free inodes are flushed before we |
| @@ -240,6 +240,7 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 239 unmodified lines | ||
| 240 | 240 | unsigned int count, sector_t oldb, |
| 241 | 241 | sector_t newb, struct page *locked_page) |
| 242 | 242 | { |
| 243 | + struct folio *folio, *locked_folio = page_folio(locked_page); | |
| 243 | 244 | const unsigned blks_per_page = |
| 244 | 245 | 1 << (PAGE_SHIFT - inode->i_blkbits); |
| 245 | 246 | const unsigned mask = blks_per_page - 1; |
| @@ -247,42 +248,39 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 1 unmodified line | ||
| 247 | 248 | pgoff_t index, cur_index, last_index; |
| 248 | 249 | unsigned pos, j, lblock; |
| 249 | 250 | sector_t end, i; |
| 250 | − struct page *page; | |
| 251 | 251 | struct buffer_head *head, *bh; |
| 252 | 252 | |
| 253 | 253 | UFSD("ENTER, ino %lu, count %u, oldb %llu, newb %llu\n", |
| 254 | 254 | inode->i_ino, count, |
| 255 | 255 | (unsigned long long)oldb, (unsigned long long)newb); |
| 256 | 256 | |
| 257 | − BUG_ON(!locked_page); | |
| 258 | − BUG_ON(!PageLocked(locked_page)); | |
| 257 | + BUG_ON(!folio_test_locked(locked_folio)); | |
| 259 | 258 | |
| 260 | − cur_index = locked_page->index; | |
| 259 | + cur_index = locked_folio->index; | |
| 261 | 260 | end = count + beg; |
| 262 | 261 | last_index = end >> (PAGE_SHIFT - inode->i_blkbits); |
| 263 | 262 | for (i = beg; i < end; i = (i | mask) + 1) { |
| 264 | 263 | index = i >> (PAGE_SHIFT - inode->i_blkbits); |
| 265 | 264 | |
| 266 | 265 | if (likely(cur_index != index)) { |
| 267 | − page = ufs_get_locked_page(mapping, index); | |
| 268 | − if (!page)/* it was truncated */ | |
| 266 | + folio = ufs_get_locked_folio(mapping, index); | |
| 267 | + if (!folio) /* it was truncated */ | |
| 269 | 268 | continue; |
| 270 | − if (IS_ERR(page)) {/* or EIO */ | |
| 269 | + if (IS_ERR(folio)) {/* or EIO */ | |
| 271 | 270 | ufs_error(inode->i_sb, __func__, |
| 272 | 271 | "read of page %llu failed\n", |
| 273 | 272 | (unsigned long long)index); |
| 274 | 273 | continue; |
| 275 | 274 | } |
| 276 | 275 | } else |
| 277 | − page = locked_page; | |
| 276 | + folio = locked_folio; | |
| 278 | 277 | |
| 279 | − head = page_buffers(page); | |
| 278 | + head = folio_buffers(folio); | |
| 280 | 279 | bh = head; |
| 281 | 280 | pos = i & mask; |
| 282 | 281 | for (j = 0; j < pos; ++j) |
| 283 | 282 | bh = bh->b_this_page; |
| 284 | 283 | |
| 285 | − | |
| 286 | 284 | if (unlikely(index == last_index)) |
| 287 | 285 | lblock = end & mask; |
| 288 | 286 | else |
| @@ -313,7 +311,7 @@ static void ufs_change_blocknr(struct inode *inode, sector_t beg, · 24 unmodified lines | ||
| 313 | 311 | } while (bh != head); |
| 314 | 312 | |
| 315 | 313 | if (likely(cur_index != index)) |
| 316 | − ufs_put_locked_page(page); | |
| 314 | + ufs_put_locked_folio(folio); | |
| 317 | 315 | } |
| 318 | 316 | UFSD("EXIT\n"); |
| 319 | 317 | } |
| @@ -1063,7 +1063,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 1062 unmodified lines | ||
| 1063 | 1063 | struct ufs_sb_private_info *uspi = UFS_SB(sb)->s_uspi; |
| 1064 | 1064 | unsigned i, end; |
| 1065 | 1065 | sector_t lastfrag; |
| 1066 | − struct page *lastpage; | |
| 1066 | + struct folio *folio; | |
| 1067 | 1067 | struct buffer_head *bh; |
| 1068 | 1068 | u64 phys64; |
| 1069 | 1069 | |
| @@ -1074,18 +1074,17 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 4 unmodified lines | ||
| 1074 | 1074 | |
| 1075 | 1075 | lastfrag--; |
| 1076 | 1076 | |
| 1077 | − lastpage = ufs_get_locked_page(mapping, lastfrag >> | |
| 1077 | + folio = ufs_get_locked_folio(mapping, lastfrag >> | |
| 1078 | 1078 | (PAGE_SHIFT - inode->i_blkbits)); |
| 1079 | − if (IS_ERR(lastpage)) { | |
| 1080 | − err = -EIO; | |
| 1081 | − goto out; | |
| 1082 | − } | |
| 1083 | − | |
| 1084 | − end = lastfrag & ((1 << (PAGE_SHIFT - inode->i_blkbits)) - 1); | |
| 1085 | − bh = page_buffers(lastpage); | |
| 1086 | − for (i = 0; i < end; ++i) | |
| 1087 | − bh = bh->b_this_page; | |
| 1079 | + if (IS_ERR(folio)) { | |
| 1080 | + err = -EIO; | |
| 1081 | + goto out; | |
| 1082 | + } | |
| 1088 | 1083 | |
| 1084 | + end = lastfrag & ((1 << (PAGE_SHIFT - inode->i_blkbits)) - 1); | |
| 1085 | + bh = folio_buffers(folio); | |
| 1086 | + for (i = 0; i < end; ++i) | |
| 1087 | + bh = bh->b_this_page; | |
| 1089 | 1088 | |
| 1090 | 1089 | err = ufs_getfrag_block(inode, lastfrag, bh, 1); |
| 1091 | 1090 | |
| @@ -1101,7 +1100,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 9 unmodified lines | ||
| 1101 | 1100 | */ |
| 1102 | 1101 | set_buffer_uptodate(bh); |
| 1103 | 1102 | mark_buffer_dirty(bh); |
| 1104 | − set_page_dirty(lastpage); | |
| 1103 | + folio_mark_dirty(folio); | |
| 1105 | 1104 | } |
| 1106 | 1105 | |
| 1107 | 1106 | if (lastfrag >= UFS_IND_FRAGMENT) { |
| @@ -1119,7 +1118,7 @@ static int ufs_alloc_lastblock(struct inode *inode, loff_t size) · 11 unmodified lines | ||
| 1119 | 1118 | } |
| 1120 | 1119 | } |
| 1121 | 1120 | out_unlock: |
| 1122 | − ufs_put_locked_page(lastpage); | |
| 1121 | + ufs_put_locked_folio(folio); | |
| 1123 | 1122 | out: |
| 1124 | 1123 | return err; |
| 1125 | 1124 | } |
| @@ -230,42 +230,40 @@ ufs_set_inode_dev(struct super_block *sb, struct ufs_inode_info *ufsi, dev_t dev · 229 unmodified lines | ||
| 230 | 230 | } |
| 231 | 231 | |
| 232 | 232 | /** |
| 233 | − * ufs_get_locked_page() - locate, pin and lock a pagecache page, if not exist | |
| 233 | + * ufs_get_locked_folio() - locate, pin and lock a pagecache folio, if not exist | |
| 234 | 234 | * read it from disk. |
| 235 | 235 | * @mapping: the address_space to search |
| 236 | 236 | * @index: the page index |
| 237 | 237 | * |
| 238 | − * Locates the desired pagecache page, if not exist we'll read it, | |
| 238 | + * Locates the desired pagecache folio, if not exist we'll read it, | |
| 239 | 239 | * locks it, increments its reference |
| 240 | 240 | * count and returns its address. |
| 241 | 241 | * |
| 242 | 242 | */ |
| 243 | − | |
| 244 | −struct page *ufs_get_locked_page(struct address_space *mapping, | |
| 243 | +struct folio *ufs_get_locked_folio(struct address_space *mapping, | |
| 245 | 244 | pgoff_t index) |
| 246 | 245 | { |
| 247 | 246 | struct inode *inode = mapping->host; |
| 248 | − struct page *page = find_lock_page(mapping, index); | |
| 249 | − if (!page) { | |
| 250 | − page = read_mapping_page(mapping, index, NULL); | |
| 247 | + struct folio *folio = filemap_lock_folio(mapping, index); | |
| 248 | + if (!folio) { | |
| 249 | + folio = read_mapping_folio(mapping, index, NULL); | |
| 251 | 250 | |
| 252 | − if (IS_ERR(page)) { | |
| 253 | − printk(KERN_ERR "ufs_change_blocknr: " | |
| 254 | − "read_mapping_page error: ino %lu, index: %lu\n", | |
| 251 | + if (IS_ERR(folio)) { | |
| 252 | + printk(KERN_ERR "ufs_change_blocknr: read_mapping_folio error: ino %lu, index: %lu\n", | |
| 255 | 253 | mapping->host->i_ino, index); |
| 256 | − return page; | |
| 254 | + return folio; | |
| 257 | 255 | } |
| 258 | 256 | |
| 259 | − lock_page(page); | |
| 257 | + folio_lock(folio); | |
| 260 | 258 | |
| 261 | − if (unlikely(page->mapping == NULL)) { | |
| 259 | + if (unlikely(folio->mapping == NULL)) { | |
| 262 | 260 | /* Truncate got there first */ |
| 263 | − unlock_page(page); | |
| 264 | − put_page(page); | |
| 261 | + folio_unlock(folio); | |
| 262 | + folio_put(folio); | |
| 265 | 263 | return NULL; |
| 266 | 264 | } |
| 267 | 265 | } |
| 268 | − if (!page_has_buffers(page)) | |
| 269 | − create_empty_buffers(page, 1 << inode->i_blkbits, 0); | |
| 270 | − return page; | |
| 266 | + if (!folio_buffers(folio)) | |
| 267 | + create_empty_buffers(folio, 1 << inode->i_blkbits, 0); | |
| 268 | + return folio; | |
| 271 | 269 | } |
| @@ -273,15 +273,13 @@ extern void _ubh_ubhcpymem_(struct ufs_sb_private_info *, unsigned char *, struc · 272 unmodified lines | ||
| 273 | 273 | extern void _ubh_memcpyubh_(struct ufs_sb_private_info *, struct ufs_buffer_head *, unsigned char *, unsigned); |
| 274 | 274 | |
| 275 | 275 | /* This functions works with cache pages*/ |
| 276 | −extern struct page *ufs_get_locked_page(struct address_space *mapping, | |
| 277 | − pgoff_t index); | |
| 278 | −static inline void ufs_put_locked_page(struct page *page) | |
| 276 | +struct folio *ufs_get_locked_folio(struct address_space *mapping, pgoff_t index); | |
| 277 | +static inline void ufs_put_locked_folio(struct folio *folio) | |
| 279 | 278 | { |
| 280 | − unlock_page(page); | |
| 281 | − put_page(page); | |
| 279 | + folio_unlock(folio); | |
| 280 | + folio_put(folio); | |
| 282 | 281 | } |
| 283 | 282 | |
| 284 | − | |
| 285 | 283 | /* |
| 286 | 284 | * macros and inline function to get important structures from ufs_sb_private_info |
| 287 | 285 | */ |
| @@ -123,6 +123,11 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx) · 122 unmodified lines | ||
| 123 | 123 | return ctx->features & UFFD_FEATURE_INITIALIZED; |
| 124 | 124 | } |
| 125 | 125 | |
| 126 | +static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx) | |
| 127 | +{ | |
| 128 | + return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC); | |
| 129 | +} | |
| 130 | + | |
| 126 | 131 | /* |
| 127 | 132 | * Whether WP_UNPOPULATED is enabled on the uffd context. It is only |
| 128 | 133 | * meaningful when userfaultfd_wp()==true on the vma and when it's |
| @@ -922,20 +927,15 @@ static int userfaultfd_release(struct inode *inode, struct file *file) · 793 unmodified lines | ||
| 922 | 927 | continue; |
| 923 | 928 | } |
| 924 | 929 | new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS; |
| 925 | − prev = vma_merge(&vmi, mm, prev, vma->vm_start, vma->vm_end, | |
| 926 | − new_flags, vma->anon_vma, | |
| 927 | − vma->vm_file, vma->vm_pgoff, | |
| 928 | − vma_policy(vma), | |
| 929 | − NULL_VM_UFFD_CTX, anon_vma_name(vma)); | |
| 930 | − if (prev) { | |
| 931 | − vma = prev; | |
| 932 | − } else { | |
| 933 | − prev = vma; | |
| 934 | − } | |
| 930 | + vma = vma_modify_flags_uffd(&vmi, prev, vma, vma->vm_start, | |
| 931 | + vma->vm_end, new_flags, | |
| 932 | + NULL_VM_UFFD_CTX); | |
| 935 | 933 | |
| 936 | 934 | vma_start_write(vma); |
| 937 | 935 | userfaultfd_set_vm_flags(vma, new_flags); |
| 938 | 936 | vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; |
| 937 | + | |
| 938 | + prev = vma; | |
| 939 | 939 | } |
| 940 | 940 | mmap_write_unlock(mm); |
| 941 | 941 | mmput(mm); |
| @@ -1325,7 +1325,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 383 unmodified lines | ||
| 1325 | 1325 | bool basic_ioctls; |
| 1326 | 1326 | unsigned long start, end, vma_end; |
| 1327 | 1327 | struct vma_iterator vmi; |
| 1328 | − pgoff_t pgoff; | |
| 1328 | + bool wp_async = userfaultfd_wp_async_ctx(ctx); | |
| 1329 | 1329 | |
| 1330 | 1330 | user_uffdio_register = (struct uffdio_register __user *) arg; |
| 1331 | 1331 | |
| @@ -1399,7 +1399,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 67 unmodified lines | ||
| 1399 | 1399 | |
| 1400 | 1400 | /* check not compatible vmas */ |
| 1401 | 1401 | ret = -EINVAL; |
| 1402 | − if (!vma_can_userfault(cur, vm_flags)) | |
| 1402 | + if (!vma_can_userfault(cur, vm_flags, wp_async)) | |
| 1403 | 1403 | goto out_unlock; |
| 1404 | 1404 | |
| 1405 | 1405 | /* |
| @@ -1460,7 +1460,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 54 unmodified lines | ||
| 1460 | 1460 | for_each_vma_range(vmi, vma, end) { |
| 1461 | 1461 | cond_resched(); |
| 1462 | 1462 | |
| 1463 | − BUG_ON(!vma_can_userfault(vma, vm_flags)); | |
| 1463 | + BUG_ON(!vma_can_userfault(vma, vm_flags, wp_async)); | |
| 1464 | 1464 | BUG_ON(vma->vm_userfaultfd_ctx.ctx && |
| 1465 | 1465 | vma->vm_userfaultfd_ctx.ctx != ctx); |
| 1466 | 1466 | WARN_ON(!(vma->vm_flags & VM_MAYWRITE)); |
| @@ -1478,28 +1478,14 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, · 11 unmodified lines | ||
| 1478 | 1478 | vma_end = min(end, vma->vm_end); |
| 1479 | 1479 | |
| 1480 | 1480 | new_flags = (vma->vm_flags & ~__VM_UFFD_FLAGS) | vm_flags; |
| 1481 | − pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT); | |
| 1482 | − prev = vma_merge(&vmi, mm, prev, start, vma_end, new_flags, | |
| 1483 | − vma->anon_vma, vma->vm_file, pgoff, | |
| 1484 | − vma_policy(vma), | |
| 1485 | − ((struct vm_userfaultfd_ctx){ ctx }), | |
| 1486 | − anon_vma_name(vma)); | |
| 1487 | − if (prev) { | |
| 1488 | − /* vma_merge() invalidated the mas */ | |
| 1489 | − vma = prev; | |
| 1490 | − goto next; | |
| 1491 | − } | |
| 1492 | − if (vma->vm_start < start) { | |
| 1493 | − ret = split_vma(&vmi, vma, start, 1); | |
| 1494 | − if (ret) | |
| 1495 | − break; | |
| 1496 | − } | |
| 1497 | − if (vma->vm_end > end) { | |
| 1498 | − ret = split_vma(&vmi, vma, end, 0); | |
| 1499 | − if (ret) | |
| 1500 | − break; | |
| 1481 | + vma = vma_modify_flags_uffd(&vmi, prev, vma, start, vma_end, | |
| 1482 | + new_flags, | |
| 1483 | + (struct vm_userfaultfd_ctx){ctx}); | |
| 1484 | + if (IS_ERR(vma)) { | |
| 1485 | + ret = PTR_ERR(vma); | |
| 1486 | + break; | |
| 1501 | 1487 | } |
| 1502 | − next: | |
| 1488 | + | |
| 1503 | 1489 | /* |
| 1504 | 1490 | * In the vma_merge() successful mprotect-like case 8: |
| 1505 | 1491 | * the next vma was merged into the current one and |
| @@ -1561,7 +1547,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 55 unmodified lines | ||
| 1561 | 1547 | unsigned long start, end, vma_end; |
| 1562 | 1548 | const void __user *buf = (void __user *)arg; |
| 1563 | 1549 | struct vma_iterator vmi; |
| 1564 | − pgoff_t pgoff; | |
| 1550 | + bool wp_async = userfaultfd_wp_async_ctx(ctx); | |
| 1565 | 1551 | |
| 1566 | 1552 | ret = -EFAULT; |
| 1567 | 1553 | if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister))) |
| @@ -1615,7 +1601,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 47 unmodified lines | ||
| 1615 | 1601 | * provides for more strict behavior to notice |
| 1616 | 1602 | * unregistration errors. |
| 1617 | 1603 | */ |
| 1618 | − if (!vma_can_userfault(cur, cur->vm_flags)) | |
| 1604 | + if (!vma_can_userfault(cur, cur->vm_flags, wp_async)) | |
| 1619 | 1605 | goto out_unlock; |
| 1620 | 1606 | |
| 1621 | 1607 | found = true; |
| @@ -1631,7 +1617,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 9 unmodified lines | ||
| 1631 | 1617 | for_each_vma_range(vmi, vma, end) { |
| 1632 | 1618 | cond_resched(); |
| 1633 | 1619 | |
| 1634 | − BUG_ON(!vma_can_userfault(vma, vma->vm_flags)); | |
| 1620 | + BUG_ON(!vma_can_userfault(vma, vma->vm_flags, wp_async)); | |
| 1635 | 1621 | |
| 1636 | 1622 | /* |
| 1637 | 1623 | * Nothing to do: this vma is already registered into this |
| @@ -1664,26 +1650,13 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · 26 unmodified lines | ||
| 1664 | 1650 | uffd_wp_range(vma, start, vma_end - start, false); |
| 1665 | 1651 | |
| 1666 | 1652 | new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS; |
| 1667 | − pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT); | |
| 1668 | − prev = vma_merge(&vmi, mm, prev, start, vma_end, new_flags, | |
| 1669 | − vma->anon_vma, vma->vm_file, pgoff, | |
| 1670 | − vma_policy(vma), | |
| 1671 | − NULL_VM_UFFD_CTX, anon_vma_name(vma)); | |
| 1672 | − if (prev) { | |
| 1673 | − vma = prev; | |
| 1674 | − goto next; | |
| 1675 | − } | |
| 1676 | − if (vma->vm_start < start) { | |
| 1677 | − ret = split_vma(&vmi, vma, start, 1); | |
| 1678 | − if (ret) | |
| 1679 | − break; | |
| 1680 | − } | |
| 1681 | − if (vma->vm_end > end) { | |
| 1682 | − ret = split_vma(&vmi, vma, end, 0); | |
| 1683 | − if (ret) | |
| 1684 | − break; | |
| 1653 | + vma = vma_modify_flags_uffd(&vmi, prev, vma, start, vma_end, | |
| 1654 | + new_flags, NULL_VM_UFFD_CTX); | |
| 1655 | + if (IS_ERR(vma)) { | |
| 1656 | + ret = PTR_ERR(vma); | |
| 1657 | + break; | |
| 1685 | 1658 | } |
| 1686 | − next: | |
| 1659 | + | |
| 1687 | 1660 | /* |
| 1688 | 1661 | * In the vma_merge() successful mprotect-like case 8: |
| 1689 | 1662 | * the next vma was merged into the current one and |
| @@ -2018,6 +1991,11 @@ out: · 328 unmodified lines | ||
| 2018 | 1991 | return ret; |
| 2019 | 1992 | } |
| 2020 | 1993 | |
| 1994 | +bool userfaultfd_wp_async(struct vm_area_struct *vma) | |
| 1995 | +{ | |
| 1996 | + return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx); | |
| 1997 | +} | |
| 1998 | + | |
| 2021 | 1999 | static inline unsigned int uffd_ctx_features(__u64 user_features) |
| 2022 | 2000 | { |
| 2023 | 2001 | /* |
| @@ -2051,6 +2029,11 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, · 27 unmodified lines | ||
| 2051 | 2029 | ret = -EPERM; |
| 2052 | 2030 | if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE)) |
| 2053 | 2031 | goto err_out; |
| 2032 | + | |
| 2033 | + /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */ | |
| 2034 | + if (features & UFFD_FEATURE_WP_ASYNC) | |
| 2035 | + features |= UFFD_FEATURE_WP_UNPOPULATED; | |
| 2036 | + | |
| 2054 | 2037 | /* report all available features and ioctls to userland */ |
| 2055 | 2038 | uffdio_api.features = UFFD_API_FEATURES; |
| 2056 | 2039 | #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR |
| @@ -2063,6 +2046,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, · 6 unmodified lines | ||
| 2063 | 2046 | #ifndef CONFIG_PTE_MARKER_UFFD_WP |
| 2064 | 2047 | uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM; |
| 2065 | 2048 | uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED; |
| 2049 | + uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC; | |
| 2066 | 2050 | #endif |
| 2067 | 2051 | uffdio_api.ioctls = UFFD_API_IOCTLS; |
| 2068 | 2052 | ret = -EFAULT; |
| @@ -1913,8 +1913,7 @@ xfs_buftarg_shrink_scan( · 1912 unmodified lines | ||
| 1913 | 1913 | struct shrinker *shrink, |
| 1914 | 1914 | struct shrink_control *sc) |
| 1915 | 1915 | { |
| 1916 | − struct xfs_buftarg *btp = container_of(shrink, | |
| 1917 | − struct xfs_buftarg, bt_shrinker); | |
| 1916 | + struct xfs_buftarg *btp = shrink->private_data; | |
| 1918 | 1917 | LIST_HEAD(dispose); |
| 1919 | 1918 | unsigned long freed; |
| 1920 | 1919 | |
| @@ -1936,8 +1935,7 @@ xfs_buftarg_shrink_count( · 15 unmodified lines | ||
| 1936 | 1935 | struct shrinker *shrink, |
| 1937 | 1936 | struct shrink_control *sc) |
| 1938 | 1937 | { |
| 1939 | − struct xfs_buftarg *btp = container_of(shrink, | |
| 1940 | − struct xfs_buftarg, bt_shrinker); | |
| 1938 | + struct xfs_buftarg *btp = shrink->private_data; | |
| 1941 | 1939 | return list_lru_shrink_count(&btp->bt_lru, sc); |
| 1942 | 1940 | } |
| 1943 | 1941 | |
| @@ -1945,7 +1943,7 @@ void · 1 unmodified line | ||
| 1945 | 1943 | xfs_free_buftarg( |
| 1946 | 1944 | struct xfs_buftarg *btp) |
| 1947 | 1945 | { |
| 1948 | − unregister_shrinker(&btp->bt_shrinker); | |
| 1946 | + shrinker_free(btp->bt_shrinker); | |
| 1949 | 1947 | ASSERT(percpu_counter_sum(&btp->bt_io_count) == 0); |
| 1950 | 1948 | percpu_counter_destroy(&btp->bt_io_count); |
| 1951 | 1949 | list_lru_destroy(&btp->bt_lru); |
| @@ -2029,13 +2027,17 @@ xfs_alloc_buftarg( · 77 unmodified lines | ||
| 2029 | 2027 | if (percpu_counter_init(&btp->bt_io_count, 0, GFP_KERNEL)) |
| 2030 | 2028 | goto error_lru; |
| 2031 | 2029 | |
| 2032 | − btp->bt_shrinker.count_objects = xfs_buftarg_shrink_count; | |
| 2033 | − btp->bt_shrinker.scan_objects = xfs_buftarg_shrink_scan; | |
| 2034 | − btp->bt_shrinker.seeks = DEFAULT_SEEKS; | |
| 2035 | − btp->bt_shrinker.flags = SHRINKER_NUMA_AWARE; | |
| 2036 | − if (register_shrinker(&btp->bt_shrinker, "xfs-buf:%s", | |
| 2037 | − mp->m_super->s_id)) | |
| 2030 | + btp->bt_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "xfs-buf:%s", | |
| 2031 | + mp->m_super->s_id); | |
| 2032 | + if (!btp->bt_shrinker) | |
| 2038 | 2033 | goto error_pcpu; |
| 2034 | + | |
| 2035 | + btp->bt_shrinker->count_objects = xfs_buftarg_shrink_count; | |
| 2036 | + btp->bt_shrinker->scan_objects = xfs_buftarg_shrink_scan; | |
| 2037 | + btp->bt_shrinker->private_data = btp; | |
| 2038 | + | |
| 2039 | + shrinker_register(btp->bt_shrinker); | |
| 2040 | + | |
| 2039 | 2041 | return btp; |
| 2040 | 2042 | |
| 2041 | 2043 | error_pcpu: |
| @@ -109,7 +109,7 @@ typedef struct xfs_buftarg { · 108 unmodified lines | ||
| 109 | 109 | size_t bt_logical_sectormask; |
| 110 | 110 | |
| 111 | 111 | /* LRU control structures */ |
| 112 | − struct shrinker bt_shrinker; | |
| 112 | + struct shrinker *bt_shrinker; | |
| 113 | 113 | struct list_lru bt_lru; |
| 114 | 114 | |
| 115 | 115 | struct percpu_counter bt_io_count; |
| @@ -2165,8 +2165,7 @@ xfs_inodegc_shrinker_count( · 2164 unmodified lines | ||
| 2165 | 2165 | struct shrinker *shrink, |
| 2166 | 2166 | struct shrink_control *sc) |
| 2167 | 2167 | { |
| 2168 | − struct xfs_mount *mp = container_of(shrink, struct xfs_mount, | |
| 2169 | − m_inodegc_shrinker); | |
| 2168 | + struct xfs_mount *mp = shrink->private_data; | |
| 2170 | 2169 | struct xfs_inodegc *gc; |
| 2171 | 2170 | int cpu; |
| 2172 | 2171 | |
| @@ -2187,8 +2186,7 @@ xfs_inodegc_shrinker_scan( · 14 unmodified lines | ||
| 2187 | 2186 | struct shrinker *shrink, |
| 2188 | 2187 | struct shrink_control *sc) |
| 2189 | 2188 | { |
| 2190 | − struct xfs_mount *mp = container_of(shrink, struct xfs_mount, | |
| 2191 | − m_inodegc_shrinker); | |
| 2189 | + struct xfs_mount *mp = shrink->private_data; | |
| 2192 | 2190 | struct xfs_inodegc *gc; |
| 2193 | 2191 | int cpu; |
| 2194 | 2192 | bool no_items = true; |
| @@ -2224,13 +2222,19 @@ int · 29 unmodified lines | ||
| 2224 | 2222 | xfs_inodegc_register_shrinker( |
| 2225 | 2223 | struct xfs_mount *mp) |
| 2226 | 2224 | { |
| 2227 | − struct shrinker *shrink = &mp->m_inodegc_shrinker; | |
| 2225 | + mp->m_inodegc_shrinker = shrinker_alloc(SHRINKER_NONSLAB, | |
| 2226 | + "xfs-inodegc:%s", | |
| 2227 | + mp->m_super->s_id); | |
| 2228 | + if (!mp->m_inodegc_shrinker) | |
| 2229 | + return -ENOMEM; | |
| 2230 | + | |
| 2231 | + mp->m_inodegc_shrinker->count_objects = xfs_inodegc_shrinker_count; | |
| 2232 | + mp->m_inodegc_shrinker->scan_objects = xfs_inodegc_shrinker_scan; | |
| 2233 | + mp->m_inodegc_shrinker->seeks = 0; | |
| 2234 | + mp->m_inodegc_shrinker->batch = XFS_INODEGC_SHRINKER_BATCH; | |
| 2235 | + mp->m_inodegc_shrinker->private_data = mp; | |
| 2228 | 2236 | |
| 2229 | − shrink->count_objects = xfs_inodegc_shrinker_count; | |
| 2230 | − shrink->scan_objects = xfs_inodegc_shrinker_scan; | |
| 2231 | − shrink->seeks = 0; | |
| 2232 | − shrink->flags = SHRINKER_NONSLAB; | |
| 2233 | − shrink->batch = XFS_INODEGC_SHRINKER_BATCH; | |
| 2237 | + shrinker_register(mp->m_inodegc_shrinker); | |
| 2234 | 2238 | |
| 2235 | − return register_shrinker(shrink, "xfs-inodegc:%s", mp->m_super->s_id); | |
| 2239 | + return 0; | |
| 2236 | 2240 | } |
| @@ -1021,7 +1021,7 @@ xfs_mountfs( · 1020 unmodified lines | ||
| 1021 | 1021 | out_log_dealloc: |
| 1022 | 1022 | xfs_log_mount_cancel(mp); |
| 1023 | 1023 | out_inodegc_shrinker: |
| 1024 | − unregister_shrinker(&mp->m_inodegc_shrinker); | |
| 1024 | + shrinker_free(mp->m_inodegc_shrinker); | |
| 1025 | 1025 | out_fail_wait: |
| 1026 | 1026 | if (mp->m_logdev_targp && mp->m_logdev_targp != mp->m_ddev_targp) |
| 1027 | 1027 | xfs_buftarg_drain(mp->m_logdev_targp); |
| @@ -1104,7 +1104,7 @@ xfs_unmountfs( · 76 unmodified lines | ||
| 1104 | 1104 | #if defined(DEBUG) |
| 1105 | 1105 | xfs_errortag_clearall(mp); |
| 1106 | 1106 | #endif |
| 1107 | − unregister_shrinker(&mp->m_inodegc_shrinker); | |
| 1107 | + shrinker_free(mp->m_inodegc_shrinker); | |
| 1108 | 1108 | xfs_free_perag(mp); |
| 1109 | 1109 | |
| 1110 | 1110 | xfs_errortag_del(mp); |
| @@ -219,7 +219,7 @@ typedef struct xfs_mount { · 218 unmodified lines | ||
| 219 | 219 | atomic_t m_agirotor; /* last ag dir inode alloced */ |
| 220 | 220 | |
| 221 | 221 | /* Memory shrinker to throttle and reprioritize inodegc */ |
| 222 | − struct shrinker m_inodegc_shrinker; | |
| 222 | + struct shrinker *m_inodegc_shrinker; | |
| 223 | 223 | /* |
| 224 | 224 | * Workqueue item so that we can coalesce multiple inode flush attempts |
| 225 | 225 | * into a single flush. |
| @@ -504,8 +504,7 @@ xfs_qm_shrink_scan( · 503 unmodified lines | ||
| 504 | 504 | struct shrinker *shrink, |
| 505 | 505 | struct shrink_control *sc) |
| 506 | 506 | { |
| 507 | − struct xfs_quotainfo *qi = container_of(shrink, | |
| 508 | − struct xfs_quotainfo, qi_shrinker); | |
| 507 | + struct xfs_quotainfo *qi = shrink->private_data; | |
| 509 | 508 | struct xfs_qm_isolate isol; |
| 510 | 509 | unsigned long freed; |
| 511 | 510 | int error; |
| @@ -539,8 +538,7 @@ xfs_qm_shrink_count( · 27 unmodified lines | ||
| 539 | 538 | struct shrinker *shrink, |
| 540 | 539 | struct shrink_control *sc) |
| 541 | 540 | { |
| 542 | − struct xfs_quotainfo *qi = container_of(shrink, | |
| 543 | − struct xfs_quotainfo, qi_shrinker); | |
| 541 | + struct xfs_quotainfo *qi = shrink->private_data; | |
| 544 | 542 | |
| 545 | 543 | return list_lru_shrink_count(&qi->qi_lru, sc); |
| 546 | 544 | } |
| @@ -680,15 +678,18 @@ xfs_qm_init_quotainfo( · 133 unmodified lines | ||
| 680 | 678 | if (XFS_IS_PQUOTA_ON(mp)) |
| 681 | 679 | xfs_qm_set_defquota(mp, XFS_DQTYPE_PROJ, qinf); |
| 682 | 680 | |
| 683 | − qinf->qi_shrinker.count_objects = xfs_qm_shrink_count; | |
| 684 | − qinf->qi_shrinker.scan_objects = xfs_qm_shrink_scan; | |
| 685 | − qinf->qi_shrinker.seeks = DEFAULT_SEEKS; | |
| 686 | − qinf->qi_shrinker.flags = SHRINKER_NUMA_AWARE; | |
| 687 | − | |
| 688 | − error = register_shrinker(&qinf->qi_shrinker, "xfs-qm:%s", | |
| 689 | − mp->m_super->s_id); | |
| 690 | − if (error) | |
| 681 | + qinf->qi_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE, "xfs-qm:%s", | |
| 682 | + mp->m_super->s_id); | |
| 683 | + if (!qinf->qi_shrinker) { | |
| 684 | + error = -ENOMEM; | |
| 691 | 685 | goto out_free_inos; |
| 686 | + } | |
| 687 | + | |
| 688 | + qinf->qi_shrinker->count_objects = xfs_qm_shrink_count; | |
| 689 | + qinf->qi_shrinker->scan_objects = xfs_qm_shrink_scan; | |
| 690 | + qinf->qi_shrinker->private_data = qinf; | |
| 691 | + | |
| 692 | + shrinker_register(qinf->qi_shrinker); | |
| 692 | 693 | |
| 693 | 694 | return 0; |
| 694 | 695 | |
| @@ -718,7 +719,7 @@ xfs_qm_destroy_quotainfo( · 23 unmodified lines | ||
| 718 | 719 | qi = mp->m_quotainfo; |
| 719 | 720 | ASSERT(qi != NULL); |
| 720 | 721 | |
| 721 | − unregister_shrinker(&qi->qi_shrinker); | |
| 722 | + shrinker_free(qi->qi_shrinker); | |
| 722 | 723 | list_lru_destroy(&qi->qi_lru); |
| 723 | 724 | xfs_qm_destroy_quotainos(qi); |
| 724 | 725 | mutex_destroy(&qi->qi_tree_lock); |
| @@ -63,7 +63,7 @@ struct xfs_quotainfo { · 62 unmodified lines | ||
| 63 | 63 | struct xfs_def_quota qi_usr_default; |
| 64 | 64 | struct xfs_def_quota qi_grp_default; |
| 65 | 65 | struct xfs_def_quota qi_prj_default; |
| 66 | − struct shrinker qi_shrinker; | |
| 66 | + struct shrinker *qi_shrinker; | |
| 67 | 67 | |
| 68 | 68 | /* Minimum and maximum quota expiration timestamp values. */ |
| 69 | 69 | time64_t qi_expiry_min; |
| @@ -58,27 +58,25 @@ static inline bool test_and_set_bit_lock(long nr, volatile unsigned long *addr) · 57 unmodified lines | ||
| 58 | 58 | return arch_test_and_set_bit_lock(nr, addr); |
| 59 | 59 | } |
| 60 | 60 | |
| 61 | −#if defined(arch_clear_bit_unlock_is_negative_byte) | |
| 62 | 61 | /** |
| 63 | − * clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom | |
| 64 | − * byte is negative, for unlock. | |
| 65 | − * @nr: the bit to clear | |
| 66 | − * @addr: the address to start counting from | |
| 62 | + * xor_unlock_is_negative_byte - XOR a single byte in memory and test if | |
| 63 | + * it is negative, for unlock. | |
| 64 | + * @mask: Change the bits which are set in this mask. | |
| 65 | + * @addr: The address of the word containing the byte to change. | |
| 67 | 66 | * |
| 67 | + * Changes some of bits 0-6 in the word pointed to by @addr. | |
| 68 | 68 | * This operation is atomic and provides release barrier semantics. |
| 69 | + * Used to optimise some folio operations which are commonly paired | |
| 70 | + * with an unlock or end of writeback. Bit 7 is used as PG_waiters to | |
| 71 | + * indicate whether anybody is waiting for the unlock. | |
| 69 | 72 | * |
| 70 | − * This is a bit of a one-trick-pony for the filemap code, which clears | |
| 71 | − * PG_locked and tests PG_waiters, | |
| 73 | + * Return: Whether the top bit of the byte is set. | |
| 72 | 74 | */ |
| 73 | −static inline bool | |
| 74 | −clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr) | |
| 75 | +static inline bool xor_unlock_is_negative_byte(unsigned long mask, | |
| 76 | + volatile unsigned long *addr) | |
| 75 | 77 | { |
| 76 | 78 | kcsan_release(); |
| 77 | − instrument_atomic_write(addr + BIT_WORD(nr), sizeof(long)); | |
| 78 | − return arch_clear_bit_unlock_is_negative_byte(nr, addr); | |
| 79 | + instrument_atomic_write(addr, sizeof(long)); | |
| 80 | + return arch_xor_unlock_is_negative_byte(mask, addr); | |
| 79 | 81 | } |
| 80 | −/* Let everybody know we have it. */ | |
| 81 | −#define clear_bit_unlock_is_negative_byte clear_bit_unlock_is_negative_byte | |
| 82 | −#endif | |
| 83 | − | |
| 84 | 82 | #endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_LOCK_H */ |
| @@ -66,27 +66,15 @@ arch___clear_bit_unlock(unsigned int nr, volatile unsigned long *p) · 65 unmodified lines | ||
| 66 | 66 | raw_atomic_long_set_release((atomic_long_t *)p, old); |
| 67 | 67 | } |
| 68 | 68 | |
| 69 | −/** | |
| 70 | − * arch_clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom | |
| 71 | − * byte is negative, for unlock. | |
| 72 | − * @nr: the bit to clear | |
| 73 | − * @addr: the address to start counting from | |
| 74 | − * | |
| 75 | − * This is a bit of a one-trick-pony for the filemap code, which clears | |
| 76 | − * PG_locked and tests PG_waiters, | |
| 77 | − */ | |
| 78 | −#ifndef arch_clear_bit_unlock_is_negative_byte | |
| 79 | −static inline bool arch_clear_bit_unlock_is_negative_byte(unsigned int nr, | |
| 80 | − volatile unsigned long *p) | |
| 69 | +#ifndef arch_xor_unlock_is_negative_byte | |
| 70 | +static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, | |
| 71 | + volatile unsigned long *p) | |
| 81 | 72 | { |
| 82 | 73 | long old; |
| 83 | − unsigned long mask = BIT_MASK(nr); | |
| 84 | 74 | |
| 85 | − p += BIT_WORD(nr); | |
| 86 | − old = raw_atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p); | |
| 75 | + old = raw_atomic_long_fetch_xor_release(mask, (atomic_long_t *)p); | |
| 87 | 76 | return !!(old & BIT(7)); |
| 88 | 77 | } |
| 89 | −#define arch_clear_bit_unlock_is_negative_byte arch_clear_bit_unlock_is_negative_byte | |
| 90 | 78 | #endif |
| 91 | 79 | |
| 92 | 80 | #include <asm-generic/bitops/instrumented-lock.h> |
| @@ -169,6 +169,8 @@ static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) · 168 unmodified lines | ||
| 169 | 169 | ptdesc = pagetable_alloc(gfp, 0); |
| 170 | 170 | if (!ptdesc) |
| 171 | 171 | return NULL; |
| 172 | + | |
| 173 | + pagetable_pud_ctor(ptdesc); | |
| 172 | 174 | return ptdesc_address(ptdesc); |
| 173 | 175 | } |
| 174 | 176 | |
| @@ -190,8 +192,11 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) · 15 unmodified lines | ||
| 190 | 192 | |
| 191 | 193 | static inline void __pud_free(struct mm_struct *mm, pud_t *pud) |
| 192 | 194 | { |
| 195 | + struct ptdesc *ptdesc = virt_to_ptdesc(pud); | |
| 196 | + | |
| 193 | 197 | BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); |
| 194 | − pagetable_free(virt_to_ptdesc(pud)); | |
| 198 | + pagetable_pud_dtor(ptdesc); | |
| 199 | + pagetable_free(ptdesc); | |
| 195 | 200 | } |
| 196 | 201 | |
| 197 | 202 | #ifndef __HAVE_ARCH_PUD_FREE |
| @@ -60,7 +60,7 @@ static inline void get_page_bootmem(unsigned long info, struct page *page, · 59 unmodified lines | ||
| 60 | 60 | |
| 61 | 61 | static inline void free_bootmem_page(struct page *page) |
| 62 | 62 | { |
| 63 | − kmemleak_free_part(page_to_virt(page), PAGE_SIZE); | |
| 63 | + kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE); | |
| 64 | 64 | free_reserved_page(page); |
| 65 | 65 | } |
| 66 | 66 | #endif |
| @@ -198,13 +198,11 @@ void touch_buffer(struct buffer_head *bh); · 197 unmodified lines | ||
| 198 | 198 | void folio_set_bh(struct buffer_head *bh, struct folio *folio, |
| 199 | 199 | unsigned long offset); |
| 200 | 200 | struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size, |
| 201 | − bool retry); | |
| 201 | + gfp_t gfp); | |
| 202 | 202 | struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size, |
| 203 | 203 | bool retry); |
| 204 | −void create_empty_buffers(struct page *, unsigned long, | |
| 205 | − unsigned long b_state); | |
| 206 | −void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize, | |
| 207 | − unsigned long b_state); | |
| 204 | +struct buffer_head *create_empty_buffers(struct folio *folio, | |
| 205 | + unsigned long blocksize, unsigned long b_state); | |
| 208 | 206 | void end_buffer_read_sync(struct buffer_head *bh, int uptodate); |
| 209 | 207 | void end_buffer_write_sync(struct buffer_head *bh, int uptodate); |
| 210 | 208 | void end_buffer_async_write(struct buffer_head *bh, int uptodate); |
| @@ -227,8 +225,8 @@ void __wait_on_buffer(struct buffer_head *); · 16 unmodified lines | ||
| 227 | 225 | wait_queue_head_t *bh_waitq_head(struct buffer_head *bh); |
| 228 | 226 | struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block, |
| 229 | 227 | unsigned size); |
| 230 | −struct buffer_head *__getblk_gfp(struct block_device *bdev, sector_t block, | |
| 231 | − unsigned size, gfp_t gfp); | |
| 228 | +struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block, | |
| 229 | + unsigned size, gfp_t gfp); | |
| 232 | 230 | void __brelse(struct buffer_head *); |
| 233 | 231 | void __bforget(struct buffer_head *); |
| 234 | 232 | void __breadahead(struct block_device *, sector_t block, unsigned int size); |
| @@ -338,17 +336,38 @@ sb_breadahead(struct super_block *sb, sector_t block) · 103 unmodified lines | ||
| 338 | 336 | __breadahead(sb->s_bdev, block, sb->s_blocksize); |
| 339 | 337 | } |
| 340 | 338 | |
| 341 | −static inline struct buffer_head * | |
| 342 | −sb_getblk(struct super_block *sb, sector_t block) | |
| 339 | +static inline struct buffer_head *getblk_unmovable(struct block_device *bdev, | |
| 340 | + sector_t block, unsigned size) | |
| 343 | 341 | { |
| 344 | − return __getblk_gfp(sb->s_bdev, block, sb->s_blocksize, __GFP_MOVABLE); | |
| 342 | + gfp_t gfp; | |
| 343 | + | |
| 344 | + gfp = mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS); | |
| 345 | + gfp |= __GFP_NOFAIL; | |
| 346 | + | |
| 347 | + return bdev_getblk(bdev, block, size, gfp); | |
| 345 | 348 | } |
| 346 | 349 | |
| 350 | +static inline struct buffer_head *__getblk(struct block_device *bdev, | |
| 351 | + sector_t block, unsigned size) | |
| 352 | +{ | |
| 353 | + gfp_t gfp; | |
| 347 | 354 | |
| 348 | −static inline struct buffer_head * | |
| 349 | −sb_getblk_gfp(struct super_block *sb, sector_t block, gfp_t gfp) | |
| 355 | + gfp = mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS); | |
| 356 | + gfp |= __GFP_MOVABLE | __GFP_NOFAIL; | |
| 357 | + | |
| 358 | + return bdev_getblk(bdev, block, size, gfp); | |
| 359 | +} | |
| 360 | + | |
| 361 | +static inline struct buffer_head *sb_getblk(struct super_block *sb, | |
| 362 | + sector_t block) | |
| 350 | 363 | { |
| 351 | − return __getblk_gfp(sb->s_bdev, block, sb->s_blocksize, gfp); | |
| 364 | + return __getblk(sb->s_bdev, block, sb->s_blocksize); | |
| 365 | +} | |
| 366 | + | |
| 367 | +static inline struct buffer_head *sb_getblk_gfp(struct super_block *sb, | |
| 368 | + sector_t block, gfp_t gfp) | |
| 369 | +{ | |
| 370 | + return bdev_getblk(sb->s_bdev, block, sb->s_blocksize, gfp); | |
| 352 | 371 | } |
| 353 | 372 | |
| 354 | 373 | static inline struct buffer_head * |
| @@ -385,20 +404,6 @@ static inline void lock_buffer(struct buffer_head *bh) · 30 unmodified lines | ||
| 385 | 404 | __lock_buffer(bh); |
| 386 | 405 | } |
| 387 | 406 | |
| 388 | −static inline struct buffer_head *getblk_unmovable(struct block_device *bdev, | |
| 389 | − sector_t block, | |
| 390 | − unsigned size) | |
| 391 | −{ | |
| 392 | − return __getblk_gfp(bdev, block, size, 0); | |
| 393 | −} | |
| 394 | − | |
| 395 | −static inline struct buffer_head *__getblk(struct block_device *bdev, | |
| 396 | − sector_t block, | |
| 397 | − unsigned size) | |
| 398 | −{ | |
| 399 | − return __getblk_gfp(bdev, block, size, __GFP_MOVABLE); | |
| 400 | −} | |
| 401 | − | |
| 402 | 407 | static inline void bh_readahead(struct buffer_head *bh, blk_opf_t op_flags) |
| 403 | 408 | { |
| 404 | 409 | if (!buffer_uptodate(bh) && trylock_buffer(bh)) { |
| @@ -450,6 +455,28 @@ __bread(struct block_device *bdev, sector_t block, unsigned size) · 45 unmodified lines | ||
| 450 | 455 | return __bread_gfp(bdev, block, size, __GFP_MOVABLE); |
| 451 | 456 | } |
| 452 | 457 | |
| 458 | +/** | |
| 459 | + * get_nth_bh - Get a reference on the n'th buffer after this one. | |
| 460 | + * @bh: The buffer to start counting from. | |
| 461 | + * @count: How many buffers to skip. | |
| 462 | + * | |
| 463 | + * This is primarily useful for finding the nth buffer in a folio; in | |
| 464 | + * that case you pass the head buffer and the byte offset in the folio | |
| 465 | + * divided by the block size. It can be used for other purposes, but | |
| 466 | + * it will wrap at the end of the folio rather than returning NULL or | |
| 467 | + * proceeding to the next folio for you. | |
| 468 | + * | |
| 469 | + * Return: The requested buffer with an elevated refcount. | |
| 470 | + */ | |
| 471 | +static inline __must_check | |
| 472 | +struct buffer_head *get_nth_bh(struct buffer_head *bh, unsigned int count) | |
| 473 | +{ | |
| 474 | + while (count--) | |
| 475 | + bh = bh->b_this_page; | |
| 476 | + get_bh(bh); | |
| 477 | + return bh; | |
| 478 | +} | |
| 479 | + | |
| 453 | 480 | bool block_dirty_folio(struct address_space *mapping, struct folio *folio); |
| 454 | 481 | |
| 455 | 482 | #ifdef CONFIG_BUFFER_HEAD |
| @@ -73,6 +73,7 @@ struct cacheinfo { · 72 unmodified lines | ||
| 73 | 73 | |
| 74 | 74 | struct cpu_cacheinfo { |
| 75 | 75 | struct cacheinfo *info_list; |
| 76 | + unsigned int per_cpu_data_slice_size; | |
| 76 | 77 | unsigned int num_levels; |
| 77 | 78 | unsigned int num_leaves; |
| 78 | 79 | bool cpu_map_populated; |
| @@ -115,6 +115,11 @@ enum { · 114 unmodified lines | ||
| 115 | 115 | * Enable recursive subtree protection |
| 116 | 116 | */ |
| 117 | 117 | CGRP_ROOT_MEMORY_RECURSIVE_PROT = (1 << 18), |
| 118 | + | |
| 119 | + /* | |
| 120 | + * Enable hugetlb accounting for the memory controller. | |
| 121 | + */ | |
| 122 | + CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING = (1 << 19), | |
| 118 | 123 | }; |
| 119 | 124 | |
| 120 | 125 | /* cftype->flags */ |
| @@ -40,9 +40,24 @@ struct damon_addr_range { · 39 unmodified lines | ||
| 40 | 40 | * @ar: The address range of the region. |
| 41 | 41 | * @sampling_addr: Address of the sample for the next access check. |
| 42 | 42 | * @nr_accesses: Access frequency of this region. |
| 43 | + * @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for | |
| 44 | + * each sampling interval. | |
| 43 | 45 | * @list: List head for siblings. |
| 44 | 46 | * @age: Age of this region. |
| 45 | 47 | * |
| 48 | + * @nr_accesses is reset to zero for every &damon_attrs->aggr_interval and be | |
| 49 | + * increased for every &damon_attrs->sample_interval if an access to the region | |
| 50 | + * during the last sampling interval is found. The update of this field should | |
| 51 | + * not be done with direct access but with the helper function, | |
| 52 | + * damon_update_region_access_rate(). | |
| 53 | + * | |
| 54 | + * @nr_accesses_bp is another representation of @nr_accesses in basis point | |
| 55 | + * (1 in 10,000) that updated for every &damon_attrs->sample_interval in a | |
| 56 | + * manner similar to moving sum. By the algorithm, this value becomes | |
| 57 | + * @nr_accesses * 10000 for every &struct damon_attrs->aggr_interval. This can | |
| 58 | + * be used when the aggregation interval is too huge and therefore cannot wait | |
| 59 | + * for it before getting the access monitoring results. | |
| 60 | + * | |
| 46 | 61 | * @age is initially zero, increased for each aggregation interval, and reset |
| 47 | 62 | * to zero again if the access frequency is significantly changed. If two |
| 48 | 63 | * regions are merged into a new region, both @nr_accesses and @age of the new |
| @@ -52,6 +67,7 @@ struct damon_region { · 3 unmodified lines | ||
| 52 | 67 | struct damon_addr_range ar; |
| 53 | 68 | unsigned long sampling_addr; |
| 54 | 69 | unsigned int nr_accesses; |
| 70 | + unsigned int nr_accesses_bp; | |
| 55 | 71 | struct list_head list; |
| 56 | 72 | |
| 57 | 73 | unsigned int age; |
| @@ -298,24 +314,24 @@ struct damos_access_pattern { · 240 unmodified lines | ||
| 298 | 314 | * struct damos - Represents a Data Access Monitoring-based Operation Scheme. |
| 299 | 315 | * @pattern: Access pattern of target regions. |
| 300 | 316 | * @action: &damo_action to be applied to the target regions. |
| 317 | + * @apply_interval_us: The time between applying the @action. | |
| 301 | 318 | * @quota: Control the aggressiveness of this scheme. |
| 302 | 319 | * @wmarks: Watermarks for automated (in)activation of this scheme. |
| 303 | 320 | * @filters: Additional set of &struct damos_filter for &action. |
| 304 | 321 | * @stat: Statistics of this scheme. |
| 305 | 322 | * @list: List head for siblings. |
| 306 | 323 | * |
| 307 | − * For each aggregation interval, DAMON finds regions which fit in the | |
| 324 | + * For each @apply_interval_us, DAMON finds regions which fit in the | |
| 308 | 325 | * &pattern and applies &action to those. To avoid consuming too much |
| 309 | 326 | * CPU time or IO resources for the &action, "a is used. |
| 310 | 327 | * |
| 328 | + * If @apply_interval_us is zero, &damon_attrs->aggr_interval is used instead. | |
| 329 | + * | |
| 311 | 330 | * To do the work only when needed, schemes can be activated for specific |
| 312 | 331 | * system situations using &wmarks. If all schemes that registered to the |
| 313 | 332 | * monitoring context are inactive, DAMON stops monitoring either, and just |
| 314 | 333 | * repeatedly checks the watermarks. |
| 315 | 334 | * |
| 316 | − * If all schemes that registered to a &struct damon_ctx are inactive, DAMON | |
| 317 | − * stops monitoring and just repeatedly checks the watermarks. | |
| 318 | − * | |
| 319 | 335 | * Before applying the &action to a memory region, &struct damon_operations |
| 320 | 336 | * implementation could check pages of the region and skip &action to respect |
| 321 | 337 | * &filters |
| @@ -327,6 +343,14 @@ struct damos_access_pattern { · 5 unmodified lines | ||
| 327 | 343 | struct damos { |
| 328 | 344 | struct damos_access_pattern pattern; |
| 329 | 345 | enum damos_action action; |
| 346 | + unsigned long apply_interval_us; | |
| 347 | +/* private: internal use only */ | |
| 348 | + /* | |
| 349 | + * number of sample intervals that should be passed before applying | |
| 350 | + * @action | |
| 351 | + */ | |
| 352 | + unsigned long next_apply_sis; | |
| 353 | +/* public: */ | |
| 330 | 354 | struct damos_quota quota; |
| 331 | 355 | struct damos_watermarks wmarks; |
| 332 | 356 | struct list_head filters; |
| @@ -472,13 +496,14 @@ struct damon_callback { · 139 unmodified lines | ||
| 472 | 496 | * regions. |
| 473 | 497 | * |
| 474 | 498 | * For each @sample_interval, DAMON checks whether each region is accessed or |
| 475 | − * not. It aggregates and keeps the access information (number of accesses to | |
| 476 | − * each region) for @aggr_interval time. DAMON also checks whether the target | |
| 477 | − * memory regions need update (e.g., by ``mmap()`` calls from the application, | |
| 478 | − * in case of virtual memory monitoring) and applies the changes for each | |
| 479 | − * @ops_update_interval. All time intervals are in micro-seconds. | |
| 480 | − * Please refer to &struct damon_operations and &struct damon_callback for more | |
| 481 | − * detail. | |
| 499 | + * not during the last @sample_interval. If such access is found, DAMON | |
| 500 | + * aggregates the information by increasing &damon_region->nr_accesses for | |
| 501 | + * @aggr_interval time. For each @aggr_interval, the count is reset. DAMON | |
| 502 | + * also checks whether the target memory regions need update (e.g., by | |
| 503 | + * ``mmap()`` calls from the application, in case of virtual memory monitoring) | |
| 504 | + * and applies the changes for each @ops_update_interval. All time intervals | |
| 505 | + * are in micro-seconds. Please refer to &struct damon_operations and &struct | |
| 506 | + * damon_callback for more detail. | |
| 482 | 507 | */ |
| 483 | 508 | struct damon_attrs { |
| 484 | 509 | unsigned long sample_interval; |
| @@ -522,8 +547,18 @@ struct damon_ctx { · 37 unmodified lines | ||
| 522 | 547 | struct damon_attrs attrs; |
| 523 | 548 | |
| 524 | 549 | /* private: internal use only */ |
| 525 | − struct timespec64 last_aggregation; | |
| 526 | − struct timespec64 last_ops_update; | |
| 550 | + /* number of sample intervals that passed since this context started */ | |
| 551 | + unsigned long passed_sample_intervals; | |
| 552 | + /* | |
| 553 | + * number of sample intervals that should be passed before next | |
| 554 | + * aggregation | |
| 555 | + */ | |
| 556 | + unsigned long next_aggregation_sis; | |
| 557 | + /* | |
| 558 | + * number of sample intervals that should be passed before next ops | |
| 559 | + * update | |
| 560 | + */ | |
| 561 | + unsigned long next_ops_update_sis; | |
| 527 | 562 | |
| 528 | 563 | /* public: */ |
| 529 | 564 | struct task_struct *kdamond; |
| @@ -608,6 +643,8 @@ void damon_add_region(struct damon_region *r, struct damon_target *t); · 78 unmodified lines | ||
| 608 | 643 | void damon_destroy_region(struct damon_region *r, struct damon_target *t); |
| 609 | 644 | int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges, |
| 610 | 645 | unsigned int nr_ranges); |
| 646 | +void damon_update_region_access_rate(struct damon_region *r, bool accessed, | |
| 647 | + struct damon_attrs *attrs); | |
| 611 | 648 | |
| 612 | 649 | struct damos_filter *damos_new_filter(enum damos_filter_type type, |
| 613 | 650 | bool matching); |
| @@ -615,7 +652,9 @@ void damos_add_filter(struct damos *s, struct damos_filter *f); · 1 unmodified line | ||
| 615 | 652 | void damos_destroy_filter(struct damos_filter *f); |
| 616 | 653 | |
| 617 | 654 | struct damos *damon_new_scheme(struct damos_access_pattern *pattern, |
| 618 | − enum damos_action action, struct damos_quota *quota, | |
| 655 | + enum damos_action action, | |
| 656 | + unsigned long apply_interval_us, | |
| 657 | + struct damos_quota *quota, | |
| 619 | 658 | struct damos_watermarks *wmarks); |
| 620 | 659 | void damon_add_scheme(struct damon_ctx *ctx, struct damos *s); |
| 621 | 660 | void damon_destroy_scheme(struct damos *s); |
| @@ -642,6 +681,13 @@ static inline bool damon_target_has_pid(const struct damon_ctx *ctx) · 20 unmodified lines | ||
| 642 | 681 | return ctx->ops.id == DAMON_OPS_VADDR || ctx->ops.id == DAMON_OPS_FVADDR; |
| 643 | 682 | } |
| 644 | 683 | |
| 684 | +static inline unsigned int damon_max_nr_accesses(const struct damon_attrs *attrs) | |
| 685 | +{ | |
| 686 | + /* {aggr,sample}_interval are unsigned long, hence could overflow */ | |
| 687 | + return min(attrs->aggr_interval / attrs->sample_interval, | |
| 688 | + (unsigned long)UINT_MAX); | |
| 689 | +} | |
| 690 | + | |
| 645 | 691 | |
| 646 | 692 | int damon_start(struct damon_ctx **ctxs, int nr_ctxs, bool exclusive); |
| 647 | 693 | int damon_stop(struct damon_ctx **ctxs, int nr_ctxs); |
| @@ -159,8 +159,8 @@ int dax_writeback_mapping_range(struct address_space *mapping, · 158 unmodified lines | ||
| 159 | 159 | |
| 160 | 160 | struct page *dax_layout_busy_page(struct address_space *mapping); |
| 161 | 161 | struct page *dax_layout_busy_page_range(struct address_space *mapping, loff_t start, loff_t end); |
| 162 | −dax_entry_t dax_lock_page(struct page *page); | |
| 163 | −void dax_unlock_page(struct page *page, dax_entry_t cookie); | |
| 162 | +dax_entry_t dax_lock_folio(struct folio *folio); | |
| 163 | +void dax_unlock_folio(struct folio *folio, dax_entry_t cookie); | |
| 164 | 164 | dax_entry_t dax_lock_mapping_entry(struct address_space *mapping, |
| 165 | 165 | unsigned long index, struct page **page); |
| 166 | 166 | void dax_unlock_mapping_entry(struct address_space *mapping, |
| @@ -182,14 +182,14 @@ static inline int dax_writeback_mapping_range(struct address_space *mapping, · 15 unmodified lines | ||
| 182 | 182 | return -EOPNOTSUPP; |
| 183 | 183 | } |
| 184 | 184 | |
| 185 | −static inline dax_entry_t dax_lock_page(struct page *page) | |
| 185 | +static inline dax_entry_t dax_lock_folio(struct folio *folio) | |
| 186 | 186 | { |
| 187 | − if (IS_DAX(page->mapping->host)) | |
| 187 | + if (IS_DAX(folio->mapping->host)) | |
| 188 | 188 | return ~0UL; |
| 189 | 189 | return 0; |
| 190 | 190 | } |
| 191 | 191 | |
| 192 | −static inline void dax_unlock_page(struct page *page, dax_entry_t cookie) | |
| 192 | +static inline void dax_unlock_folio(struct folio *folio, dax_entry_t cookie) | |
| 193 | 193 | { |
| 194 | 194 | } |
| 195 | 195 | |
| @@ -454,7 +454,7 @@ extern const struct address_space_operations empty_aops; · 453 unmodified lines | ||
| 454 | 454 | * It is also used to block modification of page cache contents through |
| 455 | 455 | * memory mappings. |
| 456 | 456 | * @gfp_mask: Memory allocation flags to use for allocating pages. |
| 457 | − * @i_mmap_writable: Number of VM_SHARED mappings. | |
| 457 | + * @i_mmap_writable: Number of VM_SHARED, VM_MAYWRITE mappings. | |
| 458 | 458 | * @nr_thps: Number of THPs in the pagecache (non-shmem only). |
| 459 | 459 | * @i_mmap: Tree of private and shared mappings. |
| 460 | 460 | * @i_mmap_rwsem: Protects @i_mmap and @i_mmap_writable. |
| @@ -557,7 +557,7 @@ static inline int mapping_mapped(struct address_space *mapping) · 96 unmodified lines | ||
| 557 | 557 | |
| 558 | 558 | /* |
| 559 | 559 | * Might pages of this file have been modified in userspace? |
| 560 | − * Note that i_mmap_writable counts all VM_SHARED vmas: do_mmap | |
| 560 | + * Note that i_mmap_writable counts all VM_SHARED, VM_MAYWRITE vmas: do_mmap | |
| 561 | 561 | * marks vma as VM_SHARED if it is shared, and the file was opened for |
| 562 | 562 | * writing i.e. vma may be mprotected writable even if now readonly. |
| 563 | 563 | * |
| @@ -1270,7 +1270,7 @@ struct super_block { · 706 unmodified lines | ||
| 1270 | 1270 | |
| 1271 | 1271 | const struct dentry_operations *s_d_op; /* default d_op for dentries */ |
| 1272 | 1272 | |
| 1273 | − struct shrinker s_shrink; /* per-sb shrinker handle */ | |
| 1273 | + struct shrinker *s_shrink; /* per-sb shrinker handle */ | |
| 1274 | 1274 | |
| 1275 | 1275 | /* Number of inodes with nlink == 0 but still referenced */ |
| 1276 | 1276 | atomic_long_t s_remove_count; |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/topology.h> |
| 9 | 9 | |
| 10 | 10 | struct vm_area_struct; |
| 11 | +struct mempolicy; | |
| 11 | 12 | |
| 12 | 13 | /* Convert GFP flags to their corresponding migrate type */ |
| 13 | 14 | #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) |
| @@ -262,7 +263,9 @@ static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, · 248 unmodified lines | ||
| 262 | 263 | |
| 263 | 264 | #ifdef CONFIG_NUMA |
| 264 | 265 | struct page *alloc_pages(gfp_t gfp, unsigned int order); |
| 265 | −struct folio *folio_alloc(gfp_t gfp, unsigned order); | |
| 266 | +struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, | |
| 267 | + struct mempolicy *mpol, pgoff_t ilx, int nid); | |
| 268 | +struct folio *folio_alloc(gfp_t gfp, unsigned int order); | |
| 266 | 269 | struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, |
| 267 | 270 | unsigned long addr, bool hugepage); |
| 268 | 271 | #else |
| @@ -270,6 +273,11 @@ static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) · 1 unmodified line | ||
| 270 | 273 | { |
| 271 | 274 | return alloc_pages_node(numa_node_id(), gfp_mask, order); |
| 272 | 275 | } |
| 276 | +static inline struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, | |
| 277 | + struct mempolicy *mpol, pgoff_t ilx, int nid) | |
| 278 | +{ | |
| 279 | + return alloc_pages(gfp, order); | |
| 280 | +} | |
| 273 | 281 | static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order) |
| 274 | 282 | { |
| 275 | 283 | return __folio_alloc_node(gfp, order, numa_node_id()); |
| @@ -320,11 +328,13 @@ extern void page_frag_free(void *addr); · 44 unmodified lines | ||
| 320 | 328 | #define free_page(addr) free_pages((addr), 0) |
| 321 | 329 | |
| 322 | 330 | void page_alloc_init_cpuhp(void); |
| 331 | +int decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp); | |
| 323 | 332 | void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp); |
| 324 | 333 | void drain_all_pages(struct zone *zone); |
| 325 | 334 | void drain_local_pages(struct zone *zone); |
| 326 | 335 | |
| 327 | 336 | void page_alloc_init_late(void); |
| 337 | +void setup_pcp_cacheinfo(void); | |
| 328 | 338 | |
| 329 | 339 | /* |
| 330 | 340 | * gfp_allowed_mask is set to GFP_BOOT_MASK during early boot to restrict what |
| @@ -30,7 +30,7 @@ void free_huge_folio(struct folio *folio); · 29 unmodified lines | ||
| 30 | 30 | |
| 31 | 31 | #ifdef CONFIG_HUGETLB_PAGE |
| 32 | 32 | |
| 33 | −#include <linux/mempolicy.h> | |
| 33 | +#include <linux/pagemap.h> | |
| 34 | 34 | #include <linux/shm.h> |
| 35 | 35 | #include <asm/tlbflush.h> |
| 36 | 36 | |
| @@ -280,6 +280,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, · 243 unmodified lines | ||
| 280 | 280 | unsigned long cp_flags); |
| 281 | 281 | |
| 282 | 282 | bool is_hugetlb_entry_migration(pte_t pte); |
| 283 | +bool is_hugetlb_entry_hwpoisoned(pte_t pte); | |
| 283 | 284 | void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); |
| 284 | 285 | |
| 285 | 286 | #else /* !CONFIG_HUGETLB_PAGE */ |
| @@ -544,7 +545,6 @@ static inline struct hugetlbfs_sb_info *HUGETLBFS_SB(struct super_block *sb) · 258 unmodified lines | ||
| 544 | 545 | } |
| 545 | 546 | |
| 546 | 547 | struct hugetlbfs_inode_info { |
| 547 | − struct shared_policy policy; | |
| 548 | 548 | struct inode vfs_inode; |
| 549 | 549 | unsigned int seals; |
| 550 | 550 | }; |
| @@ -748,8 +748,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 197 unmodified lines | ||
| 748 | 748 | unsigned long addr, int avoid_reserve); |
| 749 | 749 | struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, |
| 750 | 750 | nodemask_t *nmask, gfp_t gfp_mask); |
| 751 | −struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma, | |
| 752 | − unsigned long address); | |
| 753 | 751 | int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping, |
| 754 | 752 | pgoff_t idx); |
| 755 | 753 | void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, |
| @@ -844,6 +842,12 @@ static inline unsigned int blocks_per_huge_page(struct hstate *h) · 88 unmodified lines | ||
| 844 | 842 | return huge_page_size(h) / 512; |
| 845 | 843 | } |
| 846 | 844 | |
| 845 | +static inline struct folio *filemap_lock_hugetlb_folio(struct hstate *h, | |
| 846 | + struct address_space *mapping, pgoff_t idx) | |
| 847 | +{ | |
| 848 | + return filemap_lock_folio(mapping, idx << huge_page_order(h)); | |
| 849 | +} | |
| 850 | + | |
| 847 | 851 | #include <asm/hugetlb.h> |
| 848 | 852 | |
| 849 | 853 | #ifndef is_hugepage_only_range |
| @@ -1040,6 +1044,12 @@ static inline struct hugepage_subpool *hugetlb_folio_subpool(struct folio *folio · 190 unmodified lines | ||
| 1040 | 1044 | return NULL; |
| 1041 | 1045 | } |
| 1042 | 1046 | |
| 1047 | +static inline struct folio *filemap_lock_hugetlb_folio(struct hstate *h, | |
| 1048 | + struct address_space *mapping, pgoff_t idx) | |
| 1049 | +{ | |
| 1050 | + return NULL; | |
| 1051 | +} | |
| 1052 | + | |
| 1043 | 1053 | static inline int isolate_or_dissolve_huge_page(struct page *page, |
| 1044 | 1054 | struct list_head *list) |
| 1045 | 1055 | { |
| @@ -1060,13 +1070,6 @@ alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, · 14 unmodified lines | ||
| 1060 | 1070 | return NULL; |
| 1061 | 1071 | } |
| 1062 | 1072 | |
| 1063 | −static inline struct folio *alloc_hugetlb_folio_vma(struct hstate *h, | |
| 1064 | − struct vm_area_struct *vma, | |
| 1065 | − unsigned long address) | |
| 1066 | −{ | |
| 1067 | − return NULL; | |
| 1068 | −} | |
| 1069 | − | |
| 1070 | 1073 | static inline int __alloc_bootmem_huge_page(struct hstate *h) |
| 1071 | 1074 | { |
| 1072 | 1075 | return 0; |
| @@ -22,13 +22,6 @@ struct resv_map; · 21 unmodified lines | ||
| 22 | 22 | struct file_region; |
| 23 | 23 | |
| 24 | 24 | #ifdef CONFIG_CGROUP_HUGETLB |
| 25 | −/* | |
| 26 | − * Minimum page order trackable by hugetlb cgroup. | |
| 27 | − * At least 3 pages are necessary for all the tracking information. | |
| 28 | − * The second tail page contains all of the hugetlb-specific fields. | |
| 29 | − */ | |
| 30 | −#define HUGETLB_CGROUP_MIN_ORDER order_base_2(__NR_USED_SUBPAGE) | |
| 31 | − | |
| 32 | 25 | enum hugetlb_memory_event { |
| 33 | 26 | HUGETLB_MAX, |
| 34 | 27 | HUGETLB_NR_MEMORY_EVENTS, |
| @@ -68,8 +61,6 @@ static inline struct hugetlb_cgroup * · 33 unmodified lines | ||
| 68 | 61 | __hugetlb_cgroup_from_folio(struct folio *folio, bool rsvd) |
| 69 | 62 | { |
| 70 | 63 | VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio); |
| 71 | − if (folio_order(folio) < HUGETLB_CGROUP_MIN_ORDER) | |
| 72 | − return NULL; | |
| 73 | 64 | if (rsvd) |
| 74 | 65 | return folio->_hugetlb_cgroup_rsvd; |
| 75 | 66 | else |
| @@ -91,8 +82,6 @@ static inline void __set_hugetlb_cgroup(struct folio *folio, · 15 unmodified lines | ||
| 91 | 82 | struct hugetlb_cgroup *h_cg, bool rsvd) |
| 92 | 83 | { |
| 93 | 84 | VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio); |
| 94 | − if (folio_order(folio) < HUGETLB_CGROUP_MIN_ORDER) | |
| 95 | − return; | |
| 96 | 85 | if (rsvd) |
| 97 | 86 | folio->_hugetlb_cgroup_rsvd = h_cg; |
| 98 | 87 | else |
| @@ -886,7 +886,7 @@ struct journal_s · 885 unmodified lines | ||
| 886 | 886 | * Journal head shrinker, reclaim buffer's journal head which |
| 887 | 887 | * has been written back. |
| 888 | 888 | */ |
| 889 | − struct shrinker j_shrinker; | |
| 889 | + struct shrinker *j_shrinker; | |
| 890 | 890 | |
| 891 | 891 | /** |
| 892 | 892 | * @j_checkpoint_jh_count: |
| @@ -40,6 +40,8 @@ extern unsigned long long max_possible_pfn; · 39 unmodified lines | ||
| 40 | 40 | * via a driver, and never indicated in the firmware-provided memory map as |
| 41 | 41 | * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the |
| 42 | 42 | * kernel resource tree. |
| 43 | + * @MEMBLOCK_RSRV_NOINIT: memory region for which struct pages are | |
| 44 | + * not initialized (only for reserved regions). | |
| 43 | 45 | */ |
| 44 | 46 | enum memblock_flags { |
| 45 | 47 | MEMBLOCK_NONE = 0x0, /* No special request */ |
| @@ -47,6 +49,7 @@ enum memblock_flags { · 1 unmodified line | ||
| 47 | 49 | MEMBLOCK_MIRROR = 0x2, /* mirrored region */ |
| 48 | 50 | MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */ |
| 49 | 51 | MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */ |
| 52 | + MEMBLOCK_RSRV_NOINIT = 0x10, /* don't initialize struct pages */ | |
| 50 | 53 | }; |
| 51 | 54 | |
| 52 | 55 | /** |
| @@ -125,6 +128,7 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); · 72 unmodified lines | ||
| 125 | 128 | int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); |
| 126 | 129 | int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); |
| 127 | 130 | int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); |
| 131 | +int memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size); | |
| 128 | 132 | |
| 129 | 133 | void memblock_free_all(void); |
| 130 | 134 | void memblock_free(void *ptr, size_t size); |
| @@ -259,6 +263,11 @@ static inline bool memblock_is_nomap(struct memblock_region *m) · 128 unmodified lines | ||
| 259 | 263 | return m->flags & MEMBLOCK_NOMAP; |
| 260 | 264 | } |
| 261 | 265 | |
| 266 | +static inline bool memblock_is_reserved_noinit(struct memblock_region *m) | |
| 267 | +{ | |
| 268 | + return m->flags & MEMBLOCK_RSRV_NOINIT; | |
| 269 | +} | |
| 270 | + | |
| 262 | 271 | static inline bool memblock_is_driver_managed(struct memblock_region *m) |
| 263 | 272 | { |
| 264 | 273 | return m->flags & MEMBLOCK_DRIVER_MANAGED; |
| @@ -21,6 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <linux/vmstat.h> |
| 22 | 22 | #include <linux/writeback.h> |
| 23 | 23 | #include <linux/page-flags.h> |
| 24 | +#include <linux/shrinker.h> | |
| 24 | 25 | |
| 25 | 26 | struct mem_cgroup; |
| 26 | 27 | struct obj_cgroup; |
| @@ -88,17 +89,6 @@ struct mem_cgroup_reclaim_iter { · 61 unmodified lines | ||
| 88 | 89 | unsigned int generation; |
| 89 | 90 | }; |
| 90 | 91 | |
| 91 | −/* | |
| 92 | − * Bitmap and deferred work of shrinker::id corresponding to memcg-aware | |
| 93 | − * shrinkers, which have elements charged to this memcg. | |
| 94 | − */ | |
| 95 | −struct shrinker_info { | |
| 96 | − struct rcu_head rcu; | |
| 97 | − atomic_long_t *nr_deferred; | |
| 98 | − unsigned long *map; | |
| 99 | − int map_nr_max; | |
| 100 | −}; | |
| 101 | − | |
| 102 | 92 | struct lruvec_stats_percpu { |
| 103 | 93 | /* Local (CPU and cgroup) state */ |
| 104 | 94 | long state[NR_VM_NODE_STAT_ITEMS]; |
| @@ -153,7 +143,7 @@ struct mem_cgroup_threshold_ary { · 48 unmodified lines | ||
| 153 | 143 | /* Size of entries[] */ |
| 154 | 144 | unsigned int size; |
| 155 | 145 | /* Array of thresholds */ |
| 156 | − struct mem_cgroup_threshold entries[]; | |
| 146 | + struct mem_cgroup_threshold entries[] __counted_by(size); | |
| 157 | 147 | }; |
| 158 | 148 | |
| 159 | 149 | struct mem_cgroup_thresholds { |
| @@ -299,7 +289,13 @@ struct mem_cgroup { · 139 unmodified lines | ||
| 299 | 289 | |
| 300 | 290 | #ifdef CONFIG_MEMCG_KMEM |
| 301 | 291 | int kmemcg_id; |
| 302 | − struct obj_cgroup __rcu *objcg; | |
| 292 | + /* | |
| 293 | + * memcg->objcg is wiped out as a part of the objcg repaprenting | |
| 294 | + * process. memcg->orig_objcg preserves a pointer (and a reference) | |
| 295 | + * to the original objcg until the end of live of memcg. | |
| 296 | + */ | |
| 297 | + struct obj_cgroup __rcu *objcg; | |
| 298 | + struct obj_cgroup *orig_objcg; | |
| 303 | 299 | /* list of inherited objcgs, protected by objcg_lock */ |
| 304 | 300 | struct list_head objcg_list; |
| 305 | 301 | #endif |
| @@ -662,6 +658,8 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target, · 356 unmodified lines | ||
| 662 | 658 | page_counter_read(&memcg->memory); |
| 663 | 659 | } |
| 664 | 660 | |
| 661 | +void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg); | |
| 662 | + | |
| 665 | 663 | int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp); |
| 666 | 664 | |
| 667 | 665 | /** |
| @@ -686,6 +684,9 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, · 18 unmodified lines | ||
| 686 | 684 | return __mem_cgroup_charge(folio, mm, gfp); |
| 687 | 685 | } |
| 688 | 686 | |
| 687 | +int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp, | |
| 688 | + long nr_pages); | |
| 689 | + | |
| 689 | 690 | int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, |
| 690 | 691 | gfp_t gfp, swp_entry_t entry); |
| 691 | 692 | void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry); |
| @@ -713,6 +714,10 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) · 21 unmodified lines | ||
| 713 | 714 | __mem_cgroup_uncharge_list(page_list); |
| 714 | 715 | } |
| 715 | 716 | |
| 717 | +void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); | |
| 718 | + | |
| 719 | +void mem_cgroup_replace_folio(struct folio *old, struct folio *new); | |
| 720 | + | |
| 716 | 721 | void mem_cgroup_migrate(struct folio *old, struct folio *new); |
| 717 | 722 | |
| 718 | 723 | /** |
| @@ -769,6 +774,8 @@ struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p); · 50 unmodified lines | ||
| 769 | 774 | |
| 770 | 775 | struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm); |
| 771 | 776 | |
| 777 | +struct mem_cgroup *get_mem_cgroup_from_current(void); | |
| 778 | + | |
| 772 | 779 | struct lruvec *folio_lruvec_lock(struct folio *folio); |
| 773 | 780 | struct lruvec *folio_lruvec_lock_irq(struct folio *folio); |
| 774 | 781 | struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, |
| @@ -1080,15 +1087,6 @@ static inline void count_memcg_events(struct mem_cgroup *memcg, · 305 unmodified lines | ||
| 1080 | 1087 | local_irq_restore(flags); |
| 1081 | 1088 | } |
| 1082 | 1089 | |
| 1083 | −static inline void count_memcg_page_event(struct page *page, | |
| 1084 | − enum vm_event_item idx) | |
| 1085 | −{ | |
| 1086 | − struct mem_cgroup *memcg = page_memcg(page); | |
| 1087 | − | |
| 1088 | − if (memcg) | |
| 1089 | − count_memcg_events(memcg, idx, 1); | |
| 1090 | −} | |
| 1091 | − | |
| 1092 | 1090 | static inline void count_memcg_folio_events(struct folio *folio, |
| 1093 | 1091 | enum vm_event_item idx, unsigned long nr) |
| 1094 | 1092 | { |
| @@ -1249,12 +1247,23 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target, · 154 unmodified lines | ||
| 1249 | 1247 | return false; |
| 1250 | 1248 | } |
| 1251 | 1249 | |
| 1250 | +static inline void mem_cgroup_commit_charge(struct folio *folio, | |
| 1251 | + struct mem_cgroup *memcg) | |
| 1252 | +{ | |
| 1253 | +} | |
| 1254 | + | |
| 1252 | 1255 | static inline int mem_cgroup_charge(struct folio *folio, |
| 1253 | 1256 | struct mm_struct *mm, gfp_t gfp) |
| 1254 | 1257 | { |
| 1255 | 1258 | return 0; |
| 1256 | 1259 | } |
| 1257 | 1260 | |
| 1261 | +static inline int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, | |
| 1262 | + gfp_t gfp, long nr_pages) | |
| 1263 | +{ | |
| 1264 | + return 0; | |
| 1265 | +} | |
| 1266 | + | |
| 1258 | 1267 | static inline int mem_cgroup_swapin_charge_folio(struct folio *folio, |
| 1259 | 1268 | struct mm_struct *mm, gfp_t gfp, swp_entry_t entry) |
| 1260 | 1269 | { |
| @@ -1273,6 +1282,16 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) · 12 unmodified lines | ||
| 1273 | 1282 | { |
| 1274 | 1283 | } |
| 1275 | 1284 | |
| 1285 | +static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, | |
| 1286 | + unsigned int nr_pages) | |
| 1287 | +{ | |
| 1288 | +} | |
| 1289 | + | |
| 1290 | +static inline void mem_cgroup_replace_folio(struct folio *old, | |
| 1291 | + struct folio *new) | |
| 1292 | +{ | |
| 1293 | +} | |
| 1294 | + | |
| 1276 | 1295 | static inline void mem_cgroup_migrate(struct folio *old, struct folio *new) |
| 1277 | 1296 | { |
| 1278 | 1297 | } |
| @@ -1310,6 +1329,11 @@ static inline struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) · 31 unmodified lines | ||
| 1310 | 1329 | return NULL; |
| 1311 | 1330 | } |
| 1312 | 1331 | |
| 1332 | +static inline struct mem_cgroup *get_mem_cgroup_from_current(void) | |
| 1333 | +{ | |
| 1334 | + return NULL; | |
| 1335 | +} | |
| 1336 | + | |
| 1313 | 1337 | static inline |
| 1314 | 1338 | struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css) |
| 1315 | 1339 | { |
| @@ -1565,11 +1589,6 @@ static inline void __count_memcg_events(struct mem_cgroup *memcg, · 249 unmodified lines | ||
| 1565 | 1589 | { |
| 1566 | 1590 | } |
| 1567 | 1591 | |
| 1568 | −static inline void count_memcg_page_event(struct page *page, | |
| 1569 | − int idx) | |
| 1570 | −{ | |
| 1571 | −} | |
| 1572 | − | |
| 1573 | 1592 | static inline void count_memcg_folio_events(struct folio *folio, |
| 1574 | 1593 | enum vm_event_item idx, unsigned long nr) |
| 1575 | 1594 | { |
| @@ -1763,9 +1782,27 @@ bool mem_cgroup_kmem_disabled(void); · 187 unmodified lines | ||
| 1763 | 1782 | int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); |
| 1764 | 1783 | void __memcg_kmem_uncharge_page(struct page *page, int order); |
| 1765 | 1784 | |
| 1766 | −struct obj_cgroup *get_obj_cgroup_from_current(void); | |
| 1785 | +/* | |
| 1786 | + * The returned objcg pointer is safe to use without additional | |
| 1787 | + * protection within a scope. The scope is defined either by | |
| 1788 | + * the current task (similar to the "current" global variable) | |
| 1789 | + * or by set_active_memcg() pair. | |
| 1790 | + * Please, use obj_cgroup_get() to get a reference if the pointer | |
| 1791 | + * needs to be used outside of the local scope. | |
| 1792 | + */ | |
| 1793 | +struct obj_cgroup *current_obj_cgroup(void); | |
| 1767 | 1794 | struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio); |
| 1768 | 1795 | |
| 1796 | +static inline struct obj_cgroup *get_obj_cgroup_from_current(void) | |
| 1797 | +{ | |
| 1798 | + struct obj_cgroup *objcg = current_obj_cgroup(); | |
| 1799 | + | |
| 1800 | + if (objcg) | |
| 1801 | + obj_cgroup_get(objcg); | |
| 1802 | + | |
| 1803 | + return objcg; | |
| 1804 | +} | |
| 1805 | + | |
| 1769 | 1806 | int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size); |
| 1770 | 1807 | void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); |
| 1771 | 1808 | |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <linux/nodemask.h> |
| 7 | 7 | #include <linux/kref.h> |
| 8 | 8 | #include <linux/mmzone.h> |
| 9 | +#include <linux/notifier.h> | |
| 9 | 10 | /* |
| 10 | 11 | * Each tier cover a abstrace distance chunk size of 128 |
| 11 | 12 | */ |
| @@ -22,7 +23,9 @@ · 10 unmodified lines | ||
| 22 | 23 | struct memory_tier; |
| 23 | 24 | struct memory_dev_type { |
| 24 | 25 | /* list of memory types that are part of same tier as this type */ |
| 25 | − struct list_head tier_sibiling; | |
| 26 | + struct list_head tier_sibling; | |
| 27 | + /* list of memory types that are managed by one driver */ | |
| 28 | + struct list_head list; | |
| 26 | 29 | /* abstract distance for this specific memory type */ |
| 27 | 30 | int adistance; |
| 28 | 31 | /* Nodes of same abstract distance */ |
| @@ -30,12 +33,21 @@ struct memory_dev_type { · 1 unmodified line | ||
| 30 | 33 | struct kref kref; |
| 31 | 34 | }; |
| 32 | 35 | |
| 36 | +struct node_hmem_attrs; | |
| 37 | + | |
| 33 | 38 | #ifdef CONFIG_NUMA |
| 34 | 39 | extern bool numa_demotion_enabled; |
| 40 | +extern struct memory_dev_type *default_dram_type; | |
| 35 | 41 | struct memory_dev_type *alloc_memory_type(int adistance); |
| 36 | 42 | void put_memory_type(struct memory_dev_type *memtype); |
| 37 | 43 | void init_node_memory_type(int node, struct memory_dev_type *default_type); |
| 38 | 44 | void clear_node_memory_type(int node, struct memory_dev_type *memtype); |
| 45 | +int register_mt_adistance_algorithm(struct notifier_block *nb); | |
| 46 | +int unregister_mt_adistance_algorithm(struct notifier_block *nb); | |
| 47 | +int mt_calc_adistance(int node, int *adist); | |
| 48 | +int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf, | |
| 49 | + const char *source); | |
| 50 | +int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist); | |
| 39 | 51 | #ifdef CONFIG_MIGRATION |
| 40 | 52 | int next_demotion_node(int node); |
| 41 | 53 | void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets); |
| @@ -60,6 +72,7 @@ static inline bool node_is_toptier(int node) · 18 unmodified lines | ||
| 60 | 72 | #else |
| 61 | 73 | |
| 62 | 74 | #define numa_demotion_enabled false |
| 75 | +#define default_dram_type NULL | |
| 63 | 76 | /* |
| 64 | 77 | * CONFIG_NUMA implementation returns non NULL error. |
| 65 | 78 | */ |
| @@ -97,5 +110,31 @@ static inline bool node_is_toptier(int node) · 31 unmodified lines | ||
| 97 | 110 | { |
| 98 | 111 | return true; |
| 99 | 112 | } |
| 113 | + | |
| 114 | +static inline int register_mt_adistance_algorithm(struct notifier_block *nb) | |
| 115 | +{ | |
| 116 | + return 0; | |
| 117 | +} | |
| 118 | + | |
| 119 | +static inline int unregister_mt_adistance_algorithm(struct notifier_block *nb) | |
| 120 | +{ | |
| 121 | + return 0; | |
| 122 | +} | |
| 123 | + | |
| 124 | +static inline int mt_calc_adistance(int node, int *adist) | |
| 125 | +{ | |
| 126 | + return NOTIFY_DONE; | |
| 127 | +} | |
| 128 | + | |
| 129 | +static inline int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf, | |
| 130 | + const char *source) | |
| 131 | +{ | |
| 132 | + return -EIO; | |
| 133 | +} | |
| 134 | + | |
| 135 | +static inline int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist) | |
| 136 | +{ | |
| 137 | + return -EIO; | |
| 138 | +} | |
| 100 | 139 | #endif /* CONFIG_NUMA */ |
| 101 | 140 | #endif /* _LINUX_MEMORY_TIERS_H */ |
| @@ -17,6 +17,8 @@ · 16 unmodified lines | ||
| 17 | 17 | |
| 18 | 18 | struct mm_struct; |
| 19 | 19 | |
| 20 | +#define NO_INTERLEAVE_INDEX (-1UL) /* use task il_prev for interleaving */ | |
| 21 | + | |
| 20 | 22 | #ifdef CONFIG_NUMA |
| 21 | 23 | |
| 22 | 24 | /* |
| @@ -89,8 +91,6 @@ static inline struct mempolicy *mpol_dup(struct mempolicy *pol) · 66 unmodified lines | ||
| 89 | 91 | return pol; |
| 90 | 92 | } |
| 91 | 93 | |
| 92 | −#define vma_policy(vma) ((vma)->vm_policy) | |
| 93 | − | |
| 94 | 94 | static inline void mpol_get(struct mempolicy *pol) |
| 95 | 95 | { |
| 96 | 96 | if (pol) |
| @@ -107,35 +107,30 @@ static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b) · 10 unmodified lines | ||
| 107 | 107 | |
| 108 | 108 | /* |
| 109 | 109 | * Tree of shared policies for a shared memory region. |
| 110 | − * Maintain the policies in a pseudo mm that contains vmas. The vmas | |
| 111 | − * carry the policy. As a special twist the pseudo mm is indexed in pages, not | |
| 112 | − * bytes, so that we can work with shared memory segments bigger than | |
| 113 | − * unsigned long. | |
| 114 | 110 | */ |
| 115 | − | |
| 116 | −struct sp_node { | |
| 117 | − struct rb_node nd; | |
| 118 | − unsigned long start, end; | |
| 119 | − struct mempolicy *policy; | |
| 120 | −}; | |
| 121 | − | |
| 122 | 111 | struct shared_policy { |
| 123 | 112 | struct rb_root root; |
| 124 | 113 | rwlock_t lock; |
| 125 | 114 | }; |
| 115 | +struct sp_node { | |
| 116 | + struct rb_node nd; | |
| 117 | + pgoff_t start, end; | |
| 118 | + struct mempolicy *policy; | |
| 119 | +}; | |
| 126 | 120 | |
| 127 | 121 | int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst); |
| 128 | 122 | void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol); |
| 129 | −int mpol_set_shared_policy(struct shared_policy *info, | |
| 130 | − struct vm_area_struct *vma, | |
| 131 | − struct mempolicy *new); | |
| 132 | −void mpol_free_shared_policy(struct shared_policy *p); | |
| 123 | +int mpol_set_shared_policy(struct shared_policy *sp, | |
| 124 | + struct vm_area_struct *vma, struct mempolicy *mpol); | |
| 125 | +void mpol_free_shared_policy(struct shared_policy *sp); | |
| 133 | 126 | struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp, |
| 134 | − unsigned long idx); | |
| 127 | + pgoff_t idx); | |
| 135 | 128 | |
| 136 | 129 | struct mempolicy *get_task_policy(struct task_struct *p); |
| 137 | 130 | struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, |
| 138 | − unsigned long addr); | |
| 131 | + unsigned long addr, pgoff_t *ilx); | |
| 132 | +struct mempolicy *get_vma_policy(struct vm_area_struct *vma, | |
| 133 | + unsigned long addr, int order, pgoff_t *ilx); | |
| 139 | 134 | bool vma_policy_mof(struct vm_area_struct *vma); |
| 140 | 135 | |
| 141 | 136 | extern void numa_default_policy(void); |
| @@ -149,8 +144,6 @@ extern int huge_node(struct vm_area_struct *vma, · 7 unmodified lines | ||
| 149 | 144 | extern bool init_nodemask_of_mempolicy(nodemask_t *mask); |
| 150 | 145 | extern bool mempolicy_in_oom_domain(struct task_struct *tsk, |
| 151 | 146 | const nodemask_t *mask); |
| 152 | −extern nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy); | |
| 153 | − | |
| 154 | 147 | extern unsigned int mempolicy_slab_node(void); |
| 155 | 148 | |
| 156 | 149 | extern enum zone_type policy_zone; |
| @@ -174,7 +167,7 @@ extern void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol); · 17 unmodified lines | ||
| 174 | 167 | /* Check if a vma is migratable */ |
| 175 | 168 | extern bool vma_migratable(struct vm_area_struct *vma); |
| 176 | 169 | |
| 177 | −extern int mpol_misplaced(struct page *, struct vm_area_struct *, unsigned long); | |
| 170 | +int mpol_misplaced(struct folio *, struct vm_area_struct *, unsigned long); | |
| 178 | 171 | extern void mpol_put_task_policy(struct task_struct *); |
| 179 | 172 | |
| 180 | 173 | static inline bool mpol_is_preferred_many(struct mempolicy *pol) |
| @@ -188,12 +181,17 @@ extern bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone); · 7 unmodified lines | ||
| 188 | 181 | |
| 189 | 182 | struct mempolicy {}; |
| 190 | 183 | |
| 184 | +static inline struct mempolicy *get_task_policy(struct task_struct *p) | |
| 185 | +{ | |
| 186 | + return NULL; | |
| 187 | +} | |
| 188 | + | |
| 191 | 189 | static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b) |
| 192 | 190 | { |
| 193 | 191 | return true; |
| 194 | 192 | } |
| 195 | 193 | |
| 196 | −static inline void mpol_put(struct mempolicy *p) | |
| 194 | +static inline void mpol_put(struct mempolicy *pol) | |
| 197 | 195 | { |
| 198 | 196 | } |
| 199 | 197 | |
| @@ -212,17 +210,22 @@ static inline void mpol_shared_policy_init(struct shared_policy *sp, · 12 unmodified lines | ||
| 212 | 210 | { |
| 213 | 211 | } |
| 214 | 212 | |
| 215 | −static inline void mpol_free_shared_policy(struct shared_policy *p) | |
| 213 | +static inline void mpol_free_shared_policy(struct shared_policy *sp) | |
| 216 | 214 | { |
| 217 | 215 | } |
| 218 | 216 | |
| 219 | 217 | static inline struct mempolicy * |
| 220 | −mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx) | |
| 218 | +mpol_shared_policy_lookup(struct shared_policy *sp, pgoff_t idx) | |
| 221 | 219 | { |
| 222 | 220 | return NULL; |
| 223 | 221 | } |
| 224 | 222 | |
| 225 | −#define vma_policy(vma) NULL | |
| 223 | +static inline struct mempolicy *get_vma_policy(struct vm_area_struct *vma, | |
| 224 | + unsigned long addr, int order, pgoff_t *ilx) | |
| 225 | +{ | |
| 226 | + *ilx = 0; | |
| 227 | + return NULL; | |
| 228 | +} | |
| 226 | 229 | |
| 227 | 230 | static inline int |
| 228 | 231 | vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) |
| @@ -278,7 +281,8 @@ static inline int mpol_parse_str(char *str, struct mempolicy **mpol) · 49 unmodified lines | ||
| 278 | 281 | } |
| 279 | 282 | #endif |
| 280 | 283 | |
| 281 | −static inline int mpol_misplaced(struct page *page, struct vm_area_struct *vma, | |
| 284 | +static inline int mpol_misplaced(struct folio *folio, | |
| 285 | + struct vm_area_struct *vma, | |
| 282 | 286 | unsigned long address) |
| 283 | 287 | { |
| 284 | 288 | return -1; /* no node preference */ |
| @@ -142,10 +142,10 @@ const struct movable_operations *page_movable_ops(struct page *page) · 141 unmodified lines | ||
| 142 | 142 | } |
| 143 | 143 | |
| 144 | 144 | #ifdef CONFIG_NUMA_BALANCING |
| 145 | −int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, | |
| 145 | +int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma, | |
| 146 | 146 | int node); |
| 147 | 147 | #else |
| 148 | −static inline int migrate_misplaced_page(struct page *page, | |
| 148 | +static inline int migrate_misplaced_folio(struct folio *folio, | |
| 149 | 149 | struct vm_area_struct *vma, int node) |
| 150 | 150 | { |
| 151 | 151 | return -EAGAIN; /* can't migrate now */ |
| @@ -617,7 +617,7 @@ struct vm_operations_struct { · 616 unmodified lines | ||
| 617 | 617 | * policy. |
| 618 | 618 | */ |
| 619 | 619 | struct mempolicy *(*get_policy)(struct vm_area_struct *vma, |
| 620 | − unsigned long addr); | |
| 620 | + unsigned long addr, pgoff_t *ilx); | |
| 621 | 621 | #endif |
| 622 | 622 | /* |
| 623 | 623 | * Called by vm_normal_page() for special PTEs to find the |
| @@ -935,6 +935,17 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma) · 311 unmodified lines | ||
| 935 | 935 | return vma->vm_flags & VM_ACCESS_FLAGS; |
| 936 | 936 | } |
| 937 | 937 | |
| 938 | +static inline bool is_shared_maywrite(vm_flags_t vm_flags) | |
| 939 | +{ | |
| 940 | + return (vm_flags & (VM_SHARED | VM_MAYWRITE)) == | |
| 941 | + (VM_SHARED | VM_MAYWRITE); | |
| 942 | +} | |
| 943 | + | |
| 944 | +static inline bool vma_is_shared_maywrite(struct vm_area_struct *vma) | |
| 945 | +{ | |
| 946 | + return is_shared_maywrite(vma->vm_flags); | |
| 947 | +} | |
| 948 | + | |
| 938 | 949 | static inline |
| 939 | 950 | struct vm_area_struct *vma_find(struct vma_iterator *vmi, unsigned long max) |
| 940 | 951 | { |
| @@ -1335,7 +1346,6 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, · 394 unmodified lines | ||
| 1335 | 1346 | struct page *page, unsigned int nr, unsigned long addr); |
| 1336 | 1347 | |
| 1337 | 1348 | vm_fault_t finish_fault(struct vm_fault *vmf); |
| 1338 | −vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf); | |
| 1339 | 1349 | #endif |
| 1340 | 1350 | |
| 1341 | 1351 | /* |
| @@ -1684,26 +1694,26 @@ static inline bool __cpupid_match_pid(pid_t task_pid, int cpupid) · 342 unmodified lines | ||
| 1684 | 1694 | |
| 1685 | 1695 | #define cpupid_match_pid(task, cpupid) __cpupid_match_pid(task->pid, cpupid) |
| 1686 | 1696 | #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS |
| 1687 | −static inline int page_cpupid_xchg_last(struct page *page, int cpupid) | |
| 1697 | +static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid) | |
| 1688 | 1698 | { |
| 1689 | − return xchg(&page->_last_cpupid, cpupid & LAST_CPUPID_MASK); | |
| 1699 | + return xchg(&folio->_last_cpupid, cpupid & LAST_CPUPID_MASK); | |
| 1690 | 1700 | } |
| 1691 | 1701 | |
| 1692 | −static inline int page_cpupid_last(struct page *page) | |
| 1702 | +static inline int folio_last_cpupid(struct folio *folio) | |
| 1693 | 1703 | { |
| 1694 | − return page->_last_cpupid; | |
| 1704 | + return folio->_last_cpupid; | |
| 1695 | 1705 | } |
| 1696 | 1706 | static inline void page_cpupid_reset_last(struct page *page) |
| 1697 | 1707 | { |
| 1698 | 1708 | page->_last_cpupid = -1 & LAST_CPUPID_MASK; |
| 1699 | 1709 | } |
| 1700 | 1710 | #else |
| 1701 | −static inline int page_cpupid_last(struct page *page) | |
| 1711 | +static inline int folio_last_cpupid(struct folio *folio) | |
| 1702 | 1712 | { |
| 1703 | − return (page->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK; | |
| 1713 | + return (folio->flags >> LAST_CPUPID_PGSHIFT) & LAST_CPUPID_MASK; | |
| 1704 | 1714 | } |
| 1705 | 1715 | |
| 1706 | −extern int page_cpupid_xchg_last(struct page *page, int cpupid); | |
| 1716 | +int folio_xchg_last_cpupid(struct folio *folio, int cpupid); | |
| 1707 | 1717 | |
| 1708 | 1718 | static inline void page_cpupid_reset_last(struct page *page) |
| 1709 | 1719 | { |
| @@ -1711,11 +1721,12 @@ static inline void page_cpupid_reset_last(struct page *page) · 1 unmodified line | ||
| 1711 | 1721 | } |
| 1712 | 1722 | #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */ |
| 1713 | 1723 | |
| 1714 | −static inline int xchg_page_access_time(struct page *page, int time) | |
| 1724 | +static inline int folio_xchg_access_time(struct folio *folio, int time) | |
| 1715 | 1725 | { |
| 1716 | 1726 | int last_time; |
| 1717 | 1727 | |
| 1718 | − last_time = page_cpupid_xchg_last(page, time >> PAGE_ACCESS_TIME_BUCKETS); | |
| 1728 | + last_time = folio_xchg_last_cpupid(folio, | |
| 1729 | + time >> PAGE_ACCESS_TIME_BUCKETS); | |
| 1719 | 1730 | return last_time << PAGE_ACCESS_TIME_BUCKETS; |
| 1720 | 1731 | } |
| 1721 | 1732 | |
| @@ -1729,19 +1740,19 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) · 7 unmodified lines | ||
| 1729 | 1740 | } |
| 1730 | 1741 | } |
| 1731 | 1742 | #else /* !CONFIG_NUMA_BALANCING */ |
| 1732 | −static inline int page_cpupid_xchg_last(struct page *page, int cpupid) | |
| 1743 | +static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid) | |
| 1733 | 1744 | { |
| 1734 | − return page_to_nid(page); /* XXX */ | |
| 1745 | + return folio_nid(folio); /* XXX */ | |
| 1735 | 1746 | } |
| 1736 | 1747 | |
| 1737 | −static inline int xchg_page_access_time(struct page *page, int time) | |
| 1748 | +static inline int folio_xchg_access_time(struct folio *folio, int time) | |
| 1738 | 1749 | { |
| 1739 | 1750 | return 0; |
| 1740 | 1751 | } |
| 1741 | 1752 | |
| 1742 | −static inline int page_cpupid_last(struct page *page) | |
| 1753 | +static inline int folio_last_cpupid(struct folio *folio) | |
| 1743 | 1754 | { |
| 1744 | − return page_to_nid(page); /* XXX */ | |
| 1755 | + return folio_nid(folio); /* XXX */ | |
| 1745 | 1756 | } |
| 1746 | 1757 | |
| 1747 | 1758 | static inline int cpupid_to_nid(int cpupid) |
| @@ -2325,6 +2336,8 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr, · 577 unmodified lines | ||
| 2325 | 2336 | pte_t pte); |
| 2326 | 2337 | struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, |
| 2327 | 2338 | pte_t pte); |
| 2339 | +struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, | |
| 2340 | + unsigned long addr, pmd_t pmd); | |
| 2328 | 2341 | struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, |
| 2329 | 2342 | pmd_t pmd); |
| 2330 | 2343 | |
| @@ -2411,8 +2424,6 @@ extern int access_process_vm(struct task_struct *tsk, unsigned long addr, · 80 unmodified lines | ||
| 2411 | 2424 | void *buf, int len, unsigned int gup_flags); |
| 2412 | 2425 | extern int access_remote_vm(struct mm_struct *mm, unsigned long addr, |
| 2413 | 2426 | void *buf, int len, unsigned int gup_flags); |
| 2414 | −extern int __access_remote_vm(struct mm_struct *mm, unsigned long addr, | |
| 2415 | − void *buf, int len, unsigned int gup_flags); | |
| 2416 | 2427 | |
| 2417 | 2428 | long get_user_pages_remote(struct mm_struct *mm, |
| 2418 | 2429 | unsigned long start, unsigned long nr_pages, |
| @@ -2423,6 +2434,9 @@ long pin_user_pages_remote(struct mm_struct *mm, · 4 unmodified lines | ||
| 2423 | 2434 | unsigned int gup_flags, struct page **pages, |
| 2424 | 2435 | int *locked); |
| 2425 | 2436 | |
| 2437 | +/* | |
| 2438 | + * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT. | |
| 2439 | + */ | |
| 2426 | 2440 | static inline struct page *get_user_page_vma_remote(struct mm_struct *mm, |
| 2427 | 2441 | unsigned long addr, |
| 2428 | 2442 | int gup_flags, |
| @@ -2430,12 +2444,15 @@ static inline struct page *get_user_page_vma_remote(struct mm_struct *mm, · 1 unmodified line | ||
| 2430 | 2444 | { |
| 2431 | 2445 | struct page *page; |
| 2432 | 2446 | struct vm_area_struct *vma; |
| 2433 | − int got = get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL); | |
| 2447 | + int got; | |
| 2448 | + | |
| 2449 | + if (WARN_ON_ONCE(unlikely(gup_flags & FOLL_NOWAIT))) | |
| 2450 | + return ERR_PTR(-EINVAL); | |
| 2451 | + | |
| 2452 | + got = get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL); | |
| 2434 | 2453 | |
| 2435 | 2454 | if (got < 0) |
| 2436 | 2455 | return ERR_PTR(got); |
| 2437 | − if (got == 0) | |
| 2438 | − return NULL; | |
| 2439 | 2456 | |
| 2440 | 2457 | vma = vma_lookup(mm, addr); |
| 2441 | 2458 | if (WARN_ON_ONCE(!vma)) { |
| @@ -2478,7 +2495,7 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen); · 36 unmodified lines | ||
| 2478 | 2495 | extern unsigned long move_page_tables(struct vm_area_struct *vma, |
| 2479 | 2496 | unsigned long old_addr, struct vm_area_struct *new_vma, |
| 2480 | 2497 | unsigned long new_addr, unsigned long len, |
| 2481 | − bool need_rmap_locks); | |
| 2498 | + bool need_rmap_locks, bool for_stack); | |
| 2482 | 2499 | |
| 2483 | 2500 | /* |
| 2484 | 2501 | * Flags used by change_protection(). For now we make it a bitmap so |
| @@ -2626,14 +2643,6 @@ static inline void setmax_mm_hiwater_rss(unsigned long *maxrss, · 141 unmodified lines | ||
| 2626 | 2643 | *maxrss = hiwater_rss; |
| 2627 | 2644 | } |
| 2628 | 2645 | |
| 2629 | −#if defined(SPLIT_RSS_COUNTING) | |
| 2630 | −void sync_mm_rss(struct mm_struct *mm); | |
| 2631 | −#else | |
| 2632 | −static inline void sync_mm_rss(struct mm_struct *mm) | |
| 2633 | −{ | |
| 2634 | −} | |
| 2635 | −#endif | |
| 2636 | − | |
| 2637 | 2646 | #ifndef CONFIG_ARCH_HAS_PTE_SPECIAL |
| 2638 | 2647 | static inline int pte_special(pte_t pte) |
| 2639 | 2648 | { |
| @@ -3055,6 +3064,22 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud) · 415 unmodified lines | ||
| 3055 | 3064 | return ptl; |
| 3056 | 3065 | } |
| 3057 | 3066 | |
| 3067 | +static inline void pagetable_pud_ctor(struct ptdesc *ptdesc) | |
| 3068 | +{ | |
| 3069 | + struct folio *folio = ptdesc_folio(ptdesc); | |
| 3070 | + | |
| 3071 | + __folio_set_pgtable(folio); | |
| 3072 | + lruvec_stat_add_folio(folio, NR_PAGETABLE); | |
| 3073 | +} | |
| 3074 | + | |
| 3075 | +static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) | |
| 3076 | +{ | |
| 3077 | + struct folio *folio = ptdesc_folio(ptdesc); | |
| 3078 | + | |
| 3079 | + __folio_clear_pgtable(folio); | |
| 3080 | + lruvec_stat_sub_folio(folio, NR_PAGETABLE); | |
| 3081 | +} | |
| 3082 | + | |
| 3058 | 3083 | extern void __init pagecache_init(void); |
| 3059 | 3084 | extern void free_initmem(void); |
| 3060 | 3085 | |
| @@ -3219,22 +3244,73 @@ extern int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, · 158 unmodified lines | ||
| 3219 | 3244 | struct vm_area_struct *next); |
| 3220 | 3245 | extern int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, |
| 3221 | 3246 | unsigned long start, unsigned long end, pgoff_t pgoff); |
| 3222 | −extern struct vm_area_struct *vma_merge(struct vma_iterator *vmi, | |
| 3223 | − struct mm_struct *, struct vm_area_struct *prev, unsigned long addr, | |
| 3224 | − unsigned long end, unsigned long vm_flags, struct anon_vma *, | |
| 3225 | − struct file *, pgoff_t, struct mempolicy *, struct vm_userfaultfd_ctx, | |
| 3226 | − struct anon_vma_name *); | |
| 3227 | 3247 | extern struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *); |
| 3228 | −extern int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *, | |
| 3229 | − unsigned long addr, int new_below); | |
| 3230 | −extern int split_vma(struct vma_iterator *vmi, struct vm_area_struct *, | |
| 3231 | − unsigned long addr, int new_below); | |
| 3232 | 3248 | extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *); |
| 3233 | 3249 | extern void unlink_file_vma(struct vm_area_struct *); |
| 3234 | 3250 | extern struct vm_area_struct *copy_vma(struct vm_area_struct **, |
| 3235 | 3251 | unsigned long addr, unsigned long len, pgoff_t pgoff, |
| 3236 | 3252 | bool *need_rmap_locks); |
| 3237 | 3253 | extern void exit_mmap(struct mm_struct *); |
| 3254 | +struct vm_area_struct *vma_modify(struct vma_iterator *vmi, | |
| 3255 | + struct vm_area_struct *prev, | |
| 3256 | + struct vm_area_struct *vma, | |
| 3257 | + unsigned long start, unsigned long end, | |
| 3258 | + unsigned long vm_flags, | |
| 3259 | + struct mempolicy *policy, | |
| 3260 | + struct vm_userfaultfd_ctx uffd_ctx, | |
| 3261 | + struct anon_vma_name *anon_name); | |
| 3262 | + | |
| 3263 | +/* We are about to modify the VMA's flags. */ | |
| 3264 | +static inline struct vm_area_struct | |
| 3265 | +*vma_modify_flags(struct vma_iterator *vmi, | |
| 3266 | + struct vm_area_struct *prev, | |
| 3267 | + struct vm_area_struct *vma, | |
| 3268 | + unsigned long start, unsigned long end, | |
| 3269 | + unsigned long new_flags) | |
| 3270 | +{ | |
| 3271 | + return vma_modify(vmi, prev, vma, start, end, new_flags, | |
| 3272 | + vma_policy(vma), vma->vm_userfaultfd_ctx, | |
| 3273 | + anon_vma_name(vma)); | |
| 3274 | +} | |
| 3275 | + | |
| 3276 | +/* We are about to modify the VMA's flags and/or anon_name. */ | |
| 3277 | +static inline struct vm_area_struct | |
| 3278 | +*vma_modify_flags_name(struct vma_iterator *vmi, | |
| 3279 | + struct vm_area_struct *prev, | |
| 3280 | + struct vm_area_struct *vma, | |
| 3281 | + unsigned long start, | |
| 3282 | + unsigned long end, | |
| 3283 | + unsigned long new_flags, | |
| 3284 | + struct anon_vma_name *new_name) | |
| 3285 | +{ | |
| 3286 | + return vma_modify(vmi, prev, vma, start, end, new_flags, | |
| 3287 | + vma_policy(vma), vma->vm_userfaultfd_ctx, new_name); | |
| 3288 | +} | |
| 3289 | + | |
| 3290 | +/* We are about to modify the VMA's memory policy. */ | |
| 3291 | +static inline struct vm_area_struct | |
| 3292 | +*vma_modify_policy(struct vma_iterator *vmi, | |
| 3293 | + struct vm_area_struct *prev, | |
| 3294 | + struct vm_area_struct *vma, | |
| 3295 | + unsigned long start, unsigned long end, | |
| 3296 | + struct mempolicy *new_pol) | |
| 3297 | +{ | |
| 3298 | + return vma_modify(vmi, prev, vma, start, end, vma->vm_flags, | |
| 3299 | + new_pol, vma->vm_userfaultfd_ctx, anon_vma_name(vma)); | |
| 3300 | +} | |
| 3301 | + | |
| 3302 | +/* We are about to modify the VMA's flags and/or uffd context. */ | |
| 3303 | +static inline struct vm_area_struct | |
| 3304 | +*vma_modify_flags_uffd(struct vma_iterator *vmi, | |
| 3305 | + struct vm_area_struct *prev, | |
| 3306 | + struct vm_area_struct *vma, | |
| 3307 | + unsigned long start, unsigned long end, | |
| 3308 | + unsigned long new_flags, | |
| 3309 | + struct vm_userfaultfd_ctx new_ctx) | |
| 3310 | +{ | |
| 3311 | + return vma_modify(vmi, prev, vma, start, end, new_flags, | |
| 3312 | + vma_policy(vma), new_ctx, anon_vma_name(vma)); | |
| 3313 | +} | |
| 3238 | 3314 | |
| 3239 | 3315 | static inline int check_data_rlimit(unsigned long rlim, |
| 3240 | 3316 | unsigned long new, |
| @@ -3997,25 +4073,26 @@ static inline void mem_dump_obj(void *object) {} · 756 unmodified lines | ||
| 3997 | 4073 | #endif |
| 3998 | 4074 | |
| 3999 | 4075 | /** |
| 4000 | − * seal_check_future_write - Check for F_SEAL_FUTURE_WRITE flag and handle it | |
| 4076 | + * seal_check_write - Check for F_SEAL_WRITE or F_SEAL_FUTURE_WRITE flags and | |
| 4077 | + * handle them. | |
| 4001 | 4078 | * @seals: the seals to check |
| 4002 | 4079 | * @vma: the vma to operate on |
| 4003 | 4080 | * |
| 4004 | − * Check whether F_SEAL_FUTURE_WRITE is set; if so, do proper check/handling on | |
| 4005 | − * the vma flags. Return 0 if check pass, or <0 for errors. | |
| 4081 | + * Check whether F_SEAL_WRITE or F_SEAL_FUTURE_WRITE are set; if so, do proper | |
| 4082 | + * check/handling on the vma flags. Return 0 if check pass, or <0 for errors. | |
| 4006 | 4083 | */ |
| 4007 | −static inline int seal_check_future_write(int seals, struct vm_area_struct *vma) | |
| 4084 | +static inline int seal_check_write(int seals, struct vm_area_struct *vma) | |
| 4008 | 4085 | { |
| 4009 | − if (seals & F_SEAL_FUTURE_WRITE) { | |
| 4086 | + if (seals & (F_SEAL_WRITE | F_SEAL_FUTURE_WRITE)) { | |
| 4010 | 4087 | /* |
| 4011 | 4088 | * New PROT_WRITE and MAP_SHARED mmaps are not allowed when |
| 4012 | − * "future write" seal active. | |
| 4089 | + * write seals are active. | |
| 4013 | 4090 | */ |
| 4014 | 4091 | if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_WRITE)) |
| 4015 | 4092 | return -EPERM; |
| 4016 | 4093 | |
| 4017 | 4094 | /* |
| 4018 | − * Since an F_SEAL_FUTURE_WRITE sealed memfd can be mapped as | |
| 4095 | + * Since an F_SEAL_[FUTURE_]WRITE sealed memfd can be mapped as | |
| 4019 | 4096 | * MAP_SHARED and read-only, take care to not allow mprotect to |
| 4020 | 4097 | * revert protections on such mappings. Do this only for shared |
| 4021 | 4098 | * mappings. For private mappings, don't need to mask |
| @@ -4059,4 +4136,11 @@ static inline void accept_memory(phys_addr_t start, phys_addr_t end) · 37 unmodified lines | ||
| 4059 | 4136 | |
| 4060 | 4137 | #endif |
| 4061 | 4138 | |
| 4139 | +static inline bool pfn_is_unaccepted_memory(unsigned long pfn) | |
| 4140 | +{ | |
| 4141 | + phys_addr_t paddr = pfn << PAGE_SHIFT; | |
| 4142 | + | |
| 4143 | + return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE); | |
| 4144 | +} | |
| 4145 | + | |
| 4062 | 4146 | #endif /* _LINUX_MM_H */ |
| @@ -4,6 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | |
| 5 | 5 | #include <linux/atomic.h> |
| 6 | 6 | #include <linux/huge_mm.h> |
| 7 | +#include <linux/mm_types.h> | |
| 7 | 8 | #include <linux/swap.h> |
| 8 | 9 | #include <linux/string.h> |
| 9 | 10 | #include <linux/userfaultfd_k.h> |
| @@ -352,15 +353,6 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) · 342 unmodified lines | ||
| 352 | 353 | } |
| 353 | 354 | |
| 354 | 355 | #ifdef CONFIG_ANON_VMA_NAME |
| 355 | −/* | |
| 356 | − * mmap_lock should be read-locked when calling anon_vma_name(). Caller should | |
| 357 | − * either keep holding the lock while using the returned pointer or it should | |
| 358 | − * raise anon_vma_name refcount before releasing the lock. | |
| 359 | − */ | |
| 360 | −extern struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma); | |
| 361 | −extern struct anon_vma_name *anon_vma_name_alloc(const char *name); | |
| 362 | −extern void anon_vma_name_free(struct kref *kref); | |
| 363 | − | |
| 364 | 356 | /* mmap_lock should be read-locked */ |
| 365 | 357 | static inline void anon_vma_name_get(struct anon_vma_name *anon_name) |
| 366 | 358 | { |
| @@ -415,16 +407,6 @@ static inline bool anon_vma_name_eq(struct anon_vma_name *anon_name1, · 48 unmodified lines | ||
| 415 | 407 | } |
| 416 | 408 | |
| 417 | 409 | #else /* CONFIG_ANON_VMA_NAME */ |
| 418 | −static inline struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma) | |
| 419 | −{ | |
| 420 | − return NULL; | |
| 421 | −} | |
| 422 | − | |
| 423 | −static inline struct anon_vma_name *anon_vma_name_alloc(const char *name) | |
| 424 | −{ | |
| 425 | − return NULL; | |
| 426 | −} | |
| 427 | − | |
| 428 | 410 | static inline void anon_vma_name_get(struct anon_vma_name *anon_name) {} |
| 429 | 411 | static inline void anon_vma_name_put(struct anon_vma_name *anon_name) {} |
| 430 | 412 | static inline void dup_anon_vma_name(struct vm_area_struct *orig_vma, |
| @@ -188,6 +188,10 @@ struct page { · 187 unmodified lines | ||
| 188 | 188 | not kmapped, ie. highmem) */ |
| 189 | 189 | #endif /* WANT_PAGE_VIRTUAL */ |
| 190 | 190 | |
| 191 | +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS | |
| 192 | + int _last_cpupid; | |
| 193 | +#endif | |
| 194 | + | |
| 191 | 195 | #ifdef CONFIG_KMSAN |
| 192 | 196 | /* |
| 193 | 197 | * KMSAN metadata for this page: |
| @@ -199,10 +203,6 @@ struct page { · 5 unmodified lines | ||
| 199 | 203 | struct page *kmsan_shadow; |
| 200 | 204 | struct page *kmsan_origin; |
| 201 | 205 | #endif |
| 202 | − | |
| 203 | −#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS | |
| 204 | − int _last_cpupid; | |
| 205 | −#endif | |
| 206 | 206 | } _struct_page_alignment; |
| 207 | 207 | |
| 208 | 208 | /* |
| @@ -261,6 +261,8 @@ typedef struct { · 52 unmodified lines | ||
| 261 | 261 | * @_refcount: Do not access this member directly. Use folio_ref_count() |
| 262 | 262 | * to find how many references there are to this folio. |
| 263 | 263 | * @memcg_data: Memory Control Group data. |
| 264 | + * @virtual: Virtual address in the kernel direct map. | |
| 265 | + * @_last_cpupid: IDs of last CPU and last process that accessed the folio. | |
| 264 | 266 | * @_entire_mapcount: Do not use directly, call folio_entire_mapcount(). |
| 265 | 267 | * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). |
| 266 | 268 | * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). |
| @@ -306,6 +308,12 @@ struct folio { · 39 unmodified lines | ||
| 306 | 308 | atomic_t _refcount; |
| 307 | 309 | #ifdef CONFIG_MEMCG |
| 308 | 310 | unsigned long memcg_data; |
| 311 | +#endif | |
| 312 | +#if defined(WANT_PAGE_VIRTUAL) | |
| 313 | + void *virtual; | |
| 314 | +#endif | |
| 315 | +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS | |
| 316 | + int _last_cpupid; | |
| 309 | 317 | #endif |
| 310 | 318 | /* private: the union with struct page is transitional */ |
| 311 | 319 | }; |
| @@ -362,6 +370,12 @@ FOLIO_MATCH(_refcount, _refcount); · 50 unmodified lines | ||
| 362 | 370 | #ifdef CONFIG_MEMCG |
| 363 | 371 | FOLIO_MATCH(memcg_data, memcg_data); |
| 364 | 372 | #endif |
| 373 | +#if defined(WANT_PAGE_VIRTUAL) | |
| 374 | +FOLIO_MATCH(virtual, virtual); | |
| 375 | +#endif | |
| 376 | +#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS | |
| 377 | +FOLIO_MATCH(_last_cpupid, _last_cpupid); | |
| 378 | +#endif | |
| 365 | 379 | #undef FOLIO_MATCH |
| 366 | 380 | #define FOLIO_MATCH(pg, fl) \ |
| 367 | 381 | static_assert(offsetof(struct folio, fl) == \ |
| @@ -535,6 +549,27 @@ struct anon_vma_name { · 167 unmodified lines | ||
| 535 | 549 | char name[]; |
| 536 | 550 | }; |
| 537 | 551 | |
| 552 | +#ifdef CONFIG_ANON_VMA_NAME | |
| 553 | +/* | |
| 554 | + * mmap_lock should be read-locked when calling anon_vma_name(). Caller should | |
| 555 | + * either keep holding the lock while using the returned pointer or it should | |
| 556 | + * raise anon_vma_name refcount before releasing the lock. | |
| 557 | + */ | |
| 558 | +struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma); | |
| 559 | +struct anon_vma_name *anon_vma_name_alloc(const char *name); | |
| 560 | +void anon_vma_name_free(struct kref *kref); | |
| 561 | +#else /* CONFIG_ANON_VMA_NAME */ | |
| 562 | +static inline struct anon_vma_name *anon_vma_name(struct vm_area_struct *vma) | |
| 563 | +{ | |
| 564 | + return NULL; | |
| 565 | +} | |
| 566 | + | |
| 567 | +static inline struct anon_vma_name *anon_vma_name_alloc(const char *name) | |
| 568 | +{ | |
| 569 | + return NULL; | |
| 570 | +} | |
| 571 | +#endif | |
| 572 | + | |
| 538 | 573 | struct vma_lock { |
| 539 | 574 | struct rw_semaphore lock; |
| 540 | 575 | }; |
| @@ -678,6 +713,12 @@ struct vm_area_struct { · 137 unmodified lines | ||
| 678 | 713 | struct vm_userfaultfd_ctx vm_userfaultfd_ctx; |
| 679 | 714 | } __randomize_layout; |
| 680 | 715 | |
| 716 | +#ifdef CONFIG_NUMA | |
| 717 | +#define vma_policy(vma) ((vma)->vm_policy) | |
| 718 | +#else | |
| 719 | +#define vma_policy(vma) NULL | |
| 720 | +#endif | |
| 721 | + | |
| 681 | 722 | #ifdef CONFIG_SCHED_MM_CID |
| 682 | 723 | struct mm_cid { |
| 683 | 724 | u64 time; |
| @@ -459,7 +459,14 @@ mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) · 458 unmodified lines | ||
| 459 | 459 | lock_map_release(&__mmu_notifier_invalidate_range_start_map); |
| 460 | 460 | } |
| 461 | 461 | |
| 462 | −static inline int | |
| 462 | +/* | |
| 463 | + * This version of mmu_notifier_invalidate_range_start() avoids blocking, but it | |
| 464 | + * can return an error if a notifier can't proceed without blocking, in which | |
| 465 | + * case you're not allowed to modify PTEs in the specified range. | |
| 466 | + * | |
| 467 | + * This is mainly intended for OOM handling. | |
| 468 | + */ | |
| 469 | +static inline int __must_check | |
| 463 | 470 | mmu_notifier_invalidate_range_start_nonblock(struct mmu_notifier_range *range) |
| 464 | 471 | { |
| 465 | 472 | int ret = 0; |
| @@ -639,8 +639,6 @@ struct lruvec { · 638 unmodified lines | ||
| 639 | 639 | #endif |
| 640 | 640 | }; |
| 641 | 641 | |
| 642 | −/* Isolate unmapped pages */ | |
| 643 | −#define ISOLATE_UNMAPPED ((__force isolate_mode_t)0x2) | |
| 644 | 642 | /* Isolate for asynchronous migration */ |
| 645 | 643 | #define ISOLATE_ASYNC_MIGRATE ((__force isolate_mode_t)0x4) |
| 646 | 644 | /* Isolate unevictable pages */ |
| @@ -676,15 +674,34 @@ enum zone_watermarks { · 29 unmodified lines | ||
| 676 | 674 | #define high_wmark_pages(z) (z->_watermark[WMARK_HIGH] + z->watermark_boost) |
| 677 | 675 | #define wmark_pages(z, i) (z->_watermark[i] + z->watermark_boost) |
| 678 | 676 | |
| 677 | +/* | |
| 678 | + * Flags used in pcp->flags field. | |
| 679 | + * | |
| 680 | + * PCPF_PREV_FREE_HIGH_ORDER: a high-order page is freed in the | |
| 681 | + * previous page freeing. To avoid to drain PCP for an accident | |
| 682 | + * high-order page freeing. | |
| 683 | + * | |
| 684 | + * PCPF_FREE_HIGH_BATCH: preserve "pcp->batch" pages in PCP before | |
| 685 | + * draining PCP for consecutive high-order pages freeing without | |
| 686 | + * allocation if data cache slice of CPU is large enough. To reduce | |
| 687 | + * zone lock contention and keep cache-hot pages reusing. | |
| 688 | + */ | |
| 689 | +#define PCPF_PREV_FREE_HIGH_ORDER BIT(0) | |
| 690 | +#define PCPF_FREE_HIGH_BATCH BIT(1) | |
| 691 | + | |
| 679 | 692 | struct per_cpu_pages { |
| 680 | 693 | spinlock_t lock; /* Protects lists field */ |
| 681 | 694 | int count; /* number of pages in the list */ |
| 682 | 695 | int high; /* high watermark, emptying needed */ |
| 696 | + int high_min; /* min high watermark */ | |
| 697 | + int high_max; /* max high watermark */ | |
| 683 | 698 | int batch; /* chunk size for buddy add/remove */ |
| 684 | − short free_factor; /* batch scaling factor during free */ | |
| 699 | + u8 flags; /* protected by pcp->lock */ | |
| 700 | + u8 alloc_factor; /* batch scaling factor during allocate */ | |
| 685 | 701 | #ifdef CONFIG_NUMA |
| 686 | − short expire; /* When 0, remote pagesets are drained */ | |
| 702 | + u8 expire; /* When 0, remote pagesets are drained */ | |
| 687 | 703 | #endif |
| 704 | + short free_count; /* consecutive free count */ | |
| 688 | 705 | |
| 689 | 706 | /* Lists of pages, one per migrate type stored on the pcp-lists */ |
| 690 | 707 | struct list_head lists[NR_PCP_LISTS]; |
| @@ -837,7 +854,8 @@ struct zone { · 146 unmodified lines | ||
| 837 | 854 | * the high and batch values are copied to individual pagesets for |
| 838 | 855 | * faster access |
| 839 | 856 | */ |
| 840 | − int pageset_high; | |
| 857 | + int pageset_high_min; | |
| 858 | + int pageset_high_max; | |
| 841 | 859 | int pageset_batch; |
| 842 | 860 | |
| 843 | 861 | #ifndef CONFIG_SPARSEMEM |
| @@ -998,6 +1016,7 @@ enum zone_flags { · 154 unmodified lines | ||
| 998 | 1016 | * Cleared when kswapd is woken. |
| 999 | 1017 | */ |
| 1000 | 1018 | ZONE_RECLAIM_ACTIVE, /* kswapd may be scanning the zone. */ |
| 1019 | + ZONE_BELOW_HIGH, /* zone is below high watermark. */ | |
| 1001 | 1020 | }; |
| 1002 | 1021 | |
| 1003 | 1022 | static inline unsigned long zone_managed_pages(struct zone *zone) |
| @@ -692,6 +692,25 @@ TESTPAGEFLAG_FALSE(Ksm, ksm) · 691 unmodified lines | ||
| 692 | 692 | |
| 693 | 693 | u64 stable_page_flags(struct page *page); |
| 694 | 694 | |
| 695 | +/** | |
| 696 | + * folio_xor_flags_has_waiters - Change some folio flags. | |
| 697 | + * @folio: The folio. | |
| 698 | + * @mask: Bits set in this word will be changed. | |
| 699 | + * | |
| 700 | + * This must only be used for flags which are changed with the folio | |
| 701 | + * lock held. For example, it is unsafe to use for PG_dirty as that | |
| 702 | + * can be set without the folio lock held. It can also only be used | |
| 703 | + * on flags which are in the range 0-6 as some of the implementations | |
| 704 | + * only affect those bits. | |
| 705 | + * | |
| 706 | + * Return: Whether there are tasks waiting on the folio. | |
| 707 | + */ | |
| 708 | +static inline bool folio_xor_flags_has_waiters(struct folio *folio, | |
| 709 | + unsigned long mask) | |
| 710 | +{ | |
| 711 | + return xor_unlock_is_negative_byte(mask, folio_flags(folio, 0)); | |
| 712 | +} | |
| 713 | + | |
| 695 | 714 | /** |
| 696 | 715 | * folio_test_uptodate - Is this folio up to date? |
| 697 | 716 | * @folio: The folio. |
| @@ -789,9 +789,6 @@ static inline pgoff_t folio_next_index(struct folio *folio) · 788 unmodified lines | ||
| 789 | 789 | */ |
| 790 | 790 | static inline struct page *folio_file_page(struct folio *folio, pgoff_t index) |
| 791 | 791 | { |
| 792 | − /* HugeTLBfs indexes the page cache in units of hpage_size */ | |
| 793 | − if (folio_test_hugetlb(folio)) | |
| 794 | − return &folio->page; | |
| 795 | 792 | return folio_page(folio, index & (folio_nr_pages(folio) - 1)); |
| 796 | 793 | } |
| 797 | 794 | |
| @@ -807,9 +804,6 @@ static inline struct page *folio_file_page(struct folio *folio, pgoff_t index) · 9 unmodified lines | ||
| 807 | 804 | */ |
| 808 | 805 | static inline bool folio_contains(struct folio *folio, pgoff_t index) |
| 809 | 806 | { |
| 810 | − /* HugeTLBfs indexes the page cache in units of hpage_size */ | |
| 811 | − if (folio_test_hugetlb(folio)) | |
| 812 | − return folio->index == index; | |
| 813 | 807 | return index - folio_index(folio) < folio_nr_pages(folio); |
| 814 | 808 | } |
| 815 | 809 | |
| @@ -867,10 +861,9 @@ static inline struct folio *read_mapping_folio(struct address_space *mapping, · 51 unmodified lines | ||
| 867 | 861 | } |
| 868 | 862 | |
| 869 | 863 | /* |
| 870 | − * Get index of the page within radix-tree (but not for hugetlb pages). | |
| 871 | − * (TODO: remove once hugetlb pages will have ->index in PAGE_SIZE) | |
| 864 | + * Get the offset in PAGE_SIZE (even for hugetlb pages). | |
| 872 | 865 | */ |
| 873 | −static inline pgoff_t page_to_index(struct page *page) | |
| 866 | +static inline pgoff_t page_to_pgoff(struct page *page) | |
| 874 | 867 | { |
| 875 | 868 | struct page *head; |
| 876 | 869 | |
| @@ -885,19 +878,6 @@ static inline pgoff_t page_to_index(struct page *page) · 8 unmodified lines | ||
| 885 | 878 | return head->index + page - head; |
| 886 | 879 | } |
| 887 | 880 | |
| 888 | −extern pgoff_t hugetlb_basepage_index(struct page *page); | |
| 889 | − | |
| 890 | −/* | |
| 891 | − * Get the offset in PAGE_SIZE (even for hugetlb pages). | |
| 892 | − * (TODO: hugetlb pages should have ->index in PAGE_SIZE) | |
| 893 | − */ | |
| 894 | −static inline pgoff_t page_to_pgoff(struct page *page) | |
| 895 | −{ | |
| 896 | − if (unlikely(PageHuge(page))) | |
| 897 | − return hugetlb_basepage_index(page); | |
| 898 | − return page_to_index(page); | |
| 899 | −} | |
| 900 | − | |
| 901 | 881 | /* |
| 902 | 882 | * Return byte-offset into filesystem object for page. |
| 903 | 883 | */ |
| @@ -934,24 +914,16 @@ static inline loff_t folio_file_pos(struct folio *folio) · 30 unmodified lines | ||
| 934 | 914 | |
| 935 | 915 | /* |
| 936 | 916 | * Get the offset in PAGE_SIZE (even for hugetlb folios). |
| 937 | − * (TODO: hugetlb folios should have ->index in PAGE_SIZE) | |
| 938 | 917 | */ |
| 939 | 918 | static inline pgoff_t folio_pgoff(struct folio *folio) |
| 940 | 919 | { |
| 941 | − if (unlikely(folio_test_hugetlb(folio))) | |
| 942 | − return hugetlb_basepage_index(&folio->page); | |
| 943 | 920 | return folio->index; |
| 944 | 921 | } |
| 945 | 922 | |
| 946 | −extern pgoff_t linear_hugepage_index(struct vm_area_struct *vma, | |
| 947 | − unsigned long address); | |
| 948 | − | |
| 949 | 923 | static inline pgoff_t linear_page_index(struct vm_area_struct *vma, |
| 950 | 924 | unsigned long address) |
| 951 | 925 | { |
| 952 | 926 | pgoff_t pgoff; |
| 953 | − if (unlikely(is_vm_hugetlb_page(vma))) | |
| 954 | − return linear_hugepage_index(vma, address); | |
| 955 | 927 | pgoff = (address - vma->vm_start) >> PAGE_SHIFT; |
| 956 | 928 | pgoff += vma->vm_pgoff; |
| 957 | 929 | return pgoff; |
| @@ -1129,6 +1101,7 @@ static inline void wait_on_page_locked(struct page *page) · 171 unmodified lines | ||
| 1129 | 1101 | folio_wait_locked(page_folio(page)); |
| 1130 | 1102 | } |
| 1131 | 1103 | |
| 1104 | +void folio_end_read(struct folio *folio, bool success); | |
| 1132 | 1105 | void wait_on_page_writeback(struct page *page); |
| 1133 | 1106 | void folio_wait_writeback(struct folio *folio); |
| 1134 | 1107 | int folio_wait_writeback_killable(struct folio *folio); |
| @@ -57,6 +57,8 @@ void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, · 56 unmodified lines | ||
| 57 | 57 | s32 batch); |
| 58 | 58 | s64 __percpu_counter_sum(struct percpu_counter *fbc); |
| 59 | 59 | int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch); |
| 60 | +bool __percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, | |
| 61 | + s64 amount, s32 batch); | |
| 60 | 62 | void percpu_counter_sync(struct percpu_counter *fbc); |
| 61 | 63 | |
| 62 | 64 | static inline int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs) |
| @@ -69,6 +71,13 @@ static inline void percpu_counter_add(struct percpu_counter *fbc, s64 amount) · 6 unmodified lines | ||
| 69 | 71 | percpu_counter_add_batch(fbc, amount, percpu_counter_batch); |
| 70 | 72 | } |
| 71 | 73 | |
| 74 | +static inline bool | |
| 75 | +percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, s64 amount) | |
| 76 | +{ | |
| 77 | + return __percpu_counter_limited_add(fbc, limit, amount, | |
| 78 | + percpu_counter_batch); | |
| 79 | +} | |
| 80 | + | |
| 72 | 81 | /* |
| 73 | 82 | * With percpu_counter_add_local() and percpu_counter_sub_local(), counts |
| 74 | 83 | * are accumulated in local per cpu counter and not in fbc->count until |
| @@ -185,6 +194,27 @@ percpu_counter_add(struct percpu_counter *fbc, s64 amount) · 110 unmodified lines | ||
| 185 | 194 | local_irq_restore(flags); |
| 186 | 195 | } |
| 187 | 196 | |
| 197 | +static inline bool | |
| 198 | +percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, s64 amount) | |
| 199 | +{ | |
| 200 | + unsigned long flags; | |
| 201 | + bool good = false; | |
| 202 | + s64 count; | |
| 203 | + | |
| 204 | + if (amount == 0) | |
| 205 | + return true; | |
| 206 | + | |
| 207 | + local_irq_save(flags); | |
| 208 | + count = fbc->count + amount; | |
| 209 | + if ((amount > 0 && count <= limit) || | |
| 210 | + (amount < 0 && count >= limit)) { | |
| 211 | + fbc->count = count; | |
| 212 | + good = true; | |
| 213 | + } | |
| 214 | + local_irq_restore(flags); | |
| 215 | + return good; | |
| 216 | +} | |
| 217 | + | |
| 188 | 218 | /* non-SMP percpu_counter_add_local is the same with percpu_counter_add */ |
| 189 | 219 | static inline void |
| 190 | 220 | percpu_counter_add_local(struct percpu_counter *fbc, s64 amount) |
| @@ -189,7 +189,7 @@ typedef int __bitwise rmap_t; · 188 unmodified lines | ||
| 189 | 189 | /* |
| 190 | 190 | * rmap interfaces called when adding or removing pte of page |
| 191 | 191 | */ |
| 192 | −void page_move_anon_rmap(struct page *, struct vm_area_struct *); | |
| 192 | +void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); | |
| 193 | 193 | void page_add_anon_rmap(struct page *, struct vm_area_struct *, |
| 194 | 194 | unsigned long address, rmap_t flags); |
| 195 | 195 | void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, |
| @@ -203,7 +203,7 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, · 7 unmodified lines | ||
| 203 | 203 | void page_remove_rmap(struct page *, struct vm_area_struct *, |
| 204 | 204 | bool compound); |
| 205 | 205 | |
| 206 | −void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *, | |
| 206 | +void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *, | |
| 207 | 207 | unsigned long address, rmap_t flags); |
| 208 | 208 | void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *, |
| 209 | 209 | unsigned long address); |
| @@ -1448,6 +1448,10 @@ struct task_struct { · 1447 unmodified lines | ||
| 1448 | 1448 | struct mem_cgroup *active_memcg; |
| 1449 | 1449 | #endif |
| 1450 | 1450 | |
| 1451 | +#ifdef CONFIG_MEMCG_KMEM | |
| 1452 | + struct obj_cgroup *objcg; | |
| 1453 | +#endif | |
| 1454 | + | |
| 1451 | 1455 | #ifdef CONFIG_BLK_CGROUP |
| 1452 | 1456 | struct gendisk *throttle_disk; |
| 1453 | 1457 | #endif |
| @@ -71,6 +71,7 @@ static inline int get_dumpable(struct mm_struct *mm) · 70 unmodified lines | ||
| 71 | 71 | #define MMF_UNSTABLE 22 /* mm is unstable for copy_from_user */ |
| 72 | 72 | #define MMF_HUGE_ZERO_PAGE 23 /* mm has ever used the global huge zero page */ |
| 73 | 73 | #define MMF_DISABLE_THP 24 /* disable THP for all VMAs */ |
| 74 | +#define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP) | |
| 74 | 75 | #define MMF_OOM_REAP_QUEUED 25 /* mm was queued for oom_reaper */ |
| 75 | 76 | #define MMF_MULTIPROCESS 26 /* mm is shared between processes */ |
| 76 | 77 | /* |
| @@ -85,10 +86,22 @@ static inline int get_dumpable(struct mm_struct *mm) · 8 unmodified lines | ||
| 85 | 86 | #define MMF_HAS_MDWE 28 |
| 86 | 87 | #define MMF_HAS_MDWE_MASK (1 << MMF_HAS_MDWE) |
| 87 | 88 | |
| 88 | −#define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP) | |
| 89 | + | |
| 90 | +#define MMF_HAS_MDWE_NO_INHERIT 29 | |
| 91 | + | |
| 92 | +#define MMF_VM_MERGE_ANY 30 | |
| 93 | +#define MMF_VM_MERGE_ANY_MASK (1 << MMF_VM_MERGE_ANY) | |
| 89 | 94 | |
| 90 | 95 | #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ |
| 91 | − MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK) | |
| 96 | + MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ | |
| 97 | + MMF_VM_MERGE_ANY_MASK) | |
| 98 | + | |
| 99 | +static inline unsigned long mmf_init_flags(unsigned long flags) | |
| 100 | +{ | |
| 101 | + if (flags & (1UL << MMF_HAS_MDWE_NO_INHERIT)) | |
| 102 | + flags &= ~((1UL << MMF_HAS_MDWE) | | |
| 103 | + (1UL << MMF_HAS_MDWE_NO_INHERIT)); | |
| 104 | + return flags & MMF_INIT_MASK; | |
| 105 | +} | |
| 92 | 106 | |
| 93 | −#define MMF_VM_MERGE_ANY 29 | |
| 94 | 107 | #endif /* _LINUX_SCHED_COREDUMP_H */ |
| @@ -403,6 +403,10 @@ DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); · 402 unmodified lines | ||
| 403 | 403 | * __GFP_ACCOUNT allocations till the end of the scope will be charged to the |
| 404 | 404 | * given memcg. |
| 405 | 405 | * |
| 406 | + * Please, make sure that caller has a reference to the passed memcg structure, | |
| 407 | + * so its lifetime is guaranteed to exceed the scope between two | |
| 408 | + * set_active_memcg() calls. | |
| 409 | + * | |
| 406 | 410 | * NOTE: This function can nest. Users must save the return value and |
| 407 | 411 | * reset the previous value after their own charging scope is over. |
| 408 | 412 | */ |
| @@ -30,8 +30,8 @@ extern void task_numa_fault(int last_node, int node, int pages, int flags); · 29 unmodified lines | ||
| 30 | 30 | extern pid_t task_numa_group_id(struct task_struct *p); |
| 31 | 31 | extern void set_numabalancing_state(bool enabled); |
| 32 | 32 | extern void task_numa_free(struct task_struct *p, bool final); |
| 33 | −extern bool should_numa_migrate_memory(struct task_struct *p, struct page *page, | |
| 34 | − int src_nid, int dst_cpu); | |
| 33 | +bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, | |
| 34 | + int src_nid, int dst_cpu); | |
| 35 | 35 | #else |
| 36 | 36 | static inline void task_numa_fault(int last_node, int node, int pages, |
| 37 | 37 | int flags) |
| @@ -48,7 +48,7 @@ static inline void task_numa_free(struct task_struct *p, bool final) · 10 unmodified lines | ||
| 48 | 48 | { |
| 49 | 49 | } |
| 50 | 50 | static inline bool should_numa_migrate_memory(struct task_struct *p, |
| 51 | − struct page *page, int src_nid, int dst_cpu) | |
| 51 | + struct folio *folio, int src_nid, int dst_cpu) | |
| 52 | 52 | { |
| 53 | 53 | return true; |
| 54 | 54 | } |
| @@ -23,18 +23,22 @@ struct shmem_inode_info { · 22 unmodified lines | ||
| 23 | 23 | unsigned long flags; |
| 24 | 24 | unsigned long alloced; /* data pages alloced to file */ |
| 25 | 25 | unsigned long swapped; /* subtotal assigned to swap */ |
| 26 | − pgoff_t fallocend; /* highest fallocate endindex */ | |
| 27 | − struct list_head shrinklist; /* shrinkable hpage inodes */ | |
| 28 | − struct list_head swaplist; /* chain of maybes on swap */ | |
| 26 | + union { | |
| 27 | + struct offset_ctx dir_offsets; /* stable directory offsets */ | |
| 28 | + struct { | |
| 29 | + struct list_head shrinklist; /* shrinkable hpage inodes */ | |
| 30 | + struct list_head swaplist; /* chain of maybes on swap */ | |
| 31 | + }; | |
| 32 | + }; | |
| 33 | + struct timespec64 i_crtime; /* file creation time */ | |
| 29 | 34 | struct shared_policy policy; /* NUMA memory alloc policy */ |
| 30 | 35 | struct simple_xattrs xattrs; /* list of xattrs */ |
| 36 | + pgoff_t fallocend; /* highest fallocate endindex */ | |
| 37 | + unsigned int fsflags; /* for FS_IOC_[SG]ETFLAGS */ | |
| 31 | 38 | atomic_t stop_eviction; /* hold when working on inode */ |
| 32 | − struct timespec64 i_crtime; /* file creation time */ | |
| 33 | − unsigned int fsflags; /* flags for FS_IOC_[SG]ETFLAGS */ | |
| 34 | 39 | #ifdef CONFIG_TMPFS_QUOTA |
| 35 | 40 | struct dquot *i_dquot[MAXQUOTAS]; |
| 36 | 41 | #endif |
| 37 | − struct offset_ctx dir_offsets; /* stable entry offsets */ | |
| 38 | 42 | struct inode vfs_inode; |
| 39 | 43 | }; |
| 40 | 44 | |
| @@ -4,6 +4,25 @@ · 3 unmodified lines | ||
| 4 | 4 | |
| 5 | 5 | #include <linux/atomic.h> |
| 6 | 6 | #include <linux/types.h> |
| 7 | +#include <linux/refcount.h> | |
| 8 | +#include <linux/completion.h> | |
| 9 | + | |
| 10 | +#define SHRINKER_UNIT_BITS BITS_PER_LONG | |
| 11 | + | |
| 12 | +/* | |
| 13 | + * Bitmap and deferred work of shrinker::id corresponding to memcg-aware | |
| 14 | + * shrinkers, which have elements charged to the memcg. | |
| 15 | + */ | |
| 16 | +struct shrinker_info_unit { | |
| 17 | + atomic_long_t nr_deferred[SHRINKER_UNIT_BITS]; | |
| 18 | + DECLARE_BITMAP(map, SHRINKER_UNIT_BITS); | |
| 19 | +}; | |
| 20 | + | |
| 21 | +struct shrinker_info { | |
| 22 | + struct rcu_head rcu; | |
| 23 | + int map_nr_max; | |
| 24 | + struct shrinker_info_unit *unit[]; | |
| 25 | +}; | |
| 7 | 26 | |
| 8 | 27 | /* |
| 9 | 28 | * This struct is used to pass information from page reclaim to the shrinkers. |
| @@ -70,6 +89,19 @@ struct shrinker { · 60 unmodified lines | ||
| 70 | 89 | int seeks; /* seeks to recreate an obj */ |
| 71 | 90 | unsigned flags; |
| 72 | 91 | |
| 92 | + /* | |
| 93 | + * The reference count of this shrinker. Registered shrinker have an | |
| 94 | + * initial refcount of 1, then the lookup operations are now allowed | |
| 95 | + * to use it via shrinker_try_get(). Later in the unregistration step, | |
| 96 | + * the initial refcount will be discarded, and will free the shrinker | |
| 97 | + * asynchronously via RCU after its refcount reaches 0. | |
| 98 | + */ | |
| 99 | + refcount_t refcount; | |
| 100 | + struct completion done; /* use to wait for refcount to reach 0 */ | |
| 101 | + struct rcu_head rcu; | |
| 102 | + | |
| 103 | + void *private_data; | |
| 104 | + | |
| 73 | 105 | /* These are for internal use */ |
| 74 | 106 | struct list_head list; |
| 75 | 107 | #ifdef CONFIG_MEMCG |
| @@ -86,48 +118,39 @@ struct shrinker { · 10 unmodified lines | ||
| 86 | 118 | }; |
| 87 | 119 | #define DEFAULT_SEEKS 2 /* A good number if you don't know better. */ |
| 88 | 120 | |
| 89 | −/* Flags */ | |
| 90 | −#define SHRINKER_REGISTERED (1 << 0) | |
| 91 | −#define SHRINKER_NUMA_AWARE (1 << 1) | |
| 92 | −#define SHRINKER_MEMCG_AWARE (1 << 2) | |
| 121 | +/* Internal flags */ | |
| 122 | +#define SHRINKER_REGISTERED BIT(0) | |
| 123 | +#define SHRINKER_ALLOCATED BIT(1) | |
| 124 | + | |
| 125 | +/* Flags for users to use */ | |
| 126 | +#define SHRINKER_NUMA_AWARE BIT(2) | |
| 127 | +#define SHRINKER_MEMCG_AWARE BIT(3) | |
| 93 | 128 | /* |
| 94 | 129 | * It just makes sense when the shrinker is also MEMCG_AWARE for now, |
| 95 | 130 | * non-MEMCG_AWARE shrinker should not have this flag set. |
| 96 | 131 | */ |
| 97 | −#define SHRINKER_NONSLAB (1 << 3) | |
| 132 | +#define SHRINKER_NONSLAB BIT(4) | |
| 98 | 133 | |
| 99 | −extern int __printf(2, 3) prealloc_shrinker(struct shrinker *shrinker, | |
| 100 | − const char *fmt, ...); | |
| 101 | −extern void register_shrinker_prepared(struct shrinker *shrinker); | |
| 102 | −extern int __printf(2, 3) register_shrinker(struct shrinker *shrinker, | |
| 103 | − const char *fmt, ...); | |
| 104 | −extern void unregister_shrinker(struct shrinker *shrinker); | |
| 105 | −extern void free_prealloced_shrinker(struct shrinker *shrinker); | |
| 106 | −extern void synchronize_shrinkers(void); | |
| 134 | +__printf(2, 3) | |
| 135 | +struct shrinker *shrinker_alloc(unsigned int flags, const char *fmt, ...); | |
| 136 | +void shrinker_register(struct shrinker *shrinker); | |
| 137 | +void shrinker_free(struct shrinker *shrinker); | |
| 107 | 138 | |
| 108 | −#ifdef CONFIG_SHRINKER_DEBUG | |
| 109 | −extern int shrinker_debugfs_add(struct shrinker *shrinker); | |
| 110 | −extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, | |
| 111 | − int *debugfs_id); | |
| 112 | −extern void shrinker_debugfs_remove(struct dentry *debugfs_entry, | |
| 113 | − int debugfs_id); | |
| 114 | −extern int __printf(2, 3) shrinker_debugfs_rename(struct shrinker *shrinker, | |
| 115 | − const char *fmt, ...); | |
| 116 | −#else /* CONFIG_SHRINKER_DEBUG */ | |
| 117 | −static inline int shrinker_debugfs_add(struct shrinker *shrinker) | |
| 118 | −{ | |
| 119 | − return 0; | |
| 120 | −} | |
| 121 | −static inline struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, | |
| 122 | − int *debugfs_id) | |
| 139 | +static inline bool shrinker_try_get(struct shrinker *shrinker) | |
| 123 | 140 | { |
| 124 | − *debugfs_id = -1; | |
| 125 | − return NULL; | |
| 141 | + return refcount_inc_not_zero(&shrinker->refcount); | |
| 126 | 142 | } |
| 127 | −static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, | |
| 128 | − int debugfs_id) | |
| 143 | + | |
| 144 | +static inline void shrinker_put(struct shrinker *shrinker) | |
| 129 | 145 | { |
| 146 | + if (refcount_dec_and_test(&shrinker->refcount)) | |
| 147 | + complete(&shrinker->done); | |
| 130 | 148 | } |
| 149 | + | |
| 150 | +#ifdef CONFIG_SHRINKER_DEBUG | |
| 151 | +extern int __printf(2, 3) shrinker_debugfs_rename(struct shrinker *shrinker, | |
| 152 | + const char *fmt, ...); | |
| 153 | +#else /* CONFIG_SHRINKER_DEBUG */ | |
| 131 | 154 | static inline __printf(2, 3) |
| 132 | 155 | int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) |
| 133 | 156 | { |
| @@ -161,11 +161,22 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) · 160 unmodified lines | ||
| 161 | 161 | } |
| 162 | 162 | |
| 163 | 163 | static inline bool vma_can_userfault(struct vm_area_struct *vma, |
| 164 | − unsigned long vm_flags) | |
| 164 | + unsigned long vm_flags, | |
| 165 | + bool wp_async) | |
| 165 | 166 | { |
| 167 | + vm_flags &= __VM_UFFD_FLAGS; | |
| 168 | + | |
| 166 | 169 | if ((vm_flags & VM_UFFD_MINOR) && |
| 167 | 170 | (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma))) |
| 168 | 171 | return false; |
| 172 | + | |
| 173 | + /* | |
| 174 | + * If wp async enabled, and WP is the only mode enabled, allow any | |
| 175 | + * memory type. | |
| 176 | + */ | |
| 177 | + if (wp_async && (vm_flags == VM_UFFD_WP)) | |
| 178 | + return true; | |
| 179 | + | |
| 169 | 180 | #ifndef CONFIG_PTE_MARKER_UFFD_WP |
| 170 | 181 | /* |
| 171 | 182 | * If user requested uffd-wp but not enabled pte markers for |
| @@ -175,6 +186,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma, · 3 unmodified lines | ||
| 175 | 186 | if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma)) |
| 176 | 187 | return false; |
| 177 | 188 | #endif |
| 189 | + | |
| 190 | + /* By default, allow any of anon|shmem|hugetlb */ | |
| 178 | 191 | return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) || |
| 179 | 192 | vma_is_shmem(vma); |
| 180 | 193 | } |
| @@ -197,6 +210,7 @@ extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, · 16 unmodified lines | ||
| 197 | 210 | extern void userfaultfd_unmap_complete(struct mm_struct *mm, |
| 198 | 211 | struct list_head *uf); |
| 199 | 212 | extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma); |
| 213 | +extern bool userfaultfd_wp_async(struct vm_area_struct *vma); | |
| 200 | 214 | |
| 201 | 215 | #else /* CONFIG_USERFAULTFD */ |
| 202 | 216 | |
| @@ -207,6 +221,13 @@ static inline vm_fault_t handle_userfault(struct vm_fault *vmf, · 4 unmodified lines | ||
| 207 | 221 | return VM_FAULT_SIGBUS; |
| 208 | 222 | } |
| 209 | 223 | |
| 224 | +static inline long uffd_wp_range(struct vm_area_struct *vma, | |
| 225 | + unsigned long start, unsigned long len, | |
| 226 | + bool enable_wp) | |
| 227 | +{ | |
| 228 | + return false; | |
| 229 | +} | |
| 230 | + | |
| 210 | 231 | static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma, |
| 211 | 232 | struct vm_userfaultfd_ctx vm_ctx) |
| 212 | 233 | { |
| @@ -297,6 +318,11 @@ static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma) · 84 unmodified lines | ||
| 297 | 318 | return false; |
| 298 | 319 | } |
| 299 | 320 | |
| 321 | +static inline bool userfaultfd_wp_async(struct vm_area_struct *vma) | |
| 322 | +{ | |
| 323 | + return false; | |
| 324 | +} | |
| 325 | + | |
| 300 | 326 | #endif /* CONFIG_USERFAULTFD */ |
| 301 | 327 | |
| 302 | 328 | static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma) |
| @@ -19,10 +19,9 @@ int default_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int · 18 unmodified lines | ||
| 19 | 19 | /* wait_queue_entry::flags */ |
| 20 | 20 | #define WQ_FLAG_EXCLUSIVE 0x01 |
| 21 | 21 | #define WQ_FLAG_WOKEN 0x02 |
| 22 | −#define WQ_FLAG_BOOKMARK 0x04 | |
| 23 | −#define WQ_FLAG_CUSTOM 0x08 | |
| 24 | −#define WQ_FLAG_DONE 0x10 | |
| 25 | −#define WQ_FLAG_PRIORITY 0x20 | |
| 22 | +#define WQ_FLAG_CUSTOM 0x04 | |
| 23 | +#define WQ_FLAG_DONE 0x08 | |
| 24 | +#define WQ_FLAG_PRIORITY 0x10 | |
| 26 | 25 | |
| 27 | 26 | /* |
| 28 | 27 | * A single wait-queue entry structure: |
| @@ -212,8 +211,6 @@ __remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry *wq · 183 unmodified lines | ||
| 212 | 211 | int __wake_up(struct wait_queue_head *wq_head, unsigned int mode, int nr, void *key); |
| 213 | 212 | void __wake_up_on_current_cpu(struct wait_queue_head *wq_head, unsigned int mode, void *key); |
| 214 | 213 | void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key); |
| 215 | −void __wake_up_locked_key_bookmark(struct wait_queue_head *wq_head, | |
| 216 | − unsigned int mode, void *key, wait_queue_entry_t *bookmark); | |
| 217 | 214 | void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key); |
| 218 | 215 | void __wake_up_locked_sync_key(struct wait_queue_head *wq_head, unsigned int mode, void *key); |
| 219 | 216 | void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr); |
| @@ -9,12 +9,51 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <linux/types.h> |
| 10 | 10 | #include <linux/tracepoint.h> |
| 11 | 11 | |
| 12 | +TRACE_EVENT_CONDITION(damos_before_apply, | |
| 13 | + | |
| 14 | + TP_PROTO(unsigned int context_idx, unsigned int scheme_idx, | |
| 15 | + unsigned int target_idx, struct damon_region *r, | |
| 16 | + unsigned int nr_regions, bool do_trace), | |
| 17 | + | |
| 18 | + TP_ARGS(context_idx, target_idx, scheme_idx, r, nr_regions, do_trace), | |
| 19 | + | |
| 20 | + TP_CONDITION(do_trace), | |
| 21 | + | |
| 22 | + TP_STRUCT__entry( | |
| 23 | + __field(unsigned int, context_idx) | |
| 24 | + __field(unsigned int, scheme_idx) | |
| 25 | + __field(unsigned long, target_idx) | |
| 26 | + __field(unsigned long, start) | |
| 27 | + __field(unsigned long, end) | |
| 28 | + __field(unsigned int, nr_accesses) | |
| 29 | + __field(unsigned int, age) | |
| 30 | + __field(unsigned int, nr_regions) | |
| 31 | + ), | |
| 32 | + | |
| 33 | + TP_fast_assign( | |
| 34 | + __entry->context_idx = context_idx; | |
| 35 | + __entry->scheme_idx = scheme_idx; | |
| 36 | + __entry->target_idx = target_idx; | |
| 37 | + __entry->start = r->ar.start; | |
| 38 | + __entry->end = r->ar.end; | |
| 39 | + __entry->nr_accesses = r->nr_accesses_bp / 10000; | |
| 40 | + __entry->age = r->age; | |
| 41 | + __entry->nr_regions = nr_regions; | |
| 42 | + ), | |
| 43 | + | |
| 44 | + TP_printk("ctx_idx=%u scheme_idx=%u target_idx=%lu nr_regions=%u %lu-%lu: %u %u", | |
| 45 | + __entry->context_idx, __entry->scheme_idx, | |
| 46 | + __entry->target_idx, __entry->nr_regions, | |
| 47 | + __entry->start, __entry->end, | |
| 48 | + __entry->nr_accesses, __entry->age) | |
| 49 | +); | |
| 50 | + | |
| 12 | 51 | TRACE_EVENT(damon_aggregated, |
| 13 | 52 | |
| 14 | − TP_PROTO(struct damon_target *t, unsigned int target_id, | |
| 15 | − struct damon_region *r, unsigned int nr_regions), | |
| 53 | + TP_PROTO(unsigned int target_id, struct damon_region *r, | |
| 54 | + unsigned int nr_regions), | |
| 16 | 55 | |
| 17 | − TP_ARGS(t, target_id, r, nr_regions), | |
| 56 | + TP_ARGS(target_id, r, nr_regions), | |
| 18 | 57 | |
| 19 | 58 | TP_STRUCT__entry( |
| 20 | 59 | __field(unsigned long, target_id) |
| @@ -49,10 +49,11 @@ TRACE_EVENT(mm_migrate_pages, · 48 unmodified lines | ||
| 49 | 49 | |
| 50 | 50 | TP_PROTO(unsigned long succeeded, unsigned long failed, |
| 51 | 51 | unsigned long thp_succeeded, unsigned long thp_failed, |
| 52 | − unsigned long thp_split, enum migrate_mode mode, int reason), | |
| 52 | + unsigned long thp_split, unsigned long large_folio_split, | |
| 53 | + enum migrate_mode mode, int reason), | |
| 53 | 54 | |
| 54 | 55 | TP_ARGS(succeeded, failed, thp_succeeded, thp_failed, |
| 55 | − thp_split, mode, reason), | |
| 56 | + thp_split, large_folio_split, mode, reason), | |
| 56 | 57 | |
| 57 | 58 | TP_STRUCT__entry( |
| 58 | 59 | __field( unsigned long, succeeded) |
| @@ -60,26 +61,29 @@ TRACE_EVENT(mm_migrate_pages, · 1 unmodified line | ||
| 60 | 61 | __field( unsigned long, thp_succeeded) |
| 61 | 62 | __field( unsigned long, thp_failed) |
| 62 | 63 | __field( unsigned long, thp_split) |
| 64 | + __field( unsigned long, large_folio_split) | |
| 63 | 65 | __field( enum migrate_mode, mode) |
| 64 | 66 | __field( int, reason) |
| 65 | 67 | ), |
| 66 | 68 | |
| 67 | 69 | TP_fast_assign( |
| 68 | − __entry->succeeded = succeeded; | |
| 69 | − __entry->failed = failed; | |
| 70 | − __entry->thp_succeeded = thp_succeeded; | |
| 71 | − __entry->thp_failed = thp_failed; | |
| 72 | − __entry->thp_split = thp_split; | |
| 73 | − __entry->mode = mode; | |
| 74 | − __entry->reason = reason; | |
| 70 | + __entry->succeeded = succeeded; | |
| 71 | + __entry->failed = failed; | |
| 72 | + __entry->thp_succeeded = thp_succeeded; | |
| 73 | + __entry->thp_failed = thp_failed; | |
| 74 | + __entry->thp_split = thp_split; | |
| 75 | + __entry->large_folio_split = large_folio_split; | |
| 76 | + __entry->mode = mode; | |
| 77 | + __entry->reason = reason; | |
| 75 | 78 | ), |
| 76 | 79 | |
| 77 | − TP_printk("nr_succeeded=%lu nr_failed=%lu nr_thp_succeeded=%lu nr_thp_failed=%lu nr_thp_split=%lu mode=%s reason=%s", | |
| 80 | + TP_printk("nr_succeeded=%lu nr_failed=%lu nr_thp_succeeded=%lu nr_thp_failed=%lu nr_thp_split=%lu nr_split=%lu mode=%s reason=%s", | |
| 78 | 81 | __entry->succeeded, |
| 79 | 82 | __entry->failed, |
| 80 | 83 | __entry->thp_succeeded, |
| 81 | 84 | __entry->thp_failed, |
| 82 | 85 | __entry->thp_split, |
| 86 | + __entry->large_folio_split, | |
| 83 | 87 | __print_symbolic(__entry->mode, MIGRATE_MODE), |
| 84 | 88 | __print_symbolic(__entry->reason, MIGRATE_REASON)) |
| 85 | 89 | ); |
| @@ -285,10 +285,9 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 284 unmodified lines | ||
| 285 | 285 | unsigned long nr_scanned, |
| 286 | 286 | unsigned long nr_skipped, |
| 287 | 287 | unsigned long nr_taken, |
| 288 | − isolate_mode_t isolate_mode, | |
| 289 | 288 | int lru), |
| 290 | 289 | |
| 291 | − TP_ARGS(highest_zoneidx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, isolate_mode, lru), | |
| 290 | + TP_ARGS(highest_zoneidx, order, nr_requested, nr_scanned, nr_skipped, nr_taken, lru), | |
| 292 | 291 | |
| 293 | 292 | TP_STRUCT__entry( |
| 294 | 293 | __field(int, highest_zoneidx) |
| @@ -297,7 +296,6 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 2 unmodified lines | ||
| 297 | 296 | __field(unsigned long, nr_scanned) |
| 298 | 297 | __field(unsigned long, nr_skipped) |
| 299 | 298 | __field(unsigned long, nr_taken) |
| 300 | − __field(unsigned int, isolate_mode) | |
| 301 | 299 | __field(int, lru) |
| 302 | 300 | ), |
| 303 | 301 | |
| @@ -308,7 +306,6 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 4 unmodified lines | ||
| 308 | 306 | __entry->nr_scanned = nr_scanned; |
| 309 | 307 | __entry->nr_skipped = nr_skipped; |
| 310 | 308 | __entry->nr_taken = nr_taken; |
| 311 | − __entry->isolate_mode = (__force unsigned int)isolate_mode; | |
| 312 | 309 | __entry->lru = lru; |
| 313 | 310 | ), |
| 314 | 311 | |
| @@ -316,8 +313,7 @@ TRACE_EVENT(mm_vmscan_lru_isolate, · 1 unmodified line | ||
| 316 | 313 | * classzone is previous name of the highest_zoneidx. |
| 317 | 314 | * Reason not to change it is the ABI requirement of the tracepoint. |
| 318 | 315 | */ |
| 319 | − TP_printk("isolate_mode=%d classzone=%d order=%d nr_requested=%lu nr_scanned=%lu nr_skipped=%lu nr_taken=%lu lru=%s", | |
| 320 | − __entry->isolate_mode, | |
| 316 | + TP_printk("classzone=%d order=%d nr_requested=%lu nr_scanned=%lu nr_skipped=%lu nr_taken=%lu lru=%s", | |
| 321 | 317 | __entry->highest_zoneidx, |
| 322 | 318 | __entry->order, |
| 323 | 319 | __entry->nr_requested, |
| @@ -305,4 +305,63 @@ typedef int __bitwise __kernel_rwf_t; · 304 unmodified lines | ||
| 305 | 305 | #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ |
| 306 | 306 | RWF_APPEND) |
| 307 | 307 | |
| 308 | +/* Pagemap ioctl */ | |
| 309 | +#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg) | |
| 310 | + | |
| 311 | +/* Bitmasks provided in pm_scan_args masks and reported in page_region.categories. */ | |
| 312 | +#define PAGE_IS_WPALLOWED (1 << 0) | |
| 313 | +#define PAGE_IS_WRITTEN (1 << 1) | |
| 314 | +#define PAGE_IS_FILE (1 << 2) | |
| 315 | +#define PAGE_IS_PRESENT (1 << 3) | |
| 316 | +#define PAGE_IS_SWAPPED (1 << 4) | |
| 317 | +#define PAGE_IS_PFNZERO (1 << 5) | |
| 318 | +#define PAGE_IS_HUGE (1 << 6) | |
| 319 | + | |
| 320 | +/* | |
| 321 | + * struct page_region - Page region with flags | |
| 322 | + * @start: Start of the region | |
| 323 | + * @end: End of the region (exclusive) | |
| 324 | + * @categories: PAGE_IS_* category bitmask for the region | |
| 325 | + */ | |
| 326 | +struct page_region { | |
| 327 | + __u64 start; | |
| 328 | + __u64 end; | |
| 329 | + __u64 categories; | |
| 330 | +}; | |
| 331 | + | |
| 332 | +/* Flags for PAGEMAP_SCAN ioctl */ | |
| 333 | +#define PM_SCAN_WP_MATCHING (1 << 0) /* Write protect the pages matched. */ | |
| 334 | +#define PM_SCAN_CHECK_WPASYNC (1 << 1) /* Abort the scan when a non-WP-enabled page is found. */ | |
| 335 | + | |
| 336 | +/* | |
| 337 | + * struct pm_scan_arg - Pagemap ioctl argument | |
| 338 | + * @size: Size of the structure | |
| 339 | + * @flags: Flags for the IOCTL | |
| 340 | + * @start: Starting address of the region | |
| 341 | + * @end: Ending address of the region | |
| 342 | + * @walk_end Address where the scan stopped (written by kernel). | |
| 343 | + * walk_end == end (address tags cleared) informs that the scan completed on entire range. | |
| 344 | + * @vec: Address of page_region struct array for output | |
| 345 | + * @vec_len: Length of the page_region struct array | |
| 346 | + * @max_pages: Optional limit for number of returned pages (0 = disabled) | |
| 347 | + * @category_inverted: PAGE_IS_* categories which values match if 0 instead of 1 | |
| 348 | + * @category_mask: Skip pages for which any category doesn't match | |
| 349 | + * @category_anyof_mask: Skip pages for which no category matches | |
| 350 | + * @return_mask: PAGE_IS_* categories that are to be reported in `page_region`s returned | |
| 351 | + */ | |
| 352 | +struct pm_scan_arg { | |
| 353 | + __u64 size; | |
| 354 | + __u64 flags; | |
| 355 | + __u64 start; | |
| 356 | + __u64 end; | |
| 357 | + __u64 walk_end; | |
| 358 | + __u64 vec; | |
| 359 | + __u64 vec_len; | |
| 360 | + __u64 max_pages; | |
| 361 | + __u64 category_inverted; | |
| 362 | + __u64 category_mask; | |
| 363 | + __u64 category_anyof_mask; | |
| 364 | + __u64 return_mask; | |
| 365 | +}; | |
| 366 | + | |
| 308 | 367 | #endif /* _UAPI_LINUX_FS_H */ |
| @@ -48,7 +48,7 @@ enum { · 47 unmodified lines | ||
| 48 | 48 | #define MPOL_MF_MOVE (1<<1) /* Move pages owned by this process to conform |
| 49 | 49 | to policy */ |
| 50 | 50 | #define MPOL_MF_MOVE_ALL (1<<2) /* Move every page to conform to policy */ |
| 51 | −#define MPOL_MF_LAZY (1<<3) /* Modifies '_MOVE: lazy migrate on fault */ | |
| 51 | +#define MPOL_MF_LAZY (1<<3) /* UNSUPPORTED FLAG: Lazy migrate on fault */ | |
| 52 | 52 | #define MPOL_MF_INTERNAL (1<<4) /* Internal flags start here */ |
| 53 | 53 | |
| 54 | 54 | #define MPOL_MF_VALID (MPOL_MF_STRICT | \ |
| @@ -283,7 +283,8 @@ struct prctl_mm_map { · 282 unmodified lines | ||
| 283 | 283 | |
| 284 | 284 | /* Memory deny write / execute */ |
| 285 | 285 | #define PR_SET_MDWE 65 |
| 286 | −# define PR_MDWE_REFUSE_EXEC_GAIN 1 | |
| 286 | +# define PR_MDWE_REFUSE_EXEC_GAIN (1UL << 0) | |
| 287 | +# define PR_MDWE_NO_INHERIT (1UL << 1) | |
| 287 | 288 | |
| 288 | 289 | #define PR_GET_MDWE 66 |
| 289 | 290 | |
| @@ -40,7 +40,8 @@ · 39 unmodified lines | ||
| 40 | 40 | UFFD_FEATURE_EXACT_ADDRESS | \ |
| 41 | 41 | UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ |
| 42 | 42 | UFFD_FEATURE_WP_UNPOPULATED | \ |
| 43 | − UFFD_FEATURE_POISON) | |
| 43 | + UFFD_FEATURE_POISON | \ | |
| 44 | + UFFD_FEATURE_WP_ASYNC) | |
| 44 | 45 | #define UFFD_API_IOCTLS \ |
| 45 | 46 | ((__u64)1 << _UFFDIO_REGISTER | \ |
| 46 | 47 | (__u64)1 << _UFFDIO_UNREGISTER | \ |
| @@ -216,6 +217,11 @@ struct uffdio_api { · 169 unmodified lines | ||
| 216 | 217 | * (i.e. empty ptes). This will be the default behavior for shmem |
| 217 | 218 | * & hugetlbfs, so this flag only affects anonymous memory behavior |
| 218 | 219 | * when userfault write-protection mode is registered. |
| 220 | + * | |
| 221 | + * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection | |
| 222 | + * asynchronous mode is supported in which the write fault is | |
| 223 | + * automatically resolved and write-protection is un-set. | |
| 224 | + * It implies UFFD_FEATURE_WP_UNPOPULATED. | |
| 219 | 225 | */ |
| 220 | 226 | #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) |
| 221 | 227 | #define UFFD_FEATURE_EVENT_FORK (1<<1) |
| @@ -232,6 +238,7 @@ struct uffdio_api { · 10 unmodified lines | ||
| 232 | 238 | #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12) |
| 233 | 239 | #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) |
| 234 | 240 | #define UFFD_FEATURE_POISON (1<<14) |
| 241 | +#define UFFD_FEATURE_WP_ASYNC (1<<15) | |
| 235 | 242 | __u64 features; |
| 236 | 243 | |
| 237 | 244 | __u64 ioctls; |
| @@ -562,30 +562,25 @@ static unsigned long shm_pagesize(struct vm_area_struct *vma) · 561 unmodified lines | ||
| 562 | 562 | } |
| 563 | 563 | |
| 564 | 564 | #ifdef CONFIG_NUMA |
| 565 | −static int shm_set_policy(struct vm_area_struct *vma, struct mempolicy *new) | |
| 565 | +static int shm_set_policy(struct vm_area_struct *vma, struct mempolicy *mpol) | |
| 566 | 566 | { |
| 567 | − struct file *file = vma->vm_file; | |
| 568 | − struct shm_file_data *sfd = shm_file_data(file); | |
| 567 | + struct shm_file_data *sfd = shm_file_data(vma->vm_file); | |
| 569 | 568 | int err = 0; |
| 570 | 569 | |
| 571 | 570 | if (sfd->vm_ops->set_policy) |
| 572 | − err = sfd->vm_ops->set_policy(vma, new); | |
| 571 | + err = sfd->vm_ops->set_policy(vma, mpol); | |
| 573 | 572 | return err; |
| 574 | 573 | } |
| 575 | 574 | |
| 576 | 575 | static struct mempolicy *shm_get_policy(struct vm_area_struct *vma, |
| 577 | − unsigned long addr) | |
| 576 | + unsigned long addr, pgoff_t *ilx) | |
| 578 | 577 | { |
| 579 | − struct file *file = vma->vm_file; | |
| 580 | − struct shm_file_data *sfd = shm_file_data(file); | |
| 581 | − struct mempolicy *pol = NULL; | |
| 578 | + struct shm_file_data *sfd = shm_file_data(vma->vm_file); | |
| 579 | + struct mempolicy *mpol = vma->vm_policy; | |
| 582 | 580 | |
| 583 | 581 | if (sfd->vm_ops->get_policy) |
| 584 | − pol = sfd->vm_ops->get_policy(vma, addr); | |
| 585 | − else if (vma->vm_policy) | |
| 586 | − pol = vma->vm_policy; | |
| 587 | − | |
| 588 | − return pol; | |
| 582 | + mpol = sfd->vm_ops->get_policy(vma, addr, ilx); | |
| 583 | + return mpol; | |
| 589 | 584 | } |
| 590 | 585 | #endif |
| 591 | 586 | |
| @@ -1908,6 +1908,7 @@ enum cgroup2_param { · 1907 unmodified lines | ||
| 1908 | 1908 | Opt_favordynmods, |
| 1909 | 1909 | Opt_memory_localevents, |
| 1910 | 1910 | Opt_memory_recursiveprot, |
| 1911 | + Opt_memory_hugetlb_accounting, | |
| 1911 | 1912 | nr__cgroup2_params |
| 1912 | 1913 | }; |
| 1913 | 1914 | |
| @@ -1916,6 +1917,7 @@ static const struct fs_parameter_spec cgroup2_fs_parameters[] = { · 2 unmodified lines | ||
| 1916 | 1917 | fsparam_flag("favordynmods", Opt_favordynmods), |
| 1917 | 1918 | fsparam_flag("memory_localevents", Opt_memory_localevents), |
| 1918 | 1919 | fsparam_flag("memory_recursiveprot", Opt_memory_recursiveprot), |
| 1920 | + fsparam_flag("memory_hugetlb_accounting", Opt_memory_hugetlb_accounting), | |
| 1919 | 1921 | {} |
| 1920 | 1922 | }; |
| 1921 | 1923 | |
| @@ -1942,6 +1944,9 @@ static int cgroup2_parse_param(struct fs_context *fc, struct fs_parameter *param · 20 unmodified lines | ||
| 1942 | 1944 | case Opt_memory_recursiveprot: |
| 1943 | 1945 | ctx->flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT; |
| 1944 | 1946 | return 0; |
| 1947 | + case Opt_memory_hugetlb_accounting: | |
| 1948 | + ctx->flags |= CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING; | |
| 1949 | + return 0; | |
| 1945 | 1950 | } |
| 1946 | 1951 | return -EINVAL; |
| 1947 | 1952 | } |
| @@ -1966,6 +1971,11 @@ static void apply_cgroup_root_flags(unsigned int root_flags) · 18 unmodified lines | ||
| 1966 | 1971 | cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT; |
| 1967 | 1972 | else |
| 1968 | 1973 | cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_RECURSIVE_PROT; |
| 1974 | + | |
| 1975 | + if (root_flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING) | |
| 1976 | + cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING; | |
| 1977 | + else | |
| 1978 | + cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING; | |
| 1969 | 1979 | } |
| 1970 | 1980 | } |
| 1971 | 1981 | |
| @@ -1979,6 +1989,8 @@ static int cgroup_show_options(struct seq_file *seq, struct kernfs_root *kf_root · 7 unmodified lines | ||
| 1979 | 1989 | seq_puts(seq, ",memory_localevents"); |
| 1980 | 1990 | if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT) |
| 1981 | 1991 | seq_puts(seq, ",memory_recursiveprot"); |
| 1992 | + if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING) | |
| 1993 | + seq_puts(seq, ",memory_hugetlb_accounting"); | |
| 1982 | 1994 | return 0; |
| 1983 | 1995 | } |
| 1984 | 1996 | |
| @@ -7068,7 +7080,8 @@ static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr, · 5083 unmodified lines | ||
| 7068 | 7080 | "nsdelegate\n" |
| 7069 | 7081 | "favordynmods\n" |
| 7070 | 7082 | "memory_localevents\n" |
| 7071 | − "memory_recursiveprot\n"); | |
| 7083 | + "memory_recursiveprot\n" | |
| 7084 | + "memory_hugetlb_accounting\n"); | |
| 7072 | 7085 | } |
| 7073 | 7086 | static struct kobj_attribute cgroup_features_attr = __ATTR_RO(features); |
| 7074 | 7087 | |
| @@ -474,8 +474,8 @@ retry: · 473 unmodified lines | ||
| 474 | 474 | gup_flags |= FOLL_SPLIT_PMD; |
| 475 | 475 | /* Read the page with vaddr into memory */ |
| 476 | 476 | old_page = get_user_page_vma_remote(mm, vaddr, gup_flags, &vma); |
| 477 | − if (IS_ERR_OR_NULL(old_page)) | |
| 478 | − return old_page ? PTR_ERR(old_page) : 0; | |
| 477 | + if (IS_ERR(old_page)) | |
| 478 | + return PTR_ERR(old_page); | |
| 479 | 479 | |
| 480 | 480 | ret = verify_opcode(old_page, vaddr, &opcode); |
| 481 | 481 | if (ret <= 0) |
| @@ -541,7 +541,6 @@ static void exit_mm(void) · 540 unmodified lines | ||
| 541 | 541 | exit_mm_release(current, mm); |
| 542 | 542 | if (!mm) |
| 543 | 543 | return; |
| 544 | − sync_mm_rss(mm); | |
| 545 | 544 | mmap_read_lock(mm); |
| 546 | 545 | mmgrab_lazy_tlb(mm); |
| 547 | 546 | BUG_ON(mm != current->active_mm); |
| @@ -831,9 +830,6 @@ void __noreturn do_exit(long code) · 283 unmodified lines | ||
| 831 | 830 | io_uring_files_cancel(); |
| 832 | 831 | exit_signals(tsk); /* sets PF_EXITING */ |
| 833 | 832 | |
| 834 | − /* sync mm's RSS info before statistics gathering */ | |
| 835 | − if (tsk->mm) | |
| 836 | − sync_mm_rss(tsk->mm); | |
| 837 | 833 | acct_update_integrals(tsk); |
| 838 | 834 | group_dead = atomic_dec_and_test(&tsk->signal->live); |
| 839 | 835 | if (group_dead) { |
| @@ -733,7 +733,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, · 732 unmodified lines | ||
| 733 | 733 | |
| 734 | 734 | get_file(file); |
| 735 | 735 | i_mmap_lock_write(mapping); |
| 736 | − if (tmp->vm_flags & VM_SHARED) | |
| 736 | + if (vma_is_shared_maywrite(tmp)) | |
| 737 | 737 | mapping_allow_writable(mapping); |
| 738 | 738 | flush_dcache_mmap_lock(mapping); |
| 739 | 739 | /* insert tmp into the share list, just after mpnt */ |
| @@ -1288,7 +1288,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, · 548 unmodified lines | ||
| 1288 | 1288 | hugetlb_count_init(mm); |
| 1289 | 1289 | |
| 1290 | 1290 | if (current->mm) { |
| 1291 | − mm->flags = current->mm->flags & MMF_INIT_MASK; | |
| 1291 | + mm->flags = mmf_init_flags(current->mm->flags); | |
| 1292 | 1292 | mm->def_flags = current->mm->def_flags & VM_INIT_DEF_MASK; |
| 1293 | 1293 | } else { |
| 1294 | 1294 | mm->flags = default_dump_filter; |
| @@ -2411,10 +2411,6 @@ __latent_entropy struct task_struct *copy_process( · 1116 unmodified lines | ||
| 2411 | 2411 | p->io_uring = NULL; |
| 2412 | 2412 | #endif |
| 2413 | 2413 | |
| 2414 | −#if defined(SPLIT_RSS_COUNTING) | |
| 2415 | − memset(&p->rss_stat, 0, sizeof(p->rss_stat)); | |
| 2416 | −#endif | |
| 2417 | − | |
| 2418 | 2414 | p->default_timer_slack_ns = current->timer_slack_ns; |
| 2419 | 2415 | |
| 2420 | 2416 | #ifdef CONFIG_PSI |
| @@ -699,12 +699,9 @@ static void test_barrier_nothreads(struct kunit *test) · 698 unmodified lines | ||
| 699 | 699 | KCSAN_EXPECT_RW_BARRIER(spin_unlock(&test_spinlock), true); |
| 700 | 700 | KCSAN_EXPECT_RW_BARRIER(mutex_lock(&test_mutex), false); |
| 701 | 701 | KCSAN_EXPECT_RW_BARRIER(mutex_unlock(&test_mutex), true); |
| 702 | − | |
| 703 | −#ifdef clear_bit_unlock_is_negative_byte | |
| 704 | − KCSAN_EXPECT_READ_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true); | |
| 705 | − KCSAN_EXPECT_WRITE_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true); | |
| 706 | − KCSAN_EXPECT_RW_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true); | |
| 707 | −#endif | |
| 702 | + KCSAN_EXPECT_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); | |
| 703 | + KCSAN_EXPECT_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); | |
| 704 | + KCSAN_EXPECT_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); | |
| 708 | 705 | kcsan_nestable_atomic_end(); |
| 709 | 706 | } |
| 710 | 707 | |
| @@ -227,12 +227,9 @@ static bool __init test_barrier(void) · 226 unmodified lines | ||
| 227 | 227 | KCSAN_CHECK_RW_BARRIER(arch_spin_unlock(&arch_spinlock)); |
| 228 | 228 | spin_lock(&test_spinlock); |
| 229 | 229 | KCSAN_CHECK_RW_BARRIER(spin_unlock(&test_spinlock)); |
| 230 | − | |
| 231 | −#ifdef clear_bit_unlock_is_negative_byte | |
| 232 | − KCSAN_CHECK_RW_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var)); | |
| 233 | − KCSAN_CHECK_READ_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var)); | |
| 234 | − KCSAN_CHECK_WRITE_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var)); | |
| 235 | −#endif | |
| 230 | + KCSAN_CHECK_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); | |
| 231 | + KCSAN_CHECK_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); | |
| 232 | + KCSAN_CHECK_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); | |
| 236 | 233 | kcsan_nestable_atomic_end(); |
| 237 | 234 | |
| 238 | 235 | return ret; |
| @@ -1469,7 +1469,6 @@ void kthread_unuse_mm(struct mm_struct *mm) · 1468 unmodified lines | ||
| 1469 | 1469 | * clearing tsk->mm. |
| 1470 | 1470 | */ |
| 1471 | 1471 | smp_mb__after_spinlock(); |
| 1472 | − sync_mm_rss(mm); | |
| 1473 | 1472 | local_irq_disable(); |
| 1474 | 1473 | tsk->mm = NULL; |
| 1475 | 1474 | membarrier_update_current_mm(NULL); |
| @@ -59,7 +59,7 @@ int ptrace_access_vm(struct task_struct *tsk, unsigned long addr, · 58 unmodified lines | ||
| 59 | 59 | return 0; |
| 60 | 60 | } |
| 61 | 61 | |
| 62 | − ret = __access_remote_vm(mm, addr, buf, len, gup_flags); | |
| 62 | + ret = access_remote_vm(mm, addr, buf, len, gup_flags); | |
| 63 | 63 | mmput(mm); |
| 64 | 64 | |
| 65 | 65 | return ret; |
| @@ -3471,13 +3471,6 @@ kfree_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 3470 unmodified lines | ||
| 3471 | 3471 | return freed == 0 ? SHRINK_STOP : freed; |
| 3472 | 3472 | } |
| 3473 | 3473 | |
| 3474 | −static struct shrinker kfree_rcu_shrinker = { | |
| 3475 | − .count_objects = kfree_rcu_shrink_count, | |
| 3476 | − .scan_objects = kfree_rcu_shrink_scan, | |
| 3477 | − .batch = 0, | |
| 3478 | − .seeks = DEFAULT_SEEKS, | |
| 3479 | −}; | |
| 3480 | − | |
| 3481 | 3474 | void __init kfree_rcu_scheduler_running(void) |
| 3482 | 3475 | { |
| 3483 | 3476 | int cpu; |
| @@ -5029,6 +5022,7 @@ static void __init kfree_rcu_batch_init(void) · 1545 unmodified lines | ||
| 5029 | 5022 | { |
| 5030 | 5023 | int cpu; |
| 5031 | 5024 | int i, j; |
| 5025 | + struct shrinker *kfree_rcu_shrinker; | |
| 5032 | 5026 | |
| 5033 | 5027 | /* Clamp it to [0:100] seconds interval. */ |
| 5034 | 5028 | if (rcu_delay_page_cache_fill_msec < 0 || |
| @@ -5060,8 +5054,17 @@ static void __init kfree_rcu_batch_init(void) · 25 unmodified lines | ||
| 5060 | 5054 | INIT_DELAYED_WORK(&krcp->page_cache_work, fill_page_cache_func); |
| 5061 | 5055 | krcp->initialized = true; |
| 5062 | 5056 | } |
| 5063 | − if (register_shrinker(&kfree_rcu_shrinker, "rcu-kfree")) | |
| 5064 | − pr_err("Failed to register kfree_rcu() shrinker!\n"); | |
| 5057 | + | |
| 5058 | + kfree_rcu_shrinker = shrinker_alloc(0, "rcu-kfree"); | |
| 5059 | + if (!kfree_rcu_shrinker) { | |
| 5060 | + pr_err("Failed to allocate kfree_rcu() shrinker!\n"); | |
| 5061 | + return; | |
| 5062 | + } | |
| 5063 | + | |
| 5064 | + kfree_rcu_shrinker->count_objects = kfree_rcu_shrink_count; | |
| 5065 | + kfree_rcu_shrinker->scan_objects = kfree_rcu_shrink_scan; | |
| 5066 | + | |
| 5067 | + shrinker_register(kfree_rcu_shrinker); | |
| 5065 | 5068 | } |
| 5066 | 5069 | |
| 5067 | 5070 | void __init rcu_init(void) |
| @@ -1396,13 +1396,6 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) · 1395 unmodified lines | ||
| 1396 | 1396 | |
| 1397 | 1397 | return count ? count : SHRINK_STOP; |
| 1398 | 1398 | } |
| 1399 | − | |
| 1400 | −static struct shrinker lazy_rcu_shrinker = { | |
| 1401 | − .count_objects = lazy_rcu_shrink_count, | |
| 1402 | − .scan_objects = lazy_rcu_shrink_scan, | |
| 1403 | − .batch = 0, | |
| 1404 | − .seeks = DEFAULT_SEEKS, | |
| 1405 | −}; | |
| 1406 | 1399 | #endif // #ifdef CONFIG_RCU_LAZY |
| 1407 | 1400 | |
| 1408 | 1401 | void __init rcu_init_nohz(void) |
| @@ -1410,6 +1403,7 @@ void __init rcu_init_nohz(void) · 1 unmodified line | ||
| 1410 | 1403 | int cpu; |
| 1411 | 1404 | struct rcu_data *rdp; |
| 1412 | 1405 | const struct cpumask *cpumask = NULL; |
| 1406 | + struct shrinker * __maybe_unused lazy_rcu_shrinker; | |
| 1413 | 1407 | |
| 1414 | 1408 | #if defined(CONFIG_NO_HZ_FULL) |
| 1415 | 1409 | if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask)) |
| @@ -1436,8 +1430,15 @@ void __init rcu_init_nohz(void) · 20 unmodified lines | ||
| 1436 | 1430 | return; |
| 1437 | 1431 | |
| 1438 | 1432 | #ifdef CONFIG_RCU_LAZY |
| 1439 | − if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) | |
| 1440 | − pr_err("Failed to register lazy_rcu shrinker!\n"); | |
| 1433 | + lazy_rcu_shrinker = shrinker_alloc(0, "rcu-lazy"); | |
| 1434 | + if (!lazy_rcu_shrinker) { | |
| 1435 | + pr_err("Failed to allocate lazy_rcu shrinker!\n"); | |
| 1436 | + } else { | |
| 1437 | + lazy_rcu_shrinker->count_objects = lazy_rcu_shrink_count; | |
| 1438 | + lazy_rcu_shrinker->scan_objects = lazy_rcu_shrink_scan; | |
| 1439 | + | |
| 1440 | + shrinker_register(lazy_rcu_shrinker); | |
| 1441 | + } | |
| 1441 | 1442 | #endif // #ifdef CONFIG_RCU_LAZY |
| 1442 | 1443 | |
| 1443 | 1444 | if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { |
| @@ -1759,12 +1759,12 @@ static bool pgdat_free_space_enough(struct pglist_data *pgdat) · 1758 unmodified lines | ||
| 1759 | 1759 | * The smaller the hint page fault latency, the higher the possibility |
| 1760 | 1760 | * for the page to be hot. |
| 1761 | 1761 | */ |
| 1762 | −static int numa_hint_fault_latency(struct page *page) | |
| 1762 | +static int numa_hint_fault_latency(struct folio *folio) | |
| 1763 | 1763 | { |
| 1764 | 1764 | int last_time, time; |
| 1765 | 1765 | |
| 1766 | 1766 | time = jiffies_to_msecs(jiffies); |
| 1767 | − last_time = xchg_page_access_time(page, time); | |
| 1767 | + last_time = folio_xchg_access_time(folio, time); | |
| 1768 | 1768 | |
| 1769 | 1769 | return (time - last_time) & PAGE_ACCESS_TIME_MASK; |
| 1770 | 1770 | } |
| @@ -1821,7 +1821,7 @@ static void numa_promotion_adjust_threshold(struct pglist_data *pgdat, · 50 unmodified lines | ||
| 1821 | 1821 | } |
| 1822 | 1822 | } |
| 1823 | 1823 | |
| 1824 | −bool should_numa_migrate_memory(struct task_struct *p, struct page * page, | |
| 1824 | +bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, | |
| 1825 | 1825 | int src_nid, int dst_cpu) |
| 1826 | 1826 | { |
| 1827 | 1827 | struct numa_group *ng = deref_curr_numa_group(p); |
| @@ -1851,16 +1851,16 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, · 23 unmodified lines | ||
| 1851 | 1851 | numa_promotion_adjust_threshold(pgdat, rate_limit, def_th); |
| 1852 | 1852 | |
| 1853 | 1853 | th = pgdat->nbp_threshold ? : def_th; |
| 1854 | − latency = numa_hint_fault_latency(page); | |
| 1854 | + latency = numa_hint_fault_latency(folio); | |
| 1855 | 1855 | if (latency >= th) |
| 1856 | 1856 | return false; |
| 1857 | 1857 | |
| 1858 | 1858 | return !numa_promotion_rate_limit(pgdat, rate_limit, |
| 1859 | − thp_nr_pages(page)); | |
| 1859 | + folio_nr_pages(folio)); | |
| 1860 | 1860 | } |
| 1861 | 1861 | |
| 1862 | 1862 | this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid); |
| 1863 | − last_cpupid = page_cpupid_xchg_last(page, this_cpupid); | |
| 1863 | + last_cpupid = folio_xchg_last_cpupid(folio, this_cpupid); | |
| 1864 | 1864 | |
| 1865 | 1865 | if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && |
| 1866 | 1866 | !node_is_toptier(src_nid) && !cpupid_valid(last_cpupid)) |
| @@ -57,13 +57,6 @@ void remove_wait_queue(struct wait_queue_head *wq_head, struct wait_queue_entry · 56 unmodified lines | ||
| 57 | 57 | } |
| 58 | 58 | EXPORT_SYMBOL(remove_wait_queue); |
| 59 | 59 | |
| 60 | −/* | |
| 61 | − * Scan threshold to break wait queue walk. | |
| 62 | − * This allows a waker to take a break from holding the | |
| 63 | − * wait queue lock during the wait queue walk. | |
| 64 | − */ | |
| 65 | −#define WAITQUEUE_WALK_BREAK_CNT 64 | |
| 66 | − | |
| 67 | 60 | /* |
| 68 | 61 | * The core wakeup function. Non-exclusive wakeups (nr_exclusive == 0) just |
| 69 | 62 | * wake everything up. If it's an exclusive wakeup (nr_exclusive == small +ve |
| @@ -78,21 +71,13 @@ EXPORT_SYMBOL(remove_wait_queue); · 8 unmodified lines | ||
| 78 | 71 | * zero in this (rare) case, and we handle it by continuing to scan the queue. |
| 79 | 72 | */ |
| 80 | 73 | static int __wake_up_common(struct wait_queue_head *wq_head, unsigned int mode, |
| 81 | − int nr_exclusive, int wake_flags, void *key, | |
| 82 | − wait_queue_entry_t *bookmark) | |
| 74 | + int nr_exclusive, int wake_flags, void *key) | |
| 83 | 75 | { |
| 84 | 76 | wait_queue_entry_t *curr, *next; |
| 85 | − int cnt = 0; | |
| 86 | 77 | |
| 87 | 78 | lockdep_assert_held(&wq_head->lock); |
| 88 | 79 | |
| 89 | − if (bookmark && (bookmark->flags & WQ_FLAG_BOOKMARK)) { | |
| 90 | − curr = list_next_entry(bookmark, entry); | |
| 91 | − | |
| 92 | − list_del(&bookmark->entry); | |
| 93 | − bookmark->flags = 0; | |
| 94 | − } else | |
| 95 | − curr = list_first_entry(&wq_head->head, wait_queue_entry_t, entry); | |
| 80 | + curr = list_first_entry(&wq_head->head, wait_queue_entry_t, entry); | |
| 96 | 81 | |
| 97 | 82 | if (&curr->entry == &wq_head->head) |
| 98 | 83 | return nr_exclusive; |
| @@ -101,21 +86,11 @@ static int __wake_up_common(struct wait_queue_head *wq_head, unsigned int mode, · 2 unmodified lines | ||
| 101 | 86 | unsigned flags = curr->flags; |
| 102 | 87 | int ret; |
| 103 | 88 | |
| 104 | − if (flags & WQ_FLAG_BOOKMARK) | |
| 105 | − continue; | |
| 106 | − | |
| 107 | 89 | ret = curr->func(curr, mode, wake_flags, key); |
| 108 | 90 | if (ret < 0) |
| 109 | 91 | break; |
| 110 | 92 | if (ret && (flags & WQ_FLAG_EXCLUSIVE) && !--nr_exclusive) |
| 111 | 93 | break; |
| 112 | − | |
| 113 | − if (bookmark && (++cnt > WAITQUEUE_WALK_BREAK_CNT) && | |
| 114 | − (&next->entry != &wq_head->head)) { | |
| 115 | − bookmark->flags = WQ_FLAG_BOOKMARK; | |
| 116 | − list_add_tail(&bookmark->entry, &next->entry); | |
| 117 | − break; | |
| 118 | − } | |
| 119 | 94 | } |
| 120 | 95 | |
| 121 | 96 | return nr_exclusive; |
| @@ -125,20 +100,12 @@ static int __wake_up_common_lock(struct wait_queue_head *wq_head, unsigned int m · 3 unmodified lines | ||
| 125 | 100 | int nr_exclusive, int wake_flags, void *key) |
| 126 | 101 | { |
| 127 | 102 | unsigned long flags; |
| 128 | − wait_queue_entry_t bookmark; | |
| 129 | − int remaining = nr_exclusive; | |
| 103 | + int remaining; | |
| 130 | 104 | |
| 131 | − bookmark.flags = 0; | |
| 132 | − bookmark.private = NULL; | |
| 133 | − bookmark.func = NULL; | |
| 134 | − INIT_LIST_HEAD(&bookmark.entry); | |
| 135 | − | |
| 136 | − do { | |
| 137 | − spin_lock_irqsave(&wq_head->lock, flags); | |
| 138 | − remaining = __wake_up_common(wq_head, mode, remaining, | |
| 139 | − wake_flags, key, &bookmark); | |
| 140 | − spin_unlock_irqrestore(&wq_head->lock, flags); | |
| 141 | − } while (bookmark.flags & WQ_FLAG_BOOKMARK); | |
| 105 | + spin_lock_irqsave(&wq_head->lock, flags); | |
| 106 | + remaining = __wake_up_common(wq_head, mode, nr_exclusive, wake_flags, | |
| 107 | + key); | |
| 108 | + spin_unlock_irqrestore(&wq_head->lock, flags); | |
| 142 | 109 | |
| 143 | 110 | return nr_exclusive - remaining; |
| 144 | 111 | } |
| @@ -171,23 +138,16 @@ void __wake_up_on_current_cpu(struct wait_queue_head *wq_head, unsigned int mode · 26 unmodified lines | ||
| 171 | 138 | */ |
| 172 | 139 | void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr) |
| 173 | 140 | { |
| 174 | − __wake_up_common(wq_head, mode, nr, 0, NULL, NULL); | |
| 141 | + __wake_up_common(wq_head, mode, nr, 0, NULL); | |
| 175 | 142 | } |
| 176 | 143 | EXPORT_SYMBOL_GPL(__wake_up_locked); |
| 177 | 144 | |
| 178 | 145 | void __wake_up_locked_key(struct wait_queue_head *wq_head, unsigned int mode, void *key) |
| 179 | 146 | { |
| 180 | − __wake_up_common(wq_head, mode, 1, 0, key, NULL); | |
| 147 | + __wake_up_common(wq_head, mode, 1, 0, key); | |
| 181 | 148 | } |
| 182 | 149 | EXPORT_SYMBOL_GPL(__wake_up_locked_key); |
| 183 | 150 | |
| 184 | −void __wake_up_locked_key_bookmark(struct wait_queue_head *wq_head, | |
| 185 | − unsigned int mode, void *key, wait_queue_entry_t *bookmark) | |
| 186 | −{ | |
| 187 | − __wake_up_common(wq_head, mode, 1, 0, key, bookmark); | |
| 188 | −} | |
| 189 | −EXPORT_SYMBOL_GPL(__wake_up_locked_key_bookmark); | |
| 190 | − | |
| 191 | 151 | /** |
| 192 | 152 | * __wake_up_sync_key - wake up threads blocked on a waitqueue. |
| 193 | 153 | * @wq_head: the waitqueue |
| @@ -233,7 +193,7 @@ EXPORT_SYMBOL_GPL(__wake_up_sync_key); · 39 unmodified lines | ||
| 233 | 193 | void __wake_up_locked_sync_key(struct wait_queue_head *wq_head, |
| 234 | 194 | unsigned int mode, void *key) |
| 235 | 195 | { |
| 236 | − __wake_up_common(wq_head, mode, 1, WF_SYNC, key, NULL); | |
| 196 | + __wake_up_common(wq_head, mode, 1, WF_SYNC, key); | |
| 237 | 197 | } |
| 238 | 198 | EXPORT_SYMBOL_GPL(__wake_up_locked_sync_key); |
| 239 | 199 | |
| @@ -2368,19 +2368,41 @@ static int prctl_set_vma(unsigned long opt, unsigned long start, · 2367 unmodified lines | ||
| 2368 | 2368 | } |
| 2369 | 2369 | #endif /* CONFIG_ANON_VMA_NAME */ |
| 2370 | 2370 | |
| 2371 | +static inline unsigned long get_current_mdwe(void) | |
| 2372 | +{ | |
| 2373 | + unsigned long ret = 0; | |
| 2374 | + | |
| 2375 | + if (test_bit(MMF_HAS_MDWE, ¤t->mm->flags)) | |
| 2376 | + ret |= PR_MDWE_REFUSE_EXEC_GAIN; | |
| 2377 | + if (test_bit(MMF_HAS_MDWE_NO_INHERIT, ¤t->mm->flags)) | |
| 2378 | + ret |= PR_MDWE_NO_INHERIT; | |
| 2379 | + | |
| 2380 | + return ret; | |
| 2381 | +} | |
| 2382 | + | |
| 2371 | 2383 | static inline int prctl_set_mdwe(unsigned long bits, unsigned long arg3, |
| 2372 | 2384 | unsigned long arg4, unsigned long arg5) |
| 2373 | 2385 | { |
| 2386 | + unsigned long current_bits; | |
| 2387 | + | |
| 2374 | 2388 | if (arg3 || arg4 || arg5) |
| 2375 | 2389 | return -EINVAL; |
| 2376 | 2390 | |
| 2377 | − if (bits & ~(PR_MDWE_REFUSE_EXEC_GAIN)) | |
| 2391 | + if (bits & ~(PR_MDWE_REFUSE_EXEC_GAIN | PR_MDWE_NO_INHERIT)) | |
| 2392 | + return -EINVAL; | |
| 2393 | + | |
| 2394 | + /* NO_INHERIT only makes sense with REFUSE_EXEC_GAIN */ | |
| 2395 | + if (bits & PR_MDWE_NO_INHERIT && !(bits & PR_MDWE_REFUSE_EXEC_GAIN)) | |
| 2378 | 2396 | return -EINVAL; |
| 2379 | 2397 | |
| 2398 | + current_bits = get_current_mdwe(); | |
| 2399 | + if (current_bits && current_bits != bits) | |
| 2400 | + return -EPERM; /* Cannot unset the flags */ | |
| 2401 | + | |
| 2402 | + if (bits & PR_MDWE_NO_INHERIT) | |
| 2403 | + set_bit(MMF_HAS_MDWE_NO_INHERIT, ¤t->mm->flags); | |
| 2380 | 2404 | if (bits & PR_MDWE_REFUSE_EXEC_GAIN) |
| 2381 | 2405 | set_bit(MMF_HAS_MDWE, ¤t->mm->flags); |
| 2382 | − else if (test_bit(MMF_HAS_MDWE, ¤t->mm->flags)) | |
| 2383 | − return -EPERM; /* Cannot unset the flag */ | |
| 2384 | 2406 | |
| 2385 | 2407 | return 0; |
| 2386 | 2408 | } |
| @@ -2390,9 +2412,7 @@ static inline int prctl_get_mdwe(unsigned long arg2, unsigned long arg3, · 3 unmodified lines | ||
| 2390 | 2412 | { |
| 2391 | 2413 | if (arg2 || arg3 || arg4 || arg5) |
| 2392 | 2414 | return -EINVAL; |
| 2393 | − | |
| 2394 | − return test_bit(MMF_HAS_MDWE, ¤t->mm->flags) ? | |
| 2395 | − PR_MDWE_REFUSE_EXEC_GAIN : 0; | |
| 2415 | + return get_current_mdwe(); | |
| 2396 | 2416 | } |
| 2397 | 2417 | |
| 2398 | 2418 | static int prctl_get_auxv(void __user *addr, unsigned long len) |
| @@ -278,6 +278,85 @@ int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch) · 277 unmodified lines | ||
| 278 | 278 | } |
| 279 | 279 | EXPORT_SYMBOL(__percpu_counter_compare); |
| 280 | 280 | |
| 281 | +/* | |
| 282 | + * Compare counter, and add amount if total is: less than or equal to limit if | |
| 283 | + * amount is positive, or greater than or equal to limit if amount is negative. | |
| 284 | + * Return true if amount is added, or false if total would be beyond the limit. | |
| 285 | + * | |
| 286 | + * Negative limit is allowed, but unusual. | |
| 287 | + * When negative amounts (subs) are given to percpu_counter_limited_add(), | |
| 288 | + * the limit would most naturally be 0 - but other limits are also allowed. | |
| 289 | + * | |
| 290 | + * Overflow beyond S64_MAX is not allowed for: counter, limit and amount | |
| 291 | + * are all assumed to be sane (far from S64_MIN and S64_MAX). | |
| 292 | + */ | |
| 293 | +bool __percpu_counter_limited_add(struct percpu_counter *fbc, | |
| 294 | + s64 limit, s64 amount, s32 batch) | |
| 295 | +{ | |
| 296 | + s64 count; | |
| 297 | + s64 unknown; | |
| 298 | + unsigned long flags; | |
| 299 | + bool good = false; | |
| 300 | + | |
| 301 | + if (amount == 0) | |
| 302 | + return true; | |
| 303 | + | |
| 304 | + local_irq_save(flags); | |
| 305 | + unknown = batch * num_online_cpus(); | |
| 306 | + count = __this_cpu_read(*fbc->counters); | |
| 307 | + | |
| 308 | + /* Skip taking the lock when safe */ | |
| 309 | + if (abs(count + amount) <= batch && | |
| 310 | + ((amount > 0 && fbc->count + unknown <= limit) || | |
| 311 | + (amount < 0 && fbc->count - unknown >= limit))) { | |
| 312 | + this_cpu_add(*fbc->counters, amount); | |
| 313 | + local_irq_restore(flags); | |
| 314 | + return true; | |
| 315 | + } | |
| 316 | + | |
| 317 | + raw_spin_lock(&fbc->lock); | |
| 318 | + count = fbc->count + amount; | |
| 319 | + | |
| 320 | + /* Skip percpu_counter_sum() when safe */ | |
| 321 | + if (amount > 0) { | |
| 322 | + if (count - unknown > limit) | |
| 323 | + goto out; | |
| 324 | + if (count + unknown <= limit) | |
| 325 | + good = true; | |
| 326 | + } else { | |
| 327 | + if (count + unknown < limit) | |
| 328 | + goto out; | |
| 329 | + if (count - unknown >= limit) | |
| 330 | + good = true; | |
| 331 | + } | |
| 332 | + | |
| 333 | + if (!good) { | |
| 334 | + s32 *pcount; | |
| 335 | + int cpu; | |
| 336 | + | |
| 337 | + for_each_cpu_or(cpu, cpu_online_mask, cpu_dying_mask) { | |
| 338 | + pcount = per_cpu_ptr(fbc->counters, cpu); | |
| 339 | + count += *pcount; | |
| 340 | + } | |
| 341 | + if (amount > 0) { | |
| 342 | + if (count > limit) | |
| 343 | + goto out; | |
| 344 | + } else { | |
| 345 | + if (count < limit) | |
| 346 | + goto out; | |
| 347 | + } | |
| 348 | + good = true; | |
| 349 | + } | |
| 350 | + | |
| 351 | + count = __this_cpu_read(*fbc->counters); | |
| 352 | + fbc->count += count + amount; | |
| 353 | + __this_cpu_sub(*fbc->counters, count); | |
| 354 | +out: | |
| 355 | + raw_spin_unlock(&fbc->lock); | |
| 356 | + local_irq_restore(flags); | |
| 357 | + return good; | |
| 358 | +} | |
| 359 | + | |
| 281 | 360 | static int __init percpu_counter_startup(void) |
| 282 | 361 | { |
| 283 | 362 | int ret; |
| @@ -130,6 +130,7 @@ config ZSWAP_COMPRESSOR_DEFAULT · 129 unmodified lines | ||
| 130 | 130 | choice |
| 131 | 131 | prompt "Default allocator" |
| 132 | 132 | depends on ZSWAP |
| 133 | + default ZSWAP_ZPOOL_DEFAULT_ZSMALLOC if MMU | |
| 133 | 134 | default ZSWAP_ZPOOL_DEFAULT_ZBUD |
| 134 | 135 | help |
| 135 | 136 | Selects the default allocator for the compressed cache for |
| @@ -704,6 +705,17 @@ config HUGETLB_PAGE_SIZE_VARIABLE · 568 unmodified lines | ||
| 704 | 705 | config CONTIG_ALLOC |
| 705 | 706 | def_bool (MEMORY_ISOLATION && COMPACTION) || CMA |
| 706 | 707 | |
| 708 | +config PCP_BATCH_SCALE_MAX | |
| 709 | + int "Maximum scale factor of PCP (Per-CPU pageset) batch allocate/free" | |
| 710 | + default 5 | |
| 711 | + range 0 6 | |
| 712 | + help | |
| 713 | + In page allocator, PCP (Per-CPU pageset) is refilled and drained in | |
| 714 | + batches. The batch number is scaled automatically to improve page | |
| 715 | + allocation/free throughput. But too large scale factor may hurt | |
| 716 | + latency. This option sets the upper limit of scale factor to limit | |
| 717 | + the maximum latency. | |
| 718 | + | |
| 707 | 719 | config PHYS_ADDR_T_64BIT |
| 708 | 720 | def_bool 64BIT |
| 709 | 721 | |
| @@ -48,8 +48,8 @@ endif · 47 unmodified lines | ||
| 48 | 48 | |
| 49 | 49 | obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ |
| 50 | 50 | maccess.o page-writeback.o folio-compat.o \ |
| 51 | − readahead.o swap.o truncate.o vmscan.o shmem.o \ | |
| 52 | − util.o mmzone.o vmstat.o backing-dev.o \ | |
| 51 | + readahead.o swap.o truncate.o vmscan.o shrinker.o \ | |
| 52 | + shmem.o util.o mmzone.o vmstat.o backing-dev.o \ | |
| 53 | 53 | mm_init.o percpu.o slab_common.o \ |
| 54 | 54 | compaction.o show_mem.o shmem_quota.o\ |
| 55 | 55 | interval_tree.o list_lru.o workingset.o \ |
| @@ -34,7 +34,7 @@ void put_page_bootmem(struct page *page) · 33 unmodified lines | ||
| 34 | 34 | ClearPagePrivate(page); |
| 35 | 35 | set_page_private(page, 0); |
| 36 | 36 | INIT_LIST_HEAD(&page->lru); |
| 37 | − kmemleak_free_part(page_to_virt(page), PAGE_SIZE); | |
| 37 | + kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE); | |
| 38 | 38 | free_reserved_page(page); |
| 39 | 39 | } |
| 40 | 40 | } |
| @@ -505,7 +505,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, · 504 unmodified lines | ||
| 505 | 505 | */ |
| 506 | 506 | if (page) { |
| 507 | 507 | for (i = 0; i < count; i++) |
| 508 | − page_kasan_tag_reset(page + i); | |
| 508 | + page_kasan_tag_reset(nth_page(page, i)); | |
| 509 | 509 | } |
| 510 | 510 | |
| 511 | 511 | if (ret && !no_warn) { |
| @@ -382,6 +382,7 @@ static void __reset_isolation_suitable(struct zone *zone) · 381 unmodified lines | ||
| 382 | 382 | bool source_set = false; |
| 383 | 383 | bool free_set = false; |
| 384 | 384 | |
| 385 | + /* Only flush if a full compaction finished recently */ | |
| 385 | 386 | if (!zone->compact_blockskip_flush) |
| 386 | 387 | return; |
| 387 | 388 | |
| @@ -434,9 +435,7 @@ void reset_isolation_suitable(pg_data_t *pgdat) · 46 unmodified lines | ||
| 434 | 435 | if (!populated_zone(zone)) |
| 435 | 436 | continue; |
| 436 | 437 | |
| 437 | − /* Only flush if a full compaction finished recently */ | |
| 438 | − if (zone->compact_blockskip_flush) | |
| 439 | − __reset_isolation_suitable(zone); | |
| 438 | + __reset_isolation_suitable(zone); | |
| 440 | 439 | } |
| 441 | 440 | } |
| 442 | 441 | |
| @@ -626,11 +625,12 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, · 183 unmodified lines | ||
| 626 | 625 | if (PageCompound(page)) { |
| 627 | 626 | const unsigned int order = compound_order(page); |
| 628 | 627 | |
| 629 | − if (likely(order <= MAX_ORDER)) { | |
| 628 | + if (blockpfn + (1UL << order) <= end_pfn) { | |
| 630 | 629 | blockpfn += (1UL << order) - 1; |
| 631 | 630 | page += (1UL << order) - 1; |
| 632 | 631 | nr_scanned += (1UL << order) - 1; |
| 633 | 632 | } |
| 633 | + | |
| 634 | 634 | goto isolate_fail; |
| 635 | 635 | } |
| 636 | 636 | |
| @@ -678,8 +678,7 @@ isolate_fail: · 41 unmodified lines | ||
| 678 | 678 | spin_unlock_irqrestore(&cc->zone->lock, flags); |
| 679 | 679 | |
| 680 | 680 | /* |
| 681 | − * There is a tiny chance that we have read bogus compound_order(), | |
| 682 | − * so be careful to not go outside of the pageblock. | |
| 681 | + * Be careful to not go outside of the pageblock. | |
| 683 | 682 | */ |
| 684 | 683 | if (unlikely(blockpfn > end_pfn)) |
| 685 | 684 | blockpfn = end_pfn; |
| @@ -1395,8 +1394,8 @@ move_freelist_head(struct list_head *freelist, struct page *freepage) · 709 unmodified lines | ||
| 1395 | 1394 | { |
| 1396 | 1395 | LIST_HEAD(sublist); |
| 1397 | 1396 | |
| 1398 | − if (!list_is_last(freelist, &freepage->lru)) { | |
| 1399 | − list_cut_before(&sublist, freelist, &freepage->lru); | |
| 1397 | + if (!list_is_first(&freepage->buddy_list, freelist)) { | |
| 1398 | + list_cut_before(&sublist, freelist, &freepage->buddy_list); | |
| 1400 | 1399 | list_splice_tail(&sublist, freelist); |
| 1401 | 1400 | } |
| 1402 | 1401 | } |
| @@ -1412,8 +1411,8 @@ move_freelist_tail(struct list_head *freelist, struct page *freepage) · 9 unmodified lines | ||
| 1412 | 1411 | { |
| 1413 | 1412 | LIST_HEAD(sublist); |
| 1414 | 1413 | |
| 1415 | − if (!list_is_first(freelist, &freepage->lru)) { | |
| 1416 | − list_cut_position(&sublist, freelist, &freepage->lru); | |
| 1414 | + if (!list_is_last(&freepage->buddy_list, freelist)) { | |
| 1415 | + list_cut_position(&sublist, freelist, &freepage->buddy_list); | |
| 1417 | 1416 | list_splice_tail(&sublist, freelist); |
| 1418 | 1417 | } |
| 1419 | 1418 | } |
| @@ -2066,8 +2065,10 @@ static isolate_migrate_t isolate_migratepages(struct compact_control *cc) · 646 unmodified lines | ||
| 2066 | 2065 | } |
| 2067 | 2066 | |
| 2068 | 2067 | /* |
| 2069 | − * order == -1 is expected when compacting via | |
| 2070 | − * /proc/sys/vm/compact_memory | |
| 2068 | + * order == -1 is expected when compacting proactively via | |
| 2069 | + * 1. /proc/sys/vm/compact_memory | |
| 2070 | + * 2. /sys/devices/system/node/nodex/compact | |
| 2071 | + * 3. /proc/sys/vm/compaction_proactiveness | |
| 2071 | 2072 | */ |
| 2072 | 2073 | static inline bool is_via_compact_memory(int order) |
| 2073 | 2074 | { |
| @@ -2377,6 +2378,30 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, · 303 unmodified lines | ||
| 2377 | 2378 | return false; |
| 2378 | 2379 | } |
| 2379 | 2380 | |
| 2381 | +/* | |
| 2382 | + * Should we do compaction for target allocation order. | |
| 2383 | + * Return COMPACT_SUCCESS if allocation for target order can be already | |
| 2384 | + * satisfied | |
| 2385 | + * Return COMPACT_SKIPPED if compaction for target order is likely to fail | |
| 2386 | + * Return COMPACT_CONTINUE if compaction for target order should be ran | |
| 2387 | + */ | |
| 2388 | +static enum compact_result | |
| 2389 | +compaction_suit_allocation_order(struct zone *zone, unsigned int order, | |
| 2390 | + int highest_zoneidx, unsigned int alloc_flags) | |
| 2391 | +{ | |
| 2392 | + unsigned long watermark; | |
| 2393 | + | |
| 2394 | + watermark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); | |
| 2395 | + if (zone_watermark_ok(zone, order, watermark, highest_zoneidx, | |
| 2396 | + alloc_flags)) | |
| 2397 | + return COMPACT_SUCCESS; | |
| 2398 | + | |
| 2399 | + if (!compaction_suitable(zone, order, highest_zoneidx)) | |
| 2400 | + return COMPACT_SKIPPED; | |
| 2401 | + | |
| 2402 | + return COMPACT_CONTINUE; | |
| 2403 | +} | |
| 2404 | + | |
| 2380 | 2405 | static enum compact_result |
| 2381 | 2406 | compact_zone(struct compact_control *cc, struct capture_control *capc) |
| 2382 | 2407 | { |
| @@ -2402,19 +2427,11 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) · 19 unmodified lines | ||
| 2402 | 2427 | cc->migratetype = gfp_migratetype(cc->gfp_mask); |
| 2403 | 2428 | |
| 2404 | 2429 | if (!is_via_compact_memory(cc->order)) { |
| 2405 | − unsigned long watermark; | |
| 2406 | − | |
| 2407 | − /* Allocation can already succeed, nothing to do */ | |
| 2408 | − watermark = wmark_pages(cc->zone, | |
| 2409 | − cc->alloc_flags & ALLOC_WMARK_MASK); | |
| 2410 | − if (zone_watermark_ok(cc->zone, cc->order, watermark, | |
| 2411 | − cc->highest_zoneidx, cc->alloc_flags)) | |
| 2412 | − return COMPACT_SUCCESS; | |
| 2413 | − | |
| 2414 | − /* Compaction is likely to fail */ | |
| 2415 | − if (!compaction_suitable(cc->zone, cc->order, | |
| 2416 | − cc->highest_zoneidx)) | |
| 2417 | − return COMPACT_SKIPPED; | |
| 2430 | + ret = compaction_suit_allocation_order(cc->zone, cc->order, | |
| 2431 | + cc->highest_zoneidx, | |
| 2432 | + cc->alloc_flags); | |
| 2433 | + if (ret != COMPACT_CONTINUE) | |
| 2434 | + return ret; | |
| 2418 | 2435 | } |
| 2419 | 2436 | |
| 2420 | 2437 | /* |
| @@ -2913,6 +2930,7 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) · 492 unmodified lines | ||
| 2913 | 2930 | int zoneid; |
| 2914 | 2931 | struct zone *zone; |
| 2915 | 2932 | enum zone_type highest_zoneidx = pgdat->kcompactd_highest_zoneidx; |
| 2933 | + enum compact_result ret; | |
| 2916 | 2934 | |
| 2917 | 2935 | for (zoneid = 0; zoneid <= highest_zoneidx; zoneid++) { |
| 2918 | 2936 | zone = &pgdat->node_zones[zoneid]; |
| @@ -2920,14 +2938,10 @@ static bool kcompactd_node_suitable(pg_data_t *pgdat) · 1 unmodified line | ||
| 2920 | 2938 | if (!populated_zone(zone)) |
| 2921 | 2939 | continue; |
| 2922 | 2940 | |
| 2923 | − /* Allocation can already succeed, check other zones */ | |
| 2924 | − if (zone_watermark_ok(zone, pgdat->kcompactd_max_order, | |
| 2925 | − min_wmark_pages(zone), | |
| 2926 | − highest_zoneidx, 0)) | |
| 2927 | − continue; | |
| 2928 | − | |
| 2929 | − if (compaction_suitable(zone, pgdat->kcompactd_max_order, | |
| 2930 | − highest_zoneidx)) | |
| 2941 | + ret = compaction_suit_allocation_order(zone, | |
| 2942 | + pgdat->kcompactd_max_order, | |
| 2943 | + highest_zoneidx, ALLOC_WMARK_MIN); | |
| 2944 | + if (ret == COMPACT_CONTINUE) | |
| 2931 | 2945 | return true; |
| 2932 | 2946 | } |
| 2933 | 2947 | |
| @@ -2950,6 +2964,8 @@ static void kcompactd_do_work(pg_data_t *pgdat) · 16 unmodified lines | ||
| 2950 | 2964 | .ignore_skip_hint = false, |
| 2951 | 2965 | .gfp_mask = GFP_KERNEL, |
| 2952 | 2966 | }; |
| 2967 | + enum compact_result ret; | |
| 2968 | + | |
| 2953 | 2969 | trace_mm_compaction_kcompactd_wake(pgdat->node_id, cc.order, |
| 2954 | 2970 | cc.highest_zoneidx); |
| 2955 | 2971 | count_compact_event(KCOMPACTD_WAKE); |
| @@ -2964,12 +2980,9 @@ static void kcompactd_do_work(pg_data_t *pgdat) · 8 unmodified lines | ||
| 2964 | 2980 | if (compaction_deferred(zone, cc.order)) |
| 2965 | 2981 | continue; |
| 2966 | 2982 | |
| 2967 | − /* Allocation can already succeed, nothing to do */ | |
| 2968 | − if (zone_watermark_ok(zone, cc.order, | |
| 2969 | − min_wmark_pages(zone), zoneid, 0)) | |
| 2970 | − continue; | |
| 2971 | − | |
| 2972 | − if (!compaction_suitable(zone, cc.order, zoneid)) | |
| 2983 | + ret = compaction_suit_allocation_order(zone, | |
| 2984 | + cc.order, zoneid, ALLOC_WMARK_MIN); | |
| 2985 | + if (ret != COMPACT_CONTINUE) | |
| 2973 | 2986 | continue; |
| 2974 | 2987 | |
| 2975 | 2988 | if (kthread_should_stop()) |
| @@ -59,6 +59,18 @@ config DAMON_SYSFS · 58 unmodified lines | ||
| 59 | 59 | This builds the sysfs interface for DAMON. The user space can use |
| 60 | 60 | the interface for arbitrary data access monitoring. |
| 61 | 61 | |
| 62 | +config DAMON_SYSFS_KUNIT_TEST | |
| 63 | + bool "Test for damon debugfs interface" if !KUNIT_ALL_TESTS | |
| 64 | + depends on DAMON_SYSFS && KUNIT=y | |
| 65 | + default KUNIT_ALL_TESTS | |
| 66 | + help | |
| 67 | + This builds the DAMON sysfs interface Kunit test suite. | |
| 68 | + | |
| 69 | + For more information on KUnit and unit tests in general, please refer | |
| 70 | + to the KUnit documentation. | |
| 71 | + | |
| 72 | + If unsure, say N. | |
| 73 | + | |
| 62 | 74 | config DAMON_DBGFS |
| 63 | 75 | bool "DAMON debugfs interface (DEPRECATED!)" |
| 64 | 76 | depends on DAMON_VADDR && DAMON_PADDR && DEBUG_FS |
| @@ -30,7 +30,7 @@ static void damon_test_regions(struct kunit *test) · 29 unmodified lines | ||
| 30 | 30 | damon_add_region(r, t); |
| 31 | 31 | KUNIT_EXPECT_EQ(test, 1u, damon_nr_regions(t)); |
| 32 | 32 | |
| 33 | − damon_del_region(r, t); | |
| 33 | + damon_destroy_region(r, t); | |
| 34 | 34 | KUNIT_EXPECT_EQ(test, 0u, damon_nr_regions(t)); |
| 35 | 35 | |
| 36 | 36 | damon_free_target(t); |
| @@ -94,6 +94,7 @@ static void damon_test_aggregate(struct kunit *test) · 57 unmodified lines | ||
| 94 | 94 | for (ir = 0; ir < 3; ir++) { |
| 95 | 95 | r = damon_new_region(saddr[it][ir], eaddr[it][ir]); |
| 96 | 96 | r->nr_accesses = accesses[it][ir]; |
| 97 | + r->nr_accesses_bp = accesses[it][ir] * 10000; | |
| 97 | 98 | damon_add_region(r, t); |
| 98 | 99 | } |
| 99 | 100 | it++; |
| @@ -147,9 +148,11 @@ static void damon_test_merge_two(struct kunit *test) · 47 unmodified lines | ||
| 147 | 148 | t = damon_new_target(); |
| 148 | 149 | r = damon_new_region(0, 100); |
| 149 | 150 | r->nr_accesses = 10; |
| 151 | + r->nr_accesses_bp = 100000; | |
| 150 | 152 | damon_add_region(r, t); |
| 151 | 153 | r2 = damon_new_region(100, 300); |
| 152 | 154 | r2->nr_accesses = 20; |
| 155 | + r2->nr_accesses_bp = 200000; | |
| 153 | 156 | damon_add_region(r2, t); |
| 154 | 157 | |
| 155 | 158 | damon_merge_two_regions(t, r, r2); |
| @@ -196,6 +199,7 @@ static void damon_test_merge_regions_of(struct kunit *test) · 40 unmodified lines | ||
| 196 | 199 | for (i = 0; i < ARRAY_SIZE(sa); i++) { |
| 197 | 200 | r = damon_new_region(sa[i], ea[i]); |
| 198 | 201 | r->nr_accesses = nrs[i]; |
| 202 | + r->nr_accesses_bp = nrs[i] * 10000; | |
| 199 | 203 | damon_add_region(r, t); |
| 200 | 204 | } |
| 201 | 205 | |
| @@ -265,6 +269,8 @@ static void damon_test_ops_registration(struct kunit *test) · 63 unmodified lines | ||
| 265 | 269 | |
| 266 | 270 | /* Check double-registration failure again */ |
| 267 | 271 | KUNIT_EXPECT_EQ(test, damon_register_ops(&ops), -EINVAL); |
| 272 | + | |
| 273 | + damon_destroy_ctx(c); | |
| 268 | 274 | } |
| 269 | 275 | |
| 270 | 276 | static void damon_test_set_regions(struct kunit *test) |
| @@ -297,6 +303,7 @@ static void damon_test_update_monitoring_result(struct kunit *test) · 26 unmodified lines | ||
| 297 | 303 | struct damon_region *r = damon_new_region(3, 7); |
| 298 | 304 | |
| 299 | 305 | r->nr_accesses = 15; |
| 306 | + r->nr_accesses_bp = 150000; | |
| 300 | 307 | r->age = 20; |
| 301 | 308 | |
| 302 | 309 | new_attrs = (struct damon_attrs){ |
| @@ -316,6 +323,8 @@ static void damon_test_update_monitoring_result(struct kunit *test) · 13 unmodified lines | ||
| 316 | 323 | damon_update_monitoring_result(r, &old_attrs, &new_attrs); |
| 317 | 324 | KUNIT_EXPECT_EQ(test, r->nr_accesses, 150); |
| 318 | 325 | KUNIT_EXPECT_EQ(test, r->age, 20); |
| 326 | + | |
| 327 | + damon_free_region(r); | |
| 319 | 328 | } |
| 320 | 329 | |
| 321 | 330 | static void damon_test_set_attrs(struct kunit *test) |
| @@ -339,6 +348,23 @@ static void damon_test_set_attrs(struct kunit *test) · 17 unmodified lines | ||
| 339 | 348 | invalid_attrs = valid_attrs; |
| 340 | 349 | invalid_attrs.aggr_interval = 4999; |
| 341 | 350 | KUNIT_EXPECT_EQ(test, damon_set_attrs(c, &invalid_attrs), -EINVAL); |
| 351 | + | |
| 352 | + damon_destroy_ctx(c); | |
| 353 | +} | |
| 354 | + | |
| 355 | +static void damon_test_moving_sum(struct kunit *test) | |
| 356 | +{ | |
| 357 | + unsigned int mvsum = 50000, nomvsum = 50000, len_window = 10; | |
| 358 | + unsigned int new_values[] = {10000, 0, 10000, 0, 0, 0, 10000, 0, 0, 0}; | |
| 359 | + unsigned int expects[] = {55000, 50000, 55000, 50000, 45000, 40000, | |
| 360 | + 45000, 40000, 35000, 30000}; | |
| 361 | + int i; | |
| 362 | + | |
| 363 | + for (i = 0; i < ARRAY_SIZE(new_values); i++) { | |
| 364 | + mvsum = damon_moving_sum(mvsum, nomvsum, len_window, | |
| 365 | + new_values[i]); | |
| 366 | + KUNIT_EXPECT_EQ(test, mvsum, expects[i]); | |
| 367 | + } | |
| 342 | 368 | } |
| 343 | 369 | |
| 344 | 370 | static void damos_test_new_filter(struct kunit *test) |
| @@ -425,6 +451,7 @@ static struct kunit_case damon_test_cases[] = { · 80 unmodified lines | ||
| 425 | 451 | KUNIT_CASE(damon_test_set_regions), |
| 426 | 452 | KUNIT_CASE(damon_test_update_monitoring_result), |
| 427 | 453 | KUNIT_CASE(damon_test_set_attrs), |
| 454 | + KUNIT_CASE(damon_test_moving_sum), | |
| 428 | 455 | KUNIT_CASE(damos_test_new_filter), |
| 429 | 456 | KUNIT_CASE(damos_test_filter_out), |
| 430 | 457 | {}, |
| @@ -128,6 +128,7 @@ struct damon_region *damon_new_region(unsigned long start, unsigned long end) · 127 unmodified lines | ||
| 128 | 128 | region->ar.start = start; |
| 129 | 129 | region->ar.end = end; |
| 130 | 130 | region->nr_accesses = 0; |
| 131 | + region->nr_accesses_bp = 0; | |
| 131 | 132 | INIT_LIST_HEAD(®ion->list); |
| 132 | 133 | |
| 133 | 134 | region->age = 0; |
| @@ -312,7 +313,9 @@ static struct damos_quota *damos_quota_init_priv(struct damos_quota *quota) · 178 unmodified lines | ||
| 312 | 313 | } |
| 313 | 314 | |
| 314 | 315 | struct damos *damon_new_scheme(struct damos_access_pattern *pattern, |
| 315 | − enum damos_action action, struct damos_quota *quota, | |
| 316 | + enum damos_action action, | |
| 317 | + unsigned long apply_interval_us, | |
| 318 | + struct damos_quota *quota, | |
| 316 | 319 | struct damos_watermarks *wmarks) |
| 317 | 320 | { |
| 318 | 321 | struct damos *scheme; |
| @@ -322,6 +325,13 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 3 unmodified lines | ||
| 322 | 325 | return NULL; |
| 323 | 326 | scheme->pattern = *pattern; |
| 324 | 327 | scheme->action = action; |
| 328 | + scheme->apply_interval_us = apply_interval_us; | |
| 329 | + /* | |
| 330 | + * next_apply_sis will be set when kdamond starts. While kdamond is | |
| 331 | + * running, it will also updated when it is added to the DAMON context, | |
| 332 | + * or damon_attrs are updated. | |
| 333 | + */ | |
| 334 | + scheme->next_apply_sis = 0; | |
| 325 | 335 | INIT_LIST_HEAD(&scheme->filters); |
| 326 | 336 | scheme->stat = (struct damos_stat){}; |
| 327 | 337 | INIT_LIST_HEAD(&scheme->list); |
| @@ -334,9 +344,21 @@ struct damos *damon_new_scheme(struct damos_access_pattern *pattern, · 6 unmodified lines | ||
| 334 | 344 | return scheme; |
| 335 | 345 | } |
| 336 | 346 | |
| 347 | +static void damos_set_next_apply_sis(struct damos *s, struct damon_ctx *ctx) | |
| 348 | +{ | |
| 349 | + unsigned long sample_interval = ctx->attrs.sample_interval ? | |
| 350 | + ctx->attrs.sample_interval : 1; | |
| 351 | + unsigned long apply_interval = s->apply_interval_us ? | |
| 352 | + s->apply_interval_us : ctx->attrs.aggr_interval; | |
| 353 | + | |
| 354 | + s->next_apply_sis = ctx->passed_sample_intervals + | |
| 355 | + apply_interval / sample_interval; | |
| 356 | +} | |
| 357 | + | |
| 337 | 358 | void damon_add_scheme(struct damon_ctx *ctx, struct damos *s) |
| 338 | 359 | { |
| 339 | 360 | list_add_tail(&s->list, &ctx->schemes); |
| 361 | + damos_set_next_apply_sis(s, ctx); | |
| 340 | 362 | } |
| 341 | 363 | |
| 342 | 364 | static void damon_del_scheme(struct damos *s) |
| @@ -427,8 +449,10 @@ struct damon_ctx *damon_new_ctx(void) · 84 unmodified lines | ||
| 427 | 449 | ctx->attrs.aggr_interval = 100 * 1000; |
| 428 | 450 | ctx->attrs.ops_update_interval = 60 * 1000 * 1000; |
| 429 | 451 | |
| 430 | − ktime_get_coarse_ts64(&ctx->last_aggregation); | |
| 431 | − ctx->last_ops_update = ctx->last_aggregation; | |
| 452 | + ctx->passed_sample_intervals = 0; | |
| 453 | + /* These will be set from kdamond_init_intervals_sis() */ | |
| 454 | + ctx->next_aggregation_sis = 0; | |
| 455 | + ctx->next_ops_update_sis = 0; | |
| 432 | 456 | |
| 433 | 457 | mutex_init(&ctx->kdamond_lock); |
| 434 | 458 | |
| @@ -476,20 +500,14 @@ static unsigned int damon_age_for_new_attrs(unsigned int age, · 41 unmodified lines | ||
| 476 | 500 | static unsigned int damon_accesses_bp_to_nr_accesses( |
| 477 | 501 | unsigned int accesses_bp, struct damon_attrs *attrs) |
| 478 | 502 | { |
| 479 | − unsigned int max_nr_accesses = | |
| 480 | − attrs->aggr_interval / attrs->sample_interval; | |
| 481 | − | |
| 482 | − return accesses_bp * max_nr_accesses / 10000; | |
| 503 | + return accesses_bp * damon_max_nr_accesses(attrs) / 10000; | |
| 483 | 504 | } |
| 484 | 505 | |
| 485 | 506 | /* convert nr_accesses to access ratio in bp (per 10,000) */ |
| 486 | 507 | static unsigned int damon_nr_accesses_to_accesses_bp( |
| 487 | 508 | unsigned int nr_accesses, struct damon_attrs *attrs) |
| 488 | 509 | { |
| 489 | − unsigned int max_nr_accesses = | |
| 490 | − attrs->aggr_interval / attrs->sample_interval; | |
| 491 | − | |
| 492 | − return nr_accesses * 10000 / max_nr_accesses; | |
| 510 | + return nr_accesses * 10000 / damon_max_nr_accesses(attrs); | |
| 493 | 511 | } |
| 494 | 512 | |
| 495 | 513 | static unsigned int damon_nr_accesses_for_new_attrs(unsigned int nr_accesses, |
| @@ -506,6 +524,7 @@ static void damon_update_monitoring_result(struct damon_region *r, · 10 unmodified lines | ||
| 506 | 524 | { |
| 507 | 525 | r->nr_accesses = damon_nr_accesses_for_new_attrs(r->nr_accesses, |
| 508 | 526 | old_attrs, new_attrs); |
| 527 | + r->nr_accesses_bp = r->nr_accesses * 10000; | |
| 509 | 528 | r->age = damon_age_for_new_attrs(r->age, old_attrs, new_attrs); |
| 510 | 529 | } |
| 511 | 530 | |
| @@ -541,13 +560,21 @@ static void damon_update_monitoring_results(struct damon_ctx *ctx, · 29 unmodified lines | ||
| 541 | 560 | * @ctx: monitoring context |
| 542 | 561 | * @attrs: monitoring attributes |
| 543 | 562 | * |
| 544 | − * This function should not be called while the kdamond is running. | |
| 563 | + * This function should be called while the kdamond is not running, or an | |
| 564 | + * access check results aggregation is not ongoing (e.g., from | |
| 565 | + * &struct damon_callback->after_aggregation or | |
| 566 | + * &struct damon_callback->after_wmarks_check callbacks). | |
| 567 | + * | |
| 545 | 568 | * Every time interval is in micro-seconds. |
| 546 | 569 | * |
| 547 | 570 | * Return: 0 on success, negative error code otherwise. |
| 548 | 571 | */ |
| 549 | 572 | int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs) |
| 550 | 573 | { |
| 574 | + unsigned long sample_interval = attrs->sample_interval ? | |
| 575 | + attrs->sample_interval : 1; | |
| 576 | + struct damos *s; | |
| 577 | + | |
| 551 | 578 | if (attrs->min_nr_regions < 3) |
| 552 | 579 | return -EINVAL; |
| 553 | 580 | if (attrs->min_nr_regions > attrs->max_nr_regions) |
| @@ -555,8 +582,17 @@ int damon_set_attrs(struct damon_ctx *ctx, struct damon_attrs *attrs) · 1 unmodified line | ||
| 555 | 582 | if (attrs->sample_interval > attrs->aggr_interval) |
| 556 | 583 | return -EINVAL; |
| 557 | 584 | |
| 585 | + ctx->next_aggregation_sis = ctx->passed_sample_intervals + | |
| 586 | + attrs->aggr_interval / sample_interval; | |
| 587 | + ctx->next_ops_update_sis = ctx->passed_sample_intervals + | |
| 588 | + attrs->ops_update_interval / sample_interval; | |
| 589 | + | |
| 558 | 590 | damon_update_monitoring_results(ctx, attrs); |
| 559 | 591 | ctx->attrs = *attrs; |
| 592 | + | |
| 593 | + damon_for_each_scheme(s, ctx) | |
| 594 | + damos_set_next_apply_sis(s, ctx); | |
| 595 | + | |
| 560 | 596 | return 0; |
| 561 | 597 | } |
| 562 | 598 | |
| @@ -728,38 +764,6 @@ int damon_stop(struct damon_ctx **ctxs, int nr_ctxs) · 165 unmodified lines | ||
| 728 | 764 | return err; |
| 729 | 765 | } |
| 730 | 766 | |
| 731 | −/* | |
| 732 | − * damon_check_reset_time_interval() - Check if a time interval is elapsed. | |
| 733 | − * @baseline: the time to check whether the interval has elapsed since | |
| 734 | − * @interval: the time interval (microseconds) | |
| 735 | − * | |
| 736 | − * See whether the given time interval has passed since the given baseline | |
| 737 | − * time. If so, it also updates the baseline to current time for next check. | |
| 738 | − * | |
| 739 | − * Return: true if the time interval has passed, or false otherwise. | |
| 740 | − */ | |
| 741 | −static bool damon_check_reset_time_interval(struct timespec64 *baseline, | |
| 742 | − unsigned long interval) | |
| 743 | −{ | |
| 744 | − struct timespec64 now; | |
| 745 | − | |
| 746 | − ktime_get_coarse_ts64(&now); | |
| 747 | − if ((timespec64_to_ns(&now) - timespec64_to_ns(baseline)) < | |
| 748 | − interval * 1000) | |
| 749 | − return false; | |
| 750 | − *baseline = now; | |
| 751 | − return true; | |
| 752 | −} | |
| 753 | − | |
| 754 | −/* | |
| 755 | − * Check whether it is time to flush the aggregated information | |
| 756 | − */ | |
| 757 | −static bool kdamond_aggregate_interval_passed(struct damon_ctx *ctx) | |
| 758 | −{ | |
| 759 | − return damon_check_reset_time_interval(&ctx->last_aggregation, | |
| 760 | − ctx->attrs.aggr_interval); | |
| 761 | −} | |
| 762 | − | |
| 763 | 767 | /* |
| 764 | 768 | * Reset the aggregated monitoring results ('nr_accesses' of each region). |
| 765 | 769 | */ |
| @@ -772,7 +776,7 @@ static void kdamond_reset_aggregated(struct damon_ctx *c) · 6 unmodified lines | ||
| 772 | 776 | struct damon_region *r; |
| 773 | 777 | |
| 774 | 778 | damon_for_each_region(r, t) { |
| 775 | − trace_damon_aggregated(t, ti, r, damon_nr_regions(t)); | |
| 779 | + trace_damon_aggregated(ti, r, damon_nr_regions(t)); | |
| 776 | 780 | r->last_nr_accesses = r->nr_accesses; |
| 777 | 781 | r->nr_accesses = 0; |
| 778 | 782 | } |
| @@ -786,12 +790,13 @@ static void damon_split_region_at(struct damon_target *t, · 7 unmodified lines | ||
| 786 | 790 | static bool __damos_valid_target(struct damon_region *r, struct damos *s) |
| 787 | 791 | { |
| 788 | 792 | unsigned long sz; |
| 793 | + unsigned int nr_accesses = r->nr_accesses_bp / 10000; | |
| 789 | 794 | |
| 790 | 795 | sz = damon_sz_region(r); |
| 791 | 796 | return s->pattern.min_sz_region <= sz && |
| 792 | 797 | sz <= s->pattern.max_sz_region && |
| 793 | − s->pattern.min_nr_accesses <= r->nr_accesses && | |
| 794 | − r->nr_accesses <= s->pattern.max_nr_accesses && | |
| 798 | + s->pattern.min_nr_accesses <= nr_accesses && | |
| 799 | + nr_accesses <= s->pattern.max_nr_accesses && | |
| 795 | 800 | s->pattern.min_age_region <= r->age && |
| 796 | 801 | r->age <= s->pattern.max_age_region; |
| 797 | 802 | } |
| @@ -946,6 +951,33 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 148 unmodified lines | ||
| 946 | 951 | struct timespec64 begin, end; |
| 947 | 952 | unsigned long sz_applied = 0; |
| 948 | 953 | int err = 0; |
| 954 | + /* | |
| 955 | + * We plan to support multiple context per kdamond, as DAMON sysfs | |
| 956 | + * implies with 'nr_contexts' file. Nevertheless, only single context | |
| 957 | + * per kdamond is supported for now. So, we can simply use '0' context | |
| 958 | + * index here. | |
| 959 | + */ | |
| 960 | + unsigned int cidx = 0; | |
| 961 | + struct damos *siter; /* schemes iterator */ | |
| 962 | + unsigned int sidx = 0; | |
| 963 | + struct damon_target *titer; /* targets iterator */ | |
| 964 | + unsigned int tidx = 0; | |
| 965 | + bool do_trace = false; | |
| 966 | + | |
| 967 | + /* get indices for trace_damos_before_apply() */ | |
| 968 | + if (trace_damos_before_apply_enabled()) { | |
| 969 | + damon_for_each_scheme(siter, c) { | |
| 970 | + if (siter == s) | |
| 971 | + break; | |
| 972 | + sidx++; | |
| 973 | + } | |
| 974 | + damon_for_each_target(titer, c) { | |
| 975 | + if (titer == t) | |
| 976 | + break; | |
| 977 | + tidx++; | |
| 978 | + } | |
| 979 | + do_trace = true; | |
| 980 | + } | |
| 949 | 981 | |
| 950 | 982 | if (c->ops.apply_scheme) { |
| 951 | 983 | if (quota->esz && quota->charged_sz + sz > quota->esz) { |
| @@ -960,8 +992,11 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t, · 8 unmodified lines | ||
| 960 | 992 | ktime_get_coarse_ts64(&begin); |
| 961 | 993 | if (c->callback.before_damos_apply) |
| 962 | 994 | err = c->callback.before_damos_apply(c, t, r, s); |
| 963 | − if (!err) | |
| 995 | + if (!err) { | |
| 996 | + trace_damos_before_apply(cidx, sidx, tidx, r, | |
| 997 | + damon_nr_regions(t), do_trace); | |
| 964 | 998 | sz_applied = c->ops.apply_scheme(c, t, r, s); |
| 999 | + } | |
| 965 | 1000 | ktime_get_coarse_ts64(&end); |
| 966 | 1001 | quota->total_charged_ns += timespec64_to_ns(&end) - |
| 967 | 1002 | timespec64_to_ns(&begin); |
| @@ -1079,14 +1114,29 @@ static void kdamond_apply_schemes(struct damon_ctx *c) · 111 unmodified lines | ||
| 1079 | 1114 | struct damon_target *t; |
| 1080 | 1115 | struct damon_region *r, *next_r; |
| 1081 | 1116 | struct damos *s; |
| 1117 | + unsigned long sample_interval = c->attrs.sample_interval ? | |
| 1118 | + c->attrs.sample_interval : 1; | |
| 1119 | + bool has_schemes_to_apply = false; | |
| 1082 | 1120 | |
| 1083 | 1121 | damon_for_each_scheme(s, c) { |
| 1122 | + if (c->passed_sample_intervals != s->next_apply_sis) | |
| 1123 | + continue; | |
| 1124 | + | |
| 1125 | + s->next_apply_sis += | |
| 1126 | + (s->apply_interval_us ? s->apply_interval_us : | |
| 1127 | + c->attrs.aggr_interval) / sample_interval; | |
| 1128 | + | |
| 1084 | 1129 | if (!s->wmarks.activated) |
| 1085 | 1130 | continue; |
| 1086 | 1131 | |
| 1132 | + has_schemes_to_apply = true; | |
| 1133 | + | |
| 1087 | 1134 | damos_adjust_quota(c, s); |
| 1088 | 1135 | } |
| 1089 | 1136 | |
| 1137 | + if (!has_schemes_to_apply) | |
| 1138 | + return; | |
| 1139 | + | |
| 1090 | 1140 | damon_for_each_target(t, c) { |
| 1091 | 1141 | damon_for_each_region_safe(r, next_r, t) |
| 1092 | 1142 | damon_do_apply_schemes(c, t, r); |
| @@ -1103,6 +1153,7 @@ static void damon_merge_two_regions(struct damon_target *t, · 10 unmodified lines | ||
| 1103 | 1153 | |
| 1104 | 1154 | l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) / |
| 1105 | 1155 | (sz_l + sz_r); |
| 1156 | + l->nr_accesses_bp = l->nr_accesses * 10000; | |
| 1106 | 1157 | l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r); |
| 1107 | 1158 | l->ar.end = r->ar.end; |
| 1108 | 1159 | damon_destroy_region(r, t); |
| @@ -1174,6 +1225,7 @@ static void damon_split_region_at(struct damon_target *t, · 65 unmodified lines | ||
| 1174 | 1225 | |
| 1175 | 1226 | new->age = r->age; |
| 1176 | 1227 | new->last_nr_accesses = r->last_nr_accesses; |
| 1228 | + new->nr_accesses_bp = r->nr_accesses_bp; | |
| 1177 | 1229 | |
| 1178 | 1230 | damon_insert_region(new, r, damon_next_region(r), t); |
| 1179 | 1231 | } |
| @@ -1240,18 +1292,6 @@ static void kdamond_split_regions(struct damon_ctx *ctx) · 60 unmodified lines | ||
| 1240 | 1292 | last_nr_regions = nr_regions; |
| 1241 | 1293 | } |
| 1242 | 1294 | |
| 1243 | −/* | |
| 1244 | − * Check whether it is time to check and apply the operations-related data | |
| 1245 | − * structures. | |
| 1246 | − * | |
| 1247 | − * Returns true if it is. | |
| 1248 | − */ | |
| 1249 | −static bool kdamond_need_update_operations(struct damon_ctx *ctx) | |
| 1250 | −{ | |
| 1251 | − return damon_check_reset_time_interval(&ctx->last_ops_update, | |
| 1252 | − ctx->attrs.ops_update_interval); | |
| 1253 | −} | |
| 1254 | − | |
| 1255 | 1295 | /* |
| 1256 | 1296 | * Check whether current monitoring should be stopped |
| 1257 | 1297 | * |
| @@ -1280,12 +1320,10 @@ static bool kdamond_need_stop(struct damon_ctx *ctx) · 22 unmodified lines | ||
| 1280 | 1320 | |
| 1281 | 1321 | static unsigned long damos_wmark_metric_value(enum damos_wmark_metric metric) |
| 1282 | 1322 | { |
| 1283 | − struct sysinfo i; | |
| 1284 | − | |
| 1285 | 1323 | switch (metric) { |
| 1286 | 1324 | case DAMOS_WMARK_FREE_MEM_RATE: |
| 1287 | − si_meminfo(&i); | |
| 1288 | − return i.freeram * 1000 / i.totalram; | |
| 1325 | + return global_zone_page_state(NR_FREE_PAGES) * 1000 / | |
| 1326 | + totalram_pages(); | |
| 1289 | 1327 | default: |
| 1290 | 1328 | break; |
| 1291 | 1329 | } |
| @@ -1363,6 +1401,25 @@ static int kdamond_wait_activation(struct damon_ctx *ctx) · 71 unmodified lines | ||
| 1363 | 1401 | return -EBUSY; |
| 1364 | 1402 | } |
| 1365 | 1403 | |
| 1404 | +static void kdamond_init_intervals_sis(struct damon_ctx *ctx) | |
| 1405 | +{ | |
| 1406 | + unsigned long sample_interval = ctx->attrs.sample_interval ? | |
| 1407 | + ctx->attrs.sample_interval : 1; | |
| 1408 | + unsigned long apply_interval; | |
| 1409 | + struct damos *scheme; | |
| 1410 | + | |
| 1411 | + ctx->passed_sample_intervals = 0; | |
| 1412 | + ctx->next_aggregation_sis = ctx->attrs.aggr_interval / sample_interval; | |
| 1413 | + ctx->next_ops_update_sis = ctx->attrs.ops_update_interval / | |
| 1414 | + sample_interval; | |
| 1415 | + | |
| 1416 | + damon_for_each_scheme(scheme, ctx) { | |
| 1417 | + apply_interval = scheme->apply_interval_us ? | |
| 1418 | + scheme->apply_interval_us : ctx->attrs.aggr_interval; | |
| 1419 | + scheme->next_apply_sis = apply_interval / sample_interval; | |
| 1420 | + } | |
| 1421 | +} | |
| 1422 | + | |
| 1366 | 1423 | /* |
| 1367 | 1424 | * The monitoring daemon that runs as a kernel thread |
| 1368 | 1425 | */ |
| @@ -1376,6 +1433,8 @@ static int kdamond_fn(void *data) · 7 unmodified lines | ||
| 1376 | 1433 | |
| 1377 | 1434 | pr_debug("kdamond (%d) starts\n", current->pid); |
| 1378 | 1435 | |
| 1436 | + kdamond_init_intervals_sis(ctx); | |
| 1437 | + | |
| 1379 | 1438 | if (ctx->ops.init) |
| 1380 | 1439 | ctx->ops.init(ctx); |
| 1381 | 1440 | if (ctx->callback.before_start && ctx->callback.before_start(ctx)) |
| @@ -1384,6 +1443,17 @@ static int kdamond_fn(void *data) · 2 unmodified lines | ||
| 1384 | 1443 | sz_limit = damon_region_sz_limit(ctx); |
| 1385 | 1444 | |
| 1386 | 1445 | while (!kdamond_need_stop(ctx)) { |
| 1446 | + /* | |
| 1447 | + * ctx->attrs and ctx->next_{aggregation,ops_update}_sis could | |
| 1448 | + * be changed from after_wmarks_check() or after_aggregation() | |
| 1449 | + * callbacks. Read the values here, and use those for this | |
| 1450 | + * iteration. That is, damon_set_attrs() updated new values | |
| 1451 | + * are respected from next iteration. | |
| 1452 | + */ | |
| 1453 | + unsigned long next_aggregation_sis = ctx->next_aggregation_sis; | |
| 1454 | + unsigned long next_ops_update_sis = ctx->next_ops_update_sis; | |
| 1455 | + unsigned long sample_interval = ctx->attrs.sample_interval; | |
| 1456 | + | |
| 1387 | 1457 | if (kdamond_wait_activation(ctx)) |
| 1388 | 1458 | break; |
| 1389 | 1459 | |
| @@ -1393,27 +1463,44 @@ static int kdamond_fn(void *data) · 3 unmodified lines | ||
| 1393 | 1463 | ctx->callback.after_sampling(ctx)) |
| 1394 | 1464 | break; |
| 1395 | 1465 | |
| 1396 | − kdamond_usleep(ctx->attrs.sample_interval); | |
| 1466 | + kdamond_usleep(sample_interval); | |
| 1467 | + ctx->passed_sample_intervals++; | |
| 1397 | 1468 | |
| 1398 | 1469 | if (ctx->ops.check_accesses) |
| 1399 | 1470 | max_nr_accesses = ctx->ops.check_accesses(ctx); |
| 1400 | 1471 | |
| 1401 | − if (kdamond_aggregate_interval_passed(ctx)) { | |
| 1472 | + if (ctx->passed_sample_intervals == next_aggregation_sis) { | |
| 1402 | 1473 | kdamond_merge_regions(ctx, |
| 1403 | 1474 | max_nr_accesses / 10, |
| 1404 | 1475 | sz_limit); |
| 1405 | 1476 | if (ctx->callback.after_aggregation && |
| 1406 | 1477 | ctx->callback.after_aggregation(ctx)) |
| 1407 | 1478 | break; |
| 1408 | − if (!list_empty(&ctx->schemes)) | |
| 1409 | − kdamond_apply_schemes(ctx); | |
| 1479 | + } | |
| 1480 | + | |
| 1481 | + /* | |
| 1482 | + * do kdamond_apply_schemes() after kdamond_merge_regions() if | |
| 1483 | + * possible, to reduce overhead | |
| 1484 | + */ | |
| 1485 | + if (!list_empty(&ctx->schemes)) | |
| 1486 | + kdamond_apply_schemes(ctx); | |
| 1487 | + | |
| 1488 | + sample_interval = ctx->attrs.sample_interval ? | |
| 1489 | + ctx->attrs.sample_interval : 1; | |
| 1490 | + if (ctx->passed_sample_intervals == next_aggregation_sis) { | |
| 1491 | + ctx->next_aggregation_sis = next_aggregation_sis + | |
| 1492 | + ctx->attrs.aggr_interval / sample_interval; | |
| 1493 | + | |
| 1410 | 1494 | kdamond_reset_aggregated(ctx); |
| 1411 | 1495 | kdamond_split_regions(ctx); |
| 1412 | 1496 | if (ctx->ops.reset_aggregated) |
| 1413 | 1497 | ctx->ops.reset_aggregated(ctx); |
| 1414 | 1498 | } |
| 1415 | 1499 | |
| 1416 | − if (kdamond_need_update_operations(ctx)) { | |
| 1500 | + if (ctx->passed_sample_intervals == next_ops_update_sis) { | |
| 1501 | + ctx->next_ops_update_sis = next_ops_update_sis + | |
| 1502 | + ctx->attrs.ops_update_interval / | |
| 1503 | + sample_interval; | |
| 1417 | 1504 | if (ctx->ops.update) |
| 1418 | 1505 | ctx->ops.update(ctx); |
| 1419 | 1506 | sz_limit = damon_region_sz_limit(ctx); |
| @@ -1517,6 +1604,76 @@ int damon_set_region_biggest_system_ram_default(struct damon_target *t, · 97 unmodified lines | ||
| 1517 | 1604 | return damon_set_regions(t, &addr_range, 1); |
| 1518 | 1605 | } |
| 1519 | 1606 | |
| 1607 | +/* | |
| 1608 | + * damon_moving_sum() - Calculate an inferred moving sum value. | |
| 1609 | + * @mvsum: Inferred sum of the last @len_window values. | |
| 1610 | + * @nomvsum: Non-moving sum of the last discrete @len_window window values. | |
| 1611 | + * @len_window: The number of last values to take care of. | |
| 1612 | + * @new_value: New value that will be added to the pseudo moving sum. | |
| 1613 | + * | |
| 1614 | + * Moving sum (moving average * window size) is good for handling noise, but | |
| 1615 | + * the cost of keeping past values can be high for arbitrary window size. This | |
| 1616 | + * function implements a lightweight pseudo moving sum function that doesn't | |
| 1617 | + * keep the past window values. | |
| 1618 | + * | |
| 1619 | + * It simply assumes there was no noise in the past, and get the no-noise | |
| 1620 | + * assumed past value to drop from @nomvsum and @len_window. @nomvsum is a | |
| 1621 | + * non-moving sum of the last window. For example, if @len_window is 10 and we | |
| 1622 | + * have 25 values, @nomvsum is the sum of the 11th to 20th values of the 25 | |
| 1623 | + * values. Hence, this function simply drops @nomvsum / @len_window from | |
| 1624 | + * given @mvsum and add @new_value. | |
| 1625 | + * | |
| 1626 | + * For example, if @len_window is 10 and @nomvsum is 50, the last 10 values for | |
| 1627 | + * the last window could be vary, e.g., 0, 10, 0, 10, 0, 10, 0, 0, 0, 20. For | |
| 1628 | + * calculating next moving sum with a new value, we should drop 0 from 50 and | |
| 1629 | + * add the new value. However, this function assumes it got value 5 for each | |
| 1630 | + * of the last ten times. Based on the assumption, when the next value is | |
| 1631 | + * measured, it drops the assumed past value, 5 from the current sum, and add | |
| 1632 | + * the new value to get the updated pseduo-moving average. | |
| 1633 | + * | |
| 1634 | + * This means the value could have errors, but the errors will be disappeared | |
| 1635 | + * for every @len_window aligned calls. For example, if @len_window is 10, the | |
| 1636 | + * pseudo moving sum with 11th value to 19th value would have an error. But | |
| 1637 | + * the sum with 20th value will not have the error. | |
| 1638 | + * | |
| 1639 | + * Return: Pseudo-moving average after getting the @new_value. | |
| 1640 | + */ | |
| 1641 | +static unsigned int damon_moving_sum(unsigned int mvsum, unsigned int nomvsum, | |
| 1642 | + unsigned int len_window, unsigned int new_value) | |
| 1643 | +{ | |
| 1644 | + return mvsum - nomvsum / len_window + new_value; | |
| 1645 | +} | |
| 1646 | + | |
| 1647 | +/** | |
| 1648 | + * damon_update_region_access_rate() - Update the access rate of a region. | |
| 1649 | + * @r: The DAMON region to update for its access check result. | |
| 1650 | + * @accessed: Whether the region has accessed during last sampling interval. | |
| 1651 | + * @attrs: The damon_attrs of the DAMON context. | |
| 1652 | + * | |
| 1653 | + * Update the access rate of a region with the region's last sampling interval | |
| 1654 | + * access check result. | |
| 1655 | + * | |
| 1656 | + * Usually this will be called by &damon_operations->check_accesses callback. | |
| 1657 | + */ | |
| 1658 | +void damon_update_region_access_rate(struct damon_region *r, bool accessed, | |
| 1659 | + struct damon_attrs *attrs) | |
| 1660 | +{ | |
| 1661 | + unsigned int len_window = 1; | |
| 1662 | + | |
| 1663 | + /* | |
| 1664 | + * sample_interval can be zero, but cannot be larger than | |
| 1665 | + * aggr_interval, owing to validation of damon_set_attrs(). | |
| 1666 | + */ | |
| 1667 | + if (attrs->sample_interval) | |
| 1668 | + len_window = damon_max_nr_accesses(attrs); | |
| 1669 | + r->nr_accesses_bp = damon_moving_sum(r->nr_accesses_bp, | |
| 1670 | + r->last_nr_accesses * 10000, len_window, | |
| 1671 | + accessed ? 10000 : 0); | |
| 1672 | + | |
| 1673 | + if (accessed) | |
| 1674 | + r->nr_accesses++; | |
| 1675 | +} | |
| 1676 | + | |
| 1520 | 1677 | static int __init damon_init(void) |
| 1521 | 1678 | { |
| 1522 | 1679 | damon_region_cache = KMEM_CACHE(damon_region, 0); |
| @@ -278,7 +278,8 @@ static struct damos **str_to_schemes(const char *str, ssize_t len, · 277 unmodified lines | ||
| 278 | 278 | goto fail; |
| 279 | 279 | |
| 280 | 280 | pos += parsed; |
| 281 | − scheme = damon_new_scheme(&pattern, action, "a, &wmarks); | |
| 281 | + scheme = damon_new_scheme(&pattern, action, 0, "a, | |
| 282 | + &wmarks); | |
| 282 | 283 | if (!scheme) |
| 283 | 284 | goto fail; |
| 284 | 285 | |
| @@ -158,6 +158,8 @@ static struct damos *damon_lru_sort_new_scheme( · 157 unmodified lines | ||
| 158 | 158 | pattern, |
| 159 | 159 | /* (de)prioritize on LRU-lists */ |
| 160 | 160 | action, |
| 161 | + /* for each aggregation interval */ | |
| 162 | + 0, | |
| 161 | 163 | /* under the quota. */ |
| 162 | 164 | "a, |
| 163 | 165 | /* (De)activate this according to the watermarks. */ |
| @@ -193,9 +195,7 @@ static int damon_lru_sort_apply_parameters(void) · 29 unmodified lines | ||
| 193 | 195 | if (err) |
| 194 | 196 | return err; |
| 195 | 197 | |
| 196 | − /* aggr_interval / sample_interval is the maximum nr_accesses */ | |
| 197 | − hot_thres = damon_lru_sort_mon_attrs.aggr_interval / | |
| 198 | − damon_lru_sort_mon_attrs.sample_interval * | |
| 198 | + hot_thres = damon_max_nr_accesses(&damon_lru_sort_mon_attrs) * | |
| 199 | 199 | hot_thres_access_freq / 1000; |
| 200 | 200 | scheme = damon_lru_sort_new_hot_scheme(hot_thres); |
| 201 | 201 | if (!scheme) |
| @@ -73,7 +73,6 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr · 72 unmodified lines | ||
| 73 | 73 | int damon_hot_score(struct damon_ctx *c, struct damon_region *r, |
| 74 | 74 | struct damos *s) |
| 75 | 75 | { |
| 76 | − unsigned int max_nr_accesses; | |
| 77 | 76 | int freq_subscore; |
| 78 | 77 | unsigned int age_in_sec; |
| 79 | 78 | int age_in_log, age_subscore; |
| @@ -81,8 +80,8 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, · 1 unmodified line | ||
| 81 | 80 | unsigned int age_weight = s->quota.weight_age; |
| 82 | 81 | int hotness; |
| 83 | 82 | |
| 84 | − max_nr_accesses = c->attrs.aggr_interval / c->attrs.sample_interval; | |
| 85 | − freq_subscore = r->nr_accesses * DAMON_MAX_SUBSCORE / max_nr_accesses; | |
| 83 | + freq_subscore = r->nr_accesses * DAMON_MAX_SUBSCORE / | |
| 84 | + damon_max_nr_accesses(&c->attrs); | |
| 86 | 85 | |
| 87 | 86 | age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000; |
| 88 | 87 | for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec; |
| @@ -148,7 +148,8 @@ out: · 147 unmodified lines | ||
| 148 | 148 | return accessed; |
| 149 | 149 | } |
| 150 | 150 | |
| 151 | −static void __damon_pa_check_access(struct damon_region *r) | |
| 151 | +static void __damon_pa_check_access(struct damon_region *r, | |
| 152 | + struct damon_attrs *attrs) | |
| 152 | 153 | { |
| 153 | 154 | static unsigned long last_addr; |
| 154 | 155 | static unsigned long last_folio_sz = PAGE_SIZE; |
| @@ -157,14 +158,12 @@ static void __damon_pa_check_access(struct damon_region *r) · 2 unmodified lines | ||
| 157 | 158 | /* If the region is in the last checked page, reuse the result */ |
| 158 | 159 | if (ALIGN_DOWN(last_addr, last_folio_sz) == |
| 159 | 160 | ALIGN_DOWN(r->sampling_addr, last_folio_sz)) { |
| 160 | − if (last_accessed) | |
| 161 | − r->nr_accesses++; | |
| 161 | + damon_update_region_access_rate(r, last_accessed, attrs); | |
| 162 | 162 | return; |
| 163 | 163 | } |
| 164 | 164 | |
| 165 | 165 | last_accessed = damon_pa_young(r->sampling_addr, &last_folio_sz); |
| 166 | − if (last_accessed) | |
| 167 | − r->nr_accesses++; | |
| 166 | + damon_update_region_access_rate(r, last_accessed, attrs); | |
| 168 | 167 | |
| 169 | 168 | last_addr = r->sampling_addr; |
| 170 | 169 | } |
| @@ -177,7 +176,7 @@ static unsigned int damon_pa_check_accesses(struct damon_ctx *ctx) · 6 unmodified lines | ||
| 177 | 176 | |
| 178 | 177 | damon_for_each_target(t, ctx) { |
| 179 | 178 | damon_for_each_region(r, t) { |
| 180 | − __damon_pa_check_access(r); | |
| 179 | + __damon_pa_check_access(r, &ctx->attrs); | |
| 181 | 180 | max_nr_accesses = max(r->nr_accesses, max_nr_accesses); |
| 182 | 181 | } |
| 183 | 182 | } |
| @@ -142,6 +142,8 @@ static struct damos *damon_reclaim_new_scheme(void) · 141 unmodified lines | ||
| 142 | 142 | &pattern, |
| 143 | 143 | /* page out those, as soon as found */ |
| 144 | 144 | DAMOS_PAGEOUT, |
| 145 | + /* for each aggregation interval */ | |
| 146 | + 0, | |
| 145 | 147 | /* under the quota. */ |
| 146 | 148 | &damon_reclaim_quota, |
| 147 | 149 | /* (De)activate this according to the watermarks. */ |
| @@ -49,6 +49,8 @@ int damon_sysfs_schemes_update_regions_start( · 48 unmodified lines | ||
| 49 | 49 | struct damon_sysfs_schemes *sysfs_schemes, |
| 50 | 50 | struct damon_ctx *ctx, bool total_bytes_only); |
| 51 | 51 | |
| 52 | +bool damos_sysfs_regions_upd_done(void); | |
| 53 | + | |
| 52 | 54 | int damon_sysfs_schemes_update_regions_stop(struct damon_ctx *ctx); |
| 53 | 55 | |
| 54 | 56 | int damon_sysfs_schemes_clear_regions( |
| @@ -31,7 +31,7 @@ static struct damon_sysfs_scheme_region *damon_sysfs_scheme_region_alloc( · 30 unmodified lines | ||
| 31 | 31 | return NULL; |
| 32 | 32 | sysfs_region->kobj = (struct kobject){}; |
| 33 | 33 | sysfs_region->ar = region->ar; |
| 34 | − sysfs_region->nr_accesses = region->nr_accesses; | |
| 34 | + sysfs_region->nr_accesses = region->nr_accesses_bp / 10000; | |
| 35 | 35 | sysfs_region->age = region->age; |
| 36 | 36 | INIT_LIST_HEAD(&sysfs_region->list); |
| 37 | 37 | return sysfs_region; |
| @@ -113,11 +113,47 @@ static const struct kobj_type damon_sysfs_scheme_region_ktype = { · 75 unmodified lines | ||
| 113 | 113 | * scheme regions directory |
| 114 | 114 | */ |
| 115 | 115 | |
| 116 | +/* | |
| 117 | + * enum damos_sysfs_regions_upd_status - Represent DAMOS tried regions update | |
| 118 | + * status | |
| 119 | + * @DAMOS_TRIED_REGIONS_UPD_IDLE: Waiting for next request. | |
| 120 | + * @DAMOS_TRIED_REGIONS_UPD_STARTED: Update started. | |
| 121 | + * @DAMOS_TRIED_REGIONS_UPD_FINISHED: Update finished. | |
| 122 | + * | |
| 123 | + * Each DAMON-based operation scheme (&struct damos) has its own apply | |
| 124 | + * interval, and we need to expose the scheme tried regions based on only | |
| 125 | + * single snapshot. For this, we keep the tried regions update status for each | |
| 126 | + * scheme. The status becomes 'idle' at the beginning. | |
| 127 | + * | |
| 128 | + * Once the tried regions update request is received, the request handling | |
| 129 | + * start function (damon_sysfs_scheme_update_regions_start()) sets the status | |
| 130 | + * of all schemes as 'idle' again, and register ->before_damos_apply() and | |
| 131 | + * ->after_sampling() callbacks. | |
| 132 | + * | |
| 133 | + * Then, the first followup ->before_damos_apply() callback | |
| 134 | + * (damon_sysfs_before_damos_apply()) sets the status 'started'. The first | |
| 135 | + * ->after_sampling() callback (damon_sysfs_after_sampling()) after the call | |
| 136 | + * is called only after the scheme is completely applied | |
| 137 | + * to the given snapshot. Hence the callback knows the situation by showing | |
| 138 | + * 'started' status, and sets the status as 'finished'. Then, | |
| 139 | + * damon_sysfs_before_damos_apply() understands the situation by showing the | |
| 140 | + * 'finished' status and do nothing. | |
| 141 | + * | |
| 142 | + * Finally, the tried regions request handling finisher function | |
| 143 | + * (damon_sysfs_schemes_update_regions_stop()) unregisters the callbacks. | |
| 144 | + */ | |
| 145 | +enum damos_sysfs_regions_upd_status { | |
| 146 | + DAMOS_TRIED_REGIONS_UPD_IDLE, | |
| 147 | + DAMOS_TRIED_REGIONS_UPD_STARTED, | |
| 148 | + DAMOS_TRIED_REGIONS_UPD_FINISHED, | |
| 149 | +}; | |
| 150 | + | |
| 116 | 151 | struct damon_sysfs_scheme_regions { |
| 117 | 152 | struct kobject kobj; |
| 118 | 153 | struct list_head regions_list; |
| 119 | 154 | int nr_regions; |
| 120 | 155 | unsigned long total_bytes; |
| 156 | + enum damos_sysfs_regions_upd_status upd_status; | |
| 121 | 157 | }; |
| 122 | 158 | |
| 123 | 159 | static struct damon_sysfs_scheme_regions * |
| @@ -130,6 +166,7 @@ damon_sysfs_scheme_regions_alloc(void) · 6 unmodified lines | ||
| 130 | 166 | INIT_LIST_HEAD(®ions->regions_list); |
| 131 | 167 | regions->nr_regions = 0; |
| 132 | 168 | regions->total_bytes = 0; |
| 169 | + regions->upd_status = DAMOS_TRIED_REGIONS_UPD_IDLE; | |
| 133 | 170 | return regions; |
| 134 | 171 | } |
| 135 | 172 | |
| @@ -1121,6 +1158,7 @@ struct damon_sysfs_scheme { · 985 unmodified lines | ||
| 1121 | 1158 | struct kobject kobj; |
| 1122 | 1159 | enum damos_action action; |
| 1123 | 1160 | struct damon_sysfs_access_pattern *access_pattern; |
| 1161 | + unsigned long apply_interval_us; | |
| 1124 | 1162 | struct damon_sysfs_quotas *quotas; |
| 1125 | 1163 | struct damon_sysfs_watermarks *watermarks; |
| 1126 | 1164 | struct damon_sysfs_scheme_filters *filters; |
| @@ -1141,7 +1179,7 @@ static const char * const damon_sysfs_damos_action_strs[] = { · 14 unmodified lines | ||
| 1141 | 1179 | }; |
| 1142 | 1180 | |
| 1143 | 1181 | static struct damon_sysfs_scheme *damon_sysfs_scheme_alloc( |
| 1144 | − enum damos_action action) | |
| 1182 | + enum damos_action action, unsigned long apply_interval_us) | |
| 1145 | 1183 | { |
| 1146 | 1184 | struct damon_sysfs_scheme *scheme = kmalloc(sizeof(*scheme), |
| 1147 | 1185 | GFP_KERNEL); |
| @@ -1150,6 +1188,7 @@ static struct damon_sysfs_scheme *damon_sysfs_scheme_alloc( · 2 unmodified lines | ||
| 1150 | 1188 | return NULL; |
| 1151 | 1189 | scheme->kobj = (struct kobject){}; |
| 1152 | 1190 | scheme->action = action; |
| 1191 | + scheme->apply_interval_us = apply_interval_us; | |
| 1153 | 1192 | return scheme; |
| 1154 | 1193 | } |
| 1155 | 1194 | |
| @@ -1353,6 +1392,25 @@ static ssize_t action_store(struct kobject *kobj, struct kobj_attribute *attr, · 197 unmodified lines | ||
| 1353 | 1392 | return -EINVAL; |
| 1354 | 1393 | } |
| 1355 | 1394 | |
| 1395 | +static ssize_t apply_interval_us_show(struct kobject *kobj, | |
| 1396 | + struct kobj_attribute *attr, char *buf) | |
| 1397 | +{ | |
| 1398 | + struct damon_sysfs_scheme *scheme = container_of(kobj, | |
| 1399 | + struct damon_sysfs_scheme, kobj); | |
| 1400 | + | |
| 1401 | + return sysfs_emit(buf, "%lu\n", scheme->apply_interval_us); | |
| 1402 | +} | |
| 1403 | + | |
| 1404 | +static ssize_t apply_interval_us_store(struct kobject *kobj, | |
| 1405 | + struct kobj_attribute *attr, const char *buf, size_t count) | |
| 1406 | +{ | |
| 1407 | + struct damon_sysfs_scheme *scheme = container_of(kobj, | |
| 1408 | + struct damon_sysfs_scheme, kobj); | |
| 1409 | + int err = kstrtoul(buf, 0, &scheme->apply_interval_us); | |
| 1410 | + | |
| 1411 | + return err ? err : count; | |
| 1412 | +} | |
| 1413 | + | |
| 1356 | 1414 | static void damon_sysfs_scheme_release(struct kobject *kobj) |
| 1357 | 1415 | { |
| 1358 | 1416 | kfree(container_of(kobj, struct damon_sysfs_scheme, kobj)); |
| @@ -1361,8 +1419,12 @@ static void damon_sysfs_scheme_release(struct kobject *kobj) · 2 unmodified lines | ||
| 1361 | 1419 | static struct kobj_attribute damon_sysfs_scheme_action_attr = |
| 1362 | 1420 | __ATTR_RW_MODE(action, 0600); |
| 1363 | 1421 | |
| 1422 | +static struct kobj_attribute damon_sysfs_scheme_apply_interval_us_attr = | |
| 1423 | + __ATTR_RW_MODE(apply_interval_us, 0600); | |
| 1424 | + | |
| 1364 | 1425 | static struct attribute *damon_sysfs_scheme_attrs[] = { |
| 1365 | 1426 | &damon_sysfs_scheme_action_attr.attr, |
| 1427 | + &damon_sysfs_scheme_apply_interval_us_attr.attr, | |
| 1366 | 1428 | NULL, |
| 1367 | 1429 | }; |
| 1368 | 1430 | ATTRIBUTE_GROUPS(damon_sysfs_scheme); |
| @@ -1413,7 +1475,11 @@ static int damon_sysfs_schemes_add_dirs(struct damon_sysfs_schemes *schemes, · 44 unmodified lines | ||
| 1413 | 1475 | schemes->schemes_arr = schemes_arr; |
| 1414 | 1476 | |
| 1415 | 1477 | for (i = 0; i < nr_schemes; i++) { |
| 1416 | − scheme = damon_sysfs_scheme_alloc(DAMOS_STAT); | |
| 1478 | + /* | |
| 1479 | + * apply_interval_us as 0 means same to aggregation interval | |
| 1480 | + * (same to before-apply_interval behavior) | |
| 1481 | + */ | |
| 1482 | + scheme = damon_sysfs_scheme_alloc(DAMOS_STAT, 0); | |
| 1417 | 1483 | if (!scheme) { |
| 1418 | 1484 | damon_sysfs_schemes_rm_dirs(schemes); |
| 1419 | 1485 | return -ENOMEM; |
| @@ -1610,8 +1676,8 @@ static struct damos *damon_sysfs_mk_scheme( · 190 unmodified lines | ||
| 1610 | 1676 | .low = sysfs_wmarks->low, |
| 1611 | 1677 | }; |
| 1612 | 1678 | |
| 1613 | − scheme = damon_new_scheme(&pattern, sysfs_scheme->action, "a, | |
| 1614 | − &wmarks); | |
| 1679 | + scheme = damon_new_scheme(&pattern, sysfs_scheme->action, | |
| 1680 | + sysfs_scheme->apply_interval_us, "a, &wmarks); | |
| 1615 | 1681 | if (!scheme) |
| 1616 | 1682 | return NULL; |
| 1617 | 1683 | |
| @@ -1641,6 +1707,7 @@ static void damon_sysfs_update_scheme(struct damos *scheme, · 23 unmodified lines | ||
| 1641 | 1707 | scheme->pattern.max_age_region = access_pattern->age->max; |
| 1642 | 1708 | |
| 1643 | 1709 | scheme->action = sysfs_scheme->action; |
| 1710 | + scheme->apply_interval_us = sysfs_scheme->apply_interval_us; | |
| 1644 | 1711 | |
| 1645 | 1712 | scheme->quota.ms = sysfs_quotas->ms; |
| 1646 | 1713 | scheme->quota.sz = sysfs_quotas->sz; |
| @@ -1747,6 +1814,10 @@ static int damon_sysfs_before_damos_apply(struct damon_ctx *ctx, · 100 unmodified lines | ||
| 1747 | 1814 | return 0; |
| 1748 | 1815 | |
| 1749 | 1816 | sysfs_regions = sysfs_schemes->schemes_arr[schemes_idx]->tried_regions; |
| 1817 | + if (sysfs_regions->upd_status == DAMOS_TRIED_REGIONS_UPD_FINISHED) | |
| 1818 | + return 0; | |
| 1819 | + if (sysfs_regions->upd_status == DAMOS_TRIED_REGIONS_UPD_IDLE) | |
| 1820 | + sysfs_regions->upd_status = DAMOS_TRIED_REGIONS_UPD_STARTED; | |
| 1750 | 1821 | sysfs_regions->total_bytes += r->ar.end - r->ar.start; |
| 1751 | 1822 | if (damos_regions_upd_total_bytes_only) |
| 1752 | 1823 | return 0; |
| @@ -1763,6 +1834,29 @@ static int damon_sysfs_before_damos_apply(struct damon_ctx *ctx, · 10 unmodified lines | ||
| 1763 | 1834 | return 0; |
| 1764 | 1835 | } |
| 1765 | 1836 | |
| 1837 | +/* | |
| 1838 | + * DAMON callback that called after each accesses sampling. While this | |
| 1839 | + * callback is registered, damon_sysfs_lock should be held to ensure the | |
| 1840 | + * regions directories exist. | |
| 1841 | + */ | |
| 1842 | +static int damon_sysfs_after_sampling(struct damon_ctx *ctx) | |
| 1843 | +{ | |
| 1844 | + struct damon_sysfs_schemes *sysfs_schemes = | |
| 1845 | + damon_sysfs_schemes_for_damos_callback; | |
| 1846 | + struct damon_sysfs_scheme_regions *sysfs_regions; | |
| 1847 | + int i; | |
| 1848 | + | |
| 1849 | + for (i = 0; i < sysfs_schemes->nr; i++) { | |
| 1850 | + sysfs_regions = sysfs_schemes->schemes_arr[i]->tried_regions; | |
| 1851 | + if (sysfs_regions->upd_status == | |
| 1852 | + DAMOS_TRIED_REGIONS_UPD_STARTED) | |
| 1853 | + sysfs_regions->upd_status = | |
| 1854 | + DAMOS_TRIED_REGIONS_UPD_FINISHED; | |
| 1855 | + } | |
| 1856 | + | |
| 1857 | + return 0; | |
| 1858 | +} | |
| 1859 | + | |
| 1766 | 1860 | /* Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock */ |
| 1767 | 1861 | int damon_sysfs_schemes_clear_regions( |
| 1768 | 1862 | struct damon_sysfs_schemes *sysfs_schemes, |
| @@ -1786,6 +1880,16 @@ int damon_sysfs_schemes_clear_regions( · 17 unmodified lines | ||
| 1786 | 1880 | return 0; |
| 1787 | 1881 | } |
| 1788 | 1882 | |
| 1883 | +static void damos_tried_regions_init_upd_status( | |
| 1884 | + struct damon_sysfs_schemes *sysfs_schemes) | |
| 1885 | +{ | |
| 1886 | + int i; | |
| 1887 | + | |
| 1888 | + for (i = 0; i < sysfs_schemes->nr; i++) | |
| 1889 | + sysfs_schemes->schemes_arr[i]->tried_regions->upd_status = | |
| 1890 | + DAMOS_TRIED_REGIONS_UPD_IDLE; | |
| 1891 | +} | |
| 1892 | + | |
| 1789 | 1893 | /* Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock */ |
| 1790 | 1894 | int damon_sysfs_schemes_update_regions_start( |
| 1791 | 1895 | struct damon_sysfs_schemes *sysfs_schemes, |
| @@ -1793,11 +1897,29 @@ int damon_sysfs_schemes_update_regions_start( · 1 unmodified line | ||
| 1793 | 1897 | { |
| 1794 | 1898 | damon_sysfs_schemes_clear_regions(sysfs_schemes, ctx); |
| 1795 | 1899 | damon_sysfs_schemes_for_damos_callback = sysfs_schemes; |
| 1900 | + damos_tried_regions_init_upd_status(sysfs_schemes); | |
| 1796 | 1901 | damos_regions_upd_total_bytes_only = total_bytes_only; |
| 1797 | 1902 | ctx->callback.before_damos_apply = damon_sysfs_before_damos_apply; |
| 1903 | + ctx->callback.after_sampling = damon_sysfs_after_sampling; | |
| 1798 | 1904 | return 0; |
| 1799 | 1905 | } |
| 1800 | 1906 | |
| 1907 | +bool damos_sysfs_regions_upd_done(void) | |
| 1908 | +{ | |
| 1909 | + struct damon_sysfs_schemes *sysfs_schemes = | |
| 1910 | + damon_sysfs_schemes_for_damos_callback; | |
| 1911 | + struct damon_sysfs_scheme_regions *sysfs_regions; | |
| 1912 | + int i; | |
| 1913 | + | |
| 1914 | + for (i = 0; i < sysfs_schemes->nr; i++) { | |
| 1915 | + sysfs_regions = sysfs_schemes->schemes_arr[i]->tried_regions; | |
| 1916 | + if (sysfs_regions->upd_status != | |
| 1917 | + DAMOS_TRIED_REGIONS_UPD_FINISHED) | |
| 1918 | + return false; | |
| 1919 | + } | |
| 1920 | + return true; | |
| 1921 | +} | |
| 1922 | + | |
| 1801 | 1923 | /* |
| 1802 | 1924 | * Called from damon_sysfs_cmd_request_callback under damon_sysfs_lock. Caller |
| 1803 | 1925 | * should unlock damon_sysfs_lock which held before |
| @@ -1807,6 +1929,7 @@ int damon_sysfs_schemes_update_regions_stop(struct damon_ctx *ctx) · 3 unmodified lines | ||
| 1807 | 1929 | { |
| 1808 | 1930 | damon_sysfs_schemes_for_damos_callback = NULL; |
| 1809 | 1931 | ctx->callback.before_damos_apply = NULL; |
| 1932 | + ctx->callback.after_sampling = NULL; | |
| 1810 | 1933 | damon_sysfs_schemes_region_idx = 0; |
| 1811 | 1934 | return 0; |
| 1812 | 1935 | } |
| @@ -0,0 +1,86 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +/* | |
| 3 | + * Data Access Monitor Unit Tests | |
| 4 | + * | |
| 5 | + * Author: SeongJae Park <sj@kernel.org> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +#ifdef CONFIG_DAMON_SYSFS_KUNIT_TEST | |
| 9 | + | |
| 10 | +#ifndef _DAMON_SYSFS_TEST_H | |
| 11 | +#define _DAMON_SYSFS_TEST_H | |
| 12 | + | |
| 13 | +#include <kunit/test.h> | |
| 14 | + | |
| 15 | +static unsigned int nr_damon_targets(struct damon_ctx *ctx) | |
| 16 | +{ | |
| 17 | + struct damon_target *t; | |
| 18 | + unsigned int nr_targets = 0; | |
| 19 | + | |
| 20 | + damon_for_each_target(t, ctx) | |
| 21 | + nr_targets++; | |
| 22 | + | |
| 23 | + return nr_targets; | |
| 24 | +} | |
| 25 | + | |
| 26 | +static int __damon_sysfs_test_get_any_pid(int min, int max) | |
| 27 | +{ | |
| 28 | + struct pid *pid; | |
| 29 | + int i; | |
| 30 | + | |
| 31 | + for (i = min; i <= max; i++) { | |
| 32 | + pid = find_get_pid(i); | |
| 33 | + if (pid) { | |
| 34 | + put_pid(pid); | |
| 35 | + return i; | |
| 36 | + } | |
| 37 | + } | |
| 38 | + return -1; | |
| 39 | +} | |
| 40 | + | |
| 41 | +static void damon_sysfs_test_set_targets(struct kunit *test) | |
| 42 | +{ | |
| 43 | + struct damon_sysfs_targets *sysfs_targets; | |
| 44 | + struct damon_sysfs_target *sysfs_target; | |
| 45 | + struct damon_ctx *ctx; | |
| 46 | + | |
| 47 | + sysfs_targets = damon_sysfs_targets_alloc(); | |
| 48 | + sysfs_targets->nr = 1; | |
| 49 | + sysfs_targets->targets_arr = kmalloc_array(1, | |
| 50 | + sizeof(*sysfs_targets->targets_arr), GFP_KERNEL); | |
| 51 | + | |
| 52 | + sysfs_target = damon_sysfs_target_alloc(); | |
| 53 | + sysfs_target->pid = __damon_sysfs_test_get_any_pid(12, 100); | |
| 54 | + sysfs_target->regions = damon_sysfs_regions_alloc(); | |
| 55 | + sysfs_targets->targets_arr[0] = sysfs_target; | |
| 56 | + | |
| 57 | + ctx = damon_new_ctx(); | |
| 58 | + | |
| 59 | + damon_sysfs_set_targets(ctx, sysfs_targets); | |
| 60 | + KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx)); | |
| 61 | + | |
| 62 | + sysfs_target->pid = __damon_sysfs_test_get_any_pid( | |
| 63 | + sysfs_target->pid + 1, 200); | |
| 64 | + damon_sysfs_set_targets(ctx, sysfs_targets); | |
| 65 | + KUNIT_EXPECT_EQ(test, 1u, nr_damon_targets(ctx)); | |
| 66 | + | |
| 67 | + damon_destroy_ctx(ctx); | |
| 68 | + kfree(sysfs_targets->targets_arr); | |
| 69 | + kfree(sysfs_targets); | |
| 70 | + kfree(sysfs_target); | |
| 71 | +} | |
| 72 | + | |
| 73 | +static struct kunit_case damon_sysfs_test_cases[] = { | |
| 74 | + KUNIT_CASE(damon_sysfs_test_set_targets), | |
| 75 | + {}, | |
| 76 | +}; | |
| 77 | + | |
| 78 | +static struct kunit_suite damon_sysfs_test_suite = { | |
| 79 | + .name = "damon-sysfs", | |
| 80 | + .test_cases = damon_sysfs_test_cases, | |
| 81 | +}; | |
| 82 | +kunit_test_suite(damon_sysfs_test_suite); | |
| 83 | + | |
| 84 | +#endif /* _DAMON_SYSFS_TEST_H */ | |
| 85 | + | |
| 86 | +#endif /* CONFIG_DAMON_SYSFS_KUNIT_TEST */ | |
| @@ -1150,58 +1150,73 @@ destroy_targets_out: · 1149 unmodified lines | ||
| 1150 | 1150 | return err; |
| 1151 | 1151 | } |
| 1152 | 1152 | |
| 1153 | −/* | |
| 1154 | − * Search a target in a context that corresponds to the sysfs target input. | |
| 1155 | − * | |
| 1156 | − * Return: pointer to the target if found, NULL if not found, or negative | |
| 1157 | − * error code if the search failed. | |
| 1158 | − */ | |
| 1159 | −static struct damon_target *damon_sysfs_existing_target( | |
| 1160 | − struct damon_sysfs_target *sys_target, struct damon_ctx *ctx) | |
| 1153 | +static int damon_sysfs_update_target_pid(struct damon_target *target, int pid) | |
| 1161 | 1154 | { |
| 1162 | − struct pid *pid; | |
| 1163 | − struct damon_target *t; | |
| 1155 | + struct pid *pid_new; | |
| 1164 | 1156 | |
| 1165 | − if (!damon_target_has_pid(ctx)) { | |
| 1166 | − /* Up to only one target for paddr could exist */ | |
| 1167 | − damon_for_each_target(t, ctx) | |
| 1168 | − return t; | |
| 1169 | − return NULL; | |
| 1157 | + pid_new = find_get_pid(pid); | |
| 1158 | + if (!pid_new) | |
| 1159 | + return -EINVAL; | |
| 1160 | + | |
| 1161 | + if (pid_new == target->pid) { | |
| 1162 | + put_pid(pid_new); | |
| 1163 | + return 0; | |
| 1170 | 1164 | } |
| 1171 | 1165 | |
| 1172 | − /* ops.id should be DAMON_OPS_VADDR or DAMON_OPS_FVADDR */ | |
| 1173 | − pid = find_get_pid(sys_target->pid); | |
| 1174 | − if (!pid) | |
| 1175 | − return ERR_PTR(-EINVAL); | |
| 1176 | − damon_for_each_target(t, ctx) { | |
| 1177 | − if (t->pid == pid) { | |
| 1178 | − put_pid(pid); | |
| 1179 | − return t; | |
| 1180 | − } | |
| 1166 | + put_pid(target->pid); | |
| 1167 | + target->pid = pid_new; | |
| 1168 | + return 0; | |
| 1169 | +} | |
| 1170 | + | |
| 1171 | +static int damon_sysfs_update_target(struct damon_target *target, | |
| 1172 | + struct damon_ctx *ctx, | |
| 1173 | + struct damon_sysfs_target *sys_target) | |
| 1174 | +{ | |
| 1175 | + int err; | |
| 1176 | + | |
| 1177 | + if (damon_target_has_pid(ctx)) { | |
| 1178 | + err = damon_sysfs_update_target_pid(target, sys_target->pid); | |
| 1179 | + if (err) | |
| 1180 | + return err; | |
| 1181 | 1181 | } |
| 1182 | − put_pid(pid); | |
| 1183 | − return NULL; | |
| 1182 | + | |
| 1183 | + /* | |
| 1184 | + * Do monitoring target region boundary update only if one or more | |
| 1185 | + * regions are set by the user. This is for keeping current monitoring | |
| 1186 | + * target results and range easier, especially for dynamic monitoring | |
| 1187 | + * target regions update ops like 'vaddr'. | |
| 1188 | + */ | |
| 1189 | + if (sys_target->regions->nr) | |
| 1190 | + err = damon_sysfs_set_regions(target, sys_target->regions); | |
| 1191 | + return err; | |
| 1184 | 1192 | } |
| 1185 | 1193 | |
| 1186 | 1194 | static int damon_sysfs_set_targets(struct damon_ctx *ctx, |
| 1187 | 1195 | struct damon_sysfs_targets *sysfs_targets) |
| 1188 | 1196 | { |
| 1189 | − int i, err; | |
| 1197 | + struct damon_target *t, *next; | |
| 1198 | + int i = 0, err; | |
| 1190 | 1199 | |
| 1191 | 1200 | /* Multiple physical address space monitoring targets makes no sense */ |
| 1192 | 1201 | if (ctx->ops.id == DAMON_OPS_PADDR && sysfs_targets->nr > 1) |
| 1193 | 1202 | return -EINVAL; |
| 1194 | 1203 | |
| 1195 | − for (i = 0; i < sysfs_targets->nr; i++) { | |
| 1204 | + damon_for_each_target_safe(t, next, ctx) { | |
| 1205 | + if (i < sysfs_targets->nr) { | |
| 1206 | + damon_sysfs_update_target(t, ctx, | |
| 1207 | + sysfs_targets->targets_arr[i]); | |
| 1208 | + } else { | |
| 1209 | + if (damon_target_has_pid(ctx)) | |
| 1210 | + put_pid(t->pid); | |
| 1211 | + damon_destroy_target(t); | |
| 1212 | + } | |
| 1213 | + i++; | |
| 1214 | + } | |
| 1215 | + | |
| 1216 | + for (; i < sysfs_targets->nr; i++) { | |
| 1196 | 1217 | struct damon_sysfs_target *st = sysfs_targets->targets_arr[i]; |
| 1197 | − struct damon_target *t = damon_sysfs_existing_target(st, ctx); | |
| 1198 | − | |
| 1199 | − if (IS_ERR(t)) | |
| 1200 | − return PTR_ERR(t); | |
| 1201 | − if (!t) | |
| 1202 | − err = damon_sysfs_add_target(st, ctx); | |
| 1203 | − else | |
| 1204 | − err = damon_sysfs_set_regions(t, st->regions); | |
| 1218 | + | |
| 1219 | + err = damon_sysfs_add_target(st, ctx); | |
| 1205 | 1220 | if (err) |
| 1206 | 1221 | return err; |
| 1207 | 1222 | } |
| @@ -1336,12 +1351,13 @@ static int damon_sysfs_commit_input(struct damon_sysfs_kdamond *kdamond) · 128 unmodified lines | ||
| 1336 | 1351 | |
| 1337 | 1352 | /* |
| 1338 | 1353 | * damon_sysfs_cmd_request_callback() - DAMON callback for handling requests. |
| 1339 | − * @c: The DAMON context of the callback. | |
| 1354 | + * @c: The DAMON context of the callback. | |
| 1355 | + * @active: Whether @c is not deactivated due to watermarks. | |
| 1340 | 1356 | * |
| 1341 | 1357 | * This function is periodically called back from the kdamond thread for @c. |
| 1342 | 1358 | * Then, it checks if there is a waiting DAMON sysfs request and handles it. |
| 1343 | 1359 | */ |
| 1344 | −static int damon_sysfs_cmd_request_callback(struct damon_ctx *c) | |
| 1360 | +static int damon_sysfs_cmd_request_callback(struct damon_ctx *c, bool active) | |
| 1345 | 1361 | { |
| 1346 | 1362 | struct damon_sysfs_kdamond *kdamond; |
| 1347 | 1363 | bool total_bytes_only = false; |
| @@ -1373,6 +1389,13 @@ static int damon_sysfs_cmd_request_callback(struct damon_ctx *c) · 25 unmodified lines | ||
| 1373 | 1389 | goto keep_lock_out; |
| 1374 | 1390 | } |
| 1375 | 1391 | } else { |
| 1392 | + /* | |
| 1393 | + * Continue regions updating if DAMON is till | |
| 1394 | + * active and the update for all schemes is not | |
| 1395 | + * finished. | |
| 1396 | + */ | |
| 1397 | + if (active && !damos_sysfs_regions_upd_done()) | |
| 1398 | + goto keep_lock_out; | |
| 1376 | 1399 | err = damon_sysfs_upd_schemes_regions_stop(kdamond); |
| 1377 | 1400 | damon_sysfs_schemes_regions_updating = false; |
| 1378 | 1401 | } |
| @@ -1392,6 +1415,24 @@ keep_lock_out: · 13 unmodified lines | ||
| 1392 | 1415 | return err; |
| 1393 | 1416 | } |
| 1394 | 1417 | |
| 1418 | +static int damon_sysfs_after_wmarks_check(struct damon_ctx *c) | |
| 1419 | +{ | |
| 1420 | + /* | |
| 1421 | + * after_wmarks_check() is called back while the context is deactivated | |
| 1422 | + * by watermarks. | |
| 1423 | + */ | |
| 1424 | + return damon_sysfs_cmd_request_callback(c, false); | |
| 1425 | +} | |
| 1426 | + | |
| 1427 | +static int damon_sysfs_after_aggregation(struct damon_ctx *c) | |
| 1428 | +{ | |
| 1429 | + /* | |
| 1430 | + * after_aggregation() is called back only while the context is not | |
| 1431 | + * deactivated by watermarks. | |
| 1432 | + */ | |
| 1433 | + return damon_sysfs_cmd_request_callback(c, true); | |
| 1434 | +} | |
| 1435 | + | |
| 1395 | 1436 | static struct damon_ctx *damon_sysfs_build_ctx( |
| 1396 | 1437 | struct damon_sysfs_context *sys_ctx) |
| 1397 | 1438 | { |
| @@ -1407,8 +1448,8 @@ static struct damon_ctx *damon_sysfs_build_ctx( · 9 unmodified lines | ||
| 1407 | 1448 | return ERR_PTR(err); |
| 1408 | 1449 | } |
| 1409 | 1450 | |
| 1410 | − ctx->callback.after_wmarks_check = damon_sysfs_cmd_request_callback; | |
| 1411 | − ctx->callback.after_aggregation = damon_sysfs_cmd_request_callback; | |
| 1451 | + ctx->callback.after_wmarks_check = damon_sysfs_after_wmarks_check; | |
| 1452 | + ctx->callback.after_aggregation = damon_sysfs_after_aggregation; | |
| 1412 | 1453 | ctx->callback.before_terminate = damon_sysfs_before_terminate; |
| 1413 | 1454 | return ctx; |
| 1414 | 1455 | } |
| @@ -1810,3 +1851,5 @@ out: · 395 unmodified lines | ||
| 1810 | 1851 | return err; |
| 1811 | 1852 | } |
| 1812 | 1853 | subsys_initcall(damon_sysfs_init); |
| 1854 | + | |
| 1855 | +#include "sysfs-test.h" | |
| @@ -558,23 +558,27 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr, · 557 unmodified lines | ||
| 558 | 558 | * r the region to be checked |
| 559 | 559 | */ |
| 560 | 560 | static void __damon_va_check_access(struct mm_struct *mm, |
| 561 | − struct damon_region *r, bool same_target) | |
| 561 | + struct damon_region *r, bool same_target, | |
| 562 | + struct damon_attrs *attrs) | |
| 562 | 563 | { |
| 563 | 564 | static unsigned long last_addr; |
| 564 | 565 | static unsigned long last_folio_sz = PAGE_SIZE; |
| 565 | 566 | static bool last_accessed; |
| 566 | 567 | |
| 568 | + if (!mm) { | |
| 569 | + damon_update_region_access_rate(r, false, attrs); | |
| 570 | + return; | |
| 571 | + } | |
| 572 | + | |
| 567 | 573 | /* If the region is in the last checked page, reuse the result */ |
| 568 | 574 | if (same_target && (ALIGN_DOWN(last_addr, last_folio_sz) == |
| 569 | 575 | ALIGN_DOWN(r->sampling_addr, last_folio_sz))) { |
| 570 | − if (last_accessed) | |
| 571 | − r->nr_accesses++; | |
| 576 | + damon_update_region_access_rate(r, last_accessed, attrs); | |
| 572 | 577 | return; |
| 573 | 578 | } |
| 574 | 579 | |
| 575 | 580 | last_accessed = damon_va_young(mm, r->sampling_addr, &last_folio_sz); |
| 576 | − if (last_accessed) | |
| 577 | − r->nr_accesses++; | |
| 581 | + damon_update_region_access_rate(r, last_accessed, attrs); | |
| 578 | 582 | |
| 579 | 583 | last_addr = r->sampling_addr; |
| 580 | 584 | } |
| @@ -589,15 +593,15 @@ static unsigned int damon_va_check_accesses(struct damon_ctx *ctx) · 8 unmodified lines | ||
| 589 | 593 | |
| 590 | 594 | damon_for_each_target(t, ctx) { |
| 591 | 595 | mm = damon_get_mm(t); |
| 592 | − if (!mm) | |
| 593 | − continue; | |
| 594 | 596 | same_target = false; |
| 595 | 597 | damon_for_each_region(r, t) { |
| 596 | − __damon_va_check_access(mm, r, same_target); | |
| 598 | + __damon_va_check_access(mm, r, same_target, | |
| 599 | + &ctx->attrs); | |
| 597 | 600 | max_nr_accesses = max(r->nr_accesses, max_nr_accesses); |
| 598 | 601 | same_target = true; |
| 599 | 602 | } |
| 600 | − mmput(mm); | |
| 603 | + if (mm) | |
| 604 | + mmput(mm); | |
| 601 | 605 | } |
| 602 | 606 | |
| 603 | 607 | return max_nr_accesses; |
| @@ -1322,8 +1322,8 @@ static int __init debug_vm_pgtable(void) · 1321 unmodified lines | ||
| 1322 | 1322 | * true irrespective of the starting protection value for a |
| 1323 | 1323 | * given page table entry. |
| 1324 | 1324 | * |
| 1325 | − * Protection based vm_flags combinatins are always linear | |
| 1326 | − * and increasing i.e starting from VM_NONE and going upto | |
| 1325 | + * Protection based vm_flags combinations are always linear | |
| 1326 | + * and increasing i.e starting from VM_NONE and going up to | |
| 1327 | 1327 | * (VM_SHARED | READ | WRITE | EXEC). |
| 1328 | 1328 | */ |
| 1329 | 1329 | #define VM_FLAGS_START (VM_NONE) |
| @@ -131,11 +131,8 @@ static void page_cache_delete(struct address_space *mapping, · 130 unmodified lines | ||
| 131 | 131 | |
| 132 | 132 | mapping_set_update(&xas, mapping); |
| 133 | 133 | |
| 134 | − /* hugetlb pages are represented by a single entry in the xarray */ | |
| 135 | − if (!folio_test_hugetlb(folio)) { | |
| 136 | − xas_set_order(&xas, folio->index, folio_order(folio)); | |
| 137 | − nr = folio_nr_pages(folio); | |
| 138 | − } | |
| 134 | + xas_set_order(&xas, folio->index, folio_order(folio)); | |
| 135 | + nr = folio_nr_pages(folio); | |
| 139 | 136 | |
| 140 | 137 | VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); |
| 141 | 138 | |
| @@ -234,7 +231,7 @@ void filemap_free_folio(struct address_space *mapping, struct folio *folio) · 92 unmodified lines | ||
| 234 | 231 | if (free_folio) |
| 235 | 232 | free_folio(folio); |
| 236 | 233 | |
| 237 | − if (folio_test_large(folio) && !folio_test_hugetlb(folio)) | |
| 234 | + if (folio_test_large(folio)) | |
| 238 | 235 | refs = folio_nr_pages(folio); |
| 239 | 236 | folio_put_refs(folio, refs); |
| 240 | 237 | } |
| @@ -819,7 +816,7 @@ void replace_page_cache_folio(struct folio *old, struct folio *new) · 578 unmodified lines | ||
| 819 | 816 | new->mapping = mapping; |
| 820 | 817 | new->index = offset; |
| 821 | 818 | |
| 822 | − mem_cgroup_migrate(old, new); | |
| 819 | + mem_cgroup_replace_folio(old, new); | |
| 823 | 820 | |
| 824 | 821 | xas_lock_irq(&xas); |
| 825 | 822 | xas_store(&xas, new); |
| @@ -855,14 +852,15 @@ noinline int __filemap_add_folio(struct address_space *mapping, · 29 unmodified lines | ||
| 855 | 852 | |
| 856 | 853 | if (!huge) { |
| 857 | 854 | int error = mem_cgroup_charge(folio, NULL, gfp); |
| 858 | − VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); | |
| 859 | 855 | if (error) |
| 860 | 856 | return error; |
| 861 | 857 | charged = true; |
| 862 | − xas_set_order(&xas, index, folio_order(folio)); | |
| 863 | − nr = folio_nr_pages(folio); | |
| 864 | 858 | } |
| 865 | 859 | |
| 860 | + VM_BUG_ON_FOLIO(index & (folio_nr_pages(folio) - 1), folio); | |
| 861 | + xas_set_order(&xas, index, folio_order(folio)); | |
| 862 | + nr = folio_nr_pages(folio); | |
| 863 | + | |
| 866 | 864 | gfp &= GFP_RECLAIM_MASK; |
| 867 | 865 | folio_ref_add(folio, nr); |
| 868 | 866 | folio->mapping = mapping; |
| @@ -1135,32 +1133,13 @@ static void folio_wake_bit(struct folio *folio, int bit_nr) · 266 unmodified lines | ||
| 1135 | 1133 | wait_queue_head_t *q = folio_waitqueue(folio); |
| 1136 | 1134 | struct wait_page_key key; |
| 1137 | 1135 | unsigned long flags; |
| 1138 | − wait_queue_entry_t bookmark; | |
| 1139 | 1136 | |
| 1140 | 1137 | key.folio = folio; |
| 1141 | 1138 | key.bit_nr = bit_nr; |
| 1142 | 1139 | key.page_match = 0; |
| 1143 | 1140 | |
| 1144 | − bookmark.flags = 0; | |
| 1145 | − bookmark.private = NULL; | |
| 1146 | − bookmark.func = NULL; | |
| 1147 | − INIT_LIST_HEAD(&bookmark.entry); | |
| 1148 | − | |
| 1149 | 1141 | spin_lock_irqsave(&q->lock, flags); |
| 1150 | − __wake_up_locked_key_bookmark(q, TASK_NORMAL, &key, &bookmark); | |
| 1151 | − | |
| 1152 | − while (bookmark.flags & WQ_FLAG_BOOKMARK) { | |
| 1153 | − /* | |
| 1154 | − * Take a breather from holding the lock, | |
| 1155 | − * allow pages that finish wake up asynchronously | |
| 1156 | − * to acquire the lock and remove themselves | |
| 1157 | − * from wait queue | |
| 1158 | − */ | |
| 1159 | − spin_unlock_irqrestore(&q->lock, flags); | |
| 1160 | − cpu_relax(); | |
| 1161 | − spin_lock_irqsave(&q->lock, flags); | |
| 1162 | − __wake_up_locked_key_bookmark(q, TASK_NORMAL, &key, &bookmark); | |
| 1163 | − } | |
| 1142 | + __wake_up_locked_key(q, TASK_NORMAL, &key); | |
| 1164 | 1143 | |
| 1165 | 1144 | /* |
| 1166 | 1145 | * It's possible to miss clearing waiters here, when we woke our page |
| @@ -1177,13 +1156,6 @@ static void folio_wake_bit(struct folio *folio, int bit_nr) · 10 unmodified lines | ||
| 1177 | 1156 | spin_unlock_irqrestore(&q->lock, flags); |
| 1178 | 1157 | } |
| 1179 | 1158 | |
| 1180 | −static void folio_wake(struct folio *folio, int bit) | |
| 1181 | −{ | |
| 1182 | − if (!folio_test_waiters(folio)) | |
| 1183 | − return; | |
| 1184 | − folio_wake_bit(folio, bit); | |
| 1185 | −} | |
| 1186 | − | |
| 1187 | 1159 | /* |
| 1188 | 1160 | * A choice of three behaviors for folio_wait_bit_common(): |
| 1189 | 1161 | */ |
| @@ -1484,29 +1456,6 @@ void folio_add_wait_queue(struct folio *folio, wait_queue_entry_t *waiter) · 294 unmodified lines | ||
| 1484 | 1456 | } |
| 1485 | 1457 | EXPORT_SYMBOL_GPL(folio_add_wait_queue); |
| 1486 | 1458 | |
| 1487 | −#ifndef clear_bit_unlock_is_negative_byte | |
| 1488 | − | |
| 1489 | −/* | |
| 1490 | − * PG_waiters is the high bit in the same byte as PG_lock. | |
| 1491 | − * | |
| 1492 | − * On x86 (and on many other architectures), we can clear PG_lock and | |
| 1493 | − * test the sign bit at the same time. But if the architecture does | |
| 1494 | − * not support that special operation, we just do this all by hand | |
| 1495 | − * instead. | |
| 1496 | − * | |
| 1497 | − * The read of PG_waiters has to be after (or concurrently with) PG_locked | |
| 1498 | − * being cleared, but a memory barrier should be unnecessary since it is | |
| 1499 | − * in the same byte as PG_locked. | |
| 1500 | − */ | |
| 1501 | −static inline bool clear_bit_unlock_is_negative_byte(long nr, volatile void *mem) | |
| 1502 | −{ | |
| 1503 | − clear_bit_unlock(nr, mem); | |
| 1504 | − /* smp_mb__after_atomic(); */ | |
| 1505 | − return test_bit(PG_waiters, mem); | |
| 1506 | −} | |
| 1507 | − | |
| 1508 | −#endif | |
| 1509 | − | |
| 1510 | 1459 | /** |
| 1511 | 1460 | * folio_unlock - Unlock a locked folio. |
| 1512 | 1461 | * @folio: The folio. |
| @@ -1522,11 +1471,41 @@ void folio_unlock(struct folio *folio) · 9 unmodified lines | ||
| 1522 | 1471 | BUILD_BUG_ON(PG_waiters != 7); |
| 1523 | 1472 | BUILD_BUG_ON(PG_locked > 7); |
| 1524 | 1473 | VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); |
| 1525 | − if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio, 0))) | |
| 1474 | + if (folio_xor_flags_has_waiters(folio, 1 << PG_locked)) | |
| 1526 | 1475 | folio_wake_bit(folio, PG_locked); |
| 1527 | 1476 | } |
| 1528 | 1477 | EXPORT_SYMBOL(folio_unlock); |
| 1529 | 1478 | |
| 1479 | +/** | |
| 1480 | + * folio_end_read - End read on a folio. | |
| 1481 | + * @folio: The folio. | |
| 1482 | + * @success: True if all reads completed successfully. | |
| 1483 | + * | |
| 1484 | + * When all reads against a folio have completed, filesystems should | |
| 1485 | + * call this function to let the pagecache know that no more reads | |
| 1486 | + * are outstanding. This will unlock the folio and wake up any thread | |
| 1487 | + * sleeping on the lock. The folio will also be marked uptodate if all | |
| 1488 | + * reads succeeded. | |
| 1489 | + * | |
| 1490 | + * Context: May be called from interrupt or process context. May not be | |
| 1491 | + * called from NMI context. | |
| 1492 | + */ | |
| 1493 | +void folio_end_read(struct folio *folio, bool success) | |
| 1494 | +{ | |
| 1495 | + unsigned long mask = 1 << PG_locked; | |
| 1496 | + | |
| 1497 | + /* Must be in bottom byte for x86 to work */ | |
| 1498 | + BUILD_BUG_ON(PG_uptodate > 7); | |
| 1499 | + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); | |
| 1500 | + VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio); | |
| 1501 | + | |
| 1502 | + if (likely(success)) | |
| 1503 | + mask |= 1 << PG_uptodate; | |
| 1504 | + if (folio_xor_flags_has_waiters(folio, mask)) | |
| 1505 | + folio_wake_bit(folio, PG_locked); | |
| 1506 | +} | |
| 1507 | +EXPORT_SYMBOL(folio_end_read); | |
| 1508 | + | |
| 1530 | 1509 | /** |
| 1531 | 1510 | * folio_end_private_2 - Clear PG_private_2 and wake any waiters. |
| 1532 | 1511 | * @folio: The folio. |
| @@ -1588,9 +1567,15 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); · 55 unmodified lines | ||
| 1588 | 1567 | /** |
| 1589 | 1568 | * folio_end_writeback - End writeback against a folio. |
| 1590 | 1569 | * @folio: The folio. |
| 1570 | + * | |
| 1571 | + * The folio must actually be under writeback. | |
| 1572 | + * | |
| 1573 | + * Context: May be called from process or interrupt context. | |
| 1591 | 1574 | */ |
| 1592 | 1575 | void folio_end_writeback(struct folio *folio) |
| 1593 | 1576 | { |
| 1577 | + VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio); | |
| 1578 | + | |
| 1594 | 1579 | /* |
| 1595 | 1580 | * folio_test_clear_reclaim() could be used here but it is an |
| 1596 | 1581 | * atomic operation and overkill in this particular case. Failing |
| @@ -1607,14 +1592,11 @@ void folio_end_writeback(struct folio *folio) · 10 unmodified lines | ||
| 1607 | 1592 | * Writeback does not hold a folio reference of its own, relying |
| 1608 | 1593 | * on truncation to wait for the clearing of PG_writeback. |
| 1609 | 1594 | * But here we must make sure that the folio is not freed and |
| 1610 | − * reused before the folio_wake(). | |
| 1595 | + * reused before the folio_wake_bit(). | |
| 1611 | 1596 | */ |
| 1612 | 1597 | folio_get(folio); |
| 1613 | − if (!__folio_end_writeback(folio)) | |
| 1614 | − BUG(); | |
| 1615 | − | |
| 1616 | − smp_mb__after_atomic(); | |
| 1617 | − folio_wake(folio, PG_writeback); | |
| 1598 | + if (__folio_end_writeback(folio)) | |
| 1599 | + folio_wake_bit(folio, PG_writeback); | |
| 1618 | 1600 | acct_reclaim_writeback(folio); |
| 1619 | 1601 | folio_put(folio); |
| 1620 | 1602 | } |
| @@ -2040,7 +2022,7 @@ unsigned find_get_entries(struct address_space *mapping, pgoff_t *start, · 419 unmodified lines | ||
| 2040 | 2022 | int idx = folio_batch_count(fbatch) - 1; |
| 2041 | 2023 | |
| 2042 | 2024 | folio = fbatch->folios[idx]; |
| 2043 | − if (!xa_is_value(folio) && !folio_test_hugetlb(folio)) | |
| 2025 | + if (!xa_is_value(folio)) | |
| 2044 | 2026 | nr = folio_nr_pages(folio); |
| 2045 | 2027 | *start = indices[idx] + nr; |
| 2046 | 2028 | } |
| @@ -2104,7 +2086,7 @@ put: · 57 unmodified lines | ||
| 2104 | 2086 | int idx = folio_batch_count(fbatch) - 1; |
| 2105 | 2087 | |
| 2106 | 2088 | folio = fbatch->folios[idx]; |
| 2107 | − if (!xa_is_value(folio) && !folio_test_hugetlb(folio)) | |
| 2089 | + if (!xa_is_value(folio)) | |
| 2108 | 2090 | nr = folio_nr_pages(folio); |
| 2109 | 2091 | *start = indices[idx] + nr; |
| 2110 | 2092 | } |
| @@ -2122,51 +2104,13 @@ put: · 11 unmodified lines | ||
| 2122 | 2104 | * index @start and up to index @end (inclusive). The folios are returned |
| 2123 | 2105 | * in @fbatch with an elevated reference count. |
| 2124 | 2106 | * |
| 2125 | − * The first folio may start before @start; if it does, it will contain | |
| 2126 | − * @start. The final folio may extend beyond @end; if it does, it will | |
| 2127 | − * contain @end. The folios have ascending indices. There may be gaps | |
| 2128 | − * between the folios if there are indices which have no folio in the | |
| 2129 | − * page cache. If folios are added to or removed from the page cache | |
| 2130 | − * while this is running, they may or may not be found by this call. | |
| 2131 | − * | |
| 2132 | 2107 | * Return: The number of folios which were found. |
| 2133 | 2108 | * We also update @start to index the next folio for the traversal. |
| 2134 | 2109 | */ |
| 2135 | 2110 | unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start, |
| 2136 | 2111 | pgoff_t end, struct folio_batch *fbatch) |
| 2137 | 2112 | { |
| 2138 | − XA_STATE(xas, &mapping->i_pages, *start); | |
| 2139 | − struct folio *folio; | |
| 2140 | − | |
| 2141 | − rcu_read_lock(); | |
| 2142 | − while ((folio = find_get_entry(&xas, end, XA_PRESENT)) != NULL) { | |
| 2143 | − /* Skip over shadow, swap and DAX entries */ | |
| 2144 | − if (xa_is_value(folio)) | |
| 2145 | − continue; | |
| 2146 | − if (!folio_batch_add(fbatch, folio)) { | |
| 2147 | − unsigned long nr = folio_nr_pages(folio); | |
| 2148 | − | |
| 2149 | − if (folio_test_hugetlb(folio)) | |
| 2150 | − nr = 1; | |
| 2151 | − *start = folio->index + nr; | |
| 2152 | − goto out; | |
| 2153 | − } | |
| 2154 | − } | |
| 2155 | − | |
| 2156 | − /* | |
| 2157 | − * We come here when there is no page beyond @end. We take care to not | |
| 2158 | − * overflow the index @start as it confuses some of the callers. This | |
| 2159 | − * breaks the iteration when there is a page at index -1 but that is | |
| 2160 | − * already broken anyway. | |
| 2161 | − */ | |
| 2162 | − if (end == (pgoff_t)-1) | |
| 2163 | − *start = (pgoff_t)-1; | |
| 2164 | − else | |
| 2165 | − *start = end + 1; | |
| 2166 | −out: | |
| 2167 | − rcu_read_unlock(); | |
| 2168 | − | |
| 2169 | − return folio_batch_count(fbatch); | |
| 2113 | + return filemap_get_folios_tag(mapping, start, end, XA_PRESENT, fbatch); | |
| 2170 | 2114 | } |
| 2171 | 2115 | EXPORT_SYMBOL(filemap_get_folios); |
| 2172 | 2116 | |
| @@ -2213,9 +2157,6 @@ unsigned filemap_get_folios_contig(struct address_space *mapping, · 40 unmodified lines | ||
| 2213 | 2157 | |
| 2214 | 2158 | if (!folio_batch_add(fbatch, folio)) { |
| 2215 | 2159 | nr = folio_nr_pages(folio); |
| 2216 | − | |
| 2217 | − if (folio_test_hugetlb(folio)) | |
| 2218 | − nr = 1; | |
| 2219 | 2160 | *start = folio->index + nr; |
| 2220 | 2161 | goto out; |
| 2221 | 2162 | } |
| @@ -2232,10 +2173,7 @@ update_start: · 10 unmodified lines | ||
| 2232 | 2173 | |
| 2233 | 2174 | if (nr) { |
| 2234 | 2175 | folio = fbatch->folios[nr - 1]; |
| 2235 | − if (folio_test_hugetlb(folio)) | |
| 2236 | − *start = folio->index + 1; | |
| 2237 | − else | |
| 2238 | − *start = folio_next_index(folio); | |
| 2176 | + *start = folio->index + folio_nr_pages(folio); | |
| 2239 | 2177 | } |
| 2240 | 2178 | out: |
| 2241 | 2179 | rcu_read_unlock(); |
| @@ -2251,7 +2189,13 @@ EXPORT_SYMBOL(filemap_get_folios_contig); · 9 unmodified lines | ||
| 2251 | 2189 | * @tag: The tag index |
| 2252 | 2190 | * @fbatch: The batch to fill |
| 2253 | 2191 | * |
| 2254 | − * Same as filemap_get_folios(), but only returning folios tagged with @tag. | |
| 2192 | + * The first folio may start before @start; if it does, it will contain | |
| 2193 | + * @start. The final folio may extend beyond @end; if it does, it will | |
| 2194 | + * contain @end. The folios have ascending indices. There may be gaps | |
| 2195 | + * between the folios if there are indices which have no folio in the | |
| 2196 | + * page cache. If folios are added to or removed from the page cache | |
| 2197 | + * while this is running, they may or may not be found by this call. | |
| 2198 | + * Only returns folios that are tagged with @tag. | |
| 2255 | 2199 | * |
| 2256 | 2200 | * Return: The number of folios found. |
| 2257 | 2201 | * Also update @start to index the next folio for traversal. |
| @@ -2273,9 +2217,6 @@ unsigned filemap_get_folios_tag(struct address_space *mapping, pgoff_t *start, · 15 unmodified lines | ||
| 2273 | 2217 | continue; |
| 2274 | 2218 | if (!folio_batch_add(fbatch, folio)) { |
| 2275 | 2219 | unsigned long nr = folio_nr_pages(folio); |
| 2276 | − | |
| 2277 | − if (folio_test_hugetlb(folio)) | |
| 2278 | − nr = 1; | |
| 2279 | 2220 | *start = folio->index + nr; |
| 2280 | 2221 | goto out; |
| 2281 | 2222 | } |
| @@ -3104,7 +3045,7 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · 822 unmodified lines | ||
| 3104 | 3045 | |
| 3105 | 3046 | /* |
| 3106 | 3047 | * NOTE! This will make us return with VM_FAULT_RETRY, but with |
| 3107 | − * the mmap_lock still held. That's how FAULT_FLAG_RETRY_NOWAIT | |
| 3048 | + * the fault lock still held. That's how FAULT_FLAG_RETRY_NOWAIT | |
| 3108 | 3049 | * is supposed to work. We have way too many special cases.. |
| 3109 | 3050 | */ |
| 3110 | 3051 | if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT) |
| @@ -3114,13 +3055,14 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, · 3 unmodified lines | ||
| 3114 | 3055 | if (vmf->flags & FAULT_FLAG_KILLABLE) { |
| 3115 | 3056 | if (__folio_lock_killable(folio)) { |
| 3116 | 3057 | /* |
| 3117 | − * We didn't have the right flags to drop the mmap_lock, | |
| 3118 | − * but all fault_handlers only check for fatal signals | |
| 3119 | − * if we return VM_FAULT_RETRY, so we need to drop the | |
| 3120 | − * mmap_lock here and return 0 if we don't have a fpin. | |
| 3058 | + * We didn't have the right flags to drop the | |
| 3059 | + * fault lock, but all fault_handlers only check | |
| 3060 | + * for fatal signals if we return VM_FAULT_RETRY, | |
| 3061 | + * so we need to drop the fault lock here and | |
| 3062 | + * return 0 if we don't have a fpin. | |
| 3121 | 3063 | */ |
| 3122 | 3064 | if (*fpin == NULL) |
| 3123 | − mmap_read_unlock(vmf->vma->vm_mm); | |
| 3065 | + release_fault_lock(vmf); | |
| 3124 | 3066 | return 0; |
| 3125 | 3067 | } |
| 3126 | 3068 | } else |
| @@ -3321,21 +3263,28 @@ retry_find: · 194 unmodified lines | ||
| 3321 | 3263 | VM_BUG_ON_FOLIO(!folio_contains(folio, index), folio); |
| 3322 | 3264 | |
| 3323 | 3265 | /* |
| 3324 | − * We have a locked page in the page cache, now we need to check | |
| 3325 | − * that it's up-to-date. If not, it is going to be due to an error. | |
| 3266 | + * We have a locked folio in the page cache, now we need to check | |
| 3267 | + * that it's up-to-date. If not, it is going to be due to an error, | |
| 3268 | + * or because readahead was otherwise unable to retrieve it. | |
| 3326 | 3269 | */ |
| 3327 | 3270 | if (unlikely(!folio_test_uptodate(folio))) { |
| 3328 | 3271 | /* |
| 3329 | − * The page was in cache and uptodate and now it is not. | |
| 3330 | − * Strange but possible since we didn't hold the page lock all | |
| 3331 | − * the time. Let's drop everything get the invalidate lock and | |
| 3332 | − * try again. | |
| 3272 | + * If the invalidate lock is not held, the folio was in cache | |
| 3273 | + * and uptodate and now it is not. Strange but possible since we | |
| 3274 | + * didn't hold the page lock all the time. Let's drop | |
| 3275 | + * everything, get the invalidate lock and try again. | |
| 3333 | 3276 | */ |
| 3334 | 3277 | if (!mapping_locked) { |
| 3335 | 3278 | folio_unlock(folio); |
| 3336 | 3279 | folio_put(folio); |
| 3337 | 3280 | goto retry_find; |
| 3338 | 3281 | } |
| 3282 | + | |
| 3283 | + /* | |
| 3284 | + * OK, the folio is really not uptodate. This can be because the | |
| 3285 | + * VMA has the VM_RAND_READ flag set, or because an error | |
| 3286 | + * arose. Let's read it in directly. | |
| 3287 | + */ | |
| 3339 | 3288 | goto page_not_uptodate; |
| 3340 | 3289 | } |
| 3341 | 3290 | |
| @@ -3591,7 +3540,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, · 249 unmodified lines | ||
| 3591 | 3540 | addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT; |
| 3592 | 3541 | vmf->pte += xas.xa_index - last_pgoff; |
| 3593 | 3542 | last_pgoff = xas.xa_index; |
| 3594 | − end = folio->index + folio_nr_pages(folio) - 1; | |
| 3543 | + end = folio_next_index(folio) - 1; | |
| 3595 | 3544 | nr_pages = min(end, end_pgoff) - xas.xa_index + 1; |
| 3596 | 3545 | |
| 3597 | 3546 | if (!folio_test_large(folio)) |
| @@ -3669,7 +3618,7 @@ int generic_file_mmap(struct file *file, struct vm_area_struct *vma) · 71 unmodified lines | ||
| 3669 | 3618 | */ |
| 3670 | 3619 | int generic_file_readonly_mmap(struct file *file, struct vm_area_struct *vma) |
| 3671 | 3620 | { |
| 3672 | − if ((vma->vm_flags & VM_SHARED) && (vma->vm_flags & VM_MAYWRITE)) | |
| 3621 | + if (vma_is_shared_maywrite(vma)) | |
| 3673 | 3622 | return -EINVAL; |
| 3674 | 3623 | return generic_file_mmap(file, vma); |
| 3675 | 3624 | } |
| @@ -1471,6 +1471,9 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, · 1470 unmodified lines | ||
| 1471 | 1471 | long ret, pages_done; |
| 1472 | 1472 | bool must_unlock = false; |
| 1473 | 1473 | |
| 1474 | + if (!nr_pages) | |
| 1475 | + return 0; | |
| 1476 | + | |
| 1474 | 1477 | /* |
| 1475 | 1478 | * The internal caller expects GUP to manage the lock internally and the |
| 1476 | 1479 | * lock must be released when this returns. |
| @@ -1595,6 +1598,14 @@ retry: · 118 unmodified lines | ||
| 1595 | 1598 | mmap_read_unlock(mm); |
| 1596 | 1599 | *locked = 0; |
| 1597 | 1600 | } |
| 1601 | + | |
| 1602 | + /* | |
| 1603 | + * Failing to pin anything implies something has gone wrong (except when | |
| 1604 | + * FOLL_NOWAIT is specified). | |
| 1605 | + */ | |
| 1606 | + if (WARN_ON_ONCE(pages_done == 0 && !(flags & FOLL_NOWAIT))) | |
| 1607 | + return -EFAULT; | |
| 1608 | + | |
| 1598 | 1609 | return pages_done; |
| 1599 | 1610 | } |
| 1600 | 1611 | |
| @@ -2227,12 +2238,11 @@ static bool is_valid_gup_args(struct page **pages, int *locked, · 626 unmodified lines | ||
| 2227 | 2238 | /* |
| 2228 | 2239 | * These flags not allowed to be specified externally to the gup |
| 2229 | 2240 | * interfaces: |
| 2230 | − * - FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY are internal only | |
| 2241 | + * - FOLL_TOUCH/FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY are internal only | |
| 2231 | 2242 | * - FOLL_REMOTE is internal only and used on follow_page() |
| 2232 | 2243 | * - FOLL_UNLOCKABLE is internal only and used if locked is !NULL |
| 2233 | 2244 | */ |
| 2234 | − if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | FOLL_UNLOCKABLE | | |
| 2235 | − FOLL_REMOTE | FOLL_FAST_ONLY))) | |
| 2245 | + if (WARN_ON_ONCE(gup_flags & INTERNAL_GUP_FLAGS)) | |
| 2236 | 2246 | return false; |
| 2237 | 2247 | |
| 2238 | 2248 | gup_flags |= to_set; |
| @@ -65,7 +65,11 @@ unsigned long transparent_hugepage_flags __read_mostly = · 64 unmodified lines | ||
| 65 | 65 | (1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)| |
| 66 | 66 | (1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG); |
| 67 | 67 | |
| 68 | −static struct shrinker deferred_split_shrinker; | |
| 68 | +static struct shrinker *deferred_split_shrinker; | |
| 69 | +static unsigned long deferred_split_count(struct shrinker *shrink, | |
| 70 | + struct shrink_control *sc); | |
| 71 | +static unsigned long deferred_split_scan(struct shrinker *shrink, | |
| 72 | + struct shrink_control *sc); | |
| 69 | 73 | |
| 70 | 74 | static atomic_t huge_zero_refcount; |
| 71 | 75 | struct page *huge_zero_page __read_mostly; |
| @@ -96,11 +100,11 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, · 24 unmodified lines | ||
| 96 | 100 | return in_pf; |
| 97 | 101 | |
| 98 | 102 | /* |
| 99 | − * Special VMA and hugetlb VMA. | |
| 103 | + * khugepaged special VMA and hugetlb VMA. | |
| 100 | 104 | * Must be checked after dax since some dax mappings may have |
| 101 | 105 | * VM_MIXEDMAP set. |
| 102 | 106 | */ |
| 103 | − if (vm_flags & VM_NO_KHUGEPAGED) | |
| 107 | + if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED)) | |
| 104 | 108 | return false; |
| 105 | 109 | |
| 106 | 110 | /* |
| @@ -128,12 +132,18 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, · 21 unmodified lines | ||
| 128 | 132 | !hugepage_flags_always()))) |
| 129 | 133 | return false; |
| 130 | 134 | |
| 131 | − /* Only regular file is valid */ | |
| 132 | − if (!in_pf && file_thp_enabled(vma)) | |
| 133 | − return true; | |
| 134 | − | |
| 135 | − if (!vma_is_anonymous(vma)) | |
| 135 | + if (!vma_is_anonymous(vma)) { | |
| 136 | + /* | |
| 137 | + * Trust that ->huge_fault() handlers know what they are doing | |
| 138 | + * in fault path. | |
| 139 | + */ | |
| 140 | + if (((in_pf || smaps)) && vma->vm_ops->huge_fault) | |
| 141 | + return true; | |
| 142 | + /* Only regular file is valid in collapse path */ | |
| 143 | + if (((!in_pf || smaps)) && file_thp_enabled(vma)) | |
| 144 | + return true; | |
| 136 | 145 | return false; |
| 146 | + } | |
| 137 | 147 | |
| 138 | 148 | if (vma_is_temporary_stack(vma)) |
| 139 | 149 | return false; |
| @@ -229,11 +239,7 @@ static unsigned long shrink_huge_zero_page_scan(struct shrinker *shrink, · 89 unmodified lines | ||
| 229 | 239 | return 0; |
| 230 | 240 | } |
| 231 | 241 | |
| 232 | −static struct shrinker huge_zero_page_shrinker = { | |
| 233 | − .count_objects = shrink_huge_zero_page_count, | |
| 234 | − .scan_objects = shrink_huge_zero_page_scan, | |
| 235 | − .seeks = DEFAULT_SEEKS, | |
| 236 | −}; | |
| 242 | +static struct shrinker *huge_zero_page_shrinker; | |
| 237 | 243 | |
| 238 | 244 | #ifdef CONFIG_SYSFS |
| 239 | 245 | static ssize_t enabled_show(struct kobject *kobj, |
| @@ -454,6 +460,38 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj) · 214 unmodified lines | ||
| 454 | 460 | } |
| 455 | 461 | #endif /* CONFIG_SYSFS */ |
| 456 | 462 | |
| 463 | +static int __init thp_shrinker_init(void) | |
| 464 | +{ | |
| 465 | + huge_zero_page_shrinker = shrinker_alloc(0, "thp-zero"); | |
| 466 | + if (!huge_zero_page_shrinker) | |
| 467 | + return -ENOMEM; | |
| 468 | + | |
| 469 | + deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE | | |
| 470 | + SHRINKER_MEMCG_AWARE | | |
| 471 | + SHRINKER_NONSLAB, | |
| 472 | + "thp-deferred_split"); | |
| 473 | + if (!deferred_split_shrinker) { | |
| 474 | + shrinker_free(huge_zero_page_shrinker); | |
| 475 | + return -ENOMEM; | |
| 476 | + } | |
| 477 | + | |
| 478 | + huge_zero_page_shrinker->count_objects = shrink_huge_zero_page_count; | |
| 479 | + huge_zero_page_shrinker->scan_objects = shrink_huge_zero_page_scan; | |
| 480 | + shrinker_register(huge_zero_page_shrinker); | |
| 481 | + | |
| 482 | + deferred_split_shrinker->count_objects = deferred_split_count; | |
| 483 | + deferred_split_shrinker->scan_objects = deferred_split_scan; | |
| 484 | + shrinker_register(deferred_split_shrinker); | |
| 485 | + | |
| 486 | + return 0; | |
| 487 | +} | |
| 488 | + | |
| 489 | +static void __init thp_shrinker_exit(void) | |
| 490 | +{ | |
| 491 | + shrinker_free(huge_zero_page_shrinker); | |
| 492 | + shrinker_free(deferred_split_shrinker); | |
| 493 | +} | |
| 494 | + | |
| 457 | 495 | static int __init hugepage_init(void) |
| 458 | 496 | { |
| 459 | 497 | int err; |
| @@ -482,12 +520,9 @@ static int __init hugepage_init(void) · 22 unmodified lines | ||
| 482 | 520 | if (err) |
| 483 | 521 | goto err_slab; |
| 484 | 522 | |
| 485 | − err = register_shrinker(&huge_zero_page_shrinker, "thp-zero"); | |
| 523 | + err = thp_shrinker_init(); | |
| 486 | 524 | if (err) |
| 487 | − goto err_hzp_shrinker; | |
| 488 | − err = register_shrinker(&deferred_split_shrinker, "thp-deferred_split"); | |
| 489 | − if (err) | |
| 490 | − goto err_split_shrinker; | |
| 525 | + goto err_shrinker; | |
| 491 | 526 | |
| 492 | 527 | /* |
| 493 | 528 | * By default disable transparent hugepages on smaller systems, |
| @@ -505,10 +540,8 @@ static int __init hugepage_init(void) · 11 unmodified lines | ||
| 505 | 540 | |
| 506 | 541 | return 0; |
| 507 | 542 | err_khugepaged: |
| 508 | − unregister_shrinker(&deferred_split_shrinker); | |
| 509 | −err_split_shrinker: | |
| 510 | − unregister_shrinker(&huge_zero_page_shrinker); | |
| 511 | −err_hzp_shrinker: | |
| 543 | + thp_shrinker_exit(); | |
| 544 | +err_shrinker: | |
| 512 | 545 | khugepaged_destroy(); |
| 513 | 546 | err_slab: |
| 514 | 547 | hugepage_exit_sysfs(hugepage_kobj); |
| @@ -1349,7 +1382,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) · 834 unmodified lines | ||
| 1349 | 1382 | if (folio_ref_count(folio) == 1) { |
| 1350 | 1383 | pmd_t entry; |
| 1351 | 1384 | |
| 1352 | − page_move_anon_rmap(page, vma); | |
| 1385 | + folio_move_anon_rmap(folio, vma); | |
| 1386 | + SetPageAnonExclusive(page); | |
| 1353 | 1387 | folio_unlock(folio); |
| 1354 | 1388 | reuse: |
| 1355 | 1389 | if (unlikely(unshare)) { |
| @@ -1490,9 +1524,9 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 134 unmodified lines | ||
| 1490 | 1524 | struct vm_area_struct *vma = vmf->vma; |
| 1491 | 1525 | pmd_t oldpmd = vmf->orig_pmd; |
| 1492 | 1526 | pmd_t pmd; |
| 1493 | − struct page *page; | |
| 1527 | + struct folio *folio; | |
| 1494 | 1528 | unsigned long haddr = vmf->address & HPAGE_PMD_MASK; |
| 1495 | − int page_nid = NUMA_NO_NODE; | |
| 1529 | + int nid = NUMA_NO_NODE; | |
| 1496 | 1530 | int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK); |
| 1497 | 1531 | bool migrated = false, writable = false; |
| 1498 | 1532 | int flags = 0; |
| @@ -1514,36 +1548,34 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 15 unmodified lines | ||
| 1514 | 1548 | can_change_pmd_writable(vma, vmf->address, pmd)) |
| 1515 | 1549 | writable = true; |
| 1516 | 1550 | |
| 1517 | − page = vm_normal_page_pmd(vma, haddr, pmd); | |
| 1518 | − if (!page) | |
| 1551 | + folio = vm_normal_folio_pmd(vma, haddr, pmd); | |
| 1552 | + if (!folio) | |
| 1519 | 1553 | goto out_map; |
| 1520 | 1554 | |
| 1521 | 1555 | /* See similar comment in do_numa_page for explanation */ |
| 1522 | 1556 | if (!writable) |
| 1523 | 1557 | flags |= TNF_NO_GROUP; |
| 1524 | 1558 | |
| 1525 | − page_nid = page_to_nid(page); | |
| 1559 | + nid = folio_nid(folio); | |
| 1526 | 1560 | /* |
| 1527 | 1561 | * For memory tiering mode, cpupid of slow memory page is used |
| 1528 | 1562 | * to record page access time. So use default value. |
| 1529 | 1563 | */ |
| 1530 | − if (node_is_toptier(page_nid)) | |
| 1531 | − last_cpupid = page_cpupid_last(page); | |
| 1532 | − target_nid = numa_migrate_prep(page, vma, haddr, page_nid, | |
| 1533 | − &flags); | |
| 1534 | − | |
| 1564 | + if (node_is_toptier(nid)) | |
| 1565 | + last_cpupid = folio_last_cpupid(folio); | |
| 1566 | + target_nid = numa_migrate_prep(folio, vma, haddr, nid, &flags); | |
| 1535 | 1567 | if (target_nid == NUMA_NO_NODE) { |
| 1536 | − put_page(page); | |
| 1568 | + folio_put(folio); | |
| 1537 | 1569 | goto out_map; |
| 1538 | 1570 | } |
| 1539 | 1571 | |
| 1540 | 1572 | spin_unlock(vmf->ptl); |
| 1541 | 1573 | writable = false; |
| 1542 | 1574 | |
| 1543 | − migrated = migrate_misplaced_page(page, vma, target_nid); | |
| 1575 | + migrated = migrate_misplaced_folio(folio, vma, target_nid); | |
| 1544 | 1576 | if (migrated) { |
| 1545 | 1577 | flags |= TNF_MIGRATED; |
| 1546 | − page_nid = target_nid; | |
| 1578 | + nid = target_nid; | |
| 1547 | 1579 | } else { |
| 1548 | 1580 | flags |= TNF_MIGRATE_FAIL; |
| 1549 | 1581 | vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); |
| @@ -1555,9 +1587,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · 5 unmodified lines | ||
| 1555 | 1587 | } |
| 1556 | 1588 | |
| 1557 | 1589 | out: |
| 1558 | − if (page_nid != NUMA_NO_NODE) | |
| 1559 | − task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, | |
| 1560 | − flags); | |
| 1590 | + if (nid != NUMA_NO_NODE) | |
| 1591 | + task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags); | |
| 1561 | 1592 | |
| 1562 | 1593 | return 0; |
| 1563 | 1594 | |
| @@ -1825,7 +1856,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 261 unmodified lines | ||
| 1825 | 1856 | #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION |
| 1826 | 1857 | if (is_swap_pmd(*pmd)) { |
| 1827 | 1858 | swp_entry_t entry = pmd_to_swp_entry(*pmd); |
| 1828 | − struct page *page = pfn_swap_entry_to_page(entry); | |
| 1859 | + struct folio *folio = page_folio(pfn_swap_entry_to_page(entry)); | |
| 1829 | 1860 | pmd_t newpmd; |
| 1830 | 1861 | |
| 1831 | 1862 | VM_BUG_ON(!is_pmd_migration_entry(*pmd)); |
| @@ -1834,7 +1865,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 2 unmodified lines | ||
| 1834 | 1865 | * A protection check is difficult so |
| 1835 | 1866 | * just be safe and disable write |
| 1836 | 1867 | */ |
| 1837 | − if (PageAnon(page)) | |
| 1868 | + if (folio_test_anon(folio)) | |
| 1838 | 1869 | entry = make_readable_exclusive_migration_entry(swp_offset(entry)); |
| 1839 | 1870 | else |
| 1840 | 1871 | entry = make_readable_migration_entry(swp_offset(entry)); |
| @@ -1856,7 +1887,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 15 unmodified lines | ||
| 1856 | 1887 | #endif |
| 1857 | 1888 | |
| 1858 | 1889 | if (prot_numa) { |
| 1859 | − struct page *page; | |
| 1890 | + struct folio *folio; | |
| 1860 | 1891 | bool toptier; |
| 1861 | 1892 | /* |
| 1862 | 1893 | * Avoid trapping faults against the zero page. The read-only |
| @@ -1869,8 +1900,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 6 unmodified lines | ||
| 1869 | 1900 | if (pmd_protnone(*pmd)) |
| 1870 | 1901 | goto unlock; |
| 1871 | 1902 | |
| 1872 | − page = pmd_page(*pmd); | |
| 1873 | − toptier = node_is_toptier(page_to_nid(page)); | |
| 1903 | + folio = page_folio(pmd_page(*pmd)); | |
| 1904 | + toptier = node_is_toptier(folio_nid(folio)); | |
| 1874 | 1905 | /* |
| 1875 | 1906 | * Skip scanning top tier node if normal numa |
| 1876 | 1907 | * balancing is disabled |
| @@ -1881,7 +1912,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, · 4 unmodified lines | ||
| 1881 | 1912 | |
| 1882 | 1913 | if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && |
| 1883 | 1914 | !toptier) |
| 1884 | − xchg_page_access_time(page, jiffies_to_msecs(jiffies)); | |
| 1915 | + folio_xchg_access_time(folio, | |
| 1916 | + jiffies_to_msecs(jiffies)); | |
| 1885 | 1917 | } |
| 1886 | 1918 | /* |
| 1887 | 1919 | * In case prot_numa, we are under mmap_read_lock(mm). It's critical |
| @@ -2483,7 +2515,7 @@ static void __split_huge_page_tail(struct folio *folio, int tail, · 595 unmodified lines | ||
| 2483 | 2515 | if (page_is_idle(head)) |
| 2484 | 2516 | set_page_idle(page_tail); |
| 2485 | 2517 | |
| 2486 | − page_cpupid_xchg_last(page_tail, page_cpupid_last(head)); | |
| 2518 | + folio_xchg_last_cpupid(new_folio, folio_last_cpupid(folio)); | |
| 2487 | 2519 | |
| 2488 | 2520 | /* |
| 2489 | 2521 | * always add to the tail because some iterators expect new |
| @@ -2828,7 +2860,7 @@ void deferred_split_folio(struct folio *folio) · 338 unmodified lines | ||
| 2828 | 2860 | #ifdef CONFIG_MEMCG |
| 2829 | 2861 | if (memcg) |
| 2830 | 2862 | set_shrinker_bit(memcg, folio_nid(folio), |
| 2831 | − deferred_split_shrinker.id); | |
| 2863 | + deferred_split_shrinker->id); | |
| 2832 | 2864 | #endif |
| 2833 | 2865 | } |
| 2834 | 2866 | spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); |
| @@ -2902,14 +2934,6 @@ next: · 67 unmodified lines | ||
| 2902 | 2934 | return split; |
| 2903 | 2935 | } |
| 2904 | 2936 | |
| 2905 | −static struct shrinker deferred_split_shrinker = { | |
| 2906 | − .count_objects = deferred_split_count, | |
| 2907 | − .scan_objects = deferred_split_scan, | |
| 2908 | − .seeks = DEFAULT_SEEKS, | |
| 2909 | − .flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE | | |
| 2910 | − SHRINKER_NONSLAB, | |
| 2911 | −}; | |
| 2912 | − | |
| 2913 | 2937 | #ifdef CONFIG_DEBUG_FS |
| 2914 | 2938 | static void split_huge_pages_all(void) |
| 2915 | 2939 | { |
| @@ -984,7 +984,7 @@ static long region_count(struct resv_map *resv, long f, long t) · 983 unmodified lines | ||
| 984 | 984 | |
| 985 | 985 | /* |
| 986 | 986 | * Convert the address within this vma to the page offset within |
| 987 | − * the mapping, in pagecache page units; huge pages here. | |
| 987 | + * the mapping, huge page units here. | |
| 988 | 988 | */ |
| 989 | 989 | static pgoff_t vma_hugecache_offset(struct hstate *h, |
| 990 | 990 | struct vm_area_struct *vma, unsigned long address) |
| @@ -993,13 +993,6 @@ static pgoff_t vma_hugecache_offset(struct hstate *h, · 2 unmodified lines | ||
| 993 | 993 | (vma->vm_pgoff >> huge_page_order(h)); |
| 994 | 994 | } |
| 995 | 995 | |
| 996 | −pgoff_t linear_hugepage_index(struct vm_area_struct *vma, | |
| 997 | − unsigned long address) | |
| 998 | −{ | |
| 999 | − return vma_hugecache_offset(hstate_vma(vma), vma, address); | |
| 1000 | −} | |
| 1001 | −EXPORT_SYMBOL_GPL(linear_hugepage_index); | |
| 1002 | − | |
| 1003 | 996 | /** |
| 1004 | 997 | * vma_kernel_pagesize - Page size granularity for this VMA. |
| 1005 | 998 | * @vma: The user mapping. |
| @@ -1478,7 +1471,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, · 472 unmodified lines | ||
| 1478 | 1471 | } |
| 1479 | 1472 | |
| 1480 | 1473 | /* |
| 1481 | − * helper for remove_pool_huge_page() - return the previously saved | |
| 1474 | + * helper for remove_pool_hugetlb_folio() - return the previously saved | |
| 1482 | 1475 | * node ["this node"] from which to free a huge page. Advance the |
| 1483 | 1476 | * next node id whether or not we find a free huge page to free so |
| 1484 | 1477 | * that the next attempt to free addresses the next node. |
| @@ -1752,7 +1745,12 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, · 267 unmodified lines | ||
| 1752 | 1745 | if (folio_test_hugetlb_raw_hwp_unreliable(folio)) |
| 1753 | 1746 | return; |
| 1754 | 1747 | |
| 1755 | − if (hugetlb_vmemmap_restore(h, &folio->page)) { | |
| 1748 | + /* | |
| 1749 | + * If folio is not vmemmap optimized (!clear_dtor), then the folio | |
| 1750 | + * is no longer identified as a hugetlb page. hugetlb_vmemmap_restore_folio | |
| 1751 | + * can only be passed hugetlb pages and will BUG otherwise. | |
| 1752 | + */ | |
| 1753 | + if (clear_dtor && hugetlb_vmemmap_restore_folio(h, folio)) { | |
| 1756 | 1754 | spin_lock_irq(&hugetlb_lock); |
| 1757 | 1755 | /* |
| 1758 | 1756 | * If we cannot allocate vmemmap pages, just refuse to free the |
| @@ -1814,22 +1812,22 @@ static void free_hpage_workfn(struct work_struct *work) · 55 unmodified lines | ||
| 1814 | 1812 | node = llist_del_all(&hpage_freelist); |
| 1815 | 1813 | |
| 1816 | 1814 | while (node) { |
| 1817 | − struct page *page; | |
| 1815 | + struct folio *folio; | |
| 1818 | 1816 | struct hstate *h; |
| 1819 | 1817 | |
| 1820 | − page = container_of((struct address_space **)node, | |
| 1821 | − struct page, mapping); | |
| 1818 | + folio = container_of((struct address_space **)node, | |
| 1819 | + struct folio, mapping); | |
| 1822 | 1820 | node = node->next; |
| 1823 | − page->mapping = NULL; | |
| 1821 | + folio->mapping = NULL; | |
| 1824 | 1822 | /* |
| 1825 | 1823 | * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in |
| 1826 | 1824 | * folio_hstate() is going to trigger because a previous call to |
| 1827 | 1825 | * remove_hugetlb_folio() will clear the hugetlb bit, so do |
| 1828 | 1826 | * not use folio_hstate() directly. |
| 1829 | 1827 | */ |
| 1830 | − h = size_to_hstate(page_size(page)); | |
| 1828 | + h = size_to_hstate(folio_size(folio)); | |
| 1831 | 1829 | |
| 1832 | − __update_and_free_hugetlb_folio(h, page_folio(page)); | |
| 1830 | + __update_and_free_hugetlb_folio(h, folio); | |
| 1833 | 1831 | |
| 1834 | 1832 | cond_resched(); |
| 1835 | 1833 | } |
| @@ -1861,13 +1859,93 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, · 25 unmodified lines | ||
| 1861 | 1859 | schedule_work(&free_hpage_work); |
| 1862 | 1860 | } |
| 1863 | 1861 | |
| 1864 | −static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) | |
| 1862 | +static void bulk_vmemmap_restore_error(struct hstate *h, | |
| 1863 | + struct list_head *folio_list, | |
| 1864 | + struct list_head *non_hvo_folios) | |
| 1865 | 1865 | { |
| 1866 | − struct page *page, *t_page; | |
| 1867 | − struct folio *folio; | |
| 1866 | + struct folio *folio, *t_folio; | |
| 1867 | + | |
| 1868 | + if (!list_empty(non_hvo_folios)) { | |
| 1869 | + /* | |
| 1870 | + * Free any restored hugetlb pages so that restore of the | |
| 1871 | + * entire list can be retried. | |
| 1872 | + * The idea is that in the common case of ENOMEM errors freeing | |
| 1873 | + * hugetlb pages with vmemmap we will free up memory so that we | |
| 1874 | + * can allocate vmemmap for more hugetlb pages. | |
| 1875 | + */ | |
| 1876 | + list_for_each_entry_safe(folio, t_folio, non_hvo_folios, lru) { | |
| 1877 | + list_del(&folio->lru); | |
| 1878 | + spin_lock_irq(&hugetlb_lock); | |
| 1879 | + __clear_hugetlb_destructor(h, folio); | |
| 1880 | + spin_unlock_irq(&hugetlb_lock); | |
| 1881 | + update_and_free_hugetlb_folio(h, folio, false); | |
| 1882 | + cond_resched(); | |
| 1883 | + } | |
| 1884 | + } else { | |
| 1885 | + /* | |
| 1886 | + * In the case where there are no folios which can be | |
| 1887 | + * immediately freed, we loop through the list trying to restore | |
| 1888 | + * vmemmap individually in the hope that someone elsewhere may | |
| 1889 | + * have done something to cause success (such as freeing some | |
| 1890 | + * memory). If unable to restore a hugetlb page, the hugetlb | |
| 1891 | + * page is made a surplus page and removed from the list. | |
| 1892 | + * If are able to restore vmemmap and free one hugetlb page, we | |
| 1893 | + * quit processing the list to retry the bulk operation. | |
| 1894 | + */ | |
| 1895 | + list_for_each_entry_safe(folio, t_folio, folio_list, lru) | |
| 1896 | + if (hugetlb_vmemmap_restore_folio(h, folio)) { | |
| 1897 | + list_del(&folio->lru); | |
| 1898 | + spin_lock_irq(&hugetlb_lock); | |
| 1899 | + add_hugetlb_folio(h, folio, true); | |
| 1900 | + spin_unlock_irq(&hugetlb_lock); | |
| 1901 | + } else { | |
| 1902 | + list_del(&folio->lru); | |
| 1903 | + spin_lock_irq(&hugetlb_lock); | |
| 1904 | + __clear_hugetlb_destructor(h, folio); | |
| 1905 | + spin_unlock_irq(&hugetlb_lock); | |
| 1906 | + update_and_free_hugetlb_folio(h, folio, false); | |
| 1907 | + cond_resched(); | |
| 1908 | + break; | |
| 1909 | + } | |
| 1910 | + } | |
| 1911 | +} | |
| 1912 | + | |
| 1913 | +static void update_and_free_pages_bulk(struct hstate *h, | |
| 1914 | + struct list_head *folio_list) | |
| 1915 | +{ | |
| 1916 | + long ret; | |
| 1917 | + struct folio *folio, *t_folio; | |
| 1918 | + LIST_HEAD(non_hvo_folios); | |
| 1919 | + | |
| 1920 | + /* | |
| 1921 | + * First allocate required vmemmmap (if necessary) for all folios. | |
| 1922 | + * Carefully handle errors and free up any available hugetlb pages | |
| 1923 | + * in an effort to make forward progress. | |
| 1924 | + */ | |
| 1925 | +retry: | |
| 1926 | + ret = hugetlb_vmemmap_restore_folios(h, folio_list, &non_hvo_folios); | |
| 1927 | + if (ret < 0) { | |
| 1928 | + bulk_vmemmap_restore_error(h, folio_list, &non_hvo_folios); | |
| 1929 | + goto retry; | |
| 1930 | + } | |
| 1931 | + | |
| 1932 | + /* | |
| 1933 | + * At this point, list should be empty, ret should be >= 0 and there | |
| 1934 | + * should only be pages on the non_hvo_folios list. | |
| 1935 | + * Do note that the non_hvo_folios list could be empty. | |
| 1936 | + * Without HVO enabled, ret will be 0 and there is no need to call | |
| 1937 | + * __clear_hugetlb_destructor as this was done previously. | |
| 1938 | + */ | |
| 1939 | + VM_WARN_ON(!list_empty(folio_list)); | |
| 1940 | + VM_WARN_ON(ret < 0); | |
| 1941 | + if (!list_empty(&non_hvo_folios) && ret) { | |
| 1942 | + spin_lock_irq(&hugetlb_lock); | |
| 1943 | + list_for_each_entry(folio, &non_hvo_folios, lru) | |
| 1944 | + __clear_hugetlb_destructor(h, folio); | |
| 1945 | + spin_unlock_irq(&hugetlb_lock); | |
| 1946 | + } | |
| 1868 | 1947 | |
| 1869 | − list_for_each_entry_safe(page, t_page, list, lru) { | |
| 1870 | − folio = page_folio(page); | |
| 1948 | + list_for_each_entry_safe(folio, t_folio, &non_hvo_folios, lru) { | |
| 1871 | 1949 | update_and_free_hugetlb_folio(h, folio, false); |
| 1872 | 1950 | cond_resched(); |
| 1873 | 1951 | } |
| @@ -1931,6 +2009,7 @@ void free_huge_folio(struct folio *folio) · 57 unmodified lines | ||
| 1931 | 2009 | pages_per_huge_page(h), folio); |
| 1932 | 2010 | hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h), |
| 1933 | 2011 | pages_per_huge_page(h), folio); |
| 2012 | + mem_cgroup_uncharge(folio); | |
| 1934 | 2013 | if (restore_reserve) |
| 1935 | 2014 | h->resv_huge_pages++; |
| 1936 | 2015 | |
| @@ -1960,16 +2039,21 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) · 23 unmodified lines | ||
| 1960 | 2039 | h->nr_huge_pages_node[nid]++; |
| 1961 | 2040 | } |
| 1962 | 2041 | |
| 1963 | −static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) | |
| 2042 | +static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio) | |
| 1964 | 2043 | { |
| 1965 | − hugetlb_vmemmap_optimize(h, &folio->page); | |
| 1966 | − INIT_LIST_HEAD(&folio->lru); | |
| 1967 | 2044 | folio_set_hugetlb(folio); |
| 2045 | + INIT_LIST_HEAD(&folio->lru); | |
| 1968 | 2046 | hugetlb_set_folio_subpool(folio, NULL); |
| 1969 | 2047 | set_hugetlb_cgroup(folio, NULL); |
| 1970 | 2048 | set_hugetlb_cgroup_rsvd(folio, NULL); |
| 1971 | 2049 | } |
| 1972 | 2050 | |
| 2051 | +static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) | |
| 2052 | +{ | |
| 2053 | + init_new_hugetlb_folio(h, folio); | |
| 2054 | + hugetlb_vmemmap_optimize_folio(h, folio); | |
| 2055 | +} | |
| 2056 | + | |
| 1973 | 2057 | static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid) |
| 1974 | 2058 | { |
| 1975 | 2059 | __prep_new_hugetlb_folio(h, folio); |
| @@ -2103,20 +2187,6 @@ struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage) · 127 unmodified lines | ||
| 2103 | 2187 | return NULL; |
| 2104 | 2188 | } |
| 2105 | 2189 | |
| 2106 | −pgoff_t hugetlb_basepage_index(struct page *page) | |
| 2107 | −{ | |
| 2108 | − struct page *page_head = compound_head(page); | |
| 2109 | − pgoff_t index = page_index(page_head); | |
| 2110 | − unsigned long compound_idx; | |
| 2111 | − | |
| 2112 | − if (compound_order(page_head) > MAX_ORDER) | |
| 2113 | − compound_idx = page_to_pfn(page) - page_to_pfn(page_head); | |
| 2114 | − else | |
| 2115 | − compound_idx = page - page_head; | |
| 2116 | − | |
| 2117 | − return (index << compound_order(page_head)) + compound_idx; | |
| 2118 | −} | |
| 2119 | − | |
| 2120 | 2190 | static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, |
| 2121 | 2191 | gfp_t gfp_mask, int nid, nodemask_t *nmask, |
| 2122 | 2192 | nodemask_t *node_alloc_noretry) |
| @@ -2180,16 +2250,9 @@ retry: · 57 unmodified lines | ||
| 2180 | 2250 | return page_folio(page); |
| 2181 | 2251 | } |
| 2182 | 2252 | |
| 2183 | −/* | |
| 2184 | − * Common helper to allocate a fresh hugetlb page. All specific allocators | |
| 2185 | − * should use this function to get new hugetlb pages | |
| 2186 | − * | |
| 2187 | − * Note that returned page is 'frozen': ref count of head page and all tail | |
| 2188 | − * pages is zero. | |
| 2189 | − */ | |
| 2190 | −static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, | |
| 2191 | − gfp_t gfp_mask, int nid, nodemask_t *nmask, | |
| 2192 | − nodemask_t *node_alloc_noretry) | |
| 2253 | +static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, | |
| 2254 | + gfp_t gfp_mask, int nid, nodemask_t *nmask, | |
| 2255 | + nodemask_t *node_alloc_noretry) | |
| 2193 | 2256 | { |
| 2194 | 2257 | struct folio *folio; |
| 2195 | 2258 | bool retry = false; |
| @@ -2202,6 +2265,7 @@ retry: · 6 unmodified lines | ||
| 2202 | 2265 | nid, nmask, node_alloc_noretry); |
| 2203 | 2266 | if (!folio) |
| 2204 | 2267 | return NULL; |
| 2268 | + | |
| 2205 | 2269 | if (hstate_is_gigantic(h)) { |
| 2206 | 2270 | if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { |
| 2207 | 2271 | /* |
| @@ -2216,32 +2280,84 @@ retry: · 8 unmodified lines | ||
| 2216 | 2280 | return NULL; |
| 2217 | 2281 | } |
| 2218 | 2282 | } |
| 2219 | − prep_new_hugetlb_folio(h, folio, folio_nid(folio)); | |
| 2220 | 2283 | |
| 2221 | 2284 | return folio; |
| 2222 | 2285 | } |
| 2223 | 2286 | |
| 2287 | +static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, | |
| 2288 | + gfp_t gfp_mask, int nid, nodemask_t *nmask, | |
| 2289 | + nodemask_t *node_alloc_noretry) | |
| 2290 | +{ | |
| 2291 | + struct folio *folio; | |
| 2292 | + | |
| 2293 | + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, | |
| 2294 | + node_alloc_noretry); | |
| 2295 | + if (folio) | |
| 2296 | + init_new_hugetlb_folio(h, folio); | |
| 2297 | + return folio; | |
| 2298 | +} | |
| 2299 | + | |
| 2224 | 2300 | /* |
| 2225 | − * Allocates a fresh page to the hugetlb allocator pool in the node interleaved | |
| 2226 | − * manner. | |
| 2301 | + * Common helper to allocate a fresh hugetlb page. All specific allocators | |
| 2302 | + * should use this function to get new hugetlb pages | |
| 2303 | + * | |
| 2304 | + * Note that returned page is 'frozen': ref count of head page and all tail | |
| 2305 | + * pages is zero. | |
| 2227 | 2306 | */ |
| 2228 | −static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, | |
| 2229 | − nodemask_t *node_alloc_noretry) | |
| 2307 | +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, | |
| 2308 | + gfp_t gfp_mask, int nid, nodemask_t *nmask, | |
| 2309 | + nodemask_t *node_alloc_noretry) | |
| 2230 | 2310 | { |
| 2231 | 2311 | struct folio *folio; |
| 2232 | − int nr_nodes, node; | |
| 2312 | + | |
| 2313 | + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, | |
| 2314 | + node_alloc_noretry); | |
| 2315 | + if (!folio) | |
| 2316 | + return NULL; | |
| 2317 | + | |
| 2318 | + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); | |
| 2319 | + return folio; | |
| 2320 | +} | |
| 2321 | + | |
| 2322 | +static void prep_and_add_allocated_folios(struct hstate *h, | |
| 2323 | + struct list_head *folio_list) | |
| 2324 | +{ | |
| 2325 | + unsigned long flags; | |
| 2326 | + struct folio *folio, *tmp_f; | |
| 2327 | + | |
| 2328 | + /* Send list for bulk vmemmap optimization processing */ | |
| 2329 | + hugetlb_vmemmap_optimize_folios(h, folio_list); | |
| 2330 | + | |
| 2331 | + /* Add all new pool pages to free lists in one lock cycle */ | |
| 2332 | + spin_lock_irqsave(&hugetlb_lock, flags); | |
| 2333 | + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { | |
| 2334 | + __prep_account_new_huge_page(h, folio_nid(folio)); | |
| 2335 | + enqueue_hugetlb_folio(h, folio); | |
| 2336 | + } | |
| 2337 | + spin_unlock_irqrestore(&hugetlb_lock, flags); | |
| 2338 | +} | |
| 2339 | + | |
| 2340 | +/* | |
| 2341 | + * Allocates a fresh hugetlb page in a node interleaved manner. The page | |
| 2342 | + * will later be added to the appropriate hugetlb pool. | |
| 2343 | + */ | |
| 2344 | +static struct folio *alloc_pool_huge_folio(struct hstate *h, | |
| 2345 | + nodemask_t *nodes_allowed, | |
| 2346 | + nodemask_t *node_alloc_noretry) | |
| 2347 | +{ | |
| 2233 | 2348 | gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; |
| 2349 | + int nr_nodes, node; | |
| 2234 | 2350 | |
| 2235 | 2351 | for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { |
| 2236 | − folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node, | |
| 2352 | + struct folio *folio; | |
| 2353 | + | |
| 2354 | + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, | |
| 2237 | 2355 | nodes_allowed, node_alloc_noretry); |
| 2238 | − if (folio) { | |
| 2239 | − free_huge_folio(folio); /* free it into the hugepage allocator */ | |
| 2240 | − return 1; | |
| 2241 | − } | |
| 2356 | + if (folio) | |
| 2357 | + return folio; | |
| 2242 | 2358 | } |
| 2243 | 2359 | |
| 2244 | − return 0; | |
| 2360 | + return NULL; | |
| 2245 | 2361 | } |
| 2246 | 2362 | |
| 2247 | 2363 | /* |
| @@ -2251,13 +2367,11 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, · 3 unmodified lines | ||
| 2251 | 2367 | * an additional call to free the page to low level allocators. |
| 2252 | 2368 | * Called with hugetlb_lock locked. |
| 2253 | 2369 | */ |
| 2254 | −static struct page *remove_pool_huge_page(struct hstate *h, | |
| 2255 | − nodemask_t *nodes_allowed, | |
| 2256 | − bool acct_surplus) | |
| 2370 | +static struct folio *remove_pool_hugetlb_folio(struct hstate *h, | |
| 2371 | + nodemask_t *nodes_allowed, bool acct_surplus) | |
| 2257 | 2372 | { |
| 2258 | 2373 | int nr_nodes, node; |
| 2259 | − struct page *page = NULL; | |
| 2260 | − struct folio *folio; | |
| 2374 | + struct folio *folio = NULL; | |
| 2261 | 2375 | |
| 2262 | 2376 | lockdep_assert_held(&hugetlb_lock); |
| 2263 | 2377 | for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { |
| @@ -2267,15 +2381,14 @@ static struct page *remove_pool_huge_page(struct hstate *h, · 3 unmodified lines | ||
| 2267 | 2381 | */ |
| 2268 | 2382 | if ((!acct_surplus || h->surplus_huge_pages_node[node]) && |
| 2269 | 2383 | !list_empty(&h->hugepage_freelists[node])) { |
| 2270 | − page = list_entry(h->hugepage_freelists[node].next, | |
| 2271 | − struct page, lru); | |
| 2272 | − folio = page_folio(page); | |
| 2384 | + folio = list_entry(h->hugepage_freelists[node].next, | |
| 2385 | + struct folio, lru); | |
| 2273 | 2386 | remove_hugetlb_folio(h, folio, acct_surplus); |
| 2274 | 2387 | break; |
| 2275 | 2388 | } |
| 2276 | 2389 | } |
| 2277 | 2390 | |
| 2278 | − return page; | |
| 2391 | + return folio; | |
| 2279 | 2392 | } |
| 2280 | 2393 | |
| 2281 | 2394 | /* |
| @@ -2343,17 +2456,23 @@ retry: · 61 unmodified lines | ||
| 2343 | 2456 | * need to adjust max_huge_pages if the page is not freed. |
| 2344 | 2457 | * Attempt to allocate vmemmmap here so that we can take |
| 2345 | 2458 | * appropriate action on failure. |
| 2459 | + * | |
| 2460 | + * The folio_test_hugetlb check here is because | |
| 2461 | + * remove_hugetlb_folio will clear hugetlb folio flag for | |
| 2462 | + * non-vmemmap optimized hugetlb folios. | |
| 2346 | 2463 | */ |
| 2347 | − rc = hugetlb_vmemmap_restore(h, &folio->page); | |
| 2348 | − if (!rc) { | |
| 2349 | − update_and_free_hugetlb_folio(h, folio, false); | |
| 2350 | − } else { | |
| 2351 | − spin_lock_irq(&hugetlb_lock); | |
| 2352 | − add_hugetlb_folio(h, folio, false); | |
| 2353 | − h->max_huge_pages++; | |
| 2354 | − spin_unlock_irq(&hugetlb_lock); | |
| 2355 | − } | |
| 2464 | + if (folio_test_hugetlb(folio)) { | |
| 2465 | + rc = hugetlb_vmemmap_restore_folio(h, folio); | |
| 2466 | + if (rc) { | |
| 2467 | + spin_lock_irq(&hugetlb_lock); | |
| 2468 | + add_hugetlb_folio(h, folio, false); | |
| 2469 | + h->max_huge_pages++; | |
| 2470 | + goto out; | |
| 2471 | + } | |
| 2472 | + } else | |
| 2473 | + rc = 0; | |
| 2356 | 2474 | |
| 2475 | + update_and_free_hugetlb_folio(h, folio, false); | |
| 2357 | 2476 | return rc; |
| 2358 | 2477 | } |
| 2359 | 2478 | out: |
| @@ -2511,24 +2630,6 @@ struct folio *alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, · 151 unmodified lines | ||
| 2511 | 2630 | return alloc_migrate_hugetlb_folio(h, gfp_mask, preferred_nid, nmask); |
| 2512 | 2631 | } |
| 2513 | 2632 | |
| 2514 | −/* mempolicy aware migration callback */ | |
| 2515 | −struct folio *alloc_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma, | |
| 2516 | − unsigned long address) | |
| 2517 | −{ | |
| 2518 | − struct mempolicy *mpol; | |
| 2519 | − nodemask_t *nodemask; | |
| 2520 | − struct folio *folio; | |
| 2521 | − gfp_t gfp_mask; | |
| 2522 | − int node; | |
| 2523 | − | |
| 2524 | − gfp_mask = htlb_alloc_mask(h); | |
| 2525 | − node = huge_node(vma, address, gfp_mask, &mpol, &nodemask); | |
| 2526 | − folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask); | |
| 2527 | − mpol_cond_put(mpol); | |
| 2528 | − | |
| 2529 | − return folio; | |
| 2530 | −} | |
| 2531 | − | |
| 2532 | 2633 | /* |
| 2533 | 2634 | * Increase the hugetlb pool such that it can accommodate a reservation |
| 2534 | 2635 | * of size 'delta'. |
| @@ -2629,7 +2730,6 @@ static void return_unused_surplus_pages(struct hstate *h, · 94 unmodified lines | ||
| 2629 | 2730 | unsigned long unused_resv_pages) |
| 2630 | 2731 | { |
| 2631 | 2732 | unsigned long nr_pages; |
| 2632 | − struct page *page; | |
| 2633 | 2733 | LIST_HEAD(page_list); |
| 2634 | 2734 | |
| 2635 | 2735 | lockdep_assert_held(&hugetlb_lock); |
| @@ -2650,15 +2750,17 @@ static void return_unused_surplus_pages(struct hstate *h, · 14 unmodified lines | ||
| 2650 | 2750 | * evenly across all nodes with memory. Iterate across these nodes |
| 2651 | 2751 | * until we can no longer free unreserved surplus pages. This occurs |
| 2652 | 2752 | * when the nodes with surplus pages have no free pages. |
| 2653 | − * remove_pool_huge_page() will balance the freed pages across the | |
| 2753 | + * remove_pool_hugetlb_folio() will balance the freed pages across the | |
| 2654 | 2754 | * on-line nodes with memory and will handle the hstate accounting. |
| 2655 | 2755 | */ |
| 2656 | 2756 | while (nr_pages--) { |
| 2657 | − page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1); | |
| 2658 | − if (!page) | |
| 2757 | + struct folio *folio; | |
| 2758 | + | |
| 2759 | + folio = remove_pool_hugetlb_folio(h, &node_states[N_MEMORY], 1); | |
| 2760 | + if (!folio) | |
| 2659 | 2761 | goto out; |
| 2660 | 2762 | |
| 2661 | − list_add(&page->lru, &page_list); | |
| 2763 | + list_add(&folio->lru, &page_list); | |
| 2662 | 2764 | } |
| 2663 | 2765 | |
| 2664 | 2766 | out: |
| @@ -3040,11 +3142,20 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 375 unmodified lines | ||
| 3040 | 3142 | struct hugepage_subpool *spool = subpool_vma(vma); |
| 3041 | 3143 | struct hstate *h = hstate_vma(vma); |
| 3042 | 3144 | struct folio *folio; |
| 3043 | − long map_chg, map_commit; | |
| 3145 | + long map_chg, map_commit, nr_pages = pages_per_huge_page(h); | |
| 3044 | 3146 | long gbl_chg; |
| 3045 | − int ret, idx; | |
| 3147 | + int memcg_charge_ret, ret, idx; | |
| 3046 | 3148 | struct hugetlb_cgroup *h_cg = NULL; |
| 3149 | + struct mem_cgroup *memcg; | |
| 3047 | 3150 | bool deferred_reserve; |
| 3151 | + gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL; | |
| 3152 | + | |
| 3153 | + memcg = get_mem_cgroup_from_current(); | |
| 3154 | + memcg_charge_ret = mem_cgroup_hugetlb_try_charge(memcg, gfp, nr_pages); | |
| 3155 | + if (memcg_charge_ret == -ENOMEM) { | |
| 3156 | + mem_cgroup_put(memcg); | |
| 3157 | + return ERR_PTR(-ENOMEM); | |
| 3158 | + } | |
| 3048 | 3159 | |
| 3049 | 3160 | idx = hstate_index(h); |
| 3050 | 3161 | /* |
| @@ -3053,8 +3164,12 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 2 unmodified lines | ||
| 3053 | 3164 | * code of zero indicates a reservation exists (no change). |
| 3054 | 3165 | */ |
| 3055 | 3166 | map_chg = gbl_chg = vma_needs_reservation(h, vma, addr); |
| 3056 | − if (map_chg < 0) | |
| 3167 | + if (map_chg < 0) { | |
| 3168 | + if (!memcg_charge_ret) | |
| 3169 | + mem_cgroup_cancel_charge(memcg, nr_pages); | |
| 3170 | + mem_cgroup_put(memcg); | |
| 3057 | 3171 | return ERR_PTR(-ENOMEM); |
| 3172 | + } | |
| 3058 | 3173 | |
| 3059 | 3174 | /* |
| 3060 | 3175 | * Processes that did not create the mapping will have no |
| @@ -3065,10 +3180,8 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 4 unmodified lines | ||
| 3065 | 3180 | */ |
| 3066 | 3181 | if (map_chg || avoid_reserve) { |
| 3067 | 3182 | gbl_chg = hugepage_subpool_get_pages(spool, 1); |
| 3068 | − if (gbl_chg < 0) { | |
| 3069 | − vma_end_reservation(h, vma, addr); | |
| 3070 | − return ERR_PTR(-ENOSPC); | |
| 3071 | − } | |
| 3183 | + if (gbl_chg < 0) | |
| 3184 | + goto out_end_reservation; | |
| 3072 | 3185 | |
| 3073 | 3186 | /* |
| 3074 | 3187 | * Even though there was no reservation in the region/reserve |
| @@ -3150,6 +3263,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, · 75 unmodified lines | ||
| 3150 | 3263 | hugetlb_cgroup_uncharge_folio_rsvd(hstate_index(h), |
| 3151 | 3264 | pages_per_huge_page(h), folio); |
| 3152 | 3265 | } |
| 3266 | + | |
| 3267 | + if (!memcg_charge_ret) | |
| 3268 | + mem_cgroup_commit_charge(folio, memcg); | |
| 3269 | + mem_cgroup_put(memcg); | |
| 3270 | + | |
| 3153 | 3271 | return folio; |
| 3154 | 3272 | |
| 3155 | 3273 | out_uncharge_cgroup: |
| @@ -3161,7 +3279,11 @@ out_uncharge_cgroup_reservation: · 5 unmodified lines | ||
| 3161 | 3279 | out_subpool_put: |
| 3162 | 3280 | if (map_chg || avoid_reserve) |
| 3163 | 3281 | hugepage_subpool_put_pages(spool, 1); |
| 3282 | +out_end_reservation: | |
| 3164 | 3283 | vma_end_reservation(h, vma, addr); |
| 3284 | + if (!memcg_charge_ret) | |
| 3285 | + mem_cgroup_cancel_charge(memcg, nr_pages); | |
| 3286 | + mem_cgroup_put(memcg); | |
| 3165 | 3287 | return ERR_PTR(-ENOSPC); |
| 3166 | 3288 | } |
| 3167 | 3289 | |
| @@ -3196,6 +3318,16 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) · 28 unmodified lines | ||
| 3196 | 3318 | } |
| 3197 | 3319 | |
| 3198 | 3320 | found: |
| 3321 | + | |
| 3322 | + /* | |
| 3323 | + * Only initialize the head struct page in memmap_init_reserved_pages, | |
| 3324 | + * rest of the struct pages will be initialized by the HugeTLB | |
| 3325 | + * subsystem itself. | |
| 3326 | + * The head struct page is used to get folio information by the HugeTLB | |
| 3327 | + * subsystem like zone id and node id. | |
| 3328 | + */ | |
| 3329 | + memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE), | |
| 3330 | + huge_page_size(h) - PAGE_SIZE); | |
| 3199 | 3331 | /* Put them into a private list first because mem_map is not up yet */ |
| 3200 | 3332 | INIT_LIST_HEAD(&m->list); |
| 3201 | 3333 | list_add(&m->list, &huge_boot_pages); |
| @@ -3203,29 +3335,102 @@ found: · 1 unmodified line | ||
| 3203 | 3335 | return 1; |
| 3204 | 3336 | } |
| 3205 | 3337 | |
| 3338 | +/* Initialize [start_page:end_page_number] tail struct pages of a hugepage */ | |
| 3339 | +static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio, | |
| 3340 | + unsigned long start_page_number, | |
| 3341 | + unsigned long end_page_number) | |
| 3342 | +{ | |
| 3343 | + enum zone_type zone = zone_idx(folio_zone(folio)); | |
| 3344 | + int nid = folio_nid(folio); | |
| 3345 | + unsigned long head_pfn = folio_pfn(folio); | |
| 3346 | + unsigned long pfn, end_pfn = head_pfn + end_page_number; | |
| 3347 | + int ret; | |
| 3348 | + | |
| 3349 | + for (pfn = head_pfn + start_page_number; pfn < end_pfn; pfn++) { | |
| 3350 | + struct page *page = pfn_to_page(pfn); | |
| 3351 | + | |
| 3352 | + __init_single_page(page, pfn, zone, nid); | |
| 3353 | + prep_compound_tail((struct page *)folio, pfn - head_pfn); | |
| 3354 | + ret = page_ref_freeze(page, 1); | |
| 3355 | + VM_BUG_ON(!ret); | |
| 3356 | + } | |
| 3357 | +} | |
| 3358 | + | |
| 3359 | +static void __init hugetlb_folio_init_vmemmap(struct folio *folio, | |
| 3360 | + struct hstate *h, | |
| 3361 | + unsigned long nr_pages) | |
| 3362 | +{ | |
| 3363 | + int ret; | |
| 3364 | + | |
| 3365 | + /* Prepare folio head */ | |
| 3366 | + __folio_clear_reserved(folio); | |
| 3367 | + __folio_set_head(folio); | |
| 3368 | + ret = folio_ref_freeze(folio, 1); | |
| 3369 | + VM_BUG_ON(!ret); | |
| 3370 | + /* Initialize the necessary tail struct pages */ | |
| 3371 | + hugetlb_folio_init_tail_vmemmap(folio, 1, nr_pages); | |
| 3372 | + prep_compound_head((struct page *)folio, huge_page_order(h)); | |
| 3373 | +} | |
| 3374 | + | |
| 3375 | +static void __init prep_and_add_bootmem_folios(struct hstate *h, | |
| 3376 | + struct list_head *folio_list) | |
| 3377 | +{ | |
| 3378 | + unsigned long flags; | |
| 3379 | + struct folio *folio, *tmp_f; | |
| 3380 | + | |
| 3381 | + /* Send list for bulk vmemmap optimization processing */ | |
| 3382 | + hugetlb_vmemmap_optimize_folios(h, folio_list); | |
| 3383 | + | |
| 3384 | + /* Add all new pool pages to free lists in one lock cycle */ | |
| 3385 | + spin_lock_irqsave(&hugetlb_lock, flags); | |
| 3386 | + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { | |
| 3387 | + if (!folio_test_hugetlb_vmemmap_optimized(folio)) { | |
| 3388 | + /* | |
| 3389 | + * If HVO fails, initialize all tail struct pages | |
| 3390 | + * We do not worry about potential long lock hold | |
| 3391 | + * time as this is early in boot and there should | |
| 3392 | + * be no contention. | |
| 3393 | + */ | |
| 3394 | + hugetlb_folio_init_tail_vmemmap(folio, | |
| 3395 | + HUGETLB_VMEMMAP_RESERVE_PAGES, | |
| 3396 | + pages_per_huge_page(h)); | |
| 3397 | + } | |
| 3398 | + __prep_account_new_huge_page(h, folio_nid(folio)); | |
| 3399 | + enqueue_hugetlb_folio(h, folio); | |
| 3400 | + } | |
| 3401 | + spin_unlock_irqrestore(&hugetlb_lock, flags); | |
| 3402 | +} | |
| 3403 | + | |
| 3206 | 3404 | /* |
| 3207 | 3405 | * Put bootmem huge pages into the standard lists after mem_map is up. |
| 3208 | 3406 | * Note: This only applies to gigantic (order > MAX_ORDER) pages. |
| 3209 | 3407 | */ |
| 3210 | 3408 | static void __init gather_bootmem_prealloc(void) |
| 3211 | 3409 | { |
| 3410 | + LIST_HEAD(folio_list); | |
| 3212 | 3411 | struct huge_bootmem_page *m; |
| 3412 | + struct hstate *h = NULL, *prev_h = NULL; | |
| 3213 | 3413 | |
| 3214 | 3414 | list_for_each_entry(m, &huge_boot_pages, list) { |
| 3215 | 3415 | struct page *page = virt_to_page(m); |
| 3216 | − struct folio *folio = page_folio(page); | |
| 3217 | − struct hstate *h = m->hstate; | |
| 3416 | + struct folio *folio = (void *)page; | |
| 3417 | + | |
| 3418 | + h = m->hstate; | |
| 3419 | + /* | |
| 3420 | + * It is possible to have multiple huge page sizes (hstates) | |
| 3421 | + * in this list. If so, process each size separately. | |
| 3422 | + */ | |
| 3423 | + if (h != prev_h && prev_h != NULL) | |
| 3424 | + prep_and_add_bootmem_folios(prev_h, &folio_list); | |
| 3425 | + prev_h = h; | |
| 3218 | 3426 | |
| 3219 | 3427 | VM_BUG_ON(!hstate_is_gigantic(h)); |
| 3220 | 3428 | WARN_ON(folio_ref_count(folio) != 1); |
| 3221 | − if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { | |
| 3222 | − WARN_ON(folio_test_reserved(folio)); | |
| 3223 | − prep_new_hugetlb_folio(h, folio, folio_nid(folio)); | |
| 3224 | − free_huge_folio(folio); /* add to the hugepage allocator */ | |
| 3225 | − } else { | |
| 3226 | − /* VERY unlikely inflated ref count on a tail page */ | |
| 3227 | − free_gigantic_folio(folio, huge_page_order(h)); | |
| 3228 | − } | |
| 3429 | + | |
| 3430 | + hugetlb_folio_init_vmemmap(folio, h, | |
| 3431 | + HUGETLB_VMEMMAP_RESERVE_PAGES); | |
| 3432 | + init_new_hugetlb_folio(h, folio); | |
| 3433 | + list_add(&folio->lru, &folio_list); | |
| 3229 | 3434 | |
| 3230 | 3435 | /* |
| 3231 | 3436 | * We need to restore the 'stolen' pages to totalram_pages |
| @@ -3235,7 +3440,10 @@ static void __init gather_bootmem_prealloc(void) · 3 unmodified lines | ||
| 3235 | 3440 | adjust_managed_page_count(page, pages_per_huge_page(h)); |
| 3236 | 3441 | cond_resched(); |
| 3237 | 3442 | } |
| 3443 | + | |
| 3444 | + prep_and_add_bootmem_folios(h, &folio_list); | |
| 3238 | 3445 | } |
| 3446 | + | |
| 3239 | 3447 | static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) |
| 3240 | 3448 | { |
| 3241 | 3449 | unsigned long i; |
| @@ -3267,9 +3475,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) · 25 unmodified lines | ||
| 3267 | 3475 | h->max_huge_pages_node[nid] = i; |
| 3268 | 3476 | } |
| 3269 | 3477 | |
| 3478 | +/* | |
| 3479 | + * NOTE: this routine is called in different contexts for gigantic and | |
| 3480 | + * non-gigantic pages. | |
| 3481 | + * - For gigantic pages, this is called early in the boot process and | |
| 3482 | + * pages are allocated from memblock allocated or something similar. | |
| 3483 | + * Gigantic pages are actually added to pools later with the routine | |
| 3484 | + * gather_bootmem_prealloc. | |
| 3485 | + * - For non-gigantic pages, this is called later in the boot process after | |
| 3486 | + * all of mm is up and functional. Pages are allocated from buddy and | |
| 3487 | + * then added to hugetlb pools. | |
| 3488 | + */ | |
| 3270 | 3489 | static void __init hugetlb_hstate_alloc_pages(struct hstate *h) |
| 3271 | 3490 | { |
| 3272 | 3491 | unsigned long i; |
| 3492 | + struct folio *folio; | |
| 3493 | + LIST_HEAD(folio_list); | |
| 3273 | 3494 | nodemask_t *node_alloc_noretry; |
| 3274 | 3495 | bool node_specific_alloc = false; |
| 3275 | 3496 | |
| @@ -3311,14 +3532,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) · 35 unmodified lines | ||
| 3311 | 3532 | |
| 3312 | 3533 | for (i = 0; i < h->max_huge_pages; ++i) { |
| 3313 | 3534 | if (hstate_is_gigantic(h)) { |
| 3535 | + /* | |
| 3536 | + * gigantic pages not added to list as they are not | |
| 3537 | + * added to pools now. | |
| 3538 | + */ | |
| 3314 | 3539 | if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) |
| 3315 | 3540 | break; |
| 3316 | − } else if (!alloc_pool_huge_page(h, | |
| 3317 | − &node_states[N_MEMORY], | |
| 3318 | − node_alloc_noretry)) | |
| 3319 | − break; | |
| 3541 | + } else { | |
| 3542 | + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], | |
| 3543 | + node_alloc_noretry); | |
| 3544 | + if (!folio) | |
| 3545 | + break; | |
| 3546 | + list_add(&folio->lru, &folio_list); | |
| 3547 | + } | |
| 3320 | 3548 | cond_resched(); |
| 3321 | 3549 | } |
| 3550 | + | |
| 3551 | + /* list will be empty if hstate_is_gigantic */ | |
| 3552 | + prep_and_add_allocated_folios(h, &folio_list); | |
| 3553 | + | |
| 3322 | 3554 | if (i < h->max_huge_pages) { |
| 3323 | 3555 | char buf[32]; |
| 3324 | 3556 | |
| @@ -3391,15 +3623,15 @@ static void try_to_free_low(struct hstate *h, unsigned long count, · 66 unmodified lines | ||
| 3391 | 3623 | * Collect pages to be freed on a list, and free after dropping lock |
| 3392 | 3624 | */ |
| 3393 | 3625 | for_each_node_mask(i, *nodes_allowed) { |
| 3394 | − struct page *page, *next; | |
| 3626 | + struct folio *folio, *next; | |
| 3395 | 3627 | struct list_head *freel = &h->hugepage_freelists[i]; |
| 3396 | − list_for_each_entry_safe(page, next, freel, lru) { | |
| 3628 | + list_for_each_entry_safe(folio, next, freel, lru) { | |
| 3397 | 3629 | if (count >= h->nr_huge_pages) |
| 3398 | 3630 | goto out; |
| 3399 | − if (PageHighMem(page)) | |
| 3631 | + if (folio_test_highmem(folio)) | |
| 3400 | 3632 | continue; |
| 3401 | − remove_hugetlb_folio(h, page_folio(page), false); | |
| 3402 | − list_add(&page->lru, &page_list); | |
| 3633 | + remove_hugetlb_folio(h, folio, false); | |
| 3634 | + list_add(&folio->lru, &page_list); | |
| 3403 | 3635 | } |
| 3404 | 3636 | } |
| 3405 | 3637 | |
| @@ -3452,8 +3684,9 @@ found: · 46 unmodified lines | ||
| 3452 | 3684 | static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, |
| 3453 | 3685 | nodemask_t *nodes_allowed) |
| 3454 | 3686 | { |
| 3455 | − unsigned long min_count, ret; | |
| 3456 | − struct page *page; | |
| 3687 | + unsigned long min_count; | |
| 3688 | + unsigned long allocated; | |
| 3689 | + struct folio *folio; | |
| 3457 | 3690 | LIST_HEAD(page_list); |
| 3458 | 3691 | NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); |
| 3459 | 3692 | |
| @@ -3484,7 +3717,9 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 24 unmodified lines | ||
| 3484 | 3717 | if (nid != NUMA_NO_NODE) { |
| 3485 | 3718 | unsigned long old_count = count; |
| 3486 | 3719 | |
| 3487 | − count += h->nr_huge_pages - h->nr_huge_pages_node[nid]; | |
| 3720 | + count += persistent_huge_pages(h) - | |
| 3721 | + (h->nr_huge_pages_node[nid] - | |
| 3722 | + h->surplus_huge_pages_node[nid]); | |
| 3488 | 3723 | /* |
| 3489 | 3724 | * User may have specified a large count value which caused the |
| 3490 | 3725 | * above calculation to overflow. In this case, they wanted |
| @@ -3528,7 +3763,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 37 unmodified lines | ||
| 3528 | 3763 | break; |
| 3529 | 3764 | } |
| 3530 | 3765 | |
| 3531 | − while (count > persistent_huge_pages(h)) { | |
| 3766 | + allocated = 0; | |
| 3767 | + while (count > (persistent_huge_pages(h) + allocated)) { | |
| 3532 | 3768 | /* |
| 3533 | 3769 | * If this allocation races such that we no longer need the |
| 3534 | 3770 | * page, free_huge_folio will handle it by freeing the page |
| @@ -3539,15 +3775,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 4 unmodified lines | ||
| 3539 | 3775 | /* yield cpu to avoid soft lockup */ |
| 3540 | 3776 | cond_resched(); |
| 3541 | 3777 | |
| 3542 | − ret = alloc_pool_huge_page(h, nodes_allowed, | |
| 3778 | + folio = alloc_pool_huge_folio(h, nodes_allowed, | |
| 3543 | 3779 | node_alloc_noretry); |
| 3544 | − spin_lock_irq(&hugetlb_lock); | |
| 3545 | − if (!ret) | |
| 3780 | + if (!folio) { | |
| 3781 | + prep_and_add_allocated_folios(h, &page_list); | |
| 3782 | + spin_lock_irq(&hugetlb_lock); | |
| 3546 | 3783 | goto out; |
| 3784 | + } | |
| 3785 | + | |
| 3786 | + list_add(&folio->lru, &page_list); | |
| 3787 | + allocated++; | |
| 3547 | 3788 | |
| 3548 | 3789 | /* Bail for signals. Probably ctrl-c from user */ |
| 3549 | − if (signal_pending(current)) | |
| 3790 | + if (signal_pending(current)) { | |
| 3791 | + prep_and_add_allocated_folios(h, &page_list); | |
| 3792 | + spin_lock_irq(&hugetlb_lock); | |
| 3550 | 3793 | goto out; |
| 3794 | + } | |
| 3795 | + | |
| 3796 | + spin_lock_irq(&hugetlb_lock); | |
| 3797 | + } | |
| 3798 | + | |
| 3799 | + /* Add allocated pages to the pool */ | |
| 3800 | + if (!list_empty(&page_list)) { | |
| 3801 | + spin_unlock_irq(&hugetlb_lock); | |
| 3802 | + prep_and_add_allocated_folios(h, &page_list); | |
| 3803 | + spin_lock_irq(&hugetlb_lock); | |
| 3551 | 3804 | } |
| 3552 | 3805 | |
| 3553 | 3806 | /* |
| @@ -3573,11 +3826,11 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, · 19 unmodified lines | ||
| 3573 | 3826 | * Collect pages to be removed on list without dropping lock |
| 3574 | 3827 | */ |
| 3575 | 3828 | while (min_count < persistent_huge_pages(h)) { |
| 3576 | − page = remove_pool_huge_page(h, nodes_allowed, 0); | |
| 3577 | − if (!page) | |
| 3829 | + folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); | |
| 3830 | + if (!folio) | |
| 3578 | 3831 | break; |
| 3579 | 3832 | |
| 3580 | − list_add(&page->lru, &page_list); | |
| 3833 | + list_add(&folio->lru, &page_list); | |
| 3581 | 3834 | } |
| 3582 | 3835 | /* free the pages after dropping lock */ |
| 3583 | 3836 | spin_unlock_irq(&hugetlb_lock); |
| @@ -3612,13 +3865,21 @@ static int demote_free_hugetlb_folio(struct hstate *h, struct folio *folio) · 28 unmodified lines | ||
| 3612 | 3865 | remove_hugetlb_folio_for_demote(h, folio, false); |
| 3613 | 3866 | spin_unlock_irq(&hugetlb_lock); |
| 3614 | 3867 | |
| 3615 | − rc = hugetlb_vmemmap_restore(h, &folio->page); | |
| 3616 | − if (rc) { | |
| 3617 | − /* Allocation of vmemmmap failed, we can not demote folio */ | |
| 3618 | − spin_lock_irq(&hugetlb_lock); | |
| 3619 | − folio_ref_unfreeze(folio, 1); | |
| 3620 | − add_hugetlb_folio(h, folio, false); | |
| 3621 | − return rc; | |
| 3868 | + /* | |
| 3869 | + * If vmemmap already existed for folio, the remove routine above would | |
| 3870 | + * have cleared the hugetlb folio flag. Hence the folio is technically | |
| 3871 | + * no longer a hugetlb folio. hugetlb_vmemmap_restore_folio can only be | |
| 3872 | + * passed hugetlb folios and will BUG otherwise. | |
| 3873 | + */ | |
| 3874 | + if (folio_test_hugetlb(folio)) { | |
| 3875 | + rc = hugetlb_vmemmap_restore_folio(h, folio); | |
| 3876 | + if (rc) { | |
| 3877 | + /* Allocation of vmemmmap failed, we can not demote folio */ | |
| 3878 | + spin_lock_irq(&hugetlb_lock); | |
| 3879 | + folio_ref_unfreeze(folio, 1); | |
| 3880 | + add_hugetlb_folio(h, folio, false); | |
| 3881 | + return rc; | |
| 3882 | + } | |
| 3622 | 3883 | } |
| 3623 | 3884 | |
| 3624 | 3885 | /* |
| @@ -4314,7 +4575,7 @@ void __init hugetlb_add_hstate(unsigned int order) · 689 unmodified lines | ||
| 4314 | 4575 | return; |
| 4315 | 4576 | } |
| 4316 | 4577 | BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE); |
| 4317 | − BUG_ON(order == 0); | |
| 4578 | + BUG_ON(order < order_base_2(__NR_USED_SUBPAGE)); | |
| 4318 | 4579 | h = &hstates[hugetlb_max_hstate++]; |
| 4319 | 4580 | mutex_init(&h->resize_lock); |
| 4320 | 4581 | h->order = order; |
| @@ -4997,7 +5258,7 @@ bool is_hugetlb_entry_migration(pte_t pte) · 676 unmodified lines | ||
| 4997 | 5258 | return false; |
| 4998 | 5259 | } |
| 4999 | 5260 | |
| 5000 | −static bool is_hugetlb_entry_hwpoisoned(pte_t pte) | |
| 5261 | +bool is_hugetlb_entry_hwpoisoned(pte_t pte) | |
| 5001 | 5262 | { |
| 5002 | 5263 | swp_entry_t swp; |
| 5003 | 5264 | |
| @@ -5605,8 +5866,10 @@ retry_avoidcopy: · 601 unmodified lines | ||
| 5605 | 5866 | * owner and can reuse this page. |
| 5606 | 5867 | */ |
| 5607 | 5868 | if (folio_mapcount(old_folio) == 1 && folio_test_anon(old_folio)) { |
| 5608 | − if (!PageAnonExclusive(&old_folio->page)) | |
| 5609 | − page_move_anon_rmap(&old_folio->page, vma); | |
| 5869 | + if (!PageAnonExclusive(&old_folio->page)) { | |
| 5870 | + folio_move_anon_rmap(old_folio, vma); | |
| 5871 | + SetPageAnonExclusive(&old_folio->page); | |
| 5872 | + } | |
| 5610 | 5873 | if (likely(!unshare)) |
| 5611 | 5874 | set_huge_ptep_writable(vma, haddr, ptep); |
| 5612 | 5875 | |
| @@ -5752,7 +6015,7 @@ static bool hugetlbfs_pagecache_present(struct hstate *h, · 139 unmodified lines | ||
| 5752 | 6015 | struct vm_area_struct *vma, unsigned long address) |
| 5753 | 6016 | { |
| 5754 | 6017 | struct address_space *mapping = vma->vm_file->f_mapping; |
| 5755 | − pgoff_t idx = vma_hugecache_offset(h, vma, address); | |
| 6018 | + pgoff_t idx = linear_page_index(vma, address); | |
| 5756 | 6019 | struct folio *folio; |
| 5757 | 6020 | |
| 5758 | 6021 | folio = filemap_get_folio(mapping, idx); |
| @@ -5769,6 +6032,7 @@ int hugetlb_add_to_page_cache(struct folio *folio, struct address_space *mapping · 10 unmodified lines | ||
| 5769 | 6032 | struct hstate *h = hstate_inode(inode); |
| 5770 | 6033 | int err; |
| 5771 | 6034 | |
| 6035 | + idx <<= huge_page_order(h); | |
| 5772 | 6036 | __folio_set_locked(folio); |
| 5773 | 6037 | err = __filemap_add_folio(mapping, folio, idx, GFP_KERNEL, NULL); |
| 5774 | 6038 | |
| @@ -5876,7 +6140,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, · 101 unmodified lines | ||
| 5876 | 6140 | * before we get page_table_lock. |
| 5877 | 6141 | */ |
| 5878 | 6142 | new_folio = false; |
| 5879 | − folio = filemap_lock_folio(mapping, idx); | |
| 6143 | + folio = filemap_lock_hugetlb_folio(h, mapping, idx); | |
| 5880 | 6144 | if (IS_ERR(folio)) { |
| 5881 | 6145 | size = i_size_read(mapping->host) >> huge_page_shift(h); |
| 5882 | 6146 | if (idx >= size) |
| @@ -6185,7 +6449,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 302 unmodified lines | ||
| 6185 | 6449 | /* Just decrements count, does not deallocate */ |
| 6186 | 6450 | vma_end_reservation(h, vma, haddr); |
| 6187 | 6451 | |
| 6188 | − pagecache_folio = filemap_lock_folio(mapping, idx); | |
| 6452 | + pagecache_folio = filemap_lock_hugetlb_folio(h, mapping, idx); | |
| 6189 | 6453 | if (IS_ERR(pagecache_folio)) |
| 6190 | 6454 | pagecache_folio = NULL; |
| 6191 | 6455 | } |
| @@ -6199,21 +6463,28 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, · 7 unmodified lines | ||
| 6199 | 6463 | /* Handle userfault-wp first, before trying to lock more pages */ |
| 6200 | 6464 | if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && |
| 6201 | 6465 | (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { |
| 6202 | − struct vm_fault vmf = { | |
| 6203 | − .vma = vma, | |
| 6204 | − .address = haddr, | |
| 6205 | − .real_address = address, | |
| 6206 | − .flags = flags, | |
| 6207 | − }; | |
| 6466 | + if (!userfaultfd_wp_async(vma)) { | |
| 6467 | + struct vm_fault vmf = { | |
| 6468 | + .vma = vma, | |
| 6469 | + .address = haddr, | |
| 6470 | + .real_address = address, | |
| 6471 | + .flags = flags, | |
| 6472 | + }; | |
| 6208 | 6473 | |
| 6209 | − spin_unlock(ptl); | |
| 6210 | − if (pagecache_folio) { | |
| 6211 | − folio_unlock(pagecache_folio); | |
| 6212 | − folio_put(pagecache_folio); | |
| 6474 | + spin_unlock(ptl); | |
| 6475 | + if (pagecache_folio) { | |
| 6476 | + folio_unlock(pagecache_folio); | |
| 6477 | + folio_put(pagecache_folio); | |
| 6478 | + } | |
| 6479 | + hugetlb_vma_unlock_read(vma); | |
| 6480 | + mutex_unlock(&hugetlb_fault_mutex_table[hash]); | |
| 6481 | + return handle_userfault(&vmf, VM_UFFD_WP); | |
| 6213 | 6482 | } |
| 6214 | − hugetlb_vma_unlock_read(vma); | |
| 6215 | − mutex_unlock(&hugetlb_fault_mutex_table[hash]); | |
| 6216 | − return handle_userfault(&vmf, VM_UFFD_WP); | |
| 6483 | + | |
| 6484 | + entry = huge_pte_clear_uffd_wp(entry); | |
| 6485 | + set_huge_pte_at(mm, haddr, ptep, entry, | |
| 6486 | + huge_page_size(hstate_vma(vma))); | |
| 6487 | + /* Fallthrough to CoW */ | |
| 6217 | 6488 | } |
| 6218 | 6489 | |
| 6219 | 6490 | /* |
| @@ -6270,6 +6541,26 @@ out_mutex: · 50 unmodified lines | ||
| 6270 | 6541 | } |
| 6271 | 6542 | |
| 6272 | 6543 | #ifdef CONFIG_USERFAULTFD |
| 6544 | +/* | |
| 6545 | + * Can probably be eliminated, but still used by hugetlb_mfill_atomic_pte(). | |
| 6546 | + */ | |
| 6547 | +static struct folio *alloc_hugetlb_folio_vma(struct hstate *h, | |
| 6548 | + struct vm_area_struct *vma, unsigned long address) | |
| 6549 | +{ | |
| 6550 | + struct mempolicy *mpol; | |
| 6551 | + nodemask_t *nodemask; | |
| 6552 | + struct folio *folio; | |
| 6553 | + gfp_t gfp_mask; | |
| 6554 | + int node; | |
| 6555 | + | |
| 6556 | + gfp_mask = htlb_alloc_mask(h); | |
| 6557 | + node = huge_node(vma, address, gfp_mask, &mpol, &nodemask); | |
| 6558 | + folio = alloc_hugetlb_folio_nodemask(h, node, nodemask, gfp_mask); | |
| 6559 | + mpol_cond_put(mpol); | |
| 6560 | + | |
| 6561 | + return folio; | |
| 6562 | +} | |
| 6563 | + | |
| 6273 | 6564 | /* |
| 6274 | 6565 | * Used by userfaultfd UFFDIO_* ioctls. Based on userfaultfd's mfill_atomic_pte |
| 6275 | 6566 | * with modifications for hugetlb pages. |
| @@ -6318,7 +6609,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, · 42 unmodified lines | ||
| 6318 | 6609 | |
| 6319 | 6610 | if (is_continue) { |
| 6320 | 6611 | ret = -EFAULT; |
| 6321 | − folio = filemap_lock_folio(mapping, idx); | |
| 6612 | + folio = filemap_lock_hugetlb_folio(h, mapping, idx); | |
| 6322 | 6613 | if (IS_ERR(folio)) |
| 6323 | 6614 | goto out; |
| 6324 | 6615 | folio_in_pagecache = true; |
| @@ -6520,7 +6811,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, · 195 unmodified lines | ||
| 6520 | 6811 | } |
| 6521 | 6812 | } |
| 6522 | 6813 | |
| 6523 | − page += ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); | |
| 6814 | + page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); | |
| 6524 | 6815 | |
| 6525 | 6816 | /* |
| 6526 | 6817 | * Note that page may be a sub-page, and with vmemmap |
| @@ -262,12 +262,6 @@ static int __hugetlb_cgroup_charge_cgroup(int idx, unsigned long nr_pages, · 261 unmodified lines | ||
| 262 | 262 | |
| 263 | 263 | if (hugetlb_cgroup_disabled()) |
| 264 | 264 | goto done; |
| 265 | − /* | |
| 266 | − * We don't charge any cgroup if the compound page have less | |
| 267 | − * than 3 pages. | |
| 268 | − */ | |
| 269 | − if (huge_page_order(&hstates[idx]) < HUGETLB_CGROUP_MIN_ORDER) | |
| 270 | − goto done; | |
| 271 | 265 | again: |
| 272 | 266 | rcu_read_lock(); |
| 273 | 267 | h_cg = hugetlb_cgroup_from_task(current); |
| @@ -397,9 +391,6 @@ static void __hugetlb_cgroup_uncharge_cgroup(int idx, unsigned long nr_pages, · 123 unmodified lines | ||
| 397 | 391 | if (hugetlb_cgroup_disabled() || !h_cg) |
| 398 | 392 | return; |
| 399 | 393 | |
| 400 | − if (huge_page_order(&hstates[idx]) < HUGETLB_CGROUP_MIN_ORDER) | |
| 401 | − return; | |
| 402 | − | |
| 403 | 394 | page_counter_uncharge(__hugetlb_cgroup_counter_from_cgroup(h_cg, idx, |
| 404 | 395 | rsvd), |
| 405 | 396 | nr_pages); |
| @@ -869,15 +860,8 @@ void __init hugetlb_cgroup_file_init(void) · 463 unmodified lines | ||
| 869 | 860 | { |
| 870 | 861 | struct hstate *h; |
| 871 | 862 | |
| 872 | − for_each_hstate(h) { | |
| 873 | − /* | |
| 874 | − * Add cgroup control files only if the huge page consists | |
| 875 | − * of more than two normal pages. This is because we use | |
| 876 | − * page[2].private for storing cgroup details. | |
| 877 | − */ | |
| 878 | − if (huge_page_order(h) >= HUGETLB_CGROUP_MIN_ORDER) | |
| 879 | − __hugetlb_cgroup_file_init(hstate_index(h)); | |
| 880 | − } | |
| 863 | + for_each_hstate(h) | |
| 864 | + __hugetlb_cgroup_file_init(hstate_index(h)); | |
| 881 | 865 | } |
| 882 | 866 | |
| 883 | 867 | /* |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/pgtable.h> |
| 14 | 14 | #include <linux/moduleparam.h> |
| 15 | 15 | #include <linux/bootmem_info.h> |
| 16 | +#include <linux/mmdebug.h> | |
| 16 | 17 | #include <asm/pgalloc.h> |
| 17 | 18 | #include <asm/tlbflush.h> |
| 18 | 19 | #include "hugetlb_vmemmap.h" |
| @@ -26,6 +27,8 @@ · 7 unmodified lines | ||
| 26 | 27 | * @reuse_addr: the virtual address of the @reuse_page page. |
| 27 | 28 | * @vmemmap_pages: the list head of the vmemmap pages that can be freed |
| 28 | 29 | * or is mapped from. |
| 30 | + * @flags: used to modify behavior in vmemmap page table walking | |
| 31 | + * operations. | |
| 29 | 32 | */ |
| 30 | 33 | struct vmemmap_remap_walk { |
| 31 | 34 | void (*remap_pte)(pte_t *pte, unsigned long addr, |
| @@ -34,9 +37,15 @@ struct vmemmap_remap_walk { · 2 unmodified lines | ||
| 34 | 37 | struct page *reuse_page; |
| 35 | 38 | unsigned long reuse_addr; |
| 36 | 39 | struct list_head *vmemmap_pages; |
| 40 | + | |
| 41 | +/* Skip the TLB flush when we split the PMD */ | |
| 42 | +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) | |
| 43 | +/* Skip the TLB flush when we remap the PTE */ | |
| 44 | +#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) | |
| 45 | + unsigned long flags; | |
| 37 | 46 | }; |
| 38 | 47 | |
| 39 | −static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) | |
| 48 | +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) | |
| 40 | 49 | { |
| 41 | 50 | pmd_t __pmd; |
| 42 | 51 | int i; |
| @@ -79,7 +88,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) · 36 unmodified lines | ||
| 79 | 88 | /* Make pte visible before pmd. See comment in pmd_install(). */ |
| 80 | 89 | smp_wmb(); |
| 81 | 90 | pmd_populate_kernel(&init_mm, pmd, pgtable); |
| 82 | − flush_tlb_kernel_range(start, start + PMD_SIZE); | |
| 91 | + if (flush) | |
| 92 | + flush_tlb_kernel_range(start, start + PMD_SIZE); | |
| 83 | 93 | } else { |
| 84 | 94 | pte_free_kernel(&init_mm, pgtable); |
| 85 | 95 | } |
| @@ -126,11 +136,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, · 40 unmodified lines | ||
| 126 | 136 | do { |
| 127 | 137 | int ret; |
| 128 | 138 | |
| 129 | − ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); | |
| 139 | + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, | |
| 140 | + !(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)); | |
| 130 | 141 | if (ret) |
| 131 | 142 | return ret; |
| 132 | 143 | |
| 133 | 144 | next = pmd_addr_end(addr, end); |
| 145 | + | |
| 146 | + /* | |
| 147 | + * We are only splitting, not remapping the hugetlb vmemmap | |
| 148 | + * pages. | |
| 149 | + */ | |
| 150 | + if (!walk->remap_pte) | |
| 151 | + continue; | |
| 152 | + | |
| 134 | 153 | vmemmap_pte_range(pmd, addr, next, walk); |
| 135 | 154 | } while (pmd++, addr = next, addr != end); |
| 136 | 155 | |
| @@ -197,7 +216,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, · 60 unmodified lines | ||
| 197 | 216 | return ret; |
| 198 | 217 | } while (pgd++, addr = next, addr != end); |
| 199 | 218 | |
| 200 | − flush_tlb_kernel_range(start, end); | |
| 219 | + if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) | |
| 220 | + flush_tlb_kernel_range(start, end); | |
| 201 | 221 | |
| 202 | 222 | return 0; |
| 203 | 223 | } |
| @@ -250,7 +270,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, · 46 unmodified lines | ||
| 250 | 270 | } |
| 251 | 271 | |
| 252 | 272 | entry = mk_pte(walk->reuse_page, pgprot); |
| 253 | − list_add_tail(&page->lru, walk->vmemmap_pages); | |
| 273 | + list_add(&page->lru, walk->vmemmap_pages); | |
| 254 | 274 | set_pte_at(&init_mm, addr, pte, entry); |
| 255 | 275 | } |
| 256 | 276 | |
| @@ -296,6 +316,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, · 39 unmodified lines | ||
| 296 | 316 | set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); |
| 297 | 317 | } |
| 298 | 318 | |
| 319 | +/** | |
| 320 | + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) | |
| 321 | + * backing PMDs of the directmap into PTEs | |
| 322 | + * @start: start address of the vmemmap virtual address range that we want | |
| 323 | + * to remap. | |
| 324 | + * @end: end address of the vmemmap virtual address range that we want to | |
| 325 | + * remap. | |
| 326 | + * @reuse: reuse address. | |
| 327 | + * | |
| 328 | + * Return: %0 on success, negative error code otherwise. | |
| 329 | + */ | |
| 330 | +static int vmemmap_remap_split(unsigned long start, unsigned long end, | |
| 331 | + unsigned long reuse) | |
| 332 | +{ | |
| 333 | + int ret; | |
| 334 | + struct vmemmap_remap_walk walk = { | |
| 335 | + .remap_pte = NULL, | |
| 336 | + .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH, | |
| 337 | + }; | |
| 338 | + | |
| 339 | + /* See the comment in the vmemmap_remap_free(). */ | |
| 340 | + BUG_ON(start - reuse != PAGE_SIZE); | |
| 341 | + | |
| 342 | + mmap_read_lock(&init_mm); | |
| 343 | + ret = vmemmap_remap_range(reuse, end, &walk); | |
| 344 | + mmap_read_unlock(&init_mm); | |
| 345 | + | |
| 346 | + return ret; | |
| 347 | +} | |
| 348 | + | |
| 299 | 349 | /** |
| 300 | 350 | * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) |
| 301 | 351 | * to the page which @reuse is mapped to, then free vmemmap |
| @@ -305,22 +355,26 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, · 3 unmodified lines | ||
| 305 | 355 | * @end: end address of the vmemmap virtual address range that we want to |
| 306 | 356 | * remap. |
| 307 | 357 | * @reuse: reuse address. |
| 358 | + * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers | |
| 359 | + * responsibility to free pages. | |
| 360 | + * @flags: modifications to vmemmap_remap_walk flags | |
| 308 | 361 | * |
| 309 | 362 | * Return: %0 on success, negative error code otherwise. |
| 310 | 363 | */ |
| 311 | 364 | static int vmemmap_remap_free(unsigned long start, unsigned long end, |
| 312 | − unsigned long reuse) | |
| 365 | + unsigned long reuse, | |
| 366 | + struct list_head *vmemmap_pages, | |
| 367 | + unsigned long flags) | |
| 313 | 368 | { |
| 314 | 369 | int ret; |
| 315 | − LIST_HEAD(vmemmap_pages); | |
| 316 | 370 | struct vmemmap_remap_walk walk = { |
| 317 | 371 | .remap_pte = vmemmap_remap_pte, |
| 318 | 372 | .reuse_addr = reuse, |
| 319 | − .vmemmap_pages = &vmemmap_pages, | |
| 373 | + .vmemmap_pages = vmemmap_pages, | |
| 374 | + .flags = flags, | |
| 320 | 375 | }; |
| 321 | − int nid = page_to_nid((struct page *)start); | |
| 322 | − gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | | |
| 323 | − __GFP_NOWARN; | |
| 376 | + int nid = page_to_nid((struct page *)reuse); | |
| 377 | + gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; | |
| 324 | 378 | |
| 325 | 379 | /* |
| 326 | 380 | * Allocate a new head vmemmap page to avoid breaking a contiguous |
| @@ -334,7 +388,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, · 7 unmodified lines | ||
| 334 | 388 | if (walk.reuse_page) { |
| 335 | 389 | copy_page(page_to_virt(walk.reuse_page), |
| 336 | 390 | (void *)walk.reuse_addr); |
| 337 | − list_add(&walk.reuse_page->lru, &vmemmap_pages); | |
| 391 | + list_add(&walk.reuse_page->lru, vmemmap_pages); | |
| 338 | 392 | } |
| 339 | 393 | |
| 340 | 394 | /* |
| @@ -365,22 +419,21 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, · 24 unmodified lines | ||
| 365 | 419 | walk = (struct vmemmap_remap_walk) { |
| 366 | 420 | .remap_pte = vmemmap_restore_pte, |
| 367 | 421 | .reuse_addr = reuse, |
| 368 | − .vmemmap_pages = &vmemmap_pages, | |
| 422 | + .vmemmap_pages = vmemmap_pages, | |
| 423 | + .flags = 0, | |
| 369 | 424 | }; |
| 370 | 425 | |
| 371 | 426 | vmemmap_remap_range(reuse, end, &walk); |
| 372 | 427 | } |
| 373 | 428 | mmap_read_unlock(&init_mm); |
| 374 | 429 | |
| 375 | − free_vmemmap_page_list(&vmemmap_pages); | |
| 376 | − | |
| 377 | 430 | return ret; |
| 378 | 431 | } |
| 379 | 432 | |
| 380 | 433 | static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, |
| 381 | 434 | struct list_head *list) |
| 382 | 435 | { |
| 383 | − gfp_t gfp_mask = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_THISNODE; | |
| 436 | + gfp_t gfp_mask = GFP_KERNEL | __GFP_RETRY_MAYFAIL; | |
| 384 | 437 | unsigned long nr_pages = (end - start) >> PAGE_SHIFT; |
| 385 | 438 | int nid = page_to_nid((struct page *)start); |
| 386 | 439 | struct page *page, *next; |
| @@ -389,7 +442,7 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, · 2 unmodified lines | ||
| 389 | 442 | page = alloc_pages_node(nid, gfp_mask, 0); |
| 390 | 443 | if (!page) |
| 391 | 444 | goto out; |
| 392 | − list_add_tail(&page->lru, list); | |
| 445 | + list_add(&page->lru, list); | |
| 393 | 446 | } |
| 394 | 447 | |
| 395 | 448 | return 0; |
| @@ -408,17 +461,19 @@ out: · 12 unmodified lines | ||
| 408 | 461 | * @end: end address of the vmemmap virtual address range that we want to |
| 409 | 462 | * remap. |
| 410 | 463 | * @reuse: reuse address. |
| 464 | + * @flags: modifications to vmemmap_remap_walk flags | |
| 411 | 465 | * |
| 412 | 466 | * Return: %0 on success, negative error code otherwise. |
| 413 | 467 | */ |
| 414 | 468 | static int vmemmap_remap_alloc(unsigned long start, unsigned long end, |
| 415 | − unsigned long reuse) | |
| 469 | + unsigned long reuse, unsigned long flags) | |
| 416 | 470 | { |
| 417 | 471 | LIST_HEAD(vmemmap_pages); |
| 418 | 472 | struct vmemmap_remap_walk walk = { |
| 419 | 473 | .remap_pte = vmemmap_restore_pte, |
| 420 | 474 | .reuse_addr = reuse, |
| 421 | 475 | .vmemmap_pages = &vmemmap_pages, |
| 476 | + .flags = flags, | |
| 422 | 477 | }; |
| 423 | 478 | |
| 424 | 479 | /* See the comment in the vmemmap_remap_free(). */ |
| @@ -440,23 +495,15 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); · 15 unmodified lines | ||
| 440 | 495 | static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); |
| 441 | 496 | core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); |
| 442 | 497 | |
| 443 | −/** | |
| 444 | − * hugetlb_vmemmap_restore - restore previously optimized (by | |
| 445 | − * hugetlb_vmemmap_optimize()) vmemmap pages which | |
| 446 | − * will be reallocated and remapped. | |
| 447 | − * @h: struct hstate. | |
| 448 | − * @head: the head page whose vmemmap pages will be restored. | |
| 449 | − * | |
| 450 | − * Return: %0 if @head's vmemmap pages have been reallocated and remapped, | |
| 451 | − * negative error code otherwise. | |
| 452 | − */ | |
| 453 | −int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) | |
| 498 | +static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio, unsigned long flags) | |
| 454 | 499 | { |
| 455 | 500 | int ret; |
| 501 | + struct page *head = &folio->page; | |
| 456 | 502 | unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; |
| 457 | 503 | unsigned long vmemmap_reuse; |
| 458 | 504 | |
| 459 | − if (!HPageVmemmapOptimized(head)) | |
| 505 | + VM_WARN_ON_ONCE(!PageHuge(head)); | |
| 506 | + if (!folio_test_hugetlb_vmemmap_optimized(folio)) | |
| 460 | 507 | return 0; |
| 461 | 508 | |
| 462 | 509 | vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); |
| @@ -470,18 +517,77 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) · 7 unmodified lines | ||
| 470 | 517 | * When a HugeTLB page is freed to the buddy allocator, previously |
| 471 | 518 | * discarded vmemmap pages must be allocated and remapping. |
| 472 | 519 | */ |
| 473 | − ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); | |
| 520 | + ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags); | |
| 474 | 521 | if (!ret) { |
| 475 | − ClearHPageVmemmapOptimized(head); | |
| 522 | + folio_clear_hugetlb_vmemmap_optimized(folio); | |
| 476 | 523 | static_branch_dec(&hugetlb_optimize_vmemmap_key); |
| 477 | 524 | } |
| 478 | 525 | |
| 479 | 526 | return ret; |
| 480 | 527 | } |
| 481 | 528 | |
| 529 | +/** | |
| 530 | + * hugetlb_vmemmap_restore_folio - restore previously optimized (by | |
| 531 | + * hugetlb_vmemmap_optimize_folio()) vmemmap pages which | |
| 532 | + * will be reallocated and remapped. | |
| 533 | + * @h: struct hstate. | |
| 534 | + * @folio: the folio whose vmemmap pages will be restored. | |
| 535 | + * | |
| 536 | + * Return: %0 if @folio's vmemmap pages have been reallocated and remapped, | |
| 537 | + * negative error code otherwise. | |
| 538 | + */ | |
| 539 | +int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio) | |
| 540 | +{ | |
| 541 | + return __hugetlb_vmemmap_restore_folio(h, folio, 0); | |
| 542 | +} | |
| 543 | + | |
| 544 | +/** | |
| 545 | + * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. | |
| 546 | + * @h: hstate. | |
| 547 | + * @folio_list: list of folios. | |
| 548 | + * @non_hvo_folios: Output list of folios for which vmemmap exists. | |
| 549 | + * | |
| 550 | + * Return: number of folios for which vmemmap was restored, or an error code | |
| 551 | + * if an error was encountered restoring vmemmap for a folio. | |
| 552 | + * Folios that have vmemmap are moved to the non_hvo_folios | |
| 553 | + * list. Processing of entries stops when the first error is | |
| 554 | + * encountered. The folio that experienced the error and all | |
| 555 | + * non-processed folios will remain on folio_list. | |
| 556 | + */ | |
| 557 | +long hugetlb_vmemmap_restore_folios(const struct hstate *h, | |
| 558 | + struct list_head *folio_list, | |
| 559 | + struct list_head *non_hvo_folios) | |
| 560 | +{ | |
| 561 | + struct folio *folio, *t_folio; | |
| 562 | + long restored = 0; | |
| 563 | + long ret = 0; | |
| 564 | + | |
| 565 | + list_for_each_entry_safe(folio, t_folio, folio_list, lru) { | |
| 566 | + if (folio_test_hugetlb_vmemmap_optimized(folio)) { | |
| 567 | + ret = __hugetlb_vmemmap_restore_folio(h, folio, | |
| 568 | + VMEMMAP_REMAP_NO_TLB_FLUSH); | |
| 569 | + if (ret) | |
| 570 | + break; | |
| 571 | + restored++; | |
| 572 | + } | |
| 573 | + | |
| 574 | + /* Add non-optimized folios to output list */ | |
| 575 | + list_move(&folio->lru, non_hvo_folios); | |
| 576 | + } | |
| 577 | + | |
| 578 | + if (restored) | |
| 579 | + flush_tlb_all(); | |
| 580 | + if (!ret) | |
| 581 | + ret = restored; | |
| 582 | + return ret; | |
| 583 | +} | |
| 584 | + | |
| 482 | 585 | /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ |
| 483 | 586 | static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) |
| 484 | 587 | { |
| 588 | + if (HPageVmemmapOptimized((struct page *)head)) | |
| 589 | + return false; | |
| 590 | + | |
| 485 | 591 | if (!READ_ONCE(vmemmap_optimize_enabled)) |
| 486 | 592 | return false; |
| 487 | 593 | |
| @@ -535,39 +641,136 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h · 47 unmodified lines | ||
| 535 | 641 | return true; |
| 536 | 642 | } |
| 537 | 643 | |
| 644 | +static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, | |
| 645 | + struct folio *folio, | |
| 646 | + struct list_head *vmemmap_pages, | |
| 647 | + unsigned long flags) | |
| 648 | +{ | |
| 649 | + int ret = 0; | |
| 650 | + struct page *head = &folio->page; | |
| 651 | + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; | |
| 652 | + unsigned long vmemmap_reuse; | |
| 653 | + | |
| 654 | + VM_WARN_ON_ONCE(!PageHuge(head)); | |
| 655 | + if (!vmemmap_should_optimize(h, head)) | |
| 656 | + return ret; | |
| 657 | + | |
| 658 | + static_branch_inc(&hugetlb_optimize_vmemmap_key); | |
| 659 | + /* | |
| 660 | + * Very Subtle | |
| 661 | + * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed | |
| 662 | + * immediately after remapping. As a result, subsequent accesses | |
| 663 | + * and modifications to struct pages associated with the hugetlb | |
| 664 | + * page could be to the OLD struct pages. Set the vmemmap optimized | |
| 665 | + * flag here so that it is copied to the new head page. This keeps | |
| 666 | + * the old and new struct pages in sync. | |
| 667 | + * If there is an error during optimization, we will immediately FLUSH | |
| 668 | + * the TLB and clear the flag below. | |
| 669 | + */ | |
| 670 | + folio_set_hugetlb_vmemmap_optimized(folio); | |
| 671 | + | |
| 672 | + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); | |
| 673 | + vmemmap_reuse = vmemmap_start; | |
| 674 | + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; | |
| 675 | + | |
| 676 | + /* | |
| 677 | + * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) | |
| 678 | + * to the page which @vmemmap_reuse is mapped to. Add pages previously | |
| 679 | + * mapping the range to vmemmap_pages list so that they can be freed by | |
| 680 | + * the caller. | |
| 681 | + */ | |
| 682 | + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, | |
| 683 | + vmemmap_pages, flags); | |
| 684 | + if (ret) { | |
| 685 | + static_branch_dec(&hugetlb_optimize_vmemmap_key); | |
| 686 | + folio_clear_hugetlb_vmemmap_optimized(folio); | |
| 687 | + } | |
| 688 | + | |
| 689 | + return ret; | |
| 690 | +} | |
| 691 | + | |
| 538 | 692 | /** |
| 539 | − * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. | |
| 693 | + * hugetlb_vmemmap_optimize_folio - optimize @folio's vmemmap pages. | |
| 540 | 694 | * @h: struct hstate. |
| 541 | − * @head: the head page whose vmemmap pages will be optimized. | |
| 695 | + * @folio: the folio whose vmemmap pages will be optimized. | |
| 542 | 696 | * |
| 543 | − * This function only tries to optimize @head's vmemmap pages and does not | |
| 697 | + * This function only tries to optimize @folio's vmemmap pages and does not | |
| 544 | 698 | * guarantee that the optimization will succeed after it returns. The caller |
| 545 | − * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages | |
| 546 | − * have been optimized. | |
| 699 | + * can use folio_test_hugetlb_vmemmap_optimized(@folio) to detect if @folio's | |
| 700 | + * vmemmap pages have been optimized. | |
| 547 | 701 | */ |
| 548 | −void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) | |
| 702 | +void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio) | |
| 703 | +{ | |
| 704 | + LIST_HEAD(vmemmap_pages); | |
| 705 | + | |
| 706 | + __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, 0); | |
| 707 | + free_vmemmap_page_list(&vmemmap_pages); | |
| 708 | +} | |
| 709 | + | |
| 710 | +static int hugetlb_vmemmap_split(const struct hstate *h, struct page *head) | |
| 549 | 711 | { |
| 550 | 712 | unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; |
| 551 | 713 | unsigned long vmemmap_reuse; |
| 552 | 714 | |
| 553 | 715 | if (!vmemmap_should_optimize(h, head)) |
| 554 | − return; | |
| 555 | − | |
| 556 | − static_branch_inc(&hugetlb_optimize_vmemmap_key); | |
| 716 | + return 0; | |
| 557 | 717 | |
| 558 | 718 | vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); |
| 559 | 719 | vmemmap_reuse = vmemmap_start; |
| 560 | 720 | vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; |
| 561 | 721 | |
| 562 | 722 | /* |
| 563 | − * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) | |
| 564 | − * to the page which @vmemmap_reuse is mapped to, then free the pages | |
| 565 | − * which the range [@vmemmap_start, @vmemmap_end] is mapped to. | |
| 723 | + * Split PMDs on the vmemmap virtual address range [@vmemmap_start, | |
| 724 | + * @vmemmap_end] | |
| 566 | 725 | */ |
| 567 | − if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse)) | |
| 568 | − static_branch_dec(&hugetlb_optimize_vmemmap_key); | |
| 569 | − else | |
| 570 | − SetHPageVmemmapOptimized(head); | |
| 726 | + return vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); | |
| 727 | +} | |
| 728 | + | |
| 729 | +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) | |
| 730 | +{ | |
| 731 | + struct folio *folio; | |
| 732 | + LIST_HEAD(vmemmap_pages); | |
| 733 | + | |
| 734 | + list_for_each_entry(folio, folio_list, lru) { | |
| 735 | + int ret = hugetlb_vmemmap_split(h, &folio->page); | |
| 736 | + | |
| 737 | + /* | |
| 738 | + * Spliting the PMD requires allocating a page, thus lets fail | |
| 739 | + * early once we encounter the first OOM. No point in retrying | |
| 740 | + * as it can be dynamically done on remap with the memory | |
| 741 | + * we get back from the vmemmap deduplication. | |
| 742 | + */ | |
| 743 | + if (ret == -ENOMEM) | |
| 744 | + break; | |
| 745 | + } | |
| 746 | + | |
| 747 | + flush_tlb_all(); | |
| 748 | + | |
| 749 | + list_for_each_entry(folio, folio_list, lru) { | |
| 750 | + int ret = __hugetlb_vmemmap_optimize_folio(h, folio, | |
| 751 | + &vmemmap_pages, | |
| 752 | + VMEMMAP_REMAP_NO_TLB_FLUSH); | |
| 753 | + | |
| 754 | + /* | |
| 755 | + * Pages to be freed may have been accumulated. If we | |
| 756 | + * encounter an ENOMEM, free what we have and try again. | |
| 757 | + * This can occur in the case that both spliting fails | |
| 758 | + * halfway and head page allocation also failed. In this | |
| 759 | + * case __hugetlb_vmemmap_optimize_folio() would free memory | |
| 760 | + * allowing more vmemmap remaps to occur. | |
| 761 | + */ | |
| 762 | + if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { | |
| 763 | + flush_tlb_all(); | |
| 764 | + free_vmemmap_page_list(&vmemmap_pages); | |
| 765 | + INIT_LIST_HEAD(&vmemmap_pages); | |
| 766 | + __hugetlb_vmemmap_optimize_folio(h, folio, | |
| 767 | + &vmemmap_pages, | |
| 768 | + VMEMMAP_REMAP_NO_TLB_FLUSH); | |
| 769 | + } | |
| 770 | + } | |
| 771 | + | |
| 772 | + flush_tlb_all(); | |
| 773 | + free_vmemmap_page_list(&vmemmap_pages); | |
| 571 | 774 | } |
| 572 | 775 | |
| 573 | 776 | static struct ctl_table hugetlb_vmemmap_sysctls[] = { |
| @@ -586,7 +789,7 @@ static int __init hugetlb_vmemmap_init(void) · 12 unmodified lines | ||
| 586 | 789 | const struct hstate *h; |
| 587 | 790 | |
| 588 | 791 | /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */ |
| 589 | − BUILD_BUG_ON(__NR_USED_SUBPAGE * sizeof(struct page) > HUGETLB_VMEMMAP_RESERVE_SIZE); | |
| 792 | + BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES); | |
| 590 | 793 | |
| 591 | 794 | for_each_hstate(h) { |
| 592 | 795 | if (hugetlb_vmemmap_optimizable(h)) { |
| @@ -10,15 +10,20 @@ · 9 unmodified lines | ||
| 10 | 10 | #define _LINUX_HUGETLB_VMEMMAP_H |
| 11 | 11 | #include <linux/hugetlb.h> |
| 12 | 12 | |
| 13 | −#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP | |
| 14 | −int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); | |
| 15 | −void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); | |
| 16 | − | |
| 17 | 13 | /* |
| 18 | 14 | * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See |
| 19 | − * Documentation/vm/vmemmap_dedup.rst. | |
| 15 | + * Documentation/mm/vmemmap_dedup.rst. | |
| 20 | 16 | */ |
| 21 | 17 | #define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE |
| 18 | +#define HUGETLB_VMEMMAP_RESERVE_PAGES (HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page)) | |
| 19 | + | |
| 20 | +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP | |
| 21 | +int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio); | |
| 22 | +long hugetlb_vmemmap_restore_folios(const struct hstate *h, | |
| 23 | + struct list_head *folio_list, | |
| 24 | + struct list_head *non_hvo_folios); | |
| 25 | +void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio); | |
| 26 | +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); | |
| 22 | 27 | |
| 23 | 28 | static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) |
| 24 | 29 | { |
| @@ -38,12 +43,24 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate · 13 unmodified lines | ||
| 38 | 43 | return size > 0 ? size : 0; |
| 39 | 44 | } |
| 40 | 45 | #else |
| 41 | −static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) | |
| 46 | +static inline int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio) | |
| 42 | 47 | { |
| 43 | 48 | return 0; |
| 44 | 49 | } |
| 45 | 50 | |
| 46 | −static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) | |
| 51 | +static long hugetlb_vmemmap_restore_folios(const struct hstate *h, | |
| 52 | + struct list_head *folio_list, | |
| 53 | + struct list_head *non_hvo_folios) | |
| 54 | +{ | |
| 55 | + list_splice_init(folio_list, non_hvo_folios); | |
| 56 | + return 0; | |
| 57 | +} | |
| 58 | + | |
| 59 | +static inline void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio) | |
| 60 | +{ | |
| 61 | +} | |
| 62 | + | |
| 63 | +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) | |
| 47 | 64 | { |
| 48 | 65 | } |
| 49 | 66 | |
| @@ -415,6 +415,15 @@ static inline void folio_set_order(struct folio *folio, unsigned int order) · 414 unmodified lines | ||
| 415 | 415 | |
| 416 | 416 | void folio_undo_large_rmappable(struct folio *folio); |
| 417 | 417 | |
| 418 | +static inline struct folio *page_rmappable_folio(struct page *page) | |
| 419 | +{ | |
| 420 | + struct folio *folio = (struct folio *)page; | |
| 421 | + | |
| 422 | + if (folio && folio_order(folio) > 1) | |
| 423 | + folio_prep_large_rmappable(folio); | |
| 424 | + return folio; | |
| 425 | +} | |
| 426 | + | |
| 418 | 427 | static inline void prep_compound_head(struct page *page, unsigned int order) |
| 419 | 428 | { |
| 420 | 429 | struct folio *folio = (struct folio *)page; |
| @@ -586,6 +595,56 @@ extern long faultin_vma_page_range(struct vm_area_struct *vma, · 165 unmodified lines | ||
| 586 | 595 | bool write, int *locked); |
| 587 | 596 | extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, |
| 588 | 597 | unsigned long bytes); |
| 598 | + | |
| 599 | +/* | |
| 600 | + * NOTE: This function can't tell whether the folio is "fully mapped" in the | |
| 601 | + * range. | |
| 602 | + * "fully mapped" means all the pages of folio is associated with the page | |
| 603 | + * table of range while this function just check whether the folio range is | |
| 604 | + * within the range [start, end). Function caller needs to do page table | |
| 605 | + * check if it cares about the page table association. | |
| 606 | + * | |
| 607 | + * Typical usage (like mlock or madvise) is: | |
| 608 | + * Caller knows at least 1 page of folio is associated with page table of VMA | |
| 609 | + * and the range [start, end) is intersect with the VMA range. Caller wants | |
| 610 | + * to know whether the folio is fully associated with the range. It calls | |
| 611 | + * this function to check whether the folio is in the range first. Then checks | |
| 612 | + * the page table to know whether the folio is fully mapped to the range. | |
| 613 | + */ | |
| 614 | +static inline bool | |
| 615 | +folio_within_range(struct folio *folio, struct vm_area_struct *vma, | |
| 616 | + unsigned long start, unsigned long end) | |
| 617 | +{ | |
| 618 | + pgoff_t pgoff, addr; | |
| 619 | + unsigned long vma_pglen = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; | |
| 620 | + | |
| 621 | + VM_WARN_ON_FOLIO(folio_test_ksm(folio), folio); | |
| 622 | + if (start > end) | |
| 623 | + return false; | |
| 624 | + | |
| 625 | + if (start < vma->vm_start) | |
| 626 | + start = vma->vm_start; | |
| 627 | + | |
| 628 | + if (end > vma->vm_end) | |
| 629 | + end = vma->vm_end; | |
| 630 | + | |
| 631 | + pgoff = folio_pgoff(folio); | |
| 632 | + | |
| 633 | + /* if folio start address is not in vma range */ | |
| 634 | + if (!in_range(pgoff, vma->vm_pgoff, vma_pglen)) | |
| 635 | + return false; | |
| 636 | + | |
| 637 | + addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); | |
| 638 | + | |
| 639 | + return !(addr < start || end - addr < folio_size(folio)); | |
| 640 | +} | |
| 641 | + | |
| 642 | +static inline bool | |
| 643 | +folio_within_vma(struct folio *folio, struct vm_area_struct *vma) | |
| 644 | +{ | |
| 645 | + return folio_within_range(folio, vma, vma->vm_start, vma->vm_end); | |
| 646 | +} | |
| 647 | + | |
| 589 | 648 | /* |
| 590 | 649 | * mlock_vma_folio() and munlock_vma_folio(): |
| 591 | 650 | * should be called with vma's mmap_lock held for read or write, |
| @@ -594,14 +653,10 @@ extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, · 2 unmodified lines | ||
| 594 | 653 | * mlock is usually called at the end of page_add_*_rmap(), munlock at |
| 595 | 654 | * the end of page_remove_rmap(); but new anon folios are managed by |
| 596 | 655 | * folio_add_lru_vma() calling mlock_new_folio(). |
| 597 | − * | |
| 598 | − * @compound is used to include pmd mappings of THPs, but filter out | |
| 599 | − * pte mappings of THPs, which cannot be consistently counted: a pte | |
| 600 | − * mapping of the THP head cannot be distinguished by the page alone. | |
| 601 | 656 | */ |
| 602 | 657 | void mlock_folio(struct folio *folio); |
| 603 | 658 | static inline void mlock_vma_folio(struct folio *folio, |
| 604 | − struct vm_area_struct *vma, bool compound) | |
| 659 | + struct vm_area_struct *vma) | |
| 605 | 660 | { |
| 606 | 661 | /* |
| 607 | 662 | * The VM_SPECIAL check here serves two purposes. |
| @@ -611,17 +666,24 @@ static inline void mlock_vma_folio(struct folio *folio, · 3 unmodified lines | ||
| 611 | 666 | * file->f_op->mmap() is using vm_insert_page(s), when VM_LOCKED may |
| 612 | 667 | * still be set while VM_SPECIAL bits are added: so ignore it then. |
| 613 | 668 | */ |
| 614 | − if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED) && | |
| 615 | − (compound || !folio_test_large(folio))) | |
| 669 | + if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED)) | |
| 616 | 670 | mlock_folio(folio); |
| 617 | 671 | } |
| 618 | 672 | |
| 619 | 673 | void munlock_folio(struct folio *folio); |
| 620 | 674 | static inline void munlock_vma_folio(struct folio *folio, |
| 621 | − struct vm_area_struct *vma, bool compound) | |
| 675 | + struct vm_area_struct *vma) | |
| 622 | 676 | { |
| 623 | − if (unlikely(vma->vm_flags & VM_LOCKED) && | |
| 624 | − (compound || !folio_test_large(folio))) | |
| 677 | + /* | |
| 678 | + * munlock if the function is called. Ideally, we should only | |
| 679 | + * do munlock if any page of folio is unmapped from VMA and | |
| 680 | + * cause folio not fully mapped to VMA. | |
| 681 | + * | |
| 682 | + * But it's not easy to confirm that's the situation. So we | |
| 683 | + * always munlock the folio and page reclaim will correct it | |
| 684 | + * if it's wrong. | |
| 685 | + */ | |
| 686 | + if (unlikely(vma->vm_flags & VM_LOCKED)) | |
| 625 | 687 | munlock_folio(folio); |
| 626 | 688 | } |
| 627 | 689 | |
| @@ -930,7 +992,7 @@ void vunmap_range_noflush(unsigned long start, unsigned long end); · 302 unmodified lines | ||
| 930 | 992 | |
| 931 | 993 | void __vunmap_range_noflush(unsigned long start, unsigned long end); |
| 932 | 994 | |
| 933 | −int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, | |
| 995 | +int numa_migrate_prep(struct folio *folio, struct vm_area_struct *vma, | |
| 934 | 996 | unsigned long addr, int page_nid, int *flags); |
| 935 | 997 | |
| 936 | 998 | void free_zone_device_page(struct page *page); |
| @@ -949,6 +1011,13 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, · 12 unmodified lines | ||
| 949 | 1011 | unsigned long addr, pmd_t *pmd, |
| 950 | 1012 | unsigned int flags); |
| 951 | 1013 | |
| 1014 | +/* | |
| 1015 | + * mm/mmap.c | |
| 1016 | + */ | |
| 1017 | +struct vm_area_struct *vma_merge_extend(struct vma_iterator *vmi, | |
| 1018 | + struct vm_area_struct *vma, | |
| 1019 | + unsigned long delta); | |
| 1020 | + | |
| 952 | 1021 | enum { |
| 953 | 1022 | /* mark page accessed */ |
| 954 | 1023 | FOLL_TOUCH = 1 << 16, |
| @@ -964,6 +1033,9 @@ enum { · 9 unmodified lines | ||
| 964 | 1033 | FOLL_UNLOCKABLE = 1 << 21, |
| 965 | 1034 | }; |
| 966 | 1035 | |
| 1036 | +#define INTERNAL_GUP_FLAGS (FOLL_TOUCH | FOLL_TRIED | FOLL_REMOTE | FOLL_PIN | \ | |
| 1037 | + FOLL_FAST_ONLY | FOLL_UNLOCKABLE) | |
| 1038 | + | |
| 967 | 1039 | /* |
| 968 | 1040 | * Indicates for which pages that are write-protected in the page table, |
| 969 | 1041 | * whether GUP has to trigger unsharing via FAULT_FLAG_UNSHARE such that the |
| @@ -1154,4 +1226,57 @@ struct vma_prepare { · 184 unmodified lines | ||
| 1154 | 1226 | struct vm_area_struct *remove; |
| 1155 | 1227 | struct vm_area_struct *remove2; |
| 1156 | 1228 | }; |
| 1229 | + | |
| 1230 | +void __meminit __init_single_page(struct page *page, unsigned long pfn, | |
| 1231 | + unsigned long zone, int nid); | |
| 1232 | + | |
| 1233 | +/* shrinker related functions */ | |
| 1234 | +unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, | |
| 1235 | + int priority); | |
| 1236 | + | |
| 1237 | +#ifdef CONFIG_SHRINKER_DEBUG | |
| 1238 | +static inline __printf(2, 0) int shrinker_debugfs_name_alloc( | |
| 1239 | + struct shrinker *shrinker, const char *fmt, va_list ap) | |
| 1240 | +{ | |
| 1241 | + shrinker->name = kvasprintf_const(GFP_KERNEL, fmt, ap); | |
| 1242 | + | |
| 1243 | + return shrinker->name ? 0 : -ENOMEM; | |
| 1244 | +} | |
| 1245 | + | |
| 1246 | +static inline void shrinker_debugfs_name_free(struct shrinker *shrinker) | |
| 1247 | +{ | |
| 1248 | + kfree_const(shrinker->name); | |
| 1249 | + shrinker->name = NULL; | |
| 1250 | +} | |
| 1251 | + | |
| 1252 | +extern int shrinker_debugfs_add(struct shrinker *shrinker); | |
| 1253 | +extern struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, | |
| 1254 | + int *debugfs_id); | |
| 1255 | +extern void shrinker_debugfs_remove(struct dentry *debugfs_entry, | |
| 1256 | + int debugfs_id); | |
| 1257 | +#else /* CONFIG_SHRINKER_DEBUG */ | |
| 1258 | +static inline int shrinker_debugfs_add(struct shrinker *shrinker) | |
| 1259 | +{ | |
| 1260 | + return 0; | |
| 1261 | +} | |
| 1262 | +static inline int shrinker_debugfs_name_alloc(struct shrinker *shrinker, | |
| 1263 | + const char *fmt, va_list ap) | |
| 1264 | +{ | |
| 1265 | + return 0; | |
| 1266 | +} | |
| 1267 | +static inline void shrinker_debugfs_name_free(struct shrinker *shrinker) | |
| 1268 | +{ | |
| 1269 | +} | |
| 1270 | +static inline struct dentry *shrinker_debugfs_detach(struct shrinker *shrinker, | |
| 1271 | + int *debugfs_id) | |
| 1272 | +{ | |
| 1273 | + *debugfs_id = -1; | |
| 1274 | + return NULL; | |
| 1275 | +} | |
| 1276 | +static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, | |
| 1277 | + int debugfs_id) | |
| 1278 | +{ | |
| 1279 | +} | |
| 1280 | +#endif /* CONFIG_SHRINKER_DEBUG */ | |
| 1281 | + | |
| 1157 | 1282 | #endif /* __MM_INTERNAL_H */ |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | * Author: Andrey Ryabinin <a.ryabinin@samsung.com> |
| 6 | 6 | */ |
| 7 | 7 | |
| 8 | −#define pr_fmt(fmt) "kasan_test: " fmt | |
| 8 | +#define pr_fmt(fmt) "kasan: test: " fmt | |
| 9 | 9 | |
| 10 | 10 | #include <kunit/test.h> |
| 11 | 11 | #include <linux/bitops.h> |
| @@ -91,10 +91,11 @@ static void kasan_test_exit(struct kunit *test) · 79 unmodified lines | ||
| 91 | 91 | } |
| 92 | 92 | |
| 93 | 93 | /** |
| 94 | − * KUNIT_EXPECT_KASAN_FAIL() - check that the executed expression produces a | |
| 95 | − * KASAN report; causes a test failure otherwise. This relies on a KUnit | |
| 96 | − * resource named "kasan_status". Do not use this name for KUnit resources | |
| 97 | − * outside of KASAN tests. | |
| 94 | + * KUNIT_EXPECT_KASAN_FAIL - check that the executed expression produces a | |
| 95 | + * KASAN report; causes a KUnit test failure otherwise. | |
| 96 | + * | |
| 97 | + * @test: Currently executing KUnit test. | |
| 98 | + * @expression: Expression that must produce a KASAN report. | |
| 98 | 99 | * |
| 99 | 100 | * For hardware tag-based KASAN, when a synchronous tag fault happens, tag |
| 100 | 101 | * checking is auto-disabled. When this happens, this test handler reenables |
| @@ -1097,11 +1098,9 @@ static void kasan_bitops_test_and_modify(struct kunit *test, int nr, void *addr) · 996 unmodified lines | ||
| 1097 | 1098 | KUNIT_EXPECT_KASAN_FAIL(test, test_and_change_bit(nr, addr)); |
| 1098 | 1099 | KUNIT_EXPECT_KASAN_FAIL(test, __test_and_change_bit(nr, addr)); |
| 1099 | 1100 | KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = test_bit(nr, addr)); |
| 1100 | − | |
| 1101 | −#if defined(clear_bit_unlock_is_negative_byte) | |
| 1102 | − KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = | |
| 1103 | − clear_bit_unlock_is_negative_byte(nr, addr)); | |
| 1104 | −#endif | |
| 1101 | + if (nr < 7) | |
| 1102 | + KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = | |
| 1103 | + xor_unlock_is_negative_byte(1 << nr, addr)); | |
| 1105 | 1104 | } |
| 1106 | 1105 | |
| 1107 | 1106 | static void kasan_bitops_generic(struct kunit *test) |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | * Author: Andrey Ryabinin <a.ryabinin@samsung.com> |
| 6 | 6 | */ |
| 7 | 7 | |
| 8 | −#define pr_fmt(fmt) "kasan test: %s " fmt, __func__ | |
| 8 | +#define pr_fmt(fmt) "kasan: test: " fmt | |
| 9 | 9 | |
| 10 | 10 | #include <linux/mman.h> |
| 11 | 11 | #include <linux/module.h> |
| @@ -8,6 +8,8 @@ · 7 unmodified lines | ||
| 8 | 8 | * Based on code by Dmitry Chernenkov. |
| 9 | 9 | */ |
| 10 | 10 | |
| 11 | +#define pr_fmt(fmt) "kasan: " fmt | |
| 12 | + | |
| 11 | 13 | #include <linux/gfp.h> |
| 12 | 14 | #include <linux/hash.h> |
| 13 | 15 | #include <linux/kernel.h> |
| @@ -414,7 +416,7 @@ static int __init kasan_cpu_quarantine_init(void) · 400 unmodified lines | ||
| 414 | 416 | ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "mm/kasan:online", |
| 415 | 417 | kasan_cpu_online, kasan_cpu_offline); |
| 416 | 418 | if (ret < 0) |
| 417 | − pr_err("kasan cpu quarantine register failed [%d]\n", ret); | |
| 419 | + pr_err("cpu quarantine register failed [%d]\n", ret); | |
| 418 | 420 | return ret; |
| 419 | 421 | } |
| 420 | 422 | late_initcall(kasan_cpu_quarantine_init); |
| @@ -538,7 +538,7 @@ void kasan_report_invalid_free(void *ptr, unsigned long ip, enum kasan_report_ty · 537 unmodified lines | ||
| 538 | 538 | |
| 539 | 539 | start_report(&flags, true); |
| 540 | 540 | |
| 541 | − memset(&info, 0, sizeof(info)); | |
| 541 | + __memset(&info, 0, sizeof(info)); | |
| 542 | 542 | info.type = type; |
| 543 | 543 | info.access_addr = ptr; |
| 544 | 544 | info.access_size = 0; |
| @@ -576,7 +576,7 @@ bool kasan_report(const void *addr, size_t size, bool is_write, · 31 unmodified lines | ||
| 576 | 576 | |
| 577 | 577 | start_report(&irq_flags, true); |
| 578 | 578 | |
| 579 | − memset(&info, 0, sizeof(info)); | |
| 579 | + __memset(&info, 0, sizeof(info)); | |
| 580 | 580 | info.type = KASAN_REPORT_ACCESS; |
| 581 | 581 | info.access_addr = addr; |
| 582 | 582 | info.access_size = size; |
| @@ -220,7 +220,7 @@ static bool __must_check tokenize_frame_descr(const char **frame_descr, · 219 unmodified lines | ||
| 220 | 220 | const size_t tok_len = sep - *frame_descr; |
| 221 | 221 | |
| 222 | 222 | if (tok_len + 1 > max_tok_len) { |
| 223 | − pr_err("KASAN internal error: frame description too long: %s\n", | |
| 223 | + pr_err("internal error: frame description too long: %s\n", | |
| 224 | 224 | *frame_descr); |
| 225 | 225 | return false; |
| 226 | 226 | } |
| @@ -233,7 +233,7 @@ static bool __must_check tokenize_frame_descr(const char **frame_descr, · 6 unmodified lines | ||
| 233 | 233 | *frame_descr = sep + 1; |
| 234 | 234 | |
| 235 | 235 | if (value != NULL && kstrtoul(token, 10, value)) { |
| 236 | − pr_err("KASAN internal error: not a valid number: %s\n", token); | |
| 236 | + pr_err("internal error: not a valid number: %s\n", token); | |
| 237 | 237 | return false; |
| 238 | 238 | } |
| 239 | 239 | |
| @@ -323,7 +323,7 @@ static bool __must_check get_address_stack_frame_info(const void *addr, · 83 unmodified lines | ||
| 323 | 323 | |
| 324 | 324 | frame = (const unsigned long *)(mem_ptr + KASAN_GRANULE_SIZE); |
| 325 | 325 | if (frame[0] != KASAN_CURRENT_STACK_FRAME_MAGIC) { |
| 326 | − pr_err("KASAN internal error: frame info validation failed; invalid marker: %lu\n", | |
| 326 | + pr_err("internal error: frame has invalid marker: %lu\n", | |
| 327 | 327 | frame[0]); |
| 328 | 328 | return false; |
| 329 | 329 | } |
| @@ -324,7 +324,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, · 323 unmodified lines | ||
| 324 | 324 | if (!page) |
| 325 | 325 | return -ENOMEM; |
| 326 | 326 | |
| 327 | − memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE); | |
| 327 | + __memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE); | |
| 328 | 328 | pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL); |
| 329 | 329 | |
| 330 | 330 | spin_lock(&init_mm.page_table_lock); |
| @@ -524,15 +524,15 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte, · 523 unmodified lines | ||
| 524 | 524 | } |
| 525 | 525 | } |
| 526 | 526 | |
| 527 | −static bool is_refcount_suitable(struct page *page) | |
| 527 | +static bool is_refcount_suitable(struct folio *folio) | |
| 528 | 528 | { |
| 529 | 529 | int expected_refcount; |
| 530 | 530 | |
| 531 | − expected_refcount = total_mapcount(page); | |
| 532 | − if (PageSwapCache(page)) | |
| 533 | − expected_refcount += compound_nr(page); | |
| 531 | + expected_refcount = folio_mapcount(folio); | |
| 532 | + if (folio_test_swapcache(folio)) | |
| 533 | + expected_refcount += folio_nr_pages(folio); | |
| 534 | 534 | |
| 535 | − return page_count(page) == expected_refcount; | |
| 535 | + return folio_ref_count(folio) == expected_refcount; | |
| 536 | 536 | } |
| 537 | 537 | |
| 538 | 538 | static int __collapse_huge_page_isolate(struct vm_area_struct *vma, |
| @@ -542,6 +542,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 3 unmodified lines | ||
| 542 | 542 | struct list_head *compound_pagelist) |
| 543 | 543 | { |
| 544 | 544 | struct page *page = NULL; |
| 545 | + struct folio *folio = NULL; | |
| 545 | 546 | pte_t *_pte; |
| 546 | 547 | int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; |
| 547 | 548 | bool writable = false; |
| @@ -576,7 +577,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 28 unmodified lines | ||
| 576 | 577 | goto out; |
| 577 | 578 | } |
| 578 | 579 | |
| 579 | − VM_BUG_ON_PAGE(!PageAnon(page), page); | |
| 580 | + folio = page_folio(page); | |
| 581 | + VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); | |
| 580 | 582 | |
| 581 | 583 | if (page_mapcount(page) > 1) { |
| 582 | 584 | ++shared; |
| @@ -588,16 +590,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 5 unmodified lines | ||
| 588 | 590 | } |
| 589 | 591 | } |
| 590 | 592 | |
| 591 | − if (PageCompound(page)) { | |
| 592 | − struct page *p; | |
| 593 | − page = compound_head(page); | |
| 593 | + if (folio_test_large(folio)) { | |
| 594 | + struct folio *f; | |
| 594 | 595 | |
| 595 | 596 | /* |
| 596 | 597 | * Check if we have dealt with the compound page |
| 597 | 598 | * already |
| 598 | 599 | */ |
| 599 | − list_for_each_entry(p, compound_pagelist, lru) { | |
| 600 | − if (page == p) | |
| 600 | + list_for_each_entry(f, compound_pagelist, lru) { | |
| 601 | + if (folio == f) | |
| 601 | 602 | goto next; |
| 602 | 603 | } |
| 603 | 604 | } |
| @@ -608,7 +609,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 4 unmodified lines | ||
| 608 | 609 | * is needed to serialize against split_huge_page |
| 609 | 610 | * when invoked from the VM. |
| 610 | 611 | */ |
| 611 | − if (!trylock_page(page)) { | |
| 612 | + if (!folio_trylock(folio)) { | |
| 612 | 613 | result = SCAN_PAGE_LOCK; |
| 613 | 614 | goto out; |
| 614 | 615 | } |
| @@ -624,8 +625,8 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 9 unmodified lines | ||
| 624 | 625 | * but not from this process. The other process cannot write to |
| 625 | 626 | * the page, only trigger CoW. |
| 626 | 627 | */ |
| 627 | − if (!is_refcount_suitable(page)) { | |
| 628 | − unlock_page(page); | |
| 628 | + if (!is_refcount_suitable(folio)) { | |
| 629 | + folio_unlock(folio); | |
| 629 | 630 | result = SCAN_PAGE_COUNT; |
| 630 | 631 | goto out; |
| 631 | 632 | } |
| @@ -634,27 +635,27 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · 2 unmodified lines | ||
| 634 | 635 | * Isolate the page to avoid collapsing an hugepage |
| 635 | 636 | * currently in use by the VM. |
| 636 | 637 | */ |
| 637 | − if (!isolate_lru_page(page)) { | |
| 638 | − unlock_page(page); | |
| 638 | + if (!folio_isolate_lru(folio)) { | |
| 639 | + folio_unlock(folio); | |
| 639 | 640 | result = SCAN_DEL_PAGE_LRU; |
| 640 | 641 | goto out; |
| 641 | 642 | } |
| 642 | − mod_node_page_state(page_pgdat(page), | |
| 643 | − NR_ISOLATED_ANON + page_is_file_lru(page), | |
| 644 | − compound_nr(page)); | |
| 645 | − VM_BUG_ON_PAGE(!PageLocked(page), page); | |
| 646 | − VM_BUG_ON_PAGE(PageLRU(page), page); | |
| 643 | + node_stat_mod_folio(folio, | |
| 644 | + NR_ISOLATED_ANON + folio_is_file_lru(folio), | |
| 645 | + folio_nr_pages(folio)); | |
| 646 | + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); | |
| 647 | + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); | |
| 647 | 648 | |
| 648 | − if (PageCompound(page)) | |
| 649 | − list_add_tail(&page->lru, compound_pagelist); | |
| 649 | + if (folio_test_large(folio)) | |
| 650 | + list_add_tail(&folio->lru, compound_pagelist); | |
| 650 | 651 | next: |
| 651 | 652 | /* |
| 652 | 653 | * If collapse was initiated by khugepaged, check that there is |
| 653 | 654 | * enough young pte to justify collapsing the page |
| 654 | 655 | */ |
| 655 | 656 | if (cc->is_khugepaged && |
| 656 | − (pte_young(pteval) || page_is_young(page) || | |
| 657 | − PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, | |
| 657 | + (pte_young(pteval) || folio_test_young(folio) || | |
| 658 | + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, | |
| 658 | 659 | address))) |
| 659 | 660 | referenced++; |
| 660 | 661 | |
| @@ -668,13 +669,13 @@ next: · 7 unmodified lines | ||
| 668 | 669 | result = SCAN_LACK_REFERENCED_PAGE; |
| 669 | 670 | } else { |
| 670 | 671 | result = SCAN_SUCCEED; |
| 671 | − trace_mm_collapse_huge_page_isolate(page, none_or_zero, | |
| 672 | + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, | |
| 672 | 673 | referenced, writable, result); |
| 673 | 674 | return result; |
| 674 | 675 | } |
| 675 | 676 | out: |
| 676 | 677 | release_pte_pages(pte, _pte, compound_pagelist); |
| 677 | − trace_mm_collapse_huge_page_isolate(page, none_or_zero, | |
| 678 | + trace_mm_collapse_huge_page_isolate(&folio->page, none_or_zero, | |
| 678 | 679 | referenced, writable, result); |
| 679 | 680 | return result; |
| 680 | 681 | } |
| @@ -887,16 +888,16 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) · 206 unmodified lines | ||
| 887 | 888 | } |
| 888 | 889 | #endif |
| 889 | 890 | |
| 890 | −static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int node, | |
| 891 | +static bool hpage_collapse_alloc_folio(struct folio **folio, gfp_t gfp, int node, | |
| 891 | 892 | nodemask_t *nmask) |
| 892 | 893 | { |
| 893 | − *hpage = __alloc_pages(gfp, HPAGE_PMD_ORDER, node, nmask); | |
| 894 | − if (unlikely(!*hpage)) { | |
| 894 | + *folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, nmask); | |
| 895 | + | |
| 896 | + if (unlikely(!*folio)) { | |
| 895 | 897 | count_vm_event(THP_COLLAPSE_ALLOC_FAILED); |
| 896 | 898 | return false; |
| 897 | 899 | } |
| 898 | 900 | |
| 899 | − folio_prep_large_rmappable((struct folio *)*hpage); | |
| 900 | 901 | count_vm_event(THP_COLLAPSE_ALLOC); |
| 901 | 902 | return true; |
| 902 | 903 | } |
| @@ -1063,17 +1064,20 @@ static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, · 160 unmodified lines | ||
| 1063 | 1064 | int node = hpage_collapse_find_target_node(cc); |
| 1064 | 1065 | struct folio *folio; |
| 1065 | 1066 | |
| 1066 | − if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask)) | |
| 1067 | + if (!hpage_collapse_alloc_folio(&folio, gfp, node, &cc->alloc_nmask)) { | |
| 1068 | + *hpage = NULL; | |
| 1067 | 1069 | return SCAN_ALLOC_HUGE_PAGE_FAIL; |
| 1070 | + } | |
| 1068 | 1071 | |
| 1069 | − folio = page_folio(*hpage); | |
| 1070 | 1072 | if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { |
| 1071 | 1073 | folio_put(folio); |
| 1072 | 1074 | *hpage = NULL; |
| 1073 | 1075 | return SCAN_CGROUP_CHARGE_FAIL; |
| 1074 | 1076 | } |
| 1075 | − count_memcg_page_event(*hpage, THP_COLLAPSE_ALLOC); | |
| 1076 | 1077 | |
| 1078 | + count_memcg_folio_events(folio, THP_COLLAPSE_ALLOC, 1); | |
| 1079 | + | |
| 1080 | + *hpage = folio_page(folio, 0); | |
| 1077 | 1081 | return SCAN_SUCCEED; |
| 1078 | 1082 | } |
| 1079 | 1083 | |
| @@ -1247,6 +1251,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 167 unmodified lines | ||
| 1247 | 1251 | int result = SCAN_FAIL, referenced = 0; |
| 1248 | 1252 | int none_or_zero = 0, shared = 0; |
| 1249 | 1253 | struct page *page = NULL; |
| 1254 | + struct folio *folio = NULL; | |
| 1250 | 1255 | unsigned long _address; |
| 1251 | 1256 | spinlock_t *ptl; |
| 1252 | 1257 | int node = NUMA_NO_NODE, unmapped = 0; |
| @@ -1333,29 +1338,28 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 80 unmodified lines | ||
| 1333 | 1338 | } |
| 1334 | 1339 | } |
| 1335 | 1340 | |
| 1336 | − page = compound_head(page); | |
| 1337 | − | |
| 1341 | + folio = page_folio(page); | |
| 1338 | 1342 | /* |
| 1339 | 1343 | * Record which node the original page is from and save this |
| 1340 | 1344 | * information to cc->node_load[]. |
| 1341 | 1345 | * Khugepaged will allocate hugepage from the node has the max |
| 1342 | 1346 | * hit record. |
| 1343 | 1347 | */ |
| 1344 | − node = page_to_nid(page); | |
| 1348 | + node = folio_nid(folio); | |
| 1345 | 1349 | if (hpage_collapse_scan_abort(node, cc)) { |
| 1346 | 1350 | result = SCAN_SCAN_ABORT; |
| 1347 | 1351 | goto out_unmap; |
| 1348 | 1352 | } |
| 1349 | 1353 | cc->node_load[node]++; |
| 1350 | − if (!PageLRU(page)) { | |
| 1354 | + if (!folio_test_lru(folio)) { | |
| 1351 | 1355 | result = SCAN_PAGE_LRU; |
| 1352 | 1356 | goto out_unmap; |
| 1353 | 1357 | } |
| 1354 | − if (PageLocked(page)) { | |
| 1358 | + if (folio_test_locked(folio)) { | |
| 1355 | 1359 | result = SCAN_PAGE_LOCK; |
| 1356 | 1360 | goto out_unmap; |
| 1357 | 1361 | } |
| 1358 | − if (!PageAnon(page)) { | |
| 1362 | + if (!folio_test_anon(folio)) { | |
| 1359 | 1363 | result = SCAN_PAGE_ANON; |
| 1360 | 1364 | goto out_unmap; |
| 1361 | 1365 | } |
| @@ -1370,7 +1374,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 8 unmodified lines | ||
| 1370 | 1374 | * has excessive GUP pins (i.e. 512). Anyway the same check |
| 1371 | 1375 | * will be done again later the risk seems low. |
| 1372 | 1376 | */ |
| 1373 | − if (!is_refcount_suitable(page)) { | |
| 1377 | + if (!is_refcount_suitable(folio)) { | |
| 1374 | 1378 | result = SCAN_PAGE_COUNT; |
| 1375 | 1379 | goto out_unmap; |
| 1376 | 1380 | } |
| @@ -1380,8 +1384,8 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, · 3 unmodified lines | ||
| 1380 | 1384 | * enough young pte to justify collapsing the page |
| 1381 | 1385 | */ |
| 1382 | 1386 | if (cc->is_khugepaged && |
| 1383 | − (pte_young(pteval) || page_is_young(page) || | |
| 1384 | − PageReferenced(page) || mmu_notifier_test_young(vma->vm_mm, | |
| 1387 | + (pte_young(pteval) || folio_test_young(folio) || | |
| 1388 | + folio_test_referenced(folio) || mmu_notifier_test_young(vma->vm_mm, | |
| 1385 | 1389 | address))) |
| 1386 | 1390 | referenced++; |
| 1387 | 1391 | } |
| @@ -1403,7 +1407,7 @@ out_unmap: · 15 unmodified lines | ||
| 1403 | 1407 | *mmap_locked = false; |
| 1404 | 1408 | } |
| 1405 | 1409 | out: |
| 1406 | − trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced, | |
| 1410 | + trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced, | |
| 1407 | 1411 | none_or_zero, result, unmapped); |
| 1408 | 1412 | return result; |
| 1409 | 1413 | } |
| @@ -1473,7 +1477,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 63 unmodified lines | ||
| 1473 | 1477 | bool notified = false; |
| 1474 | 1478 | unsigned long haddr = addr & HPAGE_PMD_MASK; |
| 1475 | 1479 | struct vm_area_struct *vma = vma_lookup(mm, haddr); |
| 1476 | − struct page *hpage; | |
| 1480 | + struct folio *folio; | |
| 1477 | 1481 | pte_t *start_pte, *pte; |
| 1478 | 1482 | pmd_t *pmd, pgt_pmd; |
| 1479 | 1483 | spinlock_t *pml = NULL, *ptl; |
| @@ -1506,19 +1510,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 26 unmodified lines | ||
| 1506 | 1510 | if (userfaultfd_wp(vma)) |
| 1507 | 1511 | return SCAN_PTE_UFFD_WP; |
| 1508 | 1512 | |
| 1509 | − hpage = find_lock_page(vma->vm_file->f_mapping, | |
| 1513 | + folio = filemap_lock_folio(vma->vm_file->f_mapping, | |
| 1510 | 1514 | linear_page_index(vma, haddr)); |
| 1511 | − if (!hpage) | |
| 1515 | + if (IS_ERR(folio)) | |
| 1512 | 1516 | return SCAN_PAGE_NULL; |
| 1513 | 1517 | |
| 1514 | − if (!PageHead(hpage)) { | |
| 1515 | − result = SCAN_FAIL; | |
| 1516 | − goto drop_hpage; | |
| 1517 | − } | |
| 1518 | − | |
| 1519 | − if (compound_order(hpage) != HPAGE_PMD_ORDER) { | |
| 1518 | + if (folio_order(folio) != HPAGE_PMD_ORDER) { | |
| 1520 | 1519 | result = SCAN_PAGE_COMPOUND; |
| 1521 | − goto drop_hpage; | |
| 1520 | + goto drop_folio; | |
| 1522 | 1521 | } |
| 1523 | 1522 | |
| 1524 | 1523 | result = find_pmd_or_thp_or_none(mm, haddr, &pmd); |
| @@ -1532,13 +1531,13 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 7 unmodified lines | ||
| 1532 | 1531 | */ |
| 1533 | 1532 | goto maybe_install_pmd; |
| 1534 | 1533 | default: |
| 1535 | − goto drop_hpage; | |
| 1534 | + goto drop_folio; | |
| 1536 | 1535 | } |
| 1537 | 1536 | |
| 1538 | 1537 | result = SCAN_FAIL; |
| 1539 | 1538 | start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); |
| 1540 | 1539 | if (!start_pte) /* mmap_lock + page lock should prevent this */ |
| 1541 | − goto drop_hpage; | |
| 1540 | + goto drop_folio; | |
| 1542 | 1541 | |
| 1543 | 1542 | /* step 1: check all mapped PTEs are to the right huge page */ |
| 1544 | 1543 | for (i = 0, addr = haddr, pte = start_pte; |
| @@ -1563,7 +1562,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 18 unmodified lines | ||
| 1563 | 1562 | * Note that uprobe, debugger, or MAP_PRIVATE may change the |
| 1564 | 1563 | * page table, but the new page will not be a subpage of hpage. |
| 1565 | 1564 | */ |
| 1566 | − if (hpage + i != page) | |
| 1565 | + if (folio_page(folio, i) != page) | |
| 1567 | 1566 | goto abort; |
| 1568 | 1567 | } |
| 1569 | 1568 | |
| @@ -1578,7 +1577,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 8 unmodified lines | ||
| 1578 | 1577 | * page_table_lock) ptl nests inside pml. The less time we hold pml, |
| 1579 | 1578 | * the better; but userfaultfd's mfill_atomic_pte() on a private VMA |
| 1580 | 1579 | * inserts a valid as-if-COWed PTE without even looking up page cache. |
| 1581 | − * So page lock of hpage does not protect from it, so we must not drop | |
| 1580 | + * So page lock of folio does not protect from it, so we must not drop | |
| 1582 | 1581 | * ptl before pgt_pmd is removed, so uffd private needs pml taken now. |
| 1583 | 1582 | */ |
| 1584 | 1583 | if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) |
| @@ -1602,7 +1601,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 17 unmodified lines | ||
| 1602 | 1601 | continue; |
| 1603 | 1602 | /* |
| 1604 | 1603 | * We dropped ptl after the first scan, to do the mmu_notifier: |
| 1605 | − * page lock stops more PTEs of the hpage being faulted in, but | |
| 1604 | + * page lock stops more PTEs of the folio being faulted in, but | |
| 1606 | 1605 | * does not stop write faults COWing anon copies from existing |
| 1607 | 1606 | * PTEs; and does not stop those being swapped out or migrated. |
| 1608 | 1607 | */ |
| @@ -1611,7 +1610,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 2 unmodified lines | ||
| 1611 | 1610 | goto abort; |
| 1612 | 1611 | } |
| 1613 | 1612 | page = vm_normal_page(vma, addr, ptent); |
| 1614 | − if (hpage + i != page) | |
| 1613 | + if (folio_page(folio, i) != page) | |
| 1615 | 1614 | goto abort; |
| 1616 | 1615 | |
| 1617 | 1616 | /* |
| @@ -1630,8 +1629,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 12 unmodified lines | ||
| 1630 | 1629 | |
| 1631 | 1630 | /* step 3: set proper refcount and mm_counters. */ |
| 1632 | 1631 | if (nr_ptes) { |
| 1633 | − page_ref_sub(hpage, nr_ptes); | |
| 1634 | − add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); | |
| 1632 | + folio_ref_sub(folio, nr_ptes); | |
| 1633 | + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); | |
| 1635 | 1634 | } |
| 1636 | 1635 | |
| 1637 | 1636 | /* step 4: remove empty page table */ |
| @@ -1655,14 +1654,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, · 17 unmodified lines | ||
| 1655 | 1654 | maybe_install_pmd: |
| 1656 | 1655 | /* step 5: install pmd entry */ |
| 1657 | 1656 | result = install_pmd |
| 1658 | − ? set_huge_pmd(vma, haddr, pmd, hpage) | |
| 1657 | + ? set_huge_pmd(vma, haddr, pmd, &folio->page) | |
| 1659 | 1658 | : SCAN_SUCCEED; |
| 1660 | − goto drop_hpage; | |
| 1659 | + goto drop_folio; | |
| 1661 | 1660 | abort: |
| 1662 | 1661 | if (nr_ptes) { |
| 1663 | 1662 | flush_tlb_mm(mm); |
| 1664 | − page_ref_sub(hpage, nr_ptes); | |
| 1665 | − add_mm_counter(mm, mm_counter_file(hpage), -nr_ptes); | |
| 1663 | + folio_ref_sub(folio, nr_ptes); | |
| 1664 | + add_mm_counter(mm, mm_counter_file(&folio->page), -nr_ptes); | |
| 1666 | 1665 | } |
| 1667 | 1666 | if (start_pte) |
| 1668 | 1667 | pte_unmap_unlock(start_pte, ptl); |
| @@ -1670,9 +1669,9 @@ abort: · 1 unmodified line | ||
| 1670 | 1669 | spin_unlock(pml); |
| 1671 | 1670 | if (notified) |
| 1672 | 1671 | mmu_notifier_invalidate_range_end(&range); |
| 1673 | −drop_hpage: | |
| 1674 | − unlock_page(hpage); | |
| 1675 | − put_page(hpage); | |
| 1672 | +drop_folio: | |
| 1673 | + folio_unlock(folio); | |
| 1674 | + folio_put(folio); | |
| 1676 | 1675 | return result; |
| 1677 | 1676 | } |
| 1678 | 1677 | |
| @@ -583,6 +583,19 @@ static void __remove_object(struct kmemleak_object *object) · 582 unmodified lines | ||
| 583 | 583 | object->del_state |= DELSTATE_REMOVED; |
| 584 | 584 | } |
| 585 | 585 | |
| 586 | +static struct kmemleak_object *__find_and_remove_object(unsigned long ptr, | |
| 587 | + int alias, | |
| 588 | + bool is_phys) | |
| 589 | +{ | |
| 590 | + struct kmemleak_object *object; | |
| 591 | + | |
| 592 | + object = __lookup_object(ptr, alias, is_phys); | |
| 593 | + if (object) | |
| 594 | + __remove_object(object); | |
| 595 | + | |
| 596 | + return object; | |
| 597 | +} | |
| 598 | + | |
| 586 | 599 | /* |
| 587 | 600 | * Look up an object in the object search tree and remove it from both |
| 588 | 601 | * object_tree_root (or object_phys_tree_root) and object_list. The |
| @@ -596,9 +609,7 @@ static struct kmemleak_object *find_and_remove_object(unsigned long ptr, int ali · 7 unmodified lines | ||
| 596 | 609 | struct kmemleak_object *object; |
| 597 | 610 | |
| 598 | 611 | raw_spin_lock_irqsave(&kmemleak_lock, flags); |
| 599 | − object = __lookup_object(ptr, alias, is_phys); | |
| 600 | − if (object) | |
| 601 | − __remove_object(object); | |
| 612 | + object = __find_and_remove_object(ptr, alias, is_phys); | |
| 602 | 613 | raw_spin_unlock_irqrestore(&kmemleak_lock, flags); |
| 603 | 614 | |
| 604 | 615 | return object; |
| @@ -623,27 +634,28 @@ static noinline depot_stack_handle_t set_track_prepare(void) · 18 unmodified lines | ||
| 623 | 634 | return trace_handle; |
| 624 | 635 | } |
| 625 | 636 | |
| 626 | −/* | |
| 627 | − * Create the metadata (struct kmemleak_object) corresponding to an allocated | |
| 628 | − * memory block and add it to the object_list and object_tree_root (or | |
| 629 | − * object_phys_tree_root). | |
| 630 | − */ | |
| 631 | −static void __create_object(unsigned long ptr, size_t size, | |
| 632 | − int min_count, gfp_t gfp, bool is_phys) | |
| 637 | +static struct kmemleak_object *__alloc_object(gfp_t gfp) | |
| 633 | 638 | { |
| 634 | − unsigned long flags; | |
| 635 | − struct kmemleak_object *object, *parent; | |
| 636 | − struct rb_node **link, *rb_parent; | |
| 637 | − unsigned long untagged_ptr; | |
| 638 | − unsigned long untagged_objp; | |
| 639 | + struct kmemleak_object *object; | |
| 639 | 640 | |
| 640 | 641 | object = mem_pool_alloc(gfp); |
| 641 | 642 | if (!object) { |
| 642 | 643 | pr_warn("Cannot allocate a kmemleak_object structure\n"); |
| 643 | 644 | kmemleak_disable(); |
| 644 | − return; | |
| 645 | 645 | } |
| 646 | 646 | |
| 647 | + return object; | |
| 648 | +} | |
| 649 | + | |
| 650 | +static int __link_object(struct kmemleak_object *object, unsigned long ptr, | |
| 651 | + size_t size, int min_count, bool is_phys) | |
| 652 | +{ | |
| 653 | + | |
| 654 | + struct kmemleak_object *parent; | |
| 655 | + struct rb_node **link, *rb_parent; | |
| 656 | + unsigned long untagged_ptr; | |
| 657 | + unsigned long untagged_objp; | |
| 658 | + | |
| 647 | 659 | INIT_LIST_HEAD(&object->object_list); |
| 648 | 660 | INIT_LIST_HEAD(&object->gray_list); |
| 649 | 661 | INIT_HLIST_HEAD(&object->area_list); |
| @@ -680,8 +692,6 @@ static void __create_object(unsigned long ptr, size_t size, · 30 unmodified lines | ||
| 680 | 692 | /* kernel backtrace */ |
| 681 | 693 | object->trace_handle = set_track_prepare(); |
| 682 | 694 | |
| 683 | − raw_spin_lock_irqsave(&kmemleak_lock, flags); | |
| 684 | − | |
| 685 | 695 | untagged_ptr = (unsigned long)kasan_reset_tag((void *)ptr); |
| 686 | 696 | /* |
| 687 | 697 | * Only update min_addr and max_addr with object |
| @@ -710,16 +720,38 @@ static void __create_object(unsigned long ptr, size_t size, · 22 unmodified lines | ||
| 710 | 720 | * be freed while the kmemleak_lock is held. |
| 711 | 721 | */ |
| 712 | 722 | dump_object_info(parent); |
| 713 | − kmem_cache_free(object_cache, object); | |
| 714 | − goto out; | |
| 723 | + return -EEXIST; | |
| 715 | 724 | } |
| 716 | 725 | } |
| 717 | 726 | rb_link_node(&object->rb_node, rb_parent, link); |
| 718 | 727 | rb_insert_color(&object->rb_node, is_phys ? &object_phys_tree_root : |
| 719 | 728 | &object_tree_root); |
| 720 | 729 | list_add_tail_rcu(&object->object_list, &object_list); |
| 721 | −out: | |
| 730 | + | |
| 731 | + return 0; | |
| 732 | +} | |
| 733 | + | |
| 734 | +/* | |
| 735 | + * Create the metadata (struct kmemleak_object) corresponding to an allocated | |
| 736 | + * memory block and add it to the object_list and object_tree_root (or | |
| 737 | + * object_phys_tree_root). | |
| 738 | + */ | |
| 739 | +static void __create_object(unsigned long ptr, size_t size, | |
| 740 | + int min_count, gfp_t gfp, bool is_phys) | |
| 741 | +{ | |
| 742 | + struct kmemleak_object *object; | |
| 743 | + unsigned long flags; | |
| 744 | + int ret; | |
| 745 | + | |
| 746 | + object = __alloc_object(gfp); | |
| 747 | + if (!object) | |
| 748 | + return; | |
| 749 | + | |
| 750 | + raw_spin_lock_irqsave(&kmemleak_lock, flags); | |
| 751 | + ret = __link_object(object, ptr, size, min_count, is_phys); | |
| 722 | 752 | raw_spin_unlock_irqrestore(&kmemleak_lock, flags); |
| 753 | + if (ret) | |
| 754 | + mem_pool_free(object); | |
| 723 | 755 | } |
| 724 | 756 | |
| 725 | 757 | /* Create kmemleak object which allocated with virtual address. */ |
| @@ -782,16 +814,25 @@ static void delete_object_full(unsigned long ptr) · 56 unmodified lines | ||
| 782 | 814 | */ |
| 783 | 815 | static void delete_object_part(unsigned long ptr, size_t size, bool is_phys) |
| 784 | 816 | { |
| 785 | − struct kmemleak_object *object; | |
| 786 | − unsigned long start, end; | |
| 817 | + struct kmemleak_object *object, *object_l, *object_r; | |
| 818 | + unsigned long start, end, flags; | |
| 819 | + | |
| 820 | + object_l = __alloc_object(GFP_KERNEL); | |
| 821 | + if (!object_l) | |
| 822 | + return; | |
| 787 | 823 | |
| 788 | − object = find_and_remove_object(ptr, 1, is_phys); | |
| 824 | + object_r = __alloc_object(GFP_KERNEL); | |
| 825 | + if (!object_r) | |
| 826 | + goto out; | |
| 827 | + | |
| 828 | + raw_spin_lock_irqsave(&kmemleak_lock, flags); | |
| 829 | + object = __find_and_remove_object(ptr, 1, is_phys); | |
| 789 | 830 | if (!object) { |
| 790 | 831 | #ifdef DEBUG |
| 791 | 832 | kmemleak_warn("Partially freeing unknown object at 0x%08lx (size %zu)\n", |
| 792 | 833 | ptr, size); |
| 793 | 834 | #endif |
| 794 | − return; | |
| 835 | + goto unlock; | |
| 795 | 836 | } |
| 796 | 837 | |
| 797 | 838 | /* |
| @@ -801,14 +842,25 @@ static void delete_object_part(unsigned long ptr, size_t size, bool is_phys) · 3 unmodified lines | ||
| 801 | 842 | */ |
| 802 | 843 | start = object->pointer; |
| 803 | 844 | end = object->pointer + object->size; |
| 804 | − if (ptr > start) | |
| 805 | − __create_object(start, ptr - start, object->min_count, | |
| 806 | − GFP_KERNEL, is_phys); | |
| 807 | − if (ptr + size < end) | |
| 808 | − __create_object(ptr + size, end - ptr - size, object->min_count, | |
| 809 | − GFP_KERNEL, is_phys); | |
| 845 | + if ((ptr > start) && | |
| 846 | + !__link_object(object_l, start, ptr - start, | |
| 847 | + object->min_count, is_phys)) | |
| 848 | + object_l = NULL; | |
| 849 | + if ((ptr + size < end) && | |
| 850 | + !__link_object(object_r, ptr + size, end - ptr - size, | |
| 851 | + object->min_count, is_phys)) | |
| 852 | + object_r = NULL; | |
| 853 | + | |
| 854 | +unlock: | |
| 855 | + raw_spin_unlock_irqrestore(&kmemleak_lock, flags); | |
| 856 | + if (object) | |
| 857 | + __delete_object(object); | |
| 810 | 858 | |
| 811 | − __delete_object(object); | |
| 859 | +out: | |
| 860 | + if (object_l) | |
| 861 | + mem_pool_free(object_l); | |
| 862 | + if (object_r) | |
| 863 | + mem_pool_free(object_r); | |
| 812 | 864 | } |
| 813 | 865 | |
| 814 | 866 | static void __paint_it(struct kmemleak_object *object, int color) |
| @@ -975,7 +1027,7 @@ static void object_no_scan(unsigned long ptr) · 160 unmodified lines | ||
| 975 | 1027 | void __ref kmemleak_alloc(const void *ptr, size_t size, int min_count, |
| 976 | 1028 | gfp_t gfp) |
| 977 | 1029 | { |
| 978 | − pr_debug("%s(0x%p, %zu, %d)\n", __func__, ptr, size, min_count); | |
| 1030 | + pr_debug("%s(0x%px, %zu, %d)\n", __func__, ptr, size, min_count); | |
| 979 | 1031 | |
| 980 | 1032 | if (kmemleak_enabled && ptr && !IS_ERR(ptr)) |
| 981 | 1033 | create_object((unsigned long)ptr, size, min_count, gfp); |
| @@ -996,7 +1048,7 @@ void __ref kmemleak_alloc_percpu(const void __percpu *ptr, size_t size, · 14 unmodified lines | ||
| 996 | 1048 | { |
| 997 | 1049 | unsigned int cpu; |
| 998 | 1050 | |
| 999 | − pr_debug("%s(0x%p, %zu)\n", __func__, ptr, size); | |
| 1051 | + pr_debug("%s(0x%px, %zu)\n", __func__, ptr, size); | |
| 1000 | 1052 | |
| 1001 | 1053 | /* |
| 1002 | 1054 | * Percpu allocations are only scanned and not reported as leaks |
| @@ -1020,7 +1072,7 @@ EXPORT_SYMBOL_GPL(kmemleak_alloc_percpu); · 17 unmodified lines | ||
| 1020 | 1072 | */ |
| 1021 | 1073 | void __ref kmemleak_vmalloc(const struct vm_struct *area, size_t size, gfp_t gfp) |
| 1022 | 1074 | { |
| 1023 | − pr_debug("%s(0x%p, %zu)\n", __func__, area, size); | |
| 1075 | + pr_debug("%s(0x%px, %zu)\n", __func__, area, size); | |
| 1024 | 1076 | |
| 1025 | 1077 | /* |
| 1026 | 1078 | * A min_count = 2 is needed because vm_struct contains a reference to |
| @@ -1043,7 +1095,7 @@ EXPORT_SYMBOL_GPL(kmemleak_vmalloc); · 16 unmodified lines | ||
| 1043 | 1095 | */ |
| 1044 | 1096 | void __ref kmemleak_free(const void *ptr) |
| 1045 | 1097 | { |
| 1046 | − pr_debug("%s(0x%p)\n", __func__, ptr); | |
| 1098 | + pr_debug("%s(0x%px)\n", __func__, ptr); | |
| 1047 | 1099 | |
| 1048 | 1100 | if (kmemleak_free_enabled && ptr && !IS_ERR(ptr)) |
| 1049 | 1101 | delete_object_full((unsigned long)ptr); |
| @@ -1061,7 +1113,7 @@ EXPORT_SYMBOL_GPL(kmemleak_free); · 11 unmodified lines | ||
| 1061 | 1113 | */ |
| 1062 | 1114 | void __ref kmemleak_free_part(const void *ptr, size_t size) |
| 1063 | 1115 | { |
| 1064 | − pr_debug("%s(0x%p)\n", __func__, ptr); | |
| 1116 | + pr_debug("%s(0x%px)\n", __func__, ptr); | |
| 1065 | 1117 | |
| 1066 | 1118 | if (kmemleak_enabled && ptr && !IS_ERR(ptr)) |
| 1067 | 1119 | delete_object_part((unsigned long)ptr, size, false); |
| @@ -1079,7 +1131,7 @@ void __ref kmemleak_free_percpu(const void __percpu *ptr) · 11 unmodified lines | ||
| 1079 | 1131 | { |
| 1080 | 1132 | unsigned int cpu; |
| 1081 | 1133 | |
| 1082 | − pr_debug("%s(0x%p)\n", __func__, ptr); | |
| 1134 | + pr_debug("%s(0x%px)\n", __func__, ptr); | |
| 1083 | 1135 | |
| 1084 | 1136 | if (kmemleak_free_enabled && ptr && !IS_ERR(ptr)) |
| 1085 | 1137 | for_each_possible_cpu(cpu) |
| @@ -1100,7 +1152,7 @@ void __ref kmemleak_update_trace(const void *ptr) · 14 unmodified lines | ||
| 1100 | 1152 | struct kmemleak_object *object; |
| 1101 | 1153 | unsigned long flags; |
| 1102 | 1154 | |
| 1103 | − pr_debug("%s(0x%p)\n", __func__, ptr); | |
| 1155 | + pr_debug("%s(0x%px)\n", __func__, ptr); | |
| 1104 | 1156 | |
| 1105 | 1157 | if (!kmemleak_enabled || IS_ERR_OR_NULL(ptr)) |
| 1106 | 1158 | return; |
| @@ -1131,7 +1183,7 @@ EXPORT_SYMBOL(kmemleak_update_trace); · 24 unmodified lines | ||
| 1131 | 1183 | */ |
| 1132 | 1184 | void __ref kmemleak_not_leak(const void *ptr) |
| 1133 | 1185 | { |
| 1134 | − pr_debug("%s(0x%p)\n", __func__, ptr); | |
| 1186 | + pr_debug("%s(0x%px)\n", __func__, ptr); | |
| 1135 | 1187 | |
| 1136 | 1188 | if (kmemleak_enabled && ptr && !IS_ERR(ptr)) |
| 1137 | 1189 | make_gray_object((unsigned long)ptr); |
| @@ -1149,7 +1201,7 @@ EXPORT_SYMBOL(kmemleak_not_leak); · 11 unmodified lines | ||
| 1149 | 1201 | */ |
| 1150 | 1202 | void __ref kmemleak_ignore(const void *ptr) |
| 1151 | 1203 | { |
| 1152 | − pr_debug("%s(0x%p)\n", __func__, ptr); | |
| 1204 | + pr_debug("%s(0x%px)\n", __func__, ptr); | |
| 1153 | 1205 | |
| 1154 | 1206 | if (kmemleak_enabled && ptr && !IS_ERR(ptr)) |
| 1155 | 1207 | make_black_object((unsigned long)ptr, false); |
| @@ -1169,7 +1221,7 @@ EXPORT_SYMBOL(kmemleak_ignore); · 13 unmodified lines | ||
| 1169 | 1221 | */ |
| 1170 | 1222 | void __ref kmemleak_scan_area(const void *ptr, size_t size, gfp_t gfp) |
| 1171 | 1223 | { |
| 1172 | − pr_debug("%s(0x%p)\n", __func__, ptr); | |
| 1224 | + pr_debug("%s(0x%px)\n", __func__, ptr); | |
| 1173 | 1225 | |
| 1174 | 1226 | if (kmemleak_enabled && ptr && size && !IS_ERR(ptr)) |
| 1175 | 1227 | add_scan_area((unsigned long)ptr, size, gfp); |
| @@ -1187,7 +1239,7 @@ EXPORT_SYMBOL(kmemleak_scan_area); · 11 unmodified lines | ||
| 1187 | 1239 | */ |
| 1188 | 1240 | void __ref kmemleak_no_scan(const void *ptr) |
| 1189 | 1241 | { |
| 1190 | − pr_debug("%s(0x%p)\n", __func__, ptr); | |
| 1242 | + pr_debug("%s(0x%px)\n", __func__, ptr); | |
| 1191 | 1243 | |
| 1192 | 1244 | if (kmemleak_enabled && ptr && !IS_ERR(ptr)) |
| 1193 | 1245 | object_no_scan((unsigned long)ptr); |
| @@ -1203,7 +1255,7 @@ EXPORT_SYMBOL(kmemleak_no_scan); · 9 unmodified lines | ||
| 1203 | 1255 | */ |
| 1204 | 1256 | void __ref kmemleak_alloc_phys(phys_addr_t phys, size_t size, gfp_t gfp) |
| 1205 | 1257 | { |
| 1206 | − pr_debug("%s(0x%pa, %zu)\n", __func__, &phys, size); | |
| 1258 | + pr_debug("%s(0x%px, %zu)\n", __func__, &phys, size); | |
| 1207 | 1259 | |
| 1208 | 1260 | if (kmemleak_enabled) |
| 1209 | 1261 | /* |
| @@ -1223,7 +1275,7 @@ EXPORT_SYMBOL(kmemleak_alloc_phys); · 13 unmodified lines | ||
| 1223 | 1275 | */ |
| 1224 | 1276 | void __ref kmemleak_free_part_phys(phys_addr_t phys, size_t size) |
| 1225 | 1277 | { |
| 1226 | − pr_debug("%s(0x%pa)\n", __func__, &phys); | |
| 1278 | + pr_debug("%s(0x%px)\n", __func__, &phys); | |
| 1227 | 1279 | |
| 1228 | 1280 | if (kmemleak_enabled) |
| 1229 | 1281 | delete_object_part((unsigned long)phys, size, true); |
| @@ -1237,7 +1289,7 @@ EXPORT_SYMBOL(kmemleak_free_part_phys); · 7 unmodified lines | ||
| 1237 | 1289 | */ |
| 1238 | 1290 | void __ref kmemleak_ignore_phys(phys_addr_t phys) |
| 1239 | 1291 | { |
| 1240 | − pr_debug("%s(0x%pa)\n", __func__, &phys); | |
| 1292 | + pr_debug("%s(0x%px)\n", __func__, &phys); | |
| 1241 | 1293 | |
| 1242 | 1294 | if (kmemleak_enabled) |
| 1243 | 1295 | make_black_object((unsigned long)phys, true); |
| @@ -83,131 +83,66 @@ depot_stack_handle_t kmsan_save_stack_with_flags(gfp_t flags, · 82 unmodified lines | ||
| 83 | 83 | /* Copy the metadata following the memmove() behavior. */ |
| 84 | 84 | void kmsan_internal_memmove_metadata(void *dst, void *src, size_t n) |
| 85 | 85 | { |
| 86 | + depot_stack_handle_t prev_old_origin = 0, prev_new_origin = 0; | |
| 87 | + int i, iter, step, src_off, dst_off, oiter_src, oiter_dst; | |
| 86 | 88 | depot_stack_handle_t old_origin = 0, new_origin = 0; |
| 87 | − int src_slots, dst_slots, i, iter, step, skip_bits; | |
| 88 | 89 | depot_stack_handle_t *origin_src, *origin_dst; |
| 89 | − void *shadow_src, *shadow_dst; | |
| 90 | − u32 *align_shadow_src, shadow; | |
| 90 | + u8 *shadow_src, *shadow_dst; | |
| 91 | + u32 *align_shadow_dst; | |
| 91 | 92 | bool backwards; |
| 92 | 93 | |
| 93 | 94 | shadow_dst = kmsan_get_metadata(dst, KMSAN_META_SHADOW); |
| 94 | 95 | if (!shadow_dst) |
| 95 | 96 | return; |
| 96 | 97 | KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(dst, n)); |
| 98 | + align_shadow_dst = | |
| 99 | + (u32 *)ALIGN_DOWN((u64)shadow_dst, KMSAN_ORIGIN_SIZE); | |
| 97 | 100 | |
| 98 | 101 | shadow_src = kmsan_get_metadata(src, KMSAN_META_SHADOW); |
| 99 | 102 | if (!shadow_src) { |
| 100 | − /* | |
| 101 | − * @src is untracked: zero out destination shadow, ignore the | |
| 102 | − * origins, we're done. | |
| 103 | − */ | |
| 104 | − __memset(shadow_dst, 0, n); | |
| 103 | + /* @src is untracked: mark @dst as initialized. */ | |
| 104 | + kmsan_internal_unpoison_memory(dst, n, /*checked*/ false); | |
| 105 | 105 | return; |
| 106 | 106 | } |
| 107 | 107 | KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(src, n)); |
| 108 | 108 | |
| 109 | − __memmove(shadow_dst, shadow_src, n); | |
| 110 | − | |
| 111 | 109 | origin_dst = kmsan_get_metadata(dst, KMSAN_META_ORIGIN); |
| 112 | 110 | origin_src = kmsan_get_metadata(src, KMSAN_META_ORIGIN); |
| 113 | 111 | KMSAN_WARN_ON(!origin_dst || !origin_src); |
| 114 | − src_slots = (ALIGN((u64)src + n, KMSAN_ORIGIN_SIZE) - | |
| 115 | − ALIGN_DOWN((u64)src, KMSAN_ORIGIN_SIZE)) / | |
| 116 | − KMSAN_ORIGIN_SIZE; | |
| 117 | − dst_slots = (ALIGN((u64)dst + n, KMSAN_ORIGIN_SIZE) - | |
| 118 | − ALIGN_DOWN((u64)dst, KMSAN_ORIGIN_SIZE)) / | |
| 119 | − KMSAN_ORIGIN_SIZE; | |
| 120 | − KMSAN_WARN_ON((src_slots < 1) || (dst_slots < 1)); | |
| 121 | − KMSAN_WARN_ON((src_slots - dst_slots > 1) || | |
| 122 | − (dst_slots - src_slots < -1)); | |
| 123 | 112 | |
| 124 | 113 | backwards = dst > src; |
| 125 | − i = backwards ? min(src_slots, dst_slots) - 1 : 0; | |
| 126 | − iter = backwards ? -1 : 1; | |
| 127 | − | |
| 128 | − align_shadow_src = | |
| 129 | − (u32 *)ALIGN_DOWN((u64)shadow_src, KMSAN_ORIGIN_SIZE); | |
| 130 | − for (step = 0; step < min(src_slots, dst_slots); step++, i += iter) { | |
| 131 | − KMSAN_WARN_ON(i < 0); | |
| 132 | − shadow = align_shadow_src[i]; | |
| 133 | − if (i == 0) { | |
| 134 | − /* | |
| 135 | − * If @src isn't aligned on KMSAN_ORIGIN_SIZE, don't | |
| 136 | − * look at the first @src % KMSAN_ORIGIN_SIZE bytes | |
| 137 | − * of the first shadow slot. | |
| 138 | − */ | |
| 139 | − skip_bits = ((u64)src % KMSAN_ORIGIN_SIZE) * 8; | |
| 140 | − shadow = (shadow >> skip_bits) << skip_bits; | |
| 114 | + step = backwards ? -1 : 1; | |
| 115 | + iter = backwards ? n - 1 : 0; | |
| 116 | + src_off = (u64)src % KMSAN_ORIGIN_SIZE; | |
| 117 | + dst_off = (u64)dst % KMSAN_ORIGIN_SIZE; | |
| 118 | + | |
| 119 | + /* Copy shadow bytes one by one, updating the origins if necessary. */ | |
| 120 | + for (i = 0; i < n; i++, iter += step) { | |
| 121 | + oiter_src = (iter + src_off) / KMSAN_ORIGIN_SIZE; | |
| 122 | + oiter_dst = (iter + dst_off) / KMSAN_ORIGIN_SIZE; | |
| 123 | + if (!shadow_src[iter]) { | |
| 124 | + shadow_dst[iter] = 0; | |
| 125 | + if (!align_shadow_dst[oiter_dst]) | |
| 126 | + origin_dst[oiter_dst] = 0; | |
| 127 | + continue; | |
| 141 | 128 | } |
| 142 | − if (i == src_slots - 1) { | |
| 143 | − /* | |
| 144 | − * If @src + n isn't aligned on | |
| 145 | − * KMSAN_ORIGIN_SIZE, don't look at the last | |
| 146 | − * (@src + n) % KMSAN_ORIGIN_SIZE bytes of the | |
| 147 | − * last shadow slot. | |
| 148 | − */ | |
| 149 | − skip_bits = (((u64)src + n) % KMSAN_ORIGIN_SIZE) * 8; | |
| 150 | − shadow = (shadow << skip_bits) >> skip_bits; | |
| 151 | − } | |
| 152 | − /* | |
| 153 | − * Overwrite the origin only if the corresponding | |
| 154 | − * shadow is nonempty. | |
| 155 | − */ | |
| 156 | − if (origin_src[i] && (origin_src[i] != old_origin) && shadow) { | |
| 157 | − old_origin = origin_src[i]; | |
| 158 | − new_origin = kmsan_internal_chain_origin(old_origin); | |
| 129 | + shadow_dst[iter] = shadow_src[iter]; | |
| 130 | + old_origin = origin_src[oiter_src]; | |
| 131 | + if (old_origin == prev_old_origin) | |
| 132 | + new_origin = prev_new_origin; | |
| 133 | + else { | |
| 159 | 134 | /* |
| 160 | 135 | * kmsan_internal_chain_origin() may return |
| 161 | 136 | * NULL, but we don't want to lose the previous |
| 162 | 137 | * origin value. |
| 163 | 138 | */ |
| 139 | + new_origin = kmsan_internal_chain_origin(old_origin); | |
| 164 | 140 | if (!new_origin) |
| 165 | 141 | new_origin = old_origin; |
| 166 | 142 | } |
| 167 | − if (shadow) | |
| 168 | − origin_dst[i] = new_origin; | |
| 169 | − else | |
| 170 | − origin_dst[i] = 0; | |
| 171 | − } | |
| 172 | − /* | |
| 173 | − * If dst_slots is greater than src_slots (i.e. | |
| 174 | − * dst_slots == src_slots + 1), there is an extra origin slot at the | |
| 175 | − * beginning or end of the destination buffer, for which we take the | |
| 176 | − * origin from the previous slot. | |
| 177 | − * This is only done if the part of the source shadow corresponding to | |
| 178 | − * slot is non-zero. | |
| 179 | − * | |
| 180 | − * E.g. if we copy 8 aligned bytes that are marked as uninitialized | |
| 181 | − * and have origins o111 and o222, to an unaligned buffer with offset 1, | |
| 182 | − * these two origins are copied to three origin slots, so one of then | |
| 183 | − * needs to be duplicated, depending on the copy direction (@backwards) | |
| 184 | − * | |
| 185 | − * src shadow: |uuuu|uuuu|....| | |
| 186 | − * src origin: |o111|o222|....| | |
| 187 | − * | |
| 188 | − * backwards = 0: | |
| 189 | − * dst shadow: |.uuu|uuuu|u...| | |
| 190 | − * dst origin: |....|o111|o222| - fill the empty slot with o111 | |
| 191 | − * backwards = 1: | |
| 192 | − * dst shadow: |.uuu|uuuu|u...| | |
| 193 | − * dst origin: |o111|o222|....| - fill the empty slot with o222 | |
| 194 | − */ | |
| 195 | − if (src_slots < dst_slots) { | |
| 196 | − if (backwards) { | |
| 197 | − shadow = align_shadow_src[src_slots - 1]; | |
| 198 | − skip_bits = (((u64)dst + n) % KMSAN_ORIGIN_SIZE) * 8; | |
| 199 | − shadow = (shadow << skip_bits) >> skip_bits; | |
| 200 | − if (shadow) | |
| 201 | − /* src_slots > 0, therefore dst_slots is at least 2 */ | |
| 202 | − origin_dst[dst_slots - 1] = | |
| 203 | − origin_dst[dst_slots - 2]; | |
| 204 | − } else { | |
| 205 | − shadow = align_shadow_src[0]; | |
| 206 | − skip_bits = ((u64)dst % KMSAN_ORIGIN_SIZE) * 8; | |
| 207 | − shadow = (shadow >> skip_bits) << skip_bits; | |
| 208 | − if (shadow) | |
| 209 | − origin_dst[0] = origin_dst[1]; | |
| 210 | − } | |
| 143 | + origin_dst[oiter_dst] = new_origin; | |
| 144 | + prev_new_origin = new_origin; | |
| 145 | + prev_old_origin = old_origin; | |
| 211 | 146 | } |
| 212 | 147 | } |
| 213 | 148 | |
| @@ -67,6 +67,17 @@ static bool report_available(void) · 66 unmodified lines | ||
| 67 | 67 | return READ_ONCE(observed.available); |
| 68 | 68 | } |
| 69 | 69 | |
| 70 | +/* Reset observed.available, so that the test can trigger another report. */ | |
| 71 | +static void report_reset(void) | |
| 72 | +{ | |
| 73 | + unsigned long flags; | |
| 74 | + | |
| 75 | + spin_lock_irqsave(&observed.lock, flags); | |
| 76 | + WRITE_ONCE(observed.available, false); | |
| 77 | + observed.ignore = false; | |
| 78 | + spin_unlock_irqrestore(&observed.lock, flags); | |
| 79 | +} | |
| 80 | + | |
| 70 | 81 | /* Information we expect in a report. */ |
| 71 | 82 | struct expect_report { |
| 72 | 83 | const char *error_type; /* Error type. */ |
| @@ -407,33 +418,25 @@ static void test_printk(struct kunit *test) · 334 unmodified lines | ||
| 407 | 418 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); |
| 408 | 419 | } |
| 409 | 420 | |
| 410 | −/* | |
| 411 | − * Prevent the compiler from optimizing @var away. Without this, Clang may | |
| 412 | − * notice that @var is uninitialized and drop memcpy() calls that use it. | |
| 413 | − * | |
| 414 | − * There is OPTIMIZER_HIDE_VAR() in linux/compier.h that we cannot use here, | |
| 415 | − * because it is implemented as inline assembly receiving @var as a parameter | |
| 416 | − * and will enforce a KMSAN check. Same is true for e.g. barrier_data(var). | |
| 417 | − */ | |
| 418 | −#define DO_NOT_OPTIMIZE(var) barrier() | |
| 421 | +/* Prevent the compiler from inlining a memcpy() call. */ | |
| 422 | +static noinline void *memcpy_noinline(volatile void *dst, | |
| 423 | + const volatile void *src, size_t size) | |
| 424 | +{ | |
| 425 | + return memcpy((void *)dst, (const void *)src, size); | |
| 426 | +} | |
| 419 | 427 | |
| 420 | −/* | |
| 421 | − * Test case: ensure that memcpy() correctly copies initialized values. | |
| 422 | − * Also serves as a regression test to ensure DO_NOT_OPTIMIZE() does not cause | |
| 423 | − * extra checks. | |
| 424 | − */ | |
| 428 | +/* Test case: ensure that memcpy() correctly copies initialized values. */ | |
| 425 | 429 | static void test_init_memcpy(struct kunit *test) |
| 426 | 430 | { |
| 427 | 431 | EXPECTATION_NO_REPORT(expect); |
| 428 | − volatile int src; | |
| 429 | − volatile int dst = 0; | |
| 432 | + volatile long long src; | |
| 433 | + volatile long long dst = 0; | |
| 430 | 434 | |
| 431 | − DO_NOT_OPTIMIZE(src); | |
| 432 | 435 | src = 1; |
| 433 | 436 | kunit_info( |
| 434 | 437 | test, |
| 435 | 438 | "memcpy()ing aligned initialized src to aligned dst (no reports)\n"); |
| 436 | − memcpy((void *)&dst, (void *)&src, sizeof(src)); | |
| 439 | + memcpy_noinline((void *)&dst, (void *)&src, sizeof(src)); | |
| 437 | 440 | kmsan_check_memory((void *)&dst, sizeof(dst)); |
| 438 | 441 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); |
| 439 | 442 | } |
| @@ -451,8 +454,7 @@ static void test_memcpy_aligned_to_aligned(struct kunit *test) · 11 unmodified lines | ||
| 451 | 454 | kunit_info( |
| 452 | 455 | test, |
| 453 | 456 | "memcpy()ing aligned uninit src to aligned dst (UMR report)\n"); |
| 454 | − DO_NOT_OPTIMIZE(uninit_src); | |
| 455 | − memcpy((void *)&dst, (void *)&uninit_src, sizeof(uninit_src)); | |
| 457 | + memcpy_noinline((void *)&dst, (void *)&uninit_src, sizeof(uninit_src)); | |
| 456 | 458 | kmsan_check_memory((void *)&dst, sizeof(dst)); |
| 457 | 459 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); |
| 458 | 460 | } |
| @@ -463,7 +465,7 @@ static void test_memcpy_aligned_to_aligned(struct kunit *test) · 4 unmodified lines | ||
| 463 | 465 | * |
| 464 | 466 | * Copying aligned 4-byte value to an unaligned one leads to touching two |
| 465 | 467 | * aligned 4-byte values. This test case checks that KMSAN correctly reports an |
| 466 | − * error on the first of the two values. | |
| 468 | + * error on the mentioned two values. | |
| 467 | 469 | */ |
| 468 | 470 | static void test_memcpy_aligned_to_unaligned(struct kunit *test) |
| 469 | 471 | { |
| @@ -474,33 +476,65 @@ static void test_memcpy_aligned_to_unaligned(struct kunit *test) · 4 unmodified lines | ||
| 474 | 476 | kunit_info( |
| 475 | 477 | test, |
| 476 | 478 | "memcpy()ing aligned uninit src to unaligned dst (UMR report)\n"); |
| 477 | − DO_NOT_OPTIMIZE(uninit_src); | |
| 478 | − memcpy((void *)&dst[1], (void *)&uninit_src, sizeof(uninit_src)); | |
| 479 | + kmsan_check_memory((void *)&uninit_src, sizeof(uninit_src)); | |
| 480 | + memcpy_noinline((void *)&dst[1], (void *)&uninit_src, | |
| 481 | + sizeof(uninit_src)); | |
| 479 | 482 | kmsan_check_memory((void *)dst, 4); |
| 480 | 483 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); |
| 484 | + report_reset(); | |
| 485 | + kmsan_check_memory((void *)&dst[4], sizeof(uninit_src)); | |
| 486 | + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); | |
| 481 | 487 | } |
| 482 | 488 | |
| 483 | 489 | /* |
| 484 | − * Test case: ensure that memcpy() correctly copies uninitialized values between | |
| 485 | − * aligned `src` and unaligned `dst`. | |
| 490 | + * Test case: ensure that origin slots do not accidentally get overwritten with | |
| 491 | + * zeroes during memcpy(). | |
| 486 | 492 | * |
| 487 | − * Copying aligned 4-byte value to an unaligned one leads to touching two | |
| 488 | − * aligned 4-byte values. This test case checks that KMSAN correctly reports an | |
| 489 | − * error on the second of the two values. | |
| 493 | + * Previously, when copying memory from an aligned buffer to an unaligned one, | |
| 494 | + * if there were zero origins corresponding to zero shadow values in the source | |
| 495 | + * buffer, they could have ended up being copied to nonzero shadow values in the | |
| 496 | + * destination buffer: | |
| 497 | + * | |
| 498 | + * memcpy(0xffff888080a00000, 0xffff888080900002, 8) | |
| 499 | + * | |
| 500 | + * src (0xffff888080900002): ..xx .... xx.. | |
| 501 | + * src origins: o111 0000 o222 | |
| 502 | + * dst (0xffff888080a00000): xx.. ..xx | |
| 503 | + * dst origins: o111 0000 | |
| 504 | + * (or 0000 o222) | |
| 505 | + * | |
| 506 | + * (here . stands for an initialized byte, and x for an uninitialized one. | |
| 507 | + * | |
| 508 | + * Ensure that this does not happen anymore, and for both destination bytes | |
| 509 | + * the origin is nonzero (i.e. KMSAN reports an error). | |
| 490 | 510 | */ |
| 491 | −static void test_memcpy_aligned_to_unaligned2(struct kunit *test) | |
| 511 | +static void test_memcpy_initialized_gap(struct kunit *test) | |
| 492 | 512 | { |
| 493 | − EXPECTATION_UNINIT_VALUE_FN(expect, | |
| 494 | − "test_memcpy_aligned_to_unaligned2"); | |
| 495 | − volatile int uninit_src; | |
| 513 | + EXPECTATION_UNINIT_VALUE_FN(expect, "test_memcpy_initialized_gap"); | |
| 514 | + volatile char uninit_src[12]; | |
| 496 | 515 | volatile char dst[8] = { 0 }; |
| 497 | 516 | |
| 498 | 517 | kunit_info( |
| 499 | 518 | test, |
| 500 | − "memcpy()ing aligned uninit src to unaligned dst - part 2 (UMR report)\n"); | |
| 501 | − DO_NOT_OPTIMIZE(uninit_src); | |
| 502 | − memcpy((void *)&dst[1], (void *)&uninit_src, sizeof(uninit_src)); | |
| 503 | − kmsan_check_memory((void *)&dst[4], sizeof(uninit_src)); | |
| 519 | + "unaligned 4-byte initialized value gets a nonzero origin after memcpy() - (2 UMR reports)\n"); | |
| 520 | + | |
| 521 | + uninit_src[0] = 42; | |
| 522 | + uninit_src[1] = 42; | |
| 523 | + uninit_src[4] = 42; | |
| 524 | + uninit_src[5] = 42; | |
| 525 | + uninit_src[6] = 42; | |
| 526 | + uninit_src[7] = 42; | |
| 527 | + uninit_src[10] = 42; | |
| 528 | + uninit_src[11] = 42; | |
| 529 | + memcpy_noinline((void *)&dst[0], (void *)&uninit_src[2], 8); | |
| 530 | + | |
| 531 | + kmsan_check_memory((void *)&dst[0], 4); | |
| 532 | + KUNIT_EXPECT_TRUE(test, report_matches(&expect)); | |
| 533 | + report_reset(); | |
| 534 | + kmsan_check_memory((void *)&dst[2], 4); | |
| 535 | + KUNIT_EXPECT_FALSE(test, report_matches(&expect)); | |
| 536 | + report_reset(); | |
| 537 | + kmsan_check_memory((void *)&dst[4], 4); | |
| 504 | 538 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); |
| 505 | 539 | } |
| 506 | 540 | |
| @@ -513,7 +547,6 @@ static void test_memcpy_aligned_to_unaligned2(struct kunit *test) · 6 unmodified lines | ||
| 513 | 547 | \ |
| 514 | 548 | kunit_info(test, \ |
| 515 | 549 | "memset" #size "() should initialize memory\n"); \ |
| 516 | − DO_NOT_OPTIMIZE(uninit); \ | |
| 517 | 550 | memset##size((uint##size##_t *)&uninit, 0, 1); \ |
| 518 | 551 | kmsan_check_memory((void *)&uninit, sizeof(uninit)); \ |
| 519 | 552 | KUNIT_EXPECT_TRUE(test, report_matches(&expect)); \ |
| @@ -598,7 +631,7 @@ static struct kunit_case kmsan_test_cases[] = { · 78 unmodified lines | ||
| 598 | 631 | KUNIT_CASE(test_init_memcpy), |
| 599 | 632 | KUNIT_CASE(test_memcpy_aligned_to_aligned), |
| 600 | 633 | KUNIT_CASE(test_memcpy_aligned_to_unaligned), |
| 601 | − KUNIT_CASE(test_memcpy_aligned_to_unaligned2), | |
| 634 | + KUNIT_CASE(test_memcpy_initialized_gap), | |
| 602 | 635 | KUNIT_CASE(test_memset16), |
| 603 | 636 | KUNIT_CASE(test_memset32), |
| 604 | 637 | KUNIT_CASE(test_memset64), |
| @@ -285,12 +285,17 @@ void __init kmsan_init_alloc_meta_for_range(void *start, void *end) · 284 unmodified lines | ||
| 285 | 285 | size = PAGE_ALIGN((u64)end - (u64)start); |
| 286 | 286 | shadow = memblock_alloc(size, PAGE_SIZE); |
| 287 | 287 | origin = memblock_alloc(size, PAGE_SIZE); |
| 288 | + | |
| 289 | + if (!shadow || !origin) | |
| 290 | + panic("%s: Failed to allocate metadata memory for early boot range of size %llu", | |
| 291 | + __func__, size); | |
| 292 | + | |
| 288 | 293 | for (u64 addr = 0; addr < size; addr += PAGE_SIZE) { |
| 289 | 294 | page = virt_to_page_or_null((char *)start + addr); |
| 290 | − shadow_p = virt_to_page_or_null((char *)shadow + addr); | |
| 295 | + shadow_p = virt_to_page((char *)shadow + addr); | |
| 291 | 296 | set_no_shadow_origin_page(shadow_p); |
| 292 | 297 | shadow_page_for(page) = shadow_p; |
| 293 | − origin_p = virt_to_page_or_null((char *)origin + addr); | |
| 298 | + origin_p = virt_to_page((char *)origin + addr); | |
| 294 | 299 | set_no_shadow_origin_page(origin_p); |
| 295 | 300 | origin_page_for(page) = origin_p; |
| 296 | 301 | } |
| @@ -56,6 +56,8 @@ · 55 unmodified lines | ||
| 56 | 56 | #define DO_NUMA(x) do { } while (0) |
| 57 | 57 | #endif |
| 58 | 58 | |
| 59 | +typedef u8 rmap_age_t; | |
| 60 | + | |
| 59 | 61 | /** |
| 60 | 62 | * DOC: Overview |
| 61 | 63 | * |
| @@ -193,6 +195,8 @@ struct ksm_stable_node { · 131 unmodified lines | ||
| 193 | 195 | * @node: rb node of this rmap_item in the unstable tree |
| 194 | 196 | * @head: pointer to stable_node heading this list in the stable tree |
| 195 | 197 | * @hlist: link into hlist of rmap_items hanging off that stable_node |
| 198 | + * @age: number of scan iterations since creation | |
| 199 | + * @remaining_skips: how many scans to skip | |
| 196 | 200 | */ |
| 197 | 201 | struct ksm_rmap_item { |
| 198 | 202 | struct ksm_rmap_item *rmap_list; |
| @@ -205,6 +209,8 @@ struct ksm_rmap_item { · 6 unmodified lines | ||
| 205 | 209 | struct mm_struct *mm; |
| 206 | 210 | unsigned long address; /* + low bits used for flags below */ |
| 207 | 211 | unsigned int oldchecksum; /* when unstable */ |
| 212 | + rmap_age_t age; | |
| 213 | + rmap_age_t remaining_skips; | |
| 208 | 214 | union { |
| 209 | 215 | struct rb_node node; /* when node of unstable tree */ |
| 210 | 216 | struct { /* when listed from stable tree */ |
| @@ -281,9 +287,16 @@ static unsigned int zero_checksum __read_mostly; · 70 unmodified lines | ||
| 281 | 287 | /* Whether to merge empty (zeroed) pages with actual zero pages */ |
| 282 | 288 | static bool ksm_use_zero_pages __read_mostly; |
| 283 | 289 | |
| 290 | +/* Skip pages that couldn't be de-duplicated previously */ | |
| 291 | +/* Default to true at least temporarily, for testing */ | |
| 292 | +static bool ksm_smart_scan = true; | |
| 293 | + | |
| 284 | 294 | /* The number of zero pages which is placed by KSM */ |
| 285 | 295 | unsigned long ksm_zero_pages; |
| 286 | 296 | |
| 297 | +/* The number of pages that have been skipped due to "smart scanning" */ | |
| 298 | +static unsigned long ksm_pages_skipped; | |
| 299 | + | |
| 287 | 300 | #ifdef CONFIG_NUMA |
| 288 | 301 | /* Zeroed when merging across nodes is not allowed */ |
| 289 | 302 | static unsigned int ksm_merge_across_nodes = 1; |
| @@ -2305,6 +2318,74 @@ static struct ksm_rmap_item *get_next_rmap_item(struct ksm_mm_slot *mm_slot, · 2015 unmodified lines | ||
| 2305 | 2318 | return rmap_item; |
| 2306 | 2319 | } |
| 2307 | 2320 | |
| 2321 | +/* | |
| 2322 | + * Calculate skip age for the ksm page age. The age determines how often | |
| 2323 | + * de-duplicating has already been tried unsuccessfully. If the age is | |
| 2324 | + * smaller, the scanning of this page is skipped for less scans. | |
| 2325 | + * | |
| 2326 | + * @age: rmap_item age of page | |
| 2327 | + */ | |
| 2328 | +static unsigned int skip_age(rmap_age_t age) | |
| 2329 | +{ | |
| 2330 | + if (age <= 3) | |
| 2331 | + return 1; | |
| 2332 | + if (age <= 5) | |
| 2333 | + return 2; | |
| 2334 | + if (age <= 8) | |
| 2335 | + return 4; | |
| 2336 | + | |
| 2337 | + return 8; | |
| 2338 | +} | |
| 2339 | + | |
| 2340 | +/* | |
| 2341 | + * Determines if a page should be skipped for the current scan. | |
| 2342 | + * | |
| 2343 | + * @page: page to check | |
| 2344 | + * @rmap_item: associated rmap_item of page | |
| 2345 | + */ | |
| 2346 | +static bool should_skip_rmap_item(struct page *page, | |
| 2347 | + struct ksm_rmap_item *rmap_item) | |
| 2348 | +{ | |
| 2349 | + rmap_age_t age; | |
| 2350 | + | |
| 2351 | + if (!ksm_smart_scan) | |
| 2352 | + return false; | |
| 2353 | + | |
| 2354 | + /* | |
| 2355 | + * Never skip pages that are already KSM; pages cmp_and_merge_page() | |
| 2356 | + * will essentially ignore them, but we still have to process them | |
| 2357 | + * properly. | |
| 2358 | + */ | |
| 2359 | + if (PageKsm(page)) | |
| 2360 | + return false; | |
| 2361 | + | |
| 2362 | + age = rmap_item->age; | |
| 2363 | + if (age != U8_MAX) | |
| 2364 | + rmap_item->age++; | |
| 2365 | + | |
| 2366 | + /* | |
| 2367 | + * Smaller ages are not skipped, they need to get a chance to go | |
| 2368 | + * through the different phases of the KSM merging. | |
| 2369 | + */ | |
| 2370 | + if (age < 3) | |
| 2371 | + return false; | |
| 2372 | + | |
| 2373 | + /* | |
| 2374 | + * Are we still allowed to skip? If not, then don't skip it | |
| 2375 | + * and determine how much more often we are allowed to skip next. | |
| 2376 | + */ | |
| 2377 | + if (!rmap_item->remaining_skips) { | |
| 2378 | + rmap_item->remaining_skips = skip_age(age); | |
| 2379 | + return false; | |
| 2380 | + } | |
| 2381 | + | |
| 2382 | + /* Skip this page */ | |
| 2383 | + ksm_pages_skipped++; | |
| 2384 | + rmap_item->remaining_skips--; | |
| 2385 | + remove_rmap_item_from_tree(rmap_item); | |
| 2386 | + return true; | |
| 2387 | +} | |
| 2388 | + | |
| 2308 | 2389 | static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page) |
| 2309 | 2390 | { |
| 2310 | 2391 | struct mm_struct *mm; |
| @@ -2409,6 +2490,10 @@ next_mm: · 98 unmodified lines | ||
| 2409 | 2490 | if (rmap_item) { |
| 2410 | 2491 | ksm_scan.rmap_list = |
| 2411 | 2492 | &rmap_item->rmap_list; |
| 2493 | + | |
| 2494 | + if (should_skip_rmap_item(*page, rmap_item)) | |
| 2495 | + goto next_page; | |
| 2496 | + | |
| 2412 | 2497 | ksm_scan.address += PAGE_SIZE; |
| 2413 | 2498 | } else |
| 2414 | 2499 | put_page(*page); |
| @@ -3383,6 +3468,13 @@ static ssize_t pages_volatile_show(struct kobject *kobj, · 968 unmodified lines | ||
| 3383 | 3468 | } |
| 3384 | 3469 | KSM_ATTR_RO(pages_volatile); |
| 3385 | 3470 | |
| 3471 | +static ssize_t pages_skipped_show(struct kobject *kobj, | |
| 3472 | + struct kobj_attribute *attr, char *buf) | |
| 3473 | +{ | |
| 3474 | + return sysfs_emit(buf, "%lu\n", ksm_pages_skipped); | |
| 3475 | +} | |
| 3476 | +KSM_ATTR_RO(pages_skipped); | |
| 3477 | + | |
| 3386 | 3478 | static ssize_t ksm_zero_pages_show(struct kobject *kobj, |
| 3387 | 3479 | struct kobj_attribute *attr, char *buf) |
| 3388 | 3480 | { |
| @@ -3449,6 +3541,28 @@ static ssize_t full_scans_show(struct kobject *kobj, · 60 unmodified lines | ||
| 3449 | 3541 | } |
| 3450 | 3542 | KSM_ATTR_RO(full_scans); |
| 3451 | 3543 | |
| 3544 | +static ssize_t smart_scan_show(struct kobject *kobj, | |
| 3545 | + struct kobj_attribute *attr, char *buf) | |
| 3546 | +{ | |
| 3547 | + return sysfs_emit(buf, "%u\n", ksm_smart_scan); | |
| 3548 | +} | |
| 3549 | + | |
| 3550 | +static ssize_t smart_scan_store(struct kobject *kobj, | |
| 3551 | + struct kobj_attribute *attr, | |
| 3552 | + const char *buf, size_t count) | |
| 3553 | +{ | |
| 3554 | + int err; | |
| 3555 | + bool value; | |
| 3556 | + | |
| 3557 | + err = kstrtobool(buf, &value); | |
| 3558 | + if (err) | |
| 3559 | + return -EINVAL; | |
| 3560 | + | |
| 3561 | + ksm_smart_scan = value; | |
| 3562 | + return count; | |
| 3563 | +} | |
| 3564 | +KSM_ATTR(smart_scan); | |
| 3565 | + | |
| 3452 | 3566 | static struct attribute *ksm_attrs[] = { |
| 3453 | 3567 | &sleep_millisecs_attr.attr, |
| 3454 | 3568 | &pages_to_scan_attr.attr, |
| @@ -3458,6 +3572,7 @@ static struct attribute *ksm_attrs[] = { · 3 unmodified lines | ||
| 3458 | 3572 | &pages_sharing_attr.attr, |
| 3459 | 3573 | &pages_unshared_attr.attr, |
| 3460 | 3574 | &pages_volatile_attr.attr, |
| 3575 | + &pages_skipped_attr.attr, | |
| 3461 | 3576 | &ksm_zero_pages_attr.attr, |
| 3462 | 3577 | &full_scans_attr.attr, |
| 3463 | 3578 | #ifdef CONFIG_NUMA |
| @@ -3469,6 +3584,7 @@ static struct attribute *ksm_attrs[] = { · 5 unmodified lines | ||
| 3469 | 3584 | &stable_node_chains_prune_millisecs_attr.attr, |
| 3470 | 3585 | &use_zero_pages_attr.attr, |
| 3471 | 3586 | &general_profit_attr.attr, |
| 3587 | + &smart_scan_attr.attr, | |
| 3472 | 3588 | NULL, |
| 3473 | 3589 | }; |
| 3474 | 3590 | |
| @@ -141,7 +141,6 @@ static int madvise_update_vma(struct vm_area_struct *vma, · 140 unmodified lines | ||
| 141 | 141 | { |
| 142 | 142 | struct mm_struct *mm = vma->vm_mm; |
| 143 | 143 | int error; |
| 144 | − pgoff_t pgoff; | |
| 145 | 144 | VMA_ITERATOR(vmi, mm, start); |
| 146 | 145 | |
| 147 | 146 | if (new_flags == vma->vm_flags && anon_vma_name_eq(anon_vma_name(vma), anon_name)) { |
| @@ -149,30 +148,13 @@ static int madvise_update_vma(struct vm_area_struct *vma, · 1 unmodified line | ||
| 149 | 148 | return 0; |
| 150 | 149 | } |
| 151 | 150 | |
| 152 | − pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT); | |
| 153 | − *prev = vma_merge(&vmi, mm, *prev, start, end, new_flags, | |
| 154 | − vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma), | |
| 155 | − vma->vm_userfaultfd_ctx, anon_name); | |
| 156 | − if (*prev) { | |
| 157 | − vma = *prev; | |
| 158 | − goto success; | |
| 159 | − } | |
| 151 | + vma = vma_modify_flags_name(&vmi, *prev, vma, start, end, new_flags, | |
| 152 | + anon_name); | |
| 153 | + if (IS_ERR(vma)) | |
| 154 | + return PTR_ERR(vma); | |
| 160 | 155 | |
| 161 | 156 | *prev = vma; |
| 162 | 157 | |
| 163 | − if (start != vma->vm_start) { | |
| 164 | − error = split_vma(&vmi, vma, start, 1); | |
| 165 | − if (error) | |
| 166 | − return error; | |
| 167 | − } | |
| 168 | − | |
| 169 | − if (end != vma->vm_end) { | |
| 170 | − error = split_vma(&vmi, vma, end, 0); | |
| 171 | − if (error) | |
| 172 | − return error; | |
| 173 | − } | |
| 174 | − | |
| 175 | −success: | |
| 176 | 158 | /* vm_flags is protected by the mmap_lock held in write mode. */ |
| 177 | 159 | vma_start_write(vma); |
| 178 | 160 | vm_flags_reset(vma, new_flags); |
| @@ -746,11 +728,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, · 567 unmodified lines | ||
| 746 | 728 | folio_mark_lazyfree(folio); |
| 747 | 729 | } |
| 748 | 730 | |
| 749 | − if (nr_swap) { | |
| 750 | − if (current->mm == mm) | |
| 751 | − sync_mm_rss(mm); | |
| 731 | + if (nr_swap) | |
| 752 | 732 | add_mm_counter(mm, MM_SWAPENTS, nr_swap); |
| 753 | − } | |
| 754 | 733 | if (start_pte) { |
| 755 | 734 | arch_leave_lazy_mmu_mode(); |
| 756 | 735 | pte_unmap_unlock(start_pte, ptl); |
| @@ -991,7 +970,7 @@ static long madvise_remove(struct vm_area_struct *vma, · 234 unmodified lines | ||
| 991 | 970 | return -EINVAL; |
| 992 | 971 | } |
| 993 | 972 | |
| 994 | − if ((vma->vm_flags & (VM_SHARED|VM_WRITE)) != (VM_SHARED|VM_WRITE)) | |
| 973 | + if (!vma_is_shared_maywrite(vma)) | |
| 995 | 974 | return -EACCES; |
| 996 | 975 | |
| 997 | 976 | offset = (loff_t)(start - vma->vm_start) |
| @@ -892,6 +892,7 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) · 891 unmodified lines | ||
| 892 | 892 | |
| 893 | 893 | /** |
| 894 | 894 | * memblock_setclr_flag - set or clear flag for a memory region |
| 895 | + * @type: memblock type to set/clear flag for | |
| 895 | 896 | * @base: base address of the region |
| 896 | 897 | * @size: size of the region |
| 897 | 898 | * @set: set or clear the flag |
| @@ -901,10 +902,9 @@ int __init_memblock memblock_physmem_add(phys_addr_t base, phys_addr_t size) · 3 unmodified lines | ||
| 901 | 902 | * |
| 902 | 903 | * Return: 0 on success, -errno on failure. |
| 903 | 904 | */ |
| 904 | −static int __init_memblock memblock_setclr_flag(phys_addr_t base, | |
| 905 | − phys_addr_t size, int set, int flag) | |
| 905 | +static int __init_memblock memblock_setclr_flag(struct memblock_type *type, | |
| 906 | + phys_addr_t base, phys_addr_t size, int set, int flag) | |
| 906 | 907 | { |
| 907 | − struct memblock_type *type = &memblock.memory; | |
| 908 | 908 | int i, ret, start_rgn, end_rgn; |
| 909 | 909 | |
| 910 | 910 | ret = memblock_isolate_range(type, base, size, &start_rgn, &end_rgn); |
| @@ -933,7 +933,7 @@ static int __init_memblock memblock_setclr_flag(phys_addr_t base, · 22 unmodified lines | ||
| 933 | 933 | */ |
| 934 | 934 | int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) |
| 935 | 935 | { |
| 936 | − return memblock_setclr_flag(base, size, 1, MEMBLOCK_HOTPLUG); | |
| 936 | + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_HOTPLUG); | |
| 937 | 937 | } |
| 938 | 938 | |
| 939 | 939 | /** |
| @@ -945,7 +945,7 @@ int __init_memblock memblock_mark_hotplug(phys_addr_t base, phys_addr_t size) · 5 unmodified lines | ||
| 945 | 945 | */ |
| 946 | 946 | int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size) |
| 947 | 947 | { |
| 948 | − return memblock_setclr_flag(base, size, 0, MEMBLOCK_HOTPLUG); | |
| 948 | + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_HOTPLUG); | |
| 949 | 949 | } |
| 950 | 950 | |
| 951 | 951 | /** |
| @@ -962,7 +962,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) · 10 unmodified lines | ||
| 962 | 962 | |
| 963 | 963 | system_has_some_mirror = true; |
| 964 | 964 | |
| 965 | − return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR); | |
| 965 | + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_MIRROR); | |
| 966 | 966 | } |
| 967 | 967 | |
| 968 | 968 | /** |
| @@ -982,7 +982,7 @@ int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size) · 13 unmodified lines | ||
| 982 | 982 | */ |
| 983 | 983 | int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) |
| 984 | 984 | { |
| 985 | − return memblock_setclr_flag(base, size, 1, MEMBLOCK_NOMAP); | |
| 985 | + return memblock_setclr_flag(&memblock.memory, base, size, 1, MEMBLOCK_NOMAP); | |
| 986 | 986 | } |
| 987 | 987 | |
| 988 | 988 | /** |
| @@ -994,7 +994,25 @@ int __init_memblock memblock_mark_nomap(phys_addr_t base, phys_addr_t size) · 5 unmodified lines | ||
| 994 | 994 | */ |
| 995 | 995 | int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) |
| 996 | 996 | { |
| 997 | − return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); | |
| 997 | + return memblock_setclr_flag(&memblock.memory, base, size, 0, MEMBLOCK_NOMAP); | |
| 998 | +} | |
| 999 | + | |
| 1000 | +/** | |
| 1001 | + * memblock_reserved_mark_noinit - Mark a reserved memory region with flag | |
| 1002 | + * MEMBLOCK_RSRV_NOINIT which results in the struct pages not being initialized | |
| 1003 | + * for this region. | |
| 1004 | + * @base: the base phys addr of the region | |
| 1005 | + * @size: the size of the region | |
| 1006 | + * | |
| 1007 | + * struct pages will not be initialized for reserved memory regions marked with | |
| 1008 | + * %MEMBLOCK_RSRV_NOINIT. | |
| 1009 | + * | |
| 1010 | + * Return: 0 on success, -errno on failure. | |
| 1011 | + */ | |
| 1012 | +int __init_memblock memblock_reserved_mark_noinit(phys_addr_t base, phys_addr_t size) | |
| 1013 | +{ | |
| 1014 | + return memblock_setclr_flag(&memblock.reserved, base, size, 1, | |
| 1015 | + MEMBLOCK_RSRV_NOINIT); | |
| 998 | 1016 | } |
| 999 | 1017 | |
| 1000 | 1018 | static bool should_skip_region(struct memblock_type *type, |
| @@ -2113,13 +2131,18 @@ static void __init memmap_init_reserved_pages(void) · 1112 unmodified lines | ||
| 2113 | 2131 | memblock_set_node(start, end, &memblock.reserved, nid); |
| 2114 | 2132 | } |
| 2115 | 2133 | |
| 2116 | − /* initialize struct pages for the reserved regions */ | |
| 2134 | + /* | |
| 2135 | + * initialize struct pages for reserved regions that don't have | |
| 2136 | + * the MEMBLOCK_RSRV_NOINIT flag set | |
| 2137 | + */ | |
| 2117 | 2138 | for_each_reserved_mem_region(region) { |
| 2118 | − nid = memblock_get_region_node(region); | |
| 2119 | − start = region->base; | |
| 2120 | − end = start + region->size; | |
| 2139 | + if (!memblock_is_reserved_noinit(region)) { | |
| 2140 | + nid = memblock_get_region_node(region); | |
| 2141 | + start = region->base; | |
| 2142 | + end = start + region->size; | |
| 2121 | 2143 | |
| 2122 | − reserve_bootmem_region(start, end, nid); | |
| 2144 | + reserve_bootmem_region(start, end, nid); | |
| 2145 | + } | |
| 2123 | 2146 | } |
| 2124 | 2147 | } |
| 2125 | 2148 | |
| @@ -249,6 +249,9 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressure *vmpr) · 248 unmodified lines | ||
| 249 | 249 | return container_of(vmpr, struct mem_cgroup, vmpressure); |
| 250 | 250 | } |
| 251 | 251 | |
| 252 | +#define CURRENT_OBJCG_UPDATE_BIT 0 | |
| 253 | +#define CURRENT_OBJCG_UPDATE_FLAG (1UL << CURRENT_OBJCG_UPDATE_BIT) | |
| 254 | + | |
| 252 | 255 | #ifdef CONFIG_MEMCG_KMEM |
| 253 | 256 | static DEFINE_SPINLOCK(objcg_lock); |
| 254 | 257 | |
| @@ -704,6 +707,8 @@ static const unsigned int memcg_vm_event_stat[] = { · 449 unmodified lines | ||
| 704 | 707 | #ifdef CONFIG_TRANSPARENT_HUGEPAGE |
| 705 | 708 | THP_FAULT_ALLOC, |
| 706 | 709 | THP_COLLAPSE_ALLOC, |
| 710 | + THP_SWPOUT, | |
| 711 | + THP_SWPOUT_FALLBACK, | |
| 707 | 712 | #endif |
| 708 | 713 | }; |
| 709 | 714 | |
| @@ -761,6 +766,22 @@ unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) · 51 unmodified lines | ||
| 761 | 766 | return x; |
| 762 | 767 | } |
| 763 | 768 | |
| 769 | +static int memcg_page_state_unit(int item); | |
| 770 | + | |
| 771 | +/* | |
| 772 | + * Normalize the value passed into memcg_rstat_updated() to be in pages. Round | |
| 773 | + * up non-zero sub-page updates to 1 page as zero page updates are ignored. | |
| 774 | + */ | |
| 775 | +static int memcg_state_val_in_pages(int idx, int val) | |
| 776 | +{ | |
| 777 | + int unit = memcg_page_state_unit(idx); | |
| 778 | + | |
| 779 | + if (!val || unit == PAGE_SIZE) | |
| 780 | + return val; | |
| 781 | + else | |
| 782 | + return max(val * unit / PAGE_SIZE, 1UL); | |
| 783 | +} | |
| 784 | + | |
| 764 | 785 | /** |
| 765 | 786 | * __mod_memcg_state - update cgroup memory statistics |
| 766 | 787 | * @memcg: the memory cgroup |
| @@ -773,7 +794,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) · 6 unmodified lines | ||
| 773 | 794 | return; |
| 774 | 795 | |
| 775 | 796 | __this_cpu_add(memcg->vmstats_percpu->state[idx], val); |
| 776 | − memcg_rstat_updated(memcg, val); | |
| 797 | + memcg_rstat_updated(memcg, memcg_state_val_in_pages(idx, val)); | |
| 777 | 798 | } |
| 778 | 799 | |
| 779 | 800 | /* idx can be of type enum memcg_stat_item or node_stat_item. */ |
| @@ -798,7 +819,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, · 18 unmodified lines | ||
| 798 | 819 | memcg = pn->memcg; |
| 799 | 820 | |
| 800 | 821 | /* |
| 801 | − * The caller from rmap relay on disabled preemption becase they never | |
| 822 | + * The caller from rmap relies on disabled preemption because they never | |
| 802 | 823 | * update their counter from in-interrupt context. For these two |
| 803 | 824 | * counters we check that the update is never performed from an |
| 804 | 825 | * interrupt context while other caller need to have disabled interrupt. |
| @@ -824,7 +845,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, · 19 unmodified lines | ||
| 824 | 845 | /* Update lruvec */ |
| 825 | 846 | __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val); |
| 826 | 847 | |
| 827 | − memcg_rstat_updated(memcg, val); | |
| 848 | + memcg_rstat_updated(memcg, memcg_state_val_in_pages(idx, val)); | |
| 828 | 849 | memcg_stats_unlock(); |
| 829 | 850 | } |
| 830 | 851 | |
| @@ -1068,17 +1089,25 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) · 237 unmodified lines | ||
| 1068 | 1089 | } |
| 1069 | 1090 | EXPORT_SYMBOL(get_mem_cgroup_from_mm); |
| 1070 | 1091 | |
| 1071 | −static __always_inline bool memcg_kmem_bypass(void) | |
| 1092 | +/** | |
| 1093 | + * get_mem_cgroup_from_current - Obtain a reference on current task's memcg. | |
| 1094 | + */ | |
| 1095 | +struct mem_cgroup *get_mem_cgroup_from_current(void) | |
| 1072 | 1096 | { |
| 1073 | − /* Allow remote memcg charging from any context. */ | |
| 1074 | − if (unlikely(active_memcg())) | |
| 1075 | − return false; | |
| 1097 | + struct mem_cgroup *memcg; | |
| 1076 | 1098 | |
| 1077 | − /* Memcg to charge can't be determined. */ | |
| 1078 | − if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) | |
| 1079 | − return true; | |
| 1099 | + if (mem_cgroup_disabled()) | |
| 1100 | + return NULL; | |
| 1080 | 1101 | |
| 1081 | − return false; | |
| 1102 | +again: | |
| 1103 | + rcu_read_lock(); | |
| 1104 | + memcg = mem_cgroup_from_task(current); | |
| 1105 | + if (!css_tryget(&memcg->css)) { | |
| 1106 | + rcu_read_unlock(); | |
| 1107 | + goto again; | |
| 1108 | + } | |
| 1109 | + rcu_read_unlock(); | |
| 1110 | + return memcg; | |
| 1082 | 1111 | } |
| 1083 | 1112 | |
| 1084 | 1113 | /** |
| @@ -1533,7 +1562,7 @@ static const struct memory_stat memory_stats[] = { · 448 unmodified lines | ||
| 1533 | 1562 | { "workingset_nodereclaim", WORKINGSET_NODERECLAIM }, |
| 1534 | 1563 | }; |
| 1535 | 1564 | |
| 1536 | −/* Translate stat items to the correct unit for memory.stat output */ | |
| 1565 | +/* The actual unit of the state item, not the same as the output unit */ | |
| 1537 | 1566 | static int memcg_page_state_unit(int item) |
| 1538 | 1567 | { |
| 1539 | 1568 | switch (item) { |
| @@ -1541,6 +1570,22 @@ static int memcg_page_state_unit(int item) · 1 unmodified line | ||
| 1541 | 1570 | case MEMCG_ZSWAP_B: |
| 1542 | 1571 | case NR_SLAB_RECLAIMABLE_B: |
| 1543 | 1572 | case NR_SLAB_UNRECLAIMABLE_B: |
| 1573 | + return 1; | |
| 1574 | + case NR_KERNEL_STACK_KB: | |
| 1575 | + return SZ_1K; | |
| 1576 | + default: | |
| 1577 | + return PAGE_SIZE; | |
| 1578 | + } | |
| 1579 | +} | |
| 1580 | + | |
| 1581 | +/* Translate stat items to the correct unit for memory.stat output */ | |
| 1582 | +static int memcg_page_state_output_unit(int item) | |
| 1583 | +{ | |
| 1584 | + /* | |
| 1585 | + * Workingset state is actually in pages, but we export it to userspace | |
| 1586 | + * as a scalar count of events, so special case it here. | |
| 1587 | + */ | |
| 1588 | + switch (item) { | |
| 1544 | 1589 | case WORKINGSET_REFAULT_ANON: |
| 1545 | 1590 | case WORKINGSET_REFAULT_FILE: |
| 1546 | 1591 | case WORKINGSET_ACTIVATE_ANON: |
| @@ -1549,17 +1594,23 @@ static int memcg_page_state_unit(int item) · 2 unmodified lines | ||
| 1549 | 1594 | case WORKINGSET_RESTORE_FILE: |
| 1550 | 1595 | case WORKINGSET_NODERECLAIM: |
| 1551 | 1596 | return 1; |
| 1552 | − case NR_KERNEL_STACK_KB: | |
| 1553 | − return SZ_1K; | |
| 1554 | 1597 | default: |
| 1555 | − return PAGE_SIZE; | |
| 1598 | + return memcg_page_state_unit(item); | |
| 1556 | 1599 | } |
| 1557 | 1600 | } |
| 1558 | 1601 | |
| 1559 | 1602 | static inline unsigned long memcg_page_state_output(struct mem_cgroup *memcg, |
| 1560 | 1603 | int item) |
| 1561 | 1604 | { |
| 1562 | − return memcg_page_state(memcg, item) * memcg_page_state_unit(item); | |
| 1605 | + return memcg_page_state(memcg, item) * | |
| 1606 | + memcg_page_state_output_unit(item); | |
| 1607 | +} | |
| 1608 | + | |
| 1609 | +static inline unsigned long memcg_page_state_local_output( | |
| 1610 | + struct mem_cgroup *memcg, int item) | |
| 1611 | +{ | |
| 1612 | + return memcg_page_state_local(memcg, item) * | |
| 1613 | + memcg_page_state_output_unit(item); | |
| 1563 | 1614 | } |
| 1564 | 1615 | |
| 1565 | 1616 | static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) |
| @@ -2833,7 +2884,12 @@ static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, · 1267 unmodified lines | ||
| 2833 | 2884 | return try_charge_memcg(memcg, gfp_mask, nr_pages); |
| 2834 | 2885 | } |
| 2835 | 2886 | |
| 2836 | −static inline void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) | |
| 2887 | +/** | |
| 2888 | + * mem_cgroup_cancel_charge() - cancel an uncommitted try_charge() call. | |
| 2889 | + * @memcg: memcg previously charged. | |
| 2890 | + * @nr_pages: number of pages previously charged. | |
| 2891 | + */ | |
| 2892 | +void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) | |
| 2837 | 2893 | { |
| 2838 | 2894 | if (mem_cgroup_is_root(memcg)) |
| 2839 | 2895 | return; |
| @@ -2858,6 +2914,22 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) · 18 unmodified lines | ||
| 2858 | 2914 | folio->memcg_data = (unsigned long)memcg; |
| 2859 | 2915 | } |
| 2860 | 2916 | |
| 2917 | +/** | |
| 2918 | + * mem_cgroup_commit_charge - commit a previously successful try_charge(). | |
| 2919 | + * @folio: folio to commit the charge to. | |
| 2920 | + * @memcg: memcg previously charged. | |
| 2921 | + */ | |
| 2922 | +void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg) | |
| 2923 | +{ | |
| 2924 | + css_get(&memcg->css); | |
| 2925 | + commit_charge(folio, memcg); | |
| 2926 | + | |
| 2927 | + local_irq_disable(); | |
| 2928 | + mem_cgroup_charge_statistics(memcg, folio_nr_pages(folio)); | |
| 2929 | + memcg_check_events(memcg, folio_nid(folio)); | |
| 2930 | + local_irq_enable(); | |
| 2931 | +} | |
| 2932 | + | |
| 2861 | 2933 | #ifdef CONFIG_MEMCG_KMEM |
| 2862 | 2934 | /* |
| 2863 | 2935 | * The allocated objcg pointers array is not accounted directly. |
| @@ -3007,28 +3079,105 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) · 143 unmodified lines | ||
| 3007 | 3079 | |
| 3008 | 3080 | for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { |
| 3009 | 3081 | objcg = rcu_dereference(memcg->objcg); |
| 3010 | − if (objcg && obj_cgroup_tryget(objcg)) | |
| 3082 | + if (likely(objcg && obj_cgroup_tryget(objcg))) | |
| 3011 | 3083 | break; |
| 3012 | 3084 | objcg = NULL; |
| 3013 | 3085 | } |
| 3014 | 3086 | return objcg; |
| 3015 | 3087 | } |
| 3016 | 3088 | |
| 3017 | −__always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) | |
| 3089 | +static struct obj_cgroup *current_objcg_update(void) | |
| 3018 | 3090 | { |
| 3019 | − struct obj_cgroup *objcg = NULL; | |
| 3020 | 3091 | struct mem_cgroup *memcg; |
| 3092 | + struct obj_cgroup *old, *objcg = NULL; | |
| 3021 | 3093 | |
| 3022 | − if (memcg_kmem_bypass()) | |
| 3023 | − return NULL; | |
| 3094 | + do { | |
| 3095 | + /* Atomically drop the update bit. */ | |
| 3096 | + old = xchg(¤t->objcg, NULL); | |
| 3097 | + if (old) { | |
| 3098 | + old = (struct obj_cgroup *) | |
| 3099 | + ((unsigned long)old & ~CURRENT_OBJCG_UPDATE_FLAG); | |
| 3100 | + if (old) | |
| 3101 | + obj_cgroup_put(old); | |
| 3102 | + | |
| 3103 | + old = NULL; | |
| 3104 | + } | |
| 3024 | 3105 | |
| 3025 | − rcu_read_lock(); | |
| 3026 | − if (unlikely(active_memcg())) | |
| 3027 | − memcg = active_memcg(); | |
| 3028 | − else | |
| 3106 | + /* If new objcg is NULL, no reason for the second atomic update. */ | |
| 3107 | + if (!current->mm || (current->flags & PF_KTHREAD)) | |
| 3108 | + return NULL; | |
| 3109 | + | |
| 3110 | + /* | |
| 3111 | + * Release the objcg pointer from the previous iteration, | |
| 3112 | + * if try_cmpxcg() below fails. | |
| 3113 | + */ | |
| 3114 | + if (unlikely(objcg)) { | |
| 3115 | + obj_cgroup_put(objcg); | |
| 3116 | + objcg = NULL; | |
| 3117 | + } | |
| 3118 | + | |
| 3119 | + /* | |
| 3120 | + * Obtain the new objcg pointer. The current task can be | |
| 3121 | + * asynchronously moved to another memcg and the previous | |
| 3122 | + * memcg can be offlined. So let's get the memcg pointer | |
| 3123 | + * and try get a reference to objcg under a rcu read lock. | |
| 3124 | + */ | |
| 3125 | + | |
| 3126 | + rcu_read_lock(); | |
| 3029 | 3127 | memcg = mem_cgroup_from_task(current); |
| 3030 | − objcg = __get_obj_cgroup_from_memcg(memcg); | |
| 3031 | − rcu_read_unlock(); | |
| 3128 | + objcg = __get_obj_cgroup_from_memcg(memcg); | |
| 3129 | + rcu_read_unlock(); | |
| 3130 | + | |
| 3131 | + /* | |
| 3132 | + * Try set up a new objcg pointer atomically. If it | |
| 3133 | + * fails, it means the update flag was set concurrently, so | |
| 3134 | + * the whole procedure should be repeated. | |
| 3135 | + */ | |
| 3136 | + } while (!try_cmpxchg(¤t->objcg, &old, objcg)); | |
| 3137 | + | |
| 3138 | + return objcg; | |
| 3139 | +} | |
| 3140 | + | |
| 3141 | +__always_inline struct obj_cgroup *current_obj_cgroup(void) | |
| 3142 | +{ | |
| 3143 | + struct mem_cgroup *memcg; | |
| 3144 | + struct obj_cgroup *objcg; | |
| 3145 | + | |
| 3146 | + if (in_task()) { | |
| 3147 | + memcg = current->active_memcg; | |
| 3148 | + if (unlikely(memcg)) | |
| 3149 | + goto from_memcg; | |
| 3150 | + | |
| 3151 | + objcg = READ_ONCE(current->objcg); | |
| 3152 | + if (unlikely((unsigned long)objcg & CURRENT_OBJCG_UPDATE_FLAG)) | |
| 3153 | + objcg = current_objcg_update(); | |
| 3154 | + /* | |
| 3155 | + * Objcg reference is kept by the task, so it's safe | |
| 3156 | + * to use the objcg by the current task. | |
| 3157 | + */ | |
| 3158 | + return objcg; | |
| 3159 | + } | |
| 3160 | + | |
| 3161 | + memcg = this_cpu_read(int_active_memcg); | |
| 3162 | + if (unlikely(memcg)) | |
| 3163 | + goto from_memcg; | |
| 3164 | + | |
| 3165 | + return NULL; | |
| 3166 | + | |
| 3167 | +from_memcg: | |
| 3168 | + for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { | |
| 3169 | + /* | |
| 3170 | + * Memcg pointer is protected by scope (see set_active_memcg()) | |
| 3171 | + * and is pinning the corresponding objcg, so objcg can't go | |
| 3172 | + * away and can be used within the scope without any additional | |
| 3173 | + * protection. | |
| 3174 | + */ | |
| 3175 | + objcg = rcu_dereference_check(memcg->objcg, 1); | |
| 3176 | + if (likely(objcg)) | |
| 3177 | + break; | |
| 3178 | + objcg = NULL; | |
| 3179 | + } | |
| 3180 | + | |
| 3032 | 3181 | return objcg; |
| 3033 | 3182 | } |
| 3034 | 3183 | |
| @@ -3126,15 +3275,15 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) · 91 unmodified lines | ||
| 3126 | 3275 | struct obj_cgroup *objcg; |
| 3127 | 3276 | int ret = 0; |
| 3128 | 3277 | |
| 3129 | − objcg = get_obj_cgroup_from_current(); | |
| 3278 | + objcg = current_obj_cgroup(); | |
| 3130 | 3279 | if (objcg) { |
| 3131 | 3280 | ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); |
| 3132 | 3281 | if (!ret) { |
| 3282 | + obj_cgroup_get(objcg); | |
| 3133 | 3283 | page->memcg_data = (unsigned long)objcg | |
| 3134 | 3284 | MEMCG_DATA_KMEM; |
| 3135 | 3285 | return 0; |
| 3136 | 3286 | } |
| 3137 | − obj_cgroup_put(objcg); | |
| 3138 | 3287 | } |
| 3139 | 3288 | return ret; |
| 3140 | 3289 | } |
| @@ -3761,6 +3910,8 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) · 620 unmodified lines | ||
| 3761 | 3910 | |
| 3762 | 3911 | objcg->memcg = memcg; |
| 3763 | 3912 | rcu_assign_pointer(memcg->objcg, objcg); |
| 3913 | + obj_cgroup_get(objcg); | |
| 3914 | + memcg->orig_objcg = objcg; | |
| 3764 | 3915 | |
| 3765 | 3916 | static_branch_enable(&memcg_kmem_online_key); |
| 3766 | 3917 | |
| @@ -4066,7 +4217,10 @@ static const unsigned int memcg1_stats[] = { · 299 unmodified lines | ||
| 4066 | 4217 | NR_WRITEBACK, |
| 4067 | 4218 | WORKINGSET_REFAULT_ANON, |
| 4068 | 4219 | WORKINGSET_REFAULT_FILE, |
| 4220 | +#ifdef CONFIG_SWAP | |
| 4069 | 4221 | MEMCG_SWAP, |
| 4222 | + NR_SWAPCACHE, | |
| 4223 | +#endif | |
| 4070 | 4224 | }; |
| 4071 | 4225 | |
| 4072 | 4226 | static const char *const memcg1_stat_names[] = { |
| @@ -4081,7 +4235,10 @@ static const char *const memcg1_stat_names[] = { · 8 unmodified lines | ||
| 4081 | 4235 | "writeback", |
| 4082 | 4236 | "workingset_refault_anon", |
| 4083 | 4237 | "workingset_refault_file", |
| 4238 | +#ifdef CONFIG_SWAP | |
| 4084 | 4239 | "swap", |
| 4240 | + "swapcached", | |
| 4241 | +#endif | |
| 4085 | 4242 | }; |
| 4086 | 4243 | |
| 4087 | 4244 | /* Universal VM events cgroup1 shows, original sort order */ |
| @@ -4105,11 +4262,8 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) · 17 unmodified lines | ||
| 4105 | 4262 | for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { |
| 4106 | 4263 | unsigned long nr; |
| 4107 | 4264 | |
| 4108 | − if (memcg1_stats[i] == MEMCG_SWAP && !do_memsw_account()) | |
| 4109 | − continue; | |
| 4110 | − nr = memcg_page_state_local(memcg, memcg1_stats[i]); | |
| 4111 | − seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], | |
| 4112 | − nr * memcg_page_state_unit(memcg1_stats[i])); | |
| 4265 | + nr = memcg_page_state_local_output(memcg, memcg1_stats[i]); | |
| 4266 | + seq_buf_printf(s, "%s %lu\n", memcg1_stat_names[i], nr); | |
| 4113 | 4267 | } |
| 4114 | 4268 | |
| 4115 | 4269 | for (i = 0; i < ARRAY_SIZE(memcg1_events); i++) |
| @@ -4129,18 +4283,15 @@ static void memcg1_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) · 13 unmodified lines | ||
| 4129 | 4283 | } |
| 4130 | 4284 | seq_buf_printf(s, "hierarchical_memory_limit %llu\n", |
| 4131 | 4285 | (u64)memory * PAGE_SIZE); |
| 4132 | − if (do_memsw_account()) | |
| 4133 | − seq_buf_printf(s, "hierarchical_memsw_limit %llu\n", | |
| 4134 | − (u64)memsw * PAGE_SIZE); | |
| 4286 | + seq_buf_printf(s, "hierarchical_memsw_limit %llu\n", | |
| 4287 | + (u64)memsw * PAGE_SIZE); | |
| 4135 | 4288 | |
| 4136 | 4289 | for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { |
| 4137 | 4290 | unsigned long nr; |
| 4138 | 4291 | |
| 4139 | − if (memcg1_stats[i] == MEMCG_SWAP && !do_memsw_account()) | |
| 4140 | − continue; | |
| 4141 | − nr = memcg_page_state(memcg, memcg1_stats[i]); | |
| 4292 | + nr = memcg_page_state_output(memcg, memcg1_stats[i]); | |
| 4142 | 4293 | seq_buf_printf(s, "total_%s %llu\n", memcg1_stat_names[i], |
| 4143 | − (u64)nr * memcg_page_state_unit(memcg1_stats[i])); | |
| 4294 | + (u64)nr); | |
| 4144 | 4295 | } |
| 4145 | 4296 | |
| 4146 | 4297 | for (i = 0; i < ARRAY_SIZE(memcg1_events); i++) |
| @@ -5268,6 +5419,9 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) · 1121 unmodified lines | ||
| 5268 | 5419 | { |
| 5269 | 5420 | int node; |
| 5270 | 5421 | |
| 5422 | + if (memcg->orig_objcg) | |
| 5423 | + obj_cgroup_put(memcg->orig_objcg); | |
| 5424 | + | |
| 5271 | 5425 | for_each_node(node) |
| 5272 | 5426 | free_mem_cgroup_per_node_info(memcg, node); |
| 5273 | 5427 | kfree(memcg->vmstats); |
| @@ -6076,7 +6230,7 @@ static void __mem_cgroup_clear_mc(void) · 802 unmodified lines | ||
| 6076 | 6230 | |
| 6077 | 6231 | /* we must uncharge all the leftover precharges from mc.to */ |
| 6078 | 6232 | if (mc.precharge) { |
| 6079 | − cancel_charge(mc.to, mc.precharge); | |
| 6233 | + mem_cgroup_cancel_charge(mc.to, mc.precharge); | |
| 6080 | 6234 | mc.precharge = 0; |
| 6081 | 6235 | } |
| 6082 | 6236 | /* |
| @@ -6084,7 +6238,7 @@ static void __mem_cgroup_clear_mc(void) · 1 unmodified line | ||
| 6084 | 6238 | * we must uncharge here. |
| 6085 | 6239 | */ |
| 6086 | 6240 | if (mc.moved_charge) { |
| 6087 | − cancel_charge(mc.from, mc.moved_charge); | |
| 6241 | + mem_cgroup_cancel_charge(mc.from, mc.moved_charge); | |
| 6088 | 6242 | mc.moved_charge = 0; |
| 6089 | 6243 | } |
| 6090 | 6244 | /* we must fixup refcnts and charges */ |
| @@ -6364,6 +6518,7 @@ static void mem_cgroup_move_task(void) · 273 unmodified lines | ||
| 6364 | 6518 | mem_cgroup_clear_mc(); |
| 6365 | 6519 | } |
| 6366 | 6520 | } |
| 6521 | + | |
| 6367 | 6522 | #else /* !CONFIG_MMU */ |
| 6368 | 6523 | static int mem_cgroup_can_attach(struct cgroup_taskset *tset) |
| 6369 | 6524 | { |
| @@ -6377,8 +6532,39 @@ static void mem_cgroup_move_task(void) · 7 unmodified lines | ||
| 6377 | 6532 | } |
| 6378 | 6533 | #endif |
| 6379 | 6534 | |
| 6535 | +#ifdef CONFIG_MEMCG_KMEM | |
| 6536 | +static void mem_cgroup_fork(struct task_struct *task) | |
| 6537 | +{ | |
| 6538 | + /* | |
| 6539 | + * Set the update flag to cause task->objcg to be initialized lazily | |
| 6540 | + * on the first allocation. It can be done without any synchronization | |
| 6541 | + * because it's always performed on the current task, so does | |
| 6542 | + * current_objcg_update(). | |
| 6543 | + */ | |
| 6544 | + task->objcg = (struct obj_cgroup *)CURRENT_OBJCG_UPDATE_FLAG; | |
| 6545 | +} | |
| 6546 | + | |
| 6547 | +static void mem_cgroup_exit(struct task_struct *task) | |
| 6548 | +{ | |
| 6549 | + struct obj_cgroup *objcg = task->objcg; | |
| 6550 | + | |
| 6551 | + objcg = (struct obj_cgroup *) | |
| 6552 | + ((unsigned long)objcg & ~CURRENT_OBJCG_UPDATE_FLAG); | |
| 6553 | + if (objcg) | |
| 6554 | + obj_cgroup_put(objcg); | |
| 6555 | + | |
| 6556 | + /* | |
| 6557 | + * Some kernel allocations can happen after this point, | |
| 6558 | + * but let's ignore them. It can be done without any synchronization | |
| 6559 | + * because it's always performed on the current task, so does | |
| 6560 | + * current_objcg_update(). | |
| 6561 | + */ | |
| 6562 | + task->objcg = NULL; | |
| 6563 | +} | |
| 6564 | +#endif | |
| 6565 | + | |
| 6380 | 6566 | #ifdef CONFIG_LRU_GEN |
| 6381 | −static void mem_cgroup_attach(struct cgroup_taskset *tset) | |
| 6567 | +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) | |
| 6382 | 6568 | { |
| 6383 | 6569 | struct task_struct *task; |
| 6384 | 6570 | struct cgroup_subsys_state *css; |
| @@ -6396,10 +6582,31 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) · 11 unmodified lines | ||
| 6396 | 6582 | task_unlock(task); |
| 6397 | 6583 | } |
| 6398 | 6584 | #else |
| 6585 | +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {} | |
| 6586 | +#endif /* CONFIG_LRU_GEN */ | |
| 6587 | + | |
| 6588 | +#ifdef CONFIG_MEMCG_KMEM | |
| 6589 | +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) | |
| 6590 | +{ | |
| 6591 | + struct task_struct *task; | |
| 6592 | + struct cgroup_subsys_state *css; | |
| 6593 | + | |
| 6594 | + cgroup_taskset_for_each(task, css, tset) { | |
| 6595 | + /* atomically set the update bit */ | |
| 6596 | + set_bit(CURRENT_OBJCG_UPDATE_BIT, (unsigned long *)&task->objcg); | |
| 6597 | + } | |
| 6598 | +} | |
| 6599 | +#else | |
| 6600 | +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {} | |
| 6601 | +#endif /* CONFIG_MEMCG_KMEM */ | |
| 6602 | + | |
| 6603 | +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) | |
| 6399 | 6604 | static void mem_cgroup_attach(struct cgroup_taskset *tset) |
| 6400 | 6605 | { |
| 6606 | + mem_cgroup_lru_gen_attach(tset); | |
| 6607 | + mem_cgroup_kmem_attach(tset); | |
| 6401 | 6608 | } |
| 6402 | −#endif /* CONFIG_LRU_GEN */ | |
| 6609 | +#endif | |
| 6403 | 6610 | |
| 6404 | 6611 | static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value) |
| 6405 | 6612 | { |
| @@ -6622,7 +6829,8 @@ static int memory_stat_show(struct seq_file *m, void *v) · 216 unmodified lines | ||
| 6622 | 6829 | static inline unsigned long lruvec_page_state_output(struct lruvec *lruvec, |
| 6623 | 6830 | int item) |
| 6624 | 6831 | { |
| 6625 | − return lruvec_page_state(lruvec, item) * memcg_page_state_unit(item); | |
| 6832 | + return lruvec_page_state(lruvec, item) * | |
| 6833 | + memcg_page_state_output_unit(item); | |
| 6626 | 6834 | } |
| 6627 | 6835 | |
| 6628 | 6836 | static int memory_numa_stat_show(struct seq_file *m, void *v) |
| @@ -6808,9 +7016,15 @@ struct cgroup_subsys memory_cgrp_subsys = { · 179 unmodified lines | ||
| 6808 | 7016 | .css_reset = mem_cgroup_css_reset, |
| 6809 | 7017 | .css_rstat_flush = mem_cgroup_css_rstat_flush, |
| 6810 | 7018 | .can_attach = mem_cgroup_can_attach, |
| 7019 | +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) | |
| 6811 | 7020 | .attach = mem_cgroup_attach, |
| 7021 | +#endif | |
| 6812 | 7022 | .cancel_attach = mem_cgroup_cancel_attach, |
| 6813 | 7023 | .post_attach = mem_cgroup_move_task, |
| 7024 | +#ifdef CONFIG_MEMCG_KMEM | |
| 7025 | + .fork = mem_cgroup_fork, | |
| 7026 | + .exit = mem_cgroup_exit, | |
| 7027 | +#endif | |
| 6814 | 7028 | .dfl_cftypes = memory_files, |
| 6815 | 7029 | .legacy_cftypes = mem_cgroup_legacy_files, |
| 6816 | 7030 | .early_init = 0, |
| @@ -6990,20 +7204,13 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, · 173 unmodified lines | ||
| 6990 | 7204 | static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, |
| 6991 | 7205 | gfp_t gfp) |
| 6992 | 7206 | { |
| 6993 | − long nr_pages = folio_nr_pages(folio); | |
| 6994 | 7207 | int ret; |
| 6995 | 7208 | |
| 6996 | − ret = try_charge(memcg, gfp, nr_pages); | |
| 7209 | + ret = try_charge(memcg, gfp, folio_nr_pages(folio)); | |
| 6997 | 7210 | if (ret) |
| 6998 | 7211 | goto out; |
| 6999 | 7212 | |
| 7000 | − css_get(&memcg->css); | |
| 7001 | − commit_charge(folio, memcg); | |
| 7002 | − | |
| 7003 | − local_irq_disable(); | |
| 7004 | − mem_cgroup_charge_statistics(memcg, nr_pages); | |
| 7005 | − memcg_check_events(memcg, folio_nid(folio)); | |
| 7006 | − local_irq_enable(); | |
| 7213 | + mem_cgroup_commit_charge(folio, memcg); | |
| 7007 | 7214 | out: |
| 7008 | 7215 | return ret; |
| 7009 | 7216 | } |
| @@ -7020,6 +7227,41 @@ int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp) · 10 unmodified lines | ||
| 7020 | 7227 | return ret; |
| 7021 | 7228 | } |
| 7022 | 7229 | |
| 7230 | +/** | |
| 7231 | + * mem_cgroup_hugetlb_try_charge - try to charge the memcg for a hugetlb folio | |
| 7232 | + * @memcg: memcg to charge. | |
| 7233 | + * @gfp: reclaim mode. | |
| 7234 | + * @nr_pages: number of pages to charge. | |
| 7235 | + * | |
| 7236 | + * This function is called when allocating a huge page folio to determine if | |
| 7237 | + * the memcg has the capacity for it. It does not commit the charge yet, | |
| 7238 | + * as the hugetlb folio itself has not been obtained from the hugetlb pool. | |
| 7239 | + * | |
| 7240 | + * Once we have obtained the hugetlb folio, we can call | |
| 7241 | + * mem_cgroup_commit_charge() to commit the charge. If we fail to obtain the | |
| 7242 | + * folio, we should instead call mem_cgroup_cancel_charge() to undo the effect | |
| 7243 | + * of try_charge(). | |
| 7244 | + * | |
| 7245 | + * Returns 0 on success. Otherwise, an error code is returned. | |
| 7246 | + */ | |
| 7247 | +int mem_cgroup_hugetlb_try_charge(struct mem_cgroup *memcg, gfp_t gfp, | |
| 7248 | + long nr_pages) | |
| 7249 | +{ | |
| 7250 | + /* | |
| 7251 | + * If hugetlb memcg charging is not enabled, do not fail hugetlb allocation, | |
| 7252 | + * but do not attempt to commit charge later (or cancel on error) either. | |
| 7253 | + */ | |
| 7254 | + if (mem_cgroup_disabled() || !memcg || | |
| 7255 | + !cgroup_subsys_on_dfl(memory_cgrp_subsys) || | |
| 7256 | + !(cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING)) | |
| 7257 | + return -EOPNOTSUPP; | |
| 7258 | + | |
| 7259 | + if (try_charge(memcg, gfp, nr_pages)) | |
| 7260 | + return -ENOMEM; | |
| 7261 | + | |
| 7262 | + return 0; | |
| 7263 | +} | |
| 7264 | + | |
| 7023 | 7265 | /** |
| 7024 | 7266 | * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin. |
| 7025 | 7267 | * @folio: folio to charge. |
| @@ -7216,16 +7458,17 @@ void __mem_cgroup_uncharge_list(struct list_head *page_list) · 190 unmodified lines | ||
| 7216 | 7458 | } |
| 7217 | 7459 | |
| 7218 | 7460 | /** |
| 7219 | − * mem_cgroup_migrate - Charge a folio's replacement. | |
| 7461 | + * mem_cgroup_replace_folio - Charge a folio's replacement. | |
| 7220 | 7462 | * @old: Currently circulating folio. |
| 7221 | 7463 | * @new: Replacement folio. |
| 7222 | 7464 | * |
| 7223 | 7465 | * Charge @new as a replacement folio for @old. @old will |
| 7224 | − * be uncharged upon free. | |
| 7466 | + * be uncharged upon free. This is only used by the page cache | |
| 7467 | + * (in replace_page_cache_folio()). | |
| 7225 | 7468 | * |
| 7226 | 7469 | * Both folios must be locked, @new->mapping must be set up. |
| 7227 | 7470 | */ |
| 7228 | −void mem_cgroup_migrate(struct folio *old, struct folio *new) | |
| 7471 | +void mem_cgroup_replace_folio(struct folio *old, struct folio *new) | |
| 7229 | 7472 | { |
| 7230 | 7473 | struct mem_cgroup *memcg; |
| 7231 | 7474 | long nr_pages = folio_nr_pages(new); |
| @@ -7264,6 +7507,44 @@ void mem_cgroup_migrate(struct folio *old, struct folio *new) · 32 unmodified lines | ||
| 7264 | 7507 | local_irq_restore(flags); |
| 7265 | 7508 | } |
| 7266 | 7509 | |
| 7510 | +/** | |
| 7511 | + * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio. | |
| 7512 | + * @old: Currently circulating folio. | |
| 7513 | + * @new: Replacement folio. | |
| 7514 | + * | |
| 7515 | + * Transfer the memcg data from the old folio to the new folio for migration. | |
| 7516 | + * The old folio's data info will be cleared. Note that the memory counters | |
| 7517 | + * will remain unchanged throughout the process. | |
| 7518 | + * | |
| 7519 | + * Both folios must be locked, @new->mapping must be set up. | |
| 7520 | + */ | |
| 7521 | +void mem_cgroup_migrate(struct folio *old, struct folio *new) | |
| 7522 | +{ | |
| 7523 | + struct mem_cgroup *memcg; | |
| 7524 | + | |
| 7525 | + VM_BUG_ON_FOLIO(!folio_test_locked(old), old); | |
| 7526 | + VM_BUG_ON_FOLIO(!folio_test_locked(new), new); | |
| 7527 | + VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new); | |
| 7528 | + VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new), new); | |
| 7529 | + | |
| 7530 | + if (mem_cgroup_disabled()) | |
| 7531 | + return; | |
| 7532 | + | |
| 7533 | + memcg = folio_memcg(old); | |
| 7534 | + /* | |
| 7535 | + * Note that it is normal to see !memcg for a hugetlb folio. | |
| 7536 | + * For e.g, itt could have been allocated when memory_hugetlb_accounting | |
| 7537 | + * was not selected. | |
| 7538 | + */ | |
| 7539 | + VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old); | |
| 7540 | + if (!memcg) | |
| 7541 | + return; | |
| 7542 | + | |
| 7543 | + /* Transfer the charge and the css ref */ | |
| 7544 | + commit_charge(new, memcg); | |
| 7545 | + old->memcg_data = 0; | |
| 7546 | +} | |
| 7547 | + | |
| 7267 | 7548 | DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); |
| 7268 | 7549 | EXPORT_SYMBOL(memcg_sockets_enabled_key); |
| 7269 | 7550 | |
| @@ -7763,7 +8044,7 @@ static struct cftype memsw_files[] = { · 493 unmodified lines | ||
| 7763 | 8044 | * |
| 7764 | 8045 | * This doesn't check for specific headroom, and it is not atomic |
| 7765 | 8046 | * either. But with zswap, the size of the allocation is only known |
| 7766 | − * once compression has occured, and this optimistic pre-check avoids | |
| 8047 | + * once compression has occurred, and this optimistic pre-check avoids | |
| 7767 | 8048 | * spending cycles on compression when there is already no room left |
| 7768 | 8049 | * or zswap is disabled altogether somewhere in the hierarchy. |
| 7769 | 8050 | */ |
| @@ -315,12 +315,6 @@ SYSCALL_DEFINE2(memfd_create, · 314 unmodified lines | ||
| 315 | 315 | if ((flags & MFD_EXEC) && (flags & MFD_NOEXEC_SEAL)) |
| 316 | 316 | return -EINVAL; |
| 317 | 317 | |
| 318 | − if (!(flags & (MFD_EXEC | MFD_NOEXEC_SEAL))) { | |
| 319 | − pr_warn_once( | |
| 320 | − "%s[%d]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set\n", | |
| 321 | − current->comm, task_pid_nr(current)); | |
| 322 | − } | |
| 323 | − | |
| 324 | 318 | error = check_sysctl_memfd_noexec(&flags); |
| 325 | 319 | if (error < 0) |
| 326 | 320 | return error; |
| @@ -1713,20 +1713,23 @@ static void unmap_and_kill(struct list_head *to_kill, unsigned long pfn, · 1712 unmodified lines | ||
| 1713 | 1713 | kill_procs(to_kill, flags & MF_MUST_KILL, false, pfn, flags); |
| 1714 | 1714 | } |
| 1715 | 1715 | |
| 1716 | +/* | |
| 1717 | + * Only dev_pagemap pages get here, such as fsdax when the filesystem | |
| 1718 | + * either do not claim or fails to claim a hwpoison event, or devdax. | |
| 1719 | + * The fsdax pages are initialized per base page, and the devdax pages | |
| 1720 | + * could be initialized either as base pages, or as compound pages with | |
| 1721 | + * vmemmap optimization enabled. Devdax is simplistic in its dealing with | |
| 1722 | + * hwpoison, such that, if a subpage of a compound page is poisoned, | |
| 1723 | + * simply mark the compound head page is by far sufficient. | |
| 1724 | + */ | |
| 1716 | 1725 | static int mf_generic_kill_procs(unsigned long long pfn, int flags, |
| 1717 | 1726 | struct dev_pagemap *pgmap) |
| 1718 | 1727 | { |
| 1719 | − struct page *page = pfn_to_page(pfn); | |
| 1728 | + struct folio *folio = pfn_folio(pfn); | |
| 1720 | 1729 | LIST_HEAD(to_kill); |
| 1721 | 1730 | dax_entry_t cookie; |
| 1722 | 1731 | int rc = 0; |
| 1723 | 1732 | |
| 1724 | − /* | |
| 1725 | − * Pages instantiated by device-dax (not filesystem-dax) | |
| 1726 | − * may be compound pages. | |
| 1727 | − */ | |
| 1728 | − page = compound_head(page); | |
| 1729 | − | |
| 1730 | 1733 | /* |
| 1731 | 1734 | * Prevent the inode from being freed while we are interrogating |
| 1732 | 1735 | * the address_space, typically this would be handled by |
| @@ -1734,11 +1737,11 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 1 unmodified line | ||
| 1734 | 1737 | * also prevents changes to the mapping of this pfn until |
| 1735 | 1738 | * poison signaling is complete. |
| 1736 | 1739 | */ |
| 1737 | − cookie = dax_lock_page(page); | |
| 1740 | + cookie = dax_lock_folio(folio); | |
| 1738 | 1741 | if (!cookie) |
| 1739 | 1742 | return -EBUSY; |
| 1740 | 1743 | |
| 1741 | − if (hwpoison_filter(page)) { | |
| 1744 | + if (hwpoison_filter(&folio->page)) { | |
| 1742 | 1745 | rc = -EOPNOTSUPP; |
| 1743 | 1746 | goto unlock; |
| 1744 | 1747 | } |
| @@ -1760,7 +1763,7 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 15 unmodified lines | ||
| 1760 | 1763 | * Use this flag as an indication that the dax page has been |
| 1761 | 1764 | * remapped UC to prevent speculative consumption of poison. |
| 1762 | 1765 | */ |
| 1763 | − SetPageHWPoison(page); | |
| 1766 | + SetPageHWPoison(&folio->page); | |
| 1764 | 1767 | |
| 1765 | 1768 | /* |
| 1766 | 1769 | * Unlike System-RAM there is no possibility to swap in a |
| @@ -1769,11 +1772,11 @@ static int mf_generic_kill_procs(unsigned long long pfn, int flags, · 2 unmodified lines | ||
| 1769 | 1772 | * SIGBUS (i.e. MF_MUST_KILL) |
| 1770 | 1773 | */ |
| 1771 | 1774 | flags |= MF_ACTION_REQUIRED | MF_MUST_KILL; |
| 1772 | − collect_procs(page, &to_kill, true); | |
| 1775 | + collect_procs(&folio->page, &to_kill, true); | |
| 1773 | 1776 | |
| 1774 | − unmap_and_kill(&to_kill, pfn, page->mapping, page->index, flags); | |
| 1777 | + unmap_and_kill(&to_kill, pfn, folio->mapping, folio->index, flags); | |
| 1775 | 1778 | unlock: |
| 1776 | − dax_unlock_page(page, cookie); | |
| 1779 | + dax_unlock_folio(folio, cookie); | |
| 1777 | 1780 | return rc; |
| 1778 | 1781 | } |
| 1779 | 1782 | |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <linux/kobject.h> |
| 6 | 6 | #include <linux/memory.h> |
| 7 | 7 | #include <linux/memory-tiers.h> |
| 8 | +#include <linux/notifier.h> | |
| 8 | 9 | |
| 9 | 10 | #include "internal.h" |
| 10 | 11 | |
| @@ -36,7 +37,7 @@ struct node_memory_type_map { · 25 unmodified lines | ||
| 36 | 37 | static DEFINE_MUTEX(memory_tier_lock); |
| 37 | 38 | static LIST_HEAD(memory_tiers); |
| 38 | 39 | static struct node_memory_type_map node_memory_types[MAX_NUMNODES]; |
| 39 | −static struct memory_dev_type *default_dram_type; | |
| 40 | +struct memory_dev_type *default_dram_type; | |
| 40 | 41 | |
| 41 | 42 | static struct bus_type memory_tier_subsys = { |
| 42 | 43 | .name = "memory_tiering", |
| @@ -105,6 +106,13 @@ static int top_tier_adistance; · 62 unmodified lines | ||
| 105 | 106 | static struct demotion_nodes *node_demotion __read_mostly; |
| 106 | 107 | #endif /* CONFIG_MIGRATION */ |
| 107 | 108 | |
| 109 | +static BLOCKING_NOTIFIER_HEAD(mt_adistance_algorithms); | |
| 110 | + | |
| 111 | +static bool default_dram_perf_error; | |
| 112 | +static struct node_hmem_attrs default_dram_perf; | |
| 113 | +static int default_dram_perf_ref_nid = NUMA_NO_NODE; | |
| 114 | +static const char *default_dram_perf_ref_source; | |
| 115 | + | |
| 108 | 116 | static inline struct memory_tier *to_memory_tier(struct device *device) |
| 109 | 117 | { |
| 110 | 118 | return container_of(device, struct memory_tier, dev); |
| @@ -115,7 +123,7 @@ static __always_inline nodemask_t get_memtier_nodemask(struct memory_tier *memti · 4 unmodified lines | ||
| 115 | 123 | nodemask_t nodes = NODE_MASK_NONE; |
| 116 | 124 | struct memory_dev_type *memtype; |
| 117 | 125 | |
| 118 | − list_for_each_entry(memtype, &memtier->memory_types, tier_sibiling) | |
| 126 | + list_for_each_entry(memtype, &memtier->memory_types, tier_sibling) | |
| 119 | 127 | nodes_or(nodes, nodes, memtype->nodes); |
| 120 | 128 | |
| 121 | 129 | return nodes; |
| @@ -174,7 +182,7 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · 52 unmodified lines | ||
| 174 | 182 | * If the memtype is already part of a memory tier, |
| 175 | 183 | * just return that. |
| 176 | 184 | */ |
| 177 | − if (!list_empty(&memtype->tier_sibiling)) { | |
| 185 | + if (!list_empty(&memtype->tier_sibling)) { | |
| 178 | 186 | list_for_each_entry(memtier, &memory_tiers, list) { |
| 179 | 187 | if (adistance == memtier->adistance_start) |
| 180 | 188 | return memtier; |
| @@ -218,7 +226,7 @@ static struct memory_tier *find_create_memory_tier(struct memory_dev_type *memty · 37 unmodified lines | ||
| 218 | 226 | memtier = new_memtier; |
| 219 | 227 | |
| 220 | 228 | link_memtype: |
| 221 | − list_add(&memtype->tier_sibiling, &memtier->memory_types); | |
| 229 | + list_add(&memtype->tier_sibling, &memtier->memory_types); | |
| 222 | 230 | return memtier; |
| 223 | 231 | } |
| 224 | 232 | |
| @@ -527,7 +535,7 @@ static bool clear_node_memory_tier(int node) · 302 unmodified lines | ||
| 527 | 535 | memtype = node_memory_types[node].memtype; |
| 528 | 536 | node_clear(node, memtype->nodes); |
| 529 | 537 | if (nodes_empty(memtype->nodes)) { |
| 530 | − list_del_init(&memtype->tier_sibiling); | |
| 538 | + list_del_init(&memtype->tier_sibling); | |
| 531 | 539 | if (list_empty(&memtier->memory_types)) |
| 532 | 540 | destroy_memory_tier(memtier); |
| 533 | 541 | } |
| @@ -553,7 +561,7 @@ struct memory_dev_type *alloc_memory_type(int adistance) · 19 unmodified lines | ||
| 553 | 561 | return ERR_PTR(-ENOMEM); |
| 554 | 562 | |
| 555 | 563 | memtype->adistance = adistance; |
| 556 | − INIT_LIST_HEAD(&memtype->tier_sibiling); | |
| 564 | + INIT_LIST_HEAD(&memtype->tier_sibling); | |
| 557 | 565 | memtype->nodes = NODE_MASK_NONE; |
| 558 | 566 | kref_init(&memtype->kref); |
| 559 | 567 | return memtype; |
| @@ -578,13 +586,14 @@ EXPORT_SYMBOL_GPL(init_node_memory_type); · 18 unmodified lines | ||
| 578 | 586 | void clear_node_memory_type(int node, struct memory_dev_type *memtype) |
| 579 | 587 | { |
| 580 | 588 | mutex_lock(&memory_tier_lock); |
| 581 | − if (node_memory_types[node].memtype == memtype) | |
| 589 | + if (node_memory_types[node].memtype == memtype || !memtype) | |
| 582 | 590 | node_memory_types[node].map_count--; |
| 583 | 591 | /* |
| 584 | 592 | * If we umapped all the attached devices to this node, |
| 585 | 593 | * clear the node memory type. |
| 586 | 594 | */ |
| 587 | 595 | if (!node_memory_types[node].map_count) { |
| 596 | + memtype = node_memory_types[node].memtype; | |
| 588 | 597 | node_memory_types[node].memtype = NULL; |
| 589 | 598 | put_memory_type(memtype); |
| 590 | 599 | } |
| @@ -592,6 +601,158 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype) · 1 unmodified line | ||
| 592 | 601 | } |
| 593 | 602 | EXPORT_SYMBOL_GPL(clear_node_memory_type); |
| 594 | 603 | |
| 604 | +static void dump_hmem_attrs(struct node_hmem_attrs *attrs, const char *prefix) | |
| 605 | +{ | |
| 606 | + pr_info( | |
| 607 | +"%sread_latency: %u, write_latency: %u, read_bandwidth: %u, write_bandwidth: %u\n", | |
| 608 | + prefix, attrs->read_latency, attrs->write_latency, | |
| 609 | + attrs->read_bandwidth, attrs->write_bandwidth); | |
| 610 | +} | |
| 611 | + | |
| 612 | +int mt_set_default_dram_perf(int nid, struct node_hmem_attrs *perf, | |
| 613 | + const char *source) | |
| 614 | +{ | |
| 615 | + int rc = 0; | |
| 616 | + | |
| 617 | + mutex_lock(&memory_tier_lock); | |
| 618 | + if (default_dram_perf_error) { | |
| 619 | + rc = -EIO; | |
| 620 | + goto out; | |
| 621 | + } | |
| 622 | + | |
| 623 | + if (perf->read_latency + perf->write_latency == 0 || | |
| 624 | + perf->read_bandwidth + perf->write_bandwidth == 0) { | |
| 625 | + rc = -EINVAL; | |
| 626 | + goto out; | |
| 627 | + } | |
| 628 | + | |
| 629 | + if (default_dram_perf_ref_nid == NUMA_NO_NODE) { | |
| 630 | + default_dram_perf = *perf; | |
| 631 | + default_dram_perf_ref_nid = nid; | |
| 632 | + default_dram_perf_ref_source = kstrdup(source, GFP_KERNEL); | |
| 633 | + goto out; | |
| 634 | + } | |
| 635 | + | |
| 636 | + /* | |
| 637 | + * The performance of all default DRAM nodes is expected to be | |
| 638 | + * same (that is, the variation is less than 10%). And it | |
| 639 | + * will be used as base to calculate the abstract distance of | |
| 640 | + * other memory nodes. | |
| 641 | + */ | |
| 642 | + if (abs(perf->read_latency - default_dram_perf.read_latency) * 10 > | |
| 643 | + default_dram_perf.read_latency || | |
| 644 | + abs(perf->write_latency - default_dram_perf.write_latency) * 10 > | |
| 645 | + default_dram_perf.write_latency || | |
| 646 | + abs(perf->read_bandwidth - default_dram_perf.read_bandwidth) * 10 > | |
| 647 | + default_dram_perf.read_bandwidth || | |
| 648 | + abs(perf->write_bandwidth - default_dram_perf.write_bandwidth) * 10 > | |
| 649 | + default_dram_perf.write_bandwidth) { | |
| 650 | + pr_info( | |
| 651 | +"memory-tiers: the performance of DRAM node %d mismatches that of the reference\n" | |
| 652 | +"DRAM node %d.\n", nid, default_dram_perf_ref_nid); | |
| 653 | + pr_info(" performance of reference DRAM node %d:\n", | |
| 654 | + default_dram_perf_ref_nid); | |
| 655 | + dump_hmem_attrs(&default_dram_perf, " "); | |
| 656 | + pr_info(" performance of DRAM node %d:\n", nid); | |
| 657 | + dump_hmem_attrs(perf, " "); | |
| 658 | + pr_info( | |
| 659 | +" disable default DRAM node performance based abstract distance algorithm.\n"); | |
| 660 | + default_dram_perf_error = true; | |
| 661 | + rc = -EINVAL; | |
| 662 | + } | |
| 663 | + | |
| 664 | +out: | |
| 665 | + mutex_unlock(&memory_tier_lock); | |
| 666 | + return rc; | |
| 667 | +} | |
| 668 | + | |
| 669 | +int mt_perf_to_adistance(struct node_hmem_attrs *perf, int *adist) | |
| 670 | +{ | |
| 671 | + if (default_dram_perf_error) | |
| 672 | + return -EIO; | |
| 673 | + | |
| 674 | + if (default_dram_perf_ref_nid == NUMA_NO_NODE) | |
| 675 | + return -ENOENT; | |
| 676 | + | |
| 677 | + if (perf->read_latency + perf->write_latency == 0 || | |
| 678 | + perf->read_bandwidth + perf->write_bandwidth == 0) | |
| 679 | + return -EINVAL; | |
| 680 | + | |
| 681 | + mutex_lock(&memory_tier_lock); | |
| 682 | + /* | |
| 683 | + * The abstract distance of a memory node is in direct proportion to | |
| 684 | + * its memory latency (read + write) and inversely proportional to its | |
| 685 | + * memory bandwidth (read + write). The abstract distance, memory | |
| 686 | + * latency, and memory bandwidth of the default DRAM nodes are used as | |
| 687 | + * the base. | |
| 688 | + */ | |
| 689 | + *adist = MEMTIER_ADISTANCE_DRAM * | |
| 690 | + (perf->read_latency + perf->write_latency) / | |
| 691 | + (default_dram_perf.read_latency + default_dram_perf.write_latency) * | |
| 692 | + (default_dram_perf.read_bandwidth + default_dram_perf.write_bandwidth) / | |
| 693 | + (perf->read_bandwidth + perf->write_bandwidth); | |
| 694 | + mutex_unlock(&memory_tier_lock); | |
| 695 | + | |
| 696 | + return 0; | |
| 697 | +} | |
| 698 | +EXPORT_SYMBOL_GPL(mt_perf_to_adistance); | |
| 699 | + | |
| 700 | +/** | |
| 701 | + * register_mt_adistance_algorithm() - Register memory tiering abstract distance algorithm | |
| 702 | + * @nb: The notifier block which describe the algorithm | |
| 703 | + * | |
| 704 | + * Return: 0 on success, errno on error. | |
| 705 | + * | |
| 706 | + * Every memory tiering abstract distance algorithm provider needs to | |
| 707 | + * register the algorithm with register_mt_adistance_algorithm(). To | |
| 708 | + * calculate the abstract distance for a specified memory node, the | |
| 709 | + * notifier function will be called unless some high priority | |
| 710 | + * algorithm has provided result. The prototype of the notifier | |
| 711 | + * function is as follows, | |
| 712 | + * | |
| 713 | + * int (*algorithm_notifier)(struct notifier_block *nb, | |
| 714 | + * unsigned long nid, void *data); | |
| 715 | + * | |
| 716 | + * Where "nid" specifies the memory node, "data" is the pointer to the | |
| 717 | + * returned abstract distance (that is, "int *adist"). If the | |
| 718 | + * algorithm provides the result, NOTIFY_STOP should be returned. | |
| 719 | + * Otherwise, return_value & %NOTIFY_STOP_MASK == 0 to allow the next | |
| 720 | + * algorithm in the chain to provide the result. | |
| 721 | + */ | |
| 722 | +int register_mt_adistance_algorithm(struct notifier_block *nb) | |
| 723 | +{ | |
| 724 | + return blocking_notifier_chain_register(&mt_adistance_algorithms, nb); | |
| 725 | +} | |
| 726 | +EXPORT_SYMBOL_GPL(register_mt_adistance_algorithm); | |
| 727 | + | |
| 728 | +/** | |
| 729 | + * unregister_mt_adistance_algorithm() - Unregister memory tiering abstract distance algorithm | |
| 730 | + * @nb: the notifier block which describe the algorithm | |
| 731 | + * | |
| 732 | + * Return: 0 on success, errno on error. | |
| 733 | + */ | |
| 734 | +int unregister_mt_adistance_algorithm(struct notifier_block *nb) | |
| 735 | +{ | |
| 736 | + return blocking_notifier_chain_unregister(&mt_adistance_algorithms, nb); | |
| 737 | +} | |
| 738 | +EXPORT_SYMBOL_GPL(unregister_mt_adistance_algorithm); | |
| 739 | + | |
| 740 | +/** | |
| 741 | + * mt_calc_adistance() - Calculate abstract distance with registered algorithms | |
| 742 | + * @node: the node to calculate abstract distance for | |
| 743 | + * @adist: the returned abstract distance | |
| 744 | + * | |
| 745 | + * Return: if return_value & %NOTIFY_STOP_MASK != 0, then some | |
| 746 | + * abstract distance algorithm provides the result, and return it via | |
| 747 | + * @adist. Otherwise, no algorithm can provide the result and @adist | |
| 748 | + * will be kept as it is. | |
| 749 | + */ | |
| 750 | +int mt_calc_adistance(int node, int *adist) | |
| 751 | +{ | |
| 752 | + return blocking_notifier_call_chain(&mt_adistance_algorithms, node, adist); | |
| 753 | +} | |
| 754 | +EXPORT_SYMBOL_GPL(mt_calc_adistance); | |
| 755 | + | |
| 595 | 756 | static int __meminit memtier_hotplug_callback(struct notifier_block *self, |
| 596 | 757 | unsigned long action, void *_arg) |
| 597 | 758 | { |
| @@ -1,3 +1,4 @@ | ||
| 1 | + | |
| 1 | 2 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 3 | /* |
| 3 | 4 | * linux/mm/memory.c |
| @@ -471,8 +472,6 @@ static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss) · 467 unmodified lines | ||
| 471 | 472 | { |
| 472 | 473 | int i; |
| 473 | 474 | |
| 474 | − if (current->mm == mm) | |
| 475 | − sync_mm_rss(mm); | |
| 476 | 475 | for (i = 0; i < NR_MM_COUNTERS; i++) |
| 477 | 476 | if (rss[i]) |
| 478 | 477 | add_mm_counter(mm, i, rss[i]); |
| @@ -691,6 +690,16 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, · 212 unmodified lines | ||
| 691 | 690 | out: |
| 692 | 691 | return pfn_to_page(pfn); |
| 693 | 692 | } |
| 693 | + | |
| 694 | +struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, | |
| 695 | + unsigned long addr, pmd_t pmd) | |
| 696 | +{ | |
| 697 | + struct page *page = vm_normal_page_pmd(vma, addr, pmd); | |
| 698 | + | |
| 699 | + if (page) | |
| 700 | + return page_folio(page); | |
| 701 | + return NULL; | |
| 702 | +} | |
| 694 | 703 | #endif |
| 695 | 704 | |
| 696 | 705 | static void restore_exclusive_pte(struct vm_area_struct *vma, |
| @@ -3006,23 +3015,24 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) · 2309 unmodified lines | ||
| 3006 | 3015 | * case, all we need to do here is to mark the page as writable and update |
| 3007 | 3016 | * any related book-keeping. |
| 3008 | 3017 | */ |
| 3009 | −static inline void wp_page_reuse(struct vm_fault *vmf) | |
| 3018 | +static inline void wp_page_reuse(struct vm_fault *vmf, struct folio *folio) | |
| 3010 | 3019 | __releases(vmf->ptl) |
| 3011 | 3020 | { |
| 3012 | 3021 | struct vm_area_struct *vma = vmf->vma; |
| 3013 | − struct page *page = vmf->page; | |
| 3014 | 3022 | pte_t entry; |
| 3015 | 3023 | |
| 3016 | 3024 | VM_BUG_ON(!(vmf->flags & FAULT_FLAG_WRITE)); |
| 3017 | − VM_BUG_ON(page && PageAnon(page) && !PageAnonExclusive(page)); | |
| 3018 | 3025 | |
| 3019 | − /* | |
| 3020 | − * Clear the pages cpupid information as the existing | |
| 3021 | − * information potentially belongs to a now completely | |
| 3022 | − * unrelated process. | |
| 3023 | − */ | |
| 3024 | − if (page) | |
| 3025 | − page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1); | |
| 3026 | + if (folio) { | |
| 3027 | + VM_BUG_ON(folio_test_anon(folio) && | |
| 3028 | + !PageAnonExclusive(vmf->page)); | |
| 3029 | + /* | |
| 3030 | + * Clear the folio's cpupid information as the existing | |
| 3031 | + * information potentially belongs to a now completely | |
| 3032 | + * unrelated process. | |
| 3033 | + */ | |
| 3034 | + folio_xchg_last_cpupid(folio, (1 << LAST_CPUPID_SHIFT) - 1); | |
| 3035 | + } | |
| 3026 | 3036 | |
| 3027 | 3037 | flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); |
| 3028 | 3038 | entry = pte_mkyoung(vmf->orig_pte); |
| @@ -3033,6 +3043,36 @@ static inline void wp_page_reuse(struct vm_fault *vmf) · 4 unmodified lines | ||
| 3033 | 3043 | count_vm_event(PGREUSE); |
| 3034 | 3044 | } |
| 3035 | 3045 | |
| 3046 | +/* | |
| 3047 | + * We could add a bitflag somewhere, but for now, we know that all | |
| 3048 | + * vm_ops that have a ->map_pages have been audited and don't need | |
| 3049 | + * the mmap_lock to be held. | |
| 3050 | + */ | |
| 3051 | +static inline vm_fault_t vmf_can_call_fault(const struct vm_fault *vmf) | |
| 3052 | +{ | |
| 3053 | + struct vm_area_struct *vma = vmf->vma; | |
| 3054 | + | |
| 3055 | + if (vma->vm_ops->map_pages || !(vmf->flags & FAULT_FLAG_VMA_LOCK)) | |
| 3056 | + return 0; | |
| 3057 | + vma_end_read(vma); | |
| 3058 | + return VM_FAULT_RETRY; | |
| 3059 | +} | |
| 3060 | + | |
| 3061 | +static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) | |
| 3062 | +{ | |
| 3063 | + struct vm_area_struct *vma = vmf->vma; | |
| 3064 | + | |
| 3065 | + if (likely(vma->anon_vma)) | |
| 3066 | + return 0; | |
| 3067 | + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { | |
| 3068 | + vma_end_read(vma); | |
| 3069 | + return VM_FAULT_RETRY; | |
| 3070 | + } | |
| 3071 | + if (__anon_vma_prepare(vma)) | |
| 3072 | + return VM_FAULT_OOM; | |
| 3073 | + return 0; | |
| 3074 | +} | |
| 3075 | + | |
| 3036 | 3076 | /* |
| 3037 | 3077 | * Handle the case of a page which we actually need to copy to a new page, |
| 3038 | 3078 | * either due to COW or unsharing. |
| @@ -3060,27 +3100,29 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 21 unmodified lines | ||
| 3060 | 3100 | pte_t entry; |
| 3061 | 3101 | int page_copied = 0; |
| 3062 | 3102 | struct mmu_notifier_range range; |
| 3063 | − int ret; | |
| 3103 | + vm_fault_t ret; | |
| 3064 | 3104 | |
| 3065 | 3105 | delayacct_wpcopy_start(); |
| 3066 | 3106 | |
| 3067 | 3107 | if (vmf->page) |
| 3068 | 3108 | old_folio = page_folio(vmf->page); |
| 3069 | − if (unlikely(anon_vma_prepare(vma))) | |
| 3070 | − goto oom; | |
| 3109 | + ret = vmf_anon_prepare(vmf); | |
| 3110 | + if (unlikely(ret)) | |
| 3111 | + goto out; | |
| 3071 | 3112 | |
| 3072 | 3113 | if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { |
| 3073 | 3114 | new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); |
| 3074 | 3115 | if (!new_folio) |
| 3075 | 3116 | goto oom; |
| 3076 | 3117 | } else { |
| 3118 | + int err; | |
| 3077 | 3119 | new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, |
| 3078 | 3120 | vmf->address, false); |
| 3079 | 3121 | if (!new_folio) |
| 3080 | 3122 | goto oom; |
| 3081 | 3123 | |
| 3082 | − ret = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); | |
| 3083 | − if (ret) { | |
| 3124 | + err = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); | |
| 3125 | + if (err) { | |
| 3084 | 3126 | /* |
| 3085 | 3127 | * COW failed, if the fault was solved by other, |
| 3086 | 3128 | * it's fine. If not, userspace would re-fault on |
| @@ -3093,7 +3135,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 6 unmodified lines | ||
| 3093 | 3135 | folio_put(old_folio); |
| 3094 | 3136 | |
| 3095 | 3137 | delayacct_wpcopy_end(); |
| 3096 | − return ret == -EHWPOISON ? VM_FAULT_HWPOISON : 0; | |
| 3138 | + return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0; | |
| 3097 | 3139 | } |
| 3098 | 3140 | kmsan_copy_page_meta(&new_folio->page, vmf->page); |
| 3099 | 3141 | } |
| @@ -3203,11 +3245,13 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) · 103 unmodified lines | ||
| 3203 | 3245 | oom_free_new: |
| 3204 | 3246 | folio_put(new_folio); |
| 3205 | 3247 | oom: |
| 3248 | + ret = VM_FAULT_OOM; | |
| 3249 | +out: | |
| 3206 | 3250 | if (old_folio) |
| 3207 | 3251 | folio_put(old_folio); |
| 3208 | 3252 | |
| 3209 | 3253 | delayacct_wpcopy_end(); |
| 3210 | − return VM_FAULT_OOM; | |
| 3254 | + return ret; | |
| 3211 | 3255 | } |
| 3212 | 3256 | |
| 3213 | 3257 | /** |
| @@ -3215,6 +3259,7 @@ oom: · 1 unmodified line | ||
| 3215 | 3259 | * writeable once the page is prepared |
| 3216 | 3260 | * |
| 3217 | 3261 | * @vmf: structure describing the fault |
| 3262 | + * @folio: the folio of vmf->page | |
| 3218 | 3263 | * |
| 3219 | 3264 | * This function handles all that is needed to finish a write page fault in a |
| 3220 | 3265 | * shared mapping due to PTE being read-only once the mapped page is prepared. |
| @@ -3226,7 +3271,7 @@ oom: · 5 unmodified lines | ||
| 3226 | 3271 | * Return: %0 on success, %VM_FAULT_NOPAGE when PTE got changed before |
| 3227 | 3272 | * we acquired PTE lock. |
| 3228 | 3273 | */ |
| 3229 | −vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) | |
| 3274 | +static vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf, struct folio *folio) | |
| 3230 | 3275 | { |
| 3231 | 3276 | WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED)); |
| 3232 | 3277 | vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, |
| @@ -3242,7 +3287,7 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) · 9 unmodified lines | ||
| 3242 | 3287 | pte_unmap_unlock(vmf->pte, vmf->ptl); |
| 3243 | 3288 | return VM_FAULT_NOPAGE; |
| 3244 | 3289 | } |
| 3245 | − wp_page_reuse(vmf); | |
| 3290 | + wp_page_reuse(vmf, folio); | |
| 3246 | 3291 | return 0; |
| 3247 | 3292 | } |
| 3248 | 3293 | |
| @@ -3258,18 +3303,17 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) · 9 unmodified lines | ||
| 3258 | 3303 | vm_fault_t ret; |
| 3259 | 3304 | |
| 3260 | 3305 | pte_unmap_unlock(vmf->pte, vmf->ptl); |
| 3261 | − if (vmf->flags & FAULT_FLAG_VMA_LOCK) { | |
| 3262 | − vma_end_read(vmf->vma); | |
| 3263 | − return VM_FAULT_RETRY; | |
| 3264 | − } | |
| 3306 | + ret = vmf_can_call_fault(vmf); | |
| 3307 | + if (ret) | |
| 3308 | + return ret; | |
| 3265 | 3309 | |
| 3266 | 3310 | vmf->flags |= FAULT_FLAG_MKWRITE; |
| 3267 | 3311 | ret = vma->vm_ops->pfn_mkwrite(vmf); |
| 3268 | 3312 | if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)) |
| 3269 | 3313 | return ret; |
| 3270 | − return finish_mkwrite_fault(vmf); | |
| 3314 | + return finish_mkwrite_fault(vmf, NULL); | |
| 3271 | 3315 | } |
| 3272 | − wp_page_reuse(vmf); | |
| 3316 | + wp_page_reuse(vmf, NULL); | |
| 3273 | 3317 | return 0; |
| 3274 | 3318 | } |
| 3275 | 3319 | |
| @@ -3285,10 +3329,10 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 9 unmodified lines | ||
| 3285 | 3329 | vm_fault_t tmp; |
| 3286 | 3330 | |
| 3287 | 3331 | pte_unmap_unlock(vmf->pte, vmf->ptl); |
| 3288 | − if (vmf->flags & FAULT_FLAG_VMA_LOCK) { | |
| 3332 | + tmp = vmf_can_call_fault(vmf); | |
| 3333 | + if (tmp) { | |
| 3289 | 3334 | folio_put(folio); |
| 3290 | − vma_end_read(vmf->vma); | |
| 3291 | − return VM_FAULT_RETRY; | |
| 3335 | + return tmp; | |
| 3292 | 3336 | } |
| 3293 | 3337 | |
| 3294 | 3338 | tmp = do_page_mkwrite(vmf, folio); |
| @@ -3297,14 +3341,14 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 2 unmodified lines | ||
| 3297 | 3341 | folio_put(folio); |
| 3298 | 3342 | return tmp; |
| 3299 | 3343 | } |
| 3300 | − tmp = finish_mkwrite_fault(vmf); | |
| 3344 | + tmp = finish_mkwrite_fault(vmf, folio); | |
| 3301 | 3345 | if (unlikely(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) { |
| 3302 | 3346 | folio_unlock(folio); |
| 3303 | 3347 | folio_put(folio); |
| 3304 | 3348 | return tmp; |
| 3305 | 3349 | } |
| 3306 | 3350 | } else { |
| 3307 | − wp_page_reuse(vmf); | |
| 3351 | + wp_page_reuse(vmf, folio); | |
| 3308 | 3352 | folio_lock(folio); |
| 3309 | 3353 | } |
| 3310 | 3354 | ret |= fault_dirty_shared_page(vmf); |
| @@ -3313,6 +3357,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) · 2 unmodified lines | ||
| 3313 | 3357 | return ret; |
| 3314 | 3358 | } |
| 3315 | 3359 | |
| 3360 | +static bool wp_can_reuse_anon_folio(struct folio *folio, | |
| 3361 | + struct vm_area_struct *vma) | |
| 3362 | +{ | |
| 3363 | + /* | |
| 3364 | + * We have to verify under folio lock: these early checks are | |
| 3365 | + * just an optimization to avoid locking the folio and freeing | |
| 3366 | + * the swapcache if there is little hope that we can reuse. | |
| 3367 | + * | |
| 3368 | + * KSM doesn't necessarily raise the folio refcount. | |
| 3369 | + */ | |
| 3370 | + if (folio_test_ksm(folio) || folio_ref_count(folio) > 3) | |
| 3371 | + return false; | |
| 3372 | + if (!folio_test_lru(folio)) | |
| 3373 | + /* | |
| 3374 | + * We cannot easily detect+handle references from | |
| 3375 | + * remote LRU caches or references to LRU folios. | |
| 3376 | + */ | |
| 3377 | + lru_add_drain(); | |
| 3378 | + if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) | |
| 3379 | + return false; | |
| 3380 | + if (!folio_trylock(folio)) | |
| 3381 | + return false; | |
| 3382 | + if (folio_test_swapcache(folio)) | |
| 3383 | + folio_free_swap(folio); | |
| 3384 | + if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) { | |
| 3385 | + folio_unlock(folio); | |
| 3386 | + return false; | |
| 3387 | + } | |
| 3388 | + /* | |
| 3389 | + * Ok, we've got the only folio reference from our mapping | |
| 3390 | + * and the folio is locked, it's dark out, and we're wearing | |
| 3391 | + * sunglasses. Hit it. | |
| 3392 | + */ | |
| 3393 | + folio_move_anon_rmap(folio, vma); | |
| 3394 | + folio_unlock(folio); | |
| 3395 | + return true; | |
| 3396 | +} | |
| 3397 | + | |
| 3316 | 3398 | /* |
| 3317 | 3399 | * This routine handles present pages, when |
| 3318 | 3400 | * * users try to write to a shared page (FAULT_FLAG_WRITE) |
| @@ -3341,11 +3423,28 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) · 22 unmodified lines | ||
| 3341 | 3423 | const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; |
| 3342 | 3424 | struct vm_area_struct *vma = vmf->vma; |
| 3343 | 3425 | struct folio *folio = NULL; |
| 3426 | + pte_t pte; | |
| 3344 | 3427 | |
| 3345 | 3428 | if (likely(!unshare)) { |
| 3346 | 3429 | if (userfaultfd_pte_wp(vma, ptep_get(vmf->pte))) { |
| 3347 | − pte_unmap_unlock(vmf->pte, vmf->ptl); | |
| 3348 | − return handle_userfault(vmf, VM_UFFD_WP); | |
| 3430 | + if (!userfaultfd_wp_async(vma)) { | |
| 3431 | + pte_unmap_unlock(vmf->pte, vmf->ptl); | |
| 3432 | + return handle_userfault(vmf, VM_UFFD_WP); | |
| 3433 | + } | |
| 3434 | + | |
| 3435 | + /* | |
| 3436 | + * Nothing needed (cache flush, TLB invalidations, | |
| 3437 | + * etc.) because we're only removing the uffd-wp bit, | |
| 3438 | + * which is completely invisible to the user. | |
| 3439 | + */ | |
| 3440 | + pte = pte_clear_uffd_wp(ptep_get(vmf->pte)); | |
| 3441 | + | |
| 3442 | + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); | |
| 3443 | + /* | |
| 3444 | + * Update this to be prepared for following up CoW | |
| 3445 | + * handling | |
| 3446 | + */ | |
| 3447 | + vmf->orig_pte = pte; | |
| 3349 | 3448 | } |
| 3350 | 3449 | |
| 3351 | 3450 | /* |
| @@ -3382,62 +3481,21 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) · 30 unmodified lines | ||
| 3382 | 3481 | /* |
| 3383 | 3482 | * Private mapping: create an exclusive anonymous page copy if reuse |
| 3384 | 3483 | * is impossible. We might miss VM_WRITE for FOLL_FORCE handling. |
| 3484 | + * | |
| 3485 | + * If we encounter a page that is marked exclusive, we must reuse | |
| 3486 | + * the page without further checks. | |
| 3385 | 3487 | */ |
| 3386 | − if (folio && folio_test_anon(folio)) { | |
| 3387 | − /* | |
| 3388 | − * If the page is exclusive to this process we must reuse the | |
| 3389 | − * page without further checks. | |
| 3390 | − */ | |
| 3391 | − if (PageAnonExclusive(vmf->page)) | |
| 3392 | − goto reuse; | |
| 3393 | − | |
| 3394 | − /* | |
| 3395 | − * We have to verify under folio lock: these early checks are | |
| 3396 | − * just an optimization to avoid locking the folio and freeing | |
| 3397 | − * the swapcache if there is little hope that we can reuse. | |
| 3398 | − * | |
| 3399 | − * KSM doesn't necessarily raise the folio refcount. | |
| 3400 | − */ | |
| 3401 | − if (folio_test_ksm(folio) || folio_ref_count(folio) > 3) | |
| 3402 | − goto copy; | |
| 3403 | − if (!folio_test_lru(folio)) | |
| 3404 | − /* | |
| 3405 | − * We cannot easily detect+handle references from | |
| 3406 | − * remote LRU caches or references to LRU folios. | |
| 3407 | − */ | |
| 3408 | − lru_add_drain(); | |
| 3409 | − if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) | |
| 3410 | − goto copy; | |
| 3411 | − if (!folio_trylock(folio)) | |
| 3412 | − goto copy; | |
| 3413 | − if (folio_test_swapcache(folio)) | |
| 3414 | − folio_free_swap(folio); | |
| 3415 | − if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) { | |
| 3416 | − folio_unlock(folio); | |
| 3417 | − goto copy; | |
| 3418 | − } | |
| 3419 | − /* | |
| 3420 | − * Ok, we've got the only folio reference from our mapping | |
| 3421 | − * and the folio is locked, it's dark out, and we're wearing | |
| 3422 | − * sunglasses. Hit it. | |
| 3423 | − */ | |
| 3424 | − page_move_anon_rmap(vmf->page, vma); | |
| 3425 | − folio_unlock(folio); | |
| 3426 | −reuse: | |
| 3488 | + if (folio && folio_test_anon(folio) && | |
| 3489 | + (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) { | |
| 3490 | + if (!PageAnonExclusive(vmf->page)) | |
| 3491 | + SetPageAnonExclusive(vmf->page); | |
| 3427 | 3492 | if (unlikely(unshare)) { |
| 3428 | 3493 | pte_unmap_unlock(vmf->pte, vmf->ptl); |
| 3429 | 3494 | return 0; |
| 3430 | 3495 | } |
| 3431 | − wp_page_reuse(vmf); | |
| 3496 | + wp_page_reuse(vmf, folio); | |
| 3432 | 3497 | return 0; |
| 3433 | 3498 | } |
| 3434 | −copy: | |
| 3435 | − if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma->anon_vma) { | |
| 3436 | − pte_unmap_unlock(vmf->pte, vmf->ptl); | |
| 3437 | − vma_end_read(vmf->vma); | |
| 3438 | − return VM_FAULT_RETRY; | |
| 3439 | − } | |
| 3440 | − | |
| 3441 | 3499 | /* |
| 3442 | 3500 | * Ok, we need to copy. Oh, well.. |
| 3443 | 3501 | */ |
| @@ -4563,10 +4621,9 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) · 1119 unmodified lines | ||
| 4563 | 4621 | return ret; |
| 4564 | 4622 | } |
| 4565 | 4623 | |
| 4566 | − if (vmf->flags & FAULT_FLAG_VMA_LOCK) { | |
| 4567 | − vma_end_read(vmf->vma); | |
| 4568 | − return VM_FAULT_RETRY; | |
| 4569 | − } | |
| 4624 | + ret = vmf_can_call_fault(vmf); | |
| 4625 | + if (ret) | |
| 4626 | + return ret; | |
| 4570 | 4627 | |
| 4571 | 4628 | ret = __do_fault(vmf); |
| 4572 | 4629 | if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) |
| @@ -4585,13 +4642,11 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) · 12 unmodified lines | ||
| 4585 | 4642 | struct vm_area_struct *vma = vmf->vma; |
| 4586 | 4643 | vm_fault_t ret; |
| 4587 | 4644 | |
| 4588 | − if (vmf->flags & FAULT_FLAG_VMA_LOCK) { | |
| 4589 | − vma_end_read(vma); | |
| 4590 | − return VM_FAULT_RETRY; | |
| 4591 | − } | |
| 4592 | − | |
| 4593 | − if (unlikely(anon_vma_prepare(vma))) | |
| 4594 | − return VM_FAULT_OOM; | |
| 4645 | + ret = vmf_can_call_fault(vmf); | |
| 4646 | + if (!ret) | |
| 4647 | + ret = vmf_anon_prepare(vmf); | |
| 4648 | + if (ret) | |
| 4649 | + return ret; | |
| 4595 | 4650 | |
| 4596 | 4651 | vmf->cow_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); |
| 4597 | 4652 | if (!vmf->cow_page) |
| @@ -4630,10 +4685,9 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) · 32 unmodified lines | ||
| 4630 | 4685 | vm_fault_t ret, tmp; |
| 4631 | 4686 | struct folio *folio; |
| 4632 | 4687 | |
| 4633 | − if (vmf->flags & FAULT_FLAG_VMA_LOCK) { | |
| 4634 | − vma_end_read(vma); | |
| 4635 | − return VM_FAULT_RETRY; | |
| 4636 | − } | |
| 4688 | + ret = vmf_can_call_fault(vmf); | |
| 4689 | + if (ret) | |
| 4690 | + return ret; | |
| 4637 | 4691 | |
| 4638 | 4692 | ret = __do_fault(vmf); |
| 4639 | 4693 | if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) |
| @@ -4719,10 +4773,10 @@ static vm_fault_t do_fault(struct vm_fault *vmf) · 79 unmodified lines | ||
| 4719 | 4773 | return ret; |
| 4720 | 4774 | } |
| 4721 | 4775 | |
| 4722 | −int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, | |
| 4776 | +int numa_migrate_prep(struct folio *folio, struct vm_area_struct *vma, | |
| 4723 | 4777 | unsigned long addr, int page_nid, int *flags) |
| 4724 | 4778 | { |
| 4725 | − get_page(page); | |
| 4779 | + folio_get(folio); | |
| 4726 | 4780 | |
| 4727 | 4781 | /* Record the current PID acceesing VMA */ |
| 4728 | 4782 | vma_set_access_pid_bit(vma); |
| @@ -4733,14 +4787,14 @@ int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, · 4 unmodified lines | ||
| 4733 | 4787 | *flags |= TNF_FAULT_LOCAL; |
| 4734 | 4788 | } |
| 4735 | 4789 | |
| 4736 | − return mpol_misplaced(page, vma, addr); | |
| 4790 | + return mpol_misplaced(folio, vma, addr); | |
| 4737 | 4791 | } |
| 4738 | 4792 | |
| 4739 | 4793 | static vm_fault_t do_numa_page(struct vm_fault *vmf) |
| 4740 | 4794 | { |
| 4741 | 4795 | struct vm_area_struct *vma = vmf->vma; |
| 4742 | − struct page *page = NULL; | |
| 4743 | − int page_nid = NUMA_NO_NODE; | |
| 4796 | + struct folio *folio = NULL; | |
| 4797 | + int nid = NUMA_NO_NODE; | |
| 4744 | 4798 | bool writable = false; |
| 4745 | 4799 | int last_cpupid; |
| 4746 | 4800 | int target_nid; |
| @@ -4771,12 +4825,12 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 24 unmodified lines | ||
| 4771 | 4825 | can_change_pte_writable(vma, vmf->address, pte)) |
| 4772 | 4826 | writable = true; |
| 4773 | 4827 | |
| 4774 | − page = vm_normal_page(vma, vmf->address, pte); | |
| 4775 | − if (!page || is_zone_device_page(page)) | |
| 4828 | + folio = vm_normal_folio(vma, vmf->address, pte); | |
| 4829 | + if (!folio || folio_is_zone_device(folio)) | |
| 4776 | 4830 | goto out_map; |
| 4777 | 4831 | |
| 4778 | 4832 | /* TODO: handle PTE-mapped THP */ |
| 4779 | − if (PageCompound(page)) | |
| 4833 | + if (folio_test_large(folio)) | |
| 4780 | 4834 | goto out_map; |
| 4781 | 4835 | |
| 4782 | 4836 | /* |
| @@ -4791,34 +4845,33 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 8 unmodified lines | ||
| 4791 | 4845 | flags |= TNF_NO_GROUP; |
| 4792 | 4846 | |
| 4793 | 4847 | /* |
| 4794 | − * Flag if the page is shared between multiple address spaces. This | |
| 4848 | + * Flag if the folio is shared between multiple address spaces. This | |
| 4795 | 4849 | * is later used when determining whether to group tasks together |
| 4796 | 4850 | */ |
| 4797 | − if (page_mapcount(page) > 1 && (vma->vm_flags & VM_SHARED)) | |
| 4851 | + if (folio_estimated_sharers(folio) > 1 && (vma->vm_flags & VM_SHARED)) | |
| 4798 | 4852 | flags |= TNF_SHARED; |
| 4799 | 4853 | |
| 4800 | − page_nid = page_to_nid(page); | |
| 4854 | + nid = folio_nid(folio); | |
| 4801 | 4855 | /* |
| 4802 | 4856 | * For memory tiering mode, cpupid of slow memory page is used |
| 4803 | 4857 | * to record page access time. So use default value. |
| 4804 | 4858 | */ |
| 4805 | 4859 | if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && |
| 4806 | − !node_is_toptier(page_nid)) | |
| 4860 | + !node_is_toptier(nid)) | |
| 4807 | 4861 | last_cpupid = (-1 & LAST_CPUPID_MASK); |
| 4808 | 4862 | else |
| 4809 | − last_cpupid = page_cpupid_last(page); | |
| 4810 | − target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid, | |
| 4811 | − &flags); | |
| 4863 | + last_cpupid = folio_last_cpupid(folio); | |
| 4864 | + target_nid = numa_migrate_prep(folio, vma, vmf->address, nid, &flags); | |
| 4812 | 4865 | if (target_nid == NUMA_NO_NODE) { |
| 4813 | − put_page(page); | |
| 4866 | + folio_put(folio); | |
| 4814 | 4867 | goto out_map; |
| 4815 | 4868 | } |
| 4816 | 4869 | pte_unmap_unlock(vmf->pte, vmf->ptl); |
| 4817 | 4870 | writable = false; |
| 4818 | 4871 | |
| 4819 | 4872 | /* Migrate to the requested node */ |
| 4820 | − if (migrate_misplaced_page(page, vma, target_nid)) { | |
| 4821 | − page_nid = target_nid; | |
| 4873 | + if (migrate_misplaced_folio(folio, vma, target_nid)) { | |
| 4874 | + nid = target_nid; | |
| 4822 | 4875 | flags |= TNF_MIGRATED; |
| 4823 | 4876 | } else { |
| 4824 | 4877 | flags |= TNF_MIGRATE_FAIL; |
| @@ -4834,8 +4887,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) · 9 unmodified lines | ||
| 4834 | 4887 | } |
| 4835 | 4888 | |
| 4836 | 4889 | out: |
| 4837 | − if (page_nid != NUMA_NO_NODE) | |
| 4838 | − task_numa_fault(last_cpupid, page_nid, 1, flags); | |
| 4890 | + if (nid != NUMA_NO_NODE) | |
| 4891 | + task_numa_fault(last_cpupid, nid, 1, flags); | |
| 4839 | 4892 | return 0; |
| 4840 | 4893 | out_map: |
| 4841 | 4894 | /* |
| @@ -4872,8 +4925,11 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) · 30 unmodified lines | ||
| 4872 | 4925 | |
| 4873 | 4926 | if (vma_is_anonymous(vma)) { |
| 4874 | 4927 | if (likely(!unshare) && |
| 4875 | − userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd)) | |
| 4928 | + userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd)) { | |
| 4929 | + if (userfaultfd_wp_async(vmf->vma)) | |
| 4930 | + goto split; | |
| 4876 | 4931 | return handle_userfault(vmf, VM_UFFD_WP); |
| 4932 | + } | |
| 4877 | 4933 | return do_huge_pmd_wp_page(vmf); |
| 4878 | 4934 | } |
| 4879 | 4935 | |
| @@ -4885,6 +4941,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) · 5 unmodified lines | ||
| 4885 | 4941 | } |
| 4886 | 4942 | } |
| 4887 | 4943 | |
| 4944 | +split: | |
| 4888 | 4945 | /* COW or write-notify handled on pte level: split pmd. */ |
| 4889 | 4946 | __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL); |
| 4890 | 4947 | |
| @@ -5736,8 +5793,8 @@ EXPORT_SYMBOL_GPL(generic_access_phys); · 845 unmodified lines | ||
| 5736 | 5793 | /* |
| 5737 | 5794 | * Access another process' address space as given in mm. |
| 5738 | 5795 | */ |
| 5739 | −int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf, | |
| 5740 | − int len, unsigned int gup_flags) | |
| 5796 | +static int __access_remote_vm(struct mm_struct *mm, unsigned long addr, | |
| 5797 | + void *buf, int len, unsigned int gup_flags) | |
| 5741 | 5798 | { |
| 5742 | 5799 | void *old_buf = buf; |
| 5743 | 5800 | int write = gup_flags & FOLL_WRITE; |
| @@ -5760,7 +5817,7 @@ int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf, · 16 unmodified lines | ||
| 5760 | 5817 | struct page *page = get_user_page_vma_remote(mm, addr, |
| 5761 | 5818 | gup_flags, &vma); |
| 5762 | 5819 | |
| 5763 | − if (IS_ERR_OR_NULL(page)) { | |
| 5820 | + if (IS_ERR(page)) { | |
| 5764 | 5821 | /* We might need to expand the stack to access it */ |
| 5765 | 5822 | vma = vma_lookup(mm, addr); |
| 5766 | 5823 | if (!vma) { |
| @@ -5774,7 +5831,6 @@ int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf, · 7 unmodified lines | ||
| 5774 | 5831 | continue; |
| 5775 | 5832 | } |
| 5776 | 5833 | |
| 5777 | − | |
| 5778 | 5834 | /* |
| 5779 | 5835 | * Check if this is a VM_IO | VM_PFNMAP VMA, which |
| 5780 | 5836 | * we can access using slightly different code. |
| @@ -1689,7 +1689,7 @@ static int scan_movable_pages(unsigned long start, unsigned long end, · 1688 unmodified lines | ||
| 1689 | 1689 | */ |
| 1690 | 1690 | if (HPageMigratable(head)) |
| 1691 | 1691 | goto found; |
| 1692 | − skip = compound_nr(head) - (page - head); | |
| 1692 | + skip = compound_nr(head) - (pfn - page_to_pfn(head)); | |
| 1693 | 1693 | pfn += skip - 1; |
| 1694 | 1694 | } |
| 1695 | 1695 | return -ENOENT; |
| @@ -2012,12 +2012,16 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages, · 316 unmodified lines | ||
| 2012 | 2012 | /* reinitialise watermarks and update pcp limits */ |
| 2013 | 2013 | init_per_zone_wmark_min(); |
| 2014 | 2014 | |
| 2015 | + /* | |
| 2016 | + * Make sure to mark the node as memory-less before rebuilding the zone | |
| 2017 | + * list. Otherwise this node would still appear in the fallback lists. | |
| 2018 | + */ | |
| 2019 | + node_states_clear_node(node, &arg); | |
| 2015 | 2020 | if (!populated_zone(zone)) { |
| 2016 | 2021 | zone_pcp_reset(zone); |
| 2017 | 2022 | build_all_zonelists(NULL); |
| 2018 | 2023 | } |
| 2019 | 2024 | |
| 2020 | − node_states_clear_node(node, &arg); | |
| 2021 | 2025 | if (arg.status_change_nid >= 0) { |
| 2022 | 2026 | kcompactd_stop(node); |
| 2023 | 2027 | kswapd_stop(node); |
| @@ -25,7 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | * to the last. It would be better if bind would truly restrict |
| 26 | 26 | * the allocation to memory nodes instead |
| 27 | 27 | * |
| 28 | − * preferred Try a specific node first before normal fallback. | |
| 28 | + * preferred Try a specific node first before normal fallback. | |
| 29 | 29 | * As a special case NUMA_NO_NODE here means do the allocation |
| 30 | 30 | * on the local CPU. This is normally identical to default, |
| 31 | 31 | * but useful to set in a VMA when you have a non default |
| @@ -52,7 +52,7 @@ · 20 unmodified lines | ||
| 52 | 52 | * on systems with highmem kernel lowmem allocation don't get policied. |
| 53 | 53 | * Same with GFP_DMA allocations. |
| 54 | 54 | * |
| 55 | − * For shmfs/tmpfs/hugetlbfs shared memory the policy is shared between | |
| 55 | + * For shmem/tmpfs shared memory the policy is shared between | |
| 56 | 56 | * all users and remembered even when nobody has memory mapped. |
| 57 | 57 | */ |
| 58 | 58 | |
| @@ -111,7 +111,8 @@ · 52 unmodified lines | ||
| 111 | 111 | |
| 112 | 112 | /* Internal flags */ |
| 113 | 113 | #define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0) /* Skip checks for continuous vmas */ |
| 114 | −#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */ | |
| 114 | +#define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1) /* Invert check for nodemask */ | |
| 115 | +#define MPOL_MF_WRLOCK (MPOL_MF_INTERNAL << 2) /* Write-lock walked vmas */ | |
| 115 | 116 | |
| 116 | 117 | static struct kmem_cache *policy_cache; |
| 117 | 118 | static struct kmem_cache *sn_cache; |
| @@ -267,9 +268,6 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 149 unmodified lines | ||
| 267 | 268 | { |
| 268 | 269 | struct mempolicy *policy; |
| 269 | 270 | |
| 270 | − pr_debug("setting mode %d flags %d nodes[0] %lx\n", | |
| 271 | − mode, flags, nodes ? nodes_addr(*nodes)[0] : NUMA_NO_NODE); | |
| 272 | − | |
| 273 | 271 | if (mode == MPOL_DEFAULT) { |
| 274 | 272 | if (nodes && !nodes_empty(*nodes)) |
| 275 | 273 | return ERR_PTR(-EINVAL); |
| @@ -297,6 +295,7 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 21 unmodified lines | ||
| 297 | 295 | return ERR_PTR(-EINVAL); |
| 298 | 296 | } else if (nodes_empty(*nodes)) |
| 299 | 297 | return ERR_PTR(-EINVAL); |
| 298 | + | |
| 300 | 299 | policy = kmem_cache_alloc(policy_cache, GFP_KERNEL); |
| 301 | 300 | if (!policy) |
| 302 | 301 | return ERR_PTR(-ENOMEM); |
| @@ -309,11 +308,11 @@ static struct mempolicy *mpol_new(unsigned short mode, unsigned short flags, · 6 unmodified lines | ||
| 309 | 308 | } |
| 310 | 309 | |
| 311 | 310 | /* Slow path of a mpol destructor. */ |
| 312 | −void __mpol_put(struct mempolicy *p) | |
| 311 | +void __mpol_put(struct mempolicy *pol) | |
| 313 | 312 | { |
| 314 | − if (!atomic_dec_and_test(&p->refcnt)) | |
| 313 | + if (!atomic_dec_and_test(&pol->refcnt)) | |
| 315 | 314 | return; |
| 316 | − kmem_cache_free(policy_cache, p); | |
| 315 | + kmem_cache_free(policy_cache, pol); | |
| 317 | 316 | } |
| 318 | 317 | |
| 319 | 318 | static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes) |
| @@ -370,7 +369,6 @@ static void mpol_rebind_policy(struct mempolicy *pol, const nodemask_t *newmask) · 50 unmodified lines | ||
| 370 | 369 | * |
| 371 | 370 | * Called with task's alloc_lock held. |
| 372 | 371 | */ |
| 373 | − | |
| 374 | 372 | void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new) |
| 375 | 373 | { |
| 376 | 374 | mpol_rebind_policy(tsk->mempolicy, new); |
| @@ -381,7 +379,6 @@ void mpol_rebind_task(struct task_struct *tsk, const nodemask_t *new) · 4 unmodified lines | ||
| 381 | 379 | * |
| 382 | 380 | * Call holding a reference to mm. Takes mm->mmap_lock during call. |
| 383 | 381 | */ |
| 384 | − | |
| 385 | 382 | void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) |
| 386 | 383 | { |
| 387 | 384 | struct vm_area_struct *vma; |
| @@ -420,8 +417,25 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { · 32 unmodified lines | ||
| 420 | 417 | }, |
| 421 | 418 | }; |
| 422 | 419 | |
| 423 | −static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, | |
| 420 | +static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, | |
| 424 | 421 | unsigned long flags); |
| 422 | +static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, | |
| 423 | + pgoff_t ilx, int *nid); | |
| 424 | + | |
| 425 | +static bool strictly_unmovable(unsigned long flags) | |
| 426 | +{ | |
| 427 | + /* | |
| 428 | + * STRICT without MOVE flags lets do_mbind() fail immediately with -EIO | |
| 429 | + * if any misplaced page is found. | |
| 430 | + */ | |
| 431 | + return (flags & (MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) == | |
| 432 | + MPOL_MF_STRICT; | |
| 433 | +} | |
| 434 | + | |
| 435 | +struct migration_mpol { /* for alloc_migration_target_by_mpol() */ | |
| 436 | + struct mempolicy *pol; | |
| 437 | + pgoff_t ilx; | |
| 438 | +}; | |
| 425 | 439 | |
| 426 | 440 | struct queue_pages { |
| 427 | 441 | struct list_head *pagelist; |
| @@ -430,7 +444,8 @@ struct queue_pages { · 2 unmodified lines | ||
| 430 | 444 | unsigned long start; |
| 431 | 445 | unsigned long end; |
| 432 | 446 | struct vm_area_struct *first; |
| 433 | − bool has_unmovable; | |
| 447 | + struct folio *large; /* note last large folio encountered */ | |
| 448 | + long nr_failed; /* could not be isolated at this time */ | |
| 434 | 449 | }; |
| 435 | 450 | |
| 436 | 451 | /* |
| @@ -448,61 +463,37 @@ static inline bool queue_folio_required(struct folio *folio, · 11 unmodified lines | ||
| 448 | 463 | return node_isset(nid, *qp->nmask) == !(flags & MPOL_MF_INVERT); |
| 449 | 464 | } |
| 450 | 465 | |
| 451 | −/* | |
| 452 | − * queue_folios_pmd() has three possible return values: | |
| 453 | − * 0 - folios are placed on the right node or queued successfully, or | |
| 454 | − * special page is met, i.e. zero page, or unmovable page is found | |
| 455 | − * but continue walking (indicated by queue_pages.has_unmovable). | |
| 456 | − * -EIO - is migration entry or only MPOL_MF_STRICT was specified and an | |
| 457 | − * existing folio was already on a node that does not follow the | |
| 458 | − * policy. | |
| 459 | − */ | |
| 460 | −static int queue_folios_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr, | |
| 461 | − unsigned long end, struct mm_walk *walk) | |
| 462 | − __releases(ptl) | |
| 466 | +static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk) | |
| 463 | 467 | { |
| 464 | − int ret = 0; | |
| 465 | 468 | struct folio *folio; |
| 466 | 469 | struct queue_pages *qp = walk->private; |
| 467 | − unsigned long flags; | |
| 468 | 470 | |
| 469 | 471 | if (unlikely(is_pmd_migration_entry(*pmd))) { |
| 470 | − ret = -EIO; | |
| 471 | − goto unlock; | |
| 472 | + qp->nr_failed++; | |
| 473 | + return; | |
| 472 | 474 | } |
| 473 | 475 | folio = pfn_folio(pmd_pfn(*pmd)); |
| 474 | 476 | if (is_huge_zero_page(&folio->page)) { |
| 475 | 477 | walk->action = ACTION_CONTINUE; |
| 476 | − goto unlock; | |
| 478 | + return; | |
| 477 | 479 | } |
| 478 | 480 | if (!queue_folio_required(folio, qp)) |
| 479 | − goto unlock; | |
| 480 | − | |
| 481 | − flags = qp->flags; | |
| 482 | − /* go to folio migration */ | |
| 483 | − if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) { | |
| 484 | − if (!vma_migratable(walk->vma) || | |
| 485 | − migrate_folio_add(folio, qp->pagelist, flags)) { | |
| 486 | − qp->has_unmovable = true; | |
| 487 | − goto unlock; | |
| 488 | − } | |
| 489 | − } else | |
| 490 | − ret = -EIO; | |
| 491 | −unlock: | |
| 492 | − spin_unlock(ptl); | |
| 493 | − return ret; | |
| 481 | + return; | |
| 482 | + if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || | |
| 483 | + !vma_migratable(walk->vma) || | |
| 484 | + !migrate_folio_add(folio, qp->pagelist, qp->flags)) | |
| 485 | + qp->nr_failed++; | |
| 494 | 486 | } |
| 495 | 487 | |
| 496 | 488 | /* |
| 497 | − * Scan through pages checking if pages follow certain conditions, | |
| 498 | − * and move them to the pagelist if they do. | |
| 489 | + * Scan through folios, checking if they satisfy the required conditions, | |
| 490 | + * moving them from LRU to local pagelist for migration if they do (or not). | |
| 499 | 491 | * |
| 500 | − * queue_folios_pte_range() has three possible return values: | |
| 501 | − * 0 - folios are placed on the right node or queued successfully, or | |
| 502 | − * special page is met, i.e. zero page, or unmovable page is found | |
| 503 | − * but continue walking (indicated by queue_pages.has_unmovable). | |
| 504 | − * -EIO - only MPOL_MF_STRICT was specified and an existing folio was already | |
| 505 | − * on a node that does not follow the policy. | |
| 492 | + * queue_folios_pte_range() has two possible return values: | |
| 493 | + * 0 - continue walking to scan for more, even if an existing folio on the | |
| 494 | + * wrong node could not be isolated and queued for migration. | |
| 495 | + * -EIO - only MPOL_MF_STRICT was specified, without MPOL_MF_MOVE or ..._ALL, | |
| 496 | + * and an existing folio was on a node that does not follow the policy. | |
| 506 | 497 | */ |
| 507 | 498 | static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, |
| 508 | 499 | unsigned long end, struct mm_walk *walk) |
| @@ -516,8 +507,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 7 unmodified lines | ||
| 516 | 507 | spinlock_t *ptl; |
| 517 | 508 | |
| 518 | 509 | ptl = pmd_trans_huge_lock(pmd, vma); |
| 519 | − if (ptl) | |
| 520 | − return queue_folios_pmd(pmd, ptl, addr, end, walk); | |
| 510 | + if (ptl) { | |
| 511 | + queue_folios_pmd(pmd, walk); | |
| 512 | + spin_unlock(ptl); | |
| 513 | + goto out; | |
| 514 | + } | |
| 521 | 515 | |
| 522 | 516 | mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); |
| 523 | 517 | if (!pte) { |
| @@ -526,8 +520,13 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 2 unmodified lines | ||
| 526 | 520 | } |
| 527 | 521 | for (; addr != end; pte++, addr += PAGE_SIZE) { |
| 528 | 522 | ptent = ptep_get(pte); |
| 529 | − if (!pte_present(ptent)) | |
| 523 | + if (pte_none(ptent)) | |
| 524 | + continue; | |
| 525 | + if (!pte_present(ptent)) { | |
| 526 | + if (is_migration_entry(pte_to_swp_entry(ptent))) | |
| 527 | + qp->nr_failed++; | |
| 530 | 528 | continue; |
| 529 | + } | |
| 531 | 530 | folio = vm_normal_folio(vma, addr, ptent); |
| 532 | 531 | if (!folio || folio_is_zone_device(folio)) |
| 533 | 532 | continue; |
| @@ -539,95 +538,87 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · 5 unmodified lines | ||
| 539 | 538 | continue; |
| 540 | 539 | if (!queue_folio_required(folio, qp)) |
| 541 | 540 | continue; |
| 542 | − if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) { | |
| 543 | − /* | |
| 544 | − * MPOL_MF_STRICT must be specified if we get here. | |
| 545 | − * Continue walking vmas due to MPOL_MF_MOVE* flags. | |
| 546 | − */ | |
| 547 | − if (!vma_migratable(vma)) | |
| 548 | − qp->has_unmovable = true; | |
| 549 | − | |
| 541 | + if (folio_test_large(folio)) { | |
| 550 | 542 | /* |
| 551 | − * Do not abort immediately since there may be | |
| 552 | − * temporary off LRU pages in the range. Still | |
| 553 | − * need migrate other LRU pages. | |
| 543 | + * A large folio can only be isolated from LRU once, | |
| 544 | + * but may be mapped by many PTEs (and Copy-On-Write may | |
| 545 | + * intersperse PTEs of other, order 0, folios). This is | |
| 546 | + * a common case, so don't mistake it for failure (but | |
| 547 | + * there can be other cases of multi-mapped pages which | |
| 548 | + * this quick check does not help to filter out - and a | |
| 549 | + * search of the pagelist might grow to be prohibitive). | |
| 550 | + * | |
| 551 | + * migrate_pages(&pagelist) returns nr_failed folios, so | |
| 552 | + * check "large" now so that queue_pages_range() returns | |
| 553 | + * a comparable nr_failed folios. This does imply that | |
| 554 | + * if folio could not be isolated for some racy reason | |
| 555 | + * at its first PTE, later PTEs will not give it another | |
| 556 | + * chance of isolation; but keeps the accounting simple. | |
| 554 | 557 | */ |
| 555 | − if (migrate_folio_add(folio, qp->pagelist, flags)) | |
| 556 | − qp->has_unmovable = true; | |
| 557 | − } else | |
| 558 | − break; | |
| 558 | + if (folio == qp->large) | |
| 559 | + continue; | |
| 560 | + qp->large = folio; | |
| 561 | + } | |
| 562 | + if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || | |
| 563 | + !vma_migratable(vma) || | |
| 564 | + !migrate_folio_add(folio, qp->pagelist, flags)) { | |
| 565 | + qp->nr_failed++; | |
| 566 | + if (strictly_unmovable(flags)) | |
| 567 | + break; | |
| 568 | + } | |
| 559 | 569 | } |
| 560 | 570 | pte_unmap_unlock(mapped_pte, ptl); |
| 561 | 571 | cond_resched(); |
| 562 | − | |
| 563 | − return addr != end ? -EIO : 0; | |
| 572 | +out: | |
| 573 | + if (qp->nr_failed && strictly_unmovable(flags)) | |
| 574 | + return -EIO; | |
| 575 | + return 0; | |
| 564 | 576 | } |
| 565 | 577 | |
| 566 | 578 | static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask, |
| 567 | 579 | unsigned long addr, unsigned long end, |
| 568 | 580 | struct mm_walk *walk) |
| 569 | 581 | { |
| 570 | − int ret = 0; | |
| 571 | 582 | #ifdef CONFIG_HUGETLB_PAGE |
| 572 | 583 | struct queue_pages *qp = walk->private; |
| 573 | − unsigned long flags = (qp->flags & MPOL_MF_VALID); | |
| 584 | + unsigned long flags = qp->flags; | |
| 574 | 585 | struct folio *folio; |
| 575 | 586 | spinlock_t *ptl; |
| 576 | 587 | pte_t entry; |
| 577 | 588 | |
| 578 | 589 | ptl = huge_pte_lock(hstate_vma(walk->vma), walk->mm, pte); |
| 579 | 590 | entry = huge_ptep_get(pte); |
| 580 | − if (!pte_present(entry)) | |
| 591 | + if (!pte_present(entry)) { | |
| 592 | + if (unlikely(is_hugetlb_entry_migration(entry))) | |
| 593 | + qp->nr_failed++; | |
| 581 | 594 | goto unlock; |
| 595 | + } | |
| 582 | 596 | folio = pfn_folio(pte_pfn(entry)); |
| 583 | 597 | if (!queue_folio_required(folio, qp)) |
| 584 | 598 | goto unlock; |
| 585 | − | |
| 586 | − if (flags == MPOL_MF_STRICT) { | |
| 587 | − /* | |
| 588 | − * STRICT alone means only detecting misplaced folio and no | |
| 589 | − * need to further check other vma. | |
| 590 | − */ | |
| 591 | − ret = -EIO; | |
| 599 | + if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) || | |
| 600 | + !vma_migratable(walk->vma)) { | |
| 601 | + qp->nr_failed++; | |
| 592 | 602 | goto unlock; |
| 593 | 603 | } |
| 594 | − | |
| 595 | − if (!vma_migratable(walk->vma)) { | |
| 596 | − /* | |
| 597 | − * Must be STRICT with MOVE*, otherwise .test_walk() have | |
| 598 | − * stopped walking current vma. | |
| 599 | − * Detecting misplaced folio but allow migrating folios which | |
| 600 | − * have been queued. | |
| 601 | − */ | |
| 602 | − qp->has_unmovable = true; | |
| 603 | − goto unlock; | |
| 604 | − } | |
| 605 | − | |
| 606 | 604 | /* |
| 607 | − * With MPOL_MF_MOVE, we try to migrate only unshared folios. If it | |
| 608 | − * is shared it is likely not worth migrating. | |
| 605 | + * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. | |
| 606 | + * Choosing not to migrate a shared folio is not counted as a failure. | |
| 609 | 607 | * |
| 610 | 608 | * To check if the folio is shared, ideally we want to make sure |
| 611 | 609 | * every page is mapped to the same process. Doing that is very |
| 612 | − * expensive, so check the estimated mapcount of the folio instead. | |
| 610 | + * expensive, so check the estimated sharers of the folio instead. | |
| 613 | 611 | */ |
| 614 | − if (flags & (MPOL_MF_MOVE_ALL) || | |
| 615 | − (flags & MPOL_MF_MOVE && folio_estimated_sharers(folio) == 1 && | |
| 616 | − !hugetlb_pmd_shared(pte))) { | |
| 617 | − if (!isolate_hugetlb(folio, qp->pagelist) && | |
| 618 | − (flags & MPOL_MF_STRICT)) | |
| 619 | − /* | |
| 620 | − * Failed to isolate folio but allow migrating pages | |
| 621 | − * which have been queued. | |
| 622 | − */ | |
| 623 | − qp->has_unmovable = true; | |
| 624 | − } | |
| 612 | + if ((flags & MPOL_MF_MOVE_ALL) || | |
| 613 | + (folio_estimated_sharers(folio) == 1 && !hugetlb_pmd_shared(pte))) | |
| 614 | + if (!isolate_hugetlb(folio, qp->pagelist)) | |
| 615 | + qp->nr_failed++; | |
| 625 | 616 | unlock: |
| 626 | 617 | spin_unlock(ptl); |
| 627 | −#else | |
| 628 | − BUG(); | |
| 618 | + if (qp->nr_failed && strictly_unmovable(flags)) | |
| 619 | + return -EIO; | |
| 629 | 620 | #endif |
| 630 | − return ret; | |
| 621 | + return 0; | |
| 631 | 622 | } |
| 632 | 623 | |
| 633 | 624 | #ifdef CONFIG_NUMA_BALANCING |
| @@ -656,12 +647,6 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, · 22 unmodified lines | ||
| 656 | 647 | |
| 657 | 648 | return nr_updated; |
| 658 | 649 | } |
| 659 | −#else | |
| 660 | −static unsigned long change_prot_numa(struct vm_area_struct *vma, | |
| 661 | − unsigned long addr, unsigned long end) | |
| 662 | −{ | |
| 663 | − return 0; | |
| 664 | −} | |
| 665 | 650 | #endif /* CONFIG_NUMA_BALANCING */ |
| 666 | 651 | |
| 667 | 652 | static int queue_pages_test_walk(unsigned long start, unsigned long end, |
| @@ -700,16 +685,11 @@ static int queue_pages_test_walk(unsigned long start, unsigned long end, · 32 unmodified lines | ||
| 700 | 685 | if (endvma > end) |
| 701 | 686 | endvma = end; |
| 702 | 687 | |
| 703 | − if (flags & MPOL_MF_LAZY) { | |
| 704 | − /* Similar to task_numa_work, skip inaccessible VMAs */ | |
| 705 | − if (!is_vm_hugetlb_page(vma) && vma_is_accessible(vma) && | |
| 706 | − !(vma->vm_flags & VM_MIXEDMAP)) | |
| 707 | − change_prot_numa(vma, start, endvma); | |
| 708 | − return 1; | |
| 709 | − } | |
| 710 | − | |
| 711 | − /* queue pages from current vma */ | |
| 712 | − if (flags & MPOL_MF_VALID) | |
| 688 | + /* | |
| 689 | + * Check page nodes, and queue pages to move, in the current vma. | |
| 690 | + * But if no moving, and no strict checking, the scan can be skipped. | |
| 691 | + */ | |
| 692 | + if (flags & (MPOL_MF_STRICT | MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) | |
| 713 | 693 | return 0; |
| 714 | 694 | return 1; |
| 715 | 695 | } |
| @@ -731,22 +711,21 @@ static const struct mm_walk_ops queue_pages_lock_vma_walk_ops = { · 15 unmodified lines | ||
| 731 | 711 | /* |
| 732 | 712 | * Walk through page tables and collect pages to be migrated. |
| 733 | 713 | * |
| 734 | − * If pages found in a given range are on a set of nodes (determined by | |
| 735 | − * @nodes and @flags,) it's isolated and queued to the pagelist which is | |
| 736 | − * passed via @private. | |
| 714 | + * If pages found in a given range are not on the required set of @nodes, | |
| 715 | + * and migration is allowed, they are isolated and queued to @pagelist. | |
| 737 | 716 | * |
| 738 | − * queue_pages_range() has three possible return values: | |
| 739 | − * 1 - there is unmovable page, but MPOL_MF_MOVE* & MPOL_MF_STRICT were | |
| 740 | − * specified. | |
| 741 | − * 0 - queue pages successfully or no misplaced page. | |
| 742 | − * errno - i.e. misplaced pages with MPOL_MF_STRICT specified (-EIO) or | |
| 743 | − * memory range specified by nodemask and maxnode points outside | |
| 744 | − * your accessible address space (-EFAULT) | |
| 717 | + * queue_pages_range() may return: | |
| 718 | + * 0 - all pages already on the right node, or successfully queued for moving | |
| 719 | + * (or neither strict checking nor moving requested: only range checking). | |
| 720 | + * >0 - this number of misplaced folios could not be queued for moving | |
| 721 | + * (a hugetlbfs page or a transparent huge page being counted as 1). | |
| 722 | + * -EIO - a misplaced page found, when MPOL_MF_STRICT specified without MOVEs. | |
| 723 | + * -EFAULT - a hole in the memory range, when MPOL_MF_DISCONTIG_OK unspecified. | |
| 745 | 724 | */ |
| 746 | −static int | |
| 725 | +static long | |
| 747 | 726 | queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, |
| 748 | 727 | nodemask_t *nodes, unsigned long flags, |
| 749 | − struct list_head *pagelist, bool lock_vma) | |
| 728 | + struct list_head *pagelist) | |
| 750 | 729 | { |
| 751 | 730 | int err; |
| 752 | 731 | struct queue_pages qp = { |
| @@ -756,20 +735,17 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, · 3 unmodified lines | ||
| 756 | 735 | .start = start, |
| 757 | 736 | .end = end, |
| 758 | 737 | .first = NULL, |
| 759 | − .has_unmovable = false, | |
| 760 | 738 | }; |
| 761 | − const struct mm_walk_ops *ops = lock_vma ? | |
| 739 | + const struct mm_walk_ops *ops = (flags & MPOL_MF_WRLOCK) ? | |
| 762 | 740 | &queue_pages_lock_vma_walk_ops : &queue_pages_walk_ops; |
| 763 | 741 | |
| 764 | 742 | err = walk_page_range(mm, start, end, ops, &qp); |
| 765 | 743 | |
| 766 | − if (qp.has_unmovable) | |
| 767 | − err = 1; | |
| 768 | 744 | if (!qp.first) |
| 769 | 745 | /* whole range in hole */ |
| 770 | 746 | err = -EFAULT; |
| 771 | 747 | |
| 772 | − return err; | |
| 748 | + return err ? : qp.nr_failed; | |
| 773 | 749 | } |
| 774 | 750 | |
| 775 | 751 | /* |
| @@ -777,7 +753,7 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, · 1 unmodified line | ||
| 777 | 753 | * This must be called with the mmap_lock held for writing. |
| 778 | 754 | */ |
| 779 | 755 | static int vma_replace_policy(struct vm_area_struct *vma, |
| 780 | − struct mempolicy *pol) | |
| 756 | + struct mempolicy *pol) | |
| 781 | 757 | { |
| 782 | 758 | int err; |
| 783 | 759 | struct mempolicy *old; |
| @@ -785,11 +761,6 @@ static int vma_replace_policy(struct vm_area_struct *vma, · 1 unmodified line | ||
| 785 | 761 | |
| 786 | 762 | vma_assert_write_locked(vma); |
| 787 | 763 | |
| 788 | − pr_debug("vma %lx-%lx/%lx vm_ops %p vm_file %p set_policy %p\n", | |
| 789 | − vma->vm_start, vma->vm_end, vma->vm_pgoff, | |
| 790 | − vma->vm_ops, vma->vm_file, | |
| 791 | − vma->vm_ops ? vma->vm_ops->set_policy : NULL); | |
| 792 | − | |
| 793 | 764 | new = mpol_dup(pol); |
| 794 | 765 | if (IS_ERR(new)) |
| 795 | 766 | return PTR_ERR(new); |
| @@ -815,10 +786,7 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, · 19 unmodified lines | ||
| 815 | 786 | struct vm_area_struct **prev, unsigned long start, |
| 816 | 787 | unsigned long end, struct mempolicy *new_pol) |
| 817 | 788 | { |
| 818 | − struct vm_area_struct *merged; | |
| 819 | 789 | unsigned long vmstart, vmend; |
| 820 | − pgoff_t pgoff; | |
| 821 | − int err; | |
| 822 | 790 | |
| 823 | 791 | vmend = min(end, vma->vm_end); |
| 824 | 792 | if (start > vma->vm_start) { |
| @@ -828,31 +796,14 @@ static int mbind_range(struct vma_iterator *vmi, struct vm_area_struct *vma, · 3 unmodified lines | ||
| 828 | 796 | vmstart = vma->vm_start; |
| 829 | 797 | } |
| 830 | 798 | |
| 831 | − if (mpol_equal(vma_policy(vma), new_pol)) { | |
| 799 | + if (mpol_equal(vma->vm_policy, new_pol)) { | |
| 832 | 800 | *prev = vma; |
| 833 | 801 | return 0; |
| 834 | 802 | } |
| 835 | 803 | |
| 836 | − pgoff = vma->vm_pgoff + ((vmstart - vma->vm_start) >> PAGE_SHIFT); | |
| 837 | − merged = vma_merge(vmi, vma->vm_mm, *prev, vmstart, vmend, vma->vm_flags, | |
| 838 | − vma->anon_vma, vma->vm_file, pgoff, new_pol, | |
| 839 | − vma->vm_userfaultfd_ctx, anon_vma_name(vma)); | |
| 840 | − if (merged) { | |
| 841 | − *prev = merged; | |
| 842 | − return vma_replace_policy(merged, new_pol); | |
| 843 | − } | |
| 844 | − | |
| 845 | − if (vma->vm_start != vmstart) { | |
| 846 | − err = split_vma(vmi, vma, vmstart, 1); | |
| 847 | − if (err) | |
| 848 | − return err; | |
| 849 | − } | |
| 850 | − | |
| 851 | − if (vma->vm_end != vmend) { | |
| 852 | − err = split_vma(vmi, vma, vmend, 0); | |
| 853 | − if (err) | |
| 854 | − return err; | |
| 855 | − } | |
| 804 | + vma = vma_modify_policy(vmi, *prev, vma, vmstart, vmend, new_pol); | |
| 805 | + if (IS_ERR(vma)) | |
| 806 | + return PTR_ERR(vma); | |
| 856 | 807 | |
| 857 | 808 | *prev = vma; |
| 858 | 809 | return vma_replace_policy(vma, new_pol); |
| @@ -900,18 +851,18 @@ out: · 41 unmodified lines | ||
| 900 | 851 | * |
| 901 | 852 | * Called with task's alloc_lock held |
| 902 | 853 | */ |
| 903 | −static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes) | |
| 854 | +static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes) | |
| 904 | 855 | { |
| 905 | 856 | nodes_clear(*nodes); |
| 906 | − if (p == &default_policy) | |
| 857 | + if (pol == &default_policy) | |
| 907 | 858 | return; |
| 908 | 859 | |
| 909 | − switch (p->mode) { | |
| 860 | + switch (pol->mode) { | |
| 910 | 861 | case MPOL_BIND: |
| 911 | 862 | case MPOL_INTERLEAVE: |
| 912 | 863 | case MPOL_PREFERRED: |
| 913 | 864 | case MPOL_PREFERRED_MANY: |
| 914 | − *nodes = p->nodes; | |
| 865 | + *nodes = pol->nodes; | |
| 915 | 866 | break; |
| 916 | 867 | case MPOL_LOCAL: |
| 917 | 868 | /* return empty node mask for local allocation */ |
| @@ -958,6 +909,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 40 unmodified lines | ||
| 958 | 909 | } |
| 959 | 910 | |
| 960 | 911 | if (flags & MPOL_F_ADDR) { |
| 912 | + pgoff_t ilx; /* ignored here */ | |
| 961 | 913 | /* |
| 962 | 914 | * Do NOT fall back to task policy if the |
| 963 | 915 | * vma/shared policy at addr is NULL. We |
| @@ -969,10 +921,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 5 unmodified lines | ||
| 969 | 921 | mmap_read_unlock(mm); |
| 970 | 922 | return -EFAULT; |
| 971 | 923 | } |
| 972 | − if (vma->vm_ops && vma->vm_ops->get_policy) | |
| 973 | − pol = vma->vm_ops->get_policy(vma, addr); | |
| 974 | − else | |
| 975 | − pol = vma->vm_policy; | |
| 924 | + pol = __get_vma_policy(vma, addr, &ilx); | |
| 976 | 925 | } else if (addr) |
| 977 | 926 | return -EINVAL; |
| 978 | 927 | |
| @@ -1032,16 +981,16 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 53 unmodified lines | ||
| 1032 | 981 | } |
| 1033 | 982 | |
| 1034 | 983 | #ifdef CONFIG_MIGRATION |
| 1035 | −static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, | |
| 984 | +static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, | |
| 1036 | 985 | unsigned long flags) |
| 1037 | 986 | { |
| 1038 | 987 | /* |
| 1039 | − * We try to migrate only unshared folios. If it is shared it | |
| 1040 | − * is likely not worth migrating. | |
| 988 | + * Unless MPOL_MF_MOVE_ALL, we try to avoid migrating a shared folio. | |
| 989 | + * Choosing not to migrate a shared folio is not counted as a failure. | |
| 1041 | 990 | * |
| 1042 | 991 | * To check if the folio is shared, ideally we want to make sure |
| 1043 | 992 | * every page is mapped to the same process. Doing that is very |
| 1044 | − * expensive, so check the estimated mapcount of the folio instead. | |
| 993 | + * expensive, so check the estimated sharers of the folio instead. | |
| 1045 | 994 | */ |
| 1046 | 995 | if ((flags & MPOL_MF_MOVE_ALL) || folio_estimated_sharers(folio) == 1) { |
| 1047 | 996 | if (folio_isolate_lru(folio)) { |
| @@ -1049,32 +998,31 @@ static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, · 1 unmodified line | ||
| 1049 | 998 | node_stat_mod_folio(folio, |
| 1050 | 999 | NR_ISOLATED_ANON + folio_is_file_lru(folio), |
| 1051 | 1000 | folio_nr_pages(folio)); |
| 1052 | − } else if (flags & MPOL_MF_STRICT) { | |
| 1001 | + } else { | |
| 1053 | 1002 | /* |
| 1054 | 1003 | * Non-movable folio may reach here. And, there may be |
| 1055 | 1004 | * temporary off LRU folios or non-LRU movable folios. |
| 1056 | 1005 | * Treat them as unmovable folios since they can't be |
| 1057 | − * isolated, so they can't be moved at the moment. It | |
| 1058 | − * should return -EIO for this case too. | |
| 1006 | + * isolated, so they can't be moved at the moment. | |
| 1059 | 1007 | */ |
| 1060 | − return -EIO; | |
| 1008 | + return false; | |
| 1061 | 1009 | } |
| 1062 | 1010 | } |
| 1063 | − | |
| 1064 | − return 0; | |
| 1011 | + return true; | |
| 1065 | 1012 | } |
| 1066 | 1013 | |
| 1067 | 1014 | /* |
| 1068 | 1015 | * Migrate pages from one node to a target node. |
| 1069 | 1016 | * Returns error or the number of pages not migrated. |
| 1070 | 1017 | */ |
| 1071 | −static int migrate_to_node(struct mm_struct *mm, int source, int dest, | |
| 1072 | − int flags) | |
| 1018 | +static long migrate_to_node(struct mm_struct *mm, int source, int dest, | |
| 1019 | + int flags) | |
| 1073 | 1020 | { |
| 1074 | 1021 | nodemask_t nmask; |
| 1075 | 1022 | struct vm_area_struct *vma; |
| 1076 | 1023 | LIST_HEAD(pagelist); |
| 1077 | − int err = 0; | |
| 1024 | + long nr_failed; | |
| 1025 | + long err = 0; | |
| 1078 | 1026 | struct migration_target_control mtc = { |
| 1079 | 1027 | .nid = dest, |
| 1080 | 1028 | .gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE, |
| @@ -1083,23 +1031,30 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, · 2 unmodified lines | ||
| 1083 | 1031 | nodes_clear(nmask); |
| 1084 | 1032 | node_set(source, nmask); |
| 1085 | 1033 | |
| 1034 | + VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); | |
| 1035 | + | |
| 1036 | + mmap_read_lock(mm); | |
| 1037 | + vma = find_vma(mm, 0); | |
| 1038 | + | |
| 1086 | 1039 | /* |
| 1087 | − * This does not "check" the range but isolates all pages that | |
| 1040 | + * This does not migrate the range, but isolates all pages that | |
| 1088 | 1041 | * need migration. Between passing in the full user address |
| 1089 | − * space range and MPOL_MF_DISCONTIG_OK, this call can not fail. | |
| 1042 | + * space range and MPOL_MF_DISCONTIG_OK, this call cannot fail, | |
| 1043 | + * but passes back the count of pages which could not be isolated. | |
| 1090 | 1044 | */ |
| 1091 | − vma = find_vma(mm, 0); | |
| 1092 | − VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); | |
| 1093 | − queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, | |
| 1094 | − flags | MPOL_MF_DISCONTIG_OK, &pagelist, false); | |
| 1045 | + nr_failed = queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, | |
| 1046 | + flags | MPOL_MF_DISCONTIG_OK, &pagelist); | |
| 1047 | + mmap_read_unlock(mm); | |
| 1095 | 1048 | |
| 1096 | 1049 | if (!list_empty(&pagelist)) { |
| 1097 | 1050 | err = migrate_pages(&pagelist, alloc_migration_target, NULL, |
| 1098 | − (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL); | |
| 1051 | + (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL, NULL); | |
| 1099 | 1052 | if (err) |
| 1100 | 1053 | putback_movable_pages(&pagelist); |
| 1101 | 1054 | } |
| 1102 | 1055 | |
| 1056 | + if (err >= 0) | |
| 1057 | + err += nr_failed; | |
| 1103 | 1058 | return err; |
| 1104 | 1059 | } |
| 1105 | 1060 | |
| @@ -1112,14 +1067,12 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, · 6 unmodified lines | ||
| 1112 | 1067 | int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, |
| 1113 | 1068 | const nodemask_t *to, int flags) |
| 1114 | 1069 | { |
| 1115 | − int busy = 0; | |
| 1116 | − int err = 0; | |
| 1070 | + long nr_failed = 0; | |
| 1071 | + long err = 0; | |
| 1117 | 1072 | nodemask_t tmp; |
| 1118 | 1073 | |
| 1119 | 1074 | lru_cache_disable(); |
| 1120 | 1075 | |
| 1121 | − mmap_read_lock(mm); | |
| 1122 | − | |
| 1123 | 1076 | /* |
| 1124 | 1077 | * Find a 'source' bit set in 'tmp' whose corresponding 'dest' |
| 1125 | 1078 | * bit in 'to' is not also set in 'tmp'. Clear the found 'source' |
| @@ -1195,59 +1148,58 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, · 69 unmodified lines | ||
| 1195 | 1148 | node_clear(source, tmp); |
| 1196 | 1149 | err = migrate_to_node(mm, source, dest, flags); |
| 1197 | 1150 | if (err > 0) |
| 1198 | − busy += err; | |
| 1151 | + nr_failed += err; | |
| 1199 | 1152 | if (err < 0) |
| 1200 | 1153 | break; |
| 1201 | 1154 | } |
| 1202 | − mmap_read_unlock(mm); | |
| 1203 | 1155 | |
| 1204 | 1156 | lru_cache_enable(); |
| 1205 | 1157 | if (err < 0) |
| 1206 | 1158 | return err; |
| 1207 | − return busy; | |
| 1208 | − | |
| 1159 | + return (nr_failed < INT_MAX) ? nr_failed : INT_MAX; | |
| 1209 | 1160 | } |
| 1210 | 1161 | |
| 1211 | 1162 | /* |
| 1212 | − * Allocate a new page for page migration based on vma policy. | |
| 1213 | − * Start by assuming the page is mapped by the same vma as contains @start. | |
| 1214 | − * Search forward from there, if not. N.B., this assumes that the | |
| 1215 | − * list of pages handed to migrate_pages()--which is how we get here-- | |
| 1216 | − * is in virtual address order. | |
| 1163 | + * Allocate a new folio for page migration, according to NUMA mempolicy. | |
| 1217 | 1164 | */ |
| 1218 | −static struct folio *new_folio(struct folio *src, unsigned long start) | |
| 1165 | +static struct folio *alloc_migration_target_by_mpol(struct folio *src, | |
| 1166 | + unsigned long private) | |
| 1219 | 1167 | { |
| 1220 | − struct vm_area_struct *vma; | |
| 1221 | − unsigned long address; | |
| 1222 | − VMA_ITERATOR(vmi, current->mm, start); | |
| 1223 | − gfp_t gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL; | |
| 1168 | + struct migration_mpol *mmpol = (struct migration_mpol *)private; | |
| 1169 | + struct mempolicy *pol = mmpol->pol; | |
| 1170 | + pgoff_t ilx = mmpol->ilx; | |
| 1171 | + struct page *page; | |
| 1172 | + unsigned int order; | |
| 1173 | + int nid = numa_node_id(); | |
| 1174 | + gfp_t gfp; | |
| 1224 | 1175 | |
| 1225 | − for_each_vma(vmi, vma) { | |
| 1226 | − address = page_address_in_vma(&src->page, vma); | |
| 1227 | − if (address != -EFAULT) | |
| 1228 | − break; | |
| 1229 | − } | |
| 1176 | + order = folio_order(src); | |
| 1177 | + ilx += src->index >> order; | |
| 1230 | 1178 | |
| 1231 | 1179 | if (folio_test_hugetlb(src)) { |
| 1232 | − return alloc_hugetlb_folio_vma(folio_hstate(src), | |
| 1233 | − vma, address); | |
| 1180 | + nodemask_t *nodemask; | |
| 1181 | + struct hstate *h; | |
| 1182 | + | |
| 1183 | + h = folio_hstate(src); | |
| 1184 | + gfp = htlb_alloc_mask(h); | |
| 1185 | + nodemask = policy_nodemask(gfp, pol, ilx, &nid); | |
| 1186 | + return alloc_hugetlb_folio_nodemask(h, nid, nodemask, gfp); | |
| 1234 | 1187 | } |
| 1235 | 1188 | |
| 1236 | 1189 | if (folio_test_large(src)) |
| 1237 | 1190 | gfp = GFP_TRANSHUGE; |
| 1191 | + else | |
| 1192 | + gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL | __GFP_COMP; | |
| 1238 | 1193 | |
| 1239 | − /* | |
| 1240 | − * if !vma, vma_alloc_folio() will use task or system default policy | |
| 1241 | − */ | |
| 1242 | − return vma_alloc_folio(gfp, folio_order(src), vma, address, | |
| 1243 | − folio_test_large(src)); | |
| 1194 | + page = alloc_pages_mpol(gfp, order, pol, ilx, nid); | |
| 1195 | + return page_rmappable_folio(page); | |
| 1244 | 1196 | } |
| 1245 | 1197 | #else |
| 1246 | 1198 | |
| 1247 | −static int migrate_folio_add(struct folio *folio, struct list_head *foliolist, | |
| 1199 | +static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist, | |
| 1248 | 1200 | unsigned long flags) |
| 1249 | 1201 | { |
| 1250 | − return -EIO; | |
| 1202 | + return false; | |
| 1251 | 1203 | } |
| 1252 | 1204 | |
| 1253 | 1205 | int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, |
| @@ -1256,7 +1208,8 @@ int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from, · 2 unmodified lines | ||
| 1256 | 1208 | return -ENOSYS; |
| 1257 | 1209 | } |
| 1258 | 1210 | |
| 1259 | −static struct folio *new_folio(struct folio *src, unsigned long start) | |
| 1211 | +static struct folio *alloc_migration_target_by_mpol(struct folio *src, | |
| 1212 | + unsigned long private) | |
| 1260 | 1213 | { |
| 1261 | 1214 | return NULL; |
| 1262 | 1215 | } |
| @@ -1269,10 +1222,11 @@ static long do_mbind(unsigned long start, unsigned long len, · 6 unmodified lines | ||
| 1269 | 1222 | struct mm_struct *mm = current->mm; |
| 1270 | 1223 | struct vm_area_struct *vma, *prev; |
| 1271 | 1224 | struct vma_iterator vmi; |
| 1225 | + struct migration_mpol mmpol; | |
| 1272 | 1226 | struct mempolicy *new; |
| 1273 | 1227 | unsigned long end; |
| 1274 | − int err; | |
| 1275 | − int ret; | |
| 1228 | + long err; | |
| 1229 | + long nr_failed; | |
| 1276 | 1230 | LIST_HEAD(pagelist); |
| 1277 | 1231 | |
| 1278 | 1232 | if (flags & ~(unsigned long)MPOL_MF_VALID) |
| @@ -1298,9 +1252,6 @@ static long do_mbind(unsigned long start, unsigned long len, · 19 unmodified lines | ||
| 1298 | 1252 | if (IS_ERR(new)) |
| 1299 | 1253 | return PTR_ERR(new); |
| 1300 | 1254 | |
| 1301 | − if (flags & MPOL_MF_LAZY) | |
| 1302 | − new->flags |= MPOL_F_MOF; | |
| 1303 | − | |
| 1304 | 1255 | /* |
| 1305 | 1256 | * If we are using the default policy then operation |
| 1306 | 1257 | * on discontinuous address spaces is okay after all |
| @@ -1308,14 +1259,8 @@ static long do_mbind(unsigned long start, unsigned long len, · 1 unmodified line | ||
| 1308 | 1259 | if (!new) |
| 1309 | 1260 | flags |= MPOL_MF_DISCONTIG_OK; |
| 1310 | 1261 | |
| 1311 | − pr_debug("mbind %lx-%lx mode:%d flags:%d nodes:%lx\n", | |
| 1312 | − start, start + len, mode, mode_flags, | |
| 1313 | − nmask ? nodes_addr(*nmask)[0] : NUMA_NO_NODE); | |
| 1314 | − | |
| 1315 | − if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) { | |
| 1316 | − | |
| 1262 | + if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) | |
| 1317 | 1263 | lru_cache_disable(); |
| 1318 | − } | |
| 1319 | 1264 | { |
| 1320 | 1265 | NODEMASK_SCRATCH(scratch); |
| 1321 | 1266 | if (scratch) { |
| @@ -1331,45 +1276,81 @@ static long do_mbind(unsigned long start, unsigned long len, · 9 unmodified lines | ||
| 1331 | 1276 | goto mpol_out; |
| 1332 | 1277 | |
| 1333 | 1278 | /* |
| 1334 | − * Lock the VMAs before scanning for pages to migrate, to ensure we don't | |
| 1335 | − * miss a concurrently inserted page. | |
| 1279 | + * Lock the VMAs before scanning for pages to migrate, | |
| 1280 | + * to ensure we don't miss a concurrently inserted page. | |
| 1336 | 1281 | */ |
| 1337 | − ret = queue_pages_range(mm, start, end, nmask, | |
| 1338 | − flags | MPOL_MF_INVERT, &pagelist, true); | |
| 1282 | + nr_failed = queue_pages_range(mm, start, end, nmask, | |
| 1283 | + flags | MPOL_MF_INVERT | MPOL_MF_WRLOCK, &pagelist); | |
| 1339 | 1284 | |
| 1340 | − if (ret < 0) { | |
| 1341 | − err = ret; | |
| 1342 | − goto up_out; | |
| 1343 | − } | |
| 1344 | − | |
| 1345 | − vma_iter_init(&vmi, mm, start); | |
| 1346 | − prev = vma_prev(&vmi); | |
| 1347 | − for_each_vma_range(vmi, vma, end) { | |
| 1348 | − err = mbind_range(&vmi, vma, &prev, start, end, new); | |
| 1349 | − if (err) | |
| 1350 | − break; | |
| 1285 | + if (nr_failed < 0) { | |
| 1286 | + err = nr_failed; | |
| 1287 | + nr_failed = 0; | |
| 1288 | + } else { | |
| 1289 | + vma_iter_init(&vmi, mm, start); | |
| 1290 | + prev = vma_prev(&vmi); | |
| 1291 | + for_each_vma_range(vmi, vma, end) { | |
| 1292 | + err = mbind_range(&vmi, vma, &prev, start, end, new); | |
| 1293 | + if (err) | |
| 1294 | + break; | |
| 1295 | + } | |
| 1351 | 1296 | } |
| 1352 | 1297 | |
| 1353 | − if (!err) { | |
| 1354 | − int nr_failed = 0; | |
| 1355 | − | |
| 1356 | − if (!list_empty(&pagelist)) { | |
| 1357 | − WARN_ON_ONCE(flags & MPOL_MF_LAZY); | |
| 1358 | − nr_failed = migrate_pages(&pagelist, new_folio, NULL, | |
| 1359 | − start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND, NULL); | |
| 1360 | − if (nr_failed) | |
| 1361 | − putback_movable_pages(&pagelist); | |
| 1298 | + if (!err && !list_empty(&pagelist)) { | |
| 1299 | + /* Convert MPOL_DEFAULT's NULL to task or default policy */ | |
| 1300 | + if (!new) { | |
| 1301 | + new = get_task_policy(current); | |
| 1302 | + mpol_get(new); | |
| 1362 | 1303 | } |
| 1304 | + mmpol.pol = new; | |
| 1305 | + mmpol.ilx = 0; | |
| 1363 | 1306 | |
| 1364 | − if (((ret > 0) || nr_failed) && (flags & MPOL_MF_STRICT)) | |
| 1365 | − err = -EIO; | |
| 1366 | − } else { | |
| 1367 | −up_out: | |
| 1368 | − if (!list_empty(&pagelist)) | |
| 1369 | − putback_movable_pages(&pagelist); | |
| 1307 | + /* | |
| 1308 | + * In the interleaved case, attempt to allocate on exactly the | |
| 1309 | + * targeted nodes, for the first VMA to be migrated; for later | |
| 1310 | + * VMAs, the nodes will still be interleaved from the targeted | |
| 1311 | + * nodemask, but one by one may be selected differently. | |
| 1312 | + */ | |
| 1313 | + if (new->mode == MPOL_INTERLEAVE) { | |
| 1314 | + struct page *page; | |
| 1315 | + unsigned int order; | |
| 1316 | + unsigned long addr = -EFAULT; | |
| 1317 | + | |
| 1318 | + list_for_each_entry(page, &pagelist, lru) { | |
| 1319 | + if (!PageKsm(page)) | |
| 1320 | + break; | |
| 1321 | + } | |
| 1322 | + if (!list_entry_is_head(page, &pagelist, lru)) { | |
| 1323 | + vma_iter_init(&vmi, mm, start); | |
| 1324 | + for_each_vma_range(vmi, vma, end) { | |
| 1325 | + addr = page_address_in_vma(page, vma); | |
| 1326 | + if (addr != -EFAULT) | |
| 1327 | + break; | |
| 1328 | + } | |
| 1329 | + } | |
| 1330 | + if (addr != -EFAULT) { | |
| 1331 | + order = compound_order(page); | |
| 1332 | + /* We already know the pol, but not the ilx */ | |
| 1333 | + mpol_cond_put(get_vma_policy(vma, addr, order, | |
| 1334 | + &mmpol.ilx)); | |
| 1335 | + /* Set base from which to increment by index */ | |
| 1336 | + mmpol.ilx -= page->index >> order; | |
| 1337 | + } | |
| 1338 | + } | |
| 1370 | 1339 | } |
| 1371 | 1340 | |
| 1372 | 1341 | mmap_write_unlock(mm); |
| 1342 | + | |
| 1343 | + if (!err && !list_empty(&pagelist)) { | |
| 1344 | + nr_failed |= migrate_pages(&pagelist, | |
| 1345 | + alloc_migration_target_by_mpol, NULL, | |
| 1346 | + (unsigned long)&mmpol, MIGRATE_SYNC, | |
| 1347 | + MR_MEMPOLICY_MBIND, NULL); | |
| 1348 | + } | |
| 1349 | + | |
| 1350 | + if (nr_failed && (flags & MPOL_MF_STRICT)) | |
| 1351 | + err = -EIO; | |
| 1352 | + if (!list_empty(&pagelist)) | |
| 1353 | + putback_movable_pages(&pagelist); | |
| 1373 | 1354 | mpol_out: |
| 1374 | 1355 | mpol_put(new); |
| 1375 | 1356 | if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) |
| @@ -1690,7 +1671,6 @@ out: · 314 unmodified lines | ||
| 1690 | 1671 | out_put: |
| 1691 | 1672 | put_task_struct(task); |
| 1692 | 1673 | goto out; |
| 1693 | − | |
| 1694 | 1674 | } |
| 1695 | 1675 | |
| 1696 | 1676 | SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode, |
| @@ -1700,7 +1680,6 @@ SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode, · 3 unmodified lines | ||
| 1700 | 1680 | return kernel_migrate_pages(pid, maxnode, old_nodes, new_nodes); |
| 1701 | 1681 | } |
| 1702 | 1682 | |
| 1703 | − | |
| 1704 | 1683 | /* Retrieve NUMA policy */ |
| 1705 | 1684 | static int kernel_get_mempolicy(int __user *policy, |
| 1706 | 1685 | unsigned long __user *nmask, |
| @@ -1767,34 +1746,19 @@ bool vma_migratable(struct vm_area_struct *vma) · 60 unmodified lines | ||
| 1767 | 1746 | } |
| 1768 | 1747 | |
| 1769 | 1748 | struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, |
| 1770 | − unsigned long addr) | |
| 1749 | + unsigned long addr, pgoff_t *ilx) | |
| 1771 | 1750 | { |
| 1772 | − struct mempolicy *pol = NULL; | |
| 1773 | − | |
| 1774 | − if (vma) { | |
| 1775 | − if (vma->vm_ops && vma->vm_ops->get_policy) { | |
| 1776 | − pol = vma->vm_ops->get_policy(vma, addr); | |
| 1777 | − } else if (vma->vm_policy) { | |
| 1778 | − pol = vma->vm_policy; | |
| 1779 | − | |
| 1780 | − /* | |
| 1781 | − * shmem_alloc_page() passes MPOL_F_SHARED policy with | |
| 1782 | − * a pseudo vma whose vma->vm_ops=NULL. Take a reference | |
| 1783 | − * count on these policies which will be dropped by | |
| 1784 | − * mpol_cond_put() later | |
| 1785 | − */ | |
| 1786 | − if (mpol_needs_cond_ref(pol)) | |
| 1787 | − mpol_get(pol); | |
| 1788 | − } | |
| 1789 | − } | |
| 1790 | − | |
| 1791 | − return pol; | |
| 1751 | + *ilx = 0; | |
| 1752 | + return (vma->vm_ops && vma->vm_ops->get_policy) ? | |
| 1753 | + vma->vm_ops->get_policy(vma, addr, ilx) : vma->vm_policy; | |
| 1792 | 1754 | } |
| 1793 | 1755 | |
| 1794 | 1756 | /* |
| 1795 | − * get_vma_policy(@vma, @addr) | |
| 1757 | + * get_vma_policy(@vma, @addr, @order, @ilx) | |
| 1796 | 1758 | * @vma: virtual memory area whose policy is sought |
| 1797 | 1759 | * @addr: address in @vma for shared policy lookup |
| 1760 | + * @order: 0, or appropriate huge_page_order for interleaving | |
| 1761 | + * @ilx: interleave index (output), for use only when MPOL_INTERLEAVE | |
| 1798 | 1762 | * |
| 1799 | 1763 | * Returns effective policy for a VMA at specified address. |
| 1800 | 1764 | * Falls back to current->mempolicy or system default policy, as necessary. |
| @@ -1803,14 +1767,18 @@ struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, · 2 unmodified lines | ||
| 1803 | 1767 | * freeing by another task. It is the caller's responsibility to free the |
| 1804 | 1768 | * extra reference for shared policies. |
| 1805 | 1769 | */ |
| 1806 | −static struct mempolicy *get_vma_policy(struct vm_area_struct *vma, | |
| 1807 | − unsigned long addr) | |
| 1770 | +struct mempolicy *get_vma_policy(struct vm_area_struct *vma, | |
| 1771 | + unsigned long addr, int order, pgoff_t *ilx) | |
| 1808 | 1772 | { |
| 1809 | − struct mempolicy *pol = __get_vma_policy(vma, addr); | |
| 1773 | + struct mempolicy *pol; | |
| 1810 | 1774 | |
| 1775 | + pol = __get_vma_policy(vma, addr, ilx); | |
| 1811 | 1776 | if (!pol) |
| 1812 | 1777 | pol = get_task_policy(current); |
| 1813 | − | |
| 1778 | + if (pol->mode == MPOL_INTERLEAVE) { | |
| 1779 | + *ilx += vma->vm_pgoff >> order; | |
| 1780 | + *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order); | |
| 1781 | + } | |
| 1814 | 1782 | return pol; |
| 1815 | 1783 | } |
| 1816 | 1784 | |
| @@ -1820,8 +1788,9 @@ bool vma_policy_mof(struct vm_area_struct *vma) · 3 unmodified lines | ||
| 1820 | 1788 | |
| 1821 | 1789 | if (vma->vm_ops && vma->vm_ops->get_policy) { |
| 1822 | 1790 | bool ret = false; |
| 1791 | + pgoff_t ilx; /* ignored here */ | |
| 1823 | 1792 | |
| 1824 | − pol = vma->vm_ops->get_policy(vma, vma->vm_start); | |
| 1793 | + pol = vma->vm_ops->get_policy(vma, vma->vm_start, &ilx); | |
| 1825 | 1794 | if (pol && (pol->flags & MPOL_F_MOF)) |
| 1826 | 1795 | ret = true; |
| 1827 | 1796 | mpol_cond_put(pol); |
| @@ -1856,64 +1825,15 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) · 28 unmodified lines | ||
| 1856 | 1825 | return zone >= dynamic_policy_zone; |
| 1857 | 1826 | } |
| 1858 | 1827 | |
| 1859 | −/* | |
| 1860 | − * Return a nodemask representing a mempolicy for filtering nodes for | |
| 1861 | − * page allocation | |
| 1862 | − */ | |
| 1863 | −nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy) | |
| 1864 | −{ | |
| 1865 | − int mode = policy->mode; | |
| 1866 | − | |
| 1867 | − /* Lower zones don't get a nodemask applied for MPOL_BIND */ | |
| 1868 | − if (unlikely(mode == MPOL_BIND) && | |
| 1869 | − apply_policy_zone(policy, gfp_zone(gfp)) && | |
| 1870 | − cpuset_nodemask_valid_mems_allowed(&policy->nodes)) | |
| 1871 | − return &policy->nodes; | |
| 1872 | − | |
| 1873 | − if (mode == MPOL_PREFERRED_MANY) | |
| 1874 | − return &policy->nodes; | |
| 1875 | − | |
| 1876 | − return NULL; | |
| 1877 | −} | |
| 1878 | − | |
| 1879 | −/* | |
| 1880 | − * Return the preferred node id for 'prefer' mempolicy, and return | |
| 1881 | − * the given id for all other policies. | |
| 1882 | − * | |
| 1883 | − * policy_node() is always coupled with policy_nodemask(), which | |
| 1884 | − * secures the nodemask limit for 'bind' and 'prefer-many' policy. | |
| 1885 | − */ | |
| 1886 | −static int policy_node(gfp_t gfp, struct mempolicy *policy, int nd) | |
| 1887 | −{ | |
| 1888 | − if (policy->mode == MPOL_PREFERRED) { | |
| 1889 | − nd = first_node(policy->nodes); | |
| 1890 | − } else { | |
| 1891 | − /* | |
| 1892 | − * __GFP_THISNODE shouldn't even be used with the bind policy | |
| 1893 | − * because we might easily break the expectation to stay on the | |
| 1894 | − * requested node and not break the policy. | |
| 1895 | − */ | |
| 1896 | − WARN_ON_ONCE(policy->mode == MPOL_BIND && (gfp & __GFP_THISNODE)); | |
| 1897 | − } | |
| 1898 | − | |
| 1899 | − if ((policy->mode == MPOL_BIND || | |
| 1900 | − policy->mode == MPOL_PREFERRED_MANY) && | |
| 1901 | − policy->home_node != NUMA_NO_NODE) | |
| 1902 | − return policy->home_node; | |
| 1903 | − | |
| 1904 | − return nd; | |
| 1905 | −} | |
| 1906 | − | |
| 1907 | 1828 | /* Do dynamic interleaving for a process */ |
| 1908 | −static unsigned interleave_nodes(struct mempolicy *policy) | |
| 1829 | +static unsigned int interleave_nodes(struct mempolicy *policy) | |
| 1909 | 1830 | { |
| 1910 | − unsigned next; | |
| 1911 | − struct task_struct *me = current; | |
| 1831 | + unsigned int nid; | |
| 1912 | 1832 | |
| 1913 | − next = next_node_in(me->il_prev, policy->nodes); | |
| 1914 | − if (next < MAX_NUMNODES) | |
| 1915 | − me->il_prev = next; | |
| 1916 | − return next; | |
| 1833 | + nid = next_node_in(current->il_prev, policy->nodes); | |
| 1834 | + if (nid < MAX_NUMNODES) | |
| 1835 | + current->il_prev = nid; | |
| 1836 | + return nid; | |
| 1917 | 1837 | } |
| 1918 | 1838 | |
| 1919 | 1839 | /* |
| @@ -1964,11 +1884,11 @@ unsigned int mempolicy_slab_node(void) · 44 unmodified lines | ||
| 1964 | 1884 | } |
| 1965 | 1885 | |
| 1966 | 1886 | /* |
| 1967 | − * Do static interleaving for a VMA with known offset @n. Returns the n'th | |
| 1968 | − * node in pol->nodes (starting from n=0), wrapping around if n exceeds the | |
| 1969 | − * number of present nodes. | |
| 1887 | + * Do static interleaving for interleave index @ilx. Returns the ilx'th | |
| 1888 | + * node in pol->nodes (starting from ilx=0), wrapping around if ilx | |
| 1889 | + * exceeds the number of present nodes. | |
| 1970 | 1890 | */ |
| 1971 | −static unsigned offset_il_node(struct mempolicy *pol, unsigned long n) | |
| 1891 | +static unsigned int interleave_nid(struct mempolicy *pol, pgoff_t ilx) | |
| 1972 | 1892 | { |
| 1973 | 1893 | nodemask_t nodemask = pol->nodes; |
| 1974 | 1894 | unsigned int target, nnodes; |
| @@ -1986,33 +1906,54 @@ static unsigned offset_il_node(struct mempolicy *pol, unsigned long n) · 11 unmodified lines | ||
| 1986 | 1906 | nnodes = nodes_weight(nodemask); |
| 1987 | 1907 | if (!nnodes) |
| 1988 | 1908 | return numa_node_id(); |
| 1989 | − target = (unsigned int)n % nnodes; | |
| 1909 | + target = ilx % nnodes; | |
| 1990 | 1910 | nid = first_node(nodemask); |
| 1991 | 1911 | for (i = 0; i < target; i++) |
| 1992 | 1912 | nid = next_node(nid, nodemask); |
| 1993 | 1913 | return nid; |
| 1994 | 1914 | } |
| 1995 | 1915 | |
| 1996 | −/* Determine a node number for interleave */ | |
| 1997 | −static inline unsigned interleave_nid(struct mempolicy *pol, | |
| 1998 | − struct vm_area_struct *vma, unsigned long addr, int shift) | |
| 1916 | +/* | |
| 1917 | + * Return a nodemask representing a mempolicy for filtering nodes for | |
| 1918 | + * page allocation, together with preferred node id (or the input node id). | |
| 1919 | + */ | |
| 1920 | +static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, | |
| 1921 | + pgoff_t ilx, int *nid) | |
| 1999 | 1922 | { |
| 2000 | − if (vma) { | |
| 2001 | − unsigned long off; | |
| 1923 | + nodemask_t *nodemask = NULL; | |
| 2002 | 1924 | |
| 1925 | + switch (pol->mode) { | |
| 1926 | + case MPOL_PREFERRED: | |
| 1927 | + /* Override input node id */ | |
| 1928 | + *nid = first_node(pol->nodes); | |
| 1929 | + break; | |
| 1930 | + case MPOL_PREFERRED_MANY: | |
| 1931 | + nodemask = &pol->nodes; | |
| 1932 | + if (pol->home_node != NUMA_NO_NODE) | |
| 1933 | + *nid = pol->home_node; | |
| 1934 | + break; | |
| 1935 | + case MPOL_BIND: | |
| 1936 | + /* Restrict to nodemask (but not on lower zones) */ | |
| 1937 | + if (apply_policy_zone(pol, gfp_zone(gfp)) && | |
| 1938 | + cpuset_nodemask_valid_mems_allowed(&pol->nodes)) | |
| 1939 | + nodemask = &pol->nodes; | |
| 1940 | + if (pol->home_node != NUMA_NO_NODE) | |
| 1941 | + *nid = pol->home_node; | |
| 2003 | 1942 | /* |
| 2004 | − * for small pages, there is no difference between | |
| 2005 | − * shift and PAGE_SHIFT, so the bit-shift is safe. | |
| 2006 | − * for huge pages, since vm_pgoff is in units of small | |
| 2007 | − * pages, we need to shift off the always 0 bits to get | |
| 2008 | − * a useful offset. | |
| 1943 | + * __GFP_THISNODE shouldn't even be used with the bind policy | |
| 1944 | + * because we might easily break the expectation to stay on the | |
| 1945 | + * requested node and not break the policy. | |
| 2009 | 1946 | */ |
| 2010 | − BUG_ON(shift < PAGE_SHIFT); | |
| 2011 | − off = vma->vm_pgoff >> (shift - PAGE_SHIFT); | |
| 2012 | − off += (addr - vma->vm_start) >> shift; | |
| 2013 | − return offset_il_node(pol, off); | |
| 2014 | − } else | |
| 2015 | − return interleave_nodes(pol); | |
| 1947 | + WARN_ON_ONCE(gfp & __GFP_THISNODE); | |
| 1948 | + break; | |
| 1949 | + case MPOL_INTERLEAVE: | |
| 1950 | + /* Override input node id */ | |
| 1951 | + *nid = (ilx == NO_INTERLEAVE_INDEX) ? | |
| 1952 | + interleave_nodes(pol) : interleave_nid(pol, ilx); | |
| 1953 | + break; | |
| 1954 | + } | |
| 1955 | + | |
| 1956 | + return nodemask; | |
| 2016 | 1957 | } |
| 2017 | 1958 | |
| 2018 | 1959 | #ifdef CONFIG_HUGETLBFS |
| @@ -2028,27 +1969,16 @@ static inline unsigned interleave_nid(struct mempolicy *pol, · 9 unmodified lines | ||
| 2028 | 1969 | * to the struct mempolicy for conditional unref after allocation. |
| 2029 | 1970 | * If the effective policy is 'bind' or 'prefer-many', returns a pointer |
| 2030 | 1971 | * to the mempolicy's @nodemask for filtering the zonelist. |
| 2031 | − * | |
| 2032 | − * Must be protected by read_mems_allowed_begin() | |
| 2033 | 1972 | */ |
| 2034 | 1973 | int huge_node(struct vm_area_struct *vma, unsigned long addr, gfp_t gfp_flags, |
| 2035 | − struct mempolicy **mpol, nodemask_t **nodemask) | |
| 1974 | + struct mempolicy **mpol, nodemask_t **nodemask) | |
| 2036 | 1975 | { |
| 1976 | + pgoff_t ilx; | |
| 2037 | 1977 | int nid; |
| 2038 | − int mode; | |
| 2039 | − | |
| 2040 | − *mpol = get_vma_policy(vma, addr); | |
| 2041 | − *nodemask = NULL; | |
| 2042 | − mode = (*mpol)->mode; | |
| 2043 | 1978 | |
| 2044 | − if (unlikely(mode == MPOL_INTERLEAVE)) { | |
| 2045 | − nid = interleave_nid(*mpol, vma, addr, | |
| 2046 | − huge_page_shift(hstate_vma(vma))); | |
| 2047 | − } else { | |
| 2048 | − nid = policy_node(gfp_flags, *mpol, numa_node_id()); | |
| 2049 | − if (mode == MPOL_BIND || mode == MPOL_PREFERRED_MANY) | |
| 2050 | − *nodemask = &(*mpol)->nodes; | |
| 2051 | − } | |
| 1979 | + nid = numa_node_id(); | |
| 1980 | + *mpol = get_vma_policy(vma, addr, hstate_vma(vma)->order, &ilx); | |
| 1981 | + *nodemask = policy_nodemask(gfp_flags, *mpol, ilx, &nid); | |
| 2052 | 1982 | return nid; |
| 2053 | 1983 | } |
| 2054 | 1984 | |
| @@ -2126,27 +2056,8 @@ bool mempolicy_in_oom_domain(struct task_struct *tsk, · 71 unmodified lines | ||
| 2126 | 2056 | return ret; |
| 2127 | 2057 | } |
| 2128 | 2058 | |
| 2129 | −/* Allocate a page in interleaved policy. | |
| 2130 | − Own path because it needs to do special accounting. */ | |
| 2131 | −static struct page *alloc_page_interleave(gfp_t gfp, unsigned order, | |
| 2132 | − unsigned nid) | |
| 2133 | −{ | |
| 2134 | − struct page *page; | |
| 2135 | − | |
| 2136 | − page = __alloc_pages(gfp, order, nid, NULL); | |
| 2137 | − /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */ | |
| 2138 | − if (!static_branch_likely(&vm_numa_stat_key)) | |
| 2139 | − return page; | |
| 2140 | − if (page && page_to_nid(page) == nid) { | |
| 2141 | − preempt_disable(); | |
| 2142 | − __count_numa_event(page_zone(page), NUMA_INTERLEAVE_HIT); | |
| 2143 | − preempt_enable(); | |
| 2144 | − } | |
| 2145 | − return page; | |
| 2146 | −} | |
| 2147 | − | |
| 2148 | 2059 | static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, |
| 2149 | − int nid, struct mempolicy *pol) | |
| 2060 | + int nid, nodemask_t *nodemask) | |
| 2150 | 2061 | { |
| 2151 | 2062 | struct page *page; |
| 2152 | 2063 | gfp_t preferred_gfp; |
| @@ -2159,7 +2070,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, · 6 unmodified lines | ||
| 2159 | 2070 | */ |
| 2160 | 2071 | preferred_gfp = gfp | __GFP_NOWARN; |
| 2161 | 2072 | preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); |
| 2162 | − page = __alloc_pages(preferred_gfp, order, nid, &pol->nodes); | |
| 2073 | + page = __alloc_pages(preferred_gfp, order, nid, nodemask); | |
| 2163 | 2074 | if (!page) |
| 2164 | 2075 | page = __alloc_pages(gfp, order, nid, NULL); |
| 2165 | 2076 | |
| @@ -2167,61 +2078,29 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, · 1 unmodified line | ||
| 2167 | 2078 | } |
| 2168 | 2079 | |
| 2169 | 2080 | /** |
| 2170 | − * vma_alloc_folio - Allocate a folio for a VMA. | |
| 2081 | + * alloc_pages_mpol - Allocate pages according to NUMA mempolicy. | |
| 2171 | 2082 | * @gfp: GFP flags. |
| 2172 | − * @order: Order of the folio. | |
| 2173 | − * @vma: Pointer to VMA or NULL if not available. | |
| 2174 | − * @addr: Virtual address of the allocation. Must be inside @vma. | |
| 2175 | − * @hugepage: For hugepages try only the preferred node if possible. | |
| 2083 | + * @order: Order of the page allocation. | |
| 2084 | + * @pol: Pointer to the NUMA mempolicy. | |
| 2085 | + * @ilx: Index for interleave mempolicy (also distinguishes alloc_pages()). | |
| 2086 | + * @nid: Preferred node (usually numa_node_id() but @mpol may override it). | |
| 2176 | 2087 | * |
| 2177 | − * Allocate a folio for a specific address in @vma, using the appropriate | |
| 2178 | − * NUMA policy. When @vma is not NULL the caller must hold the mmap_lock | |
| 2179 | − * of the mm_struct of the VMA to prevent it from going away. Should be | |
| 2180 | − * used for all allocations for folios that will be mapped into user space. | |
| 2181 | − * | |
| 2182 | − * Return: The folio on success or NULL if allocation fails. | |
| 2088 | + * Return: The page on success or NULL if allocation fails. | |
| 2183 | 2089 | */ |
| 2184 | −struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, | |
| 2185 | − unsigned long addr, bool hugepage) | |
| 2090 | +struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, | |
| 2091 | + struct mempolicy *pol, pgoff_t ilx, int nid) | |
| 2186 | 2092 | { |
| 2187 | − struct mempolicy *pol; | |
| 2188 | − int node = numa_node_id(); | |
| 2189 | − struct folio *folio; | |
| 2190 | − int preferred_nid; | |
| 2191 | − nodemask_t *nmask; | |
| 2192 | − | |
| 2193 | − pol = get_vma_policy(vma, addr); | |
| 2194 | − | |
| 2195 | − if (pol->mode == MPOL_INTERLEAVE) { | |
| 2196 | − struct page *page; | |
| 2197 | − unsigned nid; | |
| 2198 | − | |
| 2199 | − nid = interleave_nid(pol, vma, addr, PAGE_SHIFT + order); | |
| 2200 | − mpol_cond_put(pol); | |
| 2201 | − gfp |= __GFP_COMP; | |
| 2202 | − page = alloc_page_interleave(gfp, order, nid); | |
| 2203 | − folio = (struct folio *)page; | |
| 2204 | − if (folio && order > 1) | |
| 2205 | − folio_prep_large_rmappable(folio); | |
| 2206 | − goto out; | |
| 2207 | − } | |
| 2208 | − | |
| 2209 | − if (pol->mode == MPOL_PREFERRED_MANY) { | |
| 2210 | − struct page *page; | |
| 2093 | + nodemask_t *nodemask; | |
| 2094 | + struct page *page; | |
| 2211 | 2095 | |
| 2212 | − node = policy_node(gfp, pol, node); | |
| 2213 | − gfp |= __GFP_COMP; | |
| 2214 | − page = alloc_pages_preferred_many(gfp, order, node, pol); | |
| 2215 | − mpol_cond_put(pol); | |
| 2216 | − folio = (struct folio *)page; | |
| 2217 | − if (folio && order > 1) | |
| 2218 | − folio_prep_large_rmappable(folio); | |
| 2219 | − goto out; | |
| 2220 | − } | |
| 2096 | + nodemask = policy_nodemask(gfp, pol, ilx, &nid); | |
| 2221 | 2097 | |
| 2222 | − if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) { | |
| 2223 | − int hpage_node = node; | |
| 2098 | + if (pol->mode == MPOL_PREFERRED_MANY) | |
| 2099 | + return alloc_pages_preferred_many(gfp, order, nid, nodemask); | |
| 2224 | 2100 | |
| 2101 | + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && | |
| 2102 | + /* filter "hugepage" allocation, unless from alloc_pages() */ | |
| 2103 | + order == HPAGE_PMD_ORDER && ilx != NO_INTERLEAVE_INDEX) { | |
| 2225 | 2104 | /* |
| 2226 | 2105 | * For hugepage allocation and non-interleave policy which |
| 2227 | 2106 | * allows the current node (or other explicitly preferred |
| @@ -2232,39 +2111,68 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, · 4 unmodified lines | ||
| 2232 | 2111 | * If the policy is interleave or does not allow the current |
| 2233 | 2112 | * node in its nodemask, we allocate the standard way. |
| 2234 | 2113 | */ |
| 2235 | − if (pol->mode == MPOL_PREFERRED) | |
| 2236 | − hpage_node = first_node(pol->nodes); | |
| 2237 | − | |
| 2238 | − nmask = policy_nodemask(gfp, pol); | |
| 2239 | − if (!nmask || node_isset(hpage_node, *nmask)) { | |
| 2240 | − mpol_cond_put(pol); | |
| 2114 | + if (pol->mode != MPOL_INTERLEAVE && | |
| 2115 | + (!nodemask || node_isset(nid, *nodemask))) { | |
| 2241 | 2116 | /* |
| 2242 | 2117 | * First, try to allocate THP only on local node, but |
| 2243 | 2118 | * don't reclaim unnecessarily, just compact. |
| 2244 | 2119 | */ |
| 2245 | − folio = __folio_alloc_node(gfp | __GFP_THISNODE | | |
| 2246 | − __GFP_NORETRY, order, hpage_node); | |
| 2247 | − | |
| 2120 | + page = __alloc_pages_node(nid, | |
| 2121 | + gfp | __GFP_THISNODE | __GFP_NORETRY, order); | |
| 2122 | + if (page || !(gfp & __GFP_DIRECT_RECLAIM)) | |
| 2123 | + return page; | |
| 2248 | 2124 | /* |
| 2249 | 2125 | * If hugepage allocations are configured to always |
| 2250 | 2126 | * synchronous compact or the vma has been madvised |
| 2251 | 2127 | * to prefer hugepage backing, retry allowing remote |
| 2252 | 2128 | * memory with both reclaim and compact as well. |
| 2253 | 2129 | */ |
| 2254 | − if (!folio && (gfp & __GFP_DIRECT_RECLAIM)) | |
| 2255 | − folio = __folio_alloc(gfp, order, hpage_node, | |
| 2256 | − nmask); | |
| 2130 | + } | |
| 2131 | + } | |
| 2257 | 2132 | |
| 2258 | − goto out; | |
| 2133 | + page = __alloc_pages(gfp, order, nid, nodemask); | |
| 2134 | + | |
| 2135 | + if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) { | |
| 2136 | + /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */ | |
| 2137 | + if (static_branch_likely(&vm_numa_stat_key) && | |
| 2138 | + page_to_nid(page) == nid) { | |
| 2139 | + preempt_disable(); | |
| 2140 | + __count_numa_event(page_zone(page), NUMA_INTERLEAVE_HIT); | |
| 2141 | + preempt_enable(); | |
| 2259 | 2142 | } |
| 2260 | 2143 | } |
| 2261 | 2144 | |
| 2262 | − nmask = policy_nodemask(gfp, pol); | |
| 2263 | − preferred_nid = policy_node(gfp, pol, node); | |
| 2264 | − folio = __folio_alloc(gfp, order, preferred_nid, nmask); | |
| 2145 | + return page; | |
| 2146 | +} | |
| 2147 | + | |
| 2148 | +/** | |
| 2149 | + * vma_alloc_folio - Allocate a folio for a VMA. | |
| 2150 | + * @gfp: GFP flags. | |
| 2151 | + * @order: Order of the folio. | |
| 2152 | + * @vma: Pointer to VMA. | |
| 2153 | + * @addr: Virtual address of the allocation. Must be inside @vma. | |
| 2154 | + * @hugepage: Unused (was: For hugepages try only preferred node if possible). | |
| 2155 | + * | |
| 2156 | + * Allocate a folio for a specific address in @vma, using the appropriate | |
| 2157 | + * NUMA policy. The caller must hold the mmap_lock of the mm_struct of the | |
| 2158 | + * VMA to prevent it from going away. Should be used for all allocations | |
| 2159 | + * for folios that will be mapped into user space, excepting hugetlbfs, and | |
| 2160 | + * excepting where direct use of alloc_pages_mpol() is more appropriate. | |
| 2161 | + * | |
| 2162 | + * Return: The folio on success or NULL if allocation fails. | |
| 2163 | + */ | |
| 2164 | +struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, | |
| 2165 | + unsigned long addr, bool hugepage) | |
| 2166 | +{ | |
| 2167 | + struct mempolicy *pol; | |
| 2168 | + pgoff_t ilx; | |
| 2169 | + struct page *page; | |
| 2170 | + | |
| 2171 | + pol = get_vma_policy(vma, addr, order, &ilx); | |
| 2172 | + page = alloc_pages_mpol(gfp | __GFP_COMP, order, | |
| 2173 | + pol, ilx, numa_node_id()); | |
| 2265 | 2174 | mpol_cond_put(pol); |
| 2266 | −out: | |
| 2267 | − return folio; | |
| 2175 | + return page_rmappable_folio(page); | |
| 2268 | 2176 | } |
| 2269 | 2177 | EXPORT_SYMBOL(vma_alloc_folio); |
| 2270 | 2178 | |
| @@ -2282,40 +2190,25 @@ EXPORT_SYMBOL(vma_alloc_folio); · 11 unmodified lines | ||
| 2282 | 2190 | * flags are used. |
| 2283 | 2191 | * Return: The page on success or NULL if allocation fails. |
| 2284 | 2192 | */ |
| 2285 | −struct page *alloc_pages(gfp_t gfp, unsigned order) | |
| 2193 | +struct page *alloc_pages(gfp_t gfp, unsigned int order) | |
| 2286 | 2194 | { |
| 2287 | 2195 | struct mempolicy *pol = &default_policy; |
| 2288 | − struct page *page; | |
| 2289 | − | |
| 2290 | − if (!in_interrupt() && !(gfp & __GFP_THISNODE)) | |
| 2291 | − pol = get_task_policy(current); | |
| 2292 | 2196 | |
| 2293 | 2197 | /* |
| 2294 | 2198 | * No reference counting needed for current->mempolicy |
| 2295 | 2199 | * nor system default_policy |
| 2296 | 2200 | */ |
| 2297 | − if (pol->mode == MPOL_INTERLEAVE) | |
| 2298 | − page = alloc_page_interleave(gfp, order, interleave_nodes(pol)); | |
| 2299 | − else if (pol->mode == MPOL_PREFERRED_MANY) | |
| 2300 | − page = alloc_pages_preferred_many(gfp, order, | |
| 2301 | − policy_node(gfp, pol, numa_node_id()), pol); | |
| 2302 | − else | |
| 2303 | − page = __alloc_pages(gfp, order, | |
| 2304 | − policy_node(gfp, pol, numa_node_id()), | |
| 2305 | − policy_nodemask(gfp, pol)); | |
| 2201 | + if (!in_interrupt() && !(gfp & __GFP_THISNODE)) | |
| 2202 | + pol = get_task_policy(current); | |
| 2306 | 2203 | |
| 2307 | − return page; | |
| 2204 | + return alloc_pages_mpol(gfp, order, | |
| 2205 | + pol, NO_INTERLEAVE_INDEX, numa_node_id()); | |
| 2308 | 2206 | } |
| 2309 | 2207 | EXPORT_SYMBOL(alloc_pages); |
| 2310 | 2208 | |
| 2311 | −struct folio *folio_alloc(gfp_t gfp, unsigned order) | |
| 2209 | +struct folio *folio_alloc(gfp_t gfp, unsigned int order) | |
| 2312 | 2210 | { |
| 2313 | − struct page *page = alloc_pages(gfp | __GFP_COMP, order); | |
| 2314 | − struct folio *folio = (struct folio *)page; | |
| 2315 | − | |
| 2316 | − if (folio && order > 1) | |
| 2317 | − folio_prep_large_rmappable(folio); | |
| 2318 | − return folio; | |
| 2211 | + return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order)); | |
| 2319 | 2212 | } |
| 2320 | 2213 | EXPORT_SYMBOL(folio_alloc); |
| 2321 | 2214 | |
| @@ -2384,6 +2277,8 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, · 62 unmodified lines | ||
| 2384 | 2277 | unsigned long nr_pages, struct page **page_array) |
| 2385 | 2278 | { |
| 2386 | 2279 | struct mempolicy *pol = &default_policy; |
| 2280 | + nodemask_t *nodemask; | |
| 2281 | + int nid; | |
| 2387 | 2282 | |
| 2388 | 2283 | if (!in_interrupt() && !(gfp & __GFP_THISNODE)) |
| 2389 | 2284 | pol = get_task_policy(current); |
| @@ -2396,14 +2291,15 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, · 6 unmodified lines | ||
| 2396 | 2291 | return alloc_pages_bulk_array_preferred_many(gfp, |
| 2397 | 2292 | numa_node_id(), pol, nr_pages, page_array); |
| 2398 | 2293 | |
| 2399 | − return __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()), | |
| 2400 | − policy_nodemask(gfp, pol), nr_pages, NULL, | |
| 2401 | − page_array); | |
| 2294 | + nid = numa_node_id(); | |
| 2295 | + nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid); | |
| 2296 | + return __alloc_pages_bulk(gfp, nid, nodemask, | |
| 2297 | + nr_pages, NULL, page_array); | |
| 2402 | 2298 | } |
| 2403 | 2299 | |
| 2404 | 2300 | int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) |
| 2405 | 2301 | { |
| 2406 | − struct mempolicy *pol = mpol_dup(vma_policy(src)); | |
| 2302 | + struct mempolicy *pol = mpol_dup(src->vm_policy); | |
| 2407 | 2303 | |
| 2408 | 2304 | if (IS_ERR(pol)) |
| 2409 | 2305 | return PTR_ERR(pol); |
| @@ -2488,8 +2384,8 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b) · 78 unmodified lines | ||
| 2488 | 2384 | * lookup first element intersecting start-end. Caller holds sp->lock for |
| 2489 | 2385 | * reading or for writing |
| 2490 | 2386 | */ |
| 2491 | −static struct sp_node * | |
| 2492 | −sp_lookup(struct shared_policy *sp, unsigned long start, unsigned long end) | |
| 2387 | +static struct sp_node *sp_lookup(struct shared_policy *sp, | |
| 2388 | + pgoff_t start, pgoff_t end) | |
| 2493 | 2389 | { |
| 2494 | 2390 | struct rb_node *n = sp->root.rb_node; |
| 2495 | 2391 | |
| @@ -2540,13 +2436,11 @@ static void sp_insert(struct shared_policy *sp, struct sp_node *new) · 44 unmodified lines | ||
| 2540 | 2436 | } |
| 2541 | 2437 | rb_link_node(&new->nd, parent, p); |
| 2542 | 2438 | rb_insert_color(&new->nd, &sp->root); |
| 2543 | − pr_debug("inserting %lx-%lx: %d\n", new->start, new->end, | |
| 2544 | − new->policy ? new->policy->mode : 0); | |
| 2545 | 2439 | } |
| 2546 | 2440 | |
| 2547 | 2441 | /* Find shared policy intersecting idx */ |
| 2548 | −struct mempolicy * | |
| 2549 | −mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx) | |
| 2442 | +struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp, | |
| 2443 | + pgoff_t idx) | |
| 2550 | 2444 | { |
| 2551 | 2445 | struct mempolicy *pol = NULL; |
| 2552 | 2446 | struct sp_node *sn; |
| @@ -2570,39 +2464,38 @@ static void sp_free(struct sp_node *n) · 17 unmodified lines | ||
| 2570 | 2464 | } |
| 2571 | 2465 | |
| 2572 | 2466 | /** |
| 2573 | − * mpol_misplaced - check whether current page node is valid in policy | |
| 2467 | + * mpol_misplaced - check whether current folio node is valid in policy | |
| 2574 | 2468 | * |
| 2575 | − * @page: page to be checked | |
| 2576 | − * @vma: vm area where page mapped | |
| 2577 | − * @addr: virtual address where page mapped | |
| 2469 | + * @folio: folio to be checked | |
| 2470 | + * @vma: vm area where folio mapped | |
| 2471 | + * @addr: virtual address in @vma for shared policy lookup and interleave policy | |
| 2578 | 2472 | * |
| 2579 | − * Lookup current policy node id for vma,addr and "compare to" page's | |
| 2473 | + * Lookup current policy node id for vma,addr and "compare to" folio's | |
| 2580 | 2474 | * node id. Policy determination "mimics" alloc_page_vma(). |
| 2581 | 2475 | * Called from fault path where we know the vma and faulting address. |
| 2582 | 2476 | * |
| 2583 | 2477 | * Return: NUMA_NO_NODE if the page is in a node that is valid for this |
| 2584 | − * policy, or a suitable node ID to allocate a replacement page from. | |
| 2478 | + * policy, or a suitable node ID to allocate a replacement folio from. | |
| 2585 | 2479 | */ |
| 2586 | −int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long addr) | |
| 2480 | +int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma, | |
| 2481 | + unsigned long addr) | |
| 2587 | 2482 | { |
| 2588 | 2483 | struct mempolicy *pol; |
| 2484 | + pgoff_t ilx; | |
| 2589 | 2485 | struct zoneref *z; |
| 2590 | − int curnid = page_to_nid(page); | |
| 2591 | − unsigned long pgoff; | |
| 2486 | + int curnid = folio_nid(folio); | |
| 2592 | 2487 | int thiscpu = raw_smp_processor_id(); |
| 2593 | 2488 | int thisnid = cpu_to_node(thiscpu); |
| 2594 | 2489 | int polnid = NUMA_NO_NODE; |
| 2595 | 2490 | int ret = NUMA_NO_NODE; |
| 2596 | 2491 | |
| 2597 | − pol = get_vma_policy(vma, addr); | |
| 2492 | + pol = get_vma_policy(vma, addr, folio_order(folio), &ilx); | |
| 2598 | 2493 | if (!(pol->flags & MPOL_F_MOF)) |
| 2599 | 2494 | goto out; |
| 2600 | 2495 | |
| 2601 | 2496 | switch (pol->mode) { |
| 2602 | 2497 | case MPOL_INTERLEAVE: |
| 2603 | − pgoff = vma->vm_pgoff; | |
| 2604 | − pgoff += (addr - vma->vm_start) >> PAGE_SHIFT; | |
| 2605 | − polnid = offset_il_node(pol, pgoff); | |
| 2498 | + polnid = interleave_nid(pol, ilx); | |
| 2606 | 2499 | break; |
| 2607 | 2500 | |
| 2608 | 2501 | case MPOL_PREFERRED: |
| @@ -2643,11 +2536,12 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long · 34 unmodified lines | ||
| 2643 | 2536 | BUG(); |
| 2644 | 2537 | } |
| 2645 | 2538 | |
| 2646 | − /* Migrate the page towards the node whose CPU is referencing it */ | |
| 2539 | + /* Migrate the folio towards the node whose CPU is referencing it */ | |
| 2647 | 2540 | if (pol->flags & MPOL_F_MORON) { |
| 2648 | 2541 | polnid = thisnid; |
| 2649 | 2542 | |
| 2650 | − if (!should_numa_migrate_memory(current, page, curnid, thiscpu)) | |
| 2543 | + if (!should_numa_migrate_memory(current, folio, curnid, | |
| 2544 | + thiscpu)) | |
| 2651 | 2545 | goto out; |
| 2652 | 2546 | } |
| 2653 | 2547 | |
| @@ -2678,7 +2572,6 @@ void mpol_put_task_policy(struct task_struct *task) · 24 unmodified lines | ||
| 2678 | 2572 | |
| 2679 | 2573 | static void sp_delete(struct shared_policy *sp, struct sp_node *n) |
| 2680 | 2574 | { |
| 2681 | − pr_debug("deleting %lx-l%lx\n", n->start, n->end); | |
| 2682 | 2575 | rb_erase(&n->nd, &sp->root); |
| 2683 | 2576 | sp_free(n); |
| 2684 | 2577 | } |
| @@ -2713,8 +2606,8 @@ static struct sp_node *sp_alloc(unsigned long start, unsigned long end, · 28 unmodified lines | ||
| 2713 | 2606 | } |
| 2714 | 2607 | |
| 2715 | 2608 | /* Replace a policy range. */ |
| 2716 | −static int shared_policy_replace(struct shared_policy *sp, unsigned long start, | |
| 2717 | − unsigned long end, struct sp_node *new) | |
| 2609 | +static int shared_policy_replace(struct shared_policy *sp, pgoff_t start, | |
| 2610 | + pgoff_t end, struct sp_node *new) | |
| 2718 | 2611 | { |
| 2719 | 2612 | struct sp_node *n; |
| 2720 | 2613 | struct sp_node *n_new = NULL; |
| @@ -2797,30 +2690,30 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol) · 76 unmodified lines | ||
| 2797 | 2690 | rwlock_init(&sp->lock); |
| 2798 | 2691 | |
| 2799 | 2692 | if (mpol) { |
| 2800 | − struct vm_area_struct pvma; | |
| 2801 | − struct mempolicy *new; | |
| 2693 | + struct sp_node *sn; | |
| 2694 | + struct mempolicy *npol; | |
| 2802 | 2695 | NODEMASK_SCRATCH(scratch); |
| 2803 | 2696 | |
| 2804 | 2697 | if (!scratch) |
| 2805 | 2698 | goto put_mpol; |
| 2806 | − /* contextualize the tmpfs mount point mempolicy */ | |
| 2807 | − new = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask); | |
| 2808 | − if (IS_ERR(new)) | |
| 2699 | + | |
| 2700 | + /* contextualize the tmpfs mount point mempolicy to this file */ | |
| 2701 | + npol = mpol_new(mpol->mode, mpol->flags, &mpol->w.user_nodemask); | |
| 2702 | + if (IS_ERR(npol)) | |
| 2809 | 2703 | goto free_scratch; /* no valid nodemask intersection */ |
| 2810 | 2704 | |
| 2811 | 2705 | task_lock(current); |
| 2812 | − ret = mpol_set_nodemask(new, &mpol->w.user_nodemask, scratch); | |
| 2706 | + ret = mpol_set_nodemask(npol, &mpol->w.user_nodemask, scratch); | |
| 2813 | 2707 | task_unlock(current); |
| 2814 | 2708 | if (ret) |
| 2815 | − goto put_new; | |
| 2816 | − | |
| 2817 | − /* Create pseudo-vma that contains just the policy */ | |
| 2818 | − vma_init(&pvma, NULL); | |
| 2819 | − pvma.vm_end = TASK_SIZE; /* policy covers entire file */ | |
| 2820 | − mpol_set_shared_policy(sp, &pvma, new); /* adds ref */ | |
| 2821 | − | |
| 2822 | −put_new: | |
| 2823 | − mpol_put(new); /* drop initial ref */ | |
| 2709 | + goto put_npol; | |
| 2710 | + | |
| 2711 | + /* alloc node covering entire file; adds ref to file's npol */ | |
| 2712 | + sn = sp_alloc(0, MAX_LFS_FILESIZE >> PAGE_SHIFT, npol); | |
| 2713 | + if (sn) | |
| 2714 | + sp_insert(sp, sn); | |
| 2715 | +put_npol: | |
| 2716 | + mpol_put(npol); /* drop initial ref on file's npol */ | |
| 2824 | 2717 | free_scratch: |
| 2825 | 2718 | NODEMASK_SCRATCH_FREE(scratch); |
| 2826 | 2719 | put_mpol: |
| @@ -2828,46 +2721,40 @@ put_mpol: · 1 unmodified line | ||
| 2828 | 2721 | } |
| 2829 | 2722 | } |
| 2830 | 2723 | |
| 2831 | −int mpol_set_shared_policy(struct shared_policy *info, | |
| 2832 | − struct vm_area_struct *vma, struct mempolicy *npol) | |
| 2724 | +int mpol_set_shared_policy(struct shared_policy *sp, | |
| 2725 | + struct vm_area_struct *vma, struct mempolicy *pol) | |
| 2833 | 2726 | { |
| 2834 | 2727 | int err; |
| 2835 | 2728 | struct sp_node *new = NULL; |
| 2836 | 2729 | unsigned long sz = vma_pages(vma); |
| 2837 | 2730 | |
| 2838 | − pr_debug("set_shared_policy %lx sz %lu %d %d %lx\n", | |
| 2839 | − vma->vm_pgoff, | |
| 2840 | − sz, npol ? npol->mode : -1, | |
| 2841 | − npol ? npol->flags : -1, | |
| 2842 | − npol ? nodes_addr(npol->nodes)[0] : NUMA_NO_NODE); | |
| 2843 | − | |
| 2844 | − if (npol) { | |
| 2845 | − new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, npol); | |
| 2731 | + if (pol) { | |
| 2732 | + new = sp_alloc(vma->vm_pgoff, vma->vm_pgoff + sz, pol); | |
| 2846 | 2733 | if (!new) |
| 2847 | 2734 | return -ENOMEM; |
| 2848 | 2735 | } |
| 2849 | − err = shared_policy_replace(info, vma->vm_pgoff, vma->vm_pgoff+sz, new); | |
| 2736 | + err = shared_policy_replace(sp, vma->vm_pgoff, vma->vm_pgoff + sz, new); | |
| 2850 | 2737 | if (err && new) |
| 2851 | 2738 | sp_free(new); |
| 2852 | 2739 | return err; |
| 2853 | 2740 | } |
| 2854 | 2741 | |
| 2855 | 2742 | /* Free a backing policy store on inode delete. */ |
| 2856 | −void mpol_free_shared_policy(struct shared_policy *p) | |
| 2743 | +void mpol_free_shared_policy(struct shared_policy *sp) | |
| 2857 | 2744 | { |
| 2858 | 2745 | struct sp_node *n; |
| 2859 | 2746 | struct rb_node *next; |
| 2860 | 2747 | |
| 2861 | − if (!p->root.rb_node) | |
| 2748 | + if (!sp->root.rb_node) | |
| 2862 | 2749 | return; |
| 2863 | − write_lock(&p->lock); | |
| 2864 | − next = rb_first(&p->root); | |
| 2750 | + write_lock(&sp->lock); | |
| 2751 | + next = rb_first(&sp->root); | |
| 2865 | 2752 | while (next) { |
| 2866 | 2753 | n = rb_entry(next, struct sp_node, nd); |
| 2867 | 2754 | next = rb_next(&n->nd); |
| 2868 | − sp_delete(p, n); | |
| 2755 | + sp_delete(sp, n); | |
| 2869 | 2756 | } |
| 2870 | − write_unlock(&p->lock); | |
| 2757 | + write_unlock(&sp->lock); | |
| 2871 | 2758 | } |
| 2872 | 2759 | |
| 2873 | 2760 | #ifdef CONFIG_NUMA_BALANCING |
| @@ -2917,7 +2804,6 @@ static inline void __init check_numabalancing_enable(void) · 43 unmodified lines | ||
| 2917 | 2804 | } |
| 2918 | 2805 | #endif /* CONFIG_NUMA_BALANCING */ |
| 2919 | 2806 | |
| 2920 | −/* assumes fs == KERNEL_DS */ | |
| 2921 | 2807 | void __init numa_policy_init(void) |
| 2922 | 2808 | { |
| 2923 | 2809 | nodemask_t interleave_nodes; |
| @@ -2980,7 +2866,6 @@ void numa_default_policy(void) · 56 unmodified lines | ||
| 2980 | 2866 | /* |
| 2981 | 2867 | * Parse and format mempolicy from/to strings |
| 2982 | 2868 | */ |
| 2983 | − | |
| 2984 | 2869 | static const char * const policy_modes[] = |
| 2985 | 2870 | { |
| 2986 | 2871 | [MPOL_DEFAULT] = "default", |
| @@ -2991,7 +2876,6 @@ static const char * const policy_modes[] = · 4 unmodified lines | ||
| 2991 | 2876 | [MPOL_PREFERRED_MANY] = "prefer (many)", |
| 2992 | 2877 | }; |
| 2993 | 2878 | |
| 2994 | − | |
| 2995 | 2879 | #ifdef CONFIG_TMPFS |
| 2996 | 2880 | /** |
| 2997 | 2881 | * mpol_parse_str - parse string to mempolicy, for tmpfs mpol mount option. |
| @@ -157,8 +157,8 @@ void putback_movable_pages(struct list_head *l) · 156 unmodified lines | ||
| 157 | 157 | list_del(&folio->lru); |
| 158 | 158 | /* |
| 159 | 159 | * We isolated non-lru movable folio so here we can use |
| 160 | − * __PageMovable because LRU folio's mapping cannot have | |
| 161 | − * PAGE_MAPPING_MOVABLE. | |
| 160 | + * __folio_test_movable because LRU folio's mapping cannot | |
| 161 | + * have PAGE_MAPPING_MOVABLE. | |
| 162 | 162 | */ |
| 163 | 163 | if (unlikely(__folio_test_movable(folio))) { |
| 164 | 164 | VM_BUG_ON_FOLIO(!folio_test_isolated(folio), folio); |
| @@ -249,7 +249,7 @@ static bool remove_migration_pte(struct folio *folio, · 84 unmodified lines | ||
| 249 | 249 | |
| 250 | 250 | pte = arch_make_huge_pte(pte, shift, vma->vm_flags); |
| 251 | 251 | if (folio_test_anon(folio)) |
| 252 | − hugepage_add_anon_rmap(new, vma, pvmw.address, | |
| 252 | + hugepage_add_anon_rmap(folio, vma, pvmw.address, | |
| 253 | 253 | rmap_flags); |
| 254 | 254 | else |
| 255 | 255 | page_dup_file_rmap(new, true); |
| @@ -524,7 +524,7 @@ int migrate_huge_page_move_mapping(struct address_space *mapping, · 268 unmodified lines | ||
| 524 | 524 | int expected_count; |
| 525 | 525 | |
| 526 | 526 | xas_lock_irq(&xas); |
| 527 | − expected_count = 2 + folio_has_private(src); | |
| 527 | + expected_count = folio_expected_refs(mapping, src); | |
| 528 | 528 | if (!folio_ref_freeze(src, expected_count)) { |
| 529 | 529 | xas_unlock_irq(&xas); |
| 530 | 530 | return -EAGAIN; |
| @@ -533,11 +533,11 @@ int migrate_huge_page_move_mapping(struct address_space *mapping, · 2 unmodified lines | ||
| 533 | 533 | dst->index = src->index; |
| 534 | 534 | dst->mapping = src->mapping; |
| 535 | 535 | |
| 536 | − folio_get(dst); | |
| 536 | + folio_ref_add(dst, folio_nr_pages(dst)); | |
| 537 | 537 | |
| 538 | 538 | xas_store(&xas, dst); |
| 539 | 539 | |
| 540 | − folio_ref_unfreeze(src, expected_count - 1); | |
| 540 | + folio_ref_unfreeze(src, expected_count - folio_nr_pages(src)); | |
| 541 | 541 | |
| 542 | 542 | xas_unlock_irq(&xas); |
| 543 | 543 | |
| @@ -588,20 +588,20 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · 44 unmodified lines | ||
| 588 | 588 | * Copy NUMA information to the new page, to prevent over-eager |
| 589 | 589 | * future migrations of this same page. |
| 590 | 590 | */ |
| 591 | − cpupid = page_cpupid_xchg_last(&folio->page, -1); | |
| 591 | + cpupid = folio_xchg_last_cpupid(folio, -1); | |
| 592 | 592 | /* |
| 593 | 593 | * For memory tiering mode, when migrate between slow and fast |
| 594 | 594 | * memory node, reset cpupid, because that is used to record |
| 595 | 595 | * page access time in slow memory node. |
| 596 | 596 | */ |
| 597 | 597 | if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) { |
| 598 | − bool f_toptier = node_is_toptier(page_to_nid(&folio->page)); | |
| 599 | − bool t_toptier = node_is_toptier(page_to_nid(&newfolio->page)); | |
| 598 | + bool f_toptier = node_is_toptier(folio_nid(folio)); | |
| 599 | + bool t_toptier = node_is_toptier(folio_nid(newfolio)); | |
| 600 | 600 | |
| 601 | 601 | if (f_toptier != t_toptier) |
| 602 | 602 | cpupid = -1; |
| 603 | 603 | } |
| 604 | − page_cpupid_xchg_last(&newfolio->page, cpupid); | |
| 604 | + folio_xchg_last_cpupid(newfolio, cpupid); | |
| 605 | 605 | |
| 606 | 606 | folio_migrate_ksm(newfolio, folio); |
| 607 | 607 | /* |
| @@ -633,8 +633,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) · 25 unmodified lines | ||
| 633 | 633 | |
| 634 | 634 | folio_copy_owner(newfolio, folio); |
| 635 | 635 | |
| 636 | − if (!folio_test_hugetlb(folio)) | |
| 637 | − mem_cgroup_migrate(folio, newfolio); | |
| 636 | + mem_cgroup_migrate(folio, newfolio); | |
| 638 | 637 | } |
| 639 | 638 | EXPORT_SYMBOL(folio_migrate_flags); |
| 640 | 639 | |
| @@ -946,7 +945,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, · 305 unmodified lines | ||
| 946 | 945 | enum migrate_mode mode) |
| 947 | 946 | { |
| 948 | 947 | int rc = -EAGAIN; |
| 949 | − bool is_lru = !__PageMovable(&src->page); | |
| 948 | + bool is_lru = !__folio_test_movable(src); | |
| 950 | 949 | |
| 951 | 950 | VM_BUG_ON_FOLIO(!folio_test_locked(src), src); |
| 952 | 951 | VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst); |
| @@ -993,7 +992,7 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, · 40 unmodified lines | ||
| 993 | 992 | * src is freed; but stats require that PageAnon be left as PageAnon. |
| 994 | 993 | */ |
| 995 | 994 | if (rc == MIGRATEPAGE_SUCCESS) { |
| 996 | − if (__PageMovable(&src->page)) { | |
| 995 | + if (__folio_test_movable(src)) { | |
| 997 | 996 | VM_BUG_ON_FOLIO(!folio_test_isolated(src), src); |
| 998 | 997 | |
| 999 | 998 | /* |
| @@ -1028,22 +1027,28 @@ union migration_ptr { · 28 unmodified lines | ||
| 1028 | 1027 | struct anon_vma *anon_vma; |
| 1029 | 1028 | struct address_space *mapping; |
| 1030 | 1029 | }; |
| 1030 | + | |
| 1031 | +enum { | |
| 1032 | + PAGE_WAS_MAPPED = BIT(0), | |
| 1033 | + PAGE_WAS_MLOCKED = BIT(1), | |
| 1034 | +}; | |
| 1035 | + | |
| 1031 | 1036 | static void __migrate_folio_record(struct folio *dst, |
| 1032 | − unsigned long page_was_mapped, | |
| 1037 | + unsigned long old_page_state, | |
| 1033 | 1038 | struct anon_vma *anon_vma) |
| 1034 | 1039 | { |
| 1035 | 1040 | union migration_ptr ptr = { .anon_vma = anon_vma }; |
| 1036 | 1041 | dst->mapping = ptr.mapping; |
| 1037 | − dst->private = (void *)page_was_mapped; | |
| 1042 | + dst->private = (void *)old_page_state; | |
| 1038 | 1043 | } |
| 1039 | 1044 | |
| 1040 | 1045 | static void __migrate_folio_extract(struct folio *dst, |
| 1041 | − int *page_was_mappedp, | |
| 1046 | + int *old_page_state, | |
| 1042 | 1047 | struct anon_vma **anon_vmap) |
| 1043 | 1048 | { |
| 1044 | 1049 | union migration_ptr ptr = { .mapping = dst->mapping }; |
| 1045 | 1050 | *anon_vmap = ptr.anon_vma; |
| 1046 | − *page_was_mappedp = (unsigned long)dst->private; | |
| 1051 | + *old_page_state = (unsigned long)dst->private; | |
| 1047 | 1052 | dst->mapping = NULL; |
| 1048 | 1053 | dst->private = NULL; |
| 1049 | 1054 | } |
| @@ -1085,7 +1090,7 @@ static void migrate_folio_done(struct folio *src, · 35 unmodified lines | ||
| 1085 | 1090 | /* |
| 1086 | 1091 | * Compaction can migrate also non-LRU pages which are |
| 1087 | 1092 | * not accounted to NR_ISOLATED_*. They can be recognized |
| 1088 | − * as __PageMovable | |
| 1093 | + * as __folio_test_movable | |
| 1089 | 1094 | */ |
| 1090 | 1095 | if (likely(!__folio_test_movable(src))) |
| 1091 | 1096 | mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + |
| @@ -1104,9 +1109,9 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 12 unmodified lines | ||
| 1104 | 1109 | { |
| 1105 | 1110 | struct folio *dst; |
| 1106 | 1111 | int rc = -EAGAIN; |
| 1107 | − int page_was_mapped = 0; | |
| 1112 | + int old_page_state = 0; | |
| 1108 | 1113 | struct anon_vma *anon_vma = NULL; |
| 1109 | − bool is_lru = !__PageMovable(&src->page); | |
| 1114 | + bool is_lru = !__folio_test_movable(src); | |
| 1110 | 1115 | bool locked = false; |
| 1111 | 1116 | bool dst_locked = false; |
| 1112 | 1117 | |
| @@ -1158,6 +1163,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 45 unmodified lines | ||
| 1158 | 1163 | folio_lock(src); |
| 1159 | 1164 | } |
| 1160 | 1165 | locked = true; |
| 1166 | + if (folio_test_mlocked(src)) | |
| 1167 | + old_page_state |= PAGE_WAS_MLOCKED; | |
| 1161 | 1168 | |
| 1162 | 1169 | if (folio_test_writeback(src)) { |
| 1163 | 1170 | /* |
| @@ -1207,7 +1214,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 43 unmodified lines | ||
| 1207 | 1214 | dst_locked = true; |
| 1208 | 1215 | |
| 1209 | 1216 | if (unlikely(!is_lru)) { |
| 1210 | − __migrate_folio_record(dst, page_was_mapped, anon_vma); | |
| 1217 | + __migrate_folio_record(dst, old_page_state, anon_vma); | |
| 1211 | 1218 | return MIGRATEPAGE_UNMAP; |
| 1212 | 1219 | } |
| 1213 | 1220 | |
| @@ -1233,11 +1240,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, · 19 unmodified lines | ||
| 1233 | 1240 | VM_BUG_ON_FOLIO(folio_test_anon(src) && |
| 1234 | 1241 | !folio_test_ksm(src) && !anon_vma, src); |
| 1235 | 1242 | try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); |
| 1236 | − page_was_mapped = 1; | |
| 1243 | + old_page_state |= PAGE_WAS_MAPPED; | |
| 1237 | 1244 | } |
| 1238 | 1245 | |
| 1239 | 1246 | if (!folio_mapped(src)) { |
| 1240 | − __migrate_folio_record(dst, page_was_mapped, anon_vma); | |
| 1247 | + __migrate_folio_record(dst, old_page_state, anon_vma); | |
| 1241 | 1248 | return MIGRATEPAGE_UNMAP; |
| 1242 | 1249 | } |
| 1243 | 1250 | |
| @@ -1249,7 +1256,8 @@ out: · 5 unmodified lines | ||
| 1249 | 1256 | if (rc == -EAGAIN) |
| 1250 | 1257 | ret = NULL; |
| 1251 | 1258 | |
| 1252 | − migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); | |
| 1259 | + migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED, | |
| 1260 | + anon_vma, locked, ret); | |
| 1253 | 1261 | migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private); |
| 1254 | 1262 | |
| 1255 | 1263 | return rc; |
| @@ -1262,12 +1270,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 6 unmodified lines | ||
| 1262 | 1270 | struct list_head *ret) |
| 1263 | 1271 | { |
| 1264 | 1272 | int rc; |
| 1265 | − int page_was_mapped = 0; | |
| 1273 | + int old_page_state = 0; | |
| 1266 | 1274 | struct anon_vma *anon_vma = NULL; |
| 1267 | − bool is_lru = !__PageMovable(&src->page); | |
| 1275 | + bool is_lru = !__folio_test_movable(src); | |
| 1268 | 1276 | struct list_head *prev; |
| 1269 | 1277 | |
| 1270 | − __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); | |
| 1278 | + __migrate_folio_extract(dst, &old_page_state, &anon_vma); | |
| 1271 | 1279 | prev = dst->lru.prev; |
| 1272 | 1280 | list_del(&dst->lru); |
| 1273 | 1281 | |
| @@ -1288,10 +1296,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, · 14 unmodified lines | ||
| 1288 | 1296 | * isolated from the unevictable LRU: but this case is the easiest. |
| 1289 | 1297 | */ |
| 1290 | 1298 | folio_add_lru(dst); |
| 1291 | − if (page_was_mapped) | |
| 1299 | + if (old_page_state & PAGE_WAS_MLOCKED) | |
| 1292 | 1300 | lru_add_drain(); |
| 1293 | 1301 | |
| 1294 | − if (page_was_mapped) | |
| 1302 | + if (old_page_state & PAGE_WAS_MAPPED) | |
| 1295 | 1303 | remove_migration_ptes(src, dst, false); |
| 1296 | 1304 | |
| 1297 | 1305 | out_unlock_both: |
| @@ -1323,11 +1331,12 @@ out: · 25 unmodified lines | ||
| 1323 | 1331 | */ |
| 1324 | 1332 | if (rc == -EAGAIN) { |
| 1325 | 1333 | list_add(&dst->lru, prev); |
| 1326 | − __migrate_folio_record(dst, page_was_mapped, anon_vma); | |
| 1334 | + __migrate_folio_record(dst, old_page_state, anon_vma); | |
| 1327 | 1335 | return rc; |
| 1328 | 1336 | } |
| 1329 | 1337 | |
| 1330 | − migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret); | |
| 1338 | + migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED, | |
| 1339 | + anon_vma, true, ret); | |
| 1331 | 1340 | migrate_folio_undo_dst(dst, true, put_new_folio, private); |
| 1332 | 1341 | |
| 1333 | 1342 | return rc; |
| @@ -1496,6 +1505,7 @@ struct migrate_pages_stats { · 162 unmodified lines | ||
| 1496 | 1505 | int nr_thp_succeeded; /* THP migrated successfully */ |
| 1497 | 1506 | int nr_thp_failed; /* THP failed to be migrated */ |
| 1498 | 1507 | int nr_thp_split; /* THP split before migrating */ |
| 1508 | + int nr_split; /* Large folio (include THP) split before migrating */ | |
| 1499 | 1509 | }; |
| 1500 | 1510 | |
| 1501 | 1511 | /* |
| @@ -1615,6 +1625,7 @@ static int migrate_pages_batch(struct list_head *from, · 113 unmodified lines | ||
| 1615 | 1625 | int nr_retry_pages = 0; |
| 1616 | 1626 | int pass = 0; |
| 1617 | 1627 | bool is_thp = false; |
| 1628 | + bool is_large = false; | |
| 1618 | 1629 | struct folio *folio, *folio2, *dst = NULL, *dst2; |
| 1619 | 1630 | int rc, rc_saved = 0, nr_pages; |
| 1620 | 1631 | LIST_HEAD(unmap_folios); |
| @@ -1630,7 +1641,8 @@ static int migrate_pages_batch(struct list_head *from, · 9 unmodified lines | ||
| 1630 | 1641 | nr_retry_pages = 0; |
| 1631 | 1642 | |
| 1632 | 1643 | list_for_each_entry_safe(folio, folio2, from, lru) { |
| 1633 | − is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); | |
| 1644 | + is_large = folio_test_large(folio); | |
| 1645 | + is_thp = is_large && folio_test_pmd_mappable(folio); | |
| 1634 | 1646 | nr_pages = folio_nr_pages(folio); |
| 1635 | 1647 | |
| 1636 | 1648 | cond_resched(); |
| @@ -1650,6 +1662,7 @@ static int migrate_pages_batch(struct list_head *from, · 13 unmodified lines | ||
| 1650 | 1662 | stats->nr_thp_failed++; |
| 1651 | 1663 | if (!try_split_folio(folio, split_folios)) { |
| 1652 | 1664 | stats->nr_thp_split++; |
| 1665 | + stats->nr_split++; | |
| 1653 | 1666 | continue; |
| 1654 | 1667 | } |
| 1655 | 1668 | stats->nr_failed_pages += nr_pages; |
| @@ -1678,11 +1691,12 @@ static int migrate_pages_batch(struct list_head *from, · 22 unmodified lines | ||
| 1678 | 1691 | nr_failed++; |
| 1679 | 1692 | stats->nr_thp_failed += is_thp; |
| 1680 | 1693 | /* Large folio NUMA faulting doesn't split to retry. */ |
| 1681 | − if (folio_test_large(folio) && !nosplit) { | |
| 1694 | + if (is_large && !nosplit) { | |
| 1682 | 1695 | int ret = try_split_folio(folio, split_folios); |
| 1683 | 1696 | |
| 1684 | 1697 | if (!ret) { |
| 1685 | 1698 | stats->nr_thp_split += is_thp; |
| 1699 | + stats->nr_split++; | |
| 1686 | 1700 | break; |
| 1687 | 1701 | } else if (reason == MR_LONGTERM_PIN && |
| 1688 | 1702 | ret == -EAGAIN) { |
| @@ -1795,12 +1809,12 @@ out: · 106 unmodified lines | ||
| 1795 | 1809 | dst = list_first_entry(&dst_folios, struct folio, lru); |
| 1796 | 1810 | dst2 = list_next_entry(dst, lru); |
| 1797 | 1811 | list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { |
| 1798 | − int page_was_mapped = 0; | |
| 1812 | + int old_page_state = 0; | |
| 1799 | 1813 | struct anon_vma *anon_vma = NULL; |
| 1800 | 1814 | |
| 1801 | − __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); | |
| 1802 | − migrate_folio_undo_src(folio, page_was_mapped, anon_vma, | |
| 1803 | − true, ret_folios); | |
| 1815 | + __migrate_folio_extract(dst, &old_page_state, &anon_vma); | |
| 1816 | + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, | |
| 1817 | + anon_vma, true, ret_folios); | |
| 1804 | 1818 | list_del(&dst->lru); |
| 1805 | 1819 | migrate_folio_undo_dst(dst, true, put_new_folio, private); |
| 1806 | 1820 | dst = dst2; |
| @@ -1828,6 +1842,7 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio, · 21 unmodified lines | ||
| 1828 | 1842 | stats->nr_succeeded += astats.nr_succeeded; |
| 1829 | 1843 | stats->nr_thp_succeeded += astats.nr_thp_succeeded; |
| 1830 | 1844 | stats->nr_thp_split += astats.nr_thp_split; |
| 1845 | + stats->nr_split += astats.nr_split; | |
| 1831 | 1846 | if (rc < 0) { |
| 1832 | 1847 | stats->nr_failed_pages += astats.nr_failed_pages; |
| 1833 | 1848 | stats->nr_thp_failed += astats.nr_thp_failed; |
| @@ -1835,7 +1850,11 @@ static int migrate_pages_sync(struct list_head *from, new_folio_t get_new_folio, · 1 unmodified line | ||
| 1835 | 1850 | return rc; |
| 1836 | 1851 | } |
| 1837 | 1852 | stats->nr_thp_failed += astats.nr_thp_split; |
| 1838 | − nr_failed += astats.nr_thp_split; | |
| 1853 | + /* | |
| 1854 | + * Do not count rc, as pages will be retried below. | |
| 1855 | + * Count nr_split only, since it includes nr_thp_split. | |
| 1856 | + */ | |
| 1857 | + nr_failed += astats.nr_split; | |
| 1839 | 1858 | /* |
| 1840 | 1859 | * Fall back to migrate all failed folios one by one synchronously. All |
| 1841 | 1860 | * failed folios except split THPs will be retried, so their failure |
| @@ -1970,7 +1989,8 @@ out: · 128 unmodified lines | ||
| 1970 | 1989 | count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split); |
| 1971 | 1990 | trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages, |
| 1972 | 1991 | stats.nr_thp_succeeded, stats.nr_thp_failed, |
| 1973 | − stats.nr_thp_split, mode, reason); | |
| 1992 | + stats.nr_thp_split, stats.nr_split, mode, | |
| 1993 | + reason); | |
| 1974 | 1994 | |
| 1975 | 1995 | if (ret_succeeded) |
| 1976 | 1996 | *ret_succeeded = stats.nr_succeeded; |
| @@ -2029,8 +2049,7 @@ static int store_status(int __user *status, int start, int value, int nr) · 52 unmodified lines | ||
| 2029 | 2049 | return 0; |
| 2030 | 2050 | } |
| 2031 | 2051 | |
| 2032 | −static int do_move_pages_to_node(struct mm_struct *mm, | |
| 2033 | − struct list_head *pagelist, int node) | |
| 2052 | +static int do_move_pages_to_node(struct list_head *pagelist, int node) | |
| 2034 | 2053 | { |
| 2035 | 2054 | int err; |
| 2036 | 2055 | struct migration_target_control mtc = { |
| @@ -2060,8 +2079,8 @@ static int add_page_for_migration(struct mm_struct *mm, const void __user *p, · 23 unmodified lines | ||
| 2060 | 2079 | struct vm_area_struct *vma; |
| 2061 | 2080 | unsigned long addr; |
| 2062 | 2081 | struct page *page; |
| 2082 | + struct folio *folio; | |
| 2063 | 2083 | int err; |
| 2064 | − bool isolated; | |
| 2065 | 2084 | |
| 2066 | 2085 | mmap_read_lock(mm); |
| 2067 | 2086 | addr = (unsigned long)untagged_addr_remote(mm, p); |
| @@ -2082,51 +2101,44 @@ static int add_page_for_migration(struct mm_struct *mm, const void __user *p, · 14 unmodified lines | ||
| 2082 | 2101 | if (!page) |
| 2083 | 2102 | goto out; |
| 2084 | 2103 | |
| 2085 | − if (is_zone_device_page(page)) | |
| 2086 | − goto out_putpage; | |
| 2104 | + folio = page_folio(page); | |
| 2105 | + if (folio_is_zone_device(folio)) | |
| 2106 | + goto out_putfolio; | |
| 2087 | 2107 | |
| 2088 | 2108 | err = 0; |
| 2089 | − if (page_to_nid(page) == node) | |
| 2090 | − goto out_putpage; | |
| 2109 | + if (folio_nid(folio) == node) | |
| 2110 | + goto out_putfolio; | |
| 2091 | 2111 | |
| 2092 | 2112 | err = -EACCES; |
| 2093 | 2113 | if (page_mapcount(page) > 1 && !migrate_all) |
| 2094 | − goto out_putpage; | |
| 2114 | + goto out_putfolio; | |
| 2095 | 2115 | |
| 2096 | − if (PageHuge(page)) { | |
| 2097 | − if (PageHead(page)) { | |
| 2098 | − isolated = isolate_hugetlb(page_folio(page), pagelist); | |
| 2099 | − err = isolated ? 1 : -EBUSY; | |
| 2100 | − } | |
| 2116 | + err = -EBUSY; | |
| 2117 | + if (folio_test_hugetlb(folio)) { | |
| 2118 | + if (isolate_hugetlb(folio, pagelist)) | |
| 2119 | + err = 1; | |
| 2101 | 2120 | } else { |
| 2102 | − struct page *head; | |
| 2103 | − | |
| 2104 | − head = compound_head(page); | |
| 2105 | − isolated = isolate_lru_page(head); | |
| 2106 | − if (!isolated) { | |
| 2107 | − err = -EBUSY; | |
| 2108 | − goto out_putpage; | |
| 2109 | − } | |
| 2121 | + if (!folio_isolate_lru(folio)) | |
| 2122 | + goto out_putfolio; | |
| 2110 | 2123 | |
| 2111 | 2124 | err = 1; |
| 2112 | − list_add_tail(&head->lru, pagelist); | |
| 2113 | − mod_node_page_state(page_pgdat(head), | |
| 2114 | − NR_ISOLATED_ANON + page_is_file_lru(head), | |
| 2115 | − thp_nr_pages(head)); | |
| 2125 | + list_add_tail(&folio->lru, pagelist); | |
| 2126 | + node_stat_mod_folio(folio, | |
| 2127 | + NR_ISOLATED_ANON + folio_is_file_lru(folio), | |
| 2128 | + folio_nr_pages(folio)); | |
| 2116 | 2129 | } |
| 2117 | −out_putpage: | |
| 2130 | +out_putfolio: | |
| 2118 | 2131 | /* |
| 2119 | − * Either remove the duplicate refcount from | |
| 2120 | − * isolate_lru_page() or drop the page ref if it was | |
| 2121 | − * not isolated. | |
| 2132 | + * Either remove the duplicate refcount from folio_isolate_lru() | |
| 2133 | + * or drop the folio ref if it was not isolated. | |
| 2122 | 2134 | */ |
| 2123 | − put_page(page); | |
| 2135 | + folio_put(folio); | |
| 2124 | 2136 | out: |
| 2125 | 2137 | mmap_read_unlock(mm); |
| 2126 | 2138 | return err; |
| 2127 | 2139 | } |
| 2128 | 2140 | |
| 2129 | −static int move_pages_and_store_status(struct mm_struct *mm, int node, | |
| 2141 | +static int move_pages_and_store_status(int node, | |
| 2130 | 2142 | struct list_head *pagelist, int __user *status, |
| 2131 | 2143 | int start, int i, unsigned long nr_pages) |
| 2132 | 2144 | { |
| @@ -2135,7 +2147,7 @@ static int move_pages_and_store_status(struct mm_struct *mm, int node, · 2 unmodified lines | ||
| 2135 | 2147 | if (list_empty(pagelist)) |
| 2136 | 2148 | return 0; |
| 2137 | 2149 | |
| 2138 | − err = do_move_pages_to_node(mm, pagelist, node); | |
| 2150 | + err = do_move_pages_to_node(pagelist, node); | |
| 2139 | 2151 | if (err) { |
| 2140 | 2152 | /* |
| 2141 | 2153 | * Positive err means the number of failed |
| @@ -2203,7 +2215,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 61 unmodified lines | ||
| 2203 | 2215 | current_node = node; |
| 2204 | 2216 | start = i; |
| 2205 | 2217 | } else if (node != current_node) { |
| 2206 | − err = move_pages_and_store_status(mm, current_node, | |
| 2218 | + err = move_pages_and_store_status(current_node, | |
| 2207 | 2219 | &pagelist, status, start, i, nr_pages); |
| 2208 | 2220 | if (err) |
| 2209 | 2221 | goto out; |
| @@ -2238,7 +2250,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 28 unmodified lines | ||
| 2238 | 2250 | if (err) |
| 2239 | 2251 | goto out_flush; |
| 2240 | 2252 | |
| 2241 | − err = move_pages_and_store_status(mm, current_node, &pagelist, | |
| 2253 | + err = move_pages_and_store_status(current_node, &pagelist, | |
| 2242 | 2254 | status, start, i, nr_pages); |
| 2243 | 2255 | if (err) { |
| 2244 | 2256 | /* We have accounted for page i */ |
| @@ -2250,7 +2262,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, · 5 unmodified lines | ||
| 2250 | 2262 | } |
| 2251 | 2263 | out_flush: |
| 2252 | 2264 | /* Make sure we do not overwrite the existing error */ |
| 2253 | − err1 = move_pages_and_store_status(mm, current_node, &pagelist, | |
| 2265 | + err1 = move_pages_and_store_status(current_node, &pagelist, | |
| 2254 | 2266 | status, start, i, nr_pages); |
| 2255 | 2267 | if (err >= 0) |
| 2256 | 2268 | err = err1; |
| @@ -2491,16 +2503,9 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src, · 234 unmodified lines | ||
| 2491 | 2503 | return __folio_alloc_node(gfp, order, nid); |
| 2492 | 2504 | } |
| 2493 | 2505 | |
| 2494 | −static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) | |
| 2506 | +static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio) | |
| 2495 | 2507 | { |
| 2496 | − int nr_pages = thp_nr_pages(page); | |
| 2497 | − int order = compound_order(page); | |
| 2498 | − | |
| 2499 | − VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); | |
| 2500 | − | |
| 2501 | − /* Do not migrate THP mapped by multiple processes */ | |
| 2502 | − if (PageTransHuge(page) && total_mapcount(page) > 1) | |
| 2503 | − return 0; | |
| 2508 | + int nr_pages = folio_nr_pages(folio); | |
| 2504 | 2509 | |
| 2505 | 2510 | /* Avoid migrating to a node that is nearly full */ |
| 2506 | 2511 | if (!migrate_balanced_pgdat(pgdat, nr_pages)) { |
| @@ -2512,75 +2517,79 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) · 5 unmodified lines | ||
| 2512 | 2517 | if (managed_zone(pgdat->node_zones + z)) |
| 2513 | 2518 | break; |
| 2514 | 2519 | } |
| 2515 | − wakeup_kswapd(pgdat->node_zones + z, 0, order, ZONE_MOVABLE); | |
| 2520 | + wakeup_kswapd(pgdat->node_zones + z, 0, | |
| 2521 | + folio_order(folio), ZONE_MOVABLE); | |
| 2516 | 2522 | return 0; |
| 2517 | 2523 | } |
| 2518 | 2524 | |
| 2519 | − if (!isolate_lru_page(page)) | |
| 2525 | + if (!folio_isolate_lru(folio)) | |
| 2520 | 2526 | return 0; |
| 2521 | 2527 | |
| 2522 | − mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + page_is_file_lru(page), | |
| 2528 | + node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio), | |
| 2523 | 2529 | nr_pages); |
| 2524 | 2530 | |
| 2525 | 2531 | /* |
| 2526 | − * Isolating the page has taken another reference, so the | |
| 2527 | − * caller's reference can be safely dropped without the page | |
| 2532 | + * Isolating the folio has taken another reference, so the | |
| 2533 | + * caller's reference can be safely dropped without the folio | |
| 2528 | 2534 | * disappearing underneath us during migration. |
| 2529 | 2535 | */ |
| 2530 | − put_page(page); | |
| 2536 | + folio_put(folio); | |
| 2531 | 2537 | return 1; |
| 2532 | 2538 | } |
| 2533 | 2539 | |
| 2534 | 2540 | /* |
| 2535 | − * Attempt to migrate a misplaced page to the specified destination | |
| 2541 | + * Attempt to migrate a misplaced folio to the specified destination | |
| 2536 | 2542 | * node. Caller is expected to have an elevated reference count on |
| 2537 | − * the page that will be dropped by this function before returning. | |
| 2543 | + * the folio that will be dropped by this function before returning. | |
| 2538 | 2544 | */ |
| 2539 | −int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, | |
| 2540 | − int node) | |
| 2545 | +int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma, | |
| 2546 | + int node) | |
| 2541 | 2547 | { |
| 2542 | 2548 | pg_data_t *pgdat = NODE_DATA(node); |
| 2543 | 2549 | int isolated; |
| 2544 | 2550 | int nr_remaining; |
| 2545 | 2551 | unsigned int nr_succeeded; |
| 2546 | 2552 | LIST_HEAD(migratepages); |
| 2547 | − int nr_pages = thp_nr_pages(page); | |
| 2553 | + int nr_pages = folio_nr_pages(folio); | |
| 2548 | 2554 | |
| 2549 | 2555 | /* |
| 2550 | − * Don't migrate file pages that are mapped in multiple processes | |
| 2556 | + * Don't migrate file folios that are mapped in multiple processes | |
| 2551 | 2557 | * with execute permissions as they are probably shared libraries. |
| 2558 | + * To check if the folio is shared, ideally we want to make sure | |
| 2559 | + * every page is mapped to the same process. Doing that is very | |
| 2560 | + * expensive, so check the estimated mapcount of the folio instead. | |
| 2552 | 2561 | */ |
| 2553 | − if (page_mapcount(page) != 1 && page_is_file_lru(page) && | |
| 2562 | + if (folio_estimated_sharers(folio) != 1 && folio_is_file_lru(folio) && | |
| 2554 | 2563 | (vma->vm_flags & VM_EXEC)) |
| 2555 | 2564 | goto out; |
| 2556 | 2565 | |
| 2557 | 2566 | /* |
| 2558 | − * Also do not migrate dirty pages as not all filesystems can move | |
| 2559 | − * dirty pages in MIGRATE_ASYNC mode which is a waste of cycles. | |
| 2567 | + * Also do not migrate dirty folios as not all filesystems can move | |
| 2568 | + * dirty folios in MIGRATE_ASYNC mode which is a waste of cycles. | |
| 2560 | 2569 | */ |
| 2561 | − if (page_is_file_lru(page) && PageDirty(page)) | |
| 2570 | + if (folio_is_file_lru(folio) && folio_test_dirty(folio)) | |
| 2562 | 2571 | goto out; |
| 2563 | 2572 | |
| 2564 | − isolated = numamigrate_isolate_page(pgdat, page); | |
| 2573 | + isolated = numamigrate_isolate_folio(pgdat, folio); | |
| 2565 | 2574 | if (!isolated) |
| 2566 | 2575 | goto out; |
| 2567 | 2576 | |
| 2568 | − list_add(&page->lru, &migratepages); | |
| 2577 | + list_add(&folio->lru, &migratepages); | |
| 2569 | 2578 | nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio, |
| 2570 | 2579 | NULL, node, MIGRATE_ASYNC, |
| 2571 | 2580 | MR_NUMA_MISPLACED, &nr_succeeded); |
| 2572 | 2581 | if (nr_remaining) { |
| 2573 | 2582 | if (!list_empty(&migratepages)) { |
| 2574 | − list_del(&page->lru); | |
| 2575 | − mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + | |
| 2576 | − page_is_file_lru(page), -nr_pages); | |
| 2577 | − putback_lru_page(page); | |
| 2583 | + list_del(&folio->lru); | |
| 2584 | + node_stat_mod_folio(folio, NR_ISOLATED_ANON + | |
| 2585 | + folio_is_file_lru(folio), -nr_pages); | |
| 2586 | + folio_putback_lru(folio); | |
| 2578 | 2587 | } |
| 2579 | 2588 | isolated = 0; |
| 2580 | 2589 | } |
| 2581 | 2590 | if (nr_succeeded) { |
| 2582 | 2591 | count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded); |
| 2583 | − if (!node_is_toptier(page_to_nid(page)) && node_is_toptier(node)) | |
| 2592 | + if (!node_is_toptier(folio_nid(folio)) && node_is_toptier(node)) | |
| 2584 | 2593 | mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, |
| 2585 | 2594 | nr_succeeded); |
| 2586 | 2595 | } |
| @@ -2588,7 +2597,7 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, · 1 unmodified line | ||
| 2588 | 2597 | return isolated; |
| 2589 | 2598 | |
| 2590 | 2599 | out: |
| 2591 | − put_page(page); | |
| 2600 | + folio_put(folio); | |
| 2592 | 2601 | return 0; |
| 2593 | 2602 | } |
| 2594 | 2603 | #endif /* CONFIG_NUMA_BALANCING */ |
| @@ -305,6 +305,62 @@ void munlock_folio(struct folio *folio) · 304 unmodified lines | ||
| 305 | 305 | local_unlock(&mlock_fbatch.lock); |
| 306 | 306 | } |
| 307 | 307 | |
| 308 | +static inline unsigned int folio_mlock_step(struct folio *folio, | |
| 309 | + pte_t *pte, unsigned long addr, unsigned long end) | |
| 310 | +{ | |
| 311 | + unsigned int count, i, nr = folio_nr_pages(folio); | |
| 312 | + unsigned long pfn = folio_pfn(folio); | |
| 313 | + pte_t ptent = ptep_get(pte); | |
| 314 | + | |
| 315 | + if (!folio_test_large(folio)) | |
| 316 | + return 1; | |
| 317 | + | |
| 318 | + count = pfn + nr - pte_pfn(ptent); | |
| 319 | + count = min_t(unsigned int, count, (end - addr) >> PAGE_SHIFT); | |
| 320 | + | |
| 321 | + for (i = 0; i < count; i++, pte++) { | |
| 322 | + pte_t entry = ptep_get(pte); | |
| 323 | + | |
| 324 | + if (!pte_present(entry)) | |
| 325 | + break; | |
| 326 | + if (pte_pfn(entry) - pfn >= nr) | |
| 327 | + break; | |
| 328 | + } | |
| 329 | + | |
| 330 | + return i; | |
| 331 | +} | |
| 332 | + | |
| 333 | +static inline bool allow_mlock_munlock(struct folio *folio, | |
| 334 | + struct vm_area_struct *vma, unsigned long start, | |
| 335 | + unsigned long end, unsigned int step) | |
| 336 | +{ | |
| 337 | + /* | |
| 338 | + * For unlock, allow munlock large folio which is partially | |
| 339 | + * mapped to VMA. As it's possible that large folio is | |
| 340 | + * mlocked and VMA is split later. | |
| 341 | + * | |
| 342 | + * During memory pressure, such kind of large folio can | |
| 343 | + * be split. And the pages are not in VM_LOCKed VMA | |
| 344 | + * can be reclaimed. | |
| 345 | + */ | |
101 more lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 273 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 130 lines aren’t shown here.
This diff is large: its 164 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 593 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 290 lines aren’t shown here.
This diff is large: its 1055 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 810 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 222 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 868 lines aren’t shown here.
This diff is large: its 112 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 398 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 235 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 219 lines aren’t shown here.
This diff is large: its 437 lines aren’t shown here.
This diff is large: its 1661 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.