History

mm: support madvise(MADV_FREE)

Linux doesn't have an ability to free pages lazy while other OS already have been supported that named by madvise(MADV_FREE). The gain is clear that kernel can discard freed pages rather than swapping out or OOM if memory pressure happens. Without memory pressure, freed pages would be reused by userspace without another additional overhead(ex, page fault + allocation + zeroing). Jason Evans said: : Facebook has been using MAP_UNINITIALIZED : (https://lkml.org/lkml/2012/1/18/308) in some of its applications for : several years, but there are operational costs to maintaining this : out-of-tree in our kernel and in jemalloc, and we are anxious to retire it : in favor of MADV_FREE. When we first enabled MAP_UNINITIALIZED it : increased throughput for much of our workload by ~5%, and although the : benefit has decreased using newer hardware and kernels, there is still : enough benefit that we cannot reasonably retire it without a replacement. : : Aside from Facebook operations, there are numerous broadly used : applications that would benefit from MADV_FREE. The ones that immediately : come to mind are redis, varnish, and MariaDB. I don't have much insight : into Android internals and development process, but I would hope to see : MADV_FREE support eventually end up there as well to benefit applications : linked with the integrated jemalloc. : : jemalloc will use MADV_FREE once it becomes available in the Linux kernel. : In fact, jemalloc already uses MADV_FREE or equivalent everywhere it's : available: *BSD, OS X, Windows, and Solaris -- every platform except Linux : (and AIX, but I'm not sure it even compiles on AIX). The lack of : MADV_FREE on Linux forced me down a long series of increasingly : sophisticated heuristics for madvise() volume reduction, and even so this : remains a common performance issue for people using jemalloc on Linux. : Please integrate MADV_FREE; many people will benefit substantially. How it works: When madvise syscall is called, VM clears dirty bit of ptes of the range. If memory pressure happens, VM checks dirty bit of page table and if it found still "clean", it means it's a "lazyfree pages" so VM could discard the page instead of swapping out. Once there was store operation for the page before VM peek a page to reclaim, dirty bit is set so VM can swap out the page instead of discarding. One thing we should notice is that basically, MADV_FREE relies on dirty bit in page table entry to decide whether VM allows to discard the page or not. IOW, if page table entry includes marked dirty bit, VM shouldn't discard the page. However, as a example, if swap-in by read fault happens, page table entry doesn't have dirty bit so MADV_FREE could discard the page wrongly. For avoiding the problem, MADV_FREE did more checks with PageDirty and PageSwapCache. It worked out because swapped-in page lives on swap cache and since it is evicted from the swap cache, the page has PG_dirty flag. So both page flags check effectively prevent wrong discarding by MADV_FREE. However, a problem in above logic is that swapped-in page has PG_dirty still after they are removed from swap cache so VM cannot consider the page as freeable any more even if madvise_free is called in future. Look at below example for detail. ptr = malloc(); memset(ptr); .. .. .. heavy memory pressure so all of pages are swapped out .. .. var = *ptr; -> a page swapped-in and could be removed from swapcache. Then, page table doesn't mark dirty bit and page descriptor includes PG_dirty .. .. madvise_free(ptr); -> It doesn't clear PG_dirty of the page. .. .. .. .. heavy memory pressure again. .. In this time, VM cannot discard the page because the page .. has *PG_dirty* To solve the problem, this patch clears PG_dirty if only the page is owned exclusively by current process when madvise is called because PG_dirty represents ptes's dirtiness in several processes so we could clear it only if we own it exclusively. Firstly, heavy users would be general allocators(ex, jemalloc, tcmalloc and hope glibc supports it) and jemalloc/tcmalloc already have supported the feature for other OS(ex, FreeBSD) barrios@blaptop:~/benchmark/ebizzy$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 12 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 2 Stepping: 3 CPU MHz: 3200.185 BogoMIPS: 6400.53 Virtualization: VT-x Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K NUMA node0 CPU(s): 0-11 ebizzy benchmark(./ebizzy -S 10 -n 512) Higher avg is better. vanilla-jemalloc MADV_free-jemalloc 1 thread records: 10 records: 10 avg: 2961.90 avg: 12069.70 std: 71.96(2.43%) std: 186.68(1.55%) max: 3070.00 max: 12385.00 min: 2796.00 min: 11746.00 2 thread records: 10 records: 10 avg: 5020.00 avg: 17827.00 std: 264.87(5.28%) std: 358.52(2.01%) max: 5244.00 max: 18760.00 min: 4251.00 min: 17382.00 4 thread records: 10 records: 10 avg: 8988.80 avg: 27930.80 std: 1175.33(13.08%) std: 3317.33(11.88%) max: 9508.00 max: 30879.00 min: 5477.00 min: 21024.00 8 thread records: 10 records: 10 avg: 13036.50 avg: 33739.40 std: 170.67(1.31%) std: 5146.22(15.25%) max: 13371.00 max: 40572.00 min: 12785.00 min: 24088.00 16 thread records: 10 records: 10 avg: 11092.40 avg: 31424.20 std: 710.60(6.41%) std: 3763.89(11.98%) max: 12446.00 max: 36635.00 min: 9949.00 min: 25669.00 32 thread records: 10 records: 10 avg: 11067.00 avg: 34495.80 std: 971.06(8.77%) std: 2721.36(7.89%) max: 12010.00 max: 38598.00 min: 9002.00 min: 30636.00 In summary, MADV_FREE is about much faster than MADV_DONTNEED. This patch (of 12): Add core MADV_FREE implementation. [akpm@linux-foundation.org: small cleanups] Signed-off-by: Minchan Kim <minchan@kernel.org> Acked-by: Michal Hocko <mhocko@suse.com> Acked-by: Hugh Dickins <hughd@google.com> Cc: Mika Penttil <mika.penttila@nextfour.com> Cc: Michael Kerrisk <mtk.manpages@gmail.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Rik van Riel <riel@redhat.com> Cc: Mel Gorman <mgorman@suse.de> Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> Cc: Jason Evans <je@fb.com> Cc: Daniel Micay <danielmicay@gmail.com> Cc: "Kirill A. Shutemov" <kirill@shutemov.name> Cc: Shaohua Li <shli@kernel.org> Cc: <yalin.wang2010@gmail.com> Cc: Andy Lutomirski <luto@amacapital.net> Cc: "James E.J. Bottomley" <jejb@parisc-linux.org> Cc: "Kirill A. Shutemov" <kirill@shutemov.name> Cc: "Shaohua Li" <shli@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Chen Gang <gang.chen.5i5j@gmail.com> Cc: Chris Zankel <chris@zankel.net> Cc: Darrick J. Wong <darrick.wong@oracle.com> Cc: David S. Miller <davem@davemloft.net> Cc: Helge Deller <deller@gmx.de> Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru> Cc: Matt Turner <mattst88@gmail.com> Cc: Max Filippov <jcmvbkbc@gmail.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Richard Henderson <rth@twiddle.net> Cc: Roland Dreier <roland@kernel.org> Cc: Russell King <rmk@arm.linux.org.uk> Cc: Shaohua Li <shli@kernel.org> Cc: Will Deacon <will.deacon@arm.com> Cc: Wu Fengguang <fengguang.wu@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Minchan Kim committed 10 years ago

Commit 854e9ed1 parent 17ec4cd8 files · +221 −9Browse files

Unified · Split

8 files
  1. include/linux/rmap.h +2 −0
  2. include/linux/vm_event_item.h +1 −0
  3. include/uapi/asm-generic/mman-common.h +1 −0
  4. mm/madvise.c +170 −0
  5. mm/rmap.c +32 −4
  6. mm/swap_state.c +2 −3
  7. mm/vmscan.c +12 −2
  8. mm/vmstat.c +1 −0
include/linux/rmap.h+2−0
@@ -85,6 +85,7 @@ enum ttu_flags { · 84 unmodified lines
8585 TTU_UNMAP = 1, /* unmap mode */
8686 TTU_MIGRATION = 2, /* migration mode */
8787 TTU_MUNLOCK = 4, /* munlock mode */
88+ TTU_LZFREE = 8, /* lazy free mode */
8889
8990 TTU_IGNORE_MLOCK = (1 << 8), /* ignore mlock */
9091 TTU_IGNORE_ACCESS = (1 << 9), /* don't age */
@@ -311,5 +312,6 @@ static inline int page_mkclean(struct page *page) · 220 unmodified lines
311312 #define SWAP_AGAIN 1
312313 #define SWAP_FAIL 2
313314 #define SWAP_MLOCK 3
315+#define SWAP_LZFREE 4
314316
315317 #endif /* _LINUX_RMAP_H */
include/linux/vm_event_item.h+1−0
@@ -25,6 +25,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, · 24 unmodified lines
2525 FOR_ALL_ZONES(PGALLOC),
2626 PGFREE, PGACTIVATE, PGDEACTIVATE,
2727 PGFAULT, PGMAJFAULT,
28+ PGLAZYFREED,
2829 FOR_ALL_ZONES(PGREFILL),
2930 FOR_ALL_ZONES(PGSTEAL_KSWAPD),
3031 FOR_ALL_ZONES(PGSTEAL_DIRECT),
include/uapi/asm-generic/mman-common.h+1−0
@@ -39,6 +39,7 @@ · 38 unmodified lines
3939 #define MADV_SEQUENTIAL 2 /* expect sequential page references */
4040 #define MADV_WILLNEED 3 /* will need these pages */
4141 #define MADV_DONTNEED 4 /* don't need these pages */
42+#define MADV_FREE 5 /* free pages only if memory pressure */
4243
4344 /* common parameters: try to keep these consistent across architectures */
4445 #define MADV_REMOVE 9 /* remove these pages & resources */
mm/madvise.c+170−0
@@ -20,6 +20,9 @@ · 19 unmodified lines
2020 #include <linux/backing-dev.h>
2121 #include <linux/swap.h>
2222 #include <linux/swapops.h>
23+#include <linux/mmu_notifier.h>
24+
25+#include <asm/tlb.h>
2326
2427 /*
2528 * Any behaviour which results in changes to the vma->vm_flags needs to
@@ -32,6 +35,7 @@ static int madvise_need_mmap_write(int behavior) · 6 unmodified lines
3235 case MADV_REMOVE:
3336 case MADV_WILLNEED:
3437 case MADV_DONTNEED:
38+ case MADV_FREE:
3539 return 0;
3640 default:
3741 /* be safe, default to 1. list exceptions explicitly */
@@ -256,6 +260,163 @@ static long madvise_willneed(struct vm_area_struct *vma, · 218 unmodified lines
256260 return 0;
257261 }
258262
263+static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
264+ unsigned long end, struct mm_walk *walk)
265+
266+{
267+ struct mmu_gather *tlb = walk->private;
268+ struct mm_struct *mm = tlb->mm;
269+ struct vm_area_struct *vma = walk->vma;
270+ spinlock_t *ptl;
271+ pte_t *orig_pte, *pte, ptent;
272+ struct page *page;
273+
274+ split_huge_pmd(vma, pmd, addr);
275+ if (pmd_trans_unstable(pmd))
276+ return 0;
277+
278+ orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
279+ arch_enter_lazy_mmu_mode();
280+ for (; addr != end; pte++, addr += PAGE_SIZE) {
281+ ptent = *pte;
282+
283+ if (!pte_present(ptent))
284+ continue;
285+
286+ page = vm_normal_page(vma, addr, ptent);
287+ if (!page)
288+ continue;
289+
290+ /*
291+ * If pmd isn't transhuge but the page is THP and
292+ * is owned by only this process, split it and
293+ * deactivate all pages.
294+ */
295+ if (PageTransCompound(page)) {
296+ if (page_mapcount(page) != 1)
297+ goto out;
298+ get_page(page);
299+ if (!trylock_page(page)) {
300+ put_page(page);
301+ goto out;
302+ }
303+ pte_unmap_unlock(orig_pte, ptl);
304+ if (split_huge_page(page)) {
305+ unlock_page(page);
306+ put_page(page);
307+ pte_offset_map_lock(mm, pmd, addr, &ptl);
308+ goto out;
309+ }
310+ put_page(page);
311+ unlock_page(page);
312+ pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
313+ pte--;
314+ addr -= PAGE_SIZE;
315+ continue;
316+ }
317+
318+ VM_BUG_ON_PAGE(PageTransCompound(page), page);
319+
320+ if (PageSwapCache(page) || PageDirty(page)) {
321+ if (!trylock_page(page))
322+ continue;
323+ /*
324+ * If page is shared with others, we couldn't clear
325+ * PG_dirty of the page.
326+ */
327+ if (page_mapcount(page) != 1) {
328+ unlock_page(page);
329+ continue;
330+ }
331+
332+ if (PageSwapCache(page) && !try_to_free_swap(page)) {
333+ unlock_page(page);
334+ continue;
335+ }
336+
337+ ClearPageDirty(page);
338+ unlock_page(page);
339+ }
340+
341+ if (pte_young(ptent) || pte_dirty(ptent)) {
342+ /*
343+ * Some of architecture(ex, PPC) don't update TLB
344+ * with set_pte_at and tlb_remove_tlb_entry so for
345+ * the portability, remap the pte with old|clean
346+ * after pte clearing.
347+ */
348+ ptent = ptep_get_and_clear_full(mm, addr, pte,
349+ tlb->fullmm);
350+
351+ ptent = pte_mkold(ptent);
352+ ptent = pte_mkclean(ptent);
353+ set_pte_at(mm, addr, pte, ptent);
354+ tlb_remove_tlb_entry(tlb, pte, addr);
355+ }
356+ }
357+out:
358+ arch_leave_lazy_mmu_mode();
359+ pte_unmap_unlock(orig_pte, ptl);
360+ cond_resched();
361+ return 0;
362+}
363+
364+static void madvise_free_page_range(struct mmu_gather *tlb,
365+ struct vm_area_struct *vma,
366+ unsigned long addr, unsigned long end)
367+{
368+ struct mm_walk free_walk = {
369+ .pmd_entry = madvise_free_pte_range,
370+ .mm = vma->vm_mm,
371+ .private = tlb,
372+ };
373+
374+ tlb_start_vma(tlb, vma);
375+ walk_page_range(addr, end, &free_walk);
376+ tlb_end_vma(tlb, vma);
377+}
378+
379+static int madvise_free_single_vma(struct vm_area_struct *vma,
380+ unsigned long start_addr, unsigned long end_addr)
381+{
382+ unsigned long start, end;
383+ struct mm_struct *mm = vma->vm_mm;
384+ struct mmu_gather tlb;
385+
386+ if (vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP))
387+ return -EINVAL;
388+
389+ /* MADV_FREE works for only anon vma at the moment */
390+ if (!vma_is_anonymous(vma))
391+ return -EINVAL;
392+
393+ start = max(vma->vm_start, start_addr);
394+ if (start >= vma->vm_end)
395+ return -EINVAL;
396+ end = min(vma->vm_end, end_addr);
397+ if (end <= vma->vm_start)
398+ return -EINVAL;
399+
400+ lru_add_drain();
401+ tlb_gather_mmu(&tlb, mm, start, end);
402+ update_hiwater_rss(mm);
403+
404+ mmu_notifier_invalidate_range_start(mm, start, end);
405+ madvise_free_page_range(&tlb, vma, start, end);
406+ mmu_notifier_invalidate_range_end(mm, start, end);
407+ tlb_finish_mmu(&tlb, start, end);
408+
409+ return 0;
410+}
411+
412+static long madvise_free(struct vm_area_struct *vma,
413+ struct vm_area_struct **prev,
414+ unsigned long start, unsigned long end)
415+{
416+ *prev = vma;
417+ return madvise_free_single_vma(vma, start, end);
418+}
419+
259420 /*
260421 * Application no longer needs these pages. If the pages are dirty,
261422 * it's OK to just throw them away. The app will be more careful about
@@ -379,6 +540,14 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, · 117 unmodified lines
379540 return madvise_remove(vma, prev, start, end);
380541 case MADV_WILLNEED:
381542 return madvise_willneed(vma, prev, start, end);
543+ case MADV_FREE:
544+ /*
545+ * XXX: In this implementation, MADV_FREE works like
546+ * MADV_DONTNEED on swapless system or full swap.
547+ */
548+ if (get_nr_swap_pages() > 0)
549+ return madvise_free(vma, prev, start, end);
550+ /* passthrough */
382551 case MADV_DONTNEED:
383552 return madvise_dontneed(vma, prev, start, end);
384553 default:
@@ -398,6 +567,7 @@ madvise_behavior_valid(int behavior) · 13 unmodified lines
398567 case MADV_REMOVE:
399568 case MADV_WILLNEED:
400569 case MADV_DONTNEED:
570+ case MADV_FREE:
401571 #ifdef CONFIG_KSM
402572 case MADV_MERGEABLE:
403573 case MADV_UNMERGEABLE:
mm/rmap.c+32−4
@@ -1411,6 +1411,11 @@ void page_remove_rmap(struct page *page, bool compound) · 1410 unmodified lines
14111411 */
14121412 }
14131413
1414+struct rmap_private {
1415+ enum ttu_flags flags;
1416+ int lazyfreed;
1417+};
1418+
14141419 /*
14151420 * @arg: enum ttu_flags will be passed to this argument
14161421 */
@@ -1422,7 +1427,8 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma, · 5 unmodified lines
14221427 pte_t pteval;
14231428 spinlock_t *ptl;
14241429 int ret = SWAP_AGAIN;
1425− enum ttu_flags flags = (enum ttu_flags)arg;
1430+ struct rmap_private *rp = arg;
1431+ enum ttu_flags flags = rp->flags;
14261432
14271433 /* munlock has nothing to gain from examining un-locked vmas */
14281434 if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED))
@@ -1514,6 +1520,14 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma, · 85 unmodified lines
15141520 * See handle_pte_fault() ...
15151521 */
15161522 VM_BUG_ON_PAGE(!PageSwapCache(page), page);
1523+
1524+ if (!PageDirty(page) && (flags & TTU_LZFREE)) {
1525+ /* It's a freeable page by MADV_FREE */
1526+ dec_mm_counter(mm, MM_ANONPAGES);
1527+ rp->lazyfreed++;
1528+ goto discard;
1529+ }
1530+
15171531 if (swap_duplicate(entry) < 0) {
15181532 set_pte_at(mm, address, pte, pteval);
15191533 ret = SWAP_FAIL;
@@ -1534,6 +1548,7 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma, · 14 unmodified lines
15341548 } else
15351549 dec_mm_counter(mm, mm_counter_file(page));
15361550
1551+discard:
15371552 page_remove_rmap(page, PageHuge(page));
15381553 page_cache_release(page);
15391554
@@ -1586,9 +1601,14 @@ static int page_not_mapped(struct page *page) · 46 unmodified lines
15861601 int try_to_unmap(struct page *page, enum ttu_flags flags)
15871602 {
15881603 int ret;
1604+ struct rmap_private rp = {
1605+ .flags = flags,
1606+ .lazyfreed = 0,
1607+ };
1608+
15891609 struct rmap_walk_control rwc = {
15901610 .rmap_one = try_to_unmap_one,
1591− .arg = (void *)flags,
1611+ .arg = &rp,
15921612 .done = page_not_mapped,
15931613 .anon_lock = page_lock_anon_vma_read,
15941614 };
@@ -1608,8 +1628,11 @@ int try_to_unmap(struct page *page, enum ttu_flags flags) · 13 unmodified lines
16081628
16091629 ret = rmap_walk(page, &rwc);
16101630
1611− if (ret != SWAP_MLOCK && !page_mapped(page))
1631+ if (ret != SWAP_MLOCK && !page_mapped(page)) {
16121632 ret = SWAP_SUCCESS;
1633+ if (rp.lazyfreed && !PageDirty(page))
1634+ ret = SWAP_LZFREE;
1635+ }
16131636 return ret;
16141637 }
16151638
@@ -1631,9 +1654,14 @@ int try_to_unmap(struct page *page, enum ttu_flags flags) · 15 unmodified lines
16311654 int try_to_munlock(struct page *page)
16321655 {
16331656 int ret;
1657+ struct rmap_private rp = {
1658+ .flags = TTU_MUNLOCK,
1659+ .lazyfreed = 0,
1660+ };
1661+
16341662 struct rmap_walk_control rwc = {
16351663 .rmap_one = try_to_unmap_one,
1636− .arg = (void *)TTU_MUNLOCK,
1664+ .arg = &rp,
16371665 .done = page_not_mapped,
16381666 .anon_lock = page_lock_anon_vma_read,
16391667
mm/swap_state.c+2−3
@@ -185,13 +185,12 @@ int add_to_swap(struct page *page, struct list_head *list) · 184 unmodified lines
185185 * deadlock in the swap out path.
186186 */
187187 /*
188− * Add it to the swap cache and mark it dirty
188+ * Add it to the swap cache.
189189 */
190190 err = add_to_swap_cache(page, entry,
191191 __GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN);
192192
193− if (!err) { /* Success */
194− SetPageDirty(page);
193+ if (!err) {
195194 return 1;
196195 } else { /* -ENOMEM radix-tree allocation failure */
197196 /*
mm/vmscan.c+12−2
@@ -906,6 +906,8 @@ static unsigned long shrink_page_list(struct list_head *page_list, · 905 unmodified lines
906906 int may_enter_fs;
907907 enum page_references references = PAGEREF_RECLAIM_CLEAN;
908908 bool dirty, writeback;
909+ bool lazyfree = false;
910+ int ret = SWAP_SUCCESS;
909911
910912 cond_resched();
911913
@@ -1049,6 +1051,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, · 137 unmodified lines
10491051 goto keep_locked;
10501052 if (!add_to_swap(page, page_list))
10511053 goto activate_locked;
1054+ lazyfree = true;
10521055 may_enter_fs = 1;
10531056
10541057 /* Adding to swap updated mapping */
@@ -1060,14 +1063,17 @@ static unsigned long shrink_page_list(struct list_head *page_list, · 5 unmodified lines
10601063 * processes. Try to unmap it here.
10611064 */
10621065 if (page_mapped(page) && mapping) {
1063− switch (try_to_unmap(page,
1064− ttu_flags|TTU_BATCH_FLUSH)) {
1066+ switch (ret = try_to_unmap(page, lazyfree ?
1067+ (ttu_flags | TTU_BATCH_FLUSH | TTU_LZFREE) :
1068+ (ttu_flags | TTU_BATCH_FLUSH))) {
10651069 case SWAP_FAIL:
10661070 goto activate_locked;
10671071 case SWAP_AGAIN:
10681072 goto keep_locked;
10691073 case SWAP_MLOCK:
10701074 goto cull_mlocked;
1075+ case SWAP_LZFREE:
1076+ goto lazyfree;
10711077 case SWAP_SUCCESS:
10721078 ; /* try to free the page below */
10731079 }
@@ -1174,6 +1180,7 @@ static unsigned long shrink_page_list(struct list_head *page_list, · 100 unmodified lines
11741180 }
11751181 }
11761182
1183+lazyfree:
11771184 if (!mapping || !__remove_mapping(mapping, page, true))
11781185 goto keep_locked;
11791186
@@ -1186,6 +1193,9 @@ static unsigned long shrink_page_list(struct list_head *page_list, · 6 unmodified lines
11861193 */
11871194 __ClearPageLocked(page);
11881195 free_it:
1196+ if (ret == SWAP_LZFREE)
1197+ count_vm_event(PGLAZYFREED);
1198+
11891199 nr_reclaimed++;
11901200
11911201 /*
mm/vmstat.c+1−0
@@ -783,6 +783,7 @@ const char * const vmstat_text[] = { · 782 unmodified lines
783783
784784 "pgfault",
785785 "pgmajfault",
786+ "pglazyfreed",
786787
787788 TEXTS_FOR_ZONES("pgrefill")
788789 TEXTS_FOR_ZONES("pgsteal_kswapd")