History

workingset: refactor LRU refault to expose refault recency check

Patch series "cachestat: a new syscall for page cache state of files", v13. There is currently no good way to query the page cache statistics of large files and directory trees. There is mincore(), but it scales poorly: the kernel writes out a lot of bitmap data that userspace has to aggregate, when the user really does not care about per-page information in that case. The user also needs to mmap and unmap each file as it goes along, which can be quite slow as well. Some use cases where this information could come in handy: * Allowing database to decide whether to perform an index scan or direct table queries based on the in-memory cache state of the index. * Visibility into the writeback algorithm, for performance issues diagnostic. * Workload-aware writeback pacing: estimating IO fulfilled by page cache (and IO to be done) within a range of a file, allowing for more frequent syncing when and where there is IO capacity, and batching when there is not. * Computing memory usage of large files/directory trees, analogous to the du tool for disk usage. More information about these use cases could be found in this thread: https://lore.kernel.org/lkml/20230315170934.GA97793@cmpxchg.org/ This series of patches introduces a new system call, cachestat, that summarizes the page cache statistics (number of cached pages, dirty pages, pages marked for writeback, evicted pages etc.) of a file, in a specified range of bytes. It also include a selftest suite that tests some typical usage. Currently, the syscall is only wired in for x86 architecture. This interface is inspired by past discussion and concerns with fincore, which has a similar design (and as a result, issues) as mincore. Relevant links: https://lkml.indiana.edu/hypermail/linux/kernel/1302.1/04207.html https://lkml.indiana.edu/hypermail/linux/kernel/1302.1/04209.html I have also developed a small tool that computes the memory usage of files and directories, analogous to the du utility. User can choose between mincore or cachestat (with cachestat exporting more information than mincore). To compare the performance of these two options, I benchmarked the tool on the root directory of a Meta's server machine, each for five runs: Using cachestat real -- Median: 33.377s, Average: 33.475s, Standard Deviation: 0.3602 user -- Median: 4.08s, Average: 4.1078s, Standard Deviation: 0.0742 sys -- Median: 28.823s, Average: 28.8866s, Standard Deviation: 0.2689 Using mincore: real -- Median: 102.352s, Average: 102.3442s, Standard Deviation: 0.2059 user -- Median: 10.149s, Average: 10.1482s, Standard Deviation: 0.0162 sys -- Median: 91.186s, Average: 91.2084s, Standard Deviation: 0.2046 I also ran both syscalls on a 2TB sparse file: Using cachestat: real 0m0.009s user 0m0.000s sys 0m0.009s Using mincore: real 0m37.510s user 0m2.934s sys 0m34.558s Very large files like this are the pathological case for mincore. In fact, to compute the stats for a single 2TB file, mincore takes as long as cachestat takes to compute the stats for the entire tree! This could easily happen inadvertently when we run it on subdirectories. Mincore is clearly not suitable for a general-purpose command line tool. Regarding security concerns, cachestat() should not pose any additional issues. The caller already has read permission to the file itself (since they need an fd to that file to call cachestat). This means that the caller can access the underlying data in its entirety, which is a much greater source of information (and as a result, a much greater security risk) than the cache status itself. The latest API change (in v13 of the patch series) is suggested by Jens Axboe. It allows for 64-bit length argument, even on 32-bit architecture (which is previously not possible due to the limit on the number of syscall arguments). Furthermore, it eliminates the need for compatibility handling - every user can use the same ABI. This patch (of 4): In preparation for computing recently evicted pages in cachestat, refactor workingset_refault and lru_gen_refault to expose a helper function that would test if an evicted page is recently evicted. [penguin-kernel@I-love.SAKURA.ne.jp: add missing rcu_read_unlock() in lru_gen_refault()] Link: https://lkml.kernel.org/r/610781bc-cf11-fc89-a46f-87cb8235d439@I-love.SAKURA.ne.jp Link: https://lkml.kernel.org/r/20230503013608.2431726-1-nphamcs@gmail.com Link: https://lkml.kernel.org/r/20230503013608.2431726-2-nphamcs@gmail.com Signed-off-by: Nhat Pham <nphamcs@gmail.com> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Brian Foster <bfoster@redhat.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Michael Kerrisk <mtk.manpages@gmail.com> Cc: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

Nhat Pham committed 3 years ago

Commit ffcb5f51 parent 18b1d182 files · +103 −48Browse files

Unified · Split

2 files
  1. include/linux/swap.h +1 −0
  2. mm/workingset.c +102 −48
include/linux/swap.h+1−0
@@ -368,6 +368,7 @@ static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry) · 367 unmodified lines
368}368}
369 369
370/* linux/mm/workingset.c */370/* linux/mm/workingset.c */
371bool workingset_test_recent(void *shadow, bool file, bool *workingset);
371void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages);372void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages);
372void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg);373void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg);
373void workingset_refault(struct folio *folio, void *shadow);374void workingset_refault(struct folio *folio, void *shadow);
mm/workingset.c+102−48
@@ -255,6 +255,29 @@ static void *lru_gen_eviction(struct folio *folio) · 254 unmodified lines
255 return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs);255 return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs);
256}256}
257 257
258/*
259 * Tests if the shadow entry is for a folio that was recently evicted.
260 * Fills in @memcgid, @pglist_data, @token, @workingset with the values
261 * unpacked from shadow.
262 */
263static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid,
264 struct pglist_data **pgdat, unsigned long *token, bool *workingset)
265{
266 struct mem_cgroup *eviction_memcg;
267 struct lruvec *lruvec;
268 struct lru_gen_folio *lrugen;
269 unsigned long min_seq;
270
271 unpack_shadow(shadow, memcgid, pgdat, token, workingset);
272 eviction_memcg = mem_cgroup_from_id(*memcgid);
273
274 lruvec = mem_cgroup_lruvec(eviction_memcg, *pgdat);
275 lrugen = &lruvec->lrugen;
276
277 min_seq = READ_ONCE(lrugen->min_seq[file]);
278 return (*token >> LRU_REFS_WIDTH) == (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH));
279}
280
258static void lru_gen_refault(struct folio *folio, void *shadow)281static void lru_gen_refault(struct folio *folio, void *shadow)
259{282{
260 int hist, tier, refs;283 int hist, tier, refs;
@@ -269,23 +292,22 @@ static void lru_gen_refault(struct folio *folio, void *shadow) · 8 unmodified lines
269 int type = folio_is_file_lru(folio);292 int type = folio_is_file_lru(folio);
270 int delta = folio_nr_pages(folio);293 int delta = folio_nr_pages(folio);
271 294
272 unpack_shadow(shadow, &memcg_id, &pgdat, &token, &workingset);
273
274 if (pgdat != folio_pgdat(folio))
275 return;
276
277 rcu_read_lock();295 rcu_read_lock();
278 296
297 if (!lru_gen_test_recent(shadow, type, &memcg_id, &pgdat, &token,
298 &workingset))
299 goto unlock;
300
279 memcg = folio_memcg_rcu(folio);301 memcg = folio_memcg_rcu(folio);
280 if (memcg_id != mem_cgroup_id(memcg))302 if (memcg_id != mem_cgroup_id(memcg))
281 goto unlock;303 goto unlock;
282 304
305 if (pgdat != folio_pgdat(folio))
306 goto unlock;
307
283 lruvec = mem_cgroup_lruvec(memcg, pgdat);308 lruvec = mem_cgroup_lruvec(memcg, pgdat);
284 lrugen = &lruvec->lrugen;309 lrugen = &lruvec->lrugen;
285
286 min_seq = READ_ONCE(lrugen->min_seq[type]);310 min_seq = READ_ONCE(lrugen->min_seq[type]);
287 if ((token >> LRU_REFS_WIDTH) != (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH)))
288 goto unlock;
289 311
290 hist = lru_hist_from_seq(min_seq);312 hist = lru_hist_from_seq(min_seq);
291 /* see the comment in folio_lru_refs() */313 /* see the comment in folio_lru_refs() */
@@ -317,6 +339,12 @@ static void *lru_gen_eviction(struct folio *folio) · 25 unmodified lines
317 return NULL;339 return NULL;
318}340}
319 341
342static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid,
343 struct pglist_data **pgdat, unsigned long *token, bool *workingset)
344{
345 return false;
346}
347
320static void lru_gen_refault(struct folio *folio, void *shadow)348static void lru_gen_refault(struct folio *folio, void *shadow)
321{349{
322}350}
@@ -385,42 +413,34 @@ void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg) · 62 unmodified lines
385}413}
386 414
387/**415/**
388 * workingset_refault - Evaluate the refault of a previously evicted folio.416 * workingset_test_recent - tests if the shadow entry is for a folio that was
389 * @folio: The freshly allocated replacement folio.417 * recently evicted. Also fills in @workingset with the value unpacked from
390 * @shadow: Shadow entry of the evicted folio.418 * shadow.
391 *419 * @shadow: the shadow entry to be tested.
392 * Calculates and evaluates the refault distance of the previously420 * @file: whether the corresponding folio is from the file lru.
393 * evicted folio in the context of the node and the memcg whose memory421 * @workingset: where the workingset value unpacked from shadow should
394 * pressure caused the eviction.422 * be stored.
423 *
424 * Return: true if the shadow is for a recently evicted folio; false otherwise.
395 */425 */
396void workingset_refault(struct folio *folio, void *shadow)426bool workingset_test_recent(void *shadow, bool file, bool *workingset)
397{427{
398 bool file = folio_is_file_lru(folio);
399 struct mem_cgroup *eviction_memcg;428 struct mem_cgroup *eviction_memcg;
400 struct lruvec *eviction_lruvec;429 struct lruvec *eviction_lruvec;
401 unsigned long refault_distance;430 unsigned long refault_distance;
402 unsigned long workingset_size;431 unsigned long workingset_size;
403 struct pglist_data *pgdat;
404 struct mem_cgroup *memcg;
405 unsigned long eviction;
406 struct lruvec *lruvec;
407 unsigned long refault;432 unsigned long refault;
408 bool workingset;
409 int memcgid;433 int memcgid;
410 long nr;434 struct pglist_data *pgdat;
435 unsigned long eviction;
411 436
412 if (lru_gen_enabled()) {437 if (lru_gen_enabled())
413 lru_gen_refault(folio, shadow);438 return lru_gen_test_recent(shadow, file, &memcgid, &pgdat, &eviction,
414 return;439 workingset);
415 }
416 440
417 unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset);441 unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset);
418 eviction <<= bucket_order;442 eviction <<= bucket_order;
419 443
420 /* Flush stats (and potentially sleep) before holding RCU read lock */
421 mem_cgroup_flush_stats_ratelimited();
422
423 rcu_read_lock();
424 /*444 /*
425 * Look up the memcg associated with the stored ID. It might445 * Look up the memcg associated with the stored ID. It might
426 * have been deleted since the folio's eviction.446 * have been deleted since the folio's eviction.
@@ -439,7 +459,8 @@ void workingset_refault(struct folio *folio, void *shadow) · 12 unmodified lines
439 */459 */
440 eviction_memcg = mem_cgroup_from_id(memcgid);460 eviction_memcg = mem_cgroup_from_id(memcgid);
441 if (!mem_cgroup_disabled() && !eviction_memcg)461 if (!mem_cgroup_disabled() && !eviction_memcg)
442 goto out;462 return false;
463
443 eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat);464 eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat);
444 refault = atomic_long_read(&eviction_lruvec->nonresident_age);465 refault = atomic_long_read(&eviction_lruvec->nonresident_age);
445 466
@@ -461,20 +482,6 @@ void workingset_refault(struct folio *folio, void *shadow) · 15 unmodified lines
461 */482 */
462 refault_distance = (refault - eviction) & EVICTION_MASK;483 refault_distance = (refault - eviction) & EVICTION_MASK;
463 484
464 /*
465 * The activation decision for this folio is made at the level
466 * where the eviction occurred, as that is where the LRU order
467 * during folio reclaim is being determined.
468 *
469 * However, the cgroup that will own the folio is the one that
470 * is actually experiencing the refault event.
471 */
472 nr = folio_nr_pages(folio);
473 memcg = folio_memcg(folio);
474 pgdat = folio_pgdat(folio);
475 lruvec = mem_cgroup_lruvec(memcg, pgdat);
476
477 mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr);
478 /*485 /*
479 * Compare the distance to the existing workingset size. We486 * Compare the distance to the existing workingset size. We
480 * don't activate pages that couldn't stay resident even if487 * don't activate pages that couldn't stay resident even if
@@ -495,7 +502,54 @@ void workingset_refault(struct folio *folio, void *shadow) · 14 unmodified lines
495 NR_INACTIVE_ANON);502 NR_INACTIVE_ANON);
496 }503 }
497 }504 }
498 if (refault_distance > workingset_size)505
506 return refault_distance <= workingset_size;
507}
508
509/**
510 * workingset_refault - Evaluate the refault of a previously evicted folio.
511 * @folio: The freshly allocated replacement folio.
512 * @shadow: Shadow entry of the evicted folio.
513 *
514 * Calculates and evaluates the refault distance of the previously
515 * evicted folio in the context of the node and the memcg whose memory
516 * pressure caused the eviction.
517 */
518void workingset_refault(struct folio *folio, void *shadow)
519{
520 bool file = folio_is_file_lru(folio);
521 struct pglist_data *pgdat;
522 struct mem_cgroup *memcg;
523 struct lruvec *lruvec;
524 bool workingset;
525 long nr;
526
527 if (lru_gen_enabled()) {
528 lru_gen_refault(folio, shadow);
529 return;
530 }
531
532 /* Flush stats (and potentially sleep) before holding RCU read lock */
533 mem_cgroup_flush_stats_ratelimited();
534
535 rcu_read_lock();
536
537 /*
538 * The activation decision for this folio is made at the level
539 * where the eviction occurred, as that is where the LRU order
540 * during folio reclaim is being determined.
541 *
542 * However, the cgroup that will own the folio is the one that
543 * is actually experiencing the refault event.
544 */
545 nr = folio_nr_pages(folio);
546 memcg = folio_memcg(folio);
547 pgdat = folio_pgdat(folio);
548 lruvec = mem_cgroup_lruvec(memcg, pgdat);
549
550 mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr);
551
552 if (!workingset_test_recent(shadow, file, &workingset))
499 goto out;553 goto out;
500 554
501 folio_set_active(folio);555 folio_set_active(folio);

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t