History

mm: delay page_remove_rmap() until after the TLB has been flushed

When we remove a page table entry, we are very careful to only free the page after we have flushed the TLB, because other CPUs could still be using the page through stale TLB entries until after the flush. However, we have removed the rmap entry for that page early, which means that functions like folio_mkclean() would end up not serializing with the page table lock because the page had already been made invisible to rmap. And that is a problem, because while the TLB entry exists, we could end up with the following situation: (a) one CPU could come in and clean it, never seeing our mapping of the page (b) another CPU could continue to use the stale and dirty TLB entry and continue to write to said page resulting in a page that has been dirtied, but then marked clean again, all while another CPU might have dirtied it some more. End result: possibly lost dirty data. This extends our current TLB gather infrastructure to optionally track a "should I do a delayed page_remove_rmap() for this page after flushing the TLB". It uses the newly introduced 'encoded page pointer' to do that without having to keep separate data around. Note, this is complicated by a couple of issues: - we want to delay the rmap removal, but not past the page table lock, because that simplifies the memcg accounting - only SMP configurations want to delay TLB flushing, since on UP there are obviously no remote TLBs to worry about, and the page table lock means there are no preemption issues either - s390 has its own mmu_gather model that doesn't delay TLB flushing, and as a result also does not want the delayed rmap. As such, we can treat S390 like the UP case and use a common fallback for the "no delays" case. - we can track an enormous number of pages in our mmu_gather structure, with MAX_GATHER_BATCH_COUNT batches of MAX_TABLE_BATCH pages each, all set up to be approximately 10k pending pages. We do not want to have a huge number of batched pages that we then need to check for delayed rmap handling inside the page table lock. Particularly that last point results in a noteworthy detail, where the normal page batch gathering is limited once we have delayed rmaps pending, in such a way that only the last batch (the so-called "active batch") in the mmu_gather structure can have any delayed entries. NOTE! While the "possibly lost dirty data" sounds catastrophic, for this all to happen you need to have a user thread doing either madvise() with MADV_DONTNEED or a full re-mmap() of the area concurrently with another thread continuing to use said mapping. So arguably this is about user space doing crazy things, but from a VM consistency standpoint it's better if we track the dirty bit properly even when user space goes off the rails. [akpm@linux-foundation.org: fix UP build, per Linus] Link: https://lore.kernel.org/all/B88D3073-440A-41C7-95F4-895D3F657EF2@gmail.com/ Link: https://lkml.kernel.org/r/20221109203051.1835763-4-torvalds@linux-foundation.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: Hugh Dickins <hughd@google.com> Reported-by: Nadav Amit <nadav.amit@gmail.com> Tested-by: Nadav Amit <nadav.amit@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

Linus Torvalds committed 4 years ago

Commit 5df397d1 parent 7cc8f9c4 files · +82 −8Browse files

Unified · Split

4 files
  1. arch/s390/include/asm/tlb.h +3 −0
  2. include/asm-generic/tlb.h +29 −2
  3. mm/memory.c +17 −6
  4. mm/mmu_gather.c +33 −0
arch/s390/include/asm/tlb.h+3−0
@@ -41,6 +41,9 @@ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, · 40 unmodified lines
41 * Release the page cache reference for a pte removed by41 * Release the page cache reference for a pte removed by
42 * tlb_ptep_clear_flush. In both flush modes the tlb for a page cache page42 * tlb_ptep_clear_flush. In both flush modes the tlb for a page cache page
43 * has already been freed, so just do free_page_and_swap_cache.43 * has already been freed, so just do free_page_and_swap_cache.
44 *
45 * s390 doesn't delay rmap removal, so there is nothing encoded in
46 * the page pointer.
44 */47 */
45static inline bool __tlb_remove_page_size(struct mmu_gather *tlb,48static inline bool __tlb_remove_page_size(struct mmu_gather *tlb,
46 struct encoded_page *page,49 struct encoded_page *page,
include/asm-generic/tlb.h+29−2
@@ -263,6 +263,28 @@ struct mmu_gather_batch { · 262 unmodified lines
263extern bool __tlb_remove_page_size(struct mmu_gather *tlb,263extern bool __tlb_remove_page_size(struct mmu_gather *tlb,
264 struct encoded_page *page,264 struct encoded_page *page,
265 int page_size);265 int page_size);
266
267#ifdef CONFIG_SMP
268/*
269 * This both sets 'delayed_rmap', and returns true. It would be an inline
270 * function, except we define it before the 'struct mmu_gather'.
271 */
272#define tlb_delay_rmap(tlb) (((tlb)->delayed_rmap = 1), true)
273extern void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma);
274#endif
275
276#endif
277
278/*
279 * We have a no-op version of the rmap removal that doesn't
280 * delay anything. That is used on S390, which flushes remote
281 * TLBs synchronously, and on UP, which doesn't have any
282 * remote TLBs to flush and is not preemptible due to this
283 * all happening under the page table lock.
284 */
285#ifndef tlb_delay_rmap
286#define tlb_delay_rmap(tlb) (false)
287static inline void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma) { }
266#endif288#endif
267 289
268/*290/*
@@ -295,6 +317,11 @@ struct mmu_gather { · 26 unmodified lines
295 */317 */
296 unsigned int freed_tables : 1;318 unsigned int freed_tables : 1;
297 319
320 /*
321 * Do we have pending delayed rmap removals?
322 */
323 unsigned int delayed_rmap : 1;
324
298 /*325 /*
299 * at which levels have we cleared entries?326 * at which levels have we cleared entries?
300 */327 */
@@ -440,9 +467,9 @@ static inline void tlb_remove_page_size(struct mmu_gather *tlb, · 139 unmodified lines
440 tlb_flush_mmu(tlb);467 tlb_flush_mmu(tlb);
441}468}
442 469
443static inline bool __tlb_remove_page(struct mmu_gather *tlb, struct page *page)470static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, struct page *page, unsigned int flags)
444{471{
445 return __tlb_remove_page_size(tlb, encode_page(page, 0), PAGE_SIZE);472 return __tlb_remove_page_size(tlb, encode_page(page, flags), PAGE_SIZE);
446}473}
447 474
448/* tlb_remove_page475/* tlb_remove_page
mm/memory.c+17−6
@@ -1374,6 +1374,8 @@ again: · 1373 unmodified lines
1374 break;1374 break;
1375 1375
1376 if (pte_present(ptent)) {1376 if (pte_present(ptent)) {
1377 unsigned int delay_rmap;
1378
1377 page = vm_normal_page(vma, addr, ptent);1379 page = vm_normal_page(vma, addr, ptent);
1378 if (unlikely(!should_zap_page(details, page)))1380 if (unlikely(!should_zap_page(details, page)))
1379 continue;1381 continue;
@@ -1385,20 +1387,26 @@ again: · 5 unmodified lines
1385 if (unlikely(!page))1387 if (unlikely(!page))
1386 continue;1388 continue;
1387 1389
1390 delay_rmap = 0;
1388 if (!PageAnon(page)) {1391 if (!PageAnon(page)) {
1389 if (pte_dirty(ptent)) {1392 if (pte_dirty(ptent)) {
1390 force_flush = 1;
1391 set_page_dirty(page);1393 set_page_dirty(page);
1394 if (tlb_delay_rmap(tlb)) {
1395 delay_rmap = 1;
1396 force_flush = 1;
1397 }
1392 }1398 }
1393 if (pte_young(ptent) &&1399 if (pte_young(ptent) &&
1394 likely(!(vma->vm_flags & VM_SEQ_READ)))1400 likely(!(vma->vm_flags & VM_SEQ_READ)))
1395 mark_page_accessed(page);1401 mark_page_accessed(page);
1396 }1402 }
1397 rss[mm_counter(page)]--;1403 rss[mm_counter(page)]--;
1398 page_remove_rmap(page, vma, false);1404 if (!delay_rmap) {
1399 if (unlikely(page_mapcount(page) < 0))1405 page_remove_rmap(page, vma, false);
1400 print_bad_pte(vma, addr, ptent, page);1406 if (unlikely(page_mapcount(page) < 0))
1401 if (unlikely(__tlb_remove_page(tlb, page))) {1407 print_bad_pte(vma, addr, ptent, page);
1408 }
1409 if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) {
1402 force_flush = 1;1410 force_flush = 1;
1403 addr += PAGE_SIZE;1411 addr += PAGE_SIZE;
1404 break;1412 break;
@@ -1455,8 +1463,11 @@ again: · 50 unmodified lines
1455 arch_leave_lazy_mmu_mode();1463 arch_leave_lazy_mmu_mode();
1456 1464
1457 /* Do the actual TLB flush before dropping ptl */1465 /* Do the actual TLB flush before dropping ptl */
1458 if (force_flush)1466 if (force_flush) {
1459 tlb_flush_mmu_tlbonly(tlb);1467 tlb_flush_mmu_tlbonly(tlb);
1468 if (tlb->delayed_rmap)
1469 tlb_flush_rmaps(tlb, vma);
1470 }
1460 pte_unmap_unlock(start_pte, ptl);1471 pte_unmap_unlock(start_pte, ptl);
1461 1472
1462 /*1473 /*
mm/mmu_gather.c+33−0
@@ -9,6 +9,7 @@ · 8 unmodified lines
9#include <linux/rcupdate.h>9#include <linux/rcupdate.h>
10#include <linux/smp.h>10#include <linux/smp.h>
11#include <linux/swap.h>11#include <linux/swap.h>
12#include <linux/rmap.h>
12 13
13#include <asm/pgalloc.h>14#include <asm/pgalloc.h>
14#include <asm/tlb.h>15#include <asm/tlb.h>
@@ -19,6 +20,10 @@ static bool tlb_next_batch(struct mmu_gather *tlb) · 4 unmodified lines
19{20{
20 struct mmu_gather_batch *batch;21 struct mmu_gather_batch *batch;
21 22
23 /* No more batching if we have delayed rmaps pending */
24 if (tlb->delayed_rmap)
25 return false;
26
22 batch = tlb->active;27 batch = tlb->active;
23 if (batch->next) {28 if (batch->next) {
24 tlb->active = batch->next;29 tlb->active = batch->next;
@@ -43,6 +48,33 @@ static bool tlb_next_batch(struct mmu_gather *tlb) · 18 unmodified lines
43 return true;48 return true;
44}49}
45 50
51#ifdef CONFIG_SMP
52/**
53 * tlb_flush_rmaps - do pending rmap removals after we have flushed the TLB
54 * @tlb: the current mmu_gather
55 *
56 * Note that because of how tlb_next_batch() above works, we will
57 * never start new batches with pending delayed rmaps, so we only
58 * need to walk through the current active batch.
59 */
60void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma)
61{
62 struct mmu_gather_batch *batch;
63
64 batch = tlb->active;
65 for (int i = 0; i < batch->nr; i++) {
66 struct encoded_page *enc = batch->encoded_pages[i];
67
68 if (encoded_page_flags(enc)) {
69 struct page *page = encoded_page_ptr(enc);
70 page_remove_rmap(page, vma, false);
71 }
72 }
73
74 tlb->delayed_rmap = 0;
75}
76#endif
77
46static void tlb_batch_pages_flush(struct mmu_gather *tlb)78static void tlb_batch_pages_flush(struct mmu_gather *tlb)
47{79{
48 struct mmu_gather_batch *batch;80 struct mmu_gather_batch *batch;
@@ -284,6 +316,7 @@ static void __tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, · 235 unmodified lines
284 tlb->active = &tlb->local;316 tlb->active = &tlb->local;
285 tlb->batch_count = 0;317 tlb->batch_count = 0;
286#endif318#endif
319 tlb->delayed_rmap = 0;
287 320
288 tlb_table_init(tlb);321 tlb_table_init(tlb);
289#ifdef CONFIG_MMU_GATHER_PAGE_SIZE322#ifdef CONFIG_MMU_GATHER_PAGE_SIZE

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t