Patch series "Keep tail page private zero at free and folio split", v3. This patchset makes sure tail_page->private is zero before compound or high-order pages are returned to the allocator. It also checks tail pages that become new folio heads during large folio split, before their private fields are used by new folios. Note on ZONE_DEVICE and DAX page/folio === ZONE_DEVICE and DAX use prep_compound_tail() to reinitialize folios, so tail_page->private was reset before this patchset. There was a concern that after this patchset stale ->private can appear after ZONE_DEVICE/DAX folio initialization. My reasoning is that no code sets ZONE_DEVICE/DAX page->private, so their page->private stays zero all the time. ZONE_DEVICE_PRIVATE page migration only supports anonymous memory without swapcache, so after the migration ->private remains zero. But let me know if my reasoning is wrong. It can be fixed by adding ->private zeroing code in ZONE_DEVICE/DAX folio initialization code. Motivation === page->private is zeroed at page free time since commit ac1ea219590c0 ("mm/page_alloc: clear page->private in free_pages_prepare()"), since we concluded that it might be too much to ask every page user to free a page with ->private zeroed. The holder of the last page reference might not know whether ->private needs to be cleared. For compound and high-order pages, tail_page->private can also leak to later users if it is left uncleared. The page allocation path does not zero every tail_page->private field, so they can be seen by new users and cause unexpected issues[1]. Check tail_page->private at page free time, and check tail pages that become new folio heads during large folio split. With those checks in place, prep_compound_tail() no longer needs to clear tail_page->private when preparing compound page metadata. Overview === 1. Patch 1 clears all pages ->private before percpu-km frees them. 2. Patch 2 removes setting page->private in compaction code when a free page is taken out of the buddy allocator. cc->freepages is indexed by page order, so storing the free page order in page->private is redundant. In alloc_contig_frozen_range_noprof(), isolate_freepages_range() is used to grab free pages from buddy allocator and it leaves the aforementioned page->private set until either split_free_frozen_pages() or prep_new_page() is called. That stale value without resetting triggers the tail_page->private nonzero check once set_page_private(0) is removed from prep_compound_tail(). 3. Patch 3 adds back the page->private check for tail pages promoted to new folio heads in __split_folio_to_order(). 4. Patch 4 adds a tail_page->private check in the page free path. 5. Patch 5 removes tail_page->private zeroing from prep_compound_tail(). This patch (of 5): page->private is cleared in free page path. In a subsequent commit, tail_page->private will be checked and ensured to be zero. Clearing percpu-km allocated pages' ->private to prevent triggering warnings later, namely undo what we did in pcpu_create_chunk().
Unified · Split
| @@ -94,8 +94,15 @@ static void pcpu_destroy_chunk(struct pcpu_chunk *chunk) · 93 unmodified lines | ||
| 94 | 94 | pcpu_stats_chunk_dealloc(); |
| 95 | 95 | trace_percpu_destroy_chunk(chunk->base_addr); |
| 96 | 96 | |
| 97 | − if (chunk->data) | |
| 97 | + if (chunk->data) { | |
| 98 | + struct page *pages = (struct page *)chunk->data; | |
| 99 | + int i; | |
| 100 | + | |
| 101 | + /* clear chunk info from each page before free them */ | |
| 102 | + for (i = 0; i < nr_pages; i++) | |
| 103 | + pcpu_set_page_chunk(pages + i, NULL); | |
| 98 | 104 | __free_pages(chunk->data, order_base_2(nr_pages)); |
| 105 | + } | |
| 99 | 106 | pcpu_free_chunk(chunk); |
| 100 | 107 | } |
| 101 | 108 | |