History

mm/memory: fix PMD/PUD checks in follow_pfnmap_start()

follow_pfnmap_start() suffers from two problems: (1) We are not re-fetching the pmd/pud after taking the PTL Therefore, we are not properly stabilizing what the lock actually protects. If there is concurrent zapping, we would indicate to the caller that we found an entry, however, that entry might already have been invalidated, or contain a different PFN after taking the lock. Properly use pmdp_get() / pudp_get() after taking the lock. (2) pmd_leaf() / pud_leaf() are not well defined on non-present entries pmd_leaf()/pud_leaf() could wrongly trigger on non-present entries. There is no real guarantee that pmd_leaf()/pud_leaf() returns something reasonable on non-present entries. Most architectures indeed either perform a present check or make it work by smart use of flags. However, for example loongarch checks the _PAGE_HUGE flag in pmd_leaf(), and always sets the _PAGE_HUGE flag in __swp_entry_to_pmd(). Whereby pmd_trans_huge() explicitly checks pmd_present(), pmd_leaf() does not do that. Let's check pmd_present()/pud_present() before assuming "the is a present PMD leaf" when spotting pmd_leaf()/pud_leaf(), like other page table handling code that traverses user page tables does. Given that non-present PMD entries are likely rare in VM_IO|VM_PFNMAP, (1) is likely more relevant than (2). It is questionable how often (1) would actually trigger, but let's CC stable to be sure. This was found by code inspection.

David Hildenbrand (Arm) committed 7 months ago

Commit ffef67b1 parent 65570041 file · +15 −3Browse files

Unified · Split

mm/memory.c+15−3
@@ -6815,11 +6815,16 @@ retry: · 6814 unmodified lines
68156815
68166816 pudp = pud_offset(p4dp, address);
68176817 pud = pudp_get(pudp);
6818− if (pud_none(pud))
6818+ if (!pud_present(pud))
68196819 goto out;
68206820 if (pud_leaf(pud)) {
68216821 lock = pud_lock(mm, pudp);
6822− if (!unlikely(pud_leaf(pud))) {
6822+ pud = pudp_get(pudp);
6823+
6824+ if (unlikely(!pud_present(pud))) {
6825+ spin_unlock(lock);
6826+ goto out;
6827+ } else if (unlikely(!pud_leaf(pud))) {
68236828 spin_unlock(lock);
68246829 goto retry;
68256830 }
@@ -6831,9 +6836,16 @@ retry: · 5 unmodified lines
68316836
68326837 pmdp = pmd_offset(pudp, address);
68336838 pmd = pmdp_get_lockless(pmdp);
6839+ if (!pmd_present(pmd))
6840+ goto out;
68346841 if (pmd_leaf(pmd)) {
68356842 lock = pmd_lock(mm, pmdp);
6836− if (!unlikely(pmd_leaf(pmd))) {
6843+ pmd = pmdp_get(pmdp);
6844+
6845+ if (unlikely(!pmd_present(pmd))) {
6846+ spin_unlock(lock);
6847+ goto out;
6848+ } else if (unlikely(!pmd_leaf(pmd))) {
68376849 spin_unlock(lock);
68386850 goto retry;
68396851 }

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t