The commit c75890700455 ("x86/entry/64: Remove unneeded kernel CR3 switching") removed a CR3 write in the faulting path of load_gs_index(). But the path's FENCE_SWAPGS_USER_ENTRY has no fence operation if PTI is enabled, see spectre_v1_select_mitigation(). Rather, it depended on the serializing CR3 write of SWITCH_TO_KERNEL_CR3 and since it got removed, add a FENCE_SWAPGS_KERNEL_ENTRY call to make sure speculation is blocked. [ bp: Massage commit message and comment. ]
Unified · Split
| @@ -987,11 +987,6 @@ SYM_CODE_START_LOCAL(error_entry) · 986 unmodified lines | ||
| 987 | 987 | pushq %r12 |
| 988 | 988 | ret |
| 989 | 989 | |
| 990 | −.Lerror_entry_done_lfence: | |
| 991 | − FENCE_SWAPGS_KERNEL_ENTRY | |
| 992 | −.Lerror_entry_done: | |
| 993 | − ret | |
| 994 | − | |
| 995 | 990 | /* |
| 996 | 991 | * There are two places in the kernel that can potentially fault with |
| 997 | 992 | * usergs. Handle them here. B stepping K8s sometimes report a |
| @@ -1014,8 +1009,14 @@ SYM_CODE_START_LOCAL(error_entry) · 16 unmodified lines | ||
| 1014 | 1009 | * .Lgs_change's error handler with kernel gsbase. |
| 1015 | 1010 | */ |
| 1016 | 1011 | SWAPGS |
| 1017 | − FENCE_SWAPGS_USER_ENTRY | |
| 1018 | − jmp .Lerror_entry_done | |
| 1012 | + | |
| 1013 | + /* | |
| 1014 | + * Issue an LFENCE to prevent GS speculation, regardless of whether it is a | |
| 1015 | + * kernel or user gsbase. | |
| 1016 | + */ | |
| 1017 | +.Lerror_entry_done_lfence: | |
| 1018 | + FENCE_SWAPGS_KERNEL_ENTRY | |
| 1019 | + ret | |
| 1019 | 1020 | |
| 1020 | 1021 | .Lbstep_iret: |
| 1021 | 1022 | /* Fix truncated RIP */ |