Pull x86 entry updates from Thomas Gleixner: "The x86 entry, exception and interrupt code rework This all started about 6 month ago with the attempt to move the Posix CPU timer heavy lifting out of the timer interrupt code and just have lockless quick checks in that code path. Trivial 5 patches. This unearthed an inconsistency in the KVM handling of task work and the review requested to move all of this into generic code so other architectures can share. Valid request and solved with another 25 patches but those unearthed inconsistencies vs. RCU and instrumentation. Digging into this made it obvious that there are quite some inconsistencies vs. instrumentation in general. The int3 text poke handling in particular was completely unprotected and with the batched update of trace events even more likely to expose to endless int3 recursion. In parallel the RCU implications of instrumenting fragile entry code came up in several discussions. The conclusion of the x86 maintainer team was to go all the way and make the protection against any form of instrumentation of fragile and dangerous code pathes enforcable and verifiable by tooling. A first batch of preparatory work hit mainline with commit d5f744f9a2ac ("Pull x86 entry code updates from Thomas Gleixner") That (almost) full solution introduced a new code section '.noinstr.text' into which all code which needs to be protected from instrumentation of all sorts goes into. Any call into instrumentable code out of this section has to be annotated. objtool has support to validate this. Kprobes now excludes this section fully which also prevents BPF from fiddling with it and all 'noinstr' annotated functions also keep ftrace off. The section, kprobes and objtool changes are already merged. The major changes coming with this are: - Preparatory cleanups - Annotating of relevant functions to move them into the noinstr.text section or enforcing inlining by marking them __always_inline so the compiler cannot misplace or instrument them. - Splitting and simplifying the idtentry macro maze so that it is now clearly separated into simple exception entries and the more interesting ones which use interrupt stacks and have the paranoid handling vs. CR3 and GS. - Move quite some of the low level ASM functionality into C code: - enter_from and exit to user space handling. The ASM code now calls into C after doing the really necessary ASM handling and the return path goes back out without bells and whistels in ASM. - exception entry/exit got the equivivalent treatment - move all IRQ tracepoints from ASM to C so they can be placed as appropriate which is especially important for the int3 recursion issue. - Consolidate the declaration and definition of entry points between 32 and 64 bit. They share a common header and macros now. - Remove the extra device interrupt entry maze and just use the regular exception entry code. - All ASM entry points except NMI are now generated from the shared header file and the corresponding macros in the 32 and 64 bit entry ASM. - The C code entry points are consolidated as well with the help of DEFINE_IDTENTRY*() macros. This allows to ensure at one central point that all corresponding entry points share the same semantics. The actual function body for most entry points is in an instrumentable and sane state. There are special macros for the more sensitive entry points, e.g. INT3 and of course the nasty paranoid #NMI, #MCE, #DB and #DF. They allow to put the whole entry instrumentation and RCU handling into safe places instead of the previous pray that it is correct approach. - The INT3 text poke handling is now completely isolated and the recursion issue banned. Aside of the entry rework this required other isolation work, e.g. the ability to force inline bsearch. - Prevent #DB on fragile entry code, entry relevant memory and disable it on NMI, #MC entry, which allowed to get rid of the nested #DB IST stack shifting hackery. - A few other cleanups and enhancements which have been made possible through this and already merged changes, e.g. consolidating and further restricting the IDT code so the IDT table becomes RO after init which removes yet another popular attack vector - About 680 lines of ASM maze are gone. There are a few open issues: - An escape out of the noinstr section in the MCE handler which needs some more thought but under the aspect that MCE is a complete trainwreck by design and the propability to survive it is low, this was not high on the priority list. - Paravirtualization When PV is enabled then objtool complains about a bunch of indirect calls out of the noinstr section. There are a few straight forward ways to fix this, but the other issues vs. general correctness were more pressing than parawitz. - KVM KVM is inconsistent as well. Patches have been posted, but they have not yet been commented on or picked up by the KVM folks. - IDLE Pretty much the same problems can be found in the low level idle code especially the parts where RCU stopped watching. This was beyond the scope of the more obvious and exposable problems and is on the todo list. The lesson learned from this brain melting exercise to morph the evolved code base into something which can be validated and understood is that once again the violation of the most important engineering principle "correctness first" has caused quite a few people to spend valuable time on problems which could have been avoided in the first place. The "features first" tinkering mindset really has to stop. With that I want to say thanks to everyone involved in contributing to this effort. Special thanks go to the following people (alphabetical order): Alexandre Chartre, Andy Lutomirski, Borislav Petkov, Brian Gerst, Frederic Weisbecker, Josh Poimboeuf, Juergen Gross, Lai Jiangshan, Macro Elver, Paolo Bonzin,i Paul McKenney, Peter Zijlstra, Vitaly Kuznetsov, and Will Deacon" * tag 'x86-entry-2020-06-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (142 commits) x86/entry: Force rcu_irq_enter() when in idle task x86/entry: Make NMI use IDTENTRY_RAW x86/entry: Treat BUG/WARN as NMI-like entries x86/entry: Unbreak __irqentry_text_start/end magic x86/entry: __always_inline CR2 for noinstr lockdep: __always_inline more for noinstr x86/entry: Re-order #DB handler to avoid *SAN instrumentation x86/entry: __always_inline arch_atomic_* for noinstr x86/entry: __always_inline irqflags for noinstr x86/entry: __always_inline debugreg for noinstr x86/idt: Consolidate idt functionality x86/idt: Cleanup trap_init() x86/idt: Use proper constants for table size x86/idt: Add comments about early #PF handling x86/idt: Mark init only functions __init x86/entry: Rename trace_hardirqs_off_prepare() x86/entry: Clarify irq_{enter,exit}_rcu() x86/entry: Remove DBn stacks x86/entry: Remove debug IDT frobbing x86/entry: Optimize local_db_save() for virt ...
Showing the changes to include/linux (7 of 111 files). Show all
Unified · Split
| @@ -4,7 +4,29 @@ · 3 unmodified lines | |||
| 4 | 4 | ||
| 5 | #include <linux/types.h> | 5 | #include <linux/types.h> |
| 6 | 6 | ||
| 7 | void *bsearch(const void *key, const void *base, size_t num, size_t size, | 7 | static __always_inline |
| 8 | cmp_func_t cmp); | 8 | void *__inline_bsearch(const void *key, const void *base, size_t num, size_t size, cmp_func_t cmp) |
| 9 | { | ||
| 10 | const char *pivot; | ||
| 11 | int result; | ||
| 12 | |||
| 13 | while (num > 0) { | ||
| 14 | pivot = base + (num >> 1) * size; | ||
| 15 | result = cmp(key, pivot); | ||
| 16 | |||
| 17 | if (result == 0) | ||
| 18 | return (void *)pivot; | ||
| 19 | |||
| 20 | if (result > 0) { | ||
| 21 | base = pivot + size; | ||
| 22 | num--; | ||
| 23 | } | ||
| 24 | num >>= 1; | ||
| 25 | } | ||
| 26 | |||
| 27 | return NULL; | ||
| 28 | } | ||
| 29 | |||
| 30 | extern void *bsearch(const void *key, const void *base, size_t num, size_t size, cmp_func_t cmp); | ||
| 9 | 31 | ||
| 10 | #endif /* _LINUX_BSEARCH_H */ | 32 | #endif /* _LINUX_BSEARCH_H */ |
| @@ -33,13 +33,13 @@ static inline void user_exit(void) · 32 unmodified lines | |||
| 33 | } | 33 | } |
| 34 | 34 | ||
| 35 | /* Called with interrupts disabled. */ | 35 | /* Called with interrupts disabled. */ |
| 36 | static inline void user_enter_irqoff(void) | 36 | static __always_inline void user_enter_irqoff(void) |
| 37 | { | 37 | { |
| 38 | if (context_tracking_enabled()) | 38 | if (context_tracking_enabled()) |
| 39 | __context_tracking_enter(CONTEXT_USER); | 39 | __context_tracking_enter(CONTEXT_USER); |
| 40 | 40 | ||
| 41 | } | 41 | } |
| 42 | static inline void user_exit_irqoff(void) | 42 | static __always_inline void user_exit_irqoff(void) |
| 43 | { | 43 | { |
| 44 | if (context_tracking_enabled()) | 44 | if (context_tracking_enabled()) |
| 45 | __context_tracking_exit(CONTEXT_USER); | 45 | __context_tracking_exit(CONTEXT_USER); |
| @@ -75,7 +75,7 @@ static inline void exception_exit(enum ctx_state prev_ctx) · 29 unmodified lines | |||
| 75 | * is enabled. If context tracking is disabled, returns | 75 | * is enabled. If context tracking is disabled, returns |
| 76 | * CONTEXT_DISABLED. This should be used primarily for debugging. | 76 | * CONTEXT_DISABLED. This should be used primarily for debugging. |
| 77 | */ | 77 | */ |
| 78 | static inline enum ctx_state ct_state(void) | 78 | static __always_inline enum ctx_state ct_state(void) |
| 79 | { | 79 | { |
| 80 | return context_tracking_enabled() ? | 80 | return context_tracking_enabled() ? |
| 81 | this_cpu_read(context_tracking.state) : CONTEXT_DISABLED; | 81 | this_cpu_read(context_tracking.state) : CONTEXT_DISABLED; |
| @@ -26,12 +26,12 @@ struct context_tracking { · 25 unmodified lines | |||
| 26 | extern struct static_key_false context_tracking_key; | 26 | extern struct static_key_false context_tracking_key; |
| 27 | DECLARE_PER_CPU(struct context_tracking, context_tracking); | 27 | DECLARE_PER_CPU(struct context_tracking, context_tracking); |
| 28 | 28 | ||
| 29 | static inline bool context_tracking_enabled(void) | 29 | static __always_inline bool context_tracking_enabled(void) |
| 30 | { | 30 | { |
| 31 | return static_branch_unlikely(&context_tracking_key); | 31 | return static_branch_unlikely(&context_tracking_key); |
| 32 | } | 32 | } |
| 33 | 33 | ||
| 34 | static inline bool context_tracking_enabled_cpu(int cpu) | 34 | static __always_inline bool context_tracking_enabled_cpu(int cpu) |
| 35 | { | 35 | { |
| 36 | return context_tracking_enabled() && per_cpu(context_tracking.active, cpu); | 36 | return context_tracking_enabled() && per_cpu(context_tracking.active, cpu); |
| 37 | } | 37 | } |
| @@ -41,7 +41,7 @@ static inline bool context_tracking_enabled_this_cpu(void) · 3 unmodified lines | |||
| 41 | return context_tracking_enabled() && __this_cpu_read(context_tracking.active); | 41 | return context_tracking_enabled() && __this_cpu_read(context_tracking.active); |
| 42 | } | 42 | } |
| 43 | 43 | ||
| 44 | static inline bool context_tracking_in_user(void) | 44 | static __always_inline bool context_tracking_in_user(void) |
| 45 | { | 45 | { |
| 46 | return __this_cpu_read(context_tracking.state) == CONTEXT_USER; | 46 | return __this_cpu_read(context_tracking.state) == CONTEXT_USER; |
| 47 | } | 47 | } |
| @@ -12,7 +12,7 @@ extern int debug_locks __read_mostly; · 11 unmodified lines | |||
| 12 | extern int debug_locks_silent __read_mostly; | 12 | extern int debug_locks_silent __read_mostly; |
| 13 | 13 | ||
| 14 | 14 | ||
| 15 | static inline int __debug_locks_off(void) | 15 | static __always_inline int __debug_locks_off(void) |
| 16 | { | 16 | { |
| 17 | return xchg(&debug_locks, 0); | 17 | return xchg(&debug_locks, 0); |
| 18 | } | 18 | } |
| @@ -37,10 +37,25 @@ static __always_inline void rcu_irq_enter_check_tick(void) · 36 unmodified lines | |||
| 37 | lockdep_hardirq_enter(); \ | 37 | lockdep_hardirq_enter(); \ |
| 38 | } while (0) | 38 | } while (0) |
| 39 | 39 | ||
| 40 | /* | ||
| 41 | * Like __irq_enter() without time accounting for fast | ||
| 42 | * interrupts, e.g. reschedule IPI where time accounting | ||
| 43 | * is more expensive than the actual interrupt. | ||
| 44 | */ | ||
| 45 | #define __irq_enter_raw() \ | ||
| 46 | do { \ | ||
| 47 | preempt_count_add(HARDIRQ_OFFSET); \ | ||
| 48 | lockdep_hardirq_enter(); \ | ||
| 49 | } while (0) | ||
| 50 | |||
| 40 | /* | 51 | /* |
| 41 | * Enter irq context (on NO_HZ, update jiffies): | 52 | * Enter irq context (on NO_HZ, update jiffies): |
| 42 | */ | 53 | */ |
| 43 | extern void irq_enter(void); | 54 | void irq_enter(void); |
| 55 | /* | ||
| 56 | * Like irq_enter(), but RCU is already watching. | ||
| 57 | */ | ||
| 58 | void irq_enter_rcu(void); | ||
| 44 | 59 | ||
| 45 | /* | 60 | /* |
| 46 | * Exit irq context without processing softirqs: | 61 | * Exit irq context without processing softirqs: |
| @@ -52,10 +67,24 @@ extern void irq_enter(void); · 5 unmodified lines | |||
| 52 | preempt_count_sub(HARDIRQ_OFFSET); \ | 67 | preempt_count_sub(HARDIRQ_OFFSET); \ |
| 53 | } while (0) | 68 | } while (0) |
| 54 | 69 | ||
| 70 | /* | ||
| 71 | * Like __irq_exit() without time accounting | ||
| 72 | */ | ||
| 73 | #define __irq_exit_raw() \ | ||
| 74 | do { \ | ||
| 75 | lockdep_hardirq_exit(); \ | ||
| 76 | preempt_count_sub(HARDIRQ_OFFSET); \ | ||
| 77 | } while (0) | ||
| 78 | |||
| 55 | /* | 79 | /* |
| 56 | * Exit irq context and process softirqs if needed: | 80 | * Exit irq context and process softirqs if needed: |
| 57 | */ | 81 | */ |
| 58 | extern void irq_exit(void); | 82 | void irq_exit(void); |
| 83 | |||
| 84 | /* | ||
| 85 | * Like irq_exit(), but return with RCU watching. | ||
| 86 | */ | ||
| 87 | void irq_exit_rcu(void); | ||
| 59 | 88 | ||
| 60 | #ifndef arch_nmi_enter | 89 | #ifndef arch_nmi_enter |
| 61 | #define arch_nmi_enter() do { } while (0) | 90 | #define arch_nmi_enter() do { } while (0) |
| @@ -87,20 +116,24 @@ extern void rcu_nmi_exit(void); · 25 unmodified lines | |||
| 87 | arch_nmi_enter(); \ | 116 | arch_nmi_enter(); \ |
| 88 | printk_nmi_enter(); \ | 117 | printk_nmi_enter(); \ |
| 89 | lockdep_off(); \ | 118 | lockdep_off(); \ |
| 90 | ftrace_nmi_enter(); \ | ||
| 91 | BUG_ON(in_nmi() == NMI_MASK); \ | 119 | BUG_ON(in_nmi() == NMI_MASK); \ |
| 92 | __preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET); \ | 120 | __preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET); \ |
| 93 | rcu_nmi_enter(); \ | 121 | rcu_nmi_enter(); \ |
| 94 | lockdep_hardirq_enter(); \ | 122 | lockdep_hardirq_enter(); \ |
| 123 | instrumentation_begin(); \ | ||
| 124 | ftrace_nmi_enter(); \ | ||
| 125 | instrumentation_end(); \ | ||
| 95 | } while (0) | 126 | } while (0) |
| 96 | 127 | ||
| 97 | #define nmi_exit() \ | 128 | #define nmi_exit() \ |
| 98 | do { \ | 129 | do { \ |
| 130 | instrumentation_begin(); \ | ||
| 131 | ftrace_nmi_exit(); \ | ||
| 132 | instrumentation_end(); \ | ||
| 99 | lockdep_hardirq_exit(); \ | 133 | lockdep_hardirq_exit(); \ |
| 100 | rcu_nmi_exit(); \ | 134 | rcu_nmi_exit(); \ |
| 101 | BUG_ON(!in_nmi()); \ | 135 | BUG_ON(!in_nmi()); \ |
| 102 | __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ | 136 | __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ |
| 103 | ftrace_nmi_exit(); \ | ||
| 104 | lockdep_on(); \ | 137 | lockdep_on(); \ |
| 105 | printk_nmi_exit(); \ | 138 | printk_nmi_exit(); \ |
| 106 | arch_nmi_exit(); \ | 139 | arch_nmi_exit(); \ |
| @@ -760,8 +760,10 @@ extern int arch_early_irq_init(void); · 759 unmodified lines | |||
| 760 | /* | 760 | /* |
| 761 | * We want to know which function is an entrypoint of a hardirq or a softirq. | 761 | * We want to know which function is an entrypoint of a hardirq or a softirq. |
| 762 | */ | 762 | */ |
| 763 | #define __irq_entry __attribute__((__section__(".irqentry.text"))) | 763 | #ifndef __irq_entry |
| 764 | #define __softirq_entry \ | 764 | # define __irq_entry __attribute__((__section__(".irqentry.text"))) |
| 765 | __attribute__((__section__(".softirqentry.text"))) | 765 | #endif |
| 766 | |||
| 767 | #define __softirq_entry __attribute__((__section__(".softirqentry.text"))) | ||
| 766 | 768 | ||
| 767 | #endif | 769 | #endif |
| @@ -32,7 +32,7 @@ · 31 unmodified lines | |||
| 32 | 32 | ||
| 33 | #ifdef CONFIG_TRACE_IRQFLAGS | 33 | #ifdef CONFIG_TRACE_IRQFLAGS |
| 34 | extern void trace_hardirqs_on_prepare(void); | 34 | extern void trace_hardirqs_on_prepare(void); |
| 35 | extern void trace_hardirqs_off_prepare(void); | 35 | extern void trace_hardirqs_off_finish(void); |
| 36 | extern void trace_hardirqs_on(void); | 36 | extern void trace_hardirqs_on(void); |
| 37 | extern void trace_hardirqs_off(void); | 37 | extern void trace_hardirqs_off(void); |
| 38 | # define lockdep_hardirq_context(p) ((p)->hardirq_context) | 38 | # define lockdep_hardirq_context(p) ((p)->hardirq_context) |
| @@ -101,7 +101,7 @@ do { \ · 62 unmodified lines | |||
| 101 | 101 | ||
| 102 | #else | 102 | #else |
| 103 | # define trace_hardirqs_on_prepare() do { } while (0) | 103 | # define trace_hardirqs_on_prepare() do { } while (0) |
| 104 | # define trace_hardirqs_off_prepare() do { } while (0) | 104 | # define trace_hardirqs_off_finish() do { } while (0) |
| 105 | # define trace_hardirqs_on() do { } while (0) | 105 | # define trace_hardirqs_on() do { } while (0) |
| 106 | # define trace_hardirqs_off() do { } while (0) | 106 | # define trace_hardirqs_off() do { } while (0) |
| 107 | # define lockdep_hardirq_context(p) 0 | 107 | # define lockdep_hardirq_context(p) 0 |