Merge tag 'x86-entry-2020-06-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 entry updates from Thomas Gleixner: "The x86 entry, exception and interrupt code rework This all started about 6 month ago with the attempt to move the Posix CPU timer heavy lifting out of the timer interrupt code and just have lockless quick checks in that code path. Trivial 5 patches. This unearthed an inconsistency in the KVM handling of task work and the review requested to move all of this into generic code so other architectures can share. Valid request and solved with another 25 patches but those unearthed inconsistencies vs. RCU and instrumentation. Digging into this made it obvious that there are quite some inconsistencies vs. instrumentation in general. The int3 text poke handling in particular was completely unprotected and with the batched update of trace events even more likely to expose to endless int3 recursion. In parallel the RCU implications of instrumenting fragile entry code came up in several discussions. The conclusion of the x86 maintainer team was to go all the way and make the protection against any form of instrumentation of fragile and dangerous code pathes enforcable and verifiable by tooling. A first batch of preparatory work hit mainline with commit d5f744f9a2ac ("Pull x86 entry code updates from Thomas Gleixner") That (almost) full solution introduced a new code section '.noinstr.text' into which all code which needs to be protected from instrumentation of all sorts goes into. Any call into instrumentable code out of this section has to be annotated. objtool has support to validate this. Kprobes now excludes this section fully which also prevents BPF from fiddling with it and all 'noinstr' annotated functions also keep ftrace off. The section, kprobes and objtool changes are already merged. The major changes coming with this are: - Preparatory cleanups - Annotating of relevant functions to move them into the noinstr.text section or enforcing inlining by marking them __always_inline so the compiler cannot misplace or instrument them. - Splitting and simplifying the idtentry macro maze so that it is now clearly separated into simple exception entries and the more interesting ones which use interrupt stacks and have the paranoid handling vs. CR3 and GS. - Move quite some of the low level ASM functionality into C code: - enter_from and exit to user space handling. The ASM code now calls into C after doing the really necessary ASM handling and the return path goes back out without bells and whistels in ASM. - exception entry/exit got the equivivalent treatment - move all IRQ tracepoints from ASM to C so they can be placed as appropriate which is especially important for the int3 recursion issue. - Consolidate the declaration and definition of entry points between 32 and 64 bit. They share a common header and macros now. - Remove the extra device interrupt entry maze and just use the regular exception entry code. - All ASM entry points except NMI are now generated from the shared header file and the corresponding macros in the 32 and 64 bit entry ASM. - The C code entry points are consolidated as well with the help of DEFINE_IDTENTRY*() macros. This allows to ensure at one central point that all corresponding entry points share the same semantics. The actual function body for most entry points is in an instrumentable and sane state. There are special macros for the more sensitive entry points, e.g. INT3 and of course the nasty paranoid #NMI, #MCE, #DB and #DF. They allow to put the whole entry instrumentation and RCU handling into safe places instead of the previous pray that it is correct approach. - The INT3 text poke handling is now completely isolated and the recursion issue banned. Aside of the entry rework this required other isolation work, e.g. the ability to force inline bsearch. - Prevent #DB on fragile entry code, entry relevant memory and disable it on NMI, #MC entry, which allowed to get rid of the nested #DB IST stack shifting hackery. - A few other cleanups and enhancements which have been made possible through this and already merged changes, e.g. consolidating and further restricting the IDT code so the IDT table becomes RO after init which removes yet another popular attack vector - About 680 lines of ASM maze are gone. There are a few open issues: - An escape out of the noinstr section in the MCE handler which needs some more thought but under the aspect that MCE is a complete trainwreck by design and the propability to survive it is low, this was not high on the priority list. - Paravirtualization When PV is enabled then objtool complains about a bunch of indirect calls out of the noinstr section. There are a few straight forward ways to fix this, but the other issues vs. general correctness were more pressing than parawitz. - KVM KVM is inconsistent as well. Patches have been posted, but they have not yet been commented on or picked up by the KVM folks. - IDLE Pretty much the same problems can be found in the low level idle code especially the parts where RCU stopped watching. This was beyond the scope of the more obvious and exposable problems and is on the todo list. The lesson learned from this brain melting exercise to morph the evolved code base into something which can be validated and understood is that once again the violation of the most important engineering principle "correctness first" has caused quite a few people to spend valuable time on problems which could have been avoided in the first place. The "features first" tinkering mindset really has to stop. With that I want to say thanks to everyone involved in contributing to this effort. Special thanks go to the following people (alphabetical order): Alexandre Chartre, Andy Lutomirski, Borislav Petkov, Brian Gerst, Frederic Weisbecker, Josh Poimboeuf, Juergen Gross, Lai Jiangshan, Macro Elver, Paolo Bonzin,i Paul McKenney, Peter Zijlstra, Vitaly Kuznetsov, and Will Deacon" * tag 'x86-entry-2020-06-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (142 commits) x86/entry: Force rcu_irq_enter() when in idle task x86/entry: Make NMI use IDTENTRY_RAW x86/entry: Treat BUG/WARN as NMI-like entries x86/entry: Unbreak __irqentry_text_start/end magic x86/entry: __always_inline CR2 for noinstr lockdep: __always_inline more for noinstr x86/entry: Re-order #DB handler to avoid *SAN instrumentation x86/entry: __always_inline arch_atomic_* for noinstr x86/entry: __always_inline irqflags for noinstr x86/entry: __always_inline debugreg for noinstr x86/idt: Consolidate idt functionality x86/idt: Cleanup trap_init() x86/idt: Use proper constants for table size x86/idt: Add comments about early #PF handling x86/idt: Mark init only functions __init x86/entry: Rename trace_hardirqs_off_prepare() x86/entry: Clarify irq_{enter,exit}_rcu() x86/entry: Remove DBn stacks x86/entry: Remove debug IDT frobbing x86/entry: Optimize local_db_save() for virt ...

Linus Torvalds committed 6 years ago

Commit 076f14b2 parents 6c32978 0bf3924111 files · +2727 −2414Browse files

Showing the changes to include/linux (7 of 111 files). Show all

Unified · Split

7 files

include/linux/bsearch.h +24 −2
include/linux/context_tracking.h +3 −3
include/linux/context_tracking_state.h +3 −3
include/linux/debug_locks.h +1 −1
include/linux/hardirq.h +37 −4
include/linux/interrupt.h +5 −3
include/linux/irqflags.h +2 −2

include/linux/bsearch.h+24−2

@@ -4,7 +4,29 @@ · 3 unmodified lines
4	4
5	5	#include <linux/types.h>
6	6
7		−void bsearch(const void key, const void *base, size_t num, size_t size,
8		− cmp_func_t cmp);
	7	+static __always_inline
	8	+void __inline_bsearch(const void key, const void *base, size_t num, size_t size, cmp_func_t cmp)
	9	+{
	10	+ const char *pivot;
	11	+ int result;
	12	+
	13	+ while (num > 0) {
	14	+ pivot = base + (num >> 1) * size;
	15	+ result = cmp(key, pivot);
	16	+
	17	+ if (result == 0)
	18	+ return (void *)pivot;
	19	+
	20	+ if (result > 0) {
	21	+ base = pivot + size;
	22	+ num--;
	23	+ }
	24	+ num >>= 1;
	25	+ }
	26	+
	27	+ return NULL;
	28	+}
	29	+
	30	+extern void bsearch(const void key, const void *base, size_t num, size_t size, cmp_func_t cmp);
9	31
10	32	#endif /* _LINUX_BSEARCH_H */

include/linux/context_tracking.h+3−3

@@ -33,13 +33,13 @@ static inline void user_exit(void) · 32 unmodified lines
33	33	}
34	34
35	35	/* Called with interrupts disabled. */
36		−static inline void user_enter_irqoff(void)
	36	+static __always_inline void user_enter_irqoff(void)
37	37	{
38	38	if (context_tracking_enabled())
39	39	__context_tracking_enter(CONTEXT_USER);
40	40
41	41	}
42		−static inline void user_exit_irqoff(void)
	42	+static __always_inline void user_exit_irqoff(void)
43	43	{
44	44	if (context_tracking_enabled())
45	45	__context_tracking_exit(CONTEXT_USER);
@@ -75,7 +75,7 @@ static inline void exception_exit(enum ctx_state prev_ctx) · 29 unmodified lines
75	75	* is enabled. If context tracking is disabled, returns
76	76	* CONTEXT_DISABLED. This should be used primarily for debugging.
77	77	*/
78		−static inline enum ctx_state ct_state(void)
	78	+static __always_inline enum ctx_state ct_state(void)
79	79	{
80	80	return context_tracking_enabled() ?
81	81	this_cpu_read(context_tracking.state) : CONTEXT_DISABLED;

include/linux/context_tracking_state.h+3−3

@@ -26,12 +26,12 @@ struct context_tracking { · 25 unmodified lines
26	26	extern struct static_key_false context_tracking_key;
27	27	DECLARE_PER_CPU(struct context_tracking, context_tracking);
28	28
29		−static inline bool context_tracking_enabled(void)
	29	+static __always_inline bool context_tracking_enabled(void)
30	30	{
31	31	return static_branch_unlikely(&context_tracking_key);
32	32	}
33	33
34		−static inline bool context_tracking_enabled_cpu(int cpu)
	34	+static __always_inline bool context_tracking_enabled_cpu(int cpu)
35	35	{
36	36	return context_tracking_enabled() && per_cpu(context_tracking.active, cpu);
37	37	}
@@ -41,7 +41,7 @@ static inline bool context_tracking_enabled_this_cpu(void) · 3 unmodified lines
41	41	return context_tracking_enabled() && __this_cpu_read(context_tracking.active);
42	42	}
43	43
44		−static inline bool context_tracking_in_user(void)
	44	+static __always_inline bool context_tracking_in_user(void)
45	45	{
46	46	return __this_cpu_read(context_tracking.state) == CONTEXT_USER;
47	47	}

include/linux/debug_locks.h+1−1

@@ -12,7 +12,7 @@ extern int debug_locks __read_mostly; · 11 unmodified lines
12	12	extern int debug_locks_silent __read_mostly;
13	13
14	14
15		−static inline int __debug_locks_off(void)
	15	+static __always_inline int __debug_locks_off(void)
16	16	{
17	17	return xchg(&debug_locks, 0);
18	18	}

include/linux/hardirq.h+37−4

@@ -37,10 +37,25 @@ static __always_inline void rcu_irq_enter_check_tick(void) · 36 unmodified lines
37	37	lockdep_hardirq_enter(); \
38	38	} while (0)
39	39
	40	+/*
	41	+ * Like __irq_enter() without time accounting for fast
	42	+ * interrupts, e.g. reschedule IPI where time accounting
	43	+ * is more expensive than the actual interrupt.
	44	+ */
	45	+#define __irq_enter_raw() \
	46	+ do { \
	47	+ preempt_count_add(HARDIRQ_OFFSET); \
	48	+ lockdep_hardirq_enter(); \
	49	+ } while (0)
	50	+
40	51	/*
41	52	* Enter irq context (on NO_HZ, update jiffies):
42	53	*/
43		−extern void irq_enter(void);
	54	+void irq_enter(void);
	55	+/*
	56	+ * Like irq_enter(), but RCU is already watching.
	57	+ */
	58	+void irq_enter_rcu(void);
44	59
45	60	/*
46	61	* Exit irq context without processing softirqs:
@@ -52,10 +67,24 @@ extern void irq_enter(void); · 5 unmodified lines
52	67	preempt_count_sub(HARDIRQ_OFFSET); \
53	68	} while (0)
54	69
	70	+/*
	71	+ * Like __irq_exit() without time accounting
	72	+ */
	73	+#define __irq_exit_raw() \
	74	+ do { \
	75	+ lockdep_hardirq_exit(); \
	76	+ preempt_count_sub(HARDIRQ_OFFSET); \
	77	+ } while (0)
	78	+
55	79	/*
56	80	* Exit irq context and process softirqs if needed:
57	81	*/
58		−extern void irq_exit(void);
	82	+void irq_exit(void);
	83	+
	84	+/*
	85	+ * Like irq_exit(), but return with RCU watching.
	86	+ */
	87	+void irq_exit_rcu(void);
59	88
60	89	#ifndef arch_nmi_enter
61	90	#define arch_nmi_enter() do { } while (0)
@@ -87,20 +116,24 @@ extern void rcu_nmi_exit(void); · 25 unmodified lines
87	116	arch_nmi_enter(); \
88	117	printk_nmi_enter(); \
89	118	lockdep_off(); \
90		− ftrace_nmi_enter(); \
91	119	BUG_ON(in_nmi() == NMI_MASK); \
92	120	__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET); \
93	121	rcu_nmi_enter(); \
94	122	lockdep_hardirq_enter(); \
	123	+ instrumentation_begin(); \
	124	+ ftrace_nmi_enter(); \
	125	+ instrumentation_end(); \
95	126	} while (0)
96	127
97	128	#define nmi_exit() \
98	129	do { \
	130	+ instrumentation_begin(); \
	131	+ ftrace_nmi_exit(); \
	132	+ instrumentation_end(); \
99	133	lockdep_hardirq_exit(); \
100	134	rcu_nmi_exit(); \
101	135	BUG_ON(!in_nmi()); \
102	136	__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \
103		− ftrace_nmi_exit(); \
104	137	lockdep_on(); \
105	138	printk_nmi_exit(); \
106	139	arch_nmi_exit(); \

include/linux/interrupt.h+5−3

@@ -760,8 +760,10 @@ extern int arch_early_irq_init(void); · 759 unmodified lines
760	760	/*
761	761	* We want to know which function is an entrypoint of a hardirq or a softirq.
762	762	*/
763		−#define __irq_entry __attribute__((__section__(".irqentry.text")))
764		−#define __softirq_entry \
765		− __attribute__((__section__(".softirqentry.text")))
	763	+#ifndef __irq_entry
	764	+# define __irq_entry __attribute__((__section__(".irqentry.text")))
	765	+#endif
	766	+
	767	+#define __softirq_entry __attribute__((__section__(".softirqentry.text")))
766	768
767	769	#endif

include/linux/irqflags.h+2−2

@@ -32,7 +32,7 @@ · 31 unmodified lines
32	32
33	33	#ifdef CONFIG_TRACE_IRQFLAGS
34	34	extern void trace_hardirqs_on_prepare(void);
35		− extern void trace_hardirqs_off_prepare(void);
	35	+ extern void trace_hardirqs_off_finish(void);
36	36	extern void trace_hardirqs_on(void);
37	37	extern void trace_hardirqs_off(void);
38	38	# define lockdep_hardirq_context(p) ((p)->hardirq_context)
@@ -101,7 +101,7 @@ do { \ · 62 unmodified lines
101	101
102	102	#else
103	103	# define trace_hardirqs_on_prepare() do { } while (0)
104		−# define trace_hardirqs_off_prepare() do { } while (0)
	104	+# define trace_hardirqs_off_finish() do { } while (0)
105	105	# define trace_hardirqs_on() do { } while (0)
106	106	# define trace_hardirqs_off() do { } while (0)
107	107	# define lockdep_hardirq_context(p) 0

Merge tag 'x86-entry-2020-06-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code