lael-labs/linux

.mailmap+4−0

 Patrick Mochel <mochel@digitalimplant.org>
 Paul Burton <paulburton@kernel.org> <paul.burton@imgtec.com>
 Paul Burton <paulburton@kernel.org> <paul.burton@mips.com>
+Paul E. McKenney <paulmck@kernel.org> <paulmck@linux.ibm.com>
+Paul E. McKenney <paulmck@kernel.org> <paulmck@linux.vnet.ibm.com>
+Paul E. McKenney <paulmck@kernel.org> <paul.mckenney@linaro.org>
+Paul E. McKenney <paulmck@kernel.org> <paulmck@us.ibm.com>
 Peter A Jonsson <pj@ludd.ltu.se>
 Peter Oruba <peter@oruba.de>
 Peter Oruba <peter.oruba@amd.com>

Documentation/RCU/NMI-RCU.rstrenamed from Documentation/RCU/NMI-RCU.txt+28−25

@@ -1,4 +1,7 @@
		1	.. _NMI_rcu_doc:
		2
1	Using RCU to Protect Dynamic NMI Handlers	3	Using RCU to Protect Dynamic NMI Handlers
		4	=========================================
2		5
3		6
4	Although RCU is usually used to protect read-mostly data structures,	7	Although RCU is usually used to protect read-mostly data structures,
@@ -9,7 +12,7 @@ work in "arch/x86/oprofile/nmi_timer_int.c" and in · 4 unmodified lines
9	"arch/x86/kernel/traps.c".	12	"arch/x86/kernel/traps.c".
10		13
11	The relevant pieces of code are listed below, each followed by a	14	The relevant pieces of code are listed below, each followed by a
12	brief explanation.	15	brief explanation::
13		16
14	static int dummy_nmi_callback(struct pt_regs *regs, int cpu)	17	static int dummy_nmi_callback(struct pt_regs *regs, int cpu)
15	{	18	{
@@ -18,12 +21,12 @@ brief explanation. · 2 unmodified lines
18		21
19	The dummy_nmi_callback() function is a "dummy" NMI handler that does	22	The dummy_nmi_callback() function is a "dummy" NMI handler that does
20	nothing, but returns zero, thus saying that it did nothing, allowing	23	nothing, but returns zero, thus saying that it did nothing, allowing
21	the NMI handler to take the default machine-specific action.	24	the NMI handler to take the default machine-specific action::
22		25
23	static nmi_callback_t nmi_callback = dummy_nmi_callback;	26	static nmi_callback_t nmi_callback = dummy_nmi_callback;
24		27
25	This nmi_callback variable is a global function pointer to the current	28	This nmi_callback variable is a global function pointer to the current
26	NMI handler.	29	NMI handler::
27		30
28	void do_nmi(struct pt_regs * regs, long error_code)	31	void do_nmi(struct pt_regs * regs, long error_code)
29	{	32	{
@@ -53,11 +56,12 @@ anyway. However, in practice it is a good documentation aid, particularly · 23 unmodified lines
53	for anyone attempting to do something similar on Alpha or on systems	56	for anyone attempting to do something similar on Alpha or on systems
54	with aggressive optimizing compilers.	57	with aggressive optimizing compilers.
55		58
56	Quick Quiz: Why might the rcu_dereference_sched() be necessary on Alpha,	59	Quick Quiz:
57	given that the code referenced by the pointer is read-only?	60	Why might the rcu_dereference_sched() be necessary on Alpha, given that the code referenced by the pointer is read-only?
58		61
		62	:ref:`Answer to Quick Quiz <answer_quick_quiz_NMI>`
59		63
60	Back to the discussion of NMI and RCU...	64	Back to the discussion of NMI and RCU::
61		65
62	void set_nmi_callback(nmi_callback_t callback)	66	void set_nmi_callback(nmi_callback_t callback)
63	{	67	{
@@ -68,7 +72,7 @@ The set_nmi_callback() function registers an NMI handler. Note that any · 4 unmodified lines
68	data that is to be used by the callback must be initialized up -before-	72	data that is to be used by the callback must be initialized up -before-
69	the call to set_nmi_callback(). On architectures that do not order	73	the call to set_nmi_callback(). On architectures that do not order
70	writes, the rcu_assign_pointer() ensures that the NMI handler sees the	74	writes, the rcu_assign_pointer() ensures that the NMI handler sees the
71	initialized values.	75	initialized values::
72		76
73	void unset_nmi_callback(void)	77	void unset_nmi_callback(void)
74	{	78	{
@@ -82,7 +86,7 @@ up any data structures used by the old NMI handler until execution · 7 unmodified lines
82	of it completes on all other CPUs.	86	of it completes on all other CPUs.
83		87
84	One way to accomplish this is via synchronize_rcu(), perhaps as	88	One way to accomplish this is via synchronize_rcu(), perhaps as
85	follows:	89	follows::
86		90
87	unset_nmi_callback();	91	unset_nmi_callback();
88	synchronize_rcu();	92	synchronize_rcu();
@@ -98,24 +102,23 @@ to free up the handler's data as soon as synchronize_rcu() returns. · 9 unmodified lines
98	Important note: for this to work, the architecture in question must	102	Important note: for this to work, the architecture in question must
99	invoke nmi_enter() and nmi_exit() on NMI entry and exit, respectively.	103	invoke nmi_enter() and nmi_exit() on NMI entry and exit, respectively.
100		104
		105	.. _answer_quick_quiz_NMI:
101		106
102	Answer to Quick Quiz	107	Answer to Quick Quiz:
103		108	Why might the rcu_dereference_sched() be necessary on Alpha, given that the code referenced by the pointer is read-only?
104	Why might the rcu_dereference_sched() be necessary on Alpha, given
105	that the code referenced by the pointer is read-only?
106		109
107	Answer: The caller to set_nmi_callback() might well have	110	The caller to set_nmi_callback() might well have
108	initialized some data that is to be used by the new NMI	111	initialized some data that is to be used by the new NMI
109	handler. In this case, the rcu_dereference_sched() would	112	handler. In this case, the rcu_dereference_sched() would
110	be needed, because otherwise a CPU that received an NMI	113	be needed, because otherwise a CPU that received an NMI
111	just after the new handler was set might see the pointer	114	just after the new handler was set might see the pointer
112	to the new NMI handler, but the old pre-initialized	115	to the new NMI handler, but the old pre-initialized
113	version of the handler's data.	116	version of the handler's data.
114		117
115	This same sad story can happen on other CPUs when using	118	This same sad story can happen on other CPUs when using
116	a compiler with aggressive pointer-value speculation	119	a compiler with aggressive pointer-value speculation
117	optimizations.	120	optimizations.
118		121
119	More important, the rcu_dereference_sched() makes it	122	More important, the rcu_dereference_sched() makes it
120	clear to someone reading the code that the pointer is	123	clear to someone reading the code that the pointer is
121	being protected by RCU-sched.	124	being protected by RCU-sched.

Documentation/RCU/arrayRCU.rstrenamed from Documentation/RCU/arrayRCU.txt+23−11

-Using RCU to Protect Read-Mostly Arrays
+.. _array_rcu_doc:
+Using RCU to Protect Read-Mostly Arrays
+=======================================
 Although RCU is more commonly used to protect linked lists, it can
 also be used to protect arrays.  Three situations are as follows:
-1.  Hash Tables
+1.  :ref:`Hash Tables <hash_tables>`
-2.  Static Arrays
+2.  :ref:`Static Arrays <static_arrays>`
-3.  Resizeable Arrays
+3.  :ref:`Resizable Arrays <resizable_arrays>`
 Each of these three situations involves an RCU-protected pointer to an
 array that is separately indexed.  It might be tempting to consider use
 of RCU to instead protect the index into an array, however, this use
-case is -not- supported.  The problem with RCU-protected indexes into
+case is **not** supported.  The problem with RCU-protected indexes into
 arrays is that compilers can play way too many optimization games with
 integers, which means that the rules governing handling of these indexes
 are far more trouble than they are worth.  If RCU-protected indexes into
 That aside, each of the three RCU-protected pointer situations are
 described in the following sections.
+.. _hash_tables:
 Situation 1: Hash Tables
+------------------------
 Hash tables are often implemented as an array, where each array entry
 has a linked-list hash chain.  Each hash chain can be protected by RCU
 as described in the listRCU.txt document.  This approach also applies
 to other array-of-list situations, such as radix trees.
+.. _static_arrays:
 Situation 2: Static Arrays
+--------------------------
 Static arrays, where the data (rather than a pointer to the data) is
 located in each array element, and where the array is never resized,
 this situation, which would also have minimal read-side overhead as long
 as updates are rare.
-Quick Quiz:  Why is it so important that updates be rare when
+Quick Quiz:
-	     using seqlock?
+		Why is it so important that updates be rare when using seqlock?
+:ref:`Answer to Quick Quiz <answer_quick_quiz_seqlock>`
+.. _resizable_arrays:
-Situation 3: Resizeable Arrays
+Situation 3: Resizable Arrays
+------------------------------
-Use of RCU for resizeable arrays is demonstrated by the grow_ary()
+Use of RCU for resizable arrays is demonstrated by the grow_ary()
 function formerly used by the System V IPC code.  The array is used
 to map from semaphore, message-queue, and shared-memory IDs to the data
 structure that represents the corresponding IPC construct.  The grow_ary()
 the new array, and invokes ipc_rcu_putref() to free up the old array.
 Note that rcu_assign_pointer() is used to update the ids->entries pointer,
 which includes any memory barriers required on whatever architecture
-you are running on.
+you are running on::
 	static int grow_ary(struct ipc_ids* ids, int newsize)
 	{
 to the desired IPC object is placed in "out", with NULL indicating
 a non-existent entry.  After acquiring "out->lock", the "out->deleted"
 flag indicates whether the IPC object is in the process of being
-deleted, and, if not, the pointer is returned.
+deleted, and, if not, the pointer is returned::
 	struct kern_ipc_perm* ipc_lock(struct ipc_ids* ids, int id)
 	{
 		return out;
 	}
+.. _answer_quick_quiz_seqlock:
 Answer to Quick Quiz:
+	Why is it so important that updates be rare when using seqlock?
 	The reason that it is important that updates be rare when
 	using seqlock is that frequent updates can livelock readers.

Documentation/RCU/index.rst+5−0

 .. toctree::
    :maxdepth: 3
+   arrayRCU
+   rcubarrier
+   rcu_dereference
+   whatisRCU
    rcu
    listRCU
+   NMI-RCU
    UP
    Design/Memory-Ordering/Tree-RCU-Memory-Ordering

Documentation/RCU/lockdep-splat.txt+1−1

@@ -99,7 +99,7 @@ With this change, the rcu_dereference() is always within an RCU · 98 unmodified lines
99	read-side critical section, which again would have suppressed the	99	read-side critical section, which again would have suppressed the
100	above lockdep-RCU splat.	100	above lockdep-RCU splat.
101		101
102	But in this particular case, we don't actually deference the pointer	102	But in this particular case, we don't actually dereference the pointer
103	returned from rcu_dereference(). Instead, that pointer is just compared	103	returned from rcu_dereference(). Instead, that pointer is just compared
104	to the cic pointer, which means that the rcu_dereference() can be replaced	104	to the cic pointer, which means that the rcu_dereference() can be replaced
105	by rcu_access_pointer() as follows:	105	by rcu_access_pointer() as follows:

Documentation/RCU/rcu_dereference.rstrenamed from Documentation/RCU/rcu_dereference.txt+41−34

+.. _rcu_dereference_doc:
 PROPER CARE AND FEEDING OF RETURN VALUES FROM rcu_dereference()
+===============================================================
 Most of the time, you can use values from rcu_dereference() or one of
 the similar primitives without worries.  Dereferencing (prefix "*"),
 It is nevertheless possible to get into trouble with other operations.
 Follow these rules to keep your RCU code working properly:
-o	You must use one of the rcu_dereference() family of primitives
+-	You must use one of the rcu_dereference() family of primitives
 	to load an RCU-protected pointer, otherwise CONFIG_PROVE_RCU
 	will complain.  Worse yet, your code can see random memory-corruption
 	bugs due to games that compilers and DEC Alpha can play.
 	for an example where the compiler can in fact deduce the exact
 	value of the pointer, and thus cause misordering.
-o	You are only permitted to use rcu_dereference on pointer values.
+-	You are only permitted to use rcu_dereference on pointer values.
 	The compiler simply knows too much about integral values to
 	trust it to carry dependencies through integer operations.
 	There are a very few exceptions, namely that you can temporarily
 	cast the pointer to uintptr_t in order to:
-	o	Set bits and clear bits down in the must-be-zero low-order
+	-	Set bits and clear bits down in the must-be-zero low-order
 		bits of that pointer.  This clearly means that the pointer
 		must have alignment constraints, for example, this does
 		-not- work in general for char* pointers.
-	o	XOR bits to translate pointers, as is done in some
+	-	XOR bits to translate pointers, as is done in some
 		classic buddy-allocator algorithms.
 	It is important to cast the value back to pointer before
 	doing much of anything else with it.
-o	Avoid cancellation when using the "+" and "-" infix arithmetic
+-	Avoid cancellation when using the "+" and "-" infix arithmetic
 	operators.  For example, for a given variable "x", avoid
 	"(x-(uintptr_t)x)" for char* pointers.	The compiler is within its
 	rights to substitute zero for this sort of expression, so that
 	"p+a-b" is safe because its value still necessarily depends on
 	the rcu_dereference(), thus maintaining proper ordering.
-o	If you are using RCU to protect JITed functions, so that the
+-	If you are using RCU to protect JITed functions, so that the
 	"()" function-invocation operator is applied to a value obtained
 	(directly or indirectly) from rcu_dereference(), you may need to
 	interact directly with the hardware to flush instruction caches.
 	This issue arises on some systems when a newly JITed function is
 	using the same memory that was used by an earlier JITed function.
-o	Do not use the results from relational operators ("==", "!=",
+-	Do not use the results from relational operators ("==", "!=",
 	">", ">=", "<", or "<=") when dereferencing.  For example,
-	the following (quite strange) code is buggy:
+	the following (quite strange) code is buggy::
 		int *p;
 		int *q;
 	after such branches, but can speculate loads, which can again
 	result in misordering bugs.
-o	Be very careful about comparing pointers obtained from
+-	Be very careful about comparing pointers obtained from
 	rcu_dereference() against non-NULL values.  As Linus Torvalds
 	explained, if the two pointers are equal, the compiler could
 	substitute the pointer you are comparing against for the pointer
-	obtained from rcu_dereference().  For example:
+	obtained from rcu_dereference().  For example::
 		p = rcu_dereference(gp);
 		if (p == &default_struct)
 	Because the compiler now knows that the value of "p" is exactly
 	the address of the variable "default_struct", it is free to
-	transform this code into the following:
+	transform this code into the following::
 		p = rcu_dereference(gp);
 		if (p == &default_struct)
 	However, comparisons are OK in the following cases:
-	o	The comparison was against the NULL pointer.  If the
+	-	The comparison was against the NULL pointer.  If the
 		compiler knows that the pointer is NULL, you had better
 		not be dereferencing it anyway.  If the comparison is
 		non-equal, the compiler is none the wiser.  Therefore,
 		it is safe to compare pointers from rcu_dereference()
 		against NULL pointers.
-	o	The pointer is never dereferenced after being compared.
+	-	The pointer is never dereferenced after being compared.
 		Since there are no subsequent dereferences, the compiler
 		cannot use anything it learned from the comparison
 		to reorder the non-existent subsequent dereferences.
 		dereferenced, rcu_access_pointer() should be used in place
 		of rcu_dereference().
-	o	The comparison is against a pointer that references memory
+	-	The comparison is against a pointer that references memory
 		that was initialized "a long time ago."  The reason
 		this is safe is that even if misordering occurs, the
 		misordering will not affect the accesses that follow
 		the comparison.  So exactly how long ago is "a long
 		time ago"?  Here are some possibilities:
-		o	Compile time.
+		-	Compile time.
-		o	Boot time.
+		-	Boot time.
-		o	Module-init time for module code.
+		-	Module-init time for module code.
-		o	Prior to kthread creation for kthread code.
+		-	Prior to kthread creation for kthread code.
-		o	During some prior acquisition of the lock that
+		-	During some prior acquisition of the lock that
 			we now hold.
-		o	Before mod_timer() time for a timer handler.
+		-	Before mod_timer() time for a timer handler.
 		There are many other possibilities involving the Linux
 		kernel's wide array of primitives that cause code to
 		be invoked at a later time.
-	o	The pointer being compared against also came from
+	-	The pointer being compared against also came from
 		rcu_dereference().  In this case, both pointers depend
 		on one rcu_dereference() or another, so you get proper
 		ordering either way.
 		of such an RCU usage bug is shown in the section titled
 		"EXAMPLE OF AMPLIFIED RCU-USAGE BUG".
-	o	All of the accesses following the comparison are stores,
+	-	All of the accesses following the comparison are stores,
 		so that a control dependency preserves the needed ordering.
 		That said, it is easy to get control dependencies wrong.
 		Please see the "CONTROL DEPENDENCIES" section of
 		Documentation/memory-barriers.txt for more details.
-	o	The pointers are not equal -and- the compiler does
+	-	The pointers are not equal -and- the compiler does
 		not have enough information to deduce the value of the
 		pointer.  Note that the volatile cast in rcu_dereference()
 		will normally prevent the compiler from knowing too much.
 		comparison will provide exactly the information that the
 		compiler needs to deduce the value of the pointer.
-o	Disable any value-speculation optimizations that your compiler
+-	Disable any value-speculation optimizations that your compiler
 	might provide, especially if you are making use of feedback-based
 	optimizations that take data collected from prior runs.  Such
 	value-speculation optimizations reorder operations by design.
 EXAMPLE OF AMPLIFIED RCU-USAGE BUG
+----------------------------------
 Because updaters can run concurrently with RCU readers, RCU readers can
 see stale and/or inconsistent values.  If RCU readers need fresh or
 consistent values, which they sometimes do, they need to take proper
-precautions.  To see this, consider the following code fragment:
+precautions.  To see this, consider the following code fragment::
 	struct foo {
 		int a;
 But suppose that the reader needs a consistent view?
-Then one approach is to use locking, for example, as follows:
+Then one approach is to use locking, for example, as follows::
 	struct foo {
 		int a;
 EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH
+-----------------------------------------
 If a pointer obtained from rcu_dereference() compares not-equal to some
 other pointer, the compiler normally has no clue what the value of the
 should prevent the compiler from guessing the value.
 But without rcu_dereference(), the compiler knows more than you might
-expect.  Consider the following code fragment:
+expect.  Consider the following code fragment::
 	struct foo {
 		int a;
 WHICH MEMBER OF THE rcu_dereference() FAMILY SHOULD YOU USE?
+------------------------------------------------------------
 First, please avoid using rcu_dereference_raw() and also please avoid
 using rcu_dereference_check() and rcu_dereference_protected() with a
 2.	If the access might be within an RCU read-side critical section
 	on the one hand, or protected by (say) my_lock on the other,
-	use rcu_dereference_check(), for example:
+	use rcu_dereference_check(), for example::
 		p1 = rcu_dereference_check(p->rcu_protected_pointer,
 					   lockdep_is_held(&my_lock));
 3.	If the access might be within an RCU read-side critical section
 	on the one hand, or protected by either my_lock or your_lock on
-	the other, again use rcu_dereference_check(), for example:
+	the other, again use rcu_dereference_check(), for example::
 		p1 = rcu_dereference_check(p->rcu_protected_pointer,
 					   lockdep_is_held(&my_lock) ||
 					   lockdep_is_held(&your_lock));
 4.	If the access is on the update side, so that it is always protected
-	by my_lock, use rcu_dereference_protected():
+	by my_lock, use rcu_dereference_protected()::
 		p1 = rcu_dereference_protected(p->rcu_protected_pointer,
 					       lockdep_is_held(&my_lock));
 SPARSE CHECKING OF RCU-PROTECTED POINTERS
+-----------------------------------------
 The sparse static-analysis tool checks for direct access to RCU-protected
 pointers, which can result in "interesting" bugs due to compiler
 optimizations involving invented loads and perhaps also load tearing.
-For example, suppose someone mistakenly does something like this:
+For example, suppose someone mistakenly does something like this::
 	p = q->rcu_protected_pointer;
 	do_something_with(p->a);
 	do_something_else_with(p->b);
 If register pressure is high, the compiler might optimize "p" out
-of existence, transforming the code to something like this:
+of existence, transforming the code to something like this::
 	do_something_with(q->rcu_protected_pointer->a);
 	do_something_else_with(q->rcu_protected_pointer->b);
 of pointers, which also might fatally disappoint your code.
 These problems could have been avoided simply by making the code instead
-read as follows:
+read as follows::
 	p = rcu_dereference(q->rcu_protected_pointer);
 	do_something_with(p->a);
 this pointer is accessed directly.  It will also cause sparse to complain
 if a pointer not marked with "__rcu" is accessed using rcu_dereference()
 and friends.  For example, ->rcu_protected_pointer might be declared as
-follows:
+follows::
 	struct foo __rcu *rcu_protected_pointer;

Documentation/RCU/rcubarrier.rstrenamed from Documentation/RCU/rcubarrier.txt+125−97

@@ -1,4 +1,7 @@
		1	.. _rcu_barrier:
		2
1	RCU and Unloadable Modules	3	RCU and Unloadable Modules
		4	==========================
2		5
3	[Originally published in LWN Jan. 14, 2007: http://lwn.net/Articles/217484/]	6	[Originally published in LWN Jan. 14, 2007: http://lwn.net/Articles/217484/]
4		7
@@ -21,7 +24,7 @@ given that readers might well leave absolutely no trace of their · 16 unmodified lines
21	presence? There is a synchronize_rcu() primitive that blocks until all	24	presence? There is a synchronize_rcu() primitive that blocks until all
22	pre-existing readers have completed. An updater wishing to delete an	25	pre-existing readers have completed. An updater wishing to delete an
23	element p from a linked list might do the following, while holding an	26	element p from a linked list might do the following, while holding an
24	appropriate lock, of course:	27	appropriate lock, of course::
25		28
26	list_del_rcu(p);	29	list_del_rcu(p);
27	synchronize_rcu();	30	synchronize_rcu();
@@ -32,13 +35,13 @@ primitive must be used instead. This primitive takes a pointer to an · 4 unmodified lines
32	rcu_head struct placed within the RCU-protected data structure and	35	rcu_head struct placed within the RCU-protected data structure and
33	another pointer to a function that may be invoked later to free that	36	another pointer to a function that may be invoked later to free that
34	structure. Code to delete an element p from the linked list from IRQ	37	structure. Code to delete an element p from the linked list from IRQ
35	context might then be as follows:	38	context might then be as follows::
36		39
37	list_del_rcu(p);	40	list_del_rcu(p);
38	call_rcu(&p->rcu, p_callback);	41	call_rcu(&p->rcu, p_callback);
39		42
40	Since call_rcu() never blocks, this code can safely be used from within	43	Since call_rcu() never blocks, this code can safely be used from within
41	IRQ context. The function p_callback() might be defined as follows:	44	IRQ context. The function p_callback() might be defined as follows::
42		45
43	static void p_callback(struct rcu_head *rp)	46	static void p_callback(struct rcu_head *rp)
44	{	47	{
@@ -49,6 +52,7 @@ IRQ context. The function p_callback() might be defined as follows: · 4 unmodified lines
49		52
50		53
51	Unloading Modules That Use call_rcu()	54	Unloading Modules That Use call_rcu()
		55	-------------------------------------
52		56
53	But what if p_callback is defined in an unloadable module?	57	But what if p_callback is defined in an unloadable module?
54		58
@@ -69,10 +73,11 @@ in realtime kernels in order to avoid excessive scheduling latencies. · 14 unmodified lines
69		73
70		74
71	rcu_barrier()	75	rcu_barrier()
		76	-------------
72		77
73	We instead need the rcu_barrier() primitive. Rather than waiting for	78	We instead need the rcu_barrier() primitive. Rather than waiting for
74	a grace period to elapse, rcu_barrier() waits for all outstanding RCU	79	a grace period to elapse, rcu_barrier() waits for all outstanding RCU
75	callbacks to complete. Please note that rcu_barrier() does -not- imply	80	callbacks to complete. Please note that rcu_barrier() does not imply
76	synchronize_rcu(), in particular, if there are no RCU callbacks queued	81	synchronize_rcu(), in particular, if there are no RCU callbacks queued
77	anywhere, rcu_barrier() is within its rights to return immediately,	82	anywhere, rcu_barrier() is within its rights to return immediately,
78	without waiting for a grace period to elapse.	83	without waiting for a grace period to elapse.
@@ -88,79 +93,79 @@ must match the flavor of rcu_barrier() with that of call_rcu(). If your · 9 unmodified lines
88	module uses multiple flavors of call_rcu(), then it must also use multiple	93	module uses multiple flavors of call_rcu(), then it must also use multiple
89	flavors of rcu_barrier() when unloading that module. For example, if	94	flavors of rcu_barrier() when unloading that module. For example, if
90	it uses call_rcu(), call_srcu() on srcu_struct_1, and call_srcu() on	95	it uses call_rcu(), call_srcu() on srcu_struct_1, and call_srcu() on
91	srcu_struct_2(), then the following three lines of code will be required	96	srcu_struct_2, then the following three lines of code will be required
92	when unloading:	97	when unloading::
93		98
94	1 rcu_barrier();	99	1 rcu_barrier();
95	2 srcu_barrier(&srcu_struct_1);	100	2 srcu_barrier(&srcu_struct_1);
96	3 srcu_barrier(&srcu_struct_2);	101	3 srcu_barrier(&srcu_struct_2);
97		102
98	The rcutorture module makes use of rcu_barrier() in its exit function	103	The rcutorture module makes use of rcu_barrier() in its exit function
99	as follows:	104	as follows::
100		105
101	1 static void	106	1 static void
102	2 rcu_torture_cleanup(void)	107	2 rcu_torture_cleanup(void)
103	3 {	108	3 {
104	4 int i;	109	4 int i;
105	5	110	5
106	6 fullstop = 1;	111	6 fullstop = 1;
107	7 if (shuffler_task != NULL) {	112	7 if (shuffler_task != NULL) {
108	8 VERBOSE_PRINTK_STRING("Stopping rcu_torture_shuffle task");	113	8 VERBOSE_PRINTK_STRING("Stopping rcu_torture_shuffle task");
109	9 kthread_stop(shuffler_task);	114	9 kthread_stop(shuffler_task);
110	10 }	115	10 }
111	11 shuffler_task = NULL;	116	11 shuffler_task = NULL;
112	12	117	12
113	13 if (writer_task != NULL) {	118	13 if (writer_task != NULL) {
114	14 VERBOSE_PRINTK_STRING("Stopping rcu_torture_writer task");	119	14 VERBOSE_PRINTK_STRING("Stopping rcu_torture_writer task");
115	15 kthread_stop(writer_task);	120	15 kthread_stop(writer_task);
116	16 }	121	16 }
117	17 writer_task = NULL;	122	17 writer_task = NULL;
118	18	123	18
119	19 if (reader_tasks != NULL) {	124	19 if (reader_tasks != NULL) {
120	20 for (i = 0; i < nrealreaders; i++) {	125	20 for (i = 0; i < nrealreaders; i++) {
121	21 if (reader_tasks[i] != NULL) {	126	21 if (reader_tasks[i] != NULL) {
122	22 VERBOSE_PRINTK_STRING(	127	22 VERBOSE_PRINTK_STRING(
123	23 "Stopping rcu_torture_reader task");	128	23 "Stopping rcu_torture_reader task");
124	24 kthread_stop(reader_tasks[i]);	129	24 kthread_stop(reader_tasks[i]);
125	25 }	130	25 }
126	26 reader_tasks[i] = NULL;	131	26 reader_tasks[i] = NULL;
127	27 }	132	27 }
128	28 kfree(reader_tasks);	133	28 kfree(reader_tasks);
129	29 reader_tasks = NULL;	134	29 reader_tasks = NULL;
130	30 }	135	30 }
131	31 rcu_torture_current = NULL;	136	31 rcu_torture_current = NULL;
132	32	137	32
133	33 if (fakewriter_tasks != NULL) {	138	33 if (fakewriter_tasks != NULL) {
134	34 for (i = 0; i < nfakewriters; i++) {	139	34 for (i = 0; i < nfakewriters; i++) {
135	35 if (fakewriter_tasks[i] != NULL) {	140	35 if (fakewriter_tasks[i] != NULL) {
136	36 VERBOSE_PRINTK_STRING(	141	36 VERBOSE_PRINTK_STRING(
137	37 "Stopping rcu_torture_fakewriter task");	142	37 "Stopping rcu_torture_fakewriter task");
138	38 kthread_stop(fakewriter_tasks[i]);	143	38 kthread_stop(fakewriter_tasks[i]);
139	39 }	144	39 }
140	40 fakewriter_tasks[i] = NULL;	145	40 fakewriter_tasks[i] = NULL;
141	41 }	146	41 }
142	42 kfree(fakewriter_tasks);	147	42 kfree(fakewriter_tasks);
143	43 fakewriter_tasks = NULL;	148	43 fakewriter_tasks = NULL;
144	44 }	149	44 }
145	45	150	45
146	46 if (stats_task != NULL) {	151	46 if (stats_task != NULL) {
147	47 VERBOSE_PRINTK_STRING("Stopping rcu_torture_stats task");	152	47 VERBOSE_PRINTK_STRING("Stopping rcu_torture_stats task");
148	48 kthread_stop(stats_task);	153	48 kthread_stop(stats_task);
149	49 }	154	49 }
150	50 stats_task = NULL;	155	50 stats_task = NULL;
151	51	156	51
152	52 /* Wait for all RCU callbacks to fire. */	157	52 /* Wait for all RCU callbacks to fire. */
153	53 rcu_barrier();	158	53 rcu_barrier();
154	54	159	54
155	55 rcu_torture_stats_print(); /* -After- the stats thread is stopped! */	160	55 rcu_torture_stats_print(); /* -After- the stats thread is stopped! */
156	56	161	56
157	57 if (cur_ops->cleanup != NULL)	162	57 if (cur_ops->cleanup != NULL)
158	58 cur_ops->cleanup();	163	58 cur_ops->cleanup();
159	59 if (atomic_read(&n_rcu_torture_error))	164	59 if (atomic_read(&n_rcu_torture_error))
160	60 rcu_torture_print_module_parms("End of test: FAILURE");	165	60 rcu_torture_print_module_parms("End of test: FAILURE");
161	61 else	166	61 else
162	62 rcu_torture_print_module_parms("End of test: SUCCESS");	167	62 rcu_torture_print_module_parms("End of test: SUCCESS");
163	63 }	168	63 }
164		169
165	Line 6 sets a global variable that prevents any RCU callbacks from	170	Line 6 sets a global variable that prevents any RCU callbacks from
166	re-posting themselves. This will not be necessary in most cases, since	171	re-posting themselves. This will not be necessary in most cases, since
@@ -176,9 +181,14 @@ for any pre-existing callbacks to complete. · 9 unmodified lines
176	Then lines 55-62 print status and do operation-specific cleanup, and	181	Then lines 55-62 print status and do operation-specific cleanup, and
177	then return, permitting the module-unload operation to be completed.	182	then return, permitting the module-unload operation to be completed.
178		183
179	Quick Quiz #1: Is there any other situation where rcu_barrier() might	184	.. _rcubarrier_quiz_1:
		185
		186	Quick Quiz #1:
		187	Is there any other situation where rcu_barrier() might
180	be required?	188	be required?
181		189
		190	:ref:`Answer to Quick Quiz #1 <answer_rcubarrier_quiz_1>`
		191
182	Your module might have additional complications. For example, if your	192	Your module might have additional complications. For example, if your
183	module invokes call_rcu() from timers, you will need to first cancel all	193	module invokes call_rcu() from timers, you will need to first cancel all
184	the timers, and only then invoke rcu_barrier() to wait for any remaining	194	the timers, and only then invoke rcu_barrier() to wait for any remaining
@@ -188,11 +198,12 @@ Of course, if you module uses call_rcu(), you will need to invoke · 3 unmodified lines
188	rcu_barrier() before unloading. Similarly, if your module uses	198	rcu_barrier() before unloading. Similarly, if your module uses
189	call_srcu(), you will need to invoke srcu_barrier() before unloading,	199	call_srcu(), you will need to invoke srcu_barrier() before unloading,
190	and on the same srcu_struct structure. If your module uses call_rcu()	200	and on the same srcu_struct structure. If your module uses call_rcu()
191	-and- call_srcu(), then you will need to invoke rcu_barrier() -and-	201	and call_srcu(), then you will need to invoke rcu_barrier() and
192	srcu_barrier().	202	srcu_barrier().
193		203
194		204
195	Implementing rcu_barrier()	205	Implementing rcu_barrier()
		206	--------------------------
196		207
197	Dipankar Sarma's implementation of rcu_barrier() makes use of the fact	208	Dipankar Sarma's implementation of rcu_barrier() makes use of the fact
198	that RCU callbacks are never reordered once queued on one of the per-CPU	209	that RCU callbacks are never reordered once queued on one of the per-CPU
@@ -200,19 +211,19 @@ queues. His implementation queues an RCU callback on each of the per-CPU · 1 unmodified line
200	callback queues, and then waits until they have all started executing, at	211	callback queues, and then waits until they have all started executing, at
201	which point, all earlier RCU callbacks are guaranteed to have completed.	212	which point, all earlier RCU callbacks are guaranteed to have completed.
202		213
203	The original code for rcu_barrier() was as follows:	214	The original code for rcu_barrier() was as follows::
204		215
205	1 void rcu_barrier(void)	216	1 void rcu_barrier(void)
206	2 {	217	2 {
207	3 BUG_ON(in_interrupt());	218	3 BUG_ON(in_interrupt());
208	4 /* Take cpucontrol mutex to protect against CPU hotplug */	219	4 /* Take cpucontrol mutex to protect against CPU hotplug */
209	5 mutex_lock(&rcu_barrier_mutex);	220	5 mutex_lock(&rcu_barrier_mutex);
210	6 init_completion(&rcu_barrier_completion);	221	6 init_completion(&rcu_barrier_completion);
211	7 atomic_set(&rcu_barrier_cpu_count, 0);	222	7 atomic_set(&rcu_barrier_cpu_count, 0);
212	8 on_each_cpu(rcu_barrier_func, NULL, 0, 1);	223	8 on_each_cpu(rcu_barrier_func, NULL, 0, 1);
213	9 wait_for_completion(&rcu_barrier_completion);	224	9 wait_for_completion(&rcu_barrier_completion);
214	10 mutex_unlock(&rcu_barrier_mutex);	225	10 mutex_unlock(&rcu_barrier_mutex);
215	11 }	226	11 }
216		227
217	Line 3 verifies that the caller is in process context, and lines 5 and 10	228	Line 3 verifies that the caller is in process context, and lines 5 and 10
218	use rcu_barrier_mutex to ensure that only one rcu_barrier() is using the	229	use rcu_barrier_mutex to ensure that only one rcu_barrier() is using the
@@ -226,18 +237,18 @@ This code was rewritten in 2008 and several times thereafter, but this · 7 unmodified lines
226	still gives the general idea.	237	still gives the general idea.
227		238
228	The rcu_barrier_func() runs on each CPU, where it invokes call_rcu()	239	The rcu_barrier_func() runs on each CPU, where it invokes call_rcu()
229	to post an RCU callback, as follows:	240	to post an RCU callback, as follows::
230		241
231	1 static void rcu_barrier_func(void *notused)	242	1 static void rcu_barrier_func(void *notused)
232	2 {	243	2 {
233	3 int cpu = smp_processor_id();	244	3 int cpu = smp_processor_id();
234	4 struct rcu_data *rdp = &per_cpu(rcu_data, cpu);	245	4 struct rcu_data *rdp = &per_cpu(rcu_data, cpu);
235	5 struct rcu_head *head;	246	5 struct rcu_head *head;
236	6	247	6
237	7 head = &rdp->barrier;	248	7 head = &rdp->barrier;
238	8 atomic_inc(&rcu_barrier_cpu_count);	249	8 atomic_inc(&rcu_barrier_cpu_count);
239	9 call_rcu(head, rcu_barrier_callback);	250	9 call_rcu(head, rcu_barrier_callback);
240	10 }	251	10 }
241		252
242	Lines 3 and 4 locate RCU's internal per-CPU rcu_data structure,	253	Lines 3 and 4 locate RCU's internal per-CPU rcu_data structure,
243	which contains the struct rcu_head that needed for the later call to	254	which contains the struct rcu_head that needed for the later call to
@@ -248,20 +259,25 @@ the current CPU's queue. · 4 unmodified lines
248		259
249	The rcu_barrier_callback() function simply atomically decrements the	260	The rcu_barrier_callback() function simply atomically decrements the
250	rcu_barrier_cpu_count variable and finalizes the completion when it	261	rcu_barrier_cpu_count variable and finalizes the completion when it
251	reaches zero, as follows:	262	reaches zero, as follows::
252		263
253	1 static void rcu_barrier_callback(struct rcu_head *notused)	264	1 static void rcu_barrier_callback(struct rcu_head *notused)
254	2 {	265	2 {
255	3 if (atomic_dec_and_test(&rcu_barrier_cpu_count))	266	3 if (atomic_dec_and_test(&rcu_barrier_cpu_count))
256	4 complete(&rcu_barrier_completion);	267	4 complete(&rcu_barrier_completion);
257	5 }	268	5 }
258		269
259	Quick Quiz #2: What happens if CPU 0's rcu_barrier_func() executes	270	.. _rcubarrier_quiz_2:
		271
		272	Quick Quiz #2:
		273	What happens if CPU 0's rcu_barrier_func() executes
260	immediately (thus incrementing rcu_barrier_cpu_count to the	274	immediately (thus incrementing rcu_barrier_cpu_count to the
261	value one), but the other CPU's rcu_barrier_func() invocations	275	value one), but the other CPU's rcu_barrier_func() invocations
262	are delayed for a full grace period? Couldn't this result in	276	are delayed for a full grace period? Couldn't this result in
263	rcu_barrier() returning prematurely?	277	rcu_barrier() returning prematurely?
264		278
		279	:ref:`Answer to Quick Quiz #2 <answer_rcubarrier_quiz_2>`
		280
265	The current rcu_barrier() implementation is more complex, due to the need	281	The current rcu_barrier() implementation is more complex, due to the need
266	to avoid disturbing idle CPUs (especially on battery-powered systems)	282	to avoid disturbing idle CPUs (especially on battery-powered systems)
267	and the need to minimally disturb non-idle CPUs in real-time systems.	283	and the need to minimally disturb non-idle CPUs in real-time systems.
@@ -269,6 +285,7 @@ However, the code above illustrates the concepts. · 1 unmodified line
269		285
270		286
271	rcu_barrier() Summary	287	rcu_barrier() Summary
		288	---------------------
272		289
273	The rcu_barrier() primitive has seen relatively little use, since most	290	The rcu_barrier() primitive has seen relatively little use, since most
274	code using RCU is in the core kernel rather than in modules. However, if	291	code using RCU is in the core kernel rather than in modules. However, if
@@ -277,8 +294,12 @@ so that your module may be safely unloaded. · 2 unmodified lines
277		294
278		295
279	Answers to Quick Quizzes	296	Answers to Quick Quizzes
		297	------------------------
		298
		299	.. _answer_rcubarrier_quiz_1:
280		300
281	Quick Quiz #1: Is there any other situation where rcu_barrier() might	301	Quick Quiz #1:
		302	Is there any other situation where rcu_barrier() might
282	be required?	303	be required?
283		304
284	Answer: Interestingly enough, rcu_barrier() was not originally	305	Answer: Interestingly enough, rcu_barrier() was not originally
@@ -292,7 +313,12 @@ Answer: Interestingly enough, rcu_barrier() was not originally · 7 unmodified lines
292	implementing rcutorture, and found that rcu_barrier() solves	313	implementing rcutorture, and found that rcu_barrier() solves
293	this problem as well.	314	this problem as well.
294		315
295	Quick Quiz #2: What happens if CPU 0's rcu_barrier_func() executes	316	:ref:`Back to Quick Quiz #1 <rcubarrier_quiz_1>`
		317
		318	.. _answer_rcubarrier_quiz_2:
		319
		320	Quick Quiz #2:
		321	What happens if CPU 0's rcu_barrier_func() executes
296	immediately (thus incrementing rcu_barrier_cpu_count to the	322	immediately (thus incrementing rcu_barrier_cpu_count to the
297	value one), but the other CPU's rcu_barrier_func() invocations	323	value one), but the other CPU's rcu_barrier_func() invocations
298	are delayed for a full grace period? Couldn't this result in	324	are delayed for a full grace period? Couldn't this result in
@@ -323,3 +349,5 @@ Answer: This cannot happen. The reason is that on_each_cpu() has its last · 24 unmodified lines
323	is to add an rcu_read_lock() before line 8 of rcu_barrier()	349	is to add an rcu_read_lock() before line 8 of rcu_barrier()
324	and an rcu_read_unlock() after line 8 of this same function. If	350	and an rcu_read_unlock() after line 8 of this same function. If
325	you can think of a better change, please let me know!	351	you can think of a better change, please let me know!
		352
		353	:ref:`Back to Quick Quiz #2 <rcubarrier_quiz_2>`

Documentation/RCU/stallwarn.txt+3−8

 In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed
 for each CPU:
-	0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 Nonlazy posted: ..D
+	0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1
 The "last_accelerate:" prints the low-order 16 bits (in hex) of the
 jiffies counter when this CPU last invoked rcu_try_advance_all_cbs()
 from rcu_needs_cpu() or last invoked rcu_accelerate_cbs() from
-rcu_prepare_for_idle().  The "Nonlazy posted:" indicates lazy-callback
+rcu_prepare_for_idle(). "dyntick_enabled: 1" indicates that dyntick-idle
-status, so that an "l" indicates that all callbacks were lazy at the start
+processing is enabled.
-of the last idle period and an "L" indicates that there are currently
-no non-lazy callbacks (in both cases, "." is printed otherwise, as
-shown above) and "D" indicates that dyntick-idle processing is enabled
-("." is printed otherwise, for example, if disabled via the "nohz="
-kernel boot parameter).
 If the grace period ends just as the stall warning starts printing,
 there will be a spurious stall-warning message, which will include

Documentation/RCU/whatisRCU.rstrenamed from Documentation/RCU/whatisRCU.txt+183−108

@@ -1,15 +1,18 @@
		1	.. _whatisrcu_doc:
		2
1	What is RCU? -- "Read, Copy, Update"	3	What is RCU? -- "Read, Copy, Update"
		4	======================================
2		5
3	Please note that the "What is RCU?" LWN series is an excellent place	6	Please note that the "What is RCU?" LWN series is an excellent place
4	to start learning about RCU:	7	to start learning about RCU:
5		8
6	1. What is RCU, Fundamentally? http://lwn.net/Articles/262464/	9	\| 1. What is RCU, Fundamentally? http://lwn.net/Articles/262464/
7	2. What is RCU? Part 2: Usage http://lwn.net/Articles/263130/	10	\| 2. What is RCU? Part 2: Usage http://lwn.net/Articles/263130/
8	3. RCU part 3: the RCU API http://lwn.net/Articles/264090/	11	\| 3. RCU part 3: the RCU API http://lwn.net/Articles/264090/
9	4. The RCU API, 2010 Edition http://lwn.net/Articles/418853/	12	\| 4. The RCU API, 2010 Edition http://lwn.net/Articles/418853/
10	2010 Big API Table http://lwn.net/Articles/419086/	13	\| 2010 Big API Table http://lwn.net/Articles/419086/
11	5. The RCU API, 2014 Edition http://lwn.net/Articles/609904/	14	\| 5. The RCU API, 2014 Edition http://lwn.net/Articles/609904/
12	2014 Big API Table http://lwn.net/Articles/609973/	15	\| 2014 Big API Table http://lwn.net/Articles/609973/
13		16
14		17
15	What is RCU?	18	What is RCU?
@@ -24,14 +27,21 @@ the experience has been that different people must take different paths · 8 unmodified lines
24	to arrive at an understanding of RCU. This document provides several	27	to arrive at an understanding of RCU. This document provides several
25	different paths, as follows:	28	different paths, as follows:
26		29
27	1. RCU OVERVIEW	30	:ref:`1. RCU OVERVIEW <1_whatisRCU>`
28	2. WHAT IS RCU'S CORE API?	31
29	3. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?	32	:ref:`2. WHAT IS RCU'S CORE API? <2_whatisRCU>`
30	4. WHAT IF MY UPDATING THREAD CANNOT BLOCK?	33
31	5. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?	34	:ref:`3. WHAT ARE SOME EXAMPLE USES OF CORE RCU API? <3_whatisRCU>`
32	6. ANALOGY WITH READER-WRITER LOCKING	35
33	7. FULL LIST OF RCU APIs	36	:ref:`4. WHAT IF MY UPDATING THREAD CANNOT BLOCK? <4_whatisRCU>`
34	8. ANSWERS TO QUICK QUIZZES	37
		38	:ref:`5. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU? <5_whatisRCU>`
		39
		40	:ref:`6. ANALOGY WITH READER-WRITER LOCKING <6_whatisRCU>`
		41
		42	:ref:`7. FULL LIST OF RCU APIs <7_whatisRCU>`
		43
		44	:ref:`8. ANSWERS TO QUICK QUIZZES <8_whatisRCU>`
35		45
36	People who prefer starting with a conceptual overview should focus on	46	People who prefer starting with a conceptual overview should focus on
37	Section 1, though most readers will profit by reading this section at	47	Section 1, though most readers will profit by reading this section at
@@ -49,8 +59,10 @@ everything, feel free to read the whole thing -- but if you are really · 11 unmodified lines
49	that type of person, you have perused the source code and will therefore	59	that type of person, you have perused the source code and will therefore
50	never need this document anyway. ;-)	60	never need this document anyway. ;-)
51		61
		62	.. _1_whatisRCU:
52		63
53	1. RCU OVERVIEW	64	1. RCU OVERVIEW
		65	----------------
54		66
55	The basic idea behind RCU is to split updates into "removal" and	67	The basic idea behind RCU is to split updates into "removal" and
56	"reclamation" phases. The removal phase removes references to data items	68	"reclamation" phases. The removal phase removes references to data items
@@ -116,8 +128,10 @@ So how the heck can a reclaimer tell when a reader is done, given · 59 unmodified lines
116	that readers are not doing any sort of synchronization operations???	128	that readers are not doing any sort of synchronization operations???
117	Read on to learn about how RCU's API makes this easy.	129	Read on to learn about how RCU's API makes this easy.
118		130
		131	.. _2_whatisRCU:
119		132
120	2. WHAT IS RCU'S CORE API?	133	2. WHAT IS RCU'S CORE API?
		134	---------------------------
121		135
122	The core RCU API is quite small:	136	The core RCU API is quite small:
123		137
@@ -136,7 +150,7 @@ later. See the kernel docbook documentation for more info, or look directly · 12 unmodified lines
136	at the function header comments.	150	at the function header comments.
137		151
138	rcu_read_lock()	152	rcu_read_lock()
139		153	^^^^^^^^^^^^^^^
140	void rcu_read_lock(void);	154	void rcu_read_lock(void);
141		155
142	Used by a reader to inform the reclaimer that the reader is	156	Used by a reader to inform the reclaimer that the reader is
@@ -150,7 +164,7 @@ rcu_read_lock() · 7 unmodified lines
150	longer-term references to data structures.	164	longer-term references to data structures.
151		165
152	rcu_read_unlock()	166	rcu_read_unlock()
153		167	^^^^^^^^^^^^^^^^^
154	void rcu_read_unlock(void);	168	void rcu_read_unlock(void);
155		169
156	Used by a reader to inform the reclaimer that the reader is	170	Used by a reader to inform the reclaimer that the reader is
@@ -158,15 +172,15 @@ rcu_read_unlock() · 1 unmodified line
158	read-side critical sections may be nested and/or overlapping.	172	read-side critical sections may be nested and/or overlapping.
159		173
160	synchronize_rcu()	174	synchronize_rcu()
161		175	^^^^^^^^^^^^^^^^^
162	void synchronize_rcu(void);	176	void synchronize_rcu(void);
163		177
164	Marks the end of updater code and the beginning of reclaimer	178	Marks the end of updater code and the beginning of reclaimer
165	code. It does this by blocking until all pre-existing RCU	179	code. It does this by blocking until all pre-existing RCU
166	read-side critical sections on all CPUs have completed.	180	read-side critical sections on all CPUs have completed.
167	Note that synchronize_rcu() will -not- necessarily wait for	181	Note that synchronize_rcu() will not necessarily wait for
168	any subsequent RCU read-side critical sections to complete.	182	any subsequent RCU read-side critical sections to complete.
169	For example, consider the following sequence of events:	183	For example, consider the following sequence of events::
170		184
171	CPU 0 CPU 1 CPU 2	185	CPU 0 CPU 1 CPU 2
172	----------------- ------------------------- ---------------	186	----------------- ------------------------- ---------------
@@ -182,7 +196,7 @@ synchronize_rcu() · 9 unmodified lines
182	any that begin after synchronize_rcu() is invoked.	196	any that begin after synchronize_rcu() is invoked.
183		197
184	Of course, synchronize_rcu() does not necessarily return	198	Of course, synchronize_rcu() does not necessarily return
185	-immediately- after the last pre-existing RCU read-side critical	199	immediately after the last pre-existing RCU read-side critical
186	section completes. For one thing, there might well be scheduling	200	section completes. For one thing, there might well be scheduling
187	delays. For another thing, many RCU implementations process	201	delays. For another thing, many RCU implementations process
188	requests in batches in order to improve efficiencies, which can	202	requests in batches in order to improve efficiencies, which can
@@ -211,10 +225,10 @@ synchronize_rcu() · 22 unmodified lines
211	checklist.txt for some approaches to limiting the update rate.	225	checklist.txt for some approaches to limiting the update rate.
212		226
213	rcu_assign_pointer()	227	rcu_assign_pointer()
214		228	^^^^^^^^^^^^^^^^^^^^
215	void rcu_assign_pointer(p, typeof(p) v);	229	void rcu_assign_pointer(p, typeof(p) v);
216		230
217	Yes, rcu_assign_pointer() -is- implemented as a macro, though it	231	Yes, rcu_assign_pointer() is implemented as a macro, though it
218	would be cool to be able to declare a function in this manner.	232	would be cool to be able to declare a function in this manner.
219	(Compiler experts will no doubt disagree.)	233	(Compiler experts will no doubt disagree.)
220		234
@@ -231,7 +245,7 @@ rcu_assign_pointer() · 10 unmodified lines
231	the _rcu list-manipulation primitives such as list_add_rcu().	245	the _rcu list-manipulation primitives such as list_add_rcu().
232		246
233	rcu_dereference()	247	rcu_dereference()
234		248	^^^^^^^^^^^^^^^^^
235	typeof(p) rcu_dereference(p);	249	typeof(p) rcu_dereference(p);
236		250
237	Like rcu_assign_pointer(), rcu_dereference() must be implemented	251	Like rcu_assign_pointer(), rcu_dereference() must be implemented
@@ -248,13 +262,13 @@ rcu_dereference() · 10 unmodified lines
248		262
249	Common coding practice uses rcu_dereference() to copy an	263	Common coding practice uses rcu_dereference() to copy an
250	RCU-protected pointer to a local variable, then dereferences	264	RCU-protected pointer to a local variable, then dereferences
251	this local variable, for example as follows:	265	this local variable, for example as follows::
252		266
253	p = rcu_dereference(head.next);	267	p = rcu_dereference(head.next);
254	return p->data;	268	return p->data;
255		269
256	However, in this case, one could just as easily combine these	270	However, in this case, one could just as easily combine these
257	into one statement:	271	into one statement::
258		272
259	return rcu_dereference(head.next)->data;	273	return rcu_dereference(head.next)->data;
260		274
@@ -266,8 +280,8 @@ rcu_dereference() · 5 unmodified lines
266	unnecessary overhead on Alpha CPUs.	280	unnecessary overhead on Alpha CPUs.
267		281
268	Note that the value returned by rcu_dereference() is valid	282	Note that the value returned by rcu_dereference() is valid
269	only within the enclosing RCU read-side critical section [1].	283	only within the enclosing RCU read-side critical section [1]_.
270	For example, the following is -not- legal:	284	For example, the following is not legal::
271		285
272	rcu_read_lock();	286	rcu_read_lock();
273	p = rcu_dereference(head.next);	287	p = rcu_dereference(head.next);
@@ -290,9 +304,9 @@ rcu_dereference() · 16 unmodified lines
290	at any time, including immediately after the rcu_dereference().	304	at any time, including immediately after the rcu_dereference().
291	And, again like rcu_assign_pointer(), rcu_dereference() is	305	And, again like rcu_assign_pointer(), rcu_dereference() is
292	typically used indirectly, via the _rcu list-manipulation	306	typically used indirectly, via the _rcu list-manipulation
293	primitives, such as list_for_each_entry_rcu() [2].	307	primitives, such as list_for_each_entry_rcu() [2]_.
294		308
295	[1] The variant rcu_dereference_protected() can be used outside	309	.. [1] The variant rcu_dereference_protected() can be used outside
296	of an RCU read-side critical section as long as the usage is	310	of an RCU read-side critical section as long as the usage is
297	protected by locks acquired by the update-side code. This variant	311	protected by locks acquired by the update-side code. This variant
298	avoids the lockdep warning that would happen when using (for	312	avoids the lockdep warning that would happen when using (for
@@ -305,7 +319,7 @@ rcu_dereference() · 6 unmodified lines
305	a lockdep splat is emitted. See Documentation/RCU/Design/Requirements/Requirements.rst	319	a lockdep splat is emitted. See Documentation/RCU/Design/Requirements/Requirements.rst
306	and the API's code comments for more details and example usage.	320	and the API's code comments for more details and example usage.
307		321
308	[2] If the list_for_each_entry_rcu() instance might be used by	322	.. [2] If the list_for_each_entry_rcu() instance might be used by
309	update-side code as well as by RCU readers, then an additional	323	update-side code as well as by RCU readers, then an additional
310	lockdep expression can be added to its list of arguments.	324	lockdep expression can be added to its list of arguments.
311	For example, given an additional "lock_is_held(&mylock)" argument,	325	For example, given an additional "lock_is_held(&mylock)" argument,
@@ -315,6 +329,7 @@ rcu_dereference() · 3 unmodified lines
315		329
316	The following diagram shows how each API communicates among the	330	The following diagram shows how each API communicates among the
317	reader, updater, and reclaimer.	331	reader, updater, and reclaimer.
		332	::
318		333
319		334
320	rcu_assign_pointer()	335	rcu_assign_pointer()
@@ -375,12 +390,16 @@ c. RCU applied to scheduler and interrupt/NMI-handler tasks. · 54 unmodified lines
375	Again, most uses will be of (a). The (b) and (c) cases are important	390	Again, most uses will be of (a). The (b) and (c) cases are important
376	for specialized uses, but are relatively uncommon.	391	for specialized uses, but are relatively uncommon.
377		392
		393	.. _3_whatisRCU:
378		394
379	3. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?	395	3. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?
		396	-----------------------------------------------
380		397
381	This section shows a simple use of the core RCU API to protect a	398	This section shows a simple use of the core RCU API to protect a
382	global pointer to a dynamically allocated structure. More-typical	399	global pointer to a dynamically allocated structure. More-typical
383	uses of RCU may be found in listRCU.txt, arrayRCU.txt, and NMI-RCU.txt.	400	uses of RCU may be found in :ref:`listRCU.rst <list_rcu_doc>`,
		401	:ref:`arrayRCU.rst <array_rcu_doc>`, and :ref:`NMI-RCU.rst <NMI_rcu_doc>`.
		402	::
384		403
385	struct foo {	404	struct foo {
386	int a;	405	int a;
@@ -440,40 +459,43 @@ uses of RCU may be found in listRCU.txt, arrayRCU.txt, and NMI-RCU.txt. · 53 unmodified lines
440		459
441	So, to sum up:	460	So, to sum up:
442		461
443	o Use rcu_read_lock() and rcu_read_unlock() to guard RCU	462	- Use rcu_read_lock() and rcu_read_unlock() to guard RCU
444	read-side critical sections.	463	read-side critical sections.
445		464
446	o Within an RCU read-side critical section, use rcu_dereference()	465	- Within an RCU read-side critical section, use rcu_dereference()
447	to dereference RCU-protected pointers.	466	to dereference RCU-protected pointers.
448		467
449	o Use some solid scheme (such as locks or semaphores) to	468	- Use some solid scheme (such as locks or semaphores) to
450	keep concurrent updates from interfering with each other.	469	keep concurrent updates from interfering with each other.
451		470
452	o Use rcu_assign_pointer() to update an RCU-protected pointer.	471	- Use rcu_assign_pointer() to update an RCU-protected pointer.
453	This primitive protects concurrent readers from the updater,	472	This primitive protects concurrent readers from the updater,
454	-not- concurrent updates from each other! You therefore still	473	not concurrent updates from each other! You therefore still
455	need to use locking (or something similar) to keep concurrent	474	need to use locking (or something similar) to keep concurrent
456	rcu_assign_pointer() primitives from interfering with each other.	475	rcu_assign_pointer() primitives from interfering with each other.
457		476
458	o Use synchronize_rcu() -after- removing a data element from an	477	- Use synchronize_rcu() after removing a data element from an
459	RCU-protected data structure, but -before- reclaiming/freeing	478	RCU-protected data structure, but before reclaiming/freeing
460	the data element, in order to wait for the completion of all	479	the data element, in order to wait for the completion of all
461	RCU read-side critical sections that might be referencing that	480	RCU read-side critical sections that might be referencing that
462	data item.	481	data item.
463		482
464	See checklist.txt for additional rules to follow when using RCU.	483	See checklist.txt for additional rules to follow when using RCU.
465	And again, more-typical uses of RCU may be found in listRCU.txt,	484	And again, more-typical uses of RCU may be found in :ref:`listRCU.rst
466	arrayRCU.txt, and NMI-RCU.txt.	485	<list_rcu_doc>`, :ref:`arrayRCU.rst <array_rcu_doc>`, and :ref:`NMI-RCU.rst
		486	<NMI_rcu_doc>`.
467		487
		488	.. _4_whatisRCU:
468		489
469	4. WHAT IF MY UPDATING THREAD CANNOT BLOCK?	490	4. WHAT IF MY UPDATING THREAD CANNOT BLOCK?
		491	--------------------------------------------
470		492
471	In the example above, foo_update_a() blocks until a grace period elapses.	493	In the example above, foo_update_a() blocks until a grace period elapses.
472	This is quite simple, but in some cases one cannot afford to wait so	494	This is quite simple, but in some cases one cannot afford to wait so
473	long -- there might be other high-priority work to be done.	495	long -- there might be other high-priority work to be done.
474		496
475	In such cases, one uses call_rcu() rather than synchronize_rcu().	497	In such cases, one uses call_rcu() rather than synchronize_rcu().
476	The call_rcu() API is as follows:	498	The call_rcu() API is as follows::
477		499
478	void call_rcu(struct rcu_head * head,	500	void call_rcu(struct rcu_head * head,
479	void (func)(struct rcu_head head));	501	void (func)(struct rcu_head head));
@@ -481,7 +503,7 @@ The call_rcu() API is as follows: · 1 unmodified line
481	This function invokes func(head) after a grace period has elapsed.	503	This function invokes func(head) after a grace period has elapsed.
482	This invocation might happen from either softirq or process context,	504	This invocation might happen from either softirq or process context,
483	so the function is not permitted to block. The foo struct needs to	505	so the function is not permitted to block. The foo struct needs to
484	have an rcu_head structure added, perhaps as follows:	506	have an rcu_head structure added, perhaps as follows::
485		507
486	struct foo {	508	struct foo {
487	int a;	509	int a;
@@ -490,7 +512,7 @@ have an rcu_head structure added, perhaps as follows: · 2 unmodified lines
490	struct rcu_head rcu;	512	struct rcu_head rcu;
491	};	513	};
492		514
493	The foo_update_a() function might then be written as follows:	515	The foo_update_a() function might then be written as follows::
494		516
495	/*	517	/*
496	* Create a new struct foo that is the same as the one currently	518	* Create a new struct foo that is the same as the one currently
@@ -520,7 +542,7 @@ The foo_update_a() function might then be written as follows: · 23 unmodified lines
520	call_rcu(&old_fp->rcu, foo_reclaim);	542	call_rcu(&old_fp->rcu, foo_reclaim);
521	}	543	}
522		544
523	The foo_reclaim() function might appear as follows:	545	The foo_reclaim() function might appear as follows::
524		546
525	void foo_reclaim(struct rcu_head *rp)	547	void foo_reclaim(struct rcu_head *rp)
526	{	548	{
@@ -544,7 +566,7 @@ namely foo_reclaim(). · 17 unmodified lines
544	The summary of advice is the same as for the previous section, except	566	The summary of advice is the same as for the previous section, except
545	that we are now using call_rcu() rather than synchronize_rcu():	567	that we are now using call_rcu() rather than synchronize_rcu():
546		568
547	o Use call_rcu() -after- removing a data element from an	569	- Use call_rcu() after removing a data element from an
548	RCU-protected data structure in order to register a callback	570	RCU-protected data structure in order to register a callback
549	function that will be invoked after the completion of all RCU	571	function that will be invoked after the completion of all RCU
550	read-side critical sections that might be referencing that	572	read-side critical sections that might be referencing that
@@ -552,14 +574,16 @@ o Use call_rcu() -after- removing a data element from an · 1 unmodified line
552		574
553	If the callback for call_rcu() is not doing anything more than calling	575	If the callback for call_rcu() is not doing anything more than calling
554	kfree() on the structure, you can use kfree_rcu() instead of call_rcu()	576	kfree() on the structure, you can use kfree_rcu() instead of call_rcu()
555	to avoid having to write your own callback:	577	to avoid having to write your own callback::
556		578
557	kfree_rcu(old_fp, rcu);	579	kfree_rcu(old_fp, rcu);
558		580
559	Again, see checklist.txt for additional rules governing the use of RCU.	581	Again, see checklist.txt for additional rules governing the use of RCU.
560		582
		583	.. _5_whatisRCU:
561		584
562	5. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?	585	5. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?
		586	------------------------------------------------
563		587
564	One of the nice things about RCU is that it has extremely simple "toy"	588	One of the nice things about RCU is that it has extremely simple "toy"
565	implementations that are a good first step towards understanding the	589	implementations that are a good first step towards understanding the
@@ -579,7 +603,7 @@ more details on the current implementation as of early 2004. · 13 unmodified lines
579		603
580		604
581	5A. "TOY" IMPLEMENTATION #1: LOCKING	605	5A. "TOY" IMPLEMENTATION #1: LOCKING
582		606	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
583	This section presents a "toy" RCU implementation that is based on	607	This section presents a "toy" RCU implementation that is based on
584	familiar locking primitives. Its overhead makes it a non-starter for	608	familiar locking primitives. Its overhead makes it a non-starter for
585	real-life use, as does its lack of scalability. It is also unsuitable	609	real-life use, as does its lack of scalability. It is also unsuitable
@@ -591,7 +615,7 @@ you allow nested rcu_read_lock() calls, you can deadlock. · 5 unmodified lines
591	However, it is probably the easiest implementation to relate to, so is	615	However, it is probably the easiest implementation to relate to, so is
592	a good starting point.	616	a good starting point.
593		617
594	It is extremely simple:	618	It is extremely simple::
595		619
596	static DEFINE_RWLOCK(rcu_gp_mutex);	620	static DEFINE_RWLOCK(rcu_gp_mutex);
597		621
@@ -614,7 +638,7 @@ It is extremely simple: · 16 unmodified lines
614		638
615	[You can ignore rcu_assign_pointer() and rcu_dereference() without missing	639	[You can ignore rcu_assign_pointer() and rcu_dereference() without missing
616	much. But here are simplified versions anyway. And whatever you do,	640	much. But here are simplified versions anyway. And whatever you do,
617	don't forget about them when submitting patches making use of RCU!]	641	don't forget about them when submitting patches making use of RCU!]::
618		642
619	#define rcu_assign_pointer(p, v) \	643	#define rcu_assign_pointer(p, v) \
620	({ \	644	({ \
@@ -647,18 +671,23 @@ that the only thing that can block rcu_read_lock() is a synchronize_rcu(). · 26 unmodified lines
647	But synchronize_rcu() does not acquire any locks while holding rcu_gp_mutex,	671	But synchronize_rcu() does not acquire any locks while holding rcu_gp_mutex,
648	so there can be no deadlock cycle.	672	so there can be no deadlock cycle.
649		673
650	Quick Quiz #1: Why is this argument naive? How could a deadlock	674	.. _quiz_1:
		675
		676	Quick Quiz #1:
		677	Why is this argument naive? How could a deadlock
651	occur when using this algorithm in a real-world Linux	678	occur when using this algorithm in a real-world Linux
652	kernel? How could this deadlock be avoided?	679	kernel? How could this deadlock be avoided?
653		680
		681	:ref:`Answers to Quick Quiz <8_whatisRCU>`
654		682
655	5B. "TOY" EXAMPLE #2: CLASSIC RCU	683	5B. "TOY" EXAMPLE #2: CLASSIC RCU
656		684	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
657	This section presents a "toy" RCU implementation that is based on	685	This section presents a "toy" RCU implementation that is based on
658	"classic RCU". It is also short on performance (but only for updates) and	686	"classic RCU". It is also short on performance (but only for updates) and
659	on features such as hotplug CPU and the ability to run in CONFIG_PREEMPT	687	on features such as hotplug CPU and the ability to run in CONFIG_PREEMPT
660	kernels. The definitions of rcu_dereference() and rcu_assign_pointer()	688	kernels. The definitions of rcu_dereference() and rcu_assign_pointer()
661	are the same as those shown in the preceding section, so they are omitted.	689	are the same as those shown in the preceding section, so they are omitted.
		690	::
662		691
663	void rcu_read_lock(void) { }	692	void rcu_read_lock(void) { }
664		693
@@ -683,14 +712,14 @@ CPU in turn. The run_on() primitive can be implemented straightforwardly · 18 unmodified lines
683	in terms of the sched_setaffinity() primitive. Of course, a somewhat less	712	in terms of the sched_setaffinity() primitive. Of course, a somewhat less
684	"toy" implementation would restore the affinity upon completion rather	713	"toy" implementation would restore the affinity upon completion rather
685	than just leaving all tasks running on the last CPU, but when I said	714	than just leaving all tasks running on the last CPU, but when I said
686	"toy", I meant -toy-!	715	"toy", I meant toy!
687		716
688	So how the heck is this supposed to work???	717	So how the heck is this supposed to work???
689		718
690	Remember that it is illegal to block while in an RCU read-side critical	719	Remember that it is illegal to block while in an RCU read-side critical
691	section. Therefore, if a given CPU executes a context switch, we know	720	section. Therefore, if a given CPU executes a context switch, we know
692	that it must have completed all preceding RCU read-side critical sections.	721	that it must have completed all preceding RCU read-side critical sections.
693	Once -all- CPUs have executed a context switch, then -all- preceding	722	Once all CPUs have executed a context switch, then all preceding
694	RCU read-side critical sections will have completed.	723	RCU read-side critical sections will have completed.
695		724
696	So, suppose that we remove a data item from its structure and then invoke	725	So, suppose that we remove a data item from its structure and then invoke
@@ -698,19 +727,32 @@ synchronize_rcu(). Once synchronize_rcu() returns, we are guaranteed · 1 unmodified line
698	that there are no RCU read-side critical sections holding a reference	727	that there are no RCU read-side critical sections holding a reference
699	to that data item, so we can safely reclaim it.	728	to that data item, so we can safely reclaim it.
700		729
701	Quick Quiz #2: Give an example where Classic RCU's read-side	730	.. _quiz_2:
702	overhead is -negative-.	731
		732	Quick Quiz #2:
		733	Give an example where Classic RCU's read-side
		734	overhead is negative.
		735
		736	:ref:`Answers to Quick Quiz <8_whatisRCU>`
703		737
704	Quick Quiz #3: If it is illegal to block in an RCU read-side	738	.. _quiz_3:
		739
		740	Quick Quiz #3:
		741	If it is illegal to block in an RCU read-side
705	critical section, what the heck do you do in	742	critical section, what the heck do you do in
706	PREEMPT_RT, where normal spinlocks can block???	743	PREEMPT_RT, where normal spinlocks can block???
707		744
		745	:ref:`Answers to Quick Quiz <8_whatisRCU>`
		746
		747	.. _6_whatisRCU:
708		748
709	6. ANALOGY WITH READER-WRITER LOCKING	749	6. ANALOGY WITH READER-WRITER LOCKING
		750	--------------------------------------
710		751
711	Although RCU can be used in many different ways, a very common use of	752	Although RCU can be used in many different ways, a very common use of
712	RCU is analogous to reader-writer locking. The following unified	753	RCU is analogous to reader-writer locking. The following unified
713	diff shows how closely related RCU and reader-writer locking can be.	754	diff shows how closely related RCU and reader-writer locking can be.
		755	::
714		756
715	@@ -5,5 +5,5 @@ struct el {	757	@@ -5,5 +5,5 @@ struct el {
716	int data;	758	int data;
@@ -762,7 +804,7 @@ diff shows how closely related RCU and reader-writer locking can be. · 45 unmodified lines
762	return 0;	804	return 0;
763	}	805	}
764		806
765	Or, for those who prefer a side-by-side listing:	807	Or, for those who prefer a side-by-side listing::
766		808
767	1 struct el { 1 struct el {	809	1 struct el { 1 struct el {
768	2 struct list_head list; 2 struct list_head list;	810	2 struct list_head list; 2 struct list_head list;
@@ -774,40 +816,44 @@ Or, for those who prefer a side-by-side listing: · 5 unmodified lines
774	8 rwlock_t listmutex; 8 spinlock_t listmutex;	816	8 rwlock_t listmutex; 8 spinlock_t listmutex;
775	9 struct el head; 9 struct el head;	817	9 struct el head; 9 struct el head;
776		818
777	1 int search(long key, int result) 1 int search(long key, int result)	819	::
778	2 { 2 {	820
779	3 struct list_head lp; 3 struct list_head lp;	821	1 int search(long key, int result) 1 int search(long key, int result)
780	4 struct el p; 4 struct el p;	822	2 { 2 {
781	5 5	823	3 struct list_head lp; 3 struct list_head lp;
782	6 read_lock(&listmutex); 6 rcu_read_lock();	824	4 struct el p; 4 struct el p;
783	7 list_for_each_entry(p, head, lp) { 7 list_for_each_entry_rcu(p, head, lp) {	825	5 5
784	8 if (p->key == key) { 8 if (p->key == key) {	826	6 read_lock(&listmutex); 6 rcu_read_lock();
785	9 result = p->data; 9 result = p->data;	827	7 list_for_each_entry(p, head, lp) { 7 list_for_each_entry_rcu(p, head, lp) {
786	10 read_unlock(&listmutex); 10 rcu_read_unlock();	828	8 if (p->key == key) { 8 if (p->key == key) {
787	11 return 1; 11 return 1;	829	9 result = p->data; 9 result = p->data;
788	12 } 12 }	830	10 read_unlock(&listmutex); 10 rcu_read_unlock();
789	13 } 13 }	831	11 return 1; 11 return 1;
790	14 read_unlock(&listmutex); 14 rcu_read_unlock();	832	12 } 12 }
791	15 return 0; 15 return 0;	833	13 } 13 }
792	16 } 16 }	834	14 read_unlock(&listmutex); 14 rcu_read_unlock();
793		835	15 return 0; 15 return 0;
794	1 int delete(long key) 1 int delete(long key)	836	16 } 16 }
795	2 { 2 {	837
796	3 struct el p; 3 struct el p;	838	::
797	4 4	839
798	5 write_lock(&listmutex); 5 spin_lock(&listmutex);	840	1 int delete(long key) 1 int delete(long key)
799	6 list_for_each_entry(p, head, lp) { 6 list_for_each_entry(p, head, lp) {	841	2 { 2 {
800	7 if (p->key == key) { 7 if (p->key == key) {	842	3 struct el p; 3 struct el p;
801	8 list_del(&p->list); 8 list_del_rcu(&p->list);	843	4 4
802	9 write_unlock(&listmutex); 9 spin_unlock(&listmutex);	844	5 write_lock(&listmutex); 5 spin_lock(&listmutex);
803	10 synchronize_rcu();	845	6 list_for_each_entry(p, head, lp) { 6 list_for_each_entry(p, head, lp) {
804	10 kfree(p); 11 kfree(p);	846	7 if (p->key == key) { 7 if (p->key == key) {
805	11 return 1; 12 return 1;	847	8 list_del(&p->list); 8 list_del_rcu(&p->list);
806	12 } 13 }	848	9 write_unlock(&listmutex); 9 spin_unlock(&listmutex);
807	13 } 14 }	849	10 synchronize_rcu();
808	14 write_unlock(&listmutex); 15 spin_unlock(&listmutex);	850	10 kfree(p); 11 kfree(p);
809	15 return 0; 16 return 0;	851	11 return 1; 12 return 1;
810	16 } 17 }	852	12 } 13 }
		853	13 } 14 }
		854	14 write_unlock(&listmutex); 15 spin_unlock(&listmutex);
		855	15 return 0; 16 return 0;
		856	16 } 17 }
811		857
812	Either way, the differences are quite small. Read-side locking moves	858	Either way, the differences are quite small. Read-side locking moves
813	to rcu_read_lock() and rcu_read_unlock, update-side locking moves from	859	to rcu_read_lock() and rcu_read_unlock, update-side locking moves from
@@ -825,22 +871,27 @@ delete() can now block. If this is a problem, there is a callback-based · 11 unmodified lines
825	mechanism that never blocks, namely call_rcu() or kfree_rcu(), that can	871	mechanism that never blocks, namely call_rcu() or kfree_rcu(), that can
826	be used in place of synchronize_rcu().	872	be used in place of synchronize_rcu().
827		873
		874	.. _7_whatisRCU:
828		875
829	7. FULL LIST OF RCU APIs	876	7. FULL LIST OF RCU APIs
		877	-------------------------
830		878
831	The RCU APIs are documented in docbook-format header comments in the	879	The RCU APIs are documented in docbook-format header comments in the
832	Linux-kernel source code, but it helps to have a full list of the	880	Linux-kernel source code, but it helps to have a full list of the
833	APIs, since there does not appear to be a way to categorize them	881	APIs, since there does not appear to be a way to categorize them
834	in docbook. Here is the list, by category.	882	in docbook. Here is the list, by category.
835		883
836	RCU list traversal:	884	RCU list traversal::
837		885
838	list_entry_rcu	886	list_entry_rcu
		887	list_entry_lockless
839	list_first_entry_rcu	888	list_first_entry_rcu
840	list_next_rcu	889	list_next_rcu
841	list_for_each_entry_rcu	890	list_for_each_entry_rcu
842	list_for_each_entry_continue_rcu	891	list_for_each_entry_continue_rcu
843	list_for_each_entry_from_rcu	892	list_for_each_entry_from_rcu
		893	list_first_or_null_rcu
		894	list_next_or_null_rcu
844	hlist_first_rcu	895	hlist_first_rcu
845	hlist_next_rcu	896	hlist_next_rcu
846	hlist_pprev_rcu	897	hlist_pprev_rcu
@@ -854,7 +905,7 @@ RCU list traversal: · 7 unmodified lines
854	hlist_bl_first_rcu	905	hlist_bl_first_rcu
855	hlist_bl_for_each_entry_rcu	906	hlist_bl_for_each_entry_rcu
856		907
857	RCU pointer/list update:	908	RCU pointer/list update::
858		909
859	rcu_assign_pointer	910	rcu_assign_pointer
860	list_add_rcu	911	list_add_rcu
@@ -864,10 +915,12 @@ RCU pointer/list update: · 3 unmodified lines
864	hlist_add_behind_rcu	915	hlist_add_behind_rcu
865	hlist_add_before_rcu	916	hlist_add_before_rcu
866	hlist_add_head_rcu	917	hlist_add_head_rcu
		918	hlist_add_tail_rcu
867	hlist_del_rcu	919	hlist_del_rcu
868	hlist_del_init_rcu	920	hlist_del_init_rcu
869	hlist_replace_rcu	921	hlist_replace_rcu
870	list_splice_init_rcu()	922	list_splice_init_rcu
		923	list_splice_tail_init_rcu
871	hlist_nulls_del_init_rcu	924	hlist_nulls_del_init_rcu
872	hlist_nulls_del_rcu	925	hlist_nulls_del_rcu
873	hlist_nulls_add_head_rcu	926	hlist_nulls_add_head_rcu
@@ -876,7 +929,9 @@ RCU pointer/list update: · 2 unmodified lines
876	hlist_bl_del_rcu	929	hlist_bl_del_rcu
877	hlist_bl_set_first_rcu	930	hlist_bl_set_first_rcu
878		931
879	RCU: Critical sections Grace period Barrier	932	RCU::
		933
		934	Critical sections Grace period Barrier
880		935
881	rcu_read_lock synchronize_net rcu_barrier	936	rcu_read_lock synchronize_net rcu_barrier
882	rcu_read_unlock synchronize_rcu	937	rcu_read_unlock synchronize_rcu
@@ -885,7 +940,9 @@ RCU: Critical sections Grace period Barrier · 2 unmodified lines
885	rcu_dereference_check kfree_rcu	940	rcu_dereference_check kfree_rcu
886	rcu_dereference_protected	941	rcu_dereference_protected
887		942
888	bh: Critical sections Grace period Barrier	943	bh::
		944
		945	Critical sections Grace period Barrier
889		946
890	rcu_read_lock_bh call_rcu rcu_barrier	947	rcu_read_lock_bh call_rcu rcu_barrier
891	rcu_read_unlock_bh synchronize_rcu	948	rcu_read_unlock_bh synchronize_rcu
@@ -896,7 +953,9 @@ bh: Critical sections Grace period Barrier · 4 unmodified lines
896	rcu_dereference_bh_protected	953	rcu_dereference_bh_protected
897	rcu_read_lock_bh_held	954	rcu_read_lock_bh_held
898		955
899	sched: Critical sections Grace period Barrier	956	sched::
		957
		958	Critical sections Grace period Barrier
900		959
901	rcu_read_lock_sched call_rcu rcu_barrier	960	rcu_read_lock_sched call_rcu rcu_barrier
902	rcu_read_unlock_sched synchronize_rcu	961	rcu_read_unlock_sched synchronize_rcu
@@ -910,7 +969,9 @@ sched: Critical sections Grace period Barrier · 7 unmodified lines
910	rcu_read_lock_sched_held	969	rcu_read_lock_sched_held
911		970
912		971
913	SRCU: Critical sections Grace period Barrier	972	SRCU::
		973
		974	Critical sections Grace period Barrier
914		975
915	srcu_read_lock call_srcu srcu_barrier	976	srcu_read_lock call_srcu srcu_barrier
916	srcu_read_unlock synchronize_srcu	977	srcu_read_unlock synchronize_srcu
@@ -918,13 +979,14 @@ SRCU: Critical sections Grace period Barrier · 1 unmodified line
918	srcu_dereference_check	979	srcu_dereference_check
919	srcu_read_lock_held	980	srcu_read_lock_held
920		981
921	SRCU: Initialization/cleanup	982	SRCU: Initialization/cleanup::
		983
922	DEFINE_SRCU	984	DEFINE_SRCU
923	DEFINE_STATIC_SRCU	985	DEFINE_STATIC_SRCU
924	init_srcu_struct	986	init_srcu_struct
925	cleanup_srcu_struct	987	cleanup_srcu_struct
926		988
927	All: lockdep-checked RCU-protected pointer access	989	All: lockdep-checked RCU-protected pointer access::
928		990
929	rcu_access_pointer	991	rcu_access_pointer
930	rcu_dereference_raw	992	rcu_dereference_raw
@@ -974,15 +1036,19 @@ g. Otherwise, use RCU. · 43 unmodified lines
974	Of course, this all assumes that you have determined that RCU is in fact	1036	Of course, this all assumes that you have determined that RCU is in fact
975	the right tool for your job.	1037	the right tool for your job.
976		1038
		1039	.. _8_whatisRCU:
977		1040
978	8. ANSWERS TO QUICK QUIZZES	1041	8. ANSWERS TO QUICK QUIZZES
		1042	----------------------------
979		1043
980	Quick Quiz #1: Why is this argument naive? How could a deadlock	1044	Quick Quiz #1:
		1045	Why is this argument naive? How could a deadlock
981	occur when using this algorithm in a real-world Linux	1046	occur when using this algorithm in a real-world Linux
982	kernel? [Referring to the lock-based "toy" RCU	1047	kernel? [Referring to the lock-based "toy" RCU
983	algorithm.]	1048	algorithm.]
984		1049
985	Answer: Consider the following sequence of events:	1050	Answer:
		1051	Consider the following sequence of events:
986		1052
987	1. CPU 0 acquires some unrelated lock, call it	1053	1. CPU 0 acquires some unrelated lock, call it
988	"problematic_lock", disabling irq via	1054	"problematic_lock", disabling irq via
@@ -1021,10 +1087,14 @@ Answer: Consider the following sequence of events: · 32 unmodified lines
1021	approach where tasks in RCU read-side critical sections	1087	approach where tasks in RCU read-side critical sections
1022	cannot be blocked by tasks executing synchronize_rcu().	1088	cannot be blocked by tasks executing synchronize_rcu().
1023		1089
1024	Quick Quiz #2: Give an example where Classic RCU's read-side	1090	:ref:`Back to Quick Quiz #1 <quiz_1>`
1025	overhead is -negative-.	1091
		1092	Quick Quiz #2:
		1093	Give an example where Classic RCU's read-side
		1094	overhead is negative.
1026		1095
1027	Answer: Imagine a single-CPU system with a non-CONFIG_PREEMPT	1096	Answer:
		1097	Imagine a single-CPU system with a non-CONFIG_PREEMPT
1028	kernel where a routing table is used by process-context	1098	kernel where a routing table is used by process-context
1029	code, but can be updated by irq-context code (for example,	1099	code, but can be updated by irq-context code (for example,
1030	by an "ICMP REDIRECT" packet). The usual way of handling	1100	by an "ICMP REDIRECT" packet). The usual way of handling
@@ -1046,11 +1116,15 @@ Answer: Imagine a single-CPU system with a non-CONFIG_PREEMPT · 15 unmodified lines
1046	even the theoretical possibility of negative overhead for	1116	even the theoretical possibility of negative overhead for
1047	a synchronization primitive is a bit unexpected. ;-)	1117	a synchronization primitive is a bit unexpected. ;-)
1048		1118
1049	Quick Quiz #3: If it is illegal to block in an RCU read-side	1119	:ref:`Back to Quick Quiz #2 <quiz_2>`
		1120
		1121	Quick Quiz #3:
		1122	If it is illegal to block in an RCU read-side
1050	critical section, what the heck do you do in	1123	critical section, what the heck do you do in
1051	PREEMPT_RT, where normal spinlocks can block???	1124	PREEMPT_RT, where normal spinlocks can block???
1052		1125
1053	Answer: Just as PREEMPT_RT permits preemption of spinlock	1126	Answer:
		1127	Just as PREEMPT_RT permits preemption of spinlock
1054	critical sections, it permits preemption of RCU	1128	critical sections, it permits preemption of RCU
1055	read-side critical sections. It also permits	1129	read-side critical sections. It also permits
1056	spinlocks blocking while in RCU read-side critical	1130	spinlocks blocking while in RCU read-side critical
@@ -1069,6 +1143,7 @@ Answer: Just as PREEMPT_RT permits preemption of spinlock · 12 unmodified lines
1069	Besides, how does the computer know what pizza parlor	1143	Besides, how does the computer know what pizza parlor
1070	the human being went to???	1144	the human being went to???
1071		1145
		1146	:ref:`Back to Quick Quiz #3 <quiz_3>`
1072		1147
1073	ACKNOWLEDGEMENTS	1148	ACKNOWLEDGEMENTS
1074		1149

Documentation/admin-guide/kernel-parameters.txt+13−0

@@ -3978,6 +3978,19 @@ · 3977 unmodified lines
3978	test until boot completes in order to avoid	3978	test until boot completes in order to avoid
3979	interference.	3979	interference.
3980		3980
		3981	rcuperf.kfree_rcu_test= [KNL]
		3982	Set to measure performance of kfree_rcu() flooding.
		3983
		3984	rcuperf.kfree_nthreads= [KNL]
		3985	The number of threads running loops of kfree_rcu().
		3986
		3987	rcuperf.kfree_alloc_num= [KNL]
		3988	Number of allocations and frees done in an iteration.
		3989
		3990	rcuperf.kfree_loops= [KNL]
		3991	Number of loops doing rcuperf.kfree_alloc_num number
		3992	of allocations and frees.
		3993
3981	rcuperf.nreaders= [KNL]	3994	rcuperf.nreaders= [KNL]
3982	Set number of RCU readers. The value -1 selects	3995	Set number of RCU readers. The value -1 selects
3983	N, where N is the number of CPUs. A value	3996	N, where N is the number of CPUs. A value

arch/powerpc/include/asm/barrier.h+0−2

  * mb() prevents loads and stores being reordered across this point.
  * rmb() prevents loads being reordered across this point.
  * wmb() prevents stores being reordered across this point.
- * read_barrier_depends() prevents data-dependent loads being reordered
- *	across this point (nop on PPC).
  *
  * *mb() variants without smp_ prefix must order all types of memory
  * operations with one another. sync is the only instruction sufficient

drivers/net/wireless/mediatek/mt76/agg-rx.c+2−2

@@ -281,8 +281,8 @@ void mt76_rx_aggr_stop(struct mt76_dev dev, struct mt76_wcid wcid, u8 tidno) · 280 unmodified lines
281	{	281	{
282	struct mt76_rx_tid *tid = NULL;	282	struct mt76_rx_tid *tid = NULL;
283		283
284	rcu_swap_protected(wcid->aggr[tidno], tid,	284	tid = rcu_replace_pointer(wcid->aggr[tidno], tid,
285	lockdep_is_held(&dev->mutex));	285	lockdep_is_held(&dev->mutex));
286	if (tid) {	286	if (tid) {
287	mt76_rx_aggr_shutdown(dev, tid);	287	mt76_rx_aggr_shutdown(dev, tid);
288	kfree_rcu(tid, rcu_head);	288	kfree_rcu(tid, rcu_head);

include/linux/list.h+112−24

 #define LIST_HEAD(name) \
 	struct list_head name = LIST_HEAD_INIT(name)
+/**
+ * INIT_LIST_HEAD - Initialize a list_head structure
+ * @list: list_head structure to be initialized.
+ *
+ * Initializes the list_head to point to itself.  If it is a list header,
+ * the result is an empty list.
+ */
 static inline void INIT_LIST_HEAD(struct list_head *list)
 {
 	WRITE_ONCE(list->next, list);
 	entry->prev = NULL;
 }
-/**
- * list_del - deletes entry from list.
- * @entry: the element to delete from the list.
- * Note: list_empty() on entry does not return true after this, the entry is
- * in an undefined state.
- */
 static inline void __list_del_entry(struct list_head *entry)
 {
 	if (!__list_del_entry_valid(entry))
 	__list_del(entry->prev, entry->next);
 }
+/**
+ * list_del - deletes entry from list.
+ * @entry: the element to delete from the list.
+ * Note: list_empty() on entry does not return true after this, the entry is
+ * in an undefined state.
+ */
 static inline void list_del(struct list_head *entry)
 {
 	__list_del_entry(entry);
 	new->prev->next = new;
 }
+/**
+ * list_replace_init - replace old entry by new one and initialize the old one
+ * @old : the element to be replaced
+ * @new : the new element to insert
+ *
+ * If @old was empty, it will be overwritten.
+ */
 static inline void list_replace_init(struct list_head *old,
-					struct list_head *new)
+				     struct list_head *new)
 {
 	list_replace(old, new);
 	INIT_LIST_HEAD(old);
 	h->pprev = NULL;
 }
+/**
+ * hlist_unhashed - Has node been removed from list and reinitialized?
+ * @h: Node to be checked
+ *
+ * Not that not all removal functions will leave a node in unhashed
+ * state.  For example, hlist_nulls_del_init_rcu() does leave the
+ * node in unhashed state, but hlist_nulls_del() does not.
+ */
 static inline int hlist_unhashed(const struct hlist_node *h)
 {
 	return !h->pprev;
 }
+/**
+ * hlist_unhashed_lockless - Version of hlist_unhashed for lockless use
+ * @h: Node to be checked
+ *
+ * This variant of hlist_unhashed() must be used in lockless contexts
+ * to avoid potential load-tearing.  The READ_ONCE() is paired with the
+ * various WRITE_ONCE() in hlist helpers that are defined below.
+ */
+static inline int hlist_unhashed_lockless(const struct hlist_node *h)
+{
+	return !READ_ONCE(h->pprev);
+}
+/**
+ * hlist_empty - Is the specified hlist_head structure an empty hlist?
+ * @h: Structure to check.
+ */
 static inline int hlist_empty(const struct hlist_head *h)
 {
 	return !READ_ONCE(h->first);
 	WRITE_ONCE(*pprev, next);
 	if (next)
-		next->pprev = pprev;
+		WRITE_ONCE(next->pprev, pprev);
 }
+/**
+ * hlist_del - Delete the specified hlist_node from its list
+ * @n: Node to delete.
+ *
+ * Note that this function leaves the node in hashed state.  Use
+ * hlist_del_init() or similar instead to unhash @n.
+ */
 static inline void hlist_del(struct hlist_node *n)
 {
 	__hlist_del(n);
 	n->pprev = LIST_POISON2;
 }
+/**
+ * hlist_del_init - Delete the specified hlist_node from its list and initialize
+ * @n: Node to delete.
+ *
+ * Note that this function leaves the node in unhashed state.
+ */
 static inline void hlist_del_init(struct hlist_node *n)
 {
 	if (!hlist_unhashed(n)) {
 	}
 }
+/**
+ * hlist_add_head - add a new entry at the beginning of the hlist
+ * @n: new entry to be added
+ * @h: hlist head to add it after
+ *
+ * Insert a new entry after the specified head.
+ * This is good for implementing stacks.
+ */
 static inline void hlist_add_head(struct hlist_node *n, struct hlist_head *h)
 {
 	struct hlist_node *first = h->first;
-	n->next = first;
+	WRITE_ONCE(n->next, first);
 	if (first)
-		first->pprev = &n->next;
+		WRITE_ONCE(first->pprev, &n->next);
 	WRITE_ONCE(h->first, n);
-	n->pprev = &h->first;
+	WRITE_ONCE(n->pprev, &h->first);
 }
-/* next must be != NULL */
+/**
+ * hlist_add_before - add a new entry before the one specified
+ * @n: new entry to be added
+ * @next: hlist node to add it before, which must be non-NULL
+ */
 static inline void hlist_add_before(struct hlist_node *n,
-					struct hlist_node *next)
+				    struct hlist_node *next)
 {
-	n->pprev = next->pprev;
+	WRITE_ONCE(n->pprev, next->pprev);
-	n->next = next;
+	WRITE_ONCE(n->next, next);
-	next->pprev = &n->next;
+	WRITE_ONCE(next->pprev, &n->next);
 	WRITE_ONCE(*(n->pprev), n);
 }
+/**
+ * hlist_add_behing - add a new entry after the one specified
+ * @n: new entry to be added
+ * @prev: hlist node to add it after, which must be non-NULL
+ */
 static inline void hlist_add_behind(struct hlist_node *n,
 				    struct hlist_node *prev)
 {
-	n->next = prev->next;
+	WRITE_ONCE(n->next, prev->next);
-	prev->next = n;
+	WRITE_ONCE(prev->next, n);
-	n->pprev = &prev->next;
+	WRITE_ONCE(n->pprev, &prev->next);
 	if (n->next)
-		n->next->pprev  = &n->next;
+		WRITE_ONCE(n->next->pprev, &n->next);
 }
-/* after that we'll appear to be on some hlist and hlist_del will work */
+/**
+ * hlist_add_fake - create a fake hlist consisting of a single headless node
+ * @n: Node to make a fake list out of
+ *
+ * This makes @n appear to be its own predecessor on a headless hlist.
+ * The point of this is to allow things like hlist_del() to work correctly
+ * in cases where there is no list.
+ */
 static inline void hlist_add_fake(struct hlist_node *n)
 {
 	n->pprev = &n->next;
 }
+/**
+ * hlist_fake: Is this node a fake hlist?
+ * @h: Node to check for being a self-referential fake hlist.
+ */
 static inline bool hlist_fake(struct hlist_node *h)
 {
 	return h->pprev == &h->next;
 }
-/*
+/**
+ * hlist_is_singular_node - is node the only element of the specified hlist?
+ * @n: Node to check for singularity.
+ * @h: Header for potentially singular list.
+ *
  * Check whether the node is the only node of the head without
- * accessing head:
+ * accessing head, thus avoiding unnecessary cache misses.
  */
 static inline bool
 hlist_is_singular_node(struct hlist_node *n, struct hlist_head *h)
 	return !n->next && n->pprev == &h->first;
 }
-/*
+/**
+ * hlist_move_list - Move an hlist
+ * @old: hlist_head for old list.
+ * @new: hlist_head for new list.
+ *
  * Move a list from one list head to another. Fixup the pprev
  * reference of the first entry if it exists.
  */

include/linux/list_nulls.h+26−4

 	return ((unsigned long)ptr) >> 1;
 }
+/**
+ * hlist_nulls_unhashed - Has node been removed and reinitialized?
+ * @h: Node to be checked
+ *
+ * Not that not all removal functions will leave a node in unhashed state.
+ * For example, hlist_del_init_rcu() leaves the node in unhashed state,
+ * but hlist_nulls_del() does not.
+ */
 static inline int hlist_nulls_unhashed(const struct hlist_nulls_node *h)
 {
 	return !h->pprev;
 }
+/**
+ * hlist_nulls_unhashed_lockless - Has node been removed and reinitialized?
+ * @h: Node to be checked
+ *
+ * Not that not all removal functions will leave a node in unhashed state.
+ * For example, hlist_del_init_rcu() leaves the node in unhashed state,
+ * but hlist_nulls_del() does not.  Unlike hlist_nulls_unhashed(), this
+ * function may be used locklessly.
+ */
+static inline int hlist_nulls_unhashed_lockless(const struct hlist_nulls_node *h)
+{
+	return !READ_ONCE(h->pprev);
+}
 static inline int hlist_nulls_empty(const struct hlist_nulls_head *h)
 {
 	return is_a_nulls(READ_ONCE(h->first));
 	struct hlist_nulls_node *first = h->first;
 	n->next = first;
-	n->pprev = &h->first;
+	WRITE_ONCE(n->pprev, &h->first);
 	h->first = n;
 	if (!is_a_nulls(first))
-		first->pprev = &n->next;
+		WRITE_ONCE(first->pprev, &n->next);
 }
 static inline void __hlist_nulls_del(struct hlist_nulls_node *n)
 	WRITE_ONCE(*pprev, next);
 	if (!is_a_nulls(next))
-		next->pprev = pprev;
+		WRITE_ONCE(next->pprev, pprev);
 }
 static inline void hlist_nulls_del(struct hlist_nulls_node *n)
 {
 	__hlist_nulls_del(n);
-	n->pprev = LIST_POISON2;
+	WRITE_ONCE(n->pprev, LIST_POISON2);
 }
 /**

include/linux/rcu_segcblist.h+0−2

 	struct rcu_head *head;
 	struct rcu_head **tail;
 	long len;
-	long len_lazy;
 };
 #define RCU_CBLIST_INITIALIZER(n) { .head = NULL, .tail = &n.head }
 #else
 	long len;
 #endif
-	long len_lazy;
 	u8 enabled;
 	u8 offloaded;
 };

include/linux/rculist.h+24−14

  */
 #define list_next_rcu(list)	(*((struct list_head __rcu **)(&(list)->next)))
+/**
+ * list_tail_rcu - returns the prev pointer of the head of the list
+ * @head: the head of the list
+ *
+ * Note: This should only be used with the list header, and even then
+ * only if list_del() and similar primitives are not also used on the
+ * list header.
+ */
+#define list_tail_rcu(head)	(*((struct list_head __rcu **)(&(head)->prev)))
 /*
  * Check during list traversal that we are within an RCU reader
  */
 {
 	if (!hlist_unhashed(n)) {
 		__hlist_del(n);
-		n->pprev = NULL;
+		WRITE_ONCE(n->pprev, NULL);
 	}
 }
  * @pos:	the type * to use as a loop cursor.
  * @head:	the head for your list.
  * @member:	the name of the list_head within the struct.
- * @cond:	optional lockdep expression if called from non-RCU protection.
+ * @cond...:	optional lockdep expression if called from non-RCU protection.
  *
  * This list-traversal primitive may safely run concurrently with
  * the _rcu list-mutation primitives such as list_add_rcu()
 static inline void hlist_del_rcu(struct hlist_node *n)
 {
 	__hlist_del(n);
-	n->pprev = LIST_POISON2;
+	WRITE_ONCE(n->pprev, LIST_POISON2);
 }
 /**
 	struct hlist_node *next = old->next;
 	new->next = next;
-	new->pprev = old->pprev;
+	WRITE_ONCE(new->pprev, old->pprev);
 	rcu_assign_pointer(*(struct hlist_node __rcu **)new->pprev, new);
 	if (next)
-		new->next->pprev = &new->next;
+		WRITE_ONCE(new->next->pprev, &new->next);
-	old->pprev = LIST_POISON2;
+	WRITE_ONCE(old->pprev, LIST_POISON2);
 }
 /*
 	struct hlist_node *first = h->first;
 	n->next = first;
-	n->pprev = &h->first;
+	WRITE_ONCE(n->pprev, &h->first);
 	rcu_assign_pointer(hlist_first_rcu(h), n);
 	if (first)
-		first->pprev = &n->next;
+		WRITE_ONCE(first->pprev, &n->next);
 }
 /**
 	if (last) {
 		n->next = last->next;
-		n->pprev = &last->next;
+		WRITE_ONCE(n->pprev, &last->next);
 		rcu_assign_pointer(hlist_next_rcu(last), n);
 	} else {
 		hlist_add_head_rcu(n, h);
 static inline void hlist_add_before_rcu(struct hlist_node *n,
 					struct hlist_node *next)
 {
-	n->pprev = next->pprev;
+	WRITE_ONCE(n->pprev, next->pprev);
 	n->next = next;
 	rcu_assign_pointer(hlist_pprev_rcu(n), n);
-	next->pprev = &n->next;
+	WRITE_ONCE(next->pprev, &n->next);
 }
 /**
 					struct hlist_node *prev)
 {
 	n->next = prev->next;
-	n->pprev = &prev->next;
+	WRITE_ONCE(n->pprev, &prev->next);
 	rcu_assign_pointer(hlist_next_rcu(prev), n);
 	if (n->next)
-		n->next->pprev = &n->next;
+		WRITE_ONCE(n->next->pprev, &n->next);
 }
 #define __hlist_for_each_rcu(pos, head)				\
  * @pos:	the type * to use as a loop cursor.
  * @head:	the head for your list.
  * @member:	the name of the hlist_node within the struct.
- * @cond:	optional lockdep expression if called from non-RCU protection.
+ * @cond...:	optional lockdep expression if called from non-RCU protection.
  *
  * This list-traversal primitive may safely run concurrently with
  * the _rcu list-mutation primitives such as hlist_add_head_rcu()

include/linux/rculist_nulls.h+14−6

 {
 	if (!hlist_nulls_unhashed(n)) {
 		__hlist_nulls_del(n);
-		n->pprev = NULL;
+		WRITE_ONCE(n->pprev, NULL);
 	}
 }
+/**
+ * hlist_nulls_first_rcu - returns the first element of the hash list.
+ * @head: the head of the list.
+ */
 #define hlist_nulls_first_rcu(head) \
 	(*((struct hlist_nulls_node __rcu __force **)&(head)->first))
+/**
+ * hlist_nulls_next_rcu - returns the element of the list after @node.
+ * @node: element of the list.
+ */
 #define hlist_nulls_next_rcu(node) \
 	(*((struct hlist_nulls_node __rcu __force **)&(node)->next))
 static inline void hlist_nulls_del_rcu(struct hlist_nulls_node *n)
 {
 	__hlist_nulls_del(n);
-	n->pprev = LIST_POISON2;
+	WRITE_ONCE(n->pprev, LIST_POISON2);
 }
 /**
 	struct hlist_nulls_node *first = h->first;
 	n->next = first;
-	n->pprev = &h->first;
+	WRITE_ONCE(n->pprev, &h->first);
 	rcu_assign_pointer(hlist_nulls_first_rcu(h), n);
 	if (!is_a_nulls(first))
-		first->pprev = &n->next;
+		WRITE_ONCE(first->pprev, &n->next);
 }
 /**
  * hlist_nulls_for_each_entry_rcu - iterate over rcu list of given type
  * @tpos:	the type * to use as a loop cursor.
  * @pos:	the &struct hlist_nulls_node to use as a loop cursor.
- * @head:	the head for your list.
+ * @head:	the head of the list.
  * @member:	the name of the hlist_nulls_node within the struct.
  *
  * The barrier() is needed to make sure compiler doesn't cache first element [1],
  *   iterate over list of given type safe against removal of list entry
  * @tpos:	the type * to use as a loop cursor.
  * @pos:	the &struct hlist_nulls_node to use as a loop cursor.
- * @head:	the head for your list.
+ * @head:	the head of the list.
  * @member:	the name of the hlist_nulls_node within the struct.
  */
 #define hlist_nulls_for_each_entry_safe(tpos, pos, head, member)		\

include/linux/rcupdate.h+8−20

  *
  * This macro resembles cond_resched(), except that it is defined to
  * report potential quiescent states to RCU-tasks even if the cond_resched()
- * machinery were to be shut off, as some advocate for PREEMPT kernels.
+ * machinery were to be shut off, as some advocate for PREEMPTION kernels.
  */
 #define cond_resched_tasks_rcu_qs() \
 do { \
  * TREE_RCU and rcu_barrier_() primitives in TINY_RCU.
  */
-#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)
+#if defined(CONFIG_TREE_RCU)
 #include <linux/rcutree.h>
 #elif defined(CONFIG_TINY_RCU)
 #include <linux/rcutiny.h>
 	__tmp;								\
 })
-/**
- * rcu_swap_protected() - swap an RCU and a regular pointer
- * @rcu_ptr: RCU pointer
- * @ptr: regular pointer
- * @c: the conditions under which the dereference will take place
- *
- * Perform swap(@rcu_ptr, @ptr) where @rcu_ptr is an RCU-annotated pointer and
- * @c is the argument that is passed to the rcu_dereference_protected() call
- * used to read that pointer.
- */
-#define rcu_swap_protected(rcu_ptr, ptr, c) do {			\
-	typeof(ptr) __tmp = rcu_dereference_protected((rcu_ptr), (c));	\
-	rcu_assign_pointer((rcu_ptr), (ptr));				\
-	(ptr) = __tmp;							\
-} while (0)
 /**
  * rcu_access_pointer() - fetch RCU pointer with no dereferencing
  * @p: The pointer to read
  *
  * You can avoid reading and understanding the next paragraph by
  * following this rule: don't put anything in an rcu_read_lock() RCU
- * read-side critical section that would block in a !PREEMPT kernel.
+ * read-side critical section that would block in a !PREEMPTION kernel.
  * But if you want the full story, read on!
  *
- * In non-preemptible RCU implementations (TREE_RCU and TINY_RCU),
+ * In non-preemptible RCU implementations (pure TREE_RCU and TINY_RCU),
  * it is illegal to block while in an RCU read-side critical section.
  * In preemptible RCU implementations (PREEMPT_RCU) in CONFIG_PREEMPTION
  * kernel builds, RCU read-side critical sections may be preempted,
 	return false;
 }
+/* kernel/ksysfs.c definitions */
+extern int rcu_expedited;
+extern int rcu_normal;
 #endif /* __LINUX_RCUPDATE_H */

include/linux/rcutiny.h+1−0

@@ -85,6 +85,7 @@ static inline void rcu_scheduler_starting(void) { } · 84 unmodified lines
85	static inline void rcu_end_inkernel_boot(void) { }	85	static inline void rcu_end_inkernel_boot(void) { }
86	static inline bool rcu_is_watching(void) { return true; }	86	static inline bool rcu_is_watching(void) { return true; }
87	static inline void rcu_momentary_dyntick_idle(void) { }	87	static inline void rcu_momentary_dyntick_idle(void) { }
		88	static inline void kfree_rcu_scheduler_running(void) { }
88		89
89	/* Avoid RCU read-side critical sections leaking across. */	90	/* Avoid RCU read-side critical sections leaking across. */
90	static inline void rcu_all_qs(void) { barrier(); }	91	static inline void rcu_all_qs(void) { barrier(); }

include/linux/rcutree.h+1−0

@@ -38,6 +38,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func); · 37 unmodified lines
38	void rcu_barrier(void);	38	void rcu_barrier(void);
39	bool rcu_eqs_special_set(int cpu);	39	bool rcu_eqs_special_set(int cpu);
40	void rcu_momentary_dyntick_idle(void);	40	void rcu_momentary_dyntick_idle(void);
		41	void kfree_rcu_scheduler_running(void);
41	unsigned long get_state_synchronize_rcu(void);	42	unsigned long get_state_synchronize_rcu(void);
42	void cond_synchronize_rcu(unsigned long oldstate);	43	void cond_synchronize_rcu(unsigned long oldstate);
43		44

include/linux/tick.h+4−1

 	TICK_DEP_BIT_PERF_EVENTS	= 1,
 	TICK_DEP_BIT_SCHED		= 2,
 	TICK_DEP_BIT_CLOCK_UNSTABLE	= 3,
-	TICK_DEP_BIT_RCU		= 4
+	TICK_DEP_BIT_RCU		= 4,
+	TICK_DEP_BIT_RCU_EXP		= 5
 };
+#define TICK_DEP_BIT_MAX TICK_DEP_BIT_RCU_EXP
 #define TICK_DEP_MASK_NONE		0
 #define TICK_DEP_MASK_POSIX_TIMER	(1 << TICK_DEP_BIT_POSIX_TIMER)
 #define TICK_DEP_MASK_SCHED		(1 << TICK_DEP_BIT_SCHED)
 #define TICK_DEP_MASK_CLOCK_UNSTABLE	(1 << TICK_DEP_BIT_CLOCK_UNSTABLE)
 #define TICK_DEP_MASK_RCU		(1 << TICK_DEP_BIT_RCU)
+#define TICK_DEP_MASK_RCU_EXP		(1 << TICK_DEP_BIT_RCU_EXP)
 #ifdef CONFIG_NO_HZ_COMMON
 extern bool tick_nohz_enabled;

include/trace/events/rcu.h+16−24

 	TP_printk("%s", __entry->s)
 );
-#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)
+#if defined(CONFIG_TREE_RCU)
 /*
  * Tracepoint for grace-period events.  Takes a string identifying the
 		  __entry->cpu, __entry->qsevent)
 );
-#endif /* #if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU) */
+#endif /* #if defined(CONFIG_TREE_RCU) */
 /*
  * Tracepoint for dyntick-idle entry/exit events.  These take a string
  */
 TRACE_EVENT_RCU(rcu_dyntick,
-	TP_PROTO(const char *polarity, long oldnesting, long newnesting, atomic_t dynticks),
+	TP_PROTO(const char *polarity, long oldnesting, long newnesting, int dynticks),
 	TP_ARGS(polarity, oldnesting, newnesting, dynticks),
 		__entry->polarity = polarity;
 		__entry->oldnesting = oldnesting;
 		__entry->newnesting = newnesting;
-		__entry->dynticks = atomic_read(&dynticks);
+		__entry->dynticks = dynticks;
 	),
 	TP_printk("%s %lx %lx %#3x", __entry->polarity,
  */
 TRACE_EVENT_RCU(rcu_callback,
-	TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen_lazy,
+	TP_PROTO(const char *rcuname, struct rcu_head *rhp, long qlen),
-		 long qlen),
-	TP_ARGS(rcuname, rhp, qlen_lazy, qlen),
+	TP_ARGS(rcuname, rhp, qlen),
 	TP_STRUCT__entry(
 		__field(const char *, rcuname)
 		__field(void *, rhp)
 		__field(void *, func)
-		__field(long, qlen_lazy)
 		__field(long, qlen)
 	),
 		__entry->rcuname = rcuname;
 		__entry->rhp = rhp;
 		__entry->func = rhp->func;
-		__entry->qlen_lazy = qlen_lazy;
 		__entry->qlen = qlen;
 	),
-	TP_printk("%s rhp=%p func=%ps %ld/%ld",
+	TP_printk("%s rhp=%p func=%ps %ld",
 		  __entry->rcuname, __entry->rhp, __entry->func,
-		  __entry->qlen_lazy, __entry->qlen)
+		  __entry->qlen)
 );
 /*
 TRACE_EVENT_RCU(rcu_kfree_callback,
 	TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset,
-		 long qlen_lazy, long qlen),
+		 long qlen),
-	TP_ARGS(rcuname, rhp, offset, qlen_lazy, qlen),
+	TP_ARGS(rcuname, rhp, offset, qlen),
 	TP_STRUCT__entry(
 		__field(const char *, rcuname)
 		__field(void *, rhp)
 		__field(unsigned long, offset)
-		__field(long, qlen_lazy)
 		__field(long, qlen)
 	),
 		__entry->rcuname = rcuname;
 		__entry->rhp = rhp;
 		__entry->offset = offset;
-		__entry->qlen_lazy = qlen_lazy;
 		__entry->qlen = qlen;
 	),
-	TP_printk("%s rhp=%p func=%ld %ld/%ld",
+	TP_printk("%s rhp=%p func=%ld %ld",
 		  __entry->rcuname, __entry->rhp, __entry->offset,
-		  __entry->qlen_lazy, __entry->qlen)
+		  __entry->qlen)
 );
 /*
  */
 TRACE_EVENT_RCU(rcu_batch_start,
-	TP_PROTO(const char *rcuname, long qlen_lazy, long qlen, long blimit),
+	TP_PROTO(const char *rcuname, long qlen, long blimit),
-	TP_ARGS(rcuname, qlen_lazy, qlen, blimit),
+	TP_ARGS(rcuname, qlen, blimit),
 	TP_STRUCT__entry(
 		__field(const char *, rcuname)
-		__field(long, qlen_lazy)
 		__field(long, qlen)
 		__field(long, blimit)
 	),
 	TP_fast_assign(
 		__entry->rcuname = rcuname;
-		__entry->qlen_lazy = qlen_lazy;
 		__entry->qlen = qlen;
 		__entry->blimit = blimit;
 	),
-	TP_printk("%s CBs=%ld/%ld bl=%ld",
+	TP_printk("%s CBs=%ld bl=%ld",
-		  __entry->rcuname, __entry->qlen_lazy, __entry->qlen,
+		  __entry->rcuname, __entry->qlen, __entry->blimit)
-		  __entry->blimit)
 );
 /*

kernel/rcu/Kconfig+9−8

 config TREE_RCU
 	bool
-	default y if !PREEMPTION && SMP
+	default y if SMP
 	help
 	  This option selects the RCU implementation that is
 	  designed for very large SMP system with hundreds or
 config PREEMPT_RCU
 	bool
 	default y if PREEMPTION
+	select TREE_RCU
 	help
 	  This option selects the RCU implementation that is
 	  designed for very large SMP systems with hundreds or
 	  user-mode execution as quiescent states.
 config RCU_STALL_COMMON
-	def_bool ( TREE_RCU || PREEMPT_RCU )
+	def_bool TREE_RCU
 	help
 	  This option enables RCU CPU stall code that is common between
 	  the TINY and TREE variants of RCU.  The purpose is to allow
 	  making these warnings mandatory for the tree variants.
 config RCU_NEED_SEGCBLIST
-	def_bool ( TREE_RCU || PREEMPT_RCU || TREE_SRCU )
+	def_bool ( TREE_RCU || TREE_SRCU )
 config RCU_FANOUT
 	int "Tree-based hierarchical RCU fanout value"
 	range 2 64 if 64BIT
 	range 2 32 if !64BIT
-	depends on (TREE_RCU || PREEMPT_RCU) && RCU_EXPERT
+	depends on TREE_RCU && RCU_EXPERT
 	default 64 if 64BIT
 	default 32 if !64BIT
 	help
 	int "Tree-based hierarchical RCU leaf-level fanout value"
 	range 2 64 if 64BIT
 	range 2 32 if !64BIT
-	depends on (TREE_RCU || PREEMPT_RCU) && RCU_EXPERT
+	depends on TREE_RCU && RCU_EXPERT
 	default 16
 	help
 	  This option controls the leaf-level fanout of hierarchical
 config RCU_NOCB_CPU
 	bool "Offload RCU callback processing from boot-selected CPUs"
-	depends on TREE_RCU || PREEMPT_RCU
+	depends on TREE_RCU
 	depends on RCU_EXPERT || NO_HZ_FULL
 	default n
 	help
 	  specified at boot time by the rcu_nocbs parameter.  For each
 	  such CPU, a kthread ("rcuox/N") will be created to invoke
 	  callbacks, where the "N" is the CPU being offloaded, and where
-	  the "p" for RCU-preempt (PREEMPT kernels) and "s" for RCU-sched
+	  the "p" for RCU-preempt (PREEMPTION kernels) and "s" for RCU-sched
-	  (!PREEMPT kernels).  Nothing prevents this kthread from running
+	  (!PREEMPTION kernels).  Nothing prevents this kthread from running
 	  on the specified CPUs, but (1) the kthreads may be preempted
 	  between each callback, and (2) affinity or cgroups can be used
 	  to force the kthreads to run on whatever set of CPUs is desired.

kernel/rcu/Makefile+0−1

@@ -9,6 +9,5 @@ obj-$(CONFIG_TINY_SRCU) += srcutiny.o · 8 unmodified lines
9	obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o	9	obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o
10	obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o	10	obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o
11	obj-$(CONFIG_TREE_RCU) += tree.o	11	obj-$(CONFIG_TREE_RCU) += tree.o
12	obj-$(CONFIG_PREEMPT_RCU) += tree.o
13	obj-$(CONFIG_TINY_RCU) += tiny.o	12	obj-$(CONFIG_TINY_RCU) += tiny.o
14	obj-$(CONFIG_RCU_NEED_SEGCBLIST) += rcu_segcblist.o	13	obj-$(CONFIG_RCU_NEED_SEGCBLIST) += rcu_segcblist.o

kernel/rcu/rcu.h+3−30

 }
 #endif	/* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
-void kfree(const void *);
-/*
- * Reclaim the specified callback, either by invoking it (non-lazy case)
- * or freeing it directly (lazy case).  Return true if lazy, false otherwise.
- */
-static inline bool __rcu_reclaim(const char *rn, struct rcu_head *head)
-{
-	rcu_callback_t f;
-	unsigned long offset = (unsigned long)head->func;
-	rcu_lock_acquire(&rcu_callback_map);
-	if (__is_kfree_rcu_offset(offset)) {
-		trace_rcu_invoke_kfree_callback(rn, head, offset);
-		kfree((void *)head - offset);
-		rcu_lock_release(&rcu_callback_map);
-		return true;
-	} else {
-		trace_rcu_invoke_callback(rn, head);
-		f = head->func;
-		WRITE_ONCE(head->func, (rcu_callback_t)0L);
-		f(head);
-		rcu_lock_release(&rcu_callback_map);
-		return false;
-	}
-}
 #ifdef CONFIG_RCU_STALL_COMMON
 extern int rcu_cpu_stall_ftrace_dump;
  */
 extern void resched_cpu(int cpu);
-#if defined(SRCU) || !defined(TINY_RCU)
+#if defined(CONFIG_SRCU) || !defined(CONFIG_TINY_RCU)
 #include <linux/rcu_node_tree.h>
 #define raw_lockdep_assert_held_rcu_node(p)				\
 	lockdep_assert_held(&ACCESS_PRIVATE(p, lock))
-#endif /* #if defined(SRCU) || !defined(TINY_RCU) */
+#endif /* #if defined(CONFIG_SRCU) || !defined(CONFIG_TINY_RCU) */
 #ifdef CONFIG_SRCU
 void srcu_init(void);
 	INVALID_RCU_FLAVOR
 };
-#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)
+#if defined(CONFIG_TREE_RCU)
 void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags,
 			    unsigned long *gp_seq);
 void do_trace_rcu_torture_read(const char *rcutorturename,

kernel/rcu/rcu_segcblist.c+3−22

 	rclp->head = NULL;
 	rclp->tail = &rclp->head;
 	rclp->len = 0;
-	rclp->len_lazy = 0;
 }
 /*
  * Enqueue an rcu_head structure onto the specified callback list.
- * This function assumes that the callback is non-lazy because it
- * is intended for use by no-CBs CPUs, which do not distinguish
- * between lazy and non-lazy RCU callbacks.
  */
 void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct rcu_head *rhp)
 {
 	else
 		drclp->tail = &drclp->head;
 	drclp->len = srclp->len;
-	drclp->len_lazy = srclp->len_lazy;
 	if (!rhp) {
 		rcu_cblist_init(srclp);
 	} else {
 		srclp->head = rhp;
 		srclp->tail = &rhp->next;
 		WRITE_ONCE(srclp->len, 1);
-		srclp->len_lazy = 0;
 	}
 }
 /*
  * Dequeue the oldest rcu_head structure from the specified callback
- * list.  This function assumes that the callback is non-lazy, but
+ * list.
- * the caller can later invoke rcu_cblist_dequeued_lazy() if it
- * finds otherwise (and if it cares about laziness).  This allows
- * different users to have different ways of determining laziness.
  */
 struct rcu_head *rcu_cblist_dequeue(struct rcu_cblist *rclp)
 {
 	for (i = 0; i < RCU_CBLIST_NSEGS; i++)
 		rsclp->tails[i] = &rsclp->head;
 	rcu_segcblist_set_len(rsclp, 0);
-	rsclp->len_lazy = 0;
 	rsclp->enabled = 1;
 }
 {
 	WARN_ON_ONCE(!rcu_segcblist_empty(rsclp));
 	WARN_ON_ONCE(rcu_segcblist_n_cbs(rsclp));
-	WARN_ON_ONCE(rcu_segcblist_n_lazy_cbs(rsclp));
 	rsclp->enabled = 0;
 }
  * absolutely not OK for it to ever miss posting a callback.
  */
 void rcu_segcblist_enqueue(struct rcu_segcblist *rsclp,
-			   struct rcu_head *rhp, bool lazy)
+			   struct rcu_head *rhp)
 {
 	rcu_segcblist_inc_len(rsclp);
-	if (lazy)
-		rsclp->len_lazy++;
 	smp_mb(); /* Ensure counts are updated before callback is enqueued. */
 	rhp->next = NULL;
 	WRITE_ONCE(*rsclp->tails[RCU_NEXT_TAIL], rhp);
  * period.  You have been warned.
  */
 bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp,
-			   struct rcu_head *rhp, bool lazy)
+			   struct rcu_head *rhp)
 {
 	int i;
 	if (rcu_segcblist_n_cbs(rsclp) == 0)
 		return false;
 	rcu_segcblist_inc_len(rsclp);
-	if (lazy)
-		rsclp->len_lazy++;
 	smp_mb(); /* Ensure counts are updated before callback is entrained. */
 	rhp->next = NULL;
 	for (i = RCU_NEXT_TAIL; i > RCU_DONE_TAIL; i--)
 void rcu_segcblist_extract_count(struct rcu_segcblist *rsclp,
 					       struct rcu_cblist *rclp)
 {
-	rclp->len_lazy += rsclp->len_lazy;
-	rsclp->len_lazy = 0;
 	rclp->len = rcu_segcblist_xchg_len(rsclp, 0);
 }
 void rcu_segcblist_insert_count(struct rcu_segcblist *rsclp,
 				struct rcu_cblist *rclp)
 {
-	rsclp->len_lazy += rclp->len_lazy;
 	rcu_segcblist_add_len(rsclp, rclp->len);
-	rclp->len_lazy = 0;
 	rclp->len = 0;
 }

kernel/rcu/rcu_segcblist.h+2−23

 	return READ_ONCE(rclp->len);
 }
-/*
- * Account for the fact that a previously dequeued callback turned out
- * to be marked as lazy.
- */
-static inline void rcu_cblist_dequeued_lazy(struct rcu_cblist *rclp)
-{
-	rclp->len_lazy--;
-}
 void rcu_cblist_init(struct rcu_cblist *rclp);
 void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct rcu_head *rhp);
 void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,
 #endif
 }
-/* Return number of lazy callbacks in segmented callback list. */
-static inline long rcu_segcblist_n_lazy_cbs(struct rcu_segcblist *rsclp)
-{
-	return rsclp->len_lazy;
-}
-/* Return number of lazy callbacks in segmented callback list. */
-static inline long rcu_segcblist_n_nonlazy_cbs(struct rcu_segcblist *rsclp)
-{
-	return rcu_segcblist_n_cbs(rsclp) - rsclp->len_lazy;
-}
 /*
  * Is the specified rcu_segcblist enabled, for example, not corresponding
  * to an offline CPU?
 struct rcu_head *rcu_segcblist_first_pend_cb(struct rcu_segcblist *rsclp);
 bool rcu_segcblist_nextgp(struct rcu_segcblist *rsclp, unsigned long *lp);
 void rcu_segcblist_enqueue(struct rcu_segcblist *rsclp,
-			   struct rcu_head *rhp, bool lazy);
+			   struct rcu_head *rhp);
 bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp,
-			   struct rcu_head *rhp, bool lazy);
+			   struct rcu_head *rhp);
 void rcu_segcblist_extract_count(struct rcu_segcblist *rsclp,
 				 struct rcu_cblist *rclp);
 void rcu_segcblist_extract_done_cbs(struct rcu_segcblist *rsclp,

kernel/rcu/rcuperf.c+165−8

@@ -86,6 +86,7 @@ torture_param(bool, shutdown, RCUPERF_SHUTDOWN, · 85 unmodified lines
86	"Shutdown at end of performance tests.");	86	"Shutdown at end of performance tests.");
87	torture_param(int, verbose, 1, "Enable verbose debugging printk()s");	87	torture_param(int, verbose, 1, "Enable verbose debugging printk()s");
88	torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable");	88	torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable");
		89	torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() perf test?");
89		90
90	static char *perf_type = "rcu";	91	static char *perf_type = "rcu";
91	module_param(perf_type, charp, 0444);	92	module_param(perf_type, charp, 0444);
@@ -105,8 +106,8 @@ static atomic_t n_rcu_perf_writer_finished; · 13 unmodified lines
105	static wait_queue_head_t shutdown_wq;	106	static wait_queue_head_t shutdown_wq;
106	static u64 t_rcu_perf_writer_started;	107	static u64 t_rcu_perf_writer_started;
107	static u64 t_rcu_perf_writer_finished;	108	static u64 t_rcu_perf_writer_finished;
108	static unsigned long b_rcu_perf_writer_started;	109	static unsigned long b_rcu_gp_test_started;
109	static unsigned long b_rcu_perf_writer_finished;	110	static unsigned long b_rcu_gp_test_finished;
110	static DEFINE_PER_CPU(atomic_t, n_async_inflight);	111	static DEFINE_PER_CPU(atomic_t, n_async_inflight);
111		112
112	#define MAX_MEAS 10000	113	#define MAX_MEAS 10000
@@ -378,10 +379,10 @@ rcu_perf_writer(void *arg) · 265 unmodified lines
378	if (atomic_inc_return(&n_rcu_perf_writer_started) >= nrealwriters) {	379	if (atomic_inc_return(&n_rcu_perf_writer_started) >= nrealwriters) {
379	t_rcu_perf_writer_started = t;	380	t_rcu_perf_writer_started = t;
380	if (gp_exp) {	381	if (gp_exp) {
381	b_rcu_perf_writer_started =	382	b_rcu_gp_test_started =
382	cur_ops->exp_completed() / 2;	383	cur_ops->exp_completed() / 2;
383	} else {	384	} else {
384	b_rcu_perf_writer_started = cur_ops->get_gp_seq();	385	b_rcu_gp_test_started = cur_ops->get_gp_seq();
385	}	386	}
386	}	387	}
387		388
@@ -429,10 +430,10 @@ retry: · 41 unmodified lines
429	PERFOUT_STRING("Test complete");	430	PERFOUT_STRING("Test complete");
430	t_rcu_perf_writer_finished = t;	431	t_rcu_perf_writer_finished = t;
431	if (gp_exp) {	432	if (gp_exp) {
432	b_rcu_perf_writer_finished =	433	b_rcu_gp_test_finished =
433	cur_ops->exp_completed() / 2;	434	cur_ops->exp_completed() / 2;
434	} else {	435	} else {
435	b_rcu_perf_writer_finished =	436	b_rcu_gp_test_finished =
436	cur_ops->get_gp_seq();	437	cur_ops->get_gp_seq();
437	}	438	}
438	if (shutdown) {	439	if (shutdown) {
@@ -515,8 +516,8 @@ rcu_perf_cleanup(void) · 76 unmodified lines
515	t_rcu_perf_writer_finished -	516	t_rcu_perf_writer_finished -
516	t_rcu_perf_writer_started,	517	t_rcu_perf_writer_started,
517	ngps,	518	ngps,
518	rcuperf_seq_diff(b_rcu_perf_writer_finished,	519	rcuperf_seq_diff(b_rcu_gp_test_finished,
519	b_rcu_perf_writer_started));	520	b_rcu_gp_test_started));
520	for (i = 0; i < nrealwriters; i++) {	521	for (i = 0; i < nrealwriters; i++) {
521	if (!writer_durations)	522	if (!writer_durations)
522	break;	523	break;
@@ -584,6 +585,159 @@ rcu_perf_shutdown(void *arg) · 61 unmodified lines
584	return -EINVAL;	585	return -EINVAL;
585	}	586	}
586		587
		588	/*
		589	* kfree_rcu() performance tests: Start a kfree_rcu() loop on all CPUs for number
		590	* of iterations and measure total time and number of GP for all iterations to complete.
		591	*/
		592
		593	torture_param(int, kfree_nthreads, -1, "Number of threads running loops of kfree_rcu().");
		594	torture_param(int, kfree_alloc_num, 8000, "Number of allocations and frees done in an iteration.");
		595	torture_param(int, kfree_loops, 10, "Number of loops doing kfree_alloc_num allocations and frees.");
		596
		597	static struct task_struct **kfree_reader_tasks;
		598	static int kfree_nrealthreads;
		599	static atomic_t n_kfree_perf_thread_started;
		600	static atomic_t n_kfree_perf_thread_ended;
		601
		602	struct kfree_obj {
		603	char kfree_obj[8];
		604	struct rcu_head rh;
		605	};
		606
		607	static int
		608	kfree_perf_thread(void *arg)
		609	{
		610	int i, loop = 0;
		611	long me = (long)arg;
		612	struct kfree_obj *alloc_ptr;
		613	u64 start_time, end_time;
		614
		615	VERBOSE_PERFOUT_STRING("kfree_perf_thread task started");
		616	set_cpus_allowed_ptr(current, cpumask_of(me % nr_cpu_ids));
		617	set_user_nice(current, MAX_NICE);
		618
		619	start_time = ktime_get_mono_fast_ns();
		620
		621	if (atomic_inc_return(&n_kfree_perf_thread_started) >= kfree_nrealthreads) {
		622	if (gp_exp)
		623	b_rcu_gp_test_started = cur_ops->exp_completed() / 2;
		624	else
		625	b_rcu_gp_test_started = cur_ops->get_gp_seq();
		626	}
		627
		628	do {
		629	for (i = 0; i < kfree_alloc_num; i++) {
		630	alloc_ptr = kmalloc(sizeof(struct kfree_obj), GFP_KERNEL);
		631	if (!alloc_ptr)
		632	return -ENOMEM;
		633
		634	kfree_rcu(alloc_ptr, rh);
		635	}
		636
		637	cond_resched();
		638	} while (!torture_must_stop() && ++loop < kfree_loops);
		639
		640	if (atomic_inc_return(&n_kfree_perf_thread_ended) >= kfree_nrealthreads) {
		641	end_time = ktime_get_mono_fast_ns();
		642
		643	if (gp_exp)
		644	b_rcu_gp_test_finished = cur_ops->exp_completed() / 2;
		645	else
		646	b_rcu_gp_test_finished = cur_ops->get_gp_seq();
		647
		648	pr_alert("Total time taken by all kfree'ers: %llu ns, loops: %d, batches: %ld\n",
		649	(unsigned long long)(end_time - start_time), kfree_loops,
		650	rcuperf_seq_diff(b_rcu_gp_test_finished, b_rcu_gp_test_started));
		651	if (shutdown) {
		652	smp_mb(); /* Assign before wake. */
		653	wake_up(&shutdown_wq);
		654	}
		655	}
		656
		657	torture_kthread_stopping("kfree_perf_thread");
		658	return 0;
		659	}
		660
		661	static void
		662	kfree_perf_cleanup(void)
		663	{
		664	int i;
		665
		666	if (torture_cleanup_begin())
		667	return;
		668
		669	if (kfree_reader_tasks) {
		670	for (i = 0; i < kfree_nrealthreads; i++)
		671	torture_stop_kthread(kfree_perf_thread,
		672	kfree_reader_tasks[i]);
		673	kfree(kfree_reader_tasks);
		674	}
		675
		676	torture_cleanup_end();
		677	}
		678
		679	/*
		680	* shutdown kthread. Just waits to be awakened, then shuts down system.
		681	*/
		682	static int
		683	kfree_perf_shutdown(void *arg)
		684	{
		685	do {
		686	wait_event(shutdown_wq,
		687	atomic_read(&n_kfree_perf_thread_ended) >=
		688	kfree_nrealthreads);
		689	} while (atomic_read(&n_kfree_perf_thread_ended) < kfree_nrealthreads);
		690
		691	smp_mb(); /* Wake before output. */
		692
		693	kfree_perf_cleanup();
		694	kernel_power_off();
		695	return -EINVAL;
		696	}
		697
		698	static int __init
		699	kfree_perf_init(void)
		700	{
		701	long i;
		702	int firsterr = 0;
		703
		704	kfree_nrealthreads = compute_real(kfree_nthreads);
		705	/* Start up the kthreads. */
		706	if (shutdown) {
		707	init_waitqueue_head(&shutdown_wq);
		708	firsterr = torture_create_kthread(kfree_perf_shutdown, NULL,
		709	shutdown_task);
		710	if (firsterr)
		711	goto unwind;
		712	schedule_timeout_uninterruptible(1);
		713	}
		714
		715	kfree_reader_tasks = kcalloc(kfree_nrealthreads, sizeof(kfree_reader_tasks[0]),
		716	GFP_KERNEL);
		717	if (kfree_reader_tasks == NULL) {
		718	firsterr = -ENOMEM;
		719	goto unwind;
		720	}
		721
		722	for (i = 0; i < kfree_nrealthreads; i++) {
		723	firsterr = torture_create_kthread(kfree_perf_thread, (void *)i,
		724	kfree_reader_tasks[i]);
		725	if (firsterr)
		726	goto unwind;
		727	}
		728
		729	while (atomic_read(&n_kfree_perf_thread_started) < kfree_nrealthreads)
		730	schedule_timeout_uninterruptible(1);
		731
		732	torture_init_end();
		733	return 0;
		734
		735	unwind:
		736	torture_init_end();
		737	kfree_perf_cleanup();
		738	return firsterr;
		739	}
		740
587	static int __init	741	static int __init
588	rcu_perf_init(void)	742	rcu_perf_init(void)
589	{	743	{
@@ -616,6 +770,9 @@ rcu_perf_init(void) · 26 unmodified lines
616	if (cur_ops->init)	770	if (cur_ops->init)
617	cur_ops->init();	771	cur_ops->init();
618		772
		773	if (kfree_rcu_test)
		774	return kfree_perf_init();
		775
619	nrealwriters = compute_real(nwriters);	776	nrealwriters = compute_real(nwriters);
620	nrealreaders = compute_real(nreaders);	777	nrealreaders = compute_real(nreaders);
621	atomic_set(&n_rcu_perf_reader_started, 0);	778	atomic_set(&n_rcu_perf_reader_started, 0);

kernel/rcu/rcutorture.c+81−60

@@ -1661,43 +1661,52 @@ static void rcu_torture_fwd_prog_cb(struct rcu_head *rhp) · 1660 unmodified lines
1661	struct rcu_fwd_cb {	1661	struct rcu_fwd_cb {
1662	struct rcu_head rh;	1662	struct rcu_head rh;
1663	struct rcu_fwd_cb *rfc_next;	1663	struct rcu_fwd_cb *rfc_next;
		1664	struct rcu_fwd *rfc_rfp;
1664	int rfc_gps;	1665	int rfc_gps;
1665	};	1666	};
1666	static DEFINE_SPINLOCK(rcu_fwd_lock);	1667
1667	static struct rcu_fwd_cb *rcu_fwd_cb_head;
1668	static struct rcu_fwd_cb **rcu_fwd_cb_tail = &rcu_fwd_cb_head;
1669	static long n_launders_cb;
1670	static unsigned long rcu_fwd_startat;
1671	static bool rcu_fwd_emergency_stop;
1672	#define MAX_FWD_CB_JIFFIES (8 * HZ) /* Maximum CB test duration. */	1668	#define MAX_FWD_CB_JIFFIES (8 * HZ) /* Maximum CB test duration. */
1673	#define MIN_FWD_CB_LAUNDERS 3 /* This many CB invocations to count. */	1669	#define MIN_FWD_CB_LAUNDERS 3 /* This many CB invocations to count. */
1674	#define MIN_FWD_CBS_LAUNDERED 100 /* Number of counted CBs. */	1670	#define MIN_FWD_CBS_LAUNDERED 100 /* Number of counted CBs. */
1675	#define FWD_CBS_HIST_DIV 10 /* Histogram buckets/second. */	1671	#define FWD_CBS_HIST_DIV 10 /* Histogram buckets/second. */
		1672	#define N_LAUNDERS_HIST (2 * MAX_FWD_CB_JIFFIES / (HZ / FWD_CBS_HIST_DIV))
		1673
1676	struct rcu_launder_hist {	1674	struct rcu_launder_hist {
1677	long n_launders;	1675	long n_launders;
1678	unsigned long launder_gp_seq;	1676	unsigned long launder_gp_seq;
1679	};	1677	};
1680	#define N_LAUNDERS_HIST (2 * MAX_FWD_CB_JIFFIES / (HZ / FWD_CBS_HIST_DIV))
1681	static struct rcu_launder_hist n_launders_hist[N_LAUNDERS_HIST];
1682	static unsigned long rcu_launder_gp_seq_start;
1683		1678
1684	static void rcu_torture_fwd_cb_hist(void)	1679	struct rcu_fwd {
		1680	spinlock_t rcu_fwd_lock;
		1681	struct rcu_fwd_cb *rcu_fwd_cb_head;
		1682	struct rcu_fwd_cb **rcu_fwd_cb_tail;
		1683	long n_launders_cb;
		1684	unsigned long rcu_fwd_startat;
		1685	struct rcu_launder_hist n_launders_hist[N_LAUNDERS_HIST];
		1686	unsigned long rcu_launder_gp_seq_start;
		1687	};
		1688
		1689	struct rcu_fwd *rcu_fwds;
		1690	bool rcu_fwd_emergency_stop;
		1691
		1692	static void rcu_torture_fwd_cb_hist(struct rcu_fwd *rfp)
1685	{	1693	{
1686	unsigned long gps;	1694	unsigned long gps;
1687	unsigned long gps_old;	1695	unsigned long gps_old;
1688	int i;	1696	int i;
1689	int j;	1697	int j;
1690		1698
1691	for (i = ARRAY_SIZE(n_launders_hist) - 1; i > 0; i--)	1699	for (i = ARRAY_SIZE(rfp->n_launders_hist) - 1; i > 0; i--)
1692	if (n_launders_hist[i].n_launders > 0)	1700	if (rfp->n_launders_hist[i].n_launders > 0)
1693	break;	1701	break;
1694	pr_alert("%s: Callback-invocation histogram (duration %lu jiffies):",	1702	pr_alert("%s: Callback-invocation histogram (duration %lu jiffies):",
1695	__func__, jiffies - rcu_fwd_startat);	1703	__func__, jiffies - rfp->rcu_fwd_startat);
1696	gps_old = rcu_launder_gp_seq_start;	1704	gps_old = rfp->rcu_launder_gp_seq_start;
1697	for (j = 0; j <= i; j++) {	1705	for (j = 0; j <= i; j++) {
1698	gps = n_launders_hist[j].launder_gp_seq;	1706	gps = rfp->n_launders_hist[j].launder_gp_seq;
1699	pr_cont(" %ds/%d: %ld:%ld",	1707	pr_cont(" %ds/%d: %ld:%ld",
1700	j + 1, FWD_CBS_HIST_DIV, n_launders_hist[j].n_launders,	1708	j + 1, FWD_CBS_HIST_DIV,
		1709	rfp->n_launders_hist[j].n_launders,
1701	rcutorture_seq_diff(gps, gps_old));	1710	rcutorture_seq_diff(gps, gps_old));
1702	gps_old = gps;	1711	gps_old = gps;
1703	}	1712	}
@@ -1711,26 +1720,27 @@ static void rcu_torture_fwd_cb_cr(struct rcu_head *rhp) · 7 unmodified lines
1711	int i;	1720	int i;
1712	struct rcu_fwd_cb *rfcp = container_of(rhp, struct rcu_fwd_cb, rh);	1721	struct rcu_fwd_cb *rfcp = container_of(rhp, struct rcu_fwd_cb, rh);
1713	struct rcu_fwd_cb **rfcpp;	1722	struct rcu_fwd_cb **rfcpp;
		1723	struct rcu_fwd *rfp = rfcp->rfc_rfp;
1714		1724
1715	rfcp->rfc_next = NULL;	1725	rfcp->rfc_next = NULL;
1716	rfcp->rfc_gps++;	1726	rfcp->rfc_gps++;
1717	spin_lock_irqsave(&rcu_fwd_lock, flags);	1727	spin_lock_irqsave(&rfp->rcu_fwd_lock, flags);
1718	rfcpp = rcu_fwd_cb_tail;	1728	rfcpp = rfp->rcu_fwd_cb_tail;
1719	rcu_fwd_cb_tail = &rfcp->rfc_next;	1729	rfp->rcu_fwd_cb_tail = &rfcp->rfc_next;
1720	WRITE_ONCE(*rfcpp, rfcp);	1730	WRITE_ONCE(*rfcpp, rfcp);
1721	WRITE_ONCE(n_launders_cb, n_launders_cb + 1);	1731	WRITE_ONCE(rfp->n_launders_cb, rfp->n_launders_cb + 1);
1722	i = ((jiffies - rcu_fwd_startat) / (HZ / FWD_CBS_HIST_DIV));	1732	i = ((jiffies - rfp->rcu_fwd_startat) / (HZ / FWD_CBS_HIST_DIV));
1723	if (i >= ARRAY_SIZE(n_launders_hist))	1733	if (i >= ARRAY_SIZE(rfp->n_launders_hist))
1724	i = ARRAY_SIZE(n_launders_hist) - 1;	1734	i = ARRAY_SIZE(rfp->n_launders_hist) - 1;
1725	n_launders_hist[i].n_launders++;	1735	rfp->n_launders_hist[i].n_launders++;
1726	n_launders_hist[i].launder_gp_seq = cur_ops->get_gp_seq();	1736	rfp->n_launders_hist[i].launder_gp_seq = cur_ops->get_gp_seq();
1727	spin_unlock_irqrestore(&rcu_fwd_lock, flags);	1737	spin_unlock_irqrestore(&rfp->rcu_fwd_lock, flags);
1728	}	1738	}
1729		1739
1730	// Give the scheduler a chance, even on nohz_full CPUs.	1740	// Give the scheduler a chance, even on nohz_full CPUs.
1731	static void rcu_torture_fwd_prog_cond_resched(unsigned long iter)	1741	static void rcu_torture_fwd_prog_cond_resched(unsigned long iter)
1732	{	1742	{
1733	if (IS_ENABLED(CONFIG_PREEMPT) && IS_ENABLED(CONFIG_NO_HZ_FULL)) {	1743	if (IS_ENABLED(CONFIG_PREEMPTION) && IS_ENABLED(CONFIG_NO_HZ_FULL)) {
1734	// Real call_rcu() floods hit userspace, so emulate that.	1744	// Real call_rcu() floods hit userspace, so emulate that.
1735	if (need_resched() \|\| (iter & 0xfff))	1745	if (need_resched() \|\| (iter & 0xfff))
1736	schedule();	1746	schedule();
@@ -1744,23 +1754,23 @@ static void rcu_torture_fwd_prog_cond_resched(unsigned long iter) · 7 unmodified lines
1744	* Free all callbacks on the rcu_fwd_cb_head list, either because the	1754	* Free all callbacks on the rcu_fwd_cb_head list, either because the
1745	* test is over or because we hit an OOM event.	1755	* test is over or because we hit an OOM event.
1746	*/	1756	*/
1747	static unsigned long rcu_torture_fwd_prog_cbfree(void)	1757	static unsigned long rcu_torture_fwd_prog_cbfree(struct rcu_fwd *rfp)
1748	{	1758	{
1749	unsigned long flags;	1759	unsigned long flags;
1750	unsigned long freed = 0;	1760	unsigned long freed = 0;
1751	struct rcu_fwd_cb *rfcp;	1761	struct rcu_fwd_cb *rfcp;
1752		1762
1753	for (;;) {	1763	for (;;) {
1754	spin_lock_irqsave(&rcu_fwd_lock, flags);	1764	spin_lock_irqsave(&rfp->rcu_fwd_lock, flags);
1755	rfcp = rcu_fwd_cb_head;	1765	rfcp = rfp->rcu_fwd_cb_head;
1756	if (!rfcp) {	1766	if (!rfcp) {
1757	spin_unlock_irqrestore(&rcu_fwd_lock, flags);	1767	spin_unlock_irqrestore(&rfp->rcu_fwd_lock, flags);
1758	break;	1768	break;
1759	}	1769	}
1760	rcu_fwd_cb_head = rfcp->rfc_next;	1770	rfp->rcu_fwd_cb_head = rfcp->rfc_next;
1761	if (!rcu_fwd_cb_head)	1771	if (!rfp->rcu_fwd_cb_head)
1762	rcu_fwd_cb_tail = &rcu_fwd_cb_head;	1772	rfp->rcu_fwd_cb_tail = &rfp->rcu_fwd_cb_head;
1763	spin_unlock_irqrestore(&rcu_fwd_lock, flags);	1773	spin_unlock_irqrestore(&rfp->rcu_fwd_lock, flags);
1764	kfree(rfcp);	1774	kfree(rfcp);
1765	freed++;	1775	freed++;
1766	rcu_torture_fwd_prog_cond_resched(freed);	1776	rcu_torture_fwd_prog_cond_resched(freed);
@@ -1774,7 +1784,8 @@ static unsigned long rcu_torture_fwd_prog_cbfree(void) · 7 unmodified lines
1774	}	1784	}
1775		1785
1776	/* Carry out need_resched()/cond_resched() forward-progress testing. */	1786	/* Carry out need_resched()/cond_resched() forward-progress testing. */
1777	static void rcu_torture_fwd_prog_nr(int tested, int tested_tries)	1787	static void rcu_torture_fwd_prog_nr(struct rcu_fwd *rfp,
		1788	int tested, int tested_tries)
1778	{	1789	{
1779	unsigned long cver;	1790	unsigned long cver;
1780	unsigned long dur;	1791	unsigned long dur;
@@ -1804,8 +1815,8 @@ static void rcu_torture_fwd_prog_nr(int tested, int tested_tries) · 23 unmodified lines
1804	sd = cur_ops->stall_dur() + 1;	1815	sd = cur_ops->stall_dur() + 1;
1805	sd4 = (sd + fwd_progress_div - 1) / fwd_progress_div;	1816	sd4 = (sd + fwd_progress_div - 1) / fwd_progress_div;
1806	dur = sd4 + torture_random(&trs) % (sd - sd4);	1817	dur = sd4 + torture_random(&trs) % (sd - sd4);
1807	WRITE_ONCE(rcu_fwd_startat, jiffies);	1818	WRITE_ONCE(rfp->rcu_fwd_startat, jiffies);
1808	stopat = rcu_fwd_startat + dur;	1819	stopat = rfp->rcu_fwd_startat + dur;
1809	while (time_before(jiffies, stopat) &&	1820	while (time_before(jiffies, stopat) &&
1810	!shutdown_time_arrived() &&	1821	!shutdown_time_arrived() &&
1811	!READ_ONCE(rcu_fwd_emergency_stop) && !torture_must_stop()) {	1822	!READ_ONCE(rcu_fwd_emergency_stop) && !torture_must_stop()) {
@@ -1840,7 +1851,7 @@ static void rcu_torture_fwd_prog_nr(int tested, int tested_tries) · 28 unmodified lines
1840	}	1851	}
1841		1852
1842	/* Carry out call_rcu() forward-progress testing. */	1853	/* Carry out call_rcu() forward-progress testing. */
1843	static void rcu_torture_fwd_prog_cr(void)	1854	static void rcu_torture_fwd_prog_cr(struct rcu_fwd *rfp)
1844	{	1855	{
1845	unsigned long cver;	1856	unsigned long cver;
1846	unsigned long flags;	1857	unsigned long flags;
@@ -1864,23 +1875,23 @@ static void rcu_torture_fwd_prog_cr(void) · 17 unmodified lines
1864	/* Loop continuously posting RCU callbacks. */	1875	/* Loop continuously posting RCU callbacks. */
1865	WRITE_ONCE(rcu_fwd_cb_nodelay, true);	1876	WRITE_ONCE(rcu_fwd_cb_nodelay, true);
1866	cur_ops->sync(); /* Later readers see above write. */	1877	cur_ops->sync(); /* Later readers see above write. */
1867	WRITE_ONCE(rcu_fwd_startat, jiffies);	1878	WRITE_ONCE(rfp->rcu_fwd_startat, jiffies);
1868	stopat = rcu_fwd_startat + MAX_FWD_CB_JIFFIES;	1879	stopat = rfp->rcu_fwd_startat + MAX_FWD_CB_JIFFIES;
1869	n_launders = 0;	1880	n_launders = 0;
1870	n_launders_cb = 0;	1881	rfp->n_launders_cb = 0; // Hoist initialization for multi-kthread
1871	n_launders_sa = 0;	1882	n_launders_sa = 0;
1872	n_max_cbs = 0;	1883	n_max_cbs = 0;
1873	n_max_gps = 0;	1884	n_max_gps = 0;
1874	for (i = 0; i < ARRAY_SIZE(n_launders_hist); i++)	1885	for (i = 0; i < ARRAY_SIZE(rfp->n_launders_hist); i++)
1875	n_launders_hist[i].n_launders = 0;	1886	rfp->n_launders_hist[i].n_launders = 0;
1876	cver = READ_ONCE(rcu_torture_current_version);	1887	cver = READ_ONCE(rcu_torture_current_version);
1877	gps = cur_ops->get_gp_seq();	1888	gps = cur_ops->get_gp_seq();
1878	rcu_launder_gp_seq_start = gps;	1889	rfp->rcu_launder_gp_seq_start = gps;
1879	tick_dep_set_task(current, TICK_DEP_BIT_RCU);	1890	tick_dep_set_task(current, TICK_DEP_BIT_RCU);
1880	while (time_before(jiffies, stopat) &&	1891	while (time_before(jiffies, stopat) &&
1881	!shutdown_time_arrived() &&	1892	!shutdown_time_arrived() &&
1882	!READ_ONCE(rcu_fwd_emergency_stop) && !torture_must_stop()) {	1893	!READ_ONCE(rcu_fwd_emergency_stop) && !torture_must_stop()) {
1883	rfcp = READ_ONCE(rcu_fwd_cb_head);	1894	rfcp = READ_ONCE(rfp->rcu_fwd_cb_head);
1884	rfcpn = NULL;	1895	rfcpn = NULL;
1885	if (rfcp)	1896	if (rfcp)
1886	rfcpn = READ_ONCE(rfcp->rfc_next);	1897	rfcpn = READ_ONCE(rfcp->rfc_next);
@@ -1888,7 +1899,7 @@ static void rcu_torture_fwd_prog_cr(void) · 1 unmodified line
1888	if (rfcp->rfc_gps >= MIN_FWD_CB_LAUNDERS &&	1899	if (rfcp->rfc_gps >= MIN_FWD_CB_LAUNDERS &&
1889	++n_max_gps >= MIN_FWD_CBS_LAUNDERED)	1900	++n_max_gps >= MIN_FWD_CBS_LAUNDERED)
1890	break;	1901	break;
1891	rcu_fwd_cb_head = rfcpn;	1902	rfp->rcu_fwd_cb_head = rfcpn;
1892	n_launders++;	1903	n_launders++;
1893	n_launders_sa++;	1904	n_launders_sa++;
1894	} else {	1905	} else {
@@ -1900,6 +1911,7 @@ static void rcu_torture_fwd_prog_cr(void) · 5 unmodified lines
1900	n_max_cbs++;	1911	n_max_cbs++;
1901	n_launders_sa = 0;	1912	n_launders_sa = 0;
1902	rfcp->rfc_gps = 0;	1913	rfcp->rfc_gps = 0;
		1914	rfcp->rfc_rfp = rfp;
1903	}	1915	}
1904	cur_ops->call(&rfcp->rh, rcu_torture_fwd_cb_cr);	1916	cur_ops->call(&rfcp->rh, rcu_torture_fwd_cb_cr);
1905	rcu_torture_fwd_prog_cond_resched(n_launders + n_max_cbs);	1917	rcu_torture_fwd_prog_cond_resched(n_launders + n_max_cbs);
@@ -1910,22 +1922,22 @@ static void rcu_torture_fwd_prog_cr(void) · 4 unmodified lines
1910	}	1922	}
1911	}	1923	}
1912	stoppedat = jiffies;	1924	stoppedat = jiffies;
1913	n_launders_cb_snap = READ_ONCE(n_launders_cb);	1925	n_launders_cb_snap = READ_ONCE(rfp->n_launders_cb);
1914	cver = READ_ONCE(rcu_torture_current_version) - cver;	1926	cver = READ_ONCE(rcu_torture_current_version) - cver;
1915	gps = rcutorture_seq_diff(cur_ops->get_gp_seq(), gps);	1927	gps = rcutorture_seq_diff(cur_ops->get_gp_seq(), gps);
1916	cur_ops->cb_barrier(); /* Wait for callbacks to be invoked. */	1928	cur_ops->cb_barrier(); /* Wait for callbacks to be invoked. */
1917	(void)rcu_torture_fwd_prog_cbfree();	1929	(void)rcu_torture_fwd_prog_cbfree(rfp);
1918		1930
1919	if (!torture_must_stop() && !READ_ONCE(rcu_fwd_emergency_stop) &&	1931	if (!torture_must_stop() && !READ_ONCE(rcu_fwd_emergency_stop) &&
1920	!shutdown_time_arrived()) {	1932	!shutdown_time_arrived()) {
1921	WARN_ON(n_max_gps < MIN_FWD_CBS_LAUNDERED);	1933	WARN_ON(n_max_gps < MIN_FWD_CBS_LAUNDERED);
1922	pr_alert("%s Duration %lu barrier: %lu pending %ld n_launders: %ld n_launders_sa: %ld n_max_gps: %ld n_max_cbs: %ld cver %ld gps %ld\n",	1934	pr_alert("%s Duration %lu barrier: %lu pending %ld n_launders: %ld n_launders_sa: %ld n_max_gps: %ld n_max_cbs: %ld cver %ld gps %ld\n",
1923	__func__,	1935	__func__,
1924	stoppedat - rcu_fwd_startat, jiffies - stoppedat,	1936	stoppedat - rfp->rcu_fwd_startat, jiffies - stoppedat,
1925	n_launders + n_max_cbs - n_launders_cb_snap,	1937	n_launders + n_max_cbs - n_launders_cb_snap,
1926	n_launders, n_launders_sa,	1938	n_launders, n_launders_sa,
1927	n_max_gps, n_max_cbs, cver, gps);	1939	n_max_gps, n_max_cbs, cver, gps);
1928	rcu_torture_fwd_cb_hist();	1940	rcu_torture_fwd_cb_hist(rfp);
1929	}	1941	}
1930	schedule_timeout_uninterruptible(HZ); /* Let CBs drain. */	1942	schedule_timeout_uninterruptible(HZ); /* Let CBs drain. */
1931	tick_dep_clear_task(current, TICK_DEP_BIT_RCU);	1943	tick_dep_clear_task(current, TICK_DEP_BIT_RCU);
@@ -1940,20 +1952,22 @@ static void rcu_torture_fwd_prog_cr(void) · 8 unmodified lines
1940	static int rcutorture_oom_notify(struct notifier_block *self,	1952	static int rcutorture_oom_notify(struct notifier_block *self,
1941	unsigned long notused, void *nfreed)	1953	unsigned long notused, void *nfreed)
1942	{	1954	{
		1955	struct rcu_fwd *rfp = rcu_fwds;
		1956
1943	WARN(1, "%s invoked upon OOM during forward-progress testing.\n",	1957	WARN(1, "%s invoked upon OOM during forward-progress testing.\n",
1944	__func__);	1958	__func__);
1945	rcu_torture_fwd_cb_hist();	1959	rcu_torture_fwd_cb_hist(rfp);
1946	rcu_fwd_progress_check(1 + (jiffies - READ_ONCE(rcu_fwd_startat)) / 2);	1960	rcu_fwd_progress_check(1 + (jiffies - READ_ONCE(rfp->rcu_fwd_startat)) / 2);
1947	WRITE_ONCE(rcu_fwd_emergency_stop, true);	1961	WRITE_ONCE(rcu_fwd_emergency_stop, true);
1948	smp_mb(); /* Emergency stop before free and wait to avoid hangs. */	1962	smp_mb(); /* Emergency stop before free and wait to avoid hangs. */
1949	pr_info("%s: Freed %lu RCU callbacks.\n",	1963	pr_info("%s: Freed %lu RCU callbacks.\n",
1950	__func__, rcu_torture_fwd_prog_cbfree());	1964	__func__, rcu_torture_fwd_prog_cbfree(rfp));
1951	rcu_barrier();	1965	rcu_barrier();
1952	pr_info("%s: Freed %lu RCU callbacks.\n",	1966	pr_info("%s: Freed %lu RCU callbacks.\n",
1953	__func__, rcu_torture_fwd_prog_cbfree());	1967	__func__, rcu_torture_fwd_prog_cbfree(rfp));
1954	rcu_barrier();	1968	rcu_barrier();
1955	pr_info("%s: Freed %lu RCU callbacks.\n",	1969	pr_info("%s: Freed %lu RCU callbacks.\n",
1956	__func__, rcu_torture_fwd_prog_cbfree());	1970	__func__, rcu_torture_fwd_prog_cbfree(rfp));
1957	smp_mb(); /* Frees before return to avoid redoing OOM. */	1971	smp_mb(); /* Frees before return to avoid redoing OOM. */
1958	((unsigned long )nfreed)++; /* Forward progress CBs freed! */	1972	((unsigned long )nfreed)++; /* Forward progress CBs freed! */
1959	pr_info("%s returning after OOM processing.\n", __func__);	1973	pr_info("%s returning after OOM processing.\n", __func__);
@@ -1967,6 +1981,7 @@ static struct notifier_block rcutorture_oom_nb = { · 7 unmodified lines
1967	/* Carry out grace-period forward-progress testing. */	1981	/* Carry out grace-period forward-progress testing. */
1968	static int rcu_torture_fwd_prog(void *args)	1982	static int rcu_torture_fwd_prog(void *args)
1969	{	1983	{
		1984	struct rcu_fwd *rfp = args;
1970	int tested = 0;	1985	int tested = 0;
1971	int tested_tries = 0;	1986	int tested_tries = 0;
1972		1987
@@ -1978,8 +1993,8 @@ static int rcu_torture_fwd_prog(void *args) · 5 unmodified lines
1978	schedule_timeout_interruptible(fwd_progress_holdoff * HZ);	1993	schedule_timeout_interruptible(fwd_progress_holdoff * HZ);
1979	WRITE_ONCE(rcu_fwd_emergency_stop, false);	1994	WRITE_ONCE(rcu_fwd_emergency_stop, false);
1980	register_oom_notifier(&rcutorture_oom_nb);	1995	register_oom_notifier(&rcutorture_oom_nb);
1981	rcu_torture_fwd_prog_nr(&tested, &tested_tries);	1996	rcu_torture_fwd_prog_nr(rfp, &tested, &tested_tries);
1982	rcu_torture_fwd_prog_cr();	1997	rcu_torture_fwd_prog_cr(rfp);
1983	unregister_oom_notifier(&rcutorture_oom_nb);	1998	unregister_oom_notifier(&rcutorture_oom_nb);
1984		1999
1985	/* Avoid slow periods, better to test when busy. */	2000	/* Avoid slow periods, better to test when busy. */
@@ -1995,6 +2010,8 @@ static int rcu_torture_fwd_prog(void *args) · 9 unmodified lines
1995	/* If forward-progress checking is requested and feasible, spawn the thread. */	2010	/* If forward-progress checking is requested and feasible, spawn the thread. */
1996	static int __init rcu_torture_fwd_prog_init(void)	2011	static int __init rcu_torture_fwd_prog_init(void)
1997	{	2012	{
		2013	struct rcu_fwd *rfp;
		2014
1998	if (!fwd_progress)	2015	if (!fwd_progress)
1999	return 0; /* Not requested, so don't do it. */	2016	return 0; /* Not requested, so don't do it. */
2000	if (!cur_ops->stall_dur \|\| cur_ops->stall_dur() <= 0 \|\|	2017	if (!cur_ops->stall_dur \|\| cur_ops->stall_dur() <= 0 \|\|
@@ -2013,8 +2030,12 @@ static int __init rcu_torture_fwd_prog_init(void) · 12 unmodified lines
2013	fwd_progress_holdoff = 1;	2030	fwd_progress_holdoff = 1;
2014	if (fwd_progress_div <= 0)	2031	if (fwd_progress_div <= 0)
2015	fwd_progress_div = 4;	2032	fwd_progress_div = 4;
2016	return torture_create_kthread(rcu_torture_fwd_prog,	2033	rfp = kzalloc(sizeof(*rfp), GFP_KERNEL);
2017	NULL, fwd_prog_task);	2034	if (!rfp)
		2035	return -ENOMEM;
		2036	spin_lock_init(&rfp->rcu_fwd_lock);
		2037	rfp->rcu_fwd_cb_tail = &rfp->rcu_fwd_cb_head;
		2038	return torture_create_kthread(rcu_torture_fwd_prog, rfp, fwd_prog_task);
2018	}	2039	}
2019		2040
2020	/* Callback function for RCU barrier testing. */	2041	/* Callback function for RCU barrier testing. */

kernel/rcu/srcutiny.c+1−1

@@ -103,7 +103,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock); · 102 unmodified lines
103		103
104	/*	104	/*
105	* Workqueue handler to drive one grace period and invoke any callbacks	105	* Workqueue handler to drive one grace period and invoke any callbacks
106	* that become ready as a result. Single-CPU and !PREEMPT operation	106	* that become ready as a result. Single-CPU and !PREEMPTION operation
107	* means that we get away with murder on synchronization. ;-)	107	* means that we get away with murder on synchronization. ;-)
108	*/	108	*/
109	void srcu_drive_gp(struct work_struct *wp)	109	void srcu_drive_gp(struct work_struct *wp)

kernel/rcu/srcutree.c+6−5

 	idx = rcu_seq_state(ssp->srcu_gp_seq);
 	WARN_ON_ONCE(idx != SRCU_STATE_SCAN2);
 	cbdelay = srcu_get_delay(ssp);
-	ssp->srcu_last_gp_end = ktime_get_mono_fast_ns();
+	WRITE_ONCE(ssp->srcu_last_gp_end, ktime_get_mono_fast_ns());
 	rcu_seq_end(&ssp->srcu_gp_seq);
 	gpseq = rcu_seq_current(&ssp->srcu_gp_seq);
 	if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, gpseq))
 	unsigned long flags;
 	struct srcu_data *sdp;
 	unsigned long t;
+	unsigned long tlast;
 	/* If the local srcu_data structure has callbacks, not idle.  */
 	local_irq_save(flags);
 	/* First, see if enough time has passed since the last GP. */
 	t = ktime_get_mono_fast_ns();
+	tlast = READ_ONCE(ssp->srcu_last_gp_end);
 	if (exp_holdoff == 0 ||
-	    time_in_range_open(t, ssp->srcu_last_gp_end,
+	    time_in_range_open(t, tlast, tlast + exp_holdoff))
-			       ssp->srcu_last_gp_end + exp_holdoff))
 		return false; /* Too soon after last GP. */
 	/* Next, check for probable idleness. */
 	local_irq_save(flags);
 	sdp = this_cpu_ptr(ssp->sda);
 	spin_lock_rcu_node(sdp);
-	rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp, false);
+	rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp);
 	rcu_segcblist_advance(&sdp->srcu_cblist,
 			      rcu_seq_current(&ssp->srcu_gp_seq));
 	s = rcu_seq_snap(&ssp->srcu_gp_seq);
 		sdp->srcu_barrier_head.func = srcu_barrier_cb;
 		debug_rcu_head_queue(&sdp->srcu_barrier_head);
 		if (!rcu_segcblist_entrain(&sdp->srcu_cblist,
-					   &sdp->srcu_barrier_head, 0)) {
+					   &sdp->srcu_barrier_head)) {
 			debug_rcu_head_unqueue(&sdp->srcu_barrier_head);
 			atomic_dec(&ssp->srcu_barrier_cpu_cnt);
 		}

kernel/rcu/tiny.c+27−1

@@ -22,6 +22,7 @@ · 21 unmodified lines
22	#include <linux/time.h>	22	#include <linux/time.h>
23	#include <linux/cpu.h>	23	#include <linux/cpu.h>
24	#include <linux/prefetch.h>	24	#include <linux/prefetch.h>
		25	#include <linux/slab.h>
25		26
26	#include "rcu.h"	27	#include "rcu.h"
27		28
@@ -73,6 +74,31 @@ void rcu_sched_clock_irq(int user) · 45 unmodified lines
73	}	74	}
74	}	75	}
75		76
		77	/*
		78	* Reclaim the specified callback, either by invoking it for non-kfree cases or
		79	* freeing it directly (for kfree). Return true if kfreeing, false otherwise.
		80	*/
		81	static inline bool rcu_reclaim_tiny(struct rcu_head *head)
		82	{
		83	rcu_callback_t f;
		84	unsigned long offset = (unsigned long)head->func;
		85
		86	rcu_lock_acquire(&rcu_callback_map);
		87	if (__is_kfree_rcu_offset(offset)) {
		88	trace_rcu_invoke_kfree_callback("", head, offset);
		89	kfree((void *)head - offset);
		90	rcu_lock_release(&rcu_callback_map);
		91	return true;
		92	}
		93
		94	trace_rcu_invoke_callback("", head);
		95	f = head->func;
		96	WRITE_ONCE(head->func, (rcu_callback_t)0L);
		97	f(head);
		98	rcu_lock_release(&rcu_callback_map);
		99	return false;
		100	}
		101
76	/* Invoke the RCU callbacks whose grace period has elapsed. */	102	/* Invoke the RCU callbacks whose grace period has elapsed. */
77	static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused)	103	static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused)
78	{	104	{
@@ -100,7 +126,7 @@ static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused · 21 unmodified lines
100	prefetch(next);	126	prefetch(next);
101	debug_rcu_head_unqueue(list);	127	debug_rcu_head_unqueue(list);
102	local_bh_disable();	128	local_bh_disable();
103	__rcu_reclaim("", list);	129	rcu_reclaim_tiny(list);
104	local_bh_enable();	130	local_bh_enable();
105	list = next;	131	list = next;
106	}	132	}

kernel/rcu/tree.c+272−52

@@ -43,7 +43,6 @@ · 42 unmodified lines
43	#include <uapi/linux/sched/types.h>	43	#include <uapi/linux/sched/types.h>
44	#include <linux/prefetch.h>	44	#include <linux/prefetch.h>
45	#include <linux/delay.h>	45	#include <linux/delay.h>
46	#include <linux/stop_machine.h>
47	#include <linux/random.h>	46	#include <linux/random.h>
48	#include <linux/trace_events.h>	47	#include <linux/trace_events.h>
49	#include <linux/suspend.h>	48	#include <linux/suspend.h>
@@ -55,6 +54,7 @@ · 5 unmodified lines
55	#include <linux/oom.h>	54	#include <linux/oom.h>
56	#include <linux/smpboot.h>	55	#include <linux/smpboot.h>
57	#include <linux/jiffies.h>	56	#include <linux/jiffies.h>
		57	#include <linux/slab.h>
58	#include <linux/sched/isolation.h>	58	#include <linux/sched/isolation.h>
59	#include <linux/sched/clock.h>	59	#include <linux/sched/clock.h>
60	#include "../time/tick-internal.h"	60	#include "../time/tick-internal.h"
@@ -84,7 +84,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = { · 23 unmodified lines
84	.dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE,	84	.dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE,
85	.dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),	85	.dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),
86	};	86	};
87	struct rcu_state rcu_state = {	87	static struct rcu_state rcu_state = {
88	.level = { &rcu_state.node[0] },	88	.level = { &rcu_state.node[0] },
89	.gp_state = RCU_GP_IDLE,	89	.gp_state = RCU_GP_IDLE,
90	.gp_seq = (0UL - 300UL) << RCU_SEQ_CTR_SHIFT,	90	.gp_seq = (0UL - 300UL) << RCU_SEQ_CTR_SHIFT,
@@ -188,7 +188,7 @@ EXPORT_SYMBOL_GPL(rcu_get_gp_kthreads_prio); · 97 unmodified lines
188	* held, but the bit corresponding to the current CPU will be stable	188	* held, but the bit corresponding to the current CPU will be stable
189	* in most contexts.	189	* in most contexts.
190	*/	190	*/
191	unsigned long rcu_rnp_online_cpus(struct rcu_node *rnp)	191	static unsigned long rcu_rnp_online_cpus(struct rcu_node *rnp)
192	{	192	{
193	return READ_ONCE(rnp->qsmaskinitnext);	193	return READ_ONCE(rnp->qsmaskinitnext);
194	}	194	}
@@ -294,7 +294,7 @@ static void rcu_dynticks_eqs_online(void) · 99 unmodified lines
294	*	294	*
295	* No ordering, as we are sampling CPU-local information.	295	* No ordering, as we are sampling CPU-local information.
296	*/	296	*/
297	bool rcu_dynticks_curr_cpu_in_eqs(void)	297	static bool rcu_dynticks_curr_cpu_in_eqs(void)
298	{	298	{
299	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);	299	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
300		300
@@ -305,7 +305,7 @@ bool rcu_dynticks_curr_cpu_in_eqs(void) · 4 unmodified lines
305	* Snapshot the ->dynticks counter with full ordering so as to allow	305	* Snapshot the ->dynticks counter with full ordering so as to allow
306	* stable comparison of this counter with past and future snapshots.	306	* stable comparison of this counter with past and future snapshots.
307	*/	307	*/
308	int rcu_dynticks_snap(struct rcu_data *rdp)	308	static int rcu_dynticks_snap(struct rcu_data *rdp)
309	{	309	{
310	int snap = atomic_add_return(0, &rdp->dynticks);	310	int snap = atomic_add_return(0, &rdp->dynticks);
311		311
@@ -528,16 +528,6 @@ static struct rcu_node *rcu_get_root(void) · 216 unmodified lines
528	return &rcu_state.node[0];	528	return &rcu_state.node[0];
529	}	529	}
530		530
531	/*
532	* Convert a ->gp_state value to a character string.
533	*/
534	static const char *gp_state_getname(short gs)
535	{
536	if (gs < 0 \|\| gs >= ARRAY_SIZE(gp_state_names))
537	return "???";
538	return gp_state_names[gs];
539	}
540
541	/*	531	/*
542	* Send along grace-period-related data for rcutorture diagnostics.	532	* Send along grace-period-related data for rcutorture diagnostics.
543	*/	533	*/
@@ -577,7 +567,7 @@ static void rcu_eqs_enter(bool user) · 33 unmodified lines
577	}	567	}
578		568
579	lockdep_assert_irqs_disabled();	569	lockdep_assert_irqs_disabled();
580	trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, rdp->dynticks);	570	trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks));
581	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));	571	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
582	rdp = this_cpu_ptr(&rcu_data);	572	rdp = this_cpu_ptr(&rcu_data);
583	do_nocb_deferred_wakeup(rdp);	573	do_nocb_deferred_wakeup(rdp);
@@ -650,14 +640,15 @@ static __always_inline void rcu_nmi_exit_common(bool irq) · 66 unmodified lines
650	* leave it in non-RCU-idle state.	640	* leave it in non-RCU-idle state.
651	*/	641	*/
652	if (rdp->dynticks_nmi_nesting != 1) {	642	if (rdp->dynticks_nmi_nesting != 1) {
653	trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2, rdp->dynticks);	643	trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2,
		644	atomic_read(&rdp->dynticks));
654	WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */	645	WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */
655	rdp->dynticks_nmi_nesting - 2);	646	rdp->dynticks_nmi_nesting - 2);
656	return;	647	return;
657	}	648	}
658		649
659	/* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */	650	/* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */
660	trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, rdp->dynticks);	651	trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, atomic_read(&rdp->dynticks));
661	WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */	652	WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */
662		653
663	if (irq)	654	if (irq)
@@ -744,7 +735,7 @@ static void rcu_eqs_exit(bool user) · 80 unmodified lines
744	rcu_dynticks_task_exit();	735	rcu_dynticks_task_exit();
745	rcu_dynticks_eqs_exit();	736	rcu_dynticks_eqs_exit();
746	rcu_cleanup_after_idle();	737	rcu_cleanup_after_idle();
747	trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, rdp->dynticks);	738	trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, atomic_read(&rdp->dynticks));
748	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));	739	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
749	WRITE_ONCE(rdp->dynticks_nesting, 1);	740	WRITE_ONCE(rdp->dynticks_nesting, 1);
750	WARN_ON_ONCE(rdp->dynticks_nmi_nesting);	741	WARN_ON_ONCE(rdp->dynticks_nmi_nesting);
@@ -800,8 +791,8 @@ void rcu_user_exit(void) · 49 unmodified lines
800	*/	791	*/
801	static __always_inline void rcu_nmi_enter_common(bool irq)	792	static __always_inline void rcu_nmi_enter_common(bool irq)
802	{	793	{
803	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
804	long incby = 2;	794	long incby = 2;
		795	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
805		796
806	/* Complain about underflow. */	797	/* Complain about underflow. */
807	WARN_ON_ONCE(rdp->dynticks_nmi_nesting < 0);	798	WARN_ON_ONCE(rdp->dynticks_nmi_nesting < 0);
@@ -828,12 +819,17 @@ static __always_inline void rcu_nmi_enter_common(bool irq) · 20 unmodified lines
828	} else if (tick_nohz_full_cpu(rdp->cpu) &&	819	} else if (tick_nohz_full_cpu(rdp->cpu) &&
829	rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&	820	rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&
830	READ_ONCE(rdp->rcu_urgent_qs) && !rdp->rcu_forced_tick) {	821	READ_ONCE(rdp->rcu_urgent_qs) && !rdp->rcu_forced_tick) {
831	rdp->rcu_forced_tick = true;	822	raw_spin_lock_rcu_node(rdp->mynode);
832	tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);	823	// Recheck under lock.
		824	if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
		825	rdp->rcu_forced_tick = true;
		826	tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
		827	}
		828	raw_spin_unlock_rcu_node(rdp->mynode);
833	}	829	}
834	trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="),	830	trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="),
835	rdp->dynticks_nmi_nesting,	831	rdp->dynticks_nmi_nesting,
836	rdp->dynticks_nmi_nesting + incby, rdp->dynticks);	832	rdp->dynticks_nmi_nesting + incby, atomic_read(&rdp->dynticks));
837	WRITE_ONCE(rdp->dynticks_nmi_nesting, /* Prevent store tearing. */	833	WRITE_ONCE(rdp->dynticks_nmi_nesting, /* Prevent store tearing. */
838	rdp->dynticks_nmi_nesting + incby);	834	rdp->dynticks_nmi_nesting + incby);
839	barrier();	835	barrier();
@@ -898,6 +894,7 @@ void rcu_irq_enter_irqson(void) · 58 unmodified lines
898	*/	894	*/
899	static void rcu_disable_urgency_upon_qs(struct rcu_data *rdp)	895	static void rcu_disable_urgency_upon_qs(struct rcu_data *rdp)
900	{	896	{
		897	raw_lockdep_assert_held_rcu_node(rdp->mynode);
901	WRITE_ONCE(rdp->rcu_urgent_qs, false);	898	WRITE_ONCE(rdp->rcu_urgent_qs, false);
902	WRITE_ONCE(rdp->rcu_need_heavy_qs, false);	899	WRITE_ONCE(rdp->rcu_need_heavy_qs, false);
903	if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick) {	900	if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick) {
@@ -1934,7 +1931,7 @@ rcu_report_unblock_qs_rnp(struct rcu_node *rnp, unsigned long flags) · 1030 unmodified lines
1934	struct rcu_node *rnp_p;	1931	struct rcu_node *rnp_p;
1935		1932
1936	raw_lockdep_assert_held_rcu_node(rnp);	1933	raw_lockdep_assert_held_rcu_node(rnp);
1937	if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_PREEMPTION)) \|\|	1934	if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_PREEMPT_RCU)) \|\|
1938	WARN_ON_ONCE(rcu_preempt_blocked_readers_cgp(rnp)) \|\|	1935	WARN_ON_ONCE(rcu_preempt_blocked_readers_cgp(rnp)) \|\|
1939	rnp->qsmask != 0) {	1936	rnp->qsmask != 0) {
1940	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	1937	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
@@ -2146,7 +2143,6 @@ static void rcu_do_batch(struct rcu_data *rdp) · 205 unmodified lines
2146	/* If no callbacks are ready, just return. */	2143	/* If no callbacks are ready, just return. */
2147	if (!rcu_segcblist_ready_cbs(&rdp->cblist)) {	2144	if (!rcu_segcblist_ready_cbs(&rdp->cblist)) {
2148	trace_rcu_batch_start(rcu_state.name,	2145	trace_rcu_batch_start(rcu_state.name,
2149	rcu_segcblist_n_lazy_cbs(&rdp->cblist),
2150	rcu_segcblist_n_cbs(&rdp->cblist), 0);	2146	rcu_segcblist_n_cbs(&rdp->cblist), 0);
2151	trace_rcu_batch_end(rcu_state.name, 0,	2147	trace_rcu_batch_end(rcu_state.name, 0,
2152	!rcu_segcblist_empty(&rdp->cblist),	2148	!rcu_segcblist_empty(&rdp->cblist),
@@ -2168,7 +2164,6 @@ static void rcu_do_batch(struct rcu_data *rdp) · 15 unmodified lines
2168	if (unlikely(bl > 100))	2164	if (unlikely(bl > 100))
2169	tlimit = local_clock() + rcu_resched_ns;	2165	tlimit = local_clock() + rcu_resched_ns;
2170	trace_rcu_batch_start(rcu_state.name,	2166	trace_rcu_batch_start(rcu_state.name,
2171	rcu_segcblist_n_lazy_cbs(&rdp->cblist),
2172	rcu_segcblist_n_cbs(&rdp->cblist), bl);	2167	rcu_segcblist_n_cbs(&rdp->cblist), bl);
2173	rcu_segcblist_extract_done_cbs(&rdp->cblist, &rcl);	2168	rcu_segcblist_extract_done_cbs(&rdp->cblist, &rcl);
2174	if (offloaded)	2169	if (offloaded)
@@ -2179,9 +2174,19 @@ static void rcu_do_batch(struct rcu_data *rdp) · 4 unmodified lines
2179	tick_dep_set_task(current, TICK_DEP_BIT_RCU);	2174	tick_dep_set_task(current, TICK_DEP_BIT_RCU);
2180	rhp = rcu_cblist_dequeue(&rcl);	2175	rhp = rcu_cblist_dequeue(&rcl);
2181	for (; rhp; rhp = rcu_cblist_dequeue(&rcl)) {	2176	for (; rhp; rhp = rcu_cblist_dequeue(&rcl)) {
		2177	rcu_callback_t f;
		2178
2182	debug_rcu_head_unqueue(rhp);	2179	debug_rcu_head_unqueue(rhp);
2183	if (__rcu_reclaim(rcu_state.name, rhp))	2180
2184	rcu_cblist_dequeued_lazy(&rcl);	2181	rcu_lock_acquire(&rcu_callback_map);
		2182	trace_rcu_invoke_callback(rcu_state.name, rhp);
		2183
		2184	f = rhp->func;
		2185	WRITE_ONCE(rhp->func, (rcu_callback_t)0L);
		2186	f(rhp);
		2187
		2188	rcu_lock_release(&rcu_callback_map);
		2189
2185	/*	2190	/*
2186	* Stop only if limit reached and CPU has something to do.	2191	* Stop only if limit reached and CPU has something to do.
2187	* Note: The rcl structure counts down from zero.	2192	* Note: The rcl structure counts down from zero.
@@ -2294,7 +2299,7 @@ static void force_qs_rnp(int (f)(struct rcu_data rdp)) · 106 unmodified lines
2294	mask = 0;	2299	mask = 0;
2295	raw_spin_lock_irqsave_rcu_node(rnp, flags);	2300	raw_spin_lock_irqsave_rcu_node(rnp, flags);
2296	if (rnp->qsmask == 0) {	2301	if (rnp->qsmask == 0) {
2297	if (!IS_ENABLED(CONFIG_PREEMPTION) \|\|	2302	if (!IS_ENABLED(CONFIG_PREEMPT_RCU) \|\|
2298	rcu_preempt_blocked_readers_cgp(rnp)) {	2303	rcu_preempt_blocked_readers_cgp(rnp)) {
2299	/*	2304	/*
2300	* No point in scanning bits because they	2305	* No point in scanning bits because they
@@ -2308,14 +2313,11 @@ static void force_qs_rnp(int (f)(struct rcu_data rdp)) · 7 unmodified lines
2308	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	2313	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
2309	continue;	2314	continue;
2310	}	2315	}
2311	for_each_leaf_node_possible_cpu(rnp, cpu) {	2316	for_each_leaf_node_cpu_mask(rnp, cpu, rnp->qsmask) {
2312	unsigned long bit = leaf_node_cpu_bit(rnp, cpu);	2317	rdp = per_cpu_ptr(&rcu_data, cpu);
2313	if ((rnp->qsmask & bit) != 0) {	2318	if (f(rdp)) {
2314	rdp = per_cpu_ptr(&rcu_data, cpu);	2319	mask \|= rdp->grpmask;
2315	if (f(rdp)) {	2320	rcu_disable_urgency_upon_qs(rdp);
2316	mask \|= bit;
2317	rcu_disable_urgency_upon_qs(rdp);
2318	}
2319	}	2321	}
2320	}	2322	}
2321	if (mask != 0) {	2323	if (mask != 0) {
@@ -2474,8 +2476,8 @@ static void rcu_cpu_kthread(unsigned int cpu) · 152 unmodified lines
2474	char work, *workp = this_cpu_ptr(&rcu_data.rcu_cpu_has_work);	2476	char work, *workp = this_cpu_ptr(&rcu_data.rcu_cpu_has_work);
2475	int spincnt;	2477	int spincnt;
2476		2478
		2479	trace_rcu_utilization(TPS("Start CPU kthread@rcu_run"));
2477	for (spincnt = 0; spincnt < 10; spincnt++) {	2480	for (spincnt = 0; spincnt < 10; spincnt++) {
2478	trace_rcu_utilization(TPS("Start CPU kthread@rcu_wait"));
2479	local_bh_disable();	2481	local_bh_disable();
2480	*statusp = RCU_KTHREAD_RUNNING;	2482	*statusp = RCU_KTHREAD_RUNNING;
2481	local_irq_disable();	2483	local_irq_disable();
@@ -2583,7 +2585,7 @@ static void rcu_leak_callback(struct rcu_head *rhp) · 101 unmodified lines
2583	* is expected to specify a CPU.	2585	* is expected to specify a CPU.
2584	*/	2586	*/
2585	static void	2587	static void
2586	__call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy)	2588	__call_rcu(struct rcu_head *head, rcu_callback_t func)
2587	{	2589	{
2588	unsigned long flags;	2590	unsigned long flags;
2589	struct rcu_data *rdp;	2591	struct rcu_data *rdp;
@@ -2618,18 +2620,17 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy) · 28 unmodified lines
2618	if (rcu_segcblist_empty(&rdp->cblist))	2620	if (rcu_segcblist_empty(&rdp->cblist))
2619	rcu_segcblist_init(&rdp->cblist);	2621	rcu_segcblist_init(&rdp->cblist);
2620	}	2622	}
		2623
2621	if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags))	2624	if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags))
2622	return; // Enqueued onto ->nocb_bypass, so just leave.	2625	return; // Enqueued onto ->nocb_bypass, so just leave.
2623	/* If we get here, rcu_nocb_try_bypass() acquired ->nocb_lock. */	2626	/* If we get here, rcu_nocb_try_bypass() acquired ->nocb_lock. */
2624	rcu_segcblist_enqueue(&rdp->cblist, head, lazy);	2627	rcu_segcblist_enqueue(&rdp->cblist, head);
2625	if (__is_kfree_rcu_offset((unsigned long)func))	2628	if (__is_kfree_rcu_offset((unsigned long)func))
2626	trace_rcu_kfree_callback(rcu_state.name, head,	2629	trace_rcu_kfree_callback(rcu_state.name, head,
2627	(unsigned long)func,	2630	(unsigned long)func,
2628	rcu_segcblist_n_lazy_cbs(&rdp->cblist),
2629	rcu_segcblist_n_cbs(&rdp->cblist));	2631	rcu_segcblist_n_cbs(&rdp->cblist));
2630	else	2632	else
2631	trace_rcu_callback(rcu_state.name, head,	2633	trace_rcu_callback(rcu_state.name, head,
2632	rcu_segcblist_n_lazy_cbs(&rdp->cblist),
2633	rcu_segcblist_n_cbs(&rdp->cblist));	2634	rcu_segcblist_n_cbs(&rdp->cblist));
2634		2635
2635	/* Go handle any RCU core processing required. */	2636	/* Go handle any RCU core processing required. */
@@ -2679,28 +2680,230 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy) · 43 unmodified lines
2679	*/	2680	*/
2680	void call_rcu(struct rcu_head *head, rcu_callback_t func)	2681	void call_rcu(struct rcu_head *head, rcu_callback_t func)
2681	{	2682	{
2682	__call_rcu(head, func, 0);	2683	__call_rcu(head, func);
2683	}	2684	}
2684	EXPORT_SYMBOL_GPL(call_rcu);	2685	EXPORT_SYMBOL_GPL(call_rcu);
2685		2686
		2687
		2688	/* Maximum number of jiffies to wait before draining a batch. */
		2689	#define KFREE_DRAIN_JIFFIES (HZ / 50)
		2690	#define KFREE_N_BATCHES 2
		2691
		2692	/**
		2693	* struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests
		2694	* @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
		2695	* @head_free: List of kfree_rcu() objects waiting for a grace period
		2696	* @krcp: Pointer to @kfree_rcu_cpu structure
		2697	*/
		2698
		2699	struct kfree_rcu_cpu_work {
		2700	struct rcu_work rcu_work;
		2701	struct rcu_head *head_free;
		2702	struct kfree_rcu_cpu *krcp;
		2703	};
		2704
		2705	/**
		2706	* struct kfree_rcu_cpu - batch up kfree_rcu() requests for RCU grace period
		2707	* @head: List of kfree_rcu() objects not yet waiting for a grace period
		2708	* @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period
		2709	* @lock: Synchronize access to this structure
		2710	* @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES
		2711	* @monitor_todo: Tracks whether a @monitor_work delayed work is pending
		2712	* @initialized: The @lock and @rcu_work fields have been initialized
		2713	*
		2714	* This is a per-CPU structure. The reason that it is not included in
		2715	* the rcu_data structure is to permit this code to be extracted from
		2716	* the RCU files. Such extraction could allow further optimization of
		2717	* the interactions with the slab allocators.
		2718	*/
		2719	struct kfree_rcu_cpu {
		2720	struct rcu_head *head;
		2721	struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES];
		2722	spinlock_t lock;
		2723	struct delayed_work monitor_work;
		2724	bool monitor_todo;
		2725	bool initialized;
		2726	};
		2727
		2728	static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc);
		2729
		2730	/*
		2731	* This function is invoked in workqueue context after a grace period.
		2732	* It frees all the objects queued on ->head_free.
		2733	*/
		2734	static void kfree_rcu_work(struct work_struct *work)
		2735	{
		2736	unsigned long flags;
		2737	struct rcu_head head, next;
		2738	struct kfree_rcu_cpu *krcp;
		2739	struct kfree_rcu_cpu_work *krwp;
		2740
		2741	krwp = container_of(to_rcu_work(work),
		2742	struct kfree_rcu_cpu_work, rcu_work);
		2743	krcp = krwp->krcp;
		2744	spin_lock_irqsave(&krcp->lock, flags);
		2745	head = krwp->head_free;
		2746	krwp->head_free = NULL;
		2747	spin_unlock_irqrestore(&krcp->lock, flags);
		2748
		2749	// List "head" is now private, so traverse locklessly.
		2750	for (; head; head = next) {
		2751	unsigned long offset = (unsigned long)head->func;
		2752
		2753	next = head->next;
		2754	// Potentially optimize with kfree_bulk in future.
		2755	debug_rcu_head_unqueue(head);
		2756	rcu_lock_acquire(&rcu_callback_map);
		2757	trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset);
		2758
		2759	if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) {
		2760	/* Could be optimized with kfree_bulk() in future. */
		2761	kfree((void *)head - offset);
		2762	}
		2763
		2764	rcu_lock_release(&rcu_callback_map);
		2765	cond_resched_tasks_rcu_qs();
		2766	}
		2767	}
		2768
2686	/*	2769	/*
2687	* Queue an RCU callback for lazy invocation after a grace period.	2770	* Schedule the kfree batch RCU work to run in workqueue context after a GP.
2688	* This will likely be later named something like "call_rcu_lazy()",	2771	*
2689	* but this change will require some way of tagging the lazy RCU	2772	* This function is invoked by kfree_rcu_monitor() when the KFREE_DRAIN_JIFFIES
2690	* callbacks in the list of pending callbacks. Until then, this	2773	* timeout has been reached.
2691	* function may only be called from __kfree_rcu().	2774	*/
		2775	static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp)
		2776	{
		2777	int i;
		2778	struct kfree_rcu_cpu_work *krwp = NULL;
		2779
		2780	lockdep_assert_held(&krcp->lock);
		2781	for (i = 0; i < KFREE_N_BATCHES; i++)
		2782	if (!krcp->krw_arr[i].head_free) {
		2783	krwp = &(krcp->krw_arr[i]);
		2784	break;
		2785	}
		2786
		2787	// If a previous RCU batch is in progress, we cannot immediately
		2788	// queue another one, so return false to tell caller to retry.
		2789	if (!krwp)
		2790	return false;
		2791
		2792	krwp->head_free = krcp->head;
		2793	krcp->head = NULL;
		2794	INIT_RCU_WORK(&krwp->rcu_work, kfree_rcu_work);
		2795	queue_rcu_work(system_wq, &krwp->rcu_work);
		2796	return true;
		2797	}
		2798
		2799	static inline void kfree_rcu_drain_unlock(struct kfree_rcu_cpu *krcp,
		2800	unsigned long flags)
		2801	{
		2802	// Attempt to start a new batch.
		2803	krcp->monitor_todo = false;
		2804	if (queue_kfree_rcu_work(krcp)) {
		2805	// Success! Our job is done here.
		2806	spin_unlock_irqrestore(&krcp->lock, flags);
		2807	return;
		2808	}
		2809
		2810	// Previous RCU batch still in progress, try again later.
		2811	krcp->monitor_todo = true;
		2812	schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES);
		2813	spin_unlock_irqrestore(&krcp->lock, flags);
		2814	}
		2815
		2816	/*
		2817	* This function is invoked after the KFREE_DRAIN_JIFFIES timeout.
		2818	* It invokes kfree_rcu_drain_unlock() to attempt to start another batch.
		2819	*/
		2820	static void kfree_rcu_monitor(struct work_struct *work)
		2821	{
		2822	unsigned long flags;
		2823	struct kfree_rcu_cpu *krcp = container_of(work, struct kfree_rcu_cpu,
		2824	monitor_work.work);
		2825
		2826	spin_lock_irqsave(&krcp->lock, flags);
		2827	if (krcp->monitor_todo)
		2828	kfree_rcu_drain_unlock(krcp, flags);
		2829	else
		2830	spin_unlock_irqrestore(&krcp->lock, flags);
		2831	}
		2832
		2833	/*
		2834	* Queue a request for lazy invocation of kfree() after a grace period.
		2835	*
		2836	* Each kfree_call_rcu() request is added to a batch. The batch will be drained
		2837	* every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch
		2838	* will be kfree'd in workqueue context. This allows us to:
		2839	*
		2840	* 1. Batch requests together to reduce the number of grace periods during
		2841	* heavy kfree_rcu() load.
		2842	*
		2843	* 2. It makes it possible to use kfree_bulk() on a large number of
		2844	* kfree_rcu() requests thus reducing cache misses and the per-object
		2845	* overhead of kfree().
2692	*/	2846	*/
2693	void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)	2847	void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
2694	{	2848	{
2695	__call_rcu(head, func, 1);	2849	unsigned long flags;
		2850	struct kfree_rcu_cpu *krcp;
		2851
		2852	local_irq_save(flags); // For safely calling this_cpu_ptr().
		2853	krcp = this_cpu_ptr(&krc);
		2854	if (krcp->initialized)
		2855	spin_lock(&krcp->lock);
		2856
		2857	// Queue the object but don't yet schedule the batch.
		2858	if (debug_rcu_head_queue(head)) {
		2859	// Probable double kfree_rcu(), just leak.
		2860	WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
		2861	__func__, head);
		2862	goto unlock_return;
		2863	}
		2864	head->func = func;
		2865	head->next = krcp->head;
		2866	krcp->head = head;
		2867
		2868	// Set timer to drain after KFREE_DRAIN_JIFFIES.
		2869	if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING &&
		2870	!krcp->monitor_todo) {
		2871	krcp->monitor_todo = true;
		2872	schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES);
		2873	}
		2874
		2875	unlock_return:
		2876	if (krcp->initialized)
		2877	spin_unlock(&krcp->lock);
		2878	local_irq_restore(flags);
2696	}	2879	}
2697	EXPORT_SYMBOL_GPL(kfree_call_rcu);	2880	EXPORT_SYMBOL_GPL(kfree_call_rcu);
2698		2881
		2882	void __init kfree_rcu_scheduler_running(void)
		2883	{
		2884	int cpu;
		2885	unsigned long flags;
		2886
		2887	for_each_online_cpu(cpu) {
		2888	struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
		2889
		2890	spin_lock_irqsave(&krcp->lock, flags);
		2891	if (!krcp->head \|\| krcp->monitor_todo) {
		2892	spin_unlock_irqrestore(&krcp->lock, flags);
		2893	continue;
		2894	}
		2895	krcp->monitor_todo = true;
		2896	schedule_delayed_work_on(cpu, &krcp->monitor_work,
		2897	KFREE_DRAIN_JIFFIES);
		2898	spin_unlock_irqrestore(&krcp->lock, flags);
		2899	}
		2900	}
		2901
2699	/*	2902	/*
2700	* During early boot, any blocking grace-period wait automatically	2903	* During early boot, any blocking grace-period wait automatically
2701	* implies a grace period. Later on, this is never the case for PREEMPT.	2904	* implies a grace period. Later on, this is never the case for PREEMPTION.
2702	*	2905	*
2703	* Howevr, because a context switch is a grace period for !PREEMPT, any	2906	* Howevr, because a context switch is a grace period for !PREEMPTION, any
2704	* blocking grace-period wait automatically implies a grace period if	2907	* blocking grace-period wait automatically implies a grace period if
2705	* there is only one CPU online at any point time during execution of	2908	* there is only one CPU online at any point time during execution of
2706	* either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to	2909	* either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to
@@ -2896,7 +3099,7 @@ static void rcu_barrier_func(void *unused) · 189 unmodified lines
2896	debug_rcu_head_queue(&rdp->barrier_head);	3099	debug_rcu_head_queue(&rdp->barrier_head);
2897	rcu_nocb_lock(rdp);	3100	rcu_nocb_lock(rdp);
2898	WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies));	3101	WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies));
2899	if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head, 0)) {	3102	if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) {
2900	atomic_inc(&rcu_state.barrier_cpu_count);	3103	atomic_inc(&rcu_state.barrier_cpu_count);
2901	} else {	3104	} else {
2902	debug_rcu_head_unqueue(&rdp->barrier_head);	3105	debug_rcu_head_unqueue(&rdp->barrier_head);
@@ -3557,12 +3760,29 @@ static void __init rcu_dump_rcu_node_tree(void) · 654 unmodified lines
3557	struct workqueue_struct *rcu_gp_wq;	3760	struct workqueue_struct *rcu_gp_wq;
3558	struct workqueue_struct *rcu_par_gp_wq;	3761	struct workqueue_struct *rcu_par_gp_wq;
3559		3762
		3763	static void __init kfree_rcu_batch_init(void)
		3764	{
		3765	int cpu;
		3766	int i;
		3767
		3768	for_each_possible_cpu(cpu) {
		3769	struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
		3770
		3771	spin_lock_init(&krcp->lock);
		3772	for (i = 0; i < KFREE_N_BATCHES; i++)
		3773	krcp->krw_arr[i].krcp = krcp;
		3774	INIT_DELAYED_WORK(&krcp->monitor_work, kfree_rcu_monitor);
		3775	krcp->initialized = true;
		3776	}
		3777	}
		3778
3560	void __init rcu_init(void)	3779	void __init rcu_init(void)
3561	{	3780	{
3562	int cpu;	3781	int cpu;
3563		3782
3564	rcu_early_boot_tests();	3783	rcu_early_boot_tests();
3565		3784
		3785	kfree_rcu_batch_init();
3566	rcu_bootup_announce();	3786	rcu_bootup_announce();
3567	rcu_init_geometry();	3787	rcu_init_geometry();
3568	rcu_init_one();	3788	rcu_init_one();

kernel/rcu/tree.h+1−17

 #include <linux/cpumask.h>
 #include <linux/seqlock.h>
 #include <linux/swait.h>
-#include <linux/stop_machine.h>
 #include <linux/rcu_node_tree.h>
 #include "rcu_segcblist.h"
 	bool rcu_need_heavy_qs;		/* GP old, so heavy quiescent state! */
 	bool rcu_urgent_qs;		/* GP old need light quiescent state. */
 	bool rcu_forced_tick;		/* Forced tick to provide QS. */
+	bool rcu_forced_tick_exp;	/*   ... provide QS to expedited GP. */
 #ifdef CONFIG_RCU_FAST_NO_HZ
-	bool all_lazy;			/* All CPU's CBs lazy at idle start? */
 	unsigned long last_accelerate;	/* Last jiffy CBs were accelerated. */
 	unsigned long last_advance_all;	/* Last jiffy CBs were all advanced. */
 	int tick_nohz_enabled_snap;	/* Previously seen value from sysfs. */
 #define RCU_GP_CLEANUP   7	/* Grace-period cleanup started. */
 #define RCU_GP_CLEANED   8	/* Grace-period cleanup complete. */
-static const char * const gp_state_names[] = {
-	"RCU_GP_IDLE",
-	"RCU_GP_WAIT_GPS",
-	"RCU_GP_DONE_GPS",
-	"RCU_GP_ONOFF",
-	"RCU_GP_INIT",
-	"RCU_GP_WAIT_FQS",
-	"RCU_GP_DOING_FQS",
-	"RCU_GP_CLEANUP",
-	"RCU_GP_CLEANED",
-};
 /*
  * In order to export the rcu_state name to the tracing tools, it
  * needs to be added in the __tracepoint_string section.
 #define RCU_NAME rcu_name
 #endif /* #else #ifdef CONFIG_TRACING */
-int rcu_dynticks_snap(struct rcu_data *rdp);
 /* Forward declarations for tree_plugin.h */
 static void rcu_bootup_announce(void);
 static void rcu_qs(void);
 static int rcu_print_task_exp_stall(struct rcu_node *rnp);
 static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp);
 static void rcu_flavor_sched_clock_irq(int user);
-void call_rcu(struct rcu_head *head, rcu_callback_t func);
 static void dump_blkd_tasks(struct rcu_node *rnp, int ncheck);
 static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags);
 static void rcu_preempt_boost_start_gp(struct rcu_node *rnp);

kernel/rcu/tree_exp.h+91−56

@@ -21,7 +21,7 @@ static void rcu_exp_gp_seq_start(void) · 20 unmodified lines
21	}	21	}
22		22
23	/*	23	/*
24	* Return then value that expedited-grace-period counter will have	24	* Return the value that the expedited-grace-period counter will have
25	* at the end of the current grace period.	25	* at the end of the current grace period.
26	*/	26	*/
27	static __maybe_unused unsigned long rcu_exp_gp_seq_endval(void)	27	static __maybe_unused unsigned long rcu_exp_gp_seq_endval(void)
@@ -39,7 +39,9 @@ static void rcu_exp_gp_seq_end(void) · 11 unmodified lines
39	}	39	}
40		40
41	/*	41	/*
42	* Take a snapshot of the expedited-grace-period counter.	42	* Take a snapshot of the expedited-grace-period counter, which is the
		43	* earliest value that will indicate that a full grace period has
		44	* elapsed since the current time.
43	*/	45	*/
44	static unsigned long rcu_exp_gp_seq_snap(void)	46	static unsigned long rcu_exp_gp_seq_snap(void)
45	{	47	{
@@ -134,7 +136,7 @@ static void __maybe_unused sync_exp_reset_tree(void) · 88 unmodified lines
134	rcu_for_each_node_breadth_first(rnp) {	136	rcu_for_each_node_breadth_first(rnp) {
135	raw_spin_lock_irqsave_rcu_node(rnp, flags);	137	raw_spin_lock_irqsave_rcu_node(rnp, flags);
136	WARN_ON_ONCE(rnp->expmask);	138	WARN_ON_ONCE(rnp->expmask);
137	rnp->expmask = rnp->expmaskinit;	139	WRITE_ONCE(rnp->expmask, rnp->expmaskinit);
138	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	140	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
139	}	141	}
140	}	142	}
@@ -143,31 +145,26 @@ static void __maybe_unused sync_exp_reset_tree(void) · 2 unmodified lines
143	* Return non-zero if there is no RCU expedited grace period in progress	145	* Return non-zero if there is no RCU expedited grace period in progress
144	* for the specified rcu_node structure, in other words, if all CPUs and	146	* for the specified rcu_node structure, in other words, if all CPUs and
145	* tasks covered by the specified rcu_node structure have done their bit	147	* tasks covered by the specified rcu_node structure have done their bit
146	* for the current expedited grace period. Works only for preemptible	148	* for the current expedited grace period.
147	* RCU -- other RCU implementation use other means.
148	*
149	* Caller must hold the specificed rcu_node structure's ->lock
150	*/	149	*/
151	static bool sync_rcu_preempt_exp_done(struct rcu_node *rnp)	150	static bool sync_rcu_exp_done(struct rcu_node *rnp)
152	{	151	{
153	raw_lockdep_assert_held_rcu_node(rnp);	152	raw_lockdep_assert_held_rcu_node(rnp);
154
155	return rnp->exp_tasks == NULL &&	153	return rnp->exp_tasks == NULL &&
156	READ_ONCE(rnp->expmask) == 0;	154	READ_ONCE(rnp->expmask) == 0;
157	}	155	}
158		156
159	/*	157	/*
160	* Like sync_rcu_preempt_exp_done(), but this function assumes the caller	158	* Like sync_rcu_exp_done(), but where the caller does not hold the
161	* doesn't hold the rcu_node's ->lock, and will acquire and release the lock	159	* rcu_node's ->lock.
162	* itself
163	*/	160	*/
164	static bool sync_rcu_preempt_exp_done_unlocked(struct rcu_node *rnp)	161	static bool sync_rcu_exp_done_unlocked(struct rcu_node *rnp)
165	{	162	{
166	unsigned long flags;	163	unsigned long flags;
167	bool ret;	164	bool ret;
168		165
169	raw_spin_lock_irqsave_rcu_node(rnp, flags);	166	raw_spin_lock_irqsave_rcu_node(rnp, flags);
170	ret = sync_rcu_preempt_exp_done(rnp);	167	ret = sync_rcu_exp_done(rnp);
171	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	168	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
172		169
173	return ret;	170	return ret;
@@ -181,8 +178,6 @@ static bool sync_rcu_preempt_exp_done_unlocked(struct rcu_node *rnp) · 7 unmodified lines
181	* which the task was queued or to one of that rcu_node structure's ancestors,	178	* which the task was queued or to one of that rcu_node structure's ancestors,
182	* recursively up the tree. (Calm down, calm down, we do the recursion	179	* recursively up the tree. (Calm down, calm down, we do the recursion
183	* iteratively!)	180	* iteratively!)
184	*
185	* Caller must hold the specified rcu_node structure's ->lock.
186	*/	181	*/
187	static void __rcu_report_exp_rnp(struct rcu_node *rnp,	182	static void __rcu_report_exp_rnp(struct rcu_node *rnp,
188	bool wake, unsigned long flags)	183	bool wake, unsigned long flags)
@@ -190,8 +185,9 @@ static void __rcu_report_exp_rnp(struct rcu_node *rnp, · 1 unmodified line
190	{	185	{
191	unsigned long mask;	186	unsigned long mask;
192		187
		188	raw_lockdep_assert_held_rcu_node(rnp);
193	for (;;) {	189	for (;;) {
194	if (!sync_rcu_preempt_exp_done(rnp)) {	190	if (!sync_rcu_exp_done(rnp)) {
195	if (!rnp->expmask)	191	if (!rnp->expmask)
196	rcu_initiate_boost(rnp, flags);	192	rcu_initiate_boost(rnp, flags);
197	else	193	else
@@ -211,7 +207,7 @@ static void __rcu_report_exp_rnp(struct rcu_node *rnp, · 13 unmodified lines
211	rnp = rnp->parent;	207	rnp = rnp->parent;
212	raw_spin_lock_rcu_node(rnp); /* irqs already disabled */	208	raw_spin_lock_rcu_node(rnp); /* irqs already disabled */
213	WARN_ON_ONCE(!(rnp->expmask & mask));	209	WARN_ON_ONCE(!(rnp->expmask & mask));
214	rnp->expmask &= ~mask;	210	WRITE_ONCE(rnp->expmask, rnp->expmask & ~mask);
215	}	211	}
216	}	212	}
217		213
@@ -234,14 +230,23 @@ static void __maybe_unused rcu_report_exp_rnp(struct rcu_node *rnp, bool wake) · 16 unmodified lines
234	static void rcu_report_exp_cpu_mult(struct rcu_node *rnp,	230	static void rcu_report_exp_cpu_mult(struct rcu_node *rnp,
235	unsigned long mask, bool wake)	231	unsigned long mask, bool wake)
236	{	232	{
		233	int cpu;
237	unsigned long flags;	234	unsigned long flags;
		235	struct rcu_data *rdp;
238		236
239	raw_spin_lock_irqsave_rcu_node(rnp, flags);	237	raw_spin_lock_irqsave_rcu_node(rnp, flags);
240	if (!(rnp->expmask & mask)) {	238	if (!(rnp->expmask & mask)) {
241	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	239	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
242	return;	240	return;
243	}	241	}
244	rnp->expmask &= ~mask;	242	WRITE_ONCE(rnp->expmask, rnp->expmask & ~mask);
		243	for_each_leaf_node_cpu_mask(rnp, cpu, mask) {
		244	rdp = per_cpu_ptr(&rcu_data, cpu);
		245	if (!IS_ENABLED(CONFIG_NO_HZ_FULL) \|\| !rdp->rcu_forced_tick_exp)
		246	continue;
		247	rdp->rcu_forced_tick_exp = false;
		248	tick_dep_clear_cpu(cpu, TICK_DEP_BIT_RCU_EXP);
		249	}
245	__rcu_report_exp_rnp(rnp, wake, flags); /* Releases rnp->lock. */	250	__rcu_report_exp_rnp(rnp, wake, flags); /* Releases rnp->lock. */
246	}	251	}
247		252
@@ -345,8 +350,8 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) · 97 unmodified lines
345	/* Each pass checks a CPU for identity, offline, and idle. */	350	/* Each pass checks a CPU for identity, offline, and idle. */
346	mask_ofl_test = 0;	351	mask_ofl_test = 0;
347	for_each_leaf_node_cpu_mask(rnp, cpu, rnp->expmask) {	352	for_each_leaf_node_cpu_mask(rnp, cpu, rnp->expmask) {
348	unsigned long mask = leaf_node_cpu_bit(rnp, cpu);
349	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);	353	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
		354	unsigned long mask = rdp->grpmask;
350	int snap;	355	int snap;
351		356
352	if (raw_smp_processor_id() == cpu \|\|	357	if (raw_smp_processor_id() == cpu \|\|
@@ -372,12 +377,10 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) · 19 unmodified lines
372	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	377	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
373		378
374	/* IPI the remaining CPUs for expedited quiescent state. */	379	/* IPI the remaining CPUs for expedited quiescent state. */
375	for_each_leaf_node_cpu_mask(rnp, cpu, rnp->expmask) {	380	for_each_leaf_node_cpu_mask(rnp, cpu, mask_ofl_ipi) {
376	unsigned long mask = leaf_node_cpu_bit(rnp, cpu);
377	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);	381	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
		382	unsigned long mask = rdp->grpmask;
378		383
379	if (!(mask_ofl_ipi & mask))
380	continue;
381	retry_ipi:	384	retry_ipi:
382	if (rcu_dynticks_in_eqs_since(rdp, rdp->exp_dynticks_snap)) {	385	if (rcu_dynticks_in_eqs_since(rdp, rdp->exp_dynticks_snap)) {
383	mask_ofl_test \|= mask;	386	mask_ofl_test \|= mask;
@@ -389,10 +392,10 @@ retry_ipi: · 5 unmodified lines
389	}	392	}
390	ret = smp_call_function_single(cpu, rcu_exp_handler, NULL, 0);	393	ret = smp_call_function_single(cpu, rcu_exp_handler, NULL, 0);
391	put_cpu();	394	put_cpu();
392	if (!ret) {	395	/* The CPU will report the QS in response to the IPI. */
393	mask_ofl_ipi &= ~mask;	396	if (!ret)
394	continue;	397	continue;
395	}	398
396	/* Failed, raced with CPU hotplug operation. */	399	/* Failed, raced with CPU hotplug operation. */
397	raw_spin_lock_irqsave_rcu_node(rnp, flags);	400	raw_spin_lock_irqsave_rcu_node(rnp, flags);
398	if ((rnp->qsmaskinitnext & mask) &&	401	if ((rnp->qsmaskinitnext & mask) &&
@@ -403,13 +406,12 @@ retry_ipi: · 4 unmodified lines
403	schedule_timeout_uninterruptible(1);	406	schedule_timeout_uninterruptible(1);
404	goto retry_ipi;	407	goto retry_ipi;
405	}	408	}
406	/* CPU really is offline, so we can ignore it. */	409	/* CPU really is offline, so we must report its QS. */
407	if (!(rnp->expmask & mask))	410	if (rnp->expmask & mask)
408	mask_ofl_ipi &= ~mask;	411	mask_ofl_test \|= mask;
409	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	412	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
410	}	413	}
411	/* Report quiescent states for those that went offline. */	414	/* Report quiescent states for those that went offline. */
412	mask_ofl_test \|= mask_ofl_ipi;
413	if (mask_ofl_test)	415	if (mask_ofl_test)
414	rcu_report_exp_cpu_mult(rnp, mask_ofl_test, false);	416	rcu_report_exp_cpu_mult(rnp, mask_ofl_test, false);
415	}	417	}
@@ -456,29 +458,62 @@ static void sync_rcu_exp_select_cpus(void) · 40 unmodified lines
456	flush_work(&rnp->rew.rew_work);	458	flush_work(&rnp->rew.rew_work);
457	}	459	}
458		460
459	static void synchronize_sched_expedited_wait(void)	461	/*
		462	* Wait for the expedited grace period to elapse, within time limit.
		463	* If the time limit is exceeded without the grace period elapsing,
		464	* return false, otherwise return true.
		465	*/
		466	static bool synchronize_rcu_expedited_wait_once(long tlimit)
		467	{
		468	int t;
		469	struct rcu_node *rnp_root = rcu_get_root();
		470
		471	t = swait_event_timeout_exclusive(rcu_state.expedited_wq,
		472	sync_rcu_exp_done_unlocked(rnp_root),
		473	tlimit);
		474	// Workqueues should not be signaled.
		475	if (t > 0 \|\| sync_rcu_exp_done_unlocked(rnp_root))
		476	return true;
		477	WARN_ON(t < 0); /* workqueues should not be signaled. */
		478	return false;
		479	}
		480
		481	/*
		482	* Wait for the expedited grace period to elapse, issuing any needed
		483	* RCU CPU stall warnings along the way.
		484	*/
		485	static void synchronize_rcu_expedited_wait(void)
460	{	486	{
461	int cpu;	487	int cpu;
462	unsigned long jiffies_stall;	488	unsigned long jiffies_stall;
463	unsigned long jiffies_start;	489	unsigned long jiffies_start;
464	unsigned long mask;	490	unsigned long mask;
465	int ndetected;	491	int ndetected;
		492	struct rcu_data *rdp;
466	struct rcu_node *rnp;	493	struct rcu_node *rnp;
467	struct rcu_node *rnp_root = rcu_get_root();	494	struct rcu_node *rnp_root = rcu_get_root();
468	int ret;
469		495
470	trace_rcu_exp_grace_period(rcu_state.name, rcu_exp_gp_seq_endval(), TPS("startwait"));	496	trace_rcu_exp_grace_period(rcu_state.name, rcu_exp_gp_seq_endval(), TPS("startwait"));
471	jiffies_stall = rcu_jiffies_till_stall_check();	497	jiffies_stall = rcu_jiffies_till_stall_check();
472	jiffies_start = jiffies;	498	jiffies_start = jiffies;
		499	if (IS_ENABLED(CONFIG_NO_HZ_FULL)) {
		500	if (synchronize_rcu_expedited_wait_once(1))
		501	return;
		502	rcu_for_each_leaf_node(rnp) {
		503	for_each_leaf_node_cpu_mask(rnp, cpu, rnp->expmask) {
		504	rdp = per_cpu_ptr(&rcu_data, cpu);
		505	if (rdp->rcu_forced_tick_exp)
		506	continue;
		507	rdp->rcu_forced_tick_exp = true;
		508	tick_dep_set_cpu(cpu, TICK_DEP_BIT_RCU_EXP);
		509	}
		510	}
		511	WARN_ON_ONCE(1);
		512	}
473		513
474	for (;;) {	514	for (;;) {
475	ret = swait_event_timeout_exclusive(	515	if (synchronize_rcu_expedited_wait_once(jiffies_stall))
476	rcu_state.expedited_wq,
477	sync_rcu_preempt_exp_done_unlocked(rnp_root),
478	jiffies_stall);
479	if (ret > 0 \|\| sync_rcu_preempt_exp_done_unlocked(rnp_root))
480	return;	516	return;
481	WARN_ON(ret < 0); /* workqueues should not be signaled. */
482	if (rcu_cpu_stall_suppress)	517	if (rcu_cpu_stall_suppress)
483	continue;	518	continue;
484	panic_on_rcu_stall();	519	panic_on_rcu_stall();
@@ -491,7 +526,7 @@ static void synchronize_sched_expedited_wait(void) · 6 unmodified lines
491	struct rcu_data *rdp;	526	struct rcu_data *rdp;
492		527
493	mask = leaf_node_cpu_bit(rnp, cpu);	528	mask = leaf_node_cpu_bit(rnp, cpu);
494	if (!(rnp->expmask & mask))	529	if (!(READ_ONCE(rnp->expmask) & mask))
495	continue;	530	continue;
496	ndetected++;	531	ndetected++;
497	rdp = per_cpu_ptr(&rcu_data, cpu);	532	rdp = per_cpu_ptr(&rcu_data, cpu);
@@ -503,17 +538,18 @@ static void synchronize_sched_expedited_wait(void) · 5 unmodified lines
503	}	538	}
504	pr_cont(" } %lu jiffies s: %lu root: %#lx/%c\n",	539	pr_cont(" } %lu jiffies s: %lu root: %#lx/%c\n",
505	jiffies - jiffies_start, rcu_state.expedited_sequence,	540	jiffies - jiffies_start, rcu_state.expedited_sequence,
506	rnp_root->expmask, ".T"[!!rnp_root->exp_tasks]);	541	READ_ONCE(rnp_root->expmask),
		542	".T"[!!rnp_root->exp_tasks]);
507	if (ndetected) {	543	if (ndetected) {
508	pr_err("blocking rcu_node structures:");	544	pr_err("blocking rcu_node structures:");
509	rcu_for_each_node_breadth_first(rnp) {	545	rcu_for_each_node_breadth_first(rnp) {
510	if (rnp == rnp_root)	546	if (rnp == rnp_root)
511	continue; /* printed unconditionally */	547	continue; /* printed unconditionally */
512	if (sync_rcu_preempt_exp_done_unlocked(rnp))	548	if (sync_rcu_exp_done_unlocked(rnp))
513	continue;	549	continue;
514	pr_cont(" l=%u:%d-%d:%#lx/%c",	550	pr_cont(" l=%u:%d-%d:%#lx/%c",
515	rnp->level, rnp->grplo, rnp->grphi,	551	rnp->level, rnp->grplo, rnp->grphi,
516	rnp->expmask,	552	READ_ONCE(rnp->expmask),
517	".T"[!!rnp->exp_tasks]);	553	".T"[!!rnp->exp_tasks]);
518	}	554	}
519	pr_cont("\n");	555	pr_cont("\n");
@@ -521,7 +557,7 @@ static void synchronize_sched_expedited_wait(void) · 1 unmodified line
521	rcu_for_each_leaf_node(rnp) {	557	rcu_for_each_leaf_node(rnp) {
522	for_each_leaf_node_possible_cpu(rnp, cpu) {	558	for_each_leaf_node_possible_cpu(rnp, cpu) {
523	mask = leaf_node_cpu_bit(rnp, cpu);	559	mask = leaf_node_cpu_bit(rnp, cpu);
524	if (!(rnp->expmask & mask))	560	if (!(READ_ONCE(rnp->expmask) & mask))
525	continue;	561	continue;
526	dump_cpu_task(cpu);	562	dump_cpu_task(cpu);
527	}	563	}
@@ -540,15 +576,14 @@ static void rcu_exp_wait_wake(unsigned long s) · 12 unmodified lines
540	{	576	{
541	struct rcu_node *rnp;	577	struct rcu_node *rnp;
542		578
543	synchronize_sched_expedited_wait();	579	synchronize_rcu_expedited_wait();
544	rcu_exp_gp_seq_end();
545	trace_rcu_exp_grace_period(rcu_state.name, s, TPS("end"));
546		580
547	/*	581	// Switch over to wakeup mode, allowing the next GP to proceed.
548	* Switch over to wakeup mode, allowing the next GP, but -only- the	582	// End the previous grace period only after acquiring the mutex
549	* next GP, to proceed.	583	// to ensure that only one GP runs concurrently with wakeups.
550	*/
551	mutex_lock(&rcu_state.exp_wake_mutex);	584	mutex_lock(&rcu_state.exp_wake_mutex);
		585	rcu_exp_gp_seq_end();
		586	trace_rcu_exp_grace_period(rcu_state.name, s, TPS("end"));
552		587
553	rcu_for_each_node_breadth_first(rnp) {	588	rcu_for_each_node_breadth_first(rnp) {
554	if (ULONG_CMP_LT(READ_ONCE(rnp->exp_seq_rq), s)) {	589	if (ULONG_CMP_LT(READ_ONCE(rnp->exp_seq_rq), s)) {
@@ -559,7 +594,7 @@ static void rcu_exp_wait_wake(unsigned long s) · 4 unmodified lines
559	spin_unlock(&rnp->exp_lock);	594	spin_unlock(&rnp->exp_lock);
560	}	595	}
561	smp_mb(); /* All above changes before wakeup. */	596	smp_mb(); /* All above changes before wakeup. */
562	wake_up_all(&rnp->exp_wq[rcu_seq_ctr(rcu_state.expedited_sequence) & 0x3]);	597	wake_up_all(&rnp->exp_wq[rcu_seq_ctr(s) & 0x3]);
563	}	598	}
564	trace_rcu_exp_grace_period(rcu_state.name, s, TPS("endwake"));	599	trace_rcu_exp_grace_period(rcu_state.name, s, TPS("endwake"));
565	mutex_unlock(&rcu_state.exp_wake_mutex);	600	mutex_unlock(&rcu_state.exp_wake_mutex);
@@ -610,7 +645,7 @@ static void rcu_exp_handler(void *unused) · 44 unmodified lines
610	* critical section. If also enabled or idle, immediately	645	* critical section. If also enabled or idle, immediately
611	* report the quiescent state, otherwise defer.	646	* report the quiescent state, otherwise defer.
612	*/	647	*/
613	if (!t->rcu_read_lock_nesting) {	648	if (!rcu_preempt_depth()) {
614	if (!(preempt_count() & (PREEMPT_MASK \| SOFTIRQ_MASK)) \|\|	649	if (!(preempt_count() & (PREEMPT_MASK \| SOFTIRQ_MASK)) \|\|
615	rcu_dynticks_curr_cpu_in_eqs()) {	650	rcu_dynticks_curr_cpu_in_eqs()) {
616	rcu_report_exp_rdp(rdp);	651	rcu_report_exp_rdp(rdp);
@@ -634,7 +669,7 @@ static void rcu_exp_handler(void *unused) · 17 unmodified lines
634	* can have caused this quiescent state to already have been	669	* can have caused this quiescent state to already have been
635	* reported, so we really do need to check ->expmask.	670	* reported, so we really do need to check ->expmask.
636	*/	671	*/
637	if (t->rcu_read_lock_nesting > 0) {	672	if (rcu_preempt_depth() > 0) {
638	raw_spin_lock_irqsave_rcu_node(rnp, flags);	673	raw_spin_lock_irqsave_rcu_node(rnp, flags);
639	if (rnp->expmask & rdp->grpmask) {	674	if (rnp->expmask & rdp->grpmask) {
640	rdp->exp_deferred_qs = true;	675	rdp->exp_deferred_qs = true;
@@ -670,7 +705,7 @@ static void rcu_exp_handler(void *unused) · 29 unmodified lines
670	}	705	}
671	}	706	}
672		707
673	/* PREEMPT=y, so no PREEMPT=n expedited grace period to clean up after. */	708	/* PREEMPTION=y, so no PREEMPTION=n expedited grace period to clean up after. */
674	static void sync_sched_exp_online_cleanup(int cpu)	709	static void sync_sched_exp_online_cleanup(int cpu)
675	{	710	{
676	}	711	}
@@ -785,7 +820,7 @@ static int rcu_print_task_exp_stall(struct rcu_node *rnp) · 108 unmodified lines
785	* implementations, it is still unfriendly to real-time workloads, so is	820	* implementations, it is still unfriendly to real-time workloads, so is
786	* thus not recommended for any sort of common-case code. In fact, if	821	* thus not recommended for any sort of common-case code. In fact, if
787	* you are using synchronize_rcu_expedited() in a loop, please restructure	822	* you are using synchronize_rcu_expedited() in a loop, please restructure
788	* your code to batch your updates, and then Use a single synchronize_rcu()	823	* your code to batch your updates, and then use a single synchronize_rcu()
789	* instead.	824	* instead.
790	*	825	*
791	* This has the same semantics as (but is more brutal than) synchronize_rcu().	826	* This has the same semantics as (but is more brutal than) synchronize_rcu().

kernel/rcu/tree_plugin.h+78−90

@@ -220,7 +220,7 @@ static void rcu_preempt_ctxt_queue(struct rcu_node rnp, struct rcu_data rdp) · 219 unmodified lines
220	* blocked tasks.	220	* blocked tasks.
221	*/	221	*/
222	if (!rnp->gp_tasks && (blkd_state & RCU_GP_BLKD)) {	222	if (!rnp->gp_tasks && (blkd_state & RCU_GP_BLKD)) {
223	rnp->gp_tasks = &t->rcu_node_entry;	223	WRITE_ONCE(rnp->gp_tasks, &t->rcu_node_entry);
224	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq);	224	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq);
225	}	225	}
226	if (!rnp->exp_tasks && (blkd_state & RCU_EXP_BLKD))	226	if (!rnp->exp_tasks && (blkd_state & RCU_EXP_BLKD))
@@ -290,8 +290,8 @@ void rcu_note_context_switch(bool preempt) · 63 unmodified lines
290		290
291	trace_rcu_utilization(TPS("Start context switch"));	291	trace_rcu_utilization(TPS("Start context switch"));
292	lockdep_assert_irqs_disabled();	292	lockdep_assert_irqs_disabled();
293	WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0);	293	WARN_ON_ONCE(!preempt && rcu_preempt_depth() > 0);
294	if (t->rcu_read_lock_nesting > 0 &&	294	if (rcu_preempt_depth() > 0 &&
295	!t->rcu_read_unlock_special.b.blocked) {	295	!t->rcu_read_unlock_special.b.blocked) {
296		296
297	/* Possibly blocking in an RCU read-side critical section. */	297	/* Possibly blocking in an RCU read-side critical section. */
@@ -340,7 +340,7 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch); · 42 unmodified lines
340	*/	340	*/
341	static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp)	341	static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp)
342	{	342	{
343	return rnp->gp_tasks != NULL;	343	return READ_ONCE(rnp->gp_tasks) != NULL;
344	}	344	}
345		345
346	/* Bias and limit values for ->rcu_read_lock_nesting. */	346	/* Bias and limit values for ->rcu_read_lock_nesting. */
@@ -348,6 +348,21 @@ static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp) · 1 unmodified line
348	#define RCU_NEST_NMAX (-INT_MAX / 2)	348	#define RCU_NEST_NMAX (-INT_MAX / 2)
349	#define RCU_NEST_PMAX (INT_MAX / 2)	349	#define RCU_NEST_PMAX (INT_MAX / 2)
350		350
		351	static void rcu_preempt_read_enter(void)
		352	{
		353	current->rcu_read_lock_nesting++;
		354	}
		355
		356	static void rcu_preempt_read_exit(void)
		357	{
		358	current->rcu_read_lock_nesting--;
		359	}
		360
		361	static void rcu_preempt_depth_set(int val)
		362	{
		363	current->rcu_read_lock_nesting = val;
		364	}
		365
351	/*	366	/*
352	* Preemptible RCU implementation for rcu_read_lock().	367	* Preemptible RCU implementation for rcu_read_lock().
353	* Just increment ->rcu_read_lock_nesting, shared state will be updated	368	* Just increment ->rcu_read_lock_nesting, shared state will be updated
@@ -355,9 +370,9 @@ static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp) · 1 unmodified line
355	*/	370	*/
356	void __rcu_read_lock(void)	371	void __rcu_read_lock(void)
357	{	372	{
358	current->rcu_read_lock_nesting++;	373	rcu_preempt_read_enter();
359	if (IS_ENABLED(CONFIG_PROVE_LOCKING))	374	if (IS_ENABLED(CONFIG_PROVE_LOCKING))
360	WARN_ON_ONCE(current->rcu_read_lock_nesting > RCU_NEST_PMAX);	375	WARN_ON_ONCE(rcu_preempt_depth() > RCU_NEST_PMAX);
361	barrier(); /* critical section after entry code. */	376	barrier(); /* critical section after entry code. */
362	}	377	}
363	EXPORT_SYMBOL_GPL(__rcu_read_lock);	378	EXPORT_SYMBOL_GPL(__rcu_read_lock);
@@ -373,19 +388,19 @@ void __rcu_read_unlock(void) · 9 unmodified lines
373	{	388	{
374	struct task_struct *t = current;	389	struct task_struct *t = current;
375		390
376	if (t->rcu_read_lock_nesting != 1) {	391	if (rcu_preempt_depth() != 1) {
377	--t->rcu_read_lock_nesting;	392	rcu_preempt_read_exit();
378	} else {	393	} else {
379	barrier(); /* critical section before exit code. */	394	barrier(); /* critical section before exit code. */
380	t->rcu_read_lock_nesting = -RCU_NEST_BIAS;	395	rcu_preempt_depth_set(-RCU_NEST_BIAS);
381	barrier(); /* assign before ->rcu_read_unlock_special load */	396	barrier(); /* assign before ->rcu_read_unlock_special load */
382	if (unlikely(READ_ONCE(t->rcu_read_unlock_special.s)))	397	if (unlikely(READ_ONCE(t->rcu_read_unlock_special.s)))
383	rcu_read_unlock_special(t);	398	rcu_read_unlock_special(t);
384	barrier(); /* ->rcu_read_unlock_special load before assign */	399	barrier(); /* ->rcu_read_unlock_special load before assign */
385	t->rcu_read_lock_nesting = 0;	400	rcu_preempt_depth_set(0);
386	}	401	}
387	if (IS_ENABLED(CONFIG_PROVE_LOCKING)) {	402	if (IS_ENABLED(CONFIG_PROVE_LOCKING)) {
388	int rrln = t->rcu_read_lock_nesting;	403	int rrln = rcu_preempt_depth();
389		404
390	WARN_ON_ONCE(rrln < 0 && rrln > RCU_NEST_NMAX);	405	WARN_ON_ONCE(rrln < 0 && rrln > RCU_NEST_NMAX);
391	}	406	}
@@ -444,15 +459,9 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 52 unmodified lines
444	local_irq_restore(flags);	459	local_irq_restore(flags);
445	return;	460	return;
446	}	461	}
447	t->rcu_read_unlock_special.b.deferred_qs = false;	462	t->rcu_read_unlock_special.s = 0;
448	if (special.b.need_qs) {	463	if (special.b.need_qs)
449	rcu_qs();	464	rcu_qs();
450	t->rcu_read_unlock_special.b.need_qs = false;
451	if (!t->rcu_read_unlock_special.s && !rdp->exp_deferred_qs) {
452	local_irq_restore(flags);
453	return;
454	}
455	}
456		465
457	/*	466	/*
458	* Respond to a request by an expedited grace period for a	467	* Respond to a request by an expedited grace period for a
@@ -460,17 +469,11 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 1 unmodified line
460	* tasks are handled when removing the task from the	469	* tasks are handled when removing the task from the
461	* blocked-tasks list below.	470	* blocked-tasks list below.
462	*/	471	*/
463	if (rdp->exp_deferred_qs) {	472	if (rdp->exp_deferred_qs)
464	rcu_report_exp_rdp(rdp);	473	rcu_report_exp_rdp(rdp);
465	if (!t->rcu_read_unlock_special.s) {
466	local_irq_restore(flags);
467	return;
468	}
469	}
470		474
471	/* Clean up if blocked during RCU read-side critical section. */	475	/* Clean up if blocked during RCU read-side critical section. */
472	if (special.b.blocked) {	476	if (special.b.blocked) {
473	t->rcu_read_unlock_special.b.blocked = false;
474		477
475	/*	478	/*
476	* Remove this task from the list it blocked on. The task	479	* Remove this task from the list it blocked on. The task
@@ -485,7 +488,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 8 unmodified lines
485	empty_norm = !rcu_preempt_blocked_readers_cgp(rnp);	488	empty_norm = !rcu_preempt_blocked_readers_cgp(rnp);
486	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq &&	489	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq &&
487	(!empty_norm \|\| rnp->qsmask));	490	(!empty_norm \|\| rnp->qsmask));
488	empty_exp = sync_rcu_preempt_exp_done(rnp);	491	empty_exp = sync_rcu_exp_done(rnp);
489	smp_mb(); /* ensure expedited fastpath sees end of RCU c-s. */	492	smp_mb(); /* ensure expedited fastpath sees end of RCU c-s. */
490	np = rcu_next_node_entry(t, rnp);	493	np = rcu_next_node_entry(t, rnp);
491	list_del_init(&t->rcu_node_entry);	494	list_del_init(&t->rcu_node_entry);
@@ -493,7 +496,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 1 unmodified line
493	trace_rcu_unlock_preempted_task(TPS("rcu_preempt"),	496	trace_rcu_unlock_preempted_task(TPS("rcu_preempt"),
494	rnp->gp_seq, t->pid);	497	rnp->gp_seq, t->pid);
495	if (&t->rcu_node_entry == rnp->gp_tasks)	498	if (&t->rcu_node_entry == rnp->gp_tasks)
496	rnp->gp_tasks = np;	499	WRITE_ONCE(rnp->gp_tasks, np);
497	if (&t->rcu_node_entry == rnp->exp_tasks)	500	if (&t->rcu_node_entry == rnp->exp_tasks)
498	rnp->exp_tasks = np;	501	rnp->exp_tasks = np;
499	if (IS_ENABLED(CONFIG_RCU_BOOST)) {	502	if (IS_ENABLED(CONFIG_RCU_BOOST)) {
@@ -509,7 +512,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 9 unmodified lines
509	* Note that rcu_report_unblock_qs_rnp() releases rnp->lock,	512	* Note that rcu_report_unblock_qs_rnp() releases rnp->lock,
510	* so we must take a snapshot of the expedited state.	513	* so we must take a snapshot of the expedited state.
511	*/	514	*/
512	empty_exp_now = sync_rcu_preempt_exp_done(rnp);	515	empty_exp_now = sync_rcu_exp_done(rnp);
513	if (!empty_norm && !rcu_preempt_blocked_readers_cgp(rnp)) {	516	if (!empty_norm && !rcu_preempt_blocked_readers_cgp(rnp)) {
514	trace_rcu_quiescent_state_report(TPS("preempt_rcu"),	517	trace_rcu_quiescent_state_report(TPS("preempt_rcu"),
515	rnp->gp_seq,	518	rnp->gp_seq,
@@ -551,7 +554,7 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t) · 35 unmodified lines
551	{	554	{
552	return (__this_cpu_read(rcu_data.exp_deferred_qs) \|\|	555	return (__this_cpu_read(rcu_data.exp_deferred_qs) \|\|
553	READ_ONCE(t->rcu_read_unlock_special.s)) &&	556	READ_ONCE(t->rcu_read_unlock_special.s)) &&
554	t->rcu_read_lock_nesting <= 0;	557	rcu_preempt_depth() <= 0;
555	}	558	}
556		559
557	/*	560	/*
@@ -564,16 +567,16 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t) · 6 unmodified lines
564	static void rcu_preempt_deferred_qs(struct task_struct *t)	567	static void rcu_preempt_deferred_qs(struct task_struct *t)
565	{	568	{
566	unsigned long flags;	569	unsigned long flags;
567	bool couldrecurse = t->rcu_read_lock_nesting >= 0;	570	bool couldrecurse = rcu_preempt_depth() >= 0;
568		571
569	if (!rcu_preempt_need_deferred_qs(t))	572	if (!rcu_preempt_need_deferred_qs(t))
570	return;	573	return;
571	if (couldrecurse)	574	if (couldrecurse)
572	t->rcu_read_lock_nesting -= RCU_NEST_BIAS;	575	rcu_preempt_depth_set(rcu_preempt_depth() - RCU_NEST_BIAS);
573	local_irq_save(flags);	576	local_irq_save(flags);
574	rcu_preempt_deferred_qs_irqrestore(t, flags);	577	rcu_preempt_deferred_qs_irqrestore(t, flags);
575	if (couldrecurse)	578	if (couldrecurse)
576	t->rcu_read_lock_nesting += RCU_NEST_BIAS;	579	rcu_preempt_depth_set(rcu_preempt_depth() + RCU_NEST_BIAS);
577	}	580	}
578		581
579	/*	582	/*
@@ -610,9 +613,8 @@ static void rcu_read_unlock_special(struct task_struct *t) · 30 unmodified lines
610	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);	613	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
611	struct rcu_node *rnp = rdp->mynode;	614	struct rcu_node *rnp = rdp->mynode;
612		615
613	t->rcu_read_unlock_special.b.exp_hint = false;
614	exp = (t->rcu_blocked_node && t->rcu_blocked_node->exp_tasks) \|\|	616	exp = (t->rcu_blocked_node && t->rcu_blocked_node->exp_tasks) \|\|
615	(rdp->grpmask & rnp->expmask) \|\|	617	(rdp->grpmask & READ_ONCE(rnp->expmask)) \|\|
616	tick_nohz_full_cpu(rdp->cpu);	618	tick_nohz_full_cpu(rdp->cpu);
617	// Need to defer quiescent state until everything is enabled.	619	// Need to defer quiescent state until everything is enabled.
618	if (irqs_were_disabled && use_softirq &&	620	if (irqs_were_disabled && use_softirq &&
@@ -640,7 +642,6 @@ static void rcu_read_unlock_special(struct task_struct *t) · 21 unmodified lines
640	local_irq_restore(flags);	642	local_irq_restore(flags);
641	return;	643	return;
642	}	644	}
643	WRITE_ONCE(t->rcu_read_unlock_special.b.exp_hint, false);
644	rcu_preempt_deferred_qs_irqrestore(t, flags);	645	rcu_preempt_deferred_qs_irqrestore(t, flags);
645	}	646	}
646		647
@@ -648,8 +649,7 @@ static void rcu_read_unlock_special(struct task_struct *t) · 1 unmodified line
648	* Check that the list of blocked tasks for the newly completed grace	649	* Check that the list of blocked tasks for the newly completed grace
649	* period is in fact empty. It is a serious bug to complete a grace	650	* period is in fact empty. It is a serious bug to complete a grace
650	* period that still has RCU readers blocked! This function must be	651	* period that still has RCU readers blocked! This function must be
651	* invoked -before- updating this rnp's ->gp_seq, and the rnp's ->lock	652	* invoked -before- updating this rnp's ->gp_seq.
652	* must be held by the caller.
653	*	653	*
654	* Also, if there are blocked tasks on the list, they automatically	654	* Also, if there are blocked tasks on the list, they automatically
655	* block the newly created grace period, so set up ->gp_tasks accordingly.	655	* block the newly created grace period, so set up ->gp_tasks accordingly.
@@ -659,11 +659,12 @@ static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp) · 3 unmodified lines
659	struct task_struct *t;	659	struct task_struct *t;
660		660
661	RCU_LOCKDEP_WARN(preemptible(), "rcu_preempt_check_blocked_tasks() invoked with preemption enabled!!!\n");	661	RCU_LOCKDEP_WARN(preemptible(), "rcu_preempt_check_blocked_tasks() invoked with preemption enabled!!!\n");
		662	raw_lockdep_assert_held_rcu_node(rnp);
662	if (WARN_ON_ONCE(rcu_preempt_blocked_readers_cgp(rnp)))	663	if (WARN_ON_ONCE(rcu_preempt_blocked_readers_cgp(rnp)))
663	dump_blkd_tasks(rnp, 10);	664	dump_blkd_tasks(rnp, 10);
664	if (rcu_preempt_has_tasks(rnp) &&	665	if (rcu_preempt_has_tasks(rnp) &&
665	(rnp->qsmaskinit \|\| rnp->wait_blkd_tasks)) {	666	(rnp->qsmaskinit \|\| rnp->wait_blkd_tasks)) {
666	rnp->gp_tasks = rnp->blkd_tasks.next;	667	WRITE_ONCE(rnp->gp_tasks, rnp->blkd_tasks.next);
667	t = container_of(rnp->gp_tasks, struct task_struct,	668	t = container_of(rnp->gp_tasks, struct task_struct,
668	rcu_node_entry);	669	rcu_node_entry);
669	trace_rcu_unlock_preempted_task(TPS("rcu_preempt-GPS"),	670	trace_rcu_unlock_preempted_task(TPS("rcu_preempt-GPS"),
@@ -686,7 +687,7 @@ static void rcu_flavor_sched_clock_irq(int user) · 16 unmodified lines
686	if (user \|\| rcu_is_cpu_rrupt_from_idle()) {	687	if (user \|\| rcu_is_cpu_rrupt_from_idle()) {
687	rcu_note_voluntary_context_switch(current);	688	rcu_note_voluntary_context_switch(current);
688	}	689	}
689	if (t->rcu_read_lock_nesting > 0 \|\|	690	if (rcu_preempt_depth() > 0 \|\|
690	(preempt_count() & (PREEMPT_MASK \| SOFTIRQ_MASK))) {	691	(preempt_count() & (PREEMPT_MASK \| SOFTIRQ_MASK))) {
691	/* No QS, force context switch if deferred. */	692	/* No QS, force context switch if deferred. */
692	if (rcu_preempt_need_deferred_qs(t)) {	693	if (rcu_preempt_need_deferred_qs(t)) {
@@ -696,13 +697,13 @@ static void rcu_flavor_sched_clock_irq(int user) · 3 unmodified lines
696	} else if (rcu_preempt_need_deferred_qs(t)) {	697	} else if (rcu_preempt_need_deferred_qs(t)) {
697	rcu_preempt_deferred_qs(t); /* Report deferred QS. */	698	rcu_preempt_deferred_qs(t); /* Report deferred QS. */
698	return;	699	return;
699	} else if (!t->rcu_read_lock_nesting) {	700	} else if (!rcu_preempt_depth()) {
700	rcu_qs(); /* Report immediate QS. */	701	rcu_qs(); /* Report immediate QS. */
701	return;	702	return;
702	}	703	}
703		704
704	/* If GP is oldish, ask for help from rcu_read_unlock_special(). */	705	/* If GP is oldish, ask for help from rcu_read_unlock_special(). */
705	if (t->rcu_read_lock_nesting > 0 &&	706	if (rcu_preempt_depth() > 0 &&
706	__this_cpu_read(rcu_data.core_needs_qs) &&	707	__this_cpu_read(rcu_data.core_needs_qs) &&
707	__this_cpu_read(rcu_data.cpu_no_qs.b.norm) &&	708	__this_cpu_read(rcu_data.cpu_no_qs.b.norm) &&
708	!t->rcu_read_unlock_special.b.need_qs &&	709	!t->rcu_read_unlock_special.b.need_qs &&
@@ -723,11 +724,11 @@ void exit_rcu(void) · 14 unmodified lines
723	struct task_struct *t = current;	724	struct task_struct *t = current;
724		725
725	if (unlikely(!list_empty(&current->rcu_node_entry))) {	726	if (unlikely(!list_empty(&current->rcu_node_entry))) {
726	t->rcu_read_lock_nesting = 1;	727	rcu_preempt_depth_set(1);
727	barrier();	728	barrier();
728	WRITE_ONCE(t->rcu_read_unlock_special.b.blocked, true);	729	WRITE_ONCE(t->rcu_read_unlock_special.b.blocked, true);
729	} else if (unlikely(t->rcu_read_lock_nesting)) {	730	} else if (unlikely(rcu_preempt_depth())) {
730	t->rcu_read_lock_nesting = 1;	731	rcu_preempt_depth_set(1);
731	} else {	732	} else {
732	return;	733	return;
733	}	734	}
@@ -757,7 +758,8 @@ dump_blkd_tasks(struct rcu_node *rnp, int ncheck) · 23 unmodified lines
757	pr_info("%s: %d:%d ->qsmask %#lx ->qsmaskinit %#lx ->qsmaskinitnext %#lx\n",	758	pr_info("%s: %d:%d ->qsmask %#lx ->qsmaskinit %#lx ->qsmaskinitnext %#lx\n",
758	__func__, rnp1->grplo, rnp1->grphi, rnp1->qsmask, rnp1->qsmaskinit, rnp1->qsmaskinitnext);	759	__func__, rnp1->grplo, rnp1->grphi, rnp1->qsmask, rnp1->qsmaskinit, rnp1->qsmaskinitnext);
759	pr_info("%s: ->gp_tasks %p ->boost_tasks %p ->exp_tasks %p\n",	760	pr_info("%s: ->gp_tasks %p ->boost_tasks %p ->exp_tasks %p\n",
760	__func__, rnp->gp_tasks, rnp->boost_tasks, rnp->exp_tasks);	761	__func__, READ_ONCE(rnp->gp_tasks), rnp->boost_tasks,
		762	rnp->exp_tasks);
761	pr_info("%s: ->blkd_tasks", __func__);	763	pr_info("%s: ->blkd_tasks", __func__);
762	i = 0;	764	i = 0;
763	list_for_each(lhp, &rnp->blkd_tasks) {	765	list_for_each(lhp, &rnp->blkd_tasks) {
@@ -788,7 +790,7 @@ static void __init rcu_bootup_announce(void) · 24 unmodified lines
788	}	790	}
789		791
790	/*	792	/*
791	* Note a quiescent state for PREEMPT=n. Because we do not need to know	793	* Note a quiescent state for PREEMPTION=n. Because we do not need to know
792	* how many quiescent states passed, just if there was at least one since	794	* how many quiescent states passed, just if there was at least one since
793	* the start of the grace period, this just sets a flag. The caller must	795	* the start of the grace period, this just sets a flag. The caller must
794	* have disabled preemption.	796	* have disabled preemption.
@@ -838,7 +840,7 @@ void rcu_all_qs(void) · 43 unmodified lines
838	EXPORT_SYMBOL_GPL(rcu_all_qs);	840	EXPORT_SYMBOL_GPL(rcu_all_qs);
839		841
840	/*	842	/*
841	* Note a PREEMPT=n context switch. The caller must have disabled interrupts.	843	* Note a PREEMPTION=n context switch. The caller must have disabled interrupts.
842	*/	844	*/
843	void rcu_note_context_switch(bool preempt)	845	void rcu_note_context_switch(bool preempt)
844	{	846	{
@@ -1262,10 +1264,9 @@ static void rcu_prepare_for_idle(void) · 417 unmodified lines
1262	/*	1264	/*
1263	* This code is invoked when a CPU goes idle, at which point we want	1265	* This code is invoked when a CPU goes idle, at which point we want
1264	* to have the CPU do everything required for RCU so that it can enter	1266	* to have the CPU do everything required for RCU so that it can enter
1265	* the energy-efficient dyntick-idle mode. This is handled by a	1267	* the energy-efficient dyntick-idle mode.
1266	* state machine implemented by rcu_prepare_for_idle() below.
1267	*	1268	*
1268	* The following three proprocessor symbols control this state machine:	1269	* The following preprocessor symbol controls this:
1269	*	1270	*
1270	* RCU_IDLE_GP_DELAY gives the number of jiffies that a CPU is permitted	1271	* RCU_IDLE_GP_DELAY gives the number of jiffies that a CPU is permitted
1271	* to sleep in dyntick-idle mode with RCU callbacks pending. This	1272	* to sleep in dyntick-idle mode with RCU callbacks pending. This
@@ -1274,21 +1275,15 @@ static void rcu_prepare_for_idle(void) · 2 unmodified lines
1274	* number, be warned: Setting RCU_IDLE_GP_DELAY too high can hang your	1275	* number, be warned: Setting RCU_IDLE_GP_DELAY too high can hang your
1275	* system. And if you are -that- concerned about energy efficiency,	1276	* system. And if you are -that- concerned about energy efficiency,
1276	* just power the system down and be done with it!	1277	* just power the system down and be done with it!
1277	* RCU_IDLE_LAZY_GP_DELAY gives the number of jiffies that a CPU is
1278	* permitted to sleep in dyntick-idle mode with only lazy RCU
1279	* callbacks pending. Setting this too high can OOM your system.
1280	*	1278	*
1281	* The values below work well in practice. If future workloads require	1279	* The value below works well in practice. If future workloads require
1282	* adjustment, they can be converted into kernel config parameters, though	1280	* adjustment, they can be converted into kernel config parameters, though
1283	* making the state machine smarter might be a better option.	1281	* making the state machine smarter might be a better option.
1284	*/	1282	*/
1285	#define RCU_IDLE_GP_DELAY 4 /* Roughly one grace period. */	1283	#define RCU_IDLE_GP_DELAY 4 /* Roughly one grace period. */
1286	#define RCU_IDLE_LAZY_GP_DELAY (6 * HZ) /* Roughly six seconds. */
1287		1284
1288	static int rcu_idle_gp_delay = RCU_IDLE_GP_DELAY;	1285	static int rcu_idle_gp_delay = RCU_IDLE_GP_DELAY;
1289	module_param(rcu_idle_gp_delay, int, 0644);	1286	module_param(rcu_idle_gp_delay, int, 0644);
1290	static int rcu_idle_lazy_gp_delay = RCU_IDLE_LAZY_GP_DELAY;
1291	module_param(rcu_idle_lazy_gp_delay, int, 0644);
1292		1287
1293	/*	1288	/*
1294	* Try to advance callbacks on the current CPU, but only if it has been	1289	* Try to advance callbacks on the current CPU, but only if it has been
@@ -1327,8 +1322,7 @@ static bool __maybe_unused rcu_try_advance_all_cbs(void) · 32 unmodified lines
1327	/*	1322	/*
1328	* Allow the CPU to enter dyntick-idle mode unless it has callbacks ready	1323	* Allow the CPU to enter dyntick-idle mode unless it has callbacks ready
1329	* to invoke. If the CPU has callbacks, try to advance them. Tell the	1324	* to invoke. If the CPU has callbacks, try to advance them. Tell the
1330	* caller to set the timeout based on whether or not there are non-lazy	1325	* caller about what to set the timeout.
1331	* callbacks.
1332	*	1326	*
1333	* The caller must have disabled interrupts.	1327	* The caller must have disabled interrupts.
1334	*/	1328	*/
@@ -1354,25 +1348,18 @@ int rcu_needs_cpu(u64 basemono, u64 *nextevt) · 19 unmodified lines
1354	}	1348	}
1355	rdp->last_accelerate = jiffies;	1349	rdp->last_accelerate = jiffies;
1356		1350
1357	/* Request timer delay depending on laziness, and round. */	1351	/* Request timer and round. */
1358	rdp->all_lazy = !rcu_segcblist_n_nonlazy_cbs(&rdp->cblist);	1352	dj = round_up(rcu_idle_gp_delay + jiffies, rcu_idle_gp_delay) - jiffies;
1359	if (rdp->all_lazy) {	1353
1360	dj = round_jiffies(rcu_idle_lazy_gp_delay + jiffies) - jiffies;
1361	} else {
1362	dj = round_up(rcu_idle_gp_delay + jiffies,
1363	rcu_idle_gp_delay) - jiffies;
1364	}
1365	nextevt = basemono + dj TICK_NSEC;	1354	nextevt = basemono + dj TICK_NSEC;
1366	return 0;	1355	return 0;
1367	}	1356	}
1368		1357
1369	/*	1358	/*
1370	* Prepare a CPU for idle from an RCU perspective. The first major task	1359	* Prepare a CPU for idle from an RCU perspective. The first major task is to
1371	* is to sense whether nohz mode has been enabled or disabled via sysfs.	1360	* sense whether nohz mode has been enabled or disabled via sysfs. The second
1372	* The second major task is to check to see if a non-lazy callback has	1361	* major task is to accelerate (that is, assign grace-period numbers to) any
1373	* arrived at a CPU that previously had only lazy callbacks. The third	1362	* recently arrived callbacks.
1374	* major task is to accelerate (that is, assign grace-period numbers to)
1375	* any recently arrived callbacks.
1376	*	1363	*
1377	* The caller must have disabled interrupts.	1364	* The caller must have disabled interrupts.
1378	*/	1365	*/
@@ -1398,17 +1385,6 @@ static void rcu_prepare_for_idle(void) · 19 unmodified lines
1398	if (!tne)	1385	if (!tne)
1399	return;	1386	return;
1400		1387
1401	/*
1402	* If a non-lazy callback arrived at a CPU having only lazy
1403	* callbacks, invoke RCU core for the side-effect of recalculating
1404	* idle duration on re-entry to idle.
1405	*/
1406	if (rdp->all_lazy && rcu_segcblist_n_nonlazy_cbs(&rdp->cblist)) {
1407	rdp->all_lazy = false;
1408	invoke_rcu_core();
1409	return;
1410	}
1411
1412	/*	1388	/*
1413	* If we have not yet accelerated this jiffy, accelerate all	1389	* If we have not yet accelerated this jiffy, accelerate all
1414	* callbacks on this CPU.	1390	* callbacks on this CPU.
@@ -2321,6 +2297,8 @@ static void __init rcu_organize_nocb_kthreads(void) · 906 unmodified lines
2321	{	2297	{
2322	int cpu;	2298	int cpu;
2323	bool firsttime = true;	2299	bool firsttime = true;
		2300	bool gotnocbs = false;
		2301	bool gotnocbscbs = true;
2324	int ls = rcu_nocb_gp_stride;	2302	int ls = rcu_nocb_gp_stride;
2325	int nl = 0; /* Next GP kthread. */	2303	int nl = 0; /* Next GP kthread. */
2326	struct rcu_data *rdp;	2304	struct rcu_data *rdp;
@@ -2343,21 +2321,31 @@ static void __init rcu_organize_nocb_kthreads(void) · 16 unmodified lines
2343	rdp = per_cpu_ptr(&rcu_data, cpu);	2321	rdp = per_cpu_ptr(&rcu_data, cpu);
2344	if (rdp->cpu >= nl) {	2322	if (rdp->cpu >= nl) {
2345	/* New GP kthread, set up for CBs & next GP. */	2323	/* New GP kthread, set up for CBs & next GP. */
		2324	gotnocbs = true;
2346	nl = DIV_ROUND_UP(rdp->cpu + 1, ls) * ls;	2325	nl = DIV_ROUND_UP(rdp->cpu + 1, ls) * ls;
2347	rdp->nocb_gp_rdp = rdp;	2326	rdp->nocb_gp_rdp = rdp;
2348	rdp_gp = rdp;	2327	rdp_gp = rdp;
2349	if (!firsttime && dump_tree)	2328	if (dump_tree) {
2350	pr_cont("\n");	2329	if (!firsttime)
2351	firsttime = false;	2330	pr_cont("%s\n", gotnocbscbs
2352	pr_alert("%s: No-CB GP kthread CPU %d:", __func__, cpu);	2331	? "" : " (self only)");
		2332	gotnocbscbs = false;
		2333	firsttime = false;
		2334	pr_alert("%s: No-CB GP kthread CPU %d:",
		2335	__func__, cpu);
		2336	}
2353	} else {	2337	} else {
2354	/* Another CB kthread, link to previous GP kthread. */	2338	/* Another CB kthread, link to previous GP kthread. */
		2339	gotnocbscbs = true;
2355	rdp->nocb_gp_rdp = rdp_gp;	2340	rdp->nocb_gp_rdp = rdp_gp;
2356	rdp_prev->nocb_next_cb_rdp = rdp;	2341	rdp_prev->nocb_next_cb_rdp = rdp;
2357	pr_alert(" %d", cpu);	2342	if (dump_tree)
		2343	pr_cont(" %d", cpu);
2358	}	2344	}
2359	rdp_prev = rdp;	2345	rdp_prev = rdp;
2360	}	2346	}
		2347	if (gotnocbs && dump_tree)
		2348	pr_cont("%s\n", gotnocbscbs ? "" : " (self only)");
2361	}	2349	}
2362		2350
2363	/*	2351	/*

kernel/rcu/tree_stall.h+27−7

 //
 // Printing RCU CPU stall warnings
-#ifdef CONFIG_PREEMPTION
+#ifdef CONFIG_PREEMPT_RCU
 /*
  * Dump detailed information for all tasks blocking the current RCU
 	return ndetected;
 }
-#else /* #ifdef CONFIG_PREEMPTION */
+#else /* #ifdef CONFIG_PREEMPT_RCU */
 /*
  * Because preemptible RCU does not exist, we never have to check for
 {
 	return 0;
 }
-#endif /* #else #ifdef CONFIG_PREEMPTION */
+#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
 /*
  * Dump stacks of all tasks running on stalled CPUs.  First try using
 {
 	struct rcu_data *rdp = &per_cpu(rcu_data, cpu);
-	sprintf(cp, "last_accelerate: %04lx/%04lx, Nonlazy posted: %c%c%c",
+	sprintf(cp, "last_accelerate: %04lx/%04lx dyntick_enabled: %d",
 		rdp->last_accelerate & 0xffff, jiffies & 0xffff,
-		".l"[rdp->all_lazy],
+		!!rdp->tick_nohz_enabled_snap);
-		".L"[!rcu_segcblist_n_nonlazy_cbs(&rdp->cblist)],
-		".D"[!!rdp->tick_nohz_enabled_snap]);
 }
 #else /* #ifdef CONFIG_RCU_FAST_NO_HZ */
 #endif /* #else #ifdef CONFIG_RCU_FAST_NO_HZ */
+static const char * const gp_state_names[] = {
+	[RCU_GP_IDLE] = "RCU_GP_IDLE",
+	[RCU_GP_WAIT_GPS] = "RCU_GP_WAIT_GPS",
+	[RCU_GP_DONE_GPS] = "RCU_GP_DONE_GPS",
+	[RCU_GP_ONOFF] = "RCU_GP_ONOFF",
+	[RCU_GP_INIT] = "RCU_GP_INIT",
+	[RCU_GP_WAIT_FQS] = "RCU_GP_WAIT_FQS",
+	[RCU_GP_DOING_FQS] = "RCU_GP_DOING_FQS",
+	[RCU_GP_CLEANUP] = "RCU_GP_CLEANUP",
+	[RCU_GP_CLEANED] = "RCU_GP_CLEANED",
+};
+/*
+ * Convert a ->gp_state value to a character string.
+ */
+static const char *gp_state_getname(short gs)
+{
+	if (gs < 0 || gs >= ARRAY_SIZE(gp_state_names))
+		return "???";
+	return gp_state_names[gs];
+}
 /*
  * Print out diagnostic information for the specified stalled CPU.
  *

kernel/rcu/update.c+11−3

 #include <linux/rcupdate_wait.h>
 #include <linux/sched/isolation.h>
 #include <linux/kprobes.h>
+#include <linux/slab.h>
 #define CREATE_TRACE_POINTS
 #define MODULE_PARAM_PREFIX "rcupdate."
 #ifndef CONFIG_TINY_RCU
-extern int rcu_expedited; /* from sysctl */
 module_param(rcu_expedited, int, 0);
-extern int rcu_normal; /* from sysctl */
 module_param(rcu_normal, int, 0);
 static int rcu_normal_after_boot;
 module_param(rcu_normal_after_boot, int, 0);
 {
 	rcu_test_sync_prims();
 	rcu_scheduler_active = RCU_SCHEDULER_RUNNING;
+	kfree_rcu_scheduler_running();
 	rcu_test_sync_prims();
 	return 0;
 }
 EXPORT_SYMBOL_GPL(rcuhead_debug_descr);
 #endif /* #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD */
-#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU) || defined(CONFIG_RCU_TRACE)
+#if defined(CONFIG_TREE_RCU) || defined(CONFIG_RCU_TRACE)
 void do_trace_rcu_torture_read(const char *rcutorturename, struct rcu_head *rhp,
 			       unsigned long secs,
 			       unsigned long c_old, unsigned long c)
 DEFINE_STATIC_SRCU(early_srcu);
+struct early_boot_kfree_rcu {
+	struct rcu_head rh;
+};
 static void early_boot_test_call_rcu(void)
 {
 	static struct rcu_head head;
 	static struct rcu_head shead;
+	struct early_boot_kfree_rcu *rhp;
 	call_rcu(&head, test_callback);
 	if (IS_ENABLED(CONFIG_SRCU))
 		call_srcu(&early_srcu, &shead, test_callback);
+	rhp = kmalloc(sizeof(*rhp), GFP_KERNEL);
+	if (!WARN_ON_ONCE(!rhp))
+		kfree_rcu(rhp, rh);
 }
 void rcu_early_boot_tests(void)

kernel/sysctl.c+1−1

@@ -1268,7 +1268,7 @@ static struct ctl_table kern_table[] = { · 1267 unmodified lines
1268	.proc_handler = proc_do_static_key,	1268	.proc_handler = proc_do_static_key,
1269	},	1269	},
1270	#endif	1270	#endif
1271	#if defined(CONFIG_TREE_RCU) \|\| defined(CONFIG_PREEMPT_RCU)	1271	#if defined(CONFIG_TREE_RCU)
1272	{	1272	{
1273	.procname = "panic_on_rcu_stall",	1273	.procname = "panic_on_rcu_stall",
1274	.data = &sysctl_panic_on_rcu_stall,	1274	.data = &sysctl_panic_on_rcu_stall,

net/tipc/crypto.c+1−4

 #define tipc_aead_rcu_ptr(rcu_ptr, lock)				\
 	rcu_dereference_protected((rcu_ptr), lockdep_is_held(lock))
-#define tipc_aead_rcu_swap(rcu_ptr, ptr, lock)				\
-	rcu_swap_protected((rcu_ptr), (ptr), lockdep_is_held(lock))
 #define tipc_aead_rcu_replace(rcu_ptr, ptr, lock)			\
 do {									\
 	typeof(rcu_ptr) __tmp = rcu_dereference_protected((rcu_ptr),	\
 	/* Move passive key if any */
 	if (key.passive) {
-		tipc_aead_rcu_swap(rx->aead[key.passive], tmp2, &rx->lock);
+		tmp2 = rcu_replace_pointer(rx->aead[key.passive], tmp2, lockdep_is_held(&rx->lock));
 		x = (key.passive - key.pending + new_pending) % KEY_MAX;
 		new_passive = (x <= 0) ? x + KEY_MAX : x;
 	}

tools/testing/selftests/rcutorture/bin/cpus2use.sh+9−2

 	exit 0
 fi
 ncpus=`grep '^processor' /proc/cpuinfo | wc -l`
-idlecpus=`mpstat | tail -1 | \
+if mpstat -V > /dev/null 2>&1
-	awk -v ncpus=$ncpus '{ print ncpus * ($7 + $NF) / 100 }'`
+then
+	idlecpus=`mpstat | tail -1 | \
+		awk -v ncpus=$ncpus '{ print ncpus * ($7 + $NF) / 100 }'`
+else
+	# No mpstat command, so use all available CPUs.
+	echo The mpstat command is not available, so greedily using all CPUs.
+	idlecpus=$ncpus
+fi
 awk -v ncpus=$ncpus -v idlecpus=$idlecpus < /dev/null '
 BEGIN {
 	cpus2use = idlecpus;

tools/testing/selftests/rcutorture/bin/jitter.sh+22−8

@@ -23,25 +23,39 @@ spinmax=${4-1000} · 22 unmodified lines
23		23
24	n=1	24	n=1
25		25
26	starttime=`awk 'BEGIN { print systime(); }' < /dev/null`	26	starttime=`gawk 'BEGIN { print systime(); }' < /dev/null`
		27
		28	nohotplugcpus=
		29	for i in /sys/devices/system/cpu/cpu[0-9]*
		30	do
		31	if test -f $i/online
		32	then
		33	:
		34	else
		35	curcpu=`echo $i \| sed -e 's/^[^0-9]*//'`
		36	nohotplugcpus="$nohotplugcpus $curcpu"
		37	fi
		38	done
27		39
28	while :	40	while :
29	do	41	do
30	# Check for done.	42	# Check for done.
31	t=`awk -v s=$starttime 'BEGIN { print systime() - s; }' < /dev/null`	43	t=`gawk -v s=$starttime 'BEGIN { print systime() - s; }' < /dev/null`
32	if test "$t" -gt "$duration"	44	if test "$t" -gt "$duration"
33	then	45	then
34	exit 0;	46	exit 0;
35	fi	47	fi
36		48
37	# Set affinity to randomly selected online CPU	49	# Set affinity to randomly selected online CPU
38	cpus=`grep 1 /sys/devices/system/cpu/*/online \|	50	if cpus=`grep 1 /sys/devices/system/cpu/*/online 2>&1 \|
39	sed -e 's,/[^/]$,,' -e 's/^[^0-9]//'`	51	sed -e 's,/[^/]$,,' -e 's/^[^0-9]//'`
40		52	then
41	# Do not leave out poor old cpu0 which may not be hot-pluggable	53	:
42	if [ ! -f "/sys/devices/system/cpu/cpu0/online" ]; then	54	else
43	cpus="0 $cpus"	55	cpus=
44	fi	56	fi
		57	# Do not leave out non-hot-pluggable CPUs
		58	cpus="$cpus $nohotplugcpus"
45		59
46	cpumask=`awk -v cpus="$cpus" -v me=$me -v n=$n 'BEGIN {	60	cpumask=`awk -v cpus="$cpus" -v me=$me -v n=$n 'BEGIN {
47	srand(n + me + systime());	61	srand(n + me + systime());

tools/testing/selftests/rcutorture/bin/kvm-recheck-rcu.sh+2−1

 	    tail -1 | sed -e 's/^\[[ 0-9.]*] //' |
 	    awk '{ print \"[\" $1 \" \" $5 \" \" $6 \" \" $7 \"]\"; }' |
 	    tr -d '\012\015'`"
+fwdprog="`grep 'rcu_torture_fwd_prog_cr Duration' $i/console.log 2> /dev/null | sed -e 's/^\[[^]]*] //' | sort -k15nr | head -1 | awk '{ print $14 " " $15 }'`"
 if test -z "$ngps"
 then
 	echo "$configfile ------- " $stopstate
 			BEGIN { print ngps / dur }' < /dev/null`
 		title="$title ($ngpsps/s)"
 	fi
-	echo $title $stopstate
+	echo $title $stopstate $fwdprog
 	nclosecalls=`grep --binary-files=text 'torture: Reader Batch' $i/console.log | tail -1 | awk '{for (i=NF-8;i<=NF;i++) sum+=$i; } END {print sum}'`
 	if test -z "$nclosecalls"
 	then

tools/testing/selftests/rcutorture/bin/kvm-test-1-run.sh+6−7

 boot_args=$6
 cd $KVM
-kstarttime=`awk 'BEGIN { print systime() }' < /dev/null`
+kstarttime=`gawk 'BEGIN { print systime() }' < /dev/null`
 if test -z "$TORTURE_BUILDONLY"
 then
 	echo ' ---' `date`: Starting kernel
 qemu_args="-enable-kvm -nographic $qemu_args"
 cpu_count=`configNR_CPUS.sh $resdir/ConfigFragment`
 cpu_count=`configfrag_boot_cpus "$boot_args" "$config_template" "$cpu_count"`
-vcpus=`identify_qemu_vcpus`
+if test "$cpu_count" -gt "$TORTURE_ALLOTED_CPUS"
-if test $cpu_count -gt $vcpus
 then
-	echo CPU count limited from $cpu_count to $vcpus | tee -a $resdir/Warnings
+	echo CPU count limited from $cpu_count to $TORTURE_ALLOTED_CPUS | tee -a $resdir/Warnings
-	cpu_count=$vcpus
+	cpu_count=$TORTURE_ALLOTED_CPUS
 fi
 qemu_args="`specify_qemu_cpus "$QEMU" "$qemu_args" "$cpu_count"`"
 	then
 		qemu_pid=`cat "$resdir/qemu_pid"`
 	fi
-	kruntime=`awk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
+	kruntime=`gawk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
 	if test -z "$qemu_pid" || kill -0 "$qemu_pid" > /dev/null 2>&1
 	then
 		if test $kruntime -ge $seconds
 	oldline="`tail $resdir/console.log`"
 	while :
 	do
-		kruntime=`awk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
+		kruntime=`gawk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
 		if kill -0 $qemu_pid > /dev/null 2>&1
 		then
 			:

tools/testing/selftests/rcutorture/bin/kvm.sh+21−9

 dryrun=""
 KVM="`pwd`/tools/testing/selftests/rcutorture"; export KVM
 PATH=${KVM}/bin:$PATH; export PATH
-TORTURE_ALLOTED_CPUS=""
+. functions.sh
+TORTURE_ALLOTED_CPUS="`identify_qemu_vcpus`"
 TORTURE_DEFCONFIG=defconfig
 TORTURE_BOOT_IMAGE=""
 TORTURE_INITRD="$KVM/initrd"; export TORTURE_INITRD
 ds=`date +%Y.%m.%d-%H:%M:%S`
 jitter="-1"
-. functions.sh
 usage () {
 	echo "Usage: $scriptname optional arguments:"
 	echo "       --bootargs kernel-boot-arguments"
 		checkarg --cpus "(number)" "$#" "$2" '^[0-9]*$' '^--'
 		cpus=$2
 		TORTURE_ALLOTED_CPUS="$2"
+		max_cpus="`identify_qemu_vcpus`"
+		if test "$TORTURE_ALLOTED_CPUS" -gt "$max_cpus"
+		then
+			TORTURE_ALLOTED_CPUS=$max_cpus
+		fi
 		shift
 		;;
 	--datestamp)
 CONFIGFRAG=${KVM}/configs/${TORTURE_SUITE}; export CONFIGFRAG
+defaultconfigs="`tr '\012' ' ' < $CONFIGFRAG/CFLIST`"
 if test -z "$configs"
 then
-	configs="`cat $CONFIGFRAG/CFLIST`"
+	configs=$defaultconfigs
 fi
 if test -z "$resdir"
 fi
 # Create a file of test-name/#cpus pairs, sorted by decreasing #cpus.
-touch $T/cfgcpu
+configs_derep=
 for CF in $configs
 do
 	case $CF in
 		CF1=$CF
 		;;
 	esac
+	for ((cur_rep=0;cur_rep<$config_reps;cur_rep++))
+	do
+		configs_derep="$configs_derep $CF1"
+	done
+done
+touch $T/cfgcpu
+configs_derep="`echo $configs_derep | sed -e "s/\<CFLIST\>/$defaultconfigs/g"`"
+for CF1 in $configs_derep
+do
 	if test -f "$CONFIGFRAG/$CF1"
 	then
 		cpu_count=`configNR_CPUS.sh $CONFIGFRAG/$CF1`
 		cpu_count=`configfrag_boot_cpus "$TORTURE_BOOTARGS" "$CONFIGFRAG/$CF1" "$cpu_count"`
 		cpu_count=`configfrag_boot_maxcpus "$TORTURE_BOOTARGS" "$CONFIGFRAG/$CF1" "$cpu_count"`
-		for ((cur_rep=0;cur_rep<$config_reps;cur_rep++))
+		echo $CF1 $cpu_count >> $T/cfgcpu
-		do
-			echo $CF1 $cpu_count >> $T/cfgcpu
-		done
 	else
 		echo "The --configs file $CF1 does not exist, terminating."
 		exit 1

tools/testing/selftests/rcutorture/bin/mkinitrd.sh+3−52

@@ -20,58 +20,9 @@ if [ -s "$D/initrd/init" ]; then · 19 unmodified lines
20	exit 0	20	exit 0
21	fi	21	fi
22		22
23	T=${TMPDIR-/tmp}/mkinitrd.sh.$$	23	# Create a C-language initrd/init infinite-loop program and statically
24	trap 'rm -rf $T' 0 2	24	# link it. This results in a very small initrd.
25	mkdir $T	25	echo "Creating a statically linked C-language initrd"
26
27	cat > $T/init << '__EOF___'
28	#!/bin/sh
29	# Run in userspace a few milliseconds every second. This helps to
30	# exercise the NO_HZ_FULL portions of RCU. The 192 instances of "a" was
31	# empirically shown to give a nice multi-millisecond burst of user-mode
32	# execution on a 2GHz CPU, as desired. Modern CPUs will vary from a
33	# couple of milliseconds up to perhaps 100 milliseconds, which is an
34	# acceptable range.
35	#
36	# Why not calibrate an exact delay? Because within this initrd, we
37	# are restricted to Bourne-shell builtins, which as far as I know do not
38	# provide any means of obtaining a fine-grained timestamp.
39
40	a4="a a a a"
41	a16="$a4 $a4 $a4 $a4"
42	a64="$a16 $a16 $a16 $a16"
43	a192="$a64 $a64 $a64"
44	while :
45	do
46	q=
47	for i in $a192
48	do
49	q="$q $i"
50	done
51	sleep 1
52	done
53	__EOF___
54
55	# Try using dracut to create initrd
56	if command -v dracut >/dev/null 2>&1
57	then
58	echo Creating $D/initrd using dracut.
59	# Filesystem creation
60	dracut --force --no-hostonly --no-hostonly-cmdline --module "base" $T/initramfs.img
61	cd $D
62	mkdir -p initrd
63	cd initrd
64	zcat $T/initramfs.img \| cpio -id
65	cp $T/init init
66	chmod +x init
67	echo Done creating $D/initrd using dracut
68	exit 0
69	fi
70
71	# No dracut, so create a C-language initrd/init program and statically
72	# link it. This results in a very small initrd, but might be a bit less
73	# future-proof than dracut.
74	echo "Could not find dracut, attempting C initrd"
75	cd $D	26	cd $D
76	mkdir -p initrd	27	mkdir -p initrd
77	cd initrd	28	cd initrd