Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull RCU updates from Ingo Molnar: "The RCU changes in this cycle were: - Expedited grace-period updates - kfree_rcu() updates - RCU list updates - Preemptible RCU updates - Torture-test updates - Miscellaneous fixes - Documentation updates" * 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (69 commits) rcu: Remove unused stop-machine #include powerpc: Remove comment about read_barrier_depends() .mailmap: Add entries for old paulmck@kernel.org addresses srcu: Apply *_ONCE() to ->srcu_last_gp_end rcu: Switch force_qs_rnp() to for_each_leaf_node_cpu_mask() rcu: Move rcu_{expedited,normal} definitions into rcupdate.h rcu: Move gp_state_names[] and gp_state_getname() to tree_stall.h rcu: Remove the declaration of call_rcu() in tree.h rcu: Fix tracepoint tracking RCU CPU kthread utilization rcu: Fix harmless omission of "CONFIG_" from #if condition rcu: Avoid tick_dep_set_cpu() misordering rcu: Provide wrappers for uses of ->rcu_read_lock_nesting rcu: Use READ_ONCE() for ->expmask in rcu_read_unlock_special() rcu: Clear ->rcu_read_unlock_special only once rcu: Clear .exp_hint only when deferred quiescent state has been reported rcu: Rename some instance of CONFIG_PREEMPTION to CONFIG_PREEMPT_RCU rcu: Remove kfree_call_rcu_nobatch() rcu: Remove kfree_rcu() special casing and lazy-callback handling rcu: Add support for debug_objects debugging for kfree_rcu() rcu: Add multiple in-flight batches of kfree_rcu() work ...

Linus Torvalds committed 6 years ago

Commit d99391e2 parents 8b56177 f8a4bb646 files · +1476 −851Browse files

Unified · Split

46 files

.mailmap +4 −0
Documentation/RCU/NMI-RCU.rst +28 −25
Documentation/RCU/arrayRCU.rst +23 −11
Documentation/RCU/index.rst +5 −0
Documentation/RCU/lockdep-splat.txt +1 −1
Documentation/RCU/rcu_dereference.rst +41 −34
Documentation/RCU/rcubarrier.rst +125 −97
Documentation/RCU/stallwarn.txt +3 −8
Documentation/RCU/whatisRCU.rst +183 −108
Documentation/admin-guide/kernel-parameters.txt +13 −0
arch/powerpc/include/asm/barrier.h +0 −2
drivers/net/wireless/mediatek/mt76/agg-rx.c +2 −2
include/linux/list.h +112 −24
include/linux/list_nulls.h +26 −4
include/linux/rcu_segcblist.h +0 −2
include/linux/rculist.h +24 −14
include/linux/rculist_nulls.h +14 −6
include/linux/rcupdate.h +8 −20
include/linux/rcutiny.h +1 −0
include/linux/rcutree.h +1 −0
include/linux/tick.h +4 −1
include/trace/events/rcu.h +16 −24
kernel/rcu/Kconfig +9 −8
kernel/rcu/Makefile +0 −1
kernel/rcu/rcu.h +3 −30
kernel/rcu/rcu_segcblist.c +3 −22
kernel/rcu/rcu_segcblist.h +2 −23
kernel/rcu/rcuperf.c +165 −8
kernel/rcu/rcutorture.c +81 −60
kernel/rcu/srcutiny.c +1 −1
kernel/rcu/srcutree.c +6 −5
kernel/rcu/tiny.c +27 −1
kernel/rcu/tree.c +272 −52
kernel/rcu/tree.h +1 −17
kernel/rcu/tree_exp.h +91 −56
kernel/rcu/tree_plugin.h +78 −90
kernel/rcu/tree_stall.h +27 −7
kernel/rcu/update.c +11 −3
kernel/sysctl.c +1 −1
net/tipc/crypto.c +1 −4
tools/testing/selftests/rcutorture/bin/cpus2use.sh +9 −2
tools/testing/selftests/rcutorture/bin/jitter.sh +22 −8
tools/testing/selftests/rcutorture/bin/kvm-recheck-rcu.sh +2 −1
tools/testing/selftests/rcutorture/bin/kvm-test-1-run.sh +6 −7
tools/testing/selftests/rcutorture/bin/kvm.sh +21 −9
tools/testing/selftests/rcutorture/bin/mkinitrd.sh +3 −52

.mailmap+4−0

@@ -210,6 +210,10 @@ Paolo 'Blaisorblade' Giarrusso <blaisorblade@yahoo.it> · 209 unmodified lines
210	210	Patrick Mochel <mochel@digitalimplant.org>
211	211	Paul Burton <paulburton@kernel.org> <paul.burton@imgtec.com>
212	212	Paul Burton <paulburton@kernel.org> <paul.burton@mips.com>
	213	+Paul E. McKenney <paulmck@kernel.org> <paulmck@linux.ibm.com>
	214	+Paul E. McKenney <paulmck@kernel.org> <paulmck@linux.vnet.ibm.com>
	215	+Paul E. McKenney <paulmck@kernel.org> <paul.mckenney@linaro.org>
	216	+Paul E. McKenney <paulmck@kernel.org> <paulmck@us.ibm.com>
213	217	Peter A Jonsson <pj@ludd.ltu.se>
214	218	Peter Oruba <peter@oruba.de>
215	219	Peter Oruba <peter.oruba@amd.com>

Documentation/RCU/NMI-RCU.rstrenamed from Documentation/RCU/NMI-RCU.txt+28−25

@@ -1,4 +1,7 @@
	1	+.. _NMI_rcu_doc:
	2	+
1	3	Using RCU to Protect Dynamic NMI Handlers
	4	+=========================================
2	5
3	6
4	7	Although RCU is usually used to protect read-mostly data structures,
@@ -9,7 +12,7 @@ work in "arch/x86/oprofile/nmi_timer_int.c" and in · 4 unmodified lines
9	12	"arch/x86/kernel/traps.c".
10	13
11	14	The relevant pieces of code are listed below, each followed by a
12		−brief explanation.
	15	+brief explanation::
13	16
14	17	static int dummy_nmi_callback(struct pt_regs *regs, int cpu)
15	18	{
@@ -18,12 +21,12 @@ brief explanation. · 2 unmodified lines
18	21
19	22	The dummy_nmi_callback() function is a "dummy" NMI handler that does
20	23	nothing, but returns zero, thus saying that it did nothing, allowing
21		−the NMI handler to take the default machine-specific action.
	24	+the NMI handler to take the default machine-specific action::
22	25
23	26	static nmi_callback_t nmi_callback = dummy_nmi_callback;
24	27
25	28	This nmi_callback variable is a global function pointer to the current
26		−NMI handler.
	29	+NMI handler::
27	30
28	31	void do_nmi(struct pt_regs * regs, long error_code)
29	32	{
@@ -53,11 +56,12 @@ anyway. However, in practice it is a good documentation aid, particularly · 23 unmodified lines
53	56	for anyone attempting to do something similar on Alpha or on systems
54	57	with aggressive optimizing compilers.
55	58
56		−Quick Quiz: Why might the rcu_dereference_sched() be necessary on Alpha,
57		− given that the code referenced by the pointer is read-only?
	59	+Quick Quiz:
	60	+ Why might the rcu_dereference_sched() be necessary on Alpha, given that the code referenced by the pointer is read-only?
58	61
	62	+:ref:`Answer to Quick Quiz <answer_quick_quiz_NMI>`
59	63
60		−Back to the discussion of NMI and RCU...
	64	+Back to the discussion of NMI and RCU::
61	65
62	66	void set_nmi_callback(nmi_callback_t callback)
63	67	{
@@ -68,7 +72,7 @@ The set_nmi_callback() function registers an NMI handler. Note that any · 4 unmodified lines
68	72	data that is to be used by the callback must be initialized up -before-
69	73	the call to set_nmi_callback(). On architectures that do not order
70	74	writes, the rcu_assign_pointer() ensures that the NMI handler sees the
71		−initialized values.
	75	+initialized values::
72	76
73	77	void unset_nmi_callback(void)
74	78	{
@@ -82,7 +86,7 @@ up any data structures used by the old NMI handler until execution · 7 unmodified lines
82	86	of it completes on all other CPUs.
83	87
84	88	One way to accomplish this is via synchronize_rcu(), perhaps as
85		−follows:
	89	+follows::
86	90
87	91	unset_nmi_callback();
88	92	synchronize_rcu();
@@ -98,24 +102,23 @@ to free up the handler's data as soon as synchronize_rcu() returns. · 9 unmodified lines
98	102	Important note: for this to work, the architecture in question must
99	103	invoke nmi_enter() and nmi_exit() on NMI entry and exit, respectively.
100	104
	105	+.. _answer_quick_quiz_NMI:
101	106
102		−Answer to Quick Quiz
103		−
104		− Why might the rcu_dereference_sched() be necessary on Alpha, given
105		− that the code referenced by the pointer is read-only?
	107	+Answer to Quick Quiz:
	108	+ Why might the rcu_dereference_sched() be necessary on Alpha, given that the code referenced by the pointer is read-only?
106	109
107		− Answer: The caller to set_nmi_callback() might well have
108		− initialized some data that is to be used by the new NMI
109		− handler. In this case, the rcu_dereference_sched() would
110		− be needed, because otherwise a CPU that received an NMI
111		− just after the new handler was set might see the pointer
112		− to the new NMI handler, but the old pre-initialized
113		− version of the handler's data.
	110	+ The caller to set_nmi_callback() might well have
	111	+ initialized some data that is to be used by the new NMI
	112	+ handler. In this case, the rcu_dereference_sched() would
	113	+ be needed, because otherwise a CPU that received an NMI
	114	+ just after the new handler was set might see the pointer
	115	+ to the new NMI handler, but the old pre-initialized
	116	+ version of the handler's data.
114	117
115		− This same sad story can happen on other CPUs when using
116		− a compiler with aggressive pointer-value speculation
117		− optimizations.
	118	+ This same sad story can happen on other CPUs when using
	119	+ a compiler with aggressive pointer-value speculation
	120	+ optimizations.
118	121
119		− More important, the rcu_dereference_sched() makes it
120		− clear to someone reading the code that the pointer is
121		− being protected by RCU-sched.
	122	+ More important, the rcu_dereference_sched() makes it
	123	+ clear to someone reading the code that the pointer is
	124	+ being protected by RCU-sched.

Documentation/RCU/arrayRCU.rstrenamed from Documentation/RCU/arrayRCU.txt+23−11

@@ -1,19 +1,21 @@
1		−Using RCU to Protect Read-Mostly Arrays
	1	+.. _array_rcu_doc:
2	2
	3	+Using RCU to Protect Read-Mostly Arrays
	4	+=======================================
3	5
4	6	Although RCU is more commonly used to protect linked lists, it can
5	7	also be used to protect arrays. Three situations are as follows:
6	8
7		−1. Hash Tables
	9	+1. :ref:`Hash Tables <hash_tables>`
8	10
9		−2. Static Arrays
	11	+2. :ref:`Static Arrays <static_arrays>`
10	12
11		−3. Resizeable Arrays
	13	+3. :ref:`Resizable Arrays <resizable_arrays>`
12	14
13	15	Each of these three situations involves an RCU-protected pointer to an
14	16	array that is separately indexed. It might be tempting to consider use
15	17	of RCU to instead protect the index into an array, however, this use
16		−case is -not- supported. The problem with RCU-protected indexes into
	18	+case is not supported. The problem with RCU-protected indexes into
17	19	arrays is that compilers can play way too many optimization games with
18	20	integers, which means that the rules governing handling of these indexes
19	21	are far more trouble than they are worth. If RCU-protected indexes into
@@ -24,16 +26,20 @@ to be safely used. · 4 unmodified lines
24	26	That aside, each of the three RCU-protected pointer situations are
25	27	described in the following sections.
26	28
	29	+.. _hash_tables:
27	30
28	31	Situation 1: Hash Tables
	32	+------------------------
29	33
30	34	Hash tables are often implemented as an array, where each array entry
31	35	has a linked-list hash chain. Each hash chain can be protected by RCU
32	36	as described in the listRCU.txt document. This approach also applies
33	37	to other array-of-list situations, such as radix trees.
34	38
	39	+.. _static_arrays:
35	40
36	41	Situation 2: Static Arrays
	42	+--------------------------
37	43
38	44	Static arrays, where the data (rather than a pointer to the data) is
39	45	located in each array element, and where the array is never resized,
@@ -41,13 +47,17 @@ have not been used with RCU. Rik van Riel recommends using seqlock in · 1 unmodified line
41	47	this situation, which would also have minimal read-side overhead as long
42	48	as updates are rare.
43	49
44		−Quick Quiz: Why is it so important that updates be rare when
45		− using seqlock?
	50	+Quick Quiz:
	51	+ Why is it so important that updates be rare when using seqlock?
	52	+
	53	+:ref:`Answer to Quick Quiz <answer_quick_quiz_seqlock>`
46	54
	55	+.. _resizable_arrays:
47	56
48		−Situation 3: Resizeable Arrays
	57	+Situation 3: Resizable Arrays
	58	+------------------------------
49	59
50		−Use of RCU for resizeable arrays is demonstrated by the grow_ary()
	60	+Use of RCU for resizable arrays is demonstrated by the grow_ary()
51	61	function formerly used by the System V IPC code. The array is used
52	62	to map from semaphore, message-queue, and shared-memory IDs to the data
53	63	structure that represents the corresponding IPC construct. The grow_ary()
@@ -60,7 +70,7 @@ the remainder of the new, updates the ids->entries pointer to point to · 6 unmodified lines
60	70	the new array, and invokes ipc_rcu_putref() to free up the old array.
61	71	Note that rcu_assign_pointer() is used to update the ids->entries pointer,
62	72	which includes any memory barriers required on whatever architecture
63		−you are running on.
	73	+you are running on::
64	74
65	75	static int grow_ary(struct ipc_ids* ids, int newsize)
66	76	{
@@ -112,7 +122,7 @@ a simple check suffices. The pointer to the structure corresponding · 45 unmodified lines
112	122	to the desired IPC object is placed in "out", with NULL indicating
113	123	a non-existent entry. After acquiring "out->lock", the "out->deleted"
114	124	flag indicates whether the IPC object is in the process of being
115		−deleted, and, if not, the pointer is returned.
	125	+deleted, and, if not, the pointer is returned::
116	126
117	127	struct kern_ipc_perm* ipc_lock(struct ipc_ids* ids, int id)
118	128	{
@@ -144,8 +154,10 @@ deleted, and, if not, the pointer is returned. · 25 unmodified lines
144	154	return out;
145	155	}
146	156
	157	+.. _answer_quick_quiz_seqlock:
147	158
148	159	Answer to Quick Quiz:
	160	+ Why is it so important that updates be rare when using seqlock?
149	161
150	162	The reason that it is important that updates be rare when
151	163	using seqlock is that frequent updates can livelock readers.

Documentation/RCU/index.rst+5−0

@@ -7,8 +7,13 @@ RCU concepts · 6 unmodified lines
7	7	.. toctree::
8	8	:maxdepth: 3
9	9
	10	+ arrayRCU
	11	+ rcubarrier
	12	+ rcu_dereference
	13	+ whatisRCU
10	14	rcu
11	15	listRCU
	16	+ NMI-RCU
12	17	UP
13	18
14	19	Design/Memory-Ordering/Tree-RCU-Memory-Ordering

Documentation/RCU/lockdep-splat.txt+1−1

@@ -99,7 +99,7 @@ With this change, the rcu_dereference() is always within an RCU · 98 unmodified lines
99	99	read-side critical section, which again would have suppressed the
100	100	above lockdep-RCU splat.
101	101
102		−But in this particular case, we don't actually deference the pointer
	102	+But in this particular case, we don't actually dereference the pointer
103	103	returned from rcu_dereference(). Instead, that pointer is just compared
104	104	to the cic pointer, which means that the rcu_dereference() can be replaced
105	105	by rcu_access_pointer() as follows:

Documentation/RCU/rcu_dereference.rstrenamed from Documentation/RCU/rcu_dereference.txt+41−34

@@ -1,4 +1,7 @@
	1	+.. _rcu_dereference_doc:
	2	+
1	3	PROPER CARE AND FEEDING OF RETURN VALUES FROM rcu_dereference()
	4	+===============================================================
2	5
3	6	Most of the time, you can use values from rcu_dereference() or one of
4	7	the similar primitives without worries. Dereferencing (prefix "*"),
@@ -8,7 +11,7 @@ subtraction of constants, and casts all work quite naturally and safely. · 3 unmodified lines
8	11	It is nevertheless possible to get into trouble with other operations.
9	12	Follow these rules to keep your RCU code working properly:
10	13
11		−o You must use one of the rcu_dereference() family of primitives
	14	+- You must use one of the rcu_dereference() family of primitives
12	15	to load an RCU-protected pointer, otherwise CONFIG_PROVE_RCU
13	16	will complain. Worse yet, your code can see random memory-corruption
14	17	bugs due to games that compilers and DEC Alpha can play.
@@ -25,24 +28,24 @@ o You must use one of the rcu_dereference() family of primitives · 10 unmodified lines
25	28	for an example where the compiler can in fact deduce the exact
26	29	value of the pointer, and thus cause misordering.
27	30
28		−o You are only permitted to use rcu_dereference on pointer values.
	31	+- You are only permitted to use rcu_dereference on pointer values.
29	32	The compiler simply knows too much about integral values to
30	33	trust it to carry dependencies through integer operations.
31	34	There are a very few exceptions, namely that you can temporarily
32	35	cast the pointer to uintptr_t in order to:
33	36
34		− o Set bits and clear bits down in the must-be-zero low-order
	37	+ - Set bits and clear bits down in the must-be-zero low-order
35	38	bits of that pointer. This clearly means that the pointer
36	39	must have alignment constraints, for example, this does
37	40	-not- work in general for char* pointers.
38	41
39		− o XOR bits to translate pointers, as is done in some
	42	+ - XOR bits to translate pointers, as is done in some
40	43	classic buddy-allocator algorithms.
41	44
42	45	It is important to cast the value back to pointer before
43	46	doing much of anything else with it.
44	47
45		−o Avoid cancellation when using the "+" and "-" infix arithmetic
	48	+- Avoid cancellation when using the "+" and "-" infix arithmetic
46	49	operators. For example, for a given variable "x", avoid
47	50	"(x-(uintptr_t)x)" for char* pointers. The compiler is within its
48	51	rights to substitute zero for this sort of expression, so that
@@ -54,16 +57,16 @@ o Avoid cancellation when using the "+" and "-" infix arithmetic · 5 unmodified lines
54	57	"p+a-b" is safe because its value still necessarily depends on
55	58	the rcu_dereference(), thus maintaining proper ordering.
56	59
57		−o If you are using RCU to protect JITed functions, so that the
	60	+- If you are using RCU to protect JITed functions, so that the
58	61	"()" function-invocation operator is applied to a value obtained
59	62	(directly or indirectly) from rcu_dereference(), you may need to
60	63	interact directly with the hardware to flush instruction caches.
61	64	This issue arises on some systems when a newly JITed function is
62	65	using the same memory that was used by an earlier JITed function.
63	66
64		−o Do not use the results from relational operators ("==", "!=",
	67	+- Do not use the results from relational operators ("==", "!=",
65	68	">", ">=", "<", or "<=") when dereferencing. For example,
66		− the following (quite strange) code is buggy:
	69	+ the following (quite strange) code is buggy::
67	70
68	71	int *p;
69	72	int *q;
@@ -81,11 +84,11 @@ o Do not use the results from relational operators ("==", "!=", · 11 unmodified lines
81	84	after such branches, but can speculate loads, which can again
82	85	result in misordering bugs.
83	86
84		−o Be very careful about comparing pointers obtained from
	87	+- Be very careful about comparing pointers obtained from
85	88	rcu_dereference() against non-NULL values. As Linus Torvalds
86	89	explained, if the two pointers are equal, the compiler could
87	90	substitute the pointer you are comparing against for the pointer
88		− obtained from rcu_dereference(). For example:
	91	+ obtained from rcu_dereference(). For example::
89	92
90	93	p = rcu_dereference(gp);
91	94	if (p == &default_struct)
@@ -93,7 +96,7 @@ o Be very careful about comparing pointers obtained from · 1 unmodified line
93	96
94	97	Because the compiler now knows that the value of "p" is exactly
95	98	the address of the variable "default_struct", it is free to
96		− transform this code into the following:
	99	+ transform this code into the following::
97	100
98	101	p = rcu_dereference(gp);
99	102	if (p == &default_struct)
@@ -105,14 +108,14 @@ o Be very careful about comparing pointers obtained from · 5 unmodified lines
105	108
106	109	However, comparisons are OK in the following cases:
107	110
108		− o The comparison was against the NULL pointer. If the
	111	+ - The comparison was against the NULL pointer. If the
109	112	compiler knows that the pointer is NULL, you had better
110	113	not be dereferencing it anyway. If the comparison is
111	114	non-equal, the compiler is none the wiser. Therefore,
112	115	it is safe to compare pointers from rcu_dereference()
113	116	against NULL pointers.
114	117
115		− o The pointer is never dereferenced after being compared.
	118	+ - The pointer is never dereferenced after being compared.
116	119	Since there are no subsequent dereferences, the compiler
117	120	cannot use anything it learned from the comparison
118	121	to reorder the non-existent subsequent dereferences.
@@ -124,31 +127,31 @@ o Be very careful about comparing pointers obtained from · 5 unmodified lines
124	127	dereferenced, rcu_access_pointer() should be used in place
125	128	of rcu_dereference().
126	129
127		− o The comparison is against a pointer that references memory
	130	+ - The comparison is against a pointer that references memory
128	131	that was initialized "a long time ago." The reason
129	132	this is safe is that even if misordering occurs, the
130	133	misordering will not affect the accesses that follow
131	134	the comparison. So exactly how long ago is "a long
132	135	time ago"? Here are some possibilities:
133	136
134		− o Compile time.
	137	+ - Compile time.
135	138
136		− o Boot time.
	139	+ - Boot time.
137	140
138		− o Module-init time for module code.
	141	+ - Module-init time for module code.
139	142
140		− o Prior to kthread creation for kthread code.
	143	+ - Prior to kthread creation for kthread code.
141	144
142		− o During some prior acquisition of the lock that
	145	+ - During some prior acquisition of the lock that
143	146	we now hold.
144	147
145		− o Before mod_timer() time for a timer handler.
	148	+ - Before mod_timer() time for a timer handler.
146	149
147	150	There are many other possibilities involving the Linux
148	151	kernel's wide array of primitives that cause code to
149	152	be invoked at a later time.
150	153
151		− o The pointer being compared against also came from
	154	+ - The pointer being compared against also came from
152	155	rcu_dereference(). In this case, both pointers depend
153	156	on one rcu_dereference() or another, so you get proper
154	157	ordering either way.
@@ -159,13 +162,13 @@ o Be very careful about comparing pointers obtained from · 4 unmodified lines
159	162	of such an RCU usage bug is shown in the section titled
160	163	"EXAMPLE OF AMPLIFIED RCU-USAGE BUG".
161	164
162		− o All of the accesses following the comparison are stores,
	165	+ - All of the accesses following the comparison are stores,
163	166	so that a control dependency preserves the needed ordering.
164	167	That said, it is easy to get control dependencies wrong.
165	168	Please see the "CONTROL DEPENDENCIES" section of
166	169	Documentation/memory-barriers.txt for more details.
167	170
168		− o The pointers are not equal -and- the compiler does
	171	+ - The pointers are not equal -and- the compiler does
169	172	not have enough information to deduce the value of the
170	173	pointer. Note that the volatile cast in rcu_dereference()
171	174	will normally prevent the compiler from knowing too much.
@@ -175,7 +178,7 @@ o Be very careful about comparing pointers obtained from · 3 unmodified lines
175	178	comparison will provide exactly the information that the
176	179	compiler needs to deduce the value of the pointer.
177	180
178		−o Disable any value-speculation optimizations that your compiler
	181	+- Disable any value-speculation optimizations that your compiler
179	182	might provide, especially if you are making use of feedback-based
180	183	optimizations that take data collected from prior runs. Such
181	184	value-speculation optimizations reorder operations by design.
@@ -188,11 +191,12 @@ o Disable any value-speculation optimizations that your compiler · 6 unmodified lines
188	191
189	192
190	193	EXAMPLE OF AMPLIFIED RCU-USAGE BUG
	194	+----------------------------------
191	195
192	196	Because updaters can run concurrently with RCU readers, RCU readers can
193	197	see stale and/or inconsistent values. If RCU readers need fresh or
194	198	consistent values, which they sometimes do, they need to take proper
195		−precautions. To see this, consider the following code fragment:
	199	+precautions. To see this, consider the following code fragment::
196	200
197	201	struct foo {
198	202	int a;
@@ -244,7 +248,7 @@ to some reordering from the compiler and CPUs is beside the point. · 45 unmodified lines
244	248
245	249	But suppose that the reader needs a consistent view?
246	250
247		−Then one approach is to use locking, for example, as follows:
	251	+Then one approach is to use locking, for example, as follows::
248	252
249	253	struct foo {
250	254	int a;
@@ -299,6 +303,7 @@ As always, use the right tool for the job! · 48 unmodified lines
299	303
300	304
301	305	EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH
	306	+-----------------------------------------
302	307
303	308	If a pointer obtained from rcu_dereference() compares not-equal to some
304	309	other pointer, the compiler normally has no clue what the value of the
@@ -308,7 +313,7 @@ guarantees that RCU depends on. And the volatile cast in rcu_dereference() · 3 unmodified lines
308	313	should prevent the compiler from guessing the value.
309	314
310	315	But without rcu_dereference(), the compiler knows more than you might
311		−expect. Consider the following code fragment:
	316	+expect. Consider the following code fragment::
312	317
313	318	struct foo {
314	319	int a;
@@ -354,6 +359,7 @@ dereference the resulting pointer. · 39 unmodified lines
354	359
355	360
356	361	WHICH MEMBER OF THE rcu_dereference() FAMILY SHOULD YOU USE?
	362	+------------------------------------------------------------
357	363
358	364	First, please avoid using rcu_dereference_raw() and also please avoid
359	365	using rcu_dereference_check() and rcu_dereference_protected() with a
@@ -370,7 +376,7 @@ member of the rcu_dereference() to use in various situations: · 10 unmodified lines
370	376
371	377	2. If the access might be within an RCU read-side critical section
372	378	on the one hand, or protected by (say) my_lock on the other,
373		− use rcu_dereference_check(), for example:
	379	+ use rcu_dereference_check(), for example::
374	380
375	381	p1 = rcu_dereference_check(p->rcu_protected_pointer,
376	382	lockdep_is_held(&my_lock));
@@ -378,14 +384,14 @@ member of the rcu_dereference() to use in various situations: · 1 unmodified line
378	384
379	385	3. If the access might be within an RCU read-side critical section
380	386	on the one hand, or protected by either my_lock or your_lock on
381		− the other, again use rcu_dereference_check(), for example:
	387	+ the other, again use rcu_dereference_check(), for example::
382	388
383	389	p1 = rcu_dereference_check(p->rcu_protected_pointer,
384	390	lockdep_is_held(&my_lock) \|\|
385	391	lockdep_is_held(&your_lock));
386	392
387	393	4. If the access is on the update side, so that it is always protected
388		− by my_lock, use rcu_dereference_protected():
	394	+ by my_lock, use rcu_dereference_protected()::
389	395
390	396	p1 = rcu_dereference_protected(p->rcu_protected_pointer,
391	397	lockdep_is_held(&my_lock));
@@ -410,18 +416,19 @@ member of the rcu_dereference() to use in various situations: · 18 unmodified lines
410	416
411	417
412	418	SPARSE CHECKING OF RCU-PROTECTED POINTERS
	419	+-----------------------------------------
413	420
414	421	The sparse static-analysis tool checks for direct access to RCU-protected
415	422	pointers, which can result in "interesting" bugs due to compiler
416	423	optimizations involving invented loads and perhaps also load tearing.
417		−For example, suppose someone mistakenly does something like this:
	424	+For example, suppose someone mistakenly does something like this::
418	425
419	426	p = q->rcu_protected_pointer;
420	427	do_something_with(p->a);
421	428	do_something_else_with(p->b);
422	429
423	430	If register pressure is high, the compiler might optimize "p" out
424		−of existence, transforming the code to something like this:
	431	+of existence, transforming the code to something like this::
425	432
426	433	do_something_with(q->rcu_protected_pointer->a);
427	434	do_something_else_with(q->rcu_protected_pointer->b);
@@ -435,7 +442,7 @@ Load tearing could of course result in dereferencing a mashup of a pair · 7 unmodified lines
435	442	of pointers, which also might fatally disappoint your code.
436	443
437	444	These problems could have been avoided simply by making the code instead
438		−read as follows:
	445	+read as follows::
439	446
440	447	p = rcu_dereference(q->rcu_protected_pointer);
441	448	do_something_with(p->a);
@@ -448,7 +455,7 @@ or as a formal parameter, with "__rcu", which tells sparse to complain if · 6 unmodified lines
448	455	this pointer is accessed directly. It will also cause sparse to complain
449	456	if a pointer not marked with "__rcu" is accessed using rcu_dereference()
450	457	and friends. For example, ->rcu_protected_pointer might be declared as
451		−follows:
	458	+follows::
452	459
453	460	struct foo __rcu *rcu_protected_pointer;
454	461

Documentation/RCU/rcubarrier.rstrenamed from Documentation/RCU/rcubarrier.txt+125−97

@@ -1,4 +1,7 @@
	1	+.. _rcu_barrier:
	2	+
1	3	RCU and Unloadable Modules
	4	+==========================
2	5
3	6	[Originally published in LWN Jan. 14, 2007: http://lwn.net/Articles/217484/]
4	7
@@ -21,7 +24,7 @@ given that readers might well leave absolutely no trace of their · 16 unmodified lines
21	24	presence? There is a synchronize_rcu() primitive that blocks until all
22	25	pre-existing readers have completed. An updater wishing to delete an
23	26	element p from a linked list might do the following, while holding an
24		−appropriate lock, of course:
	27	+appropriate lock, of course::
25	28
26	29	list_del_rcu(p);
27	30	synchronize_rcu();
@@ -32,13 +35,13 @@ primitive must be used instead. This primitive takes a pointer to an · 4 unmodified lines
32	35	rcu_head struct placed within the RCU-protected data structure and
33	36	another pointer to a function that may be invoked later to free that
34	37	structure. Code to delete an element p from the linked list from IRQ
35		−context might then be as follows:
	38	+context might then be as follows::
36	39
37	40	list_del_rcu(p);
38	41	call_rcu(&p->rcu, p_callback);
39	42
40	43	Since call_rcu() never blocks, this code can safely be used from within
41		−IRQ context. The function p_callback() might be defined as follows:
	44	+IRQ context. The function p_callback() might be defined as follows::
42	45
43	46	static void p_callback(struct rcu_head *rp)
44	47	{
@@ -49,6 +52,7 @@ IRQ context. The function p_callback() might be defined as follows: · 4 unmodified lines
49	52
50	53
51	54	Unloading Modules That Use call_rcu()
	55	+-------------------------------------
52	56
53	57	But what if p_callback is defined in an unloadable module?
54	58
@@ -69,10 +73,11 @@ in realtime kernels in order to avoid excessive scheduling latencies. · 14 unmodified lines
69	73
70	74
71	75	rcu_barrier()
	76	+-------------
72	77
73	78	We instead need the rcu_barrier() primitive. Rather than waiting for
74	79	a grace period to elapse, rcu_barrier() waits for all outstanding RCU
75		−callbacks to complete. Please note that rcu_barrier() does -not- imply
	80	+callbacks to complete. Please note that rcu_barrier() does not imply
76	81	synchronize_rcu(), in particular, if there are no RCU callbacks queued
77	82	anywhere, rcu_barrier() is within its rights to return immediately,
78	83	without waiting for a grace period to elapse.
@@ -88,79 +93,79 @@ must match the flavor of rcu_barrier() with that of call_rcu(). If your · 9 unmodified lines
88	93	module uses multiple flavors of call_rcu(), then it must also use multiple
89	94	flavors of rcu_barrier() when unloading that module. For example, if
90	95	it uses call_rcu(), call_srcu() on srcu_struct_1, and call_srcu() on
91		−srcu_struct_2(), then the following three lines of code will be required
92		−when unloading:
	96	+srcu_struct_2, then the following three lines of code will be required
	97	+when unloading::
93	98
94	99	1 rcu_barrier();
95	100	2 srcu_barrier(&srcu_struct_1);
96	101	3 srcu_barrier(&srcu_struct_2);
97	102
98	103	The rcutorture module makes use of rcu_barrier() in its exit function
99		−as follows:
	104	+as follows::
100	105
101		− 1 static void
102		− 2 rcu_torture_cleanup(void)
103		− 3 {
104		− 4 int i;
	106	+ 1 static void
	107	+ 2 rcu_torture_cleanup(void)
	108	+ 3 {
	109	+ 4 int i;
105	110	5
106		− 6 fullstop = 1;
107		− 7 if (shuffler_task != NULL) {
	111	+ 6 fullstop = 1;
	112	+ 7 if (shuffler_task != NULL) {
108	113	8 VERBOSE_PRINTK_STRING("Stopping rcu_torture_shuffle task");
109	114	9 kthread_stop(shuffler_task);
110		−10 }
111		−11 shuffler_task = NULL;
112		−12
113		−13 if (writer_task != NULL) {
114		−14 VERBOSE_PRINTK_STRING("Stopping rcu_torture_writer task");
115		−15 kthread_stop(writer_task);
116		−16 }
117		−17 writer_task = NULL;
118		−18
119		−19 if (reader_tasks != NULL) {
120		−20 for (i = 0; i < nrealreaders; i++) {
121		−21 if (reader_tasks[i] != NULL) {
122		−22 VERBOSE_PRINTK_STRING(
123		−23 "Stopping rcu_torture_reader task");
124		−24 kthread_stop(reader_tasks[i]);
125		−25 }
126		−26 reader_tasks[i] = NULL;
127		−27 }
128		−28 kfree(reader_tasks);
129		−29 reader_tasks = NULL;
130		−30 }
131		−31 rcu_torture_current = NULL;
132		−32
133		−33 if (fakewriter_tasks != NULL) {
134		−34 for (i = 0; i < nfakewriters; i++) {
135		−35 if (fakewriter_tasks[i] != NULL) {
136		−36 VERBOSE_PRINTK_STRING(
137		−37 "Stopping rcu_torture_fakewriter task");
138		−38 kthread_stop(fakewriter_tasks[i]);
139		−39 }
140		−40 fakewriter_tasks[i] = NULL;
141		−41 }
142		−42 kfree(fakewriter_tasks);
143		−43 fakewriter_tasks = NULL;
144		−44 }
145		−45
146		−46 if (stats_task != NULL) {
147		−47 VERBOSE_PRINTK_STRING("Stopping rcu_torture_stats task");
148		−48 kthread_stop(stats_task);
149		−49 }
150		−50 stats_task = NULL;
151		−51
152		−52 /* Wait for all RCU callbacks to fire. */
153		−53 rcu_barrier();
154		−54
155		−55 rcu_torture_stats_print(); /* -After- the stats thread is stopped! */
156		−56
157		−57 if (cur_ops->cleanup != NULL)
158		−58 cur_ops->cleanup();
159		−59 if (atomic_read(&n_rcu_torture_error))
160		−60 rcu_torture_print_module_parms("End of test: FAILURE");
161		−61 else
162		−62 rcu_torture_print_module_parms("End of test: SUCCESS");
163		−63 }
	115	+ 10 }
	116	+ 11 shuffler_task = NULL;
	117	+ 12
	118	+ 13 if (writer_task != NULL) {
	119	+ 14 VERBOSE_PRINTK_STRING("Stopping rcu_torture_writer task");
	120	+ 15 kthread_stop(writer_task);
	121	+ 16 }
	122	+ 17 writer_task = NULL;
	123	+ 18
	124	+ 19 if (reader_tasks != NULL) {
	125	+ 20 for (i = 0; i < nrealreaders; i++) {
	126	+ 21 if (reader_tasks[i] != NULL) {
	127	+ 22 VERBOSE_PRINTK_STRING(
	128	+ 23 "Stopping rcu_torture_reader task");
	129	+ 24 kthread_stop(reader_tasks[i]);
	130	+ 25 }
	131	+ 26 reader_tasks[i] = NULL;
	132	+ 27 }
	133	+ 28 kfree(reader_tasks);
	134	+ 29 reader_tasks = NULL;
	135	+ 30 }
	136	+ 31 rcu_torture_current = NULL;
	137	+ 32
	138	+ 33 if (fakewriter_tasks != NULL) {
	139	+ 34 for (i = 0; i < nfakewriters; i++) {
	140	+ 35 if (fakewriter_tasks[i] != NULL) {
	141	+ 36 VERBOSE_PRINTK_STRING(
	142	+ 37 "Stopping rcu_torture_fakewriter task");
	143	+ 38 kthread_stop(fakewriter_tasks[i]);
	144	+ 39 }
	145	+ 40 fakewriter_tasks[i] = NULL;
	146	+ 41 }
	147	+ 42 kfree(fakewriter_tasks);
	148	+ 43 fakewriter_tasks = NULL;
	149	+ 44 }
	150	+ 45
	151	+ 46 if (stats_task != NULL) {
	152	+ 47 VERBOSE_PRINTK_STRING("Stopping rcu_torture_stats task");
	153	+ 48 kthread_stop(stats_task);
	154	+ 49 }
	155	+ 50 stats_task = NULL;
	156	+ 51
	157	+ 52 /* Wait for all RCU callbacks to fire. */
	158	+ 53 rcu_barrier();
	159	+ 54
	160	+ 55 rcu_torture_stats_print(); /* -After- the stats thread is stopped! */
	161	+ 56
	162	+ 57 if (cur_ops->cleanup != NULL)
	163	+ 58 cur_ops->cleanup();
	164	+ 59 if (atomic_read(&n_rcu_torture_error))
	165	+ 60 rcu_torture_print_module_parms("End of test: FAILURE");
	166	+ 61 else
	167	+ 62 rcu_torture_print_module_parms("End of test: SUCCESS");
	168	+ 63 }
164	169
165	170	Line 6 sets a global variable that prevents any RCU callbacks from
166	171	re-posting themselves. This will not be necessary in most cases, since
@@ -176,9 +181,14 @@ for any pre-existing callbacks to complete. · 9 unmodified lines
176	181	Then lines 55-62 print status and do operation-specific cleanup, and
177	182	then return, permitting the module-unload operation to be completed.
178	183
179		−Quick Quiz #1: Is there any other situation where rcu_barrier() might
	184	+.. _rcubarrier_quiz_1:
	185	+
	186	+Quick Quiz #1:
	187	+ Is there any other situation where rcu_barrier() might
180	188	be required?
181	189
	190	+:ref:`Answer to Quick Quiz #1 <answer_rcubarrier_quiz_1>`
	191	+
182	192	Your module might have additional complications. For example, if your
183	193	module invokes call_rcu() from timers, you will need to first cancel all
184	194	the timers, and only then invoke rcu_barrier() to wait for any remaining
@@ -188,11 +198,12 @@ Of course, if you module uses call_rcu(), you will need to invoke · 3 unmodified lines
188	198	rcu_barrier() before unloading. Similarly, if your module uses
189	199	call_srcu(), you will need to invoke srcu_barrier() before unloading,
190	200	and on the same srcu_struct structure. If your module uses call_rcu()
191		−-and- call_srcu(), then you will need to invoke rcu_barrier() -and-
	201	+and call_srcu(), then you will need to invoke rcu_barrier() and
192	202	srcu_barrier().
193	203
194	204
195	205	Implementing rcu_barrier()
	206	+--------------------------
196	207
197	208	Dipankar Sarma's implementation of rcu_barrier() makes use of the fact
198	209	that RCU callbacks are never reordered once queued on one of the per-CPU
@@ -200,19 +211,19 @@ queues. His implementation queues an RCU callback on each of the per-CPU · 1 unmodified line
200	211	callback queues, and then waits until they have all started executing, at
201	212	which point, all earlier RCU callbacks are guaranteed to have completed.
202	213
203		−The original code for rcu_barrier() was as follows:
	214	+The original code for rcu_barrier() was as follows::
204	215
205		− 1 void rcu_barrier(void)
206		− 2 {
207		− 3 BUG_ON(in_interrupt());
208		− 4 /* Take cpucontrol mutex to protect against CPU hotplug */
209		− 5 mutex_lock(&rcu_barrier_mutex);
210		− 6 init_completion(&rcu_barrier_completion);
211		− 7 atomic_set(&rcu_barrier_cpu_count, 0);
212		− 8 on_each_cpu(rcu_barrier_func, NULL, 0, 1);
213		− 9 wait_for_completion(&rcu_barrier_completion);
214		−10 mutex_unlock(&rcu_barrier_mutex);
215		−11 }
	216	+ 1 void rcu_barrier(void)
	217	+ 2 {
	218	+ 3 BUG_ON(in_interrupt());
	219	+ 4 /* Take cpucontrol mutex to protect against CPU hotplug */
	220	+ 5 mutex_lock(&rcu_barrier_mutex);
	221	+ 6 init_completion(&rcu_barrier_completion);
	222	+ 7 atomic_set(&rcu_barrier_cpu_count, 0);
	223	+ 8 on_each_cpu(rcu_barrier_func, NULL, 0, 1);
	224	+ 9 wait_for_completion(&rcu_barrier_completion);
	225	+ 10 mutex_unlock(&rcu_barrier_mutex);
	226	+ 11 }
216	227
217	228	Line 3 verifies that the caller is in process context, and lines 5 and 10
218	229	use rcu_barrier_mutex to ensure that only one rcu_barrier() is using the
@@ -226,18 +237,18 @@ This code was rewritten in 2008 and several times thereafter, but this · 7 unmodified lines
226	237	still gives the general idea.
227	238
228	239	The rcu_barrier_func() runs on each CPU, where it invokes call_rcu()
229		−to post an RCU callback, as follows:
	240	+to post an RCU callback, as follows::
230	241
231		− 1 static void rcu_barrier_func(void *notused)
232		− 2 {
233		− 3 int cpu = smp_processor_id();
234		− 4 struct rcu_data *rdp = &per_cpu(rcu_data, cpu);
235		− 5 struct rcu_head *head;
	242	+ 1 static void rcu_barrier_func(void *notused)
	243	+ 2 {
	244	+ 3 int cpu = smp_processor_id();
	245	+ 4 struct rcu_data *rdp = &per_cpu(rcu_data, cpu);
	246	+ 5 struct rcu_head *head;
236	247	6
237		− 7 head = &rdp->barrier;
238		− 8 atomic_inc(&rcu_barrier_cpu_count);
239		− 9 call_rcu(head, rcu_barrier_callback);
240		−10 }
	248	+ 7 head = &rdp->barrier;
	249	+ 8 atomic_inc(&rcu_barrier_cpu_count);
	250	+ 9 call_rcu(head, rcu_barrier_callback);
	251	+ 10 }
241	252
242	253	Lines 3 and 4 locate RCU's internal per-CPU rcu_data structure,
243	254	which contains the struct rcu_head that needed for the later call to
@@ -248,20 +259,25 @@ the current CPU's queue. · 4 unmodified lines
248	259
249	260	The rcu_barrier_callback() function simply atomically decrements the
250	261	rcu_barrier_cpu_count variable and finalizes the completion when it
251		−reaches zero, as follows:
	262	+reaches zero, as follows::
252	263
253	264	1 static void rcu_barrier_callback(struct rcu_head *notused)
254	265	2 {
255		− 3 if (atomic_dec_and_test(&rcu_barrier_cpu_count))
256		− 4 complete(&rcu_barrier_completion);
	266	+ 3 if (atomic_dec_and_test(&rcu_barrier_cpu_count))
	267	+ 4 complete(&rcu_barrier_completion);
257	268	5 }
258	269
259		−Quick Quiz #2: What happens if CPU 0's rcu_barrier_func() executes
	270	+.. _rcubarrier_quiz_2:
	271	+
	272	+Quick Quiz #2:
	273	+ What happens if CPU 0's rcu_barrier_func() executes
260	274	immediately (thus incrementing rcu_barrier_cpu_count to the
261	275	value one), but the other CPU's rcu_barrier_func() invocations
262	276	are delayed for a full grace period? Couldn't this result in
263	277	rcu_barrier() returning prematurely?
264	278
	279	+:ref:`Answer to Quick Quiz #2 <answer_rcubarrier_quiz_2>`
	280	+
265	281	The current rcu_barrier() implementation is more complex, due to the need
266	282	to avoid disturbing idle CPUs (especially on battery-powered systems)
267	283	and the need to minimally disturb non-idle CPUs in real-time systems.
@@ -269,6 +285,7 @@ However, the code above illustrates the concepts. · 1 unmodified line
269	285
270	286
271	287	rcu_barrier() Summary
	288	+---------------------
272	289
273	290	The rcu_barrier() primitive has seen relatively little use, since most
274	291	code using RCU is in the core kernel rather than in modules. However, if
@@ -277,8 +294,12 @@ so that your module may be safely unloaded. · 2 unmodified lines
277	294
278	295
279	296	Answers to Quick Quizzes
	297	+------------------------
	298	+
	299	+.. _answer_rcubarrier_quiz_1:
280	300
281		−Quick Quiz #1: Is there any other situation where rcu_barrier() might
	301	+Quick Quiz #1:
	302	+ Is there any other situation where rcu_barrier() might
282	303	be required?
283	304
284	305	Answer: Interestingly enough, rcu_barrier() was not originally
@@ -292,7 +313,12 @@ Answer: Interestingly enough, rcu_barrier() was not originally · 7 unmodified lines
292	313	implementing rcutorture, and found that rcu_barrier() solves
293	314	this problem as well.
294	315
295		−Quick Quiz #2: What happens if CPU 0's rcu_barrier_func() executes
	316	+:ref:`Back to Quick Quiz #1 <rcubarrier_quiz_1>`
	317	+
	318	+.. _answer_rcubarrier_quiz_2:
	319	+
	320	+Quick Quiz #2:
	321	+ What happens if CPU 0's rcu_barrier_func() executes
296	322	immediately (thus incrementing rcu_barrier_cpu_count to the
297	323	value one), but the other CPU's rcu_barrier_func() invocations
298	324	are delayed for a full grace period? Couldn't this result in
@@ -323,3 +349,5 @@ Answer: This cannot happen. The reason is that on_each_cpu() has its last · 24 unmodified lines
323	349	is to add an rcu_read_lock() before line 8 of rcu_barrier()
324	350	and an rcu_read_unlock() after line 8 of this same function. If
325	351	you can think of a better change, please let me know!
	352	+
	353	+:ref:`Back to Quick Quiz #2 <rcubarrier_quiz_2>`

Documentation/RCU/stallwarn.txt+3−8

@@ -225,18 +225,13 @@ an estimate of the total number of RCU callbacks queued across all CPUs · 224 unmodified lines
225	225	In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed
226	226	for each CPU:
227	227
228		− 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 Nonlazy posted: ..D
	228	+ 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1
229	229
230	230	The "last_accelerate:" prints the low-order 16 bits (in hex) of the
231	231	jiffies counter when this CPU last invoked rcu_try_advance_all_cbs()
232	232	from rcu_needs_cpu() or last invoked rcu_accelerate_cbs() from
233		−rcu_prepare_for_idle(). The "Nonlazy posted:" indicates lazy-callback
234		−status, so that an "l" indicates that all callbacks were lazy at the start
235		−of the last idle period and an "L" indicates that there are currently
236		−no non-lazy callbacks (in both cases, "." is printed otherwise, as
237		−shown above) and "D" indicates that dyntick-idle processing is enabled
238		−("." is printed otherwise, for example, if disabled via the "nohz="
239		−kernel boot parameter).
	233	+rcu_prepare_for_idle(). "dyntick_enabled: 1" indicates that dyntick-idle
	234	+processing is enabled.
240	235
241	236	If the grace period ends just as the stall warning starts printing,
242	237	there will be a spurious stall-warning message, which will include

Documentation/RCU/whatisRCU.rstrenamed from Documentation/RCU/whatisRCU.txt+183−108

@@ -1,15 +1,18 @@
	1	+.. _whatisrcu_doc:
	2	+
1	3	What is RCU? -- "Read, Copy, Update"
	4	+======================================
2	5
3	6	Please note that the "What is RCU?" LWN series is an excellent place
4	7	to start learning about RCU:
5	8
6		−1. What is RCU, Fundamentally? http://lwn.net/Articles/262464/
7		−2. What is RCU? Part 2: Usage http://lwn.net/Articles/263130/
8		−3. RCU part 3: the RCU API http://lwn.net/Articles/264090/
9		−4. The RCU API, 2010 Edition http://lwn.net/Articles/418853/
10		− 2010 Big API Table http://lwn.net/Articles/419086/
11		−5. The RCU API, 2014 Edition http://lwn.net/Articles/609904/
12		− 2014 Big API Table http://lwn.net/Articles/609973/
	9	+\| 1. What is RCU, Fundamentally? http://lwn.net/Articles/262464/
	10	+\| 2. What is RCU? Part 2: Usage http://lwn.net/Articles/263130/
	11	+\| 3. RCU part 3: the RCU API http://lwn.net/Articles/264090/
	12	+\| 4. The RCU API, 2010 Edition http://lwn.net/Articles/418853/
	13	+\| 2010 Big API Table http://lwn.net/Articles/419086/
	14	+\| 5. The RCU API, 2014 Edition http://lwn.net/Articles/609904/
	15	+\| 2014 Big API Table http://lwn.net/Articles/609973/
13	16
14	17
15	18	What is RCU?
@@ -24,14 +27,21 @@ the experience has been that different people must take different paths · 8 unmodified lines
24	27	to arrive at an understanding of RCU. This document provides several
25	28	different paths, as follows:
26	29
27		−1. RCU OVERVIEW
28		−2. WHAT IS RCU'S CORE API?
29		−3. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?
30		−4. WHAT IF MY UPDATING THREAD CANNOT BLOCK?
31		−5. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?
32		−6. ANALOGY WITH READER-WRITER LOCKING
33		−7. FULL LIST OF RCU APIs
34		−8. ANSWERS TO QUICK QUIZZES
	30	+:ref:`1. RCU OVERVIEW <1_whatisRCU>`
	31	+
	32	+:ref:`2. WHAT IS RCU'S CORE API? <2_whatisRCU>`
	33	+
	34	+:ref:`3. WHAT ARE SOME EXAMPLE USES OF CORE RCU API? <3_whatisRCU>`
	35	+
	36	+:ref:`4. WHAT IF MY UPDATING THREAD CANNOT BLOCK? <4_whatisRCU>`
	37	+
	38	+:ref:`5. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU? <5_whatisRCU>`
	39	+
	40	+:ref:`6. ANALOGY WITH READER-WRITER LOCKING <6_whatisRCU>`
	41	+
	42	+:ref:`7. FULL LIST OF RCU APIs <7_whatisRCU>`
	43	+
	44	+:ref:`8. ANSWERS TO QUICK QUIZZES <8_whatisRCU>`
35	45
36	46	People who prefer starting with a conceptual overview should focus on
37	47	Section 1, though most readers will profit by reading this section at
@@ -49,8 +59,10 @@ everything, feel free to read the whole thing -- but if you are really · 11 unmodified lines
49	59	that type of person, you have perused the source code and will therefore
50	60	never need this document anyway. ;-)
51	61
	62	+.. _1_whatisRCU:
52	63
53	64	1. RCU OVERVIEW
	65	+----------------
54	66
55	67	The basic idea behind RCU is to split updates into "removal" and
56	68	"reclamation" phases. The removal phase removes references to data items
@@ -116,8 +128,10 @@ So how the heck can a reclaimer tell when a reader is done, given · 59 unmodified lines
116	128	that readers are not doing any sort of synchronization operations???
117	129	Read on to learn about how RCU's API makes this easy.
118	130
	131	+.. _2_whatisRCU:
119	132
120	133	2. WHAT IS RCU'S CORE API?
	134	+---------------------------
121	135
122	136	The core RCU API is quite small:
123	137
@@ -136,7 +150,7 @@ later. See the kernel docbook documentation for more info, or look directly · 12 unmodified lines
136	150	at the function header comments.
137	151
138	152	rcu_read_lock()
139		−
	153	+^^^^^^^^^^^^^^^
140	154	void rcu_read_lock(void);
141	155
142	156	Used by a reader to inform the reclaimer that the reader is
@@ -150,7 +164,7 @@ rcu_read_lock() · 7 unmodified lines
150	164	longer-term references to data structures.
151	165
152	166	rcu_read_unlock()
153		−
	167	+^^^^^^^^^^^^^^^^^
154	168	void rcu_read_unlock(void);
155	169
156	170	Used by a reader to inform the reclaimer that the reader is
@@ -158,15 +172,15 @@ rcu_read_unlock() · 1 unmodified line
158	172	read-side critical sections may be nested and/or overlapping.
159	173
160	174	synchronize_rcu()
161		−
	175	+^^^^^^^^^^^^^^^^^
162	176	void synchronize_rcu(void);
163	177
164	178	Marks the end of updater code and the beginning of reclaimer
165	179	code. It does this by blocking until all pre-existing RCU
166	180	read-side critical sections on all CPUs have completed.
167		− Note that synchronize_rcu() will -not- necessarily wait for
	181	+ Note that synchronize_rcu() will not necessarily wait for
168	182	any subsequent RCU read-side critical sections to complete.
169		− For example, consider the following sequence of events:
	183	+ For example, consider the following sequence of events::
170	184
171	185	CPU 0 CPU 1 CPU 2
172	186	----------------- ------------------------- ---------------
@@ -182,7 +196,7 @@ synchronize_rcu() · 9 unmodified lines
182	196	any that begin after synchronize_rcu() is invoked.
183	197
184	198	Of course, synchronize_rcu() does not necessarily return
185		− -immediately- after the last pre-existing RCU read-side critical
	199	+ immediately after the last pre-existing RCU read-side critical
186	200	section completes. For one thing, there might well be scheduling
187	201	delays. For another thing, many RCU implementations process
188	202	requests in batches in order to improve efficiencies, which can
@@ -211,10 +225,10 @@ synchronize_rcu() · 22 unmodified lines
211	225	checklist.txt for some approaches to limiting the update rate.
212	226
213	227	rcu_assign_pointer()
214		−
	228	+^^^^^^^^^^^^^^^^^^^^
215	229	void rcu_assign_pointer(p, typeof(p) v);
216	230
217		− Yes, rcu_assign_pointer() -is- implemented as a macro, though it
	231	+ Yes, rcu_assign_pointer() is implemented as a macro, though it
218	232	would be cool to be able to declare a function in this manner.
219	233	(Compiler experts will no doubt disagree.)
220	234
@@ -231,7 +245,7 @@ rcu_assign_pointer() · 10 unmodified lines
231	245	the _rcu list-manipulation primitives such as list_add_rcu().
232	246
233	247	rcu_dereference()
234		−
	248	+^^^^^^^^^^^^^^^^^
235	249	typeof(p) rcu_dereference(p);
236	250
237	251	Like rcu_assign_pointer(), rcu_dereference() must be implemented
@@ -248,13 +262,13 @@ rcu_dereference() · 10 unmodified lines
248	262
249	263	Common coding practice uses rcu_dereference() to copy an
250	264	RCU-protected pointer to a local variable, then dereferences
251		− this local variable, for example as follows:
	265	+ this local variable, for example as follows::
252	266
253	267	p = rcu_dereference(head.next);
254	268	return p->data;
255	269
256	270	However, in this case, one could just as easily combine these
257		− into one statement:
	271	+ into one statement::
258	272
259	273	return rcu_dereference(head.next)->data;
260	274
@@ -266,8 +280,8 @@ rcu_dereference() · 5 unmodified lines
266	280	unnecessary overhead on Alpha CPUs.
267	281
268	282	Note that the value returned by rcu_dereference() is valid
269		− only within the enclosing RCU read-side critical section [1].
270		− For example, the following is -not- legal:
	283	+ only within the enclosing RCU read-side critical section [1]_.
	284	+ For example, the following is not legal::
271	285
272	286	rcu_read_lock();
273	287	p = rcu_dereference(head.next);
@@ -290,9 +304,9 @@ rcu_dereference() · 16 unmodified lines
290	304	at any time, including immediately after the rcu_dereference().
291	305	And, again like rcu_assign_pointer(), rcu_dereference() is
292	306	typically used indirectly, via the _rcu list-manipulation
293		− primitives, such as list_for_each_entry_rcu() [2].
	307	+ primitives, such as list_for_each_entry_rcu() [2]_.
294	308
295		− [1] The variant rcu_dereference_protected() can be used outside
	309	+.. [1] The variant rcu_dereference_protected() can be used outside
296	310	of an RCU read-side critical section as long as the usage is
297	311	protected by locks acquired by the update-side code. This variant
298	312	avoids the lockdep warning that would happen when using (for
@@ -305,7 +319,7 @@ rcu_dereference() · 6 unmodified lines
305	319	a lockdep splat is emitted. See Documentation/RCU/Design/Requirements/Requirements.rst
306	320	and the API's code comments for more details and example usage.
307	321
308		− [2] If the list_for_each_entry_rcu() instance might be used by
	322	+.. [2] If the list_for_each_entry_rcu() instance might be used by
309	323	update-side code as well as by RCU readers, then an additional
310	324	lockdep expression can be added to its list of arguments.
311	325	For example, given an additional "lock_is_held(&mylock)" argument,
@@ -315,6 +329,7 @@ rcu_dereference() · 3 unmodified lines
315	329
316	330	The following diagram shows how each API communicates among the
317	331	reader, updater, and reclaimer.
	332	+::
318	333
319	334
320	335	rcu_assign_pointer()
@@ -375,12 +390,16 @@ c. RCU applied to scheduler and interrupt/NMI-handler tasks. · 54 unmodified lines
375	390	Again, most uses will be of (a). The (b) and (c) cases are important
376	391	for specialized uses, but are relatively uncommon.
377	392
	393	+.. _3_whatisRCU:
378	394
379	395	3. WHAT ARE SOME EXAMPLE USES OF CORE RCU API?
	396	+-----------------------------------------------
380	397
381	398	This section shows a simple use of the core RCU API to protect a
382	399	global pointer to a dynamically allocated structure. More-typical
383		−uses of RCU may be found in listRCU.txt, arrayRCU.txt, and NMI-RCU.txt.
	400	+uses of RCU may be found in :ref:`listRCU.rst <list_rcu_doc>`,
	401	+:ref:`arrayRCU.rst <array_rcu_doc>`, and :ref:`NMI-RCU.rst <NMI_rcu_doc>`.
	402	+::
384	403
385	404	struct foo {
386	405	int a;
@@ -440,40 +459,43 @@ uses of RCU may be found in listRCU.txt, arrayRCU.txt, and NMI-RCU.txt. · 53 unmodified lines
440	459
441	460	So, to sum up:
442	461
443		−o Use rcu_read_lock() and rcu_read_unlock() to guard RCU
	462	+- Use rcu_read_lock() and rcu_read_unlock() to guard RCU
444	463	read-side critical sections.
445	464
446		−o Within an RCU read-side critical section, use rcu_dereference()
	465	+- Within an RCU read-side critical section, use rcu_dereference()
447	466	to dereference RCU-protected pointers.
448	467
449		−o Use some solid scheme (such as locks or semaphores) to
	468	+- Use some solid scheme (such as locks or semaphores) to
450	469	keep concurrent updates from interfering with each other.
451	470
452		−o Use rcu_assign_pointer() to update an RCU-protected pointer.
	471	+- Use rcu_assign_pointer() to update an RCU-protected pointer.
453	472	This primitive protects concurrent readers from the updater,
454		− -not- concurrent updates from each other! You therefore still
	473	+ not concurrent updates from each other! You therefore still
455	474	need to use locking (or something similar) to keep concurrent
456	475	rcu_assign_pointer() primitives from interfering with each other.
457	476
458		−o Use synchronize_rcu() -after- removing a data element from an
459		− RCU-protected data structure, but -before- reclaiming/freeing
	477	+- Use synchronize_rcu() after removing a data element from an
	478	+ RCU-protected data structure, but before reclaiming/freeing
460	479	the data element, in order to wait for the completion of all
461	480	RCU read-side critical sections that might be referencing that
462	481	data item.
463	482
464	483	See checklist.txt for additional rules to follow when using RCU.
465		−And again, more-typical uses of RCU may be found in listRCU.txt,
466		−arrayRCU.txt, and NMI-RCU.txt.
	484	+And again, more-typical uses of RCU may be found in :ref:`listRCU.rst
	485	+<list_rcu_doc>`, :ref:`arrayRCU.rst <array_rcu_doc>`, and :ref:`NMI-RCU.rst
	486	+<NMI_rcu_doc>`.
467	487
	488	+.. _4_whatisRCU:
468	489
469	490	4. WHAT IF MY UPDATING THREAD CANNOT BLOCK?
	491	+--------------------------------------------
470	492
471	493	In the example above, foo_update_a() blocks until a grace period elapses.
472	494	This is quite simple, but in some cases one cannot afford to wait so
473	495	long -- there might be other high-priority work to be done.
474	496
475	497	In such cases, one uses call_rcu() rather than synchronize_rcu().
476		−The call_rcu() API is as follows:
	498	+The call_rcu() API is as follows::
477	499
478	500	void call_rcu(struct rcu_head * head,
479	501	void (func)(struct rcu_head head));
@@ -481,7 +503,7 @@ The call_rcu() API is as follows: · 1 unmodified line
481	503	This function invokes func(head) after a grace period has elapsed.
482	504	This invocation might happen from either softirq or process context,
483	505	so the function is not permitted to block. The foo struct needs to
484		−have an rcu_head structure added, perhaps as follows:
	506	+have an rcu_head structure added, perhaps as follows::
485	507
486	508	struct foo {
487	509	int a;
@@ -490,7 +512,7 @@ have an rcu_head structure added, perhaps as follows: · 2 unmodified lines
490	512	struct rcu_head rcu;
491	513	};
492	514
493		−The foo_update_a() function might then be written as follows:
	515	+The foo_update_a() function might then be written as follows::
494	516
495	517	/*
496	518	* Create a new struct foo that is the same as the one currently
@@ -520,7 +542,7 @@ The foo_update_a() function might then be written as follows: · 23 unmodified lines
520	542	call_rcu(&old_fp->rcu, foo_reclaim);
521	543	}
522	544
523		−The foo_reclaim() function might appear as follows:
	545	+The foo_reclaim() function might appear as follows::
524	546
525	547	void foo_reclaim(struct rcu_head *rp)
526	548	{
@@ -544,7 +566,7 @@ namely foo_reclaim(). · 17 unmodified lines
544	566	The summary of advice is the same as for the previous section, except
545	567	that we are now using call_rcu() rather than synchronize_rcu():
546	568
547		−o Use call_rcu() -after- removing a data element from an
	569	+- Use call_rcu() after removing a data element from an
548	570	RCU-protected data structure in order to register a callback
549	571	function that will be invoked after the completion of all RCU
550	572	read-side critical sections that might be referencing that
@@ -552,14 +574,16 @@ o Use call_rcu() -after- removing a data element from an · 1 unmodified line
552	574
553	575	If the callback for call_rcu() is not doing anything more than calling
554	576	kfree() on the structure, you can use kfree_rcu() instead of call_rcu()
555		−to avoid having to write your own callback:
	577	+to avoid having to write your own callback::
556	578
557	579	kfree_rcu(old_fp, rcu);
558	580
559	581	Again, see checklist.txt for additional rules governing the use of RCU.
560	582
	583	+.. _5_whatisRCU:
561	584
562	585	5. WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU?
	586	+------------------------------------------------
563	587
564	588	One of the nice things about RCU is that it has extremely simple "toy"
565	589	implementations that are a good first step towards understanding the
@@ -579,7 +603,7 @@ more details on the current implementation as of early 2004. · 13 unmodified lines
579	603
580	604
581	605	5A. "TOY" IMPLEMENTATION #1: LOCKING
582		−
	606	+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
583	607	This section presents a "toy" RCU implementation that is based on
584	608	familiar locking primitives. Its overhead makes it a non-starter for
585	609	real-life use, as does its lack of scalability. It is also unsuitable
@@ -591,7 +615,7 @@ you allow nested rcu_read_lock() calls, you can deadlock. · 5 unmodified lines
591	615	However, it is probably the easiest implementation to relate to, so is
592	616	a good starting point.
593	617
594		−It is extremely simple:
	618	+It is extremely simple::
595	619
596	620	static DEFINE_RWLOCK(rcu_gp_mutex);
597	621
@@ -614,7 +638,7 @@ It is extremely simple: · 16 unmodified lines
614	638
615	639	[You can ignore rcu_assign_pointer() and rcu_dereference() without missing
616	640	much. But here are simplified versions anyway. And whatever you do,
617		−don't forget about them when submitting patches making use of RCU!]
	641	+don't forget about them when submitting patches making use of RCU!]::
618	642
619	643	#define rcu_assign_pointer(p, v) \
620	644	({ \
@@ -647,18 +671,23 @@ that the only thing that can block rcu_read_lock() is a synchronize_rcu(). · 26 unmodified lines
647	671	But synchronize_rcu() does not acquire any locks while holding rcu_gp_mutex,
648	672	so there can be no deadlock cycle.
649	673
650		−Quick Quiz #1: Why is this argument naive? How could a deadlock
	674	+.. _quiz_1:
	675	+
	676	+Quick Quiz #1:
	677	+ Why is this argument naive? How could a deadlock
651	678	occur when using this algorithm in a real-world Linux
652	679	kernel? How could this deadlock be avoided?
653	680
	681	+:ref:`Answers to Quick Quiz <8_whatisRCU>`
654	682
655	683	5B. "TOY" EXAMPLE #2: CLASSIC RCU
656		−
	684	+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
657	685	This section presents a "toy" RCU implementation that is based on
658	686	"classic RCU". It is also short on performance (but only for updates) and
659	687	on features such as hotplug CPU and the ability to run in CONFIG_PREEMPT
660	688	kernels. The definitions of rcu_dereference() and rcu_assign_pointer()
661	689	are the same as those shown in the preceding section, so they are omitted.
	690	+::
662	691
663	692	void rcu_read_lock(void) { }
664	693
@@ -683,14 +712,14 @@ CPU in turn. The run_on() primitive can be implemented straightforwardly · 18 unmodified lines
683	712	in terms of the sched_setaffinity() primitive. Of course, a somewhat less
684	713	"toy" implementation would restore the affinity upon completion rather
685	714	than just leaving all tasks running on the last CPU, but when I said
686		−"toy", I meant -toy-!
	715	+"toy", I meant toy!
687	716
688	717	So how the heck is this supposed to work???
689	718
690	719	Remember that it is illegal to block while in an RCU read-side critical
691	720	section. Therefore, if a given CPU executes a context switch, we know
692	721	that it must have completed all preceding RCU read-side critical sections.
693		−Once -all- CPUs have executed a context switch, then -all- preceding
	722	+Once all CPUs have executed a context switch, then all preceding
694	723	RCU read-side critical sections will have completed.
695	724
696	725	So, suppose that we remove a data item from its structure and then invoke
@@ -698,19 +727,32 @@ synchronize_rcu(). Once synchronize_rcu() returns, we are guaranteed · 1 unmodified line
698	727	that there are no RCU read-side critical sections holding a reference
699	728	to that data item, so we can safely reclaim it.
700	729
701		−Quick Quiz #2: Give an example where Classic RCU's read-side
702		− overhead is -negative-.
	730	+.. _quiz_2:
	731	+
	732	+Quick Quiz #2:
	733	+ Give an example where Classic RCU's read-side
	734	+ overhead is negative.
	735	+
	736	+:ref:`Answers to Quick Quiz <8_whatisRCU>`
703	737
704		−Quick Quiz #3: If it is illegal to block in an RCU read-side
	738	+.. _quiz_3:
	739	+
	740	+Quick Quiz #3:
	741	+ If it is illegal to block in an RCU read-side
705	742	critical section, what the heck do you do in
706	743	PREEMPT_RT, where normal spinlocks can block???
707	744
	745	+:ref:`Answers to Quick Quiz <8_whatisRCU>`
	746	+
	747	+.. _6_whatisRCU:
708	748
709	749	6. ANALOGY WITH READER-WRITER LOCKING
	750	+--------------------------------------
710	751
711	752	Although RCU can be used in many different ways, a very common use of
712	753	RCU is analogous to reader-writer locking. The following unified
713	754	diff shows how closely related RCU and reader-writer locking can be.
	755	+::
714	756
715	757	@@ -5,5 +5,5 @@ struct el {
716	758	int data;
@@ -762,7 +804,7 @@ diff shows how closely related RCU and reader-writer locking can be. · 45 unmodified lines
762	804	return 0;
763	805	}
764	806
765		−Or, for those who prefer a side-by-side listing:
	807	+Or, for those who prefer a side-by-side listing::
766	808
767	809	1 struct el { 1 struct el {
768	810	2 struct list_head list; 2 struct list_head list;
@@ -774,40 +816,44 @@ Or, for those who prefer a side-by-side listing: · 5 unmodified lines
774	816	8 rwlock_t listmutex; 8 spinlock_t listmutex;
775	817	9 struct el head; 9 struct el head;
776	818
777		− 1 int search(long key, int result) 1 int search(long key, int result)
778		− 2 { 2 {
779		− 3 struct list_head lp; 3 struct list_head lp;
780		− 4 struct el p; 4 struct el p;
781		− 5 5
782		− 6 read_lock(&listmutex); 6 rcu_read_lock();
783		− 7 list_for_each_entry(p, head, lp) { 7 list_for_each_entry_rcu(p, head, lp) {
784		− 8 if (p->key == key) { 8 if (p->key == key) {
785		− 9 result = p->data; 9 result = p->data;
786		−10 read_unlock(&listmutex); 10 rcu_read_unlock();
787		−11 return 1; 11 return 1;
788		−12 } 12 }
789		−13 } 13 }
790		−14 read_unlock(&listmutex); 14 rcu_read_unlock();
791		−15 return 0; 15 return 0;
792		−16 } 16 }
793		−
794		− 1 int delete(long key) 1 int delete(long key)
795		− 2 { 2 {
796		− 3 struct el p; 3 struct el p;
797		− 4 4
798		− 5 write_lock(&listmutex); 5 spin_lock(&listmutex);
799		− 6 list_for_each_entry(p, head, lp) { 6 list_for_each_entry(p, head, lp) {
800		− 7 if (p->key == key) { 7 if (p->key == key) {
801		− 8 list_del(&p->list); 8 list_del_rcu(&p->list);
802		− 9 write_unlock(&listmutex); 9 spin_unlock(&listmutex);
803		− 10 synchronize_rcu();
804		−10 kfree(p); 11 kfree(p);
805		−11 return 1; 12 return 1;
806		−12 } 13 }
807		−13 } 14 }
808		−14 write_unlock(&listmutex); 15 spin_unlock(&listmutex);
809		−15 return 0; 16 return 0;
810		−16 } 17 }
	819	+::
	820	+
	821	+ 1 int search(long key, int result) 1 int search(long key, int result)
	822	+ 2 { 2 {
	823	+ 3 struct list_head lp; 3 struct list_head lp;
	824	+ 4 struct el p; 4 struct el p;
	825	+ 5 5
	826	+ 6 read_lock(&listmutex); 6 rcu_read_lock();
	827	+ 7 list_for_each_entry(p, head, lp) { 7 list_for_each_entry_rcu(p, head, lp) {
	828	+ 8 if (p->key == key) { 8 if (p->key == key) {
	829	+ 9 result = p->data; 9 result = p->data;
	830	+ 10 read_unlock(&listmutex); 10 rcu_read_unlock();
	831	+ 11 return 1; 11 return 1;
	832	+ 12 } 12 }
	833	+ 13 } 13 }
	834	+ 14 read_unlock(&listmutex); 14 rcu_read_unlock();
	835	+ 15 return 0; 15 return 0;
	836	+ 16 } 16 }
	837	+
	838	+::
	839	+
	840	+ 1 int delete(long key) 1 int delete(long key)
	841	+ 2 { 2 {
	842	+ 3 struct el p; 3 struct el p;
	843	+ 4 4
	844	+ 5 write_lock(&listmutex); 5 spin_lock(&listmutex);
	845	+ 6 list_for_each_entry(p, head, lp) { 6 list_for_each_entry(p, head, lp) {
	846	+ 7 if (p->key == key) { 7 if (p->key == key) {
	847	+ 8 list_del(&p->list); 8 list_del_rcu(&p->list);
	848	+ 9 write_unlock(&listmutex); 9 spin_unlock(&listmutex);
	849	+ 10 synchronize_rcu();
	850	+ 10 kfree(p); 11 kfree(p);
	851	+ 11 return 1; 12 return 1;
	852	+ 12 } 13 }
	853	+ 13 } 14 }
	854	+ 14 write_unlock(&listmutex); 15 spin_unlock(&listmutex);
	855	+ 15 return 0; 16 return 0;
	856	+ 16 } 17 }
811	857
812	858	Either way, the differences are quite small. Read-side locking moves
813	859	to rcu_read_lock() and rcu_read_unlock, update-side locking moves from
@@ -825,22 +871,27 @@ delete() can now block. If this is a problem, there is a callback-based · 11 unmodified lines
825	871	mechanism that never blocks, namely call_rcu() or kfree_rcu(), that can
826	872	be used in place of synchronize_rcu().
827	873
	874	+.. _7_whatisRCU:
828	875
829	876	7. FULL LIST OF RCU APIs
	877	+-------------------------
830	878
831	879	The RCU APIs are documented in docbook-format header comments in the
832	880	Linux-kernel source code, but it helps to have a full list of the
833	881	APIs, since there does not appear to be a way to categorize them
834	882	in docbook. Here is the list, by category.
835	883
836		−RCU list traversal:
	884	+RCU list traversal::
837	885
838	886	list_entry_rcu
	887	+ list_entry_lockless
839	888	list_first_entry_rcu
840	889	list_next_rcu
841	890	list_for_each_entry_rcu
842	891	list_for_each_entry_continue_rcu
843	892	list_for_each_entry_from_rcu
	893	+ list_first_or_null_rcu
	894	+ list_next_or_null_rcu
844	895	hlist_first_rcu
845	896	hlist_next_rcu
846	897	hlist_pprev_rcu
@@ -854,7 +905,7 @@ RCU list traversal: · 7 unmodified lines
854	905	hlist_bl_first_rcu
855	906	hlist_bl_for_each_entry_rcu
856	907
857		−RCU pointer/list update:
	908	+RCU pointer/list update::
858	909
859	910	rcu_assign_pointer
860	911	list_add_rcu
@@ -864,10 +915,12 @@ RCU pointer/list update: · 3 unmodified lines
864	915	hlist_add_behind_rcu
865	916	hlist_add_before_rcu
866	917	hlist_add_head_rcu
	918	+ hlist_add_tail_rcu
867	919	hlist_del_rcu
868	920	hlist_del_init_rcu
869	921	hlist_replace_rcu
870		− list_splice_init_rcu()
	922	+ list_splice_init_rcu
	923	+ list_splice_tail_init_rcu
871	924	hlist_nulls_del_init_rcu
872	925	hlist_nulls_del_rcu
873	926	hlist_nulls_add_head_rcu
@@ -876,7 +929,9 @@ RCU pointer/list update: · 2 unmodified lines
876	929	hlist_bl_del_rcu
877	930	hlist_bl_set_first_rcu
878	931
879		−RCU: Critical sections Grace period Barrier
	932	+RCU::
	933	+
	934	+ Critical sections Grace period Barrier
880	935
881	936	rcu_read_lock synchronize_net rcu_barrier
882	937	rcu_read_unlock synchronize_rcu
@@ -885,7 +940,9 @@ RCU: Critical sections Grace period Barrier · 2 unmodified lines
885	940	rcu_dereference_check kfree_rcu
886	941	rcu_dereference_protected
887	942
888		−bh: Critical sections Grace period Barrier
	943	+bh::
	944	+
	945	+ Critical sections Grace period Barrier
889	946
890	947	rcu_read_lock_bh call_rcu rcu_barrier
891	948	rcu_read_unlock_bh synchronize_rcu
@@ -896,7 +953,9 @@ bh: Critical sections Grace period Barrier · 4 unmodified lines
896	953	rcu_dereference_bh_protected
897	954	rcu_read_lock_bh_held
898	955
899		−sched: Critical sections Grace period Barrier
	956	+sched::
	957	+
	958	+ Critical sections Grace period Barrier
900	959
901	960	rcu_read_lock_sched call_rcu rcu_barrier
902	961	rcu_read_unlock_sched synchronize_rcu
@@ -910,7 +969,9 @@ sched: Critical sections Grace period Barrier · 7 unmodified lines
910	969	rcu_read_lock_sched_held
911	970
912	971
913		−SRCU: Critical sections Grace period Barrier
	972	+SRCU::
	973	+
	974	+ Critical sections Grace period Barrier
914	975
915	976	srcu_read_lock call_srcu srcu_barrier
916	977	srcu_read_unlock synchronize_srcu
@@ -918,13 +979,14 @@ SRCU: Critical sections Grace period Barrier · 1 unmodified line
918	979	srcu_dereference_check
919	980	srcu_read_lock_held
920	981
921		−SRCU: Initialization/cleanup
	982	+SRCU: Initialization/cleanup::
	983	+
922	984	DEFINE_SRCU
923	985	DEFINE_STATIC_SRCU
924	986	init_srcu_struct
925	987	cleanup_srcu_struct
926	988
927		−All: lockdep-checked RCU-protected pointer access
	989	+All: lockdep-checked RCU-protected pointer access::
928	990
929	991	rcu_access_pointer
930	992	rcu_dereference_raw
@@ -974,15 +1036,19 @@ g. Otherwise, use RCU. · 43 unmodified lines
974	1036	Of course, this all assumes that you have determined that RCU is in fact
975	1037	the right tool for your job.
976	1038
	1039	+.. _8_whatisRCU:
977	1040
978	1041	8. ANSWERS TO QUICK QUIZZES
	1042	+----------------------------
979	1043
980		−Quick Quiz #1: Why is this argument naive? How could a deadlock
	1044	+Quick Quiz #1:
	1045	+ Why is this argument naive? How could a deadlock
981	1046	occur when using this algorithm in a real-world Linux
982	1047	kernel? [Referring to the lock-based "toy" RCU
983	1048	algorithm.]
984	1049
985		−Answer: Consider the following sequence of events:
	1050	+Answer:
	1051	+ Consider the following sequence of events:
986	1052
987	1053	1. CPU 0 acquires some unrelated lock, call it
988	1054	"problematic_lock", disabling irq via
@@ -1021,10 +1087,14 @@ Answer: Consider the following sequence of events: · 32 unmodified lines
1021	1087	approach where tasks in RCU read-side critical sections
1022	1088	cannot be blocked by tasks executing synchronize_rcu().
1023	1089
1024		−Quick Quiz #2: Give an example where Classic RCU's read-side
1025		− overhead is -negative-.
	1090	+:ref:`Back to Quick Quiz #1 <quiz_1>`
	1091	+
	1092	+Quick Quiz #2:
	1093	+ Give an example where Classic RCU's read-side
	1094	+ overhead is negative.
1026	1095
1027		−Answer: Imagine a single-CPU system with a non-CONFIG_PREEMPT
	1096	+Answer:
	1097	+ Imagine a single-CPU system with a non-CONFIG_PREEMPT
1028	1098	kernel where a routing table is used by process-context
1029	1099	code, but can be updated by irq-context code (for example,
1030	1100	by an "ICMP REDIRECT" packet). The usual way of handling
@@ -1046,11 +1116,15 @@ Answer: Imagine a single-CPU system with a non-CONFIG_PREEMPT · 15 unmodified lines
1046	1116	even the theoretical possibility of negative overhead for
1047	1117	a synchronization primitive is a bit unexpected. ;-)
1048	1118
1049		−Quick Quiz #3: If it is illegal to block in an RCU read-side
	1119	+:ref:`Back to Quick Quiz #2 <quiz_2>`
	1120	+
	1121	+Quick Quiz #3:
	1122	+ If it is illegal to block in an RCU read-side
1050	1123	critical section, what the heck do you do in
1051	1124	PREEMPT_RT, where normal spinlocks can block???
1052	1125
1053		−Answer: Just as PREEMPT_RT permits preemption of spinlock
	1126	+Answer:
	1127	+ Just as PREEMPT_RT permits preemption of spinlock
1054	1128	critical sections, it permits preemption of RCU
1055	1129	read-side critical sections. It also permits
1056	1130	spinlocks blocking while in RCU read-side critical
@@ -1069,6 +1143,7 @@ Answer: Just as PREEMPT_RT permits preemption of spinlock · 12 unmodified lines
1069	1143	Besides, how does the computer know what pizza parlor
1070	1144	the human being went to???
1071	1145
	1146	+:ref:`Back to Quick Quiz #3 <quiz_3>`
1072	1147
1073	1148	ACKNOWLEDGEMENTS
1074	1149

Documentation/admin-guide/kernel-parameters.txt+13−0

@@ -4001,6 +4001,19 @@ · 4000 unmodified lines
4001	4001	test until boot completes in order to avoid
4002	4002	interference.
4003	4003
	4004	+ rcuperf.kfree_rcu_test= [KNL]
	4005	+ Set to measure performance of kfree_rcu() flooding.
	4006	+
	4007	+ rcuperf.kfree_nthreads= [KNL]
	4008	+ The number of threads running loops of kfree_rcu().
	4009	+
	4010	+ rcuperf.kfree_alloc_num= [KNL]
	4011	+ Number of allocations and frees done in an iteration.
	4012	+
	4013	+ rcuperf.kfree_loops= [KNL]
	4014	+ Number of loops doing rcuperf.kfree_alloc_num number
	4015	+ of allocations and frees.
	4016	+
4004	4017	rcuperf.nreaders= [KNL]
4005	4018	Set number of RCU readers. The value -1 selects
4006	4019	N, where N is the number of CPUs. A value

arch/powerpc/include/asm/barrier.h+0−2

@@ -18,8 +18,6 @@ · 17 unmodified lines
18	18	* mb() prevents loads and stores being reordered across this point.
19	19	* rmb() prevents loads being reordered across this point.
20	20	* wmb() prevents stores being reordered across this point.
21		− * read_barrier_depends() prevents data-dependent loads being reordered
22		− * across this point (nop on PPC).
23	21	*
24	22	* *mb() variants without smp_ prefix must order all types of memory
25	23	* operations with one another. sync is the only instruction sufficient

drivers/net/wireless/mediatek/mt76/agg-rx.c+2−2

@@ -281,8 +281,8 @@ void mt76_rx_aggr_stop(struct mt76_dev dev, struct mt76_wcid wcid, u8 tidno) · 280 unmodified lines
281	281	{
282	282	struct mt76_rx_tid *tid = NULL;
283	283
284		− rcu_swap_protected(wcid->aggr[tidno], tid,
285		− lockdep_is_held(&dev->mutex));
	284	+ tid = rcu_replace_pointer(wcid->aggr[tidno], tid,
	285	+ lockdep_is_held(&dev->mutex));
286	286	if (tid) {
287	287	mt76_rx_aggr_shutdown(dev, tid);
288	288	kfree_rcu(tid, rcu_head);

include/linux/list.h+112−24

@@ -23,6 +23,13 @@ · 22 unmodified lines
23	23	#define LIST_HEAD(name) \
24	24	struct list_head name = LIST_HEAD_INIT(name)
25	25
	26	+/**
	27	+ * INIT_LIST_HEAD - Initialize a list_head structure
	28	+ * @list: list_head structure to be initialized.
	29	+ *
	30	+ * Initializes the list_head to point to itself. If it is a list header,
	31	+ * the result is an empty list.
	32	+ */
26	33	static inline void INIT_LIST_HEAD(struct list_head *list)
27	34	{
28	35	WRITE_ONCE(list->next, list);
@@ -120,12 +127,6 @@ static inline void __list_del_clearprev(struct list_head *entry) · 91 unmodified lines
120	127	entry->prev = NULL;
121	128	}
122	129
123		−/**
124		− * list_del - deletes entry from list.
125		− * @entry: the element to delete from the list.
126		− * Note: list_empty() on entry does not return true after this, the entry is
127		− * in an undefined state.
128		− */
129	130	static inline void __list_del_entry(struct list_head *entry)
130	131	{
131	132	if (!__list_del_entry_valid(entry))
@@ -134,6 +135,12 @@ static inline void __list_del_entry(struct list_head *entry) · 2 unmodified lines
134	135	__list_del(entry->prev, entry->next);
135	136	}
136	137
	138	+/**
	139	+ * list_del - deletes entry from list.
	140	+ * @entry: the element to delete from the list.
	141	+ * Note: list_empty() on entry does not return true after this, the entry is
	142	+ * in an undefined state.
	143	+ */
137	144	static inline void list_del(struct list_head *entry)
138	145	{
139	146	__list_del_entry(entry);
@@ -157,8 +164,15 @@ static inline void list_replace(struct list_head *old, · 17 unmodified lines
157	164	new->prev->next = new;
158	165	}
159	166
	167	+/**
	168	+ * list_replace_init - replace old entry by new one and initialize the old one
	169	+ * @old : the element to be replaced
	170	+ * @new : the new element to insert
	171	+ *
	172	+ * If @old was empty, it will be overwritten.
	173	+ */
160	174	static inline void list_replace_init(struct list_head *old,
161		− struct list_head *new)
	175	+ struct list_head *new)
162	176	{
163	177	list_replace(old, new);
164	178	INIT_LIST_HEAD(old);
@@ -754,11 +768,36 @@ static inline void INIT_HLIST_NODE(struct hlist_node *h) · 589 unmodified lines
754	768	h->pprev = NULL;
755	769	}
756	770
	771	+/**
	772	+ * hlist_unhashed - Has node been removed from list and reinitialized?
	773	+ * @h: Node to be checked
	774	+ *
	775	+ * Not that not all removal functions will leave a node in unhashed
	776	+ * state. For example, hlist_nulls_del_init_rcu() does leave the
	777	+ * node in unhashed state, but hlist_nulls_del() does not.
	778	+ */
757	779	static inline int hlist_unhashed(const struct hlist_node *h)
758	780	{
759	781	return !h->pprev;
760	782	}
761	783
	784	+/**
	785	+ * hlist_unhashed_lockless - Version of hlist_unhashed for lockless use
	786	+ * @h: Node to be checked
	787	+ *
	788	+ * This variant of hlist_unhashed() must be used in lockless contexts
	789	+ * to avoid potential load-tearing. The READ_ONCE() is paired with the
	790	+ * various WRITE_ONCE() in hlist helpers that are defined below.
	791	+ */
	792	+static inline int hlist_unhashed_lockless(const struct hlist_node *h)
	793	+{
	794	+ return !READ_ONCE(h->pprev);
	795	+}
	796	+
	797	+/**
	798	+ * hlist_empty - Is the specified hlist_head structure an empty hlist?
	799	+ * @h: Structure to check.
	800	+ */
762	801	static inline int hlist_empty(const struct hlist_head *h)
763	802	{
764	803	return !READ_ONCE(h->first);
@@ -771,9 +810,16 @@ static inline void __hlist_del(struct hlist_node *n) · 6 unmodified lines
771	810
772	811	WRITE_ONCE(*pprev, next);
773	812	if (next)
774		− next->pprev = pprev;
	813	+ WRITE_ONCE(next->pprev, pprev);
775	814	}
776	815
	816	+/**
	817	+ * hlist_del - Delete the specified hlist_node from its list
	818	+ * @n: Node to delete.
	819	+ *
	820	+ * Note that this function leaves the node in hashed state. Use
	821	+ * hlist_del_init() or similar instead to unhash @n.
	822	+ */
777	823	static inline void hlist_del(struct hlist_node *n)
778	824	{
779	825	__hlist_del(n);
@@ -781,6 +827,12 @@ static inline void hlist_del(struct hlist_node *n) · 1 unmodified line
781	827	n->pprev = LIST_POISON2;
782	828	}
783	829
	830	+/**
	831	+ * hlist_del_init - Delete the specified hlist_node from its list and initialize
	832	+ * @n: Node to delete.
	833	+ *
	834	+ * Note that this function leaves the node in unhashed state.
	835	+ */
784	836	static inline void hlist_del_init(struct hlist_node *n)
785	837	{
786	838	if (!hlist_unhashed(n)) {
@@ -789,51 +841,83 @@ static inline void hlist_del_init(struct hlist_node *n) · 2 unmodified lines
789	841	}
790	842	}
791	843
	844	+/**
	845	+ * hlist_add_head - add a new entry at the beginning of the hlist
	846	+ * @n: new entry to be added
	847	+ * @h: hlist head to add it after
	848	+ *
	849	+ * Insert a new entry after the specified head.
	850	+ * This is good for implementing stacks.
	851	+ */
792	852	static inline void hlist_add_head(struct hlist_node n, struct hlist_head h)
793	853	{
794	854	struct hlist_node *first = h->first;
795		− n->next = first;
	855	+ WRITE_ONCE(n->next, first);
796	856	if (first)
797		− first->pprev = &n->next;
	857	+ WRITE_ONCE(first->pprev, &n->next);
798	858	WRITE_ONCE(h->first, n);
799		− n->pprev = &h->first;
	859	+ WRITE_ONCE(n->pprev, &h->first);
800	860	}
801	861
802		−/* next must be != NULL */
	862	+/**
	863	+ * hlist_add_before - add a new entry before the one specified
	864	+ * @n: new entry to be added
	865	+ * @next: hlist node to add it before, which must be non-NULL
	866	+ */
803	867	static inline void hlist_add_before(struct hlist_node *n,
804		− struct hlist_node *next)
	868	+ struct hlist_node *next)
805	869	{
806		− n->pprev = next->pprev;
807		− n->next = next;
808		− next->pprev = &n->next;
	870	+ WRITE_ONCE(n->pprev, next->pprev);
	871	+ WRITE_ONCE(n->next, next);
	872	+ WRITE_ONCE(next->pprev, &n->next);
809	873	WRITE_ONCE(*(n->pprev), n);
810	874	}
811	875
	876	+/**
	877	+ * hlist_add_behing - add a new entry after the one specified
	878	+ * @n: new entry to be added
	879	+ * @prev: hlist node to add it after, which must be non-NULL
	880	+ */
812	881	static inline void hlist_add_behind(struct hlist_node *n,
813	882	struct hlist_node *prev)
814	883	{
815		− n->next = prev->next;
816		− prev->next = n;
817		− n->pprev = &prev->next;
	884	+ WRITE_ONCE(n->next, prev->next);
	885	+ WRITE_ONCE(prev->next, n);
	886	+ WRITE_ONCE(n->pprev, &prev->next);
818	887
819	888	if (n->next)
820		− n->next->pprev = &n->next;
	889	+ WRITE_ONCE(n->next->pprev, &n->next);
821	890	}
822	891
823		−/* after that we'll appear to be on some hlist and hlist_del will work */
	892	+/**
	893	+ * hlist_add_fake - create a fake hlist consisting of a single headless node
	894	+ * @n: Node to make a fake list out of
	895	+ *
	896	+ * This makes @n appear to be its own predecessor on a headless hlist.
	897	+ * The point of this is to allow things like hlist_del() to work correctly
	898	+ * in cases where there is no list.
	899	+ */
824	900	static inline void hlist_add_fake(struct hlist_node *n)
825	901	{
826	902	n->pprev = &n->next;
827	903	}
828	904
	905	+/**
	906	+ * hlist_fake: Is this node a fake hlist?
	907	+ * @h: Node to check for being a self-referential fake hlist.
	908	+ */
829	909	static inline bool hlist_fake(struct hlist_node *h)
830	910	{
831	911	return h->pprev == &h->next;
832	912	}
833	913
834		−/*
	914	+/**
	915	+ * hlist_is_singular_node - is node the only element of the specified hlist?
	916	+ * @n: Node to check for singularity.
	917	+ * @h: Header for potentially singular list.
	918	+ *
835	919	* Check whether the node is the only node of the head without
836		− * accessing head:
	920	+ * accessing head, thus avoiding unnecessary cache misses.
837	921	*/
838	922	static inline bool
839	923	hlist_is_singular_node(struct hlist_node n, struct hlist_head h)
@@ -841,7 +925,11 @@ hlist_is_singular_node(struct hlist_node n, struct hlist_head h) · 1 unmodified line
841	925	return !n->next && n->pprev == &h->first;
842	926	}
843	927
844		−/*
	928	+/**
	929	+ * hlist_move_list - Move an hlist
	930	+ * @old: hlist_head for old list.
	931	+ * @new: hlist_head for new list.
	932	+ *
845	933	* Move a list from one list head to another. Fixup the pprev
846	934	* reference of the first entry if it exists.
847	935	*/

include/linux/list_nulls.h+26−4

@@ -56,11 +56,33 @@ static inline unsigned long get_nulls_value(const struct hlist_nulls_node *ptr) · 55 unmodified lines
56	56	return ((unsigned long)ptr) >> 1;
57	57	}
58	58
	59	+/**
	60	+ * hlist_nulls_unhashed - Has node been removed and reinitialized?
	61	+ * @h: Node to be checked
	62	+ *
	63	+ * Not that not all removal functions will leave a node in unhashed state.
	64	+ * For example, hlist_del_init_rcu() leaves the node in unhashed state,
	65	+ * but hlist_nulls_del() does not.
	66	+ */
59	67	static inline int hlist_nulls_unhashed(const struct hlist_nulls_node *h)
60	68	{
61	69	return !h->pprev;
62	70	}
63	71
	72	+/**
	73	+ * hlist_nulls_unhashed_lockless - Has node been removed and reinitialized?
	74	+ * @h: Node to be checked
	75	+ *
	76	+ * Not that not all removal functions will leave a node in unhashed state.
	77	+ * For example, hlist_del_init_rcu() leaves the node in unhashed state,
	78	+ * but hlist_nulls_del() does not. Unlike hlist_nulls_unhashed(), this
	79	+ * function may be used locklessly.
	80	+ */
	81	+static inline int hlist_nulls_unhashed_lockless(const struct hlist_nulls_node *h)
	82	+{
	83	+ return !READ_ONCE(h->pprev);
	84	+}
	85	+
64	86	static inline int hlist_nulls_empty(const struct hlist_nulls_head *h)
65	87	{
66	88	return is_a_nulls(READ_ONCE(h->first));
@@ -72,10 +94,10 @@ static inline void hlist_nulls_add_head(struct hlist_nulls_node *n, · 5 unmodified lines
72	94	struct hlist_nulls_node *first = h->first;
73	95
74	96	n->next = first;
75		− n->pprev = &h->first;
	97	+ WRITE_ONCE(n->pprev, &h->first);
76	98	h->first = n;
77	99	if (!is_a_nulls(first))
78		− first->pprev = &n->next;
	100	+ WRITE_ONCE(first->pprev, &n->next);
79	101	}
80	102
81	103	static inline void __hlist_nulls_del(struct hlist_nulls_node *n)
@@ -85,13 +107,13 @@ static inline void __hlist_nulls_del(struct hlist_nulls_node *n) · 3 unmodified lines
85	107
86	108	WRITE_ONCE(*pprev, next);
87	109	if (!is_a_nulls(next))
88		− next->pprev = pprev;
	110	+ WRITE_ONCE(next->pprev, pprev);
89	111	}
90	112
91	113	static inline void hlist_nulls_del(struct hlist_nulls_node *n)
92	114	{
93	115	__hlist_nulls_del(n);
94		− n->pprev = LIST_POISON2;
	116	+ WRITE_ONCE(n->pprev, LIST_POISON2);
95	117	}
96	118
97	119	/**

include/linux/rcu_segcblist.h+0−2

@@ -22,7 +22,6 @@ struct rcu_cblist { · 21 unmodified lines
22	22	struct rcu_head *head;
23	23	struct rcu_head **tail;
24	24	long len;
25		− long len_lazy;
26	25	};
27	26
28	27	#define RCU_CBLIST_INITIALIZER(n) { .head = NULL, .tail = &n.head }
@@ -73,7 +72,6 @@ struct rcu_segcblist { · 44 unmodified lines
73	72	#else
74	73	long len;
75	74	#endif
76		− long len_lazy;
77	75	u8 enabled;
78	76	u8 offloaded;
79	77	};

include/linux/rculist.h+24−14

@@ -40,6 +40,16 @@ static inline void INIT_LIST_HEAD_RCU(struct list_head *list) · 39 unmodified lines
40	40	*/
41	41	#define list_next_rcu(list) (((struct list_head __rcu *)(&(list)->next)))
42	42
	43	+/**
	44	+ * list_tail_rcu - returns the prev pointer of the head of the list
	45	+ * @head: the head of the list
	46	+ *
	47	+ * Note: This should only be used with the list header, and even then
	48	+ * only if list_del() and similar primitives are not also used on the
	49	+ * list header.
	50	+ */
	51	+#define list_tail_rcu(head) (((struct list_head __rcu *)(&(head)->prev)))
	52	+
43	53	/*
44	54	* Check during list traversal that we are within an RCU reader
45	55	*/
@@ -173,7 +183,7 @@ static inline void hlist_del_init_rcu(struct hlist_node *n) · 127 unmodified lines
173	183	{
174	184	if (!hlist_unhashed(n)) {
175	185	__hlist_del(n);
176		− n->pprev = NULL;
	186	+ WRITE_ONCE(n->pprev, NULL);
177	187	}
178	188	}
179	189
@@ -361,7 +371,7 @@ static inline void list_splice_tail_init_rcu(struct list_head *list, · 181 unmodified lines
361	371	* @pos: the type * to use as a loop cursor.
362	372	* @head: the head for your list.
363	373	* @member: the name of the list_head within the struct.
364		− * @cond: optional lockdep expression if called from non-RCU protection.
	374	+ * @cond...: optional lockdep expression if called from non-RCU protection.
365	375	*
366	376	* This list-traversal primitive may safely run concurrently with
367	377	* the _rcu list-mutation primitives such as list_add_rcu()
@@ -473,7 +483,7 @@ static inline void list_splice_tail_init_rcu(struct list_head *list, · 105 unmodified lines
473	483	static inline void hlist_del_rcu(struct hlist_node *n)
474	484	{
475	485	__hlist_del(n);
476		− n->pprev = LIST_POISON2;
	486	+ WRITE_ONCE(n->pprev, LIST_POISON2);
477	487	}
478	488
479	489	/**
@@ -489,11 +499,11 @@ static inline void hlist_replace_rcu(struct hlist_node *old, · 9 unmodified lines
489	499	struct hlist_node *next = old->next;
490	500
491	501	new->next = next;
492		− new->pprev = old->pprev;
	502	+ WRITE_ONCE(new->pprev, old->pprev);
493	503	rcu_assign_pointer((struct hlist_node __rcu *)new->pprev, new);
494	504	if (next)
495		− new->next->pprev = &new->next;
496		− old->pprev = LIST_POISON2;
	505	+ WRITE_ONCE(new->next->pprev, &new->next);
	506	+ WRITE_ONCE(old->pprev, LIST_POISON2);
497	507	}
498	508
499	509	/*
@@ -528,10 +538,10 @@ static inline void hlist_add_head_rcu(struct hlist_node *n, · 28 unmodified lines
528	538	struct hlist_node *first = h->first;
529	539
530	540	n->next = first;
531		− n->pprev = &h->first;
	541	+ WRITE_ONCE(n->pprev, &h->first);
532	542	rcu_assign_pointer(hlist_first_rcu(h), n);
533	543	if (first)
534		− first->pprev = &n->next;
	544	+ WRITE_ONCE(first->pprev, &n->next);
535	545	}
536	546
537	547	/**
@@ -564,7 +574,7 @@ static inline void hlist_add_tail_rcu(struct hlist_node *n, · 26 unmodified lines
564	574
565	575	if (last) {
566	576	n->next = last->next;
567		− n->pprev = &last->next;
	577	+ WRITE_ONCE(n->pprev, &last->next);
568	578	rcu_assign_pointer(hlist_next_rcu(last), n);
569	579	} else {
570	580	hlist_add_head_rcu(n, h);
@@ -592,10 +602,10 @@ static inline void hlist_add_tail_rcu(struct hlist_node *n, · 21 unmodified lines
592	602	static inline void hlist_add_before_rcu(struct hlist_node *n,
593	603	struct hlist_node *next)
594	604	{
595		− n->pprev = next->pprev;
	605	+ WRITE_ONCE(n->pprev, next->pprev);
596	606	n->next = next;
597	607	rcu_assign_pointer(hlist_pprev_rcu(n), n);
598		− next->pprev = &n->next;
	608	+ WRITE_ONCE(next->pprev, &n->next);
599	609	}
600	610
601	611	/**
@@ -620,10 +630,10 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n, · 18 unmodified lines
620	630	struct hlist_node *prev)
621	631	{
622	632	n->next = prev->next;
623		− n->pprev = &prev->next;
	633	+ WRITE_ONCE(n->pprev, &prev->next);
624	634	rcu_assign_pointer(hlist_next_rcu(prev), n);
625	635	if (n->next)
626		− n->next->pprev = &n->next;
	636	+ WRITE_ONCE(n->next->pprev, &n->next);
627	637	}
628	638
629	639	#define __hlist_for_each_rcu(pos, head) \
@@ -636,7 +646,7 @@ static inline void hlist_add_behind_rcu(struct hlist_node *n, · 6 unmodified lines
636	646	* @pos: the type * to use as a loop cursor.
637	647	* @head: the head for your list.
638	648	* @member: the name of the hlist_node within the struct.
639		− * @cond: optional lockdep expression if called from non-RCU protection.
	649	+ * @cond...: optional lockdep expression if called from non-RCU protection.
640	650	*
641	651	* This list-traversal primitive may safely run concurrently with
642	652	* the _rcu list-mutation primitives such as hlist_add_head_rcu()

include/linux/rculist_nulls.h+14−6

@@ -34,13 +34,21 @@ static inline void hlist_nulls_del_init_rcu(struct hlist_nulls_node *n) · 33 unmodified lines
34	34	{
35	35	if (!hlist_nulls_unhashed(n)) {
36	36	__hlist_nulls_del(n);
37		− n->pprev = NULL;
	37	+ WRITE_ONCE(n->pprev, NULL);
38	38	}
39	39	}
40	40
	41	+/**
	42	+ * hlist_nulls_first_rcu - returns the first element of the hash list.
	43	+ * @head: the head of the list.
	44	+ */
41	45	#define hlist_nulls_first_rcu(head) \
42	46	(((struct hlist_nulls_node __rcu __force *)&(head)->first))
43	47
	48	+/**
	49	+ * hlist_nulls_next_rcu - returns the element of the list after @node.
	50	+ * @node: element of the list.
	51	+ */
44	52	#define hlist_nulls_next_rcu(node) \
45	53	(((struct hlist_nulls_node __rcu __force *)&(node)->next))
46	54
@@ -66,7 +74,7 @@ static inline void hlist_nulls_del_init_rcu(struct hlist_nulls_node *n) · 19 unmodified lines
66	74	static inline void hlist_nulls_del_rcu(struct hlist_nulls_node *n)
67	75	{
68	76	__hlist_nulls_del(n);
69		− n->pprev = LIST_POISON2;
	77	+ WRITE_ONCE(n->pprev, LIST_POISON2);
70	78	}
71	79
72	80	/**
@@ -94,10 +102,10 @@ static inline void hlist_nulls_add_head_rcu(struct hlist_nulls_node *n, · 21 unmodified lines
94	102	struct hlist_nulls_node *first = h->first;
95	103
96	104	n->next = first;
97		− n->pprev = &h->first;
	105	+ WRITE_ONCE(n->pprev, &h->first);
98	106	rcu_assign_pointer(hlist_nulls_first_rcu(h), n);
99	107	if (!is_a_nulls(first))
100		− first->pprev = &n->next;
	108	+ WRITE_ONCE(first->pprev, &n->next);
101	109	}
102	110
103	111	/**
@@ -141,7 +149,7 @@ static inline void hlist_nulls_add_tail_rcu(struct hlist_nulls_node *n, · 37 unmodified lines
141	149	* hlist_nulls_for_each_entry_rcu - iterate over rcu list of given type
142	150	* @tpos: the type * to use as a loop cursor.
143	151	* @pos: the &struct hlist_nulls_node to use as a loop cursor.
144		− * @head: the head for your list.
	152	+ * @head: the head of the list.
145	153	* @member: the name of the hlist_nulls_node within the struct.
146	154	*
147	155	* The barrier() is needed to make sure compiler doesn't cache first element [1],
@@ -161,7 +169,7 @@ static inline void hlist_nulls_add_tail_rcu(struct hlist_nulls_node *n, · 13 unmodified lines
161	169	* iterate over list of given type safe against removal of list entry
162	170	* @tpos: the type * to use as a loop cursor.
163	171	* @pos: the &struct hlist_nulls_node to use as a loop cursor.
164		− * @head: the head for your list.
	172	+ * @head: the head of the list.
165	173	* @member: the name of the hlist_nulls_node within the struct.
166	174	*/
167	175	#define hlist_nulls_for_each_entry_safe(tpos, pos, head, member) \

include/linux/rcupdate.h+8−20

@@ -154,7 +154,7 @@ static inline void exit_tasks_rcu_finish(void) { } · 153 unmodified lines
154	154	*
155	155	* This macro resembles cond_resched(), except that it is defined to
156	156	* report potential quiescent states to RCU-tasks even if the cond_resched()
157		− * machinery were to be shut off, as some advocate for PREEMPT kernels.
	157	+ * machinery were to be shut off, as some advocate for PREEMPTION kernels.
158	158	*/
159	159	#define cond_resched_tasks_rcu_qs() \
160	160	do { \
@@ -167,7 +167,7 @@ do { \ · 6 unmodified lines
167	167	* TREE_RCU and rcu_barrier_() primitives in TINY_RCU.
168	168	*/
169	169
170		−#if defined(CONFIG_TREE_RCU) \|\| defined(CONFIG_PREEMPT_RCU)
	170	+#if defined(CONFIG_TREE_RCU)
171	171	#include <linux/rcutree.h>
172	172	#elif defined(CONFIG_TINY_RCU)
173	173	#include <linux/rcutiny.h>
@@ -400,22 +400,6 @@ do { \ · 226 unmodified lines
400	400	__tmp; \
401	401	})
402	402
403		−/**
404		− * rcu_swap_protected() - swap an RCU and a regular pointer
405		− * @rcu_ptr: RCU pointer
406		− * @ptr: regular pointer
407		− * @c: the conditions under which the dereference will take place
408		− *
409		− * Perform swap(@rcu_ptr, @ptr) where @rcu_ptr is an RCU-annotated pointer and
410		− * @c is the argument that is passed to the rcu_dereference_protected() call
411		− * used to read that pointer.
412		− */
413		−#define rcu_swap_protected(rcu_ptr, ptr, c) do { \
414		− typeof(ptr) __tmp = rcu_dereference_protected((rcu_ptr), (c)); \
415		− rcu_assign_pointer((rcu_ptr), (ptr)); \
416		− (ptr) = __tmp; \
417		−} while (0)
418		−
419	403	/**
420	404	* rcu_access_pointer() - fetch RCU pointer with no dereferencing
421	405	* @p: The pointer to read
@@ -598,10 +582,10 @@ do { \ · 176 unmodified lines
598	582	*
599	583	* You can avoid reading and understanding the next paragraph by
600	584	* following this rule: don't put anything in an rcu_read_lock() RCU
601		− * read-side critical section that would block in a !PREEMPT kernel.
	585	+ * read-side critical section that would block in a !PREEMPTION kernel.
602	586	* But if you want the full story, read on!
603	587	*
604		− * In non-preemptible RCU implementations (TREE_RCU and TINY_RCU),
	588	+ * In non-preemptible RCU implementations (pure TREE_RCU and TINY_RCU),
605	589	* it is illegal to block while in an RCU read-side critical section.
606	590	* In preemptible RCU implementations (PREEMPT_RCU) in CONFIG_PREEMPTION
607	591	* kernel builds, RCU read-side critical sections may be preempted,
@@ -912,4 +896,8 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f) · 304 unmodified lines
912	896	return false;
913	897	}
914	898
	899	+/* kernel/ksysfs.c definitions */
	900	+extern int rcu_expedited;
	901	+extern int rcu_normal;
	902	+
915	903	#endif /* __LINUX_RCUPDATE_H */

include/linux/rcutiny.h+1−0

@@ -85,6 +85,7 @@ static inline void rcu_scheduler_starting(void) { } · 84 unmodified lines
85	85	static inline void rcu_end_inkernel_boot(void) { }
86	86	static inline bool rcu_is_watching(void) { return true; }
87	87	static inline void rcu_momentary_dyntick_idle(void) { }
	88	+static inline void kfree_rcu_scheduler_running(void) { }
88	89
89	90	/* Avoid RCU read-side critical sections leaking across. */
90	91	static inline void rcu_all_qs(void) { barrier(); }

include/linux/rcutree.h+1−0

@@ -38,6 +38,7 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func); · 37 unmodified lines
38	38	void rcu_barrier(void);
39	39	bool rcu_eqs_special_set(int cpu);
40	40	void rcu_momentary_dyntick_idle(void);
	41	+void kfree_rcu_scheduler_running(void);
41	42	unsigned long get_state_synchronize_rcu(void);
42	43	void cond_synchronize_rcu(unsigned long oldstate);
43	44

include/linux/tick.h+4−1

@@ -109,8 +109,10 @@ enum tick_dep_bits { · 108 unmodified lines
109	109	TICK_DEP_BIT_PERF_EVENTS = 1,
110	110	TICK_DEP_BIT_SCHED = 2,
111	111	TICK_DEP_BIT_CLOCK_UNSTABLE = 3,
112		− TICK_DEP_BIT_RCU = 4
	112	+ TICK_DEP_BIT_RCU = 4,
	113	+ TICK_DEP_BIT_RCU_EXP = 5
113	114	};
	115	+#define TICK_DEP_BIT_MAX TICK_DEP_BIT_RCU_EXP
114	116
115	117	#define TICK_DEP_MASK_NONE 0
116	118	#define TICK_DEP_MASK_POSIX_TIMER (1 << TICK_DEP_BIT_POSIX_TIMER)
@@ -118,6 +120,7 @@ enum tick_dep_bits { · 1 unmodified line
118	120	#define TICK_DEP_MASK_SCHED (1 << TICK_DEP_BIT_SCHED)
119	121	#define TICK_DEP_MASK_CLOCK_UNSTABLE (1 << TICK_DEP_BIT_CLOCK_UNSTABLE)
120	122	#define TICK_DEP_MASK_RCU (1 << TICK_DEP_BIT_RCU)
	123	+#define TICK_DEP_MASK_RCU_EXP (1 << TICK_DEP_BIT_RCU_EXP)
121	124
122	125	#ifdef CONFIG_NO_HZ_COMMON
123	126	extern bool tick_nohz_enabled;

include/trace/events/rcu.h+16−24

@@ -41,7 +41,7 @@ TRACE_EVENT(rcu_utilization, · 40 unmodified lines
41	41	TP_printk("%s", __entry->s)
42	42	);
43	43
44		−#if defined(CONFIG_TREE_RCU) \|\| defined(CONFIG_PREEMPT_RCU)
	44	+#if defined(CONFIG_TREE_RCU)
45	45
46	46	/*
47	47	* Tracepoint for grace-period events. Takes a string identifying the
@@ -432,7 +432,7 @@ TRACE_EVENT_RCU(rcu_fqs, · 384 unmodified lines
432	432	__entry->cpu, __entry->qsevent)
433	433	);
434	434
435		−#endif /* #if defined(CONFIG_TREE_RCU) \|\| defined(CONFIG_PREEMPT_RCU) */
	435	+#endif /* #if defined(CONFIG_TREE_RCU) */
436	436
437	437	/*
438	438	* Tracepoint for dyntick-idle entry/exit events. These take a string
@@ -449,7 +449,7 @@ TRACE_EVENT_RCU(rcu_fqs, · 10 unmodified lines
449	449	*/
450	450	TRACE_EVENT_RCU(rcu_dyntick,
451	451
452		− TP_PROTO(const char *polarity, long oldnesting, long newnesting, atomic_t dynticks),
	452	+ TP_PROTO(const char *polarity, long oldnesting, long newnesting, int dynticks),
453	453
454	454	TP_ARGS(polarity, oldnesting, newnesting, dynticks),
455	455
@@ -464,7 +464,7 @@ TRACE_EVENT_RCU(rcu_dyntick, · 8 unmodified lines
464	464	__entry->polarity = polarity;
465	465	__entry->oldnesting = oldnesting;
466	466	__entry->newnesting = newnesting;
467		− __entry->dynticks = atomic_read(&dynticks);
	467	+ __entry->dynticks = dynticks;
468	468	),
469	469
470	470	TP_printk("%s %lx %lx %#3x", __entry->polarity,
@@ -481,16 +481,14 @@ TRACE_EVENT_RCU(rcu_dyntick, · 10 unmodified lines
481	481	*/
482	482	TRACE_EVENT_RCU(rcu_callback,
483	483
484		− TP_PROTO(const char rcuname, struct rcu_head rhp, long qlen_lazy,
485		− long qlen),
	484	+ TP_PROTO(const char rcuname, struct rcu_head rhp, long qlen),
486	485
487		− TP_ARGS(rcuname, rhp, qlen_lazy, qlen),
	486	+ TP_ARGS(rcuname, rhp, qlen),
488	487
489	488	TP_STRUCT__entry(
490	489	__field(const char *, rcuname)
491	490	__field(void *, rhp)
492	491	__field(void *, func)
493		− __field(long, qlen_lazy)
494	492	__field(long, qlen)
495	493	),
496	494
@@ -498,13 +496,12 @@ TRACE_EVENT_RCU(rcu_callback, · 1 unmodified line
498	496	__entry->rcuname = rcuname;
499	497	__entry->rhp = rhp;
500	498	__entry->func = rhp->func;
501		− __entry->qlen_lazy = qlen_lazy;
502	499	__entry->qlen = qlen;
503	500	),
504	501
505		− TP_printk("%s rhp=%p func=%ps %ld/%ld",
	502	+ TP_printk("%s rhp=%p func=%ps %ld",
506	503	__entry->rcuname, __entry->rhp, __entry->func,
507		− __entry->qlen_lazy, __entry->qlen)
	504	+ __entry->qlen)
508	505	);
509	506
510	507	/*
@@ -518,15 +515,14 @@ TRACE_EVENT_RCU(rcu_callback, · 7 unmodified lines
518	515	TRACE_EVENT_RCU(rcu_kfree_callback,
519	516
520	517	TP_PROTO(const char rcuname, struct rcu_head rhp, unsigned long offset,
521		− long qlen_lazy, long qlen),
	518	+ long qlen),
522	519
523		− TP_ARGS(rcuname, rhp, offset, qlen_lazy, qlen),
	520	+ TP_ARGS(rcuname, rhp, offset, qlen),
524	521
525	522	TP_STRUCT__entry(
526	523	__field(const char *, rcuname)
527	524	__field(void *, rhp)
528	525	__field(unsigned long, offset)
529		− __field(long, qlen_lazy)
530	526	__field(long, qlen)
531	527	),
532	528
@@ -534,13 +530,12 @@ TRACE_EVENT_RCU(rcu_kfree_callback, · 1 unmodified line
534	530	__entry->rcuname = rcuname;
535	531	__entry->rhp = rhp;
536	532	__entry->offset = offset;
537		− __entry->qlen_lazy = qlen_lazy;
538	533	__entry->qlen = qlen;
539	534	),
540	535
541		− TP_printk("%s rhp=%p func=%ld %ld/%ld",
	536	+ TP_printk("%s rhp=%p func=%ld %ld",
542	537	__entry->rcuname, __entry->rhp, __entry->offset,
543		− __entry->qlen_lazy, __entry->qlen)
	538	+ __entry->qlen)
544	539	);
545	540
546	541	/*
@@ -552,27 +547,24 @@ TRACE_EVENT_RCU(rcu_kfree_callback, · 5 unmodified lines
552	547	*/
553	548	TRACE_EVENT_RCU(rcu_batch_start,
554	549
555		− TP_PROTO(const char *rcuname, long qlen_lazy, long qlen, long blimit),
	550	+ TP_PROTO(const char *rcuname, long qlen, long blimit),
556	551
557		− TP_ARGS(rcuname, qlen_lazy, qlen, blimit),
	552	+ TP_ARGS(rcuname, qlen, blimit),
558	553
559	554	TP_STRUCT__entry(
560	555	__field(const char *, rcuname)
561		− __field(long, qlen_lazy)
562	556	__field(long, qlen)
563	557	__field(long, blimit)
564	558	),
565	559
566	560	TP_fast_assign(
567	561	__entry->rcuname = rcuname;
568		− __entry->qlen_lazy = qlen_lazy;
569	562	__entry->qlen = qlen;
570	563	__entry->blimit = blimit;
571	564	),
572	565
573		− TP_printk("%s CBs=%ld/%ld bl=%ld",
574		− __entry->rcuname, __entry->qlen_lazy, __entry->qlen,
575		− __entry->blimit)
	566	+ TP_printk("%s CBs=%ld bl=%ld",
	567	+ __entry->rcuname, __entry->qlen, __entry->blimit)
576	568	);
577	569
578	570	/*

kernel/rcu/Kconfig+9−8

@@ -7,7 +7,7 @@ menu "RCU Subsystem" · 6 unmodified lines
7	7
8	8	config TREE_RCU
9	9	bool
10		− default y if !PREEMPTION && SMP
	10	+ default y if SMP
11	11	help
12	12	This option selects the RCU implementation that is
13	13	designed for very large SMP system with hundreds or
@@ -17,6 +17,7 @@ config TREE_RCU · 3 unmodified lines
17	17	config PREEMPT_RCU
18	18	bool
19	19	default y if PREEMPTION
	20	+ select TREE_RCU
20	21	help
21	22	This option selects the RCU implementation that is
22	23	designed for very large SMP systems with hundreds or
@@ -78,7 +79,7 @@ config TASKS_RCU · 55 unmodified lines
78	79	user-mode execution as quiescent states.
79	80
80	81	config RCU_STALL_COMMON
81		− def_bool ( TREE_RCU \|\| PREEMPT_RCU )
	82	+ def_bool TREE_RCU
82	83	help
83	84	This option enables RCU CPU stall code that is common between
84	85	the TINY and TREE variants of RCU. The purpose is to allow
@@ -86,13 +87,13 @@ config RCU_STALL_COMMON · 1 unmodified line
86	87	making these warnings mandatory for the tree variants.
87	88
88	89	config RCU_NEED_SEGCBLIST
89		− def_bool ( TREE_RCU \|\| PREEMPT_RCU \|\| TREE_SRCU )
	90	+ def_bool ( TREE_RCU \|\| TREE_SRCU )
90	91
91	92	config RCU_FANOUT
92	93	int "Tree-based hierarchical RCU fanout value"
93	94	range 2 64 if 64BIT
94	95	range 2 32 if !64BIT
95		− depends on (TREE_RCU \|\| PREEMPT_RCU) && RCU_EXPERT
	96	+ depends on TREE_RCU && RCU_EXPERT
96	97	default 64 if 64BIT
97	98	default 32 if !64BIT
98	99	help
@@ -112,7 +113,7 @@ config RCU_FANOUT_LEAF · 13 unmodified lines
112	113	int "Tree-based hierarchical RCU leaf-level fanout value"
113	114	range 2 64 if 64BIT
114	115	range 2 32 if !64BIT
115		− depends on (TREE_RCU \|\| PREEMPT_RCU) && RCU_EXPERT
	116	+ depends on TREE_RCU && RCU_EXPERT
116	117	default 16
117	118	help
118	119	This option controls the leaf-level fanout of hierarchical
@@ -187,7 +188,7 @@ config RCU_BOOST_DELAY · 68 unmodified lines
187	188
188	189	config RCU_NOCB_CPU
189	190	bool "Offload RCU callback processing from boot-selected CPUs"
190		− depends on TREE_RCU \|\| PREEMPT_RCU
	191	+ depends on TREE_RCU
191	192	depends on RCU_EXPERT \|\| NO_HZ_FULL
192	193	default n
193	194	help
@@ -200,8 +201,8 @@ config RCU_NOCB_CPU · 6 unmodified lines
200	201	specified at boot time by the rcu_nocbs parameter. For each
201	202	such CPU, a kthread ("rcuox/N") will be created to invoke
202	203	callbacks, where the "N" is the CPU being offloaded, and where
203		− the "p" for RCU-preempt (PREEMPT kernels) and "s" for RCU-sched
204		− (!PREEMPT kernels). Nothing prevents this kthread from running
	204	+ the "p" for RCU-preempt (PREEMPTION kernels) and "s" for RCU-sched
	205	+ (!PREEMPTION kernels). Nothing prevents this kthread from running
205	206	on the specified CPUs, but (1) the kthreads may be preempted
206	207	between each callback, and (2) affinity or cgroups can be used
207	208	to force the kthreads to run on whatever set of CPUs is desired.

kernel/rcu/Makefile+0−1

@@ -9,6 +9,5 @@ obj-$(CONFIG_TINY_SRCU) += srcutiny.o · 8 unmodified lines
9	9	obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o
10	10	obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o
11	11	obj-$(CONFIG_TREE_RCU) += tree.o
12		−obj-$(CONFIG_PREEMPT_RCU) += tree.o
13	12	obj-$(CONFIG_TINY_RCU) += tiny.o
14	13	obj-$(CONFIG_RCU_NEED_SEGCBLIST) += rcu_segcblist.o

kernel/rcu/rcu.h+3−30

@@ -198,33 +198,6 @@ static inline void debug_rcu_head_unqueue(struct rcu_head *head) · 197 unmodified lines
198	198	}
199	199	#endif /* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
200	200
201		−void kfree(const void *);
202		−
203		−/*
204		− * Reclaim the specified callback, either by invoking it (non-lazy case)
205		− * or freeing it directly (lazy case). Return true if lazy, false otherwise.
206		− */
207		−static inline bool __rcu_reclaim(const char rn, struct rcu_head head)
208		−{
209		− rcu_callback_t f;
210		− unsigned long offset = (unsigned long)head->func;
211		−
212		− rcu_lock_acquire(&rcu_callback_map);
213		− if (__is_kfree_rcu_offset(offset)) {
214		− trace_rcu_invoke_kfree_callback(rn, head, offset);
215		− kfree((void *)head - offset);
216		− rcu_lock_release(&rcu_callback_map);
217		− return true;
218		− } else {
219		− trace_rcu_invoke_callback(rn, head);
220		− f = head->func;
221		− WRITE_ONCE(head->func, (rcu_callback_t)0L);
222		− f(head);
223		− rcu_lock_release(&rcu_callback_map);
224		− return false;
225		− }
226		−}
227		−
228	201	#ifdef CONFIG_RCU_STALL_COMMON
229	202
230	203	extern int rcu_cpu_stall_ftrace_dump;
@@ -281,7 +254,7 @@ void rcu_test_sync_prims(void); · 50 unmodified lines
281	254	*/
282	255	extern void resched_cpu(int cpu);
283	256
284		−#if defined(SRCU) \|\| !defined(TINY_RCU)
	257	+#if defined(CONFIG_SRCU) \|\| !defined(CONFIG_TINY_RCU)
285	258
286	259	#include <linux/rcu_node_tree.h>
287	260
@@ -418,7 +391,7 @@ do { \ · 130 unmodified lines
418	391	#define raw_lockdep_assert_held_rcu_node(p) \
419	392	lockdep_assert_held(&ACCESS_PRIVATE(p, lock))
420	393
421		−#endif /* #if defined(SRCU) \|\| !defined(TINY_RCU) */
	394	+#endif /* #if defined(CONFIG_SRCU) \|\| !defined(CONFIG_TINY_RCU) */
422	395
423	396	#ifdef CONFIG_SRCU
424	397	void srcu_init(void);
@@ -454,7 +427,7 @@ enum rcutorture_type { · 29 unmodified lines
454	427	INVALID_RCU_FLAVOR
455	428	};
456	429
457		−#if defined(CONFIG_TREE_RCU) \|\| defined(CONFIG_PREEMPT_RCU)
	430	+#if defined(CONFIG_TREE_RCU)
458	431	void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags,
459	432	unsigned long *gp_seq);
460	433	void do_trace_rcu_torture_read(const char *rcutorturename,

kernel/rcu/rcu_segcblist.c+3−22

@@ -20,14 +20,10 @@ void rcu_cblist_init(struct rcu_cblist *rclp) · 19 unmodified lines
20	20	rclp->head = NULL;
21	21	rclp->tail = &rclp->head;
22	22	rclp->len = 0;
23		− rclp->len_lazy = 0;
24	23	}
25	24
26	25	/*
27	26	* Enqueue an rcu_head structure onto the specified callback list.
28		− * This function assumes that the callback is non-lazy because it
29		− * is intended for use by no-CBs CPUs, which do not distinguish
30		− * between lazy and non-lazy RCU callbacks.
31	27	*/
32	28	void rcu_cblist_enqueue(struct rcu_cblist rclp, struct rcu_head rhp)
33	29	{
@@ -54,7 +50,6 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, · 20 unmodified lines
54	50	else
55	51	drclp->tail = &drclp->head;
56	52	drclp->len = srclp->len;
57		− drclp->len_lazy = srclp->len_lazy;
58	53	if (!rhp) {
59	54	rcu_cblist_init(srclp);
60	55	} else {
@@ -62,16 +57,12 @@ void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp, · 1 unmodified line
62	57	srclp->head = rhp;
63	58	srclp->tail = &rhp->next;
64	59	WRITE_ONCE(srclp->len, 1);
65		− srclp->len_lazy = 0;
66	60	}
67	61	}
68	62
69	63	/*
70	64	* Dequeue the oldest rcu_head structure from the specified callback
71		− * list. This function assumes that the callback is non-lazy, but
72		− * the caller can later invoke rcu_cblist_dequeued_lazy() if it
73		− * finds otherwise (and if it cares about laziness). This allows
74		− * different users to have different ways of determining laziness.
	65	+ * list.
75	66	*/
76	67	struct rcu_head rcu_cblist_dequeue(struct rcu_cblist rclp)
77	68	{
@@ -161,7 +152,6 @@ void rcu_segcblist_init(struct rcu_segcblist *rsclp) · 83 unmodified lines
161	152	for (i = 0; i < RCU_CBLIST_NSEGS; i++)
162	153	rsclp->tails[i] = &rsclp->head;
163	154	rcu_segcblist_set_len(rsclp, 0);
164		− rsclp->len_lazy = 0;
165	155	rsclp->enabled = 1;
166	156	}
167	157
@@ -173,7 +163,6 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp) · 5 unmodified lines
173	163	{
174	164	WARN_ON_ONCE(!rcu_segcblist_empty(rsclp));
175	165	WARN_ON_ONCE(rcu_segcblist_n_cbs(rsclp));
176		− WARN_ON_ONCE(rcu_segcblist_n_lazy_cbs(rsclp));
177	166	rsclp->enabled = 0;
178	167	}
179	168
@@ -253,11 +242,9 @@ bool rcu_segcblist_nextgp(struct rcu_segcblist rsclp, unsigned long lp) · 73 unmodified lines
253	242	* absolutely not OK for it to ever miss posting a callback.
254	243	*/
255	244	void rcu_segcblist_enqueue(struct rcu_segcblist *rsclp,
256		− struct rcu_head *rhp, bool lazy)
	245	+ struct rcu_head *rhp)
257	246	{
258	247	rcu_segcblist_inc_len(rsclp);
259		− if (lazy)
260		− rsclp->len_lazy++;
261	248	smp_mb(); /* Ensure counts are updated before callback is enqueued. */
262	249	rhp->next = NULL;
263	250	WRITE_ONCE(*rsclp->tails[RCU_NEXT_TAIL], rhp);
@@ -275,15 +262,13 @@ void rcu_segcblist_enqueue(struct rcu_segcblist *rsclp, · 11 unmodified lines
275	262	* period. You have been warned.
276	263	*/
277	264	bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp,
278		− struct rcu_head *rhp, bool lazy)
	265	+ struct rcu_head *rhp)
279	266	{
280	267	int i;
281	268
282	269	if (rcu_segcblist_n_cbs(rsclp) == 0)
283	270	return false;
284	271	rcu_segcblist_inc_len(rsclp);
285		− if (lazy)
286		− rsclp->len_lazy++;
287	272	smp_mb(); /* Ensure counts are updated before callback is entrained. */
288	273	rhp->next = NULL;
289	274	for (i = RCU_NEXT_TAIL; i > RCU_DONE_TAIL; i--)
@@ -307,8 +292,6 @@ bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp, · 17 unmodified lines
307	292	void rcu_segcblist_extract_count(struct rcu_segcblist *rsclp,
308	293	struct rcu_cblist *rclp)
309	294	{
310		− rclp->len_lazy += rsclp->len_lazy;
311		− rsclp->len_lazy = 0;
312	295	rclp->len = rcu_segcblist_xchg_len(rsclp, 0);
313	296	}
314	297
@@ -361,9 +344,7 @@ void rcu_segcblist_extract_pend_cbs(struct rcu_segcblist *rsclp, · 46 unmodified lines
361	344	void rcu_segcblist_insert_count(struct rcu_segcblist *rsclp,
362	345	struct rcu_cblist *rclp)
363	346	{
364		− rsclp->len_lazy += rclp->len_lazy;
365	347	rcu_segcblist_add_len(rsclp, rclp->len);
366		− rclp->len_lazy = 0;
367	348	rclp->len = 0;
368	349	}
369	350

kernel/rcu/rcu_segcblist.h+2−23

@@ -15,15 +15,6 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *rclp) · 14 unmodified lines
15	15	return READ_ONCE(rclp->len);
16	16	}
17	17
18		−/*
19		− * Account for the fact that a previously dequeued callback turned out
20		− * to be marked as lazy.
21		− */
22		−static inline void rcu_cblist_dequeued_lazy(struct rcu_cblist *rclp)
23		−{
24		− rclp->len_lazy--;
25		−}
26		−
27	18	void rcu_cblist_init(struct rcu_cblist *rclp);
28	19	void rcu_cblist_enqueue(struct rcu_cblist rclp, struct rcu_head rhp);
29	20	void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,
@@ -59,18 +50,6 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp) · 29 unmodified lines
59	50	#endif
60	51	}
61	52
62		−/* Return number of lazy callbacks in segmented callback list. */
63		−static inline long rcu_segcblist_n_lazy_cbs(struct rcu_segcblist *rsclp)
64		−{
65		− return rsclp->len_lazy;
66		−}
67		−
68		−/* Return number of lazy callbacks in segmented callback list. */
69		−static inline long rcu_segcblist_n_nonlazy_cbs(struct rcu_segcblist *rsclp)
70		−{
71		− return rcu_segcblist_n_cbs(rsclp) - rsclp->len_lazy;
72		−}
73		−
74	53	/*
75	54	* Is the specified rcu_segcblist enabled, for example, not corresponding
76	55	* to an offline CPU?
@@ -106,9 +85,9 @@ struct rcu_head rcu_segcblist_first_cb(struct rcu_segcblist rsclp); · 29 unmodified lines
106	85	struct rcu_head rcu_segcblist_first_pend_cb(struct rcu_segcblist rsclp);
107	86	bool rcu_segcblist_nextgp(struct rcu_segcblist rsclp, unsigned long lp);
108	87	void rcu_segcblist_enqueue(struct rcu_segcblist *rsclp,
109		− struct rcu_head *rhp, bool lazy);
	88	+ struct rcu_head *rhp);
110	89	bool rcu_segcblist_entrain(struct rcu_segcblist *rsclp,
111		− struct rcu_head *rhp, bool lazy);
	90	+ struct rcu_head *rhp);
112	91	void rcu_segcblist_extract_count(struct rcu_segcblist *rsclp,
113	92	struct rcu_cblist *rclp);
114	93	void rcu_segcblist_extract_done_cbs(struct rcu_segcblist *rsclp,

kernel/rcu/rcuperf.c+165−8

@@ -86,6 +86,7 @@ torture_param(bool, shutdown, RCUPERF_SHUTDOWN, · 85 unmodified lines
86	86	"Shutdown at end of performance tests.");
87	87	torture_param(int, verbose, 1, "Enable verbose debugging printk()s");
88	88	torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable");
	89	+torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() perf test?");
89	90
90	91	static char *perf_type = "rcu";
91	92	module_param(perf_type, charp, 0444);
@@ -105,8 +106,8 @@ static atomic_t n_rcu_perf_writer_finished; · 13 unmodified lines
105	106	static wait_queue_head_t shutdown_wq;
106	107	static u64 t_rcu_perf_writer_started;
107	108	static u64 t_rcu_perf_writer_finished;
108		−static unsigned long b_rcu_perf_writer_started;
109		−static unsigned long b_rcu_perf_writer_finished;
	109	+static unsigned long b_rcu_gp_test_started;
	110	+static unsigned long b_rcu_gp_test_finished;
110	111	static DEFINE_PER_CPU(atomic_t, n_async_inflight);
111	112
112	113	#define MAX_MEAS 10000
@@ -378,10 +379,10 @@ rcu_perf_writer(void *arg) · 265 unmodified lines
378	379	if (atomic_inc_return(&n_rcu_perf_writer_started) >= nrealwriters) {
379	380	t_rcu_perf_writer_started = t;
380	381	if (gp_exp) {
381		− b_rcu_perf_writer_started =
	382	+ b_rcu_gp_test_started =
382	383	cur_ops->exp_completed() / 2;
383	384	} else {
384		− b_rcu_perf_writer_started = cur_ops->get_gp_seq();
	385	+ b_rcu_gp_test_started = cur_ops->get_gp_seq();
385	386	}
386	387	}
387	388
@@ -429,10 +430,10 @@ retry: · 41 unmodified lines
429	430	PERFOUT_STRING("Test complete");
430	431	t_rcu_perf_writer_finished = t;
431	432	if (gp_exp) {
432		− b_rcu_perf_writer_finished =
	433	+ b_rcu_gp_test_finished =
433	434	cur_ops->exp_completed() / 2;
434	435	} else {
435		− b_rcu_perf_writer_finished =
	436	+ b_rcu_gp_test_finished =
436	437	cur_ops->get_gp_seq();
437	438	}
438	439	if (shutdown) {
@@ -515,8 +516,8 @@ rcu_perf_cleanup(void) · 76 unmodified lines
515	516	t_rcu_perf_writer_finished -
516	517	t_rcu_perf_writer_started,
517	518	ngps,
518		− rcuperf_seq_diff(b_rcu_perf_writer_finished,
519		− b_rcu_perf_writer_started));
	519	+ rcuperf_seq_diff(b_rcu_gp_test_finished,
	520	+ b_rcu_gp_test_started));
520	521	for (i = 0; i < nrealwriters; i++) {
521	522	if (!writer_durations)
522	523	break;
@@ -584,6 +585,159 @@ rcu_perf_shutdown(void *arg) · 61 unmodified lines
584	585	return -EINVAL;
585	586	}
586	587
	588	+/*
	589	+ * kfree_rcu() performance tests: Start a kfree_rcu() loop on all CPUs for number
	590	+ * of iterations and measure total time and number of GP for all iterations to complete.
	591	+ */
	592	+
	593	+torture_param(int, kfree_nthreads, -1, "Number of threads running loops of kfree_rcu().");
	594	+torture_param(int, kfree_alloc_num, 8000, "Number of allocations and frees done in an iteration.");
	595	+torture_param(int, kfree_loops, 10, "Number of loops doing kfree_alloc_num allocations and frees.");
	596	+
	597	+static struct task_struct **kfree_reader_tasks;
	598	+static int kfree_nrealthreads;
	599	+static atomic_t n_kfree_perf_thread_started;
	600	+static atomic_t n_kfree_perf_thread_ended;
	601	+
	602	+struct kfree_obj {
	603	+ char kfree_obj[8];
	604	+ struct rcu_head rh;
	605	+};
	606	+
	607	+static int
	608	+kfree_perf_thread(void *arg)
	609	+{
	610	+ int i, loop = 0;
	611	+ long me = (long)arg;
	612	+ struct kfree_obj *alloc_ptr;
	613	+ u64 start_time, end_time;
	614	+
	615	+ VERBOSE_PERFOUT_STRING("kfree_perf_thread task started");
	616	+ set_cpus_allowed_ptr(current, cpumask_of(me % nr_cpu_ids));
	617	+ set_user_nice(current, MAX_NICE);
	618	+
	619	+ start_time = ktime_get_mono_fast_ns();
	620	+
	621	+ if (atomic_inc_return(&n_kfree_perf_thread_started) >= kfree_nrealthreads) {
	622	+ if (gp_exp)
	623	+ b_rcu_gp_test_started = cur_ops->exp_completed() / 2;
	624	+ else
	625	+ b_rcu_gp_test_started = cur_ops->get_gp_seq();
	626	+ }
	627	+
	628	+ do {
	629	+ for (i = 0; i < kfree_alloc_num; i++) {
	630	+ alloc_ptr = kmalloc(sizeof(struct kfree_obj), GFP_KERNEL);
	631	+ if (!alloc_ptr)
	632	+ return -ENOMEM;
	633	+
	634	+ kfree_rcu(alloc_ptr, rh);
	635	+ }
	636	+
	637	+ cond_resched();
	638	+ } while (!torture_must_stop() && ++loop < kfree_loops);
	639	+
	640	+ if (atomic_inc_return(&n_kfree_perf_thread_ended) >= kfree_nrealthreads) {
	641	+ end_time = ktime_get_mono_fast_ns();
	642	+
	643	+ if (gp_exp)
	644	+ b_rcu_gp_test_finished = cur_ops->exp_completed() / 2;
	645	+ else
	646	+ b_rcu_gp_test_finished = cur_ops->get_gp_seq();
	647	+
	648	+ pr_alert("Total time taken by all kfree'ers: %llu ns, loops: %d, batches: %ld\n",
	649	+ (unsigned long long)(end_time - start_time), kfree_loops,
	650	+ rcuperf_seq_diff(b_rcu_gp_test_finished, b_rcu_gp_test_started));
	651	+ if (shutdown) {
	652	+ smp_mb(); /* Assign before wake. */
	653	+ wake_up(&shutdown_wq);
	654	+ }
	655	+ }
	656	+
	657	+ torture_kthread_stopping("kfree_perf_thread");
	658	+ return 0;
	659	+}
	660	+
	661	+static void
	662	+kfree_perf_cleanup(void)
	663	+{
	664	+ int i;
	665	+
	666	+ if (torture_cleanup_begin())
	667	+ return;
	668	+
	669	+ if (kfree_reader_tasks) {
	670	+ for (i = 0; i < kfree_nrealthreads; i++)
	671	+ torture_stop_kthread(kfree_perf_thread,
	672	+ kfree_reader_tasks[i]);
	673	+ kfree(kfree_reader_tasks);
	674	+ }
	675	+
	676	+ torture_cleanup_end();
	677	+}
	678	+
	679	+/*
	680	+ * shutdown kthread. Just waits to be awakened, then shuts down system.
	681	+ */
	682	+static int
	683	+kfree_perf_shutdown(void *arg)
	684	+{
	685	+ do {
	686	+ wait_event(shutdown_wq,
	687	+ atomic_read(&n_kfree_perf_thread_ended) >=
	688	+ kfree_nrealthreads);
	689	+ } while (atomic_read(&n_kfree_perf_thread_ended) < kfree_nrealthreads);
	690	+
	691	+ smp_mb(); /* Wake before output. */
	692	+
	693	+ kfree_perf_cleanup();
	694	+ kernel_power_off();
	695	+ return -EINVAL;
	696	+}
	697	+
	698	+static int __init
	699	+kfree_perf_init(void)
	700	+{
	701	+ long i;
	702	+ int firsterr = 0;
	703	+
	704	+ kfree_nrealthreads = compute_real(kfree_nthreads);
	705	+ /* Start up the kthreads. */
	706	+ if (shutdown) {
	707	+ init_waitqueue_head(&shutdown_wq);
	708	+ firsterr = torture_create_kthread(kfree_perf_shutdown, NULL,
	709	+ shutdown_task);
	710	+ if (firsterr)
	711	+ goto unwind;
	712	+ schedule_timeout_uninterruptible(1);
	713	+ }
	714	+
	715	+ kfree_reader_tasks = kcalloc(kfree_nrealthreads, sizeof(kfree_reader_tasks[0]),
	716	+ GFP_KERNEL);
	717	+ if (kfree_reader_tasks == NULL) {
	718	+ firsterr = -ENOMEM;
	719	+ goto unwind;
	720	+ }
	721	+
	722	+ for (i = 0; i < kfree_nrealthreads; i++) {
	723	+ firsterr = torture_create_kthread(kfree_perf_thread, (void *)i,
	724	+ kfree_reader_tasks[i]);
	725	+ if (firsterr)
	726	+ goto unwind;
	727	+ }
	728	+
	729	+ while (atomic_read(&n_kfree_perf_thread_started) < kfree_nrealthreads)
	730	+ schedule_timeout_uninterruptible(1);
	731	+
	732	+ torture_init_end();
	733	+ return 0;
	734	+
	735	+unwind:
	736	+ torture_init_end();
	737	+ kfree_perf_cleanup();
	738	+ return firsterr;
	739	+}
	740	+
587	741	static int __init
588	742	rcu_perf_init(void)
589	743	{
@@ -616,6 +770,9 @@ rcu_perf_init(void) · 26 unmodified lines
616	770	if (cur_ops->init)
617	771	cur_ops->init();
618	772
	773	+ if (kfree_rcu_test)
	774	+ return kfree_perf_init();
	775	+
619	776	nrealwriters = compute_real(nwriters);
620	777	nrealreaders = compute_real(nreaders);
621	778	atomic_set(&n_rcu_perf_reader_started, 0);

kernel/rcu/rcutorture.c+81−60

@@ -1661,43 +1661,52 @@ static void rcu_torture_fwd_prog_cb(struct rcu_head *rhp) · 1660 unmodified lines
1661	1661	struct rcu_fwd_cb {
1662	1662	struct rcu_head rh;
1663	1663	struct rcu_fwd_cb *rfc_next;
	1664	+ struct rcu_fwd *rfc_rfp;
1664	1665	int rfc_gps;
1665	1666	};
1666		−static DEFINE_SPINLOCK(rcu_fwd_lock);
1667		−static struct rcu_fwd_cb *rcu_fwd_cb_head;
1668		−static struct rcu_fwd_cb **rcu_fwd_cb_tail = &rcu_fwd_cb_head;
1669		−static long n_launders_cb;
1670		−static unsigned long rcu_fwd_startat;
1671		−static bool rcu_fwd_emergency_stop;
	1667	+
1672	1668	#define MAX_FWD_CB_JIFFIES (8 * HZ) /* Maximum CB test duration. */
1673	1669	#define MIN_FWD_CB_LAUNDERS 3 /* This many CB invocations to count. */
1674	1670	#define MIN_FWD_CBS_LAUNDERED 100 /* Number of counted CBs. */
1675	1671	#define FWD_CBS_HIST_DIV 10 /* Histogram buckets/second. */
	1672	+#define N_LAUNDERS_HIST (2 * MAX_FWD_CB_JIFFIES / (HZ / FWD_CBS_HIST_DIV))
	1673	+
1676	1674	struct rcu_launder_hist {
1677	1675	long n_launders;
1678	1676	unsigned long launder_gp_seq;
1679	1677	};
1680		−#define N_LAUNDERS_HIST (2 * MAX_FWD_CB_JIFFIES / (HZ / FWD_CBS_HIST_DIV))
1681		−static struct rcu_launder_hist n_launders_hist[N_LAUNDERS_HIST];
1682		−static unsigned long rcu_launder_gp_seq_start;
1683	1678
1684		−static void rcu_torture_fwd_cb_hist(void)
	1679	+struct rcu_fwd {
	1680	+ spinlock_t rcu_fwd_lock;
	1681	+ struct rcu_fwd_cb *rcu_fwd_cb_head;
	1682	+ struct rcu_fwd_cb **rcu_fwd_cb_tail;
	1683	+ long n_launders_cb;
	1684	+ unsigned long rcu_fwd_startat;
	1685	+ struct rcu_launder_hist n_launders_hist[N_LAUNDERS_HIST];
	1686	+ unsigned long rcu_launder_gp_seq_start;
	1687	+};
	1688	+
	1689	+struct rcu_fwd *rcu_fwds;
	1690	+bool rcu_fwd_emergency_stop;
	1691	+
	1692	+static void rcu_torture_fwd_cb_hist(struct rcu_fwd *rfp)
1685	1693	{
1686	1694	unsigned long gps;
1687	1695	unsigned long gps_old;
1688	1696	int i;
1689	1697	int j;
1690	1698
1691		− for (i = ARRAY_SIZE(n_launders_hist) - 1; i > 0; i--)
1692		− if (n_launders_hist[i].n_launders > 0)
	1699	+ for (i = ARRAY_SIZE(rfp->n_launders_hist) - 1; i > 0; i--)
	1700	+ if (rfp->n_launders_hist[i].n_launders > 0)
1693	1701	break;
1694	1702	pr_alert("%s: Callback-invocation histogram (duration %lu jiffies):",
1695		− __func__, jiffies - rcu_fwd_startat);
1696		− gps_old = rcu_launder_gp_seq_start;
	1703	+ __func__, jiffies - rfp->rcu_fwd_startat);
	1704	+ gps_old = rfp->rcu_launder_gp_seq_start;
1697	1705	for (j = 0; j <= i; j++) {
1698		− gps = n_launders_hist[j].launder_gp_seq;
	1706	+ gps = rfp->n_launders_hist[j].launder_gp_seq;
1699	1707	pr_cont(" %ds/%d: %ld:%ld",
1700		− j + 1, FWD_CBS_HIST_DIV, n_launders_hist[j].n_launders,
	1708	+ j + 1, FWD_CBS_HIST_DIV,
	1709	+ rfp->n_launders_hist[j].n_launders,
1701	1710	rcutorture_seq_diff(gps, gps_old));
1702	1711	gps_old = gps;
1703	1712	}
@@ -1711,26 +1720,27 @@ static void rcu_torture_fwd_cb_cr(struct rcu_head *rhp) · 7 unmodified lines
1711	1720	int i;
1712	1721	struct rcu_fwd_cb *rfcp = container_of(rhp, struct rcu_fwd_cb, rh);
1713	1722	struct rcu_fwd_cb **rfcpp;
	1723	+ struct rcu_fwd *rfp = rfcp->rfc_rfp;
1714	1724
1715	1725	rfcp->rfc_next = NULL;
1716	1726	rfcp->rfc_gps++;
1717		− spin_lock_irqsave(&rcu_fwd_lock, flags);
1718		− rfcpp = rcu_fwd_cb_tail;
1719		− rcu_fwd_cb_tail = &rfcp->rfc_next;
	1727	+ spin_lock_irqsave(&rfp->rcu_fwd_lock, flags);
	1728	+ rfcpp = rfp->rcu_fwd_cb_tail;
	1729	+ rfp->rcu_fwd_cb_tail = &rfcp->rfc_next;
1720	1730	WRITE_ONCE(*rfcpp, rfcp);
1721		− WRITE_ONCE(n_launders_cb, n_launders_cb + 1);
1722		− i = ((jiffies - rcu_fwd_startat) / (HZ / FWD_CBS_HIST_DIV));
1723		− if (i >= ARRAY_SIZE(n_launders_hist))
1724		− i = ARRAY_SIZE(n_launders_hist) - 1;
1725		− n_launders_hist[i].n_launders++;
1726		− n_launders_hist[i].launder_gp_seq = cur_ops->get_gp_seq();
1727		− spin_unlock_irqrestore(&rcu_fwd_lock, flags);
	1731	+ WRITE_ONCE(rfp->n_launders_cb, rfp->n_launders_cb + 1);
	1732	+ i = ((jiffies - rfp->rcu_fwd_startat) / (HZ / FWD_CBS_HIST_DIV));
	1733	+ if (i >= ARRAY_SIZE(rfp->n_launders_hist))
	1734	+ i = ARRAY_SIZE(rfp->n_launders_hist) - 1;
	1735	+ rfp->n_launders_hist[i].n_launders++;
	1736	+ rfp->n_launders_hist[i].launder_gp_seq = cur_ops->get_gp_seq();
	1737	+ spin_unlock_irqrestore(&rfp->rcu_fwd_lock, flags);
1728	1738	}
1729	1739
1730	1740	// Give the scheduler a chance, even on nohz_full CPUs.
1731	1741	static void rcu_torture_fwd_prog_cond_resched(unsigned long iter)
1732	1742	{
1733		− if (IS_ENABLED(CONFIG_PREEMPT) && IS_ENABLED(CONFIG_NO_HZ_FULL)) {
	1743	+ if (IS_ENABLED(CONFIG_PREEMPTION) && IS_ENABLED(CONFIG_NO_HZ_FULL)) {
1734	1744	// Real call_rcu() floods hit userspace, so emulate that.
1735	1745	if (need_resched() \|\| (iter & 0xfff))
1736	1746	schedule();
@@ -1744,23 +1754,23 @@ static void rcu_torture_fwd_prog_cond_resched(unsigned long iter) · 7 unmodified lines
1744	1754	* Free all callbacks on the rcu_fwd_cb_head list, either because the
1745	1755	* test is over or because we hit an OOM event.
1746	1756	*/
1747		−static unsigned long rcu_torture_fwd_prog_cbfree(void)
	1757	+static unsigned long rcu_torture_fwd_prog_cbfree(struct rcu_fwd *rfp)
1748	1758	{
1749	1759	unsigned long flags;
1750	1760	unsigned long freed = 0;
1751	1761	struct rcu_fwd_cb *rfcp;
1752	1762
1753	1763	for (;;) {
1754		− spin_lock_irqsave(&rcu_fwd_lock, flags);
1755		− rfcp = rcu_fwd_cb_head;
	1764	+ spin_lock_irqsave(&rfp->rcu_fwd_lock, flags);
	1765	+ rfcp = rfp->rcu_fwd_cb_head;
1756	1766	if (!rfcp) {
1757		− spin_unlock_irqrestore(&rcu_fwd_lock, flags);
	1767	+ spin_unlock_irqrestore(&rfp->rcu_fwd_lock, flags);
1758	1768	break;
1759	1769	}
1760		− rcu_fwd_cb_head = rfcp->rfc_next;
1761		− if (!rcu_fwd_cb_head)
1762		− rcu_fwd_cb_tail = &rcu_fwd_cb_head;
1763		− spin_unlock_irqrestore(&rcu_fwd_lock, flags);
	1770	+ rfp->rcu_fwd_cb_head = rfcp->rfc_next;
	1771	+ if (!rfp->rcu_fwd_cb_head)
	1772	+ rfp->rcu_fwd_cb_tail = &rfp->rcu_fwd_cb_head;
	1773	+ spin_unlock_irqrestore(&rfp->rcu_fwd_lock, flags);
1764	1774	kfree(rfcp);
1765	1775	freed++;
1766	1776	rcu_torture_fwd_prog_cond_resched(freed);
@@ -1774,7 +1784,8 @@ static unsigned long rcu_torture_fwd_prog_cbfree(void) · 7 unmodified lines
1774	1784	}
1775	1785
1776	1786	/* Carry out need_resched()/cond_resched() forward-progress testing. */
1777		−static void rcu_torture_fwd_prog_nr(int tested, int tested_tries)
	1787	+static void rcu_torture_fwd_prog_nr(struct rcu_fwd *rfp,
	1788	+ int tested, int tested_tries)
1778	1789	{
1779	1790	unsigned long cver;
1780	1791	unsigned long dur;
@@ -1804,8 +1815,8 @@ static void rcu_torture_fwd_prog_nr(int tested, int tested_tries) · 23 unmodified lines
1804	1815	sd = cur_ops->stall_dur() + 1;
1805	1816	sd4 = (sd + fwd_progress_div - 1) / fwd_progress_div;
1806	1817	dur = sd4 + torture_random(&trs) % (sd - sd4);
1807		− WRITE_ONCE(rcu_fwd_startat, jiffies);
1808		− stopat = rcu_fwd_startat + dur;
	1818	+ WRITE_ONCE(rfp->rcu_fwd_startat, jiffies);
	1819	+ stopat = rfp->rcu_fwd_startat + dur;
1809	1820	while (time_before(jiffies, stopat) &&
1810	1821	!shutdown_time_arrived() &&
1811	1822	!READ_ONCE(rcu_fwd_emergency_stop) && !torture_must_stop()) {
@@ -1840,7 +1851,7 @@ static void rcu_torture_fwd_prog_nr(int tested, int tested_tries) · 28 unmodified lines
1840	1851	}
1841	1852
1842	1853	/* Carry out call_rcu() forward-progress testing. */
1843		−static void rcu_torture_fwd_prog_cr(void)
	1854	+static void rcu_torture_fwd_prog_cr(struct rcu_fwd *rfp)
1844	1855	{
1845	1856	unsigned long cver;
1846	1857	unsigned long flags;
@@ -1864,23 +1875,23 @@ static void rcu_torture_fwd_prog_cr(void) · 17 unmodified lines
1864	1875	/* Loop continuously posting RCU callbacks. */
1865	1876	WRITE_ONCE(rcu_fwd_cb_nodelay, true);
1866	1877	cur_ops->sync(); /* Later readers see above write. */
1867		− WRITE_ONCE(rcu_fwd_startat, jiffies);
1868		− stopat = rcu_fwd_startat + MAX_FWD_CB_JIFFIES;
	1878	+ WRITE_ONCE(rfp->rcu_fwd_startat, jiffies);
	1879	+ stopat = rfp->rcu_fwd_startat + MAX_FWD_CB_JIFFIES;
1869	1880	n_launders = 0;
1870		− n_launders_cb = 0;
	1881	+ rfp->n_launders_cb = 0; // Hoist initialization for multi-kthread
1871	1882	n_launders_sa = 0;
1872	1883	n_max_cbs = 0;
1873	1884	n_max_gps = 0;
1874		− for (i = 0; i < ARRAY_SIZE(n_launders_hist); i++)
1875		− n_launders_hist[i].n_launders = 0;
	1885	+ for (i = 0; i < ARRAY_SIZE(rfp->n_launders_hist); i++)
	1886	+ rfp->n_launders_hist[i].n_launders = 0;
1876	1887	cver = READ_ONCE(rcu_torture_current_version);
1877	1888	gps = cur_ops->get_gp_seq();
1878		− rcu_launder_gp_seq_start = gps;
	1889	+ rfp->rcu_launder_gp_seq_start = gps;
1879	1890	tick_dep_set_task(current, TICK_DEP_BIT_RCU);
1880	1891	while (time_before(jiffies, stopat) &&
1881	1892	!shutdown_time_arrived() &&
1882	1893	!READ_ONCE(rcu_fwd_emergency_stop) && !torture_must_stop()) {
1883		− rfcp = READ_ONCE(rcu_fwd_cb_head);
	1894	+ rfcp = READ_ONCE(rfp->rcu_fwd_cb_head);
1884	1895	rfcpn = NULL;
1885	1896	if (rfcp)
1886	1897	rfcpn = READ_ONCE(rfcp->rfc_next);
@@ -1888,7 +1899,7 @@ static void rcu_torture_fwd_prog_cr(void) · 1 unmodified line
1888	1899	if (rfcp->rfc_gps >= MIN_FWD_CB_LAUNDERS &&
1889	1900	++n_max_gps >= MIN_FWD_CBS_LAUNDERED)
1890	1901	break;
1891		− rcu_fwd_cb_head = rfcpn;
	1902	+ rfp->rcu_fwd_cb_head = rfcpn;
1892	1903	n_launders++;
1893	1904	n_launders_sa++;
1894	1905	} else {
@@ -1900,6 +1911,7 @@ static void rcu_torture_fwd_prog_cr(void) · 5 unmodified lines
1900	1911	n_max_cbs++;
1901	1912	n_launders_sa = 0;
1902	1913	rfcp->rfc_gps = 0;
	1914	+ rfcp->rfc_rfp = rfp;
1903	1915	}
1904	1916	cur_ops->call(&rfcp->rh, rcu_torture_fwd_cb_cr);
1905	1917	rcu_torture_fwd_prog_cond_resched(n_launders + n_max_cbs);
@@ -1910,22 +1922,22 @@ static void rcu_torture_fwd_prog_cr(void) · 4 unmodified lines
1910	1922	}
1911	1923	}
1912	1924	stoppedat = jiffies;
1913		− n_launders_cb_snap = READ_ONCE(n_launders_cb);
	1925	+ n_launders_cb_snap = READ_ONCE(rfp->n_launders_cb);
1914	1926	cver = READ_ONCE(rcu_torture_current_version) - cver;
1915	1927	gps = rcutorture_seq_diff(cur_ops->get_gp_seq(), gps);
1916	1928	cur_ops->cb_barrier(); /* Wait for callbacks to be invoked. */
1917		− (void)rcu_torture_fwd_prog_cbfree();
	1929	+ (void)rcu_torture_fwd_prog_cbfree(rfp);
1918	1930
1919	1931	if (!torture_must_stop() && !READ_ONCE(rcu_fwd_emergency_stop) &&
1920	1932	!shutdown_time_arrived()) {
1921	1933	WARN_ON(n_max_gps < MIN_FWD_CBS_LAUNDERED);
1922	1934	pr_alert("%s Duration %lu barrier: %lu pending %ld n_launders: %ld n_launders_sa: %ld n_max_gps: %ld n_max_cbs: %ld cver %ld gps %ld\n",
1923	1935	__func__,
1924		− stoppedat - rcu_fwd_startat, jiffies - stoppedat,
	1936	+ stoppedat - rfp->rcu_fwd_startat, jiffies - stoppedat,
1925	1937	n_launders + n_max_cbs - n_launders_cb_snap,
1926	1938	n_launders, n_launders_sa,
1927	1939	n_max_gps, n_max_cbs, cver, gps);
1928		− rcu_torture_fwd_cb_hist();
	1940	+ rcu_torture_fwd_cb_hist(rfp);
1929	1941	}
1930	1942	schedule_timeout_uninterruptible(HZ); /* Let CBs drain. */
1931	1943	tick_dep_clear_task(current, TICK_DEP_BIT_RCU);
@@ -1940,20 +1952,22 @@ static void rcu_torture_fwd_prog_cr(void) · 8 unmodified lines
1940	1952	static int rcutorture_oom_notify(struct notifier_block *self,
1941	1953	unsigned long notused, void *nfreed)
1942	1954	{
	1955	+ struct rcu_fwd *rfp = rcu_fwds;
	1956	+
1943	1957	WARN(1, "%s invoked upon OOM during forward-progress testing.\n",
1944	1958	__func__);
1945		− rcu_torture_fwd_cb_hist();
1946		− rcu_fwd_progress_check(1 + (jiffies - READ_ONCE(rcu_fwd_startat)) / 2);
	1959	+ rcu_torture_fwd_cb_hist(rfp);
	1960	+ rcu_fwd_progress_check(1 + (jiffies - READ_ONCE(rfp->rcu_fwd_startat)) / 2);
1947	1961	WRITE_ONCE(rcu_fwd_emergency_stop, true);
1948	1962	smp_mb(); /* Emergency stop before free and wait to avoid hangs. */
1949	1963	pr_info("%s: Freed %lu RCU callbacks.\n",
1950		− __func__, rcu_torture_fwd_prog_cbfree());
	1964	+ __func__, rcu_torture_fwd_prog_cbfree(rfp));
1951	1965	rcu_barrier();
1952	1966	pr_info("%s: Freed %lu RCU callbacks.\n",
1953		− __func__, rcu_torture_fwd_prog_cbfree());
	1967	+ __func__, rcu_torture_fwd_prog_cbfree(rfp));
1954	1968	rcu_barrier();
1955	1969	pr_info("%s: Freed %lu RCU callbacks.\n",
1956		− __func__, rcu_torture_fwd_prog_cbfree());
	1970	+ __func__, rcu_torture_fwd_prog_cbfree(rfp));
1957	1971	smp_mb(); /* Frees before return to avoid redoing OOM. */
1958	1972	((unsigned long )nfreed)++; /* Forward progress CBs freed! */
1959	1973	pr_info("%s returning after OOM processing.\n", __func__);
@@ -1967,6 +1981,7 @@ static struct notifier_block rcutorture_oom_nb = { · 7 unmodified lines
1967	1981	/* Carry out grace-period forward-progress testing. */
1968	1982	static int rcu_torture_fwd_prog(void *args)
1969	1983	{
	1984	+ struct rcu_fwd *rfp = args;
1970	1985	int tested = 0;
1971	1986	int tested_tries = 0;
1972	1987
@@ -1978,8 +1993,8 @@ static int rcu_torture_fwd_prog(void *args) · 5 unmodified lines
1978	1993	schedule_timeout_interruptible(fwd_progress_holdoff * HZ);
1979	1994	WRITE_ONCE(rcu_fwd_emergency_stop, false);
1980	1995	register_oom_notifier(&rcutorture_oom_nb);
1981		− rcu_torture_fwd_prog_nr(&tested, &tested_tries);
1982		− rcu_torture_fwd_prog_cr();
	1996	+ rcu_torture_fwd_prog_nr(rfp, &tested, &tested_tries);
	1997	+ rcu_torture_fwd_prog_cr(rfp);
1983	1998	unregister_oom_notifier(&rcutorture_oom_nb);
1984	1999
1985	2000	/* Avoid slow periods, better to test when busy. */
@@ -1995,6 +2010,8 @@ static int rcu_torture_fwd_prog(void *args) · 9 unmodified lines
1995	2010	/* If forward-progress checking is requested and feasible, spawn the thread. */
1996	2011	static int __init rcu_torture_fwd_prog_init(void)
1997	2012	{
	2013	+ struct rcu_fwd *rfp;
	2014	+
1998	2015	if (!fwd_progress)
1999	2016	return 0; /* Not requested, so don't do it. */
2000	2017	if (!cur_ops->stall_dur \|\| cur_ops->stall_dur() <= 0 \|\|
@@ -2013,8 +2030,12 @@ static int __init rcu_torture_fwd_prog_init(void) · 12 unmodified lines
2013	2030	fwd_progress_holdoff = 1;
2014	2031	if (fwd_progress_div <= 0)
2015	2032	fwd_progress_div = 4;
2016		− return torture_create_kthread(rcu_torture_fwd_prog,
2017		− NULL, fwd_prog_task);
	2033	+ rfp = kzalloc(sizeof(*rfp), GFP_KERNEL);
	2034	+ if (!rfp)
	2035	+ return -ENOMEM;
	2036	+ spin_lock_init(&rfp->rcu_fwd_lock);
	2037	+ rfp->rcu_fwd_cb_tail = &rfp->rcu_fwd_cb_head;
	2038	+ return torture_create_kthread(rcu_torture_fwd_prog, rfp, fwd_prog_task);
2018	2039	}
2019	2040
2020	2041	/* Callback function for RCU barrier testing. */

kernel/rcu/srcutiny.c+1−1

@@ -103,7 +103,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock); · 102 unmodified lines
103	103
104	104	/*
105	105	* Workqueue handler to drive one grace period and invoke any callbacks
106		− * that become ready as a result. Single-CPU and !PREEMPT operation
	106	+ * that become ready as a result. Single-CPU and !PREEMPTION operation
107	107	* means that we get away with murder on synchronization. ;-)
108	108	*/
109	109	void srcu_drive_gp(struct work_struct *wp)

kernel/rcu/srcutree.c+6−5

@@ -530,7 +530,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) · 529 unmodified lines
530	530	idx = rcu_seq_state(ssp->srcu_gp_seq);
531	531	WARN_ON_ONCE(idx != SRCU_STATE_SCAN2);
532	532	cbdelay = srcu_get_delay(ssp);
533		− ssp->srcu_last_gp_end = ktime_get_mono_fast_ns();
	533	+ WRITE_ONCE(ssp->srcu_last_gp_end, ktime_get_mono_fast_ns());
534	534	rcu_seq_end(&ssp->srcu_gp_seq);
535	535	gpseq = rcu_seq_current(&ssp->srcu_gp_seq);
536	536	if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, gpseq))
@@ -762,6 +762,7 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp) · 225 unmodified lines
762	762	unsigned long flags;
763	763	struct srcu_data *sdp;
764	764	unsigned long t;
	765	+ unsigned long tlast;
765	766
766	767	/* If the local srcu_data structure has callbacks, not idle. */
767	768	local_irq_save(flags);
@@ -780,9 +781,9 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp) · 12 unmodified lines
780	781
781	782	/* First, see if enough time has passed since the last GP. */
782	783	t = ktime_get_mono_fast_ns();
	784	+ tlast = READ_ONCE(ssp->srcu_last_gp_end);
783	785	if (exp_holdoff == 0 \|\|
784		− time_in_range_open(t, ssp->srcu_last_gp_end,
785		− ssp->srcu_last_gp_end + exp_holdoff))
	786	+ time_in_range_open(t, tlast, tlast + exp_holdoff))
786	787	return false; /* Too soon after last GP. */
787	788
788	789	/* Next, check for probable idleness. */
@@ -853,7 +854,7 @@ static void __call_srcu(struct srcu_struct ssp, struct rcu_head rhp, · 64 unmodified lines
853	854	local_irq_save(flags);
854	855	sdp = this_cpu_ptr(ssp->sda);
855	856	spin_lock_rcu_node(sdp);
856		− rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp, false);
	857	+ rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp);
857	858	rcu_segcblist_advance(&sdp->srcu_cblist,
858	859	rcu_seq_current(&ssp->srcu_gp_seq));
859	860	s = rcu_seq_snap(&ssp->srcu_gp_seq);
@@ -1052,7 +1053,7 @@ void srcu_barrier(struct srcu_struct *ssp) · 192 unmodified lines
1052	1053	sdp->srcu_barrier_head.func = srcu_barrier_cb;
1053	1054	debug_rcu_head_queue(&sdp->srcu_barrier_head);
1054	1055	if (!rcu_segcblist_entrain(&sdp->srcu_cblist,
1055		− &sdp->srcu_barrier_head, 0)) {
	1056	+ &sdp->srcu_barrier_head)) {
1056	1057	debug_rcu_head_unqueue(&sdp->srcu_barrier_head);
1057	1058	atomic_dec(&ssp->srcu_barrier_cpu_cnt);
1058	1059	}

kernel/rcu/tiny.c+27−1

@@ -22,6 +22,7 @@ · 21 unmodified lines
22	22	#include <linux/time.h>
23	23	#include <linux/cpu.h>
24	24	#include <linux/prefetch.h>
	25	+#include <linux/slab.h>
25	26
26	27	#include "rcu.h"
27	28
@@ -73,6 +74,31 @@ void rcu_sched_clock_irq(int user) · 45 unmodified lines
73	74	}
74	75	}
75	76
	77	+/*
	78	+ * Reclaim the specified callback, either by invoking it for non-kfree cases or
	79	+ * freeing it directly (for kfree). Return true if kfreeing, false otherwise.
	80	+ */
	81	+static inline bool rcu_reclaim_tiny(struct rcu_head *head)
	82	+{
	83	+ rcu_callback_t f;
	84	+ unsigned long offset = (unsigned long)head->func;
	85	+
	86	+ rcu_lock_acquire(&rcu_callback_map);
	87	+ if (__is_kfree_rcu_offset(offset)) {
	88	+ trace_rcu_invoke_kfree_callback("", head, offset);
	89	+ kfree((void *)head - offset);
	90	+ rcu_lock_release(&rcu_callback_map);
	91	+ return true;
	92	+ }
	93	+
	94	+ trace_rcu_invoke_callback("", head);
	95	+ f = head->func;
	96	+ WRITE_ONCE(head->func, (rcu_callback_t)0L);
	97	+ f(head);
	98	+ rcu_lock_release(&rcu_callback_map);
	99	+ return false;
	100	+}
	101	+
76	102	/* Invoke the RCU callbacks whose grace period has elapsed. */
77	103	static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused)
78	104	{
@@ -100,7 +126,7 @@ static __latent_entropy void rcu_process_callbacks(struct softirq_action *unused · 21 unmodified lines
100	126	prefetch(next);
101	127	debug_rcu_head_unqueue(list);
102	128	local_bh_disable();
103		− __rcu_reclaim("", list);
	129	+ rcu_reclaim_tiny(list);
104	130	local_bh_enable();
105	131	list = next;
106	132	}

kernel/rcu/tree.c+272−52

@@ -43,7 +43,6 @@ · 42 unmodified lines
43	43	#include <uapi/linux/sched/types.h>
44	44	#include <linux/prefetch.h>
45	45	#include <linux/delay.h>
46		−#include <linux/stop_machine.h>
47	46	#include <linux/random.h>
48	47	#include <linux/trace_events.h>
49	48	#include <linux/suspend.h>
@@ -55,6 +54,7 @@ · 5 unmodified lines
55	54	#include <linux/oom.h>
56	55	#include <linux/smpboot.h>
57	56	#include <linux/jiffies.h>
	57	+#include <linux/slab.h>
58	58	#include <linux/sched/isolation.h>
59	59	#include <linux/sched/clock.h>
60	60	#include "../time/tick-internal.h"
@@ -84,7 +84,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = { · 23 unmodified lines
84	84	.dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE,
85	85	.dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),
86	86	};
87		−struct rcu_state rcu_state = {
	87	+static struct rcu_state rcu_state = {
88	88	.level = { &rcu_state.node[0] },
89	89	.gp_state = RCU_GP_IDLE,
90	90	.gp_seq = (0UL - 300UL) << RCU_SEQ_CTR_SHIFT,
@@ -188,7 +188,7 @@ EXPORT_SYMBOL_GPL(rcu_get_gp_kthreads_prio); · 97 unmodified lines
188	188	* held, but the bit corresponding to the current CPU will be stable
189	189	* in most contexts.
190	190	*/
191		−unsigned long rcu_rnp_online_cpus(struct rcu_node *rnp)
	191	+static unsigned long rcu_rnp_online_cpus(struct rcu_node *rnp)
192	192	{
193	193	return READ_ONCE(rnp->qsmaskinitnext);
194	194	}
@@ -294,7 +294,7 @@ static void rcu_dynticks_eqs_online(void) · 99 unmodified lines
294	294	*
295	295	* No ordering, as we are sampling CPU-local information.
296	296	*/
297		−bool rcu_dynticks_curr_cpu_in_eqs(void)
	297	+static bool rcu_dynticks_curr_cpu_in_eqs(void)
298	298	{
299	299	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
300	300
@@ -305,7 +305,7 @@ bool rcu_dynticks_curr_cpu_in_eqs(void) · 4 unmodified lines
305	305	* Snapshot the ->dynticks counter with full ordering so as to allow
306	306	* stable comparison of this counter with past and future snapshots.
307	307	*/
308		−int rcu_dynticks_snap(struct rcu_data *rdp)
	308	+static int rcu_dynticks_snap(struct rcu_data *rdp)
309	309	{
310	310	int snap = atomic_add_return(0, &rdp->dynticks);
311	311
@@ -528,16 +528,6 @@ static struct rcu_node *rcu_get_root(void) · 216 unmodified lines
528	528	return &rcu_state.node[0];
529	529	}
530	530
531		−/*
532		− * Convert a ->gp_state value to a character string.
533		− */
534		−static const char *gp_state_getname(short gs)
535		−{
536		− if (gs < 0 \|\| gs >= ARRAY_SIZE(gp_state_names))
537		− return "???";
538		− return gp_state_names[gs];
539		−}
540		−
541	531	/*
542	532	* Send along grace-period-related data for rcutorture diagnostics.
543	533	*/
@@ -577,7 +567,7 @@ static void rcu_eqs_enter(bool user) · 33 unmodified lines
577	567	}
578	568
579	569	lockdep_assert_irqs_disabled();
580		− trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, rdp->dynticks);
	570	+ trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks));
581	571	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
582	572	rdp = this_cpu_ptr(&rcu_data);
583	573	do_nocb_deferred_wakeup(rdp);
@@ -650,14 +640,15 @@ static __always_inline void rcu_nmi_exit_common(bool irq) · 66 unmodified lines
650	640	* leave it in non-RCU-idle state.
651	641	*/
652	642	if (rdp->dynticks_nmi_nesting != 1) {
653		− trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2, rdp->dynticks);
	643	+ trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2,
	644	+ atomic_read(&rdp->dynticks));
654	645	WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */
655	646	rdp->dynticks_nmi_nesting - 2);
656	647	return;
657	648	}
658	649
659	650	/* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */
660		− trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, rdp->dynticks);
	651	+ trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, atomic_read(&rdp->dynticks));
661	652	WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */
662	653
663	654	if (irq)
@@ -744,7 +735,7 @@ static void rcu_eqs_exit(bool user) · 80 unmodified lines
744	735	rcu_dynticks_task_exit();
745	736	rcu_dynticks_eqs_exit();
746	737	rcu_cleanup_after_idle();
747		− trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, rdp->dynticks);
	738	+ trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, atomic_read(&rdp->dynticks));
748	739	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
749	740	WRITE_ONCE(rdp->dynticks_nesting, 1);
750	741	WARN_ON_ONCE(rdp->dynticks_nmi_nesting);
@@ -800,8 +791,8 @@ void rcu_user_exit(void) · 49 unmodified lines
800	791	*/
801	792	static __always_inline void rcu_nmi_enter_common(bool irq)
802	793	{
803		− struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
804	794	long incby = 2;
	795	+ struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
805	796
806	797	/* Complain about underflow. */
807	798	WARN_ON_ONCE(rdp->dynticks_nmi_nesting < 0);
@@ -828,12 +819,17 @@ static __always_inline void rcu_nmi_enter_common(bool irq) · 20 unmodified lines
828	819	} else if (tick_nohz_full_cpu(rdp->cpu) &&
829	820	rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&
830	821	READ_ONCE(rdp->rcu_urgent_qs) && !rdp->rcu_forced_tick) {
831		− rdp->rcu_forced_tick = true;
832		− tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
	822	+ raw_spin_lock_rcu_node(rdp->mynode);
	823	+ // Recheck under lock.
	824	+ if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
	825	+ rdp->rcu_forced_tick = true;
	826	+ tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
	827	+ }
	828	+ raw_spin_unlock_rcu_node(rdp->mynode);
833	829	}
834	830	trace_rcu_dyntick(incby == 1 ? TPS("Endirq") : TPS("++="),
835	831	rdp->dynticks_nmi_nesting,
836		− rdp->dynticks_nmi_nesting + incby, rdp->dynticks);
	832	+ rdp->dynticks_nmi_nesting + incby, atomic_read(&rdp->dynticks));
837	833	WRITE_ONCE(rdp->dynticks_nmi_nesting, /* Prevent store tearing. */
838	834	rdp->dynticks_nmi_nesting + incby);
839	835	barrier();
@@ -898,6 +894,7 @@ void rcu_irq_enter_irqson(void) · 58 unmodified lines
898	894	*/
899	895	static void rcu_disable_urgency_upon_qs(struct rcu_data *rdp)
900	896	{
	897	+ raw_lockdep_assert_held_rcu_node(rdp->mynode);
901	898	WRITE_ONCE(rdp->rcu_urgent_qs, false);
902	899	WRITE_ONCE(rdp->rcu_need_heavy_qs, false);
903	900	if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick) {
@@ -1934,7 +1931,7 @@ rcu_report_unblock_qs_rnp(struct rcu_node *rnp, unsigned long flags) · 1030 unmodified lines
1934	1931	struct rcu_node *rnp_p;
1935	1932
1936	1933	raw_lockdep_assert_held_rcu_node(rnp);
1937		− if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_PREEMPTION)) \|\|
	1934	+ if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_PREEMPT_RCU)) \|\|
1938	1935	WARN_ON_ONCE(rcu_preempt_blocked_readers_cgp(rnp)) \|\|
1939	1936	rnp->qsmask != 0) {
1940	1937	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
@@ -2146,7 +2143,6 @@ static void rcu_do_batch(struct rcu_data *rdp) · 205 unmodified lines
2146	2143	/* If no callbacks are ready, just return. */
2147	2144	if (!rcu_segcblist_ready_cbs(&rdp->cblist)) {
2148	2145	trace_rcu_batch_start(rcu_state.name,
2149		− rcu_segcblist_n_lazy_cbs(&rdp->cblist),
2150	2146	rcu_segcblist_n_cbs(&rdp->cblist), 0);
2151	2147	trace_rcu_batch_end(rcu_state.name, 0,
2152	2148	!rcu_segcblist_empty(&rdp->cblist),
@@ -2168,7 +2164,6 @@ static void rcu_do_batch(struct rcu_data *rdp) · 15 unmodified lines
2168	2164	if (unlikely(bl > 100))
2169	2165	tlimit = local_clock() + rcu_resched_ns;
2170	2166	trace_rcu_batch_start(rcu_state.name,
2171		− rcu_segcblist_n_lazy_cbs(&rdp->cblist),
2172	2167	rcu_segcblist_n_cbs(&rdp->cblist), bl);
2173	2168	rcu_segcblist_extract_done_cbs(&rdp->cblist, &rcl);
2174	2169	if (offloaded)
@@ -2179,9 +2174,19 @@ static void rcu_do_batch(struct rcu_data *rdp) · 4 unmodified lines
2179	2174	tick_dep_set_task(current, TICK_DEP_BIT_RCU);
2180	2175	rhp = rcu_cblist_dequeue(&rcl);
2181	2176	for (; rhp; rhp = rcu_cblist_dequeue(&rcl)) {
	2177	+ rcu_callback_t f;
	2178	+
2182	2179	debug_rcu_head_unqueue(rhp);
2183		− if (__rcu_reclaim(rcu_state.name, rhp))
2184		− rcu_cblist_dequeued_lazy(&rcl);
	2180	+
	2181	+ rcu_lock_acquire(&rcu_callback_map);
	2182	+ trace_rcu_invoke_callback(rcu_state.name, rhp);
	2183	+
	2184	+ f = rhp->func;
	2185	+ WRITE_ONCE(rhp->func, (rcu_callback_t)0L);
	2186	+ f(rhp);
	2187	+
	2188	+ rcu_lock_release(&rcu_callback_map);
	2189	+
2185	2190	/*
2186	2191	* Stop only if limit reached and CPU has something to do.
2187	2192	* Note: The rcl structure counts down from zero.
@@ -2294,7 +2299,7 @@ static void force_qs_rnp(int (f)(struct rcu_data rdp)) · 106 unmodified lines
2294	2299	mask = 0;
2295	2300	raw_spin_lock_irqsave_rcu_node(rnp, flags);
2296	2301	if (rnp->qsmask == 0) {
2297		− if (!IS_ENABLED(CONFIG_PREEMPTION) \|\|
	2302	+ if (!IS_ENABLED(CONFIG_PREEMPT_RCU) \|\|
2298	2303	rcu_preempt_blocked_readers_cgp(rnp)) {
2299	2304	/*
2300	2305	* No point in scanning bits because they
@@ -2308,14 +2313,11 @@ static void force_qs_rnp(int (f)(struct rcu_data rdp)) · 7 unmodified lines
2308	2313	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
2309	2314	continue;
2310	2315	}
2311		− for_each_leaf_node_possible_cpu(rnp, cpu) {
2312		− unsigned long bit = leaf_node_cpu_bit(rnp, cpu);
2313		− if ((rnp->qsmask & bit) != 0) {
2314		− rdp = per_cpu_ptr(&rcu_data, cpu);
2315		− if (f(rdp)) {
2316		− mask \|= bit;
2317		− rcu_disable_urgency_upon_qs(rdp);
2318		− }
	2316	+ for_each_leaf_node_cpu_mask(rnp, cpu, rnp->qsmask) {
	2317	+ rdp = per_cpu_ptr(&rcu_data, cpu);
	2318	+ if (f(rdp)) {
	2319	+ mask \|= rdp->grpmask;
	2320	+ rcu_disable_urgency_upon_qs(rdp);
2319	2321	}
2320	2322	}
2321	2323	if (mask != 0) {
@@ -2474,8 +2476,8 @@ static void rcu_cpu_kthread(unsigned int cpu) · 152 unmodified lines
2474	2476	char work, *workp = this_cpu_ptr(&rcu_data.rcu_cpu_has_work);
2475	2477	int spincnt;
2476	2478
	2479	+ trace_rcu_utilization(TPS("Start CPU kthread@rcu_run"));
2477	2480	for (spincnt = 0; spincnt < 10; spincnt++) {
2478		− trace_rcu_utilization(TPS("Start CPU kthread@rcu_wait"));
2479	2481	local_bh_disable();
2480	2482	*statusp = RCU_KTHREAD_RUNNING;
2481	2483	local_irq_disable();
@@ -2583,7 +2585,7 @@ static void rcu_leak_callback(struct rcu_head *rhp) · 101 unmodified lines
2583	2585	* is expected to specify a CPU.
2584	2586	*/
2585	2587	static void
2586		−__call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy)
	2588	+__call_rcu(struct rcu_head *head, rcu_callback_t func)
2587	2589	{
2588	2590	unsigned long flags;
2589	2591	struct rcu_data *rdp;
@@ -2618,18 +2620,17 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy) · 28 unmodified lines
2618	2620	if (rcu_segcblist_empty(&rdp->cblist))
2619	2621	rcu_segcblist_init(&rdp->cblist);
2620	2622	}
	2623	+
2621	2624	if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags))
2622	2625	return; // Enqueued onto ->nocb_bypass, so just leave.
2623	2626	/* If we get here, rcu_nocb_try_bypass() acquired ->nocb_lock. */
2624		− rcu_segcblist_enqueue(&rdp->cblist, head, lazy);
	2627	+ rcu_segcblist_enqueue(&rdp->cblist, head);
2625	2628	if (__is_kfree_rcu_offset((unsigned long)func))
2626	2629	trace_rcu_kfree_callback(rcu_state.name, head,
2627	2630	(unsigned long)func,
2628		− rcu_segcblist_n_lazy_cbs(&rdp->cblist),
2629	2631	rcu_segcblist_n_cbs(&rdp->cblist));
2630	2632	else
2631	2633	trace_rcu_callback(rcu_state.name, head,
2632		− rcu_segcblist_n_lazy_cbs(&rdp->cblist),
2633	2634	rcu_segcblist_n_cbs(&rdp->cblist));
2634	2635
2635	2636	/* Go handle any RCU core processing required. */
@@ -2679,28 +2680,230 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy) · 43 unmodified lines
2679	2680	*/
2680	2681	void call_rcu(struct rcu_head *head, rcu_callback_t func)
2681	2682	{
2682		− __call_rcu(head, func, 0);
	2683	+ __call_rcu(head, func);
2683	2684	}
2684	2685	EXPORT_SYMBOL_GPL(call_rcu);
2685	2686
	2687	+
	2688	+/* Maximum number of jiffies to wait before draining a batch. */
	2689	+#define KFREE_DRAIN_JIFFIES (HZ / 50)
	2690	+#define KFREE_N_BATCHES 2
	2691	+
	2692	+/**
	2693	+ * struct kfree_rcu_cpu_work - single batch of kfree_rcu() requests
	2694	+ * @rcu_work: Let queue_rcu_work() invoke workqueue handler after grace period
	2695	+ * @head_free: List of kfree_rcu() objects waiting for a grace period
	2696	+ * @krcp: Pointer to @kfree_rcu_cpu structure
	2697	+ */
	2698	+
	2699	+struct kfree_rcu_cpu_work {
	2700	+ struct rcu_work rcu_work;
	2701	+ struct rcu_head *head_free;
	2702	+ struct kfree_rcu_cpu *krcp;
	2703	+};
	2704	+
	2705	+/**
	2706	+ * struct kfree_rcu_cpu - batch up kfree_rcu() requests for RCU grace period
	2707	+ * @head: List of kfree_rcu() objects not yet waiting for a grace period
	2708	+ * @krw_arr: Array of batches of kfree_rcu() objects waiting for a grace period
	2709	+ * @lock: Synchronize access to this structure
	2710	+ * @monitor_work: Promote @head to @head_free after KFREE_DRAIN_JIFFIES
	2711	+ * @monitor_todo: Tracks whether a @monitor_work delayed work is pending
	2712	+ * @initialized: The @lock and @rcu_work fields have been initialized
	2713	+ *
	2714	+ * This is a per-CPU structure. The reason that it is not included in
	2715	+ * the rcu_data structure is to permit this code to be extracted from
	2716	+ * the RCU files. Such extraction could allow further optimization of
	2717	+ * the interactions with the slab allocators.
	2718	+ */
	2719	+struct kfree_rcu_cpu {
	2720	+ struct rcu_head *head;
	2721	+ struct kfree_rcu_cpu_work krw_arr[KFREE_N_BATCHES];
	2722	+ spinlock_t lock;
	2723	+ struct delayed_work monitor_work;
	2724	+ bool monitor_todo;
	2725	+ bool initialized;
	2726	+};
	2727	+
	2728	+static DEFINE_PER_CPU(struct kfree_rcu_cpu, krc);
	2729	+
	2730	+/*
	2731	+ * This function is invoked in workqueue context after a grace period.
	2732	+ * It frees all the objects queued on ->head_free.
	2733	+ */
	2734	+static void kfree_rcu_work(struct work_struct *work)
	2735	+{
	2736	+ unsigned long flags;
	2737	+ struct rcu_head head, next;
	2738	+ struct kfree_rcu_cpu *krcp;
	2739	+ struct kfree_rcu_cpu_work *krwp;
	2740	+
	2741	+ krwp = container_of(to_rcu_work(work),
	2742	+ struct kfree_rcu_cpu_work, rcu_work);
	2743	+ krcp = krwp->krcp;
	2744	+ spin_lock_irqsave(&krcp->lock, flags);
	2745	+ head = krwp->head_free;
	2746	+ krwp->head_free = NULL;
	2747	+ spin_unlock_irqrestore(&krcp->lock, flags);
	2748	+
	2749	+ // List "head" is now private, so traverse locklessly.
	2750	+ for (; head; head = next) {
	2751	+ unsigned long offset = (unsigned long)head->func;
	2752	+
	2753	+ next = head->next;
	2754	+ // Potentially optimize with kfree_bulk in future.
	2755	+ debug_rcu_head_unqueue(head);
	2756	+ rcu_lock_acquire(&rcu_callback_map);
	2757	+ trace_rcu_invoke_kfree_callback(rcu_state.name, head, offset);
	2758	+
	2759	+ if (!WARN_ON_ONCE(!__is_kfree_rcu_offset(offset))) {
	2760	+ /* Could be optimized with kfree_bulk() in future. */
	2761	+ kfree((void *)head - offset);
	2762	+ }
	2763	+
	2764	+ rcu_lock_release(&rcu_callback_map);
	2765	+ cond_resched_tasks_rcu_qs();
	2766	+ }
	2767	+}
	2768	+
2686	2769	/*
2687		− * Queue an RCU callback for lazy invocation after a grace period.
2688		− * This will likely be later named something like "call_rcu_lazy()",
2689		− * but this change will require some way of tagging the lazy RCU
2690		− * callbacks in the list of pending callbacks. Until then, this
2691		− * function may only be called from __kfree_rcu().
	2770	+ * Schedule the kfree batch RCU work to run in workqueue context after a GP.
	2771	+ *
	2772	+ * This function is invoked by kfree_rcu_monitor() when the KFREE_DRAIN_JIFFIES
	2773	+ * timeout has been reached.
	2774	+ */
	2775	+static inline bool queue_kfree_rcu_work(struct kfree_rcu_cpu *krcp)
	2776	+{
	2777	+ int i;
	2778	+ struct kfree_rcu_cpu_work *krwp = NULL;
	2779	+
	2780	+ lockdep_assert_held(&krcp->lock);
	2781	+ for (i = 0; i < KFREE_N_BATCHES; i++)
	2782	+ if (!krcp->krw_arr[i].head_free) {
	2783	+ krwp = &(krcp->krw_arr[i]);
	2784	+ break;
	2785	+ }
	2786	+
	2787	+ // If a previous RCU batch is in progress, we cannot immediately
	2788	+ // queue another one, so return false to tell caller to retry.
	2789	+ if (!krwp)
	2790	+ return false;
	2791	+
	2792	+ krwp->head_free = krcp->head;
	2793	+ krcp->head = NULL;
	2794	+ INIT_RCU_WORK(&krwp->rcu_work, kfree_rcu_work);
	2795	+ queue_rcu_work(system_wq, &krwp->rcu_work);
	2796	+ return true;
	2797	+}
	2798	+
	2799	+static inline void kfree_rcu_drain_unlock(struct kfree_rcu_cpu *krcp,
	2800	+ unsigned long flags)
	2801	+{
	2802	+ // Attempt to start a new batch.
	2803	+ krcp->monitor_todo = false;
	2804	+ if (queue_kfree_rcu_work(krcp)) {
	2805	+ // Success! Our job is done here.
	2806	+ spin_unlock_irqrestore(&krcp->lock, flags);
	2807	+ return;
	2808	+ }
	2809	+
	2810	+ // Previous RCU batch still in progress, try again later.
	2811	+ krcp->monitor_todo = true;
	2812	+ schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES);
	2813	+ spin_unlock_irqrestore(&krcp->lock, flags);
	2814	+}
	2815	+
	2816	+/*
	2817	+ * This function is invoked after the KFREE_DRAIN_JIFFIES timeout.
	2818	+ * It invokes kfree_rcu_drain_unlock() to attempt to start another batch.
	2819	+ */
	2820	+static void kfree_rcu_monitor(struct work_struct *work)
	2821	+{
	2822	+ unsigned long flags;
	2823	+ struct kfree_rcu_cpu *krcp = container_of(work, struct kfree_rcu_cpu,
	2824	+ monitor_work.work);
	2825	+
	2826	+ spin_lock_irqsave(&krcp->lock, flags);
	2827	+ if (krcp->monitor_todo)
	2828	+ kfree_rcu_drain_unlock(krcp, flags);
	2829	+ else
	2830	+ spin_unlock_irqrestore(&krcp->lock, flags);
	2831	+}
	2832	+
	2833	+/*
	2834	+ * Queue a request for lazy invocation of kfree() after a grace period.
	2835	+ *
	2836	+ * Each kfree_call_rcu() request is added to a batch. The batch will be drained
	2837	+ * every KFREE_DRAIN_JIFFIES number of jiffies. All the objects in the batch
	2838	+ * will be kfree'd in workqueue context. This allows us to:
	2839	+ *
	2840	+ * 1. Batch requests together to reduce the number of grace periods during
	2841	+ * heavy kfree_rcu() load.
	2842	+ *
	2843	+ * 2. It makes it possible to use kfree_bulk() on a large number of
	2844	+ * kfree_rcu() requests thus reducing cache misses and the per-object
	2845	+ * overhead of kfree().
2692	2846	*/
2693	2847	void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func)
2694	2848	{
2695		− __call_rcu(head, func, 1);
	2849	+ unsigned long flags;
	2850	+ struct kfree_rcu_cpu *krcp;
	2851	+
	2852	+ local_irq_save(flags); // For safely calling this_cpu_ptr().
	2853	+ krcp = this_cpu_ptr(&krc);
	2854	+ if (krcp->initialized)
	2855	+ spin_lock(&krcp->lock);
	2856	+
	2857	+ // Queue the object but don't yet schedule the batch.
	2858	+ if (debug_rcu_head_queue(head)) {
	2859	+ // Probable double kfree_rcu(), just leak.
	2860	+ WARN_ONCE(1, "%s(): Double-freed call. rcu_head %p\n",
	2861	+ __func__, head);
	2862	+ goto unlock_return;
	2863	+ }
	2864	+ head->func = func;
	2865	+ head->next = krcp->head;
	2866	+ krcp->head = head;
	2867	+
	2868	+ // Set timer to drain after KFREE_DRAIN_JIFFIES.
	2869	+ if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING &&
	2870	+ !krcp->monitor_todo) {
	2871	+ krcp->monitor_todo = true;
	2872	+ schedule_delayed_work(&krcp->monitor_work, KFREE_DRAIN_JIFFIES);
	2873	+ }
	2874	+
	2875	+unlock_return:
	2876	+ if (krcp->initialized)
	2877	+ spin_unlock(&krcp->lock);
	2878	+ local_irq_restore(flags);
2696	2879	}
2697	2880	EXPORT_SYMBOL_GPL(kfree_call_rcu);
2698	2881
	2882	+void __init kfree_rcu_scheduler_running(void)
	2883	+{
	2884	+ int cpu;
	2885	+ unsigned long flags;
	2886	+
	2887	+ for_each_online_cpu(cpu) {
	2888	+ struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
	2889	+
	2890	+ spin_lock_irqsave(&krcp->lock, flags);
	2891	+ if (!krcp->head \|\| krcp->monitor_todo) {
	2892	+ spin_unlock_irqrestore(&krcp->lock, flags);
	2893	+ continue;
	2894	+ }
	2895	+ krcp->monitor_todo = true;
	2896	+ schedule_delayed_work_on(cpu, &krcp->monitor_work,
	2897	+ KFREE_DRAIN_JIFFIES);
	2898	+ spin_unlock_irqrestore(&krcp->lock, flags);
	2899	+ }
	2900	+}
	2901	+
2699	2902	/*
2700	2903	* During early boot, any blocking grace-period wait automatically
2701		− * implies a grace period. Later on, this is never the case for PREEMPT.
	2904	+ * implies a grace period. Later on, this is never the case for PREEMPTION.
2702	2905	*
2703		− * Howevr, because a context switch is a grace period for !PREEMPT, any
	2906	+ * Howevr, because a context switch is a grace period for !PREEMPTION, any
2704	2907	* blocking grace-period wait automatically implies a grace period if
2705	2908	* there is only one CPU online at any point time during execution of
2706	2909	* either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to
@@ -2896,7 +3099,7 @@ static void rcu_barrier_func(void *unused) · 189 unmodified lines
2896	3099	debug_rcu_head_queue(&rdp->barrier_head);
2897	3100	rcu_nocb_lock(rdp);
2898	3101	WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies));
2899		− if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head, 0)) {
	3102	+ if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) {
2900	3103	atomic_inc(&rcu_state.barrier_cpu_count);
2901	3104	} else {
2902	3105	debug_rcu_head_unqueue(&rdp->barrier_head);
@@ -3557,12 +3760,29 @@ static void __init rcu_dump_rcu_node_tree(void) · 654 unmodified lines
3557	3760	struct workqueue_struct *rcu_gp_wq;
3558	3761	struct workqueue_struct *rcu_par_gp_wq;
3559	3762
	3763	+static void __init kfree_rcu_batch_init(void)
	3764	+{
	3765	+ int cpu;
	3766	+ int i;
	3767	+
	3768	+ for_each_possible_cpu(cpu) {
	3769	+ struct kfree_rcu_cpu *krcp = per_cpu_ptr(&krc, cpu);
	3770	+
	3771	+ spin_lock_init(&krcp->lock);
	3772	+ for (i = 0; i < KFREE_N_BATCHES; i++)
	3773	+ krcp->krw_arr[i].krcp = krcp;
	3774	+ INIT_DELAYED_WORK(&krcp->monitor_work, kfree_rcu_monitor);
	3775	+ krcp->initialized = true;
	3776	+ }
	3777	+}
	3778	+
3560	3779	void __init rcu_init(void)
3561	3780	{
3562	3781	int cpu;
3563	3782
3564	3783	rcu_early_boot_tests();
3565	3784
	3785	+ kfree_rcu_batch_init();
3566	3786	rcu_bootup_announce();
3567	3787	rcu_init_geometry();
3568	3788	rcu_init_one();

kernel/rcu/tree.h+1−17

@@ -16,7 +16,6 @@ · 15 unmodified lines
16	16	#include <linux/cpumask.h>
17	17	#include <linux/seqlock.h>
18	18	#include <linux/swait.h>
19		−#include <linux/stop_machine.h>
20	19	#include <linux/rcu_node_tree.h>
21	20
22	21	#include "rcu_segcblist.h"
@@ -182,8 +181,8 @@ struct rcu_data { · 159 unmodified lines
182	181	bool rcu_need_heavy_qs; /* GP old, so heavy quiescent state! */
183	182	bool rcu_urgent_qs; /* GP old need light quiescent state. */
184	183	bool rcu_forced_tick; /* Forced tick to provide QS. */
	184	+ bool rcu_forced_tick_exp; /* ... provide QS to expedited GP. */
185	185	#ifdef CONFIG_RCU_FAST_NO_HZ
186		− bool all_lazy; /* All CPU's CBs lazy at idle start? */
187	186	unsigned long last_accelerate; /* Last jiffy CBs were accelerated. */
188	187	unsigned long last_advance_all; /* Last jiffy CBs were all advanced. */
189	188	int tick_nohz_enabled_snap; /* Previously seen value from sysfs. */
@@ -368,18 +367,6 @@ struct rcu_state { · 178 unmodified lines
368	367	#define RCU_GP_CLEANUP 7 /* Grace-period cleanup started. */
369	368	#define RCU_GP_CLEANED 8 /* Grace-period cleanup complete. */
370	369
371		−static const char * const gp_state_names[] = {
372		− "RCU_GP_IDLE",
373		− "RCU_GP_WAIT_GPS",
374		− "RCU_GP_DONE_GPS",
375		− "RCU_GP_ONOFF",
376		− "RCU_GP_INIT",
377		− "RCU_GP_WAIT_FQS",
378		− "RCU_GP_DOING_FQS",
379		− "RCU_GP_CLEANUP",
380		− "RCU_GP_CLEANED",
381		−};
382		−
383	370	/*
384	371	* In order to export the rcu_state name to the tracing tools, it
385	372	* needs to be added in the __tracepoint_string section.
@@ -403,8 +390,6 @@ static const char *tp_rcu_varname __used __tracepoint_string = rcu_name; · 17 unmodified lines
403	390	#define RCU_NAME rcu_name
404	391	#endif /* #else #ifdef CONFIG_TRACING */
405	392
406		−int rcu_dynticks_snap(struct rcu_data *rdp);
407		−
408	393	/* Forward declarations for tree_plugin.h */
409	394	static void rcu_bootup_announce(void);
410	395	static void rcu_qs(void);
@@ -415,7 +400,6 @@ static bool rcu_preempt_has_tasks(struct rcu_node *rnp); · 4 unmodified lines
415	400	static int rcu_print_task_exp_stall(struct rcu_node *rnp);
416	401	static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp);
417	402	static void rcu_flavor_sched_clock_irq(int user);
418		−void call_rcu(struct rcu_head *head, rcu_callback_t func);
419	403	static void dump_blkd_tasks(struct rcu_node *rnp, int ncheck);
420	404	static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags);
421	405	static void rcu_preempt_boost_start_gp(struct rcu_node *rnp);

kernel/rcu/tree_exp.h+91−56

@@ -21,7 +21,7 @@ static void rcu_exp_gp_seq_start(void) · 20 unmodified lines
21	21	}
22	22
23	23	/*
24		− * Return then value that expedited-grace-period counter will have
	24	+ * Return the value that the expedited-grace-period counter will have
25	25	* at the end of the current grace period.
26	26	*/
27	27	static __maybe_unused unsigned long rcu_exp_gp_seq_endval(void)
@@ -39,7 +39,9 @@ static void rcu_exp_gp_seq_end(void) · 11 unmodified lines
39	39	}
40	40
41	41	/*
42		− * Take a snapshot of the expedited-grace-period counter.
	42	+ * Take a snapshot of the expedited-grace-period counter, which is the
	43	+ * earliest value that will indicate that a full grace period has
	44	+ * elapsed since the current time.
43	45	*/
44	46	static unsigned long rcu_exp_gp_seq_snap(void)
45	47	{
@@ -134,7 +136,7 @@ static void __maybe_unused sync_exp_reset_tree(void) · 88 unmodified lines
134	136	rcu_for_each_node_breadth_first(rnp) {
135	137	raw_spin_lock_irqsave_rcu_node(rnp, flags);
136	138	WARN_ON_ONCE(rnp->expmask);
137		− rnp->expmask = rnp->expmaskinit;
	139	+ WRITE_ONCE(rnp->expmask, rnp->expmaskinit);
138	140	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
139	141	}
140	142	}
@@ -143,31 +145,26 @@ static void __maybe_unused sync_exp_reset_tree(void) · 2 unmodified lines
143	145	* Return non-zero if there is no RCU expedited grace period in progress
144	146	* for the specified rcu_node structure, in other words, if all CPUs and
145	147	* tasks covered by the specified rcu_node structure have done their bit
146		− * for the current expedited grace period. Works only for preemptible
147		− * RCU -- other RCU implementation use other means.
148		− *
149		− * Caller must hold the specificed rcu_node structure's ->lock
	148	+ * for the current expedited grace period.
150	149	*/
151		−static bool sync_rcu_preempt_exp_done(struct rcu_node *rnp)
	150	+static bool sync_rcu_exp_done(struct rcu_node *rnp)
152	151	{
153	152	raw_lockdep_assert_held_rcu_node(rnp);
154		−
155	153	return rnp->exp_tasks == NULL &&
156	154	READ_ONCE(rnp->expmask) == 0;
157	155	}
158	156
159	157	/*
160		− * Like sync_rcu_preempt_exp_done(), but this function assumes the caller
161		− * doesn't hold the rcu_node's ->lock, and will acquire and release the lock
162		− * itself
	158	+ * Like sync_rcu_exp_done(), but where the caller does not hold the
	159	+ * rcu_node's ->lock.
163	160	*/
164		−static bool sync_rcu_preempt_exp_done_unlocked(struct rcu_node *rnp)
	161	+static bool sync_rcu_exp_done_unlocked(struct rcu_node *rnp)
165	162	{
166	163	unsigned long flags;
167	164	bool ret;
168	165
169	166	raw_spin_lock_irqsave_rcu_node(rnp, flags);
170		− ret = sync_rcu_preempt_exp_done(rnp);
	167	+ ret = sync_rcu_exp_done(rnp);
171	168	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
172	169
173	170	return ret;
@@ -181,8 +178,6 @@ static bool sync_rcu_preempt_exp_done_unlocked(struct rcu_node *rnp) · 7 unmodified lines
181	178	* which the task was queued or to one of that rcu_node structure's ancestors,
182	179	* recursively up the tree. (Calm down, calm down, we do the recursion
183	180	* iteratively!)
184		− *
185		− * Caller must hold the specified rcu_node structure's ->lock.
186	181	*/
187	182	static void __rcu_report_exp_rnp(struct rcu_node *rnp,
188	183	bool wake, unsigned long flags)
@@ -190,8 +185,9 @@ static void __rcu_report_exp_rnp(struct rcu_node *rnp, · 1 unmodified line
190	185	{
191	186	unsigned long mask;
192	187
	188	+ raw_lockdep_assert_held_rcu_node(rnp);
193	189	for (;;) {
194		− if (!sync_rcu_preempt_exp_done(rnp)) {
	190	+ if (!sync_rcu_exp_done(rnp)) {
195	191	if (!rnp->expmask)
196	192	rcu_initiate_boost(rnp, flags);
197	193	else
@@ -211,7 +207,7 @@ static void __rcu_report_exp_rnp(struct rcu_node *rnp, · 13 unmodified lines
211	207	rnp = rnp->parent;
212	208	raw_spin_lock_rcu_node(rnp); /* irqs already disabled */
213	209	WARN_ON_ONCE(!(rnp->expmask & mask));
214		− rnp->expmask &= ~mask;
	210	+ WRITE_ONCE(rnp->expmask, rnp->expmask & ~mask);
215	211	}
216	212	}
217	213
@@ -234,14 +230,23 @@ static void __maybe_unused rcu_report_exp_rnp(struct rcu_node *rnp, bool wake) · 16 unmodified lines
234	230	static void rcu_report_exp_cpu_mult(struct rcu_node *rnp,
235	231	unsigned long mask, bool wake)
236	232	{
	233	+ int cpu;
237	234	unsigned long flags;
	235	+ struct rcu_data *rdp;
238	236
239	237	raw_spin_lock_irqsave_rcu_node(rnp, flags);
240	238	if (!(rnp->expmask & mask)) {
241	239	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
242	240	return;
243	241	}
244		− rnp->expmask &= ~mask;
	242	+ WRITE_ONCE(rnp->expmask, rnp->expmask & ~mask);
	243	+ for_each_leaf_node_cpu_mask(rnp, cpu, mask) {
	244	+ rdp = per_cpu_ptr(&rcu_data, cpu);
	245	+ if (!IS_ENABLED(CONFIG_NO_HZ_FULL) \|\| !rdp->rcu_forced_tick_exp)
	246	+ continue;
	247	+ rdp->rcu_forced_tick_exp = false;
	248	+ tick_dep_clear_cpu(cpu, TICK_DEP_BIT_RCU_EXP);
	249	+ }
245	250	__rcu_report_exp_rnp(rnp, wake, flags); /* Releases rnp->lock. */
246	251	}
247	252
@@ -345,8 +350,8 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) · 97 unmodified lines
345	350	/* Each pass checks a CPU for identity, offline, and idle. */
346	351	mask_ofl_test = 0;
347	352	for_each_leaf_node_cpu_mask(rnp, cpu, rnp->expmask) {
348		− unsigned long mask = leaf_node_cpu_bit(rnp, cpu);
349	353	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
	354	+ unsigned long mask = rdp->grpmask;
350	355	int snap;
351	356
352	357	if (raw_smp_processor_id() == cpu \|\|
@@ -372,12 +377,10 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) · 19 unmodified lines
372	377	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
373	378
374	379	/* IPI the remaining CPUs for expedited quiescent state. */
375		− for_each_leaf_node_cpu_mask(rnp, cpu, rnp->expmask) {
376		− unsigned long mask = leaf_node_cpu_bit(rnp, cpu);
	380	+ for_each_leaf_node_cpu_mask(rnp, cpu, mask_ofl_ipi) {
377	381	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
	382	+ unsigned long mask = rdp->grpmask;
378	383
379		− if (!(mask_ofl_ipi & mask))
380		− continue;
381	384	retry_ipi:
382	385	if (rcu_dynticks_in_eqs_since(rdp, rdp->exp_dynticks_snap)) {
383	386	mask_ofl_test \|= mask;
@@ -389,10 +392,10 @@ retry_ipi: · 5 unmodified lines
389	392	}
390	393	ret = smp_call_function_single(cpu, rcu_exp_handler, NULL, 0);
391	394	put_cpu();
392		− if (!ret) {
393		− mask_ofl_ipi &= ~mask;
	395	+ /* The CPU will report the QS in response to the IPI. */
	396	+ if (!ret)
394	397	continue;
395		− }
	398	+
396	399	/* Failed, raced with CPU hotplug operation. */
397	400	raw_spin_lock_irqsave_rcu_node(rnp, flags);
398	401	if ((rnp->qsmaskinitnext & mask) &&
@@ -403,13 +406,12 @@ retry_ipi: · 4 unmodified lines
403	406	schedule_timeout_uninterruptible(1);
404	407	goto retry_ipi;
405	408	}
406		− /* CPU really is offline, so we can ignore it. */
407		− if (!(rnp->expmask & mask))
408		− mask_ofl_ipi &= ~mask;
	409	+ /* CPU really is offline, so we must report its QS. */
	410	+ if (rnp->expmask & mask)
	411	+ mask_ofl_test \|= mask;
409	412	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
410	413	}
411	414	/* Report quiescent states for those that went offline. */
412		− mask_ofl_test \|= mask_ofl_ipi;
413	415	if (mask_ofl_test)
414	416	rcu_report_exp_cpu_mult(rnp, mask_ofl_test, false);
415	417	}
@@ -456,29 +458,62 @@ static void sync_rcu_exp_select_cpus(void) · 40 unmodified lines
456	458	flush_work(&rnp->rew.rew_work);
457	459	}
458	460
459		−static void synchronize_sched_expedited_wait(void)
	461	+/*
	462	+ * Wait for the expedited grace period to elapse, within time limit.
	463	+ * If the time limit is exceeded without the grace period elapsing,
	464	+ * return false, otherwise return true.
	465	+ */
	466	+static bool synchronize_rcu_expedited_wait_once(long tlimit)
	467	+{
	468	+ int t;
	469	+ struct rcu_node *rnp_root = rcu_get_root();
	470	+
	471	+ t = swait_event_timeout_exclusive(rcu_state.expedited_wq,
	472	+ sync_rcu_exp_done_unlocked(rnp_root),
	473	+ tlimit);
	474	+ // Workqueues should not be signaled.
	475	+ if (t > 0 \|\| sync_rcu_exp_done_unlocked(rnp_root))
	476	+ return true;
	477	+ WARN_ON(t < 0); /* workqueues should not be signaled. */
	478	+ return false;
	479	+}
	480	+
	481	+/*
	482	+ * Wait for the expedited grace period to elapse, issuing any needed
	483	+ * RCU CPU stall warnings along the way.
	484	+ */
	485	+static void synchronize_rcu_expedited_wait(void)
460	486	{
461	487	int cpu;
462	488	unsigned long jiffies_stall;
463	489	unsigned long jiffies_start;
464	490	unsigned long mask;
465	491	int ndetected;
	492	+ struct rcu_data *rdp;
466	493	struct rcu_node *rnp;
467	494	struct rcu_node *rnp_root = rcu_get_root();
468		− int ret;
469	495
470	496	trace_rcu_exp_grace_period(rcu_state.name, rcu_exp_gp_seq_endval(), TPS("startwait"));
471	497	jiffies_stall = rcu_jiffies_till_stall_check();
472	498	jiffies_start = jiffies;
	499	+ if (IS_ENABLED(CONFIG_NO_HZ_FULL)) {
	500	+ if (synchronize_rcu_expedited_wait_once(1))
	501	+ return;
	502	+ rcu_for_each_leaf_node(rnp) {
	503	+ for_each_leaf_node_cpu_mask(rnp, cpu, rnp->expmask) {
	504	+ rdp = per_cpu_ptr(&rcu_data, cpu);
	505	+ if (rdp->rcu_forced_tick_exp)
	506	+ continue;
	507	+ rdp->rcu_forced_tick_exp = true;
	508	+ tick_dep_set_cpu(cpu, TICK_DEP_BIT_RCU_EXP);
	509	+ }
	510	+ }
	511	+ WARN_ON_ONCE(1);
	512	+ }
473	513
474	514	for (;;) {
475		− ret = swait_event_timeout_exclusive(
476		− rcu_state.expedited_wq,
477		− sync_rcu_preempt_exp_done_unlocked(rnp_root),
478		− jiffies_stall);
479		− if (ret > 0 \|\| sync_rcu_preempt_exp_done_unlocked(rnp_root))
	515	+ if (synchronize_rcu_expedited_wait_once(jiffies_stall))
480	516	return;
481		− WARN_ON(ret < 0); /* workqueues should not be signaled. */
482	517	if (rcu_cpu_stall_suppress)
483	518	continue;
484	519	panic_on_rcu_stall();
@@ -491,7 +526,7 @@ static void synchronize_sched_expedited_wait(void) · 6 unmodified lines
491	526	struct rcu_data *rdp;
492	527
493	528	mask = leaf_node_cpu_bit(rnp, cpu);
494		− if (!(rnp->expmask & mask))
	529	+ if (!(READ_ONCE(rnp->expmask) & mask))
495	530	continue;
496	531	ndetected++;
497	532	rdp = per_cpu_ptr(&rcu_data, cpu);
@@ -503,17 +538,18 @@ static void synchronize_sched_expedited_wait(void) · 5 unmodified lines
503	538	}
504	539	pr_cont(" } %lu jiffies s: %lu root: %#lx/%c\n",
505	540	jiffies - jiffies_start, rcu_state.expedited_sequence,
506		− rnp_root->expmask, ".T"[!!rnp_root->exp_tasks]);
	541	+ READ_ONCE(rnp_root->expmask),
	542	+ ".T"[!!rnp_root->exp_tasks]);
507	543	if (ndetected) {
508	544	pr_err("blocking rcu_node structures:");
509	545	rcu_for_each_node_breadth_first(rnp) {
510	546	if (rnp == rnp_root)
511	547	continue; /* printed unconditionally */
512		− if (sync_rcu_preempt_exp_done_unlocked(rnp))
	548	+ if (sync_rcu_exp_done_unlocked(rnp))
513	549	continue;
514	550	pr_cont(" l=%u:%d-%d:%#lx/%c",
515	551	rnp->level, rnp->grplo, rnp->grphi,
516		− rnp->expmask,
	552	+ READ_ONCE(rnp->expmask),
517	553	".T"[!!rnp->exp_tasks]);
518	554	}
519	555	pr_cont("\n");
@@ -521,7 +557,7 @@ static void synchronize_sched_expedited_wait(void) · 1 unmodified line
521	557	rcu_for_each_leaf_node(rnp) {
522	558	for_each_leaf_node_possible_cpu(rnp, cpu) {
523	559	mask = leaf_node_cpu_bit(rnp, cpu);
524		− if (!(rnp->expmask & mask))
	560	+ if (!(READ_ONCE(rnp->expmask) & mask))
525	561	continue;
526	562	dump_cpu_task(cpu);
527	563	}
@@ -540,15 +576,14 @@ static void rcu_exp_wait_wake(unsigned long s) · 12 unmodified lines
540	576	{
541	577	struct rcu_node *rnp;
542	578
543		− synchronize_sched_expedited_wait();
544		− rcu_exp_gp_seq_end();
545		− trace_rcu_exp_grace_period(rcu_state.name, s, TPS("end"));
	579	+ synchronize_rcu_expedited_wait();
546	580
547		− /*
548		− * Switch over to wakeup mode, allowing the next GP, but -only- the
549		− * next GP, to proceed.
550		− */
	581	+ // Switch over to wakeup mode, allowing the next GP to proceed.
	582	+ // End the previous grace period only after acquiring the mutex
	583	+ // to ensure that only one GP runs concurrently with wakeups.
551	584	mutex_lock(&rcu_state.exp_wake_mutex);
	585	+ rcu_exp_gp_seq_end();
	586	+ trace_rcu_exp_grace_period(rcu_state.name, s, TPS("end"));
552	587
553	588	rcu_for_each_node_breadth_first(rnp) {
554	589	if (ULONG_CMP_LT(READ_ONCE(rnp->exp_seq_rq), s)) {
@@ -559,7 +594,7 @@ static void rcu_exp_wait_wake(unsigned long s) · 4 unmodified lines
559	594	spin_unlock(&rnp->exp_lock);
560	595	}
561	596	smp_mb(); /* All above changes before wakeup. */
562		− wake_up_all(&rnp->exp_wq[rcu_seq_ctr(rcu_state.expedited_sequence) & 0x3]);
	597	+ wake_up_all(&rnp->exp_wq[rcu_seq_ctr(s) & 0x3]);
563	598	}
564	599	trace_rcu_exp_grace_period(rcu_state.name, s, TPS("endwake"));
565	600	mutex_unlock(&rcu_state.exp_wake_mutex);
@@ -610,7 +645,7 @@ static void rcu_exp_handler(void *unused) · 44 unmodified lines
610	645	* critical section. If also enabled or idle, immediately
611	646	* report the quiescent state, otherwise defer.
612	647	*/
613		− if (!t->rcu_read_lock_nesting) {
	648	+ if (!rcu_preempt_depth()) {
614	649	if (!(preempt_count() & (PREEMPT_MASK \| SOFTIRQ_MASK)) \|\|
615	650	rcu_dynticks_curr_cpu_in_eqs()) {
616	651	rcu_report_exp_rdp(rdp);
@@ -634,7 +669,7 @@ static void rcu_exp_handler(void *unused) · 17 unmodified lines
634	669	* can have caused this quiescent state to already have been
635	670	* reported, so we really do need to check ->expmask.
636	671	*/
637		− if (t->rcu_read_lock_nesting > 0) {
	672	+ if (rcu_preempt_depth() > 0) {
638	673	raw_spin_lock_irqsave_rcu_node(rnp, flags);
639	674	if (rnp->expmask & rdp->grpmask) {
640	675	rdp->exp_deferred_qs = true;
@@ -670,7 +705,7 @@ static void rcu_exp_handler(void *unused) · 29 unmodified lines
670	705	}
671	706	}
672	707
673		−/* PREEMPT=y, so no PREEMPT=n expedited grace period to clean up after. */
	708	+/* PREEMPTION=y, so no PREEMPTION=n expedited grace period to clean up after. */
674	709	static void sync_sched_exp_online_cleanup(int cpu)
675	710	{
676	711	}
@@ -785,7 +820,7 @@ static int rcu_print_task_exp_stall(struct rcu_node *rnp) · 108 unmodified lines
785	820	* implementations, it is still unfriendly to real-time workloads, so is
786	821	* thus not recommended for any sort of common-case code. In fact, if
787	822	* you are using synchronize_rcu_expedited() in a loop, please restructure
788		− * your code to batch your updates, and then Use a single synchronize_rcu()
	823	+ * your code to batch your updates, and then use a single synchronize_rcu()
789	824	* instead.
790	825	*
791	826	* This has the same semantics as (but is more brutal than) synchronize_rcu().

kernel/rcu/tree_plugin.h+78−90

@@ -220,7 +220,7 @@ static void rcu_preempt_ctxt_queue(struct rcu_node rnp, struct rcu_data rdp) · 219 unmodified lines
220	220	* blocked tasks.
221	221	*/
222	222	if (!rnp->gp_tasks && (blkd_state & RCU_GP_BLKD)) {
223		− rnp->gp_tasks = &t->rcu_node_entry;
	223	+ WRITE_ONCE(rnp->gp_tasks, &t->rcu_node_entry);
224	224	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq);
225	225	}
226	226	if (!rnp->exp_tasks && (blkd_state & RCU_EXP_BLKD))
@@ -290,8 +290,8 @@ void rcu_note_context_switch(bool preempt) · 63 unmodified lines
290	290
291	291	trace_rcu_utilization(TPS("Start context switch"));
292	292	lockdep_assert_irqs_disabled();
293		− WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0);
294		− if (t->rcu_read_lock_nesting > 0 &&
	293	+ WARN_ON_ONCE(!preempt && rcu_preempt_depth() > 0);
	294	+ if (rcu_preempt_depth() > 0 &&
295	295	!t->rcu_read_unlock_special.b.blocked) {
296	296
297	297	/* Possibly blocking in an RCU read-side critical section. */
@@ -340,7 +340,7 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch); · 42 unmodified lines
340	340	*/
341	341	static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp)
342	342	{
343		− return rnp->gp_tasks != NULL;
	343	+ return READ_ONCE(rnp->gp_tasks) != NULL;
344	344	}
345	345
346	346	/* Bias and limit values for ->rcu_read_lock_nesting. */
@@ -348,6 +348,21 @@ static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp) · 1 unmodified line
348	348	#define RCU_NEST_NMAX (-INT_MAX / 2)
349	349	#define RCU_NEST_PMAX (INT_MAX / 2)
350	350
	351	+static void rcu_preempt_read_enter(void)
	352	+{
	353	+ current->rcu_read_lock_nesting++;
	354	+}
	355	+
	356	+static void rcu_preempt_read_exit(void)
	357	+{
	358	+ current->rcu_read_lock_nesting--;
	359	+}
	360	+
	361	+static void rcu_preempt_depth_set(int val)
	362	+{
	363	+ current->rcu_read_lock_nesting = val;
	364	+}
	365	+
351	366	/*
352	367	* Preemptible RCU implementation for rcu_read_lock().
353	368	* Just increment ->rcu_read_lock_nesting, shared state will be updated
@@ -355,9 +370,9 @@ static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp) · 1 unmodified line
355	370	*/
356	371	void __rcu_read_lock(void)
357	372	{
358		− current->rcu_read_lock_nesting++;
	373	+ rcu_preempt_read_enter();
359	374	if (IS_ENABLED(CONFIG_PROVE_LOCKING))
360		− WARN_ON_ONCE(current->rcu_read_lock_nesting > RCU_NEST_PMAX);
	375	+ WARN_ON_ONCE(rcu_preempt_depth() > RCU_NEST_PMAX);
361	376	barrier(); /* critical section after entry code. */
362	377	}
363	378	EXPORT_SYMBOL_GPL(__rcu_read_lock);
@@ -373,19 +388,19 @@ void __rcu_read_unlock(void) · 9 unmodified lines
373	388	{
374	389	struct task_struct *t = current;
375	390
376		− if (t->rcu_read_lock_nesting != 1) {
377		− --t->rcu_read_lock_nesting;
	391	+ if (rcu_preempt_depth() != 1) {
	392	+ rcu_preempt_read_exit();
378	393	} else {
379	394	barrier(); /* critical section before exit code. */
380		− t->rcu_read_lock_nesting = -RCU_NEST_BIAS;
	395	+ rcu_preempt_depth_set(-RCU_NEST_BIAS);
381	396	barrier(); /* assign before ->rcu_read_unlock_special load */
382	397	if (unlikely(READ_ONCE(t->rcu_read_unlock_special.s)))
383	398	rcu_read_unlock_special(t);
384	399	barrier(); /* ->rcu_read_unlock_special load before assign */
385		− t->rcu_read_lock_nesting = 0;
	400	+ rcu_preempt_depth_set(0);
386	401	}
387	402	if (IS_ENABLED(CONFIG_PROVE_LOCKING)) {
388		− int rrln = t->rcu_read_lock_nesting;
	403	+ int rrln = rcu_preempt_depth();
389	404
390	405	WARN_ON_ONCE(rrln < 0 && rrln > RCU_NEST_NMAX);
391	406	}
@@ -444,15 +459,9 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 52 unmodified lines
444	459	local_irq_restore(flags);
445	460	return;
446	461	}
447		− t->rcu_read_unlock_special.b.deferred_qs = false;
448		− if (special.b.need_qs) {
	462	+ t->rcu_read_unlock_special.s = 0;
	463	+ if (special.b.need_qs)
449	464	rcu_qs();
450		− t->rcu_read_unlock_special.b.need_qs = false;
451		− if (!t->rcu_read_unlock_special.s && !rdp->exp_deferred_qs) {
452		− local_irq_restore(flags);
453		− return;
454		− }
455		− }
456	465
457	466	/*
458	467	* Respond to a request by an expedited grace period for a
@@ -460,17 +469,11 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 1 unmodified line
460	469	* tasks are handled when removing the task from the
461	470	* blocked-tasks list below.
462	471	*/
463		− if (rdp->exp_deferred_qs) {
	472	+ if (rdp->exp_deferred_qs)
464	473	rcu_report_exp_rdp(rdp);
465		− if (!t->rcu_read_unlock_special.s) {
466		− local_irq_restore(flags);
467		− return;
468		− }
469		− }
470	474
471	475	/* Clean up if blocked during RCU read-side critical section. */
472	476	if (special.b.blocked) {
473		− t->rcu_read_unlock_special.b.blocked = false;
474	477
475	478	/*
476	479	* Remove this task from the list it blocked on. The task
@@ -485,7 +488,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 8 unmodified lines
485	488	empty_norm = !rcu_preempt_blocked_readers_cgp(rnp);
486	489	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq &&
487	490	(!empty_norm \|\| rnp->qsmask));
488		− empty_exp = sync_rcu_preempt_exp_done(rnp);
	491	+ empty_exp = sync_rcu_exp_done(rnp);
489	492	smp_mb(); /* ensure expedited fastpath sees end of RCU c-s. */
490	493	np = rcu_next_node_entry(t, rnp);
491	494	list_del_init(&t->rcu_node_entry);
@@ -493,7 +496,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 1 unmodified line
493	496	trace_rcu_unlock_preempted_task(TPS("rcu_preempt"),
494	497	rnp->gp_seq, t->pid);
495	498	if (&t->rcu_node_entry == rnp->gp_tasks)
496		− rnp->gp_tasks = np;
	499	+ WRITE_ONCE(rnp->gp_tasks, np);
497	500	if (&t->rcu_node_entry == rnp->exp_tasks)
498	501	rnp->exp_tasks = np;
499	502	if (IS_ENABLED(CONFIG_RCU_BOOST)) {
@@ -509,7 +512,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 9 unmodified lines
509	512	* Note that rcu_report_unblock_qs_rnp() releases rnp->lock,
510	513	* so we must take a snapshot of the expedited state.
511	514	*/
512		− empty_exp_now = sync_rcu_preempt_exp_done(rnp);
	515	+ empty_exp_now = sync_rcu_exp_done(rnp);
513	516	if (!empty_norm && !rcu_preempt_blocked_readers_cgp(rnp)) {
514	517	trace_rcu_quiescent_state_report(TPS("preempt_rcu"),
515	518	rnp->gp_seq,
@@ -551,7 +554,7 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t) · 35 unmodified lines
551	554	{
552	555	return (__this_cpu_read(rcu_data.exp_deferred_qs) \|\|
553	556	READ_ONCE(t->rcu_read_unlock_special.s)) &&
554		− t->rcu_read_lock_nesting <= 0;
	557	+ rcu_preempt_depth() <= 0;
555	558	}
556	559
557	560	/*
@@ -564,16 +567,16 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t) · 6 unmodified lines
564	567	static void rcu_preempt_deferred_qs(struct task_struct *t)
565	568	{
566	569	unsigned long flags;
567		− bool couldrecurse = t->rcu_read_lock_nesting >= 0;
	570	+ bool couldrecurse = rcu_preempt_depth() >= 0;
568	571
569	572	if (!rcu_preempt_need_deferred_qs(t))
570	573	return;
571	574	if (couldrecurse)
572		− t->rcu_read_lock_nesting -= RCU_NEST_BIAS;
	575	+ rcu_preempt_depth_set(rcu_preempt_depth() - RCU_NEST_BIAS);
573	576	local_irq_save(flags);
574	577	rcu_preempt_deferred_qs_irqrestore(t, flags);
575	578	if (couldrecurse)
576		− t->rcu_read_lock_nesting += RCU_NEST_BIAS;
	579	+ rcu_preempt_depth_set(rcu_preempt_depth() + RCU_NEST_BIAS);
577	580	}
578	581
579	582	/*
@@ -610,9 +613,8 @@ static void rcu_read_unlock_special(struct task_struct *t) · 30 unmodified lines
610	613	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
611	614	struct rcu_node *rnp = rdp->mynode;
612	615
613		− t->rcu_read_unlock_special.b.exp_hint = false;
614	616	exp = (t->rcu_blocked_node && t->rcu_blocked_node->exp_tasks) \|\|
615		− (rdp->grpmask & rnp->expmask) \|\|
	617	+ (rdp->grpmask & READ_ONCE(rnp->expmask)) \|\|
616	618	tick_nohz_full_cpu(rdp->cpu);
617	619	// Need to defer quiescent state until everything is enabled.
618	620	if (irqs_were_disabled && use_softirq &&
@@ -640,7 +642,6 @@ static void rcu_read_unlock_special(struct task_struct *t) · 21 unmodified lines
640	642	local_irq_restore(flags);
641	643	return;
642	644	}
643		− WRITE_ONCE(t->rcu_read_unlock_special.b.exp_hint, false);
644	645	rcu_preempt_deferred_qs_irqrestore(t, flags);
645	646	}
646	647
@@ -648,8 +649,7 @@ static void rcu_read_unlock_special(struct task_struct *t) · 1 unmodified line
648	649	* Check that the list of blocked tasks for the newly completed grace
649	650	* period is in fact empty. It is a serious bug to complete a grace
650	651	* period that still has RCU readers blocked! This function must be
651		− * invoked -before- updating this rnp's ->gp_seq, and the rnp's ->lock
652		− * must be held by the caller.
	652	+ * invoked -before- updating this rnp's ->gp_seq.
653	653	*
654	654	* Also, if there are blocked tasks on the list, they automatically
655	655	* block the newly created grace period, so set up ->gp_tasks accordingly.
@@ -659,11 +659,12 @@ static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp) · 3 unmodified lines
659	659	struct task_struct *t;
660	660
661	661	RCU_LOCKDEP_WARN(preemptible(), "rcu_preempt_check_blocked_tasks() invoked with preemption enabled!!!\n");
	662	+ raw_lockdep_assert_held_rcu_node(rnp);
662	663	if (WARN_ON_ONCE(rcu_preempt_blocked_readers_cgp(rnp)))
663	664	dump_blkd_tasks(rnp, 10);
664	665	if (rcu_preempt_has_tasks(rnp) &&
665	666	(rnp->qsmaskinit \|\| rnp->wait_blkd_tasks)) {
666		− rnp->gp_tasks = rnp->blkd_tasks.next;
	667	+ WRITE_ONCE(rnp->gp_tasks, rnp->blkd_tasks.next);
667	668	t = container_of(rnp->gp_tasks, struct task_struct,
668	669	rcu_node_entry);
669	670	trace_rcu_unlock_preempted_task(TPS("rcu_preempt-GPS"),
@@ -686,7 +687,7 @@ static void rcu_flavor_sched_clock_irq(int user) · 16 unmodified lines
686	687	if (user \|\| rcu_is_cpu_rrupt_from_idle()) {
687	688	rcu_note_voluntary_context_switch(current);
688	689	}
689		− if (t->rcu_read_lock_nesting > 0 \|\|
	690	+ if (rcu_preempt_depth() > 0 \|\|
690	691	(preempt_count() & (PREEMPT_MASK \| SOFTIRQ_MASK))) {
691	692	/* No QS, force context switch if deferred. */
692	693	if (rcu_preempt_need_deferred_qs(t)) {
@@ -696,13 +697,13 @@ static void rcu_flavor_sched_clock_irq(int user) · 3 unmodified lines
696	697	} else if (rcu_preempt_need_deferred_qs(t)) {
697	698	rcu_preempt_deferred_qs(t); /* Report deferred QS. */
698	699	return;
699		− } else if (!t->rcu_read_lock_nesting) {
	700	+ } else if (!rcu_preempt_depth()) {
700	701	rcu_qs(); /* Report immediate QS. */
701	702	return;
702	703	}
703	704
704	705	/* If GP is oldish, ask for help from rcu_read_unlock_special(). */
705		− if (t->rcu_read_lock_nesting > 0 &&
	706	+ if (rcu_preempt_depth() > 0 &&
706	707	__this_cpu_read(rcu_data.core_needs_qs) &&
707	708	__this_cpu_read(rcu_data.cpu_no_qs.b.norm) &&
708	709	!t->rcu_read_unlock_special.b.need_qs &&
@@ -723,11 +724,11 @@ void exit_rcu(void) · 14 unmodified lines
723	724	struct task_struct *t = current;
724	725
725	726	if (unlikely(!list_empty(&current->rcu_node_entry))) {
726		− t->rcu_read_lock_nesting = 1;
	727	+ rcu_preempt_depth_set(1);
727	728	barrier();
728	729	WRITE_ONCE(t->rcu_read_unlock_special.b.blocked, true);
729		− } else if (unlikely(t->rcu_read_lock_nesting)) {
730		− t->rcu_read_lock_nesting = 1;
	730	+ } else if (unlikely(rcu_preempt_depth())) {
	731	+ rcu_preempt_depth_set(1);
731	732	} else {
732	733	return;
733	734	}
@@ -757,7 +758,8 @@ dump_blkd_tasks(struct rcu_node *rnp, int ncheck) · 23 unmodified lines
757	758	pr_info("%s: %d:%d ->qsmask %#lx ->qsmaskinit %#lx ->qsmaskinitnext %#lx\n",
758	759	__func__, rnp1->grplo, rnp1->grphi, rnp1->qsmask, rnp1->qsmaskinit, rnp1->qsmaskinitnext);
759	760	pr_info("%s: ->gp_tasks %p ->boost_tasks %p ->exp_tasks %p\n",
760		− __func__, rnp->gp_tasks, rnp->boost_tasks, rnp->exp_tasks);
	761	+ __func__, READ_ONCE(rnp->gp_tasks), rnp->boost_tasks,
	762	+ rnp->exp_tasks);
761	763	pr_info("%s: ->blkd_tasks", __func__);
762	764	i = 0;
763	765	list_for_each(lhp, &rnp->blkd_tasks) {
@@ -788,7 +790,7 @@ static void __init rcu_bootup_announce(void) · 24 unmodified lines
788	790	}
789	791
790	792	/*
791		− * Note a quiescent state for PREEMPT=n. Because we do not need to know
	793	+ * Note a quiescent state for PREEMPTION=n. Because we do not need to know
792	794	* how many quiescent states passed, just if there was at least one since
793	795	* the start of the grace period, this just sets a flag. The caller must
794	796	* have disabled preemption.
@@ -838,7 +840,7 @@ void rcu_all_qs(void) · 43 unmodified lines
838	840	EXPORT_SYMBOL_GPL(rcu_all_qs);
839	841
840	842	/*
841		− * Note a PREEMPT=n context switch. The caller must have disabled interrupts.
	843	+ * Note a PREEMPTION=n context switch. The caller must have disabled interrupts.
842	844	*/
843	845	void rcu_note_context_switch(bool preempt)
844	846	{
@@ -1262,10 +1264,9 @@ static void rcu_prepare_for_idle(void) · 417 unmodified lines
1262	1264	/*
1263	1265	* This code is invoked when a CPU goes idle, at which point we want
1264	1266	* to have the CPU do everything required for RCU so that it can enter
1265		− * the energy-efficient dyntick-idle mode. This is handled by a
1266		− * state machine implemented by rcu_prepare_for_idle() below.
	1267	+ * the energy-efficient dyntick-idle mode.
1267	1268	*
1268		− * The following three proprocessor symbols control this state machine:
	1269	+ * The following preprocessor symbol controls this:
1269	1270	*
1270	1271	* RCU_IDLE_GP_DELAY gives the number of jiffies that a CPU is permitted
1271	1272	* to sleep in dyntick-idle mode with RCU callbacks pending. This
@@ -1274,21 +1275,15 @@ static void rcu_prepare_for_idle(void) · 2 unmodified lines
1274	1275	* number, be warned: Setting RCU_IDLE_GP_DELAY too high can hang your
1275	1276	* system. And if you are -that- concerned about energy efficiency,
1276	1277	* just power the system down and be done with it!
1277		− * RCU_IDLE_LAZY_GP_DELAY gives the number of jiffies that a CPU is
1278		− * permitted to sleep in dyntick-idle mode with only lazy RCU
1279		− * callbacks pending. Setting this too high can OOM your system.
1280	1278	*
1281		− * The values below work well in practice. If future workloads require
	1279	+ * The value below works well in practice. If future workloads require
1282	1280	* adjustment, they can be converted into kernel config parameters, though
1283	1281	* making the state machine smarter might be a better option.
1284	1282	*/
1285	1283	#define RCU_IDLE_GP_DELAY 4 /* Roughly one grace period. */
1286		−#define RCU_IDLE_LAZY_GP_DELAY (6 * HZ) /* Roughly six seconds. */
1287	1284
1288	1285	static int rcu_idle_gp_delay = RCU_IDLE_GP_DELAY;
1289	1286	module_param(rcu_idle_gp_delay, int, 0644);
1290		−static int rcu_idle_lazy_gp_delay = RCU_IDLE_LAZY_GP_DELAY;
1291		−module_param(rcu_idle_lazy_gp_delay, int, 0644);
1292	1287
1293	1288	/*
1294	1289	* Try to advance callbacks on the current CPU, but only if it has been
@@ -1327,8 +1322,7 @@ static bool __maybe_unused rcu_try_advance_all_cbs(void) · 32 unmodified lines
1327	1322	/*
1328	1323	* Allow the CPU to enter dyntick-idle mode unless it has callbacks ready
1329	1324	* to invoke. If the CPU has callbacks, try to advance them. Tell the
1330		− * caller to set the timeout based on whether or not there are non-lazy
1331		− * callbacks.
	1325	+ * caller about what to set the timeout.
1332	1326	*
1333	1327	* The caller must have disabled interrupts.
1334	1328	*/
@@ -1354,25 +1348,18 @@ int rcu_needs_cpu(u64 basemono, u64 *nextevt) · 19 unmodified lines
1354	1348	}
1355	1349	rdp->last_accelerate = jiffies;
1356	1350
1357		− /* Request timer delay depending on laziness, and round. */
1358		− rdp->all_lazy = !rcu_segcblist_n_nonlazy_cbs(&rdp->cblist);
1359		− if (rdp->all_lazy) {
1360		− dj = round_jiffies(rcu_idle_lazy_gp_delay + jiffies) - jiffies;
1361		− } else {
1362		− dj = round_up(rcu_idle_gp_delay + jiffies,
1363		− rcu_idle_gp_delay) - jiffies;
1364		− }
	1351	+ /* Request timer and round. */
	1352	+ dj = round_up(rcu_idle_gp_delay + jiffies, rcu_idle_gp_delay) - jiffies;
	1353	+
1365	1354	nextevt = basemono + dj TICK_NSEC;
1366	1355	return 0;
1367	1356	}
1368	1357
1369	1358	/*
1370		− * Prepare a CPU for idle from an RCU perspective. The first major task
1371		− * is to sense whether nohz mode has been enabled or disabled via sysfs.
1372		− * The second major task is to check to see if a non-lazy callback has
1373		− * arrived at a CPU that previously had only lazy callbacks. The third
1374		− * major task is to accelerate (that is, assign grace-period numbers to)
1375		− * any recently arrived callbacks.
	1359	+ * Prepare a CPU for idle from an RCU perspective. The first major task is to
	1360	+ * sense whether nohz mode has been enabled or disabled via sysfs. The second
	1361	+ * major task is to accelerate (that is, assign grace-period numbers to) any
	1362	+ * recently arrived callbacks.
1376	1363	*
1377	1364	* The caller must have disabled interrupts.
1378	1365	*/
@@ -1398,17 +1385,6 @@ static void rcu_prepare_for_idle(void) · 19 unmodified lines
1398	1385	if (!tne)
1399	1386	return;
1400	1387
1401		− /*
1402		− * If a non-lazy callback arrived at a CPU having only lazy
1403		− * callbacks, invoke RCU core for the side-effect of recalculating
1404		− * idle duration on re-entry to idle.
1405		− */
1406		− if (rdp->all_lazy && rcu_segcblist_n_nonlazy_cbs(&rdp->cblist)) {
1407		− rdp->all_lazy = false;
1408		− invoke_rcu_core();
1409		− return;
1410		− }
1411		−
1412	1388	/*
1413	1389	* If we have not yet accelerated this jiffy, accelerate all
1414	1390	* callbacks on this CPU.
@@ -2321,6 +2297,8 @@ static void __init rcu_organize_nocb_kthreads(void) · 906 unmodified lines
2321	2297	{
2322	2298	int cpu;
2323	2299	bool firsttime = true;
	2300	+ bool gotnocbs = false;
	2301	+ bool gotnocbscbs = true;
2324	2302	int ls = rcu_nocb_gp_stride;
2325	2303	int nl = 0; /* Next GP kthread. */
2326	2304	struct rcu_data *rdp;
@@ -2343,21 +2321,31 @@ static void __init rcu_organize_nocb_kthreads(void) · 16 unmodified lines
2343	2321	rdp = per_cpu_ptr(&rcu_data, cpu);
2344	2322	if (rdp->cpu >= nl) {
2345	2323	/* New GP kthread, set up for CBs & next GP. */
	2324	+ gotnocbs = true;
2346	2325	nl = DIV_ROUND_UP(rdp->cpu + 1, ls) * ls;
2347	2326	rdp->nocb_gp_rdp = rdp;
2348	2327	rdp_gp = rdp;
2349		− if (!firsttime && dump_tree)
2350		− pr_cont("\n");
2351		− firsttime = false;
2352		− pr_alert("%s: No-CB GP kthread CPU %d:", __func__, cpu);
	2328	+ if (dump_tree) {
	2329	+ if (!firsttime)
	2330	+ pr_cont("%s\n", gotnocbscbs
	2331	+ ? "" : " (self only)");
	2332	+ gotnocbscbs = false;
	2333	+ firsttime = false;
	2334	+ pr_alert("%s: No-CB GP kthread CPU %d:",
	2335	+ __func__, cpu);
	2336	+ }
2353	2337	} else {
2354	2338	/* Another CB kthread, link to previous GP kthread. */
	2339	+ gotnocbscbs = true;
2355	2340	rdp->nocb_gp_rdp = rdp_gp;
2356	2341	rdp_prev->nocb_next_cb_rdp = rdp;
2357		− pr_alert(" %d", cpu);
	2342	+ if (dump_tree)
	2343	+ pr_cont(" %d", cpu);
2358	2344	}
2359	2345	rdp_prev = rdp;
2360	2346	}
	2347	+ if (gotnocbs && dump_tree)
	2348	+ pr_cont("%s\n", gotnocbscbs ? "" : " (self only)");
2361	2349	}
2362	2350
2363	2351	/*

kernel/rcu/tree_stall.h+27−7

@@ -163,7 +163,7 @@ static void rcu_iw_handler(struct irq_work *iwp) · 162 unmodified lines
163	163	//
164	164	// Printing RCU CPU stall warnings
165	165
166		−#ifdef CONFIG_PREEMPTION
	166	+#ifdef CONFIG_PREEMPT_RCU
167	167
168	168	/*
169	169	* Dump detailed information for all tasks blocking the current RCU
@@ -215,7 +215,7 @@ static int rcu_print_task_stall(struct rcu_node *rnp) · 45 unmodified lines
215	215	return ndetected;
216	216	}
217	217
218		−#else /* #ifdef CONFIG_PREEMPTION */
	218	+#else /* #ifdef CONFIG_PREEMPT_RCU */
219	219
220	220	/*
221	221	* Because preemptible RCU does not exist, we never have to check for
@@ -233,7 +233,7 @@ static int rcu_print_task_stall(struct rcu_node *rnp) · 11 unmodified lines
233	233	{
234	234	return 0;
235	235	}
236		−#endif /* #else #ifdef CONFIG_PREEMPTION */
	236	+#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
237	237
238	238	/*
239	239	* Dump stacks of all tasks running on stalled CPUs. First try using
@@ -263,11 +263,9 @@ static void print_cpu_stall_fast_no_hz(char *cp, int cpu) · 23 unmodified lines
263	263	{
264	264	struct rcu_data *rdp = &per_cpu(rcu_data, cpu);
265	265
266		− sprintf(cp, "last_accelerate: %04lx/%04lx, Nonlazy posted: %c%c%c",
	266	+ sprintf(cp, "last_accelerate: %04lx/%04lx dyntick_enabled: %d",
267	267	rdp->last_accelerate & 0xffff, jiffies & 0xffff,
268		− ".l"[rdp->all_lazy],
269		− ".L"[!rcu_segcblist_n_nonlazy_cbs(&rdp->cblist)],
270		− ".D"[!!rdp->tick_nohz_enabled_snap]);
	268	+ !!rdp->tick_nohz_enabled_snap);
271	269	}
272	270
273	271	#else /* #ifdef CONFIG_RCU_FAST_NO_HZ */
@@ -279,6 +277,28 @@ static void print_cpu_stall_fast_no_hz(char *cp, int cpu) · 5 unmodified lines
279	277
280	278	#endif /* #else #ifdef CONFIG_RCU_FAST_NO_HZ */
281	279
	280	+static const char * const gp_state_names[] = {
	281	+ [RCU_GP_IDLE] = "RCU_GP_IDLE",
	282	+ [RCU_GP_WAIT_GPS] = "RCU_GP_WAIT_GPS",
	283	+ [RCU_GP_DONE_GPS] = "RCU_GP_DONE_GPS",
	284	+ [RCU_GP_ONOFF] = "RCU_GP_ONOFF",
	285	+ [RCU_GP_INIT] = "RCU_GP_INIT",
	286	+ [RCU_GP_WAIT_FQS] = "RCU_GP_WAIT_FQS",
	287	+ [RCU_GP_DOING_FQS] = "RCU_GP_DOING_FQS",
	288	+ [RCU_GP_CLEANUP] = "RCU_GP_CLEANUP",
	289	+ [RCU_GP_CLEANED] = "RCU_GP_CLEANED",
	290	+};
	291	+
	292	+/*
	293	+ * Convert a ->gp_state value to a character string.
	294	+ */
	295	+static const char *gp_state_getname(short gs)
	296	+{
	297	+ if (gs < 0 \|\| gs >= ARRAY_SIZE(gp_state_names))
	298	+ return "???";
	299	+ return gp_state_names[gs];
	300	+}
	301	+
282	302	/*
283	303	* Print out diagnostic information for the specified stalled CPU.
284	304	*

kernel/rcu/update.c+11−3

@@ -40,6 +40,7 @@ · 39 unmodified lines
40	40	#include <linux/rcupdate_wait.h>
41	41	#include <linux/sched/isolation.h>
42	42	#include <linux/kprobes.h>
	43	+#include <linux/slab.h>
43	44
44	45	#define CREATE_TRACE_POINTS
45	46
@@ -51,9 +52,7 @@ · 5 unmodified lines
51	52	#define MODULE_PARAM_PREFIX "rcupdate."
52	53
53	54	#ifndef CONFIG_TINY_RCU
54		−extern int rcu_expedited; /* from sysctl */
55	55	module_param(rcu_expedited, int, 0);
56		−extern int rcu_normal; /* from sysctl */
57	56	module_param(rcu_normal, int, 0);
58	57	static int rcu_normal_after_boot;
59	58	module_param(rcu_normal_after_boot, int, 0);
@@ -218,6 +217,7 @@ static int __init rcu_set_runtime_mode(void) · 158 unmodified lines
218	217	{
219	218	rcu_test_sync_prims();
220	219	rcu_scheduler_active = RCU_SCHEDULER_RUNNING;
	220	+ kfree_rcu_scheduler_running();
221	221	rcu_test_sync_prims();
222	222	return 0;
223	223	}
@@ -435,7 +435,7 @@ struct debug_obj_descr rcuhead_debug_descr = { · 211 unmodified lines
435	435	EXPORT_SYMBOL_GPL(rcuhead_debug_descr);
436	436	#endif /* #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD */
437	437
438		−#if defined(CONFIG_TREE_RCU) \|\| defined(CONFIG_PREEMPT_RCU) \|\| defined(CONFIG_RCU_TRACE)
	438	+#if defined(CONFIG_TREE_RCU) \|\| defined(CONFIG_RCU_TRACE)
439	439	void do_trace_rcu_torture_read(const char rcutorturename, struct rcu_head rhp,
440	440	unsigned long secs,
441	441	unsigned long c_old, unsigned long c)
@@ -853,14 +853,22 @@ static void test_callback(struct rcu_head *r) · 411 unmodified lines
853	853
854	854	DEFINE_STATIC_SRCU(early_srcu);
855	855
	856	+struct early_boot_kfree_rcu {
	857	+ struct rcu_head rh;
	858	+};
	859	+
856	860	static void early_boot_test_call_rcu(void)
857	861	{
858	862	static struct rcu_head head;
859	863	static struct rcu_head shead;
	864	+ struct early_boot_kfree_rcu *rhp;
860	865
861	866	call_rcu(&head, test_callback);
862	867	if (IS_ENABLED(CONFIG_SRCU))
863	868	call_srcu(&early_srcu, &shead, test_callback);
	869	+ rhp = kmalloc(sizeof(*rhp), GFP_KERNEL);
	870	+ if (!WARN_ON_ONCE(!rhp))
	871	+ kfree_rcu(rhp, rh);
864	872	}
865	873
866	874	void rcu_early_boot_tests(void)

kernel/sysctl.c+1−1

@@ -1268,7 +1268,7 @@ static struct ctl_table kern_table[] = { · 1267 unmodified lines
1268	1268	.proc_handler = proc_do_static_key,
1269	1269	},
1270	1270	#endif
1271		−#if defined(CONFIG_TREE_RCU) \|\| defined(CONFIG_PREEMPT_RCU)
	1271	+#if defined(CONFIG_TREE_RCU)
1272	1272	{
1273	1273	.procname = "panic_on_rcu_stall",
1274	1274	.data = &sysctl_panic_on_rcu_stall,

net/tipc/crypto.c+1−4

@@ -257,9 +257,6 @@ static char *tipc_key_change_dump(struct tipc_key old, struct tipc_key new, · 256 unmodified lines
257	257	#define tipc_aead_rcu_ptr(rcu_ptr, lock) \
258	258	rcu_dereference_protected((rcu_ptr), lockdep_is_held(lock))
259	259
260		−#define tipc_aead_rcu_swap(rcu_ptr, ptr, lock) \
261		− rcu_swap_protected((rcu_ptr), (ptr), lockdep_is_held(lock))
262		−
263	260	#define tipc_aead_rcu_replace(rcu_ptr, ptr, lock) \
264	261	do { \
265	262	typeof(rcu_ptr) __tmp = rcu_dereference_protected((rcu_ptr), \
@@ -1189,7 +1186,7 @@ static bool tipc_crypto_key_try_align(struct tipc_crypto *rx, u8 new_pending) · 923 unmodified lines
1189	1186
1190	1187	/* Move passive key if any */
1191	1188	if (key.passive) {
1192		− tipc_aead_rcu_swap(rx->aead[key.passive], tmp2, &rx->lock);
	1189	+ tmp2 = rcu_replace_pointer(rx->aead[key.passive], tmp2, lockdep_is_held(&rx->lock));
1193	1190	x = (key.passive - key.pending + new_pending) % KEY_MAX;
1194	1191	new_passive = (x <= 0) ? x + KEY_MAX : x;
1195	1192	}

tools/testing/selftests/rcutorture/bin/cpus2use.sh+9−2

@@ -15,8 +15,15 @@ then · 14 unmodified lines
15	15	exit 0
16	16	fi
17	17	ncpus=`grep '^processor' /proc/cpuinfo \| wc -l`
18		−idlecpus=`mpstat \| tail -1 \| \
19		− awk -v ncpus=$ncpus '{ print ncpus * ($7 + $NF) / 100 }'`
	18	+if mpstat -V > /dev/null 2>&1
	19	+then
	20	+ idlecpus=`mpstat \| tail -1 \| \
	21	+ awk -v ncpus=$ncpus '{ print ncpus * ($7 + $NF) / 100 }'`
	22	+else
	23	+ # No mpstat command, so use all available CPUs.
	24	+ echo The mpstat command is not available, so greedily using all CPUs.
	25	+ idlecpus=$ncpus
	26	+fi
20	27	awk -v ncpus=$ncpus -v idlecpus=$idlecpus < /dev/null '
21	28	BEGIN {
22	29	cpus2use = idlecpus;

tools/testing/selftests/rcutorture/bin/jitter.sh+22−8

@@ -23,25 +23,39 @@ spinmax=${4-1000} · 22 unmodified lines
23	23
24	24	n=1
25	25
26		−starttime=`awk 'BEGIN { print systime(); }' < /dev/null`
	26	+starttime=`gawk 'BEGIN { print systime(); }' < /dev/null`
	27	+
	28	+nohotplugcpus=
	29	+for i in /sys/devices/system/cpu/cpu[0-9]*
	30	+do
	31	+ if test -f $i/online
	32	+ then
	33	+ :
	34	+ else
	35	+ curcpu=`echo $i \| sed -e 's/^[^0-9]*//'`
	36	+ nohotplugcpus="$nohotplugcpus $curcpu"
	37	+ fi
	38	+done
27	39
28	40	while :
29	41	do
30	42	# Check for done.
31		− t=`awk -v s=$starttime 'BEGIN { print systime() - s; }' < /dev/null`
	43	+ t=`gawk -v s=$starttime 'BEGIN { print systime() - s; }' < /dev/null`
32	44	if test "$t" -gt "$duration"
33	45	then
34	46	exit 0;
35	47	fi
36	48
37	49	# Set affinity to randomly selected online CPU
38		− cpus=`grep 1 /sys/devices/system/cpu/*/online \|
39		− sed -e 's,/[^/]$,,' -e 's/^[^0-9]//'`
40		−
41		− # Do not leave out poor old cpu0 which may not be hot-pluggable
42		− if [ ! -f "/sys/devices/system/cpu/cpu0/online" ]; then
43		− cpus="0 $cpus"
	50	+ if cpus=`grep 1 /sys/devices/system/cpu/*/online 2>&1 \|
	51	+ sed -e 's,/[^/]$,,' -e 's/^[^0-9]//'`
	52	+ then
	53	+ :
	54	+ else
	55	+ cpus=
44	56	fi
	57	+ # Do not leave out non-hot-pluggable CPUs
	58	+ cpus="$cpus $nohotplugcpus"
45	59
46	60	cpumask=`awk -v cpus="$cpus" -v me=$me -v n=$n 'BEGIN {
47	61	srand(n + me + systime());

tools/testing/selftests/rcutorture/bin/kvm-recheck-rcu.sh+2−1

@@ -25,6 +25,7 @@ stopstate="`grep 'End-test grace-period state: g' $i/console.log 2> /dev/null \| · 24 unmodified lines
25	25	tail -1 \| sed -e 's/^\[[ 0-9.]*] //' \|
26	26	awk '{ print \"[\" $1 \" \" $5 \" \" $6 \" \" $7 \"]\"; }' \|
27	27	tr -d '\012\015'`"
	28	+fwdprog="`grep 'rcu_torture_fwd_prog_cr Duration' $i/console.log 2> /dev/null \| sed -e 's/^\[[^]]*] //' \| sort -k15nr \| head -1 \| awk '{ print $14 " " $15 }'`"
28	29	if test -z "$ngps"
29	30	then
30	31	echo "$configfile ------- " $stopstate
@@ -39,7 +40,7 @@ else · 8 unmodified lines
39	40	BEGIN { print ngps / dur }' < /dev/null`
40	41	title="$title ($ngpsps/s)"
41	42	fi
42		− echo $title $stopstate
	43	+ echo $title $stopstate $fwdprog
43	44	nclosecalls=`grep --binary-files=text 'torture: Reader Batch' $i/console.log \| tail -1 \| awk '{for (i=NF-8;i<=NF;i++) sum+=$i; } END {print sum}'`
44	45	if test -z "$nclosecalls"
45	46	then

tools/testing/selftests/rcutorture/bin/kvm-test-1-run.sh+6−7

@@ -123,7 +123,7 @@ qemu_args=$5 · 122 unmodified lines
123	123	boot_args=$6
124	124
125	125	cd $KVM
126		−kstarttime=`awk 'BEGIN { print systime() }' < /dev/null`
	126	+kstarttime=`gawk 'BEGIN { print systime() }' < /dev/null`
127	127	if test -z "$TORTURE_BUILDONLY"
128	128	then
129	129	echo ' ---' `date`: Starting kernel
@@ -133,11 +133,10 @@ fi · 3 unmodified lines
133	133	qemu_args="-enable-kvm -nographic $qemu_args"
134	134	cpu_count=`configNR_CPUS.sh $resdir/ConfigFragment`
135	135	cpu_count=`configfrag_boot_cpus "$boot_args" "$config_template" "$cpu_count"`
136		−vcpus=`identify_qemu_vcpus`
137		−if test $cpu_count -gt $vcpus
	136	+if test "$cpu_count" -gt "$TORTURE_ALLOTED_CPUS"
138	137	then
139		− echo CPU count limited from $cpu_count to $vcpus \| tee -a $resdir/Warnings
140		− cpu_count=$vcpus
	138	+ echo CPU count limited from $cpu_count to $TORTURE_ALLOTED_CPUS \| tee -a $resdir/Warnings
	139	+ cpu_count=$TORTURE_ALLOTED_CPUS
141	140	fi
142	141	qemu_args="`specify_qemu_cpus "$QEMU" "$qemu_args" "$cpu_count"`"
143	142
@@ -177,7 +176,7 @@ do · 33 unmodified lines
177	176	then
178	177	qemu_pid=`cat "$resdir/qemu_pid"`
179	178	fi
180		− kruntime=`awk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
	179	+ kruntime=`gawk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
181	180	if test -z "$qemu_pid" \|\| kill -0 "$qemu_pid" > /dev/null 2>&1
182	181	then
183	182	if test $kruntime -ge $seconds
@@ -213,7 +212,7 @@ then · 29 unmodified lines
213	212	oldline="`tail $resdir/console.log`"
214	213	while :
215	214	do
216		− kruntime=`awk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
	215	+ kruntime=`gawk 'BEGIN { print systime() - '"$kstarttime"' }' < /dev/null`
217	216	if kill -0 $qemu_pid > /dev/null 2>&1
218	217	then
219	218	:

tools/testing/selftests/rcutorture/bin/kvm.sh+21−9

@@ -24,7 +24,9 @@ dur=$((30*60)) · 23 unmodified lines
24	24	dryrun=""
25	25	KVM="`pwd`/tools/testing/selftests/rcutorture"; export KVM
26	26	PATH=${KVM}/bin:$PATH; export PATH
27		−TORTURE_ALLOTED_CPUS=""
	27	+. functions.sh
	28	+
	29	+TORTURE_ALLOTED_CPUS="`identify_qemu_vcpus`"
28	30	TORTURE_DEFCONFIG=defconfig
29	31	TORTURE_BOOT_IMAGE=""
30	32	TORTURE_INITRD="$KVM/initrd"; export TORTURE_INITRD
@@ -40,8 +42,6 @@ cpus=0 · 9 unmodified lines
40	42	ds=`date +%Y.%m.%d-%H:%M:%S`
41	43	jitter="-1"
42	44
43		−. functions.sh
44		−
45	45	usage () {
46	46	echo "Usage: $scriptname optional arguments:"
47	47	echo " --bootargs kernel-boot-arguments"
@@ -93,6 +93,11 @@ do · 45 unmodified lines
93	93	checkarg --cpus "(number)" "$#" "$2" '^[0-9]*$' '^--'
94	94	cpus=$2
95	95	TORTURE_ALLOTED_CPUS="$2"
	96	+ max_cpus="`identify_qemu_vcpus`"
	97	+ if test "$TORTURE_ALLOTED_CPUS" -gt "$max_cpus"
	98	+ then
	99	+ TORTURE_ALLOTED_CPUS=$max_cpus
	100	+ fi
96	101	shift
97	102	;;
98	103	--datestamp)
@@ -198,9 +203,10 @@ fi · 99 unmodified lines
198	203
199	204	CONFIGFRAG=${KVM}/configs/${TORTURE_SUITE}; export CONFIGFRAG
200	205
	206	+defaultconfigs="`tr '\012' ' ' < $CONFIGFRAG/CFLIST`"
201	207	if test -z "$configs"
202	208	then
203		− configs="`cat $CONFIGFRAG/CFLIST`"
	209	+ configs=$defaultconfigs
204	210	fi
205	211
206	212	if test -z "$resdir"
@@ -209,7 +215,7 @@ then · 2 unmodified lines
209	215	fi
210	216
211	217	# Create a file of test-name/#cpus pairs, sorted by decreasing #cpus.
212		−touch $T/cfgcpu
	218	+configs_derep=
213	219	for CF in $configs
214	220	do
215	221	case $CF in
@@ -222,15 +228,21 @@ do · 6 unmodified lines
222	228	CF1=$CF
223	229	;;
224	230	esac
	231	+ for ((cur_rep=0;cur_rep<$config_reps;cur_rep++))
	232	+ do
	233	+ configs_derep="$configs_derep $CF1"
	234	+ done
	235	+done
	236	+touch $T/cfgcpu
	237	+configs_derep="`echo $configs_derep \| sed -e "s/\<CFLIST\>/$defaultconfigs/g"`"
	238	+for CF1 in $configs_derep
	239	+do
225	240	if test -f "$CONFIGFRAG/$CF1"
226	241	then
227	242	cpu_count=`configNR_CPUS.sh $CONFIGFRAG/$CF1`
228	243	cpu_count=`configfrag_boot_cpus "$TORTURE_BOOTARGS" "$CONFIGFRAG/$CF1" "$cpu_count"`
229	244	cpu_count=`configfrag_boot_maxcpus "$TORTURE_BOOTARGS" "$CONFIGFRAG/$CF1" "$cpu_count"`
230		− for ((cur_rep=0;cur_rep<$config_reps;cur_rep++))
231		− do
232		− echo $CF1 $cpu_count >> $T/cfgcpu
233		− done
	245	+ echo $CF1 $cpu_count >> $T/cfgcpu
234	246	else
235	247	echo "The --configs file $CF1 does not exist, terminating."
236	248	exit 1

tools/testing/selftests/rcutorture/bin/mkinitrd.sh+3−52

@@ -20,58 +20,9 @@ if [ -s "$D/initrd/init" ]; then · 19 unmodified lines
20	20	exit 0
21	21	fi
22	22
23		−T=${TMPDIR-/tmp}/mkinitrd.sh.$$
24		−trap 'rm -rf $T' 0 2
25		−mkdir $T
26		−
27		−cat > $T/init << '__EOF___'
28		−#!/bin/sh
29		−# Run in userspace a few milliseconds every second. This helps to
30		−# exercise the NO_HZ_FULL portions of RCU. The 192 instances of "a" was
31		−# empirically shown to give a nice multi-millisecond burst of user-mode
32		−# execution on a 2GHz CPU, as desired. Modern CPUs will vary from a
33		−# couple of milliseconds up to perhaps 100 milliseconds, which is an
34		−# acceptable range.
35		−#
36		−# Why not calibrate an exact delay? Because within this initrd, we
37		−# are restricted to Bourne-shell builtins, which as far as I know do not
38		−# provide any means of obtaining a fine-grained timestamp.
39		−
40		−a4="a a a a"
41		−a16="$a4 $a4 $a4 $a4"
42		−a64="$a16 $a16 $a16 $a16"
43		−a192="$a64 $a64 $a64"
44		−while :
45		−do
46		− q=
47		− for i in $a192
48		− do
49		− q="$q $i"
50		− done
51		− sleep 1
52		−done
53		−__EOF___
54		−
55		−# Try using dracut to create initrd
56		−if command -v dracut >/dev/null 2>&1
57		−then
58		− echo Creating $D/initrd using dracut.
59		− # Filesystem creation
60		− dracut --force --no-hostonly --no-hostonly-cmdline --module "base" $T/initramfs.img
61		− cd $D
62		− mkdir -p initrd
63		− cd initrd
64		− zcat $T/initramfs.img \| cpio -id
65		− cp $T/init init
66		− chmod +x init
67		− echo Done creating $D/initrd using dracut
68		− exit 0
69		−fi
70		−
71		−# No dracut, so create a C-language initrd/init program and statically
72		−# link it. This results in a very small initrd, but might be a bit less
73		−# future-proof than dracut.
74		−echo "Could not find dracut, attempting C initrd"
	23	+# Create a C-language initrd/init infinite-loop program and statically
	24	+# link it. This results in a very small initrd.
	25	+echo "Creating a statically linked C-language initrd"
75	26	cd $D
76	27	mkdir -p initrd
77	28	cd initrd