lael-labs/linux

Documentation/RCU/stallwarn.rst+0−11

@@ -254,17 +254,6 @@ period (in this case 2603), the grace-period sequence number (7075), and · 253 unmodified lines
254	an estimate of the total number of RCU callbacks queued across all CPUs	254	an estimate of the total number of RCU callbacks queued across all CPUs
255	(625 in this case).	255	(625 in this case).
256		256
257	In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed
258	for each CPU::
259
260	0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1
261
262	The "last_accelerate:" prints the low-order 16 bits (in hex) of the
263	jiffies counter when this CPU last invoked rcu_try_advance_all_cbs()
264	from rcu_needs_cpu() or last invoked rcu_accelerate_cbs() from
265	rcu_prepare_for_idle(). "dyntick_enabled: 1" indicates that dyntick-idle
266	processing is enabled.
267
268	If the grace period ends just as the stall warning starts printing,	257	If the grace period ends just as the stall warning starts printing,
269	there will be a spurious stall-warning message, which will include	258	there will be a spurious stall-warning message, which will include
270	the following::	259	the following::

Documentation/admin-guide/kernel-parameters.txt+52−18

@@ -4343,19 +4343,30 @@ · 4342 unmodified lines
4343	Disable the Correctable Errors Collector,	4343	Disable the Correctable Errors Collector,
4344	see CONFIG_RAS_CEC help text.	4344	see CONFIG_RAS_CEC help text.
4345		4345
4346	rcu_nocbs= [KNL]	4346	rcu_nocbs[=cpu-list]
4347	The argument is a cpu list, as described above.	4347	[KNL] The optional argument is a cpu list,
4348		4348	as described above.
4349	In kernels built with CONFIG_RCU_NOCB_CPU=y, set	4349
4350	the specified list of CPUs to be no-callback CPUs.	4350	In kernels built with CONFIG_RCU_NOCB_CPU=y,
4351	Invocation of these CPUs' RCU callbacks will be	4351	enable the no-callback CPU mode, which prevents
4352	offloaded to "rcuox/N" kthreads created for that	4352	such CPUs' callbacks from being invoked in
4353	purpose, where "x" is "p" for RCU-preempt, and	4353	softirq context. Invocation of such CPUs' RCU
4354	"s" for RCU-sched, and "N" is the CPU number.	4354	callbacks will instead be offloaded to "rcuox/N"
4355	This reduces OS jitter on the offloaded CPUs,	4355	kthreads created for that purpose, where "x" is
4356	which can be useful for HPC and real-time	4356	"p" for RCU-preempt, "s" for RCU-sched, and "g"
4357	workloads. It can also improve energy efficiency	4357	for the kthreads that mediate grace periods; and
4358	for asymmetric multiprocessors.	4358	"N" is the CPU number. This reduces OS jitter on
		4359	the offloaded CPUs, which can be useful for HPC
		4360	and real-time workloads. It can also improve
		4361	energy efficiency for asymmetric multiprocessors.
		4362
		4363	If a cpulist is passed as an argument, the specified
		4364	list of CPUs is set to no-callback mode from boot.
		4365
		4366	Otherwise, if the '=' sign and the cpulist
		4367	arguments are omitted, no CPU will be set to
		4368	no-callback mode from boot but the mode may be
		4369	toggled at runtime via cpusets.
4359		4370
4360	rcu_nocb_poll [KNL]	4371	rcu_nocb_poll [KNL]
4361	Rather than requiring that offloaded CPUs	4372	Rather than requiring that offloaded CPUs
@@ -4489,10 +4500,6 @@ · 127 unmodified lines
4489	on rcutree.qhimark at boot time and to zero to	4500	on rcutree.qhimark at boot time and to zero to
4490	disable more aggressive help enlistment.	4501	disable more aggressive help enlistment.
4491		4502
4492	rcutree.rcu_idle_gp_delay= [KNL]
4493	Set wakeup interval for idle CPUs that have
4494	RCU callbacks (RCU_FAST_NO_HZ=y).
4495
4496	rcutree.rcu_kick_kthreads= [KNL]	4503	rcutree.rcu_kick_kthreads= [KNL]
4497	Cause the grace-period kthread to get an extra	4504	Cause the grace-period kthread to get an extra
4498	wake_up() if it sleeps three times longer than	4505	wake_up() if it sleeps three times longer than
@@ -4603,8 +4610,12 @@ · 104 unmodified lines
4603	in seconds.	4610	in seconds.
4604		4611
4605	rcutorture.fwd_progress= [KNL]	4612	rcutorture.fwd_progress= [KNL]
4606	Enable RCU grace-period forward-progress testing	4613	Specifies the number of kthreads to be used
		4614	for RCU grace-period forward-progress testing
4607	for the types of RCU supporting this notion.	4615	for the types of RCU supporting this notion.
		4616	Defaults to 1 kthread, values less than zero or
		4617	greater than the number of CPUs cause the number
		4618	of CPUs to be used.
4608		4619
4609	rcutorture.fwd_progress_div= [KNL]	4620	rcutorture.fwd_progress_div= [KNL]
4610	Specify the fraction of a CPU-stall-warning	4621	Specify the fraction of a CPU-stall-warning
@@ -4805,6 +4816,29 @@ · 194 unmodified lines
4805	period to instead use normal non-expedited	4816	period to instead use normal non-expedited
4806	grace-period processing.	4817	grace-period processing.
4807		4818
		4819	rcupdate.rcu_task_collapse_lim= [KNL]
		4820	Set the maximum number of callbacks present
		4821	at the beginning of a grace period that allows
		4822	the RCU Tasks flavors to collapse back to using
		4823	a single callback queue. This switching only
		4824	occurs when rcupdate.rcu_task_enqueue_lim is
		4825	set to the default value of -1.
		4826
		4827	rcupdate.rcu_task_contend_lim= [KNL]
		4828	Set the minimum number of callback-queuing-time
		4829	lock-contention events per jiffy required to
		4830	cause the RCU Tasks flavors to switch to per-CPU
		4831	callback queuing. This switching only occurs
		4832	when rcupdate.rcu_task_enqueue_lim is set to
		4833	the default value of -1.
		4834
		4835	rcupdate.rcu_task_enqueue_lim= [KNL]
		4836	Set the number of callback queues to use for the
		4837	RCU Tasks family of RCU flavors. The default
		4838	of -1 allows this to be automatically (and
		4839	dynamically) adjusted. This parameter is intended
		4840	for use in testing.
		4841
4808	rcupdate.rcu_task_ipi_delay= [KNL]	4842	rcupdate.rcu_task_ipi_delay= [KNL]
4809	Set time in jiffies during which RCU tasks will	4843	Set time in jiffies during which RCU tasks will
4810	avoid sending IPIs, starting with the beginning	4844	avoid sending IPIs, starting with the beginning

Documentation/timers/no_hz.rst+3−7

@@ -184,16 +184,12 @@ There are situations in which idle CPUs cannot be permitted to · 183 unmodified lines
184	enter either dyntick-idle mode or adaptive-tick mode, the most	184	enter either dyntick-idle mode or adaptive-tick mode, the most
185	common being when that CPU has RCU callbacks pending.	185	common being when that CPU has RCU callbacks pending.
186		186
187	The CONFIG_RCU_FAST_NO_HZ=y Kconfig option may be used to cause such CPUs	187	Avoid this by offloading RCU callback processing to "rcuo" kthreads
188	to enter dyntick-idle mode or adaptive-tick mode anyway. In this case,
189	a timer will awaken these CPUs every four jiffies in order to ensure
190	that the RCU callbacks are processed in a timely fashion.
191
192	Another approach is to offload RCU callback processing to "rcuo" kthreads
193	using the CONFIG_RCU_NOCB_CPU=y Kconfig option. The specific CPUs to	188	using the CONFIG_RCU_NOCB_CPU=y Kconfig option. The specific CPUs to
194	offload may be selected using The "rcu_nocbs=" kernel boot parameter,	189	offload may be selected using The "rcu_nocbs=" kernel boot parameter,
195	which takes a comma-separated list of CPUs and CPU ranges, for example,	190	which takes a comma-separated list of CPUs and CPU ranges, for example,
196	"1,3-5" selects CPUs 1, 3, 4, and 5.	191	"1,3-5" selects CPUs 1, 3, 4, and 5. Note that CPUs specified by
		192	the "nohz_full" kernel boot parameter are also offloaded.
197		193
198	The offloaded CPUs will never queue RCU callbacks, and therefore RCU	194	The offloaded CPUs will never queue RCU callbacks, and therefore RCU
199	never prevents offloaded CPUs from entering either dyntick-idle mode	195	never prevents offloaded CPUs from entering either dyntick-idle mode

include/linux/rcu_segcblist.h+37−14

  *
  *
  *  ----------------------------------------------------------------------------
- *  |                         SEGCBLIST_SOFTIRQ_ONLY                           |
+ *  |                              SEGCBLIST_RCU_CORE                          |
  *  |                                                                          |
  *  |  Callbacks processed by rcu_core() from softirqs or local                |
  *  |  rcuc kthread, without holding nocb_lock.                                |
  *                                         |
  *                                         v
  *  ----------------------------------------------------------------------------
- *  |                        SEGCBLIST_OFFLOADED                               |
+ *  |       SEGCBLIST_RCU_CORE | SEGCBLIST_LOCKING | SEGCBLIST_OFFLOADED       |
  *  |                                                                          |
  *  | Callbacks processed by rcu_core() from softirqs or local                 |
  *  | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads,     |
  *                        |                                 |
  *                        v                                 v
  *  ---------------------------------------  ----------------------------------|
- *  |        SEGCBLIST_OFFLOADED |        |  |     SEGCBLIST_OFFLOADED |       |
+ *  |        SEGCBLIST_RCU_CORE   |       |  |     SEGCBLIST_RCU_CORE   |      |
+ *  |        SEGCBLIST_LOCKING    |       |  |     SEGCBLIST_LOCKING    |      |
+ *  |        SEGCBLIST_OFFLOADED  |       |  |     SEGCBLIST_OFFLOADED  |      |
  *  |        SEGCBLIST_KTHREAD_CB         |  |     SEGCBLIST_KTHREAD_GP        |
  *  |                                     |  |                                 |
  *  |                                     |  |                                 |
  *                                         |
  *                                         v
  *  |--------------------------------------------------------------------------|
- *  |                           SEGCBLIST_OFFLOADED |                          |
+ *  |                           SEGCBLIST_LOCKING    |                         |
- *  |                           SEGCBLIST_KTHREAD_CB |                         |
+ *  |                           SEGCBLIST_OFFLOADED  |                         |
- *  |                           SEGCBLIST_KTHREAD_GP                           |
+ *  |                           SEGCBLIST_KTHREAD_GP |                         |
+ *  |                           SEGCBLIST_KTHREAD_CB                           |
  *  |                                                                          |
  *  |   Kthreads handle callbacks holding nocb_lock, local rcu_core() stops    |
  *  |   handling callbacks. Enable bypass queueing.                            |
  *
  *
  *  |--------------------------------------------------------------------------|
- *  |                           SEGCBLIST_OFFLOADED |                          |
+ *  |                           SEGCBLIST_LOCKING    |                         |
+ *  |                           SEGCBLIST_OFFLOADED  |                         |
  *  |                           SEGCBLIST_KTHREAD_CB |                         |
  *  |                           SEGCBLIST_KTHREAD_GP                           |
  *  |                                                                          |
  *                                      |
  *                                      v
  *  |--------------------------------------------------------------------------|
+ *  |                           SEGCBLIST_RCU_CORE   |                         |
+ *  |                           SEGCBLIST_LOCKING    |                         |
+ *  |                           SEGCBLIST_OFFLOADED  |                         |
+ *  |                           SEGCBLIST_KTHREAD_CB |                         |
+ *  |                           SEGCBLIST_KTHREAD_GP                           |
+ *  |                                                                          |
+ *  |   CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core()    |
+ *  |   handles callbacks concurrently. Bypass enqueue is enabled.             |
+ *  |   Invoke RCU core so we make sure not to preempt it in the middle with   |
+ *  |   leaving some urgent work unattended within a jiffy.                    |
+ *  ----------------------------------------------------------------------------
+ *                                      |
+ *                                      v
+ *  |--------------------------------------------------------------------------|
+ *  |                           SEGCBLIST_RCU_CORE   |                         |
+ *  |                           SEGCBLIST_LOCKING    |                         |
  *  |                           SEGCBLIST_KTHREAD_CB |                         |
  *  |                           SEGCBLIST_KTHREAD_GP                           |
  *  |                                                                          |
  *                     |                                 |
  *                     v                                 v
  *  ---------------------------------------------------------------------------|
- *  |                                                                          |
+ *  |                                     |                                    |
+ *  |        SEGCBLIST_RCU_CORE |         |       SEGCBLIST_RCU_CORE |         |
+ *  |        SEGCBLIST_LOCKING  |         |       SEGCBLIST_LOCKING  |         |
  *  |        SEGCBLIST_KTHREAD_CB         |       SEGCBLIST_KTHREAD_GP         |
  *  |                                     |                                    |
  *  | GP kthread woke up and              |   CB kthread woke up and           |
  *                                      |
  *                                      v
  *  ----------------------------------------------------------------------------
- *  |                                   0                                      |
+ *  |                SEGCBLIST_RCU_CORE | SEGCBLIST_LOCKING                    |
  *  |                                                                          |
  *  | Callbacks processed by rcu_core() from softirqs or local                 |
  *  | rcuc kthread, while holding nocb_lock. Forbid nocb_timer to be armed.    |
  *                                      |
  *                                      v
  *  ----------------------------------------------------------------------------
- *  |                         SEGCBLIST_SOFTIRQ_ONLY                           |
+ *  |                         SEGCBLIST_RCU_CORE                               |
  *  |                                                                          |
  *  |  Callbacks processed by rcu_core() from softirqs or local                |
  *  |  rcuc kthread, without holding nocb_lock.                                |
  *  ----------------------------------------------------------------------------
  */
 #define SEGCBLIST_ENABLED	BIT(0)
-#define SEGCBLIST_SOFTIRQ_ONLY	BIT(1)
+#define SEGCBLIST_RCU_CORE	BIT(1)
-#define SEGCBLIST_KTHREAD_CB	BIT(2)
+#define SEGCBLIST_LOCKING	BIT(2)
-#define SEGCBLIST_KTHREAD_GP	BIT(3)
+#define SEGCBLIST_KTHREAD_CB	BIT(3)
-#define SEGCBLIST_OFFLOADED	BIT(4)
+#define SEGCBLIST_KTHREAD_GP	BIT(4)
+#define SEGCBLIST_OFFLOADED	BIT(5)
 struct rcu_segcblist {
 	struct rcu_head *head;

include/linux/rcupdate.h+28−22

 #define rcu_check_sparse(p, space)
 #endif /* #else #ifdef __CHECKER__ */
+#define __unrcu_pointer(p, local)					\
+({									\
+	typeof(*p) *local = (typeof(*p) *__force)(p);			\
+	rcu_check_sparse(p, __rcu);					\
+	((typeof(*p) __force __kernel *)(local)); 			\
+})
 /**
  * unrcu_pointer - mark a pointer as not being RCU protected
  * @p: pointer needing to lose its __rcu property
  * Converts @p from an __rcu pointer to a __kernel pointer.
  * This allows an __rcu pointer to be used with xchg() and friends.
  */
-#define unrcu_pointer(p)						\
+#define unrcu_pointer(p) __unrcu_pointer(p, __UNIQUE_ID(rcu))
-({									\
-	typeof(*p) *_________p1 = (typeof(*p) *__force)(p);		\
-	rcu_check_sparse(p, __rcu);					\
-	((typeof(*p) __force __kernel *)(_________p1)); 		\
-})
-#define __rcu_access_pointer(p, space) \
+#define __rcu_access_pointer(p, local, space) \
 ({ \
-	typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \
+	typeof(*p) *local = (typeof(*p) *__force)READ_ONCE(p); \
 	rcu_check_sparse(p, space); \
-	((typeof(*p) __force __kernel *)(_________p1)); \
+	((typeof(*p) __force __kernel *)(local)); \
 })
-#define __rcu_dereference_check(p, c, space) \
+#define __rcu_dereference_check(p, local, c, space) \
 ({ \
 	/* Dependency order vs. p above. */ \
-	typeof(*p) *________p1 = (typeof(*p) *__force)READ_ONCE(p); \
+	typeof(*p) *local = (typeof(*p) *__force)READ_ONCE(p); \
 	RCU_LOCKDEP_WARN(!(c), "suspicious rcu_dereference_check() usage"); \
 	rcu_check_sparse(p, space); \
-	((typeof(*p) __force __kernel *)(________p1)); \
+	((typeof(*p) __force __kernel *)(local)); \
 })
-#define __rcu_dereference_protected(p, c, space) \
+#define __rcu_dereference_protected(p, local, c, space) \
 ({ \
 	RCU_LOCKDEP_WARN(!(c), "suspicious rcu_dereference_protected() usage"); \
 	rcu_check_sparse(p, space); \
 	((typeof(*p) __force __kernel *)(p)); \
 })
-#define rcu_dereference_raw(p) \
+#define __rcu_dereference_raw(p, local) \
 ({ \
 	/* Dependency order vs. p above. */ \
-	typeof(p) ________p1 = READ_ONCE(p); \
+	typeof(p) local = READ_ONCE(p); \
-	((typeof(*p) __force __kernel *)(________p1)); \
+	((typeof(*p) __force __kernel *)(local)); \
 })
+#define rcu_dereference_raw(p) __rcu_dereference_raw(p, __UNIQUE_ID(rcu))
 /**
  * RCU_INITIALIZER() - statically initialize an RCU-protected global variable
  * when tearing down multi-linked structures after a grace period
  * has elapsed.
  */
-#define rcu_access_pointer(p) __rcu_access_pointer((p), __rcu)
+#define rcu_access_pointer(p) __rcu_access_pointer((p), __UNIQUE_ID(rcu), __rcu)
 /**
  * rcu_dereference_check() - rcu_dereference with debug checking
  * annotated as __rcu.
  */
 #define rcu_dereference_check(p, c) \
-	__rcu_dereference_check((p), (c) || rcu_read_lock_held(), __rcu)
+	__rcu_dereference_check((p), __UNIQUE_ID(rcu), \
+				(c) || rcu_read_lock_held(), __rcu)
 /**
  * rcu_dereference_bh_check() - rcu_dereference_bh with debug checking
  * rcu_read_lock() but also rcu_read_lock_bh() into account.
  */
 #define rcu_dereference_bh_check(p, c) \
-	__rcu_dereference_check((p), (c) || rcu_read_lock_bh_held(), __rcu)
+	__rcu_dereference_check((p), __UNIQUE_ID(rcu), \
+				(c) || rcu_read_lock_bh_held(), __rcu)
 /**
  * rcu_dereference_sched_check() - rcu_dereference_sched with debug checking
  * only rcu_read_lock() but also rcu_read_lock_sched() into account.
  */
 #define rcu_dereference_sched_check(p, c) \
-	__rcu_dereference_check((p), (c) || rcu_read_lock_sched_held(), \
+	__rcu_dereference_check((p), __UNIQUE_ID(rcu), \
+				(c) || rcu_read_lock_sched_held(), \
 				__rcu)
 /*
  * The no-tracing version of rcu_dereference_raw() must not call
  * rcu_read_lock_held().
  */
-#define rcu_dereference_raw_check(p) __rcu_dereference_check((p), 1, __rcu)
+#define rcu_dereference_raw_check(p) \
+	__rcu_dereference_check((p), __UNIQUE_ID(rcu), 1, __rcu)
 /**
  * rcu_dereference_protected() - fetch RCU pointer when updates prevented
  * but very ugly failures.
  */
 #define rcu_dereference_protected(p, c) \
-	__rcu_dereference_protected((p), (c), __rcu)
+	__rcu_dereference_protected((p), __UNIQUE_ID(rcu), (c), __rcu)
 /**

include/linux/rcutiny.h+1−1

@@ -85,7 +85,7 @@ static inline void rcu_irq_enter_irqson(void) { } · 84 unmodified lines
85	static inline void rcu_irq_exit(void) { }	85	static inline void rcu_irq_exit(void) { }
86	static inline void rcu_irq_exit_check_preempt(void) { }	86	static inline void rcu_irq_exit_check_preempt(void) { }
87	#define rcu_is_idle_cpu(cpu) \	87	#define rcu_is_idle_cpu(cpu) \
88	(is_idle_task(current) && !in_nmi() && !in_irq() && !in_serving_softirq())	88	(is_idle_task(current) && !in_nmi() && !in_hardirq() && !in_serving_softirq())
89	static inline void exit_rcu(void) { }	89	static inline void exit_rcu(void) { }
90	static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)	90	static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)
91	{	91	{

include/linux/srcu.h+2−1

@@ -117,7 +117,8 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp) · 116 unmodified lines
117	* lockdep_is_held() calls.	117	* lockdep_is_held() calls.
118	*/	118	*/
119	#define srcu_dereference_check(p, ssp, c) \	119	#define srcu_dereference_check(p, ssp, c) \
120	__rcu_dereference_check((p), (c) \|\| srcu_read_lock_held(ssp), __rcu)	120	__rcu_dereference_check((p), __UNIQUE_ID(rcu), \
		121	(c) \|\| srcu_read_lock_held(ssp), __rcu)
121		122
122	/**	123	/**
123	* srcu_dereference - fetch SRCU-protected pointer for later dereferencing	124	* srcu_dereference - fetch SRCU-protected pointer for later dereferencing

include/linux/torture.h+2−7

 		pr_alert("%s" TORTURE_FLAG " %s\n", torture_type, s);		\
 	}									\
 } while (0)
-#define VERBOSE_TOROUT_ERRSTRING(s) \
+#define TOROUT_ERRSTRING(s) \
-do {										\
+	pr_alert("%s" TORTURE_FLAG "!!! %s\n", torture_type, s)
-	if (verbose) {								\
-		verbose_torout_sleep();						\
-		pr_alert("%s" TORTURE_FLAG "!!! %s\n", torture_type, s);	\
-	}									\
-} while (0)
 void verbose_torout_sleep(void);
 #define torture_init_error(firsterr)						\

kernel/locking/locktorture.c+2−2

@@ -1047,7 +1047,7 @@ static int __init lock_torture_init(void) · 1046 unmodified lines
1047	sizeof(writer_tasks[0]),	1047	sizeof(writer_tasks[0]),
1048	GFP_KERNEL);	1048	GFP_KERNEL);
1049	if (writer_tasks == NULL) {	1049	if (writer_tasks == NULL) {
1050	VERBOSE_TOROUT_ERRSTRING("writer_tasks: Out of memory");	1050	TOROUT_ERRSTRING("writer_tasks: Out of memory");
1051	firsterr = -ENOMEM;	1051	firsterr = -ENOMEM;
1052	goto unwind;	1052	goto unwind;
1053	}	1053	}
@@ -1058,7 +1058,7 @@ static int __init lock_torture_init(void) · 4 unmodified lines
1058	sizeof(reader_tasks[0]),	1058	sizeof(reader_tasks[0]),
1059	GFP_KERNEL);	1059	GFP_KERNEL);
1060	if (reader_tasks == NULL) {	1060	if (reader_tasks == NULL) {
1061	VERBOSE_TOROUT_ERRSTRING("reader_tasks: Out of memory");	1061	TOROUT_ERRSTRING("reader_tasks: Out of memory");
1062	kfree(writer_tasks);	1062	kfree(writer_tasks);
1063	writer_tasks = NULL;	1063	writer_tasks = NULL;
1064	firsterr = -ENOMEM;	1064	firsterr = -ENOMEM;

kernel/rcu/Kconfig+1−19

@@ -112,7 +112,7 @@ config RCU_STALL_COMMON · 111 unmodified lines
112	making these warnings mandatory for the tree variants.	112	making these warnings mandatory for the tree variants.
113		113
114	config RCU_NEED_SEGCBLIST	114	config RCU_NEED_SEGCBLIST
115	def_bool ( TREE_RCU \|\| TREE_SRCU )	115	def_bool ( TREE_RCU \|\| TREE_SRCU \|\| TASKS_RCU_GENERIC )
116		116
117	config RCU_FANOUT	117	config RCU_FANOUT
118	int "Tree-based hierarchical RCU fanout value"	118	int "Tree-based hierarchical RCU fanout value"
@@ -169,24 +169,6 @@ config RCU_FANOUT_LEAF · 50 unmodified lines
169		169
170	Take the default if unsure.	170	Take the default if unsure.
171		171
172	config RCU_FAST_NO_HZ
173	bool "Accelerate last non-dyntick-idle CPU's grace periods"
174	depends on NO_HZ_COMMON && SMP && RCU_EXPERT
175	default n
176	help
177	This option permits CPUs to enter dynticks-idle state even if
178	they have RCU callbacks queued, and prevents RCU from waking
179	these CPUs up more than roughly once every four jiffies (by
180	default, you can adjust this using the rcutree.rcu_idle_gp_delay
181	parameter), thus improving energy efficiency. On the other
182	hand, this option increases the duration of RCU grace periods,
183	for example, slowing down synchronize_rcu().
184
185	Say Y if energy efficiency is critically important, and you
186	don't care about increased grace-period durations.
187
188	Say N if you are unsure.
189
190	config RCU_BOOST	172	config RCU_BOOST
191	bool "Enable RCU priority boosting"	173	bool "Enable RCU priority boosting"
192	depends on (RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT) \|\| PREEMPT_RT	174	depends on (RT_MUTEXES && PREEMPT_RCU && RCU_EXPERT) \|\| PREEMPT_RT

kernel/rcu/rcu_segcblist.c+4−6

 }
 /*
- * Mark the specified rcu_segcblist structure as offloaded.
+ * Mark the specified rcu_segcblist structure as offloaded (or not)
  */
 void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload)
 {
-	if (offload) {
+	if (offload)
-		rcu_segcblist_clear_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY);
+		rcu_segcblist_set_flags(rsclp, SEGCBLIST_LOCKING | SEGCBLIST_OFFLOADED);
-		rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
+	else
-	} else {
 		rcu_segcblist_clear_flags(rsclp, SEGCBLIST_OFFLOADED);
-	}
 }
 /*

kernel/rcu/rcu_segcblist.h+7−5

@@ -80,11 +80,14 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp) · 79 unmodified lines
80	return rcu_segcblist_test_flags(rsclp, SEGCBLIST_ENABLED);	80	return rcu_segcblist_test_flags(rsclp, SEGCBLIST_ENABLED);
81	}	81	}
82		82
83	/* Is the specified rcu_segcblist offloaded, or is SEGCBLIST_SOFTIRQ_ONLY set? */	83	/*
		84	* Is the specified rcu_segcblist NOCB offloaded (or in the middle of the
		85	* [de]offloading process)?
		86	*/
84	static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)	87	static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
85	{	88	{
86	if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&	89	if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
87	!rcu_segcblist_test_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY))	90	rcu_segcblist_test_flags(rsclp, SEGCBLIST_LOCKING))
88	return true;	91	return true;
89		92
90	return false;	93	return false;
@@ -92,9 +95,8 @@ static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp) · 1 unmodified line
92		95
93	static inline bool rcu_segcblist_completely_offloaded(struct rcu_segcblist *rsclp)	96	static inline bool rcu_segcblist_completely_offloaded(struct rcu_segcblist *rsclp)
94	{	97	{
95	int flags = SEGCBLIST_KTHREAD_CB \| SEGCBLIST_KTHREAD_GP \| SEGCBLIST_OFFLOADED;	98	if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
96		99	!rcu_segcblist_test_flags(rsclp, SEGCBLIST_RCU_CORE))
97	if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) && (rsclp->flags & flags) == flags)
98	return true;	100	return true;
99		101
100	return false;	102	return false;

kernel/rcu/rcuscale.c+7−7

@@ -50,8 +50,8 @@ MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.ibm.com>"); · 49 unmodified lines
50	pr_alert("%s" SCALE_FLAG " %s\n", scale_type, s)	50	pr_alert("%s" SCALE_FLAG " %s\n", scale_type, s)
51	#define VERBOSE_SCALEOUT_STRING(s) \	51	#define VERBOSE_SCALEOUT_STRING(s) \
52	do { if (verbose) pr_alert("%s" SCALE_FLAG " %s\n", scale_type, s); } while (0)	52	do { if (verbose) pr_alert("%s" SCALE_FLAG " %s\n", scale_type, s); } while (0)
53	#define VERBOSE_SCALEOUT_ERRSTRING(s) \	53	#define SCALEOUT_ERRSTRING(s) \
54	do { if (verbose) pr_alert("%s" SCALE_FLAG "!!! %s\n", scale_type, s); } while (0)	54	pr_alert("%s" SCALE_FLAG "!!! %s\n", scale_type, s)
55		55
56	/*	56	/*
57	* The intended use cases for the nreaders and nwriters module parameters	57	* The intended use cases for the nreaders and nwriters module parameters
@@ -514,11 +514,11 @@ rcu_scale_cleanup(void) · 456 unmodified lines
514	* during the mid-boot phase, so have to wait till the end.	514	* during the mid-boot phase, so have to wait till the end.
515	*/	515	*/
516	if (rcu_gp_is_expedited() && !rcu_gp_is_normal() && !gp_exp)	516	if (rcu_gp_is_expedited() && !rcu_gp_is_normal() && !gp_exp)
517	VERBOSE_SCALEOUT_ERRSTRING("All grace periods expedited, no normal ones to measure!");	517	SCALEOUT_ERRSTRING("All grace periods expedited, no normal ones to measure!");
518	if (rcu_gp_is_normal() && gp_exp)	518	if (rcu_gp_is_normal() && gp_exp)
519	VERBOSE_SCALEOUT_ERRSTRING("All grace periods normal, no expedited ones to measure!");	519	SCALEOUT_ERRSTRING("All grace periods normal, no expedited ones to measure!");
520	if (gp_exp && gp_async)	520	if (gp_exp && gp_async)
521	VERBOSE_SCALEOUT_ERRSTRING("No expedited async GPs, so went with async!");	521	SCALEOUT_ERRSTRING("No expedited async GPs, so went with async!");
522		522
523	if (torture_cleanup_begin())	523	if (torture_cleanup_begin())
524	return;	524	return;
@@ -845,7 +845,7 @@ rcu_scale_init(void) · 320 unmodified lines
845	reader_tasks = kcalloc(nrealreaders, sizeof(reader_tasks[0]),	845	reader_tasks = kcalloc(nrealreaders, sizeof(reader_tasks[0]),
846	GFP_KERNEL);	846	GFP_KERNEL);
847	if (reader_tasks == NULL) {	847	if (reader_tasks == NULL) {
848	VERBOSE_SCALEOUT_ERRSTRING("out of memory");	848	SCALEOUT_ERRSTRING("out of memory");
849	firsterr = -ENOMEM;	849	firsterr = -ENOMEM;
850	goto unwind;	850	goto unwind;
851	}	851	}
@@ -865,7 +865,7 @@ rcu_scale_init(void) · 13 unmodified lines
865	kcalloc(nrealwriters, sizeof(*writer_n_durations),	865	kcalloc(nrealwriters, sizeof(*writer_n_durations),
866	GFP_KERNEL);	866	GFP_KERNEL);
867	if (!writer_tasks \|\| !writer_durations \|\| !writer_n_durations) {	867	if (!writer_tasks \|\| !writer_durations \|\| !writer_n_durations) {
868	VERBOSE_SCALEOUT_ERRSTRING("out of memory");	868	SCALEOUT_ERRSTRING("out of memory");
869	firsterr = -ENOMEM;	869	firsterr = -ENOMEM;
870	goto unwind;	870	goto unwind;
871	}	871	}

kernel/rcu/rcutorture.c+174−60

 #include <linux/oom.h>
 #include <linux/tick.h>
 #include <linux/rcupdate_trace.h>
+#include <linux/nmi.h>
 #include "rcu.h"
 MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.ibm.com> and Josh Triplett <josh@joshtriplett.org>");
 /* Bits for ->extendables field, extendables param, and related definitions. */
-#define RCUTORTURE_RDR_SHIFT	 8	/* Put SRCU index in upper bits. */
+#define RCUTORTURE_RDR_SHIFT_1	 8	/* Put SRCU index in upper bits. */
-#define RCUTORTURE_RDR_MASK	 ((1 << RCUTORTURE_RDR_SHIFT) - 1)
+#define RCUTORTURE_RDR_MASK_1	 (1 << RCUTORTURE_RDR_SHIFT_1)
+#define RCUTORTURE_RDR_SHIFT_2	 9	/* Put SRCU index in upper bits. */
+#define RCUTORTURE_RDR_MASK_2	 (1 << RCUTORTURE_RDR_SHIFT_2)
 #define RCUTORTURE_RDR_BH	 0x01	/* Extend readers by disabling bh. */
 #define RCUTORTURE_RDR_IRQ	 0x02	/*  ... disabling interrupts. */
 #define RCUTORTURE_RDR_PREEMPT	 0x04	/*  ... disabling preemption. */
 #define RCUTORTURE_RDR_RBH	 0x08	/*  ... rcu_read_lock_bh(). */
 #define RCUTORTURE_RDR_SCHED	 0x10	/*  ... rcu_read_lock_sched(). */
-#define RCUTORTURE_RDR_RCU	 0x20	/*  ... entering another RCU reader. */
+#define RCUTORTURE_RDR_RCU_1	 0x20	/*  ... entering another RCU reader. */
-#define RCUTORTURE_RDR_NBITS	 6	/* Number of bits defined above. */
+#define RCUTORTURE_RDR_RCU_2	 0x40	/*  ... entering another RCU reader. */
+#define RCUTORTURE_RDR_NBITS	 7	/* Number of bits defined above. */
 #define RCUTORTURE_MAX_EXTEND	 \
 	(RCUTORTURE_RDR_BH | RCUTORTURE_RDR_IRQ | RCUTORTURE_RDR_PREEMPT | \
 	 RCUTORTURE_RDR_RBH | RCUTORTURE_RDR_SCHED)
 	      "Duration of fqs bursts (us), 0 to disable");
 torture_param(int, fqs_holdoff, 0, "Holdoff time within fqs bursts (us)");
 torture_param(int, fqs_stutter, 3, "Wait time between fqs bursts (s)");
-torture_param(bool, fwd_progress, 1, "Test grace-period forward progress");
+torture_param(int, fwd_progress, 1, "Test grace-period forward progress");
 torture_param(int, fwd_progress_div, 4, "Fraction of CPU stall to wait");
 torture_param(int, fwd_progress_holdoff, 60,
 	      "Time between forward-progress tests (s)");
 torture_param(int, stall_cpu, 0, "Stall duration (s), zero to disable.");
 torture_param(int, stall_cpu_holdoff, 10,
 	     "Time to wait before starting stall (s).");
+torture_param(bool, stall_no_softlockup, false,
+	     "Avoid softlockup warning during cpu stall.");
 torture_param(int, stall_cpu_irqsoff, 0, "Disable interrupts while stalling.");
 torture_param(int, stall_cpu_block, 0, "Sleep while stalling.");
 torture_param(int, stall_gp_kthread, 0,
 static struct task_struct *fqs_task;
 static struct task_struct *boost_tasks[NR_CPUS];
 static struct task_struct *stall_task;
-static struct task_struct *fwd_prog_task;
+static struct task_struct **fwd_prog_tasks;
 static struct task_struct **barrier_cbs_tasks;
 static struct task_struct *barrier_task;
 static struct task_struct *read_exit_task;
 	void (*gp_kthread_dbg)(void);
 	bool (*check_boost_failed)(unsigned long gp_state, int *cpup);
 	int (*stall_dur)(void);
+	long cbflood_max;
 	int irq_capable;
 	int can_boost;
 	int extendables;
 	int slow_gps;
+	int no_pi_lock;
 	const char *name;
 };
 	.cb_barrier	= srcu_torture_barrier,
 	.stats		= srcu_torture_stats,
 	.irq_capable	= 1,
+	.no_pi_lock	= IS_ENABLED(CONFIG_TINY_SRCU),
 	.name		= "srcu"
 };
 	.cb_barrier	= srcu_torture_barrier,
 	.stats		= srcu_torture_stats,
 	.irq_capable	= 1,
+	.no_pi_lock	= IS_ENABLED(CONFIG_TINY_SRCU),
 	.name		= "srcud"
 };
 	.cb_barrier	= srcu_torture_barrier,
 	.stats		= srcu_torture_stats,
 	.irq_capable	= 1,
+	.no_pi_lock	= IS_ENABLED(CONFIG_TINY_SRCU),
 	.extendables	= RCUTORTURE_MAX_EXTEND,
 	.name		= "busted_srcud"
 };
 	.call		= call_rcu_tasks_rude,
 	.cb_barrier	= rcu_barrier_tasks_rude,
 	.gp_kthread_dbg	= show_rcu_tasks_rude_gp_kthread,
+	.cbflood_max	= 50000,
 	.fqs		= NULL,
 	.stats		= NULL,
 	.irq_capable	= 1,
 	.call		= call_rcu_tasks_trace,
 	.cb_barrier	= rcu_barrier_tasks_trace,
 	.gp_kthread_dbg	= show_rcu_tasks_trace_gp_kthread,
+	.cbflood_max	= 50000,
 	.fqs		= NULL,
 	.stats		= NULL,
 	.irq_capable	= 1,
 				  struct rt_read_seg *rtrsp)
 {
 	unsigned long flags;
-	int idxnew = -1;
+	int idxnew1 = -1;
-	int idxold = *readstate;
+	int idxnew2 = -1;
+	int idxold1 = *readstate;
+	int idxold2 = idxold1;
 	int statesnew = ~*readstate & newstate;
 	int statesold = *readstate & ~newstate;
-	WARN_ON_ONCE(idxold < 0);
+	WARN_ON_ONCE(idxold2 < 0);
-	WARN_ON_ONCE((idxold >> RCUTORTURE_RDR_SHIFT) > 1);
+	WARN_ON_ONCE((idxold2 >> RCUTORTURE_RDR_SHIFT_2) > 1);
 	rtrsp->rt_readstate = newstate;
 	/* First, put new protection in place to avoid critical-section gap. */
 		preempt_disable();
 	if (statesnew & RCUTORTURE_RDR_SCHED)
 		rcu_read_lock_sched();
-	if (statesnew & RCUTORTURE_RDR_RCU)
+	if (statesnew & RCUTORTURE_RDR_RCU_1)
-		idxnew = cur_ops->readlock() << RCUTORTURE_RDR_SHIFT;
+		idxnew1 = (cur_ops->readlock() & 0x1) << RCUTORTURE_RDR_SHIFT_1;
+	if (statesnew & RCUTORTURE_RDR_RCU_2)
+		idxnew2 = (cur_ops->readlock() & 0x1) << RCUTORTURE_RDR_SHIFT_2;
 	/*
 	 * Next, remove old protection, in decreasing order of strength
 		local_bh_enable();
 	if (statesold & RCUTORTURE_RDR_RBH)
 		rcu_read_unlock_bh();
-	if (statesold & RCUTORTURE_RDR_RCU) {
+	if (statesold & RCUTORTURE_RDR_RCU_2) {
-		bool lockit = !statesnew && !(torture_random(trsp) & 0xffff);
+		cur_ops->readunlock((idxold2 >> RCUTORTURE_RDR_SHIFT_2) & 0x1);
+		WARN_ON_ONCE(idxnew2 != -1);
+		idxold2 = 0;
+	}
+	if (statesold & RCUTORTURE_RDR_RCU_1) {
+		bool lockit;
+		lockit = !cur_ops->no_pi_lock && !statesnew && !(torture_random(trsp) & 0xffff);
 		if (lockit)
 			raw_spin_lock_irqsave(&current->pi_lock, flags);
-		cur_ops->readunlock(idxold >> RCUTORTURE_RDR_SHIFT);
+		cur_ops->readunlock((idxold1 >> RCUTORTURE_RDR_SHIFT_1) & 0x1);
+		WARN_ON_ONCE(idxnew1 != -1);
+		idxold1 = 0;
 		if (lockit)
 			raw_spin_unlock_irqrestore(&current->pi_lock, flags);
 	}
 		cur_ops->read_delay(trsp, rtrsp);
 	/* Update the reader state. */
-	if (idxnew == -1)
+	if (idxnew1 == -1)
-		idxnew = idxold & ~RCUTORTURE_RDR_MASK;
+		idxnew1 = idxold1 & RCUTORTURE_RDR_MASK_1;
-	WARN_ON_ONCE(idxnew < 0);
+	WARN_ON_ONCE(idxnew1 < 0);
-	WARN_ON_ONCE((idxnew >> RCUTORTURE_RDR_SHIFT) > 1);
+	if (WARN_ON_ONCE((idxnew1 >> RCUTORTURE_RDR_SHIFT_1) > 1))
-	*readstate = idxnew | newstate;
+		pr_info("Unexpected idxnew1 value of %#x\n", idxnew1);
-	WARN_ON_ONCE((*readstate >> RCUTORTURE_RDR_SHIFT) < 0);
+	if (idxnew2 == -1)
-	WARN_ON_ONCE((*readstate >> RCUTORTURE_RDR_SHIFT) > 1);
+		idxnew2 = idxold2 & RCUTORTURE_RDR_MASK_2;
+	WARN_ON_ONCE(idxnew2 < 0);
+	WARN_ON_ONCE((idxnew2 >> RCUTORTURE_RDR_SHIFT_2) > 1);
+	*readstate = idxnew1 | idxnew2 | newstate;
+	WARN_ON_ONCE(*readstate < 0);
+	if (WARN_ON_ONCE((*readstate >> RCUTORTURE_RDR_SHIFT_2) > 1))
+		pr_info("Unexpected idxnew2 value of %#x\n", idxnew2);
 }
 /* Return the biggest extendables mask given current RCU and boot parameters. */
 	WARN_ON_ONCE(extendables & ~RCUTORTURE_MAX_EXTEND);
 	mask = extendables & RCUTORTURE_MAX_EXTEND & cur_ops->extendables;
-	mask = mask | RCUTORTURE_RDR_RCU;
+	mask = mask | RCUTORTURE_RDR_RCU_1 | RCUTORTURE_RDR_RCU_2;
 	return mask;
 }
 	unsigned long preempts_irq = preempts | RCUTORTURE_RDR_IRQ;
 	unsigned long bhs = RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH;
-	WARN_ON_ONCE(mask >> RCUTORTURE_RDR_SHIFT);
+	WARN_ON_ONCE(mask >> RCUTORTURE_RDR_SHIFT_1);
 	/* Mostly only one bit (need preemption!), sometimes lots of bits. */
 	if (!(randmask1 & 0x7))
 		mask = mask & randmask2;
 	else
 		mask = mask & (1 << (randmask2 % RCUTORTURE_RDR_NBITS));
+	// Can't have nested RCU reader without outer RCU reader.
+	if (!(mask & RCUTORTURE_RDR_RCU_1) && (mask & RCUTORTURE_RDR_RCU_2)) {
+		if (oldmask & RCUTORTURE_RDR_RCU_1)
+			mask &= ~RCUTORTURE_RDR_RCU_2;
+		else
+			mask |= RCUTORTURE_RDR_RCU_1;
+	}
 	/*
 	 * Can't enable bh w/irq disabled.
 	 */
 			mask |= oldmask & bhs;
 	}
-	return mask ?: RCUTORTURE_RDR_RCU;
+	return mask ?: RCUTORTURE_RDR_RCU_1;
 }
 /*
 			  rcu_torture_writer_state,
 			  cookie, cur_ops->get_gp_state());
 	rcutorture_one_extend(&readstate, 0, trsp, rtrsp);
-	WARN_ON_ONCE(readstate & RCUTORTURE_RDR_MASK);
+	WARN_ON_ONCE(readstate);
 	// This next splat is expected behavior if leakpointer, especially
 	// for CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels.
 	WARN_ON_ONCE(leakpointer && READ_ONCE(p->rtort_pipe_count) > 1);
 #else
 				schedule_timeout_uninterruptible(HZ);
 #endif
+			} else if (stall_no_softlockup) {
+				touch_softlockup_watchdog();
 			}
 		if (stall_cpu_irqsoff)
 			local_irq_enable();
 	unsigned long rcu_fwd_startat;
 	struct rcu_launder_hist n_launders_hist[N_LAUNDERS_HIST];
 	unsigned long rcu_launder_gp_seq_start;
+	int rcu_fwd_id;
 };
 static DEFINE_MUTEX(rcu_fwd_mutex);
 static struct rcu_fwd *rcu_fwds;
+static unsigned long rcu_fwd_seq;
+static atomic_long_t rcu_fwd_max_cbs;
 static bool rcu_fwd_emergency_stop;
 static void rcu_torture_fwd_cb_hist(struct rcu_fwd *rfp)
 	for (i = ARRAY_SIZE(rfp->n_launders_hist) - 1; i > 0; i--)
 		if (rfp->n_launders_hist[i].n_launders > 0)
 			break;
-	pr_alert("%s: Callback-invocation histogram (duration %lu jiffies):",
+	mutex_lock(&rcu_fwd_mutex); // Serialize histograms.
-		 __func__, jiffies - rfp->rcu_fwd_startat);
+	pr_alert("%s: Callback-invocation histogram %d (duration %lu jiffies):",
+		 __func__, rfp->rcu_fwd_id, jiffies - rfp->rcu_fwd_startat);
 	gps_old = rfp->rcu_launder_gp_seq_start;
 	for (j = 0; j <= i; j++) {
 		gps = rfp->n_launders_hist[j].launder_gp_seq;
 		gps_old = gps;
 	}
 	pr_cont("\n");
+	mutex_unlock(&rcu_fwd_mutex);
 }
 /* Callback function for continuous-flood RCU callbacks. */
 		cver = READ_ONCE(rcu_torture_current_version) - cver;
 		gps = rcutorture_seq_diff(cur_ops->get_gp_seq(), gps);
 		WARN_ON(!cver && gps < 2);
-		pr_alert("%s: Duration %ld cver %ld gps %ld\n", __func__, dur, cver, gps);
+		pr_alert("%s: %d Duration %ld cver %ld gps %ld\n", __func__,
+			 rfp->rcu_fwd_id, dur, cver, gps);
 	}
 	if (selfpropcb) {
 		WRITE_ONCE(fcs.stop, 1);
 			rfp->rcu_fwd_cb_head = rfcpn;
 			n_launders++;
 			n_launders_sa++;
-		} else {
+		} else if (!cur_ops->cbflood_max || cur_ops->cbflood_max > n_max_cbs) {
 			rfcp = kmalloc(sizeof(*rfcp), GFP_KERNEL);
 			if (WARN_ON_ONCE(!rfcp)) {
 				schedule_timeout_interruptible(1);
 			n_launders_sa = 0;
 			rfcp->rfc_gps = 0;
 			rfcp->rfc_rfp = rfp;
+		} else {
+			rfcp = NULL;
 		}
-		cur_ops->call(&rfcp->rh, rcu_torture_fwd_cb_cr);
+		if (rfcp)
+			cur_ops->call(&rfcp->rh, rcu_torture_fwd_cb_cr);
 		rcu_torture_fwd_prog_cond_resched(n_launders + n_max_cbs);
 		if (tick_nohz_full_enabled()) {
 			local_irq_save(flags);
 			 n_launders + n_max_cbs - n_launders_cb_snap,
 			 n_launders, n_launders_sa,
 			 n_max_gps, n_max_cbs, cver, gps);
+		atomic_long_add(n_max_cbs, &rcu_fwd_max_cbs);
 		rcu_torture_fwd_cb_hist(rfp);
 	}
 	schedule_timeout_uninterruptible(HZ); /* Let CBs drain. */
 static int rcutorture_oom_notify(struct notifier_block *self,
 				 unsigned long notused, void *nfreed)
 {
+	int i;
+	long ncbs;
 	struct rcu_fwd *rfp;
 	mutex_lock(&rcu_fwd_mutex);
 	}
 	WARN(1, "%s invoked upon OOM during forward-progress testing.\n",
 	     __func__);
-	rcu_torture_fwd_cb_hist(rfp);
+	for (i = 0; i < fwd_progress; i++) {
-	rcu_fwd_progress_check(1 + (jiffies - READ_ONCE(rfp->rcu_fwd_startat)) / 2);
+		rcu_torture_fwd_cb_hist(&rfp[i]);
+		rcu_fwd_progress_check(1 + (jiffies - READ_ONCE(rfp[i].rcu_fwd_startat)) / 2);
+	}
 	WRITE_ONCE(rcu_fwd_emergency_stop, true);
 	smp_mb(); /* Emergency stop before free and wait to avoid hangs. */
-	pr_info("%s: Freed %lu RCU callbacks.\n",
+	ncbs = 0;
-		__func__, rcu_torture_fwd_prog_cbfree(rfp));
+	for (i = 0; i < fwd_progress; i++)
+		ncbs += rcu_torture_fwd_prog_cbfree(&rfp[i]);
+	pr_info("%s: Freed %lu RCU callbacks.\n", __func__, ncbs);
 	rcu_barrier();
-	pr_info("%s: Freed %lu RCU callbacks.\n",
+	ncbs = 0;
-		__func__, rcu_torture_fwd_prog_cbfree(rfp));
+	for (i = 0; i < fwd_progress; i++)
+		ncbs += rcu_torture_fwd_prog_cbfree(&rfp[i]);
+	pr_info("%s: Freed %lu RCU callbacks.\n", __func__, ncbs);
 	rcu_barrier();
-	pr_info("%s: Freed %lu RCU callbacks.\n",
+	ncbs = 0;
-		__func__, rcu_torture_fwd_prog_cbfree(rfp));
+	for (i = 0; i < fwd_progress; i++)
+		ncbs += rcu_torture_fwd_prog_cbfree(&rfp[i]);
+	pr_info("%s: Freed %lu RCU callbacks.\n", __func__, ncbs);
 	smp_mb(); /* Frees before return to avoid redoing OOM. */
 	(*(unsigned long *)nfreed)++; /* Forward progress CBs freed! */
 	pr_info("%s returning after OOM processing.\n", __func__);
 /* Carry out grace-period forward-progress testing. */
 static int rcu_torture_fwd_prog(void *args)
 {
+	bool firsttime = true;
+	long max_cbs;
 	int oldnice = task_nice(current);
+	unsigned long oldseq = READ_ONCE(rcu_fwd_seq);
 	struct rcu_fwd *rfp = args;
 	int tested = 0;
 	int tested_tries = 0;
 	if (!IS_ENABLED(CONFIG_SMP) || !IS_ENABLED(CONFIG_RCU_BOOST))
 		set_user_nice(current, MAX_NICE);
 	do {
-		schedule_timeout_interruptible(fwd_progress_holdoff * HZ);
+		if (!rfp->rcu_fwd_id) {
-		WRITE_ONCE(rcu_fwd_emergency_stop, false);
+			schedule_timeout_interruptible(fwd_progress_holdoff * HZ);
-		if (!IS_ENABLED(CONFIG_TINY_RCU) ||
+			WRITE_ONCE(rcu_fwd_emergency_stop, false);
-		    rcu_inkernel_boot_has_ended())
+			if (!firsttime) {
-			rcu_torture_fwd_prog_nr(rfp, &tested, &tested_tries);
+				max_cbs = atomic_long_xchg(&rcu_fwd_max_cbs, 0);
-		if (rcu_inkernel_boot_has_ended())
+				pr_alert("%s n_max_cbs: %ld\n", __func__, max_cbs);
+			}
+			firsttime = false;
+			WRITE_ONCE(rcu_fwd_seq, rcu_fwd_seq + 1);
+		} else {
+			while (READ_ONCE(rcu_fwd_seq) == oldseq)
+				schedule_timeout_interruptible(1);
+			oldseq = READ_ONCE(rcu_fwd_seq);
+		}
+		pr_alert("%s: Starting forward-progress test %d\n", __func__, rfp->rcu_fwd_id);
+		if (rcu_inkernel_boot_has_ended() && torture_num_online_cpus() > rfp->rcu_fwd_id)
 			rcu_torture_fwd_prog_cr(rfp);
+		if ((cur_ops->stall_dur && cur_ops->stall_dur() > 0) &&
+		    (!IS_ENABLED(CONFIG_TINY_RCU) ||
+		     (rcu_inkernel_boot_has_ended() &&
+		      torture_num_online_cpus() > rfp->rcu_fwd_id)))
+			rcu_torture_fwd_prog_nr(rfp, &tested, &tested_tries);
 		/* Avoid slow periods, better to test when busy. */
 		if (stutter_wait("rcu_torture_fwd_prog"))
 			sched_set_normal(current, oldnice);
 	} while (!torture_must_stop());
 	/* Short runs might not contain a valid forward-progress attempt. */
-	WARN_ON(!tested && tested_tries >= 5);
+	if (!rfp->rcu_fwd_id) {
-	pr_alert("%s: tested %d tested_tries %d\n", __func__, tested, tested_tries);
+		WARN_ON(!tested && tested_tries >= 5);
+		pr_alert("%s: tested %d tested_tries %d\n", __func__, tested, tested_tries);
+	}
 	torture_kthread_stopping("rcu_torture_fwd_prog");
 	return 0;
 }
 /* If forward-progress checking is requested and feasible, spawn the thread. */
 static int __init rcu_torture_fwd_prog_init(void)
 {
+	int i;
+	int ret = 0;
 	struct rcu_fwd *rfp;
 	if (!fwd_progress)
 		return 0; /* Not requested, so don't do it. */
+	if (fwd_progress >= nr_cpu_ids) {
+		VERBOSE_TOROUT_STRING("rcu_torture_fwd_prog_init: Limiting fwd_progress to # CPUs.\n");
+		fwd_progress = nr_cpu_ids;
+	} else if (fwd_progress < 0) {
+		fwd_progress = nr_cpu_ids;
+	}
 	if ((!cur_ops->sync && !cur_ops->call) ||
-	    !cur_ops->stall_dur || cur_ops->stall_dur() <= 0 || cur_ops == &rcu_busted_ops) {
+	    (!cur_ops->cbflood_max && (!cur_ops->stall_dur || cur_ops->stall_dur() <= 0)) ||
+	    cur_ops == &rcu_busted_ops) {
 		VERBOSE_TOROUT_STRING("rcu_torture_fwd_prog_init: Disabled, unsupported by RCU flavor under test");
+		fwd_progress = 0;
 		return 0;
 	}
 	if (stall_cpu > 0) {
 		VERBOSE_TOROUT_STRING("rcu_torture_fwd_prog_init: Disabled, conflicts with CPU-stall testing");
+		fwd_progress = 0;
 		if (IS_MODULE(CONFIG_RCU_TORTURE_TEST))
 			return -EINVAL; /* In module, can fail back to user. */
 		WARN_ON(1); /* Make sure rcutorture notices conflict. */
 		fwd_progress_holdoff = 1;
 	if (fwd_progress_div <= 0)
 		fwd_progress_div = 4;
-	rfp = kzalloc(sizeof(*rfp), GFP_KERNEL);
+	rfp = kcalloc(fwd_progress, sizeof(*rfp), GFP_KERNEL);
-	if (!rfp)
+	fwd_prog_tasks = kcalloc(fwd_progress, sizeof(*fwd_prog_tasks), GFP_KERNEL);
+	if (!rfp || !fwd_prog_tasks) {
+		kfree(rfp);
+		kfree(fwd_prog_tasks);
+		fwd_prog_tasks = NULL;
+		fwd_progress = 0;
 		return -ENOMEM;
-	spin_lock_init(&rfp->rcu_fwd_lock);
+	}
-	rfp->rcu_fwd_cb_tail = &rfp->rcu_fwd_cb_head;
+	for (i = 0; i < fwd_progress; i++) {
+		spin_lock_init(&rfp[i].rcu_fwd_lock);
+		rfp[i].rcu_fwd_cb_tail = &rfp[i].rcu_fwd_cb_head;
+		rfp[i].rcu_fwd_id = i;
+	}
 	mutex_lock(&rcu_fwd_mutex);
 	rcu_fwds = rfp;
 	mutex_unlock(&rcu_fwd_mutex);
 	register_oom_notifier(&rcutorture_oom_nb);
-	return torture_create_kthread(rcu_torture_fwd_prog, rfp, fwd_prog_task);
+	for (i = 0; i < fwd_progress; i++) {
+		ret = torture_create_kthread(rcu_torture_fwd_prog, &rcu_fwds[i], fwd_prog_tasks[i]);
+		if (ret) {
+			fwd_progress = i;
+			return ret;
+		}
+	}
+	return 0;
 }
 static void rcu_torture_fwd_prog_cleanup(void)
 {
+	int i;
 	struct rcu_fwd *rfp;
-	torture_stop_kthread(rcu_torture_fwd_prog, fwd_prog_task);
+	if (!rcu_fwds || !fwd_prog_tasks)
-	rfp = rcu_fwds;
+		return;
+	for (i = 0; i < fwd_progress; i++)
+		torture_stop_kthread(rcu_torture_fwd_prog, fwd_prog_tasks[i]);
+	unregister_oom_notifier(&rcutorture_oom_nb);
 	mutex_lock(&rcu_fwd_mutex);
+	rfp = rcu_fwds;
 	rcu_fwds = NULL;
 	mutex_unlock(&rcu_fwd_mutex);
-	unregister_oom_notifier(&rcutorture_oom_nb);
 	kfree(rfp);
+	kfree(fwd_prog_tasks);
+	fwd_prog_tasks = NULL;
 }
 /* Callback function for RCU barrier testing. */
 				     &trs, "%s",
 				     "rcu_torture_read_exit_child");
 		if (IS_ERR(tsp)) {
-			VERBOSE_TOROUT_ERRSTRING("out of memory");
+			TOROUT_ERRSTRING("out of memory");
 			errexit = true;
 			tsp = NULL;
 			break;
 					   sizeof(fakewriter_tasks[0]),
 					   GFP_KERNEL);
 		if (fakewriter_tasks == NULL) {
-			VERBOSE_TOROUT_ERRSTRING("out of memory");
+			TOROUT_ERRSTRING("out of memory");
 			firsterr = -ENOMEM;
 			goto unwind;
 		}
 	rcu_torture_reader_mbchk = kcalloc(nrealreaders, sizeof(*rcu_torture_reader_mbchk),
 					   GFP_KERNEL);
 	if (!reader_tasks || !rcu_torture_reader_mbchk) {
-		VERBOSE_TOROUT_ERRSTRING("out of memory");
+		TOROUT_ERRSTRING("out of memory");
 		firsterr = -ENOMEM;
 		goto unwind;
 	}
 	if (nrealnocbers > 0) {
 		nocb_tasks = kcalloc(nrealnocbers, sizeof(nocb_tasks[0]), GFP_KERNEL);
 		if (nocb_tasks == NULL) {
-			VERBOSE_TOROUT_ERRSTRING("out of memory");
+			TOROUT_ERRSTRING("out of memory");
 			firsterr = -ENOMEM;
 			goto unwind;
 		}

kernel/rcu/refscale.c+24−26

 	pr_alert("%s" SCALE_FLAG s, scale_type, ## x)
 #define VERBOSE_SCALEOUT(s, x...) \
-	do { if (verbose) pr_alert("%s" SCALE_FLAG s, scale_type, ## x); } while (0)
+	do { \
+		if (verbose) \
+			pr_alert("%s" SCALE_FLAG s "\n", scale_type, ## x); \
+	} while (0)
 static atomic_t verbose_batch_ctr;
 	    (verbose_batched <= 0 ||							\
 	     !(atomic_inc_return(&verbose_batch_ctr) % verbose_batched))) {		\
 		schedule_timeout_uninterruptible(1);					\
-		pr_alert("%s" SCALE_FLAG s, scale_type, ## x);				\
+		pr_alert("%s" SCALE_FLAG s "\n", scale_type, ## x);			\
 	}										\
 } while (0)
-#define VERBOSE_SCALEOUT_ERRSTRING(s, x...) \
+#define SCALEOUT_ERRSTRING(s, x...) pr_alert("%s" SCALE_FLAG "!!! " s "\n", scale_type, ## x)
-	do { if (verbose) pr_alert("%s" SCALE_FLAG "!!! " s, scale_type, ## x); } while (0)
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("Joel Fernandes (Google) <joel@joelfernandes.org>");
 	char *buf;
 	u64 sum = 0;
-	buf = kmalloc(128 + nreaders * 32, GFP_KERNEL);
+	buf = kmalloc(800 + 64, GFP_KERNEL);
 	if (!buf)
 		return 0;
 	buf[0] = 0;
 		if (i % 5 == 0)
 			strcat(buf, "\n");
+		if (strlen(buf) >= 800) {
+			pr_alert("%s", buf);
+			buf[0] = 0;
+		}
 		strcat(buf, buf1);
 		sum += rt->last_duration_ns;
 	}
-	strcat(buf, "\n");
+	pr_alert("%s\n", buf);
-	SCALEOUT("%s\n", buf);
 	kfree(buf);
 	return sum;
 // point all the timestamps are printed.
 static int main_func(void *arg)
 {
-	bool errexit = false;
 	int exp, r;
 	char buf1[64];
 	char *buf;
 	VERBOSE_SCALEOUT("main_func task started");
 	result_avg = kzalloc(nruns * sizeof(*result_avg), GFP_KERNEL);
-	buf = kzalloc(64 + nruns * 32, GFP_KERNEL);
+	buf = kzalloc(800 + 64, GFP_KERNEL);
 	if (!result_avg || !buf) {
-		VERBOSE_SCALEOUT_ERRSTRING("out of memory");
+		SCALEOUT_ERRSTRING("out of memory");
-		errexit = true;
+		goto oom_exit;
 	}
 	if (holdoff)
 		schedule_timeout_interruptible(holdoff * HZ);
 	// Start exp readers up per experiment
 	for (exp = 0; exp < nruns && !torture_must_stop(); exp++) {
-		if (errexit)
-			break;
 		if (torture_must_stop())
 			goto end;
 	// Print the average of all experiments
 	SCALEOUT("END OF TEST. Calculating average duration per loop (nanoseconds)...\n");
-	if (!errexit) {
+	pr_alert("Runs\tTime(ns)\n");
-		buf[0] = 0;
-		strcat(buf, "\n");
-		strcat(buf, "Runs\tTime(ns)\n");
-	}
 	for (exp = 0; exp < nruns; exp++) {
 		u64 avg;
 		u32 rem;
-		if (errexit)
-			break;
 		avg = div_u64_rem(result_avg[exp], 1000, &rem);
 		sprintf(buf1, "%d\t%llu.%03u\n", exp + 1, avg, rem);
 		strcat(buf, buf1);
+		if (strlen(buf) >= 800) {
+			pr_alert("%s", buf);
+			buf[0] = 0;
+		}
 	}
-	if (!errexit)
+	pr_alert("%s", buf);
-		SCALEOUT("%s", buf);
+oom_exit:
 	// This will shutdown everything including us.
 	if (shutdown) {
 		shutdown_start = 1;
 	reader_tasks = kcalloc(nreaders, sizeof(reader_tasks[0]),
 			       GFP_KERNEL);
 	if (!reader_tasks) {
-		VERBOSE_SCALEOUT_ERRSTRING("out of memory");
+		SCALEOUT_ERRSTRING("out of memory");
 		firsterr = -ENOMEM;
 		goto unwind;
 	}
-	VERBOSE_SCALEOUT("Starting %d reader threads\n", nreaders);
+	VERBOSE_SCALEOUT("Starting %d reader threads", nreaders);
 	for (i = 0; i < nreaders; i++) {
 		firsterr = torture_create_kthread(ref_scale_reader, (void *)i,

kernel/rcu/srcutiny.c+1−1

@@ -99,7 +99,7 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx) · 98 unmodified lines
99	int newval = READ_ONCE(ssp->srcu_lock_nesting[idx]) - 1;	99	int newval = READ_ONCE(ssp->srcu_lock_nesting[idx]) - 1;
100		100
101	WRITE_ONCE(ssp->srcu_lock_nesting[idx], newval);	101	WRITE_ONCE(ssp->srcu_lock_nesting[idx], newval);
102	if (!newval && READ_ONCE(ssp->srcu_gp_waiting))	102	if (!newval && READ_ONCE(ssp->srcu_gp_waiting) && in_task())
103	swake_up_one(&ssp->srcu_wq);	103	swake_up_one(&ssp->srcu_wq);
104	}	104	}
105	EXPORT_SYMBOL_GPL(__srcu_read_unlock);	105	EXPORT_SYMBOL_GPL(__srcu_read_unlock);

kernel/rcu/tasks.h+394−82

@@ -6,6 +6,7 @@ · 5 unmodified lines
6	*/	6	*/
7		7
8	#ifdef CONFIG_TASKS_RCU_GENERIC	8	#ifdef CONFIG_TASKS_RCU_GENERIC
		9	#include "rcu_segcblist.h"
9		10
10	////////////////////////////////////////////////////////////////////////	11	////////////////////////////////////////////////////////////////////////
11	//	12	//
@@ -19,12 +20,34 @@ typedef void (postscan_func_t)(struct list_head hop); · 7 unmodified lines
19	typedef void (holdouts_func_t)(struct list_head hop, bool ndrpt, bool *frptp);	20	typedef void (holdouts_func_t)(struct list_head hop, bool ndrpt, bool *frptp);
20	typedef void (postgp_func_t)(struct rcu_tasks rtp);	21	typedef void (postgp_func_t)(struct rcu_tasks rtp);
21		22
		23	/**
		24	* struct rcu_tasks_percpu - Per-CPU component of definition for a Tasks-RCU-like mechanism.
		25	* @cblist: Callback list.
		26	* @lock: Lock protecting per-CPU callback list.
		27	* @rtp_jiffies: Jiffies counter value for statistics.
		28	* @rtp_n_lock_retries: Rough lock-contention statistic.
		29	* @rtp_work: Work queue for invoking callbacks.
		30	* @rtp_irq_work: IRQ work queue for deferred wakeups.
		31	* @barrier_q_head: RCU callback for barrier operation.
		32	* @cpu: CPU number corresponding to this entry.
		33	* @rtpp: Pointer to the rcu_tasks structure.
		34	*/
		35	struct rcu_tasks_percpu {
		36	struct rcu_segcblist cblist;
		37	raw_spinlock_t __private lock;
		38	unsigned long rtp_jiffies;
		39	unsigned long rtp_n_lock_retries;
		40	struct work_struct rtp_work;
		41	struct irq_work rtp_irq_work;
		42	struct rcu_head barrier_q_head;
		43	int cpu;
		44	struct rcu_tasks *rtpp;
		45	};
		46
22	/**	47	/**
23	* struct rcu_tasks - Definition for a Tasks-RCU-like mechanism.	48	* struct rcu_tasks - Definition for a Tasks-RCU-like mechanism.
24	* @cbs_head: Head of callback list.
25	* @cbs_tail: Tail pointer for callback list.
26	* @cbs_wq: Wait queue allowing new callback to get kthread's attention.	49	* @cbs_wq: Wait queue allowing new callback to get kthread's attention.
27	* @cbs_lock: Lock protecting callback list.	50	* @cbs_gbl_lock: Lock protecting callback list.
28	* @kthread_ptr: This flavor's grace-period/callback-invocation kthread.	51	* @kthread_ptr: This flavor's grace-period/callback-invocation kthread.
29	* @gp_func: This flavor's grace-period-wait function.	52	* @gp_func: This flavor's grace-period-wait function.
30	* @gp_state: Grace period's most recent state transition (debugging).	53	* @gp_state: Grace period's most recent state transition (debugging).
@@ -32,7 +55,7 @@ typedef void (postgp_func_t)(struct rcu_tasks rtp); · 1 unmodified line
32	* @init_fract: Initial backoff sleep interval.	55	* @init_fract: Initial backoff sleep interval.
33	* @gp_jiffies: Time of last @gp_state transition.	56	* @gp_jiffies: Time of last @gp_state transition.
34	* @gp_start: Most recent grace-period start in jiffies.	57	* @gp_start: Most recent grace-period start in jiffies.
35	* @n_gps: Number of grace periods completed since boot.	58	* @tasks_gp_seq: Number of grace periods completed since boot.
36	* @n_ipis: Number of IPIs sent to encourage grace periods to end.	59	* @n_ipis: Number of IPIs sent to encourage grace periods to end.
37	* @n_ipis_fails: Number of IPI-send failures.	60	* @n_ipis_fails: Number of IPI-send failures.
38	* @pregp_func: This flavor's pre-grace-period function (optional).	61	* @pregp_func: This flavor's pre-grace-period function (optional).
@@ -41,20 +64,27 @@ typedef void (postgp_func_t)(struct rcu_tasks rtp); · 2 unmodified lines
41	* @holdouts_func: This flavor's holdout-list scan function (optional).	64	* @holdouts_func: This flavor's holdout-list scan function (optional).
42	* @postgp_func: This flavor's post-grace-period function (optional).	65	* @postgp_func: This flavor's post-grace-period function (optional).
43	* @call_func: This flavor's call_rcu()-equivalent function.	66	* @call_func: This flavor's call_rcu()-equivalent function.
		67	* @rtpcpu: This flavor's rcu_tasks_percpu structure.
		68	* @percpu_enqueue_shift: Shift down CPU ID this much when enqueuing callbacks.
		69	* @percpu_enqueue_lim: Number of per-CPU callback queues in use for enqueuing.
		70	* @percpu_dequeue_lim: Number of per-CPU callback queues in use for dequeuing.
		71	* @percpu_dequeue_gpseq: RCU grace-period number to propagate enqueue limit to dequeuers.
		72	* @barrier_q_mutex: Serialize barrier operations.
		73	* @barrier_q_count: Number of queues being waited on.
		74	* @barrier_q_completion: Barrier wait/wakeup mechanism.
		75	* @barrier_q_seq: Sequence number for barrier operations.
44	* @name: This flavor's textual name.	76	* @name: This flavor's textual name.
45	* @kname: This flavor's kthread name.	77	* @kname: This flavor's kthread name.
46	*/	78	*/
47	struct rcu_tasks {	79	struct rcu_tasks {
48	struct rcu_head *cbs_head;
49	struct rcu_head **cbs_tail;
50	struct wait_queue_head cbs_wq;	80	struct wait_queue_head cbs_wq;
51	raw_spinlock_t cbs_lock;	81	raw_spinlock_t cbs_gbl_lock;
52	int gp_state;	82	int gp_state;
53	int gp_sleep;	83	int gp_sleep;
54	int init_fract;	84	int init_fract;
55	unsigned long gp_jiffies;	85	unsigned long gp_jiffies;
56	unsigned long gp_start;	86	unsigned long gp_start;
57	unsigned long n_gps;	87	unsigned long tasks_gp_seq;
58	unsigned long n_ipis;	88	unsigned long n_ipis;
59	unsigned long n_ipis_fails;	89	unsigned long n_ipis_fails;
60	struct task_struct *kthread_ptr;	90	struct task_struct *kthread_ptr;
@@ -65,20 +95,40 @@ struct rcu_tasks { · 4 unmodified lines
65	holdouts_func_t holdouts_func;	95	holdouts_func_t holdouts_func;
66	postgp_func_t postgp_func;	96	postgp_func_t postgp_func;
67	call_rcu_func_t call_func;	97	call_rcu_func_t call_func;
		98	struct rcu_tasks_percpu __percpu *rtpcpu;
		99	int percpu_enqueue_shift;
		100	int percpu_enqueue_lim;
		101	int percpu_dequeue_lim;
		102	unsigned long percpu_dequeue_gpseq;
		103	struct mutex barrier_q_mutex;
		104	atomic_t barrier_q_count;
		105	struct completion barrier_q_completion;
		106	unsigned long barrier_q_seq;
68	char *name;	107	char *name;
69	char *kname;	108	char *kname;
70	};	109	};
71		110
72	#define DEFINE_RCU_TASKS(rt_name, gp, call, n) \	111	static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp);
73	static struct rcu_tasks rt_name = \	112
74	{ \	113	#define DEFINE_RCU_TASKS(rt_name, gp, call, n) \
75	.cbs_tail = &rt_name.cbs_head, \	114	static DEFINE_PER_CPU(struct rcu_tasks_percpu, rt_name ## __percpu) = { \
76	.cbs_wq = __WAIT_QUEUE_HEAD_INITIALIZER(rt_name.cbs_wq), \	115	.lock = __RAW_SPIN_LOCK_UNLOCKED(rt_name ## __percpu.cbs_pcpu_lock), \
77	.cbs_lock = __RAW_SPIN_LOCK_UNLOCKED(rt_name.cbs_lock), \	116	.rtp_irq_work = IRQ_WORK_INIT(call_rcu_tasks_iw_wakeup), \
78	.gp_func = gp, \	117	}; \
79	.call_func = call, \	118	static struct rcu_tasks rt_name = \
80	.name = n, \	119	{ \
81	.kname = #rt_name, \	120	.cbs_wq = __WAIT_QUEUE_HEAD_INITIALIZER(rt_name.cbs_wq), \
		121	.cbs_gbl_lock = __RAW_SPIN_LOCK_UNLOCKED(rt_name.cbs_gbl_lock), \
		122	.gp_func = gp, \
		123	.call_func = call, \
		124	.rtpcpu = &rt_name ## __percpu, \
		125	.name = n, \
		126	.percpu_enqueue_shift = ilog2(CONFIG_NR_CPUS), \
		127	.percpu_enqueue_lim = 1, \
		128	.percpu_dequeue_lim = 1, \
		129	.barrier_q_mutex = __MUTEX_INITIALIZER(rt_name.barrier_q_mutex), \
		130	.barrier_q_seq = (0UL - 50UL) << RCU_SEQ_CTR_SHIFT, \
		131	.kname = #rt_name, \
82	}	132	}
83		133
84	/* Track exiting tasks in order to allow them to be waited for. */	134	/* Track exiting tasks in order to allow them to be waited for. */
@@ -94,6 +144,15 @@ module_param(rcu_task_ipi_delay, int, 0644); · 9 unmodified lines
94	static int rcu_task_stall_timeout __read_mostly = RCU_TASK_STALL_TIMEOUT;	144	static int rcu_task_stall_timeout __read_mostly = RCU_TASK_STALL_TIMEOUT;
95	module_param(rcu_task_stall_timeout, int, 0644);	145	module_param(rcu_task_stall_timeout, int, 0644);
96		146
		147	static int rcu_task_enqueue_lim __read_mostly = -1;
		148	module_param(rcu_task_enqueue_lim, int, 0444);
		149
		150	static bool rcu_task_cb_adjust;
		151	static int rcu_task_contend_lim __read_mostly = 100;
		152	module_param(rcu_task_contend_lim, int, 0444);
		153	static int rcu_task_collapse_lim __read_mostly = 10;
		154	module_param(rcu_task_collapse_lim, int, 0444);
		155
97	/* RCU tasks grace-period state for debugging. */	156	/* RCU tasks grace-period state for debugging. */
98	#define RTGS_INIT 0	157	#define RTGS_INIT 0
99	#define RTGS_WAIT_WAIT_CBS 1	158	#define RTGS_WAIT_WAIT_CBS 1
@@ -128,6 +187,8 @@ static const char * const rcu_tasks_gp_state_names[] = { · 28 unmodified lines
128	//	187	//
129	// Generic code.	188	// Generic code.
130		189
		190	static void rcu_tasks_invoke_cbs_wq(struct work_struct *wp);
		191
131	/* Record grace-period phase and time. */	192	/* Record grace-period phase and time. */
132	static void set_tasks_gp_state(struct rcu_tasks *rtp, int newstate)	193	static void set_tasks_gp_state(struct rcu_tasks *rtp, int newstate)
133	{	194	{
@@ -148,23 +209,106 @@ static const char tasks_gp_state_getname(struct rcu_tasks rtp) · 14 unmodified lines
148	}	209	}
149	#endif /* #ifndef CONFIG_TINY_RCU */	210	#endif /* #ifndef CONFIG_TINY_RCU */
150		211
		212	// Initialize per-CPU callback lists for the specified flavor of
		213	// Tasks RCU.
		214	static void cblist_init_generic(struct rcu_tasks *rtp)
		215	{
		216	int cpu;
		217	unsigned long flags;
		218	int lim;
		219
		220	raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
		221	if (rcu_task_enqueue_lim < 0) {
		222	rcu_task_enqueue_lim = 1;
		223	rcu_task_cb_adjust = true;
		224	pr_info("%s: Setting adjustable number of callback queues.\n", __func__);
		225	} else if (rcu_task_enqueue_lim == 0) {
		226	rcu_task_enqueue_lim = 1;
		227	}
		228	lim = rcu_task_enqueue_lim;
		229
		230	if (lim > nr_cpu_ids)
		231	lim = nr_cpu_ids;
		232	WRITE_ONCE(rtp->percpu_enqueue_shift, ilog2(nr_cpu_ids / lim));
		233	WRITE_ONCE(rtp->percpu_dequeue_lim, lim);
		234	smp_store_release(&rtp->percpu_enqueue_lim, lim);
		235	for_each_possible_cpu(cpu) {
		236	struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rtp->rtpcpu, cpu);
		237
		238	WARN_ON_ONCE(!rtpcp);
		239	if (cpu)
		240	raw_spin_lock_init(&ACCESS_PRIVATE(rtpcp, lock));
		241	raw_spin_lock_rcu_node(rtpcp); // irqs already disabled.
		242	if (rcu_segcblist_empty(&rtpcp->cblist))
		243	rcu_segcblist_init(&rtpcp->cblist);
		244	INIT_WORK(&rtpcp->rtp_work, rcu_tasks_invoke_cbs_wq);
		245	rtpcp->cpu = cpu;
		246	rtpcp->rtpp = rtp;
		247	raw_spin_unlock_rcu_node(rtpcp); // irqs remain disabled.
		248	}
		249	raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
		250	pr_info("%s: Setting shift to %d and lim to %d.\n", __func__, data_race(rtp->percpu_enqueue_shift), data_race(rtp->percpu_enqueue_lim));
		251	}
		252
		253	// IRQ-work handler that does deferred wakeup for call_rcu_tasks_generic().
		254	static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp)
		255	{
		256	struct rcu_tasks *rtp;
		257	struct rcu_tasks_percpu *rtpcp = container_of(iwp, struct rcu_tasks_percpu, rtp_irq_work);
		258
		259	rtp = rtpcp->rtpp;
		260	wake_up(&rtp->cbs_wq);
		261	}
		262
151	// Enqueue a callback for the specified flavor of Tasks RCU.	263	// Enqueue a callback for the specified flavor of Tasks RCU.
152	static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,	264	static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
153	struct rcu_tasks *rtp)	265	struct rcu_tasks *rtp)
154	{	266	{
155	unsigned long flags;	267	unsigned long flags;
		268	unsigned long j;
		269	bool needadjust = false;
156	bool needwake;	270	bool needwake;
		271	struct rcu_tasks_percpu *rtpcp;
157		272
158	rhp->next = NULL;	273	rhp->next = NULL;
159	rhp->func = func;	274	rhp->func = func;
160	raw_spin_lock_irqsave(&rtp->cbs_lock, flags);	275	local_irq_save(flags);
161	needwake = !rtp->cbs_head;	276	rcu_read_lock();
162	WRITE_ONCE(*rtp->cbs_tail, rhp);	277	rtpcp = per_cpu_ptr(rtp->rtpcpu,
163	rtp->cbs_tail = &rhp->next;	278	smp_processor_id() >> READ_ONCE(rtp->percpu_enqueue_shift));
164	raw_spin_unlock_irqrestore(&rtp->cbs_lock, flags);	279	if (!raw_spin_trylock_rcu_node(rtpcp)) { // irqs already disabled.
		280	raw_spin_lock_rcu_node(rtpcp); // irqs already disabled.
		281	j = jiffies;
		282	if (rtpcp->rtp_jiffies != j) {
		283	rtpcp->rtp_jiffies = j;
		284	rtpcp->rtp_n_lock_retries = 0;
		285	}
		286	if (rcu_task_cb_adjust && ++rtpcp->rtp_n_lock_retries > rcu_task_contend_lim &&
		287	READ_ONCE(rtp->percpu_enqueue_lim) != nr_cpu_ids)
		288	needadjust = true; // Defer adjustment to avoid deadlock.
		289	}
		290	if (!rcu_segcblist_is_enabled(&rtpcp->cblist)) {
		291	raw_spin_unlock_rcu_node(rtpcp); // irqs remain disabled.
		292	cblist_init_generic(rtp);
		293	raw_spin_lock_rcu_node(rtpcp); // irqs already disabled.
		294	}
		295	needwake = rcu_segcblist_empty(&rtpcp->cblist);
		296	rcu_segcblist_enqueue(&rtpcp->cblist, rhp);
		297	raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
		298	if (unlikely(needadjust)) {
		299	raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
		300	if (rtp->percpu_enqueue_lim != nr_cpu_ids) {
		301	WRITE_ONCE(rtp->percpu_enqueue_shift, ilog2(nr_cpu_ids));
		302	WRITE_ONCE(rtp->percpu_dequeue_lim, nr_cpu_ids);
		303	smp_store_release(&rtp->percpu_enqueue_lim, nr_cpu_ids);
		304	pr_info("Switching %s to per-CPU callback queuing.\n", rtp->name);
		305	}
		306	raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
		307	}
		308	rcu_read_unlock();
165	/* We can't create the thread unless interrupts are enabled. */	309	/* We can't create the thread unless interrupts are enabled. */
166	if (needwake && READ_ONCE(rtp->kthread_ptr))	310	if (needwake && READ_ONCE(rtp->kthread_ptr))
167	wake_up(&rtp->cbs_wq);	311	irq_work_queue(&rtpcp->rtp_irq_work);
168	}	312	}
169		313
170	// Wait for a grace period for the specified flavor of Tasks RCU.	314	// Wait for a grace period for the specified flavor of Tasks RCU.
@@ -178,12 +322,173 @@ static void synchronize_rcu_tasks_generic(struct rcu_tasks *rtp) · 7 unmodified lines
178	wait_rcu_gp(rtp->call_func);	322	wait_rcu_gp(rtp->call_func);
179	}	323	}
180		324
		325	// RCU callback function for rcu_barrier_tasks_generic().
		326	static void rcu_barrier_tasks_generic_cb(struct rcu_head *rhp)
		327	{
		328	struct rcu_tasks *rtp;
		329	struct rcu_tasks_percpu *rtpcp;
		330
		331	rtpcp = container_of(rhp, struct rcu_tasks_percpu, barrier_q_head);
		332	rtp = rtpcp->rtpp;
		333	if (atomic_dec_and_test(&rtp->barrier_q_count))
		334	complete(&rtp->barrier_q_completion);
		335	}
		336
		337	// Wait for all in-flight callbacks for the specified RCU Tasks flavor.
		338	// Operates in a manner similar to rcu_barrier().
		339	static void rcu_barrier_tasks_generic(struct rcu_tasks *rtp)
		340	{
		341	int cpu;
		342	unsigned long flags;
		343	struct rcu_tasks_percpu *rtpcp;
		344	unsigned long s = rcu_seq_snap(&rtp->barrier_q_seq);
		345
		346	mutex_lock(&rtp->barrier_q_mutex);
		347	if (rcu_seq_done(&rtp->barrier_q_seq, s)) {
		348	smp_mb();
		349	mutex_unlock(&rtp->barrier_q_mutex);
		350	return;
		351	}
		352	rcu_seq_start(&rtp->barrier_q_seq);
		353	init_completion(&rtp->barrier_q_completion);
		354	atomic_set(&rtp->barrier_q_count, 2);
		355	for_each_possible_cpu(cpu) {
		356	if (cpu >= smp_load_acquire(&rtp->percpu_dequeue_lim))
		357	break;
		358	rtpcp = per_cpu_ptr(rtp->rtpcpu, cpu);
		359	rtpcp->barrier_q_head.func = rcu_barrier_tasks_generic_cb;
		360	raw_spin_lock_irqsave_rcu_node(rtpcp, flags);
		361	if (rcu_segcblist_entrain(&rtpcp->cblist, &rtpcp->barrier_q_head))
		362	atomic_inc(&rtp->barrier_q_count);
		363	raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
		364	}
		365	if (atomic_sub_and_test(2, &rtp->barrier_q_count))
		366	complete(&rtp->barrier_q_completion);
		367	wait_for_completion(&rtp->barrier_q_completion);
		368	rcu_seq_end(&rtp->barrier_q_seq);
		369	mutex_unlock(&rtp->barrier_q_mutex);
		370	}
		371
		372	// Advance callbacks and indicate whether either a grace period or
		373	// callback invocation is needed.
		374	static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)
		375	{
		376	int cpu;
		377	unsigned long flags;
		378	long n;
		379	long ncbs = 0;
		380	long ncbsnz = 0;
		381	int needgpcb = 0;
		382
		383	for (cpu = 0; cpu < smp_load_acquire(&rtp->percpu_dequeue_lim); cpu++) {
		384	struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rtp->rtpcpu, cpu);
		385
		386	/* Advance and accelerate any new callbacks. */
		387	if (!rcu_segcblist_n_cbs(&rtpcp->cblist))
		388	continue;
		389	raw_spin_lock_irqsave_rcu_node(rtpcp, flags);
		390	// Should we shrink down to a single callback queue?
		391	n = rcu_segcblist_n_cbs(&rtpcp->cblist);
		392	if (n) {
		393	ncbs += n;
		394	if (cpu > 0)
		395	ncbsnz += n;
		396	}
		397	rcu_segcblist_advance(&rtpcp->cblist, rcu_seq_current(&rtp->tasks_gp_seq));
		398	(void)rcu_segcblist_accelerate(&rtpcp->cblist, rcu_seq_snap(&rtp->tasks_gp_seq));
		399	if (rcu_segcblist_pend_cbs(&rtpcp->cblist))
		400	needgpcb \|= 0x3;
		401	if (!rcu_segcblist_empty(&rtpcp->cblist))
		402	needgpcb \|= 0x1;
		403	raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
		404	}
		405
		406	// Shrink down to a single callback queue if appropriate.
		407	// This is done in two stages: (1) If there are no more than
		408	// rcu_task_collapse_lim callbacks on CPU 0 and none on any other
		409	// CPU, limit enqueueing to CPU 0. (2) After an RCU grace period,
		410	// if there has not been an increase in callbacks, limit dequeuing
		411	// to CPU 0. Note the matching RCU read-side critical section in
		412	// call_rcu_tasks_generic().
		413	if (rcu_task_cb_adjust && ncbs <= rcu_task_collapse_lim) {
		414	raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
		415	if (rtp->percpu_enqueue_lim > 1) {
		416	WRITE_ONCE(rtp->percpu_enqueue_shift, ilog2(nr_cpu_ids));
		417	smp_store_release(&rtp->percpu_enqueue_lim, 1);
		418	rtp->percpu_dequeue_gpseq = get_state_synchronize_rcu();
		419	pr_info("Starting switch %s to CPU-0 callback queuing.\n", rtp->name);
		420	}
		421	raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
		422	}
		423	if (rcu_task_cb_adjust && !ncbsnz &&
		424	poll_state_synchronize_rcu(rtp->percpu_dequeue_gpseq)) {
		425	raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
		426	if (rtp->percpu_enqueue_lim < rtp->percpu_dequeue_lim) {
		427	WRITE_ONCE(rtp->percpu_dequeue_lim, 1);
		428	pr_info("Completing switch %s to CPU-0 callback queuing.\n", rtp->name);
		429	}
		430	raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
		431	}
		432
		433	return needgpcb;
		434	}
		435
		436	// Advance callbacks and invoke any that are ready.
		437	static void rcu_tasks_invoke_cbs(struct rcu_tasks rtp, struct rcu_tasks_percpu rtpcp)
		438	{
		439	int cpu;
		440	int cpunext;
		441	unsigned long flags;
		442	int len;
		443	struct rcu_head *rhp;
		444	struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
		445	struct rcu_tasks_percpu *rtpcp_next;
		446
		447	cpu = rtpcp->cpu;
		448	cpunext = cpu * 2 + 1;
		449	if (cpunext < smp_load_acquire(&rtp->percpu_dequeue_lim)) {
		450	rtpcp_next = per_cpu_ptr(rtp->rtpcpu, cpunext);
		451	queue_work_on(cpunext, system_wq, &rtpcp_next->rtp_work);
		452	cpunext++;
		453	if (cpunext < smp_load_acquire(&rtp->percpu_dequeue_lim)) {
		454	rtpcp_next = per_cpu_ptr(rtp->rtpcpu, cpunext);
		455	queue_work_on(cpunext, system_wq, &rtpcp_next->rtp_work);
		456	}
		457	}
		458
		459	if (rcu_segcblist_empty(&rtpcp->cblist))
		460	return;
		461	raw_spin_lock_irqsave_rcu_node(rtpcp, flags);
		462	rcu_segcblist_advance(&rtpcp->cblist, rcu_seq_current(&rtp->tasks_gp_seq));
		463	rcu_segcblist_extract_done_cbs(&rtpcp->cblist, &rcl);
		464	raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
		465	len = rcl.len;
		466	for (rhp = rcu_cblist_dequeue(&rcl); rhp; rhp = rcu_cblist_dequeue(&rcl)) {
		467	local_bh_disable();
		468	rhp->func(rhp);
		469	local_bh_enable();
		470	cond_resched();
		471	}
		472	raw_spin_lock_irqsave_rcu_node(rtpcp, flags);
		473	rcu_segcblist_add_len(&rtpcp->cblist, -len);
		474	(void)rcu_segcblist_accelerate(&rtpcp->cblist, rcu_seq_snap(&rtp->tasks_gp_seq));
		475	raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
		476	}
		477
		478	// Workqueue flood to advance callbacks and invoke any that are ready.
		479	static void rcu_tasks_invoke_cbs_wq(struct work_struct *wp)
		480	{
		481	struct rcu_tasks *rtp;
		482	struct rcu_tasks_percpu *rtpcp = container_of(wp, struct rcu_tasks_percpu, rtp_work);
		483
		484	rtp = rtpcp->rtpp;
		485	rcu_tasks_invoke_cbs(rtp, rtpcp);
		486	}
		487
181	/* RCU-tasks kthread that detects grace periods and invokes callbacks. */	488	/* RCU-tasks kthread that detects grace periods and invokes callbacks. */
182	static int __noreturn rcu_tasks_kthread(void *arg)	489	static int __noreturn rcu_tasks_kthread(void *arg)
183	{	490	{
184	unsigned long flags;	491	int needgpcb;
185	struct rcu_head *list;
186	struct rcu_head *next;
187	struct rcu_tasks *rtp = arg;	492	struct rcu_tasks *rtp = arg;
188		493
189	/* Run on housekeeping CPUs by default. Sysadm can move if desired. */	494	/* Run on housekeeping CPUs by default. Sysadm can move if desired. */
@@ -199,42 +504,22 @@ static int __noreturn rcu_tasks_kthread(void *arg) · 9 unmodified lines
199	for (;;) {	504	for (;;) {
200	set_tasks_gp_state(rtp, RTGS_WAIT_CBS);	505	set_tasks_gp_state(rtp, RTGS_WAIT_CBS);
201		506
202	/* Pick up any new callbacks. */
203	raw_spin_lock_irqsave(&rtp->cbs_lock, flags);
204	smp_mb__after_spinlock(); // Order updates vs. GP.
205	list = rtp->cbs_head;
206	rtp->cbs_head = NULL;
207	rtp->cbs_tail = &rtp->cbs_head;
208	raw_spin_unlock_irqrestore(&rtp->cbs_lock, flags);
209
210	/* If there were none, wait a bit and start over. */	507	/* If there were none, wait a bit and start over. */
211	if (!list) {	508	wait_event_idle(rtp->cbs_wq, (needgpcb = rcu_tasks_need_gpcb(rtp)));
212	wait_event_interruptible(rtp->cbs_wq,	509
213	READ_ONCE(rtp->cbs_head));	510	if (needgpcb & 0x2) {
214	if (!rtp->cbs_head) {	511	// Wait for one grace period.
215	WARN_ON(signal_pending(current));	512	set_tasks_gp_state(rtp, RTGS_WAIT_GP);
216	set_tasks_gp_state(rtp, RTGS_WAIT_WAIT_CBS);	513	rtp->gp_start = jiffies;
217	schedule_timeout_idle(HZ/10);	514	rcu_seq_start(&rtp->tasks_gp_seq);
218	}	515	rtp->gp_func(rtp);
219	continue;	516	rcu_seq_end(&rtp->tasks_gp_seq);
220	}	517	}
221		518
222	// Wait for one grace period.	519	/* Invoke callbacks. */
223	set_tasks_gp_state(rtp, RTGS_WAIT_GP);
224	rtp->gp_start = jiffies;
225	rtp->gp_func(rtp);
226	rtp->n_gps++;
227
228	/* Invoke the callbacks. */
229	set_tasks_gp_state(rtp, RTGS_INVOKE_CBS);	520	set_tasks_gp_state(rtp, RTGS_INVOKE_CBS);
230	while (list) {	521	rcu_tasks_invoke_cbs(rtp, per_cpu_ptr(rtp->rtpcpu, 0));
231	next = list->next;	522
232	local_bh_disable();
233	list->func(list);
234	local_bh_enable();
235	list = next;
236	cond_resched();
237	}
238	/* Paranoid sleep to keep this from entering a tight loop */	523	/* Paranoid sleep to keep this from entering a tight loop */
239	schedule_timeout_idle(rtp->gp_sleep);	524	schedule_timeout_idle(rtp->gp_sleep);
240	}	525	}
@@ -279,14 +564,15 @@ static void __init rcu_tasks_bootup_oddness(void) · 38 unmodified lines
279	/* Dump out rcutorture-relevant state common to all RCU-tasks flavors. */	564	/* Dump out rcutorture-relevant state common to all RCU-tasks flavors. */
280	static void show_rcu_tasks_generic_gp_kthread(struct rcu_tasks rtp, char s)	565	static void show_rcu_tasks_generic_gp_kthread(struct rcu_tasks rtp, char s)
281	{	566	{
		567	struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rtp->rtpcpu, 0); // for_each...
282	pr_info("%s: %s(%d) since %lu g:%lu i:%lu/%lu %c%c %s\n",	568	pr_info("%s: %s(%d) since %lu g:%lu i:%lu/%lu %c%c %s\n",
283	rtp->kname,	569	rtp->kname,
284	tasks_gp_state_getname(rtp), data_race(rtp->gp_state),	570	tasks_gp_state_getname(rtp), data_race(rtp->gp_state),
285	jiffies - data_race(rtp->gp_jiffies),	571	jiffies - data_race(rtp->gp_jiffies),
286	data_race(rtp->n_gps),	572	data_race(rcu_seq_current(&rtp->tasks_gp_seq)),
287	data_race(rtp->n_ipis_fails), data_race(rtp->n_ipis),	573	data_race(rtp->n_ipis_fails), data_race(rtp->n_ipis),
288	".k"[!!data_race(rtp->kthread_ptr)],	574	".k"[!!data_race(rtp->kthread_ptr)],
289	".C"[!!data_race(rtp->cbs_head)],	575	".C"[!data_race(rcu_segcblist_empty(&rtpcp->cblist))],
290	s);	576	s);
291	}	577	}
292	#endif // #ifndef CONFIG_TINY_RCU	578	#endif // #ifndef CONFIG_TINY_RCU
@@ -411,10 +697,10 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) · 118 unmodified lines
411	// exit_tasks_rcu_finish() functions begin and end, respectively, the SRCU	697	// exit_tasks_rcu_finish() functions begin and end, respectively, the SRCU
412	// read-side critical sections waited for by rcu_tasks_postscan().	698	// read-side critical sections waited for by rcu_tasks_postscan().
413	//	699	//
414	// Pre-grace-period update-side code is ordered before the grace via the	700	// Pre-grace-period update-side code is ordered before the grace
415	// ->cbs_lock and the smp_mb__after_spinlock(). Pre-grace-period read-side	701	// via the raw_spin_lock.*rcu_node(). Pre-grace-period read-side code
416	// code is ordered before the grace period via synchronize_rcu() call	702	// is ordered before the grace period via synchronize_rcu() call in
417	// in rcu_tasks_pregp_step() and by the scheduler's locks and interrupt	703	// rcu_tasks_pregp_step() and by the scheduler's locks and interrupt
418	// disabling.	704	// disabling.
419		705
420	/* Pre-grace-period preparation. */	706	/* Pre-grace-period preparation. */
@@ -586,13 +872,13 @@ EXPORT_SYMBOL_GPL(synchronize_rcu_tasks); · 165 unmodified lines
586	*/	872	*/
587	void rcu_barrier_tasks(void)	873	void rcu_barrier_tasks(void)
588	{	874	{
589	/* There is only one callback queue, so this is easy. ;-) */	875	rcu_barrier_tasks_generic(&rcu_tasks);
590	synchronize_rcu_tasks();
591	}	876	}
592	EXPORT_SYMBOL_GPL(rcu_barrier_tasks);	877	EXPORT_SYMBOL_GPL(rcu_barrier_tasks);
593		878
594	static int __init rcu_spawn_tasks_kthread(void)	879	static int __init rcu_spawn_tasks_kthread(void)
595	{	880	{
		881	cblist_init_generic(&rcu_tasks);
596	rcu_tasks.gp_sleep = HZ / 10;	882	rcu_tasks.gp_sleep = HZ / 10;
597	rcu_tasks.init_fract = HZ / 10;	883	rcu_tasks.init_fract = HZ / 10;
598	rcu_tasks.pregp_func = rcu_tasks_pregp_step;	884	rcu_tasks.pregp_func = rcu_tasks_pregp_step;
@@ -724,13 +1010,13 @@ EXPORT_SYMBOL_GPL(synchronize_rcu_tasks_rude); · 125 unmodified lines
724	*/	1010	*/
725	void rcu_barrier_tasks_rude(void)	1011	void rcu_barrier_tasks_rude(void)
726	{	1012	{
727	/* There is only one callback queue, so this is easy. ;-) */	1013	rcu_barrier_tasks_generic(&rcu_tasks_rude);
728	synchronize_rcu_tasks_rude();
729	}	1014	}
730	EXPORT_SYMBOL_GPL(rcu_barrier_tasks_rude);	1015	EXPORT_SYMBOL_GPL(rcu_barrier_tasks_rude);
731		1016
732	static int __init rcu_spawn_tasks_rude_kthread(void)	1017	static int __init rcu_spawn_tasks_rude_kthread(void)
733	{	1018	{
		1019	cblist_init_generic(&rcu_tasks_rude);
734	rcu_tasks_rude.gp_sleep = HZ / 10;	1020	rcu_tasks_rude.gp_sleep = HZ / 10;
735	rcu_spawn_tasks_kthread_generic(&rcu_tasks_rude);	1021	rcu_spawn_tasks_kthread_generic(&rcu_tasks_rude);
736	return 0;	1022	return 0;
@@ -1073,25 +1359,50 @@ static void rcu_tasks_trace_postscan(struct list_head *hop) · 336 unmodified lines
1073	// Any tasks that exit after this point will set ->trc_reader_checked.	1359	// Any tasks that exit after this point will set ->trc_reader_checked.
1074	}	1360	}
1075		1361
		1362	/* Communicate task state back to the RCU tasks trace stall warning request. */
		1363	struct trc_stall_chk_rdr {
		1364	int nesting;
		1365	int ipi_to_cpu;
		1366	u8 needqs;
		1367	};
		1368
		1369	static int trc_check_slow_task(struct task_struct t, void arg)
		1370	{
		1371	struct trc_stall_chk_rdr *trc_rdrp = arg;
		1372
		1373	if (task_curr(t))
		1374	return false; // It is running, so decline to inspect it.
		1375	trc_rdrp->nesting = READ_ONCE(t->trc_reader_nesting);
		1376	trc_rdrp->ipi_to_cpu = READ_ONCE(t->trc_ipi_to_cpu);
		1377	trc_rdrp->needqs = READ_ONCE(t->trc_reader_special.b.need_qs);
		1378	return true;
		1379	}
		1380
1076	/* Show the state of a task stalling the current RCU tasks trace GP. */	1381	/* Show the state of a task stalling the current RCU tasks trace GP. */
1077	static void show_stalled_task_trace(struct task_struct t, bool firstreport)	1382	static void show_stalled_task_trace(struct task_struct t, bool firstreport)
1078	{	1383	{
1079	int cpu;	1384	int cpu;
		1385	struct trc_stall_chk_rdr trc_rdr;
		1386	bool is_idle_tsk = is_idle_task(t);
1080		1387
1081	if (*firstreport) {	1388	if (*firstreport) {
1082	pr_err("INFO: rcu_tasks_trace detected stalls on tasks:\n");	1389	pr_err("INFO: rcu_tasks_trace detected stalls on tasks:\n");
1083	*firstreport = false;	1390	*firstreport = false;
1084	}	1391	}
1085	// FIXME: This should attempt to use try_invoke_on_nonrunning_task().
1086	cpu = task_cpu(t);	1392	cpu = task_cpu(t);
1087	pr_alert("P%d: %c%c%c nesting: %d%c cpu: %d\n",	1393	if (!task_call_func(t, trc_check_slow_task, &trc_rdr))
1088	t->pid,	1394	pr_alert("P%d: %c\n",
1089	".I"[READ_ONCE(t->trc_ipi_to_cpu) >= 0],	1395	t->pid,
1090	".i"[is_idle_task(t)],	1396	".i"[is_idle_tsk]);
1091	".N"[cpu >= 0 && tick_nohz_full_cpu(cpu)],	1397	else
1092	READ_ONCE(t->trc_reader_nesting),	1398	pr_alert("P%d: %c%c%c nesting: %d%c cpu: %d\n",
1093	" N"[!!READ_ONCE(t->trc_reader_special.b.need_qs)],	1399	t->pid,
1094	cpu);	1400	".I"[trc_rdr.ipi_to_cpu >= 0],
		1401	".i"[is_idle_tsk],
		1402	".N"[cpu >= 0 && tick_nohz_full_cpu(cpu)],
		1403	trc_rdr.nesting,
		1404	" N"[!!trc_rdr.needqs],
		1405	cpu);
1095	sched_show_task(t);	1406	sched_show_task(t);
1096	}	1407	}
1097		1408
@@ -1121,7 +1432,8 @@ static void check_all_holdout_tasks_trace(struct list_head *hop, · 23 unmodified lines
1121	trc_wait_for_one_reader(t, hop);	1432	trc_wait_for_one_reader(t, hop);
1122		1433
1123	// If check succeeded, remove this task from the list.	1434	// If check succeeded, remove this task from the list.
1124	if (READ_ONCE(t->trc_reader_checked))	1435	if (smp_load_acquire(&t->trc_ipi_to_cpu) == -1 &&
		1436	READ_ONCE(t->trc_reader_checked))
1125	trc_del_holdout(t);	1437	trc_del_holdout(t);
1126	else if (needreport)	1438	else if (needreport)
1127	show_stalled_task_trace(t, firstreport);	1439	show_stalled_task_trace(t, firstreport);
@@ -1156,7 +1468,7 @@ static void rcu_tasks_trace_postgp(struct rcu_tasks *rtp) · 28 unmodified lines
1156	// Yes, this assumes that CPUs process IPIs in order. If that ever	1468	// Yes, this assumes that CPUs process IPIs in order. If that ever
1157	// changes, there will need to be a recheck and/or timed wait.	1469	// changes, there will need to be a recheck and/or timed wait.
1158	for_each_online_cpu(cpu)	1470	for_each_online_cpu(cpu)
1159	if (smp_load_acquire(per_cpu_ptr(&trc_ipi_to_cpu, cpu)))	1471	if (WARN_ON_ONCE(smp_load_acquire(per_cpu_ptr(&trc_ipi_to_cpu, cpu))))
1160	smp_call_function_single(cpu, rcu_tasks_trace_empty_fn, NULL, 1);	1472	smp_call_function_single(cpu, rcu_tasks_trace_empty_fn, NULL, 1);
1161		1473
1162	// Remove the safety count.	1474	// Remove the safety count.
@@ -1256,13 +1568,13 @@ EXPORT_SYMBOL_GPL(synchronize_rcu_tasks_trace); · 93 unmodified lines
1256	*/	1568	*/
1257	void rcu_barrier_tasks_trace(void)	1569	void rcu_barrier_tasks_trace(void)
1258	{	1570	{
1259	/* There is only one callback queue, so this is easy. ;-) */	1571	rcu_barrier_tasks_generic(&rcu_tasks_trace);
1260	synchronize_rcu_tasks_trace();
1261	}	1572	}
1262	EXPORT_SYMBOL_GPL(rcu_barrier_tasks_trace);	1573	EXPORT_SYMBOL_GPL(rcu_barrier_tasks_trace);
1263		1574
1264	static int __init rcu_spawn_tasks_trace_kthread(void)	1575	static int __init rcu_spawn_tasks_trace_kthread(void)
1265	{	1576	{
		1577	cblist_init_generic(&rcu_tasks_trace);
1266	if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) {	1578	if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) {
1267	rcu_tasks_trace.gp_sleep = HZ / 10;	1579	rcu_tasks_trace.gp_sleep = HZ / 10;
1268	rcu_tasks_trace.init_fract = HZ / 10;	1580	rcu_tasks_trace.init_fract = HZ / 10;

kernel/rcu/tree.c+86−45

 	.dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE,
 	.dynticks = ATOMIC_INIT(1),
 #ifdef CONFIG_RCU_NOCB_CPU
-	.cblist.flags = SEGCBLIST_SOFTIRQ_ONLY,
+	.cblist.flags = SEGCBLIST_RCU_CORE,
 #endif
 };
 static struct rcu_state rcu_state = {
 	instrumentation_begin();
 	trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rdp->dynticks));
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
-	rcu_prepare_for_idle();
 	rcu_preempt_deferred_qs(current);
 	// instrumentation for the noinstr rcu_dynticks_eqs_enter()
 	trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, atomic_read(&rdp->dynticks));
 	WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */
-	if (!in_nmi())
-		rcu_prepare_for_idle();
 	// instrumentation for the noinstr rcu_dynticks_eqs_enter()
 	instrument_atomic_write(&rdp->dynticks, sizeof(rdp->dynticks));
 	instrumentation_end();
 	// instrumentation for the noinstr rcu_dynticks_eqs_exit()
 	instrument_atomic_write(&rdp->dynticks, sizeof(rdp->dynticks));
-	rcu_cleanup_after_idle();
 	trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, atomic_read(&rdp->dynticks));
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
 	WRITE_ONCE(rdp->dynticks_nesting, 1);
 		rcu_dynticks_eqs_exit();
 		// ... but is watching here.
-		if (!in_nmi()) {
-			instrumentation_begin();
-			rcu_cleanup_after_idle();
-			instrumentation_end();
-		}
 		instrumentation_begin();
 		// instrumentation for the noinstr rcu_dynticks_curr_cpu_in_eqs()
 		instrument_atomic_read(&rdp->dynticks, sizeof(rdp->dynticks));
 	local_irq_restore(flags);
 }
+/*
+ * Check to see if any future non-offloaded RCU-related work will need
+ * to be done by the current CPU, even if none need be done immediately,
+ * returning 1 if so.  This function is part of the RCU implementation;
+ * it is -not- an exported member of the RCU API.  This is used by
+ * the idle-entry code to figure out whether it is safe to disable the
+ * scheduler-clock interrupt.
+ *
+ * Just check whether or not this CPU has non-offloaded RCU callbacks
+ * queued.
+ */
+int rcu_needs_cpu(u64 basemono, u64 *nextevt)
+{
+	*nextevt = KTIME_MAX;
+	return !rcu_segcblist_empty(&this_cpu_ptr(&rcu_data)->cblist) &&
+		!rcu_rdp_is_offloaded(this_cpu_ptr(&rcu_data));
+}
 /*
  * If any sort of urgency was applied to the current CPU (for example,
  * the scheduler-clock interrupt was enabled on a nohz_full CPU) in order
 {
 	struct task_struct *t = READ_ONCE(rcu_state.gp_kthread);
-	if ((current == t && !in_irq() && !in_serving_softirq()) ||
+	if ((current == t && !in_hardirq() && !in_serving_softirq()) ||
 	    !READ_ONCE(rcu_state.gp_flags) || !t)
 		return;
 	WRITE_ONCE(rcu_state.gp_wake_time, jiffies);
 						  struct rcu_data *rdp)
 {
 	rcu_lockdep_assert_cblist_protected(rdp);
-	if (!rcu_seq_state(rcu_seq_current(&rnp->gp_seq)) ||
+	if (!rcu_seq_state(rcu_seq_current(&rnp->gp_seq)) || !raw_spin_trylock_rcu_node(rnp))
-	    !raw_spin_trylock_rcu_node(rnp))
 		return;
-	WARN_ON_ONCE(rcu_advance_cbs(rnp, rdp));
+	// The grace period cannot end while we hold the rcu_node lock.
+	if (rcu_seq_state(rcu_seq_current(&rnp->gp_seq)))
+		WARN_ON_ONCE(rcu_advance_cbs(rnp, rdp));
 	raw_spin_unlock_rcu_node(rnp);
 }
 	unsigned long flags;
 	unsigned long mask;
 	bool needwake = false;
-	const bool offloaded = rcu_rdp_is_offloaded(rdp);
+	bool needacc = false;
 	struct rcu_node *rnp;
 	WARN_ON_ONCE(rdp->cpu != smp_processor_id());
 		/*
 		 * This GP can't end until cpu checks in, so all of our
 		 * callbacks can be processed during the next GP.
+		 *
+		 * NOCB kthreads have their own way to deal with that...
 		 */
-		if (!offloaded)
+		if (!rcu_rdp_is_offloaded(rdp)) {
 			needwake = rcu_accelerate_cbs(rnp, rdp);
+		} else if (!rcu_segcblist_completely_offloaded(&rdp->cblist)) {
+			/*
+			 * ...but NOCB kthreads may miss or delay callbacks acceleration
+			 * if in the middle of a (de-)offloading process.
+			 */
+			needacc = true;
+		}
 		rcu_disable_urgency_upon_qs(rdp);
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);
 		/* ^^^ Released rnp->lock */
 		if (needwake)
 			rcu_gp_kthread_wake();
+		if (needacc) {
+			rcu_nocb_lock_irqsave(rdp, flags);
+			rcu_accelerate_cbs_unlocked(rnp, rdp);
+			rcu_nocb_unlock_irqrestore(rdp, flags);
+		}
 	}
 }
 	int div;
 	bool __maybe_unused empty;
 	unsigned long flags;
-	const bool offloaded = rcu_rdp_is_offloaded(rdp);
 	struct rcu_head *rhp;
 	struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
 	long bl, count = 0;
 	}
 	/*
-	 * Extract the list of ready callbacks, disabling to prevent
+	 * Extract the list of ready callbacks, disabling IRQs to prevent
 	 * races with call_rcu() from interrupt handlers.  Leave the
 	 * callback counts, as rcu_barrier() needs to be conservative.
 	 */
-	local_irq_save(flags);
+	rcu_nocb_lock_irqsave(rdp, flags);
-	rcu_nocb_lock(rdp);
 	WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));
 	pending = rcu_segcblist_n_cbs(&rdp->cblist);
 	div = READ_ONCE(rcu_divisor);
 	div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div;
 	bl = max(rdp->blimit, pending >> div);
-	if (unlikely(bl > 100)) {
+	if (in_serving_softirq() && unlikely(bl > 100)) {
 		long rrn = READ_ONCE(rcu_resched_ns);
 		rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn;
 	trace_rcu_batch_start(rcu_state.name,
 			      rcu_segcblist_n_cbs(&rdp->cblist), bl);
 	rcu_segcblist_extract_done_cbs(&rdp->cblist, &rcl);
-	if (offloaded)
+	if (rcu_rdp_is_offloaded(rdp))
 		rdp->qlen_last_fqs_check = rcu_segcblist_n_cbs(&rdp->cblist);
 	trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCbDequeued"));
 		/*
 		 * Stop only if limit reached and CPU has something to do.
 		 */
-		if (count >= bl && !offloaded &&
+		if (in_serving_softirq()) {
-		    (need_resched() ||
+			if (count >= bl && (need_resched() || !is_idle_task(current)))
-		     (!is_idle_task(current) && !rcu_is_callbacks_kthread())))
+				break;
-			break;
+			/*
-		if (unlikely(tlimit)) {
+			 * Make sure we don't spend too much time here and deprive other
-			/* only call local_clock() every 32 callbacks */
+			 * softirq vectors of CPU cycles.
-			if (likely((count & 31) || local_clock() < tlimit))
+			 */
-				continue;
+			if (unlikely(tlimit)) {
-			/* Exceeded the time limit, so leave. */
+				/* only call local_clock() every 32 callbacks */
-			break;
+				if (likely((count & 31) || local_clock() < tlimit))
-		}
+					continue;
-		if (!in_serving_softirq()) {
+				/* Exceeded the time limit, so leave. */
+				break;
+			}
+		} else {
 			local_bh_enable();
 			lockdep_assert_irqs_enabled();
 			cond_resched_tasks_rcu_qs();
 		}
 	}
-	local_irq_save(flags);
+	rcu_nocb_lock_irqsave(rdp, flags);
-	rcu_nocb_lock(rdp);
 	rdp->n_cbs_invoked += count;
 	trace_rcu_batch_end(rcu_state.name, count, !!rcl.head, need_resched(),
 			    is_idle_task(current), rcu_is_callbacks_kthread());
 	rcu_nocb_unlock_irqrestore(rdp, flags);
-	/* Re-invoke RCU core processing if there are callbacks remaining. */
-	if (!offloaded && rcu_segcblist_ready_cbs(&rdp->cblist))
-		invoke_rcu_core();
 	tick_dep_clear_task(current, TICK_DEP_BIT_RCU);
 }
 	unsigned long flags;
 	struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
 	struct rcu_node *rnp = rdp->mynode;
+	/*
+	 * On RT rcu_core() can be preempted when IRQs aren't disabled.
+	 * Therefore this function can race with concurrent NOCB (de-)offloading
+	 * on this CPU and the below condition must be considered volatile.
+	 * However if we race with:
+	 *
+	 * _ Offloading:   In the worst case we accelerate or process callbacks
+	 *                 concurrently with NOCB kthreads. We are guaranteed to
+	 *                 call rcu_nocb_lock() if that happens.
+	 *
+	 * _ Deoffloading: In the worst case we miss callbacks acceleration or
+	 *                 processing. This is fine because the early stage
+	 *                 of deoffloading invokes rcu_core() after setting
+	 *                 SEGCBLIST_RCU_CORE. So we guarantee that we'll process
+	 *                 what could have been dismissed without the need to wait
+	 *                 for the next rcu_pending() check in the next jiffy.
+	 */
 	const bool do_batch = !rcu_segcblist_completely_offloaded(&rdp->cblist);
 	if (cpu_is_offline(smp_processor_id()))
 	WARN_ON_ONCE(!rdp->beenonline);
 	/* Report any deferred quiescent states if preemption enabled. */
-	if (!(preempt_count() & PREEMPT_MASK)) {
+	if (IS_ENABLED(CONFIG_PREEMPT_COUNT) && (!(preempt_count() & PREEMPT_MASK))) {
 		rcu_preempt_deferred_qs(current);
 	} else if (rcu_preempt_need_deferred_qs(current)) {
 		set_tsk_need_resched(current);
 	/* If there are callbacks ready, invoke them. */
 	if (do_batch && rcu_segcblist_ready_cbs(&rdp->cblist) &&
-	    likely(READ_ONCE(rcu_scheduler_fully_active)))
+	    likely(READ_ONCE(rcu_scheduler_fully_active))) {
 		rcu_do_batch(rdp);
+		/* Re-invoke RCU core processing if there are callbacks remaining. */
+		if (rcu_segcblist_ready_cbs(&rdp->cblist))
+			invoke_rcu_core();
+	}
 	/* Do any needed deferred wakeups of rcuo kthreads. */
 	do_nocb_deferred_wakeup(rdp);
 	head->func = func;
 	head->next = NULL;
 	local_irq_save(flags);
-	kasan_record_aux_stack(head);
+	kasan_record_aux_stack_noalloc(head);
 	rdp = this_cpu_ptr(&rcu_data);
 	/* Add the callback to our list. */
 		return;
 	}
-	kasan_record_aux_stack(ptr);
+	kasan_record_aux_stack_noalloc(ptr);
 	success = add_ptr_to_bulk_krc_lock(&krcp, &flags, ptr, !head);
 	if (!success) {
 		run_page_cache_worker(krcp);

kernel/rcu/tree.h+15−16

@@ -157,7 +157,6 @@ struct rcu_data { · 156 unmodified lines
157	bool core_needs_qs; /* Core waits for quiescent state. */	157	bool core_needs_qs; /* Core waits for quiescent state. */
158	bool beenonline; /* CPU online at least once. */	158	bool beenonline; /* CPU online at least once. */
159	bool gpwrap; /* Possible ->gp_seq wrap. */	159	bool gpwrap; /* Possible ->gp_seq wrap. */
160	bool exp_deferred_qs; /* This CPU awaiting a deferred QS? */
161	bool cpu_started; /* RCU watching this onlining CPU. */	160	bool cpu_started; /* RCU watching this onlining CPU. */
162	struct rcu_node mynode; / This CPU's leaf of hierarchy */	161	struct rcu_node mynode; / This CPU's leaf of hierarchy */
163	unsigned long grpmask; /* Mask to apply to leaf qsmask. */	162	unsigned long grpmask; /* Mask to apply to leaf qsmask. */
@@ -189,11 +188,6 @@ struct rcu_data { · 25 unmodified lines
189	bool rcu_urgent_qs; /* GP old need light quiescent state. */	188	bool rcu_urgent_qs; /* GP old need light quiescent state. */
190	bool rcu_forced_tick; /* Forced tick to provide QS. */	189	bool rcu_forced_tick; /* Forced tick to provide QS. */
191	bool rcu_forced_tick_exp; /* ... provide QS to expedited GP. */	190	bool rcu_forced_tick_exp; /* ... provide QS to expedited GP. */
192	#ifdef CONFIG_RCU_FAST_NO_HZ
193	unsigned long last_accelerate; /* Last jiffy CBs were accelerated. */
194	unsigned long last_advance_all; /* Last jiffy CBs were all advanced. */
195	int tick_nohz_enabled_snap; /* Previously seen value from sysfs. */
196	#endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */
197		191
198	/* 4) rcu_barrier(), OOM callbacks, and expediting. */	192	/* 4) rcu_barrier(), OOM callbacks, and expediting. */
199	struct rcu_head barrier_head;	193	struct rcu_head barrier_head;
@@ -227,8 +221,11 @@ struct rcu_data { · 27 unmodified lines
227	struct swait_queue_head nocb_gp_wq; /* For nocb kthreads to sleep on. */	221	struct swait_queue_head nocb_gp_wq; /* For nocb kthreads to sleep on. */
228	bool nocb_cb_sleep; /* Is the nocb CB thread asleep? */	222	bool nocb_cb_sleep; /* Is the nocb CB thread asleep? */
229	struct task_struct *nocb_cb_kthread;	223	struct task_struct *nocb_cb_kthread;
230	struct rcu_data *nocb_next_cb_rdp;	224	struct list_head nocb_head_rdp; /*
231	/* Next rcu_data in wakeup chain. */	225	* Head of rcu_data list in wakeup chain,
		226	* if rdp_gp.
		227	*/
		228	struct list_head nocb_entry_rdp; /* rcu_data node in wakeup chain. */
232		229
233	/* The following fields are used by CB kthread, hence new cacheline. */	230	/* The following fields are used by CB kthread, hence new cacheline. */
234	struct rcu_data *nocb_gp_rdp ____cacheline_internodealigned_in_smp;	231	struct rcu_data *nocb_gp_rdp ____cacheline_internodealigned_in_smp;
@@ -419,8 +416,6 @@ static bool rcu_is_callbacks_kthread(void); · 184 unmodified lines
419	static void rcu_cpu_kthread_setup(unsigned int cpu);	416	static void rcu_cpu_kthread_setup(unsigned int cpu);
420	static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp);	417	static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp);
421	static void __init rcu_spawn_boost_kthreads(void);	418	static void __init rcu_spawn_boost_kthreads(void);
422	static void rcu_cleanup_after_idle(void);
423	static void rcu_prepare_for_idle(void);
424	static bool rcu_preempt_has_tasks(struct rcu_node *rnp);	419	static bool rcu_preempt_has_tasks(struct rcu_node *rnp);
425	static bool rcu_preempt_need_deferred_qs(struct task_struct *t);	420	static bool rcu_preempt_need_deferred_qs(struct task_struct *t);
426	static void rcu_preempt_deferred_qs(struct task_struct *t);	421	static void rcu_preempt_deferred_qs(struct task_struct *t);
@@ -447,12 +442,16 @@ static void rcu_nocb_unlock_irqrestore(struct rcu_data *rdp, · 20 unmodified lines
447	static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp);	442	static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp);
448	#ifdef CONFIG_RCU_NOCB_CPU	443	#ifdef CONFIG_RCU_NOCB_CPU
449	static void __init rcu_organize_nocb_kthreads(void);	444	static void __init rcu_organize_nocb_kthreads(void);
450	#define rcu_nocb_lock_irqsave(rdp, flags) \	445
451	do { \	446	/*
452	if (!rcu_segcblist_is_offloaded(&(rdp)->cblist)) \	447	* Disable IRQs before checking offloaded state so that local
453	local_irq_save(flags); \	448	* locking is safe against concurrent de-offloading.
454	else \	449	*/
455	raw_spin_lock_irqsave(&(rdp)->nocb_lock, (flags)); \	450	#define rcu_nocb_lock_irqsave(rdp, flags) \
		451	do { \
		452	local_irq_save(flags); \
		453	if (rcu_segcblist_is_offloaded(&(rdp)->cblist)) \
		454	raw_spin_lock(&(rdp)->nocb_lock); \
456	} while (0)	455	} while (0)
457	#else /* #ifdef CONFIG_RCU_NOCB_CPU */	456	#else /* #ifdef CONFIG_RCU_NOCB_CPU */
458	#define rcu_nocb_lock_irqsave(rdp, flags) local_irq_save(flags)	457	#define rcu_nocb_lock_irqsave(rdp, flags) local_irq_save(flags)

kernel/rcu/tree_exp.h+9−5

  */
 static void rcu_report_exp_rdp(struct rcu_data *rdp)
 {
-	WRITE_ONCE(rdp->exp_deferred_qs, false);
+	WRITE_ONCE(rdp->cpu_no_qs.b.exp, false);
 	rcu_report_exp_cpu_mult(rdp->mynode, rdp->grpmask, true);
 }
 			continue;
 		}
 		if (get_cpu() == cpu) {
+			mask_ofl_test |= mask;
 			put_cpu();
 			continue;
 		}
 				if (rdp->rcu_forced_tick_exp)
 					continue;
 				rdp->rcu_forced_tick_exp = true;
-				tick_dep_set_cpu(cpu, TICK_DEP_BIT_RCU_EXP);
+				preempt_disable();
+				if (cpu_online(cpu))
+					tick_dep_set_cpu(cpu, TICK_DEP_BIT_RCU_EXP);
+				preempt_enable();
 			}
 		}
 		j = READ_ONCE(jiffies_till_first_fqs);
 		    rcu_dynticks_curr_cpu_in_eqs()) {
 			rcu_report_exp_rdp(rdp);
 		} else {
-			rdp->exp_deferred_qs = true;
+			WRITE_ONCE(rdp->cpu_no_qs.b.exp, true);
 			set_tsk_need_resched(t);
 			set_preempt_need_resched();
 		}
 	if (depth > 0) {
 		raw_spin_lock_irqsave_rcu_node(rnp, flags);
 		if (rnp->expmask & rdp->grpmask) {
-			rdp->exp_deferred_qs = true;
+			WRITE_ONCE(rdp->cpu_no_qs.b.exp, true);
 			t->rcu_read_unlock_special.b.exp_hint = true;
 		}
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	my_cpu = get_cpu();
 	/* Quiescent state either not needed or already requested, leave. */
 	if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) ||
-	    rdp->cpu_no_qs.b.exp) {
+	    READ_ONCE(rdp->cpu_no_qs.b.exp)) {
 		put_cpu();
 		return;
 	}

kernel/rcu/tree_nocb.h+106−54

@@ -60,16 +60,22 @@ static inline bool rcu_current_is_nocb_kthread(struct rcu_data *rdp) · 59 unmodified lines
60	* Parse the boot-time rcu_nocb_mask CPU list from the kernel parameters.	60	* Parse the boot-time rcu_nocb_mask CPU list from the kernel parameters.
61	* If the list is invalid, a warning is emitted and all CPUs are offloaded.	61	* If the list is invalid, a warning is emitted and all CPUs are offloaded.
62	*/	62	*/
		63
		64	static bool rcu_nocb_is_setup;
		65
63	static int __init rcu_nocb_setup(char *str)	66	static int __init rcu_nocb_setup(char *str)
64	{	67	{
65	alloc_bootmem_cpumask_var(&rcu_nocb_mask);	68	alloc_bootmem_cpumask_var(&rcu_nocb_mask);
66	if (cpulist_parse(str, rcu_nocb_mask)) {	69	if (*str == '=') {
67	pr_warn("rcu_nocbs= bad CPU range, all CPUs set\n");	70	if (cpulist_parse(++str, rcu_nocb_mask)) {
68	cpumask_setall(rcu_nocb_mask);	71	pr_warn("rcu_nocbs= bad CPU range, all CPUs set\n");
		72	cpumask_setall(rcu_nocb_mask);
		73	}
69	}	74	}
		75	rcu_nocb_is_setup = true;
70	return 1;	76	return 1;
71	}	77	}
72	__setup("rcu_nocbs=", rcu_nocb_setup);	78	__setup("rcu_nocbs", rcu_nocb_setup);
73		79
74	static int __init parse_rcu_nocb_poll(char *arg)	80	static int __init parse_rcu_nocb_poll(char *arg)
75	{	81	{
@@ -625,7 +631,21 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) · 549 unmodified lines
625	* and the global grace-period kthread are awakened if needed.	631	* and the global grace-period kthread are awakened if needed.
626	*/	632	*/
627	WARN_ON_ONCE(my_rdp->nocb_gp_rdp != my_rdp);	633	WARN_ON_ONCE(my_rdp->nocb_gp_rdp != my_rdp);
628	for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {	634	/*
		635	* An rcu_data structure is removed from the list after its
		636	* CPU is de-offloaded and added to the list before that CPU is
		637	* (re-)offloaded. If the following loop happens to be referencing
		638	* that rcu_data structure during the time that the corresponding
		639	* CPU is de-offloaded and then immediately re-offloaded, this
		640	* loop's rdp pointer will be carried to the end of the list by
		641	* the resulting pair of list operations. This can cause the loop
		642	* to skip over some of the rcu_data structures that were supposed
		643	* to have been scanned. Fortunately a new iteration through the
		644	* entire loop is forced after a given CPU's rcu_data structure
		645	* is added to the list, so the skipped-over rcu_data structures
		646	* won't be ignored for long.
		647	*/
		648	list_for_each_entry_rcu(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp, 1) {
629	bool needwake_state = false;	649	bool needwake_state = false;
630		650
631	if (!nocb_gp_enabled_cb(rdp))	651	if (!nocb_gp_enabled_cb(rdp))
@@ -789,6 +809,18 @@ static void nocb_cb_wait(struct rcu_data *rdp) · 157 unmodified lines
789	bool can_sleep = true;	809	bool can_sleep = true;
790	struct rcu_node *rnp = rdp->mynode;	810	struct rcu_node *rnp = rdp->mynode;
791		811
		812	do {
		813	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
		814	nocb_cb_wait_cond(rdp));
		815
		816	// VVV Ensure CB invocation follows _sleep test.
		817	if (smp_load_acquire(&rdp->nocb_cb_sleep)) { // ^^^
		818	WARN_ON(signal_pending(current));
		819	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
		820	}
		821	} while (!nocb_cb_can_run(rdp));
		822
		823
792	local_irq_save(flags);	824	local_irq_save(flags);
793	rcu_momentary_dyntick_idle();	825	rcu_momentary_dyntick_idle();
794	local_irq_restore(flags);	826	local_irq_restore(flags);
@@ -841,17 +873,6 @@ static void nocb_cb_wait(struct rcu_data *rdp) · 46 unmodified lines
841		873
842	if (needwake_state)	874	if (needwake_state)
843	swake_up_one(&rdp->nocb_state_wq);	875	swake_up_one(&rdp->nocb_state_wq);
844
845	do {
846	swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
847	nocb_cb_wait_cond(rdp));
848
849	// VVV Ensure CB invocation follows _sleep test.
850	if (smp_load_acquire(&rdp->nocb_cb_sleep)) { // ^^^
851	WARN_ON(signal_pending(current));
852	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
853	}
854	} while (!nocb_cb_can_run(rdp));
855	}	876	}
856		877
857	/*	878	/*
@@ -990,22 +1011,33 @@ static long rcu_nocb_rdp_deoffload(void *arg) · 132 unmodified lines
990	* will refuse to put anything into the bypass.	1011	* will refuse to put anything into the bypass.
991	*/	1012	*/
992	WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies));	1013	WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies));
		1014	/*
		1015	* Start with invoking rcu_core() early. This way if the current thread
		1016	* happens to preempt an ongoing call to rcu_core() in the middle,
		1017	* leaving some work dismissed because rcu_core() still thinks the rdp is
		1018	* completely offloaded, we are guaranteed a nearby future instance of
		1019	* rcu_core() to catch up.
		1020	*/
		1021	rcu_segcblist_set_flags(cblist, SEGCBLIST_RCU_CORE);
		1022	invoke_rcu_core();
993	ret = rdp_offload_toggle(rdp, false, flags);	1023	ret = rdp_offload_toggle(rdp, false, flags);
994	swait_event_exclusive(rdp->nocb_state_wq,	1024	swait_event_exclusive(rdp->nocb_state_wq,
995	!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB \|	1025	!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB \|
996	SEGCBLIST_KTHREAD_GP));	1026	SEGCBLIST_KTHREAD_GP));
		1027	/* Stop nocb_gp_wait() from iterating over this structure. */
		1028	list_del_rcu(&rdp->nocb_entry_rdp);
997	/*	1029	/*
998	* Lock one last time to acquire latest callback updates from kthreads	1030	* Lock one last time to acquire latest callback updates from kthreads
999	* so we can later handle callbacks locally without locking.	1031	* so we can later handle callbacks locally without locking.
1000	*/	1032	*/
1001	rcu_nocb_lock_irqsave(rdp, flags);	1033	rcu_nocb_lock_irqsave(rdp, flags);
1002	/*	1034	/*
1003	* Theoretically we could set SEGCBLIST_SOFTIRQ_ONLY after the nocb	1035	* Theoretically we could clear SEGCBLIST_LOCKING after the nocb
1004	* lock is released but how about being paranoid for once?	1036	* lock is released but how about being paranoid for once?
1005	*/	1037	*/
1006	rcu_segcblist_set_flags(cblist, SEGCBLIST_SOFTIRQ_ONLY);	1038	rcu_segcblist_clear_flags(cblist, SEGCBLIST_LOCKING);
1007	/*	1039	/*
1008	* With SEGCBLIST_SOFTIRQ_ONLY, we can't use	1040	* Without SEGCBLIST_LOCKING, we can't use
1009	* rcu_nocb_unlock_irqrestore() anymore.	1041	* rcu_nocb_unlock_irqrestore() anymore.
1010	*/	1042	*/
1011	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);	1043	raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
@@ -1057,15 +1089,26 @@ static long rcu_nocb_rdp_offload(void *arg) · 45 unmodified lines
1057	return -EINVAL;	1089	return -EINVAL;
1058		1090
1059	pr_info("Offloading %d\n", rdp->cpu);	1091	pr_info("Offloading %d\n", rdp->cpu);
		1092
		1093	/*
		1094	* Cause future nocb_gp_wait() invocations to iterate over
		1095	* structure, resetting ->nocb_gp_sleep and waking up the related
		1096	* "rcuog". Since nocb_gp_wait() in turn locks ->nocb_gp_lock
		1097	* before setting ->nocb_gp_sleep again, we are guaranteed to
		1098	* iterate this newly added structure before "rcuog" goes to
		1099	* sleep again.
		1100	*/
		1101	list_add_tail_rcu(&rdp->nocb_entry_rdp, &rdp->nocb_gp_rdp->nocb_head_rdp);
		1102
1060	/*	1103	/*
1061	* Can't use rcu_nocb_lock_irqsave() while we are in	1104	* Can't use rcu_nocb_lock_irqsave() before SEGCBLIST_LOCKING
1062	* SEGCBLIST_SOFTIRQ_ONLY mode.	1105	* is set.
1063	*/	1106	*/
1064	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);	1107	raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
1065		1108
1066	/*	1109	/*
1067	* We didn't take the nocb lock while working on the	1110	* We didn't take the nocb lock while working on the
1068	* rdp->cblist in SEGCBLIST_SOFTIRQ_ONLY mode.	1111	* rdp->cblist with SEGCBLIST_LOCKING cleared (pure softirq/rcuc mode).
1069	* Every modifications that have been done previously on	1112	* Every modifications that have been done previously on
1070	* rdp->cblist must be visible remotely by the nocb kthreads	1113	* rdp->cblist must be visible remotely by the nocb kthreads
1071	* upon wake up after reading the cblist flags.	1114	* upon wake up after reading the cblist flags.
@@ -1084,6 +1127,14 @@ static long rcu_nocb_rdp_offload(void *arg) · 12 unmodified lines
1084	rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB) &&	1127	rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB) &&
1085	rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP));	1128	rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP));
1086		1129
		1130	/*
		1131	* All kthreads are ready to work, we can finally relieve rcu_core() and
		1132	* enable nocb bypass.
		1133	*/
		1134	rcu_nocb_lock_irqsave(rdp, flags);
		1135	rcu_segcblist_clear_flags(cblist, SEGCBLIST_RCU_CORE);
		1136	rcu_nocb_unlock_irqrestore(rdp, flags);
		1137
1087	return ret;	1138	return ret;
1088	}	1139	}
1089		1140
@@ -1122,13 +1173,17 @@ void __init rcu_init_nohz(void) · 32 unmodified lines
1122	need_rcu_nocb_mask = true;	1173	need_rcu_nocb_mask = true;
1123	#endif /* #if defined(CONFIG_NO_HZ_FULL) */	1174	#endif /* #if defined(CONFIG_NO_HZ_FULL) */
1124		1175
1125	if (!cpumask_available(rcu_nocb_mask) && need_rcu_nocb_mask) {	1176	if (need_rcu_nocb_mask) {
1126	if (!zalloc_cpumask_var(&rcu_nocb_mask, GFP_KERNEL)) {	1177	if (!cpumask_available(rcu_nocb_mask)) {
1127	pr_info("rcu_nocb_mask allocation failed, callback offloading disabled.\n");	1178	if (!zalloc_cpumask_var(&rcu_nocb_mask, GFP_KERNEL)) {
1128	return;	1179	pr_info("rcu_nocb_mask allocation failed, callback offloading disabled.\n");
		1180	return;
		1181	}
1129	}	1182	}
		1183	rcu_nocb_is_setup = true;
1130	}	1184	}
1131	if (!cpumask_available(rcu_nocb_mask))	1185
		1186	if (!rcu_nocb_is_setup)
1132	return;	1187	return;
1133		1188
1134	#if defined(CONFIG_NO_HZ_FULL)	1189	#if defined(CONFIG_NO_HZ_FULL)
@@ -1154,8 +1209,8 @@ void __init rcu_init_nohz(void) · 19 unmodified lines
1154	if (rcu_segcblist_empty(&rdp->cblist))	1209	if (rcu_segcblist_empty(&rdp->cblist))
1155	rcu_segcblist_init(&rdp->cblist);	1210	rcu_segcblist_init(&rdp->cblist);
1156	rcu_segcblist_offload(&rdp->cblist, true);	1211	rcu_segcblist_offload(&rdp->cblist, true);
1157	rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB);	1212	rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB \| SEGCBLIST_KTHREAD_GP);
1158	rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_GP);	1213	rcu_segcblist_clear_flags(&rdp->cblist, SEGCBLIST_RCU_CORE);
1159	}	1214	}
1160	rcu_organize_nocb_kthreads();	1215	rcu_organize_nocb_kthreads();
1161	}	1216	}
@@ -1178,17 +1233,17 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) · 16 unmodified lines
1178	* rcuo CB kthread, spawn it. Additionally, if the rcuo GP kthread	1233	* rcuo CB kthread, spawn it. Additionally, if the rcuo GP kthread
1179	* for this CPU's group has not yet been created, spawn it as well.	1234	* for this CPU's group has not yet been created, spawn it as well.
1180	*/	1235	*/
1181	static void rcu_spawn_one_nocb_kthread(int cpu)	1236	static void rcu_spawn_cpu_nocb_kthread(int cpu)
1182	{	1237	{
1183	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);	1238	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
1184	struct rcu_data *rdp_gp;	1239	struct rcu_data *rdp_gp;
1185	struct task_struct *t;	1240	struct task_struct *t;
1186		1241
1187	/*	1242	if (!rcu_scheduler_fully_active \|\| !rcu_nocb_is_setup)
1188	* If this isn't a no-CBs CPU or if it already has an rcuo kthread,	1243	return;
1189	* then nothing to do.	1244
1190	*/	1245	/* If there already is an rcuo kthread, then nothing to do. */
1191	if (!rcu_is_nocb_cpu(cpu) \|\| rdp->nocb_cb_kthread)	1246	if (rdp->nocb_cb_kthread)
1192	return;	1247	return;
1193		1248
1194	/* If we didn't spawn the GP kthread first, reorganize! */	1249	/* If we didn't spawn the GP kthread first, reorganize! */
@@ -1210,16 +1265,6 @@ static void rcu_spawn_one_nocb_kthread(int cpu) · 15 unmodified lines
1210	WRITE_ONCE(rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread);	1265	WRITE_ONCE(rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread);
1211	}	1266	}
1212		1267
1213	/*
1214	* If the specified CPU is a no-CBs CPU that does not already have its
1215	* rcuo kthread, spawn it.
1216	*/
1217	static void rcu_spawn_cpu_nocb_kthread(int cpu)
1218	{
1219	if (rcu_scheduler_fully_active)
1220	rcu_spawn_one_nocb_kthread(cpu);
1221	}
1222
1223	/*	1268	/*
1224	* Once the scheduler is running, spawn rcuo kthreads for all online	1269	* Once the scheduler is running, spawn rcuo kthreads for all online
1225	* no-CBs CPUs. This assumes that the early_initcall()s happen before	1270	* no-CBs CPUs. This assumes that the early_initcall()s happen before
@@ -1230,8 +1275,10 @@ static void __init rcu_spawn_nocb_kthreads(void) · 4 unmodified lines
1230	{	1275	{
1231	int cpu;	1276	int cpu;
1232		1277
1233	for_each_online_cpu(cpu)	1278	if (rcu_nocb_is_setup) {
1234	rcu_spawn_cpu_nocb_kthread(cpu);	1279	for_each_online_cpu(cpu)
		1280	rcu_spawn_cpu_nocb_kthread(cpu);
		1281	}
1235	}	1282	}
1236		1283
1237	/* How many CB CPU IDs per GP kthread? Default of -1 for sqrt(nr_cpu_ids). */	1284	/* How many CB CPU IDs per GP kthread? Default of -1 for sqrt(nr_cpu_ids). */
@@ -1251,7 +1298,6 @@ static void __init rcu_organize_nocb_kthreads(void) · 13 unmodified lines
1251	int nl = 0; /* Next GP kthread. */	1298	int nl = 0; /* Next GP kthread. */
1252	struct rcu_data *rdp;	1299	struct rcu_data *rdp;
1253	struct rcu_data rdp_gp = NULL; / Suppress misguided gcc warn. */	1300	struct rcu_data rdp_gp = NULL; / Suppress misguided gcc warn. */
1254	struct rcu_data *rdp_prev = NULL;
1255		1301
1256	if (!cpumask_available(rcu_nocb_mask))	1302	if (!cpumask_available(rcu_nocb_mask))
1257	return;	1303	return;
@@ -1265,14 +1311,14 @@ static void __init rcu_organize_nocb_kthreads(void) · 7 unmodified lines
1265	* Should the corresponding CPU come online in the future, then	1311	* Should the corresponding CPU come online in the future, then
1266	* we will spawn the needed set of rcu_nocb_kthread() kthreads.	1312	* we will spawn the needed set of rcu_nocb_kthread() kthreads.
1267	*/	1313	*/
1268	for_each_cpu(cpu, rcu_nocb_mask) {	1314	for_each_possible_cpu(cpu) {
1269	rdp = per_cpu_ptr(&rcu_data, cpu);	1315	rdp = per_cpu_ptr(&rcu_data, cpu);
1270	if (rdp->cpu >= nl) {	1316	if (rdp->cpu >= nl) {
1271	/* New GP kthread, set up for CBs & next GP. */	1317	/* New GP kthread, set up for CBs & next GP. */
1272	gotnocbs = true;	1318	gotnocbs = true;
1273	nl = DIV_ROUND_UP(rdp->cpu + 1, ls) * ls;	1319	nl = DIV_ROUND_UP(rdp->cpu + 1, ls) * ls;
1274	rdp->nocb_gp_rdp = rdp;
1275	rdp_gp = rdp;	1320	rdp_gp = rdp;
		1321	INIT_LIST_HEAD(&rdp->nocb_head_rdp);
1276	if (dump_tree) {	1322	if (dump_tree) {
1277	if (!firsttime)	1323	if (!firsttime)
1278	pr_cont("%s\n", gotnocbscbs	1324	pr_cont("%s\n", gotnocbscbs
@@ -1285,12 +1331,12 @@ static void __init rcu_organize_nocb_kthreads(void) · 6 unmodified lines
1285	} else {	1331	} else {
1286	/* Another CB kthread, link to previous GP kthread. */	1332	/* Another CB kthread, link to previous GP kthread. */
1287	gotnocbscbs = true;	1333	gotnocbscbs = true;
1288	rdp->nocb_gp_rdp = rdp_gp;
1289	rdp_prev->nocb_next_cb_rdp = rdp;
1290	if (dump_tree)	1334	if (dump_tree)
1291	pr_cont(" %d", cpu);	1335	pr_cont(" %d", cpu);
1292	}	1336	}
1293	rdp_prev = rdp;	1337	rdp->nocb_gp_rdp = rdp_gp;
		1338	if (cpumask_test_cpu(cpu, rcu_nocb_mask))
		1339	list_add_tail(&rdp->nocb_entry_rdp, &rdp_gp->nocb_head_rdp);
1294	}	1340	}
1295	if (gotnocbs && dump_tree)	1341	if (gotnocbs && dump_tree)
1296	pr_cont("%s\n", gotnocbscbs ? "" : " (self only)");	1342	pr_cont("%s\n", gotnocbscbs ? "" : " (self only)");
@@ -1352,6 +1398,7 @@ static void show_rcu_nocb_state(struct rcu_data *rdp) · 55 unmodified lines
1352	{	1398	{
1353	char bufw[20];	1399	char bufw[20];
1354	char bufr[20];	1400	char bufr[20];
		1401	struct rcu_data *nocb_next_rdp;
1355	struct rcu_segcblist *rsclp = &rdp->cblist;	1402	struct rcu_segcblist *rsclp = &rdp->cblist;
1356	bool waslocked;	1403	bool waslocked;
1357	bool wassleep;	1404	bool wassleep;
@@ -1359,11 +1406,16 @@ static void show_rcu_nocb_state(struct rcu_data *rdp) · 1 unmodified line
1359	if (rdp->nocb_gp_rdp == rdp)	1406	if (rdp->nocb_gp_rdp == rdp)
1360	show_rcu_nocb_gp_state(rdp);	1407	show_rcu_nocb_gp_state(rdp);
1361		1408
		1409	nocb_next_rdp = list_next_or_null_rcu(&rdp->nocb_gp_rdp->nocb_head_rdp,
		1410	&rdp->nocb_entry_rdp,
		1411	typeof(*rdp),
		1412	nocb_entry_rdp);
		1413
1362	sprintf(bufw, "%ld", rsclp->gp_seq[RCU_WAIT_TAIL]);	1414	sprintf(bufw, "%ld", rsclp->gp_seq[RCU_WAIT_TAIL]);
1363	sprintf(bufr, "%ld", rsclp->gp_seq[RCU_NEXT_READY_TAIL]);	1415	sprintf(bufr, "%ld", rsclp->gp_seq[RCU_NEXT_READY_TAIL]);
1364	pr_info(" CB %d^%d->%d %c%c%c%c%c%c F%ld L%ld C%d %c%c%s%c%s%c%c q%ld %c CPU %d%s\n",	1416	pr_info(" CB %d^%d->%d %c%c%c%c%c%c F%ld L%ld C%d %c%c%s%c%s%c%c q%ld %c CPU %d%s\n",
1365	rdp->cpu, rdp->nocb_gp_rdp->cpu,	1417	rdp->cpu, rdp->nocb_gp_rdp->cpu,
1366	rdp->nocb_next_cb_rdp ? rdp->nocb_next_cb_rdp->cpu : -1,	1418	nocb_next_rdp ? nocb_next_rdp->cpu : -1,
1367	"kK"[!!rdp->nocb_cb_kthread],	1419	"kK"[!!rdp->nocb_cb_kthread],
1368	"bB"[raw_spin_is_locked(&rdp->nocb_bypass_lock)],	1420	"bB"[raw_spin_is_locked(&rdp->nocb_bypass_lock)],
1369	"cC"[!!atomic_read(&rdp->nocb_lock_contended)],	1421	"cC"[!!atomic_read(&rdp->nocb_lock_contended)],

kernel/rcu/tree_plugin.h+33−217

@@ -16,7 +16,7 @@ · 15 unmodified lines
16	static bool rcu_rdp_is_offloaded(struct rcu_data *rdp)	16	static bool rcu_rdp_is_offloaded(struct rcu_data *rdp)
17	{	17	{
18	/*	18	/*
19	* In order to read the offloaded state of an rdp is a safe	19	* In order to read the offloaded state of an rdp in a safe
20	* and stable way and prevent from its value to be changed	20	* and stable way and prevent from its value to be changed
21	* under us, we must either hold the barrier mutex, the cpu	21	* under us, we must either hold the barrier mutex, the cpu
22	* hotplug lock (read or write) or the nocb lock. Local	22	* hotplug lock (read or write) or the nocb lock. Local
@@ -51,12 +51,10 @@ static void __init rcu_bootup_announce_oddness(void) · 28 unmodified lines
51	RCU_FANOUT);	51	RCU_FANOUT);
52	if (rcu_fanout_exact)	52	if (rcu_fanout_exact)
53	pr_info("\tHierarchical RCU autobalancing is disabled.\n");	53	pr_info("\tHierarchical RCU autobalancing is disabled.\n");
54	if (IS_ENABLED(CONFIG_RCU_FAST_NO_HZ))
55	pr_info("\tRCU dyntick-idle grace-period acceleration is enabled.\n");
56	if (IS_ENABLED(CONFIG_PROVE_RCU))	54	if (IS_ENABLED(CONFIG_PROVE_RCU))
57	pr_info("\tRCU lockdep checking is enabled.\n");	55	pr_info("\tRCU lockdep checking is enabled.\n");
58	if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD))	56	if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD))
59	pr_info("\tRCU strict (and thus non-scalable) grace periods enabled.\n");	57	pr_info("\tRCU strict (and thus non-scalable) grace periods are enabled.\n");
60	if (RCU_NUM_LVLS >= 4)	58	if (RCU_NUM_LVLS >= 4)
61	pr_info("\tFour(or more)-level hierarchy is enabled.\n");	59	pr_info("\tFour(or more)-level hierarchy is enabled.\n");
62	if (RCU_FANOUT_LEAF != 16)	60	if (RCU_FANOUT_LEAF != 16)
@@ -88,13 +86,13 @@ static void __init rcu_bootup_announce_oddness(void) · 25 unmodified lines
88	if (rcu_kick_kthreads)	86	if (rcu_kick_kthreads)
89	pr_info("\tKick kthreads if too-long grace period.\n");	87	pr_info("\tKick kthreads if too-long grace period.\n");
90	if (IS_ENABLED(CONFIG_DEBUG_OBJECTS_RCU_HEAD))	88	if (IS_ENABLED(CONFIG_DEBUG_OBJECTS_RCU_HEAD))
91	pr_info("\tRCU callback double-/use-after-free debug enabled.\n");	89	pr_info("\tRCU callback double-/use-after-free debug is enabled.\n");
92	if (gp_preinit_delay)	90	if (gp_preinit_delay)
93	pr_info("\tRCU debug GP pre-init slowdown %d jiffies.\n", gp_preinit_delay);	91	pr_info("\tRCU debug GP pre-init slowdown %d jiffies.\n", gp_preinit_delay);
94	if (gp_init_delay)	92	if (gp_init_delay)
95	pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_init_delay);	93	pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_init_delay);
96	if (gp_cleanup_delay)	94	if (gp_cleanup_delay)
97	pr_info("\tRCU debug GP init slowdown %d jiffies.\n", gp_cleanup_delay);	95	pr_info("\tRCU debug GP cleanup slowdown %d jiffies.\n", gp_cleanup_delay);
98	if (!use_softirq)	96	if (!use_softirq)
99	pr_info("\tRCU_SOFTIRQ processing moved to rcuc kthreads.\n");	97	pr_info("\tRCU_SOFTIRQ processing moved to rcuc kthreads.\n");
100	if (IS_ENABLED(CONFIG_RCU_EQS_DEBUG))	98	if (IS_ENABLED(CONFIG_RCU_EQS_DEBUG))
@@ -260,10 +258,10 @@ static void rcu_preempt_ctxt_queue(struct rcu_node rnp, struct rcu_data rdp) · 159 unmodified lines
260	* no need to check for a subsequent expedited GP. (Though we are	258	* no need to check for a subsequent expedited GP. (Though we are
261	* still in a quiescent state in any case.)	259	* still in a quiescent state in any case.)
262	*/	260	*/
263	if (blkd_state & RCU_EXP_BLKD && rdp->exp_deferred_qs)	261	if (blkd_state & RCU_EXP_BLKD && rdp->cpu_no_qs.b.exp)
264	rcu_report_exp_rdp(rdp);	262	rcu_report_exp_rdp(rdp);
265	else	263	else
266	WARN_ON_ONCE(rdp->exp_deferred_qs);	264	WARN_ON_ONCE(rdp->cpu_no_qs.b.exp);
267	}	265	}
268		266
269	/*	267	/*
@@ -277,12 +275,16 @@ static void rcu_preempt_ctxt_queue(struct rcu_node rnp, struct rcu_data rdp) · 7 unmodified lines
277	* current task, there might be any number of other tasks blocked while	275	* current task, there might be any number of other tasks blocked while
278	* in an RCU read-side critical section.	276	* in an RCU read-side critical section.
279	*	277	*
		278	* Unlike non-preemptible-RCU, quiescent state reports for expedited
		279	* grace periods are handled separately via deferred quiescent states
		280	* and context switch events.
		281	*
280	* Callers to this function must disable preemption.	282	* Callers to this function must disable preemption.
281	*/	283	*/
282	static void rcu_qs(void)	284	static void rcu_qs(void)
283	{	285	{
284	RCU_LOCKDEP_WARN(preemptible(), "rcu_qs() invoked with preemption enabled!!!\n");	286	RCU_LOCKDEP_WARN(preemptible(), "rcu_qs() invoked with preemption enabled!!!\n");
285	if (__this_cpu_read(rcu_data.cpu_no_qs.s)) {	287	if (__this_cpu_read(rcu_data.cpu_no_qs.b.norm)) {
286	trace_rcu_grace_period(TPS("rcu_preempt"),	288	trace_rcu_grace_period(TPS("rcu_preempt"),
287	__this_cpu_read(rcu_data.gp_seq),	289	__this_cpu_read(rcu_data.gp_seq),
288	TPS("cpuqs"));	290	TPS("cpuqs"));
@@ -350,7 +352,7 @@ void rcu_note_context_switch(bool preempt) · 61 unmodified lines
350	* means that we continue to block the current grace period.	352	* means that we continue to block the current grace period.
351	*/	353	*/
352	rcu_qs();	354	rcu_qs();
353	if (rdp->exp_deferred_qs)	355	if (rdp->cpu_no_qs.b.exp)
354	rcu_report_exp_rdp(rdp);	356	rcu_report_exp_rdp(rdp);
355	rcu_tasks_qs(current, preempt);	357	rcu_tasks_qs(current, preempt);
356	trace_rcu_utilization(TPS("End context switch"));	358	trace_rcu_utilization(TPS("End context switch"));
@@ -477,7 +479,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 120 unmodified lines
477	*/	479	*/
478	special = t->rcu_read_unlock_special;	480	special = t->rcu_read_unlock_special;
479	rdp = this_cpu_ptr(&rcu_data);	481	rdp = this_cpu_ptr(&rcu_data);
480	if (!special.s && !rdp->exp_deferred_qs) {	482	if (!special.s && !rdp->cpu_no_qs.b.exp) {
481	local_irq_restore(flags);	483	local_irq_restore(flags);
482	return;	484	return;
483	}	485	}
@@ -497,7 +499,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 13 unmodified lines
497	* tasks are handled when removing the task from the	499	* tasks are handled when removing the task from the
498	* blocked-tasks list below.	500	* blocked-tasks list below.
499	*/	501	*/
500	if (rdp->exp_deferred_qs)	502	if (rdp->cpu_no_qs.b.exp)
501	rcu_report_exp_rdp(rdp);	503	rcu_report_exp_rdp(rdp);
502		504
503	/* Clean up if blocked during RCU read-side critical section. */	505	/* Clean up if blocked during RCU read-side critical section. */
@@ -580,7 +582,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 76 unmodified lines
580	*/	582	*/
581	static bool rcu_preempt_need_deferred_qs(struct task_struct *t)	583	static bool rcu_preempt_need_deferred_qs(struct task_struct *t)
582	{	584	{
583	return (__this_cpu_read(rcu_data.exp_deferred_qs) \|\|	585	return (__this_cpu_read(rcu_data.cpu_no_qs.b.exp) \|\|
584	READ_ONCE(t->rcu_read_unlock_special.s)) &&	586	READ_ONCE(t->rcu_read_unlock_special.s)) &&
585	rcu_preempt_depth() == 0;	587	rcu_preempt_depth() == 0;
586	}	588	}
@@ -642,7 +644,7 @@ static void rcu_read_unlock_special(struct task_struct *t) · 55 unmodified lines
642	(IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled &&	644	(IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled &&
643	t->rcu_blocked_node);	645	t->rcu_blocked_node);
644	// Need to defer quiescent state until everything is enabled.	646	// Need to defer quiescent state until everything is enabled.
645	if (use_softirq && (in_irq() \|\| (expboost && !irqs_were_disabled))) {	647	if (use_softirq && (in_hardirq() \|\| (expboost && !irqs_were_disabled))) {
646	// Using softirq, safe to awaken, and either the	648	// Using softirq, safe to awaken, and either the
647	// wakeup is free or there is either an expedited	649	// wakeup is free or there is either an expedited
648	// GP in flight or a potential need to deboost.	650	// GP in flight or a potential need to deboost.
@@ -845,10 +847,8 @@ static void rcu_qs(void) · 196 unmodified lines
845	trace_rcu_grace_period(TPS("rcu_sched"),	847	trace_rcu_grace_period(TPS("rcu_sched"),
846	__this_cpu_read(rcu_data.gp_seq), TPS("cpuqs"));	848	__this_cpu_read(rcu_data.gp_seq), TPS("cpuqs"));
847	__this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);	849	__this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);
848	if (!__this_cpu_read(rcu_data.cpu_no_qs.b.exp))	850	if (__this_cpu_read(rcu_data.cpu_no_qs.b.exp))
849	return;	851	rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
850	__this_cpu_write(rcu_data.cpu_no_qs.b.exp, false);
851	rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
852	}	852	}
853		853
854	/*	854	/*
@@ -925,7 +925,18 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t) · 70 unmodified lines
925	{	925	{
926	return false;	926	return false;
927	}	927	}
928	static void rcu_preempt_deferred_qs(struct task_struct *t) { }	928
		929	// Except that we do need to respond to a request by an expedited grace
		930	// period for a quiescent state from this CPU. Note that requests from
		931	// tasks are handled when removing the task from the blocked-tasks list
		932	// below.
		933	static void rcu_preempt_deferred_qs(struct task_struct *t)
		934	{
		935	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
		936
		937	if (rdp->cpu_no_qs.b.exp)
		938	rcu_report_exp_rdp(rdp);
		939	}
929		940
930	/*	941	/*
931	* Because there is no preemptible RCU, there can be no readers blocked,	942	* Because there is no preemptible RCU, there can be no readers blocked,
@@ -1153,7 +1164,6 @@ static void rcu_preempt_boost_start_gp(struct rcu_node *rnp) · 221 unmodified lines
1153	/*	1164	/*
1154	* Create an RCU-boost kthread for the specified node if one does not	1165	* Create an RCU-boost kthread for the specified node if one does not
1155	* already exist. We only create this kthread for preemptible RCU.	1166	* already exist. We only create this kthread for preemptible RCU.
1156	* Returns zero if all is well, a negated errno otherwise.
1157	*/	1167	*/
1158	static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp)	1168	static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp)
1159	{	1169	{
@@ -1204,8 +1214,9 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) · 44 unmodified lines
1204	if ((mask & leaf_node_cpu_bit(rnp, cpu)) &&	1214	if ((mask & leaf_node_cpu_bit(rnp, cpu)) &&
1205	cpu != outgoingcpu)	1215	cpu != outgoingcpu)
1206	cpumask_set_cpu(cpu, cm);	1216	cpumask_set_cpu(cpu, cm);
		1217	cpumask_and(cm, cm, housekeeping_cpumask(HK_FLAG_RCU));
1207	if (cpumask_weight(cm) == 0)	1218	if (cpumask_weight(cm) == 0)
1208	cpumask_setall(cm);	1219	cpumask_copy(cm, housekeeping_cpumask(HK_FLAG_RCU));
1209	set_cpus_allowed_ptr(t, cm);	1220	set_cpus_allowed_ptr(t, cm);
1210	free_cpumask_var(cm);	1221	free_cpumask_var(cm);
1211	}	1222	}
@@ -1253,201 +1264,6 @@ static void __init rcu_spawn_boost_kthreads(void) · 41 unmodified lines
1253		1264
1254	#endif /* #else #ifdef CONFIG_RCU_BOOST */	1265	#endif /* #else #ifdef CONFIG_RCU_BOOST */
1255		1266
1256	#if !defined(CONFIG_RCU_FAST_NO_HZ)
1257
1258	/*
1259	* Check to see if any future non-offloaded RCU-related work will need
1260	* to be done by the current CPU, even if none need be done immediately,
1261	* returning 1 if so. This function is part of the RCU implementation;
1262	* it is -not- an exported member of the RCU API.
1263	*
1264	* Because we not have RCU_FAST_NO_HZ, just check whether or not this
1265	* CPU has RCU callbacks queued.
1266	*/
1267	int rcu_needs_cpu(u64 basemono, u64 *nextevt)
1268	{
1269	*nextevt = KTIME_MAX;
1270	return !rcu_segcblist_empty(&this_cpu_ptr(&rcu_data)->cblist) &&
1271	!rcu_rdp_is_offloaded(this_cpu_ptr(&rcu_data));
1272	}
1273
1274	/*
1275	* Because we do not have RCU_FAST_NO_HZ, don't bother cleaning up
1276	* after it.
1277	*/
1278	static void rcu_cleanup_after_idle(void)
1279	{
1280	}
1281
1282	/*
1283	* Do the idle-entry grace-period work, which, because CONFIG_RCU_FAST_NO_HZ=n,
1284	* is nothing.
1285	*/
1286	static void rcu_prepare_for_idle(void)
1287	{
1288	}
1289
1290	#else /* #if !defined(CONFIG_RCU_FAST_NO_HZ) */
1291
1292	/*
1293	* This code is invoked when a CPU goes idle, at which point we want
1294	* to have the CPU do everything required for RCU so that it can enter
1295	* the energy-efficient dyntick-idle mode.
1296	*
1297	* The following preprocessor symbol controls this:
1298	*
1299	* RCU_IDLE_GP_DELAY gives the number of jiffies that a CPU is permitted
1300	* to sleep in dyntick-idle mode with RCU callbacks pending. This
1301	* is sized to be roughly one RCU grace period. Those energy-efficiency
1302	* benchmarkers who might otherwise be tempted to set this to a large
1303	* number, be warned: Setting RCU_IDLE_GP_DELAY too high can hang your
1304	* system. And if you are -that- concerned about energy efficiency,
1305	* just power the system down and be done with it!
1306	*
1307	* The value below works well in practice. If future workloads require
1308	* adjustment, they can be converted into kernel config parameters, though
1309	* making the state machine smarter might be a better option.
1310	*/
1311	#define RCU_IDLE_GP_DELAY 4 /* Roughly one grace period. */
1312
1313	static int rcu_idle_gp_delay = RCU_IDLE_GP_DELAY;
1314	module_param(rcu_idle_gp_delay, int, 0644);
1315
1316	/*
1317	* Try to advance callbacks on the current CPU, but only if it has been
1318	* awhile since the last time we did so. Afterwards, if there are any
1319	* callbacks ready for immediate invocation, return true.
1320	*/
1321	static bool __maybe_unused rcu_try_advance_all_cbs(void)
1322	{
1323	bool cbs_ready = false;
1324	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
1325	struct rcu_node *rnp;
1326
1327	/* Exit early if we advanced recently. */
1328	if (jiffies == rdp->last_advance_all)
1329	return false;
1330	rdp->last_advance_all = jiffies;
1331
1332	rnp = rdp->mynode;
1333
1334	/*
1335	* Don't bother checking unless a grace period has
1336	* completed since we last checked and there are
1337	* callbacks not yet ready to invoke.
1338	*/
1339	if ((rcu_seq_completed_gp(rdp->gp_seq,
1340	rcu_seq_current(&rnp->gp_seq)) \|\|
1341	unlikely(READ_ONCE(rdp->gpwrap))) &&
1342	rcu_segcblist_pend_cbs(&rdp->cblist))
1343	note_gp_changes(rdp);
1344
1345	if (rcu_segcblist_ready_cbs(&rdp->cblist))
1346	cbs_ready = true;
1347	return cbs_ready;
1348	}
1349
1350	/*
1351	* Allow the CPU to enter dyntick-idle mode unless it has callbacks ready
1352	* to invoke. If the CPU has callbacks, try to advance them. Tell the
1353	* caller about what to set the timeout.
1354	*
1355	* The caller must have disabled interrupts.
1356	*/
1357	int rcu_needs_cpu(u64 basemono, u64 *nextevt)
1358	{
1359	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
1360	unsigned long dj;
1361
1362	lockdep_assert_irqs_disabled();
1363
1364	/* If no non-offloaded callbacks, RCU doesn't need the CPU. */
1365	if (rcu_segcblist_empty(&rdp->cblist) \|\|
1366	rcu_rdp_is_offloaded(rdp)) {
1367	*nextevt = KTIME_MAX;
1368	return 0;
1369	}
1370
1371	/* Attempt to advance callbacks. */
1372	if (rcu_try_advance_all_cbs()) {
1373	/* Some ready to invoke, so initiate later invocation. */
1374	invoke_rcu_core();
1375	return 1;
1376	}
1377	rdp->last_accelerate = jiffies;
1378
1379	/* Request timer and round. */
1380	dj = round_up(rcu_idle_gp_delay + jiffies, rcu_idle_gp_delay) - jiffies;
1381
1382	nextevt = basemono + dj TICK_NSEC;
1383	return 0;
1384	}
1385
1386	/*
1387	* Prepare a CPU for idle from an RCU perspective. The first major task is to
1388	* sense whether nohz mode has been enabled or disabled via sysfs. The second
1389	* major task is to accelerate (that is, assign grace-period numbers to) any
1390	* recently arrived callbacks.
1391	*
1392	* The caller must have disabled interrupts.
1393	*/
1394	static void rcu_prepare_for_idle(void)
1395	{
1396	bool needwake;
1397	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
1398	struct rcu_node *rnp;
1399	int tne;
1400
1401	lockdep_assert_irqs_disabled();
1402	if (rcu_rdp_is_offloaded(rdp))
1403	return;
1404
1405	/* Handle nohz enablement switches conservatively. */
1406	tne = READ_ONCE(tick_nohz_active);
1407	if (tne != rdp->tick_nohz_enabled_snap) {
1408	if (!rcu_segcblist_empty(&rdp->cblist))
1409	invoke_rcu_core(); /* force nohz to see update. */
1410	rdp->tick_nohz_enabled_snap = tne;
1411	return;
1412	}
1413	if (!tne)
1414	return;
1415
1416	/*
1417	* If we have not yet accelerated this jiffy, accelerate all
1418	* callbacks on this CPU.
1419	*/
1420	if (rdp->last_accelerate == jiffies)
1421	return;
1422	rdp->last_accelerate = jiffies;
1423	if (rcu_segcblist_pend_cbs(&rdp->cblist)) {
1424	rnp = rdp->mynode;
1425	raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */
1426	needwake = rcu_accelerate_cbs(rnp, rdp);
1427	raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
1428	if (needwake)
1429	rcu_gp_kthread_wake();
1430	}
1431	}
1432
1433	/*
1434	* Clean up for exit from idle. Attempt to advance callbacks based on
1435	* any grace periods that elapsed while the CPU was idle, and if any
1436	* callbacks are now ready to invoke, initiate invocation.
1437	*/
1438	static void rcu_cleanup_after_idle(void)
1439	{
1440	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
1441
1442	lockdep_assert_irqs_disabled();
1443	if (rcu_rdp_is_offloaded(rdp))
1444	return;
1445	if (rcu_try_advance_all_cbs())
1446	invoke_rcu_core();
1447	}
1448
1449	#endif /* #else #if !defined(CONFIG_RCU_FAST_NO_HZ) */
1450
1451	/*	1267	/*
1452	* Is this CPU a NO_HZ_FULL CPU that should ignore RCU so that the	1268	* Is this CPU a NO_HZ_FULL CPU that should ignore RCU so that the
1453	* grace-period kthread will do force_quiescent_state() processing?	1269	* grace-period kthread will do force_quiescent_state() processing?
@@ -1455,7 +1271,7 @@ static void rcu_cleanup_after_idle(void) · 1 unmodified line
1455	* CPU unless the grace period has extended for too long.	1271	* CPU unless the grace period has extended for too long.
1456	*	1272	*
1457	* This code relies on the fact that all NO_HZ_FULL CPUs are also	1273	* This code relies on the fact that all NO_HZ_FULL CPUs are also
1458	* CONFIG_RCU_NOCB_CPU CPUs.	1274	* RCU_NOCB_CPU CPUs.
1459	*/	1275	*/
1460	static bool rcu_nohz_full_cpu(void)	1276	static bool rcu_nohz_full_cpu(void)
1461	{	1277	{

kernel/rcu/tree_stall.h+2−25

 	}
 }
-#ifdef CONFIG_RCU_FAST_NO_HZ
-static void print_cpu_stall_fast_no_hz(char *cp, int cpu)
-{
-	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
-	sprintf(cp, "last_accelerate: %04lx/%04lx dyntick_enabled: %d",
-		rdp->last_accelerate & 0xffff, jiffies & 0xffff,
-		!!rdp->tick_nohz_enabled_snap);
-}
-#else /* #ifdef CONFIG_RCU_FAST_NO_HZ */
-static void print_cpu_stall_fast_no_hz(char *cp, int cpu)
-{
-	*cp = '\0';
-}
-#endif /* #else #ifdef CONFIG_RCU_FAST_NO_HZ */
 static const char * const gp_state_names[] = {
 	[RCU_GP_IDLE] = "RCU_GP_IDLE",
 	[RCU_GP_WAIT_GPS] = "RCU_GP_WAIT_GPS",
  * of RCU grace periods that this CPU is ignorant of, for example, "1"
  * if the CPU was aware of the previous grace period.
  *
- * Also print out idle and (if CONFIG_RCU_FAST_NO_HZ) idle-entry info.
+ * Also print out idle info.
  */
 static void print_cpu_stall_info(int cpu)
 {
 	unsigned long delta;
 	bool falsepositive;
-	char fast_no_hz[72];
 	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
 	char *ticks_title;
 	unsigned long ticks_value;
 		ticks_title = "ticks this GP";
 		ticks_value = rdp->ticks_this_gp;
 	}
-	print_cpu_stall_fast_no_hz(fast_no_hz, cpu);
 	delta = rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq);
 	falsepositive = rcu_is_gp_kthread_starving(NULL) &&
 			rcu_dynticks_in_eqs(rcu_dynticks_snap(rdp));
-	pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld/%#lx softirq=%u/%u fqs=%ld %s%s\n",
+	pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld/%#lx softirq=%u/%u fqs=%ld %s\n",
 	       cpu,
 	       "O."[!!cpu_online(cpu)],
 	       "o."[!!(rdp->grpmask & rdp->mynode->qsmaskinit)],
 	       rdp->dynticks_nesting, rdp->dynticks_nmi_nesting,
 	       rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu),
 	       data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart,
-	       fast_no_hz,
 	       falsepositive ? " (false positive?)" : "");
 }

kernel/scftorture.c+6−10

 #define SCFTORT_STRING "scftorture"
 #define SCFTORT_FLAG SCFTORT_STRING ": "
-#define SCFTORTOUT(s, x...) \
-	pr_alert(SCFTORT_FLAG s, ## x)
 #define VERBOSE_SCFTORTOUT(s, x...) \
-	do { if (verbose) pr_alert(SCFTORT_FLAG s, ## x); } while (0)
+	do { if (verbose) pr_alert(SCFTORT_FLAG s "\n", ## x); } while (0)
-#define VERBOSE_SCFTORTOUT_ERRSTRING(s, x...) \
+#define SCFTORTOUT_ERRSTRING(s, x...) pr_alert(SCFTORT_FLAG "!!! " s "\n", ## x)
-	do { if (verbose) pr_alert(SCFTORT_FLAG "!!! " s, ## x); } while (0)
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("Paul E. McKenney <paulmck@kernel.org>");
 	if (weight_resched1 == 0 && weight_single1 == 0 && weight_single_rpc1 == 0 &&
 	    weight_single_wait1 == 0 && weight_many1 == 0 && weight_many_wait1 == 0 &&
 	    weight_all1 == 0 && weight_all_wait1 == 0) {
-		VERBOSE_SCFTORTOUT_ERRSTRING("all zero weights makes no sense");
+		SCFTORTOUT_ERRSTRING("all zero weights makes no sense");
 		firsterr = -EINVAL;
 		goto unwind;
 	}
 	if (IS_BUILTIN(CONFIG_SCF_TORTURE_TEST))
 		scf_sel_add(weight_resched1, SCF_PRIM_RESCHED, false);
 	else if (weight_resched1)
-		VERBOSE_SCFTORTOUT_ERRSTRING("built as module, weight_resched ignored");
+		SCFTORTOUT_ERRSTRING("built as module, weight_resched ignored");
 	scf_sel_add(weight_single1, SCF_PRIM_SINGLE, false);
 	scf_sel_add(weight_single_rpc1, SCF_PRIM_SINGLE_RPC, true);
 	scf_sel_add(weight_single_wait1, SCF_PRIM_SINGLE, true);
 		nthreads = num_online_cpus();
 	scf_stats_p = kcalloc(nthreads, sizeof(scf_stats_p[0]), GFP_KERNEL);
 	if (!scf_stats_p) {
-		VERBOSE_SCFTORTOUT_ERRSTRING("out of memory");
+		SCFTORTOUT_ERRSTRING("out of memory");
 		firsterr = -ENOMEM;
 		goto unwind;
 	}
-	VERBOSE_SCFTORTOUT("Starting %d smp_call_function() threads\n", nthreads);
+	VERBOSE_SCFTORTOUT("Starting %d smp_call_function() threads", nthreads);
 	atomic_set(&n_started, nthreads);
 	for (i = 0; i < nthreads; i++) {

kernel/torture.c+2−2

@@ -570,7 +570,7 @@ int torture_shuffle_init(long shuffint) · 569 unmodified lines
570	shuffle_idle_cpu = -1;	570	shuffle_idle_cpu = -1;
571		571
572	if (!alloc_cpumask_var(&shuffle_tmp_mask, GFP_KERNEL)) {	572	if (!alloc_cpumask_var(&shuffle_tmp_mask, GFP_KERNEL)) {
573	VERBOSE_TOROUT_ERRSTRING("Failed to alloc mask");	573	TOROUT_ERRSTRING("Failed to alloc mask");
574	return -ENOMEM;	574	return -ENOMEM;
575	}	575	}
576		576
@@ -934,7 +934,7 @@ int _torture_create_kthread(int (fn)(void arg), void arg, char s, char *m, · 357 unmodified lines
934	*tp = kthread_run(fn, arg, "%s", s);	934	*tp = kthread_run(fn, arg, "%s", s);
935	if (IS_ERR(*tp)) {	935	if (IS_ERR(*tp)) {
936	ret = PTR_ERR(*tp);	936	ret = PTR_ERR(*tp);
937	VERBOSE_TOROUT_ERRSTRING(f);	937	TOROUT_ERRSTRING(f);
938	*tp = NULL;	938	*tp = NULL;
939	}	939	}
940	torture_shuffle_task_register(*tp);	940	torture_shuffle_task_register(*tp);

tools/include/nolibc/nolibc.h+60−26

  *   - arguments are in rdi, rsi, rdx, r10, r8, r9 respectively
  *   - the system call is performed by calling the syscall instruction
  *   - syscall return comes in rax
- *   - rcx and r8..r11 may be clobbered, others are preserved.
+ *   - rcx and r11 are clobbered, others are preserved.
  *   - the arguments are cast to long and assigned into the target registers
  *     which are then simply passed as registers to the asm code, so that we
  *     don't have to experience issues with register constraints.
  *   - the syscall number is always specified last in order to allow to force
  *     some registers before (gcc refuses a %-register at the last position).
+ *   - see also x86-64 ABI section A.2 AMD64 Linux Kernel Conventions, A.2.1
+ *     Calling Conventions.
+ *
+ * Link x86-64 ABI: https://gitlab.com/x86-psABIs/x86-64-ABI/-/wikis/x86-64-psABI
+ *
  */
 #define my_syscall0(num)                                                      \
 									      \
 	asm volatile (                                                        \
 		"syscall\n"                                                   \
-		: "=a" (_ret)                                                 \
+		: "=a"(_ret)                                                  \
 		: "0"(_num)                                                   \
-		: "rcx", "r8", "r9", "r10", "r11", "memory", "cc"             \
+		: "rcx", "r11", "memory", "cc"                                \
 	);                                                                    \
 	_ret;                                                                 \
 })
 									      \
 	asm volatile (                                                        \
 		"syscall\n"                                                   \
-		: "=a" (_ret)                                                 \
+		: "=a"(_ret)                                                  \
 		: "r"(_arg1),                                                 \
 		  "0"(_num)                                                   \
-		: "rcx", "r8", "r9", "r10", "r11", "memory", "cc"             \
+		: "rcx", "r11", "memory", "cc"                                \
 	);                                                                    \
 	_ret;                                                                 \
 })
 									      \
 	asm volatile (                                                        \
 		"syscall\n"                                                   \
-		: "=a" (_ret)                                                 \
+		: "=a"(_ret)                                                  \
 		: "r"(_arg1), "r"(_arg2),                                     \
 		  "0"(_num)                                                   \
-		: "rcx", "r8", "r9", "r10", "r11", "memory", "cc"             \
+		: "rcx", "r11", "memory", "cc"                                \
 	);                                                                    \
 	_ret;                                                                 \
 })
 									      \
 	asm volatile (                                                        \
 		"syscall\n"                                                   \
-		: "=a" (_ret)                                                 \
+		: "=a"(_ret)                                                  \
 		: "r"(_arg1), "r"(_arg2), "r"(_arg3),                         \
 		  "0"(_num)                                                   \
-		: "rcx", "r8", "r9", "r10", "r11", "memory", "cc"             \
+		: "rcx", "r11", "memory", "cc"                                \
 	);                                                                    \
 	_ret;                                                                 \
 })
 									      \
 	asm volatile (                                                        \
 		"syscall\n"                                                   \
-		: "=a" (_ret), "=r"(_arg4)                                    \
+		: "=a"(_ret)                                                  \
 		: "r"(_arg1), "r"(_arg2), "r"(_arg3), "r"(_arg4),             \
 		  "0"(_num)                                                   \
-		: "rcx", "r8", "r9", "r11", "memory", "cc"                    \
+		: "rcx", "r11", "memory", "cc"                                \
 	);                                                                    \
 	_ret;                                                                 \
 })
 									      \
 	asm volatile (                                                        \
 		"syscall\n"                                                   \
-		: "=a" (_ret), "=r"(_arg4), "=r"(_arg5)                       \
+		: "=a"(_ret)                                                  \
 		: "r"(_arg1), "r"(_arg2), "r"(_arg3), "r"(_arg4), "r"(_arg5), \
 		  "0"(_num)                                                   \
-		: "rcx", "r9", "r11", "memory", "cc"                          \
+		: "rcx", "r11", "memory", "cc"                                \
 	);                                                                    \
 	_ret;                                                                 \
 })
 									      \
 	asm volatile (                                                        \
 		"syscall\n"                                                   \
-		: "=a" (_ret), "=r"(_arg4), "=r"(_arg5)                       \
+		: "=a"(_ret)                                                  \
 		: "r"(_arg1), "r"(_arg2), "r"(_arg3), "r"(_arg4), "r"(_arg5), \
 		  "r"(_arg6), "0"(_num)                                       \
 		: "rcx", "r11", "memory", "cc"                                \
 })
 /* startup code */
+/*
+ * x86-64 System V ABI mandates:
+ * 1) %rsp must be 16-byte aligned right before the function call.
+ * 2) The deepest stack frame should be zero (the %rbp).
+ *
+ */
 asm(".section .text\n"
     ".global _start\n"
     "_start:\n"
     "pop %rdi\n"                // argc   (first arg, %rdi)
     "mov %rsp, %rsi\n"          // argv[] (second arg, %rsi)
     "lea 8(%rsi,%rdi,8),%rdx\n" // then a NULL then envp (third arg, %rdx)
-    "and $-16, %rsp\n"          // x86 ABI : esp must be 16-byte aligned when
+    "xor %ebp, %ebp\n"          // zero the stack frame
-    "sub $8, %rsp\n"            // entering the callee
+    "and $-16, %rsp\n"          // x86 ABI : esp must be 16-byte aligned before call
     "call main\n"               // main() returns the status code, we'll exit with it.
-    "movzb %al, %rdi\n"         // retrieve exit code from 8 lower bits
+    "mov %eax, %edi\n"          // retrieve exit code (32 bit)
-    "mov $60, %rax\n"           // NR_exit == 60
+    "mov $60, %eax\n"           // NR_exit == 60
     "syscall\n"                 // really exit
     "hlt\n"                     // ensure it does not return
     "");
 })
 /* startup code */
+/*
+ * i386 System V ABI mandates:
+ * 1) last pushed argument must be 16-byte aligned.
+ * 2) The deepest stack frame should be set to zero
+ *
+ */
 asm(".section .text\n"
     ".global _start\n"
     "_start:\n"
     "pop %eax\n"                // argc   (first arg, %eax)
     "mov %esp, %ebx\n"          // argv[] (second arg, %ebx)
     "lea 4(%ebx,%eax,4),%ecx\n" // then a NULL then envp (third arg, %ecx)
-    "and $-16, %esp\n"          // x86 ABI : esp must be 16-byte aligned when
+    "xor %ebp, %ebp\n"          // zero the stack frame
+    "and $-16, %esp\n"          // x86 ABI : esp must be 16-byte aligned before
+    "sub $4, %esp\n"            // the call instruction (args are aligned)
     "push %ecx\n"               // push all registers on the stack so that we
     "push %ebx\n"               // support both regparm and plain stack modes
     "push %eax\n"
     "call main\n"               // main() returns the status code in %eax
-    "movzbl %al, %ebx\n"        // retrieve exit code from lower 8 bits
+    "mov %eax, %ebx\n"          // retrieve exit code (32-bit int)
-    "movl   $1, %eax\n"         // NR_exit == 1
+    "movl $1, %eax\n"           // NR_exit == 1
-    "int    $0x80\n"            // exit now
+    "int $0x80\n"               // exit now
     "hlt\n"                     // ensure it does not
     "");
     "and %r3, %r1, $-8\n"         // AAPCS : sp must be 8-byte aligned in the
     "mov %sp, %r3\n"              //         callee, an bl doesn't push (lr=pc)
     "bl main\n"                   // main() returns the status code, we'll exit with it.
-    "and %r0, %r0, $0xff\n"       // limit exit code to 8 bits
     "movs r7, $1\n"               // NR_exit == 1
     "svc $0x00\n"
     "");
     "add x2, x2, x1\n"            //           + argv
     "and sp, x1, -16\n"           // sp must be 16-byte aligned in the callee
     "bl main\n"                   // main() returns the status code, we'll exit with it.
-    "and x0, x0, 0xff\n"          // limit exit code to 8 bits
     "mov x8, 93\n"                // NR_exit == 93
     "svc #0\n"
     "");
     "addiu $sp,$sp,-16\n"         // the callee expects to save a0..a3 there!
     "jal main\n"                  // main() returns the status code, we'll exit with it.
     "nop\n"                       // delayed slot
-    "and $a0, $v0, 0xff\n"        // limit exit code to 8 bits
+    "move $a0, $v0\n"             // retrieve 32-bit exit code from v0
     "li $v0, 4001\n"              // NR_exit == 4001
     "syscall\n"
     ".end __start\n"
     "add   a2,a2,a1\n"           //             + argv
     "andi  sp,a1,-16\n"          // sp must be 16-byte aligned
     "call  main\n"               // main() returns the status code, we'll exit with it.
-    "andi  a0, a0, 0xff\n"       // limit exit code to 8 bits
     "li a7, 93\n"                // NR_exit == 93
     "ecall\n"
     "");
 	return my_syscall0(__NR_getpid);
 }
+static __attribute__((unused))
+pid_t sys_gettid(void)
+{
+	return my_syscall0(__NR_gettid);
+}
 static __attribute__((unused))
 int sys_gettimeofday(struct timeval *tv, struct timezone *tz)
 {
 	return ret;
 }
+static __attribute__((unused))
+pid_t gettid(void)
+{
+	pid_t ret = sys_gettid();
+	if (ret < 0) {
+		SET_ERRNO(-ret);
+		ret = -1;
+	}
+	return ret;
+}
 static __attribute__((unused))
 int gettimeofday(struct timeval *tv, struct timezone *tz)
 {

tools/testing/selftests/rcutorture/bin/kvm-find-errors.sh+2−2

@@ -30,9 +30,9 @@ editor=${EDITOR-vi} · 29 unmodified lines
30	files=	30	files=
31	for i in ${rundir}/*/Make.out	31	for i in ${rundir}/*/Make.out
32	do	32	do
33	if egrep -q "error:\|warning:" < $i	33	if egrep -q "error:\|warning:\|^ld: .*undefined reference to" < $i
34	then	34	then
35	egrep "error:\|warning:" < $i > $i.diags	35	egrep "error:\|warning:\|^ld: .*undefined reference to" < $i > $i.diags
36	files="$files $i.diags $i"	36	files="$files $i.diags $i"
37	fi	37	fi
38	done	38	done

tools/testing/selftests/rcutorture/bin/kvm-recheck-rcu.sh+1−1

@@ -25,7 +25,7 @@ stopstate="`grep 'End-test grace-period state: g' $i/console.log 2> /dev/null \| · 24 unmodified lines
25	tail -1 \| sed -e 's/^\[[ 0-9.]*] //' \|	25	tail -1 \| sed -e 's/^\[[ 0-9.]*] //' \|
26	awk '{ print \"[\" $1 \" \" $5 \" \" $6 \" \" $7 \"]\"; }' \|	26	awk '{ print \"[\" $1 \" \" $5 \" \" $6 \" \" $7 \"]\"; }' \|
27	tr -d '\012\015'`"	27	tr -d '\012\015'`"
28	fwdprog="`grep 'rcu_torture_fwd_prog_cr Duration' $i/console.log 2> /dev/null \| sed -e 's/^\[[^]]*] //' \| sort -k15nr \| head -1 \| awk '{ print $14 " " $15 }'`"	28	fwdprog="`grep 'rcu_torture_fwd_prog n_max_cbs: ' $i/console.log 2> /dev/null \| sed -e 's/^\[[^]]*] //' \| sort -k3nr \| head -1 \| awk '{ print $2 " " $3 }'`"
29	if test -z "$ngps"	29	if test -z "$ngps"
30	then	30	then
31	echo "$configfile ------- " $stopstate	31	echo "$configfile ------- " $stopstate

tools/testing/selftests/rcutorture/bin/kvm-remote.sh+15−8

 	if test "$ret" -ne 0
 	then
 		echo System $i unreachable, giving up. | tee -a "$oldrun/remote-log"
-		exit 4 | tee -a "$oldrun/remote-log"
+		exit 4
 	fi
 done
 	ret=$?
 	if test "$ret" -ne 0
 	then
-		echo Unable to download $T/binres.tgz to system $i, giving up. | tee -a "$oldrun/remote-log"
+		echo Unable to download $T/binres.tgz to system $i, waiting and then retrying. | tee -a "$oldrun/remote-log"
-		exit 10 | tee -a "$oldrun/remote-log"
+		sleep 60
+		cat $T/binres.tgz | ssh $i "cd /tmp; tar -xzf -"
+		ret=$?
+		if test "$ret" -ne 0
+		then
+			echo Unable to download $T/binres.tgz to system $i, giving up. | tee -a "$oldrun/remote-log"
+			exit 10
+		fi
 	fi
 done
 		ret=$?
 		if test "$ret" -eq 255
 		then
-			echo " ---" ssh failure to $1 checking for file $2, retry after $sleeptime seconds. `date`
+			echo " ---" ssh failure to $1 checking for file $2, retry after $sleeptime seconds. `date` | tee -a "$oldrun/remote-log"
 		elif test "$ret" -eq 0
 		then
 			return 0
 		elif test "$ret" -eq 1
 		then
-			echo " ---" File \"$2\" not found: ssh $1 test -f \"$2\"
+			echo " ---" File \"$2\" not found: ssh $1 test -f \"$2\" | tee -a "$oldrun/remote-log"
 			return 1
 		else
-			echo " ---" Exit code $ret: ssh $1 test -f \"$2\", retry after $sleeptime seconds. `date`
+			echo " ---" Exit code $ret: ssh $1 test -f \"$2\", retry after $sleeptime seconds. `date` | tee -a "$oldrun/remote-log"
 			return $ret
 		fi
 		sleep $sleeptime
 		sleep 30
 	fi
 done
-echo All batches started. `date`
+echo All batches started. `date` | tee -a "$oldrun/remote-log"
 # Wait for all remaining scenarios to complete and collect results.
 for i in $systems
 	do
 		sleep 30
 	done
-	echo " ---" Collecting results from $i `date`
+	echo " ---" Collecting results from $i `date` | tee -a "$oldrun/remote-log"
 	( cd "$oldrun"; ssh $i "cd $rundir; tar -czf - kvm-remote-*.sh.out */console.log */kvm-test-1-run*.sh.out */qemu[_-]pid */qemu-retval */qemu-affinity; rm -rf $T > /dev/null 2>&1" | tar -xzf - )
 done

tools/testing/selftests/rcutorture/bin/kvm.sh+6−3

 	echo "       --help"
 	echo "       --interactive"
 	echo "       --jitter N [ maxsleep (us) [ maxspin (us) ] ]"
+	echo "       --kasan"
 	echo "       --kconfig Kconfig-options"
+	echo "       --kcsan"
 	echo "       --kmake-arg kernel-make-arguments"
 	echo "       --mac nn:nn:nn:nn:nn:nn"
 	echo "       --memory megabytes|nnnG"
 	echo "       --qemu-cmd qemu-system-..."
 	echo "       --remote"
 	echo "       --results absolute-pathname"
+	echo "       --shutdown-grace seconds"
 	echo "       --torture lock|rcu|rcuscale|refscale|scf"
 	echo "       --trust-make"
 	exit 1
 		jitter="$2"
 		shift
 		;;
+	--kasan)
+		TORTURE_KCONFIG_KASAN_ARG="CONFIG_DEBUG_INFO=y CONFIG_KASAN=y"; export TORTURE_KCONFIG_KASAN_ARG
+		;;
 	--kconfig|--kconfigs)
 		checkarg --kconfig "(Kconfig options)" $# "$2" '^CONFIG_[A-Z0-9_]\+=\([ynm]\|[0-9]\+\)\( CONFIG_[A-Z0-9_]\+=\([ynm]\|[0-9]\+\)\)*$' '^error$'
 		TORTURE_KCONFIG_ARG="`echo "$TORTURE_KCONFIG_ARG $2" | sed -e 's/^ *//' -e 's/ *$//'`"
 		shift
 		;;
-	--kasan)
-		TORTURE_KCONFIG_KASAN_ARG="CONFIG_DEBUG_INFO=y CONFIG_KASAN=y"; export TORTURE_KCONFIG_KASAN_ARG
-		;;
 	--kcsan)
 		TORTURE_KCONFIG_KCSAN_ARG="CONFIG_DEBUG_INFO=y CONFIG_KCSAN=y CONFIG_KCSAN_STRICT=y CONFIG_KCSAN_REPORT_ONCE_IN_MS=100000 CONFIG_KCSAN_VERBOSE=y CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y"; export TORTURE_KCONFIG_KCSAN_ARG
 		;;

tools/testing/selftests/rcutorture/bin/parse-build.sh+2−1

 grep warning: < $F > $T/warnings
 grep "include/linux/*rcu*\.h:" $T/warnings > $T/hwarnings
 grep "kernel/rcu/[^/]*:" $T/warnings > $T/cwarnings
-cat $T/hwarnings $T/cwarnings > $T/rcuwarnings
+grep "^ld: .*undefined reference to" $T/warnings | head -1 > $T/ldwarnings
+cat $T/hwarnings $T/cwarnings $T/ldwarnings > $T/rcuwarnings
 if test -s $T/rcuwarnings
 then
 	print_warning $title build errors:

tools/testing/selftests/rcutorture/configs/rcu/TASKS01.boot+1−0

@@ -1,2 +1,3 @@
1	rcutorture.torture_type=tasks	1	rcutorture.torture_type=tasks
2	rcutree.use_softirq=0	2	rcutree.use_softirq=0
		3	rcupdate.rcu_task_enqueue_lim=4

tools/testing/selftests/rcutorture/configs/rcu/TRACE01.boot+1−0

@@ -1 +1,2 @@
1	rcutorture.torture_type=tasks-tracing	1	rcutorture.torture_type=tasks-tracing
		2	rcupdate.rcu_task_enqueue_lim=2

tools/testing/selftests/rcutorture/configs/rcu/TRACE02.boot+1−0

@@ -1 +1,2 @@
1	rcutorture.torture_type=tasks-tracing	1	rcutorture.torture_type=tasks-tracing
		2	rcutorture.fwd_progress=2

tools/testing/selftests/rcutorture/configs/rcu/TREE01+0−1

@@ -6,7 +6,6 @@ CONFIG_PREEMPT=y · 5 unmodified lines
6	CONFIG_HZ_PERIODIC=n	6	CONFIG_HZ_PERIODIC=n
7	CONFIG_NO_HZ_IDLE=y	7	CONFIG_NO_HZ_IDLE=y
8	CONFIG_NO_HZ_FULL=n	8	CONFIG_NO_HZ_FULL=n
9	CONFIG_RCU_FAST_NO_HZ=y
10	CONFIG_RCU_TRACE=y	9	CONFIG_RCU_TRACE=y
11	CONFIG_HOTPLUG_CPU=y	10	CONFIG_HOTPLUG_CPU=y
12	CONFIG_MAXSMP=y	11	CONFIG_MAXSMP=y

tools/testing/selftests/rcutorture/configs/rcu/TREE02+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT=y · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=y	8	CONFIG_NO_HZ_IDLE=y
9	CONFIG_NO_HZ_FULL=n	9	CONFIG_NO_HZ_FULL=n
10	CONFIG_RCU_FAST_NO_HZ=n
11	CONFIG_RCU_TRACE=n	10	CONFIG_RCU_TRACE=n
12	CONFIG_RCU_FANOUT=3	11	CONFIG_RCU_FANOUT=3
13	CONFIG_RCU_FANOUT_LEAF=3	12	CONFIG_RCU_FANOUT_LEAF=3

tools/testing/selftests/rcutorture/configs/rcu/TREE02.boot+1−0

@@ -0,0 +1 @@
		1	rcutorture.fwd_progress=2

tools/testing/selftests/rcutorture/configs/rcu/TREE04+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT=n · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=n	8	CONFIG_NO_HZ_IDLE=n
9	CONFIG_NO_HZ_FULL=y	9	CONFIG_NO_HZ_FULL=y
10	CONFIG_RCU_FAST_NO_HZ=y
11	CONFIG_RCU_TRACE=y	10	CONFIG_RCU_TRACE=y
12	CONFIG_RCU_FANOUT=4	11	CONFIG_RCU_FANOUT=4
13	CONFIG_RCU_FANOUT_LEAF=3	12	CONFIG_RCU_FANOUT_LEAF=3

tools/testing/selftests/rcutorture/configs/rcu/TREE05+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT=n · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=y	8	CONFIG_NO_HZ_IDLE=y
9	CONFIG_NO_HZ_FULL=n	9	CONFIG_NO_HZ_FULL=n
10	CONFIG_RCU_FAST_NO_HZ=n
11	CONFIG_RCU_TRACE=n	10	CONFIG_RCU_TRACE=n
12	CONFIG_HOTPLUG_CPU=y	11	CONFIG_HOTPLUG_CPU=y
13	CONFIG_RCU_FANOUT=6	12	CONFIG_RCU_FANOUT=6

tools/testing/selftests/rcutorture/configs/rcu/TREE06+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT=n · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=y	8	CONFIG_NO_HZ_IDLE=y
9	CONFIG_NO_HZ_FULL=n	9	CONFIG_NO_HZ_FULL=n
10	CONFIG_RCU_FAST_NO_HZ=n
11	CONFIG_RCU_TRACE=n	10	CONFIG_RCU_TRACE=n
12	CONFIG_RCU_FANOUT=6	11	CONFIG_RCU_FANOUT=6
13	CONFIG_RCU_FANOUT_LEAF=6	12	CONFIG_RCU_FANOUT_LEAF=6

tools/testing/selftests/rcutorture/configs/rcu/TREE07+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT=n · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=n	8	CONFIG_NO_HZ_IDLE=n
9	CONFIG_NO_HZ_FULL=y	9	CONFIG_NO_HZ_FULL=y
10	CONFIG_RCU_FAST_NO_HZ=n
11	CONFIG_RCU_TRACE=y	10	CONFIG_RCU_TRACE=y
12	CONFIG_HOTPLUG_CPU=y	11	CONFIG_HOTPLUG_CPU=y
13	CONFIG_RCU_FANOUT=2	12	CONFIG_RCU_FANOUT=2

tools/testing/selftests/rcutorture/configs/rcu/TREE08+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT=y · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=y	8	CONFIG_NO_HZ_IDLE=y
9	CONFIG_NO_HZ_FULL=n	9	CONFIG_NO_HZ_FULL=n
10	CONFIG_RCU_FAST_NO_HZ=n
11	CONFIG_RCU_TRACE=n	10	CONFIG_RCU_TRACE=n
12	CONFIG_RCU_FANOUT=3	11	CONFIG_RCU_FANOUT=3
13	CONFIG_RCU_FANOUT_LEAF=2	12	CONFIG_RCU_FANOUT_LEAF=2

tools/testing/selftests/rcutorture/configs/rcu/TREE10+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT=n · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=y	8	CONFIG_NO_HZ_IDLE=y
9	CONFIG_NO_HZ_FULL=n	9	CONFIG_NO_HZ_FULL=n
10	CONFIG_RCU_FAST_NO_HZ=n
11	CONFIG_RCU_TRACE=n	10	CONFIG_RCU_TRACE=n
12	CONFIG_RCU_NOCB_CPU=n	11	CONFIG_RCU_NOCB_CPU=n
13	CONFIG_DEBUG_LOCK_ALLOC=n	12	CONFIG_DEBUG_LOCK_ALLOC=n

tools/testing/selftests/rcutorture/configs/rcu/TREE10.boot+1−0

@@ -0,0 +1 @@
		1	rcutorture.fwd_progress=2

tools/testing/selftests/rcutorture/configs/rcuscale/TINY+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT_DYNAMIC=n · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=y	8	CONFIG_NO_HZ_IDLE=y
9	CONFIG_NO_HZ_FULL=n	9	CONFIG_NO_HZ_FULL=n
10	CONFIG_RCU_FAST_NO_HZ=n
11	CONFIG_RCU_NOCB_CPU=n	10	CONFIG_RCU_NOCB_CPU=n
12	CONFIG_DEBUG_LOCK_ALLOC=n	11	CONFIG_DEBUG_LOCK_ALLOC=n
13	CONFIG_PROVE_LOCKING=n	12	CONFIG_PROVE_LOCKING=n

tools/testing/selftests/rcutorture/configs/rcuscale/TRACE01+0−1

@@ -5,7 +5,6 @@ CONFIG_PREEMPT=n · 4 unmodified lines
5	CONFIG_HZ_PERIODIC=n	5	CONFIG_HZ_PERIODIC=n
6	CONFIG_NO_HZ_IDLE=y	6	CONFIG_NO_HZ_IDLE=y
7	CONFIG_NO_HZ_FULL=n	7	CONFIG_NO_HZ_FULL=n
8	CONFIG_RCU_FAST_NO_HZ=n
9	CONFIG_RCU_NOCB_CPU=n	8	CONFIG_RCU_NOCB_CPU=n
10	CONFIG_DEBUG_LOCK_ALLOC=n	9	CONFIG_DEBUG_LOCK_ALLOC=n
11	CONFIG_PROVE_LOCKING=n	10	CONFIG_PROVE_LOCKING=n

tools/testing/selftests/rcutorture/configs/rcuscale/TREE+0−1

@@ -6,7 +6,6 @@ CONFIG_PREEMPT=y · 5 unmodified lines
6	CONFIG_HZ_PERIODIC=n	6	CONFIG_HZ_PERIODIC=n
7	CONFIG_NO_HZ_IDLE=y	7	CONFIG_NO_HZ_IDLE=y
8	CONFIG_NO_HZ_FULL=n	8	CONFIG_NO_HZ_FULL=n
9	CONFIG_RCU_FAST_NO_HZ=n
10	CONFIG_HOTPLUG_CPU=y	9	CONFIG_HOTPLUG_CPU=y
11	CONFIG_SUSPEND=n	10	CONFIG_SUSPEND=n
12	CONFIG_HIBERNATION=n	11	CONFIG_HIBERNATION=n

tools/testing/selftests/rcutorture/configs/rcuscale/TREE54+0−1

@@ -7,7 +7,6 @@ CONFIG_PREEMPT=y · 6 unmodified lines
7	CONFIG_HZ_PERIODIC=n	7	CONFIG_HZ_PERIODIC=n
8	CONFIG_NO_HZ_IDLE=y	8	CONFIG_NO_HZ_IDLE=y
9	CONFIG_NO_HZ_FULL=n	9	CONFIG_NO_HZ_FULL=n
10	CONFIG_RCU_FAST_NO_HZ=n
11	CONFIG_HOTPLUG_CPU=y	10	CONFIG_HOTPLUG_CPU=y
12	CONFIG_SUSPEND=n	11	CONFIG_SUSPEND=n
13	CONFIG_HIBERNATION=n	12	CONFIG_HIBERNATION=n

tools/testing/selftests/rcutorture/configs/refscale/NOPREEMPT+0−1

@@ -6,7 +6,6 @@ CONFIG_PREEMPT=n · 5 unmodified lines
6	CONFIG_HZ_PERIODIC=n	6	CONFIG_HZ_PERIODIC=n
7	CONFIG_NO_HZ_IDLE=y	7	CONFIG_NO_HZ_IDLE=y
8	CONFIG_NO_HZ_FULL=n	8	CONFIG_NO_HZ_FULL=n
9	CONFIG_RCU_FAST_NO_HZ=n
10	CONFIG_HOTPLUG_CPU=y	9	CONFIG_HOTPLUG_CPU=y
11	CONFIG_SUSPEND=n	10	CONFIG_SUSPEND=n
12	CONFIG_HIBERNATION=n	11	CONFIG_HIBERNATION=n

tools/testing/selftests/rcutorture/configs/refscale/PREEMPT+0−1

@@ -6,7 +6,6 @@ CONFIG_PREEMPT=y · 5 unmodified lines
6	CONFIG_HZ_PERIODIC=n	6	CONFIG_HZ_PERIODIC=n
7	CONFIG_NO_HZ_IDLE=y	7	CONFIG_NO_HZ_IDLE=y
8	CONFIG_NO_HZ_FULL=n	8	CONFIG_NO_HZ_FULL=n
9	CONFIG_RCU_FAST_NO_HZ=n
10	CONFIG_HOTPLUG_CPU=y	9	CONFIG_HOTPLUG_CPU=y
11	CONFIG_SUSPEND=n	10	CONFIG_SUSPEND=n
12	CONFIG_HIBERNATION=n	11	CONFIG_HIBERNATION=n

tools/testing/selftests/rcutorture/doc/TREE_RCU-kconfig.txt+0−1

@@ -15,7 +15,6 @@ CONFIG_PROVE_RCU -- Hardwired to CONFIG_PROVE_LOCKING. · 14 unmodified lines
15	CONFIG_RCU_BOOST -- one of PREEMPT_RCU.	15	CONFIG_RCU_BOOST -- one of PREEMPT_RCU.
16	CONFIG_RCU_FANOUT -- Cover hierarchy, but overlap with others.	16	CONFIG_RCU_FANOUT -- Cover hierarchy, but overlap with others.
17	CONFIG_RCU_FANOUT_LEAF -- Do one non-default.	17	CONFIG_RCU_FANOUT_LEAF -- Do one non-default.
18	CONFIG_RCU_FAST_NO_HZ -- Do one, but not with all nohz_full CPUs.
19	CONFIG_RCU_NOCB_CPU -- Do three, one with no rcu_nocbs CPUs, one with	18	CONFIG_RCU_NOCB_CPU -- Do three, one with no rcu_nocbs CPUs, one with
20	rcu_nocbs=0, and one with all rcu_nocbs CPUs.	19	rcu_nocbs=0, and one with all rcu_nocbs CPUs.
21	CONFIG_RCU_TRACE -- Do half.	20	CONFIG_RCU_TRACE -- Do half.