Merge tag 'rcu.release.v6.17' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux

Pull RCU updates from Neeraj Upadhyay: "Expedited grace period updates: - Protect against early RCU exp quiescent state reporting during exp grace period initialization - Remove superfluous barrier in task unblock path - Remove the CPU online quiescent state report optimization, which is error prone for certain scenarios - Add warning for unexpected pending requested expedited quiescent state on dying CPU Core: - Robustify rcu_is_cpu_rrupt_from_idle() by using more accurate indicators of the actual context tracking state of a CPU - Handle ->defer_qs_iw_pending field data race - Enable rcu_normal_wake_from_gp by default on systems with <= 16 CPUs - Fix lockup in rcu_read_unlock() due to recursive irq_exit() calls - Refactor expedited handling condition in rcu_read_unlock_special() - Documentation updates for hotplug and GP init scan ordering, separation of rcu_state and rnp's gp_seq states, quiescent state reporting for offline CPUs torture-scripts: - Cleanup and improve scripts : remove superfluous warnings for disabled tests; better handling of kvm.sh --kconfig arg; suppress some confusing diagnostics; tolerate bad kvm.sh args; add new diagnostic for build output; fail allmodconfig testing on warnings - Include RCU_TORTURE_TEST_CHK_RDR_STATE config for KCSAN kernels - Disable default RCU-tasks and clocksource-wdog testing on arm64 - Add EXPERT Kconfig option for arm64 KCSAN runs - Remove SRCU-lite testing rcutorture: - Start torture writer threads creation after reader threads to handle race in SRCU-P scenario - Add SRCU down_read()/up_read() test - Add diagnostics for delayed SRCU up_read(), unmatched up_read(), print number of up/down readers and the number of such readers which migrated to other CPU - Ignore certain unsupported configurations for trivial RCU test - Fix splats in RT kernels due to inaccurate checks for BH-disabled context - Enable checks and logs to capture intentionally exercised unexpected scenarios (too short readers) for BUSTED test - Remove SRCU-lite testing srcu: - Expedite SRCU-fast grace periods - Remove SRCU-lite implementation - Add guards for SRCU-fast readers rcu nocb: - Dump NOCB group leader state on stall detection - Robustify nocb_cb_kthread pointer accesses - Fix delayed execution of hurry callbacks when LAZY_RCU is enabled refscale: - Fix multiplication overflow in "loops" and "nreaders" calculations" * tag 'rcu.release.v6.17' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux: (49 commits) rcu: Document concurrent quiescent state reporting for offline CPUs rcu: Document separation of rcu_state and rnp's gp_seq rcu: Document GP init vs hotplug-scan ordering requirements srcu: Add guards for SRCU-fast readers rcu: Fix delayed execution of hurry callbacks rcu: Refactor expedited handling check in rcu_read_unlock_special() checkpatch: Remove SRCU-lite deprecation srcu: Remove SRCU-lite implementation srcu: Expedite SRCU-fast grace periods rcutorture: Remove support for SRCU-lite rcutorture: Remove SRCU-lite scenarios torture: Remove support for SRCU-lite torture: Make torture.sh --allmodconfig testing fail on warnings torture: Add "ERROR" diagnostic for testing kernel-build output torture: Make torture.sh tolerate runs having bad kvm.sh arguments torture: Add textid.txt file to --do-allmodconfig and --do-rcu-rust runs torture: Extract testid.txt generation to separate script torture: Suppress "find" diagnostics from torture.sh --do-none run torture: Provide EXPERT Kconfig option for arm64 KCSAN torture.sh runs rcu: Fix rcu_read_unlock() deadloop due to IRQ work ...

Linus Torvalds committed 1 year ago

Commit 2db4df02 parents 7dff275 cc1d13624 files · +770 −319Browse files

Unified · Split

24 files

Documentation/RCU/Design/Data-Structures/Data-Structures.rst +33 −0
Documentation/RCU/Design/Requirements/Requirements.rst +128 −0
Documentation/admin-guide/kernel-parameters.txt +2 −1
include/linux/srcu.h +8 −46
include/linux/srcutiny.h +0 −3
include/linux/srcutree.h +0 −38
kernel/rcu/rcutorture.c +298 −58
kernel/rcu/refscale.c +8 −34
kernel/rcu/srcutree.c +2 −0
kernel/rcu/tree.c +64 −16
kernel/rcu/tree.h +12 −1
kernel/rcu/tree_exp.h +9 −50
kernel/rcu/tree_nocb.h +6 −4
kernel/rcu/tree_plugin.h +102 −20
kernel/rcu/tree_stall.h +1 −2
scripts/checkpatch.pl +0 −2
tools/testing/selftests/rcutorture/bin/kvm-build.sh +1 −1
tools/testing/selftests/rcutorture/bin/kvm.sh +2 −13
tools/testing/selftests/rcutorture/bin/mktestid.sh +29 −0
tools/testing/selftests/rcutorture/bin/torture.sh +62 −16
tools/testing/selftests/rcutorture/configs/rcu/BUSTED +3 −0
tools/testing/selftests/rcutorture/configs/rcu/CFLIST +0 −1
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L +0 −10
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot +0 −3

Documentation/RCU/Design/Data-Structures/Data-Structures.rst+33−0

@@ -286,6 +286,39 @@ in order to detect the beginnings and ends of grace periods in a · 285 unmodified lines
286	distributed fashion. The values flow from ``rcu_state`` to ``rcu_node``	286	distributed fashion. The values flow from ``rcu_state`` to ``rcu_node``
287	(down the tree from the root to the leaves) to ``rcu_data``.	287	(down the tree from the root to the leaves) to ``rcu_data``.
288		288
		289	+-----------------------------------------------------------------------+
		290	\| Quick Quiz: \|
		291	+-----------------------------------------------------------------------+
		292	\| Given that the root rcu_node structure has a gp_seq field, \|
		293	\| why does RCU maintain a separate gp_seq in the rcu_state structure? \|
		294	\| Why not just use the root rcu_node's gp_seq as the official record \|
		295	\| and update it directly when starting a new grace period? \|
		296	+-----------------------------------------------------------------------+
		297	\| Answer: \|
		298	+-----------------------------------------------------------------------+
		299	\| On single-node RCU trees (where the root node is also a leaf), \|
		300	\| updating the root node's gp_seq immediately would create unnecessary \|
		301	\| lock contention. Here's why: \|
		302	\| \|
		303	\| If we did rcu_seq_start() directly on the root node's gp_seq: \|
		304	\| \|
		305	\| 1. All CPUs would immediately see their node's gp_seq from their rdp's\|
		306	\| gp_seq, in rcu_pending(). They would all then invoke the RCU-core. \|
		307	\| 2. Which calls note_gp_changes() and try to acquire the node lock. \|
		308	\| 3. But rnp->qsmask isn't initialized yet (happens later in \|
		309	\| rcu_gp_init()) \|
		310	\| 4. So each CPU would acquire the lock, find it can't determine if it \|
		311	\| needs to report quiescent state (no qsmask), update rdp->gp_seq, \|
		312	\| and release the lock. \|
		313	\| 5. Result: Lots of lock acquisitions with no grace period progress \|
		314	\| \|
		315	\| By having a separate rcu_state.gp_seq, we can increment the official \|
		316	\| grace period counter without immediately affecting what CPUs see in \|
		317	\| their nodes. The hierarchical propagation in rcu_gp_init() then \|
		318	\| updates the root node's gp_seq and qsmask together under the same lock\|
		319	\| acquisition, avoiding this useless contention. \|
		320	+-----------------------------------------------------------------------+
		321
289	Miscellaneous	322	Miscellaneous
290	'''''''''''''	323	'''''''''''''
291		324

Documentation/RCU/Design/Requirements/Requirements.rst+128−0

@@ -1970,6 +1970,134 @@ corresponding CPU's leaf node lock is held. This avoids race conditions · 1969 unmodified lines
1970	between RCU's hotplug notifier hooks, the grace period initialization	1970	between RCU's hotplug notifier hooks, the grace period initialization
1971	code, and the FQS loop, all of which refer to or modify this bookkeeping.	1971	code, and the FQS loop, all of which refer to or modify this bookkeeping.
1972		1972
		1973	Note that grace period initialization (rcu_gp_init()) must carefully sequence
		1974	CPU hotplug scanning with grace period state changes. For example, the
		1975	following race could occur in rcu_gp_init() if rcu_seq_start() were to happen
		1976	after the CPU hotplug scanning.
		1977
		1978	.. code-block:: none
		1979
		1980	CPU0 (rcu_gp_init) CPU1 CPU2
		1981	--------------------- ---- ----
		1982	// Hotplug scan first (WRONG ORDER)
		1983	rcu_for_each_leaf_node(rnp) {
		1984	rnp->qsmaskinit = rnp->qsmaskinitnext;
		1985	}
		1986	rcutree_report_cpu_starting()
		1987	rnp->qsmaskinitnext \|= mask;
		1988	rcu_read_lock()
		1989	r0 = *X;
		1990	r1 = *X;
		1991	X = NULL;
		1992	cookie = get_state_synchronize_rcu();
		1993	// cookie = 8 (future GP)
		1994	rcu_seq_start(&rcu_state.gp_seq);
		1995	// gp_seq = 5
		1996
		1997	// CPU1 now invisible to this GP!
		1998	rcu_for_each_node_breadth_first() {
		1999	rnp->qsmask = rnp->qsmaskinit;
		2000	// CPU1 not included!
		2001	}
		2002
		2003	// GP completes without CPU1
		2004	rcu_seq_end(&rcu_state.gp_seq);
		2005	// gp_seq = 8
		2006	poll_state_synchronize_rcu(cookie);
		2007	// Returns true!
		2008	kfree(r1);
		2009	r2 = *r0; // USE-AFTER-FREE!
		2010
		2011	By incrementing gp_seq first, CPU1's RCU read-side critical section
		2012	is guaranteed to not be missed by CPU2.
		2013
		2014	Concurrent Quiescent State Reporting for Offline CPUs
		2015
		2016	RCU must ensure that CPUs going offline report quiescent states to avoid
		2017	blocking grace periods. This requires careful synchronization to handle
		2018	race conditions
		2019
		2020	Race condition causing Offline CPU to hang GP
		2021
		2022	A race between CPU offlining and new GP initialization (gp_init) may occur
		2023	because `rcu_report_qs_rnp()` in `rcutree_report_cpu_dead()` must temporarily
		2024	release the `rcu_node` lock to wake the RCU grace-period kthread:
		2025
		2026	.. code-block:: none
		2027
		2028	CPU1 (going offline) CPU0 (GP kthread)
		2029	-------------------- -----------------
		2030	rcutree_report_cpu_dead()
		2031	rcu_report_qs_rnp()
		2032	// Must release rnp->lock to wake GP kthread
		2033	raw_spin_unlock_irqrestore_rcu_node()
		2034	// Wakes up and starts new GP
		2035	rcu_gp_init()
		2036	// First loop:
		2037	copies qsmaskinitnext->qsmaskinit
		2038	// CPU1 still in qsmaskinitnext!
		2039
		2040	// Second loop:
		2041	rnp->qsmask = rnp->qsmaskinit
		2042	mask = rnp->qsmask & ~rnp->qsmaskinitnext
		2043	// mask is 0! CPU1 still in both masks
		2044	// Reacquire lock (but too late)
		2045	rnp->qsmaskinitnext &= ~mask // Finally clears bit
		2046
		2047	Without `ofl_lock`, the new grace period includes the offline CPU and waits
		2048	forever for its quiescent state causing a GP hang.
		2049
		2050	A solution with ofl_lock
		2051
		2052	The `ofl_lock` (offline lock) prevents `rcu_gp_init()` from running during
		2053	the vulnerable window when `rcu_report_qs_rnp()` has released `rnp->lock`:
		2054
		2055	.. code-block:: none
		2056
		2057	CPU0 (rcu_gp_init) CPU1 (rcutree_report_cpu_dead)
		2058	------------------ ------------------------------
		2059	rcu_for_each_leaf_node(rnp) {
		2060	arch_spin_lock(&ofl_lock) -----> arch_spin_lock(&ofl_lock) [BLOCKED]
		2061
		2062	// Safe: CPU1 can't interfere
		2063	rnp->qsmaskinit = rnp->qsmaskinitnext
		2064
		2065	arch_spin_unlock(&ofl_lock) ---> // Now CPU1 can proceed
		2066	} // But snapshot already taken
		2067
		2068	Another race causing GP hangs in rcu_gpu_init(): Reporting QS for Now-offline CPUs
		2069
		2070	After the first loop takes an atomic snapshot of online CPUs, as shown above,
		2071	the second loop in `rcu_gp_init()` detects CPUs that went offline between
		2072	releasing `ofl_lock` and acquiring the per-node `rnp->lock`. This detection is
		2073	crucial because:
		2074
		2075	1. The CPU might have gone offline after the snapshot but before the second loop
		2076	2. The offline CPU cannot report its own QS if it's already dead
		2077	3. Without this detection, the grace period would wait forever for CPUs that
		2078	are now offline.
		2079
		2080	The second loop performs this detection safely:
		2081
		2082	.. code-block:: none
		2083
		2084	rcu_for_each_node_breadth_first(rnp) {
		2085	raw_spin_lock_irqsave_rcu_node(rnp, flags);
		2086	rnp->qsmask = rnp->qsmaskinit; // Apply the snapshot
		2087
		2088	// Detect CPUs offline after snapshot
		2089	mask = rnp->qsmask & ~rnp->qsmaskinitnext;
		2090
		2091	if (mask && rcu_is_leaf_node(rnp))
		2092	rcu_report_qs_rnp(mask, ...) // Report QS for offline CPUs
		2093	}
		2094
		2095	This approach ensures atomicity: quiescent state reporting for offline CPUs
		2096	happens either in `rcu_gp_init()` (second loop) or in `rcutree_report_cpu_dead()`,
		2097	never both and never neither. The `rnp->lock` held throughout the sequence
		2098	prevents races - `rcutree_report_cpu_dead()` also acquires this lock when
		2099	clearing `qsmaskinitnext`, ensuring mutual exclusion.
		2100
1973	Scheduler and RCU	2101	Scheduler and RCU
1974	~~~~~~~~~~~~~~~~~	2102	~~~~~~~~~~~~~~~~~
1975		2103

Documentation/admin-guide/kernel-parameters.txt+2−1

@@ -5508,7 +5508,8 @@ · 5507 unmodified lines
5508	echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp	5508	echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp
5509	or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1"	5509	or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1"
5510		5510
5511	Default is 0.	5511	Default is 1 if num_possible_cpus() <= 16 and it is not explicitly
		5512	disabled by the boot parameter passing 0.
5512		5513
5513	rcuscale.gp_async= [KNL]	5514	rcuscale.gp_async= [KNL]
5514	Measure performance of asynchronous	5515	Measure performance of asynchronous

include/linux/srcu.h+8−46

@@ -46,11 +46,11 @@ int init_srcu_struct(struct srcu_struct *ssp); · 45 unmodified lines
46	/* Values for SRCU Tree srcu_data ->srcu_reader_flavor, but also used by rcutorture. */	46	/* Values for SRCU Tree srcu_data ->srcu_reader_flavor, but also used by rcutorture. */
47	#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock().	47	#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock().
48	#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe().	48	#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe().
49	#define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite().	49	// 0x4 // SRCU-lite is no longer with us.
50	#define SRCU_READ_FLAVOR_FAST 0x8 // srcu_read_lock_fast().	50	#define SRCU_READ_FLAVOR_FAST 0x8 // srcu_read_lock_fast().
51	#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL \| SRCU_READ_FLAVOR_NMI \| \	51	#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL \| SRCU_READ_FLAVOR_NMI \| \
52	SRCU_READ_FLAVOR_LITE \| SRCU_READ_FLAVOR_FAST) // All of the above.	52	SRCU_READ_FLAVOR_FAST) // All of the above.
53	#define SRCU_READ_FLAVOR_SLOWGP (SRCU_READ_FLAVOR_LITE \| SRCU_READ_FLAVOR_FAST)	53	#define SRCU_READ_FLAVOR_SLOWGP SRCU_READ_FLAVOR_FAST
54	// Flavors requiring synchronize_rcu()	54	// Flavors requiring synchronize_rcu()
55	// instead of smp_mb().	55	// instead of smp_mb().
56	void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp);	56	void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp);
@@ -299,33 +299,6 @@ static inline struct srcu_ctr __percpu srcu_down_read_fast(struct srcu_struct · 242 unmodified lines
299	return __srcu_read_lock_fast(ssp);	299	return __srcu_read_lock_fast(ssp);
300	}	300	}
301		301
302	/**
303	* srcu_read_lock_lite - register a new reader for an SRCU-protected structure.
304	* @ssp: srcu_struct in which to register the new reader.
305	*
306	* Enter an SRCU read-side critical section, but for a light-weight
307	* smp_mb()-free reader. See srcu_read_lock() for more information.
308	*
309	* If srcu_read_lock_lite() is ever used on an srcu_struct structure,
310	* then none of the other flavors may be used, whether before, during,
311	* or after. Note that grace-period auto-expediting is disabled for _lite
312	* srcu_struct structures because auto-expedited grace periods invoke
313	* synchronize_rcu_expedited(), IPIs and all.
314	*
315	* Note that srcu_read_lock_lite() can be invoked only from those contexts
316	* where RCU is watching, that is, from contexts where it would be legal
317	* to invoke rcu_read_lock(). Otherwise, lockdep will complain.
318	*/
319	static inline int srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp)
320	{
321	int retval;
322
323	srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_LITE);
324	retval = __srcu_read_lock_lite(ssp);
325	rcu_try_lock_acquire(&ssp->dep_map);
326	return retval;
327	}
328
329	/**	302	/**
330	* srcu_read_lock_nmisafe - register a new reader for an SRCU-protected structure.	303	* srcu_read_lock_nmisafe - register a new reader for an SRCU-protected structure.
331	* @ssp: srcu_struct in which to register the new reader.	304	* @ssp: srcu_struct in which to register the new reader.
@@ -434,22 +407,6 @@ static inline void srcu_up_read_fast(struct srcu_struct *ssp, struct srcu_ctr __ · 102 unmodified lines
434	__srcu_read_unlock_fast(ssp, scp);	407	__srcu_read_unlock_fast(ssp, scp);
435	}	408	}
436		409
437	/**
438	* srcu_read_unlock_lite - unregister a old reader from an SRCU-protected structure.
439	* @ssp: srcu_struct in which to unregister the old reader.
440	* @idx: return value from corresponding srcu_read_lock_lite().
441	*
442	* Exit a light-weight SRCU read-side critical section.
443	*/
444	static inline void srcu_read_unlock_lite(struct srcu_struct *ssp, int idx)
445	__releases(ssp)
446	{
447	WARN_ON_ONCE(idx & ~0x1);
448	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE);
449	srcu_lock_release(&ssp->dep_map);
450	__srcu_read_unlock_lite(ssp, idx);
451	}
452
453	/**	410	/**
454	* srcu_read_unlock_nmisafe - unregister a old reader from an SRCU-protected structure.	411	* srcu_read_unlock_nmisafe - unregister a old reader from an SRCU-protected structure.
455	* @ssp: srcu_struct in which to unregister the old reader.	412	* @ssp: srcu_struct in which to unregister the old reader.
@@ -524,4 +481,9 @@ DEFINE_LOCK_GUARD_1(srcu, struct srcu_struct, · 68 unmodified lines
524	srcu_read_unlock(_T->lock, _T->idx),	481	srcu_read_unlock(_T->lock, _T->idx),
525	int idx)	482	int idx)
526		483
		484	DEFINE_LOCK_GUARD_1(srcu_fast, struct srcu_struct,
		485	_T->scp = srcu_read_lock_fast(_T->lock),
		486	srcu_read_unlock_fast(_T->lock, _T->scp),
		487	struct srcu_ctr __percpu *scp)
		488
527	#endif	489	#endif

include/linux/srcutiny.h+0−3

@@ -93,9 +93,6 @@ static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ · 92 unmodified lines
93	__srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp));	93	__srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp));
94	}	94	}
95		95
96	#define __srcu_read_lock_lite __srcu_read_lock
97	#define __srcu_read_unlock_lite __srcu_read_unlock
98
99	static inline void synchronize_srcu_expedited(struct srcu_struct *ssp)	96	static inline void synchronize_srcu_expedited(struct srcu_struct *ssp)
100	{	97	{
101	synchronize_srcu(ssp);	98	synchronize_srcu(ssp);

include/linux/srcutree.h+0−38

@@ -278,44 +278,6 @@ static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ · 277 unmodified lines
278	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_fast().");	278	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_fast().");
279	}	279	}
280		280
281	/*
282	* Counts the new reader in the appropriate per-CPU element of the
283	* srcu_struct. Returns an index that must be passed to the matching
284	* srcu_read_unlock_lite().
285	*
286	* Note that this_cpu_inc() is an RCU read-side critical section either
287	* because it disables interrupts, because it is a single instruction,
288	* or because it is a read-modify-write atomic operation, depending on
289	* the whims of the architecture.
290	*/
291	static inline int __srcu_read_lock_lite(struct srcu_struct *ssp)
292	{
293	struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp);
294
295	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite().");
296	this_cpu_inc(scp->srcu_locks.counter); /* Y */
297	barrier(); /* Avoid leaking the critical section. */
298	return __srcu_ptr_to_ctr(ssp, scp);
299	}
300
301	/*
302	* Removes the count for the old reader from the appropriate
303	* per-CPU element of the srcu_struct. Note that this may well be a
304	* different CPU than that which was incremented by the corresponding
305	* srcu_read_lock_lite(), but it must be within the same task.
306	*
307	* Note that this_cpu_inc() is an RCU read-side critical section either
308	* because it disables interrupts, because it is a single instruction,
309	* or because it is a read-modify-write atomic operation, depending on
310	* the whims of the architecture.
311	*/
312	static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx)
313	{
314	barrier(); /* Avoid leaking the critical section. */
315	this_cpu_inc(__srcu_ctr_to_ptr(ssp, idx)->srcu_unlocks.counter); /* Z */
316	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite().");
317	}
318
319	void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor);	281	void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor);
320		282
321	// Record reader usage even for CONFIG_PROVE_RCU=n kernels. This is	283	// Record reader usage even for CONFIG_PROVE_RCU=n kernels. This is

kernel/rcu/rcutorture.c+298−58

 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.ibm.com> and Josh Triplett <josh@joshtriplett.org>");
-/* Bits for ->extendables field, extendables param, and related definitions. */
+// Bits for ->extendables field, extendables param, and related definitions.
-#define RCUTORTURE_RDR_SHIFT_1	 8	/* Put SRCU index in upper bits. */
+#define RCUTORTURE_RDR_SHIFT_1	8	// Put SRCU index in upper bits.
-#define RCUTORTURE_RDR_MASK_1	 (0xff << RCUTORTURE_RDR_SHIFT_1)
+#define RCUTORTURE_RDR_MASK_1	(0xff << RCUTORTURE_RDR_SHIFT_1)
-#define RCUTORTURE_RDR_SHIFT_2	 16	/* Put SRCU index in upper bits. */
+#define RCUTORTURE_RDR_SHIFT_2	16	// Put SRCU index in upper bits.
-#define RCUTORTURE_RDR_MASK_2	 (0xff << RCUTORTURE_RDR_SHIFT_2)
+#define RCUTORTURE_RDR_MASK_2	(0xff << RCUTORTURE_RDR_SHIFT_2)
-#define RCUTORTURE_RDR_BH	 0x01	/* Extend readers by disabling bh. */
+#define RCUTORTURE_RDR_BH	0x01	// Extend readers by disabling bh.
-#define RCUTORTURE_RDR_IRQ	 0x02	/*  ... disabling interrupts. */
+#define RCUTORTURE_RDR_IRQ	0x02	//  ... disabling interrupts.
-#define RCUTORTURE_RDR_PREEMPT	 0x04	/*  ... disabling preemption. */
+#define RCUTORTURE_RDR_PREEMPT	0x04	//  ... disabling preemption.
-#define RCUTORTURE_RDR_RBH	 0x08	/*  ... rcu_read_lock_bh(). */
+#define RCUTORTURE_RDR_RBH	0x08	//  ... rcu_read_lock_bh().
-#define RCUTORTURE_RDR_SCHED	 0x10	/*  ... rcu_read_lock_sched(). */
+#define RCUTORTURE_RDR_SCHED	0x10	//  ... rcu_read_lock_sched().
-#define RCUTORTURE_RDR_RCU_1	 0x20	/*  ... entering another RCU reader. */
+#define RCUTORTURE_RDR_RCU_1	0x20	//  ... entering another RCU reader.
-#define RCUTORTURE_RDR_RCU_2	 0x40	/*  ... entering another RCU reader. */
+#define RCUTORTURE_RDR_RCU_2	0x40	//  ... entering another RCU reader.
-#define RCUTORTURE_RDR_NBITS	 7	/* Number of bits defined above. */
+#define RCUTORTURE_RDR_UPDOWN	0x80	//  ... up-read from task, down-read from timer.
-#define RCUTORTURE_MAX_EXTEND	 \
+					//	Note: Manual start, automatic end.
+#define RCUTORTURE_RDR_NBITS	8	// Number of bits defined above.
+#define RCUTORTURE_MAX_EXTEND	\
 	(RCUTORTURE_RDR_BH | RCUTORTURE_RDR_IRQ | RCUTORTURE_RDR_PREEMPT | \
-	 RCUTORTURE_RDR_RBH | RCUTORTURE_RDR_SCHED)
+	 RCUTORTURE_RDR_RBH | RCUTORTURE_RDR_SCHED)  // Intentionally omit RCUTORTURE_RDR_UPDOWN.
 #define RCUTORTURE_RDR_ALLBITS	\
 	(RCUTORTURE_MAX_EXTEND | RCUTORTURE_RDR_RCU_1 | RCUTORTURE_RDR_RCU_2 | \
 	 RCUTORTURE_RDR_MASK_1 | RCUTORTURE_RDR_MASK_2)
 torture_param(int, irqreader, 1, "Allow RCU readers from irq handlers");
 torture_param(int, leakpointer, 0, "Leak pointer dereferences from readers");
 torture_param(int, n_barrier_cbs, 0, "# of callbacks/kthreads for barrier testing");
+torture_param(int, n_up_down, 32, "# of concurrent up/down hrtimer-based RCU readers");
 torture_param(int, nfakewriters, 4, "Number of RCU fake writer threads");
 torture_param(int, nreaders, -1, "Number of RCU reader threads");
 torture_param(int, object_debug, 0, "Enable debug-object double call_rcu() testing");
 static struct task_struct *writer_task;
 static struct task_struct **fakewriter_tasks;
 static struct task_struct **reader_tasks;
+static struct task_struct *updown_task;
 static struct task_struct **nocb_tasks;
 static struct task_struct *stats_task;
 static struct task_struct *fqs_task;
 	void (*readunlock)(int idx);
 	int (*readlock_held)(void);   // lockdep.
 	int (*readlock_nesting)(void); // actual nesting, if available, -1 if not.
+	int (*down_read)(void);
+	void (*up_read)(int idx);
 	unsigned long (*get_gp_seq)(void);
 	unsigned long (*gp_diff)(unsigned long new, unsigned long old);
 	void (*deferred_free)(struct rcu_torture *p);
 	int no_pi_lock;
 	int debug_objects;
 	int start_poll_irqsoff;
+	int have_up_down;
 	const char *name;
 };
 	    !(torture_random(rrsp) % (nrealreaders * 2000 * longdelay_ms))) {
 		started = cur_ops->get_gp_seq();
 		ts = rcu_trace_clock_local();
-		if (preempt_count() & (SOFTIRQ_MASK | HARDIRQ_MASK))
+		if ((preempt_count() & HARDIRQ_MASK) || softirq_count())
 			longdelay_ms = 5; /* Avoid triggering BH limits. */
 		mdelay(longdelay_ms);
 		rtrsp->rt_delay_ms = longdelay_ms;
 		WARN_ON_ONCE(idx & ~0x1);
 		ret += idx << 1;
 	}
-	if (reader_flavor & SRCU_READ_FLAVOR_LITE) {
-		idx = srcu_read_lock_lite(srcu_ctlp);
-		WARN_ON_ONCE(idx & ~0x1);
-		ret += idx << 2;
-	}
 	if (reader_flavor & SRCU_READ_FLAVOR_FAST) {
 		scp = srcu_read_lock_fast(srcu_ctlp);
 		idx = __srcu_ptr_to_ctr(srcu_ctlp, scp);
 	WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1)));
 	if (reader_flavor & SRCU_READ_FLAVOR_FAST)
 		srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3));
-	if (reader_flavor & SRCU_READ_FLAVOR_LITE)
-		srcu_read_unlock_lite(srcu_ctlp, (idx & 0x4) >> 2);
 	if (reader_flavor & SRCU_READ_FLAVOR_NMI)
 		srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1);
 	if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_READ_FLAVOR_ALL))
 	return srcu_read_lock_held(srcu_ctlp);
 }
+static bool srcu_torture_have_up_down(void)
+{
+	int rf = reader_flavor;
+	if (!rf)
+		rf = SRCU_READ_FLAVOR_NORMAL;
+	return !!(cur_ops->have_up_down & rf);
+}
+static int srcu_torture_down_read(void)
+{
+	int idx;
+	struct srcu_ctr __percpu *scp;
+	WARN_ON_ONCE(reader_flavor & ~SRCU_READ_FLAVOR_ALL);
+	WARN_ON_ONCE(reader_flavor & (reader_flavor - 1));
+	if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_READ_FLAVOR_ALL)) {
+		idx = srcu_down_read(srcu_ctlp);
+		WARN_ON_ONCE(idx & ~0x1);
+		return idx;
+	}
+	if (reader_flavor & SRCU_READ_FLAVOR_FAST) {
+		scp = srcu_down_read_fast(srcu_ctlp);
+		idx = __srcu_ptr_to_ctr(srcu_ctlp, scp);
+		WARN_ON_ONCE(idx & ~0x1);
+		return idx << 3;
+	}
+	WARN_ON_ONCE(1);
+	return 0;
+}
+static void srcu_torture_up_read(int idx)
+{
+	WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor && (idx & ~0x1)));
+	if (reader_flavor & SRCU_READ_FLAVOR_FAST)
+		srcu_up_read_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3));
+	else if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) ||
+		 !(reader_flavor & SRCU_READ_FLAVOR_ALL))
+		srcu_up_read(srcu_ctlp, idx & 0x1);
+	else
+		WARN_ON_ONCE(1);
+}
 static unsigned long srcu_torture_completed(void)
 {
 	return srcu_batches_completed(srcu_ctlp);
 	.readlock	= srcu_torture_read_lock,
 	.read_delay	= srcu_read_delay,
 	.readunlock	= srcu_torture_read_unlock,
+	.down_read	= srcu_torture_down_read,
+	.up_read	= srcu_torture_up_read,
 	.readlock_held	= torture_srcu_read_lock_held,
 	.get_gp_seq	= srcu_torture_completed,
 	.gp_diff	= rcu_seq_diff,
 	.irq_capable	= 1,
 	.no_pi_lock	= IS_ENABLED(CONFIG_TINY_SRCU),
 	.debug_objects	= 1,
+	.have_up_down	= IS_ENABLED(CONFIG_TINY_SRCU)
+				? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST,
 	.name		= "srcu"
 };
 	.read_delay	= srcu_read_delay,
 	.readunlock	= srcu_torture_read_unlock,
 	.readlock_held	= torture_srcu_read_lock_held,
+	.down_read	= srcu_torture_down_read,
+	.up_read	= srcu_torture_up_read,
 	.get_gp_seq	= srcu_torture_completed,
 	.gp_diff	= rcu_seq_diff,
 	.deferred_free	= srcu_torture_deferred_free,
 	.irq_capable	= 1,
 	.no_pi_lock	= IS_ENABLED(CONFIG_TINY_SRCU),
 	.debug_objects	= 1,
+	.have_up_down	= IS_ENABLED(CONFIG_TINY_SRCU)
+				? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST,
 	.name		= "srcud"
 };
 /*
  * Definitions for trivial CONFIG_PREEMPT=n-only torture testing.
- * This implementation does not necessarily work well with CPU hotplug.
+ * This implementation does not work well with CPU hotplug nor
+ * with rcutorture's shuffling.
  */
 static void synchronize_rcu_trivial(void)
 	}
 }
+static void rcu_sync_torture_init_trivial(void)
+{
+	rcu_sync_torture_init();
+	// if (onoff_interval || shuffle_interval) {
+	if (WARN_ONCE(onoff_interval || shuffle_interval, "%s: Non-zero onoff_interval (%d) or shuffle_interval (%d) breaks trivial RCU, resetting to zero", __func__, onoff_interval, shuffle_interval)) {
+		onoff_interval = 0;
+		shuffle_interval = 0;
+	}
+}
 static int rcu_torture_read_lock_trivial(void)
 {
 	preempt_disable();
 static struct rcu_torture_ops trivial_ops = {
 	.ttype		= RCU_TRIVIAL_FLAVOR,
-	.init		= rcu_sync_torture_init,
+	.init		= rcu_sync_torture_init_trivial,
 	.readlock	= rcu_torture_read_lock_trivial,
 	.read_delay	= rcu_read_delay,  /* just reuse rcu's version. */
 	.readunlock	= rcu_torture_read_unlock_trivial,
 						cur_ops->gp_kthread_dbg();
 					WARN(1, "%s: rtort_pipe_count: %d\n", __func__, rcu_tortures[i].rtort_pipe_count);
 					rcu_ftrace_dump(DUMP_ALL);
+					break;
 				}
 		if (stutter_waited)
 			sched_set_normal(current, oldnice);
 // Verify the specified RCUTORTURE_RDR* state.
 #define ROEC_ARGS "%s %s: Current %#x  To add %#x  To remove %#x  preempt_count() %#x\n", __func__, s, curstate, new, old, preempt_count()
-static void rcutorture_one_extend_check(char *s, int curstate, int new, int old, bool insoftirq)
+static void rcutorture_one_extend_check(char *s, int curstate, int new, int old)
 {
 	int mask;
-	if (!IS_ENABLED(CONFIG_RCU_TORTURE_TEST_CHK_RDR_STATE))
+	if (!IS_ENABLED(CONFIG_RCU_TORTURE_TEST_CHK_RDR_STATE) || in_nmi())
 		return;
-	WARN_ONCE(!(curstate & RCUTORTURE_RDR_IRQ) && irqs_disabled(), ROEC_ARGS);
+	WARN_ONCE(!(curstate & RCUTORTURE_RDR_IRQ) && irqs_disabled() && !in_hardirq(), ROEC_ARGS);
 	WARN_ONCE((curstate & RCUTORTURE_RDR_IRQ) && !irqs_disabled(), ROEC_ARGS);
 	// If CONFIG_PREEMPT_COUNT=n, further checks are unreliable.
 		return;
 	WARN_ONCE((curstate & (RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH)) &&
-		  !(preempt_count() & SOFTIRQ_MASK), ROEC_ARGS);
+		  !softirq_count(), ROEC_ARGS);
 	WARN_ONCE((curstate & (RCUTORTURE_RDR_PREEMPT | RCUTORTURE_RDR_SCHED)) &&
 		  !(preempt_count() & PREEMPT_MASK), ROEC_ARGS);
 	WARN_ONCE(cur_ops->readlock_nesting &&
 		  (curstate & (RCUTORTURE_RDR_RCU_1 | RCUTORTURE_RDR_RCU_2)) &&
 		  cur_ops->readlock_nesting() == 0, ROEC_ARGS);
-	// Timer handlers have all sorts of stuff disabled, so ignore
+	// Interrupt handlers have all sorts of stuff disabled, so ignore
 	// unintended disabling.
-	if (insoftirq)
+	if (in_serving_softirq() || in_hardirq())
 		return;
 	WARN_ONCE(cur_ops->extendables &&
 		  !(curstate & (RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH)) &&
-		  (preempt_count() & SOFTIRQ_MASK), ROEC_ARGS);
+		  softirq_count(), ROEC_ARGS);
 	/*
 	 * non-preemptible RCU in a preemptible kernel uses preempt_disable()
 	if (!IS_ENABLED(CONFIG_PREEMPT_RCU))
 		mask |= RCUTORTURE_RDR_PREEMPT | RCUTORTURE_RDR_SCHED;
+	if (IS_ENABLED(CONFIG_PREEMPT_RT) && softirq_count())
+		mask |= RCUTORTURE_RDR_BH | RCUTORTURE_RDR_RBH;
 	WARN_ONCE(cur_ops->readlock_nesting && !(curstate & mask) &&
 		  cur_ops->readlock_nesting() > 0, ROEC_ARGS);
 }
  * beginning or end of the critical section and if there was actually a
  * change, do a ->read_delay().
  */
-static void rcutorture_one_extend(int *readstate, int newstate, bool insoftirq,
+static void rcutorture_one_extend(int *readstate, int newstate, struct torture_random_state *trsp,
-				  struct torture_random_state *trsp,
 				  struct rt_read_seg *rtrsp)
 {
 	bool first;
 	first = idxold1 == 0;
 	WARN_ON_ONCE(idxold2 < 0);
-	WARN_ON_ONCE(idxold2 & ~RCUTORTURE_RDR_ALLBITS);
+	WARN_ON_ONCE(idxold2 & ~(RCUTORTURE_RDR_ALLBITS | RCUTORTURE_RDR_UPDOWN));
-	rcutorture_one_extend_check("before change", idxold1, statesnew, statesold, insoftirq);
+	rcutorture_one_extend_check("before change", idxold1, statesnew, statesold);
 	rtrsp->rt_readstate = newstate;
 	/* First, put new protection in place to avoid critical-section gap. */
 		idxnew2 = (cur_ops->readlock() << RCUTORTURE_RDR_SHIFT_2) & RCUTORTURE_RDR_MASK_2;
 	// Complain unless both the old and the new protection is in place.
-	rcutorture_one_extend_check("during change",
+	rcutorture_one_extend_check("during change", idxold1 | statesnew, statesnew, statesold);
-				    idxold1 | statesnew, statesnew, statesold, insoftirq);
 	// Sample CPU under both sets of protections to reduce confusion.
 	if (IS_ENABLED(CONFIG_RCU_TORTURE_TEST_LOG_CPU)) {
 		if (lockit)
 			raw_spin_unlock_irqrestore(&current->pi_lock, flags);
 	}
+	if (statesold & RCUTORTURE_RDR_UPDOWN) {
+		cur_ops->up_read((idxold1 & RCUTORTURE_RDR_MASK_1) >> RCUTORTURE_RDR_SHIFT_1);
+		WARN_ON_ONCE(idxnew1 != -1);
+		idxold1 = 0;
+	}
 	/* Delay if neither beginning nor end and there was a change. */
 	if ((statesnew || statesold) && *readstate && newstate)
 	WARN_ON_ONCE(*readstate < 0);
 	if (WARN_ON_ONCE(*readstate & ~RCUTORTURE_RDR_ALLBITS))
 		pr_info("Unexpected readstate value of %#x\n", *readstate);
-	rcutorture_one_extend_check("after change", *readstate, statesnew, statesold, insoftirq);
+	rcutorture_one_extend_check("after change", *readstate, statesnew, statesold);
 }
 /* Return the biggest extendables mask given current RCU and boot parameters. */
  * critical section.
  */
 static struct rt_read_seg *
-rcutorture_loop_extend(int *readstate, bool insoftirq, struct torture_random_state *trsp,
+rcutorture_loop_extend(int *readstate, struct torture_random_state *trsp, struct rt_read_seg *rtrsp)
-		       struct rt_read_seg *rtrsp)
 {
 	int i;
 	int j;
 	i = ((i | (i >> 3)) & RCUTORTURE_RDR_MAX_LOOPS) + 1;
 	for (j = 0; j < i; j++) {
 		mask = rcutorture_extend_mask(*readstate, trsp);
-		rcutorture_one_extend(readstate, mask, insoftirq, trsp, &rtrsp[j]);
+		WARN_ON_ONCE(mask & RCUTORTURE_RDR_UPDOWN);
+		rcutorture_one_extend(readstate, mask, trsp, &rtrsp[j]);
 	}
 	return &rtrsp[j];
 }
 	rtorsp->started = cur_ops->get_gp_seq();
 	rtorsp->ts = rcu_trace_clock_local();
 	rtorsp->p = rcu_dereference_check(rcu_torture_current,
-				  !cur_ops->readlock_held || cur_ops->readlock_held());
+					  !cur_ops->readlock_held || cur_ops->readlock_held() ||
+					  (rtorsp->readstate & RCUTORTURE_RDR_UPDOWN));
 	if (rtorsp->p == NULL) {
 		/* Wait for rcu_torture_writer to get underway */
-		rcutorture_one_extend(&rtorsp->readstate, 0, myid < 0, trsp, rtorsp->rtrsp);
+		rcutorture_one_extend(&rtorsp->readstate, 0, trsp, rtorsp->rtrsp);
 		return false;
 	}
 	if (rtorsp->p->rtort_mbtest == 0)
  * critical sections and check for errors.
  */
 static void rcu_torture_one_read_end(struct rcu_torture_one_read_state *rtorsp,
-				     struct torture_random_state *trsp, long myid)
+				     struct torture_random_state *trsp)
 {
 	int i;
 	unsigned long completed;
 	}
 	if (cur_ops->reader_blocked)
 		preempted = cur_ops->reader_blocked();
-	rcutorture_one_extend(&rtorsp->readstate, 0, myid < 0, trsp, rtorsp->rtrsp);
+	rcutorture_one_extend(&rtorsp->readstate, 0, trsp, rtorsp->rtrsp);
 	WARN_ON_ONCE(rtorsp->readstate);
 	// This next splat is expected behavior if leakpointer, especially
 	// for CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels.
 	WARN_ON_ONCE(!rcu_is_watching());
 	init_rcu_torture_one_read_state(&rtors, trsp);
 	newstate = rcutorture_extend_mask(rtors.readstate, trsp);
-	rcutorture_one_extend(&rtors.readstate, newstate, myid < 0, trsp, rtors.rtrsp++);
+	WARN_ON_ONCE(newstate & RCUTORTURE_RDR_UPDOWN);
+	rcutorture_one_extend(&rtors.readstate, newstate, trsp, rtors.rtrsp++);
 	if (!rcu_torture_one_read_start(&rtors, trsp, myid)) {
-		rcutorture_one_extend(&rtors.readstate, 0, myid < 0, trsp, rtors.rtrsp);
+		rcutorture_one_extend(&rtors.readstate, 0, trsp, rtors.rtrsp);
 		return false;
 	}
-	rtors.rtrsp = rcutorture_loop_extend(&rtors.readstate, myid < 0, trsp, rtors.rtrsp);
+	rtors.rtrsp = rcutorture_loop_extend(&rtors.readstate, trsp, rtors.rtrsp);
-	rcu_torture_one_read_end(&rtors, trsp, myid);
+	rcu_torture_one_read_end(&rtors, trsp);
 	return true;
 }
 	return 0;
 }
+struct rcu_torture_one_read_state_updown {
+	struct hrtimer rtorsu_hrt;
+	bool rtorsu_inuse;
+	ktime_t rtorsu_kt;
+	int rtorsu_cpu;
+	unsigned long rtorsu_j;
+	unsigned long rtorsu_ndowns;
+	unsigned long rtorsu_nups;
+	unsigned long rtorsu_nmigrates;
+	struct torture_random_state rtorsu_trs;
+	struct rcu_torture_one_read_state rtorsu_rtors;
+};
+static struct rcu_torture_one_read_state_updown *updownreaders;
+static DEFINE_TORTURE_RANDOM(rcu_torture_updown_rand);
+static int rcu_torture_updown(void *arg);
+static enum hrtimer_restart rcu_torture_updown_hrt(struct hrtimer *hrtp)
+{
+	int cpu = raw_smp_processor_id();
+	struct rcu_torture_one_read_state_updown *rtorsup;
+	rtorsup = container_of(hrtp, struct rcu_torture_one_read_state_updown, rtorsu_hrt);
+	rcu_torture_one_read_end(&rtorsup->rtorsu_rtors, &rtorsup->rtorsu_trs);
+	WARN_ONCE(rtorsup->rtorsu_nups >= rtorsup->rtorsu_ndowns, "%s: Up without matching down #%zu.\n", __func__, rtorsup - updownreaders);
+	WRITE_ONCE(rtorsup->rtorsu_nups, rtorsup->rtorsu_nups + 1);
+	WRITE_ONCE(rtorsup->rtorsu_nmigrates,
+		   rtorsup->rtorsu_nmigrates + (cpu != rtorsup->rtorsu_cpu));
+	smp_store_release(&rtorsup->rtorsu_inuse, false);
+	return HRTIMER_NORESTART;
+}
+static int rcu_torture_updown_init(void)
+{
+	int i;
+	struct torture_random_state *rand = &rcu_torture_updown_rand;
+	int ret;
+	if (n_up_down < 0)
+		return 0;
+	if (!srcu_torture_have_up_down()) {
+		VERBOSE_TOROUT_STRING("rcu_torture_updown_init: Disabling up/down reader tests due to lack of primitives");
+		return 0;
+	}
+	updownreaders = kcalloc(n_up_down, sizeof(*updownreaders), GFP_KERNEL);
+	if (!updownreaders) {
+		VERBOSE_TOROUT_STRING("rcu_torture_updown_init: Out of memory, disabling up/down reader tests");
+		return -ENOMEM;
+	}
+	for (i = 0; i < n_up_down; i++) {
+		init_rcu_torture_one_read_state(&updownreaders[i].rtorsu_rtors, rand);
+		hrtimer_setup(&updownreaders[i].rtorsu_hrt, rcu_torture_updown_hrt, CLOCK_MONOTONIC,
+			      HRTIMER_MODE_REL | HRTIMER_MODE_HARD);
+		torture_random_init(&updownreaders[i].rtorsu_trs);
+		init_rcu_torture_one_read_state(&updownreaders[i].rtorsu_rtors,
+						&updownreaders[i].rtorsu_trs);
+	}
+	ret = torture_create_kthread(rcu_torture_updown, rand, updown_task);
+	if (ret) {
+		kfree(updownreaders);
+		updownreaders = NULL;
+	}
+	return ret;
+}
+static void rcu_torture_updown_cleanup(void)
+{
+	struct rcu_torture_one_read_state_updown *rtorsup;
+	for (rtorsup = updownreaders; rtorsup < &updownreaders[n_up_down]; rtorsup++) {
+		if (!smp_load_acquire(&rtorsup->rtorsu_inuse))
+			continue;
+		if (hrtimer_cancel(&rtorsup->rtorsu_hrt) || WARN_ON_ONCE(rtorsup->rtorsu_inuse)) {
+			rcu_torture_one_read_end(&rtorsup->rtorsu_rtors, &rtorsup->rtorsu_trs);
+			WARN_ONCE(rtorsup->rtorsu_nups >= rtorsup->rtorsu_ndowns, "%s: Up without matching down #%zu.\n", __func__, rtorsup - updownreaders);
+			WRITE_ONCE(rtorsup->rtorsu_nups, rtorsup->rtorsu_nups + 1);
+			smp_store_release(&rtorsup->rtorsu_inuse, false);
+		}
+	}
+	kfree(updownreaders);
+	updownreaders = NULL;
+}
+// Do one reader for rcu_torture_updown().
+static void rcu_torture_updown_one(struct rcu_torture_one_read_state_updown *rtorsup)
+{
+	int idx;
+	int rawidx;
+	ktime_t t;
+	init_rcu_torture_one_read_state(&rtorsup->rtorsu_rtors, &rtorsup->rtorsu_trs);
+	rawidx = cur_ops->down_read();
+	WRITE_ONCE(rtorsup->rtorsu_ndowns, rtorsup->rtorsu_ndowns + 1);
+	idx = (rawidx << RCUTORTURE_RDR_SHIFT_1) & RCUTORTURE_RDR_MASK_1;
+	rtorsup->rtorsu_rtors.readstate = idx | RCUTORTURE_RDR_UPDOWN;
+	rtorsup->rtorsu_rtors.rtrsp++;
+	rtorsup->rtorsu_cpu = raw_smp_processor_id();
+	if (!rcu_torture_one_read_start(&rtorsup->rtorsu_rtors, &rtorsup->rtorsu_trs, -1)) {
+		WARN_ONCE(rtorsup->rtorsu_nups >= rtorsup->rtorsu_ndowns, "%s: Up without matching down #%zu.\n", __func__, rtorsup - updownreaders);
+		WRITE_ONCE(rtorsup->rtorsu_nups, rtorsup->rtorsu_nups + 1);
+		schedule_timeout_idle(HZ);
+		return;
+	}
+	smp_store_release(&rtorsup->rtorsu_inuse, true);
+	t = torture_random(&rtorsup->rtorsu_trs) & 0xfffff; // One per million.
+	if (t < 10 * 1000)
+		t = 200 * 1000 * 1000;
+	hrtimer_start(&rtorsup->rtorsu_hrt, t, HRTIMER_MODE_REL | HRTIMER_MODE_HARD);
+	smp_mb(); // Sample jiffies after posting hrtimer.
+	rtorsup->rtorsu_j = jiffies;  // Not used by hrtimer handler.
+	rtorsup->rtorsu_kt = t;
+}
+/*
+ * RCU torture up/down reader kthread, starting RCU readers in kthread
+ * context and ending them in hrtimer handlers.  Otherwise similar to
+ * rcu_torture_reader().
+ */
+static int
+rcu_torture_updown(void *arg)
+{
+	unsigned long j;
+	struct rcu_torture_one_read_state_updown *rtorsup;
+	VERBOSE_TOROUT_STRING("rcu_torture_updown task started");
+	do {
+		for (rtorsup = updownreaders; rtorsup < &updownreaders[n_up_down]; rtorsup++) {
+			if (torture_must_stop())
+				break;
+			j = smp_load_acquire(&jiffies); // Time before ->rtorsu_inuse.
+			if (smp_load_acquire(&rtorsup->rtorsu_inuse)) {
+				WARN_ONCE(time_after(j, rtorsup->rtorsu_j + 1 + HZ * 10),
+					  "hrtimer queued at jiffies %lu for %lld ns took %lu jiffies\n", rtorsup->rtorsu_j, rtorsup->rtorsu_kt, j - rtorsup->rtorsu_j);
+				continue;
+			}
+			rcu_torture_updown_one(rtorsup);
+		}
+		torture_hrtimeout_ms(1, 1000, &rcu_torture_updown_rand);
+		stutter_wait("rcu_torture_updown");
+	} while (!torture_must_stop());
+	rcu_torture_updown_cleanup();
+	torture_kthread_stopping("rcu_torture_updown");
+	return 0;
+}
 /*
  * Randomly Toggle CPUs' callback-offload state.  This uses hrtimers to
  * increase race probabilities and fuzzes the interval between toggling.
 	long pipesummary[RCU_TORTURE_PIPE_LEN + 1] = { 0 };
 	long batchsummary[RCU_TORTURE_PIPE_LEN + 1] = { 0 };
 	long n_gpwraps = 0;
+	unsigned long ndowns = 0;
+	unsigned long nunexpired = 0;
+	unsigned long nmigrates = 0;
+	unsigned long nups = 0;
 	struct rcu_torture *rtcp;
 	static unsigned long rtcv_snap = ULONG_MAX;
 	static bool splatted;
 		if (cur_ops->get_gpwrap_count)
 			n_gpwraps += cur_ops->get_gpwrap_count(cpu);
 	}
+	if (updownreaders) {
+		for (i = 0; i < n_up_down; i++) {
+			ndowns += READ_ONCE(updownreaders[i].rtorsu_ndowns);
+			nups += READ_ONCE(updownreaders[i].rtorsu_nups);
+			nunexpired += READ_ONCE(updownreaders[i].rtorsu_inuse);
+			nmigrates += READ_ONCE(updownreaders[i].rtorsu_nmigrates);
+		}
+	}
 	for (i = RCU_TORTURE_PIPE_LEN; i >= 0; i--) {
 		if (pipesummary[i] != 0)
 			break;
-	}
+	} // The value of variable "i" is used later, so don't clobber it!
 	pr_alert("%s%s ", torture_type, TORTURE_FLAG);
 	rtcp = rcu_access_pointer(rcu_torture_current);
 		n_rcu_torture_boost_failure,
 		n_rcu_torture_boosts,
 		atomic_long_read(&n_rcu_torture_timers));
+	if (updownreaders)
+		pr_cont("ndowns: %lu nups: %lu nhrt: %lu nmigrates: %lu ", ndowns, nups, nunexpired,  nmigrates);
 	torture_onoff_stats();
 	pr_cont("barrier: %ld/%ld:%ld ",
 		data_race(n_barrier_successes),
 		 "reader_flavor=%x "
 		 "nocbs_nthreads=%d nocbs_toggle=%d "
 		 "test_nmis=%d "
-		 "preempt_duration=%d preempt_interval=%d\n",
+		 "preempt_duration=%d preempt_interval=%d n_up_down=%d\n",
 		 torture_type, tag, nrealreaders, nrealfakewriters,
 		 stat_interval, verbose, test_no_idle_hz, shuffle_interval,
 		 stutter, irqreader, fqs_duration, fqs_holdoff, fqs_stutter,
 		 reader_flavor,
 		 nocbs_nthreads, nocbs_toggle,
 		 test_nmis,
-		 preempt_duration, preempt_interval);
+		 preempt_duration, preempt_interval, n_up_down);
 }
 static int rcutorture_booster_cleanup(unsigned int cpu)
 		nocb_tasks = NULL;
 	}
+	if (updown_task) {
+		torture_stop_kthread(rcu_torture_updown, updown_task);
+		updown_task = NULL;
+	}
 	if (reader_tasks) {
 		for (i = 0; i < nrealreaders; i++)
 			torture_stop_kthread(rcu_torture_reader,
 	/* Start up the kthreads. */
 	rcu_torture_write_types();
-	firsterr = torture_create_kthread(rcu_torture_writer, NULL,
-					  writer_task);
-	if (torture_init_error(firsterr))
-		goto unwind;
 	if (nrealfakewriters > 0) {
 		fakewriter_tasks = kcalloc(nrealfakewriters,
 					   sizeof(fakewriter_tasks[0]),
 		if (torture_init_error(firsterr))
 			goto unwind;
 	}
+	firsterr = torture_create_kthread(rcu_torture_writer, NULL,
+					  writer_task);
+	if (torture_init_error(firsterr))
+		goto unwind;
+	firsterr = rcu_torture_updown_init();
+	if (torture_init_error(firsterr))
+		goto unwind;
 	nrealnocbers = nocbs_nthreads;
 	if (WARN_ON(nrealnocbers < 0))
 		nrealnocbers = 1;

kernel/rcu/refscale.c+8−34

 // Number of typesafe_lookup structures, that is, the degree of concurrency.
 torture_param(long, lookup_instances, 0, "Number of typesafe_lookup structures.");
 // Number of loops per experiment, all readers execute operations concurrently.
-torture_param(long, loops, 10000, "Number of loops per experiment.");
+torture_param(int, loops, 10000, "Number of loops per experiment.");
 // Number of readers, with -1 defaulting to about 75% of the CPUs.
 torture_param(int, nreaders, -1, "Number of readers, -1 for 75% of CPUs.");
 // Number of runs.
 	.name		= "srcu-fast"
 };
-static void srcu_lite_ref_scale_read_section(const int nloops)
-{
-	int i;
-	int idx;
-	for (i = nloops; i >= 0; i--) {
-		idx = srcu_read_lock_lite(srcu_ctlp);
-		srcu_read_unlock_lite(srcu_ctlp, idx);
-	}
-}
-static void srcu_lite_ref_scale_delay_section(const int nloops, const int udl, const int ndl)
-{
-	int i;
-	int idx;
-	for (i = nloops; i >= 0; i--) {
-		idx = srcu_read_lock_lite(srcu_ctlp);
-		un_delay(udl, ndl);
-		srcu_read_unlock_lite(srcu_ctlp, idx);
-	}
-}
-static const struct ref_scale_ops srcu_lite_ops = {
-	.init		= rcu_sync_scale_init,
-	.readsection	= srcu_lite_ref_scale_read_section,
-	.delaysection	= srcu_lite_ref_scale_delay_section,
-	.name		= "srcu-lite"
-};
 #ifdef CONFIG_TASKS_RCU
 // Definitions for RCU Tasks ref scale testing: Empty read markers.
 ref_scale_print_module_parms(const struct ref_scale_ops *cur_ops, const char *tag)
 {
 	pr_alert("%s" SCALE_FLAG
-		 "--- %s:  verbose=%d verbose_batched=%d shutdown=%d holdoff=%d lookup_instances=%ld loops=%ld nreaders=%d nruns=%d readdelay=%d\n", scale_type, tag,
+		 "--- %s:  verbose=%d verbose_batched=%d shutdown=%d holdoff=%d lookup_instances=%ld loops=%d nreaders=%d nruns=%d readdelay=%d\n", scale_type, tag,
 		 verbose, verbose_batched, shutdown, holdoff, lookup_instances, loops, nreaders, nruns, readdelay);
 }
 	long i;
 	int firsterr = 0;
 	static const struct ref_scale_ops *scale_ops[] = {
-		&rcu_ops, &srcu_ops, &srcu_fast_ops, &srcu_lite_ops, RCU_TRACE_OPS RCU_TASKS_OPS
+		&rcu_ops, &srcu_ops, &srcu_fast_ops, RCU_TRACE_OPS RCU_TASKS_OPS
 		&refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops,
 		&acqrel_ops, &sched_clock_ops, &clock_ops, &jiffies_ops,
 		&typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops,
 	// Reader tasks (default to ~75% of online CPUs).
 	if (nreaders < 0)
 		nreaders = (num_online_cpus() >> 1) + (num_online_cpus() >> 2);
-	if (WARN_ONCE(loops <= 0, "%s: loops = %ld, adjusted to 1\n", __func__, loops))
+	if (WARN_ONCE(loops <= 0, "%s: loops = %d, adjusted to 1\n", __func__, loops))
 		loops = 1;
 	if (WARN_ONCE(nreaders <= 0, "%s: nreaders = %d, adjusted to 1\n", __func__, nreaders))
 		nreaders = 1;
 	if (WARN_ONCE(nruns <= 0, "%s: nruns = %d, adjusted to 1\n", __func__, nruns))
 		nruns = 1;
+	if (WARN_ONCE(loops > INT_MAX / nreaders,
+		      "%s: nreaders * loops will overflow, adjusted loops to %d",
+		      __func__, INT_MAX / nreaders))
+		loops = INT_MAX / nreaders;
 	reader_tasks = kcalloc(nreaders, sizeof(reader_tasks[0]),
 			       GFP_KERNEL);
 	if (!reader_tasks) {

kernel/rcu/srcutree.c+2−0

@@ -502,6 +502,8 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) · 501 unmodified lines
502	*/	502	*/
503	if (!did_gp)	503	if (!did_gp)
504	smp_mb(); /* A */	504	smp_mb(); /* A */
		505	else if (srcu_gp_is_expedited(ssp))
		506	synchronize_rcu_expedited(); /* X */
505	else	507	else
506	synchronize_rcu(); /* X */	508	synchronize_rcu(); /* X */
507		509

kernel/rcu/tree.c+64−16

@@ -160,7 +160,6 @@ static void rcu_report_qs_rnp(unsigned long mask, struct rcu_node *rnp, · 159 unmodified lines
160	unsigned long gps, unsigned long flags);	160	unsigned long gps, unsigned long flags);
161	static void invoke_rcu_core(void);	161	static void invoke_rcu_core(void);
162	static void rcu_report_exp_rdp(struct rcu_data *rdp);	162	static void rcu_report_exp_rdp(struct rcu_data *rdp);
163	static void sync_sched_exp_online_cleanup(int cpu);
164	static void check_cb_ovld_locked(struct rcu_data rdp, struct rcu_node rnp);	163	static void check_cb_ovld_locked(struct rcu_data rdp, struct rcu_node rnp);
165	static bool rcu_rdp_is_offloaded(struct rcu_data *rdp);	164	static bool rcu_rdp_is_offloaded(struct rcu_data *rdp);
166	static bool rcu_rdp_cpu_online(struct rcu_data *rdp);	165	static bool rcu_rdp_cpu_online(struct rcu_data *rdp);
@@ -377,7 +376,7 @@ EXPORT_SYMBOL_GPL(rcu_momentary_eqs); · 210 unmodified lines
377	*/	376	*/
378	static int rcu_is_cpu_rrupt_from_idle(void)	377	static int rcu_is_cpu_rrupt_from_idle(void)
379	{	378	{
380	long nesting;	379	long nmi_nesting = ct_nmi_nesting();
381		380
382	/*	381	/*
383	* Usually called from the tick; but also used from smp_function_call()	382	* Usually called from the tick; but also used from smp_function_call()
@@ -389,21 +388,28 @@ static int rcu_is_cpu_rrupt_from_idle(void) · 5 unmodified lines
389	/* Check for counter underflows */	388	/* Check for counter underflows */
390	RCU_LOCKDEP_WARN(ct_nesting() < 0,	389	RCU_LOCKDEP_WARN(ct_nesting() < 0,
391	"RCU nesting counter underflow!");	390	"RCU nesting counter underflow!");
392	RCU_LOCKDEP_WARN(ct_nmi_nesting() <= 0,
393	"RCU nmi_nesting counter underflow/zero!");
394		391
395	/* Are we at first interrupt nesting level? */	392	/* Non-idle interrupt or nested idle interrupt */
396	nesting = ct_nmi_nesting();	393	if (nmi_nesting > 1)
397	if (nesting > 1)
398	return false;	394	return false;
399		395
400	/*	396	/*
401	* If we're not in an interrupt, we must be in the idle task!	397	* Non nested idle interrupt (interrupting section where RCU
		398	* wasn't watching).
402	*/	399	*/
403	WARN_ON_ONCE(!nesting && !is_idle_task(current));	400	if (nmi_nesting == 1)
		401	return true;
		402
		403	/* Not in an interrupt */
		404	if (!nmi_nesting) {
		405	RCU_LOCKDEP_WARN(!in_task() \|\| !is_idle_task(current),
		406	"RCU nmi_nesting counter not in idle task!");
		407	return !rcu_is_watching_curr_cpu();
		408	}
404		409
405	/* Does CPU appear to be idle from an RCU standpoint? */	410	RCU_LOCKDEP_WARN(1, "RCU nmi_nesting counter underflow/zero!");
406	return ct_nesting() == 0;	411
		412	return false;
407	}	413	}
408		414
409	#define DEFAULT_RCU_BLIMIT (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ? 1000 : 10)	415	#define DEFAULT_RCU_BLIMIT (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ? 1000 : 10)
@@ -1625,8 +1631,10 @@ static void rcu_sr_put_wait_head(struct llist_node *node) · 1215 unmodified lines
1625	atomic_set_release(&sr_wn->inuse, 0);	1631	atomic_set_release(&sr_wn->inuse, 0);
1626	}	1632	}
1627		1633
1628	/* Disabled by default. */	1634	/* Enable rcu_normal_wake_from_gp automatically on small systems. */
1629	static int rcu_normal_wake_from_gp;	1635	#define WAKE_FROM_GP_CPU_THRESHOLD 16
		1636
		1637	static int rcu_normal_wake_from_gp = -1;
1630	module_param(rcu_normal_wake_from_gp, int, 0644);	1638	module_param(rcu_normal_wake_from_gp, int, 0644);
1631	static struct workqueue_struct *sync_wq;	1639	static struct workqueue_struct *sync_wq;
1632		1640
@@ -1829,6 +1837,18 @@ static noinline_for_stack bool rcu_gp_init(void) · 196 unmodified lines
1829	start_new_poll = rcu_sr_normal_gp_init();	1837	start_new_poll = rcu_sr_normal_gp_init();
1830	/* Record GP times before starting GP, hence rcu_seq_start(). */	1838	/* Record GP times before starting GP, hence rcu_seq_start(). */
1831	old_gp_seq = rcu_state.gp_seq;	1839	old_gp_seq = rcu_state.gp_seq;
		1840	/*
		1841	* Critical ordering: rcu_seq_start() must happen BEFORE the CPU hotplug
		1842	* scan below. Otherwise we risk a race where a newly onlining CPU could
		1843	* be missed by the current grace period, potentially leading to
		1844	* use-after-free errors. For a detailed explanation of this race, see
		1845	* Documentation/RCU/Design/Requirements/Requirements.rst in the
		1846	* "Hotplug CPU" section.
		1847	*
		1848	* Also note that the root rnp's gp_seq is kept separate from, and lags,
		1849	* the rcu_state's gp_seq, for a reason. See the Quick-Quiz on
		1850	* Single-node systems for more details (in Data-Structures.rst).
		1851	*/
1832	rcu_seq_start(&rcu_state.gp_seq);	1852	rcu_seq_start(&rcu_state.gp_seq);
1833	/* Ensure that rcu_seq_done_exact() guardband doesn't give false positives. */	1853	/* Ensure that rcu_seq_done_exact() guardband doesn't give false positives. */
1834	WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) &&	1854	WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) &&
@@ -1865,6 +1885,10 @@ static noinline_for_stack bool rcu_gp_init(void) · 30 unmodified lines
1865	/* Exclude CPU hotplug operations. */	1885	/* Exclude CPU hotplug operations. */
1866	rcu_for_each_leaf_node(rnp) {	1886	rcu_for_each_leaf_node(rnp) {
1867	local_irq_disable();	1887	local_irq_disable();
		1888	/*
		1889	* Serialize with CPU offline. See Requirements.rst > Hotplug CPU >
		1890	* Concurrent Quiescent State Reporting for Offline CPUs.
		1891	*/
1868	arch_spin_lock(&rcu_state.ofl_lock);	1892	arch_spin_lock(&rcu_state.ofl_lock);
1869	raw_spin_lock_rcu_node(rnp);	1893	raw_spin_lock_rcu_node(rnp);
1870	if (rnp->qsmaskinit == rnp->qsmaskinitnext &&	1894	if (rnp->qsmaskinit == rnp->qsmaskinitnext &&
@@ -1939,7 +1963,12 @@ static noinline_for_stack bool rcu_gp_init(void) · 68 unmodified lines
1939	trace_rcu_grace_period_init(rcu_state.name, rnp->gp_seq,	1963	trace_rcu_grace_period_init(rcu_state.name, rnp->gp_seq,
1940	rnp->level, rnp->grplo,	1964	rnp->level, rnp->grplo,
1941	rnp->grphi, rnp->qsmask);	1965	rnp->grphi, rnp->qsmask);
1942	/* Quiescent states for tasks on any now-offline CPUs. */	1966	/*
		1967	* Quiescent states for tasks on any now-offline CPUs. Since we
		1968	* released the ofl and rnp lock before this loop, CPUs might
		1969	* have gone offline and we have to report QS on their behalf.
		1970	* See Requirements.rst > Hotplug CPU > Concurrent QS Reporting.
		1971	*/
1943	mask = rnp->qsmask & ~rnp->qsmaskinitnext;	1972	mask = rnp->qsmask & ~rnp->qsmaskinitnext;
1944	rnp->rcu_gp_init_mask = mask;	1973	rnp->rcu_gp_init_mask = mask;
1945	if ((mask \|\| rnp->wait_blkd_tasks) && rcu_is_leaf_node(rnp))	1974	if ((mask \|\| rnp->wait_blkd_tasks) && rcu_is_leaf_node(rnp))
@@ -3243,7 +3272,7 @@ static void synchronize_rcu_normal(void) · 1297 unmodified lines
3243		3272
3244	trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request"));	3273	trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request"));
3245		3274
3246	if (!READ_ONCE(rcu_normal_wake_from_gp)) {	3275	if (READ_ONCE(rcu_normal_wake_from_gp) < 1) {
3247	wait_rcu_gp(call_rcu_hurry);	3276	wait_rcu_gp(call_rcu_hurry);
3248	goto trace_complete_out;	3277	goto trace_complete_out;
3249	}	3278	}
@@ -4268,7 +4297,6 @@ int rcutree_online_cpu(unsigned int cpu) · 1018 unmodified lines
4268	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	4297	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
4269	if (rcu_scheduler_active == RCU_SCHEDULER_INACTIVE)	4298	if (rcu_scheduler_active == RCU_SCHEDULER_INACTIVE)
4270	return 0; /* Too early in boot for scheduler work. */	4299	return 0; /* Too early in boot for scheduler work. */
4271	sync_sched_exp_online_cleanup(cpu);
4272		4300
4273	// Stop-machine done, so allow nohz_full to disable tick.	4301	// Stop-machine done, so allow nohz_full to disable tick.
4274	tick_dep_clear(TICK_DEP_BIT_RCU);	4302	tick_dep_clear(TICK_DEP_BIT_RCU);
@@ -4358,6 +4386,12 @@ void rcutree_report_cpu_dead(void) · 83 unmodified lines
4358	* may introduce a new READ-side while it is actually off the QS masks.	4386	* may introduce a new READ-side while it is actually off the QS masks.
4359	*/	4387	*/
4360	lockdep_assert_irqs_disabled();	4388	lockdep_assert_irqs_disabled();
		4389	/*
		4390	* CPUHP_AP_SMPCFD_DYING was the last call for rcu_exp_handler() execution.
		4391	* The requested QS must have been reported on the last context switch
		4392	* from stop machine to idle.
		4393	*/
		4394	WARN_ON_ONCE(rdp->cpu_no_qs.b.exp);
4361	// Do any dangling deferred wakeups.	4395	// Do any dangling deferred wakeups.
4362	do_nocb_deferred_wakeup(rdp);	4396	do_nocb_deferred_wakeup(rdp);
4363		4397
@@ -4365,6 +4399,13 @@ void rcutree_report_cpu_dead(void) · 1 unmodified line
4365		4399
4366	/* Remove outgoing CPU from mask in the leaf rcu_node structure. */	4400	/* Remove outgoing CPU from mask in the leaf rcu_node structure. */
4367	mask = rdp->grpmask;	4401	mask = rdp->grpmask;
		4402
		4403	/*
		4404	* Hold the ofl_lock and rnp lock to avoid races between CPU going
		4405	* offline and doing a QS report (as below), versus rcu_gp_init().
		4406	* See Requirements.rst > Hotplug CPU > Concurrent QS Reporting section
		4407	* for more details.
		4408	*/
4368	arch_spin_lock(&rcu_state.ofl_lock);	4409	arch_spin_lock(&rcu_state.ofl_lock);
4369	raw_spin_lock_irqsave_rcu_node(rnp, flags); /* Enforce GP memory-order guarantee. */	4410	raw_spin_lock_irqsave_rcu_node(rnp, flags); /* Enforce GP memory-order guarantee. */
4370	rdp->rcu_ofl_gp_seq = READ_ONCE(rcu_state.gp_seq);	4411	rdp->rcu_ofl_gp_seq = READ_ONCE(rcu_state.gp_seq);
@@ -4375,6 +4416,7 @@ void rcutree_report_cpu_dead(void) · 4 unmodified lines
4375	rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);	4416	rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);
4376	raw_spin_lock_irqsave_rcu_node(rnp, flags);	4417	raw_spin_lock_irqsave_rcu_node(rnp, flags);
4377	}	4418	}
		4419	/* Clear from ->qsmaskinitnext to mark offline. */
4378	WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext & ~mask);	4420	WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext & ~mask);
4379	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	4421	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
4380	arch_spin_unlock(&rcu_state.ofl_lock);	4422	arch_spin_unlock(&rcu_state.ofl_lock);
@@ -4847,6 +4889,12 @@ void __init rcu_init(void) · 466 unmodified lines
4847	sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM, 0);	4889	sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM, 0);
4848	WARN_ON(!sync_wq);	4890	WARN_ON(!sync_wq);
4849		4891
		4892	/* Respect if explicitly disabled via a boot parameter. */
		4893	if (rcu_normal_wake_from_gp < 0) {
		4894	if (num_possible_cpus() <= WAKE_FROM_GP_CPU_THRESHOLD)
		4895	rcu_normal_wake_from_gp = 1;
		4896	}
		4897
4850	/* Fill in default value for rcutree.qovld boot parameter. */	4898	/* Fill in default value for rcutree.qovld boot parameter. */
4851	/* -After- the rcu_node ->lock fields are initialized! */	4899	/* -After- the rcu_node ->lock fields are initialized! */
4852	if (qovld < 0)	4900	if (qovld < 0)

kernel/rcu/tree.h+12−1

 	unsigned long   jiffies;	/* Track jiffies value */
 };
+/*
+ * An IRQ work (deferred_qs_iw) is used by RCU to get the scheduler's attention.
+ * to report quiescent states at the soonest possible time.
+ * The request can be in one of the following states:
+ * - DEFER_QS_IDLE: An IRQ work is yet to be scheduled.
+ * - DEFER_QS_PENDING: An IRQ work was scheduled but either not yet run, or it
+ *                     ran and we still haven't reported a quiescent state.
+ */
+#define DEFER_QS_IDLE		0
+#define DEFER_QS_PENDING	1
 /* Per-CPU data for read-copy update. */
 struct rcu_data {
 	/* 1) quiescent-state and grace-period handling : */
 					/*  during and after the last grace */
 					/* period it is aware of. */
 	struct irq_work defer_qs_iw;	/* Obtain later scheduler attention. */
-	bool defer_qs_iw_pending;	/* Scheduler attention pending? */
+	int defer_qs_iw_pending;	/* Scheduler attention pending? */
 	struct work_struct strict_work;	/* Schedule readers for strict GPs. */
 	/* 2) batch handling */

kernel/rcu/tree_exp.h+9−50

@@ -141,6 +141,13 @@ static void __maybe_unused sync_exp_reset_tree(void) · 140 unmodified lines
141	raw_spin_lock_irqsave_rcu_node(rnp, flags);	141	raw_spin_lock_irqsave_rcu_node(rnp, flags);
142	WARN_ON_ONCE(rnp->expmask);	142	WARN_ON_ONCE(rnp->expmask);
143	WRITE_ONCE(rnp->expmask, rnp->expmaskinit);	143	WRITE_ONCE(rnp->expmask, rnp->expmaskinit);
		144	/*
		145	* Need to wait for any blocked tasks as well. Note that
		146	* additional blocking tasks will also block the expedited GP
		147	* until such time as the ->expmask bits are cleared.
		148	*/
		149	if (rcu_is_leaf_node(rnp) && rcu_preempt_has_tasks(rnp))
		150	WRITE_ONCE(rnp->exp_tasks, rnp->blkd_tasks.next);
144	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	151	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
145	}	152	}
146	}	153	}
@@ -393,13 +400,6 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) · 246 unmodified lines
393	}	400	}
394	mask_ofl_ipi = rnp->expmask & ~mask_ofl_test;	401	mask_ofl_ipi = rnp->expmask & ~mask_ofl_test;
395		402
396	/*
397	* Need to wait for any blocked tasks as well. Note that
398	* additional blocking tasks will also block the expedited GP
399	* until such time as the ->expmask bits are cleared.
400	*/
401	if (rcu_preempt_has_tasks(rnp))
402	WRITE_ONCE(rnp->exp_tasks, rnp->blkd_tasks.next);
403	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);	403	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
404		404
405	/* IPI the remaining CPUs for expedited quiescent state. */	405	/* IPI the remaining CPUs for expedited quiescent state. */
@@ -751,12 +751,8 @@ static void rcu_exp_handler(void *unused) · 345 unmodified lines
751	struct task_struct *t = current;	751	struct task_struct *t = current;
752		752
753	/*	753	/*
754	* First, is there no need for a quiescent state from this CPU,	754	* WARN if the CPU is unexpectedly already looking for a
755	* or is this CPU already looking for a quiescent state for the	755	* QS or has already reported one.
756	* current grace period? If either is the case, just leave.
757	* However, this should not happen due to the preemptible
758	* sync_sched_exp_online_cleanup() implementation being a no-op,
759	* so warn if this does happen.
760	*/	756	*/
761	ASSERT_EXCLUSIVE_WRITER_SCOPED(rdp->cpu_no_qs.b.exp);	757	ASSERT_EXCLUSIVE_WRITER_SCOPED(rdp->cpu_no_qs.b.exp);
762	if (WARN_ON_ONCE(!(READ_ONCE(rnp->expmask) & rdp->grpmask) \|\|	758	if (WARN_ON_ONCE(!(READ_ONCE(rnp->expmask) & rdp->grpmask) \|\|
@@ -803,11 +799,6 @@ static void rcu_exp_handler(void *unused) · 40 unmodified lines
803	WARN_ON_ONCE(1);	799	WARN_ON_ONCE(1);
804	}	800	}
805		801
806	/* PREEMPTION=y, so no PREEMPTION=n expedited grace period to clean up after. */
807	static void sync_sched_exp_online_cleanup(int cpu)
808	{
809	}
810
811	/*	802	/*
812	* Scan the current list of tasks blocked within RCU read-side critical	803	* Scan the current list of tasks blocked within RCU read-side critical
813	* sections, printing out the tid of each that is blocking the current	804	* sections, printing out the tid of each that is blocking the current
@@ -885,38 +876,6 @@ static void rcu_exp_handler(void *unused) · 71 unmodified lines
885	rcu_exp_need_qs();	876	rcu_exp_need_qs();
886	}	877	}
887		878
888	/* Send IPI for expedited cleanup if needed at end of CPU-hotplug operation. */
889	static void sync_sched_exp_online_cleanup(int cpu)
890	{
891	unsigned long flags;
892	int my_cpu;
893	struct rcu_data *rdp;
894	int ret;
895	struct rcu_node *rnp;
896
897	rdp = per_cpu_ptr(&rcu_data, cpu);
898	rnp = rdp->mynode;
899	my_cpu = get_cpu();
900	/* Quiescent state either not needed or already requested, leave. */
901	if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) \|\|
902	READ_ONCE(rdp->cpu_no_qs.b.exp)) {
903	put_cpu();
904	return;
905	}
906	/* Quiescent state needed on current CPU, so set it up locally. */
907	if (my_cpu == cpu) {
908	local_irq_save(flags);
909	rcu_exp_need_qs();
910	local_irq_restore(flags);
911	put_cpu();
912	return;
913	}
914	/* Quiescent state needed on some other CPU, send IPI. */
915	ret = smp_call_function_single(cpu, rcu_exp_handler, NULL, 0);
916	put_cpu();
917	WARN_ON_ONCE(ret);
918	}
919
920	/*	879	/*
921	* Because preemptible RCU does not exist, we never have to check for	880	* Because preemptible RCU does not exist, we never have to check for
922	* tasks blocked within RCU read-side critical sections that are	881	* tasks blocked within RCU read-side critical sections that are

kernel/rcu/tree_nocb.h+6−4

 	 * callback storms, no need to wake up too early.
 	 */
 	if (waketype == RCU_NOCB_WAKE_LAZY &&
-	    rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) {
+	    rdp_gp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) {
 		mod_timer(&rdp_gp->nocb_timer, jiffies + rcu_get_jiffies_lazy_flush());
 		WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
 	} else if (waketype == RCU_NOCB_WAKE_BYPASS) {
 static int rcu_nocb_rdp_offload(struct rcu_data *rdp)
 {
 	int wake_gp;
-	struct rcu_data *rdp_gp = rdp->nocb_gp_rdp;
 	WARN_ON_ONCE(cpu_online(rdp->cpu));
 	/*
 	if (!rdp->nocb_gp_rdp)
 		return -EINVAL;
-	if (WARN_ON_ONCE(!rdp_gp->nocb_gp_kthread))
+	if (WARN_ON_ONCE(!rdp->nocb_gp_kthread))
 		return -EINVAL;
 	pr_info("Offloading %d\n", rdp->cpu);
 	wake_gp = rcu_nocb_queue_toggle_rdp(rdp);
 	if (wake_gp)
-		wake_up_process(rdp_gp->nocb_gp_kthread);
+		wake_up_process(rdp->nocb_gp_kthread);
 	swait_event_exclusive(rdp->nocb_state_wq,
 			      rcu_nocb_rdp_offload_wait_cond(rdp));
 	if (rdp->nocb_gp_rdp == rdp)
 		show_rcu_nocb_gp_state(rdp);
+	if (!rcu_segcblist_is_offloaded(&rdp->cblist))
+		return;
 	nocb_next_rdp = list_next_or_null_rcu(&rdp->nocb_gp_rdp->nocb_head_rdp,
 					      &rdp->nocb_entry_rdp,
 					      typeof(*rdp),

kernel/rcu/tree_plugin.h+102−20

@@ -486,13 +486,16 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 485 unmodified lines
486	struct rcu_node *rnp;	486	struct rcu_node *rnp;
487	union rcu_special special;	487	union rcu_special special;
488		488
		489	rdp = this_cpu_ptr(&rcu_data);
		490	if (rdp->defer_qs_iw_pending == DEFER_QS_PENDING)
		491	rdp->defer_qs_iw_pending = DEFER_QS_IDLE;
		492
489	/*	493	/*
490	* If RCU core is waiting for this CPU to exit its critical section,	494	* If RCU core is waiting for this CPU to exit its critical section,
491	* report the fact that it has exited. Because irqs are disabled,	495	* report the fact that it has exited. Because irqs are disabled,
492	* t->rcu_read_unlock_special cannot change.	496	* t->rcu_read_unlock_special cannot change.
493	*/	497	*/
494	special = t->rcu_read_unlock_special;	498	special = t->rcu_read_unlock_special;
495	rdp = this_cpu_ptr(&rcu_data);
496	if (!special.s && !rdp->cpu_no_qs.b.exp) {	499	if (!special.s && !rdp->cpu_no_qs.b.exp) {
497	local_irq_restore(flags);	500	local_irq_restore(flags);
498	return;	501	return;
@@ -534,7 +537,6 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 35 unmodified lines
534	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq &&	537	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq &&
535	(!empty_norm \|\| rnp->qsmask));	538	(!empty_norm \|\| rnp->qsmask));
536	empty_exp = sync_rcu_exp_done(rnp);	539	empty_exp = sync_rcu_exp_done(rnp);
537	smp_mb(); /* ensure expedited fastpath sees end of RCU c-s. */
538	np = rcu_next_node_entry(t, rnp);	540	np = rcu_next_node_entry(t, rnp);
539	list_del_init(&t->rcu_node_entry);	541	list_del_init(&t->rcu_node_entry);
540	t->rcu_blocked_node = NULL;	542	t->rcu_blocked_node = NULL;
@@ -624,10 +626,98 @@ notrace void rcu_preempt_deferred_qs(struct task_struct *t) · 83 unmodified lines
624	*/	626	*/
625	static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp)	627	static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp)
626	{	628	{
		629	unsigned long flags;
627	struct rcu_data *rdp;	630	struct rcu_data *rdp;
628		631
629	rdp = container_of(iwp, struct rcu_data, defer_qs_iw);	632	rdp = container_of(iwp, struct rcu_data, defer_qs_iw);
630	rdp->defer_qs_iw_pending = false;	633	local_irq_save(flags);
		634
		635	/*
		636	* If the IRQ work handler happens to run in the middle of RCU read-side
		637	* critical section, it could be ineffective in getting the scheduler's
		638	* attention to report a deferred quiescent state (the whole point of the
		639	* IRQ work). For this reason, requeue the IRQ work.
		640	*
		641	* Basically, we want to avoid following situation:
		642	* 1. rcu_read_unlock() queues IRQ work (state -> DEFER_QS_PENDING)
		643	* 2. CPU enters new rcu_read_lock()
		644	* 3. IRQ work runs but cannot report QS due to rcu_preempt_depth() > 0
		645	* 4. rcu_read_unlock() does not re-queue work (state still PENDING)
		646	* 5. Deferred QS reporting does not happen.
		647	*/
		648	if (rcu_preempt_depth() > 0)
		649	WRITE_ONCE(rdp->defer_qs_iw_pending, DEFER_QS_IDLE);
		650
		651	local_irq_restore(flags);
		652	}
		653
		654	/*
		655	* Check if expedited grace period processing during unlock is needed.
		656	*
		657	* This function determines whether expedited handling is required based on:
		658	* 1. Task blocking an expedited grace period (based on a heuristic, could be
		659	* false-positive, see below.)
		660	* 2. CPU participating in an expedited grace period
		661	* 3. Strict grace period mode requiring expedited handling
		662	* 4. RCU priority deboosting needs when interrupts were disabled
		663	*
		664	* @t: The task being checked
		665	* @rdp: The per-CPU RCU data
		666	* @rnp: The RCU node for this CPU
		667	* @irqs_were_disabled: Whether interrupts were disabled before rcu_read_unlock()
		668	*
		669	* Returns true if expedited processing of the rcu_read_unlock() is needed.
		670	*/
		671	static bool rcu_unlock_needs_exp_handling(struct task_struct *t,
		672	struct rcu_data *rdp,
		673	struct rcu_node *rnp,
		674	bool irqs_were_disabled)
		675	{
		676	/*
		677	* Check if this task is blocking an expedited grace period. If the
		678	* task was preempted within an RCU read-side critical section and is
		679	* on the expedited grace period blockers list (exp_tasks), we need
		680	* expedited handling to unblock the expedited GP. This is not an exact
		681	* check because 't' might not be on the exp_tasks list at all - its
		682	* just a fast heuristic that can be false-positive sometimes.
		683	*/
		684	if (t->rcu_blocked_node && READ_ONCE(t->rcu_blocked_node->exp_tasks))
		685	return true;
		686
		687	/*
		688	* Check if this CPU is participating in an expedited grace period.
		689	* The expmask bitmap tracks which CPUs need to check in for the
		690	* current expedited GP. If our CPU's bit is set, we need expedited
		691	* handling to help complete the expedited GP.
		692	*/
		693	if (rdp->grpmask & READ_ONCE(rnp->expmask))
		694	return true;
		695
		696	/*
		697	* In CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels, all grace periods
		698	* are treated as short for testing purposes even if that means
		699	* disturbing the system more. Check if either:
		700	* - This CPU has not yet reported a quiescent state, or
		701	* - This task was preempted within an RCU critical section
		702	* In either case, require expedited handling for strict GP mode.
		703	*/
		704	if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) &&
		705	((rdp->grpmask & READ_ONCE(rnp->qsmask)) \|\| t->rcu_blocked_node))
		706	return true;
		707
		708	/*
		709	* RCU priority boosting case: If a task is subject to RCU priority
		710	* boosting and exits an RCU read-side critical section with interrupts
		711	* disabled, we need expedited handling to ensure timely deboosting.
		712	* Without this, a low-priority task could incorrectly run at high
		713	* real-time priority for an extended period degrading real-time
		714	* responsiveness. This applies to all CONFIG_RCU_BOOST=y kernels,
		715	* not just to PREEMPT_RT.
		716	*/
		717	if (IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled && t->rcu_blocked_node)
		718	return true;
		719
		720	return false;
631	}	721	}
632		722
633	/*	723	/*
@@ -649,18 +739,14 @@ static void rcu_read_unlock_special(struct task_struct *t) · 15 unmodified lines
649	local_irq_save(flags);	739	local_irq_save(flags);
650	irqs_were_disabled = irqs_disabled_flags(flags);	740	irqs_were_disabled = irqs_disabled_flags(flags);
651	if (preempt_bh_were_disabled \|\| irqs_were_disabled) {	741	if (preempt_bh_were_disabled \|\| irqs_were_disabled) {
652	bool expboost; // Expedited GP in flight or possible boosting.	742	bool needs_exp; // Expedited handling needed.
653	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);	743	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
654	struct rcu_node *rnp = rdp->mynode;	744	struct rcu_node *rnp = rdp->mynode;
655		745
656	expboost = (t->rcu_blocked_node && READ_ONCE(t->rcu_blocked_node->exp_tasks)) \|\|	746	needs_exp = rcu_unlock_needs_exp_handling(t, rdp, rnp, irqs_were_disabled);
657	(rdp->grpmask & READ_ONCE(rnp->expmask)) \|\|	747
658	(IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) &&
659	((rdp->grpmask & READ_ONCE(rnp->qsmask)) \|\| t->rcu_blocked_node)) \|\|
660	(IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled &&
661	t->rcu_blocked_node);
662	// Need to defer quiescent state until everything is enabled.	748	// Need to defer quiescent state until everything is enabled.
663	if (use_softirq && (in_hardirq() \|\| (expboost && !irqs_were_disabled))) {	749	if (use_softirq && (in_hardirq() \|\| (needs_exp && !irqs_were_disabled))) {
664	// Using softirq, safe to awaken, and either the	750	// Using softirq, safe to awaken, and either the
665	// wakeup is free or there is either an expedited	751	// wakeup is free or there is either an expedited
666	// GP in flight or a potential need to deboost.	752	// GP in flight or a potential need to deboost.
@@ -673,17 +759,13 @@ static void rcu_read_unlock_special(struct task_struct *t) · 6 unmodified lines
673	set_tsk_need_resched(current);	759	set_tsk_need_resched(current);
674	set_preempt_need_resched();	760	set_preempt_need_resched();
675	if (IS_ENABLED(CONFIG_IRQ_WORK) && irqs_were_disabled &&	761	if (IS_ENABLED(CONFIG_IRQ_WORK) && irqs_were_disabled &&
676	expboost && !rdp->defer_qs_iw_pending && cpu_online(rdp->cpu)) {	762	needs_exp && rdp->defer_qs_iw_pending != DEFER_QS_PENDING &&
		763	cpu_online(rdp->cpu)) {
677	// Get scheduler to re-evaluate and call hooks.	764	// Get scheduler to re-evaluate and call hooks.
678	// If !IRQ_WORK, FQS scan will eventually IPI.	765	// If !IRQ_WORK, FQS scan will eventually IPI.
679	if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) &&	766	rdp->defer_qs_iw =
680	IS_ENABLED(CONFIG_PREEMPT_RT))	767	IRQ_WORK_INIT_HARD(rcu_preempt_deferred_qs_handler);
681	rdp->defer_qs_iw = IRQ_WORK_INIT_HARD(	768	rdp->defer_qs_iw_pending = DEFER_QS_PENDING;
682	rcu_preempt_deferred_qs_handler);
683	else
684	init_irq_work(&rdp->defer_qs_iw,
685	rcu_preempt_deferred_qs_handler);
686	rdp->defer_qs_iw_pending = true;
687	irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);	769	irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
688	}	770	}
689	}	771	}

kernel/rcu/tree_stall.h+1−2

 	for_each_possible_cpu(cpu) {
 		rdp = per_cpu_ptr(&rcu_data, cpu);
 		cbs += data_race(READ_ONCE(rdp->n_cbs_invoked));
-		if (rcu_segcblist_is_offloaded(&rdp->cblist))
+		show_rcu_nocb_state(rdp);
-			show_rcu_nocb_state(rdp);
 	}
 	pr_info("RCU callbacks invoked since boot: %lu\n", cbs);
 	show_rcu_tasks_gp_kthreads();

scripts/checkpatch.pl+0−2

@@ -857,8 +857,6 @@ our %deprecated_apis = ( · 856 unmodified lines
857	"kunmap" => "kunmap_local",	857	"kunmap" => "kunmap_local",
858	"kmap_atomic" => "kmap_local_page",	858	"kmap_atomic" => "kmap_local_page",
859	"kunmap_atomic" => "kunmap_local",	859	"kunmap_atomic" => "kunmap_local",
860	"srcu_read_lock_lite" => "srcu_read_lock_fast",
861	"srcu_read_unlock_lite" => "srcu_read_unlock_fast",
862	);	860	);
863		861
864	#Create a search pattern for all these strings to speed up a loop below	862	#Create a search pattern for all these strings to speed up a loop below

tools/testing/selftests/rcutorture/bin/kvm-build.sh+1−1

@@ -44,7 +44,7 @@ fi · 43 unmodified lines
44	ncpus="`getconf _NPROCESSORS_ONLN`"	44	ncpus="`getconf _NPROCESSORS_ONLN`"
45	make -j$((2 * ncpus)) $TORTURE_KMAKE_ARG > $resdir/Make.out 2>&1	45	make -j$((2 * ncpus)) $TORTURE_KMAKE_ARG > $resdir/Make.out 2>&1
46	retval=$?	46	retval=$?
47	if test $retval -ne 0 \|\| grep "rcu[^/]*": < $resdir/Make.out \| grep -E -q "Stop\|Error\|error:\|warning:" \|\| grep -E -q "Stop\|Error\|error:" < $resdir/Make.out	47	if test $retval -ne 0 \|\| grep "rcu[^/]*": < $resdir/Make.out \| grep -E -q "Stop\|ERROR\|Error\|error:\|warning:" \|\| grep -E -q "Stop\|ERROR\|Error\|error:" < $resdir/Make.out
48	then	48	then
49	echo Kernel build error	49	echo Kernel build error
50	grep -E "Stop\|Error\|error:\|warning:" < $resdir/Make.out	50	grep -E "Stop\|Error\|error:\|warning:" < $resdir/Make.out

tools/testing/selftests/rcutorture/bin/kvm.sh+2−13

 		fi
 		;;
 	--kconfig|--kconfigs)
-		checkarg --kconfig "(Kconfig options)" $# "$2" '^\(#CHECK#\)\?CONFIG_[A-Z0-9_]\+=\([ynm]\|[0-9]\+\|"[^"]*"\)\( \(#CHECK#\)\?CONFIG_[A-Z0-9_]\+=\([ynm]\|[0-9]\+\|"[^"]*"\)\)*$' '^error$'
+		checkarg --kconfig "(Kconfig options)" $# "$2" '^\(#CHECK#\)\?CONFIG_[A-Z0-9_]\+=\([ynm]\|[0-9]\+\|"[^"]*"\)\( \+\(#CHECK#\)\?CONFIG_[A-Z0-9_]\+=\([ynm]\|[0-9]\+\|"[^"]*"\)\)* *$' '^error$'
 		TORTURE_KCONFIG_ARG="`echo "$TORTURE_KCONFIG_ARG $2" | sed -e 's/^ *//' -e 's/ *$//'`"
 		shift
 		;;
 touch $resdir/$ds/log
 echo $scriptname $args >> $resdir/$ds/log
 echo ${TORTURE_SUITE} > $resdir/$ds/torture_suite
-echo Build directory: `pwd` > $resdir/$ds/testid.txt
+mktestid.sh $resdir/$ds
-if test -d .git
-then
-	echo Current commit: `git rev-parse HEAD` >> $resdir/$ds/testid.txt
-	echo >> $resdir/$ds/testid.txt
-	echo ' ---' Output of "'"git status"'": >> $resdir/$ds/testid.txt
-	git status >> $resdir/$ds/testid.txt
-	echo >> $resdir/$ds/testid.txt
-	echo >> $resdir/$ds/testid.txt
-	echo ' ---' Output of "'"git diff HEAD"'": >> $resdir/$ds/testid.txt
-	git diff HEAD >> $resdir/$ds/testid.txt
-fi
 ___EOF___
 kvm-assign-cpus.sh /sys/devices/system/node > $T/cpuarray.awk
 kvm-get-cpus-script.sh $T/cpuarray.awk $T/dumpbatches.awk

tools/testing/selftests/rcutorture/bin/mktestid.sh+29−0

@@ -0,0 +1,29 @@
	1	#!/bin/bash
	2	# SPDX-License-Identifier: GPL-2.0+
	3	#
	4	# Create a testid.txt file in the specified directory.
	5	#
	6	# Usage: mktestid.sh dirpath
	7	#
	8	# Copyright (C) Meta Platforms, Inc. 2025
	9	#
	10	# Author: Paul E. McKenney <paulmck@kernel.org>
	11
	12	resdir="$1"
	13	if test -z "${resdir}" \|\| ! test -d "${resdir}" \|\| ! test -w "${resdir}"
	14	then
	15	echo Path '"'${resdir}'"' not writeable directory, no ${resdir}/testid.txt.
	16	exit 1
	17	fi
	18	echo Build directory: `pwd` > ${resdir}/testid.txt
	19	if test -d .git
	20	then
	21	echo Current commit: `git rev-parse HEAD` >> ${resdir}/testid.txt
	22	echo >> ${resdir}/testid.txt
	23	echo ' ---' Output of "'"git status"'": >> ${resdir}/testid.txt
	24	git status >> ${resdir}/testid.txt
	25	echo >> ${resdir}/testid.txt
	26	echo >> ${resdir}/testid.txt
	27	echo ' ---' Output of "'"git diff HEAD"'": >> ${resdir}/testid.txt
	28	git diff HEAD >> ${resdir}/testid.txt
	29	fi

tools/testing/selftests/rcutorture/bin/torture.sh+62−16

 	VERBOSE_BATCH_CPUS=0
 fi
+# Machine architecture?  ("uname -p" is said to be less portable.)1
+thisarch="`uname -m`"
+if test "${thisarch}" = aarch64
+then
+	ifnotaarch64=no
+else
+	ifnotaarch64=yes
+fi
 # Configurations/scenarios.
 configs_rcutorture=
 configs_locktorture=
 explicit_normal=no
 do_kasan=yes
 do_kcsan=no
-do_clocksourcewd=yes
+do_clocksourcewd="${ifnotaarch64}"
 do_rt=yes
-do_rcutasksflavors=yes
+do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
 do_srcu_lockdep=yes
 do_rcu_rust=no
 		;;
 	--do-all|--doall)
 		do_allmodconfig=yes
-		do_rcutasksflavor=yes
+		do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
 		do_rcutorture=yes
 		do_locktorture=yes
 		do_scftorture=yes
 		explicit_normal=no
 		do_kasan=yes
 		do_kcsan=yes
-		do_clocksourcewd=yes
+		do_clocksourcewd="${ifnotaarch64}"
 		do_srcu_lockdep=yes
 		;;
 	--do-allmodconfig|--do-no-allmodconfig|--no-allmodconfig)
 	configs_rcutorture=CFLIST
 fi
 duration_rcutorture=$((duration_base*duration_rcutorture_frac/10))
-if test "$duration_rcutorture" -eq 0
+if test "$duration_rcutorture" -eq 0 && test "$do_locktorture" = "yes"
 then
 	echo " --- Zero time for rcutorture, disabling" | tee -a $T/log
 	do_rcutorture=no
 	configs_locktorture=CFLIST
 fi
 duration_locktorture=$((duration_base*duration_locktorture_frac/10))
-if test "$duration_locktorture" -eq 0
+if test "$duration_locktorture" -eq 0 && test "$do_locktorture" = "yes"
 then
 	echo " --- Zero time for locktorture, disabling" | tee -a $T/log
 	do_locktorture=no
 	configs_scftorture=CFLIST
 fi
 duration_scftorture=$((duration_base*duration_scftorture_frac/10))
-if test "$duration_scftorture" -eq 0
+if test "$duration_scftorture" -eq 0 && test "$do_scftorture" = "yes"
 then
 	echo " --- Zero time for scftorture, disabling" | tee -a $T/log
 	do_scftorture=no
 fi
+# CONFIG_EXPERT=y is currently required for arm64 KCSAN runs.
+kcsan_expert=
+if test "${thisarch}" = aarch64
+then
+	kcsan_expert="CONFIG_EXPERT=y"
+fi
 touch $T/failures
 touch $T/successes
 	then
 		curflavor=$flavor
 		torture_one "$@"
-		mv $T/last-resdir $T/last-resdir-nodebug || :
+		if test -e $T/last-resdir
+		then
+			mv $T/last-resdir $T/last-resdir-nodebug || :
+		fi
 	fi
 	if test "$do_kasan" = "yes"
 	then
 		curflavor=${flavor}-kasan
 		torture_one "$@" --kasan
-		mv $T/last-resdir $T/last-resdir-kasan || :
+		if test -e $T/last-resdir
+		then
+			mv $T/last-resdir $T/last-resdir-kasan || :
+		fi
 	fi
 	if test "$do_kcsan" = "yes"
 	then
 			kcsan_kmake_tag="--kmake-args"
 			cur_kcsan_kmake_args="$kcsan_kmake_args"
 		fi
-		torture_one "$@" --kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y" $kcsan_kmake_tag $cur_kcsan_kmake_args --kcsan
+		chk_rdr_state=
-		mv $T/last-resdir $T/last-resdir-kcsan || :
+		if test "${flavor}" = rcutorture
+		then
+			chk_rdr_state="CONFIG_RCU_TORTURE_TEST_CHK_RDR_STATE=y"
+		fi
+		torture_one "$@" --kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y ${kcsan_expert} ${chk_rdr_state}" $kcsan_kmake_tag $cur_kcsan_kmake_args --kcsan
+		if test -e $T/last-resdir
+		then
+			mv $T/last-resdir $T/last-resdir-kcsan || :
+		fi
 	fi
 }
 	echo " --- allmodconfig:" Start `date` | tee -a $T/log
 	amcdir="tools/testing/selftests/rcutorture/res/$ds/allmodconfig"
 	mkdir -p "$amcdir"
+	mktestid.sh "$amcdir"
 	echo " --- make clean" | tee $amcdir/log > "$amcdir/Make.out" 2>&1
 	make -j$MAKE_ALLOTED_CPUS clean >> "$amcdir/Make.out" 2>&1
 	retcode=$?
 		make -j$MAKE_ALLOTED_CPUS >> "$amcdir/Make.out" 2>&1
 		retcode="$?"
 		echo $retcode > "$amcdir/Make.exitcode"
+		if grep -E -q "Stop|ERROR|Error|error:|warning:" < "$amcdir/Make.out"
+		then
+			retcode=99
+		fi
 		buildphase='"make"'
 	fi
 	if test "$retcode" -eq 0
 	echo " --- do-rcu-rust:" Start `date` | tee -a $T/log
 	rrdir="tools/testing/selftests/rcutorture/res/$ds/results-rcu-rust"
 	mkdir -p "$rrdir"
+	mktestid.sh "$rrdir"
 	echo " --- make LLVM=1 rustavailable " | tee -a $rrdir/log > $rrdir/rustavailable.out
 	make LLVM=1 rustavailable > $T/rustavailable.out 2>&1
 	retcode=$?
 echo FAILURES: | tee -a $T/log
 if test -s "$T/failures"
 then
-	awk < "$T/failures" -v sq="'" '{ print "echo " sq $0 sq; print "sed -e " sq "1,/^ --- .* Test summary:$/d" sq " " $2 "/log | grep Summary: | sed -e " sq "s/^[^S]*/  /" sq; }' | sh | tee -a $T/log | tee "$T/failuresum"
+	awk < "$T/failures" -v sq="'" '
+	{
+		print "echo " sq $0 sq;
+		if ($2 != "")
+			print "sed -e " sq "1,/^ --- .* Test summary:$/d" sq " " $2 "/log | grep Summary: | sed -e " sq "s/^[^S]*/  /" sq;
+		else
+			print "echo " sq "  " sq "Run failed to produce results directory.";
+	}' | sh | tee -a $T/log | tee "$T/failuresum"
 	nfailures="`wc -l "$T/failures" | awk '{ print $1 }'`"
 	grep "^  Summary: " "$T/failuresum" |
 		grep -v '^  Summary: Bugs: [0-9]* (all bugs kcsan)$' > "$T/nonkcsan"
 	fi
 	ret=2
 fi
-if test "$do_kcsan" = "yes"
+if test "$do_kcsan" = "yes" && test -e tools/testing/selftests/rcutorture/res/$ds
 then
 	TORTURE_KCONFIG_KCSAN_ARG=1 tools/testing/selftests/rcutorture/bin/kcsan-collapse.sh tools/testing/selftests/rcutorture/res/$ds > tools/testing/selftests/rcutorture/res/$ds/kcsan.sum
 fi
 echo Started at $startdate, ended at `date`, duration `get_starttime_duration $starttime`. | tee -a $T/log
 echo Summary: Successes: $nsuccesses Failures: $nfailures. | tee -a $T/log
-tdir="`cat $T/successes $T/failures | head -1 | awk '{ print $NF }' | sed -e 's,/[^/]\+/*$,,'`"
+tdir="`cat $T/successes $T/failures | awk 'NF > 1 { print $NF }' | head -1 | sed -e 's,/[^/]\+/*$,,'`"
-find "$tdir" -name 'ConfigFragment.diags' -print > $T/configerrors
+if test -n "$tdir"
-find "$tdir" -name 'Make.out.diags' -print > $T/builderrors
+then
+	find "$tdir" -name 'ConfigFragment.diags' -print > $T/configerrors
+	find "$tdir" -name 'Make.out.diags' -print > $T/builderrors
+fi
 if test -s "$T/configerrors"
 then
 	echo "  Scenarios with .config errors: `wc -l "$T/configerrors" | awk '{ print $1 }'`"

tools/testing/selftests/rcutorture/configs/rcu/BUSTED+3−0

@@ -5,3 +5,6 @@ CONFIG_HOTPLUG_CPU=y · 4 unmodified lines
5	CONFIG_PREEMPT_NONE=n	5	CONFIG_PREEMPT_NONE=n
6	CONFIG_PREEMPT_VOLUNTARY=n	6	CONFIG_PREEMPT_VOLUNTARY=n
7	CONFIG_PREEMPT=y	7	CONFIG_PREEMPT=y
		8	CONFIG_RCU_TORTURE_TEST_CHK_RDR_STATE=y
		9	CONFIG_RCU_TORTURE_TEST_LOG_CPU=y
		10	CONFIG_RCU_TORTURE_TEST_LOG_GP=y

tools/testing/selftests/rcutorture/configs/rcu/CFLIST+0−1

@@ -5,7 +5,6 @@ TREE04 · 4 unmodified lines
5	TREE05	5	TREE05
6	TREE07	6	TREE07
7	TREE09	7	TREE09
8	SRCU-L
9	SRCU-N	8	SRCU-N
10	SRCU-P	9	SRCU-P
11	SRCU-T	10	SRCU-T

tools/testing/selftests/rcutorture/configs/rcu/SRCU-L+0−10

@@ -1,10 +0,0 @@
1	CONFIG_RCU_TRACE=n
2	CONFIG_SMP=y
3	CONFIG_NR_CPUS=6
4	CONFIG_HOTPLUG_CPU=y
5	CONFIG_PREEMPT_NONE=y
6	CONFIG_PREEMPT_VOLUNTARY=n
7	CONFIG_PREEMPT=n
8	#CHECK#CONFIG_RCU_EXPERT=n
9	CONFIG_KPROBES=n
10	CONFIG_FTRACE=n

tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot+0−3

@@ -1,3 +0,0 @@
1	rcutorture.torture_type=srcu
2	rcutorture.reader_flavor=0x4
3	rcutorture.fwd_progress=3

Merge tag 'rcu.release.v6.17' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code