Merge tag 'rcu.release.v6.17' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux

Pull RCU updates from Neeraj Upadhyay: "Expedited grace period updates: - Protect against early RCU exp quiescent state reporting during exp grace period initialization - Remove superfluous barrier in task unblock path - Remove the CPU online quiescent state report optimization, which is error prone for certain scenarios - Add warning for unexpected pending requested expedited quiescent state on dying CPU Core: - Robustify rcu_is_cpu_rrupt_from_idle() by using more accurate indicators of the actual context tracking state of a CPU - Handle ->defer_qs_iw_pending field data race - Enable rcu_normal_wake_from_gp by default on systems with <= 16 CPUs - Fix lockup in rcu_read_unlock() due to recursive irq_exit() calls - Refactor expedited handling condition in rcu_read_unlock_special() - Documentation updates for hotplug and GP init scan ordering, separation of rcu_state and rnp's gp_seq states, quiescent state reporting for offline CPUs torture-scripts: - Cleanup and improve scripts : remove superfluous warnings for disabled tests; better handling of kvm.sh --kconfig arg; suppress some confusing diagnostics; tolerate bad kvm.sh args; add new diagnostic for build output; fail allmodconfig testing on warnings - Include RCU_TORTURE_TEST_CHK_RDR_STATE config for KCSAN kernels - Disable default RCU-tasks and clocksource-wdog testing on arm64 - Add EXPERT Kconfig option for arm64 KCSAN runs - Remove SRCU-lite testing rcutorture: - Start torture writer threads creation after reader threads to handle race in SRCU-P scenario - Add SRCU down_read()/up_read() test - Add diagnostics for delayed SRCU up_read(), unmatched up_read(), print number of up/down readers and the number of such readers which migrated to other CPU - Ignore certain unsupported configurations for trivial RCU test - Fix splats in RT kernels due to inaccurate checks for BH-disabled context - Enable checks and logs to capture intentionally exercised unexpected scenarios (too short readers) for BUSTED test - Remove SRCU-lite testing srcu: - Expedite SRCU-fast grace periods - Remove SRCU-lite implementation - Add guards for SRCU-fast readers rcu nocb: - Dump NOCB group leader state on stall detection - Robustify nocb_cb_kthread pointer accesses - Fix delayed execution of hurry callbacks when LAZY_RCU is enabled refscale: - Fix multiplication overflow in "loops" and "nreaders" calculations" * tag 'rcu.release.v6.17' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux: (49 commits) rcu: Document concurrent quiescent state reporting for offline CPUs rcu: Document separation of rcu_state and rnp's gp_seq rcu: Document GP init vs hotplug-scan ordering requirements srcu: Add guards for SRCU-fast readers rcu: Fix delayed execution of hurry callbacks rcu: Refactor expedited handling check in rcu_read_unlock_special() checkpatch: Remove SRCU-lite deprecation srcu: Remove SRCU-lite implementation srcu: Expedite SRCU-fast grace periods rcutorture: Remove support for SRCU-lite rcutorture: Remove SRCU-lite scenarios torture: Remove support for SRCU-lite torture: Make torture.sh --allmodconfig testing fail on warnings torture: Add "ERROR" diagnostic for testing kernel-build output torture: Make torture.sh tolerate runs having bad kvm.sh arguments torture: Add textid.txt file to --do-allmodconfig and --do-rcu-rust runs torture: Extract testid.txt generation to separate script torture: Suppress "find" diagnostics from torture.sh --do-none run torture: Provide EXPERT Kconfig option for arm64 KCSAN torture.sh runs rcu: Fix rcu_read_unlock() deadloop due to IRQ work ...

Linus Torvalds committed 1 year ago

Commit 2db4df02 parents 7dff275 cc1d13624 files · +770 −319Browse files

Unified · Split

24 files

Documentation/RCU/Design/Data-Structures/Data-Structures.rst +33 −0
Documentation/RCU/Design/Requirements/Requirements.rst +128 −0
Documentation/admin-guide/kernel-parameters.txt +2 −1
include/linux/srcu.h +8 −46
include/linux/srcutiny.h +0 −3
include/linux/srcutree.h +0 −38
kernel/rcu/rcutorture.c +298 −58
kernel/rcu/refscale.c +8 −34
kernel/rcu/srcutree.c +2 −0
kernel/rcu/tree.c +64 −16
kernel/rcu/tree.h +12 −1
kernel/rcu/tree_exp.h +9 −50
kernel/rcu/tree_nocb.h +6 −4
kernel/rcu/tree_plugin.h +102 −20
kernel/rcu/tree_stall.h +1 −2
scripts/checkpatch.pl +0 −2
tools/testing/selftests/rcutorture/bin/kvm-build.sh +1 −1
tools/testing/selftests/rcutorture/bin/kvm.sh +2 −13
tools/testing/selftests/rcutorture/bin/mktestid.sh +29 −0
tools/testing/selftests/rcutorture/bin/torture.sh +62 −16
tools/testing/selftests/rcutorture/configs/rcu/BUSTED +3 −0
tools/testing/selftests/rcutorture/configs/rcu/CFLIST +0 −1
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L +0 −10
tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot +0 −3

Documentation/RCU/Design/Data-Structures/Data-Structures.rst+33−0

@@ -286,6 +286,39 @@ in order to detect the beginnings and ends of grace periods in a · 285 unmodified lines
286	286	distributed fashion. The values flow from ``rcu_state`` to ``rcu_node``
287	287	(down the tree from the root to the leaves) to ``rcu_data``.
288	288
	289	++-----------------------------------------------------------------------+
	290	+\| Quick Quiz: \|
	291	++-----------------------------------------------------------------------+
	292	+\| Given that the root rcu_node structure has a gp_seq field, \|
	293	+\| why does RCU maintain a separate gp_seq in the rcu_state structure? \|
	294	+\| Why not just use the root rcu_node's gp_seq as the official record \|
	295	+\| and update it directly when starting a new grace period? \|
	296	++-----------------------------------------------------------------------+
	297	+\| Answer: \|
	298	++-----------------------------------------------------------------------+
	299	+\| On single-node RCU trees (where the root node is also a leaf), \|
	300	+\| updating the root node's gp_seq immediately would create unnecessary \|
	301	+\| lock contention. Here's why: \|
	302	+\| \|
	303	+\| If we did rcu_seq_start() directly on the root node's gp_seq: \|
	304	+\| \|
	305	+\| 1. All CPUs would immediately see their node's gp_seq from their rdp's\|
	306	+\| gp_seq, in rcu_pending(). They would all then invoke the RCU-core. \|
	307	+\| 2. Which calls note_gp_changes() and try to acquire the node lock. \|
	308	+\| 3. But rnp->qsmask isn't initialized yet (happens later in \|
	309	+\| rcu_gp_init()) \|
	310	+\| 4. So each CPU would acquire the lock, find it can't determine if it \|
	311	+\| needs to report quiescent state (no qsmask), update rdp->gp_seq, \|
	312	+\| and release the lock. \|
	313	+\| 5. Result: Lots of lock acquisitions with no grace period progress \|
	314	+\| \|
	315	+\| By having a separate rcu_state.gp_seq, we can increment the official \|
	316	+\| grace period counter without immediately affecting what CPUs see in \|
	317	+\| their nodes. The hierarchical propagation in rcu_gp_init() then \|
	318	+\| updates the root node's gp_seq and qsmask together under the same lock\|
	319	+\| acquisition, avoiding this useless contention. \|
	320	++-----------------------------------------------------------------------+
	321	+
289	322	Miscellaneous
290	323	'''''''''''''
291	324

Documentation/RCU/Design/Requirements/Requirements.rst+128−0

@@ -1970,6 +1970,134 @@ corresponding CPU's leaf node lock is held. This avoids race conditions · 1969 unmodified lines
1970	1970	between RCU's hotplug notifier hooks, the grace period initialization
1971	1971	code, and the FQS loop, all of which refer to or modify this bookkeeping.
1972	1972
	1973	+Note that grace period initialization (rcu_gp_init()) must carefully sequence
	1974	+CPU hotplug scanning with grace period state changes. For example, the
	1975	+following race could occur in rcu_gp_init() if rcu_seq_start() were to happen
	1976	+after the CPU hotplug scanning.
	1977	+
	1978	+.. code-block:: none
	1979	+
	1980	+ CPU0 (rcu_gp_init) CPU1 CPU2
	1981	+ --------------------- ---- ----
	1982	+ // Hotplug scan first (WRONG ORDER)
	1983	+ rcu_for_each_leaf_node(rnp) {
	1984	+ rnp->qsmaskinit = rnp->qsmaskinitnext;
	1985	+ }
	1986	+ rcutree_report_cpu_starting()
	1987	+ rnp->qsmaskinitnext \|= mask;
	1988	+ rcu_read_lock()
	1989	+ r0 = *X;
	1990	+ r1 = *X;
	1991	+ X = NULL;
	1992	+ cookie = get_state_synchronize_rcu();
	1993	+ // cookie = 8 (future GP)
	1994	+ rcu_seq_start(&rcu_state.gp_seq);
	1995	+ // gp_seq = 5
	1996	+
	1997	+ // CPU1 now invisible to this GP!
	1998	+ rcu_for_each_node_breadth_first() {
	1999	+ rnp->qsmask = rnp->qsmaskinit;
	2000	+ // CPU1 not included!
	2001	+ }
	2002	+
	2003	+ // GP completes without CPU1
	2004	+ rcu_seq_end(&rcu_state.gp_seq);
	2005	+ // gp_seq = 8
	2006	+ poll_state_synchronize_rcu(cookie);
	2007	+ // Returns true!
	2008	+ kfree(r1);
	2009	+ r2 = *r0; // USE-AFTER-FREE!
	2010	+
	2011	+By incrementing gp_seq first, CPU1's RCU read-side critical section
	2012	+is guaranteed to not be missed by CPU2.
	2013	+
	2014	+Concurrent Quiescent State Reporting for Offline CPUs
	2015	+
	2016	+RCU must ensure that CPUs going offline report quiescent states to avoid
	2017	+blocking grace periods. This requires careful synchronization to handle
	2018	+race conditions
	2019	+
	2020	+Race condition causing Offline CPU to hang GP
	2021	+
	2022	+A race between CPU offlining and new GP initialization (gp_init) may occur
	2023	+because `rcu_report_qs_rnp()` in `rcutree_report_cpu_dead()` must temporarily
	2024	+release the `rcu_node` lock to wake the RCU grace-period kthread:
	2025	+
	2026	+.. code-block:: none
	2027	+
	2028	+ CPU1 (going offline) CPU0 (GP kthread)
	2029	+ -------------------- -----------------
	2030	+ rcutree_report_cpu_dead()
	2031	+ rcu_report_qs_rnp()
	2032	+ // Must release rnp->lock to wake GP kthread
	2033	+ raw_spin_unlock_irqrestore_rcu_node()
	2034	+ // Wakes up and starts new GP
	2035	+ rcu_gp_init()
	2036	+ // First loop:
	2037	+ copies qsmaskinitnext->qsmaskinit
	2038	+ // CPU1 still in qsmaskinitnext!
	2039	+
	2040	+ // Second loop:
	2041	+ rnp->qsmask = rnp->qsmaskinit
	2042	+ mask = rnp->qsmask & ~rnp->qsmaskinitnext
	2043	+ // mask is 0! CPU1 still in both masks
	2044	+ // Reacquire lock (but too late)
	2045	+ rnp->qsmaskinitnext &= ~mask // Finally clears bit
	2046	+
	2047	+Without `ofl_lock`, the new grace period includes the offline CPU and waits
	2048	+forever for its quiescent state causing a GP hang.
	2049	+
	2050	+A solution with ofl_lock
	2051	+
	2052	+The `ofl_lock` (offline lock) prevents `rcu_gp_init()` from running during
	2053	+the vulnerable window when `rcu_report_qs_rnp()` has released `rnp->lock`:
	2054	+
	2055	+.. code-block:: none
	2056	+
	2057	+ CPU0 (rcu_gp_init) CPU1 (rcutree_report_cpu_dead)
	2058	+ ------------------ ------------------------------
	2059	+ rcu_for_each_leaf_node(rnp) {
	2060	+ arch_spin_lock(&ofl_lock) -----> arch_spin_lock(&ofl_lock) [BLOCKED]
	2061	+
	2062	+ // Safe: CPU1 can't interfere
	2063	+ rnp->qsmaskinit = rnp->qsmaskinitnext
	2064	+
	2065	+ arch_spin_unlock(&ofl_lock) ---> // Now CPU1 can proceed
	2066	+ } // But snapshot already taken
	2067	+
	2068	+Another race causing GP hangs in rcu_gpu_init(): Reporting QS for Now-offline CPUs
	2069	+
	2070	+After the first loop takes an atomic snapshot of online CPUs, as shown above,
	2071	+the second loop in `rcu_gp_init()` detects CPUs that went offline between
	2072	+releasing `ofl_lock` and acquiring the per-node `rnp->lock`. This detection is
	2073	+crucial because:
	2074	+
	2075	+1. The CPU might have gone offline after the snapshot but before the second loop
	2076	+2. The offline CPU cannot report its own QS if it's already dead
	2077	+3. Without this detection, the grace period would wait forever for CPUs that
	2078	+ are now offline.
	2079	+
	2080	+The second loop performs this detection safely:
	2081	+
	2082	+.. code-block:: none
	2083	+
	2084	+ rcu_for_each_node_breadth_first(rnp) {
	2085	+ raw_spin_lock_irqsave_rcu_node(rnp, flags);
	2086	+ rnp->qsmask = rnp->qsmaskinit; // Apply the snapshot
	2087	+
	2088	+ // Detect CPUs offline after snapshot
	2089	+ mask = rnp->qsmask & ~rnp->qsmaskinitnext;
	2090	+
	2091	+ if (mask && rcu_is_leaf_node(rnp))
	2092	+ rcu_report_qs_rnp(mask, ...) // Report QS for offline CPUs
	2093	+ }
	2094	+
	2095	+This approach ensures atomicity: quiescent state reporting for offline CPUs
	2096	+happens either in `rcu_gp_init()` (second loop) or in `rcutree_report_cpu_dead()`,
	2097	+never both and never neither. The `rnp->lock` held throughout the sequence
	2098	+prevents races - `rcutree_report_cpu_dead()` also acquires this lock when
	2099	+clearing `qsmaskinitnext`, ensuring mutual exclusion.
	2100	+
1973	2101	Scheduler and RCU
1974	2102	~~~~~~~~~~~~~~~~~
1975	2103

Documentation/admin-guide/kernel-parameters.txt+2−1

@@ -5508,7 +5508,8 @@ · 5507 unmodified lines
5508	5508	echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp
5509	5509	or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1"
5510	5510
5511		− Default is 0.
	5511	+ Default is 1 if num_possible_cpus() <= 16 and it is not explicitly
	5512	+ disabled by the boot parameter passing 0.
5512	5513
5513	5514	rcuscale.gp_async= [KNL]
5514	5515	Measure performance of asynchronous

include/linux/srcu.h+8−46

@@ -46,11 +46,11 @@ int init_srcu_struct(struct srcu_struct *ssp); · 45 unmodified lines
46	46	/* Values for SRCU Tree srcu_data ->srcu_reader_flavor, but also used by rcutorture. */
47	47	#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock().
48	48	#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe().
49		−#define SRCU_READ_FLAVOR_LITE 0x4 // srcu_read_lock_lite().
	49	+// 0x4 // SRCU-lite is no longer with us.
50	50	#define SRCU_READ_FLAVOR_FAST 0x8 // srcu_read_lock_fast().
51	51	#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL \| SRCU_READ_FLAVOR_NMI \| \
52		− SRCU_READ_FLAVOR_LITE \| SRCU_READ_FLAVOR_FAST) // All of the above.
53		−#define SRCU_READ_FLAVOR_SLOWGP (SRCU_READ_FLAVOR_LITE \| SRCU_READ_FLAVOR_FAST)
	52	+ SRCU_READ_FLAVOR_FAST) // All of the above.
	53	+#define SRCU_READ_FLAVOR_SLOWGP SRCU_READ_FLAVOR_FAST
54	54	// Flavors requiring synchronize_rcu()
55	55	// instead of smp_mb().
56	56	void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp);
@@ -299,33 +299,6 @@ static inline struct srcu_ctr __percpu srcu_down_read_fast(struct srcu_struct · 242 unmodified lines
299	299	return __srcu_read_lock_fast(ssp);
300	300	}
301	301
302		−/**
303		− * srcu_read_lock_lite - register a new reader for an SRCU-protected structure.
304		− * @ssp: srcu_struct in which to register the new reader.
305		− *
306		− * Enter an SRCU read-side critical section, but for a light-weight
307		− * smp_mb()-free reader. See srcu_read_lock() for more information.
308		− *
309		− * If srcu_read_lock_lite() is ever used on an srcu_struct structure,
310		− * then none of the other flavors may be used, whether before, during,
311		− * or after. Note that grace-period auto-expediting is disabled for _lite
312		− * srcu_struct structures because auto-expedited grace periods invoke
313		− * synchronize_rcu_expedited(), IPIs and all.
314		− *
315		− * Note that srcu_read_lock_lite() can be invoked only from those contexts
316		− * where RCU is watching, that is, from contexts where it would be legal
317		− * to invoke rcu_read_lock(). Otherwise, lockdep will complain.
318		− */
319		−static inline int srcu_read_lock_lite(struct srcu_struct *ssp) __acquires(ssp)
320		−{
321		− int retval;
322		−
323		− srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_LITE);
324		− retval = __srcu_read_lock_lite(ssp);
325		− rcu_try_lock_acquire(&ssp->dep_map);
326		− return retval;
327		−}
328		−
329	302	/**
330	303	* srcu_read_lock_nmisafe - register a new reader for an SRCU-protected structure.
331	304	* @ssp: srcu_struct in which to register the new reader.
@@ -434,22 +407,6 @@ static inline void srcu_up_read_fast(struct srcu_struct *ssp, struct srcu_ctr __ · 102 unmodified lines
434	407	__srcu_read_unlock_fast(ssp, scp);
435	408	}
436	409
437		−/**
438		− * srcu_read_unlock_lite - unregister a old reader from an SRCU-protected structure.
439		− * @ssp: srcu_struct in which to unregister the old reader.
440		− * @idx: return value from corresponding srcu_read_lock_lite().
441		− *
442		− * Exit a light-weight SRCU read-side critical section.
443		− */
444		−static inline void srcu_read_unlock_lite(struct srcu_struct *ssp, int idx)
445		− __releases(ssp)
446		−{
447		− WARN_ON_ONCE(idx & ~0x1);
448		− srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_LITE);
449		− srcu_lock_release(&ssp->dep_map);
450		− __srcu_read_unlock_lite(ssp, idx);
451		−}
452		−
453	410	/**
454	411	* srcu_read_unlock_nmisafe - unregister a old reader from an SRCU-protected structure.
455	412	* @ssp: srcu_struct in which to unregister the old reader.
@@ -524,4 +481,9 @@ DEFINE_LOCK_GUARD_1(srcu, struct srcu_struct, · 68 unmodified lines
524	481	srcu_read_unlock(_T->lock, _T->idx),
525	482	int idx)
526	483
	484	+DEFINE_LOCK_GUARD_1(srcu_fast, struct srcu_struct,
	485	+ _T->scp = srcu_read_lock_fast(_T->lock),
	486	+ srcu_read_unlock_fast(_T->lock, _T->scp),
	487	+ struct srcu_ctr __percpu *scp)
	488	+
527	489	#endif

include/linux/srcutiny.h+0−3

@@ -93,9 +93,6 @@ static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ · 92 unmodified lines
93	93	__srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp));
94	94	}
95	95
96		−#define __srcu_read_lock_lite __srcu_read_lock
97		−#define __srcu_read_unlock_lite __srcu_read_unlock
98		−
99	96	static inline void synchronize_srcu_expedited(struct srcu_struct *ssp)
100	97	{
101	98	synchronize_srcu(ssp);

include/linux/srcutree.h+0−38

@@ -278,44 +278,6 @@ static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ · 277 unmodified lines
278	278	RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_fast().");
279	279	}
280	280
281		−/*
282		− * Counts the new reader in the appropriate per-CPU element of the
283		− * srcu_struct. Returns an index that must be passed to the matching
284		− * srcu_read_unlock_lite().
285		− *
286		− * Note that this_cpu_inc() is an RCU read-side critical section either
287		− * because it disables interrupts, because it is a single instruction,
288		− * or because it is a read-modify-write atomic operation, depending on
289		− * the whims of the architecture.
290		− */
291		−static inline int __srcu_read_lock_lite(struct srcu_struct *ssp)
292		−{
293		− struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp);
294		−
295		− RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_lite().");
296		− this_cpu_inc(scp->srcu_locks.counter); /* Y */
297		− barrier(); /* Avoid leaking the critical section. */
298		− return __srcu_ptr_to_ctr(ssp, scp);
299		−}
300		−
301		−/*
302		− * Removes the count for the old reader from the appropriate
303		− * per-CPU element of the srcu_struct. Note that this may well be a
304		− * different CPU than that which was incremented by the corresponding
305		− * srcu_read_lock_lite(), but it must be within the same task.
306		− *
307		− * Note that this_cpu_inc() is an RCU read-side critical section either
308		− * because it disables interrupts, because it is a single instruction,
309		− * or because it is a read-modify-write atomic operation, depending on
310		− * the whims of the architecture.
311		− */
312		−static inline void __srcu_read_unlock_lite(struct srcu_struct *ssp, int idx)
313		−{
314		− barrier(); /* Avoid leaking the critical section. */
315		− this_cpu_inc(__srcu_ctr_to_ptr(ssp, idx)->srcu_unlocks.counter); /* Z */
316		− RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_lite().");
317		−}
318		−
319	281	void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor);
320	282
321	283	// Record reader usage even for CONFIG_PROVE_RCU=n kernels. This is

kernel/rcu/rcutorture.c+298−58

@@ -55,22 +55,24 @@ MODULE_DESCRIPTION("Read-Copy Update module-based torture test facility"); · 54 unmodified lines
55	55	MODULE_LICENSE("GPL");
56	56	MODULE_AUTHOR("Paul E. McKenney <paulmck@linux.ibm.com> and Josh Triplett <josh@joshtriplett.org>");
57	57
58		−/* Bits for ->extendables field, extendables param, and related definitions. */
59		−#define RCUTORTURE_RDR_SHIFT_1 8 /* Put SRCU index in upper bits. */
60		−#define RCUTORTURE_RDR_MASK_1 (0xff << RCUTORTURE_RDR_SHIFT_1)
61		−#define RCUTORTURE_RDR_SHIFT_2 16 /* Put SRCU index in upper bits. */
62		−#define RCUTORTURE_RDR_MASK_2 (0xff << RCUTORTURE_RDR_SHIFT_2)
63		−#define RCUTORTURE_RDR_BH 0x01 /* Extend readers by disabling bh. */
64		−#define RCUTORTURE_RDR_IRQ 0x02 /* ... disabling interrupts. */
65		−#define RCUTORTURE_RDR_PREEMPT 0x04 /* ... disabling preemption. */
66		−#define RCUTORTURE_RDR_RBH 0x08 /* ... rcu_read_lock_bh(). */
67		−#define RCUTORTURE_RDR_SCHED 0x10 /* ... rcu_read_lock_sched(). */
68		−#define RCUTORTURE_RDR_RCU_1 0x20 /* ... entering another RCU reader. */
69		−#define RCUTORTURE_RDR_RCU_2 0x40 /* ... entering another RCU reader. */
70		−#define RCUTORTURE_RDR_NBITS 7 /* Number of bits defined above. */
71		−#define RCUTORTURE_MAX_EXTEND \
	58	+// Bits for ->extendables field, extendables param, and related definitions.
	59	+#define RCUTORTURE_RDR_SHIFT_1 8 // Put SRCU index in upper bits.
	60	+#define RCUTORTURE_RDR_MASK_1 (0xff << RCUTORTURE_RDR_SHIFT_1)
	61	+#define RCUTORTURE_RDR_SHIFT_2 16 // Put SRCU index in upper bits.
	62	+#define RCUTORTURE_RDR_MASK_2 (0xff << RCUTORTURE_RDR_SHIFT_2)
	63	+#define RCUTORTURE_RDR_BH 0x01 // Extend readers by disabling bh.
	64	+#define RCUTORTURE_RDR_IRQ 0x02 // ... disabling interrupts.
	65	+#define RCUTORTURE_RDR_PREEMPT 0x04 // ... disabling preemption.
	66	+#define RCUTORTURE_RDR_RBH 0x08 // ... rcu_read_lock_bh().
	67	+#define RCUTORTURE_RDR_SCHED 0x10 // ... rcu_read_lock_sched().
	68	+#define RCUTORTURE_RDR_RCU_1 0x20 // ... entering another RCU reader.
	69	+#define RCUTORTURE_RDR_RCU_2 0x40 // ... entering another RCU reader.
	70	+#define RCUTORTURE_RDR_UPDOWN 0x80 // ... up-read from task, down-read from timer.
	71	+ // Note: Manual start, automatic end.
	72	+#define RCUTORTURE_RDR_NBITS 8 // Number of bits defined above.
	73	+#define RCUTORTURE_MAX_EXTEND \
72	74	(RCUTORTURE_RDR_BH \| RCUTORTURE_RDR_IRQ \| RCUTORTURE_RDR_PREEMPT \| \
73		− RCUTORTURE_RDR_RBH \| RCUTORTURE_RDR_SCHED)
	75	+ RCUTORTURE_RDR_RBH \| RCUTORTURE_RDR_SCHED) // Intentionally omit RCUTORTURE_RDR_UPDOWN.
74	76	#define RCUTORTURE_RDR_ALLBITS \
75	77	(RCUTORTURE_MAX_EXTEND \| RCUTORTURE_RDR_RCU_1 \| RCUTORTURE_RDR_RCU_2 \| \
76	78	RCUTORTURE_RDR_MASK_1 \| RCUTORTURE_RDR_MASK_2)
@@ -110,6 +112,7 @@ torture_param(bool, gp_sync, false, "Use synchronous GP wait primitives"); · 33 unmodified lines
110	112	torture_param(int, irqreader, 1, "Allow RCU readers from irq handlers");
111	113	torture_param(int, leakpointer, 0, "Leak pointer dereferences from readers");
112	114	torture_param(int, n_barrier_cbs, 0, "# of callbacks/kthreads for barrier testing");
	115	+torture_param(int, n_up_down, 32, "# of concurrent up/down hrtimer-based RCU readers");
113	116	torture_param(int, nfakewriters, 4, "Number of RCU fake writer threads");
114	117	torture_param(int, nreaders, -1, "Number of RCU reader threads");
115	118	torture_param(int, object_debug, 0, "Enable debug-object double call_rcu() testing");
@@ -156,6 +159,7 @@ static int nrealfakewriters; · 40 unmodified lines
156	159	static struct task_struct *writer_task;
157	160	static struct task_struct **fakewriter_tasks;
158	161	static struct task_struct **reader_tasks;
	162	+static struct task_struct *updown_task;
159	163	static struct task_struct **nocb_tasks;
160	164	static struct task_struct *stats_task;
161	165	static struct task_struct *fqs_task;
@@ -378,6 +382,8 @@ struct rcu_torture_ops { · 216 unmodified lines
378	382	void (*readunlock)(int idx);
379	383	int (*readlock_held)(void); // lockdep.
380	384	int (*readlock_nesting)(void); // actual nesting, if available, -1 if not.
	385	+ int (*down_read)(void);
	386	+ void (*up_read)(int idx);
381	387	unsigned long (*get_gp_seq)(void);
382	388	unsigned long (*gp_diff)(unsigned long new, unsigned long old);
383	389	void (deferred_free)(struct rcu_torture p);
@@ -427,6 +433,7 @@ struct rcu_torture_ops { · 43 unmodified lines
427	433	int no_pi_lock;
428	434	int debug_objects;
429	435	int start_poll_irqsoff;
	436	+ int have_up_down;
430	437	const char *name;
431	438	};
432	439
@@ -464,7 +471,7 @@ rcu_read_delay(struct torture_random_state rrsp, struct rt_read_seg rtrsp) · 31 unmodified lines
464	471	!(torture_random(rrsp) % (nrealreaders * 2000 * longdelay_ms))) {
465	472	started = cur_ops->get_gp_seq();
466	473	ts = rcu_trace_clock_local();
467		− if (preempt_count() & (SOFTIRQ_MASK \| HARDIRQ_MASK))
	474	+ if ((preempt_count() & HARDIRQ_MASK) \|\| softirq_count())
468	475	longdelay_ms = 5; /* Avoid triggering BH limits. */
469	476	mdelay(longdelay_ms);
470	477	rtrsp->rt_delay_ms = longdelay_ms;
@@ -711,11 +718,6 @@ static int srcu_torture_read_lock(void) · 240 unmodified lines
711	718	WARN_ON_ONCE(idx & ~0x1);
712	719	ret += idx << 1;
713	720	}
714		− if (reader_flavor & SRCU_READ_FLAVOR_LITE) {
715		− idx = srcu_read_lock_lite(srcu_ctlp);
716		− WARN_ON_ONCE(idx & ~0x1);
717		− ret += idx << 2;
718		− }
719	721	if (reader_flavor & SRCU_READ_FLAVOR_FAST) {
720	722	scp = srcu_read_lock_fast(srcu_ctlp);
721	723	idx = __srcu_ptr_to_ctr(srcu_ctlp, scp);
@@ -749,8 +751,6 @@ static void srcu_torture_read_unlock(int idx) · 27 unmodified lines
749	751	WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) \|\| (!reader_flavor && (idx & ~0x1)));
750	752	if (reader_flavor & SRCU_READ_FLAVOR_FAST)
751	753	srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3));
752		− if (reader_flavor & SRCU_READ_FLAVOR_LITE)
753		− srcu_read_unlock_lite(srcu_ctlp, (idx & 0x4) >> 2);
754	754	if (reader_flavor & SRCU_READ_FLAVOR_NMI)
755	755	srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1);
756	756	if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) \|\| !(reader_flavor & SRCU_READ_FLAVOR_ALL))
@@ -762,6 +762,50 @@ static int torture_srcu_read_lock_held(void) · 5 unmodified lines
762	762	return srcu_read_lock_held(srcu_ctlp);
763	763	}
764	764
	765	+static bool srcu_torture_have_up_down(void)
	766	+{
	767	+ int rf = reader_flavor;
	768	+
	769	+ if (!rf)
	770	+ rf = SRCU_READ_FLAVOR_NORMAL;
	771	+ return !!(cur_ops->have_up_down & rf);
	772	+}
	773	+
	774	+static int srcu_torture_down_read(void)
	775	+{
	776	+ int idx;
	777	+ struct srcu_ctr __percpu *scp;
	778	+
	779	+ WARN_ON_ONCE(reader_flavor & ~SRCU_READ_FLAVOR_ALL);
	780	+ WARN_ON_ONCE(reader_flavor & (reader_flavor - 1));
	781	+
	782	+ if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) \|\| !(reader_flavor & SRCU_READ_FLAVOR_ALL)) {
	783	+ idx = srcu_down_read(srcu_ctlp);
	784	+ WARN_ON_ONCE(idx & ~0x1);
	785	+ return idx;
	786	+ }
	787	+ if (reader_flavor & SRCU_READ_FLAVOR_FAST) {
	788	+ scp = srcu_down_read_fast(srcu_ctlp);
	789	+ idx = __srcu_ptr_to_ctr(srcu_ctlp, scp);
	790	+ WARN_ON_ONCE(idx & ~0x1);
	791	+ return idx << 3;
	792	+ }
	793	+ WARN_ON_ONCE(1);
	794	+ return 0;
	795	+}
	796	+
	797	+static void srcu_torture_up_read(int idx)
	798	+{
	799	+ WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) \|\| (!reader_flavor && (idx & ~0x1)));
	800	+ if (reader_flavor & SRCU_READ_FLAVOR_FAST)
	801	+ srcu_up_read_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3));
	802	+ else if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) \|\|
	803	+ !(reader_flavor & SRCU_READ_FLAVOR_ALL))
	804	+ srcu_up_read(srcu_ctlp, idx & 0x1);
	805	+ else
	806	+ WARN_ON_ONCE(1);
	807	+}
	808	+
765	809	static unsigned long srcu_torture_completed(void)
766	810	{
767	811	return srcu_batches_completed(srcu_ctlp);
@@ -819,6 +863,8 @@ static struct rcu_torture_ops srcu_ops = { · 51 unmodified lines
819	863	.readlock = srcu_torture_read_lock,
820	864	.read_delay = srcu_read_delay,
821	865	.readunlock = srcu_torture_read_unlock,
	866	+ .down_read = srcu_torture_down_read,
	867	+ .up_read = srcu_torture_up_read,
822	868	.readlock_held = torture_srcu_read_lock_held,
823	869	.get_gp_seq = srcu_torture_completed,
824	870	.gp_diff = rcu_seq_diff,
@@ -839,6 +885,8 @@ static struct rcu_torture_ops srcu_ops = { · 14 unmodified lines
839	885	.irq_capable = 1,
840	886	.no_pi_lock = IS_ENABLED(CONFIG_TINY_SRCU),
841	887	.debug_objects = 1,
	888	+ .have_up_down = IS_ENABLED(CONFIG_TINY_SRCU)
	889	+ ? 0 : SRCU_READ_FLAVOR_NORMAL \| SRCU_READ_FLAVOR_FAST,
842	890	.name = "srcu"
843	891	};
844	892
@@ -864,6 +912,8 @@ static struct rcu_torture_ops srcud_ops = { · 19 unmodified lines
864	912	.read_delay = srcu_read_delay,
865	913	.readunlock = srcu_torture_read_unlock,
866	914	.readlock_held = torture_srcu_read_lock_held,
	915	+ .down_read = srcu_torture_down_read,
	916	+ .up_read = srcu_torture_up_read,
867	917	.get_gp_seq = srcu_torture_completed,
868	918	.gp_diff = rcu_seq_diff,
869	919	.deferred_free = srcu_torture_deferred_free,
@@ -883,6 +933,8 @@ static struct rcu_torture_ops srcud_ops = { · 13 unmodified lines
883	933	.irq_capable = 1,
884	934	.no_pi_lock = IS_ENABLED(CONFIG_TINY_SRCU),
885	935	.debug_objects = 1,
	936	+ .have_up_down = IS_ENABLED(CONFIG_TINY_SRCU)
	937	+ ? 0 : SRCU_READ_FLAVOR_NORMAL \| SRCU_READ_FLAVOR_FAST,
886	938	.name = "srcud"
887	939	};
888	940
@@ -910,7 +962,8 @@ static struct rcu_torture_ops busted_srcud_ops = { · 21 unmodified lines
910	962
911	963	/*
912	964	* Definitions for trivial CONFIG_PREEMPT=n-only torture testing.
913		− * This implementation does not necessarily work well with CPU hotplug.
	965	+ * This implementation does not work well with CPU hotplug nor
	966	+ * with rcutorture's shuffling.
914	967	*/
915	968
916	969	static void synchronize_rcu_trivial(void)
@@ -923,6 +976,16 @@ static void synchronize_rcu_trivial(void) · 6 unmodified lines
923	976	}
924	977	}
925	978
	979	+static void rcu_sync_torture_init_trivial(void)
	980	+{
	981	+ rcu_sync_torture_init();
	982	+ // if (onoff_interval \|\| shuffle_interval) {
	983	+ if (WARN_ONCE(onoff_interval \|\| shuffle_interval, "%s: Non-zero onoff_interval (%d) or shuffle_interval (%d) breaks trivial RCU, resetting to zero", __func__, onoff_interval, shuffle_interval)) {
	984	+ onoff_interval = 0;
	985	+ shuffle_interval = 0;
	986	+ }
	987	+}
	988	+
926	989	static int rcu_torture_read_lock_trivial(void)
927	990	{
928	991	preempt_disable();
@@ -936,7 +999,7 @@ static void rcu_torture_read_unlock_trivial(int idx) · 7 unmodified lines
936	999
937	1000	static struct rcu_torture_ops trivial_ops = {
938	1001	.ttype = RCU_TRIVIAL_FLAVOR,
939		− .init = rcu_sync_torture_init,
	1002	+ .init = rcu_sync_torture_init_trivial,
940	1003	.readlock = rcu_torture_read_lock_trivial,
941	1004	.read_delay = rcu_read_delay, /* just reuse rcu's version. */
942	1005	.readunlock = rcu_torture_read_unlock_trivial,
@@ -1722,6 +1785,7 @@ rcu_torture_writer(void *arg) · 779 unmodified lines
1722	1785	cur_ops->gp_kthread_dbg();
1723	1786	WARN(1, "%s: rtort_pipe_count: %d\n", __func__, rcu_tortures[i].rtort_pipe_count);
1724	1787	rcu_ftrace_dump(DUMP_ALL);
	1788	+ break;
1725	1789	}
1726	1790	if (stutter_waited)
1727	1791	sched_set_normal(current, oldnice);
@@ -1915,14 +1979,14 @@ static void rcu_torture_reader_do_mbchk(long myid, struct rcu_torture *rtp, · 187 unmodified lines
1915	1979
1916	1980	// Verify the specified RCUTORTURE_RDR* state.
1917	1981	#define ROEC_ARGS "%s %s: Current %#x To add %#x To remove %#x preempt_count() %#x\n", __func__, s, curstate, new, old, preempt_count()
1918		−static void rcutorture_one_extend_check(char *s, int curstate, int new, int old, bool insoftirq)
	1982	+static void rcutorture_one_extend_check(char *s, int curstate, int new, int old)
1919	1983	{
1920	1984	int mask;
1921	1985
1922		− if (!IS_ENABLED(CONFIG_RCU_TORTURE_TEST_CHK_RDR_STATE))
	1986	+ if (!IS_ENABLED(CONFIG_RCU_TORTURE_TEST_CHK_RDR_STATE) \|\| in_nmi())
1923	1987	return;
1924	1988
1925		− WARN_ONCE(!(curstate & RCUTORTURE_RDR_IRQ) && irqs_disabled(), ROEC_ARGS);
	1989	+ WARN_ONCE(!(curstate & RCUTORTURE_RDR_IRQ) && irqs_disabled() && !in_hardirq(), ROEC_ARGS);
1926	1990	WARN_ONCE((curstate & RCUTORTURE_RDR_IRQ) && !irqs_disabled(), ROEC_ARGS);
1927	1991
1928	1992	// If CONFIG_PREEMPT_COUNT=n, further checks are unreliable.
@@ -1930,21 +1994,21 @@ static void rcutorture_one_extend_check(char *s, int curstate, int new, int old, · 1 unmodified line
1930	1994	return;
1931	1995
1932	1996	WARN_ONCE((curstate & (RCUTORTURE_RDR_BH \| RCUTORTURE_RDR_RBH)) &&
1933		− !(preempt_count() & SOFTIRQ_MASK), ROEC_ARGS);
	1997	+ !softirq_count(), ROEC_ARGS);
1934	1998	WARN_ONCE((curstate & (RCUTORTURE_RDR_PREEMPT \| RCUTORTURE_RDR_SCHED)) &&
1935	1999	!(preempt_count() & PREEMPT_MASK), ROEC_ARGS);
1936	2000	WARN_ONCE(cur_ops->readlock_nesting &&
1937	2001	(curstate & (RCUTORTURE_RDR_RCU_1 \| RCUTORTURE_RDR_RCU_2)) &&
1938	2002	cur_ops->readlock_nesting() == 0, ROEC_ARGS);
1939	2003
1940		− // Timer handlers have all sorts of stuff disabled, so ignore
	2004	+ // Interrupt handlers have all sorts of stuff disabled, so ignore
1941	2005	// unintended disabling.
1942		− if (insoftirq)
	2006	+ if (in_serving_softirq() \|\| in_hardirq())
1943	2007	return;
1944	2008
1945	2009	WARN_ONCE(cur_ops->extendables &&
1946	2010	!(curstate & (RCUTORTURE_RDR_BH \| RCUTORTURE_RDR_RBH)) &&
1947		− (preempt_count() & SOFTIRQ_MASK), ROEC_ARGS);
	2011	+ softirq_count(), ROEC_ARGS);
1948	2012
1949	2013	/*
1950	2014	* non-preemptible RCU in a preemptible kernel uses preempt_disable()
@@ -1965,6 +2029,9 @@ static void rcutorture_one_extend_check(char *s, int curstate, int new, int old, · 14 unmodified lines
1965	2029	if (!IS_ENABLED(CONFIG_PREEMPT_RCU))
1966	2030	mask \|= RCUTORTURE_RDR_PREEMPT \| RCUTORTURE_RDR_SCHED;
1967	2031
	2032	+ if (IS_ENABLED(CONFIG_PREEMPT_RT) && softirq_count())
	2033	+ mask \|= RCUTORTURE_RDR_BH \| RCUTORTURE_RDR_RBH;
	2034	+
1968	2035	WARN_ONCE(cur_ops->readlock_nesting && !(curstate & mask) &&
1969	2036	cur_ops->readlock_nesting() > 0, ROEC_ARGS);
1970	2037	}
@@ -1978,8 +2045,7 @@ static void rcutorture_one_extend_check(char *s, int curstate, int new, int old, · 7 unmodified lines
1978	2045	* beginning or end of the critical section and if there was actually a
1979	2046	* change, do a ->read_delay().
1980	2047	*/
1981		−static void rcutorture_one_extend(int *readstate, int newstate, bool insoftirq,
1982		− struct torture_random_state *trsp,
	2048	+static void rcutorture_one_extend(int readstate, int newstate, struct torture_random_state trsp,
1983	2049	struct rt_read_seg *rtrsp)
1984	2050	{
1985	2051	bool first;
@@ -1993,8 +2059,8 @@ static void rcutorture_one_extend(int *readstate, int newstate, bool insoftirq, · 7 unmodified lines
1993	2059
1994	2060	first = idxold1 == 0;
1995	2061	WARN_ON_ONCE(idxold2 < 0);
1996		− WARN_ON_ONCE(idxold2 & ~RCUTORTURE_RDR_ALLBITS);
1997		− rcutorture_one_extend_check("before change", idxold1, statesnew, statesold, insoftirq);
	2062	+ WARN_ON_ONCE(idxold2 & ~(RCUTORTURE_RDR_ALLBITS \| RCUTORTURE_RDR_UPDOWN));
	2063	+ rcutorture_one_extend_check("before change", idxold1, statesnew, statesold);
1998	2064	rtrsp->rt_readstate = newstate;
1999	2065
2000	2066	/* First, put new protection in place to avoid critical-section gap. */
@@ -2014,8 +2080,7 @@ static void rcutorture_one_extend(int *readstate, int newstate, bool insoftirq, · 13 unmodified lines
2014	2080	idxnew2 = (cur_ops->readlock() << RCUTORTURE_RDR_SHIFT_2) & RCUTORTURE_RDR_MASK_2;
2015	2081
2016	2082	// Complain unless both the old and the new protection is in place.
2017		− rcutorture_one_extend_check("during change",
2018		− idxold1 \| statesnew, statesnew, statesold, insoftirq);
	2083	+ rcutorture_one_extend_check("during change", idxold1 \| statesnew, statesnew, statesold);
2019	2084
2020	2085	// Sample CPU under both sets of protections to reduce confusion.
2021	2086	if (IS_ENABLED(CONFIG_RCU_TORTURE_TEST_LOG_CPU)) {
@@ -2069,6 +2134,11 @@ static void rcutorture_one_extend(int *readstate, int newstate, bool insoftirq, · 47 unmodified lines
2069	2134	if (lockit)
2070	2135	raw_spin_unlock_irqrestore(&current->pi_lock, flags);
2071	2136	}
	2137	+ if (statesold & RCUTORTURE_RDR_UPDOWN) {
	2138	+ cur_ops->up_read((idxold1 & RCUTORTURE_RDR_MASK_1) >> RCUTORTURE_RDR_SHIFT_1);
	2139	+ WARN_ON_ONCE(idxnew1 != -1);
	2140	+ idxold1 = 0;
	2141	+ }
2072	2142
2073	2143	/* Delay if neither beginning nor end and there was a change. */
2074	2144	if ((statesnew \|\| statesold) && *readstate && newstate)
@@ -2085,7 +2155,7 @@ static void rcutorture_one_extend(int *readstate, int newstate, bool insoftirq, · 10 unmodified lines
2085	2155	WARN_ON_ONCE(*readstate < 0);
2086	2156	if (WARN_ON_ONCE(*readstate & ~RCUTORTURE_RDR_ALLBITS))
2087	2157	pr_info("Unexpected readstate value of %#x\n", *readstate);
2088		− rcutorture_one_extend_check("after change", *readstate, statesnew, statesold, insoftirq);
	2158	+ rcutorture_one_extend_check("after change", *readstate, statesnew, statesold);
2089	2159	}
2090	2160
2091	2161	/* Return the biggest extendables mask given current RCU and boot parameters. */
@@ -2152,8 +2222,7 @@ rcutorture_extend_mask(int oldmask, struct torture_random_state *trsp) · 60 unmodified lines
2152	2222	* critical section.
2153	2223	*/
2154	2224	static struct rt_read_seg *
2155		−rcutorture_loop_extend(int readstate, bool insoftirq, struct torture_random_state trsp,
2156		− struct rt_read_seg *rtrsp)
	2225	+rcutorture_loop_extend(int readstate, struct torture_random_state trsp, struct rt_read_seg *rtrsp)
2157	2226	{
2158	2227	int i;
2159	2228	int j;
@@ -2167,7 +2236,8 @@ rcutorture_loop_extend(int *readstate, bool insoftirq, struct torture_random_sta · 7 unmodified lines
2167	2236	i = ((i \| (i >> 3)) & RCUTORTURE_RDR_MAX_LOOPS) + 1;
2168	2237	for (j = 0; j < i; j++) {
2169	2238	mask = rcutorture_extend_mask(*readstate, trsp);
2170		− rcutorture_one_extend(readstate, mask, insoftirq, trsp, &rtrsp[j]);
	2239	+ WARN_ON_ONCE(mask & RCUTORTURE_RDR_UPDOWN);
	2240	+ rcutorture_one_extend(readstate, mask, trsp, &rtrsp[j]);
2171	2241	}
2172	2242	return &rtrsp[j];
2173	2243	}
@@ -2209,10 +2279,11 @@ static bool rcu_torture_one_read_start(struct rcu_torture_one_read_state *rtorsp · 35 unmodified lines
2209	2279	rtorsp->started = cur_ops->get_gp_seq();
2210	2280	rtorsp->ts = rcu_trace_clock_local();
2211	2281	rtorsp->p = rcu_dereference_check(rcu_torture_current,
2212		− !cur_ops->readlock_held \|\| cur_ops->readlock_held());
	2282	+ !cur_ops->readlock_held \|\| cur_ops->readlock_held() \|\|
	2283	+ (rtorsp->readstate & RCUTORTURE_RDR_UPDOWN));
2213	2284	if (rtorsp->p == NULL) {
2214	2285	/* Wait for rcu_torture_writer to get underway */
2215		− rcutorture_one_extend(&rtorsp->readstate, 0, myid < 0, trsp, rtorsp->rtrsp);
	2286	+ rcutorture_one_extend(&rtorsp->readstate, 0, trsp, rtorsp->rtrsp);
2216	2287	return false;
2217	2288	}
2218	2289	if (rtorsp->p->rtort_mbtest == 0)
@@ -2226,7 +2297,7 @@ static bool rcu_torture_one_read_start(struct rcu_torture_one_read_state *rtorsp · 7 unmodified lines
2226	2297	* critical sections and check for errors.
2227	2298	*/
2228	2299	static void rcu_torture_one_read_end(struct rcu_torture_one_read_state *rtorsp,
2229		− struct torture_random_state *trsp, long myid)
	2300	+ struct torture_random_state *trsp)
2230	2301	{
2231	2302	int i;
2232	2303	unsigned long completed;
@@ -2273,7 +2344,7 @@ static void rcu_torture_one_read_end(struct rcu_torture_one_read_state *rtorsp, · 40 unmodified lines
2273	2344	}
2274	2345	if (cur_ops->reader_blocked)
2275	2346	preempted = cur_ops->reader_blocked();
2276		− rcutorture_one_extend(&rtorsp->readstate, 0, myid < 0, trsp, rtorsp->rtrsp);
	2347	+ rcutorture_one_extend(&rtorsp->readstate, 0, trsp, rtorsp->rtrsp);
2277	2348	WARN_ON_ONCE(rtorsp->readstate);
2278	2349	// This next splat is expected behavior if leakpointer, especially
2279	2350	// for CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels.
@@ -2302,13 +2373,14 @@ static bool rcu_torture_one_read(struct torture_random_state *trsp, long myid) · 22 unmodified lines
2302	2373	WARN_ON_ONCE(!rcu_is_watching());
2303	2374	init_rcu_torture_one_read_state(&rtors, trsp);
2304	2375	newstate = rcutorture_extend_mask(rtors.readstate, trsp);
2305		− rcutorture_one_extend(&rtors.readstate, newstate, myid < 0, trsp, rtors.rtrsp++);
	2376	+ WARN_ON_ONCE(newstate & RCUTORTURE_RDR_UPDOWN);
	2377	+ rcutorture_one_extend(&rtors.readstate, newstate, trsp, rtors.rtrsp++);
2306	2378	if (!rcu_torture_one_read_start(&rtors, trsp, myid)) {
2307		− rcutorture_one_extend(&rtors.readstate, 0, myid < 0, trsp, rtors.rtrsp);
	2379	+ rcutorture_one_extend(&rtors.readstate, 0, trsp, rtors.rtrsp);
2308	2380	return false;
2309	2381	}
2310		− rtors.rtrsp = rcutorture_loop_extend(&rtors.readstate, myid < 0, trsp, rtors.rtrsp);
2311		− rcu_torture_one_read_end(&rtors, trsp, myid);
	2382	+ rtors.rtrsp = rcutorture_loop_extend(&rtors.readstate, trsp, rtors.rtrsp);
	2383	+ rcu_torture_one_read_end(&rtors, trsp);
2312	2384	return true;
2313	2385	}
2314	2386
@@ -2378,6 +2450,152 @@ rcu_torture_reader(void *arg) · 63 unmodified lines
2378	2450	return 0;
2379	2451	}
2380	2452
	2453	+struct rcu_torture_one_read_state_updown {
	2454	+ struct hrtimer rtorsu_hrt;
	2455	+ bool rtorsu_inuse;
	2456	+ ktime_t rtorsu_kt;
	2457	+ int rtorsu_cpu;
	2458	+ unsigned long rtorsu_j;
	2459	+ unsigned long rtorsu_ndowns;
	2460	+ unsigned long rtorsu_nups;
	2461	+ unsigned long rtorsu_nmigrates;
	2462	+ struct torture_random_state rtorsu_trs;
	2463	+ struct rcu_torture_one_read_state rtorsu_rtors;
	2464	+};
	2465	+
	2466	+static struct rcu_torture_one_read_state_updown *updownreaders;
	2467	+static DEFINE_TORTURE_RANDOM(rcu_torture_updown_rand);
	2468	+static int rcu_torture_updown(void *arg);
	2469	+
	2470	+static enum hrtimer_restart rcu_torture_updown_hrt(struct hrtimer *hrtp)
	2471	+{
	2472	+ int cpu = raw_smp_processor_id();
	2473	+ struct rcu_torture_one_read_state_updown *rtorsup;
	2474	+
	2475	+ rtorsup = container_of(hrtp, struct rcu_torture_one_read_state_updown, rtorsu_hrt);
	2476	+ rcu_torture_one_read_end(&rtorsup->rtorsu_rtors, &rtorsup->rtorsu_trs);
	2477	+ WARN_ONCE(rtorsup->rtorsu_nups >= rtorsup->rtorsu_ndowns, "%s: Up without matching down #%zu.\n", __func__, rtorsup - updownreaders);
	2478	+ WRITE_ONCE(rtorsup->rtorsu_nups, rtorsup->rtorsu_nups + 1);
	2479	+ WRITE_ONCE(rtorsup->rtorsu_nmigrates,
	2480	+ rtorsup->rtorsu_nmigrates + (cpu != rtorsup->rtorsu_cpu));
	2481	+ smp_store_release(&rtorsup->rtorsu_inuse, false);
	2482	+ return HRTIMER_NORESTART;
	2483	+}
	2484	+
	2485	+static int rcu_torture_updown_init(void)
	2486	+{
	2487	+ int i;
	2488	+ struct torture_random_state *rand = &rcu_torture_updown_rand;
	2489	+ int ret;
	2490	+
	2491	+ if (n_up_down < 0)
	2492	+ return 0;
	2493	+ if (!srcu_torture_have_up_down()) {
	2494	+ VERBOSE_TOROUT_STRING("rcu_torture_updown_init: Disabling up/down reader tests due to lack of primitives");
	2495	+ return 0;
	2496	+ }
	2497	+ updownreaders = kcalloc(n_up_down, sizeof(*updownreaders), GFP_KERNEL);
	2498	+ if (!updownreaders) {
	2499	+ VERBOSE_TOROUT_STRING("rcu_torture_updown_init: Out of memory, disabling up/down reader tests");
	2500	+ return -ENOMEM;
	2501	+ }
	2502	+ for (i = 0; i < n_up_down; i++) {
	2503	+ init_rcu_torture_one_read_state(&updownreaders[i].rtorsu_rtors, rand);
	2504	+ hrtimer_setup(&updownreaders[i].rtorsu_hrt, rcu_torture_updown_hrt, CLOCK_MONOTONIC,
	2505	+ HRTIMER_MODE_REL \| HRTIMER_MODE_HARD);
	2506	+ torture_random_init(&updownreaders[i].rtorsu_trs);
	2507	+ init_rcu_torture_one_read_state(&updownreaders[i].rtorsu_rtors,
	2508	+ &updownreaders[i].rtorsu_trs);
	2509	+ }
	2510	+ ret = torture_create_kthread(rcu_torture_updown, rand, updown_task);
	2511	+ if (ret) {
	2512	+ kfree(updownreaders);
	2513	+ updownreaders = NULL;
	2514	+ }
	2515	+ return ret;
	2516	+}
	2517	+
	2518	+static void rcu_torture_updown_cleanup(void)
	2519	+{
	2520	+ struct rcu_torture_one_read_state_updown *rtorsup;
	2521	+
	2522	+ for (rtorsup = updownreaders; rtorsup < &updownreaders[n_up_down]; rtorsup++) {
	2523	+ if (!smp_load_acquire(&rtorsup->rtorsu_inuse))
	2524	+ continue;
	2525	+ if (hrtimer_cancel(&rtorsup->rtorsu_hrt) \|\| WARN_ON_ONCE(rtorsup->rtorsu_inuse)) {
	2526	+ rcu_torture_one_read_end(&rtorsup->rtorsu_rtors, &rtorsup->rtorsu_trs);
	2527	+ WARN_ONCE(rtorsup->rtorsu_nups >= rtorsup->rtorsu_ndowns, "%s: Up without matching down #%zu.\n", __func__, rtorsup - updownreaders);
	2528	+ WRITE_ONCE(rtorsup->rtorsu_nups, rtorsup->rtorsu_nups + 1);
	2529	+ smp_store_release(&rtorsup->rtorsu_inuse, false);
	2530	+ }
	2531	+
	2532	+ }
	2533	+ kfree(updownreaders);
	2534	+ updownreaders = NULL;
	2535	+}
	2536	+
	2537	+// Do one reader for rcu_torture_updown().
	2538	+static void rcu_torture_updown_one(struct rcu_torture_one_read_state_updown *rtorsup)
	2539	+{
	2540	+ int idx;
	2541	+ int rawidx;
	2542	+ ktime_t t;
	2543	+
	2544	+ init_rcu_torture_one_read_state(&rtorsup->rtorsu_rtors, &rtorsup->rtorsu_trs);
	2545	+ rawidx = cur_ops->down_read();
	2546	+ WRITE_ONCE(rtorsup->rtorsu_ndowns, rtorsup->rtorsu_ndowns + 1);
	2547	+ idx = (rawidx << RCUTORTURE_RDR_SHIFT_1) & RCUTORTURE_RDR_MASK_1;
	2548	+ rtorsup->rtorsu_rtors.readstate = idx \| RCUTORTURE_RDR_UPDOWN;
	2549	+ rtorsup->rtorsu_rtors.rtrsp++;
	2550	+ rtorsup->rtorsu_cpu = raw_smp_processor_id();
	2551	+ if (!rcu_torture_one_read_start(&rtorsup->rtorsu_rtors, &rtorsup->rtorsu_trs, -1)) {
	2552	+ WARN_ONCE(rtorsup->rtorsu_nups >= rtorsup->rtorsu_ndowns, "%s: Up without matching down #%zu.\n", __func__, rtorsup - updownreaders);
	2553	+ WRITE_ONCE(rtorsup->rtorsu_nups, rtorsup->rtorsu_nups + 1);
	2554	+ schedule_timeout_idle(HZ);
	2555	+ return;
	2556	+ }
	2557	+ smp_store_release(&rtorsup->rtorsu_inuse, true);
	2558	+ t = torture_random(&rtorsup->rtorsu_trs) & 0xfffff; // One per million.
	2559	+ if (t < 10 * 1000)
	2560	+ t = 200 * 1000 * 1000;
	2561	+ hrtimer_start(&rtorsup->rtorsu_hrt, t, HRTIMER_MODE_REL \| HRTIMER_MODE_HARD);
	2562	+ smp_mb(); // Sample jiffies after posting hrtimer.
	2563	+ rtorsup->rtorsu_j = jiffies; // Not used by hrtimer handler.
	2564	+ rtorsup->rtorsu_kt = t;
	2565	+}
	2566	+
	2567	+/*
	2568	+ * RCU torture up/down reader kthread, starting RCU readers in kthread
	2569	+ * context and ending them in hrtimer handlers. Otherwise similar to
	2570	+ * rcu_torture_reader().
	2571	+ */
	2572	+static int
	2573	+rcu_torture_updown(void *arg)
	2574	+{
	2575	+ unsigned long j;
	2576	+ struct rcu_torture_one_read_state_updown *rtorsup;
	2577	+
	2578	+ VERBOSE_TOROUT_STRING("rcu_torture_updown task started");
	2579	+ do {
	2580	+ for (rtorsup = updownreaders; rtorsup < &updownreaders[n_up_down]; rtorsup++) {
	2581	+ if (torture_must_stop())
	2582	+ break;
	2583	+ j = smp_load_acquire(&jiffies); // Time before ->rtorsu_inuse.
	2584	+ if (smp_load_acquire(&rtorsup->rtorsu_inuse)) {
	2585	+ WARN_ONCE(time_after(j, rtorsup->rtorsu_j + 1 + HZ * 10),
	2586	+ "hrtimer queued at jiffies %lu for %lld ns took %lu jiffies\n", rtorsup->rtorsu_j, rtorsup->rtorsu_kt, j - rtorsup->rtorsu_j);
	2587	+ continue;
	2588	+ }
	2589	+ rcu_torture_updown_one(rtorsup);
	2590	+ }
	2591	+ torture_hrtimeout_ms(1, 1000, &rcu_torture_updown_rand);
	2592	+ stutter_wait("rcu_torture_updown");
	2593	+ } while (!torture_must_stop());
	2594	+ rcu_torture_updown_cleanup();
	2595	+ torture_kthread_stopping("rcu_torture_updown");
	2596	+ return 0;
	2597	+}
	2598	+
2381	2599	/*
2382	2600	* Randomly Toggle CPUs' callback-offload state. This uses hrtimers to
2383	2601	* increase race probabilities and fuzzes the interval between toggling.
@@ -2441,6 +2659,10 @@ rcu_torture_stats_print(void) · 57 unmodified lines
2441	2659	long pipesummary[RCU_TORTURE_PIPE_LEN + 1] = { 0 };
2442	2660	long batchsummary[RCU_TORTURE_PIPE_LEN + 1] = { 0 };
2443	2661	long n_gpwraps = 0;
	2662	+ unsigned long ndowns = 0;
	2663	+ unsigned long nunexpired = 0;
	2664	+ unsigned long nmigrates = 0;
	2665	+ unsigned long nups = 0;
2444	2666	struct rcu_torture *rtcp;
2445	2667	static unsigned long rtcv_snap = ULONG_MAX;
2446	2668	static bool splatted;
@@ -2454,10 +2676,18 @@ rcu_torture_stats_print(void) · 7 unmodified lines
2454	2676	if (cur_ops->get_gpwrap_count)
2455	2677	n_gpwraps += cur_ops->get_gpwrap_count(cpu);
2456	2678	}
	2679	+ if (updownreaders) {
	2680	+ for (i = 0; i < n_up_down; i++) {
	2681	+ ndowns += READ_ONCE(updownreaders[i].rtorsu_ndowns);
	2682	+ nups += READ_ONCE(updownreaders[i].rtorsu_nups);
	2683	+ nunexpired += READ_ONCE(updownreaders[i].rtorsu_inuse);
	2684	+ nmigrates += READ_ONCE(updownreaders[i].rtorsu_nmigrates);
	2685	+ }
	2686	+ }
2457	2687	for (i = RCU_TORTURE_PIPE_LEN; i >= 0; i--) {
2458	2688	if (pipesummary[i] != 0)
2459	2689	break;
2460		− }
	2690	+ } // The value of variable "i" is used later, so don't clobber it!
2461	2691
2462	2692	pr_alert("%s%s ", torture_type, TORTURE_FLAG);
2463	2693	rtcp = rcu_access_pointer(rcu_torture_current);
@@ -2478,6 +2708,8 @@ rcu_torture_stats_print(void) · 14 unmodified lines
2478	2708	n_rcu_torture_boost_failure,
2479	2709	n_rcu_torture_boosts,
2480	2710	atomic_long_read(&n_rcu_torture_timers));
	2711	+ if (updownreaders)
	2712	+ pr_cont("ndowns: %lu nups: %lu nhrt: %lu nmigrates: %lu ", ndowns, nups, nunexpired, nmigrates);
2481	2713	torture_onoff_stats();
2482	2714	pr_cont("barrier: %ld/%ld:%ld ",
2483	2715	data_race(n_barrier_successes),
@@ -2632,7 +2864,7 @@ rcu_torture_print_module_parms(struct rcu_torture_ops cur_ops, const char tag) · 148 unmodified lines
2632	2864	"reader_flavor=%x "
2633	2865	"nocbs_nthreads=%d nocbs_toggle=%d "
2634	2866	"test_nmis=%d "
2635		− "preempt_duration=%d preempt_interval=%d\n",
	2867	+ "preempt_duration=%d preempt_interval=%d n_up_down=%d\n",
2636	2868	torture_type, tag, nrealreaders, nrealfakewriters,
2637	2869	stat_interval, verbose, test_no_idle_hz, shuffle_interval,
2638	2870	stutter, irqreader, fqs_duration, fqs_holdoff, fqs_stutter,
@@ -2646,7 +2878,7 @@ rcu_torture_print_module_parms(struct rcu_torture_ops cur_ops, const char tag) · 7 unmodified lines
2646	2878	reader_flavor,
2647	2879	nocbs_nthreads, nocbs_toggle,
2648	2880	test_nmis,
2649		− preempt_duration, preempt_interval);
	2881	+ preempt_duration, preempt_interval, n_up_down);
2650	2882	}
2651	2883
2652	2884	static int rcutorture_booster_cleanup(unsigned int cpu)
@@ -3749,6 +3981,10 @@ rcu_torture_cleanup(void) · 1096 unmodified lines
3749	3981	nocb_tasks = NULL;
3750	3982	}
3751	3983
	3984	+ if (updown_task) {
	3985	+ torture_stop_kthread(rcu_torture_updown, updown_task);
	3986	+ updown_task = NULL;
	3987	+ }
3752	3988	if (reader_tasks) {
3753	3989	for (i = 0; i < nrealreaders; i++)
3754	3990	torture_stop_kthread(rcu_torture_reader,
@@ -4245,11 +4481,6 @@ rcu_torture_init(void) · 490 unmodified lines
4245	4481	/* Start up the kthreads. */
4246	4482
4247	4483	rcu_torture_write_types();
4248		− firsterr = torture_create_kthread(rcu_torture_writer, NULL,
4249		− writer_task);
4250		− if (torture_init_error(firsterr))
4251		− goto unwind;
4252		−
4253	4484	if (nrealfakewriters > 0) {
4254	4485	fakewriter_tasks = kcalloc(nrealfakewriters,
4255	4486	sizeof(fakewriter_tasks[0]),
@@ -4282,6 +4513,15 @@ rcu_torture_init(void) · 26 unmodified lines
4282	4513	if (torture_init_error(firsterr))
4283	4514	goto unwind;
4284	4515	}
	4516	+
	4517	+ firsterr = torture_create_kthread(rcu_torture_writer, NULL,
	4518	+ writer_task);
	4519	+ if (torture_init_error(firsterr))
	4520	+ goto unwind;
	4521	+
	4522	+ firsterr = rcu_torture_updown_init();
	4523	+ if (torture_init_error(firsterr))
	4524	+ goto unwind;
4285	4525	nrealnocbers = nocbs_nthreads;
4286	4526	if (WARN_ON(nrealnocbers < 0))
4287	4527	nrealnocbers = 1;

kernel/rcu/refscale.c+8−34

@@ -85,7 +85,7 @@ torture_param(int, holdoff, IS_BUILTIN(CONFIG_RCU_REF_SCALE_TEST) ? 10 : 0, · 84 unmodified lines
85	85	// Number of typesafe_lookup structures, that is, the degree of concurrency.
86	86	torture_param(long, lookup_instances, 0, "Number of typesafe_lookup structures.");
87	87	// Number of loops per experiment, all readers execute operations concurrently.
88		−torture_param(long, loops, 10000, "Number of loops per experiment.");
	88	+torture_param(int, loops, 10000, "Number of loops per experiment.");
89	89	// Number of readers, with -1 defaulting to about 75% of the CPUs.
90	90	torture_param(int, nreaders, -1, "Number of readers, -1 for 75% of CPUs.");
91	91	// Number of runs.
@@ -246,36 +246,6 @@ static const struct ref_scale_ops srcu_fast_ops = { · 154 unmodified lines
246	246	.name = "srcu-fast"
247	247	};
248	248
249		−static void srcu_lite_ref_scale_read_section(const int nloops)
250		−{
251		− int i;
252		− int idx;
253		−
254		− for (i = nloops; i >= 0; i--) {
255		− idx = srcu_read_lock_lite(srcu_ctlp);
256		− srcu_read_unlock_lite(srcu_ctlp, idx);
257		− }
258		−}
259		−
260		−static void srcu_lite_ref_scale_delay_section(const int nloops, const int udl, const int ndl)
261		−{
262		− int i;
263		− int idx;
264		−
265		− for (i = nloops; i >= 0; i--) {
266		− idx = srcu_read_lock_lite(srcu_ctlp);
267		− un_delay(udl, ndl);
268		− srcu_read_unlock_lite(srcu_ctlp, idx);
269		− }
270		−}
271		−
272		−static const struct ref_scale_ops srcu_lite_ops = {
273		− .init = rcu_sync_scale_init,
274		− .readsection = srcu_lite_ref_scale_read_section,
275		− .delaysection = srcu_lite_ref_scale_delay_section,
276		− .name = "srcu-lite"
277		−};
278		−
279	249	#ifdef CONFIG_TASKS_RCU
280	250
281	251	// Definitions for RCU Tasks ref scale testing: Empty read markers.
@@ -1140,7 +1110,7 @@ static void · 858 unmodified lines
1140	1110	ref_scale_print_module_parms(const struct ref_scale_ops cur_ops, const char tag)
1141	1111	{
1142	1112	pr_alert("%s" SCALE_FLAG
1143		− "--- %s: verbose=%d verbose_batched=%d shutdown=%d holdoff=%d lookup_instances=%ld loops=%ld nreaders=%d nruns=%d readdelay=%d\n", scale_type, tag,
	1113	+ "--- %s: verbose=%d verbose_batched=%d shutdown=%d holdoff=%d lookup_instances=%ld loops=%d nreaders=%d nruns=%d readdelay=%d\n", scale_type, tag,
1144	1114	verbose, verbose_batched, shutdown, holdoff, lookup_instances, loops, nreaders, nruns, readdelay);
1145	1115	}
1146	1116
@@ -1193,7 +1163,7 @@ ref_scale_init(void) · 46 unmodified lines
1193	1163	long i;
1194	1164	int firsterr = 0;
1195	1165	static const struct ref_scale_ops *scale_ops[] = {
1196		− &rcu_ops, &srcu_ops, &srcu_fast_ops, &srcu_lite_ops, RCU_TRACE_OPS RCU_TASKS_OPS
	1166	+ &rcu_ops, &srcu_ops, &srcu_fast_ops, RCU_TRACE_OPS RCU_TASKS_OPS
1197	1167	&refcnt_ops, &rwlock_ops, &rwsem_ops, &lock_ops, &lock_irq_ops,
1198	1168	&acqrel_ops, &sched_clock_ops, &clock_ops, &jiffies_ops,
1199	1169	&typesafe_ref_ops, &typesafe_lock_ops, &typesafe_seqlock_ops,
@@ -1238,12 +1208,16 @@ ref_scale_init(void) · 38 unmodified lines
1238	1208	// Reader tasks (default to ~75% of online CPUs).
1239	1209	if (nreaders < 0)
1240	1210	nreaders = (num_online_cpus() >> 1) + (num_online_cpus() >> 2);
1241		− if (WARN_ONCE(loops <= 0, "%s: loops = %ld, adjusted to 1\n", __func__, loops))
	1211	+ if (WARN_ONCE(loops <= 0, "%s: loops = %d, adjusted to 1\n", __func__, loops))
1242	1212	loops = 1;
1243	1213	if (WARN_ONCE(nreaders <= 0, "%s: nreaders = %d, adjusted to 1\n", __func__, nreaders))
1244	1214	nreaders = 1;
1245	1215	if (WARN_ONCE(nruns <= 0, "%s: nruns = %d, adjusted to 1\n", __func__, nruns))
1246	1216	nruns = 1;
	1217	+ if (WARN_ONCE(loops > INT_MAX / nreaders,
	1218	+ "%s: nreaders * loops will overflow, adjusted loops to %d",
	1219	+ __func__, INT_MAX / nreaders))
	1220	+ loops = INT_MAX / nreaders;
1247	1221	reader_tasks = kcalloc(nreaders, sizeof(reader_tasks[0]),
1248	1222	GFP_KERNEL);
1249	1223	if (!reader_tasks) {

kernel/rcu/srcutree.c+2−0

@@ -502,6 +502,8 @@ static bool srcu_readers_active_idx_check(struct srcu_struct *ssp, int idx) · 501 unmodified lines
502	502	*/
503	503	if (!did_gp)
504	504	smp_mb(); /* A */
	505	+ else if (srcu_gp_is_expedited(ssp))
	506	+ synchronize_rcu_expedited(); /* X */
505	507	else
506	508	synchronize_rcu(); /* X */
507	509

kernel/rcu/tree.c+64−16

@@ -160,7 +160,6 @@ static void rcu_report_qs_rnp(unsigned long mask, struct rcu_node *rnp, · 159 unmodified lines
160	160	unsigned long gps, unsigned long flags);
161	161	static void invoke_rcu_core(void);
162	162	static void rcu_report_exp_rdp(struct rcu_data *rdp);
163		−static void sync_sched_exp_online_cleanup(int cpu);
164	163	static void check_cb_ovld_locked(struct rcu_data rdp, struct rcu_node rnp);
165	164	static bool rcu_rdp_is_offloaded(struct rcu_data *rdp);
166	165	static bool rcu_rdp_cpu_online(struct rcu_data *rdp);
@@ -377,7 +376,7 @@ EXPORT_SYMBOL_GPL(rcu_momentary_eqs); · 210 unmodified lines
377	376	*/
378	377	static int rcu_is_cpu_rrupt_from_idle(void)
379	378	{
380		− long nesting;
	379	+ long nmi_nesting = ct_nmi_nesting();
381	380
382	381	/*
383	382	* Usually called from the tick; but also used from smp_function_call()
@@ -389,21 +388,28 @@ static int rcu_is_cpu_rrupt_from_idle(void) · 5 unmodified lines
389	388	/* Check for counter underflows */
390	389	RCU_LOCKDEP_WARN(ct_nesting() < 0,
391	390	"RCU nesting counter underflow!");
392		− RCU_LOCKDEP_WARN(ct_nmi_nesting() <= 0,
393		− "RCU nmi_nesting counter underflow/zero!");
394	391
395		− /* Are we at first interrupt nesting level? */
396		− nesting = ct_nmi_nesting();
397		− if (nesting > 1)
	392	+ /* Non-idle interrupt or nested idle interrupt */
	393	+ if (nmi_nesting > 1)
398	394	return false;
399	395
400	396	/*
401		− * If we're not in an interrupt, we must be in the idle task!
	397	+ * Non nested idle interrupt (interrupting section where RCU
	398	+ * wasn't watching).
402	399	*/
403		− WARN_ON_ONCE(!nesting && !is_idle_task(current));
	400	+ if (nmi_nesting == 1)
	401	+ return true;
	402	+
	403	+ /* Not in an interrupt */
	404	+ if (!nmi_nesting) {
	405	+ RCU_LOCKDEP_WARN(!in_task() \|\| !is_idle_task(current),
	406	+ "RCU nmi_nesting counter not in idle task!");
	407	+ return !rcu_is_watching_curr_cpu();
	408	+ }
404	409
405		− /* Does CPU appear to be idle from an RCU standpoint? */
406		− return ct_nesting() == 0;
	410	+ RCU_LOCKDEP_WARN(1, "RCU nmi_nesting counter underflow/zero!");
	411	+
	412	+ return false;
407	413	}
408	414
409	415	#define DEFAULT_RCU_BLIMIT (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ? 1000 : 10)
@@ -1625,8 +1631,10 @@ static void rcu_sr_put_wait_head(struct llist_node *node) · 1215 unmodified lines
1625	1631	atomic_set_release(&sr_wn->inuse, 0);
1626	1632	}
1627	1633
1628		−/* Disabled by default. */
1629		−static int rcu_normal_wake_from_gp;
	1634	+/* Enable rcu_normal_wake_from_gp automatically on small systems. */
	1635	+#define WAKE_FROM_GP_CPU_THRESHOLD 16
	1636	+
	1637	+static int rcu_normal_wake_from_gp = -1;
1630	1638	module_param(rcu_normal_wake_from_gp, int, 0644);
1631	1639	static struct workqueue_struct *sync_wq;
1632	1640
@@ -1829,6 +1837,18 @@ static noinline_for_stack bool rcu_gp_init(void) · 196 unmodified lines
1829	1837	start_new_poll = rcu_sr_normal_gp_init();
1830	1838	/* Record GP times before starting GP, hence rcu_seq_start(). */
1831	1839	old_gp_seq = rcu_state.gp_seq;
	1840	+ /*
	1841	+ * Critical ordering: rcu_seq_start() must happen BEFORE the CPU hotplug
	1842	+ * scan below. Otherwise we risk a race where a newly onlining CPU could
	1843	+ * be missed by the current grace period, potentially leading to
	1844	+ * use-after-free errors. For a detailed explanation of this race, see
	1845	+ * Documentation/RCU/Design/Requirements/Requirements.rst in the
	1846	+ * "Hotplug CPU" section.
	1847	+ *
	1848	+ * Also note that the root rnp's gp_seq is kept separate from, and lags,
	1849	+ * the rcu_state's gp_seq, for a reason. See the Quick-Quiz on
	1850	+ * Single-node systems for more details (in Data-Structures.rst).
	1851	+ */
1832	1852	rcu_seq_start(&rcu_state.gp_seq);
1833	1853	/* Ensure that rcu_seq_done_exact() guardband doesn't give false positives. */
1834	1854	WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) &&
@@ -1865,6 +1885,10 @@ static noinline_for_stack bool rcu_gp_init(void) · 30 unmodified lines
1865	1885	/* Exclude CPU hotplug operations. */
1866	1886	rcu_for_each_leaf_node(rnp) {
1867	1887	local_irq_disable();
	1888	+ /*
	1889	+ * Serialize with CPU offline. See Requirements.rst > Hotplug CPU >
	1890	+ * Concurrent Quiescent State Reporting for Offline CPUs.
	1891	+ */
1868	1892	arch_spin_lock(&rcu_state.ofl_lock);
1869	1893	raw_spin_lock_rcu_node(rnp);
1870	1894	if (rnp->qsmaskinit == rnp->qsmaskinitnext &&
@@ -1939,7 +1963,12 @@ static noinline_for_stack bool rcu_gp_init(void) · 68 unmodified lines
1939	1963	trace_rcu_grace_period_init(rcu_state.name, rnp->gp_seq,
1940	1964	rnp->level, rnp->grplo,
1941	1965	rnp->grphi, rnp->qsmask);
1942		− /* Quiescent states for tasks on any now-offline CPUs. */
	1966	+ /*
	1967	+ * Quiescent states for tasks on any now-offline CPUs. Since we
	1968	+ * released the ofl and rnp lock before this loop, CPUs might
	1969	+ * have gone offline and we have to report QS on their behalf.
	1970	+ * See Requirements.rst > Hotplug CPU > Concurrent QS Reporting.
	1971	+ */
1943	1972	mask = rnp->qsmask & ~rnp->qsmaskinitnext;
1944	1973	rnp->rcu_gp_init_mask = mask;
1945	1974	if ((mask \|\| rnp->wait_blkd_tasks) && rcu_is_leaf_node(rnp))
@@ -3243,7 +3272,7 @@ static void synchronize_rcu_normal(void) · 1297 unmodified lines
3243	3272
3244	3273	trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request"));
3245	3274
3246		− if (!READ_ONCE(rcu_normal_wake_from_gp)) {
	3275	+ if (READ_ONCE(rcu_normal_wake_from_gp) < 1) {
3247	3276	wait_rcu_gp(call_rcu_hurry);
3248	3277	goto trace_complete_out;
3249	3278	}
@@ -4268,7 +4297,6 @@ int rcutree_online_cpu(unsigned int cpu) · 1018 unmodified lines
4268	4297	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
4269	4298	if (rcu_scheduler_active == RCU_SCHEDULER_INACTIVE)
4270	4299	return 0; /* Too early in boot for scheduler work. */
4271		− sync_sched_exp_online_cleanup(cpu);
4272	4300
4273	4301	// Stop-machine done, so allow nohz_full to disable tick.
4274	4302	tick_dep_clear(TICK_DEP_BIT_RCU);
@@ -4358,6 +4386,12 @@ void rcutree_report_cpu_dead(void) · 83 unmodified lines
4358	4386	* may introduce a new READ-side while it is actually off the QS masks.
4359	4387	*/
4360	4388	lockdep_assert_irqs_disabled();
	4389	+ /*
	4390	+ * CPUHP_AP_SMPCFD_DYING was the last call for rcu_exp_handler() execution.
	4391	+ * The requested QS must have been reported on the last context switch
	4392	+ * from stop machine to idle.
	4393	+ */
	4394	+ WARN_ON_ONCE(rdp->cpu_no_qs.b.exp);
4361	4395	// Do any dangling deferred wakeups.
4362	4396	do_nocb_deferred_wakeup(rdp);
4363	4397
@@ -4365,6 +4399,13 @@ void rcutree_report_cpu_dead(void) · 1 unmodified line
4365	4399
4366	4400	/* Remove outgoing CPU from mask in the leaf rcu_node structure. */
4367	4401	mask = rdp->grpmask;
	4402	+
	4403	+ /*
	4404	+ * Hold the ofl_lock and rnp lock to avoid races between CPU going
	4405	+ * offline and doing a QS report (as below), versus rcu_gp_init().
	4406	+ * See Requirements.rst > Hotplug CPU > Concurrent QS Reporting section
	4407	+ * for more details.
	4408	+ */
4368	4409	arch_spin_lock(&rcu_state.ofl_lock);
4369	4410	raw_spin_lock_irqsave_rcu_node(rnp, flags); /* Enforce GP memory-order guarantee. */
4370	4411	rdp->rcu_ofl_gp_seq = READ_ONCE(rcu_state.gp_seq);
@@ -4375,6 +4416,7 @@ void rcutree_report_cpu_dead(void) · 4 unmodified lines
4375	4416	rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);
4376	4417	raw_spin_lock_irqsave_rcu_node(rnp, flags);
4377	4418	}
	4419	+ /* Clear from ->qsmaskinitnext to mark offline. */
4378	4420	WRITE_ONCE(rnp->qsmaskinitnext, rnp->qsmaskinitnext & ~mask);
4379	4421	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
4380	4422	arch_spin_unlock(&rcu_state.ofl_lock);
@@ -4847,6 +4889,12 @@ void __init rcu_init(void) · 466 unmodified lines
4847	4889	sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM, 0);
4848	4890	WARN_ON(!sync_wq);
4849	4891
	4892	+ /* Respect if explicitly disabled via a boot parameter. */
	4893	+ if (rcu_normal_wake_from_gp < 0) {
	4894	+ if (num_possible_cpus() <= WAKE_FROM_GP_CPU_THRESHOLD)
	4895	+ rcu_normal_wake_from_gp = 1;
	4896	+ }
	4897	+
4850	4898	/* Fill in default value for rcutree.qovld boot parameter. */
4851	4899	/* -After- the rcu_node ->lock fields are initialized! */
4852	4900	if (qovld < 0)

kernel/rcu/tree.h+12−1

@@ -174,6 +174,17 @@ struct rcu_snap_record { · 173 unmodified lines
174	174	unsigned long jiffies; /* Track jiffies value */
175	175	};
176	176
	177	+/*
	178	+ * An IRQ work (deferred_qs_iw) is used by RCU to get the scheduler's attention.
	179	+ * to report quiescent states at the soonest possible time.
	180	+ * The request can be in one of the following states:
	181	+ * - DEFER_QS_IDLE: An IRQ work is yet to be scheduled.
	182	+ * - DEFER_QS_PENDING: An IRQ work was scheduled but either not yet run, or it
	183	+ * ran and we still haven't reported a quiescent state.
	184	+ */
	185	+#define DEFER_QS_IDLE 0
	186	+#define DEFER_QS_PENDING 1
	187	+
177	188	/* Per-CPU data for read-copy update. */
178	189	struct rcu_data {
179	190	/* 1) quiescent-state and grace-period handling : */
@@ -192,7 +203,7 @@ struct rcu_data { · 12 unmodified lines
192	203	/* during and after the last grace */
193	204	/* period it is aware of. */
194	205	struct irq_work defer_qs_iw; /* Obtain later scheduler attention. */
195		− bool defer_qs_iw_pending; /* Scheduler attention pending? */
	206	+ int defer_qs_iw_pending; /* Scheduler attention pending? */
196	207	struct work_struct strict_work; /* Schedule readers for strict GPs. */
197	208
198	209	/* 2) batch handling */

kernel/rcu/tree_exp.h+9−50

@@ -141,6 +141,13 @@ static void __maybe_unused sync_exp_reset_tree(void) · 140 unmodified lines
141	141	raw_spin_lock_irqsave_rcu_node(rnp, flags);
142	142	WARN_ON_ONCE(rnp->expmask);
143	143	WRITE_ONCE(rnp->expmask, rnp->expmaskinit);
	144	+ /*
	145	+ * Need to wait for any blocked tasks as well. Note that
	146	+ * additional blocking tasks will also block the expedited GP
	147	+ * until such time as the ->expmask bits are cleared.
	148	+ */
	149	+ if (rcu_is_leaf_node(rnp) && rcu_preempt_has_tasks(rnp))
	150	+ WRITE_ONCE(rnp->exp_tasks, rnp->blkd_tasks.next);
144	151	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
145	152	}
146	153	}
@@ -393,13 +400,6 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) · 246 unmodified lines
393	400	}
394	401	mask_ofl_ipi = rnp->expmask & ~mask_ofl_test;
395	402
396		− /*
397		− * Need to wait for any blocked tasks as well. Note that
398		− * additional blocking tasks will also block the expedited GP
399		− * until such time as the ->expmask bits are cleared.
400		− */
401		− if (rcu_preempt_has_tasks(rnp))
402		− WRITE_ONCE(rnp->exp_tasks, rnp->blkd_tasks.next);
403	403	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
404	404
405	405	/* IPI the remaining CPUs for expedited quiescent state. */
@@ -751,12 +751,8 @@ static void rcu_exp_handler(void *unused) · 345 unmodified lines
751	751	struct task_struct *t = current;
752	752
753	753	/*
754		− * First, is there no need for a quiescent state from this CPU,
755		− * or is this CPU already looking for a quiescent state for the
756		− * current grace period? If either is the case, just leave.
757		− * However, this should not happen due to the preemptible
758		− * sync_sched_exp_online_cleanup() implementation being a no-op,
759		− * so warn if this does happen.
	754	+ * WARN if the CPU is unexpectedly already looking for a
	755	+ * QS or has already reported one.
760	756	*/
761	757	ASSERT_EXCLUSIVE_WRITER_SCOPED(rdp->cpu_no_qs.b.exp);
762	758	if (WARN_ON_ONCE(!(READ_ONCE(rnp->expmask) & rdp->grpmask) \|\|
@@ -803,11 +799,6 @@ static void rcu_exp_handler(void *unused) · 40 unmodified lines
803	799	WARN_ON_ONCE(1);
804	800	}
805	801
806		−/* PREEMPTION=y, so no PREEMPTION=n expedited grace period to clean up after. */
807		−static void sync_sched_exp_online_cleanup(int cpu)
808		−{
809		−}
810		−
811	802	/*
812	803	* Scan the current list of tasks blocked within RCU read-side critical
813	804	* sections, printing out the tid of each that is blocking the current
@@ -885,38 +876,6 @@ static void rcu_exp_handler(void *unused) · 71 unmodified lines
885	876	rcu_exp_need_qs();
886	877	}
887	878
888		−/* Send IPI for expedited cleanup if needed at end of CPU-hotplug operation. */
889		−static void sync_sched_exp_online_cleanup(int cpu)
890		−{
891		− unsigned long flags;
892		− int my_cpu;
893		− struct rcu_data *rdp;
894		− int ret;
895		− struct rcu_node *rnp;
896		−
897		− rdp = per_cpu_ptr(&rcu_data, cpu);
898		− rnp = rdp->mynode;
899		− my_cpu = get_cpu();
900		− /* Quiescent state either not needed or already requested, leave. */
901		− if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) \|\|
902		− READ_ONCE(rdp->cpu_no_qs.b.exp)) {
903		− put_cpu();
904		− return;
905		− }
906		− /* Quiescent state needed on current CPU, so set it up locally. */
907		− if (my_cpu == cpu) {
908		− local_irq_save(flags);
909		− rcu_exp_need_qs();
910		− local_irq_restore(flags);
911		− put_cpu();
912		− return;
913		− }
914		− /* Quiescent state needed on some other CPU, send IPI. */
915		− ret = smp_call_function_single(cpu, rcu_exp_handler, NULL, 0);
916		− put_cpu();
917		− WARN_ON_ONCE(ret);
918		−}
919		−
920	879	/*
921	880	* Because preemptible RCU does not exist, we never have to check for
922	881	* tasks blocked within RCU read-side critical sections that are

kernel/rcu/tree_nocb.h+6−4

@@ -276,7 +276,7 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, · 275 unmodified lines
276	276	* callback storms, no need to wake up too early.
277	277	*/
278	278	if (waketype == RCU_NOCB_WAKE_LAZY &&
279		− rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) {
	279	+ rdp_gp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) {
280	280	mod_timer(&rdp_gp->nocb_timer, jiffies + rcu_get_jiffies_lazy_flush());
281	281	WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype);
282	282	} else if (waketype == RCU_NOCB_WAKE_BYPASS) {
@@ -1146,7 +1146,6 @@ static bool rcu_nocb_rdp_offload_wait_cond(struct rcu_data *rdp) · 863 unmodified lines
1146	1146	static int rcu_nocb_rdp_offload(struct rcu_data *rdp)
1147	1147	{
1148	1148	int wake_gp;
1149		− struct rcu_data *rdp_gp = rdp->nocb_gp_rdp;
1150	1149
1151	1150	WARN_ON_ONCE(cpu_online(rdp->cpu));
1152	1151	/*
@@ -1156,7 +1155,7 @@ static int rcu_nocb_rdp_offload(struct rcu_data *rdp) · 3 unmodified lines
1156	1155	if (!rdp->nocb_gp_rdp)
1157	1156	return -EINVAL;
1158	1157
1159		− if (WARN_ON_ONCE(!rdp_gp->nocb_gp_kthread))
	1158	+ if (WARN_ON_ONCE(!rdp->nocb_gp_kthread))
1160	1159	return -EINVAL;
1161	1160
1162	1161	pr_info("Offloading %d\n", rdp->cpu);
@@ -1166,7 +1165,7 @@ static int rcu_nocb_rdp_offload(struct rcu_data *rdp) · 3 unmodified lines
1166	1165
1167	1166	wake_gp = rcu_nocb_queue_toggle_rdp(rdp);
1168	1167	if (wake_gp)
1169		− wake_up_process(rdp_gp->nocb_gp_kthread);
	1168	+ wake_up_process(rdp->nocb_gp_kthread);
1170	1169
1171	1170	swait_event_exclusive(rdp->nocb_state_wq,
1172	1171	rcu_nocb_rdp_offload_wait_cond(rdp));
@@ -1564,6 +1563,9 @@ static void show_rcu_nocb_state(struct rcu_data *rdp) · 391 unmodified lines
1564	1563	if (rdp->nocb_gp_rdp == rdp)
1565	1564	show_rcu_nocb_gp_state(rdp);
1566	1565
	1566	+ if (!rcu_segcblist_is_offloaded(&rdp->cblist))
	1567	+ return;
	1568	+
1567	1569	nocb_next_rdp = list_next_or_null_rcu(&rdp->nocb_gp_rdp->nocb_head_rdp,
1568	1570	&rdp->nocb_entry_rdp,
1569	1571	typeof(*rdp),

kernel/rcu/tree_plugin.h+102−20

@@ -486,13 +486,16 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 485 unmodified lines
486	486	struct rcu_node *rnp;
487	487	union rcu_special special;
488	488
	489	+ rdp = this_cpu_ptr(&rcu_data);
	490	+ if (rdp->defer_qs_iw_pending == DEFER_QS_PENDING)
	491	+ rdp->defer_qs_iw_pending = DEFER_QS_IDLE;
	492	+
489	493	/*
490	494	* If RCU core is waiting for this CPU to exit its critical section,
491	495	* report the fact that it has exited. Because irqs are disabled,
492	496	* t->rcu_read_unlock_special cannot change.
493	497	*/
494	498	special = t->rcu_read_unlock_special;
495		− rdp = this_cpu_ptr(&rcu_data);
496	499	if (!special.s && !rdp->cpu_no_qs.b.exp) {
497	500	local_irq_restore(flags);
498	501	return;
@@ -534,7 +537,6 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags) · 35 unmodified lines
534	537	WARN_ON_ONCE(rnp->completedqs == rnp->gp_seq &&
535	538	(!empty_norm \|\| rnp->qsmask));
536	539	empty_exp = sync_rcu_exp_done(rnp);
537		− smp_mb(); /* ensure expedited fastpath sees end of RCU c-s. */
538	540	np = rcu_next_node_entry(t, rnp);
539	541	list_del_init(&t->rcu_node_entry);
540	542	t->rcu_blocked_node = NULL;
@@ -624,10 +626,98 @@ notrace void rcu_preempt_deferred_qs(struct task_struct *t) · 83 unmodified lines
624	626	*/
625	627	static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp)
626	628	{
	629	+ unsigned long flags;
627	630	struct rcu_data *rdp;
628	631
629	632	rdp = container_of(iwp, struct rcu_data, defer_qs_iw);
630		− rdp->defer_qs_iw_pending = false;
	633	+ local_irq_save(flags);
	634	+
	635	+ /*
	636	+ * If the IRQ work handler happens to run in the middle of RCU read-side
	637	+ * critical section, it could be ineffective in getting the scheduler's
	638	+ * attention to report a deferred quiescent state (the whole point of the
	639	+ * IRQ work). For this reason, requeue the IRQ work.
	640	+ *
	641	+ * Basically, we want to avoid following situation:
	642	+ * 1. rcu_read_unlock() queues IRQ work (state -> DEFER_QS_PENDING)
	643	+ * 2. CPU enters new rcu_read_lock()
	644	+ * 3. IRQ work runs but cannot report QS due to rcu_preempt_depth() > 0
	645	+ * 4. rcu_read_unlock() does not re-queue work (state still PENDING)
	646	+ * 5. Deferred QS reporting does not happen.
	647	+ */
	648	+ if (rcu_preempt_depth() > 0)
	649	+ WRITE_ONCE(rdp->defer_qs_iw_pending, DEFER_QS_IDLE);
	650	+
	651	+ local_irq_restore(flags);
	652	+}
	653	+
	654	+/*
	655	+ * Check if expedited grace period processing during unlock is needed.
	656	+ *
	657	+ * This function determines whether expedited handling is required based on:
	658	+ * 1. Task blocking an expedited grace period (based on a heuristic, could be
	659	+ * false-positive, see below.)
	660	+ * 2. CPU participating in an expedited grace period
	661	+ * 3. Strict grace period mode requiring expedited handling
	662	+ * 4. RCU priority deboosting needs when interrupts were disabled
	663	+ *
	664	+ * @t: The task being checked
	665	+ * @rdp: The per-CPU RCU data
	666	+ * @rnp: The RCU node for this CPU
	667	+ * @irqs_were_disabled: Whether interrupts were disabled before rcu_read_unlock()
	668	+ *
	669	+ * Returns true if expedited processing of the rcu_read_unlock() is needed.
	670	+ */
	671	+static bool rcu_unlock_needs_exp_handling(struct task_struct *t,
	672	+ struct rcu_data *rdp,
	673	+ struct rcu_node *rnp,
	674	+ bool irqs_were_disabled)
	675	+{
	676	+ /*
	677	+ * Check if this task is blocking an expedited grace period. If the
	678	+ * task was preempted within an RCU read-side critical section and is
	679	+ * on the expedited grace period blockers list (exp_tasks), we need
	680	+ * expedited handling to unblock the expedited GP. This is not an exact
	681	+ * check because 't' might not be on the exp_tasks list at all - its
	682	+ * just a fast heuristic that can be false-positive sometimes.
	683	+ */
	684	+ if (t->rcu_blocked_node && READ_ONCE(t->rcu_blocked_node->exp_tasks))
	685	+ return true;
	686	+
	687	+ /*
	688	+ * Check if this CPU is participating in an expedited grace period.
	689	+ * The expmask bitmap tracks which CPUs need to check in for the
	690	+ * current expedited GP. If our CPU's bit is set, we need expedited
	691	+ * handling to help complete the expedited GP.
	692	+ */
	693	+ if (rdp->grpmask & READ_ONCE(rnp->expmask))
	694	+ return true;
	695	+
	696	+ /*
	697	+ * In CONFIG_RCU_STRICT_GRACE_PERIOD=y kernels, all grace periods
	698	+ * are treated as short for testing purposes even if that means
	699	+ * disturbing the system more. Check if either:
	700	+ * - This CPU has not yet reported a quiescent state, or
	701	+ * - This task was preempted within an RCU critical section
	702	+ * In either case, require expedited handling for strict GP mode.
	703	+ */
	704	+ if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) &&
	705	+ ((rdp->grpmask & READ_ONCE(rnp->qsmask)) \|\| t->rcu_blocked_node))
	706	+ return true;
	707	+
	708	+ /*
	709	+ * RCU priority boosting case: If a task is subject to RCU priority
	710	+ * boosting and exits an RCU read-side critical section with interrupts
	711	+ * disabled, we need expedited handling to ensure timely deboosting.
	712	+ * Without this, a low-priority task could incorrectly run at high
	713	+ * real-time priority for an extended period degrading real-time
	714	+ * responsiveness. This applies to all CONFIG_RCU_BOOST=y kernels,
	715	+ * not just to PREEMPT_RT.
	716	+ */
	717	+ if (IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled && t->rcu_blocked_node)
	718	+ return true;
	719	+
	720	+ return false;
631	721	}
632	722
633	723	/*
@@ -649,18 +739,14 @@ static void rcu_read_unlock_special(struct task_struct *t) · 15 unmodified lines
649	739	local_irq_save(flags);
650	740	irqs_were_disabled = irqs_disabled_flags(flags);
651	741	if (preempt_bh_were_disabled \|\| irqs_were_disabled) {
652		− bool expboost; // Expedited GP in flight or possible boosting.
	742	+ bool needs_exp; // Expedited handling needed.
653	743	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
654	744	struct rcu_node *rnp = rdp->mynode;
655	745
656		− expboost = (t->rcu_blocked_node && READ_ONCE(t->rcu_blocked_node->exp_tasks)) \|\|
657		− (rdp->grpmask & READ_ONCE(rnp->expmask)) \|\|
658		− (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) &&
659		− ((rdp->grpmask & READ_ONCE(rnp->qsmask)) \|\| t->rcu_blocked_node)) \|\|
660		− (IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled &&
661		− t->rcu_blocked_node);
	746	+ needs_exp = rcu_unlock_needs_exp_handling(t, rdp, rnp, irqs_were_disabled);
	747	+
662	748	// Need to defer quiescent state until everything is enabled.
663		− if (use_softirq && (in_hardirq() \|\| (expboost && !irqs_were_disabled))) {
	749	+ if (use_softirq && (in_hardirq() \|\| (needs_exp && !irqs_were_disabled))) {
664	750	// Using softirq, safe to awaken, and either the
665	751	// wakeup is free or there is either an expedited
666	752	// GP in flight or a potential need to deboost.
@@ -673,17 +759,13 @@ static void rcu_read_unlock_special(struct task_struct *t) · 6 unmodified lines
673	759	set_tsk_need_resched(current);
674	760	set_preempt_need_resched();
675	761	if (IS_ENABLED(CONFIG_IRQ_WORK) && irqs_were_disabled &&
676		− expboost && !rdp->defer_qs_iw_pending && cpu_online(rdp->cpu)) {
	762	+ needs_exp && rdp->defer_qs_iw_pending != DEFER_QS_PENDING &&
	763	+ cpu_online(rdp->cpu)) {
677	764	// Get scheduler to re-evaluate and call hooks.
678	765	// If !IRQ_WORK, FQS scan will eventually IPI.
679		− if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) &&
680		− IS_ENABLED(CONFIG_PREEMPT_RT))
681		− rdp->defer_qs_iw = IRQ_WORK_INIT_HARD(
682		− rcu_preempt_deferred_qs_handler);
683		− else
684		− init_irq_work(&rdp->defer_qs_iw,
685		− rcu_preempt_deferred_qs_handler);
686		− rdp->defer_qs_iw_pending = true;
	766	+ rdp->defer_qs_iw =
	767	+ IRQ_WORK_INIT_HARD(rcu_preempt_deferred_qs_handler);
	768	+ rdp->defer_qs_iw_pending = DEFER_QS_PENDING;
687	769	irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
688	770	}
689	771	}

kernel/rcu/tree_stall.h+1−2

@@ -982,8 +982,7 @@ void show_rcu_gp_kthreads(void) · 981 unmodified lines
982	982	for_each_possible_cpu(cpu) {
983	983	rdp = per_cpu_ptr(&rcu_data, cpu);
984	984	cbs += data_race(READ_ONCE(rdp->n_cbs_invoked));
985		− if (rcu_segcblist_is_offloaded(&rdp->cblist))
986		− show_rcu_nocb_state(rdp);
	985	+ show_rcu_nocb_state(rdp);
987	986	}
988	987	pr_info("RCU callbacks invoked since boot: %lu\n", cbs);
989	988	show_rcu_tasks_gp_kthreads();

scripts/checkpatch.pl+0−2

@@ -857,8 +857,6 @@ our %deprecated_apis = ( · 856 unmodified lines
857	857	"kunmap" => "kunmap_local",
858	858	"kmap_atomic" => "kmap_local_page",
859	859	"kunmap_atomic" => "kunmap_local",
860		− "srcu_read_lock_lite" => "srcu_read_lock_fast",
861		− "srcu_read_unlock_lite" => "srcu_read_unlock_fast",
862	860	);
863	861
864	862	#Create a search pattern for all these strings to speed up a loop below

tools/testing/selftests/rcutorture/bin/kvm-build.sh+1−1

@@ -44,7 +44,7 @@ fi · 43 unmodified lines
44	44	ncpus="`getconf _NPROCESSORS_ONLN`"
45	45	make -j$((2 * ncpus)) $TORTURE_KMAKE_ARG > $resdir/Make.out 2>&1
46	46	retval=$?
47		−if test $retval -ne 0 \|\| grep "rcu[^/]*": < $resdir/Make.out \| grep -E -q "Stop\|Error\|error:\|warning:" \|\| grep -E -q "Stop\|Error\|error:" < $resdir/Make.out
	47	+if test $retval -ne 0 \|\| grep "rcu[^/]*": < $resdir/Make.out \| grep -E -q "Stop\|ERROR\|Error\|error:\|warning:" \|\| grep -E -q "Stop\|ERROR\|Error\|error:" < $resdir/Make.out
48	48	then
49	49	echo Kernel build error
50	50	grep -E "Stop\|Error\|error:\|warning:" < $resdir/Make.out

tools/testing/selftests/rcutorture/bin/kvm.sh+2−13

@@ -199,7 +199,7 @@ do · 198 unmodified lines
199	199	fi
200	200	;;
201	201	--kconfig\|--kconfigs)
202		− checkarg --kconfig "(Kconfig options)" $# "$2" '^$#CHECK#$\?CONFIG_[A-Z0-9_]\+=$[ynm]\\|[0-9]\+\\|"[^"]"$$ \(#CHECK#$\?CONFIG_[A-Z0-9_]\+=$[ynm]\\|[0-9]\+\\|"[^"]"$\)*$' '^error$'
	202	+ checkarg --kconfig "(Kconfig options)" $# "$2" '^$#CHECK#$\?CONFIG_[A-Z0-9_]\+=$[ynm]\\|[0-9]\+\\|"[^"]"$$ \+\(#CHECK#$\?CONFIG_[A-Z0-9_]\+=$[ynm]\\|[0-9]\+\\|"[^"]"$\)* *$' '^error$'
203	203	TORTURE_KCONFIG_ARG="`echo "$TORTURE_KCONFIG_ARG $2" \| sed -e 's/^ //' -e 's/ $//'`"
204	204	shift
205	205	;;
@@ -442,18 +442,7 @@ echo $scriptname $args · 236 unmodified lines
442	442	touch $resdir/$ds/log
443	443	echo $scriptname $args >> $resdir/$ds/log
444	444	echo ${TORTURE_SUITE} > $resdir/$ds/torture_suite
445		−echo Build directory: `pwd` > $resdir/$ds/testid.txt
446		−if test -d .git
447		−then
448		− echo Current commit: `git rev-parse HEAD` >> $resdir/$ds/testid.txt
449		− echo >> $resdir/$ds/testid.txt
450		− echo ' ---' Output of "'"git status"'": >> $resdir/$ds/testid.txt
451		− git status >> $resdir/$ds/testid.txt
452		− echo >> $resdir/$ds/testid.txt
453		− echo >> $resdir/$ds/testid.txt
454		− echo ' ---' Output of "'"git diff HEAD"'": >> $resdir/$ds/testid.txt
455		− git diff HEAD >> $resdir/$ds/testid.txt
456		−fi
	445	+mktestid.sh $resdir/$ds
457	446	___EOF___
458	447	kvm-assign-cpus.sh /sys/devices/system/node > $T/cpuarray.awk
459	448	kvm-get-cpus-script.sh $T/cpuarray.awk $T/dumpbatches.awk

tools/testing/selftests/rcutorture/bin/mktestid.sh+29−0

@@ -0,0 +1,29 @@
	1	+#!/bin/bash
	2	+# SPDX-License-Identifier: GPL-2.0+
	3	+#
	4	+# Create a testid.txt file in the specified directory.
	5	+#
	6	+# Usage: mktestid.sh dirpath
	7	+#
	8	+# Copyright (C) Meta Platforms, Inc. 2025
	9	+#
	10	+# Author: Paul E. McKenney <paulmck@kernel.org>
	11	+
	12	+resdir="$1"
	13	+if test -z "${resdir}" \|\| ! test -d "${resdir}" \|\| ! test -w "${resdir}"
	14	+then
	15	+ echo Path '"'${resdir}'"' not writeable directory, no ${resdir}/testid.txt.
	16	+ exit 1
	17	+fi
	18	+echo Build directory: `pwd` > ${resdir}/testid.txt
	19	+if test -d .git
	20	+then
	21	+ echo Current commit: `git rev-parse HEAD` >> ${resdir}/testid.txt
	22	+ echo >> ${resdir}/testid.txt
	23	+ echo ' ---' Output of "'"git status"'": >> ${resdir}/testid.txt
	24	+ git status >> ${resdir}/testid.txt
	25	+ echo >> ${resdir}/testid.txt
	26	+ echo >> ${resdir}/testid.txt
	27	+ echo ' ---' Output of "'"git diff HEAD"'": >> ${resdir}/testid.txt
	28	+ git diff HEAD >> ${resdir}/testid.txt
	29	+fi

tools/testing/selftests/rcutorture/bin/torture.sh+62−16

@@ -30,6 +30,15 @@ then · 29 unmodified lines
30	30	VERBOSE_BATCH_CPUS=0
31	31	fi
32	32
	33	+# Machine architecture? ("uname -p" is said to be less portable.)1
	34	+thisarch="`uname -m`"
	35	+if test "${thisarch}" = aarch64
	36	+then
	37	+ ifnotaarch64=no
	38	+else
	39	+ ifnotaarch64=yes
	40	+fi
	41	+
33	42	# Configurations/scenarios.
34	43	configs_rcutorture=
35	44	configs_locktorture=
@@ -55,9 +64,9 @@ do_normal=yes · 19 unmodified lines
55	64	explicit_normal=no
56	65	do_kasan=yes
57	66	do_kcsan=no
58		−do_clocksourcewd=yes
	67	+do_clocksourcewd="${ifnotaarch64}"
59	68	do_rt=yes
60		−do_rcutasksflavors=yes
	69	+do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
61	70	do_srcu_lockdep=yes
62	71	do_rcu_rust=no
63	72
@@ -124,7 +133,7 @@ do · 60 unmodified lines
124	133	;;
125	134	--do-all\|--doall)
126	135	do_allmodconfig=yes
127		− do_rcutasksflavor=yes
	136	+ do_rcutasksflavors="${ifnotaarch64}" # FIXME: Back to "yes" when SMP=n auto-avoided
128	137	do_rcutorture=yes
129	138	do_locktorture=yes
130	139	do_scftorture=yes
@@ -136,7 +145,7 @@ do · 5 unmodified lines
136	145	explicit_normal=no
137	146	do_kasan=yes
138	147	do_kcsan=yes
139		− do_clocksourcewd=yes
	148	+ do_clocksourcewd="${ifnotaarch64}"
140	149	do_srcu_lockdep=yes
141	150	;;
142	151	--do-allmodconfig\|--do-no-allmodconfig\|--no-allmodconfig)
@@ -274,7 +283,7 @@ then · 131 unmodified lines
274	283	configs_rcutorture=CFLIST
275	284	fi
276	285	duration_rcutorture=$((duration_base*duration_rcutorture_frac/10))
277		−if test "$duration_rcutorture" -eq 0
	286	+if test "$duration_rcutorture" -eq 0 && test "$do_locktorture" = "yes"
278	287	then
279	288	echo " --- Zero time for rcutorture, disabling" \| tee -a $T/log
280	289	do_rcutorture=no
@@ -286,7 +295,7 @@ then · 5 unmodified lines
286	295	configs_locktorture=CFLIST
287	296	fi
288	297	duration_locktorture=$((duration_base*duration_locktorture_frac/10))
289		−if test "$duration_locktorture" -eq 0
	298	+if test "$duration_locktorture" -eq 0 && test "$do_locktorture" = "yes"
290	299	then
291	300	echo " --- Zero time for locktorture, disabling" \| tee -a $T/log
292	301	do_locktorture=no
@@ -298,12 +307,19 @@ then · 5 unmodified lines
298	307	configs_scftorture=CFLIST
299	308	fi
300	309	duration_scftorture=$((duration_base*duration_scftorture_frac/10))
301		−if test "$duration_scftorture" -eq 0
	310	+if test "$duration_scftorture" -eq 0 && test "$do_scftorture" = "yes"
302	311	then
303	312	echo " --- Zero time for scftorture, disabling" \| tee -a $T/log
304	313	do_scftorture=no
305	314	fi
306	315
	316	+# CONFIG_EXPERT=y is currently required for arm64 KCSAN runs.
	317	+kcsan_expert=
	318	+if test "${thisarch}" = aarch64
	319	+then
	320	+ kcsan_expert="CONFIG_EXPERT=y"
	321	+fi
	322	+
307	323	touch $T/failures
308	324	touch $T/successes
309	325
@@ -362,13 +378,19 @@ function torture_set { · 52 unmodified lines
362	378	then
363	379	curflavor=$flavor
364	380	torture_one "$@"
365		− mv $T/last-resdir $T/last-resdir-nodebug \|\| :
	381	+ if test -e $T/last-resdir
	382	+ then
	383	+ mv $T/last-resdir $T/last-resdir-nodebug \|\| :
	384	+ fi
366	385	fi
367	386	if test "$do_kasan" = "yes"
368	387	then
369	388	curflavor=${flavor}-kasan
370	389	torture_one "$@" --kasan
371		− mv $T/last-resdir $T/last-resdir-kasan \|\| :
	390	+ if test -e $T/last-resdir
	391	+ then
	392	+ mv $T/last-resdir $T/last-resdir-kasan \|\| :
	393	+ fi
372	394	fi
373	395	if test "$do_kcsan" = "yes"
374	396	then
@@ -378,8 +400,16 @@ function torture_set { · 3 unmodified lines
378	400	kcsan_kmake_tag="--kmake-args"
379	401	cur_kcsan_kmake_args="$kcsan_kmake_args"
380	402	fi
381		− torture_one "$@" --kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y" $kcsan_kmake_tag $cur_kcsan_kmake_args --kcsan
382		− mv $T/last-resdir $T/last-resdir-kcsan \|\| :
	403	+ chk_rdr_state=
	404	+ if test "${flavor}" = rcutorture
	405	+ then
	406	+ chk_rdr_state="CONFIG_RCU_TORTURE_TEST_CHK_RDR_STATE=y"
	407	+ fi
	408	+ torture_one "$@" --kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y ${kcsan_expert} ${chk_rdr_state}" $kcsan_kmake_tag $cur_kcsan_kmake_args --kcsan
	409	+ if test -e $T/last-resdir
	410	+ then
	411	+ mv $T/last-resdir $T/last-resdir-kcsan \|\| :
	412	+ fi
383	413	fi
384	414	}
385	415
@@ -389,6 +419,7 @@ then · 3 unmodified lines
389	419	echo " --- allmodconfig:" Start `date` \| tee -a $T/log
390	420	amcdir="tools/testing/selftests/rcutorture/res/$ds/allmodconfig"
391	421	mkdir -p "$amcdir"
	422	+ mktestid.sh "$amcdir"
392	423	echo " --- make clean" \| tee $amcdir/log > "$amcdir/Make.out" 2>&1
393	424	make -j$MAKE_ALLOTED_CPUS clean >> "$amcdir/Make.out" 2>&1
394	425	retcode=$?
@@ -407,6 +438,10 @@ then · 12 unmodified lines
407	438	make -j$MAKE_ALLOTED_CPUS >> "$amcdir/Make.out" 2>&1
408	439	retcode="$?"
409	440	echo $retcode > "$amcdir/Make.exitcode"
	441	+ if grep -E -q "Stop\|ERROR\|Error\|error:\|warning:" < "$amcdir/Make.out"
	442	+ then
	443	+ retcode=99
	444	+ fi
410	445	buildphase='"make"'
411	446	fi
412	447	if test "$retcode" -eq 0
@@ -495,6 +530,7 @@ then · 82 unmodified lines
495	530	echo " --- do-rcu-rust:" Start `date` \| tee -a $T/log
496	531	rrdir="tools/testing/selftests/rcutorture/res/$ds/results-rcu-rust"
497	532	mkdir -p "$rrdir"
	533	+ mktestid.sh "$rrdir"
498	534	echo " --- make LLVM=1 rustavailable " \| tee -a $rrdir/log > $rrdir/rustavailable.out
499	535	make LLVM=1 rustavailable > $T/rustavailable.out 2>&1
500	536	retcode=$?
@@ -681,7 +717,14 @@ nfailures=0 · 180 unmodified lines
681	717	echo FAILURES: \| tee -a $T/log
682	718	if test -s "$T/failures"
683	719	then
684		− awk < "$T/failures" -v sq="'" '{ print "echo " sq $0 sq; print "sed -e " sq "1,/^ --- .* Test summary:$/d" sq " " $2 "/log \| grep Summary: \| sed -e " sq "s/^[^S]*/ /" sq; }' \| sh \| tee -a $T/log \| tee "$T/failuresum"
	720	+ awk < "$T/failures" -v sq="'" '
	721	+ {
	722	+ print "echo " sq $0 sq;
	723	+ if ($2 != "")
	724	+ print "sed -e " sq "1,/^ --- .* Test summary:$/d" sq " " $2 "/log \| grep Summary: \| sed -e " sq "s/^[^S]*/ /" sq;
	725	+ else
	726	+ print "echo " sq " " sq "Run failed to produce results directory.";
	727	+ }' \| sh \| tee -a $T/log \| tee "$T/failuresum"
685	728	nfailures="`wc -l "$T/failures" \| awk '{ print $1 }'`"
686	729	grep "^ Summary: " "$T/failuresum" \|
687	730	grep -v '^ Summary: Bugs: [0-9]* (all bugs kcsan)$' > "$T/nonkcsan"
@@ -691,15 +734,18 @@ then · 3 unmodified lines
691	734	fi
692	735	ret=2
693	736	fi
694		−if test "$do_kcsan" = "yes"
	737	+if test "$do_kcsan" = "yes" && test -e tools/testing/selftests/rcutorture/res/$ds
695	738	then
696	739	TORTURE_KCONFIG_KCSAN_ARG=1 tools/testing/selftests/rcutorture/bin/kcsan-collapse.sh tools/testing/selftests/rcutorture/res/$ds > tools/testing/selftests/rcutorture/res/$ds/kcsan.sum
697	740	fi
698	741	echo Started at $startdate, ended at `date`, duration `get_starttime_duration $starttime`. \| tee -a $T/log
699	742	echo Summary: Successes: $nsuccesses Failures: $nfailures. \| tee -a $T/log
700		−tdir="`cat $T/successes $T/failures \| head -1 \| awk '{ print $NF }' \| sed -e 's,/[^/]\+/*$,,'`"
701		−find "$tdir" -name 'ConfigFragment.diags' -print > $T/configerrors
702		−find "$tdir" -name 'Make.out.diags' -print > $T/builderrors
	743	+tdir="`cat $T/successes $T/failures \| awk 'NF > 1 { print $NF }' \| head -1 \| sed -e 's,/[^/]\+/*$,,'`"
	744	+if test -n "$tdir"
	745	+then
	746	+ find "$tdir" -name 'ConfigFragment.diags' -print > $T/configerrors
	747	+ find "$tdir" -name 'Make.out.diags' -print > $T/builderrors
	748	+fi
703	749	if test -s "$T/configerrors"
704	750	then
705	751	echo " Scenarios with .config errors: `wc -l "$T/configerrors" \| awk '{ print $1 }'`"

tools/testing/selftests/rcutorture/configs/rcu/BUSTED+3−0

@@ -5,3 +5,6 @@ CONFIG_HOTPLUG_CPU=y · 4 unmodified lines
5	5	CONFIG_PREEMPT_NONE=n
6	6	CONFIG_PREEMPT_VOLUNTARY=n
7	7	CONFIG_PREEMPT=y
	8	+CONFIG_RCU_TORTURE_TEST_CHK_RDR_STATE=y
	9	+CONFIG_RCU_TORTURE_TEST_LOG_CPU=y
	10	+CONFIG_RCU_TORTURE_TEST_LOG_GP=y

tools/testing/selftests/rcutorture/configs/rcu/CFLIST+0−1

@@ -5,7 +5,6 @@ TREE04 · 4 unmodified lines
5	5	TREE05
6	6	TREE07
7	7	TREE09
8		−SRCU-L
9	8	SRCU-N
10	9	SRCU-P
11	10	SRCU-T

tools/testing/selftests/rcutorture/configs/rcu/SRCU-L+0−10

@@ -1,10 +0,0 @@
1		−CONFIG_RCU_TRACE=n
2		−CONFIG_SMP=y
3		−CONFIG_NR_CPUS=6
4		−CONFIG_HOTPLUG_CPU=y
5		−CONFIG_PREEMPT_NONE=y
6		−CONFIG_PREEMPT_VOLUNTARY=n
7		−CONFIG_PREEMPT=n
8		−#CHECK#CONFIG_RCU_EXPERT=n
9		−CONFIG_KPROBES=n
10		−CONFIG_FTRACE=n

tools/testing/selftests/rcutorture/configs/rcu/SRCU-L.boot+0−3

@@ -1,3 +0,0 @@
1		−rcutorture.torture_type=srcu
2		−rcutorture.reader_flavor=0x4
3		−rcutorture.fwd_progress=3

Merge tag 'rcu.release.v6.17' of git://git.kernel.org/pub/scm/linux/kernel/git/rcu/linux

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code