Merge tag 'wq-for-6.9' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq

Pull workqueue updates from Tejun Heo: "This cycle, a lot of workqueue changes including some that are significant and invasive. - During v6.6 cycle, unbound workqueues were updated so that they are more topology aware and flexible, which among other things improved workqueue behavior on modern multi-L3 CPUs. In the process, commit 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") switched unbound workqueues to use per-CPU frontend pool_workqueues as a part of increasing front-back mapping flexibility. An unwelcome side effect of this change was that this made max concurrency enforcement per-CPU blowing up the maximum number of allowed concurrent executions. I incorrectly assumed that this wouldn't cause practical problems as most unbound workqueue users are self-regulate max concurrency; however, there definitely are which don't (e.g. on IO paths) and the drastic increase in the allowed max concurrency led to noticeable perf regressions in some use cases. This is now addressed by separating out max concurrency enforcement to a separate struct - wq_node_nr_active - which makes @max_active consistently mean system-wide max concurrency regardless of the number of CPUs or (finally) NUMA nodes. This is a rather invasive and, in places, a bit clunky; however, the clunkiness rises from the the inherent requirement to handle the disagreement between the execution locality domain and max concurrency enforcement domain on some modern machines. See commit 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues") for more details. - BH workqueue support is added. They are similar to per-CPU workqueues but execute work items in the softirq context. This is expected to replace tasklet. However, currently, it's missing the ability to disable and enable work items which is needed to convert many tasklet users. To avoid crowding this merge window too much, this will be included in the next merge window. A separate pull request will be sent for the couple conversion patches that are currently pending. - Waiman plugged a long-standing hole in workqueue CPU isolation where ordered workqueues didn't follow wq_unbound_cpumask updates. Ordered workqueues now follow the same rules as other unbound workqueues. - More CPU isolation improvements: Juri fixed another deficit in workqueue isolation where unbound rescuers don't respect wq_unbound_cpumask. Leonardo fixed delayed_work timers firing on isolated CPUs. - Other misc changes" * tag 'wq-for-6.9' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq: (54 commits) workqueue: Drain BH work items on hot-unplugged CPUs workqueue: Introduce from_work() helper for cleaner callback declarations workqueue: Control intensive warning threshold through cmdline workqueue: Make @flags handling consistent across set_work_data() and friends workqueue: Remove clear_work_data() workqueue: Factor out work_grab_pending() from __cancel_work_sync() workqueue: Clean up enum work_bits and related constants workqueue: Introduce work_cancel_flags workqueue: Use variable name irq_flags for saving local irq flags workqueue: Reorganize flush and cancel[_sync] functions workqueue: Rename __cancel_work_timer() to __cancel_timer_sync() workqueue: Use rcu_read_lock_any_held() instead of rcu_read_lock_held() workqueue: Cosmetic changes workqueue, irq_work: Build fix for !CONFIG_IRQ_WORK workqueue: Fix queue_work_on() with BH workqueues async: Use a dedicated unbound workqueue with raised min_active workqueue: Implement workqueue_set_min_active() workqueue: Fix kernel-doc comment of unplug_oldest_pwq() workqueue: Bind unbound workqueue rescuer to wq_unbound_cpumask kernel/workqueue: Let rescuers follow unbound wq cpumask changes ...

Linus Torvalds committed 2 years ago

Commit ff887eb2 parents 8ede842 1acd92d11 files · +1690 −530Browse files

Unified · Split

11 files

Documentation/admin-guide/kernel-parameters.txt +9 −0
Documentation/core-api/workqueue.rst +29 −14
include/linux/async.h +1 −0
include/linux/workqueue.h +99 −42
init/Kconfig +1 −1
init/main.c +1 −0
kernel/async.c +16 −1
kernel/softirq.c +5 −0
kernel/workqueue.c +1434 −457
rust/kernel/workqueue.rs +5 −1
tools/workqueue/wq_dump.py +90 −14

Documentation/admin-guide/kernel-parameters.txt+9−0

@@ -7244,6 +7244,15 @@ · 7243 unmodified lines
7244	7244	threshold repeatedly. They are likely good
7245	7245	candidates for using WQ_UNBOUND workqueues instead.
7246	7246
	7247	+ workqueue.cpu_intensive_warning_thresh=<uint>
	7248	+ If CONFIG_WQ_CPU_INTENSIVE_REPORT is set, the kernel
	7249	+ will report the work functions which violate the
	7250	+ intensive_threshold_us repeatedly. In order to prevent
	7251	+ spurious warnings, start printing only after a work
	7252	+ function has violated this threshold number of times.
	7253	+
	7254	+ The default is 4 times. 0 disables the warning.
	7255	+
7247	7256	workqueue.power_efficient
7248	7257	Per-cpu workqueues are generally preferred because
7249	7258	they show better performance thanks to cache

Documentation/core-api/workqueue.rst+29−14

@@ -77,10 +77,12 @@ wants a function to be executed asynchronously it has to set up a work · 76 unmodified lines
77	77	item pointing to that function and queue that work item on a
78	78	workqueue.
79	79
80		−Special purpose threads, called worker threads, execute the functions
81		−off of the queue, one after the other. If no work is queued, the
82		−worker threads become idle. These worker threads are managed in so
83		−called worker-pools.
	80	+A work item can be executed in either a thread or the BH (softirq) context.
	81	+
	82	+For threaded workqueues, special purpose threads, called [k]workers, execute
	83	+the functions off of the queue, one after the other. If no work is queued,
	84	+the worker threads become idle. These worker threads are managed in
	85	+worker-pools.
84	86
85	87	The cmwq design differentiates between the user-facing workqueues that
86	88	subsystems and drivers queue work items on and the backend mechanism
@@ -91,6 +93,12 @@ for high priority ones, for each possible CPU and some extra · 4 unmodified lines
91	93	worker-pools to serve work items queued on unbound workqueues - the
92	94	number of these backing pools is dynamic.
93	95
	96	+BH workqueues use the same framework. However, as there can only be one
	97	+concurrent execution context, there's no need to worry about concurrency.
	98	+Each per-CPU BH worker pool contains only one pseudo worker which represents
	99	+the BH execution context. A BH workqueue can be considered a convenience
	100	+interface to softirq.
	101	+
94	102	Subsystems and drivers can create and queue work items through special
95	103	workqueue API functions as they see fit. They can influence some
96	104	aspects of the way the work items are executed by setting flags on the
@@ -106,7 +114,7 @@ unless specifically overridden, a work item of a bound workqueue will · 9 unmodified lines
106	114	be queued on the worklist of either normal or highpri worker-pool that
107	115	is associated to the CPU the issuer is running on.
108	116
109		−For any worker pool implementation, managing the concurrency level
	117	+For any thread pool implementation, managing the concurrency level
110	118	(how many execution contexts are active) is an important issue. cmwq
111	119	tries to keep the concurrency at a minimal but sufficient level.
112	120	Minimal to save resources and sufficient in that the system is used at
@@ -164,6 +172,17 @@ resources, scheduled and executed. · 51 unmodified lines
164	172	``flags``
165	173	---------
166	174
	175	+``WQ_BH``
	176	+ BH workqueues can be considered a convenience interface to softirq. BH
	177	+ workqueues are always per-CPU and all BH work items are executed in the
	178	+ queueing CPU's softirq context in the queueing order.
	179	+
	180	+ All BH workqueues must have 0 ``max_active`` and ``WQ_HIGHPRI`` is the
	181	+ only allowed additional flag.
	182	+
	183	+ BH work items cannot sleep. All other features such as delayed queueing,
	184	+ flushing and canceling are supported.
	185	+
167	186	``WQ_UNBOUND``
168	187	Work items queued to an unbound wq are served by the special
169	188	worker-pools which host workers which are not bound to any
@@ -237,15 +256,11 @@ may queue at the same time. Unless there is a specific need for · 67 unmodified lines
237	256	throttling the number of active work items, specifying '0' is
238	257	recommended.
239	258
240		−Some users depend on the strict execution ordering of ST wq. The
241		−combination of ``@max_active`` of 1 and ``WQ_UNBOUND`` used to
242		−achieve this behavior. Work items on such wq were always queued to the
243		−unbound worker-pools and only one work item could be active at any given
244		−time thus achieving the same ordering property as ST wq.
245		−
246		−In the current implementation the above configuration only guarantees
247		−ST behavior within a given NUMA node. Instead ``alloc_ordered_workqueue()`` should
248		−be used to achieve system-wide ST behavior.
	259	+Some users depend on strict execution ordering where only one work item
	260	+is in flight at any given time and the work items are processed in
	261	+queueing order. While the combination of ``@max_active`` of 1 and
	262	+``WQ_UNBOUND`` used to achieve this behavior, this is no longer the
	263	+case. Use ``alloc_ordered_queue()`` instead.
249	264
250	265
251	266	Example Execution Scenarios

include/linux/async.h+1−0

@@ -120,4 +120,5 @@ extern void async_synchronize_cookie(async_cookie_t cookie); · 119 unmodified lines
120	120	extern void async_synchronize_cookie_domain(async_cookie_t cookie,
121	121	struct async_domain *domain);
122	122	extern bool current_is_async(void);
	123	+extern void async_init(void);
123	124	#endif

include/linux/workqueue.h+99−42

@@ -22,20 +22,54 @@ · 21 unmodified lines
22	22	*/
23	23	#define work_data_bits(work) ((unsigned long *)(&(work)->data))
24	24
25		−enum {
	25	+enum work_bits {
26	26	WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */
27		− WORK_STRUCT_INACTIVE_BIT= 1, /* work item is inactive */
28		− WORK_STRUCT_PWQ_BIT = 2, /* data points to pwq */
29		− WORK_STRUCT_LINKED_BIT = 3, /* next work is linked to this one */
	27	+ WORK_STRUCT_INACTIVE_BIT, /* work item is inactive */
	28	+ WORK_STRUCT_PWQ_BIT, /* data points to pwq */
	29	+ WORK_STRUCT_LINKED_BIT, /* next work is linked to this one */
30	30	#ifdef CONFIG_DEBUG_OBJECTS_WORK
31		− WORK_STRUCT_STATIC_BIT = 4, /* static initializer (debugobjects) */
32		− WORK_STRUCT_COLOR_SHIFT = 5, /* color for workqueue flushing */
33		−#else
34		− WORK_STRUCT_COLOR_SHIFT = 4, /* color for workqueue flushing */
	31	+ WORK_STRUCT_STATIC_BIT, /* static initializer (debugobjects) */
35	32	#endif
	33	+ WORK_STRUCT_FLAG_BITS,
36	34
	35	+ /* color for workqueue flushing */
	36	+ WORK_STRUCT_COLOR_SHIFT = WORK_STRUCT_FLAG_BITS,
37	37	WORK_STRUCT_COLOR_BITS = 4,
38	38
	39	+ /*
	40	+ * When WORK_STRUCT_PWQ is set, reserve 8 bits off of pwq pointer w/
	41	+ * debugobjects turned off. This makes pwqs aligned to 256 bytes (512
	42	+ * bytes w/ DEBUG_OBJECTS_WORK) and allows 16 workqueue flush colors.
	43	+ *
	44	+ * MSB
	45	+ * [ pwq pointer ] [ flush color ] [ STRUCT flags ]
	46	+ * 4 bits 4 or 5 bits
	47	+ */
	48	+ WORK_STRUCT_PWQ_SHIFT = WORK_STRUCT_COLOR_SHIFT + WORK_STRUCT_COLOR_BITS,
	49	+
	50	+ /*
	51	+ * data contains off-queue information when !WORK_STRUCT_PWQ.
	52	+ *
	53	+ * MSB
	54	+ * [ pool ID ] [ OFFQ flags ] [ STRUCT flags ]
	55	+ * 1 bit 4 or 5 bits
	56	+ */
	57	+ WORK_OFFQ_FLAG_SHIFT = WORK_STRUCT_FLAG_BITS,
	58	+ WORK_OFFQ_CANCELING_BIT = WORK_OFFQ_FLAG_SHIFT,
	59	+ WORK_OFFQ_FLAG_END,
	60	+ WORK_OFFQ_FLAG_BITS = WORK_OFFQ_FLAG_END - WORK_OFFQ_FLAG_SHIFT,
	61	+
	62	+ /*
	63	+ * When a work item is off queue, the high bits encode off-queue flags
	64	+ * and the last pool it was on. Cap pool ID to 31 bits and use the
	65	+ * highest number to indicate that no pool is associated.
	66	+ */
	67	+ WORK_OFFQ_POOL_SHIFT = WORK_OFFQ_FLAG_SHIFT + WORK_OFFQ_FLAG_BITS,
	68	+ WORK_OFFQ_LEFT = BITS_PER_LONG - WORK_OFFQ_POOL_SHIFT,
	69	+ WORK_OFFQ_POOL_BITS = WORK_OFFQ_LEFT <= 31 ? WORK_OFFQ_LEFT : 31,
	70	+};
	71	+
	72	+enum work_flags {
39	73	WORK_STRUCT_PENDING = 1 << WORK_STRUCT_PENDING_BIT,
40	74	WORK_STRUCT_INACTIVE = 1 << WORK_STRUCT_INACTIVE_BIT,
41	75	WORK_STRUCT_PWQ = 1 << WORK_STRUCT_PWQ_BIT,
@@ -45,35 +79,14 @@ enum { · 3 unmodified lines
45	79	#else
46	80	WORK_STRUCT_STATIC = 0,
47	81	#endif
	82	+};
48	83
	84	+enum wq_misc_consts {
49	85	WORK_NR_COLORS = (1 << WORK_STRUCT_COLOR_BITS),
50	86
51	87	/* not bound to any CPU, prefer the local CPU */
52	88	WORK_CPU_UNBOUND = NR_CPUS,
53	89
54		− /*
55		− * Reserve 8 bits off of pwq pointer w/ debugobjects turned off.
56		− * This makes pwqs aligned to 256 bytes and allows 16 workqueue
57		− * flush colors.
58		− */
59		− WORK_STRUCT_FLAG_BITS = WORK_STRUCT_COLOR_SHIFT +
60		− WORK_STRUCT_COLOR_BITS,
61		−
62		− /* data contains off-queue information when !WORK_STRUCT_PWQ */
63		− WORK_OFFQ_FLAG_BASE = WORK_STRUCT_COLOR_SHIFT,
64		−
65		− __WORK_OFFQ_CANCELING = WORK_OFFQ_FLAG_BASE,
66		−
67		− /*
68		− * When a work item is off queue, its high bits point to the last
69		− * pool it was on. Cap at 31 bits and use the highest number to
70		− * indicate that no pool is associated.
71		− */
72		− WORK_OFFQ_FLAG_BITS = 1,
73		− WORK_OFFQ_POOL_SHIFT = WORK_OFFQ_FLAG_BASE + WORK_OFFQ_FLAG_BITS,
74		− WORK_OFFQ_LEFT = BITS_PER_LONG - WORK_OFFQ_POOL_SHIFT,
75		− WORK_OFFQ_POOL_BITS = WORK_OFFQ_LEFT <= 31 ? WORK_OFFQ_LEFT : 31,
76		−
77	90	/* bit mask for work_busy() return values */
78	91	WORK_BUSY_PENDING = 1 << 0,
79	92	WORK_BUSY_RUNNING = 1 << 1,
@@ -83,12 +96,10 @@ enum { · 3 unmodified lines
83	96	};
84	97
85	98	/* Convenience constants - of type 'unsigned long', not 'enum'! */
86		−#define WORK_OFFQ_CANCELING (1ul << __WORK_OFFQ_CANCELING)
	99	+#define WORK_OFFQ_CANCELING (1ul << WORK_OFFQ_CANCELING_BIT)
87	100	#define WORK_OFFQ_POOL_NONE ((1ul << WORK_OFFQ_POOL_BITS) - 1)
88	101	#define WORK_STRUCT_NO_POOL (WORK_OFFQ_POOL_NONE << WORK_OFFQ_POOL_SHIFT)
89		−
90		−#define WORK_STRUCT_FLAG_MASK ((1ul << WORK_STRUCT_FLAG_BITS) - 1)
91		−#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)
	102	+#define WORK_STRUCT_PWQ_MASK (~((1ul << WORK_STRUCT_PWQ_SHIFT) - 1))
92	103
93	104	#define WORK_DATA_INIT() ATOMIC_LONG_INIT((unsigned long)WORK_STRUCT_NO_POOL)
94	105	#define WORK_DATA_STATIC_INIT() \
@@ -347,7 +358,8 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; } · 252 unmodified lines
347	358	* Workqueue flags and constants. For details, please refer to
348	359	* Documentation/core-api/workqueue.rst.
349	360	*/
350		−enum {
	361	+enum wq_flags {
	362	+ WQ_BH = 1 << 0, /* execute in bottom half (softirq) context */
351	363	WQ_UNBOUND = 1 << 1, /* not bound to any cpu */
352	364	WQ_FREEZABLE = 1 << 2, /* freeze during suspend */
353	365	WQ_MEM_RECLAIM = 1 << 3, /* may be used for memory reclaim */
@@ -386,11 +398,22 @@ enum { · 32 unmodified lines
386	398	__WQ_DRAINING = 1 << 16, /* internal: workqueue is draining */
387	399	__WQ_ORDERED = 1 << 17, /* internal: workqueue is ordered */
388	400	__WQ_LEGACY = 1 << 18, /* internal: create_workqueue() /
389		− __WQ_ORDERED_EXPLICIT = 1 << 19, /* internal: alloc_ordered_workqueue() */
390	401
	402	+ /* BH wq only allows the following flags */
	403	+ __WQ_BH_ALLOWS = WQ_BH \| WQ_HIGHPRI,
	404	+};
	405	+
	406	+enum wq_consts {
391	407	WQ_MAX_ACTIVE = 512, /* I like 512, better ideas? */
392	408	WQ_UNBOUND_MAX_ACTIVE = WQ_MAX_ACTIVE,
393	409	WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2,
	410	+
	411	+ /*
	412	+ * Per-node default cap on min_active. Unless explicitly set, min_active
	413	+ * is set to min(max_active, WQ_DFL_MIN_ACTIVE). For more details, see
	414	+ * workqueue_struct->min_active definition.
	415	+ */
	416	+ WQ_DFL_MIN_ACTIVE = 8,
394	417	};
395	418
396	419	/*
@@ -420,6 +443,9 @@ enum { · 23 unmodified lines
420	443	* they are same as their non-power-efficient counterparts - e.g.
421	444	* system_power_efficient_wq is identical to system_wq if
422	445	* 'wq_power_efficient' is disabled. See WQ_POWER_EFFICIENT for more info.
	446	+ *
	447	+ * system_bh[_highpri]_wq are convenience interface to softirq. BH work items
	448	+ * are executed in the queueing CPU's BH context in the queueing order.
423	449	*/
424	450	extern struct workqueue_struct *system_wq;
425	451	extern struct workqueue_struct *system_highpri_wq;
@@ -428,16 +454,43 @@ extern struct workqueue_struct *system_unbound_wq; · 2 unmodified lines
428	454	extern struct workqueue_struct *system_freezable_wq;
429	455	extern struct workqueue_struct *system_power_efficient_wq;
430	456	extern struct workqueue_struct *system_freezable_power_efficient_wq;
	457	+extern struct workqueue_struct *system_bh_wq;
	458	+extern struct workqueue_struct *system_bh_highpri_wq;
	459	+
	460	+void workqueue_softirq_action(bool highpri);
	461	+void workqueue_softirq_dead(unsigned int cpu);
431	462
432	463	/**
433	464	* alloc_workqueue - allocate a workqueue
434	465	* @fmt: printf format for the name of the workqueue
435	466	* @flags: WQ_* flags
436		− * @max_active: max in-flight work items per CPU, 0 for default
	467	+ * @max_active: max in-flight work items, 0 for default
437	468	* remaining args: args for @fmt
438	469	*
439		− * Allocate a workqueue with the specified parameters. For detailed
440		− * information on WQ_* flags, please refer to
	470	+ * For a per-cpu workqueue, @max_active limits the number of in-flight work
	471	+ * items for each CPU. e.g. @max_active of 1 indicates that each CPU can be
	472	+ * executing at most one work item for the workqueue.
	473	+ *
	474	+ * For unbound workqueues, @max_active limits the number of in-flight work items
	475	+ * for the whole system. e.g. @max_active of 16 indicates that that there can be
	476	+ * at most 16 work items executing for the workqueue in the whole system.
	477	+ *
	478	+ * As sharing the same active counter for an unbound workqueue across multiple
	479	+ * NUMA nodes can be expensive, @max_active is distributed to each NUMA node
	480	+ * according to the proportion of the number of online CPUs and enforced
	481	+ * independently.
	482	+ *
	483	+ * Depending on online CPU distribution, a node may end up with per-node
	484	+ * max_active which is significantly lower than @max_active, which can lead to
	485	+ * deadlocks if the per-node concurrency limit is lower than the maximum number
	486	+ * of interdependent work items for the workqueue.
	487	+ *
	488	+ * To guarantee forward progress regardless of online CPU distribution, the
	489	+ * concurrency limit on every node is guaranteed to be equal to or greater than
	490	+ * min_active which is set to min(@max_active, %WQ_DFL_MIN_ACTIVE). This means
	491	+ * that the sum of per-node max_active's may be larger than @max_active.
	492	+ *
	493	+ * For detailed information on %WQ_* flags, please refer to
441	494	* Documentation/core-api/workqueue.rst.
442	495	*
443	496	* RETURNS:
@@ -460,8 +513,7 @@ alloc_workqueue(const char *fmt, unsigned int flags, int max_active, ...); · 16 unmodified lines
460	513	* Pointer to the allocated workqueue on success, %NULL on failure.
461	514	*/
462	515	#define alloc_ordered_workqueue(fmt, flags, args...) \
463		− alloc_workqueue(fmt, WQ_UNBOUND \| __WQ_ORDERED \| \
464		− __WQ_ORDERED_EXPLICIT \| (flags), 1, ##args)
	516	+ alloc_workqueue(fmt, WQ_UNBOUND \| __WQ_ORDERED \| (flags), 1, ##args)
465	517
466	518	#define create_workqueue(name) \
467	519	alloc_workqueue("%s", __WQ_LEGACY \| WQ_MEM_RECLAIM, 1, (name))
@@ -471,6 +523,9 @@ alloc_workqueue(const char *fmt, unsigned int flags, int max_active, ...); · 3 unmodified lines
471	523	#define create_singlethread_workqueue(name) \
472	524	alloc_ordered_workqueue("%s", __WQ_LEGACY \| WQ_MEM_RECLAIM, name)
473	525
	526	+#define from_work(var, callback_work, work_fieldname) \
	527	+ container_of(callback_work, typeof(*var), work_fieldname)
	528	+
474	529	extern void destroy_workqueue(struct workqueue_struct *wq);
475	530
476	531	struct workqueue_attrs *alloc_workqueue_attrs(void);
@@ -508,6 +563,8 @@ extern bool flush_rcu_work(struct rcu_work *rwork); · 31 unmodified lines
508	563
509	564	extern void workqueue_set_max_active(struct workqueue_struct *wq,
510	565	int max_active);
	566	+extern void workqueue_set_min_active(struct workqueue_struct *wq,
	567	+ int min_active);
511	568	extern struct work_struct *current_work(void);
512	569	extern bool current_is_workqueue_rescuer(void);
513	570	extern bool workqueue_congested(int cpu, struct workqueue_struct *wq);

init/Kconfig+1−1

@@ -115,7 +115,7 @@ config CONSTRUCTORS · 114 unmodified lines
115	115	bool
116	116
117	117	config IRQ_WORK
118		− bool
	118	+ def_bool y if SMP
119	119
120	120	config BUILDTIME_TABLE_SORT
121	121	bool

init/main.c+1−0

@@ -1547,6 +1547,7 @@ static noinline void __init kernel_init_freeable(void) · 1546 unmodified lines
1547	1547	sched_init_smp();
1548	1548
1549	1549	workqueue_init_topology();
	1550	+ async_init();
1550	1551	padata_init();
1551	1552	page_alloc_init_late();
1552	1553

kernel/async.c+16−1

@@ -64,6 +64,7 @@ static async_cookie_t next_cookie = 1; · 63 unmodified lines
64	64	static LIST_HEAD(async_global_pending); /* pending from all registered doms */
65	65	static ASYNC_DOMAIN(async_dfl_domain);
66	66	static DEFINE_SPINLOCK(async_lock);
	67	+static struct workqueue_struct *async_wq;
67	68
68	69	struct async_entry {
69	70	struct list_head domain_list;
@@ -174,7 +175,7 @@ static async_cookie_t __async_schedule_node_domain(async_func_t func, · 104 unmodified lines
174	175	spin_unlock_irqrestore(&async_lock, flags);
175	176
176	177	/* schedule for execution */
177		− queue_work_node(node, system_unbound_wq, &entry->work);
	178	+ queue_work_node(node, async_wq, &entry->work);
178	179
179	180	return newcookie;
180	181	}
@@ -345,3 +346,17 @@ bool current_is_async(void) · 164 unmodified lines
345	346	return worker && worker->current_func == async_run_entry_fn;
346	347	}
347	348	EXPORT_SYMBOL_GPL(current_is_async);
	349	+
	350	+void __init async_init(void)
	351	+{
	352	+ /*
	353	+ * Async can schedule a number of interdependent work items. However,
	354	+ * unbound workqueues can handle only upto min_active interdependent
	355	+ * work items. The default min_active of 8 isn't sufficient for async
	356	+ * and can lead to stalls. Let's use a dedicated workqueue with raised
	357	+ * min_active.
	358	+ */
	359	+ async_wq = alloc_workqueue("async", WQ_UNBOUND, 0);
	360	+ BUG_ON(!async_wq);
	361	+ workqueue_set_min_active(async_wq, WQ_DFL_ACTIVE);
	362	+}

kernel/softirq.c+5−0

@@ -27,6 +27,7 @@ · 26 unmodified lines
27	27	#include <linux/tick.h>
28	28	#include <linux/irq.h>
29	29	#include <linux/wait_bit.h>
	30	+#include <linux/workqueue.h>
30	31
31	32	#include <asm/softirq_stack.h>
32	33
@@ -802,11 +803,13 @@ static void tasklet_action_common(struct softirq_action *a, · 769 unmodified lines
802	803
803	804	static __latent_entropy void tasklet_action(struct softirq_action *a)
804	805	{
	806	+ workqueue_softirq_action(false);
805	807	tasklet_action_common(a, this_cpu_ptr(&tasklet_vec), TASKLET_SOFTIRQ);
806	808	}
807	809
808	810	static __latent_entropy void tasklet_hi_action(struct softirq_action *a)
809	811	{
	812	+ workqueue_softirq_action(true);
810	813	tasklet_action_common(a, this_cpu_ptr(&tasklet_hi_vec), HI_SOFTIRQ);
811	814	}
812	815
@@ -929,6 +932,8 @@ static void run_ksoftirqd(unsigned int cpu) · 116 unmodified lines
929	932	#ifdef CONFIG_HOTPLUG_CPU
930	933	static int takeover_tasklets(unsigned int cpu)
931	934	{
	935	+ workqueue_softirq_dead(cpu);
	936	+
932	937	/* CPU is dead, so no lock needed. */
933	938	local_irq_disable();
934	939

kernel/workqueue.c+1434−457

@@ -29,6 +29,7 @@ · 28 unmodified lines
29	29	#include <linux/kernel.h>
30	30	#include <linux/sched.h>
31	31	#include <linux/init.h>
	32	+#include <linux/interrupt.h>
32	33	#include <linux/signal.h>
33	34	#include <linux/completion.h>
34	35	#include <linux/workqueue.h>
@@ -53,10 +54,11 @@ · 18 unmodified lines
53	54	#include <linux/nmi.h>
54	55	#include <linux/kvm_para.h>
55	56	#include <linux/delay.h>
	57	+#include <linux/irq_work.h>
56	58
57	59	#include "workqueue_internal.h"
58	60
59		−enum {
	61	+enum worker_pool_flags {
60	62	/*
61	63	* worker_pool flags
62	64	*
@@ -72,10 +74,17 @@ enum { · 9 unmodified lines
72	74	* Note that DISASSOCIATED should be flipped only while holding
73	75	* wq_pool_attach_mutex to avoid changing binding state while
74	76	* worker_attach_to_pool() is in progress.
	77	+ *
	78	+ * As there can only be one concurrent BH execution context per CPU, a
	79	+ * BH pool is per-CPU and always DISASSOCIATED.
75	80	*/
76		− POOL_MANAGER_ACTIVE = 1 << 0, /* being managed */
	81	+ POOL_BH = 1 << 0, /* is a BH pool */
	82	+ POOL_MANAGER_ACTIVE = 1 << 1, /* being managed */
77	83	POOL_DISASSOCIATED = 1 << 2, /* cpu can't serve workers */
	84	+ POOL_BH_DRAINING = 1 << 3, /* draining after CPU offline */
	85	+};
78	86
	87	+enum worker_flags {
79	88	/* worker flags */
80	89	WORKER_DIE = 1 << 1, /* die die die */
81	90	WORKER_IDLE = 1 << 2, /* is idle */
@@ -86,7 +95,13 @@ enum { · 4 unmodified lines
86	95
87	96	WORKER_NOT_RUNNING = WORKER_PREP \| WORKER_CPU_INTENSIVE \|
88	97	WORKER_UNBOUND \| WORKER_REBOUND,
	98	+};
	99	+
	100	+enum work_cancel_flags {
	101	+ WORK_CANCEL_DELAYED = 1 << 0, /* canceling a delayed_work */
	102	+};
89	103
	104	+enum wq_internal_consts {
90	105	NR_STD_WORKER_POOLS = 2, /* # standard pools per cpu */
91	106
92	107	UNBOUND_POOL_HASH_ORDER = 6, /* hashed by pool->attrs */
@@ -108,9 +123,17 @@ enum { · 15 unmodified lines
108	123	RESCUER_NICE_LEVEL = MIN_NICE,
109	124	HIGHPRI_NICE_LEVEL = MIN_NICE,
110	125
111		− WQ_NAME_LEN = 24,
	126	+ WQ_NAME_LEN = 32,
112	127	};
113	128
	129	+/*
	130	+ * We don't want to trap softirq for too long. See MAX_SOFTIRQ_TIME and
	131	+ * MAX_SOFTIRQ_RESTART in kernel/softirq.c. These are macros because
	132	+ * msecs_to_jiffies() can't be an initializer.
	133	+ */
	134	+#define BH_WORKER_JIFFIES msecs_to_jiffies(2)
	135	+#define BH_WORKER_RESTARTS 10
	136	+
114	137	/*
115	138	* Structure fields follow one of the following exclusion rules.
116	139	*
@@ -122,6 +145,9 @@ enum { · 5 unmodified lines
122	145	*
123	146	* L: pool->lock protected. Access with pool->lock held.
124	147	*
	148	+ * LN: pool->lock and wq_node_nr_active->lock protected for writes. Either for
	149	+ * reads.
	150	+ *
125	151	* K: Only modified by worker while holding pool->lock. Can be safely read by
126	152	* self, while holding pool->lock or from IRQ context if %current is the
127	153	* kworker.
@@ -143,6 +169,9 @@ enum { · 15 unmodified lines
143	169	*
144	170	* WR: wq->mutex protected for writes. RCU protected for reads.
145	171	*
	172	+ * WO: wq->mutex protected for writes. Updated with WRITE_ONCE() and can be read
	173	+ * with READ_ONCE() without locking.
	174	+ *
146	175	* MD: wq_mayday_lock protected.
147	176	*
148	177	* WD: Used internally by the watchdog.
@@ -219,7 +248,7 @@ enum pool_workqueue_stats { · 70 unmodified lines
219	248	};
220	249
221	250	/*
222		− * The per-pool workqueue. While queued, the lower WORK_STRUCT_FLAG_BITS
	251	+ * The per-pool workqueue. While queued, bits below WORK_PWQ_SHIFT
223	252	* of work_struct->data are used for flags and the remaining high bits
224	253	* point to the pwq; thus, pwqs need to be aligned at two's power of the
225	254	* number of flag bits.
@@ -232,6 +261,7 @@ struct pool_workqueue { · 6 unmodified lines
232	261	int refcnt; /* L: reference count */
233	262	int nr_in_flight[WORK_NR_COLORS];
234	263	/* L: nr of in_flight works */
	264	+ bool plugged; /* L: execution suspended */
235	265
236	266	/*
237	267	* nr_active management and WORK_STRUCT_INACTIVE:
@@ -240,18 +270,18 @@ struct pool_workqueue { · 2 unmodified lines
240	270	* pwq->inactive_works instead of pool->worklist and marked with
241	271	* WORK_STRUCT_INACTIVE.
242	272	*
243		− * All work items marked with WORK_STRUCT_INACTIVE do not participate
244		− * in pwq->nr_active and all work items in pwq->inactive_works are
245		− * marked with WORK_STRUCT_INACTIVE. But not all WORK_STRUCT_INACTIVE
246		− * work items are in pwq->inactive_works. Some of them are ready to
247		− * run in pool->worklist or worker->scheduled. Those work itmes are
248		− * only struct wq_barrier which is used for flush_work() and should
249		− * not participate in pwq->nr_active. For non-barrier work item, it
250		− * is marked with WORK_STRUCT_INACTIVE iff it is in pwq->inactive_works.
	273	+ * All work items marked with WORK_STRUCT_INACTIVE do not participate in
	274	+ * nr_active and all work items in pwq->inactive_works are marked with
	275	+ * WORK_STRUCT_INACTIVE. But not all WORK_STRUCT_INACTIVE work items are
	276	+ * in pwq->inactive_works. Some of them are ready to run in
	277	+ * pool->worklist or worker->scheduled. Those work itmes are only struct
	278	+ * wq_barrier which is used for flush_work() and should not participate
	279	+ * in nr_active. For non-barrier work item, it is marked with
	280	+ * WORK_STRUCT_INACTIVE iff it is in pwq->inactive_works.
251	281	*/
252	282	int nr_active; /* L: nr of active works */
253		− int max_active; /* L: max active works */
254	283	struct list_head inactive_works; /* L: inactive works */
	284	+ struct list_head pending_node; /* LN: node on wq_node_nr_active->pending_pwqs */
255	285	struct list_head pwqs_node; /* WR: node on wq->pwqs */
256	286	struct list_head mayday_node; /* MD: node on wq->maydays */
257	287
@@ -265,7 +295,7 @@ struct pool_workqueue { · 7 unmodified lines
265	295	*/
266	296	struct kthread_work release_work;
267	297	struct rcu_head rcu;
268		−} __aligned(1 << WORK_STRUCT_FLAG_BITS);
	298	+} __aligned(1 << WORK_STRUCT_PWQ_SHIFT);
269	299
270	300	/*
271	301	* Structure used to wait for workqueue flush.
@@ -278,6 +308,26 @@ struct wq_flusher { · 6 unmodified lines
278	308
279	309	struct wq_device;
280	310
	311	+/*
	312	+ * Unlike in a per-cpu workqueue where max_active limits its concurrency level
	313	+ * on each CPU, in an unbound workqueue, max_active applies to the whole system.
	314	+ * As sharing a single nr_active across multiple sockets can be very expensive,
	315	+ * the counting and enforcement is per NUMA node.
	316	+ *
	317	+ * The following struct is used to enforce per-node max_active. When a pwq wants
	318	+ * to start executing a work item, it should increment ->nr using
	319	+ * tryinc_node_nr_active(). If acquisition fails due to ->nr already being over
	320	+ * ->max, the pwq is queued on ->pending_pwqs. As in-flight work items finish
	321	+ * and decrement ->nr, node_activate_pending_pwq() activates the pending pwqs in
	322	+ * round-robin order.
	323	+ */
	324	+struct wq_node_nr_active {
	325	+ int max; /* per-node max_active */
	326	+ atomic_t nr; /* per-node nr_active */
	327	+ raw_spinlock_t lock; /* nests inside pool locks */
	328	+ struct list_head pending_pwqs; /* LN: pwqs with inactive works */
	329	+};
	330	+
281	331	/*
282	332	* The externally visible workqueue. It relays the issued work items to
283	333	* the appropriate worker_pool through its pool_workqueues.
@@ -298,10 +348,15 @@ struct workqueue_struct { · 14 unmodified lines
298	348	struct worker rescuer; / MD: rescue worker */
299	349
300	350	int nr_drainers; /* WQ: drain in progress */
301		− int saved_max_active; /* WQ: saved pwq max_active */
	351	+
	352	+ /* See alloc_workqueue() function comment for info on min/max_active */
	353	+ int max_active; /* WO: max active works */
	354	+ int min_active; /* WO: min active works */
	355	+ int saved_max_active; /* WQ: saved max_active */
	356	+ int saved_min_active; /* WQ: saved min_active */
302	357
303	358	struct workqueue_attrs unbound_attrs; / PW: only for unbound wqs */
304		− struct pool_workqueue dfl_pwq; / PW: only for unbound wqs */
	359	+ struct pool_workqueue __rcu dfl_pwq; / PW: only for unbound wqs */
305	360
306	361	#ifdef CONFIG_SYSFS
307	362	struct wq_device wq_dev; / I: for sysfs interface */
@@ -323,10 +378,9 @@ struct workqueue_struct { · 15 unmodified lines
323	378	/* hot fields used during command issue, aligned to cacheline */
324	379	unsigned int flags ____cacheline_aligned; /* WQ: WQ_* flags */
325	380	struct pool_workqueue __percpu __rcu *cpu_pwq; / I: per-cpu pwqs */
	381	+ struct wq_node_nr_active node_nr_active[]; / I: per-node nr_active */
326	382	};
327	383
328		−static struct kmem_cache *pwq_cache;
329		−
330	384	/*
331	385	* Each pod type describes how CPUs should be grouped for unbound workqueues.
332	386	* See the comment above workqueue_attrs->affn_scope.
@@ -338,16 +392,13 @@ struct wq_pod_type { · 5 unmodified lines
338	392	int cpu_pod; / cpu -> pod */
339	393	};
340	394
341		−static struct wq_pod_type wq_pod_types[WQ_AFFN_NR_TYPES];
342		−static enum wq_affn_scope wq_affn_dfl = WQ_AFFN_CACHE;
343		−
344	395	static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = {
345		− [WQ_AFFN_DFL] = "default",
346		− [WQ_AFFN_CPU] = "cpu",
347		− [WQ_AFFN_SMT] = "smt",
348		− [WQ_AFFN_CACHE] = "cache",
349		− [WQ_AFFN_NUMA] = "numa",
350		− [WQ_AFFN_SYSTEM] = "system",
	396	+ [WQ_AFFN_DFL] = "default",
	397	+ [WQ_AFFN_CPU] = "cpu",
	398	+ [WQ_AFFN_SMT] = "smt",
	399	+ [WQ_AFFN_CACHE] = "cache",
	400	+ [WQ_AFFN_NUMA] = "numa",
	401	+ [WQ_AFFN_SYSTEM] = "system",
351	402	};
352	403
353	404	/*
@@ -359,12 +410,22 @@ static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = { · 5 unmodified lines
359	410	*/
360	411	static unsigned long wq_cpu_intensive_thresh_us = ULONG_MAX;
361	412	module_param_named(cpu_intensive_thresh_us, wq_cpu_intensive_thresh_us, ulong, 0644);
	413	+#ifdef CONFIG_WQ_CPU_INTENSIVE_REPORT
	414	+static unsigned int wq_cpu_intensive_warning_thresh = 4;
	415	+module_param_named(cpu_intensive_warning_thresh, wq_cpu_intensive_warning_thresh, uint, 0644);
	416	+#endif
362	417
363	418	/* see the comment above the definition of WQ_POWER_EFFICIENT */
364	419	static bool wq_power_efficient = IS_ENABLED(CONFIG_WQ_POWER_EFFICIENT_DEFAULT);
365	420	module_param_named(power_efficient, wq_power_efficient, bool, 0444);
366	421
367	422	static bool wq_online; /* can kworkers be created yet? */
	423	+static bool wq_topo_initialized __read_mostly = false;
	424	+
	425	+static struct kmem_cache *pwq_cache;
	426	+
	427	+static struct wq_pod_type wq_pod_types[WQ_AFFN_NR_TYPES];
	428	+static enum wq_affn_scope wq_affn_dfl = WQ_AFFN_CACHE;
368	429
369	430	/* buf for wq_update_unbound_pod_attrs(), protected by CPU hotplug exclusion */
370	431	static struct workqueue_attrs *wq_update_pod_attrs_buf;
@@ -405,8 +466,17 @@ static bool wq_debug_force_rr_cpu = false; · 34 unmodified lines
405	466	#endif
406	467	module_param_named(debug_force_rr_cpu, wq_debug_force_rr_cpu, bool, 0644);
407	468
	469	+/* to raise softirq for the BH worker pools on other CPUs */
	470	+static DEFINE_PER_CPU_SHARED_ALIGNED(struct irq_work [NR_STD_WORKER_POOLS],
	471	+ bh_pool_irq_works);
	472	+
	473	+/* the BH worker pools */
	474	+static DEFINE_PER_CPU_SHARED_ALIGNED(struct worker_pool [NR_STD_WORKER_POOLS],
	475	+ bh_worker_pools);
	476	+
408	477	/* the per-cpu worker pools */
409		−static DEFINE_PER_CPU_SHARED_ALIGNED(struct worker_pool [NR_STD_WORKER_POOLS], cpu_worker_pools);
	478	+static DEFINE_PER_CPU_SHARED_ALIGNED(struct worker_pool [NR_STD_WORKER_POOLS],
	479	+ cpu_worker_pools);
410	480
411	481	static DEFINE_IDR(worker_pool_idr); /* PR: idr of all pools */
412	482
@@ -419,6 +489,12 @@ static struct workqueue_attrs *unbound_std_wq_attrs[NR_STD_WORKER_POOLS]; · 6 unmodified lines
419	489	/* I: attributes used when instantiating ordered pools on demand */
420	490	static struct workqueue_attrs *ordered_wq_attrs[NR_STD_WORKER_POOLS];
421	491
	492	+/*
	493	+ * Used to synchronize multiple cancel_sync attempts on the same work item. See
	494	+ * work_grab_pending() and __cancel_work_sync().
	495	+ */
	496	+static DECLARE_WAIT_QUEUE_HEAD(wq_cancel_waitq);
	497	+
422	498	/*
423	499	* I: kthread_worker to release pwq's. pwq release needs to be bounced to a
424	500	* process context while holding a pool lock. Bounce to a dedicated kthread
@@ -440,6 +516,10 @@ struct workqueue_struct *system_power_efficient_wq __ro_after_init; · 15 unmodified lines
440	516	EXPORT_SYMBOL_GPL(system_power_efficient_wq);
441	517	struct workqueue_struct *system_freezable_power_efficient_wq __ro_after_init;
442	518	EXPORT_SYMBOL_GPL(system_freezable_power_efficient_wq);
	519	+struct workqueue_struct *system_bh_wq;
	520	+EXPORT_SYMBOL_GPL(system_bh_wq);
	521	+struct workqueue_struct *system_bh_highpri_wq;
	522	+EXPORT_SYMBOL_GPL(system_bh_highpri_wq);
443	523
444	524	static int worker_thread(void *__worker);
445	525	static void workqueue_sysfs_unregister(struct workqueue_struct *wq);
@@ -450,16 +530,21 @@ static void show_one_worker_pool(struct worker_pool *pool); · 4 unmodified lines
450	530	#include <trace/events/workqueue.h>
451	531
452	532	#define assert_rcu_or_pool_mutex() \
453		− RCU_LOCKDEP_WARN(!rcu_read_lock_held() && \
	533	+ RCU_LOCKDEP_WARN(!rcu_read_lock_any_held() && \
454	534	!lockdep_is_held(&wq_pool_mutex), \
455	535	"RCU or wq_pool_mutex should be held")
456	536
457	537	#define assert_rcu_or_wq_mutex_or_pool_mutex(wq) \
458		− RCU_LOCKDEP_WARN(!rcu_read_lock_held() && \
	538	+ RCU_LOCKDEP_WARN(!rcu_read_lock_any_held() && \
459	539	!lockdep_is_held(&wq->mutex) && \
460	540	!lockdep_is_held(&wq_pool_mutex), \
461	541	"RCU, wq->mutex or wq_pool_mutex should be held")
462	542
	543	+#define for_each_bh_worker_pool(pool, cpu) \
	544	+ for ((pool) = &per_cpu(bh_worker_pools, cpu)[0]; \
	545	+ (pool) < &per_cpu(bh_worker_pools, cpu)[NR_STD_WORKER_POOLS]; \
	546	+ (pool)++)
	547	+
463	548	#define for_each_cpu_worker_pool(pool, cpu) \
464	549	for ((pool) = &per_cpu(cpu_worker_pools, cpu)[0]; \
465	550	(pool) < &per_cpu(cpu_worker_pools, cpu)[NR_STD_WORKER_POOLS]; \
@@ -632,6 +717,36 @@ static int worker_pool_assign_id(struct worker_pool *pool) · 166 unmodified lines
632	717	return ret;
633	718	}
634	719
	720	+static struct pool_workqueue __rcu **
	721	+unbound_pwq_slot(struct workqueue_struct *wq, int cpu)
	722	+{
	723	+ if (cpu >= 0)
	724	+ return per_cpu_ptr(wq->cpu_pwq, cpu);
	725	+ else
	726	+ return &wq->dfl_pwq;
	727	+}
	728	+
	729	+/* @cpu < 0 for dfl_pwq */
	730	+static struct pool_workqueue unbound_pwq(struct workqueue_struct wq, int cpu)
	731	+{
	732	+ return rcu_dereference_check(*unbound_pwq_slot(wq, cpu),
	733	+ lockdep_is_held(&wq_pool_mutex) \|\|
	734	+ lockdep_is_held(&wq->mutex));
	735	+}
	736	+
	737	+/**
	738	+ * unbound_effective_cpumask - effective cpumask of an unbound workqueue
	739	+ * @wq: workqueue of interest
	740	+ *
	741	+ * @wq->unbound_attrs->cpumask contains the cpumask requested by the user which
	742	+ * is masked with wq_unbound_cpumask to determine the effective cpumask. The
	743	+ * default pwq is always mapped to the pool with the current effective cpumask.
	744	+ */
	745	+static struct cpumask unbound_effective_cpumask(struct workqueue_struct wq)
	746	+{
	747	+ return unbound_pwq(wq, -1)->pool->attrs->__pod_cpumask;
	748	+}
	749	+
635	750	static unsigned int work_color_to_flags(int color)
636	751	{
637	752	return color << WORK_STRUCT_COLOR_SHIFT;
@@ -653,10 +768,9 @@ static int work_next_color(int color) · 15 unmodified lines
653	768	* contain the pointer to the queued pwq. Once execution starts, the flag
654	769	* is cleared and the high bits contain OFFQ flags and pool ID.
655	770	*
656		− * set_work_pwq(), set_work_pool_and_clear_pending(), mark_work_canceling()
657		− * and clear_work_data() can be used to set the pwq, pool or clear
658		− * work->data. These functions should only be called while the work is
659		− * owned - ie. while the PENDING bit is set.
	771	+ * set_work_pwq(), set_work_pool_and_clear_pending() and mark_work_canceling()
	772	+ * can be used to set the pwq, pool or clear work->data. These functions should
	773	+ * only be called while the work is owned - ie. while the PENDING bit is set.
660	774	*
661	775	* get_work_pool() and get_work_pwq() can be used to obtain the pool or pwq
662	776	* corresponding to a work. Pool is available once the work has been
@@ -668,29 +782,28 @@ static int work_next_color(int color) · 5 unmodified lines
668	782	* but stay off timer and worklist for arbitrarily long and nobody should
669	783	* try to steal the PENDING bit.
670	784	*/
671		−static inline void set_work_data(struct work_struct *work, unsigned long data,
672		− unsigned long flags)
	785	+static inline void set_work_data(struct work_struct *work, unsigned long data)
673	786	{
674	787	WARN_ON_ONCE(!work_pending(work));
675		− atomic_long_set(&work->data, data \| flags \| work_static(work));
	788	+ atomic_long_set(&work->data, data \| work_static(work));
676	789	}
677	790
678	791	static void set_work_pwq(struct work_struct work, struct pool_workqueue pwq,
679		− unsigned long extra_flags)
	792	+ unsigned long flags)
680	793	{
681		− set_work_data(work, (unsigned long)pwq,
682		− WORK_STRUCT_PENDING \| WORK_STRUCT_PWQ \| extra_flags);
	794	+ set_work_data(work, (unsigned long)pwq \| WORK_STRUCT_PENDING \|
	795	+ WORK_STRUCT_PWQ \| flags);
683	796	}
684	797
685	798	static void set_work_pool_and_keep_pending(struct work_struct *work,
686		− int pool_id)
	799	+ int pool_id, unsigned long flags)
687	800	{
688		− set_work_data(work, (unsigned long)pool_id << WORK_OFFQ_POOL_SHIFT,
689		− WORK_STRUCT_PENDING);
	801	+ set_work_data(work, ((unsigned long)pool_id << WORK_OFFQ_POOL_SHIFT) \|
	802	+ WORK_STRUCT_PENDING \| flags);
690	803	}
691	804
692	805	static void set_work_pool_and_clear_pending(struct work_struct *work,
693		− int pool_id)
	806	+ int pool_id, unsigned long flags)
694	807	{
695	808	/*
696	809	* The following wmb is paired with the implied mb in
@@ -699,7 +812,8 @@ static void set_work_pool_and_clear_pending(struct work_struct *work, · 2 unmodified lines
699	812	* owner.
700	813	*/
701	814	smp_wmb();
702		− set_work_data(work, (unsigned long)pool_id << WORK_OFFQ_POOL_SHIFT, 0);
	815	+ set_work_data(work, ((unsigned long)pool_id << WORK_OFFQ_POOL_SHIFT) \|
	816	+ flags);
703	817	/*
704	818	* The following mb guarantees that previous clear of a PENDING bit
705	819	* will not be reordered with any speculative LOADS or STORES from
@@ -731,15 +845,9 @@ static void set_work_pool_and_clear_pending(struct work_struct *work, · 25 unmodified lines
731	845	smp_mb();
732	846	}
733	847
734		−static void clear_work_data(struct work_struct *work)
735		−{
736		− smp_wmb(); /* see set_work_pool_and_clear_pending() */
737		− set_work_data(work, WORK_STRUCT_NO_POOL, 0);
738		−}
739		−
740	848	static inline struct pool_workqueue *work_struct_pwq(unsigned long data)
741	849	{
742		− return (struct pool_workqueue *)(data & WORK_STRUCT_WQ_DATA_MASK);
	850	+ return (struct pool_workqueue *)(data & WORK_STRUCT_PWQ_MASK);
743	851	}
744	852
745	853	static struct pool_workqueue get_work_pwq(struct work_struct work)
@@ -806,7 +914,7 @@ static void mark_work_canceling(struct work_struct *work) · 60 unmodified lines
806	914	unsigned long pool_id = get_work_pool_id(work);
807	915
808	916	pool_id <<= WORK_OFFQ_POOL_SHIFT;
809		− set_work_data(work, pool_id \| WORK_OFFQ_CANCELING, WORK_STRUCT_PENDING);
	917	+ set_work_data(work, pool_id \| WORK_STRUCT_PENDING \| WORK_OFFQ_CANCELING);
810	918	}
811	919
812	920	static bool work_is_canceling(struct work_struct *work)
@@ -1101,6 +1209,29 @@ static bool assign_work(struct work_struct work, struct worker worker, · 288 unmodified lines
1101	1209	return true;
1102	1210	}
1103	1211
	1212	+static struct irq_work bh_pool_irq_work(struct worker_pool pool)
	1213	+{
	1214	+ int high = pool->attrs->nice == HIGHPRI_NICE_LEVEL ? 1 : 0;
	1215	+
	1216	+ return &per_cpu(bh_pool_irq_works, pool->cpu)[high];
	1217	+}
	1218	+
	1219	+static void kick_bh_pool(struct worker_pool *pool)
	1220	+{
	1221	+#ifdef CONFIG_SMP
	1222	+ /* see drain_dead_softirq_workfn() for BH_DRAINING */
	1223	+ if (unlikely(pool->cpu != smp_processor_id() &&
	1224	+ !(pool->flags & POOL_BH_DRAINING))) {
	1225	+ irq_work_queue_on(bh_pool_irq_work(pool), pool->cpu);
	1226	+ return;
	1227	+ }
	1228	+#endif
	1229	+ if (pool->attrs->nice == HIGHPRI_NICE_LEVEL)
	1230	+ raise_softirq_irqoff(HI_SOFTIRQ);
	1231	+ else
	1232	+ raise_softirq_irqoff(TASKLET_SOFTIRQ);
	1233	+}
	1234	+
1104	1235	/**
1105	1236	* kick_pool - wake up an idle worker if necessary
1106	1237	* @pool: pool to kick
@@ -1118,6 +1249,11 @@ static bool kick_pool(struct worker_pool *pool) · 11 unmodified lines
1118	1249	if (!need_more_worker(pool) \|\| !worker)
1119	1250	return false;
1120	1251
	1252	+ if (pool->flags & POOL_BH) {
	1253	+ kick_bh_pool(pool);
	1254	+ return true;
	1255	+ }
	1256	+
1121	1257	p = worker->task;
1122	1258
1123	1259	#ifdef CONFIG_SMP
@@ -1198,11 +1334,13 @@ restart: · 74 unmodified lines
1198	1334	u64 cnt;
1199	1335
1200	1336	/*
1201		− * Start reporting from the fourth time and back off
	1337	+ * Start reporting from the warning_thresh and back off
1202	1338	* exponentially.
1203	1339	*/
1204	1340	cnt = atomic64_inc_return_relaxed(&ent->cnt);
1205		− if (cnt >= 4 && is_power_of_2(cnt))
	1341	+ if (wq_cpu_intensive_warning_thresh &&
	1342	+ cnt >= wq_cpu_intensive_warning_thresh &&
	1343	+ is_power_of_2(cnt + 1 - wq_cpu_intensive_warning_thresh))
1206	1344	printk_deferred(KERN_WARNING "workqueue: %ps hogged CPU for >%luus %llu times, consider switching to WQ_UNBOUND\n",
1207	1345	ent->func, wq_cpu_intensive_thresh_us,
1208	1346	atomic64_read(&ent->cnt));
@@ -1231,10 +1369,12 @@ restart: · 22 unmodified lines
1231	1369
1232	1370	ent = &wci_ents[wci_nr_ents++];
1233	1371	ent->func = func;
1234		− atomic64_set(&ent->cnt, 1);
	1372	+ atomic64_set(&ent->cnt, 0);
1235	1373	hash_add_rcu(wci_hash, &ent->hash_node, (unsigned long)func);
1236	1374
1237	1375	raw_spin_unlock(&wci_lock);
	1376	+
	1377	+ goto restart;
1238	1378	}
1239	1379
1240	1380	#else /* CONFIG_WQ_CPU_INTENSIVE_REPORT */
@@ -1401,6 +1541,74 @@ work_func_t wq_worker_last_func(struct task_struct *task) · 160 unmodified lines
1401	1541	return worker->last_func;
1402	1542	}
1403	1543
	1544	+/**
	1545	+ * wq_node_nr_active - Determine wq_node_nr_active to use
	1546	+ * @wq: workqueue of interest
	1547	+ * @node: NUMA node, can be %NUMA_NO_NODE
	1548	+ *
	1549	+ * Determine wq_node_nr_active to use for @wq on @node. Returns:
	1550	+ *
	1551	+ * - %NULL for per-cpu workqueues as they don't need to use shared nr_active.
	1552	+ *
	1553	+ * - node_nr_active[nr_node_ids] if @node is %NUMA_NO_NODE.
	1554	+ *
	1555	+ * - Otherwise, node_nr_active[@node].
	1556	+ */
	1557	+static struct wq_node_nr_active wq_node_nr_active(struct workqueue_struct wq,
	1558	+ int node)
	1559	+{
	1560	+ if (!(wq->flags & WQ_UNBOUND))
	1561	+ return NULL;
	1562	+
	1563	+ if (node == NUMA_NO_NODE)
	1564	+ node = nr_node_ids;
	1565	+
	1566	+ return wq->node_nr_active[node];
	1567	+}
	1568	+
	1569	+/**
	1570	+ * wq_update_node_max_active - Update per-node max_actives to use
	1571	+ * @wq: workqueue to update
	1572	+ * @off_cpu: CPU that's going down, -1 if a CPU is not going down
	1573	+ *
	1574	+ * Update @wq->node_nr_active[]->max. @wq must be unbound. max_active is
	1575	+ * distributed among nodes according to the proportions of numbers of online
	1576	+ * cpus. The result is always between @wq->min_active and max_active.
	1577	+ */
	1578	+static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)
	1579	+{
	1580	+ struct cpumask *effective = unbound_effective_cpumask(wq);
	1581	+ int min_active = READ_ONCE(wq->min_active);
	1582	+ int max_active = READ_ONCE(wq->max_active);
	1583	+ int total_cpus, node;
	1584	+
	1585	+ lockdep_assert_held(&wq->mutex);
	1586	+
	1587	+ if (!wq_topo_initialized)
	1588	+ return;
	1589	+
	1590	+ if (off_cpu >= 0 && !cpumask_test_cpu(off_cpu, effective))
	1591	+ off_cpu = -1;
	1592	+
	1593	+ total_cpus = cpumask_weight_and(effective, cpu_online_mask);
	1594	+ if (off_cpu >= 0)
	1595	+ total_cpus--;
	1596	+
	1597	+ for_each_node(node) {
	1598	+ int node_cpus;
	1599	+
	1600	+ node_cpus = cpumask_weight_and(effective, cpumask_of_node(node));
	1601	+ if (off_cpu >= 0 && cpu_to_node(off_cpu) == node)
	1602	+ node_cpus--;
	1603	+
	1604	+ wq_node_nr_active(wq, node)->max =
	1605	+ clamp(DIV_ROUND_UP(max_active * node_cpus, total_cpus),
	1606	+ min_active, max_active);
	1607	+ }
	1608	+
	1609	+ wq_node_nr_active(wq, NUMA_NO_NODE)->max = min_active;
	1610	+}
	1611	+
1404	1612	/**
1405	1613	* get_pwq - get an extra reference on the specified pool_workqueue
1406	1614	* @pwq: pool_workqueue to get
@@ -1453,24 +1661,336 @@ static void put_pwq_unlocked(struct pool_workqueue *pwq) · 46 unmodified lines
1453	1661	}
1454	1662	}
1455	1663
1456		−static void pwq_activate_inactive_work(struct work_struct *work)
	1664	+static bool pwq_is_empty(struct pool_workqueue *pwq)
1457	1665	{
1458		− struct pool_workqueue *pwq = get_work_pwq(work);
	1666	+ return !pwq->nr_active && list_empty(&pwq->inactive_works);
	1667	+}
1459	1668
	1669	+static void __pwq_activate_work(struct pool_workqueue *pwq,
	1670	+ struct work_struct *work)
	1671	+{
	1672	+ unsigned long *wdb = work_data_bits(work);
	1673	+
	1674	+ WARN_ON_ONCE(!(*wdb & WORK_STRUCT_INACTIVE));
1460	1675	trace_workqueue_activate_work(work);
1461	1676	if (list_empty(&pwq->pool->worklist))
1462	1677	pwq->pool->watchdog_ts = jiffies;
1463	1678	move_linked_works(work, &pwq->pool->worklist, NULL);
1464		− __clear_bit(WORK_STRUCT_INACTIVE_BIT, work_data_bits(work));
	1679	+ __clear_bit(WORK_STRUCT_INACTIVE_BIT, wdb);
	1680	+}
	1681	+
	1682	+/**
	1683	+ * pwq_activate_work - Activate a work item if inactive
	1684	+ * @pwq: pool_workqueue @work belongs to
	1685	+ * @work: work item to activate
	1686	+ *
	1687	+ * Returns %true if activated. %false if already active.
	1688	+ */
	1689	+static bool pwq_activate_work(struct pool_workqueue *pwq,
	1690	+ struct work_struct *work)
	1691	+{
	1692	+ struct worker_pool *pool = pwq->pool;
	1693	+ struct wq_node_nr_active *nna;
	1694	+
	1695	+ lockdep_assert_held(&pool->lock);
	1696	+
	1697	+ if (!(*work_data_bits(work) & WORK_STRUCT_INACTIVE))
	1698	+ return false;
	1699	+
	1700	+ nna = wq_node_nr_active(pwq->wq, pool->node);
	1701	+ if (nna)
	1702	+ atomic_inc(&nna->nr);
	1703	+
1465	1704	pwq->nr_active++;
	1705	+ __pwq_activate_work(pwq, work);
	1706	+ return true;
	1707	+}
	1708	+
	1709	+static bool tryinc_node_nr_active(struct wq_node_nr_active *nna)
	1710	+{
	1711	+ int max = READ_ONCE(nna->max);
	1712	+
	1713	+ while (true) {
	1714	+ int old, tmp;
	1715	+
	1716	+ old = atomic_read(&nna->nr);
	1717	+ if (old >= max)
	1718	+ return false;
	1719	+ tmp = atomic_cmpxchg_relaxed(&nna->nr, old, old + 1);
	1720	+ if (tmp == old)
	1721	+ return true;
	1722	+ }
1466	1723	}
1467	1724
1468		−static void pwq_activate_first_inactive(struct pool_workqueue *pwq)
	1725	+/**
	1726	+ * pwq_tryinc_nr_active - Try to increment nr_active for a pwq
	1727	+ * @pwq: pool_workqueue of interest
	1728	+ * @fill: max_active may have increased, try to increase concurrency level
	1729	+ *
	1730	+ * Try to increment nr_active for @pwq. Returns %true if an nr_active count is
	1731	+ * successfully obtained. %false otherwise.
	1732	+ */
	1733	+static bool pwq_tryinc_nr_active(struct pool_workqueue *pwq, bool fill)
1469	1734	{
1470		− struct work_struct *work = list_first_entry(&pwq->inactive_works,
1471		− struct work_struct, entry);
	1735	+ struct workqueue_struct *wq = pwq->wq;
	1736	+ struct worker_pool *pool = pwq->pool;
	1737	+ struct wq_node_nr_active *nna = wq_node_nr_active(wq, pool->node);
	1738	+ bool obtained = false;
	1739	+
	1740	+ lockdep_assert_held(&pool->lock);
	1741	+
	1742	+ if (!nna) {
	1743	+ /* BH or per-cpu workqueue, pwq->nr_active is sufficient */
	1744	+ obtained = pwq->nr_active < READ_ONCE(wq->max_active);
	1745	+ goto out;
	1746	+ }
	1747	+
	1748	+ if (unlikely(pwq->plugged))
	1749	+ return false;
	1750	+
	1751	+ /*
	1752	+ * Unbound workqueue uses per-node shared nr_active $nna. If @pwq is
	1753	+ * already waiting on $nna, pwq_dec_nr_active() will maintain the
	1754	+ * concurrency level. Don't jump the line.
	1755	+ *
	1756	+ * We need to ignore the pending test after max_active has increased as
	1757	+ * pwq_dec_nr_active() can only maintain the concurrency level but not
	1758	+ * increase it. This is indicated by @fill.
	1759	+ */
	1760	+ if (!list_empty(&pwq->pending_node) && likely(!fill))
	1761	+ goto out;
	1762	+
	1763	+ obtained = tryinc_node_nr_active(nna);
	1764	+ if (obtained)
	1765	+ goto out;
	1766	+
	1767	+ /*
	1768	+ * Lockless acquisition failed. Lock, add ourself to $nna->pending_pwqs
	1769	+ * and try again. The smp_mb() is paired with the implied memory barrier
	1770	+ * of atomic_dec_return() in pwq_dec_nr_active() to ensure that either
	1771	+ * we see the decremented $nna->nr or they see non-empty
	1772	+ * $nna->pending_pwqs.
	1773	+ */
	1774	+ raw_spin_lock(&nna->lock);
	1775	+
	1776	+ if (list_empty(&pwq->pending_node))
	1777	+ list_add_tail(&pwq->pending_node, &nna->pending_pwqs);
	1778	+ else if (likely(!fill))
	1779	+ goto out_unlock;
	1780	+
	1781	+ smp_mb();
	1782	+
	1783	+ obtained = tryinc_node_nr_active(nna);
	1784	+
	1785	+ /*
	1786	+ * If @fill, @pwq might have already been pending. Being spuriously
	1787	+ * pending in cold paths doesn't affect anything. Let's leave it be.
	1788	+ */
	1789	+ if (obtained && likely(!fill))
	1790	+ list_del_init(&pwq->pending_node);
	1791	+
	1792	+out_unlock:
	1793	+ raw_spin_unlock(&nna->lock);
	1794	+out:
	1795	+ if (obtained)
	1796	+ pwq->nr_active++;
	1797	+ return obtained;
	1798	+}
	1799	+
	1800	+/**
	1801	+ * pwq_activate_first_inactive - Activate the first inactive work item on a pwq
	1802	+ * @pwq: pool_workqueue of interest
	1803	+ * @fill: max_active may have increased, try to increase concurrency level
	1804	+ *
	1805	+ * Activate the first inactive work item of @pwq if available and allowed by
	1806	+ * max_active limit.
	1807	+ *
	1808	+ * Returns %true if an inactive work item has been activated. %false if no
	1809	+ * inactive work item is found or max_active limit is reached.
	1810	+ */
	1811	+static bool pwq_activate_first_inactive(struct pool_workqueue *pwq, bool fill)
	1812	+{
	1813	+ struct work_struct *work =
	1814	+ list_first_entry_or_null(&pwq->inactive_works,
	1815	+ struct work_struct, entry);
	1816	+
	1817	+ if (work && pwq_tryinc_nr_active(pwq, fill)) {
	1818	+ __pwq_activate_work(pwq, work);
	1819	+ return true;
	1820	+ } else {
	1821	+ return false;
	1822	+ }
	1823	+}
	1824	+
	1825	+/**
	1826	+ * unplug_oldest_pwq - unplug the oldest pool_workqueue
	1827	+ * @wq: workqueue_struct where its oldest pwq is to be unplugged
	1828	+ *
	1829	+ * This function should only be called for ordered workqueues where only the
	1830	+ * oldest pwq is unplugged, the others are plugged to suspend execution to
	1831	+ * ensure proper work item ordering::
	1832	+ *
	1833	+ * dfl_pwq --------------+ [P] - plugged
	1834	+ * \|
	1835	+ * v
	1836	+ * pwqs -> A -> B [P] -> C [P] (newest)
	1837	+ * \| \| \|
	1838	+ * 1 3 5
	1839	+ * \| \| \|
	1840	+ * 2 4 6
	1841	+ *
	1842	+ * When the oldest pwq is drained and removed, this function should be called
	1843	+ * to unplug the next oldest one to start its work item execution. Note that
	1844	+ * pwq's are linked into wq->pwqs with the oldest first, so the first one in
	1845	+ * the list is the oldest.
	1846	+ */
	1847	+static void unplug_oldest_pwq(struct workqueue_struct *wq)
	1848	+{
	1849	+ struct pool_workqueue *pwq;
	1850	+
	1851	+ lockdep_assert_held(&wq->mutex);
	1852	+
	1853	+ /* Caller should make sure that pwqs isn't empty before calling */
	1854	+ pwq = list_first_entry_or_null(&wq->pwqs, struct pool_workqueue,
	1855	+ pwqs_node);
	1856	+ raw_spin_lock_irq(&pwq->pool->lock);
	1857	+ if (pwq->plugged) {
	1858	+ pwq->plugged = false;
	1859	+ if (pwq_activate_first_inactive(pwq, true))
	1860	+ kick_pool(pwq->pool);
	1861	+ }
	1862	+ raw_spin_unlock_irq(&pwq->pool->lock);
	1863	+}
	1864	+
	1865	+/**
	1866	+ * node_activate_pending_pwq - Activate a pending pwq on a wq_node_nr_active
	1867	+ * @nna: wq_node_nr_active to activate a pending pwq for
	1868	+ * @caller_pool: worker_pool the caller is locking
	1869	+ *
	1870	+ * Activate a pwq in @nna->pending_pwqs. Called with @caller_pool locked.
	1871	+ * @caller_pool may be unlocked and relocked to lock other worker_pools.
	1872	+ */
	1873	+static void node_activate_pending_pwq(struct wq_node_nr_active *nna,
	1874	+ struct worker_pool *caller_pool)
	1875	+{
	1876	+ struct worker_pool *locked_pool = caller_pool;
	1877	+ struct pool_workqueue *pwq;
	1878	+ struct work_struct *work;
	1879	+
	1880	+ lockdep_assert_held(&caller_pool->lock);
	1881	+
	1882	+ raw_spin_lock(&nna->lock);
	1883	+retry:
	1884	+ pwq = list_first_entry_or_null(&nna->pending_pwqs,
	1885	+ struct pool_workqueue, pending_node);
	1886	+ if (!pwq)
	1887	+ goto out_unlock;
	1888	+
	1889	+ /*
	1890	+ * If @pwq is for a different pool than @locked_pool, we need to lock
	1891	+ * @pwq->pool->lock. Let's trylock first. If unsuccessful, do the unlock
	1892	+ * / lock dance. For that, we also need to release @nna->lock as it's
	1893	+ * nested inside pool locks.
	1894	+ */
	1895	+ if (pwq->pool != locked_pool) {
	1896	+ raw_spin_unlock(&locked_pool->lock);
	1897	+ locked_pool = pwq->pool;
	1898	+ if (!raw_spin_trylock(&locked_pool->lock)) {
	1899	+ raw_spin_unlock(&nna->lock);
	1900	+ raw_spin_lock(&locked_pool->lock);
	1901	+ raw_spin_lock(&nna->lock);
	1902	+ goto retry;
	1903	+ }
	1904	+ }
	1905	+
	1906	+ /*
	1907	+ * $pwq may not have any inactive work items due to e.g. cancellations.
	1908	+ * Drop it from pending_pwqs and see if there's another one.
	1909	+ */
	1910	+ work = list_first_entry_or_null(&pwq->inactive_works,
	1911	+ struct work_struct, entry);
	1912	+ if (!work) {
	1913	+ list_del_init(&pwq->pending_node);
	1914	+ goto retry;
	1915	+ }
	1916	+
	1917	+ /*
	1918	+ * Acquire an nr_active count and activate the inactive work item. If
	1919	+ * $pwq still has inactive work items, rotate it to the end of the
	1920	+ * pending_pwqs so that we round-robin through them. This means that
	1921	+ * inactive work items are not activated in queueing order which is fine
	1922	+ * given that there has never been any ordering across different pwqs.
	1923	+ */
	1924	+ if (likely(tryinc_node_nr_active(nna))) {
	1925	+ pwq->nr_active++;
	1926	+ __pwq_activate_work(pwq, work);
	1927	+
	1928	+ if (list_empty(&pwq->inactive_works))
	1929	+ list_del_init(&pwq->pending_node);
	1930	+ else
	1931	+ list_move_tail(&pwq->pending_node, &nna->pending_pwqs);
	1932	+
	1933	+ /* if activating a foreign pool, make sure it's running */
	1934	+ if (pwq->pool != caller_pool)
	1935	+ kick_pool(pwq->pool);
	1936	+ }
	1937	+
	1938	+out_unlock:
	1939	+ raw_spin_unlock(&nna->lock);
	1940	+ if (locked_pool != caller_pool) {
	1941	+ raw_spin_unlock(&locked_pool->lock);
	1942	+ raw_spin_lock(&caller_pool->lock);
	1943	+ }
	1944	+}
	1945	+
	1946	+/**
	1947	+ * pwq_dec_nr_active - Retire an active count
	1948	+ * @pwq: pool_workqueue of interest
	1949	+ *
	1950	+ * Decrement @pwq's nr_active and try to activate the first inactive work item.
	1951	+ * For unbound workqueues, this function may temporarily drop @pwq->pool->lock.
	1952	+ */
	1953	+static void pwq_dec_nr_active(struct pool_workqueue *pwq)
	1954	+{
	1955	+ struct worker_pool *pool = pwq->pool;
	1956	+ struct wq_node_nr_active *nna = wq_node_nr_active(pwq->wq, pool->node);
	1957	+
	1958	+ lockdep_assert_held(&pool->lock);
	1959	+
	1960	+ /*
	1961	+ * @pwq->nr_active should be decremented for both percpu and unbound
	1962	+ * workqueues.
	1963	+ */
	1964	+ pwq->nr_active--;
	1965	+
	1966	+ /*
	1967	+ * For a percpu workqueue, it's simple. Just need to kick the first
	1968	+ * inactive work item on @pwq itself.
	1969	+ */
	1970	+ if (!nna) {
	1971	+ pwq_activate_first_inactive(pwq, false);
	1972	+ return;
	1973	+ }
	1974	+
	1975	+ /*
	1976	+ * If @pwq is for an unbound workqueue, it's more complicated because
	1977	+ * multiple pwqs and pools may be sharing the nr_active count. When a
	1978	+ * pwq needs to wait for an nr_active count, it puts itself on
	1979	+ * $nna->pending_pwqs. The following atomic_dec_return()'s implied
	1980	+ * memory barrier is paired with smp_mb() in pwq_tryinc_nr_active() to
	1981	+ * guarantee that either we see non-empty pending_pwqs or they see
	1982	+ * decremented $nna->nr.
	1983	+ *
	1984	+ * $nna->max may change as CPUs come online/offline and @pwq->wq's
	1985	+ * max_active gets updated. However, it is guaranteed to be equal to or
	1986	+ * larger than @pwq->wq->min_active which is above zero unless freezing.
	1987	+ * This maintains the forward progress guarantee.
	1988	+ */
	1989	+ if (atomic_dec_return(&nna->nr) >= READ_ONCE(nna->max))
	1990	+ return;
1472	1991
1473		− pwq_activate_inactive_work(work);
	1992	+ if (!list_empty(&nna->pending_pwqs))
	1993	+ node_activate_pending_pwq(nna, pool);
1474	1994	}
1475	1995
1476	1996	/**
@@ -1481,6 +2001,11 @@ static void pwq_activate_first_inactive(struct pool_workqueue *pwq) · 4 unmodified lines
1481	2001	* A work either has completed or is removed from pending queue,
1482	2002	* decrement nr_in_flight of its pwq and handle workqueue flushing.
1483	2003	*
	2004	+ * NOTE:
	2005	+ * For unbound workqueues, this function may temporarily drop @pwq->pool->lock
	2006	+ * and thus should be called after all other state updates for the in-flight
	2007	+ * work item is complete.
	2008	+ *
1484	2009	* CONTEXT:
1485	2010	* raw_spin_lock_irq(pool->lock).
1486	2011	*/
@@ -1488,14 +2013,8 @@ static void pwq_dec_nr_in_flight(struct pool_workqueue *pwq, unsigned long work_ · 1 unmodified line
1488	2013	{
1489	2014	int color = get_work_color(work_data);
1490	2015
1491		− if (!(work_data & WORK_STRUCT_INACTIVE)) {
1492		− pwq->nr_active--;
1493		− if (!list_empty(&pwq->inactive_works)) {
1494		− /* one down, submit an inactive one */
1495		− if (pwq->nr_active < pwq->max_active)
1496		− pwq_activate_first_inactive(pwq);
1497		− }
1498		− }
	2016	+ if (!(work_data & WORK_STRUCT_INACTIVE))
	2017	+ pwq_dec_nr_active(pwq);
1499	2018
1500	2019	pwq->nr_in_flight[color]--;
1501	2020
@@ -1523,8 +2042,8 @@ out_put: · 21 unmodified lines
1523	2042	/**
1524	2043	* try_to_grab_pending - steal work item from worklist and disable irq
1525	2044	* @work: work item to steal
1526		− * @is_dwork: @work is a delayed_work
1527		− * @flags: place to store irq state
	2045	+ * @cflags: %WORK_CANCEL_ flags
	2046	+ * @irq_flags: place to store irq state
1528	2047	*
1529	2048	* Try to grab PENDING bit of @work. This function can handle @work in any
1530	2049	* stable state - idle, on timer or on worklist.
@@ -1546,20 +2065,20 @@ out_put: · 15 unmodified lines
1546	2065	* irqsafe, ensures that we return -EAGAIN for finite short period of time.
1547	2066	*
1548	2067	* On successful return, >= 0, irq is disabled and the caller is
1549		− * responsible for releasing it using local_irq_restore(*@flags).
	2068	+ * responsible for releasing it using local_irq_restore(*@irq_flags).
1550	2069	*
1551	2070	* This function is safe to call from any context including IRQ handler.
1552	2071	*/
1553		−static int try_to_grab_pending(struct work_struct *work, bool is_dwork,
1554		− unsigned long *flags)
	2072	+static int try_to_grab_pending(struct work_struct *work, u32 cflags,
	2073	+ unsigned long *irq_flags)
1555	2074	{
1556	2075	struct worker_pool *pool;
1557	2076	struct pool_workqueue *pwq;
1558	2077
1559		− local_irq_save(*flags);
	2078	+ local_irq_save(*irq_flags);
1560	2079
1561	2080	/* try to steal the timer if it exists */
1562		− if (is_dwork) {
	2081	+ if (cflags & WORK_CANCEL_DELAYED) {
1563	2082	struct delayed_work *dwork = to_delayed_work(work);
1564	2083
1565	2084	/*
@@ -1595,6 +2114,8 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork, · 29 unmodified lines
1595	2114	*/
1596	2115	pwq = get_work_pwq(work);
1597	2116	if (pwq && pwq->pool == pool) {
	2117	+ unsigned long work_data;
	2118	+
1598	2119	debug_work_deactivate(work);
1599	2120
1600	2121	/*
@@ -1608,14 +2129,19 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork, · 7 unmodified lines
1608	2129	* management later on and cause stall. Make sure the work
1609	2130	* item is activated before grabbing.
1610	2131	*/
1611		− if (*work_data_bits(work) & WORK_STRUCT_INACTIVE)
1612		− pwq_activate_inactive_work(work);
	2132	+ pwq_activate_work(pwq, work);
1613	2133
1614	2134	list_del_init(&work->entry);
1615		− pwq_dec_nr_in_flight(pwq, *work_data_bits(work));
1616	2135
1617		− /* work->data points to pwq iff queued, point to pool */
1618		− set_work_pool_and_keep_pending(work, pool->id);
	2136	+ /*
	2137	+ * work->data points to pwq iff queued. Let's point to pool. As
	2138	+ * this destroys work->data needed by the next step, stash it.
	2139	+ */
	2140	+ work_data = *work_data_bits(work);
	2141	+ set_work_pool_and_keep_pending(work, pool->id, 0);
	2142	+
	2143	+ /* must be the last step, see the function comment */
	2144	+ pwq_dec_nr_in_flight(pwq, work_data);
1619	2145
1620	2146	raw_spin_unlock(&pool->lock);
1621	2147	rcu_read_unlock();
@@ -1624,13 +2150,82 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork, · 2 unmodified lines
1624	2150	raw_spin_unlock(&pool->lock);
1625	2151	fail:
1626	2152	rcu_read_unlock();
1627		− local_irq_restore(*flags);
	2153	+ local_irq_restore(*irq_flags);
1628	2154	if (work_is_canceling(work))
1629	2155	return -ENOENT;
1630	2156	cpu_relax();
1631	2157	return -EAGAIN;
1632	2158	}
1633	2159
	2160	+struct cwt_wait {
	2161	+ wait_queue_entry_t wait;
	2162	+ struct work_struct *work;
	2163	+};
	2164	+
	2165	+static int cwt_wakefn(wait_queue_entry_t wait, unsigned mode, int sync, void key)
	2166	+{
	2167	+ struct cwt_wait *cwait = container_of(wait, struct cwt_wait, wait);
	2168	+
	2169	+ if (cwait->work != key)
	2170	+ return 0;
	2171	+ return autoremove_wake_function(wait, mode, sync, key);
	2172	+}
	2173	+
	2174	+/**
	2175	+ * work_grab_pending - steal work item from worklist and disable irq
	2176	+ * @work: work item to steal
	2177	+ * @cflags: %WORK_CANCEL_ flags
	2178	+ * @irq_flags: place to store IRQ state
	2179	+ *
	2180	+ * Grab PENDING bit of @work. @work can be in any stable state - idle, on timer
	2181	+ * or on worklist.
	2182	+ *
	2183	+ * Must be called in process context. IRQ is disabled on return with IRQ state
	2184	+ * stored in *@irq_flags. The caller is responsible for re-enabling it using
	2185	+ * local_irq_restore().
	2186	+ *
	2187	+ * Returns %true if @work was pending. %false if idle.
	2188	+ */
	2189	+static bool work_grab_pending(struct work_struct *work, u32 cflags,
	2190	+ unsigned long *irq_flags)
	2191	+{
	2192	+ struct cwt_wait cwait;
	2193	+ int ret;
	2194	+
	2195	+ might_sleep();
	2196	+repeat:
	2197	+ ret = try_to_grab_pending(work, cflags, irq_flags);
	2198	+ if (likely(ret >= 0))
	2199	+ return ret;
	2200	+ if (ret != -ENOENT)
	2201	+ goto repeat;
	2202	+
	2203	+ /*
	2204	+ * Someone is already canceling. Wait for it to finish. flush_work()
	2205	+ * doesn't work for PREEMPT_NONE because we may get woken up between
	2206	+ * @work's completion and the other canceling task resuming and clearing
	2207	+ * CANCELING - flush_work() will return false immediately as @work is no
	2208	+ * longer busy, try_to_grab_pending() will return -ENOENT as @work is
	2209	+ * still being canceled and the other canceling task won't be able to
	2210	+ * clear CANCELING as we're hogging the CPU.
	2211	+ *
	2212	+ * Let's wait for completion using a waitqueue. As this may lead to the
	2213	+ * thundering herd problem, use a custom wake function which matches
	2214	+ * @work along with exclusive wait and wakeup.
	2215	+ */
	2216	+ init_wait(&cwait.wait);
	2217	+ cwait.wait.func = cwt_wakefn;
	2218	+ cwait.work = work;
	2219	+
	2220	+ prepare_to_wait_exclusive(&wq_cancel_waitq, &cwait.wait,
	2221	+ TASK_UNINTERRUPTIBLE);
	2222	+ if (work_is_canceling(work))
	2223	+ schedule();
	2224	+ finish_wait(&wq_cancel_waitq, &cwait.wait);
	2225	+
	2226	+ goto repeat;
	2227	+}
	2228	+
1634	2229	/**
1635	2230	* insert_work - insert a work into a pool
1636	2231	* @pwq: pwq @work belongs to
@@ -1718,7 +2313,6 @@ static void __queue_work(int cpu, struct workqueue_struct *wq, · 81 unmodified lines
1718	2313	*/
1719	2314	lockdep_assert_irqs_disabled();
1720	2315
1721		−
1722	2316	/*
1723	2317	* For a draining wq, only works from the same workqueue are
1724	2318	* allowed. The __WQ_DESTROYING helps to spot the issue that
@@ -1793,12 +2387,16 @@ retry: · 68 unmodified lines
1793	2387	pwq->nr_in_flight[pwq->work_color]++;
1794	2388	work_flags = work_color_to_flags(pwq->work_color);
1795	2389
1796		− if (likely(pwq->nr_active < pwq->max_active)) {
	2390	+ /*
	2391	+ * Limit the number of concurrently active work items to max_active.
	2392	+ * @work must also queue behind existing inactive work items to maintain
	2393	+ * ordering when max_active changes. See wq_adjust_max_active().
	2394	+ */
	2395	+ if (list_empty(&pwq->inactive_works) && pwq_tryinc_nr_active(pwq, false)) {
1797	2396	if (list_empty(&pool->worklist))
1798	2397	pool->watchdog_ts = jiffies;
1799	2398
1800	2399	trace_workqueue_activate_work(work);
1801		− pwq->nr_active++;
1802	2400	insert_work(pwq, work, &pool->worklist, work_flags);
1803	2401	kick_pool(pool);
1804	2402	} else {
@@ -1829,16 +2427,16 @@ bool queue_work_on(int cpu, struct workqueue_struct *wq, · 24 unmodified lines
1829	2427	struct work_struct *work)
1830	2428	{
1831	2429	bool ret = false;
1832		− unsigned long flags;
	2430	+ unsigned long irq_flags;
1833	2431
1834		− local_irq_save(flags);
	2432	+ local_irq_save(irq_flags);
1835	2433
1836	2434	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {
1837	2435	__queue_work(cpu, wq, work);
1838	2436	ret = true;
1839	2437	}
1840	2438
1841		− local_irq_restore(flags);
	2439	+ local_irq_restore(irq_flags);
1842	2440	return ret;
1843	2441	}
1844	2442	EXPORT_SYMBOL(queue_work_on);
@@ -1895,7 +2493,7 @@ static int select_numa_node_cpu(int node) · 50 unmodified lines
1895	2493	bool queue_work_node(int node, struct workqueue_struct *wq,
1896	2494	struct work_struct *work)
1897	2495	{
1898		− unsigned long flags;
	2496	+ unsigned long irq_flags;
1899	2497	bool ret = false;
1900	2498
1901	2499	/*
@@ -1909,7 +2507,7 @@ bool queue_work_node(int node, struct workqueue_struct *wq, · 7 unmodified lines
1909	2507	*/
1910	2508	WARN_ON_ONCE(!(wq->flags & WQ_UNBOUND));
1911	2509
1912		− local_irq_save(flags);
	2510	+ local_irq_save(irq_flags);
1913	2511
1914	2512	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {
1915	2513	int cpu = select_numa_node_cpu(node);
@@ -1918,7 +2516,7 @@ bool queue_work_node(int node, struct workqueue_struct *wq, · 2 unmodified lines
1918	2516	ret = true;
1919	2517	}
1920	2518
1921		− local_irq_restore(flags);
	2519	+ local_irq_restore(irq_flags);
1922	2520	return ret;
1923	2521	}
1924	2522	EXPORT_SYMBOL_GPL(queue_work_node);
@@ -1958,10 +2556,18 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, · 33 unmodified lines
1958	2556	dwork->cpu = cpu;
1959	2557	timer->expires = jiffies + delay;
1960	2558
1961		− if (unlikely(cpu != WORK_CPU_UNBOUND))
	2559	+ if (housekeeping_enabled(HK_TYPE_TIMER)) {
	2560	+ /* If the current cpu is a housekeeping cpu, use it. */
	2561	+ cpu = smp_processor_id();
	2562	+ if (!housekeeping_test_cpu(cpu, HK_TYPE_TIMER))
	2563	+ cpu = housekeeping_any_cpu(HK_TYPE_TIMER);
1962	2564	add_timer_on(timer, cpu);
1963		− else
1964		− add_timer(timer);
	2565	+ } else {
	2566	+ if (likely(cpu == WORK_CPU_UNBOUND))
	2567	+ add_timer(timer);
	2568	+ else
	2569	+ add_timer_on(timer, cpu);
	2570	+ }
1965	2571	}
1966	2572
1967	2573	/**
@@ -1980,17 +2586,17 @@ bool queue_delayed_work_on(int cpu, struct workqueue_struct *wq, · 12 unmodified lines
1980	2586	{
1981	2587	struct work_struct *work = &dwork->work;
1982	2588	bool ret = false;
1983		− unsigned long flags;
	2589	+ unsigned long irq_flags;
1984	2590
1985	2591	/* read the comment in __queue_work() */
1986		− local_irq_save(flags);
	2592	+ local_irq_save(irq_flags);
1987	2593
1988	2594	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {
1989	2595	__queue_delayed_work(cpu, wq, dwork, delay);
1990	2596	ret = true;
1991	2597	}
1992	2598
1993		− local_irq_restore(flags);
	2599	+ local_irq_restore(irq_flags);
1994	2600	return ret;
1995	2601	}
1996	2602	EXPORT_SYMBOL(queue_delayed_work_on);
@@ -2016,16 +2622,17 @@ EXPORT_SYMBOL(queue_delayed_work_on); · 19 unmodified lines
2016	2622	bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq,
2017	2623	struct delayed_work *dwork, unsigned long delay)
2018	2624	{
2019		− unsigned long flags;
	2625	+ unsigned long irq_flags;
2020	2626	int ret;
2021	2627
2022	2628	do {
2023		− ret = try_to_grab_pending(&dwork->work, true, &flags);
	2629	+ ret = try_to_grab_pending(&dwork->work, WORK_CANCEL_DELAYED,
	2630	+ &irq_flags);
2024	2631	} while (unlikely(ret == -EAGAIN));
2025	2632
2026	2633	if (likely(ret >= 0)) {
2027	2634	__queue_delayed_work(cpu, wq, dwork, delay);
2028		− local_irq_restore(flags);
	2635	+ local_irq_restore(irq_flags);
2029	2636	}
2030	2637
2031	2638	/* -ENOENT from try_to_grab_pending() becomes %true */
@@ -2100,19 +2707,21 @@ static cpumask_t pool_allowed_cpus(struct worker_pool pool) · 68 unmodified lines
2100	2707	* cpu-[un]hotplugs.
2101	2708	*/
2102	2709	static void worker_attach_to_pool(struct worker *worker,
2103		− struct worker_pool *pool)
	2710	+ struct worker_pool *pool)
2104	2711	{
2105	2712	mutex_lock(&wq_pool_attach_mutex);
2106	2713
2107	2714	/*
2108		− * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains
2109		− * stable across this function. See the comments above the flag
2110		− * definition for details.
	2715	+ * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains stable
	2716	+ * across this function. See the comments above the flag definition for
	2717	+ * details. BH workers are, while per-CPU, always DISASSOCIATED.
2111	2718	*/
2112		− if (pool->flags & POOL_DISASSOCIATED)
	2719	+ if (pool->flags & POOL_DISASSOCIATED) {
2113	2720	worker->flags \|= WORKER_UNBOUND;
2114		− else
	2721	+ } else {
	2722	+ WARN_ON_ONCE(pool->flags & POOL_BH);
2115	2723	kthread_set_per_cpu(worker->task, pool->cpu);
	2724	+ }
2116	2725
2117	2726	if (worker->rescue_wq)
2118	2727	set_cpus_allowed_ptr(worker->task, pool_allowed_cpus(pool));
@@ -2136,6 +2745,9 @@ static void worker_detach_from_pool(struct worker *worker) · 17 unmodified lines
2136	2745	struct worker_pool *pool = worker->pool;
2137	2746	struct completion *detach_completion = NULL;
2138	2747
	2748	+ /* there is one permanent BH worker per CPU which should never detach */
	2749	+ WARN_ON_ONCE(pool->flags & POOL_BH);
	2750	+
2139	2751	mutex_lock(&wq_pool_attach_mutex);
2140	2752
2141	2753	kthread_set_per_cpu(worker->task, -1);
@@ -2187,27 +2799,29 @@ static struct worker create_worker(struct worker_pool pool) · 45 unmodified lines
2187	2799
2188	2800	worker->id = id;
2189	2801
2190		− if (pool->cpu >= 0)
2191		− snprintf(id_buf, sizeof(id_buf), "%d:%d%s", pool->cpu, id,
2192		− pool->attrs->nice < 0 ? "H" : "");
2193		− else
2194		− snprintf(id_buf, sizeof(id_buf), "u%d:%d", pool->id, id);
2195		−
2196		− worker->task = kthread_create_on_node(worker_thread, worker, pool->node,
2197		− "kworker/%s", id_buf);
2198		− if (IS_ERR(worker->task)) {
2199		− if (PTR_ERR(worker->task) == -EINTR) {
2200		− pr_err("workqueue: Interrupted when creating a worker thread \"kworker/%s\"\n",
2201		− id_buf);
2202		− } else {
2203		− pr_err_once("workqueue: Failed to create a worker thread: %pe",
2204		− worker->task);
	2802	+ if (!(pool->flags & POOL_BH)) {
	2803	+ if (pool->cpu >= 0)
	2804	+ snprintf(id_buf, sizeof(id_buf), "%d:%d%s", pool->cpu, id,
	2805	+ pool->attrs->nice < 0 ? "H" : "");
	2806	+ else
	2807	+ snprintf(id_buf, sizeof(id_buf), "u%d:%d", pool->id, id);
	2808	+
	2809	+ worker->task = kthread_create_on_node(worker_thread, worker,
	2810	+ pool->node, "kworker/%s", id_buf);
	2811	+ if (IS_ERR(worker->task)) {
	2812	+ if (PTR_ERR(worker->task) == -EINTR) {
	2813	+ pr_err("workqueue: Interrupted when creating a worker thread \"kworker/%s\"\n",
	2814	+ id_buf);
	2815	+ } else {
	2816	+ pr_err_once("workqueue: Failed to create a worker thread: %pe",
	2817	+ worker->task);
	2818	+ }
	2819	+ goto fail;
2205	2820	}
2206		− goto fail;
2207		− }
2208	2821
2209		− set_user_nice(worker->task, pool->attrs->nice);
2210		− kthread_bind_mask(worker->task, pool_allowed_cpus(pool));
	2822	+ set_user_nice(worker->task, pool->attrs->nice);
	2823	+ kthread_bind_mask(worker->task, pool_allowed_cpus(pool));
	2824	+ }
2211	2825
2212	2826	/* successful, attach the worker to the pool */
2213	2827	worker_attach_to_pool(worker, pool);
@@ -2217,14 +2831,14 @@ static struct worker create_worker(struct worker_pool pool) · 3 unmodified lines
2217	2831
2218	2832	worker->pool->nr_workers++;
2219	2833	worker_enter_idle(worker);
2220		− kick_pool(pool);
2221	2834
2222	2835	/*
2223	2836	* @worker is waiting on a completion in kthread() and will trigger hung
2224		− * check if not woken up soon. As kick_pool() might not have waken it
2225		− * up, wake it up explicitly once more.
	2837	+ * check if not woken up soon. As kick_pool() is noop if @pool is empty,
	2838	+ * wake it up explicitly.
2226	2839	*/
2227		− wake_up_process(worker->task);
	2840	+ if (worker->task)
	2841	+ wake_up_process(worker->task);
2228	2842
2229	2843	raw_spin_unlock_irq(&pool->lock);
2230	2844
@@ -2543,6 +3157,8 @@ __acquires(&pool->lock) · 312 unmodified lines
2543	3157	struct pool_workqueue *pwq = get_work_pwq(work);
2544	3158	struct worker_pool *pool = worker->pool;
2545	3159	unsigned long work_data;
	3160	+ int lockdep_start_depth, rcu_start_depth;
	3161	+ bool bh_draining = pool->flags & POOL_BH_DRAINING;
2546	3162	#ifdef CONFIG_LOCKDEP
2547	3163	/*
2548	3164	* It is permissible to free the struct work_struct from
@@ -2565,7 +3181,8 @@ __acquires(&pool->lock) · 16 unmodified lines
2565	3181	worker->current_work = work;
2566	3182	worker->current_func = work->func;
2567	3183	worker->current_pwq = pwq;
2568		− worker->current_at = worker->task->se.sum_exec_runtime;
	3184	+ if (worker->task)
	3185	+ worker->current_at = worker->task->se.sum_exec_runtime;
2569	3186	work_data = *work_data_bits(work);
2570	3187	worker->current_color = get_work_color(work_data);
2571	3188
@@ -2600,12 +3217,16 @@ __acquires(&pool->lock) · 28 unmodified lines
2600	3217	* PENDING and queued state changes happen together while IRQ is
2601	3218	* disabled.
2602	3219	*/
2603		− set_work_pool_and_clear_pending(work, pool->id);
	3220	+ set_work_pool_and_clear_pending(work, pool->id, 0);
2604	3221
2605	3222	pwq->stats[PWQ_STAT_STARTED]++;
2606	3223	raw_spin_unlock_irq(&pool->lock);
2607	3224
2608		− lock_map_acquire(&pwq->wq->lockdep_map);
	3225	+ rcu_start_depth = rcu_preempt_depth();
	3226	+ lockdep_start_depth = lockdep_depth(current);
	3227	+ /* see drain_dead_softirq_workfn() */
	3228	+ if (!bh_draining)
	3229	+ lock_map_acquire(&pwq->wq->lockdep_map);
2609	3230	lock_map_acquire(&lockdep_map);
2610	3231	/*
2611	3232	* Strictly speaking we should mark the invariant state without holding
@@ -2638,12 +3259,17 @@ __acquires(&pool->lock) · 26 unmodified lines
2638	3259	trace_workqueue_execute_end(work, worker->current_func);
2639	3260	pwq->stats[PWQ_STAT_COMPLETED]++;
2640	3261	lock_map_release(&lockdep_map);
2641		− lock_map_release(&pwq->wq->lockdep_map);
	3262	+ if (!bh_draining)
	3263	+ lock_map_release(&pwq->wq->lockdep_map);
2642	3264
2643		− if (unlikely(in_atomic() \|\| lockdep_depth(current) > 0)) {
2644		− pr_err("BUG: workqueue leaked lock or atomic: %s/0x%08x/%d\n"
2645		− " last function: %ps\n",
2646		− current->comm, preempt_count(), task_pid_nr(current),
	3265	+ if (unlikely((worker->task && in_atomic()) \|\|
	3266	+ lockdep_depth(current) != lockdep_start_depth \|\|
	3267	+ rcu_preempt_depth() != rcu_start_depth)) {
	3268	+ pr_err("BUG: workqueue leaked atomic, lock or RCU: %s[%d]\n"
	3269	+ " preempt=0x%08x lock=%d->%d RCU=%d->%d workfn=%ps\n",
	3270	+ current->comm, task_pid_nr(current), preempt_count(),
	3271	+ lockdep_start_depth, lockdep_depth(current),
	3272	+ rcu_start_depth, rcu_preempt_depth(),
2647	3273	worker->current_func);
2648	3274	debug_show_held_locks(current);
2649	3275	dump_stack();
@@ -2657,7 +3283,8 @@ __acquires(&pool->lock) · 7 unmodified lines
2657	3283	* stop_machine. At the same time, report a quiescent RCU state so
2658	3284	* the same condition doesn't freeze RCU.
2659	3285	*/
2660		− cond_resched();
	3286	+ if (worker->task)
	3287	+ cond_resched();
2661	3288
2662	3289	raw_spin_lock_irq(&pool->lock);
2663	3290
@@ -2677,6 +3304,8 @@ __acquires(&pool->lock) · 13 unmodified lines
2677	3304	worker->current_func = NULL;
2678	3305	worker->current_pwq = NULL;
2679	3306	worker->current_color = INT_MAX;
	3307	+
	3308	+ /* must be the last step, see the function comment */
2680	3309	pwq_dec_nr_in_flight(pwq, work_data);
2681	3310	}
2682	3311
@@ -2938,6 +3567,139 @@ repeat: · 255 unmodified lines
2938	3567	goto repeat;
2939	3568	}
2940	3569
	3570	+static void bh_worker(struct worker *worker)
	3571	+{
	3572	+ struct worker_pool *pool = worker->pool;
	3573	+ int nr_restarts = BH_WORKER_RESTARTS;
	3574	+ unsigned long end = jiffies + BH_WORKER_JIFFIES;
	3575	+
	3576	+ raw_spin_lock_irq(&pool->lock);
	3577	+ worker_leave_idle(worker);
	3578	+
	3579	+ /*
	3580	+ * This function follows the structure of worker_thread(). See there for
	3581	+ * explanations on each step.
	3582	+ */
	3583	+ if (!need_more_worker(pool))
	3584	+ goto done;
	3585	+
	3586	+ WARN_ON_ONCE(!list_empty(&worker->scheduled));
	3587	+ worker_clr_flags(worker, WORKER_PREP \| WORKER_REBOUND);
	3588	+
	3589	+ do {
	3590	+ struct work_struct *work =
	3591	+ list_first_entry(&pool->worklist,
	3592	+ struct work_struct, entry);
	3593	+
	3594	+ if (assign_work(work, worker, NULL))
	3595	+ process_scheduled_works(worker);
	3596	+ } while (keep_working(pool) &&
	3597	+ --nr_restarts && time_before(jiffies, end));
	3598	+
	3599	+ worker_set_flags(worker, WORKER_PREP);
	3600	+done:
	3601	+ worker_enter_idle(worker);
	3602	+ kick_pool(pool);
	3603	+ raw_spin_unlock_irq(&pool->lock);
	3604	+}
	3605	+
	3606	+/*
	3607	+ * TODO: Convert all tasklet users to workqueue and use softirq directly.
	3608	+ *
	3609	+ * This is currently called from tasklet[_hi]action() and thus is also called
	3610	+ * whenever there are tasklets to run. Let's do an early exit if there's nothing
	3611	+ * queued. Once conversion from tasklet is complete, the need_more_worker() test
	3612	+ * can be dropped.
	3613	+ *
	3614	+ * After full conversion, we'll add worker->softirq_action, directly use the
	3615	+ * softirq action and obtain the worker pointer from the softirq_action pointer.
	3616	+ */
	3617	+void workqueue_softirq_action(bool highpri)
	3618	+{
	3619	+ struct worker_pool *pool =
	3620	+ &per_cpu(bh_worker_pools, smp_processor_id())[highpri];
	3621	+ if (need_more_worker(pool))
	3622	+ bh_worker(list_first_entry(&pool->workers, struct worker, node));
	3623	+}
	3624	+
	3625	+struct wq_drain_dead_softirq_work {
	3626	+ struct work_struct work;
	3627	+ struct worker_pool *pool;
	3628	+ struct completion done;
	3629	+};
	3630	+
	3631	+static void drain_dead_softirq_workfn(struct work_struct *work)
	3632	+{
	3633	+ struct wq_drain_dead_softirq_work *dead_work =
	3634	+ container_of(work, struct wq_drain_dead_softirq_work, work);
	3635	+ struct worker_pool *pool = dead_work->pool;
	3636	+ bool repeat;
	3637	+
	3638	+ /*
	3639	+ * @pool's CPU is dead and we want to execute its still pending work
	3640	+ * items from this BH work item which is running on a different CPU. As
	3641	+ * its CPU is dead, @pool can't be kicked and, as work execution path
	3642	+ * will be nested, a lockdep annotation needs to be suppressed. Mark
	3643	+ * @pool with %POOL_BH_DRAINING for the special treatments.
	3644	+ */
	3645	+ raw_spin_lock_irq(&pool->lock);
	3646	+ pool->flags \|= POOL_BH_DRAINING;
	3647	+ raw_spin_unlock_irq(&pool->lock);
	3648	+
	3649	+ bh_worker(list_first_entry(&pool->workers, struct worker, node));
	3650	+
	3651	+ raw_spin_lock_irq(&pool->lock);
	3652	+ pool->flags &= ~POOL_BH_DRAINING;
	3653	+ repeat = need_more_worker(pool);
	3654	+ raw_spin_unlock_irq(&pool->lock);
	3655	+
	3656	+ /*
	3657	+ * bh_worker() might hit consecutive execution limit and bail. If there
	3658	+ * still are pending work items, reschedule self and return so that we
	3659	+ * don't hog this CPU's BH.
	3660	+ */
	3661	+ if (repeat) {
	3662	+ if (pool->attrs->nice == HIGHPRI_NICE_LEVEL)
	3663	+ queue_work(system_bh_highpri_wq, work);
	3664	+ else
	3665	+ queue_work(system_bh_wq, work);
	3666	+ } else {
	3667	+ complete(&dead_work->done);
	3668	+ }
	3669	+}
	3670	+
	3671	+/*
	3672	+ * @cpu is dead. Drain the remaining BH work items on the current CPU. It's
	3673	+ * possible to allocate dead_work per CPU and avoid flushing. However, then we
	3674	+ * have to worry about draining overlapping with CPU coming back online or
	3675	+ * nesting (one CPU's dead_work queued on another CPU which is also dead and so
	3676	+ * on). Let's keep it simple and drain them synchronously. These are BH work
	3677	+ * items which shouldn't be requeued on the same pool. Shouldn't take long.
	3678	+ */
	3679	+void workqueue_softirq_dead(unsigned int cpu)
	3680	+{
	3681	+ int i;
	3682	+
	3683	+ for (i = 0; i < NR_STD_WORKER_POOLS; i++) {
	3684	+ struct worker_pool *pool = &per_cpu(bh_worker_pools, cpu)[i];
	3685	+ struct wq_drain_dead_softirq_work dead_work;
	3686	+
	3687	+ if (!need_more_worker(pool))
	3688	+ continue;
	3689	+
	3690	+ INIT_WORK(&dead_work.work, drain_dead_softirq_workfn);
	3691	+ dead_work.pool = pool;
	3692	+ init_completion(&dead_work.done);
	3693	+
	3694	+ if (pool->attrs->nice == HIGHPRI_NICE_LEVEL)
	3695	+ queue_work(system_bh_highpri_wq, &dead_work.work);
	3696	+ else
	3697	+ queue_work(system_bh_wq, &dead_work.work);
	3698	+
	3699	+ wait_for_completion(&dead_work.done);
	3700	+ }
	3701	+}
	3702	+
2941	3703	/**
2942	3704	* check_flush_dependency - check for flush dependency sanity
2943	3705	* @target_wq: workqueue being flushed
@@ -3010,6 +3772,7 @@ static void insert_wq_barrier(struct pool_workqueue *pwq, · 66 unmodified lines
3010	3772	struct wq_barrier *barr,
3011	3773	struct work_struct target, struct worker worker)
3012	3774	{
	3775	+ static __maybe_unused struct lock_class_key bh_key, thr_key;
3013	3776	unsigned int work_flags = 0;
3014	3777	unsigned int work_color;
3015	3778	struct list_head *head;
@@ -3019,15 +3782,20 @@ static void insert_wq_barrier(struct pool_workqueue *pwq, · 3 unmodified lines
3019	3782	* as we know for sure that this will not trigger any of the
3020	3783	* checks and call back into the fixup functions where we
3021	3784	* might deadlock.
	3785	+ *
	3786	+ * BH and threaded workqueues need separate lockdep keys to avoid
	3787	+ * spuriously triggering "inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W}
	3788	+ * usage".
3022	3789	*/
3023		− INIT_WORK_ONSTACK(&barr->work, wq_barrier_func);
	3790	+ INIT_WORK_ONSTACK_KEY(&barr->work, wq_barrier_func,
	3791	+ (pwq->wq->flags & WQ_BH) ? &bh_key : &thr_key);
3024	3792	__set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(&barr->work));
3025	3793
3026	3794	init_completion_map(&barr->done, &target->lockdep_map);
3027	3795
3028	3796	barr->task = current;
3029	3797
3030		− /* The barrier work item does not participate in pwq->nr_active. */
	3798	+ /* The barrier work item does not participate in nr_active. */
3031	3799	work_flags \|= WORK_STRUCT_INACTIVE;
3032	3800
3033	3801	/*
@@ -3124,6 +3892,35 @@ static bool flush_workqueue_prep_pwqs(struct workqueue_struct *wq, · 90 unmodified lines
3124	3892	return wait;
3125	3893	}
3126	3894
	3895	+static void touch_wq_lockdep_map(struct workqueue_struct *wq)
	3896	+{
	3897	+#ifdef CONFIG_LOCKDEP
	3898	+ if (wq->flags & WQ_BH)
	3899	+ local_bh_disable();
	3900	+
	3901	+ lock_map_acquire(&wq->lockdep_map);
	3902	+ lock_map_release(&wq->lockdep_map);
	3903	+
	3904	+ if (wq->flags & WQ_BH)
	3905	+ local_bh_enable();
	3906	+#endif
	3907	+}
	3908	+
	3909	+static void touch_work_lockdep_map(struct work_struct *work,
	3910	+ struct workqueue_struct *wq)
	3911	+{
	3912	+#ifdef CONFIG_LOCKDEP
	3913	+ if (wq->flags & WQ_BH)
	3914	+ local_bh_disable();
	3915	+
	3916	+ lock_map_acquire(&work->lockdep_map);
	3917	+ lock_map_release(&work->lockdep_map);
	3918	+
	3919	+ if (wq->flags & WQ_BH)
	3920	+ local_bh_enable();
	3921	+#endif
	3922	+}
	3923	+
3127	3924	/**
3128	3925	* __flush_workqueue - ensure that any scheduled work has run to completion.
3129	3926	* @wq: workqueue to flush
@@ -3143,8 +3940,7 @@ void __flush_workqueue(struct workqueue_struct *wq) · 13 unmodified lines
3143	3940	if (WARN_ON(!wq_online))
3144	3941	return;
3145	3942
3146		− lock_map_acquire(&wq->lockdep_map);
3147		− lock_map_release(&wq->lockdep_map);
	3943	+ touch_wq_lockdep_map(wq);
3148	3944
3149	3945	mutex_lock(&wq->mutex);
3150	3946
@@ -3316,7 +4112,7 @@ reflush: · 165 unmodified lines
3316	4112	bool drained;
3317	4113
3318	4114	raw_spin_lock_irq(&pwq->pool->lock);
3319		− drained = !pwq->nr_active && list_empty(&pwq->inactive_works);
	4115	+ drained = pwq_is_empty(pwq);
3320	4116	raw_spin_unlock_irq(&pwq->pool->lock);
3321	4117
3322	4118	if (drained)
@@ -3343,6 +4139,7 @@ static bool start_flush_work(struct work_struct work, struct wq_barrier barr, · 20 unmodified lines
3343	4139	struct worker *worker = NULL;
3344	4140	struct worker_pool *pool;
3345	4141	struct pool_workqueue *pwq;
	4142	+ struct workqueue_struct *wq;
3346	4143
3347	4144	might_sleep();
3348	4145
@@ -3366,11 +4163,14 @@ static bool start_flush_work(struct work_struct work, struct wq_barrier barr, · 17 unmodified lines
3366	4163	pwq = worker->current_pwq;
3367	4164	}
3368	4165
3369		− check_flush_dependency(pwq->wq, work);
	4166	+ wq = pwq->wq;
	4167	+ check_flush_dependency(wq, work);
3370	4168
3371	4169	insert_wq_barrier(pwq, barr, work, worker);
3372	4170	raw_spin_unlock_irq(&pool->lock);
3373	4171
	4172	+ touch_work_lockdep_map(work, wq);
	4173	+
3374	4174	/*
3375	4175	* Force a lock recursion deadlock when using flush_work() inside a
3376	4176	* single-threaded or rescuer equipped workqueue.
@@ -3380,11 +4180,9 @@ static bool start_flush_work(struct work_struct work, struct wq_barrier barr, · 3 unmodified lines
3380	4180	* workqueues the deadlock happens when the rescuer stalls, blocking
3381	4181	* forward progress.
3382	4182	*/
3383		− if (!from_cancel &&
3384		− (pwq->wq->saved_max_active == 1 \|\| pwq->wq->rescuer)) {
3385		− lock_map_acquire(&pwq->wq->lockdep_map);
3386		− lock_map_release(&pwq->wq->lockdep_map);
3387		− }
	4183	+ if (!from_cancel && (wq->saved_max_active == 1 \|\| wq->rescuer))
	4184	+ touch_wq_lockdep_map(wq);
	4185	+
3388	4186	rcu_read_unlock();
3389	4187	return true;
3390	4188	already_gone:
@@ -3393,146 +4191,41 @@ already_gone: · 2 unmodified lines
3393	4191	return false;
3394	4192	}
3395	4193
3396		−static bool __flush_work(struct work_struct *work, bool from_cancel)
3397		−{
3398		− struct wq_barrier barr;
3399		−
3400		− if (WARN_ON(!wq_online))
3401		− return false;
3402		−
3403		− if (WARN_ON(!work->func))
3404		− return false;
3405		−
3406		− lock_map_acquire(&work->lockdep_map);
3407		− lock_map_release(&work->lockdep_map);
3408		−
3409		− if (start_flush_work(work, &barr, from_cancel)) {
3410		− wait_for_completion(&barr.done);
3411		− destroy_work_on_stack(&barr.work);
3412		− return true;
3413		− } else {
3414		− return false;
3415		− }
3416		−}
3417		−
3418		−/**
3419		− * flush_work - wait for a work to finish executing the last queueing instance
3420		− * @work: the work to flush
3421		− *
3422		− * Wait until @work has finished execution. @work is guaranteed to be idle
3423		− * on return if it hasn't been requeued since flush started.
3424		− *
3425		− * Return:
3426		− * %true if flush_work() waited for the work to finish execution,
3427		− * %false if it was already idle.
3428		− */
3429		−bool flush_work(struct work_struct *work)
3430		−{
3431		− return __flush_work(work, false);
3432		−}
3433		−EXPORT_SYMBOL_GPL(flush_work);
3434		−
3435		−struct cwt_wait {
3436		− wait_queue_entry_t wait;
3437		− struct work_struct *work;
3438		−};
3439		−
3440		−static int cwt_wakefn(wait_queue_entry_t wait, unsigned mode, int sync, void key)
3441		−{
3442		− struct cwt_wait *cwait = container_of(wait, struct cwt_wait, wait);
3443		−
3444		− if (cwait->work != key)
3445		− return 0;
3446		− return autoremove_wake_function(wait, mode, sync, key);
3447		−}
3448		−
3449		−static bool __cancel_work_timer(struct work_struct *work, bool is_dwork)
3450		−{
3451		− static DECLARE_WAIT_QUEUE_HEAD(cancel_waitq);
3452		− unsigned long flags;
3453		− int ret;
3454		−
3455		− do {
3456		− ret = try_to_grab_pending(work, is_dwork, &flags);
3457		− /*
3458		− * If someone else is already canceling, wait for it to
3459		− * finish. flush_work() doesn't work for PREEMPT_NONE
3460		− * because we may get scheduled between @work's completion
3461		− * and the other canceling task resuming and clearing
3462		− * CANCELING - flush_work() will return false immediately
3463		− * as @work is no longer busy, try_to_grab_pending() will
3464		− * return -ENOENT as @work is still being canceled and the
3465		− * other canceling task won't be able to clear CANCELING as
3466		− * we're hogging the CPU.
3467		− *
3468		− * Let's wait for completion using a waitqueue. As this
3469		− * may lead to the thundering herd problem, use a custom
3470		− * wake function which matches @work along with exclusive
3471		− * wait and wakeup.
3472		− */
3473		− if (unlikely(ret == -ENOENT)) {
3474		− struct cwt_wait cwait;
3475		−
3476		− init_wait(&cwait.wait);
3477		− cwait.wait.func = cwt_wakefn;
3478		− cwait.work = work;
3479		−
3480		− prepare_to_wait_exclusive(&cancel_waitq, &cwait.wait,
3481		− TASK_UNINTERRUPTIBLE);
3482		− if (work_is_canceling(work))
3483		− schedule();
3484		− finish_wait(&cancel_waitq, &cwait.wait);
3485		− }
3486		− } while (unlikely(ret < 0));
3487		−
3488		− /* tell other tasks trying to grab @work to back off */
3489		− mark_work_canceling(work);
3490		− local_irq_restore(flags);
3491		−
3492		− /*
3493		− * This allows canceling during early boot. We know that @work
3494		− * isn't executing.
3495		− */
3496		− if (wq_online)
3497		− __flush_work(work, true);
	4194	+static bool __flush_work(struct work_struct *work, bool from_cancel)
	4195	+{
	4196	+ struct wq_barrier barr;
3498	4197
3499		− clear_work_data(work);
	4198	+ if (WARN_ON(!wq_online))
	4199	+ return false;
3500	4200
3501		− /*
3502		− * Paired with prepare_to_wait() above so that either
3503		− * waitqueue_active() is visible here or !work_is_canceling() is
3504		− * visible there.
3505		− */
3506		− smp_mb();
3507		− if (waitqueue_active(&cancel_waitq))
3508		− __wake_up(&cancel_waitq, TASK_NORMAL, 1, work);
	4201	+ if (WARN_ON(!work->func))
	4202	+ return false;
3509	4203
3510		− return ret;
	4204	+ if (start_flush_work(work, &barr, from_cancel)) {
	4205	+ wait_for_completion(&barr.done);
	4206	+ destroy_work_on_stack(&barr.work);
	4207	+ return true;
	4208	+ } else {
	4209	+ return false;
	4210	+ }
3511	4211	}
3512	4212
3513	4213	/**
3514		− * cancel_work_sync - cancel a work and wait for it to finish
3515		− * @work: the work to cancel
3516		− *
3517		− * Cancel @work and wait for its execution to finish. This function
3518		− * can be used even if the work re-queues itself or migrates to
3519		− * another workqueue. On return from this function, @work is
3520		− * guaranteed to be not pending or executing on any CPU.
3521		− *
3522		− * cancel_work_sync(&delayed_work->work) must not be used for
3523		− * delayed_work's. Use cancel_delayed_work_sync() instead.
	4214	+ * flush_work - wait for a work to finish executing the last queueing instance
	4215	+ * @work: the work to flush
3524	4216	*
3525		− * The caller must ensure that the workqueue on which @work was last
3526		− * queued can't be destroyed before this function returns.
	4217	+ * Wait until @work has finished execution. @work is guaranteed to be idle
	4218	+ * on return if it hasn't been requeued since flush started.
3527	4219	*
3528	4220	* Return:
3529		− * %true if @work was pending, %false otherwise.
	4221	+ * %true if flush_work() waited for the work to finish execution,
	4222	+ * %false if it was already idle.
3530	4223	*/
3531		−bool cancel_work_sync(struct work_struct *work)
	4224	+bool flush_work(struct work_struct *work)
3532	4225	{
3533		− return __cancel_work_timer(work, false);
	4226	+ return __flush_work(work, false);
3534	4227	}
3535		−EXPORT_SYMBOL_GPL(cancel_work_sync);
	4228	+EXPORT_SYMBOL_GPL(flush_work);
3536	4229
3537	4230	/**
3538	4231	* flush_delayed_work - wait for a dwork to finish executing the last queueing
@@ -3576,20 +4269,50 @@ bool flush_rcu_work(struct rcu_work *rwork) · 37 unmodified lines
3576	4269	}
3577	4270	EXPORT_SYMBOL(flush_rcu_work);
3578	4271
3579		−static bool __cancel_work(struct work_struct *work, bool is_dwork)
	4272	+static bool __cancel_work(struct work_struct *work, u32 cflags)
3580	4273	{
3581		− unsigned long flags;
	4274	+ unsigned long irq_flags;
3582	4275	int ret;
3583	4276
3584	4277	do {
3585		− ret = try_to_grab_pending(work, is_dwork, &flags);
	4278	+ ret = try_to_grab_pending(work, cflags, &irq_flags);
3586	4279	} while (unlikely(ret == -EAGAIN));
3587	4280
3588	4281	if (unlikely(ret < 0))
3589	4282	return false;
3590	4283
3591		− set_work_pool_and_clear_pending(work, get_work_pool_id(work));
3592		− local_irq_restore(flags);
	4284	+ set_work_pool_and_clear_pending(work, get_work_pool_id(work), 0);
	4285	+ local_irq_restore(irq_flags);
	4286	+ return ret;
	4287	+}
	4288	+
	4289	+static bool __cancel_work_sync(struct work_struct *work, u32 cflags)
	4290	+{
	4291	+ unsigned long irq_flags;
	4292	+ bool ret;
	4293	+
	4294	+ /* claim @work and tell other tasks trying to grab @work to back off */
	4295	+ ret = work_grab_pending(work, cflags, &irq_flags);
	4296	+ mark_work_canceling(work);
	4297	+ local_irq_restore(irq_flags);
	4298	+
	4299	+ /*
	4300	+ * Skip __flush_work() during early boot when we know that @work isn't
	4301	+ * executing. This allows canceling during early boot.
	4302	+ */
	4303	+ if (wq_online)
	4304	+ __flush_work(work, true);
	4305	+
	4306	+ /*
	4307	+ * smp_mb() at the end of set_work_pool_and_clear_pending() is paired
	4308	+ * with prepare_to_wait() above so that either waitqueue_active() is
	4309	+ * visible here or !work_is_canceling() is visible there.
	4310	+ */
	4311	+ set_work_pool_and_clear_pending(work, WORK_OFFQ_POOL_NONE, 0);
	4312	+
	4313	+ if (waitqueue_active(&wq_cancel_waitq))
	4314	+ __wake_up(&wq_cancel_waitq, TASK_NORMAL, 1, work);
	4315	+
3593	4316	return ret;
3594	4317	}
3595	4318
@@ -3598,10 +4321,34 @@ static bool __cancel_work(struct work_struct *work, bool is_dwork) · 2 unmodified lines
3598	4321	*/
3599	4322	bool cancel_work(struct work_struct *work)
3600	4323	{
3601		− return __cancel_work(work, false);
	4324	+ return __cancel_work(work, 0);
3602	4325	}
3603	4326	EXPORT_SYMBOL(cancel_work);
3604	4327
	4328	+/**
	4329	+ * cancel_work_sync - cancel a work and wait for it to finish
	4330	+ * @work: the work to cancel
	4331	+ *
	4332	+ * Cancel @work and wait for its execution to finish. This function
	4333	+ * can be used even if the work re-queues itself or migrates to
	4334	+ * another workqueue. On return from this function, @work is
	4335	+ * guaranteed to be not pending or executing on any CPU.
	4336	+ *
	4337	+ * cancel_work_sync(&delayed_work->work) must not be used for
	4338	+ * delayed_work's. Use cancel_delayed_work_sync() instead.
	4339	+ *
	4340	+ * The caller must ensure that the workqueue on which @work was last
	4341	+ * queued can't be destroyed before this function returns.
	4342	+ *
	4343	+ * Return:
	4344	+ * %true if @work was pending, %false otherwise.
	4345	+ */
	4346	+bool cancel_work_sync(struct work_struct *work)
	4347	+{
	4348	+ return __cancel_work_sync(work, 0);
	4349	+}
	4350	+EXPORT_SYMBOL_GPL(cancel_work_sync);
	4351	+
3605	4352	/**
3606	4353	* cancel_delayed_work - cancel a delayed work
3607	4354	* @dwork: delayed_work to cancel
@@ -3620,7 +4367,7 @@ EXPORT_SYMBOL(cancel_work); · 12 unmodified lines
3620	4367	*/
3621	4368	bool cancel_delayed_work(struct delayed_work *dwork)
3622	4369	{
3623		− return __cancel_work(&dwork->work, true);
	4370	+ return __cancel_work(&dwork->work, WORK_CANCEL_DELAYED);
3624	4371	}
3625	4372	EXPORT_SYMBOL(cancel_delayed_work);
3626	4373
@@ -3635,7 +4382,7 @@ EXPORT_SYMBOL(cancel_delayed_work); · 8 unmodified lines
3635	4382	*/
3636	4383	bool cancel_delayed_work_sync(struct delayed_work *dwork)
3637	4384	{
3638		− return __cancel_work_timer(&dwork->work, true);
	4385	+ return __cancel_work_sync(&dwork->work, WORK_CANCEL_DELAYED);
3639	4386	}
3640	4387	EXPORT_SYMBOL(cancel_delayed_work_sync);
3641	4388
@@ -3927,11 +4674,66 @@ static void wq_free_lockdep(struct workqueue_struct *wq) · 285 unmodified lines
3927	4674	}
3928	4675	#endif
3929	4676
	4677	+static void free_node_nr_active(struct wq_node_nr_active **nna_ar)
	4678	+{
	4679	+ int node;
	4680	+
	4681	+ for_each_node(node) {
	4682	+ kfree(nna_ar[node]);
	4683	+ nna_ar[node] = NULL;
	4684	+ }
	4685	+
	4686	+ kfree(nna_ar[nr_node_ids]);
	4687	+ nna_ar[nr_node_ids] = NULL;
	4688	+}
	4689	+
	4690	+static void init_node_nr_active(struct wq_node_nr_active *nna)
	4691	+{
	4692	+ nna->max = WQ_DFL_MIN_ACTIVE;
	4693	+ atomic_set(&nna->nr, 0);
	4694	+ raw_spin_lock_init(&nna->lock);
	4695	+ INIT_LIST_HEAD(&nna->pending_pwqs);
	4696	+}
	4697	+
	4698	+/*
	4699	+ * Each node's nr_active counter will be accessed mostly from its own node and
	4700	+ * should be allocated in the node.
	4701	+ */
	4702	+static int alloc_node_nr_active(struct wq_node_nr_active **nna_ar)
	4703	+{
	4704	+ struct wq_node_nr_active *nna;
	4705	+ int node;
	4706	+
	4707	+ for_each_node(node) {
	4708	+ nna = kzalloc_node(sizeof(*nna), GFP_KERNEL, node);
	4709	+ if (!nna)
	4710	+ goto err_free;
	4711	+ init_node_nr_active(nna);
	4712	+ nna_ar[node] = nna;
	4713	+ }
	4714	+
	4715	+ /* [nr_node_ids] is used as the fallback */
	4716	+ nna = kzalloc_node(sizeof(*nna), GFP_KERNEL, NUMA_NO_NODE);
	4717	+ if (!nna)
	4718	+ goto err_free;
	4719	+ init_node_nr_active(nna);
	4720	+ nna_ar[nr_node_ids] = nna;
	4721	+
	4722	+ return 0;
	4723	+
	4724	+err_free:
	4725	+ free_node_nr_active(nna_ar);
	4726	+ return -ENOMEM;
	4727	+}
	4728	+
3930	4729	static void rcu_free_wq(struct rcu_head *rcu)
3931	4730	{
3932	4731	struct workqueue_struct *wq =
3933	4732	container_of(rcu, struct workqueue_struct, rcu);
3934	4733
	4734	+ if (wq->flags & WQ_UNBOUND)
	4735	+ free_node_nr_active(wq->node_nr_active);
	4736	+
3935	4737	wq_free_lockdep(wq);
3936	4738	free_percpu(wq->cpu_pwq);
3937	4739	free_workqueue_attrs(wq->unbound_attrs);
@@ -4121,6 +4923,13 @@ static void pwq_release_workfn(struct kthread_work *work) · 183 unmodified lines
4121	4923	mutex_lock(&wq->mutex);
4122	4924	list_del_rcu(&pwq->pwqs_node);
4123	4925	is_last = list_empty(&wq->pwqs);
	4926	+
	4927	+ /*
	4928	+ * For ordered workqueue with a plugged dfl_pwq, restart it now.
	4929	+ */
	4930	+ if (!is_last && (wq->flags & __WQ_ORDERED))
	4931	+ unplug_oldest_pwq(wq);
	4932	+
4124	4933	mutex_unlock(&wq->mutex);
4125	4934	}
4126	4935
@@ -4130,6 +4939,15 @@ static void pwq_release_workfn(struct kthread_work *work) · 3 unmodified lines
4130	4939	mutex_unlock(&wq_pool_mutex);
4131	4940	}
4132	4941
	4942	+ if (!list_empty(&pwq->pending_node)) {
	4943	+ struct wq_node_nr_active *nna =
	4944	+ wq_node_nr_active(pwq->wq, pwq->pool->node);
	4945	+
	4946	+ raw_spin_lock_irq(&nna->lock);
	4947	+ list_del_init(&pwq->pending_node);
	4948	+ raw_spin_unlock_irq(&nna->lock);
	4949	+ }
	4950	+
4133	4951	call_rcu(&pwq->rcu, rcu_free_pwq);
4134	4952
4135	4953	/*
@@ -4142,55 +4960,11 @@ static void pwq_release_workfn(struct kthread_work *work) · 6 unmodified lines
4142	4960	}
4143	4961	}
4144	4962
4145		−/**
4146		− * pwq_adjust_max_active - update a pwq's max_active to the current setting
4147		− * @pwq: target pool_workqueue
4148		− *
4149		− * If @pwq isn't freezing, set @pwq->max_active to the associated
4150		− * workqueue's saved_max_active and activate inactive work items
4151		− * accordingly. If @pwq is freezing, clear @pwq->max_active to zero.
4152		− */
4153		−static void pwq_adjust_max_active(struct pool_workqueue *pwq)
4154		−{
4155		− struct workqueue_struct *wq = pwq->wq;
4156		− bool freezable = wq->flags & WQ_FREEZABLE;
4157		− unsigned long flags;
4158		−
4159		− /* for @wq->saved_max_active */
4160		− lockdep_assert_held(&wq->mutex);
4161		−
4162		− /* fast exit for non-freezable wqs */
4163		− if (!freezable && pwq->max_active == wq->saved_max_active)
4164		− return;
4165		−
4166		− /* this function can be called during early boot w/ irq disabled */
4167		− raw_spin_lock_irqsave(&pwq->pool->lock, flags);
4168		−
4169		− /*
4170		− * During [un]freezing, the caller is responsible for ensuring that
4171		− * this function is called at least once after @workqueue_freezing
4172		− * is updated and visible.
4173		− */
4174		− if (!freezable \|\| !workqueue_freezing) {
4175		− pwq->max_active = wq->saved_max_active;
4176		−
4177		− while (!list_empty(&pwq->inactive_works) &&
4178		− pwq->nr_active < pwq->max_active)
4179		− pwq_activate_first_inactive(pwq);
4180		−
4181		− kick_pool(pwq->pool);
4182		− } else {
4183		− pwq->max_active = 0;
4184		− }
4185		−
4186		− raw_spin_unlock_irqrestore(&pwq->pool->lock, flags);
4187		−}
4188		−
4189	4963	/* initialize newly allocated @pwq which is associated with @wq and @pool */
4190	4964	static void init_pwq(struct pool_workqueue pwq, struct workqueue_struct wq,
4191	4965	struct worker_pool *pool)
4192	4966	{
4193		− BUG_ON((unsigned long)pwq & WORK_STRUCT_FLAG_MASK);
	4967	+ BUG_ON((unsigned long)pwq & ~WORK_STRUCT_PWQ_MASK);
4194	4968
4195	4969	memset(pwq, 0, sizeof(*pwq));
4196	4970
@@ -4199,6 +4973,7 @@ static void init_pwq(struct pool_workqueue pwq, struct workqueue_struct wq, · 2 unmodified lines
4199	4973	pwq->flush_color = -1;
4200	4974	pwq->refcnt = 1;
4201	4975	INIT_LIST_HEAD(&pwq->inactive_works);
	4976	+ INIT_LIST_HEAD(&pwq->pending_node);
4202	4977	INIT_LIST_HEAD(&pwq->pwqs_node);
4203	4978	INIT_LIST_HEAD(&pwq->mayday_node);
4204	4979	kthread_init_work(&pwq->release_work, pwq_release_workfn);
@@ -4218,11 +4993,8 @@ static void link_pwq(struct pool_workqueue *pwq) · 13 unmodified lines
4218	4993	/* set the matching work_color */
4219	4994	pwq->work_color = wq->work_color;
4220	4995
4221		− /* sync max_active to the current setting */
4222		− pwq_adjust_max_active(pwq);
4223		−
4224	4996	/* link in @pwq */
4225		− list_add_rcu(&pwq->pwqs_node, &wq->pwqs);
	4997	+ list_add_tail_rcu(&pwq->pwqs_node, &wq->pwqs);
4226	4998	}
4227	4999
4228	5000	/* obtain a pool matching @attr and create a pwq associating the pool and @wq */
@@ -4289,10 +5061,11 @@ static void wq_calc_pod_cpumask(struct workqueue_attrs *attrs, int cpu, · 60 unmodified lines
4289	5061	"possible intersect\n");
4290	5062	}
4291	5063
4292		−/* install @pwq into @wq's cpu_pwq and return the old pwq */
	5064	+/* install @pwq into @wq and return the old pwq, @cpu < 0 for dfl_pwq */
4293	5065	static struct pool_workqueue install_unbound_pwq(struct workqueue_struct wq,
4294	5066	int cpu, struct pool_workqueue *pwq)
4295	5067	{
	5068	+ struct pool_workqueue __rcu **slot = unbound_pwq_slot(wq, cpu);
4296	5069	struct pool_workqueue *old_pwq;
4297	5070
4298	5071	lockdep_assert_held(&wq_pool_mutex);
@@ -4301,8 +5074,8 @@ static struct pool_workqueue install_unbound_pwq(struct workqueue_struct wq, · 2 unmodified lines
4301	5074	/* link_pwq() can handle duplicate calls */
4302	5075	link_pwq(pwq);
4303	5076
4304		− old_pwq = rcu_access_pointer(*per_cpu_ptr(wq->cpu_pwq, cpu));
4305		− rcu_assign_pointer(*per_cpu_ptr(wq->cpu_pwq, cpu), pwq);
	5077	+ old_pwq = rcu_access_pointer(*slot);
	5078	+ rcu_assign_pointer(*slot, pwq);
4306	5079	return old_pwq;
4307	5080	}
4308	5081
@@ -4383,6 +5156,15 @@ apply_wqattrs_prepare(struct workqueue_struct *wq, · 74 unmodified lines
4383	5156	cpumask_copy(new_attrs->__pod_cpumask, new_attrs->cpumask);
4384	5157	ctx->attrs = new_attrs;
4385	5158
	5159	+ /*
	5160	+ * For initialized ordered workqueues, there should only be one pwq
	5161	+ * (dfl_pwq). Set the plugged flag of ctx->dfl_pwq to suspend execution
	5162	+ * of newly queued work items until execution of older work items in
	5163	+ * the old pwq's have completed.
	5164	+ */
	5165	+ if ((wq->flags & __WQ_ORDERED) && !list_empty(&wq->pwqs))
	5166	+ ctx->dfl_pwq->plugged = true;
	5167	+
4386	5168	ctx->wq = wq;
4387	5169	return ctx;
4388	5170
@@ -4402,14 +5184,19 @@ static void apply_wqattrs_commit(struct apply_wqattrs_ctx *ctx) · 13 unmodified lines
4402	5184
4403	5185	copy_workqueue_attrs(ctx->wq->unbound_attrs, ctx->attrs);
4404	5186
4405		− /* save the previous pwq and install the new one */
	5187	+ /* save the previous pwqs and install the new ones */
4406	5188	for_each_possible_cpu(cpu)
4407	5189	ctx->pwq_tbl[cpu] = install_unbound_pwq(ctx->wq, cpu,
4408	5190	ctx->pwq_tbl[cpu]);
	5191	+ ctx->dfl_pwq = install_unbound_pwq(ctx->wq, -1, ctx->dfl_pwq);
	5192	+
	5193	+ /* update node_nr_active->max */
	5194	+ wq_update_node_max_active(ctx->wq, -1);
4409	5195
4410		− /* @dfl_pwq might not have been used, ensure it's linked */
4411		− link_pwq(ctx->dfl_pwq);
4412		− swap(ctx->wq->dfl_pwq, ctx->dfl_pwq);
	5196	+ /* rescuer needs to respect wq cpumask changes */
	5197	+ if (ctx->wq->rescuer)
	5198	+ set_cpus_allowed_ptr(ctx->wq->rescuer->task,
	5199	+ unbound_effective_cpumask(ctx->wq));
4413	5200
4414	5201	mutex_unlock(&ctx->wq->mutex);
4415	5202	}
@@ -4423,14 +5210,6 @@ static int apply_workqueue_attrs_locked(struct workqueue_struct *wq, · 7 unmodified lines
4423	5210	if (WARN_ON(!(wq->flags & WQ_UNBOUND)))
4424	5211	return -EINVAL;
4425	5212
4426		− /* creating multiple pwqs breaks ordering guarantee */
4427		− if (!list_empty(&wq->pwqs)) {
4428		− if (WARN_ON(wq->flags & __WQ_ORDERED_EXPLICIT))
4429		− return -EINVAL;
4430		−
4431		− wq->flags &= ~__WQ_ORDERED;
4432		− }
4433		−
4434	5213	ctx = apply_wqattrs_prepare(wq, attrs, wq_unbound_cpumask);
4435	5214	if (IS_ERR(ctx))
4436	5215	return PTR_ERR(ctx);
@@ -4519,9 +5298,7 @@ static void wq_update_pod(struct workqueue_struct *wq, int cpu, · 82 unmodified lines
4519	5298
4520	5299	/* nothing to do if the target cpumask matches the current pwq */
4521	5300	wq_calc_pod_cpumask(target_attrs, cpu, off_cpu);
4522		− pwq = rcu_dereference_protected(*per_cpu_ptr(wq->cpu_pwq, cpu),
4523		− lockdep_is_held(&wq_pool_mutex));
4524		− if (wqattrs_equal(target_attrs, pwq->pool->attrs))
	5301	+ if (wqattrs_equal(target_attrs, unbound_pwq(wq, cpu)->pool->attrs))
4525	5302	return;
4526	5303
4527	5304	/* create a new pwq */
@@ -4539,10 +5316,11 @@ static void wq_update_pod(struct workqueue_struct *wq, int cpu, · 11 unmodified lines
4539	5316
4540	5317	use_dfl_pwq:
4541	5318	mutex_lock(&wq->mutex);
4542		− raw_spin_lock_irq(&wq->dfl_pwq->pool->lock);
4543		− get_pwq(wq->dfl_pwq);
4544		− raw_spin_unlock_irq(&wq->dfl_pwq->pool->lock);
4545		− old_pwq = install_unbound_pwq(wq, cpu, wq->dfl_pwq);
	5319	+ pwq = unbound_pwq(wq, -1);
	5320	+ raw_spin_lock_irq(&pwq->pool->lock);
	5321	+ get_pwq(pwq);
	5322	+ raw_spin_unlock_irq(&pwq->pool->lock);
	5323	+ old_pwq = install_unbound_pwq(wq, cpu, pwq);
4546	5324	out_unlock:
4547	5325	mutex_unlock(&wq->mutex);
4548	5326	put_pwq_unlocked(old_pwq);
@@ -4559,10 +5337,17 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq) · 10 unmodified lines
4559	5337
4560	5338	if (!(wq->flags & WQ_UNBOUND)) {
4561	5339	for_each_possible_cpu(cpu) {
4562		− struct pool_workqueue **pwq_p =
4563		− per_cpu_ptr(wq->cpu_pwq, cpu);
4564		− struct worker_pool *pool =
4565		− &(per_cpu_ptr(cpu_worker_pools, cpu)[highpri]);
	5340	+ struct pool_workqueue **pwq_p;
	5341	+ struct worker_pool __percpu *pools;
	5342	+ struct worker_pool *pool;
	5343	+
	5344	+ if (wq->flags & WQ_BH)
	5345	+ pools = bh_worker_pools;
	5346	+ else
	5347	+ pools = cpu_worker_pools;
	5348	+
	5349	+ pool = &(per_cpu_ptr(pools, cpu)[highpri]);
	5350	+ pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
4566	5351
4567	5352	*pwq_p = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL,
4568	5353	pool->node);
@@ -4580,10 +5365,13 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq) · 11 unmodified lines
4580	5365
4581	5366	cpus_read_lock();
4582	5367	if (wq->flags & __WQ_ORDERED) {
	5368	+ struct pool_workqueue *dfl_pwq;
	5369	+
4583	5370	ret = apply_workqueue_attrs(wq, ordered_wq_attrs[highpri]);
4584	5371	/* there should only be single pwq for ordering guarantee */
4585		− WARN(!ret && (wq->pwqs.next != &wq->dfl_pwq->pwqs_node \|\|
4586		− wq->pwqs.prev != &wq->dfl_pwq->pwqs_node),
	5372	+ dfl_pwq = rcu_access_pointer(wq->dfl_pwq);
	5373	+ WARN(!ret && (wq->pwqs.next != &dfl_pwq->pwqs_node \|\|
	5374	+ wq->pwqs.prev != &dfl_pwq->pwqs_node),
4587	5375	"ordering guarantee broken for workqueue %s\n", wq->name);
4588	5376	} else {
4589	5377	ret = apply_workqueue_attrs(wq, unbound_std_wq_attrs[highpri]);
@@ -4652,12 +5440,78 @@ static int init_rescuer(struct workqueue_struct *wq) · 62 unmodified lines
4652	5440	}
4653	5441
4654	5442	wq->rescuer = rescuer;
4655		− kthread_bind_mask(rescuer->task, cpu_possible_mask);
	5443	+ if (wq->flags & WQ_UNBOUND)
	5444	+ kthread_bind_mask(rescuer->task, wq_unbound_cpumask);
	5445	+ else
	5446	+ kthread_bind_mask(rescuer->task, cpu_possible_mask);
4656	5447	wake_up_process(rescuer->task);
4657	5448
4658	5449	return 0;
4659	5450	}
4660	5451
	5452	+/**
	5453	+ * wq_adjust_max_active - update a wq's max_active to the current setting
	5454	+ * @wq: target workqueue
	5455	+ *
	5456	+ * If @wq isn't freezing, set @wq->max_active to the saved_max_active and
	5457	+ * activate inactive work items accordingly. If @wq is freezing, clear
	5458	+ * @wq->max_active to zero.
	5459	+ */
	5460	+static void wq_adjust_max_active(struct workqueue_struct *wq)
	5461	+{
	5462	+ bool activated;
	5463	+ int new_max, new_min;
	5464	+
	5465	+ lockdep_assert_held(&wq->mutex);
	5466	+
	5467	+ if ((wq->flags & WQ_FREEZABLE) && workqueue_freezing) {
	5468	+ new_max = 0;
	5469	+ new_min = 0;
	5470	+ } else {
	5471	+ new_max = wq->saved_max_active;
	5472	+ new_min = wq->saved_min_active;
	5473	+ }
	5474	+
	5475	+ if (wq->max_active == new_max && wq->min_active == new_min)
	5476	+ return;
	5477	+
	5478	+ /*
	5479	+ * Update @wq->max/min_active and then kick inactive work items if more
	5480	+ * active work items are allowed. This doesn't break work item ordering
	5481	+ * because new work items are always queued behind existing inactive
	5482	+ * work items if there are any.
	5483	+ */
	5484	+ WRITE_ONCE(wq->max_active, new_max);
	5485	+ WRITE_ONCE(wq->min_active, new_min);
	5486	+
	5487	+ if (wq->flags & WQ_UNBOUND)
	5488	+ wq_update_node_max_active(wq, -1);
	5489	+
	5490	+ if (new_max == 0)
	5491	+ return;
	5492	+
	5493	+ /*
	5494	+ * Round-robin through pwq's activating the first inactive work item
	5495	+ * until max_active is filled.
	5496	+ */
	5497	+ do {
	5498	+ struct pool_workqueue *pwq;
	5499	+
	5500	+ activated = false;
	5501	+ for_each_pwq(pwq, wq) {
	5502	+ unsigned long irq_flags;
	5503	+
	5504	+ /* can be called during early boot w/ irq disabled */
	5505	+ raw_spin_lock_irqsave(&pwq->pool->lock, irq_flags);
	5506	+ if (pwq_activate_first_inactive(pwq, true)) {
	5507	+ activated = true;
	5508	+ kick_pool(pwq->pool);
	5509	+ }
	5510	+ raw_spin_unlock_irqrestore(&pwq->pool->lock, irq_flags);
	5511	+ }
	5512	+ } while (activated);
	5513	+}
	5514	+
4661	5515	__printf(1, 4)
4662	5516	struct workqueue_struct alloc_workqueue(const char fmt,
4663	5517	unsigned int flags,
@@ -4665,23 +5519,27 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 1 unmodified line
4665	5519	{
4666	5520	va_list args;
4667	5521	struct workqueue_struct *wq;
4668		− struct pool_workqueue *pwq;
	5522	+ size_t wq_size;
	5523	+ int name_len;
4669	5524
4670		− /*
4671		− * Unbound && max_active == 1 used to imply ordered, which is no longer
4672		− * the case on many machines due to per-pod pools. While
4673		− * alloc_ordered_workqueue() is the right way to create an ordered
4674		− * workqueue, keep the previous behavior to avoid subtle breakages.
4675		− */
4676		− if ((flags & WQ_UNBOUND) && max_active == 1)
4677		− flags \|= __WQ_ORDERED;
	5525	+ if (flags & WQ_BH) {
	5526	+ if (WARN_ON_ONCE(flags & ~__WQ_BH_ALLOWS))
	5527	+ return NULL;
	5528	+ if (WARN_ON_ONCE(max_active))
	5529	+ return NULL;
	5530	+ }
4678	5531
4679	5532	/* see the comment above the definition of WQ_POWER_EFFICIENT */
4680	5533	if ((flags & WQ_POWER_EFFICIENT) && wq_power_efficient)
4681	5534	flags \|= WQ_UNBOUND;
4682	5535
4683	5536	/* allocate wq and format name */
4684		− wq = kzalloc(sizeof(*wq), GFP_KERNEL);
	5537	+ if (flags & WQ_UNBOUND)
	5538	+ wq_size = struct_size(wq, node_nr_active, nr_node_ids + 1);
	5539	+ else
	5540	+ wq_size = sizeof(*wq);
	5541	+
	5542	+ wq = kzalloc(wq_size, GFP_KERNEL);
4685	5543	if (!wq)
4686	5544	return NULL;
4687	5545
@@ -4692,15 +5550,30 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 4 unmodified lines
4692	5550	}
4693	5551
4694	5552	va_start(args, max_active);
4695		− vsnprintf(wq->name, sizeof(wq->name), fmt, args);
	5553	+ name_len = vsnprintf(wq->name, sizeof(wq->name), fmt, args);
4696	5554	va_end(args);
4697	5555
4698		− max_active = max_active ?: WQ_DFL_ACTIVE;
4699		− max_active = wq_clamp_max_active(max_active, flags, wq->name);
	5556	+ if (name_len >= WQ_NAME_LEN)
	5557	+ pr_warn_once("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n",
	5558	+ wq->name);
	5559	+
	5560	+ if (flags & WQ_BH) {
	5561	+ /*
	5562	+ * BH workqueues always share a single execution context per CPU
	5563	+ * and don't impose any max_active limit.
	5564	+ */
	5565	+ max_active = INT_MAX;
	5566	+ } else {
	5567	+ max_active = max_active ?: WQ_DFL_ACTIVE;
	5568	+ max_active = wq_clamp_max_active(max_active, flags, wq->name);
	5569	+ }
4700	5570
4701	5571	/* init wq */
4702	5572	wq->flags = flags;
4703		− wq->saved_max_active = max_active;
	5573	+ wq->max_active = max_active;
	5574	+ wq->min_active = min(max_active, WQ_DFL_MIN_ACTIVE);
	5575	+ wq->saved_max_active = wq->max_active;
	5576	+ wq->saved_min_active = wq->min_active;
4704	5577	mutex_init(&wq->mutex);
4705	5578	atomic_set(&wq->nr_pwqs_to_flush, 0);
4706	5579	INIT_LIST_HEAD(&wq->pwqs);
@@ -4711,8 +5584,13 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 4 unmodified lines
4711	5584	wq_init_lockdep(wq);
4712	5585	INIT_LIST_HEAD(&wq->list);
4713	5586
	5587	+ if (flags & WQ_UNBOUND) {
	5588	+ if (alloc_node_nr_active(wq->node_nr_active) < 0)
	5589	+ goto err_unreg_lockdep;
	5590	+ }
	5591	+
4714	5592	if (alloc_and_link_pwqs(wq) < 0)
4715		− goto err_unreg_lockdep;
	5593	+ goto err_free_node_nr_active;
4716	5594
4717	5595	if (wq_online && init_rescuer(wq) < 0)
4718	5596	goto err_destroy;
@@ -4728,8 +5606,7 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 9 unmodified lines
4728	5606	mutex_lock(&wq_pool_mutex);
4729	5607
4730	5608	mutex_lock(&wq->mutex);
4731		− for_each_pwq(pwq, wq)
4732		− pwq_adjust_max_active(pwq);
	5609	+ wq_adjust_max_active(wq);
4733	5610	mutex_unlock(&wq->mutex);
4734	5611
4735	5612	list_add_tail_rcu(&wq->list, &workqueues);
@@ -4738,6 +5615,9 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 2 unmodified lines
4738	5615
4739	5616	return wq;
4740	5617
	5618	+err_free_node_nr_active:
	5619	+ if (wq->flags & WQ_UNBOUND)
	5620	+ free_node_nr_active(wq->node_nr_active);
4741	5621	err_unreg_lockdep:
4742	5622	wq_unregister_lockdep(wq);
4743	5623	wq_free_lockdep(wq);
@@ -4759,9 +5639,9 @@ static bool pwq_busy(struct pool_workqueue *pwq) · 15 unmodified lines
4759	5639	if (pwq->nr_in_flight[i])
4760	5640	return true;
4761	5641
4762		− if ((pwq != pwq->wq->dfl_pwq) && (pwq->refcnt > 1))
	5642	+ if ((pwq != rcu_access_pointer(pwq->wq->dfl_pwq)) && (pwq->refcnt > 1))
4763	5643	return true;
4764		− if (pwq->nr_active \|\| !list_empty(&pwq->inactive_works))
	5644	+ if (!pwq_is_empty(pwq))
4765	5645	return true;
4766	5646
4767	5647	return false;
@@ -4843,13 +5723,12 @@ void destroy_workqueue(struct workqueue_struct *wq) · 75 unmodified lines
4843	5723	rcu_read_lock();
4844	5724
4845	5725	for_each_possible_cpu(cpu) {
4846		− pwq = rcu_access_pointer(*per_cpu_ptr(wq->cpu_pwq, cpu));
4847		− RCU_INIT_POINTER(*per_cpu_ptr(wq->cpu_pwq, cpu), NULL);
4848		− put_pwq_unlocked(pwq);
	5726	+ put_pwq_unlocked(unbound_pwq(wq, cpu));
	5727	+ RCU_INIT_POINTER(*unbound_pwq_slot(wq, cpu), NULL);
4849	5728	}
4850	5729
4851		− put_pwq_unlocked(wq->dfl_pwq);
4852		− wq->dfl_pwq = NULL;
	5730	+ put_pwq_unlocked(unbound_pwq(wq, -1));
	5731	+ RCU_INIT_POINTER(*unbound_pwq_slot(wq, -1), NULL);
4853	5732
4854	5733	rcu_read_unlock();
4855	5734	}
@@ -4860,33 +5739,62 @@ EXPORT_SYMBOL_GPL(destroy_workqueue); · 4 unmodified lines
4860	5739	* @wq: target workqueue
4861	5740	* @max_active: new max_active value.
4862	5741	*
4863		− * Set max_active of @wq to @max_active.
	5742	+ * Set max_active of @wq to @max_active. See the alloc_workqueue() function
	5743	+ * comment.
4864	5744	*
4865	5745	* CONTEXT:
4866	5746	* Don't call from IRQ context.
4867	5747	*/
4868	5748	void workqueue_set_max_active(struct workqueue_struct *wq, int max_active)
4869	5749	{
4870		− struct pool_workqueue *pwq;
4871		−
	5750	+ /* max_active doesn't mean anything for BH workqueues */
	5751	+ if (WARN_ON(wq->flags & WQ_BH))
	5752	+ return;
4872	5753	/* disallow meddling with max_active for ordered workqueues */
4873		− if (WARN_ON(wq->flags & __WQ_ORDERED_EXPLICIT))
	5754	+ if (WARN_ON(wq->flags & __WQ_ORDERED))
4874	5755	return;
4875	5756
4876	5757	max_active = wq_clamp_max_active(max_active, wq->flags, wq->name);
4877	5758
4878	5759	mutex_lock(&wq->mutex);
4879	5760
4880		− wq->flags &= ~__WQ_ORDERED;
4881	5761	wq->saved_max_active = max_active;
	5762	+ if (wq->flags & WQ_UNBOUND)
	5763	+ wq->saved_min_active = min(wq->saved_min_active, max_active);
4882	5764
4883		− for_each_pwq(pwq, wq)
4884		− pwq_adjust_max_active(pwq);
	5765	+ wq_adjust_max_active(wq);
4885	5766
4886	5767	mutex_unlock(&wq->mutex);
4887	5768	}
4888	5769	EXPORT_SYMBOL_GPL(workqueue_set_max_active);
4889	5770
	5771	+/**
	5772	+ * workqueue_set_min_active - adjust min_active of an unbound workqueue
	5773	+ * @wq: target unbound workqueue
	5774	+ * @min_active: new min_active value
	5775	+ *
	5776	+ * Set min_active of an unbound workqueue. Unlike other types of workqueues, an
	5777	+ * unbound workqueue is not guaranteed to be able to process max_active
	5778	+ * interdependent work items. Instead, an unbound workqueue is guaranteed to be
	5779	+ * able to process min_active number of interdependent work items which is
	5780	+ * %WQ_DFL_MIN_ACTIVE by default.
	5781	+ *
	5782	+ * Use this function to adjust the min_active value between 0 and the current
	5783	+ * max_active.
	5784	+ */
	5785	+void workqueue_set_min_active(struct workqueue_struct *wq, int min_active)
	5786	+{
	5787	+ /* min_active is only meaningful for non-ordered unbound workqueues */
	5788	+ if (WARN_ON((wq->flags & (WQ_BH \| WQ_UNBOUND \| __WQ_ORDERED)) !=
	5789	+ WQ_UNBOUND))
	5790	+ return;
	5791	+
	5792	+ mutex_lock(&wq->mutex);
	5793	+ wq->saved_min_active = clamp(min_active, 0, wq->saved_max_active);
	5794	+ wq_adjust_max_active(wq);
	5795	+ mutex_unlock(&wq->mutex);
	5796	+}
	5797	+
4890	5798	/**
4891	5799	* current_work - retrieve %current task's work struct
4892	5800	*
@@ -4972,7 +5880,7 @@ EXPORT_SYMBOL_GPL(workqueue_congested); · 79 unmodified lines
4972	5880	unsigned int work_busy(struct work_struct *work)
4973	5881	{
4974	5882	struct worker_pool *pool;
4975		− unsigned long flags;
	5883	+ unsigned long irq_flags;
4976	5884	unsigned int ret = 0;
4977	5885
4978	5886	if (work_pending(work))
@@ -4981,10 +5889,10 @@ unsigned int work_busy(struct work_struct *work) · 2 unmodified lines
4981	5889	rcu_read_lock();
4982	5890	pool = get_work_pool(work);
4983	5891	if (pool) {
4984		− raw_spin_lock_irqsave(&pool->lock, flags);
	5892	+ raw_spin_lock_irqsave(&pool->lock, irq_flags);
4985	5893	if (find_worker_executing_work(pool, work))
4986	5894	ret \|= WORK_BUSY_RUNNING;
4987		− raw_spin_unlock_irqrestore(&pool->lock, flags);
	5895	+ raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
4988	5896	}
4989	5897	rcu_read_unlock();
4990	5898
@@ -5069,7 +5977,24 @@ static void pr_cont_pool_info(struct worker_pool *pool) · 78 unmodified lines
5069	5977	pr_cont(" cpus=%*pbl", nr_cpumask_bits, pool->attrs->cpumask);
5070	5978	if (pool->node != NUMA_NO_NODE)
5071	5979	pr_cont(" node=%d", pool->node);
5072		− pr_cont(" flags=0x%x nice=%d", pool->flags, pool->attrs->nice);
	5980	+ pr_cont(" flags=0x%x", pool->flags);
	5981	+ if (pool->flags & POOL_BH)
	5982	+ pr_cont(" bh%s",
	5983	+ pool->attrs->nice == HIGHPRI_NICE_LEVEL ? "-hi" : "");
	5984	+ else
	5985	+ pr_cont(" nice=%d", pool->attrs->nice);
	5986	+}
	5987	+
	5988	+static void pr_cont_worker_id(struct worker *worker)
	5989	+{
	5990	+ struct worker_pool *pool = worker->pool;
	5991	+
	5992	+ if (pool->flags & WQ_BH)
	5993	+ pr_cont("bh%s",
	5994	+ pool->attrs->nice == HIGHPRI_NICE_LEVEL ? "-hi" : "");
	5995	+ else
	5996	+ pr_cont("%d%s", task_pid_nr(worker->task),
	5997	+ worker->rescue_wq ? "(RESCUER)" : "");
5073	5998	}
5074	5999
5075	6000	struct pr_cont_work_struct {
@@ -5128,8 +6053,8 @@ static void show_pwq(struct pool_workqueue *pwq) · 52 unmodified lines
5128	6053	pr_info(" pwq %d:", pool->id);
5129	6054	pr_cont_pool_info(pool);
5130	6055
5131		− pr_cont(" active=%d/%d refcnt=%d%s\n",
5132		− pwq->nr_active, pwq->max_active, pwq->refcnt,
	6056	+ pr_cont(" active=%d refcnt=%d%s\n",
	6057	+ pwq->nr_active, pwq->refcnt,
5133	6058	!list_empty(&pwq->mayday_node) ? " MAYDAY" : "");
5134	6059
5135	6060	hash_for_each(pool->busy_hash, bkt, worker, hentry) {
@@ -5146,10 +6071,9 @@ static void show_pwq(struct pool_workqueue *pwq) · 10 unmodified lines
5146	6071	if (worker->current_pwq != pwq)
5147	6072	continue;
5148	6073
5149		− pr_cont("%s %d%s:%ps", comma ? "," : "",
5150		− task_pid_nr(worker->task),
5151		− worker->rescue_wq ? "(RESCUER)" : "",
5152		− worker->current_func);
	6074	+ pr_cont(" %s", comma ? "," : "");
	6075	+ pr_cont_worker_id(worker);
	6076	+ pr_cont(":%ps", worker->current_func);
5153	6077	list_for_each_entry(work, &worker->scheduled, entry)
5154	6078	pr_cont_work(false, work, &pcws);
5155	6079	pr_cont_work_flush(comma, (work_func_t)-1L, &pcws);
@@ -5200,10 +6124,10 @@ void show_one_workqueue(struct workqueue_struct *wq) · 44 unmodified lines
5200	6124	{
5201	6125	struct pool_workqueue *pwq;
5202	6126	bool idle = true;
5203		− unsigned long flags;
	6127	+ unsigned long irq_flags;
5204	6128
5205	6129	for_each_pwq(pwq, wq) {
5206		− if (pwq->nr_active \|\| !list_empty(&pwq->inactive_works)) {
	6130	+ if (!pwq_is_empty(pwq)) {
5207	6131	idle = false;
5208	6132	break;
5209	6133	}
@@ -5214,8 +6138,8 @@ void show_one_workqueue(struct workqueue_struct *wq) · 4 unmodified lines
5214	6138	pr_info("workqueue %s: flags=0x%x\n", wq->name, wq->flags);
5215	6139
5216	6140	for_each_pwq(pwq, wq) {
5217		− raw_spin_lock_irqsave(&pwq->pool->lock, flags);
5218		− if (pwq->nr_active \|\| !list_empty(&pwq->inactive_works)) {
	6141	+ raw_spin_lock_irqsave(&pwq->pool->lock, irq_flags);
	6142	+ if (!pwq_is_empty(pwq)) {
5219	6143	/*
5220	6144	* Defer printing to avoid deadlocks in console
5221	6145	* drivers that queue work while holding locks
@@ -5225,7 +6149,7 @@ void show_one_workqueue(struct workqueue_struct *wq) · 3 unmodified lines
5225	6149	show_pwq(pwq);
5226	6150	printk_deferred_exit();
5227	6151	}
5228		− raw_spin_unlock_irqrestore(&pwq->pool->lock, flags);
	6152	+ raw_spin_unlock_irqrestore(&pwq->pool->lock, irq_flags);
5229	6153	/*
5230	6154	* We could be printing a lot from atomic context, e.g.
5231	6155	* sysrq-t -> show_all_workqueues(). Avoid triggering
@@ -5244,10 +6168,10 @@ static void show_one_worker_pool(struct worker_pool *pool) · 12 unmodified lines
5244	6168	{
5245	6169	struct worker *worker;
5246	6170	bool first = true;
5247		− unsigned long flags;
	6171	+ unsigned long irq_flags;
5248	6172	unsigned long hung = 0;
5249	6173
5250		− raw_spin_lock_irqsave(&pool->lock, flags);
	6174	+ raw_spin_lock_irqsave(&pool->lock, irq_flags);
5251	6175	if (pool->nr_workers == pool->nr_idle)
5252	6176	goto next_pool;
5253	6177
@@ -5268,14 +6192,14 @@ static void show_one_worker_pool(struct worker_pool *pool) · 14 unmodified lines
5268	6192	pr_cont(" manager: %d",
5269	6193	task_pid_nr(pool->manager->task));
5270	6194	list_for_each_entry(worker, &pool->idle_list, entry) {
5271		− pr_cont(" %s%d", first ? "idle: " : "",
5272		− task_pid_nr(worker->task));
	6195	+ pr_cont(" %s", first ? "idle: " : "");
	6196	+ pr_cont_worker_id(worker);
5273	6197	first = false;
5274	6198	}
5275	6199	pr_cont("\n");
5276	6200	printk_deferred_exit();
5277	6201	next_pool:
5278		− raw_spin_unlock_irqrestore(&pool->lock, flags);
	6202	+ raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
5279	6203	/*
5280	6204	* We could be printing a lot from atomic context, e.g.
5281	6205	* sysrq-t -> show_all_workqueues(). Avoid triggering
@@ -5542,13 +6466,15 @@ int workqueue_online_cpu(unsigned int cpu) · 260 unmodified lines
5542	6466	mutex_lock(&wq_pool_mutex);
5543	6467
5544	6468	for_each_pool(pool, pi) {
5545		− mutex_lock(&wq_pool_attach_mutex);
	6469	+ /* BH pools aren't affected by hotplug */
	6470	+ if (pool->flags & POOL_BH)
	6471	+ continue;
5546	6472
	6473	+ mutex_lock(&wq_pool_attach_mutex);
5547	6474	if (pool->cpu == cpu)
5548	6475	rebind_workers(pool);
5549	6476	else if (pool->cpu < 0)
5550	6477	restore_unbound_workers_cpumask(pool, cpu);
5551		−
5552	6478	mutex_unlock(&wq_pool_attach_mutex);
5553	6479	}
5554	6480
@@ -5562,6 +6488,10 @@ int workqueue_online_cpu(unsigned int cpu) · 7 unmodified lines
5562	6488
5563	6489	for_each_cpu(tcpu, pt->pod_cpus[pt->cpu_pod[cpu]])
5564	6490	wq_update_pod(wq, tcpu, cpu, true);
	6491	+
	6492	+ mutex_lock(&wq->mutex);
	6493	+ wq_update_node_max_active(wq, -1);
	6494	+ mutex_unlock(&wq->mutex);
5565	6495	}
5566	6496	}
5567	6497
@@ -5590,6 +6520,10 @@ int workqueue_offline_cpu(unsigned int cpu) · 22 unmodified lines
5590	6520
5591	6521	for_each_cpu(tcpu, pt->pod_cpus[pt->cpu_pod[cpu]])
5592	6522	wq_update_pod(wq, tcpu, cpu, false);
	6523	+
	6524	+ mutex_lock(&wq->mutex);
	6525	+ wq_update_node_max_active(wq, cpu);
	6526	+ mutex_unlock(&wq->mutex);
5593	6527	}
5594	6528	}
5595	6529	mutex_unlock(&wq_pool_mutex);
@@ -5677,7 +6611,6 @@ EXPORT_SYMBOL_GPL(work_on_cpu_safe_key); · 81 unmodified lines
5677	6611	void freeze_workqueues_begin(void)
5678	6612	{
5679	6613	struct workqueue_struct *wq;
5680		− struct pool_workqueue *pwq;
5681	6614
5682	6615	mutex_lock(&wq_pool_mutex);
5683	6616
@@ -5686,8 +6619,7 @@ void freeze_workqueues_begin(void) · 2 unmodified lines
5686	6619
5687	6620	list_for_each_entry(wq, &workqueues, list) {
5688	6621	mutex_lock(&wq->mutex);
5689		− for_each_pwq(pwq, wq)
5690		− pwq_adjust_max_active(pwq);
	6622	+ wq_adjust_max_active(wq);
5691	6623	mutex_unlock(&wq->mutex);
5692	6624	}
5693	6625
@@ -5752,7 +6684,6 @@ out_unlock: · 58 unmodified lines
5752	6684	void thaw_workqueues(void)
5753	6685	{
5754	6686	struct workqueue_struct *wq;
5755		− struct pool_workqueue *pwq;
5756	6687
5757	6688	mutex_lock(&wq_pool_mutex);
5758	6689
@@ -5764,8 +6695,7 @@ void thaw_workqueues(void) · 5 unmodified lines
5764	6695	/* restore max_active and repopulate worklist */
5765	6696	list_for_each_entry(wq, &workqueues, list) {
5766	6697	mutex_lock(&wq->mutex);
5767		− for_each_pwq(pwq, wq)
5768		− pwq_adjust_max_active(pwq);
	6698	+ wq_adjust_max_active(wq);
5769	6699	mutex_unlock(&wq->mutex);
5770	6700	}
5771	6701
@@ -5784,10 +6714,7 @@ static int workqueue_apply_unbound_cpumask(const cpumask_var_t unbound_cpumask) · 12 unmodified lines
5784	6714	lockdep_assert_held(&wq_pool_mutex);
5785	6715
5786	6716	list_for_each_entry(wq, &workqueues, list) {
5787		− if (!(wq->flags & WQ_UNBOUND))
5788		− continue;
5789		− /* creating multiple pwqs breaks ordering guarantee */
5790		− if (wq->flags & __WQ_ORDERED)
	6717	+ if (!(wq->flags & WQ_UNBOUND) \|\| (wq->flags & __WQ_DESTROYING))
5791	6718	continue;
5792	6719
5793	6720	ctx = apply_wqattrs_prepare(wq, wq->unbound_attrs, unbound_cpumask);
@@ -6303,11 +7230,10 @@ int workqueue_sysfs_register(struct workqueue_struct *wq) · 509 unmodified lines
6303	7230	int ret;
6304	7231
6305	7232	/*
6306		− * Adjusting max_active or creating new pwqs by applying
6307		− * attributes breaks ordering guarantee. Disallow exposing ordered
6308		− * workqueues.
	7233	+ * Adjusting max_active breaks ordering guarantee. Disallow exposing
	7234	+ * ordered workqueues.
6309	7235	*/
6310		− if (WARN_ON(wq->flags & __WQ_ORDERED_EXPLICIT))
	7236	+ if (WARN_ON(wq->flags & __WQ_ORDERED))
6311	7237	return -EINVAL;
6312	7238
6313	7239	wq->wq_dev = wq_dev = kzalloc(sizeof(*wq_dev), GFP_KERNEL);
@@ -6404,10 +7330,10 @@ static DEFINE_PER_CPU(unsigned long, wq_watchdog_touched_cpu) = INITIAL_JIFFIES; · 90 unmodified lines
6404	7330	static void show_cpu_pool_hog(struct worker_pool *pool)
6405	7331	{
6406	7332	struct worker *worker;
6407		− unsigned long flags;
	7333	+ unsigned long irq_flags;
6408	7334	int bkt;
6409	7335
6410		− raw_spin_lock_irqsave(&pool->lock, flags);
	7336	+ raw_spin_lock_irqsave(&pool->lock, irq_flags);
6411	7337
6412	7338	hash_for_each(pool->busy_hash, bkt, worker, hentry) {
6413	7339	if (task_is_running(worker->task)) {
@@ -6425,7 +7351,7 @@ static void show_cpu_pool_hog(struct worker_pool *pool) · 11 unmodified lines
6425	7351	}
6426	7352	}
6427	7353
6428		− raw_spin_unlock_irqrestore(&pool->lock, flags);
	7354	+ raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
6429	7355	}
6430	7356
6431	7357	static void show_cpu_pools_hogs(void)
@@ -6497,7 +7423,7 @@ static void wq_watchdog_timer_fn(struct timer_list *unused) · 65 unmodified lines
6497	7423	/* did we stall? */
6498	7424	if (time_after(now, ts + thresh)) {
6499	7425	lockup_detected = true;
6500		− if (pool->cpu >= 0) {
	7426	+ if (pool->cpu >= 0 && !(pool->flags & POOL_BH)) {
6501	7427	pool->cpu_stall = true;
6502	7428	cpu_pool_stall = true;
6503	7429	}
@@ -6580,6 +7506,16 @@ static inline void wq_watchdog_init(void) { } · 76 unmodified lines
6580	7506
6581	7507	#endif /* CONFIG_WQ_WATCHDOG */
6582	7508
	7509	+static void bh_pool_kick_normal(struct irq_work *irq_work)
	7510	+{
	7511	+ raise_softirq_irqoff(TASKLET_SOFTIRQ);
	7512	+}
	7513	+
	7514	+static void bh_pool_kick_highpri(struct irq_work *irq_work)
	7515	+{
	7516	+ raise_softirq_irqoff(HI_SOFTIRQ);
	7517	+}
	7518	+
6583	7519	static void __init restrict_unbound_cpumask(const char name, const struct cpumask mask)
6584	7520	{
6585	7521	if (!cpumask_intersects(wq_unbound_cpumask, mask)) {
@@ -6591,6 +7527,22 @@ static void __init restrict_unbound_cpumask(const char *name, const struct cpuma · 5 unmodified lines
6591	7527	cpumask_and(wq_unbound_cpumask, wq_unbound_cpumask, mask);
6592	7528	}
6593	7529
	7530	+static void __init init_cpu_worker_pool(struct worker_pool *pool, int cpu, int nice)
	7531	+{
	7532	+ BUG_ON(init_worker_pool(pool));
	7533	+ pool->cpu = cpu;
	7534	+ cpumask_copy(pool->attrs->cpumask, cpumask_of(cpu));
	7535	+ cpumask_copy(pool->attrs->__pod_cpumask, cpumask_of(cpu));
	7536	+ pool->attrs->nice = nice;
	7537	+ pool->attrs->affn_strict = true;
	7538	+ pool->node = cpu_to_node(cpu);
	7539	+
	7540	+ /* alloc pool ID */
	7541	+ mutex_lock(&wq_pool_mutex);
	7542	+ BUG_ON(worker_pool_assign_id(pool));
	7543	+ mutex_unlock(&wq_pool_mutex);
	7544	+}
	7545	+
6594	7546	/**
6595	7547	* workqueue_init_early - early init for workqueue subsystem
6596	7548	*
@@ -6605,6 +7557,8 @@ void __init workqueue_init_early(void) · 8 unmodified lines
6605	7557	{
6606	7558	struct wq_pod_type *pt = &wq_pod_types[WQ_AFFN_SYSTEM];
6607	7559	int std_nice[NR_STD_WORKER_POOLS] = { 0, HIGHPRI_NICE_LEVEL };
	7560	+ void (irq_work_fns[2])(struct irq_work ) = { bh_pool_kick_normal,
	7561	+ bh_pool_kick_highpri };
6608	7562	int i, cpu;
6609	7563
6610	7564	BUILD_BUG_ON(__alignof__(struct pool_workqueue) < __alignof__(long long));
@@ -6626,6 +7580,13 @@ void __init workqueue_init_early(void) · 15 unmodified lines
6626	7580	wq_update_pod_attrs_buf = alloc_workqueue_attrs();
6627	7581	BUG_ON(!wq_update_pod_attrs_buf);
6628	7582
	7583	+ /*
	7584	+ * If nohz_full is enabled, set power efficient workqueue as unbound.
	7585	+ * This allows workqueue items to be moved to HK CPUs.
	7586	+ */
	7587	+ if (housekeeping_enabled(HK_TYPE_TICK))
	7588	+ wq_power_efficient = true;
	7589	+
6629	7590	/* initialize WQ_AFFN_SYSTEM pods */
6630	7591	pt->pod_cpus = kcalloc(1, sizeof(pt->pod_cpus[0]), GFP_KERNEL);
6631	7592	pt->pod_node = kcalloc(1, sizeof(pt->pod_node[0]), GFP_KERNEL);
@@ -6639,25 +7600,21 @@ void __init workqueue_init_early(void) · 7 unmodified lines
6639	7600	pt->pod_node[0] = NUMA_NO_NODE;
6640	7601	pt->cpu_pod[0] = 0;
6641	7602
6642		− /* initialize CPU pools */
	7603	+ /* initialize BH and CPU pools */
6643	7604	for_each_possible_cpu(cpu) {
6644	7605	struct worker_pool *pool;
6645	7606
6646	7607	i = 0;
6647		− for_each_cpu_worker_pool(pool, cpu) {
6648		− BUG_ON(init_worker_pool(pool));
6649		− pool->cpu = cpu;
6650		− cpumask_copy(pool->attrs->cpumask, cpumask_of(cpu));
6651		− cpumask_copy(pool->attrs->__pod_cpumask, cpumask_of(cpu));
6652		− pool->attrs->nice = std_nice[i++];
6653		− pool->attrs->affn_strict = true;
6654		− pool->node = cpu_to_node(cpu);
6655		−
6656		− /* alloc pool ID */
6657		− mutex_lock(&wq_pool_mutex);
6658		− BUG_ON(worker_pool_assign_id(pool));
6659		− mutex_unlock(&wq_pool_mutex);
	7608	+ for_each_bh_worker_pool(pool, cpu) {
	7609	+ init_cpu_worker_pool(pool, cpu, std_nice[i]);
	7610	+ pool->flags \|= POOL_BH;
	7611	+ init_irq_work(bh_pool_irq_work(pool), irq_work_fns[i]);
	7612	+ i++;
6660	7613	}
	7614	+
	7615	+ i = 0;
	7616	+ for_each_cpu_worker_pool(pool, cpu)
	7617	+ init_cpu_worker_pool(pool, cpu, std_nice[i++]);
6661	7618	}
6662	7619
6663	7620	/* create default unbound and ordered wq attrs */
@@ -6687,13 +7644,17 @@ void __init workqueue_init_early(void) · 23 unmodified lines
6687	7644	WQ_FREEZABLE, 0);
6688	7645	system_power_efficient_wq = alloc_workqueue("events_power_efficient",
6689	7646	WQ_POWER_EFFICIENT, 0);
6690		− system_freezable_power_efficient_wq = alloc_workqueue("events_freezable_power_efficient",
	7647	+ system_freezable_power_efficient_wq = alloc_workqueue("events_freezable_pwr_efficient",
6691	7648	WQ_FREEZABLE \| WQ_POWER_EFFICIENT,
6692	7649	0);
	7650	+ system_bh_wq = alloc_workqueue("events_bh", WQ_BH, 0);
	7651	+ system_bh_highpri_wq = alloc_workqueue("events_bh_highpri",
	7652	+ WQ_BH \| WQ_HIGHPRI, 0);
6693	7653	BUG_ON(!system_wq \|\| !system_highpri_wq \|\| !system_long_wq \|\|
6694	7654	!system_unbound_wq \|\| !system_freezable_wq \|\|
6695	7655	!system_power_efficient_wq \|\|
6696		− !system_freezable_power_efficient_wq);
	7656	+ !system_freezable_power_efficient_wq \|\|
	7657	+ !system_bh_wq \|\| !system_bh_highpri_wq);
6697	7658	}
6698	7659
6699	7660	static void __init wq_cpu_intensive_thresh_init(void)
@@ -6759,9 +7720,10 @@ void __init workqueue_init(void) · 59 unmodified lines
6759	7720	* up. Also, create a rescuer for workqueues that requested it.
6760	7721	*/
6761	7722	for_each_possible_cpu(cpu) {
6762		− for_each_cpu_worker_pool(pool, cpu) {
	7723	+ for_each_bh_worker_pool(pool, cpu)
	7724	+ pool->node = cpu_to_node(cpu);
	7725	+ for_each_cpu_worker_pool(pool, cpu)
6763	7726	pool->node = cpu_to_node(cpu);
6764		− }
6765	7727	}
6766	7728
6767	7729	list_for_each_entry(wq, &workqueues, list) {
@@ -6772,7 +7734,16 @@ void __init workqueue_init(void) · 4 unmodified lines
6772	7734
6773	7735	mutex_unlock(&wq_pool_mutex);
6774	7736
6775		− /* create the initial workers */
	7737	+ /*
	7738	+ * Create the initial workers. A BH pool has one pseudo worker that
	7739	+ * represents the shared BH execution context and thus doesn't get
	7740	+ * affected by hotplug events. Create the BH pseudo workers for all
	7741	+ * possible CPUs here.
	7742	+ */
	7743	+ for_each_possible_cpu(cpu)
	7744	+ for_each_bh_worker_pool(pool, cpu)
	7745	+ BUG_ON(!create_worker(pool));
	7746	+
6776	7747	for_each_online_cpu(cpu) {
6777	7748	for_each_cpu_worker_pool(pool, cpu) {
6778	7749	pool->flags &= ~POOL_DISASSOCIATED;
@@ -6852,7 +7823,7 @@ static bool __init cpus_share_numa(int cpu0, int cpu1) · 73 unmodified lines
6852	7823	/**
6853	7824	* workqueue_init_topology - initialize CPU pods for unbound workqueues
6854	7825	*
6855		− * This is the third step of there-staged workqueue subsystem initialization and
	7826	+ * This is the third step of three-staged workqueue subsystem initialization and
6856	7827	* invoked after SMP and topology information are fully initialized. It
6857	7828	* initializes the unbound CPU pods accordingly.
6858	7829	*/
@@ -6866,6 +7837,8 @@ void __init workqueue_init_topology(void) · 7 unmodified lines
6866	7837	init_pod_type(&wq_pod_types[WQ_AFFN_CACHE], cpus_share_cache);
6867	7838	init_pod_type(&wq_pod_types[WQ_AFFN_NUMA], cpus_share_numa);
6868	7839
	7840	+ wq_topo_initialized = true;
	7841	+
6869	7842	mutex_lock(&wq_pool_mutex);
6870	7843
6871	7844	/*
@@ -6874,8 +7847,12 @@ void __init workqueue_init_topology(void) · 2 unmodified lines
6874	7847	* combinations to apply per-pod sharing.
6875	7848	*/
6876	7849	list_for_each_entry(wq, &workqueues, list) {
6877		− for_each_online_cpu(cpu) {
	7850	+ for_each_online_cpu(cpu)
6878	7851	wq_update_pod(wq, cpu, cpu, true);
	7852	+ if (wq->flags & WQ_UNBOUND) {
	7853	+ mutex_lock(&wq->mutex);
	7854	+ wq_update_node_max_active(wq, -1);
	7855	+ mutex_unlock(&wq->mutex);
6879	7856	}
6880	7857	}
6881	7858

rust/kernel/workqueue.rs+5−1

@@ -198,7 +198,11 @@ impl Queue { · 197 unmodified lines
198	198	// stay valid until we call the function pointer in the `work_struct`, so the access is ok.
199	199	unsafe {
200	200	w.__enqueue(move \|work_ptr\| {
201		− bindings::queue_work_on(bindings::WORK_CPU_UNBOUND as _, queue_ptr, work_ptr)
	201	+ bindings::queue_work_on(
	202	+ bindings::wq_misc_consts_WORK_CPU_UNBOUND as _,
	203	+ queue_ptr,
	204	+ work_ptr,
	205	+ )
202	206	})
203	207	}
204	208	}

tools/workqueue/wq_dump.py+90−14

@@ -50,6 +50,7 @@ import drgn · 49 unmodified lines
50	50	from drgn.helpers.linux.list import list_for_each_entry,list_empty
51	51	from drgn.helpers.linux.percpu import per_cpu_ptr
52	52	from drgn.helpers.linux.cpumask import for_each_cpu,for_each_possible_cpu
	53	+from drgn.helpers.linux.nodemask import for_each_node
53	54	from drgn.helpers.linux.idr import idr_for_each
54	55
55	56	import argparse
@@ -75,6 +76,22 @@ def cpumask_str(cpumask): · 19 unmodified lines
75	76	output += f'{v:08x}'
76	77	return output.strip()
77	78
	79	+wq_type_len = 9
	80	+
	81	+def wq_type_str(wq):
	82	+ if wq.flags & WQ_BH:
	83	+ return f'{"bh":{wq_type_len}}'
	84	+ elif wq.flags & WQ_UNBOUND:
	85	+ if wq.flags & WQ_ORDERED:
	86	+ return f'{"ordered":{wq_type_len}}'
	87	+ else:
	88	+ if wq.unbound_attrs.affn_strict:
	89	+ return f'{"unbound,S":{wq_type_len}}'
	90	+ else:
	91	+ return f'{"unbound":{wq_type_len}}'
	92	+ else:
	93	+ return f'{"percpu":{wq_type_len}}'
	94	+
78	95	worker_pool_idr = prog['worker_pool_idr']
79	96	workqueues = prog['workqueues']
80	97	wq_unbound_cpumask = prog['wq_unbound_cpumask']
@@ -82,6 +99,7 @@ wq_pod_types = prog['wq_pod_types'] · 1 unmodified line
82	99	wq_affn_dfl = prog['wq_affn_dfl']
83	100	wq_affn_names = prog['wq_affn_names']
84	101
	102	+WQ_BH = prog['WQ_BH']
85	103	WQ_UNBOUND = prog['WQ_UNBOUND']
86	104	WQ_ORDERED = prog['__WQ_ORDERED']
87	105	WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM']
@@ -92,6 +110,11 @@ WQ_AFFN_CACHE = prog['WQ_AFFN_CACHE'] · 4 unmodified lines
92	110	WQ_AFFN_NUMA = prog['WQ_AFFN_NUMA']
93	111	WQ_AFFN_SYSTEM = prog['WQ_AFFN_SYSTEM']
94	112
	113	+POOL_BH = prog['POOL_BH']
	114	+
	115	+WQ_NAME_LEN = prog['WQ_NAME_LEN'].value_()
	116	+cpumask_str_len = len(cpumask_str(wq_unbound_cpumask))
	117	+
95	118	print('Affinity Scopes')
96	119	print('===============')
97	120
@@ -133,10 +156,12 @@ for pi, pool in idr_for_each(worker_pool_idr): · 35 unmodified lines
133	156
134	157	for pi, pool in idr_for_each(worker_pool_idr):
135	158	pool = drgn.Object(prog, 'struct worker_pool', address=pool)
136		− print(f'pool[{pi:0{max_pool_id_len}}] ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
	159	+ print(f'pool[{pi:0{max_pool_id_len}}] flags=0x{pool.flags.value_():02x} ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
137	160	print(f'idle/workers={pool.nr_idle.value_():3}/{pool.nr_workers.value_():3} ', end='')
138	161	if pool.cpu >= 0:
139	162	print(f'cpu={pool.cpu.value_():3}', end='')
	163	+ if pool.flags & POOL_BH:
	164	+ print(' bh', end='')
140	165	else:
141	166	print(f'cpus={cpumask_str(pool.attrs.cpumask)}', end='')
142	167	print(f' pod_cpus={cpumask_str(pool.attrs.__pod_cpumask)}', end='')
@@ -148,24 +173,13 @@ print('') · 5 unmodified lines
148	173	print('Workqueue CPU -> pool')
149	174	print('=====================')
150	175
151		−print('[ workqueue \ type CPU', end='')
	176	+print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ {"type CPU":{wq_type_len}}', end='')
152	177	for cpu in for_each_possible_cpu(prog):
153	178	print(f' {cpu:{max_pool_id_len}}', end='')
154	179	print(' dfl]')
155	180
156	181	for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
157		− print(f'{wq.name.string_().decode()[-24:]:24}', end='')
158		− if wq.flags & WQ_UNBOUND:
159		− if wq.flags & WQ_ORDERED:
160		− print(' ordered ', end='')
161		− else:
162		− print(' unbound', end='')
163		− if wq.unbound_attrs.affn_strict:
164		− print(',S ', end='')
165		− else:
166		− print(' ', end='')
167		− else:
168		− print(' percpu ', end='')
	182	+ print(f'{wq.name.string_().decode():{WQ_NAME_LEN}} {wq_type_str(wq):10}', end='')
169	183
170	184	for cpu in for_each_possible_cpu(prog):
171	185	pool_id = per_cpu_ptr(wq.cpu_pwq, cpu)[0].pool.id.value_()
@@ -175,3 +189,65 @@ for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_( · 3 unmodified lines
175	189	if wq.flags & WQ_UNBOUND:
176	190	print(f' {wq.dfl_pwq.pool.id.value_():{max_pool_id_len}}', end='')
177	191	print('')
	192	+
	193	+print('')
	194	+print('Workqueue -> rescuer')
	195	+print('====================')
	196	+
	197	+ucpus_len = max(cpumask_str_len, len("unbound_cpus"))
	198	+rcpus_len = max(cpumask_str_len, len("rescuer_cpus"))
	199	+
	200	+print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ {"unbound_cpus":{ucpus_len}} pid {"rescuer_cpus":{rcpus_len}} ]')
	201	+
	202	+for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
	203	+ if not (wq.flags & WQ_MEM_RECLAIM):
	204	+ continue
	205	+
	206	+ print(f'{wq.name.string_().decode():{WQ_NAME_LEN}}', end='')
	207	+ if wq.unbound_attrs.value_() != 0:
	208	+ print(f' {cpumask_str(wq.unbound_attrs.cpumask):{ucpus_len}}', end='')
	209	+ else:
	210	+ print(f' {"":{ucpus_len}}', end='')
	211	+
	212	+ print(f' {wq.rescuer.task.pid.value_():6}', end='')
	213	+ print(f' {cpumask_str(wq.rescuer.task.cpus_ptr):{rcpus_len}}', end='')
	214	+ print('')
	215	+
	216	+print('')
	217	+print('Unbound workqueue -> node_nr/max_active')
	218	+print('=======================================')
	219	+
	220	+if 'node_to_cpumask_map' in prog:
	221	+ __cpu_online_mask = prog['__cpu_online_mask']
	222	+ node_to_cpumask_map = prog['node_to_cpumask_map']
	223	+ nr_node_ids = prog['nr_node_ids'].value_()
	224	+
	225	+ print(f'online_cpus={cpumask_str(__cpu_online_mask.address_of_())}')
	226	+ for node in for_each_node():
	227	+ print(f'NODE[{node:02}]={cpumask_str(node_to_cpumask_map[node])}')
	228	+ print('')
	229	+
	230	+ print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ min max', end='')
	231	+ first = True
	232	+ for node in for_each_node():
	233	+ if first:
	234	+ print(f' NODE {node}', end='')
	235	+ first = False
	236	+ else:
	237	+ print(f' {node:7}', end='')
	238	+ print(f' {"dfl":>7} ]')
	239	+ print('')
	240	+
	241	+ for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
	242	+ if not (wq.flags & WQ_UNBOUND):
	243	+ continue
	244	+
	245	+ print(f'{wq.name.string_().decode():{WQ_NAME_LEN}} ', end='')
	246	+ print(f'{wq.min_active.value_():3} {wq.max_active.value_():3}', end='')
	247	+ for node in for_each_node():
	248	+ nna = wq.node_nr_active[node]
	249	+ print(f' {nna.nr.counter.value_():3}/{nna.max.value_():3}', end='')
	250	+ nna = wq.node_nr_active[nr_node_ids]
	251	+ print(f' {nna.nr.counter.value_():3}/{nna.max.value_():3}')
	252	+else:
	253	+ printf(f'node_to_cpumask_map not present, is NUMA enabled?')