Merge tag 'wq-for-6.9' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq

Pull workqueue updates from Tejun Heo: "This cycle, a lot of workqueue changes including some that are significant and invasive. - During v6.6 cycle, unbound workqueues were updated so that they are more topology aware and flexible, which among other things improved workqueue behavior on modern multi-L3 CPUs. In the process, commit 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu pool_workqueues") switched unbound workqueues to use per-CPU frontend pool_workqueues as a part of increasing front-back mapping flexibility. An unwelcome side effect of this change was that this made max concurrency enforcement per-CPU blowing up the maximum number of allowed concurrent executions. I incorrectly assumed that this wouldn't cause practical problems as most unbound workqueue users are self-regulate max concurrency; however, there definitely are which don't (e.g. on IO paths) and the drastic increase in the allowed max concurrency led to noticeable perf regressions in some use cases. This is now addressed by separating out max concurrency enforcement to a separate struct - wq_node_nr_active - which makes @max_active consistently mean system-wide max concurrency regardless of the number of CPUs or (finally) NUMA nodes. This is a rather invasive and, in places, a bit clunky; however, the clunkiness rises from the the inherent requirement to handle the disagreement between the execution locality domain and max concurrency enforcement domain on some modern machines. See commit 5797b1c18919 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues") for more details. - BH workqueue support is added. They are similar to per-CPU workqueues but execute work items in the softirq context. This is expected to replace tasklet. However, currently, it's missing the ability to disable and enable work items which is needed to convert many tasklet users. To avoid crowding this merge window too much, this will be included in the next merge window. A separate pull request will be sent for the couple conversion patches that are currently pending. - Waiman plugged a long-standing hole in workqueue CPU isolation where ordered workqueues didn't follow wq_unbound_cpumask updates. Ordered workqueues now follow the same rules as other unbound workqueues. - More CPU isolation improvements: Juri fixed another deficit in workqueue isolation where unbound rescuers don't respect wq_unbound_cpumask. Leonardo fixed delayed_work timers firing on isolated CPUs. - Other misc changes" * tag 'wq-for-6.9' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq: (54 commits) workqueue: Drain BH work items on hot-unplugged CPUs workqueue: Introduce from_work() helper for cleaner callback declarations workqueue: Control intensive warning threshold through cmdline workqueue: Make @flags handling consistent across set_work_data() and friends workqueue: Remove clear_work_data() workqueue: Factor out work_grab_pending() from __cancel_work_sync() workqueue: Clean up enum work_bits and related constants workqueue: Introduce work_cancel_flags workqueue: Use variable name irq_flags for saving local irq flags workqueue: Reorganize flush and cancel[_sync] functions workqueue: Rename __cancel_work_timer() to __cancel_timer_sync() workqueue: Use rcu_read_lock_any_held() instead of rcu_read_lock_held() workqueue: Cosmetic changes workqueue, irq_work: Build fix for !CONFIG_IRQ_WORK workqueue: Fix queue_work_on() with BH workqueues async: Use a dedicated unbound workqueue with raised min_active workqueue: Implement workqueue_set_min_active() workqueue: Fix kernel-doc comment of unplug_oldest_pwq() workqueue: Bind unbound workqueue rescuer to wq_unbound_cpumask kernel/workqueue: Let rescuers follow unbound wq cpumask changes ...

Linus Torvalds committed 2 years ago

Commit ff887eb2 parents 8ede842 1acd92d11 files · +1690 −530Browse files

Unified · Split

11 files

Documentation/admin-guide/kernel-parameters.txt +9 −0
Documentation/core-api/workqueue.rst +29 −14
include/linux/async.h +1 −0
include/linux/workqueue.h +99 −42
init/Kconfig +1 −1
init/main.c +1 −0
kernel/async.c +16 −1
kernel/softirq.c +5 −0
kernel/workqueue.c +1434 −457
rust/kernel/workqueue.rs +5 −1
tools/workqueue/wq_dump.py +90 −14

Documentation/admin-guide/kernel-parameters.txt+9−0

@@ -7244,6 +7244,15 @@ · 7243 unmodified lines
7244	threshold repeatedly. They are likely good	7244	threshold repeatedly. They are likely good
7245	candidates for using WQ_UNBOUND workqueues instead.	7245	candidates for using WQ_UNBOUND workqueues instead.
7246		7246
		7247	workqueue.cpu_intensive_warning_thresh=<uint>
		7248	If CONFIG_WQ_CPU_INTENSIVE_REPORT is set, the kernel
		7249	will report the work functions which violate the
		7250	intensive_threshold_us repeatedly. In order to prevent
		7251	spurious warnings, start printing only after a work
		7252	function has violated this threshold number of times.
		7253
		7254	The default is 4 times. 0 disables the warning.
		7255
7247	workqueue.power_efficient	7256	workqueue.power_efficient
7248	Per-cpu workqueues are generally preferred because	7257	Per-cpu workqueues are generally preferred because
7249	they show better performance thanks to cache	7258	they show better performance thanks to cache

Documentation/core-api/workqueue.rst+29−14

@@ -77,10 +77,12 @@ wants a function to be executed asynchronously it has to set up a work · 76 unmodified lines
77	item pointing to that function and queue that work item on a	77	item pointing to that function and queue that work item on a
78	workqueue.	78	workqueue.
79		79
80	Special purpose threads, called worker threads, execute the functions	80	A work item can be executed in either a thread or the BH (softirq) context.
81	off of the queue, one after the other. If no work is queued, the	81
82	worker threads become idle. These worker threads are managed in so	82	For threaded workqueues, special purpose threads, called [k]workers, execute
83	called worker-pools.	83	the functions off of the queue, one after the other. If no work is queued,
		84	the worker threads become idle. These worker threads are managed in
		85	worker-pools.
84		86
85	The cmwq design differentiates between the user-facing workqueues that	87	The cmwq design differentiates between the user-facing workqueues that
86	subsystems and drivers queue work items on and the backend mechanism	88	subsystems and drivers queue work items on and the backend mechanism
@@ -91,6 +93,12 @@ for high priority ones, for each possible CPU and some extra · 4 unmodified lines
91	worker-pools to serve work items queued on unbound workqueues - the	93	worker-pools to serve work items queued on unbound workqueues - the
92	number of these backing pools is dynamic.	94	number of these backing pools is dynamic.
93		95
		96	BH workqueues use the same framework. However, as there can only be one
		97	concurrent execution context, there's no need to worry about concurrency.
		98	Each per-CPU BH worker pool contains only one pseudo worker which represents
		99	the BH execution context. A BH workqueue can be considered a convenience
		100	interface to softirq.
		101
94	Subsystems and drivers can create and queue work items through special	102	Subsystems and drivers can create and queue work items through special
95	workqueue API functions as they see fit. They can influence some	103	workqueue API functions as they see fit. They can influence some
96	aspects of the way the work items are executed by setting flags on the	104	aspects of the way the work items are executed by setting flags on the
@@ -106,7 +114,7 @@ unless specifically overridden, a work item of a bound workqueue will · 9 unmodified lines
106	be queued on the worklist of either normal or highpri worker-pool that	114	be queued on the worklist of either normal or highpri worker-pool that
107	is associated to the CPU the issuer is running on.	115	is associated to the CPU the issuer is running on.
108		116
109	For any worker pool implementation, managing the concurrency level	117	For any thread pool implementation, managing the concurrency level
110	(how many execution contexts are active) is an important issue. cmwq	118	(how many execution contexts are active) is an important issue. cmwq
111	tries to keep the concurrency at a minimal but sufficient level.	119	tries to keep the concurrency at a minimal but sufficient level.
112	Minimal to save resources and sufficient in that the system is used at	120	Minimal to save resources and sufficient in that the system is used at
@@ -164,6 +172,17 @@ resources, scheduled and executed. · 51 unmodified lines
164	``flags``	172	``flags``
165	---------	173	---------
166		174
		175	``WQ_BH``
		176	BH workqueues can be considered a convenience interface to softirq. BH
		177	workqueues are always per-CPU and all BH work items are executed in the
		178	queueing CPU's softirq context in the queueing order.
		179
		180	All BH workqueues must have 0 ``max_active`` and ``WQ_HIGHPRI`` is the
		181	only allowed additional flag.
		182
		183	BH work items cannot sleep. All other features such as delayed queueing,
		184	flushing and canceling are supported.
		185
167	``WQ_UNBOUND``	186	``WQ_UNBOUND``
168	Work items queued to an unbound wq are served by the special	187	Work items queued to an unbound wq are served by the special
169	worker-pools which host workers which are not bound to any	188	worker-pools which host workers which are not bound to any
@@ -237,15 +256,11 @@ may queue at the same time. Unless there is a specific need for · 67 unmodified lines
237	throttling the number of active work items, specifying '0' is	256	throttling the number of active work items, specifying '0' is
238	recommended.	257	recommended.
239		258
240	Some users depend on the strict execution ordering of ST wq. The	259	Some users depend on strict execution ordering where only one work item
241	combination of ``@max_active`` of 1 and ``WQ_UNBOUND`` used to	260	is in flight at any given time and the work items are processed in
242	achieve this behavior. Work items on such wq were always queued to the	261	queueing order. While the combination of ``@max_active`` of 1 and
243	unbound worker-pools and only one work item could be active at any given	262	``WQ_UNBOUND`` used to achieve this behavior, this is no longer the
244	time thus achieving the same ordering property as ST wq.	263	case. Use ``alloc_ordered_queue()`` instead.
245
246	In the current implementation the above configuration only guarantees
247	ST behavior within a given NUMA node. Instead ``alloc_ordered_workqueue()`` should
248	be used to achieve system-wide ST behavior.
249		264
250		265
251	Example Execution Scenarios	266	Example Execution Scenarios

include/linux/async.h+1−0

@@ -120,4 +120,5 @@ extern void async_synchronize_cookie(async_cookie_t cookie); · 119 unmodified lines
120	extern void async_synchronize_cookie_domain(async_cookie_t cookie,	120	extern void async_synchronize_cookie_domain(async_cookie_t cookie,
121	struct async_domain *domain);	121	struct async_domain *domain);
122	extern bool current_is_async(void);	122	extern bool current_is_async(void);
		123	extern void async_init(void);
123	#endif	124	#endif

include/linux/workqueue.h+99−42

@@ -22,20 +22,54 @@ · 21 unmodified lines
22	*/	22	*/
23	#define work_data_bits(work) ((unsigned long *)(&(work)->data))	23	#define work_data_bits(work) ((unsigned long *)(&(work)->data))
24		24
25	enum {	25	enum work_bits {
26	WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */	26	WORK_STRUCT_PENDING_BIT = 0, /* work item is pending execution */
27	WORK_STRUCT_INACTIVE_BIT= 1, /* work item is inactive */	27	WORK_STRUCT_INACTIVE_BIT, /* work item is inactive */
28	WORK_STRUCT_PWQ_BIT = 2, /* data points to pwq */	28	WORK_STRUCT_PWQ_BIT, /* data points to pwq */
29	WORK_STRUCT_LINKED_BIT = 3, /* next work is linked to this one */	29	WORK_STRUCT_LINKED_BIT, /* next work is linked to this one */
30	#ifdef CONFIG_DEBUG_OBJECTS_WORK	30	#ifdef CONFIG_DEBUG_OBJECTS_WORK
31	WORK_STRUCT_STATIC_BIT = 4, /* static initializer (debugobjects) */	31	WORK_STRUCT_STATIC_BIT, /* static initializer (debugobjects) */
32	WORK_STRUCT_COLOR_SHIFT = 5, /* color for workqueue flushing */
33	#else
34	WORK_STRUCT_COLOR_SHIFT = 4, /* color for workqueue flushing */
35	#endif	32	#endif
		33	WORK_STRUCT_FLAG_BITS,
36		34
		35	/* color for workqueue flushing */
		36	WORK_STRUCT_COLOR_SHIFT = WORK_STRUCT_FLAG_BITS,
37	WORK_STRUCT_COLOR_BITS = 4,	37	WORK_STRUCT_COLOR_BITS = 4,
38		38
		39	/*
		40	* When WORK_STRUCT_PWQ is set, reserve 8 bits off of pwq pointer w/
		41	* debugobjects turned off. This makes pwqs aligned to 256 bytes (512
		42	* bytes w/ DEBUG_OBJECTS_WORK) and allows 16 workqueue flush colors.
		43	*
		44	* MSB
		45	* [ pwq pointer ] [ flush color ] [ STRUCT flags ]
		46	* 4 bits 4 or 5 bits
		47	*/
		48	WORK_STRUCT_PWQ_SHIFT = WORK_STRUCT_COLOR_SHIFT + WORK_STRUCT_COLOR_BITS,
		49
		50	/*
		51	* data contains off-queue information when !WORK_STRUCT_PWQ.
		52	*
		53	* MSB
		54	* [ pool ID ] [ OFFQ flags ] [ STRUCT flags ]
		55	* 1 bit 4 or 5 bits
		56	*/
		57	WORK_OFFQ_FLAG_SHIFT = WORK_STRUCT_FLAG_BITS,
		58	WORK_OFFQ_CANCELING_BIT = WORK_OFFQ_FLAG_SHIFT,
		59	WORK_OFFQ_FLAG_END,
		60	WORK_OFFQ_FLAG_BITS = WORK_OFFQ_FLAG_END - WORK_OFFQ_FLAG_SHIFT,
		61
		62	/*
		63	* When a work item is off queue, the high bits encode off-queue flags
		64	* and the last pool it was on. Cap pool ID to 31 bits and use the
		65	* highest number to indicate that no pool is associated.
		66	*/
		67	WORK_OFFQ_POOL_SHIFT = WORK_OFFQ_FLAG_SHIFT + WORK_OFFQ_FLAG_BITS,
		68	WORK_OFFQ_LEFT = BITS_PER_LONG - WORK_OFFQ_POOL_SHIFT,
		69	WORK_OFFQ_POOL_BITS = WORK_OFFQ_LEFT <= 31 ? WORK_OFFQ_LEFT : 31,
		70	};
		71
		72	enum work_flags {
39	WORK_STRUCT_PENDING = 1 << WORK_STRUCT_PENDING_BIT,	73	WORK_STRUCT_PENDING = 1 << WORK_STRUCT_PENDING_BIT,
40	WORK_STRUCT_INACTIVE = 1 << WORK_STRUCT_INACTIVE_BIT,	74	WORK_STRUCT_INACTIVE = 1 << WORK_STRUCT_INACTIVE_BIT,
41	WORK_STRUCT_PWQ = 1 << WORK_STRUCT_PWQ_BIT,	75	WORK_STRUCT_PWQ = 1 << WORK_STRUCT_PWQ_BIT,
@@ -45,35 +79,14 @@ enum { · 3 unmodified lines
45	#else	79	#else
46	WORK_STRUCT_STATIC = 0,	80	WORK_STRUCT_STATIC = 0,
47	#endif	81	#endif
		82	};
48		83
		84	enum wq_misc_consts {
49	WORK_NR_COLORS = (1 << WORK_STRUCT_COLOR_BITS),	85	WORK_NR_COLORS = (1 << WORK_STRUCT_COLOR_BITS),
50		86
51	/* not bound to any CPU, prefer the local CPU */	87	/* not bound to any CPU, prefer the local CPU */
52	WORK_CPU_UNBOUND = NR_CPUS,	88	WORK_CPU_UNBOUND = NR_CPUS,
53		89
54	/*
55	* Reserve 8 bits off of pwq pointer w/ debugobjects turned off.
56	* This makes pwqs aligned to 256 bytes and allows 16 workqueue
57	* flush colors.
58	*/
59	WORK_STRUCT_FLAG_BITS = WORK_STRUCT_COLOR_SHIFT +
60	WORK_STRUCT_COLOR_BITS,
61
62	/* data contains off-queue information when !WORK_STRUCT_PWQ */
63	WORK_OFFQ_FLAG_BASE = WORK_STRUCT_COLOR_SHIFT,
64
65	__WORK_OFFQ_CANCELING = WORK_OFFQ_FLAG_BASE,
66
67	/*
68	* When a work item is off queue, its high bits point to the last
69	* pool it was on. Cap at 31 bits and use the highest number to
70	* indicate that no pool is associated.
71	*/
72	WORK_OFFQ_FLAG_BITS = 1,
73	WORK_OFFQ_POOL_SHIFT = WORK_OFFQ_FLAG_BASE + WORK_OFFQ_FLAG_BITS,
74	WORK_OFFQ_LEFT = BITS_PER_LONG - WORK_OFFQ_POOL_SHIFT,
75	WORK_OFFQ_POOL_BITS = WORK_OFFQ_LEFT <= 31 ? WORK_OFFQ_LEFT : 31,
76
77	/* bit mask for work_busy() return values */	90	/* bit mask for work_busy() return values */
78	WORK_BUSY_PENDING = 1 << 0,	91	WORK_BUSY_PENDING = 1 << 0,
79	WORK_BUSY_RUNNING = 1 << 1,	92	WORK_BUSY_RUNNING = 1 << 1,
@@ -83,12 +96,10 @@ enum { · 3 unmodified lines
83	};	96	};
84		97
85	/* Convenience constants - of type 'unsigned long', not 'enum'! */	98	/* Convenience constants - of type 'unsigned long', not 'enum'! */
86	#define WORK_OFFQ_CANCELING (1ul << __WORK_OFFQ_CANCELING)	99	#define WORK_OFFQ_CANCELING (1ul << WORK_OFFQ_CANCELING_BIT)
87	#define WORK_OFFQ_POOL_NONE ((1ul << WORK_OFFQ_POOL_BITS) - 1)	100	#define WORK_OFFQ_POOL_NONE ((1ul << WORK_OFFQ_POOL_BITS) - 1)
88	#define WORK_STRUCT_NO_POOL (WORK_OFFQ_POOL_NONE << WORK_OFFQ_POOL_SHIFT)	101	#define WORK_STRUCT_NO_POOL (WORK_OFFQ_POOL_NONE << WORK_OFFQ_POOL_SHIFT)
89		102	#define WORK_STRUCT_PWQ_MASK (~((1ul << WORK_STRUCT_PWQ_SHIFT) - 1))
90	#define WORK_STRUCT_FLAG_MASK ((1ul << WORK_STRUCT_FLAG_BITS) - 1)
91	#define WORK_STRUCT_WQ_DATA_MASK (~WORK_STRUCT_FLAG_MASK)
92		103
93	#define WORK_DATA_INIT() ATOMIC_LONG_INIT((unsigned long)WORK_STRUCT_NO_POOL)	104	#define WORK_DATA_INIT() ATOMIC_LONG_INIT((unsigned long)WORK_STRUCT_NO_POOL)
94	#define WORK_DATA_STATIC_INIT() \	105	#define WORK_DATA_STATIC_INIT() \
@@ -347,7 +358,8 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; } · 252 unmodified lines
347	* Workqueue flags and constants. For details, please refer to	358	* Workqueue flags and constants. For details, please refer to
348	* Documentation/core-api/workqueue.rst.	359	* Documentation/core-api/workqueue.rst.
349	*/	360	*/
350	enum {	361	enum wq_flags {
		362	WQ_BH = 1 << 0, /* execute in bottom half (softirq) context */
351	WQ_UNBOUND = 1 << 1, /* not bound to any cpu */	363	WQ_UNBOUND = 1 << 1, /* not bound to any cpu */
352	WQ_FREEZABLE = 1 << 2, /* freeze during suspend */	364	WQ_FREEZABLE = 1 << 2, /* freeze during suspend */
353	WQ_MEM_RECLAIM = 1 << 3, /* may be used for memory reclaim */	365	WQ_MEM_RECLAIM = 1 << 3, /* may be used for memory reclaim */
@@ -386,11 +398,22 @@ enum { · 32 unmodified lines
386	__WQ_DRAINING = 1 << 16, /* internal: workqueue is draining */	398	__WQ_DRAINING = 1 << 16, /* internal: workqueue is draining */
387	__WQ_ORDERED = 1 << 17, /* internal: workqueue is ordered */	399	__WQ_ORDERED = 1 << 17, /* internal: workqueue is ordered */
388	__WQ_LEGACY = 1 << 18, /* internal: create_workqueue() /	400	__WQ_LEGACY = 1 << 18, /* internal: create_workqueue() /
389	__WQ_ORDERED_EXPLICIT = 1 << 19, /* internal: alloc_ordered_workqueue() */
390		401
		402	/* BH wq only allows the following flags */
		403	__WQ_BH_ALLOWS = WQ_BH \| WQ_HIGHPRI,
		404	};
		405
		406	enum wq_consts {
391	WQ_MAX_ACTIVE = 512, /* I like 512, better ideas? */	407	WQ_MAX_ACTIVE = 512, /* I like 512, better ideas? */
392	WQ_UNBOUND_MAX_ACTIVE = WQ_MAX_ACTIVE,	408	WQ_UNBOUND_MAX_ACTIVE = WQ_MAX_ACTIVE,
393	WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2,	409	WQ_DFL_ACTIVE = WQ_MAX_ACTIVE / 2,
		410
		411	/*
		412	* Per-node default cap on min_active. Unless explicitly set, min_active
		413	* is set to min(max_active, WQ_DFL_MIN_ACTIVE). For more details, see
		414	* workqueue_struct->min_active definition.
		415	*/
		416	WQ_DFL_MIN_ACTIVE = 8,
394	};	417	};
395		418
396	/*	419	/*
@@ -420,6 +443,9 @@ enum { · 23 unmodified lines
420	* they are same as their non-power-efficient counterparts - e.g.	443	* they are same as their non-power-efficient counterparts - e.g.
421	* system_power_efficient_wq is identical to system_wq if	444	* system_power_efficient_wq is identical to system_wq if
422	* 'wq_power_efficient' is disabled. See WQ_POWER_EFFICIENT for more info.	445	* 'wq_power_efficient' is disabled. See WQ_POWER_EFFICIENT for more info.
		446	*
		447	* system_bh[_highpri]_wq are convenience interface to softirq. BH work items
		448	* are executed in the queueing CPU's BH context in the queueing order.
423	*/	449	*/
424	extern struct workqueue_struct *system_wq;	450	extern struct workqueue_struct *system_wq;
425	extern struct workqueue_struct *system_highpri_wq;	451	extern struct workqueue_struct *system_highpri_wq;
@@ -428,16 +454,43 @@ extern struct workqueue_struct *system_unbound_wq; · 2 unmodified lines
428	extern struct workqueue_struct *system_freezable_wq;	454	extern struct workqueue_struct *system_freezable_wq;
429	extern struct workqueue_struct *system_power_efficient_wq;	455	extern struct workqueue_struct *system_power_efficient_wq;
430	extern struct workqueue_struct *system_freezable_power_efficient_wq;	456	extern struct workqueue_struct *system_freezable_power_efficient_wq;
		457	extern struct workqueue_struct *system_bh_wq;
		458	extern struct workqueue_struct *system_bh_highpri_wq;
		459
		460	void workqueue_softirq_action(bool highpri);
		461	void workqueue_softirq_dead(unsigned int cpu);
431		462
432	/**	463	/**
433	* alloc_workqueue - allocate a workqueue	464	* alloc_workqueue - allocate a workqueue
434	* @fmt: printf format for the name of the workqueue	465	* @fmt: printf format for the name of the workqueue
435	* @flags: WQ_* flags	466	* @flags: WQ_* flags
436	* @max_active: max in-flight work items per CPU, 0 for default	467	* @max_active: max in-flight work items, 0 for default
437	* remaining args: args for @fmt	468	* remaining args: args for @fmt
438	*	469	*
439	* Allocate a workqueue with the specified parameters. For detailed	470	* For a per-cpu workqueue, @max_active limits the number of in-flight work
440	* information on WQ_* flags, please refer to	471	* items for each CPU. e.g. @max_active of 1 indicates that each CPU can be
		472	* executing at most one work item for the workqueue.
		473	*
		474	* For unbound workqueues, @max_active limits the number of in-flight work items
		475	* for the whole system. e.g. @max_active of 16 indicates that that there can be
		476	* at most 16 work items executing for the workqueue in the whole system.
		477	*
		478	* As sharing the same active counter for an unbound workqueue across multiple
		479	* NUMA nodes can be expensive, @max_active is distributed to each NUMA node
		480	* according to the proportion of the number of online CPUs and enforced
		481	* independently.
		482	*
		483	* Depending on online CPU distribution, a node may end up with per-node
		484	* max_active which is significantly lower than @max_active, which can lead to
		485	* deadlocks if the per-node concurrency limit is lower than the maximum number
		486	* of interdependent work items for the workqueue.
		487	*
		488	* To guarantee forward progress regardless of online CPU distribution, the
		489	* concurrency limit on every node is guaranteed to be equal to or greater than
		490	* min_active which is set to min(@max_active, %WQ_DFL_MIN_ACTIVE). This means
		491	* that the sum of per-node max_active's may be larger than @max_active.
		492	*
		493	* For detailed information on %WQ_* flags, please refer to
441	* Documentation/core-api/workqueue.rst.	494	* Documentation/core-api/workqueue.rst.
442	*	495	*
443	* RETURNS:	496	* RETURNS:
@@ -460,8 +513,7 @@ alloc_workqueue(const char *fmt, unsigned int flags, int max_active, ...); · 16 unmodified lines
460	* Pointer to the allocated workqueue on success, %NULL on failure.	513	* Pointer to the allocated workqueue on success, %NULL on failure.
461	*/	514	*/
462	#define alloc_ordered_workqueue(fmt, flags, args...) \	515	#define alloc_ordered_workqueue(fmt, flags, args...) \
463	alloc_workqueue(fmt, WQ_UNBOUND \| __WQ_ORDERED \| \	516	alloc_workqueue(fmt, WQ_UNBOUND \| __WQ_ORDERED \| (flags), 1, ##args)
464	__WQ_ORDERED_EXPLICIT \| (flags), 1, ##args)
465		517
466	#define create_workqueue(name) \	518	#define create_workqueue(name) \
467	alloc_workqueue("%s", __WQ_LEGACY \| WQ_MEM_RECLAIM, 1, (name))	519	alloc_workqueue("%s", __WQ_LEGACY \| WQ_MEM_RECLAIM, 1, (name))
@@ -471,6 +523,9 @@ alloc_workqueue(const char *fmt, unsigned int flags, int max_active, ...); · 3 unmodified lines
471	#define create_singlethread_workqueue(name) \	523	#define create_singlethread_workqueue(name) \
472	alloc_ordered_workqueue("%s", __WQ_LEGACY \| WQ_MEM_RECLAIM, name)	524	alloc_ordered_workqueue("%s", __WQ_LEGACY \| WQ_MEM_RECLAIM, name)
473		525
		526	#define from_work(var, callback_work, work_fieldname) \
		527	container_of(callback_work, typeof(*var), work_fieldname)
		528
474	extern void destroy_workqueue(struct workqueue_struct *wq);	529	extern void destroy_workqueue(struct workqueue_struct *wq);
475		530
476	struct workqueue_attrs *alloc_workqueue_attrs(void);	531	struct workqueue_attrs *alloc_workqueue_attrs(void);
@@ -508,6 +563,8 @@ extern bool flush_rcu_work(struct rcu_work *rwork); · 31 unmodified lines
508		563
509	extern void workqueue_set_max_active(struct workqueue_struct *wq,	564	extern void workqueue_set_max_active(struct workqueue_struct *wq,
510	int max_active);	565	int max_active);
		566	extern void workqueue_set_min_active(struct workqueue_struct *wq,
		567	int min_active);
511	extern struct work_struct *current_work(void);	568	extern struct work_struct *current_work(void);
512	extern bool current_is_workqueue_rescuer(void);	569	extern bool current_is_workqueue_rescuer(void);
513	extern bool workqueue_congested(int cpu, struct workqueue_struct *wq);	570	extern bool workqueue_congested(int cpu, struct workqueue_struct *wq);

init/Kconfig+1−1

@@ -115,7 +115,7 @@ config CONSTRUCTORS · 114 unmodified lines
115	bool	115	bool
116		116
117	config IRQ_WORK	117	config IRQ_WORK
118	bool	118	def_bool y if SMP
119		119
120	config BUILDTIME_TABLE_SORT	120	config BUILDTIME_TABLE_SORT
121	bool	121	bool

init/main.c+1−0

@@ -1547,6 +1547,7 @@ static noinline void __init kernel_init_freeable(void) · 1546 unmodified lines
1547	sched_init_smp();	1547	sched_init_smp();
1548		1548
1549	workqueue_init_topology();	1549	workqueue_init_topology();
		1550	async_init();
1550	padata_init();	1551	padata_init();
1551	page_alloc_init_late();	1552	page_alloc_init_late();
1552		1553

kernel/async.c+16−1

 static LIST_HEAD(async_global_pending);	/* pending from all registered doms */
 static ASYNC_DOMAIN(async_dfl_domain);
 static DEFINE_SPINLOCK(async_lock);
+static struct workqueue_struct *async_wq;
 struct async_entry {
 	struct list_head	domain_list;
 	spin_unlock_irqrestore(&async_lock, flags);
 	/* schedule for execution */
-	queue_work_node(node, system_unbound_wq, &entry->work);
+	queue_work_node(node, async_wq, &entry->work);
 	return newcookie;
 }
 	return worker && worker->current_func == async_run_entry_fn;
 }
 EXPORT_SYMBOL_GPL(current_is_async);
+void __init async_init(void)
+{
+	/*
+	 * Async can schedule a number of interdependent work items. However,
+	 * unbound workqueues can handle only upto min_active interdependent
+	 * work items. The default min_active of 8 isn't sufficient for async
+	 * and can lead to stalls. Let's use a dedicated workqueue with raised
+	 * min_active.
+	 */
+	async_wq = alloc_workqueue("async", WQ_UNBOUND, 0);
+	BUG_ON(!async_wq);
+	workqueue_set_min_active(async_wq, WQ_DFL_ACTIVE);
+}

kernel/softirq.c+5−0

 #include <linux/tick.h>
 #include <linux/irq.h>
 #include <linux/wait_bit.h>
+#include <linux/workqueue.h>
 #include <asm/softirq_stack.h>
 static __latent_entropy void tasklet_action(struct softirq_action *a)
 {
+	workqueue_softirq_action(false);
 	tasklet_action_common(a, this_cpu_ptr(&tasklet_vec), TASKLET_SOFTIRQ);
 }
 static __latent_entropy void tasklet_hi_action(struct softirq_action *a)
 {
+	workqueue_softirq_action(true);
 	tasklet_action_common(a, this_cpu_ptr(&tasklet_hi_vec), HI_SOFTIRQ);
 }
 #ifdef CONFIG_HOTPLUG_CPU
 static int takeover_tasklets(unsigned int cpu)
 {
+	workqueue_softirq_dead(cpu);
 	/* CPU is dead, so no lock needed. */
 	local_irq_disable();

kernel/workqueue.c+1434−457

@@ -29,6 +29,7 @@ · 28 unmodified lines
29	#include <linux/kernel.h>	29	#include <linux/kernel.h>
30	#include <linux/sched.h>	30	#include <linux/sched.h>
31	#include <linux/init.h>	31	#include <linux/init.h>
		32	#include <linux/interrupt.h>
32	#include <linux/signal.h>	33	#include <linux/signal.h>
33	#include <linux/completion.h>	34	#include <linux/completion.h>
34	#include <linux/workqueue.h>	35	#include <linux/workqueue.h>
@@ -53,10 +54,11 @@ · 18 unmodified lines
53	#include <linux/nmi.h>	54	#include <linux/nmi.h>
54	#include <linux/kvm_para.h>	55	#include <linux/kvm_para.h>
55	#include <linux/delay.h>	56	#include <linux/delay.h>
		57	#include <linux/irq_work.h>
56		58
57	#include "workqueue_internal.h"	59	#include "workqueue_internal.h"
58		60
59	enum {	61	enum worker_pool_flags {
60	/*	62	/*
61	* worker_pool flags	63	* worker_pool flags
62	*	64	*
@@ -72,10 +74,17 @@ enum { · 9 unmodified lines
72	* Note that DISASSOCIATED should be flipped only while holding	74	* Note that DISASSOCIATED should be flipped only while holding
73	* wq_pool_attach_mutex to avoid changing binding state while	75	* wq_pool_attach_mutex to avoid changing binding state while
74	* worker_attach_to_pool() is in progress.	76	* worker_attach_to_pool() is in progress.
		77	*
		78	* As there can only be one concurrent BH execution context per CPU, a
		79	* BH pool is per-CPU and always DISASSOCIATED.
75	*/	80	*/
76	POOL_MANAGER_ACTIVE = 1 << 0, /* being managed */	81	POOL_BH = 1 << 0, /* is a BH pool */
		82	POOL_MANAGER_ACTIVE = 1 << 1, /* being managed */
77	POOL_DISASSOCIATED = 1 << 2, /* cpu can't serve workers */	83	POOL_DISASSOCIATED = 1 << 2, /* cpu can't serve workers */
		84	POOL_BH_DRAINING = 1 << 3, /* draining after CPU offline */
		85	};
78		86
		87	enum worker_flags {
79	/* worker flags */	88	/* worker flags */
80	WORKER_DIE = 1 << 1, /* die die die */	89	WORKER_DIE = 1 << 1, /* die die die */
81	WORKER_IDLE = 1 << 2, /* is idle */	90	WORKER_IDLE = 1 << 2, /* is idle */
@@ -86,7 +95,13 @@ enum { · 4 unmodified lines
86		95
87	WORKER_NOT_RUNNING = WORKER_PREP \| WORKER_CPU_INTENSIVE \|	96	WORKER_NOT_RUNNING = WORKER_PREP \| WORKER_CPU_INTENSIVE \|
88	WORKER_UNBOUND \| WORKER_REBOUND,	97	WORKER_UNBOUND \| WORKER_REBOUND,
		98	};
		99
		100	enum work_cancel_flags {
		101	WORK_CANCEL_DELAYED = 1 << 0, /* canceling a delayed_work */
		102	};
89		103
		104	enum wq_internal_consts {
90	NR_STD_WORKER_POOLS = 2, /* # standard pools per cpu */	105	NR_STD_WORKER_POOLS = 2, /* # standard pools per cpu */
91		106
92	UNBOUND_POOL_HASH_ORDER = 6, /* hashed by pool->attrs */	107	UNBOUND_POOL_HASH_ORDER = 6, /* hashed by pool->attrs */
@@ -108,9 +123,17 @@ enum { · 15 unmodified lines
108	RESCUER_NICE_LEVEL = MIN_NICE,	123	RESCUER_NICE_LEVEL = MIN_NICE,
109	HIGHPRI_NICE_LEVEL = MIN_NICE,	124	HIGHPRI_NICE_LEVEL = MIN_NICE,
110		125
111	WQ_NAME_LEN = 24,	126	WQ_NAME_LEN = 32,
112	};	127	};
113		128
		129	/*
		130	* We don't want to trap softirq for too long. See MAX_SOFTIRQ_TIME and
		131	* MAX_SOFTIRQ_RESTART in kernel/softirq.c. These are macros because
		132	* msecs_to_jiffies() can't be an initializer.
		133	*/
		134	#define BH_WORKER_JIFFIES msecs_to_jiffies(2)
		135	#define BH_WORKER_RESTARTS 10
		136
114	/*	137	/*
115	* Structure fields follow one of the following exclusion rules.	138	* Structure fields follow one of the following exclusion rules.
116	*	139	*
@@ -122,6 +145,9 @@ enum { · 5 unmodified lines
122	*	145	*
123	* L: pool->lock protected. Access with pool->lock held.	146	* L: pool->lock protected. Access with pool->lock held.
124	*	147	*
		148	* LN: pool->lock and wq_node_nr_active->lock protected for writes. Either for
		149	* reads.
		150	*
125	* K: Only modified by worker while holding pool->lock. Can be safely read by	151	* K: Only modified by worker while holding pool->lock. Can be safely read by
126	* self, while holding pool->lock or from IRQ context if %current is the	152	* self, while holding pool->lock or from IRQ context if %current is the
127	* kworker.	153	* kworker.
@@ -143,6 +169,9 @@ enum { · 15 unmodified lines
143	*	169	*
144	* WR: wq->mutex protected for writes. RCU protected for reads.	170	* WR: wq->mutex protected for writes. RCU protected for reads.
145	*	171	*
		172	* WO: wq->mutex protected for writes. Updated with WRITE_ONCE() and can be read
		173	* with READ_ONCE() without locking.
		174	*
146	* MD: wq_mayday_lock protected.	175	* MD: wq_mayday_lock protected.
147	*	176	*
148	* WD: Used internally by the watchdog.	177	* WD: Used internally by the watchdog.
@@ -219,7 +248,7 @@ enum pool_workqueue_stats { · 70 unmodified lines
219	};	248	};
220		249
221	/*	250	/*
222	* The per-pool workqueue. While queued, the lower WORK_STRUCT_FLAG_BITS	251	* The per-pool workqueue. While queued, bits below WORK_PWQ_SHIFT
223	* of work_struct->data are used for flags and the remaining high bits	252	* of work_struct->data are used for flags and the remaining high bits
224	* point to the pwq; thus, pwqs need to be aligned at two's power of the	253	* point to the pwq; thus, pwqs need to be aligned at two's power of the
225	* number of flag bits.	254	* number of flag bits.
@@ -232,6 +261,7 @@ struct pool_workqueue { · 6 unmodified lines
232	int refcnt; /* L: reference count */	261	int refcnt; /* L: reference count */
233	int nr_in_flight[WORK_NR_COLORS];	262	int nr_in_flight[WORK_NR_COLORS];
234	/* L: nr of in_flight works */	263	/* L: nr of in_flight works */
		264	bool plugged; /* L: execution suspended */
235		265
236	/*	266	/*
237	* nr_active management and WORK_STRUCT_INACTIVE:	267	* nr_active management and WORK_STRUCT_INACTIVE:
@@ -240,18 +270,18 @@ struct pool_workqueue { · 2 unmodified lines
240	* pwq->inactive_works instead of pool->worklist and marked with	270	* pwq->inactive_works instead of pool->worklist and marked with
241	* WORK_STRUCT_INACTIVE.	271	* WORK_STRUCT_INACTIVE.
242	*	272	*
243	* All work items marked with WORK_STRUCT_INACTIVE do not participate	273	* All work items marked with WORK_STRUCT_INACTIVE do not participate in
244	* in pwq->nr_active and all work items in pwq->inactive_works are	274	* nr_active and all work items in pwq->inactive_works are marked with
245	* marked with WORK_STRUCT_INACTIVE. But not all WORK_STRUCT_INACTIVE	275	* WORK_STRUCT_INACTIVE. But not all WORK_STRUCT_INACTIVE work items are
246	* work items are in pwq->inactive_works. Some of them are ready to	276	* in pwq->inactive_works. Some of them are ready to run in
247	* run in pool->worklist or worker->scheduled. Those work itmes are	277	* pool->worklist or worker->scheduled. Those work itmes are only struct
248	* only struct wq_barrier which is used for flush_work() and should	278	* wq_barrier which is used for flush_work() and should not participate
249	* not participate in pwq->nr_active. For non-barrier work item, it	279	* in nr_active. For non-barrier work item, it is marked with
250	* is marked with WORK_STRUCT_INACTIVE iff it is in pwq->inactive_works.	280	* WORK_STRUCT_INACTIVE iff it is in pwq->inactive_works.
251	*/	281	*/
252	int nr_active; /* L: nr of active works */	282	int nr_active; /* L: nr of active works */
253	int max_active; /* L: max active works */
254	struct list_head inactive_works; /* L: inactive works */	283	struct list_head inactive_works; /* L: inactive works */
		284	struct list_head pending_node; /* LN: node on wq_node_nr_active->pending_pwqs */
255	struct list_head pwqs_node; /* WR: node on wq->pwqs */	285	struct list_head pwqs_node; /* WR: node on wq->pwqs */
256	struct list_head mayday_node; /* MD: node on wq->maydays */	286	struct list_head mayday_node; /* MD: node on wq->maydays */
257		287
@@ -265,7 +295,7 @@ struct pool_workqueue { · 7 unmodified lines
265	*/	295	*/
266	struct kthread_work release_work;	296	struct kthread_work release_work;
267	struct rcu_head rcu;	297	struct rcu_head rcu;
268	} __aligned(1 << WORK_STRUCT_FLAG_BITS);	298	} __aligned(1 << WORK_STRUCT_PWQ_SHIFT);
269		299
270	/*	300	/*
271	* Structure used to wait for workqueue flush.	301	* Structure used to wait for workqueue flush.
@@ -278,6 +308,26 @@ struct wq_flusher { · 6 unmodified lines
278		308
279	struct wq_device;	309	struct wq_device;
280		310
		311	/*
		312	* Unlike in a per-cpu workqueue where max_active limits its concurrency level
		313	* on each CPU, in an unbound workqueue, max_active applies to the whole system.
		314	* As sharing a single nr_active across multiple sockets can be very expensive,
		315	* the counting and enforcement is per NUMA node.
		316	*
		317	* The following struct is used to enforce per-node max_active. When a pwq wants
		318	* to start executing a work item, it should increment ->nr using
		319	* tryinc_node_nr_active(). If acquisition fails due to ->nr already being over
		320	* ->max, the pwq is queued on ->pending_pwqs. As in-flight work items finish
		321	* and decrement ->nr, node_activate_pending_pwq() activates the pending pwqs in
		322	* round-robin order.
		323	*/
		324	struct wq_node_nr_active {
		325	int max; /* per-node max_active */
		326	atomic_t nr; /* per-node nr_active */
		327	raw_spinlock_t lock; /* nests inside pool locks */
		328	struct list_head pending_pwqs; /* LN: pwqs with inactive works */
		329	};
		330
281	/*	331	/*
282	* The externally visible workqueue. It relays the issued work items to	332	* The externally visible workqueue. It relays the issued work items to
283	* the appropriate worker_pool through its pool_workqueues.	333	* the appropriate worker_pool through its pool_workqueues.
@@ -298,10 +348,15 @@ struct workqueue_struct { · 14 unmodified lines
298	struct worker rescuer; / MD: rescue worker */	348	struct worker rescuer; / MD: rescue worker */
299		349
300	int nr_drainers; /* WQ: drain in progress */	350	int nr_drainers; /* WQ: drain in progress */
301	int saved_max_active; /* WQ: saved pwq max_active */	351
		352	/* See alloc_workqueue() function comment for info on min/max_active */
		353	int max_active; /* WO: max active works */
		354	int min_active; /* WO: min active works */
		355	int saved_max_active; /* WQ: saved max_active */
		356	int saved_min_active; /* WQ: saved min_active */
302		357
303	struct workqueue_attrs unbound_attrs; / PW: only for unbound wqs */	358	struct workqueue_attrs unbound_attrs; / PW: only for unbound wqs */
304	struct pool_workqueue dfl_pwq; / PW: only for unbound wqs */	359	struct pool_workqueue __rcu dfl_pwq; / PW: only for unbound wqs */
305		360
306	#ifdef CONFIG_SYSFS	361	#ifdef CONFIG_SYSFS
307	struct wq_device wq_dev; / I: for sysfs interface */	362	struct wq_device wq_dev; / I: for sysfs interface */
@@ -323,10 +378,9 @@ struct workqueue_struct { · 15 unmodified lines
323	/* hot fields used during command issue, aligned to cacheline */	378	/* hot fields used during command issue, aligned to cacheline */
324	unsigned int flags ____cacheline_aligned; /* WQ: WQ_* flags */	379	unsigned int flags ____cacheline_aligned; /* WQ: WQ_* flags */
325	struct pool_workqueue __percpu __rcu *cpu_pwq; / I: per-cpu pwqs */	380	struct pool_workqueue __percpu __rcu *cpu_pwq; / I: per-cpu pwqs */
		381	struct wq_node_nr_active node_nr_active[]; / I: per-node nr_active */
326	};	382	};
327		383
328	static struct kmem_cache *pwq_cache;
329
330	/*	384	/*
331	* Each pod type describes how CPUs should be grouped for unbound workqueues.	385	* Each pod type describes how CPUs should be grouped for unbound workqueues.
332	* See the comment above workqueue_attrs->affn_scope.	386	* See the comment above workqueue_attrs->affn_scope.
@@ -338,16 +392,13 @@ struct wq_pod_type { · 5 unmodified lines
338	int cpu_pod; / cpu -> pod */	392	int cpu_pod; / cpu -> pod */
339	};	393	};
340		394
341	static struct wq_pod_type wq_pod_types[WQ_AFFN_NR_TYPES];
342	static enum wq_affn_scope wq_affn_dfl = WQ_AFFN_CACHE;
343
344	static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = {	395	static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = {
345	[WQ_AFFN_DFL] = "default",	396	[WQ_AFFN_DFL] = "default",
346	[WQ_AFFN_CPU] = "cpu",	397	[WQ_AFFN_CPU] = "cpu",
347	[WQ_AFFN_SMT] = "smt",	398	[WQ_AFFN_SMT] = "smt",
348	[WQ_AFFN_CACHE] = "cache",	399	[WQ_AFFN_CACHE] = "cache",
349	[WQ_AFFN_NUMA] = "numa",	400	[WQ_AFFN_NUMA] = "numa",
350	[WQ_AFFN_SYSTEM] = "system",	401	[WQ_AFFN_SYSTEM] = "system",
351	};	402	};
352		403
353	/*	404	/*
@@ -359,12 +410,22 @@ static const char *wq_affn_names[WQ_AFFN_NR_TYPES] = { · 5 unmodified lines
359	*/	410	*/
360	static unsigned long wq_cpu_intensive_thresh_us = ULONG_MAX;	411	static unsigned long wq_cpu_intensive_thresh_us = ULONG_MAX;
361	module_param_named(cpu_intensive_thresh_us, wq_cpu_intensive_thresh_us, ulong, 0644);	412	module_param_named(cpu_intensive_thresh_us, wq_cpu_intensive_thresh_us, ulong, 0644);
		413	#ifdef CONFIG_WQ_CPU_INTENSIVE_REPORT
		414	static unsigned int wq_cpu_intensive_warning_thresh = 4;
		415	module_param_named(cpu_intensive_warning_thresh, wq_cpu_intensive_warning_thresh, uint, 0644);
		416	#endif
362		417
363	/* see the comment above the definition of WQ_POWER_EFFICIENT */	418	/* see the comment above the definition of WQ_POWER_EFFICIENT */
364	static bool wq_power_efficient = IS_ENABLED(CONFIG_WQ_POWER_EFFICIENT_DEFAULT);	419	static bool wq_power_efficient = IS_ENABLED(CONFIG_WQ_POWER_EFFICIENT_DEFAULT);
365	module_param_named(power_efficient, wq_power_efficient, bool, 0444);	420	module_param_named(power_efficient, wq_power_efficient, bool, 0444);
366		421
367	static bool wq_online; /* can kworkers be created yet? */	422	static bool wq_online; /* can kworkers be created yet? */
		423	static bool wq_topo_initialized __read_mostly = false;
		424
		425	static struct kmem_cache *pwq_cache;
		426
		427	static struct wq_pod_type wq_pod_types[WQ_AFFN_NR_TYPES];
		428	static enum wq_affn_scope wq_affn_dfl = WQ_AFFN_CACHE;
368		429
369	/* buf for wq_update_unbound_pod_attrs(), protected by CPU hotplug exclusion */	430	/* buf for wq_update_unbound_pod_attrs(), protected by CPU hotplug exclusion */
370	static struct workqueue_attrs *wq_update_pod_attrs_buf;	431	static struct workqueue_attrs *wq_update_pod_attrs_buf;
@@ -405,8 +466,17 @@ static bool wq_debug_force_rr_cpu = false; · 34 unmodified lines
405	#endif	466	#endif
406	module_param_named(debug_force_rr_cpu, wq_debug_force_rr_cpu, bool, 0644);	467	module_param_named(debug_force_rr_cpu, wq_debug_force_rr_cpu, bool, 0644);
407		468
		469	/* to raise softirq for the BH worker pools on other CPUs */
		470	static DEFINE_PER_CPU_SHARED_ALIGNED(struct irq_work [NR_STD_WORKER_POOLS],
		471	bh_pool_irq_works);
		472
		473	/* the BH worker pools */
		474	static DEFINE_PER_CPU_SHARED_ALIGNED(struct worker_pool [NR_STD_WORKER_POOLS],
		475	bh_worker_pools);
		476
408	/* the per-cpu worker pools */	477	/* the per-cpu worker pools */
409	static DEFINE_PER_CPU_SHARED_ALIGNED(struct worker_pool [NR_STD_WORKER_POOLS], cpu_worker_pools);	478	static DEFINE_PER_CPU_SHARED_ALIGNED(struct worker_pool [NR_STD_WORKER_POOLS],
		479	cpu_worker_pools);
410		480
411	static DEFINE_IDR(worker_pool_idr); /* PR: idr of all pools */	481	static DEFINE_IDR(worker_pool_idr); /* PR: idr of all pools */
412		482
@@ -419,6 +489,12 @@ static struct workqueue_attrs *unbound_std_wq_attrs[NR_STD_WORKER_POOLS]; · 6 unmodified lines
419	/* I: attributes used when instantiating ordered pools on demand */	489	/* I: attributes used when instantiating ordered pools on demand */
420	static struct workqueue_attrs *ordered_wq_attrs[NR_STD_WORKER_POOLS];	490	static struct workqueue_attrs *ordered_wq_attrs[NR_STD_WORKER_POOLS];
421		491
		492	/*
		493	* Used to synchronize multiple cancel_sync attempts on the same work item. See
		494	* work_grab_pending() and __cancel_work_sync().
		495	*/
		496	static DECLARE_WAIT_QUEUE_HEAD(wq_cancel_waitq);
		497
422	/*	498	/*
423	* I: kthread_worker to release pwq's. pwq release needs to be bounced to a	499	* I: kthread_worker to release pwq's. pwq release needs to be bounced to a
424	* process context while holding a pool lock. Bounce to a dedicated kthread	500	* process context while holding a pool lock. Bounce to a dedicated kthread
@@ -440,6 +516,10 @@ struct workqueue_struct *system_power_efficient_wq __ro_after_init; · 15 unmodified lines
440	EXPORT_SYMBOL_GPL(system_power_efficient_wq);	516	EXPORT_SYMBOL_GPL(system_power_efficient_wq);
441	struct workqueue_struct *system_freezable_power_efficient_wq __ro_after_init;	517	struct workqueue_struct *system_freezable_power_efficient_wq __ro_after_init;
442	EXPORT_SYMBOL_GPL(system_freezable_power_efficient_wq);	518	EXPORT_SYMBOL_GPL(system_freezable_power_efficient_wq);
		519	struct workqueue_struct *system_bh_wq;
		520	EXPORT_SYMBOL_GPL(system_bh_wq);
		521	struct workqueue_struct *system_bh_highpri_wq;
		522	EXPORT_SYMBOL_GPL(system_bh_highpri_wq);
443		523
444	static int worker_thread(void *__worker);	524	static int worker_thread(void *__worker);
445	static void workqueue_sysfs_unregister(struct workqueue_struct *wq);	525	static void workqueue_sysfs_unregister(struct workqueue_struct *wq);
@@ -450,16 +530,21 @@ static void show_one_worker_pool(struct worker_pool *pool); · 4 unmodified lines
450	#include <trace/events/workqueue.h>	530	#include <trace/events/workqueue.h>
451		531
452	#define assert_rcu_or_pool_mutex() \	532	#define assert_rcu_or_pool_mutex() \
453	RCU_LOCKDEP_WARN(!rcu_read_lock_held() && \	533	RCU_LOCKDEP_WARN(!rcu_read_lock_any_held() && \
454	!lockdep_is_held(&wq_pool_mutex), \	534	!lockdep_is_held(&wq_pool_mutex), \
455	"RCU or wq_pool_mutex should be held")	535	"RCU or wq_pool_mutex should be held")
456		536
457	#define assert_rcu_or_wq_mutex_or_pool_mutex(wq) \	537	#define assert_rcu_or_wq_mutex_or_pool_mutex(wq) \
458	RCU_LOCKDEP_WARN(!rcu_read_lock_held() && \	538	RCU_LOCKDEP_WARN(!rcu_read_lock_any_held() && \
459	!lockdep_is_held(&wq->mutex) && \	539	!lockdep_is_held(&wq->mutex) && \
460	!lockdep_is_held(&wq_pool_mutex), \	540	!lockdep_is_held(&wq_pool_mutex), \
461	"RCU, wq->mutex or wq_pool_mutex should be held")	541	"RCU, wq->mutex or wq_pool_mutex should be held")
462		542
		543	#define for_each_bh_worker_pool(pool, cpu) \
		544	for ((pool) = &per_cpu(bh_worker_pools, cpu)[0]; \
		545	(pool) < &per_cpu(bh_worker_pools, cpu)[NR_STD_WORKER_POOLS]; \
		546	(pool)++)
		547
463	#define for_each_cpu_worker_pool(pool, cpu) \	548	#define for_each_cpu_worker_pool(pool, cpu) \
464	for ((pool) = &per_cpu(cpu_worker_pools, cpu)[0]; \	549	for ((pool) = &per_cpu(cpu_worker_pools, cpu)[0]; \
465	(pool) < &per_cpu(cpu_worker_pools, cpu)[NR_STD_WORKER_POOLS]; \	550	(pool) < &per_cpu(cpu_worker_pools, cpu)[NR_STD_WORKER_POOLS]; \
@@ -632,6 +717,36 @@ static int worker_pool_assign_id(struct worker_pool *pool) · 166 unmodified lines
632	return ret;	717	return ret;
633	}	718	}
634		719
		720	static struct pool_workqueue __rcu **
		721	unbound_pwq_slot(struct workqueue_struct *wq, int cpu)
		722	{
		723	if (cpu >= 0)
		724	return per_cpu_ptr(wq->cpu_pwq, cpu);
		725	else
		726	return &wq->dfl_pwq;
		727	}
		728
		729	/* @cpu < 0 for dfl_pwq */
		730	static struct pool_workqueue unbound_pwq(struct workqueue_struct wq, int cpu)
		731	{
		732	return rcu_dereference_check(*unbound_pwq_slot(wq, cpu),
		733	lockdep_is_held(&wq_pool_mutex) \|\|
		734	lockdep_is_held(&wq->mutex));
		735	}
		736
		737	/**
		738	* unbound_effective_cpumask - effective cpumask of an unbound workqueue
		739	* @wq: workqueue of interest
		740	*
		741	* @wq->unbound_attrs->cpumask contains the cpumask requested by the user which
		742	* is masked with wq_unbound_cpumask to determine the effective cpumask. The
		743	* default pwq is always mapped to the pool with the current effective cpumask.
		744	*/
		745	static struct cpumask unbound_effective_cpumask(struct workqueue_struct wq)
		746	{
		747	return unbound_pwq(wq, -1)->pool->attrs->__pod_cpumask;
		748	}
		749
635	static unsigned int work_color_to_flags(int color)	750	static unsigned int work_color_to_flags(int color)
636	{	751	{
637	return color << WORK_STRUCT_COLOR_SHIFT;	752	return color << WORK_STRUCT_COLOR_SHIFT;
@@ -653,10 +768,9 @@ static int work_next_color(int color) · 15 unmodified lines
653	* contain the pointer to the queued pwq. Once execution starts, the flag	768	* contain the pointer to the queued pwq. Once execution starts, the flag
654	* is cleared and the high bits contain OFFQ flags and pool ID.	769	* is cleared and the high bits contain OFFQ flags and pool ID.
655	*	770	*
656	* set_work_pwq(), set_work_pool_and_clear_pending(), mark_work_canceling()	771	* set_work_pwq(), set_work_pool_and_clear_pending() and mark_work_canceling()
657	* and clear_work_data() can be used to set the pwq, pool or clear	772	* can be used to set the pwq, pool or clear work->data. These functions should
658	* work->data. These functions should only be called while the work is	773	* only be called while the work is owned - ie. while the PENDING bit is set.
659	* owned - ie. while the PENDING bit is set.
660	*	774	*
661	* get_work_pool() and get_work_pwq() can be used to obtain the pool or pwq	775	* get_work_pool() and get_work_pwq() can be used to obtain the pool or pwq
662	* corresponding to a work. Pool is available once the work has been	776	* corresponding to a work. Pool is available once the work has been
@@ -668,29 +782,28 @@ static int work_next_color(int color) · 5 unmodified lines
668	* but stay off timer and worklist for arbitrarily long and nobody should	782	* but stay off timer and worklist for arbitrarily long and nobody should
669	* try to steal the PENDING bit.	783	* try to steal the PENDING bit.
670	*/	784	*/
671	static inline void set_work_data(struct work_struct *work, unsigned long data,	785	static inline void set_work_data(struct work_struct *work, unsigned long data)
672	unsigned long flags)
673	{	786	{
674	WARN_ON_ONCE(!work_pending(work));	787	WARN_ON_ONCE(!work_pending(work));
675	atomic_long_set(&work->data, data \| flags \| work_static(work));	788	atomic_long_set(&work->data, data \| work_static(work));
676	}	789	}
677		790
678	static void set_work_pwq(struct work_struct work, struct pool_workqueue pwq,	791	static void set_work_pwq(struct work_struct work, struct pool_workqueue pwq,
679	unsigned long extra_flags)	792	unsigned long flags)
680	{	793	{
681	set_work_data(work, (unsigned long)pwq,	794	set_work_data(work, (unsigned long)pwq \| WORK_STRUCT_PENDING \|
682	WORK_STRUCT_PENDING \| WORK_STRUCT_PWQ \| extra_flags);	795	WORK_STRUCT_PWQ \| flags);
683	}	796	}
684		797
685	static void set_work_pool_and_keep_pending(struct work_struct *work,	798	static void set_work_pool_and_keep_pending(struct work_struct *work,
686	int pool_id)	799	int pool_id, unsigned long flags)
687	{	800	{
688	set_work_data(work, (unsigned long)pool_id << WORK_OFFQ_POOL_SHIFT,	801	set_work_data(work, ((unsigned long)pool_id << WORK_OFFQ_POOL_SHIFT) \|
689	WORK_STRUCT_PENDING);	802	WORK_STRUCT_PENDING \| flags);
690	}	803	}
691		804
692	static void set_work_pool_and_clear_pending(struct work_struct *work,	805	static void set_work_pool_and_clear_pending(struct work_struct *work,
693	int pool_id)	806	int pool_id, unsigned long flags)
694	{	807	{
695	/*	808	/*
696	* The following wmb is paired with the implied mb in	809	* The following wmb is paired with the implied mb in
@@ -699,7 +812,8 @@ static void set_work_pool_and_clear_pending(struct work_struct *work, · 2 unmodified lines
699	* owner.	812	* owner.
700	*/	813	*/
701	smp_wmb();	814	smp_wmb();
702	set_work_data(work, (unsigned long)pool_id << WORK_OFFQ_POOL_SHIFT, 0);	815	set_work_data(work, ((unsigned long)pool_id << WORK_OFFQ_POOL_SHIFT) \|
		816	flags);
703	/*	817	/*
704	* The following mb guarantees that previous clear of a PENDING bit	818	* The following mb guarantees that previous clear of a PENDING bit
705	* will not be reordered with any speculative LOADS or STORES from	819	* will not be reordered with any speculative LOADS or STORES from
@@ -731,15 +845,9 @@ static void set_work_pool_and_clear_pending(struct work_struct *work, · 25 unmodified lines
731	smp_mb();	845	smp_mb();
732	}	846	}
733		847
734	static void clear_work_data(struct work_struct *work)
735	{
736	smp_wmb(); /* see set_work_pool_and_clear_pending() */
737	set_work_data(work, WORK_STRUCT_NO_POOL, 0);
738	}
739
740	static inline struct pool_workqueue *work_struct_pwq(unsigned long data)	848	static inline struct pool_workqueue *work_struct_pwq(unsigned long data)
741	{	849	{
742	return (struct pool_workqueue *)(data & WORK_STRUCT_WQ_DATA_MASK);	850	return (struct pool_workqueue *)(data & WORK_STRUCT_PWQ_MASK);
743	}	851	}
744		852
745	static struct pool_workqueue get_work_pwq(struct work_struct work)	853	static struct pool_workqueue get_work_pwq(struct work_struct work)
@@ -806,7 +914,7 @@ static void mark_work_canceling(struct work_struct *work) · 60 unmodified lines
806	unsigned long pool_id = get_work_pool_id(work);	914	unsigned long pool_id = get_work_pool_id(work);
807		915
808	pool_id <<= WORK_OFFQ_POOL_SHIFT;	916	pool_id <<= WORK_OFFQ_POOL_SHIFT;
809	set_work_data(work, pool_id \| WORK_OFFQ_CANCELING, WORK_STRUCT_PENDING);	917	set_work_data(work, pool_id \| WORK_STRUCT_PENDING \| WORK_OFFQ_CANCELING);
810	}	918	}
811		919
812	static bool work_is_canceling(struct work_struct *work)	920	static bool work_is_canceling(struct work_struct *work)
@@ -1101,6 +1209,29 @@ static bool assign_work(struct work_struct work, struct worker worker, · 288 unmodified lines
1101	return true;	1209	return true;
1102	}	1210	}
1103		1211
		1212	static struct irq_work bh_pool_irq_work(struct worker_pool pool)
		1213	{
		1214	int high = pool->attrs->nice == HIGHPRI_NICE_LEVEL ? 1 : 0;
		1215
		1216	return &per_cpu(bh_pool_irq_works, pool->cpu)[high];
		1217	}
		1218
		1219	static void kick_bh_pool(struct worker_pool *pool)
		1220	{
		1221	#ifdef CONFIG_SMP
		1222	/* see drain_dead_softirq_workfn() for BH_DRAINING */
		1223	if (unlikely(pool->cpu != smp_processor_id() &&
		1224	!(pool->flags & POOL_BH_DRAINING))) {
		1225	irq_work_queue_on(bh_pool_irq_work(pool), pool->cpu);
		1226	return;
		1227	}
		1228	#endif
		1229	if (pool->attrs->nice == HIGHPRI_NICE_LEVEL)
		1230	raise_softirq_irqoff(HI_SOFTIRQ);
		1231	else
		1232	raise_softirq_irqoff(TASKLET_SOFTIRQ);
		1233	}
		1234
1104	/**	1235	/**
1105	* kick_pool - wake up an idle worker if necessary	1236	* kick_pool - wake up an idle worker if necessary
1106	* @pool: pool to kick	1237	* @pool: pool to kick
@@ -1118,6 +1249,11 @@ static bool kick_pool(struct worker_pool *pool) · 11 unmodified lines
1118	if (!need_more_worker(pool) \|\| !worker)	1249	if (!need_more_worker(pool) \|\| !worker)
1119	return false;	1250	return false;
1120		1251
		1252	if (pool->flags & POOL_BH) {
		1253	kick_bh_pool(pool);
		1254	return true;
		1255	}
		1256
1121	p = worker->task;	1257	p = worker->task;
1122		1258
1123	#ifdef CONFIG_SMP	1259	#ifdef CONFIG_SMP
@@ -1198,11 +1334,13 @@ restart: · 74 unmodified lines
1198	u64 cnt;	1334	u64 cnt;
1199		1335
1200	/*	1336	/*
1201	* Start reporting from the fourth time and back off	1337	* Start reporting from the warning_thresh and back off
1202	* exponentially.	1338	* exponentially.
1203	*/	1339	*/
1204	cnt = atomic64_inc_return_relaxed(&ent->cnt);	1340	cnt = atomic64_inc_return_relaxed(&ent->cnt);
1205	if (cnt >= 4 && is_power_of_2(cnt))	1341	if (wq_cpu_intensive_warning_thresh &&
		1342	cnt >= wq_cpu_intensive_warning_thresh &&
		1343	is_power_of_2(cnt + 1 - wq_cpu_intensive_warning_thresh))
1206	printk_deferred(KERN_WARNING "workqueue: %ps hogged CPU for >%luus %llu times, consider switching to WQ_UNBOUND\n",	1344	printk_deferred(KERN_WARNING "workqueue: %ps hogged CPU for >%luus %llu times, consider switching to WQ_UNBOUND\n",
1207	ent->func, wq_cpu_intensive_thresh_us,	1345	ent->func, wq_cpu_intensive_thresh_us,
1208	atomic64_read(&ent->cnt));	1346	atomic64_read(&ent->cnt));
@@ -1231,10 +1369,12 @@ restart: · 22 unmodified lines
1231		1369
1232	ent = &wci_ents[wci_nr_ents++];	1370	ent = &wci_ents[wci_nr_ents++];
1233	ent->func = func;	1371	ent->func = func;
1234	atomic64_set(&ent->cnt, 1);	1372	atomic64_set(&ent->cnt, 0);
1235	hash_add_rcu(wci_hash, &ent->hash_node, (unsigned long)func);	1373	hash_add_rcu(wci_hash, &ent->hash_node, (unsigned long)func);
1236		1374
1237	raw_spin_unlock(&wci_lock);	1375	raw_spin_unlock(&wci_lock);
		1376
		1377	goto restart;
1238	}	1378	}
1239		1379
1240	#else /* CONFIG_WQ_CPU_INTENSIVE_REPORT */	1380	#else /* CONFIG_WQ_CPU_INTENSIVE_REPORT */
@@ -1401,6 +1541,74 @@ work_func_t wq_worker_last_func(struct task_struct *task) · 160 unmodified lines
1401	return worker->last_func;	1541	return worker->last_func;
1402	}	1542	}
1403		1543
		1544	/**
		1545	* wq_node_nr_active - Determine wq_node_nr_active to use
		1546	* @wq: workqueue of interest
		1547	* @node: NUMA node, can be %NUMA_NO_NODE
		1548	*
		1549	* Determine wq_node_nr_active to use for @wq on @node. Returns:
		1550	*
		1551	* - %NULL for per-cpu workqueues as they don't need to use shared nr_active.
		1552	*
		1553	* - node_nr_active[nr_node_ids] if @node is %NUMA_NO_NODE.
		1554	*
		1555	* - Otherwise, node_nr_active[@node].
		1556	*/
		1557	static struct wq_node_nr_active wq_node_nr_active(struct workqueue_struct wq,
		1558	int node)
		1559	{
		1560	if (!(wq->flags & WQ_UNBOUND))
		1561	return NULL;
		1562
		1563	if (node == NUMA_NO_NODE)
		1564	node = nr_node_ids;
		1565
		1566	return wq->node_nr_active[node];
		1567	}
		1568
		1569	/**
		1570	* wq_update_node_max_active - Update per-node max_actives to use
		1571	* @wq: workqueue to update
		1572	* @off_cpu: CPU that's going down, -1 if a CPU is not going down
		1573	*
		1574	* Update @wq->node_nr_active[]->max. @wq must be unbound. max_active is
		1575	* distributed among nodes according to the proportions of numbers of online
		1576	* cpus. The result is always between @wq->min_active and max_active.
		1577	*/
		1578	static void wq_update_node_max_active(struct workqueue_struct *wq, int off_cpu)
		1579	{
		1580	struct cpumask *effective = unbound_effective_cpumask(wq);
		1581	int min_active = READ_ONCE(wq->min_active);
		1582	int max_active = READ_ONCE(wq->max_active);
		1583	int total_cpus, node;
		1584
		1585	lockdep_assert_held(&wq->mutex);
		1586
		1587	if (!wq_topo_initialized)
		1588	return;
		1589
		1590	if (off_cpu >= 0 && !cpumask_test_cpu(off_cpu, effective))
		1591	off_cpu = -1;
		1592
		1593	total_cpus = cpumask_weight_and(effective, cpu_online_mask);
		1594	if (off_cpu >= 0)
		1595	total_cpus--;
		1596
		1597	for_each_node(node) {
		1598	int node_cpus;
		1599
		1600	node_cpus = cpumask_weight_and(effective, cpumask_of_node(node));
		1601	if (off_cpu >= 0 && cpu_to_node(off_cpu) == node)
		1602	node_cpus--;
		1603
		1604	wq_node_nr_active(wq, node)->max =
		1605	clamp(DIV_ROUND_UP(max_active * node_cpus, total_cpus),
		1606	min_active, max_active);
		1607	}
		1608
		1609	wq_node_nr_active(wq, NUMA_NO_NODE)->max = min_active;
		1610	}
		1611
1404	/**	1612	/**
1405	* get_pwq - get an extra reference on the specified pool_workqueue	1613	* get_pwq - get an extra reference on the specified pool_workqueue
1406	* @pwq: pool_workqueue to get	1614	* @pwq: pool_workqueue to get
@@ -1453,24 +1661,336 @@ static void put_pwq_unlocked(struct pool_workqueue *pwq) · 46 unmodified lines
1453	}	1661	}
1454	}	1662	}
1455		1663
1456	static void pwq_activate_inactive_work(struct work_struct *work)	1664	static bool pwq_is_empty(struct pool_workqueue *pwq)
1457	{	1665	{
1458	struct pool_workqueue *pwq = get_work_pwq(work);	1666	return !pwq->nr_active && list_empty(&pwq->inactive_works);
		1667	}
1459		1668
		1669	static void __pwq_activate_work(struct pool_workqueue *pwq,
		1670	struct work_struct *work)
		1671	{
		1672	unsigned long *wdb = work_data_bits(work);
		1673
		1674	WARN_ON_ONCE(!(*wdb & WORK_STRUCT_INACTIVE));
1460	trace_workqueue_activate_work(work);	1675	trace_workqueue_activate_work(work);
1461	if (list_empty(&pwq->pool->worklist))	1676	if (list_empty(&pwq->pool->worklist))
1462	pwq->pool->watchdog_ts = jiffies;	1677	pwq->pool->watchdog_ts = jiffies;
1463	move_linked_works(work, &pwq->pool->worklist, NULL);	1678	move_linked_works(work, &pwq->pool->worklist, NULL);
1464	__clear_bit(WORK_STRUCT_INACTIVE_BIT, work_data_bits(work));	1679	__clear_bit(WORK_STRUCT_INACTIVE_BIT, wdb);
		1680	}
		1681
		1682	/**
		1683	* pwq_activate_work - Activate a work item if inactive
		1684	* @pwq: pool_workqueue @work belongs to
		1685	* @work: work item to activate
		1686	*
		1687	* Returns %true if activated. %false if already active.
		1688	*/
		1689	static bool pwq_activate_work(struct pool_workqueue *pwq,
		1690	struct work_struct *work)
		1691	{
		1692	struct worker_pool *pool = pwq->pool;
		1693	struct wq_node_nr_active *nna;
		1694
		1695	lockdep_assert_held(&pool->lock);
		1696
		1697	if (!(*work_data_bits(work) & WORK_STRUCT_INACTIVE))
		1698	return false;
		1699
		1700	nna = wq_node_nr_active(pwq->wq, pool->node);
		1701	if (nna)
		1702	atomic_inc(&nna->nr);
		1703
1465	pwq->nr_active++;	1704	pwq->nr_active++;
		1705	__pwq_activate_work(pwq, work);
		1706	return true;
		1707	}
		1708
		1709	static bool tryinc_node_nr_active(struct wq_node_nr_active *nna)
		1710	{
		1711	int max = READ_ONCE(nna->max);
		1712
		1713	while (true) {
		1714	int old, tmp;
		1715
		1716	old = atomic_read(&nna->nr);
		1717	if (old >= max)
		1718	return false;
		1719	tmp = atomic_cmpxchg_relaxed(&nna->nr, old, old + 1);
		1720	if (tmp == old)
		1721	return true;
		1722	}
1466	}	1723	}
1467		1724
1468	static void pwq_activate_first_inactive(struct pool_workqueue *pwq)	1725	/**
		1726	* pwq_tryinc_nr_active - Try to increment nr_active for a pwq
		1727	* @pwq: pool_workqueue of interest
		1728	* @fill: max_active may have increased, try to increase concurrency level
		1729	*
		1730	* Try to increment nr_active for @pwq. Returns %true if an nr_active count is
		1731	* successfully obtained. %false otherwise.
		1732	*/
		1733	static bool pwq_tryinc_nr_active(struct pool_workqueue *pwq, bool fill)
1469	{	1734	{
1470	struct work_struct *work = list_first_entry(&pwq->inactive_works,	1735	struct workqueue_struct *wq = pwq->wq;
1471	struct work_struct, entry);	1736	struct worker_pool *pool = pwq->pool;
		1737	struct wq_node_nr_active *nna = wq_node_nr_active(wq, pool->node);
		1738	bool obtained = false;
		1739
		1740	lockdep_assert_held(&pool->lock);
		1741
		1742	if (!nna) {
		1743	/* BH or per-cpu workqueue, pwq->nr_active is sufficient */
		1744	obtained = pwq->nr_active < READ_ONCE(wq->max_active);
		1745	goto out;
		1746	}
		1747
		1748	if (unlikely(pwq->plugged))
		1749	return false;
		1750
		1751	/*
		1752	* Unbound workqueue uses per-node shared nr_active $nna. If @pwq is
		1753	* already waiting on $nna, pwq_dec_nr_active() will maintain the
		1754	* concurrency level. Don't jump the line.
		1755	*
		1756	* We need to ignore the pending test after max_active has increased as
		1757	* pwq_dec_nr_active() can only maintain the concurrency level but not
		1758	* increase it. This is indicated by @fill.
		1759	*/
		1760	if (!list_empty(&pwq->pending_node) && likely(!fill))
		1761	goto out;
		1762
		1763	obtained = tryinc_node_nr_active(nna);
		1764	if (obtained)
		1765	goto out;
		1766
		1767	/*
		1768	* Lockless acquisition failed. Lock, add ourself to $nna->pending_pwqs
		1769	* and try again. The smp_mb() is paired with the implied memory barrier
		1770	* of atomic_dec_return() in pwq_dec_nr_active() to ensure that either
		1771	* we see the decremented $nna->nr or they see non-empty
		1772	* $nna->pending_pwqs.
		1773	*/
		1774	raw_spin_lock(&nna->lock);
		1775
		1776	if (list_empty(&pwq->pending_node))
		1777	list_add_tail(&pwq->pending_node, &nna->pending_pwqs);
		1778	else if (likely(!fill))
		1779	goto out_unlock;
		1780
		1781	smp_mb();
		1782
		1783	obtained = tryinc_node_nr_active(nna);
		1784
		1785	/*
		1786	* If @fill, @pwq might have already been pending. Being spuriously
		1787	* pending in cold paths doesn't affect anything. Let's leave it be.
		1788	*/
		1789	if (obtained && likely(!fill))
		1790	list_del_init(&pwq->pending_node);
		1791
		1792	out_unlock:
		1793	raw_spin_unlock(&nna->lock);
		1794	out:
		1795	if (obtained)
		1796	pwq->nr_active++;
		1797	return obtained;
		1798	}
		1799
		1800	/**
		1801	* pwq_activate_first_inactive - Activate the first inactive work item on a pwq
		1802	* @pwq: pool_workqueue of interest
		1803	* @fill: max_active may have increased, try to increase concurrency level
		1804	*
		1805	* Activate the first inactive work item of @pwq if available and allowed by
		1806	* max_active limit.
		1807	*
		1808	* Returns %true if an inactive work item has been activated. %false if no
		1809	* inactive work item is found or max_active limit is reached.
		1810	*/
		1811	static bool pwq_activate_first_inactive(struct pool_workqueue *pwq, bool fill)
		1812	{
		1813	struct work_struct *work =
		1814	list_first_entry_or_null(&pwq->inactive_works,
		1815	struct work_struct, entry);
		1816
		1817	if (work && pwq_tryinc_nr_active(pwq, fill)) {
		1818	__pwq_activate_work(pwq, work);
		1819	return true;
		1820	} else {
		1821	return false;
		1822	}
		1823	}
		1824
		1825	/**
		1826	* unplug_oldest_pwq - unplug the oldest pool_workqueue
		1827	* @wq: workqueue_struct where its oldest pwq is to be unplugged
		1828	*
		1829	* This function should only be called for ordered workqueues where only the
		1830	* oldest pwq is unplugged, the others are plugged to suspend execution to
		1831	* ensure proper work item ordering::
		1832	*
		1833	* dfl_pwq --------------+ [P] - plugged
		1834	* \|
		1835	* v
		1836	* pwqs -> A -> B [P] -> C [P] (newest)
		1837	* \| \| \|
		1838	* 1 3 5
		1839	* \| \| \|
		1840	* 2 4 6
		1841	*
		1842	* When the oldest pwq is drained and removed, this function should be called
		1843	* to unplug the next oldest one to start its work item execution. Note that
		1844	* pwq's are linked into wq->pwqs with the oldest first, so the first one in
		1845	* the list is the oldest.
		1846	*/
		1847	static void unplug_oldest_pwq(struct workqueue_struct *wq)
		1848	{
		1849	struct pool_workqueue *pwq;
		1850
		1851	lockdep_assert_held(&wq->mutex);
		1852
		1853	/* Caller should make sure that pwqs isn't empty before calling */
		1854	pwq = list_first_entry_or_null(&wq->pwqs, struct pool_workqueue,
		1855	pwqs_node);
		1856	raw_spin_lock_irq(&pwq->pool->lock);
		1857	if (pwq->plugged) {
		1858	pwq->plugged = false;
		1859	if (pwq_activate_first_inactive(pwq, true))
		1860	kick_pool(pwq->pool);
		1861	}
		1862	raw_spin_unlock_irq(&pwq->pool->lock);
		1863	}
		1864
		1865	/**
		1866	* node_activate_pending_pwq - Activate a pending pwq on a wq_node_nr_active
		1867	* @nna: wq_node_nr_active to activate a pending pwq for
		1868	* @caller_pool: worker_pool the caller is locking
		1869	*
		1870	* Activate a pwq in @nna->pending_pwqs. Called with @caller_pool locked.
		1871	* @caller_pool may be unlocked and relocked to lock other worker_pools.
		1872	*/
		1873	static void node_activate_pending_pwq(struct wq_node_nr_active *nna,
		1874	struct worker_pool *caller_pool)
		1875	{
		1876	struct worker_pool *locked_pool = caller_pool;
		1877	struct pool_workqueue *pwq;
		1878	struct work_struct *work;
		1879
		1880	lockdep_assert_held(&caller_pool->lock);
		1881
		1882	raw_spin_lock(&nna->lock);
		1883	retry:
		1884	pwq = list_first_entry_or_null(&nna->pending_pwqs,
		1885	struct pool_workqueue, pending_node);
		1886	if (!pwq)
		1887	goto out_unlock;
		1888
		1889	/*
		1890	* If @pwq is for a different pool than @locked_pool, we need to lock
		1891	* @pwq->pool->lock. Let's trylock first. If unsuccessful, do the unlock
		1892	* / lock dance. For that, we also need to release @nna->lock as it's
		1893	* nested inside pool locks.
		1894	*/
		1895	if (pwq->pool != locked_pool) {
		1896	raw_spin_unlock(&locked_pool->lock);
		1897	locked_pool = pwq->pool;
		1898	if (!raw_spin_trylock(&locked_pool->lock)) {
		1899	raw_spin_unlock(&nna->lock);
		1900	raw_spin_lock(&locked_pool->lock);
		1901	raw_spin_lock(&nna->lock);
		1902	goto retry;
		1903	}
		1904	}
		1905
		1906	/*
		1907	* $pwq may not have any inactive work items due to e.g. cancellations.
		1908	* Drop it from pending_pwqs and see if there's another one.
		1909	*/
		1910	work = list_first_entry_or_null(&pwq->inactive_works,
		1911	struct work_struct, entry);
		1912	if (!work) {
		1913	list_del_init(&pwq->pending_node);
		1914	goto retry;
		1915	}
		1916
		1917	/*
		1918	* Acquire an nr_active count and activate the inactive work item. If
		1919	* $pwq still has inactive work items, rotate it to the end of the
		1920	* pending_pwqs so that we round-robin through them. This means that
		1921	* inactive work items are not activated in queueing order which is fine
		1922	* given that there has never been any ordering across different pwqs.
		1923	*/
		1924	if (likely(tryinc_node_nr_active(nna))) {
		1925	pwq->nr_active++;
		1926	__pwq_activate_work(pwq, work);
		1927
		1928	if (list_empty(&pwq->inactive_works))
		1929	list_del_init(&pwq->pending_node);
		1930	else
		1931	list_move_tail(&pwq->pending_node, &nna->pending_pwqs);
		1932
		1933	/* if activating a foreign pool, make sure it's running */
		1934	if (pwq->pool != caller_pool)
		1935	kick_pool(pwq->pool);
		1936	}
		1937
		1938	out_unlock:
		1939	raw_spin_unlock(&nna->lock);
		1940	if (locked_pool != caller_pool) {
		1941	raw_spin_unlock(&locked_pool->lock);
		1942	raw_spin_lock(&caller_pool->lock);
		1943	}
		1944	}
		1945
		1946	/**
		1947	* pwq_dec_nr_active - Retire an active count
		1948	* @pwq: pool_workqueue of interest
		1949	*
		1950	* Decrement @pwq's nr_active and try to activate the first inactive work item.
		1951	* For unbound workqueues, this function may temporarily drop @pwq->pool->lock.
		1952	*/
		1953	static void pwq_dec_nr_active(struct pool_workqueue *pwq)
		1954	{
		1955	struct worker_pool *pool = pwq->pool;
		1956	struct wq_node_nr_active *nna = wq_node_nr_active(pwq->wq, pool->node);
		1957
		1958	lockdep_assert_held(&pool->lock);
		1959
		1960	/*
		1961	* @pwq->nr_active should be decremented for both percpu and unbound
		1962	* workqueues.
		1963	*/
		1964	pwq->nr_active--;
		1965
		1966	/*
		1967	* For a percpu workqueue, it's simple. Just need to kick the first
		1968	* inactive work item on @pwq itself.
		1969	*/
		1970	if (!nna) {
		1971	pwq_activate_first_inactive(pwq, false);
		1972	return;
		1973	}
		1974
		1975	/*
		1976	* If @pwq is for an unbound workqueue, it's more complicated because
		1977	* multiple pwqs and pools may be sharing the nr_active count. When a
		1978	* pwq needs to wait for an nr_active count, it puts itself on
		1979	* $nna->pending_pwqs. The following atomic_dec_return()'s implied
		1980	* memory barrier is paired with smp_mb() in pwq_tryinc_nr_active() to
		1981	* guarantee that either we see non-empty pending_pwqs or they see
		1982	* decremented $nna->nr.
		1983	*
		1984	* $nna->max may change as CPUs come online/offline and @pwq->wq's
		1985	* max_active gets updated. However, it is guaranteed to be equal to or
		1986	* larger than @pwq->wq->min_active which is above zero unless freezing.
		1987	* This maintains the forward progress guarantee.
		1988	*/
		1989	if (atomic_dec_return(&nna->nr) >= READ_ONCE(nna->max))
		1990	return;
1472		1991
1473	pwq_activate_inactive_work(work);	1992	if (!list_empty(&nna->pending_pwqs))
		1993	node_activate_pending_pwq(nna, pool);
1474	}	1994	}
1475		1995
1476	/**	1996	/**
@@ -1481,6 +2001,11 @@ static void pwq_activate_first_inactive(struct pool_workqueue *pwq) · 4 unmodified lines
1481	* A work either has completed or is removed from pending queue,	2001	* A work either has completed or is removed from pending queue,
1482	* decrement nr_in_flight of its pwq and handle workqueue flushing.	2002	* decrement nr_in_flight of its pwq and handle workqueue flushing.
1483	*	2003	*
		2004	* NOTE:
		2005	* For unbound workqueues, this function may temporarily drop @pwq->pool->lock
		2006	* and thus should be called after all other state updates for the in-flight
		2007	* work item is complete.
		2008	*
1484	* CONTEXT:	2009	* CONTEXT:
1485	* raw_spin_lock_irq(pool->lock).	2010	* raw_spin_lock_irq(pool->lock).
1486	*/	2011	*/
@@ -1488,14 +2013,8 @@ static void pwq_dec_nr_in_flight(struct pool_workqueue *pwq, unsigned long work_ · 1 unmodified line
1488	{	2013	{
1489	int color = get_work_color(work_data);	2014	int color = get_work_color(work_data);
1490		2015
1491	if (!(work_data & WORK_STRUCT_INACTIVE)) {	2016	if (!(work_data & WORK_STRUCT_INACTIVE))
1492	pwq->nr_active--;	2017	pwq_dec_nr_active(pwq);
1493	if (!list_empty(&pwq->inactive_works)) {
1494	/* one down, submit an inactive one */
1495	if (pwq->nr_active < pwq->max_active)
1496	pwq_activate_first_inactive(pwq);
1497	}
1498	}
1499		2018
1500	pwq->nr_in_flight[color]--;	2019	pwq->nr_in_flight[color]--;
1501		2020
@@ -1523,8 +2042,8 @@ out_put: · 21 unmodified lines
1523	/**	2042	/**
1524	* try_to_grab_pending - steal work item from worklist and disable irq	2043	* try_to_grab_pending - steal work item from worklist and disable irq
1525	* @work: work item to steal	2044	* @work: work item to steal
1526	* @is_dwork: @work is a delayed_work	2045	* @cflags: %WORK_CANCEL_ flags
1527	* @flags: place to store irq state	2046	* @irq_flags: place to store irq state
1528	*	2047	*
1529	* Try to grab PENDING bit of @work. This function can handle @work in any	2048	* Try to grab PENDING bit of @work. This function can handle @work in any
1530	* stable state - idle, on timer or on worklist.	2049	* stable state - idle, on timer or on worklist.
@@ -1546,20 +2065,20 @@ out_put: · 15 unmodified lines
1546	* irqsafe, ensures that we return -EAGAIN for finite short period of time.	2065	* irqsafe, ensures that we return -EAGAIN for finite short period of time.
1547	*	2066	*
1548	* On successful return, >= 0, irq is disabled and the caller is	2067	* On successful return, >= 0, irq is disabled and the caller is
1549	* responsible for releasing it using local_irq_restore(*@flags).	2068	* responsible for releasing it using local_irq_restore(*@irq_flags).
1550	*	2069	*
1551	* This function is safe to call from any context including IRQ handler.	2070	* This function is safe to call from any context including IRQ handler.
1552	*/	2071	*/
1553	static int try_to_grab_pending(struct work_struct *work, bool is_dwork,	2072	static int try_to_grab_pending(struct work_struct *work, u32 cflags,
1554	unsigned long *flags)	2073	unsigned long *irq_flags)
1555	{	2074	{
1556	struct worker_pool *pool;	2075	struct worker_pool *pool;
1557	struct pool_workqueue *pwq;	2076	struct pool_workqueue *pwq;
1558		2077
1559	local_irq_save(*flags);	2078	local_irq_save(*irq_flags);
1560		2079
1561	/* try to steal the timer if it exists */	2080	/* try to steal the timer if it exists */
1562	if (is_dwork) {	2081	if (cflags & WORK_CANCEL_DELAYED) {
1563	struct delayed_work *dwork = to_delayed_work(work);	2082	struct delayed_work *dwork = to_delayed_work(work);
1564		2083
1565	/*	2084	/*
@@ -1595,6 +2114,8 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork, · 29 unmodified lines
1595	*/	2114	*/
1596	pwq = get_work_pwq(work);	2115	pwq = get_work_pwq(work);
1597	if (pwq && pwq->pool == pool) {	2116	if (pwq && pwq->pool == pool) {
		2117	unsigned long work_data;
		2118
1598	debug_work_deactivate(work);	2119	debug_work_deactivate(work);
1599		2120
1600	/*	2121	/*
@@ -1608,14 +2129,19 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork, · 7 unmodified lines
1608	* management later on and cause stall. Make sure the work	2129	* management later on and cause stall. Make sure the work
1609	* item is activated before grabbing.	2130	* item is activated before grabbing.
1610	*/	2131	*/
1611	if (*work_data_bits(work) & WORK_STRUCT_INACTIVE)	2132	pwq_activate_work(pwq, work);
1612	pwq_activate_inactive_work(work);
1613		2133
1614	list_del_init(&work->entry);	2134	list_del_init(&work->entry);
1615	pwq_dec_nr_in_flight(pwq, *work_data_bits(work));
1616		2135
1617	/* work->data points to pwq iff queued, point to pool */	2136	/*
1618	set_work_pool_and_keep_pending(work, pool->id);	2137	* work->data points to pwq iff queued. Let's point to pool. As
		2138	* this destroys work->data needed by the next step, stash it.
		2139	*/
		2140	work_data = *work_data_bits(work);
		2141	set_work_pool_and_keep_pending(work, pool->id, 0);
		2142
		2143	/* must be the last step, see the function comment */
		2144	pwq_dec_nr_in_flight(pwq, work_data);
1619		2145
1620	raw_spin_unlock(&pool->lock);	2146	raw_spin_unlock(&pool->lock);
1621	rcu_read_unlock();	2147	rcu_read_unlock();
@@ -1624,13 +2150,82 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork, · 2 unmodified lines
1624	raw_spin_unlock(&pool->lock);	2150	raw_spin_unlock(&pool->lock);
1625	fail:	2151	fail:
1626	rcu_read_unlock();	2152	rcu_read_unlock();
1627	local_irq_restore(*flags);	2153	local_irq_restore(*irq_flags);
1628	if (work_is_canceling(work))	2154	if (work_is_canceling(work))
1629	return -ENOENT;	2155	return -ENOENT;
1630	cpu_relax();	2156	cpu_relax();
1631	return -EAGAIN;	2157	return -EAGAIN;
1632	}	2158	}
1633		2159
		2160	struct cwt_wait {
		2161	wait_queue_entry_t wait;
		2162	struct work_struct *work;
		2163	};
		2164
		2165	static int cwt_wakefn(wait_queue_entry_t wait, unsigned mode, int sync, void key)
		2166	{
		2167	struct cwt_wait *cwait = container_of(wait, struct cwt_wait, wait);
		2168
		2169	if (cwait->work != key)
		2170	return 0;
		2171	return autoremove_wake_function(wait, mode, sync, key);
		2172	}
		2173
		2174	/**
		2175	* work_grab_pending - steal work item from worklist and disable irq
		2176	* @work: work item to steal
		2177	* @cflags: %WORK_CANCEL_ flags
		2178	* @irq_flags: place to store IRQ state
		2179	*
		2180	* Grab PENDING bit of @work. @work can be in any stable state - idle, on timer
		2181	* or on worklist.
		2182	*
		2183	* Must be called in process context. IRQ is disabled on return with IRQ state
		2184	* stored in *@irq_flags. The caller is responsible for re-enabling it using
		2185	* local_irq_restore().
		2186	*
		2187	* Returns %true if @work was pending. %false if idle.
		2188	*/
		2189	static bool work_grab_pending(struct work_struct *work, u32 cflags,
		2190	unsigned long *irq_flags)
		2191	{
		2192	struct cwt_wait cwait;
		2193	int ret;
		2194
		2195	might_sleep();
		2196	repeat:
		2197	ret = try_to_grab_pending(work, cflags, irq_flags);
		2198	if (likely(ret >= 0))
		2199	return ret;
		2200	if (ret != -ENOENT)
		2201	goto repeat;
		2202
		2203	/*
		2204	* Someone is already canceling. Wait for it to finish. flush_work()
		2205	* doesn't work for PREEMPT_NONE because we may get woken up between
		2206	* @work's completion and the other canceling task resuming and clearing
		2207	* CANCELING - flush_work() will return false immediately as @work is no
		2208	* longer busy, try_to_grab_pending() will return -ENOENT as @work is
		2209	* still being canceled and the other canceling task won't be able to
		2210	* clear CANCELING as we're hogging the CPU.
		2211	*
		2212	* Let's wait for completion using a waitqueue. As this may lead to the
		2213	* thundering herd problem, use a custom wake function which matches
		2214	* @work along with exclusive wait and wakeup.
		2215	*/
		2216	init_wait(&cwait.wait);
		2217	cwait.wait.func = cwt_wakefn;
		2218	cwait.work = work;
		2219
		2220	prepare_to_wait_exclusive(&wq_cancel_waitq, &cwait.wait,
		2221	TASK_UNINTERRUPTIBLE);
		2222	if (work_is_canceling(work))
		2223	schedule();
		2224	finish_wait(&wq_cancel_waitq, &cwait.wait);
		2225
		2226	goto repeat;
		2227	}
		2228
1634	/**	2229	/**
1635	* insert_work - insert a work into a pool	2230	* insert_work - insert a work into a pool
1636	* @pwq: pwq @work belongs to	2231	* @pwq: pwq @work belongs to
@@ -1718,7 +2313,6 @@ static void __queue_work(int cpu, struct workqueue_struct *wq, · 81 unmodified lines
1718	*/	2313	*/
1719	lockdep_assert_irqs_disabled();	2314	lockdep_assert_irqs_disabled();
1720		2315
1721
1722	/*	2316	/*
1723	* For a draining wq, only works from the same workqueue are	2317	* For a draining wq, only works from the same workqueue are
1724	* allowed. The __WQ_DESTROYING helps to spot the issue that	2318	* allowed. The __WQ_DESTROYING helps to spot the issue that
@@ -1793,12 +2387,16 @@ retry: · 68 unmodified lines
1793	pwq->nr_in_flight[pwq->work_color]++;	2387	pwq->nr_in_flight[pwq->work_color]++;
1794	work_flags = work_color_to_flags(pwq->work_color);	2388	work_flags = work_color_to_flags(pwq->work_color);
1795		2389
1796	if (likely(pwq->nr_active < pwq->max_active)) {	2390	/*
		2391	* Limit the number of concurrently active work items to max_active.
		2392	* @work must also queue behind existing inactive work items to maintain
		2393	* ordering when max_active changes. See wq_adjust_max_active().
		2394	*/
		2395	if (list_empty(&pwq->inactive_works) && pwq_tryinc_nr_active(pwq, false)) {
1797	if (list_empty(&pool->worklist))	2396	if (list_empty(&pool->worklist))
1798	pool->watchdog_ts = jiffies;	2397	pool->watchdog_ts = jiffies;
1799		2398
1800	trace_workqueue_activate_work(work);	2399	trace_workqueue_activate_work(work);
1801	pwq->nr_active++;
1802	insert_work(pwq, work, &pool->worklist, work_flags);	2400	insert_work(pwq, work, &pool->worklist, work_flags);
1803	kick_pool(pool);	2401	kick_pool(pool);
1804	} else {	2402	} else {
@@ -1829,16 +2427,16 @@ bool queue_work_on(int cpu, struct workqueue_struct *wq, · 24 unmodified lines
1829	struct work_struct *work)	2427	struct work_struct *work)
1830	{	2428	{
1831	bool ret = false;	2429	bool ret = false;
1832	unsigned long flags;	2430	unsigned long irq_flags;
1833		2431
1834	local_irq_save(flags);	2432	local_irq_save(irq_flags);
1835		2433
1836	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {	2434	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {
1837	__queue_work(cpu, wq, work);	2435	__queue_work(cpu, wq, work);
1838	ret = true;	2436	ret = true;
1839	}	2437	}
1840		2438
1841	local_irq_restore(flags);	2439	local_irq_restore(irq_flags);
1842	return ret;	2440	return ret;
1843	}	2441	}
1844	EXPORT_SYMBOL(queue_work_on);	2442	EXPORT_SYMBOL(queue_work_on);
@@ -1895,7 +2493,7 @@ static int select_numa_node_cpu(int node) · 50 unmodified lines
1895	bool queue_work_node(int node, struct workqueue_struct *wq,	2493	bool queue_work_node(int node, struct workqueue_struct *wq,
1896	struct work_struct *work)	2494	struct work_struct *work)
1897	{	2495	{
1898	unsigned long flags;	2496	unsigned long irq_flags;
1899	bool ret = false;	2497	bool ret = false;
1900		2498
1901	/*	2499	/*
@@ -1909,7 +2507,7 @@ bool queue_work_node(int node, struct workqueue_struct *wq, · 7 unmodified lines
1909	*/	2507	*/
1910	WARN_ON_ONCE(!(wq->flags & WQ_UNBOUND));	2508	WARN_ON_ONCE(!(wq->flags & WQ_UNBOUND));
1911		2509
1912	local_irq_save(flags);	2510	local_irq_save(irq_flags);
1913		2511
1914	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {	2512	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {
1915	int cpu = select_numa_node_cpu(node);	2513	int cpu = select_numa_node_cpu(node);
@@ -1918,7 +2516,7 @@ bool queue_work_node(int node, struct workqueue_struct *wq, · 2 unmodified lines
1918	ret = true;	2516	ret = true;
1919	}	2517	}
1920		2518
1921	local_irq_restore(flags);	2519	local_irq_restore(irq_flags);
1922	return ret;	2520	return ret;
1923	}	2521	}
1924	EXPORT_SYMBOL_GPL(queue_work_node);	2522	EXPORT_SYMBOL_GPL(queue_work_node);
@@ -1958,10 +2556,18 @@ static void __queue_delayed_work(int cpu, struct workqueue_struct *wq, · 33 unmodified lines
1958	dwork->cpu = cpu;	2556	dwork->cpu = cpu;
1959	timer->expires = jiffies + delay;	2557	timer->expires = jiffies + delay;
1960		2558
1961	if (unlikely(cpu != WORK_CPU_UNBOUND))	2559	if (housekeeping_enabled(HK_TYPE_TIMER)) {
		2560	/* If the current cpu is a housekeeping cpu, use it. */
		2561	cpu = smp_processor_id();
		2562	if (!housekeeping_test_cpu(cpu, HK_TYPE_TIMER))
		2563	cpu = housekeeping_any_cpu(HK_TYPE_TIMER);
1962	add_timer_on(timer, cpu);	2564	add_timer_on(timer, cpu);
1963	else	2565	} else {
1964	add_timer(timer);	2566	if (likely(cpu == WORK_CPU_UNBOUND))
		2567	add_timer(timer);
		2568	else
		2569	add_timer_on(timer, cpu);
		2570	}
1965	}	2571	}
1966		2572
1967	/**	2573	/**
@@ -1980,17 +2586,17 @@ bool queue_delayed_work_on(int cpu, struct workqueue_struct *wq, · 12 unmodified lines
1980	{	2586	{
1981	struct work_struct *work = &dwork->work;	2587	struct work_struct *work = &dwork->work;
1982	bool ret = false;	2588	bool ret = false;
1983	unsigned long flags;	2589	unsigned long irq_flags;
1984		2590
1985	/* read the comment in __queue_work() */	2591	/* read the comment in __queue_work() */
1986	local_irq_save(flags);	2592	local_irq_save(irq_flags);
1987		2593
1988	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {	2594	if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) {
1989	__queue_delayed_work(cpu, wq, dwork, delay);	2595	__queue_delayed_work(cpu, wq, dwork, delay);
1990	ret = true;	2596	ret = true;
1991	}	2597	}
1992		2598
1993	local_irq_restore(flags);	2599	local_irq_restore(irq_flags);
1994	return ret;	2600	return ret;
1995	}	2601	}
1996	EXPORT_SYMBOL(queue_delayed_work_on);	2602	EXPORT_SYMBOL(queue_delayed_work_on);
@@ -2016,16 +2622,17 @@ EXPORT_SYMBOL(queue_delayed_work_on); · 19 unmodified lines
2016	bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq,	2622	bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq,
2017	struct delayed_work *dwork, unsigned long delay)	2623	struct delayed_work *dwork, unsigned long delay)
2018	{	2624	{
2019	unsigned long flags;	2625	unsigned long irq_flags;
2020	int ret;	2626	int ret;
2021		2627
2022	do {	2628	do {
2023	ret = try_to_grab_pending(&dwork->work, true, &flags);	2629	ret = try_to_grab_pending(&dwork->work, WORK_CANCEL_DELAYED,
		2630	&irq_flags);
2024	} while (unlikely(ret == -EAGAIN));	2631	} while (unlikely(ret == -EAGAIN));
2025		2632
2026	if (likely(ret >= 0)) {	2633	if (likely(ret >= 0)) {
2027	__queue_delayed_work(cpu, wq, dwork, delay);	2634	__queue_delayed_work(cpu, wq, dwork, delay);
2028	local_irq_restore(flags);	2635	local_irq_restore(irq_flags);
2029	}	2636	}
2030		2637
2031	/* -ENOENT from try_to_grab_pending() becomes %true */	2638	/* -ENOENT from try_to_grab_pending() becomes %true */
@@ -2100,19 +2707,21 @@ static cpumask_t pool_allowed_cpus(struct worker_pool pool) · 68 unmodified lines
2100	* cpu-[un]hotplugs.	2707	* cpu-[un]hotplugs.
2101	*/	2708	*/
2102	static void worker_attach_to_pool(struct worker *worker,	2709	static void worker_attach_to_pool(struct worker *worker,
2103	struct worker_pool *pool)	2710	struct worker_pool *pool)
2104	{	2711	{
2105	mutex_lock(&wq_pool_attach_mutex);	2712	mutex_lock(&wq_pool_attach_mutex);
2106		2713
2107	/*	2714	/*
2108	* The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains	2715	* The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains stable
2109	* stable across this function. See the comments above the flag	2716	* across this function. See the comments above the flag definition for
2110	* definition for details.	2717	* details. BH workers are, while per-CPU, always DISASSOCIATED.
2111	*/	2718	*/
2112	if (pool->flags & POOL_DISASSOCIATED)	2719	if (pool->flags & POOL_DISASSOCIATED) {
2113	worker->flags \|= WORKER_UNBOUND;	2720	worker->flags \|= WORKER_UNBOUND;
2114	else	2721	} else {
		2722	WARN_ON_ONCE(pool->flags & POOL_BH);
2115	kthread_set_per_cpu(worker->task, pool->cpu);	2723	kthread_set_per_cpu(worker->task, pool->cpu);
		2724	}
2116		2725
2117	if (worker->rescue_wq)	2726	if (worker->rescue_wq)
2118	set_cpus_allowed_ptr(worker->task, pool_allowed_cpus(pool));	2727	set_cpus_allowed_ptr(worker->task, pool_allowed_cpus(pool));
@@ -2136,6 +2745,9 @@ static void worker_detach_from_pool(struct worker *worker) · 17 unmodified lines
2136	struct worker_pool *pool = worker->pool;	2745	struct worker_pool *pool = worker->pool;
2137	struct completion *detach_completion = NULL;	2746	struct completion *detach_completion = NULL;
2138		2747
		2748	/* there is one permanent BH worker per CPU which should never detach */
		2749	WARN_ON_ONCE(pool->flags & POOL_BH);
		2750
2139	mutex_lock(&wq_pool_attach_mutex);	2751	mutex_lock(&wq_pool_attach_mutex);
2140		2752
2141	kthread_set_per_cpu(worker->task, -1);	2753	kthread_set_per_cpu(worker->task, -1);
@@ -2187,27 +2799,29 @@ static struct worker create_worker(struct worker_pool pool) · 45 unmodified lines
2187		2799
2188	worker->id = id;	2800	worker->id = id;
2189		2801
2190	if (pool->cpu >= 0)	2802	if (!(pool->flags & POOL_BH)) {
2191	snprintf(id_buf, sizeof(id_buf), "%d:%d%s", pool->cpu, id,	2803	if (pool->cpu >= 0)
2192	pool->attrs->nice < 0 ? "H" : "");	2804	snprintf(id_buf, sizeof(id_buf), "%d:%d%s", pool->cpu, id,
2193	else	2805	pool->attrs->nice < 0 ? "H" : "");
2194	snprintf(id_buf, sizeof(id_buf), "u%d:%d", pool->id, id);	2806	else
2195		2807	snprintf(id_buf, sizeof(id_buf), "u%d:%d", pool->id, id);
2196	worker->task = kthread_create_on_node(worker_thread, worker, pool->node,	2808
2197	"kworker/%s", id_buf);	2809	worker->task = kthread_create_on_node(worker_thread, worker,
2198	if (IS_ERR(worker->task)) {	2810	pool->node, "kworker/%s", id_buf);
2199	if (PTR_ERR(worker->task) == -EINTR) {	2811	if (IS_ERR(worker->task)) {
2200	pr_err("workqueue: Interrupted when creating a worker thread \"kworker/%s\"\n",	2812	if (PTR_ERR(worker->task) == -EINTR) {
2201	id_buf);	2813	pr_err("workqueue: Interrupted when creating a worker thread \"kworker/%s\"\n",
2202	} else {	2814	id_buf);
2203	pr_err_once("workqueue: Failed to create a worker thread: %pe",	2815	} else {
2204	worker->task);	2816	pr_err_once("workqueue: Failed to create a worker thread: %pe",
		2817	worker->task);
		2818	}
		2819	goto fail;
2205	}	2820	}
2206	goto fail;
2207	}
2208		2821
2209	set_user_nice(worker->task, pool->attrs->nice);	2822	set_user_nice(worker->task, pool->attrs->nice);
2210	kthread_bind_mask(worker->task, pool_allowed_cpus(pool));	2823	kthread_bind_mask(worker->task, pool_allowed_cpus(pool));
		2824	}
2211		2825
2212	/* successful, attach the worker to the pool */	2826	/* successful, attach the worker to the pool */
2213	worker_attach_to_pool(worker, pool);	2827	worker_attach_to_pool(worker, pool);
@@ -2217,14 +2831,14 @@ static struct worker create_worker(struct worker_pool pool) · 3 unmodified lines
2217		2831
2218	worker->pool->nr_workers++;	2832	worker->pool->nr_workers++;
2219	worker_enter_idle(worker);	2833	worker_enter_idle(worker);
2220	kick_pool(pool);
2221		2834
2222	/*	2835	/*
2223	* @worker is waiting on a completion in kthread() and will trigger hung	2836	* @worker is waiting on a completion in kthread() and will trigger hung
2224	* check if not woken up soon. As kick_pool() might not have waken it	2837	* check if not woken up soon. As kick_pool() is noop if @pool is empty,
2225	* up, wake it up explicitly once more.	2838	* wake it up explicitly.
2226	*/	2839	*/
2227	wake_up_process(worker->task);	2840	if (worker->task)
		2841	wake_up_process(worker->task);
2228		2842
2229	raw_spin_unlock_irq(&pool->lock);	2843	raw_spin_unlock_irq(&pool->lock);
2230		2844
@@ -2543,6 +3157,8 @@ __acquires(&pool->lock) · 312 unmodified lines
2543	struct pool_workqueue *pwq = get_work_pwq(work);	3157	struct pool_workqueue *pwq = get_work_pwq(work);
2544	struct worker_pool *pool = worker->pool;	3158	struct worker_pool *pool = worker->pool;
2545	unsigned long work_data;	3159	unsigned long work_data;
		3160	int lockdep_start_depth, rcu_start_depth;
		3161	bool bh_draining = pool->flags & POOL_BH_DRAINING;
2546	#ifdef CONFIG_LOCKDEP	3162	#ifdef CONFIG_LOCKDEP
2547	/*	3163	/*
2548	* It is permissible to free the struct work_struct from	3164	* It is permissible to free the struct work_struct from
@@ -2565,7 +3181,8 @@ __acquires(&pool->lock) · 16 unmodified lines
2565	worker->current_work = work;	3181	worker->current_work = work;
2566	worker->current_func = work->func;	3182	worker->current_func = work->func;
2567	worker->current_pwq = pwq;	3183	worker->current_pwq = pwq;
2568	worker->current_at = worker->task->se.sum_exec_runtime;	3184	if (worker->task)
		3185	worker->current_at = worker->task->se.sum_exec_runtime;
2569	work_data = *work_data_bits(work);	3186	work_data = *work_data_bits(work);
2570	worker->current_color = get_work_color(work_data);	3187	worker->current_color = get_work_color(work_data);
2571		3188
@@ -2600,12 +3217,16 @@ __acquires(&pool->lock) · 28 unmodified lines
2600	* PENDING and queued state changes happen together while IRQ is	3217	* PENDING and queued state changes happen together while IRQ is
2601	* disabled.	3218	* disabled.
2602	*/	3219	*/
2603	set_work_pool_and_clear_pending(work, pool->id);	3220	set_work_pool_and_clear_pending(work, pool->id, 0);
2604		3221
2605	pwq->stats[PWQ_STAT_STARTED]++;	3222	pwq->stats[PWQ_STAT_STARTED]++;
2606	raw_spin_unlock_irq(&pool->lock);	3223	raw_spin_unlock_irq(&pool->lock);
2607		3224
2608	lock_map_acquire(&pwq->wq->lockdep_map);	3225	rcu_start_depth = rcu_preempt_depth();
		3226	lockdep_start_depth = lockdep_depth(current);
		3227	/* see drain_dead_softirq_workfn() */
		3228	if (!bh_draining)
		3229	lock_map_acquire(&pwq->wq->lockdep_map);
2609	lock_map_acquire(&lockdep_map);	3230	lock_map_acquire(&lockdep_map);
2610	/*	3231	/*
2611	* Strictly speaking we should mark the invariant state without holding	3232	* Strictly speaking we should mark the invariant state without holding
@@ -2638,12 +3259,17 @@ __acquires(&pool->lock) · 26 unmodified lines
2638	trace_workqueue_execute_end(work, worker->current_func);	3259	trace_workqueue_execute_end(work, worker->current_func);
2639	pwq->stats[PWQ_STAT_COMPLETED]++;	3260	pwq->stats[PWQ_STAT_COMPLETED]++;
2640	lock_map_release(&lockdep_map);	3261	lock_map_release(&lockdep_map);
2641	lock_map_release(&pwq->wq->lockdep_map);	3262	if (!bh_draining)
		3263	lock_map_release(&pwq->wq->lockdep_map);
2642		3264
2643	if (unlikely(in_atomic() \|\| lockdep_depth(current) > 0)) {	3265	if (unlikely((worker->task && in_atomic()) \|\|
2644	pr_err("BUG: workqueue leaked lock or atomic: %s/0x%08x/%d\n"	3266	lockdep_depth(current) != lockdep_start_depth \|\|
2645	" last function: %ps\n",	3267	rcu_preempt_depth() != rcu_start_depth)) {
2646	current->comm, preempt_count(), task_pid_nr(current),	3268	pr_err("BUG: workqueue leaked atomic, lock or RCU: %s[%d]\n"
		3269	" preempt=0x%08x lock=%d->%d RCU=%d->%d workfn=%ps\n",
		3270	current->comm, task_pid_nr(current), preempt_count(),
		3271	lockdep_start_depth, lockdep_depth(current),
		3272	rcu_start_depth, rcu_preempt_depth(),
2647	worker->current_func);	3273	worker->current_func);
2648	debug_show_held_locks(current);	3274	debug_show_held_locks(current);
2649	dump_stack();	3275	dump_stack();
@@ -2657,7 +3283,8 @@ __acquires(&pool->lock) · 7 unmodified lines
2657	* stop_machine. At the same time, report a quiescent RCU state so	3283	* stop_machine. At the same time, report a quiescent RCU state so
2658	* the same condition doesn't freeze RCU.	3284	* the same condition doesn't freeze RCU.
2659	*/	3285	*/
2660	cond_resched();	3286	if (worker->task)
		3287	cond_resched();
2661		3288
2662	raw_spin_lock_irq(&pool->lock);	3289	raw_spin_lock_irq(&pool->lock);
2663		3290
@@ -2677,6 +3304,8 @@ __acquires(&pool->lock) · 13 unmodified lines
2677	worker->current_func = NULL;	3304	worker->current_func = NULL;
2678	worker->current_pwq = NULL;	3305	worker->current_pwq = NULL;
2679	worker->current_color = INT_MAX;	3306	worker->current_color = INT_MAX;
		3307
		3308	/* must be the last step, see the function comment */
2680	pwq_dec_nr_in_flight(pwq, work_data);	3309	pwq_dec_nr_in_flight(pwq, work_data);
2681	}	3310	}
2682		3311
@@ -2938,6 +3567,139 @@ repeat: · 255 unmodified lines
2938	goto repeat;	3567	goto repeat;
2939	}	3568	}
2940		3569
		3570	static void bh_worker(struct worker *worker)
		3571	{
		3572	struct worker_pool *pool = worker->pool;
		3573	int nr_restarts = BH_WORKER_RESTARTS;
		3574	unsigned long end = jiffies + BH_WORKER_JIFFIES;
		3575
		3576	raw_spin_lock_irq(&pool->lock);
		3577	worker_leave_idle(worker);
		3578
		3579	/*
		3580	* This function follows the structure of worker_thread(). See there for
		3581	* explanations on each step.
		3582	*/
		3583	if (!need_more_worker(pool))
		3584	goto done;
		3585
		3586	WARN_ON_ONCE(!list_empty(&worker->scheduled));
		3587	worker_clr_flags(worker, WORKER_PREP \| WORKER_REBOUND);
		3588
		3589	do {
		3590	struct work_struct *work =
		3591	list_first_entry(&pool->worklist,
		3592	struct work_struct, entry);
		3593
		3594	if (assign_work(work, worker, NULL))
		3595	process_scheduled_works(worker);
		3596	} while (keep_working(pool) &&
		3597	--nr_restarts && time_before(jiffies, end));
		3598
		3599	worker_set_flags(worker, WORKER_PREP);
		3600	done:
		3601	worker_enter_idle(worker);
		3602	kick_pool(pool);
		3603	raw_spin_unlock_irq(&pool->lock);
		3604	}
		3605
		3606	/*
		3607	* TODO: Convert all tasklet users to workqueue and use softirq directly.
		3608	*
		3609	* This is currently called from tasklet[_hi]action() and thus is also called
		3610	* whenever there are tasklets to run. Let's do an early exit if there's nothing
		3611	* queued. Once conversion from tasklet is complete, the need_more_worker() test
		3612	* can be dropped.
		3613	*
		3614	* After full conversion, we'll add worker->softirq_action, directly use the
		3615	* softirq action and obtain the worker pointer from the softirq_action pointer.
		3616	*/
		3617	void workqueue_softirq_action(bool highpri)
		3618	{
		3619	struct worker_pool *pool =
		3620	&per_cpu(bh_worker_pools, smp_processor_id())[highpri];
		3621	if (need_more_worker(pool))
		3622	bh_worker(list_first_entry(&pool->workers, struct worker, node));
		3623	}
		3624
		3625	struct wq_drain_dead_softirq_work {
		3626	struct work_struct work;
		3627	struct worker_pool *pool;
		3628	struct completion done;
		3629	};
		3630
		3631	static void drain_dead_softirq_workfn(struct work_struct *work)
		3632	{
		3633	struct wq_drain_dead_softirq_work *dead_work =
		3634	container_of(work, struct wq_drain_dead_softirq_work, work);
		3635	struct worker_pool *pool = dead_work->pool;
		3636	bool repeat;
		3637
		3638	/*
		3639	* @pool's CPU is dead and we want to execute its still pending work
		3640	* items from this BH work item which is running on a different CPU. As
		3641	* its CPU is dead, @pool can't be kicked and, as work execution path
		3642	* will be nested, a lockdep annotation needs to be suppressed. Mark
		3643	* @pool with %POOL_BH_DRAINING for the special treatments.
		3644	*/
		3645	raw_spin_lock_irq(&pool->lock);
		3646	pool->flags \|= POOL_BH_DRAINING;
		3647	raw_spin_unlock_irq(&pool->lock);
		3648
		3649	bh_worker(list_first_entry(&pool->workers, struct worker, node));
		3650
		3651	raw_spin_lock_irq(&pool->lock);
		3652	pool->flags &= ~POOL_BH_DRAINING;
		3653	repeat = need_more_worker(pool);
		3654	raw_spin_unlock_irq(&pool->lock);
		3655
		3656	/*
		3657	* bh_worker() might hit consecutive execution limit and bail. If there
		3658	* still are pending work items, reschedule self and return so that we
		3659	* don't hog this CPU's BH.
		3660	*/
		3661	if (repeat) {
		3662	if (pool->attrs->nice == HIGHPRI_NICE_LEVEL)
		3663	queue_work(system_bh_highpri_wq, work);
		3664	else
		3665	queue_work(system_bh_wq, work);
		3666	} else {
		3667	complete(&dead_work->done);
		3668	}
		3669	}
		3670
		3671	/*
		3672	* @cpu is dead. Drain the remaining BH work items on the current CPU. It's
		3673	* possible to allocate dead_work per CPU and avoid flushing. However, then we
		3674	* have to worry about draining overlapping with CPU coming back online or
		3675	* nesting (one CPU's dead_work queued on another CPU which is also dead and so
		3676	* on). Let's keep it simple and drain them synchronously. These are BH work
		3677	* items which shouldn't be requeued on the same pool. Shouldn't take long.
		3678	*/
		3679	void workqueue_softirq_dead(unsigned int cpu)
		3680	{
		3681	int i;
		3682
		3683	for (i = 0; i < NR_STD_WORKER_POOLS; i++) {
		3684	struct worker_pool *pool = &per_cpu(bh_worker_pools, cpu)[i];
		3685	struct wq_drain_dead_softirq_work dead_work;
		3686
		3687	if (!need_more_worker(pool))
		3688	continue;
		3689
		3690	INIT_WORK(&dead_work.work, drain_dead_softirq_workfn);
		3691	dead_work.pool = pool;
		3692	init_completion(&dead_work.done);
		3693
		3694	if (pool->attrs->nice == HIGHPRI_NICE_LEVEL)
		3695	queue_work(system_bh_highpri_wq, &dead_work.work);
		3696	else
		3697	queue_work(system_bh_wq, &dead_work.work);
		3698
		3699	wait_for_completion(&dead_work.done);
		3700	}
		3701	}
		3702
2941	/**	3703	/**
2942	* check_flush_dependency - check for flush dependency sanity	3704	* check_flush_dependency - check for flush dependency sanity
2943	* @target_wq: workqueue being flushed	3705	* @target_wq: workqueue being flushed
@@ -3010,6 +3772,7 @@ static void insert_wq_barrier(struct pool_workqueue *pwq, · 66 unmodified lines
3010	struct wq_barrier *barr,	3772	struct wq_barrier *barr,
3011	struct work_struct target, struct worker worker)	3773	struct work_struct target, struct worker worker)
3012	{	3774	{
		3775	static __maybe_unused struct lock_class_key bh_key, thr_key;
3013	unsigned int work_flags = 0;	3776	unsigned int work_flags = 0;
3014	unsigned int work_color;	3777	unsigned int work_color;
3015	struct list_head *head;	3778	struct list_head *head;
@@ -3019,15 +3782,20 @@ static void insert_wq_barrier(struct pool_workqueue *pwq, · 3 unmodified lines
3019	* as we know for sure that this will not trigger any of the	3782	* as we know for sure that this will not trigger any of the
3020	* checks and call back into the fixup functions where we	3783	* checks and call back into the fixup functions where we
3021	* might deadlock.	3784	* might deadlock.
		3785	*
		3786	* BH and threaded workqueues need separate lockdep keys to avoid
		3787	* spuriously triggering "inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W}
		3788	* usage".
3022	*/	3789	*/
3023	INIT_WORK_ONSTACK(&barr->work, wq_barrier_func);	3790	INIT_WORK_ONSTACK_KEY(&barr->work, wq_barrier_func,
		3791	(pwq->wq->flags & WQ_BH) ? &bh_key : &thr_key);
3024	__set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(&barr->work));	3792	__set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(&barr->work));
3025		3793
3026	init_completion_map(&barr->done, &target->lockdep_map);	3794	init_completion_map(&barr->done, &target->lockdep_map);
3027		3795
3028	barr->task = current;	3796	barr->task = current;
3029		3797
3030	/* The barrier work item does not participate in pwq->nr_active. */	3798	/* The barrier work item does not participate in nr_active. */
3031	work_flags \|= WORK_STRUCT_INACTIVE;	3799	work_flags \|= WORK_STRUCT_INACTIVE;
3032		3800
3033	/*	3801	/*
@@ -3124,6 +3892,35 @@ static bool flush_workqueue_prep_pwqs(struct workqueue_struct *wq, · 90 unmodified lines
3124	return wait;	3892	return wait;
3125	}	3893	}
3126		3894
		3895	static void touch_wq_lockdep_map(struct workqueue_struct *wq)
		3896	{
		3897	#ifdef CONFIG_LOCKDEP
		3898	if (wq->flags & WQ_BH)
		3899	local_bh_disable();
		3900
		3901	lock_map_acquire(&wq->lockdep_map);
		3902	lock_map_release(&wq->lockdep_map);
		3903
		3904	if (wq->flags & WQ_BH)
		3905	local_bh_enable();
		3906	#endif
		3907	}
		3908
		3909	static void touch_work_lockdep_map(struct work_struct *work,
		3910	struct workqueue_struct *wq)
		3911	{
		3912	#ifdef CONFIG_LOCKDEP
		3913	if (wq->flags & WQ_BH)
		3914	local_bh_disable();
		3915
		3916	lock_map_acquire(&work->lockdep_map);
		3917	lock_map_release(&work->lockdep_map);
		3918
		3919	if (wq->flags & WQ_BH)
		3920	local_bh_enable();
		3921	#endif
		3922	}
		3923
3127	/**	3924	/**
3128	* __flush_workqueue - ensure that any scheduled work has run to completion.	3925	* __flush_workqueue - ensure that any scheduled work has run to completion.
3129	* @wq: workqueue to flush	3926	* @wq: workqueue to flush
@@ -3143,8 +3940,7 @@ void __flush_workqueue(struct workqueue_struct *wq) · 13 unmodified lines
3143	if (WARN_ON(!wq_online))	3940	if (WARN_ON(!wq_online))
3144	return;	3941	return;
3145		3942
3146	lock_map_acquire(&wq->lockdep_map);	3943	touch_wq_lockdep_map(wq);
3147	lock_map_release(&wq->lockdep_map);
3148		3944
3149	mutex_lock(&wq->mutex);	3945	mutex_lock(&wq->mutex);
3150		3946
@@ -3316,7 +4112,7 @@ reflush: · 165 unmodified lines
3316	bool drained;	4112	bool drained;
3317		4113
3318	raw_spin_lock_irq(&pwq->pool->lock);	4114	raw_spin_lock_irq(&pwq->pool->lock);
3319	drained = !pwq->nr_active && list_empty(&pwq->inactive_works);	4115	drained = pwq_is_empty(pwq);
3320	raw_spin_unlock_irq(&pwq->pool->lock);	4116	raw_spin_unlock_irq(&pwq->pool->lock);
3321		4117
3322	if (drained)	4118	if (drained)
@@ -3343,6 +4139,7 @@ static bool start_flush_work(struct work_struct work, struct wq_barrier barr, · 20 unmodified lines
3343	struct worker *worker = NULL;	4139	struct worker *worker = NULL;
3344	struct worker_pool *pool;	4140	struct worker_pool *pool;
3345	struct pool_workqueue *pwq;	4141	struct pool_workqueue *pwq;
		4142	struct workqueue_struct *wq;
3346		4143
3347	might_sleep();	4144	might_sleep();
3348		4145
@@ -3366,11 +4163,14 @@ static bool start_flush_work(struct work_struct work, struct wq_barrier barr, · 17 unmodified lines
3366	pwq = worker->current_pwq;	4163	pwq = worker->current_pwq;
3367	}	4164	}
3368		4165
3369	check_flush_dependency(pwq->wq, work);	4166	wq = pwq->wq;
		4167	check_flush_dependency(wq, work);
3370		4168
3371	insert_wq_barrier(pwq, barr, work, worker);	4169	insert_wq_barrier(pwq, barr, work, worker);
3372	raw_spin_unlock_irq(&pool->lock);	4170	raw_spin_unlock_irq(&pool->lock);
3373		4171
		4172	touch_work_lockdep_map(work, wq);
		4173
3374	/*	4174	/*
3375	* Force a lock recursion deadlock when using flush_work() inside a	4175	* Force a lock recursion deadlock when using flush_work() inside a
3376	* single-threaded or rescuer equipped workqueue.	4176	* single-threaded or rescuer equipped workqueue.
@@ -3380,11 +4180,9 @@ static bool start_flush_work(struct work_struct work, struct wq_barrier barr, · 3 unmodified lines
3380	* workqueues the deadlock happens when the rescuer stalls, blocking	4180	* workqueues the deadlock happens when the rescuer stalls, blocking
3381	* forward progress.	4181	* forward progress.
3382	*/	4182	*/
3383	if (!from_cancel &&	4183	if (!from_cancel && (wq->saved_max_active == 1 \|\| wq->rescuer))
3384	(pwq->wq->saved_max_active == 1 \|\| pwq->wq->rescuer)) {	4184	touch_wq_lockdep_map(wq);
3385	lock_map_acquire(&pwq->wq->lockdep_map);	4185
3386	lock_map_release(&pwq->wq->lockdep_map);
3387	}
3388	rcu_read_unlock();	4186	rcu_read_unlock();
3389	return true;	4187	return true;
3390	already_gone:	4188	already_gone:
@@ -3393,146 +4191,41 @@ already_gone: · 2 unmodified lines
3393	return false;	4191	return false;
3394	}	4192	}
3395		4193
3396	static bool __flush_work(struct work_struct *work, bool from_cancel)	4194	static bool __flush_work(struct work_struct *work, bool from_cancel)
3397	{	4195	{
3398	struct wq_barrier barr;	4196	struct wq_barrier barr;
3399
3400	if (WARN_ON(!wq_online))
3401	return false;
3402
3403	if (WARN_ON(!work->func))
3404	return false;
3405
3406	lock_map_acquire(&work->lockdep_map);
3407	lock_map_release(&work->lockdep_map);
3408
3409	if (start_flush_work(work, &barr, from_cancel)) {
3410	wait_for_completion(&barr.done);
3411	destroy_work_on_stack(&barr.work);
3412	return true;
3413	} else {
3414	return false;
3415	}
3416	}
3417
3418	/**
3419	* flush_work - wait for a work to finish executing the last queueing instance
3420	* @work: the work to flush
3421	*
3422	* Wait until @work has finished execution. @work is guaranteed to be idle
3423	* on return if it hasn't been requeued since flush started.
3424	*
3425	* Return:
3426	* %true if flush_work() waited for the work to finish execution,
3427	* %false if it was already idle.
3428	*/
3429	bool flush_work(struct work_struct *work)
3430	{
3431	return __flush_work(work, false);
3432	}
3433	EXPORT_SYMBOL_GPL(flush_work);
3434
3435	struct cwt_wait {
3436	wait_queue_entry_t wait;
3437	struct work_struct *work;
3438	};
3439
3440	static int cwt_wakefn(wait_queue_entry_t wait, unsigned mode, int sync, void key)
3441	{
3442	struct cwt_wait *cwait = container_of(wait, struct cwt_wait, wait);
3443
3444	if (cwait->work != key)
3445	return 0;
3446	return autoremove_wake_function(wait, mode, sync, key);
3447	}
3448
3449	static bool __cancel_work_timer(struct work_struct *work, bool is_dwork)
3450	{
3451	static DECLARE_WAIT_QUEUE_HEAD(cancel_waitq);
3452	unsigned long flags;
3453	int ret;
3454
3455	do {
3456	ret = try_to_grab_pending(work, is_dwork, &flags);
3457	/*
3458	* If someone else is already canceling, wait for it to
3459	* finish. flush_work() doesn't work for PREEMPT_NONE
3460	* because we may get scheduled between @work's completion
3461	* and the other canceling task resuming and clearing
3462	* CANCELING - flush_work() will return false immediately
3463	* as @work is no longer busy, try_to_grab_pending() will
3464	* return -ENOENT as @work is still being canceled and the
3465	* other canceling task won't be able to clear CANCELING as
3466	* we're hogging the CPU.
3467	*
3468	* Let's wait for completion using a waitqueue. As this
3469	* may lead to the thundering herd problem, use a custom
3470	* wake function which matches @work along with exclusive
3471	* wait and wakeup.
3472	*/
3473	if (unlikely(ret == -ENOENT)) {
3474	struct cwt_wait cwait;
3475
3476	init_wait(&cwait.wait);
3477	cwait.wait.func = cwt_wakefn;
3478	cwait.work = work;
3479
3480	prepare_to_wait_exclusive(&cancel_waitq, &cwait.wait,
3481	TASK_UNINTERRUPTIBLE);
3482	if (work_is_canceling(work))
3483	schedule();
3484	finish_wait(&cancel_waitq, &cwait.wait);
3485	}
3486	} while (unlikely(ret < 0));
3487
3488	/* tell other tasks trying to grab @work to back off */
3489	mark_work_canceling(work);
3490	local_irq_restore(flags);
3491
3492	/*
3493	* This allows canceling during early boot. We know that @work
3494	* isn't executing.
3495	*/
3496	if (wq_online)
3497	__flush_work(work, true);
3498		4197
3499	clear_work_data(work);	4198	if (WARN_ON(!wq_online))
		4199	return false;
3500		4200
3501	/*	4201	if (WARN_ON(!work->func))
3502	* Paired with prepare_to_wait() above so that either	4202	return false;
3503	* waitqueue_active() is visible here or !work_is_canceling() is
3504	* visible there.
3505	*/
3506	smp_mb();
3507	if (waitqueue_active(&cancel_waitq))
3508	__wake_up(&cancel_waitq, TASK_NORMAL, 1, work);
3509		4203
3510	return ret;	4204	if (start_flush_work(work, &barr, from_cancel)) {
		4205	wait_for_completion(&barr.done);
		4206	destroy_work_on_stack(&barr.work);
		4207	return true;
		4208	} else {
		4209	return false;
		4210	}
3511	}	4211	}
3512		4212
3513	/**	4213	/**
3514	* cancel_work_sync - cancel a work and wait for it to finish	4214	* flush_work - wait for a work to finish executing the last queueing instance
3515	* @work: the work to cancel	4215	* @work: the work to flush
3516	*
3517	* Cancel @work and wait for its execution to finish. This function
3518	* can be used even if the work re-queues itself or migrates to
3519	* another workqueue. On return from this function, @work is
3520	* guaranteed to be not pending or executing on any CPU.
3521	*
3522	* cancel_work_sync(&delayed_work->work) must not be used for
3523	* delayed_work's. Use cancel_delayed_work_sync() instead.
3524	*	4216	*
3525	* The caller must ensure that the workqueue on which @work was last	4217	* Wait until @work has finished execution. @work is guaranteed to be idle
3526	* queued can't be destroyed before this function returns.	4218	* on return if it hasn't been requeued since flush started.
3527	*	4219	*
3528	* Return:	4220	* Return:
3529	* %true if @work was pending, %false otherwise.	4221	* %true if flush_work() waited for the work to finish execution,
		4222	* %false if it was already idle.
3530	*/	4223	*/
3531	bool cancel_work_sync(struct work_struct *work)	4224	bool flush_work(struct work_struct *work)
3532	{	4225	{
3533	return __cancel_work_timer(work, false);	4226	return __flush_work(work, false);
3534	}	4227	}
3535	EXPORT_SYMBOL_GPL(cancel_work_sync);	4228	EXPORT_SYMBOL_GPL(flush_work);
3536		4229
3537	/**	4230	/**
3538	* flush_delayed_work - wait for a dwork to finish executing the last queueing	4231	* flush_delayed_work - wait for a dwork to finish executing the last queueing
@@ -3576,20 +4269,50 @@ bool flush_rcu_work(struct rcu_work *rwork) · 37 unmodified lines
3576	}	4269	}
3577	EXPORT_SYMBOL(flush_rcu_work);	4270	EXPORT_SYMBOL(flush_rcu_work);
3578		4271
3579	static bool __cancel_work(struct work_struct *work, bool is_dwork)	4272	static bool __cancel_work(struct work_struct *work, u32 cflags)
3580	{	4273	{
3581	unsigned long flags;	4274	unsigned long irq_flags;
3582	int ret;	4275	int ret;
3583		4276
3584	do {	4277	do {
3585	ret = try_to_grab_pending(work, is_dwork, &flags);	4278	ret = try_to_grab_pending(work, cflags, &irq_flags);
3586	} while (unlikely(ret == -EAGAIN));	4279	} while (unlikely(ret == -EAGAIN));
3587		4280
3588	if (unlikely(ret < 0))	4281	if (unlikely(ret < 0))
3589	return false;	4282	return false;
3590		4283
3591	set_work_pool_and_clear_pending(work, get_work_pool_id(work));	4284	set_work_pool_and_clear_pending(work, get_work_pool_id(work), 0);
3592	local_irq_restore(flags);	4285	local_irq_restore(irq_flags);
		4286	return ret;
		4287	}
		4288
		4289	static bool __cancel_work_sync(struct work_struct *work, u32 cflags)
		4290	{
		4291	unsigned long irq_flags;
		4292	bool ret;
		4293
		4294	/* claim @work and tell other tasks trying to grab @work to back off */
		4295	ret = work_grab_pending(work, cflags, &irq_flags);
		4296	mark_work_canceling(work);
		4297	local_irq_restore(irq_flags);
		4298
		4299	/*
		4300	* Skip __flush_work() during early boot when we know that @work isn't
		4301	* executing. This allows canceling during early boot.
		4302	*/
		4303	if (wq_online)
		4304	__flush_work(work, true);
		4305
		4306	/*
		4307	* smp_mb() at the end of set_work_pool_and_clear_pending() is paired
		4308	* with prepare_to_wait() above so that either waitqueue_active() is
		4309	* visible here or !work_is_canceling() is visible there.
		4310	*/
		4311	set_work_pool_and_clear_pending(work, WORK_OFFQ_POOL_NONE, 0);
		4312
		4313	if (waitqueue_active(&wq_cancel_waitq))
		4314	__wake_up(&wq_cancel_waitq, TASK_NORMAL, 1, work);
		4315
3593	return ret;	4316	return ret;
3594	}	4317	}
3595		4318
@@ -3598,10 +4321,34 @@ static bool __cancel_work(struct work_struct *work, bool is_dwork) · 2 unmodified lines
3598	*/	4321	*/
3599	bool cancel_work(struct work_struct *work)	4322	bool cancel_work(struct work_struct *work)
3600	{	4323	{
3601	return __cancel_work(work, false);	4324	return __cancel_work(work, 0);
3602	}	4325	}
3603	EXPORT_SYMBOL(cancel_work);	4326	EXPORT_SYMBOL(cancel_work);
3604		4327
		4328	/**
		4329	* cancel_work_sync - cancel a work and wait for it to finish
		4330	* @work: the work to cancel
		4331	*
		4332	* Cancel @work and wait for its execution to finish. This function
		4333	* can be used even if the work re-queues itself or migrates to
		4334	* another workqueue. On return from this function, @work is
		4335	* guaranteed to be not pending or executing on any CPU.
		4336	*
		4337	* cancel_work_sync(&delayed_work->work) must not be used for
		4338	* delayed_work's. Use cancel_delayed_work_sync() instead.
		4339	*
		4340	* The caller must ensure that the workqueue on which @work was last
		4341	* queued can't be destroyed before this function returns.
		4342	*
		4343	* Return:
		4344	* %true if @work was pending, %false otherwise.
		4345	*/
		4346	bool cancel_work_sync(struct work_struct *work)
		4347	{
		4348	return __cancel_work_sync(work, 0);
		4349	}
		4350	EXPORT_SYMBOL_GPL(cancel_work_sync);
		4351
3605	/**	4352	/**
3606	* cancel_delayed_work - cancel a delayed work	4353	* cancel_delayed_work - cancel a delayed work
3607	* @dwork: delayed_work to cancel	4354	* @dwork: delayed_work to cancel
@@ -3620,7 +4367,7 @@ EXPORT_SYMBOL(cancel_work); · 12 unmodified lines
3620	*/	4367	*/
3621	bool cancel_delayed_work(struct delayed_work *dwork)	4368	bool cancel_delayed_work(struct delayed_work *dwork)
3622	{	4369	{
3623	return __cancel_work(&dwork->work, true);	4370	return __cancel_work(&dwork->work, WORK_CANCEL_DELAYED);
3624	}	4371	}
3625	EXPORT_SYMBOL(cancel_delayed_work);	4372	EXPORT_SYMBOL(cancel_delayed_work);
3626		4373
@@ -3635,7 +4382,7 @@ EXPORT_SYMBOL(cancel_delayed_work); · 8 unmodified lines
3635	*/	4382	*/
3636	bool cancel_delayed_work_sync(struct delayed_work *dwork)	4383	bool cancel_delayed_work_sync(struct delayed_work *dwork)
3637	{	4384	{
3638	return __cancel_work_timer(&dwork->work, true);	4385	return __cancel_work_sync(&dwork->work, WORK_CANCEL_DELAYED);
3639	}	4386	}
3640	EXPORT_SYMBOL(cancel_delayed_work_sync);	4387	EXPORT_SYMBOL(cancel_delayed_work_sync);
3641		4388
@@ -3927,11 +4674,66 @@ static void wq_free_lockdep(struct workqueue_struct *wq) · 285 unmodified lines
3927	}	4674	}
3928	#endif	4675	#endif
3929		4676
		4677	static void free_node_nr_active(struct wq_node_nr_active **nna_ar)
		4678	{
		4679	int node;
		4680
		4681	for_each_node(node) {
		4682	kfree(nna_ar[node]);
		4683	nna_ar[node] = NULL;
		4684	}
		4685
		4686	kfree(nna_ar[nr_node_ids]);
		4687	nna_ar[nr_node_ids] = NULL;
		4688	}
		4689
		4690	static void init_node_nr_active(struct wq_node_nr_active *nna)
		4691	{
		4692	nna->max = WQ_DFL_MIN_ACTIVE;
		4693	atomic_set(&nna->nr, 0);
		4694	raw_spin_lock_init(&nna->lock);
		4695	INIT_LIST_HEAD(&nna->pending_pwqs);
		4696	}
		4697
		4698	/*
		4699	* Each node's nr_active counter will be accessed mostly from its own node and
		4700	* should be allocated in the node.
		4701	*/
		4702	static int alloc_node_nr_active(struct wq_node_nr_active **nna_ar)
		4703	{
		4704	struct wq_node_nr_active *nna;
		4705	int node;
		4706
		4707	for_each_node(node) {
		4708	nna = kzalloc_node(sizeof(*nna), GFP_KERNEL, node);
		4709	if (!nna)
		4710	goto err_free;
		4711	init_node_nr_active(nna);
		4712	nna_ar[node] = nna;
		4713	}
		4714
		4715	/* [nr_node_ids] is used as the fallback */
		4716	nna = kzalloc_node(sizeof(*nna), GFP_KERNEL, NUMA_NO_NODE);
		4717	if (!nna)
		4718	goto err_free;
		4719	init_node_nr_active(nna);
		4720	nna_ar[nr_node_ids] = nna;
		4721
		4722	return 0;
		4723
		4724	err_free:
		4725	free_node_nr_active(nna_ar);
		4726	return -ENOMEM;
		4727	}
		4728
3930	static void rcu_free_wq(struct rcu_head *rcu)	4729	static void rcu_free_wq(struct rcu_head *rcu)
3931	{	4730	{
3932	struct workqueue_struct *wq =	4731	struct workqueue_struct *wq =
3933	container_of(rcu, struct workqueue_struct, rcu);	4732	container_of(rcu, struct workqueue_struct, rcu);
3934		4733
		4734	if (wq->flags & WQ_UNBOUND)
		4735	free_node_nr_active(wq->node_nr_active);
		4736
3935	wq_free_lockdep(wq);	4737	wq_free_lockdep(wq);
3936	free_percpu(wq->cpu_pwq);	4738	free_percpu(wq->cpu_pwq);
3937	free_workqueue_attrs(wq->unbound_attrs);	4739	free_workqueue_attrs(wq->unbound_attrs);
@@ -4121,6 +4923,13 @@ static void pwq_release_workfn(struct kthread_work *work) · 183 unmodified lines
4121	mutex_lock(&wq->mutex);	4923	mutex_lock(&wq->mutex);
4122	list_del_rcu(&pwq->pwqs_node);	4924	list_del_rcu(&pwq->pwqs_node);
4123	is_last = list_empty(&wq->pwqs);	4925	is_last = list_empty(&wq->pwqs);
		4926
		4927	/*
		4928	* For ordered workqueue with a plugged dfl_pwq, restart it now.
		4929	*/
		4930	if (!is_last && (wq->flags & __WQ_ORDERED))
		4931	unplug_oldest_pwq(wq);
		4932
4124	mutex_unlock(&wq->mutex);	4933	mutex_unlock(&wq->mutex);
4125	}	4934	}
4126		4935
@@ -4130,6 +4939,15 @@ static void pwq_release_workfn(struct kthread_work *work) · 3 unmodified lines
4130	mutex_unlock(&wq_pool_mutex);	4939	mutex_unlock(&wq_pool_mutex);
4131	}	4940	}
4132		4941
		4942	if (!list_empty(&pwq->pending_node)) {
		4943	struct wq_node_nr_active *nna =
		4944	wq_node_nr_active(pwq->wq, pwq->pool->node);
		4945
		4946	raw_spin_lock_irq(&nna->lock);
		4947	list_del_init(&pwq->pending_node);
		4948	raw_spin_unlock_irq(&nna->lock);
		4949	}
		4950
4133	call_rcu(&pwq->rcu, rcu_free_pwq);	4951	call_rcu(&pwq->rcu, rcu_free_pwq);
4134		4952
4135	/*	4953	/*
@@ -4142,55 +4960,11 @@ static void pwq_release_workfn(struct kthread_work *work) · 6 unmodified lines
4142	}	4960	}
4143	}	4961	}
4144		4962
4145	/**
4146	* pwq_adjust_max_active - update a pwq's max_active to the current setting
4147	* @pwq: target pool_workqueue
4148	*
4149	* If @pwq isn't freezing, set @pwq->max_active to the associated
4150	* workqueue's saved_max_active and activate inactive work items
4151	* accordingly. If @pwq is freezing, clear @pwq->max_active to zero.
4152	*/
4153	static void pwq_adjust_max_active(struct pool_workqueue *pwq)
4154	{
4155	struct workqueue_struct *wq = pwq->wq;
4156	bool freezable = wq->flags & WQ_FREEZABLE;
4157	unsigned long flags;
4158
4159	/* for @wq->saved_max_active */
4160	lockdep_assert_held(&wq->mutex);
4161
4162	/* fast exit for non-freezable wqs */
4163	if (!freezable && pwq->max_active == wq->saved_max_active)
4164	return;
4165
4166	/* this function can be called during early boot w/ irq disabled */
4167	raw_spin_lock_irqsave(&pwq->pool->lock, flags);
4168
4169	/*
4170	* During [un]freezing, the caller is responsible for ensuring that
4171	* this function is called at least once after @workqueue_freezing
4172	* is updated and visible.
4173	*/
4174	if (!freezable \|\| !workqueue_freezing) {
4175	pwq->max_active = wq->saved_max_active;
4176
4177	while (!list_empty(&pwq->inactive_works) &&
4178	pwq->nr_active < pwq->max_active)
4179	pwq_activate_first_inactive(pwq);
4180
4181	kick_pool(pwq->pool);
4182	} else {
4183	pwq->max_active = 0;
4184	}
4185
4186	raw_spin_unlock_irqrestore(&pwq->pool->lock, flags);
4187	}
4188
4189	/* initialize newly allocated @pwq which is associated with @wq and @pool */	4963	/* initialize newly allocated @pwq which is associated with @wq and @pool */
4190	static void init_pwq(struct pool_workqueue pwq, struct workqueue_struct wq,	4964	static void init_pwq(struct pool_workqueue pwq, struct workqueue_struct wq,
4191	struct worker_pool *pool)	4965	struct worker_pool *pool)
4192	{	4966	{
4193	BUG_ON((unsigned long)pwq & WORK_STRUCT_FLAG_MASK);	4967	BUG_ON((unsigned long)pwq & ~WORK_STRUCT_PWQ_MASK);
4194		4968
4195	memset(pwq, 0, sizeof(*pwq));	4969	memset(pwq, 0, sizeof(*pwq));
4196		4970
@@ -4199,6 +4973,7 @@ static void init_pwq(struct pool_workqueue pwq, struct workqueue_struct wq, · 2 unmodified lines
4199	pwq->flush_color = -1;	4973	pwq->flush_color = -1;
4200	pwq->refcnt = 1;	4974	pwq->refcnt = 1;
4201	INIT_LIST_HEAD(&pwq->inactive_works);	4975	INIT_LIST_HEAD(&pwq->inactive_works);
		4976	INIT_LIST_HEAD(&pwq->pending_node);
4202	INIT_LIST_HEAD(&pwq->pwqs_node);	4977	INIT_LIST_HEAD(&pwq->pwqs_node);
4203	INIT_LIST_HEAD(&pwq->mayday_node);	4978	INIT_LIST_HEAD(&pwq->mayday_node);
4204	kthread_init_work(&pwq->release_work, pwq_release_workfn);	4979	kthread_init_work(&pwq->release_work, pwq_release_workfn);
@@ -4218,11 +4993,8 @@ static void link_pwq(struct pool_workqueue *pwq) · 13 unmodified lines
4218	/* set the matching work_color */	4993	/* set the matching work_color */
4219	pwq->work_color = wq->work_color;	4994	pwq->work_color = wq->work_color;
4220		4995
4221	/* sync max_active to the current setting */
4222	pwq_adjust_max_active(pwq);
4223
4224	/* link in @pwq */	4996	/* link in @pwq */
4225	list_add_rcu(&pwq->pwqs_node, &wq->pwqs);	4997	list_add_tail_rcu(&pwq->pwqs_node, &wq->pwqs);
4226	}	4998	}
4227		4999
4228	/* obtain a pool matching @attr and create a pwq associating the pool and @wq */	5000	/* obtain a pool matching @attr and create a pwq associating the pool and @wq */
@@ -4289,10 +5061,11 @@ static void wq_calc_pod_cpumask(struct workqueue_attrs *attrs, int cpu, · 60 unmodified lines
4289	"possible intersect\n");	5061	"possible intersect\n");
4290	}	5062	}
4291		5063
4292	/* install @pwq into @wq's cpu_pwq and return the old pwq */	5064	/* install @pwq into @wq and return the old pwq, @cpu < 0 for dfl_pwq */
4293	static struct pool_workqueue install_unbound_pwq(struct workqueue_struct wq,	5065	static struct pool_workqueue install_unbound_pwq(struct workqueue_struct wq,
4294	int cpu, struct pool_workqueue *pwq)	5066	int cpu, struct pool_workqueue *pwq)
4295	{	5067	{
		5068	struct pool_workqueue __rcu **slot = unbound_pwq_slot(wq, cpu);
4296	struct pool_workqueue *old_pwq;	5069	struct pool_workqueue *old_pwq;
4297		5070
4298	lockdep_assert_held(&wq_pool_mutex);	5071	lockdep_assert_held(&wq_pool_mutex);
@@ -4301,8 +5074,8 @@ static struct pool_workqueue install_unbound_pwq(struct workqueue_struct wq, · 2 unmodified lines
4301	/* link_pwq() can handle duplicate calls */	5074	/* link_pwq() can handle duplicate calls */
4302	link_pwq(pwq);	5075	link_pwq(pwq);
4303		5076
4304	old_pwq = rcu_access_pointer(*per_cpu_ptr(wq->cpu_pwq, cpu));	5077	old_pwq = rcu_access_pointer(*slot);
4305	rcu_assign_pointer(*per_cpu_ptr(wq->cpu_pwq, cpu), pwq);	5078	rcu_assign_pointer(*slot, pwq);
4306	return old_pwq;	5079	return old_pwq;
4307	}	5080	}
4308		5081
@@ -4383,6 +5156,15 @@ apply_wqattrs_prepare(struct workqueue_struct *wq, · 74 unmodified lines
4383	cpumask_copy(new_attrs->__pod_cpumask, new_attrs->cpumask);	5156	cpumask_copy(new_attrs->__pod_cpumask, new_attrs->cpumask);
4384	ctx->attrs = new_attrs;	5157	ctx->attrs = new_attrs;
4385		5158
		5159	/*
		5160	* For initialized ordered workqueues, there should only be one pwq
		5161	* (dfl_pwq). Set the plugged flag of ctx->dfl_pwq to suspend execution
		5162	* of newly queued work items until execution of older work items in
		5163	* the old pwq's have completed.
		5164	*/
		5165	if ((wq->flags & __WQ_ORDERED) && !list_empty(&wq->pwqs))
		5166	ctx->dfl_pwq->plugged = true;
		5167
4386	ctx->wq = wq;	5168	ctx->wq = wq;
4387	return ctx;	5169	return ctx;
4388		5170
@@ -4402,14 +5184,19 @@ static void apply_wqattrs_commit(struct apply_wqattrs_ctx *ctx) · 13 unmodified lines
4402		5184
4403	copy_workqueue_attrs(ctx->wq->unbound_attrs, ctx->attrs);	5185	copy_workqueue_attrs(ctx->wq->unbound_attrs, ctx->attrs);
4404		5186
4405	/* save the previous pwq and install the new one */	5187	/* save the previous pwqs and install the new ones */
4406	for_each_possible_cpu(cpu)	5188	for_each_possible_cpu(cpu)
4407	ctx->pwq_tbl[cpu] = install_unbound_pwq(ctx->wq, cpu,	5189	ctx->pwq_tbl[cpu] = install_unbound_pwq(ctx->wq, cpu,
4408	ctx->pwq_tbl[cpu]);	5190	ctx->pwq_tbl[cpu]);
		5191	ctx->dfl_pwq = install_unbound_pwq(ctx->wq, -1, ctx->dfl_pwq);
		5192
		5193	/* update node_nr_active->max */
		5194	wq_update_node_max_active(ctx->wq, -1);
4409		5195
4410	/* @dfl_pwq might not have been used, ensure it's linked */	5196	/* rescuer needs to respect wq cpumask changes */
4411	link_pwq(ctx->dfl_pwq);	5197	if (ctx->wq->rescuer)
4412	swap(ctx->wq->dfl_pwq, ctx->dfl_pwq);	5198	set_cpus_allowed_ptr(ctx->wq->rescuer->task,
		5199	unbound_effective_cpumask(ctx->wq));
4413		5200
4414	mutex_unlock(&ctx->wq->mutex);	5201	mutex_unlock(&ctx->wq->mutex);
4415	}	5202	}
@@ -4423,14 +5210,6 @@ static int apply_workqueue_attrs_locked(struct workqueue_struct *wq, · 7 unmodified lines
4423	if (WARN_ON(!(wq->flags & WQ_UNBOUND)))	5210	if (WARN_ON(!(wq->flags & WQ_UNBOUND)))
4424	return -EINVAL;	5211	return -EINVAL;
4425		5212
4426	/* creating multiple pwqs breaks ordering guarantee */
4427	if (!list_empty(&wq->pwqs)) {
4428	if (WARN_ON(wq->flags & __WQ_ORDERED_EXPLICIT))
4429	return -EINVAL;
4430
4431	wq->flags &= ~__WQ_ORDERED;
4432	}
4433
4434	ctx = apply_wqattrs_prepare(wq, attrs, wq_unbound_cpumask);	5213	ctx = apply_wqattrs_prepare(wq, attrs, wq_unbound_cpumask);
4435	if (IS_ERR(ctx))	5214	if (IS_ERR(ctx))
4436	return PTR_ERR(ctx);	5215	return PTR_ERR(ctx);
@@ -4519,9 +5298,7 @@ static void wq_update_pod(struct workqueue_struct *wq, int cpu, · 82 unmodified lines
4519		5298
4520	/* nothing to do if the target cpumask matches the current pwq */	5299	/* nothing to do if the target cpumask matches the current pwq */
4521	wq_calc_pod_cpumask(target_attrs, cpu, off_cpu);	5300	wq_calc_pod_cpumask(target_attrs, cpu, off_cpu);
4522	pwq = rcu_dereference_protected(*per_cpu_ptr(wq->cpu_pwq, cpu),	5301	if (wqattrs_equal(target_attrs, unbound_pwq(wq, cpu)->pool->attrs))
4523	lockdep_is_held(&wq_pool_mutex));
4524	if (wqattrs_equal(target_attrs, pwq->pool->attrs))
4525	return;	5302	return;
4526		5303
4527	/* create a new pwq */	5304	/* create a new pwq */
@@ -4539,10 +5316,11 @@ static void wq_update_pod(struct workqueue_struct *wq, int cpu, · 11 unmodified lines
4539		5316
4540	use_dfl_pwq:	5317	use_dfl_pwq:
4541	mutex_lock(&wq->mutex);	5318	mutex_lock(&wq->mutex);
4542	raw_spin_lock_irq(&wq->dfl_pwq->pool->lock);	5319	pwq = unbound_pwq(wq, -1);
4543	get_pwq(wq->dfl_pwq);	5320	raw_spin_lock_irq(&pwq->pool->lock);
4544	raw_spin_unlock_irq(&wq->dfl_pwq->pool->lock);	5321	get_pwq(pwq);
4545	old_pwq = install_unbound_pwq(wq, cpu, wq->dfl_pwq);	5322	raw_spin_unlock_irq(&pwq->pool->lock);
		5323	old_pwq = install_unbound_pwq(wq, cpu, pwq);
4546	out_unlock:	5324	out_unlock:
4547	mutex_unlock(&wq->mutex);	5325	mutex_unlock(&wq->mutex);
4548	put_pwq_unlocked(old_pwq);	5326	put_pwq_unlocked(old_pwq);
@@ -4559,10 +5337,17 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq) · 10 unmodified lines
4559		5337
4560	if (!(wq->flags & WQ_UNBOUND)) {	5338	if (!(wq->flags & WQ_UNBOUND)) {
4561	for_each_possible_cpu(cpu) {	5339	for_each_possible_cpu(cpu) {
4562	struct pool_workqueue **pwq_p =	5340	struct pool_workqueue **pwq_p;
4563	per_cpu_ptr(wq->cpu_pwq, cpu);	5341	struct worker_pool __percpu *pools;
4564	struct worker_pool *pool =	5342	struct worker_pool *pool;
4565	&(per_cpu_ptr(cpu_worker_pools, cpu)[highpri]);	5343
		5344	if (wq->flags & WQ_BH)
		5345	pools = bh_worker_pools;
		5346	else
		5347	pools = cpu_worker_pools;
		5348
		5349	pool = &(per_cpu_ptr(pools, cpu)[highpri]);
		5350	pwq_p = per_cpu_ptr(wq->cpu_pwq, cpu);
4566		5351
4567	*pwq_p = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL,	5352	*pwq_p = kmem_cache_alloc_node(pwq_cache, GFP_KERNEL,
4568	pool->node);	5353	pool->node);
@@ -4580,10 +5365,13 @@ static int alloc_and_link_pwqs(struct workqueue_struct *wq) · 11 unmodified lines
4580		5365
4581	cpus_read_lock();	5366	cpus_read_lock();
4582	if (wq->flags & __WQ_ORDERED) {	5367	if (wq->flags & __WQ_ORDERED) {
		5368	struct pool_workqueue *dfl_pwq;
		5369
4583	ret = apply_workqueue_attrs(wq, ordered_wq_attrs[highpri]);	5370	ret = apply_workqueue_attrs(wq, ordered_wq_attrs[highpri]);
4584	/* there should only be single pwq for ordering guarantee */	5371	/* there should only be single pwq for ordering guarantee */
4585	WARN(!ret && (wq->pwqs.next != &wq->dfl_pwq->pwqs_node \|\|	5372	dfl_pwq = rcu_access_pointer(wq->dfl_pwq);
4586	wq->pwqs.prev != &wq->dfl_pwq->pwqs_node),	5373	WARN(!ret && (wq->pwqs.next != &dfl_pwq->pwqs_node \|\|
		5374	wq->pwqs.prev != &dfl_pwq->pwqs_node),
4587	"ordering guarantee broken for workqueue %s\n", wq->name);	5375	"ordering guarantee broken for workqueue %s\n", wq->name);
4588	} else {	5376	} else {
4589	ret = apply_workqueue_attrs(wq, unbound_std_wq_attrs[highpri]);	5377	ret = apply_workqueue_attrs(wq, unbound_std_wq_attrs[highpri]);
@@ -4652,12 +5440,78 @@ static int init_rescuer(struct workqueue_struct *wq) · 62 unmodified lines
4652	}	5440	}
4653		5441
4654	wq->rescuer = rescuer;	5442	wq->rescuer = rescuer;
4655	kthread_bind_mask(rescuer->task, cpu_possible_mask);	5443	if (wq->flags & WQ_UNBOUND)
		5444	kthread_bind_mask(rescuer->task, wq_unbound_cpumask);
		5445	else
		5446	kthread_bind_mask(rescuer->task, cpu_possible_mask);
4656	wake_up_process(rescuer->task);	5447	wake_up_process(rescuer->task);
4657		5448
4658	return 0;	5449	return 0;
4659	}	5450	}
4660		5451
		5452	/**
		5453	* wq_adjust_max_active - update a wq's max_active to the current setting
		5454	* @wq: target workqueue
		5455	*
		5456	* If @wq isn't freezing, set @wq->max_active to the saved_max_active and
		5457	* activate inactive work items accordingly. If @wq is freezing, clear
		5458	* @wq->max_active to zero.
		5459	*/
		5460	static void wq_adjust_max_active(struct workqueue_struct *wq)
		5461	{
		5462	bool activated;
		5463	int new_max, new_min;
		5464
		5465	lockdep_assert_held(&wq->mutex);
		5466
		5467	if ((wq->flags & WQ_FREEZABLE) && workqueue_freezing) {
		5468	new_max = 0;
		5469	new_min = 0;
		5470	} else {
		5471	new_max = wq->saved_max_active;
		5472	new_min = wq->saved_min_active;
		5473	}
		5474
		5475	if (wq->max_active == new_max && wq->min_active == new_min)
		5476	return;
		5477
		5478	/*
		5479	* Update @wq->max/min_active and then kick inactive work items if more
		5480	* active work items are allowed. This doesn't break work item ordering
		5481	* because new work items are always queued behind existing inactive
		5482	* work items if there are any.
		5483	*/
		5484	WRITE_ONCE(wq->max_active, new_max);
		5485	WRITE_ONCE(wq->min_active, new_min);
		5486
		5487	if (wq->flags & WQ_UNBOUND)
		5488	wq_update_node_max_active(wq, -1);
		5489
		5490	if (new_max == 0)
		5491	return;
		5492
		5493	/*
		5494	* Round-robin through pwq's activating the first inactive work item
		5495	* until max_active is filled.
		5496	*/
		5497	do {
		5498	struct pool_workqueue *pwq;
		5499
		5500	activated = false;
		5501	for_each_pwq(pwq, wq) {
		5502	unsigned long irq_flags;
		5503
		5504	/* can be called during early boot w/ irq disabled */
		5505	raw_spin_lock_irqsave(&pwq->pool->lock, irq_flags);
		5506	if (pwq_activate_first_inactive(pwq, true)) {
		5507	activated = true;
		5508	kick_pool(pwq->pool);
		5509	}
		5510	raw_spin_unlock_irqrestore(&pwq->pool->lock, irq_flags);
		5511	}
		5512	} while (activated);
		5513	}
		5514
4661	__printf(1, 4)	5515	__printf(1, 4)
4662	struct workqueue_struct alloc_workqueue(const char fmt,	5516	struct workqueue_struct alloc_workqueue(const char fmt,
4663	unsigned int flags,	5517	unsigned int flags,
@@ -4665,23 +5519,27 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 1 unmodified line
4665	{	5519	{
4666	va_list args;	5520	va_list args;
4667	struct workqueue_struct *wq;	5521	struct workqueue_struct *wq;
4668	struct pool_workqueue *pwq;	5522	size_t wq_size;
		5523	int name_len;
4669		5524
4670	/*	5525	if (flags & WQ_BH) {
4671	* Unbound && max_active == 1 used to imply ordered, which is no longer	5526	if (WARN_ON_ONCE(flags & ~__WQ_BH_ALLOWS))
4672	* the case on many machines due to per-pod pools. While	5527	return NULL;
4673	* alloc_ordered_workqueue() is the right way to create an ordered	5528	if (WARN_ON_ONCE(max_active))
4674	* workqueue, keep the previous behavior to avoid subtle breakages.	5529	return NULL;
4675	*/	5530	}
4676	if ((flags & WQ_UNBOUND) && max_active == 1)
4677	flags \|= __WQ_ORDERED;
4678		5531
4679	/* see the comment above the definition of WQ_POWER_EFFICIENT */	5532	/* see the comment above the definition of WQ_POWER_EFFICIENT */
4680	if ((flags & WQ_POWER_EFFICIENT) && wq_power_efficient)	5533	if ((flags & WQ_POWER_EFFICIENT) && wq_power_efficient)
4681	flags \|= WQ_UNBOUND;	5534	flags \|= WQ_UNBOUND;
4682		5535
4683	/* allocate wq and format name */	5536	/* allocate wq and format name */
4684	wq = kzalloc(sizeof(*wq), GFP_KERNEL);	5537	if (flags & WQ_UNBOUND)
		5538	wq_size = struct_size(wq, node_nr_active, nr_node_ids + 1);
		5539	else
		5540	wq_size = sizeof(*wq);
		5541
		5542	wq = kzalloc(wq_size, GFP_KERNEL);
4685	if (!wq)	5543	if (!wq)
4686	return NULL;	5544	return NULL;
4687		5545
@@ -4692,15 +5550,30 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 4 unmodified lines
4692	}	5550	}
4693		5551
4694	va_start(args, max_active);	5552	va_start(args, max_active);
4695	vsnprintf(wq->name, sizeof(wq->name), fmt, args);	5553	name_len = vsnprintf(wq->name, sizeof(wq->name), fmt, args);
4696	va_end(args);	5554	va_end(args);
4697		5555
4698	max_active = max_active ?: WQ_DFL_ACTIVE;	5556	if (name_len >= WQ_NAME_LEN)
4699	max_active = wq_clamp_max_active(max_active, flags, wq->name);	5557	pr_warn_once("workqueue: name exceeds WQ_NAME_LEN. Truncating to: %s\n",
		5558	wq->name);
		5559
		5560	if (flags & WQ_BH) {
		5561	/*
		5562	* BH workqueues always share a single execution context per CPU
		5563	* and don't impose any max_active limit.
		5564	*/
		5565	max_active = INT_MAX;
		5566	} else {
		5567	max_active = max_active ?: WQ_DFL_ACTIVE;
		5568	max_active = wq_clamp_max_active(max_active, flags, wq->name);
		5569	}
4700		5570
4701	/* init wq */	5571	/* init wq */
4702	wq->flags = flags;	5572	wq->flags = flags;
4703	wq->saved_max_active = max_active;	5573	wq->max_active = max_active;
		5574	wq->min_active = min(max_active, WQ_DFL_MIN_ACTIVE);
		5575	wq->saved_max_active = wq->max_active;
		5576	wq->saved_min_active = wq->min_active;
4704	mutex_init(&wq->mutex);	5577	mutex_init(&wq->mutex);
4705	atomic_set(&wq->nr_pwqs_to_flush, 0);	5578	atomic_set(&wq->nr_pwqs_to_flush, 0);
4706	INIT_LIST_HEAD(&wq->pwqs);	5579	INIT_LIST_HEAD(&wq->pwqs);
@@ -4711,8 +5584,13 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 4 unmodified lines
4711	wq_init_lockdep(wq);	5584	wq_init_lockdep(wq);
4712	INIT_LIST_HEAD(&wq->list);	5585	INIT_LIST_HEAD(&wq->list);
4713		5586
		5587	if (flags & WQ_UNBOUND) {
		5588	if (alloc_node_nr_active(wq->node_nr_active) < 0)
		5589	goto err_unreg_lockdep;
		5590	}
		5591
4714	if (alloc_and_link_pwqs(wq) < 0)	5592	if (alloc_and_link_pwqs(wq) < 0)
4715	goto err_unreg_lockdep;	5593	goto err_free_node_nr_active;
4716		5594
4717	if (wq_online && init_rescuer(wq) < 0)	5595	if (wq_online && init_rescuer(wq) < 0)
4718	goto err_destroy;	5596	goto err_destroy;
@@ -4728,8 +5606,7 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 9 unmodified lines
4728	mutex_lock(&wq_pool_mutex);	5606	mutex_lock(&wq_pool_mutex);
4729		5607
4730	mutex_lock(&wq->mutex);	5608	mutex_lock(&wq->mutex);
4731	for_each_pwq(pwq, wq)	5609	wq_adjust_max_active(wq);
4732	pwq_adjust_max_active(pwq);
4733	mutex_unlock(&wq->mutex);	5610	mutex_unlock(&wq->mutex);
4734		5611
4735	list_add_tail_rcu(&wq->list, &workqueues);	5612	list_add_tail_rcu(&wq->list, &workqueues);
@@ -4738,6 +5615,9 @@ struct workqueue_struct alloc_workqueue(const char fmt, · 2 unmodified lines
4738		5615
4739	return wq;	5616	return wq;
4740		5617
		5618	err_free_node_nr_active:
		5619	if (wq->flags & WQ_UNBOUND)
		5620	free_node_nr_active(wq->node_nr_active);
4741	err_unreg_lockdep:	5621	err_unreg_lockdep:
4742	wq_unregister_lockdep(wq);	5622	wq_unregister_lockdep(wq);
4743	wq_free_lockdep(wq);	5623	wq_free_lockdep(wq);
@@ -4759,9 +5639,9 @@ static bool pwq_busy(struct pool_workqueue *pwq) · 15 unmodified lines
4759	if (pwq->nr_in_flight[i])	5639	if (pwq->nr_in_flight[i])
4760	return true;	5640	return true;
4761		5641
4762	if ((pwq != pwq->wq->dfl_pwq) && (pwq->refcnt > 1))	5642	if ((pwq != rcu_access_pointer(pwq->wq->dfl_pwq)) && (pwq->refcnt > 1))
4763	return true;	5643	return true;
4764	if (pwq->nr_active \|\| !list_empty(&pwq->inactive_works))	5644	if (!pwq_is_empty(pwq))
4765	return true;	5645	return true;
4766		5646
4767	return false;	5647	return false;
@@ -4843,13 +5723,12 @@ void destroy_workqueue(struct workqueue_struct *wq) · 75 unmodified lines
4843	rcu_read_lock();	5723	rcu_read_lock();
4844		5724
4845	for_each_possible_cpu(cpu) {	5725	for_each_possible_cpu(cpu) {
4846	pwq = rcu_access_pointer(*per_cpu_ptr(wq->cpu_pwq, cpu));	5726	put_pwq_unlocked(unbound_pwq(wq, cpu));
4847	RCU_INIT_POINTER(*per_cpu_ptr(wq->cpu_pwq, cpu), NULL);	5727	RCU_INIT_POINTER(*unbound_pwq_slot(wq, cpu), NULL);
4848	put_pwq_unlocked(pwq);
4849	}	5728	}
4850		5729
4851	put_pwq_unlocked(wq->dfl_pwq);	5730	put_pwq_unlocked(unbound_pwq(wq, -1));
4852	wq->dfl_pwq = NULL;	5731	RCU_INIT_POINTER(*unbound_pwq_slot(wq, -1), NULL);
4853		5732
4854	rcu_read_unlock();	5733	rcu_read_unlock();
4855	}	5734	}
@@ -4860,33 +5739,62 @@ EXPORT_SYMBOL_GPL(destroy_workqueue); · 4 unmodified lines
4860	* @wq: target workqueue	5739	* @wq: target workqueue
4861	* @max_active: new max_active value.	5740	* @max_active: new max_active value.
4862	*	5741	*
4863	* Set max_active of @wq to @max_active.	5742	* Set max_active of @wq to @max_active. See the alloc_workqueue() function
		5743	* comment.
4864	*	5744	*
4865	* CONTEXT:	5745	* CONTEXT:
4866	* Don't call from IRQ context.	5746	* Don't call from IRQ context.
4867	*/	5747	*/
4868	void workqueue_set_max_active(struct workqueue_struct *wq, int max_active)	5748	void workqueue_set_max_active(struct workqueue_struct *wq, int max_active)
4869	{	5749	{
4870	struct pool_workqueue *pwq;	5750	/* max_active doesn't mean anything for BH workqueues */
4871		5751	if (WARN_ON(wq->flags & WQ_BH))
		5752	return;
4872	/* disallow meddling with max_active for ordered workqueues */	5753	/* disallow meddling with max_active for ordered workqueues */
4873	if (WARN_ON(wq->flags & __WQ_ORDERED_EXPLICIT))	5754	if (WARN_ON(wq->flags & __WQ_ORDERED))
4874	return;	5755	return;
4875		5756
4876	max_active = wq_clamp_max_active(max_active, wq->flags, wq->name);	5757	max_active = wq_clamp_max_active(max_active, wq->flags, wq->name);
4877		5758
4878	mutex_lock(&wq->mutex);	5759	mutex_lock(&wq->mutex);
4879		5760
4880	wq->flags &= ~__WQ_ORDERED;
4881	wq->saved_max_active = max_active;	5761	wq->saved_max_active = max_active;
		5762	if (wq->flags & WQ_UNBOUND)
		5763	wq->saved_min_active = min(wq->saved_min_active, max_active);
4882		5764
4883	for_each_pwq(pwq, wq)	5765	wq_adjust_max_active(wq);
4884	pwq_adjust_max_active(pwq);
4885		5766
4886	mutex_unlock(&wq->mutex);	5767	mutex_unlock(&wq->mutex);
4887	}	5768	}
4888	EXPORT_SYMBOL_GPL(workqueue_set_max_active);	5769	EXPORT_SYMBOL_GPL(workqueue_set_max_active);
4889		5770
		5771	/**
		5772	* workqueue_set_min_active - adjust min_active of an unbound workqueue
		5773	* @wq: target unbound workqueue
		5774	* @min_active: new min_active value
		5775	*
		5776	* Set min_active of an unbound workqueue. Unlike other types of workqueues, an
		5777	* unbound workqueue is not guaranteed to be able to process max_active
		5778	* interdependent work items. Instead, an unbound workqueue is guaranteed to be
		5779	* able to process min_active number of interdependent work items which is
		5780	* %WQ_DFL_MIN_ACTIVE by default.
		5781	*
		5782	* Use this function to adjust the min_active value between 0 and the current
		5783	* max_active.
		5784	*/
		5785	void workqueue_set_min_active(struct workqueue_struct *wq, int min_active)
		5786	{
		5787	/* min_active is only meaningful for non-ordered unbound workqueues */
		5788	if (WARN_ON((wq->flags & (WQ_BH \| WQ_UNBOUND \| __WQ_ORDERED)) !=
		5789	WQ_UNBOUND))
		5790	return;
		5791
		5792	mutex_lock(&wq->mutex);
		5793	wq->saved_min_active = clamp(min_active, 0, wq->saved_max_active);
		5794	wq_adjust_max_active(wq);
		5795	mutex_unlock(&wq->mutex);
		5796	}
		5797
4890	/**	5798	/**
4891	* current_work - retrieve %current task's work struct	5799	* current_work - retrieve %current task's work struct
4892	*	5800	*
@@ -4972,7 +5880,7 @@ EXPORT_SYMBOL_GPL(workqueue_congested); · 79 unmodified lines
4972	unsigned int work_busy(struct work_struct *work)	5880	unsigned int work_busy(struct work_struct *work)
4973	{	5881	{
4974	struct worker_pool *pool;	5882	struct worker_pool *pool;
4975	unsigned long flags;	5883	unsigned long irq_flags;
4976	unsigned int ret = 0;	5884	unsigned int ret = 0;
4977		5885
4978	if (work_pending(work))	5886	if (work_pending(work))
@@ -4981,10 +5889,10 @@ unsigned int work_busy(struct work_struct *work) · 2 unmodified lines
4981	rcu_read_lock();	5889	rcu_read_lock();
4982	pool = get_work_pool(work);	5890	pool = get_work_pool(work);
4983	if (pool) {	5891	if (pool) {
4984	raw_spin_lock_irqsave(&pool->lock, flags);	5892	raw_spin_lock_irqsave(&pool->lock, irq_flags);
4985	if (find_worker_executing_work(pool, work))	5893	if (find_worker_executing_work(pool, work))
4986	ret \|= WORK_BUSY_RUNNING;	5894	ret \|= WORK_BUSY_RUNNING;
4987	raw_spin_unlock_irqrestore(&pool->lock, flags);	5895	raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
4988	}	5896	}
4989	rcu_read_unlock();	5897	rcu_read_unlock();
4990		5898
@@ -5069,7 +5977,24 @@ static void pr_cont_pool_info(struct worker_pool *pool) · 78 unmodified lines
5069	pr_cont(" cpus=%*pbl", nr_cpumask_bits, pool->attrs->cpumask);	5977	pr_cont(" cpus=%*pbl", nr_cpumask_bits, pool->attrs->cpumask);
5070	if (pool->node != NUMA_NO_NODE)	5978	if (pool->node != NUMA_NO_NODE)
5071	pr_cont(" node=%d", pool->node);	5979	pr_cont(" node=%d", pool->node);
5072	pr_cont(" flags=0x%x nice=%d", pool->flags, pool->attrs->nice);	5980	pr_cont(" flags=0x%x", pool->flags);
		5981	if (pool->flags & POOL_BH)
		5982	pr_cont(" bh%s",
		5983	pool->attrs->nice == HIGHPRI_NICE_LEVEL ? "-hi" : "");
		5984	else
		5985	pr_cont(" nice=%d", pool->attrs->nice);
		5986	}
		5987
		5988	static void pr_cont_worker_id(struct worker *worker)
		5989	{
		5990	struct worker_pool *pool = worker->pool;
		5991
		5992	if (pool->flags & WQ_BH)
		5993	pr_cont("bh%s",
		5994	pool->attrs->nice == HIGHPRI_NICE_LEVEL ? "-hi" : "");
		5995	else
		5996	pr_cont("%d%s", task_pid_nr(worker->task),
		5997	worker->rescue_wq ? "(RESCUER)" : "");
5073	}	5998	}
5074		5999
5075	struct pr_cont_work_struct {	6000	struct pr_cont_work_struct {
@@ -5128,8 +6053,8 @@ static void show_pwq(struct pool_workqueue *pwq) · 52 unmodified lines
5128	pr_info(" pwq %d:", pool->id);	6053	pr_info(" pwq %d:", pool->id);
5129	pr_cont_pool_info(pool);	6054	pr_cont_pool_info(pool);
5130		6055
5131	pr_cont(" active=%d/%d refcnt=%d%s\n",	6056	pr_cont(" active=%d refcnt=%d%s\n",
5132	pwq->nr_active, pwq->max_active, pwq->refcnt,	6057	pwq->nr_active, pwq->refcnt,
5133	!list_empty(&pwq->mayday_node) ? " MAYDAY" : "");	6058	!list_empty(&pwq->mayday_node) ? " MAYDAY" : "");
5134		6059
5135	hash_for_each(pool->busy_hash, bkt, worker, hentry) {	6060	hash_for_each(pool->busy_hash, bkt, worker, hentry) {
@@ -5146,10 +6071,9 @@ static void show_pwq(struct pool_workqueue *pwq) · 10 unmodified lines
5146	if (worker->current_pwq != pwq)	6071	if (worker->current_pwq != pwq)
5147	continue;	6072	continue;
5148		6073
5149	pr_cont("%s %d%s:%ps", comma ? "," : "",	6074	pr_cont(" %s", comma ? "," : "");
5150	task_pid_nr(worker->task),	6075	pr_cont_worker_id(worker);
5151	worker->rescue_wq ? "(RESCUER)" : "",	6076	pr_cont(":%ps", worker->current_func);
5152	worker->current_func);
5153	list_for_each_entry(work, &worker->scheduled, entry)	6077	list_for_each_entry(work, &worker->scheduled, entry)
5154	pr_cont_work(false, work, &pcws);	6078	pr_cont_work(false, work, &pcws);
5155	pr_cont_work_flush(comma, (work_func_t)-1L, &pcws);	6079	pr_cont_work_flush(comma, (work_func_t)-1L, &pcws);
@@ -5200,10 +6124,10 @@ void show_one_workqueue(struct workqueue_struct *wq) · 44 unmodified lines
5200	{	6124	{
5201	struct pool_workqueue *pwq;	6125	struct pool_workqueue *pwq;
5202	bool idle = true;	6126	bool idle = true;
5203	unsigned long flags;	6127	unsigned long irq_flags;
5204		6128
5205	for_each_pwq(pwq, wq) {	6129	for_each_pwq(pwq, wq) {
5206	if (pwq->nr_active \|\| !list_empty(&pwq->inactive_works)) {	6130	if (!pwq_is_empty(pwq)) {
5207	idle = false;	6131	idle = false;
5208	break;	6132	break;
5209	}	6133	}
@@ -5214,8 +6138,8 @@ void show_one_workqueue(struct workqueue_struct *wq) · 4 unmodified lines
5214	pr_info("workqueue %s: flags=0x%x\n", wq->name, wq->flags);	6138	pr_info("workqueue %s: flags=0x%x\n", wq->name, wq->flags);
5215		6139
5216	for_each_pwq(pwq, wq) {	6140	for_each_pwq(pwq, wq) {
5217	raw_spin_lock_irqsave(&pwq->pool->lock, flags);	6141	raw_spin_lock_irqsave(&pwq->pool->lock, irq_flags);
5218	if (pwq->nr_active \|\| !list_empty(&pwq->inactive_works)) {	6142	if (!pwq_is_empty(pwq)) {
5219	/*	6143	/*
5220	* Defer printing to avoid deadlocks in console	6144	* Defer printing to avoid deadlocks in console
5221	* drivers that queue work while holding locks	6145	* drivers that queue work while holding locks
@@ -5225,7 +6149,7 @@ void show_one_workqueue(struct workqueue_struct *wq) · 3 unmodified lines
5225	show_pwq(pwq);	6149	show_pwq(pwq);
5226	printk_deferred_exit();	6150	printk_deferred_exit();
5227	}	6151	}
5228	raw_spin_unlock_irqrestore(&pwq->pool->lock, flags);	6152	raw_spin_unlock_irqrestore(&pwq->pool->lock, irq_flags);
5229	/*	6153	/*
5230	* We could be printing a lot from atomic context, e.g.	6154	* We could be printing a lot from atomic context, e.g.
5231	* sysrq-t -> show_all_workqueues(). Avoid triggering	6155	* sysrq-t -> show_all_workqueues(). Avoid triggering
@@ -5244,10 +6168,10 @@ static void show_one_worker_pool(struct worker_pool *pool) · 12 unmodified lines
5244	{	6168	{
5245	struct worker *worker;	6169	struct worker *worker;
5246	bool first = true;	6170	bool first = true;
5247	unsigned long flags;	6171	unsigned long irq_flags;
5248	unsigned long hung = 0;	6172	unsigned long hung = 0;
5249		6173
5250	raw_spin_lock_irqsave(&pool->lock, flags);	6174	raw_spin_lock_irqsave(&pool->lock, irq_flags);
5251	if (pool->nr_workers == pool->nr_idle)	6175	if (pool->nr_workers == pool->nr_idle)
5252	goto next_pool;	6176	goto next_pool;
5253		6177
@@ -5268,14 +6192,14 @@ static void show_one_worker_pool(struct worker_pool *pool) · 14 unmodified lines
5268	pr_cont(" manager: %d",	6192	pr_cont(" manager: %d",
5269	task_pid_nr(pool->manager->task));	6193	task_pid_nr(pool->manager->task));
5270	list_for_each_entry(worker, &pool->idle_list, entry) {	6194	list_for_each_entry(worker, &pool->idle_list, entry) {
5271	pr_cont(" %s%d", first ? "idle: " : "",	6195	pr_cont(" %s", first ? "idle: " : "");
5272	task_pid_nr(worker->task));	6196	pr_cont_worker_id(worker);
5273	first = false;	6197	first = false;
5274	}	6198	}
5275	pr_cont("\n");	6199	pr_cont("\n");
5276	printk_deferred_exit();	6200	printk_deferred_exit();
5277	next_pool:	6201	next_pool:
5278	raw_spin_unlock_irqrestore(&pool->lock, flags);	6202	raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
5279	/*	6203	/*
5280	* We could be printing a lot from atomic context, e.g.	6204	* We could be printing a lot from atomic context, e.g.
5281	* sysrq-t -> show_all_workqueues(). Avoid triggering	6205	* sysrq-t -> show_all_workqueues(). Avoid triggering
@@ -5542,13 +6466,15 @@ int workqueue_online_cpu(unsigned int cpu) · 260 unmodified lines
5542	mutex_lock(&wq_pool_mutex);	6466	mutex_lock(&wq_pool_mutex);
5543		6467
5544	for_each_pool(pool, pi) {	6468	for_each_pool(pool, pi) {
5545	mutex_lock(&wq_pool_attach_mutex);	6469	/* BH pools aren't affected by hotplug */
		6470	if (pool->flags & POOL_BH)
		6471	continue;
5546		6472
		6473	mutex_lock(&wq_pool_attach_mutex);
5547	if (pool->cpu == cpu)	6474	if (pool->cpu == cpu)
5548	rebind_workers(pool);	6475	rebind_workers(pool);
5549	else if (pool->cpu < 0)	6476	else if (pool->cpu < 0)
5550	restore_unbound_workers_cpumask(pool, cpu);	6477	restore_unbound_workers_cpumask(pool, cpu);
5551
5552	mutex_unlock(&wq_pool_attach_mutex);	6478	mutex_unlock(&wq_pool_attach_mutex);
5553	}	6479	}
5554		6480
@@ -5562,6 +6488,10 @@ int workqueue_online_cpu(unsigned int cpu) · 7 unmodified lines
5562		6488
5563	for_each_cpu(tcpu, pt->pod_cpus[pt->cpu_pod[cpu]])	6489	for_each_cpu(tcpu, pt->pod_cpus[pt->cpu_pod[cpu]])
5564	wq_update_pod(wq, tcpu, cpu, true);	6490	wq_update_pod(wq, tcpu, cpu, true);
		6491
		6492	mutex_lock(&wq->mutex);
		6493	wq_update_node_max_active(wq, -1);
		6494	mutex_unlock(&wq->mutex);
5565	}	6495	}
5566	}	6496	}
5567		6497
@@ -5590,6 +6520,10 @@ int workqueue_offline_cpu(unsigned int cpu) · 22 unmodified lines
5590		6520
5591	for_each_cpu(tcpu, pt->pod_cpus[pt->cpu_pod[cpu]])	6521	for_each_cpu(tcpu, pt->pod_cpus[pt->cpu_pod[cpu]])
5592	wq_update_pod(wq, tcpu, cpu, false);	6522	wq_update_pod(wq, tcpu, cpu, false);
		6523
		6524	mutex_lock(&wq->mutex);
		6525	wq_update_node_max_active(wq, cpu);
		6526	mutex_unlock(&wq->mutex);
5593	}	6527	}
5594	}	6528	}
5595	mutex_unlock(&wq_pool_mutex);	6529	mutex_unlock(&wq_pool_mutex);
@@ -5677,7 +6611,6 @@ EXPORT_SYMBOL_GPL(work_on_cpu_safe_key); · 81 unmodified lines
5677	void freeze_workqueues_begin(void)	6611	void freeze_workqueues_begin(void)
5678	{	6612	{
5679	struct workqueue_struct *wq;	6613	struct workqueue_struct *wq;
5680	struct pool_workqueue *pwq;
5681		6614
5682	mutex_lock(&wq_pool_mutex);	6615	mutex_lock(&wq_pool_mutex);
5683		6616
@@ -5686,8 +6619,7 @@ void freeze_workqueues_begin(void) · 2 unmodified lines
5686		6619
5687	list_for_each_entry(wq, &workqueues, list) {	6620	list_for_each_entry(wq, &workqueues, list) {
5688	mutex_lock(&wq->mutex);	6621	mutex_lock(&wq->mutex);
5689	for_each_pwq(pwq, wq)	6622	wq_adjust_max_active(wq);
5690	pwq_adjust_max_active(pwq);
5691	mutex_unlock(&wq->mutex);	6623	mutex_unlock(&wq->mutex);
5692	}	6624	}
5693		6625
@@ -5752,7 +6684,6 @@ out_unlock: · 58 unmodified lines
5752	void thaw_workqueues(void)	6684	void thaw_workqueues(void)
5753	{	6685	{
5754	struct workqueue_struct *wq;	6686	struct workqueue_struct *wq;
5755	struct pool_workqueue *pwq;
5756		6687
5757	mutex_lock(&wq_pool_mutex);	6688	mutex_lock(&wq_pool_mutex);
5758		6689
@@ -5764,8 +6695,7 @@ void thaw_workqueues(void) · 5 unmodified lines
5764	/* restore max_active and repopulate worklist */	6695	/* restore max_active and repopulate worklist */
5765	list_for_each_entry(wq, &workqueues, list) {	6696	list_for_each_entry(wq, &workqueues, list) {
5766	mutex_lock(&wq->mutex);	6697	mutex_lock(&wq->mutex);
5767	for_each_pwq(pwq, wq)	6698	wq_adjust_max_active(wq);
5768	pwq_adjust_max_active(pwq);
5769	mutex_unlock(&wq->mutex);	6699	mutex_unlock(&wq->mutex);
5770	}	6700	}
5771		6701
@@ -5784,10 +6714,7 @@ static int workqueue_apply_unbound_cpumask(const cpumask_var_t unbound_cpumask) · 12 unmodified lines
5784	lockdep_assert_held(&wq_pool_mutex);	6714	lockdep_assert_held(&wq_pool_mutex);
5785		6715
5786	list_for_each_entry(wq, &workqueues, list) {	6716	list_for_each_entry(wq, &workqueues, list) {
5787	if (!(wq->flags & WQ_UNBOUND))	6717	if (!(wq->flags & WQ_UNBOUND) \|\| (wq->flags & __WQ_DESTROYING))
5788	continue;
5789	/* creating multiple pwqs breaks ordering guarantee */
5790	if (wq->flags & __WQ_ORDERED)
5791	continue;	6718	continue;
5792		6719
5793	ctx = apply_wqattrs_prepare(wq, wq->unbound_attrs, unbound_cpumask);	6720	ctx = apply_wqattrs_prepare(wq, wq->unbound_attrs, unbound_cpumask);
@@ -6303,11 +7230,10 @@ int workqueue_sysfs_register(struct workqueue_struct *wq) · 509 unmodified lines
6303	int ret;	7230	int ret;
6304		7231
6305	/*	7232	/*
6306	* Adjusting max_active or creating new pwqs by applying	7233	* Adjusting max_active breaks ordering guarantee. Disallow exposing
6307	* attributes breaks ordering guarantee. Disallow exposing ordered	7234	* ordered workqueues.
6308	* workqueues.
6309	*/	7235	*/
6310	if (WARN_ON(wq->flags & __WQ_ORDERED_EXPLICIT))	7236	if (WARN_ON(wq->flags & __WQ_ORDERED))
6311	return -EINVAL;	7237	return -EINVAL;
6312		7238
6313	wq->wq_dev = wq_dev = kzalloc(sizeof(*wq_dev), GFP_KERNEL);	7239	wq->wq_dev = wq_dev = kzalloc(sizeof(*wq_dev), GFP_KERNEL);
@@ -6404,10 +7330,10 @@ static DEFINE_PER_CPU(unsigned long, wq_watchdog_touched_cpu) = INITIAL_JIFFIES; · 90 unmodified lines
6404	static void show_cpu_pool_hog(struct worker_pool *pool)	7330	static void show_cpu_pool_hog(struct worker_pool *pool)
6405	{	7331	{
6406	struct worker *worker;	7332	struct worker *worker;
6407	unsigned long flags;	7333	unsigned long irq_flags;
6408	int bkt;	7334	int bkt;
6409		7335
6410	raw_spin_lock_irqsave(&pool->lock, flags);	7336	raw_spin_lock_irqsave(&pool->lock, irq_flags);
6411		7337
6412	hash_for_each(pool->busy_hash, bkt, worker, hentry) {	7338	hash_for_each(pool->busy_hash, bkt, worker, hentry) {
6413	if (task_is_running(worker->task)) {	7339	if (task_is_running(worker->task)) {
@@ -6425,7 +7351,7 @@ static void show_cpu_pool_hog(struct worker_pool *pool) · 11 unmodified lines
6425	}	7351	}
6426	}	7352	}
6427		7353
6428	raw_spin_unlock_irqrestore(&pool->lock, flags);	7354	raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
6429	}	7355	}
6430		7356
6431	static void show_cpu_pools_hogs(void)	7357	static void show_cpu_pools_hogs(void)
@@ -6497,7 +7423,7 @@ static void wq_watchdog_timer_fn(struct timer_list *unused) · 65 unmodified lines
6497	/* did we stall? */	7423	/* did we stall? */
6498	if (time_after(now, ts + thresh)) {	7424	if (time_after(now, ts + thresh)) {
6499	lockup_detected = true;	7425	lockup_detected = true;
6500	if (pool->cpu >= 0) {	7426	if (pool->cpu >= 0 && !(pool->flags & POOL_BH)) {
6501	pool->cpu_stall = true;	7427	pool->cpu_stall = true;
6502	cpu_pool_stall = true;	7428	cpu_pool_stall = true;
6503	}	7429	}
@@ -6580,6 +7506,16 @@ static inline void wq_watchdog_init(void) { } · 76 unmodified lines
6580		7506
6581	#endif /* CONFIG_WQ_WATCHDOG */	7507	#endif /* CONFIG_WQ_WATCHDOG */
6582		7508
		7509	static void bh_pool_kick_normal(struct irq_work *irq_work)
		7510	{
		7511	raise_softirq_irqoff(TASKLET_SOFTIRQ);
		7512	}
		7513
		7514	static void bh_pool_kick_highpri(struct irq_work *irq_work)
		7515	{
		7516	raise_softirq_irqoff(HI_SOFTIRQ);
		7517	}
		7518
6583	static void __init restrict_unbound_cpumask(const char name, const struct cpumask mask)	7519	static void __init restrict_unbound_cpumask(const char name, const struct cpumask mask)
6584	{	7520	{
6585	if (!cpumask_intersects(wq_unbound_cpumask, mask)) {	7521	if (!cpumask_intersects(wq_unbound_cpumask, mask)) {
@@ -6591,6 +7527,22 @@ static void __init restrict_unbound_cpumask(const char *name, const struct cpuma · 5 unmodified lines
6591	cpumask_and(wq_unbound_cpumask, wq_unbound_cpumask, mask);	7527	cpumask_and(wq_unbound_cpumask, wq_unbound_cpumask, mask);
6592	}	7528	}
6593		7529
		7530	static void __init init_cpu_worker_pool(struct worker_pool *pool, int cpu, int nice)
		7531	{
		7532	BUG_ON(init_worker_pool(pool));
		7533	pool->cpu = cpu;
		7534	cpumask_copy(pool->attrs->cpumask, cpumask_of(cpu));
		7535	cpumask_copy(pool->attrs->__pod_cpumask, cpumask_of(cpu));
		7536	pool->attrs->nice = nice;
		7537	pool->attrs->affn_strict = true;
		7538	pool->node = cpu_to_node(cpu);
		7539
		7540	/* alloc pool ID */
		7541	mutex_lock(&wq_pool_mutex);
		7542	BUG_ON(worker_pool_assign_id(pool));
		7543	mutex_unlock(&wq_pool_mutex);
		7544	}
		7545
6594	/**	7546	/**
6595	* workqueue_init_early - early init for workqueue subsystem	7547	* workqueue_init_early - early init for workqueue subsystem
6596	*	7548	*
@@ -6605,6 +7557,8 @@ void __init workqueue_init_early(void) · 8 unmodified lines
6605	{	7557	{
6606	struct wq_pod_type *pt = &wq_pod_types[WQ_AFFN_SYSTEM];	7558	struct wq_pod_type *pt = &wq_pod_types[WQ_AFFN_SYSTEM];
6607	int std_nice[NR_STD_WORKER_POOLS] = { 0, HIGHPRI_NICE_LEVEL };	7559	int std_nice[NR_STD_WORKER_POOLS] = { 0, HIGHPRI_NICE_LEVEL };
		7560	void (irq_work_fns[2])(struct irq_work ) = { bh_pool_kick_normal,
		7561	bh_pool_kick_highpri };
6608	int i, cpu;	7562	int i, cpu;
6609		7563
6610	BUILD_BUG_ON(__alignof__(struct pool_workqueue) < __alignof__(long long));	7564	BUILD_BUG_ON(__alignof__(struct pool_workqueue) < __alignof__(long long));
@@ -6626,6 +7580,13 @@ void __init workqueue_init_early(void) · 15 unmodified lines
6626	wq_update_pod_attrs_buf = alloc_workqueue_attrs();	7580	wq_update_pod_attrs_buf = alloc_workqueue_attrs();
6627	BUG_ON(!wq_update_pod_attrs_buf);	7581	BUG_ON(!wq_update_pod_attrs_buf);
6628		7582
		7583	/*
		7584	* If nohz_full is enabled, set power efficient workqueue as unbound.
		7585	* This allows workqueue items to be moved to HK CPUs.
		7586	*/
		7587	if (housekeeping_enabled(HK_TYPE_TICK))
		7588	wq_power_efficient = true;
		7589
6629	/* initialize WQ_AFFN_SYSTEM pods */	7590	/* initialize WQ_AFFN_SYSTEM pods */
6630	pt->pod_cpus = kcalloc(1, sizeof(pt->pod_cpus[0]), GFP_KERNEL);	7591	pt->pod_cpus = kcalloc(1, sizeof(pt->pod_cpus[0]), GFP_KERNEL);
6631	pt->pod_node = kcalloc(1, sizeof(pt->pod_node[0]), GFP_KERNEL);	7592	pt->pod_node = kcalloc(1, sizeof(pt->pod_node[0]), GFP_KERNEL);
@@ -6639,25 +7600,21 @@ void __init workqueue_init_early(void) · 7 unmodified lines
6639	pt->pod_node[0] = NUMA_NO_NODE;	7600	pt->pod_node[0] = NUMA_NO_NODE;
6640	pt->cpu_pod[0] = 0;	7601	pt->cpu_pod[0] = 0;
6641		7602
6642	/* initialize CPU pools */	7603	/* initialize BH and CPU pools */
6643	for_each_possible_cpu(cpu) {	7604	for_each_possible_cpu(cpu) {
6644	struct worker_pool *pool;	7605	struct worker_pool *pool;
6645		7606
6646	i = 0;	7607	i = 0;
6647	for_each_cpu_worker_pool(pool, cpu) {	7608	for_each_bh_worker_pool(pool, cpu) {
6648	BUG_ON(init_worker_pool(pool));	7609	init_cpu_worker_pool(pool, cpu, std_nice[i]);
6649	pool->cpu = cpu;	7610	pool->flags \|= POOL_BH;
6650	cpumask_copy(pool->attrs->cpumask, cpumask_of(cpu));	7611	init_irq_work(bh_pool_irq_work(pool), irq_work_fns[i]);
6651	cpumask_copy(pool->attrs->__pod_cpumask, cpumask_of(cpu));	7612	i++;
6652	pool->attrs->nice = std_nice[i++];
6653	pool->attrs->affn_strict = true;
6654	pool->node = cpu_to_node(cpu);
6655
6656	/* alloc pool ID */
6657	mutex_lock(&wq_pool_mutex);
6658	BUG_ON(worker_pool_assign_id(pool));
6659	mutex_unlock(&wq_pool_mutex);
6660	}	7613	}
		7614
		7615	i = 0;
		7616	for_each_cpu_worker_pool(pool, cpu)
		7617	init_cpu_worker_pool(pool, cpu, std_nice[i++]);
6661	}	7618	}
6662		7619
6663	/* create default unbound and ordered wq attrs */	7620	/* create default unbound and ordered wq attrs */
@@ -6687,13 +7644,17 @@ void __init workqueue_init_early(void) · 23 unmodified lines
6687	WQ_FREEZABLE, 0);	7644	WQ_FREEZABLE, 0);
6688	system_power_efficient_wq = alloc_workqueue("events_power_efficient",	7645	system_power_efficient_wq = alloc_workqueue("events_power_efficient",
6689	WQ_POWER_EFFICIENT, 0);	7646	WQ_POWER_EFFICIENT, 0);
6690	system_freezable_power_efficient_wq = alloc_workqueue("events_freezable_power_efficient",	7647	system_freezable_power_efficient_wq = alloc_workqueue("events_freezable_pwr_efficient",
6691	WQ_FREEZABLE \| WQ_POWER_EFFICIENT,	7648	WQ_FREEZABLE \| WQ_POWER_EFFICIENT,
6692	0);	7649	0);
		7650	system_bh_wq = alloc_workqueue("events_bh", WQ_BH, 0);
		7651	system_bh_highpri_wq = alloc_workqueue("events_bh_highpri",
		7652	WQ_BH \| WQ_HIGHPRI, 0);
6693	BUG_ON(!system_wq \|\| !system_highpri_wq \|\| !system_long_wq \|\|	7653	BUG_ON(!system_wq \|\| !system_highpri_wq \|\| !system_long_wq \|\|
6694	!system_unbound_wq \|\| !system_freezable_wq \|\|	7654	!system_unbound_wq \|\| !system_freezable_wq \|\|
6695	!system_power_efficient_wq \|\|	7655	!system_power_efficient_wq \|\|
6696	!system_freezable_power_efficient_wq);	7656	!system_freezable_power_efficient_wq \|\|
		7657	!system_bh_wq \|\| !system_bh_highpri_wq);
6697	}	7658	}
6698		7659
6699	static void __init wq_cpu_intensive_thresh_init(void)	7660	static void __init wq_cpu_intensive_thresh_init(void)
@@ -6759,9 +7720,10 @@ void __init workqueue_init(void) · 59 unmodified lines
6759	* up. Also, create a rescuer for workqueues that requested it.	7720	* up. Also, create a rescuer for workqueues that requested it.
6760	*/	7721	*/
6761	for_each_possible_cpu(cpu) {	7722	for_each_possible_cpu(cpu) {
6762	for_each_cpu_worker_pool(pool, cpu) {	7723	for_each_bh_worker_pool(pool, cpu)
		7724	pool->node = cpu_to_node(cpu);
		7725	for_each_cpu_worker_pool(pool, cpu)
6763	pool->node = cpu_to_node(cpu);	7726	pool->node = cpu_to_node(cpu);
6764	}
6765	}	7727	}
6766		7728
6767	list_for_each_entry(wq, &workqueues, list) {	7729	list_for_each_entry(wq, &workqueues, list) {
@@ -6772,7 +7734,16 @@ void __init workqueue_init(void) · 4 unmodified lines
6772		7734
6773	mutex_unlock(&wq_pool_mutex);	7735	mutex_unlock(&wq_pool_mutex);
6774		7736
6775	/* create the initial workers */	7737	/*
		7738	* Create the initial workers. A BH pool has one pseudo worker that
		7739	* represents the shared BH execution context and thus doesn't get
		7740	* affected by hotplug events. Create the BH pseudo workers for all
		7741	* possible CPUs here.
		7742	*/
		7743	for_each_possible_cpu(cpu)
		7744	for_each_bh_worker_pool(pool, cpu)
		7745	BUG_ON(!create_worker(pool));
		7746
6776	for_each_online_cpu(cpu) {	7747	for_each_online_cpu(cpu) {
6777	for_each_cpu_worker_pool(pool, cpu) {	7748	for_each_cpu_worker_pool(pool, cpu) {
6778	pool->flags &= ~POOL_DISASSOCIATED;	7749	pool->flags &= ~POOL_DISASSOCIATED;
@@ -6852,7 +7823,7 @@ static bool __init cpus_share_numa(int cpu0, int cpu1) · 73 unmodified lines
6852	/**	7823	/**
6853	* workqueue_init_topology - initialize CPU pods for unbound workqueues	7824	* workqueue_init_topology - initialize CPU pods for unbound workqueues
6854	*	7825	*
6855	* This is the third step of there-staged workqueue subsystem initialization and	7826	* This is the third step of three-staged workqueue subsystem initialization and
6856	* invoked after SMP and topology information are fully initialized. It	7827	* invoked after SMP and topology information are fully initialized. It
6857	* initializes the unbound CPU pods accordingly.	7828	* initializes the unbound CPU pods accordingly.
6858	*/	7829	*/
@@ -6866,6 +7837,8 @@ void __init workqueue_init_topology(void) · 7 unmodified lines
6866	init_pod_type(&wq_pod_types[WQ_AFFN_CACHE], cpus_share_cache);	7837	init_pod_type(&wq_pod_types[WQ_AFFN_CACHE], cpus_share_cache);
6867	init_pod_type(&wq_pod_types[WQ_AFFN_NUMA], cpus_share_numa);	7838	init_pod_type(&wq_pod_types[WQ_AFFN_NUMA], cpus_share_numa);
6868		7839
		7840	wq_topo_initialized = true;
		7841
6869	mutex_lock(&wq_pool_mutex);	7842	mutex_lock(&wq_pool_mutex);
6870		7843
6871	/*	7844	/*
@@ -6874,8 +7847,12 @@ void __init workqueue_init_topology(void) · 2 unmodified lines
6874	* combinations to apply per-pod sharing.	7847	* combinations to apply per-pod sharing.
6875	*/	7848	*/
6876	list_for_each_entry(wq, &workqueues, list) {	7849	list_for_each_entry(wq, &workqueues, list) {
6877	for_each_online_cpu(cpu) {	7850	for_each_online_cpu(cpu)
6878	wq_update_pod(wq, cpu, cpu, true);	7851	wq_update_pod(wq, cpu, cpu, true);
		7852	if (wq->flags & WQ_UNBOUND) {
		7853	mutex_lock(&wq->mutex);
		7854	wq_update_node_max_active(wq, -1);
		7855	mutex_unlock(&wq->mutex);
6879	}	7856	}
6880	}	7857	}
6881		7858

rust/kernel/workqueue.rs+5−1

         // stay valid until we call the function pointer in the `work_struct`, so the access is ok.
         unsafe {
             w.__enqueue(move |work_ptr| {
-                bindings::queue_work_on(bindings::WORK_CPU_UNBOUND as _, queue_ptr, work_ptr)
+                bindings::queue_work_on(
+                    bindings::wq_misc_consts_WORK_CPU_UNBOUND as _,
+                    queue_ptr,
+                    work_ptr,
+                )
             })
         }
     }

tools/workqueue/wq_dump.py+90−14

@@ -50,6 +50,7 @@ import drgn · 49 unmodified lines
50	from drgn.helpers.linux.list import list_for_each_entry,list_empty	50	from drgn.helpers.linux.list import list_for_each_entry,list_empty
51	from drgn.helpers.linux.percpu import per_cpu_ptr	51	from drgn.helpers.linux.percpu import per_cpu_ptr
52	from drgn.helpers.linux.cpumask import for_each_cpu,for_each_possible_cpu	52	from drgn.helpers.linux.cpumask import for_each_cpu,for_each_possible_cpu
		53	from drgn.helpers.linux.nodemask import for_each_node
53	from drgn.helpers.linux.idr import idr_for_each	54	from drgn.helpers.linux.idr import idr_for_each
54		55
55	import argparse	56	import argparse
@@ -75,6 +76,22 @@ def cpumask_str(cpumask): · 19 unmodified lines
75	output += f'{v:08x}'	76	output += f'{v:08x}'
76	return output.strip()	77	return output.strip()
77		78
		79	wq_type_len = 9
		80
		81	def wq_type_str(wq):
		82	if wq.flags & WQ_BH:
		83	return f'{"bh":{wq_type_len}}'
		84	elif wq.flags & WQ_UNBOUND:
		85	if wq.flags & WQ_ORDERED:
		86	return f'{"ordered":{wq_type_len}}'
		87	else:
		88	if wq.unbound_attrs.affn_strict:
		89	return f'{"unbound,S":{wq_type_len}}'
		90	else:
		91	return f'{"unbound":{wq_type_len}}'
		92	else:
		93	return f'{"percpu":{wq_type_len}}'
		94
78	worker_pool_idr = prog['worker_pool_idr']	95	worker_pool_idr = prog['worker_pool_idr']
79	workqueues = prog['workqueues']	96	workqueues = prog['workqueues']
80	wq_unbound_cpumask = prog['wq_unbound_cpumask']	97	wq_unbound_cpumask = prog['wq_unbound_cpumask']
@@ -82,6 +99,7 @@ wq_pod_types = prog['wq_pod_types'] · 1 unmodified line
82	wq_affn_dfl = prog['wq_affn_dfl']	99	wq_affn_dfl = prog['wq_affn_dfl']
83	wq_affn_names = prog['wq_affn_names']	100	wq_affn_names = prog['wq_affn_names']
84		101
		102	WQ_BH = prog['WQ_BH']
85	WQ_UNBOUND = prog['WQ_UNBOUND']	103	WQ_UNBOUND = prog['WQ_UNBOUND']
86	WQ_ORDERED = prog['__WQ_ORDERED']	104	WQ_ORDERED = prog['__WQ_ORDERED']
87	WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM']	105	WQ_MEM_RECLAIM = prog['WQ_MEM_RECLAIM']
@@ -92,6 +110,11 @@ WQ_AFFN_CACHE = prog['WQ_AFFN_CACHE'] · 4 unmodified lines
92	WQ_AFFN_NUMA = prog['WQ_AFFN_NUMA']	110	WQ_AFFN_NUMA = prog['WQ_AFFN_NUMA']
93	WQ_AFFN_SYSTEM = prog['WQ_AFFN_SYSTEM']	111	WQ_AFFN_SYSTEM = prog['WQ_AFFN_SYSTEM']
94		112
		113	POOL_BH = prog['POOL_BH']
		114
		115	WQ_NAME_LEN = prog['WQ_NAME_LEN'].value_()
		116	cpumask_str_len = len(cpumask_str(wq_unbound_cpumask))
		117
95	print('Affinity Scopes')	118	print('Affinity Scopes')
96	print('===============')	119	print('===============')
97		120
@@ -133,10 +156,12 @@ for pi, pool in idr_for_each(worker_pool_idr): · 35 unmodified lines
133		156
134	for pi, pool in idr_for_each(worker_pool_idr):	157	for pi, pool in idr_for_each(worker_pool_idr):
135	pool = drgn.Object(prog, 'struct worker_pool', address=pool)	158	pool = drgn.Object(prog, 'struct worker_pool', address=pool)
136	print(f'pool[{pi:0{max_pool_id_len}}] ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')	159	print(f'pool[{pi:0{max_pool_id_len}}] flags=0x{pool.flags.value_():02x} ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
137	print(f'idle/workers={pool.nr_idle.value_():3}/{pool.nr_workers.value_():3} ', end='')	160	print(f'idle/workers={pool.nr_idle.value_():3}/{pool.nr_workers.value_():3} ', end='')
138	if pool.cpu >= 0:	161	if pool.cpu >= 0:
139	print(f'cpu={pool.cpu.value_():3}', end='')	162	print(f'cpu={pool.cpu.value_():3}', end='')
		163	if pool.flags & POOL_BH:
		164	print(' bh', end='')
140	else:	165	else:
141	print(f'cpus={cpumask_str(pool.attrs.cpumask)}', end='')	166	print(f'cpus={cpumask_str(pool.attrs.cpumask)}', end='')
142	print(f' pod_cpus={cpumask_str(pool.attrs.__pod_cpumask)}', end='')	167	print(f' pod_cpus={cpumask_str(pool.attrs.__pod_cpumask)}', end='')
@@ -148,24 +173,13 @@ print('') · 5 unmodified lines
148	print('Workqueue CPU -> pool')	173	print('Workqueue CPU -> pool')
149	print('=====================')	174	print('=====================')
150		175
151	print('[ workqueue \ type CPU', end='')	176	print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ {"type CPU":{wq_type_len}}', end='')
152	for cpu in for_each_possible_cpu(prog):	177	for cpu in for_each_possible_cpu(prog):
153	print(f' {cpu:{max_pool_id_len}}', end='')	178	print(f' {cpu:{max_pool_id_len}}', end='')
154	print(' dfl]')	179	print(' dfl]')
155		180
156	for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):	181	for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
157	print(f'{wq.name.string_().decode()[-24:]:24}', end='')	182	print(f'{wq.name.string_().decode():{WQ_NAME_LEN}} {wq_type_str(wq):10}', end='')
158	if wq.flags & WQ_UNBOUND:
159	if wq.flags & WQ_ORDERED:
160	print(' ordered ', end='')
161	else:
162	print(' unbound', end='')
163	if wq.unbound_attrs.affn_strict:
164	print(',S ', end='')
165	else:
166	print(' ', end='')
167	else:
168	print(' percpu ', end='')
169		183
170	for cpu in for_each_possible_cpu(prog):	184	for cpu in for_each_possible_cpu(prog):
171	pool_id = per_cpu_ptr(wq.cpu_pwq, cpu)[0].pool.id.value_()	185	pool_id = per_cpu_ptr(wq.cpu_pwq, cpu)[0].pool.id.value_()
@@ -175,3 +189,65 @@ for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_( · 3 unmodified lines
175	if wq.flags & WQ_UNBOUND:	189	if wq.flags & WQ_UNBOUND:
176	print(f' {wq.dfl_pwq.pool.id.value_():{max_pool_id_len}}', end='')	190	print(f' {wq.dfl_pwq.pool.id.value_():{max_pool_id_len}}', end='')
177	print('')	191	print('')
		192
		193	print('')
		194	print('Workqueue -> rescuer')
		195	print('====================')
		196
		197	ucpus_len = max(cpumask_str_len, len("unbound_cpus"))
		198	rcpus_len = max(cpumask_str_len, len("rescuer_cpus"))
		199
		200	print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ {"unbound_cpus":{ucpus_len}} pid {"rescuer_cpus":{rcpus_len}} ]')
		201
		202	for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
		203	if not (wq.flags & WQ_MEM_RECLAIM):
		204	continue
		205
		206	print(f'{wq.name.string_().decode():{WQ_NAME_LEN}}', end='')
		207	if wq.unbound_attrs.value_() != 0:
		208	print(f' {cpumask_str(wq.unbound_attrs.cpumask):{ucpus_len}}', end='')
		209	else:
		210	print(f' {"":{ucpus_len}}', end='')
		211
		212	print(f' {wq.rescuer.task.pid.value_():6}', end='')
		213	print(f' {cpumask_str(wq.rescuer.task.cpus_ptr):{rcpus_len}}', end='')
		214	print('')
		215
		216	print('')
		217	print('Unbound workqueue -> node_nr/max_active')
		218	print('=======================================')
		219
		220	if 'node_to_cpumask_map' in prog:
		221	__cpu_online_mask = prog['__cpu_online_mask']
		222	node_to_cpumask_map = prog['node_to_cpumask_map']
		223	nr_node_ids = prog['nr_node_ids'].value_()
		224
		225	print(f'online_cpus={cpumask_str(__cpu_online_mask.address_of_())}')
		226	for node in for_each_node():
		227	print(f'NODE[{node:02}]={cpumask_str(node_to_cpumask_map[node])}')
		228	print('')
		229
		230	print(f'[{"workqueue":^{WQ_NAME_LEN-2}}\\ min max', end='')
		231	first = True
		232	for node in for_each_node():
		233	if first:
		234	print(f' NODE {node}', end='')
		235	first = False
		236	else:
		237	print(f' {node:7}', end='')
		238	print(f' {"dfl":>7} ]')
		239	print('')
		240
		241	for wq in list_for_each_entry('struct workqueue_struct', workqueues.address_of_(), 'list'):
		242	if not (wq.flags & WQ_UNBOUND):
		243	continue
		244
		245	print(f'{wq.name.string_().decode():{WQ_NAME_LEN}} ', end='')
		246	print(f'{wq.min_active.value_():3} {wq.max_active.value_():3}', end='')
		247	for node in for_each_node():
		248	nna = wq.node_nr_active[node]
		249	print(f' {nna.nr.counter.value_():3}/{nna.max.value_():3}', end='')
		250	nna = wq.node_nr_active[nr_node_ids]
		251	print(f' {nna.nr.counter.value_():3}/{nna.max.value_():3}')
		252	else:
		253	printf(f'node_to_cpumask_map not present, is NUMA enabled?')