History

workqueue: Show all busy workers in stall diagnostics

show_cpu_pool_hog() only prints workers whose task is currently running on the CPU (task_is_running()). This misses workers that are busy processing a work item but are sleeping or blocked — for example, a worker that clears PF_WQ_WORKER and enters wait_event_idle(). Such a worker still occupies a pool slot and prevents progress, yet produces an empty backtrace section in the watchdog output. This is happening on real arm64 systems, where toggle_allocation_gate() IPIs every single CPU in the machine (which lacks NMI), causing workqueue stalls that show empty backtraces because toggle_allocation_gate() is sleeping in wait_event_idle(). Remove the task_is_running() filter so every in-flight worker in the pool's busy_hash is dumped. The busy_hash is protected by pool->lock, which is already held.

Breno Leitao committed 8 months ago

Commit 8823eae1 parent e8e14ac1 file · +13 −15Browse files

Unified · Split

kernel/workqueue.c+13−15
@@ -7583,9 +7583,9 @@ MODULE_PARM_DESC(panic_on_stall_time, "Panic if stall exceeds this many seconds · 7582 unmodified lines
7583 7583
7584/*7584/*
7585 * Show workers that might prevent the processing of pending work items.7585 * Show workers that might prevent the processing of pending work items.
7586 * The only candidates are CPU-bound workers in the running state.7586 * A busy worker that is not running on the CPU (e.g. sleeping in
7587 * Pending work items should be handled by another idle worker7587 * wait_event_idle() with PF_WQ_WORKER cleared) can stall the pool just as
7588 * in all other situations.7588 * effectively as a CPU-bound one, so dump every in-flight worker.
7589 */7589 */
7590static void show_cpu_pool_hog(struct worker_pool *pool)7590static void show_cpu_pool_hog(struct worker_pool *pool)
7591{7591{
@@ -7596,19 +7596,17 @@ static void show_cpu_pool_hog(struct worker_pool *pool) · 4 unmodified lines
7596 raw_spin_lock_irqsave(&pool->lock, irq_flags);7596 raw_spin_lock_irqsave(&pool->lock, irq_flags);
7597 7597
7598 hash_for_each(pool->busy_hash, bkt, worker, hentry) {7598 hash_for_each(pool->busy_hash, bkt, worker, hentry) {
7599 if (task_is_running(worker->task)) {7599 /*
7600 /*7600 * Defer printing to avoid deadlocks in console
7601 * Defer printing to avoid deadlocks in console7601 * drivers that queue work while holding locks
7602 * drivers that queue work while holding locks7602 * also taken in their write paths.
7603 * also taken in their write paths.7603 */
7604 */7604 printk_deferred_enter();
7605 printk_deferred_enter();
7606 7605
7607 pr_info("pool %d:\n", pool->id);7606 pr_info("pool %d:\n", pool->id);
7608 sched_show_task(worker->task);7607 sched_show_task(worker->task);
7609 7608
7610 printk_deferred_exit();7609 printk_deferred_exit();
7611 }
7612 }7610 }
7613 7611
7614 raw_spin_unlock_irqrestore(&pool->lock, irq_flags);7612 raw_spin_unlock_irqrestore(&pool->lock, irq_flags);
@@ -7619,7 +7617,7 @@ static void show_cpu_pools_hogs(void) · 4 unmodified lines
7619 struct worker_pool *pool;7617 struct worker_pool *pool;
7620 int pi;7618 int pi;
7621 7619
7622 pr_info("Showing backtraces of running workers in stalled CPU-bound worker pools:\n");7620 pr_info("Showing backtraces of busy workers in stalled CPU-bound worker pools:\n");
7623 7621
7624 rcu_read_lock();7622 rcu_read_lock();
7625 7623