Entering RCU idle mode may cause a deferred wake up of an RCU NOCB_GP kthread (rcuog) to be serviced. Usually a local wake up happening while running the idle task is handled in one of the need_resched() checks carefully placed within the idle loop that can break to the scheduler. Unfortunately the call to rcu_idle_enter() is already beyond the last generic need_resched() check and we may halt the CPU with a resched request unhandled, leaving the task hanging. Fix this with splitting the rcuog wakeup handling from rcu_idle_enter() and place it before the last generic need_resched() check in the idle loop. It is then assumed that no call to call_rcu() will be performed after that in the idle loop until the CPU is put in low power mode.
Showing the changes to kernel (3 of 4 files). Show all
Unified · Split
| @@ -671,10 +671,7 @@ static noinstr void rcu_eqs_enter(bool user) · 670 unmodified lines | ||
| 671 | 671 | */ |
| 672 | 672 | void rcu_idle_enter(void) |
| 673 | 673 | { |
| 674 | − struct rcu_data *rdp = this_cpu_ptr(&rcu_data); | |
| 675 | − | |
| 676 | 674 | lockdep_assert_irqs_disabled(); |
| 677 | − do_nocb_deferred_wakeup(rdp); | |
| 678 | 675 | rcu_eqs_enter(false); |
| 679 | 676 | } |
| 680 | 677 | EXPORT_SYMBOL_GPL(rcu_idle_enter); |
| @@ -2187,6 +2187,11 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · 2186 unmodified lines | ||
| 2187 | 2187 | do_nocb_deferred_wakeup_common(rdp); |
| 2188 | 2188 | } |
| 2189 | 2189 | |
| 2190 | +void rcu_nocb_flush_deferred_wakeup(void) | |
| 2191 | +{ | |
| 2192 | + do_nocb_deferred_wakeup(this_cpu_ptr(&rcu_data)); | |
| 2193 | +} | |
| 2194 | + | |
| 2190 | 2195 | void __init rcu_init_nohz(void) |
| 2191 | 2196 | { |
| 2192 | 2197 | int cpu; |
| @@ -285,6 +285,7 @@ static void do_idle(void) · 284 unmodified lines | ||
| 285 | 285 | } |
| 286 | 286 | |
| 287 | 287 | arch_cpu_idle_enter(); |
| 288 | + rcu_nocb_flush_deferred_wakeup(); | |
| 288 | 289 | |
| 289 | 290 | /* |
| 290 | 291 | * In poll mode we reenable interrupts and spin. Also if we |