Taking a cpu offline removes the cpu from the online mask before the CPU_DEAD notification is done. The clock events layer does the cleanup of the dead CPU from the CPU_DEAD notifier chain. tick_do_timer_cpu is used to avoid xtime lock contention by assigning the task of jiffies xtime updates to one CPU. If a CPU is taken offline, then this assignment becomes stale. This went unnoticed because most of the time the offline CPU went dead before the online CPU reached __cpu_die(), where the CPU_DEAD state is checked. In the case that the offline CPU did not reach the DEAD state before we reach __cpu_die(), the code in there goes to sleep for 100ms. Due to the stale time update assignment, the system is stuck forever. Take the assignment away when a cpu is not longer in the cpu_online_mask. We do this in the last call to tick_nohz_stop_sched_tick() when the offline CPU is on the way to the final play_dead() idle entry.
Showing the changes to kernel (1 of 1 file). Show all
Unified · Split
| @@ -160,6 +160,18 @@ void tick_nohz_stop_sched_tick(void) · 159 unmodified lines | ||
| 160 | 160 | cpu = smp_processor_id(); |
| 161 | 161 | ts = &per_cpu(tick_cpu_sched, cpu); |
| 162 | 162 | |
| 163 | + /* | |
| 164 | + * If this cpu is offline and it is the one which updates | |
| 165 | + * jiffies, then give up the assignment and let it be taken by | |
| 166 | + * the cpu which runs the tick timer next. If we don't drop | |
| 167 | + * this here the jiffies might be stale and do_timer() never | |
| 168 | + * invoked. | |
| 169 | + */ | |
| 170 | + if (unlikely(!cpu_online(cpu))) { | |
| 171 | + if (cpu == tick_do_timer_cpu) | |
| 172 | + tick_do_timer_cpu = -1; | |
| 173 | + } | |
| 174 | + | |
| 163 | 175 | if (unlikely(ts->nohz_mode == NOHZ_MODE_INACTIVE)) |
| 164 | 176 | goto end; |
| 165 | 177 | |