The try_to_wake_up function has an optimization where it can queue a task for wakeup on its previous CPU, if the task is still in the middle of going to sleep inside schedule(). Once schedule() re-enables IRQs, the task will be woken up with an IPI, and placed back on the runqueue. If we have such a wakeup pending, there is no need to search other CPUs for runnable tasks. Just skip (or bail out early from) newidle balancing, and run the just woken up task. For a memcache like workload test, this reduces total CPU use by about 2%, proportionally split between user and system time, and p99 and p95 application response time by 10% on average. The schedstats run_delay number shows a similar improvement.
Showing the changes to kernel (1 of 1 file). Show all
Unified · Split
| @@ -10592,6 +10592,14 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) · 10591 unmodified lines | ||
| 10592 | 10592 | u64 curr_cost = 0; |
| 10593 | 10593 | |
| 10594 | 10594 | update_misfit_status(NULL, this_rq); |
| 10595 | + | |
| 10596 | + /* | |
| 10597 | + * There is a task waiting to run. No need to search for one. | |
| 10598 | + * Return 0; the task will be enqueued when switching to idle. | |
| 10599 | + */ | |
| 10600 | + if (this_rq->ttwu_pending) | |
| 10601 | + return 0; | |
| 10602 | + | |
| 10595 | 10603 | /* |
| 10596 | 10604 | * We must set idle_stamp _before_ calling idle_balance(), such that we |
| 10597 | 10605 | * measure the duration of idle_balance() as idle time. |
| @@ -10657,7 +10665,8 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) · 59 unmodified lines | ||
| 10657 | 10665 | * Stop searching for tasks to pull if there are |
| 10658 | 10666 | * now runnable tasks on this rq. |
| 10659 | 10667 | */ |
| 10660 | − if (pulled_task || this_rq->nr_running > 0) | |
| 10668 | + if (pulled_task || this_rq->nr_running > 0 || | |
| 10669 | + this_rq->ttwu_pending) | |
| 10661 | 10670 | break; |
| 10662 | 10671 | } |
| 10663 | 10672 | rcu_read_unlock(); |