The try_to_wake_up function has an optimization where it can queue a task for wakeup on its previous CPU, if the task is still in the middle of going to sleep inside schedule(). Once schedule() re-enables IRQs, the task will be woken up with an IPI, and placed back on the runqueue. If we have such a wakeup pending, there is no need to search other CPUs for runnable tasks. Just skip (or bail out early from) newidle balancing, and run the just woken up task. For a memcache like workload test, this reduces total CPU use by about 2%, proportionally split between user and system time, and p99 and p95 application response time by 10% on average. The schedstats run_delay number shows a similar improvement.
Showing the changes to kernel (1 of 1 file). Show all
Unified · Split
| @@ -10592,6 +10592,14 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) · 10591 unmodified lines | |||
| 10592 | u64 curr_cost = 0; | 10592 | u64 curr_cost = 0; |
| 10593 | 10593 | ||
| 10594 | update_misfit_status(NULL, this_rq); | 10594 | update_misfit_status(NULL, this_rq); |
| 10595 | |||
| 10596 | /* | ||
| 10597 | * There is a task waiting to run. No need to search for one. | ||
| 10598 | * Return 0; the task will be enqueued when switching to idle. | ||
| 10599 | */ | ||
| 10600 | if (this_rq->ttwu_pending) | ||
| 10601 | return 0; | ||
| 10602 | |||
| 10595 | /* | 10603 | /* |
| 10596 | * We must set idle_stamp _before_ calling idle_balance(), such that we | 10604 | * We must set idle_stamp _before_ calling idle_balance(), such that we |
| 10597 | * measure the duration of idle_balance() as idle time. | 10605 | * measure the duration of idle_balance() as idle time. |
| @@ -10657,7 +10665,8 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) · 59 unmodified lines | |||
| 10657 | * Stop searching for tasks to pull if there are | 10665 | * Stop searching for tasks to pull if there are |
| 10658 | * now runnable tasks on this rq. | 10666 | * now runnable tasks on this rq. |
| 10659 | */ | 10667 | */ |
| 10660 | if (pulled_task || this_rq->nr_running > 0) | 10668 | if (pulled_task || this_rq->nr_running > 0 || |
| 10669 | this_rq->ttwu_pending) | ||
| 10661 | break; | 10670 | break; |
| 10662 | } | 10671 | } |
| 10663 | rcu_read_unlock(); | 10672 | rcu_read_unlock(); |