hrtimer_get_next_event() runs on every tick-stop decision via get_next_timer_interrupt() -> cmp_next_hrtimer_event(). When high resolution timers are active it must return KTIME_MAX -- the caller documents and depends on this. The function takes cpu_base->lock, checks hres_active, and returns KTIME_MAX. Taking the lock is not required because cpu_base->hres_active is only written by the local CPU in hrtimer_switch_to_hres() from hard interrupt context and in hrtimers_cpu_starting() during bring-up. All callers of hrtimer_get_next_event() reach it from the tick-stop / cpuidle paths with interrupts disabled on that CPU. No writer can therefore race with the read, so an unlocked hres_active check is stable and the lock can be skipped in this case. On a 176-thread AMD EPYC 9D64 running a production workload, bucketing callers of native_queued_spin_lock_slowpath(), the slowpath had 199 samples in total, of which 54 are attributed to hrtimer_get_next_event(), i.e. this accounts for ~27% of slowpath hits on this specific workload. Move the hres_active check before the lock guard region to address this. [ tglx: Massaged change log and comments ]
Unified · Split
| @@ -1786,13 +1786,21 @@ EXPORT_SYMBOL_GPL(__hrtimer_get_remaining); · 1785 unmodified lines | |||
| 1786 | ktime_t hrtimer_get_next_event(void) | 1786 | ktime_t hrtimer_get_next_event(void) |
| 1787 | { | 1787 | { |
| 1788 | struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases); | 1788 | struct hrtimer_cpu_base *cpu_base = this_cpu_ptr(&hrtimer_bases); |
| 1789 | ktime_t expires = KTIME_MAX; | ||
| 1790 | 1789 | ||
| 1791 | guard(raw_spinlock_irqsave)(&cpu_base->lock); | 1790 | /* |
| 1792 | if (!hrtimer_hres_active(cpu_base)) | 1791 | * When HRES is active cmp_next_hrtimer_event() expects KTIME_MAX. |
| 1793 | expires = __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_ALL); | 1792 | * |
| 1793 | * cpu_base->hres_active is written only by the local CPU in | ||
| 1794 | * hrtimer_switch_to_hres() from hard interrupt context and in | ||
| 1795 | * hrtimers_cpu_starting() during CPU bring-up, and all callers reach | ||
| 1796 | * this with interrupts disabled on the same CPU, so an unlocked read is | ||
| 1797 | * stable without holding the lock. | ||
| 1798 | */ | ||
| 1799 | if (hrtimer_hres_active(cpu_base)) | ||
| 1800 | return KTIME_MAX; | ||
| 1794 | 1801 | ||
| 1795 | return expires; | 1802 | guard(raw_spinlock_irqsave)(&cpu_base->lock); |
| 1803 | return __hrtimer_get_next_event(cpu_base, HRTIMER_ACTIVE_ALL); | ||
| 1796 | } | 1804 | } |
| 1797 | 1805 | ||
| 1798 | /** | 1806 | /** |