History

io_uring: defer eventfd signaling when queued from a wakeup handler

io_req_local_work_add() signals the CQ ring eventfd inline when it is the one to push the first entry onto ->work_list. For DEFER_TASKRUN rings that add is frequently done from a waitqueue wakeup handler, where an arbitrary waitqueue lock is held. eventfd_signal_mask() only refuses to recurse when current->in_eventfd is set, but that bit is set by eventfd_signal_mask() itself. If the wake chain starts somewhere else, signal goes out inline and can feed back into epoll. Add IOU_F_TWQ_IN_WAKE, set it on the task_work add done from the three waitqueue callbacks, and use it to force io_eventfd_signal() down the existing call_rcu_hurry() deferral instead of signaling inline.

Jens Axboe committed 3 months ago

Commit cd305ee1 parent 3f3a6a18 files · +30 −21Browse files

Showing the changes to include (1 of 8 files). Show all

Unified · Split

include/linux/io_uring_types.h+8−0
@@ -20,6 +20,14 @@ enum { · 19 unmodified lines
20 * It's also ignored unless IORING_SETUP_DEFER_TASKRUN is set.20 * It's also ignored unless IORING_SETUP_DEFER_TASKRUN is set.
21 */21 */
22 IOU_F_TWQ_LAZY_WAKE = 1,22 IOU_F_TWQ_LAZY_WAKE = 1,
23
24 /*
25 * Set when task_work is queued from a waitqueue wakeup handler, where
26 * an arbitrary provider waitqueue lock is held. Signaling the CQ ring
27 * eventfd inline from there can recurse back into that lock through
28 * epoll, so the eventfd signal must be deferred.
29 */
30 IOU_F_TWQ_IN_WAKE = 2,
23};31};
24 32
25enum io_uring_cmd_flags {33enum io_uring_cmd_flags {

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t