io_req_local_work_add() signals the CQ ring eventfd inline when it is the one to push the first entry onto ->work_list. For DEFER_TASKRUN rings that add is frequently done from a waitqueue wakeup handler, where an arbitrary waitqueue lock is held. eventfd_signal_mask() only refuses to recurse when current->in_eventfd is set, but that bit is set by eventfd_signal_mask() itself. If the wake chain starts somewhere else, signal goes out inline and can feed back into epoll. Add IOU_F_TWQ_IN_WAKE, set it on the task_work add done from the three waitqueue callbacks, and use it to force io_eventfd_signal() down the existing call_rcu_hurry() deferral instead of signaling inline.
Showing the changes to include (1 of 8 files). Show all
Unified · Split
| @@ -20,6 +20,14 @@ enum { · 19 unmodified lines | |||
| 20 | * It's also ignored unless IORING_SETUP_DEFER_TASKRUN is set. | 20 | * It's also ignored unless IORING_SETUP_DEFER_TASKRUN is set. |
| 21 | */ | 21 | */ |
| 22 | IOU_F_TWQ_LAZY_WAKE = 1, | 22 | IOU_F_TWQ_LAZY_WAKE = 1, |
| 23 | |||
| 24 | /* | ||
| 25 | * Set when task_work is queued from a waitqueue wakeup handler, where | ||
| 26 | * an arbitrary provider waitqueue lock is held. Signaling the CQ ring | ||
| 27 | * eventfd inline from there can recurse back into that lock through | ||
| 28 | * epoll, so the eventfd signal must be deferred. | ||
| 29 | */ | ||
| 30 | IOU_F_TWQ_IN_WAKE = 2, | ||
| 23 | }; | 31 | }; |
| 24 | 32 | ||
| 25 | enum io_uring_cmd_flags { | 33 | enum io_uring_cmd_flags { |