Every LOCALIO read and write is currently bounced through the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue. That bounce is only actually required when the submitting context is a memory-reclaim context: LOCALIO issues IO directly into a stacked local filesystem (e.g. XFS) which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing that from a WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and risks a forward-progress deadlock, which is why commit b9f5dd57f4a5 ("nfs/localio: use dedicated workqueues for filesystem read and write") introduced the intermediate workqueue. Outside of reclaim context -- ordinary application/task submission such as O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and merely adds a context switch and scheduling latency per IO while discarding the NFS client's inherent application-context parallelism. Add current_is_workqueue_mem_reclaim(), which reports whether %current is a WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency() warns on. Use it, together with the PF_MEMALLOC check, in the new nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read() and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue the IO inline. Buffered writeback continues to bounce (wb_workfn is a WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline. Running nfs_local_call_write() inline is safe: it already saves and restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it sets and scopes the file opener's creds. The async O_DIRECT completion path is likewise unaffected: when the underlying filesystem returns -EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete / nfs_local_write_aio_complete) can run in bottom-half context and so must still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to nfsiod_workqueue via nfs_local_pgio_aio_complete(). That completion hop is independent of how the IO was submitted, and this change leaves it as-is; only the submission side stops unconditionally hopping through nfslocaliod_workqueue.
Showing the changes to kernel/workqueue.c (1 of 3 files). Show all
Unified · Split
| @@ -6215,6 +6215,30 @@ bool current_is_workqueue_rescuer(void) · 6214 unmodified lines | ||
| 6215 | 6215 | return worker && worker->rescue_wq; |
| 6216 | 6216 | } |
| 6217 | 6217 | |
| 6218 | +/** | |
| 6219 | + * current_is_workqueue_mem_reclaim - is %current a %WQ_MEM_RECLAIM worker? | |
| 6220 | + * | |
| 6221 | + * Determine whether %current is a workqueue worker executing on a workqueue | |
| 6222 | + * created with %WQ_MEM_RECLAIM. This mirrors the condition that | |
| 6223 | + * check_flush_dependency() warns on: flushing (or otherwise waiting on) a | |
| 6224 | + * !WQ_MEM_RECLAIM workqueue from such a context breaks the forward-progress | |
| 6225 | + * guarantee and can deadlock. Callers that may recurse into such a flush -- | |
| 6226 | + * e.g. NFS LOCALIO submitting into a stacked filesystem that flushes its own | |
| 6227 | + * !WQ_MEM_RECLAIM workqueue -- can use this to decide whether they must defer | |
| 6228 | + * the work to a !WQ_MEM_RECLAIM workqueue rather than run it inline. | |
| 6229 | + * | |
| 6230 | + * Return: %true if %current is a %WQ_MEM_RECLAIM worker. %false otherwise. | |
| 6231 | + */ | |
| 6232 | +bool current_is_workqueue_mem_reclaim(void) | |
| 6233 | +{ | |
| 6234 | + struct worker *worker = current_wq_worker(); | |
| 6235 | + | |
| 6236 | + return worker && | |
| 6237 | + ((worker->current_pwq->wq->flags & | |
| 6238 | + (WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM); | |
| 6239 | +} | |
| 6240 | +EXPORT_SYMBOL_GPL(current_is_workqueue_mem_reclaim); | |
| 6241 | + | |
| 6218 | 6242 | /** |
| 6219 | 6243 | * workqueue_congested - test whether a workqueue is congested |
| 6220 | 6244 | * @cpu: CPU in question |