History

NFS/localio: issue IO inline when not in a memory-reclaim context

Every LOCALIO read and write is currently bounced through the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue. That bounce is only actually required when the submitting context is a memory-reclaim context: LOCALIO issues IO directly into a stacked local filesystem (e.g. XFS) which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing that from a WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and risks a forward-progress deadlock, which is why commit b9f5dd57f4a5 ("nfs/localio: use dedicated workqueues for filesystem read and write") introduced the intermediate workqueue. Outside of reclaim context -- ordinary application/task submission such as O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and merely adds a context switch and scheduling latency per IO while discarding the NFS client's inherent application-context parallelism. Add current_is_workqueue_mem_reclaim(), which reports whether %current is a WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency() warns on. Use it, together with the PF_MEMALLOC check, in the new nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read() and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue the IO inline. Buffered writeback continues to bounce (wb_workfn is a WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline. Running nfs_local_call_write() inline is safe: it already saves and restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it sets and scopes the file opener's creds. The async O_DIRECT completion path is likewise unaffected: when the underlying filesystem returns -EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete / nfs_local_write_aio_complete) can run in bottom-half context and so must still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to nfsiod_workqueue via nfs_local_pgio_aio_complete(). That completion hop is independent of how the IO was submitted, and this change leaves it as-is; only the submission side stops unconditionally hopping through nfslocaliod_workqueue.

Mike Snitzer committed 4 months ago

Commit da729dd1 parent 68c03753 files · +56 −2Browse files

Showing the changes to kernel/workqueue.c (1 of 3 files). Show all

Unified · Split

kernel/workqueue.c+24−0
@@ -6215,6 +6215,30 @@ bool current_is_workqueue_rescuer(void) · 6214 unmodified lines
62156215 return worker && worker->rescue_wq;
62166216 }
62176217
6218+/**
6219+ * current_is_workqueue_mem_reclaim - is %current a %WQ_MEM_RECLAIM worker?
6220+ *
6221+ * Determine whether %current is a workqueue worker executing on a workqueue
6222+ * created with %WQ_MEM_RECLAIM. This mirrors the condition that
6223+ * check_flush_dependency() warns on: flushing (or otherwise waiting on) a
6224+ * !WQ_MEM_RECLAIM workqueue from such a context breaks the forward-progress
6225+ * guarantee and can deadlock. Callers that may recurse into such a flush --
6226+ * e.g. NFS LOCALIO submitting into a stacked filesystem that flushes its own
6227+ * !WQ_MEM_RECLAIM workqueue -- can use this to decide whether they must defer
6228+ * the work to a !WQ_MEM_RECLAIM workqueue rather than run it inline.
6229+ *
6230+ * Return: %true if %current is a %WQ_MEM_RECLAIM worker. %false otherwise.
6231+ */
6232+bool current_is_workqueue_mem_reclaim(void)
6233+{
6234+ struct worker *worker = current_wq_worker();
6235+
6236+ return worker &&
6237+ ((worker->current_pwq->wq->flags &
6238+ (WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM);
6239+}
6240+EXPORT_SYMBOL_GPL(current_is_workqueue_mem_reclaim);
6241+
62186242 /**
62196243 * workqueue_congested - test whether a workqueue is congested
62206244 * @cpu: CPU in question

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t