Merge tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse

Pull fuse updates from Miklos Szeredi: - Improve performance of the io-uring transport by introducing buffer pools and zero-copy (Joanne) - Fix lots of bugs (Baokun Li) - Fix io-uring initialization issues (Joanne, Bernd) - More prep work for large folios (Joanne) - Don't limit buffered read to 128k (Jim Harris) - Fix zeroing of page end (dirtied with mmap) on file size extension (Jimmy Zuber) - Improve performance in certain cases with wake_up_sync() when queuing request (Xuewen Yan) - Misc fixes and cleanups (Xuewen Yan) * tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse: (35 commits) fuse: zero the partial EOF page when extending a file io_uring: Add missing include for ITER_SOURCE and ITER_DEST fuse: Fix the condition to enable over-io-uring fuse: invalidate the correct range after O_APPEND direct write selftests/fuse: test post-EOF page zeroing when a file is extended fuse: wake one waiter per freed slot when raising max_background fuse: use min_not_zero() in fuse_init_server_timeout() fuse: copy request headers via a stack buffer for io-uring fuse: give wakeup hints to the scheduler for synchronous requests fuse: check for NULL root inode in fuse_fill_super_submount fuse: reject a duplicate fd= mount option cuse: wait for pending RCU callbacks on module exit fuse: fix invalidate lock leak on open O_TRUNC DAX failure fuse: fix invalidate lock leak on setattr writeback failure fuse: wait for FR_FINISHED on abort_on_kill to prevent use-after-free fuse: make dentry_tree_work static docs: fuse: document io-uring buffer pool and zero-copy uapi fuse: add zero-copy over io-uring fuse: support registered buffer pools in io-uring fuse: add io-uring buffer pools ...

Linus Torvalds committed 2 months ago

Commit 9cebfe62 parents ce14fe4 34b5c4a27 files · +1474 −236Browse files

Unified · Split

27 files

Documentation/block/ublk.rst +7 −7
Documentation/filesystems/fuse/fuse-io-uring.rst +34 −2
Documentation/filesystems/fuse/index.rst +1 −0
Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst +126 −0
drivers/block/ublk_drv.c +11 −11
fs/fuse/args.h +2 −0
fs/fuse/cuse.c +7 −1
fs/fuse/dev.c +67 −37
fs/fuse/dev.h +1 −1
fs/fuse/dev_uring.c +541 −92
fs/fuse/dev_uring_i.h +55 −1
fs/fuse/dir.c +24 −13
fs/fuse/file.c +16 −4
fs/fuse/fuse_dev_i.h +5 −0
fs/fuse/fuse_i.h +2 −0
fs/fuse/inode.c +10 −6
fs/fuse/readdir.c +1 −1
fs/fuse/req_timeout.c +3 −9
include/linux/io_uring/cmd.h +32 −6
include/linux/io_uring_types.h +5 −0
include/uapi/linux/fuse.h +61 −2
io_uring/io_uring.c +1 −1
io_uring/rsrc.c +90 −37
io_uring/rsrc.h +0 −5
tools/testing/selftests/filesystems/fuse/.gitignore +1 −0
tools/testing/selftests/filesystems/fuse/Makefile +3 −0
tools/testing/selftests/filesystems/fuse/write_extend_eof_test.c +368 −0

Documentation/block/ublk.rst+7−7

@@ -382,17 +382,17 @@ Zero copy · 381 unmodified lines
382	382	---------
383	383
384	384	ublk zero copy relies on io_uring's fixed kernel buffer, which provides
385		−two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`.
	385	+two APIs: `io_buffer_register_request()` and `io_buffer_unregister`.
386	386
387	387	ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call
388		−`io_buffer_register_bvec()` for ublk server to register client request
	388	+`io_buffer_register_request()` for ublk server to register client request
389	389	buffer into io_uring buffer table, then ublk server can submit io_uring
390	390	IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF`
391		−calls `io_buffer_unregister_bvec()` to unregister the buffer, which is
392		−guaranteed to be live between calling `io_buffer_register_bvec()` and
393		−`io_buffer_unregister_bvec()`. Any io_uring operation which supports this
394		−kind of kernel buffer will grab one reference of the buffer until the
395		−operation is completed.
	391	+calls `io_buffer_unregister()` to unregister the buffer, which is guaranteed
	392	+to be live between calling `io_buffer_register_request()` and
	393	+`io_buffer_unregister()`. Any io_uring operation which supports this kind of
	394	+kernel buffer will grab one reference of the buffer until the operation is
	395	+completed.
396	396
397	397	ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and
398	398	be trusted, because it is ublk server's responsibility to make sure IO buffer

Documentation/filesystems/fuse/fuse-io-uring.rst+34−2

@@ -11,6 +11,9 @@ and works. For generic details about FUSE see fuse.rst. · 10 unmodified lines
11	11	This document also covers the current interface, which is
12	12	still in development and might change.
13	13
	14	+For the userspace protocol, see
	15	+Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst.
	16	+
14	17	Limitations
15	18	===========
16	19	As of now not all requests types are supported through io-uring, userspace
@@ -95,5 +98,34 @@ Sending requests with CQEs · 78 unmodified lines
95	98	\| <fuse_unlink() \|
96	99	\| <sys_unlink() \|
97	100
98		−
99		−
	101	+Buffer pools
	102	+============
	103	+
	104	+Without a buffer pool, every entry needs to pass a dedicated payload buffer
	105	+large enough for the maximum payload size. A buffer pool decouples entries
	106	+from payload buffers. The server hands the kernel one contiguous buffer pool
	107	+of memory and when the kernel sends the server a request, it indicates the
	108	+offset into the pool for that request's payload. Internally, the kernel is
	109	+able to manage/optimize the buffer pool memory however it likes.
	110	+
	111	+A server may also register the pool region with io_uring as a fixed buffer.
	112	+The backing pages are then pinned once, avoiding per-request pinning and
	113	+address translation. This also allows servers to use the same registered
	114	+buffers for subsequent backing store I/O through io-uring, keeping data
	115	+in the same pinned pages without additional pinning / mapping overhead.
	116	+
	117	+Zero-copy
	118	+=========
	119	+
	120	+Zero-copy lets the server read from / write to the client's pages (pinned
	121	+user pages for direct I/O, or page-cache folios for buffered I/O) without an
	122	+intermediary payload copy. This requires CAP_SYS_ADMIN privileges.
	123	+
	124	+When a fuse request arrives for a file that opted into zero-copy, the kernel
	125	+registers the relevant pages (pinned user pages for direct i/o or underlying
	126	+page cache folios for buffered i/o) into a sparse slot in the server's
	127	+io_uring registered buffer table. The server can then operate on these pages
	128	+directly using io-uring fixed buffer operations (eg read_fixed / write_fixed)
	129	+and the kernel unregisters these pages when the request completes.
	130	+Non-page-backed args (eg op out headers) will go through the payload buffer as
	131	+normal.

Documentation/filesystems/fuse/index.rst+1−0

@@ -12,3 +12,4 @@ FUSE (Filesystem in Userspace) Technical Documentation · 11 unmodified lines
12	12	fuse-io
13	13	fuse-io-uring
14	14	fuse-passthrough
	15	+ uapi/fuse-uapi-io-uring

Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst+126−0

@@ -0,0 +1,126 @@
	1	+.. SPDX-License-Identifier: GPL-2.0
	2	+
	3	+=====================================
	4	+FUSE-over-io-uring uapi documentation
	5	+=====================================
	6	+
	7	+Commands
	8	+========
	9	+
	10	+``enum fuse_uring_cmd``:
	11	+
	12	+``FUSE_IO_URING_CMD_ADD_QUEUE``
	13	+ Create a queue identified by ``fuse_uring_cmd_req.qid``. Queue-wide
	14	+ options are passed in ``fuse_uring_cmd_req.flags``:
	15	+
	16	+ ``FUSE_URING_ZERO_COPY``
	17	+ Enable zero-copy on this queue. Requires ``CAP_SYS_ADMIN`` and a buffer
	18	+ pool, which is added separately via ``ADD_BUFPOOL`` before registering
	19	+ entries (see `Zero-copy`_).
	20	+
	21	+``FUSE_IO_URING_CMD_ADD_BUFPOOL``
	22	+ Register the payload buffer pool for an existing queue. The server provides
	23	+ a single contiguous region in ``fuse_uring_cmd_req.bufpool.uaddr`` /
	24	+ ``.len``. This command must be issued after ``ADD_QUEUE`` and before
	25	+ registering any payload-carrying entries on that queue.
	26	+ ``fuse_uring_cmd_req.flags`` must be 0. Submitting this command with
	27	+ ``IORING_URING_CMD_FIXED`` marks the pool as registered, which avoids per
	28	+ i/o pinning/unpinning and mapping overhead (see `Buffer pools`_).
	29	+
	30	+``FUSE_IO_URING_CMD_REGISTER``
	31	+ Register a ring entry (a long-lived SQE that carries the request header
	32	+ iovec). For a zero-copy queue, ``fuse_uring_cmd_req.ent_zero_copy_buf_index``
	33	+ indicates the reserved registered buffer table slot this entry uses for
	34	+ zero-copy (see `Zero-copy`_).
	35	+
	36	+``FUSE_IO_URING_CMD_COMMIT_AND_FETCH``
	37	+ Commit the reply for a completed request and fetch the next one. The
	38	+ request is identified by ``fuse_uring_cmd_req.commit_id`` (the value the
	39	+ kernel reported in ``fuse_uring_ent_in_out.commit_id``).
	40	+
	41	+Structures
	42	+==========
	43	+
	44	+``struct fuse_uring_cmd_req`` (80-byte SQE command area):
	45	+
	46	+============================ ==================================================
	47	+Field Meaning
	48	+============================ ==================================================
	49	+``flags`` Command-specific flags (see each command).
	50	+``commit_id`` Request id, for ``COMMIT_AND_FETCH``.
	51	+``qid`` Queue index.
	52	+``bufpool.uaddr`` Pool base address, for ``ADD_BUFPOOL``.
	53	+``bufpool.len`` Pool length in bytes, for ``ADD_BUFPOOL``.
	54	+``bufpool.reserved`` Must be 0, for ``ADD_BUFPOOL``.
	55	+``ent_zero_copy_buf_index`` Per-entry zero-copy slot, for ``REGISTER``.
	56	+============================ ==================================================
	57	+
	58	+``struct fuse_uring_ent_in_out`` (reported by the kernel per request):
	59	+
	60	+============================ ==================================================
	61	+Field Meaning
	62	+============================ ==================================================
	63	+``flags`` ``FUSE_URING_ENT_ZERO_COPY`` if zero-copied.
	64	+``commit_id`` Id to echo back in ``COMMIT_AND_FETCH``.
	65	+``payload_sz`` Total payload size in bytes (see `Zero-copy`_).
	66	+``offset`` Payload buffer offset within the pool.
	67	+============================ ==================================================
	68	+
	69	+Buffer pools
	70	+============
	71	+Setup:
	72	+
	73	+* Issue ``ADD_QUEUE`` for the qid.
	74	+* Issue ``ADD_BUFPOOL`` with ``bufpool.uaddr`` and ``bufpool.len`` pointing
	75	+ at the region.
	76	+* Register entries with ``REGISTER``.
	77	+
	78	+For every request that has a payload, the kernel reports where the payload
	79	+lives in ``struct fuse_uring_ent_in_out`` (part of
	80	+``struct fuse_uring_req_header``):
	81	+
	82	+``offset``
	83	+ Byte offset, within the pool region, for this request's payload buffer.
	84	+ The server adds this to the pool base address to locate the payload.
	85	+
	86	+``payload_sz``
	87	+ Number of payload bytes for this request.
	88	+
	89	+To use registered buffers, the server registers the pool region with io_uring
	90	+and submits ``ADD_BUFPOOL`` with ``IORING_URING_CMD_FIXED`` set in
	91	+``sqe->uring_cmd_flags`` and the index of the registered bufpool in
	92	+``sqe->buf_index``. Every SQE the server submits afterwards must follow the
	93	+same fixed-buffer protocol, carrying ``IORING_URING_CMD_FIXED`` and that same
	94	+``sqe->buf_index``. The same registered buffer can be reused for the server's
	95	+backing-store I/O as well (e.g. ``IORING_OP_READ_FIXED`` /
	96	+``IORING_OP_WRITE_FIXED``).
	97	+
	98	+Zero-copy
	99	+=========
	100	+Requirements:
	101	+
	102	+* The server must be privileged (``CAP_SYS_ADMIN``).
	103	+* A zero-copy queue: ``ADD_QUEUE`` with the ``FUSE_URING_ZERO_COPY`` flag set.
	104	+* A buffer pool: ``ADD_BUFPOOL``.
	105	+* For each entry, ``REGISTER`` with ``ent_zero_copy_buf_index`` set to the
	106	+ index this entry uses in the server's io_uring registered-buffer table.
	107	+ This is where the kernel registers the request's pages for the server to
	108	+ access (it is separate from the payload pool). On a non-zero-copy queue this
	109	+ field must be 0.
	110	+
	111	+Zero-copy is selected per open file. The server sets the open-file flag in
	112	+the ``FUSE_OPEN`` / ``FUSE_CREATE`` reply:
	113	+
	114	+``FOPEN_IO_URING_ZERO_COPY``
	115	+ Reads/writes on this open file should use zero-copy.
	116	+
	117	+For a request that is zero-copied, the kernel sets ``FUSE_URING_ENT_ZERO_COPY``
	118	+in ``fuse_uring_ent_in_out.flags`` and places the request's pages at the
	119	+entry's ``ent_zero_copy_buf_index``. The server then issues
	120	+``IORING_OP_READ_FIXED`` / ``IORING_OP_WRITE_FIXED`` against that index to
	121	+transfer the data directly to/from the client's pages.
	122	+
	123	+For such a request, ``payload_sz`` includes the zero-copied page bytes
	124	+(transferred via the registered buffer at ``ent_zero_copy_buf_index``). Any
	125	+non-page-backed args (e.g. op headers) are still copied through the pool
	126	+payload buffer at ``offset``.

drivers/block/ublk_drv.c+11−11

@@ -1699,8 +1699,8 @@ ublk_auto_buf_register(const struct ublk_queue ubq, struct request req, · 1698 unmodified lines
1699	1699	{
1700	1700	int ret;
1701	1701
1702		− ret = io_buffer_register_bvec(cmd, req, ublk_io_release,
1703		− io->buf.auto_reg.index, issue_flags);
	1702	+ ret = io_buffer_register_request(cmd, req, ublk_io_release,
	1703	+ io->buf.auto_reg.index, issue_flags);
1704	1704	if (ret) {
1705	1705	if (io->buf.auto_reg.flags & UBLK_AUTO_BUF_REG_FALLBACK) {
1706	1706	ublk_auto_buf_reg_fallback(ubq, req->tag);
@@ -1909,7 +1909,7 @@ static noinline void ublk_batch_dispatch_fail(struct ublk_queue *ubq, · 202 unmodified lines
1909	1909	ublk_io_unlock(io);
1910	1910
1911	1911	if (index != -1)
1912		− io_buffer_unregister_bvec(data->cmd, index,
	1912	+ io_buffer_unregister(data->cmd, index,
1913	1913	data->issue_flags);
1914	1914	}
1915	1915
@@ -3192,8 +3192,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd, · 1276 unmodified lines
3192	3192	if (!req)
3193	3193	return -EINVAL;
3194	3194
3195		− ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,
3196		− issue_flags);
	3195	+ ret = io_buffer_register_request(cmd, req, ublk_io_release, index,
	3196	+ issue_flags);
3197	3197	if (ret) {
3198	3198	ublk_put_req_ref(io, req);
3199	3199	return ret;
@@ -3224,8 +3224,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd, · 24 unmodified lines
3224	3224	if (!ublk_dev_support_zero_copy(ub) \|\| !blk_rq_has_data(req))
3225	3225	return -EINVAL;
3226	3226
3227		− ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,
3228		− issue_flags);
	3227	+ ret = io_buffer_register_request(cmd, req, ublk_io_release, index,
	3228	+ issue_flags);
3229	3229	if (ret)
3230	3230	return ret;
3231	3231
@@ -3240,7 +3240,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd, · 8 unmodified lines
3240	3240	if (!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY))
3241	3241	return -EINVAL;
3242	3242
3243		− return io_buffer_unregister_bvec(cmd, index, issue_flags);
	3243	+ return io_buffer_unregister(cmd, index, issue_flags);
3244	3244	}
3245	3245
3246	3246	static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr)
@@ -3384,7 +3384,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, · 137 unmodified lines
3384	3384	goto out;
3385	3385
3386	3386	/*
3387		− * io_buffer_unregister_bvec() doesn't access the ubq or io,
	3387	+ * io_buffer_unregister() doesn't access the ubq or io,
3388	3388	* so no need to validate the q_id, tag, or task
3389	3389	*/
3390	3390	if (_IOC_NR(cmd_op) == UBLK_IO_UNREGISTER_IO_BUF)
@@ -3456,7 +3456,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, · 65 unmodified lines
3456	3456	req = ublk_fill_io_cmd(io, cmd);
3457	3457	ublk_apply_io_buf(ub, io, cmd, addr, &auto_buf, &buf_idx);
3458	3458	if (buf_idx != UBLK_INVALID_BUF_IDX)
3459		− io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
	3459	+ io_buffer_unregister(cmd, buf_idx, issue_flags);
3460	3460	compl = ublk_need_complete_req(ub, io);
3461	3461
3462	3462	if (req_op(req) == REQ_OP_ZONE_APPEND)
@@ -3801,7 +3801,7 @@ static int ublk_batch_commit_io(struct ublk_queue *ubq, · 338 unmodified lines
3801	3801	}
3802	3802
3803	3803	if (buf_idx != UBLK_INVALID_BUF_IDX)
3804		− io_buffer_unregister_bvec(data->cmd, buf_idx, data->issue_flags);
	3804	+ io_buffer_unregister(data->cmd, buf_idx, data->issue_flags);
3805	3805	if (req_op(req) == REQ_OP_ZONE_APPEND)
3806	3806	req->__sector = ublk_batch_zone_lba(uc, elem);
3807	3807	if (compl)

fs/fuse/args.h+2−0

@@ -42,6 +42,8 @@ struct fuse_args { · 41 unmodified lines
42	42	bool is_pinned:1;
43	43	bool invalidate_vmap:1;
44	44	bool abort_on_kill:1;
	45	+ /* server requested io-uring zero-copy for this op */
	46	+ bool zero_copy:1;
45	47	struct fuse_in_arg in_args[4];
46	48	struct fuse_arg out_args[2];
47	49	void (end)(struct fuse_args args, int error);

fs/fuse/cuse.c+7−1

@@ -530,7 +530,8 @@ static int cuse_channel_open(struct inode inode, struct file file) · 529 unmodified lines
530	530
531	531	INIT_LIST_HEAD(&cc->list);
532	532
533		− cc->fc.chan->initialized = 1;
	533	+ /* Pairs with smp_load_acquire() readers of fch->initialized */
	534	+ smp_store_release(&cc->fc.chan->initialized, 1);
534	535	rc = cuse_send_init(cc);
535	536	if (rc) {
536	537	fuse_dev_put(fud);
@@ -653,6 +654,11 @@ static void __exit cuse_exit(void) · 116 unmodified lines
653	654	{
654	655	misc_deregister(&cuse_miscdev);
655	656	class_destroy(cuse_class);
	657	+ /*
	658	+ * Wait for pending call_rcu() callbacks that call back into
	659	+ * this module via fc->release (cuse_fc_release).
	660	+ */
	661	+ rcu_barrier();
656	662	}
657	663
658	664	module_init(cuse_init);

fs/fuse/dev.c+67−37

@@ -75,17 +75,23 @@ void fuse_chan_set_initialized(struct fuse_chan fch, struct fuse_chan_param pa · 74 unmodified lines
75	75	fch->minor = param->minor;
76	76	fch->max_write = param->max_write;
77	77	fch->max_pages = param->max_pages;
	78	+
	79	+ if (param->io_uring_enabled)
	80	+ fuse_uring_conn_init(fch);
78	81	}
79	82
80		− /* Make sure stores before this are seen on another CPU */
81		− smp_wmb();
82		− fch->initialized = 1;
	83	+ /* Pairs with smp_load_acquire() readers of fch->initialized */
	84	+ smp_store_release(&fch->initialized, 1);
83	85	wake_up_all(&fch->blocked_waitq);
84	86	}
85	87
86	88	static bool fuse_block_alloc(struct fuse_chan *fch, bool for_background)
87	89	{
88		− return !fch->initialized \|\| (for_background && fch->blocked) \|\|
	90	+ /* Pairs with smp_store_release() in fuse_chan_set_initialized() */
	91	+ if (!smp_load_acquire(&fch->initialized))
	92	+ return true;
	93	+
	94	+ return (for_background && fch->blocked) \|\|
89	95	(fch->io_uring && fch->connected && !fuse_uring_ready(fch));
90	96	}
91	97
@@ -120,9 +126,6 @@ static struct fuse_req fuse_get_req(struct fuse_chan fch, bool for_background) · 28 unmodified lines
120	126	goto out;
121	127	}
122	128
123		− /* Matches smp_wmb() in fuse_chan_set_initialized() */
124		− smp_rmb();
125		−
126	129	err = -ENOTCONN;
127	130	if (!fch->connected)
128	131	goto out;
@@ -210,10 +213,13 @@ EXPORT_SYMBOL_GPL(fuse_req_hash); · 81 unmodified lines
210	213	/*
211	214	* A new request is available, wake fiq->waitq
212	215	*/
213		−static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq)
	216	+static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq, bool sync)
214	217	__releases(fiq->lock)
215	218	{
216		− wake_up(&fiq->waitq);
	219	+ if (sync)
	220	+ wake_up_sync(&fiq->waitq);
	221	+ else
	222	+ wake_up(&fiq->waitq);
217	223	kill_fasync(&fiq->fasync, SIGIO, POLL_IN);
218	224	spin_unlock(&fiq->lock);
219	225	}
@@ -230,7 +236,7 @@ void fuse_dev_queue_forget(struct fuse_iqueue *fiq, · 10 unmodified lines
230	236	if (fiq->connected) {
231	237	fiq->forget_list_tail->next = forget;
232	238	fiq->forget_list_tail = forget;
233		− fuse_dev_wake_and_unlock(fiq);
	239	+ fuse_dev_wake_and_unlock(fiq, false);
234	240	} else {
235	241	kfree(forget);
236	242	spin_unlock(&fiq->lock);
@@ -240,7 +246,8 @@ void fuse_dev_queue_forget(struct fuse_iqueue *fiq, · 3 unmodified lines
240	246	void fuse_dev_queue_interrupt(struct fuse_iqueue fiq, struct fuse_req req)
241	247	{
242	248	spin_lock(&fiq->lock);
243		− if (list_empty(&req->intr_entry)) {
	249	+ /* Repeat FR_SENT test after obtaining the lock to prevent race with fuse_resend() */
	250	+ if (list_empty(&req->intr_entry) && test_bit(FR_SENT, &req->flags)) {
244	251	list_add_tail(&req->intr_entry, &fiq->interrupts);
245	252	/*
246	253	* Pairs with smp_mb() implied by test_and_set_bit()
@@ -251,7 +258,7 @@ void fuse_dev_queue_interrupt(struct fuse_iqueue fiq, struct fuse_req req) · 4 unmodified lines
251	258	list_del_init(&req->intr_entry);
252	259	spin_unlock(&fiq->lock);
253	260	} else {
254		− fuse_dev_wake_and_unlock(fiq);
	261	+ fuse_dev_wake_and_unlock(fiq, false);
255	262	}
256	263	} else {
257	264	spin_unlock(&fiq->lock);
@@ -281,11 +288,13 @@ EXPORT_SYMBOL_GPL(fuse_request_assign_unique); · 23 unmodified lines
281	288
282	289	static void fuse_dev_queue_req(struct fuse_iqueue fiq, struct fuse_req req)
283	290	{
	291	+ bool sync = test_and_clear_bit(FR_SYNC_WAKEUP, &req->flags);
	292	+
284	293	spin_lock(&fiq->lock);
285	294	if (fiq->connected) {
286	295	fuse_request_assign_unique_locked(fiq, req);
287	296	list_add_tail(&req->list, &fiq->pending);
288		− fuse_dev_wake_and_unlock(fiq);
	297	+ fuse_dev_wake_and_unlock(fiq, sync);
289	298	} else {
290	299	spin_unlock(&fiq->lock);
291	300	req->out.h.error = -ENOTCONN;
@@ -397,7 +406,8 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val) · 105 unmodified lines
397	406	fch->max_background = val;
398	407	fch->blocked = fch->num_background >= fch->max_background;
399	408	if (!fch->blocked)
400		− wake_up(&fch->blocked_waitq);
	409	+ wake_up_nr(&fch->blocked_waitq,
	410	+ fch->max_background - fch->num_background);
401	411	spin_unlock(&fch->bg_lock);
402	412	}
403	413
@@ -411,11 +421,6 @@ void fuse_chan_set_fc(struct fuse_chan fch, struct fuse_conn fc) · 7 unmodified lines
411	421	fch->conn = fc;
412	422	}
413	423
414		−void fuse_chan_io_uring_enable(struct fuse_chan *fch)
415		−{
416		− fch->io_uring = 1;
417		−}
418		−
419	424	void fuse_pqueue_init(struct fuse_pqueue *fpq)
420	425	{
421	426	spin_lock_init(&fpq->lock);
@@ -725,7 +730,7 @@ static void request_wait_answer(struct fuse_req *req) · 303 unmodified lines
725	730
726	731	if (req->args->abort_on_kill) {
727	732	fuse_chan_abort(fch, false);
728		− return;
	733	+ goto wait_for_finish;
729	734	}
730	735
731	736	if (test_bit(FR_URING, &req->flags))
@@ -736,6 +741,7 @@ static void request_wait_answer(struct fuse_req *req) · 4 unmodified lines
736	741	return;
737	742	}
738	743
	744	+wait_for_finish:
739	745	/*
740	746	* Either request is already in userspace, or it was forced.
741	747	* Wait it out.
@@ -752,6 +758,11 @@ static void __fuse_request_send(struct fuse_req *req) · 10 unmodified lines
752	758	/* acquire extra reference, since request is still needed after
753	759	fuse_request_end() */
754	760	__fuse_get_request(req);
	761	+ /*
	762	+ * This is a synchronous request: the caller will block waiting for
	763	+ * the answer. Hint the scheduler via wake_up_sync().
	764	+ */
	765	+ set_bit(FR_SYNC_WAKEUP, &req->flags);
755	766	fuse_send_one(fiq, req);
756	767
757	768	request_wait_answer(req);
@@ -1249,11 +1260,25 @@ int fuse_copy_folio(struct fuse_copy_state cs, struct folio *foliop, · 491 unmodified lines
1249	1260
1250	1261	if (folio) {
1251	1262	size = folio_size(folio);
1252		− if (zeroing && count < size)
1253		− folio_zero_range(folio, 0, size);
	1263	+ if (zeroing && count < size) {
	1264	+ /*
	1265	+ * When the copy is skipped the folio already holds the
	1266	+ * payload, so only the bytes outside [offset, offset +
	1267	+ * count) may be zeroed.
	1268	+ *
	1269	+ * Otherwise, the whole folio is cleared first so that a
	1270	+ * failed copy leaves zeros rather than stale folio
	1271	+ * contents.
	1272	+ */
	1273	+ if (cs->skip_folio_copy)
	1274	+ folio_zero_segments(folio, 0, offset,
	1275	+ offset + count, size);
	1276	+ else
	1277	+ folio_zero_range(folio, 0, size);
	1278	+ }
1254	1279	}
1255	1280
1256		− while (count) {
	1281	+ while (!cs->skip_folio_copy && count) {
1257	1282	if (cs->write && cs->pipebufs && folio) {
1258	1283	/*
1259	1284	* Can't control lifetime of pipe buffers, so always
@@ -1346,6 +1371,10 @@ int fuse_copy_args(struct fuse_copy_state *cs, unsigned numargs, · 86 unmodified lines
1346	1371	for (i = 0; !err && i < numargs; i++) {
1347	1372	struct fuse_arg *arg = &args[i];
1348	1373	if (i == numargs - 1 && argpages)
	1374	+ /*
	1375	+ * if cs->skip_folio_copy is set, this just does any
	1376	+ * needed zeroing. No copying is involved.
	1377	+ */
1349	1378	err = fuse_copy_folios(cs, arg->size, zeroing);
1350	1379	else
1351	1380	err = fuse_copy_one(cs, arg->value, arg->size);
@@ -1760,7 +1789,7 @@ out: · 408 unmodified lines
1760	1789	void fuse_chan_resend(struct fuse_chan *fch)
1761	1790	{
1762	1791	struct fuse_dev *fud;
1763		− struct fuse_req req, next;
	1792	+ struct fuse_req *req;
1764	1793	struct fuse_iqueue *fiq = &fch->iq;
1765	1794	LIST_HEAD(to_queue);
1766	1795	unsigned int i;
@@ -1775,24 +1804,20 @@ void fuse_chan_resend(struct fuse_chan *fch) · 8 unmodified lines
1775	1804	struct fuse_pqueue *fpq = &fud->pq;
1776	1805
1777	1806	spin_lock(&fpq->lock);
1778		− for (i = 0; i < FUSE_PQ_HASH_SIZE; i++)
1779		− list_splice_tail_init(&fpq->processing[i], &to_queue);
	1807	+ for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) {
	1808	+ struct list_head *this_queue = &fpq->processing[i];
	1809	+
	1810	+ list_for_each_entry(req, this_queue, list)
	1811	+ clear_bit(FR_SENT, &req->flags);
	1812	+ list_splice_tail_init(this_queue, &to_queue);
	1813	+ }
1780	1814	spin_unlock(&fpq->lock);
1781	1815	}
1782	1816	spin_unlock(&fch->lock);
1783	1817
1784		− list_for_each_entry_safe(req, next, &to_queue, list) {
1785		− set_bit(FR_PENDING, &req->flags);
1786		− clear_bit(FR_SENT, &req->flags);
1787		− /* mark the request as resend request */
1788		− req->in.h.unique \|= FUSE_UNIQUE_RESEND;
1789		− }
1790		−
1791	1818	spin_lock(&fiq->lock);
1792	1819	if (!fiq->connected) {
1793	1820	spin_unlock(&fiq->lock);
1794		− list_for_each_entry(req, &to_queue, list)
1795		− clear_bit(FR_PENDING, &req->flags);
1796	1821	fuse_dev_end_requests(&to_queue);
1797	1822	return;
1798	1823	}
@@ -1801,12 +1826,16 @@ void fuse_chan_resend(struct fuse_chan *fch) · 2 unmodified lines
1801	1826	* intr_entry on fiq->interrupts after the request is re-queued.
1802	1827	*/
1803	1828	list_for_each_entry(req, &to_queue, list) {
	1829	+ set_bit(FR_PENDING, &req->flags);
	1830	+ /* mark the request as resend request */
	1831	+ req->in.h.unique \|= FUSE_UNIQUE_RESEND;
	1832	+
1804	1833	if (test_bit(FR_INTERRUPTED, &req->flags))
1805	1834	list_del_init(&req->intr_entry);
1806	1835	}
1807	1836	/* iq and pq requests are both oldest to newest */
1808	1837	list_splice(&to_queue, &fiq->pending);
1809		− fuse_dev_wake_and_unlock(fiq);
	1838	+ fuse_dev_wake_and_unlock(fiq, false);
1810	1839	}
1811	1840
1812	1841	/* Look up request on processing list by unique ID */
@@ -1888,7 +1917,8 @@ static ssize_t fuse_dev_do_write(struct fuse_dev *fud, · 75 unmodified lines
1888	1917	* initialized and connected state
1889	1918	*/
1890	1919	err = -EINVAL;
1891		− if (!fch->initialized \|\| !fch->connected)
	1920	+ /* Pairs with smp_store_release() in fuse_chan_set_initialized() */
	1921	+ if (!smp_load_acquire(&fch->initialized) \|\| !fch->connected)
1892	1922	goto copy_finish;
1893	1923
1894	1924	/* Don't try to move folios (yet) */

fs/fuse/dev.h+1−1

@@ -22,6 +22,7 @@ struct fuse_chan_param { · 21 unmodified lines
22	22	unsigned int minor;
23	23	unsigned int max_write;
24	24	unsigned int max_pages;
	25	+ bool io_uring_enabled;
25	26	};
26	27
27	28	struct fuse_chan *fuse_chan_new(void);
@@ -34,7 +35,6 @@ void fuse_chan_max_background_set(struct fuse_chan *fch, unsigned int val); · 6 unmodified lines
34	35	unsigned int fuse_chan_num_waiting(struct fuse_chan *fch);
35	36	void fuse_chan_set_fc(struct fuse_chan fch, struct fuse_conn fc);
36	37	void fuse_chan_set_initialized(struct fuse_chan fch, struct fuse_chan_param param);
37		−void fuse_chan_io_uring_enable(struct fuse_chan *fch);
38	38	ssize_t fuse_chan_send(struct fuse_chan fch, struct fuse_args args);
39	39	int fuse_chan_send_bg(struct fuse_chan fch, struct fuse_args args, gfp_t gfp_flags);
40	40	int fuse_chan_send_notify_reply(struct fuse_chan fch, struct fuse_args args, u64 unique);

fs/fuse/dev_uring.c+541−92

@@ -9,6 +9,7 @@ · 8 unmodified lines
9	9	#include "dev_uring_i.h"
10	10	#include "fuse_trace.h"
11	11
	12	+#include <linux/bitmap.h>
12	13	#include <linux/fs.h>
13	14	#include <linux/io_uring/cmd.h>
14	15
@@ -21,6 +22,8 @@ MODULE_PARM_DESC(enable_uring, · 6 unmodified lines
21	22	#define FUSE_URING_IOV_HEADERS 0
22	23	#define FUSE_URING_IOV_PAYLOAD 1
23	24
	25	+#define FUSE_URING_ADD_QUEUE_FLAGS (FUSE_URING_ZERO_COPY)
	26	+
24	27	bool fuse_uring_enabled(void)
25	28	{
26	29	return enable_uring;
@@ -30,6 +33,11 @@ struct fuse_uring_pdu { · 3 unmodified lines
30	33	struct fuse_ring_ent *ent;
31	34	};
32	35
	36	+struct fuse_zero_copy_bvs {
	37	+ unsigned int nr_bvs;
	38	+ struct bio_vec bvs[];
	39	+};
	40	+
33	41	static const struct fuse_iqueue_ops fuse_io_uring_ops;
34	42
35	43	enum fuse_uring_header_type {
@@ -41,6 +49,32 @@ enum fuse_uring_header_type { · 5 unmodified lines
41	49	FUSE_URING_HEADER_RING_ENT,
42	50	};
43	51
	52	+static inline bool bufpool_enabled(struct fuse_ring_queue *queue)
	53	+{
	54	+ return queue->payload_mode == FUSE_PAYLOAD_BUFPOOL;
	55	+}
	56	+
	57	+static inline bool bufpool_registered(struct fuse_ring_queue *queue)
	58	+{
	59	+ return queue->bufpool && queue->bufpool->registered;
	60	+}
	61	+
	62	+/*
	63	+ * For a registered bufpool, every sqe that drives a payload import (REGISTER,
	64	+ * COMMIT_AND_FETCH) must carry the registered buffer index of the pool.
	65	+ * This also must be called from the command's issue handler, where cmd->sqe is
	66	+ * still valid
	67	+ */
	68	+static inline bool fuse_uring_cmd_index_ok(struct io_uring_cmd *cmd,
	69	+ struct fuse_ring_queue *queue)
	70	+{
	71	+ if (!bufpool_registered(queue))
	72	+ return true;
	73	+
	74	+ return (cmd->flags & IORING_URING_CMD_FIXED) &&
	75	+ READ_ONCE(cmd->sqe->buf_index) == queue->bufpool->registered_index;
	76	+}
	77	+
44	78	static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
45	79	struct fuse_ring_ent *ring_ent)
46	80	{
@@ -86,8 +120,36 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue) · 39 unmodified lines
86	120	}
87	121	}
88	122
	123	+static bool can_zero_copy_req(struct fuse_ring_ent ent, struct fuse_req req)
	124	+{
	125	+ struct fuse_args *args = req->args;
	126	+
	127	+ if (!ent->queue->zero_copy \|\| !args->zero_copy)
	128	+ return false;
	129	+
	130	+ if (args->opcode != FUSE_READ && args->opcode != FUSE_WRITE)
	131	+ return false;
	132	+
	133	+ return args->in_pages \|\| args->out_pages;
	134	+}
	135	+
	136	+static void zero_copy_unregister(struct io_uring_cmd *cmd,
	137	+ struct fuse_ring_ent *ent,
	138	+ unsigned int issue_flags)
	139	+{
	140	+ if (ent->zero_copied) {
	141	+ int err = io_buffer_unregister(cmd, ent->zero_copy_index,
	142	+ issue_flags);
	143	+
	144	+ if (err)
	145	+ pr_warn_ratelimited("qid=%d zero-copy unregister failed: %d\n",
	146	+ ent->queue->qid, err);
	147	+ ent->zero_copied = false;
	148	+ }
	149	+}
	150	+
89	151	static void fuse_uring_req_end(struct fuse_ring_ent ent, struct fuse_req req,
90		− int error)
	152	+ int error, unsigned int issue_flags)
91	153	{
92	154	struct fuse_ring_queue *queue = ent->queue;
93	155	struct fuse_ring *ring = queue->ring;
@@ -107,6 +169,8 @@ static void fuse_uring_req_end(struct fuse_ring_ent ent, struct fuse_req req, · 13 unmodified lines
107	169
108	170	spin_unlock(&queue->lock);
109	171
	172	+ zero_copy_unregister(ent->cmd, ent, issue_flags);
	173	+
110	174	if (error)
111	175	req->out.h.error = error;
112	176
@@ -204,7 +268,7 @@ void fuse_uring_destruct(struct fuse_chan *fch) · 91 unmodified lines
204	268	return;
205	269
206	270	for (qid = 0; qid < ring->nr_queues; qid++) {
207		− struct fuse_ring_queue *queue = ring->queues[qid];
	271	+ struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]);
208	272	struct fuse_ring_ent ent, next;
209	273
210	274	if (!queue)
@@ -222,8 +286,9 @@ void fuse_uring_destruct(struct fuse_chan *fch) · 11 unmodified lines
222	286	}
223	287
224	288	kfree(queue->fpq.processing);
	289	+ kfree(queue->bufpool);
225	290	kfree(queue);
226		− ring->queues[qid] = NULL;
	291	+ WRITE_ONCE(ring->queues[qid], NULL);
227	292	}
228	293
229	294	kfree(ring->queues);
@@ -238,7 +303,6 @@ static struct fuse_ring fuse_uring_create(struct fuse_chan fch) · 8 unmodified lines
238	303	{
239	304	struct fuse_ring *ring;
240	305	size_t nr_queues = num_possible_cpus();
241		− struct fuse_ring *res = NULL;
242	306	size_t max_payload_size;
243	307
244	308	ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT);
@@ -258,12 +322,6 @@ static struct fuse_ring fuse_uring_create(struct fuse_chan fch) · 13 unmodified lines
258	322	spin_unlock(&fch->lock);
259	323	goto out_err;
260	324	}
261		− if (fch->ring) {
262		− /* race, another thread created the ring in the meantime */
263		− spin_unlock(&fch->lock);
264		− res = fch->ring;
265		− goto out_err;
266		− }
267	325
268	326	init_waitqueue_head(&ring->stop_waitq);
269	327
@@ -278,11 +336,18 @@ static struct fuse_ring fuse_uring_create(struct fuse_chan fch) · 8 unmodified lines
278	336	out_err:
279	337	kfree(ring->queues);
280	338	kfree(ring);
281		− return res;
	339	+ return NULL;
	340	+}
	341	+
	342	+void fuse_uring_conn_init(struct fuse_chan *fch)
	343	+{
	344	+ if (fuse_uring_create(fch))
	345	+ fch->io_uring = 1;
282	346	}
283	347
284	348	static struct fuse_ring_queue fuse_uring_create_queue(struct fuse_ring ring,
285		− int qid)
	349	+ int qid, bool zero_copy,
	350	+ bool fail_if_exists)
286	351	{
287	352	struct fuse_chan *fch = ring->chan;
288	353	struct fuse_ring_queue *queue;
@@ -290,16 +355,17 @@ static struct fuse_ring_queue fuse_uring_create_queue(struct fuse_ring ring, · 1 unmodified line
290	355
291	356	queue = kzalloc_obj(*queue, GFP_KERNEL_ACCOUNT);
292	357	if (!queue)
293		− return NULL;
	358	+ return ERR_PTR(-ENOMEM);
294	359	pq = fuse_pqueue_alloc();
295	360	if (!pq) {
296	361	kfree(queue);
297		− return NULL;
	362	+ return ERR_PTR(-ENOMEM);
298	363	}
299	364
300	365	queue->qid = qid;
301	366	queue->ring = ring;
302	367	spin_lock_init(&queue->lock);
	368	+ queue->zero_copy = zero_copy;
303	369
304	370	INIT_LIST_HEAD(&queue->ent_avail_queue);
305	371	INIT_LIST_HEAD(&queue->ent_commit_queue);
@@ -316,14 +382,17 @@ static struct fuse_ring_queue fuse_uring_create_queue(struct fuse_ring ring, · 10 unmodified lines
316	382	if (ring->queues[qid]) {
317	383	spin_unlock(&fch->lock);
318	384	kfree(queue->fpq.processing);
	385	+ kfree(queue->bufpool);
319	386	kfree(queue);
320		− return ring->queues[qid];
	387	+ return fail_if_exists ? ERR_PTR(-EEXIST) : ring->queues[qid];
321	388	}
322	389
323	390	/*
324		− * write_once and lock as the caller mostly doesn't take the lock at all
	391	+ * fch->lock serializes concurrent creators for this qid.
	392	+ * smp_store_release() are for the lockless readers who must see a
	393	+ * fully initialized queue after &ring->queues[qid] is set
325	394	*/
326		− WRITE_ONCE(ring->queues[qid], queue);
	395	+ smp_store_release(&ring->queues[qid], queue);
327	396	spin_unlock(&fch->lock);
328	397
329	398	return queue;
@@ -434,7 +503,7 @@ static void fuse_uring_log_ent_state(struct fuse_ring *ring) · 104 unmodified lines
434	503	struct fuse_ring_ent *ent;
435	504
436	505	for (qid = 0; qid < ring->nr_queues; qid++) {
437		− struct fuse_ring_queue *queue = ring->queues[qid];
	506	+ struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]);
438	507
439	508	if (!queue)
440	509	continue;
@@ -643,30 +712,57 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent, · 202 unmodified lines
643	712	return 0;
644	713	}
645	714
	715	+static int fuse_uring_import_payload(struct fuse_ring_ent *ent, int dir,
	716	+ struct iov_iter *iter,
	717	+ unsigned int issue_flags)
	718	+{
	719	+ void __user *base = ent->payload.iov_base;
	720	+ size_t len = ent->payload.iov_len;
	721	+ int err = 0;
	722	+
	723	+ if (!base) {
	724	+ memset(iter, 0, sizeof(*iter));
	725	+ return 0;
	726	+ }
	727	+
	728	+ if (bufpool_registered(ent->queue))
	729	+ err = io_uring_cmd_import_fixed((u64)(uintptr_t)base, len, dir,
	730	+ iter, ent->cmd, issue_flags);
	731	+ else
	732	+ err = import_ubuf(dir, base, len, iter);
	733	+
	734	+ if (err)
	735	+ pr_info_ratelimited("fuse: Import of user buffer failed\n");
	736	+
	737	+ return err;
	738	+}
	739	+
646	740	static int setup_fuse_copy_state(struct fuse_copy_state *cs,
647		− struct fuse_ring ring, struct fuse_req req,
	741	+ struct fuse_req *req,
648	742	struct fuse_ring_ent *ent, int dir,
649		− struct iov_iter *iter)
	743	+ struct iov_iter *iter,
	744	+ unsigned int issue_flags)
650	745	{
651	746	int err;
652	747
653		− err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);
654		− if (err) {
655		− pr_info_ratelimited("fuse: Import of user buffer failed\n");
	748	+ err = fuse_uring_import_payload(ent, dir, iter, issue_flags);
	749	+ if (err)
656	750	return err;
657		− }
658	751
659	752	fuse_copy_init(cs, dir == ITER_DEST, iter);
660	753
	754	+ if (ent->zero_copied)
	755	+ cs->skip_folio_copy = true;
	756	+
661	757	cs->is_uring = true;
662	758	cs->req = req;
663	759
664	760	return 0;
665	761	}
666	762
667		−static int fuse_uring_copy_from_ring(struct fuse_ring *ring,
668		− struct fuse_req *req,
669		− struct fuse_ring_ent *ent)
	763	+static int fuse_uring_copy_from_ring(struct fuse_req *req,
	764	+ struct fuse_ring_ent *ent,
	765	+ unsigned int issue_flags)
670	766	{
671	767	struct fuse_copy_state cs;
672	768	struct fuse_args *args = req->args;
@@ -679,7 +775,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring, · 6 unmodified lines
679	775	if (err)
680	776	return err;
681	777
682		− err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_SOURCE, &iter);
	778	+ err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter,
	779	+ issue_flags);
683	780	if (err)
684	781	return err;
685	782
@@ -688,11 +785,68 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring, · 2 unmodified lines
688	785	return err;
689	786	}
690	787
	788	+static void fuse_zero_copy_release(void *priv)
	789	+{
	790	+ struct fuse_zero_copy_bvs *zc_bvs = priv;
	791	+ unsigned int i;
	792	+
	793	+ for (i = 0; i < zc_bvs->nr_bvs; i++)
	794	+ folio_put(page_folio(zc_bvs->bvs[i].bv_page));
	795	+
	796	+ kvfree(zc_bvs);
	797	+}
	798	+
	799	+static int fuse_uring_set_up_zero_copy(struct fuse_ring_ent *ent,
	800	+ struct fuse_req *req,
	801	+ unsigned int issue_flags)
	802	+{
	803	+ struct fuse_args_pages *ap;
	804	+ int err, i, ddir = 0;
	805	+ struct fuse_zero_copy_bvs *zc_bvs;
	806	+ struct bio_vec *bvs;
	807	+
	808	+ /* out_pages indicates a read, in_pages indicates a write */
	809	+ if (req->args->out_pages)
	810	+ ddir \|= IO_BUF_DEST;
	811	+ if (req->args->in_pages)
	812	+ ddir \|= IO_BUF_SOURCE;
	813	+
	814	+ ap = container_of(req->args, typeof(*ap), args);
	815	+
	816	+ zc_bvs = kvmalloc_flex(*zc_bvs, bvs, ap->num_folios,
	817	+ GFP_KERNEL_ACCOUNT);
	818	+ if (!zc_bvs)
	819	+ return -ENOMEM;
	820	+
	821	+ zc_bvs->nr_bvs = ap->num_folios;
	822	+ bvs = zc_bvs->bvs;
	823	+ for (i = 0; i < ap->num_folios; i++) {
	824	+ bvs[i].bv_page = folio_page(ap->folios[i], 0);
	825	+ bvs[i].bv_offset = ap->descs[i].offset;
	826	+ bvs[i].bv_len = ap->descs[i].length;
	827	+ folio_get(ap->folios[i]);
	828	+ }
	829	+
	830	+ err = io_buffer_register_bvec(ent->cmd, bvs, ap->num_folios,
	831	+ fuse_zero_copy_release, zc_bvs,
	832	+ ddir, ent->zero_copy_index,
	833	+ issue_flags);
	834	+ if (err) {
	835	+ fuse_zero_copy_release(zc_bvs);
	836	+ return err;
	837	+ }
	838	+
	839	+ ent->zero_copied = true;
	840	+
	841	+ return 0;
	842	+}
	843	+
691	844	/*
692	845	* Copy data from the req to the ring buffer
693	846	*/
694		−static int fuse_uring_args_to_ring(struct fuse_ring ring, struct fuse_req req,
695		− struct fuse_ring_ent *ent)
	847	+static int fuse_uring_args_to_ring(struct fuse_req *req,
	848	+ struct fuse_ring_ent *ent,
	849	+ unsigned int issue_flags)
696	850	{
697	851	struct fuse_copy_state cs;
698	852	struct fuse_args *args = req->args;
@@ -705,7 +859,15 @@ static int fuse_uring_args_to_ring(struct fuse_ring ring, struct fuse_req req, · 6 unmodified lines
705	859	.commit_id = req->in.h.unique,
706	860	};
707	861
708		− err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_DEST, &iter);
	862	+ if (can_zero_copy_req(ent, req)) {
	863	+ ent_in_out.flags \|= FUSE_URING_ENT_ZERO_COPY;
	864	+ err = fuse_uring_set_up_zero_copy(ent, req, issue_flags);
	865	+ if (err)
	866	+ return err;
	867	+ }
	868	+
	869	+ err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter,
	870	+ issue_flags);
709	871	if (err)
710	872	return err;
711	873
@@ -735,15 +897,32 @@ static int fuse_uring_args_to_ring(struct fuse_ring ring, struct fuse_req req, · 23 unmodified lines
735	897	}
736	898
737	899	ent_in_out.payload_sz = cs.ring.copied_sz;
	900	+ /*
	901	+ * on a zero-copied write the pages are registered for the server to
	902	+ * read via a fixed-buffer op rather than copied into the payload
	903	+ * buffer, so copied_sz does not account for it. The server still needs
	904	+ * the total inbound size to know how many bytes to read from the
	905	+ * registered buffer, so add the page arg (always the last in-arg) back
	906	+ * in
	907	+ */
	908	+ if (cs.skip_folio_copy && args->in_pages)
	909	+ ent_in_out.payload_sz +=
	910	+ args->in_args[args->in_numargs - 1].size;
	911	+
	912	+ if (bufpool_enabled(ent->queue) && ent->payload.iov_base)
	913	+ ent_in_out.offset =
	914	+ (uintptr_t)ent->payload.iov_base - ent->queue->bufpool->base_uaddr;
	915	+
738	916	return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
739	917	&ent_in_out, sizeof(ent_in_out));
740	918	}
741	919
742	920	static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
743		− struct fuse_req *req)
	921	+ struct fuse_req *req,
	922	+ unsigned int issue_flags)
744	923	{
745	924	struct fuse_ring_queue *queue = ent->queue;
746		− struct fuse_ring *ring = queue->ring;
	925	+ struct fuse_in_header in_header;
747	926	int err;
748	927
749	928	err = -EIO;
@@ -758,23 +937,124 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · 8 unmodified lines
758	937	return err;
759	938
760	939	/* copy the request */
761		− err = fuse_uring_args_to_ring(ring, req, ent);
	940	+ err = fuse_uring_args_to_ring(req, ent, issue_flags);
762	941	if (unlikely(err)) {
763	942	pr_info_ratelimited("Copy to ring failed: %d\n", err);
764	943	return err;
765	944	}
766	945
767	946	/* copy fuse_in_header */
768		− return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->in.h,
769		− sizeof(req->in.h));
	947	+ in_header = req->in.h;
	948	+ return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &in_header,
	949	+ sizeof(in_header));
	950	+}
	951	+
	952	+static bool fuse_uring_req_has_copyable_payload(struct fuse_ring_ent *ent,
	953	+ struct fuse_req *req)
	954	+{
	955	+ struct fuse_args *args = req->args;
	956	+
	957	+ if (!can_zero_copy_req(ent, req))
	958	+ return args->in_numargs > 1 \|\| args->out_numargs;
	959	+
	960	+ /*
	961	+ * the asymmetry between in_numargs > 2 and out_numargs > 1 is because
	962	+ * the per-op header is extracted before fuse_copy_args() for inargs but
	963	+ * not for outargs
	964	+ */
	965	+ if ((args->in_numargs > 1) && (!args->in_pages \|\| args->in_numargs > 2))
	966	+ return true;
	967	+ if (args->out_numargs && (!args->out_pages \|\| args->out_numargs > 1))
	968	+ return true;
	969	+
	970	+ return false;
	971	+}
	972	+
	973	+static int fuse_uring_select_buffer(struct fuse_ring_ent *ent)
	974	+{
	975	+ struct fuse_ring_queue *queue = ent->queue;
	976	+ struct fuse_bufpool *pool = queue->bufpool;
	977	+ unsigned int id;
	978	+
	979	+ lockdep_assert_held(&queue->lock);
	980	+
	981	+ id = find_first_bit(pool->free_map, pool->nr_bufs);
	982	+ if (id >= pool->nr_bufs)
	983	+ return -ENOBUFS;
	984	+
	985	+ WARN_ON_ONCE(ent->payload.iov_base);
	986	+ __clear_bit(id, pool->free_map);
	987	+
	988	+ ent->buf_id = id;
	989	+ ent->payload.iov_base =
	990	+ (void __user )(pool->base_uaddr + id pool->buf_size);
	991	+ ent->payload.iov_len = pool->buf_size;
	992	+
	993	+ return 0;
	994	+}
	995	+
	996	+static void fuse_uring_recycle_buffer(struct fuse_ring_ent *ent)
	997	+{
	998	+ struct iovec *ent_payload = &ent->payload;
	999	+ struct fuse_ring_queue *queue = ent->queue;
	1000	+ struct fuse_bufpool *pool;
	1001	+
	1002	+ lockdep_assert_held(&queue->lock);
	1003	+
	1004	+ if (!bufpool_enabled(queue) \|\| !ent_payload->iov_base)
	1005	+ return;
	1006	+
	1007	+ pool = queue->bufpool;
	1008	+
	1009	+ /* a buffer should never be recycled twice */
	1010	+ WARN_ON_ONCE(test_bit(ent->buf_id, pool->free_map));
	1011	+ __set_bit(ent->buf_id, pool->free_map);
	1012	+
	1013	+ memset(ent_payload, 0, sizeof(*ent_payload));
	1014	+ ent->buf_id = 0;
	1015	+}
	1016	+
	1017	+static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
	1018	+ struct fuse_req *req)
	1019	+{
	1020	+ bool buffer_selected;
	1021	+ bool has_payload;
	1022	+
	1023	+ if (!bufpool_enabled(ent->queue))
	1024	+ return 0;
	1025	+
	1026	+ buffer_selected = !!ent->payload.iov_base;
	1027	+ has_payload = fuse_uring_req_has_copyable_payload(ent, req);
	1028	+
	1029	+ if (has_payload && !buffer_selected)
	1030	+ return fuse_uring_select_buffer(ent);
	1031	+
	1032	+ if (!has_payload && buffer_selected)
	1033	+ fuse_uring_recycle_buffer(ent);
	1034	+
	1035	+ return 0;
	1036	+}
	1037	+
	1038	+static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
	1039	+ struct fuse_req *req)
	1040	+{
	1041	+ if (!bufpool_enabled(ent->queue))
	1042	+ return 0;
	1043	+
	1044	+ /* no payload to copy, can skip selecting a buffer */
	1045	+ if (!fuse_uring_req_has_copyable_payload(ent, req))
	1046	+ return 0;
	1047	+
	1048	+ return fuse_uring_select_buffer(ent);
770	1049	}
771	1050
772	1051	static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
773		− struct fuse_req *req)
	1052	+ struct fuse_req *req,
	1053	+ unsigned int issue_flags)
774	1054	{
775	1055	int err;
776	1056
777		− err = fuse_uring_copy_to_ring(ent, req);
	1057	+ err = fuse_uring_copy_to_ring(ent, req, issue_flags);
778	1058	if (!err) {
779	1059	set_bit(FR_SENT, &req->flags);
780	1060	trace_fuse_request_sent(req);
@@ -788,7 +1068,7 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent, · 7 unmodified lines
788	1068	ent->state = FRRS_INVALID;
789	1069	spin_unlock(&ent->queue->lock);
790	1070
791		− fuse_uring_req_end(ent, req, err);
	1071	+ fuse_uring_req_end(ent, req, err, issue_flags);
792	1072	}
793	1073
794	1074	return err;
@@ -856,9 +1136,12 @@ static struct fuse_req fuse_uring_ent_assign_req(struct fuse_ring_ent ent) · 61 unmodified lines
856	1136
857	1137	/* get and assign the next entry while it is still holding the lock */
858	1138	req = list_first_entry_or_null(req_queue, struct fuse_req, list);
859		− if (req)
860		− fuse_uring_add_req_to_ring_ent(ent, req);
	1139	+ if (!req \|\| fuse_uring_next_req_update_buffer(ent, req)) {
	1140	+ fuse_uring_recycle_buffer(ent);
	1141	+ return NULL;
	1142	+ }
861	1143
	1144	+ fuse_uring_add_req_to_ring_ent(ent, req);
862	1145	return req;
863	1146	}
864	1147
@@ -870,12 +1153,13 @@ static struct fuse_req fuse_uring_ent_assign_req(struct fuse_ring_ent ent) · 5 unmodified lines
870	1153	static void fuse_uring_commit(struct fuse_ring_ent ent, struct fuse_req req,
871	1154	unsigned int issue_flags)
872	1155	{
873		− struct fuse_ring *ring = ent->queue->ring;
	1156	+ struct fuse_out_header out_header;
874	1157	ssize_t err = -EFAULT;
875	1158
876		− if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,
877		− sizeof(req->out.h)))
	1159	+ if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &out_header,
	1160	+ sizeof(out_header)))
878	1161	goto out;
	1162	+ req->out.h = out_header;
879	1163
880	1164	err = fuse_uring_out_header_has_err(&req->out.h, req);
881	1165	if (err) {
@@ -883,9 +1167,9 @@ static void fuse_uring_commit(struct fuse_ring_ent ent, struct fuse_req req, · 1 unmodified line
883	1167	goto out;
884	1168	}
885	1169
886		− err = fuse_uring_copy_from_ring(ring, req, ent);
	1170	+ err = fuse_uring_copy_from_ring(req, ent, issue_flags);
887	1171	out:
888		− fuse_uring_req_end(ent, req, err);
	1172	+ fuse_uring_req_end(ent, req, err, issue_flags);
889	1173	}
890	1174
891	1175	/*
@@ -895,7 +1179,8 @@ out: · 3 unmodified lines
895	1179	* Else, there is no next fuse request and this returns false.
896	1180	*/
897	1181	static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
898		− struct fuse_ring_queue *queue)
	1182	+ struct fuse_ring_queue *queue,
	1183	+ unsigned int issue_flags)
899	1184	{
900	1185	int err;
901	1186	struct fuse_req *req;
@@ -907,7 +1192,7 @@ retry: · 5 unmodified lines
907	1192	spin_unlock(&queue->lock);
908	1193
909	1194	if (req) {
910		− err = fuse_uring_prepare_send(ent, req);
	1195	+ err = fuse_uring_prepare_send(ent, req, issue_flags);
911	1196	if (err)
912	1197	goto retry;
913	1198	}
@@ -967,7 +1252,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · 53 unmodified lines
967	1252	if (qid >= ring->nr_queues)
968	1253	return -EINVAL;
969	1254
970		− queue = ring->queues[qid];
	1255	+ queue = READ_ONCE(ring->queues[qid]);
971	1256	if (!queue)
972	1257	return err;
973	1258	fpq = &queue->fpq;
@@ -981,6 +1266,11 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · 7 unmodified lines
981	1266	return err;
982	1267	}
983	1268
	1269	+ if (!fuse_uring_cmd_index_ok(cmd, queue)) {
	1270	+ spin_unlock(&queue->lock);
	1271	+ return -EINVAL;
	1272	+ }
	1273	+
984	1274	/* Find a request based on the unique ID of the fuse request
985	1275	* This should get revised, as it needs a hash calculation and list
986	1276	* search. And full struct fuse_pqueue is needed (memory overhead).
@@ -1002,8 +1292,14 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · 15 unmodified lines
1002	1292	if (err != 0) {
1003	1293	pr_info_ratelimited("qid=%d commit_id %llu state %d",
1004	1294	queue->qid, commit_id, ent->state);
	1295	+ fuse_uring_recycle_buffer(ent);
1005	1296	spin_unlock(&queue->lock);
1006		− fuse_uring_req_end(ent, req, err);
	1297	+ /*
	1298	+ * Unregister any zero copyable pages since ent->cmd is null
	1299	+ * when it hits fuse_uring_req_end() in this path
	1300	+ */
	1301	+ zero_copy_unregister(cmd, ent, issue_flags);
	1302	+ fuse_uring_req_end(ent, req, err, issue_flags);
1007	1303	return err;
1008	1304	}
1009	1305
@@ -1019,8 +1315,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · 9 unmodified lines
1019	1315	* fuse requests would otherwise not get processed - committing
1020	1316	* and fetching is done in one step vs legacy fuse, which has separated
1021	1317	* read (fetch request) and write (commit result).
	1318	+ *
	1319	+ * If there is no next request or if all buffers are busy (if using a
	1320	+ * bufpool), the cmd is not returned to userspace. The entry is left
	1321	+ * available and the cmd only returns to userspace when there's a
	1322	+ * next request and an available buffer.
1022	1323	*/
1023		− if (fuse_uring_get_next_fuse_req(ent, queue))
	1324	+ if (fuse_uring_get_next_fuse_req(ent, queue, issue_flags))
1024	1325	fuse_uring_send(ent, cmd, 0, issue_flags);
1025	1326	return 0;
1026	1327	}
@@ -1035,7 +1336,7 @@ static bool is_ring_ready(struct fuse_ring *ring, int current_qid) · 8 unmodified lines
1035	1336	if (current_qid == qid)
1036	1337	continue;
1037	1338
1038		− queue = ring->queues[qid];
	1339	+ queue = READ_ONCE(ring->queues[qid]);
1039	1340	if (!queue) {
1040	1341	ready = false;
1041	1342	break;
@@ -1122,10 +1423,14 @@ static struct fuse_ring_ent * · 80 unmodified lines
1122	1423	fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
1123	1424	struct fuse_ring_queue *queue)
1124	1425	{
	1426	+ const struct fuse_uring_cmd_req *cmd_req =
	1427	+ io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
1125	1428	struct fuse_ring *ring = queue->ring;
1126	1429	struct fuse_ring_ent *ent;
1127	1430	struct iovec iov[FUSE_URING_IOV_SEGS];
1128	1431	struct iovec headers, payload;
	1432	+ unsigned int zero_copy_index;
	1433	+
1129	1434	int err;
1130	1435
1131	1436	err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
@@ -1135,6 +1440,10 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · 3 unmodified lines
1135	1440	return ERR_PTR(err);
1136	1441	}
1137	1442
	1443	+ zero_copy_index = READ_ONCE(cmd_req->ent_zero_copy_buf_index);
	1444	+ if (zero_copy_index && !queue->zero_copy)
	1445	+ return ERR_PTR(-EINVAL);
	1446	+
1138	1447	err = -EINVAL;
1139	1448	headers = &iov[FUSE_URING_IOV_HEADERS];
1140	1449	if (headers->iov_len < sizeof(struct fuse_uring_req_header)) {
@@ -1143,11 +1452,29 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · 2 unmodified lines
1143	1452	}
1144	1453
1145	1454	payload = &iov[FUSE_URING_IOV_PAYLOAD];
1146		− if (payload->iov_len < ring->max_payload_sz) {
1147		− pr_info_ratelimited("Invalid req payload len %zu\n",
1148		− payload->iov_len);
1149		− return ERR_PTR(err);
	1455	+
	1456	+ spin_lock(&queue->lock);
	1457	+ if (bufpool_enabled(queue)) {
	1458	+ if (payload->iov_base \|\| payload->iov_len \|\|
	1459	+ !fuse_uring_cmd_index_ok(cmd, queue)) {
	1460	+ spin_unlock(&queue->lock);
	1461	+ return ERR_PTR(err);
	1462	+ }
	1463	+ } else {
	1464	+ if (payload->iov_len < ring->max_payload_sz) {
	1465	+ spin_unlock(&queue->lock);
	1466	+ pr_info_ratelimited("Invalid req payload len %zu\n",
	1467	+ payload->iov_len);
	1468	+ return ERR_PTR(err);
	1469	+ }
	1470	+ if (queue->zero_copy) {
	1471	+ spin_unlock(&queue->lock);
	1472	+ pr_info_ratelimited("Can only use zero copy with bufpools\n");
	1473	+ return ERR_PTR(err);
	1474	+ }
	1475	+ queue->payload_mode = FUSE_PAYLOAD_PER_ENT;
1150	1476	}
	1477	+ spin_unlock(&queue->lock);
1151	1478
1152	1479	err = -ENOMEM;
1153	1480	ent = kzalloc_obj(*ent, GFP_KERNEL_ACCOUNT);
@@ -1158,7 +1485,9 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · 4 unmodified lines
1158	1485
1159	1486	ent->queue = queue;
1160	1487	ent->headers = headers->iov_base;
1161		− ent->payload = payload->iov_base;
	1488	+ if (queue->payload_mode == FUSE_PAYLOAD_PER_ENT)
	1489	+ ent->payload = *payload;
	1490	+ ent->zero_copy_index = zero_copy_index;
1162	1491
1163	1492	atomic_inc(&ring->queue_refs);
1164	1493	return ent;
@@ -1176,26 +1505,21 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, · 11 unmodified lines
1176	1505	struct fuse_ring *ring = smp_load_acquire(&fch->ring);
1177	1506	struct fuse_ring_queue *queue;
1178	1507	struct fuse_ring_ent *ent;
1179		− int err;
1180	1508	unsigned int qid = READ_ONCE(cmd_req->qid);
1181	1509
1182		− err = -ENOMEM;
1183		− if (!ring) {
1184		− ring = fuse_uring_create(fch);
1185		− if (!ring)
1186		− return err;
1187		− }
	1510	+ if (!ring)
	1511	+ return -EINVAL;
1188	1512
1189	1513	if (qid >= ring->nr_queues) {
1190	1514	pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);
1191	1515	return -EINVAL;
1192	1516	}
1193	1517
1194		− queue = ring->queues[qid];
	1518	+ queue = READ_ONCE(ring->queues[qid]);
1195	1519	if (!queue) {
1196		− queue = fuse_uring_create_queue(ring, qid);
1197		− if (!queue)
1198		− return err;
	1520	+ queue = fuse_uring_create_queue(ring, qid, false, false);
	1521	+ if (IS_ERR(queue))
	1522	+ return PTR_ERR(queue);
1199	1523	}
1200	1524
1201	1525	/*
@@ -1210,6 +1534,110 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, · 8 unmodified lines
1210	1534	return fuse_uring_do_register(ent, cmd, issue_flags);
1211	1535	}
1212	1536
	1537	+static int fuse_uring_add_queue(struct io_uring_cmd cmd, struct fuse_chan fch)
	1538	+{
	1539	+ const struct fuse_uring_cmd_req *cmd_req =
	1540	+ io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
	1541	+ struct fuse_ring *ring = smp_load_acquire(&fch->ring);
	1542	+ unsigned int qid = READ_ONCE(cmd_req->qid);
	1543	+ uint64_t flags = READ_ONCE(cmd_req->flags);
	1544	+ struct fuse_ring_queue *queue;
	1545	+ bool zero_copy = flags & FUSE_URING_ZERO_COPY;
	1546	+
	1547	+ if (!ring)
	1548	+ return -EINVAL;
	1549	+
	1550	+ if (qid >= ring->nr_queues) {
	1551	+ pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);
	1552	+ return -EINVAL;
	1553	+ }
	1554	+
	1555	+ if (flags & ~FUSE_URING_ADD_QUEUE_FLAGS)
	1556	+ return -EINVAL;
	1557	+
	1558	+ if (zero_copy && !capable(CAP_SYS_ADMIN))
	1559	+ return -EPERM;
	1560	+
	1561	+ queue = fuse_uring_create_queue(ring, qid, zero_copy, true);
	1562	+ if (IS_ERR(queue))
	1563	+ return PTR_ERR(queue);
	1564	+
	1565	+ return 0;
	1566	+}
	1567	+
	1568	+static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd,
	1569	+ struct fuse_chan *fch)
	1570	+{
	1571	+ const struct fuse_uring_cmd_req *cmd_req =
	1572	+ io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
	1573	+ unsigned int qid = READ_ONCE(cmd_req->qid);
	1574	+ uint64_t flags = READ_ONCE(cmd_req->flags);
	1575	+ /* paired with the smp_store_release() in fuse_uring_create */
	1576	+ struct fuse_ring *ring = smp_load_acquire(&fch->ring);
	1577	+ struct fuse_ring_queue *queue;
	1578	+ struct fuse_bufpool *pool;
	1579	+ uintptr_t pool_uaddr;
	1580	+ unsigned int pool_len, nr_bufs;
	1581	+ size_t pool_size, buf_size;
	1582	+ bool registered = cmd->flags & IORING_URING_CMD_FIXED;
	1583	+
	1584	+ if (!ring \|\| qid >= ring->nr_queues \|\| flags)
	1585	+ return -EINVAL;
	1586	+
	1587	+ /* reserved for future use, must be zero */
	1588	+ if (READ_ONCE(cmd_req->bufpool.reserved))
	1589	+ return -EINVAL;
	1590	+
	1591	+ /* Pairs with smp_store_release() in fuse_uring_create_queue() */
	1592	+ queue = smp_load_acquire(&ring->queues[qid]);
	1593	+ if (!queue)
	1594	+ return -EINVAL;
	1595	+
	1596	+ pool_uaddr = READ_ONCE(cmd_req->bufpool.uaddr);
	1597	+ pool_len = READ_ONCE(cmd_req->bufpool.len);
	1598	+
	1599	+ /* each buffer holds the max payload size */
	1600	+ buf_size = queue->ring->max_payload_sz;
	1601	+
	1602	+ nr_bufs = pool_len / buf_size;
	1603	+ if (!nr_bufs)
	1604	+ return -EINVAL;
	1605	+
	1606	+ pool_size = struct_size(pool, free_map, BITS_TO_LONGS(nr_bufs));
	1607	+ pool = kzalloc(pool_size, GFP_KERNEL_ACCOUNT);
	1608	+ if (!pool)
	1609	+ return -ENOMEM;
	1610	+
	1611	+ pool->base_uaddr = pool_uaddr;
	1612	+ pool->buf_size = buf_size;
	1613	+ pool->nr_bufs = nr_bufs;
	1614	+ /* all buffers are free */
	1615	+ bitmap_set(pool->free_map, 0, nr_bufs);
	1616	+
	1617	+ /*
	1618	+ * A registered bufpool is reached through an io_uring fixed buffer, so
	1619	+ * the pool is registered iff this command was submitted with
	1620	+ * IORING_URING_CMD_FIXED. The registered buffer index is taken from
	1621	+ * sqe->buf_index.
	1622	+ */
	1623	+ if (registered) {
	1624	+ pool->registered = true;
	1625	+ pool->registered_index = READ_ONCE(cmd->sqe->buf_index);
	1626	+ }
	1627	+
	1628	+ spin_lock(&queue->lock);
	1629	+ if (queue->payload_mode != FUSE_PAYLOAD_UNSET) {
	1630	+ spin_unlock(&queue->lock);
	1631	+ kfree(pool);
	1632	+ return -EINVAL;
	1633	+ }
	1634	+ queue->bufpool = pool;
	1635	+ queue->payload_mode = FUSE_PAYLOAD_BUFPOOL;
	1636	+ spin_unlock(&queue->lock);
	1637	+
	1638	+ return 0;
	1639	+}
	1640	+
1213	1641	/*
1214	1642	* Entry function from io_uring to handle the given passthrough command
1215	1643	* (op code IORING_OP_URING_CMD)
@@ -1237,23 +1665,30 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) · 21 unmodified lines
1237	1665	}
1238	1666	fch = fud->chan;
1239	1667
1240		− /* Once a connection has io-uring enabled on it, it can't be disabled */
1241		− if (!enable_uring && !fch->io_uring) {
1242		− pr_info_ratelimited("fuse-io-uring is disabled\n");
1243		− return -EOPNOTSUPP;
1244		− }
	1668	+ /*
	1669	+ * The ring is sized from values negotiated by FUSE_INIT
	1670	+ *
	1671	+ * Pairs with smp_store_release() in fuse_chan_set_initialized()
	1672	+ */
	1673	+ if (!smp_load_acquire(&fch->initialized))
	1674	+ return -EAGAIN;
1245	1675
1246	1676	if (fch->abort_with_err)
1247	1677	return -ECONNABORTED;
1248	1678	if (!fch->connected)
1249	1679	return -ENOTCONN;
1250	1680
1251		− /*
1252		− * fuse_uring_register() needs the ring to be initialized,
1253		− * we need to know the max payload size
1254		− */
1255		− if (!fch->initialized)
1256		− return -EAGAIN;
	1681	+ /* Once a connection has io-uring enabled on it, it can't be disabled */
	1682	+ if (!enable_uring && !fch->io_uring) {
	1683	+ pr_info_ratelimited("fuse-io-uring is disabled by module parameter\n");
	1684	+ return -EOPNOTSUPP;
	1685	+ }
	1686	+
	1687	+ if (!fch->io_uring) {
	1688	+ pr_info_ratelimited(
	1689	+ "fuse-io-uring not enabled on this connection\n");
	1690	+ return -EOPNOTSUPP;
	1691	+ }
1257	1692
1258	1693	switch (cmd_op) {
1259	1694	case FUSE_IO_URING_CMD_REGISTER:
@@ -1274,6 +1709,18 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) · 14 unmodified lines
1274	1709	return err;
1275	1710	}
1276	1711	break;
	1712	+ case FUSE_IO_URING_CMD_ADD_QUEUE:
	1713	+ err = fuse_uring_add_queue(cmd, fch);
	1714	+ if (err)
	1715	+ pr_info_once("FUSE_IO_URING_CMD_ADD_QUEUE failed err=%d\n",
	1716	+ err);
	1717	+ return err;
	1718	+ case FUSE_IO_URING_CMD_ADD_BUFPOOL:
	1719	+ err = fuse_uring_add_bufpool(cmd, fch);
	1720	+ if (err)
	1721	+ pr_info_once("FUSE_IO_URING_ADD_BUFPOOL failed err=%d\n",
	1722	+ err);
	1723	+ return err;
1277	1724	default:
1278	1725	return -EINVAL;
1279	1726	}
@@ -1295,9 +1742,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw) · 15 unmodified lines
1295	1742	int err;
1296	1743
1297	1744	if (!tw.cancel) {
1298		− err = fuse_uring_prepare_send(ent, ent->fuse_req);
	1745	+ err = fuse_uring_prepare_send(ent, ent->fuse_req, issue_flags);
1299	1746	if (err) {
1300		− if (!fuse_uring_get_next_fuse_req(ent, queue))
	1747	+ if (!fuse_uring_get_next_fuse_req(ent, queue,
	1748	+ issue_flags))
1301	1749	return;
1302	1750	err = 0;
1303	1751	}
@@ -1307,11 +1755,12 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw) · 3 unmodified lines
1307	1755
1308	1756	spin_lock(&queue->lock);
1309	1757	list_del_init(&ent->list);
	1758	+ fuse_uring_recycle_buffer(ent);
1310	1759	spin_unlock(&queue->lock);
1311	1760
1312	1761	io_uring_cmd_done(cmd, err, issue_flags);
1313	1762
1314		− fuse_uring_req_end(ent, ent->fuse_req, err);
	1763	+ fuse_uring_req_end(ent, ent->fuse_req, err, issue_flags);
1315	1764	kfree(ent);
1316	1765	if (atomic_dec_and_test(&queue->ring->queue_refs))
1317	1766	wake_up_all(&queue->ring->stop_waitq);
@@ -1330,7 +1779,7 @@ static struct fuse_ring_queue fuse_uring_task_to_queue(struct fuse_ring ring) · 12 unmodified lines
1330	1779	ring->nr_queues))
1331	1780	qid = 0;
1332	1781
1333		− queue = ring->queues[qid];
	1782	+ queue = READ_ONCE(ring->queues[qid]);
1334	1783	WARN_ONCE(!queue, "Missing queue for qid %d\n", qid);
1335	1784
1336	1785	return queue;
@@ -1368,15 +1817,16 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue fiq, struct fuse_req req) · 31 unmodified lines
1368	1817	req->ring_queue = queue;
1369	1818	ent = list_first_entry_or_null(&queue->ent_avail_queue,
1370	1819	struct fuse_ring_ent, list);
1371		− if (ent)
1372		− fuse_uring_add_req_to_ring_ent(ent, req);
1373		− else
1374		− list_add_tail(&req->list, &queue->fuse_req_queue);
1375		− spin_unlock(&queue->lock);
1376	1820
1377		− if (ent)
1378		− fuse_uring_dispatch_ent(ent);
	1821	+ if (!ent \|\| fuse_uring_prep_buffer(ent, req)) {
	1822	+ list_add_tail(&req->list, &queue->fuse_req_queue);
	1823	+ spin_unlock(&queue->lock);
	1824	+ return;
	1825	+ }
1379	1826
	1827	+ fuse_uring_add_req_to_ring_ent(ent, req);
	1828	+ spin_unlock(&queue->lock);
	1829	+ fuse_uring_dispatch_ent(ent);
1380	1830	return;
1381	1831
1382	1832	err_unlock:
@@ -1424,10 +1874,9 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) · 41 unmodified lines
1424	1874	*/
1425	1875	req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req,
1426	1876	list);
1427		− if (ent && req) {
	1877	+ if (ent && req && !fuse_uring_prep_buffer(ent, req)) {
1428	1878	fuse_uring_add_req_to_ring_ent(ent, req);
1429	1879	spin_unlock(&queue->lock);
1430		−
1431	1880	fuse_uring_dispatch_ent(ent);
1432	1881	} else {
1433	1882	spin_unlock(&queue->lock);

fs/fuse/dev_uring_i.h+55−1

@@ -7,6 +7,8 @@ · 6 unmodified lines
7	7	#ifndef _FS_FUSE_DEV_URING_I_H
8	8	#define _FS_FUSE_DEV_URING_I_H
9	9
	10	+#include <linux/uio.h>
	11	+
10	12	#include "fuse_dev_i.h"
11	13
12	14	#ifdef CONFIG_FUSE_IO_URING
@@ -36,11 +38,50 @@ enum fuse_ring_req_state { · 23 unmodified lines
36	38	FRRS_RELEASED,
37	39	};
38	40
	41	+/* how a queue's payload buffers are provided */
	42	+enum fuse_queue_payload_mode {
	43	+ /* not yet committed (a bufpool may still be added) */
	44	+ FUSE_PAYLOAD_UNSET = 0,
	45	+ /* each entry registers its own payload buffer */
	46	+ FUSE_PAYLOAD_PER_ENT,
	47	+ /* each entry's payload buffer is assigned from a bufpool */
	48	+ FUSE_PAYLOAD_BUFPOOL,
	49	+};
	50	+
	51	+struct fuse_bufpool {
	52	+ bool registered;
	53	+
	54	+ /*
	55	+ * io_uring registered buffer table index for this pool, bound at
	56	+ * ADD_BUFPOOL time. Only valid if the bufpool is registered
	57	+ */
	58	+ u16 registered_index;
	59	+
	60	+ /* starting uaddr of the bufpool */
	61	+ uintptr_t base_uaddr;
	62	+
	63	+ /* size of each buffer in the pool */
	64	+ size_t buf_size;
	65	+
	66	+ /* total number of buffers in the pool */
	67	+ unsigned int nr_bufs;
	68	+
	69	+ /* bitmap tracking which buffers are free */
	70	+ unsigned long free_map[];
	71	+};
	72	+
39	73	/** A fuse ring entry, part of the ring queue */
40	74	struct fuse_ring_ent {
41	75	/* userspace buffer */
42	76	struct fuse_uring_req_header __user *headers;
43		− void __user *payload;
	77	+ struct iovec payload;
	78	+
	79	+ /* buffer id in the pool, if bufpools are used. ignored otherwise */
	80	+ unsigned int buf_id;
	81	+
	82	+ /* true if the request's pages are being zero-copied */
	83	+ bool zero_copied;
	84	+ unsigned int zero_copy_index;
44	85
45	86	/* the ring queue that owns the request */
46	87	struct fuse_ring_queue *queue;
@@ -99,6 +140,14 @@ struct fuse_ring_queue { · 52 unmodified lines
99	140	unsigned int active_background;
100	141
101	142	bool stopped;
	143	+
	144	+ /* how this queue's payload buffers are provided */
	145	+ enum fuse_queue_payload_mode payload_mode;
	146	+
	147	+ /* only allocated when payload_mode == FUSE_PAYLOAD_BUFPOOL */
	148	+ struct fuse_bufpool *bufpool;
	149	+
	150	+ bool zero_copy;
102	151	};
103	152
104	153	/*
@@ -135,6 +184,7 @@ struct fuse_ring { · 30 unmodified lines
135	184	bool ready;
136	185	};
137	186
	187	+void fuse_uring_conn_init(struct fuse_chan *fch);
138	188	void fuse_uring_stop_queues(struct fuse_ring *ring);
139	189	void fuse_uring_abort_end_requests(struct fuse_ring *ring);
140	190	int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags);
@@ -174,6 +224,10 @@ static inline bool fuse_uring_ready(struct fuse_chan *fch) · 33 unmodified lines
174	224
175	225	#else /* CONFIG_FUSE_IO_URING */
176	226
	227	+static inline void fuse_uring_conn_init(struct fuse_chan *fch)
	228	+{
	229	+}
	230	+
177	231	static inline void fuse_uring_abort(struct fuse_chan *fch)
178	232	{
179	233	}

fs/fuse/dir.c+24−13

@@ -34,7 +34,7 @@ struct dentry_bucket { · 33 unmodified lines
34	34	#define FUSE_HASH_BITS 5
35	35	#define FUSE_HASH_SIZE (1 << FUSE_HASH_BITS)
36	36	static struct dentry_bucket dentry_hash[FUSE_HASH_SIZE];
37		−struct delayed_work dentry_tree_work;
	37	+static struct delayed_work dentry_tree_work;
38	38
39	39	/* Minimum invalidation work queue frequency */
40	40	#define FUSE_DENTRY_INVAL_FREQ_MIN 5
@@ -96,6 +96,7 @@ static void fuse_advise_use_readdirplus(struct inode *dir) · 55 unmodified lines
96	96
97	97	struct fuse_dentry {
98	98	u64 time;
	99	+ u64 epoch;
99	100	union {
100	101	struct rcu_head rcu;
101	102	struct rb_node node;
@@ -236,6 +237,13 @@ void fuse_dentry_tree_cleanup(void) · 134 unmodified lines
236	237	WARN_ON_ONCE(!RB_EMPTY_ROOT(&dentry_hash[i].tree));
237	238	}
238	239
	240	+void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch)
	241	+{
	242	+ struct fuse_dentry *fd = dentry->d_fsdata;
	243	+
	244	+ fd->epoch = epoch;
	245	+}
	246	+
239	247	static inline void __fuse_dentry_settime(struct dentry *dentry, u64 time)
240	248	{
241	249	((struct fuse_dentry *) dentry->d_fsdata)->time = time;
@@ -387,10 +395,11 @@ static int fuse_dentry_revalidate(struct inode dir, const struct qstr name, · 145 unmodified lines
387	395	struct fuse_mount *fm;
388	396	struct fuse_conn *fc;
389	397	struct fuse_inode *fi;
	398	+ struct fuse_dentry *fd = entry->d_fsdata;
390	399	int ret;
391	400
392	401	fc = get_fuse_conn_super(dir->i_sb);
393		− if (entry->d_time < atomic_read(&fc->epoch))
	402	+ if (fd->epoch < atomic_read(&fc->epoch))
394	403	goto invalid;
395	404
396	405	inode = d_inode_rcu(entry);
@@ -480,10 +489,10 @@ static int fuse_dentry_init(struct dentry *dentry) · 83 unmodified lines
480	489	RB_CLEAR_NODE(&fd->node);
481	490	dentry->d_fsdata = fd;
482	491	/*
483		− * Initialising d_time (epoch) to '0' ensures the dentry is invalid
	492	+ * Initialising epoch to '0' ensures the dentry is invalid
484	493	* if compared to fc->epoch, which is initialized to '1'.
485	494	*/
486		− dentry->d_time = 0;
	495	+ fuse_dentry_set_epoch(dentry, 0);
487	496
488	497	return 0;
489	498	}
@@ -641,7 +650,7 @@ static struct dentry fuse_lookup(struct inode dir, struct dentry *entry, · 151 unmodified lines
641	650	goto out_err;
642	651
643	652	entry = newent ? newent : entry;
644		− entry->d_time = epoch;
	653	+ fuse_dentry_set_epoch(entry, epoch);
645	654	if (outarg_valid)
646	655	fuse_change_entry_timeout(entry, &outarg);
647	656	else
@@ -898,7 +907,7 @@ static int fuse_create_open(struct mnt_idmap idmap, struct inode dir, · 250 unmodified lines
898	907	}
899	908	kfree(forget);
900	909	d_instantiate(entry, inode);
901		− entry->d_time = epoch;
	910	+ fuse_dentry_set_epoch(entry, epoch);
902	911	fuse_change_entry_timeout(entry, &outentry);
903	912	fuse_dir_changed(dir);
904	913	err = generic_file_open(inode, file);
@@ -1028,10 +1037,10 @@ static struct dentry create_new_entry(struct mnt_idmap idmap, struct fuse_moun · 123 unmodified lines
1028	1037	return d;
1029	1038
1030	1039	if (d) {
1031		− d->d_time = epoch;
	1040	+ fuse_dentry_set_epoch(d, epoch);
1032	1041	fuse_change_entry_timeout(d, &outarg);
1033	1042	} else {
1034		− entry->d_time = epoch;
	1043	+ fuse_dentry_set_epoch(entry, epoch);
1035	1044	fuse_change_entry_timeout(entry, &outarg);
1036	1045	}
1037	1046	fuse_dir_changed(dir);
@@ -2169,10 +2178,8 @@ int fuse_do_setattr(struct mnt_idmap idmap, struct dentry dentry, · 1131 unmodified lines
2169	2178	filemap_invalidate_lock(mapping);
2170	2179	fault_blocked = true;
2171	2180	err = fuse_dax_break_layouts(inode, 0, -1);
2172		− if (err) {
2173		− filemap_invalidate_unlock(mapping);
2174		− return err;
2175		− }
	2181	+ if (err)
	2182	+ goto unlock;
2176	2183	}
2177	2184
2178	2185	if (attr->ia_valid & ATTR_OPEN) {
@@ -2199,7 +2206,7 @@ int fuse_do_setattr(struct mnt_idmap idmap, struct dentry dentry, · 20 unmodified lines
2199	2206	ATTR_TIMES_SET)) {
2200	2207	err = write_inode_now(inode, true);
2201	2208	if (err)
2202		− return err;
	2209	+ goto unlock;
2203	2210
2204	2211	fuse_set_nowrite(inode);
2205	2212	fuse_release_nowrite(inode);
@@ -2290,6 +2297,9 @@ int fuse_do_setattr(struct mnt_idmap idmap, struct dentry dentry, · 84 unmodified lines
2290	2297	*/
2291	2298	if ((is_truncate \|\| !is_wb) &&
2292	2299	S_ISREG(inode->i_mode) && oldsize != outarg.attr.size) {
	2300	+ if (outarg.attr.size > oldsize)
	2301	+ truncate_pagecache_range(inode, oldsize,
	2302	+ outarg.attr.size - 1);
2293	2303	truncate_pagecache(inode, outarg.attr.size);
2294	2304	invalidate_inode_pages2(mapping);
2295	2305	}
@@ -2307,6 +2317,7 @@ error: · 11 unmodified lines
2307	2317
2308	2318	clear_bit(FUSE_I_SIZE_UNSTABLE, &fi->state);
2309	2319
	2320	+unlock:
2310	2321	if (fault_blocked)
2311	2322	filemap_invalidate_unlock(mapping);
2312	2323	return err;

fs/fuse/file.c+16−4

@@ -272,7 +272,7 @@ static int fuse_open(struct inode inode, struct file file) · 271 unmodified lines
272	272	filemap_invalidate_lock(inode->i_mapping);
273	273	err = fuse_dax_break_layouts(inode, 0, -1);
274	274	if (err)
275		− goto out_inode_unlock;
	275	+ goto out_unlock;
276	276	}
277	277
278	278	if (is_wb_truncate \|\| dax_truncate)
@@ -296,9 +296,9 @@ static int fuse_open(struct inode inode, struct file file) · 17 unmodified lines
296	296	else if (!(ff->open_flags & FOPEN_KEEP_CACHE))
297	297	invalidate_inode_pages2(inode->i_mapping);
298	298	}
	299	+out_unlock:
299	300	if (dax_truncate)
300	301	filemap_invalidate_unlock(inode->i_mapping);
301		−out_inode_unlock:
302	302	if (is_wb_truncate \|\| dax_truncate)
303	303	inode_unlock(inode);
304	304
@@ -605,6 +605,7 @@ void fuse_read_args_fill(struct fuse_io_args ia, struct file file, loff_t pos, · 300 unmodified lines
605	605	args->out_argvar = true;
606	606	args->out_numargs = 1;
607	607	args->out_args[0].size = count;
	608	+ args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
608	609	}
609	610
610	611	static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres,
@@ -1153,6 +1154,7 @@ static void fuse_write_args_fill(struct fuse_io_args ia, struct fuse_file ff, · 542 unmodified lines
1153	1154	args->out_numargs = 1;
1154	1155	args->out_args[0].size = sizeof(ia->write.out);
1155	1156	args->out_args[0].value = &ia->write.out;
	1157	+ args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
1156	1158	}
1157	1159
1158	1160	static unsigned int fuse_write_flags(struct kiocb *iocb)
@@ -1346,9 +1348,13 @@ static ssize_t fuse_perform_write(struct kiocb iocb, struct iov_iter ii) · 187 unmodified lines
1346	1348	struct fuse_conn *fc = get_fuse_conn(inode);
1347	1349	struct fuse_inode *fi = get_fuse_inode(inode);
1348	1350	loff_t pos = iocb->ki_pos;
	1351	+ loff_t old_size = i_size_read(inode);
1349	1352	int err = 0;
1350	1353	ssize_t res = 0;
1351	1354
	1355	+ if (pos > old_size)
	1356	+ truncate_pagecache_range(inode, old_size, pos - 1);
	1357	+
1352	1358	if (inode->i_size < pos + iov_iter_count(ii))
1353	1359	set_bit(FUSE_I_SIZE_UNSTABLE, &fi->state);
1354	1360
@@ -1787,13 +1793,14 @@ static ssize_t fuse_direct_write_iter(struct kiocb iocb, struct iov_iter from) · 432 unmodified lines
1787	1793	{
1788	1794	struct inode *inode = file_inode(iocb->ki_filp);
1789	1795	struct address_space *mapping = inode->i_mapping;
1790		− loff_t pos = iocb->ki_pos;
1791	1796	ssize_t res;
1792	1797	bool exclusive;
1793	1798
1794	1799	fuse_dio_lock(iocb, from, &exclusive);
1795	1800	res = generic_write_checks(iocb, from);
1796	1801	if (res > 0) {
	1802	+ loff_t pos = iocb->ki_pos;
	1803	+
1797	1804	task_io_account_write(res);
1798	1805	if (!is_sync_kiocb(iocb)) {
1799	1806	res = fuse_direct_IO(iocb, from);
@@ -1808,7 +1815,7 @@ static ssize_t fuse_direct_write_iter(struct kiocb iocb, struct iov_iter from) · 8 unmodified lines
1808	1815	/*
1809	1816	* As in generic_file_direct_write(), invalidate after
1810	1817	* write, to invalidate read-ahead cache that may have
1811		− * with the write.
	1818	+ * competed with the write.
1812	1819	*/
1813	1820	invalidate_inode_pages2_range(mapping,
1814	1821	pos >> PAGE_SHIFT,
@@ -2894,6 +2901,11 @@ static long fuse_file_fallocate(struct file *file, int mode, loff_t offset, · 1079 unmodified lines
2894	2901
2895	2902	/* we could have extended the file */
2896	2903	if (!(mode & FALLOC_FL_KEEP_SIZE)) {
	2904	+ loff_t oldsize = i_size_read(inode);
	2905	+
	2906	+ if (offset + length > oldsize)
	2907	+ truncate_pagecache_range(inode, oldsize,
	2908	+ offset + length - 1);
2897	2909	if (fuse_write_update_attr(inode, offset + length, length))
2898	2910	file_update_time(file);
2899	2911	}

fs/fuse/fuse_dev_i.h+5−0

@@ -38,6 +38,8 @@ struct fuse_iqueue; · 37 unmodified lines
38	38	* @FR_PRIVATE: request is on private list
39	39	* @FR_ASYNC: request is asynchronous
40	40	* @FR_URING: request is handled through fuse-io-uring
	41	+ * @FR_SYNC_WAKEUP: use synchronous wakeup when queueing this request to
	42	+ * give the scheduler a hint about the waker task
41	43	*/
42	44	enum fuse_req_flag {
43	45	FR_ISREPLY,
@@ -53,6 +55,7 @@ enum fuse_req_flag { · 9 unmodified lines
53	55	FR_PRIVATE,
54	56	FR_ASYNC,
55	57	FR_URING,
	58	+ FR_SYNC_WAKEUP,
56	59	};
57	60
58	61	/**
@@ -325,6 +328,8 @@ struct fuse_copy_state { · 266 unmodified lines
325	328	bool write:1;
326	329	bool move_folios:1;
327	330	bool is_uring:1;
	331	+ /* set when the payload is zero-copied. folios are filled in place */
	332	+ bool skip_folio_copy:1;
328	333	struct {
329	334	unsigned int copied_sz; /* copied size into the user buffer */
330	335	} ring;

fs/fuse/fuse_i.h+2−0

@@ -1054,6 +1054,8 @@ u64 fuse_time_to_jiffies(u64 sec, u32 nsec); · 1053 unmodified lines
1054	1054
1055	1055	void fuse_change_entry_timeout(struct dentry entry, struct fuse_entry_out o);
1056	1056
	1057	+void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch);
	1058	+
1057	1059	/*
1058	1060	* Initialize fuse_conn
1059	1061	*/

fs/fuse/inode.c+10−6

@@ -791,6 +791,9 @@ static int fuse_opt_fd(struct fs_context fsc, struct file file) · 790 unmodified lines
791	791	{
792	792	struct fuse_fs_context *ctx = fsc->fs_private;
793	793
	794	+ if (ctx->fud)
	795	+ return invalfc(fsc, "Multiple fd specified");
	796	+
794	797	if (file->f_op != &fuse_dev_operations)
795	798	return invalfc(fsc, "fd is not a fuse device");
796	799	/*
@@ -1272,6 +1275,7 @@ static void process_init_reply(struct fuse_args *args, int error) · 475 unmodified lines
1272	1275	struct fuse_mount *fm = ia->fm;
1273	1276	struct fuse_conn *fc = fm->fc;
1274	1277	struct fuse_init_out *arg = &ia->out;
	1278	+ bool io_uring_enabled = false;
1275	1279	bool ok = true;
1276	1280
1277	1281	if (error \|\| arg->major != FUSE_KERNEL_VERSION)
@@ -1402,7 +1406,7 @@ static void process_init_reply(struct fuse_args *args, int error) · 124 unmodified lines
1402	1406	ok = false;
1403	1407	}
1404	1408	if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled())
1405		− fuse_chan_io_uring_enable(fc->chan);
	1409	+ io_uring_enabled = true;
1406	1410
1407	1411	if (flags & FUSE_REQUEST_TIMEOUT)
1408	1412	timeout = arg->request_timeout;
@@ -1416,6 +1420,7 @@ static void process_init_reply(struct fuse_args *args, int error) · 7 unmodified lines
1416	1420
1417	1421	fm->sb->s_bdi->ra_pages =
1418	1422	min(fm->sb->s_bdi->ra_pages, ra_pages);
	1423	+ fm->sb->s_bdi->io_pages = fc->max_pages;
1419	1424	fc->minor = arg->minor;
1420	1425	fc->max_write = arg->minor < 5 ? 4096 : arg->max_write;
1421	1426	fc->max_write = max_t(unsigned, 4096, fc->max_write);
@@ -1432,6 +1437,7 @@ static void process_init_reply(struct fuse_args *args, int error) · 10 unmodified lines
1432	1437	.minor = fc->minor,
1433	1438	.max_write = fc->max_write,
1434	1439	.max_pages = fc->max_pages,
	1440	+ .io_uring_enabled = io_uring_enabled,
1435	1441	};
1436	1442	fuse_chan_set_initialized(fc->chan, &cp);
1437	1443	}
@@ -1474,12 +1480,8 @@ static struct fuse_init_args fuse_new_init(struct fuse_mount fm) · 36 unmodified lines
1474	1480	if (IS_ENABLED(CONFIG_FUSE_PASSTHROUGH))
1475	1481	flags \|= FUSE_PASSTHROUGH;
1476	1482
1477		− /*
1478		− * This is just an information flag for fuse server. No need to check
1479		− * the reply - server is either sending IORING_OP_URING_CMD or not.
1480		− */
1481	1483	if (fuse_uring_enabled())
1482		− flags \|= FUSE_OVER_IO_URING;
	1484	+ flags \|= FUSE_OVER_IO_URING \| FUSE_HAS_IO_URING_BUFPOOL;
1483	1485
1484	1486	ia->in.flags = flags;
1485	1487	ia->in.flags2 = flags >> 32;
@@ -1639,6 +1641,8 @@ static int fuse_fill_super_submount(struct super_block *sb, · 153 unmodified lines
1639	1641	fuse_fill_attr_from_inode(&root_attr, parent_fi);
1640	1642	root = fuse_iget(sb, parent_fi->nodeid, 0, &root_attr, 0, 0,
1641	1643	fuse_get_evict_ctr(fm->fc));
	1644	+ if (!root)
	1645	+ return -ENOMEM;
1642	1646	/*
1643	1647	* This inode is just a duplicate, so it is not looked up and
1644	1648	* its nlookup should not be incremented. fuse_iget() does

fs/fuse/readdir.c+1−1

@@ -260,7 +260,7 @@ retry: · 259 unmodified lines
260	260	}
261	261	if (fc->readdirplus_auto)
262	262	set_bit(FUSE_I_INIT_RDPLUS, &get_fuse_inode(inode)->state);
263		− dentry->d_time = epoch;
	263	+ fuse_dentry_set_epoch(dentry, epoch);
264	264	fuse_change_entry_timeout(dentry, o);
265	265
266	266	dput(dentry);

fs/fuse/req_timeout.c+3−9

@@ -128,18 +128,12 @@ static void set_request_timeout(struct fuse_chan *fch, unsigned int timeout) · 127 unmodified lines
128	128
129	129	void fuse_init_server_timeout(struct fuse_chan *fch, unsigned int timeout)
130	130	{
131		− if (!timeout && !fuse_max_req_timeout && !fuse_default_req_timeout)
132		− return;
133		−
134	131	if (!timeout)
135	132	timeout = fuse_default_req_timeout;
136	133
137		− if (fuse_max_req_timeout) {
138		− if (timeout)
139		− timeout = min(fuse_max_req_timeout, timeout);
140		− else
141		− timeout = fuse_max_req_timeout;
142		− }
	134	+ timeout = min_not_zero(timeout, fuse_max_req_timeout);
	135	+ if (!timeout)
	136	+ return;
143	137
144	138	timeout = max(FUSE_TIMEOUT_TIMER_FREQ, timeout);
145	139

include/linux/io_uring/cmd.h+32−6

@@ -91,6 +91,15 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, · 90 unmodified lines
91	91	bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
92	92	struct io_br_sel *sel, unsigned int issue_flags);
93	93
	94	+int io_buffer_register_request(struct io_uring_cmd cmd, struct request rq,
	95	+ void (release)(void ), unsigned int index,
	96	+ unsigned int issue_flags);
	97	+int io_buffer_register_bvec(struct io_uring_cmd cmd, const struct bio_vec bvs,
	98	+ unsigned int nr_bvecs, void (release)(void ),
	99	+ void *priv, u8 dir, unsigned int index,
	100	+ unsigned int issue_flags);
	101	+int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
	102	+ unsigned int issue_flags);
94	103	#else
95	104	static inline int
96	105	io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -133,6 +142,29 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, · 36 unmodified lines
133	142	{
134	143	return true;
135	144	}
	145	+static inline int io_buffer_register_request(struct io_uring_cmd *cmd,
	146	+ struct request *rq,
	147	+ void (release)(void ),
	148	+ unsigned int index,
	149	+ unsigned int issue_flags)
	150	+{
	151	+ return -EOPNOTSUPP;
	152	+}
	153	+static inline int io_buffer_register_bvec(struct io_uring_cmd *cmd,
	154	+ const struct bio_vec *bvs,
	155	+ unsigned int nr_bvecs,
	156	+ void (release)(void ), void *priv,
	157	+ u8 dir, unsigned int index,
	158	+ unsigned int issue_flags)
	159	+{
	160	+ return -EOPNOTSUPP;
	161	+}
	162	+static inline int io_buffer_unregister(struct io_uring_cmd *cmd,
	163	+ unsigned int index,
	164	+ unsigned int issue_flags)
	165	+{
	166	+ return -EOPNOTSUPP;
	167	+}
136	168	#endif
137	169
138	170	static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
@@ -182,10 +214,4 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret, · 43 unmodified lines
182	214	return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true);
183	215	}
184	216
185		−int io_buffer_register_bvec(struct io_uring_cmd cmd, struct request rq,
186		− void (release)(void ), unsigned int index,
187		− unsigned int issue_flags);
188		−int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
189		− unsigned int issue_flags);
190		−
191	217	#endif /* _LINUX_IO_URING_CMD_H */

include/linux/io_uring_types.h+5−0

@@ -53,6 +53,11 @@ enum io_uring_cmd_flags { · 52 unmodified lines
53	53	IO_URING_F_COMPAT = (1 << 12),
54	54	};
55	55
	56	+enum {
	57	+ IO_BUF_DEST = 1 << ITER_DEST,
	58	+ IO_BUF_SOURCE = 1 << ITER_SOURCE,
	59	+};
	60	+
56	61	struct iou_loop_params;
57	62
58	63	struct io_wq_work_node {

include/uapi/linux/fuse.h+61−2

@@ -240,6 +240,14 @@ · 239 unmodified lines
240	240	* - add FUSE_COPY_FILE_RANGE_64
241	241	* - add struct fuse_copy_file_range_out
242	242	* - add FUSE_NOTIFY_PRUNE
	243	+ *
	244	+ * 7.46
	245	+ * - add FUSE_IO_URING_CMD_ADD_QUEUE
	246	+ * - add FUSE_HAS_IO_URING_BUFPOOL
	247	+ * - add fuse_uring_cmd_req bufpool struct
	248	+ * - add bufpool offset field to fuse_uring_ent_in_out struct
	249	+ * - add FUSE_URING_ZERO_COPY, FUSE_URING_ENT_ZERO_COPY, and
	250	+ * FOPEN_IO_URING_ZERO_COPY flag
243	251	*/
244	252
245	253	#ifndef _LINUX_FUSE_H
@@ -275,7 +283,7 @@ · 29 unmodified lines
275	283	#define FUSE_KERNEL_VERSION 7
276	284
277	285	/** Minor version number of this interface */
278		−#define FUSE_KERNEL_MINOR_VERSION 45
	286	+#define FUSE_KERNEL_MINOR_VERSION 46
279	287
280	288	/** The node ID of the root inode */
281	289	#define FUSE_ROOT_ID 1
@@ -383,6 +391,12 @@ struct fuse_file_lock { · 101 unmodified lines
383	391	* FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE)
384	392	* FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode
385	393	* FOPEN_PASSTHROUGH: passthrough read/write io for this open file
	394	+ * FOPEN_IO_URING_ZERO_COPY: use io-uring zero-copy for reads/writes on this
	395	+ * open file. Honored only when the serving io-uring
	396	+ * queue was set up for zero-copy
	397	+ * (FUSE_URING_ZERO_COPY) and the request carries page
	398	+ * payload. Otherwise reads/writes fall back to
	399	+ * copying.
386	400	*/
387	401	#define FOPEN_DIRECT_IO (1 << 0)
388	402	#define FOPEN_KEEP_CACHE (1 << 1)
@@ -392,6 +406,7 @@ struct fuse_file_lock { · 3 unmodified lines
392	406	#define FOPEN_NOFLUSH (1 << 5)
393	407	#define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6)
394	408	#define FOPEN_PASSTHROUGH (1 << 7)
	409	+#define FOPEN_IO_URING_ZERO_COPY (1 << 8)
395	410
396	411	/**
397	412	* INIT request/reply flags
@@ -448,6 +463,7 @@ struct fuse_file_lock { · 50 unmodified lines
448	463	* FUSE_OVER_IO_URING: Indicate that client supports io-uring
449	464	* FUSE_REQUEST_TIMEOUT: kernel supports timing out requests.
450	465	* init_out.request_timeout contains the timeout (in secs)
	466	+ * FUSE_HAS_IO_URING_BUFPOOL: kernel supports io-uring buffer pools
451	467	*/
452	468	#define FUSE_ASYNC_READ (1 << 0)
453	469	#define FUSE_POSIX_LOCKS (1 << 1)
@@ -495,6 +511,7 @@ struct fuse_file_lock { · 41 unmodified lines
495	511	#define FUSE_ALLOW_IDMAP (1ULL << 40)
496	512	#define FUSE_OVER_IO_URING (1ULL << 41)
497	513	#define FUSE_REQUEST_TIMEOUT (1ULL << 42)
	514	+#define FUSE_HAS_IO_URING_BUFPOOL (1ULL << 43)
498	515
499	516	/**
500	517	* CUSE INIT request/reply flags
@@ -1251,6 +1268,13 @@ struct fuse_supp_groups { · 750 unmodified lines
1251	1268	#define FUSE_URING_IN_OUT_HEADER_SZ 128
1252	1269	#define FUSE_URING_OP_IN_OUT_SZ 128
1253	1270
	1271	+/**
	1272	+ * fuse_uring_ent_in_out flags
	1273	+ *
	1274	+ * FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied
	1275	+ */
	1276	+#define FUSE_URING_ENT_ZERO_COPY (1 << 0)
	1277	+
1254	1278	/* Used as part of the fuse_uring_req_header */
1255	1279	struct fuse_uring_ent_in_out {
1256	1280	uint64_t flags;
@@ -1263,7 +1287,9 @@ struct fuse_uring_ent_in_out { · 6 unmodified lines
1263	1287
1264	1288	/* size of user payload buffer */
1265	1289	uint32_t payload_sz;
1266		− uint32_t padding;
	1290	+
	1291	+ /* Offset into the bufpool, if bufpools are used */
	1292	+ uint32_t offset;
1267	1293
1268	1294	uint64_t reserved;
1269	1295	};
@@ -1292,8 +1318,22 @@ enum fuse_uring_cmd { · 22 unmodified lines
1292	1318
1293	1319	/* commit fuse request result and fetch next request */
1294	1320	FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2,
	1321	+
	1322	+ /* add a queue */
	1323	+ FUSE_IO_URING_CMD_ADD_QUEUE = 3,
	1324	+
	1325	+ /* add a bufpool to a queue */
	1326	+ FUSE_IO_URING_CMD_ADD_BUFPOOL = 4,
1295	1327	};
1296	1328
	1329	+/*
	1330	+ * fuse_uring_cmd_req flags for FUSE_IO_URING_CMD_ADD_QUEUE
	1331	+ *
	1332	+ * FUSE_URING_ZERO_COPY is only supported for queues with bufpools on privileged
	1333	+ * servers
	1334	+ */
	1335	+#define FUSE_URING_ZERO_COPY (1 << 0)
	1336	+
1297	1337	/**
1298	1338	* In the 80B command area of the SQE.
1299	1339	*/
@@ -1306,6 +1346,25 @@ struct fuse_uring_cmd_req { · 6 unmodified lines
1306	1346	/* queue the command is for (queue index) */
1307	1347	uint16_t qid;
1308	1348	uint8_t padding[6];
	1349	+
	1350	+ union {
	1351	+ struct {
	1352	+ /* base address of bufpool */
	1353	+ uint64_t uaddr;
	1354	+ uint32_t len;
	1355	+ uint32_t reserved;
	1356	+ } bufpool;
	1357	+
	1358	+ /*
	1359	+ * Index of this entry's slot in the server's io_uring
	1360	+ * registered buffer table, where the kernel registers the
	1361	+ * request's pages for zero-copy. Set for
	1362	+ * FUSE_IO_URING_CMD_REGISTER cmds only, and only on queues
	1363	+ * created with FUSE_URING_ZERO_COPY. On a non-zero-copy queue
	1364	+ * this must be 0
	1365	+ */
	1366	+ uint16_t ent_zero_copy_buf_index;
	1367	+ };
1309	1368	};
1310	1369
1311	1370	#endif /* _LINUX_FUSE_H */

io_uring/io_uring.c+1−1

@@ -3236,7 +3236,7 @@ static int __init io_uring_init(void) · 3235 unmodified lines
3236	3236	io_uring_optable_init();
3237	3237
3238	3238	/* imu->dir is u8 */
3239		− BUILD_BUG_ON((IO_IMU_DEST \| IO_IMU_SOURCE) > U8_MAX);
	3239	+ BUILD_BUG_ON((IO_BUF_DEST \| IO_BUF_SOURCE) > U8_MAX);
3240	3240
3241	3241	/*
3242	3242	* Allow user copy in the per-command field, which starts after the

io_uring/rsrc.c+90−37

@@ -912,7 +912,7 @@ static struct io_rsrc_node io_sqe_buffer_register(struct io_ring_ctx ctx, · 911 unmodified lines
912	912	imu->release = io_release_ubuf;
913	913	imu->priv = imu;
914	914	imu->flags = 0;
915		− imu->dir = IO_IMU_DEST \| IO_IMU_SOURCE;
	915	+ imu->dir = IO_BUF_DEST \| IO_BUF_SOURCE;
916	916	if (coalesced)
917	917	imu->folio_shift = data.folio_shift;
918	918	refcount_set(&imu->refs, 1);
@@ -1015,71 +1015,124 @@ int io_sqe_buffers_register(struct io_ring_ctx ctx, void __user arg, · 96 unmodified lines
1015	1015	return ret;
1016	1016	}
1017	1017
1018		−int io_buffer_register_bvec(struct io_uring_cmd cmd, struct request rq,
1019		− void (release)(void ), unsigned int index,
1020		− unsigned int issue_flags)
	1018	+static struct io_mapped_ubuf io_kernel_buffer_init(struct io_ring_ctx ctx,
	1019	+ unsigned int nr_bvecs,
	1020	+ unsigned int total_bytes,
	1021	+ u8 dir,
	1022	+ void (release)(void ),
	1023	+ void *priv,
	1024	+ unsigned int index)
1021	1025	{
1022		− struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
1023	1026	struct io_rsrc_data *data = &ctx->buf_table;
1024		− struct req_iterator rq_iter;
1025	1027	struct io_mapped_ubuf *imu;
1026	1028	struct io_rsrc_node *node;
1027		− struct bio_vec bv;
1028		− unsigned int nr_bvecs = 0;
1029		− int ret = 0;
1030	1029
1031		− io_ring_submit_lock(ctx, issue_flags);
1032		− if (index >= data->nr) {
1033		− ret = -EINVAL;
1034		− goto unlock;
1035		− }
	1030	+ if (index >= data->nr)
	1031	+ return ERR_PTR(-EINVAL);
1036	1032	index = array_index_nospec(index, data->nr);
1037	1033
1038		− if (data->nodes[index]) {
1039		− ret = -EBUSY;
1040		− goto unlock;
1041		− }
	1034	+ if (data->nodes[index])
	1035	+ return ERR_PTR(-EBUSY);
1042	1036
1043	1037	node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);
1044		− if (!node) {
1045		− ret = -ENOMEM;
1046		− goto unlock;
1047		− }
	1038	+ if (!node)
	1039	+ return ERR_PTR(-ENOMEM);
1048	1040
1049		− /*
1050		− * blk_rq_nr_phys_segments() may overestimate the number of bvecs
1051		− * but avoids needing to iterate over the bvecs
1052		− */
1053		− imu = io_alloc_imu(ctx, blk_rq_nr_phys_segments(rq));
	1041	+ imu = io_alloc_imu(ctx, nr_bvecs);
1054	1042	if (!imu) {
1055	1043	io_cache_free(&ctx->node_cache, node);
1056		− ret = -ENOMEM;
1057		− goto unlock;
	1044	+ return ERR_PTR(-ENOMEM);
1058	1045	}
1059	1046
1060	1047	imu->ubuf = 0;
1061		− imu->len = blk_rq_bytes(rq);
	1048	+ imu->len = total_bytes;
1062	1049	imu->folio_shift = PAGE_SHIFT;
	1050	+ imu->nr_bvecs = nr_bvecs;
1063	1051	refcount_set(&imu->refs, 1);
1064	1052	imu->release = release;
1065		− imu->priv = rq;
	1053	+ imu->priv = priv;
	1054	+ imu->dir = dir;
1066	1055	imu->flags = IO_REGBUF_F_KBUF;
1067		− imu->dir = 1 << rq_data_dir(rq);
1068	1056
	1057	+ node->buf = imu;
	1058	+ data->nodes[index] = node;
	1059	+
	1060	+ return imu;
	1061	+}
	1062	+
	1063	+int io_buffer_register_request(struct io_uring_cmd cmd, struct request rq,
	1064	+ void (release)(void ), unsigned int index,
	1065	+ unsigned int issue_flags)
	1066	+{
	1067	+ struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
	1068	+ struct req_iterator rq_iter;
	1069	+ struct io_mapped_ubuf *imu;
	1070	+ struct bio_vec bv;
	1071	+ /*
	1072	+ * blk_rq_nr_phys_segments() may overestimate the number of bvecs
	1073	+ * but avoids needing to iterate over the bvecs
	1074	+ */
	1075	+ unsigned int nr_bvecs = blk_rq_nr_phys_segments(rq);
	1076	+ unsigned int total_bytes = blk_rq_bytes(rq);
	1077	+ int ret = 0;
	1078	+
	1079	+ io_ring_submit_lock(ctx, issue_flags);
	1080	+
	1081	+ imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes,
	1082	+ 1 << rq_data_dir(rq), release, rq, index);
	1083	+ if (IS_ERR(imu)) {
	1084	+ ret = PTR_ERR(imu);
	1085	+ goto unlock;
	1086	+ }
	1087	+
	1088	+ nr_bvecs = 0;
1069	1089	rq_for_each_bvec(bv, rq, rq_iter)
1070	1090	imu->bvec[nr_bvecs++] = bv;
1071	1091	imu->nr_bvecs = nr_bvecs;
1072	1092
1073		− node->buf = imu;
1074		− data->nodes[index] = node;
	1093	+unlock:
	1094	+ io_ring_submit_unlock(ctx, issue_flags);
	1095	+ return ret;
	1096	+}
	1097	+EXPORT_SYMBOL_GPL(io_buffer_register_request);
	1098	+
	1099	+/*
	1100	+ * bvs is copied internally. caller may free it on return.
	1101	+ */
	1102	+int io_buffer_register_bvec(struct io_uring_cmd cmd, const struct bio_vec bvs,
	1103	+ unsigned int nr_bvecs, void (release)(void ),
	1104	+ void *priv, u8 dir, unsigned int index,
	1105	+ unsigned int issue_flags)
	1106	+{
	1107	+ struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
	1108	+ struct io_mapped_ubuf *imu;
	1109	+ struct bio_vec *bvec;
	1110	+ unsigned int i, total_bytes = 0;
	1111	+ int ret = 0;
	1112	+
	1113	+ for (i = 0; i < nr_bvecs; i++)
	1114	+ total_bytes += bvs[i].bv_len;
	1115	+
	1116	+ io_ring_submit_lock(ctx, issue_flags);
	1117	+ imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, dir, release,
	1118	+ priv, index);
	1119	+ if (IS_ERR(imu)) {
	1120	+ ret = PTR_ERR(imu);
	1121	+ goto unlock;
	1122	+ }
	1123	+
	1124	+ bvec = imu->bvec;
	1125	+ for (i = 0; i < nr_bvecs; i++)
	1126	+ bvec[i] = bvs[i];
	1127	+
1075	1128	unlock:
1076	1129	io_ring_submit_unlock(ctx, issue_flags);
1077	1130	return ret;
1078	1131	}
1079	1132	EXPORT_SYMBOL_GPL(io_buffer_register_bvec);
1080	1133
1081		−int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
1082		− unsigned int issue_flags)
	1134	+int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
	1135	+ unsigned int issue_flags)
1083	1136	{
1084	1137	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
1085	1138	struct io_rsrc_data *data = &ctx->buf_table;
@@ -1109,7 +1162,7 @@ unlock: · 23 unmodified lines
1109	1162	io_ring_submit_unlock(ctx, issue_flags);
1110	1163	return ret;
1111	1164	}
1112		−EXPORT_SYMBOL_GPL(io_buffer_unregister_bvec);
	1165	+EXPORT_SYMBOL_GPL(io_buffer_unregister);
1113	1166
1114	1167	static int validate_fixed_range(u64 buf_addr, size_t len,
1115	1168	const struct io_mapped_ubuf *imu)

io_uring/rsrc.h+0−5

@@ -25,11 +25,6 @@ struct io_rsrc_node { · 24 unmodified lines
25	25	};
26	26	};
27	27
28		−enum {
29		− IO_IMU_DEST = 1 << ITER_DEST,
30		− IO_IMU_SOURCE = 1 << ITER_SOURCE,
31		−};
32		−
33	28	enum {
34	29	IO_REGBUF_F_KBUF = 1,
35	30	};

tools/testing/selftests/filesystems/fuse/.gitignore+1−0

@@ -1,3 +1,4 @@
1	1	# SPDX-License-Identifier: GPL-2.0-only
2	2	fuse_mnt
3	3	fusectl_test
	4	+write_extend_eof_test

tools/testing/selftests/filesystems/fuse/Makefile+3−0

@@ -3,6 +3,7 @@ · 2 unmodified lines
3	3	CFLAGS += -Wall -O2 -g $(KHDR_INCLUDES)
4	4
5	5	TEST_GEN_PROGS := fusectl_test
	6	+TEST_GEN_PROGS += write_extend_eof_test
6	7	TEST_GEN_FILES := fuse_mnt
7	8
8	9	# fuse_acl_cache_test requires libfuse3; add it only when the library is present.
@@ -14,6 +15,8 @@ endif · 5 unmodified lines
14	15
15	16	include ../../lib.mk
16	17
	18	+$(OUTPUT)/write_extend_eof_test: LDLIBS += -lpthread
	19	+
17	20	VAR_CFLAGS := $(shell pkg-config fuse --cflags 2>/dev/null)
18	21	ifeq ($(VAR_CFLAGS),)
19	22	VAR_CFLAGS := -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse

tools/testing/selftests/filesystems/fuse/write_extend_eof_test.c+368−0

@@ -0,0 +1,368 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/*
	3	+ * Regression test for the fuse write-extend partial-EOF-page zeroing bug.
	4	+ *
	5	+ * A buffered write that extends i_size past a non-page-aligned EOF must zero
	6	+ * the tail of the old last page. If an application has mmap'd that page and
	7	+ * stored into the post-EOF region (undefined until the file grows), the
	8	+ * now-in-bounds tail must read back as zero, not as the stale stored bytes.
	9	+ *
	10	+ * The bug is exposed on a non-writeback_cache server that keeps the page cache
	11	+ * across the write (FOPEN_KEEP_CACHE without FOPEN_DIRECT_IO). This test is a
	12	+ * raw /dev/fuse server in that mode; the backing data is always zero in the
	13	+ * hole, so any non-zero byte a read sees is stale page-cache data.
	14	+ *
	15	+ * Requires root to mount fuse.
	16	+ */
	17	+#define _GNU_SOURCE
	18	+#include <errno.h>
	19	+#include <fcntl.h>
	20	+#include <linux/falloc.h>
	21	+#include <pthread.h>
	22	+#include <stdint.h>
	23	+#include <string.h>
	24	+#include <unistd.h>
	25	+#include <sys/mman.h>
	26	+#include <sys/mount.h>
	27	+#include <sys/stat.h>
	28	+#include <sys/uio.h>
	29	+#include <linux/fuse.h>
	30	+
	31	+#include "../../kselftest_harness.h"
	32	+
	33	+#define FUSE_ROOT_ID 1
	34	+#define FILE_INO 2
	35	+#define MAX_WRITE (128 * 1024)
	36	+#define BACKING_SIZE (4 * 1024 * 1024)
	37	+#define POLLUTE 0xee
	38	+
	39	+/* Server-side state, shared with the responder thread. */
	40	+struct server {
	41	+ int fd;
	42	+ unsigned char backing[BACKING_SIZE]; /* authoritative bytes */
	43	+ uint64_t size;
	44	+};
	45	+
	46	+static void reply(int fd, uint64_t unique, int error, void *data, size_t len)
	47	+{
	48	+ struct fuse_out_header oh = {
	49	+ .len = sizeof(oh) + (data ? len : 0),
	50	+ .error = error,
	51	+ .unique = unique,
	52	+ };
	53	+ struct iovec iov[2] = { { &oh, sizeof(oh) }, { data, len } };
	54	+
	55	+ /* Errors here are teardown races (device closed on unmount); ignore. */
	56	+ if (writev(fd, iov, data ? 2 : 1) < 0)
	57	+ return;
	58	+}
	59	+
	60	+static void fill_attr(struct fuse_attr *a, uint64_t ino, uint32_t mode,
	61	+ uint64_t size)
	62	+{
	63	+ memset(a, 0, sizeof(*a));
	64	+ a->ino = ino;
	65	+ a->mode = mode;
	66	+ a->nlink = 1;
	67	+ a->size = size;
	68	+ a->blksize = sysconf(_SC_PAGESIZE);
	69	+}
	70	+
	71	+static void server_thread(void arg)
	72	+{
	73	+ struct server *s = arg;
	74	+ static char buf[MAX_WRITE + 4096];
	75	+
	76	+ for (;;) {
	77	+ ssize_t n = read(s->fd, buf, sizeof(buf));
	78	+ struct fuse_in_header ih = (void )buf;
	79	+
	80	+ if (n < 0) {
	81	+ if (errno == EINTR \|\| errno == EAGAIN)
	82	+ continue;
	83	+ return NULL; /* device closed on unmount */
	84	+ }
	85	+ if (n < (ssize_t)sizeof(*ih))
	86	+ continue;
	87	+
	88	+ switch (ih->opcode) {
	89	+ case FUSE_INIT: {
	90	+ struct fuse_init_in in = (void )(ih + 1);
	91	+ struct fuse_init_out out = {0};
	92	+
	93	+ /* No FUSE_WRITEBACK_CACHE: the exposed configuration. */
	94	+ out.major = FUSE_KERNEL_VERSION;
	95	+ out.minor = FUSE_KERNEL_MINOR_VERSION;
	96	+ out.max_readahead = in->max_readahead;
	97	+ out.max_write = MAX_WRITE;
	98	+ out.max_background = 16;
	99	+ out.congestion_threshold = 12;
	100	+ out.flags = FUSE_MAX_PAGES;
	101	+ out.max_pages = MAX_WRITE / sysconf(_SC_PAGESIZE);
	102	+ reply(s->fd, ih->unique, 0, &out, sizeof(out));
	103	+ break;
	104	+ }
	105	+ case FUSE_GETATTR: {
	106	+ struct fuse_attr_out out = {0};
	107	+ int root = ih->nodeid == FUSE_ROOT_ID;
	108	+
	109	+ out.attr_valid = 3600;
	110	+ fill_attr(&out.attr, ih->nodeid,
	111	+ root ? (S_IFDIR \| 0755) : (S_IFREG \| 0644),
	112	+ root ? 0 : s->size);
	113	+ reply(s->fd, ih->unique, 0, &out, sizeof(out));
	114	+ break;
	115	+ }
	116	+ case FUSE_LOOKUP: {
	117	+ struct fuse_entry_out out = {0};
	118	+
	119	+ out.nodeid = FILE_INO;
	120	+ out.attr_valid = 3600;
	121	+ out.entry_valid = 3600;
	122	+ fill_attr(&out.attr, FILE_INO, S_IFREG \| 0644, s->size);
	123	+ reply(s->fd, ih->unique, 0, &out, sizeof(out));
	124	+ break;
	125	+ }
	126	+ case FUSE_OPEN:
	127	+ case FUSE_OPENDIR: {
	128	+ struct fuse_open_out out = {0};
	129	+
	130	+ /* Keep the cache across the write, but not direct I/O. */
	131	+ out.open_flags = FOPEN_KEEP_CACHE;
	132	+ reply(s->fd, ih->unique, 0, &out, sizeof(out));
	133	+ break;
	134	+ }
	135	+ case FUSE_READ: {
	136	+ struct fuse_read_in in = (void )(ih + 1);
	137	+ uint64_t off = in->offset;
	138	+ uint32_t size = in->size;
	139	+
	140	+ if (off >= BACKING_SIZE)
	141	+ size = 0;
	142	+ else if (off + size > BACKING_SIZE)
	143	+ size = BACKING_SIZE - off;
	144	+ reply(s->fd, ih->unique, 0, s->backing + off, size);
	145	+ break;
	146	+ }
	147	+ case FUSE_WRITE: {
	148	+ struct fuse_write_in in = (void )(ih + 1);
	149	+ struct fuse_write_out out = {0};
	150	+ uint64_t off = in->offset;
	151	+ uint32_t size = in->size;
	152	+
	153	+ if (off < BACKING_SIZE) {
	154	+ uint32_t c = size;
	155	+
	156	+ if (off + c > BACKING_SIZE)
	157	+ c = BACKING_SIZE - off;
	158	+ memcpy(s->backing + off, in + 1, c);
	159	+ if (off + c > s->size)
	160	+ s->size = off + c;
	161	+ }
	162	+ out.size = size;
	163	+ reply(s->fd, ih->unique, 0, &out, sizeof(out));
	164	+ break;
	165	+ }
	166	+ case FUSE_SETATTR: {
	167	+ struct fuse_setattr_in in = (void )(ih + 1);
	168	+ struct fuse_attr_out out = {0};
	169	+
	170	+ if ((in->valid & FATTR_SIZE) && in->size <= BACKING_SIZE) {
	171	+ if (in->size > s->size)
	172	+ memset(s->backing + s->size, 0,
	173	+ in->size - s->size);
	174	+ s->size = in->size;
	175	+ }
	176	+ out.attr_valid = 3600;
	177	+ fill_attr(&out.attr, ih->nodeid, S_IFREG \| 0644, s->size);
	178	+ reply(s->fd, ih->unique, 0, &out, sizeof(out));
	179	+ break;
	180	+ }
	181	+ case FUSE_FALLOCATE: {
	182	+ struct fuse_fallocate_in in = (void )(ih + 1);
	183	+ uint64_t end = in->offset + in->length;
	184	+
	185	+ /* Only plain (size-extending) fallocate is used here. */
	186	+ if (!(in->mode & FALLOC_FL_KEEP_SIZE) &&
	187	+ end <= BACKING_SIZE && end > s->size) {
	188	+ memset(s->backing + s->size, 0, end - s->size);
	189	+ s->size = end;
	190	+ }
	191	+ reply(s->fd, ih->unique, 0, NULL, 0);
	192	+ break;
	193	+ }
	194	+ case FUSE_FLUSH:
	195	+ case FUSE_RELEASE:
	196	+ case FUSE_RELEASEDIR:
	197	+ case FUSE_FSYNC:
	198	+ case FUSE_ACCESS:
	199	+ reply(s->fd, ih->unique, 0, NULL, 0);
	200	+ break;
	201	+ case FUSE_FORGET:
	202	+ break;
	203	+ default:
	204	+ reply(s->fd, ih->unique, -EOPNOTSUPP, NULL, 0);
	205	+ break;
	206	+ }
	207	+ }
	208	+}
	209	+
	210	+FIXTURE(fuse)
	211	+{
	212	+ struct server *srv;
	213	+ pthread_t thread;
	214	+ char dir[64];
	215	+ long page; /* runtime page size */
	216	+ off_t eof; /* mid-page EOF, page-relative */
	217	+ int fd; /* open test file */
	218	+ char map; / mmap of the EOF page */
	219	+ int mounted;
	220	+};
	221	+
	222	+FIXTURE_SETUP(fuse)
	223	+{
	224	+ char opts[128];
	225	+ pthread_t t;
	226	+
	227	+ if (geteuid() != 0)
	228	+ SKIP(return, "need root to mount fuse");
	229	+
	230	+ self->page = sysconf(_SC_PAGESIZE);
	231	+ self->fd = -1;
	232	+ self->map = MAP_FAILED;
	233	+
	234	+ self->srv = mmap(NULL, sizeof(*self->srv), PROT_READ \| PROT_WRITE,
	235	+ MAP_SHARED \| MAP_ANONYMOUS, -1, 0);
	236	+ ASSERT_NE(MAP_FAILED, self->srv);
	237	+
	238	+ self->srv->fd = open("/dev/fuse", O_RDWR);
	239	+ ASSERT_GE(self->srv->fd, 0);
	240	+
	241	+ strcpy(self->dir, "/tmp/fuse_weof_XXXXXX");
	242	+ ASSERT_NE(NULL, mkdtemp(self->dir));
	243	+
	244	+ snprintf(opts, sizeof(opts),
	245	+ "fd=%d,rootmode=40000,user_id=0,group_id=0",
	246	+ self->srv->fd);
	247	+ ASSERT_EQ(0, mount("fuse", self->dir, "fuse", 0, opts));
	248	+ self->mounted = 1;
	249	+
	250	+ ASSERT_EQ(0, pthread_create(&t, NULL, server_thread, self->srv));
	251	+ self->thread = t;
	252	+}
	253	+
	254	+FIXTURE_TEARDOWN(fuse)
	255	+{
	256	+ if (self->map != MAP_FAILED)
	257	+ munmap(self->map, self->page);
	258	+ if (self->fd >= 0)
	259	+ close(self->fd);
	260	+ if (self->mounted)
	261	+ umount2(self->dir, MNT_DETACH);
	262	+ if (self->srv && self->srv != MAP_FAILED) {
	263	+ if (self->srv->fd > 0)
	264	+ close(self->srv->fd);
	265	+ munmap(self->srv, sizeof(*self->srv));
	266	+ }
	267	+ if (self->dir[0])
	268	+ rmdir(self->dir);
	269	+}
	270	+
	271	+/*
	272	+ * Create the test file with a mid-page EOF and mmap-store POLLUTE into its
	273	+ * post-EOF tail (a legal store, undefined until the file grows). Leaves the
	274	+ * file open and the EOF page mapped in the fixture for the caller to extend.
	275	+ */
	276	+static void pollute_eof_tail(struct __test_metadata *_metadata,
	277	+ FIXTURE_DATA(fuse) * self)
	278	+{
	279	+ off_t eof = 2 * self->page + self->page / 4;
	280	+ char path[128];
	281	+ char *buf;
	282	+
	283	+ snprintf(path, sizeof(path), "%s/file", self->dir);
	284	+ self->fd = open(path, O_RDWR \| O_CREAT \| O_TRUNC, 0644);
	285	+ ASSERT_GE(self->fd, 0);
	286	+ self->eof = eof;
	287	+
	288	+ buf = malloc(eof);
	289	+ ASSERT_NE(NULL, buf);
	290	+ memset(buf, 'A', eof);
	291	+ ASSERT_EQ(eof, pwrite(self->fd, buf, eof, 0));
	292	+ free(buf);
	293	+
	294	+ self->map = mmap(NULL, self->page, PROT_READ \| PROT_WRITE, MAP_SHARED,
	295	+ self->fd, eof & ~(self->page - 1));
	296	+ ASSERT_NE(MAP_FAILED, self->map);
	297	+ memset(self->map + (eof & (self->page - 1)), POLLUTE,
	298	+ self->page - (eof & (self->page - 1)));
	299	+}
	300	+
	301	+/* Assert the old post-EOF tail [eof, end of its page) now reads back as zero. */
	302	+static void assert_tail_zeroed(struct __test_metadata *_metadata,
	303	+ FIXTURE_DATA(fuse) * self)
	304	+{
	305	+ off_t base = self->eof & ~(self->page - 1);
	306	+ char *tail = malloc(self->page);
	307	+ int i;
	308	+
	309	+ ASSERT_NE(NULL, tail);
	310	+ ASSERT_EQ(self->page, pread(self->fd, tail, self->page, base));
	311	+ for (i = self->eof & (self->page - 1); i < self->page; i++)
	312	+ ASSERT_EQ(0, tail[i]);
	313	+ free(tail);
	314	+}
	315	+
	316	+/* Basic: pollute the post-EOF tail, extend past it by a later write. */
	317	+TEST_F(fuse, write_extend)
	318	+{
	319	+ pollute_eof_tail(_metadata, self);
	320	+ ASSERT_EQ(4, pwrite(self->fd, "data", 4, 5 * self->page + self->page / 3));
	321	+ assert_tail_zeroed(_metadata, self);
	322	+}
	323	+
	324	+/* Extend via ftruncate() rather than a write. */
	325	+TEST_F(fuse, ftruncate_extend)
	326	+{
	327	+ pollute_eof_tail(_metadata, self);
	328	+ ASSERT_EQ(0, ftruncate(self->fd, 8 * self->page));
	329	+ assert_tail_zeroed(_metadata, self);
	330	+}
	331	+
	332	+/* Extend via fallocate() starting at the old EOF. */
	333	+TEST_F(fuse, fallocate_extend)
	334	+{
	335	+ pollute_eof_tail(_metadata, self);
	336	+ ASSERT_EQ(0, fallocate(self->fd, 0, self->eof, 4 * self->page));
	337	+ assert_tail_zeroed(_metadata, self);
	338	+}
	339	+
	340	+/* A write landing inside the old EOF page must not clobber its own data. */
	341	+TEST_F(fuse, extend_into_eof_page_preserves_data)
	342	+{
	343	+ off_t base, wr;
	344	+ char buf, rd;
	345	+ int i;
	346	+
	347	+ pollute_eof_tail(_metadata, self);
	348	+ base = self->eof & ~(self->page - 1);
	349	+ wr = base + 3 * self->page / 4; /* starts in the EOF page */
	350	+
	351	+ buf = malloc(2 * self->page);
	352	+ ASSERT_NE(NULL, buf);
	353	+ memset(buf, 'B', 2 * self->page);
	354	+ ASSERT_EQ(2 * self->page, pwrite(self->fd, buf, 2 * self->page, wr));
	355	+ free(buf);
	356	+
	357	+ rd = malloc(self->page);
	358	+ ASSERT_NE(NULL, rd);
	359	+ ASSERT_EQ(self->page, pread(self->fd, rd, self->page, base));
	360	+ /* [eof, wr) is hole -> zero; [wr, page) is written data -> 'B'. */
	361	+ for (i = self->eof & (self->page - 1); i < wr - base; i++)
	362	+ ASSERT_EQ(0, rd[i]);
	363	+ for (i = wr - base; i < self->page; i++)
	364	+ ASSERT_EQ('B', rd[i]);
	365	+ free(rd);
	366	+}
	367	+
	368	+TEST_HARNESS_MAIN