lael-labs/linux

Documentation/block/ublk.rst+7−7

Documentation/filesystems/fuse/fuse-io-uring.rst+34−2

@@ -11,6 +11,9 @@ and works. For generic details about FUSE see fuse.rst. · 10 unmodified lines
11	This document also covers the current interface, which is	11	This document also covers the current interface, which is
12	still in development and might change.	12	still in development and might change.
13		13
		14	For the userspace protocol, see
		15	Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst.
		16
14	Limitations	17	Limitations
15	===========	18	===========
16	As of now not all requests types are supported through io-uring, userspace	19	As of now not all requests types are supported through io-uring, userspace
@@ -95,5 +98,34 @@ Sending requests with CQEs · 78 unmodified lines
95	\| <fuse_unlink() \|	98	\| <fuse_unlink() \|
96	\| <sys_unlink() \|	99	\| <sys_unlink() \|
97		100
98		101	Buffer pools
99		102	============
		103
		104	Without a buffer pool, every entry needs to pass a dedicated payload buffer
		105	large enough for the maximum payload size. A buffer pool decouples entries
		106	from payload buffers. The server hands the kernel one contiguous buffer pool
		107	of memory and when the kernel sends the server a request, it indicates the
		108	offset into the pool for that request's payload. Internally, the kernel is
		109	able to manage/optimize the buffer pool memory however it likes.
		110
		111	A server may also register the pool region with io_uring as a fixed buffer.
		112	The backing pages are then pinned once, avoiding per-request pinning and
		113	address translation. This also allows servers to use the same registered
		114	buffers for subsequent backing store I/O through io-uring, keeping data
		115	in the same pinned pages without additional pinning / mapping overhead.
		116
		117	Zero-copy
		118	=========
		119
		120	Zero-copy lets the server read from / write to the client's pages (pinned
		121	user pages for direct I/O, or page-cache folios for buffered I/O) without an
		122	intermediary payload copy. This requires CAP_SYS_ADMIN privileges.
		123
		124	When a fuse request arrives for a file that opted into zero-copy, the kernel
		125	registers the relevant pages (pinned user pages for direct i/o or underlying
		126	page cache folios for buffered i/o) into a sparse slot in the server's
		127	io_uring registered buffer table. The server can then operate on these pages
		128	directly using io-uring fixed buffer operations (eg read_fixed / write_fixed)
		129	and the kernel unregisters these pages when the request completes.
		130	Non-page-backed args (eg op out headers) will go through the payload buffer as
		131	normal.

Documentation/filesystems/fuse/index.rst+1−0

@@ -12,3 +12,4 @@ FUSE (Filesystem in Userspace) Technical Documentation · 11 unmodified lines
12	fuse-io	12	fuse-io
13	fuse-io-uring	13	fuse-io-uring
14	fuse-passthrough	14	fuse-passthrough
		15	uapi/fuse-uapi-io-uring

Documentation/filesystems/fuse/uapi/fuse-uapi-io-uring.rst+126−0

@@ -0,0 +1,126 @@
	1	.. SPDX-License-Identifier: GPL-2.0
	2
	3	=====================================
	4	FUSE-over-io-uring uapi documentation
	5	=====================================
	6
	7	Commands
	8	========
	9
	10	``enum fuse_uring_cmd``:
	11
	12	``FUSE_IO_URING_CMD_ADD_QUEUE``
	13	Create a queue identified by ``fuse_uring_cmd_req.qid``. Queue-wide
	14	options are passed in ``fuse_uring_cmd_req.flags``:
	15
	16	``FUSE_URING_ZERO_COPY``
	17	Enable zero-copy on this queue. Requires ``CAP_SYS_ADMIN`` and a buffer
	18	pool, which is added separately via ``ADD_BUFPOOL`` before registering
	19	entries (see `Zero-copy`_).
	20
	21	``FUSE_IO_URING_CMD_ADD_BUFPOOL``
	22	Register the payload buffer pool for an existing queue. The server provides
	23	a single contiguous region in ``fuse_uring_cmd_req.bufpool.uaddr`` /
	24	``.len``. This command must be issued after ``ADD_QUEUE`` and before
	25	registering any payload-carrying entries on that queue.
	26	``fuse_uring_cmd_req.flags`` must be 0. Submitting this command with
	27	``IORING_URING_CMD_FIXED`` marks the pool as registered, which avoids per
	28	i/o pinning/unpinning and mapping overhead (see `Buffer pools`_).
	29
	30	``FUSE_IO_URING_CMD_REGISTER``
	31	Register a ring entry (a long-lived SQE that carries the request header
	32	iovec). For a zero-copy queue, ``fuse_uring_cmd_req.ent_zero_copy_buf_index``
	33	indicates the reserved registered buffer table slot this entry uses for
	34	zero-copy (see `Zero-copy`_).
	35
	36	``FUSE_IO_URING_CMD_COMMIT_AND_FETCH``
	37	Commit the reply for a completed request and fetch the next one. The
	38	request is identified by ``fuse_uring_cmd_req.commit_id`` (the value the
	39	kernel reported in ``fuse_uring_ent_in_out.commit_id``).
	40
	41	Structures
	42	==========
	43
	44	``struct fuse_uring_cmd_req`` (80-byte SQE command area):
	45
	46	============================ ==================================================
	47	Field Meaning
	48	============================ ==================================================
	49	``flags`` Command-specific flags (see each command).
	50	``commit_id`` Request id, for ``COMMIT_AND_FETCH``.
	51	``qid`` Queue index.
	52	``bufpool.uaddr`` Pool base address, for ``ADD_BUFPOOL``.
	53	``bufpool.len`` Pool length in bytes, for ``ADD_BUFPOOL``.
	54	``bufpool.reserved`` Must be 0, for ``ADD_BUFPOOL``.
	55	``ent_zero_copy_buf_index`` Per-entry zero-copy slot, for ``REGISTER``.
	56	============================ ==================================================
	57
	58	``struct fuse_uring_ent_in_out`` (reported by the kernel per request):
	59
	60	============================ ==================================================
	61	Field Meaning
	62	============================ ==================================================
	63	``flags`` ``FUSE_URING_ENT_ZERO_COPY`` if zero-copied.
	64	``commit_id`` Id to echo back in ``COMMIT_AND_FETCH``.
	65	``payload_sz`` Total payload size in bytes (see `Zero-copy`_).
	66	``offset`` Payload buffer offset within the pool.
	67	============================ ==================================================
	68
	69	Buffer pools
	70	============
	71	Setup:
	72
	73	* Issue ``ADD_QUEUE`` for the qid.
	74	* Issue ``ADD_BUFPOOL`` with ``bufpool.uaddr`` and ``bufpool.len`` pointing
	75	at the region.
	76	* Register entries with ``REGISTER``.
	77
	78	For every request that has a payload, the kernel reports where the payload
	79	lives in ``struct fuse_uring_ent_in_out`` (part of
	80	``struct fuse_uring_req_header``):
	81
	82	``offset``
	83	Byte offset, within the pool region, for this request's payload buffer.
	84	The server adds this to the pool base address to locate the payload.
	85
	86	``payload_sz``
	87	Number of payload bytes for this request.
	88
	89	To use registered buffers, the server registers the pool region with io_uring
	90	and submits ``ADD_BUFPOOL`` with ``IORING_URING_CMD_FIXED`` set in
	91	``sqe->uring_cmd_flags`` and the index of the registered bufpool in
	92	``sqe->buf_index``. Every SQE the server submits afterwards must follow the
	93	same fixed-buffer protocol, carrying ``IORING_URING_CMD_FIXED`` and that same
	94	``sqe->buf_index``. The same registered buffer can be reused for the server's
	95	backing-store I/O as well (e.g. ``IORING_OP_READ_FIXED`` /
	96	``IORING_OP_WRITE_FIXED``).
	97
	98	Zero-copy
	99	=========
	100	Requirements:
	101
	102	* The server must be privileged (``CAP_SYS_ADMIN``).
	103	* A zero-copy queue: ``ADD_QUEUE`` with the ``FUSE_URING_ZERO_COPY`` flag set.
	104	* A buffer pool: ``ADD_BUFPOOL``.
	105	* For each entry, ``REGISTER`` with ``ent_zero_copy_buf_index`` set to the
	106	index this entry uses in the server's io_uring registered-buffer table.
	107	This is where the kernel registers the request's pages for the server to
	108	access (it is separate from the payload pool). On a non-zero-copy queue this
	109	field must be 0.
	110
	111	Zero-copy is selected per open file. The server sets the open-file flag in
	112	the ``FUSE_OPEN`` / ``FUSE_CREATE`` reply:
	113
	114	``FOPEN_IO_URING_ZERO_COPY``
	115	Reads/writes on this open file should use zero-copy.
	116
	117	For a request that is zero-copied, the kernel sets ``FUSE_URING_ENT_ZERO_COPY``
	118	in ``fuse_uring_ent_in_out.flags`` and places the request's pages at the
	119	entry's ``ent_zero_copy_buf_index``. The server then issues
	120	``IORING_OP_READ_FIXED`` / ``IORING_OP_WRITE_FIXED`` against that index to
	121	transfer the data directly to/from the client's pages.
	122
	123	For such a request, ``payload_sz`` includes the zero-copied page bytes
	124	(transferred via the registered buffer at ``ent_zero_copy_buf_index``). Any
	125	non-page-backed args (e.g. op headers) are still copied through the pool
	126	payload buffer at ``offset``.

drivers/block/ublk_drv.c+11−11

@@ -1699,8 +1699,8 @@ ublk_auto_buf_register(const struct ublk_queue ubq, struct request req, · 1698 unmodified lines
1699	{	1699	{
1700	int ret;	1700	int ret;
1701		1701
1702	ret = io_buffer_register_bvec(cmd, req, ublk_io_release,	1702	ret = io_buffer_register_request(cmd, req, ublk_io_release,
1703	io->buf.auto_reg.index, issue_flags);	1703	io->buf.auto_reg.index, issue_flags);
1704	if (ret) {	1704	if (ret) {
1705	if (io->buf.auto_reg.flags & UBLK_AUTO_BUF_REG_FALLBACK) {	1705	if (io->buf.auto_reg.flags & UBLK_AUTO_BUF_REG_FALLBACK) {
1706	ublk_auto_buf_reg_fallback(ubq, req->tag);	1706	ublk_auto_buf_reg_fallback(ubq, req->tag);
@@ -1909,7 +1909,7 @@ static noinline void ublk_batch_dispatch_fail(struct ublk_queue *ubq, · 202 unmodified lines
1909	ublk_io_unlock(io);	1909	ublk_io_unlock(io);
1910		1910
1911	if (index != -1)	1911	if (index != -1)
1912	io_buffer_unregister_bvec(data->cmd, index,	1912	io_buffer_unregister(data->cmd, index,
1913	data->issue_flags);	1913	data->issue_flags);
1914	}	1914	}
1915		1915
@@ -3192,8 +3192,8 @@ static int ublk_register_io_buf(struct io_uring_cmd *cmd, · 1276 unmodified lines
3192	if (!req)	3192	if (!req)
3193	return -EINVAL;	3193	return -EINVAL;
3194		3194
3195	ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,	3195	ret = io_buffer_register_request(cmd, req, ublk_io_release, index,
3196	issue_flags);	3196	issue_flags);
3197	if (ret) {	3197	if (ret) {
3198	ublk_put_req_ref(io, req);	3198	ublk_put_req_ref(io, req);
3199	return ret;	3199	return ret;
@@ -3224,8 +3224,8 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd, · 24 unmodified lines
3224	if (!ublk_dev_support_zero_copy(ub) \|\| !blk_rq_has_data(req))	3224	if (!ublk_dev_support_zero_copy(ub) \|\| !blk_rq_has_data(req))
3225	return -EINVAL;	3225	return -EINVAL;
3226		3226
3227	ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index,	3227	ret = io_buffer_register_request(cmd, req, ublk_io_release, index,
3228	issue_flags);	3228	issue_flags);
3229	if (ret)	3229	if (ret)
3230	return ret;	3230	return ret;
3231		3231
@@ -3240,7 +3240,7 @@ static int ublk_unregister_io_buf(struct io_uring_cmd *cmd, · 8 unmodified lines
3240	if (!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY))	3240	if (!(ub->dev_info.flags & UBLK_F_SUPPORT_ZERO_COPY))
3241	return -EINVAL;	3241	return -EINVAL;
3242		3242
3243	return io_buffer_unregister_bvec(cmd, index, issue_flags);	3243	return io_buffer_unregister(cmd, index, issue_flags);
3244	}	3244	}
3245		3245
3246	static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr)	3246	static int ublk_check_fetch_buf(const struct ublk_device *ub, __u64 buf_addr)
@@ -3384,7 +3384,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, · 137 unmodified lines
3384	goto out;	3384	goto out;
3385		3385
3386	/*	3386	/*
3387	* io_buffer_unregister_bvec() doesn't access the ubq or io,	3387	* io_buffer_unregister() doesn't access the ubq or io,
3388	* so no need to validate the q_id, tag, or task	3388	* so no need to validate the q_id, tag, or task
3389	*/	3389	*/
3390	if (_IOC_NR(cmd_op) == UBLK_IO_UNREGISTER_IO_BUF)	3390	if (_IOC_NR(cmd_op) == UBLK_IO_UNREGISTER_IO_BUF)
@@ -3456,7 +3456,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd, · 65 unmodified lines
3456	req = ublk_fill_io_cmd(io, cmd);	3456	req = ublk_fill_io_cmd(io, cmd);
3457	ublk_apply_io_buf(ub, io, cmd, addr, &auto_buf, &buf_idx);	3457	ublk_apply_io_buf(ub, io, cmd, addr, &auto_buf, &buf_idx);
3458	if (buf_idx != UBLK_INVALID_BUF_IDX)	3458	if (buf_idx != UBLK_INVALID_BUF_IDX)
3459	io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);	3459	io_buffer_unregister(cmd, buf_idx, issue_flags);
3460	compl = ublk_need_complete_req(ub, io);	3460	compl = ublk_need_complete_req(ub, io);
3461		3461
3462	if (req_op(req) == REQ_OP_ZONE_APPEND)	3462	if (req_op(req) == REQ_OP_ZONE_APPEND)
@@ -3801,7 +3801,7 @@ static int ublk_batch_commit_io(struct ublk_queue *ubq, · 338 unmodified lines
3801	}	3801	}
3802		3802
3803	if (buf_idx != UBLK_INVALID_BUF_IDX)	3803	if (buf_idx != UBLK_INVALID_BUF_IDX)
3804	io_buffer_unregister_bvec(data->cmd, buf_idx, data->issue_flags);	3804	io_buffer_unregister(data->cmd, buf_idx, data->issue_flags);
3805	if (req_op(req) == REQ_OP_ZONE_APPEND)	3805	if (req_op(req) == REQ_OP_ZONE_APPEND)
3806	req->__sector = ublk_batch_zone_lba(uc, elem);	3806	req->__sector = ublk_batch_zone_lba(uc, elem);
3807	if (compl)	3807	if (compl)

fs/fuse/args.h+2−0

 	bool is_pinned:1;
 	bool invalidate_vmap:1;
 	bool abort_on_kill:1;
+	/* server requested io-uring zero-copy for this op */
+	bool zero_copy:1;
 	struct fuse_in_arg in_args[4];
 	struct fuse_arg out_args[2];
 	void (*end)(struct fuse_args *args, int error);

fs/fuse/cuse.c+7−1

 	INIT_LIST_HEAD(&cc->list);
-	cc->fc.chan->initialized = 1;
+	/* Pairs with smp_load_acquire() readers of fch->initialized */
+	smp_store_release(&cc->fc.chan->initialized, 1);
 	rc = cuse_send_init(cc);
 	if (rc) {
 		fuse_dev_put(fud);
 {
 	misc_deregister(&cuse_miscdev);
 	class_destroy(cuse_class);
+	/*
+	 * Wait for pending call_rcu() callbacks that call back into
+	 * this module via fc->release (cuse_fc_release).
+	 */
+	rcu_barrier();
 }
 module_init(cuse_init);

fs/fuse/dev.c+67−37

 		fch->minor = param->minor;
 		fch->max_write = param->max_write;
 		fch->max_pages = param->max_pages;
+		if (param->io_uring_enabled)
+			fuse_uring_conn_init(fch);
 	}
-	/* Make sure stores before this are seen on another CPU */
+	/* Pairs with smp_load_acquire() readers of fch->initialized */
-	smp_wmb();
+	smp_store_release(&fch->initialized, 1);
-	fch->initialized = 1;
 	wake_up_all(&fch->blocked_waitq);
 }
 static bool fuse_block_alloc(struct fuse_chan *fch, bool for_background)
 {
-	return !fch->initialized || (for_background && fch->blocked) ||
+	/* Pairs with smp_store_release() in fuse_chan_set_initialized() */
+	if (!smp_load_acquire(&fch->initialized))
+		return true;
+	return (for_background && fch->blocked) ||
 	       (fch->io_uring && fch->connected && !fuse_uring_ready(fch));
 }
 			goto out;
 	}
-	/* Matches smp_wmb() in fuse_chan_set_initialized() */
-	smp_rmb();
 	err = -ENOTCONN;
 	if (!fch->connected)
 		goto out;
 /*
  * A new request is available, wake fiq->waitq
  */
-static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq)
+static void fuse_dev_wake_and_unlock(struct fuse_iqueue *fiq, bool sync)
 __releases(fiq->lock)
 {
-	wake_up(&fiq->waitq);
+	if (sync)
+		wake_up_sync(&fiq->waitq);
+	else
+		wake_up(&fiq->waitq);
 	kill_fasync(&fiq->fasync, SIGIO, POLL_IN);
 	spin_unlock(&fiq->lock);
 }
 	if (fiq->connected) {
 		fiq->forget_list_tail->next = forget;
 		fiq->forget_list_tail = forget;
-		fuse_dev_wake_and_unlock(fiq);
+		fuse_dev_wake_and_unlock(fiq, false);
 	} else {
 		kfree(forget);
 		spin_unlock(&fiq->lock);
 void fuse_dev_queue_interrupt(struct fuse_iqueue *fiq, struct fuse_req *req)
 {
 	spin_lock(&fiq->lock);
-	if (list_empty(&req->intr_entry)) {
+	/* Repeat FR_SENT test after obtaining the lock to prevent race with fuse_resend() */
+	if (list_empty(&req->intr_entry) && test_bit(FR_SENT, &req->flags)) {
 		list_add_tail(&req->intr_entry, &fiq->interrupts);
 		/*
 		 * Pairs with smp_mb() implied by test_and_set_bit()
 			list_del_init(&req->intr_entry);
 			spin_unlock(&fiq->lock);
 		} else  {
-			fuse_dev_wake_and_unlock(fiq);
+			fuse_dev_wake_and_unlock(fiq, false);
 		}
 	} else {
 		spin_unlock(&fiq->lock);
 static void fuse_dev_queue_req(struct fuse_iqueue *fiq, struct fuse_req *req)
 {
+	bool sync = test_and_clear_bit(FR_SYNC_WAKEUP, &req->flags);
 	spin_lock(&fiq->lock);
 	if (fiq->connected) {
 		fuse_request_assign_unique_locked(fiq, req);
 		list_add_tail(&req->list, &fiq->pending);
-		fuse_dev_wake_and_unlock(fiq);
+		fuse_dev_wake_and_unlock(fiq, sync);
 	} else {
 		spin_unlock(&fiq->lock);
 		req->out.h.error = -ENOTCONN;
 	fch->max_background = val;
 	fch->blocked = fch->num_background >= fch->max_background;
 	if (!fch->blocked)
-		wake_up(&fch->blocked_waitq);
+		wake_up_nr(&fch->blocked_waitq,
+			   fch->max_background - fch->num_background);
 	spin_unlock(&fch->bg_lock);
 }
 	fch->conn = fc;
 }
-void fuse_chan_io_uring_enable(struct fuse_chan *fch)
-{
-	fch->io_uring = 1;
-}
 void fuse_pqueue_init(struct fuse_pqueue *fpq)
 {
 	spin_lock_init(&fpq->lock);
 		if (req->args->abort_on_kill) {
 			fuse_chan_abort(fch, false);
-			return;
+			goto wait_for_finish;
 		}
 		if (test_bit(FR_URING, &req->flags))
 			return;
 	}
+wait_for_finish:
 	/*
 	 * Either request is already in userspace, or it was forced.
 	 * Wait it out.
 	/* acquire extra reference, since request is still needed after
 	   fuse_request_end() */
 	__fuse_get_request(req);
+	/*
+	 * This is a synchronous request: the caller will block waiting for
+	 * the answer. Hint the scheduler via wake_up_sync().
+	 */
+	set_bit(FR_SYNC_WAKEUP, &req->flags);
 	fuse_send_one(fiq, req);
 	request_wait_answer(req);
 	if (folio) {
 		size = folio_size(folio);
-		if (zeroing && count < size)
+		if (zeroing && count < size) {
-			folio_zero_range(folio, 0, size);
+			/*
+			 * When the copy is skipped the folio already holds the
+			 * payload, so only the bytes outside [offset, offset +
+			 * count) may be zeroed.
+			 *
+			 * Otherwise, the whole folio is cleared first so that a
+			 * failed copy leaves zeros rather than stale folio
+			 * contents.
+			 */
+			if (cs->skip_folio_copy)
+				folio_zero_segments(folio, 0, offset,
+						    offset + count, size);
+			else
+				folio_zero_range(folio, 0, size);
+		}
 	}
-	while (count) {
+	while (!cs->skip_folio_copy && count) {
 		if (cs->write && cs->pipebufs && folio) {
 			/*
 			 * Can't control lifetime of pipe buffers, so always
 	for (i = 0; !err && i < numargs; i++)  {
 		struct fuse_arg *arg = &args[i];
 		if (i == numargs - 1 && argpages)
+			/*
+			 * if cs->skip_folio_copy is set, this just does any
+			 * needed zeroing. No copying is involved.
+			 */
 			err = fuse_copy_folios(cs, arg->size, zeroing);
 		else
 			err = fuse_copy_one(cs, arg->value, arg->size);
 void fuse_chan_resend(struct fuse_chan *fch)
 {
 	struct fuse_dev *fud;
-	struct fuse_req *req, *next;
+	struct fuse_req *req;
 	struct fuse_iqueue *fiq = &fch->iq;
 	LIST_HEAD(to_queue);
 	unsigned int i;
 		struct fuse_pqueue *fpq = &fud->pq;
 		spin_lock(&fpq->lock);
-		for (i = 0; i < FUSE_PQ_HASH_SIZE; i++)
+		for (i = 0; i < FUSE_PQ_HASH_SIZE; i++) {
-			list_splice_tail_init(&fpq->processing[i], &to_queue);
+			struct list_head *this_queue = &fpq->processing[i];
+			list_for_each_entry(req, this_queue, list)
+				clear_bit(FR_SENT, &req->flags);
+			list_splice_tail_init(this_queue, &to_queue);
+		}
 		spin_unlock(&fpq->lock);
 	}
 	spin_unlock(&fch->lock);
-	list_for_each_entry_safe(req, next, &to_queue, list) {
-		set_bit(FR_PENDING, &req->flags);
-		clear_bit(FR_SENT, &req->flags);
-		/* mark the request as resend request */
-		req->in.h.unique |= FUSE_UNIQUE_RESEND;
-	}
 	spin_lock(&fiq->lock);
 	if (!fiq->connected) {
 		spin_unlock(&fiq->lock);
-		list_for_each_entry(req, &to_queue, list)
-			clear_bit(FR_PENDING, &req->flags);
 		fuse_dev_end_requests(&to_queue);
 		return;
 	}
 	 * intr_entry on fiq->interrupts after the request is re-queued.
 	 */
 	list_for_each_entry(req, &to_queue, list) {
+		set_bit(FR_PENDING, &req->flags);
+		/* mark the request as resend request */
+		req->in.h.unique |= FUSE_UNIQUE_RESEND;
 		if (test_bit(FR_INTERRUPTED, &req->flags))
 			list_del_init(&req->intr_entry);
 	}
 	/* iq and pq requests are both oldest to newest */
 	list_splice(&to_queue, &fiq->pending);
-	fuse_dev_wake_and_unlock(fiq);
+	fuse_dev_wake_and_unlock(fiq, false);
 }
 /* Look up request on processing list by unique ID */
 		 * initialized and connected state
 		 */
 		err = -EINVAL;
-		if (!fch->initialized || !fch->connected)
+		/* Pairs with smp_store_release() in fuse_chan_set_initialized() */
+		if (!smp_load_acquire(&fch->initialized) || !fch->connected)
 			goto copy_finish;
 		/* Don't try to move folios (yet) */

fs/fuse/dev.h+1−1

 	unsigned int minor;
 	unsigned int max_write;
 	unsigned int max_pages;
+	bool io_uring_enabled;
 };
 struct fuse_chan *fuse_chan_new(void);
 unsigned int fuse_chan_num_waiting(struct fuse_chan *fch);
 void fuse_chan_set_fc(struct fuse_chan *fch, struct fuse_conn *fc);
 void fuse_chan_set_initialized(struct fuse_chan *fch, struct fuse_chan_param *param);
-void fuse_chan_io_uring_enable(struct fuse_chan *fch);
 ssize_t fuse_chan_send(struct fuse_chan *fch, struct fuse_args *args);
 int fuse_chan_send_bg(struct fuse_chan *fch, struct fuse_args *args, gfp_t gfp_flags);
 int fuse_chan_send_notify_reply(struct fuse_chan *fch, struct fuse_args *args, u64 unique);

fs/fuse/dev_uring.c+541−92

@@ -9,6 +9,7 @@ · 8 unmodified lines
9	#include "dev_uring_i.h"	9	#include "dev_uring_i.h"
10	#include "fuse_trace.h"	10	#include "fuse_trace.h"
11		11
		12	#include <linux/bitmap.h>
12	#include <linux/fs.h>	13	#include <linux/fs.h>
13	#include <linux/io_uring/cmd.h>	14	#include <linux/io_uring/cmd.h>
14		15
@@ -21,6 +22,8 @@ MODULE_PARM_DESC(enable_uring, · 6 unmodified lines
21	#define FUSE_URING_IOV_HEADERS 0	22	#define FUSE_URING_IOV_HEADERS 0
22	#define FUSE_URING_IOV_PAYLOAD 1	23	#define FUSE_URING_IOV_PAYLOAD 1
23		24
		25	#define FUSE_URING_ADD_QUEUE_FLAGS (FUSE_URING_ZERO_COPY)
		26
24	bool fuse_uring_enabled(void)	27	bool fuse_uring_enabled(void)
25	{	28	{
26	return enable_uring;	29	return enable_uring;
@@ -30,6 +33,11 @@ struct fuse_uring_pdu { · 3 unmodified lines
30	struct fuse_ring_ent *ent;	33	struct fuse_ring_ent *ent;
31	};	34	};
32		35
		36	struct fuse_zero_copy_bvs {
		37	unsigned int nr_bvs;
		38	struct bio_vec bvs[];
		39	};
		40
33	static const struct fuse_iqueue_ops fuse_io_uring_ops;	41	static const struct fuse_iqueue_ops fuse_io_uring_ops;
34		42
35	enum fuse_uring_header_type {	43	enum fuse_uring_header_type {
@@ -41,6 +49,32 @@ enum fuse_uring_header_type { · 5 unmodified lines
41	FUSE_URING_HEADER_RING_ENT,	49	FUSE_URING_HEADER_RING_ENT,
42	};	50	};
43		51
		52	static inline bool bufpool_enabled(struct fuse_ring_queue *queue)
		53	{
		54	return queue->payload_mode == FUSE_PAYLOAD_BUFPOOL;
		55	}
		56
		57	static inline bool bufpool_registered(struct fuse_ring_queue *queue)
		58	{
		59	return queue->bufpool && queue->bufpool->registered;
		60	}
		61
		62	/*
		63	* For a registered bufpool, every sqe that drives a payload import (REGISTER,
		64	* COMMIT_AND_FETCH) must carry the registered buffer index of the pool.
		65	* This also must be called from the command's issue handler, where cmd->sqe is
		66	* still valid
		67	*/
		68	static inline bool fuse_uring_cmd_index_ok(struct io_uring_cmd *cmd,
		69	struct fuse_ring_queue *queue)
		70	{
		71	if (!bufpool_registered(queue))
		72	return true;
		73
		74	return (cmd->flags & IORING_URING_CMD_FIXED) &&
		75	READ_ONCE(cmd->sqe->buf_index) == queue->bufpool->registered_index;
		76	}
		77
44	static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,	78	static void uring_cmd_set_ring_ent(struct io_uring_cmd *cmd,
45	struct fuse_ring_ent *ring_ent)	79	struct fuse_ring_ent *ring_ent)
46	{	80	{
@@ -86,8 +120,36 @@ static void fuse_uring_flush_bg(struct fuse_ring_queue *queue) · 39 unmodified lines
86	}	120	}
87	}	121	}
88		122
		123	static bool can_zero_copy_req(struct fuse_ring_ent ent, struct fuse_req req)
		124	{
		125	struct fuse_args *args = req->args;
		126
		127	if (!ent->queue->zero_copy \|\| !args->zero_copy)
		128	return false;
		129
		130	if (args->opcode != FUSE_READ && args->opcode != FUSE_WRITE)
		131	return false;
		132
		133	return args->in_pages \|\| args->out_pages;
		134	}
		135
		136	static void zero_copy_unregister(struct io_uring_cmd *cmd,
		137	struct fuse_ring_ent *ent,
		138	unsigned int issue_flags)
		139	{
		140	if (ent->zero_copied) {
		141	int err = io_buffer_unregister(cmd, ent->zero_copy_index,
		142	issue_flags);
		143
		144	if (err)
		145	pr_warn_ratelimited("qid=%d zero-copy unregister failed: %d\n",
		146	ent->queue->qid, err);
		147	ent->zero_copied = false;
		148	}
		149	}
		150
89	static void fuse_uring_req_end(struct fuse_ring_ent ent, struct fuse_req req,	151	static void fuse_uring_req_end(struct fuse_ring_ent ent, struct fuse_req req,
90	int error)	152	int error, unsigned int issue_flags)
91	{	153	{
92	struct fuse_ring_queue *queue = ent->queue;	154	struct fuse_ring_queue *queue = ent->queue;
93	struct fuse_ring *ring = queue->ring;	155	struct fuse_ring *ring = queue->ring;
@@ -107,6 +169,8 @@ static void fuse_uring_req_end(struct fuse_ring_ent ent, struct fuse_req req, · 13 unmodified lines
107		169
108	spin_unlock(&queue->lock);	170	spin_unlock(&queue->lock);
109		171
		172	zero_copy_unregister(ent->cmd, ent, issue_flags);
		173
110	if (error)	174	if (error)
111	req->out.h.error = error;	175	req->out.h.error = error;
112		176
@@ -204,7 +268,7 @@ void fuse_uring_destruct(struct fuse_chan *fch) · 91 unmodified lines
204	return;	268	return;
205		269
206	for (qid = 0; qid < ring->nr_queues; qid++) {	270	for (qid = 0; qid < ring->nr_queues; qid++) {
207	struct fuse_ring_queue *queue = ring->queues[qid];	271	struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]);
208	struct fuse_ring_ent ent, next;	272	struct fuse_ring_ent ent, next;
209		273
210	if (!queue)	274	if (!queue)
@@ -222,8 +286,9 @@ void fuse_uring_destruct(struct fuse_chan *fch) · 11 unmodified lines
222	}	286	}
223		287
224	kfree(queue->fpq.processing);	288	kfree(queue->fpq.processing);
		289	kfree(queue->bufpool);
225	kfree(queue);	290	kfree(queue);
226	ring->queues[qid] = NULL;	291	WRITE_ONCE(ring->queues[qid], NULL);
227	}	292	}
228		293
229	kfree(ring->queues);	294	kfree(ring->queues);
@@ -238,7 +303,6 @@ static struct fuse_ring fuse_uring_create(struct fuse_chan fch) · 8 unmodified lines
238	{	303	{
239	struct fuse_ring *ring;	304	struct fuse_ring *ring;
240	size_t nr_queues = num_possible_cpus();	305	size_t nr_queues = num_possible_cpus();
241	struct fuse_ring *res = NULL;
242	size_t max_payload_size;	306	size_t max_payload_size;
243		307
244	ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT);	308	ring = kzalloc_obj(*ring, GFP_KERNEL_ACCOUNT);
@@ -258,12 +322,6 @@ static struct fuse_ring fuse_uring_create(struct fuse_chan fch) · 13 unmodified lines
258	spin_unlock(&fch->lock);	322	spin_unlock(&fch->lock);
259	goto out_err;	323	goto out_err;
260	}	324	}
261	if (fch->ring) {
262	/* race, another thread created the ring in the meantime */
263	spin_unlock(&fch->lock);
264	res = fch->ring;
265	goto out_err;
266	}
267		325
268	init_waitqueue_head(&ring->stop_waitq);	326	init_waitqueue_head(&ring->stop_waitq);
269		327
@@ -278,11 +336,18 @@ static struct fuse_ring fuse_uring_create(struct fuse_chan fch) · 8 unmodified lines
278	out_err:	336	out_err:
279	kfree(ring->queues);	337	kfree(ring->queues);
280	kfree(ring);	338	kfree(ring);
281	return res;	339	return NULL;
		340	}
		341
		342	void fuse_uring_conn_init(struct fuse_chan *fch)
		343	{
		344	if (fuse_uring_create(fch))
		345	fch->io_uring = 1;
282	}	346	}
283		347
284	static struct fuse_ring_queue fuse_uring_create_queue(struct fuse_ring ring,	348	static struct fuse_ring_queue fuse_uring_create_queue(struct fuse_ring ring,
285	int qid)	349	int qid, bool zero_copy,
		350	bool fail_if_exists)
286	{	351	{
287	struct fuse_chan *fch = ring->chan;	352	struct fuse_chan *fch = ring->chan;
288	struct fuse_ring_queue *queue;	353	struct fuse_ring_queue *queue;
@@ -290,16 +355,17 @@ static struct fuse_ring_queue fuse_uring_create_queue(struct fuse_ring ring, · 1 unmodified line
290		355
291	queue = kzalloc_obj(*queue, GFP_KERNEL_ACCOUNT);	356	queue = kzalloc_obj(*queue, GFP_KERNEL_ACCOUNT);
292	if (!queue)	357	if (!queue)
293	return NULL;	358	return ERR_PTR(-ENOMEM);
294	pq = fuse_pqueue_alloc();	359	pq = fuse_pqueue_alloc();
295	if (!pq) {	360	if (!pq) {
296	kfree(queue);	361	kfree(queue);
297	return NULL;	362	return ERR_PTR(-ENOMEM);
298	}	363	}
299		364
300	queue->qid = qid;	365	queue->qid = qid;
301	queue->ring = ring;	366	queue->ring = ring;
302	spin_lock_init(&queue->lock);	367	spin_lock_init(&queue->lock);
		368	queue->zero_copy = zero_copy;
303		369
304	INIT_LIST_HEAD(&queue->ent_avail_queue);	370	INIT_LIST_HEAD(&queue->ent_avail_queue);
305	INIT_LIST_HEAD(&queue->ent_commit_queue);	371	INIT_LIST_HEAD(&queue->ent_commit_queue);
@@ -316,14 +382,17 @@ static struct fuse_ring_queue fuse_uring_create_queue(struct fuse_ring ring, · 10 unmodified lines
316	if (ring->queues[qid]) {	382	if (ring->queues[qid]) {
317	spin_unlock(&fch->lock);	383	spin_unlock(&fch->lock);
318	kfree(queue->fpq.processing);	384	kfree(queue->fpq.processing);
		385	kfree(queue->bufpool);
319	kfree(queue);	386	kfree(queue);
320	return ring->queues[qid];	387	return fail_if_exists ? ERR_PTR(-EEXIST) : ring->queues[qid];
321	}	388	}
322		389
323	/*	390	/*
324	* write_once and lock as the caller mostly doesn't take the lock at all	391	* fch->lock serializes concurrent creators for this qid.
		392	* smp_store_release() are for the lockless readers who must see a
		393	* fully initialized queue after &ring->queues[qid] is set
325	*/	394	*/
326	WRITE_ONCE(ring->queues[qid], queue);	395	smp_store_release(&ring->queues[qid], queue);
327	spin_unlock(&fch->lock);	396	spin_unlock(&fch->lock);
328		397
329	return queue;	398	return queue;
@@ -434,7 +503,7 @@ static void fuse_uring_log_ent_state(struct fuse_ring *ring) · 104 unmodified lines
434	struct fuse_ring_ent *ent;	503	struct fuse_ring_ent *ent;
435		504
436	for (qid = 0; qid < ring->nr_queues; qid++) {	505	for (qid = 0; qid < ring->nr_queues; qid++) {
437	struct fuse_ring_queue *queue = ring->queues[qid];	506	struct fuse_ring_queue *queue = READ_ONCE(ring->queues[qid]);
438		507
439	if (!queue)	508	if (!queue)
440	continue;	509	continue;
@@ -643,30 +712,57 @@ static int copy_header_from_ring(struct fuse_ring_ent *ent, · 202 unmodified lines
643	return 0;	712	return 0;
644	}	713	}
645		714
		715	static int fuse_uring_import_payload(struct fuse_ring_ent *ent, int dir,
		716	struct iov_iter *iter,
		717	unsigned int issue_flags)
		718	{
		719	void __user *base = ent->payload.iov_base;
		720	size_t len = ent->payload.iov_len;
		721	int err = 0;
		722
		723	if (!base) {
		724	memset(iter, 0, sizeof(*iter));
		725	return 0;
		726	}
		727
		728	if (bufpool_registered(ent->queue))
		729	err = io_uring_cmd_import_fixed((u64)(uintptr_t)base, len, dir,
		730	iter, ent->cmd, issue_flags);
		731	else
		732	err = import_ubuf(dir, base, len, iter);
		733
		734	if (err)
		735	pr_info_ratelimited("fuse: Import of user buffer failed\n");
		736
		737	return err;
		738	}
		739
646	static int setup_fuse_copy_state(struct fuse_copy_state *cs,	740	static int setup_fuse_copy_state(struct fuse_copy_state *cs,
647	struct fuse_ring ring, struct fuse_req req,	741	struct fuse_req *req,
648	struct fuse_ring_ent *ent, int dir,	742	struct fuse_ring_ent *ent, int dir,
649	struct iov_iter *iter)	743	struct iov_iter *iter,
		744	unsigned int issue_flags)
650	{	745	{
651	int err;	746	int err;
652		747
653	err = import_ubuf(dir, ent->payload, ring->max_payload_sz, iter);	748	err = fuse_uring_import_payload(ent, dir, iter, issue_flags);
654	if (err) {	749	if (err)
655	pr_info_ratelimited("fuse: Import of user buffer failed\n");
656	return err;	750	return err;
657	}
658		751
659	fuse_copy_init(cs, dir == ITER_DEST, iter);	752	fuse_copy_init(cs, dir == ITER_DEST, iter);
660		753
		754	if (ent->zero_copied)
		755	cs->skip_folio_copy = true;
		756
661	cs->is_uring = true;	757	cs->is_uring = true;
662	cs->req = req;	758	cs->req = req;
663		759
664	return 0;	760	return 0;
665	}	761	}
666		762
667	static int fuse_uring_copy_from_ring(struct fuse_ring *ring,	763	static int fuse_uring_copy_from_ring(struct fuse_req *req,
668	struct fuse_req *req,	764	struct fuse_ring_ent *ent,
669	struct fuse_ring_ent *ent)	765	unsigned int issue_flags)
670	{	766	{
671	struct fuse_copy_state cs;	767	struct fuse_copy_state cs;
672	struct fuse_args *args = req->args;	768	struct fuse_args *args = req->args;
@@ -679,7 +775,8 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring, · 6 unmodified lines
679	if (err)	775	if (err)
680	return err;	776	return err;
681		777
682	err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_SOURCE, &iter);	778	err = setup_fuse_copy_state(&cs, req, ent, ITER_SOURCE, &iter,
		779	issue_flags);
683	if (err)	780	if (err)
684	return err;	781	return err;
685		782
@@ -688,11 +785,68 @@ static int fuse_uring_copy_from_ring(struct fuse_ring *ring, · 2 unmodified lines
688	return err;	785	return err;
689	}	786	}
690		787
		788	static void fuse_zero_copy_release(void *priv)
		789	{
		790	struct fuse_zero_copy_bvs *zc_bvs = priv;
		791	unsigned int i;
		792
		793	for (i = 0; i < zc_bvs->nr_bvs; i++)
		794	folio_put(page_folio(zc_bvs->bvs[i].bv_page));
		795
		796	kvfree(zc_bvs);
		797	}
		798
		799	static int fuse_uring_set_up_zero_copy(struct fuse_ring_ent *ent,
		800	struct fuse_req *req,
		801	unsigned int issue_flags)
		802	{
		803	struct fuse_args_pages *ap;
		804	int err, i, ddir = 0;
		805	struct fuse_zero_copy_bvs *zc_bvs;
		806	struct bio_vec *bvs;
		807
		808	/* out_pages indicates a read, in_pages indicates a write */
		809	if (req->args->out_pages)
		810	ddir \|= IO_BUF_DEST;
		811	if (req->args->in_pages)
		812	ddir \|= IO_BUF_SOURCE;
		813
		814	ap = container_of(req->args, typeof(*ap), args);
		815
		816	zc_bvs = kvmalloc_flex(*zc_bvs, bvs, ap->num_folios,
		817	GFP_KERNEL_ACCOUNT);
		818	if (!zc_bvs)
		819	return -ENOMEM;
		820
		821	zc_bvs->nr_bvs = ap->num_folios;
		822	bvs = zc_bvs->bvs;
		823	for (i = 0; i < ap->num_folios; i++) {
		824	bvs[i].bv_page = folio_page(ap->folios[i], 0);
		825	bvs[i].bv_offset = ap->descs[i].offset;
		826	bvs[i].bv_len = ap->descs[i].length;
		827	folio_get(ap->folios[i]);
		828	}
		829
		830	err = io_buffer_register_bvec(ent->cmd, bvs, ap->num_folios,
		831	fuse_zero_copy_release, zc_bvs,
		832	ddir, ent->zero_copy_index,
		833	issue_flags);
		834	if (err) {
		835	fuse_zero_copy_release(zc_bvs);
		836	return err;
		837	}
		838
		839	ent->zero_copied = true;
		840
		841	return 0;
		842	}
		843
691	/*	844	/*
692	* Copy data from the req to the ring buffer	845	* Copy data from the req to the ring buffer
693	*/	846	*/
694	static int fuse_uring_args_to_ring(struct fuse_ring ring, struct fuse_req req,	847	static int fuse_uring_args_to_ring(struct fuse_req *req,
695	struct fuse_ring_ent *ent)	848	struct fuse_ring_ent *ent,
		849	unsigned int issue_flags)
696	{	850	{
697	struct fuse_copy_state cs;	851	struct fuse_copy_state cs;
698	struct fuse_args *args = req->args;	852	struct fuse_args *args = req->args;
@@ -705,7 +859,15 @@ static int fuse_uring_args_to_ring(struct fuse_ring ring, struct fuse_req req, · 6 unmodified lines
705	.commit_id = req->in.h.unique,	859	.commit_id = req->in.h.unique,
706	};	860	};
707		861
708	err = setup_fuse_copy_state(&cs, ring, req, ent, ITER_DEST, &iter);	862	if (can_zero_copy_req(ent, req)) {
		863	ent_in_out.flags \|= FUSE_URING_ENT_ZERO_COPY;
		864	err = fuse_uring_set_up_zero_copy(ent, req, issue_flags);
		865	if (err)
		866	return err;
		867	}
		868
		869	err = setup_fuse_copy_state(&cs, req, ent, ITER_DEST, &iter,
		870	issue_flags);
709	if (err)	871	if (err)
710	return err;	872	return err;
711		873
@@ -735,15 +897,32 @@ static int fuse_uring_args_to_ring(struct fuse_ring ring, struct fuse_req req, · 23 unmodified lines
735	}	897	}
736		898
737	ent_in_out.payload_sz = cs.ring.copied_sz;	899	ent_in_out.payload_sz = cs.ring.copied_sz;
		900	/*
		901	* on a zero-copied write the pages are registered for the server to
		902	* read via a fixed-buffer op rather than copied into the payload
		903	* buffer, so copied_sz does not account for it. The server still needs
		904	* the total inbound size to know how many bytes to read from the
		905	* registered buffer, so add the page arg (always the last in-arg) back
		906	* in
		907	*/
		908	if (cs.skip_folio_copy && args->in_pages)
		909	ent_in_out.payload_sz +=
		910	args->in_args[args->in_numargs - 1].size;
		911
		912	if (bufpool_enabled(ent->queue) && ent->payload.iov_base)
		913	ent_in_out.offset =
		914	(uintptr_t)ent->payload.iov_base - ent->queue->bufpool->base_uaddr;
		915
738	return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,	916	return copy_header_to_ring(ent, FUSE_URING_HEADER_RING_ENT,
739	&ent_in_out, sizeof(ent_in_out));	917	&ent_in_out, sizeof(ent_in_out));
740	}	918	}
741		919
742	static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,	920	static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent,
743	struct fuse_req *req)	921	struct fuse_req *req,
		922	unsigned int issue_flags)
744	{	923	{
745	struct fuse_ring_queue *queue = ent->queue;	924	struct fuse_ring_queue *queue = ent->queue;
746	struct fuse_ring *ring = queue->ring;	925	struct fuse_in_header in_header;
747	int err;	926	int err;
748		927
749	err = -EIO;	928	err = -EIO;
@@ -758,23 +937,124 @@ static int fuse_uring_copy_to_ring(struct fuse_ring_ent *ent, · 8 unmodified lines
758	return err;	937	return err;
759		938
760	/* copy the request */	939	/* copy the request */
761	err = fuse_uring_args_to_ring(ring, req, ent);	940	err = fuse_uring_args_to_ring(req, ent, issue_flags);
762	if (unlikely(err)) {	941	if (unlikely(err)) {
763	pr_info_ratelimited("Copy to ring failed: %d\n", err);	942	pr_info_ratelimited("Copy to ring failed: %d\n", err);
764	return err;	943	return err;
765	}	944	}
766		945
767	/* copy fuse_in_header */	946	/* copy fuse_in_header */
768	return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->in.h,	947	in_header = req->in.h;
769	sizeof(req->in.h));	948	return copy_header_to_ring(ent, FUSE_URING_HEADER_IN_OUT, &in_header,
		949	sizeof(in_header));
		950	}
		951
		952	static bool fuse_uring_req_has_copyable_payload(struct fuse_ring_ent *ent,
		953	struct fuse_req *req)
		954	{
		955	struct fuse_args *args = req->args;
		956
		957	if (!can_zero_copy_req(ent, req))
		958	return args->in_numargs > 1 \|\| args->out_numargs;
		959
		960	/*
		961	* the asymmetry between in_numargs > 2 and out_numargs > 1 is because
		962	* the per-op header is extracted before fuse_copy_args() for inargs but
		963	* not for outargs
		964	*/
		965	if ((args->in_numargs > 1) && (!args->in_pages \|\| args->in_numargs > 2))
		966	return true;
		967	if (args->out_numargs && (!args->out_pages \|\| args->out_numargs > 1))
		968	return true;
		969
		970	return false;
		971	}
		972
		973	static int fuse_uring_select_buffer(struct fuse_ring_ent *ent)
		974	{
		975	struct fuse_ring_queue *queue = ent->queue;
		976	struct fuse_bufpool *pool = queue->bufpool;
		977	unsigned int id;
		978
		979	lockdep_assert_held(&queue->lock);
		980
		981	id = find_first_bit(pool->free_map, pool->nr_bufs);
		982	if (id >= pool->nr_bufs)
		983	return -ENOBUFS;
		984
		985	WARN_ON_ONCE(ent->payload.iov_base);
		986	__clear_bit(id, pool->free_map);
		987
		988	ent->buf_id = id;
		989	ent->payload.iov_base =
		990	(void __user )(pool->base_uaddr + id pool->buf_size);
		991	ent->payload.iov_len = pool->buf_size;
		992
		993	return 0;
		994	}
		995
		996	static void fuse_uring_recycle_buffer(struct fuse_ring_ent *ent)
		997	{
		998	struct iovec *ent_payload = &ent->payload;
		999	struct fuse_ring_queue *queue = ent->queue;
		1000	struct fuse_bufpool *pool;
		1001
		1002	lockdep_assert_held(&queue->lock);
		1003
		1004	if (!bufpool_enabled(queue) \|\| !ent_payload->iov_base)
		1005	return;
		1006
		1007	pool = queue->bufpool;
		1008
		1009	/* a buffer should never be recycled twice */
		1010	WARN_ON_ONCE(test_bit(ent->buf_id, pool->free_map));
		1011	__set_bit(ent->buf_id, pool->free_map);
		1012
		1013	memset(ent_payload, 0, sizeof(*ent_payload));
		1014	ent->buf_id = 0;
		1015	}
		1016
		1017	static int fuse_uring_next_req_update_buffer(struct fuse_ring_ent *ent,
		1018	struct fuse_req *req)
		1019	{
		1020	bool buffer_selected;
		1021	bool has_payload;
		1022
		1023	if (!bufpool_enabled(ent->queue))
		1024	return 0;
		1025
		1026	buffer_selected = !!ent->payload.iov_base;
		1027	has_payload = fuse_uring_req_has_copyable_payload(ent, req);
		1028
		1029	if (has_payload && !buffer_selected)
		1030	return fuse_uring_select_buffer(ent);
		1031
		1032	if (!has_payload && buffer_selected)
		1033	fuse_uring_recycle_buffer(ent);
		1034
		1035	return 0;
		1036	}
		1037
		1038	static int fuse_uring_prep_buffer(struct fuse_ring_ent *ent,
		1039	struct fuse_req *req)
		1040	{
		1041	if (!bufpool_enabled(ent->queue))
		1042	return 0;
		1043
		1044	/* no payload to copy, can skip selecting a buffer */
		1045	if (!fuse_uring_req_has_copyable_payload(ent, req))
		1046	return 0;
		1047
		1048	return fuse_uring_select_buffer(ent);
770	}	1049	}
771		1050
772	static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,	1051	static int fuse_uring_prepare_send(struct fuse_ring_ent *ent,
773	struct fuse_req *req)	1052	struct fuse_req *req,
		1053	unsigned int issue_flags)
774	{	1054	{
775	int err;	1055	int err;
776		1056
777	err = fuse_uring_copy_to_ring(ent, req);	1057	err = fuse_uring_copy_to_ring(ent, req, issue_flags);
778	if (!err) {	1058	if (!err) {
779	set_bit(FR_SENT, &req->flags);	1059	set_bit(FR_SENT, &req->flags);
780	trace_fuse_request_sent(req);	1060	trace_fuse_request_sent(req);
@@ -788,7 +1068,7 @@ static int fuse_uring_prepare_send(struct fuse_ring_ent *ent, · 7 unmodified lines
788	ent->state = FRRS_INVALID;	1068	ent->state = FRRS_INVALID;
789	spin_unlock(&ent->queue->lock);	1069	spin_unlock(&ent->queue->lock);
790		1070
791	fuse_uring_req_end(ent, req, err);	1071	fuse_uring_req_end(ent, req, err, issue_flags);
792	}	1072	}
793		1073
794	return err;	1074	return err;
@@ -856,9 +1136,12 @@ static struct fuse_req fuse_uring_ent_assign_req(struct fuse_ring_ent ent) · 61 unmodified lines
856		1136
857	/* get and assign the next entry while it is still holding the lock */	1137	/* get and assign the next entry while it is still holding the lock */
858	req = list_first_entry_or_null(req_queue, struct fuse_req, list);	1138	req = list_first_entry_or_null(req_queue, struct fuse_req, list);
859	if (req)	1139	if (!req \|\| fuse_uring_next_req_update_buffer(ent, req)) {
860	fuse_uring_add_req_to_ring_ent(ent, req);	1140	fuse_uring_recycle_buffer(ent);
		1141	return NULL;
		1142	}
861		1143
		1144	fuse_uring_add_req_to_ring_ent(ent, req);
862	return req;	1145	return req;
863	}	1146	}
864		1147
@@ -870,12 +1153,13 @@ static struct fuse_req fuse_uring_ent_assign_req(struct fuse_ring_ent ent) · 5 unmodified lines
870	static void fuse_uring_commit(struct fuse_ring_ent ent, struct fuse_req req,	1153	static void fuse_uring_commit(struct fuse_ring_ent ent, struct fuse_req req,
871	unsigned int issue_flags)	1154	unsigned int issue_flags)
872	{	1155	{
873	struct fuse_ring *ring = ent->queue->ring;	1156	struct fuse_out_header out_header;
874	ssize_t err = -EFAULT;	1157	ssize_t err = -EFAULT;
875		1158
876	if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &req->out.h,	1159	if (copy_header_from_ring(ent, FUSE_URING_HEADER_IN_OUT, &out_header,
877	sizeof(req->out.h)))	1160	sizeof(out_header)))
878	goto out;	1161	goto out;
		1162	req->out.h = out_header;
879		1163
880	err = fuse_uring_out_header_has_err(&req->out.h, req);	1164	err = fuse_uring_out_header_has_err(&req->out.h, req);
881	if (err) {	1165	if (err) {
@@ -883,9 +1167,9 @@ static void fuse_uring_commit(struct fuse_ring_ent ent, struct fuse_req req, · 1 unmodified line
883	goto out;	1167	goto out;
884	}	1168	}
885		1169
886	err = fuse_uring_copy_from_ring(ring, req, ent);	1170	err = fuse_uring_copy_from_ring(req, ent, issue_flags);
887	out:	1171	out:
888	fuse_uring_req_end(ent, req, err);	1172	fuse_uring_req_end(ent, req, err, issue_flags);
889	}	1173	}
890		1174
891	/*	1175	/*
@@ -895,7 +1179,8 @@ out: · 3 unmodified lines
895	* Else, there is no next fuse request and this returns false.	1179	* Else, there is no next fuse request and this returns false.
896	*/	1180	*/
897	static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,	1181	static bool fuse_uring_get_next_fuse_req(struct fuse_ring_ent *ent,
898	struct fuse_ring_queue *queue)	1182	struct fuse_ring_queue *queue,
		1183	unsigned int issue_flags)
899	{	1184	{
900	int err;	1185	int err;
901	struct fuse_req *req;	1186	struct fuse_req *req;
@@ -907,7 +1192,7 @@ retry: · 5 unmodified lines
907	spin_unlock(&queue->lock);	1192	spin_unlock(&queue->lock);
908		1193
909	if (req) {	1194	if (req) {
910	err = fuse_uring_prepare_send(ent, req);	1195	err = fuse_uring_prepare_send(ent, req, issue_flags);
911	if (err)	1196	if (err)
912	goto retry;	1197	goto retry;
913	}	1198	}
@@ -967,7 +1252,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · 53 unmodified lines
967	if (qid >= ring->nr_queues)	1252	if (qid >= ring->nr_queues)
968	return -EINVAL;	1253	return -EINVAL;
969		1254
970	queue = ring->queues[qid];	1255	queue = READ_ONCE(ring->queues[qid]);
971	if (!queue)	1256	if (!queue)
972	return err;	1257	return err;
973	fpq = &queue->fpq;	1258	fpq = &queue->fpq;
@@ -981,6 +1266,11 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · 7 unmodified lines
981	return err;	1266	return err;
982	}	1267	}
983		1268
		1269	if (!fuse_uring_cmd_index_ok(cmd, queue)) {
		1270	spin_unlock(&queue->lock);
		1271	return -EINVAL;
		1272	}
		1273
984	/* Find a request based on the unique ID of the fuse request	1274	/* Find a request based on the unique ID of the fuse request
985	* This should get revised, as it needs a hash calculation and list	1275	* This should get revised, as it needs a hash calculation and list
986	* search. And full struct fuse_pqueue is needed (memory overhead).	1276	* search. And full struct fuse_pqueue is needed (memory overhead).
@@ -1002,8 +1292,14 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · 15 unmodified lines
1002	if (err != 0) {	1292	if (err != 0) {
1003	pr_info_ratelimited("qid=%d commit_id %llu state %d",	1293	pr_info_ratelimited("qid=%d commit_id %llu state %d",
1004	queue->qid, commit_id, ent->state);	1294	queue->qid, commit_id, ent->state);
		1295	fuse_uring_recycle_buffer(ent);
1005	spin_unlock(&queue->lock);	1296	spin_unlock(&queue->lock);
1006	fuse_uring_req_end(ent, req, err);	1297	/*
		1298	* Unregister any zero copyable pages since ent->cmd is null
		1299	* when it hits fuse_uring_req_end() in this path
		1300	*/
		1301	zero_copy_unregister(cmd, ent, issue_flags);
		1302	fuse_uring_req_end(ent, req, err, issue_flags);
1007	return err;	1303	return err;
1008	}	1304	}
1009		1305
@@ -1019,8 +1315,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags, · 9 unmodified lines
1019	* fuse requests would otherwise not get processed - committing	1315	* fuse requests would otherwise not get processed - committing
1020	* and fetching is done in one step vs legacy fuse, which has separated	1316	* and fetching is done in one step vs legacy fuse, which has separated
1021	* read (fetch request) and write (commit result).	1317	* read (fetch request) and write (commit result).
		1318	*
		1319	* If there is no next request or if all buffers are busy (if using a
		1320	* bufpool), the cmd is not returned to userspace. The entry is left
		1321	* available and the cmd only returns to userspace when there's a
		1322	* next request and an available buffer.
1022	*/	1323	*/
1023	if (fuse_uring_get_next_fuse_req(ent, queue))	1324	if (fuse_uring_get_next_fuse_req(ent, queue, issue_flags))
1024	fuse_uring_send(ent, cmd, 0, issue_flags);	1325	fuse_uring_send(ent, cmd, 0, issue_flags);
1025	return 0;	1326	return 0;
1026	}	1327	}
@@ -1035,7 +1336,7 @@ static bool is_ring_ready(struct fuse_ring *ring, int current_qid) · 8 unmodified lines
1035	if (current_qid == qid)	1336	if (current_qid == qid)
1036	continue;	1337	continue;
1037		1338
1038	queue = ring->queues[qid];	1339	queue = READ_ONCE(ring->queues[qid]);
1039	if (!queue) {	1340	if (!queue) {
1040	ready = false;	1341	ready = false;
1041	break;	1342	break;
@@ -1122,10 +1423,14 @@ static struct fuse_ring_ent * · 80 unmodified lines
1122	fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,	1423	fuse_uring_create_ring_ent(struct io_uring_cmd *cmd,
1123	struct fuse_ring_queue *queue)	1424	struct fuse_ring_queue *queue)
1124	{	1425	{
		1426	const struct fuse_uring_cmd_req *cmd_req =
		1427	io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
1125	struct fuse_ring *ring = queue->ring;	1428	struct fuse_ring *ring = queue->ring;
1126	struct fuse_ring_ent *ent;	1429	struct fuse_ring_ent *ent;
1127	struct iovec iov[FUSE_URING_IOV_SEGS];	1430	struct iovec iov[FUSE_URING_IOV_SEGS];
1128	struct iovec headers, payload;	1431	struct iovec headers, payload;
		1432	unsigned int zero_copy_index;
		1433
1129	int err;	1434	int err;
1130		1435
1131	err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);	1436	err = fuse_uring_get_iovec_from_sqe(cmd->sqe, iov);
@@ -1135,6 +1440,10 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · 3 unmodified lines
1135	return ERR_PTR(err);	1440	return ERR_PTR(err);
1136	}	1441	}
1137		1442
		1443	zero_copy_index = READ_ONCE(cmd_req->ent_zero_copy_buf_index);
		1444	if (zero_copy_index && !queue->zero_copy)
		1445	return ERR_PTR(-EINVAL);
		1446
1138	err = -EINVAL;	1447	err = -EINVAL;
1139	headers = &iov[FUSE_URING_IOV_HEADERS];	1448	headers = &iov[FUSE_URING_IOV_HEADERS];
1140	if (headers->iov_len < sizeof(struct fuse_uring_req_header)) {	1449	if (headers->iov_len < sizeof(struct fuse_uring_req_header)) {
@@ -1143,11 +1452,29 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · 2 unmodified lines
1143	}	1452	}
1144		1453
1145	payload = &iov[FUSE_URING_IOV_PAYLOAD];	1454	payload = &iov[FUSE_URING_IOV_PAYLOAD];
1146	if (payload->iov_len < ring->max_payload_sz) {	1455
1147	pr_info_ratelimited("Invalid req payload len %zu\n",	1456	spin_lock(&queue->lock);
1148	payload->iov_len);	1457	if (bufpool_enabled(queue)) {
1149	return ERR_PTR(err);	1458	if (payload->iov_base \|\| payload->iov_len \|\|
		1459	!fuse_uring_cmd_index_ok(cmd, queue)) {
		1460	spin_unlock(&queue->lock);
		1461	return ERR_PTR(err);
		1462	}
		1463	} else {
		1464	if (payload->iov_len < ring->max_payload_sz) {
		1465	spin_unlock(&queue->lock);
		1466	pr_info_ratelimited("Invalid req payload len %zu\n",
		1467	payload->iov_len);
		1468	return ERR_PTR(err);
		1469	}
		1470	if (queue->zero_copy) {
		1471	spin_unlock(&queue->lock);
		1472	pr_info_ratelimited("Can only use zero copy with bufpools\n");
		1473	return ERR_PTR(err);
		1474	}
		1475	queue->payload_mode = FUSE_PAYLOAD_PER_ENT;
1150	}	1476	}
		1477	spin_unlock(&queue->lock);
1151		1478
1152	err = -ENOMEM;	1479	err = -ENOMEM;
1153	ent = kzalloc_obj(*ent, GFP_KERNEL_ACCOUNT);	1480	ent = kzalloc_obj(*ent, GFP_KERNEL_ACCOUNT);
@@ -1158,7 +1485,9 @@ fuse_uring_create_ring_ent(struct io_uring_cmd *cmd, · 4 unmodified lines
1158		1485
1159	ent->queue = queue;	1486	ent->queue = queue;
1160	ent->headers = headers->iov_base;	1487	ent->headers = headers->iov_base;
1161	ent->payload = payload->iov_base;	1488	if (queue->payload_mode == FUSE_PAYLOAD_PER_ENT)
		1489	ent->payload = *payload;
		1490	ent->zero_copy_index = zero_copy_index;
1162		1491
1163	atomic_inc(&ring->queue_refs);	1492	atomic_inc(&ring->queue_refs);
1164	return ent;	1493	return ent;
@@ -1176,26 +1505,21 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, · 11 unmodified lines
1176	struct fuse_ring *ring = smp_load_acquire(&fch->ring);	1505	struct fuse_ring *ring = smp_load_acquire(&fch->ring);
1177	struct fuse_ring_queue *queue;	1506	struct fuse_ring_queue *queue;
1178	struct fuse_ring_ent *ent;	1507	struct fuse_ring_ent *ent;
1179	int err;
1180	unsigned int qid = READ_ONCE(cmd_req->qid);	1508	unsigned int qid = READ_ONCE(cmd_req->qid);
1181		1509
1182	err = -ENOMEM;	1510	if (!ring)
1183	if (!ring) {	1511	return -EINVAL;
1184	ring = fuse_uring_create(fch);
1185	if (!ring)
1186	return err;
1187	}
1188		1512
1189	if (qid >= ring->nr_queues) {	1513	if (qid >= ring->nr_queues) {
1190	pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);	1514	pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);
1191	return -EINVAL;	1515	return -EINVAL;
1192	}	1516	}
1193		1517
1194	queue = ring->queues[qid];	1518	queue = READ_ONCE(ring->queues[qid]);
1195	if (!queue) {	1519	if (!queue) {
1196	queue = fuse_uring_create_queue(ring, qid);	1520	queue = fuse_uring_create_queue(ring, qid, false, false);
1197	if (!queue)	1521	if (IS_ERR(queue))
1198	return err;	1522	return PTR_ERR(queue);
1199	}	1523	}
1200		1524
1201	/*	1525	/*
@@ -1210,6 +1534,110 @@ static int fuse_uring_register(struct io_uring_cmd *cmd, · 8 unmodified lines
1210	return fuse_uring_do_register(ent, cmd, issue_flags);	1534	return fuse_uring_do_register(ent, cmd, issue_flags);
1211	}	1535	}
1212		1536
		1537	static int fuse_uring_add_queue(struct io_uring_cmd cmd, struct fuse_chan fch)
		1538	{
		1539	const struct fuse_uring_cmd_req *cmd_req =
		1540	io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
		1541	struct fuse_ring *ring = smp_load_acquire(&fch->ring);
		1542	unsigned int qid = READ_ONCE(cmd_req->qid);
		1543	uint64_t flags = READ_ONCE(cmd_req->flags);
		1544	struct fuse_ring_queue *queue;
		1545	bool zero_copy = flags & FUSE_URING_ZERO_COPY;
		1546
		1547	if (!ring)
		1548	return -EINVAL;
		1549
		1550	if (qid >= ring->nr_queues) {
		1551	pr_info_ratelimited("fuse: Invalid ring qid %u\n", qid);
		1552	return -EINVAL;
		1553	}
		1554
		1555	if (flags & ~FUSE_URING_ADD_QUEUE_FLAGS)
		1556	return -EINVAL;
		1557
		1558	if (zero_copy && !capable(CAP_SYS_ADMIN))
		1559	return -EPERM;
		1560
		1561	queue = fuse_uring_create_queue(ring, qid, zero_copy, true);
		1562	if (IS_ERR(queue))
		1563	return PTR_ERR(queue);
		1564
		1565	return 0;
		1566	}
		1567
		1568	static int fuse_uring_add_bufpool(struct io_uring_cmd *cmd,
		1569	struct fuse_chan *fch)
		1570	{
		1571	const struct fuse_uring_cmd_req *cmd_req =
		1572	io_uring_sqe128_cmd(cmd->sqe, struct fuse_uring_cmd_req);
		1573	unsigned int qid = READ_ONCE(cmd_req->qid);
		1574	uint64_t flags = READ_ONCE(cmd_req->flags);
		1575	/* paired with the smp_store_release() in fuse_uring_create */
		1576	struct fuse_ring *ring = smp_load_acquire(&fch->ring);
		1577	struct fuse_ring_queue *queue;
		1578	struct fuse_bufpool *pool;
		1579	uintptr_t pool_uaddr;
		1580	unsigned int pool_len, nr_bufs;
		1581	size_t pool_size, buf_size;
		1582	bool registered = cmd->flags & IORING_URING_CMD_FIXED;
		1583
		1584	if (!ring \|\| qid >= ring->nr_queues \|\| flags)
		1585	return -EINVAL;
		1586
		1587	/* reserved for future use, must be zero */
		1588	if (READ_ONCE(cmd_req->bufpool.reserved))
		1589	return -EINVAL;
		1590
		1591	/* Pairs with smp_store_release() in fuse_uring_create_queue() */
		1592	queue = smp_load_acquire(&ring->queues[qid]);
		1593	if (!queue)
		1594	return -EINVAL;
		1595
		1596	pool_uaddr = READ_ONCE(cmd_req->bufpool.uaddr);
		1597	pool_len = READ_ONCE(cmd_req->bufpool.len);
		1598
		1599	/* each buffer holds the max payload size */
		1600	buf_size = queue->ring->max_payload_sz;
		1601
		1602	nr_bufs = pool_len / buf_size;
		1603	if (!nr_bufs)
		1604	return -EINVAL;
		1605
		1606	pool_size = struct_size(pool, free_map, BITS_TO_LONGS(nr_bufs));
		1607	pool = kzalloc(pool_size, GFP_KERNEL_ACCOUNT);
		1608	if (!pool)
		1609	return -ENOMEM;
		1610
		1611	pool->base_uaddr = pool_uaddr;
		1612	pool->buf_size = buf_size;
		1613	pool->nr_bufs = nr_bufs;
		1614	/* all buffers are free */
		1615	bitmap_set(pool->free_map, 0, nr_bufs);
		1616
		1617	/*
		1618	* A registered bufpool is reached through an io_uring fixed buffer, so
		1619	* the pool is registered iff this command was submitted with
		1620	* IORING_URING_CMD_FIXED. The registered buffer index is taken from
		1621	* sqe->buf_index.
		1622	*/
		1623	if (registered) {
		1624	pool->registered = true;
		1625	pool->registered_index = READ_ONCE(cmd->sqe->buf_index);
		1626	}
		1627
		1628	spin_lock(&queue->lock);
		1629	if (queue->payload_mode != FUSE_PAYLOAD_UNSET) {
		1630	spin_unlock(&queue->lock);
		1631	kfree(pool);
		1632	return -EINVAL;
		1633	}
		1634	queue->bufpool = pool;
		1635	queue->payload_mode = FUSE_PAYLOAD_BUFPOOL;
		1636	spin_unlock(&queue->lock);
		1637
		1638	return 0;
		1639	}
		1640
1213	/*	1641	/*
1214	* Entry function from io_uring to handle the given passthrough command	1642	* Entry function from io_uring to handle the given passthrough command
1215	* (op code IORING_OP_URING_CMD)	1643	* (op code IORING_OP_URING_CMD)
@@ -1237,23 +1665,30 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) · 21 unmodified lines
1237	}	1665	}
1238	fch = fud->chan;	1666	fch = fud->chan;
1239		1667
1240	/* Once a connection has io-uring enabled on it, it can't be disabled */	1668	/*
1241	if (!enable_uring && !fch->io_uring) {	1669	* The ring is sized from values negotiated by FUSE_INIT
1242	pr_info_ratelimited("fuse-io-uring is disabled\n");	1670	*
1243	return -EOPNOTSUPP;	1671	* Pairs with smp_store_release() in fuse_chan_set_initialized()
1244	}	1672	*/
		1673	if (!smp_load_acquire(&fch->initialized))
		1674	return -EAGAIN;
1245		1675
1246	if (fch->abort_with_err)	1676	if (fch->abort_with_err)
1247	return -ECONNABORTED;	1677	return -ECONNABORTED;
1248	if (!fch->connected)	1678	if (!fch->connected)
1249	return -ENOTCONN;	1679	return -ENOTCONN;
1250		1680
1251	/*	1681	/* Once a connection has io-uring enabled on it, it can't be disabled */
1252	* fuse_uring_register() needs the ring to be initialized,	1682	if (!enable_uring && !fch->io_uring) {
1253	* we need to know the max payload size	1683	pr_info_ratelimited("fuse-io-uring is disabled by module parameter\n");
1254	*/	1684	return -EOPNOTSUPP;
1255	if (!fch->initialized)	1685	}
1256	return -EAGAIN;	1686
		1687	if (!fch->io_uring) {
		1688	pr_info_ratelimited(
		1689	"fuse-io-uring not enabled on this connection\n");
		1690	return -EOPNOTSUPP;
		1691	}
1257		1692
1258	switch (cmd_op) {	1693	switch (cmd_op) {
1259	case FUSE_IO_URING_CMD_REGISTER:	1694	case FUSE_IO_URING_CMD_REGISTER:
@@ -1274,6 +1709,18 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) · 14 unmodified lines
1274	return err;	1709	return err;
1275	}	1710	}
1276	break;	1711	break;
		1712	case FUSE_IO_URING_CMD_ADD_QUEUE:
		1713	err = fuse_uring_add_queue(cmd, fch);
		1714	if (err)
		1715	pr_info_once("FUSE_IO_URING_CMD_ADD_QUEUE failed err=%d\n",
		1716	err);
		1717	return err;
		1718	case FUSE_IO_URING_CMD_ADD_BUFPOOL:
		1719	err = fuse_uring_add_bufpool(cmd, fch);
		1720	if (err)
		1721	pr_info_once("FUSE_IO_URING_ADD_BUFPOOL failed err=%d\n",
		1722	err);
		1723	return err;
1277	default:	1724	default:
1278	return -EINVAL;	1725	return -EINVAL;
1279	}	1726	}
@@ -1295,9 +1742,10 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw) · 15 unmodified lines
1295	int err;	1742	int err;
1296		1743
1297	if (!tw.cancel) {	1744	if (!tw.cancel) {
1298	err = fuse_uring_prepare_send(ent, ent->fuse_req);	1745	err = fuse_uring_prepare_send(ent, ent->fuse_req, issue_flags);
1299	if (err) {	1746	if (err) {
1300	if (!fuse_uring_get_next_fuse_req(ent, queue))	1747	if (!fuse_uring_get_next_fuse_req(ent, queue,
		1748	issue_flags))
1301	return;	1749	return;
1302	err = 0;	1750	err = 0;
1303	}	1751	}
@@ -1307,11 +1755,12 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw) · 3 unmodified lines
1307		1755
1308	spin_lock(&queue->lock);	1756	spin_lock(&queue->lock);
1309	list_del_init(&ent->list);	1757	list_del_init(&ent->list);
		1758	fuse_uring_recycle_buffer(ent);
1310	spin_unlock(&queue->lock);	1759	spin_unlock(&queue->lock);
1311		1760
1312	io_uring_cmd_done(cmd, err, issue_flags);	1761	io_uring_cmd_done(cmd, err, issue_flags);
1313		1762
1314	fuse_uring_req_end(ent, ent->fuse_req, err);	1763	fuse_uring_req_end(ent, ent->fuse_req, err, issue_flags);
1315	kfree(ent);	1764	kfree(ent);
1316	if (atomic_dec_and_test(&queue->ring->queue_refs))	1765	if (atomic_dec_and_test(&queue->ring->queue_refs))
1317	wake_up_all(&queue->ring->stop_waitq);	1766	wake_up_all(&queue->ring->stop_waitq);
@@ -1330,7 +1779,7 @@ static struct fuse_ring_queue fuse_uring_task_to_queue(struct fuse_ring ring) · 12 unmodified lines
1330	ring->nr_queues))	1779	ring->nr_queues))
1331	qid = 0;	1780	qid = 0;
1332		1781
1333	queue = ring->queues[qid];	1782	queue = READ_ONCE(ring->queues[qid]);
1334	WARN_ONCE(!queue, "Missing queue for qid %d\n", qid);	1783	WARN_ONCE(!queue, "Missing queue for qid %d\n", qid);
1335		1784
1336	return queue;	1785	return queue;
@@ -1368,15 +1817,16 @@ void fuse_uring_queue_fuse_req(struct fuse_iqueue fiq, struct fuse_req req) · 31 unmodified lines
1368	req->ring_queue = queue;	1817	req->ring_queue = queue;
1369	ent = list_first_entry_or_null(&queue->ent_avail_queue,	1818	ent = list_first_entry_or_null(&queue->ent_avail_queue,
1370	struct fuse_ring_ent, list);	1819	struct fuse_ring_ent, list);
1371	if (ent)
1372	fuse_uring_add_req_to_ring_ent(ent, req);
1373	else
1374	list_add_tail(&req->list, &queue->fuse_req_queue);
1375	spin_unlock(&queue->lock);
1376		1820
1377	if (ent)	1821	if (!ent \|\| fuse_uring_prep_buffer(ent, req)) {
1378	fuse_uring_dispatch_ent(ent);	1822	list_add_tail(&req->list, &queue->fuse_req_queue);
		1823	spin_unlock(&queue->lock);
		1824	return;
		1825	}
1379		1826
		1827	fuse_uring_add_req_to_ring_ent(ent, req);
		1828	spin_unlock(&queue->lock);
		1829	fuse_uring_dispatch_ent(ent);
1380	return;	1830	return;
1381		1831
1382	err_unlock:	1832	err_unlock:
@@ -1424,10 +1874,9 @@ bool fuse_uring_queue_bq_req(struct fuse_req *req) · 41 unmodified lines
1424	*/	1874	*/
1425	req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req,	1875	req = list_first_entry_or_null(&queue->fuse_req_queue, struct fuse_req,
1426	list);	1876	list);
1427	if (ent && req) {	1877	if (ent && req && !fuse_uring_prep_buffer(ent, req)) {
1428	fuse_uring_add_req_to_ring_ent(ent, req);	1878	fuse_uring_add_req_to_ring_ent(ent, req);
1429	spin_unlock(&queue->lock);	1879	spin_unlock(&queue->lock);
1430
1431	fuse_uring_dispatch_ent(ent);	1880	fuse_uring_dispatch_ent(ent);
1432	} else {	1881	} else {
1433	spin_unlock(&queue->lock);	1882	spin_unlock(&queue->lock);

fs/fuse/dev_uring_i.h+55−1

@@ -7,6 +7,8 @@ · 6 unmodified lines
7	#ifndef _FS_FUSE_DEV_URING_I_H	7	#ifndef _FS_FUSE_DEV_URING_I_H
8	#define _FS_FUSE_DEV_URING_I_H	8	#define _FS_FUSE_DEV_URING_I_H
9		9
		10	#include <linux/uio.h>
		11
10	#include "fuse_dev_i.h"	12	#include "fuse_dev_i.h"
11		13
12	#ifdef CONFIG_FUSE_IO_URING	14	#ifdef CONFIG_FUSE_IO_URING
@@ -36,11 +38,50 @@ enum fuse_ring_req_state { · 23 unmodified lines
36	FRRS_RELEASED,	38	FRRS_RELEASED,
37	};	39	};
38		40
		41	/* how a queue's payload buffers are provided */
		42	enum fuse_queue_payload_mode {
		43	/* not yet committed (a bufpool may still be added) */
		44	FUSE_PAYLOAD_UNSET = 0,
		45	/* each entry registers its own payload buffer */
		46	FUSE_PAYLOAD_PER_ENT,
		47	/* each entry's payload buffer is assigned from a bufpool */
		48	FUSE_PAYLOAD_BUFPOOL,
		49	};
		50
		51	struct fuse_bufpool {
		52	bool registered;
		53
		54	/*
		55	* io_uring registered buffer table index for this pool, bound at
		56	* ADD_BUFPOOL time. Only valid if the bufpool is registered
		57	*/
		58	u16 registered_index;
		59
		60	/* starting uaddr of the bufpool */
		61	uintptr_t base_uaddr;
		62
		63	/* size of each buffer in the pool */
		64	size_t buf_size;
		65
		66	/* total number of buffers in the pool */
		67	unsigned int nr_bufs;
		68
		69	/* bitmap tracking which buffers are free */
		70	unsigned long free_map[];
		71	};
		72
39	/** A fuse ring entry, part of the ring queue */	73	/** A fuse ring entry, part of the ring queue */
40	struct fuse_ring_ent {	74	struct fuse_ring_ent {
41	/* userspace buffer */	75	/* userspace buffer */
42	struct fuse_uring_req_header __user *headers;	76	struct fuse_uring_req_header __user *headers;
43	void __user *payload;	77	struct iovec payload;
		78
		79	/* buffer id in the pool, if bufpools are used. ignored otherwise */
		80	unsigned int buf_id;
		81
		82	/* true if the request's pages are being zero-copied */
		83	bool zero_copied;
		84	unsigned int zero_copy_index;
44		85
45	/* the ring queue that owns the request */	86	/* the ring queue that owns the request */
46	struct fuse_ring_queue *queue;	87	struct fuse_ring_queue *queue;
@@ -99,6 +140,14 @@ struct fuse_ring_queue { · 52 unmodified lines
99	unsigned int active_background;	140	unsigned int active_background;
100		141
101	bool stopped;	142	bool stopped;
		143
		144	/* how this queue's payload buffers are provided */
		145	enum fuse_queue_payload_mode payload_mode;
		146
		147	/* only allocated when payload_mode == FUSE_PAYLOAD_BUFPOOL */
		148	struct fuse_bufpool *bufpool;
		149
		150	bool zero_copy;
102	};	151	};
103		152
104	/*	153	/*
@@ -135,6 +184,7 @@ struct fuse_ring { · 30 unmodified lines
135	bool ready;	184	bool ready;
136	};	185	};
137		186
		187	void fuse_uring_conn_init(struct fuse_chan *fch);
138	void fuse_uring_stop_queues(struct fuse_ring *ring);	188	void fuse_uring_stop_queues(struct fuse_ring *ring);
139	void fuse_uring_abort_end_requests(struct fuse_ring *ring);	189	void fuse_uring_abort_end_requests(struct fuse_ring *ring);
140	int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags);	190	int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags);
@@ -174,6 +224,10 @@ static inline bool fuse_uring_ready(struct fuse_chan *fch) · 33 unmodified lines
174		224
175	#else /* CONFIG_FUSE_IO_URING */	225	#else /* CONFIG_FUSE_IO_URING */
176		226
		227	static inline void fuse_uring_conn_init(struct fuse_chan *fch)
		228	{
		229	}
		230
177	static inline void fuse_uring_abort(struct fuse_chan *fch)	231	static inline void fuse_uring_abort(struct fuse_chan *fch)
178	{	232	{
179	}	233	}

fs/fuse/dir.c+24−13

 #define FUSE_HASH_BITS	5
 #define FUSE_HASH_SIZE	(1 << FUSE_HASH_BITS)
 static struct dentry_bucket dentry_hash[FUSE_HASH_SIZE];
-struct delayed_work dentry_tree_work;
+static struct delayed_work dentry_tree_work;
 /* Minimum invalidation work queue frequency */
 #define FUSE_DENTRY_INVAL_FREQ_MIN 5
 struct fuse_dentry {
 	u64 time;
+	u64 epoch;
 	union {
 		struct rcu_head rcu;
 		struct rb_node node;
 		WARN_ON_ONCE(!RB_EMPTY_ROOT(&dentry_hash[i].tree));
 }
+void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch)
+{
+	struct fuse_dentry *fd = dentry->d_fsdata;
+	fd->epoch = epoch;
+}
 static inline void __fuse_dentry_settime(struct dentry *dentry, u64 time)
 {
 	((struct fuse_dentry *) dentry->d_fsdata)->time = time;
 	struct fuse_mount *fm;
 	struct fuse_conn *fc;
 	struct fuse_inode *fi;
+	struct fuse_dentry *fd = entry->d_fsdata;
 	int ret;
 	fc = get_fuse_conn_super(dir->i_sb);
-	if (entry->d_time < atomic_read(&fc->epoch))
+	if (fd->epoch < atomic_read(&fc->epoch))
 		goto invalid;
 	inode = d_inode_rcu(entry);
 	RB_CLEAR_NODE(&fd->node);
 	dentry->d_fsdata = fd;
 	/*
-	 * Initialising d_time (epoch) to '0' ensures the dentry is invalid
+	 * Initialising epoch to '0' ensures the dentry is invalid
 	 * if compared to fc->epoch, which is initialized to '1'.
 	 */
-	dentry->d_time = 0;
+	fuse_dentry_set_epoch(dentry, 0);
 	return 0;
 }
 		goto out_err;
 	entry = newent ? newent : entry;
-	entry->d_time = epoch;
+	fuse_dentry_set_epoch(entry, epoch);
 	if (outarg_valid)
 		fuse_change_entry_timeout(entry, &outarg);
 	else
 	}
 	kfree(forget);
 	d_instantiate(entry, inode);
-	entry->d_time = epoch;
+	fuse_dentry_set_epoch(entry, epoch);
 	fuse_change_entry_timeout(entry, &outentry);
 	fuse_dir_changed(dir);
 	err = generic_file_open(inode, file);
 		return d;
 	if (d) {
-		d->d_time = epoch;
+		fuse_dentry_set_epoch(d, epoch);
 		fuse_change_entry_timeout(d, &outarg);
 	} else {
-		entry->d_time = epoch;
+		fuse_dentry_set_epoch(entry, epoch);
 		fuse_change_entry_timeout(entry, &outarg);
 	}
 	fuse_dir_changed(dir);
 		filemap_invalidate_lock(mapping);
 		fault_blocked = true;
 		err = fuse_dax_break_layouts(inode, 0, -1);
-		if (err) {
+		if (err)
-			filemap_invalidate_unlock(mapping);
+			goto unlock;
-			return err;
-		}
 	}
 	if (attr->ia_valid & ATTR_OPEN) {
 			 ATTR_TIMES_SET)) {
 		err = write_inode_now(inode, true);
 		if (err)
-			return err;
+			goto unlock;
 		fuse_set_nowrite(inode);
 		fuse_release_nowrite(inode);
 	 */
 	if ((is_truncate || !is_wb) &&
 	    S_ISREG(inode->i_mode) && oldsize != outarg.attr.size) {
+		if (outarg.attr.size > oldsize)
+			truncate_pagecache_range(inode, oldsize,
+						 outarg.attr.size - 1);
 		truncate_pagecache(inode, outarg.attr.size);
 		invalidate_inode_pages2(mapping);
 	}
 	clear_bit(FUSE_I_SIZE_UNSTABLE, &fi->state);
+unlock:
 	if (fault_blocked)
 		filemap_invalidate_unlock(mapping);
 	return err;

fs/fuse/file.c+16−4

 		filemap_invalidate_lock(inode->i_mapping);
 		err = fuse_dax_break_layouts(inode, 0, -1);
 		if (err)
-			goto out_inode_unlock;
+			goto out_unlock;
 	}
 	if (is_wb_truncate || dax_truncate)
 		else if (!(ff->open_flags & FOPEN_KEEP_CACHE))
 			invalidate_inode_pages2(inode->i_mapping);
 	}
+out_unlock:
 	if (dax_truncate)
 		filemap_invalidate_unlock(inode->i_mapping);
-out_inode_unlock:
 	if (is_wb_truncate || dax_truncate)
 		inode_unlock(inode);
 	args->out_argvar = true;
 	args->out_numargs = 1;
 	args->out_args[0].size = count;
+	args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
 }
 static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres,
 	args->out_numargs = 1;
 	args->out_args[0].size = sizeof(ia->write.out);
 	args->out_args[0].value = &ia->write.out;
+	args->zero_copy = ff->open_flags & FOPEN_IO_URING_ZERO_COPY;
 }
 static unsigned int fuse_write_flags(struct kiocb *iocb)
 	struct fuse_conn *fc = get_fuse_conn(inode);
 	struct fuse_inode *fi = get_fuse_inode(inode);
 	loff_t pos = iocb->ki_pos;
+	loff_t old_size = i_size_read(inode);
 	int err = 0;
 	ssize_t res = 0;
+	if (pos > old_size)
+		truncate_pagecache_range(inode, old_size, pos - 1);
 	if (inode->i_size < pos + iov_iter_count(ii))
 		set_bit(FUSE_I_SIZE_UNSTABLE, &fi->state);
 {
 	struct inode *inode = file_inode(iocb->ki_filp);
 	struct address_space *mapping = inode->i_mapping;
-	loff_t pos = iocb->ki_pos;
 	ssize_t res;
 	bool exclusive;
 	fuse_dio_lock(iocb, from, &exclusive);
 	res = generic_write_checks(iocb, from);
 	if (res > 0) {
+		loff_t pos = iocb->ki_pos;
 		task_io_account_write(res);
 		if (!is_sync_kiocb(iocb)) {
 			res = fuse_direct_IO(iocb, from);
 			/*
 			 * As in generic_file_direct_write(), invalidate after
 			 * write, to invalidate read-ahead cache that may have
-			 * with the write.
+			 * competed with the write.
 			 */
 			invalidate_inode_pages2_range(mapping,
 				pos >> PAGE_SHIFT,
 	/* we could have extended the file */
 	if (!(mode & FALLOC_FL_KEEP_SIZE)) {
+		loff_t oldsize = i_size_read(inode);
+		if (offset + length > oldsize)
+			truncate_pagecache_range(inode, oldsize,
+						 offset + length - 1);
 		if (fuse_write_update_attr(inode, offset + length, length))
 			file_update_time(file);
 	}

fs/fuse/fuse_dev_i.h+5−0

  * @FR_PRIVATE:		request is on private list
  * @FR_ASYNC:		request is asynchronous
  * @FR_URING:		request is handled through fuse-io-uring
+ * @FR_SYNC_WAKEUP:	use synchronous wakeup when queueing this request to
+ *			give the scheduler a hint about the waker task
  */
 enum fuse_req_flag {
 	FR_ISREPLY,
 	FR_PRIVATE,
 	FR_ASYNC,
 	FR_URING,
+	FR_SYNC_WAKEUP,
 };
 /**
 	bool write:1;
 	bool move_folios:1;
 	bool is_uring:1;
+	/* set when the payload is zero-copied. folios are filled in place */
+	bool skip_folio_copy:1;
 	struct {
 		unsigned int copied_sz; /* copied size into the user buffer */
 	} ring;

fs/fuse/fuse_i.h+2−0

@@ -1054,6 +1054,8 @@ u64 fuse_time_to_jiffies(u64 sec, u32 nsec); · 1053 unmodified lines
1054		1054
1055	void fuse_change_entry_timeout(struct dentry entry, struct fuse_entry_out o);	1055	void fuse_change_entry_timeout(struct dentry entry, struct fuse_entry_out o);
1056		1056
		1057	void fuse_dentry_set_epoch(struct dentry *dentry, u64 epoch);
		1058
1057	/*	1059	/*
1058	* Initialize fuse_conn	1060	* Initialize fuse_conn
1059	*/	1061	*/

fs/fuse/inode.c+10−6

 {
 	struct fuse_fs_context *ctx = fsc->fs_private;
+	if (ctx->fud)
+		return invalfc(fsc, "Multiple fd specified");
 	if (file->f_op != &fuse_dev_operations)
 		return invalfc(fsc, "fd is not a fuse device");
 	/*
 	struct fuse_mount *fm = ia->fm;
 	struct fuse_conn *fc = fm->fc;
 	struct fuse_init_out *arg = &ia->out;
+	bool io_uring_enabled = false;
 	bool ok = true;
 	if (error || arg->major != FUSE_KERNEL_VERSION)
 					ok = false;
 			}
 			if (flags & FUSE_OVER_IO_URING && fuse_uring_enabled())
-				fuse_chan_io_uring_enable(fc->chan);
+				io_uring_enabled = true;
 			if (flags & FUSE_REQUEST_TIMEOUT)
 				timeout = arg->request_timeout;
 		fm->sb->s_bdi->ra_pages =
 				min(fm->sb->s_bdi->ra_pages, ra_pages);
+		fm->sb->s_bdi->io_pages = fc->max_pages;
 		fc->minor = arg->minor;
 		fc->max_write = arg->minor < 5 ? 4096 : arg->max_write;
 		fc->max_write = max_t(unsigned, 4096, fc->max_write);
 			.minor = fc->minor,
 			.max_write = fc->max_write,
 			.max_pages = fc->max_pages,
+			.io_uring_enabled = io_uring_enabled,
 		};
 		fuse_chan_set_initialized(fc->chan, &cp);
 	}
 	if (IS_ENABLED(CONFIG_FUSE_PASSTHROUGH))
 		flags |= FUSE_PASSTHROUGH;
-	/*
-	 * This is just an information flag for fuse server. No need to check
-	 * the reply - server is either sending IORING_OP_URING_CMD or not.
-	 */
 	if (fuse_uring_enabled())
-		flags |= FUSE_OVER_IO_URING;
+		flags |= FUSE_OVER_IO_URING | FUSE_HAS_IO_URING_BUFPOOL;
 	ia->in.flags = flags;
 	ia->in.flags2 = flags >> 32;
 	fuse_fill_attr_from_inode(&root_attr, parent_fi);
 	root = fuse_iget(sb, parent_fi->nodeid, 0, &root_attr, 0, 0,
 			 fuse_get_evict_ctr(fm->fc));
+	if (!root)
+		return -ENOMEM;
 	/*
 	 * This inode is just a duplicate, so it is not looked up and
 	 * its nlookup should not be incremented.  fuse_iget() does

fs/fuse/readdir.c+1−1

@@ -260,7 +260,7 @@ retry: · 259 unmodified lines
260	}	260	}
261	if (fc->readdirplus_auto)	261	if (fc->readdirplus_auto)
262	set_bit(FUSE_I_INIT_RDPLUS, &get_fuse_inode(inode)->state);	262	set_bit(FUSE_I_INIT_RDPLUS, &get_fuse_inode(inode)->state);
263	dentry->d_time = epoch;	263	fuse_dentry_set_epoch(dentry, epoch);
264	fuse_change_entry_timeout(dentry, o);	264	fuse_change_entry_timeout(dentry, o);
265		265
266	dput(dentry);	266	dput(dentry);

fs/fuse/req_timeout.c+3−9

 void fuse_init_server_timeout(struct fuse_chan *fch, unsigned int timeout)
 {
-	if (!timeout && !fuse_max_req_timeout && !fuse_default_req_timeout)
-		return;
 	if (!timeout)
 		timeout = fuse_default_req_timeout;
-	if (fuse_max_req_timeout) {
+	timeout = min_not_zero(timeout, fuse_max_req_timeout);
-		if (timeout)
+	if (!timeout)
-			timeout = min(fuse_max_req_timeout, timeout);
+		return;
-		else
-			timeout = fuse_max_req_timeout;
-	}
 	timeout = max(FUSE_TIMEOUT_TIMER_FREQ, timeout);

include/linux/io_uring/cmd.h+32−6

@@ -91,6 +91,15 @@ struct io_br_sel io_uring_cmd_buffer_select(struct io_uring_cmd *ioucmd, · 90 unmodified lines
91	bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,	91	bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd,
92	struct io_br_sel *sel, unsigned int issue_flags);	92	struct io_br_sel *sel, unsigned int issue_flags);
93		93
		94	int io_buffer_register_request(struct io_uring_cmd cmd, struct request rq,
		95	void (release)(void ), unsigned int index,
		96	unsigned int issue_flags);
		97	int io_buffer_register_bvec(struct io_uring_cmd cmd, const struct bio_vec bvs,
		98	unsigned int nr_bvecs, void (release)(void ),
		99	void *priv, u8 dir, unsigned int index,
		100	unsigned int issue_flags);
		101	int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
		102	unsigned int issue_flags);
94	#else	103	#else
95	static inline int	104	static inline int
96	io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,	105	io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
@@ -133,6 +142,29 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, · 36 unmodified lines
133	{	142	{
134	return true;	143	return true;
135	}	144	}
		145	static inline int io_buffer_register_request(struct io_uring_cmd *cmd,
		146	struct request *rq,
		147	void (release)(void ),
		148	unsigned int index,
		149	unsigned int issue_flags)
		150	{
		151	return -EOPNOTSUPP;
		152	}
		153	static inline int io_buffer_register_bvec(struct io_uring_cmd *cmd,
		154	const struct bio_vec *bvs,
		155	unsigned int nr_bvecs,
		156	void (release)(void ), void *priv,
		157	u8 dir, unsigned int index,
		158	unsigned int issue_flags)
		159	{
		160	return -EOPNOTSUPP;
		161	}
		162	static inline int io_buffer_unregister(struct io_uring_cmd *cmd,
		163	unsigned int index,
		164	unsigned int issue_flags)
		165	{
		166	return -EOPNOTSUPP;
		167	}
136	#endif	168	#endif
137		169
138	static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)	170	static inline struct io_uring_cmd *io_uring_cmd_from_tw(struct io_tw_req tw_req)
@@ -182,10 +214,4 @@ static inline void io_uring_cmd_done32(struct io_uring_cmd *ioucmd, s32 ret, · 43 unmodified lines
182	return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true);	214	return __io_uring_cmd_done(ioucmd, ret, res2, issue_flags, true);
183	}	215	}
184		216
185	int io_buffer_register_bvec(struct io_uring_cmd cmd, struct request rq,
186	void (release)(void ), unsigned int index,
187	unsigned int issue_flags);
188	int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
189	unsigned int issue_flags);
190
191	#endif /* _LINUX_IO_URING_CMD_H */	217	#endif /* _LINUX_IO_URING_CMD_H */

include/linux/io_uring_types.h+5−0

 	IO_URING_F_COMPAT		= (1 << 12),
 };
+enum {
+	IO_BUF_DEST	= 1 << ITER_DEST,
+	IO_BUF_SOURCE	= 1 << ITER_SOURCE,
+};
 struct iou_loop_params;
 struct io_wq_work_node {

include/uapi/linux/fuse.h+61−2

@@ -240,6 +240,14 @@ · 239 unmodified lines
240	* - add FUSE_COPY_FILE_RANGE_64	240	* - add FUSE_COPY_FILE_RANGE_64
241	* - add struct fuse_copy_file_range_out	241	* - add struct fuse_copy_file_range_out
242	* - add FUSE_NOTIFY_PRUNE	242	* - add FUSE_NOTIFY_PRUNE
		243	*
		244	* 7.46
		245	* - add FUSE_IO_URING_CMD_ADD_QUEUE
		246	* - add FUSE_HAS_IO_URING_BUFPOOL
		247	* - add fuse_uring_cmd_req bufpool struct
		248	* - add bufpool offset field to fuse_uring_ent_in_out struct
		249	* - add FUSE_URING_ZERO_COPY, FUSE_URING_ENT_ZERO_COPY, and
		250	* FOPEN_IO_URING_ZERO_COPY flag
243	*/	251	*/
244		252
245	#ifndef _LINUX_FUSE_H	253	#ifndef _LINUX_FUSE_H
@@ -275,7 +283,7 @@ · 29 unmodified lines
275	#define FUSE_KERNEL_VERSION 7	283	#define FUSE_KERNEL_VERSION 7
276		284
277	/** Minor version number of this interface */	285	/** Minor version number of this interface */
278	#define FUSE_KERNEL_MINOR_VERSION 45	286	#define FUSE_KERNEL_MINOR_VERSION 46
279		287
280	/** The node ID of the root inode */	288	/** The node ID of the root inode */
281	#define FUSE_ROOT_ID 1	289	#define FUSE_ROOT_ID 1
@@ -383,6 +391,12 @@ struct fuse_file_lock { · 101 unmodified lines
383	* FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE)	391	* FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE)
384	* FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode	392	* FOPEN_PARALLEL_DIRECT_WRITES: Allow concurrent direct writes on the same inode
385	* FOPEN_PASSTHROUGH: passthrough read/write io for this open file	393	* FOPEN_PASSTHROUGH: passthrough read/write io for this open file
		394	* FOPEN_IO_URING_ZERO_COPY: use io-uring zero-copy for reads/writes on this
		395	* open file. Honored only when the serving io-uring
		396	* queue was set up for zero-copy
		397	* (FUSE_URING_ZERO_COPY) and the request carries page
		398	* payload. Otherwise reads/writes fall back to
		399	* copying.
386	*/	400	*/
387	#define FOPEN_DIRECT_IO (1 << 0)	401	#define FOPEN_DIRECT_IO (1 << 0)
388	#define FOPEN_KEEP_CACHE (1 << 1)	402	#define FOPEN_KEEP_CACHE (1 << 1)
@@ -392,6 +406,7 @@ struct fuse_file_lock { · 3 unmodified lines
392	#define FOPEN_NOFLUSH (1 << 5)	406	#define FOPEN_NOFLUSH (1 << 5)
393	#define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6)	407	#define FOPEN_PARALLEL_DIRECT_WRITES (1 << 6)
394	#define FOPEN_PASSTHROUGH (1 << 7)	408	#define FOPEN_PASSTHROUGH (1 << 7)
		409	#define FOPEN_IO_URING_ZERO_COPY (1 << 8)
395		410
396	/**	411	/**
397	* INIT request/reply flags	412	* INIT request/reply flags
@@ -448,6 +463,7 @@ struct fuse_file_lock { · 50 unmodified lines
448	* FUSE_OVER_IO_URING: Indicate that client supports io-uring	463	* FUSE_OVER_IO_URING: Indicate that client supports io-uring
449	* FUSE_REQUEST_TIMEOUT: kernel supports timing out requests.	464	* FUSE_REQUEST_TIMEOUT: kernel supports timing out requests.
450	* init_out.request_timeout contains the timeout (in secs)	465	* init_out.request_timeout contains the timeout (in secs)
		466	* FUSE_HAS_IO_URING_BUFPOOL: kernel supports io-uring buffer pools
451	*/	467	*/
452	#define FUSE_ASYNC_READ (1 << 0)	468	#define FUSE_ASYNC_READ (1 << 0)
453	#define FUSE_POSIX_LOCKS (1 << 1)	469	#define FUSE_POSIX_LOCKS (1 << 1)
@@ -495,6 +511,7 @@ struct fuse_file_lock { · 41 unmodified lines
495	#define FUSE_ALLOW_IDMAP (1ULL << 40)	511	#define FUSE_ALLOW_IDMAP (1ULL << 40)
496	#define FUSE_OVER_IO_URING (1ULL << 41)	512	#define FUSE_OVER_IO_URING (1ULL << 41)
497	#define FUSE_REQUEST_TIMEOUT (1ULL << 42)	513	#define FUSE_REQUEST_TIMEOUT (1ULL << 42)
		514	#define FUSE_HAS_IO_URING_BUFPOOL (1ULL << 43)
498		515
499	/**	516	/**
500	* CUSE INIT request/reply flags	517	* CUSE INIT request/reply flags
@@ -1251,6 +1268,13 @@ struct fuse_supp_groups { · 750 unmodified lines
1251	#define FUSE_URING_IN_OUT_HEADER_SZ 128	1268	#define FUSE_URING_IN_OUT_HEADER_SZ 128
1252	#define FUSE_URING_OP_IN_OUT_SZ 128	1269	#define FUSE_URING_OP_IN_OUT_SZ 128
1253		1270
		1271	/**
		1272	* fuse_uring_ent_in_out flags
		1273	*
		1274	* FUSE_URING_ENT_ZERO_COPY: Set if the ent's payload is zero-copied
		1275	*/
		1276	#define FUSE_URING_ENT_ZERO_COPY (1 << 0)
		1277
1254	/* Used as part of the fuse_uring_req_header */	1278	/* Used as part of the fuse_uring_req_header */
1255	struct fuse_uring_ent_in_out {	1279	struct fuse_uring_ent_in_out {
1256	uint64_t flags;	1280	uint64_t flags;
@@ -1263,7 +1287,9 @@ struct fuse_uring_ent_in_out { · 6 unmodified lines
1263		1287
1264	/* size of user payload buffer */	1288	/* size of user payload buffer */
1265	uint32_t payload_sz;	1289	uint32_t payload_sz;
1266	uint32_t padding;	1290
		1291	/* Offset into the bufpool, if bufpools are used */
		1292	uint32_t offset;
1267		1293
1268	uint64_t reserved;	1294	uint64_t reserved;
1269	};	1295	};
@@ -1292,8 +1318,22 @@ enum fuse_uring_cmd { · 22 unmodified lines
1292		1318
1293	/* commit fuse request result and fetch next request */	1319	/* commit fuse request result and fetch next request */
1294	FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2,	1320	FUSE_IO_URING_CMD_COMMIT_AND_FETCH = 2,
		1321
		1322	/* add a queue */
		1323	FUSE_IO_URING_CMD_ADD_QUEUE = 3,
		1324
		1325	/* add a bufpool to a queue */
		1326	FUSE_IO_URING_CMD_ADD_BUFPOOL = 4,
1295	};	1327	};
1296		1328
		1329	/*
		1330	* fuse_uring_cmd_req flags for FUSE_IO_URING_CMD_ADD_QUEUE
		1331	*
		1332	* FUSE_URING_ZERO_COPY is only supported for queues with bufpools on privileged
		1333	* servers
		1334	*/
		1335	#define FUSE_URING_ZERO_COPY (1 << 0)
		1336
1297	/**	1337	/**
1298	* In the 80B command area of the SQE.	1338	* In the 80B command area of the SQE.
1299	*/	1339	*/
@@ -1306,6 +1346,25 @@ struct fuse_uring_cmd_req { · 6 unmodified lines
1306	/* queue the command is for (queue index) */	1346	/* queue the command is for (queue index) */
1307	uint16_t qid;	1347	uint16_t qid;
1308	uint8_t padding[6];	1348	uint8_t padding[6];
		1349
		1350	union {
		1351	struct {
		1352	/* base address of bufpool */
		1353	uint64_t uaddr;
		1354	uint32_t len;
		1355	uint32_t reserved;
		1356	} bufpool;
		1357
		1358	/*
		1359	* Index of this entry's slot in the server's io_uring
		1360	* registered buffer table, where the kernel registers the
		1361	* request's pages for zero-copy. Set for
		1362	* FUSE_IO_URING_CMD_REGISTER cmds only, and only on queues
		1363	* created with FUSE_URING_ZERO_COPY. On a non-zero-copy queue
		1364	* this must be 0
		1365	*/
		1366	uint16_t ent_zero_copy_buf_index;
		1367	};
1309	};	1368	};
1310		1369
1311	#endif /* _LINUX_FUSE_H */	1370	#endif /* _LINUX_FUSE_H */

io_uring/io_uring.c+1−1

@@ -3236,7 +3236,7 @@ static int __init io_uring_init(void) · 3235 unmodified lines
3236	io_uring_optable_init();	3236	io_uring_optable_init();
3237		3237
3238	/* imu->dir is u8 */	3238	/* imu->dir is u8 */
3239	BUILD_BUG_ON((IO_IMU_DEST \| IO_IMU_SOURCE) > U8_MAX);	3239	BUILD_BUG_ON((IO_BUF_DEST \| IO_BUF_SOURCE) > U8_MAX);
3240		3240
3241	/*	3241	/*
3242	* Allow user copy in the per-command field, which starts after the	3242	* Allow user copy in the per-command field, which starts after the

io_uring/rsrc.c+90−37

 	imu->release = io_release_ubuf;
 	imu->priv = imu;
 	imu->flags = 0;
-	imu->dir = IO_IMU_DEST | IO_IMU_SOURCE;
+	imu->dir = IO_BUF_DEST | IO_BUF_SOURCE;
 	if (coalesced)
 		imu->folio_shift = data.folio_shift;
 	refcount_set(&imu->refs, 1);
 	return ret;
 }
-int io_buffer_register_bvec(struct io_uring_cmd *cmd, struct request *rq,
+static struct io_mapped_ubuf *io_kernel_buffer_init(struct io_ring_ctx *ctx,
-			    void (*release)(void *), unsigned int index,
+						    unsigned int nr_bvecs,
-			    unsigned int issue_flags)
+						    unsigned int total_bytes,
+						    u8 dir,
+						    void (*release)(void *),
+						    void *priv,
+						    unsigned int index)
 {
-	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
 	struct io_rsrc_data *data = &ctx->buf_table;
-	struct req_iterator rq_iter;
 	struct io_mapped_ubuf *imu;
 	struct io_rsrc_node *node;
-	struct bio_vec bv;
-	unsigned int nr_bvecs = 0;
-	int ret = 0;
-	io_ring_submit_lock(ctx, issue_flags);
+	if (index >= data->nr)
-	if (index >= data->nr) {
+		return ERR_PTR(-EINVAL);
-		ret = -EINVAL;
-		goto unlock;
-	}
 	index = array_index_nospec(index, data->nr);
-	if (data->nodes[index]) {
+	if (data->nodes[index])
-		ret = -EBUSY;
+		return ERR_PTR(-EBUSY);
-		goto unlock;
-	}
 	node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);
-	if (!node) {
+	if (!node)
-		ret = -ENOMEM;
+		return ERR_PTR(-ENOMEM);
-		goto unlock;
-	}
-	/*
+	imu = io_alloc_imu(ctx, nr_bvecs);
-	 * blk_rq_nr_phys_segments() may overestimate the number of bvecs
-	 * but avoids needing to iterate over the bvecs
-	 */
-	imu = io_alloc_imu(ctx, blk_rq_nr_phys_segments(rq));
 	if (!imu) {
 		io_cache_free(&ctx->node_cache, node);
-		ret = -ENOMEM;
+		return ERR_PTR(-ENOMEM);
-		goto unlock;
 	}
 	imu->ubuf = 0;
-	imu->len = blk_rq_bytes(rq);
+	imu->len = total_bytes;
 	imu->folio_shift = PAGE_SHIFT;
+	imu->nr_bvecs = nr_bvecs;
 	refcount_set(&imu->refs, 1);
 	imu->release = release;
-	imu->priv = rq;
+	imu->priv = priv;
+	imu->dir = dir;
 	imu->flags = IO_REGBUF_F_KBUF;
-	imu->dir = 1 << rq_data_dir(rq);
+	node->buf = imu;
+	data->nodes[index] = node;
+	return imu;
+}
+int io_buffer_register_request(struct io_uring_cmd *cmd, struct request *rq,
+			       void (*release)(void *), unsigned int index,
+			       unsigned int issue_flags)
+{
+	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
+	struct req_iterator rq_iter;
+	struct io_mapped_ubuf *imu;
+	struct bio_vec bv;
+	/*
+	 * blk_rq_nr_phys_segments() may overestimate the number of bvecs
+	 * but avoids needing to iterate over the bvecs
+	 */
+	unsigned int nr_bvecs = blk_rq_nr_phys_segments(rq);
+	unsigned int total_bytes = blk_rq_bytes(rq);
+	int ret = 0;
+	io_ring_submit_lock(ctx, issue_flags);
+	imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes,
+				    1 << rq_data_dir(rq), release, rq, index);
+	if (IS_ERR(imu)) {
+		ret = PTR_ERR(imu);
+		goto unlock;
+	}
+	nr_bvecs = 0;
 	rq_for_each_bvec(bv, rq, rq_iter)
 		imu->bvec[nr_bvecs++] = bv;
 	imu->nr_bvecs = nr_bvecs;
-	node->buf = imu;
+unlock:
-	data->nodes[index] = node;
+	io_ring_submit_unlock(ctx, issue_flags);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(io_buffer_register_request);
+/*
+ * bvs is copied internally. caller may free it on return.
+ */
+int io_buffer_register_bvec(struct io_uring_cmd *cmd, const struct bio_vec *bvs,
+			    unsigned int nr_bvecs, void (*release)(void *),
+			    void *priv, u8 dir, unsigned int index,
+			    unsigned int issue_flags)
+{
+	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
+	struct io_mapped_ubuf *imu;
+	struct bio_vec *bvec;
+	unsigned int i, total_bytes = 0;
+	int ret = 0;
+	for (i = 0; i < nr_bvecs; i++)
+		total_bytes += bvs[i].bv_len;
+	io_ring_submit_lock(ctx, issue_flags);
+	imu = io_kernel_buffer_init(ctx, nr_bvecs, total_bytes, dir, release,
+				    priv, index);
+	if (IS_ERR(imu)) {
+		ret = PTR_ERR(imu);
+		goto unlock;
+	}
+	bvec = imu->bvec;
+	for (i = 0; i < nr_bvecs; i++)
+		bvec[i] = bvs[i];
 unlock:
 	io_ring_submit_unlock(ctx, issue_flags);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(io_buffer_register_bvec);
-int io_buffer_unregister_bvec(struct io_uring_cmd *cmd, unsigned int index,
+int io_buffer_unregister(struct io_uring_cmd *cmd, unsigned int index,
-			      unsigned int issue_flags)
+			 unsigned int issue_flags)
 {
 	struct io_ring_ctx *ctx = cmd_to_io_kiocb(cmd)->ctx;
 	struct io_rsrc_data *data = &ctx->buf_table;
 	io_ring_submit_unlock(ctx, issue_flags);
 	return ret;
 }
-EXPORT_SYMBOL_GPL(io_buffer_unregister_bvec);
+EXPORT_SYMBOL_GPL(io_buffer_unregister);
 static int validate_fixed_range(u64 buf_addr, size_t len,
 				const struct io_mapped_ubuf *imu)

io_uring/rsrc.h+0−5

 	};
 };
-enum {
-	IO_IMU_DEST	= 1 << ITER_DEST,
-	IO_IMU_SOURCE	= 1 << ITER_SOURCE,
-};
 enum {
 	IO_REGBUF_F_KBUF		= 1,
 };

tools/testing/selftests/filesystems/fuse/.gitignore+1−0

@@ -1,3 +1,4 @@
1	# SPDX-License-Identifier: GPL-2.0-only	1	# SPDX-License-Identifier: GPL-2.0-only
2	fuse_mnt	2	fuse_mnt
3	fusectl_test	3	fusectl_test
		4	write_extend_eof_test

tools/testing/selftests/filesystems/fuse/Makefile+3−0

 CFLAGS += -Wall -O2 -g $(KHDR_INCLUDES)
 TEST_GEN_PROGS := fusectl_test
+TEST_GEN_PROGS += write_extend_eof_test
 TEST_GEN_FILES := fuse_mnt
 # fuse_acl_cache_test requires libfuse3; add it only when the library is present.
 include ../../lib.mk
+$(OUTPUT)/write_extend_eof_test: LDLIBS += -lpthread
 VAR_CFLAGS := $(shell pkg-config fuse --cflags 2>/dev/null)
 ifeq ($(VAR_CFLAGS),)
 VAR_CFLAGS := -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse

tools/testing/selftests/filesystems/fuse/write_extend_eof_test.c+368−0

@@ -0,0 +1,368 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/*
	3	* Regression test for the fuse write-extend partial-EOF-page zeroing bug.
	4	*
	5	* A buffered write that extends i_size past a non-page-aligned EOF must zero
	6	* the tail of the old last page. If an application has mmap'd that page and
	7	* stored into the post-EOF region (undefined until the file grows), the
	8	* now-in-bounds tail must read back as zero, not as the stale stored bytes.
	9	*
	10	* The bug is exposed on a non-writeback_cache server that keeps the page cache
	11	* across the write (FOPEN_KEEP_CACHE without FOPEN_DIRECT_IO). This test is a
	12	* raw /dev/fuse server in that mode; the backing data is always zero in the
	13	* hole, so any non-zero byte a read sees is stale page-cache data.
	14	*
	15	* Requires root to mount fuse.
	16	*/
	17	#define _GNU_SOURCE
	18	#include <errno.h>
	19	#include <fcntl.h>
	20	#include <linux/falloc.h>
	21	#include <pthread.h>
	22	#include <stdint.h>
	23	#include <string.h>
	24	#include <unistd.h>
	25	#include <sys/mman.h>
	26	#include <sys/mount.h>
	27	#include <sys/stat.h>
	28	#include <sys/uio.h>
	29	#include <linux/fuse.h>
	30
	31	#include "../../kselftest_harness.h"
	32
	33	#define FUSE_ROOT_ID 1
	34	#define FILE_INO 2
	35	#define MAX_WRITE (128 * 1024)
	36	#define BACKING_SIZE (4 * 1024 * 1024)
	37	#define POLLUTE 0xee
	38
	39	/* Server-side state, shared with the responder thread. */
	40	struct server {
	41	int fd;
	42	unsigned char backing[BACKING_SIZE]; /* authoritative bytes */
	43	uint64_t size;
	44	};
	45
	46	static void reply(int fd, uint64_t unique, int error, void *data, size_t len)
	47	{
	48	struct fuse_out_header oh = {
	49	.len = sizeof(oh) + (data ? len : 0),
	50	.error = error,
	51	.unique = unique,
	52	};
	53	struct iovec iov[2] = { { &oh, sizeof(oh) }, { data, len } };
	54
	55	/* Errors here are teardown races (device closed on unmount); ignore. */
	56	if (writev(fd, iov, data ? 2 : 1) < 0)
	57	return;
	58	}
	59
	60	static void fill_attr(struct fuse_attr *a, uint64_t ino, uint32_t mode,
	61	uint64_t size)
	62	{
	63	memset(a, 0, sizeof(*a));
	64	a->ino = ino;
	65	a->mode = mode;
	66	a->nlink = 1;
	67	a->size = size;
	68	a->blksize = sysconf(_SC_PAGESIZE);
	69	}
	70
	71	static void server_thread(void arg)
	72	{
	73	struct server *s = arg;
	74	static char buf[MAX_WRITE + 4096];
	75
	76	for (;;) {
	77	ssize_t n = read(s->fd, buf, sizeof(buf));
	78	struct fuse_in_header ih = (void )buf;
	79
	80	if (n < 0) {
	81	if (errno == EINTR \|\| errno == EAGAIN)
	82	continue;
	83	return NULL; /* device closed on unmount */
	84	}
	85	if (n < (ssize_t)sizeof(*ih))
	86	continue;
	87
	88	switch (ih->opcode) {
	89	case FUSE_INIT: {
	90	struct fuse_init_in in = (void )(ih + 1);
	91	struct fuse_init_out out = {0};
	92
	93	/* No FUSE_WRITEBACK_CACHE: the exposed configuration. */
	94	out.major = FUSE_KERNEL_VERSION;
	95	out.minor = FUSE_KERNEL_MINOR_VERSION;
	96	out.max_readahead = in->max_readahead;
	97	out.max_write = MAX_WRITE;
	98	out.max_background = 16;
	99	out.congestion_threshold = 12;
	100	out.flags = FUSE_MAX_PAGES;
	101	out.max_pages = MAX_WRITE / sysconf(_SC_PAGESIZE);
	102	reply(s->fd, ih->unique, 0, &out, sizeof(out));
	103	break;
	104	}
	105	case FUSE_GETATTR: {
	106	struct fuse_attr_out out = {0};
	107	int root = ih->nodeid == FUSE_ROOT_ID;
	108
	109	out.attr_valid = 3600;
	110	fill_attr(&out.attr, ih->nodeid,
	111	root ? (S_IFDIR \| 0755) : (S_IFREG \| 0644),
	112	root ? 0 : s->size);
	113	reply(s->fd, ih->unique, 0, &out, sizeof(out));
	114	break;
	115	}
	116	case FUSE_LOOKUP: {
	117	struct fuse_entry_out out = {0};
	118
	119	out.nodeid = FILE_INO;
	120	out.attr_valid = 3600;
	121	out.entry_valid = 3600;
	122	fill_attr(&out.attr, FILE_INO, S_IFREG \| 0644, s->size);
	123	reply(s->fd, ih->unique, 0, &out, sizeof(out));
	124	break;
	125	}
	126	case FUSE_OPEN:
	127	case FUSE_OPENDIR: {
	128	struct fuse_open_out out = {0};
	129
	130	/* Keep the cache across the write, but not direct I/O. */
	131	out.open_flags = FOPEN_KEEP_CACHE;
	132	reply(s->fd, ih->unique, 0, &out, sizeof(out));
	133	break;
	134	}
	135	case FUSE_READ: {
	136	struct fuse_read_in in = (void )(ih + 1);
	137	uint64_t off = in->offset;
	138	uint32_t size = in->size;
	139
	140	if (off >= BACKING_SIZE)
	141	size = 0;
	142	else if (off + size > BACKING_SIZE)
	143	size = BACKING_SIZE - off;
	144	reply(s->fd, ih->unique, 0, s->backing + off, size);
	145	break;
	146	}
	147	case FUSE_WRITE: {
	148	struct fuse_write_in in = (void )(ih + 1);
	149	struct fuse_write_out out = {0};
	150	uint64_t off = in->offset;
	151	uint32_t size = in->size;
	152
	153	if (off < BACKING_SIZE) {
	154	uint32_t c = size;
	155
	156	if (off + c > BACKING_SIZE)
	157	c = BACKING_SIZE - off;
	158	memcpy(s->backing + off, in + 1, c);
	159	if (off + c > s->size)
	160	s->size = off + c;
	161	}
	162	out.size = size;
	163	reply(s->fd, ih->unique, 0, &out, sizeof(out));
	164	break;
	165	}
	166	case FUSE_SETATTR: {
	167	struct fuse_setattr_in in = (void )(ih + 1);
	168	struct fuse_attr_out out = {0};
	169
	170	if ((in->valid & FATTR_SIZE) && in->size <= BACKING_SIZE) {
	171	if (in->size > s->size)
	172	memset(s->backing + s->size, 0,
	173	in->size - s->size);
	174	s->size = in->size;
	175	}
	176	out.attr_valid = 3600;
	177	fill_attr(&out.attr, ih->nodeid, S_IFREG \| 0644, s->size);
	178	reply(s->fd, ih->unique, 0, &out, sizeof(out));
	179	break;
	180	}
	181	case FUSE_FALLOCATE: {
	182	struct fuse_fallocate_in in = (void )(ih + 1);
	183	uint64_t end = in->offset + in->length;
	184
	185	/* Only plain (size-extending) fallocate is used here. */
	186	if (!(in->mode & FALLOC_FL_KEEP_SIZE) &&
	187	end <= BACKING_SIZE && end > s->size) {
	188	memset(s->backing + s->size, 0, end - s->size);
	189	s->size = end;
	190	}
	191	reply(s->fd, ih->unique, 0, NULL, 0);
	192	break;
	193	}
	194	case FUSE_FLUSH:
	195	case FUSE_RELEASE:
	196	case FUSE_RELEASEDIR:
	197	case FUSE_FSYNC:
	198	case FUSE_ACCESS:
	199	reply(s->fd, ih->unique, 0, NULL, 0);
	200	break;
	201	case FUSE_FORGET:
	202	break;
	203	default:
	204	reply(s->fd, ih->unique, -EOPNOTSUPP, NULL, 0);
	205	break;
	206	}
	207	}
	208	}
	209
	210	FIXTURE(fuse)
	211	{
	212	struct server *srv;
	213	pthread_t thread;
	214	char dir[64];
	215	long page; /* runtime page size */
	216	off_t eof; /* mid-page EOF, page-relative */
	217	int fd; /* open test file */
	218	char map; / mmap of the EOF page */
	219	int mounted;
	220	};
	221
	222	FIXTURE_SETUP(fuse)
	223	{
	224	char opts[128];
	225	pthread_t t;
	226
	227	if (geteuid() != 0)
	228	SKIP(return, "need root to mount fuse");
	229
	230	self->page = sysconf(_SC_PAGESIZE);
	231	self->fd = -1;
	232	self->map = MAP_FAILED;
	233
	234	self->srv = mmap(NULL, sizeof(*self->srv), PROT_READ \| PROT_WRITE,
	235	MAP_SHARED \| MAP_ANONYMOUS, -1, 0);
	236	ASSERT_NE(MAP_FAILED, self->srv);
	237
	238	self->srv->fd = open("/dev/fuse", O_RDWR);
	239	ASSERT_GE(self->srv->fd, 0);
	240
	241	strcpy(self->dir, "/tmp/fuse_weof_XXXXXX");
	242	ASSERT_NE(NULL, mkdtemp(self->dir));
	243
	244	snprintf(opts, sizeof(opts),
	245	"fd=%d,rootmode=40000,user_id=0,group_id=0",
	246	self->srv->fd);
	247	ASSERT_EQ(0, mount("fuse", self->dir, "fuse", 0, opts));
	248	self->mounted = 1;
	249
	250	ASSERT_EQ(0, pthread_create(&t, NULL, server_thread, self->srv));
	251	self->thread = t;
	252	}
	253
	254	FIXTURE_TEARDOWN(fuse)
	255	{
	256	if (self->map != MAP_FAILED)
	257	munmap(self->map, self->page);
	258	if (self->fd >= 0)
	259	close(self->fd);
	260	if (self->mounted)
	261	umount2(self->dir, MNT_DETACH);
	262	if (self->srv && self->srv != MAP_FAILED) {
	263	if (self->srv->fd > 0)
	264	close(self->srv->fd);
	265	munmap(self->srv, sizeof(*self->srv));
	266	}
	267	if (self->dir[0])
	268	rmdir(self->dir);
	269	}
	270
	271	/*
	272	* Create the test file with a mid-page EOF and mmap-store POLLUTE into its
	273	* post-EOF tail (a legal store, undefined until the file grows). Leaves the
	274	* file open and the EOF page mapped in the fixture for the caller to extend.
	275	*/
	276	static void pollute_eof_tail(struct __test_metadata *_metadata,
	277	FIXTURE_DATA(fuse) * self)
	278	{
	279	off_t eof = 2 * self->page + self->page / 4;
	280	char path[128];
	281	char *buf;
	282
	283	snprintf(path, sizeof(path), "%s/file", self->dir);
	284	self->fd = open(path, O_RDWR \| O_CREAT \| O_TRUNC, 0644);
	285	ASSERT_GE(self->fd, 0);
	286	self->eof = eof;
	287
	288	buf = malloc(eof);
	289	ASSERT_NE(NULL, buf);
	290	memset(buf, 'A', eof);
	291	ASSERT_EQ(eof, pwrite(self->fd, buf, eof, 0));
	292	free(buf);
	293
	294	self->map = mmap(NULL, self->page, PROT_READ \| PROT_WRITE, MAP_SHARED,
	295	self->fd, eof & ~(self->page - 1));
	296	ASSERT_NE(MAP_FAILED, self->map);
	297	memset(self->map + (eof & (self->page - 1)), POLLUTE,
	298	self->page - (eof & (self->page - 1)));
	299	}
	300
	301	/* Assert the old post-EOF tail [eof, end of its page) now reads back as zero. */
	302	static void assert_tail_zeroed(struct __test_metadata *_metadata,
	303	FIXTURE_DATA(fuse) * self)
	304	{
	305	off_t base = self->eof & ~(self->page - 1);
	306	char *tail = malloc(self->page);
	307	int i;
	308
	309	ASSERT_NE(NULL, tail);
	310	ASSERT_EQ(self->page, pread(self->fd, tail, self->page, base));
	311	for (i = self->eof & (self->page - 1); i < self->page; i++)
	312	ASSERT_EQ(0, tail[i]);
	313	free(tail);
	314	}
	315
	316	/* Basic: pollute the post-EOF tail, extend past it by a later write. */
	317	TEST_F(fuse, write_extend)
	318	{
	319	pollute_eof_tail(_metadata, self);
	320	ASSERT_EQ(4, pwrite(self->fd, "data", 4, 5 * self->page + self->page / 3));
	321	assert_tail_zeroed(_metadata, self);
	322	}
	323
	324	/* Extend via ftruncate() rather than a write. */
	325	TEST_F(fuse, ftruncate_extend)
	326	{
	327	pollute_eof_tail(_metadata, self);
	328	ASSERT_EQ(0, ftruncate(self->fd, 8 * self->page));
	329	assert_tail_zeroed(_metadata, self);
	330	}
	331
	332	/* Extend via fallocate() starting at the old EOF. */
	333	TEST_F(fuse, fallocate_extend)
	334	{
	335	pollute_eof_tail(_metadata, self);
	336	ASSERT_EQ(0, fallocate(self->fd, 0, self->eof, 4 * self->page));
	337	assert_tail_zeroed(_metadata, self);
	338	}
	339
	340	/* A write landing inside the old EOF page must not clobber its own data. */
	341	TEST_F(fuse, extend_into_eof_page_preserves_data)
	342	{
	343	off_t base, wr;
	344	char buf, rd;
	345	int i;
	346
	347	pollute_eof_tail(_metadata, self);
	348	base = self->eof & ~(self->page - 1);
	349	wr = base + 3 * self->page / 4; /* starts in the EOF page */
	350
	351	buf = malloc(2 * self->page);
	352	ASSERT_NE(NULL, buf);
	353	memset(buf, 'B', 2 * self->page);
	354	ASSERT_EQ(2 * self->page, pwrite(self->fd, buf, 2 * self->page, wr));
	355	free(buf);
	356
	357	rd = malloc(self->page);
	358	ASSERT_NE(NULL, rd);
	359	ASSERT_EQ(self->page, pread(self->fd, rd, self->page, base));
	360	/* [eof, wr) is hole -> zero; [wr, page) is written data -> 'B'. */
	361	for (i = self->eof & (self->page - 1); i < wr - base; i++)
	362	ASSERT_EQ(0, rd[i]);
	363	for (i = wr - base; i < self->page; i++)
	364	ASSERT_EQ('B', rd[i]);
	365	free(rd);
	366	}
	367
	368	TEST_HARNESS_MAIN

@@ -382,17 +382,17 @@ Zero copy · 381 unmodified lines
382	---------	382	---------
383		383
384	ublk zero copy relies on io_uring's fixed kernel buffer, which provides	384	ublk zero copy relies on io_uring's fixed kernel buffer, which provides
385	two APIs: `io_buffer_register_bvec()` and `io_buffer_unregister_bvec`.	385	two APIs: `io_buffer_register_request()` and `io_buffer_unregister`.
386		386
387	ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call	387	ublk adds IO command of `UBLK_IO_REGISTER_IO_BUF` to call
388	`io_buffer_register_bvec()` for ublk server to register client request	388	`io_buffer_register_request()` for ublk server to register client request
389	buffer into io_uring buffer table, then ublk server can submit io_uring	389	buffer into io_uring buffer table, then ublk server can submit io_uring
390	IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF`	390	IOs with the registered buffer index. IO command of `UBLK_IO_UNREGISTER_IO_BUF`
391	calls `io_buffer_unregister_bvec()` to unregister the buffer, which is	391	calls `io_buffer_unregister()` to unregister the buffer, which is guaranteed
392	guaranteed to be live between calling `io_buffer_register_bvec()` and	392	to be live between calling `io_buffer_register_request()` and
393	`io_buffer_unregister_bvec()`. Any io_uring operation which supports this	393	`io_buffer_unregister()`. Any io_uring operation which supports this kind of
394	kind of kernel buffer will grab one reference of the buffer until the	394	kernel buffer will grab one reference of the buffer until the operation is
395	operation is completed.	395	completed.
396		396
397	ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and	397	ublk server implementing zero copy or user copy has to be CAP_SYS_ADMIN and
398	be trusted, because it is ublk server's responsibility to make sure IO buffer	398	be trusted, because it is ublk server's responsibility to make sure IO buffer