lael-labs/linux

Documentation/ABI/stable/sysfs-block+49−0

@@ -155,6 +155,55 @@ Description: · 154 unmodified lines
155	last zone of the device which may be smaller.	155	last zone of the device which may be smaller.
156		156
157		157
		158	What: /sys/block/<disk>/queue/crypto/
		159	Date: February 2022
		160	Contact: linux-block@vger.kernel.org
		161	Description:
		162	The presence of this subdirectory of /sys/block/<disk>/queue/
		163	indicates that the device supports inline encryption. This
		164	subdirectory contains files which describe the inline encryption
		165	capabilities of the device. For more information about inline
		166	encryption, refer to Documentation/block/inline-encryption.rst.
		167
		168
		169	What: /sys/block/<disk>/queue/crypto/max_dun_bits
		170	Date: February 2022
		171	Contact: linux-block@vger.kernel.org
		172	Description:
		173	[RO] This file shows the maximum length, in bits, of data unit
		174	numbers accepted by the device in inline encryption requests.
		175
		176
		177	What: /sys/block/<disk>/queue/crypto/modes/<mode>
		178	Date: February 2022
		179	Contact: linux-block@vger.kernel.org
		180	Description:
		181	[RO] For each crypto mode (i.e., encryption/decryption
		182	algorithm) the device supports with inline encryption, a file
		183	will exist at this location. It will contain a hexadecimal
		184	number that is a bitmask of the supported data unit sizes, in
		185	bytes, for that crypto mode.
		186
		187	Currently, the crypto modes that may be supported are:
		188
		189	* AES-256-XTS
		190	* AES-128-CBC-ESSIV
		191	* Adiantum
		192
		193	For example, if a device supports AES-256-XTS inline encryption
		194	with data unit sizes of 512 and 4096 bytes, the file
		195	/sys/block/<disk>/queue/crypto/modes/AES-256-XTS will exist and
		196	will contain "0x1200".
		197
		198
		199	What: /sys/block/<disk>/queue/crypto/num_keyslots
		200	Date: February 2022
		201	Contact: linux-block@vger.kernel.org
		202	Description:
		203	[RO] This file shows the number of keyslots the device has for
		204	use with inline encryption.
		205
		206
158	What: /sys/block/<disk>/queue/dax	207	What: /sys/block/<disk>/queue/dax
159	Date: June 2016	208	Date: June 2016
160	Contact: linux-block@vger.kernel.org	209	Contact: linux-block@vger.kernel.org

Documentation/block/biodoc.rst+0−1164

@@ -1,1164 +0,0 @@
1	=====================================================
2	Notes on the Generic Block Layer Rewrite in Linux 2.5
3	=====================================================
4
5	.. note::
6
7	It seems that there are lot of outdated stuff here. This seems
8	to be written somewhat as a task list. Yet, eventually, something
9	here might still be useful.
10
11	Notes Written on Jan 15, 2002:
12
13	- Jens Axboe <jens.axboe@oracle.com>
14	- Suparna Bhattacharya <suparna@in.ibm.com>
15
16	Last Updated May 2, 2002
17
18	September 2003: Updated I/O Scheduler portions
19	- Nick Piggin <npiggin@kernel.dk>
20
21	Introduction
22	============
23
24	These are some notes describing some aspects of the 2.5 block layer in the
25	context of the bio rewrite. The idea is to bring out some of the key
26	changes and a glimpse of the rationale behind those changes.
27
28	Please mail corrections & suggestions to suparna@in.ibm.com.
29
30	Credits
31	=======
32
33	2.5 bio rewrite:
34	- Jens Axboe <jens.axboe@oracle.com>
35
36	Many aspects of the generic block layer redesign were driven by and evolved
37	over discussions, prior patches and the collective experience of several
38	people. See sections 8 and 9 for a list of some related references.
39
40	The following people helped with review comments and inputs for this
41	document:
42
43	- Christoph Hellwig <hch@infradead.org>
44	- Arjan van de Ven <arjanv@redhat.com>
45	- Randy Dunlap <rdunlap@xenotime.net>
46	- Andre Hedrick <andre@linux-ide.org>
47
48	The following people helped with fixes/contributions to the bio patches
49	while it was still work-in-progress:
50
51	- David S. Miller <davem@redhat.com>
52
53
54	.. Description of Contents:
55
56	1. Scope for tuning of logic to various needs
57	1.1 Tuning based on device or low level driver capabilities
58	- Per-queue parameters
59	- Highmem I/O support
60	- I/O scheduler modularization
61	1.2 Tuning based on high level requirements/capabilities
62	1.2.1 Request Priority/Latency
63	1.3 Direct access/bypass to lower layers for diagnostics and special
64	device operations
65	1.3.1 Pre-built commands
66	2. New flexible and generic but minimalist i/o structure or descriptor
67	(instead of using buffer heads at the i/o layer)
68	2.1 Requirements/Goals addressed
69	2.2 The bio struct in detail (multi-page io unit)
70	2.3 Changes in the request structure
71	3. Using bios
72	3.1 Setup/teardown (allocation, splitting)
73	3.2 Generic bio helper routines
74	3.2.1 Traversing segments and completion units in a request
75	3.2.2 Setting up DMA scatterlists
76	3.2.3 I/O completion
77	3.2.4 Implications for drivers that do not interpret bios (don't handle
78	multiple segments)
79	3.3 I/O submission
80	4. The I/O scheduler
81	5. Scalability related changes
82	5.1 Granular locking: Removal of io_request_lock
83	5.2 Prepare for transition to 64 bit sector_t
84	6. Other Changes/Implications
85	6.1 Partition re-mapping handled by the generic block layer
86	7. A few tips on migration of older drivers
87	8. A list of prior/related/impacted patches/ideas
88	9. Other References/Discussion Threads
89
90
91	Bio Notes
92	=========
93
94	Let us discuss the changes in the context of how some overall goals for the
95	block layer are addressed.
96
97	1. Scope for tuning the generic logic to satisfy various requirements
98	=====================================================================
99
100	The block layer design supports adaptable abstractions to handle common
101	processing with the ability to tune the logic to an appropriate extent
102	depending on the nature of the device and the requirements of the caller.
103	One of the objectives of the rewrite was to increase the degree of tunability
104	and to enable higher level code to utilize underlying device/driver
105	capabilities to the maximum extent for better i/o performance. This is
106	important especially in the light of ever improving hardware capabilities
107	and application/middleware software designed to take advantage of these
108	capabilities.
109
110	1.1 Tuning based on low level device / driver capabilities
111	----------------------------------------------------------
112
113	Sophisticated devices with large built-in caches, intelligent i/o scheduling
114	optimizations, high memory DMA support, etc may find some of the
115	generic processing an overhead, while for less capable devices the
116	generic functionality is essential for performance or correctness reasons.
117	Knowledge of some of the capabilities or parameters of the device should be
118	used at the generic block layer to take the right decisions on
119	behalf of the driver.
120
121	How is this achieved ?
122
123	Tuning at a per-queue level:
124
125	i. Per-queue limits/values exported to the generic layer by the driver
126
127	Various parameters that the generic i/o scheduler logic uses are set at
128	a per-queue level (e.g maximum request size, maximum number of segments in
129	a scatter-gather list, logical block size)
130
131	Some parameters that were earlier available as global arrays indexed by
132	major/minor are now directly associated with the queue. Some of these may
133	move into the block device structure in the future. Some characteristics
134	have been incorporated into a queue flags field rather than separate fields
135	in themselves. There are blk_queue_xxx functions to set the parameters,
136	rather than update the fields directly
137
138	Some new queue property settings:
139
140	blk_queue_bounce_limit(q, u64 dma_address)
141	Enable I/O to highmem pages, dma_address being the
142	limit. No highmem default.
143
144	blk_queue_max_sectors(q, max_sectors)
145	Sets two variables that limit the size of the request.
146
147	- The request queue's max_sectors, which is a soft size in
148	units of 512 byte sectors, and could be dynamically varied
149	by the core kernel.
150
151	- The request queue's max_hw_sectors, which is a hard limit
152	and reflects the maximum size request a driver can handle
153	in units of 512 byte sectors.
154
155	The default for both max_sectors and max_hw_sectors is
156	255. The upper limit of max_sectors is 1024.
157
158	blk_queue_max_phys_segments(q, max_segments)
159	Maximum physical segments you can handle in a request. 128
160	default (driver limit). (See 3.2.2)
161
162	blk_queue_max_hw_segments(q, max_segments)
163	Maximum dma segments the hardware can handle in a request. 128
164	default (host adapter limit, after dma remapping).
165	(See 3.2.2)
166
167	blk_queue_max_segment_size(q, max_seg_size)
168	Maximum size of a clustered segment, 64kB default.
169
170	blk_queue_logical_block_size(q, logical_block_size)
171	Lowest possible sector size that the hardware can operate
172	on, 512 bytes default.
173
174	New queue flags:
175
176	- QUEUE_FLAG_CLUSTER (see 3.2.2)
177	- QUEUE_FLAG_QUEUED (see 3.2.4)
178
179
180	ii. High-mem i/o capabilities are now considered the default
181
182	The generic bounce buffer logic, present in 2.4, where the block layer would
183	by default copyin/out i/o requests on high-memory buffers to low-memory buffers
184	assuming that the driver wouldn't be able to handle it directly, has been
185	changed in 2.5. The bounce logic is now applied only for memory ranges
186	for which the device cannot handle i/o. A driver can specify this by
187	setting the queue bounce limit for the request queue for the device
188	(blk_queue_bounce_limit()). This avoids the inefficiencies of the copyin/out
189	where a device is capable of handling high memory i/o.
190
191	In order to enable high-memory i/o where the device is capable of supporting
192	it, the pci dma mapping routines and associated data structures have now been
193	modified to accomplish a direct page -> bus translation, without requiring
194	a virtual address mapping (unlike the earlier scheme of virtual address
195	-> bus translation). So this works uniformly for high-memory pages (which
196	do not have a corresponding kernel virtual address space mapping) and
197	low-memory pages.
198
199	Note: Please refer to Documentation/core-api/dma-api-howto.rst for a discussion
200	on PCI high mem DMA aspects and mapping of scatter gather lists, and support
201	for 64 bit PCI.
202
203	Special handling is required only for cases where i/o needs to happen on
204	pages at physical memory addresses beyond what the device can support. In these
205	cases, a bounce bio representing a buffer from the supported memory range
206	is used for performing the i/o with copyin/copyout as needed depending on
207	the type of the operation. For example, in case of a read operation, the
208	data read has to be copied to the original buffer on i/o completion, so a
209	callback routine is set up to do this, while for write, the data is copied
210	from the original buffer to the bounce buffer prior to issuing the
211	operation. Since an original buffer may be in a high memory area that's not
212	mapped in kernel virtual addr, a kmap operation may be required for
213	performing the copy, and special care may be needed in the completion path
214	as it may not be in irq context. Special care is also required (by way of
215	GFP flags) when allocating bounce buffers, to avoid certain highmem
216	deadlock possibilities.
217
218	It is also possible that a bounce buffer may be allocated from high-memory
219	area that's not mapped in kernel virtual addr, but within the range that the
220	device can use directly; so the bounce page may need to be kmapped during
221	copy operations. [Note: This does not hold in the current implementation,
222	though]
223
224	There are some situations when pages from high memory may need to
225	be kmapped, even if bounce buffers are not necessary. For example a device
226	may need to abort DMA operations and revert to PIO for the transfer, in
227	which case a virtual mapping of the page is required. For SCSI it is also
228	done in some scenarios where the low level driver cannot be trusted to
229	handle a single sg entry correctly. The driver is expected to perform the
230	kmaps as needed on such occasions as appropriate. A driver could also use
231	the blk_queue_bounce() routine on its own to bounce highmem i/o to low
232	memory for specific requests if so desired.
233
234	iii. The i/o scheduler algorithm itself can be replaced/set as appropriate
235
236	As in 2.4, it is possible to plugin a brand new i/o scheduler for a particular
237	queue or pick from (copy) existing generic schedulers and replace/override
238	certain portions of it. The 2.5 rewrite provides improved modularization
239	of the i/o scheduler. There are more pluggable callbacks, e.g for init,
240	add request, extract request, which makes it possible to abstract specific
241	i/o scheduling algorithm aspects and details outside of the generic loop.
242	It also makes it possible to completely hide the implementation details of
243	the i/o scheduler from block drivers.
244
245	I/O scheduler wrappers are to be used instead of accessing the queue directly.
246	See section 4. The I/O scheduler for details.
247
248	1.2 Tuning Based on High level code capabilities
249	------------------------------------------------
250
251	i. Application capabilities for raw i/o
252
253	This comes from some of the high-performance database/middleware
254	requirements where an application prefers to make its own i/o scheduling
255	decisions based on an understanding of the access patterns and i/o
256	characteristics
257
258	ii. High performance filesystems or other higher level kernel code's
259	capabilities
260
261	Kernel components like filesystems could also take their own i/o scheduling
262	decisions for optimizing performance. Journalling filesystems may need
263	some control over i/o ordering.
264
265	What kind of support exists at the generic block layer for this ?
266
267	The flags and rw fields in the bio structure can be used for some tuning
268	from above e.g indicating that an i/o is just a readahead request, or priority
269	settings (currently unused). As far as user applications are concerned they
270	would need an additional mechanism either via open flags or ioctls, or some
271	other upper level mechanism to communicate such settings to block.
272
273	1.2.1 Request Priority/Latency
274	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
275
276	Todo/Under discussion::
277
278	Arjan's proposed request priority scheme allows higher levels some broad
279	control (high/med/low) over the priority of an i/o request vs other pending
280	requests in the queue. For example it allows reads for bringing in an
281	executable page on demand to be given a higher priority over pending write
282	requests which haven't aged too much on the queue. Potentially this priority
283	could even be exposed to applications in some manner, providing higher level
284	tunability. Time based aging avoids starvation of lower priority
285	requests. Some bits in the bi_opf flags field in the bio structure are
286	intended to be used for this priority information.
287
288
289	1.3 Direct Access to Low level Device/Driver Capabilities (Bypass mode)
290	-----------------------------------------------------------------------
291
292	(e.g Diagnostics, Systems Management)
293
294	There are situations where high-level code needs to have direct access to
295	the low level device capabilities or requires the ability to issue commands
296	to the device bypassing some of the intermediate i/o layers.
297	These could, for example, be special control commands issued through ioctl
298	interfaces, or could be raw read/write commands that stress the drive's
299	capabilities for certain kinds of fitness tests. Having direct interfaces at
300	multiple levels without having to pass through upper layers makes
301	it possible to perform bottom up validation of the i/o path, layer by
302	layer, starting from the media.
303
304	The normal i/o submission interfaces, e.g submit_bio, could be bypassed
305	for specially crafted requests which such ioctl or diagnostics
306	interfaces would typically use, and the elevator add_request routine
307	can instead be used to directly insert such requests in the queue or preferably
308	the blk_do_rq routine can be used to place the request on the queue and
309	wait for completion. Alternatively, sometimes the caller might just
310	invoke a lower level driver specific interface with the request as a
311	parameter.
312
313	If the request is a means for passing on special information associated with
314	the command, then such information is associated with the request->special
315	field (rather than misuse the request->buffer field which is meant for the
316	request data buffer's virtual mapping).
317
318	For passing request data, the caller must build up a bio descriptor
319	representing the concerned memory buffer if the underlying driver interprets
320	bio segments or uses the block layer endrequest functions for i/o
321	completion. Alternatively one could directly use the request->buffer field to
322	specify the virtual address of the buffer, if the driver expects buffer
323	addresses passed in this way and ignores bio entries for the request type
324	involved. In the latter case, the driver would modify and manage the
325	request->buffer, request->sector and request->nr_sectors or
326	request->current_nr_sectors fields itself rather than using the block layer
327	end_request or end_that_request_first completion interfaces.
328	(See 2.3 or Documentation/block/request.rst for a brief explanation of
329	the request structure fields)
330
331	::
332
333	[TBD: end_that_request_last should be usable even in this case;
334	Perhaps an end_that_direct_request_first routine could be implemented to make
335	handling direct requests easier for such drivers; Also for drivers that
336	expect bios, a helper function could be provided for setting up a bio
337	corresponding to a data buffer]
338
339	<JENS: I dont understand the above, why is end_that_request_first() not
340	usable? Or _last for that matter. I must be missing something>
341
342	<SUP: What I meant here was that if the request doesn't have a bio, then
343	end_that_request_first doesn't modify nr_sectors or current_nr_sectors,
344	and hence can't be used for advancing request state settings on the
345	completion of partial transfers. The driver has to modify these fields
346	directly by hand.
347	This is because end_that_request_first only iterates over the bio list,
348	and always returns 0 if there are none associated with the request.
349	_last works OK in this case, and is not a problem, as I mentioned earlier
350	>
351
352	1.3.1 Pre-built Commands
353	^^^^^^^^^^^^^^^^^^^^^^^^
354
355	A request can be created with a pre-built custom command to be sent directly
356	to the device. The cmd block in the request structure has room for filling
357	in the command bytes. (i.e rq->cmd is now 16 bytes in size, and meant for
358	command pre-building, and the type of the request is now indicated
359	through rq->flags instead of via rq->cmd)
360
361	The request structure flags can be set up to indicate the type of request
362	in such cases (REQ_PC: direct packet command passed to driver, REQ_BLOCK_PC:
363	packet command issued via blk_do_rq, REQ_SPECIAL: special request).
364
365	It can help to pre-build device commands for requests in advance.
366	Drivers can now specify a request prepare function (q->prep_rq_fn) that the
367	block layer would invoke to pre-build device commands for a given request,
368	or perform other preparatory processing for the request. This is routine is
369	called by elv_next_request(), i.e. typically just before servicing a request.
370	(The prepare function would not be called for requests that have RQF_DONTPREP
371	enabled)
372
373	Aside:
374	Pre-building could possibly even be done early, i.e before placing the
375	request on the queue, rather than construct the command on the fly in the
376	driver while servicing the request queue when it may affect latencies in
377	interrupt context or responsiveness in general. One way to add early
378	pre-building would be to do it whenever we fail to merge on a request.
379	Now REQ_NOMERGE is set in the request flags to skip this one in the future,
380	which means that it will not change before we feed it to the device. So
381	the pre-builder hook can be invoked there.
382
383
384	2. Flexible and generic but minimalist i/o structure/descriptor
385	===============================================================
386
387	2.1 Reason for a new structure and requirements addressed
388	---------------------------------------------------------
389
390	Prior to 2.5, buffer heads were used as the unit of i/o at the generic block
391	layer, and the low level request structure was associated with a chain of
392	buffer heads for a contiguous i/o request. This led to certain inefficiencies
393	when it came to large i/o requests and readv/writev style operations, as it
394	forced such requests to be broken up into small chunks before being passed
395	on to the generic block layer, only to be merged by the i/o scheduler
396	when the underlying device was capable of handling the i/o in one shot.
397	Also, using the buffer head as an i/o structure for i/os that didn't originate
398	from the buffer cache unnecessarily added to the weight of the descriptors
399	which were generated for each such chunk.
400
401	The following were some of the goals and expectations considered in the
402	redesign of the block i/o data structure in 2.5.
403
404	1. Should be appropriate as a descriptor for both raw and buffered i/o -
405	avoid cache related fields which are irrelevant in the direct/page i/o path,
406	or filesystem block size alignment restrictions which may not be relevant
407	for raw i/o.
408	2. Ability to represent high-memory buffers (which do not have a virtual
409	address mapping in kernel address space).
410	3. Ability to represent large i/os w/o unnecessarily breaking them up (i.e
411	greater than PAGE_SIZE chunks in one shot)
412	4. At the same time, ability to retain independent identity of i/os from
413	different sources or i/o units requiring individual completion (e.g. for
414	latency reasons)
415	5. Ability to represent an i/o involving multiple physical memory segments
416	(including non-page aligned page fragments, as specified via readv/writev)
417	without unnecessarily breaking it up, if the underlying device is capable of
418	handling it.
419	6. Preferably should be based on a memory descriptor structure that can be
420	passed around different types of subsystems or layers, maybe even
421	networking, without duplication or extra copies of data/descriptor fields
422	themselves in the process
423	7. Ability to handle the possibility of splits/merges as the structure passes
424	through layered drivers (lvm, md, evms), with minimal overhead.
425
426	The solution was to define a new structure (bio) for the block layer,
427	instead of using the buffer head structure (bh) directly, the idea being
428	avoidance of some associated baggage and limitations. The bio structure
429	is uniformly used for all i/o at the block layer ; it forms a part of the
430	bh structure for buffered i/o, and in the case of raw/direct i/o kiobufs are
431	mapped to bio structures.
432
433	2.2 The bio struct
434	------------------
435
436	The bio structure uses a vector representation pointing to an array of tuples
437	of <page, offset, len> to describe the i/o buffer, and has various other
438	fields describing i/o parameters and state that needs to be maintained for
439	performing the i/o.
440
441	Notice that this representation means that a bio has no virtual address
442	mapping at all (unlike buffer heads).
443
444	::
445
446	struct bio_vec {
447	struct page *bv_page;
448	unsigned short bv_len;
449	unsigned short bv_offset;
450	};
451
452	/*
453	* main unit of I/O for the block layer and lower layers (ie drivers)
454	*/
455	struct bio {
456	struct bio bi_next; / request queue link */
457	struct block_device bi_bdev; / target device */
458	unsigned long bi_flags; /* status, command, etc */
459	unsigned long bi_opf; /* low bits: r/w, high: priority */
460
461	unsigned int bi_vcnt; /* how may bio_vec's */
462	struct bvec_iter bi_iter; /* current index into bio_vec array */
463
464	unsigned int bi_size; /* total size in bytes */
465	unsigned short bi_hw_segments; /* segments after DMA remapping */
466	unsigned int bi_max; /* max bio_vecs we can hold
467	used as index into pool */
468	struct bio_vec bi_io_vec; / the actual vec list */
469	bio_end_io_t bi_end_io; / bi_end_io (bio) */
470	atomic_t bi_cnt; /* pin count: free when it hits zero */
471	void *bi_private;
472	};
473
474	With this multipage bio design:
475
476	- Large i/os can be sent down in one go using a bio_vec list consisting
477	of an array of <page, offset, len> fragments (similar to the way fragments
478	are represented in the zero-copy network code)
479	- Splitting of an i/o request across multiple devices (as in the case of
480	lvm or raid) is achieved by cloning the bio (where the clone points to
481	the same bi_io_vec array, but with the index and size accordingly modified)
482	- A linked list of bios is used as before for unrelated merges [#]_ - this
483	avoids reallocs and makes independent completions easier to handle.
484	- Code that traverses the req list can find all the segments of a bio
485	by using rq_for_each_segment. This handles the fact that a request
486	has multiple bios, each of which can have multiple segments.
487	- Drivers which can't process a large bio in one shot can use the bi_iter
488	field to keep track of the next bio_vec entry to process.
489	(e.g a 1MB bio_vec needs to be handled in max 128kB chunks for IDE)
490	[TBD: Should preferably also have a bi_voffset and bi_vlen to avoid modifying
491	bi_offset an len fields]
492
493	.. [#]
494
495	unrelated merges -- a request ends up containing two or more bios that
496	didn't originate from the same place.
497
498	bi_end_io() i/o callback gets called on i/o completion of the entire bio.
499
500	At a lower level, drivers build a scatter gather list from the merged bios.
501	The scatter gather list is in the form of an array of <page, offset, len>
502	entries with their corresponding dma address mappings filled in at the
503	appropriate time. As an optimization, contiguous physical pages can be
504	covered by a single entry where <page> refers to the first page and <len>
505	covers the range of pages (up to 16 contiguous pages could be covered this
506	way). There is a helper routine (blk_rq_map_sg) which drivers can use to build
507	the sg list.
508
509	Note: Right now the only user of bios with more than one page is ll_rw_kio,
510	which in turn means that only raw I/O uses it (direct i/o may not work
511	right now). The intent however is to enable clustering of pages etc to
512	become possible. The pagebuf abstraction layer from SGI also uses multi-page
513	bios, but that is currently not included in the stock development kernels.
514	The same is true of Andrew Morton's work-in-progress multipage bio writeout
515	and readahead patches.
516
517	2.3 Changes in the Request Structure
518	------------------------------------
519
520	The request structure is the structure that gets passed down to low level
521	drivers. The block layer make_request function builds up a request structure,
522	places it on the queue and invokes the drivers request_fn. The driver makes
523	use of block layer helper routine elv_next_request to pull the next request
524	off the queue. Control or diagnostic functions might bypass block and directly
525	invoke underlying driver entry points passing in a specially constructed
526	request structure.
527
528	Only some relevant fields (mainly those which changed or may be referred
529	to in some of the discussion here) are listed below, not necessarily in
530	the order in which they occur in the structure (see include/linux/blkdev.h)
531	Refer to Documentation/block/request.rst for details about all the request
532	structure fields and a quick reference about the layers which are
533	supposed to use or modify those fields::
534
535	struct request {
536	struct list_head queuelist; /* Not meant to be directly accessed by
537	the driver.
538	Used by q->elv_next_request_fn
539	rq->queue is gone
540	*/
541	.
542	.
543	unsigned char cmd[16]; /* prebuilt command data block */
544	unsigned long flags; /* also includes earlier rq->cmd settings */
545	.
546	.
547	sector_t sector; /* this field is now of type sector_t instead of int
548	preparation for 64 bit sectors */
549	.
550	.
551
552	/* Number of scatter-gather DMA addr+len pairs after
553	* physical address coalescing is performed.
554	*/
555	unsigned short nr_phys_segments;
556
557	/* Number of scatter-gather addr+len pairs after
558	* physical and DMA remapping hardware coalescing is performed.
559	* This is the number of scatter-gather entries the driver
560	* will actually have to deal with after DMA mapping is done.
561	*/
562	unsigned short nr_hw_segments;
563
564	/* Various sector counts */
565	unsigned long nr_sectors; /* no. of sectors left: driver modifiable */
566	unsigned long hard_nr_sectors; /* block internal copy of above */
567	unsigned int current_nr_sectors; /* no. of sectors left in the
568	current segment:driver modifiable */
569	unsigned long hard_cur_sectors; /* block internal copy of the above */
570	.
571	.
572	int tag; /* command tag associated with request */
573	void special; / same as before */
574	char buffer; / valid only for low memory buffers up to
575	current_nr_sectors */
576	.
577	.
578	struct bio bio, biotail; /* bio list instead of bh */
579	struct request_list *rl;
580	}
581
582	See the req_ops and req_flag_bits definitions for an explanation of the various
583	flags available. Some bits are used by the block layer or i/o scheduler.
584
585	The behaviour of the various sector counts are almost the same as before,
586	except that since we have multi-segment bios, current_nr_sectors refers
587	to the numbers of sectors in the current segment being processed which could
588	be one of the many segments in the current bio (i.e i/o completion unit).
589	The nr_sectors value refers to the total number of sectors in the whole
590	request that remain to be transferred (no change). The purpose of the
591	hard_xxx values is for block to remember these counts every time it hands
592	over the request to the driver. These values are updated by block on
593	end_that_request_first, i.e. every time the driver completes a part of the
594	transfer and invokes block end*request helpers to mark this. The
595	driver should not modify these values. The block layer sets up the
596	nr_sectors and current_nr_sectors fields (based on the corresponding
597	hard_xxx values and the number of bytes transferred) and updates it on
598	every transfer that invokes end_that_request_first. It does the same for the
599	buffer, bio, bio->bi_iter fields too.
600
601	The buffer field is just a virtual address mapping of the current segment
602	of the i/o buffer in cases where the buffer resides in low-memory. For high
603	memory i/o, this field is not valid and must not be used by drivers.
604
605	Code that sets up its own request structures and passes them down to
606	a driver needs to be careful about interoperation with the block layer helper
607	functions which the driver uses. (Section 1.3)
608
609	3. Using bios
610	=============
611
612	3.1 Setup/Teardown
613	------------------
614
615	There are routines for managing the allocation, and reference counting, and
616	freeing of bios (bio_alloc, bio_get, bio_put).
617
618	This makes use of Ingo Molnar's mempool implementation, which enables
619	subsystems like bio to maintain their own reserve memory pools for guaranteed
620	deadlock-free allocations during extreme VM load. For example, the VM
621	subsystem makes use of the block layer to writeout dirty pages in order to be
622	able to free up memory space, a case which needs careful handling. The
623	allocation logic draws from the preallocated emergency reserve in situations
624	where it cannot allocate through normal means. If the pool is empty and it
625	can wait, then it would trigger action that would help free up memory or
626	replenish the pool (without deadlocking) and wait for availability in the pool.
627	If it is in IRQ context, and hence not in a position to do this, allocation
628	could fail if the pool is empty. In general mempool always first tries to
629	perform allocation without having to wait, even if it means digging into the
630	pool as long it is not less that 50% full.
631
632	On a free, memory is released to the pool or directly freed depending on
633	the current availability in the pool. The mempool interface lets the
634	subsystem specify the routines to be used for normal alloc and free. In the
635	case of bio, these routines make use of the standard slab allocator.
636
637	The caller of bio_alloc is expected to taken certain steps to avoid
638	deadlocks, e.g. avoid trying to allocate more memory from the pool while
639	already holding memory obtained from the pool.
640
641	::
642
643	[TBD: This is a potential issue, though a rare possibility
644	in the bounce bio allocation that happens in the current code, since
645	it ends up allocating a second bio from the same pool while
646	holding the original bio ]
647
648	Memory allocated from the pool should be released back within a limited
649	amount of time (in the case of bio, that would be after the i/o is completed).
650	This ensures that if part of the pool has been used up, some work (in this
651	case i/o) must already be in progress and memory would be available when it
652	is over. If allocating from multiple pools in the same code path, the order
653	or hierarchy of allocation needs to be consistent, just the way one deals
654	with multiple locks.
655
656	The bio_alloc routine also needs to allocate the bio_vec_list (bvec_alloc())
657	for a non-clone bio. There are the 6 pools setup for different size biovecs,
658	so bio_alloc(gfp_mask, nr_iovecs) will allocate a vec_list of the
659	given size from these slabs.
660
661	The bio_get() routine may be used to hold an extra reference on a bio prior
662	to i/o submission, if the bio fields are likely to be accessed after the
663	i/o is issued (since the bio may otherwise get freed in case i/o completion
664	happens in the meantime).
665
666	The bio_clone_fast() routine may be used to duplicate a bio, where the clone
667	shares the bio_vec_list with the original bio (i.e. both point to the
668	same bio_vec_list). This would typically be used for splitting i/o requests
669	in lvm or md.
670
671	3.2 Generic bio helper Routines
672	-------------------------------
673
674	3.2.1 Traversing segments and completion units in a request
675	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
676
677	The macro rq_for_each_segment() should be used for traversing the bios
678	in the request list (drivers should avoid directly trying to do it
679	themselves). Using these helpers should also make it easier to cope
680	with block changes in the future.
681
682	::
683
684	struct req_iterator iter;
685	rq_for_each_segment(bio_vec, rq, iter)
686	/* bio_vec is now current segment */
687
688	I/O completion callbacks are per-bio rather than per-segment, so drivers
689	that traverse bio chains on completion need to keep that in mind. Drivers
690	which don't make a distinction between segments and completion units would
691	need to be reorganized to support multi-segment bios.
692
693	3.2.2 Setting up DMA scatterlists
694	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
695
696	The blk_rq_map_sg() helper routine would be used for setting up scatter
697	gather lists from a request, so a driver need not do it on its own.
698
699	nr_segments = blk_rq_map_sg(q, rq, scatterlist);
700
701	The helper routine provides a level of abstraction which makes it easier
702	to modify the internals of request to scatterlist conversion down the line
703	without breaking drivers. The blk_rq_map_sg routine takes care of several
704	things like collapsing physically contiguous segments (if QUEUE_FLAG_CLUSTER
705	is set) and correct segment accounting to avoid exceeding the limits which
706	the i/o hardware can handle, based on various queue properties.
707
708	- Prevents a clustered segment from crossing a 4GB mem boundary
709	- Avoids building segments that would exceed the number of physical
710	memory segments that the driver can handle (phys_segments) and the
711	number that the underlying hardware can handle at once, accounting for
712	DMA remapping (hw_segments) (i.e. IOMMU aware limits).
713
714	Routines which the low level driver can use to set up the segment limits:
715
716	blk_queue_max_hw_segments() : Sets an upper limit of the maximum number of
717	hw data segments in a request (i.e. the maximum number of address/length
718	pairs the host adapter can actually hand to the device at once)
719
720	blk_queue_max_phys_segments() : Sets an upper limit on the maximum number
721	of physical data segments in a request (i.e. the largest sized scatter list
722	a driver could handle)
723
724	3.2.3 I/O completion
725	^^^^^^^^^^^^^^^^^^^^
726
727	The existing generic block layer helper routines end_request,
728	end_that_request_first and end_that_request_last can be used for i/o
729	completion (and setting things up so the rest of the i/o or the next
730	request can be kicked of) as before. With the introduction of multi-page
731	bio support, end_that_request_first requires an additional argument indicating
732	the number of sectors completed.
733
734	3.2.4 Implications for drivers that do not interpret bios
735	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
736
737	(don't handle multiple segments)
738
739	Drivers that do not interpret bios e.g those which do not handle multiple
740	segments and do not support i/o into high memory addresses (require bounce
741	buffers) and expect only virtually mapped buffers, can access the rq->buffer
742	field. As before the driver should use current_nr_sectors to determine the
743	size of remaining data in the current segment (that is the maximum it can
744	transfer in one go unless it interprets segments), and rely on the block layer
745	end_request, or end_that_request_first/last to take care of all accounting
746	and transparent mapping of the next bio segment when a segment boundary
747	is crossed on completion of a transfer. (The endrequest functions should
748	be used if only if the request has come down from block/bio path, not for
749	direct access requests which only specify rq->buffer without a valid rq->bio)
750
751	3.3 I/O Submission
752	------------------
753
754	The routine submit_bio() is used to submit a single io. Higher level i/o
755	routines make use of this:
756
757	(a) Buffered i/o:
758
759	The routine submit_bh() invokes submit_bio() on a bio corresponding to the
760	bh, allocating the bio if required. ll_rw_block() uses submit_bh() as before.
761
762	(b) Kiobuf i/o (for raw/direct i/o):
763
764	The ll_rw_kio() routine breaks up the kiobuf into page sized chunks and
765	maps the array to one or more multi-page bios, issuing submit_bio() to
766	perform the i/o on each of these.
767
768	The embedded bh array in the kiobuf structure has been removed and no
769	preallocation of bios is done for kiobufs. [The intent is to remove the
770	blocks array as well, but it's currently in there to kludge around direct i/o.]
771	Thus kiobuf allocation has switched back to using kmalloc rather than vmalloc.
772
773	Todo/Observation:
774
775	A single kiobuf structure is assumed to correspond to a contiguous range
776	of data, so brw_kiovec() invokes ll_rw_kio for each kiobuf in a kiovec.
777	So right now it wouldn't work for direct i/o on non-contiguous blocks.
778	This is to be resolved. The eventual direction is to replace kiobuf
779	by kvec's.
780
781	Badari Pulavarty has a patch to implement direct i/o correctly using
782	bio and kvec.
783
784
785	(c) Page i/o:
786
787	Todo/Under discussion:
788
789	Andrew Morton's multi-page bio patches attempt to issue multi-page
790	writeouts (and reads) from the page cache, by directly building up
791	large bios for submission completely bypassing the usage of buffer
792	heads. This work is still in progress.
793
794	Christoph Hellwig had some code that uses bios for page-io (rather than
795	bh). This isn't included in bio as yet. Christoph was also working on a
796	design for representing virtual/real extents as an entity and modifying
797	some of the address space ops interfaces to utilize this abstraction rather
798	than buffer_heads. (This is somewhat along the lines of the SGI XFS pagebuf
799	abstraction, but intended to be as lightweight as possible).
800
801	(d) Direct access i/o:
802
803	Direct access requests that do not contain bios would be submitted differently
804	as discussed earlier in section 1.3.
805
806	Aside:
807
808	Kvec i/o:
809
810	Ben LaHaise's aio code uses a slightly different structure instead
811	of kiobufs, called a kvec_cb. This contains an array of <page, offset, len>
812	tuples (very much like the networking code), together with a callback function
813	and data pointer. This is embedded into a brw_cb structure when passed
814	to brw_kvec_async().
815
816	Now it should be possible to directly map these kvecs to a bio. Just as while
817	cloning, in this case rather than PRE_BUILT bio_vecs, we set the bi_io_vec
818	array pointer to point to the veclet array in kvecs.
819
820	TBD: In order for this to work, some changes are needed in the way multi-page
821	bios are handled today. The values of the tuples in such a vector passed in
822	from higher level code should not be modified by the block layer in the course
823	of its request processing, since that would make it hard for the higher layer
824	to continue to use the vector descriptor (kvec) after i/o completes. Instead,
825	all such transient state should either be maintained in the request structure,
826	and passed on in some way to the endio completion routine.
827
828
829	4. The I/O scheduler
830	====================
831
832	I/O scheduler, a.k.a. elevator, is implemented in two layers. Generic dispatch
833	queue and specific I/O schedulers. Unless stated otherwise, elevator is used
834	to refer to both parts and I/O scheduler to specific I/O schedulers.
835
836	Block layer implements generic dispatch queue in `block/*.c`.
837	The generic dispatch queue is responsible for requeueing, handling non-fs
838	requests and all other subtleties.
839
840	Specific I/O schedulers are responsible for ordering normal filesystem
841	requests. They can also choose to delay certain requests to improve
842	throughput or whatever purpose. As the plural form indicates, there are
843	multiple I/O schedulers. They can be built as modules but at least one should
844	be built inside the kernel. Each queue can choose different one and can also
845	change to another one dynamically.
846
847	A block layer call to the i/o scheduler follows the convention elv_xxx(). This
848	calls elevator_xxx_fn in the elevator switch (block/elevator.c). Oh, xxx
849	and xxx might not match exactly, but use your imagination. If an elevator
850	doesn't implement a function, the switch does nothing or some minimal house
851	keeping work.
852
853	4.1. I/O scheduler API
854	----------------------
855
856	The functions an elevator may implement are: (* are mandatory)
857
858	=============================== ================================================
859	elevator_merge_fn called to query requests for merge with a bio
860
861	elevator_merge_req_fn called when two requests get merged. the one
862	which gets merged into the other one will be
863	never seen by I/O scheduler again. IOW, after
864	being merged, the request is gone.
865
866	elevator_merged_fn called when a request in the scheduler has been
867	involved in a merge. It is used in the deadline
868	scheduler for example, to reposition the request
869	if its sorting order has changed.
870
871	elevator_allow_merge_fn called whenever the block layer determines
872	that a bio can be merged into an existing
873	request safely. The io scheduler may still
874	want to stop a merge at this point if it
875	results in some sort of conflict internally,
876	this hook allows it to do that. Note however
877	that two requests can still be merged at later
878	time. Currently the io scheduler has no way to
879	prevent that. It can only learn about the fact
880	from elevator_merge_req_fn callback.
881
882	elevator_dispatch_fn* fills the dispatch queue with ready requests.
883	I/O schedulers are free to postpone requests by
884	not filling the dispatch queue unless @force
885	is non-zero. Once dispatched, I/O schedulers
886	are not allowed to manipulate the requests -
887	they belong to generic dispatch queue.
888
889	elevator_add_req_fn* called to add a new request into the scheduler
890
891	elevator_former_req_fn
892	elevator_latter_req_fn These return the request before or after the
893	one specified in disk sort order. Used by the
894	block layer to find merge possibilities.
895
896	elevator_completed_req_fn called when a request is completed.
897
898	elevator_set_req_fn
899	elevator_put_req_fn Must be used to allocate and free any elevator
900	specific storage for a request.
901
902	elevator_activate_req_fn Called when device driver first sees a request.
903	I/O schedulers can use this callback to
904	determine when actual execution of a request
905	starts.
906	elevator_deactivate_req_fn Called when device driver decides to delay
907	a request by requeueing it.
908
909	elevator_init_fn*
910	elevator_exit_fn Allocate and free any elevator specific storage
911	for a queue.
912	=============================== ================================================
913
914	4.2 Request flows seen by I/O schedulers
915	----------------------------------------
916
917	All requests seen by I/O schedulers strictly follow one of the following three
918	flows.
919
920	set_req_fn ->
921
922	i. add_req_fn -> (merged_fn ->)* -> dispatch_fn -> activate_req_fn ->
923	(deactivate_req_fn -> activate_req_fn ->)* -> completed_req_fn
924	ii. add_req_fn -> (merged_fn ->)* -> merge_req_fn
925	iii. [none]
926
927	-> put_req_fn
928
929	4.3 I/O scheduler implementation
930	--------------------------------
931
932	The generic i/o scheduler algorithm attempts to sort/merge/batch requests for
933	optimal disk scan and request servicing performance (based on generic
934	principles and device capabilities), optimized for:
935
936	i. improved throughput
937	ii. improved latency
938	iii. better utilization of h/w & CPU time
939
940	Characteristics:
941
942	i. Binary tree
943	AS and deadline i/o schedulers use red black binary trees for disk position
944	sorting and searching, and a fifo linked list for time-based searching. This
945	gives good scalability and good availability of information. Requests are
946	almost always dispatched in disk sort order, so a cache is kept of the next
947	request in sort order to prevent binary tree lookups.
948
949	This arrangement is not a generic block layer characteristic however, so
950	elevators may implement queues as they please.
951
952	ii. Merge hash
953	AS and deadline use a hash table indexed by the last sector of a request. This
954	enables merging code to quickly look up "back merge" candidates, even when
955	multiple I/O streams are being performed at once on one disk.
956
957	"Front merges", a new request being merged at the front of an existing request,
958	are far less common than "back merges" due to the nature of most I/O patterns.
959	Front merges are handled by the binary trees in AS and deadline schedulers.
960
961	iii. Plugging the queue to batch requests in anticipation of opportunities for
962	merge/sort optimizations
963
964	Plugging is an approach that the current i/o scheduling algorithm resorts to so
965	that it collects up enough requests in the queue to be able to take
966	advantage of the sorting/merging logic in the elevator. If the
967	queue is empty when a request comes in, then it plugs the request queue
968	(sort of like plugging the bath tub of a vessel to get fluid to build up)
969	till it fills up with a few more requests, before starting to service
970	the requests. This provides an opportunity to merge/sort the requests before
971	passing them down to the device. There are various conditions when the queue is
972	unplugged (to open up the flow again), either through a scheduled task or
973	could be on demand. For example wait_on_buffer sets the unplugging going
974	through sync_buffer() running blk_run_address_space(mapping). Or the caller
975	can do it explicity through blk_unplug(bdev). So in the read case,
976	the queue gets explicitly unplugged as part of waiting for completion on that
977	buffer.
978
979	Aside:
980	This is kind of controversial territory, as it's not clear if plugging is
981	always the right thing to do. Devices typically have their own queues,
982	and allowing a big queue to build up in software, while letting the device be
983	idle for a while may not always make sense. The trick is to handle the fine
984	balance between when to plug and when to open up. Also now that we have
985	multi-page bios being queued in one shot, we may not need to wait to merge
986	a big request from the broken up pieces coming by.
987
988	4.4 I/O contexts
989	----------------
990
991	I/O contexts provide a dynamically allocated per process data area. They may
992	be used in I/O schedulers, and in the block layer (could be used for IO statis,
993	priorities for example). See `*io_context` in block/ll_rw_blk.c, and as-iosched.c
994	for an example of usage in an i/o scheduler.
995
996
997	5. Scalability related changes
998	==============================
999
1000	5.1 Granular Locking: io_request_lock replaced by a per-queue lock
1001	------------------------------------------------------------------
1002
1003	The global io_request_lock has been removed as of 2.5, to avoid
1004	the scalability bottleneck it was causing, and has been replaced by more
1005	granular locking. The request queue structure has a pointer to the
1006	lock to be used for that queue. As a result, locking can now be
1007	per-queue, with a provision for sharing a lock across queues if
1008	necessary (e.g the scsi layer sets the queue lock pointers to the
1009	corresponding adapter lock, which results in a per host locking
1010	granularity). The locking semantics are the same, i.e. locking is
1011	still imposed by the block layer, grabbing the lock before
1012	request_fn execution which it means that lots of older drivers
1013	should still be SMP safe. Drivers are free to drop the queue
1014	lock themselves, if required. Drivers that explicitly used the
1015	io_request_lock for serialization need to be modified accordingly.
1016	Usually it's as easy as adding a global lock::
1017
1018	static DEFINE_SPINLOCK(my_driver_lock);
1019
1020	and passing the address to that lock to blk_init_queue().
1021
1022	5.2 64 bit sector numbers (sector_t prepares for 64 bit support)
1023	----------------------------------------------------------------
1024
1025	The sector number used in the bio structure has been changed to sector_t,
1026	which could be defined as 64 bit in preparation for 64 bit sector support.
1027
1028	6. Other Changes/Implications
1029	=============================
1030
1031	6.1 Partition re-mapping handled by the generic block layer
1032	-----------------------------------------------------------
1033
1034	In 2.5 some of the gendisk/partition related code has been reorganized.
1035	Now the generic block layer performs partition-remapping early and thus
1036	provides drivers with a sector number relative to whole device, rather than
1037	having to take partition number into account in order to arrive at the true
1038	sector number. The routine blk_partition_remap() is invoked by
1039	submit_bio_noacct even before invoking the queue specific ->submit_bio,
1040	so the i/o scheduler also gets to operate on whole disk sector numbers. This
1041	should typically not require changes to block drivers, it just never gets
1042	to invoke its own partition sector offset calculations since all bios
1043	sent are offset from the beginning of the device.
1044
1045
1046	7. A Few Tips on Migration of older drivers
1047	===========================================
1048
1049	Old-style drivers that just use CURRENT and ignores clustered requests,
1050	may not need much change. The generic layer will automatically handle
1051	clustered requests, multi-page bios, etc for the driver.
1052
1053	For a low performance driver or hardware that is PIO driven or just doesn't
1054	support scatter-gather changes should be minimal too.
1055
1056	The following are some points to keep in mind when converting old drivers
1057	to bio.
1058
1059	Drivers should use elv_next_request to pick up requests and are no longer
1060	supposed to handle looping directly over the request list.
1061	(struct request->queue has been removed)
1062
1063	Now end_that_request_first takes an additional number_of_sectors argument.
1064	It used to handle always just the first buffer_head in a request, now
1065	it will loop and handle as many sectors (on a bio-segment granularity)
1066	as specified.
1067
1068	Now bh->b_end_io is replaced by bio->bi_end_io, but most of the time the
1069	right thing to use is bio_endio(bio) instead.
1070
1071	If the driver is dropping the io_request_lock from its request_fn strategy,
1072	then it just needs to replace that with q->queue_lock instead.
1073
1074	As described in Sec 1.1, drivers can set max sector size, max segment size
1075	etc per queue now. Drivers that used to define their own merge functions i
1076	to handle things like this can now just use the blk_queue_* functions at
1077	blk_init_queue time.
1078
1079	Drivers no longer have to map a {partition, sector offset} into the
1080	correct absolute location anymore, this is done by the block layer, so
1081	where a driver received a request ala this before::
1082
1083	rq->rq_dev = mk_kdev(3, 5); /* /dev/hda5 */
1084	rq->sector = 0; /* first sector on hda5 */
1085
1086	it will now see::
1087
1088	rq->rq_dev = mk_kdev(3, 0); /* /dev/hda */
1089	rq->sector = 123128; /* offset from start of disk */
1090
1091	As mentioned, there is no virtual mapping of a bio. For DMA, this is
1092	not a problem as the driver probably never will need a virtual mapping.
1093	Instead it needs a bus mapping (dma_map_page for a single segment or
1094	use dma_map_sg for scatter gather) to be able to ship it to the driver. For
1095	PIO drivers (or drivers that need to revert to PIO transfer once in a
1096	while (IDE for example)), where the CPU is doing the actual data
1097	transfer a virtual mapping is needed. If the driver supports highmem I/O,
1098	(Sec 1.1, (ii) ) it needs to use kmap_atomic or similar to temporarily map
1099	a bio into the virtual address space.
1100
1101
1102	8. Prior/Related/Impacted patches
1103	=================================
1104
1105	8.1. Earlier kiobuf patches (sct/axboe/chait/hch/mkp)
1106	-----------------------------------------------------
1107
1108	- orig kiobuf & raw i/o patches (now in 2.4 tree)
1109	- direct kiobuf based i/o to devices (no intermediate bh's)
1110	- page i/o using kiobuf
1111	- kiobuf splitting for lvm (mkp)
1112	- elevator support for kiobuf request merging (axboe)
1113
1114	8.2. Zero-copy networking (Dave Miller)
1115	---------------------------------------
1116
1117	8.3. SGI XFS - pagebuf patches - use of kiobufs
1118	-----------------------------------------------
1119	8.4. Multi-page pioent patch for bio (Christoph Hellwig)
1120	--------------------------------------------------------
1121	8.5. Direct i/o implementation (Andrea Arcangeli) since 2.4.10-pre11
1122	--------------------------------------------------------------------
1123	8.6. Async i/o implementation patch (Ben LaHaise)
1124	-------------------------------------------------
1125	8.7. EVMS layering design (IBM EVMS team)
1126	-----------------------------------------
1127	8.8. Larger page cache size patch (Ben LaHaise) and Large page size (Daniel Phillips)
1128	-------------------------------------------------------------------------------------
1129
1130	=> larger contiguous physical memory buffers
1131
1132	8.9. VM reservations patch (Ben LaHaise)
1133	----------------------------------------
1134	8.10. Write clustering patches ? (Marcelo/Quintela/Riel ?)
1135	----------------------------------------------------------
1136	8.11. Block device in page cache patch (Andrea Archangeli) - now in 2.4.10+
1137	---------------------------------------------------------------------------
1138	8.12. Multiple block-size transfers for faster raw i/o (Shailabh Nagar, Badari)
1139	-------------------------------------------------------------------------------
1140	8.13 Priority based i/o scheduler - prepatches (Arjan van de Ven)
1141	------------------------------------------------------------------
1142	8.14 IDE Taskfile i/o patch (Andre Hedrick)
1143	--------------------------------------------
1144	8.15 Multi-page writeout and readahead patches (Andrew Morton)
1145	---------------------------------------------------------------
1146	8.16 Direct i/o patches for 2.5 using kvec and bio (Badari Pulavarthy)
1147	-----------------------------------------------------------------------
1148
1149	9. Other References
1150	===================
1151
1152	9.1 The Splice I/O Model
1153	------------------------
1154
1155	Larry McVoy (and subsequent discussions on lkml, and Linus' comments - Jan 2001
1156
1157	9.2 Discussions about kiobuf and bh design
1158	------------------------------------------
1159
1160	On lkml between sct, linus, alan et al - Feb-March 2001 (many of the
1161	initial thoughts that led to bio were brought up in this discussion thread)
1162
1163	9.3 Discussions on mempool on lkml - Dec 2001.
1164	----------------------------------------------

Documentation/block/capability.rst+1−1

@@ -7,4 +7,4 @@ This file documents the sysfs file ``block/<disk>/capability``. · 6 unmodified lines
7	``capability`` is a bitfield, printed in hexadecimal, indicating which	7	``capability`` is a bitfield, printed in hexadecimal, indicating which
8	capabilities a specific block device supports:	8	capabilities a specific block device supports:
9		9
10	.. kernel-doc:: include/linux/genhd.h	10	.. kernel-doc:: include/linux/blkdev.h

Documentation/block/index.rst+0−1

@@ -8,7 +8,6 @@ Block · 7 unmodified lines
8	:maxdepth: 1	8	:maxdepth: 1
9		9
10	bfq-iosched	10	bfq-iosched
11	biodoc
12	biovecs	11	biovecs
13	blk-mq	12	blk-mq
14	capability	13	capability

MAINTAINERS+1−0

@@ -3441,6 +3441,7 @@ F: Documentation/ABI/stable/sysfs-block · 3440 unmodified lines
3441	F: Documentation/block/	3441	F: Documentation/block/
3442	F: block/	3442	F: block/
3443	F: drivers/block/	3443	F: drivers/block/
		3444	F: include/linux/bio.h
3444	F: include/linux/blk*	3445	F: include/linux/blk*
3445	F: kernel/trace/blktrace.c	3446	F: kernel/trace/blktrace.c
3446	F: lib/sbitmap.c	3447	F: lib/sbitmap.c

arch/m68k/atari/stdma.c+0−1

@@ -30,7 +30,6 @@ · 29 unmodified lines
30		30
31	#include <linux/types.h>	31	#include <linux/types.h>
32	#include <linux/kdev_t.h>	32	#include <linux/kdev_t.h>
33	#include <linux/genhd.h>
34	#include <linux/sched.h>	33	#include <linux/sched.h>
35	#include <linux/init.h>	34	#include <linux/init.h>
36	#include <linux/interrupt.h>	35	#include <linux/interrupt.h>

arch/m68k/bvme6000/config.c+0−1

@@ -23,7 +23,6 @@ · 22 unmodified lines
23	#include <linux/linkage.h>	23	#include <linux/linkage.h>
24	#include <linux/init.h>	24	#include <linux/init.h>
25	#include <linux/major.h>	25	#include <linux/major.h>
26	#include <linux/genhd.h>
27	#include <linux/rtc.h>	26	#include <linux/rtc.h>
28	#include <linux/interrupt.h>	27	#include <linux/interrupt.h>
29	#include <linux/bcd.h>	28	#include <linux/bcd.h>

arch/m68k/emu/nfblock.c+0−1

@@ -13,7 +13,6 @@ · 12 unmodified lines
13	#include <linux/kernel.h>	13	#include <linux/kernel.h>
14	#include <linux/errno.h>	14	#include <linux/errno.h>
15	#include <linux/types.h>	15	#include <linux/types.h>
16	#include <linux/genhd.h>
17	#include <linux/blkdev.h>	16	#include <linux/blkdev.h>
18	#include <linux/hdreg.h>	17	#include <linux/hdreg.h>
19	#include <linux/slab.h>	18	#include <linux/slab.h>

arch/m68k/kernel/setup_mm.c+0−1

@@ -16,7 +16,6 @@ · 15 unmodified lines
16	#include <linux/interrupt.h>	16	#include <linux/interrupt.h>
17	#include <linux/fs.h>	17	#include <linux/fs.h>
18	#include <linux/console.h>	18	#include <linux/console.h>
19	#include <linux/genhd.h>
20	#include <linux/errno.h>	19	#include <linux/errno.h>
21	#include <linux/string.h>	20	#include <linux/string.h>
22	#include <linux/init.h>	21	#include <linux/init.h>

arch/m68k/mvme147/config.c+0−1

@@ -22,7 +22,6 @@ · 21 unmodified lines
22	#include <linux/linkage.h>	22	#include <linux/linkage.h>
23	#include <linux/init.h>	23	#include <linux/init.h>
24	#include <linux/major.h>	24	#include <linux/major.h>
25	#include <linux/genhd.h>
26	#include <linux/rtc.h>	25	#include <linux/rtc.h>
27	#include <linux/interrupt.h>	26	#include <linux/interrupt.h>
28		27

arch/m68k/mvme16x/config.c+0−1

@@ -24,7 +24,6 @@ · 23 unmodified lines
24	#include <linux/linkage.h>	24	#include <linux/linkage.h>
25	#include <linux/init.h>	25	#include <linux/init.h>
26	#include <linux/major.h>	26	#include <linux/major.h>
27	#include <linux/genhd.h>
28	#include <linux/rtc.h>	27	#include <linux/rtc.h>
29	#include <linux/interrupt.h>	28	#include <linux/interrupt.h>
30	#include <linux/module.h>	29	#include <linux/module.h>

block/Kconfig+13−0

@@ -26,6 +26,16 @@ menuconfig BLOCK · 25 unmodified lines
26		26
27	if BLOCK	27	if BLOCK
28		28
		29	config BLOCK_LEGACY_AUTOLOAD
		30	bool "Legacy autoloading support"
		31	default y
		32	help
		33	Enable loading modules and creating block device instances based on
		34	accesses through their device special file. This is a historic Linux
		35	feature and makes no sense in a udev world where device files are
		36	created on demand, but scripts that manually create device nodes and
		37	then call losetup might rely on this behavior.
		38
29	config BLK_RQ_ALLOC_TIME	39	config BLK_RQ_ALLOC_TIME
30	bool	40	bool
31		41
@@ -218,6 +228,9 @@ config BLK_PM · 186 unmodified lines
218	config BLOCK_HOLDER_DEPRECATED	228	config BLOCK_HOLDER_DEPRECATED
219	bool	229	bool
220		230
		231	config BLK_MQ_STACKING
		232	bool
		233
221	source "block/Kconfig.iosched"	234	source "block/Kconfig.iosched"
222		235
223	endif # BLOCK	236	endif # BLOCK

block/Makefile+2−1

@@ -36,6 +36,7 @@ obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o · 35 unmodified lines
36	obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o	36	obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o
37	obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o	37	obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o
38	obj-$(CONFIG_BLK_PM) += blk-pm.o	38	obj-$(CONFIG_BLK_PM) += blk-pm.o
39	obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += blk-crypto.o blk-crypto-profile.o	39	obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += blk-crypto.o blk-crypto-profile.o \
		40	blk-crypto-sysfs.o
40	obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) += blk-crypto-fallback.o	41	obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) += blk-crypto-fallback.o
41	obj-$(CONFIG_BLOCK_HOLDER_DEPRECATED) += holder.o	42	obj-$(CONFIG_BLOCK_HOLDER_DEPRECATED) += holder.o

block/bdev.c+7−4

 	if (test_bit(GD_NEED_PART_SCAN, &disk->state))
 		bdev_disk_changed(disk, false);
 	bdev->bd_openers++;
-	return 0;;
+	return 0;
 }
 static void blkdev_put_whole(struct block_device *bdev, fmode_t mode)
 	struct inode *inode;
 	inode = ilookup(blockdev_superblock, dev);
-	if (!inode) {
+	if (!inode && IS_ENABLED(CONFIG_BLOCK_LEGACY_AUTOLOAD)) {
 		blk_request_module(dev);
 		inode = ilookup(blockdev_superblock, dev);
-		if (!inode)
+		if (inode)
-			return NULL;
+			pr_warn_ratelimited(
+"block device autoloading is deprecated and will be removed.\n");
 	}
+	if (!inode)
+		return NULL;
 	/* switch from the inode reference to a device mode one: */
 	bdev = &BDEV_I(inode)->bdev;

block/bfq-cgroup.c+15−1

 		   struct bfq_group *bfqg)
 {
 	struct bfq_entity *entity = &bfqq->entity;
+	struct bfq_group *old_parent = bfqq_group(bfqq);
+	/*
+	 * No point to move bfqq to the same group, which can happen when
+	 * root group is offlined
+	 */
+	if (old_parent == bfqg)
+		return;
+	/*
+	 * oom_bfqq is not allowed to move, oom_bfqq will hold ref to root_group
+	 * until elevator exit.
+	 */
+	if (bfqq == &bfqd->oom_bfqq)
+		return;
 	/*
 	 * Get extra reference to prevent bfqq from being freed in
 	 * next possible expire or deactivate.
 		bfq_deactivate_bfqq(bfqd, bfqq, false, false);
 	else if (entity->on_st_or_in_serv)
 		bfq_put_idle_entity(bfq_entity_service_tree(entity), entity);
-	bfqg_and_blkg_put(bfqq_group(bfqq));
+	bfqg_and_blkg_put(old_parent);
 	if (entity->parent &&
 	    entity->parent->last_bfqq_created == bfqq)

block/bfq-iosched.c+24−13

@@ -774,7 +774,7 @@ bfq_pos_tree_add_move(struct bfq_data bfqd, struct bfq_queue bfqq) · 773 unmodified lines
774	if (!bfqq->next_rq)	774	if (!bfqq->next_rq)
775	return;	775	return;
776		776
777	bfqq->pos_root = &bfq_bfqq_to_bfqg(bfqq)->rq_pos_tree;	777	bfqq->pos_root = &bfqq_group(bfqq)->rq_pos_tree;
778	__bfqq = bfq_rq_pos_tree_lookup(bfqd, bfqq->pos_root,	778	__bfqq = bfq_rq_pos_tree_lookup(bfqd, bfqq->pos_root,
779	blk_rq_pos(bfqq->next_rq), &parent, &p);	779	blk_rq_pos(bfqq->next_rq), &parent, &p);
780	if (!__bfqq) {	780	if (!__bfqq) {
@@ -2153,7 +2153,7 @@ static void bfq_check_waker(struct bfq_data bfqd, struct bfq_queue bfqq, · 1372 unmodified lines
2153	bfqq->waker_detection_started = now_ns;	2153	bfqq->waker_detection_started = now_ns;
2154	bfq_bfqq_name(bfqq->tentative_waker_bfqq, waker_name,	2154	bfq_bfqq_name(bfqq->tentative_waker_bfqq, waker_name,
2155	MAX_BFQQ_NAME_LENGTH);	2155	MAX_BFQQ_NAME_LENGTH);
2156	bfq_log_bfqq(bfqd, bfqq, "set tenative waker %s", waker_name);	2156	bfq_log_bfqq(bfqd, bfqq, "set tentative waker %s", waker_name);
2157	} else /* Same tentative waker queue detected again */	2157	} else /* Same tentative waker queue detected again */
2158	bfqq->num_waker_detections++;	2158	bfqq->num_waker_detections++;
2159		2159
@@ -2669,7 +2669,7 @@ static struct bfq_queue bfqq_find_close(struct bfq_data bfqd, · 509 unmodified lines
2669	struct bfq_queue *bfqq,	2669	struct bfq_queue *bfqq,
2670	sector_t sector)	2670	sector_t sector)
2671	{	2671	{
2672	struct rb_root *root = &bfq_bfqq_to_bfqg(bfqq)->rq_pos_tree;	2672	struct rb_root *root = &bfqq_group(bfqq)->rq_pos_tree;
2673	struct rb_node parent, node;	2673	struct rb_node parent, node;
2674	struct bfq_queue *__bfqq;	2674	struct bfq_queue *__bfqq;
2675		2675
@@ -2782,6 +2782,15 @@ bfq_setup_merge(struct bfq_queue bfqq, struct bfq_queue new_bfqq) · 106 unmodified lines
2782	* are likely to increase the throughput.	2782	* are likely to increase the throughput.
2783	*/	2783	*/
2784	bfqq->new_bfqq = new_bfqq;	2784	bfqq->new_bfqq = new_bfqq;
		2785	/*
		2786	* The above assignment schedules the following redirections:
		2787	* each time some I/O for bfqq arrives, the process that
		2788	* generated that I/O is disassociated from bfqq and
		2789	* associated with new_bfqq. Here we increases new_bfqq->ref
		2790	* in advance, adding the number of processes that are
		2791	* expected to be associated with new_bfqq as they happen to
		2792	* issue I/O.
		2793	*/
2785	new_bfqq->ref += process_refs;	2794	new_bfqq->ref += process_refs;
2786	return new_bfqq;	2795	return new_bfqq;
2787	}	2796	}
@@ -2844,6 +2853,10 @@ bfq_setup_cooperator(struct bfq_data bfqd, struct bfq_queue bfqq, · 56 unmodified lines
2844	{	2853	{
2845	struct bfq_queue in_service_bfqq, new_bfqq;	2854	struct bfq_queue in_service_bfqq, new_bfqq;
2846		2855
		2856	/* if a merge has already been setup, then proceed with that first */
		2857	if (bfqq->new_bfqq)
		2858	return bfqq->new_bfqq;
		2859
2847	/*	2860	/*
2848	* Check delayed stable merge for rotational or non-queueing	2861	* Check delayed stable merge for rotational or non-queueing
2849	* devs. For this branch to be executed, bfqq must not be	2862	* devs. For this branch to be executed, bfqq must not be
@@ -2945,9 +2958,6 @@ bfq_setup_cooperator(struct bfq_data bfqd, struct bfq_queue bfqq, · 95 unmodified lines
2945	if (bfq_too_late_for_merging(bfqq))	2958	if (bfq_too_late_for_merging(bfqq))
2946	return NULL;	2959	return NULL;
2947		2960
2948	if (bfqq->new_bfqq)
2949	return bfqq->new_bfqq;
2950
2951	if (!io_struct \|\| unlikely(bfqq == &bfqd->oom_bfqq))	2961	if (!io_struct \|\| unlikely(bfqq == &bfqd->oom_bfqq))
2952	return NULL;	2962	return NULL;
2953		2963
@@ -5181,7 +5191,7 @@ static struct request bfq_dispatch_request(struct blk_mq_hw_ctx hctx) · 2227 unmodified lines
5181	struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;	5191	struct bfq_data *bfqd = hctx->queue->elevator->elevator_data;
5182	struct request *rq;	5192	struct request *rq;
5183	struct bfq_queue *in_serv_queue;	5193	struct bfq_queue *in_serv_queue;
5184	bool waiting_rq, idle_timer_disabled;	5194	bool waiting_rq, idle_timer_disabled = false;
5185		5195
5186	spin_lock_irq(&bfqd->lock);	5196	spin_lock_irq(&bfqd->lock);
5187		5197
@@ -5189,14 +5199,15 @@ static struct request bfq_dispatch_request(struct blk_mq_hw_ctx hctx) · 1 unmodified line
5189	waiting_rq = in_serv_queue && bfq_bfqq_wait_request(in_serv_queue);	5199	waiting_rq = in_serv_queue && bfq_bfqq_wait_request(in_serv_queue);
5190		5200
5191	rq = __bfq_dispatch_request(hctx);	5201	rq = __bfq_dispatch_request(hctx);
5192		5202	if (in_serv_queue == bfqd->in_service_queue) {
5193	idle_timer_disabled =	5203	idle_timer_disabled =
5194	waiting_rq && !bfq_bfqq_wait_request(in_serv_queue);	5204	waiting_rq && !bfq_bfqq_wait_request(in_serv_queue);
		5205	}
5195		5206
5196	spin_unlock_irq(&bfqd->lock);	5207	spin_unlock_irq(&bfqd->lock);
5197		5208	bfq_update_dispatch_stats(hctx->queue, rq,
5198	bfq_update_dispatch_stats(hctx->queue, rq, in_serv_queue,	5209	idle_timer_disabled ? in_serv_queue : NULL,
5199	idle_timer_disabled);	5210	idle_timer_disabled);
5200		5211
5201	return rq;	5212	return rq;
5202	}	5213	}

block/bfq-iosched.h+0−2

 #include <linux/blktrace_api.h>
 #include <linux/hrtimer.h>
-#include <linux/blk-cgroup.h>
 #include "blk-cgroup-rwstat.h"
 	for (parent = NULL; entity ; entity = parent)
 #endif /* CONFIG_BFQ_GROUP_IOSCHED */
-struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq);
 struct bfq_queue *bfq_entity_to_bfqq(struct bfq_entity *entity);
 unsigned int bfq_tot_busy_queues(struct bfq_data *bfqd);
 struct bfq_service_tree *bfq_entity_service_tree(struct bfq_entity *entity);

block/bfq-wf2q.c+1−16

 #ifdef CONFIG_BFQ_GROUP_IOSCHED
-struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq)
-{
-	struct bfq_entity *group_entity = bfqq->entity.parent;
-	if (!group_entity)
-		group_entity = &bfqq->bfqd->root_group->entity;
-	return container_of(group_entity, struct bfq_group, entity);
-}
 /*
  * Returns true if this budget changes may let next_in_service->parent
  * become the next_in_service entity for its parent entity.
 #else /* CONFIG_BFQ_GROUP_IOSCHED */
-struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq)
-{
-	return bfqq->bfqd->root_group;
-}
 static bool bfq_update_parent_budget(struct bfq_entity *next_in_service)
 {
 	return false;
 static unsigned short bfq_weight_to_ioprio(int weight)
 {
 	return max_t(int, 0,
-		     IOPRIO_NR_LEVELS * BFQ_WEIGHT_CONVERSION_COEFF - weight);
+		     IOPRIO_NR_LEVELS - weight / BFQ_WEIGHT_CONVERSION_COEFF);
 }
 static void bfq_get_entity(struct bfq_entity *entity)

block/bio-integrity.c+0−1

@@ -420,7 +420,6 @@ int bio_integrity_clone(struct bio bio, struct bio bio_src, · 419 unmodified lines
420		420
421	return 0;	421	return 0;
422	}	422	}
423	EXPORT_SYMBOL(bio_integrity_clone);
424		423
425	int bioset_integrity_create(struct bio_set *bs, int pool_size)	424	int bioset_integrity_create(struct bio_set *bs, int pool_size)
426	{	425	{

block/bio.c+111−79

@@ -15,7 +15,6 @@ · 14 unmodified lines
15	#include <linux/mempool.h>	15	#include <linux/mempool.h>
16	#include <linux/workqueue.h>	16	#include <linux/workqueue.h>
17	#include <linux/cgroup.h>	17	#include <linux/cgroup.h>
18	#include <linux/blk-cgroup.h>
19	#include <linux/highmem.h>	18	#include <linux/highmem.h>
20	#include <linux/sched/sysctl.h>	19	#include <linux/sched/sysctl.h>
21	#include <linux/blk-crypto.h>	20	#include <linux/blk-crypto.h>
@@ -24,6 +23,7 @@ · 2 unmodified lines
24	#include <trace/events/block.h>	23	#include <trace/events/block.h>
25	#include "blk.h"	24	#include "blk.h"
26	#include "blk-rq-qos.h"	25	#include "blk-rq-qos.h"
		26	#include "blk-cgroup.h"
27		27
28	struct bio_alloc_cache {	28	struct bio_alloc_cache {
29	struct bio *free_list;	29	struct bio *free_list;
@@ -249,12 +249,12 @@ static void bio_free(struct bio *bio) · 219 unmodified lines
249	* they must remember to pair any call to bio_init() with bio_uninit()	249	* they must remember to pair any call to bio_init() with bio_uninit()
250	* when IO has completed, or when the bio is released.	250	* when IO has completed, or when the bio is released.
251	*/	251	*/
252	void bio_init(struct bio bio, struct bio_vec table,	252	void bio_init(struct bio bio, struct block_device bdev, struct bio_vec *table,
253	unsigned short max_vecs)	253	unsigned short max_vecs, unsigned int opf)
254	{	254	{
255	bio->bi_next = NULL;	255	bio->bi_next = NULL;
256	bio->bi_bdev = NULL;	256	bio->bi_bdev = bdev;
257	bio->bi_opf = 0;	257	bio->bi_opf = opf;
258	bio->bi_flags = 0;	258	bio->bi_flags = 0;
259	bio->bi_ioprio = 0;	259	bio->bi_ioprio = 0;
260	bio->bi_write_hint = 0;	260	bio->bi_write_hint = 0;
@@ -268,6 +268,8 @@ void bio_init(struct bio bio, struct bio_vec table, · 7 unmodified lines
268	#ifdef CONFIG_BLK_CGROUP	268	#ifdef CONFIG_BLK_CGROUP
269	bio->bi_blkg = NULL;	269	bio->bi_blkg = NULL;
270	bio->bi_issue.value = 0;	270	bio->bi_issue.value = 0;
		271	if (bdev)
		272	bio_associate_blkg(bio);
271	#ifdef CONFIG_BLK_CGROUP_IOCOST	273	#ifdef CONFIG_BLK_CGROUP_IOCOST
272	bio->bi_iocost_cost = 0;	274	bio->bi_iocost_cost = 0;
273	#endif	275	#endif
@@ -293,6 +295,8 @@ EXPORT_SYMBOL(bio_init); · 19 unmodified lines
293	/**	295	/**
294	* bio_reset - reinitialize a bio	296	* bio_reset - reinitialize a bio
295	* @bio: bio to reset	297	* @bio: bio to reset
		298	* @bdev: block device to use the bio for
		299	* @opf: operation and flags for bio
296	*	300	*
297	* Description:	301	* Description:
298	* After calling bio_reset(), @bio will be in the same state as a freshly	302	* After calling bio_reset(), @bio will be in the same state as a freshly
@@ -300,11 +304,15 @@ EXPORT_SYMBOL(bio_init); · 1 unmodified line
300	* preserved are the ones that are initialized by bio_alloc_bioset(). See	304	* preserved are the ones that are initialized by bio_alloc_bioset(). See
301	* comment in struct bio.	305	* comment in struct bio.
302	*/	306	*/
303	void bio_reset(struct bio *bio)	307	void bio_reset(struct bio bio, struct block_device bdev, unsigned int opf)
304	{	308	{
305	bio_uninit(bio);	309	bio_uninit(bio);
306	memset(bio, 0, BIO_RESET_BYTES);	310	memset(bio, 0, BIO_RESET_BYTES);
307	atomic_set(&bio->__bi_remaining, 1);	311	atomic_set(&bio->__bi_remaining, 1);
		312	bio->bi_bdev = bdev;
		313	if (bio->bi_bdev)
		314	bio_associate_blkg(bio);
		315	bio->bi_opf = opf;
308	}	316	}
309	EXPORT_SYMBOL(bio_reset);	317	EXPORT_SYMBOL(bio_reset);
310		318
@@ -344,6 +352,20 @@ void bio_chain(struct bio bio, struct bio parent) · 33 unmodified lines
344	}	352	}
345	EXPORT_SYMBOL(bio_chain);	353	EXPORT_SYMBOL(bio_chain);
346		354
		355	struct bio blk_next_bio(struct bio bio, struct block_device *bdev,
		356	unsigned int nr_pages, unsigned int opf, gfp_t gfp)
		357	{
		358	struct bio *new = bio_alloc(bdev, nr_pages, opf, gfp);
		359
		360	if (bio) {
		361	bio_chain(bio, new);
		362	submit_bio(bio);
		363	}
		364
		365	return new;
		366	}
		367	EXPORT_SYMBOL_GPL(blk_next_bio);
		368
347	static void bio_alloc_rescue(struct work_struct *work)	369	static void bio_alloc_rescue(struct work_struct *work)
348	{	370	{
349	struct bio_set *bs = container_of(work, struct bio_set, rescue_work);	371	struct bio_set *bs = container_of(work, struct bio_set, rescue_work);
@@ -400,8 +422,10 @@ static void punt_bios_to_rescuer(struct bio_set *bs) · 50 unmodified lines
400		422
401	/**	423	/**
402	* bio_alloc_bioset - allocate a bio for I/O	424	* bio_alloc_bioset - allocate a bio for I/O
		425	* @bdev: block device to allocate the bio for (can be %NULL)
		426	* @nr_vecs: number of bvecs to pre-allocate
		427	* @opf: operation and flags for bio
403	* @gfp_mask: the GFP_* mask given to the slab allocator	428	* @gfp_mask: the GFP_* mask given to the slab allocator
404	* @nr_iovecs: number of iovecs to pre-allocate
405	* @bs: the bio_set to allocate from.	429	* @bs: the bio_set to allocate from.
406	*	430	*
407	* Allocate a bio from the mempools in @bs.	431	* Allocate a bio from the mempools in @bs.
@@ -430,15 +454,16 @@ static void punt_bios_to_rescuer(struct bio_set *bs) · 22 unmodified lines
430	*	454	*
431	* Returns: Pointer to new bio on success, NULL on failure.	455	* Returns: Pointer to new bio on success, NULL on failure.
432	*/	456	*/
433	struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned short nr_iovecs,	457	struct bio bio_alloc_bioset(struct block_device bdev, unsigned short nr_vecs,
		458	unsigned int opf, gfp_t gfp_mask,
434	struct bio_set *bs)	459	struct bio_set *bs)
435	{	460	{
436	gfp_t saved_gfp = gfp_mask;	461	gfp_t saved_gfp = gfp_mask;
437	struct bio *bio;	462	struct bio *bio;
438	void *p;	463	void *p;
439		464
440	/* should not use nobvec bioset for nr_iovecs > 0 */	465	/* should not use nobvec bioset for nr_vecs > 0 */
441	if (WARN_ON_ONCE(!mempool_initialized(&bs->bvec_pool) && nr_iovecs > 0))	466	if (WARN_ON_ONCE(!mempool_initialized(&bs->bvec_pool) && nr_vecs > 0))
442	return NULL;	467	return NULL;
443		468
444	/*	469	/*
@@ -475,23 +500,23 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned short nr_iovecs, · 30 unmodified lines
475	return NULL;	500	return NULL;
476		501
477	bio = p + bs->front_pad;	502	bio = p + bs->front_pad;
478	if (nr_iovecs > BIO_INLINE_VECS) {	503	if (nr_vecs > BIO_INLINE_VECS) {
479	struct bio_vec *bvl = NULL;	504	struct bio_vec *bvl = NULL;
480		505
481	bvl = bvec_alloc(&bs->bvec_pool, &nr_iovecs, gfp_mask);	506	bvl = bvec_alloc(&bs->bvec_pool, &nr_vecs, gfp_mask);
482	if (!bvl && gfp_mask != saved_gfp) {	507	if (!bvl && gfp_mask != saved_gfp) {
483	punt_bios_to_rescuer(bs);	508	punt_bios_to_rescuer(bs);
484	gfp_mask = saved_gfp;	509	gfp_mask = saved_gfp;
485	bvl = bvec_alloc(&bs->bvec_pool, &nr_iovecs, gfp_mask);	510	bvl = bvec_alloc(&bs->bvec_pool, &nr_vecs, gfp_mask);
486	}	511	}
487	if (unlikely(!bvl))	512	if (unlikely(!bvl))
488	goto err_free;	513	goto err_free;
489		514
490	bio_init(bio, bvl, nr_iovecs);	515	bio_init(bio, bdev, bvl, nr_vecs, opf);
491	} else if (nr_iovecs) {	516	} else if (nr_vecs) {
492	bio_init(bio, bio->bi_inline_vecs, BIO_INLINE_VECS);	517	bio_init(bio, bdev, bio->bi_inline_vecs, BIO_INLINE_VECS, opf);
493	} else {	518	} else {
494	bio_init(bio, NULL, 0);	519	bio_init(bio, bdev, NULL, 0, opf);
495	}	520	}
496		521
497	bio->bi_pool = bs;	522	bio->bi_pool = bs;
@@ -522,7 +547,8 @@ struct bio *bio_kmalloc(gfp_t gfp_mask, unsigned short nr_iovecs) · 24 unmodified lines
522	bio = kmalloc(struct_size(bio, bi_inline_vecs, nr_iovecs), gfp_mask);	547	bio = kmalloc(struct_size(bio, bi_inline_vecs, nr_iovecs), gfp_mask);
523	if (unlikely(!bio))	548	if (unlikely(!bio))
524	return NULL;	549	return NULL;
525	bio_init(bio, nr_iovecs ? bio->bi_inline_vecs : NULL, nr_iovecs);	550	bio_init(bio, NULL, nr_iovecs ? bio->bi_inline_vecs : NULL, nr_iovecs,
		551	0);
526	bio->bi_pool = NULL;	552	bio->bi_pool = NULL;
527	return bio;	553	return bio;
528	}	554	}
@@ -702,80 +728,84 @@ void bio_put(struct bio *bio) · 173 unmodified lines
702	}	728	}
703	EXPORT_SYMBOL(bio_put);	729	EXPORT_SYMBOL(bio_put);
704		730
705	/**	731	static int __bio_clone(struct bio bio, struct bio bio_src, gfp_t gfp)
706	* __bio_clone_fast - clone a bio that shares the original bio's biovec
707	* @bio: destination bio
708	* @bio_src: bio to clone
709	*
710	* Clone a &bio. Caller will own the returned bio, but not
711	* the actual data it points to. Reference count of returned
712	* bio will be one.
713	*
714	* Caller must ensure that @bio_src is not freed before @bio.
715	*/
716	void __bio_clone_fast(struct bio bio, struct bio bio_src)
717	{	732	{
718	WARN_ON_ONCE(bio->bi_pool && bio->bi_max_vecs);
719
720	/*
721	* most users will be overriding ->bi_bdev with a new target,
722	* so we don't set nor calculate new physical/hw segment counts here
723	*/
724	bio->bi_bdev = bio_src->bi_bdev;
725	bio_set_flag(bio, BIO_CLONED);	733	bio_set_flag(bio, BIO_CLONED);
726	if (bio_flagged(bio_src, BIO_THROTTLED))	734	if (bio_flagged(bio_src, BIO_THROTTLED))
727	bio_set_flag(bio, BIO_THROTTLED);	735	bio_set_flag(bio, BIO_THROTTLED);
728	if (bio_flagged(bio_src, BIO_REMAPPED))	736	if (bio->bi_bdev == bio_src->bi_bdev &&
		737	bio_flagged(bio_src, BIO_REMAPPED))
729	bio_set_flag(bio, BIO_REMAPPED);	738	bio_set_flag(bio, BIO_REMAPPED);
730	bio->bi_opf = bio_src->bi_opf;
731	bio->bi_ioprio = bio_src->bi_ioprio;	739	bio->bi_ioprio = bio_src->bi_ioprio;
732	bio->bi_write_hint = bio_src->bi_write_hint;	740	bio->bi_write_hint = bio_src->bi_write_hint;
733	bio->bi_iter = bio_src->bi_iter;	741	bio->bi_iter = bio_src->bi_iter;
734	bio->bi_io_vec = bio_src->bi_io_vec;
735		742
736	bio_clone_blkg_association(bio, bio_src);	743	bio_clone_blkg_association(bio, bio_src);
737	blkcg_bio_issue_init(bio);	744	blkcg_bio_issue_init(bio);
		745
		746	if (bio_crypt_clone(bio, bio_src, gfp) < 0)
		747	return -ENOMEM;
		748	if (bio_integrity(bio_src) &&
		749	bio_integrity_clone(bio, bio_src, gfp) < 0)
		750	return -ENOMEM;
		751	return 0;
738	}	752	}
739	EXPORT_SYMBOL(__bio_clone_fast);
740		753
741	/**	754	/**
742	* bio_clone_fast - clone a bio that shares the original bio's biovec	755	* bio_alloc_clone - clone a bio that shares the original bio's biovec
743	* @bio: bio to clone	756	* @bdev: block_device to clone onto
744	* @gfp_mask: allocation priority	757	* @bio_src: bio to clone from
745	* @bs: bio_set to allocate from	758	* @gfp: allocation priority
		759	* @bs: bio_set to allocate from
		760	*
		761	* Allocate a new bio that is a clone of @bio_src. The caller owns the returned
		762	* bio, but not the actual data it points to.
746	*	763	*
747	* Like __bio_clone_fast, only also allocates the returned bio	764	* The caller must ensure that the return bio is not freed before @bio_src.
748	*/	765	*/
749	struct bio bio_clone_fast(struct bio bio, gfp_t gfp_mask, struct bio_set *bs)	766	struct bio bio_alloc_clone(struct block_device bdev, struct bio *bio_src,
		767	gfp_t gfp, struct bio_set *bs)
750	{	768	{
751	struct bio *b;	769	struct bio *bio;
752		770
753	b = bio_alloc_bioset(gfp_mask, 0, bs);	771	bio = bio_alloc_bioset(bdev, 0, bio_src->bi_opf, gfp, bs);
754	if (!b)	772	if (!bio)
755	return NULL;	773	return NULL;
756		774
757	__bio_clone_fast(b, bio);	775	if (__bio_clone(bio, bio_src, gfp) < 0) {
758		776	bio_put(bio);
759	if (bio_crypt_clone(b, bio, gfp_mask) < 0)	777	return NULL;
760	goto err_put;	778	}
761		779	bio->bi_io_vec = bio_src->bi_io_vec;
762	if (bio_integrity(bio) &&
763	bio_integrity_clone(b, bio, gfp_mask) < 0)
764	goto err_put;
765
766	return b;
767		780
768	err_put:	781	return bio;
769	bio_put(b);
770	return NULL;
771	}	782	}
772	EXPORT_SYMBOL(bio_clone_fast);	783	EXPORT_SYMBOL(bio_alloc_clone);
773		784
774	const char bio_devname(struct bio bio, char *buf)	785	/**
		786	* bio_init_clone - clone a bio that shares the original bio's biovec
		787	* @bdev: block_device to clone onto
		788	* @bio: bio to clone into
		789	* @bio_src: bio to clone from
		790	* @gfp: allocation priority
		791	*
		792	* Initialize a new bio in caller provided memory that is a clone of @bio_src.
		793	* The caller owns the returned bio, but not the actual data it points to.
		794	*
		795	* The caller must ensure that @bio_src is not freed before @bio.
		796	*/
		797	int bio_init_clone(struct block_device bdev, struct bio bio,
		798	struct bio *bio_src, gfp_t gfp)
775	{	799	{
776	return bdevname(bio->bi_bdev, buf);	800	int ret;
		801
		802	bio_init(bio, bdev, bio_src->bi_io_vec, 0, bio_src->bi_opf);
		803	ret = __bio_clone(bio, bio_src, gfp);
		804	if (ret)
		805	bio_uninit(bio);
		806	return ret;
777	}	807	}
778	EXPORT_SYMBOL(bio_devname);	808	EXPORT_SYMBOL(bio_init_clone);
779		809
780	/**	810	/**
781	* bio_full - check if the bio is full	811	* bio_full - check if the bio is full
@@ -1054,7 +1084,7 @@ bool bio_add_folio(struct bio bio, struct folio folio, size_t len, · 272 unmodified lines
1054	size_t off)	1084	size_t off)
1055	{	1085	{
1056	if (len > UINT_MAX \|\| off > UINT_MAX)	1086	if (len > UINT_MAX \|\| off > UINT_MAX)
1057	return 0;	1087	return false;
1058	return bio_add_page(bio, &folio->page, len, off) > 0;	1088	return bio_add_page(bio, &folio->page, len, off) > 0;
1059	}	1089	}
1060		1090
@@ -1486,8 +1516,7 @@ again: · 425 unmodified lines
1486	if (!bio_integrity_endio(bio))	1516	if (!bio_integrity_endio(bio))
1487	return;	1517	return;
1488		1518
1489	if (bio->bi_bdev && bio_flagged(bio, BIO_TRACKED))	1519	rq_qos_done_bio(bio);
1490	rq_qos_done_bio(bdev_get_queue(bio->bi_bdev), bio);
1491		1520
1492	if (bio->bi_bdev && bio_flagged(bio, BIO_TRACE_COMPLETION)) {	1521	if (bio->bi_bdev && bio_flagged(bio, BIO_TRACE_COMPLETION)) {
1493	trace_block_bio_complete(bdev_get_queue(bio->bi_bdev), bio);	1522	trace_block_bio_complete(bdev_get_queue(bio->bi_bdev), bio);
@@ -1541,7 +1570,7 @@ struct bio bio_split(struct bio bio, int sectors, · 47 unmodified lines
1541	if (WARN_ON_ONCE(bio_op(bio) == REQ_OP_ZONE_APPEND))	1570	if (WARN_ON_ONCE(bio_op(bio) == REQ_OP_ZONE_APPEND))
1542	return NULL;	1571	return NULL;
1543		1572
1544	split = bio_clone_fast(bio, gfp, bs);	1573	split = bio_alloc_clone(bio->bi_bdev, bio, gfp, bs);
1545	if (!split)	1574	if (!split)
1546	return NULL;	1575	return NULL;
1547		1576
@@ -1636,9 +1665,9 @@ EXPORT_SYMBOL(bioset_exit); · 88 unmodified lines
1636	* Note that the bio must be embedded at the END of that structure always,	1665	* Note that the bio must be embedded at the END of that structure always,
1637	* or things will break badly.	1666	* or things will break badly.
1638	* If %BIOSET_NEED_BVECS is set in @flags, a separate pool will be allocated	1667	* If %BIOSET_NEED_BVECS is set in @flags, a separate pool will be allocated
1639	* for allocating iovecs. This pool is not needed e.g. for bio_clone_fast().	1668	* for allocating iovecs. This pool is not needed e.g. for bio_init_clone().
1640	* If %BIOSET_NEED_RESCUER is set, a workqueue is created which can be used to	1669	* If %BIOSET_NEED_RESCUER is set, a workqueue is created which can be used
1641	* dispatch queued requests when the mempool runs out of space.	1670	* to dispatch queued requests when the mempool runs out of space.
1642	*	1671	*
1643	*/	1672	*/
1644	int bioset_init(struct bio_set *bs,	1673	int bioset_init(struct bio_set *bs,
@@ -1708,7 +1737,9 @@ EXPORT_SYMBOL(bioset_init_from_src); · 63 unmodified lines
1708	/**	1737	/**
1709	* bio_alloc_kiocb - Allocate a bio from bio_set based on kiocb	1738	* bio_alloc_kiocb - Allocate a bio from bio_set based on kiocb
1710	* @kiocb: kiocb describing the IO	1739	* @kiocb: kiocb describing the IO
		1740	* @bdev: block device to allocate the bio for (can be %NULL)
1711	* @nr_vecs: number of iovecs to pre-allocate	1741	* @nr_vecs: number of iovecs to pre-allocate
		1742	* @opf: operation and flags for bio
1712	* @bs: bio_set to allocate from	1743	* @bs: bio_set to allocate from
1713	*	1744	*
1714	* Description:	1745	* Description:
@@ -1719,14 +1750,14 @@ EXPORT_SYMBOL(bioset_init_from_src); · 4 unmodified lines
1719	* MUST be done from process context, not hard/soft IRQ.	1750	* MUST be done from process context, not hard/soft IRQ.
1720	*	1751	*
1721	*/	1752	*/
1722	struct bio bio_alloc_kiocb(struct kiocb kiocb, unsigned short nr_vecs,	1753	struct bio bio_alloc_kiocb(struct kiocb kiocb, struct block_device *bdev,
1723	struct bio_set *bs)	1754	unsigned short nr_vecs, unsigned int opf, struct bio_set *bs)
1724	{	1755	{
1725	struct bio_alloc_cache *cache;	1756	struct bio_alloc_cache *cache;
1726	struct bio *bio;	1757	struct bio *bio;
1727		1758
1728	if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE) \|\| nr_vecs > BIO_INLINE_VECS)	1759	if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE) \|\| nr_vecs > BIO_INLINE_VECS)
1729	return bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs);	1760	return bio_alloc_bioset(bdev, nr_vecs, opf, GFP_KERNEL, bs);
1730		1761
1731	cache = per_cpu_ptr(bs->cache, get_cpu());	1762	cache = per_cpu_ptr(bs->cache, get_cpu());
1732	if (cache->free_list) {	1763	if (cache->free_list) {
@@ -1734,13 +1765,14 @@ struct bio bio_alloc_kiocb(struct kiocb kiocb, unsigned short nr_vecs, · 1 unmodified line
1734	cache->free_list = bio->bi_next;	1765	cache->free_list = bio->bi_next;
1735	cache->nr--;	1766	cache->nr--;
1736	put_cpu();	1767	put_cpu();
1737	bio_init(bio, nr_vecs ? bio->bi_inline_vecs : NULL, nr_vecs);	1768	bio_init(bio, bdev, nr_vecs ? bio->bi_inline_vecs : NULL,
		1769	nr_vecs, opf);
1738	bio->bi_pool = bs;	1770	bio->bi_pool = bs;
1739	bio_set_flag(bio, BIO_PERCPU_CACHE);	1771	bio_set_flag(bio, BIO_PERCPU_CACHE);
1740	return bio;	1772	return bio;
1741	}	1773	}
1742	put_cpu();	1774	put_cpu();
1743	bio = bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs);	1775	bio = bio_alloc_bioset(bdev, nr_vecs, opf, GFP_KERNEL, bs);
1744	bio_set_flag(bio, BIO_PERCPU_CACHE);	1776	bio_set_flag(bio, BIO_PERCPU_CACHE);
1745	return bio;	1777	return bio;
1746	}	1778	}

block/blk-cgroup-rwstat.h+1−1

@@ -6,7 +6,7 @@ · 5 unmodified lines
6	#ifndef _BLK_CGROUP_RWSTAT_H	6	#ifndef _BLK_CGROUP_RWSTAT_H
7	#define _BLK_CGROUP_RWSTAT_H	7	#define _BLK_CGROUP_RWSTAT_H
8		8
9	#include <linux/blk-cgroup.h>	9	#include "blk-cgroup.h"
10		10
11	enum blkg_rwstat_type {	11	enum blkg_rwstat_type {
12	BLKG_RWSTAT_READ,	12	BLKG_RWSTAT_READ,

block/blk-cgroup.c+13−7

 #include <linux/blkdev.h>
 #include <linux/backing-dev.h>
 #include <linux/slab.h>
-#include <linux/genhd.h>
 #include <linux/delay.h>
 #include <linux/atomic.h>
 #include <linux/ctype.h>
-#include <linux/blk-cgroup.h>
 #include <linux/tracehook.h>
 #include <linux/psi.h>
 #include <linux/part_stat.h>
 #include "blk.h"
+#include "blk-cgroup.h"
 #include "blk-ioprio.h"
 #include "blk-throttle.h"
 		if (blkg->pd[i])
 			blkcg_policy[i]->pd_free_fn(blkg->pd[i]);
+	if (blkg->q)
+		blk_put_queue(blkg->q);
 	free_percpu(blkg->iostat_cpu);
 	percpu_ref_exit(&blkg->refcnt);
 	kfree(blkg);
 	if (!blkg->iostat_cpu)
 		goto err_free;
+	if (!blk_get_queue(q))
+		goto err_free;
 	blkg->q = q;
 	INIT_LIST_HEAD(&blkg->q_node);
 	spin_lock_init(&blkg->async_bio_lock);
 			blk_queue_root_blkg(bdev_get_queue(bdev));
 		struct blkg_iostat tmp;
 		int cpu;
+		unsigned long flags;
 		memset(&tmp, 0, sizeof(tmp));
 		for_each_possible_cpu(cpu) {
 			struct disk_stats *cpu_dkstats;
-			unsigned long flags;
 			cpu_dkstats = per_cpu_ptr(bdev->bd_stats, cpu);
 			tmp.ios[BLKG_IOSTAT_READ] +=
 				cpu_dkstats->sectors[STAT_WRITE] << 9;
 			tmp.bytes[BLKG_IOSTAT_DISCARD] +=
 				cpu_dkstats->sectors[STAT_DISCARD] << 9;
-			flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync);
-			blkg_iostat_set(&blkg->iostat.cur, &tmp);
-			u64_stats_update_end_irqrestore(&blkg->iostat.sync, flags);
 		}
+		flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync);
+		blkg_iostat_set(&blkg->iostat.cur, &tmp);
+		u64_stats_update_end_irqrestore(&blkg->iostat.sync, flags);
 	}
 }
 	bool preloaded;
 	int ret;
+	INIT_LIST_HEAD(&q->blkg_list);
 	new_blkg = blkg_alloc(&blkcg_root, q, GFP_KERNEL);
 	if (!new_blkg)
 		return -ENOMEM;

block/blk-cgroup.h+494−0

@@ -0,0 +1,494 @@
	1	/* SPDX-License-Identifier: GPL-2.0 */
	2	#ifndef _BLK_CGROUP_PRIVATE_H
	3	#define _BLK_CGROUP_PRIVATE_H
	4	/*
	5	* block cgroup private header
	6	*
	7	* Based on ideas and code from CFQ, CFS and BFQ:
	8	* Copyright (C) 2003 Jens Axboe <axboe@kernel.dk>
	9	*
	10	* Copyright (C) 2008 Fabio Checconi <fabio@gandalf.sssup.it>
	11	* Paolo Valente <paolo.valente@unimore.it>
	12	*
	13	* Copyright (C) 2009 Vivek Goyal <vgoyal@redhat.com>
	14	* Nauman Rafique <nauman@google.com>
	15	*/
	16
	17	#include <linux/blk-cgroup.h>
	18	#include <linux/blk-mq.h>
	19
	20	/* percpu_counter batch for blkg_[rw]stats, per-cpu drift doesn't matter */
	21	#define BLKG_STAT_CPU_BATCH (INT_MAX / 2)
	22
	23	#ifdef CONFIG_BLK_CGROUP
	24
	25	/*
	26	* A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a
	27	* request_queue (q). This is used by blkcg policies which need to track
	28	* information per blkcg - q pair.
	29	*
	30	* There can be multiple active blkcg policies and each blkg:policy pair is
	31	* represented by a blkg_policy_data which is allocated and freed by each
	32	* policy's pd_alloc/free_fn() methods. A policy can allocate private data
	33	* area by allocating larger data structure which embeds blkg_policy_data
	34	* at the beginning.
	35	*/
	36	struct blkg_policy_data {
	37	/* the blkg and policy id this per-policy data belongs to */
	38	struct blkcg_gq *blkg;
	39	int plid;
	40	};
	41
	42	/*
	43	* Policies that need to keep per-blkcg data which is independent from any
	44	* request_queue associated to it should implement cpd_alloc/free_fn()
	45	* methods. A policy can allocate private data area by allocating larger
	46	* data structure which embeds blkcg_policy_data at the beginning.
	47	* cpd_init() is invoked to let each policy handle per-blkcg data.
	48	*/
	49	struct blkcg_policy_data {
	50	/* the blkcg and policy id this per-policy data belongs to */
	51	struct blkcg *blkcg;
	52	int plid;
	53	};
	54
	55	typedef struct blkcg_policy_data *(blkcg_pol_alloc_cpd_fn)(gfp_t gfp);
	56	typedef void (blkcg_pol_init_cpd_fn)(struct blkcg_policy_data *cpd);
	57	typedef void (blkcg_pol_free_cpd_fn)(struct blkcg_policy_data *cpd);
	58	typedef void (blkcg_pol_bind_cpd_fn)(struct blkcg_policy_data *cpd);
	59	typedef struct blkg_policy_data *(blkcg_pol_alloc_pd_fn)(gfp_t gfp,
	60	struct request_queue q, struct blkcg blkcg);
	61	typedef void (blkcg_pol_init_pd_fn)(struct blkg_policy_data *pd);
	62	typedef void (blkcg_pol_online_pd_fn)(struct blkg_policy_data *pd);
	63	typedef void (blkcg_pol_offline_pd_fn)(struct blkg_policy_data *pd);
	64	typedef void (blkcg_pol_free_pd_fn)(struct blkg_policy_data *pd);
	65	typedef void (blkcg_pol_reset_pd_stats_fn)(struct blkg_policy_data *pd);
	66	typedef bool (blkcg_pol_stat_pd_fn)(struct blkg_policy_data *pd,
	67	struct seq_file *s);
	68
	69	struct blkcg_policy {
	70	int plid;
	71	/* cgroup files for the policy */
	72	struct cftype *dfl_cftypes;
	73	struct cftype *legacy_cftypes;
	74
	75	/* operations */
	76	blkcg_pol_alloc_cpd_fn *cpd_alloc_fn;
	77	blkcg_pol_init_cpd_fn *cpd_init_fn;
	78	blkcg_pol_free_cpd_fn *cpd_free_fn;
	79	blkcg_pol_bind_cpd_fn *cpd_bind_fn;
	80
	81	blkcg_pol_alloc_pd_fn *pd_alloc_fn;
	82	blkcg_pol_init_pd_fn *pd_init_fn;
	83	blkcg_pol_online_pd_fn *pd_online_fn;
	84	blkcg_pol_offline_pd_fn *pd_offline_fn;
	85	blkcg_pol_free_pd_fn *pd_free_fn;
	86	blkcg_pol_reset_pd_stats_fn *pd_reset_stats_fn;
	87	blkcg_pol_stat_pd_fn *pd_stat_fn;
	88	};
	89
	90	extern struct blkcg blkcg_root;
	91	extern bool blkcg_debug_stats;
	92
	93	struct blkcg_gq blkg_lookup_slowpath(struct blkcg blkcg,
	94	struct request_queue *q, bool update_hint);
	95	int blkcg_init_queue(struct request_queue *q);
	96	void blkcg_exit_queue(struct request_queue *q);
	97
	98	/* Blkio controller policy registration */
	99	int blkcg_policy_register(struct blkcg_policy *pol);
	100	void blkcg_policy_unregister(struct blkcg_policy *pol);
	101	int blkcg_activate_policy(struct request_queue *q,
	102	const struct blkcg_policy *pol);
	103	void blkcg_deactivate_policy(struct request_queue *q,
	104	const struct blkcg_policy *pol);
	105
	106	const char blkg_dev_name(struct blkcg_gq blkg);
	107	void blkcg_print_blkgs(struct seq_file sf, struct blkcg blkcg,
	108	u64 (prfill)(struct seq_file ,
	109	struct blkg_policy_data *, int),
	110	const struct blkcg_policy *pol, int data,
	111	bool show_total);
	112	u64 __blkg_prfill_u64(struct seq_file sf, struct blkg_policy_data pd, u64 v);
	113
	114	struct blkg_conf_ctx {
	115	struct block_device *bdev;
	116	struct blkcg_gq *blkg;
	117	char *body;
	118	};
	119
	120	struct block_device blkcg_conf_open_bdev(char *inputp);
	121	int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol,
	122	char input, struct blkg_conf_ctx ctx);
	123	void blkg_conf_finish(struct blkg_conf_ctx *ctx);
	124
	125	/**
	126	* blkcg_css - find the current css
	127	*
	128	* Find the css associated with either the kthread or the current task.
	129	* This may return a dying css, so it is up to the caller to use tryget logic
	130	* to confirm it is alive and well.
	131	*/
	132	static inline struct cgroup_subsys_state *blkcg_css(void)
	133	{
	134	struct cgroup_subsys_state *css;
	135
	136	css = kthread_blkcg();
	137	if (css)
	138	return css;
	139	return task_css(current, io_cgrp_id);
	140	}
	141
	142	/**
	143	* __bio_blkcg - internal, inconsistent version to get blkcg
	144	*
	145	* DO NOT USE.
	146	* This function is inconsistent and consequently is dangerous to use. The
	147	* first part of the function returns a blkcg where a reference is owned by the
	148	* bio. This means it does not need to be rcu protected as it cannot go away
	149	* with the bio owning a reference to it. However, the latter potentially gets
	150	* it from task_css(). This can race against task migration and the cgroup
	151	* dying. It is also semantically different as it must be called rcu protected
	152	* and is susceptible to failure when trying to get a reference to it.
	153	* Therefore, it is not ok to assume that *_get() will always succeed on the
	154	* blkcg returned here.
	155	*/
	156	static inline struct blkcg __bio_blkcg(struct bio bio)
	157	{
	158	if (bio && bio->bi_blkg)
	159	return bio->bi_blkg->blkcg;
	160	return css_to_blkcg(blkcg_css());
	161	}
	162
	163	/**
	164	* bio_issue_as_root_blkg - see if this bio needs to be issued as root blkg
	165	* @return: true if this bio needs to be submitted with the root blkg context.
	166	*
	167	* In order to avoid priority inversions we sometimes need to issue a bio as if
	168	* it were attached to the root blkg, and then backcharge to the actual owning
	169	* blkg. The idea is we do bio_blkcg() to look up the actual context for the
	170	* bio and attach the appropriate blkg to the bio. Then we call this helper and
	171	* if it is true run with the root blkg for that queue and then do any
	172	* backcharging to the originating cgroup once the io is complete.
	173	*/
	174	static inline bool bio_issue_as_root_blkg(struct bio *bio)
	175	{
	176	return (bio->bi_opf & (REQ_META \| REQ_SWAP)) != 0;
	177	}
	178
	179	/**
	180	* __blkg_lookup - internal version of blkg_lookup()
	181	* @blkcg: blkcg of interest
	182	* @q: request_queue of interest
	183	* @update_hint: whether to update lookup hint with the result or not
	184	*
	185	* This is internal version and shouldn't be used by policy
	186	* implementations. Looks up blkgs for the @blkcg - @q pair regardless of
	187	* @q's bypass state. If @update_hint is %true, the caller should be
	188	* holding @q->queue_lock and lookup hint is updated on success.
	189	*/
	190	static inline struct blkcg_gq __blkg_lookup(struct blkcg blkcg,
	191	struct request_queue *q,
	192	bool update_hint)
	193	{
	194	struct blkcg_gq *blkg;
	195
	196	if (blkcg == &blkcg_root)
	197	return q->root_blkg;
	198
	199	blkg = rcu_dereference(blkcg->blkg_hint);
	200	if (blkg && blkg->q == q)
	201	return blkg;
	202
	203	return blkg_lookup_slowpath(blkcg, q, update_hint);
	204	}
	205
	206	/**
	207	* blkg_lookup - lookup blkg for the specified blkcg - q pair
	208	* @blkcg: blkcg of interest
	209	* @q: request_queue of interest
	210	*
	211	* Lookup blkg for the @blkcg - @q pair. This function should be called
	212	* under RCU read lock.
	213	*/
	214	static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg,
	215	struct request_queue *q)
	216	{
	217	WARN_ON_ONCE(!rcu_read_lock_held());
	218	return __blkg_lookup(blkcg, q, false);
	219	}
	220
	221	/**
	222	* blk_queue_root_blkg - return blkg for the (blkcg_root, @q) pair
	223	* @q: request_queue of interest
	224	*
	225	* Lookup blkg for @q at the root level. See also blkg_lookup().
	226	*/
	227	static inline struct blkcg_gq blk_queue_root_blkg(struct request_queue q)
	228	{
	229	return q->root_blkg;
	230	}
	231
	232	/**
	233	* blkg_to_pdata - get policy private data
	234	* @blkg: blkg of interest
	235	* @pol: policy of interest
	236	*
	237	* Return pointer to private data associated with the @blkg-@pol pair.
	238	*/
	239	static inline struct blkg_policy_data blkg_to_pd(struct blkcg_gq blkg,
	240	struct blkcg_policy *pol)
	241	{
	242	return blkg ? blkg->pd[pol->plid] : NULL;
	243	}
	244
	245	static inline struct blkcg_policy_data blkcg_to_cpd(struct blkcg blkcg,
	246	struct blkcg_policy *pol)
	247	{
	248	return blkcg ? blkcg->cpd[pol->plid] : NULL;
	249	}
	250
	251	/**
	252	* pdata_to_blkg - get blkg associated with policy private data
	253	* @pd: policy private data of interest
	254	*
	255	* @pd is policy private data. Determine the blkg it's associated with.
	256	*/
	257	static inline struct blkcg_gq pd_to_blkg(struct blkg_policy_data pd)
	258	{
	259	return pd ? pd->blkg : NULL;
	260	}
	261
	262	static inline struct blkcg cpd_to_blkcg(struct blkcg_policy_data cpd)
	263	{
	264	return cpd ? cpd->blkcg : NULL;
	265	}
	266
	267	/**
	268	* blkg_path - format cgroup path of blkg
	269	* @blkg: blkg of interest
	270	* @buf: target buffer
	271	* @buflen: target buffer length
	272	*
	273	* Format the path of the cgroup of @blkg into @buf.
	274	*/
	275	static inline int blkg_path(struct blkcg_gq blkg, char buf, int buflen)
	276	{
	277	return cgroup_path(blkg->blkcg->css.cgroup, buf, buflen);
	278	}
	279
	280	/**
	281	* blkg_get - get a blkg reference
	282	* @blkg: blkg to get
	283	*
	284	* The caller should be holding an existing reference.
	285	*/
	286	static inline void blkg_get(struct blkcg_gq *blkg)
	287	{
	288	percpu_ref_get(&blkg->refcnt);
	289	}
	290
	291	/**
	292	* blkg_tryget - try and get a blkg reference
	293	* @blkg: blkg to get
	294	*
	295	* This is for use when doing an RCU lookup of the blkg. We may be in the midst
	296	* of freeing this blkg, so we can only use it if the refcnt is not zero.
	297	*/
	298	static inline bool blkg_tryget(struct blkcg_gq *blkg)
	299	{
	300	return blkg && percpu_ref_tryget(&blkg->refcnt);
	301	}
	302
	303	/**
	304	* blkg_put - put a blkg reference
	305	* @blkg: blkg to put
	306	*/
	307	static inline void blkg_put(struct blkcg_gq *blkg)
	308	{
	309	percpu_ref_put(&blkg->refcnt);
	310	}
	311
	312	/**
	313	* blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants
	314	* @d_blkg: loop cursor pointing to the current descendant
	315	* @pos_css: used for iteration
	316	* @p_blkg: target blkg to walk descendants of
	317	*
	318	* Walk @c_blkg through the descendants of @p_blkg. Must be used with RCU
	319	* read locked. If called under either blkcg or queue lock, the iteration
	320	* is guaranteed to include all and only online blkgs. The caller may
	321	* update @pos_css by calling css_rightmost_descendant() to skip subtree.
	322	* @p_blkg is included in the iteration and the first node to be visited.
	323	*/
	324	#define blkg_for_each_descendant_pre(d_blkg, pos_css, p_blkg) \
	325	css_for_each_descendant_pre((pos_css), &(p_blkg)->blkcg->css) \
	326	if (((d_blkg) = __blkg_lookup(css_to_blkcg(pos_css), \
	327	(p_blkg)->q, false)))
	328
	329	/**
	330	* blkg_for_each_descendant_post - post-order walk of a blkg's descendants
	331	* @d_blkg: loop cursor pointing to the current descendant
	332	* @pos_css: used for iteration
	333	* @p_blkg: target blkg to walk descendants of
	334	*
	335	* Similar to blkg_for_each_descendant_pre() but performs post-order
	336	* traversal instead. Synchronization rules are the same. @p_blkg is
	337	* included in the iteration and the last node to be visited.
	338	*/
	339	#define blkg_for_each_descendant_post(d_blkg, pos_css, p_blkg) \
	340	css_for_each_descendant_post((pos_css), &(p_blkg)->blkcg->css) \
	341	if (((d_blkg) = __blkg_lookup(css_to_blkcg(pos_css), \
	342	(p_blkg)->q, false)))
	343
	344	bool __blkcg_punt_bio_submit(struct bio *bio);
	345
	346	static inline bool blkcg_punt_bio_submit(struct bio *bio)
	347	{
	348	if (bio->bi_opf & REQ_CGROUP_PUNT)
	349	return __blkcg_punt_bio_submit(bio);
	350	else
	351	return false;
	352	}
	353
	354	static inline void blkcg_bio_issue_init(struct bio *bio)
	355	{
	356	bio_issue_init(&bio->bi_issue, bio_sectors(bio));
	357	}
	358
	359	static inline void blkcg_use_delay(struct blkcg_gq *blkg)
	360	{
	361	if (WARN_ON_ONCE(atomic_read(&blkg->use_delay) < 0))
	362	return;
	363	if (atomic_add_return(1, &blkg->use_delay) == 1)
	364	atomic_inc(&blkg->blkcg->css.cgroup->congestion_count);
	365	}
	366
	367	static inline int blkcg_unuse_delay(struct blkcg_gq *blkg)
	368	{
	369	int old = atomic_read(&blkg->use_delay);
	370
	371	if (WARN_ON_ONCE(old < 0))
	372	return 0;
	373	if (old == 0)
	374	return 0;
	375
	376	/*
	377	* We do this song and dance because we can race with somebody else
	378	* adding or removing delay. If we just did an atomic_dec we'd end up
	379	* negative and we'd already be in trouble. We need to subtract 1 and
	380	* then check to see if we were the last delay so we can drop the
	381	* congestion count on the cgroup.
	382	*/
	383	while (old) {
	384	int cur = atomic_cmpxchg(&blkg->use_delay, old, old - 1);
	385	if (cur == old)
	386	break;
	387	old = cur;
	388	}
	389
	390	if (old == 0)
	391	return 0;
	392	if (old == 1)
	393	atomic_dec(&blkg->blkcg->css.cgroup->congestion_count);
	394	return 1;
	395	}
	396
	397	/**
	398	* blkcg_set_delay - Enable allocator delay mechanism with the specified delay amount
	399	* @blkg: target blkg
	400	* @delay: delay duration in nsecs
	401	*
	402	* When enabled with this function, the delay is not decayed and must be
	403	* explicitly cleared with blkcg_clear_delay(). Must not be mixed with
	404	* blkcg_[un]use_delay() and blkcg_add_delay() usages.
	405	*/
	406	static inline void blkcg_set_delay(struct blkcg_gq *blkg, u64 delay)
	407	{
	408	int old = atomic_read(&blkg->use_delay);
	409
	410	/* We only want 1 person setting the congestion count for this blkg. */
	411	if (!old && atomic_cmpxchg(&blkg->use_delay, old, -1) == old)
	412	atomic_inc(&blkg->blkcg->css.cgroup->congestion_count);
	413
	414	atomic64_set(&blkg->delay_nsec, delay);
	415	}
	416
	417	/**
	418	* blkcg_clear_delay - Disable allocator delay mechanism
	419	* @blkg: target blkg
	420	*
	421	* Disable use_delay mechanism. See blkcg_set_delay().
	422	*/
	423	static inline void blkcg_clear_delay(struct blkcg_gq *blkg)
	424	{
	425	int old = atomic_read(&blkg->use_delay);
	426
	427	/* We only want 1 person clearing the congestion count for this blkg. */
	428	if (old && atomic_cmpxchg(&blkg->use_delay, old, 0) == old)
	429	atomic_dec(&blkg->blkcg->css.cgroup->congestion_count);
	430	}
	431
	432	/**
	433	* blk_cgroup_mergeable - Determine whether to allow or disallow merges
	434	* @rq: request to merge into
	435	* @bio: bio to merge
	436	*
	437	* @bio and @rq should belong to the same cgroup and their issue_as_root should
	438	* match. The latter is necessary as we don't want to throttle e.g. a metadata
	439	* update because it happens to be next to a regular IO.
	440	*/
	441	static inline bool blk_cgroup_mergeable(struct request rq, struct bio bio)
	442	{
	443	return rq->bio->bi_blkg == bio->bi_blkg &&
	444	bio_issue_as_root_blkg(rq->bio) == bio_issue_as_root_blkg(bio);
	445	}
	446
	447	void blk_cgroup_bio_start(struct bio *bio);
	448	void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta);
	449	#else /* CONFIG_BLK_CGROUP */
	450
	451	struct blkg_policy_data {
	452	};
	453
	454	struct blkcg_policy_data {
	455	};
	456
	457	struct blkcg_policy {
	458	};
	459
	460	#ifdef CONFIG_BLOCK
	461
	462	static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg, void *key) { return NULL; }
	463	static inline struct blkcg_gq blk_queue_root_blkg(struct request_queue q)
	464	{ return NULL; }
	465	static inline int blkcg_init_queue(struct request_queue *q) { return 0; }
	466	static inline void blkcg_exit_queue(struct request_queue *q) { }
	467	static inline int blkcg_policy_register(struct blkcg_policy *pol) { return 0; }
	468	static inline void blkcg_policy_unregister(struct blkcg_policy *pol) { }
	469	static inline int blkcg_activate_policy(struct request_queue *q,
	470	const struct blkcg_policy *pol) { return 0; }
	471	static inline void blkcg_deactivate_policy(struct request_queue *q,
	472	const struct blkcg_policy *pol) { }
	473
	474	static inline struct blkcg __bio_blkcg(struct bio bio) { return NULL; }
	475
	476	static inline struct blkg_policy_data blkg_to_pd(struct blkcg_gq blkg,
	477	struct blkcg_policy *pol) { return NULL; }
	478	static inline struct blkcg_gq pd_to_blkg(struct blkg_policy_data pd) { return NULL; }
	479	static inline char blkg_path(struct blkcg_gq blkg) { return NULL; }
	480	static inline void blkg_get(struct blkcg_gq *blkg) { }
	481	static inline void blkg_put(struct blkcg_gq *blkg) { }
	482
	483	static inline bool blkcg_punt_bio_submit(struct bio *bio) { return false; }
	484	static inline void blkcg_bio_issue_init(struct bio *bio) { }
	485	static inline void blk_cgroup_bio_start(struct bio *bio) { }
	486	static inline bool blk_cgroup_mergeable(struct request rq, struct bio bio) { return true; }
	487
	488	#define blk_queue_for_each_rl(rl, q) \
	489	for ((rl) = &(q)->root_rl; (rl); (rl) = NULL)
	490
	491	#endif /* CONFIG_BLOCK */
	492	#endif /* CONFIG_BLK_CGROUP */
	493
	494	#endif /* _BLK_CGROUP_PRIVATE_H */

block/blk-core.c+132−161

@@ -34,7 +34,6 @@ · 33 unmodified lines
34	#include <linux/delay.h>	34	#include <linux/delay.h>
35	#include <linux/ratelimit.h>	35	#include <linux/ratelimit.h>
36	#include <linux/pm_runtime.h>	36	#include <linux/pm_runtime.h>
37	#include <linux/blk-cgroup.h>
38	#include <linux/t10-pi.h>	37	#include <linux/t10-pi.h>
39	#include <linux/debugfs.h>	38	#include <linux/debugfs.h>
40	#include <linux/bpf.h>	39	#include <linux/bpf.h>
@@ -49,6 +48,7 @@ · 8 unmodified lines
49	#include "blk.h"	48	#include "blk.h"
50	#include "blk-mq-sched.h"	49	#include "blk-mq-sched.h"
51	#include "blk-pm.h"	50	#include "blk-pm.h"
		51	#include "blk-cgroup.h"
52	#include "blk-throttle.h"	52	#include "blk-throttle.h"
53	#include "blk-rq-qos.h"	53	#include "blk-rq-qos.h"
54		54
@@ -165,6 +165,7 @@ static const struct { · 110 unmodified lines
165	[BLK_STS_RESOURCE] = { -ENOMEM, "kernel resource" },	165	[BLK_STS_RESOURCE] = { -ENOMEM, "kernel resource" },
166	[BLK_STS_DEV_RESOURCE] = { -EBUSY, "device resource" },	166	[BLK_STS_DEV_RESOURCE] = { -EBUSY, "device resource" },
167	[BLK_STS_AGAIN] = { -EAGAIN, "nonblocking retry" },	167	[BLK_STS_AGAIN] = { -EAGAIN, "nonblocking retry" },
		168	[BLK_STS_OFFLINE] = { -ENODEV, "device offline" },
168		169
169	/* device mapper special case, should not leak out: */	170	/* device mapper special case, should not leak out: */
170	[BLK_STS_DM_REQUEUE] = { -EREMCHG, "dm internal retry" },	171	[BLK_STS_DM_REQUEUE] = { -EREMCHG, "dm internal retry" },
@@ -339,8 +340,6 @@ void blk_cleanup_queue(struct request_queue *q) · 168 unmodified lines
339	blk_mq_sched_free_rqs(q);	340	blk_mq_sched_free_rqs(q);
340	mutex_unlock(&q->sysfs_lock);	341	mutex_unlock(&q->sysfs_lock);
341		342
342	percpu_ref_exit(&q->q_usage_counter);
343
344	/* @q is and will stay empty, shutdown and put */	343	/* @q is and will stay empty, shutdown and put */
345	blk_put_queue(q);	344	blk_put_queue(q);
346	}	345	}
@@ -473,9 +472,6 @@ struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu) · 126 unmodified lines
473	timer_setup(&q->timeout, blk_rq_timed_out_timer, 0);	472	timer_setup(&q->timeout, blk_rq_timed_out_timer, 0);
474	INIT_WORK(&q->timeout_work, blk_timeout_work);	473	INIT_WORK(&q->timeout_work, blk_timeout_work);
475	INIT_LIST_HEAD(&q->icq_list);	474	INIT_LIST_HEAD(&q->icq_list);
476	#ifdef CONFIG_BLK_CGROUP
477	INIT_LIST_HEAD(&q->blkg_list);
478	#endif
479		475
480	kobject_init(&q->kobj, &blk_queue_ktype);	476	kobject_init(&q->kobj, &blk_queue_ktype);
481		477
@@ -496,17 +492,12 @@ struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu) · 14 unmodified lines
496	PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))	492	PERCPU_REF_INIT_ATOMIC, GFP_KERNEL))
497	goto fail_stats;	493	goto fail_stats;
498		494
499	if (blkcg_init_queue(q))
500	goto fail_ref;
501
502	blk_queue_dma_alignment(q, 511);	495	blk_queue_dma_alignment(q, 511);
503	blk_set_default_limits(&q->limits);	496	blk_set_default_limits(&q->limits);
504	q->nr_requests = BLKDEV_DEFAULT_RQ;	497	q->nr_requests = BLKDEV_DEFAULT_RQ;
505		498
506	return q;	499	return q;
507		500
508	fail_ref:
509	percpu_ref_exit(&q->q_usage_counter);
510	fail_stats:	501	fail_stats:
511	blk_free_queue_stats(q->stats);	502	blk_free_queue_stats(q->stats);
512	fail_split:	503	fail_split:
@@ -540,17 +531,6 @@ bool blk_get_queue(struct request_queue *q) · 27 unmodified lines
540	}	531	}
541	EXPORT_SYMBOL(blk_get_queue);	532	EXPORT_SYMBOL(blk_get_queue);
542		533
543	static void handle_bad_sector(struct bio *bio, sector_t maxsector)
544	{
545	char b[BDEVNAME_SIZE];
546
547	pr_info_ratelimited("%s: attempt to access beyond end of device\n"
548	"%s: rw=%d, want=%llu, limit=%llu\n",
549	current->comm,
550	bio_devname(bio, b), bio->bi_opf,
551	bio_end_sector(bio), maxsector);
552	}
553
554	#ifdef CONFIG_FAIL_MAKE_REQUEST	534	#ifdef CONFIG_FAIL_MAKE_REQUEST
555		535
556	static DECLARE_FAULT_ATTR(fail_make_request);	536	static DECLARE_FAULT_ATTR(fail_make_request);
@@ -580,14 +560,10 @@ late_initcall(fail_make_request_debugfs); · 23 unmodified lines
580	static inline bool bio_check_ro(struct bio *bio)	560	static inline bool bio_check_ro(struct bio *bio)
581	{	561	{
582	if (op_is_write(bio_op(bio)) && bdev_read_only(bio->bi_bdev)) {	562	if (op_is_write(bio_op(bio)) && bdev_read_only(bio->bi_bdev)) {
583	char b[BDEVNAME_SIZE];
584
585	if (op_is_flush(bio->bi_opf) && !bio_sectors(bio))	563	if (op_is_flush(bio->bi_opf) && !bio_sectors(bio))
586	return false;	564	return false;
587		565	pr_warn("Trying to write to read-only block-device %pg\n",
588	WARN_ONCE(1,	566	bio->bi_bdev);
589	"Trying to write to read-only block-device %s (partno %d)\n",
590	bio_devname(bio, b), bio->bi_bdev->bd_partno);
591	/* Older lvm-tools actually trigger this */	567	/* Older lvm-tools actually trigger this */
592	return false;	568	return false;
593	}	569	}
@@ -616,7 +592,11 @@ static inline int bio_check_eod(struct bio *bio) · 22 unmodified lines
616	if (nr_sectors && maxsector &&	592	if (nr_sectors && maxsector &&
617	(nr_sectors > maxsector \|\|	593	(nr_sectors > maxsector \|\|
618	bio->bi_iter.bi_sector > maxsector - nr_sectors)) {	594	bio->bi_iter.bi_sector > maxsector - nr_sectors)) {
619	handle_bad_sector(bio, maxsector);	595	pr_info_ratelimited("%s: attempt to access beyond end of device\n"
		596	"%pg: rw=%d, want=%llu, limit=%llu\n",
		597	current->comm,
		598	bio->bi_bdev, bio->bi_opf,
		599	bio_end_sector(bio), maxsector);
620	return -EIO;	600	return -EIO;
621	}	601	}
622	return 0;	602	return 0;
@@ -676,7 +656,123 @@ static inline blk_status_t blk_check_zone_append(struct request_queue *q, · 53 unmodified lines
676	return BLK_STS_OK;	656	return BLK_STS_OK;
677	}	657	}
678		658
679	noinline_for_stack bool submit_bio_checks(struct bio *bio)	659	static void __submit_bio(struct bio *bio)
		660	{
		661	struct gendisk *disk = bio->bi_bdev->bd_disk;
		662
		663	if (unlikely(!blk_crypto_bio_prep(&bio)))
		664	return;
		665
		666	if (!disk->fops->submit_bio) {
		667	blk_mq_submit_bio(bio);
		668	} else if (likely(bio_queue_enter(bio) == 0)) {
		669	disk->fops->submit_bio(bio);
		670	blk_queue_exit(disk->queue);
		671	}
		672	}
		673
		674	/*
		675	* The loop in this function may be a bit non-obvious, and so deserves some
		676	* explanation:
		677	*
		678	* - Before entering the loop, bio->bi_next is NULL (as all callers ensure
		679	* that), so we have a list with a single bio.
		680	* - We pretend that we have just taken it off a longer list, so we assign
		681	* bio_list to a pointer to the bio_list_on_stack, thus initialising the
		682	* bio_list of new bios to be added. ->submit_bio() may indeed add some more
		683	* bios through a recursive call to submit_bio_noacct. If it did, we find a
		684	* non-NULL value in bio_list and re-enter the loop from the top.
		685	* - In this case we really did just take the bio of the top of the list (no
		686	* pretending) and so remove it from bio_list, and call into ->submit_bio()
		687	* again.
		688	*
		689	* bio_list_on_stack[0] contains bios submitted by the current ->submit_bio.
		690	* bio_list_on_stack[1] contains bios that were submitted before the current
		691	* ->submit_bio_bio, but that haven't been processed yet.
		692	*/
		693	static void __submit_bio_noacct(struct bio *bio)
		694	{
		695	struct bio_list bio_list_on_stack[2];
		696
		697	BUG_ON(bio->bi_next);
		698
		699	bio_list_init(&bio_list_on_stack[0]);
		700	current->bio_list = bio_list_on_stack;
		701
		702	do {
		703	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
		704	struct bio_list lower, same;
		705
		706	/*
		707	* Create a fresh bio_list for all subordinate requests.
		708	*/
		709	bio_list_on_stack[1] = bio_list_on_stack[0];
		710	bio_list_init(&bio_list_on_stack[0]);
		711
		712	__submit_bio(bio);
		713
		714	/*
		715	* Sort new bios into those for a lower level and those for the
		716	* same level.
		717	*/
		718	bio_list_init(&lower);
		719	bio_list_init(&same);
		720	while ((bio = bio_list_pop(&bio_list_on_stack[0])) != NULL)
		721	if (q == bdev_get_queue(bio->bi_bdev))
		722	bio_list_add(&same, bio);
		723	else
		724	bio_list_add(&lower, bio);
		725
		726	/*
		727	* Now assemble so we handle the lowest level first.
		728	*/
		729	bio_list_merge(&bio_list_on_stack[0], &lower);
		730	bio_list_merge(&bio_list_on_stack[0], &same);
		731	bio_list_merge(&bio_list_on_stack[0], &bio_list_on_stack[1]);
		732	} while ((bio = bio_list_pop(&bio_list_on_stack[0])));
		733
		734	current->bio_list = NULL;
		735	}
		736
		737	static void __submit_bio_noacct_mq(struct bio *bio)
		738	{
		739	struct bio_list bio_list[2] = { };
		740
		741	current->bio_list = bio_list;
		742
		743	do {
		744	__submit_bio(bio);
		745	} while ((bio = bio_list_pop(&bio_list[0])));
		746
		747	current->bio_list = NULL;
		748	}
		749
		750	void submit_bio_noacct_nocheck(struct bio *bio)
		751	{
		752	/*
		753	* We only want one ->submit_bio to be active at a time, else stack
		754	* usage with stacked devices could be a problem. Use current->bio_list
		755	* to collect a list of requests submited by a ->submit_bio method while
		756	* it is active, and then process them after it returned.
		757	*/
		758	if (current->bio_list)
		759	bio_list_add(&current->bio_list[0], bio);
		760	else if (!bio->bi_bdev->bd_disk->fops->submit_bio)
		761	__submit_bio_noacct_mq(bio);
		762	else
		763	__submit_bio_noacct(bio);
		764	}
		765
		766	/**
		767	* submit_bio_noacct - re-submit a bio to the block device layer for I/O
		768	* @bio: The bio describing the location in memory and on the device.
		769	*
		770	* This is a version of submit_bio() that shall only be used for I/O that is
		771	* resubmitted to lower level drivers by stacking block drivers. All file
		772	* systems and other upper level users of the block layer should use
		773	* submit_bio() instead.
		774	*/
		775	void submit_bio_noacct(struct bio *bio)
680	{	776	{
681	struct block_device *bdev = bio->bi_bdev;	777	struct block_device *bdev = bio->bi_bdev;
682	struct request_queue *q = bdev_get_queue(bdev);	778	struct request_queue *q = bdev_get_queue(bdev);
@@ -761,7 +857,7 @@ noinline_for_stack bool submit_bio_checks(struct bio *bio) · 78 unmodified lines
761	}	857	}
762		858
763	if (blk_throtl_bio(bio))	859	if (blk_throtl_bio(bio))
764	return false;	860	return;
765		861
766	blk_cgroup_bio_start(bio);	862	blk_cgroup_bio_start(bio);
767	blkcg_bio_issue_init(bio);	863	blkcg_bio_issue_init(bio);
@@ -773,138 +869,14 @@ noinline_for_stack bool submit_bio_checks(struct bio *bio) · 5 unmodified lines
773	*/	869	*/
774	bio_set_flag(bio, BIO_TRACE_COMPLETION);	870	bio_set_flag(bio, BIO_TRACE_COMPLETION);
775	}	871	}
776	return true;	872	submit_bio_noacct_nocheck(bio);
		873	return;
777		874
778	not_supported:	875	not_supported:
779	status = BLK_STS_NOTSUPP;	876	status = BLK_STS_NOTSUPP;
780	end_io:	877	end_io:
781	bio->bi_status = status;	878	bio->bi_status = status;
782	bio_endio(bio);	879	bio_endio(bio);
783	return false;
784	}
785
786	static void __submit_bio_fops(struct gendisk disk, struct bio bio)
787	{
788	if (blk_crypto_bio_prep(&bio)) {
789	if (likely(bio_queue_enter(bio) == 0)) {
790	disk->fops->submit_bio(bio);
791	blk_queue_exit(disk->queue);
792	}
793	}
794	}
795
796	static void __submit_bio(struct bio *bio)
797	{
798	struct gendisk *disk = bio->bi_bdev->bd_disk;
799
800	if (unlikely(!submit_bio_checks(bio)))
801	return;
802
803	if (!disk->fops->submit_bio)
804	blk_mq_submit_bio(bio);
805	else
806	__submit_bio_fops(disk, bio);
807	}
808
809	/*
810	* The loop in this function may be a bit non-obvious, and so deserves some
811	* explanation:
812	*
813	* - Before entering the loop, bio->bi_next is NULL (as all callers ensure
814	* that), so we have a list with a single bio.
815	* - We pretend that we have just taken it off a longer list, so we assign
816	* bio_list to a pointer to the bio_list_on_stack, thus initialising the
817	* bio_list of new bios to be added. ->submit_bio() may indeed add some more
818	* bios through a recursive call to submit_bio_noacct. If it did, we find a
819	* non-NULL value in bio_list and re-enter the loop from the top.
820	* - In this case we really did just take the bio of the top of the list (no
821	* pretending) and so remove it from bio_list, and call into ->submit_bio()
822	* again.
823	*
824	* bio_list_on_stack[0] contains bios submitted by the current ->submit_bio.
825	* bio_list_on_stack[1] contains bios that were submitted before the current
826	* ->submit_bio_bio, but that haven't been processed yet.
827	*/
828	static void __submit_bio_noacct(struct bio *bio)
829	{
830	struct bio_list bio_list_on_stack[2];
831
832	BUG_ON(bio->bi_next);
833
834	bio_list_init(&bio_list_on_stack[0]);
835	current->bio_list = bio_list_on_stack;
836
837	do {
838	struct request_queue *q = bdev_get_queue(bio->bi_bdev);
839	struct bio_list lower, same;
840
841	/*
842	* Create a fresh bio_list for all subordinate requests.
843	*/
844	bio_list_on_stack[1] = bio_list_on_stack[0];
845	bio_list_init(&bio_list_on_stack[0]);
846
847	__submit_bio(bio);
848
849	/*
850	* Sort new bios into those for a lower level and those for the
851	* same level.
852	*/
853	bio_list_init(&lower);
854	bio_list_init(&same);
855	while ((bio = bio_list_pop(&bio_list_on_stack[0])) != NULL)
856	if (q == bdev_get_queue(bio->bi_bdev))
857	bio_list_add(&same, bio);
858	else
859	bio_list_add(&lower, bio);
860
861	/*
862	* Now assemble so we handle the lowest level first.
863	*/
864	bio_list_merge(&bio_list_on_stack[0], &lower);
865	bio_list_merge(&bio_list_on_stack[0], &same);
866	bio_list_merge(&bio_list_on_stack[0], &bio_list_on_stack[1]);
867	} while ((bio = bio_list_pop(&bio_list_on_stack[0])));
868
869	current->bio_list = NULL;
870	}
871
872	static void __submit_bio_noacct_mq(struct bio *bio)
873	{
874	struct bio_list bio_list[2] = { };
875
876	current->bio_list = bio_list;
877
878	do {
879	__submit_bio(bio);
880	} while ((bio = bio_list_pop(&bio_list[0])));
881
882	current->bio_list = NULL;
883	}
884
885	/**
886	* submit_bio_noacct - re-submit a bio to the block device layer for I/O
887	* @bio: The bio describing the location in memory and on the device.
888	*
889	* This is a version of submit_bio() that shall only be used for I/O that is
890	* resubmitted to lower level drivers by stacking block drivers. All file
891	* systems and other upper level users of the block layer should use
892	* submit_bio() instead.
893	*/
894	void submit_bio_noacct(struct bio *bio)
895	{
896	/*
897	* We only want one ->submit_bio to be active at a time, else stack
898	* usage with stacked devices could be a problem. Use current->bio_list
899	* to collect a list of requests submited by a ->submit_bio method while
900	* it is active, and then process them after it returned.
901	*/
902	if (current->bio_list)
903	bio_list_add(&current->bio_list[0], bio);
904	else if (!bio->bi_bdev->bd_disk->fops->submit_bio)
905	__submit_bio_noacct_mq(bio);
906	else
907	__submit_bio_noacct(bio);
908	}	880	}
909	EXPORT_SYMBOL(submit_bio_noacct);	881	EXPORT_SYMBOL(submit_bio_noacct);
910		882
@@ -989,8 +961,7 @@ int bio_poll(struct bio bio, struct io_comp_batch iob, unsigned int flags) · 78 unmodified lines
989	!test_bit(QUEUE_FLAG_POLL, &q->queue_flags))	961	!test_bit(QUEUE_FLAG_POLL, &q->queue_flags))
990	return 0;	962	return 0;
991		963
992	if (current->plug)	964	blk_flush_plug(current->plug, false);
993	blk_flush_plug(current->plug, false);
994		965
995	if (blk_queue_enter(q, BLK_MQ_REQ_NOWAIT))	966	if (blk_queue_enter(q, BLK_MQ_REQ_NOWAIT))
996	return 0;	967	return 0;
@@ -1272,7 +1243,7 @@ struct blk_plug_cb blk_check_plugged(blk_plug_cb_fn unplug, void data, · 275 unmodified lines
1272	}	1243	}
1273	EXPORT_SYMBOL(blk_check_plugged);	1244	EXPORT_SYMBOL(blk_check_plugged);
1274		1245
1275	void blk_flush_plug(struct blk_plug *plug, bool from_schedule)	1246	void __blk_flush_plug(struct blk_plug *plug, bool from_schedule)
1276	{	1247	{
1277	if (!list_empty(&plug->cb_list))	1248	if (!list_empty(&plug->cb_list))
1278	flush_plug_callbacks(plug, from_schedule);	1249	flush_plug_callbacks(plug, from_schedule);
@@ -1301,7 +1272,7 @@ void blk_flush_plug(struct blk_plug *plug, bool from_schedule) · 22 unmodified lines
1301	void blk_finish_plug(struct blk_plug *plug)	1272	void blk_finish_plug(struct blk_plug *plug)
1302	{	1273	{
1303	if (plug == current->plug) {	1274	if (plug == current->plug) {
1304	blk_flush_plug(plug, false);	1275	__blk_flush_plug(plug, false);
1305	current->plug = NULL;	1276	current->plug = NULL;
1306	}	1277	}
1307	}	1278	}

block/blk-crypto-fallback.c+1−1

 #define pr_fmt(fmt) "blk-crypto-fallback: " fmt
 #include <crypto/skcipher.h>
-#include <linux/blk-cgroup.h>
 #include <linux/blk-crypto.h>
 #include <linux/blk-crypto-profile.h>
 #include <linux/blkdev.h>
 #include <linux/random.h>
 #include <linux/scatterlist.h>
+#include "blk-cgroup.h"
 #include "blk-crypto-internal.h"
 static unsigned int num_prealloc_bounce_pg = 32;

block/blk-crypto-internal.h+12−0

 /* Represents a crypto mode supported by blk-crypto  */
 struct blk_crypto_mode {
+	const char *name; /* name of this mode, shown in sysfs */
 	const char *cipher_str; /* crypto API name (for fallback case) */
 	unsigned int keysize; /* key size in bytes */
 	unsigned int ivsize; /* iv size in bytes */
 #ifdef CONFIG_BLK_INLINE_ENCRYPTION
+int blk_crypto_sysfs_register(struct request_queue *q);
+void blk_crypto_sysfs_unregister(struct request_queue *q);
 void bio_crypt_dun_increment(u64 dun[BLK_CRYPTO_DUN_ARRAY_SIZE],
 			     unsigned int inc);
 #else /* CONFIG_BLK_INLINE_ENCRYPTION */
+static inline int blk_crypto_sysfs_register(struct request_queue *q)
+{
+	return 0;
+}
+static inline void blk_crypto_sysfs_unregister(struct request_queue *q) { }
 static inline bool bio_crypt_rq_ctx_compatible(struct request *rq,
 					       struct bio *bio)
 {

block/blk-crypto-sysfs.c+172−0

@@ -0,0 +1,172 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/*
	3	* Copyright 2021 Google LLC
	4	*
	5	* sysfs support for blk-crypto. This file contains the code which exports the
	6	* crypto capabilities of devices via /sys/block/$disk/queue/crypto/.
	7	*/
	8
	9	#include <linux/blk-crypto-profile.h>
	10
	11	#include "blk-crypto-internal.h"
	12
	13	struct blk_crypto_kobj {
	14	struct kobject kobj;
	15	struct blk_crypto_profile *profile;
	16	};
	17
	18	struct blk_crypto_attr {
	19	struct attribute attr;
	20	ssize_t (show)(struct blk_crypto_profile profile,
	21	struct blk_crypto_attr attr, char page);
	22	};
	23
	24	static struct blk_crypto_profile kobj_to_crypto_profile(struct kobject kobj)
	25	{
	26	return container_of(kobj, struct blk_crypto_kobj, kobj)->profile;
	27	}
	28
	29	static struct blk_crypto_attr attr_to_crypto_attr(struct attribute attr)
	30	{
	31	return container_of(attr, struct blk_crypto_attr, attr);
	32	}
	33
	34	static ssize_t max_dun_bits_show(struct blk_crypto_profile *profile,
	35	struct blk_crypto_attr attr, char page)
	36	{
	37	return sysfs_emit(page, "%u\n", 8 * profile->max_dun_bytes_supported);
	38	}
	39
	40	static ssize_t num_keyslots_show(struct blk_crypto_profile *profile,
	41	struct blk_crypto_attr attr, char page)
	42	{
	43	return sysfs_emit(page, "%u\n", profile->num_slots);
	44	}
	45
	46	#define BLK_CRYPTO_RO_ATTR(_name) \
	47	static struct blk_crypto_attr _name##_attr = __ATTR_RO(_name)
	48
	49	BLK_CRYPTO_RO_ATTR(max_dun_bits);
	50	BLK_CRYPTO_RO_ATTR(num_keyslots);
	51
	52	static struct attribute *blk_crypto_attrs[] = {
	53	&max_dun_bits_attr.attr,
	54	&num_keyslots_attr.attr,
	55	NULL,
	56	};
	57
	58	static const struct attribute_group blk_crypto_attr_group = {
	59	.attrs = blk_crypto_attrs,
	60	};
	61
	62	/*
	63	* The encryption mode attributes. To avoid hard-coding the list of encryption
	64	* modes, these are initialized at boot time by blk_crypto_sysfs_init().
	65	*/
	66	static struct blk_crypto_attr __blk_crypto_mode_attrs[BLK_ENCRYPTION_MODE_MAX];
	67	static struct attribute *blk_crypto_mode_attrs[BLK_ENCRYPTION_MODE_MAX + 1];
	68
	69	static umode_t blk_crypto_mode_is_visible(struct kobject *kobj,
	70	struct attribute *attr, int n)
	71	{
	72	struct blk_crypto_profile *profile = kobj_to_crypto_profile(kobj);
	73	struct blk_crypto_attr *a = attr_to_crypto_attr(attr);
	74	int mode_num = a - __blk_crypto_mode_attrs;
	75
	76	if (profile->modes_supported[mode_num])
	77	return 0444;
	78	return 0;
	79	}
	80
	81	static ssize_t blk_crypto_mode_show(struct blk_crypto_profile *profile,
	82	struct blk_crypto_attr attr, char page)
	83	{
	84	int mode_num = attr - __blk_crypto_mode_attrs;
	85
	86	return sysfs_emit(page, "0x%x\n", profile->modes_supported[mode_num]);
	87	}
	88
	89	static const struct attribute_group blk_crypto_modes_attr_group = {
	90	.name = "modes",
	91	.attrs = blk_crypto_mode_attrs,
	92	.is_visible = blk_crypto_mode_is_visible,
	93	};
	94
	95	static const struct attribute_group *blk_crypto_attr_groups[] = {
	96	&blk_crypto_attr_group,
	97	&blk_crypto_modes_attr_group,
	98	NULL,
	99	};
	100
	101	static ssize_t blk_crypto_attr_show(struct kobject *kobj,
	102	struct attribute attr, char page)
	103	{
	104	struct blk_crypto_profile *profile = kobj_to_crypto_profile(kobj);
	105	struct blk_crypto_attr *a = attr_to_crypto_attr(attr);
	106
	107	return a->show(profile, a, page);
	108	}
	109
	110	static const struct sysfs_ops blk_crypto_attr_ops = {
	111	.show = blk_crypto_attr_show,
	112	};
	113
	114	static void blk_crypto_release(struct kobject *kobj)
	115	{
	116	kfree(container_of(kobj, struct blk_crypto_kobj, kobj));
	117	}
	118
	119	static struct kobj_type blk_crypto_ktype = {
	120	.default_groups = blk_crypto_attr_groups,
	121	.sysfs_ops = &blk_crypto_attr_ops,
	122	.release = blk_crypto_release,
	123	};
	124
	125	/*
	126	* If the request_queue has a blk_crypto_profile, create the "crypto"
	127	* subdirectory in sysfs (/sys/block/$disk/queue/crypto/).
	128	*/
	129	int blk_crypto_sysfs_register(struct request_queue *q)
	130	{
	131	struct blk_crypto_kobj *obj;
	132	int err;
	133
	134	if (!q->crypto_profile)
	135	return 0;
	136
	137	obj = kzalloc(sizeof(*obj), GFP_KERNEL);
	138	if (!obj)
	139	return -ENOMEM;
	140	obj->profile = q->crypto_profile;
	141
	142	err = kobject_init_and_add(&obj->kobj, &blk_crypto_ktype, &q->kobj,
	143	"crypto");
	144	if (err) {
	145	kobject_put(&obj->kobj);
	146	return err;
	147	}
	148	q->crypto_kobject = &obj->kobj;
	149	return 0;
	150	}
	151
	152	void blk_crypto_sysfs_unregister(struct request_queue *q)
	153	{
	154	kobject_put(q->crypto_kobject);
	155	}
	156
	157	static int __init blk_crypto_sysfs_init(void)
	158	{
	159	int i;
	160
	161	BUILD_BUG_ON(BLK_ENCRYPTION_MODE_INVALID != 0);
	162	for (i = 1; i < BLK_ENCRYPTION_MODE_MAX; i++) {
	163	struct blk_crypto_attr *attr = &__blk_crypto_mode_attrs[i];
	164
	165	attr->attr.name = blk_crypto_modes[i].name;
	166	attr->attr.mode = 0444;
	167	attr->show = blk_crypto_mode_show;
	168	blk_crypto_mode_attrs[i - 1] = &attr->attr;
	169	}
	170	return 0;
	171	}
	172	subsys_initcall(blk_crypto_sysfs_init);

block/blk-crypto.c+3−1

 const struct blk_crypto_mode blk_crypto_modes[] = {
 	[BLK_ENCRYPTION_MODE_AES_256_XTS] = {
+		.name = "AES-256-XTS",
 		.cipher_str = "xts(aes)",
 		.keysize = 64,
 		.ivsize = 16,
 	},
 	[BLK_ENCRYPTION_MODE_AES_128_CBC_ESSIV] = {
+		.name = "AES-128-CBC-ESSIV",
 		.cipher_str = "essiv(cbc(aes),sha256)",
 		.keysize = 16,
 		.ivsize = 16,
 	},
 	[BLK_ENCRYPTION_MODE_ADIANTUM] = {
+		.name = "Adiantum",
 		.cipher_str = "adiantum(xchacha12,aes)",
 		.keysize = 32,
 		.ivsize = 32,
 	*dst->bi_crypt_context = *src->bi_crypt_context;
 	return 0;
 }
-EXPORT_SYMBOL_GPL(__bio_crypt_clone);
 /* Increments @dun by @inc, treating @dun as a multi-limb integer. */
 void bio_crypt_dun_increment(u64 dun[BLK_CRYPTO_DUN_ARRAY_SIZE],

block/blk-flush.c+1−3

 {
 	struct bio bio;
-	bio_init(&bio, NULL, 0);
+	bio_init(&bio, bdev, NULL, 0, REQ_OP_WRITE | REQ_PREFLUSH);
-	bio_set_dev(&bio, bdev);
-	bio.bi_opf = REQ_OP_WRITE | REQ_PREFLUSH;
 	return submit_bio_wait(&bio);
 }
 EXPORT_SYMBOL(blkdev_issue_flush);

block/blk-iocost.c+1−1

 #include <linux/time64.h>
 #include <linux/parser.h>
 #include <linux/sched/signal.h>
-#include <linux/blk-cgroup.h>
 #include <asm/local.h>
 #include <asm/local64.h>
 #include "blk-rq-qos.h"
 #include "blk-stat.h"
 #include "blk-wbt.h"
+#include "blk-cgroup.h"
 #ifdef CONFIG_TRACEPOINTS

block/blk-iolatency.c+2−2

 #include <linux/sched/signal.h>
 #include <trace/events/block.h>
 #include <linux/blk-mq.h>
-#include <linux/blk-cgroup.h>
 #include "blk-rq-qos.h"
 #include "blk-stat.h"
+#include "blk-cgroup.h"
 #include "blk.h"
 #define DEFAULT_SCALE_COOKIE 1000000U
 	int inflight = 0;
 	blkg = bio->bi_blkg;
-	if (!blkg || !bio_flagged(bio, BIO_TRACKED))
+	if (!blkg || !bio_flagged(bio, BIO_QOS_THROTTLED))
 		return;
 	iolat = blkg_to_lat(bio->bi_blkg);

block/blk-ioprio.c+1−1

  *   Documentation/admin-guide/cgroup-v2.rst.
  */
-#include <linux/blk-cgroup.h>
 #include <linux/blk-mq.h>
 #include <linux/blk_types.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include "blk-cgroup.h"
 #include "blk-ioprio.h"
 #include "blk-rq-qos.h"

block/blk-lib.c+5−41

 #include "blk.h"
-struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp)
-{
-	struct bio *new = bio_alloc(gfp, nr_pages);
-	if (bio) {
-		bio_chain(bio, new);
-		submit_bio(bio);
-	}
-	return new;
-}
-EXPORT_SYMBOL_GPL(blk_next_bio);
 int __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
 		sector_t nr_sects, gfp_t gfp_mask, int flags,
 		struct bio **biop)
 	unsigned int op;
 	sector_t bs_mask, part_offset = 0;
-	if (!q)
-		return -ENXIO;
 	if (bdev_read_only(bdev))
 		return -EPERM;
 		WARN_ON_ONCE((req_sects << 9) > UINT_MAX);
-		bio = blk_next_bio(bio, 0, gfp_mask);
+		bio = blk_next_bio(bio, bdev, 0, op, gfp_mask);
 		bio->bi_iter.bi_sector = sector;
-		bio_set_dev(bio, bdev);
-		bio_set_op_attrs(bio, op, 0);
 		bio->bi_iter.bi_size = req_sects << 9;
 		sector += req_sects;
 		nr_sects -= req_sects;
 	struct bio *bio = *biop;
 	sector_t bs_mask;
-	if (!q)
-		return -ENXIO;
 	if (bdev_read_only(bdev))
 		return -EPERM;
 	max_write_same_sectors = bio_allowed_max_sectors(q);
 	while (nr_sects) {
-		bio = blk_next_bio(bio, 1, gfp_mask);
+		bio = blk_next_bio(bio, bdev, 1, REQ_OP_WRITE_SAME, gfp_mask);
 		bio->bi_iter.bi_sector = sector;
-		bio_set_dev(bio, bdev);
 		bio->bi_vcnt = 1;
 		bio->bi_io_vec->bv_page = page;
 		bio->bi_io_vec->bv_offset = 0;
 		bio->bi_io_vec->bv_len = bdev_logical_block_size(bdev);
-		bio_set_op_attrs(bio, REQ_OP_WRITE_SAME, 0);
 		if (nr_sects > max_write_same_sectors) {
 			bio->bi_iter.bi_size = max_write_same_sectors << 9;
 {
 	struct bio *bio = *biop;
 	unsigned int max_write_zeroes_sectors;
-	struct request_queue *q = bdev_get_queue(bdev);
-	if (!q)
-		return -ENXIO;
 	if (bdev_read_only(bdev))
 		return -EPERM;
 		return -EOPNOTSUPP;
 	while (nr_sects) {
-		bio = blk_next_bio(bio, 0, gfp_mask);
+		bio = blk_next_bio(bio, bdev, 0, REQ_OP_WRITE_ZEROES, gfp_mask);
 		bio->bi_iter.bi_sector = sector;
-		bio_set_dev(bio, bdev);
-		bio->bi_opf = REQ_OP_WRITE_ZEROES;
 		if (flags & BLKDEV_ZERO_NOUNMAP)
 			bio->bi_opf |= REQ_NOUNMAP;
 		sector_t sector, sector_t nr_sects, gfp_t gfp_mask,
 		struct bio **biop)
 {
-	struct request_queue *q = bdev_get_queue(bdev);
 	struct bio *bio = *biop;
 	int bi_size = 0;
 	unsigned int sz;
-	if (!q)
-		return -ENXIO;
 	if (bdev_read_only(bdev))
 		return -EPERM;
 	while (nr_sects != 0) {
-		bio = blk_next_bio(bio, __blkdev_sectors_to_bio_pages(nr_sects),
+		bio = blk_next_bio(bio, bdev, __blkdev_sectors_to_bio_pages(nr_sects),
-				   gfp_mask);
+				   REQ_OP_WRITE, gfp_mask);
 		bio->bi_iter.bi_sector = sector;
-		bio_set_dev(bio, bdev);
-		bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
 		while (nr_sects != 0) {
 			sz = min((sector_t) PAGE_SIZE, nr_sects << 9);

block/blk-merge.c+25−8

 #include <linux/blk-integrity.h>
 #include <linux/scatterlist.h>
 #include <linux/part_stat.h>
+#include <linux/blk-cgroup.h>
 #include <trace/events/block.h>
 		trace_block_split(split, (*bio)->bi_iter.bi_sector);
 		submit_bio_noacct(*bio);
 		*bio = split;
-		blk_throtl_charge_bio_split(*bio);
 	}
 }
 static inline int ll_new_hw_segment(struct request *req, struct bio *bio,
 		unsigned int nr_phys_segs)
 {
+	if (!blk_cgroup_mergeable(req, bio))
+		goto no_merge;
 	if (blk_integrity_merge_bio(req->q, req, bio) == false)
 		goto no_merge;
 	if (total_phys_segments > blk_rq_get_max_segments(req))
 		return 0;
+	if (!blk_cgroup_mergeable(req, next->bio))
+		return 0;
 	if (blk_integrity_merge_rq(q, req, next) == false)
 		return 0;
 	if (bio_data_dir(bio) != rq_data_dir(rq))
 		return false;
+	/* don't merge across cgroup boundaries */
+	if (!blk_cgroup_mergeable(rq, bio))
+		return false;
 	/* only merge integrity protected bio into ditto rq */
 	if (blk_integrity_merge_bio(rq->q, rq, bio) == false)
 		return false;
 	if (!plug || rq_list_empty(plug->mq_list))
 		return false;
-	/* check the previously added entry for a quick merge attempt */
+	rq_list_for_each(&plug->mq_list, rq) {
-	rq = rq_list_peek(&plug->mq_list);
+		if (rq->q == q) {
-	if (rq->q == q) {
+			if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) ==
-		if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) ==
+			    BIO_MERGE_OK)
-				BIO_MERGE_OK)
+				return true;
-			return true;
+			break;
+		}
+		/*
+		 * Only keep iterating plug list for merges if we have multiple
+		 * queues
+		 */
+		if (!plug->multiple_queues)
+			break;
 	}
 	return false;
 }

block/blk-mq-debugfs.c+3−3

@@ -707,7 +707,7 @@ static void debugfs_create_files(struct dentry parent, void data, · 706 unmodified lines
707	void blk_mq_debugfs_register(struct request_queue *q)	707	void blk_mq_debugfs_register(struct request_queue *q)
708	{	708	{
709	struct blk_mq_hw_ctx *hctx;	709	struct blk_mq_hw_ctx *hctx;
710	int i;	710	unsigned long i;
711		711
712	debugfs_create_files(q->debugfs_dir, q, blk_mq_debugfs_queue_attrs);	712	debugfs_create_files(q->debugfs_dir, q, blk_mq_debugfs_queue_attrs);
713		713
@@ -780,7 +780,7 @@ void blk_mq_debugfs_unregister_hctx(struct blk_mq_hw_ctx *hctx) · 66 unmodified lines
780	void blk_mq_debugfs_register_hctxs(struct request_queue *q)	780	void blk_mq_debugfs_register_hctxs(struct request_queue *q)
781	{	781	{
782	struct blk_mq_hw_ctx *hctx;	782	struct blk_mq_hw_ctx *hctx;
783	int i;	783	unsigned long i;
784		784
785	queue_for_each_hw_ctx(q, hctx, i)	785	queue_for_each_hw_ctx(q, hctx, i)
786	blk_mq_debugfs_register_hctx(q, hctx);	786	blk_mq_debugfs_register_hctx(q, hctx);
@@ -789,7 +789,7 @@ void blk_mq_debugfs_register_hctxs(struct request_queue *q) · 2 unmodified lines
789	void blk_mq_debugfs_unregister_hctxs(struct request_queue *q)	789	void blk_mq_debugfs_unregister_hctxs(struct request_queue *q)
790	{	790	{
791	struct blk_mq_hw_ctx *hctx;	791	struct blk_mq_hw_ctx *hctx;
792	int i;	792	unsigned long i;
793		793
794	queue_for_each_hw_ctx(q, hctx, i)	794	queue_for_each_hw_ctx(q, hctx, i)
795	blk_mq_debugfs_unregister_hctx(hctx);	795	blk_mq_debugfs_unregister_hctx(hctx);

block/blk-mq-debugfs.h+2−0

@@ -6,6 +6,8 @@ · 5 unmodified lines
6		6
7	#include <linux/seq_file.h>	7	#include <linux/seq_file.h>
8		8
		9	struct blk_mq_hw_ctx;
		10
9	struct blk_mq_debugfs_attr {	11	struct blk_mq_debugfs_attr {
10	const char *name;	12	const char *name;
11	umode_t mode;	13	umode_t mode;

block/blk-mq-sched.c+13−5

 static int blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)
 {
+	unsigned long end = jiffies + HZ;
 	int ret;
 	do {
 		ret = __blk_mq_do_dispatch_sched(hctx);
-	} while (ret == 1);
+		if (ret != 1)
+			break;
+		if (need_resched() || time_is_before_jiffies(end)) {
+			blk_mq_delay_run_hw_queue(hctx, 0);
+			break;
+		}
+	} while (1);
 	return ret;
 }
 static void blk_mq_sched_tags_teardown(struct request_queue *q, unsigned int flags)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i) {
 		if (hctx->sched_tags) {
 int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e)
 {
-	unsigned int i, flags = q->tag_set->flags;
+	unsigned int flags = q->tag_set->flags;
 	struct blk_mq_hw_ctx *hctx;
 	struct elevator_queue *eq;
+	unsigned long i;
 	int ret;
 	if (!e) {
 void blk_mq_sched_free_rqs(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	if (blk_mq_is_shared_tags(q->tag_set->flags)) {
 		blk_mq_free_rqs(q->tag_set, q->sched_shared_tags,
 void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e)
 {
 	struct blk_mq_hw_ctx *hctx;
-	unsigned int i;
+	unsigned long i;
 	unsigned int flags = 0;
 	queue_for_each_hw_ctx(q, hctx, i) {

block/blk-mq-sysfs.c+10−6

 void blk_mq_unregister_dev(struct device *dev, struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	lockdep_assert_held(&q->sysfs_dir_lock);
 int __blk_mq_register_dev(struct device *dev, struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int ret, i;
+	unsigned long i, j;
+	int ret;
 	WARN_ON_ONCE(!q->kobj.parent);
 	lockdep_assert_held(&q->sysfs_dir_lock);
 	return ret;
 unreg:
-	while (--i >= 0)
+	queue_for_each_hw_ctx(q, hctx, j) {
-		blk_mq_unregister_hctx(q->queue_hw_ctx[i]);
+		if (j < i)
+			blk_mq_unregister_hctx(hctx);
+	}
 	kobject_uevent(q->mq_kobj, KOBJ_REMOVE);
 	kobject_del(q->mq_kobj);
 void blk_mq_sysfs_unregister(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	mutex_lock(&q->sysfs_dir_lock);
 	if (!q->mq_sysfs_init_done)
 int blk_mq_sysfs_register(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i, ret = 0;
+	unsigned long i;
+	int ret = 0;
 	mutex_lock(&q->sysfs_dir_lock);
 	if (!q->mq_sysfs_init_done)

block/blk-mq-tag.c+3−3

@@ -107,7 +107,7 @@ static int __blk_mq_get_tag(struct blk_mq_alloc_data *data, · 106 unmodified lines
107	return BLK_MQ_NO_TAG;	107	return BLK_MQ_NO_TAG;
108		108
109	if (data->shallow_depth)	109	if (data->shallow_depth)
110	return __sbitmap_queue_get_shallow(bt, data->shallow_depth);	110	return sbitmap_queue_get_shallow(bt, data->shallow_depth);
111	else	111	else
112	return __sbitmap_queue_get(bt);	112	return __sbitmap_queue_get(bt);
113	}	113	}
@@ -498,7 +498,7 @@ void blk_mq_queue_tag_busy_iter(struct request_queue q, busy_tag_iter_fn fn, · 384 unmodified lines
498	void *priv)	498	void *priv)
499	{	499	{
500	/*	500	/*
501	* __blk_mq_update_nr_hw_queues() updates nr_hw_queues and queue_hw_ctx	501	* __blk_mq_update_nr_hw_queues() updates nr_hw_queues and hctx_table
502	* while the queue is frozen. So we can use q_usage_counter to avoid	502	* while the queue is frozen. So we can use q_usage_counter to avoid
503	* racing with it.	503	* racing with it.
504	*/	504	*/
@@ -515,7 +515,7 @@ void blk_mq_queue_tag_busy_iter(struct request_queue q, busy_tag_iter_fn fn, · 10 unmodified lines
515	bt_for_each(NULL, q, btags, fn, priv, false);	515	bt_for_each(NULL, q, btags, fn, priv, false);
516	} else {	516	} else {
517	struct blk_mq_hw_ctx *hctx;	517	struct blk_mq_hw_ctx *hctx;
518	int i;	518	unsigned long i;
519		519
520	queue_for_each_hw_ctx(q, hctx, i) {	520	queue_for_each_hw_ctx(q, hctx, i) {
521	struct blk_mq_tags *tags = hctx->tags;	521	struct blk_mq_tags *tags = hctx->tags;

block/blk-mq.c+152−151

 static inline struct blk_mq_hw_ctx *blk_qc_to_hctx(struct request_queue *q,
 		blk_qc_t qc)
 {
-	return q->queue_hw_ctx[(qc & ~BLK_QC_T_INTERNAL) >> BLK_QC_T_SHIFT];
+	return xa_load(&q->hctx_table,
+			(qc & ~BLK_QC_T_INTERNAL) >> BLK_QC_T_SHIFT);
 }
 static inline struct request *blk_qc_to_rq(struct blk_mq_hw_ctx *hctx,
 void blk_mq_wake_waiters(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	unsigned int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i)
 		if (blk_mq_hw_queue_mapped(hctx))
 	 * If not tell the caller that it should skip this queue.
 	 */
 	ret = -EXDEV;
-	data.hctx = q->queue_hw_ctx[hctx_idx];
+	data.hctx = xa_load(&q->hctx_table, hctx_idx);
 	if (!blk_mq_hw_queue_mapped(data.hctx))
 		goto out_queue_exit;
 	cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
 #endif
 	if (unlikely(error && !blk_rq_is_passthrough(req) &&
-		     !(req->rq_flags & RQF_QUIET)))
+		     !(req->rq_flags & RQF_QUIET))) {
 		blk_print_req_error(req, error);
+		trace_block_rq_error(req, error, nr_bytes);
+	}
 	blk_account_io_completion(req, nr_bytes);
 static void __blk_account_io_start(struct request *rq)
 {
-	/* passthrough requests can hold bios that do not have ->bi_bdev set */
+	/*
-	if (rq->bio && rq->bio->bi_bdev)
+	 * All non-passthrough requests are created from a bio with one
+	 * exception: when a flush command that is part of a flush sequence
+	 * generated by the state machine in blk-flush.c is cloned onto the
+	 * lower device by dm-multipath we can get here without a bio.
+	 */
+	if (rq->bio)
 		rq->part = rq->bio->bi_bdev;
-	else if (rq->q->disk)
+	else
 		rq->part = rq->q->disk->part0;
 	part_stat_lock();
 		container_of(work, struct request_queue, timeout_work);
 	unsigned long next = 0;
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	/* A deadlock might occur if a request is stuck requiring a
 	 * timeout at the same time a queue freeze is waiting
 void blk_mq_run_hw_queues(struct request_queue *q, bool async)
 {
 	struct blk_mq_hw_ctx *hctx, *sq_hctx;
-	int i;
+	unsigned long i;
 	sq_hctx = NULL;
 	if (blk_mq_has_sqsched(q))
 void blk_mq_delay_run_hw_queues(struct request_queue *q, unsigned long msecs)
 {
 	struct blk_mq_hw_ctx *hctx, *sq_hctx;
-	int i;
+	unsigned long i;
 	sq_hctx = NULL;
 	if (blk_mq_has_sqsched(q))
 	queue_for_each_hw_ctx(q, hctx, i) {
 		if (blk_mq_hctx_stopped(hctx))
 			continue;
+		/*
+		 * If there is already a run_work pending, leave the
+		 * pending delay untouched. Otherwise, a hctx can stall
+		 * if another hctx is re-delaying the other's work
+		 * before the work executes.
+		 */
+		if (delayed_work_pending(&hctx->run_work))
+			continue;
 		/*
 		 * Dispatch from this hctx either if there's no hctx preferred
 		 * by IO scheduler or if it has requests that bypass the
 bool blk_mq_queue_stopped(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i)
 		if (blk_mq_hctx_stopped(hctx))
 void blk_mq_stop_hw_queues(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i)
 		blk_mq_stop_hw_queue(hctx);
 void blk_mq_start_hw_queues(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i)
 		blk_mq_start_hw_queue(hctx);
 void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i)
 		blk_mq_start_stopped_hw_queue(hctx, async);
 	q->mq_ops->queue_rqs(&plug->mq_list);
 }
+static void blk_mq_dispatch_plug_list(struct blk_plug *plug, bool from_sched)
+{
+	struct blk_mq_hw_ctx *this_hctx = NULL;
+	struct blk_mq_ctx *this_ctx = NULL;
+	struct request *requeue_list = NULL;
+	unsigned int depth = 0;
+	LIST_HEAD(list);
+	do {
+		struct request *rq = rq_list_pop(&plug->mq_list);
+		if (!this_hctx) {
+			this_hctx = rq->mq_hctx;
+			this_ctx = rq->mq_ctx;
+		} else if (this_hctx != rq->mq_hctx || this_ctx != rq->mq_ctx) {
+			rq_list_add(&requeue_list, rq);
+			continue;
+		}
+		list_add_tail(&rq->queuelist, &list);
+		depth++;
+	} while (!rq_list_empty(plug->mq_list));
+	plug->mq_list = requeue_list;
+	trace_block_unplug(this_hctx->queue, depth, !from_sched);
+	blk_mq_sched_insert_requests(this_hctx, this_ctx, &list, from_sched);
+}
 void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule)
 {
-	struct blk_mq_hw_ctx *this_hctx;
-	struct blk_mq_ctx *this_ctx;
 	struct request *rq;
-	unsigned int depth;
-	LIST_HEAD(list);
 	if (rq_list_empty(plug->mq_list))
 		return;
 			return;
 	}
-	this_hctx = NULL;
-	this_ctx = NULL;
-	depth = 0;
 	do {
-		rq = rq_list_pop(&plug->mq_list);
+		blk_mq_dispatch_plug_list(plug, from_schedule);
-		if (!this_hctx) {
-			this_hctx = rq->mq_hctx;
-			this_ctx = rq->mq_ctx;
-		} else if (this_hctx != rq->mq_hctx || this_ctx != rq->mq_ctx) {
-			trace_block_unplug(this_hctx->queue, depth,
-						!from_schedule);
-			blk_mq_sched_insert_requests(this_hctx, this_ctx,
-						&list, from_schedule);
-			depth = 0;
-			this_hctx = rq->mq_hctx;
-			this_ctx = rq->mq_ctx;
-		}
-		list_add(&rq->queuelist, &list);
-		depth++;
 	} while (!rq_list_empty(plug->mq_list));
-	if (!list_empty(&list)) {
-		trace_block_unplug(this_hctx->queue, depth, !from_schedule);
-		blk_mq_sched_insert_requests(this_hctx, this_ctx, &list,
-						from_schedule);
-	}
 }
 void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx,
 	unsigned int nr_segs = 1;
 	blk_status_t ret;
-	if (unlikely(!blk_crypto_bio_prep(&bio)))
-		return;
 	blk_queue_bounce(q, &bio);
 	if (blk_may_split(q, bio))
 		__blk_queue_split(q, &bio, &nr_segs);
 				blk_mq_try_issue_directly(rq->mq_hctx, rq));
 }
+#ifdef CONFIG_BLK_MQ_STACKING
 /**
- * blk_cloned_rq_check_limits - Helper function to check a cloned request
+ * blk_insert_cloned_request - Helper for stacking drivers to submit a request
- *                              for the new queue limits
+ * @rq: the request being queued
- * @q:  the queue
- * @rq: the request being checked
- *
- * Description:
- *    @rq may have been made based on weaker limitations of upper-level queues
- *    in request stacking drivers, and it may violate the limitation of @q.
- *    Since the block layer and the underlying device driver trust @rq
- *    after it is inserted to @q, it should be checked against @q before
- *    the insertion using this generic function.
- *
- *    Request stacking drivers like request-based dm may change the queue
- *    limits when retrying requests on other queues. Those requests need
- *    to be checked against the new queue limits again during dispatch.
  */
-static blk_status_t blk_cloned_rq_check_limits(struct request_queue *q,
+blk_status_t blk_insert_cloned_request(struct request *rq)
-				      struct request *rq)
 {
+	struct request_queue *q = rq->q;
 	unsigned int max_sectors = blk_queue_get_max_sectors(q, req_op(rq));
+	blk_status_t ret;
 	if (blk_rq_sectors(rq) > max_sectors) {
 		/*
 		return BLK_STS_IOERR;
 	}
-	return BLK_STS_OK;
+	if (q->disk && should_fail_request(q->disk->part0, blk_rq_bytes(rq)))
-}
-/**
- * blk_insert_cloned_request - Helper for stacking drivers to submit a request
- * @q:  the queue to submit the request
- * @rq: the request being queued
- */
-blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request *rq)
-{
-	blk_status_t ret;
-	ret = blk_cloned_rq_check_limits(q, rq);
-	if (ret != BLK_STS_OK)
-		return ret;
-	if (rq->q->disk &&
-	    should_fail_request(rq->q->disk->part0, blk_rq_bytes(rq)))
 		return BLK_STS_IOERR;
 	if (blk_crypto_insert_cloned_request(rq))
 	 * bypass a potential scheduler on the bottom device for
 	 * insert.
 	 */
-	blk_mq_run_dispatch_ops(rq->q,
+	blk_mq_run_dispatch_ops(q,
 			ret = blk_mq_request_issue_directly(rq, true));
 	if (ret)
 		blk_account_io_done(rq, ktime_get_ns());
 		bs = &fs_bio_set;
 	__rq_for_each_bio(bio_src, rq_src) {
-		bio = bio_clone_fast(bio_src, gfp_mask, bs);
+		bio = bio_alloc_clone(rq->q->disk->part0, bio_src, gfp_mask,
+				      bs);
 		if (!bio)
 			goto free_and_out;
-		bio->bi_bdev = rq->q->disk->part0;
 		if (bio_ctr && bio_ctr(bio, bio_src, data))
 			goto free_and_out;
 	return -ENOMEM;
 }
 EXPORT_SYMBOL_GPL(blk_rq_prep_clone);
+#endif /* CONFIG_BLK_MQ_STACKING */
 /*
  * Steal bios from a request and add them to a bio list.
 	struct blk_mq_tags *drv_tags;
 	struct page *page;
+	if (list_empty(&tags->page_list))
+		return;
 	if (blk_mq_is_shared_tags(set->flags))
 		drv_tags = set->shared_tags;
 	else
 	blk_mq_free_tags(tags);
 }
+static enum hctx_type hctx_idx_to_type(struct blk_mq_tag_set *set,
+		unsigned int hctx_idx)
+{
+	int i;
+	for (i = 0; i < set->nr_maps; i++) {
+		unsigned int start = set->map[i].queue_offset;
+		unsigned int end = start + set->map[i].nr_queues;
+		if (hctx_idx >= start && hctx_idx < end)
+			break;
+	}
+	if (i >= set->nr_maps)
+		i = HCTX_TYPE_DEFAULT;
+	return i;
+}
+static int blk_mq_get_hctx_node(struct blk_mq_tag_set *set,
+		unsigned int hctx_idx)
+{
+	enum hctx_type type = hctx_idx_to_type(set, hctx_idx);
+	return blk_mq_hw_queue_to_node(&set->map[type], hctx_idx);
+}
 static struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set,
 					       unsigned int hctx_idx,
 					       unsigned int nr_tags,
 					       unsigned int reserved_tags)
 {
+	int node = blk_mq_get_hctx_node(set, hctx_idx);
 	struct blk_mq_tags *tags;
-	int node;
-	node = blk_mq_hw_queue_to_node(&set->map[HCTX_TYPE_DEFAULT], hctx_idx);
 	if (node == NUMA_NO_NODE)
 		node = set->numa_node;
 			    unsigned int hctx_idx, unsigned int depth)
 {
 	unsigned int i, j, entries_per_page, max_order = 4;
+	int node = blk_mq_get_hctx_node(set, hctx_idx);
 	size_t rq_size, left;
-	int node;
-	node = blk_mq_hw_queue_to_node(&set->map[HCTX_TYPE_DEFAULT], hctx_idx);
 	if (node == NUMA_NO_NODE)
 		node = set->numa_node;
 	blk_mq_remove_cpuhp(hctx);
+	xa_erase(&q->hctx_table, hctx_idx);
 	spin_lock(&q->unused_hctx_lock);
 	list_add(&hctx->hctx_list, &q->unused_hctx_list);
 	spin_unlock(&q->unused_hctx_lock);
 		struct blk_mq_tag_set *set, int nr_queue)
 {
 	struct blk_mq_hw_ctx *hctx;
-	unsigned int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i) {
 		if (i == nr_queue)
 			break;
-		blk_mq_debugfs_unregister_hctx(hctx);
 		blk_mq_exit_hctx(q, set, hctx, i);
 	}
 }
 	if (blk_mq_init_request(set, hctx->fq->flush_rq, hctx_idx,
 				hctx->numa_node))
 		goto exit_hctx;
+	if (xa_insert(&q->hctx_table, hctx_idx, hctx, GFP_KERNEL))
+		goto exit_flush_rq;
 	return 0;
+ exit_flush_rq:
+	if (set->ops->exit_request)
+		set->ops->exit_request(set, hctx->fq->flush_rq, hctx_idx);
  exit_hctx:
 	if (set->ops->exit_hctx)
 		set->ops->exit_hctx(hctx, hctx_idx);
 static void blk_mq_map_swqueue(struct request_queue *q)
 {
-	unsigned int i, j, hctx_idx;
+	unsigned int j, hctx_idx;
+	unsigned long i;
 	struct blk_mq_hw_ctx *hctx;
 	struct blk_mq_ctx *ctx;
 	struct blk_mq_tag_set *set = q->tag_set;
 static void queue_set_hctx_shared(struct request_queue *q, bool shared)
 {
 	struct blk_mq_hw_ctx *hctx;
-	int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i) {
 		if (shared) {
 void blk_mq_release(struct request_queue *q)
 {
 	struct blk_mq_hw_ctx *hctx, *next;
-	int i;
+	unsigned long i;
 	queue_for_each_hw_ctx(q, hctx, i)
 		WARN_ON_ONCE(hctx && list_empty(&hctx->hctx_list));
 		kobject_put(&hctx->kobj);
 	}
-	kfree(q->queue_hw_ctx);
+	xa_destroy(&q->hctx_table);
 	/*
 	 * release .mq_kobj and sw queue's kobject now because
 static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
 						struct request_queue *q)
 {
-	int i, j, end;
+	struct blk_mq_hw_ctx *hctx;
-	struct blk_mq_hw_ctx **hctxs = q->queue_hw_ctx;
+	unsigned long i, j;
-	if (q->nr_hw_queues < set->nr_hw_queues) {
-		struct blk_mq_hw_ctx **new_hctxs;
-		new_hctxs = kcalloc_node(set->nr_hw_queues,
-				       sizeof(*new_hctxs), GFP_KERNEL,
-				       set->numa_node);
-		if (!new_hctxs)
-			return;
-		if (hctxs)
-			memcpy(new_hctxs, hctxs, q->nr_hw_queues *
-			       sizeof(*hctxs));
-		q->queue_hw_ctx = new_hctxs;
-		kfree(hctxs);
-		hctxs = new_hctxs;
-	}
 	/* protect against switching io scheduler  */
 	mutex_lock(&q->sysfs_lock);
 	for (i = 0; i < set->nr_hw_queues; i++) {
-		int node;
+		int old_node;
-		struct blk_mq_hw_ctx *hctx;
+		int node = blk_mq_get_hctx_node(set, i);
+		struct blk_mq_hw_ctx *old_hctx = xa_load(&q->hctx_table, i);
-		node = blk_mq_hw_queue_to_node(&set->map[HCTX_TYPE_DEFAULT], i);
+		if (old_hctx) {
-		/*
+			old_node = old_hctx->numa_node;
-		 * If the hw queue has been mapped to another numa node,
+			blk_mq_exit_hctx(q, set, old_hctx, i);
-		 * we need to realloc the hctx. If allocation fails, fallback
+		}
-		 * to use the previous one.
-		 */
-		if (hctxs[i] && (hctxs[i]->numa_node == node))
-			continue;
-		hctx = blk_mq_alloc_and_init_hctx(set, q, i, node);
+		if (!blk_mq_alloc_and_init_hctx(set, q, i, node)) {
-		if (hctx) {
+			if (!old_hctx)
-			if (hctxs[i])
-				blk_mq_exit_hctx(q, set, hctxs[i], i);
-			hctxs[i] = hctx;
-		} else {
-			if (hctxs[i])
-				pr_warn("Allocate new hctx on node %d fails,\
-						fallback to previous one on node %d\n",
-						node, hctxs[i]->numa_node);
-			else
 				break;
+			pr_warn("Allocate new hctx on node %d fails, fallback to previous one on node %d\n",
+					node, old_node);
+			hctx = blk_mq_alloc_and_init_hctx(set, q, i, old_node);
+			WARN_ON_ONCE(!hctx);
 		}
 	}
 	/*
 	 */
 	if (i != set->nr_hw_queues) {
 		j = q->nr_hw_queues;
-		end = i;
 	} else {
 		j = i;
-		end = q->nr_hw_queues;
 		q->nr_hw_queues = set->nr_hw_queues;
 	}
-	for (; j < end; j++) {
+	xa_for_each_start(&q->hctx_table, j, hctx, j)
-		struct blk_mq_hw_ctx *hctx = hctxs[j];
+		blk_mq_exit_hctx(q, set, hctx, j);
-		if (hctx) {
-			blk_mq_exit_hctx(q, set, hctx, j);
-			hctxs[j] = NULL;
-		}
-	}
 	mutex_unlock(&q->sysfs_lock);
 }
+static void blk_mq_update_poll_flag(struct request_queue *q)
+{
+	struct blk_mq_tag_set *set = q->tag_set;
+	if (set->nr_maps > HCTX_TYPE_POLL &&
+	    set->map[HCTX_TYPE_POLL].nr_queues)
+		blk_queue_flag_set(QUEUE_FLAG_POLL, q);
+	else
+		blk_queue_flag_clear(QUEUE_FLAG_POLL, q);
+}
 int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 		struct request_queue *q)
 {
 	INIT_LIST_HEAD(&q->unused_hctx_list);
 	spin_lock_init(&q->unused_hctx_lock);
+	xa_init(&q->hctx_table);
 	blk_mq_realloc_hw_ctxs(set, q);
 	if (!q->nr_hw_queues)
 		goto err_hctxs;
 	q->tag_set = set;
 	q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT;
-	if (set->nr_maps > HCTX_TYPE_POLL &&
+	blk_mq_update_poll_flag(q);
-	    set->map[HCTX_TYPE_POLL].nr_queues)
-		blk_queue_flag_set(QUEUE_FLAG_POLL, q);
 	INIT_DELAYED_WORK(&q->requeue_work, blk_mq_requeue_work);
 	INIT_LIST_HEAD(&q->requeue_list);
 	return 0;
 err_hctxs:
-	kfree(q->queue_hw_ctx);
+	xa_destroy(&q->hctx_table);
 	q->nr_hw_queues = 0;
 	blk_mq_sysfs_deinit(q);
 err_poll:
 {
 	struct blk_mq_tag_set *set = q->tag_set;
 	struct blk_mq_hw_ctx *hctx;
-	int i, ret;
+	int ret;
+	unsigned long i;
 	if (!set)
 		return -EINVAL;
 	blk_mq_update_queue_map(set);
 	list_for_each_entry(q, &set->tag_list, tag_set_list) {
 		blk_mq_realloc_hw_ctxs(set, q);
+		blk_mq_update_poll_flag(q);
 		if (q->nr_hw_queues != set->nr_hw_queues) {
 			int i = prev_nr_hw_queues;
 {
 	if (queue_is_mq(q)) {
 		struct blk_mq_hw_ctx *hctx;
-		int i;
+		unsigned long i;
 		cancel_delayed_work_sync(&q->requeue_work);

block/blk-mq.h+1−1

@@ -83,7 +83,7 @@ static inline struct blk_mq_hw_ctx blk_mq_map_queue_type(struct request_queue · 82 unmodified lines
83	enum hctx_type type,	83	enum hctx_type type,
84	unsigned int cpu)	84	unsigned int cpu)
85	{	85	{
86	return q->queue_hw_ctx[q->tag_set->map[type].mq_map[cpu]];	86	return xa_load(&q->hctx_table, q->tag_set->map[type].mq_map[cpu]);
87	}	87	}
88		88
89	static inline enum hctx_type blk_mq_get_hctx_type(unsigned int flags)	89	static inline enum hctx_type blk_mq_get_hctx_type(unsigned int flags)

block/blk-rq-qos.h+11−9

 		__rq_qos_requeue(q->rq_qos, rq);
 }
-static inline void rq_qos_done_bio(struct request_queue *q, struct bio *bio)
+static inline void rq_qos_done_bio(struct bio *bio)
 {
-	if (q->rq_qos)
+	if (bio->bi_bdev && (bio_flagged(bio, BIO_QOS_THROTTLED) ||
-		__rq_qos_done_bio(q->rq_qos, bio);
+			     bio_flagged(bio, BIO_QOS_MERGED))) {
+		struct request_queue *q = bdev_get_queue(bio->bi_bdev);
+		if (q->rq_qos)
+			__rq_qos_done_bio(q->rq_qos, bio);
+	}
 }
 static inline void rq_qos_throttle(struct request_queue *q, struct bio *bio)
 {
-	/*
-	 * BIO_TRACKED lets controllers know that a bio went through the
-	 * normal rq_qos path.
-	 */
 	if (q->rq_qos) {
-		bio_set_flag(bio, BIO_TRACKED);
+		bio_set_flag(bio, BIO_QOS_THROTTLED);
 		__rq_qos_throttle(q->rq_qos, bio);
 	}
 }
 static inline void rq_qos_merge(struct request_queue *q, struct request *rq,
 				struct bio *bio)
 {
-	if (q->rq_qos)
+	if (q->rq_qos) {
+		bio_set_flag(bio, BIO_QOS_MERGED);
 		__rq_qos_merge(q->rq_qos, rq, bio);
+	}
 }
 static inline void rq_qos_queue_depth_changed(struct request_queue *q)

block/blk-sysfs.c+15−29

@@ -10,7 +10,6 @@ · 9 unmodified lines
10	#include <linux/backing-dev.h>	10	#include <linux/backing-dev.h>
11	#include <linux/blktrace_api.h>	11	#include <linux/blktrace_api.h>
12	#include <linux/blk-mq.h>	12	#include <linux/blk-mq.h>
13	#include <linux/blk-cgroup.h>
14	#include <linux/debugfs.h>	13	#include <linux/debugfs.h>
15		14
16	#include "blk.h"	15	#include "blk.h"
@@ -18,6 +17,7 @@ · 1 unmodified line
18	#include "blk-mq-debugfs.h"	17	#include "blk-mq-debugfs.h"
19	#include "blk-mq-sched.h"	18	#include "blk-mq-sched.h"
20	#include "blk-wbt.h"	19	#include "blk-wbt.h"
		20	#include "blk-cgroup.h"
21	#include "blk-throttle.h"	21	#include "blk-throttle.h"
22		22
23	struct queue_sysfs_entry {	23	struct queue_sysfs_entry {
@@ -739,27 +739,6 @@ static void blk_free_queue_rcu(struct rcu_head *rcu_head) · 715 unmodified lines
739	kmem_cache_free(blk_get_queue_kmem_cache(blk_queue_has_srcu(q)), q);	739	kmem_cache_free(blk_get_queue_kmem_cache(blk_queue_has_srcu(q)), q);
740	}	740	}
741		741
742	/* Unconfigure the I/O scheduler and dissociate from the cgroup controller. */
743	static void blk_exit_queue(struct request_queue *q)
744	{
745	/*
746	* Since the I/O scheduler exit code may access cgroup information,
747	* perform I/O scheduler exit before disassociating from the block
748	* cgroup controller.
749	*/
750	if (q->elevator) {
751	ioc_clear_queue(q);
752	elevator_exit(q);
753	}
754
755	/*
756	* Remove all references to @q from the block cgroup controller before
757	* restoring @q->queue_lock to avoid that restoring this pointer causes
758	* e.g. blkcg_print_blkgs() to crash.
759	*/
760	blkcg_exit_queue(q);
761	}
762
763	/**	742	/**
764	* blk_release_queue - releases all allocated resources of the request_queue	743	* blk_release_queue - releases all allocated resources of the request_queue
765	* @kobj: pointer to a kobject, whose container is a request_queue	744	* @kobj: pointer to a kobject, whose container is a request_queue
@@ -787,12 +766,12 @@ static void blk_release_queue(struct kobject *kobj) · 21 unmodified lines
787		766
788	might_sleep();	767	might_sleep();
789		768
		769	percpu_ref_exit(&q->q_usage_counter);
		770
790	if (q->poll_stat)	771	if (q->poll_stat)
791	blk_stat_remove_callback(q, q->poll_cb);	772	blk_stat_remove_callback(q, q->poll_cb);
792	blk_stat_free_callback(q->poll_cb);	773	blk_stat_free_callback(q->poll_cb);
793		774
794	blk_exit_queue(q);
795
796	blk_free_queue_stats(q->stats);	775	blk_free_queue_stats(q->stats);
797	kfree(q->poll_stat);	776	kfree(q->poll_stat);
798		777
@@ -880,6 +859,10 @@ int blk_register_queue(struct gendisk *disk) · 81 unmodified lines
880	goto put_dev;	859	goto put_dev;
881	}	860	}
882		861
		862	ret = blk_crypto_sysfs_register(q);
		863	if (ret)
		864	goto put_dev;
		865
883	blk_queue_flag_set(QUEUE_FLAG_REGISTERED, q);	866	blk_queue_flag_set(QUEUE_FLAG_REGISTERED, q);
884	wbt_enable_default(q);	867	wbt_enable_default(q);
885	blk_throtl_register_queue(q);	868	blk_throtl_register_queue(q);
@@ -910,6 +893,7 @@ unlock: · 24 unmodified lines
910	return ret;	893	return ret;
911		894
912	put_dev:	895	put_dev:
		896	elv_unregister_queue(q);
913	disk_unregister_independent_access_ranges(disk);	897	disk_unregister_independent_access_ranges(disk);
914	mutex_unlock(&q->sysfs_lock);	898	mutex_unlock(&q->sysfs_lock);
915	mutex_unlock(&q->sysfs_dir_lock);	899	mutex_unlock(&q->sysfs_dir_lock);
@@ -954,16 +938,18 @@ void blk_unregister_queue(struct gendisk *disk) · 38 unmodified lines
954	*/	938	*/
955	if (queue_is_mq(q))	939	if (queue_is_mq(q))
956	blk_mq_unregister_dev(disk_to_dev(disk), q);	940	blk_mq_unregister_dev(disk_to_dev(disk), q);
957		941	blk_crypto_sysfs_unregister(q);
958	kobject_uevent(&q->kobj, KOBJ_REMOVE);
959	kobject_del(&q->kobj);
960	blk_trace_remove_sysfs(disk_to_dev(disk));	942	blk_trace_remove_sysfs(disk_to_dev(disk));
961		943
962	mutex_lock(&q->sysfs_lock);	944	mutex_lock(&q->sysfs_lock);
963	if (q->elevator)	945	elv_unregister_queue(q);
964	elv_unregister_queue(q);
965	disk_unregister_independent_access_ranges(disk);	946	disk_unregister_independent_access_ranges(disk);
966	mutex_unlock(&q->sysfs_lock);	947	mutex_unlock(&q->sysfs_lock);
		948
		949	/* Now that we've deleted all child objects, we can delete the queue. */
		950	kobject_uevent(&q->kobj, KOBJ_REMOVE);
		951	kobject_del(&q->kobj);
		952
967	mutex_unlock(&q->sysfs_dir_lock);	953	mutex_unlock(&q->sysfs_dir_lock);
968		954
969	kobject_put(&disk_to_dev(disk)->kobj);	955	kobject_put(&disk_to_dev(disk)->kobj);

block/blk-throttle.c+68−42

 #include <linux/blkdev.h>
 #include <linux/bio.h>
 #include <linux/blktrace_api.h>
-#include <linux/blk-cgroup.h>
 #include "blk.h"
 #include "blk-cgroup-rwstat.h"
 #include "blk-stat.h"
 /* A workqueue to queue throttle related work */
 static struct workqueue_struct *kthrotld_workqueue;
-enum tg_state_flags {
-	THROTL_TG_PENDING	= 1 << 0,	/* on parent's pending tree */
-	THROTL_TG_WAS_EMPTY	= 1 << 1,	/* bio_lists[] became non-empty */
-};
 #define rb_entry_tg(node)	rb_entry((node), struct throtl_grp, rb_node)
 /* We measure latency for request size from <= 4k to >= 1M */
 	struct throtl_grp *parent_tg = sq_to_tg(tg->service_queue.parent_sq);
 	struct throtl_data *td = tg->td;
 	int rw;
+	int has_iops_limit = 0;
+	for (rw = READ; rw <= WRITE; rw++) {
+		unsigned int iops_limit = tg_iops_limit(tg, rw);
-	for (rw = READ; rw <= WRITE; rw++)
 		tg->has_rules[rw] = (parent_tg && parent_tg->has_rules[rw]) ||
 			(td->limit_valid[td->limit_index] &&
 			 (tg_bps_limit(tg, rw) != U64_MAX ||
-			  tg_iops_limit(tg, rw) != UINT_MAX));
+			  iops_limit != UINT_MAX));
+		if (iops_limit != UINT_MAX)
+			has_iops_limit = 1;
+	}
+	if (has_iops_limit)
+		tg->flags |= THROTL_TG_HAS_IOPS_LIMIT;
+	else
+		tg->flags &= ~THROTL_TG_HAS_IOPS_LIMIT;
 }
 static void throtl_pd_online(struct blkg_policy_data *pd)
 	tg->bytes_disp[rw] = 0;
 	tg->io_disp[rw] = 0;
-	atomic_set(&tg->io_split_cnt[rw], 0);
 	/*
 	 * Previous slice has expired. We must have trimmed it after last
 	 * bio dispatch. That means since start of last slice, we never used
 	tg->slice_start[rw] = jiffies;
 	tg->slice_end[rw] = jiffies + tg->td->throtl_slice;
-	atomic_set(&tg->io_split_cnt[rw], 0);
 	throtl_log(&tg->service_queue,
 		   "[%c] new slice start=%lu end=%lu jiffies=%lu",
 		   rw == READ ? 'R' : 'W', tg->slice_start[rw],
 	unsigned long jiffy_elapsed, jiffy_wait, jiffy_elapsed_rnd;
 	unsigned int bio_size = throtl_bio_data_size(bio);
-	if (bps_limit == U64_MAX) {
+	/* no need to throttle if this bio's bytes have been accounted */
+	if (bps_limit == U64_MAX || bio_flagged(bio, BIO_THROTTLED)) {
 		if (wait)
 			*wait = 0;
 		return true;
 	       bio != throtl_peek_queued(&tg->service_queue.queued[rw]));
 	/* If tg->bps = -1, then BW is unlimited */
-	if (bps_limit == U64_MAX && iops_limit == UINT_MAX) {
+	if ((bps_limit == U64_MAX && iops_limit == UINT_MAX) ||
+	    tg->flags & THROTL_TG_CANCELING) {
 		if (wait)
 			*wait = 0;
 		return true;
 				jiffies + tg->td->throtl_slice);
 	}
-	if (iops_limit != UINT_MAX)
-		tg->io_disp[rw] += atomic_xchg(&tg->io_split_cnt[rw], 0);
 	if (tg_with_in_bps_limit(tg, bio, bps_limit, &bps_wait) &&
 	    tg_with_in_iops_limit(tg, bio, iops_limit, &iops_wait)) {
 		if (wait)
 	unsigned int bio_size = throtl_bio_data_size(bio);
 	/* Charge the bio to the group */
-	tg->bytes_disp[rw] += bio_size;
+	if (!bio_flagged(bio, BIO_THROTTLED)) {
+		tg->bytes_disp[rw] += bio_size;
+		tg->last_bytes_disp[rw] += bio_size;
+	}
 	tg->io_disp[rw]++;
-	tg->last_bytes_disp[rw] += bio_size;
 	tg->last_io_disp[rw]++;
 	/*
 	struct throtl_service_queue *sq = from_timer(sq, t, pending_timer);
 	struct throtl_grp *tg = sq_to_tg(sq);
 	struct throtl_data *td = sq_to_td(sq);
-	struct request_queue *q = td->queue;
 	struct throtl_service_queue *parent_sq;
+	struct request_queue *q;
 	bool dispatched;
 	int ret;
+	/* throtl_data may be gone, so figure out request queue by blkg */
+	if (tg)
+		q = tg->pd.blkg->q;
+	else
+		q = td->queue;
 	spin_lock_irq(&q->queue_lock);
+	if (!q->root_blkg)
+		goto out_unlock;
 	if (throtl_can_upgrade(td, NULL))
 		throtl_upgrade_state(td);
 	if (!bio_list_empty(&bio_list_on_stack)) {
 		blk_start_plug(&plug);
 		while ((bio = bio_list_pop(&bio_list_on_stack)))
-			submit_bio_noacct(bio);
+			submit_bio_noacct_nocheck(bio);
 		blk_finish_plug(&plug);
 	}
 }
 	return false;
 }
+void blk_throtl_cancel_bios(struct request_queue *q)
+{
+	struct cgroup_subsys_state *pos_css;
+	struct blkcg_gq *blkg;
+	spin_lock_irq(&q->queue_lock);
+	/*
+	 * queue_lock is held, rcu lock is not needed here technically.
+	 * However, rcu lock is still held to emphasize that following
+	 * path need RCU protection and to prevent warning from lockdep.
+	 */
+	rcu_read_lock();
+	blkg_for_each_descendant_post(blkg, pos_css, q->root_blkg) {
+		struct throtl_grp *tg = blkg_to_tg(blkg);
+		struct throtl_service_queue *sq = &tg->service_queue;
+		/*
+		 * Set the flag to make sure throtl_pending_timer_fn() won't
+		 * stop until all throttled bios are dispatched.
+		 */
+		blkg_to_tg(blkg)->flags |= THROTL_TG_CANCELING;
+		/*
+		 * Update disptime after setting the above flag to make sure
+		 * throtl_select_dispatch() won't exit without dispatching.
+		 */
+		tg_update_disptime(tg);
+		throtl_schedule_pending_timer(sq, jiffies + 1);
+	}
+	rcu_read_unlock();
+	spin_unlock_irq(&q->queue_lock);
+}
 static bool throtl_can_upgrade(struct throtl_data *td,
 	struct throtl_grp *this_tg)
 {
 	}
 	if (tg->iops[READ][LIMIT_LOW]) {
-		tg->last_io_disp[READ] += atomic_xchg(&tg->last_io_split_cnt[READ], 0);
 		iops = tg->last_io_disp[READ] * HZ / elapsed_time;
 		if (iops >= tg->iops[READ][LIMIT_LOW])
 			tg->last_low_overflow_time[READ] = now;
 	}
 	if (tg->iops[WRITE][LIMIT_LOW]) {
-		tg->last_io_disp[WRITE] += atomic_xchg(&tg->last_io_split_cnt[WRITE], 0);
 		iops = tg->last_io_disp[WRITE] * HZ / elapsed_time;
 		if (iops >= tg->iops[WRITE][LIMIT_LOW])
 			tg->last_low_overflow_time[WRITE] = now;
 }
 #endif
-void blk_throtl_charge_bio_split(struct bio *bio)
-{
-	struct blkcg_gq *blkg = bio->bi_blkg;
-	struct throtl_grp *parent = blkg_to_tg(blkg);
-	struct throtl_service_queue *parent_sq;
-	bool rw = bio_data_dir(bio);
-	do {
-		if (!parent->has_rules[rw])
-			break;
-		atomic_inc(&parent->io_split_cnt[rw]);
-		atomic_inc(&parent->last_io_split_cnt[rw]);
-		parent_sq = parent->service_queue.parent_sq;
-		parent = sq_to_tg(parent_sq);
-	} while (parent);
-}
 bool __blk_throtl_bio(struct bio *bio)
 {
 	struct request_queue *q = bdev_get_queue(bio->bi_bdev);

block/blk-throttle.h+13−6

 	struct timer_list	pending_timer;	/* fires on first_pending_disptime */
 };
+enum tg_state_flags {
+	THROTL_TG_PENDING	= 1 << 0,	/* on parent's pending tree */
+	THROTL_TG_WAS_EMPTY	= 1 << 1,	/* bio_lists[] became non-empty */
+	THROTL_TG_HAS_IOPS_LIMIT = 1 << 2,	/* tg has iops limit */
+	THROTL_TG_CANCELING	= 1 << 3,	/* starts to cancel bio */
+};
 enum {
 	LIMIT_LOW,
 	LIMIT_MAX,
 	unsigned int bad_bio_cnt; /* bios exceeding latency threshold */
 	unsigned long bio_cnt_reset_time;
-	atomic_t io_split_cnt[2];
-	atomic_t last_io_split_cnt[2];
 	struct blkg_rwstat stat_bytes;
 	struct blkg_rwstat stat_ios;
 };
 static inline int blk_throtl_init(struct request_queue *q) { return 0; }
 static inline void blk_throtl_exit(struct request_queue *q) { }
 static inline void blk_throtl_register_queue(struct request_queue *q) { }
-static inline void blk_throtl_charge_bio_split(struct bio *bio) { }
 static inline bool blk_throtl_bio(struct bio *bio) { return false; }
+static inline void blk_throtl_cancel_bios(struct request_queue *q) { }
 #else /* CONFIG_BLK_DEV_THROTTLING */
 int blk_throtl_init(struct request_queue *q);
 void blk_throtl_exit(struct request_queue *q);
 void blk_throtl_register_queue(struct request_queue *q);
-void blk_throtl_charge_bio_split(struct bio *bio);
 bool __blk_throtl_bio(struct bio *bio);
+void blk_throtl_cancel_bios(struct request_queue *q);
 static inline bool blk_throtl_bio(struct bio *bio)
 {
 	struct throtl_grp *tg = blkg_to_tg(bio->bi_blkg);
-	if (bio_flagged(bio, BIO_THROTTLED))
+	/* no need to throttle bps any more if the bio has been throttled */
+	if (bio_flagged(bio, BIO_THROTTLED) &&
+	    !(tg->flags & THROTL_TG_HAS_IOPS_LIMIT))
 		return false;
 	if (!tg->has_rules[bio_data_dir(bio)])
 		return false;

block/blk-zoned.c+4−10

 			continue;
 		}
-		bio = blk_next_bio(bio, 0, gfp_mask);
+		bio = blk_next_bio(bio, bdev, 0, REQ_OP_ZONE_RESET | REQ_SYNC,
-		bio_set_dev(bio, bdev);
+				   gfp_mask);
-		bio->bi_opf = REQ_OP_ZONE_RESET | REQ_SYNC;
 		bio->bi_iter.bi_sector = sector;
 		sector += zone_sectors;
 {
 	struct bio bio;
-	bio_init(&bio, NULL, 0);
+	bio_init(&bio, bdev, NULL, 0, REQ_OP_ZONE_RESET_ALL | REQ_SYNC);
-	bio_set_dev(&bio, bdev);
-	bio.bi_opf = REQ_OP_ZONE_RESET_ALL | REQ_SYNC;
 	return submit_bio_wait(&bio);
 }
 	}
 	while (sector < end_sector) {
-		bio = blk_next_bio(bio, 0, gfp_mask);
+		bio = blk_next_bio(bio, bdev, 0, op | REQ_SYNC, gfp_mask);
-		bio_set_dev(bio, bdev);
-		bio->bi_opf = op | REQ_SYNC;
 		bio->bi_iter.bi_sector = sector;
 		sector += zone_sectors;

block/blk.h+6−4

 void __blk_mq_unfreeze_queue(struct request_queue *q, bool force_atomic);
 void blk_queue_start_drain(struct request_queue *q);
 int __bio_queue_enter(struct request_queue *q, struct bio *bio);
-bool submit_bio_checks(struct bio *bio);
+void submit_bio_noacct_nocheck(struct bio *bio);
 static inline bool blk_try_enter_queue(struct request_queue *q, bool pm)
 {
  */
 static inline bool blk_do_io_stat(struct request *rq)
 {
-	return (rq->rq_flags & RQF_IO_STAT) && rq->q->disk;
+	return (rq->rq_flags & RQF_IO_STAT) && !blk_rq_is_passthrough(rq);
 }
 void update_io_ticks(struct block_device *part, unsigned long now, bool end);
 static inline int blk_iolatency_init(struct request_queue *q) { return 0; }
 #endif
-struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp);
 #ifdef CONFIG_BLK_DEV_ZONED
 void blk_queue_free_zone_bitmaps(struct request_queue *q);
 void blk_queue_clear_zone_settings(struct request_queue *q);
 int bdev_del_partition(struct gendisk *disk, int partno);
 int bdev_resize_partition(struct gendisk *disk, int partno, sector_t start,
 		sector_t length);
+void blk_drop_partitions(struct gendisk *disk);
 int bio_add_hw_page(struct request_queue *q, struct bio *bio,
 		struct page *page, unsigned int len, unsigned int offset,
 void disk_add_events(struct gendisk *disk);
 void disk_del_events(struct gendisk *disk);
 void disk_release_events(struct gendisk *disk);
+void disk_block_events(struct gendisk *disk);
+void disk_unblock_events(struct gendisk *disk);
+void disk_flush_events(struct gendisk *disk, unsigned int mask);
 extern struct device_attribute dev_attr_events;
 extern struct device_attribute dev_attr_events_async;
 extern struct device_attribute dev_attr_events_poll_msecs;

block/bounce.c+4−7

 #include <linux/pagemap.h>
 #include <linux/mempool.h>
 #include <linux/blkdev.h>
-#include <linux/blk-cgroup.h>
 #include <linux/backing-dev.h>
 #include <linux/init.h>
 #include <linux/hash.h>
 #include <trace/events/block.h>
 #include "blk.h"
+#include "blk-cgroup.h"
 #define POOL_SIZE	64
 #define ISA_POOL_SIZE	16
 	 *    that does not own the bio - reason being drivers don't use it for
 	 *    iterating over the biovec anymore, so expecting it to be kept up
 	 *    to date (i.e. for clones that share the parent biovec) is just
-	 *    asking for trouble and would force extra work on
+	 *    asking for trouble and would force extra work.
-	 *    __bio_clone_fast() anyways.
 	 */
-	bio = bio_alloc_bioset(GFP_NOIO, bio_segments(bio_src),
+	bio = bio_alloc_bioset(bio_src->bi_bdev, bio_segments(bio_src),
-			       &bounce_bio_set);
+			       bio_src->bi_opf, GFP_NOIO, &bounce_bio_set);
-	bio->bi_bdev		= bio_src->bi_bdev;
 	if (bio_flagged(bio_src, BIO_REMAPPED))
 		bio_set_flag(bio, BIO_REMAPPED);
-	bio->bi_opf		= bio_src->bi_opf;
 	bio->bi_ioprio		= bio_src->bi_ioprio;
 	bio->bi_write_hint	= bio_src->bi_write_hint;
 	bio->bi_iter.bi_sector	= bio_src->bi_iter.bi_sector;

block/disk-events.c+1−1

@@ -4,7 +4,7 @@ · 3 unmodified lines
4	*/	4	*/
5	#include <linux/export.h>	5	#include <linux/export.h>
6	#include <linux/moduleparam.h>	6	#include <linux/moduleparam.h>
7	#include <linux/genhd.h>	7	#include <linux/blkdev.h>
8	#include "blk.h"	8	#include "blk.h"
9		9
10	struct disk_events {	10	struct disk_events {

block/elevator.c+8−8

 #include <linux/hash.h>
 #include <linux/uaccess.h>
 #include <linux/pm_runtime.h>
-#include <linux/blk-cgroup.h>
 #include <trace/events/block.h>
 #include "blk-mq-sched.h"
 #include "blk-pm.h"
 #include "blk-wbt.h"
+#include "blk-cgroup.h"
 static DEFINE_SPINLOCK(elv_list_lock);
 static LIST_HEAD(elv_list);
 {
 	struct elevator_queue *e = q->elevator;
+	ioc_clear_queue(q);
+	blk_mq_sched_free_rqs(q);
 	mutex_lock(&e->sysfs_lock);
 	blk_mq_exit_sched(q, e);
 	mutex_unlock(&e->sysfs_lock);
 void elv_unregister_queue(struct request_queue *q)
 {
+	struct elevator_queue *e = q->elevator;
 	lockdep_assert_held(&q->sysfs_lock);
-	if (q) {
+	if (e && e->registered) {
 		struct elevator_queue *e = q->elevator;
 		kobject_uevent(&e->kobj, KOBJ_REMOVE);
 	lockdep_assert_held(&q->sysfs_lock);
 	if (q->elevator) {
-		if (q->elevator->registered)
+		elv_unregister_queue(q);
-			elv_unregister_queue(q);
-		ioc_clear_queue(q);
-		blk_mq_sched_free_rqs(q);
 		elevator_exit(q);
 	}
 	if (new_e) {
 		ret = elv_register_queue(q, true);
 		if (ret) {
-			blk_mq_sched_free_rqs(q);
 			elevator_exit(q);
 			goto out;
 		}

block/fops.c+16−19

@@ -75,8 +75,13 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, · 74 unmodified lines
75	return -ENOMEM;	75	return -ENOMEM;
76	}	76	}
77		77
78	bio_init(&bio, vecs, nr_pages);	78	if (iov_iter_rw(iter) == READ) {
79	bio_set_dev(&bio, bdev);	79	bio_init(&bio, bdev, vecs, nr_pages, REQ_OP_READ);
		80	if (iter_is_iovec(iter))
		81	should_dirty = true;
		82	} else {
		83	bio_init(&bio, bdev, vecs, nr_pages, dio_bio_write_op(iocb));
		84	}
80	bio.bi_iter.bi_sector = pos >> SECTOR_SHIFT;	85	bio.bi_iter.bi_sector = pos >> SECTOR_SHIFT;
81	bio.bi_write_hint = iocb->ki_hint;	86	bio.bi_write_hint = iocb->ki_hint;
82	bio.bi_private = current;	87	bio.bi_private = current;
@@ -88,14 +93,9 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, · 5 unmodified lines
88	goto out;	93	goto out;
89	ret = bio.bi_iter.bi_size;	94	ret = bio.bi_iter.bi_size;
90		95
91	if (iov_iter_rw(iter) == READ) {	96	if (iov_iter_rw(iter) == WRITE)
92	bio.bi_opf = REQ_OP_READ;
93	if (iter_is_iovec(iter))
94	should_dirty = true;
95	} else {
96	bio.bi_opf = dio_bio_write_op(iocb);
97	task_io_account_write(ret);	97	task_io_account_write(ret);
98	}	98
99	if (iocb->ki_flags & IOCB_NOWAIT)	99	if (iocb->ki_flags & IOCB_NOWAIT)
100	bio.bi_opf \|= REQ_NOWAIT;	100	bio.bi_opf \|= REQ_NOWAIT;
101	if (iocb->ki_flags & IOCB_HIPRI)	101	if (iocb->ki_flags & IOCB_HIPRI)
@@ -190,6 +190,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb iocb, struct iov_iter iter, · 88 unmodified lines
190	struct blkdev_dio *dio;	190	struct blkdev_dio *dio;
191	struct bio *bio;	191	struct bio *bio;
192	bool is_read = (iov_iter_rw(iter) == READ), is_sync;	192	bool is_read = (iov_iter_rw(iter) == READ), is_sync;
		193	unsigned int opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb);
193	loff_t pos = iocb->ki_pos;	194	loff_t pos = iocb->ki_pos;
194	int ret = 0;	195	int ret = 0;
195		196
@@ -197,7 +198,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb iocb, struct iov_iter iter, · 1 unmodified line
197	(bdev_logical_block_size(bdev) - 1))	198	(bdev_logical_block_size(bdev) - 1))
198	return -EINVAL;	199	return -EINVAL;
199		200
200	bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool);	201	bio = bio_alloc_kiocb(iocb, bdev, nr_pages, opf, &blkdev_dio_pool);
201		202
202	dio = container_of(bio, struct blkdev_dio, bio);	203	dio = container_of(bio, struct blkdev_dio, bio);
203	atomic_set(&dio->ref, 1);	204	atomic_set(&dio->ref, 1);
@@ -223,7 +224,6 @@ static ssize_t __blkdev_direct_IO(struct kiocb iocb, struct iov_iter iter, · 19 unmodified lines
223	blk_start_plug(&plug);	224	blk_start_plug(&plug);
224		225
225	for (;;) {	226	for (;;) {
226	bio_set_dev(bio, bdev);
227	bio->bi_iter.bi_sector = pos >> SECTOR_SHIFT;	227	bio->bi_iter.bi_sector = pos >> SECTOR_SHIFT;
228	bio->bi_write_hint = iocb->ki_hint;	228	bio->bi_write_hint = iocb->ki_hint;
229	bio->bi_private = dio;	229	bio->bi_private = dio;
@@ -238,11 +238,9 @@ static ssize_t __blkdev_direct_IO(struct kiocb iocb, struct iov_iter iter, · 8 unmodified lines
238	}	238	}
239		239
240	if (is_read) {	240	if (is_read) {
241	bio->bi_opf = REQ_OP_READ;
242	if (dio->flags & DIO_SHOULD_DIRTY)	241	if (dio->flags & DIO_SHOULD_DIRTY)
243	bio_set_pages_dirty(bio);	242	bio_set_pages_dirty(bio);
244	} else {	243	} else {
245	bio->bi_opf = dio_bio_write_op(iocb);
246	task_io_account_write(bio->bi_iter.bi_size);	244	task_io_account_write(bio->bi_iter.bi_size);
247	}	245	}
248	if (iocb->ki_flags & IOCB_NOWAIT)	246	if (iocb->ki_flags & IOCB_NOWAIT)
@@ -258,7 +256,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb iocb, struct iov_iter iter, · 9 unmodified lines
258	}	256	}
259	atomic_inc(&dio->ref);	257	atomic_inc(&dio->ref);
260	submit_bio(bio);	258	submit_bio(bio);
261	bio = bio_alloc(GFP_KERNEL, nr_pages);	259	bio = bio_alloc(bdev, nr_pages, opf, GFP_KERNEL);
262	}	260	}
263		261
264	blk_finish_plug(&plug);	262	blk_finish_plug(&plug);
@@ -313,6 +311,8 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, · 48 unmodified lines
313	unsigned int nr_pages)	311	unsigned int nr_pages)
314	{	312	{
315	struct block_device *bdev = iocb->ki_filp->private_data;	313	struct block_device *bdev = iocb->ki_filp->private_data;
		314	bool is_read = iov_iter_rw(iter) == READ;
		315	unsigned int opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb);
316	struct blkdev_dio *dio;	316	struct blkdev_dio *dio;
317	struct bio *bio;	317	struct bio *bio;
318	loff_t pos = iocb->ki_pos;	318	loff_t pos = iocb->ki_pos;
@@ -322,11 +322,10 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, · 3 unmodified lines
322	(bdev_logical_block_size(bdev) - 1))	322	(bdev_logical_block_size(bdev) - 1))
323	return -EINVAL;	323	return -EINVAL;
324		324
325	bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool);	325	bio = bio_alloc_kiocb(iocb, bdev, nr_pages, opf, &blkdev_dio_pool);
326	dio = container_of(bio, struct blkdev_dio, bio);	326	dio = container_of(bio, struct blkdev_dio, bio);
327	dio->flags = 0;	327	dio->flags = 0;
328	dio->iocb = iocb;	328	dio->iocb = iocb;
329	bio_set_dev(bio, bdev);
330	bio->bi_iter.bi_sector = pos >> SECTOR_SHIFT;	329	bio->bi_iter.bi_sector = pos >> SECTOR_SHIFT;
331	bio->bi_write_hint = iocb->ki_hint;	330	bio->bi_write_hint = iocb->ki_hint;
332	bio->bi_end_io = blkdev_bio_end_io_async;	331	bio->bi_end_io = blkdev_bio_end_io_async;
@@ -349,14 +348,12 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, · 16 unmodified lines
349	}	348	}
350	dio->size = bio->bi_iter.bi_size;	349	dio->size = bio->bi_iter.bi_size;
351		350
352	if (iov_iter_rw(iter) == READ) {	351	if (is_read) {
353	bio->bi_opf = REQ_OP_READ;
354	if (iter_is_iovec(iter)) {	352	if (iter_is_iovec(iter)) {
355	dio->flags \|= DIO_SHOULD_DIRTY;	353	dio->flags \|= DIO_SHOULD_DIRTY;
356	bio_set_pages_dirty(bio);	354	bio_set_pages_dirty(bio);
357	}	355	}
358	} else {	356	} else {
359	bio->bi_opf = dio_bio_write_op(iocb);
360	task_io_account_write(bio->bi_iter.bi_size);	357	task_io_account_write(bio->bi_iter.bi_size);
361	}	358	}
362		359

block/genhd.c+62−5

 #include <linux/module.h>
 #include <linux/ctype.h>
 #include <linux/fs.h>
-#include <linux/genhd.h>
 #include <linux/kdev_t.h>
 #include <linux/kernel.h>
 #include <linux/blkdev.h>
 #include <linux/pm_runtime.h>
 #include <linux/badblocks.h>
 #include <linux/part_stat.h>
+#include "blk-throttle.h"
 #include "blk.h"
 #include "blk-mq-sched.h"
 #include "blk-rq-qos.h"
+#include "blk-cgroup.h"
 static struct kobject *block_depr;
 	struct blk_major_name *next;
 	int major;
 	char name[16];
+#ifdef CONFIG_BLOCK_LEGACY_AUTOLOAD
 	void (*probe)(dev_t devt);
+#endif
 } *major_names[BLKDEV_MAJOR_HASH_SIZE];
 static DEFINE_MUTEX(major_names_lock);
 static DEFINE_SPINLOCK(major_names_spinlock);
 	}
 	p->major = major;
+#ifdef CONFIG_BLOCK_LEGACY_AUTOLOAD
 	p->probe = probe;
+#endif
 	strlcpy(p->name, name, sizeof(p->name));
 	p->next = NULL;
 	index = major_to_index(major);
 	disk_update_readahead(disk);
 	disk_add_events(disk);
+	set_bit(GD_ADDED, &disk->state);
 	return 0;
 out_unregister_bdi:
 	blk_mq_freeze_queue_wait(q);
-	rq_qos_exit(q);
+	blk_throtl_cancel_bios(disk->queue);
 	blk_sync_queue(q);
 	blk_flush_integrity();
 	/*
 	return badblocks_store(disk->bb, page, len, 0);
 }
+#ifdef CONFIG_BLOCK_LEGACY_AUTOLOAD
 void blk_request_module(dev_t devt)
 {
 	unsigned int major = MAJOR(devt);
 		/* Make old-style 2.4 aliases work */
 		request_module("block-major-%d", MAJOR(devt));
 }
+#endif /* CONFIG_BLOCK_LEGACY_AUTOLOAD */
 /*
  * print a full list of all partitions - intended for places where the root
 	struct disk_stats stat;
 	unsigned int inflight;
-	part_stat_read_all(bdev, &stat);
 	if (queue_is_mq(q))
 		inflight = blk_mq_in_flight(q, bdev);
 	else
 		inflight = part_in_flight(bdev);
+	if (inflight) {
+		part_stat_lock();
+		update_io_ticks(bdev, jiffies, true);
+		part_stat_unlock();
+	}
+	part_stat_read_all(bdev, &stat);
 	return sprintf(buf,
 		"%8lu %8lu %8llu %8u "
 		"%8lu %8lu %8llu %8u "
 	NULL
 };
+static void disk_release_mq(struct request_queue *q)
+{
+	blk_mq_cancel_work_sync(q);
+	/*
+	 * There can't be any non non-passthrough bios in flight here, but
+	 * requests stay around longer, including passthrough ones so we
+	 * still need to freeze the queue here.
+	 */
+	blk_mq_freeze_queue(q);
+	/*
+	 * Since the I/O scheduler exit code may access cgroup information,
+	 * perform I/O scheduler exit before disassociating from the block
+	 * cgroup controller.
+	 */
+	if (q->elevator) {
+		mutex_lock(&q->sysfs_lock);
+		elevator_exit(q);
+		mutex_unlock(&q->sysfs_lock);
+	}
+	rq_qos_exit(q);
+	__blk_mq_unfreeze_queue(q, true);
+}
 /**
  * disk_release - releases all allocated resources of the gendisk
  * @dev: the device representing this disk
 	might_sleep();
 	WARN_ON_ONCE(disk_live(disk));
-	blk_mq_cancel_work_sync(disk->queue);
+	if (queue_is_mq(disk->queue))
+		disk_release_mq(disk->queue);
+	blkcg_exit_queue(disk->queue);
 	disk_release_events(disk);
 	kfree(disk->random);
 	xa_destroy(&disk->part_tbl);
 	disk->queue->disk = NULL;
 	blk_put_queue(disk->queue);
+	if (test_bit(GD_ADDED, &disk->state) && disk->fops->free_disk)
+		disk->fops->free_disk(disk);
 	iput(disk->part0->bd_inode);	/* frees the disk */
 }
 	xa_for_each(&gp->part_tbl, idx, hd) {
 		if (bdev_is_partition(hd) && !bdev_nr_sectors(hd))
 			continue;
-		part_stat_read_all(hd, &stat);
 		if (queue_is_mq(gp->queue))
 			inflight = blk_mq_in_flight(gp->queue, hd);
 		else
 			inflight = part_in_flight(hd);
+		if (inflight) {
+			part_stat_lock();
+			update_io_ticks(hd, jiffies, true);
+			part_stat_unlock();
+		}
+		part_stat_read_all(hd, &stat);
 		seq_printf(seqf, "%4d %7d %pg "
 			   "%lu %lu %lu %u "
 			   "%lu %lu %lu %u "
 	if (xa_insert(&disk->part_tbl, 0, disk->part0, GFP_KERNEL))
 		goto out_destroy_part_tbl;
+	if (blkcg_init_queue(q))
+		goto out_erase_part0;
 	rand_initialize_disk(disk);
 	disk_to_dev(disk)->class = &block_class;
 	disk_to_dev(disk)->type = &disk_type;
 #endif
 	return disk;
+out_erase_part0:
+	xa_erase(&disk->part_tbl, 0);
 out_destroy_part_tbl:
 	xa_destroy(&disk->part_tbl);
 	disk->part0->bd_disk = NULL;

block/holder.c+1−1

@@ -1,5 +1,5 @@
1	// SPDX-License-Identifier: GPL-2.0-only	1	// SPDX-License-Identifier: GPL-2.0-only
2	#include <linux/genhd.h>	2	#include <linux/blkdev.h>
3	#include <linux/slab.h>	3	#include <linux/slab.h>
4		4
5	struct bd_holder_disk {	5	struct bd_holder_disk {

block/partitions/check.h+0−1

@@ -1,7 +1,6 @@
1	/* SPDX-License-Identifier: GPL-2.0 */	1	/* SPDX-License-Identifier: GPL-2.0 */
2	#include <linux/pagemap.h>	2	#include <linux/pagemap.h>
3	#include <linux/blkdev.h>	3	#include <linux/blkdev.h>
4	#include <linux/genhd.h>
5	#include "../blk.h"	4	#include "../blk.h"
6		5
7	/*	6	/*

block/partitions/core.c+0−1

@@ -8,7 +8,6 @@ · 7 unmodified lines
8	#include <linux/major.h>	8	#include <linux/major.h>
9	#include <linux/slab.h>	9	#include <linux/slab.h>
10	#include <linux/ctype.h>	10	#include <linux/ctype.h>
11	#include <linux/genhd.h>
12	#include <linux/vmalloc.h>	11	#include <linux/vmalloc.h>
13	#include <linux/blktrace_api.h>	12	#include <linux/blktrace_api.h>
14	#include <linux/raid/detect.h>	13	#include <linux/raid/detect.h>

block/partitions/efi.h+0−1

@@ -13,7 +13,6 @@ · 12 unmodified lines
13		13
14	#include <linux/types.h>	14	#include <linux/types.h>
15	#include <linux/fs.h>	15	#include <linux/fs.h>
16	#include <linux/genhd.h>
17	#include <linux/kernel.h>	16	#include <linux/kernel.h>
18	#include <linux/major.h>	17	#include <linux/major.h>
19	#include <linux/string.h>	18	#include <linux/string.h>

block/partitions/ldm.h+0−1

@@ -14,7 +14,6 @@ · 13 unmodified lines
14		14
15	#include <linux/types.h>	15	#include <linux/types.h>
16	#include <linux/list.h>	16	#include <linux/list.h>
17	#include <linux/genhd.h>
18	#include <linux/fs.h>	17	#include <linux/fs.h>
19	#include <asm/unaligned.h>	18	#include <asm/unaligned.h>
20	#include <asm/byteorder.h>	19	#include <asm/byteorder.h>

block/sed-opal.c+1−1

@@ -13,7 +13,7 @@ · 12 unmodified lines
13	#include <linux/device.h>	13	#include <linux/device.h>
14	#include <linux/kernel.h>	14	#include <linux/kernel.h>
15	#include <linux/list.h>	15	#include <linux/list.h>
16	#include <linux/genhd.h>	16	#include <linux/blkdev.h>
17	#include <linux/slab.h>	17	#include <linux/slab.h>
18	#include <linux/uaccess.h>	18	#include <linux/uaccess.h>
19	#include <uapi/linux/sed-opal.h>	19	#include <uapi/linux/sed-opal.h>

drivers/base/class.c+1−1

@@ -16,7 +16,7 @@ · 15 unmodified lines
16	#include <linux/kdev_t.h>	16	#include <linux/kdev_t.h>
17	#include <linux/err.h>	17	#include <linux/err.h>
18	#include <linux/slab.h>	18	#include <linux/slab.h>
19	#include <linux/genhd.h>	19	#include <linux/blkdev.h>
20	#include <linux/mutex.h>	20	#include <linux/mutex.h>
21	#include "base.h"	21	#include "base.h"
22		22

drivers/base/core.c+1−1

@@ -21,7 +21,7 @@ · 20 unmodified lines
21	#include <linux/notifier.h>	21	#include <linux/notifier.h>
22	#include <linux/of.h>	22	#include <linux/of.h>
23	#include <linux/of_device.h>	23	#include <linux/of_device.h>
24	#include <linux/genhd.h>	24	#include <linux/blkdev.h>
25	#include <linux/mutex.h>	25	#include <linux/mutex.h>
26	#include <linux/pm_runtime.h>	26	#include <linux/pm_runtime.h>
27	#include <linux/netdevice.h>	27	#include <linux/netdevice.h>

drivers/base/devtmpfs.c+1−1

@@ -17,7 +17,7 @@ · 16 unmodified lines
17	#include <linux/syscalls.h>	17	#include <linux/syscalls.h>
18	#include <linux/mount.h>	18	#include <linux/mount.h>
19	#include <linux/device.h>	19	#include <linux/device.h>
20	#include <linux/genhd.h>	20	#include <linux/blkdev.h>
21	#include <linux/namei.h>	21	#include <linux/namei.h>
22	#include <linux/fs.h>	22	#include <linux/fs.h>
23	#include <linux/shmem_fs.h>	23	#include <linux/shmem_fs.h>

drivers/block/aoe/aoeblk.c+0−1

@@ -12,7 +12,6 @@ · 11 unmodified lines
12	#include <linux/ioctl.h>	12	#include <linux/ioctl.h>
13	#include <linux/slab.h>	13	#include <linux/slab.h>
14	#include <linux/ratelimit.h>	14	#include <linux/ratelimit.h>
15	#include <linux/genhd.h>
16	#include <linux/netdevice.h>	15	#include <linux/netdevice.h>
17	#include <linux/mutex.h>	16	#include <linux/mutex.h>
18	#include <linux/export.h>	17	#include <linux/export.h>

drivers/block/aoe/aoecmd.c+0−1

@@ -10,7 +10,6 @@ · 9 unmodified lines
10	#include <linux/blk-mq.h>	10	#include <linux/blk-mq.h>
11	#include <linux/skbuff.h>	11	#include <linux/skbuff.h>
12	#include <linux/netdevice.h>	12	#include <linux/netdevice.h>
13	#include <linux/genhd.h>
14	#include <linux/moduleparam.h>	13	#include <linux/moduleparam.h>
15	#include <linux/workqueue.h>	14	#include <linux/workqueue.h>
16	#include <linux/kthread.h>	15	#include <linux/kthread.h>

drivers/block/drbd/drbd_actlog.c+2−3

 		op_flags |= REQ_FUA | REQ_PREFLUSH;
 	op_flags |= REQ_SYNC;
-	bio = bio_alloc_bioset(GFP_NOIO, 1, &drbd_md_io_bio_set);
+	bio = bio_alloc_bioset(bdev->md_bdev, 1, op | op_flags, GFP_NOIO,
-	bio_set_dev(bio, bdev->md_bdev);
+			       &drbd_md_io_bio_set);
 	bio->bi_iter.bi_sector = sector;
 	err = -EIO;
 	if (bio_add_page(bio, device->md_io.page, size, 0) != size)
 		goto out;
 	bio->bi_private = device;
 	bio->bi_end_io = drbd_md_endio;
-	bio_set_op_attrs(bio, op, op_flags);
 	if (op != REQ_OP_WRITE && device->state.disk == D_DISKLESS && device->ldev == NULL)
 		/* special case, drbd_md_read() during drbd_adm_attach(): no get_ldev */

drivers/block/drbd/drbd_bitmap.c+3−4

 static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_hold(local)
 {
-	struct bio *bio = bio_alloc_bioset(GFP_NOIO, 1, &drbd_md_io_bio_set);
 	struct drbd_device *device = ctx->device;
+	unsigned int op = (ctx->flags & BM_AIO_READ) ? REQ_OP_READ : REQ_OP_WRITE;
+	struct bio *bio = bio_alloc_bioset(device->ldev->md_bdev, 1, op,
+					   GFP_NOIO, &drbd_md_io_bio_set);
 	struct drbd_bitmap *b = device->bitmap;
 	struct page *page;
 	unsigned int len;
-	unsigned int op = (ctx->flags & BM_AIO_READ) ? REQ_OP_READ : REQ_OP_WRITE;
 	sector_t on_disk_sector =
 		device->ldev->md.md_offset + device->ldev->md.bm_offset;
 		bm_store_page_idx(page, page_nr);
 	} else
 		page = b->bm_pages[page_nr];
-	bio_set_dev(bio, device->ldev->md_bdev);
 	bio->bi_iter.bi_sector = on_disk_sector;
 	/* bio_add_page of a single page to an empty bio will always succeed,
 	 * according to api.  Do we want to assert that? */
 	bio_add_page(bio, page, len, 0);
 	bio->bi_private = ctx;
 	bio->bi_end_io = drbd_bm_endio;
-	bio_set_op_attrs(bio, op, 0);
 	if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) {
 		bio_io_error(bio);

drivers/block/drbd/drbd_int.h+0−1

@@ -27,7 +27,6 @@ · 26 unmodified lines
27	#include <linux/major.h>	27	#include <linux/major.h>
28	#include <linux/blkdev.h>	28	#include <linux/blkdev.h>
29	#include <linux/backing-dev.h>	29	#include <linux/backing-dev.h>
30	#include <linux/genhd.h>
31	#include <linux/idr.h>	30	#include <linux/idr.h>
32	#include <linux/dynamic_debug.h>	31	#include <linux/dynamic_debug.h>
33	#include <net/tcp.h>	32	#include <net/tcp.h>

drivers/block/drbd/drbd_receiver.c+8−24

@@ -1279,16 +1279,16 @@ static void one_flush_endio(struct bio *bio) · 1278 unmodified lines
1279		1279
1280	static void submit_one_flush(struct drbd_device device, struct issue_flush_context ctx)	1280	static void submit_one_flush(struct drbd_device device, struct issue_flush_context ctx)
1281	{	1281	{
1282	struct bio *bio = bio_alloc(GFP_NOIO, 0);	1282	struct bio *bio = bio_alloc(device->ldev->backing_bdev, 0,
		1283	REQ_OP_FLUSH \| REQ_PREFLUSH, GFP_NOIO);
1283	struct one_flush_context octx = kmalloc(sizeof(octx), GFP_NOIO);	1284	struct one_flush_context octx = kmalloc(sizeof(octx), GFP_NOIO);
1284	if (!bio \|\| !octx) {	1285
1285	drbd_warn(device, "Could not allocate a bio, CANNOT ISSUE FLUSH\n");	1286	if (!octx) {
		1287	drbd_warn(device, "Could not allocate a octx, CANNOT ISSUE FLUSH\n");
1286	/* FIXME: what else can I do now? disconnecting or detaching	1288	/* FIXME: what else can I do now? disconnecting or detaching
1287	* really does not help to improve the state of the world, either.	1289	* really does not help to improve the state of the world, either.
1288	*/	1290	*/
1289	kfree(octx);	1291	bio_put(bio);
1290	if (bio)
1291	bio_put(bio);
1292		1292
1293	ctx->error = -ENOMEM;	1293	ctx->error = -ENOMEM;
1294	put_ldev(device);	1294	put_ldev(device);
@@ -1298,10 +1298,8 @@ static void submit_one_flush(struct drbd_device *device, struct issue_flush_cont · 3 unmodified lines
1298		1298
1299	octx->device = device;	1299	octx->device = device;
1300	octx->ctx = ctx;	1300	octx->ctx = ctx;
1301	bio_set_dev(bio, device->ldev->backing_bdev);
1302	bio->bi_private = octx;	1301	bio->bi_private = octx;
1303	bio->bi_end_io = one_flush_endio;	1302	bio->bi_end_io = one_flush_endio;
1304	bio->bi_opf = REQ_OP_FLUSH \| REQ_PREFLUSH;
1305		1303
1306	device->flush_jif = jiffies;	1304	device->flush_jif = jiffies;
1307	set_bit(FLUSH_PENDING, &device->flags);	1305	set_bit(FLUSH_PENDING, &device->flags);
@@ -1646,7 +1644,6 @@ int drbd_submit_peer_request(struct drbd_device *device, · 338 unmodified lines
1646	unsigned data_size = peer_req->i.size;	1644	unsigned data_size = peer_req->i.size;
1647	unsigned n_bios = 0;	1645	unsigned n_bios = 0;
1648	unsigned nr_pages = (data_size + PAGE_SIZE -1) >> PAGE_SHIFT;	1646	unsigned nr_pages = (data_size + PAGE_SIZE -1) >> PAGE_SHIFT;
1649	int err = -ENOMEM;
1650		1647
1651	/* TRIM/DISCARD: for now, always use the helper function	1648	/* TRIM/DISCARD: for now, always use the helper function
1652	* blkdev_issue_zeroout(..., discard=true).	1649	* blkdev_issue_zeroout(..., discard=true).
@@ -1687,15 +1684,10 @@ int drbd_submit_peer_request(struct drbd_device *device, · 34 unmodified lines
1687	* generated bio, but a bio allocated on behalf of the peer.	1684	* generated bio, but a bio allocated on behalf of the peer.
1688	*/	1685	*/
1689	next_bio:	1686	next_bio:
1690	bio = bio_alloc(GFP_NOIO, nr_pages);	1687	bio = bio_alloc(device->ldev->backing_bdev, nr_pages, op \| op_flags,
1691	if (!bio) {	1688	GFP_NOIO);
1692	drbd_err(device, "submit_ee: Allocation of a bio failed (nr_pages=%u)\n", nr_pages);
1693	goto fail;
1694	}
1695	/* > peer_req->i.sector, unless this is the first bio */	1689	/* > peer_req->i.sector, unless this is the first bio */
1696	bio->bi_iter.bi_sector = sector;	1690	bio->bi_iter.bi_sector = sector;
1697	bio_set_dev(bio, device->ldev->backing_bdev);
1698	bio_set_op_attrs(bio, op, op_flags);
1699	bio->bi_private = peer_req;	1691	bio->bi_private = peer_req;
1700	bio->bi_end_io = drbd_peer_request_endio;	1692	bio->bi_end_io = drbd_peer_request_endio;
1701		1693
@@ -1726,14 +1718,6 @@ next_bio: · 24 unmodified lines
1726	drbd_submit_bio_noacct(device, fault_type, bio);	1718	drbd_submit_bio_noacct(device, fault_type, bio);
1727	} while (bios);	1719	} while (bios);
1728	return 0;	1720	return 0;
1729
1730	fail:
1731	while (bios) {
1732	bio = bios;
1733	bios = bios->bi_next;
1734	bio_put(bio);
1735	}
1736	return err;
1737	}	1721	}
1738		1722
1739	static void drbd_remove_epoch_entry_interval(struct drbd_device *device,	1723	static void drbd_remove_epoch_entry_interval(struct drbd_device *device,

drivers/block/drbd/drbd_req.c+2−3

 		return NULL;
 	memset(req, 0, sizeof(*req));
-	req->private_bio = bio_clone_fast(bio_src, GFP_NOIO, &drbd_io_bio_set);
+	req->private_bio = bio_alloc_clone(device->ldev->backing_bdev, bio_src,
+					   GFP_NOIO, &drbd_io_bio_set);
 	req->private_bio->bi_private = req;
 	req->private_bio->bi_end_io = drbd_request_endio;
 	else
 		type = DRBD_FAULT_DT_RD;
-	bio_set_dev(bio, device->ldev->backing_bdev);
 	/* State may have changed since we grabbed our reference on the
 	 * ->ldev member. Double check, and short-circuit to endio.
 	 * In case the last activity log transaction failed to get on

drivers/block/drbd/drbd_worker.c+2−2

 	if (bio_data_dir(req->master_bio) == WRITE && req->rq_state & RQ_IN_ACT_LOG)
 		drbd_al_begin_io(device, &req->i);
-	req->private_bio = bio_clone_fast(req->master_bio, GFP_NOIO,
+	req->private_bio = bio_alloc_clone(device->ldev->backing_bdev,
+					   req->master_bio, GFP_NOIO,
 					  &drbd_io_bio_set);
-	bio_set_dev(req->private_bio, device->ldev->backing_bdev);
 	req->private_bio->bi_private = req;
 	req->private_bio->bi_end_io = drbd_request_endio;
 	submit_bio_noacct(req->private_bio);

drivers/block/floppy.c+1−3

 	cbdata.drive = drive;
-	bio_init(&bio, &bio_vec, 1);
+	bio_init(&bio, bdev, &bio_vec, 1, REQ_OP_READ);
-	bio_set_dev(&bio, bdev);
 	bio_add_page(&bio, page, block_size(bdev), 0);
 	bio.bi_iter.bi_sector = 0;
 	bio.bi_flags |= (1 << BIO_QUIET);
 	bio.bi_private = &cbdata;
 	bio.bi_end_io = floppy_rb0_cb;
-	bio_set_op_attrs(&bio, REQ_OP_READ, 0);
 	init_completion(&cbdata.complete);

drivers/block/mtip32xx/mtip32xx.c+1−4

 #include <linux/compat.h>
 #include <linux/fs.h>
 #include <linux/module.h>
-#include <linux/genhd.h>
 #include <linux/blkdev.h>
 #include <linux/blk-mq.h>
 #include <linux/bio.h>
 static struct mtip_cmd *mtip_cmd_from_tag(struct driver_data *dd,
 					  unsigned int tag)
 {
-	struct blk_mq_hw_ctx *hctx = dd->queue->queue_hw_ctx[0];
+	return blk_mq_rq_to_pdu(blk_mq_tag_to_rq(dd->tags.tags[0], tag));
-	return blk_mq_rq_to_pdu(blk_mq_tag_to_rq(hctx->tags, tag));
 }
 /*

drivers/block/mtip32xx/mtip32xx.h+0−1

@@ -15,7 +15,6 @@ · 14 unmodified lines
15	#include <linux/rwsem.h>	15	#include <linux/rwsem.h>
16	#include <linux/ata.h>	16	#include <linux/ata.h>
17	#include <linux/interrupt.h>	17	#include <linux/interrupt.h>
18	#include <linux/genhd.h>
19		18
20	/* Offset of Subsystem Device ID in pci confoguration space */	19	/* Offset of Subsystem Device ID in pci confoguration space */
21	#define PCI_SUBSYSTEM_DEVICEID 0x2E	20	#define PCI_SUBSYSTEM_DEVICEID 0x2E

drivers/block/pktcdvd.c+5−16

 			continue;
 		bio = pkt->r_bios[f];
-		bio_reset(bio);
+		bio_reset(bio, pd->bdev, REQ_OP_READ);
 		bio->bi_iter.bi_sector = pkt->sector + f * (CD_FRAMESIZE >> 9);
-		bio_set_dev(bio, pd->bdev);
 		bio->bi_end_io = pkt_end_io_read;
 		bio->bi_private = pkt;
 			BUG();
 		atomic_inc(&pkt->io_wait);
-		bio_set_op_attrs(bio, REQ_OP_READ, 0);
 		pkt_queue_bio(pd, bio);
 		frames_read++;
 	}
 {
 	int f;
-	bio_reset(pkt->w_bio);
+	bio_reset(pkt->w_bio, pd->bdev, REQ_OP_WRITE);
 	pkt->w_bio->bi_iter.bi_sector = pkt->sector;
-	bio_set_dev(pkt->w_bio, pd->bdev);
 	pkt->w_bio->bi_end_io = pkt_end_io_packet_write;
 	pkt->w_bio->bi_private = pkt;
 	/* Start the write request */
 	atomic_set(&pkt->io_wait, 1);
-	bio_set_op_attrs(pkt->w_bio, REQ_OP_WRITE, 0);
 	pkt_queue_bio(pd, pkt->w_bio);
 }
 static void pkt_make_request_read(struct pktcdvd_device *pd, struct bio *bio)
 {
-	struct bio *cloned_bio = bio_clone_fast(bio, GFP_NOIO, &pkt_bio_set);
+	struct bio *cloned_bio =
+		bio_alloc_clone(pd->bdev, bio, GFP_NOIO, &pkt_bio_set);
 	struct packet_stacked_data *psd = mempool_alloc(&psd_pool, GFP_NOIO);
 	psd->pd = pd;
 	psd->bio = bio;
-	bio_set_dev(cloned_bio, pd->bdev);
 	cloned_bio->bi_private = psd;
 	cloned_bio->bi_end_io = pkt_end_io_read_cloned;
 	pd->stats.secs_r += bio_sectors(bio);
 static void pkt_submit_bio(struct bio *bio)
 {
-	struct pktcdvd_device *pd;
+	struct pktcdvd_device *pd = bio->bi_bdev->bd_disk->queue->queuedata;
-	char b[BDEVNAME_SIZE];
 	struct bio *split;
 	blk_queue_split(&bio);
-	pd = bio->bi_bdev->bd_disk->queue->queuedata;
-	if (!pd) {
-		pr_err("%s incorrect request queue\n", bio_devname(bio, b));
-		goto end_io;
-	}
 	pkt_dbg(2, pd, "start = %6llx stop = %6llx\n",
 		(unsigned long long)bio->bi_iter.bi_sector,
 		(unsigned long long)bio_end_sector(bio));

drivers/block/rnbd/rnbd-clt.c+1−1

@@ -1343,7 +1343,7 @@ static inline void rnbd_init_hw_queue(struct rnbd_clt_dev *dev, · 1342 unmodified lines
1343		1343
1344	static void rnbd_init_mq_hw_queues(struct rnbd_clt_dev *dev)	1344	static void rnbd_init_mq_hw_queues(struct rnbd_clt_dev *dev)
1345	{	1345	{
1346	int i;	1346	unsigned long i;
1347	struct blk_mq_hw_ctx *hctx;	1347	struct blk_mq_hw_ctx *hctx;
1348	struct rnbd_queue *q;	1348	struct rnbd_queue *q;
1349		1349

drivers/block/rnbd/rnbd-srv-dev.c+1−60

@@ -12,8 +12,7 @@ · 11 unmodified lines
12	#include "rnbd-srv-dev.h"	12	#include "rnbd-srv-dev.h"
13	#include "rnbd-log.h"	13	#include "rnbd-log.h"
14		14
15	struct rnbd_dev rnbd_dev_open(const char path, fmode_t flags,	15	struct rnbd_dev rnbd_dev_open(const char path, fmode_t flags)
16	struct bio_set *bs)
17	{	16	{
18	struct rnbd_dev *dev;	17	struct rnbd_dev *dev;
19	int ret;	18	int ret;
@@ -30,7 +29,6 @@ struct rnbd_dev rnbd_dev_open(const char path, fmode_t flags, · 10 unmodified lines
30		29
31	dev->blk_open_flags = flags;	30	dev->blk_open_flags = flags;
32	bdevname(dev->bdev, dev->name);	31	bdevname(dev->bdev, dev->name);
33	dev->ibd_bio_set = bs;
34		32
35	return dev;	33	return dev;
36		34
@@ -44,60 +42,3 @@ void rnbd_dev_close(struct rnbd_dev *dev) · 7 unmodified lines
44	blkdev_put(dev->bdev, dev->blk_open_flags);	42	blkdev_put(dev->bdev, dev->blk_open_flags);
45	kfree(dev);	43	kfree(dev);
46	}	44	}
47
48	void rnbd_dev_bi_end_io(struct bio *bio)
49	{
50	struct rnbd_dev_blk_io *io = bio->bi_private;
51
52	rnbd_endio(io->priv, blk_status_to_errno(bio->bi_status));
53	bio_put(bio);
54	}
55
56	/**
57	* rnbd_bio_map_kern - map kernel address into bio
58	* @data: pointer to buffer to map
59	* @bs: bio_set to use.
60	* @len: length in bytes
61	* @gfp_mask: allocation flags for bio allocation
62	*
63	* Map the kernel address into a bio suitable for io to a block
64	* device. Returns an error pointer in case of error.
65	*/
66	struct bio rnbd_bio_map_kern(void data, struct bio_set *bs,
67	unsigned int len, gfp_t gfp_mask)
68	{
69	unsigned long kaddr = (unsigned long)data;
70	unsigned long end = (kaddr + len + PAGE_SIZE - 1) >> PAGE_SHIFT;
71	unsigned long start = kaddr >> PAGE_SHIFT;
72	const int nr_pages = end - start;
73	int offset, i;
74	struct bio *bio;
75
76	bio = bio_alloc_bioset(gfp_mask, nr_pages, bs);
77	if (!bio)
78	return ERR_PTR(-ENOMEM);
79
80	offset = offset_in_page(kaddr);
81	for (i = 0; i < nr_pages; i++) {
82	unsigned int bytes = PAGE_SIZE - offset;
83
84	if (len <= 0)
85	break;
86
87	if (bytes > len)
88	bytes = len;
89
90	if (bio_add_page(bio, virt_to_page(data), bytes,
91	offset) < bytes) {
92	/* we don't support partial mappings */
93	bio_put(bio);
94	return ERR_PTR(-EINVAL);
95	}
96
97	data += bytes;
98	len -= bytes;
99	offset = 0;
100	}
101
102	return bio;
103	}

drivers/block/rnbd/rnbd-srv-dev.h+2−16

 struct rnbd_dev {
 	struct block_device	*bdev;
-	struct bio_set		*ibd_bio_set;
 	fmode_t			blk_open_flags;
 	char			name[BDEVNAME_SIZE];
 };
-struct rnbd_dev_blk_io {
-	struct rnbd_dev *dev;
-	void		 *priv;
-	/* have to be last member for front_pad usage of bioset_init */
-	struct bio	bio;
-};
 /**
  * rnbd_dev_open() - Open a device
+ * @path:	path to open
  * @flags:	open flags
- * @bs:		bio_set to use during block io,
  */
-struct rnbd_dev *rnbd_dev_open(const char *path, fmode_t flags,
+struct rnbd_dev *rnbd_dev_open(const char *path, fmode_t flags);
-			       struct bio_set *bs);
 /**
  * rnbd_dev_close() - Close a device
 void rnbd_endio(void *priv, int error);
-void rnbd_dev_bi_end_io(struct bio *bio);
-struct bio *rnbd_bio_map_kern(void *data, struct bio_set *bs,
-			      unsigned int len, gfp_t gfp_mask);
 static inline int rnbd_dev_get_max_segs(const struct rnbd_dev *dev)
 {
 	return queue_max_segments(bdev_get_queue(dev->bdev));

drivers/block/rnbd/rnbd-srv-sysfs.c+0−1

@@ -13,7 +13,6 @@ · 12 unmodified lines
13	#include <linux/kobject.h>	13	#include <linux/kobject.h>
14	#include <linux/sysfs.h>	14	#include <linux/sysfs.h>
15	#include <linux/stat.h>	15	#include <linux/stat.h>
16	#include <linux/genhd.h>
17	#include <linux/list.h>	16	#include <linux/list.h>
18	#include <linux/moduleparam.h>	17	#include <linux/moduleparam.h>
19	#include <linux/device.h>	18	#include <linux/device.h>

drivers/block/rnbd/rnbd-srv.c+17−28

 	return sess_dev;
 }
+static void rnbd_dev_bi_end_io(struct bio *bio)
+{
+	rnbd_endio(bio->bi_private, blk_status_to_errno(bio->bi_status));
+	bio_put(bio);
+}
 static int process_rdma(struct rnbd_srv_session *srv_sess,
 			struct rtrs_srv_op *id, void *data, u32 datalen,
 			const void *usr, size_t usrlen)
 	struct rnbd_srv_sess_dev *sess_dev;
 	u32 dev_id;
 	int err;
-	struct rnbd_dev_blk_io *io;
 	struct bio *bio;
 	short prio;
 	priv->sess_dev = sess_dev;
 	priv->id = id;
-	/* Generate bio with pages pointing to the rdma buffer */
+	bio = bio_alloc(sess_dev->rnbd_dev->bdev, 1,
-	bio = rnbd_bio_map_kern(data, sess_dev->rnbd_dev->ibd_bio_set, datalen, GFP_KERNEL);
+			rnbd_to_bio_flags(le32_to_cpu(msg->rw)), GFP_KERNEL);
-	if (IS_ERR(bio)) {
+	if (bio_add_page(bio, virt_to_page(data), datalen,
-		err = PTR_ERR(bio);
+			offset_in_page(data)) != datalen) {
-		rnbd_srv_err(sess_dev, "Failed to generate bio, err: %d\n", err);
+		rnbd_srv_err(sess_dev, "Failed to map data to bio\n");
-		goto sess_dev_put;
+		err = -EINVAL;
+		goto bio_put;
 	}
-	io = container_of(bio, struct rnbd_dev_blk_io, bio);
-	io->dev = sess_dev->rnbd_dev;
-	io->priv = priv;
 	bio->bi_end_io = rnbd_dev_bi_end_io;
-	bio->bi_private = io;
+	bio->bi_private = priv;
-	bio->bi_opf = rnbd_to_bio_flags(le32_to_cpu(msg->rw));
 	bio->bi_iter.bi_sector = le64_to_cpu(msg->sector);
 	bio->bi_iter.bi_size = le32_to_cpu(msg->bi_size);
 	prio = srv_sess->ver < RNBD_PROTO_VER_MAJOR ||
 	       usrlen < sizeof(*msg) ? 0 : le16_to_cpu(msg->prio);
 	bio_set_prio(bio, prio);
-	bio_set_dev(bio, sess_dev->rnbd_dev->bdev);
 	submit_bio(bio);
 	return 0;
-sess_dev_put:
+bio_put:
+	bio_put(bio);
 	rnbd_put_sess_dev(sess_dev);
 err:
 	kfree(priv);
 out:
 	xa_destroy(&srv_sess->index_idr);
-	bioset_exit(&srv_sess->sess_bio_set);
 	pr_info("RTRS Session %s disconnected\n", srv_sess->sessname);
 		return -ENOMEM;
 	srv_sess->queue_depth = rtrs_srv_get_queue_depth(rtrs);
-	err = bioset_init(&srv_sess->sess_bio_set, srv_sess->queue_depth,
-			  offsetof(struct rnbd_dev_blk_io, bio),
-			  BIOSET_NEED_BVECS);
-	if (err) {
-		pr_err("Allocating srv_session for path %s failed\n",
-		       pathname);
-		kfree(srv_sess);
-		return err;
-	}
 	xa_init_flags(&srv_sess->index_idr, XA_FLAGS_ALLOC);
 	INIT_LIST_HEAD(&srv_sess->sess_dev_list);
 	mutex_init(&srv_sess->lock);
 		goto reject;
 	}
-	rnbd_dev = rnbd_dev_open(full_path, open_flags,
+	rnbd_dev = rnbd_dev_open(full_path, open_flags);
-				 &srv_sess->sess_bio_set);
 	if (IS_ERR(rnbd_dev)) {
 		pr_err("Opening device '%s' on session %s failed, failed to open the block device, err: %ld\n",
 		       full_path, srv_sess->sessname, PTR_ERR(rnbd_dev));

drivers/block/rnbd/rnbd-srv.h+0−1

@@ -23,7 +23,6 @@ struct rnbd_srv_session { · 22 unmodified lines
23	struct rtrs_srv_sess *rtrs;	23	struct rtrs_srv_sess *rtrs;
24	char sessname[NAME_MAX];	24	char sessname[NAME_MAX];
25	int queue_depth;	25	int queue_depth;
26	struct bio_set sess_bio_set;
27		26
28	struct xarray index_idr;	27	struct xarray index_idr;
29	/* List of struct rnbd_srv_sess_dev */	28	/* List of struct rnbd_srv_sess_dev */

drivers/block/sunvdc.c+0−1

@@ -9,7 +9,6 @@ · 8 unmodified lines
9	#include <linux/types.h>	9	#include <linux/types.h>
10	#include <linux/blk-mq.h>	10	#include <linux/blk-mq.h>
11	#include <linux/hdreg.h>	11	#include <linux/hdreg.h>
12	#include <linux/genhd.h>
13	#include <linux/cdrom.h>	12	#include <linux/cdrom.h>
14	#include <linux/slab.h>	13	#include <linux/slab.h>
15	#include <linux/spinlock.h>	14	#include <linux/spinlock.h>

drivers/block/virtio_blk.c+14−52

 	/* Process context for config space updates */
 	struct work_struct config_work;
-	/*
-	 * Tracks references from block_device_operations open/release and
-	 * virtio_driver probe/remove so this object can be freed once no
-	 * longer in use.
-	 */
-	refcount_t refs;
 	/* Ida index - used to track minor number allocations. */
 	int index;
 	return err;
 }
-static void virtblk_get(struct virtio_blk *vblk)
-{
-	refcount_inc(&vblk->refs);
-}
-static void virtblk_put(struct virtio_blk *vblk)
-{
-	if (refcount_dec_and_test(&vblk->refs)) {
-		ida_simple_remove(&vd_index_ida, vblk->index);
-		mutex_destroy(&vblk->vdev_mutex);
-		kfree(vblk);
-	}
-}
-static int virtblk_open(struct block_device *bd, fmode_t mode)
-{
-	struct virtio_blk *vblk = bd->bd_disk->private_data;
-	int ret = 0;
-	mutex_lock(&vblk->vdev_mutex);
-	if (vblk->vdev)
-		virtblk_get(vblk);
-	else
-		ret = -ENXIO;
-	mutex_unlock(&vblk->vdev_mutex);
-	return ret;
-}
-static void virtblk_release(struct gendisk *disk, fmode_t mode)
-{
-	struct virtio_blk *vblk = disk->private_data;
-	virtblk_put(vblk);
-}
 /* We provide getgeo only to please some old bootloader/partitioning tools */
 static int virtblk_getgeo(struct block_device *bd, struct hd_geometry *geo)
 {
 	return ret;
 }
+static void virtblk_free_disk(struct gendisk *disk)
+{
+	struct virtio_blk *vblk = disk->private_data;
+	ida_simple_remove(&vd_index_ida, vblk->index);
+	mutex_destroy(&vblk->vdev_mutex);
+	kfree(vblk);
+}
 static const struct block_device_operations virtblk_fops = {
-	.owner  = THIS_MODULE,
+	.owner  	= THIS_MODULE,
-	.open = virtblk_open,
+	.getgeo		= virtblk_getgeo,
-	.release = virtblk_release,
+	.free_disk	= virtblk_free_disk,
-	.getgeo = virtblk_getgeo,
 };
 static int index_to_minor(int index)
 		goto out_free_index;
 	}
-	/* This reference is dropped in virtblk_remove(). */
-	refcount_set(&vblk->refs, 1);
 	mutex_init(&vblk->vdev_mutex);
 	vblk->vdev = vdev;
 	flush_work(&vblk->config_work);
 	del_gendisk(vblk->disk);
-	blk_cleanup_disk(vblk->disk);
+	blk_cleanup_queue(vblk->disk->queue);
 	blk_mq_free_tag_set(&vblk->tag_set);
 	mutex_lock(&vblk->vdev_mutex);
 	mutex_unlock(&vblk->vdev_mutex);
-	virtblk_put(vblk);
+	put_disk(vblk->disk);
 }
 #ifdef CONFIG_PM_SLEEP

drivers/block/xen-blkback/blkback.c+5−20

 				     pages[i]->page,
 				     seg[i].nsec << 9,
 				     seg[i].offset) == 0)) {
-			bio = bio_alloc(GFP_KERNEL, bio_max_segs(nseg - i));
+			bio = bio_alloc(preq.bdev, bio_max_segs(nseg - i),
-			if (unlikely(bio == NULL))
+					operation | operation_flags,
-				goto fail_put_bio;
+					GFP_KERNEL);
 			biolist[nbio++] = bio;
-			bio_set_dev(bio, preq.bdev);
 			bio->bi_private = pending_req;
 			bio->bi_end_io  = end_block_io_op;
 			bio->bi_iter.bi_sector  = preq.sector_number;
-			bio_set_op_attrs(bio, operation, operation_flags);
 		}
 		preq.sector_number += seg[i].nsec;
 	if (!bio) {
 		BUG_ON(operation_flags != REQ_PREFLUSH);
-		bio = bio_alloc(GFP_KERNEL, 0);
+		bio = bio_alloc(preq.bdev, 0, operation | operation_flags,
-		if (unlikely(bio == NULL))
+				GFP_KERNEL);
-			goto fail_put_bio;
 		biolist[nbio++] = bio;
-		bio_set_dev(bio, preq.bdev);
 		bio->bi_private = pending_req;
 		bio->bi_end_io  = end_block_io_op;
-		bio_set_op_attrs(bio, operation, operation_flags);
 	}
 	atomic_set(&pending_req->pendcnt, nbio);
 	free_req(ring, pending_req);
 	msleep(1); /* back off a bit */
 	return -EIO;
- fail_put_bio:
-	for (i = 0; i < nbio; i++)
-		bio_put(biolist[i]);
-	atomic_set(&pending_req->pendcnt, 1);
-	__end_block_io_op(pending_req, BLK_STS_RESOURCE);
-	msleep(1); /* back off a bit */
-	return -EIO;
 }

drivers/block/zram/zram_drv.c+6−11

 #include <linux/blkdev.h>
 #include <linux/buffer_head.h>
 #include <linux/device.h>
-#include <linux/genhd.h>
 #include <linux/highmem.h>
 #include <linux/slab.h>
 #include <linux/backing-dev.h>
 {
 	struct bio *bio;
-	bio = bio_alloc(GFP_NOIO, 1);
+	bio = bio_alloc(zram->bdev, 1, parent ? parent->bi_opf : REQ_OP_READ,
+			GFP_NOIO);
 	if (!bio)
 		return -ENOMEM;
 	bio->bi_iter.bi_sector = entry * (PAGE_SIZE >> 9);
-	bio_set_dev(bio, zram->bdev);
 	if (!bio_add_page(bio, bvec->bv_page, bvec->bv_len, bvec->bv_offset)) {
 		bio_put(bio);
 		return -EIO;
 	}
-	if (!parent) {
+	if (!parent)
-		bio->bi_opf = REQ_OP_READ;
 		bio->bi_end_io = zram_page_end_io;
-	} else {
+	else
-		bio->bi_opf = parent->bi_opf;
 		bio_chain(bio, parent);
-	}
 	submit_bio(bio);
 	return 1;
 			continue;
 		}
-		bio_init(&bio, &bio_vec, 1);
+		bio_init(&bio, zram->bdev, &bio_vec, 1,
-		bio_set_dev(&bio, zram->bdev);
+			 REQ_OP_WRITE | REQ_SYNC);
 		bio.bi_iter.bi_sector = blk_idx * (PAGE_SIZE >> 9);
-		bio.bi_opf = REQ_OP_WRITE | REQ_SYNC;
 		bio_add_page(&bio, bvec.bv_page, bvec.bv_len,
 				bvec.bv_offset);

drivers/cdrom/gdrom.c+0−1

@@ -15,7 +15,6 @@ · 14 unmodified lines
15	#include <linux/slab.h>	15	#include <linux/slab.h>
16	#include <linux/dma-mapping.h>	16	#include <linux/dma-mapping.h>
17	#include <linux/cdrom.h>	17	#include <linux/cdrom.h>
18	#include <linux/genhd.h>
19	#include <linux/bio.h>	18	#include <linux/bio.h>
20	#include <linux/blk-mq.h>	19	#include <linux/blk-mq.h>
21	#include <linux/interrupt.h>	20	#include <linux/interrupt.h>

drivers/char/random.c+1−1

@@ -38,7 +38,7 @@ · 37 unmodified lines
38	#include <linux/poll.h>	38	#include <linux/poll.h>
39	#include <linux/init.h>	39	#include <linux/init.h>
40	#include <linux/fs.h>	40	#include <linux/fs.h>
41	#include <linux/genhd.h>	41	#include <linux/blkdev.h>
42	#include <linux/interrupt.h>	42	#include <linux/interrupt.h>
43	#include <linux/mm.h>	43	#include <linux/mm.h>
44	#include <linux/nodemask.h>	44	#include <linux/nodemask.h>

drivers/md/Kconfig+1−0

@@ -204,6 +204,7 @@ config BLK_DEV_DM · 203 unmodified lines
204	tristate "Device mapper support"	204	tristate "Device mapper support"
205	select BLOCK_HOLDER_DEPRECATED if SYSFS	205	select BLOCK_HOLDER_DEPRECATED if SYSFS
206	select BLK_DEV_DM_BUILTIN	206	select BLK_DEV_DM_BUILTIN
		207	select BLK_MQ_STACKING
207	depends on DAX \|\| DAX=n	208	depends on DAX \|\| DAX=n
208	help	209	help
209	Device-mapper is a low level volume manager. It works by allowing	210	Device-mapper is a low level volume manager. It works by allowing

drivers/md/bcache/io.c+2−1

@@ -26,7 +26,8 @@ struct bio bch_bbio_alloc(struct cache_set c) · 25 unmodified lines
26	struct bbio *b = mempool_alloc(&c->bio_meta, GFP_NOIO);	26	struct bbio *b = mempool_alloc(&c->bio_meta, GFP_NOIO);
27	struct bio *bio = &b->bio;	27	struct bio *bio = &b->bio;
28		28
29	bio_init(bio, bio->bi_inline_vecs, meta_bucket_pages(&c->cache->sb));	29	bio_init(bio, NULL, bio->bi_inline_vecs,
		30	meta_bucket_pages(&c->cache->sb), 0);
30		31
31	return bio;	32	return bio;
32	}	33	}

drivers/md/bcache/journal.c+5−11

 reread:		left = ca->sb.bucket_size - offset;
 		len = min_t(unsigned int, left, PAGE_SECTORS << JSET_BITS);
-		bio_reset(bio);
+		bio_reset(bio, ca->bdev, REQ_OP_READ);
 		bio->bi_iter.bi_sector	= bucket + offset;
-		bio_set_dev(bio, ca->bdev);
 		bio->bi_iter.bi_size	= len << 9;
 		bio->bi_end_io	= journal_read_endio;
 		bio->bi_private = &cl;
-		bio_set_op_attrs(bio, REQ_OP_READ, 0);
 		bch_bio_map(bio, data);
 		closure_bio_submit(ca->set, bio, &cl);
 		atomic_set(&ja->discard_in_flight, DISCARD_IN_FLIGHT);
-		bio_init(bio, bio->bi_inline_vecs, 1);
+		bio_init(bio, ca->bdev, bio->bi_inline_vecs, 1, REQ_OP_DISCARD);
-		bio_set_op_attrs(bio, REQ_OP_DISCARD, 0);
 		bio->bi_iter.bi_sector	= bucket_to_sector(ca->set,
 						ca->sb.d[ja->discard_idx]);
-		bio_set_dev(bio, ca->bdev);
 		bio->bi_iter.bi_size	= bucket_bytes(ca);
 		bio->bi_end_io		= journal_discard_endio;
 		atomic_long_add(sectors, &ca->meta_sectors_written);
-		bio_reset(bio);
+		bio_reset(bio, ca->bdev, REQ_OP_WRITE |
+			  REQ_SYNC | REQ_META | REQ_PREFLUSH | REQ_FUA);
+		bch_bio_map(bio, w->data);
 		bio->bi_iter.bi_sector	= PTR_OFFSET(k, i);
-		bio_set_dev(bio, ca->bdev);
 		bio->bi_iter.bi_size = sectors << 9;
 		bio->bi_end_io	= journal_write_endio;
 		bio->bi_private = w;
-		bio_set_op_attrs(bio, REQ_OP_WRITE,
-				 REQ_SYNC|REQ_META|REQ_PREFLUSH|REQ_FUA);
-		bch_bio_map(bio, w->data);
 		trace_bcache_journal_write(bio, w->data->keys);
 		bio_list_add(&list, bio);

drivers/md/bcache/movinggc.c+2−2

@@ -79,8 +79,8 @@ static void moving_init(struct moving_io *io) · 78 unmodified lines
79	{	79	{
80	struct bio *bio = &io->bio.bio;	80	struct bio *bio = &io->bio.bio;
81		81
82	bio_init(bio, bio->bi_inline_vecs,	82	bio_init(bio, NULL, bio->bi_inline_vecs,
83	DIV_ROUND_UP(KEY_SIZE(&io->w->key), PAGE_SECTORS));	83	DIV_ROUND_UP(KEY_SIZE(&io->w->key), PAGE_SECTORS), 0);
84	bio_get(bio);	84	bio_get(bio);
85	bio_set_prio(bio, IOPRIO_PRIO_VALUE(IOPRIO_CLASS_IDLE, 0));	85	bio_set_prio(bio, IOPRIO_PRIO_VALUE(IOPRIO_CLASS_IDLE, 0));
86		86

drivers/md/bcache/request.c+10−12

 {
 	struct bio *bio = &s->bio.bio;
-	bio_init(bio, NULL, 0);
+	bio_init_clone(bio->bi_bdev, bio, orig_bio, GFP_NOIO);
-	__bio_clone_fast(bio, orig_bio);
 	/*
 	 * bi_end_io can be set separately somewhere else, e.g. the
 	 * variants in,
 	 */
 	if (s->iop.bio) {
-		bio_reset(s->iop.bio);
+		bio_reset(s->iop.bio, s->cache_miss->bi_bdev, REQ_OP_READ);
 		s->iop.bio->bi_iter.bi_sector =
 			s->cache_miss->bi_iter.bi_sector;
-		bio_copy_dev(s->iop.bio, s->cache_miss);
 		s->iop.bio->bi_iter.bi_size = s->insert_bio_sectors << 9;
+		bio_clone_blkg_association(s->iop.bio, s->cache_miss);
 		bch_bio_map(s->iop.bio, NULL);
 		bio_copy_data(s->cache_miss, s->iop.bio);
 	/* btree_search_recurse()'s btree iterator is no good anymore */
 	ret = miss == bio ? MAP_DONE : -EINTR;
-	cache_bio = bio_alloc_bioset(GFP_NOWAIT,
+	cache_bio = bio_alloc_bioset(miss->bi_bdev,
 			DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS),
-			&dc->disk.bio_split);
+			0, GFP_NOWAIT, &dc->disk.bio_split);
 	if (!cache_bio)
 		goto out_submit;
 	cache_bio->bi_iter.bi_sector	= miss->bi_iter.bi_sector;
-	bio_copy_dev(cache_bio, miss);
 	cache_bio->bi_iter.bi_size	= s->insert_bio_sectors << 9;
 	cache_bio->bi_end_io	= backing_request_endio;
 			 */
 			struct bio *flush;
-			flush = bio_alloc_bioset(GFP_NOIO, 0,
+			flush = bio_alloc_bioset(bio->bi_bdev, 0,
-						 &dc->disk.bio_split);
+						 REQ_OP_WRITE | REQ_PREFLUSH,
+						 GFP_NOIO, &dc->disk.bio_split);
 			if (!flush) {
 				s->iop.status = BLK_STS_RESOURCE;
 				goto insert_data;
 			}
-			bio_copy_dev(flush, bio);
 			flush->bi_end_io = backing_request_endio;
 			flush->bi_private = cl;
-			flush->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH;
 			/* I/O request sent to backing device */
 			closure_bio_submit(s->iop.c, flush, cl);
 		}
 	} else {
-		s->iop.bio = bio_clone_fast(bio, GFP_NOIO, &dc->disk.bio_split);
+		s->iop.bio = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOIO,
+					     &dc->disk.bio_split);
 		/* I/O request sent to backing device */
 		bio->bi_end_io = backing_request_endio;
 		closure_bio_submit(s->iop.c, bio, cl);

drivers/md/bcache/super.c+3−6

 #include <linux/blkdev.h>
 #include <linux/pagemap.h>
 #include <linux/debugfs.h>
-#include <linux/genhd.h>
 #include <linux/idr.h>
 #include <linux/kthread.h>
 #include <linux/workqueue.h>
 	down(&dc->sb_write_mutex);
 	closure_init(cl, parent);
-	bio_init(bio, dc->sb_bv, 1);
+	bio_init(bio, dc->bdev, dc->sb_bv, 1, 0);
-	bio_set_dev(bio, dc->bdev);
 	bio->bi_end_io	= write_bdev_super_endio;
 	bio->bi_private = dc;
 	if (ca->sb.version < version)
 		ca->sb.version = version;
-	bio_init(bio, ca->sb_bv, 1);
+	bio_init(bio, ca->bdev, ca->sb_bv, 1, 0);
-	bio_set_dev(bio, ca->bdev);
 	bio->bi_end_io	= write_super_endio;
 	bio->bi_private = ca;
 	__module_get(THIS_MODULE);
 	kobject_init(&ca->kobj, &bch_cache_ktype);
-	bio_init(&ca->journal.bio, ca->journal.bio.bi_inline_vecs, 8);
+	bio_init(&ca->journal.bio, NULL, ca->journal.bio.bi_inline_vecs, 8, 0);
 	/*
 	 * when ca->sb.njournal_buckets is not zero, journal exists,

drivers/md/bcache/writeback.c+2−2

@@ -292,8 +292,8 @@ static void dirty_init(struct keybuf_key *w) · 291 unmodified lines
292	struct dirty_io *io = w->private;	292	struct dirty_io *io = w->private;
293	struct bio *bio = &io->bio;	293	struct bio *bio = &io->bio;
294		294
295	bio_init(bio, bio->bi_inline_vecs,	295	bio_init(bio, NULL, bio->bi_inline_vecs,
296	DIV_ROUND_UP(KEY_SIZE(&w->key), PAGE_SECTORS));	296	DIV_ROUND_UP(KEY_SIZE(&w->key), PAGE_SECTORS), 0);
297	if (!io->dc->writeback_percent)	297	if (!io->dc->writeback_percent)
298	bio_set_prio(bio, IOPRIO_PRIO_VALUE(IOPRIO_CLASS_IDLE, 0));	298	bio_set_prio(bio, IOPRIO_PRIO_VALUE(IOPRIO_CLASS_IDLE, 0));
299		299

drivers/md/dm-cache-target.c+9−17

@@ -744,21 +744,14 @@ static void check_if_tick_bio_needed(struct cache cache, struct bio bio) · 743 unmodified lines
744	spin_unlock_irq(&cache->lock);	744	spin_unlock_irq(&cache->lock);
745	}	745	}
746		746
747	static void __remap_to_origin_clear_discard(struct cache cache, struct bio bio,
748	dm_oblock_t oblock, bool bio_has_pbd)
749	{
750	if (bio_has_pbd)
751	check_if_tick_bio_needed(cache, bio);
752	remap_to_origin(cache, bio);
753	if (bio_data_dir(bio) == WRITE)
754	clear_discard(cache, oblock_to_dblock(cache, oblock));
755	}
756
757	static void remap_to_origin_clear_discard(struct cache cache, struct bio bio,	747	static void remap_to_origin_clear_discard(struct cache cache, struct bio bio,
758	dm_oblock_t oblock)	748	dm_oblock_t oblock)
759	{	749	{
760	// FIXME: check_if_tick_bio_needed() is called way too much through this interface	750	// FIXME: check_if_tick_bio_needed() is called way too much through this interface
761	__remap_to_origin_clear_discard(cache, bio, oblock, true);	751	check_if_tick_bio_needed(cache, bio);
		752	remap_to_origin(cache, bio);
		753	if (bio_data_dir(bio) == WRITE)
		754	clear_discard(cache, oblock_to_dblock(cache, oblock));
762	}	755	}
763		756
764	static void remap_to_cache_dirty(struct cache cache, struct bio bio,	757	static void remap_to_cache_dirty(struct cache cache, struct bio bio,
@@ -826,16 +819,15 @@ static void issue_op(struct bio bio, void context) · 61 unmodified lines
826	static void remap_to_origin_and_cache(struct cache cache, struct bio bio,	819	static void remap_to_origin_and_cache(struct cache cache, struct bio bio,
827	dm_oblock_t oblock, dm_cblock_t cblock)	820	dm_oblock_t oblock, dm_cblock_t cblock)
828	{	821	{
829	struct bio *origin_bio = bio_clone_fast(bio, GFP_NOIO, &cache->bs);	822	struct bio *origin_bio = bio_alloc_clone(cache->origin_dev->bdev, bio,
		823	GFP_NOIO, &cache->bs);
830		824
831	BUG_ON(!origin_bio);	825	BUG_ON(!origin_bio);
832		826
833	bio_chain(origin_bio, bio);	827	bio_chain(origin_bio, bio);
834	/*	828
835	* Passing false to __remap_to_origin_clear_discard() skips	829	if (bio_data_dir(origin_bio) == WRITE)
836	* all code that might use per_bio_data (since clone doesn't have it)	830	clear_discard(cache, oblock_to_dblock(cache, oblock));
837	*/
838	__remap_to_origin_clear_discard(cache, origin_bio, oblock, false);
839	submit_bio(origin_bio);	831	submit_bio(origin_bio);
840		832
841	remap_to_cache(cache, bio, cblock);	833	remap_to_cache(cache, bio, cblock);

drivers/md/dm-core.h+0−1

@@ -11,7 +11,6 @@ · 10 unmodified lines
11		11
12	#include <linux/kthread.h>	12	#include <linux/kthread.h>
13	#include <linux/ktime.h>	13	#include <linux/ktime.h>
14	#include <linux/genhd.h>
15	#include <linux/blk-mq.h>	14	#include <linux/blk-mq.h>
16	#include <linux/blk-crypto-profile.h>	15	#include <linux/blk-crypto-profile.h>
17		16

drivers/md/dm-crypt.c+17−29

@@ -234,7 +234,7 @@ static volatile unsigned long dm_crypt_pages_per_client; · 233 unmodified lines
234	#define DM_CRYPT_MEMORY_PERCENT 2	234	#define DM_CRYPT_MEMORY_PERCENT 2
235	#define DM_CRYPT_MIN_PAGES_PER_CLIENT (BIO_MAX_VECS * 16)	235	#define DM_CRYPT_MIN_PAGES_PER_CLIENT (BIO_MAX_VECS * 16)
236		236
237	static void clone_init(struct dm_crypt_io , struct bio );	237	static void crypt_endio(struct bio *clone);
238	static void kcryptd_queue_crypt(struct dm_crypt_io *io);	238	static void kcryptd_queue_crypt(struct dm_crypt_io *io);
239	static struct scatterlist crypt_get_sg_data(struct crypt_config cc,	239	static struct scatterlist crypt_get_sg_data(struct crypt_config cc,
240	struct scatterlist *sg);	240	struct scatterlist *sg);
@@ -1364,11 +1364,10 @@ static int crypt_convert_block_aead(struct crypt_config *cc, · 1123 unmodified lines
1364	}	1364	}
1365		1365
1366	if (r == -EBADMSG) {	1366	if (r == -EBADMSG) {
1367	char b[BDEVNAME_SIZE];
1368	sector_t s = le64_to_cpu(*sector);	1367	sector_t s = le64_to_cpu(*sector);
1369		1368
1370	DMERR_LIMIT("%s: INTEGRITY AEAD ERROR, sector %llu",	1369	DMERR_LIMIT("%pg: INTEGRITY AEAD ERROR, sector %llu",
1371	bio_devname(ctx->bio_in, b), s);	1370	ctx->bio_in->bi_bdev, s);
1372	dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead",	1371	dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead",
1373	ctx->bio_in, s, 0);	1372	ctx->bio_in, s, 0);
1374	}	1373	}
@@ -1672,11 +1671,10 @@ retry: · 297 unmodified lines
1672	if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM))	1671	if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM))
1673	mutex_lock(&cc->bio_alloc_lock);	1672	mutex_lock(&cc->bio_alloc_lock);
1674		1673
1675	clone = bio_alloc_bioset(GFP_NOIO, nr_iovecs, &cc->bs);	1674	clone = bio_alloc_bioset(cc->dev->bdev, nr_iovecs, io->base_bio->bi_opf,
1676	if (!clone)	1675	GFP_NOIO, &cc->bs);
1677	goto out;	1676	clone->bi_private = io;
1678		1677	clone->bi_end_io = crypt_endio;
1679	clone_init(io, clone);
1680		1678
1681	remaining_size = size;	1679	remaining_size = size;
1682		1680
@@ -1702,7 +1700,7 @@ retry: · 19 unmodified lines
1702	bio_put(clone);	1700	bio_put(clone);
1703	clone = NULL;	1701	clone = NULL;
1704	}	1702	}
1705	out:	1703
1706	if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM))	1704	if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM))
1707	mutex_unlock(&cc->bio_alloc_lock);	1705	mutex_unlock(&cc->bio_alloc_lock);
1708		1706
@@ -1829,34 +1827,25 @@ static void crypt_endio(struct bio *clone) · 120 unmodified lines
1829	crypt_dec_pending(io);	1827	crypt_dec_pending(io);
1830	}	1828	}
1831		1829
1832	static void clone_init(struct dm_crypt_io io, struct bio clone)
1833	{
1834	struct crypt_config *cc = io->cc;
1835
1836	clone->bi_private = io;
1837	clone->bi_end_io = crypt_endio;
1838	bio_set_dev(clone, cc->dev->bdev);
1839	clone->bi_opf = io->base_bio->bi_opf;
1840	}
1841
1842	static int kcryptd_io_read(struct dm_crypt_io *io, gfp_t gfp)	1830	static int kcryptd_io_read(struct dm_crypt_io *io, gfp_t gfp)
1843	{	1831	{
1844	struct crypt_config *cc = io->cc;	1832	struct crypt_config *cc = io->cc;
1845	struct bio *clone;	1833	struct bio *clone;
1846		1834
1847	/*	1835	/*
1848	* We need the original biovec array in order to decrypt	1836	* We need the original biovec array in order to decrypt the whole bio
1849	* the whole bio data afterwards -- thanks to immutable	1837	* data afterwards -- thanks to immutable biovecs we don't need to
1850	* biovecs we don't need to worry about the block layer	1838	* worry about the block layer modifying the biovec array; so leverage
1851	* modifying the biovec array; so leverage bio_clone_fast().	1839	* bio_alloc_clone().
1852	*/	1840	*/
1853	clone = bio_clone_fast(io->base_bio, gfp, &cc->bs);	1841	clone = bio_alloc_clone(cc->dev->bdev, io->base_bio, gfp, &cc->bs);
1854	if (!clone)	1842	if (!clone)
1855	return 1;	1843	return 1;
		1844	clone->bi_private = io;
		1845	clone->bi_end_io = crypt_endio;
1856		1846
1857	crypt_inc_pending(io);	1847	crypt_inc_pending(io);
1858		1848
1859	clone_init(io, clone);
1860	clone->bi_iter.bi_sector = cc->start + io->sector;	1849	clone->bi_iter.bi_sector = cc->start + io->sector;
1861		1850
1862	if (dm_crypt_integrity_io_alloc(io, clone)) {	1851	if (dm_crypt_integrity_io_alloc(io, clone)) {
@@ -2179,11 +2168,10 @@ static void kcryptd_async_done(struct crypto_async_request *async_req, · 316 unmodified lines
2179	error = cc->iv_gen_ops->post(cc, org_iv_of_dmreq(cc, dmreq), dmreq);	2168	error = cc->iv_gen_ops->post(cc, org_iv_of_dmreq(cc, dmreq), dmreq);
2180		2169
2181	if (error == -EBADMSG) {	2170	if (error == -EBADMSG) {
2182	char b[BDEVNAME_SIZE];
2183	sector_t s = le64_to_cpu(*org_sector_of_dmreq(cc, dmreq));	2171	sector_t s = le64_to_cpu(*org_sector_of_dmreq(cc, dmreq));
2184		2172
2185	DMERR_LIMIT("%s: INTEGRITY AEAD ERROR, sector %llu",	2173	DMERR_LIMIT("%pg: INTEGRITY AEAD ERROR, sector %llu",
2186	bio_devname(ctx->bio_in, b), s);	2174	ctx->bio_in->bi_bdev, s);
2187	dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead",	2175	dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead",
2188	ctx->bio_in, s, 0);	2176	ctx->bio_in, s, 0);
2189	io->error = BLK_STS_PROTECTION;	2177	io->error = BLK_STS_PROTECTION;

drivers/md/dm-integrity.c+2−3

 						checksums_ptr - checksums, dio->op == REQ_OP_READ ? TAG_CMP : TAG_WRITE);
 			if (unlikely(r)) {
 				if (r > 0) {
-					char b[BDEVNAME_SIZE];
 					sector_t s;
 					s = sector - ((r + ic->tag_size - 1) / ic->tag_size);
-					DMERR_LIMIT("%s: Checksum failed at sector 0x%llx",
+					DMERR_LIMIT("%pg: Checksum failed at sector 0x%llx",
-						    bio_devname(bio, b), s);
+						    bio->bi_bdev, s);
 					r = -EILSEQ;
 					atomic64_inc(&ic->number_of_mismatches);
 					dm_audit_log_bio(DM_MSG_PREFIX, "integrity-checksum",

drivers/md/dm-io.c+2−3

 						(PAGE_SIZE >> SECTOR_SHIFT)));
 		}
-		bio = bio_alloc_bioset(GFP_NOIO, num_bvecs, &io->client->bios);
+		bio = bio_alloc_bioset(where->bdev, num_bvecs, op | op_flags,
+				       GFP_NOIO, &io->client->bios);
 		bio->bi_iter.bi_sector = where->sector + (where->count - remaining);
-		bio_set_dev(bio, where->bdev);
 		bio->bi_end_io = endio;
-		bio_set_op_attrs(bio, op, op_flags);
 		store_io_and_region_in_bio(bio, io, region);
 		if (op == REQ_OP_DISCARD || op == REQ_OP_WRITE_ZEROES) {

drivers/md/dm-log-writes.c+8−31

 	void *ptr;
 	size_t ret;
-	bio = bio_alloc(GFP_KERNEL, 1);
+	bio = bio_alloc(lc->logdev->bdev, 1, REQ_OP_WRITE, GFP_KERNEL);
-	if (!bio) {
-		DMERR("Couldn't alloc log bio");
-		goto error;
-	}
 	bio->bi_iter.bi_size = 0;
 	bio->bi_iter.bi_sector = sector;
-	bio_set_dev(bio, lc->logdev->bdev);
 	bio->bi_end_io = (sector == WRITE_LOG_SUPER_SECTOR) ?
 			  log_end_super : log_end_io;
 	bio->bi_private = lc;
-	bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
 	page = alloc_page(GFP_KERNEL);
 	if (!page) {
 		atomic_inc(&lc->io_blocks);
-		bio = bio_alloc(GFP_KERNEL, bio_pages);
+		bio = bio_alloc(lc->logdev->bdev, bio_pages, REQ_OP_WRITE,
-		if (!bio) {
+				GFP_KERNEL);
-			DMERR("Couldn't alloc inline data bio");
-			goto error;
-		}
 		bio->bi_iter.bi_size = 0;
 		bio->bi_iter.bi_sector = sector;
-		bio_set_dev(bio, lc->logdev->bdev);
 		bio->bi_end_io = log_end_io;
 		bio->bi_private = lc;
-		bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
 		for (i = 0; i < bio_pages; i++) {
 			pg_datalen = min_t(int, datalen, PAGE_SIZE);
 error_bio:
 	bio_free_pages(bio);
 	bio_put(bio);
-error:
 	put_io_block(lc);
 	return -1;
 }
 		goto out;
 	atomic_inc(&lc->io_blocks);
-	bio = bio_alloc(GFP_KERNEL, bio_max_segs(block->vec_cnt));
+	bio = bio_alloc(lc->logdev->bdev, bio_max_segs(block->vec_cnt),
-	if (!bio) {
+			REQ_OP_WRITE, GFP_KERNEL);
-		DMERR("Couldn't alloc log bio");
-		goto error;
-	}
 	bio->bi_iter.bi_size = 0;
 	bio->bi_iter.bi_sector = sector;
-	bio_set_dev(bio, lc->logdev->bdev);
 	bio->bi_end_io = log_end_io;
 	bio->bi_private = lc;
-	bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
 	for (i = 0; i < block->vec_cnt; i++) {
 		/*
 		if (ret != block->vecs[i].bv_len) {
 			atomic_inc(&lc->io_blocks);
 			submit_bio(bio);
-			bio = bio_alloc(GFP_KERNEL,
+			bio = bio_alloc(lc->logdev->bdev,
-					bio_max_segs(block->vec_cnt - i));
+					bio_max_segs(block->vec_cnt - i),
-			if (!bio) {
+					REQ_OP_WRITE, GFP_KERNEL);
-				DMERR("Couldn't alloc log bio");
-				goto error;
-			}
 			bio->bi_iter.bi_size = 0;
 			bio->bi_iter.bi_sector = sector;
-			bio_set_dev(bio, lc->logdev->bdev);
 			bio->bi_end_io = log_end_io;
 			bio->bi_private = lc;
-			bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
 			ret = bio_add_page(bio, block->vecs[i].bv_page,
 					   block->vecs[i].bv_len, 0);

drivers/md/dm-rq.c+9−17

 	dm_complete_request(tio->orig, error);
 }
-static blk_status_t dm_dispatch_clone_request(struct request *clone, struct request *rq)
-{
-	blk_status_t r;
-	if (blk_queue_io_stat(clone->q))
-		clone->rq_flags |= RQF_IO_STAT;
-	clone->start_time_ns = ktime_get_ns();
-	r = blk_insert_cloned_request(clone->q, clone);
-	if (r != BLK_STS_OK && r != BLK_STS_RESOURCE && r != BLK_STS_DEV_RESOURCE)
-		/* must complete clone in terms of original request */
-		dm_complete_request(rq, r);
-	return r;
-}
 static int dm_rq_bio_constructor(struct bio *bio, struct bio *bio_orig,
 				 void *data)
 {
 		/* The target has remapped the I/O so dispatch it */
 		trace_block_rq_remap(clone, disk_devt(dm_disk(md)),
 				     blk_rq_pos(rq));
-		ret = dm_dispatch_clone_request(clone, rq);
+		ret = blk_insert_cloned_request(clone);
-		if (ret == BLK_STS_RESOURCE || ret == BLK_STS_DEV_RESOURCE) {
+		switch (ret) {
+		case BLK_STS_OK:
+			break;
+		case BLK_STS_RESOURCE:
+		case BLK_STS_DEV_RESOURCE:
 			blk_rq_unprep_clone(clone);
 			blk_mq_cleanup_rq(clone);
 			tio->ti->type->release_clone_rq(clone, &tio->info);
 			tio->clone = NULL;
 			return DM_MAPIO_REQUEUE;
+		default:
+			/* must complete clone in terms of original request */
+			dm_complete_request(rq, ret);
 		}
 		break;
 	case DM_MAPIO_REQUEUE:

drivers/md/dm-snap.c+1−20

 	 * for them to be committed.
 	 */
 	struct bio_list bios_queued_during_merge;
-	/*
-	 * Flush data after merge.
-	 */
-	struct bio flush_bio;
 };
 /*
 static void error_bios(struct bio *bio);
-static int flush_data(struct dm_snapshot *s)
-{
-	struct bio *flush_bio = &s->flush_bio;
-	bio_reset(flush_bio);
-	bio_set_dev(flush_bio, s->origin->bdev);
-	flush_bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH;
-	return submit_bio_wait(flush_bio);
-}
 static void merge_callback(int read_err, unsigned long write_err, void *context)
 {
 	struct dm_snapshot *s = context;
 		goto shut;
 	}
-	if (flush_data(s) < 0) {
+	if (blkdev_issue_flush(s->origin->bdev) < 0) {
 		DMERR("Flush after merge failed: shutting down merge");
 		goto shut;
 	}
 	s->first_merging_chunk = 0;
 	s->num_merging_chunks = 0;
 	bio_list_init(&s->bios_queued_during_merge);
-	bio_init(&s->flush_bio, NULL, 0);
 	/* Allocate hash table for COW data */
 	if (init_hash_tables(s)) {
 	dm_exception_store_destroy(s->store);
-	bio_uninit(&s->flush_bio);
 	dm_put_device(ti, s->cow);
 	dm_put_device(ti, s->origin);

drivers/md/dm-thin.c+12−29

@@ -282,8 +282,6 @@ struct pool { · 281 unmodified lines
282	struct dm_bio_prison_cell **cell_sort_array;	282	struct dm_bio_prison_cell **cell_sort_array;
283		283
284	mempool_t mapping_pool;	284	mempool_t mapping_pool;
285
286	struct bio flush_bio;
287	};	285	};
288		286
289	static void metadata_operation_failed(struct pool pool, const char op, int r);	287	static void metadata_operation_failed(struct pool pool, const char op, int r);
@@ -1179,25 +1177,17 @@ static void process_prepared_discard_passdown_pt1(struct dm_thin_new_mapping *m) · 889 unmodified lines
1179	return;	1177	return;
1180	}	1178	}
1181		1179
1182	discard_parent = bio_alloc(GFP_NOIO, 1);	1180	discard_parent = bio_alloc(NULL, 1, 0, GFP_NOIO);
1183	if (!discard_parent) {	1181	discard_parent->bi_end_io = passdown_endio;
1184	DMWARN("%s: unable to allocate top level discard bio for passdown. Skipping passdown.",	1182	discard_parent->bi_private = m;
1185	dm_device_name(tc->pool->pool_md));	1183	if (m->maybe_shared)
1186	queue_passdown_pt2(m);	1184	passdown_double_checking_shared_status(m, discard_parent);
1187		1185	else {
1188	} else {	1186	struct discard_op op;
1189	discard_parent->bi_end_io = passdown_endio;	1187
1190	discard_parent->bi_private = m;	1188	begin_discard(&op, tc, discard_parent);
1191		1189	r = issue_discard(&op, m->data_block, data_end);
1192	if (m->maybe_shared)	1190	end_discard(&op, r);
1193	passdown_double_checking_shared_status(m, discard_parent);
1194	else {
1195	struct discard_op op;
1196
1197	begin_discard(&op, tc, discard_parent);
1198	r = issue_discard(&op, m->data_block, data_end);
1199	end_discard(&op, r);
1200	}
1201	}	1191	}
1202	}	1192	}
1203		1193
@@ -2913,7 +2903,6 @@ static void __pool_destroy(struct pool *pool) · 1709 unmodified lines
2913	if (pool->next_mapping)	2903	if (pool->next_mapping)
2914	mempool_free(pool->next_mapping, &pool->mapping_pool);	2904	mempool_free(pool->next_mapping, &pool->mapping_pool);
2915	mempool_exit(&pool->mapping_pool);	2905	mempool_exit(&pool->mapping_pool);
2916	bio_uninit(&pool->flush_bio);
2917	dm_deferred_set_destroy(pool->shared_read_ds);	2906	dm_deferred_set_destroy(pool->shared_read_ds);
2918	dm_deferred_set_destroy(pool->all_io_ds);	2907	dm_deferred_set_destroy(pool->all_io_ds);
2919	kfree(pool);	2908	kfree(pool);
@@ -2994,7 +2983,6 @@ static struct pool pool_create(struct mapped_device pool_md, · 74 unmodified lines
2994	pool->low_water_triggered = false;	2983	pool->low_water_triggered = false;
2995	pool->suspended = true;	2984	pool->suspended = true;
2996	pool->out_of_data_space = false;	2985	pool->out_of_data_space = false;
2997	bio_init(&pool->flush_bio, NULL, 0);
2998		2986
2999	pool->shared_read_ds = dm_deferred_set_create();	2987	pool->shared_read_ds = dm_deferred_set_create();
3000	if (!pool->shared_read_ds) {	2988	if (!pool->shared_read_ds) {
@@ -3201,13 +3189,8 @@ static void metadata_low_callback(void *context) · 200 unmodified lines
3201	static int metadata_pre_commit_callback(void *context)	3189	static int metadata_pre_commit_callback(void *context)
3202	{	3190	{
3203	struct pool *pool = context;	3191	struct pool *pool = context;
3204	struct bio *flush_bio = &pool->flush_bio;
3205
3206	bio_reset(flush_bio);
3207	bio_set_dev(flush_bio, pool->data_dev);
3208	flush_bio->bi_opf = REQ_OP_WRITE \| REQ_PREFLUSH;
3209		3192
3210	return submit_bio_wait(flush_bio);	3193	return blkdev_issue_flush(pool->data_dev);
3211	}	3194	}
3212		3195
3213	static sector_t get_dev_size(struct block_device *bdev)	3196	static sector_t get_dev_size(struct block_device *bdev)

drivers/md/dm-writecache.c+4−3

 		max_pages = e->wc_list_contiguous;
-		bio = bio_alloc_bioset(GFP_NOIO, max_pages, &wc->bio_set);
+		bio = bio_alloc_bioset(wc->dev->bdev, max_pages, REQ_OP_WRITE,
+				       GFP_NOIO, &wc->bio_set);
 		wb = container_of(bio, struct writeback_struct, bio);
 		wb->wc = wc;
 		bio->bi_end_io = writecache_writeback_endio;
-		bio_set_dev(bio, wc->dev->bdev);
 		bio->bi_iter.bi_sector = read_original_sector(wc, e);
 		if (max_pages <= WB_LIST_INLINE ||
 		    unlikely(!(wb->wc_list = kmalloc_array(max_pages, sizeof(struct wc_entry *),
 			wb->wc_list[wb->wc_list_n++] = f;
 			e = f;
 		}
-		bio_set_op_attrs(bio, REQ_OP_WRITE, WC_MODE_FUA(wc) * REQ_FUA);
+		if (WC_MODE_FUA(wc))
+			bio->bi_opf |= REQ_FUA;
 		if (writecache_has_error(wc)) {
 			bio->bi_status = BLK_STS_IOERR;
 			bio_endio(bio);

drivers/md/dm-zoned-metadata.c+6−20

 	if (!mblk)
 		return ERR_PTR(-ENOMEM);
-	bio = bio_alloc(GFP_NOIO, 1);
+	bio = bio_alloc(dev->bdev, 1, REQ_OP_READ | REQ_META | REQ_PRIO,
-	if (!bio) {
+			GFP_NOIO);
-		dmz_free_mblock(zmd, mblk);
-		return ERR_PTR(-ENOMEM);
-	}
 	spin_lock(&zmd->mblk_lock);
 	/* Submit read BIO */
 	bio->bi_iter.bi_sector = dmz_blk2sect(block);
-	bio_set_dev(bio, dev->bdev);
 	bio->bi_private = mblk;
 	bio->bi_end_io = dmz_mblock_bio_end_io;
-	bio_set_op_attrs(bio, REQ_OP_READ, REQ_META | REQ_PRIO);
 	bio_add_page(bio, mblk->page, DMZ_BLOCK_SIZE, 0);
 	submit_bio(bio);
 	if (dmz_bdev_is_dying(dev))
 		return -EIO;
-	bio = bio_alloc(GFP_NOIO, 1);
+	bio = bio_alloc(dev->bdev, 1, REQ_OP_WRITE | REQ_META | REQ_PRIO,
-	if (!bio) {
+			GFP_NOIO);
-		set_bit(DMZ_META_ERROR, &mblk->state);
-		return -ENOMEM;
-	}
 	set_bit(DMZ_META_WRITING, &mblk->state);
 	bio->bi_iter.bi_sector = dmz_blk2sect(block);
-	bio_set_dev(bio, dev->bdev);
 	bio->bi_private = mblk;
 	bio->bi_end_io = dmz_mblock_bio_end_io;
-	bio_set_op_attrs(bio, REQ_OP_WRITE, REQ_META | REQ_PRIO);
 	bio_add_page(bio, mblk->page, DMZ_BLOCK_SIZE, 0);
 	submit_bio(bio);
 	if (dmz_bdev_is_dying(dev))
 		return -EIO;
-	bio = bio_alloc(GFP_NOIO, 1);
+	bio = bio_alloc(dev->bdev, 1, op | REQ_SYNC | REQ_META | REQ_PRIO,
-	if (!bio)
+			GFP_NOIO);
-		return -ENOMEM;
 	bio->bi_iter.bi_sector = dmz_blk2sect(block);
-	bio_set_dev(bio, dev->bdev);
-	bio_set_op_attrs(bio, op, REQ_SYNC | REQ_META | REQ_PRIO);
 	bio_add_page(bio, page, DMZ_BLOCK_SIZE, 0);
 	ret = submit_bio_wait(bio);
 	bio_put(bio);

drivers/md/dm-zoned-target.c+1−2

 	if (dev->flags & DMZ_BDEV_DYING)
 		return -EIO;
-	clone = bio_clone_fast(bio, GFP_NOIO, &dmz->bio_set);
+	clone = bio_alloc_clone(dev->bdev, bio, GFP_NOIO, &dmz->bio_set);
 	if (!clone)
 		return -ENOMEM;
-	bio_set_dev(clone, dev->bdev);
 	bioctx->dev = dev;
 	clone->bi_iter.bi_sector =
 		dmz_start_sect(dmz->metadata, zone) + dmz_blk2sect(chunk_block);

drivers/md/dm.c+61−111

 #define DM_IO_BIO_OFFSET \
 	(offsetof(struct dm_target_io, clone) + offsetof(struct dm_io, tio))
+static inline struct dm_target_io *clone_to_tio(struct bio *clone)
+{
+	return container_of(clone, struct dm_target_io, clone);
+}
 void *dm_per_bio_data(struct bio *bio, size_t data_size)
 {
-	struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone);
+	if (!clone_to_tio(bio)->inside_dm_io)
-	if (!tio->inside_dm_io)
 		return (char *)bio - DM_TARGET_IO_BIO_OFFSET - data_size;
 	return (char *)bio - DM_IO_BIO_OFFSET - data_size;
 }
 u64 dm_start_time_ns_from_clone(struct bio *bio)
 {
-	struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone);
+	return jiffies_to_nsecs(clone_to_tio(bio)->io->start_time);
-	struct dm_io *io = tio->io;
-	return jiffies_to_nsecs(io->start_time);
 }
 EXPORT_SYMBOL_GPL(dm_start_time_ns_from_clone);
 	struct dm_target_io *tio;
 	struct bio *clone;
-	clone = bio_alloc_bioset(GFP_NOIO, 0, &md->io_bs);
+	clone = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOIO, &md->io_bs);
-	if (!clone)
-		return NULL;
-	tio = container_of(clone, struct dm_target_io, clone);
+	tio = clone_to_tio(clone);
 	tio->inside_dm_io = true;
 	tio->io = NULL;
 	bio_put(&io->tio.clone);
 }
-static struct dm_target_io *alloc_tio(struct clone_info *ci, struct dm_target *ti,
+static struct bio *alloc_tio(struct clone_info *ci, struct dm_target *ti,
-				      unsigned target_bio_nr, gfp_t gfp_mask)
+		unsigned target_bio_nr, unsigned *len, gfp_t gfp_mask)
 {
 	struct dm_target_io *tio;
 		/* the dm_target_io embedded in ci->io is available */
 		tio = &ci->io->tio;
 	} else {
-		struct bio *clone = bio_alloc_bioset(gfp_mask, 0, &ci->io->md->bs);
+		struct bio *clone = bio_alloc_clone(ci->bio->bi_bdev, ci->bio,
+						    gfp_mask, &ci->io->md->bs);
 		if (!clone)
 			return NULL;
-		tio = container_of(clone, struct dm_target_io, clone);
+		tio = clone_to_tio(clone);
 		tio->inside_dm_io = false;
 	}
 	tio->io = ci->io;
 	tio->ti = ti;
 	tio->target_bio_nr = target_bio_nr;
+	tio->len_ptr = len;
-	return tio;
+	return &tio->clone;
 }
-static void free_tio(struct dm_target_io *tio)
+static void free_tio(struct bio *clone)
 {
-	if (tio->inside_dm_io)
+	if (clone_to_tio(clone)->inside_dm_io)
 		return;
-	bio_put(&tio->clone);
+	bio_put(clone);
 }
 /*
 static void clone_endio(struct bio *bio)
 {
 	blk_status_t error = bio->bi_status;
-	struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone);
+	struct dm_target_io *tio = clone_to_tio(bio);
 	struct dm_io *io = tio->io;
 	struct mapped_device *md = tio->io->md;
 	dm_endio_fn endio = tio->ti->type->end_io;
 		up(&md->swap_bios_semaphore);
 	}
-	free_tio(tio);
+	free_tio(bio);
 	dm_io_dec_pending(io, error);
 }
  */
 void dm_accept_partial_bio(struct bio *bio, unsigned n_sectors)
 {
-	struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone);
+	struct dm_target_io *tio = clone_to_tio(bio);
 	unsigned bi_size = bio->bi_iter.bi_size >> SECTOR_SHIFT;
 	BUG_ON(bio->bi_opf & REQ_PREFLUSH);
 	mutex_unlock(&md->swap_bios_lock);
 }
-static void __map_bio(struct dm_target_io *tio)
+static void __map_bio(struct bio *clone)
 {
+	struct dm_target_io *tio = clone_to_tio(clone);
 	int r;
 	sector_t sector;
-	struct bio *clone = &tio->clone;
 	struct dm_io *io = tio->io;
 	struct dm_target *ti = tio->ti;
 			struct mapped_device *md = io->md;
 			up(&md->swap_bios_semaphore);
 		}
-		free_tio(tio);
+		free_tio(clone);
 		dm_io_dec_pending(io, BLK_STS_IOERR);
 		break;
 	case DM_MAPIO_REQUEUE:
 			struct mapped_device *md = io->md;
 			up(&md->swap_bios_semaphore);
 		}
-		free_tio(tio);
+		free_tio(clone);
 		dm_io_dec_pending(io, BLK_STS_DM_REQUEUE);
 		break;
 	default:
 /*
  * Creates a bio that consists of range of complete bvecs.
  */
-static int clone_bio(struct dm_target_io *tio, struct bio *bio,
+static int __clone_and_map_data_bio(struct clone_info *ci, struct dm_target *ti,
-		     sector_t sector, unsigned len)
+				    sector_t sector, unsigned *len)
 {
-	struct bio *clone = &tio->clone;
+	struct bio *bio = ci->bio, *clone;
-	int r;
-	__bio_clone_fast(clone, bio);
-	r = bio_crypt_clone(clone, bio, GFP_NOIO);
-	if (r < 0)
-		return r;
-	if (bio_integrity(bio)) {
-		if (unlikely(!dm_target_has_integrity(tio->ti->type) &&
-			     !dm_target_passes_integrity(tio->ti->type))) {
-			DMWARN("%s: the target %s doesn't support integrity data.",
-				dm_device_name(tio->io->md),
-				tio->ti->type->name);
-			return -EIO;
-		}
-		r = bio_integrity_clone(clone, bio, GFP_NOIO);
-		if (r < 0)
-			return r;
-	}
+	clone = alloc_tio(ci, ti, 0, len, GFP_NOIO);
 	bio_advance(clone, to_bytes(sector - clone->bi_iter.bi_sector));
-	clone->bi_iter.bi_size = to_bytes(len);
+	clone->bi_iter.bi_size = to_bytes(*len);
 	if (bio_integrity(bio))
 		bio_integrity_trim(clone);
+	__map_bio(clone);
 	return 0;
 }
 static void alloc_multiple_bios(struct bio_list *blist, struct clone_info *ci,
-				struct dm_target *ti, unsigned num_bios)
+				struct dm_target *ti, unsigned num_bios,
+				unsigned *len)
 {
-	struct dm_target_io *tio;
+	struct bio *bio;
 	int try;
-	if (!num_bios)
-		return;
-	if (num_bios == 1) {
-		tio = alloc_tio(ci, ti, 0, GFP_NOIO);
-		bio_list_add(blist, &tio->clone);
-		return;
-	}
 	for (try = 0; try < 2; try++) {
 		int bio_nr;
-		struct bio *bio;
 		if (try)
 			mutex_lock(&ci->io->md->table_devices_lock);
 		for (bio_nr = 0; bio_nr < num_bios; bio_nr++) {
-			tio = alloc_tio(ci, ti, bio_nr, try ? GFP_NOIO : GFP_NOWAIT);
+			bio = alloc_tio(ci, ti, bio_nr, len,
-			if (!tio)
+					try ? GFP_NOIO : GFP_NOWAIT);
+			if (!bio)
 				break;
-			bio_list_add(blist, &tio->clone);
+			bio_list_add(blist, bio);
 		}
 		if (try)
 			mutex_unlock(&ci->io->md->table_devices_lock);
 		if (bio_nr == num_bios)
 			return;
-		while ((bio = bio_list_pop(blist))) {
+		while ((bio = bio_list_pop(blist)))
-			tio = container_of(bio, struct dm_target_io, clone);
+			free_tio(bio);
-			free_tio(tio);
-		}
 	}
 }
-static void __clone_and_map_simple_bio(struct clone_info *ci,
-					   struct dm_target_io *tio, unsigned *len)
-{
-	struct bio *clone = &tio->clone;
-	tio->len_ptr = len;
-	__bio_clone_fast(clone, ci->bio);
-	if (len)
-		bio_setup_sector(clone, ci->sector, *len);
-	__map_bio(tio);
-}
 static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti,
 				  unsigned num_bios, unsigned *len)
 {
 	struct bio_list blist = BIO_EMPTY_LIST;
-	struct bio *bio;
+	struct bio *clone;
-	struct dm_target_io *tio;
-	alloc_multiple_bios(&blist, ci, ti, num_bios);
-	while ((bio = bio_list_pop(&blist))) {
+	switch (num_bios) {
-		tio = container_of(bio, struct dm_target_io, clone);
+	case 0:
-		__clone_and_map_simple_bio(ci, tio, len);
+		break;
+	case 1:
+		clone = alloc_tio(ci, ti, 0, len, GFP_NOIO);
+		if (len)
+			bio_setup_sector(clone, ci->sector, *len);
+		__map_bio(clone);
+		break;
+	default:
+		alloc_multiple_bios(&blist, ci, ti, num_bios, len);
+		while ((clone = bio_list_pop(&blist))) {
+			if (len)
+				bio_setup_sector(clone, ci->sector, *len);
+			__map_bio(clone);
+		}
+		break;
 	}
 }
 	 * need to reference it after submit. It's just used as
 	 * the basis for the clone(s).
 	 */
-	bio_init(&flush_bio, NULL, 0);
+	bio_init(&flush_bio, ci->io->md->disk->part0, NULL, 0,
-	flush_bio.bi_opf = REQ_OP_WRITE | REQ_PREFLUSH | REQ_SYNC;
+		 REQ_OP_WRITE | REQ_PREFLUSH | REQ_SYNC);
-	bio_set_dev(&flush_bio, ci->io->md->disk->part0);
 	ci->bio = &flush_bio;
 	ci->sector_count = 0;
 	return 0;
 }
-static int __clone_and_map_data_bio(struct clone_info *ci, struct dm_target *ti,
-				    sector_t sector, unsigned *len)
-{
-	struct bio *bio = ci->bio;
-	struct dm_target_io *tio;
-	int r;
-	tio = alloc_tio(ci, ti, 0, GFP_NOIO);
-	tio->len_ptr = len;
-	r = clone_bio(tio, bio, sector, *len);
-	if (r < 0) {
-		free_tio(tio);
-		return r;
-	}
-	__map_bio(tio);
-	return 0;
-}
 static int __send_changing_extent_only(struct clone_info *ci, struct dm_target *ti,
 				       unsigned num_bios)
 {

drivers/md/md-faulty.c+2−2

 		}
 	}
 	if (failit) {
-		struct bio *b = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set);
+		struct bio *b = bio_alloc_clone(conf->rdev->bdev, bio, GFP_NOIO,
+						&mddev->bio_set);
-		bio_set_dev(b, conf->rdev->bdev);
 		b->bi_private = bio;
 		b->bi_end_io = faulty_fail;
 		bio = b;

drivers/md/md-multipath.c+5−8

 	}
 	multipath = conf->multipaths + mp_bh->path;
-	bio_init(&mp_bh->bio, NULL, 0);
+	bio_init_clone(multipath->rdev->bdev, &mp_bh->bio, bio, GFP_NOIO);
-	__bio_clone_fast(&mp_bh->bio, bio);
 	mp_bh->bio.bi_iter.bi_sector += multipath->rdev->data_offset;
-	bio_set_dev(&mp_bh->bio, multipath->rdev->bdev);
 	mp_bh->bio.bi_opf |= REQ_FAILFAST_TRANSPORT;
 	mp_bh->bio.bi_end_io = multipath_end_request;
 	mp_bh->bio.bi_private = mp_bh;
 	md_check_recovery(mddev);
 	for (;;) {
-		char b[BDEVNAME_SIZE];
 		spin_lock_irqsave(&conf->device_lock, flags);
 		if (list_empty(head))
 			break;
 		bio->bi_iter.bi_sector = mp_bh->master_bio->bi_iter.bi_sector;
 		if ((mp_bh->path = multipath_map (conf))<0) {
-			pr_err("multipath: %s: unrecoverable IO read error for block %llu\n",
+			pr_err("multipath: %pg: unrecoverable IO read error for block %llu\n",
-			       bio_devname(bio, b),
+			       bio->bi_bdev,
 			       (unsigned long long)bio->bi_iter.bi_sector);
 			multipath_end_bh_io(mp_bh, BLK_STS_IOERR);
 		} else {
-			pr_err("multipath: %s: redirecting sector %llu to another IO path\n",
+			pr_err("multipath: %pg: redirecting sector %llu to another IO path\n",
-			       bio_devname(bio, b),
+			       bio->bi_bdev,
 			       (unsigned long long)bio->bi_iter.bi_sector);
 			*bio = *(mp_bh->master_bio);
 			bio->bi_iter.bi_sector +=

drivers/md/md.c+14−15

@@ -562,11 +562,11 @@ static void submit_flushes(struct work_struct *ws) · 561 unmodified lines
562	atomic_inc(&rdev->nr_pending);	562	atomic_inc(&rdev->nr_pending);
563	atomic_inc(&rdev->nr_pending);	563	atomic_inc(&rdev->nr_pending);
564	rcu_read_unlock();	564	rcu_read_unlock();
565	bi = bio_alloc_bioset(GFP_NOIO, 0, &mddev->bio_set);	565	bi = bio_alloc_bioset(rdev->bdev, 0,
		566	REQ_OP_WRITE \| REQ_PREFLUSH,
		567	GFP_NOIO, &mddev->bio_set);
566	bi->bi_end_io = md_end_flush;	568	bi->bi_end_io = md_end_flush;
567	bi->bi_private = rdev;	569	bi->bi_private = rdev;
568	bio_set_dev(bi, rdev->bdev);
569	bi->bi_opf = REQ_OP_WRITE \| REQ_PREFLUSH;
570	atomic_inc(&mddev->flush_pending);	570	atomic_inc(&mddev->flush_pending);
571	submit_bio(bi);	571	submit_bio(bi);
572	rcu_read_lock();	572	rcu_read_lock();
@@ -955,7 +955,6 @@ void md_super_write(struct mddev mddev, struct md_rdev rdev, · 382 unmodified lines
955	* If an error occurred, call md_error	955	* If an error occurred, call md_error
956	*/	956	*/
957	struct bio *bio;	957	struct bio *bio;
958	int ff = 0;
959		958
960	if (!page)	959	if (!page)
961	return;	960	return;
@@ -963,11 +962,13 @@ void md_super_write(struct mddev mddev, struct md_rdev rdev, · 1 unmodified line
963	if (test_bit(Faulty, &rdev->flags))	962	if (test_bit(Faulty, &rdev->flags))
964	return;	963	return;
965		964
966	bio = bio_alloc_bioset(GFP_NOIO, 1, &mddev->sync_set);	965	bio = bio_alloc_bioset(rdev->meta_bdev ? rdev->meta_bdev : rdev->bdev,
		966	1,
		967	REQ_OP_WRITE \| REQ_SYNC \| REQ_PREFLUSH \| REQ_FUA,
		968	GFP_NOIO, &mddev->sync_set);
967		969
968	atomic_inc(&rdev->nr_pending);	970	atomic_inc(&rdev->nr_pending);
969		971
970	bio_set_dev(bio, rdev->meta_bdev ? rdev->meta_bdev : rdev->bdev);
971	bio->bi_iter.bi_sector = sector;	972	bio->bi_iter.bi_sector = sector;
972	bio_add_page(bio, page, size, 0);	973	bio_add_page(bio, page, size, 0);
973	bio->bi_private = rdev;	974	bio->bi_private = rdev;
@@ -976,8 +977,7 @@ void md_super_write(struct mddev mddev, struct md_rdev rdev, · 2 unmodified lines
976	if (test_bit(MD_FAILFAST_SUPPORTED, &mddev->flags) &&	977	if (test_bit(MD_FAILFAST_SUPPORTED, &mddev->flags) &&
977	test_bit(FailFast, &rdev->flags) &&	978	test_bit(FailFast, &rdev->flags) &&
978	!test_bit(LastDev, &rdev->flags))	979	!test_bit(LastDev, &rdev->flags))
979	ff = MD_FAILFAST;	980	bio->bi_opf \|= MD_FAILFAST;
980	bio->bi_opf = REQ_OP_WRITE \| REQ_SYNC \| REQ_PREFLUSH \| REQ_FUA \| ff;
981		981
982	atomic_inc(&mddev->pending_writes);	982	atomic_inc(&mddev->pending_writes);
983	submit_bio(bio);	983	submit_bio(bio);
@@ -998,13 +998,11 @@ int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, · 14 unmodified lines
998	struct bio bio;	998	struct bio bio;
999	struct bio_vec bvec;	999	struct bio_vec bvec;
1000		1000
1001	bio_init(&bio, &bvec, 1);
1002
1003	if (metadata_op && rdev->meta_bdev)	1001	if (metadata_op && rdev->meta_bdev)
1004	bio_set_dev(&bio, rdev->meta_bdev);	1002	bio_init(&bio, rdev->meta_bdev, &bvec, 1, op \| op_flags);
1005	else	1003	else
1006	bio_set_dev(&bio, rdev->bdev);	1004	bio_init(&bio, rdev->bdev, &bvec, 1, op \| op_flags);
1007	bio.bi_opf = op \| op_flags;	1005
1008	if (metadata_op)	1006	if (metadata_op)
1009	bio.bi_iter.bi_sector = sector + rdev->sb_start;	1007	bio.bi_iter.bi_sector = sector + rdev->sb_start;
1010	else if (rdev->mddev->reshape_position != MaxSector &&	1008	else if (rdev->mddev->reshape_position != MaxSector &&
@@ -8636,13 +8634,14 @@ static void md_end_io_acct(struct bio *bio) · 7625 unmodified lines
8636	*/	8634	*/
8637	void md_account_bio(struct mddev mddev, struct bio *bio)	8635	void md_account_bio(struct mddev mddev, struct bio *bio)
8638	{	8636	{
		8637	struct block_device bdev = (bio)->bi_bdev;
8639	struct md_io_acct *md_io_acct;	8638	struct md_io_acct *md_io_acct;
8640	struct bio *clone;	8639	struct bio *clone;
8641		8640
8642	if (!blk_queue_io_stat((*bio)->bi_bdev->bd_disk->queue))	8641	if (!blk_queue_io_stat(bdev->bd_disk->queue))
8643	return;	8642	return;
8644		8643
8645	clone = bio_clone_fast(*bio, GFP_NOIO, &mddev->io_acct_set);	8644	clone = bio_alloc_clone(bdev, *bio, GFP_NOIO, &mddev->io_acct_set);
8646	md_io_acct = container_of(clone, struct md_io_acct, bio_clone);	8645	md_io_acct = container_of(clone, struct md_io_acct, bio_clone);
8647	md_io_acct->orig_bio = *bio;	8646	md_io_acct->orig_bio = *bio;
8648	md_io_acct->start_time = bio_start_io_acct(*bio);	8647	md_io_acct->start_time = bio_start_io_acct(*bio);

drivers/md/raid1.c+23−24

 	int i = 0;
 	struct bio *behind_bio = NULL;
-	behind_bio = bio_alloc_bioset(GFP_NOIO, vcnt, &r1_bio->mddev->bio_set);
+	behind_bio = bio_alloc_bioset(NULL, vcnt, 0, GFP_NOIO,
+				      &r1_bio->mddev->bio_set);
 	if (!behind_bio)
 		return;
 	if (!r1bio_existed && blk_queue_io_stat(bio->bi_bdev->bd_disk->queue))
 		r1_bio->start_time = bio_start_io_acct(bio);
-	read_bio = bio_clone_fast(bio, gfp, &mddev->bio_set);
+	read_bio = bio_alloc_clone(mirror->rdev->bdev, bio, gfp,
+				   &mddev->bio_set);
 	r1_bio->bios[rdisk] = read_bio;
 	read_bio->bi_iter.bi_sector = r1_bio->sector +
 		mirror->rdev->data_offset;
-	bio_set_dev(read_bio, mirror->rdev->bdev);
 	read_bio->bi_end_io = raid1_end_read_request;
 	bio_set_op_attrs(read_bio, op, do_sync);
 	if (test_bit(FailFast, &mirror->rdev->flags) &&
 			first_clone = 0;
 		}
-		if (r1_bio->behind_master_bio)
-			mbio = bio_clone_fast(r1_bio->behind_master_bio,
-					      GFP_NOIO, &mddev->bio_set);
-		else
-			mbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set);
 		if (r1_bio->behind_master_bio) {
+			mbio = bio_alloc_clone(rdev->bdev,
+					       r1_bio->behind_master_bio,
+					       GFP_NOIO, &mddev->bio_set);
 			if (test_bit(CollisionCheck, &rdev->flags))
 				wait_for_serialization(rdev, r1_bio);
 			if (test_bit(WriteMostly, &rdev->flags))
 				atomic_inc(&r1_bio->behind_remaining);
-		} else if (mddev->serialize_policy)
+		} else {
-			wait_for_serialization(rdev, r1_bio);
+			mbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO,
+					       &mddev->bio_set);
+			if (mddev->serialize_policy)
+				wait_for_serialization(rdev, r1_bio);
+		}
 		r1_bio->bios[i] = mbio;
 		mbio->bi_iter.bi_sector	= (r1_bio->sector + rdev->data_offset);
-		bio_set_dev(mbio, rdev->bdev);
 		mbio->bi_end_io	= raid1_end_write_request;
 		mbio->bi_opf = bio_op(bio) | (bio->bi_opf & (REQ_SYNC | REQ_FUA));
 		if (test_bit(FailFast, &rdev->flags) &&
 		} while (!success && d != r1_bio->read_disk);
 		if (!success) {
-			char b[BDEVNAME_SIZE];
 			int abort = 0;
 			/* Cannot read from anywhere, this block is lost.
 			 * Record a bad block on each device.  If that doesn't
 			 * work just disable and interrupt the recovery.
 			 * Don't fail devices as that won't really help.
 			 */
-			pr_crit_ratelimited("md/raid1:%s: %s: unrecoverable I/O read error for block %llu\n",
+			pr_crit_ratelimited("md/raid1:%s: %pg: unrecoverable I/O read error for block %llu\n",
-					    mdname(mddev), bio_devname(bio, b),
+					    mdname(mddev), bio->bi_bdev,
 					    (unsigned long long)r1_bio->sector);
 			for (d = 0; d < conf->raid_disks * 2; d++) {
 				rdev = conf->mirrors[d].rdev;
 			continue;
 		/* fixup the bio for reuse, but preserve errno */
 		status = b->bi_status;
-		bio_reset(b);
+		bio_reset(b, conf->mirrors[i].rdev->bdev, REQ_OP_READ);
 		b->bi_status = status;
 		b->bi_iter.bi_sector = r1_bio->sector +
 			conf->mirrors[i].rdev->data_offset;
-		bio_set_dev(b, conf->mirrors[i].rdev->bdev);
 		b->bi_end_io = end_sync_read;
 		rp->raid_bio = r1_bio;
 		b->bi_private = rp;
 		/* Write at 'sector' for 'sectors'*/
 		if (test_bit(R1BIO_BehindIO, &r1_bio->state)) {
-			wbio = bio_clone_fast(r1_bio->behind_master_bio,
+			wbio = bio_alloc_clone(rdev->bdev,
-					      GFP_NOIO,
+					       r1_bio->behind_master_bio,
-					      &mddev->bio_set);
+					       GFP_NOIO, &mddev->bio_set);
 		} else {
-			wbio = bio_clone_fast(r1_bio->master_bio, GFP_NOIO,
+			wbio = bio_alloc_clone(rdev->bdev, r1_bio->master_bio,
-					      &mddev->bio_set);
+					       GFP_NOIO, &mddev->bio_set);
 		}
 		bio_set_op_attrs(wbio, REQ_OP_WRITE, 0);
 		bio_trim(wbio, sector - r1_bio->sector, sectors);
 		wbio->bi_iter.bi_sector += rdev->data_offset;
-		bio_set_dev(wbio, rdev->bdev);
 		if (submit_bio_wait(wbio) < 0)
 			/* failure! */
 	for (i = conf->poolinfo->raid_disks; i--; ) {
 		bio = r1bio->bios[i];
 		rps = bio->bi_private;
-		bio_reset(bio);
+		bio_reset(bio, NULL, 0);
 		bio->bi_private = rps;
 	}
 	r1bio->master_bio = NULL;

drivers/md/raid10.c+13−17

@@ -1208,14 +1208,13 @@ static void raid10_read_request(struct mddev mddev, struct bio bio, · 1207 unmodified lines
1208		1208
1209	if (blk_queue_io_stat(bio->bi_bdev->bd_disk->queue))	1209	if (blk_queue_io_stat(bio->bi_bdev->bd_disk->queue))
1210	r10_bio->start_time = bio_start_io_acct(bio);	1210	r10_bio->start_time = bio_start_io_acct(bio);
1211	read_bio = bio_clone_fast(bio, gfp, &mddev->bio_set);	1211	read_bio = bio_alloc_clone(rdev->bdev, bio, gfp, &mddev->bio_set);
1212		1212
1213	r10_bio->devs[slot].bio = read_bio;	1213	r10_bio->devs[slot].bio = read_bio;
1214	r10_bio->devs[slot].rdev = rdev;	1214	r10_bio->devs[slot].rdev = rdev;
1215		1215
1216	read_bio->bi_iter.bi_sector = r10_bio->devs[slot].addr +	1216	read_bio->bi_iter.bi_sector = r10_bio->devs[slot].addr +
1217	choose_data_offset(r10_bio, rdev);	1217	choose_data_offset(r10_bio, rdev);
1218	bio_set_dev(read_bio, rdev->bdev);
1219	read_bio->bi_end_io = raid10_end_read_request;	1218	read_bio->bi_end_io = raid10_end_read_request;
1220	bio_set_op_attrs(read_bio, op, do_sync);	1219	bio_set_op_attrs(read_bio, op, do_sync);
1221	if (test_bit(FailFast, &rdev->flags) &&	1220	if (test_bit(FailFast, &rdev->flags) &&
@@ -1255,7 +1254,7 @@ static void raid10_write_one_disk(struct mddev mddev, struct r10bio r10_bio, · 33 unmodified lines
1255	} else	1254	} else
1256	rdev = conf->mirrors[devnum].rdev;	1255	rdev = conf->mirrors[devnum].rdev;
1257		1256
1258	mbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set);	1257	mbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, &mddev->bio_set);
1259	if (replacement)	1258	if (replacement)
1260	r10_bio->devs[n_copy].repl_bio = mbio;	1259	r10_bio->devs[n_copy].repl_bio = mbio;
1261	else	1260	else
@@ -1263,7 +1262,6 @@ static void raid10_write_one_disk(struct mddev mddev, struct r10bio r10_bio, · 1 unmodified line
1263		1262
1264	mbio->bi_iter.bi_sector = (r10_bio->devs[n_copy].addr +	1263	mbio->bi_iter.bi_sector = (r10_bio->devs[n_copy].addr +
1265	choose_data_offset(r10_bio, rdev));	1264	choose_data_offset(r10_bio, rdev));
1266	bio_set_dev(mbio, rdev->bdev);
1267	mbio->bi_end_io = raid10_end_write_request;	1265	mbio->bi_end_io = raid10_end_write_request;
1268	bio_set_op_attrs(mbio, op, do_sync \| do_fua);	1266	bio_set_op_attrs(mbio, op, do_sync \| do_fua);
1269	if (!replacement && test_bit(FailFast,	1267	if (!replacement && test_bit(FailFast,
@@ -1812,7 +1810,8 @@ retry_discard: · 542 unmodified lines
1812	*/	1810	*/
1813	if (r10_bio->devs[disk].bio) {	1811	if (r10_bio->devs[disk].bio) {
1814	struct md_rdev *rdev = conf->mirrors[disk].rdev;	1812	struct md_rdev *rdev = conf->mirrors[disk].rdev;
1815	mbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set);	1813	mbio = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOIO,
		1814	&mddev->bio_set);
1816	mbio->bi_end_io = raid10_end_discard_request;	1815	mbio->bi_end_io = raid10_end_discard_request;
1817	mbio->bi_private = r10_bio;	1816	mbio->bi_private = r10_bio;
1818	r10_bio->devs[disk].bio = mbio;	1817	r10_bio->devs[disk].bio = mbio;
@@ -1825,7 +1824,8 @@ retry_discard: · 6 unmodified lines
1825	}	1824	}
1826	if (r10_bio->devs[disk].repl_bio) {	1825	if (r10_bio->devs[disk].repl_bio) {
1827	struct md_rdev *rrdev = conf->mirrors[disk].replacement;	1826	struct md_rdev *rrdev = conf->mirrors[disk].replacement;
1828	rbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set);	1827	rbio = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOIO,
		1828	&mddev->bio_set);
1829	rbio->bi_end_io = raid10_end_discard_request;	1829	rbio->bi_end_io = raid10_end_discard_request;
1830	rbio->bi_private = r10_bio;	1830	rbio->bi_private = r10_bio;
1831	r10_bio->devs[disk].repl_bio = rbio;	1831	r10_bio->devs[disk].repl_bio = rbio;
@@ -2422,7 +2422,7 @@ static void sync_request_write(struct mddev mddev, struct r10bio r10_bio) · 590 unmodified lines
2422	* bi_vecs, as the read request might have corrupted these	2422	* bi_vecs, as the read request might have corrupted these
2423	*/	2423	*/
2424	rp = get_resync_pages(tbio);	2424	rp = get_resync_pages(tbio);
2425	bio_reset(tbio);	2425	bio_reset(tbio, conf->mirrors[d].rdev->bdev, REQ_OP_WRITE);
2426		2426
2427	md_bio_reset_resync_pages(tbio, rp, fbio->bi_iter.bi_size);	2427	md_bio_reset_resync_pages(tbio, rp, fbio->bi_iter.bi_size);
2428		2428
@@ -2430,7 +2430,6 @@ static void sync_request_write(struct mddev mddev, struct r10bio r10_bio) · 1 unmodified line
2430	tbio->bi_private = rp;	2430	tbio->bi_private = rp;
2431	tbio->bi_iter.bi_sector = r10_bio->devs[i].addr;	2431	tbio->bi_iter.bi_sector = r10_bio->devs[i].addr;
2432	tbio->bi_end_io = end_sync_write;	2432	tbio->bi_end_io = end_sync_write;
2433	bio_set_op_attrs(tbio, REQ_OP_WRITE, 0);
2434		2433
2435	bio_copy_data(tbio, fbio);	2434	bio_copy_data(tbio, fbio);
2436		2435
@@ -2441,7 +2440,6 @@ static void sync_request_write(struct mddev mddev, struct r10bio r10_bio) · 4 unmodified lines
2441	if (test_bit(FailFast, &conf->mirrors[d].rdev->flags))	2440	if (test_bit(FailFast, &conf->mirrors[d].rdev->flags))
2442	tbio->bi_opf \|= MD_FAILFAST;	2441	tbio->bi_opf \|= MD_FAILFAST;
2443	tbio->bi_iter.bi_sector += conf->mirrors[d].rdev->data_offset;	2442	tbio->bi_iter.bi_sector += conf->mirrors[d].rdev->data_offset;
2444	bio_set_dev(tbio, conf->mirrors[d].rdev->bdev);
2445	submit_bio_noacct(tbio);	2443	submit_bio_noacct(tbio);
2446	}	2444	}
2447		2445
@@ -2894,12 +2892,12 @@ static int narrow_write_error(struct r10bio *r10_bio, int i) · 446 unmodified lines
2894	if (sectors > sect_to_write)	2892	if (sectors > sect_to_write)
2895	sectors = sect_to_write;	2893	sectors = sect_to_write;
2896	/* Write at 'sector' for 'sectors' */	2894	/* Write at 'sector' for 'sectors' */
2897	wbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set);	2895	wbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO,
		2896	&mddev->bio_set);
2898	bio_trim(wbio, sector - bio->bi_iter.bi_sector, sectors);	2897	bio_trim(wbio, sector - bio->bi_iter.bi_sector, sectors);
2899	wsector = r10_bio->devs[i].addr + (sector - r10_bio->sector);	2898	wsector = r10_bio->devs[i].addr + (sector - r10_bio->sector);
2900	wbio->bi_iter.bi_sector = wsector +	2899	wbio->bi_iter.bi_sector = wsector +
2901	choose_data_offset(r10_bio, rdev);	2900	choose_data_offset(r10_bio, rdev);
2902	bio_set_dev(wbio, rdev->bdev);
2903	bio_set_op_attrs(wbio, REQ_OP_WRITE, 0);	2901	bio_set_op_attrs(wbio, REQ_OP_WRITE, 0);
2904		2902
2905	if (submit_bio_wait(wbio) < 0)	2903	if (submit_bio_wait(wbio) < 0)
@@ -3160,12 +3158,12 @@ static struct r10bio raid10_alloc_init_r10buf(struct r10conf conf) · 254 unmodified lines
3160	for (i = 0; i < nalloc; i++) {	3158	for (i = 0; i < nalloc; i++) {
3161	bio = r10bio->devs[i].bio;	3159	bio = r10bio->devs[i].bio;
3162	rp = bio->bi_private;	3160	rp = bio->bi_private;
3163	bio_reset(bio);	3161	bio_reset(bio, NULL, 0);
3164	bio->bi_private = rp;	3162	bio->bi_private = rp;
3165	bio = r10bio->devs[i].repl_bio;	3163	bio = r10bio->devs[i].repl_bio;
3166	if (bio) {	3164	if (bio) {
3167	rp = bio->bi_private;	3165	rp = bio->bi_private;
3168	bio_reset(bio);	3166	bio_reset(bio, NULL, 0);
3169	bio->bi_private = rp;	3167	bio->bi_private = rp;
3170	}	3168	}
3171	}	3169	}
@@ -4892,14 +4890,12 @@ read_more: · 1720 unmodified lines
4892	return sectors_done;	4890	return sectors_done;
4893	}	4891	}
4894		4892
4895	read_bio = bio_alloc_bioset(GFP_KERNEL, RESYNC_PAGES, &mddev->bio_set);	4893	read_bio = bio_alloc_bioset(rdev->bdev, RESYNC_PAGES, REQ_OP_READ,
4896		4894	GFP_KERNEL, &mddev->bio_set);
4897	bio_set_dev(read_bio, rdev->bdev);
4898	read_bio->bi_iter.bi_sector = (r10_bio->devs[r10_bio->read_slot].addr	4895	read_bio->bi_iter.bi_sector = (r10_bio->devs[r10_bio->read_slot].addr
4899	+ rdev->data_offset);	4896	+ rdev->data_offset);
4900	read_bio->bi_private = r10_bio;	4897	read_bio->bi_private = r10_bio;
4901	read_bio->bi_end_io = end_reshape_read;	4898	read_bio->bi_end_io = end_reshape_read;
4902	bio_set_op_attrs(read_bio, REQ_OP_READ, 0);
4903	r10_bio->master_bio = read_bio;	4899	r10_bio->master_bio = read_bio;
4904	r10_bio->read_slot = r10_bio->devs[r10_bio->read_slot].devnum;	4900	r10_bio->read_slot = r10_bio->devs[r10_bio->read_slot].devnum;
4905		4901

drivers/md/raid5-cache.c+8−11

 static struct bio *r5l_bio_alloc(struct r5l_log *log)
 {
-	struct bio *bio = bio_alloc_bioset(GFP_NOIO, BIO_MAX_VECS, &log->bs);
+	struct bio *bio = bio_alloc_bioset(log->rdev->bdev, BIO_MAX_VECS,
+					   REQ_OP_WRITE, GFP_NOIO, &log->bs);
-	bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
-	bio_set_dev(bio, log->rdev->bdev);
 	bio->bi_iter.bi_sector = log->rdev->data_offset + log->log_start;
 	return bio;
 	if (!do_flush)
 		return;
-	bio_reset(&log->flush_bio);
+	bio_reset(&log->flush_bio, log->rdev->bdev,
-	bio_set_dev(&log->flush_bio, log->rdev->bdev);
+		  REQ_OP_WRITE | REQ_PREFLUSH);
 	log->flush_bio.bi_end_io = r5l_log_flush_endio;
-	log->flush_bio.bi_opf = REQ_OP_WRITE | REQ_PREFLUSH;
 	submit_bio(&log->flush_bio);
 }
 {
 	struct page *page;
-	ctx->ra_bio = bio_alloc_bioset(GFP_KERNEL, BIO_MAX_VECS, &log->bs);
+	ctx->ra_bio = bio_alloc_bioset(NULL, BIO_MAX_VECS, 0, GFP_KERNEL,
+				       &log->bs);
 	if (!ctx->ra_bio)
 		return -ENOMEM;
 				      struct r5l_recovery_ctx *ctx,
 				      sector_t offset)
 {
-	bio_reset(ctx->ra_bio);
+	bio_reset(ctx->ra_bio, log->rdev->bdev, REQ_OP_READ);
-	bio_set_dev(ctx->ra_bio, log->rdev->bdev);
-	bio_set_op_attrs(ctx->ra_bio, REQ_OP_READ, 0);
 	ctx->ra_bio->bi_iter.bi_sector = log->rdev->data_offset + offset;
 	ctx->valid_pages = 0;
 	INIT_LIST_HEAD(&log->io_end_ios);
 	INIT_LIST_HEAD(&log->flushing_ios);
 	INIT_LIST_HEAD(&log->finished_ios);
-	bio_init(&log->flush_bio, NULL, 0);
+	bio_init(&log->flush_bio, NULL, NULL, 0, 0);
 	log->io_kc = KMEM_CACHE(r5l_io_unit, 0);
 	if (!log->io_kc)

drivers/md/raid5-ppl.c+10−16

 	INIT_LIST_HEAD(&io->stripe_list);
 	atomic_set(&io->pending_stripes, 0);
 	atomic_set(&io->pending_flushes, 0);
-	bio_init(&io->bio, io->biovec, PPL_IO_INLINE_BVECS);
+	bio_init(&io->bio, NULL, io->biovec, PPL_IO_INLINE_BVECS, 0);
 	pplhdr = page_address(io->header_page);
 	clear_page(pplhdr);
 static void ppl_submit_iounit_bio(struct ppl_io_unit *io, struct bio *bio)
 {
-	char b[BDEVNAME_SIZE];
+	pr_debug("%s: seq: %llu size: %u sector: %llu dev: %pg\n",
-	pr_debug("%s: seq: %llu size: %u sector: %llu dev: %s\n",
 		 __func__, io->seq, bio->bi_iter.bi_size,
 		 (unsigned long long)bio->bi_iter.bi_sector,
-		 bio_devname(bio, b));
+		 bio->bi_bdev);
 	submit_bio(bio);
 }
 		if (!bio_add_page(bio, sh->ppl_page, PAGE_SIZE, 0)) {
 			struct bio *prev = bio;
-			bio = bio_alloc_bioset(GFP_NOIO, BIO_MAX_VECS,
+			bio = bio_alloc_bioset(prev->bi_bdev, BIO_MAX_VECS,
+					       prev->bi_opf, GFP_NOIO,
 					       &ppl_conf->bs);
-			bio->bi_opf = prev->bi_opf;
 			bio->bi_write_hint = prev->bi_write_hint;
-			bio_copy_dev(bio, prev);
 			bio->bi_iter.bi_sector = bio_end_sector(prev);
 			bio_add_page(bio, sh->ppl_page, PAGE_SIZE, 0);
 	struct ppl_log *log = io->log;
 	struct ppl_conf *ppl_conf = log->ppl_conf;
 	struct r5conf *conf = ppl_conf->mddev->private;
-	char b[BDEVNAME_SIZE];
-	pr_debug("%s: dev: %s\n", __func__, bio_devname(bio, b));
+	pr_debug("%s: dev: %pg\n", __func__, bio->bi_bdev);
 	if (bio->bi_status) {
 		struct md_rdev *rdev;
 		if (bdev) {
 			struct bio *bio;
-			char b[BDEVNAME_SIZE];
-			bio = bio_alloc_bioset(GFP_NOIO, 0, &ppl_conf->flush_bs);
+			bio = bio_alloc_bioset(bdev, 0, GFP_NOIO,
-			bio_set_dev(bio, bdev);
+					       REQ_OP_WRITE | REQ_PREFLUSH,
+					       &ppl_conf->flush_bs);
 			bio->bi_private = io;
-			bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH;
 			bio->bi_end_io = ppl_flush_endio;
-			pr_debug("%s: dev: %s\n", __func__,
+			pr_debug("%s: dev: %ps\n", __func__, bio->bi_bdev);
-				 bio_devname(bio, b));
 			submit_bio(bio);
 			flushed_disks++;

drivers/md/raid5.c+8−8

 		for (i = 0; i < disks; i++) {
 			struct r5dev *dev = &sh->dev[i];
-			bio_init(&dev->req, &dev->vec, 1);
+			bio_init(&dev->req, NULL, &dev->vec, 1, 0);
-			bio_init(&dev->rreq, &dev->rvec, 1);
+			bio_init(&dev->rreq, NULL, &dev->rvec, 1, 0);
 		}
 		if (raid5_has_ppl(conf)) {
 		(unsigned long long)sh->sector, i, atomic_read(&sh->count),
 		bi->bi_status);
 	if (i == disks) {
-		bio_reset(bi);
+		bio_reset(bi, NULL, 0);
 		BUG();
 		return;
 	}
 		}
 	}
 	rdev_dec_pending(rdev, conf->mddev);
-	bio_reset(bi);
+	bio_reset(bi, NULL, 0);
 	clear_bit(R5_LOCKED, &sh->dev[i].flags);
 	set_bit(STRIPE_HANDLE, &sh->state);
 	raid5_release_stripe(sh);
 		(unsigned long long)sh->sector, i, atomic_read(&sh->count),
 		bi->bi_status);
 	if (i == disks) {
-		bio_reset(bi);
+		bio_reset(bi, NULL, 0);
 		BUG();
 		return;
 	}
 	if (sh->batch_head && bi->bi_status && !replacement)
 		set_bit(STRIPE_BATCH_ERR, &sh->batch_head->state);
-	bio_reset(bi);
+	bio_reset(bi, NULL, 0);
 	if (!test_and_clear_bit(R5_DOUBLE_LOCKED, &sh->dev[i].flags))
 		clear_bit(R5_LOCKED, &sh->dev[i].flags);
 	set_bit(STRIPE_HANDLE, &sh->state);
 		return 0;
 	}
-	align_bio = bio_clone_fast(raid_bio, GFP_NOIO, &mddev->io_acct_set);
+	align_bio = bio_alloc_clone(rdev->bdev, raid_bio, GFP_NOIO,
+				    &mddev->io_acct_set);
 	md_io_acct = container_of(align_bio, struct md_io_acct, bio_clone);
 	raid_bio->bi_next = (void *)rdev;
 	if (blk_queue_io_stat(raid_bio->bi_bdev->bd_disk->queue))
 		md_io_acct->start_time = bio_start_io_acct(raid_bio);
 	md_io_acct->orig_bio = raid_bio;
-	bio_set_dev(align_bio, rdev->bdev);
 	align_bio->bi_end_io = raid5_align_endio;
 	align_bio->bi_private = md_io_acct;
 	align_bio->bi_iter.bi_sector = sector;

drivers/memstick/core/ms_block.c+15−49

 static DEFINE_IDR(msb_disk_idr); /*set of used disk numbers */
 static DEFINE_MUTEX(msb_disk_lock); /* protects against races in open/release */
-static int msb_bd_open(struct block_device *bdev, fmode_t mode)
-{
-	struct gendisk *disk = bdev->bd_disk;
-	struct msb_data *msb = disk->private_data;
-	dbg_verbose("block device open");
-	mutex_lock(&msb_disk_lock);
-	if (msb && msb->card)
-		msb->usage_count++;
-	mutex_unlock(&msb_disk_lock);
-	return 0;
-}
 static void msb_data_clear(struct msb_data *msb)
 {
 	kfree(msb->boot_page);
 	msb->card = NULL;
 }
-static int msb_disk_release(struct gendisk *disk)
-{
-	struct msb_data *msb = disk->private_data;
-	dbg_verbose("block device release");
-	mutex_lock(&msb_disk_lock);
-	if (msb) {
-		if (msb->usage_count)
-			msb->usage_count--;
-		if (!msb->usage_count) {
-			disk->private_data = NULL;
-			idr_remove(&msb_disk_idr, msb->disk_id);
-			put_disk(disk);
-			kfree(msb);
-		}
-	}
-	mutex_unlock(&msb_disk_lock);
-	return 0;
-}
-static void msb_bd_release(struct gendisk *disk, fmode_t mode)
-{
-	msb_disk_release(disk);
-}
 static int msb_bd_getgeo(struct block_device *bdev,
 				 struct hd_geometry *geo)
 {
 	return 0;
 }
+static void msb_bd_free_disk(struct gendisk *disk)
+{
+	struct msb_data *msb = disk->private_data;
+	mutex_lock(&msb_disk_lock);
+	idr_remove(&msb_disk_idr, msb->disk_id);
+	mutex_unlock(&msb_disk_lock);
+	kfree(msb);
+}
 static blk_status_t msb_queue_rq(struct blk_mq_hw_ctx *hctx,
 				 const struct blk_mq_queue_data *bd)
 {
 }
 static const struct block_device_operations msb_bdops = {
-	.open    = msb_bd_open,
+	.owner		= THIS_MODULE,
-	.release = msb_bd_release,
+	.getgeo		= msb_bd_getgeo,
-	.getgeo  = msb_bd_getgeo,
+	.free_disk	= msb_bd_free_disk,
-	.owner   = THIS_MODULE
 };
 static const struct blk_mq_ops msb_mq_ops = {
 	set_capacity(msb->disk, capacity);
 	dbg("Set total disk size to %lu sectors", capacity);
-	msb->usage_count = 1;
 	msb->io_queue = alloc_ordered_workqueue("ms_block", WQ_MEM_RECLAIM);
 	INIT_WORK(&msb->io_work, msb_io_work);
 	sg_init_table(msb->prealloc_sg, MS_BLOCK_MAX_SEGS+1);
 	msb_data_clear(msb);
 	mutex_unlock(&msb_disk_lock);
-	msb_disk_release(msb->disk);
+	put_disk(msb->disk);
 	memstick_set_drvdata(card, NULL);
 }

drivers/memstick/core/ms_block.h+0−1

@@ -143,7 +143,6 @@ struct ms_boot_page { · 142 unmodified lines
143	} __packed;	143	} __packed;
144		144
145	struct msb_data {	145	struct msb_data {
146	unsigned int usage_count;
147	struct memstick_dev *card;	146	struct memstick_dev *card;
148	struct gendisk *disk;	147	struct gendisk *disk;
149	struct request_queue *queue;	148	struct request_queue *queue;

drivers/memstick/core/mspro_block.c+10−47

@@ -133,7 +133,6 @@ struct mspro_devinfo { · 132 unmodified lines
133		133
134	struct mspro_block_data {	134	struct mspro_block_data {
135	struct memstick_dev *card;	135	struct memstick_dev *card;
136	unsigned int usage_count;
137	unsigned int caps;	136	unsigned int caps;
138	struct gendisk *disk;	137	struct gendisk *disk;
139	struct request_queue *queue;	138	struct request_queue *queue;
@@ -178,53 +177,16 @@ static int mspro_block_complete_req(struct memstick_dev *card, int error); · 38 unmodified lines
178		177
179	/* Block device */	178	/* Block device */
180		179
181	static int mspro_block_bd_open(struct block_device *bdev, fmode_t mode)	180	static void mspro_block_bd_free_disk(struct gendisk *disk)
182	{
183	struct gendisk *disk = bdev->bd_disk;
184	struct mspro_block_data *msb = disk->private_data;
185	int rc = -ENXIO;
186
187	mutex_lock(&mspro_block_disk_lock);
188
189	if (msb && msb->card) {
190	msb->usage_count++;
191	if ((mode & FMODE_WRITE) && msb->read_only)
192	rc = -EROFS;
193	else
194	rc = 0;
195	}
196
197	mutex_unlock(&mspro_block_disk_lock);
198
199	return rc;
200	}
201
202
203	static void mspro_block_disk_release(struct gendisk *disk)
204	{	181	{
205	struct mspro_block_data *msb = disk->private_data;	182	struct mspro_block_data *msb = disk->private_data;
206	int disk_id = MINOR(disk_devt(disk)) >> MSPRO_BLOCK_PART_SHIFT;	183	int disk_id = MINOR(disk_devt(disk)) >> MSPRO_BLOCK_PART_SHIFT;
207		184
208	mutex_lock(&mspro_block_disk_lock);	185	mutex_lock(&mspro_block_disk_lock);
209		186	idr_remove(&mspro_block_disk_idr, disk_id);
210	if (msb) {
211	if (msb->usage_count)
212	msb->usage_count--;
213
214	if (!msb->usage_count) {
215	kfree(msb);
216	disk->private_data = NULL;
217	idr_remove(&mspro_block_disk_idr, disk_id);
218	put_disk(disk);
219	}
220	}
221
222	mutex_unlock(&mspro_block_disk_lock);	187	mutex_unlock(&mspro_block_disk_lock);
223	}
224		188
225	static void mspro_block_bd_release(struct gendisk *disk, fmode_t mode)	189	kfree(msb);
226	{
227	mspro_block_disk_release(disk);
228	}	190	}
229		191
230	static int mspro_block_bd_getgeo(struct block_device *bdev,	192	static int mspro_block_bd_getgeo(struct block_device *bdev,
@@ -240,10 +202,9 @@ static int mspro_block_bd_getgeo(struct block_device *bdev, · 9 unmodified lines
240	}	202	}
241		203
242	static const struct block_device_operations ms_block_bdops = {	204	static const struct block_device_operations ms_block_bdops = {
243	.open = mspro_block_bd_open,	205	.owner = THIS_MODULE,
244	.release = mspro_block_bd_release,	206	.getgeo = mspro_block_bd_getgeo,
245	.getgeo = mspro_block_bd_getgeo,	207	.free_disk = mspro_block_bd_free_disk,
246	.owner = THIS_MODULE
247	};	208	};
248		209
249	/* Information */	210	/* Information */
@@ -1226,7 +1187,6 @@ static int mspro_block_init_disk(struct memstick_dev *card) · 976 unmodified lines
1226	msb->disk->first_minor = disk_id << MSPRO_BLOCK_PART_SHIFT;	1187	msb->disk->first_minor = disk_id << MSPRO_BLOCK_PART_SHIFT;
1227	msb->disk->minors = 1 << MSPRO_BLOCK_PART_SHIFT;	1188	msb->disk->minors = 1 << MSPRO_BLOCK_PART_SHIFT;
1228	msb->disk->fops = &ms_block_bdops;	1189	msb->disk->fops = &ms_block_bdops;
1229	msb->usage_count = 1;
1230	msb->disk->private_data = msb;	1190	msb->disk->private_data = msb;
1231		1191
1232	sprintf(msb->disk->disk_name, "mspblk%d", disk_id);	1192	sprintf(msb->disk->disk_name, "mspblk%d", disk_id);
@@ -1239,6 +1199,9 @@ static int mspro_block_init_disk(struct memstick_dev *card) · 6 unmodified lines
1239	set_capacity(msb->disk, capacity);	1199	set_capacity(msb->disk, capacity);
1240	dev_dbg(&card->dev, "capacity set %ld\n", capacity);	1200	dev_dbg(&card->dev, "capacity set %ld\n", capacity);
1241		1201
		1202	if (msb->read_only)
		1203	set_disk_ro(msb->disk, true);
		1204
1242	rc = device_add_disk(&card->dev, msb->disk, NULL);	1205	rc = device_add_disk(&card->dev, msb->disk, NULL);
1243	if (rc)	1206	if (rc)
1244	goto out_cleanup_disk;	1207	goto out_cleanup_disk;
@@ -1341,7 +1304,7 @@ static void mspro_block_remove(struct memstick_dev *card) · 96 unmodified lines
1341	mspro_block_data_clear(msb);	1304	mspro_block_data_clear(msb);
1342	mutex_unlock(&mspro_block_disk_lock);	1305	mutex_unlock(&mspro_block_disk_lock);
1343		1306
1344	mspro_block_disk_release(msb->disk);	1307	put_disk(msb->disk);
1345	memstick_set_drvdata(card, NULL);	1308	memstick_set_drvdata(card, NULL);
1346	}	1309	}
1347		1310

drivers/mtd/mtdswap.c+1−1

@@ -19,7 +19,7 @@ · 18 unmodified lines
19	#include <linux/sched.h>	19	#include <linux/sched.h>
20	#include <linux/slab.h>	20	#include <linux/slab.h>
21	#include <linux/vmalloc.h>	21	#include <linux/vmalloc.h>
22	#include <linux/genhd.h>	22	#include <linux/blkdev.h>
23	#include <linux/swap.h>	23	#include <linux/swap.h>
24	#include <linux/debugfs.h>	24	#include <linux/debugfs.h>
25	#include <linux/seq_file.h>	25	#include <linux/seq_file.h>

drivers/mtd/nand/raw/sharpsl.c+0−1

@@ -6,7 +6,6 @@ · 5 unmodified lines
6	* Based on Sharp's NAND driver sharp_sl.c	6	* Based on Sharp's NAND driver sharp_sl.c
7	*/	7	*/
8		8
9	#include <linux/genhd.h>
10	#include <linux/slab.h>	9	#include <linux/slab.h>
11	#include <linux/module.h>	10	#include <linux/module.h>
12	#include <linux/delay.h>	11	#include <linux/delay.h>

drivers/nvdimm/blk.c+0−1

@@ -6,7 +6,6 @@ · 5 unmodified lines
6		6
7	#include <linux/blkdev.h>	7	#include <linux/blkdev.h>
8	#include <linux/fs.h>	8	#include <linux/fs.h>
9	#include <linux/genhd.h>
10	#include <linux/module.h>	9	#include <linux/module.h>
11	#include <linux/moduleparam.h>	10	#include <linux/moduleparam.h>
12	#include <linux/nd.h>	11	#include <linux/nd.h>

drivers/nvdimm/btt.c+0−1

@@ -11,7 +11,6 @@ · 10 unmodified lines
11	#include <linux/device.h>	11	#include <linux/device.h>
12	#include <linux/mutex.h>	12	#include <linux/mutex.h>
13	#include <linux/hdreg.h>	13	#include <linux/hdreg.h>
14	#include <linux/genhd.h>
15	#include <linux/sizes.h>	14	#include <linux/sizes.h>
16	#include <linux/ndctl.h>	15	#include <linux/ndctl.h>
17	#include <linux/fs.h>	16	#include <linux/fs.h>

drivers/nvdimm/btt_devs.c+0−1

@@ -4,7 +4,6 @@ · 3 unmodified lines
4	*/	4	*/
5	#include <linux/blkdev.h>	5	#include <linux/blkdev.h>
6	#include <linux/device.h>	6	#include <linux/device.h>
7	#include <linux/genhd.h>
8	#include <linux/sizes.h>	7	#include <linux/sizes.h>
9	#include <linux/slab.h>	8	#include <linux/slab.h>
10	#include <linux/fs.h>	9	#include <linux/fs.h>

drivers/nvdimm/bus.c+0−1

@@ -11,7 +11,6 @@ · 10 unmodified lines
11	#include <linux/blkdev.h>	11	#include <linux/blkdev.h>
12	#include <linux/fcntl.h>	12	#include <linux/fcntl.h>
13	#include <linux/async.h>	13	#include <linux/async.h>
14	#include <linux/genhd.h>
15	#include <linux/ndctl.h>	14	#include <linux/ndctl.h>
16	#include <linux/sched.h>	15	#include <linux/sched.h>
17	#include <linux/slab.h>	16	#include <linux/slab.h>

drivers/nvdimm/nd_virtio.c+3−3

 	 * parent bio. Otherwise directly call nd_region flush.
 	 */
 	if (bio && bio->bi_iter.bi_sector != -1) {
-		struct bio *child = bio_alloc(GFP_ATOMIC, 0);
+		struct bio *child = bio_alloc(bio->bi_bdev, 0, REQ_PREFLUSH,
+					      GFP_ATOMIC);
 		if (!child)
 			return -ENOMEM;
-		bio_copy_dev(child, bio);
+		bio_clone_blkg_association(child, bio);
-		child->bi_opf = REQ_PREFLUSH;
 		child->bi_iter.bi_sector = -1;
 		bio_chain(child, bio);
 		submit_bio(child);

drivers/nvdimm/pfn_devs.c+0−1

@@ -5,7 +5,6 @@ · 4 unmodified lines
5	#include <linux/memremap.h>	5	#include <linux/memremap.h>
6	#include <linux/blkdev.h>	6	#include <linux/blkdev.h>
7	#include <linux/device.h>	7	#include <linux/device.h>
8	#include <linux/genhd.h>
9	#include <linux/sizes.h>	8	#include <linux/sizes.h>
10	#include <linux/slab.h>	9	#include <linux/slab.h>
11	#include <linux/fs.h>	10	#include <linux/fs.h>

drivers/nvme/target/io-cmd-bdev.c+8−10

 	if (nvmet_use_inline_bvec(req)) {
 		bio = &req->b.inline_bio;
-		bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
+		bio_init(bio, req->ns->bdev, req->inline_bvec,
+			 ARRAY_SIZE(req->inline_bvec), op);
 	} else {
-		bio = bio_alloc(GFP_KERNEL, bio_max_segs(sg_cnt));
+		bio = bio_alloc(req->ns->bdev, bio_max_segs(sg_cnt), op,
+				GFP_KERNEL);
 	}
-	bio_set_dev(bio, req->ns->bdev);
 	bio->bi_iter.bi_sector = sector;
 	bio->bi_private = req;
 	bio->bi_end_io = nvmet_bio_done;
-	bio->bi_opf = op;
 	blk_start_plug(&plug);
 	if (req->metadata_len)
 				}
 			}
-			bio = bio_alloc(GFP_KERNEL, bio_max_segs(sg_cnt));
+			bio = bio_alloc(req->ns->bdev, bio_max_segs(sg_cnt),
-			bio_set_dev(bio, req->ns->bdev);
+					op, GFP_KERNEL);
 			bio->bi_iter.bi_sector = sector;
-			bio->bi_opf = op;
 			bio_chain(bio, prev);
 			submit_bio(prev);
 	if (!nvmet_check_transfer_len(req, 0))
 		return;
-	bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
+	bio_init(bio, req->ns->bdev, req->inline_bvec,
-	bio_set_dev(bio, req->ns->bdev);
+		 ARRAY_SIZE(req->inline_bvec), REQ_OP_WRITE | REQ_PREFLUSH);
 	bio->bi_private = req;
 	bio->bi_end_io = nvmet_bio_done;
-	bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH;
 	submit_bio(bio);
 }

drivers/nvme/target/passthru.c+4−3

 	if (nvmet_use_inline_bvec(req)) {
 		bio = &req->p.inline_bio;
-		bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
+		bio_init(bio, NULL, req->inline_bvec,
+			 ARRAY_SIZE(req->inline_bvec), req_op(rq));
 	} else {
-		bio = bio_alloc(GFP_KERNEL, bio_max_segs(req->sg_cnt));
+		bio = bio_alloc(NULL, bio_max_segs(req->sg_cnt), req_op(rq),
+				GFP_KERNEL);
 		bio->bi_end_io = bio_put;
 	}
-	bio->bi_opf = req_op(rq);
 	for_each_sg(req->sg, sg, req->sg_cnt, i) {
 		if (bio_add_pc_page(rq->q, bio, sg_page(sg), sg->length,

drivers/nvme/target/zns.c+7−7

 	while (sector < get_capacity(bdev->bd_disk)) {
 		if (test_bit(blk_queue_zone_no(q, sector), d.zbitmap)) {
-			bio = blk_next_bio(bio, 0, GFP_KERNEL);
+			bio = blk_next_bio(bio, bdev, 0,
-			bio->bi_opf = zsa_req_op(req->cmd->zms.zsa) | REQ_SYNC;
+				zsa_req_op(req->cmd->zms.zsa) | REQ_SYNC,
+				GFP_KERNEL);
 			bio->bi_iter.bi_sector = sector;
-			bio_set_dev(bio, bdev);
 			/* This may take a while, so be nice to others */
 			cond_resched();
 		}
 void nvmet_bdev_execute_zone_append(struct nvmet_req *req)
 {
 	sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->rw.slba);
+	const unsigned int op = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE;
 	u16 status = NVME_SC_SUCCESS;
 	unsigned int total_len = 0;
 	struct scatterlist *sg;
 	if (nvmet_use_inline_bvec(req)) {
 		bio = &req->z.inline_bio;
-		bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec));
+		bio_init(bio, req->ns->bdev, req->inline_bvec,
+			 ARRAY_SIZE(req->inline_bvec), op);
 	} else {
-		bio = bio_alloc(GFP_KERNEL, req->sg_cnt);
+		bio = bio_alloc(req->ns->bdev, req->sg_cnt, op, GFP_KERNEL);
 	}
-	bio->bi_opf = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE;
 	bio->bi_end_io = nvmet_bdev_zone_append_bio_done;
-	bio_set_dev(bio, req->ns->bdev);
 	bio->bi_iter.bi_sector = sect;
 	bio->bi_private = req;
 	if (req->cmd->rw.control & cpu_to_le16(NVME_RW_FUA))

drivers/s390/block/dasd_int.h+0−1

@@ -47,7 +47,6 @@ · 46 unmodified lines
47	#include <linux/module.h>	47	#include <linux/module.h>
48	#include <linux/wait.h>	48	#include <linux/wait.h>
49	#include <linux/blkdev.h>	49	#include <linux/blkdev.h>
50	#include <linux/genhd.h>
51	#include <linux/hdreg.h>	50	#include <linux/hdreg.h>
52	#include <linux/interrupt.h>	51	#include <linux/interrupt.h>
53	#include <linux/log2.h>	52	#include <linux/log2.h>

drivers/s390/block/scm_blk.c+0−1

@@ -15,7 +15,6 @@ · 14 unmodified lines
15	#include <linux/module.h>	15	#include <linux/module.h>
16	#include <linux/blkdev.h>	16	#include <linux/blkdev.h>
17	#include <linux/blk-mq.h>	17	#include <linux/blk-mq.h>
18	#include <linux/genhd.h>
19	#include <linux/slab.h>	18	#include <linux/slab.h>
20	#include <linux/list.h>	19	#include <linux/list.h>
21	#include <asm/eadm.h>	20	#include <asm/eadm.h>

drivers/s390/block/scm_blk.h+0−1

@@ -6,7 +6,6 @@ · 5 unmodified lines
6	#include <linux/spinlock.h>	6	#include <linux/spinlock.h>
7	#include <linux/blkdev.h>	7	#include <linux/blkdev.h>
8	#include <linux/blk-mq.h>	8	#include <linux/blk-mq.h>
9	#include <linux/genhd.h>
10	#include <linux/list.h>	9	#include <linux/list.h>
11		10
12	#include <asm/debug.h>	11	#include <asm/debug.h>

drivers/scsi/scsi_debug.c+0−1

@@ -23,7 +23,6 @@ · 22 unmodified lines
23	#include <linux/slab.h>	23	#include <linux/slab.h>
24	#include <linux/types.h>	24	#include <linux/types.h>
25	#include <linux/string.h>	25	#include <linux/string.h>
26	#include <linux/genhd.h>
27	#include <linux/fs.h>	26	#include <linux/fs.h>
28	#include <linux/init.h>	27	#include <linux/init.h>
29	#include <linux/proc_fs.h>	28	#include <linux/proc_fs.h>

drivers/scsi/scsi_lib.c+1−1

@@ -1276,7 +1276,7 @@ scsi_device_state_check(struct scsi_device sdev, struct request req) · 1275 unmodified lines
1276	* power management commands.	1276	* power management commands.
1277	*/	1277	*/
1278	if (req && !(req->rq_flags & RQF_PM))	1278	if (req && !(req->rq_flags & RQF_PM))
1279	return BLK_STS_IOERR;	1279	return BLK_STS_OFFLINE;
1280	return BLK_STS_OK;	1280	return BLK_STS_OK;
1281	}	1281	}
1282	}	1282	}

drivers/scsi/scsicam.c+0−1

@@ -14,7 +14,6 @@ · 13 unmodified lines
14	#include <linux/module.h>	14	#include <linux/module.h>
15	#include <linux/slab.h>	15	#include <linux/slab.h>
16	#include <linux/fs.h>	16	#include <linux/fs.h>
17	#include <linux/genhd.h>
18	#include <linux/kernel.h>	17	#include <linux/kernel.h>
19	#include <linux/blkdev.h>	18	#include <linux/blkdev.h>
20	#include <linux/pagemap.h>	19	#include <linux/pagemap.h>

drivers/scsi/sd.c+27−88

 #include <linux/kernel.h>
 #include <linux/mm.h>
 #include <linux/bio.h>
-#include <linux/genhd.h>
 #include <linux/hdreg.h>
 #include <linux/errno.h>
 #include <linux/idr.h>
 static DEFINE_IDA(sd_index_ida);
-/* This semaphore is used to mediate the 0->1 reference get in the
- * face of object destruction (i.e. we can't allow a get on an
- * object after last put) */
-static DEFINE_MUTEX(sd_ref_mutex);
 static struct kmem_cache *sd_cdb_cache;
 static mempool_t *sd_cdb_pool;
 static mempool_t *sd_page_pool;
 	}
 }
-static struct scsi_disk *scsi_disk_get(struct gendisk *disk)
-{
-	struct scsi_disk *sdkp = NULL;
-	mutex_lock(&sd_ref_mutex);
-	if (disk->private_data) {
-		sdkp = scsi_disk(disk);
-		if (scsi_device_get(sdkp->device) == 0)
-			get_device(&sdkp->dev);
-		else
-			sdkp = NULL;
-	}
-	mutex_unlock(&sd_ref_mutex);
-	return sdkp;
-}
-static void scsi_disk_put(struct scsi_disk *sdkp)
-{
-	struct scsi_device *sdev = sdkp->device;
-	mutex_lock(&sd_ref_mutex);
-	put_device(&sdkp->dev);
-	scsi_device_put(sdev);
-	mutex_unlock(&sd_ref_mutex);
-}
 #ifdef CONFIG_BLK_SED_OPAL
 static int sd_sec_submit(void *data, u16 spsp, u8 secp, void *buffer,
 		size_t len, bool send)
  **/
 static int sd_open(struct block_device *bdev, fmode_t mode)
 {
-	struct scsi_disk *sdkp = scsi_disk_get(bdev->bd_disk);
+	struct scsi_disk *sdkp = scsi_disk(bdev->bd_disk);
-	struct scsi_device *sdev;
+	struct scsi_device *sdev = sdkp->device;
 	int retval;
-	if (!sdkp)
+	if (scsi_device_get(sdev))
 		return -ENXIO;
 	SCSI_LOG_HLQUEUE(3, sd_printk(KERN_INFO, sdkp, "sd_open\n"));
-	sdev = sdkp->device;
 	/*
 	 * If the device is in error recovery, wait until it is done.
 	 * If the device is offline, then disallow any access to it.
 	return 0;
 error_out:
-	scsi_disk_put(sdkp);
+	scsi_device_put(sdev);
 	return retval;
 }
 			scsi_set_medium_removal(sdev, SCSI_REMOVAL_ALLOW);
 	}
-	scsi_disk_put(sdkp);
+	scsi_device_put(sdev);
 }
 static int sd_getgeo(struct block_device *bdev, struct hd_geometry *geo)
  **/
 static unsigned int sd_check_events(struct gendisk *disk, unsigned int clearing)
 {
-	struct scsi_disk *sdkp = scsi_disk_get(disk);
+	struct scsi_disk *sdkp = disk->private_data;
 	struct scsi_device *sdp;
 	int retval;
 	bool disk_changed;
 	 */
 	disk_changed = sdp->changed;
 	sdp->changed = 0;
-	scsi_disk_put(sdkp);
 	return disk_changed ? DISK_EVENT_MEDIA_CHANGE : 0;
 }
 	.pr_clear	= sd_pr_clear,
 };
+static void scsi_disk_free_disk(struct gendisk *disk)
+{
+	struct scsi_disk *sdkp = scsi_disk(disk);
+	put_device(&sdkp->disk_dev);
+}
 static const struct block_device_operations sd_fops = {
 	.owner			= THIS_MODULE,
 	.open			= sd_open,
 	.unlock_native_capacity	= sd_unlock_native_capacity,
 	.report_zones		= sd_zbc_report_zones,
 	.get_unique_id		= sd_get_unique_id,
+	.free_disk		= scsi_disk_free_disk,
 	.pr_ops			= &sd_pr_ops,
 };
 	}
 	sdkp->device = sdp;
-	sdkp->driver = &sd_template;
 	sdkp->disk = gd;
 	sdkp->index = index;
 	sdkp->max_retries = SD_MAX_RETRIES;
 					     SD_MOD_TIMEOUT);
 	}
-	device_initialize(&sdkp->dev);
+	device_initialize(&sdkp->disk_dev);
-	sdkp->dev.parent = get_device(dev);
+	sdkp->disk_dev.parent = get_device(dev);
-	sdkp->dev.class = &sd_disk_class;
+	sdkp->disk_dev.class = &sd_disk_class;
-	dev_set_name(&sdkp->dev, "%s", dev_name(dev));
+	dev_set_name(&sdkp->disk_dev, "%s", dev_name(dev));
-	error = device_add(&sdkp->dev);
+	error = device_add(&sdkp->disk_dev);
 	if (error) {
-		put_device(&sdkp->dev);
+		put_device(&sdkp->disk_dev);
 		goto out;
 	}
 	gd->minors = SD_MINORS;
 	gd->fops = &sd_fops;
-	gd->private_data = &sdkp->driver;
+	gd->private_data = sdkp;
 	/* defaults, until the device tells us otherwise */
 	sdp->sector_size = 512;
 	error = device_add_disk(dev, gd, NULL);
 	if (error) {
-		put_device(&sdkp->dev);
+		put_device(&sdkp->disk_dev);
 		goto out;
 	}
  **/
 static int sd_remove(struct device *dev)
 {
-	struct scsi_disk *sdkp;
+	struct scsi_disk *sdkp = dev_get_drvdata(dev);
-	sdkp = dev_get_drvdata(dev);
 	scsi_autopm_get_device(sdkp->device);
-	device_del(&sdkp->dev);
+	device_del(&sdkp->disk_dev);
 	del_gendisk(sdkp->disk);
 	sd_shutdown(dev);
-	free_opal_dev(sdkp->opal_dev);
+	put_disk(sdkp->disk);
-	mutex_lock(&sd_ref_mutex);
-	dev_set_drvdata(dev, NULL);
-	put_device(&sdkp->dev);
-	mutex_unlock(&sd_ref_mutex);
 	return 0;
 }
-/**
- *	scsi_disk_release - Called to free the scsi_disk structure
- *	@dev: pointer to embedded class device
- *
- *	sd_ref_mutex must be held entering this routine.  Because it is
- *	called on last put, you should always use the scsi_disk_get()
- *	scsi_disk_put() helpers which manipulate the semaphore directly
- *	and never do a direct put_device.
- **/
 static void scsi_disk_release(struct device *dev)
 {
 	struct scsi_disk *sdkp = to_scsi_disk(dev);
-	struct gendisk *disk = sdkp->disk;
-	struct request_queue *q = disk->queue;
 	ida_free(&sd_index_ida, sdkp->index);
-	/*
-	 * Wait until all requests that are in progress have completed.
-	 * This is necessary to avoid that e.g. scsi_end_request() crashes
-	 * due to clearing the disk->private_data pointer. Wait from inside
-	 * scsi_disk_release() instead of from sd_release() to avoid that
-	 * freezing and unfreezing the request queue affects user space I/O
-	 * in case multiple processes open a /dev/sd... node concurrently.
-	 */
-	blk_mq_freeze_queue(q);
-	blk_mq_unfreeze_queue(q);
-	disk->private_data = NULL;
-	put_disk(disk);
-	put_device(&sdkp->device->sdev_gendev);
 	sd_zbc_release_disk(sdkp);
+	put_device(&sdkp->device->sdev_gendev);
+	free_opal_dev(sdkp->opal_dev);
 	kfree(sdkp);
 }

drivers/scsi/sd.h+8−4

@@ -68,9 +68,13 @@ enum { · 67 unmodified lines
68	};	68	};
69		69
70	struct scsi_disk {	70	struct scsi_disk {
71	struct scsi_driver driver; / always &sd_template */
72	struct scsi_device *device;	71	struct scsi_device *device;
73	struct device dev;	72
		73	/*
		74	* disk_dev is used to show attributes in /sys/class/scsi_disk/,
		75	* but otherwise not really needed. Do not use for refcounting.
		76	*/
		77	struct device disk_dev;
74	struct gendisk *disk;	78	struct gendisk *disk;
75	struct opal_dev *opal_dev;	79	struct opal_dev *opal_dev;
76	#ifdef CONFIG_BLK_DEV_ZONED	80	#ifdef CONFIG_BLK_DEV_ZONED
@@ -127,11 +131,11 @@ struct scsi_disk { · 50 unmodified lines
127	unsigned security : 1;	131	unsigned security : 1;
128	unsigned ignore_medium_access_errors : 1;	132	unsigned ignore_medium_access_errors : 1;
129	};	133	};
130	#define to_scsi_disk(obj) container_of(obj,struct scsi_disk,dev)	134	#define to_scsi_disk(obj) container_of(obj, struct scsi_disk, disk_dev)
131		135
132	static inline struct scsi_disk scsi_disk(struct gendisk disk)	136	static inline struct scsi_disk scsi_disk(struct gendisk disk)
133	{	137	{
134	return container_of(disk->private_data, struct scsi_disk, driver);	138	return disk->private_data;
135	}	139	}
136		140
137	#define sd_printk(prefix, sdsk, fmt, a...) \	141	#define sd_printk(prefix, sdsk, fmt, a...) \

drivers/scsi/sr.c+29−102

 static struct lock_class_key sr_bio_compl_lkclass;
-/* This semaphore is used to mediate the 0->1 reference get in the
- * face of object destruction (i.e. we can't allow a get on an
- * object after last put) */
-static DEFINE_MUTEX(sr_ref_mutex);
 static int sr_open(struct cdrom_device_info *, int);
 static void sr_release(struct cdrom_device_info *);
 	.capability		= SR_CAPABILITIES,
 };
-static void sr_kref_release(struct kref *kref);
 static inline struct scsi_cd *scsi_cd(struct gendisk *disk)
 {
-	return container_of(disk->private_data, struct scsi_cd, driver);
+	return disk->private_data;
 }
 static int sr_runtime_suspend(struct device *dev)
 		return 0;
 }
-/*
- * The get and put routines for the struct scsi_cd.  Note this entity
- * has a scsi_device pointer and owns a reference to this.
- */
-static inline struct scsi_cd *scsi_cd_get(struct gendisk *disk)
-{
-	struct scsi_cd *cd = NULL;
-	mutex_lock(&sr_ref_mutex);
-	if (disk->private_data == NULL)
-		goto out;
-	cd = scsi_cd(disk);
-	kref_get(&cd->kref);
-	if (scsi_device_get(cd->device)) {
-		kref_put(&cd->kref, sr_kref_release);
-		cd = NULL;
-	}
- out:
-	mutex_unlock(&sr_ref_mutex);
-	return cd;
-}
-static void scsi_cd_put(struct scsi_cd *cd)
-{
-	struct scsi_device *sdev = cd->device;
-	mutex_lock(&sr_ref_mutex);
-	kref_put(&cd->kref, sr_kref_release);
-	scsi_device_put(sdev);
-	mutex_unlock(&sr_ref_mutex);
-}
 static unsigned int sr_get_events(struct scsi_device *sdev)
 {
 	u8 buf[8];
 static int sr_block_open(struct block_device *bdev, fmode_t mode)
 {
-	struct scsi_cd *cd;
+	struct scsi_cd *cd = scsi_cd(bdev->bd_disk);
-	struct scsi_device *sdev;
+	struct scsi_device *sdev = cd->device;
-	int ret = -ENXIO;
+	int ret;
-	cd = scsi_cd_get(bdev->bd_disk);
+	if (scsi_device_get(cd->device))
-	if (!cd)
+		return -ENXIO;
-		goto out;
-	sdev = cd->device;
 	scsi_autopm_get_device(sdev);
 	if (bdev_check_media_change(bdev))
 		sr_revalidate_disk(cd);
 	scsi_autopm_put_device(sdev);
 	if (ret)
-		scsi_cd_put(cd);
+		scsi_device_put(cd->device);
-out:
 	return ret;
 }
 	cdrom_release(&cd->cdi, mode);
 	mutex_unlock(&cd->lock);
-	scsi_cd_put(cd);
+	scsi_device_put(cd->device);
 }
 static int sr_block_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd,
 static unsigned int sr_block_check_events(struct gendisk *disk,
 					  unsigned int clearing)
 {
-	unsigned int ret = 0;
+	struct scsi_cd *cd = disk->private_data;
-	struct scsi_cd *cd;
-	cd = scsi_cd_get(disk);
+	if (atomic_read(&cd->device->disk_events_disable_depth))
-	if (!cd)
 		return 0;
+	return cdrom_check_events(&cd->cdi, clearing);
+}
-	if (!atomic_read(&cd->device->disk_events_disable_depth))
+static void sr_free_disk(struct gendisk *disk)
-		ret = cdrom_check_events(&cd->cdi, clearing);
+{
+	struct scsi_cd *cd = disk->private_data;
-	scsi_cd_put(cd);
+	spin_lock(&sr_index_lock);
-	return ret;
+	clear_bit(MINOR(disk_devt(disk)), sr_index_bits);
+	spin_unlock(&sr_index_lock);
+	unregister_cdrom(&cd->cdi);
+	mutex_destroy(&cd->lock);
+	kfree(cd);
 }
 static const struct block_device_operations sr_bdops =
 	.ioctl		= sr_block_ioctl,
 	.compat_ioctl	= blkdev_compat_ptr_ioctl,
 	.check_events	= sr_block_check_events,
+	.free_disk	= sr_free_disk,
 };
 static int sr_open(struct cdrom_device_info *cdi, int purpose)
 	if (!cd)
 		goto fail;
-	kref_init(&cd->kref);
 	disk = __alloc_disk_node(sdev->request_queue, NUMA_NO_NODE,
 				 &sr_bio_compl_lkclass);
 	if (!disk)
 	cd->device = sdev;
 	cd->disk = disk;
-	cd->driver = &sr_template;
 	cd->capacity = 0x1fffff;
 	cd->device->changed = 1;	/* force recheck CD type */
 	cd->media_present = 1;
 	sr_vendor_init(cd);
 	set_capacity(disk, cd->capacity);
-	disk->private_data = &cd->driver;
+	disk->private_data = cd;
 	if (register_cdrom(disk, &cd->cdi))
 		goto fail_minor;
 	sr_revalidate_disk(cd);
 	error = device_add_disk(&sdev->sdev_gendev, disk, NULL);
-	if (error) {
+	if (error)
-		kref_put(&cd->kref, sr_kref_release);
+		goto unregister_cdrom;
-		goto fail;
-	}
 	sdev_printk(KERN_DEBUG, sdev,
 		    "Attached scsi CD-ROM %s\n", cd->cdi.name);
 	return 0;
+unregister_cdrom:
+	unregister_cdrom(&cd->cdi);
 fail_minor:
 	spin_lock(&sr_index_lock);
 	clear_bit(minor, sr_index_bits);
 	return ret;
 }
-/**
- *	sr_kref_release - Called to free the scsi_cd structure
- *	@kref: pointer to embedded kref
- *
- *	sr_ref_mutex must be held entering this routine.  Because it is
- *	called on last put, you should always use the scsi_cd_get()
- *	scsi_cd_put() helpers which manipulate the semaphore directly
- *	and never do a direct kref_put().
- **/
-static void sr_kref_release(struct kref *kref)
-{
-	struct scsi_cd *cd = container_of(kref, struct scsi_cd, kref);
-	struct gendisk *disk = cd->disk;
-	spin_lock(&sr_index_lock);
-	clear_bit(MINOR(disk_devt(disk)), sr_index_bits);
-	spin_unlock(&sr_index_lock);
-	unregister_cdrom(&cd->cdi);
-	disk->private_data = NULL;
-	put_disk(disk);
-	mutex_destroy(&cd->lock);
-	kfree(cd);
-}
 static int sr_remove(struct device *dev)
 {
 	struct scsi_cd *cd = dev_get_drvdata(dev);
 	scsi_autopm_get_device(cd->device);
 	del_gendisk(cd->disk);
-	dev_set_drvdata(dev, NULL);
+	put_disk(cd->disk);
-	mutex_lock(&sr_ref_mutex);
-	kref_put(&cd->kref, sr_kref_release);
-	mutex_unlock(&sr_ref_mutex);
 	return 0;
 }

drivers/scsi/sr.h+0−6

 #ifndef _SR_H
 #define _SR_H
-#include <linux/genhd.h>
-#include <linux/kref.h>
 #include <linux/mutex.h>
 #define MAX_RETRIES	3
 typedef struct scsi_cd {
-	struct scsi_driver *driver;
 	unsigned capacity;	/* size in blocks                       */
 	struct scsi_device *device;
 	unsigned int vendor;	/* vendor code, see sr_vendor.c         */
 	struct cdrom_device_info cdi;
 	struct mutex lock;
-	/* We hold gendisk and scsi_device references on probe and use
-	 * the refs on this kref to decide when to release them */
-	struct kref kref;
 	struct gendisk *disk;
 } Scsi_CD;

drivers/scsi/st.c+0−1

@@ -4276,7 +4276,6 @@ static int st_probe(struct device *dev) · 4275 unmodified lines
4276	goto out_buffer_free;	4276	goto out_buffer_free;
4277	}	4277	}
4278	kref_init(&tpnt->kref);	4278	kref_init(&tpnt->kref);
4279	tpnt->driver = &st_template;
4280		4279
4281	tpnt->device = SDp;	4280	tpnt->device = SDp;
4282	if (SDp->scsi_level <= 2)	4281	if (SDp->scsi_level <= 2)

drivers/scsi/st.h+0−1

@@ -117,7 +117,6 @@ struct scsi_tape_stats { · 116 unmodified lines
117		117
118	/* The tape drive descriptor */	118	/* The tape drive descriptor */
119	struct scsi_tape {	119	struct scsi_tape {
120	struct scsi_driver *driver;
121	struct scsi_device *device;	120	struct scsi_device *device;
122	struct mutex lock; /* For serialization */	121	struct mutex lock; /* For serialization */
123	struct completion wait; /* For SCSI commands */	122	struct completion wait; /* For SCSI commands */

drivers/scsi/ufs/ufshpb.c+2−2

@@ -494,7 +494,7 @@ static struct ufshpb_req ufshpb_get_map_req(struct ufshpb_lu hpb, · 493 unmodified lines
494	if (!map_req)	494	if (!map_req)
495	return NULL;	495	return NULL;
496		496
497	bio = bio_alloc(GFP_KERNEL, hpb->pages_per_srgn);	497	bio = bio_alloc(NULL, hpb->pages_per_srgn, 0, GFP_KERNEL);
498	if (!bio) {	498	if (!bio) {
499	ufshpb_put_req(hpb, map_req);	499	ufshpb_put_req(hpb, map_req);
500	return NULL;	500	return NULL;
@@ -2050,7 +2050,7 @@ static int ufshpb_pre_req_mempool_init(struct ufshpb_lu *hpb) · 1549 unmodified lines
2050	INIT_LIST_HEAD(&pre_req->list_req);	2050	INIT_LIST_HEAD(&pre_req->list_req);
2051	pre_req->req = NULL;	2051	pre_req->req = NULL;
2052		2052
2053	pre_req->bio = bio_alloc(GFP_KERNEL, 1);	2053	pre_req->bio = bio_alloc(NULL, 1, 0, GFP_KERNEL);
2054	if (!pre_req->bio)	2054	if (!pre_req->bio)
2055	goto release_mem;	2055	goto release_mem;
2056		2056

drivers/target/target_core_iblock.c+4−8

 #include <linux/slab.h>
 #include <linux/spinlock.h>
 #include <linux/bio.h>
-#include <linux/genhd.h>
 #include <linux/file.h>
 #include <linux/module.h>
 #include <linux/scatterlist.h>
 	 * Only allocate as many vector entries as the bio code allows us to,
 	 * we'll loop later on until we have handled the whole request.
 	 */
-	bio = bio_alloc_bioset(GFP_NOIO, bio_max_segs(sg_num),
+	bio = bio_alloc_bioset(ib_dev->ibd_bd, bio_max_segs(sg_num), opf,
-				&ib_dev->ibd_bio_set);
+			       GFP_NOIO, &ib_dev->ibd_bio_set);
 	if (!bio) {
 		pr_err("Unable to allocate memory for bio\n");
 		return NULL;
 	}
-	bio_set_dev(bio, ib_dev->ibd_bd);
 	bio->bi_private = cmd;
 	bio->bi_end_io = &iblock_bio_done;
 	bio->bi_iter.bi_sector = lba;
-	bio->bi_opf = opf;
 	return bio;
 }
 	if (immed)
 		target_complete_cmd(cmd, SAM_STAT_GOOD);
-	bio = bio_alloc(GFP_KERNEL, 0);
+	bio = bio_alloc(ib_dev->ibd_bd, 0, REQ_OP_WRITE | REQ_PREFLUSH,
+			GFP_KERNEL);
 	bio->bi_end_io = iblock_end_io_flush;
-	bio_set_dev(bio, ib_dev->ibd_bd);
-	bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH;
 	if (!immed)
 		bio->bi_private = cmd;
 	submit_bio(bio);

drivers/target/target_core_pscsi.c+0−1

@@ -17,7 +17,6 @@ · 16 unmodified lines
17	#include <linux/blk_types.h>	17	#include <linux/blk_types.h>
18	#include <linux/slab.h>	18	#include <linux/slab.h>
19	#include <linux/spinlock.h>	19	#include <linux/spinlock.h>
20	#include <linux/genhd.h>
21	#include <linux/cdrom.h>	20	#include <linux/cdrom.h>
22	#include <linux/ratelimit.h>	21	#include <linux/ratelimit.h>
23	#include <linux/module.h>	22	#include <linux/module.h>

fs/btrfs/check-integrity.c+0−1

@@ -78,7 +78,6 @@ · 77 unmodified lines
78	#include <linux/sched.h>	78	#include <linux/sched.h>
79	#include <linux/slab.h>	79	#include <linux/slab.h>
80	#include <linux/mutex.h>	80	#include <linux/mutex.h>
81	#include <linux/genhd.h>
82	#include <linux/blkdev.h>	81	#include <linux/blkdev.h>
83	#include <linux/mm.h>	82	#include <linux/mm.h>
84	#include <linux/string.h>	83	#include <linux/string.h>

fs/btrfs/disk-io.c+4−6

 		 * to do I/O, so we don't lose the ability to do integrity
 		 * checking.
 		 */
-		bio = bio_alloc(GFP_NOFS, 1);
+		bio = bio_alloc(device->bdev, 1,
-		bio_set_dev(bio, device->bdev);
+				REQ_OP_WRITE | REQ_SYNC | REQ_META | REQ_PRIO,
+				GFP_NOFS);
 		bio->bi_iter.bi_sector = bytenr >> SECTOR_SHIFT;
 		bio->bi_private = device;
 		bio->bi_end_io = btrfs_end_super_write;
 		 * go down lazy and there's a short window where the on-disk
 		 * copies might still contain the older version.
 		 */
-		bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_META | REQ_PRIO;
 		if (i == 0 && !btrfs_test_opt(device->fs_info, NOBARRIER))
 			bio->bi_opf |= REQ_FUA;
 		return;
 #endif
-	bio_reset(bio);
+	bio_reset(bio, device->bdev, REQ_OP_WRITE | REQ_SYNC | REQ_PREFLUSH);
 	bio->bi_end_io = btrfs_end_empty_barrier;
-	bio_set_dev(bio, device->bdev);
-	bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_PREFLUSH;
 	init_completion(&device->flush_wait);
 	bio->bi_private = &device->flush_wait;

fs/btrfs/extent_io.c+3−3

@@ -3143,7 +3143,7 @@ struct bio *btrfs_bio_alloc(unsigned int nr_iovecs) · 3142 unmodified lines
3143	struct bio *bio;	3143	struct bio *bio;
3144		3144
3145	ASSERT(0 < nr_iovecs && nr_iovecs <= BIO_MAX_VECS);	3145	ASSERT(0 < nr_iovecs && nr_iovecs <= BIO_MAX_VECS);
3146	bio = bio_alloc_bioset(GFP_NOFS, nr_iovecs, &btrfs_bioset);	3146	bio = bio_alloc_bioset(NULL, nr_iovecs, 0, GFP_NOFS, &btrfs_bioset);
3147	btrfs_bio_init(btrfs_bio(bio));	3147	btrfs_bio_init(btrfs_bio(bio));
3148	return bio;	3148	return bio;
3149	}	3149	}
@@ -3154,7 +3154,7 @@ struct bio btrfs_bio_clone(struct bio bio) · 4 unmodified lines
3154	struct bio *new;	3154	struct bio *new;
3155		3155
3156	/* Bio allocation backed by a bioset does not fail */	3156	/* Bio allocation backed by a bioset does not fail */
3157	new = bio_clone_fast(bio, GFP_NOFS, &btrfs_bioset);	3157	new = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOFS, &btrfs_bioset);
3158	bbio = btrfs_bio(new);	3158	bbio = btrfs_bio(new);
3159	btrfs_bio_init(bbio);	3159	btrfs_bio_init(bbio);
3160	bbio->iter = bio->bi_iter;	3160	bbio->iter = bio->bi_iter;
@@ -3169,7 +3169,7 @@ struct bio btrfs_bio_clone_partial(struct bio orig, u64 offset, u64 size) · 8 unmodified lines
3169	ASSERT(offset <= UINT_MAX && size <= UINT_MAX);	3169	ASSERT(offset <= UINT_MAX && size <= UINT_MAX);
3170		3170
3171	/* this will never fail when it's backed by a bioset */	3171	/* this will never fail when it's backed by a bioset */
3172	bio = bio_clone_fast(orig, GFP_NOFS, &btrfs_bioset);	3172	bio = bio_alloc_clone(orig->bi_bdev, orig, GFP_NOFS, &btrfs_bioset);
3173	ASSERT(bio);	3173	ASSERT(bio);
3174		3174
3175	bbio = btrfs_bio(bio);	3175	bbio = btrfs_bio(bio);

fs/buffer.c+6−8

 	if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE))
 		clear_buffer_write_io_error(bh);
-	bio = bio_alloc(GFP_NOIO, 1);
+	if (buffer_meta(bh))
+		op_flags |= REQ_META;
+	if (buffer_prio(bh))
+		op_flags |= REQ_PRIO;
+	bio = bio_alloc(bh->b_bdev, 1, op | op_flags, GFP_NOIO);
 	fscrypt_set_bio_crypt_ctx_bh(bio, bh, GFP_NOIO);
 	bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
-	bio_set_dev(bio, bh->b_bdev);
 	bio->bi_write_hint = write_hint;
 	bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh));
 	bio->bi_end_io = end_bio_bh_io_sync;
 	bio->bi_private = bh;
-	if (buffer_meta(bh))
-		op_flags |= REQ_META;
-	if (buffer_prio(bh))
-		op_flags |= REQ_PRIO;
-	bio_set_op_attrs(bio, op, op_flags);
 	/* Take care of bh's that straddle the end of the device */
 	guard_bio_eod(bio);

fs/crypto/bio.c+5−8

 	int num_pages = 0;
 	/* This always succeeds since __GFP_DIRECT_RECLAIM is set. */
-	bio = bio_alloc(GFP_NOFS, BIO_MAX_VECS);
+	bio = bio_alloc(inode->i_sb->s_bdev, BIO_MAX_VECS, REQ_OP_WRITE,
+			GFP_NOFS);
 	while (len) {
 		unsigned int blocks_this_page = min(len, blocks_per_page);
 		if (num_pages == 0) {
 			fscrypt_set_bio_crypt_ctx(bio, inode, lblk, GFP_NOFS);
-			bio_set_dev(bio, inode->i_sb->s_bdev);
 			bio->bi_iter.bi_sector =
 					pblk << (blockbits - SECTOR_SHIFT);
-			bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
 		}
 		ret = bio_add_page(bio, ZERO_PAGE(0), bytes_this_page, 0);
 		if (WARN_ON(ret != bytes_this_page)) {
 			err = submit_bio_wait(bio);
 			if (err)
 				goto out;
-			bio_reset(bio);
+			bio_reset(bio, inode->i_sb->s_bdev, REQ_OP_WRITE);
 			num_pages = 0;
 		}
 	}
 		return -EINVAL;
 	/* This always succeeds since __GFP_DIRECT_RECLAIM is set. */
-	bio = bio_alloc(GFP_NOFS, nr_pages);
+	bio = bio_alloc(inode->i_sb->s_bdev, nr_pages, REQ_OP_WRITE, GFP_NOFS);
 	do {
-		bio_set_dev(bio, inode->i_sb->s_bdev);
 		bio->bi_iter.bi_sector = pblk << (blockbits - 9);
-		bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
 		i = 0;
 		offset = 0;
 		err = submit_bio_wait(bio);
 		if (err)
 			goto out;
-		bio_reset(bio);
+		bio_reset(bio, inode->i_sb->s_bdev, REQ_OP_WRITE);
 	} while (len != 0);
 	err = 0;
 out:

fs/dax.c+0−1

@@ -11,7 +11,6 @@ · 10 unmodified lines
11	#include <linux/buffer_head.h>	11	#include <linux/buffer_head.h>
12	#include <linux/dax.h>	12	#include <linux/dax.h>
13	#include <linux/fs.h>	13	#include <linux/fs.h>
14	#include <linux/genhd.h>
15	#include <linux/highmem.h>	14	#include <linux/highmem.h>
16	#include <linux/memcontrol.h>	15	#include <linux/memcontrol.h>
17	#include <linux/mm.h>	16	#include <linux/mm.h>

fs/direct-io.c+1−4

 	 * bio_alloc() is guaranteed to return a bio when allowed to sleep and
 	 * we request a valid number of vectors.
 	 */
-	bio = bio_alloc(GFP_KERNEL, nr_vecs);
+	bio = bio_alloc(bdev, nr_vecs, dio->op | dio->op_flags, GFP_KERNEL);
-	bio_set_dev(bio, bdev);
 	bio->bi_iter.bi_sector = first_sector;
-	bio_set_op_attrs(bio, dio->op, dio->op_flags);
 	if (dio->is_async)
 		bio->bi_end_io = dio_bio_end_aio;
 	else

fs/erofs/zdata.c+2−3

 			}
 			if (!bio) {
-				bio = bio_alloc(GFP_NOIO, BIO_MAX_VECS);
+				bio = bio_alloc(mdev.m_bdev, BIO_MAX_VECS,
+						REQ_OP_READ, GFP_NOIO);
 				bio->bi_end_io = z_erofs_decompressqueue_endio;
-				bio_set_dev(bio, mdev.m_bdev);
 				last_bdev = mdev.m_bdev;
 				bio->bi_iter.bi_sector = (sector_t)cur <<
 					LOG_SECTORS_PER_BLOCK;
 				bio->bi_private = bi_private;
-				bio->bi_opf = REQ_OP_READ;
 				if (f->readahead)
 					bio->bi_opf |= REQ_RAHEAD;
 				++nr_bios;

fs/ext4/page-io.c+3−5

 {
 	ext4_io_end_t *io_end = bio->bi_private;
 	sector_t bi_sector = bio->bi_iter.bi_sector;
-	char b[BDEVNAME_SIZE];
-	if (WARN_ONCE(!io_end, "io_end is NULL: %s: sector %Lu len %u err %d\n",
+	if (WARN_ONCE(!io_end, "io_end is NULL: %pg: sector %Lu len %u err %d\n",
-		      bio_devname(bio, b),
+		      bio->bi_bdev,
 		      (long long) bio->bi_iter.bi_sector,
 		      (unsigned) bio_sectors(bio),
 		      bio->bi_status)) {
 	 * bio_alloc will _always_ be able to allocate a bio if
 	 * __GFP_DIRECT_RECLAIM is set, see comments for bio_alloc_bioset().
 	 */
-	bio = bio_alloc(GFP_NOIO, BIO_MAX_VECS);
+	bio = bio_alloc(bh->b_bdev, BIO_MAX_VECS, 0, GFP_NOIO);
 	fscrypt_set_bio_crypt_ctx_bh(bio, bh, GFP_NOIO);
 	bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
-	bio_set_dev(bio, bh->b_bdev);
 	bio->bi_end_io = ext4_end_bio;
 	bio->bi_private = ext4_get_io_end(io->io_end);
 	io->io_bio = bio;

fs/ext4/readpage.c+4−4

 			 * bio_alloc will _always_ be able to allocate a bio if
 			 * __GFP_DIRECT_RECLAIM is set, see bio_alloc_bioset().
 			 */
-			bio = bio_alloc(GFP_KERNEL, bio_max_segs(nr_pages));
+			bio = bio_alloc(bdev, bio_max_segs(nr_pages),
+					REQ_OP_READ, GFP_KERNEL);
 			fscrypt_set_bio_crypt_ctx(bio, inode, next_block,
 						  GFP_KERNEL);
 			ext4_set_bio_post_read_ctx(bio, inode, page->index);
-			bio_set_dev(bio, bdev);
 			bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9);
 			bio->bi_end_io = mpage_end_io;
-			bio_set_op_attrs(bio, REQ_OP_READ,
+			if (rac)
-						rac ? REQ_RAHEAD : 0);
+				bio->bi_opf |= REQ_RAHEAD;
 		}
 		length = first_hole << blkbits;

fs/f2fs/data.c+3−4

 	struct f2fs_sb_info *sbi = fio->sbi;
 	struct bio *bio;
-	bio = bio_alloc_bioset(GFP_NOIO, npages, &f2fs_bioset);
+	bio = bio_alloc_bioset(NULL, npages, 0, GFP_NOIO, &f2fs_bioset);
 	f2fs_target_device(sbi, fio->new_blkaddr, bio);
 	if (is_read_io(fio->op)) {
 	struct bio_post_read_ctx *ctx = NULL;
 	unsigned int post_read_steps = 0;
-	bio = bio_alloc_bioset(for_write ? GFP_NOIO : GFP_KERNEL,
+	bio = bio_alloc_bioset(NULL, bio_max_segs(nr_pages), REQ_OP_READ,
-			       bio_max_segs(nr_pages), &f2fs_bioset);
+			       for_write ? GFP_NOIO : GFP_KERNEL, &f2fs_bioset);
 	if (!bio)
 		return ERR_PTR(-ENOMEM);
 	f2fs_target_device(sbi, blkaddr, bio);
 	bio->bi_end_io = f2fs_read_end_io;
-	bio_set_op_attrs(bio, REQ_OP_READ, op_flag);
 	if (fscrypt_inode_uses_fs_layer_crypto(inode))
 		post_read_steps |= STEP_DECRYPT;

fs/fs-writeback.c+2−4

 			 * unplug, so get our IOs out the door before we
 			 * give up the CPU.
 			 */
-			if (current->plug)
+			blk_flush_plug(current->plug, false);
-				blk_flush_plug(current->plug, false);
 			cond_resched();
 		}
 	/*
 	 * If we are expecting writeback progress we must submit plugged IO.
 	 */
-	if (blk_needs_flush_plug(current))
+	blk_flush_plug(current->plug, true);
-		blk_flush_plug(current->plug, true);
 	rcu_read_lock();
 	list_for_each_entry_rcu(bdi, &bdi_list, bdi_list)

fs/gfs2/lops.c+3−5

 				      bio_end_io_t *end_io)
 {
 	struct super_block *sb = sdp->sd_vfs;
-	struct bio *bio = bio_alloc(GFP_NOIO, BIO_MAX_VECS);
+	struct bio *bio = bio_alloc(sb->s_bdev, BIO_MAX_VECS, 0, GFP_NOIO);
 	bio->bi_iter.bi_sector = blkno << sdp->sd_fsb2bb_shift;
-	bio_set_dev(bio, sb->s_bdev);
 	bio->bi_end_io = end_io;
 	bio->bi_private = sdp;
 {
 	struct bio *new;
-	new = bio_alloc(GFP_NOIO, nr_iovecs);
+	new = bio_alloc(prev->bi_bdev, nr_iovecs, prev->bi_opf, GFP_NOIO);
-	bio_copy_dev(new, prev);
+	bio_clone_blkg_association(new, prev);
 	new->bi_iter.bi_sector = bio_end_sector(prev);
-	new->bi_opf = prev->bi_opf;
 	new->bi_write_hint = prev->bi_write_hint;
 	bio_chain(new, prev);
 	submit_bio(prev);

fs/gfs2/meta_io.c+1−3

 		struct buffer_head *bh = *bhs;
 		struct bio *bio;
-		bio = bio_alloc(GFP_NOIO, num);
+		bio = bio_alloc(bh->b_bdev, num, op | op_flags, GFP_NOIO);
 		bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
-		bio_set_dev(bio, bh->b_bdev);
 		while (num > 0) {
 			bh = *bhs;
 			if (!bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh))) {
 			num--;
 		}
 		bio->bi_end_io = gfs2_meta_read_endio;
-		bio_set_op_attrs(bio, op, op_flags);
 		submit_bio(bio);
 	}
 }

fs/gfs2/ops_fstype.c+1−3

 	ClearPageDirty(page);
 	lock_page(page);
-	bio = bio_alloc(GFP_NOFS, 1);
+	bio = bio_alloc(sb->s_bdev, 1, REQ_OP_READ | REQ_META, GFP_NOFS);
 	bio->bi_iter.bi_sector = sector * (sb->s_blocksize >> 9);
-	bio_set_dev(bio, sb->s_bdev);
 	bio_add_page(bio, page, PAGE_SIZE, 0);
 	bio->bi_end_io = end_bio_io_page;
 	bio->bi_private = page;
-	bio_set_op_attrs(bio, REQ_OP_READ, REQ_META);
 	submit_bio(bio);
 	wait_on_page_locked(page);
 	bio_put(bio);

fs/gfs2/sys.c+1−1

@@ -15,7 +15,7 @@ · 14 unmodified lines
15	#include <linux/kobject.h>	15	#include <linux/kobject.h>
16	#include <linux/uaccess.h>	16	#include <linux/uaccess.h>
17	#include <linux/gfs2_ondisk.h>	17	#include <linux/gfs2_ondisk.h>
18	#include <linux/genhd.h>	18	#include <linux/blkdev.h>
19		19
20	#include "gfs2.h"	20	#include "gfs2.h"
21	#include "incore.h"	21	#include "incore.h"

fs/hfs/mdb.c+1−1

@@ -9,7 +9,7 @@ · 8 unmodified lines
9	*/	9	*/
10		10
11	#include <linux/cdrom.h>	11	#include <linux/cdrom.h>
12	#include <linux/genhd.h>	12	#include <linux/blkdev.h>
13	#include <linux/nls.h>	13	#include <linux/nls.h>
14	#include <linux/slab.h>	14	#include <linux/slab.h>
15		15

fs/hfsplus/wrapper.c+1−4

 #include <linux/fs.h>
 #include <linux/blkdev.h>
 #include <linux/cdrom.h>
-#include <linux/genhd.h>
 #include <asm/unaligned.h>
 #include "hfsplus_fs.h"
 	offset = start & (io_size - 1);
 	sector &= ~((io_size >> HFSPLUS_SECTOR_SHIFT) - 1);
-	bio = bio_alloc(GFP_NOIO, 1);
+	bio = bio_alloc(sb->s_bdev, 1, op | op_flags, GFP_NOIO);
 	bio->bi_iter.bi_sector = sector;
-	bio_set_dev(bio, sb->s_bdev);
-	bio_set_op_attrs(bio, op, op_flags);
 	if (op != WRITE && data)
 		*data = (u8 *)buf + offset;

fs/iomap/buffered-io.c+12−14

 		if (ctx->rac) /* same as readahead_gfp_mask */
 			gfp |= __GFP_NORETRY | __GFP_NOWARN;
-		ctx->bio = bio_alloc(gfp, bio_max_segs(nr_vecs));
+		ctx->bio = bio_alloc(iomap->bdev, bio_max_segs(nr_vecs),
+				     REQ_OP_READ, gfp);
 		/*
 		 * If the bio_alloc fails, try it again for a single page to
 		 * avoid having to deal with partial page reads.  This emulates
 		 * what do_mpage_readpage does.
 		 */
-		if (!ctx->bio)
+		if (!ctx->bio) {
-			ctx->bio = bio_alloc(orig_gfp, 1);
+			ctx->bio = bio_alloc(iomap->bdev, 1, REQ_OP_READ,
-		ctx->bio->bi_opf = REQ_OP_READ;
+					     orig_gfp);
+		}
 		if (ctx->rac)
 			ctx->bio->bi_opf |= REQ_RAHEAD;
 		ctx->bio->bi_iter.bi_sector = sector;
-		bio_set_dev(ctx->bio, iomap->bdev);
 		ctx->bio->bi_end_io = iomap_read_end_io;
 		bio_add_folio(ctx->bio, folio, plen, poff);
 	}
 	struct bio_vec bvec;
 	struct bio bio;
-	bio_init(&bio, &bvec, 1);
+	bio_init(&bio, iomap->bdev, &bvec, 1, REQ_OP_READ);
-	bio.bi_opf = REQ_OP_READ;
 	bio.bi_iter.bi_sector = iomap_sector(iomap, block_start);
-	bio_set_dev(&bio, iomap->bdev);
 	bio_add_folio(&bio, folio, plen, poff);
 	return submit_bio_wait(&bio);
 }
 	struct iomap_ioend *ioend;
 	struct bio *bio;
-	bio = bio_alloc_bioset(GFP_NOFS, BIO_MAX_VECS, &iomap_ioend_bioset);
+	bio = bio_alloc_bioset(wpc->iomap.bdev, BIO_MAX_VECS,
-	bio_set_dev(bio, wpc->iomap.bdev);
+			       REQ_OP_WRITE | wbc_to_write_flags(wbc),
+			       GFP_NOFS, &iomap_ioend_bioset);
 	bio->bi_iter.bi_sector = sector;
-	bio->bi_opf = REQ_OP_WRITE | wbc_to_write_flags(wbc);
 	bio->bi_write_hint = inode->i_write_hint;
 	wbc_init_bio(wbc, bio);
 {
 	struct bio *new;
-	new = bio_alloc(GFP_NOFS, BIO_MAX_VECS);
+	new = bio_alloc(prev->bi_bdev, BIO_MAX_VECS, prev->bi_opf, GFP_NOFS);
-	bio_copy_dev(new, prev);/* also copies over blkcg information */
+	bio_clone_blkg_association(new, prev);
 	new->bi_iter.bi_sector = bio_end_sector(prev);
-	new->bi_opf = prev->bi_opf;
 	new->bi_write_hint = prev->bi_write_hint;
 	bio_chain(prev, new);

fs/iomap/direct-io.c+2−6

 	int flags = REQ_SYNC | REQ_IDLE;
 	struct bio *bio;
-	bio = bio_alloc(GFP_KERNEL, 1);
+	bio = bio_alloc(iter->iomap.bdev, 1, REQ_OP_WRITE | flags, GFP_KERNEL);
-	bio_set_dev(bio, iter->iomap.bdev);
 	bio->bi_iter.bi_sector = iomap_sector(&iter->iomap, pos);
 	bio->bi_private = dio;
 	bio->bi_end_io = iomap_dio_bio_end_io;
 	get_page(page);
 	__bio_add_page(bio, page, len, 0);
-	bio_set_op_attrs(bio, REQ_OP_WRITE, flags);
 	iomap_dio_submit_bio(iter, dio, bio, pos);
 }
 			goto out;
 		}
-		bio = bio_alloc(GFP_KERNEL, nr_pages);
+		bio = bio_alloc(iomap->bdev, nr_pages, bio_opf, GFP_KERNEL);
-		bio_set_dev(bio, iomap->bdev);
 		bio->bi_iter.bi_sector = iomap_sector(iomap, pos);
 		bio->bi_write_hint = dio->iocb->ki_hint;
 		bio->bi_ioprio = dio->iocb->ki_ioprio;
 		bio->bi_private = dio;
 		bio->bi_end_io = iomap_dio_bio_end_io;
-		bio->bi_opf = bio_opf;
 		ret = bio_iov_iter_get_pages(bio, dio->submit.iter);
 		if (unlikely(ret)) {

fs/jfs/jfs_logmgr.c+2−9

 	bp->l_flag |= lbmREAD;
-	bio = bio_alloc(GFP_NOFS, 1);
+	bio = bio_alloc(log->bdev, 1, REQ_OP_READ, GFP_NOFS);
 	bio->bi_iter.bi_sector = bp->l_blkno << (log->l2bsize - 9);
-	bio_set_dev(bio, log->bdev);
 	bio_add_page(bio, bp->l_page, LOGPSIZE, bp->l_offset);
 	BUG_ON(bio->bi_iter.bi_size != LOGPSIZE);
 	bio->bi_end_io = lbmIODone;
 	bio->bi_private = bp;
-	bio->bi_opf = REQ_OP_READ;
 	/*check if journaling to disk has been disabled*/
 	if (log->no_integrity) {
 		bio->bi_iter.bi_size = 0;
 	jfs_info("lbmStartIO");
-	bio = bio_alloc(GFP_NOFS, 1);
+	bio = bio_alloc(log->bdev, 1, REQ_OP_WRITE | REQ_SYNC, GFP_NOFS);
 	bio->bi_iter.bi_sector = bp->l_blkno << (log->l2bsize - 9);
-	bio_set_dev(bio, log->bdev);
 	bio_add_page(bio, bp->l_page, LOGPSIZE, bp->l_offset);
 	BUG_ON(bio->bi_iter.bi_size != LOGPSIZE);
 	bio->bi_end_io = lbmIODone;
 	bio->bi_private = bp;
-	bio->bi_opf = REQ_OP_WRITE | REQ_SYNC;
 	/* check if journaling to disk has been disabled */
 	if (log->no_integrity) {

fs/jfs/jfs_metapage.c+3−6

 		}
 		len = min(xlen, (int)JFS_SBI(inode->i_sb)->nbperpage);
-		bio = bio_alloc(GFP_NOFS, 1);
+		bio = bio_alloc(inode->i_sb->s_bdev, 1, REQ_OP_WRITE, GFP_NOFS);
-		bio_set_dev(bio, inode->i_sb->s_bdev);
 		bio->bi_iter.bi_sector = pblock << (inode->i_blkbits - 9);
 		bio->bi_end_io = metapage_write_end_io;
 		bio->bi_private = page;
-		bio_set_op_attrs(bio, REQ_OP_WRITE, 0);
 		/* Don't call bio_add_page yet, we may add to this vec */
 		bio_offset = offset;
 			if (bio)
 				submit_bio(bio);
-			bio = bio_alloc(GFP_NOFS, 1);
+			bio = bio_alloc(inode->i_sb->s_bdev, 1, REQ_OP_READ,
-			bio_set_dev(bio, inode->i_sb->s_bdev);
+					GFP_NOFS);
 			bio->bi_iter.bi_sector =
 				pblock << (inode->i_blkbits - 9);
 			bio->bi_end_io = metapage_read_end_io;
 			bio->bi_private = page;
-			bio_set_op_attrs(bio, REQ_OP_READ, 0);
 			len = xlen << inode->i_blkbits;
 			offset = block_offset << inode->i_blkbits;
 			if (bio_add_page(bio, page, len, offset) < len)

fs/ksmbd/vfs.c+0−1

@@ -11,7 +11,6 @@ · 10 unmodified lines
11	#include <linux/writeback.h>	11	#include <linux/writeback.h>
12	#include <linux/xattr.h>	12	#include <linux/xattr.h>
13	#include <linux/falloc.h>	13	#include <linux/falloc.h>
14	#include <linux/genhd.h>
15	#include <linux/fsnotify.h>	14	#include <linux/fsnotify.h>
16	#include <linux/dcache.h>	15	#include <linux/dcache.h>
17	#include <linux/slab.h>	16	#include <linux/slab.h>

fs/mpage.c+5−29

 	return NULL;
 }
-static struct bio *
-mpage_alloc(struct block_device *bdev,
-		sector_t first_sector, int nr_vecs,
-		gfp_t gfp_flags)
-{
-	struct bio *bio;
-	/* Restrict the given (page cache) mask for slab allocations */
-	gfp_flags &= GFP_KERNEL;
-	bio = bio_alloc(gfp_flags, nr_vecs);
-	if (bio == NULL && (current->flags & PF_MEMALLOC)) {
-		while (!bio && (nr_vecs /= 2))
-			bio = bio_alloc(gfp_flags, nr_vecs);
-	}
-	if (bio) {
-		bio_set_dev(bio, bdev);
-		bio->bi_iter.bi_sector = first_sector;
-	}
-	return bio;
-}
 /*
  * support function for mpage_readahead.  The fs supplied get_block might
  * return an up to date buffer.  This is used to map that buffer into
 								page))
 				goto out;
 		}
-		args->bio = mpage_alloc(bdev, blocks[0] << (blkbits - 9),
+		args->bio = bio_alloc(bdev, bio_max_segs(args->nr_pages), 0,
-					bio_max_segs(args->nr_pages), gfp);
+				      gfp);
 		if (args->bio == NULL)
 			goto confused;
+		args->bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9);
 	}
 	length = first_hole << blkbits;
 								page, wbc))
 				goto out;
 		}
-		bio = mpage_alloc(bdev, blocks[0] << (blkbits - 9),
+		bio = bio_alloc(bdev, BIO_MAX_VECS, 0, GFP_NOFS);
-				BIO_MAX_VECS, GFP_NOFS|__GFP_HIGH);
+		bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9);
-		if (bio == NULL)
-			goto confused;
 		wbc_init_bio(wbc, bio);
 		bio->bi_write_hint = inode->i_write_hint;

fs/nfs/blocklayout/blocklayout.c+4−22

@@ -115,23 +115,6 @@ bl_submit_bio(struct bio *bio) · 114 unmodified lines
115	return NULL;	115	return NULL;
116	}	116	}
117		117
118	static struct bio *bl_alloc_init_bio(unsigned int npg,
119	struct block_device *bdev, sector_t disk_sector,
120	bio_end_io_t end_io, struct parallel_io *par)
121	{
122	struct bio *bio;
123
124	npg = bio_max_segs(npg);
125	bio = bio_alloc(GFP_NOIO, npg);
126	if (bio) {
127	bio->bi_iter.bi_sector = disk_sector;
128	bio_set_dev(bio, bdev);
129	bio->bi_end_io = end_io;
130	bio->bi_private = par;
131	}
132	return bio;
133	}
134
135	static bool offset_in_map(u64 offset, struct pnfs_block_dev_map *map)	118	static bool offset_in_map(u64 offset, struct pnfs_block_dev_map *map)
136	{	119	{
137	return offset >= map->start && offset < map->start + map->len;	120	return offset >= map->start && offset < map->start + map->len;
@@ -171,11 +154,10 @@ do_add_page_to_bio(struct bio *bio, int npg, int rw, sector_t isect, · 33 unmodified lines
171		154
172	retry:	155	retry:
173	if (!bio) {	156	if (!bio) {
174	bio = bl_alloc_init_bio(npg, map->bdev,	157	bio = bio_alloc(map->bdev, bio_max_segs(npg), rw, GFP_NOIO);
175	disk_addr >> SECTOR_SHIFT, end_io, par);	158	bio->bi_iter.bi_sector = disk_addr >> SECTOR_SHIFT;
176	if (!bio)	159	bio->bi_end_io = end_io;
177	return ERR_PTR(-ENOMEM);	160	bio->bi_private = par;
178	bio_set_op_attrs(bio, rw, 0);
179	}	161	}
180	if (bio_add_page(bio, page, len, offset) < len) {	162	if (bio_add_page(bio, page, len, offset) < len) {
181	bio = bl_submit_bio(bio);	163	bio = bl_submit_bio(bio);

fs/nfs/blocklayout/rpc_pipefs.c+0−1

@@ -27,7 +27,6 @@ · 26 unmodified lines
27	*/	27	*/
28		28
29	#include <linux/module.h>	29	#include <linux/module.h>
30	#include <linux/genhd.h>
31	#include <linux/blkdev.h>	30	#include <linux/blkdev.h>
32		31
33	#include "blocklayout.h"	32	#include "blocklayout.h"

fs/nfsd/blocklayout.c+0−1

@@ -4,7 +4,6 @@ · 3 unmodified lines
4	*/	4	*/
5	#include <linux/exportfs.h>	5	#include <linux/exportfs.h>
6	#include <linux/iomap.h>	6	#include <linux/iomap.h>
7	#include <linux/genhd.h>
8	#include <linux/slab.h>	7	#include <linux/slab.h>
9	#include <linux/pr.h>	8	#include <linux/pr.h>
10		9

fs/nilfs2/segbuf.c+4−27

@@ -371,29 +371,6 @@ static int nilfs_segbuf_submit_bio(struct nilfs_segment_buffer *segbuf, · 370 unmodified lines
371	return err;	371	return err;
372	}	372	}
373		373
374	/**
375	* nilfs_alloc_seg_bio - allocate a new bio for writing log
376	* @nilfs: nilfs object
377	* @start: start block number of the bio
378	* @nr_vecs: request size of page vector.
379	*
380	* Return Value: On success, pointer to the struct bio is returned.
381	* On error, NULL is returned.
382	*/
383	static struct bio nilfs_alloc_seg_bio(struct the_nilfs nilfs, sector_t start,
384	int nr_vecs)
385	{
386	struct bio *bio;
387
388	bio = bio_alloc(GFP_NOIO, nr_vecs);
389	if (likely(bio)) {
390	bio_set_dev(bio, nilfs->ns_bdev);
391	bio->bi_iter.bi_sector =
392	start << (nilfs->ns_blocksize_bits - 9);
393	}
394	return bio;
395	}
396
397	static void nilfs_segbuf_prepare_write(struct nilfs_segment_buffer *segbuf,	374	static void nilfs_segbuf_prepare_write(struct nilfs_segment_buffer *segbuf,
398	struct nilfs_write_info *wi)	375	struct nilfs_write_info *wi)
399	{	376	{
@@ -414,10 +391,10 @@ static int nilfs_segbuf_submit_bh(struct nilfs_segment_buffer *segbuf, · 14 unmodified lines
414	BUG_ON(wi->nr_vecs <= 0);	391	BUG_ON(wi->nr_vecs <= 0);
415	repeat:	392	repeat:
416	if (!wi->bio) {	393	if (!wi->bio) {
417	wi->bio = nilfs_alloc_seg_bio(wi->nilfs, wi->blocknr + wi->end,	394	wi->bio = bio_alloc(wi->nilfs->ns_bdev, wi->nr_vecs, 0,
418	wi->nr_vecs);	395	GFP_NOIO);
419	if (unlikely(!wi->bio))	396	wi->bio->bi_iter.bi_sector = (wi->blocknr + wi->end) <<
420	return -ENOMEM;	397	(wi->nilfs->ns_blocksize_bits - 9);
421	}	398	}
422		399
423	len = bio_add_page(wi->bio, bh->b_page, bh->b_size, bh_offset(bh));	400	len = bio_add_page(wi->bio, bh->b_page, bh->b_size, bh_offset(bh));

fs/ntfs3/fsntfs.c+6−30

@@ -1443,17 +1443,6 @@ int ntfs_write_bh(struct ntfs_sb_info sbi, struct NTFS_RECORD_HEADER rhdr, · 1442 unmodified lines
1443	return err;	1443	return err;
1444	}	1444	}
1445		1445
1446	static inline struct bio *ntfs_alloc_bio(u32 nr_vecs)
1447	{
1448	struct bio *bio = bio_alloc(GFP_NOFS \| __GFP_HIGH, nr_vecs);
1449
1450	if (!bio && (current->flags & PF_MEMALLOC)) {
1451	while (!bio && (nr_vecs /= 2))
1452	bio = bio_alloc(GFP_NOFS \| __GFP_HIGH, nr_vecs);
1453	}
1454	return bio;
1455	}
1456
1457	/*	1446	/*
1458	* ntfs_bio_pages - Read/write pages from/to disk.	1447	* ntfs_bio_pages - Read/write pages from/to disk.
1459	*/	1448	*/
@@ -1496,19 +1485,13 @@ int ntfs_bio_pages(struct ntfs_sb_info sbi, const struct runs_tree run, · 36 unmodified lines
1496	lbo = ((u64)lcn << cluster_bits) + off;	1485	lbo = ((u64)lcn << cluster_bits) + off;
1497	len = ((u64)clen << cluster_bits) - off;	1486	len = ((u64)clen << cluster_bits) - off;
1498	new_bio:	1487	new_bio:
1499	new = ntfs_alloc_bio(nr_pages - page_idx);	1488	new = bio_alloc(bdev, nr_pages - page_idx, op, GFP_NOFS);
1500	if (!new) {
1501	err = -ENOMEM;
1502	goto out;
1503	}
1504	if (bio) {	1489	if (bio) {
1505	bio_chain(bio, new);	1490	bio_chain(bio, new);
1506	submit_bio(bio);	1491	submit_bio(bio);
1507	}	1492	}
1508	bio = new;	1493	bio = new;
1509	bio_set_dev(bio, bdev);
1510	bio->bi_iter.bi_sector = lbo >> 9;	1494	bio->bi_iter.bi_sector = lbo >> 9;
1511	bio->bi_opf = op;
1512		1495
1513	while (len) {	1496	while (len) {
1514	off = vbo & (PAGE_SIZE - 1);	1497	off = vbo & (PAGE_SIZE - 1);
@@ -1599,18 +1582,12 @@ int ntfs_bio_fill_1(struct ntfs_sb_info sbi, const struct runs_tree run) · 84 unmodified lines
1599	lbo = (u64)lcn << cluster_bits;	1582	lbo = (u64)lcn << cluster_bits;
1600	len = (u64)clen << cluster_bits;	1583	len = (u64)clen << cluster_bits;
1601	new_bio:	1584	new_bio:
1602	new = ntfs_alloc_bio(BIO_MAX_VECS);	1585	new = bio_alloc(bdev, BIO_MAX_VECS, REQ_OP_WRITE, GFP_NOFS);
1603	if (!new) {
1604	err = -ENOMEM;
1605	break;
1606	}
1607	if (bio) {	1586	if (bio) {
1608	bio_chain(bio, new);	1587	bio_chain(bio, new);
1609	submit_bio(bio);	1588	submit_bio(bio);
1610	}	1589	}
1611	bio = new;	1590	bio = new;
1612	bio_set_dev(bio, bdev);
1613	bio->bi_opf = REQ_OP_WRITE;
1614	bio->bi_iter.bi_sector = lbo >> 9;	1591	bio->bi_iter.bi_sector = lbo >> 9;
1615		1592
1616	for (;;) {	1593	for (;;) {
@@ -1626,11 +1603,10 @@ new_bio: · 9 unmodified lines
1626	}	1603	}
1627	} while (run_get_entry(run, ++run_idx, NULL, &lcn, &clen));	1604	} while (run_get_entry(run, ++run_idx, NULL, &lcn, &clen));
1628		1605
1629	if (bio) {	1606	if (!err)
1630	if (!err)	1607	err = submit_bio_wait(bio);
1631	err = submit_bio_wait(bio);	1608	bio_put(bio);
1632	bio_put(bio);	1609
1633	}
1634	blk_finish_plug(&plug);	1610	blk_finish_plug(&plug);
1635	out:	1611	out:
1636	unlock_page(fill);	1612	unlock_page(fill);

fs/ocfs2/cluster/heartbeat.c+1−3

 	 * GFP_KERNEL that the local node can get fenced. It would be
 	 * nicest if we could pre-allocate these bios and avoid this
 	 * all together. */
-	bio = bio_alloc(GFP_ATOMIC, 16);
+	bio = bio_alloc(reg->hr_bdev, 16, op | op_flags, GFP_ATOMIC);
 	if (!bio) {
 		mlog(ML_ERROR, "Could not alloc slots BIO!\n");
 		bio = ERR_PTR(-ENOMEM);
 	/* Must put everything in 512 byte sectors for the bio... */
 	bio->bi_iter.bi_sector = (reg->hr_start_block + cs) << (bits - 9);
-	bio_set_dev(bio, reg->hr_bdev);
 	bio->bi_private = wc;
 	bio->bi_end_io = o2hb_bio_end_io;
-	bio_set_op_attrs(bio, op, op_flags);
 	vec_start = (cs << bits) % PAGE_SIZE;
 	while(cs < max_slots) {

fs/squashfs/block.c+6−5

 	int error, i;
 	struct bio *bio;
-	if (page_count <= BIO_MAX_VECS)
+	if (page_count <= BIO_MAX_VECS) {
-		bio = bio_alloc(GFP_NOIO, page_count);
+		bio = bio_alloc(sb->s_bdev, page_count, REQ_OP_READ, GFP_NOIO);
-	else
+	} else {
 		bio = bio_kmalloc(GFP_NOIO, page_count);
+		bio_set_dev(bio, sb->s_bdev);
+		bio->bi_opf = REQ_OP_READ;
+	}
 	if (!bio)
 		return -ENOMEM;
-	bio_set_dev(bio, sb->s_bdev);
-	bio->bi_opf = READ;
 	bio->bi_iter.bi_sector = block * (msblk->devblksize >> SECTOR_SHIFT);
 	for (i = 0; i < page_count; ++i) {

fs/xfs/xfs_bio_io.c+5−9

 		return;
 	}
-	bio_init(bio, NULL, 0);
+	bio_init(bio, bdev, NULL, 0, REQ_OP_WRITE | REQ_PREFLUSH | REQ_SYNC);
-	bio_set_dev(bio, bdev);
-	bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH | REQ_SYNC;
 	bio->bi_private = done;
 	bio->bi_end_io = xfs_flush_bdev_async_endio;
 	if (is_vmalloc && op == REQ_OP_WRITE)
 		flush_kernel_vmap_range(data, count);
-	bio = bio_alloc(GFP_KERNEL, bio_max_vecs(left));
+	bio = bio_alloc(bdev, bio_max_vecs(left), op | REQ_META | REQ_SYNC,
-	bio_set_dev(bio, bdev);
+			GFP_KERNEL);
 	bio->bi_iter.bi_sector = sector;
-	bio->bi_opf = op | REQ_META | REQ_SYNC;
 	do {
 		struct page	*page = kmem_to_page(data);
 		while (bio_add_page(bio, page, len, off) != len) {
 			struct bio	*prev = bio;
-			bio = bio_alloc(GFP_KERNEL, bio_max_vecs(left));
+			bio = bio_alloc(prev->bi_bdev, bio_max_vecs(left),
-			bio_copy_dev(bio, prev);
+					prev->bi_opf, GFP_KERNEL);
 			bio->bi_iter.bi_sector = bio_end_sector(prev);
-			bio->bi_opf = prev->bi_opf;
 			bio_chain(prev, bio);
 			submit_bio(prev);

fs/xfs/xfs_buf.c+1−3

 	atomic_inc(&bp->b_io_remaining);
 	nr_pages = bio_max_segs(total_nr_pages);
-	bio = bio_alloc(GFP_NOIO, nr_pages);
+	bio = bio_alloc(bp->b_target->bt_bdev, nr_pages, op, GFP_NOIO);
-	bio_set_dev(bio, bp->b_target->bt_bdev);
 	bio->bi_iter.bi_sector = sector;
 	bio->bi_end_io = xfs_buf_bio_end_io;
 	bio->bi_private = bp;
-	bio->bi_opf = op;
 	for (; size && nr_pages; nr_pages--, page_index++) {
 		int	rbytes, nbytes = PAGE_SIZE - offset;

fs/xfs/xfs_log.c+7−7

 		return;
 	}
-	bio_init(&iclog->ic_bio, iclog->ic_bvec, howmany(count, PAGE_SIZE));
-	bio_set_dev(&iclog->ic_bio, log->l_targ->bt_bdev);
-	iclog->ic_bio.bi_iter.bi_sector = log->l_logBBstart + bno;
-	iclog->ic_bio.bi_end_io = xlog_bio_end_io;
-	iclog->ic_bio.bi_private = iclog;
 	/*
 	 * We use REQ_SYNC | REQ_IDLE here to tell the block layer the are more
 	 * IOs coming immediately after this one. This prevents the block layer
 	 * writeback throttle from throttling log writes behind background
 	 * metadata writeback and causing priority inversions.
 	 */
-	iclog->ic_bio.bi_opf = REQ_OP_WRITE | REQ_META | REQ_SYNC | REQ_IDLE;
+	bio_init(&iclog->ic_bio, log->l_targ->bt_bdev, iclog->ic_bvec,
+		 howmany(count, PAGE_SIZE),
+		 REQ_OP_WRITE | REQ_META | REQ_SYNC | REQ_IDLE);
+	iclog->ic_bio.bi_iter.bi_sector = log->l_logBBstart + bno;
+	iclog->ic_bio.bi_end_io = xlog_bio_end_io;
+	iclog->ic_bio.bi_private = iclog;
 	if (iclog->ic_flags & XLOG_ICL_NEED_FLUSH) {
 		iclog->ic_bio.bi_opf |= REQ_PREFLUSH;
 		/*

fs/zonefs/super.c+3−6

 	if (!nr_pages)
 		return 0;
-	bio = bio_alloc(GFP_NOFS, nr_pages);
+	bio = bio_alloc(bdev, nr_pages,
-	bio_set_dev(bio, bdev);
+			REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE, GFP_NOFS);
 	bio->bi_iter.bi_sector = zi->i_zsector;
 	bio->bi_write_hint = iocb->ki_hint;
 	bio->bi_ioprio = iocb->ki_ioprio;
-	bio->bi_opf = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE;
 	if (iocb->ki_flags & IOCB_DSYNC)
 		bio->bi_opf |= REQ_FUA;
 	if (!page)
 		return -ENOMEM;
-	bio_init(&bio, &bio_vec, 1);
+	bio_init(&bio, sb->s_bdev, &bio_vec, 1, REQ_OP_READ);
 	bio.bi_iter.bi_sector = 0;
-	bio.bi_opf = REQ_OP_READ;
-	bio_set_dev(&bio, sb->s_bdev);
 	bio_add_page(&bio, page, PAGE_SIZE, 0);
 	ret = submit_bio_wait(&bio);

include/linux/bio.h+17−21

 extern int biovec_init_pool(mempool_t *pool, int pool_entries);
 extern int bioset_init_from_src(struct bio_set *bs, struct bio_set *src);
-struct bio *bio_alloc_bioset(gfp_t gfp, unsigned short nr_iovecs,
+struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs,
-		struct bio_set *bs);
+			     unsigned int opf, gfp_t gfp_mask,
-struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs,
+			     struct bio_set *bs);
-		struct bio_set *bs);
+struct bio *bio_alloc_kiocb(struct kiocb *kiocb, struct block_device *bdev,
+		unsigned short nr_vecs, unsigned int opf, struct bio_set *bs);
 struct bio *bio_kmalloc(gfp_t gfp_mask, unsigned short nr_iovecs);
 extern void bio_put(struct bio *);
-extern void __bio_clone_fast(struct bio *, struct bio *);
+struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src,
-extern struct bio *bio_clone_fast(struct bio *, gfp_t, struct bio_set *);
+		gfp_t gfp, struct bio_set *bs);
+int bio_init_clone(struct block_device *bdev, struct bio *bio,
+		struct bio *bio_src, gfp_t gfp);
 extern struct bio_set fs_bio_set;
-static inline struct bio *bio_alloc(gfp_t gfp_mask, unsigned short nr_iovecs)
+static inline struct bio *bio_alloc(struct block_device *bdev,
+		unsigned short nr_vecs, unsigned int opf, gfp_t gfp_mask)
 {
-	return bio_alloc_bioset(gfp_mask, nr_iovecs, &fs_bio_set);
+	return bio_alloc_bioset(bdev, nr_vecs, opf, gfp_mask, &fs_bio_set);
 }
 void submit_bio(struct bio *bio);
 struct request_queue;
 extern int submit_bio_wait(struct bio *bio);
-extern void bio_init(struct bio *bio, struct bio_vec *table,
+void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,
-		     unsigned short max_vecs);
+	      unsigned short max_vecs, unsigned int opf);
 extern void bio_uninit(struct bio *);
-extern void bio_reset(struct bio *);
+void bio_reset(struct bio *bio, struct block_device *bdev, unsigned int opf);
 void bio_chain(struct bio *, struct bio *);
 int bio_add_page(struct bio *, struct page *, unsigned len, unsigned off);
 		__bio_release_pages(bio, mark_dirty);
 }
-extern const char *bio_devname(struct bio *bio, char *buffer);
 #define bio_dev(bio) \
 	disk_devt((bio)->bi_bdev->bd_disk)
 	bio_associate_blkg(bio);
 }
-static inline void bio_copy_dev(struct bio *dst, struct bio *src)
-{
-	bio_clear_flag(dst, BIO_REMAPPED);
-	dst->bi_bdev = src->bi_bdev;
-	bio_clone_blkg_association(dst, src);
-}
 /*
  * BIO list management for use by remapping drivers (e.g. DM or MD) and loop.
  *
 		bio->bi_opf |= REQ_NOWAIT;
 }
-struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp);
+struct bio *blk_next_bio(struct bio *bio, struct block_device *bdev,
+		unsigned int nr_pages, unsigned int opf, gfp_t gfp);
 #endif /* __LINUX_BIO_H */

include/linux/blk-cgroup.h+5−456

@@ -25,14 +25,8 @@ · 24 unmodified lines
25	#include <linux/kthread.h>	25	#include <linux/kthread.h>
26	#include <linux/fs.h>	26	#include <linux/fs.h>
27		27
28	/* percpu_counter batch for blkg_[rw]stats, per-cpu drift doesn't matter */
29	#define BLKG_STAT_CPU_BATCH (INT_MAX / 2)
30
31	/* Max limits for throttle policy */
32	#define THROTL_IOPS_MAX UINT_MAX
33	#define FC_APPID_LEN 129	28	#define FC_APPID_LEN 129
34		29
35
36	#ifdef CONFIG_BLK_CGROUP	30	#ifdef CONFIG_BLK_CGROUP
37		31
38	enum blkg_iostat_type {	32	enum blkg_iostat_type {
@@ -44,6 +38,7 @@ enum blkg_iostat_type { · 5 unmodified lines
44	};	38	};
45		39
46	struct blkcg_gq;	40	struct blkcg_gq;
		41	struct blkg_policy_data;
47		42
48	struct blkcg {	43	struct blkcg {
49	struct cgroup_subsys_state css;	44	struct cgroup_subsys_state css;
@@ -76,36 +71,6 @@ struct blkg_iostat_set { · 26 unmodified lines
76	struct blkg_iostat last;	71	struct blkg_iostat last;
77	};	72	};
78		73
79	/*
80	* A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a
81	* request_queue (q). This is used by blkcg policies which need to track
82	* information per blkcg - q pair.
83	*
84	* There can be multiple active blkcg policies and each blkg:policy pair is
85	* represented by a blkg_policy_data which is allocated and freed by each
86	* policy's pd_alloc/free_fn() methods. A policy can allocate private data
87	* area by allocating larger data structure which embeds blkg_policy_data
88	* at the beginning.
89	*/
90	struct blkg_policy_data {
91	/* the blkg and policy id this per-policy data belongs to */
92	struct blkcg_gq *blkg;
93	int plid;
94	};
95
96	/*
97	* Policies that need to keep per-blkcg data which is independent from any
98	* request_queue associated to it should implement cpd_alloc/free_fn()
99	* methods. A policy can allocate private data area by allocating larger
100	* data structure which embeds blkcg_policy_data at the beginning.
101	* cpd_init() is invoked to let each policy handle per-blkcg data.
102	*/
103	struct blkcg_policy_data {
104	/* the blkcg and policy id this per-policy data belongs to */
105	struct blkcg *blkcg;
106	int plid;
107	};
108
109	/* association between a blk cgroup and a request queue */	74	/* association between a blk cgroup and a request queue */
110	struct blkcg_gq {	75	struct blkcg_gq {
111	/* Pointer to the associated request_queue */	76	/* Pointer to the associated request_queue */
@@ -141,120 +106,17 @@ struct blkcg_gq { · 29 unmodified lines
141	struct rcu_head rcu_head;	106	struct rcu_head rcu_head;
142	};	107	};
143		108
144	typedef struct blkcg_policy_data *(blkcg_pol_alloc_cpd_fn)(gfp_t gfp);
145	typedef void (blkcg_pol_init_cpd_fn)(struct blkcg_policy_data *cpd);
146	typedef void (blkcg_pol_free_cpd_fn)(struct blkcg_policy_data *cpd);
147	typedef void (blkcg_pol_bind_cpd_fn)(struct blkcg_policy_data *cpd);
148	typedef struct blkg_policy_data *(blkcg_pol_alloc_pd_fn)(gfp_t gfp,
149	struct request_queue q, struct blkcg blkcg);
150	typedef void (blkcg_pol_init_pd_fn)(struct blkg_policy_data *pd);
151	typedef void (blkcg_pol_online_pd_fn)(struct blkg_policy_data *pd);
152	typedef void (blkcg_pol_offline_pd_fn)(struct blkg_policy_data *pd);
153	typedef void (blkcg_pol_free_pd_fn)(struct blkg_policy_data *pd);
154	typedef void (blkcg_pol_reset_pd_stats_fn)(struct blkg_policy_data *pd);
155	typedef bool (blkcg_pol_stat_pd_fn)(struct blkg_policy_data *pd,
156	struct seq_file *s);
157
158	struct blkcg_policy {
159	int plid;
160	/* cgroup files for the policy */
161	struct cftype *dfl_cftypes;
162	struct cftype *legacy_cftypes;
163
164	/* operations */
165	blkcg_pol_alloc_cpd_fn *cpd_alloc_fn;
166	blkcg_pol_init_cpd_fn *cpd_init_fn;
167	blkcg_pol_free_cpd_fn *cpd_free_fn;
168	blkcg_pol_bind_cpd_fn *cpd_bind_fn;
169
170	blkcg_pol_alloc_pd_fn *pd_alloc_fn;
171	blkcg_pol_init_pd_fn *pd_init_fn;
172	blkcg_pol_online_pd_fn *pd_online_fn;
173	blkcg_pol_offline_pd_fn *pd_offline_fn;
174	blkcg_pol_free_pd_fn *pd_free_fn;
175	blkcg_pol_reset_pd_stats_fn *pd_reset_stats_fn;
176	blkcg_pol_stat_pd_fn *pd_stat_fn;
177	};
178
179	extern struct blkcg blkcg_root;
180	extern struct cgroup_subsys_state * const blkcg_root_css;	109	extern struct cgroup_subsys_state * const blkcg_root_css;
181	extern bool blkcg_debug_stats;
182
183	struct blkcg_gq blkg_lookup_slowpath(struct blkcg blkcg,
184	struct request_queue *q, bool update_hint);
185	int blkcg_init_queue(struct request_queue *q);
186	void blkcg_exit_queue(struct request_queue *q);
187
188	/* Blkio controller policy registration */
189	int blkcg_policy_register(struct blkcg_policy *pol);
190	void blkcg_policy_unregister(struct blkcg_policy *pol);
191	int blkcg_activate_policy(struct request_queue *q,
192	const struct blkcg_policy *pol);
193	void blkcg_deactivate_policy(struct request_queue *q,
194	const struct blkcg_policy *pol);
195
196	const char blkg_dev_name(struct blkcg_gq blkg);
197	void blkcg_print_blkgs(struct seq_file sf, struct blkcg blkcg,
198	u64 (prfill)(struct seq_file ,
199	struct blkg_policy_data *, int),
200	const struct blkcg_policy *pol, int data,
201	bool show_total);
202	u64 __blkg_prfill_u64(struct seq_file sf, struct blkg_policy_data pd, u64 v);
203
204	struct blkg_conf_ctx {
205	struct block_device *bdev;
206	struct blkcg_gq *blkg;
207	char *body;
208	};
209
210	struct block_device blkcg_conf_open_bdev(char *inputp);
211	int blkg_conf_prep(struct blkcg blkcg, const struct blkcg_policy pol,
212	char input, struct blkg_conf_ctx ctx);
213	void blkg_conf_finish(struct blkg_conf_ctx *ctx);
214		110
215	/**	111	void blkcg_destroy_blkgs(struct blkcg *blkcg);
216	* blkcg_css - find the current css	112	void blkcg_schedule_throttle(struct request_queue *q, bool use_memdelay);
217	*	113	void blkcg_maybe_throttle_current(void);
218	* Find the css associated with either the kthread or the current task.
219	* This may return a dying css, so it is up to the caller to use tryget logic
220	* to confirm it is alive and well.
221	*/
222	static inline struct cgroup_subsys_state *blkcg_css(void)
223	{
224	struct cgroup_subsys_state *css;
225
226	css = kthread_blkcg();
227	if (css)
228	return css;
229	return task_css(current, io_cgrp_id);
230	}
231		114
232	static inline struct blkcg css_to_blkcg(struct cgroup_subsys_state css)	115	static inline struct blkcg css_to_blkcg(struct cgroup_subsys_state css)
233	{	116	{
234	return css ? container_of(css, struct blkcg, css) : NULL;	117	return css ? container_of(css, struct blkcg, css) : NULL;
235	}	118	}
236		119
237	/**
238	* __bio_blkcg - internal, inconsistent version to get blkcg
239	*
240	* DO NOT USE.
241	* This function is inconsistent and consequently is dangerous to use. The
242	* first part of the function returns a blkcg where a reference is owned by the
243	* bio. This means it does not need to be rcu protected as it cannot go away
244	* with the bio owning a reference to it. However, the latter potentially gets
245	* it from task_css(). This can race against task migration and the cgroup
246	* dying. It is also semantically different as it must be called rcu protected
247	* and is susceptible to failure when trying to get a reference to it.
248	* Therefore, it is not ok to assume that *_get() will always succeed on the
249	* blkcg returned here.
250	*/
251	static inline struct blkcg __bio_blkcg(struct bio bio)
252	{
253	if (bio && bio->bi_blkg)
254	return bio->bi_blkg->blkcg;
255	return css_to_blkcg(blkcg_css());
256	}
257
258	/**	120	/**
259	* bio_blkcg - grab the blkcg associated with a bio	121	* bio_blkcg - grab the blkcg associated with a bio
260	* @bio: target bio	122	* @bio: target bio
@@ -290,22 +152,6 @@ static inline bool blk_cgroup_congested(void) · 29 unmodified lines
290	return ret;	152	return ret;
291	}	153	}
292		154
293	/**
294	* bio_issue_as_root_blkg - see if this bio needs to be issued as root blkg
295	* @return: true if this bio needs to be submitted with the root blkg context.
296	*
297	* In order to avoid priority inversions we sometimes need to issue a bio as if
298	* it were attached to the root blkg, and then backcharge to the actual owning
299	* blkg. The idea is we do bio_blkcg() to look up the actual context for the
300	* bio and attach the appropriate blkg to the bio. Then we call this helper and
301	* if it is true run with the root blkg for that queue and then do any
302	* backcharging to the originating cgroup once the io is complete.
303	*/
304	static inline bool bio_issue_as_root_blkg(struct bio *bio)
305	{
306	return (bio->bi_opf & (REQ_META \| REQ_SWAP)) != 0;
307	}
308
309	/**	155	/**
310	* blkcg_parent - get the parent of a blkcg	156	* blkcg_parent - get the parent of a blkcg
311	* @blkcg: blkcg of interest	157	* @blkcg: blkcg of interest
@@ -317,96 +163,6 @@ static inline struct blkcg blkcg_parent(struct blkcg blkcg) · 5 unmodified lines
317	return css_to_blkcg(blkcg->css.parent);	163	return css_to_blkcg(blkcg->css.parent);
318	}	164	}
319		165
320	/**
321	* __blkg_lookup - internal version of blkg_lookup()
322	* @blkcg: blkcg of interest
323	* @q: request_queue of interest
324	* @update_hint: whether to update lookup hint with the result or not
325	*
326	* This is internal version and shouldn't be used by policy
327	* implementations. Looks up blkgs for the @blkcg - @q pair regardless of
328	* @q's bypass state. If @update_hint is %true, the caller should be
329	* holding @q->queue_lock and lookup hint is updated on success.
330	*/
331	static inline struct blkcg_gq __blkg_lookup(struct blkcg blkcg,
332	struct request_queue *q,
333	bool update_hint)
334	{
335	struct blkcg_gq *blkg;
336
337	if (blkcg == &blkcg_root)
338	return q->root_blkg;
339
340	blkg = rcu_dereference(blkcg->blkg_hint);
341	if (blkg && blkg->q == q)
342	return blkg;
343
344	return blkg_lookup_slowpath(blkcg, q, update_hint);
345	}
346
347	/**
348	* blkg_lookup - lookup blkg for the specified blkcg - q pair
349	* @blkcg: blkcg of interest
350	* @q: request_queue of interest
351	*
352	* Lookup blkg for the @blkcg - @q pair. This function should be called
353	* under RCU read lock.
354	*/
355	static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg,
356	struct request_queue *q)
357	{
358	WARN_ON_ONCE(!rcu_read_lock_held());
359	return __blkg_lookup(blkcg, q, false);
360	}
361
362	/**
363	* blk_queue_root_blkg - return blkg for the (blkcg_root, @q) pair
364	* @q: request_queue of interest
365	*
366	* Lookup blkg for @q at the root level. See also blkg_lookup().
367	*/
368	static inline struct blkcg_gq blk_queue_root_blkg(struct request_queue q)
369	{
370	return q->root_blkg;
371	}
372
373	/**
374	* blkg_to_pdata - get policy private data
375	* @blkg: blkg of interest
376	* @pol: policy of interest
377	*
378	* Return pointer to private data associated with the @blkg-@pol pair.
379	*/
380	static inline struct blkg_policy_data blkg_to_pd(struct blkcg_gq blkg,
381	struct blkcg_policy *pol)
382	{
383	return blkg ? blkg->pd[pol->plid] : NULL;
384	}
385
386	static inline struct blkcg_policy_data blkcg_to_cpd(struct blkcg blkcg,
387	struct blkcg_policy *pol)
388	{
389	return blkcg ? blkcg->cpd[pol->plid] : NULL;
390	}
391
392	/**
393	* pdata_to_blkg - get blkg associated with policy private data
394	* @pd: policy private data of interest
395	*
396	* @pd is policy private data. Determine the blkg it's associated with.
397	*/
398	static inline struct blkcg_gq pd_to_blkg(struct blkg_policy_data pd)
399	{
400	return pd ? pd->blkg : NULL;
401	}
402
403	static inline struct blkcg cpd_to_blkcg(struct blkcg_policy_data cpd)
404	{
405	return cpd ? cpd->blkcg : NULL;
406	}
407
408	extern void blkcg_destroy_blkgs(struct blkcg *blkcg);
409
410	/**	166	/**
411	* blkcg_pin_online - pin online state	167	* blkcg_pin_online - pin online state
412	* @blkcg: blkcg of interest	168	* @blkcg: blkcg of interest
@@ -439,231 +195,24 @@ static inline void blkcg_unpin_online(struct blkcg *blkcg) · 26 unmodified lines
439	} while (blkcg);	195	} while (blkcg);
440	}	196	}
441		197
442	/**
443	* blkg_path - format cgroup path of blkg
444	* @blkg: blkg of interest
445	* @buf: target buffer
446	* @buflen: target buffer length
447	*
448	* Format the path of the cgroup of @blkg into @buf.
449	*/
450	static inline int blkg_path(struct blkcg_gq blkg, char buf, int buflen)
451	{
452	return cgroup_path(blkg->blkcg->css.cgroup, buf, buflen);
453	}
454
455	/**
456	* blkg_get - get a blkg reference
457	* @blkg: blkg to get
458	*
459	* The caller should be holding an existing reference.
460	*/
461	static inline void blkg_get(struct blkcg_gq *blkg)
462	{
463	percpu_ref_get(&blkg->refcnt);
464	}
465
466	/**
467	* blkg_tryget - try and get a blkg reference
468	* @blkg: blkg to get
469	*
470	* This is for use when doing an RCU lookup of the blkg. We may be in the midst
471	* of freeing this blkg, so we can only use it if the refcnt is not zero.
472	*/
473	static inline bool blkg_tryget(struct blkcg_gq *blkg)
474	{
475	return blkg && percpu_ref_tryget(&blkg->refcnt);
476	}
477
478	/**
479	* blkg_put - put a blkg reference
480	* @blkg: blkg to put
481	*/
482	static inline void blkg_put(struct blkcg_gq *blkg)
483	{
484	percpu_ref_put(&blkg->refcnt);
485	}
486
487	/**
488	* blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants
489	* @d_blkg: loop cursor pointing to the current descendant
490	* @pos_css: used for iteration
491	* @p_blkg: target blkg to walk descendants of
492	*
493	* Walk @c_blkg through the descendants of @p_blkg. Must be used with RCU
494	* read locked. If called under either blkcg or queue lock, the iteration
495	* is guaranteed to include all and only online blkgs. The caller may
496	* update @pos_css by calling css_rightmost_descendant() to skip subtree.
497	* @p_blkg is included in the iteration and the first node to be visited.
498	*/
499	#define blkg_for_each_descendant_pre(d_blkg, pos_css, p_blkg) \
500	css_for_each_descendant_pre((pos_css), &(p_blkg)->blkcg->css) \
501	if (((d_blkg) = __blkg_lookup(css_to_blkcg(pos_css), \
502	(p_blkg)->q, false)))
503
504	/**
505	* blkg_for_each_descendant_post - post-order walk of a blkg's descendants
506	* @d_blkg: loop cursor pointing to the current descendant
507	* @pos_css: used for iteration
508	* @p_blkg: target blkg to walk descendants of
509	*
510	* Similar to blkg_for_each_descendant_pre() but performs post-order
511	* traversal instead. Synchronization rules are the same. @p_blkg is
512	* included in the iteration and the last node to be visited.
513	*/
514	#define blkg_for_each_descendant_post(d_blkg, pos_css, p_blkg) \
515	css_for_each_descendant_post((pos_css), &(p_blkg)->blkcg->css) \
516	if (((d_blkg) = __blkg_lookup(css_to_blkcg(pos_css), \
517	(p_blkg)->q, false)))
518
519	bool __blkcg_punt_bio_submit(struct bio *bio);
520
521	static inline bool blkcg_punt_bio_submit(struct bio *bio)
522	{
523	if (bio->bi_opf & REQ_CGROUP_PUNT)
524	return __blkcg_punt_bio_submit(bio);
525	else
526	return false;
527	}
528
529	static inline void blkcg_bio_issue_init(struct bio *bio)
530	{
531	bio_issue_init(&bio->bi_issue, bio_sectors(bio));
532	}
533
534	static inline void blkcg_use_delay(struct blkcg_gq *blkg)
535	{
536	if (WARN_ON_ONCE(atomic_read(&blkg->use_delay) < 0))
537	return;
538	if (atomic_add_return(1, &blkg->use_delay) == 1)
539	atomic_inc(&blkg->blkcg->css.cgroup->congestion_count);
540	}
541
542	static inline int blkcg_unuse_delay(struct blkcg_gq *blkg)
543	{
544	int old = atomic_read(&blkg->use_delay);
545
546	if (WARN_ON_ONCE(old < 0))
547	return 0;
548	if (old == 0)
549	return 0;
550
551	/*
552	* We do this song and dance because we can race with somebody else
553	* adding or removing delay. If we just did an atomic_dec we'd end up
554	* negative and we'd already be in trouble. We need to subtract 1 and
555	* then check to see if we were the last delay so we can drop the
556	* congestion count on the cgroup.
557	*/
558	while (old) {
559	int cur = atomic_cmpxchg(&blkg->use_delay, old, old - 1);
560	if (cur == old)
561	break;
562	old = cur;
563	}
564
565	if (old == 0)
566	return 0;
567	if (old == 1)
568	atomic_dec(&blkg->blkcg->css.cgroup->congestion_count);
569	return 1;
570	}
571
572	/**
573	* blkcg_set_delay - Enable allocator delay mechanism with the specified delay amount
574	* @blkg: target blkg
575	* @delay: delay duration in nsecs
576	*
577	* When enabled with this function, the delay is not decayed and must be
578	* explicitly cleared with blkcg_clear_delay(). Must not be mixed with
579	* blkcg_[un]use_delay() and blkcg_add_delay() usages.
580	*/
581	static inline void blkcg_set_delay(struct blkcg_gq *blkg, u64 delay)
582	{
583	int old = atomic_read(&blkg->use_delay);
584
585	/* We only want 1 person setting the congestion count for this blkg. */
586	if (!old && atomic_cmpxchg(&blkg->use_delay, old, -1) == old)
587	atomic_inc(&blkg->blkcg->css.cgroup->congestion_count);
588
589	atomic64_set(&blkg->delay_nsec, delay);
590	}
591
592	/**
593	* blkcg_clear_delay - Disable allocator delay mechanism
594	* @blkg: target blkg
595	*
596	* Disable use_delay mechanism. See blkcg_set_delay().
597	*/
598	static inline void blkcg_clear_delay(struct blkcg_gq *blkg)
599	{
600	int old = atomic_read(&blkg->use_delay);
601
602	/* We only want 1 person clearing the congestion count for this blkg. */
603	if (old && atomic_cmpxchg(&blkg->use_delay, old, 0) == old)
604	atomic_dec(&blkg->blkcg->css.cgroup->congestion_count);
605	}
606
607	void blk_cgroup_bio_start(struct bio *bio);
608	void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta);
609	void blkcg_schedule_throttle(struct request_queue *q, bool use_memdelay);
610	void blkcg_maybe_throttle_current(void);
611	#else /* CONFIG_BLK_CGROUP */	198	#else /* CONFIG_BLK_CGROUP */
612		199
613	struct blkcg {	200	struct blkcg {
614	};	201	};
615		202
616	struct blkg_policy_data {
617	};
618
619	struct blkcg_policy_data {
620	};
621
622	struct blkcg_gq {	203	struct blkcg_gq {
623	};	204	};
624		205
625	struct blkcg_policy {
626	};
627
628	#define blkcg_root_css ((struct cgroup_subsys_state *)ERR_PTR(-EINVAL))	206	#define blkcg_root_css ((struct cgroup_subsys_state *)ERR_PTR(-EINVAL))
629		207
630	static inline void blkcg_maybe_throttle_current(void) { }	208	static inline void blkcg_maybe_throttle_current(void) { }
631	static inline bool blk_cgroup_congested(void) { return false; }	209	static inline bool blk_cgroup_congested(void) { return false; }
632		210
633	#ifdef CONFIG_BLOCK	211	#ifdef CONFIG_BLOCK
634
635	static inline void blkcg_schedule_throttle(struct request_queue *q, bool use_memdelay) { }	212	static inline void blkcg_schedule_throttle(struct request_queue *q, bool use_memdelay) { }
636
637	static inline struct blkcg_gq blkg_lookup(struct blkcg blkcg, void *key) { return NULL; }
638	static inline struct blkcg_gq blk_queue_root_blkg(struct request_queue q)
639	{ return NULL; }
640	static inline int blkcg_init_queue(struct request_queue *q) { return 0; }
641	static inline void blkcg_exit_queue(struct request_queue *q) { }
642	static inline int blkcg_policy_register(struct blkcg_policy *pol) { return 0; }
643	static inline void blkcg_policy_unregister(struct blkcg_policy *pol) { }
644	static inline int blkcg_activate_policy(struct request_queue *q,
645	const struct blkcg_policy *pol) { return 0; }
646	static inline void blkcg_deactivate_policy(struct request_queue *q,
647	const struct blkcg_policy *pol) { }
648
649	static inline struct blkcg __bio_blkcg(struct bio bio) { return NULL; }
650	static inline struct blkcg bio_blkcg(struct bio bio) { return NULL; }	213	static inline struct blkcg bio_blkcg(struct bio bio) { return NULL; }
		214	#endif /* CONFIG_BLOCK */
651		215
652	static inline struct blkg_policy_data blkg_to_pd(struct blkcg_gq blkg,
653	struct blkcg_policy *pol) { return NULL; }
654	static inline struct blkcg_gq pd_to_blkg(struct blkg_policy_data pd) { return NULL; }
655	static inline char blkg_path(struct blkcg_gq blkg) { return NULL; }
656	static inline void blkg_get(struct blkcg_gq *blkg) { }
657	static inline void blkg_put(struct blkcg_gq *blkg) { }
658
659	static inline bool blkcg_punt_bio_submit(struct bio *bio) { return false; }
660	static inline void blkcg_bio_issue_init(struct bio *bio) { }
661	static inline void blk_cgroup_bio_start(struct bio *bio) { }
662
663	#define blk_queue_for_each_rl(rl, q) \
664	for ((rl) = &(q)->root_rl; (rl); (rl) = NULL)
665
666	#endif /* CONFIG_BLOCK */
667	#endif /* CONFIG_BLK_CGROUP */	216	#endif /* CONFIG_BLK_CGROUP */
668		217
669	#ifdef CONFIG_BLK_CGROUP_FC_APPID	218	#ifdef CONFIG_BLK_CGROUP_FC_APPID

include/linux/blk-mq.h+2−4

 }
 #define queue_for_each_hw_ctx(q, hctx, i)				\
-	for ((i) = 0; (i) < (q)->nr_hw_queues &&			\
+	xa_for_each(&(q)->hctx_table, (i), (hctx))
-	     ({ hctx = (q)->queue_hw_ctx[i]; 1; }); (i)++)
 #define hctx_for_each_ctx(hctx, ctx, i)					\
 	for ((i) = 0; (i) < (hctx)->nr_ctx &&				\
 		struct bio_set *bs, gfp_t gfp_mask,
 		int (*bio_ctr)(struct bio *, struct bio *, void *), void *data);
 void blk_rq_unprep_clone(struct request *rq);
-blk_status_t blk_insert_cloned_request(struct request_queue *q,
+blk_status_t blk_insert_cloned_request(struct request *rq);
-		struct request *rq);
 struct rq_map_data {
 	struct page **pages;

include/linux/blk_types.h+9−1

  */
 #define BLK_STS_ZONE_ACTIVE_RESOURCE	((__force blk_status_t)16)
+/*
+ * BLK_STS_OFFLINE is returned from the driver when the target device is offline
+ * or is being taken offline. This could help differentiate the case where a
+ * device is intentionally being shut down from a real I/O error.
+ */
+#define BLK_STS_OFFLINE		((__force blk_status_t)17)
 /**
  * blk_path_error - returns true if error may be path related
  * @error: status the request was completed with
 	BIO_TRACE_COMPLETION,	/* bio_endio() should trace the final completion
 				 * of this bio. */
 	BIO_CGROUP_ACCT,	/* has been accounted to a cgroup */
-	BIO_TRACKED,		/* set if bio goes through the rq_qos path */
+	BIO_QOS_THROTTLED,	/* bio went through rq_qos throttle path */
+	BIO_QOS_MERGED,		/* but went through rq_qos merge path */
 	BIO_REMAPPED,
 	BIO_ZONE_WRITE_LOCKED,	/* Owns a zoned device zone write lock */
 	BIO_PERCPU_CACHE,	/* can participate in per-cpu alloc cache */

include/linux/blkdev.h+279−17

@@ -1,9 +1,13 @@
1	/* SPDX-License-Identifier: GPL-2.0 */	1	/* SPDX-License-Identifier: GPL-2.0 */
		2	/*
		3	* Portions Copyright (C) 1992 Drew Eckhardt
		4	*/
2	#ifndef _LINUX_BLKDEV_H	5	#ifndef _LINUX_BLKDEV_H
3	#define _LINUX_BLKDEV_H	6	#define _LINUX_BLKDEV_H
4		7
5	#include <linux/sched.h>	8	#include <linux/types.h>
6	#include <linux/genhd.h>	9	#include <linux/blk_types.h>
		10	#include <linux/device.h>
7	#include <linux/list.h>	11	#include <linux/list.h>
8	#include <linux/llist.h>	12	#include <linux/llist.h>
9	#include <linux/minmax.h>	13	#include <linux/minmax.h>
@@ -12,11 +16,15 @@ · 2 unmodified lines
12	#include <linux/wait.h>	16	#include <linux/wait.h>
13	#include <linux/bio.h>	17	#include <linux/bio.h>
14	#include <linux/gfp.h>	18	#include <linux/gfp.h>
		19	#include <linux/kdev_t.h>
15	#include <linux/rcupdate.h>	20	#include <linux/rcupdate.h>
16	#include <linux/percpu-refcount.h>	21	#include <linux/percpu-refcount.h>
17	#include <linux/blkzoned.h>	22	#include <linux/blkzoned.h>
		23	#include <linux/sched.h>
18	#include <linux/sbitmap.h>	24	#include <linux/sbitmap.h>
19	#include <linux/srcu.h>	25	#include <linux/srcu.h>
		26	#include <linux/uuid.h>
		27	#include <linux/xarray.h>
20		28
21	struct module;	29	struct module;
22	struct request_queue;	30	struct request_queue;
@@ -33,6 +41,10 @@ struct blk_queue_stats; · 10 unmodified lines
33	struct blk_stat_callback;	41	struct blk_stat_callback;
34	struct blk_crypto_profile;	42	struct blk_crypto_profile;
35		43
		44	extern const struct device_type disk_type;
		45	extern struct device_type part_type;
		46	extern struct class block_class;
		47
36	/* Must be consistent with blk_mq_poll_stats_bkt() */	48	/* Must be consistent with blk_mq_poll_stats_bkt() */
37	#define BLK_MQ_POLL_STATS_BKTS 16	49	#define BLK_MQ_POLL_STATS_BKTS 16
38		50
@@ -45,6 +57,145 @@ struct blk_crypto_profile; · 6 unmodified lines
45	*/	57	*/
46	#define BLKCG_MAX_POLS 6	58	#define BLKCG_MAX_POLS 6
47		59
		60	#define DISK_MAX_PARTS 256
		61	#define DISK_NAME_LEN 32
		62
		63	#define PARTITION_META_INFO_VOLNAMELTH 64
		64	/*
		65	* Enough for the string representation of any kind of UUID plus NULL.
		66	* EFI UUID is 36 characters. MSDOS UUID is 11 characters.
		67	*/
		68	#define PARTITION_META_INFO_UUIDLTH (UUID_STRING_LEN + 1)
		69
		70	struct partition_meta_info {
		71	char uuid[PARTITION_META_INFO_UUIDLTH];
		72	u8 volname[PARTITION_META_INFO_VOLNAMELTH];
		73	};
		74
		75	/**
		76	* DOC: genhd capability flags
		77	*
		78	* ``GENHD_FL_REMOVABLE``: indicates that the block device gives access to
		79	* removable media. When set, the device remains present even when media is not
		80	* inserted. Shall not be set for devices which are removed entirely when the
		81	* media is removed.
		82	*
		83	* ``GENHD_FL_HIDDEN``: the block device is hidden; it doesn't produce events,
		84	* doesn't appear in sysfs, and can't be opened from userspace or using
		85	* blkdev_get*. Used for the underlying components of multipath devices.
		86	*
		87	* ``GENHD_FL_NO_PART``: partition support is disabled. The kernel will not
		88	* scan for partitions from add_disk, and users can't add partitions manually.
		89	*
		90	*/
		91	enum {
		92	GENHD_FL_REMOVABLE = 1 << 0,
		93	GENHD_FL_HIDDEN = 1 << 1,
		94	GENHD_FL_NO_PART = 1 << 2,
		95	};
		96
		97	enum {
		98	DISK_EVENT_MEDIA_CHANGE = 1 << 0, /* media changed */
		99	DISK_EVENT_EJECT_REQUEST = 1 << 1, /* eject requested */
		100	};
		101
		102	enum {
		103	/* Poll even if events_poll_msecs is unset */
		104	DISK_EVENT_FLAG_POLL = 1 << 0,
		105	/* Forward events to udev */
		106	DISK_EVENT_FLAG_UEVENT = 1 << 1,
		107	/* Block event polling when open for exclusive write */
		108	DISK_EVENT_FLAG_BLOCK_ON_EXCL_WRITE = 1 << 2,
		109	};
		110
		111	struct disk_events;
		112	struct badblocks;
		113
		114	struct blk_integrity {
		115	const struct blk_integrity_profile *profile;
		116	unsigned char flags;
		117	unsigned char tuple_size;
		118	unsigned char interval_exp;
		119	unsigned char tag_size;
		120	};
		121
		122	struct gendisk {
		123	/*
		124	* major/first_minor/minors should not be set by any new driver, the
		125	* block core will take care of allocating them automatically.
		126	*/
		127	int major;
		128	int first_minor;
		129	int minors;
		130
		131	char disk_name[DISK_NAME_LEN]; /* name of major driver */
		132
		133	unsigned short events; /* supported events */
		134	unsigned short event_flags; /* flags related to event processing */
		135
		136	struct xarray part_tbl;
		137	struct block_device *part0;
		138
		139	const struct block_device_operations *fops;
		140	struct request_queue *queue;
		141	void *private_data;
		142
		143	int flags;
		144	unsigned long state;
		145	#define GD_NEED_PART_SCAN 0
		146	#define GD_READ_ONLY 1
		147	#define GD_DEAD 2
		148	#define GD_NATIVE_CAPACITY 3
		149	#define GD_ADDED 4
		150
		151	struct mutex open_mutex; /* open/close mutex */
		152	unsigned open_partitions; /* number of open partitions */
		153
		154	struct backing_dev_info *bdi;
		155	struct kobject *slave_dir;
		156	#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED
		157	struct list_head slave_bdevs;
		158	#endif
		159	struct timer_rand_state *random;
		160	atomic_t sync_io; /* RAID */
		161	struct disk_events *ev;
		162	#ifdef CONFIG_BLK_DEV_INTEGRITY
		163	struct kobject integrity_kobj;
		164	#endif /* CONFIG_BLK_DEV_INTEGRITY */
		165	#if IS_ENABLED(CONFIG_CDROM)
		166	struct cdrom_device_info *cdi;
		167	#endif
		168	int node_id;
		169	struct badblocks *bb;
		170	struct lockdep_map lockdep_map;
		171	u64 diskseq;
		172	};
		173
		174	static inline bool disk_live(struct gendisk *disk)
		175	{
		176	return !inode_unhashed(disk->part0->bd_inode);
		177	}
		178
		179	/*
		180	* The gendisk is refcounted by the part0 block_device, and the bd_device
		181	* therein is also used for device model presentation in sysfs.
		182	*/
		183	#define dev_to_disk(device) \
		184	(dev_to_bdev(device)->bd_disk)
		185	#define disk_to_dev(disk) \
		186	(&((disk)->part0->bd_device))
		187
		188	#if IS_REACHABLE(CONFIG_CDROM)
		189	#define disk_to_cdi(disk) ((disk)->cdi)
		190	#else
		191	#define disk_to_cdi(disk) NULL
		192	#endif
		193
		194	static inline dev_t disk_devt(struct gendisk *disk)
		195	{
		196	return MKDEV(disk->major, disk->first_minor);
		197	}
		198
48	static inline int blk_validate_block_size(unsigned long bsize)	199	static inline int blk_validate_block_size(unsigned long bsize)
49	{	200	{
50	if (bsize < 512 \|\| bsize > PAGE_SIZE \|\| !is_power_of_2(bsize))	201	if (bsize < 512 \|\| bsize > PAGE_SIZE \|\| !is_power_of_2(bsize))
@@ -204,7 +355,7 @@ struct request_queue { · 153 unmodified lines
204	unsigned int queue_depth;	355	unsigned int queue_depth;
205		356
206	/* hw dispatch queues */	357	/* hw dispatch queues */
207	struct blk_mq_hw_ctx **queue_hw_ctx;	358	struct xarray hctx_table;
208	unsigned int nr_hw_queues;	359	unsigned int nr_hw_queues;
209		360
210	/*	361	/*
@@ -262,6 +413,7 @@ struct request_queue { · 51 unmodified lines
262		413
263	#ifdef CONFIG_BLK_INLINE_ENCRYPTION	414	#ifdef CONFIG_BLK_INLINE_ENCRYPTION
264	struct blk_crypto_profile *crypto_profile;	415	struct blk_crypto_profile *crypto_profile;
		416	struct kobject *crypto_kobject;
265	#endif	417	#endif
266		418
267	unsigned int rq_timeout;	419	unsigned int rq_timeout;
@@ -596,6 +748,118 @@ static inline unsigned int blk_queue_depth(struct request_queue *q) · 328 unmodified lines
596	#define for_each_bio(_bio) \	748	#define for_each_bio(_bio) \
597	for (; _bio; _bio = _bio->bi_next)	749	for (; _bio; _bio = _bio->bi_next)
598		750
		751	int __must_check device_add_disk(struct device parent, struct gendisk disk,
		752	const struct attribute_group **groups);
		753	static inline int __must_check add_disk(struct gendisk *disk)
		754	{
		755	return device_add_disk(NULL, disk, NULL);
		756	}
		757	void del_gendisk(struct gendisk *gp);
		758	void invalidate_disk(struct gendisk *disk);
		759	void set_disk_ro(struct gendisk *disk, bool read_only);
		760	void disk_uevent(struct gendisk *disk, enum kobject_action action);
		761
		762	static inline int get_disk_ro(struct gendisk *disk)
		763	{
		764	return disk->part0->bd_read_only \|\|
		765	test_bit(GD_READ_ONLY, &disk->state);
		766	}
		767
		768	static inline int bdev_read_only(struct block_device *bdev)
		769	{
		770	return bdev->bd_read_only \|\| get_disk_ro(bdev->bd_disk);
		771	}
		772
		773	bool set_capacity_and_notify(struct gendisk *disk, sector_t size);
		774	bool disk_force_media_change(struct gendisk *disk, unsigned int events);
		775
		776	void add_disk_randomness(struct gendisk *disk) __latent_entropy;
		777	void rand_initialize_disk(struct gendisk *disk);
		778
		779	static inline sector_t get_start_sect(struct block_device *bdev)
		780	{
		781	return bdev->bd_start_sect;
		782	}
		783
		784	static inline sector_t bdev_nr_sectors(struct block_device *bdev)
		785	{
		786	return bdev->bd_nr_sectors;
		787	}
		788
		789	static inline loff_t bdev_nr_bytes(struct block_device *bdev)
		790	{
		791	return (loff_t)bdev_nr_sectors(bdev) << SECTOR_SHIFT;
		792	}
		793
		794	static inline sector_t get_capacity(struct gendisk *disk)
		795	{
		796	return bdev_nr_sectors(disk->part0);
		797	}
		798
		799	static inline u64 sb_bdev_nr_blocks(struct super_block *sb)
		800	{
		801	return bdev_nr_sectors(sb->s_bdev) >>
		802	(sb->s_blocksize_bits - SECTOR_SHIFT);
		803	}
		804
		805	int bdev_disk_changed(struct gendisk *disk, bool invalidate);
		806
		807	struct gendisk __alloc_disk_node(struct request_queue q, int node_id,
		808	struct lock_class_key *lkclass);
		809	void put_disk(struct gendisk *disk);
		810	struct gendisk __blk_alloc_disk(int node, struct lock_class_key lkclass);
		811
		812	/**
		813	* blk_alloc_disk - allocate a gendisk structure
		814	* @node_id: numa node to allocate on
		815	*
		816	* Allocate and pre-initialize a gendisk structure for use with BIO based
		817	* drivers.
		818	*
		819	* Context: can sleep
		820	*/
		821	#define blk_alloc_disk(node_id) \
		822	({ \
		823	static struct lock_class_key __key; \
		824	\
		825	__blk_alloc_disk(node_id, &__key); \
		826	})
		827	void blk_cleanup_disk(struct gendisk *disk);
		828
		829	int __register_blkdev(unsigned int major, const char *name,
		830	void (*probe)(dev_t devt));
		831	#define register_blkdev(major, name) \
		832	__register_blkdev(major, name, NULL)
		833	void unregister_blkdev(unsigned int major, const char *name);
		834
		835	bool bdev_check_media_change(struct block_device *bdev);
		836	int __invalidate_device(struct block_device *bdev, bool kill_dirty);
		837	void set_capacity(struct gendisk *disk, sector_t size);
		838
		839	#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED
		840	int bd_link_disk_holder(struct block_device bdev, struct gendisk disk);
		841	void bd_unlink_disk_holder(struct block_device bdev, struct gendisk disk);
		842	int bd_register_pending_holders(struct gendisk *disk);
		843	#else
		844	static inline int bd_link_disk_holder(struct block_device *bdev,
		845	struct gendisk *disk)
		846	{
		847	return 0;
		848	}
		849	static inline void bd_unlink_disk_holder(struct block_device *bdev,
		850	struct gendisk *disk)
		851	{
		852	}
		853	static inline int bd_register_pending_holders(struct gendisk *disk)
		854	{
		855	return 0;
		856	}
		857	#endif /* CONFIG_BLOCK_HOLDER_DEPRECATED */
		858
		859	dev_t part_devt(struct gendisk *disk, u8 partno);
		860	void inc_diskseq(struct gendisk *disk);
		861	dev_t blk_lookup_devt(const char *name, int partno);
		862	void blk_request_module(dev_t devt);
599		863
600	extern int blk_register_queue(struct gendisk *disk);	864	extern int blk_register_queue(struct gendisk *disk);
601	extern void blk_unregister_queue(struct gendisk *disk);	865	extern void blk_unregister_queue(struct gendisk *disk);
@@ -792,14 +1056,11 @@ extern void blk_start_plug(struct blk_plug *); · 190 unmodified lines
792	extern void blk_start_plug_nr_ios(struct blk_plug *, unsigned short);	1056	extern void blk_start_plug_nr_ios(struct blk_plug *, unsigned short);
793	extern void blk_finish_plug(struct blk_plug *);	1057	extern void blk_finish_plug(struct blk_plug *);
794		1058
795	void blk_flush_plug(struct blk_plug *plug, bool from_schedule);	1059	void __blk_flush_plug(struct blk_plug *plug, bool from_schedule);
796		1060	static inline void blk_flush_plug(struct blk_plug *plug, bool async)
797	static inline bool blk_needs_flush_plug(struct task_struct *tsk)
798	{	1061	{
799	struct blk_plug *plug = tsk->plug;	1062	if (plug)
800		1063	__blk_flush_plug(plug, async);
801	return plug &&
802	(plug->mq_list \|\| !list_empty(&plug->cb_list));
803	}	1064	}
804		1065
805	int blkdev_issue_flush(struct block_device *bdev);	1066	int blkdev_issue_flush(struct block_device *bdev);
@@ -825,11 +1086,6 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) · 19 unmodified lines
825	{	1086	{
826	}	1087	}
827		1088
828	static inline bool blk_needs_flush_plug(struct task_struct *tsk)
829	{
830	return false;
831	}
832
833	static inline int blkdev_issue_flush(struct block_device *bdev)	1089	static inline int blkdev_issue_flush(struct block_device *bdev)
834	{	1090	{
835	return 0;	1091	return 0;
@@ -1211,6 +1467,7 @@ struct block_device_operations { · 375 unmodified lines
1211	void (unlock_native_capacity) (struct gendisk );	1467	void (unlock_native_capacity) (struct gendisk );
1212	int (getgeo)(struct block_device , struct hd_geometry *);	1468	int (getgeo)(struct block_device , struct hd_geometry *);
1213	int (set_read_only)(struct block_device bdev, bool ro);	1469	int (set_read_only)(struct block_device bdev, bool ro);
		1470	void (free_disk)(struct gendisk disk);
1214	/* this callback is with swap_lock and sometimes page table lock held */	1471	/* this callback is with swap_lock and sometimes page table lock held */
1215	void (swap_slot_free_notify) (struct block_device , unsigned long);	1472	void (swap_slot_free_notify) (struct block_device , unsigned long);
1216	int (report_zones)(struct gendisk , sector_t sector,	1473	int (report_zones)(struct gendisk , sector_t sector,
@@ -1267,7 +1524,7 @@ void bio_end_io_acct_remapped(struct bio *bio, unsigned long start_time, · 50 unmodified lines
1267	/**	1524	/**
1268	* bio_end_io_acct - end I/O accounting for bio based drivers	1525	* bio_end_io_acct - end I/O accounting for bio based drivers
1269	* @bio: bio to end account for	1526	* @bio: bio to end account for
1270	* @start: start time returned by bio_start_io_acct()	1527	* @start_time: start time returned by bio_start_io_acct()
1271	*/	1528	*/
1272	static inline void bio_end_io_acct(struct bio *bio, unsigned long start_time)	1529	static inline void bio_end_io_acct(struct bio *bio, unsigned long start_time)
1273	{	1530	{
@@ -1312,6 +1569,7 @@ void invalidate_bdev(struct block_device *bdev); · 38 unmodified lines
1312	int sync_blockdev(struct block_device *bdev);	1569	int sync_blockdev(struct block_device *bdev);
1313	int sync_blockdev_nowait(struct block_device *bdev);	1570	int sync_blockdev_nowait(struct block_device *bdev);
1314	void sync_bdevs(bool wait);	1571	void sync_bdevs(bool wait);
		1572	void printk_all_partitions(void);
1315	#else	1573	#else
1316	static inline void invalidate_bdev(struct block_device *bdev)	1574	static inline void invalidate_bdev(struct block_device *bdev)
1317	{	1575	{
@@ -1327,7 +1585,11 @@ static inline int sync_blockdev_nowait(struct block_device *bdev) · 9 unmodified lines
1327	static inline void sync_bdevs(bool wait)	1585	static inline void sync_bdevs(bool wait)
1328	{	1586	{
1329	}	1587	}
1330	#endif	1588	static inline void printk_all_partitions(void)
		1589	{
		1590	}
		1591	#endif /* CONFIG_BLOCK */
		1592
1331	int fsync_bdev(struct block_device *bdev);	1593	int fsync_bdev(struct block_device *bdev);
1332		1594
1333	int freeze_bdev(struct block_device *bdev);	1595	int freeze_bdev(struct block_device *bdev);

include/linux/genhd.h+0−291

@@ -1,291 +0,0 @@
1	/* SPDX-License-Identifier: GPL-2.0 */
2	#ifndef _LINUX_GENHD_H
3	#define _LINUX_GENHD_H
4
5	/*
6	* genhd.h Copyright (C) 1992 Drew Eckhardt
7	* Generic hard disk header file by
8	* Drew Eckhardt
9	*
10	* <drew@colorado.edu>
11	*/
12
13	#include <linux/types.h>
14	#include <linux/kdev_t.h>
15	#include <linux/uuid.h>
16	#include <linux/blk_types.h>
17	#include <linux/device.h>
18	#include <linux/xarray.h>
19
20	extern const struct device_type disk_type;
21	extern struct device_type part_type;
22	extern struct class block_class;
23
24	#define DISK_MAX_PARTS 256
25	#define DISK_NAME_LEN 32
26
27	#define PARTITION_META_INFO_VOLNAMELTH 64
28	/*
29	* Enough for the string representation of any kind of UUID plus NULL.
30	* EFI UUID is 36 characters. MSDOS UUID is 11 characters.
31	*/
32	#define PARTITION_META_INFO_UUIDLTH (UUID_STRING_LEN + 1)
33
34	struct partition_meta_info {
35	char uuid[PARTITION_META_INFO_UUIDLTH];
36	u8 volname[PARTITION_META_INFO_VOLNAMELTH];
37	};
38
39	/**
40	* DOC: genhd capability flags
41	*
42	* ``GENHD_FL_REMOVABLE``: indicates that the block device gives access to
43	* removable media. When set, the device remains present even when media is not
44	* inserted. Shall not be set for devices which are removed entirely when the
45	* media is removed.
46	*
47	* ``GENHD_FL_HIDDEN``: the block device is hidden; it doesn't produce events,
48	* doesn't appear in sysfs, and can't be opened from userspace or using
49	* blkdev_get*. Used for the underlying components of multipath devices.
50	*
51	* ``GENHD_FL_NO_PART``: partition support is disabled. The kernel will not
52	* scan for partitions from add_disk, and users can't add partitions manually.
53	*
54	*/
55	enum {
56	GENHD_FL_REMOVABLE = 1 << 0,
57	GENHD_FL_HIDDEN = 1 << 1,
58	GENHD_FL_NO_PART = 1 << 2,
59	};
60
61	enum {
62	DISK_EVENT_MEDIA_CHANGE = 1 << 0, /* media changed */
63	DISK_EVENT_EJECT_REQUEST = 1 << 1, /* eject requested */
64	};
65
66	enum {
67	/* Poll even if events_poll_msecs is unset */
68	DISK_EVENT_FLAG_POLL = 1 << 0,
69	/* Forward events to udev */
70	DISK_EVENT_FLAG_UEVENT = 1 << 1,
71	/* Block event polling when open for exclusive write */
72	DISK_EVENT_FLAG_BLOCK_ON_EXCL_WRITE = 1 << 2,
73	};
74
75	struct disk_events;
76	struct badblocks;
77
78	struct blk_integrity {
79	const struct blk_integrity_profile *profile;
80	unsigned char flags;
81	unsigned char tuple_size;
82	unsigned char interval_exp;
83	unsigned char tag_size;
84	};
85
86	struct gendisk {
87	/*
88	* major/first_minor/minors should not be set by any new driver, the
89	* block core will take care of allocating them automatically.
90	*/
91	int major;
92	int first_minor;
93	int minors;
94
95	char disk_name[DISK_NAME_LEN]; /* name of major driver */
96
97	unsigned short events; /* supported events */
98	unsigned short event_flags; /* flags related to event processing */
99
100	struct xarray part_tbl;
101	struct block_device *part0;
102
103	const struct block_device_operations *fops;
104	struct request_queue *queue;
105	void *private_data;
106
107	int flags;
108	unsigned long state;
109	#define GD_NEED_PART_SCAN 0
110	#define GD_READ_ONLY 1
111	#define GD_DEAD 2
112	#define GD_NATIVE_CAPACITY 3
113
114	struct mutex open_mutex; /* open/close mutex */
115	unsigned open_partitions; /* number of open partitions */
116
117	struct backing_dev_info *bdi;
118	struct kobject *slave_dir;
119	#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED
120	struct list_head slave_bdevs;
121	#endif
122	struct timer_rand_state *random;
123	atomic_t sync_io; /* RAID */
124	struct disk_events *ev;
125	#ifdef CONFIG_BLK_DEV_INTEGRITY
126	struct kobject integrity_kobj;
127	#endif /* CONFIG_BLK_DEV_INTEGRITY */
128	#if IS_ENABLED(CONFIG_CDROM)
129	struct cdrom_device_info *cdi;
130	#endif
131	int node_id;
132	struct badblocks *bb;
133	struct lockdep_map lockdep_map;
134	u64 diskseq;
135	};
136
137	static inline bool disk_live(struct gendisk *disk)
138	{
139	return !inode_unhashed(disk->part0->bd_inode);
140	}
141
142	/*
143	* The gendisk is refcounted by the part0 block_device, and the bd_device
144	* therein is also used for device model presentation in sysfs.
145	*/
146	#define dev_to_disk(device) \
147	(dev_to_bdev(device)->bd_disk)
148	#define disk_to_dev(disk) \
149	(&((disk)->part0->bd_device))
150
151	#if IS_REACHABLE(CONFIG_CDROM)
152	#define disk_to_cdi(disk) ((disk)->cdi)
153	#else
154	#define disk_to_cdi(disk) NULL
155	#endif
156
157	static inline dev_t disk_devt(struct gendisk *disk)
158	{
159	return MKDEV(disk->major, disk->first_minor);
160	}
161
162	void disk_uevent(struct gendisk *disk, enum kobject_action action);
163
164	/* block/genhd.c */
165	int __must_check device_add_disk(struct device parent, struct gendisk disk,
166	const struct attribute_group **groups);
167	static inline int __must_check add_disk(struct gendisk *disk)
168	{
169	return device_add_disk(NULL, disk, NULL);
170	}
171	extern void del_gendisk(struct gendisk *gp);
172
173	void invalidate_disk(struct gendisk *disk);
174
175	void set_disk_ro(struct gendisk *disk, bool read_only);
176
177	static inline int get_disk_ro(struct gendisk *disk)
178	{
179	return disk->part0->bd_read_only \|\|
180	test_bit(GD_READ_ONLY, &disk->state);
181	}
182
183	static inline int bdev_read_only(struct block_device *bdev)
184	{
185	return bdev->bd_read_only \|\| get_disk_ro(bdev->bd_disk);
186	}
187
188	extern void disk_block_events(struct gendisk *disk);
189	extern void disk_unblock_events(struct gendisk *disk);
190	extern void disk_flush_events(struct gendisk *disk, unsigned int mask);
191	bool set_capacity_and_notify(struct gendisk *disk, sector_t size);
192	bool disk_force_media_change(struct gendisk *disk, unsigned int events);
193
194	/* drivers/char/random.c */
195	extern void add_disk_randomness(struct gendisk *disk) __latent_entropy;
196	extern void rand_initialize_disk(struct gendisk *disk);
197
198	static inline sector_t get_start_sect(struct block_device *bdev)
199	{
200	return bdev->bd_start_sect;
201	}
202
203	static inline sector_t bdev_nr_sectors(struct block_device *bdev)
204	{
205	return bdev->bd_nr_sectors;
206	}
207
208	static inline loff_t bdev_nr_bytes(struct block_device *bdev)
209	{
210	return (loff_t)bdev_nr_sectors(bdev) << SECTOR_SHIFT;
211	}
212
213	static inline sector_t get_capacity(struct gendisk *disk)
214	{
215	return bdev_nr_sectors(disk->part0);
216	}
217
218	static inline u64 sb_bdev_nr_blocks(struct super_block *sb)
219	{
220	return bdev_nr_sectors(sb->s_bdev) >>
221	(sb->s_blocksize_bits - SECTOR_SHIFT);
222	}
223
224	int bdev_disk_changed(struct gendisk *disk, bool invalidate);
225	void blk_drop_partitions(struct gendisk *disk);
226
227	struct gendisk __alloc_disk_node(struct request_queue q, int node_id,
228	struct lock_class_key *lkclass);
229	extern void put_disk(struct gendisk *disk);
230	struct gendisk __blk_alloc_disk(int node, struct lock_class_key lkclass);
231
232	/**
233	* blk_alloc_disk - allocate a gendisk structure
234	* @node_id: numa node to allocate on
235	*
236	* Allocate and pre-initialize a gendisk structure for use with BIO based
237	* drivers.
238	*
239	* Context: can sleep
240	*/
241	#define blk_alloc_disk(node_id) \
242	({ \
243	static struct lock_class_key __key; \
244	\
245	__blk_alloc_disk(node_id, &__key); \
246	})
247	void blk_cleanup_disk(struct gendisk *disk);
248
249	int __register_blkdev(unsigned int major, const char *name,
250	void (*probe)(dev_t devt));
251	#define register_blkdev(major, name) \
252	__register_blkdev(major, name, NULL)
253	void unregister_blkdev(unsigned int major, const char *name);
254
255	bool bdev_check_media_change(struct block_device *bdev);
256	int __invalidate_device(struct block_device *bdev, bool kill_dirty);
257	void set_capacity(struct gendisk *disk, sector_t size);
258
259	#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED
260	int bd_link_disk_holder(struct block_device bdev, struct gendisk disk);
261	void bd_unlink_disk_holder(struct block_device bdev, struct gendisk disk);
262	int bd_register_pending_holders(struct gendisk *disk);
263	#else
264	static inline int bd_link_disk_holder(struct block_device *bdev,
265	struct gendisk *disk)
266	{
267	return 0;
268	}
269	static inline void bd_unlink_disk_holder(struct block_device *bdev,
270	struct gendisk *disk)
271	{
272	}
273	static inline int bd_register_pending_holders(struct gendisk *disk)
274	{
275	return 0;
276	}
277	#endif /* CONFIG_BLOCK_HOLDER_DEPRECATED */
278
279	dev_t part_devt(struct gendisk *disk, u8 partno);
280	void inc_diskseq(struct gendisk *disk);
281	dev_t blk_lookup_devt(const char *name, int partno);
282	void blk_request_module(dev_t devt);
283	#ifdef CONFIG_BLOCK
284	void printk_all_partitions(void);
285	#else /* CONFIG_BLOCK */
286	static inline void printk_all_partitions(void)
287	{
288	}
289	#endif /* CONFIG_BLOCK */
290
291	#endif /* _LINUX_GENHD_H */

include/linux/part_stat.h+1−1

@@ -2,7 +2,7 @@ · 1 unmodified line
2	#ifndef _LINUX_PART_STAT_H	2	#ifndef _LINUX_PART_STAT_H
3	#define _LINUX_PART_STAT_H	3	#define _LINUX_PART_STAT_H
4		4
5	#include <linux/genhd.h>	5	#include <linux/blkdev.h>
6	#include <asm/local.h>	6	#include <asm/local.h>
7		7
8	struct disk_stats {	8	struct disk_stats {

include/linux/sbitmap.h+14−37

  * struct sbitmap_word - Word in a &struct sbitmap.
  */
 struct sbitmap_word {
-	/**
-	 * @depth: Number of bits being used in @word/@cleared
-	 */
-	unsigned long depth;
 	/**
 	 * @word: word holding free bits
 	 */
-	unsigned long word ____cacheline_aligned_in_smp;
+	unsigned long word;
 	/**
 	 * @cleared: word holding cleared bits
 	/**
 	 * @min_shallow_depth: The minimum shallow depth which may be passed to
-	 * sbitmap_queue_get_shallow() or __sbitmap_queue_get_shallow().
+	 * sbitmap_queue_get_shallow()
 	 */
 	unsigned int min_shallow_depth;
 };
 int sbitmap_init_node(struct sbitmap *sb, unsigned int depth, int shift,
 		      gfp_t flags, int node, bool round_robin, bool alloc_hint);
+/* sbitmap internal helper */
+static inline unsigned int __map_depth(const struct sbitmap *sb, int index)
+{
+	if (index == sb->map_nr - 1)
+		return sb->depth - (index << sb->shift);
+	return 1U << sb->shift;
+}
 /**
  * sbitmap_free() - Free memory used by a &struct sbitmap.
  * @sb: Bitmap to free.
 	while (scanned < sb->depth) {
 		unsigned long word;
 		unsigned int depth = min_t(unsigned int,
-					   sb->map[index].depth - nr,
+					   __map_depth(sb, index) - nr,
 					   sb->depth - scanned);
 		scanned += depth;
 					unsigned int *offset);
 /**
- * __sbitmap_queue_get_shallow() - Try to allocate a free bit from a &struct
+ * sbitmap_queue_get_shallow() - Try to allocate a free bit from a &struct
  * sbitmap_queue, limiting the depth used from each word, with preemption
  * already disabled.
  * @sbq: Bitmap queue to allocate from.
  *
  * Return: Non-negative allocated bit number if successful, -1 otherwise.
  */
-int __sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
+int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
-				unsigned int shallow_depth);
+			      unsigned int shallow_depth);
 /**
  * sbitmap_queue_get() - Try to allocate a free bit from a &struct
 	return nr;
 }
-/**
- * sbitmap_queue_get_shallow() - Try to allocate a free bit from a &struct
- * sbitmap_queue, limiting the depth used from each word.
- * @sbq: Bitmap queue to allocate from.
- * @cpu: Output parameter; will contain the CPU we ran on (e.g., to be passed to
- *       sbitmap_queue_clear()).
- * @shallow_depth: The maximum number of bits to allocate from a single word.
- * See sbitmap_get_shallow().
- *
- * If you call this, make sure to call sbitmap_queue_min_shallow_depth() after
- * initializing @sbq.
- *
- * Return: Non-negative allocated bit number if successful, -1 otherwise.
- */
-static inline int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
-					    unsigned int *cpu,
-					    unsigned int shallow_depth)
-{
-	int nr;
-	*cpu = get_cpu();
-	nr = __sbitmap_queue_get_shallow(sbq, shallow_depth);
-	put_cpu();
-	return nr;
-}
 /**
  * sbitmap_queue_min_shallow_depth() - Inform a &struct sbitmap_queue of the
  * minimum shallow depth that will be used.

include/scsi/scsi_cmnd.h+0−9

 #include <scsi/scsi_request.h>
 struct Scsi_Host;
-struct scsi_driver;
 /*
  * MAX_COMMAND_SIZE is:
 	return cmd + 1;
 }
-/* make sure not to use it with passthrough commands */
-static inline struct scsi_driver *scsi_cmd_to_driver(struct scsi_cmnd *cmd)
-{
-	struct request *rq = scsi_cmd_to_rq(cmd);
-	return *(struct scsi_driver **)rq->q->disk->private_data;
-}
 void scsi_done(struct scsi_cmnd *cmd);
 extern void scsi_finish_command(struct scsi_cmnd *cmd);

include/scsi/scsi_driver.h+7−2

 #include <linux/blk_types.h>
 #include <linux/device.h>
+#include <scsi/scsi_cmnd.h>
 struct module;
 struct request;
-struct scsi_cmnd;
-struct scsi_device;
 struct scsi_driver {
 	struct device_driver	gendrv;
 #define scsi_unregister_interface(intf) \
 	class_interface_unregister(intf)
+/* make sure not to use it with passthrough commands */
+static inline struct scsi_driver *scsi_cmd_to_driver(struct scsi_cmnd *cmd)
+{
+	return to_scsi_driver(cmd->device->sdev_gendev.driver);
+}
 #endif /* _SCSI_SCSI_DRIVER_H */

include/trace/events/block.h+36−13

@@ -100,19 +100,7 @@ TRACE_EVENT(block_rq_requeue, · 99 unmodified lines
100	__entry->nr_sector, 0)	100	__entry->nr_sector, 0)
101	);	101	);
102		102
103	/**	103	DECLARE_EVENT_CLASS(block_rq_completion,
104	* block_rq_complete - block IO operation completed by device driver
105	* @rq: block operations request
106	* @error: status code
107	* @nr_bytes: number of completed bytes
108	*
109	* The block_rq_complete tracepoint event indicates that some portion
110	* of operation request has been completed by the device driver. If
111	* the @rq->bio is %NULL, then there is absolutely no additional work to
112	* do for the request. If @rq->bio is non-NULL then there is
113	* additional work required to complete the request.
114	*/
115	TRACE_EVENT(block_rq_complete,
116		104
117	TP_PROTO(struct request *rq, blk_status_t error, unsigned int nr_bytes),	105	TP_PROTO(struct request *rq, blk_status_t error, unsigned int nr_bytes),
118		106
@@ -144,6 +132,41 @@ TRACE_EVENT(block_rq_complete, · 25 unmodified lines
144	__entry->nr_sector, __entry->error)	132	__entry->nr_sector, __entry->error)
145	);	133	);
146		134
		135	/**
		136	* block_rq_complete - block IO operation completed by device driver
		137	* @rq: block operations request
		138	* @error: status code
		139	* @nr_bytes: number of completed bytes
		140	*
		141	* The block_rq_complete tracepoint event indicates that some portion
		142	* of operation request has been completed by the device driver. If
		143	* the @rq->bio is %NULL, then there is absolutely no additional work to
		144	* do for the request. If @rq->bio is non-NULL then there is
		145	* additional work required to complete the request.
		146	*/
		147	DEFINE_EVENT(block_rq_completion, block_rq_complete,
		148
		149	TP_PROTO(struct request *rq, blk_status_t error, unsigned int nr_bytes),
		150
		151	TP_ARGS(rq, error, nr_bytes)
		152	);
		153
		154	/**
		155	* block_rq_error - block IO operation error reported by device driver
		156	* @rq: block operations request
		157	* @error: status code
		158	* @nr_bytes: number of completed bytes
		159	*
		160	* The block_rq_error tracepoint event indicates that some portion
		161	* of operation request has failed as reported by the device driver.
		162	*/
		163	DEFINE_EVENT(block_rq_completion, block_rq_error,
		164
		165	TP_PROTO(struct request *rq, blk_status_t error, unsigned int nr_bytes),
		166
		167	TP_ARGS(rq, error, nr_bytes)
		168	);
		169
147	DECLARE_EVENT_CLASS(block_rq,	170	DECLARE_EVENT_CLASS(block_rq,
148		171
149	TP_PROTO(struct request *rq),	172	TP_PROTO(struct request *rq),

init/do_mounts.c+0−1

@@ -8,7 +8,6 @@ · 7 unmodified lines
8	#include <linux/root_dev.h>	8	#include <linux/root_dev.h>
9	#include <linux/security.h>	9	#include <linux/security.h>
10	#include <linux/delay.h>	10	#include <linux/delay.h>
11	#include <linux/genhd.h>
12	#include <linux/mount.h>	11	#include <linux/mount.h>
13	#include <linux/device.h>	12	#include <linux/device.h>
14	#include <linux/init.h>	13	#include <linux/init.h>

kernel/exit.c+1−1

@@ -735,7 +735,7 @@ void __noreturn do_exit(long code) · 734 unmodified lines
735	struct task_struct *tsk = current;	735	struct task_struct *tsk = current;
736	int group_dead;	736	int group_dead;
737		737
738	WARN_ON(blk_needs_flush_plug(tsk));	738	WARN_ON(tsk->plug);
739		739
740	/*	740	/*
741	* If do_dead is called because this processes oopsed, it's possible	741	* If do_dead is called because this processes oopsed, it's possible

kernel/power/hibernate.c+0−1

@@ -28,7 +28,6 @@ · 27 unmodified lines
28	#include <linux/gfp.h>	28	#include <linux/gfp.h>
29	#include <linux/syscore_ops.h>	29	#include <linux/syscore_ops.h>
30	#include <linux/ctype.h>	30	#include <linux/ctype.h>
31	#include <linux/genhd.h>
32	#include <linux/ktime.h>	31	#include <linux/ktime.h>
33	#include <linux/security.h>	32	#include <linux/security.h>
34	#include <linux/secretmem.h>	33	#include <linux/secretmem.h>

kernel/power/swap.c+2−4

 #include <linux/file.h>
 #include <linux/delay.h>
 #include <linux/bitops.h>
-#include <linux/genhd.h>
 #include <linux/device.h>
 #include <linux/bio.h>
 #include <linux/blkdev.h>
 	struct bio *bio;
 	int error = 0;
-	bio = bio_alloc(GFP_NOIO | __GFP_HIGH, 1);
+	bio = bio_alloc(hib_resume_bdev, 1, op | op_flags,
+			GFP_NOIO | __GFP_HIGH);
 	bio->bi_iter.bi_sector = page_off * (PAGE_SIZE >> 9);
-	bio_set_dev(bio, hib_resume_bdev);
-	bio_set_op_attrs(bio, op, op_flags);
 	if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) {
 		pr_err("Adding page to bio failed at %llu\n",

kernel/sched/core.c+2−5

 	 * If we are going to sleep and we have plugged IO queued,
 	 * make sure to submit it to avoid deadlocks.
 	 */
-	if (blk_needs_flush_plug(tsk))
+	blk_flush_plug(tsk->plug, true);
-		blk_flush_plug(tsk->plug, true);
 }
 static void sched_update_worker(struct task_struct *tsk)
 	int old_iowait = current->in_iowait;
 	current->in_iowait = 1;
-	if (current->plug)
+	blk_flush_plug(current->plug, true);
-		blk_flush_plug(current->plug, true);
 	return old_iowait;
 }

lib/sbitmap.c+17−23

 		      bool alloc_hint)
 {
 	unsigned int bits_per_word;
-	unsigned int i;
 	if (shift < 0)
 		shift = sbitmap_calculate_shift(depth);
 		return -ENOMEM;
 	}
-	for (i = 0; i < sb->map_nr; i++) {
-		sb->map[i].depth = min(depth, bits_per_word);
-		depth -= sb->map[i].depth;
-	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(sbitmap_init_node);
 	sb->depth = depth;
 	sb->map_nr = DIV_ROUND_UP(sb->depth, bits_per_word);
-	for (i = 0; i < sb->map_nr; i++) {
-		sb->map[i].depth = min(depth, bits_per_word);
-		depth -= sb->map[i].depth;
-	}
 }
 EXPORT_SYMBOL_GPL(sbitmap_resize);
 	int nr;
 	do {
-		nr = __sbitmap_get_word(&map->word, map->depth, alloc_hint,
+		nr = __sbitmap_get_word(&map->word, __map_depth(sb, index),
-					!sb->round_robin);
+					alloc_hint, !sb->round_robin);
 		if (nr != -1)
 			break;
 		if (!sbitmap_deferred_clear(map))
 	for (i = 0; i < sb->map_nr; i++) {
 again:
 		nr = __sbitmap_get_word(&sb->map[index].word,
-					min(sb->map[index].depth, shallow_depth),
+					min_t(unsigned int,
+					      __map_depth(sb, index),
+					      shallow_depth),
 					SB_NR_TO_BIT(sb, alloc_hint), true);
 		if (nr != -1) {
 			nr += index << sb->shift;
 	for (i = 0; i < sb->map_nr; i++) {
 		const struct sbitmap_word *word = &sb->map[i];
+		unsigned int word_depth = __map_depth(sb, i);
 		if (set)
-			weight += bitmap_weight(&word->word, word->depth);
+			weight += bitmap_weight(&word->word, word_depth);
 		else
-			weight += bitmap_weight(&word->cleared, word->depth);
+			weight += bitmap_weight(&word->cleared, word_depth);
 	}
 	return weight;
 }
 	for (i = 0; i < sb->map_nr; i++) {
 		unsigned long word = READ_ONCE(sb->map[i].word);
 		unsigned long cleared = READ_ONCE(sb->map[i].cleared);
-		unsigned int word_bits = READ_ONCE(sb->map[i].depth);
+		unsigned int word_bits = __map_depth(sb, i);
 		word &= ~cleared;
 	for (i = 0; i < sb->map_nr; i++) {
 		struct sbitmap_word *map = &sb->map[index];
 		unsigned long get_mask;
+		unsigned int map_depth = __map_depth(sb, index);
 		sbitmap_deferred_clear(map);
-		if (map->word == (1UL << (map->depth - 1)) - 1)
+		if (map->word == (1UL << (map_depth - 1)) - 1)
 			continue;
-		nr = find_first_zero_bit(&map->word, map->depth);
+		nr = find_first_zero_bit(&map->word, map_depth);
-		if (nr + nr_tags <= map->depth) {
+		if (nr + nr_tags <= map_depth) {
 			atomic_long_t *ptr = (atomic_long_t *) &map->word;
-			int map_tags = min_t(int, nr_tags, map->depth);
+			int map_tags = min_t(int, nr_tags, map_depth);
 			unsigned long val, ret;
 			get_mask = ((1UL << map_tags) - 1) << nr;
 	return 0;
 }
-int __sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
+int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq,
-				unsigned int shallow_depth)
+			      unsigned int shallow_depth)
 {
 	WARN_ON_ONCE(shallow_depth < sbq->min_shallow_depth);
 	return sbitmap_get_shallow(&sbq->sb, shallow_depth);
 }
-EXPORT_SYMBOL_GPL(__sbitmap_queue_get_shallow);
+EXPORT_SYMBOL_GPL(sbitmap_queue_get_shallow);
 void sbitmap_queue_min_shallow_depth(struct sbitmap_queue *sbq,
 				     unsigned int min_shallow_depth)

mm/page_io.c+4−6

 		return 0;
 	}
-	bio = bio_alloc(GFP_NOIO, 1);
+	bio = bio_alloc(sis->bdev, 1,
-	bio_set_dev(bio, sis->bdev);
+			REQ_OP_WRITE | REQ_SWAP | wbc_to_write_flags(wbc),
+			GFP_NOIO);
 	bio->bi_iter.bi_sector = swap_page_sector(page);
-	bio->bi_opf = REQ_OP_WRITE | REQ_SWAP | wbc_to_write_flags(wbc);
 	bio->bi_end_io = end_write_func;
 	bio_add_page(bio, page, thp_size(page), 0);
 	}
 	ret = 0;
-	bio = bio_alloc(GFP_KERNEL, 1);
+	bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL);
-	bio_set_dev(bio, sis->bdev);
-	bio->bi_opf = REQ_OP_READ;
 	bio->bi_iter.bi_sector = swap_page_sector(page);
 	bio->bi_end_io = end_swap_bio_read;
 	bio_add_page(bio, page, thp_size(page), 0);

security/integrity/ima/ima_policy.c+0−1

@@ -16,7 +16,6 @@ · 15 unmodified lines
16	#include <linux/parser.h>	16	#include <linux/parser.h>
17	#include <linux/slab.h>	17	#include <linux/slab.h>
18	#include <linux/rculist.h>	18	#include <linux/rculist.h>
19	#include <linux/genhd.h>
20	#include <linux/seq_file.h>	19	#include <linux/seq_file.h>
21	#include <linux/ima.h>	20	#include <linux/ima.h>
22		21

Merge tag 'for-5.18/block-2022-03-18' of git://git.kernel.dk/linux-block

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code