Pull block updates from Jens Axboe: - BFQ cleanups and fixes (Yu, Zhang, Yahu, Paolo) - blk-rq-qos completion fix (Tejun) - blk-cgroup merge fix (Tejun) - Add offline error return value to distinguish it from an IO error on the device (Song) - IO stats fixes (Zhang, Christoph) - blkcg refcount fixes (Ming, Yu) - Fix for indefinite dispatch loop softlockup (Shin'ichiro) - blk-mq hardware queue management improvements (Ming) - sbitmap dead code removal (Ming, John) - Plugging merge improvements (me) - Show blk-crypto capabilities in sysfs (Eric) - Multiple delayed queue run improvement (David) - Block throttling fixes (Ming) - Start deprecating auto module loading based on dev_t (Christoph) - bio allocation improvements (Christoph, Chaitanya) - Get rid of bio_devname (Christoph) - bio clone improvements (Christoph) - Block plugging improvements (Christoph) - Get rid of genhd.h header (Christoph) - Ensure drivers use appropriate flush helpers (Christoph) - Refcounting improvements (Christoph) - Queue initialization and teardown improvements (Ming, Christoph) - Misc fixes/improvements (Barry, Chaitanya, Colin, Dan, Jiapeng, Lukas, Nian, Yang, Eric, Chengming) * tag 'for-5.18/block-2022-03-18' of git://git.kernel.dk/linux-block: (127 commits) block: cancel all throttled bios in del_gendisk() block: let blkcg_gq grab request queue's refcnt block: avoid use-after-free on throttle data block: limit request dispatch loop duration block/bfq-iosched: Fix spelling mistake "tenative" -> "tentative" sr: simplify the local variable initialization in sr_block_open() block: don't merge across cgroup boundaries if blkcg is enabled block: fix rq-qos breakage from skipping rq_qos_done_bio() block: flush plug based on hardware and software queue order block: ensure plug merging checks the correct queue at least once block: move rq_qos_exit() into disk_release() block: do more work in elevator_exit block: move blk_exit_queue into disk_release block: move q_usage_counter release into blk_queue_release block: don't remove hctx debugfs dir from blk_mq_exit_queue block: move blkcg initialization/destroy into disk allocation/release handler sr: implement ->free_disk to simplify refcounting sd: implement ->free_disk to simplify refcounting sd: delay calling free_opal_dev sd: call sd_zbc_release_disk before releasing the scsi_device reference ...
Unified · Split
| @@ -155,6 +155,55 @@ Description: · 154 unmodified lines | ||
| 155 | 155 | last zone of the device which may be smaller. |
| 156 | 156 | |
| 157 | 157 | |
| 158 | +What: /sys/block/<disk>/queue/crypto/ | |
| 159 | +Date: February 2022 | |
| 160 | +Contact: linux-block@vger.kernel.org | |
| 161 | +Description: | |
| 162 | + The presence of this subdirectory of /sys/block/<disk>/queue/ | |
| 163 | + indicates that the device supports inline encryption. This | |
| 164 | + subdirectory contains files which describe the inline encryption | |
| 165 | + capabilities of the device. For more information about inline | |
| 166 | + encryption, refer to Documentation/block/inline-encryption.rst. | |
| 167 | + | |
| 168 | + | |
| 169 | +What: /sys/block/<disk>/queue/crypto/max_dun_bits | |
| 170 | +Date: February 2022 | |
| 171 | +Contact: linux-block@vger.kernel.org | |
| 172 | +Description: | |
| 173 | + [RO] This file shows the maximum length, in bits, of data unit | |
| 174 | + numbers accepted by the device in inline encryption requests. | |
| 175 | + | |
| 176 | + | |
| 177 | +What: /sys/block/<disk>/queue/crypto/modes/<mode> | |
| 178 | +Date: February 2022 | |
| 179 | +Contact: linux-block@vger.kernel.org | |
| 180 | +Description: | |
| 181 | + [RO] For each crypto mode (i.e., encryption/decryption | |
| 182 | + algorithm) the device supports with inline encryption, a file | |
| 183 | + will exist at this location. It will contain a hexadecimal | |
| 184 | + number that is a bitmask of the supported data unit sizes, in | |
| 185 | + bytes, for that crypto mode. | |
| 186 | + | |
| 187 | + Currently, the crypto modes that may be supported are: | |
| 188 | + | |
| 189 | + * AES-256-XTS | |
| 190 | + * AES-128-CBC-ESSIV | |
| 191 | + * Adiantum | |
| 192 | + | |
| 193 | + For example, if a device supports AES-256-XTS inline encryption | |
| 194 | + with data unit sizes of 512 and 4096 bytes, the file | |
| 195 | + /sys/block/<disk>/queue/crypto/modes/AES-256-XTS will exist and | |
| 196 | + will contain "0x1200". | |
| 197 | + | |
| 198 | + | |
| 199 | +What: /sys/block/<disk>/queue/crypto/num_keyslots | |
| 200 | +Date: February 2022 | |
| 201 | +Contact: linux-block@vger.kernel.org | |
| 202 | +Description: | |
| 203 | + [RO] This file shows the number of keyslots the device has for | |
| 204 | + use with inline encryption. | |
| 205 | + | |
| 206 | + | |
| 158 | 207 | What: /sys/block/<disk>/queue/dax |
| 159 | 208 | Date: June 2016 |
| 160 | 209 | Contact: linux-block@vger.kernel.org |
| @@ -1,1164 +0,0 @@ | ||
| 1 | −===================================================== | |
| 2 | −Notes on the Generic Block Layer Rewrite in Linux 2.5 | |
| 3 | −===================================================== | |
| 4 | − | |
| 5 | −.. note:: | |
| 6 | − | |
| 7 | − It seems that there are lot of outdated stuff here. This seems | |
| 8 | − to be written somewhat as a task list. Yet, eventually, something | |
| 9 | − here might still be useful. | |
| 10 | − | |
| 11 | −Notes Written on Jan 15, 2002: | |
| 12 | − | |
| 13 | − - Jens Axboe <jens.axboe@oracle.com> | |
| 14 | − - Suparna Bhattacharya <suparna@in.ibm.com> | |
| 15 | − | |
| 16 | −Last Updated May 2, 2002 | |
| 17 | − | |
| 18 | −September 2003: Updated I/O Scheduler portions | |
| 19 | − - Nick Piggin <npiggin@kernel.dk> | |
| 20 | − | |
| 21 | −Introduction | |
| 22 | −============ | |
| 23 | − | |
| 24 | −These are some notes describing some aspects of the 2.5 block layer in the | |
| 25 | −context of the bio rewrite. The idea is to bring out some of the key | |
| 26 | −changes and a glimpse of the rationale behind those changes. | |
| 27 | − | |
| 28 | −Please mail corrections & suggestions to suparna@in.ibm.com. | |
| 29 | − | |
| 30 | −Credits | |
| 31 | −======= | |
| 32 | − | |
| 33 | −2.5 bio rewrite: | |
| 34 | − - Jens Axboe <jens.axboe@oracle.com> | |
| 35 | − | |
| 36 | −Many aspects of the generic block layer redesign were driven by and evolved | |
| 37 | −over discussions, prior patches and the collective experience of several | |
| 38 | −people. See sections 8 and 9 for a list of some related references. | |
| 39 | − | |
| 40 | −The following people helped with review comments and inputs for this | |
| 41 | −document: | |
| 42 | − | |
| 43 | − - Christoph Hellwig <hch@infradead.org> | |
| 44 | − - Arjan van de Ven <arjanv@redhat.com> | |
| 45 | − - Randy Dunlap <rdunlap@xenotime.net> | |
| 46 | − - Andre Hedrick <andre@linux-ide.org> | |
| 47 | − | |
| 48 | −The following people helped with fixes/contributions to the bio patches | |
| 49 | −while it was still work-in-progress: | |
| 50 | − | |
| 51 | − - David S. Miller <davem@redhat.com> | |
| 52 | − | |
| 53 | − | |
| 54 | −.. Description of Contents: | |
| 55 | − | |
| 56 | − 1. Scope for tuning of logic to various needs | |
| 57 | − 1.1 Tuning based on device or low level driver capabilities | |
| 58 | − - Per-queue parameters | |
| 59 | − - Highmem I/O support | |
| 60 | − - I/O scheduler modularization | |
| 61 | − 1.2 Tuning based on high level requirements/capabilities | |
| 62 | − 1.2.1 Request Priority/Latency | |
| 63 | − 1.3 Direct access/bypass to lower layers for diagnostics and special | |
| 64 | − device operations | |
| 65 | − 1.3.1 Pre-built commands | |
| 66 | − 2. New flexible and generic but minimalist i/o structure or descriptor | |
| 67 | − (instead of using buffer heads at the i/o layer) | |
| 68 | − 2.1 Requirements/Goals addressed | |
| 69 | − 2.2 The bio struct in detail (multi-page io unit) | |
| 70 | − 2.3 Changes in the request structure | |
| 71 | − 3. Using bios | |
| 72 | − 3.1 Setup/teardown (allocation, splitting) | |
| 73 | − 3.2 Generic bio helper routines | |
| 74 | − 3.2.1 Traversing segments and completion units in a request | |
| 75 | − 3.2.2 Setting up DMA scatterlists | |
| 76 | − 3.2.3 I/O completion | |
| 77 | − 3.2.4 Implications for drivers that do not interpret bios (don't handle | |
| 78 | − multiple segments) | |
| 79 | − 3.3 I/O submission | |
| 80 | − 4. The I/O scheduler | |
| 81 | − 5. Scalability related changes | |
| 82 | − 5.1 Granular locking: Removal of io_request_lock | |
| 83 | − 5.2 Prepare for transition to 64 bit sector_t | |
| 84 | − 6. Other Changes/Implications | |
| 85 | − 6.1 Partition re-mapping handled by the generic block layer | |
| 86 | − 7. A few tips on migration of older drivers | |
| 87 | − 8. A list of prior/related/impacted patches/ideas | |
| 88 | − 9. Other References/Discussion Threads | |
| 89 | − | |
| 90 | − | |
| 91 | −Bio Notes | |
| 92 | −========= | |
| 93 | − | |
| 94 | −Let us discuss the changes in the context of how some overall goals for the | |
| 95 | −block layer are addressed. | |
| 96 | − | |
| 97 | −1. Scope for tuning the generic logic to satisfy various requirements | |
| 98 | −===================================================================== | |
| 99 | − | |
| 100 | −The block layer design supports adaptable abstractions to handle common | |
| 101 | −processing with the ability to tune the logic to an appropriate extent | |
| 102 | −depending on the nature of the device and the requirements of the caller. | |
| 103 | −One of the objectives of the rewrite was to increase the degree of tunability | |
| 104 | −and to enable higher level code to utilize underlying device/driver | |
| 105 | −capabilities to the maximum extent for better i/o performance. This is | |
| 106 | −important especially in the light of ever improving hardware capabilities | |
| 107 | −and application/middleware software designed to take advantage of these | |
| 108 | −capabilities. | |
| 109 | − | |
| 110 | −1.1 Tuning based on low level device / driver capabilities | |
| 111 | −---------------------------------------------------------- | |
| 112 | − | |
| 113 | −Sophisticated devices with large built-in caches, intelligent i/o scheduling | |
| 114 | −optimizations, high memory DMA support, etc may find some of the | |
| 115 | −generic processing an overhead, while for less capable devices the | |
| 116 | −generic functionality is essential for performance or correctness reasons. | |
| 117 | −Knowledge of some of the capabilities or parameters of the device should be | |
| 118 | −used at the generic block layer to take the right decisions on | |
| 119 | −behalf of the driver. | |
| 120 | − | |
| 121 | −How is this achieved ? | |
| 122 | − | |
| 123 | −Tuning at a per-queue level: | |
| 124 | − | |
| 125 | −i. Per-queue limits/values exported to the generic layer by the driver | |
| 126 | − | |
| 127 | −Various parameters that the generic i/o scheduler logic uses are set at | |
| 128 | −a per-queue level (e.g maximum request size, maximum number of segments in | |
| 129 | −a scatter-gather list, logical block size) | |
| 130 | − | |
| 131 | −Some parameters that were earlier available as global arrays indexed by | |
| 132 | −major/minor are now directly associated with the queue. Some of these may | |
| 133 | −move into the block device structure in the future. Some characteristics | |
| 134 | −have been incorporated into a queue flags field rather than separate fields | |
| 135 | −in themselves. There are blk_queue_xxx functions to set the parameters, | |
| 136 | −rather than update the fields directly | |
| 137 | − | |
| 138 | −Some new queue property settings: | |
| 139 | − | |
| 140 | − blk_queue_bounce_limit(q, u64 dma_address) | |
| 141 | − Enable I/O to highmem pages, dma_address being the | |
| 142 | − limit. No highmem default. | |
| 143 | − | |
| 144 | − blk_queue_max_sectors(q, max_sectors) | |
| 145 | − Sets two variables that limit the size of the request. | |
| 146 | − | |
| 147 | − - The request queue's max_sectors, which is a soft size in | |
| 148 | − units of 512 byte sectors, and could be dynamically varied | |
| 149 | − by the core kernel. | |
| 150 | − | |
| 151 | − - The request queue's max_hw_sectors, which is a hard limit | |
| 152 | − and reflects the maximum size request a driver can handle | |
| 153 | − in units of 512 byte sectors. | |
| 154 | − | |
| 155 | − The default for both max_sectors and max_hw_sectors is | |
| 156 | − 255. The upper limit of max_sectors is 1024. | |
| 157 | − | |
| 158 | − blk_queue_max_phys_segments(q, max_segments) | |
| 159 | − Maximum physical segments you can handle in a request. 128 | |
| 160 | − default (driver limit). (See 3.2.2) | |
| 161 | − | |
| 162 | − blk_queue_max_hw_segments(q, max_segments) | |
| 163 | − Maximum dma segments the hardware can handle in a request. 128 | |
| 164 | − default (host adapter limit, after dma remapping). | |
| 165 | − (See 3.2.2) | |
| 166 | − | |
| 167 | − blk_queue_max_segment_size(q, max_seg_size) | |
| 168 | − Maximum size of a clustered segment, 64kB default. | |
| 169 | − | |
| 170 | − blk_queue_logical_block_size(q, logical_block_size) | |
| 171 | − Lowest possible sector size that the hardware can operate | |
| 172 | − on, 512 bytes default. | |
| 173 | − | |
| 174 | −New queue flags: | |
| 175 | − | |
| 176 | − - QUEUE_FLAG_CLUSTER (see 3.2.2) | |
| 177 | − - QUEUE_FLAG_QUEUED (see 3.2.4) | |
| 178 | − | |
| 179 | − | |
| 180 | −ii. High-mem i/o capabilities are now considered the default | |
| 181 | − | |
| 182 | −The generic bounce buffer logic, present in 2.4, where the block layer would | |
| 183 | −by default copyin/out i/o requests on high-memory buffers to low-memory buffers | |
| 184 | −assuming that the driver wouldn't be able to handle it directly, has been | |
| 185 | −changed in 2.5. The bounce logic is now applied only for memory ranges | |
| 186 | −for which the device cannot handle i/o. A driver can specify this by | |
| 187 | −setting the queue bounce limit for the request queue for the device | |
| 188 | −(blk_queue_bounce_limit()). This avoids the inefficiencies of the copyin/out | |
| 189 | −where a device is capable of handling high memory i/o. | |
| 190 | − | |
| 191 | −In order to enable high-memory i/o where the device is capable of supporting | |
| 192 | −it, the pci dma mapping routines and associated data structures have now been | |
| 193 | −modified to accomplish a direct page -> bus translation, without requiring | |
| 194 | −a virtual address mapping (unlike the earlier scheme of virtual address | |
| 195 | −-> bus translation). So this works uniformly for high-memory pages (which | |
| 196 | −do not have a corresponding kernel virtual address space mapping) and | |
| 197 | −low-memory pages. | |
| 198 | − | |
| 199 | −Note: Please refer to Documentation/core-api/dma-api-howto.rst for a discussion | |
| 200 | −on PCI high mem DMA aspects and mapping of scatter gather lists, and support | |
| 201 | −for 64 bit PCI. | |
| 202 | − | |
| 203 | −Special handling is required only for cases where i/o needs to happen on | |
| 204 | −pages at physical memory addresses beyond what the device can support. In these | |
| 205 | −cases, a bounce bio representing a buffer from the supported memory range | |
| 206 | −is used for performing the i/o with copyin/copyout as needed depending on | |
| 207 | −the type of the operation. For example, in case of a read operation, the | |
| 208 | −data read has to be copied to the original buffer on i/o completion, so a | |
| 209 | −callback routine is set up to do this, while for write, the data is copied | |
| 210 | −from the original buffer to the bounce buffer prior to issuing the | |
| 211 | −operation. Since an original buffer may be in a high memory area that's not | |
| 212 | −mapped in kernel virtual addr, a kmap operation may be required for | |
| 213 | −performing the copy, and special care may be needed in the completion path | |
| 214 | −as it may not be in irq context. Special care is also required (by way of | |
| 215 | −GFP flags) when allocating bounce buffers, to avoid certain highmem | |
| 216 | −deadlock possibilities. | |
| 217 | − | |
| 218 | −It is also possible that a bounce buffer may be allocated from high-memory | |
| 219 | −area that's not mapped in kernel virtual addr, but within the range that the | |
| 220 | −device can use directly; so the bounce page may need to be kmapped during | |
| 221 | −copy operations. [Note: This does not hold in the current implementation, | |
| 222 | −though] | |
| 223 | − | |
| 224 | −There are some situations when pages from high memory may need to | |
| 225 | −be kmapped, even if bounce buffers are not necessary. For example a device | |
| 226 | −may need to abort DMA operations and revert to PIO for the transfer, in | |
| 227 | −which case a virtual mapping of the page is required. For SCSI it is also | |
| 228 | −done in some scenarios where the low level driver cannot be trusted to | |
| 229 | −handle a single sg entry correctly. The driver is expected to perform the | |
| 230 | −kmaps as needed on such occasions as appropriate. A driver could also use | |
| 231 | −the blk_queue_bounce() routine on its own to bounce highmem i/o to low | |
| 232 | −memory for specific requests if so desired. | |
| 233 | − | |
| 234 | −iii. The i/o scheduler algorithm itself can be replaced/set as appropriate | |
| 235 | − | |
| 236 | −As in 2.4, it is possible to plugin a brand new i/o scheduler for a particular | |
| 237 | −queue or pick from (copy) existing generic schedulers and replace/override | |
| 238 | −certain portions of it. The 2.5 rewrite provides improved modularization | |
| 239 | −of the i/o scheduler. There are more pluggable callbacks, e.g for init, | |
| 240 | −add request, extract request, which makes it possible to abstract specific | |
| 241 | −i/o scheduling algorithm aspects and details outside of the generic loop. | |
| 242 | −It also makes it possible to completely hide the implementation details of | |
| 243 | −the i/o scheduler from block drivers. | |
| 244 | − | |
| 245 | −I/O scheduler wrappers are to be used instead of accessing the queue directly. | |
| 246 | −See section 4. The I/O scheduler for details. | |
| 247 | − | |
| 248 | −1.2 Tuning Based on High level code capabilities | |
| 249 | −------------------------------------------------ | |
| 250 | − | |
| 251 | −i. Application capabilities for raw i/o | |
| 252 | − | |
| 253 | −This comes from some of the high-performance database/middleware | |
| 254 | −requirements where an application prefers to make its own i/o scheduling | |
| 255 | −decisions based on an understanding of the access patterns and i/o | |
| 256 | −characteristics | |
| 257 | − | |
| 258 | −ii. High performance filesystems or other higher level kernel code's | |
| 259 | −capabilities | |
| 260 | − | |
| 261 | −Kernel components like filesystems could also take their own i/o scheduling | |
| 262 | −decisions for optimizing performance. Journalling filesystems may need | |
| 263 | −some control over i/o ordering. | |
| 264 | − | |
| 265 | −What kind of support exists at the generic block layer for this ? | |
| 266 | − | |
| 267 | −The flags and rw fields in the bio structure can be used for some tuning | |
| 268 | −from above e.g indicating that an i/o is just a readahead request, or priority | |
| 269 | −settings (currently unused). As far as user applications are concerned they | |
| 270 | −would need an additional mechanism either via open flags or ioctls, or some | |
| 271 | −other upper level mechanism to communicate such settings to block. | |
| 272 | − | |
| 273 | −1.2.1 Request Priority/Latency | |
| 274 | −^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 275 | − | |
| 276 | −Todo/Under discussion:: | |
| 277 | − | |
| 278 | − Arjan's proposed request priority scheme allows higher levels some broad | |
| 279 | − control (high/med/low) over the priority of an i/o request vs other pending | |
| 280 | − requests in the queue. For example it allows reads for bringing in an | |
| 281 | − executable page on demand to be given a higher priority over pending write | |
| 282 | − requests which haven't aged too much on the queue. Potentially this priority | |
| 283 | − could even be exposed to applications in some manner, providing higher level | |
| 284 | − tunability. Time based aging avoids starvation of lower priority | |
| 285 | − requests. Some bits in the bi_opf flags field in the bio structure are | |
| 286 | − intended to be used for this priority information. | |
| 287 | − | |
| 288 | − | |
| 289 | −1.3 Direct Access to Low level Device/Driver Capabilities (Bypass mode) | |
| 290 | −----------------------------------------------------------------------- | |
| 291 | − | |
| 292 | −(e.g Diagnostics, Systems Management) | |
| 293 | − | |
| 294 | −There are situations where high-level code needs to have direct access to | |
| 295 | −the low level device capabilities or requires the ability to issue commands | |
| 296 | −to the device bypassing some of the intermediate i/o layers. | |
| 297 | −These could, for example, be special control commands issued through ioctl | |
| 298 | −interfaces, or could be raw read/write commands that stress the drive's | |
| 299 | −capabilities for certain kinds of fitness tests. Having direct interfaces at | |
| 300 | −multiple levels without having to pass through upper layers makes | |
| 301 | −it possible to perform bottom up validation of the i/o path, layer by | |
| 302 | −layer, starting from the media. | |
| 303 | − | |
| 304 | −The normal i/o submission interfaces, e.g submit_bio, could be bypassed | |
| 305 | −for specially crafted requests which such ioctl or diagnostics | |
| 306 | −interfaces would typically use, and the elevator add_request routine | |
| 307 | −can instead be used to directly insert such requests in the queue or preferably | |
| 308 | −the blk_do_rq routine can be used to place the request on the queue and | |
| 309 | −wait for completion. Alternatively, sometimes the caller might just | |
| 310 | −invoke a lower level driver specific interface with the request as a | |
| 311 | −parameter. | |
| 312 | − | |
| 313 | −If the request is a means for passing on special information associated with | |
| 314 | −the command, then such information is associated with the request->special | |
| 315 | −field (rather than misuse the request->buffer field which is meant for the | |
| 316 | −request data buffer's virtual mapping). | |
| 317 | − | |
| 318 | −For passing request data, the caller must build up a bio descriptor | |
| 319 | −representing the concerned memory buffer if the underlying driver interprets | |
| 320 | −bio segments or uses the block layer end*request* functions for i/o | |
| 321 | −completion. Alternatively one could directly use the request->buffer field to | |
| 322 | −specify the virtual address of the buffer, if the driver expects buffer | |
| 323 | −addresses passed in this way and ignores bio entries for the request type | |
| 324 | −involved. In the latter case, the driver would modify and manage the | |
| 325 | −request->buffer, request->sector and request->nr_sectors or | |
| 326 | −request->current_nr_sectors fields itself rather than using the block layer | |
| 327 | −end_request or end_that_request_first completion interfaces. | |
| 328 | −(See 2.3 or Documentation/block/request.rst for a brief explanation of | |
| 329 | −the request structure fields) | |
| 330 | − | |
| 331 | −:: | |
| 332 | − | |
| 333 | − [TBD: end_that_request_last should be usable even in this case; | |
| 334 | − Perhaps an end_that_direct_request_first routine could be implemented to make | |
| 335 | − handling direct requests easier for such drivers; Also for drivers that | |
| 336 | − expect bios, a helper function could be provided for setting up a bio | |
| 337 | − corresponding to a data buffer] | |
| 338 | − | |
| 339 | − <JENS: I dont understand the above, why is end_that_request_first() not | |
| 340 | − usable? Or _last for that matter. I must be missing something> | |
| 341 | − | |
| 342 | − <SUP: What I meant here was that if the request doesn't have a bio, then | |
| 343 | − end_that_request_first doesn't modify nr_sectors or current_nr_sectors, | |
| 344 | − and hence can't be used for advancing request state settings on the | |
| 345 | − completion of partial transfers. The driver has to modify these fields | |
| 346 | − directly by hand. | |
| 347 | − This is because end_that_request_first only iterates over the bio list, | |
| 348 | − and always returns 0 if there are none associated with the request. | |
| 349 | − _last works OK in this case, and is not a problem, as I mentioned earlier | |
| 350 | − > | |
| 351 | − | |
| 352 | −1.3.1 Pre-built Commands | |
| 353 | −^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 354 | − | |
| 355 | −A request can be created with a pre-built custom command to be sent directly | |
| 356 | −to the device. The cmd block in the request structure has room for filling | |
| 357 | −in the command bytes. (i.e rq->cmd is now 16 bytes in size, and meant for | |
| 358 | −command pre-building, and the type of the request is now indicated | |
| 359 | −through rq->flags instead of via rq->cmd) | |
| 360 | − | |
| 361 | −The request structure flags can be set up to indicate the type of request | |
| 362 | −in such cases (REQ_PC: direct packet command passed to driver, REQ_BLOCK_PC: | |
| 363 | −packet command issued via blk_do_rq, REQ_SPECIAL: special request). | |
| 364 | − | |
| 365 | −It can help to pre-build device commands for requests in advance. | |
| 366 | −Drivers can now specify a request prepare function (q->prep_rq_fn) that the | |
| 367 | −block layer would invoke to pre-build device commands for a given request, | |
| 368 | −or perform other preparatory processing for the request. This is routine is | |
| 369 | −called by elv_next_request(), i.e. typically just before servicing a request. | |
| 370 | −(The prepare function would not be called for requests that have RQF_DONTPREP | |
| 371 | −enabled) | |
| 372 | − | |
| 373 | −Aside: | |
| 374 | − Pre-building could possibly even be done early, i.e before placing the | |
| 375 | − request on the queue, rather than construct the command on the fly in the | |
| 376 | − driver while servicing the request queue when it may affect latencies in | |
| 377 | − interrupt context or responsiveness in general. One way to add early | |
| 378 | − pre-building would be to do it whenever we fail to merge on a request. | |
| 379 | − Now REQ_NOMERGE is set in the request flags to skip this one in the future, | |
| 380 | − which means that it will not change before we feed it to the device. So | |
| 381 | − the pre-builder hook can be invoked there. | |
| 382 | − | |
| 383 | − | |
| 384 | −2. Flexible and generic but minimalist i/o structure/descriptor | |
| 385 | −=============================================================== | |
| 386 | − | |
| 387 | −2.1 Reason for a new structure and requirements addressed | |
| 388 | −--------------------------------------------------------- | |
| 389 | − | |
| 390 | −Prior to 2.5, buffer heads were used as the unit of i/o at the generic block | |
| 391 | −layer, and the low level request structure was associated with a chain of | |
| 392 | −buffer heads for a contiguous i/o request. This led to certain inefficiencies | |
| 393 | −when it came to large i/o requests and readv/writev style operations, as it | |
| 394 | −forced such requests to be broken up into small chunks before being passed | |
| 395 | −on to the generic block layer, only to be merged by the i/o scheduler | |
| 396 | −when the underlying device was capable of handling the i/o in one shot. | |
| 397 | −Also, using the buffer head as an i/o structure for i/os that didn't originate | |
| 398 | −from the buffer cache unnecessarily added to the weight of the descriptors | |
| 399 | −which were generated for each such chunk. | |
| 400 | − | |
| 401 | −The following were some of the goals and expectations considered in the | |
| 402 | −redesign of the block i/o data structure in 2.5. | |
| 403 | − | |
| 404 | −1. Should be appropriate as a descriptor for both raw and buffered i/o - | |
| 405 | − avoid cache related fields which are irrelevant in the direct/page i/o path, | |
| 406 | − or filesystem block size alignment restrictions which may not be relevant | |
| 407 | − for raw i/o. | |
| 408 | −2. Ability to represent high-memory buffers (which do not have a virtual | |
| 409 | − address mapping in kernel address space). | |
| 410 | −3. Ability to represent large i/os w/o unnecessarily breaking them up (i.e | |
| 411 | − greater than PAGE_SIZE chunks in one shot) | |
| 412 | −4. At the same time, ability to retain independent identity of i/os from | |
| 413 | − different sources or i/o units requiring individual completion (e.g. for | |
| 414 | − latency reasons) | |
| 415 | −5. Ability to represent an i/o involving multiple physical memory segments | |
| 416 | − (including non-page aligned page fragments, as specified via readv/writev) | |
| 417 | − without unnecessarily breaking it up, if the underlying device is capable of | |
| 418 | − handling it. | |
| 419 | −6. Preferably should be based on a memory descriptor structure that can be | |
| 420 | − passed around different types of subsystems or layers, maybe even | |
| 421 | − networking, without duplication or extra copies of data/descriptor fields | |
| 422 | − themselves in the process | |
| 423 | −7. Ability to handle the possibility of splits/merges as the structure passes | |
| 424 | − through layered drivers (lvm, md, evms), with minimal overhead. | |
| 425 | − | |
| 426 | −The solution was to define a new structure (bio) for the block layer, | |
| 427 | −instead of using the buffer head structure (bh) directly, the idea being | |
| 428 | −avoidance of some associated baggage and limitations. The bio structure | |
| 429 | −is uniformly used for all i/o at the block layer ; it forms a part of the | |
| 430 | −bh structure for buffered i/o, and in the case of raw/direct i/o kiobufs are | |
| 431 | −mapped to bio structures. | |
| 432 | − | |
| 433 | −2.2 The bio struct | |
| 434 | −------------------ | |
| 435 | − | |
| 436 | −The bio structure uses a vector representation pointing to an array of tuples | |
| 437 | −of <page, offset, len> to describe the i/o buffer, and has various other | |
| 438 | −fields describing i/o parameters and state that needs to be maintained for | |
| 439 | −performing the i/o. | |
| 440 | − | |
| 441 | −Notice that this representation means that a bio has no virtual address | |
| 442 | −mapping at all (unlike buffer heads). | |
| 443 | − | |
| 444 | −:: | |
| 445 | − | |
| 446 | − struct bio_vec { | |
| 447 | − struct page *bv_page; | |
| 448 | − unsigned short bv_len; | |
| 449 | − unsigned short bv_offset; | |
| 450 | − }; | |
| 451 | − | |
| 452 | − /* | |
| 453 | − * main unit of I/O for the block layer and lower layers (ie drivers) | |
| 454 | − */ | |
| 455 | − struct bio { | |
| 456 | − struct bio *bi_next; /* request queue link */ | |
| 457 | − struct block_device *bi_bdev; /* target device */ | |
| 458 | − unsigned long bi_flags; /* status, command, etc */ | |
| 459 | − unsigned long bi_opf; /* low bits: r/w, high: priority */ | |
| 460 | − | |
| 461 | − unsigned int bi_vcnt; /* how may bio_vec's */ | |
| 462 | − struct bvec_iter bi_iter; /* current index into bio_vec array */ | |
| 463 | − | |
| 464 | − unsigned int bi_size; /* total size in bytes */ | |
| 465 | − unsigned short bi_hw_segments; /* segments after DMA remapping */ | |
| 466 | − unsigned int bi_max; /* max bio_vecs we can hold | |
| 467 | − used as index into pool */ | |
| 468 | − struct bio_vec *bi_io_vec; /* the actual vec list */ | |
| 469 | − bio_end_io_t *bi_end_io; /* bi_end_io (bio) */ | |
| 470 | − atomic_t bi_cnt; /* pin count: free when it hits zero */ | |
| 471 | − void *bi_private; | |
| 472 | − }; | |
| 473 | − | |
| 474 | −With this multipage bio design: | |
| 475 | − | |
| 476 | −- Large i/os can be sent down in one go using a bio_vec list consisting | |
| 477 | − of an array of <page, offset, len> fragments (similar to the way fragments | |
| 478 | − are represented in the zero-copy network code) | |
| 479 | −- Splitting of an i/o request across multiple devices (as in the case of | |
| 480 | − lvm or raid) is achieved by cloning the bio (where the clone points to | |
| 481 | − the same bi_io_vec array, but with the index and size accordingly modified) | |
| 482 | −- A linked list of bios is used as before for unrelated merges [#]_ - this | |
| 483 | − avoids reallocs and makes independent completions easier to handle. | |
| 484 | −- Code that traverses the req list can find all the segments of a bio | |
| 485 | − by using rq_for_each_segment. This handles the fact that a request | |
| 486 | − has multiple bios, each of which can have multiple segments. | |
| 487 | −- Drivers which can't process a large bio in one shot can use the bi_iter | |
| 488 | − field to keep track of the next bio_vec entry to process. | |
| 489 | − (e.g a 1MB bio_vec needs to be handled in max 128kB chunks for IDE) | |
| 490 | − [TBD: Should preferably also have a bi_voffset and bi_vlen to avoid modifying | |
| 491 | − bi_offset an len fields] | |
| 492 | − | |
| 493 | −.. [#] | |
| 494 | − | |
| 495 | − unrelated merges -- a request ends up containing two or more bios that | |
| 496 | − didn't originate from the same place. | |
| 497 | − | |
| 498 | −bi_end_io() i/o callback gets called on i/o completion of the entire bio. | |
| 499 | − | |
| 500 | −At a lower level, drivers build a scatter gather list from the merged bios. | |
| 501 | −The scatter gather list is in the form of an array of <page, offset, len> | |
| 502 | −entries with their corresponding dma address mappings filled in at the | |
| 503 | −appropriate time. As an optimization, contiguous physical pages can be | |
| 504 | −covered by a single entry where <page> refers to the first page and <len> | |
| 505 | −covers the range of pages (up to 16 contiguous pages could be covered this | |
| 506 | −way). There is a helper routine (blk_rq_map_sg) which drivers can use to build | |
| 507 | −the sg list. | |
| 508 | − | |
| 509 | −Note: Right now the only user of bios with more than one page is ll_rw_kio, | |
| 510 | −which in turn means that only raw I/O uses it (direct i/o may not work | |
| 511 | −right now). The intent however is to enable clustering of pages etc to | |
| 512 | −become possible. The pagebuf abstraction layer from SGI also uses multi-page | |
| 513 | −bios, but that is currently not included in the stock development kernels. | |
| 514 | −The same is true of Andrew Morton's work-in-progress multipage bio writeout | |
| 515 | −and readahead patches. | |
| 516 | − | |
| 517 | −2.3 Changes in the Request Structure | |
| 518 | −------------------------------------ | |
| 519 | − | |
| 520 | −The request structure is the structure that gets passed down to low level | |
| 521 | −drivers. The block layer make_request function builds up a request structure, | |
| 522 | −places it on the queue and invokes the drivers request_fn. The driver makes | |
| 523 | −use of block layer helper routine elv_next_request to pull the next request | |
| 524 | −off the queue. Control or diagnostic functions might bypass block and directly | |
| 525 | −invoke underlying driver entry points passing in a specially constructed | |
| 526 | −request structure. | |
| 527 | − | |
| 528 | −Only some relevant fields (mainly those which changed or may be referred | |
| 529 | −to in some of the discussion here) are listed below, not necessarily in | |
| 530 | −the order in which they occur in the structure (see include/linux/blkdev.h) | |
| 531 | −Refer to Documentation/block/request.rst for details about all the request | |
| 532 | −structure fields and a quick reference about the layers which are | |
| 533 | −supposed to use or modify those fields:: | |
| 534 | − | |
| 535 | − struct request { | |
| 536 | − struct list_head queuelist; /* Not meant to be directly accessed by | |
| 537 | − the driver. | |
| 538 | − Used by q->elv_next_request_fn | |
| 539 | − rq->queue is gone | |
| 540 | − */ | |
| 541 | − . | |
| 542 | − . | |
| 543 | − unsigned char cmd[16]; /* prebuilt command data block */ | |
| 544 | − unsigned long flags; /* also includes earlier rq->cmd settings */ | |
| 545 | − . | |
| 546 | − . | |
| 547 | − sector_t sector; /* this field is now of type sector_t instead of int | |
| 548 | − preparation for 64 bit sectors */ | |
| 549 | − . | |
| 550 | − . | |
| 551 | − | |
| 552 | − /* Number of scatter-gather DMA addr+len pairs after | |
| 553 | − * physical address coalescing is performed. | |
| 554 | − */ | |
| 555 | − unsigned short nr_phys_segments; | |
| 556 | − | |
| 557 | − /* Number of scatter-gather addr+len pairs after | |
| 558 | − * physical and DMA remapping hardware coalescing is performed. | |
| 559 | − * This is the number of scatter-gather entries the driver | |
| 560 | − * will actually have to deal with after DMA mapping is done. | |
| 561 | − */ | |
| 562 | − unsigned short nr_hw_segments; | |
| 563 | − | |
| 564 | − /* Various sector counts */ | |
| 565 | − unsigned long nr_sectors; /* no. of sectors left: driver modifiable */ | |
| 566 | − unsigned long hard_nr_sectors; /* block internal copy of above */ | |
| 567 | − unsigned int current_nr_sectors; /* no. of sectors left in the | |
| 568 | − current segment:driver modifiable */ | |
| 569 | − unsigned long hard_cur_sectors; /* block internal copy of the above */ | |
| 570 | − . | |
| 571 | − . | |
| 572 | − int tag; /* command tag associated with request */ | |
| 573 | − void *special; /* same as before */ | |
| 574 | − char *buffer; /* valid only for low memory buffers up to | |
| 575 | − current_nr_sectors */ | |
| 576 | − . | |
| 577 | − . | |
| 578 | − struct bio *bio, *biotail; /* bio list instead of bh */ | |
| 579 | − struct request_list *rl; | |
| 580 | − } | |
| 581 | − | |
| 582 | −See the req_ops and req_flag_bits definitions for an explanation of the various | |
| 583 | −flags available. Some bits are used by the block layer or i/o scheduler. | |
| 584 | − | |
| 585 | −The behaviour of the various sector counts are almost the same as before, | |
| 586 | −except that since we have multi-segment bios, current_nr_sectors refers | |
| 587 | −to the numbers of sectors in the current segment being processed which could | |
| 588 | −be one of the many segments in the current bio (i.e i/o completion unit). | |
| 589 | −The nr_sectors value refers to the total number of sectors in the whole | |
| 590 | −request that remain to be transferred (no change). The purpose of the | |
| 591 | −hard_xxx values is for block to remember these counts every time it hands | |
| 592 | −over the request to the driver. These values are updated by block on | |
| 593 | −end_that_request_first, i.e. every time the driver completes a part of the | |
| 594 | −transfer and invokes block end*request helpers to mark this. The | |
| 595 | −driver should not modify these values. The block layer sets up the | |
| 596 | −nr_sectors and current_nr_sectors fields (based on the corresponding | |
| 597 | −hard_xxx values and the number of bytes transferred) and updates it on | |
| 598 | −every transfer that invokes end_that_request_first. It does the same for the | |
| 599 | −buffer, bio, bio->bi_iter fields too. | |
| 600 | − | |
| 601 | −The buffer field is just a virtual address mapping of the current segment | |
| 602 | −of the i/o buffer in cases where the buffer resides in low-memory. For high | |
| 603 | −memory i/o, this field is not valid and must not be used by drivers. | |
| 604 | − | |
| 605 | −Code that sets up its own request structures and passes them down to | |
| 606 | −a driver needs to be careful about interoperation with the block layer helper | |
| 607 | −functions which the driver uses. (Section 1.3) | |
| 608 | − | |
| 609 | −3. Using bios | |
| 610 | −============= | |
| 611 | − | |
| 612 | −3.1 Setup/Teardown | |
| 613 | −------------------ | |
| 614 | − | |
| 615 | −There are routines for managing the allocation, and reference counting, and | |
| 616 | −freeing of bios (bio_alloc, bio_get, bio_put). | |
| 617 | − | |
| 618 | −This makes use of Ingo Molnar's mempool implementation, which enables | |
| 619 | −subsystems like bio to maintain their own reserve memory pools for guaranteed | |
| 620 | −deadlock-free allocations during extreme VM load. For example, the VM | |
| 621 | −subsystem makes use of the block layer to writeout dirty pages in order to be | |
| 622 | −able to free up memory space, a case which needs careful handling. The | |
| 623 | −allocation logic draws from the preallocated emergency reserve in situations | |
| 624 | −where it cannot allocate through normal means. If the pool is empty and it | |
| 625 | −can wait, then it would trigger action that would help free up memory or | |
| 626 | −replenish the pool (without deadlocking) and wait for availability in the pool. | |
| 627 | −If it is in IRQ context, and hence not in a position to do this, allocation | |
| 628 | −could fail if the pool is empty. In general mempool always first tries to | |
| 629 | −perform allocation without having to wait, even if it means digging into the | |
| 630 | −pool as long it is not less that 50% full. | |
| 631 | − | |
| 632 | −On a free, memory is released to the pool or directly freed depending on | |
| 633 | −the current availability in the pool. The mempool interface lets the | |
| 634 | −subsystem specify the routines to be used for normal alloc and free. In the | |
| 635 | −case of bio, these routines make use of the standard slab allocator. | |
| 636 | − | |
| 637 | −The caller of bio_alloc is expected to taken certain steps to avoid | |
| 638 | −deadlocks, e.g. avoid trying to allocate more memory from the pool while | |
| 639 | −already holding memory obtained from the pool. | |
| 640 | − | |
| 641 | −:: | |
| 642 | − | |
| 643 | − [TBD: This is a potential issue, though a rare possibility | |
| 644 | − in the bounce bio allocation that happens in the current code, since | |
| 645 | − it ends up allocating a second bio from the same pool while | |
| 646 | − holding the original bio ] | |
| 647 | − | |
| 648 | −Memory allocated from the pool should be released back within a limited | |
| 649 | −amount of time (in the case of bio, that would be after the i/o is completed). | |
| 650 | −This ensures that if part of the pool has been used up, some work (in this | |
| 651 | −case i/o) must already be in progress and memory would be available when it | |
| 652 | −is over. If allocating from multiple pools in the same code path, the order | |
| 653 | −or hierarchy of allocation needs to be consistent, just the way one deals | |
| 654 | −with multiple locks. | |
| 655 | − | |
| 656 | −The bio_alloc routine also needs to allocate the bio_vec_list (bvec_alloc()) | |
| 657 | −for a non-clone bio. There are the 6 pools setup for different size biovecs, | |
| 658 | −so bio_alloc(gfp_mask, nr_iovecs) will allocate a vec_list of the | |
| 659 | −given size from these slabs. | |
| 660 | − | |
| 661 | −The bio_get() routine may be used to hold an extra reference on a bio prior | |
| 662 | −to i/o submission, if the bio fields are likely to be accessed after the | |
| 663 | −i/o is issued (since the bio may otherwise get freed in case i/o completion | |
| 664 | −happens in the meantime). | |
| 665 | − | |
| 666 | −The bio_clone_fast() routine may be used to duplicate a bio, where the clone | |
| 667 | −shares the bio_vec_list with the original bio (i.e. both point to the | |
| 668 | −same bio_vec_list). This would typically be used for splitting i/o requests | |
| 669 | −in lvm or md. | |
| 670 | − | |
| 671 | −3.2 Generic bio helper Routines | |
| 672 | −------------------------------- | |
| 673 | − | |
| 674 | −3.2.1 Traversing segments and completion units in a request | |
| 675 | −^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 676 | − | |
| 677 | −The macro rq_for_each_segment() should be used for traversing the bios | |
| 678 | −in the request list (drivers should avoid directly trying to do it | |
| 679 | −themselves). Using these helpers should also make it easier to cope | |
| 680 | −with block changes in the future. | |
| 681 | − | |
| 682 | −:: | |
| 683 | − | |
| 684 | − struct req_iterator iter; | |
| 685 | − rq_for_each_segment(bio_vec, rq, iter) | |
| 686 | − /* bio_vec is now current segment */ | |
| 687 | − | |
| 688 | −I/O completion callbacks are per-bio rather than per-segment, so drivers | |
| 689 | −that traverse bio chains on completion need to keep that in mind. Drivers | |
| 690 | −which don't make a distinction between segments and completion units would | |
| 691 | −need to be reorganized to support multi-segment bios. | |
| 692 | − | |
| 693 | −3.2.2 Setting up DMA scatterlists | |
| 694 | −^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 695 | − | |
| 696 | −The blk_rq_map_sg() helper routine would be used for setting up scatter | |
| 697 | −gather lists from a request, so a driver need not do it on its own. | |
| 698 | − | |
| 699 | − nr_segments = blk_rq_map_sg(q, rq, scatterlist); | |
| 700 | − | |
| 701 | −The helper routine provides a level of abstraction which makes it easier | |
| 702 | −to modify the internals of request to scatterlist conversion down the line | |
| 703 | −without breaking drivers. The blk_rq_map_sg routine takes care of several | |
| 704 | −things like collapsing physically contiguous segments (if QUEUE_FLAG_CLUSTER | |
| 705 | −is set) and correct segment accounting to avoid exceeding the limits which | |
| 706 | −the i/o hardware can handle, based on various queue properties. | |
| 707 | − | |
| 708 | −- Prevents a clustered segment from crossing a 4GB mem boundary | |
| 709 | −- Avoids building segments that would exceed the number of physical | |
| 710 | − memory segments that the driver can handle (phys_segments) and the | |
| 711 | − number that the underlying hardware can handle at once, accounting for | |
| 712 | − DMA remapping (hw_segments) (i.e. IOMMU aware limits). | |
| 713 | − | |
| 714 | −Routines which the low level driver can use to set up the segment limits: | |
| 715 | − | |
| 716 | −blk_queue_max_hw_segments() : Sets an upper limit of the maximum number of | |
| 717 | −hw data segments in a request (i.e. the maximum number of address/length | |
| 718 | −pairs the host adapter can actually hand to the device at once) | |
| 719 | − | |
| 720 | −blk_queue_max_phys_segments() : Sets an upper limit on the maximum number | |
| 721 | −of physical data segments in a request (i.e. the largest sized scatter list | |
| 722 | −a driver could handle) | |
| 723 | − | |
| 724 | −3.2.3 I/O completion | |
| 725 | −^^^^^^^^^^^^^^^^^^^^ | |
| 726 | − | |
| 727 | −The existing generic block layer helper routines end_request, | |
| 728 | −end_that_request_first and end_that_request_last can be used for i/o | |
| 729 | −completion (and setting things up so the rest of the i/o or the next | |
| 730 | −request can be kicked of) as before. With the introduction of multi-page | |
| 731 | −bio support, end_that_request_first requires an additional argument indicating | |
| 732 | −the number of sectors completed. | |
| 733 | − | |
| 734 | −3.2.4 Implications for drivers that do not interpret bios | |
| 735 | −^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 736 | − | |
| 737 | −(don't handle multiple segments) | |
| 738 | − | |
| 739 | −Drivers that do not interpret bios e.g those which do not handle multiple | |
| 740 | −segments and do not support i/o into high memory addresses (require bounce | |
| 741 | −buffers) and expect only virtually mapped buffers, can access the rq->buffer | |
| 742 | −field. As before the driver should use current_nr_sectors to determine the | |
| 743 | −size of remaining data in the current segment (that is the maximum it can | |
| 744 | −transfer in one go unless it interprets segments), and rely on the block layer | |
| 745 | −end_request, or end_that_request_first/last to take care of all accounting | |
| 746 | −and transparent mapping of the next bio segment when a segment boundary | |
| 747 | −is crossed on completion of a transfer. (The end*request* functions should | |
| 748 | −be used if only if the request has come down from block/bio path, not for | |
| 749 | −direct access requests which only specify rq->buffer without a valid rq->bio) | |
| 750 | − | |
| 751 | −3.3 I/O Submission | |
| 752 | −------------------ | |
| 753 | − | |
| 754 | −The routine submit_bio() is used to submit a single io. Higher level i/o | |
| 755 | −routines make use of this: | |
| 756 | − | |
| 757 | −(a) Buffered i/o: | |
| 758 | − | |
| 759 | −The routine submit_bh() invokes submit_bio() on a bio corresponding to the | |
| 760 | −bh, allocating the bio if required. ll_rw_block() uses submit_bh() as before. | |
| 761 | − | |
| 762 | −(b) Kiobuf i/o (for raw/direct i/o): | |
| 763 | − | |
| 764 | −The ll_rw_kio() routine breaks up the kiobuf into page sized chunks and | |
| 765 | −maps the array to one or more multi-page bios, issuing submit_bio() to | |
| 766 | −perform the i/o on each of these. | |
| 767 | − | |
| 768 | −The embedded bh array in the kiobuf structure has been removed and no | |
| 769 | −preallocation of bios is done for kiobufs. [The intent is to remove the | |
| 770 | −blocks array as well, but it's currently in there to kludge around direct i/o.] | |
| 771 | −Thus kiobuf allocation has switched back to using kmalloc rather than vmalloc. | |
| 772 | − | |
| 773 | −Todo/Observation: | |
| 774 | − | |
| 775 | − A single kiobuf structure is assumed to correspond to a contiguous range | |
| 776 | − of data, so brw_kiovec() invokes ll_rw_kio for each kiobuf in a kiovec. | |
| 777 | − So right now it wouldn't work for direct i/o on non-contiguous blocks. | |
| 778 | − This is to be resolved. The eventual direction is to replace kiobuf | |
| 779 | − by kvec's. | |
| 780 | − | |
| 781 | − Badari Pulavarty has a patch to implement direct i/o correctly using | |
| 782 | − bio and kvec. | |
| 783 | − | |
| 784 | − | |
| 785 | −(c) Page i/o: | |
| 786 | − | |
| 787 | −Todo/Under discussion: | |
| 788 | − | |
| 789 | − Andrew Morton's multi-page bio patches attempt to issue multi-page | |
| 790 | − writeouts (and reads) from the page cache, by directly building up | |
| 791 | − large bios for submission completely bypassing the usage of buffer | |
| 792 | − heads. This work is still in progress. | |
| 793 | − | |
| 794 | − Christoph Hellwig had some code that uses bios for page-io (rather than | |
| 795 | − bh). This isn't included in bio as yet. Christoph was also working on a | |
| 796 | − design for representing virtual/real extents as an entity and modifying | |
| 797 | − some of the address space ops interfaces to utilize this abstraction rather | |
| 798 | − than buffer_heads. (This is somewhat along the lines of the SGI XFS pagebuf | |
| 799 | − abstraction, but intended to be as lightweight as possible). | |
| 800 | − | |
| 801 | −(d) Direct access i/o: | |
| 802 | − | |
| 803 | −Direct access requests that do not contain bios would be submitted differently | |
| 804 | −as discussed earlier in section 1.3. | |
| 805 | − | |
| 806 | −Aside: | |
| 807 | − | |
| 808 | − Kvec i/o: | |
| 809 | − | |
| 810 | − Ben LaHaise's aio code uses a slightly different structure instead | |
| 811 | − of kiobufs, called a kvec_cb. This contains an array of <page, offset, len> | |
| 812 | − tuples (very much like the networking code), together with a callback function | |
| 813 | − and data pointer. This is embedded into a brw_cb structure when passed | |
| 814 | − to brw_kvec_async(). | |
| 815 | − | |
| 816 | − Now it should be possible to directly map these kvecs to a bio. Just as while | |
| 817 | − cloning, in this case rather than PRE_BUILT bio_vecs, we set the bi_io_vec | |
| 818 | − array pointer to point to the veclet array in kvecs. | |
| 819 | − | |
| 820 | − TBD: In order for this to work, some changes are needed in the way multi-page | |
| 821 | − bios are handled today. The values of the tuples in such a vector passed in | |
| 822 | − from higher level code should not be modified by the block layer in the course | |
| 823 | − of its request processing, since that would make it hard for the higher layer | |
| 824 | − to continue to use the vector descriptor (kvec) after i/o completes. Instead, | |
| 825 | − all such transient state should either be maintained in the request structure, | |
| 826 | − and passed on in some way to the endio completion routine. | |
| 827 | − | |
| 828 | − | |
| 829 | −4. The I/O scheduler | |
| 830 | −==================== | |
| 831 | − | |
| 832 | −I/O scheduler, a.k.a. elevator, is implemented in two layers. Generic dispatch | |
| 833 | −queue and specific I/O schedulers. Unless stated otherwise, elevator is used | |
| 834 | −to refer to both parts and I/O scheduler to specific I/O schedulers. | |
| 835 | − | |
| 836 | −Block layer implements generic dispatch queue in `block/*.c`. | |
| 837 | −The generic dispatch queue is responsible for requeueing, handling non-fs | |
| 838 | −requests and all other subtleties. | |
| 839 | − | |
| 840 | −Specific I/O schedulers are responsible for ordering normal filesystem | |
| 841 | −requests. They can also choose to delay certain requests to improve | |
| 842 | −throughput or whatever purpose. As the plural form indicates, there are | |
| 843 | −multiple I/O schedulers. They can be built as modules but at least one should | |
| 844 | −be built inside the kernel. Each queue can choose different one and can also | |
| 845 | −change to another one dynamically. | |
| 846 | − | |
| 847 | −A block layer call to the i/o scheduler follows the convention elv_xxx(). This | |
| 848 | −calls elevator_xxx_fn in the elevator switch (block/elevator.c). Oh, xxx | |
| 849 | −and xxx might not match exactly, but use your imagination. If an elevator | |
| 850 | −doesn't implement a function, the switch does nothing or some minimal house | |
| 851 | −keeping work. | |
| 852 | − | |
| 853 | −4.1. I/O scheduler API | |
| 854 | −---------------------- | |
| 855 | − | |
| 856 | −The functions an elevator may implement are: (* are mandatory) | |
| 857 | − | |
| 858 | −=============================== ================================================ | |
| 859 | −elevator_merge_fn called to query requests for merge with a bio | |
| 860 | − | |
| 861 | −elevator_merge_req_fn called when two requests get merged. the one | |
| 862 | − which gets merged into the other one will be | |
| 863 | − never seen by I/O scheduler again. IOW, after | |
| 864 | − being merged, the request is gone. | |
| 865 | − | |
| 866 | −elevator_merged_fn called when a request in the scheduler has been | |
| 867 | − involved in a merge. It is used in the deadline | |
| 868 | − scheduler for example, to reposition the request | |
| 869 | − if its sorting order has changed. | |
| 870 | − | |
| 871 | −elevator_allow_merge_fn called whenever the block layer determines | |
| 872 | − that a bio can be merged into an existing | |
| 873 | − request safely. The io scheduler may still | |
| 874 | − want to stop a merge at this point if it | |
| 875 | − results in some sort of conflict internally, | |
| 876 | − this hook allows it to do that. Note however | |
| 877 | − that two *requests* can still be merged at later | |
| 878 | − time. Currently the io scheduler has no way to | |
| 879 | − prevent that. It can only learn about the fact | |
| 880 | − from elevator_merge_req_fn callback. | |
| 881 | − | |
| 882 | −elevator_dispatch_fn* fills the dispatch queue with ready requests. | |
| 883 | − I/O schedulers are free to postpone requests by | |
| 884 | − not filling the dispatch queue unless @force | |
| 885 | − is non-zero. Once dispatched, I/O schedulers | |
| 886 | − are not allowed to manipulate the requests - | |
| 887 | − they belong to generic dispatch queue. | |
| 888 | − | |
| 889 | −elevator_add_req_fn* called to add a new request into the scheduler | |
| 890 | − | |
| 891 | −elevator_former_req_fn | |
| 892 | −elevator_latter_req_fn These return the request before or after the | |
| 893 | − one specified in disk sort order. Used by the | |
| 894 | − block layer to find merge possibilities. | |
| 895 | − | |
| 896 | −elevator_completed_req_fn called when a request is completed. | |
| 897 | − | |
| 898 | −elevator_set_req_fn | |
| 899 | −elevator_put_req_fn Must be used to allocate and free any elevator | |
| 900 | − specific storage for a request. | |
| 901 | − | |
| 902 | −elevator_activate_req_fn Called when device driver first sees a request. | |
| 903 | − I/O schedulers can use this callback to | |
| 904 | − determine when actual execution of a request | |
| 905 | − starts. | |
| 906 | −elevator_deactivate_req_fn Called when device driver decides to delay | |
| 907 | − a request by requeueing it. | |
| 908 | − | |
| 909 | −elevator_init_fn* | |
| 910 | −elevator_exit_fn Allocate and free any elevator specific storage | |
| 911 | − for a queue. | |
| 912 | −=============================== ================================================ | |
| 913 | − | |
| 914 | −4.2 Request flows seen by I/O schedulers | |
| 915 | −---------------------------------------- | |
| 916 | − | |
| 917 | −All requests seen by I/O schedulers strictly follow one of the following three | |
| 918 | −flows. | |
| 919 | − | |
| 920 | − set_req_fn -> | |
| 921 | − | |
| 922 | − i. add_req_fn -> (merged_fn ->)* -> dispatch_fn -> activate_req_fn -> | |
| 923 | − (deactivate_req_fn -> activate_req_fn ->)* -> completed_req_fn | |
| 924 | − ii. add_req_fn -> (merged_fn ->)* -> merge_req_fn | |
| 925 | − iii. [none] | |
| 926 | − | |
| 927 | − -> put_req_fn | |
| 928 | − | |
| 929 | −4.3 I/O scheduler implementation | |
| 930 | −-------------------------------- | |
| 931 | − | |
| 932 | −The generic i/o scheduler algorithm attempts to sort/merge/batch requests for | |
| 933 | −optimal disk scan and request servicing performance (based on generic | |
| 934 | −principles and device capabilities), optimized for: | |
| 935 | − | |
| 936 | −i. improved throughput | |
| 937 | −ii. improved latency | |
| 938 | −iii. better utilization of h/w & CPU time | |
| 939 | − | |
| 940 | −Characteristics: | |
| 941 | − | |
| 942 | −i. Binary tree | |
| 943 | −AS and deadline i/o schedulers use red black binary trees for disk position | |
| 944 | −sorting and searching, and a fifo linked list for time-based searching. This | |
| 945 | −gives good scalability and good availability of information. Requests are | |
| 946 | −almost always dispatched in disk sort order, so a cache is kept of the next | |
| 947 | −request in sort order to prevent binary tree lookups. | |
| 948 | − | |
| 949 | −This arrangement is not a generic block layer characteristic however, so | |
| 950 | −elevators may implement queues as they please. | |
| 951 | − | |
| 952 | −ii. Merge hash | |
| 953 | −AS and deadline use a hash table indexed by the last sector of a request. This | |
| 954 | −enables merging code to quickly look up "back merge" candidates, even when | |
| 955 | −multiple I/O streams are being performed at once on one disk. | |
| 956 | − | |
| 957 | −"Front merges", a new request being merged at the front of an existing request, | |
| 958 | −are far less common than "back merges" due to the nature of most I/O patterns. | |
| 959 | −Front merges are handled by the binary trees in AS and deadline schedulers. | |
| 960 | − | |
| 961 | −iii. Plugging the queue to batch requests in anticipation of opportunities for | |
| 962 | − merge/sort optimizations | |
| 963 | − | |
| 964 | −Plugging is an approach that the current i/o scheduling algorithm resorts to so | |
| 965 | −that it collects up enough requests in the queue to be able to take | |
| 966 | −advantage of the sorting/merging logic in the elevator. If the | |
| 967 | −queue is empty when a request comes in, then it plugs the request queue | |
| 968 | −(sort of like plugging the bath tub of a vessel to get fluid to build up) | |
| 969 | −till it fills up with a few more requests, before starting to service | |
| 970 | −the requests. This provides an opportunity to merge/sort the requests before | |
| 971 | −passing them down to the device. There are various conditions when the queue is | |
| 972 | −unplugged (to open up the flow again), either through a scheduled task or | |
| 973 | −could be on demand. For example wait_on_buffer sets the unplugging going | |
| 974 | −through sync_buffer() running blk_run_address_space(mapping). Or the caller | |
| 975 | −can do it explicity through blk_unplug(bdev). So in the read case, | |
| 976 | −the queue gets explicitly unplugged as part of waiting for completion on that | |
| 977 | −buffer. | |
| 978 | − | |
| 979 | −Aside: | |
| 980 | − This is kind of controversial territory, as it's not clear if plugging is | |
| 981 | − always the right thing to do. Devices typically have their own queues, | |
| 982 | − and allowing a big queue to build up in software, while letting the device be | |
| 983 | − idle for a while may not always make sense. The trick is to handle the fine | |
| 984 | − balance between when to plug and when to open up. Also now that we have | |
| 985 | − multi-page bios being queued in one shot, we may not need to wait to merge | |
| 986 | − a big request from the broken up pieces coming by. | |
| 987 | − | |
| 988 | −4.4 I/O contexts | |
| 989 | −---------------- | |
| 990 | − | |
| 991 | −I/O contexts provide a dynamically allocated per process data area. They may | |
| 992 | −be used in I/O schedulers, and in the block layer (could be used for IO statis, | |
| 993 | −priorities for example). See `*io_context` in block/ll_rw_blk.c, and as-iosched.c | |
| 994 | −for an example of usage in an i/o scheduler. | |
| 995 | − | |
| 996 | − | |
| 997 | −5. Scalability related changes | |
| 998 | −============================== | |
| 999 | − | |
| 1000 | −5.1 Granular Locking: io_request_lock replaced by a per-queue lock | |
| 1001 | −------------------------------------------------------------------ | |
| 1002 | − | |
| 1003 | −The global io_request_lock has been removed as of 2.5, to avoid | |
| 1004 | −the scalability bottleneck it was causing, and has been replaced by more | |
| 1005 | −granular locking. The request queue structure has a pointer to the | |
| 1006 | −lock to be used for that queue. As a result, locking can now be | |
| 1007 | −per-queue, with a provision for sharing a lock across queues if | |
| 1008 | −necessary (e.g the scsi layer sets the queue lock pointers to the | |
| 1009 | −corresponding adapter lock, which results in a per host locking | |
| 1010 | −granularity). The locking semantics are the same, i.e. locking is | |
| 1011 | −still imposed by the block layer, grabbing the lock before | |
| 1012 | −request_fn execution which it means that lots of older drivers | |
| 1013 | −should still be SMP safe. Drivers are free to drop the queue | |
| 1014 | −lock themselves, if required. Drivers that explicitly used the | |
| 1015 | −io_request_lock for serialization need to be modified accordingly. | |
| 1016 | −Usually it's as easy as adding a global lock:: | |
| 1017 | − | |
| 1018 | − static DEFINE_SPINLOCK(my_driver_lock); | |
| 1019 | − | |
| 1020 | −and passing the address to that lock to blk_init_queue(). | |
| 1021 | − | |
| 1022 | −5.2 64 bit sector numbers (sector_t prepares for 64 bit support) | |
| 1023 | −---------------------------------------------------------------- | |
| 1024 | − | |
| 1025 | −The sector number used in the bio structure has been changed to sector_t, | |
| 1026 | −which could be defined as 64 bit in preparation for 64 bit sector support. | |
| 1027 | − | |
| 1028 | −6. Other Changes/Implications | |
| 1029 | −============================= | |
| 1030 | − | |
| 1031 | −6.1 Partition re-mapping handled by the generic block layer | |
| 1032 | −----------------------------------------------------------- | |
| 1033 | − | |
| 1034 | −In 2.5 some of the gendisk/partition related code has been reorganized. | |
| 1035 | −Now the generic block layer performs partition-remapping early and thus | |
| 1036 | −provides drivers with a sector number relative to whole device, rather than | |
| 1037 | −having to take partition number into account in order to arrive at the true | |
| 1038 | −sector number. The routine blk_partition_remap() is invoked by | |
| 1039 | −submit_bio_noacct even before invoking the queue specific ->submit_bio, | |
| 1040 | −so the i/o scheduler also gets to operate on whole disk sector numbers. This | |
| 1041 | −should typically not require changes to block drivers, it just never gets | |
| 1042 | −to invoke its own partition sector offset calculations since all bios | |
| 1043 | −sent are offset from the beginning of the device. | |
| 1044 | − | |
| 1045 | − | |
| 1046 | −7. A Few Tips on Migration of older drivers | |
| 1047 | −=========================================== | |
| 1048 | − | |
| 1049 | −Old-style drivers that just use CURRENT and ignores clustered requests, | |
| 1050 | −may not need much change. The generic layer will automatically handle | |
| 1051 | −clustered requests, multi-page bios, etc for the driver. | |
| 1052 | − | |
| 1053 | −For a low performance driver or hardware that is PIO driven or just doesn't | |
| 1054 | −support scatter-gather changes should be minimal too. | |
| 1055 | − | |
| 1056 | −The following are some points to keep in mind when converting old drivers | |
| 1057 | −to bio. | |
| 1058 | − | |
| 1059 | −Drivers should use elv_next_request to pick up requests and are no longer | |
| 1060 | −supposed to handle looping directly over the request list. | |
| 1061 | −(struct request->queue has been removed) | |
| 1062 | − | |
| 1063 | −Now end_that_request_first takes an additional number_of_sectors argument. | |
| 1064 | −It used to handle always just the first buffer_head in a request, now | |
| 1065 | −it will loop and handle as many sectors (on a bio-segment granularity) | |
| 1066 | −as specified. | |
| 1067 | − | |
| 1068 | −Now bh->b_end_io is replaced by bio->bi_end_io, but most of the time the | |
| 1069 | −right thing to use is bio_endio(bio) instead. | |
| 1070 | − | |
| 1071 | −If the driver is dropping the io_request_lock from its request_fn strategy, | |
| 1072 | −then it just needs to replace that with q->queue_lock instead. | |
| 1073 | − | |
| 1074 | −As described in Sec 1.1, drivers can set max sector size, max segment size | |
| 1075 | −etc per queue now. Drivers that used to define their own merge functions i | |
| 1076 | −to handle things like this can now just use the blk_queue_* functions at | |
| 1077 | −blk_init_queue time. | |
| 1078 | − | |
| 1079 | −Drivers no longer have to map a {partition, sector offset} into the | |
| 1080 | −correct absolute location anymore, this is done by the block layer, so | |
| 1081 | −where a driver received a request ala this before:: | |
| 1082 | − | |
| 1083 | − rq->rq_dev = mk_kdev(3, 5); /* /dev/hda5 */ | |
| 1084 | − rq->sector = 0; /* first sector on hda5 */ | |
| 1085 | − | |
| 1086 | −it will now see:: | |
| 1087 | − | |
| 1088 | − rq->rq_dev = mk_kdev(3, 0); /* /dev/hda */ | |
| 1089 | − rq->sector = 123128; /* offset from start of disk */ | |
| 1090 | − | |
| 1091 | −As mentioned, there is no virtual mapping of a bio. For DMA, this is | |
| 1092 | −not a problem as the driver probably never will need a virtual mapping. | |
| 1093 | −Instead it needs a bus mapping (dma_map_page for a single segment or | |
| 1094 | −use dma_map_sg for scatter gather) to be able to ship it to the driver. For | |
| 1095 | −PIO drivers (or drivers that need to revert to PIO transfer once in a | |
| 1096 | −while (IDE for example)), where the CPU is doing the actual data | |
| 1097 | −transfer a virtual mapping is needed. If the driver supports highmem I/O, | |
| 1098 | −(Sec 1.1, (ii) ) it needs to use kmap_atomic or similar to temporarily map | |
| 1099 | −a bio into the virtual address space. | |
| 1100 | − | |
| 1101 | − | |
| 1102 | −8. Prior/Related/Impacted patches | |
| 1103 | −================================= | |
| 1104 | − | |
| 1105 | −8.1. Earlier kiobuf patches (sct/axboe/chait/hch/mkp) | |
| 1106 | −----------------------------------------------------- | |
| 1107 | − | |
| 1108 | −- orig kiobuf & raw i/o patches (now in 2.4 tree) | |
| 1109 | −- direct kiobuf based i/o to devices (no intermediate bh's) | |
| 1110 | −- page i/o using kiobuf | |
| 1111 | −- kiobuf splitting for lvm (mkp) | |
| 1112 | −- elevator support for kiobuf request merging (axboe) | |
| 1113 | − | |
| 1114 | −8.2. Zero-copy networking (Dave Miller) | |
| 1115 | −--------------------------------------- | |
| 1116 | − | |
| 1117 | −8.3. SGI XFS - pagebuf patches - use of kiobufs | |
| 1118 | −----------------------------------------------- | |
| 1119 | −8.4. Multi-page pioent patch for bio (Christoph Hellwig) | |
| 1120 | −-------------------------------------------------------- | |
| 1121 | −8.5. Direct i/o implementation (Andrea Arcangeli) since 2.4.10-pre11 | |
| 1122 | −-------------------------------------------------------------------- | |
| 1123 | −8.6. Async i/o implementation patch (Ben LaHaise) | |
| 1124 | −------------------------------------------------- | |
| 1125 | −8.7. EVMS layering design (IBM EVMS team) | |
| 1126 | −----------------------------------------- | |
| 1127 | −8.8. Larger page cache size patch (Ben LaHaise) and Large page size (Daniel Phillips) | |
| 1128 | −------------------------------------------------------------------------------------- | |
| 1129 | − | |
| 1130 | − => larger contiguous physical memory buffers | |
| 1131 | − | |
| 1132 | −8.9. VM reservations patch (Ben LaHaise) | |
| 1133 | −---------------------------------------- | |
| 1134 | −8.10. Write clustering patches ? (Marcelo/Quintela/Riel ?) | |
| 1135 | −---------------------------------------------------------- | |
| 1136 | −8.11. Block device in page cache patch (Andrea Archangeli) - now in 2.4.10+ | |
| 1137 | −--------------------------------------------------------------------------- | |
| 1138 | −8.12. Multiple block-size transfers for faster raw i/o (Shailabh Nagar, Badari) | |
| 1139 | −------------------------------------------------------------------------------- | |
| 1140 | −8.13 Priority based i/o scheduler - prepatches (Arjan van de Ven) | |
| 1141 | −------------------------------------------------------------------ | |
| 1142 | −8.14 IDE Taskfile i/o patch (Andre Hedrick) | |
| 1143 | −-------------------------------------------- | |
| 1144 | −8.15 Multi-page writeout and readahead patches (Andrew Morton) | |
| 1145 | −--------------------------------------------------------------- | |
| 1146 | −8.16 Direct i/o patches for 2.5 using kvec and bio (Badari Pulavarthy) | |
| 1147 | −----------------------------------------------------------------------- | |
| 1148 | − | |
| 1149 | −9. Other References | |
| 1150 | −=================== | |
| 1151 | − | |
| 1152 | −9.1 The Splice I/O Model | |
| 1153 | −------------------------ | |
| 1154 | − | |
| 1155 | −Larry McVoy (and subsequent discussions on lkml, and Linus' comments - Jan 2001 | |
| 1156 | − | |
| 1157 | −9.2 Discussions about kiobuf and bh design | |
| 1158 | −------------------------------------------ | |
| 1159 | − | |
| 1160 | −On lkml between sct, linus, alan et al - Feb-March 2001 (many of the | |
| 1161 | −initial thoughts that led to bio were brought up in this discussion thread) | |
| 1162 | − | |
| 1163 | −9.3 Discussions on mempool on lkml - Dec 2001. | |
| 1164 | −---------------------------------------------- | |
| @@ -7,4 +7,4 @@ This file documents the sysfs file ``block/<disk>/capability``. · 6 unmodified lines | ||
| 7 | 7 | ``capability`` is a bitfield, printed in hexadecimal, indicating which |
| 8 | 8 | capabilities a specific block device supports: |
| 9 | 9 | |
| 10 | −.. kernel-doc:: include/linux/genhd.h | |
| 10 | +.. kernel-doc:: include/linux/blkdev.h | |
| @@ -8,7 +8,6 @@ Block · 7 unmodified lines | ||
| 8 | 8 | :maxdepth: 1 |
| 9 | 9 | |
| 10 | 10 | bfq-iosched |
| 11 | − biodoc | |
| 12 | 11 | biovecs |
| 13 | 12 | blk-mq |
| 14 | 13 | capability |
| @@ -3441,6 +3441,7 @@ F: Documentation/ABI/stable/sysfs-block · 3440 unmodified lines | ||
| 3441 | 3441 | F: Documentation/block/ |
| 3442 | 3442 | F: block/ |
| 3443 | 3443 | F: drivers/block/ |
| 3444 | +F: include/linux/bio.h | |
| 3444 | 3445 | F: include/linux/blk* |
| 3445 | 3446 | F: kernel/trace/blktrace.c |
| 3446 | 3447 | F: lib/sbitmap.c |
| @@ -30,7 +30,6 @@ · 29 unmodified lines | ||
| 30 | 30 | |
| 31 | 31 | #include <linux/types.h> |
| 32 | 32 | #include <linux/kdev_t.h> |
| 33 | −#include <linux/genhd.h> | |
| 34 | 33 | #include <linux/sched.h> |
| 35 | 34 | #include <linux/init.h> |
| 36 | 35 | #include <linux/interrupt.h> |
| @@ -23,7 +23,6 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <linux/linkage.h> |
| 24 | 24 | #include <linux/init.h> |
| 25 | 25 | #include <linux/major.h> |
| 26 | −#include <linux/genhd.h> | |
| 27 | 26 | #include <linux/rtc.h> |
| 28 | 27 | #include <linux/interrupt.h> |
| 29 | 28 | #include <linux/bcd.h> |
| @@ -13,7 +13,6 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/kernel.h> |
| 14 | 14 | #include <linux/errno.h> |
| 15 | 15 | #include <linux/types.h> |
| 16 | −#include <linux/genhd.h> | |
| 17 | 16 | #include <linux/blkdev.h> |
| 18 | 17 | #include <linux/hdreg.h> |
| 19 | 18 | #include <linux/slab.h> |
| @@ -16,7 +16,6 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <linux/interrupt.h> |
| 17 | 17 | #include <linux/fs.h> |
| 18 | 18 | #include <linux/console.h> |
| 19 | −#include <linux/genhd.h> | |
| 20 | 19 | #include <linux/errno.h> |
| 21 | 20 | #include <linux/string.h> |
| 22 | 21 | #include <linux/init.h> |
| @@ -22,7 +22,6 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <linux/linkage.h> |
| 23 | 23 | #include <linux/init.h> |
| 24 | 24 | #include <linux/major.h> |
| 25 | −#include <linux/genhd.h> | |
| 26 | 25 | #include <linux/rtc.h> |
| 27 | 26 | #include <linux/interrupt.h> |
| 28 | 27 | |
| @@ -24,7 +24,6 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <linux/linkage.h> |
| 25 | 25 | #include <linux/init.h> |
| 26 | 26 | #include <linux/major.h> |
| 27 | −#include <linux/genhd.h> | |
| 28 | 27 | #include <linux/rtc.h> |
| 29 | 28 | #include <linux/interrupt.h> |
| 30 | 29 | #include <linux/module.h> |
| @@ -26,6 +26,16 @@ menuconfig BLOCK · 25 unmodified lines | ||
| 26 | 26 | |
| 27 | 27 | if BLOCK |
| 28 | 28 | |
| 29 | +config BLOCK_LEGACY_AUTOLOAD | |
| 30 | + bool "Legacy autoloading support" | |
| 31 | + default y | |
| 32 | + help | |
| 33 | + Enable loading modules and creating block device instances based on | |
| 34 | + accesses through their device special file. This is a historic Linux | |
| 35 | + feature and makes no sense in a udev world where device files are | |
| 36 | + created on demand, but scripts that manually create device nodes and | |
| 37 | + then call losetup might rely on this behavior. | |
| 38 | + | |
| 29 | 39 | config BLK_RQ_ALLOC_TIME |
| 30 | 40 | bool |
| 31 | 41 | |
| @@ -218,6 +228,9 @@ config BLK_PM · 186 unmodified lines | ||
| 218 | 228 | config BLOCK_HOLDER_DEPRECATED |
| 219 | 229 | bool |
| 220 | 230 | |
| 231 | +config BLK_MQ_STACKING | |
| 232 | + bool | |
| 233 | + | |
| 221 | 234 | source "block/Kconfig.iosched" |
| 222 | 235 | |
| 223 | 236 | endif # BLOCK |
| @@ -36,6 +36,7 @@ obj-$(CONFIG_BLK_DEBUG_FS) += blk-mq-debugfs.o · 35 unmodified lines | ||
| 36 | 36 | obj-$(CONFIG_BLK_DEBUG_FS_ZONED)+= blk-mq-debugfs-zoned.o |
| 37 | 37 | obj-$(CONFIG_BLK_SED_OPAL) += sed-opal.o |
| 38 | 38 | obj-$(CONFIG_BLK_PM) += blk-pm.o |
| 39 | −obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += blk-crypto.o blk-crypto-profile.o | |
| 39 | +obj-$(CONFIG_BLK_INLINE_ENCRYPTION) += blk-crypto.o blk-crypto-profile.o \ | |
| 40 | + blk-crypto-sysfs.o | |
| 40 | 41 | obj-$(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK) += blk-crypto-fallback.o |
| 41 | 42 | obj-$(CONFIG_BLOCK_HOLDER_DEPRECATED) += holder.o |
| @@ -678,7 +678,7 @@ static int blkdev_get_whole(struct block_device *bdev, fmode_t mode) · 677 unmodified lines | ||
| 678 | 678 | if (test_bit(GD_NEED_PART_SCAN, &disk->state)) |
| 679 | 679 | bdev_disk_changed(disk, false); |
| 680 | 680 | bdev->bd_openers++; |
| 681 | − return 0;; | |
| 681 | + return 0; | |
| 682 | 682 | } |
| 683 | 683 | |
| 684 | 684 | static void blkdev_put_whole(struct block_device *bdev, fmode_t mode) |
| @@ -733,12 +733,15 @@ struct block_device *blkdev_get_no_open(dev_t dev) · 48 unmodified lines | ||
| 733 | 733 | struct inode *inode; |
| 734 | 734 | |
| 735 | 735 | inode = ilookup(blockdev_superblock, dev); |
| 736 | − if (!inode) { | |
| 736 | + if (!inode && IS_ENABLED(CONFIG_BLOCK_LEGACY_AUTOLOAD)) { | |
| 737 | 737 | blk_request_module(dev); |
| 738 | 738 | inode = ilookup(blockdev_superblock, dev); |
| 739 | − if (!inode) | |
| 740 | − return NULL; | |
| 739 | + if (inode) | |
| 740 | + pr_warn_ratelimited( | |
| 741 | +"block device autoloading is deprecated and will be removed.\n"); | |
| 741 | 742 | } |
| 743 | + if (!inode) | |
| 744 | + return NULL; | |
| 742 | 745 | |
| 743 | 746 | /* switch from the inode reference to a device mode one: */ |
| 744 | 747 | bdev = &BDEV_I(inode)->bdev; |
| @@ -645,7 +645,21 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq, · 644 unmodified lines | ||
| 645 | 645 | struct bfq_group *bfqg) |
| 646 | 646 | { |
| 647 | 647 | struct bfq_entity *entity = &bfqq->entity; |
| 648 | + struct bfq_group *old_parent = bfqq_group(bfqq); | |
| 648 | 649 | |
| 650 | + /* | |
| 651 | + * No point to move bfqq to the same group, which can happen when | |
| 652 | + * root group is offlined | |
| 653 | + */ | |
| 654 | + if (old_parent == bfqg) | |
| 655 | + return; | |
| 656 | + | |
| 657 | + /* | |
| 658 | + * oom_bfqq is not allowed to move, oom_bfqq will hold ref to root_group | |
| 659 | + * until elevator exit. | |
| 660 | + */ | |
| 661 | + if (bfqq == &bfqd->oom_bfqq) | |
| 662 | + return; | |
| 649 | 663 | /* |
| 650 | 664 | * Get extra reference to prevent bfqq from being freed in |
| 651 | 665 | * next possible expire or deactivate. |
| @@ -666,7 +680,7 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq, · 14 unmodified lines | ||
| 666 | 680 | bfq_deactivate_bfqq(bfqd, bfqq, false, false); |
| 667 | 681 | else if (entity->on_st_or_in_serv) |
| 668 | 682 | bfq_put_idle_entity(bfq_entity_service_tree(entity), entity); |
| 669 | − bfqg_and_blkg_put(bfqq_group(bfqq)); | |
| 683 | + bfqg_and_blkg_put(old_parent); | |
| 670 | 684 | |
| 671 | 685 | if (entity->parent && |
| 672 | 686 | entity->parent->last_bfqq_created == bfqq) |
| @@ -774,7 +774,7 @@ bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq) · 773 unmodified lines | ||
| 774 | 774 | if (!bfqq->next_rq) |
| 775 | 775 | return; |
| 776 | 776 | |
| 777 | − bfqq->pos_root = &bfq_bfqq_to_bfqg(bfqq)->rq_pos_tree; | |
| 777 | + bfqq->pos_root = &bfqq_group(bfqq)->rq_pos_tree; | |
| 778 | 778 | __bfqq = bfq_rq_pos_tree_lookup(bfqd, bfqq->pos_root, |
| 779 | 779 | blk_rq_pos(bfqq->next_rq), &parent, &p); |
| 780 | 780 | if (!__bfqq) { |
| @@ -2153,7 +2153,7 @@ static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq, · 1372 unmodified lines | ||
| 2153 | 2153 | bfqq->waker_detection_started = now_ns; |
| 2154 | 2154 | bfq_bfqq_name(bfqq->tentative_waker_bfqq, waker_name, |
| 2155 | 2155 | MAX_BFQQ_NAME_LENGTH); |
| 2156 | − bfq_log_bfqq(bfqd, bfqq, "set tenative waker %s", waker_name); | |
| 2156 | + bfq_log_bfqq(bfqd, bfqq, "set tentative waker %s", waker_name); | |
| 2157 | 2157 | } else /* Same tentative waker queue detected again */ |
| 2158 | 2158 | bfqq->num_waker_detections++; |
| 2159 | 2159 | |
| @@ -2669,7 +2669,7 @@ static struct bfq_queue *bfqq_find_close(struct bfq_data *bfqd, · 509 unmodified lines | ||
| 2669 | 2669 | struct bfq_queue *bfqq, |
| 2670 | 2670 | sector_t sector) |
| 2671 | 2671 | { |
| 2672 | − struct rb_root *root = &bfq_bfqq_to_bfqg(bfqq)->rq_pos_tree; | |
| 2672 | + struct rb_root *root = &bfqq_group(bfqq)->rq_pos_tree; | |
| 2673 | 2673 | struct rb_node *parent, *node; |
| 2674 | 2674 | struct bfq_queue *__bfqq; |
| 2675 | 2675 | |
| @@ -2782,6 +2782,15 @@ bfq_setup_merge(struct bfq_queue *bfqq, struct bfq_queue *new_bfqq) · 106 unmodified lines | ||
| 2782 | 2782 | * are likely to increase the throughput. |
| 2783 | 2783 | */ |
| 2784 | 2784 | bfqq->new_bfqq = new_bfqq; |
| 2785 | + /* | |
| 2786 | + * The above assignment schedules the following redirections: | |
| 2787 | + * each time some I/O for bfqq arrives, the process that | |
| 2788 | + * generated that I/O is disassociated from bfqq and | |
| 2789 | + * associated with new_bfqq. Here we increases new_bfqq->ref | |
| 2790 | + * in advance, adding the number of processes that are | |
| 2791 | + * expected to be associated with new_bfqq as they happen to | |
| 2792 | + * issue I/O. | |
| 2793 | + */ | |
| 2785 | 2794 | new_bfqq->ref += process_refs; |
| 2786 | 2795 | return new_bfqq; |
| 2787 | 2796 | } |
| @@ -2844,6 +2853,10 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, · 56 unmodified lines | ||
| 2844 | 2853 | { |
| 2845 | 2854 | struct bfq_queue *in_service_bfqq, *new_bfqq; |
| 2846 | 2855 | |
| 2856 | + /* if a merge has already been setup, then proceed with that first */ | |
| 2857 | + if (bfqq->new_bfqq) | |
| 2858 | + return bfqq->new_bfqq; | |
| 2859 | + | |
| 2847 | 2860 | /* |
| 2848 | 2861 | * Check delayed stable merge for rotational or non-queueing |
| 2849 | 2862 | * devs. For this branch to be executed, bfqq must not be |
| @@ -2945,9 +2958,6 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, · 95 unmodified lines | ||
| 2945 | 2958 | if (bfq_too_late_for_merging(bfqq)) |
| 2946 | 2959 | return NULL; |
| 2947 | 2960 | |
| 2948 | − if (bfqq->new_bfqq) | |
| 2949 | − return bfqq->new_bfqq; | |
| 2950 | − | |
| 2951 | 2961 | if (!io_struct || unlikely(bfqq == &bfqd->oom_bfqq)) |
| 2952 | 2962 | return NULL; |
| 2953 | 2963 | |
| @@ -5181,7 +5191,7 @@ static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) · 2227 unmodified lines | ||
| 5181 | 5191 | struct bfq_data *bfqd = hctx->queue->elevator->elevator_data; |
| 5182 | 5192 | struct request *rq; |
| 5183 | 5193 | struct bfq_queue *in_serv_queue; |
| 5184 | − bool waiting_rq, idle_timer_disabled; | |
| 5194 | + bool waiting_rq, idle_timer_disabled = false; | |
| 5185 | 5195 | |
| 5186 | 5196 | spin_lock_irq(&bfqd->lock); |
| 5187 | 5197 | |
| @@ -5189,14 +5199,15 @@ static struct request *bfq_dispatch_request(struct blk_mq_hw_ctx *hctx) · 1 unmodified line | ||
| 5189 | 5199 | waiting_rq = in_serv_queue && bfq_bfqq_wait_request(in_serv_queue); |
| 5190 | 5200 | |
| 5191 | 5201 | rq = __bfq_dispatch_request(hctx); |
| 5192 | − | |
| 5193 | − idle_timer_disabled = | |
| 5194 | − waiting_rq && !bfq_bfqq_wait_request(in_serv_queue); | |
| 5202 | + if (in_serv_queue == bfqd->in_service_queue) { | |
| 5203 | + idle_timer_disabled = | |
| 5204 | + waiting_rq && !bfq_bfqq_wait_request(in_serv_queue); | |
| 5205 | + } | |
| 5195 | 5206 | |
| 5196 | 5207 | spin_unlock_irq(&bfqd->lock); |
| 5197 | − | |
| 5198 | − bfq_update_dispatch_stats(hctx->queue, rq, in_serv_queue, | |
| 5199 | − idle_timer_disabled); | |
| 5208 | + bfq_update_dispatch_stats(hctx->queue, rq, | |
| 5209 | + idle_timer_disabled ? in_serv_queue : NULL, | |
| 5210 | + idle_timer_disabled); | |
| 5200 | 5211 | |
| 5201 | 5212 | return rq; |
| 5202 | 5213 | } |
| @@ -8,7 +8,6 @@ · 7 unmodified lines | ||
| 8 | 8 | |
| 9 | 9 | #include <linux/blktrace_api.h> |
| 10 | 10 | #include <linux/hrtimer.h> |
| 11 | −#include <linux/blk-cgroup.h> | |
| 12 | 11 | |
| 13 | 12 | #include "blk-cgroup-rwstat.h" |
| 14 | 13 | |
| @@ -1051,7 +1050,6 @@ extern struct blkcg_policy blkcg_policy_bfq; · 1036 unmodified lines | ||
| 1051 | 1050 | for (parent = NULL; entity ; entity = parent) |
| 1052 | 1051 | #endif /* CONFIG_BFQ_GROUP_IOSCHED */ |
| 1053 | 1052 | |
| 1054 | −struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq); | |
| 1055 | 1053 | struct bfq_queue *bfq_entity_to_bfqq(struct bfq_entity *entity); |
| 1056 | 1054 | unsigned int bfq_tot_busy_queues(struct bfq_data *bfqd); |
| 1057 | 1055 | struct bfq_service_tree *bfq_entity_service_tree(struct bfq_entity *entity); |
| @@ -142,16 +142,6 @@ static bool bfq_update_next_in_service(struct bfq_sched_data *sd, · 141 unmodified lines | ||
| 142 | 142 | |
| 143 | 143 | #ifdef CONFIG_BFQ_GROUP_IOSCHED |
| 144 | 144 | |
| 145 | −struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq) | |
| 146 | −{ | |
| 147 | − struct bfq_entity *group_entity = bfqq->entity.parent; | |
| 148 | − | |
| 149 | − if (!group_entity) | |
| 150 | − group_entity = &bfqq->bfqd->root_group->entity; | |
| 151 | − | |
| 152 | − return container_of(group_entity, struct bfq_group, entity); | |
| 153 | −} | |
| 154 | − | |
| 155 | 145 | /* |
| 156 | 146 | * Returns true if this budget changes may let next_in_service->parent |
| 157 | 147 | * become the next_in_service entity for its parent entity. |
| @@ -230,11 +220,6 @@ static bool bfq_no_longer_next_in_service(struct bfq_entity *entity) · 72 unmodified lines | ||
| 230 | 220 | |
| 231 | 221 | #else /* CONFIG_BFQ_GROUP_IOSCHED */ |
| 232 | 222 | |
| 233 | −struct bfq_group *bfq_bfqq_to_bfqg(struct bfq_queue *bfqq) | |
| 234 | −{ | |
| 235 | − return bfqq->bfqd->root_group; | |
| 236 | −} | |
| 237 | − | |
| 238 | 223 | static bool bfq_update_parent_budget(struct bfq_entity *next_in_service) |
| 239 | 224 | { |
| 240 | 225 | return false; |
| @@ -519,7 +504,7 @@ unsigned short bfq_ioprio_to_weight(int ioprio) · 278 unmodified lines | ||
| 519 | 504 | static unsigned short bfq_weight_to_ioprio(int weight) |
| 520 | 505 | { |
| 521 | 506 | return max_t(int, 0, |
| 522 | − IOPRIO_NR_LEVELS * BFQ_WEIGHT_CONVERSION_COEFF - weight); | |
| 507 | + IOPRIO_NR_LEVELS - weight / BFQ_WEIGHT_CONVERSION_COEFF); | |
| 523 | 508 | } |
| 524 | 509 | |
| 525 | 510 | static void bfq_get_entity(struct bfq_entity *entity) |
| @@ -420,7 +420,6 @@ int bio_integrity_clone(struct bio *bio, struct bio *bio_src, · 419 unmodified lines | ||
| 420 | 420 | |
| 421 | 421 | return 0; |
| 422 | 422 | } |
| 423 | −EXPORT_SYMBOL(bio_integrity_clone); | |
| 424 | 423 | |
| 425 | 424 | int bioset_integrity_create(struct bio_set *bs, int pool_size) |
| 426 | 425 | { |
| @@ -15,7 +15,6 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/mempool.h> |
| 16 | 16 | #include <linux/workqueue.h> |
| 17 | 17 | #include <linux/cgroup.h> |
| 18 | −#include <linux/blk-cgroup.h> | |
| 19 | 18 | #include <linux/highmem.h> |
| 20 | 19 | #include <linux/sched/sysctl.h> |
| 21 | 20 | #include <linux/blk-crypto.h> |
| @@ -24,6 +23,7 @@ · 2 unmodified lines | ||
| 24 | 23 | #include <trace/events/block.h> |
| 25 | 24 | #include "blk.h" |
| 26 | 25 | #include "blk-rq-qos.h" |
| 26 | +#include "blk-cgroup.h" | |
| 27 | 27 | |
| 28 | 28 | struct bio_alloc_cache { |
| 29 | 29 | struct bio *free_list; |
| @@ -249,12 +249,12 @@ static void bio_free(struct bio *bio) · 219 unmodified lines | ||
| 249 | 249 | * they must remember to pair any call to bio_init() with bio_uninit() |
| 250 | 250 | * when IO has completed, or when the bio is released. |
| 251 | 251 | */ |
| 252 | −void bio_init(struct bio *bio, struct bio_vec *table, | |
| 253 | − unsigned short max_vecs) | |
| 252 | +void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table, | |
| 253 | + unsigned short max_vecs, unsigned int opf) | |
| 254 | 254 | { |
| 255 | 255 | bio->bi_next = NULL; |
| 256 | − bio->bi_bdev = NULL; | |
| 257 | − bio->bi_opf = 0; | |
| 256 | + bio->bi_bdev = bdev; | |
| 257 | + bio->bi_opf = opf; | |
| 258 | 258 | bio->bi_flags = 0; |
| 259 | 259 | bio->bi_ioprio = 0; |
| 260 | 260 | bio->bi_write_hint = 0; |
| @@ -268,6 +268,8 @@ void bio_init(struct bio *bio, struct bio_vec *table, · 7 unmodified lines | ||
| 268 | 268 | #ifdef CONFIG_BLK_CGROUP |
| 269 | 269 | bio->bi_blkg = NULL; |
| 270 | 270 | bio->bi_issue.value = 0; |
| 271 | + if (bdev) | |
| 272 | + bio_associate_blkg(bio); | |
| 271 | 273 | #ifdef CONFIG_BLK_CGROUP_IOCOST |
| 272 | 274 | bio->bi_iocost_cost = 0; |
| 273 | 275 | #endif |
| @@ -293,6 +295,8 @@ EXPORT_SYMBOL(bio_init); · 19 unmodified lines | ||
| 293 | 295 | /** |
| 294 | 296 | * bio_reset - reinitialize a bio |
| 295 | 297 | * @bio: bio to reset |
| 298 | + * @bdev: block device to use the bio for | |
| 299 | + * @opf: operation and flags for bio | |
| 296 | 300 | * |
| 297 | 301 | * Description: |
| 298 | 302 | * After calling bio_reset(), @bio will be in the same state as a freshly |
| @@ -300,11 +304,15 @@ EXPORT_SYMBOL(bio_init); · 1 unmodified line | ||
| 300 | 304 | * preserved are the ones that are initialized by bio_alloc_bioset(). See |
| 301 | 305 | * comment in struct bio. |
| 302 | 306 | */ |
| 303 | −void bio_reset(struct bio *bio) | |
| 307 | +void bio_reset(struct bio *bio, struct block_device *bdev, unsigned int opf) | |
| 304 | 308 | { |
| 305 | 309 | bio_uninit(bio); |
| 306 | 310 | memset(bio, 0, BIO_RESET_BYTES); |
| 307 | 311 | atomic_set(&bio->__bi_remaining, 1); |
| 312 | + bio->bi_bdev = bdev; | |
| 313 | + if (bio->bi_bdev) | |
| 314 | + bio_associate_blkg(bio); | |
| 315 | + bio->bi_opf = opf; | |
| 308 | 316 | } |
| 309 | 317 | EXPORT_SYMBOL(bio_reset); |
| 310 | 318 | |
| @@ -344,6 +352,20 @@ void bio_chain(struct bio *bio, struct bio *parent) · 33 unmodified lines | ||
| 344 | 352 | } |
| 345 | 353 | EXPORT_SYMBOL(bio_chain); |
| 346 | 354 | |
| 355 | +struct bio *blk_next_bio(struct bio *bio, struct block_device *bdev, | |
| 356 | + unsigned int nr_pages, unsigned int opf, gfp_t gfp) | |
| 357 | +{ | |
| 358 | + struct bio *new = bio_alloc(bdev, nr_pages, opf, gfp); | |
| 359 | + | |
| 360 | + if (bio) { | |
| 361 | + bio_chain(bio, new); | |
| 362 | + submit_bio(bio); | |
| 363 | + } | |
| 364 | + | |
| 365 | + return new; | |
| 366 | +} | |
| 367 | +EXPORT_SYMBOL_GPL(blk_next_bio); | |
| 368 | + | |
| 347 | 369 | static void bio_alloc_rescue(struct work_struct *work) |
| 348 | 370 | { |
| 349 | 371 | struct bio_set *bs = container_of(work, struct bio_set, rescue_work); |
| @@ -400,8 +422,10 @@ static void punt_bios_to_rescuer(struct bio_set *bs) · 50 unmodified lines | ||
| 400 | 422 | |
| 401 | 423 | /** |
| 402 | 424 | * bio_alloc_bioset - allocate a bio for I/O |
| 425 | + * @bdev: block device to allocate the bio for (can be %NULL) | |
| 426 | + * @nr_vecs: number of bvecs to pre-allocate | |
| 427 | + * @opf: operation and flags for bio | |
| 403 | 428 | * @gfp_mask: the GFP_* mask given to the slab allocator |
| 404 | − * @nr_iovecs: number of iovecs to pre-allocate | |
| 405 | 429 | * @bs: the bio_set to allocate from. |
| 406 | 430 | * |
| 407 | 431 | * Allocate a bio from the mempools in @bs. |
| @@ -430,15 +454,16 @@ static void punt_bios_to_rescuer(struct bio_set *bs) · 22 unmodified lines | ||
| 430 | 454 | * |
| 431 | 455 | * Returns: Pointer to new bio on success, NULL on failure. |
| 432 | 456 | */ |
| 433 | −struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned short nr_iovecs, | |
| 457 | +struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, | |
| 458 | + unsigned int opf, gfp_t gfp_mask, | |
| 434 | 459 | struct bio_set *bs) |
| 435 | 460 | { |
| 436 | 461 | gfp_t saved_gfp = gfp_mask; |
| 437 | 462 | struct bio *bio; |
| 438 | 463 | void *p; |
| 439 | 464 | |
| 440 | − /* should not use nobvec bioset for nr_iovecs > 0 */ | |
| 441 | − if (WARN_ON_ONCE(!mempool_initialized(&bs->bvec_pool) && nr_iovecs > 0)) | |
| 465 | + /* should not use nobvec bioset for nr_vecs > 0 */ | |
| 466 | + if (WARN_ON_ONCE(!mempool_initialized(&bs->bvec_pool) && nr_vecs > 0)) | |
| 442 | 467 | return NULL; |
| 443 | 468 | |
| 444 | 469 | /* |
| @@ -475,23 +500,23 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, unsigned short nr_iovecs, · 30 unmodified lines | ||
| 475 | 500 | return NULL; |
| 476 | 501 | |
| 477 | 502 | bio = p + bs->front_pad; |
| 478 | − if (nr_iovecs > BIO_INLINE_VECS) { | |
| 503 | + if (nr_vecs > BIO_INLINE_VECS) { | |
| 479 | 504 | struct bio_vec *bvl = NULL; |
| 480 | 505 | |
| 481 | − bvl = bvec_alloc(&bs->bvec_pool, &nr_iovecs, gfp_mask); | |
| 506 | + bvl = bvec_alloc(&bs->bvec_pool, &nr_vecs, gfp_mask); | |
| 482 | 507 | if (!bvl && gfp_mask != saved_gfp) { |
| 483 | 508 | punt_bios_to_rescuer(bs); |
| 484 | 509 | gfp_mask = saved_gfp; |
| 485 | − bvl = bvec_alloc(&bs->bvec_pool, &nr_iovecs, gfp_mask); | |
| 510 | + bvl = bvec_alloc(&bs->bvec_pool, &nr_vecs, gfp_mask); | |
| 486 | 511 | } |
| 487 | 512 | if (unlikely(!bvl)) |
| 488 | 513 | goto err_free; |
| 489 | 514 | |
| 490 | − bio_init(bio, bvl, nr_iovecs); | |
| 491 | − } else if (nr_iovecs) { | |
| 492 | − bio_init(bio, bio->bi_inline_vecs, BIO_INLINE_VECS); | |
| 515 | + bio_init(bio, bdev, bvl, nr_vecs, opf); | |
| 516 | + } else if (nr_vecs) { | |
| 517 | + bio_init(bio, bdev, bio->bi_inline_vecs, BIO_INLINE_VECS, opf); | |
| 493 | 518 | } else { |
| 494 | − bio_init(bio, NULL, 0); | |
| 519 | + bio_init(bio, bdev, NULL, 0, opf); | |
| 495 | 520 | } |
| 496 | 521 | |
| 497 | 522 | bio->bi_pool = bs; |
| @@ -522,7 +547,8 @@ struct bio *bio_kmalloc(gfp_t gfp_mask, unsigned short nr_iovecs) · 24 unmodified lines | ||
| 522 | 547 | bio = kmalloc(struct_size(bio, bi_inline_vecs, nr_iovecs), gfp_mask); |
| 523 | 548 | if (unlikely(!bio)) |
| 524 | 549 | return NULL; |
| 525 | − bio_init(bio, nr_iovecs ? bio->bi_inline_vecs : NULL, nr_iovecs); | |
| 550 | + bio_init(bio, NULL, nr_iovecs ? bio->bi_inline_vecs : NULL, nr_iovecs, | |
| 551 | + 0); | |
| 526 | 552 | bio->bi_pool = NULL; |
| 527 | 553 | return bio; |
| 528 | 554 | } |
| @@ -702,80 +728,84 @@ void bio_put(struct bio *bio) · 173 unmodified lines | ||
| 702 | 728 | } |
| 703 | 729 | EXPORT_SYMBOL(bio_put); |
| 704 | 730 | |
| 705 | −/** | |
| 706 | − * __bio_clone_fast - clone a bio that shares the original bio's biovec | |
| 707 | − * @bio: destination bio | |
| 708 | − * @bio_src: bio to clone | |
| 709 | − * | |
| 710 | − * Clone a &bio. Caller will own the returned bio, but not | |
| 711 | − * the actual data it points to. Reference count of returned | |
| 712 | − * bio will be one. | |
| 713 | − * | |
| 714 | − * Caller must ensure that @bio_src is not freed before @bio. | |
| 715 | − */ | |
| 716 | −void __bio_clone_fast(struct bio *bio, struct bio *bio_src) | |
| 731 | +static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp) | |
| 717 | 732 | { |
| 718 | − WARN_ON_ONCE(bio->bi_pool && bio->bi_max_vecs); | |
| 719 | − | |
| 720 | − /* | |
| 721 | − * most users will be overriding ->bi_bdev with a new target, | |
| 722 | − * so we don't set nor calculate new physical/hw segment counts here | |
| 723 | − */ | |
| 724 | − bio->bi_bdev = bio_src->bi_bdev; | |
| 725 | 733 | bio_set_flag(bio, BIO_CLONED); |
| 726 | 734 | if (bio_flagged(bio_src, BIO_THROTTLED)) |
| 727 | 735 | bio_set_flag(bio, BIO_THROTTLED); |
| 728 | − if (bio_flagged(bio_src, BIO_REMAPPED)) | |
| 736 | + if (bio->bi_bdev == bio_src->bi_bdev && | |
| 737 | + bio_flagged(bio_src, BIO_REMAPPED)) | |
| 729 | 738 | bio_set_flag(bio, BIO_REMAPPED); |
| 730 | − bio->bi_opf = bio_src->bi_opf; | |
| 731 | 739 | bio->bi_ioprio = bio_src->bi_ioprio; |
| 732 | 740 | bio->bi_write_hint = bio_src->bi_write_hint; |
| 733 | 741 | bio->bi_iter = bio_src->bi_iter; |
| 734 | − bio->bi_io_vec = bio_src->bi_io_vec; | |
| 735 | 742 | |
| 736 | 743 | bio_clone_blkg_association(bio, bio_src); |
| 737 | 744 | blkcg_bio_issue_init(bio); |
| 745 | + | |
| 746 | + if (bio_crypt_clone(bio, bio_src, gfp) < 0) | |
| 747 | + return -ENOMEM; | |
| 748 | + if (bio_integrity(bio_src) && | |
| 749 | + bio_integrity_clone(bio, bio_src, gfp) < 0) | |
| 750 | + return -ENOMEM; | |
| 751 | + return 0; | |
| 738 | 752 | } |
| 739 | −EXPORT_SYMBOL(__bio_clone_fast); | |
| 740 | 753 | |
| 741 | 754 | /** |
| 742 | − * bio_clone_fast - clone a bio that shares the original bio's biovec | |
| 743 | − * @bio: bio to clone | |
| 744 | − * @gfp_mask: allocation priority | |
| 745 | − * @bs: bio_set to allocate from | |
| 755 | + * bio_alloc_clone - clone a bio that shares the original bio's biovec | |
| 756 | + * @bdev: block_device to clone onto | |
| 757 | + * @bio_src: bio to clone from | |
| 758 | + * @gfp: allocation priority | |
| 759 | + * @bs: bio_set to allocate from | |
| 760 | + * | |
| 761 | + * Allocate a new bio that is a clone of @bio_src. The caller owns the returned | |
| 762 | + * bio, but not the actual data it points to. | |
| 746 | 763 | * |
| 747 | − * Like __bio_clone_fast, only also allocates the returned bio | |
| 764 | + * The caller must ensure that the return bio is not freed before @bio_src. | |
| 748 | 765 | */ |
| 749 | −struct bio *bio_clone_fast(struct bio *bio, gfp_t gfp_mask, struct bio_set *bs) | |
| 766 | +struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src, | |
| 767 | + gfp_t gfp, struct bio_set *bs) | |
| 750 | 768 | { |
| 751 | − struct bio *b; | |
| 769 | + struct bio *bio; | |
| 752 | 770 | |
| 753 | − b = bio_alloc_bioset(gfp_mask, 0, bs); | |
| 754 | − if (!b) | |
| 771 | + bio = bio_alloc_bioset(bdev, 0, bio_src->bi_opf, gfp, bs); | |
| 772 | + if (!bio) | |
| 755 | 773 | return NULL; |
| 756 | 774 | |
| 757 | − __bio_clone_fast(b, bio); | |
| 758 | − | |
| 759 | − if (bio_crypt_clone(b, bio, gfp_mask) < 0) | |
| 760 | − goto err_put; | |
| 761 | − | |
| 762 | − if (bio_integrity(bio) && | |
| 763 | − bio_integrity_clone(b, bio, gfp_mask) < 0) | |
| 764 | − goto err_put; | |
| 765 | − | |
| 766 | − return b; | |
| 775 | + if (__bio_clone(bio, bio_src, gfp) < 0) { | |
| 776 | + bio_put(bio); | |
| 777 | + return NULL; | |
| 778 | + } | |
| 779 | + bio->bi_io_vec = bio_src->bi_io_vec; | |
| 767 | 780 | |
| 768 | −err_put: | |
| 769 | − bio_put(b); | |
| 770 | − return NULL; | |
| 781 | + return bio; | |
| 771 | 782 | } |
| 772 | −EXPORT_SYMBOL(bio_clone_fast); | |
| 783 | +EXPORT_SYMBOL(bio_alloc_clone); | |
| 773 | 784 | |
| 774 | −const char *bio_devname(struct bio *bio, char *buf) | |
| 785 | +/** | |
| 786 | + * bio_init_clone - clone a bio that shares the original bio's biovec | |
| 787 | + * @bdev: block_device to clone onto | |
| 788 | + * @bio: bio to clone into | |
| 789 | + * @bio_src: bio to clone from | |
| 790 | + * @gfp: allocation priority | |
| 791 | + * | |
| 792 | + * Initialize a new bio in caller provided memory that is a clone of @bio_src. | |
| 793 | + * The caller owns the returned bio, but not the actual data it points to. | |
| 794 | + * | |
| 795 | + * The caller must ensure that @bio_src is not freed before @bio. | |
| 796 | + */ | |
| 797 | +int bio_init_clone(struct block_device *bdev, struct bio *bio, | |
| 798 | + struct bio *bio_src, gfp_t gfp) | |
| 775 | 799 | { |
| 776 | − return bdevname(bio->bi_bdev, buf); | |
| 800 | + int ret; | |
| 801 | + | |
| 802 | + bio_init(bio, bdev, bio_src->bi_io_vec, 0, bio_src->bi_opf); | |
| 803 | + ret = __bio_clone(bio, bio_src, gfp); | |
| 804 | + if (ret) | |
| 805 | + bio_uninit(bio); | |
| 806 | + return ret; | |
| 777 | 807 | } |
| 778 | −EXPORT_SYMBOL(bio_devname); | |
| 808 | +EXPORT_SYMBOL(bio_init_clone); | |
| 779 | 809 | |
| 780 | 810 | /** |
| 781 | 811 | * bio_full - check if the bio is full |
| @@ -1054,7 +1084,7 @@ bool bio_add_folio(struct bio *bio, struct folio *folio, size_t len, · 272 unmodified lines | ||
| 1054 | 1084 | size_t off) |
| 1055 | 1085 | { |
| 1056 | 1086 | if (len > UINT_MAX || off > UINT_MAX) |
| 1057 | − return 0; | |
| 1087 | + return false; | |
| 1058 | 1088 | return bio_add_page(bio, &folio->page, len, off) > 0; |
| 1059 | 1089 | } |
| 1060 | 1090 | |
| @@ -1486,8 +1516,7 @@ again: · 425 unmodified lines | ||
| 1486 | 1516 | if (!bio_integrity_endio(bio)) |
| 1487 | 1517 | return; |
| 1488 | 1518 | |
| 1489 | − if (bio->bi_bdev && bio_flagged(bio, BIO_TRACKED)) | |
| 1490 | − rq_qos_done_bio(bdev_get_queue(bio->bi_bdev), bio); | |
| 1519 | + rq_qos_done_bio(bio); | |
| 1491 | 1520 | |
| 1492 | 1521 | if (bio->bi_bdev && bio_flagged(bio, BIO_TRACE_COMPLETION)) { |
| 1493 | 1522 | trace_block_bio_complete(bdev_get_queue(bio->bi_bdev), bio); |
| @@ -1541,7 +1570,7 @@ struct bio *bio_split(struct bio *bio, int sectors, · 47 unmodified lines | ||
| 1541 | 1570 | if (WARN_ON_ONCE(bio_op(bio) == REQ_OP_ZONE_APPEND)) |
| 1542 | 1571 | return NULL; |
| 1543 | 1572 | |
| 1544 | − split = bio_clone_fast(bio, gfp, bs); | |
| 1573 | + split = bio_alloc_clone(bio->bi_bdev, bio, gfp, bs); | |
| 1545 | 1574 | if (!split) |
| 1546 | 1575 | return NULL; |
| 1547 | 1576 | |
| @@ -1636,9 +1665,9 @@ EXPORT_SYMBOL(bioset_exit); · 88 unmodified lines | ||
| 1636 | 1665 | * Note that the bio must be embedded at the END of that structure always, |
| 1637 | 1666 | * or things will break badly. |
| 1638 | 1667 | * If %BIOSET_NEED_BVECS is set in @flags, a separate pool will be allocated |
| 1639 | − * for allocating iovecs. This pool is not needed e.g. for bio_clone_fast(). | |
| 1640 | − * If %BIOSET_NEED_RESCUER is set, a workqueue is created which can be used to | |
| 1641 | − * dispatch queued requests when the mempool runs out of space. | |
| 1668 | + * for allocating iovecs. This pool is not needed e.g. for bio_init_clone(). | |
| 1669 | + * If %BIOSET_NEED_RESCUER is set, a workqueue is created which can be used | |
| 1670 | + * to dispatch queued requests when the mempool runs out of space. | |
| 1642 | 1671 | * |
| 1643 | 1672 | */ |
| 1644 | 1673 | int bioset_init(struct bio_set *bs, |
| @@ -1708,7 +1737,9 @@ EXPORT_SYMBOL(bioset_init_from_src); · 63 unmodified lines | ||
| 1708 | 1737 | /** |
| 1709 | 1738 | * bio_alloc_kiocb - Allocate a bio from bio_set based on kiocb |
| 1710 | 1739 | * @kiocb: kiocb describing the IO |
| 1740 | + * @bdev: block device to allocate the bio for (can be %NULL) | |
| 1711 | 1741 | * @nr_vecs: number of iovecs to pre-allocate |
| 1742 | + * @opf: operation and flags for bio | |
| 1712 | 1743 | * @bs: bio_set to allocate from |
| 1713 | 1744 | * |
| 1714 | 1745 | * Description: |
| @@ -1719,14 +1750,14 @@ EXPORT_SYMBOL(bioset_init_from_src); · 4 unmodified lines | ||
| 1719 | 1750 | * MUST be done from process context, not hard/soft IRQ. |
| 1720 | 1751 | * |
| 1721 | 1752 | */ |
| 1722 | −struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, | |
| 1723 | − struct bio_set *bs) | |
| 1753 | +struct bio *bio_alloc_kiocb(struct kiocb *kiocb, struct block_device *bdev, | |
| 1754 | + unsigned short nr_vecs, unsigned int opf, struct bio_set *bs) | |
| 1724 | 1755 | { |
| 1725 | 1756 | struct bio_alloc_cache *cache; |
| 1726 | 1757 | struct bio *bio; |
| 1727 | 1758 | |
| 1728 | 1759 | if (!(kiocb->ki_flags & IOCB_ALLOC_CACHE) || nr_vecs > BIO_INLINE_VECS) |
| 1729 | − return bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs); | |
| 1760 | + return bio_alloc_bioset(bdev, nr_vecs, opf, GFP_KERNEL, bs); | |
| 1730 | 1761 | |
| 1731 | 1762 | cache = per_cpu_ptr(bs->cache, get_cpu()); |
| 1732 | 1763 | if (cache->free_list) { |
| @@ -1734,13 +1765,14 @@ struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, · 1 unmodified line | ||
| 1734 | 1765 | cache->free_list = bio->bi_next; |
| 1735 | 1766 | cache->nr--; |
| 1736 | 1767 | put_cpu(); |
| 1737 | − bio_init(bio, nr_vecs ? bio->bi_inline_vecs : NULL, nr_vecs); | |
| 1768 | + bio_init(bio, bdev, nr_vecs ? bio->bi_inline_vecs : NULL, | |
| 1769 | + nr_vecs, opf); | |
| 1738 | 1770 | bio->bi_pool = bs; |
| 1739 | 1771 | bio_set_flag(bio, BIO_PERCPU_CACHE); |
| 1740 | 1772 | return bio; |
| 1741 | 1773 | } |
| 1742 | 1774 | put_cpu(); |
| 1743 | − bio = bio_alloc_bioset(GFP_KERNEL, nr_vecs, bs); | |
| 1775 | + bio = bio_alloc_bioset(bdev, nr_vecs, opf, GFP_KERNEL, bs); | |
| 1744 | 1776 | bio_set_flag(bio, BIO_PERCPU_CACHE); |
| 1745 | 1777 | return bio; |
| 1746 | 1778 | } |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #ifndef _BLK_CGROUP_RWSTAT_H |
| 7 | 7 | #define _BLK_CGROUP_RWSTAT_H |
| 8 | 8 | |
| 9 | −#include <linux/blk-cgroup.h> | |
| 9 | +#include "blk-cgroup.h" | |
| 10 | 10 | |
| 11 | 11 | enum blkg_rwstat_type { |
| 12 | 12 | BLKG_RWSTAT_READ, |
| @@ -23,15 +23,14 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <linux/blkdev.h> |
| 24 | 24 | #include <linux/backing-dev.h> |
| 25 | 25 | #include <linux/slab.h> |
| 26 | −#include <linux/genhd.h> | |
| 27 | 26 | #include <linux/delay.h> |
| 28 | 27 | #include <linux/atomic.h> |
| 29 | 28 | #include <linux/ctype.h> |
| 30 | −#include <linux/blk-cgroup.h> | |
| 31 | 29 | #include <linux/tracehook.h> |
| 32 | 30 | #include <linux/psi.h> |
| 33 | 31 | #include <linux/part_stat.h> |
| 34 | 32 | #include "blk.h" |
| 33 | +#include "blk-cgroup.h" | |
| 35 | 34 | #include "blk-ioprio.h" |
| 36 | 35 | #include "blk-throttle.h" |
| 37 | 36 | |
| @@ -83,6 +82,8 @@ static void blkg_free(struct blkcg_gq *blkg) · 45 unmodified lines | ||
| 83 | 82 | if (blkg->pd[i]) |
| 84 | 83 | blkcg_policy[i]->pd_free_fn(blkg->pd[i]); |
| 85 | 84 | |
| 85 | + if (blkg->q) | |
| 86 | + blk_put_queue(blkg->q); | |
| 86 | 87 | free_percpu(blkg->iostat_cpu); |
| 87 | 88 | percpu_ref_exit(&blkg->refcnt); |
| 88 | 89 | kfree(blkg); |
| @@ -168,6 +169,9 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct request_queue *q, · 79 unmodified lines | ||
| 168 | 169 | if (!blkg->iostat_cpu) |
| 169 | 170 | goto err_free; |
| 170 | 171 | |
| 172 | + if (!blk_get_queue(q)) | |
| 173 | + goto err_free; | |
| 174 | + | |
| 171 | 175 | blkg->q = q; |
| 172 | 176 | INIT_LIST_HEAD(&blkg->q_node); |
| 173 | 177 | spin_lock_init(&blkg->async_bio_lock); |
| @@ -857,11 +861,11 @@ static void blkcg_fill_root_iostats(void) · 683 unmodified lines | ||
| 857 | 861 | blk_queue_root_blkg(bdev_get_queue(bdev)); |
| 858 | 862 | struct blkg_iostat tmp; |
| 859 | 863 | int cpu; |
| 864 | + unsigned long flags; | |
| 860 | 865 | |
| 861 | 866 | memset(&tmp, 0, sizeof(tmp)); |
| 862 | 867 | for_each_possible_cpu(cpu) { |
| 863 | 868 | struct disk_stats *cpu_dkstats; |
| 864 | − unsigned long flags; | |
| 865 | 869 | |
| 866 | 870 | cpu_dkstats = per_cpu_ptr(bdev->bd_stats, cpu); |
| 867 | 871 | tmp.ios[BLKG_IOSTAT_READ] += |
| @@ -877,11 +881,11 @@ static void blkcg_fill_root_iostats(void) · 9 unmodified lines | ||
| 877 | 881 | cpu_dkstats->sectors[STAT_WRITE] << 9; |
| 878 | 882 | tmp.bytes[BLKG_IOSTAT_DISCARD] += |
| 879 | 883 | cpu_dkstats->sectors[STAT_DISCARD] << 9; |
| 880 | − | |
| 881 | − flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync); | |
| 882 | − blkg_iostat_set(&blkg->iostat.cur, &tmp); | |
| 883 | − u64_stats_update_end_irqrestore(&blkg->iostat.sync, flags); | |
| 884 | 884 | } |
| 885 | + | |
| 886 | + flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync); | |
| 887 | + blkg_iostat_set(&blkg->iostat.cur, &tmp); | |
| 888 | + u64_stats_update_end_irqrestore(&blkg->iostat.sync, flags); | |
| 885 | 889 | } |
| 886 | 890 | } |
| 887 | 891 | |
| @@ -1176,6 +1180,8 @@ int blkcg_init_queue(struct request_queue *q) · 288 unmodified lines | ||
| 1176 | 1180 | bool preloaded; |
| 1177 | 1181 | int ret; |
| 1178 | 1182 | |
| 1183 | + INIT_LIST_HEAD(&q->blkg_list); | |
| 1184 | + | |
| 1179 | 1185 | new_blkg = blkg_alloc(&blkcg_root, q, GFP_KERNEL); |
| 1180 | 1186 | if (!new_blkg) |
| 1181 | 1187 | return -ENOMEM; |
| @@ -0,0 +1,494 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +#ifndef _BLK_CGROUP_PRIVATE_H | |
| 3 | +#define _BLK_CGROUP_PRIVATE_H | |
| 4 | +/* | |
| 5 | + * block cgroup private header | |
| 6 | + * | |
| 7 | + * Based on ideas and code from CFQ, CFS and BFQ: | |
| 8 | + * Copyright (C) 2003 Jens Axboe <axboe@kernel.dk> | |
| 9 | + * | |
| 10 | + * Copyright (C) 2008 Fabio Checconi <fabio@gandalf.sssup.it> | |
| 11 | + * Paolo Valente <paolo.valente@unimore.it> | |
| 12 | + * | |
| 13 | + * Copyright (C) 2009 Vivek Goyal <vgoyal@redhat.com> | |
| 14 | + * Nauman Rafique <nauman@google.com> | |
| 15 | + */ | |
| 16 | + | |
| 17 | +#include <linux/blk-cgroup.h> | |
| 18 | +#include <linux/blk-mq.h> | |
| 19 | + | |
| 20 | +/* percpu_counter batch for blkg_[rw]stats, per-cpu drift doesn't matter */ | |
| 21 | +#define BLKG_STAT_CPU_BATCH (INT_MAX / 2) | |
| 22 | + | |
| 23 | +#ifdef CONFIG_BLK_CGROUP | |
| 24 | + | |
| 25 | +/* | |
| 26 | + * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a | |
| 27 | + * request_queue (q). This is used by blkcg policies which need to track | |
| 28 | + * information per blkcg - q pair. | |
| 29 | + * | |
| 30 | + * There can be multiple active blkcg policies and each blkg:policy pair is | |
| 31 | + * represented by a blkg_policy_data which is allocated and freed by each | |
| 32 | + * policy's pd_alloc/free_fn() methods. A policy can allocate private data | |
| 33 | + * area by allocating larger data structure which embeds blkg_policy_data | |
| 34 | + * at the beginning. | |
| 35 | + */ | |
| 36 | +struct blkg_policy_data { | |
| 37 | + /* the blkg and policy id this per-policy data belongs to */ | |
| 38 | + struct blkcg_gq *blkg; | |
| 39 | + int plid; | |
| 40 | +}; | |
| 41 | + | |
| 42 | +/* | |
| 43 | + * Policies that need to keep per-blkcg data which is independent from any | |
| 44 | + * request_queue associated to it should implement cpd_alloc/free_fn() | |
| 45 | + * methods. A policy can allocate private data area by allocating larger | |
| 46 | + * data structure which embeds blkcg_policy_data at the beginning. | |
| 47 | + * cpd_init() is invoked to let each policy handle per-blkcg data. | |
| 48 | + */ | |
| 49 | +struct blkcg_policy_data { | |
| 50 | + /* the blkcg and policy id this per-policy data belongs to */ | |
| 51 | + struct blkcg *blkcg; | |
| 52 | + int plid; | |
| 53 | +}; | |
| 54 | + | |
| 55 | +typedef struct blkcg_policy_data *(blkcg_pol_alloc_cpd_fn)(gfp_t gfp); | |
| 56 | +typedef void (blkcg_pol_init_cpd_fn)(struct blkcg_policy_data *cpd); | |
| 57 | +typedef void (blkcg_pol_free_cpd_fn)(struct blkcg_policy_data *cpd); | |
| 58 | +typedef void (blkcg_pol_bind_cpd_fn)(struct blkcg_policy_data *cpd); | |
| 59 | +typedef struct blkg_policy_data *(blkcg_pol_alloc_pd_fn)(gfp_t gfp, | |
| 60 | + struct request_queue *q, struct blkcg *blkcg); | |
| 61 | +typedef void (blkcg_pol_init_pd_fn)(struct blkg_policy_data *pd); | |
| 62 | +typedef void (blkcg_pol_online_pd_fn)(struct blkg_policy_data *pd); | |
| 63 | +typedef void (blkcg_pol_offline_pd_fn)(struct blkg_policy_data *pd); | |
| 64 | +typedef void (blkcg_pol_free_pd_fn)(struct blkg_policy_data *pd); | |
| 65 | +typedef void (blkcg_pol_reset_pd_stats_fn)(struct blkg_policy_data *pd); | |
| 66 | +typedef bool (blkcg_pol_stat_pd_fn)(struct blkg_policy_data *pd, | |
| 67 | + struct seq_file *s); | |
| 68 | + | |
| 69 | +struct blkcg_policy { | |
| 70 | + int plid; | |
| 71 | + /* cgroup files for the policy */ | |
| 72 | + struct cftype *dfl_cftypes; | |
| 73 | + struct cftype *legacy_cftypes; | |
| 74 | + | |
| 75 | + /* operations */ | |
| 76 | + blkcg_pol_alloc_cpd_fn *cpd_alloc_fn; | |
| 77 | + blkcg_pol_init_cpd_fn *cpd_init_fn; | |
| 78 | + blkcg_pol_free_cpd_fn *cpd_free_fn; | |
| 79 | + blkcg_pol_bind_cpd_fn *cpd_bind_fn; | |
| 80 | + | |
| 81 | + blkcg_pol_alloc_pd_fn *pd_alloc_fn; | |
| 82 | + blkcg_pol_init_pd_fn *pd_init_fn; | |
| 83 | + blkcg_pol_online_pd_fn *pd_online_fn; | |
| 84 | + blkcg_pol_offline_pd_fn *pd_offline_fn; | |
| 85 | + blkcg_pol_free_pd_fn *pd_free_fn; | |
| 86 | + blkcg_pol_reset_pd_stats_fn *pd_reset_stats_fn; | |
| 87 | + blkcg_pol_stat_pd_fn *pd_stat_fn; | |
| 88 | +}; | |
| 89 | + | |
| 90 | +extern struct blkcg blkcg_root; | |
| 91 | +extern bool blkcg_debug_stats; | |
| 92 | + | |
| 93 | +struct blkcg_gq *blkg_lookup_slowpath(struct blkcg *blkcg, | |
| 94 | + struct request_queue *q, bool update_hint); | |
| 95 | +int blkcg_init_queue(struct request_queue *q); | |
| 96 | +void blkcg_exit_queue(struct request_queue *q); | |
| 97 | + | |
| 98 | +/* Blkio controller policy registration */ | |
| 99 | +int blkcg_policy_register(struct blkcg_policy *pol); | |
| 100 | +void blkcg_policy_unregister(struct blkcg_policy *pol); | |
| 101 | +int blkcg_activate_policy(struct request_queue *q, | |
| 102 | + const struct blkcg_policy *pol); | |
| 103 | +void blkcg_deactivate_policy(struct request_queue *q, | |
| 104 | + const struct blkcg_policy *pol); | |
| 105 | + | |
| 106 | +const char *blkg_dev_name(struct blkcg_gq *blkg); | |
| 107 | +void blkcg_print_blkgs(struct seq_file *sf, struct blkcg *blkcg, | |
| 108 | + u64 (*prfill)(struct seq_file *, | |
| 109 | + struct blkg_policy_data *, int), | |
| 110 | + const struct blkcg_policy *pol, int data, | |
| 111 | + bool show_total); | |
| 112 | +u64 __blkg_prfill_u64(struct seq_file *sf, struct blkg_policy_data *pd, u64 v); | |
| 113 | + | |
| 114 | +struct blkg_conf_ctx { | |
| 115 | + struct block_device *bdev; | |
| 116 | + struct blkcg_gq *blkg; | |
| 117 | + char *body; | |
| 118 | +}; | |
| 119 | + | |
| 120 | +struct block_device *blkcg_conf_open_bdev(char **inputp); | |
| 121 | +int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, | |
| 122 | + char *input, struct blkg_conf_ctx *ctx); | |
| 123 | +void blkg_conf_finish(struct blkg_conf_ctx *ctx); | |
| 124 | + | |
| 125 | +/** | |
| 126 | + * blkcg_css - find the current css | |
| 127 | + * | |
| 128 | + * Find the css associated with either the kthread or the current task. | |
| 129 | + * This may return a dying css, so it is up to the caller to use tryget logic | |
| 130 | + * to confirm it is alive and well. | |
| 131 | + */ | |
| 132 | +static inline struct cgroup_subsys_state *blkcg_css(void) | |
| 133 | +{ | |
| 134 | + struct cgroup_subsys_state *css; | |
| 135 | + | |
| 136 | + css = kthread_blkcg(); | |
| 137 | + if (css) | |
| 138 | + return css; | |
| 139 | + return task_css(current, io_cgrp_id); | |
| 140 | +} | |
| 141 | + | |
| 142 | +/** | |
| 143 | + * __bio_blkcg - internal, inconsistent version to get blkcg | |
| 144 | + * | |
| 145 | + * DO NOT USE. | |
| 146 | + * This function is inconsistent and consequently is dangerous to use. The | |
| 147 | + * first part of the function returns a blkcg where a reference is owned by the | |
| 148 | + * bio. This means it does not need to be rcu protected as it cannot go away | |
| 149 | + * with the bio owning a reference to it. However, the latter potentially gets | |
| 150 | + * it from task_css(). This can race against task migration and the cgroup | |
| 151 | + * dying. It is also semantically different as it must be called rcu protected | |
| 152 | + * and is susceptible to failure when trying to get a reference to it. | |
| 153 | + * Therefore, it is not ok to assume that *_get() will always succeed on the | |
| 154 | + * blkcg returned here. | |
| 155 | + */ | |
| 156 | +static inline struct blkcg *__bio_blkcg(struct bio *bio) | |
| 157 | +{ | |
| 158 | + if (bio && bio->bi_blkg) | |
| 159 | + return bio->bi_blkg->blkcg; | |
| 160 | + return css_to_blkcg(blkcg_css()); | |
| 161 | +} | |
| 162 | + | |
| 163 | +/** | |
| 164 | + * bio_issue_as_root_blkg - see if this bio needs to be issued as root blkg | |
| 165 | + * @return: true if this bio needs to be submitted with the root blkg context. | |
| 166 | + * | |
| 167 | + * In order to avoid priority inversions we sometimes need to issue a bio as if | |
| 168 | + * it were attached to the root blkg, and then backcharge to the actual owning | |
| 169 | + * blkg. The idea is we do bio_blkcg() to look up the actual context for the | |
| 170 | + * bio and attach the appropriate blkg to the bio. Then we call this helper and | |
| 171 | + * if it is true run with the root blkg for that queue and then do any | |
| 172 | + * backcharging to the originating cgroup once the io is complete. | |
| 173 | + */ | |
| 174 | +static inline bool bio_issue_as_root_blkg(struct bio *bio) | |
| 175 | +{ | |
| 176 | + return (bio->bi_opf & (REQ_META | REQ_SWAP)) != 0; | |
| 177 | +} | |
| 178 | + | |
| 179 | +/** | |
| 180 | + * __blkg_lookup - internal version of blkg_lookup() | |
| 181 | + * @blkcg: blkcg of interest | |
| 182 | + * @q: request_queue of interest | |
| 183 | + * @update_hint: whether to update lookup hint with the result or not | |
| 184 | + * | |
| 185 | + * This is internal version and shouldn't be used by policy | |
| 186 | + * implementations. Looks up blkgs for the @blkcg - @q pair regardless of | |
| 187 | + * @q's bypass state. If @update_hint is %true, the caller should be | |
| 188 | + * holding @q->queue_lock and lookup hint is updated on success. | |
| 189 | + */ | |
| 190 | +static inline struct blkcg_gq *__blkg_lookup(struct blkcg *blkcg, | |
| 191 | + struct request_queue *q, | |
| 192 | + bool update_hint) | |
| 193 | +{ | |
| 194 | + struct blkcg_gq *blkg; | |
| 195 | + | |
| 196 | + if (blkcg == &blkcg_root) | |
| 197 | + return q->root_blkg; | |
| 198 | + | |
| 199 | + blkg = rcu_dereference(blkcg->blkg_hint); | |
| 200 | + if (blkg && blkg->q == q) | |
| 201 | + return blkg; | |
| 202 | + | |
| 203 | + return blkg_lookup_slowpath(blkcg, q, update_hint); | |
| 204 | +} | |
| 205 | + | |
| 206 | +/** | |
| 207 | + * blkg_lookup - lookup blkg for the specified blkcg - q pair | |
| 208 | + * @blkcg: blkcg of interest | |
| 209 | + * @q: request_queue of interest | |
| 210 | + * | |
| 211 | + * Lookup blkg for the @blkcg - @q pair. This function should be called | |
| 212 | + * under RCU read lock. | |
| 213 | + */ | |
| 214 | +static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, | |
| 215 | + struct request_queue *q) | |
| 216 | +{ | |
| 217 | + WARN_ON_ONCE(!rcu_read_lock_held()); | |
| 218 | + return __blkg_lookup(blkcg, q, false); | |
| 219 | +} | |
| 220 | + | |
| 221 | +/** | |
| 222 | + * blk_queue_root_blkg - return blkg for the (blkcg_root, @q) pair | |
| 223 | + * @q: request_queue of interest | |
| 224 | + * | |
| 225 | + * Lookup blkg for @q at the root level. See also blkg_lookup(). | |
| 226 | + */ | |
| 227 | +static inline struct blkcg_gq *blk_queue_root_blkg(struct request_queue *q) | |
| 228 | +{ | |
| 229 | + return q->root_blkg; | |
| 230 | +} | |
| 231 | + | |
| 232 | +/** | |
| 233 | + * blkg_to_pdata - get policy private data | |
| 234 | + * @blkg: blkg of interest | |
| 235 | + * @pol: policy of interest | |
| 236 | + * | |
| 237 | + * Return pointer to private data associated with the @blkg-@pol pair. | |
| 238 | + */ | |
| 239 | +static inline struct blkg_policy_data *blkg_to_pd(struct blkcg_gq *blkg, | |
| 240 | + struct blkcg_policy *pol) | |
| 241 | +{ | |
| 242 | + return blkg ? blkg->pd[pol->plid] : NULL; | |
| 243 | +} | |
| 244 | + | |
| 245 | +static inline struct blkcg_policy_data *blkcg_to_cpd(struct blkcg *blkcg, | |
| 246 | + struct blkcg_policy *pol) | |
| 247 | +{ | |
| 248 | + return blkcg ? blkcg->cpd[pol->plid] : NULL; | |
| 249 | +} | |
| 250 | + | |
| 251 | +/** | |
| 252 | + * pdata_to_blkg - get blkg associated with policy private data | |
| 253 | + * @pd: policy private data of interest | |
| 254 | + * | |
| 255 | + * @pd is policy private data. Determine the blkg it's associated with. | |
| 256 | + */ | |
| 257 | +static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) | |
| 258 | +{ | |
| 259 | + return pd ? pd->blkg : NULL; | |
| 260 | +} | |
| 261 | + | |
| 262 | +static inline struct blkcg *cpd_to_blkcg(struct blkcg_policy_data *cpd) | |
| 263 | +{ | |
| 264 | + return cpd ? cpd->blkcg : NULL; | |
| 265 | +} | |
| 266 | + | |
| 267 | +/** | |
| 268 | + * blkg_path - format cgroup path of blkg | |
| 269 | + * @blkg: blkg of interest | |
| 270 | + * @buf: target buffer | |
| 271 | + * @buflen: target buffer length | |
| 272 | + * | |
| 273 | + * Format the path of the cgroup of @blkg into @buf. | |
| 274 | + */ | |
| 275 | +static inline int blkg_path(struct blkcg_gq *blkg, char *buf, int buflen) | |
| 276 | +{ | |
| 277 | + return cgroup_path(blkg->blkcg->css.cgroup, buf, buflen); | |
| 278 | +} | |
| 279 | + | |
| 280 | +/** | |
| 281 | + * blkg_get - get a blkg reference | |
| 282 | + * @blkg: blkg to get | |
| 283 | + * | |
| 284 | + * The caller should be holding an existing reference. | |
| 285 | + */ | |
| 286 | +static inline void blkg_get(struct blkcg_gq *blkg) | |
| 287 | +{ | |
| 288 | + percpu_ref_get(&blkg->refcnt); | |
| 289 | +} | |
| 290 | + | |
| 291 | +/** | |
| 292 | + * blkg_tryget - try and get a blkg reference | |
| 293 | + * @blkg: blkg to get | |
| 294 | + * | |
| 295 | + * This is for use when doing an RCU lookup of the blkg. We may be in the midst | |
| 296 | + * of freeing this blkg, so we can only use it if the refcnt is not zero. | |
| 297 | + */ | |
| 298 | +static inline bool blkg_tryget(struct blkcg_gq *blkg) | |
| 299 | +{ | |
| 300 | + return blkg && percpu_ref_tryget(&blkg->refcnt); | |
| 301 | +} | |
| 302 | + | |
| 303 | +/** | |
| 304 | + * blkg_put - put a blkg reference | |
| 305 | + * @blkg: blkg to put | |
| 306 | + */ | |
| 307 | +static inline void blkg_put(struct blkcg_gq *blkg) | |
| 308 | +{ | |
| 309 | + percpu_ref_put(&blkg->refcnt); | |
| 310 | +} | |
| 311 | + | |
| 312 | +/** | |
| 313 | + * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants | |
| 314 | + * @d_blkg: loop cursor pointing to the current descendant | |
| 315 | + * @pos_css: used for iteration | |
| 316 | + * @p_blkg: target blkg to walk descendants of | |
| 317 | + * | |
| 318 | + * Walk @c_blkg through the descendants of @p_blkg. Must be used with RCU | |
| 319 | + * read locked. If called under either blkcg or queue lock, the iteration | |
| 320 | + * is guaranteed to include all and only online blkgs. The caller may | |
| 321 | + * update @pos_css by calling css_rightmost_descendant() to skip subtree. | |
| 322 | + * @p_blkg is included in the iteration and the first node to be visited. | |
| 323 | + */ | |
| 324 | +#define blkg_for_each_descendant_pre(d_blkg, pos_css, p_blkg) \ | |
| 325 | + css_for_each_descendant_pre((pos_css), &(p_blkg)->blkcg->css) \ | |
| 326 | + if (((d_blkg) = __blkg_lookup(css_to_blkcg(pos_css), \ | |
| 327 | + (p_blkg)->q, false))) | |
| 328 | + | |
| 329 | +/** | |
| 330 | + * blkg_for_each_descendant_post - post-order walk of a blkg's descendants | |
| 331 | + * @d_blkg: loop cursor pointing to the current descendant | |
| 332 | + * @pos_css: used for iteration | |
| 333 | + * @p_blkg: target blkg to walk descendants of | |
| 334 | + * | |
| 335 | + * Similar to blkg_for_each_descendant_pre() but performs post-order | |
| 336 | + * traversal instead. Synchronization rules are the same. @p_blkg is | |
| 337 | + * included in the iteration and the last node to be visited. | |
| 338 | + */ | |
| 339 | +#define blkg_for_each_descendant_post(d_blkg, pos_css, p_blkg) \ | |
| 340 | + css_for_each_descendant_post((pos_css), &(p_blkg)->blkcg->css) \ | |
| 341 | + if (((d_blkg) = __blkg_lookup(css_to_blkcg(pos_css), \ | |
| 342 | + (p_blkg)->q, false))) | |
| 343 | + | |
| 344 | +bool __blkcg_punt_bio_submit(struct bio *bio); | |
| 345 | + | |
| 346 | +static inline bool blkcg_punt_bio_submit(struct bio *bio) | |
| 347 | +{ | |
| 348 | + if (bio->bi_opf & REQ_CGROUP_PUNT) | |
| 349 | + return __blkcg_punt_bio_submit(bio); | |
| 350 | + else | |
| 351 | + return false; | |
| 352 | +} | |
| 353 | + | |
| 354 | +static inline void blkcg_bio_issue_init(struct bio *bio) | |
| 355 | +{ | |
| 356 | + bio_issue_init(&bio->bi_issue, bio_sectors(bio)); | |
| 357 | +} | |
| 358 | + | |
| 359 | +static inline void blkcg_use_delay(struct blkcg_gq *blkg) | |
| 360 | +{ | |
| 361 | + if (WARN_ON_ONCE(atomic_read(&blkg->use_delay) < 0)) | |
| 362 | + return; | |
| 363 | + if (atomic_add_return(1, &blkg->use_delay) == 1) | |
| 364 | + atomic_inc(&blkg->blkcg->css.cgroup->congestion_count); | |
| 365 | +} | |
| 366 | + | |
| 367 | +static inline int blkcg_unuse_delay(struct blkcg_gq *blkg) | |
| 368 | +{ | |
| 369 | + int old = atomic_read(&blkg->use_delay); | |
| 370 | + | |
| 371 | + if (WARN_ON_ONCE(old < 0)) | |
| 372 | + return 0; | |
| 373 | + if (old == 0) | |
| 374 | + return 0; | |
| 375 | + | |
| 376 | + /* | |
| 377 | + * We do this song and dance because we can race with somebody else | |
| 378 | + * adding or removing delay. If we just did an atomic_dec we'd end up | |
| 379 | + * negative and we'd already be in trouble. We need to subtract 1 and | |
| 380 | + * then check to see if we were the last delay so we can drop the | |
| 381 | + * congestion count on the cgroup. | |
| 382 | + */ | |
| 383 | + while (old) { | |
| 384 | + int cur = atomic_cmpxchg(&blkg->use_delay, old, old - 1); | |
| 385 | + if (cur == old) | |
| 386 | + break; | |
| 387 | + old = cur; | |
| 388 | + } | |
| 389 | + | |
| 390 | + if (old == 0) | |
| 391 | + return 0; | |
| 392 | + if (old == 1) | |
| 393 | + atomic_dec(&blkg->blkcg->css.cgroup->congestion_count); | |
| 394 | + return 1; | |
| 395 | +} | |
| 396 | + | |
| 397 | +/** | |
| 398 | + * blkcg_set_delay - Enable allocator delay mechanism with the specified delay amount | |
| 399 | + * @blkg: target blkg | |
| 400 | + * @delay: delay duration in nsecs | |
| 401 | + * | |
| 402 | + * When enabled with this function, the delay is not decayed and must be | |
| 403 | + * explicitly cleared with blkcg_clear_delay(). Must not be mixed with | |
| 404 | + * blkcg_[un]use_delay() and blkcg_add_delay() usages. | |
| 405 | + */ | |
| 406 | +static inline void blkcg_set_delay(struct blkcg_gq *blkg, u64 delay) | |
| 407 | +{ | |
| 408 | + int old = atomic_read(&blkg->use_delay); | |
| 409 | + | |
| 410 | + /* We only want 1 person setting the congestion count for this blkg. */ | |
| 411 | + if (!old && atomic_cmpxchg(&blkg->use_delay, old, -1) == old) | |
| 412 | + atomic_inc(&blkg->blkcg->css.cgroup->congestion_count); | |
| 413 | + | |
| 414 | + atomic64_set(&blkg->delay_nsec, delay); | |
| 415 | +} | |
| 416 | + | |
| 417 | +/** | |
| 418 | + * blkcg_clear_delay - Disable allocator delay mechanism | |
| 419 | + * @blkg: target blkg | |
| 420 | + * | |
| 421 | + * Disable use_delay mechanism. See blkcg_set_delay(). | |
| 422 | + */ | |
| 423 | +static inline void blkcg_clear_delay(struct blkcg_gq *blkg) | |
| 424 | +{ | |
| 425 | + int old = atomic_read(&blkg->use_delay); | |
| 426 | + | |
| 427 | + /* We only want 1 person clearing the congestion count for this blkg. */ | |
| 428 | + if (old && atomic_cmpxchg(&blkg->use_delay, old, 0) == old) | |
| 429 | + atomic_dec(&blkg->blkcg->css.cgroup->congestion_count); | |
| 430 | +} | |
| 431 | + | |
| 432 | +/** | |
| 433 | + * blk_cgroup_mergeable - Determine whether to allow or disallow merges | |
| 434 | + * @rq: request to merge into | |
| 435 | + * @bio: bio to merge | |
| 436 | + * | |
| 437 | + * @bio and @rq should belong to the same cgroup and their issue_as_root should | |
| 438 | + * match. The latter is necessary as we don't want to throttle e.g. a metadata | |
| 439 | + * update because it happens to be next to a regular IO. | |
| 440 | + */ | |
| 441 | +static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bio) | |
| 442 | +{ | |
| 443 | + return rq->bio->bi_blkg == bio->bi_blkg && | |
| 444 | + bio_issue_as_root_blkg(rq->bio) == bio_issue_as_root_blkg(bio); | |
| 445 | +} | |
| 446 | + | |
| 447 | +void blk_cgroup_bio_start(struct bio *bio); | |
| 448 | +void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta); | |
| 449 | +#else /* CONFIG_BLK_CGROUP */ | |
| 450 | + | |
| 451 | +struct blkg_policy_data { | |
| 452 | +}; | |
| 453 | + | |
| 454 | +struct blkcg_policy_data { | |
| 455 | +}; | |
| 456 | + | |
| 457 | +struct blkcg_policy { | |
| 458 | +}; | |
| 459 | + | |
| 460 | +#ifdef CONFIG_BLOCK | |
| 461 | + | |
| 462 | +static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, void *key) { return NULL; } | |
| 463 | +static inline struct blkcg_gq *blk_queue_root_blkg(struct request_queue *q) | |
| 464 | +{ return NULL; } | |
| 465 | +static inline int blkcg_init_queue(struct request_queue *q) { return 0; } | |
| 466 | +static inline void blkcg_exit_queue(struct request_queue *q) { } | |
| 467 | +static inline int blkcg_policy_register(struct blkcg_policy *pol) { return 0; } | |
| 468 | +static inline void blkcg_policy_unregister(struct blkcg_policy *pol) { } | |
| 469 | +static inline int blkcg_activate_policy(struct request_queue *q, | |
| 470 | + const struct blkcg_policy *pol) { return 0; } | |
| 471 | +static inline void blkcg_deactivate_policy(struct request_queue *q, | |
| 472 | + const struct blkcg_policy *pol) { } | |
| 473 | + | |
| 474 | +static inline struct blkcg *__bio_blkcg(struct bio *bio) { return NULL; } | |
| 475 | + | |
| 476 | +static inline struct blkg_policy_data *blkg_to_pd(struct blkcg_gq *blkg, | |
| 477 | + struct blkcg_policy *pol) { return NULL; } | |
| 478 | +static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) { return NULL; } | |
| 479 | +static inline char *blkg_path(struct blkcg_gq *blkg) { return NULL; } | |
| 480 | +static inline void blkg_get(struct blkcg_gq *blkg) { } | |
| 481 | +static inline void blkg_put(struct blkcg_gq *blkg) { } | |
| 482 | + | |
| 483 | +static inline bool blkcg_punt_bio_submit(struct bio *bio) { return false; } | |
| 484 | +static inline void blkcg_bio_issue_init(struct bio *bio) { } | |
| 485 | +static inline void blk_cgroup_bio_start(struct bio *bio) { } | |
| 486 | +static inline bool blk_cgroup_mergeable(struct request *rq, struct bio *bio) { return true; } | |
| 487 | + | |
| 488 | +#define blk_queue_for_each_rl(rl, q) \ | |
| 489 | + for ((rl) = &(q)->root_rl; (rl); (rl) = NULL) | |
| 490 | + | |
| 491 | +#endif /* CONFIG_BLOCK */ | |
| 492 | +#endif /* CONFIG_BLK_CGROUP */ | |
| 493 | + | |
| 494 | +#endif /* _BLK_CGROUP_PRIVATE_H */ | |
| @@ -34,7 +34,6 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <linux/delay.h> |
| 35 | 35 | #include <linux/ratelimit.h> |
| 36 | 36 | #include <linux/pm_runtime.h> |
| 37 | −#include <linux/blk-cgroup.h> | |
| 38 | 37 | #include <linux/t10-pi.h> |
| 39 | 38 | #include <linux/debugfs.h> |
| 40 | 39 | #include <linux/bpf.h> |
| @@ -49,6 +48,7 @@ · 8 unmodified lines | ||
| 49 | 48 | #include "blk.h" |
| 50 | 49 | #include "blk-mq-sched.h" |
| 51 | 50 | #include "blk-pm.h" |
| 51 | +#include "blk-cgroup.h" | |
| 52 | 52 | #include "blk-throttle.h" |
| 53 | 53 | #include "blk-rq-qos.h" |
| 54 | 54 | |
| @@ -165,6 +165,7 @@ static const struct { · 110 unmodified lines | ||
| 165 | 165 | [BLK_STS_RESOURCE] = { -ENOMEM, "kernel resource" }, |
| 166 | 166 | [BLK_STS_DEV_RESOURCE] = { -EBUSY, "device resource" }, |
| 167 | 167 | [BLK_STS_AGAIN] = { -EAGAIN, "nonblocking retry" }, |
| 168 | + [BLK_STS_OFFLINE] = { -ENODEV, "device offline" }, | |
| 168 | 169 | |
| 169 | 170 | /* device mapper special case, should not leak out: */ |
| 170 | 171 | [BLK_STS_DM_REQUEUE] = { -EREMCHG, "dm internal retry" }, |
| @@ -339,8 +340,6 @@ void blk_cleanup_queue(struct request_queue *q) · 168 unmodified lines | ||
| 339 | 340 | blk_mq_sched_free_rqs(q); |
| 340 | 341 | mutex_unlock(&q->sysfs_lock); |
| 341 | 342 | |
| 342 | − percpu_ref_exit(&q->q_usage_counter); | |
| 343 | − | |
| 344 | 343 | /* @q is and will stay empty, shutdown and put */ |
| 345 | 344 | blk_put_queue(q); |
| 346 | 345 | } |
| @@ -473,9 +472,6 @@ struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu) · 126 unmodified lines | ||
| 473 | 472 | timer_setup(&q->timeout, blk_rq_timed_out_timer, 0); |
| 474 | 473 | INIT_WORK(&q->timeout_work, blk_timeout_work); |
| 475 | 474 | INIT_LIST_HEAD(&q->icq_list); |
| 476 | −#ifdef CONFIG_BLK_CGROUP | |
| 477 | − INIT_LIST_HEAD(&q->blkg_list); | |
| 478 | −#endif | |
| 479 | 475 | |
| 480 | 476 | kobject_init(&q->kobj, &blk_queue_ktype); |
| 481 | 477 | |
| @@ -496,17 +492,12 @@ struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu) · 14 unmodified lines | ||
| 496 | 492 | PERCPU_REF_INIT_ATOMIC, GFP_KERNEL)) |
| 497 | 493 | goto fail_stats; |
| 498 | 494 | |
| 499 | − if (blkcg_init_queue(q)) | |
| 500 | − goto fail_ref; | |
| 501 | − | |
| 502 | 495 | blk_queue_dma_alignment(q, 511); |
| 503 | 496 | blk_set_default_limits(&q->limits); |
| 504 | 497 | q->nr_requests = BLKDEV_DEFAULT_RQ; |
| 505 | 498 | |
| 506 | 499 | return q; |
| 507 | 500 | |
| 508 | −fail_ref: | |
| 509 | − percpu_ref_exit(&q->q_usage_counter); | |
| 510 | 501 | fail_stats: |
| 511 | 502 | blk_free_queue_stats(q->stats); |
| 512 | 503 | fail_split: |
| @@ -540,17 +531,6 @@ bool blk_get_queue(struct request_queue *q) · 27 unmodified lines | ||
| 540 | 531 | } |
| 541 | 532 | EXPORT_SYMBOL(blk_get_queue); |
| 542 | 533 | |
| 543 | −static void handle_bad_sector(struct bio *bio, sector_t maxsector) | |
| 544 | −{ | |
| 545 | − char b[BDEVNAME_SIZE]; | |
| 546 | − | |
| 547 | − pr_info_ratelimited("%s: attempt to access beyond end of device\n" | |
| 548 | − "%s: rw=%d, want=%llu, limit=%llu\n", | |
| 549 | − current->comm, | |
| 550 | − bio_devname(bio, b), bio->bi_opf, | |
| 551 | − bio_end_sector(bio), maxsector); | |
| 552 | −} | |
| 553 | − | |
| 554 | 534 | #ifdef CONFIG_FAIL_MAKE_REQUEST |
| 555 | 535 | |
| 556 | 536 | static DECLARE_FAULT_ATTR(fail_make_request); |
| @@ -580,14 +560,10 @@ late_initcall(fail_make_request_debugfs); · 23 unmodified lines | ||
| 580 | 560 | static inline bool bio_check_ro(struct bio *bio) |
| 581 | 561 | { |
| 582 | 562 | if (op_is_write(bio_op(bio)) && bdev_read_only(bio->bi_bdev)) { |
| 583 | − char b[BDEVNAME_SIZE]; | |
| 584 | − | |
| 585 | 563 | if (op_is_flush(bio->bi_opf) && !bio_sectors(bio)) |
| 586 | 564 | return false; |
| 587 | − | |
| 588 | − WARN_ONCE(1, | |
| 589 | − "Trying to write to read-only block-device %s (partno %d)\n", | |
| 590 | − bio_devname(bio, b), bio->bi_bdev->bd_partno); | |
| 565 | + pr_warn("Trying to write to read-only block-device %pg\n", | |
| 566 | + bio->bi_bdev); | |
| 591 | 567 | /* Older lvm-tools actually trigger this */ |
| 592 | 568 | return false; |
| 593 | 569 | } |
| @@ -616,7 +592,11 @@ static inline int bio_check_eod(struct bio *bio) · 22 unmodified lines | ||
| 616 | 592 | if (nr_sectors && maxsector && |
| 617 | 593 | (nr_sectors > maxsector || |
| 618 | 594 | bio->bi_iter.bi_sector > maxsector - nr_sectors)) { |
| 619 | − handle_bad_sector(bio, maxsector); | |
| 595 | + pr_info_ratelimited("%s: attempt to access beyond end of device\n" | |
| 596 | + "%pg: rw=%d, want=%llu, limit=%llu\n", | |
| 597 | + current->comm, | |
| 598 | + bio->bi_bdev, bio->bi_opf, | |
| 599 | + bio_end_sector(bio), maxsector); | |
| 620 | 600 | return -EIO; |
| 621 | 601 | } |
| 622 | 602 | return 0; |
| @@ -676,7 +656,123 @@ static inline blk_status_t blk_check_zone_append(struct request_queue *q, · 53 unmodified lines | ||
| 676 | 656 | return BLK_STS_OK; |
| 677 | 657 | } |
| 678 | 658 | |
| 679 | −noinline_for_stack bool submit_bio_checks(struct bio *bio) | |
| 659 | +static void __submit_bio(struct bio *bio) | |
| 660 | +{ | |
| 661 | + struct gendisk *disk = bio->bi_bdev->bd_disk; | |
| 662 | + | |
| 663 | + if (unlikely(!blk_crypto_bio_prep(&bio))) | |
| 664 | + return; | |
| 665 | + | |
| 666 | + if (!disk->fops->submit_bio) { | |
| 667 | + blk_mq_submit_bio(bio); | |
| 668 | + } else if (likely(bio_queue_enter(bio) == 0)) { | |
| 669 | + disk->fops->submit_bio(bio); | |
| 670 | + blk_queue_exit(disk->queue); | |
| 671 | + } | |
| 672 | +} | |
| 673 | + | |
| 674 | +/* | |
| 675 | + * The loop in this function may be a bit non-obvious, and so deserves some | |
| 676 | + * explanation: | |
| 677 | + * | |
| 678 | + * - Before entering the loop, bio->bi_next is NULL (as all callers ensure | |
| 679 | + * that), so we have a list with a single bio. | |
| 680 | + * - We pretend that we have just taken it off a longer list, so we assign | |
| 681 | + * bio_list to a pointer to the bio_list_on_stack, thus initialising the | |
| 682 | + * bio_list of new bios to be added. ->submit_bio() may indeed add some more | |
| 683 | + * bios through a recursive call to submit_bio_noacct. If it did, we find a | |
| 684 | + * non-NULL value in bio_list and re-enter the loop from the top. | |
| 685 | + * - In this case we really did just take the bio of the top of the list (no | |
| 686 | + * pretending) and so remove it from bio_list, and call into ->submit_bio() | |
| 687 | + * again. | |
| 688 | + * | |
| 689 | + * bio_list_on_stack[0] contains bios submitted by the current ->submit_bio. | |
| 690 | + * bio_list_on_stack[1] contains bios that were submitted before the current | |
| 691 | + * ->submit_bio_bio, but that haven't been processed yet. | |
| 692 | + */ | |
| 693 | +static void __submit_bio_noacct(struct bio *bio) | |
| 694 | +{ | |
| 695 | + struct bio_list bio_list_on_stack[2]; | |
| 696 | + | |
| 697 | + BUG_ON(bio->bi_next); | |
| 698 | + | |
| 699 | + bio_list_init(&bio_list_on_stack[0]); | |
| 700 | + current->bio_list = bio_list_on_stack; | |
| 701 | + | |
| 702 | + do { | |
| 703 | + struct request_queue *q = bdev_get_queue(bio->bi_bdev); | |
| 704 | + struct bio_list lower, same; | |
| 705 | + | |
| 706 | + /* | |
| 707 | + * Create a fresh bio_list for all subordinate requests. | |
| 708 | + */ | |
| 709 | + bio_list_on_stack[1] = bio_list_on_stack[0]; | |
| 710 | + bio_list_init(&bio_list_on_stack[0]); | |
| 711 | + | |
| 712 | + __submit_bio(bio); | |
| 713 | + | |
| 714 | + /* | |
| 715 | + * Sort new bios into those for a lower level and those for the | |
| 716 | + * same level. | |
| 717 | + */ | |
| 718 | + bio_list_init(&lower); | |
| 719 | + bio_list_init(&same); | |
| 720 | + while ((bio = bio_list_pop(&bio_list_on_stack[0])) != NULL) | |
| 721 | + if (q == bdev_get_queue(bio->bi_bdev)) | |
| 722 | + bio_list_add(&same, bio); | |
| 723 | + else | |
| 724 | + bio_list_add(&lower, bio); | |
| 725 | + | |
| 726 | + /* | |
| 727 | + * Now assemble so we handle the lowest level first. | |
| 728 | + */ | |
| 729 | + bio_list_merge(&bio_list_on_stack[0], &lower); | |
| 730 | + bio_list_merge(&bio_list_on_stack[0], &same); | |
| 731 | + bio_list_merge(&bio_list_on_stack[0], &bio_list_on_stack[1]); | |
| 732 | + } while ((bio = bio_list_pop(&bio_list_on_stack[0]))); | |
| 733 | + | |
| 734 | + current->bio_list = NULL; | |
| 735 | +} | |
| 736 | + | |
| 737 | +static void __submit_bio_noacct_mq(struct bio *bio) | |
| 738 | +{ | |
| 739 | + struct bio_list bio_list[2] = { }; | |
| 740 | + | |
| 741 | + current->bio_list = bio_list; | |
| 742 | + | |
| 743 | + do { | |
| 744 | + __submit_bio(bio); | |
| 745 | + } while ((bio = bio_list_pop(&bio_list[0]))); | |
| 746 | + | |
| 747 | + current->bio_list = NULL; | |
| 748 | +} | |
| 749 | + | |
| 750 | +void submit_bio_noacct_nocheck(struct bio *bio) | |
| 751 | +{ | |
| 752 | + /* | |
| 753 | + * We only want one ->submit_bio to be active at a time, else stack | |
| 754 | + * usage with stacked devices could be a problem. Use current->bio_list | |
| 755 | + * to collect a list of requests submited by a ->submit_bio method while | |
| 756 | + * it is active, and then process them after it returned. | |
| 757 | + */ | |
| 758 | + if (current->bio_list) | |
| 759 | + bio_list_add(¤t->bio_list[0], bio); | |
| 760 | + else if (!bio->bi_bdev->bd_disk->fops->submit_bio) | |
| 761 | + __submit_bio_noacct_mq(bio); | |
| 762 | + else | |
| 763 | + __submit_bio_noacct(bio); | |
| 764 | +} | |
| 765 | + | |
| 766 | +/** | |
| 767 | + * submit_bio_noacct - re-submit a bio to the block device layer for I/O | |
| 768 | + * @bio: The bio describing the location in memory and on the device. | |
| 769 | + * | |
| 770 | + * This is a version of submit_bio() that shall only be used for I/O that is | |
| 771 | + * resubmitted to lower level drivers by stacking block drivers. All file | |
| 772 | + * systems and other upper level users of the block layer should use | |
| 773 | + * submit_bio() instead. | |
| 774 | + */ | |
| 775 | +void submit_bio_noacct(struct bio *bio) | |
| 680 | 776 | { |
| 681 | 777 | struct block_device *bdev = bio->bi_bdev; |
| 682 | 778 | struct request_queue *q = bdev_get_queue(bdev); |
| @@ -761,7 +857,7 @@ noinline_for_stack bool submit_bio_checks(struct bio *bio) · 78 unmodified lines | ||
| 761 | 857 | } |
| 762 | 858 | |
| 763 | 859 | if (blk_throtl_bio(bio)) |
| 764 | − return false; | |
| 860 | + return; | |
| 765 | 861 | |
| 766 | 862 | blk_cgroup_bio_start(bio); |
| 767 | 863 | blkcg_bio_issue_init(bio); |
| @@ -773,138 +869,14 @@ noinline_for_stack bool submit_bio_checks(struct bio *bio) · 5 unmodified lines | ||
| 773 | 869 | */ |
| 774 | 870 | bio_set_flag(bio, BIO_TRACE_COMPLETION); |
| 775 | 871 | } |
| 776 | − return true; | |
| 872 | + submit_bio_noacct_nocheck(bio); | |
| 873 | + return; | |
| 777 | 874 | |
| 778 | 875 | not_supported: |
| 779 | 876 | status = BLK_STS_NOTSUPP; |
| 780 | 877 | end_io: |
| 781 | 878 | bio->bi_status = status; |
| 782 | 879 | bio_endio(bio); |
| 783 | − return false; | |
| 784 | −} | |
| 785 | − | |
| 786 | −static void __submit_bio_fops(struct gendisk *disk, struct bio *bio) | |
| 787 | −{ | |
| 788 | − if (blk_crypto_bio_prep(&bio)) { | |
| 789 | − if (likely(bio_queue_enter(bio) == 0)) { | |
| 790 | − disk->fops->submit_bio(bio); | |
| 791 | − blk_queue_exit(disk->queue); | |
| 792 | − } | |
| 793 | − } | |
| 794 | −} | |
| 795 | − | |
| 796 | −static void __submit_bio(struct bio *bio) | |
| 797 | −{ | |
| 798 | − struct gendisk *disk = bio->bi_bdev->bd_disk; | |
| 799 | − | |
| 800 | − if (unlikely(!submit_bio_checks(bio))) | |
| 801 | − return; | |
| 802 | − | |
| 803 | − if (!disk->fops->submit_bio) | |
| 804 | − blk_mq_submit_bio(bio); | |
| 805 | − else | |
| 806 | − __submit_bio_fops(disk, bio); | |
| 807 | −} | |
| 808 | − | |
| 809 | −/* | |
| 810 | − * The loop in this function may be a bit non-obvious, and so deserves some | |
| 811 | − * explanation: | |
| 812 | − * | |
| 813 | − * - Before entering the loop, bio->bi_next is NULL (as all callers ensure | |
| 814 | − * that), so we have a list with a single bio. | |
| 815 | − * - We pretend that we have just taken it off a longer list, so we assign | |
| 816 | − * bio_list to a pointer to the bio_list_on_stack, thus initialising the | |
| 817 | − * bio_list of new bios to be added. ->submit_bio() may indeed add some more | |
| 818 | − * bios through a recursive call to submit_bio_noacct. If it did, we find a | |
| 819 | − * non-NULL value in bio_list and re-enter the loop from the top. | |
| 820 | − * - In this case we really did just take the bio of the top of the list (no | |
| 821 | − * pretending) and so remove it from bio_list, and call into ->submit_bio() | |
| 822 | − * again. | |
| 823 | − * | |
| 824 | − * bio_list_on_stack[0] contains bios submitted by the current ->submit_bio. | |
| 825 | − * bio_list_on_stack[1] contains bios that were submitted before the current | |
| 826 | − * ->submit_bio_bio, but that haven't been processed yet. | |
| 827 | − */ | |
| 828 | −static void __submit_bio_noacct(struct bio *bio) | |
| 829 | −{ | |
| 830 | − struct bio_list bio_list_on_stack[2]; | |
| 831 | − | |
| 832 | − BUG_ON(bio->bi_next); | |
| 833 | − | |
| 834 | − bio_list_init(&bio_list_on_stack[0]); | |
| 835 | − current->bio_list = bio_list_on_stack; | |
| 836 | − | |
| 837 | − do { | |
| 838 | − struct request_queue *q = bdev_get_queue(bio->bi_bdev); | |
| 839 | − struct bio_list lower, same; | |
| 840 | − | |
| 841 | − /* | |
| 842 | − * Create a fresh bio_list for all subordinate requests. | |
| 843 | − */ | |
| 844 | − bio_list_on_stack[1] = bio_list_on_stack[0]; | |
| 845 | − bio_list_init(&bio_list_on_stack[0]); | |
| 846 | − | |
| 847 | − __submit_bio(bio); | |
| 848 | − | |
| 849 | − /* | |
| 850 | − * Sort new bios into those for a lower level and those for the | |
| 851 | − * same level. | |
| 852 | − */ | |
| 853 | − bio_list_init(&lower); | |
| 854 | − bio_list_init(&same); | |
| 855 | − while ((bio = bio_list_pop(&bio_list_on_stack[0])) != NULL) | |
| 856 | − if (q == bdev_get_queue(bio->bi_bdev)) | |
| 857 | − bio_list_add(&same, bio); | |
| 858 | − else | |
| 859 | − bio_list_add(&lower, bio); | |
| 860 | − | |
| 861 | − /* | |
| 862 | − * Now assemble so we handle the lowest level first. | |
| 863 | − */ | |
| 864 | − bio_list_merge(&bio_list_on_stack[0], &lower); | |
| 865 | − bio_list_merge(&bio_list_on_stack[0], &same); | |
| 866 | − bio_list_merge(&bio_list_on_stack[0], &bio_list_on_stack[1]); | |
| 867 | − } while ((bio = bio_list_pop(&bio_list_on_stack[0]))); | |
| 868 | − | |
| 869 | − current->bio_list = NULL; | |
| 870 | −} | |
| 871 | − | |
| 872 | −static void __submit_bio_noacct_mq(struct bio *bio) | |
| 873 | −{ | |
| 874 | − struct bio_list bio_list[2] = { }; | |
| 875 | − | |
| 876 | − current->bio_list = bio_list; | |
| 877 | − | |
| 878 | − do { | |
| 879 | − __submit_bio(bio); | |
| 880 | − } while ((bio = bio_list_pop(&bio_list[0]))); | |
| 881 | − | |
| 882 | − current->bio_list = NULL; | |
| 883 | −} | |
| 884 | − | |
| 885 | −/** | |
| 886 | − * submit_bio_noacct - re-submit a bio to the block device layer for I/O | |
| 887 | − * @bio: The bio describing the location in memory and on the device. | |
| 888 | − * | |
| 889 | − * This is a version of submit_bio() that shall only be used for I/O that is | |
| 890 | − * resubmitted to lower level drivers by stacking block drivers. All file | |
| 891 | − * systems and other upper level users of the block layer should use | |
| 892 | − * submit_bio() instead. | |
| 893 | − */ | |
| 894 | −void submit_bio_noacct(struct bio *bio) | |
| 895 | −{ | |
| 896 | − /* | |
| 897 | − * We only want one ->submit_bio to be active at a time, else stack | |
| 898 | − * usage with stacked devices could be a problem. Use current->bio_list | |
| 899 | − * to collect a list of requests submited by a ->submit_bio method while | |
| 900 | − * it is active, and then process them after it returned. | |
| 901 | − */ | |
| 902 | − if (current->bio_list) | |
| 903 | − bio_list_add(¤t->bio_list[0], bio); | |
| 904 | − else if (!bio->bi_bdev->bd_disk->fops->submit_bio) | |
| 905 | − __submit_bio_noacct_mq(bio); | |
| 906 | − else | |
| 907 | − __submit_bio_noacct(bio); | |
| 908 | 880 | } |
| 909 | 881 | EXPORT_SYMBOL(submit_bio_noacct); |
| 910 | 882 | |
| @@ -989,8 +961,7 @@ int bio_poll(struct bio *bio, struct io_comp_batch *iob, unsigned int flags) · 78 unmodified lines | ||
| 989 | 961 | !test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) |
| 990 | 962 | return 0; |
| 991 | 963 | |
| 992 | − if (current->plug) | |
| 993 | − blk_flush_plug(current->plug, false); | |
| 964 | + blk_flush_plug(current->plug, false); | |
| 994 | 965 | |
| 995 | 966 | if (blk_queue_enter(q, BLK_MQ_REQ_NOWAIT)) |
| 996 | 967 | return 0; |
| @@ -1272,7 +1243,7 @@ struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug, void *data, · 275 unmodified lines | ||
| 1272 | 1243 | } |
| 1273 | 1244 | EXPORT_SYMBOL(blk_check_plugged); |
| 1274 | 1245 | |
| 1275 | −void blk_flush_plug(struct blk_plug *plug, bool from_schedule) | |
| 1246 | +void __blk_flush_plug(struct blk_plug *plug, bool from_schedule) | |
| 1276 | 1247 | { |
| 1277 | 1248 | if (!list_empty(&plug->cb_list)) |
| 1278 | 1249 | flush_plug_callbacks(plug, from_schedule); |
| @@ -1301,7 +1272,7 @@ void blk_flush_plug(struct blk_plug *plug, bool from_schedule) · 22 unmodified lines | ||
| 1301 | 1272 | void blk_finish_plug(struct blk_plug *plug) |
| 1302 | 1273 | { |
| 1303 | 1274 | if (plug == current->plug) { |
| 1304 | − blk_flush_plug(plug, false); | |
| 1275 | + __blk_flush_plug(plug, false); | |
| 1305 | 1276 | current->plug = NULL; |
| 1306 | 1277 | } |
| 1307 | 1278 | } |
| @@ -10,7 +10,6 @@ · 9 unmodified lines | ||
| 10 | 10 | #define pr_fmt(fmt) "blk-crypto-fallback: " fmt |
| 11 | 11 | |
| 12 | 12 | #include <crypto/skcipher.h> |
| 13 | −#include <linux/blk-cgroup.h> | |
| 14 | 13 | #include <linux/blk-crypto.h> |
| 15 | 14 | #include <linux/blk-crypto-profile.h> |
| 16 | 15 | #include <linux/blkdev.h> |
| @@ -20,6 +19,7 @@ · 3 unmodified lines | ||
| 20 | 19 | #include <linux/random.h> |
| 21 | 20 | #include <linux/scatterlist.h> |
| 22 | 21 | |
| 22 | +#include "blk-cgroup.h" | |
| 23 | 23 | #include "blk-crypto-internal.h" |
| 24 | 24 | |
| 25 | 25 | static unsigned int num_prealloc_bounce_pg = 32; |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | |
| 12 | 12 | /* Represents a crypto mode supported by blk-crypto */ |
| 13 | 13 | struct blk_crypto_mode { |
| 14 | + const char *name; /* name of this mode, shown in sysfs */ | |
| 14 | 15 | const char *cipher_str; /* crypto API name (for fallback case) */ |
| 15 | 16 | unsigned int keysize; /* key size in bytes */ |
| 16 | 17 | unsigned int ivsize; /* iv size in bytes */ |
| @@ -20,6 +21,10 @@ extern const struct blk_crypto_mode blk_crypto_modes[]; · 3 unmodified lines | ||
| 20 | 21 | |
| 21 | 22 | #ifdef CONFIG_BLK_INLINE_ENCRYPTION |
| 22 | 23 | |
| 24 | +int blk_crypto_sysfs_register(struct request_queue *q); | |
| 25 | + | |
| 26 | +void blk_crypto_sysfs_unregister(struct request_queue *q); | |
| 27 | + | |
| 23 | 28 | void bio_crypt_dun_increment(u64 dun[BLK_CRYPTO_DUN_ARRAY_SIZE], |
| 24 | 29 | unsigned int inc); |
| 25 | 30 | |
| @@ -62,6 +67,13 @@ static inline bool blk_crypto_rq_is_encrypted(struct request *rq) · 36 unmodified lines | ||
| 62 | 67 | |
| 63 | 68 | #else /* CONFIG_BLK_INLINE_ENCRYPTION */ |
| 64 | 69 | |
| 70 | +static inline int blk_crypto_sysfs_register(struct request_queue *q) | |
| 71 | +{ | |
| 72 | + return 0; | |
| 73 | +} | |
| 74 | + | |
| 75 | +static inline void blk_crypto_sysfs_unregister(struct request_queue *q) { } | |
| 76 | + | |
| 65 | 77 | static inline bool bio_crypt_rq_ctx_compatible(struct request *rq, |
| 66 | 78 | struct bio *bio) |
| 67 | 79 | { |
| @@ -0,0 +1,172 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | +/* | |
| 3 | + * Copyright 2021 Google LLC | |
| 4 | + * | |
| 5 | + * sysfs support for blk-crypto. This file contains the code which exports the | |
| 6 | + * crypto capabilities of devices via /sys/block/$disk/queue/crypto/. | |
| 7 | + */ | |
| 8 | + | |
| 9 | +#include <linux/blk-crypto-profile.h> | |
| 10 | + | |
| 11 | +#include "blk-crypto-internal.h" | |
| 12 | + | |
| 13 | +struct blk_crypto_kobj { | |
| 14 | + struct kobject kobj; | |
| 15 | + struct blk_crypto_profile *profile; | |
| 16 | +}; | |
| 17 | + | |
| 18 | +struct blk_crypto_attr { | |
| 19 | + struct attribute attr; | |
| 20 | + ssize_t (*show)(struct blk_crypto_profile *profile, | |
| 21 | + struct blk_crypto_attr *attr, char *page); | |
| 22 | +}; | |
| 23 | + | |
| 24 | +static struct blk_crypto_profile *kobj_to_crypto_profile(struct kobject *kobj) | |
| 25 | +{ | |
| 26 | + return container_of(kobj, struct blk_crypto_kobj, kobj)->profile; | |
| 27 | +} | |
| 28 | + | |
| 29 | +static struct blk_crypto_attr *attr_to_crypto_attr(struct attribute *attr) | |
| 30 | +{ | |
| 31 | + return container_of(attr, struct blk_crypto_attr, attr); | |
| 32 | +} | |
| 33 | + | |
| 34 | +static ssize_t max_dun_bits_show(struct blk_crypto_profile *profile, | |
| 35 | + struct blk_crypto_attr *attr, char *page) | |
| 36 | +{ | |
| 37 | + return sysfs_emit(page, "%u\n", 8 * profile->max_dun_bytes_supported); | |
| 38 | +} | |
| 39 | + | |
| 40 | +static ssize_t num_keyslots_show(struct blk_crypto_profile *profile, | |
| 41 | + struct blk_crypto_attr *attr, char *page) | |
| 42 | +{ | |
| 43 | + return sysfs_emit(page, "%u\n", profile->num_slots); | |
| 44 | +} | |
| 45 | + | |
| 46 | +#define BLK_CRYPTO_RO_ATTR(_name) \ | |
| 47 | + static struct blk_crypto_attr _name##_attr = __ATTR_RO(_name) | |
| 48 | + | |
| 49 | +BLK_CRYPTO_RO_ATTR(max_dun_bits); | |
| 50 | +BLK_CRYPTO_RO_ATTR(num_keyslots); | |
| 51 | + | |
| 52 | +static struct attribute *blk_crypto_attrs[] = { | |
| 53 | + &max_dun_bits_attr.attr, | |
| 54 | + &num_keyslots_attr.attr, | |
| 55 | + NULL, | |
| 56 | +}; | |
| 57 | + | |
| 58 | +static const struct attribute_group blk_crypto_attr_group = { | |
| 59 | + .attrs = blk_crypto_attrs, | |
| 60 | +}; | |
| 61 | + | |
| 62 | +/* | |
| 63 | + * The encryption mode attributes. To avoid hard-coding the list of encryption | |
| 64 | + * modes, these are initialized at boot time by blk_crypto_sysfs_init(). | |
| 65 | + */ | |
| 66 | +static struct blk_crypto_attr __blk_crypto_mode_attrs[BLK_ENCRYPTION_MODE_MAX]; | |
| 67 | +static struct attribute *blk_crypto_mode_attrs[BLK_ENCRYPTION_MODE_MAX + 1]; | |
| 68 | + | |
| 69 | +static umode_t blk_crypto_mode_is_visible(struct kobject *kobj, | |
| 70 | + struct attribute *attr, int n) | |
| 71 | +{ | |
| 72 | + struct blk_crypto_profile *profile = kobj_to_crypto_profile(kobj); | |
| 73 | + struct blk_crypto_attr *a = attr_to_crypto_attr(attr); | |
| 74 | + int mode_num = a - __blk_crypto_mode_attrs; | |
| 75 | + | |
| 76 | + if (profile->modes_supported[mode_num]) | |
| 77 | + return 0444; | |
| 78 | + return 0; | |
| 79 | +} | |
| 80 | + | |
| 81 | +static ssize_t blk_crypto_mode_show(struct blk_crypto_profile *profile, | |
| 82 | + struct blk_crypto_attr *attr, char *page) | |
| 83 | +{ | |
| 84 | + int mode_num = attr - __blk_crypto_mode_attrs; | |
| 85 | + | |
| 86 | + return sysfs_emit(page, "0x%x\n", profile->modes_supported[mode_num]); | |
| 87 | +} | |
| 88 | + | |
| 89 | +static const struct attribute_group blk_crypto_modes_attr_group = { | |
| 90 | + .name = "modes", | |
| 91 | + .attrs = blk_crypto_mode_attrs, | |
| 92 | + .is_visible = blk_crypto_mode_is_visible, | |
| 93 | +}; | |
| 94 | + | |
| 95 | +static const struct attribute_group *blk_crypto_attr_groups[] = { | |
| 96 | + &blk_crypto_attr_group, | |
| 97 | + &blk_crypto_modes_attr_group, | |
| 98 | + NULL, | |
| 99 | +}; | |
| 100 | + | |
| 101 | +static ssize_t blk_crypto_attr_show(struct kobject *kobj, | |
| 102 | + struct attribute *attr, char *page) | |
| 103 | +{ | |
| 104 | + struct blk_crypto_profile *profile = kobj_to_crypto_profile(kobj); | |
| 105 | + struct blk_crypto_attr *a = attr_to_crypto_attr(attr); | |
| 106 | + | |
| 107 | + return a->show(profile, a, page); | |
| 108 | +} | |
| 109 | + | |
| 110 | +static const struct sysfs_ops blk_crypto_attr_ops = { | |
| 111 | + .show = blk_crypto_attr_show, | |
| 112 | +}; | |
| 113 | + | |
| 114 | +static void blk_crypto_release(struct kobject *kobj) | |
| 115 | +{ | |
| 116 | + kfree(container_of(kobj, struct blk_crypto_kobj, kobj)); | |
| 117 | +} | |
| 118 | + | |
| 119 | +static struct kobj_type blk_crypto_ktype = { | |
| 120 | + .default_groups = blk_crypto_attr_groups, | |
| 121 | + .sysfs_ops = &blk_crypto_attr_ops, | |
| 122 | + .release = blk_crypto_release, | |
| 123 | +}; | |
| 124 | + | |
| 125 | +/* | |
| 126 | + * If the request_queue has a blk_crypto_profile, create the "crypto" | |
| 127 | + * subdirectory in sysfs (/sys/block/$disk/queue/crypto/). | |
| 128 | + */ | |
| 129 | +int blk_crypto_sysfs_register(struct request_queue *q) | |
| 130 | +{ | |
| 131 | + struct blk_crypto_kobj *obj; | |
| 132 | + int err; | |
| 133 | + | |
| 134 | + if (!q->crypto_profile) | |
| 135 | + return 0; | |
| 136 | + | |
| 137 | + obj = kzalloc(sizeof(*obj), GFP_KERNEL); | |
| 138 | + if (!obj) | |
| 139 | + return -ENOMEM; | |
| 140 | + obj->profile = q->crypto_profile; | |
| 141 | + | |
| 142 | + err = kobject_init_and_add(&obj->kobj, &blk_crypto_ktype, &q->kobj, | |
| 143 | + "crypto"); | |
| 144 | + if (err) { | |
| 145 | + kobject_put(&obj->kobj); | |
| 146 | + return err; | |
| 147 | + } | |
| 148 | + q->crypto_kobject = &obj->kobj; | |
| 149 | + return 0; | |
| 150 | +} | |
| 151 | + | |
| 152 | +void blk_crypto_sysfs_unregister(struct request_queue *q) | |
| 153 | +{ | |
| 154 | + kobject_put(q->crypto_kobject); | |
| 155 | +} | |
| 156 | + | |
| 157 | +static int __init blk_crypto_sysfs_init(void) | |
| 158 | +{ | |
| 159 | + int i; | |
| 160 | + | |
| 161 | + BUILD_BUG_ON(BLK_ENCRYPTION_MODE_INVALID != 0); | |
| 162 | + for (i = 1; i < BLK_ENCRYPTION_MODE_MAX; i++) { | |
| 163 | + struct blk_crypto_attr *attr = &__blk_crypto_mode_attrs[i]; | |
| 164 | + | |
| 165 | + attr->attr.name = blk_crypto_modes[i].name; | |
| 166 | + attr->attr.mode = 0444; | |
| 167 | + attr->show = blk_crypto_mode_show; | |
| 168 | + blk_crypto_mode_attrs[i - 1] = &attr->attr; | |
| 169 | + } | |
| 170 | + return 0; | |
| 171 | +} | |
| 172 | +subsys_initcall(blk_crypto_sysfs_init); | |
| @@ -19,16 +19,19 @@ · 18 unmodified lines | ||
| 19 | 19 | |
| 20 | 20 | const struct blk_crypto_mode blk_crypto_modes[] = { |
| 21 | 21 | [BLK_ENCRYPTION_MODE_AES_256_XTS] = { |
| 22 | + .name = "AES-256-XTS", | |
| 22 | 23 | .cipher_str = "xts(aes)", |
| 23 | 24 | .keysize = 64, |
| 24 | 25 | .ivsize = 16, |
| 25 | 26 | }, |
| 26 | 27 | [BLK_ENCRYPTION_MODE_AES_128_CBC_ESSIV] = { |
| 28 | + .name = "AES-128-CBC-ESSIV", | |
| 27 | 29 | .cipher_str = "essiv(cbc(aes),sha256)", |
| 28 | 30 | .keysize = 16, |
| 29 | 31 | .ivsize = 16, |
| 30 | 32 | }, |
| 31 | 33 | [BLK_ENCRYPTION_MODE_ADIANTUM] = { |
| 34 | + .name = "Adiantum", | |
| 32 | 35 | .cipher_str = "adiantum(xchacha12,aes)", |
| 33 | 36 | .keysize = 32, |
| 34 | 37 | .ivsize = 32, |
| @@ -111,7 +114,6 @@ int __bio_crypt_clone(struct bio *dst, struct bio *src, gfp_t gfp_mask) · 76 unmodified lines | ||
| 111 | 114 | *dst->bi_crypt_context = *src->bi_crypt_context; |
| 112 | 115 | return 0; |
| 113 | 116 | } |
| 114 | −EXPORT_SYMBOL_GPL(__bio_crypt_clone); | |
| 115 | 117 | |
| 116 | 118 | /* Increments @dun by @inc, treating @dun as a multi-limb integer. */ |
| 117 | 119 | void bio_crypt_dun_increment(u64 dun[BLK_CRYPTO_DUN_ARRAY_SIZE], |
| @@ -460,9 +460,7 @@ int blkdev_issue_flush(struct block_device *bdev) · 459 unmodified lines | ||
| 460 | 460 | { |
| 461 | 461 | struct bio bio; |
| 462 | 462 | |
| 463 | − bio_init(&bio, NULL, 0); | |
| 464 | − bio_set_dev(&bio, bdev); | |
| 465 | − bio.bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 463 | + bio_init(&bio, bdev, NULL, 0, REQ_OP_WRITE | REQ_PREFLUSH); | |
| 466 | 464 | return submit_bio_wait(&bio); |
| 467 | 465 | } |
| 468 | 466 | EXPORT_SYMBOL(blkdev_issue_flush); |
| @@ -178,12 +178,12 @@ · 177 unmodified lines | ||
| 178 | 178 | #include <linux/time64.h> |
| 179 | 179 | #include <linux/parser.h> |
| 180 | 180 | #include <linux/sched/signal.h> |
| 181 | −#include <linux/blk-cgroup.h> | |
| 182 | 181 | #include <asm/local.h> |
| 183 | 182 | #include <asm/local64.h> |
| 184 | 183 | #include "blk-rq-qos.h" |
| 185 | 184 | #include "blk-stat.h" |
| 186 | 185 | #include "blk-wbt.h" |
| 186 | +#include "blk-cgroup.h" | |
| 187 | 187 | |
| 188 | 188 | #ifdef CONFIG_TRACEPOINTS |
| 189 | 189 | |
| @@ -74,9 +74,9 @@ · 73 unmodified lines | ||
| 74 | 74 | #include <linux/sched/signal.h> |
| 75 | 75 | #include <trace/events/block.h> |
| 76 | 76 | #include <linux/blk-mq.h> |
| 77 | −#include <linux/blk-cgroup.h> | |
| 78 | 77 | #include "blk-rq-qos.h" |
| 79 | 78 | #include "blk-stat.h" |
| 79 | +#include "blk-cgroup.h" | |
| 80 | 80 | #include "blk.h" |
| 81 | 81 | |
| 82 | 82 | #define DEFAULT_SCALE_COOKIE 1000000U |
| @@ -598,7 +598,7 @@ static void blkcg_iolatency_done_bio(struct rq_qos *rqos, struct bio *bio) · 515 unmodified lines | ||
| 598 | 598 | int inflight = 0; |
| 599 | 599 | |
| 600 | 600 | blkg = bio->bi_blkg; |
| 601 | − if (!blkg || !bio_flagged(bio, BIO_TRACKED)) | |
| 601 | + if (!blkg || !bio_flagged(bio, BIO_QOS_THROTTLED)) | |
| 602 | 602 | return; |
| 603 | 603 | |
| 604 | 604 | iolat = blkg_to_lat(bio->bi_blkg); |
| @@ -12,11 +12,11 @@ · 11 unmodified lines | ||
| 12 | 12 | * Documentation/admin-guide/cgroup-v2.rst. |
| 13 | 13 | */ |
| 14 | 14 | |
| 15 | −#include <linux/blk-cgroup.h> | |
| 16 | 15 | #include <linux/blk-mq.h> |
| 17 | 16 | #include <linux/blk_types.h> |
| 18 | 17 | #include <linux/kernel.h> |
| 19 | 18 | #include <linux/module.h> |
| 19 | +#include "blk-cgroup.h" | |
| 20 | 20 | #include "blk-ioprio.h" |
| 21 | 21 | #include "blk-rq-qos.h" |
| 22 | 22 | |
| @@ -10,19 +10,6 @@ · 9 unmodified lines | ||
| 10 | 10 | |
| 11 | 11 | #include "blk.h" |
| 12 | 12 | |
| 13 | −struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp) | |
| 14 | −{ | |
| 15 | − struct bio *new = bio_alloc(gfp, nr_pages); | |
| 16 | − | |
| 17 | − if (bio) { | |
| 18 | − bio_chain(bio, new); | |
| 19 | − submit_bio(bio); | |
| 20 | − } | |
| 21 | − | |
| 22 | − return new; | |
| 23 | −} | |
| 24 | −EXPORT_SYMBOL_GPL(blk_next_bio); | |
| 25 | − | |
| 26 | 13 | int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, |
| 27 | 14 | sector_t nr_sects, gfp_t gfp_mask, int flags, |
| 28 | 15 | struct bio **biop) |
| @@ -32,9 +19,6 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, · 3 unmodified lines | ||
| 32 | 19 | unsigned int op; |
| 33 | 20 | sector_t bs_mask, part_offset = 0; |
| 34 | 21 | |
| 35 | − if (!q) | |
| 36 | − return -ENXIO; | |
| 37 | − | |
| 38 | 22 | if (bdev_read_only(bdev)) |
| 39 | 23 | return -EPERM; |
| 40 | 24 | |
| @@ -95,11 +79,8 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, · 54 unmodified lines | ||
| 95 | 79 | |
| 96 | 80 | WARN_ON_ONCE((req_sects << 9) > UINT_MAX); |
| 97 | 81 | |
| 98 | − bio = blk_next_bio(bio, 0, gfp_mask); | |
| 82 | + bio = blk_next_bio(bio, bdev, 0, op, gfp_mask); | |
| 99 | 83 | bio->bi_iter.bi_sector = sector; |
| 100 | − bio_set_dev(bio, bdev); | |
| 101 | − bio_set_op_attrs(bio, op, 0); | |
| 102 | − | |
| 103 | 84 | bio->bi_iter.bi_size = req_sects << 9; |
| 104 | 85 | sector += req_sects; |
| 105 | 86 | nr_sects -= req_sects; |
| @@ -172,9 +153,6 @@ static int __blkdev_issue_write_same(struct block_device *bdev, sector_t sector, · 66 unmodified lines | ||
| 172 | 153 | struct bio *bio = *biop; |
| 173 | 154 | sector_t bs_mask; |
| 174 | 155 | |
| 175 | − if (!q) | |
| 176 | − return -ENXIO; | |
| 177 | − | |
| 178 | 156 | if (bdev_read_only(bdev)) |
| 179 | 157 | return -EPERM; |
| 180 | 158 | |
| @@ -189,14 +167,12 @@ static int __blkdev_issue_write_same(struct block_device *bdev, sector_t sector, · 8 unmodified lines | ||
| 189 | 167 | max_write_same_sectors = bio_allowed_max_sectors(q); |
| 190 | 168 | |
| 191 | 169 | while (nr_sects) { |
| 192 | − bio = blk_next_bio(bio, 1, gfp_mask); | |
| 170 | + bio = blk_next_bio(bio, bdev, 1, REQ_OP_WRITE_SAME, gfp_mask); | |
| 193 | 171 | bio->bi_iter.bi_sector = sector; |
| 194 | − bio_set_dev(bio, bdev); | |
| 195 | 172 | bio->bi_vcnt = 1; |
| 196 | 173 | bio->bi_io_vec->bv_page = page; |
| 197 | 174 | bio->bi_io_vec->bv_offset = 0; |
| 198 | 175 | bio->bi_io_vec->bv_len = bdev_logical_block_size(bdev); |
| 199 | − bio_set_op_attrs(bio, REQ_OP_WRITE_SAME, 0); | |
| 200 | 176 | |
| 201 | 177 | if (nr_sects > max_write_same_sectors) { |
| 202 | 178 | bio->bi_iter.bi_size = max_write_same_sectors << 9; |
| @@ -250,10 +226,6 @@ static int __blkdev_issue_write_zeroes(struct block_device *bdev, · 47 unmodified lines | ||
| 250 | 226 | { |
| 251 | 227 | struct bio *bio = *biop; |
| 252 | 228 | unsigned int max_write_zeroes_sectors; |
| 253 | − struct request_queue *q = bdev_get_queue(bdev); | |
| 254 | − | |
| 255 | − if (!q) | |
| 256 | − return -ENXIO; | |
| 257 | 229 | |
| 258 | 230 | if (bdev_read_only(bdev)) |
| 259 | 231 | return -EPERM; |
| @@ -265,10 +237,8 @@ static int __blkdev_issue_write_zeroes(struct block_device *bdev, · 5 unmodified lines | ||
| 265 | 237 | return -EOPNOTSUPP; |
| 266 | 238 | |
| 267 | 239 | while (nr_sects) { |
| 268 | − bio = blk_next_bio(bio, 0, gfp_mask); | |
| 240 | + bio = blk_next_bio(bio, bdev, 0, REQ_OP_WRITE_ZEROES, gfp_mask); | |
| 269 | 241 | bio->bi_iter.bi_sector = sector; |
| 270 | − bio_set_dev(bio, bdev); | |
| 271 | − bio->bi_opf = REQ_OP_WRITE_ZEROES; | |
| 272 | 242 | if (flags & BLKDEV_ZERO_NOUNMAP) |
| 273 | 243 | bio->bi_opf |= REQ_NOUNMAP; |
| 274 | 244 | |
| @@ -304,23 +274,17 @@ static int __blkdev_issue_zero_pages(struct block_device *bdev, · 29 unmodified lines | ||
| 304 | 274 | sector_t sector, sector_t nr_sects, gfp_t gfp_mask, |
| 305 | 275 | struct bio **biop) |
| 306 | 276 | { |
| 307 | − struct request_queue *q = bdev_get_queue(bdev); | |
| 308 | 277 | struct bio *bio = *biop; |
| 309 | 278 | int bi_size = 0; |
| 310 | 279 | unsigned int sz; |
| 311 | 280 | |
| 312 | − if (!q) | |
| 313 | − return -ENXIO; | |
| 314 | − | |
| 315 | 281 | if (bdev_read_only(bdev)) |
| 316 | 282 | return -EPERM; |
| 317 | 283 | |
| 318 | 284 | while (nr_sects != 0) { |
| 319 | − bio = blk_next_bio(bio, __blkdev_sectors_to_bio_pages(nr_sects), | |
| 320 | − gfp_mask); | |
| 285 | + bio = blk_next_bio(bio, bdev, __blkdev_sectors_to_bio_pages(nr_sects), | |
| 286 | + REQ_OP_WRITE, gfp_mask); | |
| 321 | 287 | bio->bi_iter.bi_sector = sector; |
| 322 | − bio_set_dev(bio, bdev); | |
| 323 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 324 | 288 | |
| 325 | 289 | while (nr_sects != 0) { |
| 326 | 290 | sz = min((sector_t) PAGE_SIZE, nr_sects << 9); |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <linux/blk-integrity.h> |
| 10 | 10 | #include <linux/scatterlist.h> |
| 11 | 11 | #include <linux/part_stat.h> |
| 12 | +#include <linux/blk-cgroup.h> | |
| 12 | 13 | |
| 13 | 14 | #include <trace/events/block.h> |
| 14 | 15 | |
| @@ -368,8 +369,6 @@ void __blk_queue_split(struct request_queue *q, struct bio **bio, · 353 unmodified lines | ||
| 368 | 369 | trace_block_split(split, (*bio)->bi_iter.bi_sector); |
| 369 | 370 | submit_bio_noacct(*bio); |
| 370 | 371 | *bio = split; |
| 371 | − | |
| 372 | − blk_throtl_charge_bio_split(*bio); | |
| 373 | 372 | } |
| 374 | 373 | } |
| 375 | 374 | |
| @@ -600,6 +599,9 @@ static inline unsigned int blk_rq_get_max_sectors(struct request *rq, · 224 unmodified lines | ||
| 600 | 599 | static inline int ll_new_hw_segment(struct request *req, struct bio *bio, |
| 601 | 600 | unsigned int nr_phys_segs) |
| 602 | 601 | { |
| 602 | + if (!blk_cgroup_mergeable(req, bio)) | |
| 603 | + goto no_merge; | |
| 604 | + | |
| 603 | 605 | if (blk_integrity_merge_bio(req->q, req, bio) == false) |
| 604 | 606 | goto no_merge; |
| 605 | 607 | |
| @@ -696,6 +698,9 @@ static int ll_merge_requests_fn(struct request_queue *q, struct request *req, · 90 unmodified lines | ||
| 696 | 698 | if (total_phys_segments > blk_rq_get_max_segments(req)) |
| 697 | 699 | return 0; |
| 698 | 700 | |
| 701 | + if (!blk_cgroup_mergeable(req, next->bio)) | |
| 702 | + return 0; | |
| 703 | + | |
| 699 | 704 | if (blk_integrity_merge_rq(q, req, next) == false) |
| 700 | 705 | return 0; |
| 701 | 706 | |
| @@ -904,6 +909,10 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) · 202 unmodified lines | ||
| 904 | 909 | if (bio_data_dir(bio) != rq_data_dir(rq)) |
| 905 | 910 | return false; |
| 906 | 911 | |
| 912 | + /* don't merge across cgroup boundaries */ | |
| 913 | + if (!blk_cgroup_mergeable(rq, bio)) | |
| 914 | + return false; | |
| 915 | + | |
| 907 | 916 | /* only merge integrity protected bio into ditto rq */ |
| 908 | 917 | if (blk_integrity_merge_bio(rq->q, rq, bio) == false) |
| 909 | 918 | return false; |
| @@ -1089,12 +1098,20 @@ bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, · 179 unmodified lines | ||
| 1089 | 1098 | if (!plug || rq_list_empty(plug->mq_list)) |
| 1090 | 1099 | return false; |
| 1091 | 1100 | |
| 1092 | − /* check the previously added entry for a quick merge attempt */ | |
| 1093 | − rq = rq_list_peek(&plug->mq_list); | |
| 1094 | − if (rq->q == q) { | |
| 1095 | − if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) == | |
| 1096 | − BIO_MERGE_OK) | |
| 1097 | − return true; | |
| 1101 | + rq_list_for_each(&plug->mq_list, rq) { | |
| 1102 | + if (rq->q == q) { | |
| 1103 | + if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) == | |
| 1104 | + BIO_MERGE_OK) | |
| 1105 | + return true; | |
| 1106 | + break; | |
| 1107 | + } | |
| 1108 | + | |
| 1109 | + /* | |
| 1110 | + * Only keep iterating plug list for merges if we have multiple | |
| 1111 | + * queues | |
| 1112 | + */ | |
| 1113 | + if (!plug->multiple_queues) | |
| 1114 | + break; | |
| 1098 | 1115 | } |
| 1099 | 1116 | return false; |
| 1100 | 1117 | } |
| @@ -707,7 +707,7 @@ static void debugfs_create_files(struct dentry *parent, void *data, · 706 unmodified lines | ||
| 707 | 707 | void blk_mq_debugfs_register(struct request_queue *q) |
| 708 | 708 | { |
| 709 | 709 | struct blk_mq_hw_ctx *hctx; |
| 710 | − int i; | |
| 710 | + unsigned long i; | |
| 711 | 711 | |
| 712 | 712 | debugfs_create_files(q->debugfs_dir, q, blk_mq_debugfs_queue_attrs); |
| 713 | 713 | |
| @@ -780,7 +780,7 @@ void blk_mq_debugfs_unregister_hctx(struct blk_mq_hw_ctx *hctx) · 66 unmodified lines | ||
| 780 | 780 | void blk_mq_debugfs_register_hctxs(struct request_queue *q) |
| 781 | 781 | { |
| 782 | 782 | struct blk_mq_hw_ctx *hctx; |
| 783 | − int i; | |
| 783 | + unsigned long i; | |
| 784 | 784 | |
| 785 | 785 | queue_for_each_hw_ctx(q, hctx, i) |
| 786 | 786 | blk_mq_debugfs_register_hctx(q, hctx); |
| @@ -789,7 +789,7 @@ void blk_mq_debugfs_register_hctxs(struct request_queue *q) · 2 unmodified lines | ||
| 789 | 789 | void blk_mq_debugfs_unregister_hctxs(struct request_queue *q) |
| 790 | 790 | { |
| 791 | 791 | struct blk_mq_hw_ctx *hctx; |
| 792 | − int i; | |
| 792 | + unsigned long i; | |
| 793 | 793 | |
| 794 | 794 | queue_for_each_hw_ctx(q, hctx, i) |
| 795 | 795 | blk_mq_debugfs_unregister_hctx(hctx); |
| @@ -6,6 +6,8 @@ · 5 unmodified lines | ||
| 6 | 6 | |
| 7 | 7 | #include <linux/seq_file.h> |
| 8 | 8 | |
| 9 | +struct blk_mq_hw_ctx; | |
| 10 | + | |
| 9 | 11 | struct blk_mq_debugfs_attr { |
| 10 | 12 | const char *name; |
| 11 | 13 | umode_t mode; |
| @@ -180,11 +180,18 @@ static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) · 179 unmodified lines | ||
| 180 | 180 | |
| 181 | 181 | static int blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) |
| 182 | 182 | { |
| 183 | + unsigned long end = jiffies + HZ; | |
| 183 | 184 | int ret; |
| 184 | 185 | |
| 185 | 186 | do { |
| 186 | 187 | ret = __blk_mq_do_dispatch_sched(hctx); |
| 187 | − } while (ret == 1); | |
| 188 | + if (ret != 1) | |
| 189 | + break; | |
| 190 | + if (need_resched() || time_is_before_jiffies(end)) { | |
| 191 | + blk_mq_delay_run_hw_queue(hctx, 0); | |
| 192 | + break; | |
| 193 | + } | |
| 194 | + } while (1); | |
| 188 | 195 | |
| 189 | 196 | return ret; |
| 190 | 197 | } |
| @@ -515,7 +522,7 @@ static void blk_mq_exit_sched_shared_tags(struct request_queue *queue) · 324 unmodified lines | ||
| 515 | 522 | static void blk_mq_sched_tags_teardown(struct request_queue *q, unsigned int flags) |
| 516 | 523 | { |
| 517 | 524 | struct blk_mq_hw_ctx *hctx; |
| 518 | − int i; | |
| 525 | + unsigned long i; | |
| 519 | 526 | |
| 520 | 527 | queue_for_each_hw_ctx(q, hctx, i) { |
| 521 | 528 | if (hctx->sched_tags) { |
| @@ -550,9 +557,10 @@ static int blk_mq_init_sched_shared_tags(struct request_queue *queue) · 28 unmodified lines | ||
| 550 | 557 | |
| 551 | 558 | int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e) |
| 552 | 559 | { |
| 553 | − unsigned int i, flags = q->tag_set->flags; | |
| 560 | + unsigned int flags = q->tag_set->flags; | |
| 554 | 561 | struct blk_mq_hw_ctx *hctx; |
| 555 | 562 | struct elevator_queue *eq; |
| 563 | + unsigned long i; | |
| 556 | 564 | int ret; |
| 557 | 565 | |
| 558 | 566 | if (!e) { |
| @@ -618,7 +626,7 @@ err_free_map_and_rqs: · 59 unmodified lines | ||
| 618 | 626 | void blk_mq_sched_free_rqs(struct request_queue *q) |
| 619 | 627 | { |
| 620 | 628 | struct blk_mq_hw_ctx *hctx; |
| 621 | − int i; | |
| 629 | + unsigned long i; | |
| 622 | 630 | |
| 623 | 631 | if (blk_mq_is_shared_tags(q->tag_set->flags)) { |
| 624 | 632 | blk_mq_free_rqs(q->tag_set, q->sched_shared_tags, |
| @@ -635,7 +643,7 @@ void blk_mq_sched_free_rqs(struct request_queue *q) · 10 unmodified lines | ||
| 635 | 643 | void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e) |
| 636 | 644 | { |
| 637 | 645 | struct blk_mq_hw_ctx *hctx; |
| 638 | − unsigned int i; | |
| 646 | + unsigned long i; | |
| 639 | 647 | unsigned int flags = 0; |
| 640 | 648 | |
| 641 | 649 | queue_for_each_hw_ctx(q, hctx, i) { |
| @@ -206,7 +206,7 @@ static int blk_mq_register_hctx(struct blk_mq_hw_ctx *hctx) · 205 unmodified lines | ||
| 206 | 206 | void blk_mq_unregister_dev(struct device *dev, struct request_queue *q) |
| 207 | 207 | { |
| 208 | 208 | struct blk_mq_hw_ctx *hctx; |
| 209 | − int i; | |
| 209 | + unsigned long i; | |
| 210 | 210 | |
| 211 | 211 | lockdep_assert_held(&q->sysfs_dir_lock); |
| 212 | 212 | |
| @@ -255,7 +255,8 @@ void blk_mq_sysfs_init(struct request_queue *q) · 42 unmodified lines | ||
| 255 | 255 | int __blk_mq_register_dev(struct device *dev, struct request_queue *q) |
| 256 | 256 | { |
| 257 | 257 | struct blk_mq_hw_ctx *hctx; |
| 258 | − int ret, i; | |
| 258 | + unsigned long i, j; | |
| 259 | + int ret; | |
| 259 | 260 | |
| 260 | 261 | WARN_ON_ONCE(!q->kobj.parent); |
| 261 | 262 | lockdep_assert_held(&q->sysfs_dir_lock); |
| @@ -278,8 +279,10 @@ out: · 16 unmodified lines | ||
| 278 | 279 | return ret; |
| 279 | 280 | |
| 280 | 281 | unreg: |
| 281 | − while (--i >= 0) | |
| 282 | − blk_mq_unregister_hctx(q->queue_hw_ctx[i]); | |
| 282 | + queue_for_each_hw_ctx(q, hctx, j) { | |
| 283 | + if (j < i) | |
| 284 | + blk_mq_unregister_hctx(hctx); | |
| 285 | + } | |
| 283 | 286 | |
| 284 | 287 | kobject_uevent(q->mq_kobj, KOBJ_REMOVE); |
| 285 | 288 | kobject_del(q->mq_kobj); |
| @@ -290,7 +293,7 @@ unreg: · 4 unmodified lines | ||
| 290 | 293 | void blk_mq_sysfs_unregister(struct request_queue *q) |
| 291 | 294 | { |
| 292 | 295 | struct blk_mq_hw_ctx *hctx; |
| 293 | − int i; | |
| 296 | + unsigned long i; | |
| 294 | 297 | |
| 295 | 298 | mutex_lock(&q->sysfs_dir_lock); |
| 296 | 299 | if (!q->mq_sysfs_init_done) |
| @@ -306,7 +309,8 @@ unlock: · 9 unmodified lines | ||
| 306 | 309 | int blk_mq_sysfs_register(struct request_queue *q) |
| 307 | 310 | { |
| 308 | 311 | struct blk_mq_hw_ctx *hctx; |
| 309 | − int i, ret = 0; | |
| 312 | + unsigned long i; | |
| 313 | + int ret = 0; | |
| 310 | 314 | |
| 311 | 315 | mutex_lock(&q->sysfs_dir_lock); |
| 312 | 316 | if (!q->mq_sysfs_init_done) |
| @@ -107,7 +107,7 @@ static int __blk_mq_get_tag(struct blk_mq_alloc_data *data, · 106 unmodified lines | ||
| 107 | 107 | return BLK_MQ_NO_TAG; |
| 108 | 108 | |
| 109 | 109 | if (data->shallow_depth) |
| 110 | − return __sbitmap_queue_get_shallow(bt, data->shallow_depth); | |
| 110 | + return sbitmap_queue_get_shallow(bt, data->shallow_depth); | |
| 111 | 111 | else |
| 112 | 112 | return __sbitmap_queue_get(bt); |
| 113 | 113 | } |
| @@ -498,7 +498,7 @@ void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_tag_iter_fn *fn, · 384 unmodified lines | ||
| 498 | 498 | void *priv) |
| 499 | 499 | { |
| 500 | 500 | /* |
| 501 | − * __blk_mq_update_nr_hw_queues() updates nr_hw_queues and queue_hw_ctx | |
| 501 | + * __blk_mq_update_nr_hw_queues() updates nr_hw_queues and hctx_table | |
| 502 | 502 | * while the queue is frozen. So we can use q_usage_counter to avoid |
| 503 | 503 | * racing with it. |
| 504 | 504 | */ |
| @@ -515,7 +515,7 @@ void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_tag_iter_fn *fn, · 10 unmodified lines | ||
| 515 | 515 | bt_for_each(NULL, q, btags, fn, priv, false); |
| 516 | 516 | } else { |
| 517 | 517 | struct blk_mq_hw_ctx *hctx; |
| 518 | − int i; | |
| 518 | + unsigned long i; | |
| 519 | 519 | |
| 520 | 520 | queue_for_each_hw_ctx(q, hctx, i) { |
| 521 | 521 | struct blk_mq_tags *tags = hctx->tags; |
| @@ -71,7 +71,8 @@ static int blk_mq_poll_stats_bkt(const struct request *rq) · 70 unmodified lines | ||
| 71 | 71 | static inline struct blk_mq_hw_ctx *blk_qc_to_hctx(struct request_queue *q, |
| 72 | 72 | blk_qc_t qc) |
| 73 | 73 | { |
| 74 | − return q->queue_hw_ctx[(qc & ~BLK_QC_T_INTERNAL) >> BLK_QC_T_SHIFT]; | |
| 74 | + return xa_load(&q->hctx_table, | |
| 75 | + (qc & ~BLK_QC_T_INTERNAL) >> BLK_QC_T_SHIFT); | |
| 75 | 76 | } |
| 76 | 77 | |
| 77 | 78 | static inline struct request *blk_qc_to_rq(struct blk_mq_hw_ctx *hctx, |
| @@ -312,7 +313,7 @@ EXPORT_SYMBOL_GPL(blk_mq_unquiesce_queue); · 234 unmodified lines | ||
| 312 | 313 | void blk_mq_wake_waiters(struct request_queue *q) |
| 313 | 314 | { |
| 314 | 315 | struct blk_mq_hw_ctx *hctx; |
| 315 | − unsigned int i; | |
| 316 | + unsigned long i; | |
| 316 | 317 | |
| 317 | 318 | queue_for_each_hw_ctx(q, hctx, i) |
| 318 | 319 | if (blk_mq_hw_queue_mapped(hctx)) |
| @@ -573,7 +574,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, · 254 unmodified lines | ||
| 573 | 574 | * If not tell the caller that it should skip this queue. |
| 574 | 575 | */ |
| 575 | 576 | ret = -EXDEV; |
| 576 | − data.hctx = q->queue_hw_ctx[hctx_idx]; | |
| 577 | + data.hctx = xa_load(&q->hctx_table, hctx_idx); | |
| 577 | 578 | if (!blk_mq_hw_queue_mapped(data.hctx)) |
| 578 | 579 | goto out_queue_exit; |
| 579 | 580 | cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask); |
| @@ -793,8 +794,10 @@ bool blk_update_request(struct request *req, blk_status_t error, · 213 unmodified lines | ||
| 793 | 794 | #endif |
| 794 | 795 | |
| 795 | 796 | if (unlikely(error && !blk_rq_is_passthrough(req) && |
| 796 | − !(req->rq_flags & RQF_QUIET))) | |
| 797 | + !(req->rq_flags & RQF_QUIET))) { | |
| 797 | 798 | blk_print_req_error(req, error); |
| 799 | + trace_block_rq_error(req, error, nr_bytes); | |
| 800 | + } | |
| 798 | 801 | |
| 799 | 802 | blk_account_io_completion(req, nr_bytes); |
| 800 | 803 | |
| @@ -885,10 +888,15 @@ static inline void blk_account_io_done(struct request *req, u64 now) · 84 unmodified lines | ||
| 885 | 888 | |
| 886 | 889 | static void __blk_account_io_start(struct request *rq) |
| 887 | 890 | { |
| 888 | − /* passthrough requests can hold bios that do not have ->bi_bdev set */ | |
| 889 | − if (rq->bio && rq->bio->bi_bdev) | |
| 891 | + /* | |
| 892 | + * All non-passthrough requests are created from a bio with one | |
| 893 | + * exception: when a flush command that is part of a flush sequence | |
| 894 | + * generated by the state machine in blk-flush.c is cloned onto the | |
| 895 | + * lower device by dm-multipath we can get here without a bio. | |
| 896 | + */ | |
| 897 | + if (rq->bio) | |
| 890 | 898 | rq->part = rq->bio->bi_bdev; |
| 891 | − else if (rq->q->disk) | |
| 899 | + else | |
| 892 | 900 | rq->part = rq->q->disk->part0; |
| 893 | 901 | |
| 894 | 902 | part_stat_lock(); |
| @@ -1444,7 +1452,7 @@ static void blk_mq_timeout_work(struct work_struct *work) · 549 unmodified lines | ||
| 1444 | 1452 | container_of(work, struct request_queue, timeout_work); |
| 1445 | 1453 | unsigned long next = 0; |
| 1446 | 1454 | struct blk_mq_hw_ctx *hctx; |
| 1447 | − int i; | |
| 1455 | + unsigned long i; | |
| 1448 | 1456 | |
| 1449 | 1457 | /* A deadlock might occur if a request is stuck requiring a |
| 1450 | 1458 | * timeout at the same time a queue freeze is waiting |
| @@ -2145,7 +2153,7 @@ static struct blk_mq_hw_ctx *blk_mq_get_sq_hctx(struct request_queue *q) · 694 unmodified lines | ||
| 2145 | 2153 | void blk_mq_run_hw_queues(struct request_queue *q, bool async) |
| 2146 | 2154 | { |
| 2147 | 2155 | struct blk_mq_hw_ctx *hctx, *sq_hctx; |
| 2148 | − int i; | |
| 2156 | + unsigned long i; | |
| 2149 | 2157 | |
| 2150 | 2158 | sq_hctx = NULL; |
| 2151 | 2159 | if (blk_mq_has_sqsched(q)) |
| @@ -2173,7 +2181,7 @@ EXPORT_SYMBOL(blk_mq_run_hw_queues); · 21 unmodified lines | ||
| 2173 | 2181 | void blk_mq_delay_run_hw_queues(struct request_queue *q, unsigned long msecs) |
| 2174 | 2182 | { |
| 2175 | 2183 | struct blk_mq_hw_ctx *hctx, *sq_hctx; |
| 2176 | − int i; | |
| 2184 | + unsigned long i; | |
| 2177 | 2185 | |
| 2178 | 2186 | sq_hctx = NULL; |
| 2179 | 2187 | if (blk_mq_has_sqsched(q)) |
| @@ -2181,6 +2189,14 @@ void blk_mq_delay_run_hw_queues(struct request_queue *q, unsigned long msecs) · 1 unmodified line | ||
| 2181 | 2189 | queue_for_each_hw_ctx(q, hctx, i) { |
| 2182 | 2190 | if (blk_mq_hctx_stopped(hctx)) |
| 2183 | 2191 | continue; |
| 2192 | + /* | |
| 2193 | + * If there is already a run_work pending, leave the | |
| 2194 | + * pending delay untouched. Otherwise, a hctx can stall | |
| 2195 | + * if another hctx is re-delaying the other's work | |
| 2196 | + * before the work executes. | |
| 2197 | + */ | |
| 2198 | + if (delayed_work_pending(&hctx->run_work)) | |
| 2199 | + continue; | |
| 2184 | 2200 | /* |
| 2185 | 2201 | * Dispatch from this hctx either if there's no hctx preferred |
| 2186 | 2202 | * by IO scheduler or if it has requests that bypass the |
| @@ -2203,7 +2219,7 @@ EXPORT_SYMBOL(blk_mq_delay_run_hw_queues); · 16 unmodified lines | ||
| 2203 | 2219 | bool blk_mq_queue_stopped(struct request_queue *q) |
| 2204 | 2220 | { |
| 2205 | 2221 | struct blk_mq_hw_ctx *hctx; |
| 2206 | − int i; | |
| 2222 | + unsigned long i; | |
| 2207 | 2223 | |
| 2208 | 2224 | queue_for_each_hw_ctx(q, hctx, i) |
| 2209 | 2225 | if (blk_mq_hctx_stopped(hctx)) |
| @@ -2242,7 +2258,7 @@ EXPORT_SYMBOL(blk_mq_stop_hw_queue); · 32 unmodified lines | ||
| 2242 | 2258 | void blk_mq_stop_hw_queues(struct request_queue *q) |
| 2243 | 2259 | { |
| 2244 | 2260 | struct blk_mq_hw_ctx *hctx; |
| 2245 | − int i; | |
| 2261 | + unsigned long i; | |
| 2246 | 2262 | |
| 2247 | 2263 | queue_for_each_hw_ctx(q, hctx, i) |
| 2248 | 2264 | blk_mq_stop_hw_queue(hctx); |
| @@ -2260,7 +2276,7 @@ EXPORT_SYMBOL(blk_mq_start_hw_queue); · 11 unmodified lines | ||
| 2260 | 2276 | void blk_mq_start_hw_queues(struct request_queue *q) |
| 2261 | 2277 | { |
| 2262 | 2278 | struct blk_mq_hw_ctx *hctx; |
| 2263 | − int i; | |
| 2279 | + unsigned long i; | |
| 2264 | 2280 | |
| 2265 | 2281 | queue_for_each_hw_ctx(q, hctx, i) |
| 2266 | 2282 | blk_mq_start_hw_queue(hctx); |
| @@ -2280,7 +2296,7 @@ EXPORT_SYMBOL_GPL(blk_mq_start_stopped_hw_queue); · 13 unmodified lines | ||
| 2280 | 2296 | void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async) |
| 2281 | 2297 | { |
| 2282 | 2298 | struct blk_mq_hw_ctx *hctx; |
| 2283 | − int i; | |
| 2299 | + unsigned long i; | |
| 2284 | 2300 | |
| 2285 | 2301 | queue_for_each_hw_ctx(q, hctx, i) |
| 2286 | 2302 | blk_mq_start_stopped_hw_queue(hctx, async); |
| @@ -2561,13 +2577,36 @@ static void __blk_mq_flush_plug_list(struct request_queue *q, · 274 unmodified lines | ||
| 2561 | 2577 | q->mq_ops->queue_rqs(&plug->mq_list); |
| 2562 | 2578 | } |
| 2563 | 2579 | |
| 2580 | +static void blk_mq_dispatch_plug_list(struct blk_plug *plug, bool from_sched) | |
| 2581 | +{ | |
| 2582 | + struct blk_mq_hw_ctx *this_hctx = NULL; | |
| 2583 | + struct blk_mq_ctx *this_ctx = NULL; | |
| 2584 | + struct request *requeue_list = NULL; | |
| 2585 | + unsigned int depth = 0; | |
| 2586 | + LIST_HEAD(list); | |
| 2587 | + | |
| 2588 | + do { | |
| 2589 | + struct request *rq = rq_list_pop(&plug->mq_list); | |
| 2590 | + | |
| 2591 | + if (!this_hctx) { | |
| 2592 | + this_hctx = rq->mq_hctx; | |
| 2593 | + this_ctx = rq->mq_ctx; | |
| 2594 | + } else if (this_hctx != rq->mq_hctx || this_ctx != rq->mq_ctx) { | |
| 2595 | + rq_list_add(&requeue_list, rq); | |
| 2596 | + continue; | |
| 2597 | + } | |
| 2598 | + list_add_tail(&rq->queuelist, &list); | |
| 2599 | + depth++; | |
| 2600 | + } while (!rq_list_empty(plug->mq_list)); | |
| 2601 | + | |
| 2602 | + plug->mq_list = requeue_list; | |
| 2603 | + trace_block_unplug(this_hctx->queue, depth, !from_sched); | |
| 2604 | + blk_mq_sched_insert_requests(this_hctx, this_ctx, &list, from_sched); | |
| 2605 | +} | |
| 2606 | + | |
| 2564 | 2607 | void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) |
| 2565 | 2608 | { |
| 2566 | − struct blk_mq_hw_ctx *this_hctx; | |
| 2567 | − struct blk_mq_ctx *this_ctx; | |
| 2568 | 2609 | struct request *rq; |
| 2569 | − unsigned int depth; | |
| 2570 | − LIST_HEAD(list); | |
| 2571 | 2610 | |
| 2572 | 2611 | if (rq_list_empty(plug->mq_list)) |
| 2573 | 2612 | return; |
| @@ -2603,35 +2642,9 @@ void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule) · 29 unmodified lines | ||
| 2603 | 2642 | return; |
| 2604 | 2643 | } |
| 2605 | 2644 | |
| 2606 | − this_hctx = NULL; | |
| 2607 | − this_ctx = NULL; | |
| 2608 | − depth = 0; | |
| 2609 | 2645 | do { |
| 2610 | − rq = rq_list_pop(&plug->mq_list); | |
| 2611 | − | |
| 2612 | − if (!this_hctx) { | |
| 2613 | − this_hctx = rq->mq_hctx; | |
| 2614 | − this_ctx = rq->mq_ctx; | |
| 2615 | − } else if (this_hctx != rq->mq_hctx || this_ctx != rq->mq_ctx) { | |
| 2616 | − trace_block_unplug(this_hctx->queue, depth, | |
| 2617 | − !from_schedule); | |
| 2618 | − blk_mq_sched_insert_requests(this_hctx, this_ctx, | |
| 2619 | − &list, from_schedule); | |
| 2620 | − depth = 0; | |
| 2621 | − this_hctx = rq->mq_hctx; | |
| 2622 | − this_ctx = rq->mq_ctx; | |
| 2623 | − | |
| 2624 | − } | |
| 2625 | − | |
| 2626 | − list_add(&rq->queuelist, &list); | |
| 2627 | − depth++; | |
| 2646 | + blk_mq_dispatch_plug_list(plug, from_schedule); | |
| 2628 | 2647 | } while (!rq_list_empty(plug->mq_list)); |
| 2629 | − | |
| 2630 | − if (!list_empty(&list)) { | |
| 2631 | − trace_block_unplug(this_hctx->queue, depth, !from_schedule); | |
| 2632 | − blk_mq_sched_insert_requests(this_hctx, this_ctx, &list, | |
| 2633 | − from_schedule); | |
| 2634 | − } | |
| 2635 | 2648 | } |
| 2636 | 2649 | |
| 2637 | 2650 | void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx, |
| @@ -2804,9 +2817,6 @@ void blk_mq_submit_bio(struct bio *bio) · 166 unmodified lines | ||
| 2804 | 2817 | unsigned int nr_segs = 1; |
| 2805 | 2818 | blk_status_t ret; |
| 2806 | 2819 | |
| 2807 | − if (unlikely(!blk_crypto_bio_prep(&bio))) | |
| 2808 | − return; | |
| 2809 | − | |
| 2810 | 2820 | blk_queue_bounce(q, &bio); |
| 2811 | 2821 | if (blk_may_split(q, bio)) |
| 2812 | 2822 | __blk_queue_split(q, &bio, &nr_segs); |
| @@ -2853,27 +2863,16 @@ void blk_mq_submit_bio(struct bio *bio) · 40 unmodified lines | ||
| 2853 | 2863 | blk_mq_try_issue_directly(rq->mq_hctx, rq)); |
| 2854 | 2864 | } |
| 2855 | 2865 | |
| 2866 | +#ifdef CONFIG_BLK_MQ_STACKING | |
| 2856 | 2867 | /** |
| 2857 | − * blk_cloned_rq_check_limits - Helper function to check a cloned request | |
| 2858 | − * for the new queue limits | |
| 2859 | − * @q: the queue | |
| 2860 | − * @rq: the request being checked | |
| 2861 | − * | |
| 2862 | − * Description: | |
| 2863 | − * @rq may have been made based on weaker limitations of upper-level queues | |
| 2864 | − * in request stacking drivers, and it may violate the limitation of @q. | |
| 2865 | − * Since the block layer and the underlying device driver trust @rq | |
| 2866 | − * after it is inserted to @q, it should be checked against @q before | |
| 2867 | − * the insertion using this generic function. | |
| 2868 | − * | |
| 2869 | − * Request stacking drivers like request-based dm may change the queue | |
| 2870 | − * limits when retrying requests on other queues. Those requests need | |
| 2871 | − * to be checked against the new queue limits again during dispatch. | |
| 2868 | + * blk_insert_cloned_request - Helper for stacking drivers to submit a request | |
| 2869 | + * @rq: the request being queued | |
| 2872 | 2870 | */ |
| 2873 | −static blk_status_t blk_cloned_rq_check_limits(struct request_queue *q, | |
| 2874 | − struct request *rq) | |
| 2871 | +blk_status_t blk_insert_cloned_request(struct request *rq) | |
| 2875 | 2872 | { |
| 2873 | + struct request_queue *q = rq->q; | |
| 2876 | 2874 | unsigned int max_sectors = blk_queue_get_max_sectors(q, req_op(rq)); |
| 2875 | + blk_status_t ret; | |
| 2877 | 2876 | |
| 2878 | 2877 | if (blk_rq_sectors(rq) > max_sectors) { |
| 2879 | 2878 | /* |
| @@ -2905,24 +2904,7 @@ static blk_status_t blk_cloned_rq_check_limits(struct request_queue *q, · 25 unmodified lines | ||
| 2905 | 2904 | return BLK_STS_IOERR; |
| 2906 | 2905 | } |
| 2907 | 2906 | |
| 2908 | − return BLK_STS_OK; | |
| 2909 | −} | |
| 2910 | − | |
| 2911 | −/** | |
| 2912 | − * blk_insert_cloned_request - Helper for stacking drivers to submit a request | |
| 2913 | − * @q: the queue to submit the request | |
| 2914 | − * @rq: the request being queued | |
| 2915 | − */ | |
| 2916 | −blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request *rq) | |
| 2917 | −{ | |
| 2918 | − blk_status_t ret; | |
| 2919 | − | |
| 2920 | − ret = blk_cloned_rq_check_limits(q, rq); | |
| 2921 | − if (ret != BLK_STS_OK) | |
| 2922 | − return ret; | |
| 2923 | − | |
| 2924 | − if (rq->q->disk && | |
| 2925 | − should_fail_request(rq->q->disk->part0, blk_rq_bytes(rq))) | |
| 2907 | + if (q->disk && should_fail_request(q->disk->part0, blk_rq_bytes(rq))) | |
| 2926 | 2908 | return BLK_STS_IOERR; |
| 2927 | 2909 | |
| 2928 | 2910 | if (blk_crypto_insert_cloned_request(rq)) |
| @@ -2935,7 +2917,7 @@ blk_status_t blk_insert_cloned_request(struct request_queue *q, struct request * · 6 unmodified lines | ||
| 2935 | 2917 | * bypass a potential scheduler on the bottom device for |
| 2936 | 2918 | * insert. |
| 2937 | 2919 | */ |
| 2938 | − blk_mq_run_dispatch_ops(rq->q, | |
| 2920 | + blk_mq_run_dispatch_ops(q, | |
| 2939 | 2921 | ret = blk_mq_request_issue_directly(rq, true)); |
| 2940 | 2922 | if (ret) |
| 2941 | 2923 | blk_account_io_done(rq, ktime_get_ns()); |
| @@ -2990,10 +2972,10 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, · 48 unmodified lines | ||
| 2990 | 2972 | bs = &fs_bio_set; |
| 2991 | 2973 | |
| 2992 | 2974 | __rq_for_each_bio(bio_src, rq_src) { |
| 2993 | − bio = bio_clone_fast(bio_src, gfp_mask, bs); | |
| 2975 | + bio = bio_alloc_clone(rq->q->disk->part0, bio_src, gfp_mask, | |
| 2976 | + bs); | |
| 2994 | 2977 | if (!bio) |
| 2995 | 2978 | goto free_and_out; |
| 2996 | − bio->bi_bdev = rq->q->disk->part0; | |
| 2997 | 2979 | |
| 2998 | 2980 | if (bio_ctr && bio_ctr(bio, bio_src, data)) |
| 2999 | 2981 | goto free_and_out; |
| @@ -3030,6 +3012,7 @@ free_and_out: · 30 unmodified lines | ||
| 3030 | 3012 | return -ENOMEM; |
| 3031 | 3013 | } |
| 3032 | 3014 | EXPORT_SYMBOL_GPL(blk_rq_prep_clone); |
| 3015 | +#endif /* CONFIG_BLK_MQ_STACKING */ | |
| 3033 | 3016 | |
| 3034 | 3017 | /* |
| 3035 | 3018 | * Steal bios from a request and add them to a bio list. |
| @@ -3100,6 +3083,9 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags, · 64 unmodified lines | ||
| 3100 | 3083 | struct blk_mq_tags *drv_tags; |
| 3101 | 3084 | struct page *page; |
| 3102 | 3085 | |
| 3086 | + if (list_empty(&tags->page_list)) | |
| 3087 | + return; | |
| 3088 | + | |
| 3103 | 3089 | if (blk_mq_is_shared_tags(set->flags)) |
| 3104 | 3090 | drv_tags = set->shared_tags; |
| 3105 | 3091 | else |
| @@ -3142,15 +3128,41 @@ void blk_mq_free_rq_map(struct blk_mq_tags *tags) · 36 unmodified lines | ||
| 3142 | 3128 | blk_mq_free_tags(tags); |
| 3143 | 3129 | } |
| 3144 | 3130 | |
| 3131 | +static enum hctx_type hctx_idx_to_type(struct blk_mq_tag_set *set, | |
| 3132 | + unsigned int hctx_idx) | |
| 3133 | +{ | |
| 3134 | + int i; | |
| 3135 | + | |
| 3136 | + for (i = 0; i < set->nr_maps; i++) { | |
| 3137 | + unsigned int start = set->map[i].queue_offset; | |
| 3138 | + unsigned int end = start + set->map[i].nr_queues; | |
| 3139 | + | |
| 3140 | + if (hctx_idx >= start && hctx_idx < end) | |
| 3141 | + break; | |
| 3142 | + } | |
| 3143 | + | |
| 3144 | + if (i >= set->nr_maps) | |
| 3145 | + i = HCTX_TYPE_DEFAULT; | |
| 3146 | + | |
| 3147 | + return i; | |
| 3148 | +} | |
| 3149 | + | |
| 3150 | +static int blk_mq_get_hctx_node(struct blk_mq_tag_set *set, | |
| 3151 | + unsigned int hctx_idx) | |
| 3152 | +{ | |
| 3153 | + enum hctx_type type = hctx_idx_to_type(set, hctx_idx); | |
| 3154 | + | |
| 3155 | + return blk_mq_hw_queue_to_node(&set->map[type], hctx_idx); | |
| 3156 | +} | |
| 3157 | + | |
| 3145 | 3158 | static struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set, |
| 3146 | 3159 | unsigned int hctx_idx, |
| 3147 | 3160 | unsigned int nr_tags, |
| 3148 | 3161 | unsigned int reserved_tags) |
| 3149 | 3162 | { |
| 3163 | + int node = blk_mq_get_hctx_node(set, hctx_idx); | |
| 3150 | 3164 | struct blk_mq_tags *tags; |
| 3151 | − int node; | |
| 3152 | 3165 | |
| 3153 | − node = blk_mq_hw_queue_to_node(&set->map[HCTX_TYPE_DEFAULT], hctx_idx); | |
| 3154 | 3166 | if (node == NUMA_NO_NODE) |
| 3155 | 3167 | node = set->numa_node; |
| 3156 | 3168 | |
| @@ -3199,10 +3211,9 @@ static int blk_mq_alloc_rqs(struct blk_mq_tag_set *set, · 42 unmodified lines | ||
| 3199 | 3211 | unsigned int hctx_idx, unsigned int depth) |
| 3200 | 3212 | { |
| 3201 | 3213 | unsigned int i, j, entries_per_page, max_order = 4; |
| 3214 | + int node = blk_mq_get_hctx_node(set, hctx_idx); | |
| 3202 | 3215 | size_t rq_size, left; |
| 3203 | − int node; | |
| 3204 | 3216 | |
| 3205 | − node = blk_mq_hw_queue_to_node(&set->map[HCTX_TYPE_DEFAULT], hctx_idx); | |
| 3206 | 3217 | if (node == NUMA_NO_NODE) |
| 3207 | 3218 | node = set->numa_node; |
| 3208 | 3219 | |
| @@ -3447,6 +3458,8 @@ static void blk_mq_exit_hctx(struct request_queue *q, · 238 unmodified lines | ||
| 3447 | 3458 | |
| 3448 | 3459 | blk_mq_remove_cpuhp(hctx); |
| 3449 | 3460 | |
| 3461 | + xa_erase(&q->hctx_table, hctx_idx); | |
| 3462 | + | |
| 3450 | 3463 | spin_lock(&q->unused_hctx_lock); |
| 3451 | 3464 | list_add(&hctx->hctx_list, &q->unused_hctx_list); |
| 3452 | 3465 | spin_unlock(&q->unused_hctx_lock); |
| @@ -3456,12 +3469,11 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, · 3 unmodified lines | ||
| 3456 | 3469 | struct blk_mq_tag_set *set, int nr_queue) |
| 3457 | 3470 | { |
| 3458 | 3471 | struct blk_mq_hw_ctx *hctx; |
| 3459 | − unsigned int i; | |
| 3472 | + unsigned long i; | |
| 3460 | 3473 | |
| 3461 | 3474 | queue_for_each_hw_ctx(q, hctx, i) { |
| 3462 | 3475 | if (i == nr_queue) |
| 3463 | 3476 | break; |
| 3464 | − blk_mq_debugfs_unregister_hctx(hctx); | |
| 3465 | 3477 | blk_mq_exit_hctx(q, set, hctx, i); |
| 3466 | 3478 | } |
| 3467 | 3479 | } |
| @@ -3486,8 +3498,15 @@ static int blk_mq_init_hctx(struct request_queue *q, · 18 unmodified lines | ||
| 3486 | 3498 | if (blk_mq_init_request(set, hctx->fq->flush_rq, hctx_idx, |
| 3487 | 3499 | hctx->numa_node)) |
| 3488 | 3500 | goto exit_hctx; |
| 3501 | + | |
| 3502 | + if (xa_insert(&q->hctx_table, hctx_idx, hctx, GFP_KERNEL)) | |
| 3503 | + goto exit_flush_rq; | |
| 3504 | + | |
| 3489 | 3505 | return 0; |
| 3490 | 3506 | |
| 3507 | + exit_flush_rq: | |
| 3508 | + if (set->ops->exit_request) | |
| 3509 | + set->ops->exit_request(set, hctx->fq->flush_rq, hctx_idx); | |
| 3491 | 3510 | exit_hctx: |
| 3492 | 3511 | if (set->ops->exit_hctx) |
| 3493 | 3512 | set->ops->exit_hctx(hctx, hctx_idx); |
| @@ -3647,7 +3666,8 @@ static void __blk_mq_free_map_and_rqs(struct blk_mq_tag_set *set, · 153 unmodified lines | ||
| 3647 | 3666 | |
| 3648 | 3667 | static void blk_mq_map_swqueue(struct request_queue *q) |
| 3649 | 3668 | { |
| 3650 | − unsigned int i, j, hctx_idx; | |
| 3669 | + unsigned int j, hctx_idx; | |
| 3670 | + unsigned long i; | |
| 3651 | 3671 | struct blk_mq_hw_ctx *hctx; |
| 3652 | 3672 | struct blk_mq_ctx *ctx; |
| 3653 | 3673 | struct blk_mq_tag_set *set = q->tag_set; |
| @@ -3754,7 +3774,7 @@ static void blk_mq_map_swqueue(struct request_queue *q) · 100 unmodified lines | ||
| 3754 | 3774 | static void queue_set_hctx_shared(struct request_queue *q, bool shared) |
| 3755 | 3775 | { |
| 3756 | 3776 | struct blk_mq_hw_ctx *hctx; |
| 3757 | − int i; | |
| 3777 | + unsigned long i; | |
| 3758 | 3778 | |
| 3759 | 3779 | queue_for_each_hw_ctx(q, hctx, i) { |
| 3760 | 3780 | if (shared) { |
| @@ -3854,7 +3874,7 @@ static int blk_mq_alloc_ctxs(struct request_queue *q) · 93 unmodified lines | ||
| 3854 | 3874 | void blk_mq_release(struct request_queue *q) |
| 3855 | 3875 | { |
| 3856 | 3876 | struct blk_mq_hw_ctx *hctx, *next; |
| 3857 | − int i; | |
| 3877 | + unsigned long i; | |
| 3858 | 3878 | |
| 3859 | 3879 | queue_for_each_hw_ctx(q, hctx, i) |
| 3860 | 3880 | WARN_ON_ONCE(hctx && list_empty(&hctx->hctx_list)); |
| @@ -3865,7 +3885,7 @@ void blk_mq_release(struct request_queue *q) · 4 unmodified lines | ||
| 3865 | 3885 | kobject_put(&hctx->kobj); |
| 3866 | 3886 | } |
| 3867 | 3887 | |
| 3868 | − kfree(q->queue_hw_ctx); | |
| 3888 | + xa_destroy(&q->hctx_table); | |
| 3869 | 3889 | |
| 3870 | 3890 | /* |
| 3871 | 3891 | * release .mq_kobj and sw queue's kobject now because |
| @@ -3954,52 +3974,28 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx( · 82 unmodified lines | ||
| 3954 | 3974 | static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, |
| 3955 | 3975 | struct request_queue *q) |
| 3956 | 3976 | { |
| 3957 | − int i, j, end; | |
| 3958 | − struct blk_mq_hw_ctx **hctxs = q->queue_hw_ctx; | |
| 3959 | − | |
| 3960 | − if (q->nr_hw_queues < set->nr_hw_queues) { | |
| 3961 | − struct blk_mq_hw_ctx **new_hctxs; | |
| 3962 | − | |
| 3963 | − new_hctxs = kcalloc_node(set->nr_hw_queues, | |
| 3964 | − sizeof(*new_hctxs), GFP_KERNEL, | |
| 3965 | − set->numa_node); | |
| 3966 | − if (!new_hctxs) | |
| 3967 | − return; | |
| 3968 | − if (hctxs) | |
| 3969 | − memcpy(new_hctxs, hctxs, q->nr_hw_queues * | |
| 3970 | − sizeof(*hctxs)); | |
| 3971 | − q->queue_hw_ctx = new_hctxs; | |
| 3972 | − kfree(hctxs); | |
| 3973 | − hctxs = new_hctxs; | |
| 3974 | − } | |
| 3977 | + struct blk_mq_hw_ctx *hctx; | |
| 3978 | + unsigned long i, j; | |
| 3975 | 3979 | |
| 3976 | 3980 | /* protect against switching io scheduler */ |
| 3977 | 3981 | mutex_lock(&q->sysfs_lock); |
| 3978 | 3982 | for (i = 0; i < set->nr_hw_queues; i++) { |
| 3979 | − int node; | |
| 3980 | − struct blk_mq_hw_ctx *hctx; | |
| 3983 | + int old_node; | |
| 3984 | + int node = blk_mq_get_hctx_node(set, i); | |
| 3985 | + struct blk_mq_hw_ctx *old_hctx = xa_load(&q->hctx_table, i); | |
| 3981 | 3986 | |
| 3982 | − node = blk_mq_hw_queue_to_node(&set->map[HCTX_TYPE_DEFAULT], i); | |
| 3983 | − /* | |
| 3984 | − * If the hw queue has been mapped to another numa node, | |
| 3985 | − * we need to realloc the hctx. If allocation fails, fallback | |
| 3986 | − * to use the previous one. | |
| 3987 | − */ | |
| 3988 | − if (hctxs[i] && (hctxs[i]->numa_node == node)) | |
| 3989 | − continue; | |
| 3987 | + if (old_hctx) { | |
| 3988 | + old_node = old_hctx->numa_node; | |
| 3989 | + blk_mq_exit_hctx(q, set, old_hctx, i); | |
| 3990 | + } | |
| 3990 | 3991 | |
| 3991 | − hctx = blk_mq_alloc_and_init_hctx(set, q, i, node); | |
| 3992 | − if (hctx) { | |
| 3993 | − if (hctxs[i]) | |
| 3994 | − blk_mq_exit_hctx(q, set, hctxs[i], i); | |
| 3995 | − hctxs[i] = hctx; | |
| 3996 | − } else { | |
| 3997 | − if (hctxs[i]) | |
| 3998 | − pr_warn("Allocate new hctx on node %d fails,\ | |
| 3999 | − fallback to previous one on node %d\n", | |
| 4000 | − node, hctxs[i]->numa_node); | |
| 4001 | − else | |
| 3992 | + if (!blk_mq_alloc_and_init_hctx(set, q, i, node)) { | |
| 3993 | + if (!old_hctx) | |
| 4002 | 3994 | break; |
| 3995 | + pr_warn("Allocate new hctx on node %d fails, fallback to previous one on node %d\n", | |
| 3996 | + node, old_node); | |
| 3997 | + hctx = blk_mq_alloc_and_init_hctx(set, q, i, old_node); | |
| 3998 | + WARN_ON_ONCE(!hctx); | |
| 4003 | 3999 | } |
| 4004 | 4000 | } |
| 4005 | 4001 | /* |
| @@ -4008,24 +4004,27 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, · 2 unmodified lines | ||
| 4008 | 4004 | */ |
| 4009 | 4005 | if (i != set->nr_hw_queues) { |
| 4010 | 4006 | j = q->nr_hw_queues; |
| 4011 | − end = i; | |
| 4012 | 4007 | } else { |
| 4013 | 4008 | j = i; |
| 4014 | − end = q->nr_hw_queues; | |
| 4015 | 4009 | q->nr_hw_queues = set->nr_hw_queues; |
| 4016 | 4010 | } |
| 4017 | 4011 | |
| 4018 | − for (; j < end; j++) { | |
| 4019 | − struct blk_mq_hw_ctx *hctx = hctxs[j]; | |
| 4020 | − | |
| 4021 | − if (hctx) { | |
| 4022 | − blk_mq_exit_hctx(q, set, hctx, j); | |
| 4023 | − hctxs[j] = NULL; | |
| 4024 | − } | |
| 4025 | − } | |
| 4012 | + xa_for_each_start(&q->hctx_table, j, hctx, j) | |
| 4013 | + blk_mq_exit_hctx(q, set, hctx, j); | |
| 4026 | 4014 | mutex_unlock(&q->sysfs_lock); |
| 4027 | 4015 | } |
| 4028 | 4016 | |
| 4017 | +static void blk_mq_update_poll_flag(struct request_queue *q) | |
| 4018 | +{ | |
| 4019 | + struct blk_mq_tag_set *set = q->tag_set; | |
| 4020 | + | |
| 4021 | + if (set->nr_maps > HCTX_TYPE_POLL && | |
| 4022 | + set->map[HCTX_TYPE_POLL].nr_queues) | |
| 4023 | + blk_queue_flag_set(QUEUE_FLAG_POLL, q); | |
| 4024 | + else | |
| 4025 | + blk_queue_flag_clear(QUEUE_FLAG_POLL, q); | |
| 4026 | +} | |
| 4027 | + | |
| 4029 | 4028 | int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, |
| 4030 | 4029 | struct request_queue *q) |
| 4031 | 4030 | { |
| @@ -4050,6 +4049,8 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, · 18 unmodified lines | ||
| 4050 | 4049 | INIT_LIST_HEAD(&q->unused_hctx_list); |
| 4051 | 4050 | spin_lock_init(&q->unused_hctx_lock); |
| 4052 | 4051 | |
| 4052 | + xa_init(&q->hctx_table); | |
| 4053 | + | |
| 4053 | 4054 | blk_mq_realloc_hw_ctxs(set, q); |
| 4054 | 4055 | if (!q->nr_hw_queues) |
| 4055 | 4056 | goto err_hctxs; |
| @@ -4060,9 +4061,7 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, · 4 unmodified lines | ||
| 4060 | 4061 | q->tag_set = set; |
| 4061 | 4062 | |
| 4062 | 4063 | q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT; |
| 4063 | − if (set->nr_maps > HCTX_TYPE_POLL && | |
| 4064 | − set->map[HCTX_TYPE_POLL].nr_queues) | |
| 4065 | − blk_queue_flag_set(QUEUE_FLAG_POLL, q); | |
| 4064 | + blk_mq_update_poll_flag(q); | |
| 4066 | 4065 | |
| 4067 | 4066 | INIT_DELAYED_WORK(&q->requeue_work, blk_mq_requeue_work); |
| 4068 | 4067 | INIT_LIST_HEAD(&q->requeue_list); |
| @@ -4081,7 +4080,7 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, · 12 unmodified lines | ||
| 4081 | 4080 | return 0; |
| 4082 | 4081 | |
| 4083 | 4082 | err_hctxs: |
| 4084 | − kfree(q->queue_hw_ctx); | |
| 4083 | + xa_destroy(&q->hctx_table); | |
| 4085 | 4084 | q->nr_hw_queues = 0; |
| 4086 | 4085 | blk_mq_sysfs_deinit(q); |
| 4087 | 4086 | err_poll: |
| @@ -4369,7 +4368,8 @@ int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) · 281 unmodified lines | ||
| 4369 | 4368 | { |
| 4370 | 4369 | struct blk_mq_tag_set *set = q->tag_set; |
| 4371 | 4370 | struct blk_mq_hw_ctx *hctx; |
| 4372 | − int i, ret; | |
| 4371 | + int ret; | |
| 4372 | + unsigned long i; | |
| 4373 | 4373 | |
| 4374 | 4374 | if (!set) |
| 4375 | 4375 | return -EINVAL; |
| @@ -4528,6 +4528,7 @@ fallback: · 152 unmodified lines | ||
| 4528 | 4528 | blk_mq_update_queue_map(set); |
| 4529 | 4529 | list_for_each_entry(q, &set->tag_list, tag_set_list) { |
| 4530 | 4530 | blk_mq_realloc_hw_ctxs(set, q); |
| 4531 | + blk_mq_update_poll_flag(q); | |
| 4531 | 4532 | if (q->nr_hw_queues != set->nr_hw_queues) { |
| 4532 | 4533 | int i = prev_nr_hw_queues; |
| 4533 | 4534 | |
| @@ -4744,7 +4745,7 @@ void blk_mq_cancel_work_sync(struct request_queue *q) · 210 unmodified lines | ||
| 4744 | 4745 | { |
| 4745 | 4746 | if (queue_is_mq(q)) { |
| 4746 | 4747 | struct blk_mq_hw_ctx *hctx; |
| 4747 | − int i; | |
| 4748 | + unsigned long i; | |
| 4748 | 4749 | |
| 4749 | 4750 | cancel_delayed_work_sync(&q->requeue_work); |
| 4750 | 4751 | |
| @@ -83,7 +83,7 @@ static inline struct blk_mq_hw_ctx *blk_mq_map_queue_type(struct request_queue * · 82 unmodified lines | ||
| 83 | 83 | enum hctx_type type, |
| 84 | 84 | unsigned int cpu) |
| 85 | 85 | { |
| 86 | − return q->queue_hw_ctx[q->tag_set->map[type].mq_map[cpu]]; | |
| 86 | + return xa_load(&q->hctx_table, q->tag_set->map[type].mq_map[cpu]); | |
| 87 | 87 | } |
| 88 | 88 | |
| 89 | 89 | static inline enum hctx_type blk_mq_get_hctx_type(unsigned int flags) |
| @@ -177,20 +177,20 @@ static inline void rq_qos_requeue(struct request_queue *q, struct request *rq) · 176 unmodified lines | ||
| 177 | 177 | __rq_qos_requeue(q->rq_qos, rq); |
| 178 | 178 | } |
| 179 | 179 | |
| 180 | −static inline void rq_qos_done_bio(struct request_queue *q, struct bio *bio) | |
| 180 | +static inline void rq_qos_done_bio(struct bio *bio) | |
| 181 | 181 | { |
| 182 | − if (q->rq_qos) | |
| 183 | − __rq_qos_done_bio(q->rq_qos, bio); | |
| 182 | + if (bio->bi_bdev && (bio_flagged(bio, BIO_QOS_THROTTLED) || | |
| 183 | + bio_flagged(bio, BIO_QOS_MERGED))) { | |
| 184 | + struct request_queue *q = bdev_get_queue(bio->bi_bdev); | |
| 185 | + if (q->rq_qos) | |
| 186 | + __rq_qos_done_bio(q->rq_qos, bio); | |
| 187 | + } | |
| 184 | 188 | } |
| 185 | 189 | |
| 186 | 190 | static inline void rq_qos_throttle(struct request_queue *q, struct bio *bio) |
| 187 | 191 | { |
| 188 | − /* | |
| 189 | − * BIO_TRACKED lets controllers know that a bio went through the | |
| 190 | − * normal rq_qos path. | |
| 191 | − */ | |
| 192 | 192 | if (q->rq_qos) { |
| 193 | − bio_set_flag(bio, BIO_TRACKED); | |
| 193 | + bio_set_flag(bio, BIO_QOS_THROTTLED); | |
| 194 | 194 | __rq_qos_throttle(q->rq_qos, bio); |
| 195 | 195 | } |
| 196 | 196 | } |
| @@ -205,8 +205,10 @@ static inline void rq_qos_track(struct request_queue *q, struct request *rq, · 8 unmodified lines | ||
| 205 | 205 | static inline void rq_qos_merge(struct request_queue *q, struct request *rq, |
| 206 | 206 | struct bio *bio) |
| 207 | 207 | { |
| 208 | − if (q->rq_qos) | |
| 208 | + if (q->rq_qos) { | |
| 209 | + bio_set_flag(bio, BIO_QOS_MERGED); | |
| 209 | 210 | __rq_qos_merge(q->rq_qos, rq, bio); |
| 211 | + } | |
| 210 | 212 | } |
| 211 | 213 | |
| 212 | 214 | static inline void rq_qos_queue_depth_changed(struct request_queue *q) |
| @@ -10,7 +10,6 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/backing-dev.h> |
| 11 | 11 | #include <linux/blktrace_api.h> |
| 12 | 12 | #include <linux/blk-mq.h> |
| 13 | −#include <linux/blk-cgroup.h> | |
| 14 | 13 | #include <linux/debugfs.h> |
| 15 | 14 | |
| 16 | 15 | #include "blk.h" |
| @@ -18,6 +17,7 @@ · 1 unmodified line | ||
| 18 | 17 | #include "blk-mq-debugfs.h" |
| 19 | 18 | #include "blk-mq-sched.h" |
| 20 | 19 | #include "blk-wbt.h" |
| 20 | +#include "blk-cgroup.h" | |
| 21 | 21 | #include "blk-throttle.h" |
| 22 | 22 | |
| 23 | 23 | struct queue_sysfs_entry { |
| @@ -739,27 +739,6 @@ static void blk_free_queue_rcu(struct rcu_head *rcu_head) · 715 unmodified lines | ||
| 739 | 739 | kmem_cache_free(blk_get_queue_kmem_cache(blk_queue_has_srcu(q)), q); |
| 740 | 740 | } |
| 741 | 741 | |
| 742 | −/* Unconfigure the I/O scheduler and dissociate from the cgroup controller. */ | |
| 743 | −static void blk_exit_queue(struct request_queue *q) | |
| 744 | −{ | |
| 745 | − /* | |
| 746 | − * Since the I/O scheduler exit code may access cgroup information, | |
| 747 | − * perform I/O scheduler exit before disassociating from the block | |
| 748 | − * cgroup controller. | |
| 749 | − */ | |
| 750 | − if (q->elevator) { | |
| 751 | − ioc_clear_queue(q); | |
| 752 | − elevator_exit(q); | |
| 753 | − } | |
| 754 | − | |
| 755 | − /* | |
| 756 | − * Remove all references to @q from the block cgroup controller before | |
| 757 | − * restoring @q->queue_lock to avoid that restoring this pointer causes | |
| 758 | − * e.g. blkcg_print_blkgs() to crash. | |
| 759 | − */ | |
| 760 | − blkcg_exit_queue(q); | |
| 761 | −} | |
| 762 | − | |
| 763 | 742 | /** |
| 764 | 743 | * blk_release_queue - releases all allocated resources of the request_queue |
| 765 | 744 | * @kobj: pointer to a kobject, whose container is a request_queue |
| @@ -787,12 +766,12 @@ static void blk_release_queue(struct kobject *kobj) · 21 unmodified lines | ||
| 787 | 766 | |
| 788 | 767 | might_sleep(); |
| 789 | 768 | |
| 769 | + percpu_ref_exit(&q->q_usage_counter); | |
| 770 | + | |
| 790 | 771 | if (q->poll_stat) |
| 791 | 772 | blk_stat_remove_callback(q, q->poll_cb); |
| 792 | 773 | blk_stat_free_callback(q->poll_cb); |
| 793 | 774 | |
| 794 | − blk_exit_queue(q); | |
| 795 | − | |
| 796 | 775 | blk_free_queue_stats(q->stats); |
| 797 | 776 | kfree(q->poll_stat); |
| 798 | 777 | |
| @@ -880,6 +859,10 @@ int blk_register_queue(struct gendisk *disk) · 81 unmodified lines | ||
| 880 | 859 | goto put_dev; |
| 881 | 860 | } |
| 882 | 861 | |
| 862 | + ret = blk_crypto_sysfs_register(q); | |
| 863 | + if (ret) | |
| 864 | + goto put_dev; | |
| 865 | + | |
| 883 | 866 | blk_queue_flag_set(QUEUE_FLAG_REGISTERED, q); |
| 884 | 867 | wbt_enable_default(q); |
| 885 | 868 | blk_throtl_register_queue(q); |
| @@ -910,6 +893,7 @@ unlock: · 24 unmodified lines | ||
| 910 | 893 | return ret; |
| 911 | 894 | |
| 912 | 895 | put_dev: |
| 896 | + elv_unregister_queue(q); | |
| 913 | 897 | disk_unregister_independent_access_ranges(disk); |
| 914 | 898 | mutex_unlock(&q->sysfs_lock); |
| 915 | 899 | mutex_unlock(&q->sysfs_dir_lock); |
| @@ -954,16 +938,18 @@ void blk_unregister_queue(struct gendisk *disk) · 38 unmodified lines | ||
| 954 | 938 | */ |
| 955 | 939 | if (queue_is_mq(q)) |
| 956 | 940 | blk_mq_unregister_dev(disk_to_dev(disk), q); |
| 957 | − | |
| 958 | − kobject_uevent(&q->kobj, KOBJ_REMOVE); | |
| 959 | − kobject_del(&q->kobj); | |
| 941 | + blk_crypto_sysfs_unregister(q); | |
| 960 | 942 | blk_trace_remove_sysfs(disk_to_dev(disk)); |
| 961 | 943 | |
| 962 | 944 | mutex_lock(&q->sysfs_lock); |
| 963 | − if (q->elevator) | |
| 964 | − elv_unregister_queue(q); | |
| 945 | + elv_unregister_queue(q); | |
| 965 | 946 | disk_unregister_independent_access_ranges(disk); |
| 966 | 947 | mutex_unlock(&q->sysfs_lock); |
| 948 | + | |
| 949 | + /* Now that we've deleted all child objects, we can delete the queue. */ | |
| 950 | + kobject_uevent(&q->kobj, KOBJ_REMOVE); | |
| 951 | + kobject_del(&q->kobj); | |
| 952 | + | |
| 967 | 953 | mutex_unlock(&q->sysfs_dir_lock); |
| 968 | 954 | |
| 969 | 955 | kobject_put(&disk_to_dev(disk)->kobj); |
| @@ -10,7 +10,6 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/blkdev.h> |
| 11 | 11 | #include <linux/bio.h> |
| 12 | 12 | #include <linux/blktrace_api.h> |
| 13 | −#include <linux/blk-cgroup.h> | |
| 14 | 13 | #include "blk.h" |
| 15 | 14 | #include "blk-cgroup-rwstat.h" |
| 16 | 15 | #include "blk-stat.h" |
| @@ -42,11 +41,6 @@ · 25 unmodified lines | ||
| 42 | 41 | /* A workqueue to queue throttle related work */ |
| 43 | 42 | static struct workqueue_struct *kthrotld_workqueue; |
| 44 | 43 | |
| 45 | −enum tg_state_flags { | |
| 46 | − THROTL_TG_PENDING = 1 << 0, /* on parent's pending tree */ | |
| 47 | − THROTL_TG_WAS_EMPTY = 1 << 1, /* bio_lists[] became non-empty */ | |
| 48 | −}; | |
| 49 | − | |
| 50 | 44 | #define rb_entry_tg(node) rb_entry((node), struct throtl_grp, rb_node) |
| 51 | 45 | |
| 52 | 46 | /* We measure latency for request size from <= 4k to >= 1M */ |
| @@ -426,12 +420,24 @@ static void tg_update_has_rules(struct throtl_grp *tg) · 373 unmodified lines | ||
| 426 | 420 | struct throtl_grp *parent_tg = sq_to_tg(tg->service_queue.parent_sq); |
| 427 | 421 | struct throtl_data *td = tg->td; |
| 428 | 422 | int rw; |
| 423 | + int has_iops_limit = 0; | |
| 424 | + | |
| 425 | + for (rw = READ; rw <= WRITE; rw++) { | |
| 426 | + unsigned int iops_limit = tg_iops_limit(tg, rw); | |
| 429 | 427 | |
| 430 | − for (rw = READ; rw <= WRITE; rw++) | |
| 431 | 428 | tg->has_rules[rw] = (parent_tg && parent_tg->has_rules[rw]) || |
| 432 | 429 | (td->limit_valid[td->limit_index] && |
| 433 | 430 | (tg_bps_limit(tg, rw) != U64_MAX || |
| 434 | − tg_iops_limit(tg, rw) != UINT_MAX)); | |
| 431 | + iops_limit != UINT_MAX)); | |
| 432 | + | |
| 433 | + if (iops_limit != UINT_MAX) | |
| 434 | + has_iops_limit = 1; | |
| 435 | + } | |
| 436 | + | |
| 437 | + if (has_iops_limit) | |
| 438 | + tg->flags |= THROTL_TG_HAS_IOPS_LIMIT; | |
| 439 | + else | |
| 440 | + tg->flags &= ~THROTL_TG_HAS_IOPS_LIMIT; | |
| 435 | 441 | } |
| 436 | 442 | |
| 437 | 443 | static void throtl_pd_online(struct blkg_policy_data *pd) |
| @@ -634,8 +640,6 @@ static inline void throtl_start_new_slice_with_credit(struct throtl_grp *tg, · 196 unmodified lines | ||
| 634 | 640 | tg->bytes_disp[rw] = 0; |
| 635 | 641 | tg->io_disp[rw] = 0; |
| 636 | 642 | |
| 637 | − atomic_set(&tg->io_split_cnt[rw], 0); | |
| 638 | − | |
| 639 | 643 | /* |
| 640 | 644 | * Previous slice has expired. We must have trimmed it after last |
| 641 | 645 | * bio dispatch. That means since start of last slice, we never used |
| @@ -659,8 +663,6 @@ static inline void throtl_start_new_slice(struct throtl_grp *tg, bool rw) · 17 unmodified lines | ||
| 659 | 663 | tg->slice_start[rw] = jiffies; |
| 660 | 664 | tg->slice_end[rw] = jiffies + tg->td->throtl_slice; |
| 661 | 665 | |
| 662 | − atomic_set(&tg->io_split_cnt[rw], 0); | |
| 663 | − | |
| 664 | 666 | throtl_log(&tg->service_queue, |
| 665 | 667 | "[%c] new slice start=%lu end=%lu jiffies=%lu", |
| 666 | 668 | rw == READ ? 'R' : 'W', tg->slice_start[rw], |
| @@ -808,7 +810,8 @@ static bool tg_with_in_bps_limit(struct throtl_grp *tg, struct bio *bio, · 141 unmodified lines | ||
| 808 | 810 | unsigned long jiffy_elapsed, jiffy_wait, jiffy_elapsed_rnd; |
| 809 | 811 | unsigned int bio_size = throtl_bio_data_size(bio); |
| 810 | 812 | |
| 811 | − if (bps_limit == U64_MAX) { | |
| 813 | + /* no need to throttle if this bio's bytes have been accounted */ | |
| 814 | + if (bps_limit == U64_MAX || bio_flagged(bio, BIO_THROTTLED)) { | |
| 812 | 815 | if (wait) |
| 813 | 816 | *wait = 0; |
| 814 | 817 | return true; |
| @@ -871,7 +874,8 @@ static bool tg_may_dispatch(struct throtl_grp *tg, struct bio *bio, · 56 unmodified lines | ||
| 871 | 874 | bio != throtl_peek_queued(&tg->service_queue.queued[rw])); |
| 872 | 875 | |
| 873 | 876 | /* If tg->bps = -1, then BW is unlimited */ |
| 874 | − if (bps_limit == U64_MAX && iops_limit == UINT_MAX) { | |
| 877 | + if ((bps_limit == U64_MAX && iops_limit == UINT_MAX) || | |
| 878 | + tg->flags & THROTL_TG_CANCELING) { | |
| 875 | 879 | if (wait) |
| 876 | 880 | *wait = 0; |
| 877 | 881 | return true; |
| @@ -893,9 +897,6 @@ static bool tg_may_dispatch(struct throtl_grp *tg, struct bio *bio, · 15 unmodified lines | ||
| 893 | 897 | jiffies + tg->td->throtl_slice); |
| 894 | 898 | } |
| 895 | 899 | |
| 896 | − if (iops_limit != UINT_MAX) | |
| 897 | − tg->io_disp[rw] += atomic_xchg(&tg->io_split_cnt[rw], 0); | |
| 898 | − | |
| 899 | 900 | if (tg_with_in_bps_limit(tg, bio, bps_limit, &bps_wait) && |
| 900 | 901 | tg_with_in_iops_limit(tg, bio, iops_limit, &iops_wait)) { |
| 901 | 902 | if (wait) |
| @@ -920,9 +921,12 @@ static void throtl_charge_bio(struct throtl_grp *tg, struct bio *bio) · 18 unmodified lines | ||
| 920 | 921 | unsigned int bio_size = throtl_bio_data_size(bio); |
| 921 | 922 | |
| 922 | 923 | /* Charge the bio to the group */ |
| 923 | − tg->bytes_disp[rw] += bio_size; | |
| 924 | + if (!bio_flagged(bio, BIO_THROTTLED)) { | |
| 925 | + tg->bytes_disp[rw] += bio_size; | |
| 926 | + tg->last_bytes_disp[rw] += bio_size; | |
| 927 | + } | |
| 928 | + | |
| 924 | 929 | tg->io_disp[rw]++; |
| 925 | − tg->last_bytes_disp[rw] += bio_size; | |
| 926 | 930 | tg->last_io_disp[rw]++; |
| 927 | 931 | |
| 928 | 932 | /* |
| @@ -1134,12 +1138,22 @@ static void throtl_pending_timer_fn(struct timer_list *t) · 205 unmodified lines | ||
| 1134 | 1138 | struct throtl_service_queue *sq = from_timer(sq, t, pending_timer); |
| 1135 | 1139 | struct throtl_grp *tg = sq_to_tg(sq); |
| 1136 | 1140 | struct throtl_data *td = sq_to_td(sq); |
| 1137 | − struct request_queue *q = td->queue; | |
| 1138 | 1141 | struct throtl_service_queue *parent_sq; |
| 1142 | + struct request_queue *q; | |
| 1139 | 1143 | bool dispatched; |
| 1140 | 1144 | int ret; |
| 1141 | 1145 | |
| 1146 | + /* throtl_data may be gone, so figure out request queue by blkg */ | |
| 1147 | + if (tg) | |
| 1148 | + q = tg->pd.blkg->q; | |
| 1149 | + else | |
| 1150 | + q = td->queue; | |
| 1151 | + | |
| 1142 | 1152 | spin_lock_irq(&q->queue_lock); |
| 1153 | + | |
| 1154 | + if (!q->root_blkg) | |
| 1155 | + goto out_unlock; | |
| 1156 | + | |
| 1143 | 1157 | if (throtl_can_upgrade(td, NULL)) |
| 1144 | 1158 | throtl_upgrade_state(td); |
| 1145 | 1159 | |
| @@ -1219,7 +1233,7 @@ static void blk_throtl_dispatch_work_fn(struct work_struct *work) · 73 unmodified lines | ||
| 1219 | 1233 | if (!bio_list_empty(&bio_list_on_stack)) { |
| 1220 | 1234 | blk_start_plug(&plug); |
| 1221 | 1235 | while ((bio = bio_list_pop(&bio_list_on_stack))) |
| 1222 | − submit_bio_noacct(bio); | |
| 1236 | + submit_bio_noacct_nocheck(bio); | |
| 1223 | 1237 | blk_finish_plug(&plug); |
| 1224 | 1238 | } |
| 1225 | 1239 | } |
| @@ -1763,6 +1777,39 @@ static bool throtl_hierarchy_can_upgrade(struct throtl_grp *tg) · 537 unmodified lines | ||
| 1763 | 1777 | return false; |
| 1764 | 1778 | } |
| 1765 | 1779 | |
| 1780 | +void blk_throtl_cancel_bios(struct request_queue *q) | |
| 1781 | +{ | |
| 1782 | + struct cgroup_subsys_state *pos_css; | |
| 1783 | + struct blkcg_gq *blkg; | |
| 1784 | + | |
| 1785 | + spin_lock_irq(&q->queue_lock); | |
| 1786 | + /* | |
| 1787 | + * queue_lock is held, rcu lock is not needed here technically. | |
| 1788 | + * However, rcu lock is still held to emphasize that following | |
| 1789 | + * path need RCU protection and to prevent warning from lockdep. | |
| 1790 | + */ | |
| 1791 | + rcu_read_lock(); | |
| 1792 | + blkg_for_each_descendant_post(blkg, pos_css, q->root_blkg) { | |
| 1793 | + struct throtl_grp *tg = blkg_to_tg(blkg); | |
| 1794 | + struct throtl_service_queue *sq = &tg->service_queue; | |
| 1795 | + | |
| 1796 | + /* | |
| 1797 | + * Set the flag to make sure throtl_pending_timer_fn() won't | |
| 1798 | + * stop until all throttled bios are dispatched. | |
| 1799 | + */ | |
| 1800 | + blkg_to_tg(blkg)->flags |= THROTL_TG_CANCELING; | |
| 1801 | + /* | |
| 1802 | + * Update disptime after setting the above flag to make sure | |
| 1803 | + * throtl_select_dispatch() won't exit without dispatching. | |
| 1804 | + */ | |
| 1805 | + tg_update_disptime(tg); | |
| 1806 | + | |
| 1807 | + throtl_schedule_pending_timer(sq, jiffies + 1); | |
| 1808 | + } | |
| 1809 | + rcu_read_unlock(); | |
| 1810 | + spin_unlock_irq(&q->queue_lock); | |
| 1811 | +} | |
| 1812 | + | |
| 1766 | 1813 | static bool throtl_can_upgrade(struct throtl_data *td, |
| 1767 | 1814 | struct throtl_grp *this_tg) |
| 1768 | 1815 | { |
| @@ -1917,14 +1964,12 @@ static void throtl_downgrade_check(struct throtl_grp *tg) · 148 unmodified lines | ||
| 1917 | 1964 | } |
| 1918 | 1965 | |
| 1919 | 1966 | if (tg->iops[READ][LIMIT_LOW]) { |
| 1920 | − tg->last_io_disp[READ] += atomic_xchg(&tg->last_io_split_cnt[READ], 0); | |
| 1921 | 1967 | iops = tg->last_io_disp[READ] * HZ / elapsed_time; |
| 1922 | 1968 | if (iops >= tg->iops[READ][LIMIT_LOW]) |
| 1923 | 1969 | tg->last_low_overflow_time[READ] = now; |
| 1924 | 1970 | } |
| 1925 | 1971 | |
| 1926 | 1972 | if (tg->iops[WRITE][LIMIT_LOW]) { |
| 1927 | − tg->last_io_disp[WRITE] += atomic_xchg(&tg->last_io_split_cnt[WRITE], 0); | |
| 1928 | 1973 | iops = tg->last_io_disp[WRITE] * HZ / elapsed_time; |
| 1929 | 1974 | if (iops >= tg->iops[WRITE][LIMIT_LOW]) |
| 1930 | 1975 | tg->last_low_overflow_time[WRITE] = now; |
| @@ -2043,25 +2088,6 @@ static inline void throtl_update_latency_buckets(struct throtl_data *td) · 112 unmodified lines | ||
| 2043 | 2088 | } |
| 2044 | 2089 | #endif |
| 2045 | 2090 | |
| 2046 | −void blk_throtl_charge_bio_split(struct bio *bio) | |
| 2047 | −{ | |
| 2048 | − struct blkcg_gq *blkg = bio->bi_blkg; | |
| 2049 | − struct throtl_grp *parent = blkg_to_tg(blkg); | |
| 2050 | − struct throtl_service_queue *parent_sq; | |
| 2051 | − bool rw = bio_data_dir(bio); | |
| 2052 | − | |
| 2053 | − do { | |
| 2054 | − if (!parent->has_rules[rw]) | |
| 2055 | − break; | |
| 2056 | − | |
| 2057 | − atomic_inc(&parent->io_split_cnt[rw]); | |
| 2058 | − atomic_inc(&parent->last_io_split_cnt[rw]); | |
| 2059 | − | |
| 2060 | − parent_sq = parent->service_queue.parent_sq; | |
| 2061 | − parent = sq_to_tg(parent_sq); | |
| 2062 | − } while (parent); | |
| 2063 | −} | |
| 2064 | − | |
| 2065 | 2091 | bool __blk_throtl_bio(struct bio *bio) |
| 2066 | 2092 | { |
| 2067 | 2093 | struct request_queue *q = bdev_get_queue(bio->bi_bdev); |
| @@ -52,6 +52,13 @@ struct throtl_service_queue { · 51 unmodified lines | ||
| 52 | 52 | struct timer_list pending_timer; /* fires on first_pending_disptime */ |
| 53 | 53 | }; |
| 54 | 54 | |
| 55 | +enum tg_state_flags { | |
| 56 | + THROTL_TG_PENDING = 1 << 0, /* on parent's pending tree */ | |
| 57 | + THROTL_TG_WAS_EMPTY = 1 << 1, /* bio_lists[] became non-empty */ | |
| 58 | + THROTL_TG_HAS_IOPS_LIMIT = 1 << 2, /* tg has iops limit */ | |
| 59 | + THROTL_TG_CANCELING = 1 << 3, /* starts to cancel bio */ | |
| 60 | +}; | |
| 61 | + | |
| 55 | 62 | enum { |
| 56 | 63 | LIMIT_LOW, |
| 57 | 64 | LIMIT_MAX, |
| @@ -132,9 +139,6 @@ struct throtl_grp { · 74 unmodified lines | ||
| 132 | 139 | unsigned int bad_bio_cnt; /* bios exceeding latency threshold */ |
| 133 | 140 | unsigned long bio_cnt_reset_time; |
| 134 | 141 | |
| 135 | − atomic_t io_split_cnt[2]; | |
| 136 | − atomic_t last_io_split_cnt[2]; | |
| 137 | − | |
| 138 | 142 | struct blkg_rwstat stat_bytes; |
| 139 | 143 | struct blkg_rwstat stat_ios; |
| 140 | 144 | }; |
| @@ -158,20 +162,23 @@ static inline struct throtl_grp *blkg_to_tg(struct blkcg_gq *blkg) · 17 unmodified lines | ||
| 158 | 162 | static inline int blk_throtl_init(struct request_queue *q) { return 0; } |
| 159 | 163 | static inline void blk_throtl_exit(struct request_queue *q) { } |
| 160 | 164 | static inline void blk_throtl_register_queue(struct request_queue *q) { } |
| 161 | −static inline void blk_throtl_charge_bio_split(struct bio *bio) { } | |
| 162 | 165 | static inline bool blk_throtl_bio(struct bio *bio) { return false; } |
| 166 | +static inline void blk_throtl_cancel_bios(struct request_queue *q) { } | |
| 163 | 167 | #else /* CONFIG_BLK_DEV_THROTTLING */ |
| 164 | 168 | int blk_throtl_init(struct request_queue *q); |
| 165 | 169 | void blk_throtl_exit(struct request_queue *q); |
| 166 | 170 | void blk_throtl_register_queue(struct request_queue *q); |
| 167 | −void blk_throtl_charge_bio_split(struct bio *bio); | |
| 168 | 171 | bool __blk_throtl_bio(struct bio *bio); |
| 172 | +void blk_throtl_cancel_bios(struct request_queue *q); | |
| 169 | 173 | static inline bool blk_throtl_bio(struct bio *bio) |
| 170 | 174 | { |
| 171 | 175 | struct throtl_grp *tg = blkg_to_tg(bio->bi_blkg); |
| 172 | 176 | |
| 173 | − if (bio_flagged(bio, BIO_THROTTLED)) | |
| 177 | + /* no need to throttle bps any more if the bio has been throttled */ | |
| 178 | + if (bio_flagged(bio, BIO_THROTTLED) && | |
| 179 | + !(tg->flags & THROTL_TG_HAS_IOPS_LIMIT)) | |
| 174 | 180 | return false; |
| 181 | + | |
| 175 | 182 | if (!tg->has_rules[bio_data_dir(bio)]) |
| 176 | 183 | return false; |
| 177 | 184 | |
| @@ -215,9 +215,8 @@ static int blkdev_zone_reset_all_emulated(struct block_device *bdev, · 214 unmodified lines | ||
| 215 | 215 | continue; |
| 216 | 216 | } |
| 217 | 217 | |
| 218 | − bio = blk_next_bio(bio, 0, gfp_mask); | |
| 219 | − bio_set_dev(bio, bdev); | |
| 220 | − bio->bi_opf = REQ_OP_ZONE_RESET | REQ_SYNC; | |
| 218 | + bio = blk_next_bio(bio, bdev, 0, REQ_OP_ZONE_RESET | REQ_SYNC, | |
| 219 | + gfp_mask); | |
| 221 | 220 | bio->bi_iter.bi_sector = sector; |
| 222 | 221 | sector += zone_sectors; |
| 223 | 222 | |
| @@ -239,10 +238,7 @@ static int blkdev_zone_reset_all(struct block_device *bdev, gfp_t gfp_mask) · 15 unmodified lines | ||
| 239 | 238 | { |
| 240 | 239 | struct bio bio; |
| 241 | 240 | |
| 242 | − bio_init(&bio, NULL, 0); | |
| 243 | − bio_set_dev(&bio, bdev); | |
| 244 | − bio.bi_opf = REQ_OP_ZONE_RESET_ALL | REQ_SYNC; | |
| 245 | − | |
| 241 | + bio_init(&bio, bdev, NULL, 0, REQ_OP_ZONE_RESET_ALL | REQ_SYNC); | |
| 246 | 242 | return submit_bio_wait(&bio); |
| 247 | 243 | } |
| 248 | 244 | |
| @@ -306,9 +302,7 @@ int blkdev_zone_mgmt(struct block_device *bdev, enum req_opf op, · 57 unmodified lines | ||
| 306 | 302 | } |
| 307 | 303 | |
| 308 | 304 | while (sector < end_sector) { |
| 309 | − bio = blk_next_bio(bio, 0, gfp_mask); | |
| 310 | − bio_set_dev(bio, bdev); | |
| 311 | − bio->bi_opf = op | REQ_SYNC; | |
| 305 | + bio = blk_next_bio(bio, bdev, 0, op | REQ_SYNC, gfp_mask); | |
| 312 | 306 | bio->bi_iter.bi_sector = sector; |
| 313 | 307 | sector += zone_sectors; |
| 314 | 308 | |
| @@ -46,7 +46,7 @@ void blk_freeze_queue(struct request_queue *q); · 45 unmodified lines | ||
| 46 | 46 | void __blk_mq_unfreeze_queue(struct request_queue *q, bool force_atomic); |
| 47 | 47 | void blk_queue_start_drain(struct request_queue *q); |
| 48 | 48 | int __bio_queue_enter(struct request_queue *q, struct bio *bio); |
| 49 | −bool submit_bio_checks(struct bio *bio); | |
| 49 | +void submit_bio_noacct_nocheck(struct bio *bio); | |
| 50 | 50 | |
| 51 | 51 | static inline bool blk_try_enter_queue(struct request_queue *q, bool pm) |
| 52 | 52 | { |
| @@ -325,7 +325,7 @@ int blk_dev_init(void); · 272 unmodified lines | ||
| 325 | 325 | */ |
| 326 | 326 | static inline bool blk_do_io_stat(struct request *rq) |
| 327 | 327 | { |
| 328 | − return (rq->rq_flags & RQF_IO_STAT) && rq->q->disk; | |
| 328 | + return (rq->rq_flags & RQF_IO_STAT) && !blk_rq_is_passthrough(rq); | |
| 329 | 329 | } |
| 330 | 330 | |
| 331 | 331 | void update_io_ticks(struct block_device *part, unsigned long now, bool end); |
| @@ -406,8 +406,6 @@ extern int blk_iolatency_init(struct request_queue *q); · 74 unmodified lines | ||
| 406 | 406 | static inline int blk_iolatency_init(struct request_queue *q) { return 0; } |
| 407 | 407 | #endif |
| 408 | 408 | |
| 409 | −struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp); | |
| 410 | − | |
| 411 | 409 | #ifdef CONFIG_BLK_DEV_ZONED |
| 412 | 410 | void blk_queue_free_zone_bitmaps(struct request_queue *q); |
| 413 | 411 | void blk_queue_clear_zone_settings(struct request_queue *q); |
| @@ -426,6 +424,7 @@ int bdev_add_partition(struct gendisk *disk, int partno, sector_t start, · 12 unmodified lines | ||
| 426 | 424 | int bdev_del_partition(struct gendisk *disk, int partno); |
| 427 | 425 | int bdev_resize_partition(struct gendisk *disk, int partno, sector_t start, |
| 428 | 426 | sector_t length); |
| 427 | +void blk_drop_partitions(struct gendisk *disk); | |
| 429 | 428 | |
| 430 | 429 | int bio_add_hw_page(struct request_queue *q, struct bio *bio, |
| 431 | 430 | struct page *page, unsigned int len, unsigned int offset, |
| @@ -445,6 +444,9 @@ int disk_alloc_events(struct gendisk *disk); · 13 unmodified lines | ||
| 445 | 444 | void disk_add_events(struct gendisk *disk); |
| 446 | 445 | void disk_del_events(struct gendisk *disk); |
| 447 | 446 | void disk_release_events(struct gendisk *disk); |
| 447 | +void disk_block_events(struct gendisk *disk); | |
| 448 | +void disk_unblock_events(struct gendisk *disk); | |
| 449 | +void disk_flush_events(struct gendisk *disk, unsigned int mask); | |
| 448 | 450 | extern struct device_attribute dev_attr_events; |
| 449 | 451 | extern struct device_attribute dev_attr_events_async; |
| 450 | 452 | extern struct device_attribute dev_attr_events_poll_msecs; |
| @@ -14,7 +14,6 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <linux/pagemap.h> |
| 15 | 15 | #include <linux/mempool.h> |
| 16 | 16 | #include <linux/blkdev.h> |
| 17 | −#include <linux/blk-cgroup.h> | |
| 18 | 17 | #include <linux/backing-dev.h> |
| 19 | 18 | #include <linux/init.h> |
| 20 | 19 | #include <linux/hash.h> |
| @@ -24,6 +23,7 @@ · 3 unmodified lines | ||
| 24 | 23 | |
| 25 | 24 | #include <trace/events/block.h> |
| 26 | 25 | #include "blk.h" |
| 26 | +#include "blk-cgroup.h" | |
| 27 | 27 | |
| 28 | 28 | #define POOL_SIZE 64 |
| 29 | 29 | #define ISA_POOL_SIZE 16 |
| @@ -162,15 +162,12 @@ static struct bio *bounce_clone_bio(struct bio *bio_src) · 132 unmodified lines | ||
| 162 | 162 | * that does not own the bio - reason being drivers don't use it for |
| 163 | 163 | * iterating over the biovec anymore, so expecting it to be kept up |
| 164 | 164 | * to date (i.e. for clones that share the parent biovec) is just |
| 165 | − * asking for trouble and would force extra work on | |
| 166 | − * __bio_clone_fast() anyways. | |
| 165 | + * asking for trouble and would force extra work. | |
| 167 | 166 | */ |
| 168 | − bio = bio_alloc_bioset(GFP_NOIO, bio_segments(bio_src), | |
| 169 | − &bounce_bio_set); | |
| 170 | − bio->bi_bdev = bio_src->bi_bdev; | |
| 167 | + bio = bio_alloc_bioset(bio_src->bi_bdev, bio_segments(bio_src), | |
| 168 | + bio_src->bi_opf, GFP_NOIO, &bounce_bio_set); | |
| 171 | 169 | if (bio_flagged(bio_src, BIO_REMAPPED)) |
| 172 | 170 | bio_set_flag(bio, BIO_REMAPPED); |
| 173 | − bio->bi_opf = bio_src->bi_opf; | |
| 174 | 171 | bio->bi_ioprio = bio_src->bi_ioprio; |
| 175 | 172 | bio->bi_write_hint = bio_src->bi_write_hint; |
| 176 | 173 | bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | */ |
| 5 | 5 | #include <linux/export.h> |
| 6 | 6 | #include <linux/moduleparam.h> |
| 7 | −#include <linux/genhd.h> | |
| 7 | +#include <linux/blkdev.h> | |
| 8 | 8 | #include "blk.h" |
| 9 | 9 | |
| 10 | 10 | struct disk_events { |
| @@ -35,7 +35,6 @@ · 34 unmodified lines | ||
| 35 | 35 | #include <linux/hash.h> |
| 36 | 36 | #include <linux/uaccess.h> |
| 37 | 37 | #include <linux/pm_runtime.h> |
| 38 | −#include <linux/blk-cgroup.h> | |
| 39 | 38 | |
| 40 | 39 | #include <trace/events/block.h> |
| 41 | 40 | |
| @@ -44,6 +43,7 @@ · 2 unmodified lines | ||
| 44 | 43 | #include "blk-mq-sched.h" |
| 45 | 44 | #include "blk-pm.h" |
| 46 | 45 | #include "blk-wbt.h" |
| 46 | +#include "blk-cgroup.h" | |
| 47 | 47 | |
| 48 | 48 | static DEFINE_SPINLOCK(elv_list_lock); |
| 49 | 49 | static LIST_HEAD(elv_list); |
| @@ -192,6 +192,9 @@ void elevator_exit(struct request_queue *q) · 142 unmodified lines | ||
| 192 | 192 | { |
| 193 | 193 | struct elevator_queue *e = q->elevator; |
| 194 | 194 | |
| 195 | + ioc_clear_queue(q); | |
| 196 | + blk_mq_sched_free_rqs(q); | |
| 197 | + | |
| 195 | 198 | mutex_lock(&e->sysfs_lock); |
| 196 | 199 | blk_mq_exit_sched(q, e); |
| 197 | 200 | mutex_unlock(&e->sysfs_lock); |
| @@ -516,9 +519,11 @@ int elv_register_queue(struct request_queue *q, bool uevent) · 318 unmodified lines | ||
| 516 | 519 | |
| 517 | 520 | void elv_unregister_queue(struct request_queue *q) |
| 518 | 521 | { |
| 522 | + struct elevator_queue *e = q->elevator; | |
| 523 | + | |
| 519 | 524 | lockdep_assert_held(&q->sysfs_lock); |
| 520 | 525 | |
| 521 | − if (q) { | |
| 526 | + if (e && e->registered) { | |
| 522 | 527 | struct elevator_queue *e = q->elevator; |
| 523 | 528 | |
| 524 | 529 | kobject_uevent(&e->kobj, KOBJ_REMOVE); |
| @@ -591,11 +596,7 @@ int elevator_switch_mq(struct request_queue *q, · 66 unmodified lines | ||
| 591 | 596 | lockdep_assert_held(&q->sysfs_lock); |
| 592 | 597 | |
| 593 | 598 | if (q->elevator) { |
| 594 | − if (q->elevator->registered) | |
| 595 | − elv_unregister_queue(q); | |
| 596 | − | |
| 597 | − ioc_clear_queue(q); | |
| 598 | − blk_mq_sched_free_rqs(q); | |
| 599 | + elv_unregister_queue(q); | |
| 599 | 600 | elevator_exit(q); |
| 600 | 601 | } |
| 601 | 602 | |
| @@ -606,7 +607,6 @@ int elevator_switch_mq(struct request_queue *q, · 4 unmodified lines | ||
| 606 | 607 | if (new_e) { |
| 607 | 608 | ret = elv_register_queue(q, true); |
| 608 | 609 | if (ret) { |
| 609 | − blk_mq_sched_free_rqs(q); | |
| 610 | 610 | elevator_exit(q); |
| 611 | 611 | goto out; |
| 612 | 612 | } |
| @@ -75,8 +75,13 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, · 74 unmodified lines | ||
| 75 | 75 | return -ENOMEM; |
| 76 | 76 | } |
| 77 | 77 | |
| 78 | − bio_init(&bio, vecs, nr_pages); | |
| 79 | − bio_set_dev(&bio, bdev); | |
| 78 | + if (iov_iter_rw(iter) == READ) { | |
| 79 | + bio_init(&bio, bdev, vecs, nr_pages, REQ_OP_READ); | |
| 80 | + if (iter_is_iovec(iter)) | |
| 81 | + should_dirty = true; | |
| 82 | + } else { | |
| 83 | + bio_init(&bio, bdev, vecs, nr_pages, dio_bio_write_op(iocb)); | |
| 84 | + } | |
| 80 | 85 | bio.bi_iter.bi_sector = pos >> SECTOR_SHIFT; |
| 81 | 86 | bio.bi_write_hint = iocb->ki_hint; |
| 82 | 87 | bio.bi_private = current; |
| @@ -88,14 +93,9 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, · 5 unmodified lines | ||
| 88 | 93 | goto out; |
| 89 | 94 | ret = bio.bi_iter.bi_size; |
| 90 | 95 | |
| 91 | − if (iov_iter_rw(iter) == READ) { | |
| 92 | − bio.bi_opf = REQ_OP_READ; | |
| 93 | − if (iter_is_iovec(iter)) | |
| 94 | − should_dirty = true; | |
| 95 | − } else { | |
| 96 | − bio.bi_opf = dio_bio_write_op(iocb); | |
| 96 | + if (iov_iter_rw(iter) == WRITE) | |
| 97 | 97 | task_io_account_write(ret); |
| 98 | − } | |
| 98 | + | |
| 99 | 99 | if (iocb->ki_flags & IOCB_NOWAIT) |
| 100 | 100 | bio.bi_opf |= REQ_NOWAIT; |
| 101 | 101 | if (iocb->ki_flags & IOCB_HIPRI) |
| @@ -190,6 +190,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, · 88 unmodified lines | ||
| 190 | 190 | struct blkdev_dio *dio; |
| 191 | 191 | struct bio *bio; |
| 192 | 192 | bool is_read = (iov_iter_rw(iter) == READ), is_sync; |
| 193 | + unsigned int opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb); | |
| 193 | 194 | loff_t pos = iocb->ki_pos; |
| 194 | 195 | int ret = 0; |
| 195 | 196 | |
| @@ -197,7 +198,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, · 1 unmodified line | ||
| 197 | 198 | (bdev_logical_block_size(bdev) - 1)) |
| 198 | 199 | return -EINVAL; |
| 199 | 200 | |
| 200 | − bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool); | |
| 201 | + bio = bio_alloc_kiocb(iocb, bdev, nr_pages, opf, &blkdev_dio_pool); | |
| 201 | 202 | |
| 202 | 203 | dio = container_of(bio, struct blkdev_dio, bio); |
| 203 | 204 | atomic_set(&dio->ref, 1); |
| @@ -223,7 +224,6 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, · 19 unmodified lines | ||
| 223 | 224 | blk_start_plug(&plug); |
| 224 | 225 | |
| 225 | 226 | for (;;) { |
| 226 | − bio_set_dev(bio, bdev); | |
| 227 | 227 | bio->bi_iter.bi_sector = pos >> SECTOR_SHIFT; |
| 228 | 228 | bio->bi_write_hint = iocb->ki_hint; |
| 229 | 229 | bio->bi_private = dio; |
| @@ -238,11 +238,9 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, · 8 unmodified lines | ||
| 238 | 238 | } |
| 239 | 239 | |
| 240 | 240 | if (is_read) { |
| 241 | − bio->bi_opf = REQ_OP_READ; | |
| 242 | 241 | if (dio->flags & DIO_SHOULD_DIRTY) |
| 243 | 242 | bio_set_pages_dirty(bio); |
| 244 | 243 | } else { |
| 245 | − bio->bi_opf = dio_bio_write_op(iocb); | |
| 246 | 244 | task_io_account_write(bio->bi_iter.bi_size); |
| 247 | 245 | } |
| 248 | 246 | if (iocb->ki_flags & IOCB_NOWAIT) |
| @@ -258,7 +256,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, · 9 unmodified lines | ||
| 258 | 256 | } |
| 259 | 257 | atomic_inc(&dio->ref); |
| 260 | 258 | submit_bio(bio); |
| 261 | − bio = bio_alloc(GFP_KERNEL, nr_pages); | |
| 259 | + bio = bio_alloc(bdev, nr_pages, opf, GFP_KERNEL); | |
| 262 | 260 | } |
| 263 | 261 | |
| 264 | 262 | blk_finish_plug(&plug); |
| @@ -313,6 +311,8 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, · 48 unmodified lines | ||
| 313 | 311 | unsigned int nr_pages) |
| 314 | 312 | { |
| 315 | 313 | struct block_device *bdev = iocb->ki_filp->private_data; |
| 314 | + bool is_read = iov_iter_rw(iter) == READ; | |
| 315 | + unsigned int opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb); | |
| 316 | 316 | struct blkdev_dio *dio; |
| 317 | 317 | struct bio *bio; |
| 318 | 318 | loff_t pos = iocb->ki_pos; |
| @@ -322,11 +322,10 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, · 3 unmodified lines | ||
| 322 | 322 | (bdev_logical_block_size(bdev) - 1)) |
| 323 | 323 | return -EINVAL; |
| 324 | 324 | |
| 325 | − bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool); | |
| 325 | + bio = bio_alloc_kiocb(iocb, bdev, nr_pages, opf, &blkdev_dio_pool); | |
| 326 | 326 | dio = container_of(bio, struct blkdev_dio, bio); |
| 327 | 327 | dio->flags = 0; |
| 328 | 328 | dio->iocb = iocb; |
| 329 | − bio_set_dev(bio, bdev); | |
| 330 | 329 | bio->bi_iter.bi_sector = pos >> SECTOR_SHIFT; |
| 331 | 330 | bio->bi_write_hint = iocb->ki_hint; |
| 332 | 331 | bio->bi_end_io = blkdev_bio_end_io_async; |
| @@ -349,14 +348,12 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, · 16 unmodified lines | ||
| 349 | 348 | } |
| 350 | 349 | dio->size = bio->bi_iter.bi_size; |
| 351 | 350 | |
| 352 | − if (iov_iter_rw(iter) == READ) { | |
| 353 | − bio->bi_opf = REQ_OP_READ; | |
| 351 | + if (is_read) { | |
| 354 | 352 | if (iter_is_iovec(iter)) { |
| 355 | 353 | dio->flags |= DIO_SHOULD_DIRTY; |
| 356 | 354 | bio_set_pages_dirty(bio); |
| 357 | 355 | } |
| 358 | 356 | } else { |
| 359 | − bio->bi_opf = dio_bio_write_op(iocb); | |
| 360 | 357 | task_io_account_write(bio->bi_iter.bi_size); |
| 361 | 358 | } |
| 362 | 359 | |
| @@ -8,7 +8,6 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/module.h> |
| 9 | 9 | #include <linux/ctype.h> |
| 10 | 10 | #include <linux/fs.h> |
| 11 | −#include <linux/genhd.h> | |
| 12 | 11 | #include <linux/kdev_t.h> |
| 13 | 12 | #include <linux/kernel.h> |
| 14 | 13 | #include <linux/blkdev.h> |
| @@ -26,10 +25,12 @@ · 11 unmodified lines | ||
| 26 | 25 | #include <linux/pm_runtime.h> |
| 27 | 26 | #include <linux/badblocks.h> |
| 28 | 27 | #include <linux/part_stat.h> |
| 28 | +#include "blk-throttle.h" | |
| 29 | 29 | |
| 30 | 30 | #include "blk.h" |
| 31 | 31 | #include "blk-mq-sched.h" |
| 32 | 32 | #include "blk-rq-qos.h" |
| 33 | +#include "blk-cgroup.h" | |
| 33 | 34 | |
| 34 | 35 | static struct kobject *block_depr; |
| 35 | 36 | |
| @@ -185,7 +186,9 @@ static struct blk_major_name { · 149 unmodified lines | ||
| 185 | 186 | struct blk_major_name *next; |
| 186 | 187 | int major; |
| 187 | 188 | char name[16]; |
| 189 | +#ifdef CONFIG_BLOCK_LEGACY_AUTOLOAD | |
| 188 | 190 | void (*probe)(dev_t devt); |
| 191 | +#endif | |
| 189 | 192 | } *major_names[BLKDEV_MAJOR_HASH_SIZE]; |
| 190 | 193 | static DEFINE_MUTEX(major_names_lock); |
| 191 | 194 | static DEFINE_SPINLOCK(major_names_spinlock); |
| @@ -275,7 +278,9 @@ int __register_blkdev(unsigned int major, const char *name, · 83 unmodified lines | ||
| 275 | 278 | } |
| 276 | 279 | |
| 277 | 280 | p->major = major; |
| 281 | +#ifdef CONFIG_BLOCK_LEGACY_AUTOLOAD | |
| 278 | 282 | p->probe = probe; |
| 283 | +#endif | |
| 279 | 284 | strlcpy(p->name, name, sizeof(p->name)); |
| 280 | 285 | p->next = NULL; |
| 281 | 286 | index = major_to_index(major); |
| @@ -523,6 +528,7 @@ int __must_check device_add_disk(struct device *parent, struct gendisk *disk, · 241 unmodified lines | ||
| 523 | 528 | |
| 524 | 529 | disk_update_readahead(disk); |
| 525 | 530 | disk_add_events(disk); |
| 531 | + set_bit(GD_ADDED, &disk->state); | |
| 526 | 532 | return 0; |
| 527 | 533 | |
| 528 | 534 | out_unregister_bdi: |
| @@ -636,7 +642,8 @@ void del_gendisk(struct gendisk *disk) · 107 unmodified lines | ||
| 636 | 642 | |
| 637 | 643 | blk_mq_freeze_queue_wait(q); |
| 638 | 644 | |
| 639 | − rq_qos_exit(q); | |
| 645 | + blk_throtl_cancel_bios(disk->queue); | |
| 646 | + | |
| 640 | 647 | blk_sync_queue(q); |
| 641 | 648 | blk_flush_integrity(); |
| 642 | 649 | /* |
| @@ -693,6 +700,7 @@ static ssize_t disk_badblocks_store(struct device *dev, · 50 unmodified lines | ||
| 693 | 700 | return badblocks_store(disk->bb, page, len, 0); |
| 694 | 701 | } |
| 695 | 702 | |
| 703 | +#ifdef CONFIG_BLOCK_LEGACY_AUTOLOAD | |
| 696 | 704 | void blk_request_module(dev_t devt) |
| 697 | 705 | { |
| 698 | 706 | unsigned int major = MAJOR(devt); |
| @@ -712,6 +720,7 @@ void blk_request_module(dev_t devt) · 13 unmodified lines | ||
| 712 | 720 | /* Make old-style 2.4 aliases work */ |
| 713 | 721 | request_module("block-major-%d", MAJOR(devt)); |
| 714 | 722 | } |
| 723 | +#endif /* CONFIG_BLOCK_LEGACY_AUTOLOAD */ | |
| 715 | 724 | |
| 716 | 725 | /* |
| 717 | 726 | * print a full list of all partitions - intended for places where the root |
| @@ -927,12 +936,17 @@ ssize_t part_stat_show(struct device *dev, · 209 unmodified lines | ||
| 927 | 936 | struct disk_stats stat; |
| 928 | 937 | unsigned int inflight; |
| 929 | 938 | |
| 930 | − part_stat_read_all(bdev, &stat); | |
| 931 | 939 | if (queue_is_mq(q)) |
| 932 | 940 | inflight = blk_mq_in_flight(q, bdev); |
| 933 | 941 | else |
| 934 | 942 | inflight = part_in_flight(bdev); |
| 935 | 943 | |
| 944 | + if (inflight) { | |
| 945 | + part_stat_lock(); | |
| 946 | + update_io_ticks(bdev, jiffies, true); | |
| 947 | + part_stat_unlock(); | |
| 948 | + } | |
| 949 | + part_stat_read_all(bdev, &stat); | |
| 936 | 950 | return sprintf(buf, |
| 937 | 951 | "%8lu %8lu %8llu %8u " |
| 938 | 952 | "%8lu %8lu %8llu %8u " |
| @@ -1100,6 +1114,31 @@ static const struct attribute_group *disk_attr_groups[] = { · 161 unmodified lines | ||
| 1100 | 1114 | NULL |
| 1101 | 1115 | }; |
| 1102 | 1116 | |
| 1117 | +static void disk_release_mq(struct request_queue *q) | |
| 1118 | +{ | |
| 1119 | + blk_mq_cancel_work_sync(q); | |
| 1120 | + | |
| 1121 | + /* | |
| 1122 | + * There can't be any non non-passthrough bios in flight here, but | |
| 1123 | + * requests stay around longer, including passthrough ones so we | |
| 1124 | + * still need to freeze the queue here. | |
| 1125 | + */ | |
| 1126 | + blk_mq_freeze_queue(q); | |
| 1127 | + | |
| 1128 | + /* | |
| 1129 | + * Since the I/O scheduler exit code may access cgroup information, | |
| 1130 | + * perform I/O scheduler exit before disassociating from the block | |
| 1131 | + * cgroup controller. | |
| 1132 | + */ | |
| 1133 | + if (q->elevator) { | |
| 1134 | + mutex_lock(&q->sysfs_lock); | |
| 1135 | + elevator_exit(q); | |
| 1136 | + mutex_unlock(&q->sysfs_lock); | |
| 1137 | + } | |
| 1138 | + rq_qos_exit(q); | |
| 1139 | + __blk_mq_unfreeze_queue(q, true); | |
| 1140 | +} | |
| 1141 | + | |
| 1103 | 1142 | /** |
| 1104 | 1143 | * disk_release - releases all allocated resources of the gendisk |
| 1105 | 1144 | * @dev: the device representing this disk |
| @@ -1121,13 +1160,21 @@ static void disk_release(struct device *dev) · 15 unmodified lines | ||
| 1121 | 1160 | might_sleep(); |
| 1122 | 1161 | WARN_ON_ONCE(disk_live(disk)); |
| 1123 | 1162 | |
| 1124 | − blk_mq_cancel_work_sync(disk->queue); | |
| 1163 | + if (queue_is_mq(disk->queue)) | |
| 1164 | + disk_release_mq(disk->queue); | |
| 1165 | + | |
| 1166 | + blkcg_exit_queue(disk->queue); | |
| 1125 | 1167 | |
| 1126 | 1168 | disk_release_events(disk); |
| 1127 | 1169 | kfree(disk->random); |
| 1128 | 1170 | xa_destroy(&disk->part_tbl); |
| 1171 | + | |
| 1129 | 1172 | disk->queue->disk = NULL; |
| 1130 | 1173 | blk_put_queue(disk->queue); |
| 1174 | + | |
| 1175 | + if (test_bit(GD_ADDED, &disk->state) && disk->fops->free_disk) | |
| 1176 | + disk->fops->free_disk(disk); | |
| 1177 | + | |
| 1131 | 1178 | iput(disk->part0->bd_inode); /* frees the disk */ |
| 1132 | 1179 | } |
| 1133 | 1180 | |
| @@ -1188,12 +1235,17 @@ static int diskstats_show(struct seq_file *seqf, void *v) · 54 unmodified lines | ||
| 1188 | 1235 | xa_for_each(&gp->part_tbl, idx, hd) { |
| 1189 | 1236 | if (bdev_is_partition(hd) && !bdev_nr_sectors(hd)) |
| 1190 | 1237 | continue; |
| 1191 | − part_stat_read_all(hd, &stat); | |
| 1192 | 1238 | if (queue_is_mq(gp->queue)) |
| 1193 | 1239 | inflight = blk_mq_in_flight(gp->queue, hd); |
| 1194 | 1240 | else |
| 1195 | 1241 | inflight = part_in_flight(hd); |
| 1196 | 1242 | |
| 1243 | + if (inflight) { | |
| 1244 | + part_stat_lock(); | |
| 1245 | + update_io_ticks(hd, jiffies, true); | |
| 1246 | + part_stat_unlock(); | |
| 1247 | + } | |
| 1248 | + part_stat_read_all(hd, &stat); | |
| 1197 | 1249 | seq_printf(seqf, "%4d %7d %pg " |
| 1198 | 1250 | "%lu %lu %lu %u " |
| 1199 | 1251 | "%lu %lu %lu %u " |
| @@ -1322,6 +1374,9 @@ struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id, · 122 unmodified lines | ||
| 1322 | 1374 | if (xa_insert(&disk->part_tbl, 0, disk->part0, GFP_KERNEL)) |
| 1323 | 1375 | goto out_destroy_part_tbl; |
| 1324 | 1376 | |
| 1377 | + if (blkcg_init_queue(q)) | |
| 1378 | + goto out_erase_part0; | |
| 1379 | + | |
| 1325 | 1380 | rand_initialize_disk(disk); |
| 1326 | 1381 | disk_to_dev(disk)->class = &block_class; |
| 1327 | 1382 | disk_to_dev(disk)->type = &disk_type; |
| @@ -1334,6 +1389,8 @@ struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id, · 6 unmodified lines | ||
| 1334 | 1389 | #endif |
| 1335 | 1390 | return disk; |
| 1336 | 1391 | |
| 1392 | +out_erase_part0: | |
| 1393 | + xa_erase(&disk->part_tbl, 0); | |
| 1337 | 1394 | out_destroy_part_tbl: |
| 1338 | 1395 | xa_destroy(&disk->part_tbl); |
| 1339 | 1396 | disk->part0->bd_disk = NULL; |
| @@ -1,5 +1,5 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | −#include <linux/genhd.h> | |
| 2 | +#include <linux/blkdev.h> | |
| 3 | 3 | #include <linux/slab.h> |
| 4 | 4 | |
| 5 | 5 | struct bd_holder_disk { |
| @@ -1,7 +1,6 @@ | ||
| 1 | 1 | /* SPDX-License-Identifier: GPL-2.0 */ |
| 2 | 2 | #include <linux/pagemap.h> |
| 3 | 3 | #include <linux/blkdev.h> |
| 4 | −#include <linux/genhd.h> | |
| 5 | 4 | #include "../blk.h" |
| 6 | 5 | |
| 7 | 6 | /* |
| @@ -8,7 +8,6 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/major.h> |
| 9 | 9 | #include <linux/slab.h> |
| 10 | 10 | #include <linux/ctype.h> |
| 11 | −#include <linux/genhd.h> | |
| 12 | 11 | #include <linux/vmalloc.h> |
| 13 | 12 | #include <linux/blktrace_api.h> |
| 14 | 13 | #include <linux/raid/detect.h> |
| @@ -13,7 +13,6 @@ · 12 unmodified lines | ||
| 13 | 13 | |
| 14 | 14 | #include <linux/types.h> |
| 15 | 15 | #include <linux/fs.h> |
| 16 | −#include <linux/genhd.h> | |
| 17 | 16 | #include <linux/kernel.h> |
| 18 | 17 | #include <linux/major.h> |
| 19 | 18 | #include <linux/string.h> |
| @@ -14,7 +14,6 @@ · 13 unmodified lines | ||
| 14 | 14 | |
| 15 | 15 | #include <linux/types.h> |
| 16 | 16 | #include <linux/list.h> |
| 17 | −#include <linux/genhd.h> | |
| 18 | 17 | #include <linux/fs.h> |
| 19 | 18 | #include <asm/unaligned.h> |
| 20 | 19 | #include <asm/byteorder.h> |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/device.h> |
| 14 | 14 | #include <linux/kernel.h> |
| 15 | 15 | #include <linux/list.h> |
| 16 | −#include <linux/genhd.h> | |
| 16 | +#include <linux/blkdev.h> | |
| 17 | 17 | #include <linux/slab.h> |
| 18 | 18 | #include <linux/uaccess.h> |
| 19 | 19 | #include <uapi/linux/sed-opal.h> |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <linux/kdev_t.h> |
| 17 | 17 | #include <linux/err.h> |
| 18 | 18 | #include <linux/slab.h> |
| 19 | −#include <linux/genhd.h> | |
| 19 | +#include <linux/blkdev.h> | |
| 20 | 20 | #include <linux/mutex.h> |
| 21 | 21 | #include "base.h" |
| 22 | 22 | |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <linux/notifier.h> |
| 22 | 22 | #include <linux/of.h> |
| 23 | 23 | #include <linux/of_device.h> |
| 24 | −#include <linux/genhd.h> | |
| 24 | +#include <linux/blkdev.h> | |
| 25 | 25 | #include <linux/mutex.h> |
| 26 | 26 | #include <linux/pm_runtime.h> |
| 27 | 27 | #include <linux/netdevice.h> |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <linux/syscalls.h> |
| 18 | 18 | #include <linux/mount.h> |
| 19 | 19 | #include <linux/device.h> |
| 20 | −#include <linux/genhd.h> | |
| 20 | +#include <linux/blkdev.h> | |
| 21 | 21 | #include <linux/namei.h> |
| 22 | 22 | #include <linux/fs.h> |
| 23 | 23 | #include <linux/shmem_fs.h> |
| @@ -12,7 +12,6 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <linux/ioctl.h> |
| 13 | 13 | #include <linux/slab.h> |
| 14 | 14 | #include <linux/ratelimit.h> |
| 15 | −#include <linux/genhd.h> | |
| 16 | 15 | #include <linux/netdevice.h> |
| 17 | 16 | #include <linux/mutex.h> |
| 18 | 17 | #include <linux/export.h> |
| @@ -10,7 +10,6 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/blk-mq.h> |
| 11 | 11 | #include <linux/skbuff.h> |
| 12 | 12 | #include <linux/netdevice.h> |
| 13 | −#include <linux/genhd.h> | |
| 14 | 13 | #include <linux/moduleparam.h> |
| 15 | 14 | #include <linux/workqueue.h> |
| 16 | 15 | #include <linux/kthread.h> |
| @@ -138,15 +138,14 @@ static int _drbd_md_sync_page_io(struct drbd_device *device, · 137 unmodified lines | ||
| 138 | 138 | op_flags |= REQ_FUA | REQ_PREFLUSH; |
| 139 | 139 | op_flags |= REQ_SYNC; |
| 140 | 140 | |
| 141 | − bio = bio_alloc_bioset(GFP_NOIO, 1, &drbd_md_io_bio_set); | |
| 142 | − bio_set_dev(bio, bdev->md_bdev); | |
| 141 | + bio = bio_alloc_bioset(bdev->md_bdev, 1, op | op_flags, GFP_NOIO, | |
| 142 | + &drbd_md_io_bio_set); | |
| 143 | 143 | bio->bi_iter.bi_sector = sector; |
| 144 | 144 | err = -EIO; |
| 145 | 145 | if (bio_add_page(bio, device->md_io.page, size, 0) != size) |
| 146 | 146 | goto out; |
| 147 | 147 | bio->bi_private = device; |
| 148 | 148 | bio->bi_end_io = drbd_md_endio; |
| 149 | − bio_set_op_attrs(bio, op, op_flags); | |
| 150 | 149 | |
| 151 | 150 | if (op != REQ_OP_WRITE && device->state.disk == D_DISKLESS && device->ldev == NULL) |
| 152 | 151 | /* special case, drbd_md_read() during drbd_adm_attach(): no get_ldev */ |
| @@ -976,12 +976,13 @@ static void drbd_bm_endio(struct bio *bio) · 975 unmodified lines | ||
| 976 | 976 | |
| 977 | 977 | static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_hold(local) |
| 978 | 978 | { |
| 979 | − struct bio *bio = bio_alloc_bioset(GFP_NOIO, 1, &drbd_md_io_bio_set); | |
| 980 | 979 | struct drbd_device *device = ctx->device; |
| 980 | + unsigned int op = (ctx->flags & BM_AIO_READ) ? REQ_OP_READ : REQ_OP_WRITE; | |
| 981 | + struct bio *bio = bio_alloc_bioset(device->ldev->md_bdev, 1, op, | |
| 982 | + GFP_NOIO, &drbd_md_io_bio_set); | |
| 981 | 983 | struct drbd_bitmap *b = device->bitmap; |
| 982 | 984 | struct page *page; |
| 983 | 985 | unsigned int len; |
| 984 | − unsigned int op = (ctx->flags & BM_AIO_READ) ? REQ_OP_READ : REQ_OP_WRITE; | |
| 985 | 986 | |
| 986 | 987 | sector_t on_disk_sector = |
| 987 | 988 | device->ldev->md.md_offset + device->ldev->md.bm_offset; |
| @@ -1006,14 +1007,12 @@ static void bm_page_io_async(struct drbd_bm_aio_ctx *ctx, int page_nr) __must_ho · 18 unmodified lines | ||
| 1006 | 1007 | bm_store_page_idx(page, page_nr); |
| 1007 | 1008 | } else |
| 1008 | 1009 | page = b->bm_pages[page_nr]; |
| 1009 | − bio_set_dev(bio, device->ldev->md_bdev); | |
| 1010 | 1010 | bio->bi_iter.bi_sector = on_disk_sector; |
| 1011 | 1011 | /* bio_add_page of a single page to an empty bio will always succeed, |
| 1012 | 1012 | * according to api. Do we want to assert that? */ |
| 1013 | 1013 | bio_add_page(bio, page, len, 0); |
| 1014 | 1014 | bio->bi_private = ctx; |
| 1015 | 1015 | bio->bi_end_io = drbd_bm_endio; |
| 1016 | − bio_set_op_attrs(bio, op, 0); | |
| 1017 | 1016 | |
| 1018 | 1017 | if (drbd_insert_fault(device, (op == REQ_OP_WRITE) ? DRBD_FAULT_MD_WR : DRBD_FAULT_MD_RD)) { |
| 1019 | 1018 | bio_io_error(bio); |
| @@ -27,7 +27,6 @@ · 26 unmodified lines | ||
| 27 | 27 | #include <linux/major.h> |
| 28 | 28 | #include <linux/blkdev.h> |
| 29 | 29 | #include <linux/backing-dev.h> |
| 30 | −#include <linux/genhd.h> | |
| 31 | 30 | #include <linux/idr.h> |
| 32 | 31 | #include <linux/dynamic_debug.h> |
| 33 | 32 | #include <net/tcp.h> |
| @@ -1279,16 +1279,16 @@ static void one_flush_endio(struct bio *bio) · 1278 unmodified lines | ||
| 1279 | 1279 | |
| 1280 | 1280 | static void submit_one_flush(struct drbd_device *device, struct issue_flush_context *ctx) |
| 1281 | 1281 | { |
| 1282 | − struct bio *bio = bio_alloc(GFP_NOIO, 0); | |
| 1282 | + struct bio *bio = bio_alloc(device->ldev->backing_bdev, 0, | |
| 1283 | + REQ_OP_FLUSH | REQ_PREFLUSH, GFP_NOIO); | |
| 1283 | 1284 | struct one_flush_context *octx = kmalloc(sizeof(*octx), GFP_NOIO); |
| 1284 | − if (!bio || !octx) { | |
| 1285 | − drbd_warn(device, "Could not allocate a bio, CANNOT ISSUE FLUSH\n"); | |
| 1285 | + | |
| 1286 | + if (!octx) { | |
| 1287 | + drbd_warn(device, "Could not allocate a octx, CANNOT ISSUE FLUSH\n"); | |
| 1286 | 1288 | /* FIXME: what else can I do now? disconnecting or detaching |
| 1287 | 1289 | * really does not help to improve the state of the world, either. |
| 1288 | 1290 | */ |
| 1289 | − kfree(octx); | |
| 1290 | − if (bio) | |
| 1291 | − bio_put(bio); | |
| 1291 | + bio_put(bio); | |
| 1292 | 1292 | |
| 1293 | 1293 | ctx->error = -ENOMEM; |
| 1294 | 1294 | put_ldev(device); |
| @@ -1298,10 +1298,8 @@ static void submit_one_flush(struct drbd_device *device, struct issue_flush_cont · 3 unmodified lines | ||
| 1298 | 1298 | |
| 1299 | 1299 | octx->device = device; |
| 1300 | 1300 | octx->ctx = ctx; |
| 1301 | − bio_set_dev(bio, device->ldev->backing_bdev); | |
| 1302 | 1301 | bio->bi_private = octx; |
| 1303 | 1302 | bio->bi_end_io = one_flush_endio; |
| 1304 | − bio->bi_opf = REQ_OP_FLUSH | REQ_PREFLUSH; | |
| 1305 | 1303 | |
| 1306 | 1304 | device->flush_jif = jiffies; |
| 1307 | 1305 | set_bit(FLUSH_PENDING, &device->flags); |
| @@ -1646,7 +1644,6 @@ int drbd_submit_peer_request(struct drbd_device *device, · 338 unmodified lines | ||
| 1646 | 1644 | unsigned data_size = peer_req->i.size; |
| 1647 | 1645 | unsigned n_bios = 0; |
| 1648 | 1646 | unsigned nr_pages = (data_size + PAGE_SIZE -1) >> PAGE_SHIFT; |
| 1649 | − int err = -ENOMEM; | |
| 1650 | 1647 | |
| 1651 | 1648 | /* TRIM/DISCARD: for now, always use the helper function |
| 1652 | 1649 | * blkdev_issue_zeroout(..., discard=true). |
| @@ -1687,15 +1684,10 @@ int drbd_submit_peer_request(struct drbd_device *device, · 34 unmodified lines | ||
| 1687 | 1684 | * generated bio, but a bio allocated on behalf of the peer. |
| 1688 | 1685 | */ |
| 1689 | 1686 | next_bio: |
| 1690 | − bio = bio_alloc(GFP_NOIO, nr_pages); | |
| 1691 | − if (!bio) { | |
| 1692 | − drbd_err(device, "submit_ee: Allocation of a bio failed (nr_pages=%u)\n", nr_pages); | |
| 1693 | − goto fail; | |
| 1694 | − } | |
| 1687 | + bio = bio_alloc(device->ldev->backing_bdev, nr_pages, op | op_flags, | |
| 1688 | + GFP_NOIO); | |
| 1695 | 1689 | /* > peer_req->i.sector, unless this is the first bio */ |
| 1696 | 1690 | bio->bi_iter.bi_sector = sector; |
| 1697 | − bio_set_dev(bio, device->ldev->backing_bdev); | |
| 1698 | − bio_set_op_attrs(bio, op, op_flags); | |
| 1699 | 1691 | bio->bi_private = peer_req; |
| 1700 | 1692 | bio->bi_end_io = drbd_peer_request_endio; |
| 1701 | 1693 | |
| @@ -1726,14 +1718,6 @@ next_bio: · 24 unmodified lines | ||
| 1726 | 1718 | drbd_submit_bio_noacct(device, fault_type, bio); |
| 1727 | 1719 | } while (bios); |
| 1728 | 1720 | return 0; |
| 1729 | − | |
| 1730 | −fail: | |
| 1731 | − while (bios) { | |
| 1732 | − bio = bios; | |
| 1733 | − bios = bios->bi_next; | |
| 1734 | − bio_put(bio); | |
| 1735 | − } | |
| 1736 | − return err; | |
| 1737 | 1721 | } |
| 1738 | 1722 | |
| 1739 | 1723 | static void drbd_remove_epoch_entry_interval(struct drbd_device *device, |
| @@ -30,7 +30,8 @@ static struct drbd_request *drbd_req_new(struct drbd_device *device, struct bio · 29 unmodified lines | ||
| 30 | 30 | return NULL; |
| 31 | 31 | memset(req, 0, sizeof(*req)); |
| 32 | 32 | |
| 33 | − req->private_bio = bio_clone_fast(bio_src, GFP_NOIO, &drbd_io_bio_set); | |
| 33 | + req->private_bio = bio_alloc_clone(device->ldev->backing_bdev, bio_src, | |
| 34 | + GFP_NOIO, &drbd_io_bio_set); | |
| 34 | 35 | req->private_bio->bi_private = req; |
| 35 | 36 | req->private_bio->bi_end_io = drbd_request_endio; |
| 36 | 37 | |
| @@ -1151,8 +1152,6 @@ drbd_submit_req_private_bio(struct drbd_request *req) · 1114 unmodified lines | ||
| 1151 | 1152 | else |
| 1152 | 1153 | type = DRBD_FAULT_DT_RD; |
| 1153 | 1154 | |
| 1154 | − bio_set_dev(bio, device->ldev->backing_bdev); | |
| 1155 | − | |
| 1156 | 1155 | /* State may have changed since we grabbed our reference on the |
| 1157 | 1156 | * ->ldev member. Double check, and short-circuit to endio. |
| 1158 | 1157 | * In case the last activity log transaction failed to get on |
| @@ -1523,9 +1523,9 @@ int w_restart_disk_io(struct drbd_work *w, int cancel) · 1522 unmodified lines | ||
| 1523 | 1523 | if (bio_data_dir(req->master_bio) == WRITE && req->rq_state & RQ_IN_ACT_LOG) |
| 1524 | 1524 | drbd_al_begin_io(device, &req->i); |
| 1525 | 1525 | |
| 1526 | − req->private_bio = bio_clone_fast(req->master_bio, GFP_NOIO, | |
| 1526 | + req->private_bio = bio_alloc_clone(device->ldev->backing_bdev, | |
| 1527 | + req->master_bio, GFP_NOIO, | |
| 1527 | 1528 | &drbd_io_bio_set); |
| 1528 | − bio_set_dev(req->private_bio, device->ldev->backing_bdev); | |
| 1529 | 1529 | req->private_bio->bi_private = req; |
| 1530 | 1530 | req->private_bio->bi_end_io = drbd_request_endio; |
| 1531 | 1531 | submit_bio_noacct(req->private_bio); |
| @@ -4129,15 +4129,13 @@ static int __floppy_read_block_0(struct block_device *bdev, int drive) · 4128 unmodified lines | ||
| 4129 | 4129 | |
| 4130 | 4130 | cbdata.drive = drive; |
| 4131 | 4131 | |
| 4132 | − bio_init(&bio, &bio_vec, 1); | |
| 4133 | − bio_set_dev(&bio, bdev); | |
| 4132 | + bio_init(&bio, bdev, &bio_vec, 1, REQ_OP_READ); | |
| 4134 | 4133 | bio_add_page(&bio, page, block_size(bdev), 0); |
| 4135 | 4134 | |
| 4136 | 4135 | bio.bi_iter.bi_sector = 0; |
| 4137 | 4136 | bio.bi_flags |= (1 << BIO_QUIET); |
| 4138 | 4137 | bio.bi_private = &cbdata; |
| 4139 | 4138 | bio.bi_end_io = floppy_rb0_cb; |
| 4140 | − bio_set_op_attrs(&bio, REQ_OP_READ, 0); | |
| 4141 | 4139 | |
| 4142 | 4140 | init_completion(&cbdata.complete); |
| 4143 | 4141 | |
| @@ -19,7 +19,6 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <linux/compat.h> |
| 20 | 20 | #include <linux/fs.h> |
| 21 | 21 | #include <linux/module.h> |
| 22 | −#include <linux/genhd.h> | |
| 23 | 22 | #include <linux/blkdev.h> |
| 24 | 23 | #include <linux/blk-mq.h> |
| 25 | 24 | #include <linux/bio.h> |
| @@ -161,9 +160,7 @@ static bool mtip_check_surprise_removal(struct driver_data *dd) · 135 unmodified lines | ||
| 161 | 160 | static struct mtip_cmd *mtip_cmd_from_tag(struct driver_data *dd, |
| 162 | 161 | unsigned int tag) |
| 163 | 162 | { |
| 164 | − struct blk_mq_hw_ctx *hctx = dd->queue->queue_hw_ctx[0]; | |
| 165 | − | |
| 166 | − return blk_mq_rq_to_pdu(blk_mq_tag_to_rq(hctx->tags, tag)); | |
| 163 | + return blk_mq_rq_to_pdu(blk_mq_tag_to_rq(dd->tags.tags[0], tag)); | |
| 167 | 164 | } |
| 168 | 165 | |
| 169 | 166 | /* |
| @@ -15,7 +15,6 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/rwsem.h> |
| 16 | 16 | #include <linux/ata.h> |
| 17 | 17 | #include <linux/interrupt.h> |
| 18 | −#include <linux/genhd.h> | |
| 19 | 18 | |
| 20 | 19 | /* Offset of Subsystem Device ID in pci confoguration space */ |
| 21 | 20 | #define PCI_SUBSYSTEM_DEVICEID 0x2E |
| @@ -1020,9 +1020,8 @@ static void pkt_gather_data(struct pktcdvd_device *pd, struct packet_data *pkt) · 1019 unmodified lines | ||
| 1020 | 1020 | continue; |
| 1021 | 1021 | |
| 1022 | 1022 | bio = pkt->r_bios[f]; |
| 1023 | − bio_reset(bio); | |
| 1023 | + bio_reset(bio, pd->bdev, REQ_OP_READ); | |
| 1024 | 1024 | bio->bi_iter.bi_sector = pkt->sector + f * (CD_FRAMESIZE >> 9); |
| 1025 | − bio_set_dev(bio, pd->bdev); | |
| 1026 | 1025 | bio->bi_end_io = pkt_end_io_read; |
| 1027 | 1026 | bio->bi_private = pkt; |
| 1028 | 1027 | |
| @@ -1034,7 +1033,6 @@ static void pkt_gather_data(struct pktcdvd_device *pd, struct packet_data *pkt) · 5 unmodified lines | ||
| 1034 | 1033 | BUG(); |
| 1035 | 1034 | |
| 1036 | 1035 | atomic_inc(&pkt->io_wait); |
| 1037 | − bio_set_op_attrs(bio, REQ_OP_READ, 0); | |
| 1038 | 1036 | pkt_queue_bio(pd, bio); |
| 1039 | 1037 | frames_read++; |
| 1040 | 1038 | } |
| @@ -1235,9 +1233,8 @@ static void pkt_start_write(struct pktcdvd_device *pd, struct packet_data *pkt) · 194 unmodified lines | ||
| 1235 | 1233 | { |
| 1236 | 1234 | int f; |
| 1237 | 1235 | |
| 1238 | − bio_reset(pkt->w_bio); | |
| 1236 | + bio_reset(pkt->w_bio, pd->bdev, REQ_OP_WRITE); | |
| 1239 | 1237 | pkt->w_bio->bi_iter.bi_sector = pkt->sector; |
| 1240 | − bio_set_dev(pkt->w_bio, pd->bdev); | |
| 1241 | 1238 | pkt->w_bio->bi_end_io = pkt_end_io_packet_write; |
| 1242 | 1239 | pkt->w_bio->bi_private = pkt; |
| 1243 | 1240 | |
| @@ -1270,7 +1267,6 @@ static void pkt_start_write(struct pktcdvd_device *pd, struct packet_data *pkt) · 26 unmodified lines | ||
| 1270 | 1267 | |
| 1271 | 1268 | /* Start the write request */ |
| 1272 | 1269 | atomic_set(&pkt->io_wait, 1); |
| 1273 | − bio_set_op_attrs(pkt->w_bio, REQ_OP_WRITE, 0); | |
| 1274 | 1270 | pkt_queue_bio(pd, pkt->w_bio); |
| 1275 | 1271 | } |
| 1276 | 1272 | |
| @@ -2298,12 +2294,12 @@ static void pkt_end_io_read_cloned(struct bio *bio) · 1021 unmodified lines | ||
| 2298 | 2294 | |
| 2299 | 2295 | static void pkt_make_request_read(struct pktcdvd_device *pd, struct bio *bio) |
| 2300 | 2296 | { |
| 2301 | − struct bio *cloned_bio = bio_clone_fast(bio, GFP_NOIO, &pkt_bio_set); | |
| 2297 | + struct bio *cloned_bio = | |
| 2298 | + bio_alloc_clone(pd->bdev, bio, GFP_NOIO, &pkt_bio_set); | |
| 2302 | 2299 | struct packet_stacked_data *psd = mempool_alloc(&psd_pool, GFP_NOIO); |
| 2303 | 2300 | |
| 2304 | 2301 | psd->pd = pd; |
| 2305 | 2302 | psd->bio = bio; |
| 2306 | − bio_set_dev(cloned_bio, pd->bdev); | |
| 2307 | 2303 | cloned_bio->bi_private = psd; |
| 2308 | 2304 | cloned_bio->bi_end_io = pkt_end_io_read_cloned; |
| 2309 | 2305 | pd->stats.secs_r += bio_sectors(bio); |
| @@ -2404,18 +2400,11 @@ static void pkt_make_request_write(struct request_queue *q, struct bio *bio) · 94 unmodified lines | ||
| 2404 | 2400 | |
| 2405 | 2401 | static void pkt_submit_bio(struct bio *bio) |
| 2406 | 2402 | { |
| 2407 | − struct pktcdvd_device *pd; | |
| 2408 | − char b[BDEVNAME_SIZE]; | |
| 2403 | + struct pktcdvd_device *pd = bio->bi_bdev->bd_disk->queue->queuedata; | |
| 2409 | 2404 | struct bio *split; |
| 2410 | 2405 | |
| 2411 | 2406 | blk_queue_split(&bio); |
| 2412 | 2407 | |
| 2413 | − pd = bio->bi_bdev->bd_disk->queue->queuedata; | |
| 2414 | − if (!pd) { | |
| 2415 | − pr_err("%s incorrect request queue\n", bio_devname(bio, b)); | |
| 2416 | − goto end_io; | |
| 2417 | − } | |
| 2418 | − | |
| 2419 | 2408 | pkt_dbg(2, pd, "start = %6llx stop = %6llx\n", |
| 2420 | 2409 | (unsigned long long)bio->bi_iter.bi_sector, |
| 2421 | 2410 | (unsigned long long)bio_end_sector(bio)); |
| @@ -1343,7 +1343,7 @@ static inline void rnbd_init_hw_queue(struct rnbd_clt_dev *dev, · 1342 unmodified lines | ||
| 1343 | 1343 | |
| 1344 | 1344 | static void rnbd_init_mq_hw_queues(struct rnbd_clt_dev *dev) |
| 1345 | 1345 | { |
| 1346 | − int i; | |
| 1346 | + unsigned long i; | |
| 1347 | 1347 | struct blk_mq_hw_ctx *hctx; |
| 1348 | 1348 | struct rnbd_queue *q; |
| 1349 | 1349 | |
| @@ -12,8 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include "rnbd-srv-dev.h" |
| 13 | 13 | #include "rnbd-log.h" |
| 14 | 14 | |
| 15 | −struct rnbd_dev *rnbd_dev_open(const char *path, fmode_t flags, | |
| 16 | − struct bio_set *bs) | |
| 15 | +struct rnbd_dev *rnbd_dev_open(const char *path, fmode_t flags) | |
| 17 | 16 | { |
| 18 | 17 | struct rnbd_dev *dev; |
| 19 | 18 | int ret; |
| @@ -30,7 +29,6 @@ struct rnbd_dev *rnbd_dev_open(const char *path, fmode_t flags, · 10 unmodified lines | ||
| 30 | 29 | |
| 31 | 30 | dev->blk_open_flags = flags; |
| 32 | 31 | bdevname(dev->bdev, dev->name); |
| 33 | − dev->ibd_bio_set = bs; | |
| 34 | 32 | |
| 35 | 33 | return dev; |
| 36 | 34 | |
| @@ -44,60 +42,3 @@ void rnbd_dev_close(struct rnbd_dev *dev) · 7 unmodified lines | ||
| 44 | 42 | blkdev_put(dev->bdev, dev->blk_open_flags); |
| 45 | 43 | kfree(dev); |
| 46 | 44 | } |
| 47 | − | |
| 48 | −void rnbd_dev_bi_end_io(struct bio *bio) | |
| 49 | −{ | |
| 50 | − struct rnbd_dev_blk_io *io = bio->bi_private; | |
| 51 | − | |
| 52 | − rnbd_endio(io->priv, blk_status_to_errno(bio->bi_status)); | |
| 53 | − bio_put(bio); | |
| 54 | −} | |
| 55 | − | |
| 56 | −/** | |
| 57 | − * rnbd_bio_map_kern - map kernel address into bio | |
| 58 | − * @data: pointer to buffer to map | |
| 59 | − * @bs: bio_set to use. | |
| 60 | − * @len: length in bytes | |
| 61 | − * @gfp_mask: allocation flags for bio allocation | |
| 62 | − * | |
| 63 | − * Map the kernel address into a bio suitable for io to a block | |
| 64 | − * device. Returns an error pointer in case of error. | |
| 65 | − */ | |
| 66 | −struct bio *rnbd_bio_map_kern(void *data, struct bio_set *bs, | |
| 67 | − unsigned int len, gfp_t gfp_mask) | |
| 68 | −{ | |
| 69 | − unsigned long kaddr = (unsigned long)data; | |
| 70 | − unsigned long end = (kaddr + len + PAGE_SIZE - 1) >> PAGE_SHIFT; | |
| 71 | − unsigned long start = kaddr >> PAGE_SHIFT; | |
| 72 | − const int nr_pages = end - start; | |
| 73 | − int offset, i; | |
| 74 | − struct bio *bio; | |
| 75 | − | |
| 76 | − bio = bio_alloc_bioset(gfp_mask, nr_pages, bs); | |
| 77 | − if (!bio) | |
| 78 | − return ERR_PTR(-ENOMEM); | |
| 79 | − | |
| 80 | − offset = offset_in_page(kaddr); | |
| 81 | − for (i = 0; i < nr_pages; i++) { | |
| 82 | − unsigned int bytes = PAGE_SIZE - offset; | |
| 83 | − | |
| 84 | − if (len <= 0) | |
| 85 | − break; | |
| 86 | − | |
| 87 | − if (bytes > len) | |
| 88 | − bytes = len; | |
| 89 | − | |
| 90 | − if (bio_add_page(bio, virt_to_page(data), bytes, | |
| 91 | − offset) < bytes) { | |
| 92 | − /* we don't support partial mappings */ | |
| 93 | − bio_put(bio); | |
| 94 | − return ERR_PTR(-EINVAL); | |
| 95 | − } | |
| 96 | − | |
| 97 | − data += bytes; | |
| 98 | − len -= bytes; | |
| 99 | − offset = 0; | |
| 100 | − } | |
| 101 | − | |
| 102 | − return bio; | |
| 103 | −} | |
| @@ -14,25 +14,16 @@ · 13 unmodified lines | ||
| 14 | 14 | |
| 15 | 15 | struct rnbd_dev { |
| 16 | 16 | struct block_device *bdev; |
| 17 | − struct bio_set *ibd_bio_set; | |
| 18 | 17 | fmode_t blk_open_flags; |
| 19 | 18 | char name[BDEVNAME_SIZE]; |
| 20 | 19 | }; |
| 21 | 20 | |
| 22 | −struct rnbd_dev_blk_io { | |
| 23 | − struct rnbd_dev *dev; | |
| 24 | − void *priv; | |
| 25 | − /* have to be last member for front_pad usage of bioset_init */ | |
| 26 | − struct bio bio; | |
| 27 | −}; | |
| 28 | − | |
| 29 | 21 | /** |
| 30 | 22 | * rnbd_dev_open() - Open a device |
| 23 | + * @path: path to open | |
| 31 | 24 | * @flags: open flags |
| 32 | − * @bs: bio_set to use during block io, | |
| 33 | 25 | */ |
| 34 | −struct rnbd_dev *rnbd_dev_open(const char *path, fmode_t flags, | |
| 35 | − struct bio_set *bs); | |
| 26 | +struct rnbd_dev *rnbd_dev_open(const char *path, fmode_t flags); | |
| 36 | 27 | |
| 37 | 28 | /** |
| 38 | 29 | * rnbd_dev_close() - Close a device |
| @@ -41,11 +32,6 @@ void rnbd_dev_close(struct rnbd_dev *dev); · 2 unmodified lines | ||
| 41 | 32 | |
| 42 | 33 | void rnbd_endio(void *priv, int error); |
| 43 | 34 | |
| 44 | −void rnbd_dev_bi_end_io(struct bio *bio); | |
| 45 | − | |
| 46 | −struct bio *rnbd_bio_map_kern(void *data, struct bio_set *bs, | |
| 47 | − unsigned int len, gfp_t gfp_mask); | |
| 48 | − | |
| 49 | 35 | static inline int rnbd_dev_get_max_segs(const struct rnbd_dev *dev) |
| 50 | 36 | { |
| 51 | 37 | return queue_max_segments(bdev_get_queue(dev->bdev)); |
| @@ -13,7 +13,6 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/kobject.h> |
| 14 | 14 | #include <linux/sysfs.h> |
| 15 | 15 | #include <linux/stat.h> |
| 16 | −#include <linux/genhd.h> | |
| 17 | 16 | #include <linux/list.h> |
| 18 | 17 | #include <linux/moduleparam.h> |
| 19 | 18 | #include <linux/device.h> |
| @@ -114,6 +114,12 @@ rnbd_get_sess_dev(int dev_id, struct rnbd_srv_session *srv_sess) · 113 unmodified lines | ||
| 114 | 114 | return sess_dev; |
| 115 | 115 | } |
| 116 | 116 | |
| 117 | +static void rnbd_dev_bi_end_io(struct bio *bio) | |
| 118 | +{ | |
| 119 | + rnbd_endio(bio->bi_private, blk_status_to_errno(bio->bi_status)); | |
| 120 | + bio_put(bio); | |
| 121 | +} | |
| 122 | + | |
| 117 | 123 | static int process_rdma(struct rnbd_srv_session *srv_sess, |
| 118 | 124 | struct rtrs_srv_op *id, void *data, u32 datalen, |
| 119 | 125 | const void *usr, size_t usrlen) |
| @@ -123,7 +129,6 @@ static int process_rdma(struct rnbd_srv_session *srv_sess, · 3 unmodified lines | ||
| 123 | 129 | struct rnbd_srv_sess_dev *sess_dev; |
| 124 | 130 | u32 dev_id; |
| 125 | 131 | int err; |
| 126 | − struct rnbd_dev_blk_io *io; | |
| 127 | 132 | struct bio *bio; |
| 128 | 133 | short prio; |
| 129 | 134 | |
| @@ -144,33 +149,29 @@ static int process_rdma(struct rnbd_srv_session *srv_sess, · 14 unmodified lines | ||
| 144 | 149 | priv->sess_dev = sess_dev; |
| 145 | 150 | priv->id = id; |
| 146 | 151 | |
| 147 | − /* Generate bio with pages pointing to the rdma buffer */ | |
| 148 | − bio = rnbd_bio_map_kern(data, sess_dev->rnbd_dev->ibd_bio_set, datalen, GFP_KERNEL); | |
| 149 | − if (IS_ERR(bio)) { | |
| 150 | − err = PTR_ERR(bio); | |
| 151 | − rnbd_srv_err(sess_dev, "Failed to generate bio, err: %d\n", err); | |
| 152 | − goto sess_dev_put; | |
| 152 | + bio = bio_alloc(sess_dev->rnbd_dev->bdev, 1, | |
| 153 | + rnbd_to_bio_flags(le32_to_cpu(msg->rw)), GFP_KERNEL); | |
| 154 | + if (bio_add_page(bio, virt_to_page(data), datalen, | |
| 155 | + offset_in_page(data)) != datalen) { | |
| 156 | + rnbd_srv_err(sess_dev, "Failed to map data to bio\n"); | |
| 157 | + err = -EINVAL; | |
| 158 | + goto bio_put; | |
| 153 | 159 | } |
| 154 | 160 | |
| 155 | − io = container_of(bio, struct rnbd_dev_blk_io, bio); | |
| 156 | − io->dev = sess_dev->rnbd_dev; | |
| 157 | − io->priv = priv; | |
| 158 | − | |
| 159 | 161 | bio->bi_end_io = rnbd_dev_bi_end_io; |
| 160 | − bio->bi_private = io; | |
| 161 | − bio->bi_opf = rnbd_to_bio_flags(le32_to_cpu(msg->rw)); | |
| 162 | + bio->bi_private = priv; | |
| 162 | 163 | bio->bi_iter.bi_sector = le64_to_cpu(msg->sector); |
| 163 | 164 | bio->bi_iter.bi_size = le32_to_cpu(msg->bi_size); |
| 164 | 165 | prio = srv_sess->ver < RNBD_PROTO_VER_MAJOR || |
| 165 | 166 | usrlen < sizeof(*msg) ? 0 : le16_to_cpu(msg->prio); |
| 166 | 167 | bio_set_prio(bio, prio); |
| 167 | − bio_set_dev(bio, sess_dev->rnbd_dev->bdev); | |
| 168 | 168 | |
| 169 | 169 | submit_bio(bio); |
| 170 | 170 | |
| 171 | 171 | return 0; |
| 172 | 172 | |
| 173 | −sess_dev_put: | |
| 173 | +bio_put: | |
| 174 | + bio_put(bio); | |
| 174 | 175 | rnbd_put_sess_dev(sess_dev); |
| 175 | 176 | err: |
| 176 | 177 | kfree(priv); |
| @@ -251,7 +252,6 @@ static void destroy_sess(struct rnbd_srv_session *srv_sess) · 74 unmodified lines | ||
| 251 | 252 | |
| 252 | 253 | out: |
| 253 | 254 | xa_destroy(&srv_sess->index_idr); |
| 254 | − bioset_exit(&srv_sess->sess_bio_set); | |
| 255 | 255 | |
| 256 | 256 | pr_info("RTRS Session %s disconnected\n", srv_sess->sessname); |
| 257 | 257 | |
| @@ -280,16 +280,6 @@ static int create_sess(struct rtrs_srv_sess *rtrs) · 22 unmodified lines | ||
| 280 | 280 | return -ENOMEM; |
| 281 | 281 | |
| 282 | 282 | srv_sess->queue_depth = rtrs_srv_get_queue_depth(rtrs); |
| 283 | − err = bioset_init(&srv_sess->sess_bio_set, srv_sess->queue_depth, | |
| 284 | − offsetof(struct rnbd_dev_blk_io, bio), | |
| 285 | − BIOSET_NEED_BVECS); | |
| 286 | − if (err) { | |
| 287 | − pr_err("Allocating srv_session for path %s failed\n", | |
| 288 | − pathname); | |
| 289 | − kfree(srv_sess); | |
| 290 | − return err; | |
| 291 | − } | |
| 292 | − | |
| 293 | 283 | xa_init_flags(&srv_sess->index_idr, XA_FLAGS_ALLOC); |
| 294 | 284 | INIT_LIST_HEAD(&srv_sess->sess_dev_list); |
| 295 | 285 | mutex_init(&srv_sess->lock); |
| @@ -738,8 +728,7 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess, · 442 unmodified lines | ||
| 738 | 728 | goto reject; |
| 739 | 729 | } |
| 740 | 730 | |
| 741 | − rnbd_dev = rnbd_dev_open(full_path, open_flags, | |
| 742 | − &srv_sess->sess_bio_set); | |
| 731 | + rnbd_dev = rnbd_dev_open(full_path, open_flags); | |
| 743 | 732 | if (IS_ERR(rnbd_dev)) { |
| 744 | 733 | pr_err("Opening device '%s' on session %s failed, failed to open the block device, err: %ld\n", |
| 745 | 734 | full_path, srv_sess->sessname, PTR_ERR(rnbd_dev)); |
| @@ -23,7 +23,6 @@ struct rnbd_srv_session { · 22 unmodified lines | ||
| 23 | 23 | struct rtrs_srv_sess *rtrs; |
| 24 | 24 | char sessname[NAME_MAX]; |
| 25 | 25 | int queue_depth; |
| 26 | − struct bio_set sess_bio_set; | |
| 27 | 26 | |
| 28 | 27 | struct xarray index_idr; |
| 29 | 28 | /* List of struct rnbd_srv_sess_dev */ |
| @@ -9,7 +9,6 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <linux/types.h> |
| 10 | 10 | #include <linux/blk-mq.h> |
| 11 | 11 | #include <linux/hdreg.h> |
| 12 | −#include <linux/genhd.h> | |
| 13 | 12 | #include <linux/cdrom.h> |
| 14 | 13 | #include <linux/slab.h> |
| 15 | 14 | #include <linux/spinlock.h> |
| @@ -69,13 +69,6 @@ struct virtio_blk { · 68 unmodified lines | ||
| 69 | 69 | /* Process context for config space updates */ |
| 70 | 70 | struct work_struct config_work; |
| 71 | 71 | |
| 72 | − /* | |
| 73 | − * Tracks references from block_device_operations open/release and | |
| 74 | − * virtio_driver probe/remove so this object can be freed once no | |
| 75 | − * longer in use. | |
| 76 | − */ | |
| 77 | − refcount_t refs; | |
| 78 | − | |
| 79 | 72 | /* Ida index - used to track minor number allocations. */ |
| 80 | 73 | int index; |
| 81 | 74 | |
| @@ -386,43 +379,6 @@ out: · 304 unmodified lines | ||
| 386 | 379 | return err; |
| 387 | 380 | } |
| 388 | 381 | |
| 389 | −static void virtblk_get(struct virtio_blk *vblk) | |
| 390 | −{ | |
| 391 | − refcount_inc(&vblk->refs); | |
| 392 | −} | |
| 393 | − | |
| 394 | −static void virtblk_put(struct virtio_blk *vblk) | |
| 395 | −{ | |
| 396 | − if (refcount_dec_and_test(&vblk->refs)) { | |
| 397 | − ida_simple_remove(&vd_index_ida, vblk->index); | |
| 398 | − mutex_destroy(&vblk->vdev_mutex); | |
| 399 | − kfree(vblk); | |
| 400 | − } | |
| 401 | −} | |
| 402 | − | |
| 403 | −static int virtblk_open(struct block_device *bd, fmode_t mode) | |
| 404 | −{ | |
| 405 | − struct virtio_blk *vblk = bd->bd_disk->private_data; | |
| 406 | − int ret = 0; | |
| 407 | − | |
| 408 | − mutex_lock(&vblk->vdev_mutex); | |
| 409 | − | |
| 410 | − if (vblk->vdev) | |
| 411 | − virtblk_get(vblk); | |
| 412 | − else | |
| 413 | − ret = -ENXIO; | |
| 414 | − | |
| 415 | − mutex_unlock(&vblk->vdev_mutex); | |
| 416 | − return ret; | |
| 417 | −} | |
| 418 | − | |
| 419 | −static void virtblk_release(struct gendisk *disk, fmode_t mode) | |
| 420 | −{ | |
| 421 | − struct virtio_blk *vblk = disk->private_data; | |
| 422 | − | |
| 423 | − virtblk_put(vblk); | |
| 424 | −} | |
| 425 | − | |
| 426 | 382 | /* We provide getgeo only to please some old bootloader/partitioning tools */ |
| 427 | 383 | static int virtblk_getgeo(struct block_device *bd, struct hd_geometry *geo) |
| 428 | 384 | { |
| @@ -455,11 +411,19 @@ out: · 26 unmodified lines | ||
| 455 | 411 | return ret; |
| 456 | 412 | } |
| 457 | 413 | |
| 414 | +static void virtblk_free_disk(struct gendisk *disk) | |
| 415 | +{ | |
| 416 | + struct virtio_blk *vblk = disk->private_data; | |
| 417 | + | |
| 418 | + ida_simple_remove(&vd_index_ida, vblk->index); | |
| 419 | + mutex_destroy(&vblk->vdev_mutex); | |
| 420 | + kfree(vblk); | |
| 421 | +} | |
| 422 | + | |
| 458 | 423 | static const struct block_device_operations virtblk_fops = { |
| 459 | − .owner = THIS_MODULE, | |
| 460 | − .open = virtblk_open, | |
| 461 | − .release = virtblk_release, | |
| 462 | − .getgeo = virtblk_getgeo, | |
| 424 | + .owner = THIS_MODULE, | |
| 425 | + .getgeo = virtblk_getgeo, | |
| 426 | + .free_disk = virtblk_free_disk, | |
| 463 | 427 | }; |
| 464 | 428 | |
| 465 | 429 | static int index_to_minor(int index) |
| @@ -784,8 +748,6 @@ static int virtblk_probe(struct virtio_device *vdev) · 318 unmodified lines | ||
| 784 | 748 | goto out_free_index; |
| 785 | 749 | } |
| 786 | 750 | |
| 787 | − /* This reference is dropped in virtblk_remove(). */ | |
| 788 | − refcount_set(&vblk->refs, 1); | |
| 789 | 751 | mutex_init(&vblk->vdev_mutex); |
| 790 | 752 | |
| 791 | 753 | vblk->vdev = vdev; |
| @@ -968,7 +930,7 @@ static void virtblk_remove(struct virtio_device *vdev) · 176 unmodified lines | ||
| 968 | 930 | flush_work(&vblk->config_work); |
| 969 | 931 | |
| 970 | 932 | del_gendisk(vblk->disk); |
| 971 | − blk_cleanup_disk(vblk->disk); | |
| 933 | + blk_cleanup_queue(vblk->disk->queue); | |
| 972 | 934 | blk_mq_free_tag_set(&vblk->tag_set); |
| 973 | 935 | |
| 974 | 936 | mutex_lock(&vblk->vdev_mutex); |
| @@ -984,7 +946,7 @@ static void virtblk_remove(struct virtio_device *vdev) · 9 unmodified lines | ||
| 984 | 946 | |
| 985 | 947 | mutex_unlock(&vblk->vdev_mutex); |
| 986 | 948 | |
| 987 | − virtblk_put(vblk); | |
| 949 | + put_disk(vblk->disk); | |
| 988 | 950 | } |
| 989 | 951 | |
| 990 | 952 | #ifdef CONFIG_PM_SLEEP |
| @@ -1326,16 +1326,13 @@ static int dispatch_rw_block_io(struct xen_blkif_ring *ring, · 1325 unmodified lines | ||
| 1326 | 1326 | pages[i]->page, |
| 1327 | 1327 | seg[i].nsec << 9, |
| 1328 | 1328 | seg[i].offset) == 0)) { |
| 1329 | − bio = bio_alloc(GFP_KERNEL, bio_max_segs(nseg - i)); | |
| 1330 | − if (unlikely(bio == NULL)) | |
| 1331 | − goto fail_put_bio; | |
| 1332 | − | |
| 1329 | + bio = bio_alloc(preq.bdev, bio_max_segs(nseg - i), | |
| 1330 | + operation | operation_flags, | |
| 1331 | + GFP_KERNEL); | |
| 1333 | 1332 | biolist[nbio++] = bio; |
| 1334 | − bio_set_dev(bio, preq.bdev); | |
| 1335 | 1333 | bio->bi_private = pending_req; |
| 1336 | 1334 | bio->bi_end_io = end_block_io_op; |
| 1337 | 1335 | bio->bi_iter.bi_sector = preq.sector_number; |
| 1338 | − bio_set_op_attrs(bio, operation, operation_flags); | |
| 1339 | 1336 | } |
| 1340 | 1337 | |
| 1341 | 1338 | preq.sector_number += seg[i].nsec; |
| @@ -1345,15 +1342,11 @@ static int dispatch_rw_block_io(struct xen_blkif_ring *ring, · 3 unmodified lines | ||
| 1345 | 1342 | if (!bio) { |
| 1346 | 1343 | BUG_ON(operation_flags != REQ_PREFLUSH); |
| 1347 | 1344 | |
| 1348 | − bio = bio_alloc(GFP_KERNEL, 0); | |
| 1349 | − if (unlikely(bio == NULL)) | |
| 1350 | − goto fail_put_bio; | |
| 1351 | − | |
| 1345 | + bio = bio_alloc(preq.bdev, 0, operation | operation_flags, | |
| 1346 | + GFP_KERNEL); | |
| 1352 | 1347 | biolist[nbio++] = bio; |
| 1353 | − bio_set_dev(bio, preq.bdev); | |
| 1354 | 1348 | bio->bi_private = pending_req; |
| 1355 | 1349 | bio->bi_end_io = end_block_io_op; |
| 1356 | − bio_set_op_attrs(bio, operation, operation_flags); | |
| 1357 | 1350 | } |
| 1358 | 1351 | |
| 1359 | 1352 | atomic_set(&pending_req->pendcnt, nbio); |
| @@ -1381,14 +1374,6 @@ static int dispatch_rw_block_io(struct xen_blkif_ring *ring, · 21 unmodified lines | ||
| 1381 | 1374 | free_req(ring, pending_req); |
| 1382 | 1375 | msleep(1); /* back off a bit */ |
| 1383 | 1376 | return -EIO; |
| 1384 | − | |
| 1385 | − fail_put_bio: | |
| 1386 | − for (i = 0; i < nbio; i++) | |
| 1387 | − bio_put(biolist[i]); | |
| 1388 | − atomic_set(&pending_req->pendcnt, 1); | |
| 1389 | − __end_block_io_op(pending_req, BLK_STS_RESOURCE); | |
| 1390 | − msleep(1); /* back off a bit */ | |
| 1391 | − return -EIO; | |
| 1392 | 1377 | } |
| 1393 | 1378 | |
| 1394 | 1379 | |
| @@ -22,7 +22,6 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <linux/blkdev.h> |
| 23 | 23 | #include <linux/buffer_head.h> |
| 24 | 24 | #include <linux/device.h> |
| 25 | −#include <linux/genhd.h> | |
| 26 | 25 | #include <linux/highmem.h> |
| 27 | 26 | #include <linux/slab.h> |
| 28 | 27 | #include <linux/backing-dev.h> |
| @@ -617,24 +616,21 @@ static int read_from_bdev_async(struct zram *zram, struct bio_vec *bvec, · 588 unmodified lines | ||
| 617 | 616 | { |
| 618 | 617 | struct bio *bio; |
| 619 | 618 | |
| 620 | − bio = bio_alloc(GFP_NOIO, 1); | |
| 619 | + bio = bio_alloc(zram->bdev, 1, parent ? parent->bi_opf : REQ_OP_READ, | |
| 620 | + GFP_NOIO); | |
| 621 | 621 | if (!bio) |
| 622 | 622 | return -ENOMEM; |
| 623 | 623 | |
| 624 | 624 | bio->bi_iter.bi_sector = entry * (PAGE_SIZE >> 9); |
| 625 | − bio_set_dev(bio, zram->bdev); | |
| 626 | 625 | if (!bio_add_page(bio, bvec->bv_page, bvec->bv_len, bvec->bv_offset)) { |
| 627 | 626 | bio_put(bio); |
| 628 | 627 | return -EIO; |
| 629 | 628 | } |
| 630 | 629 | |
| 631 | − if (!parent) { | |
| 632 | − bio->bi_opf = REQ_OP_READ; | |
| 630 | + if (!parent) | |
| 633 | 631 | bio->bi_end_io = zram_page_end_io; |
| 634 | − } else { | |
| 635 | − bio->bi_opf = parent->bi_opf; | |
| 632 | + else | |
| 636 | 633 | bio_chain(bio, parent); |
| 637 | − } | |
| 638 | 634 | |
| 639 | 635 | submit_bio(bio); |
| 640 | 636 | return 1; |
| @@ -747,10 +743,9 @@ static ssize_t writeback_store(struct device *dev, · 106 unmodified lines | ||
| 747 | 743 | continue; |
| 748 | 744 | } |
| 749 | 745 | |
| 750 | − bio_init(&bio, &bio_vec, 1); | |
| 751 | − bio_set_dev(&bio, zram->bdev); | |
| 746 | + bio_init(&bio, zram->bdev, &bio_vec, 1, | |
| 747 | + REQ_OP_WRITE | REQ_SYNC); | |
| 752 | 748 | bio.bi_iter.bi_sector = blk_idx * (PAGE_SIZE >> 9); |
| 753 | − bio.bi_opf = REQ_OP_WRITE | REQ_SYNC; | |
| 754 | 749 | |
| 755 | 750 | bio_add_page(&bio, bvec.bv_page, bvec.bv_len, |
| 756 | 751 | bvec.bv_offset); |
| @@ -15,7 +15,6 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/slab.h> |
| 16 | 16 | #include <linux/dma-mapping.h> |
| 17 | 17 | #include <linux/cdrom.h> |
| 18 | −#include <linux/genhd.h> | |
| 19 | 18 | #include <linux/bio.h> |
| 20 | 19 | #include <linux/blk-mq.h> |
| 21 | 20 | #include <linux/interrupt.h> |
| @@ -38,7 +38,7 @@ · 37 unmodified lines | ||
| 38 | 38 | #include <linux/poll.h> |
| 39 | 39 | #include <linux/init.h> |
| 40 | 40 | #include <linux/fs.h> |
| 41 | −#include <linux/genhd.h> | |
| 41 | +#include <linux/blkdev.h> | |
| 42 | 42 | #include <linux/interrupt.h> |
| 43 | 43 | #include <linux/mm.h> |
| 44 | 44 | #include <linux/nodemask.h> |
| @@ -204,6 +204,7 @@ config BLK_DEV_DM · 203 unmodified lines | ||
| 204 | 204 | tristate "Device mapper support" |
| 205 | 205 | select BLOCK_HOLDER_DEPRECATED if SYSFS |
| 206 | 206 | select BLK_DEV_DM_BUILTIN |
| 207 | + select BLK_MQ_STACKING | |
| 207 | 208 | depends on DAX || DAX=n |
| 208 | 209 | help |
| 209 | 210 | Device-mapper is a low level volume manager. It works by allowing |
| @@ -26,7 +26,8 @@ struct bio *bch_bbio_alloc(struct cache_set *c) · 25 unmodified lines | ||
| 26 | 26 | struct bbio *b = mempool_alloc(&c->bio_meta, GFP_NOIO); |
| 27 | 27 | struct bio *bio = &b->bio; |
| 28 | 28 | |
| 29 | − bio_init(bio, bio->bi_inline_vecs, meta_bucket_pages(&c->cache->sb)); | |
| 29 | + bio_init(bio, NULL, bio->bi_inline_vecs, | |
| 30 | + meta_bucket_pages(&c->cache->sb), 0); | |
| 30 | 31 | |
| 31 | 32 | return bio; |
| 32 | 33 | } |
| @@ -53,14 +53,12 @@ static int journal_read_bucket(struct cache *ca, struct list_head *list, · 52 unmodified lines | ||
| 53 | 53 | reread: left = ca->sb.bucket_size - offset; |
| 54 | 54 | len = min_t(unsigned int, left, PAGE_SECTORS << JSET_BITS); |
| 55 | 55 | |
| 56 | − bio_reset(bio); | |
| 56 | + bio_reset(bio, ca->bdev, REQ_OP_READ); | |
| 57 | 57 | bio->bi_iter.bi_sector = bucket + offset; |
| 58 | − bio_set_dev(bio, ca->bdev); | |
| 59 | 58 | bio->bi_iter.bi_size = len << 9; |
| 60 | 59 | |
| 61 | 60 | bio->bi_end_io = journal_read_endio; |
| 62 | 61 | bio->bi_private = &cl; |
| 63 | − bio_set_op_attrs(bio, REQ_OP_READ, 0); | |
| 64 | 62 | bch_bio_map(bio, data); |
| 65 | 63 | |
| 66 | 64 | closure_bio_submit(ca->set, bio, &cl); |
| @@ -611,11 +609,9 @@ static void do_journal_discard(struct cache *ca) · 544 unmodified lines | ||
| 611 | 609 | |
| 612 | 610 | atomic_set(&ja->discard_in_flight, DISCARD_IN_FLIGHT); |
| 613 | 611 | |
| 614 | − bio_init(bio, bio->bi_inline_vecs, 1); | |
| 615 | − bio_set_op_attrs(bio, REQ_OP_DISCARD, 0); | |
| 612 | + bio_init(bio, ca->bdev, bio->bi_inline_vecs, 1, REQ_OP_DISCARD); | |
| 616 | 613 | bio->bi_iter.bi_sector = bucket_to_sector(ca->set, |
| 617 | 614 | ca->sb.d[ja->discard_idx]); |
| 618 | − bio_set_dev(bio, ca->bdev); | |
| 619 | 615 | bio->bi_iter.bi_size = bucket_bytes(ca); |
| 620 | 616 | bio->bi_end_io = journal_discard_endio; |
| 621 | 617 | |
| @@ -773,16 +769,14 @@ static void journal_write_unlocked(struct closure *cl) · 151 unmodified lines | ||
| 773 | 769 | |
| 774 | 770 | atomic_long_add(sectors, &ca->meta_sectors_written); |
| 775 | 771 | |
| 776 | − bio_reset(bio); | |
| 772 | + bio_reset(bio, ca->bdev, REQ_OP_WRITE | | |
| 773 | + REQ_SYNC | REQ_META | REQ_PREFLUSH | REQ_FUA); | |
| 774 | + bch_bio_map(bio, w->data); | |
| 777 | 775 | bio->bi_iter.bi_sector = PTR_OFFSET(k, i); |
| 778 | − bio_set_dev(bio, ca->bdev); | |
| 779 | 776 | bio->bi_iter.bi_size = sectors << 9; |
| 780 | 777 | |
| 781 | 778 | bio->bi_end_io = journal_write_endio; |
| 782 | 779 | bio->bi_private = w; |
| 783 | − bio_set_op_attrs(bio, REQ_OP_WRITE, | |
| 784 | − REQ_SYNC|REQ_META|REQ_PREFLUSH|REQ_FUA); | |
| 785 | − bch_bio_map(bio, w->data); | |
| 786 | 780 | |
| 787 | 781 | trace_bcache_journal_write(bio, w->data->keys); |
| 788 | 782 | bio_list_add(&list, bio); |
| @@ -79,8 +79,8 @@ static void moving_init(struct moving_io *io) · 78 unmodified lines | ||
| 79 | 79 | { |
| 80 | 80 | struct bio *bio = &io->bio.bio; |
| 81 | 81 | |
| 82 | − bio_init(bio, bio->bi_inline_vecs, | |
| 83 | − DIV_ROUND_UP(KEY_SIZE(&io->w->key), PAGE_SECTORS)); | |
| 82 | + bio_init(bio, NULL, bio->bi_inline_vecs, | |
| 83 | + DIV_ROUND_UP(KEY_SIZE(&io->w->key), PAGE_SECTORS), 0); | |
| 84 | 84 | bio_get(bio); |
| 85 | 85 | bio_set_prio(bio, IOPRIO_PRIO_VALUE(IOPRIO_CLASS_IDLE, 0)); |
| 86 | 86 | |
| @@ -685,8 +685,7 @@ static void do_bio_hook(struct search *s, · 684 unmodified lines | ||
| 685 | 685 | { |
| 686 | 686 | struct bio *bio = &s->bio.bio; |
| 687 | 687 | |
| 688 | − bio_init(bio, NULL, 0); | |
| 689 | − __bio_clone_fast(bio, orig_bio); | |
| 688 | + bio_init_clone(bio->bi_bdev, bio, orig_bio, GFP_NOIO); | |
| 690 | 689 | /* |
| 691 | 690 | * bi_end_io can be set separately somewhere else, e.g. the |
| 692 | 691 | * variants in, |
| @@ -831,11 +830,11 @@ static void cached_dev_read_done(struct closure *cl) · 138 unmodified lines | ||
| 831 | 830 | */ |
| 832 | 831 | |
| 833 | 832 | if (s->iop.bio) { |
| 834 | − bio_reset(s->iop.bio); | |
| 833 | + bio_reset(s->iop.bio, s->cache_miss->bi_bdev, REQ_OP_READ); | |
| 835 | 834 | s->iop.bio->bi_iter.bi_sector = |
| 836 | 835 | s->cache_miss->bi_iter.bi_sector; |
| 837 | − bio_copy_dev(s->iop.bio, s->cache_miss); | |
| 838 | 836 | s->iop.bio->bi_iter.bi_size = s->insert_bio_sectors << 9; |
| 837 | + bio_clone_blkg_association(s->iop.bio, s->cache_miss); | |
| 839 | 838 | bch_bio_map(s->iop.bio, NULL); |
| 840 | 839 | |
| 841 | 840 | bio_copy_data(s->cache_miss, s->iop.bio); |
| @@ -913,14 +912,13 @@ static int cached_dev_cache_miss(struct btree *b, struct search *s, · 71 unmodified lines | ||
| 913 | 912 | /* btree_search_recurse()'s btree iterator is no good anymore */ |
| 914 | 913 | ret = miss == bio ? MAP_DONE : -EINTR; |
| 915 | 914 | |
| 916 | − cache_bio = bio_alloc_bioset(GFP_NOWAIT, | |
| 915 | + cache_bio = bio_alloc_bioset(miss->bi_bdev, | |
| 917 | 916 | DIV_ROUND_UP(s->insert_bio_sectors, PAGE_SECTORS), |
| 918 | − &dc->disk.bio_split); | |
| 917 | + 0, GFP_NOWAIT, &dc->disk.bio_split); | |
| 919 | 918 | if (!cache_bio) |
| 920 | 919 | goto out_submit; |
| 921 | 920 | |
| 922 | 921 | cache_bio->bi_iter.bi_sector = miss->bi_iter.bi_sector; |
| 923 | − bio_copy_dev(cache_bio, miss); | |
| 924 | 922 | cache_bio->bi_iter.bi_size = s->insert_bio_sectors << 9; |
| 925 | 923 | |
| 926 | 924 | cache_bio->bi_end_io = backing_request_endio; |
| @@ -1025,21 +1023,21 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) · 98 unmodified lines | ||
| 1025 | 1023 | */ |
| 1026 | 1024 | struct bio *flush; |
| 1027 | 1025 | |
| 1028 | − flush = bio_alloc_bioset(GFP_NOIO, 0, | |
| 1029 | − &dc->disk.bio_split); | |
| 1026 | + flush = bio_alloc_bioset(bio->bi_bdev, 0, | |
| 1027 | + REQ_OP_WRITE | REQ_PREFLUSH, | |
| 1028 | + GFP_NOIO, &dc->disk.bio_split); | |
| 1030 | 1029 | if (!flush) { |
| 1031 | 1030 | s->iop.status = BLK_STS_RESOURCE; |
| 1032 | 1031 | goto insert_data; |
| 1033 | 1032 | } |
| 1034 | − bio_copy_dev(flush, bio); | |
| 1035 | 1033 | flush->bi_end_io = backing_request_endio; |
| 1036 | 1034 | flush->bi_private = cl; |
| 1037 | − flush->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 1038 | 1035 | /* I/O request sent to backing device */ |
| 1039 | 1036 | closure_bio_submit(s->iop.c, flush, cl); |
| 1040 | 1037 | } |
| 1041 | 1038 | } else { |
| 1042 | − s->iop.bio = bio_clone_fast(bio, GFP_NOIO, &dc->disk.bio_split); | |
| 1039 | + s->iop.bio = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOIO, | |
| 1040 | + &dc->disk.bio_split); | |
| 1043 | 1041 | /* I/O request sent to backing device */ |
| 1044 | 1042 | bio->bi_end_io = backing_request_endio; |
| 1045 | 1043 | closure_bio_submit(s->iop.c, bio, cl); |
| @@ -18,7 +18,6 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <linux/blkdev.h> |
| 19 | 19 | #include <linux/pagemap.h> |
| 20 | 20 | #include <linux/debugfs.h> |
| 21 | −#include <linux/genhd.h> | |
| 22 | 21 | #include <linux/idr.h> |
| 23 | 22 | #include <linux/kthread.h> |
| 24 | 23 | #include <linux/workqueue.h> |
| @@ -343,8 +342,7 @@ void bch_write_bdev_super(struct cached_dev *dc, struct closure *parent) · 318 unmodified lines | ||
| 343 | 342 | down(&dc->sb_write_mutex); |
| 344 | 343 | closure_init(cl, parent); |
| 345 | 344 | |
| 346 | − bio_init(bio, dc->sb_bv, 1); | |
| 347 | − bio_set_dev(bio, dc->bdev); | |
| 345 | + bio_init(bio, dc->bdev, dc->sb_bv, 1, 0); | |
| 348 | 346 | bio->bi_end_io = write_bdev_super_endio; |
| 349 | 347 | bio->bi_private = dc; |
| 350 | 348 | |
| @@ -387,8 +385,7 @@ void bcache_write_super(struct cache_set *c) · 36 unmodified lines | ||
| 387 | 385 | if (ca->sb.version < version) |
| 388 | 386 | ca->sb.version = version; |
| 389 | 387 | |
| 390 | − bio_init(bio, ca->sb_bv, 1); | |
| 391 | − bio_set_dev(bio, ca->bdev); | |
| 388 | + bio_init(bio, ca->bdev, ca->sb_bv, 1, 0); | |
| 392 | 389 | bio->bi_end_io = write_super_endio; |
| 393 | 390 | bio->bi_private = ca; |
| 394 | 391 | |
| @@ -2240,7 +2237,7 @@ static int cache_alloc(struct cache *ca) · 1845 unmodified lines | ||
| 2240 | 2237 | __module_get(THIS_MODULE); |
| 2241 | 2238 | kobject_init(&ca->kobj, &bch_cache_ktype); |
| 2242 | 2239 | |
| 2243 | − bio_init(&ca->journal.bio, ca->journal.bio.bi_inline_vecs, 8); | |
| 2240 | + bio_init(&ca->journal.bio, NULL, ca->journal.bio.bi_inline_vecs, 8, 0); | |
| 2244 | 2241 | |
| 2245 | 2242 | /* |
| 2246 | 2243 | * when ca->sb.njournal_buckets is not zero, journal exists, |
| @@ -292,8 +292,8 @@ static void dirty_init(struct keybuf_key *w) · 291 unmodified lines | ||
| 292 | 292 | struct dirty_io *io = w->private; |
| 293 | 293 | struct bio *bio = &io->bio; |
| 294 | 294 | |
| 295 | − bio_init(bio, bio->bi_inline_vecs, | |
| 296 | − DIV_ROUND_UP(KEY_SIZE(&w->key), PAGE_SECTORS)); | |
| 295 | + bio_init(bio, NULL, bio->bi_inline_vecs, | |
| 296 | + DIV_ROUND_UP(KEY_SIZE(&w->key), PAGE_SECTORS), 0); | |
| 297 | 297 | if (!io->dc->writeback_percent) |
| 298 | 298 | bio_set_prio(bio, IOPRIO_PRIO_VALUE(IOPRIO_CLASS_IDLE, 0)); |
| 299 | 299 | |
| @@ -744,21 +744,14 @@ static void check_if_tick_bio_needed(struct cache *cache, struct bio *bio) · 743 unmodified lines | ||
| 744 | 744 | spin_unlock_irq(&cache->lock); |
| 745 | 745 | } |
| 746 | 746 | |
| 747 | −static void __remap_to_origin_clear_discard(struct cache *cache, struct bio *bio, | |
| 748 | − dm_oblock_t oblock, bool bio_has_pbd) | |
| 749 | −{ | |
| 750 | − if (bio_has_pbd) | |
| 751 | − check_if_tick_bio_needed(cache, bio); | |
| 752 | − remap_to_origin(cache, bio); | |
| 753 | − if (bio_data_dir(bio) == WRITE) | |
| 754 | − clear_discard(cache, oblock_to_dblock(cache, oblock)); | |
| 755 | −} | |
| 756 | − | |
| 757 | 747 | static void remap_to_origin_clear_discard(struct cache *cache, struct bio *bio, |
| 758 | 748 | dm_oblock_t oblock) |
| 759 | 749 | { |
| 760 | 750 | // FIXME: check_if_tick_bio_needed() is called way too much through this interface |
| 761 | − __remap_to_origin_clear_discard(cache, bio, oblock, true); | |
| 751 | + check_if_tick_bio_needed(cache, bio); | |
| 752 | + remap_to_origin(cache, bio); | |
| 753 | + if (bio_data_dir(bio) == WRITE) | |
| 754 | + clear_discard(cache, oblock_to_dblock(cache, oblock)); | |
| 762 | 755 | } |
| 763 | 756 | |
| 764 | 757 | static void remap_to_cache_dirty(struct cache *cache, struct bio *bio, |
| @@ -826,16 +819,15 @@ static void issue_op(struct bio *bio, void *context) · 61 unmodified lines | ||
| 826 | 819 | static void remap_to_origin_and_cache(struct cache *cache, struct bio *bio, |
| 827 | 820 | dm_oblock_t oblock, dm_cblock_t cblock) |
| 828 | 821 | { |
| 829 | − struct bio *origin_bio = bio_clone_fast(bio, GFP_NOIO, &cache->bs); | |
| 822 | + struct bio *origin_bio = bio_alloc_clone(cache->origin_dev->bdev, bio, | |
| 823 | + GFP_NOIO, &cache->bs); | |
| 830 | 824 | |
| 831 | 825 | BUG_ON(!origin_bio); |
| 832 | 826 | |
| 833 | 827 | bio_chain(origin_bio, bio); |
| 834 | − /* | |
| 835 | − * Passing false to __remap_to_origin_clear_discard() skips | |
| 836 | − * all code that might use per_bio_data (since clone doesn't have it) | |
| 837 | − */ | |
| 838 | − __remap_to_origin_clear_discard(cache, origin_bio, oblock, false); | |
| 828 | + | |
| 829 | + if (bio_data_dir(origin_bio) == WRITE) | |
| 830 | + clear_discard(cache, oblock_to_dblock(cache, oblock)); | |
| 839 | 831 | submit_bio(origin_bio); |
| 840 | 832 | |
| 841 | 833 | remap_to_cache(cache, bio, cblock); |
| @@ -11,7 +11,6 @@ · 10 unmodified lines | ||
| 11 | 11 | |
| 12 | 12 | #include <linux/kthread.h> |
| 13 | 13 | #include <linux/ktime.h> |
| 14 | −#include <linux/genhd.h> | |
| 15 | 14 | #include <linux/blk-mq.h> |
| 16 | 15 | #include <linux/blk-crypto-profile.h> |
| 17 | 16 | |
| @@ -234,7 +234,7 @@ static volatile unsigned long dm_crypt_pages_per_client; · 233 unmodified lines | ||
| 234 | 234 | #define DM_CRYPT_MEMORY_PERCENT 2 |
| 235 | 235 | #define DM_CRYPT_MIN_PAGES_PER_CLIENT (BIO_MAX_VECS * 16) |
| 236 | 236 | |
| 237 | −static void clone_init(struct dm_crypt_io *, struct bio *); | |
| 237 | +static void crypt_endio(struct bio *clone); | |
| 238 | 238 | static void kcryptd_queue_crypt(struct dm_crypt_io *io); |
| 239 | 239 | static struct scatterlist *crypt_get_sg_data(struct crypt_config *cc, |
| 240 | 240 | struct scatterlist *sg); |
| @@ -1364,11 +1364,10 @@ static int crypt_convert_block_aead(struct crypt_config *cc, · 1123 unmodified lines | ||
| 1364 | 1364 | } |
| 1365 | 1365 | |
| 1366 | 1366 | if (r == -EBADMSG) { |
| 1367 | − char b[BDEVNAME_SIZE]; | |
| 1368 | 1367 | sector_t s = le64_to_cpu(*sector); |
| 1369 | 1368 | |
| 1370 | − DMERR_LIMIT("%s: INTEGRITY AEAD ERROR, sector %llu", | |
| 1371 | − bio_devname(ctx->bio_in, b), s); | |
| 1369 | + DMERR_LIMIT("%pg: INTEGRITY AEAD ERROR, sector %llu", | |
| 1370 | + ctx->bio_in->bi_bdev, s); | |
| 1372 | 1371 | dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead", |
| 1373 | 1372 | ctx->bio_in, s, 0); |
| 1374 | 1373 | } |
| @@ -1672,11 +1671,10 @@ retry: · 297 unmodified lines | ||
| 1672 | 1671 | if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM)) |
| 1673 | 1672 | mutex_lock(&cc->bio_alloc_lock); |
| 1674 | 1673 | |
| 1675 | − clone = bio_alloc_bioset(GFP_NOIO, nr_iovecs, &cc->bs); | |
| 1676 | − if (!clone) | |
| 1677 | − goto out; | |
| 1678 | − | |
| 1679 | − clone_init(io, clone); | |
| 1674 | + clone = bio_alloc_bioset(cc->dev->bdev, nr_iovecs, io->base_bio->bi_opf, | |
| 1675 | + GFP_NOIO, &cc->bs); | |
| 1676 | + clone->bi_private = io; | |
| 1677 | + clone->bi_end_io = crypt_endio; | |
| 1680 | 1678 | |
| 1681 | 1679 | remaining_size = size; |
| 1682 | 1680 | |
| @@ -1702,7 +1700,7 @@ retry: · 19 unmodified lines | ||
| 1702 | 1700 | bio_put(clone); |
| 1703 | 1701 | clone = NULL; |
| 1704 | 1702 | } |
| 1705 | −out: | |
| 1703 | + | |
| 1706 | 1704 | if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM)) |
| 1707 | 1705 | mutex_unlock(&cc->bio_alloc_lock); |
| 1708 | 1706 | |
| @@ -1829,34 +1827,25 @@ static void crypt_endio(struct bio *clone) · 120 unmodified lines | ||
| 1829 | 1827 | crypt_dec_pending(io); |
| 1830 | 1828 | } |
| 1831 | 1829 | |
| 1832 | −static void clone_init(struct dm_crypt_io *io, struct bio *clone) | |
| 1833 | −{ | |
| 1834 | − struct crypt_config *cc = io->cc; | |
| 1835 | − | |
| 1836 | − clone->bi_private = io; | |
| 1837 | − clone->bi_end_io = crypt_endio; | |
| 1838 | − bio_set_dev(clone, cc->dev->bdev); | |
| 1839 | − clone->bi_opf = io->base_bio->bi_opf; | |
| 1840 | −} | |
| 1841 | − | |
| 1842 | 1830 | static int kcryptd_io_read(struct dm_crypt_io *io, gfp_t gfp) |
| 1843 | 1831 | { |
| 1844 | 1832 | struct crypt_config *cc = io->cc; |
| 1845 | 1833 | struct bio *clone; |
| 1846 | 1834 | |
| 1847 | 1835 | /* |
| 1848 | − * We need the original biovec array in order to decrypt | |
| 1849 | − * the whole bio data *afterwards* -- thanks to immutable | |
| 1850 | − * biovecs we don't need to worry about the block layer | |
| 1851 | − * modifying the biovec array; so leverage bio_clone_fast(). | |
| 1836 | + * We need the original biovec array in order to decrypt the whole bio | |
| 1837 | + * data *afterwards* -- thanks to immutable biovecs we don't need to | |
| 1838 | + * worry about the block layer modifying the biovec array; so leverage | |
| 1839 | + * bio_alloc_clone(). | |
| 1852 | 1840 | */ |
| 1853 | − clone = bio_clone_fast(io->base_bio, gfp, &cc->bs); | |
| 1841 | + clone = bio_alloc_clone(cc->dev->bdev, io->base_bio, gfp, &cc->bs); | |
| 1854 | 1842 | if (!clone) |
| 1855 | 1843 | return 1; |
| 1844 | + clone->bi_private = io; | |
| 1845 | + clone->bi_end_io = crypt_endio; | |
| 1856 | 1846 | |
| 1857 | 1847 | crypt_inc_pending(io); |
| 1858 | 1848 | |
| 1859 | − clone_init(io, clone); | |
| 1860 | 1849 | clone->bi_iter.bi_sector = cc->start + io->sector; |
| 1861 | 1850 | |
| 1862 | 1851 | if (dm_crypt_integrity_io_alloc(io, clone)) { |
| @@ -2179,11 +2168,10 @@ static void kcryptd_async_done(struct crypto_async_request *async_req, · 316 unmodified lines | ||
| 2179 | 2168 | error = cc->iv_gen_ops->post(cc, org_iv_of_dmreq(cc, dmreq), dmreq); |
| 2180 | 2169 | |
| 2181 | 2170 | if (error == -EBADMSG) { |
| 2182 | − char b[BDEVNAME_SIZE]; | |
| 2183 | 2171 | sector_t s = le64_to_cpu(*org_sector_of_dmreq(cc, dmreq)); |
| 2184 | 2172 | |
| 2185 | − DMERR_LIMIT("%s: INTEGRITY AEAD ERROR, sector %llu", | |
| 2186 | − bio_devname(ctx->bio_in, b), s); | |
| 2173 | + DMERR_LIMIT("%pg: INTEGRITY AEAD ERROR, sector %llu", | |
| 2174 | + ctx->bio_in->bi_bdev, s); | |
| 2187 | 2175 | dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead", |
| 2188 | 2176 | ctx->bio_in, s, 0); |
| 2189 | 2177 | io->error = BLK_STS_PROTECTION; |
| @@ -1788,12 +1788,11 @@ again: · 1787 unmodified lines | ||
| 1788 | 1788 | checksums_ptr - checksums, dio->op == REQ_OP_READ ? TAG_CMP : TAG_WRITE); |
| 1789 | 1789 | if (unlikely(r)) { |
| 1790 | 1790 | if (r > 0) { |
| 1791 | − char b[BDEVNAME_SIZE]; | |
| 1792 | 1791 | sector_t s; |
| 1793 | 1792 | |
| 1794 | 1793 | s = sector - ((r + ic->tag_size - 1) / ic->tag_size); |
| 1795 | − DMERR_LIMIT("%s: Checksum failed at sector 0x%llx", | |
| 1796 | − bio_devname(bio, b), s); | |
| 1794 | + DMERR_LIMIT("%pg: Checksum failed at sector 0x%llx", | |
| 1795 | + bio->bi_bdev, s); | |
| 1797 | 1796 | r = -EILSEQ; |
| 1798 | 1797 | atomic64_inc(&ic->number_of_mismatches); |
| 1799 | 1798 | dm_audit_log_bio(DM_MSG_PREFIX, "integrity-checksum", |
| @@ -345,11 +345,10 @@ static void do_region(int op, int op_flags, unsigned region, · 344 unmodified lines | ||
| 345 | 345 | (PAGE_SIZE >> SECTOR_SHIFT))); |
| 346 | 346 | } |
| 347 | 347 | |
| 348 | − bio = bio_alloc_bioset(GFP_NOIO, num_bvecs, &io->client->bios); | |
| 348 | + bio = bio_alloc_bioset(where->bdev, num_bvecs, op | op_flags, | |
| 349 | + GFP_NOIO, &io->client->bios); | |
| 349 | 350 | bio->bi_iter.bi_sector = where->sector + (where->count - remaining); |
| 350 | − bio_set_dev(bio, where->bdev); | |
| 351 | 351 | bio->bi_end_io = endio; |
| 352 | − bio_set_op_attrs(bio, op, op_flags); | |
| 353 | 352 | store_io_and_region_in_bio(bio, io, region); |
| 354 | 353 | |
| 355 | 354 | if (op == REQ_OP_DISCARD || op == REQ_OP_WRITE_ZEROES) { |
| @@ -217,18 +217,12 @@ static int write_metadata(struct log_writes_c *lc, void *entry, · 216 unmodified lines | ||
| 217 | 217 | void *ptr; |
| 218 | 218 | size_t ret; |
| 219 | 219 | |
| 220 | − bio = bio_alloc(GFP_KERNEL, 1); | |
| 221 | − if (!bio) { | |
| 222 | − DMERR("Couldn't alloc log bio"); | |
| 223 | − goto error; | |
| 224 | − } | |
| 220 | + bio = bio_alloc(lc->logdev->bdev, 1, REQ_OP_WRITE, GFP_KERNEL); | |
| 225 | 221 | bio->bi_iter.bi_size = 0; |
| 226 | 222 | bio->bi_iter.bi_sector = sector; |
| 227 | − bio_set_dev(bio, lc->logdev->bdev); | |
| 228 | 223 | bio->bi_end_io = (sector == WRITE_LOG_SUPER_SECTOR) ? |
| 229 | 224 | log_end_super : log_end_io; |
| 230 | 225 | bio->bi_private = lc; |
| 231 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 232 | 226 | |
| 233 | 227 | page = alloc_page(GFP_KERNEL); |
| 234 | 228 | if (!page) { |
| @@ -275,18 +269,12 @@ static int write_inline_data(struct log_writes_c *lc, void *entry, · 40 unmodified lines | ||
| 275 | 269 | |
| 276 | 270 | atomic_inc(&lc->io_blocks); |
| 277 | 271 | |
| 278 | − bio = bio_alloc(GFP_KERNEL, bio_pages); | |
| 279 | − if (!bio) { | |
| 280 | − DMERR("Couldn't alloc inline data bio"); | |
| 281 | − goto error; | |
| 282 | − } | |
| 283 | − | |
| 272 | + bio = bio_alloc(lc->logdev->bdev, bio_pages, REQ_OP_WRITE, | |
| 273 | + GFP_KERNEL); | |
| 284 | 274 | bio->bi_iter.bi_size = 0; |
| 285 | 275 | bio->bi_iter.bi_sector = sector; |
| 286 | − bio_set_dev(bio, lc->logdev->bdev); | |
| 287 | 276 | bio->bi_end_io = log_end_io; |
| 288 | 277 | bio->bi_private = lc; |
| 289 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 290 | 278 | |
| 291 | 279 | for (i = 0; i < bio_pages; i++) { |
| 292 | 280 | pg_datalen = min_t(int, datalen, PAGE_SIZE); |
| @@ -322,7 +310,6 @@ static int write_inline_data(struct log_writes_c *lc, void *entry, · 29 unmodified lines | ||
| 322 | 310 | error_bio: |
| 323 | 311 | bio_free_pages(bio); |
| 324 | 312 | bio_put(bio); |
| 325 | −error: | |
| 326 | 313 | put_io_block(lc); |
| 327 | 314 | return -1; |
| 328 | 315 | } |
| @@ -363,17 +350,12 @@ static int log_one_block(struct log_writes_c *lc, · 34 unmodified lines | ||
| 363 | 350 | goto out; |
| 364 | 351 | |
| 365 | 352 | atomic_inc(&lc->io_blocks); |
| 366 | − bio = bio_alloc(GFP_KERNEL, bio_max_segs(block->vec_cnt)); | |
| 367 | − if (!bio) { | |
| 368 | − DMERR("Couldn't alloc log bio"); | |
| 369 | − goto error; | |
| 370 | − } | |
| 353 | + bio = bio_alloc(lc->logdev->bdev, bio_max_segs(block->vec_cnt), | |
| 354 | + REQ_OP_WRITE, GFP_KERNEL); | |
| 371 | 355 | bio->bi_iter.bi_size = 0; |
| 372 | 356 | bio->bi_iter.bi_sector = sector; |
| 373 | − bio_set_dev(bio, lc->logdev->bdev); | |
| 374 | 357 | bio->bi_end_io = log_end_io; |
| 375 | 358 | bio->bi_private = lc; |
| 376 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 377 | 359 | |
| 378 | 360 | for (i = 0; i < block->vec_cnt; i++) { |
| 379 | 361 | /* |
| @@ -385,18 +367,13 @@ static int log_one_block(struct log_writes_c *lc, · 5 unmodified lines | ||
| 385 | 367 | if (ret != block->vecs[i].bv_len) { |
| 386 | 368 | atomic_inc(&lc->io_blocks); |
| 387 | 369 | submit_bio(bio); |
| 388 | − bio = bio_alloc(GFP_KERNEL, | |
| 389 | − bio_max_segs(block->vec_cnt - i)); | |
| 390 | − if (!bio) { | |
| 391 | − DMERR("Couldn't alloc log bio"); | |
| 392 | − goto error; | |
| 393 | − } | |
| 370 | + bio = bio_alloc(lc->logdev->bdev, | |
| 371 | + bio_max_segs(block->vec_cnt - i), | |
| 372 | + REQ_OP_WRITE, GFP_KERNEL); | |
| 394 | 373 | bio->bi_iter.bi_size = 0; |
| 395 | 374 | bio->bi_iter.bi_sector = sector; |
| 396 | − bio_set_dev(bio, lc->logdev->bdev); | |
| 397 | 375 | bio->bi_end_io = log_end_io; |
| 398 | 376 | bio->bi_private = lc; |
| 399 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 400 | 377 | |
| 401 | 378 | ret = bio_add_page(bio, block->vecs[i].bv_page, |
| 402 | 379 | block->vecs[i].bv_len, 0); |
| @@ -303,21 +303,6 @@ static void end_clone_request(struct request *clone, blk_status_t error) · 302 unmodified lines | ||
| 303 | 303 | dm_complete_request(tio->orig, error); |
| 304 | 304 | } |
| 305 | 305 | |
| 306 | −static blk_status_t dm_dispatch_clone_request(struct request *clone, struct request *rq) | |
| 307 | −{ | |
| 308 | − blk_status_t r; | |
| 309 | − | |
| 310 | − if (blk_queue_io_stat(clone->q)) | |
| 311 | − clone->rq_flags |= RQF_IO_STAT; | |
| 312 | − | |
| 313 | − clone->start_time_ns = ktime_get_ns(); | |
| 314 | − r = blk_insert_cloned_request(clone->q, clone); | |
| 315 | − if (r != BLK_STS_OK && r != BLK_STS_RESOURCE && r != BLK_STS_DEV_RESOURCE) | |
| 316 | − /* must complete clone in terms of original request */ | |
| 317 | − dm_complete_request(rq, r); | |
| 318 | − return r; | |
| 319 | −} | |
| 320 | − | |
| 321 | 306 | static int dm_rq_bio_constructor(struct bio *bio, struct bio *bio_orig, |
| 322 | 307 | void *data) |
| 323 | 308 | { |
| @@ -398,13 +383,20 @@ static int map_request(struct dm_rq_target_io *tio) · 74 unmodified lines | ||
| 398 | 383 | /* The target has remapped the I/O so dispatch it */ |
| 399 | 384 | trace_block_rq_remap(clone, disk_devt(dm_disk(md)), |
| 400 | 385 | blk_rq_pos(rq)); |
| 401 | − ret = dm_dispatch_clone_request(clone, rq); | |
| 402 | − if (ret == BLK_STS_RESOURCE || ret == BLK_STS_DEV_RESOURCE) { | |
| 386 | + ret = blk_insert_cloned_request(clone); | |
| 387 | + switch (ret) { | |
| 388 | + case BLK_STS_OK: | |
| 389 | + break; | |
| 390 | + case BLK_STS_RESOURCE: | |
| 391 | + case BLK_STS_DEV_RESOURCE: | |
| 403 | 392 | blk_rq_unprep_clone(clone); |
| 404 | 393 | blk_mq_cleanup_rq(clone); |
| 405 | 394 | tio->ti->type->release_clone_rq(clone, &tio->info); |
| 406 | 395 | tio->clone = NULL; |
| 407 | 396 | return DM_MAPIO_REQUEUE; |
| 397 | + default: | |
| 398 | + /* must complete clone in terms of original request */ | |
| 399 | + dm_complete_request(rq, ret); | |
| 408 | 400 | } |
| 409 | 401 | break; |
| 410 | 402 | case DM_MAPIO_REQUEUE: |
| @@ -141,11 +141,6 @@ struct dm_snapshot { · 140 unmodified lines | ||
| 141 | 141 | * for them to be committed. |
| 142 | 142 | */ |
| 143 | 143 | struct bio_list bios_queued_during_merge; |
| 144 | − | |
| 145 | − /* | |
| 146 | − * Flush data after merge. | |
| 147 | − */ | |
| 148 | − struct bio flush_bio; | |
| 149 | 144 | }; |
| 150 | 145 | |
| 151 | 146 | /* |
| @@ -1127,17 +1122,6 @@ shut: · 975 unmodified lines | ||
| 1127 | 1122 | |
| 1128 | 1123 | static void error_bios(struct bio *bio); |
| 1129 | 1124 | |
| 1130 | −static int flush_data(struct dm_snapshot *s) | |
| 1131 | −{ | |
| 1132 | − struct bio *flush_bio = &s->flush_bio; | |
| 1133 | − | |
| 1134 | − bio_reset(flush_bio); | |
| 1135 | − bio_set_dev(flush_bio, s->origin->bdev); | |
| 1136 | − flush_bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 1137 | − | |
| 1138 | − return submit_bio_wait(flush_bio); | |
| 1139 | −} | |
| 1140 | − | |
| 1141 | 1125 | static void merge_callback(int read_err, unsigned long write_err, void *context) |
| 1142 | 1126 | { |
| 1143 | 1127 | struct dm_snapshot *s = context; |
| @@ -1151,7 +1135,7 @@ static void merge_callback(int read_err, unsigned long write_err, void *context) · 7 unmodified lines | ||
| 1151 | 1135 | goto shut; |
| 1152 | 1136 | } |
| 1153 | 1137 | |
| 1154 | − if (flush_data(s) < 0) { | |
| 1138 | + if (blkdev_issue_flush(s->origin->bdev) < 0) { | |
| 1155 | 1139 | DMERR("Flush after merge failed: shutting down merge"); |
| 1156 | 1140 | goto shut; |
| 1157 | 1141 | } |
| @@ -1340,7 +1324,6 @@ static int snapshot_ctr(struct dm_target *ti, unsigned int argc, char **argv) · 182 unmodified lines | ||
| 1340 | 1324 | s->first_merging_chunk = 0; |
| 1341 | 1325 | s->num_merging_chunks = 0; |
| 1342 | 1326 | bio_list_init(&s->bios_queued_during_merge); |
| 1343 | − bio_init(&s->flush_bio, NULL, 0); | |
| 1344 | 1327 | |
| 1345 | 1328 | /* Allocate hash table for COW data */ |
| 1346 | 1329 | if (init_hash_tables(s)) { |
| @@ -1528,8 +1511,6 @@ static void snapshot_dtr(struct dm_target *ti) · 181 unmodified lines | ||
| 1528 | 1511 | |
| 1529 | 1512 | dm_exception_store_destroy(s->store); |
| 1530 | 1513 | |
| 1531 | − bio_uninit(&s->flush_bio); | |
| 1532 | − | |
| 1533 | 1514 | dm_put_device(ti, s->cow); |
| 1534 | 1515 | |
| 1535 | 1516 | dm_put_device(ti, s->origin); |
| @@ -282,8 +282,6 @@ struct pool { · 281 unmodified lines | ||
| 282 | 282 | struct dm_bio_prison_cell **cell_sort_array; |
| 283 | 283 | |
| 284 | 284 | mempool_t mapping_pool; |
| 285 | − | |
| 286 | − struct bio flush_bio; | |
| 287 | 285 | }; |
| 288 | 286 | |
| 289 | 287 | static void metadata_operation_failed(struct pool *pool, const char *op, int r); |
| @@ -1179,25 +1177,17 @@ static void process_prepared_discard_passdown_pt1(struct dm_thin_new_mapping *m) · 889 unmodified lines | ||
| 1179 | 1177 | return; |
| 1180 | 1178 | } |
| 1181 | 1179 | |
| 1182 | − discard_parent = bio_alloc(GFP_NOIO, 1); | |
| 1183 | − if (!discard_parent) { | |
| 1184 | − DMWARN("%s: unable to allocate top level discard bio for passdown. Skipping passdown.", | |
| 1185 | − dm_device_name(tc->pool->pool_md)); | |
| 1186 | − queue_passdown_pt2(m); | |
| 1187 | − | |
| 1188 | − } else { | |
| 1189 | − discard_parent->bi_end_io = passdown_endio; | |
| 1190 | − discard_parent->bi_private = m; | |
| 1191 | − | |
| 1192 | − if (m->maybe_shared) | |
| 1193 | − passdown_double_checking_shared_status(m, discard_parent); | |
| 1194 | − else { | |
| 1195 | − struct discard_op op; | |
| 1196 | − | |
| 1197 | − begin_discard(&op, tc, discard_parent); | |
| 1198 | − r = issue_discard(&op, m->data_block, data_end); | |
| 1199 | − end_discard(&op, r); | |
| 1200 | − } | |
| 1180 | + discard_parent = bio_alloc(NULL, 1, 0, GFP_NOIO); | |
| 1181 | + discard_parent->bi_end_io = passdown_endio; | |
| 1182 | + discard_parent->bi_private = m; | |
| 1183 | + if (m->maybe_shared) | |
| 1184 | + passdown_double_checking_shared_status(m, discard_parent); | |
| 1185 | + else { | |
| 1186 | + struct discard_op op; | |
| 1187 | + | |
| 1188 | + begin_discard(&op, tc, discard_parent); | |
| 1189 | + r = issue_discard(&op, m->data_block, data_end); | |
| 1190 | + end_discard(&op, r); | |
| 1201 | 1191 | } |
| 1202 | 1192 | } |
| 1203 | 1193 | |
| @@ -2913,7 +2903,6 @@ static void __pool_destroy(struct pool *pool) · 1709 unmodified lines | ||
| 2913 | 2903 | if (pool->next_mapping) |
| 2914 | 2904 | mempool_free(pool->next_mapping, &pool->mapping_pool); |
| 2915 | 2905 | mempool_exit(&pool->mapping_pool); |
| 2916 | − bio_uninit(&pool->flush_bio); | |
| 2917 | 2906 | dm_deferred_set_destroy(pool->shared_read_ds); |
| 2918 | 2907 | dm_deferred_set_destroy(pool->all_io_ds); |
| 2919 | 2908 | kfree(pool); |
| @@ -2994,7 +2983,6 @@ static struct pool *pool_create(struct mapped_device *pool_md, · 74 unmodified lines | ||
| 2994 | 2983 | pool->low_water_triggered = false; |
| 2995 | 2984 | pool->suspended = true; |
| 2996 | 2985 | pool->out_of_data_space = false; |
| 2997 | − bio_init(&pool->flush_bio, NULL, 0); | |
| 2998 | 2986 | |
| 2999 | 2987 | pool->shared_read_ds = dm_deferred_set_create(); |
| 3000 | 2988 | if (!pool->shared_read_ds) { |
| @@ -3201,13 +3189,8 @@ static void metadata_low_callback(void *context) · 200 unmodified lines | ||
| 3201 | 3189 | static int metadata_pre_commit_callback(void *context) |
| 3202 | 3190 | { |
| 3203 | 3191 | struct pool *pool = context; |
| 3204 | − struct bio *flush_bio = &pool->flush_bio; | |
| 3205 | − | |
| 3206 | − bio_reset(flush_bio); | |
| 3207 | − bio_set_dev(flush_bio, pool->data_dev); | |
| 3208 | − flush_bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 3209 | 3192 | |
| 3210 | − return submit_bio_wait(flush_bio); | |
| 3193 | + return blkdev_issue_flush(pool->data_dev); | |
| 3211 | 3194 | } |
| 3212 | 3195 | |
| 3213 | 3196 | static sector_t get_dev_size(struct block_device *bdev) |
| @@ -1821,11 +1821,11 @@ static void __writecache_writeback_pmem(struct dm_writecache *wc, struct writeba · 1820 unmodified lines | ||
| 1821 | 1821 | |
| 1822 | 1822 | max_pages = e->wc_list_contiguous; |
| 1823 | 1823 | |
| 1824 | − bio = bio_alloc_bioset(GFP_NOIO, max_pages, &wc->bio_set); | |
| 1824 | + bio = bio_alloc_bioset(wc->dev->bdev, max_pages, REQ_OP_WRITE, | |
| 1825 | + GFP_NOIO, &wc->bio_set); | |
| 1825 | 1826 | wb = container_of(bio, struct writeback_struct, bio); |
| 1826 | 1827 | wb->wc = wc; |
| 1827 | 1828 | bio->bi_end_io = writecache_writeback_endio; |
| 1828 | − bio_set_dev(bio, wc->dev->bdev); | |
| 1829 | 1829 | bio->bi_iter.bi_sector = read_original_sector(wc, e); |
| 1830 | 1830 | if (max_pages <= WB_LIST_INLINE || |
| 1831 | 1831 | unlikely(!(wb->wc_list = kmalloc_array(max_pages, sizeof(struct wc_entry *), |
| @@ -1852,7 +1852,8 @@ static void __writecache_writeback_pmem(struct dm_writecache *wc, struct writeba · 20 unmodified lines | ||
| 1852 | 1852 | wb->wc_list[wb->wc_list_n++] = f; |
| 1853 | 1853 | e = f; |
| 1854 | 1854 | } |
| 1855 | − bio_set_op_attrs(bio, REQ_OP_WRITE, WC_MODE_FUA(wc) * REQ_FUA); | |
| 1855 | + if (WC_MODE_FUA(wc)) | |
| 1856 | + bio->bi_opf |= REQ_FUA; | |
| 1856 | 1857 | if (writecache_has_error(wc)) { |
| 1857 | 1858 | bio->bi_status = BLK_STS_IOERR; |
| 1858 | 1859 | bio_endio(bio); |
| @@ -550,11 +550,8 @@ static struct dmz_mblock *dmz_get_mblock_slow(struct dmz_metadata *zmd, · 549 unmodified lines | ||
| 550 | 550 | if (!mblk) |
| 551 | 551 | return ERR_PTR(-ENOMEM); |
| 552 | 552 | |
| 553 | − bio = bio_alloc(GFP_NOIO, 1); | |
| 554 | − if (!bio) { | |
| 555 | − dmz_free_mblock(zmd, mblk); | |
| 556 | − return ERR_PTR(-ENOMEM); | |
| 557 | − } | |
| 553 | + bio = bio_alloc(dev->bdev, 1, REQ_OP_READ | REQ_META | REQ_PRIO, | |
| 554 | + GFP_NOIO); | |
| 558 | 555 | |
| 559 | 556 | spin_lock(&zmd->mblk_lock); |
| 560 | 557 | |
| @@ -578,10 +575,8 @@ static struct dmz_mblock *dmz_get_mblock_slow(struct dmz_metadata *zmd, · 17 unmodified lines | ||
| 578 | 575 | |
| 579 | 576 | /* Submit read BIO */ |
| 580 | 577 | bio->bi_iter.bi_sector = dmz_blk2sect(block); |
| 581 | − bio_set_dev(bio, dev->bdev); | |
| 582 | 578 | bio->bi_private = mblk; |
| 583 | 579 | bio->bi_end_io = dmz_mblock_bio_end_io; |
| 584 | − bio_set_op_attrs(bio, REQ_OP_READ, REQ_META | REQ_PRIO); | |
| 585 | 580 | bio_add_page(bio, mblk->page, DMZ_BLOCK_SIZE, 0); |
| 586 | 581 | submit_bio(bio); |
| 587 | 582 | |
| @@ -725,19 +720,14 @@ static int dmz_write_mblock(struct dmz_metadata *zmd, struct dmz_mblock *mblk, · 137 unmodified lines | ||
| 725 | 720 | if (dmz_bdev_is_dying(dev)) |
| 726 | 721 | return -EIO; |
| 727 | 722 | |
| 728 | − bio = bio_alloc(GFP_NOIO, 1); | |
| 729 | − if (!bio) { | |
| 730 | − set_bit(DMZ_META_ERROR, &mblk->state); | |
| 731 | − return -ENOMEM; | |
| 732 | − } | |
| 723 | + bio = bio_alloc(dev->bdev, 1, REQ_OP_WRITE | REQ_META | REQ_PRIO, | |
| 724 | + GFP_NOIO); | |
| 733 | 725 | |
| 734 | 726 | set_bit(DMZ_META_WRITING, &mblk->state); |
| 735 | 727 | |
| 736 | 728 | bio->bi_iter.bi_sector = dmz_blk2sect(block); |
| 737 | − bio_set_dev(bio, dev->bdev); | |
| 738 | 729 | bio->bi_private = mblk; |
| 739 | 730 | bio->bi_end_io = dmz_mblock_bio_end_io; |
| 740 | − bio_set_op_attrs(bio, REQ_OP_WRITE, REQ_META | REQ_PRIO); | |
| 741 | 731 | bio_add_page(bio, mblk->page, DMZ_BLOCK_SIZE, 0); |
| 742 | 732 | submit_bio(bio); |
| 743 | 733 | |
| @@ -759,13 +749,9 @@ static int dmz_rdwr_block(struct dmz_dev *dev, int op, · 15 unmodified lines | ||
| 759 | 749 | if (dmz_bdev_is_dying(dev)) |
| 760 | 750 | return -EIO; |
| 761 | 751 | |
| 762 | − bio = bio_alloc(GFP_NOIO, 1); | |
| 763 | − if (!bio) | |
| 764 | − return -ENOMEM; | |
| 765 | − | |
| 752 | + bio = bio_alloc(dev->bdev, 1, op | REQ_SYNC | REQ_META | REQ_PRIO, | |
| 753 | + GFP_NOIO); | |
| 766 | 754 | bio->bi_iter.bi_sector = dmz_blk2sect(block); |
| 767 | − bio_set_dev(bio, dev->bdev); | |
| 768 | − bio_set_op_attrs(bio, op, REQ_SYNC | REQ_META | REQ_PRIO); | |
| 769 | 755 | bio_add_page(bio, page, DMZ_BLOCK_SIZE, 0); |
| 770 | 756 | ret = submit_bio_wait(bio); |
| 771 | 757 | bio_put(bio); |
| @@ -125,11 +125,10 @@ static int dmz_submit_bio(struct dmz_target *dmz, struct dm_zone *zone, · 124 unmodified lines | ||
| 125 | 125 | if (dev->flags & DMZ_BDEV_DYING) |
| 126 | 126 | return -EIO; |
| 127 | 127 | |
| 128 | − clone = bio_clone_fast(bio, GFP_NOIO, &dmz->bio_set); | |
| 128 | + clone = bio_alloc_clone(dev->bdev, bio, GFP_NOIO, &dmz->bio_set); | |
| 129 | 129 | if (!clone) |
| 130 | 130 | return -ENOMEM; |
| 131 | 131 | |
| 132 | − bio_set_dev(clone, dev->bdev); | |
| 133 | 132 | bioctx->dev = dev; |
| 134 | 133 | clone->bi_iter.bi_sector = |
| 135 | 134 | dmz_start_sect(dmz->metadata, zone) + dmz_blk2sect(chunk_block); |
| @@ -79,10 +79,14 @@ struct clone_info { · 78 unmodified lines | ||
| 79 | 79 | #define DM_IO_BIO_OFFSET \ |
| 80 | 80 | (offsetof(struct dm_target_io, clone) + offsetof(struct dm_io, tio)) |
| 81 | 81 | |
| 82 | +static inline struct dm_target_io *clone_to_tio(struct bio *clone) | |
| 83 | +{ | |
| 84 | + return container_of(clone, struct dm_target_io, clone); | |
| 85 | +} | |
| 86 | + | |
| 82 | 87 | void *dm_per_bio_data(struct bio *bio, size_t data_size) |
| 83 | 88 | { |
| 84 | − struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone); | |
| 85 | − if (!tio->inside_dm_io) | |
| 89 | + if (!clone_to_tio(bio)->inside_dm_io) | |
| 86 | 90 | return (char *)bio - DM_TARGET_IO_BIO_OFFSET - data_size; |
| 87 | 91 | return (char *)bio - DM_IO_BIO_OFFSET - data_size; |
| 88 | 92 | } |
| @@ -477,10 +481,7 @@ out: · 388 unmodified lines | ||
| 477 | 481 | |
| 478 | 482 | u64 dm_start_time_ns_from_clone(struct bio *bio) |
| 479 | 483 | { |
| 480 | − struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone); | |
| 481 | − struct dm_io *io = tio->io; | |
| 482 | − | |
| 483 | − return jiffies_to_nsecs(io->start_time); | |
| 484 | + return jiffies_to_nsecs(clone_to_tio(bio)->io->start_time); | |
| 484 | 485 | } |
| 485 | 486 | EXPORT_SYMBOL_GPL(dm_start_time_ns_from_clone); |
| 486 | 487 | |
| @@ -519,11 +520,9 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio) · 32 unmodified lines | ||
| 519 | 520 | struct dm_target_io *tio; |
| 520 | 521 | struct bio *clone; |
| 521 | 522 | |
| 522 | − clone = bio_alloc_bioset(GFP_NOIO, 0, &md->io_bs); | |
| 523 | − if (!clone) | |
| 524 | − return NULL; | |
| 523 | + clone = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOIO, &md->io_bs); | |
| 525 | 524 | |
| 526 | − tio = container_of(clone, struct dm_target_io, clone); | |
| 525 | + tio = clone_to_tio(clone); | |
| 527 | 526 | tio->inside_dm_io = true; |
| 528 | 527 | tio->io = NULL; |
| 529 | 528 | |
| @@ -545,8 +544,8 @@ static void free_io(struct mapped_device *md, struct dm_io *io) · 15 unmodified lines | ||
| 545 | 544 | bio_put(&io->tio.clone); |
| 546 | 545 | } |
| 547 | 546 | |
| 548 | −static struct dm_target_io *alloc_tio(struct clone_info *ci, struct dm_target *ti, | |
| 549 | − unsigned target_bio_nr, gfp_t gfp_mask) | |
| 547 | +static struct bio *alloc_tio(struct clone_info *ci, struct dm_target *ti, | |
| 548 | + unsigned target_bio_nr, unsigned *len, gfp_t gfp_mask) | |
| 550 | 549 | { |
| 551 | 550 | struct dm_target_io *tio; |
| 552 | 551 | |
| @@ -554,11 +553,12 @@ static struct dm_target_io *alloc_tio(struct clone_info *ci, struct dm_target *t · 1 unmodified line | ||
| 554 | 553 | /* the dm_target_io embedded in ci->io is available */ |
| 555 | 554 | tio = &ci->io->tio; |
| 556 | 555 | } else { |
| 557 | − struct bio *clone = bio_alloc_bioset(gfp_mask, 0, &ci->io->md->bs); | |
| 556 | + struct bio *clone = bio_alloc_clone(ci->bio->bi_bdev, ci->bio, | |
| 557 | + gfp_mask, &ci->io->md->bs); | |
| 558 | 558 | if (!clone) |
| 559 | 559 | return NULL; |
| 560 | 560 | |
| 561 | − tio = container_of(clone, struct dm_target_io, clone); | |
| 561 | + tio = clone_to_tio(clone); | |
| 562 | 562 | tio->inside_dm_io = false; |
| 563 | 563 | } |
| 564 | 564 | |
| @@ -566,15 +566,16 @@ static struct dm_target_io *alloc_tio(struct clone_info *ci, struct dm_target *t · 1 unmodified line | ||
| 566 | 566 | tio->io = ci->io; |
| 567 | 567 | tio->ti = ti; |
| 568 | 568 | tio->target_bio_nr = target_bio_nr; |
| 569 | + tio->len_ptr = len; | |
| 569 | 570 | |
| 570 | − return tio; | |
| 571 | + return &tio->clone; | |
| 571 | 572 | } |
| 572 | 573 | |
| 573 | −static void free_tio(struct dm_target_io *tio) | |
| 574 | +static void free_tio(struct bio *clone) | |
| 574 | 575 | { |
| 575 | − if (tio->inside_dm_io) | |
| 576 | + if (clone_to_tio(clone)->inside_dm_io) | |
| 576 | 577 | return; |
| 577 | − bio_put(&tio->clone); | |
| 578 | + bio_put(clone); | |
| 578 | 579 | } |
| 579 | 580 | |
| 580 | 581 | /* |
| @@ -879,7 +880,7 @@ static bool swap_bios_limit(struct dm_target *ti, struct bio *bio) · 298 unmodified lines | ||
| 879 | 880 | static void clone_endio(struct bio *bio) |
| 880 | 881 | { |
| 881 | 882 | blk_status_t error = bio->bi_status; |
| 882 | − struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone); | |
| 883 | + struct dm_target_io *tio = clone_to_tio(bio); | |
| 883 | 884 | struct dm_io *io = tio->io; |
| 884 | 885 | struct mapped_device *md = tio->io->md; |
| 885 | 886 | dm_endio_fn endio = tio->ti->type->end_io; |
| @@ -930,7 +931,7 @@ static void clone_endio(struct bio *bio) · 44 unmodified lines | ||
| 930 | 931 | up(&md->swap_bios_semaphore); |
| 931 | 932 | } |
| 932 | 933 | |
| 933 | − free_tio(tio); | |
| 934 | + free_tio(bio); | |
| 934 | 935 | dm_io_dec_pending(io, error); |
| 935 | 936 | } |
| 936 | 937 | |
| @@ -1085,7 +1086,7 @@ static int dm_dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff, · 148 unmodified lines | ||
| 1085 | 1086 | */ |
| 1086 | 1087 | void dm_accept_partial_bio(struct bio *bio, unsigned n_sectors) |
| 1087 | 1088 | { |
| 1088 | − struct dm_target_io *tio = container_of(bio, struct dm_target_io, clone); | |
| 1089 | + struct dm_target_io *tio = clone_to_tio(bio); | |
| 1089 | 1090 | unsigned bi_size = bio->bi_iter.bi_size >> SECTOR_SHIFT; |
| 1090 | 1091 | |
| 1091 | 1092 | BUG_ON(bio->bi_opf & REQ_PREFLUSH); |
| @@ -1115,11 +1116,11 @@ static noinline void __set_swap_bios_limit(struct mapped_device *md, int latch) · 23 unmodified lines | ||
| 1115 | 1116 | mutex_unlock(&md->swap_bios_lock); |
| 1116 | 1117 | } |
| 1117 | 1118 | |
| 1118 | −static void __map_bio(struct dm_target_io *tio) | |
| 1119 | +static void __map_bio(struct bio *clone) | |
| 1119 | 1120 | { |
| 1121 | + struct dm_target_io *tio = clone_to_tio(clone); | |
| 1120 | 1122 | int r; |
| 1121 | 1123 | sector_t sector; |
| 1122 | − struct bio *clone = &tio->clone; | |
| 1123 | 1124 | struct dm_io *io = tio->io; |
| 1124 | 1125 | struct dm_target *ti = tio->ti; |
| 1125 | 1126 | |
| @@ -1164,7 +1165,7 @@ static void __map_bio(struct dm_target_io *tio) · 38 unmodified lines | ||
| 1164 | 1165 | struct mapped_device *md = io->md; |
| 1165 | 1166 | up(&md->swap_bios_semaphore); |
| 1166 | 1167 | } |
| 1167 | − free_tio(tio); | |
| 1168 | + free_tio(clone); | |
| 1168 | 1169 | dm_io_dec_pending(io, BLK_STS_IOERR); |
| 1169 | 1170 | break; |
| 1170 | 1171 | case DM_MAPIO_REQUEUE: |
| @@ -1172,7 +1173,7 @@ static void __map_bio(struct dm_target_io *tio) · 1 unmodified line | ||
| 1172 | 1173 | struct mapped_device *md = io->md; |
| 1173 | 1174 | up(&md->swap_bios_semaphore); |
| 1174 | 1175 | } |
| 1175 | − free_tio(tio); | |
| 1176 | + free_tio(clone); | |
| 1176 | 1177 | dm_io_dec_pending(io, BLK_STS_DM_REQUEUE); |
| 1177 | 1178 | break; |
| 1178 | 1179 | default: |
| @@ -1190,106 +1191,75 @@ static void bio_setup_sector(struct bio *bio, sector_t sector, unsigned len) · 11 unmodified lines | ||
| 1190 | 1191 | /* |
| 1191 | 1192 | * Creates a bio that consists of range of complete bvecs. |
| 1192 | 1193 | */ |
| 1193 | −static int clone_bio(struct dm_target_io *tio, struct bio *bio, | |
| 1194 | − sector_t sector, unsigned len) | |
| 1194 | +static int __clone_and_map_data_bio(struct clone_info *ci, struct dm_target *ti, | |
| 1195 | + sector_t sector, unsigned *len) | |
| 1195 | 1196 | { |
| 1196 | − struct bio *clone = &tio->clone; | |
| 1197 | − int r; | |
| 1198 | − | |
| 1199 | − __bio_clone_fast(clone, bio); | |
| 1200 | − | |
| 1201 | − r = bio_crypt_clone(clone, bio, GFP_NOIO); | |
| 1202 | − if (r < 0) | |
| 1203 | − return r; | |
| 1204 | − | |
| 1205 | − if (bio_integrity(bio)) { | |
| 1206 | − if (unlikely(!dm_target_has_integrity(tio->ti->type) && | |
| 1207 | − !dm_target_passes_integrity(tio->ti->type))) { | |
| 1208 | − DMWARN("%s: the target %s doesn't support integrity data.", | |
| 1209 | − dm_device_name(tio->io->md), | |
| 1210 | − tio->ti->type->name); | |
| 1211 | − return -EIO; | |
| 1212 | − } | |
| 1213 | − | |
| 1214 | − r = bio_integrity_clone(clone, bio, GFP_NOIO); | |
| 1215 | − if (r < 0) | |
| 1216 | − return r; | |
| 1217 | − } | |
| 1197 | + struct bio *bio = ci->bio, *clone; | |
| 1218 | 1198 | |
| 1199 | + clone = alloc_tio(ci, ti, 0, len, GFP_NOIO); | |
| 1219 | 1200 | bio_advance(clone, to_bytes(sector - clone->bi_iter.bi_sector)); |
| 1220 | − clone->bi_iter.bi_size = to_bytes(len); | |
| 1201 | + clone->bi_iter.bi_size = to_bytes(*len); | |
| 1221 | 1202 | |
| 1222 | 1203 | if (bio_integrity(bio)) |
| 1223 | 1204 | bio_integrity_trim(clone); |
| 1224 | 1205 | |
| 1206 | + __map_bio(clone); | |
| 1225 | 1207 | return 0; |
| 1226 | 1208 | } |
| 1227 | 1209 | |
| 1228 | 1210 | static void alloc_multiple_bios(struct bio_list *blist, struct clone_info *ci, |
| 1229 | − struct dm_target *ti, unsigned num_bios) | |
| 1211 | + struct dm_target *ti, unsigned num_bios, | |
| 1212 | + unsigned *len) | |
| 1230 | 1213 | { |
| 1231 | − struct dm_target_io *tio; | |
| 1214 | + struct bio *bio; | |
| 1232 | 1215 | int try; |
| 1233 | 1216 | |
| 1234 | − if (!num_bios) | |
| 1235 | − return; | |
| 1236 | − | |
| 1237 | − if (num_bios == 1) { | |
| 1238 | − tio = alloc_tio(ci, ti, 0, GFP_NOIO); | |
| 1239 | − bio_list_add(blist, &tio->clone); | |
| 1240 | − return; | |
| 1241 | − } | |
| 1242 | − | |
| 1243 | 1217 | for (try = 0; try < 2; try++) { |
| 1244 | 1218 | int bio_nr; |
| 1245 | − struct bio *bio; | |
| 1246 | 1219 | |
| 1247 | 1220 | if (try) |
| 1248 | 1221 | mutex_lock(&ci->io->md->table_devices_lock); |
| 1249 | 1222 | for (bio_nr = 0; bio_nr < num_bios; bio_nr++) { |
| 1250 | − tio = alloc_tio(ci, ti, bio_nr, try ? GFP_NOIO : GFP_NOWAIT); | |
| 1251 | − if (!tio) | |
| 1223 | + bio = alloc_tio(ci, ti, bio_nr, len, | |
| 1224 | + try ? GFP_NOIO : GFP_NOWAIT); | |
| 1225 | + if (!bio) | |
| 1252 | 1226 | break; |
| 1253 | 1227 | |
| 1254 | − bio_list_add(blist, &tio->clone); | |
| 1228 | + bio_list_add(blist, bio); | |
| 1255 | 1229 | } |
| 1256 | 1230 | if (try) |
| 1257 | 1231 | mutex_unlock(&ci->io->md->table_devices_lock); |
| 1258 | 1232 | if (bio_nr == num_bios) |
| 1259 | 1233 | return; |
| 1260 | 1234 | |
| 1261 | − while ((bio = bio_list_pop(blist))) { | |
| 1262 | − tio = container_of(bio, struct dm_target_io, clone); | |
| 1263 | − free_tio(tio); | |
| 1264 | − } | |
| 1235 | + while ((bio = bio_list_pop(blist))) | |
| 1236 | + free_tio(bio); | |
| 1265 | 1237 | } |
| 1266 | 1238 | } |
| 1267 | 1239 | |
| 1268 | −static void __clone_and_map_simple_bio(struct clone_info *ci, | |
| 1269 | − struct dm_target_io *tio, unsigned *len) | |
| 1270 | −{ | |
| 1271 | − struct bio *clone = &tio->clone; | |
| 1272 | − | |
| 1273 | − tio->len_ptr = len; | |
| 1274 | − | |
| 1275 | − __bio_clone_fast(clone, ci->bio); | |
| 1276 | − if (len) | |
| 1277 | − bio_setup_sector(clone, ci->sector, *len); | |
| 1278 | − __map_bio(tio); | |
| 1279 | −} | |
| 1280 | − | |
| 1281 | 1240 | static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti, |
| 1282 | 1241 | unsigned num_bios, unsigned *len) |
| 1283 | 1242 | { |
| 1284 | 1243 | struct bio_list blist = BIO_EMPTY_LIST; |
| 1285 | − struct bio *bio; | |
| 1286 | − struct dm_target_io *tio; | |
| 1287 | − | |
| 1288 | − alloc_multiple_bios(&blist, ci, ti, num_bios); | |
| 1244 | + struct bio *clone; | |
| 1289 | 1245 | |
| 1290 | − while ((bio = bio_list_pop(&blist))) { | |
| 1291 | − tio = container_of(bio, struct dm_target_io, clone); | |
| 1292 | − __clone_and_map_simple_bio(ci, tio, len); | |
| 1246 | + switch (num_bios) { | |
| 1247 | + case 0: | |
| 1248 | + break; | |
| 1249 | + case 1: | |
| 1250 | + clone = alloc_tio(ci, ti, 0, len, GFP_NOIO); | |
| 1251 | + if (len) | |
| 1252 | + bio_setup_sector(clone, ci->sector, *len); | |
| 1253 | + __map_bio(clone); | |
| 1254 | + break; | |
| 1255 | + default: | |
| 1256 | + alloc_multiple_bios(&blist, ci, ti, num_bios, len); | |
| 1257 | + while ((clone = bio_list_pop(&blist))) { | |
| 1258 | + if (len) | |
| 1259 | + bio_setup_sector(clone, ci->sector, *len); | |
| 1260 | + __map_bio(clone); | |
| 1261 | + } | |
| 1262 | + break; | |
| 1293 | 1263 | } |
| 1294 | 1264 | } |
| 1295 | 1265 | |
| @@ -1304,9 +1274,8 @@ static int __send_empty_flush(struct clone_info *ci) · 8 unmodified lines | ||
| 1304 | 1274 | * need to reference it after submit. It's just used as |
| 1305 | 1275 | * the basis for the clone(s). |
| 1306 | 1276 | */ |
| 1307 | − bio_init(&flush_bio, NULL, 0); | |
| 1308 | − flush_bio.bi_opf = REQ_OP_WRITE | REQ_PREFLUSH | REQ_SYNC; | |
| 1309 | − bio_set_dev(&flush_bio, ci->io->md->disk->part0); | |
| 1277 | + bio_init(&flush_bio, ci->io->md->disk->part0, NULL, 0, | |
| 1278 | + REQ_OP_WRITE | REQ_PREFLUSH | REQ_SYNC); | |
| 1310 | 1279 | |
| 1311 | 1280 | ci->bio = &flush_bio; |
| 1312 | 1281 | ci->sector_count = 0; |
| @@ -1319,25 +1288,6 @@ static int __send_empty_flush(struct clone_info *ci) · 6 unmodified lines | ||
| 1319 | 1288 | return 0; |
| 1320 | 1289 | } |
| 1321 | 1290 | |
| 1322 | −static int __clone_and_map_data_bio(struct clone_info *ci, struct dm_target *ti, | |
| 1323 | − sector_t sector, unsigned *len) | |
| 1324 | −{ | |
| 1325 | − struct bio *bio = ci->bio; | |
| 1326 | − struct dm_target_io *tio; | |
| 1327 | − int r; | |
| 1328 | − | |
| 1329 | − tio = alloc_tio(ci, ti, 0, GFP_NOIO); | |
| 1330 | − tio->len_ptr = len; | |
| 1331 | − r = clone_bio(tio, bio, sector, *len); | |
| 1332 | − if (r < 0) { | |
| 1333 | − free_tio(tio); | |
| 1334 | − return r; | |
| 1335 | − } | |
| 1336 | − __map_bio(tio); | |
| 1337 | − | |
| 1338 | − return 0; | |
| 1339 | −} | |
| 1340 | − | |
| 1341 | 1291 | static int __send_changing_extent_only(struct clone_info *ci, struct dm_target *ti, |
| 1342 | 1292 | unsigned num_bios) |
| 1343 | 1293 | { |
| @@ -205,9 +205,9 @@ static bool faulty_make_request(struct mddev *mddev, struct bio *bio) · 204 unmodified lines | ||
| 205 | 205 | } |
| 206 | 206 | } |
| 207 | 207 | if (failit) { |
| 208 | − struct bio *b = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set); | |
| 208 | + struct bio *b = bio_alloc_clone(conf->rdev->bdev, bio, GFP_NOIO, | |
| 209 | + &mddev->bio_set); | |
| 209 | 210 | |
| 210 | − bio_set_dev(b, conf->rdev->bdev); | |
| 211 | 211 | b->bi_private = bio; |
| 212 | 212 | b->bi_end_io = faulty_fail; |
| 213 | 213 | bio = b; |
| @@ -121,11 +121,9 @@ static bool multipath_make_request(struct mddev *mddev, struct bio * bio) · 120 unmodified lines | ||
| 121 | 121 | } |
| 122 | 122 | multipath = conf->multipaths + mp_bh->path; |
| 123 | 123 | |
| 124 | − bio_init(&mp_bh->bio, NULL, 0); | |
| 125 | − __bio_clone_fast(&mp_bh->bio, bio); | |
| 124 | + bio_init_clone(multipath->rdev->bdev, &mp_bh->bio, bio, GFP_NOIO); | |
| 126 | 125 | |
| 127 | 126 | mp_bh->bio.bi_iter.bi_sector += multipath->rdev->data_offset; |
| 128 | − bio_set_dev(&mp_bh->bio, multipath->rdev->bdev); | |
| 129 | 127 | mp_bh->bio.bi_opf |= REQ_FAILFAST_TRANSPORT; |
| 130 | 128 | mp_bh->bio.bi_end_io = multipath_end_request; |
| 131 | 129 | mp_bh->bio.bi_private = mp_bh; |
| @@ -299,7 +297,6 @@ static void multipathd(struct md_thread *thread) · 167 unmodified lines | ||
| 299 | 297 | |
| 300 | 298 | md_check_recovery(mddev); |
| 301 | 299 | for (;;) { |
| 302 | − char b[BDEVNAME_SIZE]; | |
| 303 | 300 | spin_lock_irqsave(&conf->device_lock, flags); |
| 304 | 301 | if (list_empty(head)) |
| 305 | 302 | break; |
| @@ -311,13 +308,13 @@ static void multipathd(struct md_thread *thread) · 5 unmodified lines | ||
| 311 | 308 | bio->bi_iter.bi_sector = mp_bh->master_bio->bi_iter.bi_sector; |
| 312 | 309 | |
| 313 | 310 | if ((mp_bh->path = multipath_map (conf))<0) { |
| 314 | − pr_err("multipath: %s: unrecoverable IO read error for block %llu\n", | |
| 315 | − bio_devname(bio, b), | |
| 311 | + pr_err("multipath: %pg: unrecoverable IO read error for block %llu\n", | |
| 312 | + bio->bi_bdev, | |
| 316 | 313 | (unsigned long long)bio->bi_iter.bi_sector); |
| 317 | 314 | multipath_end_bh_io(mp_bh, BLK_STS_IOERR); |
| 318 | 315 | } else { |
| 319 | − pr_err("multipath: %s: redirecting sector %llu to another IO path\n", | |
| 320 | − bio_devname(bio, b), | |
| 316 | + pr_err("multipath: %pg: redirecting sector %llu to another IO path\n", | |
| 317 | + bio->bi_bdev, | |
| 321 | 318 | (unsigned long long)bio->bi_iter.bi_sector); |
| 322 | 319 | *bio = *(mp_bh->master_bio); |
| 323 | 320 | bio->bi_iter.bi_sector += |
| @@ -562,11 +562,11 @@ static void submit_flushes(struct work_struct *ws) · 561 unmodified lines | ||
| 562 | 562 | atomic_inc(&rdev->nr_pending); |
| 563 | 563 | atomic_inc(&rdev->nr_pending); |
| 564 | 564 | rcu_read_unlock(); |
| 565 | − bi = bio_alloc_bioset(GFP_NOIO, 0, &mddev->bio_set); | |
| 565 | + bi = bio_alloc_bioset(rdev->bdev, 0, | |
| 566 | + REQ_OP_WRITE | REQ_PREFLUSH, | |
| 567 | + GFP_NOIO, &mddev->bio_set); | |
| 566 | 568 | bi->bi_end_io = md_end_flush; |
| 567 | 569 | bi->bi_private = rdev; |
| 568 | − bio_set_dev(bi, rdev->bdev); | |
| 569 | − bi->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 570 | 570 | atomic_inc(&mddev->flush_pending); |
| 571 | 571 | submit_bio(bi); |
| 572 | 572 | rcu_read_lock(); |
| @@ -955,7 +955,6 @@ void md_super_write(struct mddev *mddev, struct md_rdev *rdev, · 382 unmodified lines | ||
| 955 | 955 | * If an error occurred, call md_error |
| 956 | 956 | */ |
| 957 | 957 | struct bio *bio; |
| 958 | − int ff = 0; | |
| 959 | 958 | |
| 960 | 959 | if (!page) |
| 961 | 960 | return; |
| @@ -963,11 +962,13 @@ void md_super_write(struct mddev *mddev, struct md_rdev *rdev, · 1 unmodified line | ||
| 963 | 962 | if (test_bit(Faulty, &rdev->flags)) |
| 964 | 963 | return; |
| 965 | 964 | |
| 966 | − bio = bio_alloc_bioset(GFP_NOIO, 1, &mddev->sync_set); | |
| 965 | + bio = bio_alloc_bioset(rdev->meta_bdev ? rdev->meta_bdev : rdev->bdev, | |
| 966 | + 1, | |
| 967 | + REQ_OP_WRITE | REQ_SYNC | REQ_PREFLUSH | REQ_FUA, | |
| 968 | + GFP_NOIO, &mddev->sync_set); | |
| 967 | 969 | |
| 968 | 970 | atomic_inc(&rdev->nr_pending); |
| 969 | 971 | |
| 970 | − bio_set_dev(bio, rdev->meta_bdev ? rdev->meta_bdev : rdev->bdev); | |
| 971 | 972 | bio->bi_iter.bi_sector = sector; |
| 972 | 973 | bio_add_page(bio, page, size, 0); |
| 973 | 974 | bio->bi_private = rdev; |
| @@ -976,8 +977,7 @@ void md_super_write(struct mddev *mddev, struct md_rdev *rdev, · 2 unmodified lines | ||
| 976 | 977 | if (test_bit(MD_FAILFAST_SUPPORTED, &mddev->flags) && |
| 977 | 978 | test_bit(FailFast, &rdev->flags) && |
| 978 | 979 | !test_bit(LastDev, &rdev->flags)) |
| 979 | − ff = MD_FAILFAST; | |
| 980 | − bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_PREFLUSH | REQ_FUA | ff; | |
| 980 | + bio->bi_opf |= MD_FAILFAST; | |
| 981 | 981 | |
| 982 | 982 | atomic_inc(&mddev->pending_writes); |
| 983 | 983 | submit_bio(bio); |
| @@ -998,13 +998,11 @@ int sync_page_io(struct md_rdev *rdev, sector_t sector, int size, · 14 unmodified lines | ||
| 998 | 998 | struct bio bio; |
| 999 | 999 | struct bio_vec bvec; |
| 1000 | 1000 | |
| 1001 | − bio_init(&bio, &bvec, 1); | |
| 1002 | − | |
| 1003 | 1001 | if (metadata_op && rdev->meta_bdev) |
| 1004 | − bio_set_dev(&bio, rdev->meta_bdev); | |
| 1002 | + bio_init(&bio, rdev->meta_bdev, &bvec, 1, op | op_flags); | |
| 1005 | 1003 | else |
| 1006 | − bio_set_dev(&bio, rdev->bdev); | |
| 1007 | − bio.bi_opf = op | op_flags; | |
| 1004 | + bio_init(&bio, rdev->bdev, &bvec, 1, op | op_flags); | |
| 1005 | + | |
| 1008 | 1006 | if (metadata_op) |
| 1009 | 1007 | bio.bi_iter.bi_sector = sector + rdev->sb_start; |
| 1010 | 1008 | else if (rdev->mddev->reshape_position != MaxSector && |
| @@ -8636,13 +8634,14 @@ static void md_end_io_acct(struct bio *bio) · 7625 unmodified lines | ||
| 8636 | 8634 | */ |
| 8637 | 8635 | void md_account_bio(struct mddev *mddev, struct bio **bio) |
| 8638 | 8636 | { |
| 8637 | + struct block_device *bdev = (*bio)->bi_bdev; | |
| 8639 | 8638 | struct md_io_acct *md_io_acct; |
| 8640 | 8639 | struct bio *clone; |
| 8641 | 8640 | |
| 8642 | − if (!blk_queue_io_stat((*bio)->bi_bdev->bd_disk->queue)) | |
| 8641 | + if (!blk_queue_io_stat(bdev->bd_disk->queue)) | |
| 8643 | 8642 | return; |
| 8644 | 8643 | |
| 8645 | − clone = bio_clone_fast(*bio, GFP_NOIO, &mddev->io_acct_set); | |
| 8644 | + clone = bio_alloc_clone(bdev, *bio, GFP_NOIO, &mddev->io_acct_set); | |
| 8646 | 8645 | md_io_acct = container_of(clone, struct md_io_acct, bio_clone); |
| 8647 | 8646 | md_io_acct->orig_bio = *bio; |
| 8648 | 8647 | md_io_acct->start_time = bio_start_io_acct(*bio); |
| @@ -1126,7 +1126,8 @@ static void alloc_behind_master_bio(struct r1bio *r1_bio, · 1125 unmodified lines | ||
| 1126 | 1126 | int i = 0; |
| 1127 | 1127 | struct bio *behind_bio = NULL; |
| 1128 | 1128 | |
| 1129 | − behind_bio = bio_alloc_bioset(GFP_NOIO, vcnt, &r1_bio->mddev->bio_set); | |
| 1129 | + behind_bio = bio_alloc_bioset(NULL, vcnt, 0, GFP_NOIO, | |
| 1130 | + &r1_bio->mddev->bio_set); | |
| 1130 | 1131 | if (!behind_bio) |
| 1131 | 1132 | return; |
| 1132 | 1133 | |
| @@ -1319,13 +1320,13 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, · 186 unmodified lines | ||
| 1319 | 1320 | if (!r1bio_existed && blk_queue_io_stat(bio->bi_bdev->bd_disk->queue)) |
| 1320 | 1321 | r1_bio->start_time = bio_start_io_acct(bio); |
| 1321 | 1322 | |
| 1322 | − read_bio = bio_clone_fast(bio, gfp, &mddev->bio_set); | |
| 1323 | + read_bio = bio_alloc_clone(mirror->rdev->bdev, bio, gfp, | |
| 1324 | + &mddev->bio_set); | |
| 1323 | 1325 | |
| 1324 | 1326 | r1_bio->bios[rdisk] = read_bio; |
| 1325 | 1327 | |
| 1326 | 1328 | read_bio->bi_iter.bi_sector = r1_bio->sector + |
| 1327 | 1329 | mirror->rdev->data_offset; |
| 1328 | − bio_set_dev(read_bio, mirror->rdev->bdev); | |
| 1329 | 1330 | read_bio->bi_end_io = raid1_end_read_request; |
| 1330 | 1331 | bio_set_op_attrs(read_bio, op, do_sync); |
| 1331 | 1332 | if (test_bit(FailFast, &mirror->rdev->flags) && |
| @@ -1545,24 +1546,25 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, · 213 unmodified lines | ||
| 1545 | 1546 | first_clone = 0; |
| 1546 | 1547 | } |
| 1547 | 1548 | |
| 1548 | − if (r1_bio->behind_master_bio) | |
| 1549 | − mbio = bio_clone_fast(r1_bio->behind_master_bio, | |
| 1550 | − GFP_NOIO, &mddev->bio_set); | |
| 1551 | − else | |
| 1552 | − mbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set); | |
| 1553 | − | |
| 1554 | 1549 | if (r1_bio->behind_master_bio) { |
| 1550 | + mbio = bio_alloc_clone(rdev->bdev, | |
| 1551 | + r1_bio->behind_master_bio, | |
| 1552 | + GFP_NOIO, &mddev->bio_set); | |
| 1555 | 1553 | if (test_bit(CollisionCheck, &rdev->flags)) |
| 1556 | 1554 | wait_for_serialization(rdev, r1_bio); |
| 1557 | 1555 | if (test_bit(WriteMostly, &rdev->flags)) |
| 1558 | 1556 | atomic_inc(&r1_bio->behind_remaining); |
| 1559 | − } else if (mddev->serialize_policy) | |
| 1560 | − wait_for_serialization(rdev, r1_bio); | |
| 1557 | + } else { | |
| 1558 | + mbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, | |
| 1559 | + &mddev->bio_set); | |
| 1560 | + | |
| 1561 | + if (mddev->serialize_policy) | |
| 1562 | + wait_for_serialization(rdev, r1_bio); | |
| 1563 | + } | |
| 1561 | 1564 | |
| 1562 | 1565 | r1_bio->bios[i] = mbio; |
| 1563 | 1566 | |
| 1564 | 1567 | mbio->bi_iter.bi_sector = (r1_bio->sector + rdev->data_offset); |
| 1565 | − bio_set_dev(mbio, rdev->bdev); | |
| 1566 | 1568 | mbio->bi_end_io = raid1_end_write_request; |
| 1567 | 1569 | mbio->bi_opf = bio_op(bio) | (bio->bi_opf & (REQ_SYNC | REQ_FUA)); |
| 1568 | 1570 | if (test_bit(FailFast, &rdev->flags) && |
| @@ -2070,15 +2072,14 @@ static int fix_sync_read_error(struct r1bio *r1_bio) · 501 unmodified lines | ||
| 2070 | 2072 | } while (!success && d != r1_bio->read_disk); |
| 2071 | 2073 | |
| 2072 | 2074 | if (!success) { |
| 2073 | − char b[BDEVNAME_SIZE]; | |
| 2074 | 2075 | int abort = 0; |
| 2075 | 2076 | /* Cannot read from anywhere, this block is lost. |
| 2076 | 2077 | * Record a bad block on each device. If that doesn't |
| 2077 | 2078 | * work just disable and interrupt the recovery. |
| 2078 | 2079 | * Don't fail devices as that won't really help. |
| 2079 | 2080 | */ |
| 2080 | − pr_crit_ratelimited("md/raid1:%s: %s: unrecoverable I/O read error for block %llu\n", | |
| 2081 | − mdname(mddev), bio_devname(bio, b), | |
| 2081 | + pr_crit_ratelimited("md/raid1:%s: %pg: unrecoverable I/O read error for block %llu\n", | |
| 2082 | + mdname(mddev), bio->bi_bdev, | |
| 2082 | 2083 | (unsigned long long)r1_bio->sector); |
| 2083 | 2084 | for (d = 0; d < conf->raid_disks * 2; d++) { |
| 2084 | 2085 | rdev = conf->mirrors[d].rdev; |
| @@ -2165,11 +2166,10 @@ static void process_checks(struct r1bio *r1_bio) · 80 unmodified lines | ||
| 2165 | 2166 | continue; |
| 2166 | 2167 | /* fixup the bio for reuse, but preserve errno */ |
| 2167 | 2168 | status = b->bi_status; |
| 2168 | − bio_reset(b); | |
| 2169 | + bio_reset(b, conf->mirrors[i].rdev->bdev, REQ_OP_READ); | |
| 2169 | 2170 | b->bi_status = status; |
| 2170 | 2171 | b->bi_iter.bi_sector = r1_bio->sector + |
| 2171 | 2172 | conf->mirrors[i].rdev->data_offset; |
| 2172 | − bio_set_dev(b, conf->mirrors[i].rdev->bdev); | |
| 2173 | 2173 | b->bi_end_io = end_sync_read; |
| 2174 | 2174 | rp->raid_bio = r1_bio; |
| 2175 | 2175 | b->bi_private = rp; |
| @@ -2416,12 +2416,12 @@ static int narrow_write_error(struct r1bio *r1_bio, int i) · 240 unmodified lines | ||
| 2416 | 2416 | /* Write at 'sector' for 'sectors'*/ |
| 2417 | 2417 | |
| 2418 | 2418 | if (test_bit(R1BIO_BehindIO, &r1_bio->state)) { |
| 2419 | − wbio = bio_clone_fast(r1_bio->behind_master_bio, | |
| 2420 | − GFP_NOIO, | |
| 2421 | − &mddev->bio_set); | |
| 2419 | + wbio = bio_alloc_clone(rdev->bdev, | |
| 2420 | + r1_bio->behind_master_bio, | |
| 2421 | + GFP_NOIO, &mddev->bio_set); | |
| 2422 | 2422 | } else { |
| 2423 | − wbio = bio_clone_fast(r1_bio->master_bio, GFP_NOIO, | |
| 2424 | − &mddev->bio_set); | |
| 2423 | + wbio = bio_alloc_clone(rdev->bdev, r1_bio->master_bio, | |
| 2424 | + GFP_NOIO, &mddev->bio_set); | |
| 2425 | 2425 | } |
| 2426 | 2426 | |
| 2427 | 2427 | bio_set_op_attrs(wbio, REQ_OP_WRITE, 0); |
| @@ -2430,7 +2430,6 @@ static int narrow_write_error(struct r1bio *r1_bio, int i) · 2 unmodified lines | ||
| 2430 | 2430 | |
| 2431 | 2431 | bio_trim(wbio, sector - r1_bio->sector, sectors); |
| 2432 | 2432 | wbio->bi_iter.bi_sector += rdev->data_offset; |
| 2433 | − bio_set_dev(wbio, rdev->bdev); | |
| 2434 | 2433 | |
| 2435 | 2434 | if (submit_bio_wait(wbio) < 0) |
| 2436 | 2435 | /* failure! */ |
| @@ -2650,7 +2649,7 @@ static struct r1bio *raid1_alloc_init_r1buf(struct r1conf *conf) · 213 unmodified lines | ||
| 2650 | 2649 | for (i = conf->poolinfo->raid_disks; i--; ) { |
| 2651 | 2650 | bio = r1bio->bios[i]; |
| 2652 | 2651 | rps = bio->bi_private; |
| 2653 | − bio_reset(bio); | |
| 2652 | + bio_reset(bio, NULL, 0); | |
| 2654 | 2653 | bio->bi_private = rps; |
| 2655 | 2654 | } |
| 2656 | 2655 | r1bio->master_bio = NULL; |
| @@ -1208,14 +1208,13 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, · 1207 unmodified lines | ||
| 1208 | 1208 | |
| 1209 | 1209 | if (blk_queue_io_stat(bio->bi_bdev->bd_disk->queue)) |
| 1210 | 1210 | r10_bio->start_time = bio_start_io_acct(bio); |
| 1211 | − read_bio = bio_clone_fast(bio, gfp, &mddev->bio_set); | |
| 1211 | + read_bio = bio_alloc_clone(rdev->bdev, bio, gfp, &mddev->bio_set); | |
| 1212 | 1212 | |
| 1213 | 1213 | r10_bio->devs[slot].bio = read_bio; |
| 1214 | 1214 | r10_bio->devs[slot].rdev = rdev; |
| 1215 | 1215 | |
| 1216 | 1216 | read_bio->bi_iter.bi_sector = r10_bio->devs[slot].addr + |
| 1217 | 1217 | choose_data_offset(r10_bio, rdev); |
| 1218 | − bio_set_dev(read_bio, rdev->bdev); | |
| 1219 | 1218 | read_bio->bi_end_io = raid10_end_read_request; |
| 1220 | 1219 | bio_set_op_attrs(read_bio, op, do_sync); |
| 1221 | 1220 | if (test_bit(FailFast, &rdev->flags) && |
| @@ -1255,7 +1254,7 @@ static void raid10_write_one_disk(struct mddev *mddev, struct r10bio *r10_bio, · 33 unmodified lines | ||
| 1255 | 1254 | } else |
| 1256 | 1255 | rdev = conf->mirrors[devnum].rdev; |
| 1257 | 1256 | |
| 1258 | − mbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set); | |
| 1257 | + mbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, &mddev->bio_set); | |
| 1259 | 1258 | if (replacement) |
| 1260 | 1259 | r10_bio->devs[n_copy].repl_bio = mbio; |
| 1261 | 1260 | else |
| @@ -1263,7 +1262,6 @@ static void raid10_write_one_disk(struct mddev *mddev, struct r10bio *r10_bio, · 1 unmodified line | ||
| 1263 | 1262 | |
| 1264 | 1263 | mbio->bi_iter.bi_sector = (r10_bio->devs[n_copy].addr + |
| 1265 | 1264 | choose_data_offset(r10_bio, rdev)); |
| 1266 | − bio_set_dev(mbio, rdev->bdev); | |
| 1267 | 1265 | mbio->bi_end_io = raid10_end_write_request; |
| 1268 | 1266 | bio_set_op_attrs(mbio, op, do_sync | do_fua); |
| 1269 | 1267 | if (!replacement && test_bit(FailFast, |
| @@ -1812,7 +1810,8 @@ retry_discard: · 542 unmodified lines | ||
| 1812 | 1810 | */ |
| 1813 | 1811 | if (r10_bio->devs[disk].bio) { |
| 1814 | 1812 | struct md_rdev *rdev = conf->mirrors[disk].rdev; |
| 1815 | − mbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set); | |
| 1813 | + mbio = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOIO, | |
| 1814 | + &mddev->bio_set); | |
| 1816 | 1815 | mbio->bi_end_io = raid10_end_discard_request; |
| 1817 | 1816 | mbio->bi_private = r10_bio; |
| 1818 | 1817 | r10_bio->devs[disk].bio = mbio; |
| @@ -1825,7 +1824,8 @@ retry_discard: · 6 unmodified lines | ||
| 1825 | 1824 | } |
| 1826 | 1825 | if (r10_bio->devs[disk].repl_bio) { |
| 1827 | 1826 | struct md_rdev *rrdev = conf->mirrors[disk].replacement; |
| 1828 | − rbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set); | |
| 1827 | + rbio = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOIO, | |
| 1828 | + &mddev->bio_set); | |
| 1829 | 1829 | rbio->bi_end_io = raid10_end_discard_request; |
| 1830 | 1830 | rbio->bi_private = r10_bio; |
| 1831 | 1831 | r10_bio->devs[disk].repl_bio = rbio; |
| @@ -2422,7 +2422,7 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio) · 590 unmodified lines | ||
| 2422 | 2422 | * bi_vecs, as the read request might have corrupted these |
| 2423 | 2423 | */ |
| 2424 | 2424 | rp = get_resync_pages(tbio); |
| 2425 | − bio_reset(tbio); | |
| 2425 | + bio_reset(tbio, conf->mirrors[d].rdev->bdev, REQ_OP_WRITE); | |
| 2426 | 2426 | |
| 2427 | 2427 | md_bio_reset_resync_pages(tbio, rp, fbio->bi_iter.bi_size); |
| 2428 | 2428 | |
| @@ -2430,7 +2430,6 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio) · 1 unmodified line | ||
| 2430 | 2430 | tbio->bi_private = rp; |
| 2431 | 2431 | tbio->bi_iter.bi_sector = r10_bio->devs[i].addr; |
| 2432 | 2432 | tbio->bi_end_io = end_sync_write; |
| 2433 | − bio_set_op_attrs(tbio, REQ_OP_WRITE, 0); | |
| 2434 | 2433 | |
| 2435 | 2434 | bio_copy_data(tbio, fbio); |
| 2436 | 2435 | |
| @@ -2441,7 +2440,6 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio) · 4 unmodified lines | ||
| 2441 | 2440 | if (test_bit(FailFast, &conf->mirrors[d].rdev->flags)) |
| 2442 | 2441 | tbio->bi_opf |= MD_FAILFAST; |
| 2443 | 2442 | tbio->bi_iter.bi_sector += conf->mirrors[d].rdev->data_offset; |
| 2444 | − bio_set_dev(tbio, conf->mirrors[d].rdev->bdev); | |
| 2445 | 2443 | submit_bio_noacct(tbio); |
| 2446 | 2444 | } |
| 2447 | 2445 | |
| @@ -2894,12 +2892,12 @@ static int narrow_write_error(struct r10bio *r10_bio, int i) · 446 unmodified lines | ||
| 2894 | 2892 | if (sectors > sect_to_write) |
| 2895 | 2893 | sectors = sect_to_write; |
| 2896 | 2894 | /* Write at 'sector' for 'sectors' */ |
| 2897 | − wbio = bio_clone_fast(bio, GFP_NOIO, &mddev->bio_set); | |
| 2895 | + wbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, | |
| 2896 | + &mddev->bio_set); | |
| 2898 | 2897 | bio_trim(wbio, sector - bio->bi_iter.bi_sector, sectors); |
| 2899 | 2898 | wsector = r10_bio->devs[i].addr + (sector - r10_bio->sector); |
| 2900 | 2899 | wbio->bi_iter.bi_sector = wsector + |
| 2901 | 2900 | choose_data_offset(r10_bio, rdev); |
| 2902 | − bio_set_dev(wbio, rdev->bdev); | |
| 2903 | 2901 | bio_set_op_attrs(wbio, REQ_OP_WRITE, 0); |
| 2904 | 2902 | |
| 2905 | 2903 | if (submit_bio_wait(wbio) < 0) |
| @@ -3160,12 +3158,12 @@ static struct r10bio *raid10_alloc_init_r10buf(struct r10conf *conf) · 254 unmodified lines | ||
| 3160 | 3158 | for (i = 0; i < nalloc; i++) { |
| 3161 | 3159 | bio = r10bio->devs[i].bio; |
| 3162 | 3160 | rp = bio->bi_private; |
| 3163 | − bio_reset(bio); | |
| 3161 | + bio_reset(bio, NULL, 0); | |
| 3164 | 3162 | bio->bi_private = rp; |
| 3165 | 3163 | bio = r10bio->devs[i].repl_bio; |
| 3166 | 3164 | if (bio) { |
| 3167 | 3165 | rp = bio->bi_private; |
| 3168 | − bio_reset(bio); | |
| 3166 | + bio_reset(bio, NULL, 0); | |
| 3169 | 3167 | bio->bi_private = rp; |
| 3170 | 3168 | } |
| 3171 | 3169 | } |
| @@ -4892,14 +4890,12 @@ read_more: · 1720 unmodified lines | ||
| 4892 | 4890 | return sectors_done; |
| 4893 | 4891 | } |
| 4894 | 4892 | |
| 4895 | − read_bio = bio_alloc_bioset(GFP_KERNEL, RESYNC_PAGES, &mddev->bio_set); | |
| 4896 | − | |
| 4897 | − bio_set_dev(read_bio, rdev->bdev); | |
| 4893 | + read_bio = bio_alloc_bioset(rdev->bdev, RESYNC_PAGES, REQ_OP_READ, | |
| 4894 | + GFP_KERNEL, &mddev->bio_set); | |
| 4898 | 4895 | read_bio->bi_iter.bi_sector = (r10_bio->devs[r10_bio->read_slot].addr |
| 4899 | 4896 | + rdev->data_offset); |
| 4900 | 4897 | read_bio->bi_private = r10_bio; |
| 4901 | 4898 | read_bio->bi_end_io = end_reshape_read; |
| 4902 | − bio_set_op_attrs(read_bio, REQ_OP_READ, 0); | |
| 4903 | 4899 | r10_bio->master_bio = read_bio; |
| 4904 | 4900 | r10_bio->read_slot = r10_bio->devs[r10_bio->read_slot].devnum; |
| 4905 | 4901 | |
| @@ -735,10 +735,9 @@ static void r5l_submit_current_io(struct r5l_log *log) · 734 unmodified lines | ||
| 735 | 735 | |
| 736 | 736 | static struct bio *r5l_bio_alloc(struct r5l_log *log) |
| 737 | 737 | { |
| 738 | − struct bio *bio = bio_alloc_bioset(GFP_NOIO, BIO_MAX_VECS, &log->bs); | |
| 738 | + struct bio *bio = bio_alloc_bioset(log->rdev->bdev, BIO_MAX_VECS, | |
| 739 | + REQ_OP_WRITE, GFP_NOIO, &log->bs); | |
| 739 | 740 | |
| 740 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 741 | − bio_set_dev(bio, log->rdev->bdev); | |
| 742 | 741 | bio->bi_iter.bi_sector = log->rdev->data_offset + log->log_start; |
| 743 | 742 | |
| 744 | 743 | return bio; |
| @@ -1302,10 +1301,9 @@ void r5l_flush_stripe_to_raid(struct r5l_log *log) · 557 unmodified lines | ||
| 1302 | 1301 | |
| 1303 | 1302 | if (!do_flush) |
| 1304 | 1303 | return; |
| 1305 | − bio_reset(&log->flush_bio); | |
| 1306 | − bio_set_dev(&log->flush_bio, log->rdev->bdev); | |
| 1304 | + bio_reset(&log->flush_bio, log->rdev->bdev, | |
| 1305 | + REQ_OP_WRITE | REQ_PREFLUSH); | |
| 1307 | 1306 | log->flush_bio.bi_end_io = r5l_log_flush_endio; |
| 1308 | − log->flush_bio.bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 1309 | 1307 | submit_bio(&log->flush_bio); |
| 1310 | 1308 | } |
| 1311 | 1309 | |
| @@ -1634,7 +1632,8 @@ static int r5l_recovery_allocate_ra_pool(struct r5l_log *log, · 322 unmodified lines | ||
| 1634 | 1632 | { |
| 1635 | 1633 | struct page *page; |
| 1636 | 1634 | |
| 1637 | − ctx->ra_bio = bio_alloc_bioset(GFP_KERNEL, BIO_MAX_VECS, &log->bs); | |
| 1635 | + ctx->ra_bio = bio_alloc_bioset(NULL, BIO_MAX_VECS, 0, GFP_KERNEL, | |
| 1636 | + &log->bs); | |
| 1638 | 1637 | if (!ctx->ra_bio) |
| 1639 | 1638 | return -ENOMEM; |
| 1640 | 1639 | |
| @@ -1678,9 +1677,7 @@ static int r5l_recovery_fetch_ra_pool(struct r5l_log *log, · 37 unmodified lines | ||
| 1678 | 1677 | struct r5l_recovery_ctx *ctx, |
| 1679 | 1678 | sector_t offset) |
| 1680 | 1679 | { |
| 1681 | − bio_reset(ctx->ra_bio); | |
| 1682 | − bio_set_dev(ctx->ra_bio, log->rdev->bdev); | |
| 1683 | − bio_set_op_attrs(ctx->ra_bio, REQ_OP_READ, 0); | |
| 1680 | + bio_reset(ctx->ra_bio, log->rdev->bdev, REQ_OP_READ); | |
| 1684 | 1681 | ctx->ra_bio->bi_iter.bi_sector = log->rdev->data_offset + offset; |
| 1685 | 1682 | |
| 1686 | 1683 | ctx->valid_pages = 0; |
| @@ -3108,7 +3105,7 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev) · 1421 unmodified lines | ||
| 3108 | 3105 | INIT_LIST_HEAD(&log->io_end_ios); |
| 3109 | 3106 | INIT_LIST_HEAD(&log->flushing_ios); |
| 3110 | 3107 | INIT_LIST_HEAD(&log->finished_ios); |
| 3111 | − bio_init(&log->flush_bio, NULL, 0); | |
| 3108 | + bio_init(&log->flush_bio, NULL, NULL, 0, 0); | |
| 3112 | 3109 | |
| 3113 | 3110 | log->io_kc = KMEM_CACHE(r5l_io_unit, 0); |
| 3114 | 3111 | if (!log->io_kc) |
| @@ -250,7 +250,7 @@ static struct ppl_io_unit *ppl_new_iounit(struct ppl_log *log, · 249 unmodified lines | ||
| 250 | 250 | INIT_LIST_HEAD(&io->stripe_list); |
| 251 | 251 | atomic_set(&io->pending_stripes, 0); |
| 252 | 252 | atomic_set(&io->pending_flushes, 0); |
| 253 | − bio_init(&io->bio, io->biovec, PPL_IO_INLINE_BVECS); | |
| 253 | + bio_init(&io->bio, NULL, io->biovec, PPL_IO_INLINE_BVECS, 0); | |
| 254 | 254 | |
| 255 | 255 | pplhdr = page_address(io->header_page); |
| 256 | 256 | clear_page(pplhdr); |
| @@ -416,12 +416,10 @@ static void ppl_log_endio(struct bio *bio) · 159 unmodified lines | ||
| 416 | 416 | |
| 417 | 417 | static void ppl_submit_iounit_bio(struct ppl_io_unit *io, struct bio *bio) |
| 418 | 418 | { |
| 419 | − char b[BDEVNAME_SIZE]; | |
| 420 | − | |
| 421 | − pr_debug("%s: seq: %llu size: %u sector: %llu dev: %s\n", | |
| 419 | + pr_debug("%s: seq: %llu size: %u sector: %llu dev: %pg\n", | |
| 422 | 420 | __func__, io->seq, bio->bi_iter.bi_size, |
| 423 | 421 | (unsigned long long)bio->bi_iter.bi_sector, |
| 424 | − bio_devname(bio, b)); | |
| 422 | + bio->bi_bdev); | |
| 425 | 423 | |
| 426 | 424 | submit_bio(bio); |
| 427 | 425 | } |
| @@ -496,11 +494,10 @@ static void ppl_submit_iounit(struct ppl_io_unit *io) · 68 unmodified lines | ||
| 496 | 494 | if (!bio_add_page(bio, sh->ppl_page, PAGE_SIZE, 0)) { |
| 497 | 495 | struct bio *prev = bio; |
| 498 | 496 | |
| 499 | − bio = bio_alloc_bioset(GFP_NOIO, BIO_MAX_VECS, | |
| 497 | + bio = bio_alloc_bioset(prev->bi_bdev, BIO_MAX_VECS, | |
| 498 | + prev->bi_opf, GFP_NOIO, | |
| 500 | 499 | &ppl_conf->bs); |
| 501 | − bio->bi_opf = prev->bi_opf; | |
| 502 | 500 | bio->bi_write_hint = prev->bi_write_hint; |
| 503 | − bio_copy_dev(bio, prev); | |
| 504 | 501 | bio->bi_iter.bi_sector = bio_end_sector(prev); |
| 505 | 502 | bio_add_page(bio, sh->ppl_page, PAGE_SIZE, 0); |
| 506 | 503 | |
| @@ -590,9 +587,8 @@ static void ppl_flush_endio(struct bio *bio) · 83 unmodified lines | ||
| 590 | 587 | struct ppl_log *log = io->log; |
| 591 | 588 | struct ppl_conf *ppl_conf = log->ppl_conf; |
| 592 | 589 | struct r5conf *conf = ppl_conf->mddev->private; |
| 593 | − char b[BDEVNAME_SIZE]; | |
| 594 | 590 | |
| 595 | − pr_debug("%s: dev: %s\n", __func__, bio_devname(bio, b)); | |
| 591 | + pr_debug("%s: dev: %pg\n", __func__, bio->bi_bdev); | |
| 596 | 592 | |
| 597 | 593 | if (bio->bi_status) { |
| 598 | 594 | struct md_rdev *rdev; |
| @@ -635,16 +631,14 @@ static void ppl_do_flush(struct ppl_io_unit *io) · 36 unmodified lines | ||
| 635 | 631 | |
| 636 | 632 | if (bdev) { |
| 637 | 633 | struct bio *bio; |
| 638 | − char b[BDEVNAME_SIZE]; | |
| 639 | 634 | |
| 640 | − bio = bio_alloc_bioset(GFP_NOIO, 0, &ppl_conf->flush_bs); | |
| 641 | − bio_set_dev(bio, bdev); | |
| 635 | + bio = bio_alloc_bioset(bdev, 0, GFP_NOIO, | |
| 636 | + REQ_OP_WRITE | REQ_PREFLUSH, | |
| 637 | + &ppl_conf->flush_bs); | |
| 642 | 638 | bio->bi_private = io; |
| 643 | − bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 644 | 639 | bio->bi_end_io = ppl_flush_endio; |
| 645 | 640 | |
| 646 | − pr_debug("%s: dev: %s\n", __func__, | |
| 647 | − bio_devname(bio, b)); | |
| 641 | + pr_debug("%s: dev: %ps\n", __func__, bio->bi_bdev); | |
| 648 | 642 | |
| 649 | 643 | submit_bio(bio); |
| 650 | 644 | flushed_disks++; |
| @@ -2310,8 +2310,8 @@ static struct stripe_head *alloc_stripe(struct kmem_cache *sc, gfp_t gfp, · 2309 unmodified lines | ||
| 2310 | 2310 | for (i = 0; i < disks; i++) { |
| 2311 | 2311 | struct r5dev *dev = &sh->dev[i]; |
| 2312 | 2312 | |
| 2313 | − bio_init(&dev->req, &dev->vec, 1); | |
| 2314 | − bio_init(&dev->rreq, &dev->rvec, 1); | |
| 2313 | + bio_init(&dev->req, NULL, &dev->vec, 1, 0); | |
| 2314 | + bio_init(&dev->rreq, NULL, &dev->rvec, 1, 0); | |
| 2315 | 2315 | } |
| 2316 | 2316 | |
| 2317 | 2317 | if (raid5_has_ppl(conf)) { |
| @@ -2677,7 +2677,7 @@ static void raid5_end_read_request(struct bio * bi) · 359 unmodified lines | ||
| 2677 | 2677 | (unsigned long long)sh->sector, i, atomic_read(&sh->count), |
| 2678 | 2678 | bi->bi_status); |
| 2679 | 2679 | if (i == disks) { |
| 2680 | − bio_reset(bi); | |
| 2680 | + bio_reset(bi, NULL, 0); | |
| 2681 | 2681 | BUG(); |
| 2682 | 2682 | return; |
| 2683 | 2683 | } |
| @@ -2785,7 +2785,7 @@ static void raid5_end_read_request(struct bio * bi) · 101 unmodified lines | ||
| 2785 | 2785 | } |
| 2786 | 2786 | } |
| 2787 | 2787 | rdev_dec_pending(rdev, conf->mddev); |
| 2788 | − bio_reset(bi); | |
| 2788 | + bio_reset(bi, NULL, 0); | |
| 2789 | 2789 | clear_bit(R5_LOCKED, &sh->dev[i].flags); |
| 2790 | 2790 | set_bit(STRIPE_HANDLE, &sh->state); |
| 2791 | 2791 | raid5_release_stripe(sh); |
| @@ -2823,7 +2823,7 @@ static void raid5_end_write_request(struct bio *bi) · 31 unmodified lines | ||
| 2823 | 2823 | (unsigned long long)sh->sector, i, atomic_read(&sh->count), |
| 2824 | 2824 | bi->bi_status); |
| 2825 | 2825 | if (i == disks) { |
| 2826 | − bio_reset(bi); | |
| 2826 | + bio_reset(bi, NULL, 0); | |
| 2827 | 2827 | BUG(); |
| 2828 | 2828 | return; |
| 2829 | 2829 | } |
| @@ -2860,7 +2860,7 @@ static void raid5_end_write_request(struct bio *bi) · 30 unmodified lines | ||
| 2860 | 2860 | if (sh->batch_head && bi->bi_status && !replacement) |
| 2861 | 2861 | set_bit(STRIPE_BATCH_ERR, &sh->batch_head->state); |
| 2862 | 2862 | |
| 2863 | − bio_reset(bi); | |
| 2863 | + bio_reset(bi, NULL, 0); | |
| 2864 | 2864 | if (!test_and_clear_bit(R5_DOUBLE_LOCKED, &sh->dev[i].flags)) |
| 2865 | 2865 | clear_bit(R5_LOCKED, &sh->dev[i].flags); |
| 2866 | 2866 | set_bit(STRIPE_HANDLE, &sh->state); |
| @@ -5438,14 +5438,14 @@ static int raid5_read_one_chunk(struct mddev *mddev, struct bio *raid_bio) · 2571 unmodified lines | ||
| 5438 | 5438 | return 0; |
| 5439 | 5439 | } |
| 5440 | 5440 | |
| 5441 | − align_bio = bio_clone_fast(raid_bio, GFP_NOIO, &mddev->io_acct_set); | |
| 5441 | + align_bio = bio_alloc_clone(rdev->bdev, raid_bio, GFP_NOIO, | |
| 5442 | + &mddev->io_acct_set); | |
| 5442 | 5443 | md_io_acct = container_of(align_bio, struct md_io_acct, bio_clone); |
| 5443 | 5444 | raid_bio->bi_next = (void *)rdev; |
| 5444 | 5445 | if (blk_queue_io_stat(raid_bio->bi_bdev->bd_disk->queue)) |
| 5445 | 5446 | md_io_acct->start_time = bio_start_io_acct(raid_bio); |
| 5446 | 5447 | md_io_acct->orig_bio = raid_bio; |
| 5447 | 5448 | |
| 5448 | − bio_set_dev(align_bio, rdev->bdev); | |
| 5449 | 5449 | align_bio->bi_end_io = raid5_align_endio; |
| 5450 | 5450 | align_bio->bi_private = md_io_acct; |
| 5451 | 5451 | align_bio->bi_iter.bi_sector = sector; |
| @@ -1943,22 +1943,6 @@ static void msb_io_work(struct work_struct *work) · 1942 unmodified lines | ||
| 1943 | 1943 | static DEFINE_IDR(msb_disk_idr); /*set of used disk numbers */ |
| 1944 | 1944 | static DEFINE_MUTEX(msb_disk_lock); /* protects against races in open/release */ |
| 1945 | 1945 | |
| 1946 | −static int msb_bd_open(struct block_device *bdev, fmode_t mode) | |
| 1947 | −{ | |
| 1948 | − struct gendisk *disk = bdev->bd_disk; | |
| 1949 | − struct msb_data *msb = disk->private_data; | |
| 1950 | − | |
| 1951 | − dbg_verbose("block device open"); | |
| 1952 | − | |
| 1953 | − mutex_lock(&msb_disk_lock); | |
| 1954 | − | |
| 1955 | − if (msb && msb->card) | |
| 1956 | − msb->usage_count++; | |
| 1957 | − | |
| 1958 | − mutex_unlock(&msb_disk_lock); | |
| 1959 | − return 0; | |
| 1960 | −} | |
| 1961 | − | |
| 1962 | 1946 | static void msb_data_clear(struct msb_data *msb) |
| 1963 | 1947 | { |
| 1964 | 1948 | kfree(msb->boot_page); |
| @@ -1968,33 +1952,6 @@ static void msb_data_clear(struct msb_data *msb) · 3 unmodified lines | ||
| 1968 | 1952 | msb->card = NULL; |
| 1969 | 1953 | } |
| 1970 | 1954 | |
| 1971 | −static int msb_disk_release(struct gendisk *disk) | |
| 1972 | −{ | |
| 1973 | − struct msb_data *msb = disk->private_data; | |
| 1974 | − | |
| 1975 | − dbg_verbose("block device release"); | |
| 1976 | − mutex_lock(&msb_disk_lock); | |
| 1977 | − | |
| 1978 | − if (msb) { | |
| 1979 | − if (msb->usage_count) | |
| 1980 | − msb->usage_count--; | |
| 1981 | − | |
| 1982 | − if (!msb->usage_count) { | |
| 1983 | − disk->private_data = NULL; | |
| 1984 | − idr_remove(&msb_disk_idr, msb->disk_id); | |
| 1985 | − put_disk(disk); | |
| 1986 | − kfree(msb); | |
| 1987 | − } | |
| 1988 | − } | |
| 1989 | − mutex_unlock(&msb_disk_lock); | |
| 1990 | − return 0; | |
| 1991 | −} | |
| 1992 | − | |
| 1993 | −static void msb_bd_release(struct gendisk *disk, fmode_t mode) | |
| 1994 | −{ | |
| 1995 | − msb_disk_release(disk); | |
| 1996 | −} | |
| 1997 | − | |
| 1998 | 1955 | static int msb_bd_getgeo(struct block_device *bdev, |
| 1999 | 1956 | struct hd_geometry *geo) |
| 2000 | 1957 | { |
| @@ -2003,6 +1960,17 @@ static int msb_bd_getgeo(struct block_device *bdev, · 2 unmodified lines | ||
| 2003 | 1960 | return 0; |
| 2004 | 1961 | } |
| 2005 | 1962 | |
| 1963 | +static void msb_bd_free_disk(struct gendisk *disk) | |
| 1964 | +{ | |
| 1965 | + struct msb_data *msb = disk->private_data; | |
| 1966 | + | |
| 1967 | + mutex_lock(&msb_disk_lock); | |
| 1968 | + idr_remove(&msb_disk_idr, msb->disk_id); | |
| 1969 | + mutex_unlock(&msb_disk_lock); | |
| 1970 | + | |
| 1971 | + kfree(msb); | |
| 1972 | +} | |
| 1973 | + | |
| 2006 | 1974 | static blk_status_t msb_queue_rq(struct blk_mq_hw_ctx *hctx, |
| 2007 | 1975 | const struct blk_mq_queue_data *bd) |
| 2008 | 1976 | { |
| @@ -2096,10 +2064,9 @@ static void msb_start(struct memstick_dev *card) · 87 unmodified lines | ||
| 2096 | 2064 | } |
| 2097 | 2065 | |
| 2098 | 2066 | static const struct block_device_operations msb_bdops = { |
| 2099 | − .open = msb_bd_open, | |
| 2100 | − .release = msb_bd_release, | |
| 2101 | − .getgeo = msb_bd_getgeo, | |
| 2102 | − .owner = THIS_MODULE | |
| 2067 | + .owner = THIS_MODULE, | |
| 2068 | + .getgeo = msb_bd_getgeo, | |
| 2069 | + .free_disk = msb_bd_free_disk, | |
| 2103 | 2070 | }; |
| 2104 | 2071 | |
| 2105 | 2072 | static const struct blk_mq_ops msb_mq_ops = { |
| @@ -2147,7 +2114,6 @@ static int msb_init_disk(struct memstick_dev *card) · 41 unmodified lines | ||
| 2147 | 2114 | set_capacity(msb->disk, capacity); |
| 2148 | 2115 | dbg("Set total disk size to %lu sectors", capacity); |
| 2149 | 2116 | |
| 2150 | − msb->usage_count = 1; | |
| 2151 | 2117 | msb->io_queue = alloc_ordered_workqueue("ms_block", WQ_MEM_RECLAIM); |
| 2152 | 2118 | INIT_WORK(&msb->io_work, msb_io_work); |
| 2153 | 2119 | sg_init_table(msb->prealloc_sg, MS_BLOCK_MAX_SEGS+1); |
| @@ -2229,7 +2195,7 @@ static void msb_remove(struct memstick_dev *card) · 75 unmodified lines | ||
| 2229 | 2195 | msb_data_clear(msb); |
| 2230 | 2196 | mutex_unlock(&msb_disk_lock); |
| 2231 | 2197 | |
| 2232 | − msb_disk_release(msb->disk); | |
| 2198 | + put_disk(msb->disk); | |
| 2233 | 2199 | memstick_set_drvdata(card, NULL); |
| 2234 | 2200 | } |
| 2235 | 2201 | |
| @@ -143,7 +143,6 @@ struct ms_boot_page { · 142 unmodified lines | ||
| 143 | 143 | } __packed; |
| 144 | 144 | |
| 145 | 145 | struct msb_data { |
| 146 | − unsigned int usage_count; | |
| 147 | 146 | struct memstick_dev *card; |
| 148 | 147 | struct gendisk *disk; |
| 149 | 148 | struct request_queue *queue; |
| @@ -133,7 +133,6 @@ struct mspro_devinfo { · 132 unmodified lines | ||
| 133 | 133 | |
| 134 | 134 | struct mspro_block_data { |
| 135 | 135 | struct memstick_dev *card; |
| 136 | − unsigned int usage_count; | |
| 137 | 136 | unsigned int caps; |
| 138 | 137 | struct gendisk *disk; |
| 139 | 138 | struct request_queue *queue; |
| @@ -178,53 +177,16 @@ static int mspro_block_complete_req(struct memstick_dev *card, int error); · 38 unmodified lines | ||
| 178 | 177 | |
| 179 | 178 | /*** Block device ***/ |
| 180 | 179 | |
| 181 | −static int mspro_block_bd_open(struct block_device *bdev, fmode_t mode) | |
| 182 | −{ | |
| 183 | − struct gendisk *disk = bdev->bd_disk; | |
| 184 | − struct mspro_block_data *msb = disk->private_data; | |
| 185 | − int rc = -ENXIO; | |
| 186 | − | |
| 187 | − mutex_lock(&mspro_block_disk_lock); | |
| 188 | − | |
| 189 | − if (msb && msb->card) { | |
| 190 | − msb->usage_count++; | |
| 191 | − if ((mode & FMODE_WRITE) && msb->read_only) | |
| 192 | − rc = -EROFS; | |
| 193 | − else | |
| 194 | − rc = 0; | |
| 195 | − } | |
| 196 | − | |
| 197 | − mutex_unlock(&mspro_block_disk_lock); | |
| 198 | − | |
| 199 | − return rc; | |
| 200 | −} | |
| 201 | − | |
| 202 | − | |
| 203 | −static void mspro_block_disk_release(struct gendisk *disk) | |
| 180 | +static void mspro_block_bd_free_disk(struct gendisk *disk) | |
| 204 | 181 | { |
| 205 | 182 | struct mspro_block_data *msb = disk->private_data; |
| 206 | 183 | int disk_id = MINOR(disk_devt(disk)) >> MSPRO_BLOCK_PART_SHIFT; |
| 207 | 184 | |
| 208 | 185 | mutex_lock(&mspro_block_disk_lock); |
| 209 | − | |
| 210 | − if (msb) { | |
| 211 | − if (msb->usage_count) | |
| 212 | − msb->usage_count--; | |
| 213 | − | |
| 214 | − if (!msb->usage_count) { | |
| 215 | − kfree(msb); | |
| 216 | − disk->private_data = NULL; | |
| 217 | − idr_remove(&mspro_block_disk_idr, disk_id); | |
| 218 | − put_disk(disk); | |
| 219 | − } | |
| 220 | − } | |
| 221 | − | |
| 186 | + idr_remove(&mspro_block_disk_idr, disk_id); | |
| 222 | 187 | mutex_unlock(&mspro_block_disk_lock); |
| 223 | −} | |
| 224 | 188 | |
| 225 | −static void mspro_block_bd_release(struct gendisk *disk, fmode_t mode) | |
| 226 | −{ | |
| 227 | − mspro_block_disk_release(disk); | |
| 189 | + kfree(msb); | |
| 228 | 190 | } |
| 229 | 191 | |
| 230 | 192 | static int mspro_block_bd_getgeo(struct block_device *bdev, |
| @@ -240,10 +202,9 @@ static int mspro_block_bd_getgeo(struct block_device *bdev, · 9 unmodified lines | ||
| 240 | 202 | } |
| 241 | 203 | |
| 242 | 204 | static const struct block_device_operations ms_block_bdops = { |
| 243 | − .open = mspro_block_bd_open, | |
| 244 | − .release = mspro_block_bd_release, | |
| 245 | − .getgeo = mspro_block_bd_getgeo, | |
| 246 | − .owner = THIS_MODULE | |
| 205 | + .owner = THIS_MODULE, | |
| 206 | + .getgeo = mspro_block_bd_getgeo, | |
| 207 | + .free_disk = mspro_block_bd_free_disk, | |
| 247 | 208 | }; |
| 248 | 209 | |
| 249 | 210 | /*** Information ***/ |
| @@ -1226,7 +1187,6 @@ static int mspro_block_init_disk(struct memstick_dev *card) · 976 unmodified lines | ||
| 1226 | 1187 | msb->disk->first_minor = disk_id << MSPRO_BLOCK_PART_SHIFT; |
| 1227 | 1188 | msb->disk->minors = 1 << MSPRO_BLOCK_PART_SHIFT; |
| 1228 | 1189 | msb->disk->fops = &ms_block_bdops; |
| 1229 | − msb->usage_count = 1; | |
| 1230 | 1190 | msb->disk->private_data = msb; |
| 1231 | 1191 | |
| 1232 | 1192 | sprintf(msb->disk->disk_name, "mspblk%d", disk_id); |
| @@ -1239,6 +1199,9 @@ static int mspro_block_init_disk(struct memstick_dev *card) · 6 unmodified lines | ||
| 1239 | 1199 | set_capacity(msb->disk, capacity); |
| 1240 | 1200 | dev_dbg(&card->dev, "capacity set %ld\n", capacity); |
| 1241 | 1201 | |
| 1202 | + if (msb->read_only) | |
| 1203 | + set_disk_ro(msb->disk, true); | |
| 1204 | + | |
| 1242 | 1205 | rc = device_add_disk(&card->dev, msb->disk, NULL); |
| 1243 | 1206 | if (rc) |
| 1244 | 1207 | goto out_cleanup_disk; |
| @@ -1341,7 +1304,7 @@ static void mspro_block_remove(struct memstick_dev *card) · 96 unmodified lines | ||
| 1341 | 1304 | mspro_block_data_clear(msb); |
| 1342 | 1305 | mutex_unlock(&mspro_block_disk_lock); |
| 1343 | 1306 | |
| 1344 | − mspro_block_disk_release(msb->disk); | |
| 1307 | + put_disk(msb->disk); | |
| 1345 | 1308 | memstick_set_drvdata(card, NULL); |
| 1346 | 1309 | } |
| 1347 | 1310 | |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <linux/sched.h> |
| 20 | 20 | #include <linux/slab.h> |
| 21 | 21 | #include <linux/vmalloc.h> |
| 22 | −#include <linux/genhd.h> | |
| 22 | +#include <linux/blkdev.h> | |
| 23 | 23 | #include <linux/swap.h> |
| 24 | 24 | #include <linux/debugfs.h> |
| 25 | 25 | #include <linux/seq_file.h> |
| @@ -6,7 +6,6 @@ · 5 unmodified lines | ||
| 6 | 6 | * Based on Sharp's NAND driver sharp_sl.c |
| 7 | 7 | */ |
| 8 | 8 | |
| 9 | −#include <linux/genhd.h> | |
| 10 | 9 | #include <linux/slab.h> |
| 11 | 10 | #include <linux/module.h> |
| 12 | 11 | #include <linux/delay.h> |
| @@ -6,7 +6,6 @@ · 5 unmodified lines | ||
| 6 | 6 | |
| 7 | 7 | #include <linux/blkdev.h> |
| 8 | 8 | #include <linux/fs.h> |
| 9 | −#include <linux/genhd.h> | |
| 10 | 9 | #include <linux/module.h> |
| 11 | 10 | #include <linux/moduleparam.h> |
| 12 | 11 | #include <linux/nd.h> |
| @@ -11,7 +11,6 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/device.h> |
| 12 | 12 | #include <linux/mutex.h> |
| 13 | 13 | #include <linux/hdreg.h> |
| 14 | −#include <linux/genhd.h> | |
| 15 | 14 | #include <linux/sizes.h> |
| 16 | 15 | #include <linux/ndctl.h> |
| 17 | 16 | #include <linux/fs.h> |
| @@ -4,7 +4,6 @@ · 3 unmodified lines | ||
| 4 | 4 | */ |
| 5 | 5 | #include <linux/blkdev.h> |
| 6 | 6 | #include <linux/device.h> |
| 7 | −#include <linux/genhd.h> | |
| 8 | 7 | #include <linux/sizes.h> |
| 9 | 8 | #include <linux/slab.h> |
| 10 | 9 | #include <linux/fs.h> |
| @@ -11,7 +11,6 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/blkdev.h> |
| 12 | 12 | #include <linux/fcntl.h> |
| 13 | 13 | #include <linux/async.h> |
| 14 | −#include <linux/genhd.h> | |
| 15 | 14 | #include <linux/ndctl.h> |
| 16 | 15 | #include <linux/sched.h> |
| 17 | 16 | #include <linux/slab.h> |
| @@ -105,12 +105,12 @@ int async_pmem_flush(struct nd_region *nd_region, struct bio *bio) · 104 unmodified lines | ||
| 105 | 105 | * parent bio. Otherwise directly call nd_region flush. |
| 106 | 106 | */ |
| 107 | 107 | if (bio && bio->bi_iter.bi_sector != -1) { |
| 108 | − struct bio *child = bio_alloc(GFP_ATOMIC, 0); | |
| 108 | + struct bio *child = bio_alloc(bio->bi_bdev, 0, REQ_PREFLUSH, | |
| 109 | + GFP_ATOMIC); | |
| 109 | 110 | |
| 110 | 111 | if (!child) |
| 111 | 112 | return -ENOMEM; |
| 112 | − bio_copy_dev(child, bio); | |
| 113 | − child->bi_opf = REQ_PREFLUSH; | |
| 113 | + bio_clone_blkg_association(child, bio); | |
| 114 | 114 | child->bi_iter.bi_sector = -1; |
| 115 | 115 | bio_chain(child, bio); |
| 116 | 116 | submit_bio(child); |
| @@ -5,7 +5,6 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <linux/memremap.h> |
| 6 | 6 | #include <linux/blkdev.h> |
| 7 | 7 | #include <linux/device.h> |
| 8 | −#include <linux/genhd.h> | |
| 9 | 8 | #include <linux/sizes.h> |
| 10 | 9 | #include <linux/slab.h> |
| 11 | 10 | #include <linux/fs.h> |
| @@ -267,15 +267,15 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req) · 266 unmodified lines | ||
| 267 | 267 | |
| 268 | 268 | if (nvmet_use_inline_bvec(req)) { |
| 269 | 269 | bio = &req->b.inline_bio; |
| 270 | − bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec)); | |
| 270 | + bio_init(bio, req->ns->bdev, req->inline_bvec, | |
| 271 | + ARRAY_SIZE(req->inline_bvec), op); | |
| 271 | 272 | } else { |
| 272 | − bio = bio_alloc(GFP_KERNEL, bio_max_segs(sg_cnt)); | |
| 273 | + bio = bio_alloc(req->ns->bdev, bio_max_segs(sg_cnt), op, | |
| 274 | + GFP_KERNEL); | |
| 273 | 275 | } |
| 274 | − bio_set_dev(bio, req->ns->bdev); | |
| 275 | 276 | bio->bi_iter.bi_sector = sector; |
| 276 | 277 | bio->bi_private = req; |
| 277 | 278 | bio->bi_end_io = nvmet_bio_done; |
| 278 | − bio->bi_opf = op; | |
| 279 | 279 | |
| 280 | 280 | blk_start_plug(&plug); |
| 281 | 281 | if (req->metadata_len) |
| @@ -296,10 +296,9 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req) · 14 unmodified lines | ||
| 296 | 296 | } |
| 297 | 297 | } |
| 298 | 298 | |
| 299 | − bio = bio_alloc(GFP_KERNEL, bio_max_segs(sg_cnt)); | |
| 300 | − bio_set_dev(bio, req->ns->bdev); | |
| 299 | + bio = bio_alloc(req->ns->bdev, bio_max_segs(sg_cnt), | |
| 300 | + op, GFP_KERNEL); | |
| 301 | 301 | bio->bi_iter.bi_sector = sector; |
| 302 | − bio->bi_opf = op; | |
| 303 | 302 | |
| 304 | 303 | bio_chain(bio, prev); |
| 305 | 304 | submit_bio(prev); |
| @@ -328,11 +327,10 @@ static void nvmet_bdev_execute_flush(struct nvmet_req *req) · 22 unmodified lines | ||
| 328 | 327 | if (!nvmet_check_transfer_len(req, 0)) |
| 329 | 328 | return; |
| 330 | 329 | |
| 331 | − bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec)); | |
| 332 | − bio_set_dev(bio, req->ns->bdev); | |
| 330 | + bio_init(bio, req->ns->bdev, req->inline_bvec, | |
| 331 | + ARRAY_SIZE(req->inline_bvec), REQ_OP_WRITE | REQ_PREFLUSH); | |
| 333 | 332 | bio->bi_private = req; |
| 334 | 333 | bio->bi_end_io = nvmet_bio_done; |
| 335 | − bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 336 | 334 | |
| 337 | 335 | submit_bio(bio); |
| 338 | 336 | } |
| @@ -206,12 +206,13 @@ static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq) · 205 unmodified lines | ||
| 206 | 206 | |
| 207 | 207 | if (nvmet_use_inline_bvec(req)) { |
| 208 | 208 | bio = &req->p.inline_bio; |
| 209 | − bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec)); | |
| 209 | + bio_init(bio, NULL, req->inline_bvec, | |
| 210 | + ARRAY_SIZE(req->inline_bvec), req_op(rq)); | |
| 210 | 211 | } else { |
| 211 | − bio = bio_alloc(GFP_KERNEL, bio_max_segs(req->sg_cnt)); | |
| 212 | + bio = bio_alloc(NULL, bio_max_segs(req->sg_cnt), req_op(rq), | |
| 213 | + GFP_KERNEL); | |
| 212 | 214 | bio->bi_end_io = bio_put; |
| 213 | 215 | } |
| 214 | − bio->bi_opf = req_op(rq); | |
| 215 | 216 | |
| 216 | 217 | for_each_sg(req->sg, sg, req->sg_cnt, i) { |
| 217 | 218 | if (bio_add_pc_page(rq->q, bio, sg_page(sg), sg->length, |
| @@ -412,10 +412,10 @@ static u16 nvmet_bdev_zone_mgmt_emulate_all(struct nvmet_req *req) · 411 unmodified lines | ||
| 412 | 412 | |
| 413 | 413 | while (sector < get_capacity(bdev->bd_disk)) { |
| 414 | 414 | if (test_bit(blk_queue_zone_no(q, sector), d.zbitmap)) { |
| 415 | − bio = blk_next_bio(bio, 0, GFP_KERNEL); | |
| 416 | − bio->bi_opf = zsa_req_op(req->cmd->zms.zsa) | REQ_SYNC; | |
| 415 | + bio = blk_next_bio(bio, bdev, 0, | |
| 416 | + zsa_req_op(req->cmd->zms.zsa) | REQ_SYNC, | |
| 417 | + GFP_KERNEL); | |
| 417 | 418 | bio->bi_iter.bi_sector = sector; |
| 418 | − bio_set_dev(bio, bdev); | |
| 419 | 419 | /* This may take a while, so be nice to others */ |
| 420 | 420 | cond_resched(); |
| 421 | 421 | } |
| @@ -522,6 +522,7 @@ static void nvmet_bdev_zone_append_bio_done(struct bio *bio) · 100 unmodified lines | ||
| 522 | 522 | void nvmet_bdev_execute_zone_append(struct nvmet_req *req) |
| 523 | 523 | { |
| 524 | 524 | sector_t sect = nvmet_lba_to_sect(req->ns, req->cmd->rw.slba); |
| 525 | + const unsigned int op = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE; | |
| 525 | 526 | u16 status = NVME_SC_SUCCESS; |
| 526 | 527 | unsigned int total_len = 0; |
| 527 | 528 | struct scatterlist *sg; |
| @@ -551,14 +552,13 @@ void nvmet_bdev_execute_zone_append(struct nvmet_req *req) · 23 unmodified lines | ||
| 551 | 552 | |
| 552 | 553 | if (nvmet_use_inline_bvec(req)) { |
| 553 | 554 | bio = &req->z.inline_bio; |
| 554 | − bio_init(bio, req->inline_bvec, ARRAY_SIZE(req->inline_bvec)); | |
| 555 | + bio_init(bio, req->ns->bdev, req->inline_bvec, | |
| 556 | + ARRAY_SIZE(req->inline_bvec), op); | |
| 555 | 557 | } else { |
| 556 | − bio = bio_alloc(GFP_KERNEL, req->sg_cnt); | |
| 558 | + bio = bio_alloc(req->ns->bdev, req->sg_cnt, op, GFP_KERNEL); | |
| 557 | 559 | } |
| 558 | 560 | |
| 559 | − bio->bi_opf = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE; | |
| 560 | 561 | bio->bi_end_io = nvmet_bdev_zone_append_bio_done; |
| 561 | − bio_set_dev(bio, req->ns->bdev); | |
| 562 | 562 | bio->bi_iter.bi_sector = sect; |
| 563 | 563 | bio->bi_private = req; |
| 564 | 564 | if (req->cmd->rw.control & cpu_to_le16(NVME_RW_FUA)) |
| @@ -47,7 +47,6 @@ · 46 unmodified lines | ||
| 47 | 47 | #include <linux/module.h> |
| 48 | 48 | #include <linux/wait.h> |
| 49 | 49 | #include <linux/blkdev.h> |
| 50 | −#include <linux/genhd.h> | |
| 51 | 50 | #include <linux/hdreg.h> |
| 52 | 51 | #include <linux/interrupt.h> |
| 53 | 52 | #include <linux/log2.h> |
| @@ -15,7 +15,6 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/module.h> |
| 16 | 16 | #include <linux/blkdev.h> |
| 17 | 17 | #include <linux/blk-mq.h> |
| 18 | −#include <linux/genhd.h> | |
| 19 | 18 | #include <linux/slab.h> |
| 20 | 19 | #include <linux/list.h> |
| 21 | 20 | #include <asm/eadm.h> |
| @@ -6,7 +6,6 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <linux/spinlock.h> |
| 7 | 7 | #include <linux/blkdev.h> |
| 8 | 8 | #include <linux/blk-mq.h> |
| 9 | −#include <linux/genhd.h> | |
| 10 | 9 | #include <linux/list.h> |
| 11 | 10 | |
| 12 | 11 | #include <asm/debug.h> |
| @@ -23,7 +23,6 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <linux/slab.h> |
| 24 | 24 | #include <linux/types.h> |
| 25 | 25 | #include <linux/string.h> |
| 26 | −#include <linux/genhd.h> | |
| 27 | 26 | #include <linux/fs.h> |
| 28 | 27 | #include <linux/init.h> |
| 29 | 28 | #include <linux/proc_fs.h> |
| @@ -1276,7 +1276,7 @@ scsi_device_state_check(struct scsi_device *sdev, struct request *req) · 1275 unmodified lines | ||
| 1276 | 1276 | * power management commands. |
| 1277 | 1277 | */ |
| 1278 | 1278 | if (req && !(req->rq_flags & RQF_PM)) |
| 1279 | − return BLK_STS_IOERR; | |
| 1279 | + return BLK_STS_OFFLINE; | |
| 1280 | 1280 | return BLK_STS_OK; |
| 1281 | 1281 | } |
| 1282 | 1282 | } |
| @@ -14,7 +14,6 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <linux/module.h> |
| 15 | 15 | #include <linux/slab.h> |
| 16 | 16 | #include <linux/fs.h> |
| 17 | −#include <linux/genhd.h> | |
| 18 | 17 | #include <linux/kernel.h> |
| 19 | 18 | #include <linux/blkdev.h> |
| 20 | 19 | #include <linux/pagemap.h> |
| @@ -38,7 +38,6 @@ · 37 unmodified lines | ||
| 38 | 38 | #include <linux/kernel.h> |
| 39 | 39 | #include <linux/mm.h> |
| 40 | 40 | #include <linux/bio.h> |
| 41 | −#include <linux/genhd.h> | |
| 42 | 41 | #include <linux/hdreg.h> |
| 43 | 42 | #include <linux/errno.h> |
| 44 | 43 | #include <linux/idr.h> |
| @@ -122,11 +121,6 @@ static void scsi_disk_release(struct device *cdev); · 77 unmodified lines | ||
| 122 | 121 | |
| 123 | 122 | static DEFINE_IDA(sd_index_ida); |
| 124 | 123 | |
| 125 | −/* This semaphore is used to mediate the 0->1 reference get in the | |
| 126 | − * face of object destruction (i.e. we can't allow a get on an | |
| 127 | − * object after last put) */ | |
| 128 | −static DEFINE_MUTEX(sd_ref_mutex); | |
| 129 | − | |
| 130 | 124 | static struct kmem_cache *sd_cdb_cache; |
| 131 | 125 | static mempool_t *sd_cdb_pool; |
| 132 | 126 | static mempool_t *sd_page_pool; |
| @@ -664,33 +658,6 @@ static int sd_major(int major_idx) · 531 unmodified lines | ||
| 664 | 658 | } |
| 665 | 659 | } |
| 666 | 660 | |
| 667 | −static struct scsi_disk *scsi_disk_get(struct gendisk *disk) | |
| 668 | −{ | |
| 669 | − struct scsi_disk *sdkp = NULL; | |
| 670 | − | |
| 671 | − mutex_lock(&sd_ref_mutex); | |
| 672 | − | |
| 673 | − if (disk->private_data) { | |
| 674 | − sdkp = scsi_disk(disk); | |
| 675 | − if (scsi_device_get(sdkp->device) == 0) | |
| 676 | − get_device(&sdkp->dev); | |
| 677 | − else | |
| 678 | − sdkp = NULL; | |
| 679 | − } | |
| 680 | − mutex_unlock(&sd_ref_mutex); | |
| 681 | − return sdkp; | |
| 682 | −} | |
| 683 | − | |
| 684 | −static void scsi_disk_put(struct scsi_disk *sdkp) | |
| 685 | −{ | |
| 686 | − struct scsi_device *sdev = sdkp->device; | |
| 687 | − | |
| 688 | − mutex_lock(&sd_ref_mutex); | |
| 689 | − put_device(&sdkp->dev); | |
| 690 | − scsi_device_put(sdev); | |
| 691 | − mutex_unlock(&sd_ref_mutex); | |
| 692 | −} | |
| 693 | − | |
| 694 | 661 | #ifdef CONFIG_BLK_SED_OPAL |
| 695 | 662 | static int sd_sec_submit(void *data, u16 spsp, u8 secp, void *buffer, |
| 696 | 663 | size_t len, bool send) |
| @@ -1419,17 +1386,15 @@ static bool sd_need_revalidate(struct block_device *bdev, · 722 unmodified lines | ||
| 1419 | 1386 | **/ |
| 1420 | 1387 | static int sd_open(struct block_device *bdev, fmode_t mode) |
| 1421 | 1388 | { |
| 1422 | − struct scsi_disk *sdkp = scsi_disk_get(bdev->bd_disk); | |
| 1423 | − struct scsi_device *sdev; | |
| 1389 | + struct scsi_disk *sdkp = scsi_disk(bdev->bd_disk); | |
| 1390 | + struct scsi_device *sdev = sdkp->device; | |
| 1424 | 1391 | int retval; |
| 1425 | 1392 | |
| 1426 | − if (!sdkp) | |
| 1393 | + if (scsi_device_get(sdev)) | |
| 1427 | 1394 | return -ENXIO; |
| 1428 | 1395 | |
| 1429 | 1396 | SCSI_LOG_HLQUEUE(3, sd_printk(KERN_INFO, sdkp, "sd_open\n")); |
| 1430 | 1397 | |
| 1431 | − sdev = sdkp->device; | |
| 1432 | − | |
| 1433 | 1398 | /* |
| 1434 | 1399 | * If the device is in error recovery, wait until it is done. |
| 1435 | 1400 | * If the device is offline, then disallow any access to it. |
| @@ -1474,7 +1439,7 @@ static int sd_open(struct block_device *bdev, fmode_t mode) · 38 unmodified lines | ||
| 1474 | 1439 | return 0; |
| 1475 | 1440 | |
| 1476 | 1441 | error_out: |
| 1477 | − scsi_disk_put(sdkp); | |
| 1442 | + scsi_device_put(sdev); | |
| 1478 | 1443 | return retval; |
| 1479 | 1444 | } |
| 1480 | 1445 | |
| @@ -1503,7 +1468,7 @@ static void sd_release(struct gendisk *disk, fmode_t mode) · 22 unmodified lines | ||
| 1503 | 1468 | scsi_set_medium_removal(sdev, SCSI_REMOVAL_ALLOW); |
| 1504 | 1469 | } |
| 1505 | 1470 | |
| 1506 | − scsi_disk_put(sdkp); | |
| 1471 | + scsi_device_put(sdev); | |
| 1507 | 1472 | } |
| 1508 | 1473 | |
| 1509 | 1474 | static int sd_getgeo(struct block_device *bdev, struct hd_geometry *geo) |
| @@ -1617,7 +1582,7 @@ static int media_not_present(struct scsi_disk *sdkp, · 107 unmodified lines | ||
| 1617 | 1582 | **/ |
| 1618 | 1583 | static unsigned int sd_check_events(struct gendisk *disk, unsigned int clearing) |
| 1619 | 1584 | { |
| 1620 | − struct scsi_disk *sdkp = scsi_disk_get(disk); | |
| 1585 | + struct scsi_disk *sdkp = disk->private_data; | |
| 1621 | 1586 | struct scsi_device *sdp; |
| 1622 | 1587 | int retval; |
| 1623 | 1588 | bool disk_changed; |
| @@ -1680,7 +1645,6 @@ out: · 56 unmodified lines | ||
| 1680 | 1645 | */ |
| 1681 | 1646 | disk_changed = sdp->changed; |
| 1682 | 1647 | sdp->changed = 0; |
| 1683 | − scsi_disk_put(sdkp); | |
| 1684 | 1648 | return disk_changed ? DISK_EVENT_MEDIA_CHANGE : 0; |
| 1685 | 1649 | } |
| 1686 | 1650 | |
| @@ -1888,6 +1852,13 @@ static const struct pr_ops sd_pr_ops = { · 201 unmodified lines | ||
| 1888 | 1852 | .pr_clear = sd_pr_clear, |
| 1889 | 1853 | }; |
| 1890 | 1854 | |
| 1855 | +static void scsi_disk_free_disk(struct gendisk *disk) | |
| 1856 | +{ | |
| 1857 | + struct scsi_disk *sdkp = scsi_disk(disk); | |
| 1858 | + | |
| 1859 | + put_device(&sdkp->disk_dev); | |
| 1860 | +} | |
| 1861 | + | |
| 1891 | 1862 | static const struct block_device_operations sd_fops = { |
| 1892 | 1863 | .owner = THIS_MODULE, |
| 1893 | 1864 | .open = sd_open, |
| @@ -1899,6 +1870,7 @@ static const struct block_device_operations sd_fops = { · 5 unmodified lines | ||
| 1899 | 1870 | .unlock_native_capacity = sd_unlock_native_capacity, |
| 1900 | 1871 | .report_zones = sd_zbc_report_zones, |
| 1901 | 1872 | .get_unique_id = sd_get_unique_id, |
| 1873 | + .free_disk = scsi_disk_free_disk, | |
| 1902 | 1874 | .pr_ops = &sd_pr_ops, |
| 1903 | 1875 | }; |
| 1904 | 1876 | |
| @@ -3516,7 +3488,6 @@ static int sd_probe(struct device *dev) · 1611 unmodified lines | ||
| 3516 | 3488 | } |
| 3517 | 3489 | |
| 3518 | 3490 | sdkp->device = sdp; |
| 3519 | − sdkp->driver = &sd_template; | |
| 3520 | 3491 | sdkp->disk = gd; |
| 3521 | 3492 | sdkp->index = index; |
| 3522 | 3493 | sdkp->max_retries = SD_MAX_RETRIES; |
| @@ -3531,14 +3502,14 @@ static int sd_probe(struct device *dev) · 8 unmodified lines | ||
| 3531 | 3502 | SD_MOD_TIMEOUT); |
| 3532 | 3503 | } |
| 3533 | 3504 | |
| 3534 | − device_initialize(&sdkp->dev); | |
| 3535 | − sdkp->dev.parent = get_device(dev); | |
| 3536 | − sdkp->dev.class = &sd_disk_class; | |
| 3537 | − dev_set_name(&sdkp->dev, "%s", dev_name(dev)); | |
| 3505 | + device_initialize(&sdkp->disk_dev); | |
| 3506 | + sdkp->disk_dev.parent = get_device(dev); | |
| 3507 | + sdkp->disk_dev.class = &sd_disk_class; | |
| 3508 | + dev_set_name(&sdkp->disk_dev, "%s", dev_name(dev)); | |
| 3538 | 3509 | |
| 3539 | − error = device_add(&sdkp->dev); | |
| 3510 | + error = device_add(&sdkp->disk_dev); | |
| 3540 | 3511 | if (error) { |
| 3541 | − put_device(&sdkp->dev); | |
| 3512 | + put_device(&sdkp->disk_dev); | |
| 3542 | 3513 | goto out; |
| 3543 | 3514 | } |
| 3544 | 3515 | |
| @@ -3549,7 +3520,7 @@ static int sd_probe(struct device *dev) · 4 unmodified lines | ||
| 3549 | 3520 | gd->minors = SD_MINORS; |
| 3550 | 3521 | |
| 3551 | 3522 | gd->fops = &sd_fops; |
| 3552 | − gd->private_data = &sdkp->driver; | |
| 3523 | + gd->private_data = sdkp; | |
| 3553 | 3524 | |
| 3554 | 3525 | /* defaults, until the device tells us otherwise */ |
| 3555 | 3526 | sdp->sector_size = 512; |
| @@ -3579,7 +3550,7 @@ static int sd_probe(struct device *dev) · 23 unmodified lines | ||
| 3579 | 3550 | |
| 3580 | 3551 | error = device_add_disk(dev, gd, NULL); |
| 3581 | 3552 | if (error) { |
| 3582 | − put_device(&sdkp->dev); | |
| 3553 | + put_device(&sdkp->disk_dev); | |
| 3583 | 3554 | goto out; |
| 3584 | 3555 | } |
| 3585 | 3556 | |
| @@ -3625,58 +3596,26 @@ static int sd_probe(struct device *dev) · 39 unmodified lines | ||
| 3625 | 3596 | **/ |
| 3626 | 3597 | static int sd_remove(struct device *dev) |
| 3627 | 3598 | { |
| 3628 | − struct scsi_disk *sdkp; | |
| 3599 | + struct scsi_disk *sdkp = dev_get_drvdata(dev); | |
| 3629 | 3600 | |
| 3630 | − sdkp = dev_get_drvdata(dev); | |
| 3631 | 3601 | scsi_autopm_get_device(sdkp->device); |
| 3632 | 3602 | |
| 3633 | − device_del(&sdkp->dev); | |
| 3603 | + device_del(&sdkp->disk_dev); | |
| 3634 | 3604 | del_gendisk(sdkp->disk); |
| 3635 | 3605 | sd_shutdown(dev); |
| 3636 | 3606 | |
| 3637 | − free_opal_dev(sdkp->opal_dev); | |
| 3638 | − | |
| 3639 | − mutex_lock(&sd_ref_mutex); | |
| 3640 | − dev_set_drvdata(dev, NULL); | |
| 3641 | − put_device(&sdkp->dev); | |
| 3642 | − mutex_unlock(&sd_ref_mutex); | |
| 3643 | − | |
| 3607 | + put_disk(sdkp->disk); | |
| 3644 | 3608 | return 0; |
| 3645 | 3609 | } |
| 3646 | 3610 | |
| 3647 | −/** | |
| 3648 | − * scsi_disk_release - Called to free the scsi_disk structure | |
| 3649 | − * @dev: pointer to embedded class device | |
| 3650 | − * | |
| 3651 | − * sd_ref_mutex must be held entering this routine. Because it is | |
| 3652 | − * called on last put, you should always use the scsi_disk_get() | |
| 3653 | − * scsi_disk_put() helpers which manipulate the semaphore directly | |
| 3654 | − * and never do a direct put_device. | |
| 3655 | − **/ | |
| 3656 | 3611 | static void scsi_disk_release(struct device *dev) |
| 3657 | 3612 | { |
| 3658 | 3613 | struct scsi_disk *sdkp = to_scsi_disk(dev); |
| 3659 | − struct gendisk *disk = sdkp->disk; | |
| 3660 | − struct request_queue *q = disk->queue; | |
| 3661 | 3614 | |
| 3662 | 3615 | ida_free(&sd_index_ida, sdkp->index); |
| 3663 | − | |
| 3664 | − /* | |
| 3665 | − * Wait until all requests that are in progress have completed. | |
| 3666 | − * This is necessary to avoid that e.g. scsi_end_request() crashes | |
| 3667 | − * due to clearing the disk->private_data pointer. Wait from inside | |
| 3668 | − * scsi_disk_release() instead of from sd_release() to avoid that | |
| 3669 | − * freezing and unfreezing the request queue affects user space I/O | |
| 3670 | − * in case multiple processes open a /dev/sd... node concurrently. | |
| 3671 | − */ | |
| 3672 | − blk_mq_freeze_queue(q); | |
| 3673 | − blk_mq_unfreeze_queue(q); | |
| 3674 | − | |
| 3675 | − disk->private_data = NULL; | |
| 3676 | − put_disk(disk); | |
| 3677 | − put_device(&sdkp->device->sdev_gendev); | |
| 3678 | − | |
| 3679 | 3616 | sd_zbc_release_disk(sdkp); |
| 3617 | + put_device(&sdkp->device->sdev_gendev); | |
| 3618 | + free_opal_dev(sdkp->opal_dev); | |
| 3680 | 3619 | |
| 3681 | 3620 | kfree(sdkp); |
| 3682 | 3621 | } |
| @@ -68,9 +68,13 @@ enum { · 67 unmodified lines | ||
| 68 | 68 | }; |
| 69 | 69 | |
| 70 | 70 | struct scsi_disk { |
| 71 | − struct scsi_driver *driver; /* always &sd_template */ | |
| 72 | 71 | struct scsi_device *device; |
| 73 | − struct device dev; | |
| 72 | + | |
| 73 | + /* | |
| 74 | + * disk_dev is used to show attributes in /sys/class/scsi_disk/, | |
| 75 | + * but otherwise not really needed. Do not use for refcounting. | |
| 76 | + */ | |
| 77 | + struct device disk_dev; | |
| 74 | 78 | struct gendisk *disk; |
| 75 | 79 | struct opal_dev *opal_dev; |
| 76 | 80 | #ifdef CONFIG_BLK_DEV_ZONED |
| @@ -127,11 +131,11 @@ struct scsi_disk { · 50 unmodified lines | ||
| 127 | 131 | unsigned security : 1; |
| 128 | 132 | unsigned ignore_medium_access_errors : 1; |
| 129 | 133 | }; |
| 130 | −#define to_scsi_disk(obj) container_of(obj,struct scsi_disk,dev) | |
| 134 | +#define to_scsi_disk(obj) container_of(obj, struct scsi_disk, disk_dev) | |
| 131 | 135 | |
| 132 | 136 | static inline struct scsi_disk *scsi_disk(struct gendisk *disk) |
| 133 | 137 | { |
| 134 | − return container_of(disk->private_data, struct scsi_disk, driver); | |
| 138 | + return disk->private_data; | |
| 135 | 139 | } |
| 136 | 140 | |
| 137 | 141 | #define sd_printk(prefix, sdsk, fmt, a...) \ |
| @@ -109,11 +109,6 @@ static DEFINE_SPINLOCK(sr_index_lock); · 108 unmodified lines | ||
| 109 | 109 | |
| 110 | 110 | static struct lock_class_key sr_bio_compl_lkclass; |
| 111 | 111 | |
| 112 | −/* This semaphore is used to mediate the 0->1 reference get in the | |
| 113 | − * face of object destruction (i.e. we can't allow a get on an | |
| 114 | − * object after last put) */ | |
| 115 | −static DEFINE_MUTEX(sr_ref_mutex); | |
| 116 | − | |
| 117 | 112 | static int sr_open(struct cdrom_device_info *, int); |
| 118 | 113 | static void sr_release(struct cdrom_device_info *); |
| 119 | 114 | |
| @@ -143,11 +138,9 @@ static const struct cdrom_device_ops sr_dops = { · 23 unmodified lines | ||
| 143 | 138 | .capability = SR_CAPABILITIES, |
| 144 | 139 | }; |
| 145 | 140 | |
| 146 | −static void sr_kref_release(struct kref *kref); | |
| 147 | − | |
| 148 | 141 | static inline struct scsi_cd *scsi_cd(struct gendisk *disk) |
| 149 | 142 | { |
| 150 | − return container_of(disk->private_data, struct scsi_cd, driver); | |
| 143 | + return disk->private_data; | |
| 151 | 144 | } |
| 152 | 145 | |
| 153 | 146 | static int sr_runtime_suspend(struct device *dev) |
| @@ -163,38 +156,6 @@ static int sr_runtime_suspend(struct device *dev) · 9 unmodified lines | ||
| 163 | 156 | return 0; |
| 164 | 157 | } |
| 165 | 158 | |
| 166 | −/* | |
| 167 | − * The get and put routines for the struct scsi_cd. Note this entity | |
| 168 | − * has a scsi_device pointer and owns a reference to this. | |
| 169 | − */ | |
| 170 | −static inline struct scsi_cd *scsi_cd_get(struct gendisk *disk) | |
| 171 | −{ | |
| 172 | − struct scsi_cd *cd = NULL; | |
| 173 | − | |
| 174 | − mutex_lock(&sr_ref_mutex); | |
| 175 | − if (disk->private_data == NULL) | |
| 176 | − goto out; | |
| 177 | − cd = scsi_cd(disk); | |
| 178 | − kref_get(&cd->kref); | |
| 179 | − if (scsi_device_get(cd->device)) { | |
| 180 | − kref_put(&cd->kref, sr_kref_release); | |
| 181 | − cd = NULL; | |
| 182 | − } | |
| 183 | − out: | |
| 184 | − mutex_unlock(&sr_ref_mutex); | |
| 185 | − return cd; | |
| 186 | −} | |
| 187 | − | |
| 188 | −static void scsi_cd_put(struct scsi_cd *cd) | |
| 189 | −{ | |
| 190 | − struct scsi_device *sdev = cd->device; | |
| 191 | − | |
| 192 | − mutex_lock(&sr_ref_mutex); | |
| 193 | − kref_put(&cd->kref, sr_kref_release); | |
| 194 | − scsi_device_put(sdev); | |
| 195 | − mutex_unlock(&sr_ref_mutex); | |
| 196 | −} | |
| 197 | − | |
| 198 | 159 | static unsigned int sr_get_events(struct scsi_device *sdev) |
| 199 | 160 | { |
| 200 | 161 | u8 buf[8]; |
| @@ -522,15 +483,13 @@ static void sr_revalidate_disk(struct scsi_cd *cd) · 321 unmodified lines | ||
| 522 | 483 | |
| 523 | 484 | static int sr_block_open(struct block_device *bdev, fmode_t mode) |
| 524 | 485 | { |
| 525 | − struct scsi_cd *cd; | |
| 526 | − struct scsi_device *sdev; | |
| 527 | − int ret = -ENXIO; | |
| 486 | + struct scsi_cd *cd = scsi_cd(bdev->bd_disk); | |
| 487 | + struct scsi_device *sdev = cd->device; | |
| 488 | + int ret; | |
| 528 | 489 | |
| 529 | − cd = scsi_cd_get(bdev->bd_disk); | |
| 530 | − if (!cd) | |
| 531 | − goto out; | |
| 490 | + if (scsi_device_get(cd->device)) | |
| 491 | + return -ENXIO; | |
| 532 | 492 | |
| 533 | − sdev = cd->device; | |
| 534 | 493 | scsi_autopm_get_device(sdev); |
| 535 | 494 | if (bdev_check_media_change(bdev)) |
| 536 | 495 | sr_revalidate_disk(cd); |
| @@ -541,9 +500,7 @@ static int sr_block_open(struct block_device *bdev, fmode_t mode) · 4 unmodified lines | ||
| 541 | 500 | |
| 542 | 501 | scsi_autopm_put_device(sdev); |
| 543 | 502 | if (ret) |
| 544 | − scsi_cd_put(cd); | |
| 545 | − | |
| 546 | −out: | |
| 503 | + scsi_device_put(cd->device); | |
| 547 | 504 | return ret; |
| 548 | 505 | } |
| 549 | 506 | |
| @@ -555,7 +512,7 @@ static void sr_block_release(struct gendisk *disk, fmode_t mode) · 5 unmodified lines | ||
| 555 | 512 | cdrom_release(&cd->cdi, mode); |
| 556 | 513 | mutex_unlock(&cd->lock); |
| 557 | 514 | |
| 558 | − scsi_cd_put(cd); | |
| 515 | + scsi_device_put(cd->device); | |
| 559 | 516 | } |
| 560 | 517 | |
| 561 | 518 | static int sr_block_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd, |
| @@ -595,18 +552,24 @@ out: · 33 unmodified lines | ||
| 595 | 552 | static unsigned int sr_block_check_events(struct gendisk *disk, |
| 596 | 553 | unsigned int clearing) |
| 597 | 554 | { |
| 598 | − unsigned int ret = 0; | |
| 599 | − struct scsi_cd *cd; | |
| 555 | + struct scsi_cd *cd = disk->private_data; | |
| 600 | 556 | |
| 601 | − cd = scsi_cd_get(disk); | |
| 602 | − if (!cd) | |
| 557 | + if (atomic_read(&cd->device->disk_events_disable_depth)) | |
| 603 | 558 | return 0; |
| 559 | + return cdrom_check_events(&cd->cdi, clearing); | |
| 560 | +} | |
| 604 | 561 | |
| 605 | − if (!atomic_read(&cd->device->disk_events_disable_depth)) | |
| 606 | − ret = cdrom_check_events(&cd->cdi, clearing); | |
| 562 | +static void sr_free_disk(struct gendisk *disk) | |
| 563 | +{ | |
| 564 | + struct scsi_cd *cd = disk->private_data; | |
| 607 | 565 | |
| 608 | − scsi_cd_put(cd); | |
| 609 | − return ret; | |
| 566 | + spin_lock(&sr_index_lock); | |
| 567 | + clear_bit(MINOR(disk_devt(disk)), sr_index_bits); | |
| 568 | + spin_unlock(&sr_index_lock); | |
| 569 | + | |
| 570 | + unregister_cdrom(&cd->cdi); | |
| 571 | + mutex_destroy(&cd->lock); | |
| 572 | + kfree(cd); | |
| 610 | 573 | } |
| 611 | 574 | |
| 612 | 575 | static const struct block_device_operations sr_bdops = |
| @@ -617,6 +580,7 @@ static const struct block_device_operations sr_bdops = · 4 unmodified lines | ||
| 617 | 580 | .ioctl = sr_block_ioctl, |
| 618 | 581 | .compat_ioctl = blkdev_compat_ptr_ioctl, |
| 619 | 582 | .check_events = sr_block_check_events, |
| 583 | + .free_disk = sr_free_disk, | |
| 620 | 584 | }; |
| 621 | 585 | |
| 622 | 586 | static int sr_open(struct cdrom_device_info *cdi, int purpose) |
| @@ -660,8 +624,6 @@ static int sr_probe(struct device *dev) · 37 unmodified lines | ||
| 660 | 624 | if (!cd) |
| 661 | 625 | goto fail; |
| 662 | 626 | |
| 663 | − kref_init(&cd->kref); | |
| 664 | − | |
| 665 | 627 | disk = __alloc_disk_node(sdev->request_queue, NUMA_NO_NODE, |
| 666 | 628 | &sr_bio_compl_lkclass); |
| 667 | 629 | if (!disk) |
| @@ -692,7 +654,6 @@ static int sr_probe(struct device *dev) · 24 unmodified lines | ||
| 692 | 654 | |
| 693 | 655 | cd->device = sdev; |
| 694 | 656 | cd->disk = disk; |
| 695 | − cd->driver = &sr_template; | |
| 696 | 657 | cd->capacity = 0x1fffff; |
| 697 | 658 | cd->device->changed = 1; /* force recheck CD type */ |
| 698 | 659 | cd->media_present = 1; |
| @@ -713,7 +674,7 @@ static int sr_probe(struct device *dev) · 14 unmodified lines | ||
| 713 | 674 | sr_vendor_init(cd); |
| 714 | 675 | |
| 715 | 676 | set_capacity(disk, cd->capacity); |
| 716 | − disk->private_data = &cd->driver; | |
| 677 | + disk->private_data = cd; | |
| 717 | 678 | |
| 718 | 679 | if (register_cdrom(disk, &cd->cdi)) |
| 719 | 680 | goto fail_minor; |
| @@ -728,10 +689,8 @@ static int sr_probe(struct device *dev) · 8 unmodified lines | ||
| 728 | 689 | sr_revalidate_disk(cd); |
| 729 | 690 | |
| 730 | 691 | error = device_add_disk(&sdev->sdev_gendev, disk, NULL); |
| 731 | − if (error) { | |
| 732 | − kref_put(&cd->kref, sr_kref_release); | |
| 733 | − goto fail; | |
| 734 | − } | |
| 692 | + if (error) | |
| 693 | + goto unregister_cdrom; | |
| 735 | 694 | |
| 736 | 695 | sdev_printk(KERN_DEBUG, sdev, |
| 737 | 696 | "Attached scsi CD-ROM %s\n", cd->cdi.name); |
| @@ -739,6 +698,8 @@ static int sr_probe(struct device *dev) · 1 unmodified line | ||
| 739 | 698 | |
| 740 | 699 | return 0; |
| 741 | 700 | |
| 701 | +unregister_cdrom: | |
| 702 | + unregister_cdrom(&cd->cdi); | |
| 742 | 703 | fail_minor: |
| 743 | 704 | spin_lock(&sr_index_lock); |
| 744 | 705 | clear_bit(minor, sr_index_bits); |
| @@ -1010,36 +971,6 @@ out_put_request: · 265 unmodified lines | ||
| 1010 | 971 | return ret; |
| 1011 | 972 | } |
| 1012 | 973 | |
| 1013 | − | |
| 1014 | −/** | |
| 1015 | − * sr_kref_release - Called to free the scsi_cd structure | |
| 1016 | − * @kref: pointer to embedded kref | |
| 1017 | − * | |
| 1018 | − * sr_ref_mutex must be held entering this routine. Because it is | |
| 1019 | − * called on last put, you should always use the scsi_cd_get() | |
| 1020 | − * scsi_cd_put() helpers which manipulate the semaphore directly | |
| 1021 | − * and never do a direct kref_put(). | |
| 1022 | − **/ | |
| 1023 | −static void sr_kref_release(struct kref *kref) | |
| 1024 | −{ | |
| 1025 | − struct scsi_cd *cd = container_of(kref, struct scsi_cd, kref); | |
| 1026 | − struct gendisk *disk = cd->disk; | |
| 1027 | − | |
| 1028 | − spin_lock(&sr_index_lock); | |
| 1029 | − clear_bit(MINOR(disk_devt(disk)), sr_index_bits); | |
| 1030 | − spin_unlock(&sr_index_lock); | |
| 1031 | − | |
| 1032 | − unregister_cdrom(&cd->cdi); | |
| 1033 | − | |
| 1034 | − disk->private_data = NULL; | |
| 1035 | − | |
| 1036 | − put_disk(disk); | |
| 1037 | − | |
| 1038 | − mutex_destroy(&cd->lock); | |
| 1039 | − | |
| 1040 | − kfree(cd); | |
| 1041 | −} | |
| 1042 | − | |
| 1043 | 974 | static int sr_remove(struct device *dev) |
| 1044 | 975 | { |
| 1045 | 976 | struct scsi_cd *cd = dev_get_drvdata(dev); |
| @@ -1047,11 +978,7 @@ static int sr_remove(struct device *dev) · 1 unmodified line | ||
| 1047 | 978 | scsi_autopm_get_device(cd->device); |
| 1048 | 979 | |
| 1049 | 980 | del_gendisk(cd->disk); |
| 1050 | − dev_set_drvdata(dev, NULL); | |
| 1051 | − | |
| 1052 | − mutex_lock(&sr_ref_mutex); | |
| 1053 | − kref_put(&cd->kref, sr_kref_release); | |
| 1054 | − mutex_unlock(&sr_ref_mutex); | |
| 981 | + put_disk(cd->disk); | |
| 1055 | 982 | |
| 1056 | 983 | return 0; |
| 1057 | 984 | } |
| @@ -18,8 +18,6 @@ · 17 unmodified lines | ||
| 18 | 18 | #ifndef _SR_H |
| 19 | 19 | #define _SR_H |
| 20 | 20 | |
| 21 | −#include <linux/genhd.h> | |
| 22 | −#include <linux/kref.h> | |
| 23 | 21 | #include <linux/mutex.h> |
| 24 | 22 | |
| 25 | 23 | #define MAX_RETRIES 3 |
| @@ -33,7 +31,6 @@ struct scsi_device; · 7 unmodified lines | ||
| 33 | 31 | |
| 34 | 32 | |
| 35 | 33 | typedef struct scsi_cd { |
| 36 | − struct scsi_driver *driver; | |
| 37 | 34 | unsigned capacity; /* size in blocks */ |
| 38 | 35 | struct scsi_device *device; |
| 39 | 36 | unsigned int vendor; /* vendor code, see sr_vendor.c */ |
| @@ -53,9 +50,6 @@ typedef struct scsi_cd { · 13 unmodified lines | ||
| 53 | 50 | |
| 54 | 51 | struct cdrom_device_info cdi; |
| 55 | 52 | struct mutex lock; |
| 56 | − /* We hold gendisk and scsi_device references on probe and use | |
| 57 | − * the refs on this kref to decide when to release them */ | |
| 58 | − struct kref kref; | |
| 59 | 53 | struct gendisk *disk; |
| 60 | 54 | } Scsi_CD; |
| 61 | 55 | |
| @@ -4276,7 +4276,6 @@ static int st_probe(struct device *dev) · 4275 unmodified lines | ||
| 4276 | 4276 | goto out_buffer_free; |
| 4277 | 4277 | } |
| 4278 | 4278 | kref_init(&tpnt->kref); |
| 4279 | − tpnt->driver = &st_template; | |
| 4280 | 4279 | |
| 4281 | 4280 | tpnt->device = SDp; |
| 4282 | 4281 | if (SDp->scsi_level <= 2) |
| @@ -117,7 +117,6 @@ struct scsi_tape_stats { · 116 unmodified lines | ||
| 117 | 117 | |
| 118 | 118 | /* The tape drive descriptor */ |
| 119 | 119 | struct scsi_tape { |
| 120 | − struct scsi_driver *driver; | |
| 121 | 120 | struct scsi_device *device; |
| 122 | 121 | struct mutex lock; /* For serialization */ |
| 123 | 122 | struct completion wait; /* For SCSI commands */ |
| @@ -494,7 +494,7 @@ static struct ufshpb_req *ufshpb_get_map_req(struct ufshpb_lu *hpb, · 493 unmodified lines | ||
| 494 | 494 | if (!map_req) |
| 495 | 495 | return NULL; |
| 496 | 496 | |
| 497 | − bio = bio_alloc(GFP_KERNEL, hpb->pages_per_srgn); | |
| 497 | + bio = bio_alloc(NULL, hpb->pages_per_srgn, 0, GFP_KERNEL); | |
| 498 | 498 | if (!bio) { |
| 499 | 499 | ufshpb_put_req(hpb, map_req); |
| 500 | 500 | return NULL; |
| @@ -2050,7 +2050,7 @@ static int ufshpb_pre_req_mempool_init(struct ufshpb_lu *hpb) · 1549 unmodified lines | ||
| 2050 | 2050 | INIT_LIST_HEAD(&pre_req->list_req); |
| 2051 | 2051 | pre_req->req = NULL; |
| 2052 | 2052 | |
| 2053 | − pre_req->bio = bio_alloc(GFP_KERNEL, 1); | |
| 2053 | + pre_req->bio = bio_alloc(NULL, 1, 0, GFP_KERNEL); | |
| 2054 | 2054 | if (!pre_req->bio) |
| 2055 | 2055 | goto release_mem; |
| 2056 | 2056 | |
| @@ -20,7 +20,6 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <linux/slab.h> |
| 21 | 21 | #include <linux/spinlock.h> |
| 22 | 22 | #include <linux/bio.h> |
| 23 | −#include <linux/genhd.h> | |
| 24 | 23 | #include <linux/file.h> |
| 25 | 24 | #include <linux/module.h> |
| 26 | 25 | #include <linux/scatterlist.h> |
| @@ -353,18 +352,16 @@ static struct bio *iblock_get_bio(struct se_cmd *cmd, sector_t lba, u32 sg_num, · 326 unmodified lines | ||
| 353 | 352 | * Only allocate as many vector entries as the bio code allows us to, |
| 354 | 353 | * we'll loop later on until we have handled the whole request. |
| 355 | 354 | */ |
| 356 | − bio = bio_alloc_bioset(GFP_NOIO, bio_max_segs(sg_num), | |
| 357 | − &ib_dev->ibd_bio_set); | |
| 355 | + bio = bio_alloc_bioset(ib_dev->ibd_bd, bio_max_segs(sg_num), opf, | |
| 356 | + GFP_NOIO, &ib_dev->ibd_bio_set); | |
| 358 | 357 | if (!bio) { |
| 359 | 358 | pr_err("Unable to allocate memory for bio\n"); |
| 360 | 359 | return NULL; |
| 361 | 360 | } |
| 362 | 361 | |
| 363 | − bio_set_dev(bio, ib_dev->ibd_bd); | |
| 364 | 362 | bio->bi_private = cmd; |
| 365 | 363 | bio->bi_end_io = &iblock_bio_done; |
| 366 | 364 | bio->bi_iter.bi_sector = lba; |
| 367 | − bio->bi_opf = opf; | |
| 368 | 365 | |
| 369 | 366 | return bio; |
| 370 | 367 | } |
| @@ -418,10 +415,9 @@ iblock_execute_sync_cache(struct se_cmd *cmd) · 47 unmodified lines | ||
| 418 | 415 | if (immed) |
| 419 | 416 | target_complete_cmd(cmd, SAM_STAT_GOOD); |
| 420 | 417 | |
| 421 | − bio = bio_alloc(GFP_KERNEL, 0); | |
| 418 | + bio = bio_alloc(ib_dev->ibd_bd, 0, REQ_OP_WRITE | REQ_PREFLUSH, | |
| 419 | + GFP_KERNEL); | |
| 422 | 420 | bio->bi_end_io = iblock_end_io_flush; |
| 423 | − bio_set_dev(bio, ib_dev->ibd_bd); | |
| 424 | − bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; | |
| 425 | 421 | if (!immed) |
| 426 | 422 | bio->bi_private = cmd; |
| 427 | 423 | submit_bio(bio); |
| @@ -17,7 +17,6 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <linux/blk_types.h> |
| 18 | 18 | #include <linux/slab.h> |
| 19 | 19 | #include <linux/spinlock.h> |
| 20 | −#include <linux/genhd.h> | |
| 21 | 20 | #include <linux/cdrom.h> |
| 22 | 21 | #include <linux/ratelimit.h> |
| 23 | 22 | #include <linux/module.h> |
| @@ -78,7 +78,6 @@ · 77 unmodified lines | ||
| 78 | 78 | #include <linux/sched.h> |
| 79 | 79 | #include <linux/slab.h> |
| 80 | 80 | #include <linux/mutex.h> |
| 81 | −#include <linux/genhd.h> | |
| 82 | 81 | #include <linux/blkdev.h> |
| 83 | 82 | #include <linux/mm.h> |
| 84 | 83 | #include <linux/string.h> |
| @@ -4033,8 +4033,9 @@ static int write_dev_supers(struct btrfs_device *device, · 4032 unmodified lines | ||
| 4033 | 4033 | * to do I/O, so we don't lose the ability to do integrity |
| 4034 | 4034 | * checking. |
| 4035 | 4035 | */ |
| 4036 | − bio = bio_alloc(GFP_NOFS, 1); | |
| 4037 | − bio_set_dev(bio, device->bdev); | |
| 4036 | + bio = bio_alloc(device->bdev, 1, | |
| 4037 | + REQ_OP_WRITE | REQ_SYNC | REQ_META | REQ_PRIO, | |
| 4038 | + GFP_NOFS); | |
| 4038 | 4039 | bio->bi_iter.bi_sector = bytenr >> SECTOR_SHIFT; |
| 4039 | 4040 | bio->bi_private = device; |
| 4040 | 4041 | bio->bi_end_io = btrfs_end_super_write; |
| @@ -4046,7 +4047,6 @@ static int write_dev_supers(struct btrfs_device *device, · 5 unmodified lines | ||
| 4046 | 4047 | * go down lazy and there's a short window where the on-disk |
| 4047 | 4048 | * copies might still contain the older version. |
| 4048 | 4049 | */ |
| 4049 | − bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_META | REQ_PRIO; | |
| 4050 | 4050 | if (i == 0 && !btrfs_test_opt(device->fs_info, NOBARRIER)) |
| 4051 | 4051 | bio->bi_opf |= REQ_FUA; |
| 4052 | 4052 | |
| @@ -4158,10 +4158,8 @@ static void write_dev_flush(struct btrfs_device *device) · 105 unmodified lines | ||
| 4158 | 4158 | return; |
| 4159 | 4159 | #endif |
| 4160 | 4160 | |
| 4161 | − bio_reset(bio); | |
| 4161 | + bio_reset(bio, device->bdev, REQ_OP_WRITE | REQ_SYNC | REQ_PREFLUSH); | |
| 4162 | 4162 | bio->bi_end_io = btrfs_end_empty_barrier; |
| 4163 | − bio_set_dev(bio, device->bdev); | |
| 4164 | − bio->bi_opf = REQ_OP_WRITE | REQ_SYNC | REQ_PREFLUSH; | |
| 4165 | 4163 | init_completion(&device->flush_wait); |
| 4166 | 4164 | bio->bi_private = &device->flush_wait; |
| 4167 | 4165 | |
| @@ -3143,7 +3143,7 @@ struct bio *btrfs_bio_alloc(unsigned int nr_iovecs) · 3142 unmodified lines | ||
| 3143 | 3143 | struct bio *bio; |
| 3144 | 3144 | |
| 3145 | 3145 | ASSERT(0 < nr_iovecs && nr_iovecs <= BIO_MAX_VECS); |
| 3146 | − bio = bio_alloc_bioset(GFP_NOFS, nr_iovecs, &btrfs_bioset); | |
| 3146 | + bio = bio_alloc_bioset(NULL, nr_iovecs, 0, GFP_NOFS, &btrfs_bioset); | |
| 3147 | 3147 | btrfs_bio_init(btrfs_bio(bio)); |
| 3148 | 3148 | return bio; |
| 3149 | 3149 | } |
| @@ -3154,7 +3154,7 @@ struct bio *btrfs_bio_clone(struct bio *bio) · 4 unmodified lines | ||
| 3154 | 3154 | struct bio *new; |
| 3155 | 3155 | |
| 3156 | 3156 | /* Bio allocation backed by a bioset does not fail */ |
| 3157 | − new = bio_clone_fast(bio, GFP_NOFS, &btrfs_bioset); | |
| 3157 | + new = bio_alloc_clone(bio->bi_bdev, bio, GFP_NOFS, &btrfs_bioset); | |
| 3158 | 3158 | bbio = btrfs_bio(new); |
| 3159 | 3159 | btrfs_bio_init(bbio); |
| 3160 | 3160 | bbio->iter = bio->bi_iter; |
| @@ -3169,7 +3169,7 @@ struct bio *btrfs_bio_clone_partial(struct bio *orig, u64 offset, u64 size) · 8 unmodified lines | ||
| 3169 | 3169 | ASSERT(offset <= UINT_MAX && size <= UINT_MAX); |
| 3170 | 3170 | |
| 3171 | 3171 | /* this will never fail when it's backed by a bioset */ |
| 3172 | − bio = bio_clone_fast(orig, GFP_NOFS, &btrfs_bioset); | |
| 3172 | + bio = bio_alloc_clone(orig->bi_bdev, orig, GFP_NOFS, &btrfs_bioset); | |
| 3173 | 3173 | ASSERT(bio); |
| 3174 | 3174 | |
| 3175 | 3175 | bbio = btrfs_bio(bio); |
| @@ -3024,12 +3024,16 @@ static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, · 3023 unmodified lines | ||
| 3024 | 3024 | if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE)) |
| 3025 | 3025 | clear_buffer_write_io_error(bh); |
| 3026 | 3026 | |
| 3027 | − bio = bio_alloc(GFP_NOIO, 1); | |
| 3027 | + if (buffer_meta(bh)) | |
| 3028 | + op_flags |= REQ_META; | |
| 3029 | + if (buffer_prio(bh)) | |
| 3030 | + op_flags |= REQ_PRIO; | |
| 3031 | + | |
| 3032 | + bio = bio_alloc(bh->b_bdev, 1, op | op_flags, GFP_NOIO); | |
| 3028 | 3033 | |
| 3029 | 3034 | fscrypt_set_bio_crypt_ctx_bh(bio, bh, GFP_NOIO); |
| 3030 | 3035 | |
| 3031 | 3036 | bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9); |
| 3032 | − bio_set_dev(bio, bh->b_bdev); | |
| 3033 | 3037 | bio->bi_write_hint = write_hint; |
| 3034 | 3038 | |
| 3035 | 3039 | bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh)); |
| @@ -3038,12 +3042,6 @@ static int submit_bh_wbc(int op, int op_flags, struct buffer_head *bh, · 2 unmodified lines | ||
| 3038 | 3042 | bio->bi_end_io = end_bio_bh_io_sync; |
| 3039 | 3043 | bio->bi_private = bh; |
| 3040 | 3044 | |
| 3041 | − if (buffer_meta(bh)) | |
| 3042 | − op_flags |= REQ_META; | |
| 3043 | − if (buffer_prio(bh)) | |
| 3044 | − op_flags |= REQ_PRIO; | |
| 3045 | − bio_set_op_attrs(bio, op, op_flags); | |
| 3046 | − | |
| 3047 | 3045 | /* Take care of bh's that straddle the end of the device */ |
| 3048 | 3046 | guard_bio_eod(bio); |
| 3049 | 3047 | |
| @@ -54,7 +54,8 @@ static int fscrypt_zeroout_range_inline_crypt(const struct inode *inode, · 53 unmodified lines | ||
| 54 | 54 | int num_pages = 0; |
| 55 | 55 | |
| 56 | 56 | /* This always succeeds since __GFP_DIRECT_RECLAIM is set. */ |
| 57 | − bio = bio_alloc(GFP_NOFS, BIO_MAX_VECS); | |
| 57 | + bio = bio_alloc(inode->i_sb->s_bdev, BIO_MAX_VECS, REQ_OP_WRITE, | |
| 58 | + GFP_NOFS); | |
| 58 | 59 | |
| 59 | 60 | while (len) { |
| 60 | 61 | unsigned int blocks_this_page = min(len, blocks_per_page); |
| @@ -62,10 +63,8 @@ static int fscrypt_zeroout_range_inline_crypt(const struct inode *inode, · 1 unmodified line | ||
| 62 | 63 | |
| 63 | 64 | if (num_pages == 0) { |
| 64 | 65 | fscrypt_set_bio_crypt_ctx(bio, inode, lblk, GFP_NOFS); |
| 65 | − bio_set_dev(bio, inode->i_sb->s_bdev); | |
| 66 | 66 | bio->bi_iter.bi_sector = |
| 67 | 67 | pblk << (blockbits - SECTOR_SHIFT); |
| 68 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 69 | 68 | } |
| 70 | 69 | ret = bio_add_page(bio, ZERO_PAGE(0), bytes_this_page, 0); |
| 71 | 70 | if (WARN_ON(ret != bytes_this_page)) { |
| @@ -81,7 +80,7 @@ static int fscrypt_zeroout_range_inline_crypt(const struct inode *inode, · 9 unmodified lines | ||
| 81 | 80 | err = submit_bio_wait(bio); |
| 82 | 81 | if (err) |
| 83 | 82 | goto out; |
| 84 | − bio_reset(bio); | |
| 83 | + bio_reset(bio, inode->i_sb->s_bdev, REQ_OP_WRITE); | |
| 85 | 84 | num_pages = 0; |
| 86 | 85 | } |
| 87 | 86 | } |
| @@ -150,12 +149,10 @@ int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk, · 62 unmodified lines | ||
| 150 | 149 | return -EINVAL; |
| 151 | 150 | |
| 152 | 151 | /* This always succeeds since __GFP_DIRECT_RECLAIM is set. */ |
| 153 | − bio = bio_alloc(GFP_NOFS, nr_pages); | |
| 152 | + bio = bio_alloc(inode->i_sb->s_bdev, nr_pages, REQ_OP_WRITE, GFP_NOFS); | |
| 154 | 153 | |
| 155 | 154 | do { |
| 156 | − bio_set_dev(bio, inode->i_sb->s_bdev); | |
| 157 | 155 | bio->bi_iter.bi_sector = pblk << (blockbits - 9); |
| 158 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 159 | 156 | |
| 160 | 157 | i = 0; |
| 161 | 158 | offset = 0; |
| @@ -182,7 +179,7 @@ int fscrypt_zeroout_range(const struct inode *inode, pgoff_t lblk, · 20 unmodified lines | ||
| 182 | 179 | err = submit_bio_wait(bio); |
| 183 | 180 | if (err) |
| 184 | 181 | goto out; |
| 185 | − bio_reset(bio); | |
| 182 | + bio_reset(bio, inode->i_sb->s_bdev, REQ_OP_WRITE); | |
| 186 | 183 | } while (len != 0); |
| 187 | 184 | err = 0; |
| 188 | 185 | out: |
| @@ -11,7 +11,6 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/buffer_head.h> |
| 12 | 12 | #include <linux/dax.h> |
| 13 | 13 | #include <linux/fs.h> |
| 14 | −#include <linux/genhd.h> | |
| 15 | 14 | #include <linux/highmem.h> |
| 16 | 15 | #include <linux/memcontrol.h> |
| 17 | 16 | #include <linux/mm.h> |
| @@ -396,11 +396,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, · 395 unmodified lines | ||
| 396 | 396 | * bio_alloc() is guaranteed to return a bio when allowed to sleep and |
| 397 | 397 | * we request a valid number of vectors. |
| 398 | 398 | */ |
| 399 | − bio = bio_alloc(GFP_KERNEL, nr_vecs); | |
| 400 | − | |
| 401 | − bio_set_dev(bio, bdev); | |
| 399 | + bio = bio_alloc(bdev, nr_vecs, dio->op | dio->op_flags, GFP_KERNEL); | |
| 402 | 400 | bio->bi_iter.bi_sector = first_sector; |
| 403 | − bio_set_op_attrs(bio, dio->op, dio->op_flags); | |
| 404 | 401 | if (dio->is_async) |
| 405 | 402 | bio->bi_end_io = dio_bio_end_aio; |
| 406 | 403 | else |
| @@ -1370,15 +1370,14 @@ submit_bio_retry: · 1369 unmodified lines | ||
| 1370 | 1370 | } |
| 1371 | 1371 | |
| 1372 | 1372 | if (!bio) { |
| 1373 | − bio = bio_alloc(GFP_NOIO, BIO_MAX_VECS); | |
| 1373 | + bio = bio_alloc(mdev.m_bdev, BIO_MAX_VECS, | |
| 1374 | + REQ_OP_READ, GFP_NOIO); | |
| 1374 | 1375 | bio->bi_end_io = z_erofs_decompressqueue_endio; |
| 1375 | 1376 | |
| 1376 | − bio_set_dev(bio, mdev.m_bdev); | |
| 1377 | 1377 | last_bdev = mdev.m_bdev; |
| 1378 | 1378 | bio->bi_iter.bi_sector = (sector_t)cur << |
| 1379 | 1379 | LOG_SECTORS_PER_BLOCK; |
| 1380 | 1380 | bio->bi_private = bi_private; |
| 1381 | − bio->bi_opf = REQ_OP_READ; | |
| 1382 | 1381 | if (f->readahead) |
| 1383 | 1382 | bio->bi_opf |= REQ_RAHEAD; |
| 1384 | 1383 | ++nr_bios; |
| @@ -323,10 +323,9 @@ static void ext4_end_bio(struct bio *bio) · 322 unmodified lines | ||
| 323 | 323 | { |
| 324 | 324 | ext4_io_end_t *io_end = bio->bi_private; |
| 325 | 325 | sector_t bi_sector = bio->bi_iter.bi_sector; |
| 326 | − char b[BDEVNAME_SIZE]; | |
| 327 | 326 | |
| 328 | − if (WARN_ONCE(!io_end, "io_end is NULL: %s: sector %Lu len %u err %d\n", | |
| 329 | − bio_devname(bio, b), | |
| 327 | + if (WARN_ONCE(!io_end, "io_end is NULL: %pg: sector %Lu len %u err %d\n", | |
| 328 | + bio->bi_bdev, | |
| 330 | 329 | (long long) bio->bi_iter.bi_sector, |
| 331 | 330 | (unsigned) bio_sectors(bio), |
| 332 | 331 | bio->bi_status)) { |
| @@ -398,10 +397,9 @@ static void io_submit_init_bio(struct ext4_io_submit *io, · 65 unmodified lines | ||
| 398 | 397 | * bio_alloc will _always_ be able to allocate a bio if |
| 399 | 398 | * __GFP_DIRECT_RECLAIM is set, see comments for bio_alloc_bioset(). |
| 400 | 399 | */ |
| 401 | − bio = bio_alloc(GFP_NOIO, BIO_MAX_VECS); | |
| 400 | + bio = bio_alloc(bh->b_bdev, BIO_MAX_VECS, 0, GFP_NOIO); | |
| 402 | 401 | fscrypt_set_bio_crypt_ctx_bh(bio, bh, GFP_NOIO); |
| 403 | 402 | bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9); |
| 404 | − bio_set_dev(bio, bh->b_bdev); | |
| 405 | 403 | bio->bi_end_io = ext4_end_bio; |
| 406 | 404 | bio->bi_private = ext4_get_io_end(io->io_end); |
| 407 | 405 | io->io_bio = bio; |
| @@ -365,15 +365,15 @@ int ext4_mpage_readpages(struct inode *inode, · 364 unmodified lines | ||
| 365 | 365 | * bio_alloc will _always_ be able to allocate a bio if |
| 366 | 366 | * __GFP_DIRECT_RECLAIM is set, see bio_alloc_bioset(). |
| 367 | 367 | */ |
| 368 | − bio = bio_alloc(GFP_KERNEL, bio_max_segs(nr_pages)); | |
| 368 | + bio = bio_alloc(bdev, bio_max_segs(nr_pages), | |
| 369 | + REQ_OP_READ, GFP_KERNEL); | |
| 369 | 370 | fscrypt_set_bio_crypt_ctx(bio, inode, next_block, |
| 370 | 371 | GFP_KERNEL); |
| 371 | 372 | ext4_set_bio_post_read_ctx(bio, inode, page->index); |
| 372 | − bio_set_dev(bio, bdev); | |
| 373 | 373 | bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9); |
| 374 | 374 | bio->bi_end_io = mpage_end_io; |
| 375 | − bio_set_op_attrs(bio, REQ_OP_READ, | |
| 376 | − rac ? REQ_RAHEAD : 0); | |
| 375 | + if (rac) | |
| 376 | + bio->bi_opf |= REQ_RAHEAD; | |
| 377 | 377 | } |
| 378 | 378 | |
| 379 | 379 | length = first_hole << blkbits; |
| @@ -394,7 +394,7 @@ static struct bio *__bio_alloc(struct f2fs_io_info *fio, int npages) · 393 unmodified lines | ||
| 394 | 394 | struct f2fs_sb_info *sbi = fio->sbi; |
| 395 | 395 | struct bio *bio; |
| 396 | 396 | |
| 397 | − bio = bio_alloc_bioset(GFP_NOIO, npages, &f2fs_bioset); | |
| 397 | + bio = bio_alloc_bioset(NULL, npages, 0, GFP_NOIO, &f2fs_bioset); | |
| 398 | 398 | |
| 399 | 399 | f2fs_target_device(sbi, fio->new_blkaddr, bio); |
| 400 | 400 | if (is_read_io(fio->op)) { |
| @@ -985,8 +985,8 @@ static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr, · 584 unmodified lines | ||
| 985 | 985 | struct bio_post_read_ctx *ctx = NULL; |
| 986 | 986 | unsigned int post_read_steps = 0; |
| 987 | 987 | |
| 988 | − bio = bio_alloc_bioset(for_write ? GFP_NOIO : GFP_KERNEL, | |
| 989 | − bio_max_segs(nr_pages), &f2fs_bioset); | |
| 988 | + bio = bio_alloc_bioset(NULL, bio_max_segs(nr_pages), REQ_OP_READ, | |
| 989 | + for_write ? GFP_NOIO : GFP_KERNEL, &f2fs_bioset); | |
| 990 | 990 | if (!bio) |
| 991 | 991 | return ERR_PTR(-ENOMEM); |
| 992 | 992 | |
| @@ -994,7 +994,6 @@ static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr, · 1 unmodified line | ||
| 994 | 994 | |
| 995 | 995 | f2fs_target_device(sbi, blkaddr, bio); |
| 996 | 996 | bio->bi_end_io = f2fs_read_end_io; |
| 997 | − bio_set_op_attrs(bio, REQ_OP_READ, op_flag); | |
| 998 | 997 | |
| 999 | 998 | if (fscrypt_inode_uses_fs_layer_crypto(inode)) |
| 1000 | 999 | post_read_steps |= STEP_DECRYPT; |
| @@ -1903,8 +1903,7 @@ static long writeback_sb_inodes(struct super_block *sb, · 1902 unmodified lines | ||
| 1903 | 1903 | * unplug, so get our IOs out the door before we |
| 1904 | 1904 | * give up the CPU. |
| 1905 | 1905 | */ |
| 1906 | − if (current->plug) | |
| 1907 | − blk_flush_plug(current->plug, false); | |
| 1906 | + blk_flush_plug(current->plug, false); | |
| 1908 | 1907 | cond_resched(); |
| 1909 | 1908 | } |
| 1910 | 1909 | |
| @@ -2301,8 +2300,7 @@ void wakeup_flusher_threads(enum wb_reason reason) · 390 unmodified lines | ||
| 2301 | 2300 | /* |
| 2302 | 2301 | * If we are expecting writeback progress we must submit plugged IO. |
| 2303 | 2302 | */ |
| 2304 | − if (blk_needs_flush_plug(current)) | |
| 2305 | − blk_flush_plug(current->plug, true); | |
| 2303 | + blk_flush_plug(current->plug, true); | |
| 2306 | 2304 | |
| 2307 | 2305 | rcu_read_lock(); |
| 2308 | 2306 | list_for_each_entry_rcu(bdi, &bdi_list, bdi_list) |
| @@ -265,10 +265,9 @@ static struct bio *gfs2_log_alloc_bio(struct gfs2_sbd *sdp, u64 blkno, · 264 unmodified lines | ||
| 265 | 265 | bio_end_io_t *end_io) |
| 266 | 266 | { |
| 267 | 267 | struct super_block *sb = sdp->sd_vfs; |
| 268 | − struct bio *bio = bio_alloc(GFP_NOIO, BIO_MAX_VECS); | |
| 268 | + struct bio *bio = bio_alloc(sb->s_bdev, BIO_MAX_VECS, 0, GFP_NOIO); | |
| 269 | 269 | |
| 270 | 270 | bio->bi_iter.bi_sector = blkno << sdp->sd_fsb2bb_shift; |
| 271 | − bio_set_dev(bio, sb->s_bdev); | |
| 272 | 271 | bio->bi_end_io = end_io; |
| 273 | 272 | bio->bi_private = sdp; |
| 274 | 273 | |
| @@ -489,10 +488,9 @@ static struct bio *gfs2_chain_bio(struct bio *prev, unsigned int nr_iovecs) · 214 unmodified lines | ||
| 489 | 488 | { |
| 490 | 489 | struct bio *new; |
| 491 | 490 | |
| 492 | − new = bio_alloc(GFP_NOIO, nr_iovecs); | |
| 493 | − bio_copy_dev(new, prev); | |
| 491 | + new = bio_alloc(prev->bi_bdev, nr_iovecs, prev->bi_opf, GFP_NOIO); | |
| 492 | + bio_clone_blkg_association(new, prev); | |
| 494 | 493 | new->bi_iter.bi_sector = bio_end_sector(prev); |
| 495 | − new->bi_opf = prev->bi_opf; | |
| 496 | 494 | new->bi_write_hint = prev->bi_write_hint; |
| 497 | 495 | bio_chain(new, prev); |
| 498 | 496 | submit_bio(prev); |
| @@ -222,9 +222,8 @@ static void gfs2_submit_bhs(int op, int op_flags, struct buffer_head *bhs[], · 221 unmodified lines | ||
| 222 | 222 | struct buffer_head *bh = *bhs; |
| 223 | 223 | struct bio *bio; |
| 224 | 224 | |
| 225 | − bio = bio_alloc(GFP_NOIO, num); | |
| 225 | + bio = bio_alloc(bh->b_bdev, num, op | op_flags, GFP_NOIO); | |
| 226 | 226 | bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9); |
| 227 | − bio_set_dev(bio, bh->b_bdev); | |
| 228 | 227 | while (num > 0) { |
| 229 | 228 | bh = *bhs; |
| 230 | 229 | if (!bio_add_page(bio, bh->b_page, bh->b_size, bh_offset(bh))) { |
| @@ -235,7 +234,6 @@ static void gfs2_submit_bhs(int op, int op_flags, struct buffer_head *bhs[], · 4 unmodified lines | ||
| 235 | 234 | num--; |
| 236 | 235 | } |
| 237 | 236 | bio->bi_end_io = gfs2_meta_read_endio; |
| 238 | − bio_set_op_attrs(bio, op, op_flags); | |
| 239 | 237 | submit_bio(bio); |
| 240 | 238 | } |
| 241 | 239 | } |
| @@ -251,14 +251,12 @@ static int gfs2_read_super(struct gfs2_sbd *sdp, sector_t sector, int silent) · 250 unmodified lines | ||
| 251 | 251 | ClearPageDirty(page); |
| 252 | 252 | lock_page(page); |
| 253 | 253 | |
| 254 | − bio = bio_alloc(GFP_NOFS, 1); | |
| 254 | + bio = bio_alloc(sb->s_bdev, 1, REQ_OP_READ | REQ_META, GFP_NOFS); | |
| 255 | 255 | bio->bi_iter.bi_sector = sector * (sb->s_blocksize >> 9); |
| 256 | − bio_set_dev(bio, sb->s_bdev); | |
| 257 | 256 | bio_add_page(bio, page, PAGE_SIZE, 0); |
| 258 | 257 | |
| 259 | 258 | bio->bi_end_io = end_bio_io_page; |
| 260 | 259 | bio->bi_private = page; |
| 261 | − bio_set_op_attrs(bio, REQ_OP_READ, REQ_META); | |
| 262 | 260 | submit_bio(bio); |
| 263 | 261 | wait_on_page_locked(page); |
| 264 | 262 | bio_put(bio); |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/kobject.h> |
| 16 | 16 | #include <linux/uaccess.h> |
| 17 | 17 | #include <linux/gfs2_ondisk.h> |
| 18 | −#include <linux/genhd.h> | |
| 18 | +#include <linux/blkdev.h> | |
| 19 | 19 | |
| 20 | 20 | #include "gfs2.h" |
| 21 | 21 | #include "incore.h" |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | */ |
| 10 | 10 | |
| 11 | 11 | #include <linux/cdrom.h> |
| 12 | −#include <linux/genhd.h> | |
| 12 | +#include <linux/blkdev.h> | |
| 13 | 13 | #include <linux/nls.h> |
| 14 | 14 | #include <linux/slab.h> |
| 15 | 15 | |
| @@ -12,7 +12,6 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <linux/fs.h> |
| 13 | 13 | #include <linux/blkdev.h> |
| 14 | 14 | #include <linux/cdrom.h> |
| 15 | −#include <linux/genhd.h> | |
| 16 | 15 | #include <asm/unaligned.h> |
| 17 | 16 | |
| 18 | 17 | #include "hfsplus_fs.h" |
| @@ -64,10 +63,8 @@ int hfsplus_submit_bio(struct super_block *sb, sector_t sector, · 45 unmodified lines | ||
| 64 | 63 | offset = start & (io_size - 1); |
| 65 | 64 | sector &= ~((io_size >> HFSPLUS_SECTOR_SHIFT) - 1); |
| 66 | 65 | |
| 67 | − bio = bio_alloc(GFP_NOIO, 1); | |
| 66 | + bio = bio_alloc(sb->s_bdev, 1, op | op_flags, GFP_NOIO); | |
| 68 | 67 | bio->bi_iter.bi_sector = sector; |
| 69 | − bio_set_dev(bio, sb->s_bdev); | |
| 70 | − bio_set_op_attrs(bio, op, op_flags); | |
| 71 | 68 | |
| 72 | 69 | if (op != WRITE && data) |
| 73 | 70 | *data = (u8 *)buf + offset; |
| @@ -292,19 +292,20 @@ static loff_t iomap_readpage_iter(const struct iomap_iter *iter, · 291 unmodified lines | ||
| 292 | 292 | |
| 293 | 293 | if (ctx->rac) /* same as readahead_gfp_mask */ |
| 294 | 294 | gfp |= __GFP_NORETRY | __GFP_NOWARN; |
| 295 | − ctx->bio = bio_alloc(gfp, bio_max_segs(nr_vecs)); | |
| 295 | + ctx->bio = bio_alloc(iomap->bdev, bio_max_segs(nr_vecs), | |
| 296 | + REQ_OP_READ, gfp); | |
| 296 | 297 | /* |
| 297 | 298 | * If the bio_alloc fails, try it again for a single page to |
| 298 | 299 | * avoid having to deal with partial page reads. This emulates |
| 299 | 300 | * what do_mpage_readpage does. |
| 300 | 301 | */ |
| 301 | − if (!ctx->bio) | |
| 302 | − ctx->bio = bio_alloc(orig_gfp, 1); | |
| 303 | − ctx->bio->bi_opf = REQ_OP_READ; | |
| 302 | + if (!ctx->bio) { | |
| 303 | + ctx->bio = bio_alloc(iomap->bdev, 1, REQ_OP_READ, | |
| 304 | + orig_gfp); | |
| 305 | + } | |
| 304 | 306 | if (ctx->rac) |
| 305 | 307 | ctx->bio->bi_opf |= REQ_RAHEAD; |
| 306 | 308 | ctx->bio->bi_iter.bi_sector = sector; |
| 307 | − bio_set_dev(ctx->bio, iomap->bdev); | |
| 308 | 309 | ctx->bio->bi_end_io = iomap_read_end_io; |
| 309 | 310 | bio_add_folio(ctx->bio, folio, plen, poff); |
| 310 | 311 | } |
| @@ -550,10 +551,8 @@ static int iomap_read_folio_sync(loff_t block_start, struct folio *folio, · 239 unmodified lines | ||
| 550 | 551 | struct bio_vec bvec; |
| 551 | 552 | struct bio bio; |
| 552 | 553 | |
| 553 | − bio_init(&bio, &bvec, 1); | |
| 554 | − bio.bi_opf = REQ_OP_READ; | |
| 554 | + bio_init(&bio, iomap->bdev, &bvec, 1, REQ_OP_READ); | |
| 555 | 555 | bio.bi_iter.bi_sector = iomap_sector(iomap, block_start); |
| 556 | − bio_set_dev(&bio, iomap->bdev); | |
| 557 | 556 | bio_add_folio(&bio, folio, plen, poff); |
| 558 | 557 | return submit_bio_wait(&bio); |
| 559 | 558 | } |
| @@ -1229,10 +1228,10 @@ iomap_alloc_ioend(struct inode *inode, struct iomap_writepage_ctx *wpc, · 669 unmodified lines | ||
| 1229 | 1228 | struct iomap_ioend *ioend; |
| 1230 | 1229 | struct bio *bio; |
| 1231 | 1230 | |
| 1232 | − bio = bio_alloc_bioset(GFP_NOFS, BIO_MAX_VECS, &iomap_ioend_bioset); | |
| 1233 | − bio_set_dev(bio, wpc->iomap.bdev); | |
| 1231 | + bio = bio_alloc_bioset(wpc->iomap.bdev, BIO_MAX_VECS, | |
| 1232 | + REQ_OP_WRITE | wbc_to_write_flags(wbc), | |
| 1233 | + GFP_NOFS, &iomap_ioend_bioset); | |
| 1234 | 1234 | bio->bi_iter.bi_sector = sector; |
| 1235 | − bio->bi_opf = REQ_OP_WRITE | wbc_to_write_flags(wbc); | |
| 1236 | 1235 | bio->bi_write_hint = inode->i_write_hint; |
| 1237 | 1236 | wbc_init_bio(wbc, bio); |
| 1238 | 1237 | |
| @@ -1261,10 +1260,9 @@ iomap_chain_bio(struct bio *prev) · 22 unmodified lines | ||
| 1261 | 1260 | { |
| 1262 | 1261 | struct bio *new; |
| 1263 | 1262 | |
| 1264 | − new = bio_alloc(GFP_NOFS, BIO_MAX_VECS); | |
| 1265 | − bio_copy_dev(new, prev);/* also copies over blkcg information */ | |
| 1263 | + new = bio_alloc(prev->bi_bdev, BIO_MAX_VECS, prev->bi_opf, GFP_NOFS); | |
| 1264 | + bio_clone_blkg_association(new, prev); | |
| 1266 | 1265 | new->bi_iter.bi_sector = bio_end_sector(prev); |
| 1267 | − new->bi_opf = prev->bi_opf; | |
| 1268 | 1266 | new->bi_write_hint = prev->bi_write_hint; |
| 1269 | 1267 | |
| 1270 | 1268 | bio_chain(prev, new); |
| @@ -183,15 +183,13 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, · 182 unmodified lines | ||
| 183 | 183 | int flags = REQ_SYNC | REQ_IDLE; |
| 184 | 184 | struct bio *bio; |
| 185 | 185 | |
| 186 | − bio = bio_alloc(GFP_KERNEL, 1); | |
| 187 | − bio_set_dev(bio, iter->iomap.bdev); | |
| 186 | + bio = bio_alloc(iter->iomap.bdev, 1, REQ_OP_WRITE | flags, GFP_KERNEL); | |
| 188 | 187 | bio->bi_iter.bi_sector = iomap_sector(&iter->iomap, pos); |
| 189 | 188 | bio->bi_private = dio; |
| 190 | 189 | bio->bi_end_io = iomap_dio_bio_end_io; |
| 191 | 190 | |
| 192 | 191 | get_page(page); |
| 193 | 192 | __bio_add_page(bio, page, len, 0); |
| 194 | − bio_set_op_attrs(bio, REQ_OP_WRITE, flags); | |
| 195 | 193 | iomap_dio_submit_bio(iter, dio, bio, pos); |
| 196 | 194 | } |
| 197 | 195 | |
| @@ -309,14 +307,12 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, · 111 unmodified lines | ||
| 309 | 307 | goto out; |
| 310 | 308 | } |
| 311 | 309 | |
| 312 | − bio = bio_alloc(GFP_KERNEL, nr_pages); | |
| 313 | − bio_set_dev(bio, iomap->bdev); | |
| 310 | + bio = bio_alloc(iomap->bdev, nr_pages, bio_opf, GFP_KERNEL); | |
| 314 | 311 | bio->bi_iter.bi_sector = iomap_sector(iomap, pos); |
| 315 | 312 | bio->bi_write_hint = dio->iocb->ki_hint; |
| 316 | 313 | bio->bi_ioprio = dio->iocb->ki_ioprio; |
| 317 | 314 | bio->bi_private = dio; |
| 318 | 315 | bio->bi_end_io = iomap_dio_bio_end_io; |
| 319 | − bio->bi_opf = bio_opf; | |
| 320 | 316 | |
| 321 | 317 | ret = bio_iov_iter_get_pages(bio, dio->submit.iter); |
| 322 | 318 | if (unlikely(ret)) { |
| @@ -1980,17 +1980,13 @@ static int lbmRead(struct jfs_log * log, int pn, struct lbuf ** bpp) · 1979 unmodified lines | ||
| 1980 | 1980 | |
| 1981 | 1981 | bp->l_flag |= lbmREAD; |
| 1982 | 1982 | |
| 1983 | − bio = bio_alloc(GFP_NOFS, 1); | |
| 1984 | − | |
| 1983 | + bio = bio_alloc(log->bdev, 1, REQ_OP_READ, GFP_NOFS); | |
| 1985 | 1984 | bio->bi_iter.bi_sector = bp->l_blkno << (log->l2bsize - 9); |
| 1986 | − bio_set_dev(bio, log->bdev); | |
| 1987 | − | |
| 1988 | 1985 | bio_add_page(bio, bp->l_page, LOGPSIZE, bp->l_offset); |
| 1989 | 1986 | BUG_ON(bio->bi_iter.bi_size != LOGPSIZE); |
| 1990 | 1987 | |
| 1991 | 1988 | bio->bi_end_io = lbmIODone; |
| 1992 | 1989 | bio->bi_private = bp; |
| 1993 | − bio->bi_opf = REQ_OP_READ; | |
| 1994 | 1990 | /*check if journaling to disk has been disabled*/ |
| 1995 | 1991 | if (log->no_integrity) { |
| 1996 | 1992 | bio->bi_iter.bi_size = 0; |
| @@ -2125,16 +2121,13 @@ static void lbmStartIO(struct lbuf * bp) · 128 unmodified lines | ||
| 2125 | 2121 | |
| 2126 | 2122 | jfs_info("lbmStartIO"); |
| 2127 | 2123 | |
| 2128 | − bio = bio_alloc(GFP_NOFS, 1); | |
| 2124 | + bio = bio_alloc(log->bdev, 1, REQ_OP_WRITE | REQ_SYNC, GFP_NOFS); | |
| 2129 | 2125 | bio->bi_iter.bi_sector = bp->l_blkno << (log->l2bsize - 9); |
| 2130 | − bio_set_dev(bio, log->bdev); | |
| 2131 | − | |
| 2132 | 2126 | bio_add_page(bio, bp->l_page, LOGPSIZE, bp->l_offset); |
| 2133 | 2127 | BUG_ON(bio->bi_iter.bi_size != LOGPSIZE); |
| 2134 | 2128 | |
| 2135 | 2129 | bio->bi_end_io = lbmIODone; |
| 2136 | 2130 | bio->bi_private = bp; |
| 2137 | − bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; | |
| 2138 | 2131 | |
| 2139 | 2132 | /* check if journaling to disk has been disabled */ |
| 2140 | 2133 | if (log->no_integrity) { |
| @@ -417,12 +417,10 @@ static int metapage_writepage(struct page *page, struct writeback_control *wbc) · 416 unmodified lines | ||
| 417 | 417 | } |
| 418 | 418 | len = min(xlen, (int)JFS_SBI(inode->i_sb)->nbperpage); |
| 419 | 419 | |
| 420 | − bio = bio_alloc(GFP_NOFS, 1); | |
| 421 | − bio_set_dev(bio, inode->i_sb->s_bdev); | |
| 420 | + bio = bio_alloc(inode->i_sb->s_bdev, 1, REQ_OP_WRITE, GFP_NOFS); | |
| 422 | 421 | bio->bi_iter.bi_sector = pblock << (inode->i_blkbits - 9); |
| 423 | 422 | bio->bi_end_io = metapage_write_end_io; |
| 424 | 423 | bio->bi_private = page; |
| 425 | − bio_set_op_attrs(bio, REQ_OP_WRITE, 0); | |
| 426 | 424 | |
| 427 | 425 | /* Don't call bio_add_page yet, we may add to this vec */ |
| 428 | 426 | bio_offset = offset; |
| @@ -497,13 +495,12 @@ static int metapage_readpage(struct file *fp, struct page *page) · 68 unmodified lines | ||
| 497 | 495 | if (bio) |
| 498 | 496 | submit_bio(bio); |
| 499 | 497 | |
| 500 | − bio = bio_alloc(GFP_NOFS, 1); | |
| 501 | − bio_set_dev(bio, inode->i_sb->s_bdev); | |
| 498 | + bio = bio_alloc(inode->i_sb->s_bdev, 1, REQ_OP_READ, | |
| 499 | + GFP_NOFS); | |
| 502 | 500 | bio->bi_iter.bi_sector = |
| 503 | 501 | pblock << (inode->i_blkbits - 9); |
| 504 | 502 | bio->bi_end_io = metapage_read_end_io; |
| 505 | 503 | bio->bi_private = page; |
| 506 | − bio_set_op_attrs(bio, REQ_OP_READ, 0); | |
| 507 | 504 | len = xlen << inode->i_blkbits; |
| 508 | 505 | offset = block_offset << inode->i_blkbits; |
| 509 | 506 | if (bio_add_page(bio, page, len, offset) < len) |
| @@ -11,7 +11,6 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <linux/writeback.h> |
| 12 | 12 | #include <linux/xattr.h> |
| 13 | 13 | #include <linux/falloc.h> |
| 14 | −#include <linux/genhd.h> | |
| 15 | 14 | #include <linux/fsnotify.h> |
| 16 | 15 | #include <linux/dcache.h> |
| 17 | 16 | #include <linux/slab.h> |
| @@ -66,29 +66,6 @@ static struct bio *mpage_bio_submit(int op, int op_flags, struct bio *bio) · 65 unmodified lines | ||
| 66 | 66 | return NULL; |
| 67 | 67 | } |
| 68 | 68 | |
| 69 | −static struct bio * | |
| 70 | −mpage_alloc(struct block_device *bdev, | |
| 71 | − sector_t first_sector, int nr_vecs, | |
| 72 | − gfp_t gfp_flags) | |
| 73 | −{ | |
| 74 | − struct bio *bio; | |
| 75 | − | |
| 76 | − /* Restrict the given (page cache) mask for slab allocations */ | |
| 77 | − gfp_flags &= GFP_KERNEL; | |
| 78 | − bio = bio_alloc(gfp_flags, nr_vecs); | |
| 79 | − | |
| 80 | − if (bio == NULL && (current->flags & PF_MEMALLOC)) { | |
| 81 | − while (!bio && (nr_vecs /= 2)) | |
| 82 | − bio = bio_alloc(gfp_flags, nr_vecs); | |
| 83 | − } | |
| 84 | − | |
| 85 | − if (bio) { | |
| 86 | − bio_set_dev(bio, bdev); | |
| 87 | − bio->bi_iter.bi_sector = first_sector; | |
| 88 | − } | |
| 89 | − return bio; | |
| 90 | −} | |
| 91 | − | |
| 92 | 69 | /* |
| 93 | 70 | * support function for mpage_readahead. The fs supplied get_block might |
| 94 | 71 | * return an up to date buffer. This is used to map that buffer into |
| @@ -296,10 +273,11 @@ alloc_new: · 201 unmodified lines | ||
| 296 | 273 | page)) |
| 297 | 274 | goto out; |
| 298 | 275 | } |
| 299 | − args->bio = mpage_alloc(bdev, blocks[0] << (blkbits - 9), | |
| 300 | − bio_max_segs(args->nr_pages), gfp); | |
| 276 | + args->bio = bio_alloc(bdev, bio_max_segs(args->nr_pages), 0, | |
| 277 | + gfp); | |
| 301 | 278 | if (args->bio == NULL) |
| 302 | 279 | goto confused; |
| 280 | + args->bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9); | |
| 303 | 281 | } |
| 304 | 282 | |
| 305 | 283 | length = first_hole << blkbits; |
| @@ -608,10 +586,8 @@ alloc_new: · 302 unmodified lines | ||
| 608 | 586 | page, wbc)) |
| 609 | 587 | goto out; |
| 610 | 588 | } |
| 611 | − bio = mpage_alloc(bdev, blocks[0] << (blkbits - 9), | |
| 612 | − BIO_MAX_VECS, GFP_NOFS|__GFP_HIGH); | |
| 613 | − if (bio == NULL) | |
| 614 | − goto confused; | |
| 589 | + bio = bio_alloc(bdev, BIO_MAX_VECS, 0, GFP_NOFS); | |
| 590 | + bio->bi_iter.bi_sector = blocks[0] << (blkbits - 9); | |
| 615 | 591 | |
| 616 | 592 | wbc_init_bio(wbc, bio); |
| 617 | 593 | bio->bi_write_hint = inode->i_write_hint; |
| @@ -115,23 +115,6 @@ bl_submit_bio(struct bio *bio) · 114 unmodified lines | ||
| 115 | 115 | return NULL; |
| 116 | 116 | } |
| 117 | 117 | |
| 118 | −static struct bio *bl_alloc_init_bio(unsigned int npg, | |
| 119 | − struct block_device *bdev, sector_t disk_sector, | |
| 120 | − bio_end_io_t end_io, struct parallel_io *par) | |
| 121 | −{ | |
| 122 | − struct bio *bio; | |
| 123 | − | |
| 124 | − npg = bio_max_segs(npg); | |
| 125 | − bio = bio_alloc(GFP_NOIO, npg); | |
| 126 | − if (bio) { | |
| 127 | − bio->bi_iter.bi_sector = disk_sector; | |
| 128 | − bio_set_dev(bio, bdev); | |
| 129 | − bio->bi_end_io = end_io; | |
| 130 | − bio->bi_private = par; | |
| 131 | − } | |
| 132 | − return bio; | |
| 133 | −} | |
| 134 | − | |
| 135 | 118 | static bool offset_in_map(u64 offset, struct pnfs_block_dev_map *map) |
| 136 | 119 | { |
| 137 | 120 | return offset >= map->start && offset < map->start + map->len; |
| @@ -171,11 +154,10 @@ do_add_page_to_bio(struct bio *bio, int npg, int rw, sector_t isect, · 33 unmodified lines | ||
| 171 | 154 | |
| 172 | 155 | retry: |
| 173 | 156 | if (!bio) { |
| 174 | − bio = bl_alloc_init_bio(npg, map->bdev, | |
| 175 | − disk_addr >> SECTOR_SHIFT, end_io, par); | |
| 176 | − if (!bio) | |
| 177 | − return ERR_PTR(-ENOMEM); | |
| 178 | − bio_set_op_attrs(bio, rw, 0); | |
| 157 | + bio = bio_alloc(map->bdev, bio_max_segs(npg), rw, GFP_NOIO); | |
| 158 | + bio->bi_iter.bi_sector = disk_addr >> SECTOR_SHIFT; | |
| 159 | + bio->bi_end_io = end_io; | |
| 160 | + bio->bi_private = par; | |
| 179 | 161 | } |
| 180 | 162 | if (bio_add_page(bio, page, *len, offset) < *len) { |
| 181 | 163 | bio = bl_submit_bio(bio); |
| @@ -27,7 +27,6 @@ · 26 unmodified lines | ||
| 27 | 27 | */ |
| 28 | 28 | |
| 29 | 29 | #include <linux/module.h> |
| 30 | −#include <linux/genhd.h> | |
| 31 | 30 | #include <linux/blkdev.h> |
| 32 | 31 | |
| 33 | 32 | #include "blocklayout.h" |
| @@ -4,7 +4,6 @@ · 3 unmodified lines | ||
| 4 | 4 | */ |
| 5 | 5 | #include <linux/exportfs.h> |
| 6 | 6 | #include <linux/iomap.h> |
| 7 | −#include <linux/genhd.h> | |
| 8 | 7 | #include <linux/slab.h> |
| 9 | 8 | #include <linux/pr.h> |
| 10 | 9 | |
| @@ -371,29 +371,6 @@ static int nilfs_segbuf_submit_bio(struct nilfs_segment_buffer *segbuf, · 370 unmodified lines | ||
| 371 | 371 | return err; |
| 372 | 372 | } |
| 373 | 373 | |
| 374 | −/** | |
| 375 | − * nilfs_alloc_seg_bio - allocate a new bio for writing log | |
| 376 | − * @nilfs: nilfs object | |
| 377 | − * @start: start block number of the bio | |
| 378 | − * @nr_vecs: request size of page vector. | |
| 379 | − * | |
| 380 | − * Return Value: On success, pointer to the struct bio is returned. | |
| 381 | − * On error, NULL is returned. | |
| 382 | − */ | |
| 383 | −static struct bio *nilfs_alloc_seg_bio(struct the_nilfs *nilfs, sector_t start, | |
| 384 | − int nr_vecs) | |
| 385 | −{ | |
| 386 | − struct bio *bio; | |
| 387 | − | |
| 388 | − bio = bio_alloc(GFP_NOIO, nr_vecs); | |
| 389 | − if (likely(bio)) { | |
| 390 | − bio_set_dev(bio, nilfs->ns_bdev); | |
| 391 | − bio->bi_iter.bi_sector = | |
| 392 | − start << (nilfs->ns_blocksize_bits - 9); | |
| 393 | − } | |
| 394 | − return bio; | |
| 395 | −} | |
| 396 | − | |
| 397 | 374 | static void nilfs_segbuf_prepare_write(struct nilfs_segment_buffer *segbuf, |
| 398 | 375 | struct nilfs_write_info *wi) |
| 399 | 376 | { |
| @@ -414,10 +391,10 @@ static int nilfs_segbuf_submit_bh(struct nilfs_segment_buffer *segbuf, · 14 unmodified lines | ||
| 414 | 391 | BUG_ON(wi->nr_vecs <= 0); |
| 415 | 392 | repeat: |
| 416 | 393 | if (!wi->bio) { |
| 417 | − wi->bio = nilfs_alloc_seg_bio(wi->nilfs, wi->blocknr + wi->end, | |
| 418 | − wi->nr_vecs); | |
| 419 | − if (unlikely(!wi->bio)) | |
| 420 | − return -ENOMEM; | |
| 394 | + wi->bio = bio_alloc(wi->nilfs->ns_bdev, wi->nr_vecs, 0, | |
| 395 | + GFP_NOIO); | |
| 396 | + wi->bio->bi_iter.bi_sector = (wi->blocknr + wi->end) << | |
| 397 | + (wi->nilfs->ns_blocksize_bits - 9); | |
| 421 | 398 | } |
| 422 | 399 | |
| 423 | 400 | len = bio_add_page(wi->bio, bh->b_page, bh->b_size, bh_offset(bh)); |
| @@ -1443,17 +1443,6 @@ int ntfs_write_bh(struct ntfs_sb_info *sbi, struct NTFS_RECORD_HEADER *rhdr, · 1442 unmodified lines | ||
| 1443 | 1443 | return err; |
| 1444 | 1444 | } |
| 1445 | 1445 | |
| 1446 | −static inline struct bio *ntfs_alloc_bio(u32 nr_vecs) | |
| 1447 | −{ | |
| 1448 | − struct bio *bio = bio_alloc(GFP_NOFS | __GFP_HIGH, nr_vecs); | |
| 1449 | − | |
| 1450 | − if (!bio && (current->flags & PF_MEMALLOC)) { | |
| 1451 | − while (!bio && (nr_vecs /= 2)) | |
| 1452 | − bio = bio_alloc(GFP_NOFS | __GFP_HIGH, nr_vecs); | |
| 1453 | − } | |
| 1454 | − return bio; | |
| 1455 | −} | |
| 1456 | − | |
| 1457 | 1446 | /* |
| 1458 | 1447 | * ntfs_bio_pages - Read/write pages from/to disk. |
| 1459 | 1448 | */ |
| @@ -1496,19 +1485,13 @@ int ntfs_bio_pages(struct ntfs_sb_info *sbi, const struct runs_tree *run, · 36 unmodified lines | ||
| 1496 | 1485 | lbo = ((u64)lcn << cluster_bits) + off; |
| 1497 | 1486 | len = ((u64)clen << cluster_bits) - off; |
| 1498 | 1487 | new_bio: |
| 1499 | − new = ntfs_alloc_bio(nr_pages - page_idx); | |
| 1500 | − if (!new) { | |
| 1501 | − err = -ENOMEM; | |
| 1502 | − goto out; | |
| 1503 | − } | |
| 1488 | + new = bio_alloc(bdev, nr_pages - page_idx, op, GFP_NOFS); | |
| 1504 | 1489 | if (bio) { |
| 1505 | 1490 | bio_chain(bio, new); |
| 1506 | 1491 | submit_bio(bio); |
| 1507 | 1492 | } |
| 1508 | 1493 | bio = new; |
| 1509 | − bio_set_dev(bio, bdev); | |
| 1510 | 1494 | bio->bi_iter.bi_sector = lbo >> 9; |
| 1511 | − bio->bi_opf = op; | |
| 1512 | 1495 | |
| 1513 | 1496 | while (len) { |
| 1514 | 1497 | off = vbo & (PAGE_SIZE - 1); |
| @@ -1599,18 +1582,12 @@ int ntfs_bio_fill_1(struct ntfs_sb_info *sbi, const struct runs_tree *run) · 84 unmodified lines | ||
| 1599 | 1582 | lbo = (u64)lcn << cluster_bits; |
| 1600 | 1583 | len = (u64)clen << cluster_bits; |
| 1601 | 1584 | new_bio: |
| 1602 | − new = ntfs_alloc_bio(BIO_MAX_VECS); | |
| 1603 | − if (!new) { | |
| 1604 | − err = -ENOMEM; | |
| 1605 | − break; | |
| 1606 | − } | |
| 1585 | + new = bio_alloc(bdev, BIO_MAX_VECS, REQ_OP_WRITE, GFP_NOFS); | |
| 1607 | 1586 | if (bio) { |
| 1608 | 1587 | bio_chain(bio, new); |
| 1609 | 1588 | submit_bio(bio); |
| 1610 | 1589 | } |
| 1611 | 1590 | bio = new; |
| 1612 | − bio_set_dev(bio, bdev); | |
| 1613 | − bio->bi_opf = REQ_OP_WRITE; | |
| 1614 | 1591 | bio->bi_iter.bi_sector = lbo >> 9; |
| 1615 | 1592 | |
| 1616 | 1593 | for (;;) { |
| @@ -1626,11 +1603,10 @@ new_bio: · 9 unmodified lines | ||
| 1626 | 1603 | } |
| 1627 | 1604 | } while (run_get_entry(run, ++run_idx, NULL, &lcn, &clen)); |
| 1628 | 1605 | |
| 1629 | − if (bio) { | |
| 1630 | − if (!err) | |
| 1631 | − err = submit_bio_wait(bio); | |
| 1632 | − bio_put(bio); | |
| 1633 | − } | |
| 1606 | + if (!err) | |
| 1607 | + err = submit_bio_wait(bio); | |
| 1608 | + bio_put(bio); | |
| 1609 | + | |
| 1634 | 1610 | blk_finish_plug(&plug); |
| 1635 | 1611 | out: |
| 1636 | 1612 | unlock_page(fill); |
| @@ -518,7 +518,7 @@ static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, · 517 unmodified lines | ||
| 518 | 518 | * GFP_KERNEL that the local node can get fenced. It would be |
| 519 | 519 | * nicest if we could pre-allocate these bios and avoid this |
| 520 | 520 | * all together. */ |
| 521 | − bio = bio_alloc(GFP_ATOMIC, 16); | |
| 521 | + bio = bio_alloc(reg->hr_bdev, 16, op | op_flags, GFP_ATOMIC); | |
| 522 | 522 | if (!bio) { |
| 523 | 523 | mlog(ML_ERROR, "Could not alloc slots BIO!\n"); |
| 524 | 524 | bio = ERR_PTR(-ENOMEM); |
| @@ -527,10 +527,8 @@ static struct bio *o2hb_setup_one_bio(struct o2hb_region *reg, · 2 unmodified lines | ||
| 527 | 527 | |
| 528 | 528 | /* Must put everything in 512 byte sectors for the bio... */ |
| 529 | 529 | bio->bi_iter.bi_sector = (reg->hr_start_block + cs) << (bits - 9); |
| 530 | − bio_set_dev(bio, reg->hr_bdev); | |
| 531 | 530 | bio->bi_private = wc; |
| 532 | 531 | bio->bi_end_io = o2hb_bio_end_io; |
| 533 | − bio_set_op_attrs(bio, op, op_flags); | |
| 534 | 532 | |
| 535 | 533 | vec_start = (cs << bits) % PAGE_SIZE; |
| 536 | 534 | while(cs < max_slots) { |
| @@ -86,16 +86,17 @@ static int squashfs_bio_read(struct super_block *sb, u64 index, int length, · 85 unmodified lines | ||
| 86 | 86 | int error, i; |
| 87 | 87 | struct bio *bio; |
| 88 | 88 | |
| 89 | − if (page_count <= BIO_MAX_VECS) | |
| 90 | − bio = bio_alloc(GFP_NOIO, page_count); | |
| 91 | − else | |
| 89 | + if (page_count <= BIO_MAX_VECS) { | |
| 90 | + bio = bio_alloc(sb->s_bdev, page_count, REQ_OP_READ, GFP_NOIO); | |
| 91 | + } else { | |
| 92 | 92 | bio = bio_kmalloc(GFP_NOIO, page_count); |
| 93 | + bio_set_dev(bio, sb->s_bdev); | |
| 94 | + bio->bi_opf = REQ_OP_READ; | |
| 95 | + } | |
| 93 | 96 | |
| 94 | 97 | if (!bio) |
| 95 | 98 | return -ENOMEM; |
| 96 | 99 | |
| 97 | − bio_set_dev(bio, sb->s_bdev); | |
| 98 | − bio->bi_opf = READ; | |
| 99 | 100 | bio->bi_iter.bi_sector = block * (msblk->devblksize >> SECTOR_SHIFT); |
| 100 | 101 | |
| 101 | 102 | for (i = 0; i < page_count; ++i) { |
| @@ -36,9 +36,7 @@ xfs_flush_bdev_async( · 35 unmodified lines | ||
| 36 | 36 | return; |
| 37 | 37 | } |
| 38 | 38 | |
| 39 | − bio_init(bio, NULL, 0); | |
| 40 | − bio_set_dev(bio, bdev); | |
| 41 | − bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH | REQ_SYNC; | |
| 39 | + bio_init(bio, bdev, NULL, 0, REQ_OP_WRITE | REQ_PREFLUSH | REQ_SYNC); | |
| 42 | 40 | bio->bi_private = done; |
| 43 | 41 | bio->bi_end_io = xfs_flush_bdev_async_endio; |
| 44 | 42 | |
| @@ -61,10 +59,9 @@ xfs_rw_bdev( · 16 unmodified lines | ||
| 61 | 59 | if (is_vmalloc && op == REQ_OP_WRITE) |
| 62 | 60 | flush_kernel_vmap_range(data, count); |
| 63 | 61 | |
| 64 | − bio = bio_alloc(GFP_KERNEL, bio_max_vecs(left)); | |
| 65 | − bio_set_dev(bio, bdev); | |
| 62 | + bio = bio_alloc(bdev, bio_max_vecs(left), op | REQ_META | REQ_SYNC, | |
| 63 | + GFP_KERNEL); | |
| 66 | 64 | bio->bi_iter.bi_sector = sector; |
| 67 | − bio->bi_opf = op | REQ_META | REQ_SYNC; | |
| 68 | 65 | |
| 69 | 66 | do { |
| 70 | 67 | struct page *page = kmem_to_page(data); |
| @@ -74,10 +71,9 @@ xfs_rw_bdev( · 3 unmodified lines | ||
| 74 | 71 | while (bio_add_page(bio, page, len, off) != len) { |
| 75 | 72 | struct bio *prev = bio; |
| 76 | 73 | |
| 77 | − bio = bio_alloc(GFP_KERNEL, bio_max_vecs(left)); | |
| 78 | − bio_copy_dev(bio, prev); | |
| 74 | + bio = bio_alloc(prev->bi_bdev, bio_max_vecs(left), | |
| 75 | + prev->bi_opf, GFP_KERNEL); | |
| 79 | 76 | bio->bi_iter.bi_sector = bio_end_sector(prev); |
| 80 | − bio->bi_opf = prev->bi_opf; | |
| 81 | 77 | bio_chain(prev, bio); |
| 82 | 78 | |
| 83 | 79 | submit_bio(prev); |
| @@ -1440,12 +1440,10 @@ next_chunk: · 1439 unmodified lines | ||
| 1440 | 1440 | atomic_inc(&bp->b_io_remaining); |
| 1441 | 1441 | nr_pages = bio_max_segs(total_nr_pages); |
| 1442 | 1442 | |
| 1443 | − bio = bio_alloc(GFP_NOIO, nr_pages); | |
| 1444 | − bio_set_dev(bio, bp->b_target->bt_bdev); | |
| 1443 | + bio = bio_alloc(bp->b_target->bt_bdev, nr_pages, op, GFP_NOIO); | |
| 1445 | 1444 | bio->bi_iter.bi_sector = sector; |
| 1446 | 1445 | bio->bi_end_io = xfs_buf_bio_end_io; |
| 1447 | 1446 | bio->bi_private = bp; |
| 1448 | − bio->bi_opf = op; | |
| 1449 | 1447 | |
| 1450 | 1448 | for (; size && nr_pages; nr_pages--, page_index++) { |
| 1451 | 1449 | int rbytes, nbytes = PAGE_SIZE - offset; |
| @@ -1883,19 +1883,19 @@ xlog_write_iclog( · 1882 unmodified lines | ||
| 1883 | 1883 | return; |
| 1884 | 1884 | } |
| 1885 | 1885 | |
| 1886 | − bio_init(&iclog->ic_bio, iclog->ic_bvec, howmany(count, PAGE_SIZE)); | |
| 1887 | − bio_set_dev(&iclog->ic_bio, log->l_targ->bt_bdev); | |
| 1888 | − iclog->ic_bio.bi_iter.bi_sector = log->l_logBBstart + bno; | |
| 1889 | − iclog->ic_bio.bi_end_io = xlog_bio_end_io; | |
| 1890 | − iclog->ic_bio.bi_private = iclog; | |
| 1891 | − | |
| 1892 | 1886 | /* |
| 1893 | 1887 | * We use REQ_SYNC | REQ_IDLE here to tell the block layer the are more |
| 1894 | 1888 | * IOs coming immediately after this one. This prevents the block layer |
| 1895 | 1889 | * writeback throttle from throttling log writes behind background |
| 1896 | 1890 | * metadata writeback and causing priority inversions. |
| 1897 | 1891 | */ |
| 1898 | − iclog->ic_bio.bi_opf = REQ_OP_WRITE | REQ_META | REQ_SYNC | REQ_IDLE; | |
| 1892 | + bio_init(&iclog->ic_bio, log->l_targ->bt_bdev, iclog->ic_bvec, | |
| 1893 | + howmany(count, PAGE_SIZE), | |
| 1894 | + REQ_OP_WRITE | REQ_META | REQ_SYNC | REQ_IDLE); | |
| 1895 | + iclog->ic_bio.bi_iter.bi_sector = log->l_logBBstart + bno; | |
| 1896 | + iclog->ic_bio.bi_end_io = xlog_bio_end_io; | |
| 1897 | + iclog->ic_bio.bi_private = iclog; | |
| 1898 | + | |
| 1899 | 1899 | if (iclog->ic_flags & XLOG_ICL_NEED_FLUSH) { |
| 1900 | 1900 | iclog->ic_bio.bi_opf |= REQ_PREFLUSH; |
| 1901 | 1901 | /* |
| @@ -692,12 +692,11 @@ static ssize_t zonefs_file_dio_append(struct kiocb *iocb, struct iov_iter *from) · 691 unmodified lines | ||
| 692 | 692 | if (!nr_pages) |
| 693 | 693 | return 0; |
| 694 | 694 | |
| 695 | − bio = bio_alloc(GFP_NOFS, nr_pages); | |
| 696 | − bio_set_dev(bio, bdev); | |
| 695 | + bio = bio_alloc(bdev, nr_pages, | |
| 696 | + REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE, GFP_NOFS); | |
| 697 | 697 | bio->bi_iter.bi_sector = zi->i_zsector; |
| 698 | 698 | bio->bi_write_hint = iocb->ki_hint; |
| 699 | 699 | bio->bi_ioprio = iocb->ki_ioprio; |
| 700 | − bio->bi_opf = REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE; | |
| 701 | 700 | if (iocb->ki_flags & IOCB_DSYNC) |
| 702 | 701 | bio->bi_opf |= REQ_FUA; |
| 703 | 702 | |
| @@ -1541,10 +1540,8 @@ static int zonefs_read_super(struct super_block *sb) · 837 unmodified lines | ||
| 1541 | 1540 | if (!page) |
| 1542 | 1541 | return -ENOMEM; |
| 1543 | 1542 | |
| 1544 | − bio_init(&bio, &bio_vec, 1); | |
| 1543 | + bio_init(&bio, sb->s_bdev, &bio_vec, 1, REQ_OP_READ); | |
| 1545 | 1544 | bio.bi_iter.bi_sector = 0; |
| 1546 | − bio.bi_opf = REQ_OP_READ; | |
| 1547 | − bio_set_dev(&bio, sb->s_bdev); | |
| 1548 | 1545 | bio_add_page(&bio, page, PAGE_SIZE, 0); |
| 1549 | 1546 | |
| 1550 | 1547 | ret = submit_bio_wait(&bio); |
| @@ -405,21 +405,25 @@ extern void bioset_exit(struct bio_set *); · 404 unmodified lines | ||
| 405 | 405 | extern int biovec_init_pool(mempool_t *pool, int pool_entries); |
| 406 | 406 | extern int bioset_init_from_src(struct bio_set *bs, struct bio_set *src); |
| 407 | 407 | |
| 408 | −struct bio *bio_alloc_bioset(gfp_t gfp, unsigned short nr_iovecs, | |
| 409 | − struct bio_set *bs); | |
| 410 | −struct bio *bio_alloc_kiocb(struct kiocb *kiocb, unsigned short nr_vecs, | |
| 411 | − struct bio_set *bs); | |
| 408 | +struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, | |
| 409 | + unsigned int opf, gfp_t gfp_mask, | |
| 410 | + struct bio_set *bs); | |
| 411 | +struct bio *bio_alloc_kiocb(struct kiocb *kiocb, struct block_device *bdev, | |
| 412 | + unsigned short nr_vecs, unsigned int opf, struct bio_set *bs); | |
| 412 | 413 | struct bio *bio_kmalloc(gfp_t gfp_mask, unsigned short nr_iovecs); |
| 413 | 414 | extern void bio_put(struct bio *); |
| 414 | 415 | |
| 415 | −extern void __bio_clone_fast(struct bio *, struct bio *); | |
| 416 | −extern struct bio *bio_clone_fast(struct bio *, gfp_t, struct bio_set *); | |
| 416 | +struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src, | |
| 417 | + gfp_t gfp, struct bio_set *bs); | |
| 418 | +int bio_init_clone(struct block_device *bdev, struct bio *bio, | |
| 419 | + struct bio *bio_src, gfp_t gfp); | |
| 417 | 420 | |
| 418 | 421 | extern struct bio_set fs_bio_set; |
| 419 | 422 | |
| 420 | −static inline struct bio *bio_alloc(gfp_t gfp_mask, unsigned short nr_iovecs) | |
| 423 | +static inline struct bio *bio_alloc(struct block_device *bdev, | |
| 424 | + unsigned short nr_vecs, unsigned int opf, gfp_t gfp_mask) | |
| 421 | 425 | { |
| 422 | − return bio_alloc_bioset(gfp_mask, nr_iovecs, &fs_bio_set); | |
| 426 | + return bio_alloc_bioset(bdev, nr_vecs, opf, gfp_mask, &fs_bio_set); | |
| 423 | 427 | } |
| 424 | 428 | |
| 425 | 429 | void submit_bio(struct bio *bio); |
| @@ -454,10 +458,10 @@ static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs) · 28 unmodified lines | ||
| 454 | 458 | struct request_queue; |
| 455 | 459 | |
| 456 | 460 | extern int submit_bio_wait(struct bio *bio); |
| 457 | −extern void bio_init(struct bio *bio, struct bio_vec *table, | |
| 458 | − unsigned short max_vecs); | |
| 461 | +void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table, | |
| 462 | + unsigned short max_vecs, unsigned int opf); | |
| 459 | 463 | extern void bio_uninit(struct bio *); |
| 460 | −extern void bio_reset(struct bio *); | |
| 464 | +void bio_reset(struct bio *bio, struct block_device *bdev, unsigned int opf); | |
| 461 | 465 | void bio_chain(struct bio *, struct bio *); |
| 462 | 466 | |
| 463 | 467 | int bio_add_page(struct bio *, struct page *, unsigned len, unsigned off); |
| @@ -487,8 +491,6 @@ static inline void bio_release_pages(struct bio *bio, bool mark_dirty) · 23 unmodified lines | ||
| 487 | 491 | __bio_release_pages(bio, mark_dirty); |
| 488 | 492 | } |
| 489 | 493 | |
| 490 | −extern const char *bio_devname(struct bio *bio, char *buffer); | |
| 491 | − | |
| 492 | 494 | #define bio_dev(bio) \ |
| 493 | 495 | disk_devt((bio)->bi_bdev->bd_disk) |
| 494 | 496 | |
| @@ -515,13 +517,6 @@ static inline void bio_set_dev(struct bio *bio, struct block_device *bdev) · 20 unmodified lines | ||
| 515 | 517 | bio_associate_blkg(bio); |
| 516 | 518 | } |
| 517 | 519 | |
| 518 | −static inline void bio_copy_dev(struct bio *dst, struct bio *src) | |
| 519 | −{ | |
| 520 | − bio_clear_flag(dst, BIO_REMAPPED); | |
| 521 | − dst->bi_bdev = src->bi_bdev; | |
| 522 | − bio_clone_blkg_association(dst, src); | |
| 523 | −} | |
| 524 | − | |
| 525 | 520 | /* |
| 526 | 521 | * BIO list management for use by remapping drivers (e.g. DM or MD) and loop. |
| 527 | 522 | * |
| @@ -790,6 +785,7 @@ static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb) · 262 unmodified lines | ||
| 790 | 785 | bio->bi_opf |= REQ_NOWAIT; |
| 791 | 786 | } |
| 792 | 787 | |
| 793 | −struct bio *blk_next_bio(struct bio *bio, unsigned int nr_pages, gfp_t gfp); | |
| 788 | +struct bio *blk_next_bio(struct bio *bio, struct block_device *bdev, | |
| 789 | + unsigned int nr_pages, unsigned int opf, gfp_t gfp); | |
| 794 | 790 | |
| 795 | 791 | #endif /* __LINUX_BIO_H */ |
| @@ -25,14 +25,8 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <linux/kthread.h> |
| 26 | 26 | #include <linux/fs.h> |
| 27 | 27 | |
| 28 | −/* percpu_counter batch for blkg_[rw]stats, per-cpu drift doesn't matter */ | |
| 29 | −#define BLKG_STAT_CPU_BATCH (INT_MAX / 2) | |
| 30 | − | |
| 31 | −/* Max limits for throttle policy */ | |
| 32 | −#define THROTL_IOPS_MAX UINT_MAX | |
| 33 | 28 | #define FC_APPID_LEN 129 |
| 34 | 29 | |
| 35 | − | |
| 36 | 30 | #ifdef CONFIG_BLK_CGROUP |
| 37 | 31 | |
| 38 | 32 | enum blkg_iostat_type { |
| @@ -44,6 +38,7 @@ enum blkg_iostat_type { · 5 unmodified lines | ||
| 44 | 38 | }; |
| 45 | 39 | |
| 46 | 40 | struct blkcg_gq; |
| 41 | +struct blkg_policy_data; | |
| 47 | 42 | |
| 48 | 43 | struct blkcg { |
| 49 | 44 | struct cgroup_subsys_state css; |
| @@ -76,36 +71,6 @@ struct blkg_iostat_set { · 26 unmodified lines | ||
| 76 | 71 | struct blkg_iostat last; |
| 77 | 72 | }; |
| 78 | 73 | |
| 79 | −/* | |
| 80 | − * A blkcg_gq (blkg) is association between a block cgroup (blkcg) and a | |
| 81 | − * request_queue (q). This is used by blkcg policies which need to track | |
| 82 | − * information per blkcg - q pair. | |
| 83 | − * | |
| 84 | − * There can be multiple active blkcg policies and each blkg:policy pair is | |
| 85 | − * represented by a blkg_policy_data which is allocated and freed by each | |
| 86 | − * policy's pd_alloc/free_fn() methods. A policy can allocate private data | |
| 87 | − * area by allocating larger data structure which embeds blkg_policy_data | |
| 88 | − * at the beginning. | |
| 89 | − */ | |
| 90 | −struct blkg_policy_data { | |
| 91 | − /* the blkg and policy id this per-policy data belongs to */ | |
| 92 | − struct blkcg_gq *blkg; | |
| 93 | − int plid; | |
| 94 | −}; | |
| 95 | − | |
| 96 | −/* | |
| 97 | − * Policies that need to keep per-blkcg data which is independent from any | |
| 98 | − * request_queue associated to it should implement cpd_alloc/free_fn() | |
| 99 | − * methods. A policy can allocate private data area by allocating larger | |
| 100 | − * data structure which embeds blkcg_policy_data at the beginning. | |
| 101 | − * cpd_init() is invoked to let each policy handle per-blkcg data. | |
| 102 | − */ | |
| 103 | −struct blkcg_policy_data { | |
| 104 | − /* the blkcg and policy id this per-policy data belongs to */ | |
| 105 | − struct blkcg *blkcg; | |
| 106 | − int plid; | |
| 107 | −}; | |
| 108 | − | |
| 109 | 74 | /* association between a blk cgroup and a request queue */ |
| 110 | 75 | struct blkcg_gq { |
| 111 | 76 | /* Pointer to the associated request_queue */ |
| @@ -141,120 +106,17 @@ struct blkcg_gq { · 29 unmodified lines | ||
| 141 | 106 | struct rcu_head rcu_head; |
| 142 | 107 | }; |
| 143 | 108 | |
| 144 | −typedef struct blkcg_policy_data *(blkcg_pol_alloc_cpd_fn)(gfp_t gfp); | |
| 145 | −typedef void (blkcg_pol_init_cpd_fn)(struct blkcg_policy_data *cpd); | |
| 146 | −typedef void (blkcg_pol_free_cpd_fn)(struct blkcg_policy_data *cpd); | |
| 147 | −typedef void (blkcg_pol_bind_cpd_fn)(struct blkcg_policy_data *cpd); | |
| 148 | −typedef struct blkg_policy_data *(blkcg_pol_alloc_pd_fn)(gfp_t gfp, | |
| 149 | − struct request_queue *q, struct blkcg *blkcg); | |
| 150 | −typedef void (blkcg_pol_init_pd_fn)(struct blkg_policy_data *pd); | |
| 151 | −typedef void (blkcg_pol_online_pd_fn)(struct blkg_policy_data *pd); | |
| 152 | −typedef void (blkcg_pol_offline_pd_fn)(struct blkg_policy_data *pd); | |
| 153 | −typedef void (blkcg_pol_free_pd_fn)(struct blkg_policy_data *pd); | |
| 154 | −typedef void (blkcg_pol_reset_pd_stats_fn)(struct blkg_policy_data *pd); | |
| 155 | −typedef bool (blkcg_pol_stat_pd_fn)(struct blkg_policy_data *pd, | |
| 156 | − struct seq_file *s); | |
| 157 | − | |
| 158 | −struct blkcg_policy { | |
| 159 | − int plid; | |
| 160 | − /* cgroup files for the policy */ | |
| 161 | − struct cftype *dfl_cftypes; | |
| 162 | − struct cftype *legacy_cftypes; | |
| 163 | − | |
| 164 | − /* operations */ | |
| 165 | − blkcg_pol_alloc_cpd_fn *cpd_alloc_fn; | |
| 166 | − blkcg_pol_init_cpd_fn *cpd_init_fn; | |
| 167 | − blkcg_pol_free_cpd_fn *cpd_free_fn; | |
| 168 | − blkcg_pol_bind_cpd_fn *cpd_bind_fn; | |
| 169 | − | |
| 170 | − blkcg_pol_alloc_pd_fn *pd_alloc_fn; | |
| 171 | − blkcg_pol_init_pd_fn *pd_init_fn; | |
| 172 | − blkcg_pol_online_pd_fn *pd_online_fn; | |
| 173 | − blkcg_pol_offline_pd_fn *pd_offline_fn; | |
| 174 | − blkcg_pol_free_pd_fn *pd_free_fn; | |
| 175 | − blkcg_pol_reset_pd_stats_fn *pd_reset_stats_fn; | |
| 176 | − blkcg_pol_stat_pd_fn *pd_stat_fn; | |
| 177 | −}; | |
| 178 | − | |
| 179 | −extern struct blkcg blkcg_root; | |
| 180 | 109 | extern struct cgroup_subsys_state * const blkcg_root_css; |
| 181 | −extern bool blkcg_debug_stats; | |
| 182 | − | |
| 183 | −struct blkcg_gq *blkg_lookup_slowpath(struct blkcg *blkcg, | |
| 184 | − struct request_queue *q, bool update_hint); | |
| 185 | −int blkcg_init_queue(struct request_queue *q); | |
| 186 | −void blkcg_exit_queue(struct request_queue *q); | |
| 187 | − | |
| 188 | −/* Blkio controller policy registration */ | |
| 189 | −int blkcg_policy_register(struct blkcg_policy *pol); | |
| 190 | −void blkcg_policy_unregister(struct blkcg_policy *pol); | |
| 191 | −int blkcg_activate_policy(struct request_queue *q, | |
| 192 | − const struct blkcg_policy *pol); | |
| 193 | −void blkcg_deactivate_policy(struct request_queue *q, | |
| 194 | − const struct blkcg_policy *pol); | |
| 195 | − | |
| 196 | −const char *blkg_dev_name(struct blkcg_gq *blkg); | |
| 197 | −void blkcg_print_blkgs(struct seq_file *sf, struct blkcg *blkcg, | |
| 198 | − u64 (*prfill)(struct seq_file *, | |
| 199 | − struct blkg_policy_data *, int), | |
| 200 | − const struct blkcg_policy *pol, int data, | |
| 201 | − bool show_total); | |
| 202 | −u64 __blkg_prfill_u64(struct seq_file *sf, struct blkg_policy_data *pd, u64 v); | |
| 203 | − | |
| 204 | −struct blkg_conf_ctx { | |
| 205 | − struct block_device *bdev; | |
| 206 | − struct blkcg_gq *blkg; | |
| 207 | − char *body; | |
| 208 | −}; | |
| 209 | − | |
| 210 | −struct block_device *blkcg_conf_open_bdev(char **inputp); | |
| 211 | −int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, | |
| 212 | − char *input, struct blkg_conf_ctx *ctx); | |
| 213 | −void blkg_conf_finish(struct blkg_conf_ctx *ctx); | |
| 214 | 110 | |
| 215 | −/** | |
| 216 | − * blkcg_css - find the current css | |
| 217 | − * | |
| 218 | − * Find the css associated with either the kthread or the current task. | |
| 219 | − * This may return a dying css, so it is up to the caller to use tryget logic | |
| 220 | − * to confirm it is alive and well. | |
| 221 | − */ | |
| 222 | −static inline struct cgroup_subsys_state *blkcg_css(void) | |
| 223 | −{ | |
| 224 | − struct cgroup_subsys_state *css; | |
| 225 | − | |
| 226 | − css = kthread_blkcg(); | |
| 227 | − if (css) | |
| 228 | − return css; | |
| 229 | − return task_css(current, io_cgrp_id); | |
| 230 | −} | |
| 111 | +void blkcg_destroy_blkgs(struct blkcg *blkcg); | |
| 112 | +void blkcg_schedule_throttle(struct request_queue *q, bool use_memdelay); | |
| 113 | +void blkcg_maybe_throttle_current(void); | |
| 231 | 114 | |
| 232 | 115 | static inline struct blkcg *css_to_blkcg(struct cgroup_subsys_state *css) |
| 233 | 116 | { |
| 234 | 117 | return css ? container_of(css, struct blkcg, css) : NULL; |
| 235 | 118 | } |
| 236 | 119 | |
| 237 | −/** | |
| 238 | − * __bio_blkcg - internal, inconsistent version to get blkcg | |
| 239 | − * | |
| 240 | − * DO NOT USE. | |
| 241 | − * This function is inconsistent and consequently is dangerous to use. The | |
| 242 | − * first part of the function returns a blkcg where a reference is owned by the | |
| 243 | − * bio. This means it does not need to be rcu protected as it cannot go away | |
| 244 | − * with the bio owning a reference to it. However, the latter potentially gets | |
| 245 | − * it from task_css(). This can race against task migration and the cgroup | |
| 246 | − * dying. It is also semantically different as it must be called rcu protected | |
| 247 | − * and is susceptible to failure when trying to get a reference to it. | |
| 248 | − * Therefore, it is not ok to assume that *_get() will always succeed on the | |
| 249 | − * blkcg returned here. | |
| 250 | − */ | |
| 251 | −static inline struct blkcg *__bio_blkcg(struct bio *bio) | |
| 252 | −{ | |
| 253 | − if (bio && bio->bi_blkg) | |
| 254 | − return bio->bi_blkg->blkcg; | |
| 255 | − return css_to_blkcg(blkcg_css()); | |
| 256 | −} | |
| 257 | − | |
| 258 | 120 | /** |
| 259 | 121 | * bio_blkcg - grab the blkcg associated with a bio |
| 260 | 122 | * @bio: target bio |
| @@ -290,22 +152,6 @@ static inline bool blk_cgroup_congested(void) · 29 unmodified lines | ||
| 290 | 152 | return ret; |
| 291 | 153 | } |
| 292 | 154 | |
| 293 | −/** | |
| 294 | − * bio_issue_as_root_blkg - see if this bio needs to be issued as root blkg | |
| 295 | − * @return: true if this bio needs to be submitted with the root blkg context. | |
| 296 | − * | |
| 297 | − * In order to avoid priority inversions we sometimes need to issue a bio as if | |
| 298 | − * it were attached to the root blkg, and then backcharge to the actual owning | |
| 299 | − * blkg. The idea is we do bio_blkcg() to look up the actual context for the | |
| 300 | − * bio and attach the appropriate blkg to the bio. Then we call this helper and | |
| 301 | − * if it is true run with the root blkg for that queue and then do any | |
| 302 | − * backcharging to the originating cgroup once the io is complete. | |
| 303 | − */ | |
| 304 | −static inline bool bio_issue_as_root_blkg(struct bio *bio) | |
| 305 | −{ | |
| 306 | − return (bio->bi_opf & (REQ_META | REQ_SWAP)) != 0; | |
| 307 | −} | |
| 308 | − | |
| 309 | 155 | /** |
| 310 | 156 | * blkcg_parent - get the parent of a blkcg |
| 311 | 157 | * @blkcg: blkcg of interest |
| @@ -317,96 +163,6 @@ static inline struct blkcg *blkcg_parent(struct blkcg *blkcg) · 5 unmodified lines | ||
| 317 | 163 | return css_to_blkcg(blkcg->css.parent); |
| 318 | 164 | } |
| 319 | 165 | |
| 320 | −/** | |
| 321 | − * __blkg_lookup - internal version of blkg_lookup() | |
| 322 | − * @blkcg: blkcg of interest | |
| 323 | − * @q: request_queue of interest | |
| 324 | − * @update_hint: whether to update lookup hint with the result or not | |
| 325 | − * | |
| 326 | − * This is internal version and shouldn't be used by policy | |
| 327 | − * implementations. Looks up blkgs for the @blkcg - @q pair regardless of | |
| 328 | − * @q's bypass state. If @update_hint is %true, the caller should be | |
| 329 | − * holding @q->queue_lock and lookup hint is updated on success. | |
| 330 | − */ | |
| 331 | −static inline struct blkcg_gq *__blkg_lookup(struct blkcg *blkcg, | |
| 332 | − struct request_queue *q, | |
| 333 | − bool update_hint) | |
| 334 | −{ | |
| 335 | − struct blkcg_gq *blkg; | |
| 336 | − | |
| 337 | − if (blkcg == &blkcg_root) | |
| 338 | − return q->root_blkg; | |
| 339 | − | |
| 340 | − blkg = rcu_dereference(blkcg->blkg_hint); | |
| 341 | − if (blkg && blkg->q == q) | |
| 342 | − return blkg; | |
| 343 | − | |
| 344 | − return blkg_lookup_slowpath(blkcg, q, update_hint); | |
| 345 | −} | |
| 346 | − | |
| 347 | −/** | |
| 348 | − * blkg_lookup - lookup blkg for the specified blkcg - q pair | |
| 349 | − * @blkcg: blkcg of interest | |
| 350 | − * @q: request_queue of interest | |
| 351 | − * | |
| 352 | − * Lookup blkg for the @blkcg - @q pair. This function should be called | |
| 353 | − * under RCU read lock. | |
| 354 | − */ | |
| 355 | −static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, | |
| 356 | − struct request_queue *q) | |
| 357 | −{ | |
| 358 | − WARN_ON_ONCE(!rcu_read_lock_held()); | |
| 359 | − return __blkg_lookup(blkcg, q, false); | |
| 360 | −} | |
| 361 | − | |
| 362 | −/** | |
| 363 | − * blk_queue_root_blkg - return blkg for the (blkcg_root, @q) pair | |
| 364 | − * @q: request_queue of interest | |
| 365 | − * | |
| 366 | − * Lookup blkg for @q at the root level. See also blkg_lookup(). | |
| 367 | − */ | |
| 368 | −static inline struct blkcg_gq *blk_queue_root_blkg(struct request_queue *q) | |
| 369 | −{ | |
| 370 | − return q->root_blkg; | |
| 371 | −} | |
| 372 | − | |
| 373 | −/** | |
| 374 | − * blkg_to_pdata - get policy private data | |
| 375 | − * @blkg: blkg of interest | |
| 376 | − * @pol: policy of interest | |
| 377 | − * | |
| 378 | − * Return pointer to private data associated with the @blkg-@pol pair. | |
| 379 | − */ | |
| 380 | −static inline struct blkg_policy_data *blkg_to_pd(struct blkcg_gq *blkg, | |
| 381 | − struct blkcg_policy *pol) | |
| 382 | −{ | |
| 383 | − return blkg ? blkg->pd[pol->plid] : NULL; | |
| 384 | −} | |
| 385 | − | |
| 386 | −static inline struct blkcg_policy_data *blkcg_to_cpd(struct blkcg *blkcg, | |
| 387 | − struct blkcg_policy *pol) | |
| 388 | −{ | |
| 389 | − return blkcg ? blkcg->cpd[pol->plid] : NULL; | |
| 390 | −} | |
| 391 | − | |
| 392 | −/** | |
| 393 | − * pdata_to_blkg - get blkg associated with policy private data | |
| 394 | − * @pd: policy private data of interest | |
| 395 | − * | |
| 396 | − * @pd is policy private data. Determine the blkg it's associated with. | |
| 397 | − */ | |
| 398 | −static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) | |
| 399 | −{ | |
| 400 | − return pd ? pd->blkg : NULL; | |
| 401 | −} | |
| 402 | − | |
| 403 | −static inline struct blkcg *cpd_to_blkcg(struct blkcg_policy_data *cpd) | |
| 404 | −{ | |
| 405 | − return cpd ? cpd->blkcg : NULL; | |
| 406 | −} | |
| 407 | − | |
| 408 | −extern void blkcg_destroy_blkgs(struct blkcg *blkcg); | |
| 409 | − | |
| 410 | 166 | /** |
| 411 | 167 | * blkcg_pin_online - pin online state |
| 412 | 168 | * @blkcg: blkcg of interest |
| @@ -439,231 +195,24 @@ static inline void blkcg_unpin_online(struct blkcg *blkcg) · 26 unmodified lines | ||
| 439 | 195 | } while (blkcg); |
| 440 | 196 | } |
| 441 | 197 | |
| 442 | −/** | |
| 443 | − * blkg_path - format cgroup path of blkg | |
| 444 | − * @blkg: blkg of interest | |
| 445 | − * @buf: target buffer | |
| 446 | − * @buflen: target buffer length | |
| 447 | − * | |
| 448 | − * Format the path of the cgroup of @blkg into @buf. | |
| 449 | − */ | |
| 450 | −static inline int blkg_path(struct blkcg_gq *blkg, char *buf, int buflen) | |
| 451 | −{ | |
| 452 | − return cgroup_path(blkg->blkcg->css.cgroup, buf, buflen); | |
| 453 | −} | |
| 454 | − | |
| 455 | −/** | |
| 456 | − * blkg_get - get a blkg reference | |
| 457 | − * @blkg: blkg to get | |
| 458 | − * | |
| 459 | − * The caller should be holding an existing reference. | |
| 460 | − */ | |
| 461 | −static inline void blkg_get(struct blkcg_gq *blkg) | |
| 462 | −{ | |
| 463 | − percpu_ref_get(&blkg->refcnt); | |
| 464 | −} | |
| 465 | − | |
| 466 | −/** | |
| 467 | − * blkg_tryget - try and get a blkg reference | |
| 468 | − * @blkg: blkg to get | |
| 469 | − * | |
| 470 | − * This is for use when doing an RCU lookup of the blkg. We may be in the midst | |
| 471 | − * of freeing this blkg, so we can only use it if the refcnt is not zero. | |
| 472 | − */ | |
| 473 | −static inline bool blkg_tryget(struct blkcg_gq *blkg) | |
| 474 | −{ | |
| 475 | − return blkg && percpu_ref_tryget(&blkg->refcnt); | |
| 476 | −} | |
| 477 | − | |
| 478 | −/** | |
| 479 | − * blkg_put - put a blkg reference | |
| 480 | − * @blkg: blkg to put | |
| 481 | − */ | |
| 482 | −static inline void blkg_put(struct blkcg_gq *blkg) | |
| 483 | −{ | |
| 484 | − percpu_ref_put(&blkg->refcnt); | |
| 485 | −} | |
| 486 | − | |
| 487 | −/** | |
| 488 | − * blkg_for_each_descendant_pre - pre-order walk of a blkg's descendants | |
| 489 | − * @d_blkg: loop cursor pointing to the current descendant | |
| 490 | − * @pos_css: used for iteration | |
| 491 | − * @p_blkg: target blkg to walk descendants of | |
| 492 | − * | |
| 493 | − * Walk @c_blkg through the descendants of @p_blkg. Must be used with RCU | |
| 494 | − * read locked. If called under either blkcg or queue lock, the iteration | |
| 495 | − * is guaranteed to include all and only online blkgs. The caller may | |
| 496 | − * update @pos_css by calling css_rightmost_descendant() to skip subtree. | |
| 497 | − * @p_blkg is included in the iteration and the first node to be visited. | |
| 498 | − */ | |
| 499 | −#define blkg_for_each_descendant_pre(d_blkg, pos_css, p_blkg) \ | |
| 500 | − css_for_each_descendant_pre((pos_css), &(p_blkg)->blkcg->css) \ | |
| 501 | − if (((d_blkg) = __blkg_lookup(css_to_blkcg(pos_css), \ | |
| 502 | − (p_blkg)->q, false))) | |
| 503 | − | |
| 504 | −/** | |
| 505 | − * blkg_for_each_descendant_post - post-order walk of a blkg's descendants | |
| 506 | − * @d_blkg: loop cursor pointing to the current descendant | |
| 507 | − * @pos_css: used for iteration | |
| 508 | − * @p_blkg: target blkg to walk descendants of | |
| 509 | − * | |
| 510 | − * Similar to blkg_for_each_descendant_pre() but performs post-order | |
| 511 | − * traversal instead. Synchronization rules are the same. @p_blkg is | |
| 512 | − * included in the iteration and the last node to be visited. | |
| 513 | − */ | |
| 514 | −#define blkg_for_each_descendant_post(d_blkg, pos_css, p_blkg) \ | |
| 515 | − css_for_each_descendant_post((pos_css), &(p_blkg)->blkcg->css) \ | |
| 516 | − if (((d_blkg) = __blkg_lookup(css_to_blkcg(pos_css), \ | |
| 517 | − (p_blkg)->q, false))) | |
| 518 | − | |
| 519 | −bool __blkcg_punt_bio_submit(struct bio *bio); | |
| 520 | − | |
| 521 | −static inline bool blkcg_punt_bio_submit(struct bio *bio) | |
| 522 | −{ | |
| 523 | − if (bio->bi_opf & REQ_CGROUP_PUNT) | |
| 524 | − return __blkcg_punt_bio_submit(bio); | |
| 525 | − else | |
| 526 | − return false; | |
| 527 | −} | |
| 528 | − | |
| 529 | −static inline void blkcg_bio_issue_init(struct bio *bio) | |
| 530 | −{ | |
| 531 | − bio_issue_init(&bio->bi_issue, bio_sectors(bio)); | |
| 532 | −} | |
| 533 | − | |
| 534 | −static inline void blkcg_use_delay(struct blkcg_gq *blkg) | |
| 535 | −{ | |
| 536 | − if (WARN_ON_ONCE(atomic_read(&blkg->use_delay) < 0)) | |
| 537 | − return; | |
| 538 | − if (atomic_add_return(1, &blkg->use_delay) == 1) | |
| 539 | − atomic_inc(&blkg->blkcg->css.cgroup->congestion_count); | |
| 540 | −} | |
| 541 | − | |
| 542 | −static inline int blkcg_unuse_delay(struct blkcg_gq *blkg) | |
| 543 | −{ | |
| 544 | − int old = atomic_read(&blkg->use_delay); | |
| 545 | − | |
| 546 | − if (WARN_ON_ONCE(old < 0)) | |
| 547 | − return 0; | |
| 548 | − if (old == 0) | |
| 549 | − return 0; | |
| 550 | − | |
| 551 | − /* | |
| 552 | − * We do this song and dance because we can race with somebody else | |
| 553 | − * adding or removing delay. If we just did an atomic_dec we'd end up | |
| 554 | − * negative and we'd already be in trouble. We need to subtract 1 and | |
| 555 | − * then check to see if we were the last delay so we can drop the | |
| 556 | − * congestion count on the cgroup. | |
| 557 | − */ | |
| 558 | − while (old) { | |
| 559 | − int cur = atomic_cmpxchg(&blkg->use_delay, old, old - 1); | |
| 560 | − if (cur == old) | |
| 561 | − break; | |
| 562 | − old = cur; | |
| 563 | − } | |
| 564 | − | |
| 565 | − if (old == 0) | |
| 566 | − return 0; | |
| 567 | − if (old == 1) | |
| 568 | − atomic_dec(&blkg->blkcg->css.cgroup->congestion_count); | |
| 569 | − return 1; | |
| 570 | −} | |
| 571 | − | |
| 572 | −/** | |
| 573 | − * blkcg_set_delay - Enable allocator delay mechanism with the specified delay amount | |
| 574 | − * @blkg: target blkg | |
| 575 | − * @delay: delay duration in nsecs | |
| 576 | − * | |
| 577 | − * When enabled with this function, the delay is not decayed and must be | |
| 578 | − * explicitly cleared with blkcg_clear_delay(). Must not be mixed with | |
| 579 | − * blkcg_[un]use_delay() and blkcg_add_delay() usages. | |
| 580 | − */ | |
| 581 | −static inline void blkcg_set_delay(struct blkcg_gq *blkg, u64 delay) | |
| 582 | −{ | |
| 583 | − int old = atomic_read(&blkg->use_delay); | |
| 584 | − | |
| 585 | − /* We only want 1 person setting the congestion count for this blkg. */ | |
| 586 | − if (!old && atomic_cmpxchg(&blkg->use_delay, old, -1) == old) | |
| 587 | − atomic_inc(&blkg->blkcg->css.cgroup->congestion_count); | |
| 588 | − | |
| 589 | − atomic64_set(&blkg->delay_nsec, delay); | |
| 590 | −} | |
| 591 | − | |
| 592 | −/** | |
| 593 | − * blkcg_clear_delay - Disable allocator delay mechanism | |
| 594 | − * @blkg: target blkg | |
| 595 | − * | |
| 596 | − * Disable use_delay mechanism. See blkcg_set_delay(). | |
| 597 | − */ | |
| 598 | −static inline void blkcg_clear_delay(struct blkcg_gq *blkg) | |
| 599 | −{ | |
| 600 | − int old = atomic_read(&blkg->use_delay); | |
| 601 | − | |
| 602 | − /* We only want 1 person clearing the congestion count for this blkg. */ | |
| 603 | − if (old && atomic_cmpxchg(&blkg->use_delay, old, 0) == old) | |
| 604 | − atomic_dec(&blkg->blkcg->css.cgroup->congestion_count); | |
| 605 | −} | |
| 606 | − | |
| 607 | −void blk_cgroup_bio_start(struct bio *bio); | |
| 608 | −void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta); | |
| 609 | −void blkcg_schedule_throttle(struct request_queue *q, bool use_memdelay); | |
| 610 | −void blkcg_maybe_throttle_current(void); | |
| 611 | 198 | #else /* CONFIG_BLK_CGROUP */ |
| 612 | 199 | |
| 613 | 200 | struct blkcg { |
| 614 | 201 | }; |
| 615 | 202 | |
| 616 | −struct blkg_policy_data { | |
| 617 | −}; | |
| 618 | − | |
| 619 | −struct blkcg_policy_data { | |
| 620 | −}; | |
| 621 | − | |
| 622 | 203 | struct blkcg_gq { |
| 623 | 204 | }; |
| 624 | 205 | |
| 625 | −struct blkcg_policy { | |
| 626 | −}; | |
| 627 | − | |
| 628 | 206 | #define blkcg_root_css ((struct cgroup_subsys_state *)ERR_PTR(-EINVAL)) |
| 629 | 207 | |
| 630 | 208 | static inline void blkcg_maybe_throttle_current(void) { } |
| 631 | 209 | static inline bool blk_cgroup_congested(void) { return false; } |
| 632 | 210 | |
| 633 | 211 | #ifdef CONFIG_BLOCK |
| 634 | − | |
| 635 | 212 | static inline void blkcg_schedule_throttle(struct request_queue *q, bool use_memdelay) { } |
| 636 | − | |
| 637 | −static inline struct blkcg_gq *blkg_lookup(struct blkcg *blkcg, void *key) { return NULL; } | |
| 638 | −static inline struct blkcg_gq *blk_queue_root_blkg(struct request_queue *q) | |
| 639 | −{ return NULL; } | |
| 640 | −static inline int blkcg_init_queue(struct request_queue *q) { return 0; } | |
| 641 | −static inline void blkcg_exit_queue(struct request_queue *q) { } | |
| 642 | −static inline int blkcg_policy_register(struct blkcg_policy *pol) { return 0; } | |
| 643 | −static inline void blkcg_policy_unregister(struct blkcg_policy *pol) { } | |
| 644 | −static inline int blkcg_activate_policy(struct request_queue *q, | |
| 645 | − const struct blkcg_policy *pol) { return 0; } | |
| 646 | −static inline void blkcg_deactivate_policy(struct request_queue *q, | |
| 647 | − const struct blkcg_policy *pol) { } | |
| 648 | − | |
| 649 | −static inline struct blkcg *__bio_blkcg(struct bio *bio) { return NULL; } | |
| 650 | 213 | static inline struct blkcg *bio_blkcg(struct bio *bio) { return NULL; } |
| 214 | +#endif /* CONFIG_BLOCK */ | |
| 651 | 215 | |
| 652 | −static inline struct blkg_policy_data *blkg_to_pd(struct blkcg_gq *blkg, | |
| 653 | − struct blkcg_policy *pol) { return NULL; } | |
| 654 | −static inline struct blkcg_gq *pd_to_blkg(struct blkg_policy_data *pd) { return NULL; } | |
| 655 | −static inline char *blkg_path(struct blkcg_gq *blkg) { return NULL; } | |
| 656 | −static inline void blkg_get(struct blkcg_gq *blkg) { } | |
| 657 | −static inline void blkg_put(struct blkcg_gq *blkg) { } | |
| 658 | − | |
| 659 | −static inline bool blkcg_punt_bio_submit(struct bio *bio) { return false; } | |
| 660 | −static inline void blkcg_bio_issue_init(struct bio *bio) { } | |
| 661 | −static inline void blk_cgroup_bio_start(struct bio *bio) { } | |
| 662 | − | |
| 663 | −#define blk_queue_for_each_rl(rl, q) \ | |
| 664 | − for ((rl) = &(q)->root_rl; (rl); (rl) = NULL) | |
| 665 | − | |
| 666 | −#endif /* CONFIG_BLOCK */ | |
| 667 | 216 | #endif /* CONFIG_BLK_CGROUP */ |
| 668 | 217 | |
| 669 | 218 | #ifdef CONFIG_BLK_CGROUP_FC_APPID |
| @@ -917,8 +917,7 @@ static inline void *blk_mq_rq_to_pdu(struct request *rq) · 916 unmodified lines | ||
| 917 | 917 | } |
| 918 | 918 | |
| 919 | 919 | #define queue_for_each_hw_ctx(q, hctx, i) \ |
| 920 | − for ((i) = 0; (i) < (q)->nr_hw_queues && \ | |
| 921 | − ({ hctx = (q)->queue_hw_ctx[i]; 1; }); (i)++) | |
| 920 | + xa_for_each(&(q)->hctx_table, (i), (hctx)) | |
| 922 | 921 | |
| 923 | 922 | #define hctx_for_each_ctx(hctx, ctx, i) \ |
| 924 | 923 | for ((i) = 0; (i) < (hctx)->nr_ctx && \ |
| @@ -952,8 +951,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, · 27 unmodified lines | ||
| 952 | 951 | struct bio_set *bs, gfp_t gfp_mask, |
| 953 | 952 | int (*bio_ctr)(struct bio *, struct bio *, void *), void *data); |
| 954 | 953 | void blk_rq_unprep_clone(struct request *rq); |
| 955 | −blk_status_t blk_insert_cloned_request(struct request_queue *q, | |
| 956 | − struct request *rq); | |
| 954 | +blk_status_t blk_insert_cloned_request(struct request *rq); | |
| 957 | 955 | |
| 958 | 956 | struct rq_map_data { |
| 959 | 957 | struct page **pages; |
| @@ -153,6 +153,13 @@ typedef u8 __bitwise blk_status_t; · 152 unmodified lines | ||
| 153 | 153 | */ |
| 154 | 154 | #define BLK_STS_ZONE_ACTIVE_RESOURCE ((__force blk_status_t)16) |
| 155 | 155 | |
| 156 | +/* | |
| 157 | + * BLK_STS_OFFLINE is returned from the driver when the target device is offline | |
| 158 | + * or is being taken offline. This could help differentiate the case where a | |
| 159 | + * device is intentionally being shut down from a real I/O error. | |
| 160 | + */ | |
| 161 | +#define BLK_STS_OFFLINE ((__force blk_status_t)17) | |
| 162 | + | |
| 156 | 163 | /** |
| 157 | 164 | * blk_path_error - returns true if error may be path related |
| 158 | 165 | * @error: status the request was completed with |
| @@ -317,7 +324,8 @@ enum { · 158 unmodified lines | ||
| 317 | 324 | BIO_TRACE_COMPLETION, /* bio_endio() should trace the final completion |
| 318 | 325 | * of this bio. */ |
| 319 | 326 | BIO_CGROUP_ACCT, /* has been accounted to a cgroup */ |
| 320 | − BIO_TRACKED, /* set if bio goes through the rq_qos path */ | |
| 327 | + BIO_QOS_THROTTLED, /* bio went through rq_qos throttle path */ | |
| 328 | + BIO_QOS_MERGED, /* but went through rq_qos merge path */ | |
| 321 | 329 | BIO_REMAPPED, |
| 322 | 330 | BIO_ZONE_WRITE_LOCKED, /* Owns a zoned device zone write lock */ |
| 323 | 331 | BIO_PERCPU_CACHE, /* can participate in per-cpu alloc cache */ |
| @@ -1,9 +1,13 @@ | ||
| 1 | 1 | /* SPDX-License-Identifier: GPL-2.0 */ |
| 2 | +/* | |
| 3 | + * Portions Copyright (C) 1992 Drew Eckhardt | |
| 4 | + */ | |
| 2 | 5 | #ifndef _LINUX_BLKDEV_H |
| 3 | 6 | #define _LINUX_BLKDEV_H |
| 4 | 7 | |
| 5 | −#include <linux/sched.h> | |
| 6 | −#include <linux/genhd.h> | |
| 8 | +#include <linux/types.h> | |
| 9 | +#include <linux/blk_types.h> | |
| 10 | +#include <linux/device.h> | |
| 7 | 11 | #include <linux/list.h> |
| 8 | 12 | #include <linux/llist.h> |
| 9 | 13 | #include <linux/minmax.h> |
| @@ -12,11 +16,15 @@ · 2 unmodified lines | ||
| 12 | 16 | #include <linux/wait.h> |
| 13 | 17 | #include <linux/bio.h> |
| 14 | 18 | #include <linux/gfp.h> |
| 19 | +#include <linux/kdev_t.h> | |
| 15 | 20 | #include <linux/rcupdate.h> |
| 16 | 21 | #include <linux/percpu-refcount.h> |
| 17 | 22 | #include <linux/blkzoned.h> |
| 23 | +#include <linux/sched.h> | |
| 18 | 24 | #include <linux/sbitmap.h> |
| 19 | 25 | #include <linux/srcu.h> |
| 26 | +#include <linux/uuid.h> | |
| 27 | +#include <linux/xarray.h> | |
| 20 | 28 | |
| 21 | 29 | struct module; |
| 22 | 30 | struct request_queue; |
| @@ -33,6 +41,10 @@ struct blk_queue_stats; · 10 unmodified lines | ||
| 33 | 41 | struct blk_stat_callback; |
| 34 | 42 | struct blk_crypto_profile; |
| 35 | 43 | |
| 44 | +extern const struct device_type disk_type; | |
| 45 | +extern struct device_type part_type; | |
| 46 | +extern struct class block_class; | |
| 47 | + | |
| 36 | 48 | /* Must be consistent with blk_mq_poll_stats_bkt() */ |
| 37 | 49 | #define BLK_MQ_POLL_STATS_BKTS 16 |
| 38 | 50 | |
| @@ -45,6 +57,145 @@ struct blk_crypto_profile; · 6 unmodified lines | ||
| 45 | 57 | */ |
| 46 | 58 | #define BLKCG_MAX_POLS 6 |
| 47 | 59 | |
| 60 | +#define DISK_MAX_PARTS 256 | |
| 61 | +#define DISK_NAME_LEN 32 | |
| 62 | + | |
| 63 | +#define PARTITION_META_INFO_VOLNAMELTH 64 | |
| 64 | +/* | |
| 65 | + * Enough for the string representation of any kind of UUID plus NULL. | |
| 66 | + * EFI UUID is 36 characters. MSDOS UUID is 11 characters. | |
| 67 | + */ | |
| 68 | +#define PARTITION_META_INFO_UUIDLTH (UUID_STRING_LEN + 1) | |
| 69 | + | |
| 70 | +struct partition_meta_info { | |
| 71 | + char uuid[PARTITION_META_INFO_UUIDLTH]; | |
| 72 | + u8 volname[PARTITION_META_INFO_VOLNAMELTH]; | |
| 73 | +}; | |
| 74 | + | |
| 75 | +/** | |
| 76 | + * DOC: genhd capability flags | |
| 77 | + * | |
| 78 | + * ``GENHD_FL_REMOVABLE``: indicates that the block device gives access to | |
| 79 | + * removable media. When set, the device remains present even when media is not | |
| 80 | + * inserted. Shall not be set for devices which are removed entirely when the | |
| 81 | + * media is removed. | |
| 82 | + * | |
| 83 | + * ``GENHD_FL_HIDDEN``: the block device is hidden; it doesn't produce events, | |
| 84 | + * doesn't appear in sysfs, and can't be opened from userspace or using | |
| 85 | + * blkdev_get*. Used for the underlying components of multipath devices. | |
| 86 | + * | |
| 87 | + * ``GENHD_FL_NO_PART``: partition support is disabled. The kernel will not | |
| 88 | + * scan for partitions from add_disk, and users can't add partitions manually. | |
| 89 | + * | |
| 90 | + */ | |
| 91 | +enum { | |
| 92 | + GENHD_FL_REMOVABLE = 1 << 0, | |
| 93 | + GENHD_FL_HIDDEN = 1 << 1, | |
| 94 | + GENHD_FL_NO_PART = 1 << 2, | |
| 95 | +}; | |
| 96 | + | |
| 97 | +enum { | |
| 98 | + DISK_EVENT_MEDIA_CHANGE = 1 << 0, /* media changed */ | |
| 99 | + DISK_EVENT_EJECT_REQUEST = 1 << 1, /* eject requested */ | |
| 100 | +}; | |
| 101 | + | |
| 102 | +enum { | |
| 103 | + /* Poll even if events_poll_msecs is unset */ | |
| 104 | + DISK_EVENT_FLAG_POLL = 1 << 0, | |
| 105 | + /* Forward events to udev */ | |
| 106 | + DISK_EVENT_FLAG_UEVENT = 1 << 1, | |
| 107 | + /* Block event polling when open for exclusive write */ | |
| 108 | + DISK_EVENT_FLAG_BLOCK_ON_EXCL_WRITE = 1 << 2, | |
| 109 | +}; | |
| 110 | + | |
| 111 | +struct disk_events; | |
| 112 | +struct badblocks; | |
| 113 | + | |
| 114 | +struct blk_integrity { | |
| 115 | + const struct blk_integrity_profile *profile; | |
| 116 | + unsigned char flags; | |
| 117 | + unsigned char tuple_size; | |
| 118 | + unsigned char interval_exp; | |
| 119 | + unsigned char tag_size; | |
| 120 | +}; | |
| 121 | + | |
| 122 | +struct gendisk { | |
| 123 | + /* | |
| 124 | + * major/first_minor/minors should not be set by any new driver, the | |
| 125 | + * block core will take care of allocating them automatically. | |
| 126 | + */ | |
| 127 | + int major; | |
| 128 | + int first_minor; | |
| 129 | + int minors; | |
| 130 | + | |
| 131 | + char disk_name[DISK_NAME_LEN]; /* name of major driver */ | |
| 132 | + | |
| 133 | + unsigned short events; /* supported events */ | |
| 134 | + unsigned short event_flags; /* flags related to event processing */ | |
| 135 | + | |
| 136 | + struct xarray part_tbl; | |
| 137 | + struct block_device *part0; | |
| 138 | + | |
| 139 | + const struct block_device_operations *fops; | |
| 140 | + struct request_queue *queue; | |
| 141 | + void *private_data; | |
| 142 | + | |
| 143 | + int flags; | |
| 144 | + unsigned long state; | |
| 145 | +#define GD_NEED_PART_SCAN 0 | |
| 146 | +#define GD_READ_ONLY 1 | |
| 147 | +#define GD_DEAD 2 | |
| 148 | +#define GD_NATIVE_CAPACITY 3 | |
| 149 | +#define GD_ADDED 4 | |
| 150 | + | |
| 151 | + struct mutex open_mutex; /* open/close mutex */ | |
| 152 | + unsigned open_partitions; /* number of open partitions */ | |
| 153 | + | |
| 154 | + struct backing_dev_info *bdi; | |
| 155 | + struct kobject *slave_dir; | |
| 156 | +#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED | |
| 157 | + struct list_head slave_bdevs; | |
| 158 | +#endif | |
| 159 | + struct timer_rand_state *random; | |
| 160 | + atomic_t sync_io; /* RAID */ | |
| 161 | + struct disk_events *ev; | |
| 162 | +#ifdef CONFIG_BLK_DEV_INTEGRITY | |
| 163 | + struct kobject integrity_kobj; | |
| 164 | +#endif /* CONFIG_BLK_DEV_INTEGRITY */ | |
| 165 | +#if IS_ENABLED(CONFIG_CDROM) | |
| 166 | + struct cdrom_device_info *cdi; | |
| 167 | +#endif | |
| 168 | + int node_id; | |
| 169 | + struct badblocks *bb; | |
| 170 | + struct lockdep_map lockdep_map; | |
| 171 | + u64 diskseq; | |
| 172 | +}; | |
| 173 | + | |
| 174 | +static inline bool disk_live(struct gendisk *disk) | |
| 175 | +{ | |
| 176 | + return !inode_unhashed(disk->part0->bd_inode); | |
| 177 | +} | |
| 178 | + | |
| 179 | +/* | |
| 180 | + * The gendisk is refcounted by the part0 block_device, and the bd_device | |
| 181 | + * therein is also used for device model presentation in sysfs. | |
| 182 | + */ | |
| 183 | +#define dev_to_disk(device) \ | |
| 184 | + (dev_to_bdev(device)->bd_disk) | |
| 185 | +#define disk_to_dev(disk) \ | |
| 186 | + (&((disk)->part0->bd_device)) | |
| 187 | + | |
| 188 | +#if IS_REACHABLE(CONFIG_CDROM) | |
| 189 | +#define disk_to_cdi(disk) ((disk)->cdi) | |
| 190 | +#else | |
| 191 | +#define disk_to_cdi(disk) NULL | |
| 192 | +#endif | |
| 193 | + | |
| 194 | +static inline dev_t disk_devt(struct gendisk *disk) | |
| 195 | +{ | |
| 196 | + return MKDEV(disk->major, disk->first_minor); | |
| 197 | +} | |
| 198 | + | |
| 48 | 199 | static inline int blk_validate_block_size(unsigned long bsize) |
| 49 | 200 | { |
| 50 | 201 | if (bsize < 512 || bsize > PAGE_SIZE || !is_power_of_2(bsize)) |
| @@ -204,7 +355,7 @@ struct request_queue { · 153 unmodified lines | ||
| 204 | 355 | unsigned int queue_depth; |
| 205 | 356 | |
| 206 | 357 | /* hw dispatch queues */ |
| 207 | − struct blk_mq_hw_ctx **queue_hw_ctx; | |
| 358 | + struct xarray hctx_table; | |
| 208 | 359 | unsigned int nr_hw_queues; |
| 209 | 360 | |
| 210 | 361 | /* |
| @@ -262,6 +413,7 @@ struct request_queue { · 51 unmodified lines | ||
| 262 | 413 | |
| 263 | 414 | #ifdef CONFIG_BLK_INLINE_ENCRYPTION |
| 264 | 415 | struct blk_crypto_profile *crypto_profile; |
| 416 | + struct kobject *crypto_kobject; | |
| 265 | 417 | #endif |
| 266 | 418 | |
| 267 | 419 | unsigned int rq_timeout; |
| @@ -596,6 +748,118 @@ static inline unsigned int blk_queue_depth(struct request_queue *q) · 328 unmodified lines | ||
| 596 | 748 | #define for_each_bio(_bio) \ |
| 597 | 749 | for (; _bio; _bio = _bio->bi_next) |
| 598 | 750 | |
| 751 | +int __must_check device_add_disk(struct device *parent, struct gendisk *disk, | |
| 752 | + const struct attribute_group **groups); | |
| 753 | +static inline int __must_check add_disk(struct gendisk *disk) | |
| 754 | +{ | |
| 755 | + return device_add_disk(NULL, disk, NULL); | |
| 756 | +} | |
| 757 | +void del_gendisk(struct gendisk *gp); | |
| 758 | +void invalidate_disk(struct gendisk *disk); | |
| 759 | +void set_disk_ro(struct gendisk *disk, bool read_only); | |
| 760 | +void disk_uevent(struct gendisk *disk, enum kobject_action action); | |
| 761 | + | |
| 762 | +static inline int get_disk_ro(struct gendisk *disk) | |
| 763 | +{ | |
| 764 | + return disk->part0->bd_read_only || | |
| 765 | + test_bit(GD_READ_ONLY, &disk->state); | |
| 766 | +} | |
| 767 | + | |
| 768 | +static inline int bdev_read_only(struct block_device *bdev) | |
| 769 | +{ | |
| 770 | + return bdev->bd_read_only || get_disk_ro(bdev->bd_disk); | |
| 771 | +} | |
| 772 | + | |
| 773 | +bool set_capacity_and_notify(struct gendisk *disk, sector_t size); | |
| 774 | +bool disk_force_media_change(struct gendisk *disk, unsigned int events); | |
| 775 | + | |
| 776 | +void add_disk_randomness(struct gendisk *disk) __latent_entropy; | |
| 777 | +void rand_initialize_disk(struct gendisk *disk); | |
| 778 | + | |
| 779 | +static inline sector_t get_start_sect(struct block_device *bdev) | |
| 780 | +{ | |
| 781 | + return bdev->bd_start_sect; | |
| 782 | +} | |
| 783 | + | |
| 784 | +static inline sector_t bdev_nr_sectors(struct block_device *bdev) | |
| 785 | +{ | |
| 786 | + return bdev->bd_nr_sectors; | |
| 787 | +} | |
| 788 | + | |
| 789 | +static inline loff_t bdev_nr_bytes(struct block_device *bdev) | |
| 790 | +{ | |
| 791 | + return (loff_t)bdev_nr_sectors(bdev) << SECTOR_SHIFT; | |
| 792 | +} | |
| 793 | + | |
| 794 | +static inline sector_t get_capacity(struct gendisk *disk) | |
| 795 | +{ | |
| 796 | + return bdev_nr_sectors(disk->part0); | |
| 797 | +} | |
| 798 | + | |
| 799 | +static inline u64 sb_bdev_nr_blocks(struct super_block *sb) | |
| 800 | +{ | |
| 801 | + return bdev_nr_sectors(sb->s_bdev) >> | |
| 802 | + (sb->s_blocksize_bits - SECTOR_SHIFT); | |
| 803 | +} | |
| 804 | + | |
| 805 | +int bdev_disk_changed(struct gendisk *disk, bool invalidate); | |
| 806 | + | |
| 807 | +struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id, | |
| 808 | + struct lock_class_key *lkclass); | |
| 809 | +void put_disk(struct gendisk *disk); | |
| 810 | +struct gendisk *__blk_alloc_disk(int node, struct lock_class_key *lkclass); | |
| 811 | + | |
| 812 | +/** | |
| 813 | + * blk_alloc_disk - allocate a gendisk structure | |
| 814 | + * @node_id: numa node to allocate on | |
| 815 | + * | |
| 816 | + * Allocate and pre-initialize a gendisk structure for use with BIO based | |
| 817 | + * drivers. | |
| 818 | + * | |
| 819 | + * Context: can sleep | |
| 820 | + */ | |
| 821 | +#define blk_alloc_disk(node_id) \ | |
| 822 | +({ \ | |
| 823 | + static struct lock_class_key __key; \ | |
| 824 | + \ | |
| 825 | + __blk_alloc_disk(node_id, &__key); \ | |
| 826 | +}) | |
| 827 | +void blk_cleanup_disk(struct gendisk *disk); | |
| 828 | + | |
| 829 | +int __register_blkdev(unsigned int major, const char *name, | |
| 830 | + void (*probe)(dev_t devt)); | |
| 831 | +#define register_blkdev(major, name) \ | |
| 832 | + __register_blkdev(major, name, NULL) | |
| 833 | +void unregister_blkdev(unsigned int major, const char *name); | |
| 834 | + | |
| 835 | +bool bdev_check_media_change(struct block_device *bdev); | |
| 836 | +int __invalidate_device(struct block_device *bdev, bool kill_dirty); | |
| 837 | +void set_capacity(struct gendisk *disk, sector_t size); | |
| 838 | + | |
| 839 | +#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED | |
| 840 | +int bd_link_disk_holder(struct block_device *bdev, struct gendisk *disk); | |
| 841 | +void bd_unlink_disk_holder(struct block_device *bdev, struct gendisk *disk); | |
| 842 | +int bd_register_pending_holders(struct gendisk *disk); | |
| 843 | +#else | |
| 844 | +static inline int bd_link_disk_holder(struct block_device *bdev, | |
| 845 | + struct gendisk *disk) | |
| 846 | +{ | |
| 847 | + return 0; | |
| 848 | +} | |
| 849 | +static inline void bd_unlink_disk_holder(struct block_device *bdev, | |
| 850 | + struct gendisk *disk) | |
| 851 | +{ | |
| 852 | +} | |
| 853 | +static inline int bd_register_pending_holders(struct gendisk *disk) | |
| 854 | +{ | |
| 855 | + return 0; | |
| 856 | +} | |
| 857 | +#endif /* CONFIG_BLOCK_HOLDER_DEPRECATED */ | |
| 858 | + | |
| 859 | +dev_t part_devt(struct gendisk *disk, u8 partno); | |
| 860 | +void inc_diskseq(struct gendisk *disk); | |
| 861 | +dev_t blk_lookup_devt(const char *name, int partno); | |
| 862 | +void blk_request_module(dev_t devt); | |
| 599 | 863 | |
| 600 | 864 | extern int blk_register_queue(struct gendisk *disk); |
| 601 | 865 | extern void blk_unregister_queue(struct gendisk *disk); |
| @@ -792,14 +1056,11 @@ extern void blk_start_plug(struct blk_plug *); · 190 unmodified lines | ||
| 792 | 1056 | extern void blk_start_plug_nr_ios(struct blk_plug *, unsigned short); |
| 793 | 1057 | extern void blk_finish_plug(struct blk_plug *); |
| 794 | 1058 | |
| 795 | −void blk_flush_plug(struct blk_plug *plug, bool from_schedule); | |
| 796 | − | |
| 797 | −static inline bool blk_needs_flush_plug(struct task_struct *tsk) | |
| 1059 | +void __blk_flush_plug(struct blk_plug *plug, bool from_schedule); | |
| 1060 | +static inline void blk_flush_plug(struct blk_plug *plug, bool async) | |
| 798 | 1061 | { |
| 799 | − struct blk_plug *plug = tsk->plug; | |
| 800 | − | |
| 801 | − return plug && | |
| 802 | − (plug->mq_list || !list_empty(&plug->cb_list)); | |
| 1062 | + if (plug) | |
| 1063 | + __blk_flush_plug(plug, async); | |
| 803 | 1064 | } |
| 804 | 1065 | |
| 805 | 1066 | int blkdev_issue_flush(struct block_device *bdev); |
| @@ -825,11 +1086,6 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) · 19 unmodified lines | ||
| 825 | 1086 | { |
| 826 | 1087 | } |
| 827 | 1088 | |
| 828 | −static inline bool blk_needs_flush_plug(struct task_struct *tsk) | |
| 829 | −{ | |
| 830 | − return false; | |
| 831 | −} | |
| 832 | − | |
| 833 | 1089 | static inline int blkdev_issue_flush(struct block_device *bdev) |
| 834 | 1090 | { |
| 835 | 1091 | return 0; |
| @@ -1211,6 +1467,7 @@ struct block_device_operations { · 375 unmodified lines | ||
| 1211 | 1467 | void (*unlock_native_capacity) (struct gendisk *); |
| 1212 | 1468 | int (*getgeo)(struct block_device *, struct hd_geometry *); |
| 1213 | 1469 | int (*set_read_only)(struct block_device *bdev, bool ro); |
| 1470 | + void (*free_disk)(struct gendisk *disk); | |
| 1214 | 1471 | /* this callback is with swap_lock and sometimes page table lock held */ |
| 1215 | 1472 | void (*swap_slot_free_notify) (struct block_device *, unsigned long); |
| 1216 | 1473 | int (*report_zones)(struct gendisk *, sector_t sector, |
| @@ -1267,7 +1524,7 @@ void bio_end_io_acct_remapped(struct bio *bio, unsigned long start_time, · 50 unmodified lines | ||
| 1267 | 1524 | /** |
| 1268 | 1525 | * bio_end_io_acct - end I/O accounting for bio based drivers |
| 1269 | 1526 | * @bio: bio to end account for |
| 1270 | − * @start: start time returned by bio_start_io_acct() | |
| 1527 | + * @start_time: start time returned by bio_start_io_acct() | |
| 1271 | 1528 | */ |
| 1272 | 1529 | static inline void bio_end_io_acct(struct bio *bio, unsigned long start_time) |
| 1273 | 1530 | { |
| @@ -1312,6 +1569,7 @@ void invalidate_bdev(struct block_device *bdev); · 38 unmodified lines | ||
| 1312 | 1569 | int sync_blockdev(struct block_device *bdev); |
| 1313 | 1570 | int sync_blockdev_nowait(struct block_device *bdev); |
| 1314 | 1571 | void sync_bdevs(bool wait); |
| 1572 | +void printk_all_partitions(void); | |
| 1315 | 1573 | #else |
| 1316 | 1574 | static inline void invalidate_bdev(struct block_device *bdev) |
| 1317 | 1575 | { |
| @@ -1327,7 +1585,11 @@ static inline int sync_blockdev_nowait(struct block_device *bdev) · 9 unmodified lines | ||
| 1327 | 1585 | static inline void sync_bdevs(bool wait) |
| 1328 | 1586 | { |
| 1329 | 1587 | } |
| 1330 | −#endif | |
| 1588 | +static inline void printk_all_partitions(void) | |
| 1589 | +{ | |
| 1590 | +} | |
| 1591 | +#endif /* CONFIG_BLOCK */ | |
| 1592 | + | |
| 1331 | 1593 | int fsync_bdev(struct block_device *bdev); |
| 1332 | 1594 | |
| 1333 | 1595 | int freeze_bdev(struct block_device *bdev); |
| @@ -1,291 +0,0 @@ | ||
| 1 | −/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | −#ifndef _LINUX_GENHD_H | |
| 3 | −#define _LINUX_GENHD_H | |
| 4 | − | |
| 5 | −/* | |
| 6 | − * genhd.h Copyright (C) 1992 Drew Eckhardt | |
| 7 | − * Generic hard disk header file by | |
| 8 | − * Drew Eckhardt | |
| 9 | − * | |
| 10 | − * <drew@colorado.edu> | |
| 11 | − */ | |
| 12 | − | |
| 13 | −#include <linux/types.h> | |
| 14 | −#include <linux/kdev_t.h> | |
| 15 | −#include <linux/uuid.h> | |
| 16 | −#include <linux/blk_types.h> | |
| 17 | −#include <linux/device.h> | |
| 18 | −#include <linux/xarray.h> | |
| 19 | − | |
| 20 | −extern const struct device_type disk_type; | |
| 21 | −extern struct device_type part_type; | |
| 22 | −extern struct class block_class; | |
| 23 | − | |
| 24 | −#define DISK_MAX_PARTS 256 | |
| 25 | −#define DISK_NAME_LEN 32 | |
| 26 | − | |
| 27 | −#define PARTITION_META_INFO_VOLNAMELTH 64 | |
| 28 | −/* | |
| 29 | − * Enough for the string representation of any kind of UUID plus NULL. | |
| 30 | − * EFI UUID is 36 characters. MSDOS UUID is 11 characters. | |
| 31 | − */ | |
| 32 | −#define PARTITION_META_INFO_UUIDLTH (UUID_STRING_LEN + 1) | |
| 33 | − | |
| 34 | −struct partition_meta_info { | |
| 35 | − char uuid[PARTITION_META_INFO_UUIDLTH]; | |
| 36 | − u8 volname[PARTITION_META_INFO_VOLNAMELTH]; | |
| 37 | −}; | |
| 38 | − | |
| 39 | −/** | |
| 40 | − * DOC: genhd capability flags | |
| 41 | − * | |
| 42 | − * ``GENHD_FL_REMOVABLE``: indicates that the block device gives access to | |
| 43 | − * removable media. When set, the device remains present even when media is not | |
| 44 | − * inserted. Shall not be set for devices which are removed entirely when the | |
| 45 | − * media is removed. | |
| 46 | − * | |
| 47 | − * ``GENHD_FL_HIDDEN``: the block device is hidden; it doesn't produce events, | |
| 48 | − * doesn't appear in sysfs, and can't be opened from userspace or using | |
| 49 | − * blkdev_get*. Used for the underlying components of multipath devices. | |
| 50 | − * | |
| 51 | − * ``GENHD_FL_NO_PART``: partition support is disabled. The kernel will not | |
| 52 | − * scan for partitions from add_disk, and users can't add partitions manually. | |
| 53 | − * | |
| 54 | − */ | |
| 55 | −enum { | |
| 56 | − GENHD_FL_REMOVABLE = 1 << 0, | |
| 57 | − GENHD_FL_HIDDEN = 1 << 1, | |
| 58 | − GENHD_FL_NO_PART = 1 << 2, | |
| 59 | −}; | |
| 60 | − | |
| 61 | −enum { | |
| 62 | − DISK_EVENT_MEDIA_CHANGE = 1 << 0, /* media changed */ | |
| 63 | − DISK_EVENT_EJECT_REQUEST = 1 << 1, /* eject requested */ | |
| 64 | −}; | |
| 65 | − | |
| 66 | −enum { | |
| 67 | − /* Poll even if events_poll_msecs is unset */ | |
| 68 | − DISK_EVENT_FLAG_POLL = 1 << 0, | |
| 69 | − /* Forward events to udev */ | |
| 70 | − DISK_EVENT_FLAG_UEVENT = 1 << 1, | |
| 71 | − /* Block event polling when open for exclusive write */ | |
| 72 | − DISK_EVENT_FLAG_BLOCK_ON_EXCL_WRITE = 1 << 2, | |
| 73 | −}; | |
| 74 | − | |
| 75 | −struct disk_events; | |
| 76 | −struct badblocks; | |
| 77 | − | |
| 78 | −struct blk_integrity { | |
| 79 | − const struct blk_integrity_profile *profile; | |
| 80 | − unsigned char flags; | |
| 81 | − unsigned char tuple_size; | |
| 82 | − unsigned char interval_exp; | |
| 83 | − unsigned char tag_size; | |
| 84 | −}; | |
| 85 | − | |
| 86 | −struct gendisk { | |
| 87 | − /* | |
| 88 | − * major/first_minor/minors should not be set by any new driver, the | |
| 89 | − * block core will take care of allocating them automatically. | |
| 90 | − */ | |
| 91 | − int major; | |
| 92 | − int first_minor; | |
| 93 | − int minors; | |
| 94 | − | |
| 95 | − char disk_name[DISK_NAME_LEN]; /* name of major driver */ | |
| 96 | − | |
| 97 | − unsigned short events; /* supported events */ | |
| 98 | − unsigned short event_flags; /* flags related to event processing */ | |
| 99 | − | |
| 100 | − struct xarray part_tbl; | |
| 101 | − struct block_device *part0; | |
| 102 | − | |
| 103 | − const struct block_device_operations *fops; | |
| 104 | − struct request_queue *queue; | |
| 105 | − void *private_data; | |
| 106 | − | |
| 107 | − int flags; | |
| 108 | − unsigned long state; | |
| 109 | −#define GD_NEED_PART_SCAN 0 | |
| 110 | −#define GD_READ_ONLY 1 | |
| 111 | −#define GD_DEAD 2 | |
| 112 | −#define GD_NATIVE_CAPACITY 3 | |
| 113 | − | |
| 114 | − struct mutex open_mutex; /* open/close mutex */ | |
| 115 | − unsigned open_partitions; /* number of open partitions */ | |
| 116 | − | |
| 117 | − struct backing_dev_info *bdi; | |
| 118 | − struct kobject *slave_dir; | |
| 119 | −#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED | |
| 120 | − struct list_head slave_bdevs; | |
| 121 | −#endif | |
| 122 | − struct timer_rand_state *random; | |
| 123 | − atomic_t sync_io; /* RAID */ | |
| 124 | − struct disk_events *ev; | |
| 125 | −#ifdef CONFIG_BLK_DEV_INTEGRITY | |
| 126 | − struct kobject integrity_kobj; | |
| 127 | −#endif /* CONFIG_BLK_DEV_INTEGRITY */ | |
| 128 | −#if IS_ENABLED(CONFIG_CDROM) | |
| 129 | − struct cdrom_device_info *cdi; | |
| 130 | −#endif | |
| 131 | − int node_id; | |
| 132 | − struct badblocks *bb; | |
| 133 | − struct lockdep_map lockdep_map; | |
| 134 | − u64 diskseq; | |
| 135 | −}; | |
| 136 | − | |
| 137 | −static inline bool disk_live(struct gendisk *disk) | |
| 138 | −{ | |
| 139 | − return !inode_unhashed(disk->part0->bd_inode); | |
| 140 | −} | |
| 141 | − | |
| 142 | −/* | |
| 143 | − * The gendisk is refcounted by the part0 block_device, and the bd_device | |
| 144 | − * therein is also used for device model presentation in sysfs. | |
| 145 | − */ | |
| 146 | −#define dev_to_disk(device) \ | |
| 147 | − (dev_to_bdev(device)->bd_disk) | |
| 148 | −#define disk_to_dev(disk) \ | |
| 149 | − (&((disk)->part0->bd_device)) | |
| 150 | − | |
| 151 | −#if IS_REACHABLE(CONFIG_CDROM) | |
| 152 | −#define disk_to_cdi(disk) ((disk)->cdi) | |
| 153 | −#else | |
| 154 | −#define disk_to_cdi(disk) NULL | |
| 155 | −#endif | |
| 156 | − | |
| 157 | −static inline dev_t disk_devt(struct gendisk *disk) | |
| 158 | −{ | |
| 159 | − return MKDEV(disk->major, disk->first_minor); | |
| 160 | −} | |
| 161 | − | |
| 162 | −void disk_uevent(struct gendisk *disk, enum kobject_action action); | |
| 163 | − | |
| 164 | −/* block/genhd.c */ | |
| 165 | −int __must_check device_add_disk(struct device *parent, struct gendisk *disk, | |
| 166 | − const struct attribute_group **groups); | |
| 167 | −static inline int __must_check add_disk(struct gendisk *disk) | |
| 168 | −{ | |
| 169 | − return device_add_disk(NULL, disk, NULL); | |
| 170 | −} | |
| 171 | −extern void del_gendisk(struct gendisk *gp); | |
| 172 | − | |
| 173 | −void invalidate_disk(struct gendisk *disk); | |
| 174 | − | |
| 175 | −void set_disk_ro(struct gendisk *disk, bool read_only); | |
| 176 | − | |
| 177 | −static inline int get_disk_ro(struct gendisk *disk) | |
| 178 | −{ | |
| 179 | − return disk->part0->bd_read_only || | |
| 180 | − test_bit(GD_READ_ONLY, &disk->state); | |
| 181 | −} | |
| 182 | − | |
| 183 | −static inline int bdev_read_only(struct block_device *bdev) | |
| 184 | −{ | |
| 185 | − return bdev->bd_read_only || get_disk_ro(bdev->bd_disk); | |
| 186 | −} | |
| 187 | − | |
| 188 | −extern void disk_block_events(struct gendisk *disk); | |
| 189 | −extern void disk_unblock_events(struct gendisk *disk); | |
| 190 | −extern void disk_flush_events(struct gendisk *disk, unsigned int mask); | |
| 191 | −bool set_capacity_and_notify(struct gendisk *disk, sector_t size); | |
| 192 | −bool disk_force_media_change(struct gendisk *disk, unsigned int events); | |
| 193 | − | |
| 194 | −/* drivers/char/random.c */ | |
| 195 | −extern void add_disk_randomness(struct gendisk *disk) __latent_entropy; | |
| 196 | −extern void rand_initialize_disk(struct gendisk *disk); | |
| 197 | − | |
| 198 | −static inline sector_t get_start_sect(struct block_device *bdev) | |
| 199 | −{ | |
| 200 | − return bdev->bd_start_sect; | |
| 201 | −} | |
| 202 | − | |
| 203 | −static inline sector_t bdev_nr_sectors(struct block_device *bdev) | |
| 204 | −{ | |
| 205 | − return bdev->bd_nr_sectors; | |
| 206 | −} | |
| 207 | − | |
| 208 | −static inline loff_t bdev_nr_bytes(struct block_device *bdev) | |
| 209 | −{ | |
| 210 | − return (loff_t)bdev_nr_sectors(bdev) << SECTOR_SHIFT; | |
| 211 | −} | |
| 212 | − | |
| 213 | −static inline sector_t get_capacity(struct gendisk *disk) | |
| 214 | −{ | |
| 215 | − return bdev_nr_sectors(disk->part0); | |
| 216 | −} | |
| 217 | − | |
| 218 | −static inline u64 sb_bdev_nr_blocks(struct super_block *sb) | |
| 219 | −{ | |
| 220 | − return bdev_nr_sectors(sb->s_bdev) >> | |
| 221 | − (sb->s_blocksize_bits - SECTOR_SHIFT); | |
| 222 | −} | |
| 223 | − | |
| 224 | −int bdev_disk_changed(struct gendisk *disk, bool invalidate); | |
| 225 | −void blk_drop_partitions(struct gendisk *disk); | |
| 226 | − | |
| 227 | −struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id, | |
| 228 | − struct lock_class_key *lkclass); | |
| 229 | −extern void put_disk(struct gendisk *disk); | |
| 230 | −struct gendisk *__blk_alloc_disk(int node, struct lock_class_key *lkclass); | |
| 231 | − | |
| 232 | −/** | |
| 233 | − * blk_alloc_disk - allocate a gendisk structure | |
| 234 | − * @node_id: numa node to allocate on | |
| 235 | − * | |
| 236 | − * Allocate and pre-initialize a gendisk structure for use with BIO based | |
| 237 | − * drivers. | |
| 238 | − * | |
| 239 | − * Context: can sleep | |
| 240 | − */ | |
| 241 | −#define blk_alloc_disk(node_id) \ | |
| 242 | −({ \ | |
| 243 | − static struct lock_class_key __key; \ | |
| 244 | − \ | |
| 245 | − __blk_alloc_disk(node_id, &__key); \ | |
| 246 | −}) | |
| 247 | −void blk_cleanup_disk(struct gendisk *disk); | |
| 248 | − | |
| 249 | −int __register_blkdev(unsigned int major, const char *name, | |
| 250 | − void (*probe)(dev_t devt)); | |
| 251 | −#define register_blkdev(major, name) \ | |
| 252 | − __register_blkdev(major, name, NULL) | |
| 253 | −void unregister_blkdev(unsigned int major, const char *name); | |
| 254 | − | |
| 255 | −bool bdev_check_media_change(struct block_device *bdev); | |
| 256 | −int __invalidate_device(struct block_device *bdev, bool kill_dirty); | |
| 257 | −void set_capacity(struct gendisk *disk, sector_t size); | |
| 258 | − | |
| 259 | −#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED | |
| 260 | −int bd_link_disk_holder(struct block_device *bdev, struct gendisk *disk); | |
| 261 | −void bd_unlink_disk_holder(struct block_device *bdev, struct gendisk *disk); | |
| 262 | −int bd_register_pending_holders(struct gendisk *disk); | |
| 263 | −#else | |
| 264 | −static inline int bd_link_disk_holder(struct block_device *bdev, | |
| 265 | − struct gendisk *disk) | |
| 266 | −{ | |
| 267 | − return 0; | |
| 268 | −} | |
| 269 | −static inline void bd_unlink_disk_holder(struct block_device *bdev, | |
| 270 | − struct gendisk *disk) | |
| 271 | −{ | |
| 272 | −} | |
| 273 | −static inline int bd_register_pending_holders(struct gendisk *disk) | |
| 274 | −{ | |
| 275 | − return 0; | |
| 276 | −} | |
| 277 | −#endif /* CONFIG_BLOCK_HOLDER_DEPRECATED */ | |
| 278 | − | |
| 279 | −dev_t part_devt(struct gendisk *disk, u8 partno); | |
| 280 | −void inc_diskseq(struct gendisk *disk); | |
| 281 | −dev_t blk_lookup_devt(const char *name, int partno); | |
| 282 | −void blk_request_module(dev_t devt); | |
| 283 | −#ifdef CONFIG_BLOCK | |
| 284 | −void printk_all_partitions(void); | |
| 285 | −#else /* CONFIG_BLOCK */ | |
| 286 | −static inline void printk_all_partitions(void) | |
| 287 | −{ | |
| 288 | −} | |
| 289 | −#endif /* CONFIG_BLOCK */ | |
| 290 | − | |
| 291 | −#endif /* _LINUX_GENHD_H */ | |
| @@ -2,7 +2,7 @@ · 1 unmodified line | ||
| 2 | 2 | #ifndef _LINUX_PART_STAT_H |
| 3 | 3 | #define _LINUX_PART_STAT_H |
| 4 | 4 | |
| 5 | −#include <linux/genhd.h> | |
| 5 | +#include <linux/blkdev.h> | |
| 6 | 6 | #include <asm/local.h> |
| 7 | 7 | |
| 8 | 8 | struct disk_stats { |
| @@ -27,15 +27,10 @@ struct seq_file; · 26 unmodified lines | ||
| 27 | 27 | * struct sbitmap_word - Word in a &struct sbitmap. |
| 28 | 28 | */ |
| 29 | 29 | struct sbitmap_word { |
| 30 | − /** | |
| 31 | − * @depth: Number of bits being used in @word/@cleared | |
| 32 | − */ | |
| 33 | − unsigned long depth; | |
| 34 | − | |
| 35 | 30 | /** |
| 36 | 31 | * @word: word holding free bits |
| 37 | 32 | */ |
| 38 | − unsigned long word ____cacheline_aligned_in_smp; | |
| 33 | + unsigned long word; | |
| 39 | 34 | |
| 40 | 35 | /** |
| 41 | 36 | * @cleared: word holding cleared bits |
| @@ -140,7 +135,7 @@ struct sbitmap_queue { · 98 unmodified lines | ||
| 140 | 135 | |
| 141 | 136 | /** |
| 142 | 137 | * @min_shallow_depth: The minimum shallow depth which may be passed to |
| 143 | − * sbitmap_queue_get_shallow() or __sbitmap_queue_get_shallow(). | |
| 138 | + * sbitmap_queue_get_shallow() | |
| 144 | 139 | */ |
| 145 | 140 | unsigned int min_shallow_depth; |
| 146 | 141 | }; |
| @@ -164,6 +159,14 @@ struct sbitmap_queue { · 17 unmodified lines | ||
| 164 | 159 | int sbitmap_init_node(struct sbitmap *sb, unsigned int depth, int shift, |
| 165 | 160 | gfp_t flags, int node, bool round_robin, bool alloc_hint); |
| 166 | 161 | |
| 162 | +/* sbitmap internal helper */ | |
| 163 | +static inline unsigned int __map_depth(const struct sbitmap *sb, int index) | |
| 164 | +{ | |
| 165 | + if (index == sb->map_nr - 1) | |
| 166 | + return sb->depth - (index << sb->shift); | |
| 167 | + return 1U << sb->shift; | |
| 168 | +} | |
| 169 | + | |
| 167 | 170 | /** |
| 168 | 171 | * sbitmap_free() - Free memory used by a &struct sbitmap. |
| 169 | 172 | * @sb: Bitmap to free. |
| @@ -251,7 +254,7 @@ static inline void __sbitmap_for_each_set(struct sbitmap *sb, · 81 unmodified lines | ||
| 251 | 254 | while (scanned < sb->depth) { |
| 252 | 255 | unsigned long word; |
| 253 | 256 | unsigned int depth = min_t(unsigned int, |
| 254 | − sb->map[index].depth - nr, | |
| 257 | + __map_depth(sb, index) - nr, | |
| 255 | 258 | sb->depth - scanned); |
| 256 | 259 | |
| 257 | 260 | scanned += depth; |
| @@ -460,7 +463,7 @@ unsigned long __sbitmap_queue_get_batch(struct sbitmap_queue *sbq, int nr_tags, · 202 unmodified lines | ||
| 460 | 463 | unsigned int *offset); |
| 461 | 464 | |
| 462 | 465 | /** |
| 463 | − * __sbitmap_queue_get_shallow() - Try to allocate a free bit from a &struct | |
| 466 | + * sbitmap_queue_get_shallow() - Try to allocate a free bit from a &struct | |
| 464 | 467 | * sbitmap_queue, limiting the depth used from each word, with preemption |
| 465 | 468 | * already disabled. |
| 466 | 469 | * @sbq: Bitmap queue to allocate from. |
| @@ -472,8 +475,8 @@ unsigned long __sbitmap_queue_get_batch(struct sbitmap_queue *sbq, int nr_tags, · 5 unmodified lines | ||
| 472 | 475 | * |
| 473 | 476 | * Return: Non-negative allocated bit number if successful, -1 otherwise. |
| 474 | 477 | */ |
| 475 | −int __sbitmap_queue_get_shallow(struct sbitmap_queue *sbq, | |
| 476 | − unsigned int shallow_depth); | |
| 478 | +int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq, | |
| 479 | + unsigned int shallow_depth); | |
| 477 | 480 | |
| 478 | 481 | /** |
| 479 | 482 | * sbitmap_queue_get() - Try to allocate a free bit from a &struct |
| @@ -495,32 +498,6 @@ static inline int sbitmap_queue_get(struct sbitmap_queue *sbq, · 15 unmodified lines | ||
| 495 | 498 | return nr; |
| 496 | 499 | } |
| 497 | 500 | |
| 498 | −/** | |
| 499 | − * sbitmap_queue_get_shallow() - Try to allocate a free bit from a &struct | |
| 500 | − * sbitmap_queue, limiting the depth used from each word. | |
| 501 | − * @sbq: Bitmap queue to allocate from. | |
| 502 | − * @cpu: Output parameter; will contain the CPU we ran on (e.g., to be passed to | |
| 503 | − * sbitmap_queue_clear()). | |
| 504 | − * @shallow_depth: The maximum number of bits to allocate from a single word. | |
| 505 | − * See sbitmap_get_shallow(). | |
| 506 | − * | |
| 507 | − * If you call this, make sure to call sbitmap_queue_min_shallow_depth() after | |
| 508 | − * initializing @sbq. | |
| 509 | − * | |
| 510 | − * Return: Non-negative allocated bit number if successful, -1 otherwise. | |
| 511 | − */ | |
| 512 | −static inline int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq, | |
| 513 | − unsigned int *cpu, | |
| 514 | − unsigned int shallow_depth) | |
| 515 | −{ | |
| 516 | − int nr; | |
| 517 | − | |
| 518 | − *cpu = get_cpu(); | |
| 519 | − nr = __sbitmap_queue_get_shallow(sbq, shallow_depth); | |
| 520 | − put_cpu(); | |
| 521 | − return nr; | |
| 522 | −} | |
| 523 | − | |
| 524 | 501 | /** |
| 525 | 502 | * sbitmap_queue_min_shallow_depth() - Inform a &struct sbitmap_queue of the |
| 526 | 503 | * minimum shallow depth that will be used. |
| @@ -13,7 +13,6 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <scsi/scsi_request.h> |
| 14 | 14 | |
| 15 | 15 | struct Scsi_Host; |
| 16 | −struct scsi_driver; | |
| 17 | 16 | |
| 18 | 17 | /* |
| 19 | 18 | * MAX_COMMAND_SIZE is: |
| @@ -159,14 +158,6 @@ static inline void *scsi_cmd_priv(struct scsi_cmnd *cmd) · 139 unmodified lines | ||
| 159 | 158 | return cmd + 1; |
| 160 | 159 | } |
| 161 | 160 | |
| 162 | −/* make sure not to use it with passthrough commands */ | |
| 163 | −static inline struct scsi_driver *scsi_cmd_to_driver(struct scsi_cmnd *cmd) | |
| 164 | −{ | |
| 165 | − struct request *rq = scsi_cmd_to_rq(cmd); | |
| 166 | − | |
| 167 | − return *(struct scsi_driver **)rq->q->disk->private_data; | |
| 168 | −} | |
| 169 | − | |
| 170 | 161 | void scsi_done(struct scsi_cmnd *cmd); |
| 171 | 162 | |
| 172 | 163 | extern void scsi_finish_command(struct scsi_cmnd *cmd); |
| @@ -4,11 +4,10 @@ · 3 unmodified lines | ||
| 4 | 4 | |
| 5 | 5 | #include <linux/blk_types.h> |
| 6 | 6 | #include <linux/device.h> |
| 7 | +#include <scsi/scsi_cmnd.h> | |
| 7 | 8 | |
| 8 | 9 | struct module; |
| 9 | 10 | struct request; |
| 10 | −struct scsi_cmnd; | |
| 11 | −struct scsi_device; | |
| 12 | 11 | |
| 13 | 12 | struct scsi_driver { |
| 14 | 13 | struct device_driver gendrv; |
| @@ -31,4 +30,10 @@ extern int scsi_register_interface(struct class_interface *); · 16 unmodified lines | ||
| 31 | 30 | #define scsi_unregister_interface(intf) \ |
| 32 | 31 | class_interface_unregister(intf) |
| 33 | 32 | |
| 33 | +/* make sure not to use it with passthrough commands */ | |
| 34 | +static inline struct scsi_driver *scsi_cmd_to_driver(struct scsi_cmnd *cmd) | |
| 35 | +{ | |
| 36 | + return to_scsi_driver(cmd->device->sdev_gendev.driver); | |
| 37 | +} | |
| 38 | + | |
| 34 | 39 | #endif /* _SCSI_SCSI_DRIVER_H */ |
| @@ -100,19 +100,7 @@ TRACE_EVENT(block_rq_requeue, · 99 unmodified lines | ||
| 100 | 100 | __entry->nr_sector, 0) |
| 101 | 101 | ); |
| 102 | 102 | |
| 103 | −/** | |
| 104 | − * block_rq_complete - block IO operation completed by device driver | |
| 105 | − * @rq: block operations request | |
| 106 | − * @error: status code | |
| 107 | − * @nr_bytes: number of completed bytes | |
| 108 | − * | |
| 109 | − * The block_rq_complete tracepoint event indicates that some portion | |
| 110 | − * of operation request has been completed by the device driver. If | |
| 111 | − * the @rq->bio is %NULL, then there is absolutely no additional work to | |
| 112 | − * do for the request. If @rq->bio is non-NULL then there is | |
| 113 | − * additional work required to complete the request. | |
| 114 | − */ | |
| 115 | −TRACE_EVENT(block_rq_complete, | |
| 103 | +DECLARE_EVENT_CLASS(block_rq_completion, | |
| 116 | 104 | |
| 117 | 105 | TP_PROTO(struct request *rq, blk_status_t error, unsigned int nr_bytes), |
| 118 | 106 | |
| @@ -144,6 +132,41 @@ TRACE_EVENT(block_rq_complete, · 25 unmodified lines | ||
| 144 | 132 | __entry->nr_sector, __entry->error) |
| 145 | 133 | ); |
| 146 | 134 | |
| 135 | +/** | |
| 136 | + * block_rq_complete - block IO operation completed by device driver | |
| 137 | + * @rq: block operations request | |
| 138 | + * @error: status code | |
| 139 | + * @nr_bytes: number of completed bytes | |
| 140 | + * | |
| 141 | + * The block_rq_complete tracepoint event indicates that some portion | |
| 142 | + * of operation request has been completed by the device driver. If | |
| 143 | + * the @rq->bio is %NULL, then there is absolutely no additional work to | |
| 144 | + * do for the request. If @rq->bio is non-NULL then there is | |
| 145 | + * additional work required to complete the request. | |
| 146 | + */ | |
| 147 | +DEFINE_EVENT(block_rq_completion, block_rq_complete, | |
| 148 | + | |
| 149 | + TP_PROTO(struct request *rq, blk_status_t error, unsigned int nr_bytes), | |
| 150 | + | |
| 151 | + TP_ARGS(rq, error, nr_bytes) | |
| 152 | +); | |
| 153 | + | |
| 154 | +/** | |
| 155 | + * block_rq_error - block IO operation error reported by device driver | |
| 156 | + * @rq: block operations request | |
| 157 | + * @error: status code | |
| 158 | + * @nr_bytes: number of completed bytes | |
| 159 | + * | |
| 160 | + * The block_rq_error tracepoint event indicates that some portion | |
| 161 | + * of operation request has failed as reported by the device driver. | |
| 162 | + */ | |
| 163 | +DEFINE_EVENT(block_rq_completion, block_rq_error, | |
| 164 | + | |
| 165 | + TP_PROTO(struct request *rq, blk_status_t error, unsigned int nr_bytes), | |
| 166 | + | |
| 167 | + TP_ARGS(rq, error, nr_bytes) | |
| 168 | +); | |
| 169 | + | |
| 147 | 170 | DECLARE_EVENT_CLASS(block_rq, |
| 148 | 171 | |
| 149 | 172 | TP_PROTO(struct request *rq), |
| @@ -8,7 +8,6 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/root_dev.h> |
| 9 | 9 | #include <linux/security.h> |
| 10 | 10 | #include <linux/delay.h> |
| 11 | −#include <linux/genhd.h> | |
| 12 | 11 | #include <linux/mount.h> |
| 13 | 12 | #include <linux/device.h> |
| 14 | 13 | #include <linux/init.h> |
| @@ -735,7 +735,7 @@ void __noreturn do_exit(long code) · 734 unmodified lines | ||
| 735 | 735 | struct task_struct *tsk = current; |
| 736 | 736 | int group_dead; |
| 737 | 737 | |
| 738 | − WARN_ON(blk_needs_flush_plug(tsk)); | |
| 738 | + WARN_ON(tsk->plug); | |
| 739 | 739 | |
| 740 | 740 | /* |
| 741 | 741 | * If do_dead is called because this processes oopsed, it's possible |
| @@ -28,7 +28,6 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <linux/gfp.h> |
| 29 | 29 | #include <linux/syscore_ops.h> |
| 30 | 30 | #include <linux/ctype.h> |
| 31 | −#include <linux/genhd.h> | |
| 32 | 31 | #include <linux/ktime.h> |
| 33 | 32 | #include <linux/security.h> |
| 34 | 33 | #include <linux/secretmem.h> |
| @@ -16,7 +16,6 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <linux/file.h> |
| 17 | 17 | #include <linux/delay.h> |
| 18 | 18 | #include <linux/bitops.h> |
| 19 | −#include <linux/genhd.h> | |
| 20 | 19 | #include <linux/device.h> |
| 21 | 20 | #include <linux/bio.h> |
| 22 | 21 | #include <linux/blkdev.h> |
| @@ -277,10 +276,9 @@ static int hib_submit_io(int op, int op_flags, pgoff_t page_off, void *addr, · 254 unmodified lines | ||
| 277 | 276 | struct bio *bio; |
| 278 | 277 | int error = 0; |
| 279 | 278 | |
| 280 | − bio = bio_alloc(GFP_NOIO | __GFP_HIGH, 1); | |
| 279 | + bio = bio_alloc(hib_resume_bdev, 1, op | op_flags, | |
| 280 | + GFP_NOIO | __GFP_HIGH); | |
| 281 | 281 | bio->bi_iter.bi_sector = page_off * (PAGE_SIZE >> 9); |
| 282 | − bio_set_dev(bio, hib_resume_bdev); | |
| 283 | − bio_set_op_attrs(bio, op, op_flags); | |
| 284 | 282 | |
| 285 | 283 | if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) { |
| 286 | 284 | pr_err("Adding page to bio failed at %llu\n", |
| @@ -6353,8 +6353,7 @@ static inline void sched_submit_work(struct task_struct *tsk) · 6352 unmodified lines | ||
| 6353 | 6353 | * If we are going to sleep and we have plugged IO queued, |
| 6354 | 6354 | * make sure to submit it to avoid deadlocks. |
| 6355 | 6355 | */ |
| 6356 | − if (blk_needs_flush_plug(tsk)) | |
| 6357 | − blk_flush_plug(tsk->plug, true); | |
| 6356 | + blk_flush_plug(tsk->plug, true); | |
| 6358 | 6357 | } |
| 6359 | 6358 | |
| 6360 | 6359 | static void sched_update_worker(struct task_struct *tsk) |
| @@ -8380,9 +8379,7 @@ int io_schedule_prepare(void) · 2019 unmodified lines | ||
| 8380 | 8379 | int old_iowait = current->in_iowait; |
| 8381 | 8380 | |
| 8382 | 8381 | current->in_iowait = 1; |
| 8383 | − if (current->plug) | |
| 8384 | − blk_flush_plug(current->plug, true); | |
| 8385 | − | |
| 8382 | + blk_flush_plug(current->plug, true); | |
| 8386 | 8383 | return old_iowait; |
| 8387 | 8384 | } |
| 8388 | 8385 | |
| @@ -85,7 +85,6 @@ int sbitmap_init_node(struct sbitmap *sb, unsigned int depth, int shift, · 84 unmodified lines | ||
| 85 | 85 | bool alloc_hint) |
| 86 | 86 | { |
| 87 | 87 | unsigned int bits_per_word; |
| 88 | − unsigned int i; | |
| 89 | 88 | |
| 90 | 89 | if (shift < 0) |
| 91 | 90 | shift = sbitmap_calculate_shift(depth); |
| @@ -117,10 +116,6 @@ int sbitmap_init_node(struct sbitmap *sb, unsigned int depth, int shift, · 25 unmodified lines | ||
| 117 | 116 | return -ENOMEM; |
| 118 | 117 | } |
| 119 | 118 | |
| 120 | − for (i = 0; i < sb->map_nr; i++) { | |
| 121 | − sb->map[i].depth = min(depth, bits_per_word); | |
| 122 | − depth -= sb->map[i].depth; | |
| 123 | − } | |
| 124 | 119 | return 0; |
| 125 | 120 | } |
| 126 | 121 | EXPORT_SYMBOL_GPL(sbitmap_init_node); |
| @@ -135,11 +130,6 @@ void sbitmap_resize(struct sbitmap *sb, unsigned int depth) · 8 unmodified lines | ||
| 135 | 130 | |
| 136 | 131 | sb->depth = depth; |
| 137 | 132 | sb->map_nr = DIV_ROUND_UP(sb->depth, bits_per_word); |
| 138 | − | |
| 139 | − for (i = 0; i < sb->map_nr; i++) { | |
| 140 | − sb->map[i].depth = min(depth, bits_per_word); | |
| 141 | − depth -= sb->map[i].depth; | |
| 142 | − } | |
| 143 | 133 | } |
| 144 | 134 | EXPORT_SYMBOL_GPL(sbitmap_resize); |
| 145 | 135 | |
| @@ -184,8 +174,8 @@ static int sbitmap_find_bit_in_index(struct sbitmap *sb, int index, · 38 unmodified lines | ||
| 184 | 174 | int nr; |
| 185 | 175 | |
| 186 | 176 | do { |
| 187 | − nr = __sbitmap_get_word(&map->word, map->depth, alloc_hint, | |
| 188 | − !sb->round_robin); | |
| 177 | + nr = __sbitmap_get_word(&map->word, __map_depth(sb, index), | |
| 178 | + alloc_hint, !sb->round_robin); | |
| 189 | 179 | if (nr != -1) |
| 190 | 180 | break; |
| 191 | 181 | if (!sbitmap_deferred_clear(map)) |
| @@ -257,7 +247,9 @@ static int __sbitmap_get_shallow(struct sbitmap *sb, · 65 unmodified lines | ||
| 257 | 247 | for (i = 0; i < sb->map_nr; i++) { |
| 258 | 248 | again: |
| 259 | 249 | nr = __sbitmap_get_word(&sb->map[index].word, |
| 260 | − min(sb->map[index].depth, shallow_depth), | |
| 250 | + min_t(unsigned int, | |
| 251 | + __map_depth(sb, index), | |
| 252 | + shallow_depth), | |
| 261 | 253 | SB_NR_TO_BIT(sb, alloc_hint), true); |
| 262 | 254 | if (nr != -1) { |
| 263 | 255 | nr += index << sb->shift; |
| @@ -315,11 +307,12 @@ static unsigned int __sbitmap_weight(const struct sbitmap *sb, bool set) · 51 unmodified lines | ||
| 315 | 307 | |
| 316 | 308 | for (i = 0; i < sb->map_nr; i++) { |
| 317 | 309 | const struct sbitmap_word *word = &sb->map[i]; |
| 310 | + unsigned int word_depth = __map_depth(sb, i); | |
| 318 | 311 | |
| 319 | 312 | if (set) |
| 320 | − weight += bitmap_weight(&word->word, word->depth); | |
| 313 | + weight += bitmap_weight(&word->word, word_depth); | |
| 321 | 314 | else |
| 322 | − weight += bitmap_weight(&word->cleared, word->depth); | |
| 315 | + weight += bitmap_weight(&word->cleared, word_depth); | |
| 323 | 316 | } |
| 324 | 317 | return weight; |
| 325 | 318 | } |
| @@ -367,7 +360,7 @@ void sbitmap_bitmap_show(struct sbitmap *sb, struct seq_file *m) · 41 unmodified lines | ||
| 367 | 360 | for (i = 0; i < sb->map_nr; i++) { |
| 368 | 361 | unsigned long word = READ_ONCE(sb->map[i].word); |
| 369 | 362 | unsigned long cleared = READ_ONCE(sb->map[i].cleared); |
| 370 | − unsigned int word_bits = READ_ONCE(sb->map[i].depth); | |
| 363 | + unsigned int word_bits = __map_depth(sb, i); | |
| 371 | 364 | |
| 372 | 365 | word &= ~cleared; |
| 373 | 366 | |
| @@ -531,15 +524,16 @@ unsigned long __sbitmap_queue_get_batch(struct sbitmap_queue *sbq, int nr_tags, · 157 unmodified lines | ||
| 531 | 524 | for (i = 0; i < sb->map_nr; i++) { |
| 532 | 525 | struct sbitmap_word *map = &sb->map[index]; |
| 533 | 526 | unsigned long get_mask; |
| 527 | + unsigned int map_depth = __map_depth(sb, index); | |
| 534 | 528 | |
| 535 | 529 | sbitmap_deferred_clear(map); |
| 536 | − if (map->word == (1UL << (map->depth - 1)) - 1) | |
| 530 | + if (map->word == (1UL << (map_depth - 1)) - 1) | |
| 537 | 531 | continue; |
| 538 | 532 | |
| 539 | − nr = find_first_zero_bit(&map->word, map->depth); | |
| 540 | − if (nr + nr_tags <= map->depth) { | |
| 533 | + nr = find_first_zero_bit(&map->word, map_depth); | |
| 534 | + if (nr + nr_tags <= map_depth) { | |
| 541 | 535 | atomic_long_t *ptr = (atomic_long_t *) &map->word; |
| 542 | − int map_tags = min_t(int, nr_tags, map->depth); | |
| 536 | + int map_tags = min_t(int, nr_tags, map_depth); | |
| 543 | 537 | unsigned long val, ret; |
| 544 | 538 | |
| 545 | 539 | get_mask = ((1UL << map_tags) - 1) << nr; |
| @@ -563,14 +557,14 @@ unsigned long __sbitmap_queue_get_batch(struct sbitmap_queue *sbq, int nr_tags, · 17 unmodified lines | ||
| 563 | 557 | return 0; |
| 564 | 558 | } |
| 565 | 559 | |
| 566 | −int __sbitmap_queue_get_shallow(struct sbitmap_queue *sbq, | |
| 567 | − unsigned int shallow_depth) | |
| 560 | +int sbitmap_queue_get_shallow(struct sbitmap_queue *sbq, | |
| 561 | + unsigned int shallow_depth) | |
| 568 | 562 | { |
| 569 | 563 | WARN_ON_ONCE(shallow_depth < sbq->min_shallow_depth); |
| 570 | 564 | |
| 571 | 565 | return sbitmap_get_shallow(&sbq->sb, shallow_depth); |
| 572 | 566 | } |
| 573 | −EXPORT_SYMBOL_GPL(__sbitmap_queue_get_shallow); | |
| 567 | +EXPORT_SYMBOL_GPL(sbitmap_queue_get_shallow); | |
| 574 | 568 | |
| 575 | 569 | void sbitmap_queue_min_shallow_depth(struct sbitmap_queue *sbq, |
| 576 | 570 | unsigned int min_shallow_depth) |
| @@ -338,10 +338,10 @@ int __swap_writepage(struct page *page, struct writeback_control *wbc, · 337 unmodified lines | ||
| 338 | 338 | return 0; |
| 339 | 339 | } |
| 340 | 340 | |
| 341 | − bio = bio_alloc(GFP_NOIO, 1); | |
| 342 | − bio_set_dev(bio, sis->bdev); | |
| 341 | + bio = bio_alloc(sis->bdev, 1, | |
| 342 | + REQ_OP_WRITE | REQ_SWAP | wbc_to_write_flags(wbc), | |
| 343 | + GFP_NOIO); | |
| 343 | 344 | bio->bi_iter.bi_sector = swap_page_sector(page); |
| 344 | − bio->bi_opf = REQ_OP_WRITE | REQ_SWAP | wbc_to_write_flags(wbc); | |
| 345 | 345 | bio->bi_end_io = end_write_func; |
| 346 | 346 | bio_add_page(bio, page, thp_size(page), 0); |
| 347 | 347 | |
| @@ -403,9 +403,7 @@ int swap_readpage(struct page *page, bool synchronous) · 55 unmodified lines | ||
| 403 | 403 | } |
| 404 | 404 | |
| 405 | 405 | ret = 0; |
| 406 | − bio = bio_alloc(GFP_KERNEL, 1); | |
| 407 | − bio_set_dev(bio, sis->bdev); | |
| 408 | − bio->bi_opf = REQ_OP_READ; | |
| 406 | + bio = bio_alloc(sis->bdev, 1, REQ_OP_READ, GFP_KERNEL); | |
| 409 | 407 | bio->bi_iter.bi_sector = swap_page_sector(page); |
| 410 | 408 | bio->bi_end_io = end_swap_bio_read; |
| 411 | 409 | bio_add_page(bio, page, thp_size(page), 0); |
| @@ -16,7 +16,6 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <linux/parser.h> |
| 17 | 17 | #include <linux/slab.h> |
| 18 | 18 | #include <linux/rculist.h> |
| 19 | −#include <linux/genhd.h> | |
| 20 | 19 | #include <linux/seq_file.h> |
| 21 | 20 | #include <linux/ima.h> |
| 22 | 21 | |