Pull core block layer changes from Jens Axboe: "This is the core block IO pull request for 3.18. Apart from the new and improved flush machinery for blk-mq, this is all mostly bug fixes and cleanups. - blk-mq timeout updates and fixes from Christoph. - Removal of REQ_END, also from Christoph. We pass it through the ->queue_rq() hook for blk-mq instead, freeing up one of the request bits. The space was overly tight on 32-bit, so Martin also killed REQ_KERNEL since it's no longer used. - blk integrity updates and fixes from Martin and Gu Zheng. - Update to the flush machinery for blk-mq from Ming Lei. Now we have a per hardware context flush request, which both cleans up the code should scale better for flush intensive workloads on blk-mq. - Improve the error printing, from Rob Elliott. - Backing device improvements and cleanups from Tejun. - Fixup of a misplaced rq_complete() tracepoint from Hannes. - Make blk_get_request() return error pointers, fixing up issues where we NULL deref when a device goes bad or missing. From Joe Lawrence. - Prep work for drastically reducing the memory consumption of dm devices from Junichi Nomura. This allows creating clone bio sets without preallocating a lot of memory. - Fix a blk-mq hang on certain combinations of queue depths and hardware queues from me. - Limit memory consumption for blk-mq devices for crash dump scenarios and drivers that use crazy high depths (certain SCSI shared tag setups). We now just use a single queue and limited depth for that" * 'for-3.18/core' of git://git.kernel.dk/linux-block: (58 commits) block: Remove REQ_KERNEL blk-mq: allocate cpumask on the home node bio-integrity: remove the needless fail handle of bip_slab creating block: include func name in __get_request prints block: make blk_update_request print prefix match ratelimited prefix blk-merge: don't compute bi_phys_segments from bi_vcnt for cloned bio block: fix alignment_offset math that assumes io_min is a power-of-2 blk-mq: Make bt_clear_tag() easier to read blk-mq: fix potential hang if rolling wakeup depth is too high block: add bioset_create_nobvec() block: use bio_clone_fast() in blk_rq_prep_clone() block: misplaced rq_complete tracepoint sd: Honor block layer integrity handling flags block: Replace strnicmp with strncasecmp block: Add T10 Protection Information functions block: Don't merge requests if integrity flags differ block: Integrity checksum flag block: Relocate bio integrity flags block: Add a disk flag to block integrity profile block: Add prefix to block integrity profile flags ...
Unified · Split
| @@ -53,6 +53,14 @@ Description: · 52 unmodified lines | ||
| 53 | 53 | 512 bytes of data. |
| 54 | 54 | |
| 55 | 55 | |
| 56 | +What: /sys/block/<disk>/integrity/device_is_integrity_capable | |
| 57 | +Date: July 2014 | |
| 58 | +Contact: Martin K. Petersen <martin.petersen@oracle.com> | |
| 59 | +Description: | |
| 60 | + Indicates whether a storage device is capable of storing | |
| 61 | + integrity metadata. Set if the device is T10 PI-capable. | |
| 62 | + | |
| 63 | + | |
| 56 | 64 | What: /sys/block/<disk>/integrity/write_generate |
| 57 | 65 | Date: June 2008 |
| 58 | 66 | Contact: Martin K. Petersen <martin.petersen@oracle.com> |
| @@ -129,11 +129,11 @@ interface for this is being worked on. · 128 unmodified lines | ||
| 129 | 129 | 4.1 BIO |
| 130 | 130 | |
| 131 | 131 | The data integrity patches add a new field to struct bio when |
| 132 | −CONFIG_BLK_DEV_INTEGRITY is enabled. bio->bi_integrity is a pointer | |
| 133 | −to a struct bip which contains the bio integrity payload. Essentially | |
| 134 | −a bip is a trimmed down struct bio which holds a bio_vec containing | |
| 135 | −the integrity metadata and the required housekeeping information (bvec | |
| 136 | −pool, vector count, etc.) | |
| 132 | +CONFIG_BLK_DEV_INTEGRITY is enabled. bio_integrity(bio) returns a | |
| 133 | +pointer to a struct bip which contains the bio integrity payload. | |
| 134 | +Essentially a bip is a trimmed down struct bio which holds a bio_vec | |
| 135 | +containing the integrity metadata and the required housekeeping | |
| 136 | +information (bvec pool, vector count, etc.) | |
| 137 | 137 | |
| 138 | 138 | A kernel subsystem can enable data integrity protection on a bio by |
| 139 | 139 | calling bio_integrity_alloc(bio). This will allocate and attach the |
| @@ -192,16 +192,6 @@ will require extra work due to the application tag. · 52 unmodified lines | ||
| 192 | 192 | supported by the block device. |
| 193 | 193 | |
| 194 | 194 | |
| 195 | − int bdev_integrity_enabled(block_device, int rw); | |
| 196 | − | |
| 197 | − bdev_integrity_enabled() will return 1 if the block device | |
| 198 | − supports integrity metadata transfer for the data direction | |
| 199 | − specified in 'rw'. | |
| 200 | − | |
| 201 | − bdev_integrity_enabled() honors the write_generate and | |
| 202 | − read_verify flags in sysfs and will respond accordingly. | |
| 203 | − | |
| 204 | − | |
| 205 | 195 | int bio_integrity_prep(bio); |
| 206 | 196 | |
| 207 | 197 | To generate IMD for WRITE and to set up buffers for READ, the |
| @@ -216,36 +206,6 @@ will require extra work due to the application tag. · 8 unmodified lines | ||
| 216 | 206 | bio_integrity_enabled() returned 1. |
| 217 | 207 | |
| 218 | 208 | |
| 219 | − int bio_integrity_tag_size(bio); | |
| 220 | − | |
| 221 | − If the filesystem wants to use the application tag space it will | |
| 222 | − first have to find out how much storage space is available. | |
| 223 | − Because tag space is generally limited (usually 2 bytes per | |
| 224 | − sector regardless of sector size), the integrity framework | |
| 225 | − supports interleaving the information between the sectors in an | |
| 226 | − I/O. | |
| 227 | − | |
| 228 | − Filesystems can call bio_integrity_tag_size(bio) to find out how | |
| 229 | − many bytes of storage are available for that particular bio. | |
| 230 | − | |
| 231 | − Another option is bdev_get_tag_size(block_device) which will | |
| 232 | − return the number of available bytes per hardware sector. | |
| 233 | − | |
| 234 | − | |
| 235 | − int bio_integrity_set_tag(bio, void *tag_buf, len); | |
| 236 | − | |
| 237 | − After a successful return from bio_integrity_prep(), | |
| 238 | − bio_integrity_set_tag() can be used to attach an opaque tag | |
| 239 | − buffer to a bio. Obviously this only makes sense if the I/O is | |
| 240 | − a WRITE. | |
| 241 | − | |
| 242 | − | |
| 243 | − int bio_integrity_get_tag(bio, void *tag_buf, len); | |
| 244 | − | |
| 245 | − Similarly, at READ I/O completion time the filesystem can | |
| 246 | − retrieve the tag buffer using bio_integrity_get_tag(). | |
| 247 | − | |
| 248 | − | |
| 249 | 209 | 5.3 PASSING EXISTING INTEGRITY METADATA |
| 250 | 210 | |
| 251 | 211 | Filesystems that either generate their own integrity metadata or |
| @@ -298,8 +258,6 @@ will require extra work due to the application tag. · 46 unmodified lines | ||
| 298 | 258 | .name = "STANDARDSBODY-TYPE-VARIANT-CSUM", |
| 299 | 259 | .generate_fn = my_generate_fn, |
| 300 | 260 | .verify_fn = my_verify_fn, |
| 301 | − .get_tag_fn = my_get_tag_fn, | |
| 302 | − .set_tag_fn = my_set_tag_fn, | |
| 303 | 261 | .tuple_size = sizeof(struct my_tuple_size), |
| 304 | 262 | .tag_size = <tag bytes per hw sector>, |
| 305 | 263 | }; |
| @@ -321,7 +279,5 @@ will require extra work due to the application tag. · 15 unmodified lines | ||
| 321 | 279 | are available per hardware sector. For DIF this is either 2 or |
| 322 | 280 | 0 depending on the value of the Control Mode Page ATO bit. |
| 323 | 281 | |
| 324 | − See 6.2 for a description of get_tag_fn and set_tag_fn. | |
| 325 | − | |
| 326 | 282 | ---------------------------------------------------------------------- |
| 327 | 283 | 2007-12-24 Martin K. Petersen <martin.petersen@oracle.com> |
| @@ -77,6 +77,7 @@ config BLK_DEV_BSGLIB · 76 unmodified lines | ||
| 77 | 77 | |
| 78 | 78 | config BLK_DEV_INTEGRITY |
| 79 | 79 | bool "Block layer data integrity support" |
| 80 | + select CRC_T10DIF if BLK_DEV_INTEGRITY | |
| 80 | 81 | ---help--- |
| 81 | 82 | Some storage devices allow extra information to be |
| 82 | 83 | stored/retrieved to help protect the data. The block layer |
| @@ -20,6 +20,6 @@ obj-$(CONFIG_IOSCHED_DEADLINE) += deadline-iosched.o · 19 unmodified lines | ||
| 20 | 20 | obj-$(CONFIG_IOSCHED_CFQ) += cfq-iosched.o |
| 21 | 21 | |
| 22 | 22 | obj-$(CONFIG_BLOCK_COMPAT) += compat_ioctl.o |
| 23 | −obj-$(CONFIG_BLK_DEV_INTEGRITY) += blk-integrity.o | |
| 24 | 23 | obj-$(CONFIG_BLK_CMDLINE_PARSER) += cmdline-parser.o |
| 25 | −obj-$(CONFIG_BLK_DEV_INTEGRITY) += bio-integrity.o | |
| 24 | +obj-$(CONFIG_BLK_DEV_INTEGRITY) += bio-integrity.o blk-integrity.o t10-pi.o | |
| 25 | + | |
| @@ -79,6 +79,7 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, · 78 unmodified lines | ||
| 79 | 79 | bip->bip_slab = idx; |
| 80 | 80 | bip->bip_bio = bio; |
| 81 | 81 | bio->bi_integrity = bip; |
| 82 | + bio->bi_rw |= REQ_INTEGRITY; | |
| 82 | 83 | |
| 83 | 84 | return bip; |
| 84 | 85 | err: |
| @@ -96,11 +97,12 @@ EXPORT_SYMBOL(bio_integrity_alloc); · 11 unmodified lines | ||
| 96 | 97 | */ |
| 97 | 98 | void bio_integrity_free(struct bio *bio) |
| 98 | 99 | { |
| 99 | − struct bio_integrity_payload *bip = bio->bi_integrity; | |
| 100 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 100 | 101 | struct bio_set *bs = bio->bi_pool; |
| 101 | 102 | |
| 102 | − if (bip->bip_owns_buf) | |
| 103 | − kfree(bip->bip_buf); | |
| 103 | + if (bip->bip_flags & BIP_BLOCK_INTEGRITY) | |
| 104 | + kfree(page_address(bip->bip_vec->bv_page) + | |
| 105 | + bip->bip_vec->bv_offset); | |
| 104 | 106 | |
| 105 | 107 | if (bs) { |
| 106 | 108 | if (bip->bip_slab != BIO_POOL_NONE) |
| @@ -128,7 +130,7 @@ EXPORT_SYMBOL(bio_integrity_free); · 21 unmodified lines | ||
| 128 | 130 | int bio_integrity_add_page(struct bio *bio, struct page *page, |
| 129 | 131 | unsigned int len, unsigned int offset) |
| 130 | 132 | { |
| 131 | − struct bio_integrity_payload *bip = bio->bi_integrity; | |
| 133 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 132 | 134 | struct bio_vec *iv; |
| 133 | 135 | |
| 134 | 136 | if (bip->bip_vcnt >= bip->bip_max_vcnt) { |
| @@ -147,24 +149,6 @@ int bio_integrity_add_page(struct bio *bio, struct page *page, · 12 unmodified lines | ||
| 147 | 149 | } |
| 148 | 150 | EXPORT_SYMBOL(bio_integrity_add_page); |
| 149 | 151 | |
| 150 | −static int bdev_integrity_enabled(struct block_device *bdev, int rw) | |
| 151 | −{ | |
| 152 | − struct blk_integrity *bi = bdev_get_integrity(bdev); | |
| 153 | − | |
| 154 | − if (bi == NULL) | |
| 155 | − return 0; | |
| 156 | − | |
| 157 | − if (rw == READ && bi->verify_fn != NULL && | |
| 158 | − (bi->flags & INTEGRITY_FLAG_READ)) | |
| 159 | − return 1; | |
| 160 | − | |
| 161 | − if (rw == WRITE && bi->generate_fn != NULL && | |
| 162 | − (bi->flags & INTEGRITY_FLAG_WRITE)) | |
| 163 | − return 1; | |
| 164 | − | |
| 165 | − return 0; | |
| 166 | −} | |
| 167 | − | |
| 168 | 152 | /** |
| 169 | 153 | * bio_integrity_enabled - Check whether integrity can be passed |
| 170 | 154 | * @bio: bio to check |
| @@ -174,199 +158,92 @@ static int bdev_integrity_enabled(struct block_device *bdev, int rw) · 3 unmodified lines | ||
| 174 | 158 | * set prior to calling. The functions honors the write_generate and |
| 175 | 159 | * read_verify flags in sysfs. |
| 176 | 160 | */ |
| 177 | −int bio_integrity_enabled(struct bio *bio) | |
| 161 | +bool bio_integrity_enabled(struct bio *bio) | |
| 178 | 162 | { |
| 163 | + struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev); | |
| 164 | + | |
| 179 | 165 | if (!bio_is_rw(bio)) |
| 180 | − return 0; | |
| 166 | + return false; | |
| 181 | 167 | |
| 182 | 168 | /* Already protected? */ |
| 183 | 169 | if (bio_integrity(bio)) |
| 184 | − return 0; | |
| 170 | + return false; | |
| 171 | + | |
| 172 | + if (bi == NULL) | |
| 173 | + return false; | |
| 174 | + | |
| 175 | + if (bio_data_dir(bio) == READ && bi->verify_fn != NULL && | |
| 176 | + (bi->flags & BLK_INTEGRITY_VERIFY)) | |
| 177 | + return true; | |
| 178 | + | |
| 179 | + if (bio_data_dir(bio) == WRITE && bi->generate_fn != NULL && | |
| 180 | + (bi->flags & BLK_INTEGRITY_GENERATE)) | |
| 181 | + return true; | |
| 185 | 182 | |
| 186 | − return bdev_integrity_enabled(bio->bi_bdev, bio_data_dir(bio)); | |
| 183 | + return false; | |
| 187 | 184 | } |
| 188 | 185 | EXPORT_SYMBOL(bio_integrity_enabled); |
| 189 | 186 | |
| 190 | 187 | /** |
| 191 | − * bio_integrity_hw_sectors - Convert 512b sectors to hardware ditto | |
| 188 | + * bio_integrity_intervals - Return number of integrity intervals for a bio | |
| 192 | 189 | * @bi: blk_integrity profile for device |
| 193 | − * @sectors: Number of 512 sectors to convert | |
| 190 | + * @sectors: Size of the bio in 512-byte sectors | |
| 194 | 191 | * |
| 195 | 192 | * Description: The block layer calculates everything in 512 byte |
| 196 | − * sectors but integrity metadata is done in terms of the hardware | |
| 197 | − * sector size of the storage device. Convert the block layer sectors | |
| 198 | − * to physical sectors. | |
| 193 | + * sectors but integrity metadata is done in terms of the data integrity | |
| 194 | + * interval size of the storage device. Convert the block layer sectors | |
| 195 | + * to the appropriate number of integrity intervals. | |
| 199 | 196 | */ |
| 200 | −static inline unsigned int bio_integrity_hw_sectors(struct blk_integrity *bi, | |
| 201 | − unsigned int sectors) | |
| 197 | +static inline unsigned int bio_integrity_intervals(struct blk_integrity *bi, | |
| 198 | + unsigned int sectors) | |
| 202 | 199 | { |
| 203 | − /* At this point there are only 512b or 4096b DIF/EPP devices */ | |
| 204 | − if (bi->sector_size == 4096) | |
| 205 | − return sectors >>= 3; | |
| 206 | − | |
| 207 | − return sectors; | |
| 200 | + return sectors >> (ilog2(bi->interval) - 9); | |
| 208 | 201 | } |
| 209 | 202 | |
| 210 | 203 | static inline unsigned int bio_integrity_bytes(struct blk_integrity *bi, |
| 211 | 204 | unsigned int sectors) |
| 212 | 205 | { |
| 213 | − return bio_integrity_hw_sectors(bi, sectors) * bi->tuple_size; | |
| 206 | + return bio_integrity_intervals(bi, sectors) * bi->tuple_size; | |
| 214 | 207 | } |
| 215 | 208 | |
| 216 | 209 | /** |
| 217 | − * bio_integrity_tag_size - Retrieve integrity tag space | |
| 218 | − * @bio: bio to inspect | |
| 219 | − * | |
| 220 | − * Description: Returns the maximum number of tag bytes that can be | |
| 221 | − * attached to this bio. Filesystems can use this to determine how | |
| 222 | − * much metadata to attach to an I/O. | |
| 223 | − */ | |
| 224 | −unsigned int bio_integrity_tag_size(struct bio *bio) | |
| 225 | −{ | |
| 226 | − struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev); | |
| 227 | − | |
| 228 | − BUG_ON(bio->bi_iter.bi_size == 0); | |
| 229 | − | |
| 230 | − return bi->tag_size * (bio->bi_iter.bi_size / bi->sector_size); | |
| 231 | −} | |
| 232 | −EXPORT_SYMBOL(bio_integrity_tag_size); | |
| 233 | − | |
| 234 | −static int bio_integrity_tag(struct bio *bio, void *tag_buf, unsigned int len, | |
| 235 | − int set) | |
| 236 | −{ | |
| 237 | − struct bio_integrity_payload *bip = bio->bi_integrity; | |
| 238 | − struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev); | |
| 239 | − unsigned int nr_sectors; | |
| 240 | − | |
| 241 | − BUG_ON(bip->bip_buf == NULL); | |
| 242 | − | |
| 243 | − if (bi->tag_size == 0) | |
| 244 | − return -1; | |
| 245 | − | |
| 246 | − nr_sectors = bio_integrity_hw_sectors(bi, | |
| 247 | − DIV_ROUND_UP(len, bi->tag_size)); | |
| 248 | − | |
| 249 | − if (nr_sectors * bi->tuple_size > bip->bip_iter.bi_size) { | |
| 250 | − printk(KERN_ERR "%s: tag too big for bio: %u > %u\n", __func__, | |
| 251 | − nr_sectors * bi->tuple_size, bip->bip_iter.bi_size); | |
| 252 | − return -1; | |
| 253 | − } | |
| 254 | − | |
| 255 | − if (set) | |
| 256 | − bi->set_tag_fn(bip->bip_buf, tag_buf, nr_sectors); | |
| 257 | − else | |
| 258 | − bi->get_tag_fn(bip->bip_buf, tag_buf, nr_sectors); | |
| 259 | − | |
| 260 | − return 0; | |
| 261 | −} | |
| 262 | − | |
| 263 | −/** | |
| 264 | − * bio_integrity_set_tag - Attach a tag buffer to a bio | |
| 265 | − * @bio: bio to attach buffer to | |
| 266 | − * @tag_buf: Pointer to a buffer containing tag data | |
| 267 | − * @len: Length of the included buffer | |
| 268 | − * | |
| 269 | − * Description: Use this function to tag a bio by leveraging the extra | |
| 270 | − * space provided by devices formatted with integrity protection. The | |
| 271 | − * size of the integrity buffer must be <= to the size reported by | |
| 272 | − * bio_integrity_tag_size(). | |
| 273 | − */ | |
| 274 | −int bio_integrity_set_tag(struct bio *bio, void *tag_buf, unsigned int len) | |
| 275 | −{ | |
| 276 | − BUG_ON(bio_data_dir(bio) != WRITE); | |
| 277 | − | |
| 278 | − return bio_integrity_tag(bio, tag_buf, len, 1); | |
| 279 | −} | |
| 280 | −EXPORT_SYMBOL(bio_integrity_set_tag); | |
| 281 | − | |
| 282 | −/** | |
| 283 | − * bio_integrity_get_tag - Retrieve a tag buffer from a bio | |
| 284 | − * @bio: bio to retrieve buffer from | |
| 285 | − * @tag_buf: Pointer to a buffer for the tag data | |
| 286 | − * @len: Length of the target buffer | |
| 287 | − * | |
| 288 | − * Description: Use this function to retrieve the tag buffer from a | |
| 289 | − * completed I/O. The size of the integrity buffer must be <= to the | |
| 290 | − * size reported by bio_integrity_tag_size(). | |
| 291 | − */ | |
| 292 | −int bio_integrity_get_tag(struct bio *bio, void *tag_buf, unsigned int len) | |
| 293 | −{ | |
| 294 | − BUG_ON(bio_data_dir(bio) != READ); | |
| 295 | − | |
| 296 | − return bio_integrity_tag(bio, tag_buf, len, 0); | |
| 297 | −} | |
| 298 | −EXPORT_SYMBOL(bio_integrity_get_tag); | |
| 299 | − | |
| 300 | −/** | |
| 301 | − * bio_integrity_generate_verify - Generate/verify integrity metadata for a bio | |
| 210 | + * bio_integrity_process - Process integrity metadata for a bio | |
| 302 | 211 | * @bio: bio to generate/verify integrity metadata for |
| 303 | − * @operate: operate number, 1 for generate, 0 for verify | |
| 212 | + * @proc_fn: Pointer to the relevant processing function | |
| 304 | 213 | */ |
| 305 | −static int bio_integrity_generate_verify(struct bio *bio, int operate) | |
| 214 | +static int bio_integrity_process(struct bio *bio, | |
| 215 | + integrity_processing_fn *proc_fn) | |
| 306 | 216 | { |
| 307 | 217 | struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev); |
| 308 | − struct blk_integrity_exchg bix; | |
| 218 | + struct blk_integrity_iter iter; | |
| 309 | 219 | struct bio_vec *bv; |
| 310 | − sector_t sector; | |
| 311 | − unsigned int sectors, ret = 0, i; | |
| 312 | − void *prot_buf = bio->bi_integrity->bip_buf; | |
| 313 | − | |
| 314 | − if (operate) | |
| 315 | − sector = bio->bi_iter.bi_sector; | |
| 316 | − else | |
| 317 | − sector = bio->bi_integrity->bip_iter.bi_sector; | |
| 220 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 221 | + unsigned int i, ret = 0; | |
| 222 | + void *prot_buf = page_address(bip->bip_vec->bv_page) + | |
| 223 | + bip->bip_vec->bv_offset; | |
| 318 | 224 | |
| 319 | − bix.disk_name = bio->bi_bdev->bd_disk->disk_name; | |
| 320 | − bix.sector_size = bi->sector_size; | |
| 225 | + iter.disk_name = bio->bi_bdev->bd_disk->disk_name; | |
| 226 | + iter.interval = bi->interval; | |
| 227 | + iter.seed = bip_get_seed(bip); | |
| 228 | + iter.prot_buf = prot_buf; | |
| 321 | 229 | |
| 322 | 230 | bio_for_each_segment_all(bv, bio, i) { |
| 323 | 231 | void *kaddr = kmap_atomic(bv->bv_page); |
| 324 | − bix.data_buf = kaddr + bv->bv_offset; | |
| 325 | − bix.data_size = bv->bv_len; | |
| 326 | − bix.prot_buf = prot_buf; | |
| 327 | − bix.sector = sector; | |
| 328 | − | |
| 329 | − if (operate) | |
| 330 | − bi->generate_fn(&bix); | |
| 331 | − else { | |
| 332 | − ret = bi->verify_fn(&bix); | |
| 333 | − if (ret) { | |
| 334 | − kunmap_atomic(kaddr); | |
| 335 | − return ret; | |
| 336 | − } | |
| 337 | − } | |
| 338 | 232 | |
| 339 | − sectors = bv->bv_len / bi->sector_size; | |
| 340 | − sector += sectors; | |
| 341 | − prot_buf += sectors * bi->tuple_size; | |
| 233 | + iter.data_buf = kaddr + bv->bv_offset; | |
| 234 | + iter.data_size = bv->bv_len; | |
| 235 | + | |
| 236 | + ret = proc_fn(&iter); | |
| 237 | + if (ret) { | |
| 238 | + kunmap_atomic(kaddr); | |
| 239 | + return ret; | |
| 240 | + } | |
| 342 | 241 | |
| 343 | 242 | kunmap_atomic(kaddr); |
| 344 | 243 | } |
| 345 | 244 | return ret; |
| 346 | 245 | } |
| 347 | 246 | |
| 348 | −/** | |
| 349 | − * bio_integrity_generate - Generate integrity metadata for a bio | |
| 350 | − * @bio: bio to generate integrity metadata for | |
| 351 | − * | |
| 352 | − * Description: Generates integrity metadata for a bio by calling the | |
| 353 | − * block device's generation callback function. The bio must have a | |
| 354 | − * bip attached with enough room to accommodate the generated | |
| 355 | − * integrity metadata. | |
| 356 | − */ | |
| 357 | −static void bio_integrity_generate(struct bio *bio) | |
| 358 | −{ | |
| 359 | − bio_integrity_generate_verify(bio, 1); | |
| 360 | −} | |
| 361 | − | |
| 362 | −static inline unsigned short blk_integrity_tuple_size(struct blk_integrity *bi) | |
| 363 | −{ | |
| 364 | − if (bi) | |
| 365 | − return bi->tuple_size; | |
| 366 | − | |
| 367 | − return 0; | |
| 368 | −} | |
| 369 | − | |
| 370 | 247 | /** |
| 371 | 248 | * bio_integrity_prep - Prepare bio for integrity I/O |
| 372 | 249 | * @bio: bio to prepare |
| @@ -387,17 +264,17 @@ int bio_integrity_prep(struct bio *bio) · 14 unmodified lines | ||
| 387 | 264 | unsigned long start, end; |
| 388 | 265 | unsigned int len, nr_pages; |
| 389 | 266 | unsigned int bytes, offset, i; |
| 390 | − unsigned int sectors; | |
| 267 | + unsigned int intervals; | |
| 391 | 268 | |
| 392 | 269 | bi = bdev_get_integrity(bio->bi_bdev); |
| 393 | 270 | q = bdev_get_queue(bio->bi_bdev); |
| 394 | 271 | BUG_ON(bi == NULL); |
| 395 | 272 | BUG_ON(bio_integrity(bio)); |
| 396 | 273 | |
| 397 | − sectors = bio_integrity_hw_sectors(bi, bio_sectors(bio)); | |
| 274 | + intervals = bio_integrity_intervals(bi, bio_sectors(bio)); | |
| 398 | 275 | |
| 399 | 276 | /* Allocate kernel buffer for protection data */ |
| 400 | − len = sectors * blk_integrity_tuple_size(bi); | |
| 277 | + len = intervals * bi->tuple_size; | |
| 401 | 278 | buf = kmalloc(len, GFP_NOIO | q->bounce_gfp); |
| 402 | 279 | if (unlikely(buf == NULL)) { |
| 403 | 280 | printk(KERN_ERR "could not allocate integrity buffer\n"); |
| @@ -416,10 +293,12 @@ int bio_integrity_prep(struct bio *bio) · 12 unmodified lines | ||
| 416 | 293 | return -EIO; |
| 417 | 294 | } |
| 418 | 295 | |
| 419 | − bip->bip_owns_buf = 1; | |
| 420 | − bip->bip_buf = buf; | |
| 296 | + bip->bip_flags |= BIP_BLOCK_INTEGRITY; | |
| 421 | 297 | bip->bip_iter.bi_size = len; |
| 422 | − bip->bip_iter.bi_sector = bio->bi_iter.bi_sector; | |
| 298 | + bip_set_seed(bip, bio->bi_iter.bi_sector); | |
| 299 | + | |
| 300 | + if (bi->flags & BLK_INTEGRITY_IP_CHECKSUM) | |
| 301 | + bip->bip_flags |= BIP_IP_CHECKSUM; | |
| 423 | 302 | |
| 424 | 303 | /* Map it */ |
| 425 | 304 | offset = offset_in_page(buf); |
| @@ -455,25 +334,12 @@ int bio_integrity_prep(struct bio *bio) · 29 unmodified lines | ||
| 455 | 334 | |
| 456 | 335 | /* Auto-generate integrity metadata if this is a write */ |
| 457 | 336 | if (bio_data_dir(bio) == WRITE) |
| 458 | − bio_integrity_generate(bio); | |
| 337 | + bio_integrity_process(bio, bi->generate_fn); | |
| 459 | 338 | |
| 460 | 339 | return 0; |
| 461 | 340 | } |
| 462 | 341 | EXPORT_SYMBOL(bio_integrity_prep); |
| 463 | 342 | |
| 464 | −/** | |
| 465 | − * bio_integrity_verify - Verify integrity metadata for a bio | |
| 466 | − * @bio: bio to verify | |
| 467 | − * | |
| 468 | − * Description: This function is called to verify the integrity of a | |
| 469 | − * bio. The data in the bio io_vec is compared to the integrity | |
| 470 | − * metadata returned by the HBA. | |
| 471 | − */ | |
| 472 | −static int bio_integrity_verify(struct bio *bio) | |
| 473 | −{ | |
| 474 | − return bio_integrity_generate_verify(bio, 0); | |
| 475 | −} | |
| 476 | − | |
| 477 | 343 | /** |
| 478 | 344 | * bio_integrity_verify_fn - Integrity I/O completion worker |
| 479 | 345 | * @work: Work struct stored in bio to be verified |
| @@ -487,9 +353,10 @@ static void bio_integrity_verify_fn(struct work_struct *work) · 7 unmodified lines | ||
| 487 | 353 | struct bio_integrity_payload *bip = |
| 488 | 354 | container_of(work, struct bio_integrity_payload, bip_work); |
| 489 | 355 | struct bio *bio = bip->bip_bio; |
| 356 | + struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev); | |
| 490 | 357 | int error; |
| 491 | 358 | |
| 492 | − error = bio_integrity_verify(bio); | |
| 359 | + error = bio_integrity_process(bio, bi->verify_fn); | |
| 493 | 360 | |
| 494 | 361 | /* Restore original bio completion handler */ |
| 495 | 362 | bio->bi_end_io = bip->bip_end_io; |
| @@ -510,7 +377,7 @@ static void bio_integrity_verify_fn(struct work_struct *work) · 14 unmodified lines | ||
| 510 | 377 | */ |
| 511 | 378 | void bio_integrity_endio(struct bio *bio, int error) |
| 512 | 379 | { |
| 513 | − struct bio_integrity_payload *bip = bio->bi_integrity; | |
| 380 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 514 | 381 | |
| 515 | 382 | BUG_ON(bip->bip_bio != bio); |
| 516 | 383 | |
| @@ -541,7 +408,7 @@ EXPORT_SYMBOL(bio_integrity_endio); · 24 unmodified lines | ||
| 541 | 408 | */ |
| 542 | 409 | void bio_integrity_advance(struct bio *bio, unsigned int bytes_done) |
| 543 | 410 | { |
| 544 | − struct bio_integrity_payload *bip = bio->bi_integrity; | |
| 411 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 545 | 412 | struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev); |
| 546 | 413 | unsigned bytes = bio_integrity_bytes(bi, bytes_done >> 9); |
| 547 | 414 | |
| @@ -563,7 +430,7 @@ EXPORT_SYMBOL(bio_integrity_advance); · 15 unmodified lines | ||
| 563 | 430 | void bio_integrity_trim(struct bio *bio, unsigned int offset, |
| 564 | 431 | unsigned int sectors) |
| 565 | 432 | { |
| 566 | − struct bio_integrity_payload *bip = bio->bi_integrity; | |
| 433 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 567 | 434 | struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev); |
| 568 | 435 | |
| 569 | 436 | bio_integrity_advance(bio, offset << 9); |
| @@ -582,7 +449,7 @@ EXPORT_SYMBOL(bio_integrity_trim); · 12 unmodified lines | ||
| 582 | 449 | int bio_integrity_clone(struct bio *bio, struct bio *bio_src, |
| 583 | 450 | gfp_t gfp_mask) |
| 584 | 451 | { |
| 585 | − struct bio_integrity_payload *bip_src = bio_src->bi_integrity; | |
| 452 | + struct bio_integrity_payload *bip_src = bio_integrity(bio_src); | |
| 586 | 453 | struct bio_integrity_payload *bip; |
| 587 | 454 | |
| 588 | 455 | BUG_ON(bip_src == NULL); |
| @@ -646,6 +513,4 @@ void __init bio_integrity_init(void) · 57 unmodified lines | ||
| 646 | 513 | sizeof(struct bio_integrity_payload) + |
| 647 | 514 | sizeof(struct bio_vec) * BIP_INLINE_VECS, |
| 648 | 515 | 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL); |
| 649 | − if (!bip_slab) | |
| 650 | − panic("Failed to create slab\n"); | |
| 651 | 516 | } |
| @@ -428,6 +428,9 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, int nr_iovecs, struct bio_set *bs) · 427 unmodified lines | ||
| 428 | 428 | front_pad = 0; |
| 429 | 429 | inline_vecs = nr_iovecs; |
| 430 | 430 | } else { |
| 431 | + /* should not use nobvec bioset for nr_iovecs > 0 */ | |
| 432 | + if (WARN_ON_ONCE(!bs->bvec_pool && nr_iovecs > 0)) | |
| 433 | + return NULL; | |
| 431 | 434 | /* |
| 432 | 435 | * generic_make_request() converts recursion to iteration; this |
| 433 | 436 | * means if we're running beneath it, any bios we allocate and |
| @@ -1900,20 +1903,9 @@ void bioset_free(struct bio_set *bs) · 1466 unmodified lines | ||
| 1900 | 1903 | } |
| 1901 | 1904 | EXPORT_SYMBOL(bioset_free); |
| 1902 | 1905 | |
| 1903 | −/** | |
| 1904 | − * bioset_create - Create a bio_set | |
| 1905 | − * @pool_size: Number of bio and bio_vecs to cache in the mempool | |
| 1906 | − * @front_pad: Number of bytes to allocate in front of the returned bio | |
| 1907 | − * | |
| 1908 | − * Description: | |
| 1909 | − * Set up a bio_set to be used with @bio_alloc_bioset. Allows the caller | |
| 1910 | − * to ask for a number of bytes to be allocated in front of the bio. | |
| 1911 | − * Front pad allocation is useful for embedding the bio inside | |
| 1912 | − * another structure, to avoid allocating extra data to go with the bio. | |
| 1913 | − * Note that the bio must be embedded at the END of that structure always, | |
| 1914 | − * or things will break badly. | |
| 1915 | − */ | |
| 1916 | −struct bio_set *bioset_create(unsigned int pool_size, unsigned int front_pad) | |
| 1906 | +static struct bio_set *__bioset_create(unsigned int pool_size, | |
| 1907 | + unsigned int front_pad, | |
| 1908 | + bool create_bvec_pool) | |
| 1917 | 1909 | { |
| 1918 | 1910 | unsigned int back_pad = BIO_INLINE_VECS * sizeof(struct bio_vec); |
| 1919 | 1911 | struct bio_set *bs; |
| @@ -1938,9 +1930,11 @@ struct bio_set *bioset_create(unsigned int pool_size, unsigned int front_pad) · 18 unmodified lines | ||
| 1938 | 1930 | if (!bs->bio_pool) |
| 1939 | 1931 | goto bad; |
| 1940 | 1932 | |
| 1941 | − bs->bvec_pool = biovec_create_pool(pool_size); | |
| 1942 | − if (!bs->bvec_pool) | |
| 1943 | − goto bad; | |
| 1933 | + if (create_bvec_pool) { | |
| 1934 | + bs->bvec_pool = biovec_create_pool(pool_size); | |
| 1935 | + if (!bs->bvec_pool) | |
| 1936 | + goto bad; | |
| 1937 | + } | |
| 1944 | 1938 | |
| 1945 | 1939 | bs->rescue_workqueue = alloc_workqueue("bioset", WQ_MEM_RECLAIM, 0); |
| 1946 | 1940 | if (!bs->rescue_workqueue) |
| @@ -1951,8 +1945,41 @@ bad: · 4 unmodified lines | ||
| 1951 | 1945 | bioset_free(bs); |
| 1952 | 1946 | return NULL; |
| 1953 | 1947 | } |
| 1948 | + | |
| 1949 | +/** | |
| 1950 | + * bioset_create - Create a bio_set | |
| 1951 | + * @pool_size: Number of bio and bio_vecs to cache in the mempool | |
| 1952 | + * @front_pad: Number of bytes to allocate in front of the returned bio | |
| 1953 | + * | |
| 1954 | + * Description: | |
| 1955 | + * Set up a bio_set to be used with @bio_alloc_bioset. Allows the caller | |
| 1956 | + * to ask for a number of bytes to be allocated in front of the bio. | |
| 1957 | + * Front pad allocation is useful for embedding the bio inside | |
| 1958 | + * another structure, to avoid allocating extra data to go with the bio. | |
| 1959 | + * Note that the bio must be embedded at the END of that structure always, | |
| 1960 | + * or things will break badly. | |
| 1961 | + */ | |
| 1962 | +struct bio_set *bioset_create(unsigned int pool_size, unsigned int front_pad) | |
| 1963 | +{ | |
| 1964 | + return __bioset_create(pool_size, front_pad, true); | |
| 1965 | +} | |
| 1954 | 1966 | EXPORT_SYMBOL(bioset_create); |
| 1955 | 1967 | |
| 1968 | +/** | |
| 1969 | + * bioset_create_nobvec - Create a bio_set without bio_vec mempool | |
| 1970 | + * @pool_size: Number of bio to cache in the mempool | |
| 1971 | + * @front_pad: Number of bytes to allocate in front of the returned bio | |
| 1972 | + * | |
| 1973 | + * Description: | |
| 1974 | + * Same functionality as bioset_create() except that mempool is not | |
| 1975 | + * created for bio_vecs. Saving some memory for bio_clone_fast() users. | |
| 1976 | + */ | |
| 1977 | +struct bio_set *bioset_create_nobvec(unsigned int pool_size, unsigned int front_pad) | |
| 1978 | +{ | |
| 1979 | + return __bioset_create(pool_size, front_pad, false); | |
| 1980 | +} | |
| 1981 | +EXPORT_SYMBOL(bioset_create_nobvec); | |
| 1982 | + | |
| 1956 | 1983 | #ifdef CONFIG_BLK_CGROUP |
| 1957 | 1984 | /** |
| 1958 | 1985 | * bio_associate_current - associate a bio with %current |
| @@ -822,7 +822,6 @@ static void blkcg_css_free(struct cgroup_subsys_state *css) · 821 unmodified lines | ||
| 822 | 822 | static struct cgroup_subsys_state * |
| 823 | 823 | blkcg_css_alloc(struct cgroup_subsys_state *parent_css) |
| 824 | 824 | { |
| 825 | − static atomic64_t id_seq = ATOMIC64_INIT(0); | |
| 826 | 825 | struct blkcg *blkcg; |
| 827 | 826 | |
| 828 | 827 | if (!parent_css) { |
| @@ -836,7 +835,6 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_css) · 7 unmodified lines | ||
| 836 | 835 | |
| 837 | 836 | blkcg->cfq_weight = CFQ_WEIGHT_DEFAULT; |
| 838 | 837 | blkcg->cfq_leaf_weight = CFQ_WEIGHT_DEFAULT; |
| 839 | − blkcg->id = atomic64_inc_return(&id_seq); /* root is 0, start from 1 */ | |
| 840 | 838 | done: |
| 841 | 839 | spin_lock_init(&blkcg->lock); |
| 842 | 840 | INIT_RADIX_TREE(&blkcg->blkg_tree, GFP_ATOMIC); |
| @@ -50,9 +50,6 @@ struct blkcg { · 49 unmodified lines | ||
| 50 | 50 | struct blkcg_gq *blkg_hint; |
| 51 | 51 | struct hlist_head blkg_list; |
| 52 | 52 | |
| 53 | − /* for policies to test whether associated blkcg has changed */ | |
| 54 | − uint64_t id; | |
| 55 | − | |
| 56 | 53 | /* TODO: per-policy storage in blkcg */ |
| 57 | 54 | unsigned int cfq_weight; /* belongs to cfq */ |
| 58 | 55 | unsigned int cfq_leaf_weight; |
| @@ -83,18 +83,14 @@ void blk_queue_congestion_threshold(struct request_queue *q) · 82 unmodified lines | ||
| 83 | 83 | * @bdev: device |
| 84 | 84 | * |
| 85 | 85 | * Locates the passed device's request queue and returns the address of its |
| 86 | − * backing_dev_info | |
| 87 | − * | |
| 88 | − * Will return NULL if the request queue cannot be located. | |
| 86 | + * backing_dev_info. This function can only be called if @bdev is opened | |
| 87 | + * and the return value is never NULL. | |
| 89 | 88 | */ |
| 90 | 89 | struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev) |
| 91 | 90 | { |
| 92 | − struct backing_dev_info *ret = NULL; | |
| 93 | 91 | struct request_queue *q = bdev_get_queue(bdev); |
| 94 | 92 | |
| 95 | − if (q) | |
| 96 | − ret = &q->backing_dev_info; | |
| 97 | − return ret; | |
| 93 | + return &q->backing_dev_info; | |
| 98 | 94 | } |
| 99 | 95 | EXPORT_SYMBOL(blk_get_backing_dev_info); |
| 100 | 96 | |
| @@ -394,11 +390,13 @@ static void __blk_drain_queue(struct request_queue *q, bool drain_all) · 293 unmodified lines | ||
| 394 | 390 | * be drained. Check all the queues and counters. |
| 395 | 391 | */ |
| 396 | 392 | if (drain_all) { |
| 393 | + struct blk_flush_queue *fq = blk_get_flush_queue(q, NULL); | |
| 397 | 394 | drain |= !list_empty(&q->queue_head); |
| 398 | 395 | for (i = 0; i < 2; i++) { |
| 399 | 396 | drain |= q->nr_rqs[i]; |
| 400 | 397 | drain |= q->in_flight[i]; |
| 401 | − drain |= !list_empty(&q->flush_queue[i]); | |
| 398 | + if (fq) | |
| 399 | + drain |= !list_empty(&fq->flush_queue[i]); | |
| 402 | 400 | } |
| 403 | 401 | } |
| 404 | 402 | |
| @@ -604,9 +602,6 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) · 199 unmodified lines | ||
| 604 | 602 | #ifdef CONFIG_BLK_CGROUP |
| 605 | 603 | INIT_LIST_HEAD(&q->blkg_list); |
| 606 | 604 | #endif |
| 607 | − INIT_LIST_HEAD(&q->flush_queue[0]); | |
| 608 | − INIT_LIST_HEAD(&q->flush_queue[1]); | |
| 609 | − INIT_LIST_HEAD(&q->flush_data_in_flight); | |
| 610 | 605 | INIT_DELAYED_WORK(&q->delay_work, blk_delay_work); |
| 611 | 606 | |
| 612 | 607 | kobject_init(&q->kobj, &blk_queue_ktype); |
| @@ -709,8 +704,8 @@ blk_init_allocated_queue(struct request_queue *q, request_fn_proc *rfn, · 96 unmodified lines | ||
| 709 | 704 | if (!q) |
| 710 | 705 | return NULL; |
| 711 | 706 | |
| 712 | − q->flush_rq = kzalloc(sizeof(struct request), GFP_KERNEL); | |
| 713 | − if (!q->flush_rq) | |
| 707 | + q->fq = blk_alloc_flush_queue(q, NUMA_NO_NODE, 0); | |
| 708 | + if (!q->fq) | |
| 714 | 709 | return NULL; |
| 715 | 710 | |
| 716 | 711 | if (blk_init_rl(&q->root_rl, q, GFP_KERNEL)) |
| @@ -746,7 +741,7 @@ blk_init_allocated_queue(struct request_queue *q, request_fn_proc *rfn, · 29 unmodified lines | ||
| 746 | 741 | return q; |
| 747 | 742 | |
| 748 | 743 | fail: |
| 749 | − kfree(q->flush_rq); | |
| 744 | + blk_free_flush_queue(q->fq); | |
| 750 | 745 | return NULL; |
| 751 | 746 | } |
| 752 | 747 | EXPORT_SYMBOL(blk_init_allocated_queue); |
| @@ -934,8 +929,8 @@ static struct io_context *rq_ioc(struct bio *bio) · 181 unmodified lines | ||
| 934 | 929 | * pressure or if @q is dead. |
| 935 | 930 | * |
| 936 | 931 | * Must be called with @q->queue_lock held and, |
| 937 | − * Returns %NULL on failure, with @q->queue_lock held. | |
| 938 | − * Returns !%NULL on success, with @q->queue_lock *not held*. | |
| 932 | + * Returns ERR_PTR on failure, with @q->queue_lock held. | |
| 933 | + * Returns request pointer on success, with @q->queue_lock *not held*. | |
| 939 | 934 | */ |
| 940 | 935 | static struct request *__get_request(struct request_list *rl, int rw_flags, |
| 941 | 936 | struct bio *bio, gfp_t gfp_mask) |
| @@ -949,7 +944,7 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, · 7 unmodified lines | ||
| 949 | 944 | int may_queue; |
| 950 | 945 | |
| 951 | 946 | if (unlikely(blk_queue_dying(q))) |
| 952 | − return NULL; | |
| 947 | + return ERR_PTR(-ENODEV); | |
| 953 | 948 | |
| 954 | 949 | may_queue = elv_may_queue(q, rw_flags); |
| 955 | 950 | if (may_queue == ELV_MQUEUE_NO) |
| @@ -974,7 +969,7 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, · 18 unmodified lines | ||
| 974 | 969 | * process is not a "batcher", and not |
| 975 | 970 | * exempted by the IO scheduler |
| 976 | 971 | */ |
| 977 | − return NULL; | |
| 972 | + return ERR_PTR(-ENOMEM); | |
| 978 | 973 | } |
| 979 | 974 | } |
| 980 | 975 | } |
| @@ -992,7 +987,7 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, · 11 unmodified lines | ||
| 992 | 987 | * allocated with any setting of ->nr_requests |
| 993 | 988 | */ |
| 994 | 989 | if (rl->count[is_sync] >= (3 * q->nr_requests / 2)) |
| 995 | − return NULL; | |
| 990 | + return ERR_PTR(-ENOMEM); | |
| 996 | 991 | |
| 997 | 992 | q->nr_rqs[is_sync]++; |
| 998 | 993 | rl->count[is_sync]++; |
| @@ -1065,8 +1060,8 @@ fail_elvpriv: · 66 unmodified lines | ||
| 1065 | 1060 | * shouldn't stall IO. Treat this request as !elvpriv. This will |
| 1066 | 1061 | * disturb iosched and blkcg but weird is bettern than dead. |
| 1067 | 1062 | */ |
| 1068 | − printk_ratelimited(KERN_WARNING "%s: request aux data allocation failed, iosched may be disturbed\n", | |
| 1069 | − dev_name(q->backing_dev_info.dev)); | |
| 1063 | + printk_ratelimited(KERN_WARNING "%s: dev %s: request aux data allocation failed, iosched may be disturbed\n", | |
| 1064 | + __func__, dev_name(q->backing_dev_info.dev)); | |
| 1070 | 1065 | |
| 1071 | 1066 | rq->cmd_flags &= ~REQ_ELVPRIV; |
| 1072 | 1067 | rq->elv.icq = NULL; |
| @@ -1097,7 +1092,7 @@ fail_alloc: · 24 unmodified lines | ||
| 1097 | 1092 | rq_starved: |
| 1098 | 1093 | if (unlikely(rl->count[is_sync] == 0)) |
| 1099 | 1094 | rl->starved[is_sync] = 1; |
| 1100 | − return NULL; | |
| 1095 | + return ERR_PTR(-ENOMEM); | |
| 1101 | 1096 | } |
| 1102 | 1097 | |
| 1103 | 1098 | /** |
| @@ -1111,8 +1106,8 @@ rq_starved: · 7 unmodified lines | ||
| 1111 | 1106 | * function keeps retrying under memory pressure and fails iff @q is dead. |
| 1112 | 1107 | * |
| 1113 | 1108 | * Must be called with @q->queue_lock held and, |
| 1114 | − * Returns %NULL on failure, with @q->queue_lock held. | |
| 1115 | − * Returns !%NULL on success, with @q->queue_lock *not held*. | |
| 1109 | + * Returns ERR_PTR on failure, with @q->queue_lock held. | |
| 1110 | + * Returns request pointer on success, with @q->queue_lock *not held*. | |
| 1116 | 1111 | */ |
| 1117 | 1112 | static struct request *get_request(struct request_queue *q, int rw_flags, |
| 1118 | 1113 | struct bio *bio, gfp_t gfp_mask) |
| @@ -1125,12 +1120,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags, · 6 unmodified lines | ||
| 1125 | 1120 | rl = blk_get_rl(q, bio); /* transferred to @rq on success */ |
| 1126 | 1121 | retry: |
| 1127 | 1122 | rq = __get_request(rl, rw_flags, bio, gfp_mask); |
| 1128 | − if (rq) | |
| 1123 | + if (!IS_ERR(rq)) | |
| 1129 | 1124 | return rq; |
| 1130 | 1125 | |
| 1131 | 1126 | if (!(gfp_mask & __GFP_WAIT) || unlikely(blk_queue_dying(q))) { |
| 1132 | 1127 | blk_put_rl(rl); |
| 1133 | − return NULL; | |
| 1128 | + return rq; | |
| 1134 | 1129 | } |
| 1135 | 1130 | |
| 1136 | 1131 | /* wait on @rl and retry */ |
| @@ -1167,7 +1162,7 @@ static struct request *blk_old_get_request(struct request_queue *q, int rw, · 30 unmodified lines | ||
| 1167 | 1162 | |
| 1168 | 1163 | spin_lock_irq(q->queue_lock); |
| 1169 | 1164 | rq = get_request(q, rw, NULL, gfp_mask); |
| 1170 | − if (!rq) | |
| 1165 | + if (IS_ERR(rq)) | |
| 1171 | 1166 | spin_unlock_irq(q->queue_lock); |
| 1172 | 1167 | /* q->queue_lock is unlocked at this point */ |
| 1173 | 1168 | |
| @@ -1219,8 +1214,8 @@ struct request *blk_make_request(struct request_queue *q, struct bio *bio, · 45 unmodified lines | ||
| 1219 | 1214 | { |
| 1220 | 1215 | struct request *rq = blk_get_request(q, bio_data_dir(bio), gfp_mask); |
| 1221 | 1216 | |
| 1222 | − if (unlikely(!rq)) | |
| 1223 | − return ERR_PTR(-ENOMEM); | |
| 1217 | + if (IS_ERR(rq)) | |
| 1218 | + return rq; | |
| 1224 | 1219 | |
| 1225 | 1220 | blk_rq_set_block_pc(rq); |
| 1226 | 1221 | |
| @@ -1614,8 +1609,8 @@ get_rq: · 387 unmodified lines | ||
| 1614 | 1609 | * Returns with the queue unlocked. |
| 1615 | 1610 | */ |
| 1616 | 1611 | req = get_request(q, rw_flags, bio, GFP_NOIO); |
| 1617 | − if (unlikely(!req)) { | |
| 1618 | − bio_endio(bio, -ENODEV); /* @q is dead */ | |
| 1612 | + if (IS_ERR(req)) { | |
| 1613 | + bio_endio(bio, PTR_ERR(req)); /* @q is dead */ | |
| 1619 | 1614 | goto out_unlock; |
| 1620 | 1615 | } |
| 1621 | 1616 | |
| @@ -2405,11 +2400,11 @@ bool blk_update_request(struct request *req, int error, unsigned int nr_bytes) · 783 unmodified lines | ||
| 2405 | 2400 | { |
| 2406 | 2401 | int total_bytes; |
| 2407 | 2402 | |
| 2403 | + trace_block_rq_complete(req->q, req, nr_bytes); | |
| 2404 | + | |
| 2408 | 2405 | if (!req->bio) |
| 2409 | 2406 | return false; |
| 2410 | 2407 | |
| 2411 | − trace_block_rq_complete(req->q, req, nr_bytes); | |
| 2412 | − | |
| 2413 | 2408 | /* |
| 2414 | 2409 | * For fs requests, rq is just carrier of independent bio's |
| 2415 | 2410 | * and each partial completion should be handled separately. |
| @@ -2449,8 +2444,8 @@ bool blk_update_request(struct request *req, int error, unsigned int nr_bytes) · 33 unmodified lines | ||
| 2449 | 2444 | error_type = "I/O"; |
| 2450 | 2445 | break; |
| 2451 | 2446 | } |
| 2452 | − printk_ratelimited(KERN_ERR "end_request: %s error, dev %s, sector %llu\n", | |
| 2453 | − error_type, req->rq_disk ? | |
| 2447 | + printk_ratelimited(KERN_ERR "%s: %s error, dev %s, sector %llu\n", | |
| 2448 | + __func__, error_type, req->rq_disk ? | |
| 2454 | 2449 | req->rq_disk->disk_name : "?", |
| 2455 | 2450 | (unsigned long long)blk_rq_pos(req)); |
| 2456 | 2451 | |
| @@ -2931,7 +2926,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, · 474 unmodified lines | ||
| 2931 | 2926 | blk_rq_init(NULL, rq); |
| 2932 | 2927 | |
| 2933 | 2928 | __rq_for_each_bio(bio_src, rq_src) { |
| 2934 | − bio = bio_clone_bioset(bio_src, gfp_mask, bs); | |
| 2929 | + bio = bio_clone_fast(bio_src, gfp_mask, bs); | |
| 2935 | 2930 | if (!bio) |
| 2936 | 2931 | goto free_and_out; |
| 2937 | 2932 | |
| @@ -28,7 +28,7 @@ · 27 unmodified lines | ||
| 28 | 28 | * |
| 29 | 29 | * The actual execution of flush is double buffered. Whenever a request |
| 30 | 30 | * needs to execute PRE or POSTFLUSH, it queues at |
| 31 | − * q->flush_queue[q->flush_pending_idx]. Once certain criteria are met, a | |
| 31 | + * fq->flush_queue[fq->flush_pending_idx]. Once certain criteria are met, a | |
| 32 | 32 | * flush is issued and the pending_idx is toggled. When the flush |
| 33 | 33 | * completes, all the requests which were pending are proceeded to the next |
| 34 | 34 | * step. This allows arbitrary merging of different types of FLUSH/FUA |
| @@ -91,7 +91,8 @@ enum { · 56 unmodified lines | ||
| 91 | 91 | FLUSH_PENDING_TIMEOUT = 5 * HZ, |
| 92 | 92 | }; |
| 93 | 93 | |
| 94 | −static bool blk_kick_flush(struct request_queue *q); | |
| 94 | +static bool blk_kick_flush(struct request_queue *q, | |
| 95 | + struct blk_flush_queue *fq); | |
| 95 | 96 | |
| 96 | 97 | static unsigned int blk_flush_policy(unsigned int fflags, struct request *rq) |
| 97 | 98 | { |
| @@ -126,8 +127,6 @@ static void blk_flush_restore_request(struct request *rq) · 28 unmodified lines | ||
| 126 | 127 | /* make @rq a normal request */ |
| 127 | 128 | rq->cmd_flags &= ~REQ_FLUSH_SEQ; |
| 128 | 129 | rq->end_io = rq->flush.saved_end_io; |
| 129 | − | |
| 130 | − blk_clear_rq_complete(rq); | |
| 131 | 130 | } |
| 132 | 131 | |
| 133 | 132 | static bool blk_flush_queue_rq(struct request *rq, bool add_front) |
| @@ -150,6 +149,7 @@ static bool blk_flush_queue_rq(struct request *rq, bool add_front) · 16 unmodified lines | ||
| 150 | 149 | /** |
| 151 | 150 | * blk_flush_complete_seq - complete flush sequence |
| 152 | 151 | * @rq: FLUSH/FUA request being sequenced |
| 152 | + * @fq: flush queue | |
| 153 | 153 | * @seq: sequences to complete (mask of %REQ_FSEQ_*, can be zero) |
| 154 | 154 | * @error: whether an error occurred |
| 155 | 155 | * |
| @@ -157,16 +157,17 @@ static bool blk_flush_queue_rq(struct request *rq, bool add_front) · 1 unmodified line | ||
| 157 | 157 | * completion and trigger the next step. |
| 158 | 158 | * |
| 159 | 159 | * CONTEXT: |
| 160 | − * spin_lock_irq(q->queue_lock or q->mq_flush_lock) | |
| 160 | + * spin_lock_irq(q->queue_lock or fq->mq_flush_lock) | |
| 161 | 161 | * |
| 162 | 162 | * RETURNS: |
| 163 | 163 | * %true if requests were added to the dispatch queue, %false otherwise. |
| 164 | 164 | */ |
| 165 | −static bool blk_flush_complete_seq(struct request *rq, unsigned int seq, | |
| 166 | − int error) | |
| 165 | +static bool blk_flush_complete_seq(struct request *rq, | |
| 166 | + struct blk_flush_queue *fq, | |
| 167 | + unsigned int seq, int error) | |
| 167 | 168 | { |
| 168 | 169 | struct request_queue *q = rq->q; |
| 169 | − struct list_head *pending = &q->flush_queue[q->flush_pending_idx]; | |
| 170 | + struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx]; | |
| 170 | 171 | bool queued = false, kicked; |
| 171 | 172 | |
| 172 | 173 | BUG_ON(rq->flush.seq & seq); |
| @@ -182,12 +183,12 @@ static bool blk_flush_complete_seq(struct request *rq, unsigned int seq, · 9 unmodified lines | ||
| 182 | 183 | case REQ_FSEQ_POSTFLUSH: |
| 183 | 184 | /* queue for flush */ |
| 184 | 185 | if (list_empty(pending)) |
| 185 | − q->flush_pending_since = jiffies; | |
| 186 | + fq->flush_pending_since = jiffies; | |
| 186 | 187 | list_move_tail(&rq->flush.list, pending); |
| 187 | 188 | break; |
| 188 | 189 | |
| 189 | 190 | case REQ_FSEQ_DATA: |
| 190 | − list_move_tail(&rq->flush.list, &q->flush_data_in_flight); | |
| 191 | + list_move_tail(&rq->flush.list, &fq->flush_data_in_flight); | |
| 191 | 192 | queued = blk_flush_queue_rq(rq, true); |
| 192 | 193 | break; |
| 193 | 194 | |
| @@ -202,7 +203,7 @@ static bool blk_flush_complete_seq(struct request *rq, unsigned int seq, · 8 unmodified lines | ||
| 202 | 203 | list_del_init(&rq->flush.list); |
| 203 | 204 | blk_flush_restore_request(rq); |
| 204 | 205 | if (q->mq_ops) |
| 205 | − blk_mq_end_io(rq, error); | |
| 206 | + blk_mq_end_request(rq, error); | |
| 206 | 207 | else |
| 207 | 208 | __blk_end_request_all(rq, error); |
| 208 | 209 | break; |
| @@ -211,7 +212,7 @@ static bool blk_flush_complete_seq(struct request *rq, unsigned int seq, · 2 unmodified lines | ||
| 211 | 212 | BUG(); |
| 212 | 213 | } |
| 213 | 214 | |
| 214 | − kicked = blk_kick_flush(q); | |
| 215 | + kicked = blk_kick_flush(q, fq); | |
| 215 | 216 | return kicked | queued; |
| 216 | 217 | } |
| 217 | 218 | |
| @@ -222,17 +223,18 @@ static void flush_end_io(struct request *flush_rq, int error) · 4 unmodified lines | ||
| 222 | 223 | bool queued = false; |
| 223 | 224 | struct request *rq, *n; |
| 224 | 225 | unsigned long flags = 0; |
| 226 | + struct blk_flush_queue *fq = blk_get_flush_queue(q, flush_rq->mq_ctx); | |
| 225 | 227 | |
| 226 | 228 | if (q->mq_ops) { |
| 227 | − spin_lock_irqsave(&q->mq_flush_lock, flags); | |
| 228 | − q->flush_rq->tag = -1; | |
| 229 | + spin_lock_irqsave(&fq->mq_flush_lock, flags); | |
| 230 | + flush_rq->tag = -1; | |
| 229 | 231 | } |
| 230 | 232 | |
| 231 | − running = &q->flush_queue[q->flush_running_idx]; | |
| 232 | − BUG_ON(q->flush_pending_idx == q->flush_running_idx); | |
| 233 | + running = &fq->flush_queue[fq->flush_running_idx]; | |
| 234 | + BUG_ON(fq->flush_pending_idx == fq->flush_running_idx); | |
| 233 | 235 | |
| 234 | 236 | /* account completion of the flush request */ |
| 235 | − q->flush_running_idx ^= 1; | |
| 237 | + fq->flush_running_idx ^= 1; | |
| 236 | 238 | |
| 237 | 239 | if (!q->mq_ops) |
| 238 | 240 | elv_completed_request(q, flush_rq); |
| @@ -242,7 +244,7 @@ static void flush_end_io(struct request *flush_rq, int error) · 3 unmodified lines | ||
| 242 | 244 | unsigned int seq = blk_flush_cur_seq(rq); |
| 243 | 245 | |
| 244 | 246 | BUG_ON(seq != REQ_FSEQ_PREFLUSH && seq != REQ_FSEQ_POSTFLUSH); |
| 245 | − queued |= blk_flush_complete_seq(rq, seq, error); | |
| 247 | + queued |= blk_flush_complete_seq(rq, fq, seq, error); | |
| 246 | 248 | } |
| 247 | 249 | |
| 248 | 250 | /* |
| @@ -256,71 +258,81 @@ static void flush_end_io(struct request *flush_rq, int error) · 7 unmodified lines | ||
| 256 | 258 | * directly into request_fn may confuse the driver. Always use |
| 257 | 259 | * kblockd. |
| 258 | 260 | */ |
| 259 | − if (queued || q->flush_queue_delayed) { | |
| 261 | + if (queued || fq->flush_queue_delayed) { | |
| 260 | 262 | WARN_ON(q->mq_ops); |
| 261 | 263 | blk_run_queue_async(q); |
| 262 | 264 | } |
| 263 | − q->flush_queue_delayed = 0; | |
| 265 | + fq->flush_queue_delayed = 0; | |
| 264 | 266 | if (q->mq_ops) |
| 265 | − spin_unlock_irqrestore(&q->mq_flush_lock, flags); | |
| 267 | + spin_unlock_irqrestore(&fq->mq_flush_lock, flags); | |
| 266 | 268 | } |
| 267 | 269 | |
| 268 | 270 | /** |
| 269 | 271 | * blk_kick_flush - consider issuing flush request |
| 270 | 272 | * @q: request_queue being kicked |
| 273 | + * @fq: flush queue | |
| 271 | 274 | * |
| 272 | 275 | * Flush related states of @q have changed, consider issuing flush request. |
| 273 | 276 | * Please read the comment at the top of this file for more info. |
| 274 | 277 | * |
| 275 | 278 | * CONTEXT: |
| 276 | − * spin_lock_irq(q->queue_lock or q->mq_flush_lock) | |
| 279 | + * spin_lock_irq(q->queue_lock or fq->mq_flush_lock) | |
| 277 | 280 | * |
| 278 | 281 | * RETURNS: |
| 279 | 282 | * %true if flush was issued, %false otherwise. |
| 280 | 283 | */ |
| 281 | −static bool blk_kick_flush(struct request_queue *q) | |
| 284 | +static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq) | |
| 282 | 285 | { |
| 283 | − struct list_head *pending = &q->flush_queue[q->flush_pending_idx]; | |
| 286 | + struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx]; | |
| 284 | 287 | struct request *first_rq = |
| 285 | 288 | list_first_entry(pending, struct request, flush.list); |
| 289 | + struct request *flush_rq = fq->flush_rq; | |
| 286 | 290 | |
| 287 | 291 | /* C1 described at the top of this file */ |
| 288 | − if (q->flush_pending_idx != q->flush_running_idx || list_empty(pending)) | |
| 292 | + if (fq->flush_pending_idx != fq->flush_running_idx || list_empty(pending)) | |
| 289 | 293 | return false; |
| 290 | 294 | |
| 291 | 295 | /* C2 and C3 */ |
| 292 | − if (!list_empty(&q->flush_data_in_flight) && | |
| 296 | + if (!list_empty(&fq->flush_data_in_flight) && | |
| 293 | 297 | time_before(jiffies, |
| 294 | − q->flush_pending_since + FLUSH_PENDING_TIMEOUT)) | |
| 298 | + fq->flush_pending_since + FLUSH_PENDING_TIMEOUT)) | |
| 295 | 299 | return false; |
| 296 | 300 | |
| 297 | 301 | /* |
| 298 | 302 | * Issue flush and toggle pending_idx. This makes pending_idx |
| 299 | 303 | * different from running_idx, which means flush is in flight. |
| 300 | 304 | */ |
| 301 | − q->flush_pending_idx ^= 1; | |
| 305 | + fq->flush_pending_idx ^= 1; | |
| 302 | 306 | |
| 303 | − blk_rq_init(q, q->flush_rq); | |
| 304 | − if (q->mq_ops) | |
| 305 | − blk_mq_clone_flush_request(q->flush_rq, first_rq); | |
| 307 | + blk_rq_init(q, flush_rq); | |
| 308 | + | |
| 309 | + /* | |
| 310 | + * Borrow tag from the first request since they can't | |
| 311 | + * be in flight at the same time. | |
| 312 | + */ | |
| 313 | + if (q->mq_ops) { | |
| 314 | + flush_rq->mq_ctx = first_rq->mq_ctx; | |
| 315 | + flush_rq->tag = first_rq->tag; | |
| 316 | + } | |
| 306 | 317 | |
| 307 | − q->flush_rq->cmd_type = REQ_TYPE_FS; | |
| 308 | − q->flush_rq->cmd_flags = WRITE_FLUSH | REQ_FLUSH_SEQ; | |
| 309 | − q->flush_rq->rq_disk = first_rq->rq_disk; | |
| 310 | − q->flush_rq->end_io = flush_end_io; | |
| 318 | + flush_rq->cmd_type = REQ_TYPE_FS; | |
| 319 | + flush_rq->cmd_flags = WRITE_FLUSH | REQ_FLUSH_SEQ; | |
| 320 | + flush_rq->rq_disk = first_rq->rq_disk; | |
| 321 | + flush_rq->end_io = flush_end_io; | |
| 311 | 322 | |
| 312 | − return blk_flush_queue_rq(q->flush_rq, false); | |
| 323 | + return blk_flush_queue_rq(flush_rq, false); | |
| 313 | 324 | } |
| 314 | 325 | |
| 315 | 326 | static void flush_data_end_io(struct request *rq, int error) |
| 316 | 327 | { |
| 317 | 328 | struct request_queue *q = rq->q; |
| 329 | + struct blk_flush_queue *fq = blk_get_flush_queue(q, NULL); | |
| 318 | 330 | |
| 319 | 331 | /* |
| 320 | 332 | * After populating an empty queue, kick it to avoid stall. Read |
| 321 | 333 | * the comment in flush_end_io(). |
| 322 | 334 | */ |
| 323 | − if (blk_flush_complete_seq(rq, REQ_FSEQ_DATA, error)) | |
| 335 | + if (blk_flush_complete_seq(rq, fq, REQ_FSEQ_DATA, error)) | |
| 324 | 336 | blk_run_queue_async(q); |
| 325 | 337 | } |
| 326 | 338 | |
| @@ -328,20 +340,20 @@ static void mq_flush_data_end_io(struct request *rq, int error) · 1 unmodified line | ||
| 328 | 340 | { |
| 329 | 341 | struct request_queue *q = rq->q; |
| 330 | 342 | struct blk_mq_hw_ctx *hctx; |
| 331 | − struct blk_mq_ctx *ctx; | |
| 343 | + struct blk_mq_ctx *ctx = rq->mq_ctx; | |
| 332 | 344 | unsigned long flags; |
| 345 | + struct blk_flush_queue *fq = blk_get_flush_queue(q, ctx); | |
| 333 | 346 | |
| 334 | − ctx = rq->mq_ctx; | |
| 335 | 347 | hctx = q->mq_ops->map_queue(q, ctx->cpu); |
| 336 | 348 | |
| 337 | 349 | /* |
| 338 | 350 | * After populating an empty queue, kick it to avoid stall. Read |
| 339 | 351 | * the comment in flush_end_io(). |
| 340 | 352 | */ |
| 341 | − spin_lock_irqsave(&q->mq_flush_lock, flags); | |
| 342 | − if (blk_flush_complete_seq(rq, REQ_FSEQ_DATA, error)) | |
| 353 | + spin_lock_irqsave(&fq->mq_flush_lock, flags); | |
| 354 | + if (blk_flush_complete_seq(rq, fq, REQ_FSEQ_DATA, error)) | |
| 343 | 355 | blk_mq_run_hw_queue(hctx, true); |
| 344 | − spin_unlock_irqrestore(&q->mq_flush_lock, flags); | |
| 356 | + spin_unlock_irqrestore(&fq->mq_flush_lock, flags); | |
| 345 | 357 | } |
| 346 | 358 | |
| 347 | 359 | /** |
| @@ -361,6 +373,7 @@ void blk_insert_flush(struct request *rq) · 13 unmodified lines | ||
| 361 | 373 | struct request_queue *q = rq->q; |
| 362 | 374 | unsigned int fflags = q->flush_flags; /* may change, cache */ |
| 363 | 375 | unsigned int policy = blk_flush_policy(fflags, rq); |
| 376 | + struct blk_flush_queue *fq = blk_get_flush_queue(q, rq->mq_ctx); | |
| 364 | 377 | |
| 365 | 378 | /* |
| 366 | 379 | * @policy now records what operations need to be done. Adjust |
| @@ -378,7 +391,7 @@ void blk_insert_flush(struct request *rq) · 11 unmodified lines | ||
| 378 | 391 | */ |
| 379 | 392 | if (!policy) { |
| 380 | 393 | if (q->mq_ops) |
| 381 | − blk_mq_end_io(rq, 0); | |
| 394 | + blk_mq_end_request(rq, 0); | |
| 382 | 395 | else |
| 383 | 396 | __blk_end_bidi_request(rq, 0, 0, 0); |
| 384 | 397 | return; |
| @@ -411,14 +424,14 @@ void blk_insert_flush(struct request *rq) · 26 unmodified lines | ||
| 411 | 424 | if (q->mq_ops) { |
| 412 | 425 | rq->end_io = mq_flush_data_end_io; |
| 413 | 426 | |
| 414 | − spin_lock_irq(&q->mq_flush_lock); | |
| 415 | − blk_flush_complete_seq(rq, REQ_FSEQ_ACTIONS & ~policy, 0); | |
| 416 | − spin_unlock_irq(&q->mq_flush_lock); | |
| 427 | + spin_lock_irq(&fq->mq_flush_lock); | |
| 428 | + blk_flush_complete_seq(rq, fq, REQ_FSEQ_ACTIONS & ~policy, 0); | |
| 429 | + spin_unlock_irq(&fq->mq_flush_lock); | |
| 417 | 430 | return; |
| 418 | 431 | } |
| 419 | 432 | rq->end_io = flush_data_end_io; |
| 420 | 433 | |
| 421 | − blk_flush_complete_seq(rq, REQ_FSEQ_ACTIONS & ~policy, 0); | |
| 434 | + blk_flush_complete_seq(rq, fq, REQ_FSEQ_ACTIONS & ~policy, 0); | |
| 422 | 435 | } |
| 423 | 436 | |
| 424 | 437 | /** |
| @@ -474,7 +487,43 @@ int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, · 49 unmodified lines | ||
| 474 | 487 | } |
| 475 | 488 | EXPORT_SYMBOL(blkdev_issue_flush); |
| 476 | 489 | |
| 477 | −void blk_mq_init_flush(struct request_queue *q) | |
| 490 | +struct blk_flush_queue *blk_alloc_flush_queue(struct request_queue *q, | |
| 491 | + int node, int cmd_size) | |
| 492 | +{ | |
| 493 | + struct blk_flush_queue *fq; | |
| 494 | + int rq_sz = sizeof(struct request); | |
| 495 | + | |
| 496 | + fq = kzalloc_node(sizeof(*fq), GFP_KERNEL, node); | |
| 497 | + if (!fq) | |
| 498 | + goto fail; | |
| 499 | + | |
| 500 | + if (q->mq_ops) { | |
| 501 | + spin_lock_init(&fq->mq_flush_lock); | |
| 502 | + rq_sz = round_up(rq_sz + cmd_size, cache_line_size()); | |
| 503 | + } | |
| 504 | + | |
| 505 | + fq->flush_rq = kzalloc_node(rq_sz, GFP_KERNEL, node); | |
| 506 | + if (!fq->flush_rq) | |
| 507 | + goto fail_rq; | |
| 508 | + | |
| 509 | + INIT_LIST_HEAD(&fq->flush_queue[0]); | |
| 510 | + INIT_LIST_HEAD(&fq->flush_queue[1]); | |
| 511 | + INIT_LIST_HEAD(&fq->flush_data_in_flight); | |
| 512 | + | |
| 513 | + return fq; | |
| 514 | + | |
| 515 | + fail_rq: | |
| 516 | + kfree(fq); | |
| 517 | + fail: | |
| 518 | + return NULL; | |
| 519 | +} | |
| 520 | + | |
| 521 | +void blk_free_flush_queue(struct blk_flush_queue *fq) | |
| 478 | 522 | { |
| 479 | − spin_lock_init(&q->mq_flush_lock); | |
| 523 | + /* bio based request queue hasn't flush queue */ | |
| 524 | + if (!fq) | |
| 525 | + return; | |
| 526 | + | |
| 527 | + kfree(fq->flush_rq); | |
| 528 | + kfree(fq); | |
| 480 | 529 | } |
| @@ -154,10 +154,10 @@ int blk_integrity_compare(struct gendisk *gd1, struct gendisk *gd2) · 153 unmodified lines | ||
| 154 | 154 | if (!b1 || !b2) |
| 155 | 155 | return -1; |
| 156 | 156 | |
| 157 | − if (b1->sector_size != b2->sector_size) { | |
| 158 | − printk(KERN_ERR "%s: %s/%s sector sz %u != %u\n", __func__, | |
| 159 | − gd1->disk_name, gd2->disk_name, | |
| 160 | − b1->sector_size, b2->sector_size); | |
| 157 | + if (b1->interval != b2->interval) { | |
| 158 | + pr_err("%s: %s/%s protection interval %u != %u\n", | |
| 159 | + __func__, gd1->disk_name, gd2->disk_name, | |
| 160 | + b1->interval, b2->interval); | |
| 161 | 161 | return -1; |
| 162 | 162 | } |
| 163 | 163 | |
| @@ -186,37 +186,53 @@ int blk_integrity_compare(struct gendisk *gd1, struct gendisk *gd2) · 22 unmodified lines | ||
| 186 | 186 | } |
| 187 | 187 | EXPORT_SYMBOL(blk_integrity_compare); |
| 188 | 188 | |
| 189 | −int blk_integrity_merge_rq(struct request_queue *q, struct request *req, | |
| 190 | − struct request *next) | |
| 189 | +bool blk_integrity_merge_rq(struct request_queue *q, struct request *req, | |
| 190 | + struct request *next) | |
| 191 | 191 | { |
| 192 | − if (blk_integrity_rq(req) != blk_integrity_rq(next)) | |
| 193 | − return -1; | |
| 192 | + if (blk_integrity_rq(req) == 0 && blk_integrity_rq(next) == 0) | |
| 193 | + return true; | |
| 194 | + | |
| 195 | + if (blk_integrity_rq(req) == 0 || blk_integrity_rq(next) == 0) | |
| 196 | + return false; | |
| 197 | + | |
| 198 | + if (bio_integrity(req->bio)->bip_flags != | |
| 199 | + bio_integrity(next->bio)->bip_flags) | |
| 200 | + return false; | |
| 194 | 201 | |
| 195 | 202 | if (req->nr_integrity_segments + next->nr_integrity_segments > |
| 196 | 203 | q->limits.max_integrity_segments) |
| 197 | − return -1; | |
| 204 | + return false; | |
| 198 | 205 | |
| 199 | − return 0; | |
| 206 | + return true; | |
| 200 | 207 | } |
| 201 | 208 | EXPORT_SYMBOL(blk_integrity_merge_rq); |
| 202 | 209 | |
| 203 | −int blk_integrity_merge_bio(struct request_queue *q, struct request *req, | |
| 204 | − struct bio *bio) | |
| 210 | +bool blk_integrity_merge_bio(struct request_queue *q, struct request *req, | |
| 211 | + struct bio *bio) | |
| 205 | 212 | { |
| 206 | 213 | int nr_integrity_segs; |
| 207 | 214 | struct bio *next = bio->bi_next; |
| 208 | 215 | |
| 216 | + if (blk_integrity_rq(req) == 0 && bio_integrity(bio) == NULL) | |
| 217 | + return true; | |
| 218 | + | |
| 219 | + if (blk_integrity_rq(req) == 0 || bio_integrity(bio) == NULL) | |
| 220 | + return false; | |
| 221 | + | |
| 222 | + if (bio_integrity(req->bio)->bip_flags != bio_integrity(bio)->bip_flags) | |
| 223 | + return false; | |
| 224 | + | |
| 209 | 225 | bio->bi_next = NULL; |
| 210 | 226 | nr_integrity_segs = blk_rq_count_integrity_sg(q, bio); |
| 211 | 227 | bio->bi_next = next; |
| 212 | 228 | |
| 213 | 229 | if (req->nr_integrity_segments + nr_integrity_segs > |
| 214 | 230 | q->limits.max_integrity_segments) |
| 215 | − return -1; | |
| 231 | + return false; | |
| 216 | 232 | |
| 217 | 233 | req->nr_integrity_segments += nr_integrity_segs; |
| 218 | 234 | |
| 219 | − return 0; | |
| 235 | + return true; | |
| 220 | 236 | } |
| 221 | 237 | EXPORT_SYMBOL(blk_integrity_merge_bio); |
| 222 | 238 | |
| @@ -269,42 +285,48 @@ static ssize_t integrity_tag_size_show(struct blk_integrity *bi, char *page) · 46 unmodified lines | ||
| 269 | 285 | return sprintf(page, "0\n"); |
| 270 | 286 | } |
| 271 | 287 | |
| 272 | −static ssize_t integrity_read_store(struct blk_integrity *bi, | |
| 273 | − const char *page, size_t count) | |
| 288 | +static ssize_t integrity_verify_store(struct blk_integrity *bi, | |
| 289 | + const char *page, size_t count) | |
| 274 | 290 | { |
| 275 | 291 | char *p = (char *) page; |
| 276 | 292 | unsigned long val = simple_strtoul(p, &p, 10); |
| 277 | 293 | |
| 278 | 294 | if (val) |
| 279 | − bi->flags |= INTEGRITY_FLAG_READ; | |
| 295 | + bi->flags |= BLK_INTEGRITY_VERIFY; | |
| 280 | 296 | else |
| 281 | − bi->flags &= ~INTEGRITY_FLAG_READ; | |
| 297 | + bi->flags &= ~BLK_INTEGRITY_VERIFY; | |
| 282 | 298 | |
| 283 | 299 | return count; |
| 284 | 300 | } |
| 285 | 301 | |
| 286 | −static ssize_t integrity_read_show(struct blk_integrity *bi, char *page) | |
| 302 | +static ssize_t integrity_verify_show(struct blk_integrity *bi, char *page) | |
| 287 | 303 | { |
| 288 | − return sprintf(page, "%d\n", (bi->flags & INTEGRITY_FLAG_READ) != 0); | |
| 304 | + return sprintf(page, "%d\n", (bi->flags & BLK_INTEGRITY_VERIFY) != 0); | |
| 289 | 305 | } |
| 290 | 306 | |
| 291 | −static ssize_t integrity_write_store(struct blk_integrity *bi, | |
| 292 | − const char *page, size_t count) | |
| 307 | +static ssize_t integrity_generate_store(struct blk_integrity *bi, | |
| 308 | + const char *page, size_t count) | |
| 293 | 309 | { |
| 294 | 310 | char *p = (char *) page; |
| 295 | 311 | unsigned long val = simple_strtoul(p, &p, 10); |
| 296 | 312 | |
| 297 | 313 | if (val) |
| 298 | − bi->flags |= INTEGRITY_FLAG_WRITE; | |
| 314 | + bi->flags |= BLK_INTEGRITY_GENERATE; | |
| 299 | 315 | else |
| 300 | − bi->flags &= ~INTEGRITY_FLAG_WRITE; | |
| 316 | + bi->flags &= ~BLK_INTEGRITY_GENERATE; | |
| 301 | 317 | |
| 302 | 318 | return count; |
| 303 | 319 | } |
| 304 | 320 | |
| 305 | −static ssize_t integrity_write_show(struct blk_integrity *bi, char *page) | |
| 321 | +static ssize_t integrity_generate_show(struct blk_integrity *bi, char *page) | |
| 322 | +{ | |
| 323 | + return sprintf(page, "%d\n", (bi->flags & BLK_INTEGRITY_GENERATE) != 0); | |
| 324 | +} | |
| 325 | + | |
| 326 | +static ssize_t integrity_device_show(struct blk_integrity *bi, char *page) | |
| 306 | 327 | { |
| 307 | − return sprintf(page, "%d\n", (bi->flags & INTEGRITY_FLAG_WRITE) != 0); | |
| 328 | + return sprintf(page, "%u\n", | |
| 329 | + (bi->flags & BLK_INTEGRITY_DEVICE_CAPABLE) != 0); | |
| 308 | 330 | } |
| 309 | 331 | |
| 310 | 332 | static struct integrity_sysfs_entry integrity_format_entry = { |
| @@ -317,23 +339,29 @@ static struct integrity_sysfs_entry integrity_tag_size_entry = { · 6 unmodified lines | ||
| 317 | 339 | .show = integrity_tag_size_show, |
| 318 | 340 | }; |
| 319 | 341 | |
| 320 | −static struct integrity_sysfs_entry integrity_read_entry = { | |
| 342 | +static struct integrity_sysfs_entry integrity_verify_entry = { | |
| 321 | 343 | .attr = { .name = "read_verify", .mode = S_IRUGO | S_IWUSR }, |
| 322 | − .show = integrity_read_show, | |
| 323 | − .store = integrity_read_store, | |
| 344 | + .show = integrity_verify_show, | |
| 345 | + .store = integrity_verify_store, | |
| 324 | 346 | }; |
| 325 | 347 | |
| 326 | −static struct integrity_sysfs_entry integrity_write_entry = { | |
| 348 | +static struct integrity_sysfs_entry integrity_generate_entry = { | |
| 327 | 349 | .attr = { .name = "write_generate", .mode = S_IRUGO | S_IWUSR }, |
| 328 | − .show = integrity_write_show, | |
| 329 | − .store = integrity_write_store, | |
| 350 | + .show = integrity_generate_show, | |
| 351 | + .store = integrity_generate_store, | |
| 352 | +}; | |
| 353 | + | |
| 354 | +static struct integrity_sysfs_entry integrity_device_entry = { | |
| 355 | + .attr = { .name = "device_is_integrity_capable", .mode = S_IRUGO }, | |
| 356 | + .show = integrity_device_show, | |
| 330 | 357 | }; |
| 331 | 358 | |
| 332 | 359 | static struct attribute *integrity_attrs[] = { |
| 333 | 360 | &integrity_format_entry.attr, |
| 334 | 361 | &integrity_tag_size_entry.attr, |
| 335 | − &integrity_read_entry.attr, | |
| 336 | − &integrity_write_entry.attr, | |
| 362 | + &integrity_verify_entry.attr, | |
| 363 | + &integrity_generate_entry.attr, | |
| 364 | + &integrity_device_entry.attr, | |
| 337 | 365 | NULL, |
| 338 | 366 | }; |
| 339 | 367 | |
| @@ -406,8 +434,8 @@ int blk_integrity_register(struct gendisk *disk, struct blk_integrity *template) · 66 unmodified lines | ||
| 406 | 434 | |
| 407 | 435 | kobject_uevent(&bi->kobj, KOBJ_ADD); |
| 408 | 436 | |
| 409 | − bi->flags |= INTEGRITY_FLAG_READ | INTEGRITY_FLAG_WRITE; | |
| 410 | − bi->sector_size = queue_logical_block_size(disk->queue); | |
| 437 | + bi->flags |= BLK_INTEGRITY_VERIFY | BLK_INTEGRITY_GENERATE; | |
| 438 | + bi->interval = queue_logical_block_size(disk->queue); | |
| 411 | 439 | disk->integrity = bi; |
| 412 | 440 | } else |
| 413 | 441 | bi = disk->integrity; |
| @@ -418,9 +446,8 @@ int blk_integrity_register(struct gendisk *disk, struct blk_integrity *template) · 4 unmodified lines | ||
| 418 | 446 | bi->generate_fn = template->generate_fn; |
| 419 | 447 | bi->verify_fn = template->verify_fn; |
| 420 | 448 | bi->tuple_size = template->tuple_size; |
| 421 | − bi->set_tag_fn = template->set_tag_fn; | |
| 422 | − bi->get_tag_fn = template->get_tag_fn; | |
| 423 | 449 | bi->tag_size = template->tag_size; |
| 450 | + bi->flags |= template->flags; | |
| 424 | 451 | } else |
| 425 | 452 | bi->name = bi_unsupported_name; |
| 426 | 453 | |
| @@ -97,14 +97,18 @@ void blk_recalc_rq_segments(struct request *rq) · 96 unmodified lines | ||
| 97 | 97 | |
| 98 | 98 | void blk_recount_segments(struct request_queue *q, struct bio *bio) |
| 99 | 99 | { |
| 100 | − if (test_bit(QUEUE_FLAG_NO_SG_MERGE, &q->queue_flags) && | |
| 100 | + bool no_sg_merge = !!test_bit(QUEUE_FLAG_NO_SG_MERGE, | |
| 101 | + &q->queue_flags); | |
| 102 | + | |
| 103 | + if (no_sg_merge && !bio_flagged(bio, BIO_CLONED) && | |
| 101 | 104 | bio->bi_vcnt < queue_max_segments(q)) |
| 102 | 105 | bio->bi_phys_segments = bio->bi_vcnt; |
| 103 | 106 | else { |
| 104 | 107 | struct bio *nxt = bio->bi_next; |
| 105 | 108 | |
| 106 | 109 | bio->bi_next = NULL; |
| 107 | − bio->bi_phys_segments = __blk_recalc_rq_segments(q, bio, false); | |
| 110 | + bio->bi_phys_segments = __blk_recalc_rq_segments(q, bio, | |
| 111 | + no_sg_merge); | |
| 108 | 112 | bio->bi_next = nxt; |
| 109 | 113 | } |
| 110 | 114 | |
| @@ -313,7 +317,7 @@ static inline int ll_new_hw_segment(struct request_queue *q, · 202 unmodified lines | ||
| 313 | 317 | if (req->nr_phys_segments + nr_phys_segs > queue_max_segments(q)) |
| 314 | 318 | goto no_merge; |
| 315 | 319 | |
| 316 | − if (bio_integrity(bio) && blk_integrity_merge_bio(q, req, bio)) | |
| 320 | + if (blk_integrity_merge_bio(q, req, bio) == false) | |
| 317 | 321 | goto no_merge; |
| 318 | 322 | |
| 319 | 323 | /* |
| @@ -410,7 +414,7 @@ static int ll_merge_requests_fn(struct request_queue *q, struct request *req, · 90 unmodified lines | ||
| 410 | 414 | if (total_phys_segments > queue_max_segments(q)) |
| 411 | 415 | return 0; |
| 412 | 416 | |
| 413 | − if (blk_integrity_rq(req) && blk_integrity_merge_rq(q, req, next)) | |
| 417 | + if (blk_integrity_merge_rq(q, req, next) == false) | |
| 414 | 418 | return 0; |
| 415 | 419 | |
| 416 | 420 | /* Merge is OK... */ |
| @@ -590,7 +594,7 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) · 173 unmodified lines | ||
| 590 | 594 | return false; |
| 591 | 595 | |
| 592 | 596 | /* only merge integrity protected bio into ditto rq */ |
| 593 | − if (bio_integrity(bio) != blk_integrity_rq(rq)) | |
| 597 | + if (blk_integrity_merge_bio(rq->q, rq, bio) == false) | |
| 594 | 598 | return false; |
| 595 | 599 | |
| 596 | 600 | /* must be using the same buffer */ |
| @@ -351,15 +351,12 @@ static void bt_clear_tag(struct blk_mq_bitmap_tags *bt, unsigned int tag) · 350 unmodified lines | ||
| 351 | 351 | return; |
| 352 | 352 | |
| 353 | 353 | wait_cnt = atomic_dec_return(&bs->wait_cnt); |
| 354 | + if (unlikely(wait_cnt < 0)) | |
| 355 | + wait_cnt = atomic_inc_return(&bs->wait_cnt); | |
| 354 | 356 | if (wait_cnt == 0) { |
| 355 | −wake: | |
| 356 | 357 | atomic_add(bt->wake_cnt, &bs->wait_cnt); |
| 357 | 358 | bt_index_atomic_inc(&bt->wake_index); |
| 358 | 359 | wake_up(&bs->wait); |
| 359 | − } else if (wait_cnt < 0) { | |
| 360 | − wait_cnt = atomic_inc_return(&bs->wait_cnt); | |
| 361 | − if (!wait_cnt) | |
| 362 | − goto wake; | |
| 363 | 360 | } |
| 364 | 361 | } |
| 365 | 362 | |
| @@ -392,45 +389,37 @@ void blk_mq_put_tag(struct blk_mq_hw_ctx *hctx, unsigned int tag, · 26 unmodified lines | ||
| 392 | 389 | __blk_mq_put_reserved_tag(tags, tag); |
| 393 | 390 | } |
| 394 | 391 | |
| 395 | −static void bt_for_each_free(struct blk_mq_bitmap_tags *bt, | |
| 396 | − unsigned long *free_map, unsigned int off) | |
| 392 | +static void bt_for_each(struct blk_mq_hw_ctx *hctx, | |
| 393 | + struct blk_mq_bitmap_tags *bt, unsigned int off, | |
| 394 | + busy_iter_fn *fn, void *data, bool reserved) | |
| 397 | 395 | { |
| 398 | − int i; | |
| 396 | + struct request *rq; | |
| 397 | + int bit, i; | |
| 399 | 398 | |
| 400 | 399 | for (i = 0; i < bt->map_nr; i++) { |
| 401 | 400 | struct blk_align_bitmap *bm = &bt->map[i]; |
| 402 | − int bit = 0; | |
| 403 | 401 | |
| 404 | − do { | |
| 405 | − bit = find_next_zero_bit(&bm->word, bm->depth, bit); | |
| 406 | − if (bit >= bm->depth) | |
| 407 | − break; | |
| 408 | − | |
| 409 | − __set_bit(bit + off, free_map); | |
| 410 | − bit++; | |
| 411 | − } while (1); | |
| 402 | + for (bit = find_first_bit(&bm->word, bm->depth); | |
| 403 | + bit < bm->depth; | |
| 404 | + bit = find_next_bit(&bm->word, bm->depth, bit + 1)) { | |
| 405 | + rq = blk_mq_tag_to_rq(hctx->tags, off + bit); | |
| 406 | + if (rq->q == hctx->queue) | |
| 407 | + fn(hctx, rq, data, reserved); | |
| 408 | + } | |
| 412 | 409 | |
| 413 | 410 | off += (1 << bt->bits_per_word); |
| 414 | 411 | } |
| 415 | 412 | } |
| 416 | 413 | |
| 417 | −void blk_mq_tag_busy_iter(struct blk_mq_tags *tags, | |
| 418 | − void (*fn)(void *, unsigned long *), void *data) | |
| 414 | +void blk_mq_tag_busy_iter(struct blk_mq_hw_ctx *hctx, busy_iter_fn *fn, | |
| 415 | + void *priv) | |
| 419 | 416 | { |
| 420 | − unsigned long *tag_map; | |
| 421 | − size_t map_size; | |
| 422 | − | |
| 423 | − map_size = ALIGN(tags->nr_tags, BITS_PER_LONG) / BITS_PER_LONG; | |
| 424 | − tag_map = kzalloc(map_size * sizeof(unsigned long), GFP_ATOMIC); | |
| 425 | − if (!tag_map) | |
| 426 | − return; | |
| 417 | + struct blk_mq_tags *tags = hctx->tags; | |
| 427 | 418 | |
| 428 | − bt_for_each_free(&tags->bitmap_tags, tag_map, tags->nr_reserved_tags); | |
| 429 | 419 | if (tags->nr_reserved_tags) |
| 430 | − bt_for_each_free(&tags->breserved_tags, tag_map, 0); | |
| 431 | − | |
| 432 | − fn(data, tag_map); | |
| 433 | − kfree(tag_map); | |
| 420 | + bt_for_each(hctx, &tags->breserved_tags, 0, fn, priv, true); | |
| 421 | + bt_for_each(hctx, &tags->bitmap_tags, tags->nr_reserved_tags, fn, priv, | |
| 422 | + false); | |
| 434 | 423 | } |
| 435 | 424 | EXPORT_SYMBOL(blk_mq_tag_busy_iter); |
| 436 | 425 | |
| @@ -463,8 +452,8 @@ static void bt_update_count(struct blk_mq_bitmap_tags *bt, · 26 unmodified lines | ||
| 463 | 452 | } |
| 464 | 453 | |
| 465 | 454 | bt->wake_cnt = BT_WAIT_BATCH; |
| 466 | − if (bt->wake_cnt > depth / 4) | |
| 467 | − bt->wake_cnt = max(1U, depth / 4); | |
| 455 | + if (bt->wake_cnt > depth / BT_WAIT_QUEUES) | |
| 456 | + bt->wake_cnt = max(1U, depth / BT_WAIT_QUEUES); | |
| 468 | 457 | |
| 469 | 458 | bt->depth = depth; |
| 470 | 459 | } |
| @@ -20,6 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <linux/cache.h> |
| 21 | 21 | #include <linux/sched/sysctl.h> |
| 22 | 22 | #include <linux/delay.h> |
| 23 | +#include <linux/crash_dump.h> | |
| 23 | 24 | |
| 24 | 25 | #include <trace/events/block.h> |
| 25 | 26 | |
| @@ -223,9 +224,11 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, gfp_t gfp, · 197 unmodified lines | ||
| 223 | 224 | struct blk_mq_hw_ctx *hctx; |
| 224 | 225 | struct request *rq; |
| 225 | 226 | struct blk_mq_alloc_data alloc_data; |
| 227 | + int ret; | |
| 226 | 228 | |
| 227 | − if (blk_mq_queue_enter(q)) | |
| 228 | − return NULL; | |
| 229 | + ret = blk_mq_queue_enter(q); | |
| 230 | + if (ret) | |
| 231 | + return ERR_PTR(ret); | |
| 229 | 232 | |
| 230 | 233 | ctx = blk_mq_get_ctx(q); |
| 231 | 234 | hctx = q->mq_ops->map_queue(q, ctx->cpu); |
| @@ -245,6 +248,8 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, gfp_t gfp, · 13 unmodified lines | ||
| 245 | 248 | ctx = alloc_data.ctx; |
| 246 | 249 | } |
| 247 | 250 | blk_mq_put_ctx(ctx); |
| 251 | + if (!rq) | |
| 252 | + return ERR_PTR(-EWOULDBLOCK); | |
| 248 | 253 | return rq; |
| 249 | 254 | } |
| 250 | 255 | EXPORT_SYMBOL(blk_mq_alloc_request); |
| @@ -276,27 +281,7 @@ void blk_mq_free_request(struct request *rq) · 25 unmodified lines | ||
| 276 | 281 | __blk_mq_free_request(hctx, ctx, rq); |
| 277 | 282 | } |
| 278 | 283 | |
| 279 | −/* | |
| 280 | − * Clone all relevant state from a request that has been put on hold in | |
| 281 | − * the flush state machine into the preallocated flush request that hangs | |
| 282 | − * off the request queue. | |
| 283 | − * | |
| 284 | − * For a driver the flush request should be invisible, that's why we are | |
| 285 | − * impersonating the original request here. | |
| 286 | − */ | |
| 287 | −void blk_mq_clone_flush_request(struct request *flush_rq, | |
| 288 | − struct request *orig_rq) | |
| 289 | −{ | |
| 290 | − struct blk_mq_hw_ctx *hctx = | |
| 291 | − orig_rq->q->mq_ops->map_queue(orig_rq->q, orig_rq->mq_ctx->cpu); | |
| 292 | − | |
| 293 | − flush_rq->mq_ctx = orig_rq->mq_ctx; | |
| 294 | − flush_rq->tag = orig_rq->tag; | |
| 295 | − memcpy(blk_mq_rq_to_pdu(flush_rq), blk_mq_rq_to_pdu(orig_rq), | |
| 296 | − hctx->cmd_size); | |
| 297 | −} | |
| 298 | − | |
| 299 | −inline void __blk_mq_end_io(struct request *rq, int error) | |
| 284 | +inline void __blk_mq_end_request(struct request *rq, int error) | |
| 300 | 285 | { |
| 301 | 286 | blk_account_io_done(rq); |
| 302 | 287 | |
| @@ -308,15 +293,15 @@ inline void __blk_mq_end_io(struct request *rq, int error) · 5 unmodified lines | ||
| 308 | 293 | blk_mq_free_request(rq); |
| 309 | 294 | } |
| 310 | 295 | } |
| 311 | −EXPORT_SYMBOL(__blk_mq_end_io); | |
| 296 | +EXPORT_SYMBOL(__blk_mq_end_request); | |
| 312 | 297 | |
| 313 | −void blk_mq_end_io(struct request *rq, int error) | |
| 298 | +void blk_mq_end_request(struct request *rq, int error) | |
| 314 | 299 | { |
| 315 | 300 | if (blk_update_request(rq, error, blk_rq_bytes(rq))) |
| 316 | 301 | BUG(); |
| 317 | − __blk_mq_end_io(rq, error); | |
| 302 | + __blk_mq_end_request(rq, error); | |
| 318 | 303 | } |
| 319 | −EXPORT_SYMBOL(blk_mq_end_io); | |
| 304 | +EXPORT_SYMBOL(blk_mq_end_request); | |
| 320 | 305 | |
| 321 | 306 | static void __blk_mq_complete_request_remote(void *data) |
| 322 | 307 | { |
| @@ -356,7 +341,7 @@ void __blk_mq_complete_request(struct request *rq) · 33 unmodified lines | ||
| 356 | 341 | struct request_queue *q = rq->q; |
| 357 | 342 | |
| 358 | 343 | if (!q->softirq_done_fn) |
| 359 | − blk_mq_end_io(rq, rq->errors); | |
| 344 | + blk_mq_end_request(rq, rq->errors); | |
| 360 | 345 | else |
| 361 | 346 | blk_mq_ipi_complete_request(rq); |
| 362 | 347 | } |
| @@ -380,7 +365,7 @@ void blk_mq_complete_request(struct request *rq) · 17 unmodified lines | ||
| 380 | 365 | } |
| 381 | 366 | EXPORT_SYMBOL(blk_mq_complete_request); |
| 382 | 367 | |
| 383 | −static void blk_mq_start_request(struct request *rq, bool last) | |
| 368 | +void blk_mq_start_request(struct request *rq) | |
| 384 | 369 | { |
| 385 | 370 | struct request_queue *q = rq->q; |
| 386 | 371 | |
| @@ -417,35 +402,24 @@ static void blk_mq_start_request(struct request *rq, bool last) · 30 unmodified lines | ||
| 417 | 402 | */ |
| 418 | 403 | rq->nr_phys_segments++; |
| 419 | 404 | } |
| 420 | − | |
| 421 | − /* | |
| 422 | − * Flag the last request in the series so that drivers know when IO | |
| 423 | − * should be kicked off, if they don't do it on a per-request basis. | |
| 424 | − * | |
| 425 | − * Note: the flag isn't the only condition drivers should do kick off. | |
| 426 | − * If drive is busy, the last request might not have the bit set. | |
| 427 | − */ | |
| 428 | − if (last) | |
| 429 | − rq->cmd_flags |= REQ_END; | |
| 430 | 405 | } |
| 406 | +EXPORT_SYMBOL(blk_mq_start_request); | |
| 431 | 407 | |
| 432 | 408 | static void __blk_mq_requeue_request(struct request *rq) |
| 433 | 409 | { |
| 434 | 410 | struct request_queue *q = rq->q; |
| 435 | 411 | |
| 436 | 412 | trace_block_rq_requeue(q, rq); |
| 437 | − clear_bit(REQ_ATOM_STARTED, &rq->atomic_flags); | |
| 438 | − | |
| 439 | − rq->cmd_flags &= ~REQ_END; | |
| 440 | 413 | |
| 441 | − if (q->dma_drain_size && blk_rq_bytes(rq)) | |
| 442 | − rq->nr_phys_segments--; | |
| 414 | + if (test_and_clear_bit(REQ_ATOM_STARTED, &rq->atomic_flags)) { | |
| 415 | + if (q->dma_drain_size && blk_rq_bytes(rq)) | |
| 416 | + rq->nr_phys_segments--; | |
| 417 | + } | |
| 443 | 418 | } |
| 444 | 419 | |
| 445 | 420 | void blk_mq_requeue_request(struct request *rq) |
| 446 | 421 | { |
| 447 | 422 | __blk_mq_requeue_request(rq); |
| 448 | − blk_clear_rq_complete(rq); | |
| 449 | 423 | |
| 450 | 424 | BUG_ON(blk_queued_rq(rq)); |
| 451 | 425 | blk_mq_add_to_requeue_list(rq, true); |
| @@ -514,78 +488,35 @@ void blk_mq_kick_requeue_list(struct request_queue *q) · 62 unmodified lines | ||
| 514 | 488 | } |
| 515 | 489 | EXPORT_SYMBOL(blk_mq_kick_requeue_list); |
| 516 | 490 | |
| 517 | −static inline bool is_flush_request(struct request *rq, unsigned int tag) | |
| 491 | +static inline bool is_flush_request(struct request *rq, | |
| 492 | + struct blk_flush_queue *fq, unsigned int tag) | |
| 518 | 493 | { |
| 519 | 494 | return ((rq->cmd_flags & REQ_FLUSH_SEQ) && |
| 520 | − rq->q->flush_rq->tag == tag); | |
| 495 | + fq->flush_rq->tag == tag); | |
| 521 | 496 | } |
| 522 | 497 | |
| 523 | 498 | struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag) |
| 524 | 499 | { |
| 525 | 500 | struct request *rq = tags->rqs[tag]; |
| 501 | + /* mq_ctx of flush rq is always cloned from the corresponding req */ | |
| 502 | + struct blk_flush_queue *fq = blk_get_flush_queue(rq->q, rq->mq_ctx); | |
| 526 | 503 | |
| 527 | − if (!is_flush_request(rq, tag)) | |
| 504 | + if (!is_flush_request(rq, fq, tag)) | |
| 528 | 505 | return rq; |
| 529 | 506 | |
| 530 | − return rq->q->flush_rq; | |
| 507 | + return fq->flush_rq; | |
| 531 | 508 | } |
| 532 | 509 | EXPORT_SYMBOL(blk_mq_tag_to_rq); |
| 533 | 510 | |
| 534 | 511 | struct blk_mq_timeout_data { |
| 535 | − struct blk_mq_hw_ctx *hctx; | |
| 536 | − unsigned long *next; | |
| 537 | − unsigned int *next_set; | |
| 512 | + unsigned long next; | |
| 513 | + unsigned int next_set; | |
| 538 | 514 | }; |
| 539 | 515 | |
| 540 | −static void blk_mq_timeout_check(void *__data, unsigned long *free_tags) | |
| 516 | +void blk_mq_rq_timed_out(struct request *req, bool reserved) | |
| 541 | 517 | { |
| 542 | − struct blk_mq_timeout_data *data = __data; | |
| 543 | − struct blk_mq_hw_ctx *hctx = data->hctx; | |
| 544 | − unsigned int tag; | |
| 545 | − | |
| 546 | − /* It may not be in flight yet (this is where | |
| 547 | − * the REQ_ATOMIC_STARTED flag comes in). The requests are | |
| 548 | − * statically allocated, so we know it's always safe to access the | |
| 549 | − * memory associated with a bit offset into ->rqs[]. | |
| 550 | − */ | |
| 551 | − tag = 0; | |
| 552 | − do { | |
| 553 | − struct request *rq; | |
| 554 | − | |
| 555 | − tag = find_next_zero_bit(free_tags, hctx->tags->nr_tags, tag); | |
| 556 | − if (tag >= hctx->tags->nr_tags) | |
| 557 | − break; | |
| 558 | − | |
| 559 | − rq = blk_mq_tag_to_rq(hctx->tags, tag++); | |
| 560 | − if (rq->q != hctx->queue) | |
| 561 | − continue; | |
| 562 | − if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags)) | |
| 563 | − continue; | |
| 564 | − | |
| 565 | − blk_rq_check_expired(rq, data->next, data->next_set); | |
| 566 | − } while (1); | |
| 567 | −} | |
| 568 | − | |
| 569 | −static void blk_mq_hw_ctx_check_timeout(struct blk_mq_hw_ctx *hctx, | |
| 570 | − unsigned long *next, | |
| 571 | − unsigned int *next_set) | |
| 572 | −{ | |
| 573 | − struct blk_mq_timeout_data data = { | |
| 574 | − .hctx = hctx, | |
| 575 | − .next = next, | |
| 576 | − .next_set = next_set, | |
| 577 | − }; | |
| 578 | − | |
| 579 | − /* | |
| 580 | − * Ask the tagging code to iterate busy requests, so we can | |
| 581 | − * check them for timeout. | |
| 582 | − */ | |
| 583 | − blk_mq_tag_busy_iter(hctx->tags, blk_mq_timeout_check, &data); | |
| 584 | −} | |
| 585 | − | |
| 586 | −static enum blk_eh_timer_return blk_mq_rq_timed_out(struct request *rq) | |
| 587 | −{ | |
| 588 | − struct request_queue *q = rq->q; | |
| 518 | + struct blk_mq_ops *ops = req->q->mq_ops; | |
| 519 | + enum blk_eh_timer_return ret = BLK_EH_RESET_TIMER; | |
| 589 | 520 | |
| 590 | 521 | /* |
| 591 | 522 | * We know that complete is set at this point. If STARTED isn't set |
| @@ -596,21 +527,54 @@ static enum blk_eh_timer_return blk_mq_rq_timed_out(struct request *rq) · 4 unmodified lines | ||
| 596 | 527 | * we both flags will get cleared. So check here again, and ignore |
| 597 | 528 | * a timeout event with a request that isn't active. |
| 598 | 529 | */ |
| 599 | − if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags)) | |
| 600 | − return BLK_EH_NOT_HANDLED; | |
| 530 | + if (!test_bit(REQ_ATOM_STARTED, &req->atomic_flags)) | |
| 531 | + return; | |
| 532 | + | |
| 533 | + if (ops->timeout) | |
| 534 | + ret = ops->timeout(req, reserved); | |
| 535 | + | |
| 536 | + switch (ret) { | |
| 537 | + case BLK_EH_HANDLED: | |
| 538 | + __blk_mq_complete_request(req); | |
| 539 | + break; | |
| 540 | + case BLK_EH_RESET_TIMER: | |
| 541 | + blk_add_timer(req); | |
| 542 | + blk_clear_rq_complete(req); | |
| 543 | + break; | |
| 544 | + case BLK_EH_NOT_HANDLED: | |
| 545 | + break; | |
| 546 | + default: | |
| 547 | + printk(KERN_ERR "block: bad eh return: %d\n", ret); | |
| 548 | + break; | |
| 549 | + } | |
| 550 | +} | |
| 551 | + | |
| 552 | +static void blk_mq_check_expired(struct blk_mq_hw_ctx *hctx, | |
| 553 | + struct request *rq, void *priv, bool reserved) | |
| 554 | +{ | |
| 555 | + struct blk_mq_timeout_data *data = priv; | |
| 601 | 556 | |
| 602 | − if (!q->mq_ops->timeout) | |
| 603 | − return BLK_EH_RESET_TIMER; | |
| 557 | + if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags)) | |
| 558 | + return; | |
| 604 | 559 | |
| 605 | − return q->mq_ops->timeout(rq); | |
| 560 | + if (time_after_eq(jiffies, rq->deadline)) { | |
| 561 | + if (!blk_mark_rq_complete(rq)) | |
| 562 | + blk_mq_rq_timed_out(rq, reserved); | |
| 563 | + } else if (!data->next_set || time_after(data->next, rq->deadline)) { | |
| 564 | + data->next = rq->deadline; | |
| 565 | + data->next_set = 1; | |
| 566 | + } | |
| 606 | 567 | } |
| 607 | 568 | |
| 608 | −static void blk_mq_rq_timer(unsigned long data) | |
| 569 | +static void blk_mq_rq_timer(unsigned long priv) | |
| 609 | 570 | { |
| 610 | − struct request_queue *q = (struct request_queue *) data; | |
| 571 | + struct request_queue *q = (struct request_queue *)priv; | |
| 572 | + struct blk_mq_timeout_data data = { | |
| 573 | + .next = 0, | |
| 574 | + .next_set = 0, | |
| 575 | + }; | |
| 611 | 576 | struct blk_mq_hw_ctx *hctx; |
| 612 | − unsigned long next = 0; | |
| 613 | − int i, next_set = 0; | |
| 577 | + int i; | |
| 614 | 578 | |
| 615 | 579 | queue_for_each_hw_ctx(q, hctx, i) { |
| 616 | 580 | /* |
| @@ -620,12 +584,12 @@ static void blk_mq_rq_timer(unsigned long data) · 3 unmodified lines | ||
| 620 | 584 | if (!hctx->nr_ctx || !hctx->tags) |
| 621 | 585 | continue; |
| 622 | 586 | |
| 623 | − blk_mq_hw_ctx_check_timeout(hctx, &next, &next_set); | |
| 587 | + blk_mq_tag_busy_iter(hctx, blk_mq_check_expired, &data); | |
| 624 | 588 | } |
| 625 | 589 | |
| 626 | − if (next_set) { | |
| 627 | − next = blk_rq_timeout(round_jiffies_up(next)); | |
| 628 | − mod_timer(&q->timeout, next); | |
| 590 | + if (data.next_set) { | |
| 591 | + data.next = blk_rq_timeout(round_jiffies_up(data.next)); | |
| 592 | + mod_timer(&q->timeout, data.next); | |
| 629 | 593 | } else { |
| 630 | 594 | queue_for_each_hw_ctx(q, hctx, i) |
| 631 | 595 | blk_mq_tag_idle(hctx); |
| @@ -751,9 +715,7 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx) · 119 unmodified lines | ||
| 751 | 715 | rq = list_first_entry(&rq_list, struct request, queuelist); |
| 752 | 716 | list_del_init(&rq->queuelist); |
| 753 | 717 | |
| 754 | − blk_mq_start_request(rq, list_empty(&rq_list)); | |
| 755 | − | |
| 756 | − ret = q->mq_ops->queue_rq(hctx, rq); | |
| 718 | + ret = q->mq_ops->queue_rq(hctx, rq, list_empty(&rq_list)); | |
| 757 | 719 | switch (ret) { |
| 758 | 720 | case BLK_MQ_RQ_QUEUE_OK: |
| 759 | 721 | queued++; |
| @@ -766,7 +728,7 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx) · 6 unmodified lines | ||
| 766 | 728 | pr_err("blk-mq: bad return on queue: %d\n", ret); |
| 767 | 729 | case BLK_MQ_RQ_QUEUE_ERROR: |
| 768 | 730 | rq->errors = -EIO; |
| 769 | − blk_mq_end_io(rq, rq->errors); | |
| 731 | + blk_mq_end_request(rq, rq->errors); | |
| 770 | 732 | break; |
| 771 | 733 | } |
| 772 | 734 | |
| @@ -1194,14 +1156,13 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio) · 421 unmodified lines | ||
| 1194 | 1156 | int ret; |
| 1195 | 1157 | |
| 1196 | 1158 | blk_mq_bio_to_request(rq, bio); |
| 1197 | − blk_mq_start_request(rq, true); | |
| 1198 | 1159 | |
| 1199 | 1160 | /* |
| 1200 | 1161 | * For OK queue, we are done. For error, kill it. Any other |
| 1201 | 1162 | * error (busy), just add it to our list as we previously |
| 1202 | 1163 | * would have done |
| 1203 | 1164 | */ |
| 1204 | − ret = q->mq_ops->queue_rq(data.hctx, rq); | |
| 1165 | + ret = q->mq_ops->queue_rq(data.hctx, rq, true); | |
| 1205 | 1166 | if (ret == BLK_MQ_RQ_QUEUE_OK) |
| 1206 | 1167 | goto done; |
| 1207 | 1168 | else { |
| @@ -1209,7 +1170,7 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio) · 1 unmodified line | ||
| 1209 | 1170 | |
| 1210 | 1171 | if (ret == BLK_MQ_RQ_QUEUE_ERROR) { |
| 1211 | 1172 | rq->errors = -EIO; |
| 1212 | − blk_mq_end_io(rq, rq->errors); | |
| 1173 | + blk_mq_end_request(rq, rq->errors); | |
| 1213 | 1174 | goto done; |
| 1214 | 1175 | } |
| 1215 | 1176 | } |
| @@ -1531,6 +1492,28 @@ static int blk_mq_hctx_notify(void *data, unsigned long action, · 315 unmodified lines | ||
| 1531 | 1492 | return NOTIFY_OK; |
| 1532 | 1493 | } |
| 1533 | 1494 | |
| 1495 | +static void blk_mq_exit_hctx(struct request_queue *q, | |
| 1496 | + struct blk_mq_tag_set *set, | |
| 1497 | + struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) | |
| 1498 | +{ | |
| 1499 | + unsigned flush_start_tag = set->queue_depth; | |
| 1500 | + | |
| 1501 | + blk_mq_tag_idle(hctx); | |
| 1502 | + | |
| 1503 | + if (set->ops->exit_request) | |
| 1504 | + set->ops->exit_request(set->driver_data, | |
| 1505 | + hctx->fq->flush_rq, hctx_idx, | |
| 1506 | + flush_start_tag + hctx_idx); | |
| 1507 | + | |
| 1508 | + if (set->ops->exit_hctx) | |
| 1509 | + set->ops->exit_hctx(hctx, hctx_idx); | |
| 1510 | + | |
| 1511 | + blk_mq_unregister_cpu_notifier(&hctx->cpu_notifier); | |
| 1512 | + blk_free_flush_queue(hctx->fq); | |
| 1513 | + kfree(hctx->ctxs); | |
| 1514 | + blk_mq_free_bitmap(&hctx->ctx_map); | |
| 1515 | +} | |
| 1516 | + | |
| 1534 | 1517 | static void blk_mq_exit_hw_queues(struct request_queue *q, |
| 1535 | 1518 | struct blk_mq_tag_set *set, int nr_queue) |
| 1536 | 1519 | { |
| @@ -1540,17 +1523,8 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, · 3 unmodified lines | ||
| 1540 | 1523 | queue_for_each_hw_ctx(q, hctx, i) { |
| 1541 | 1524 | if (i == nr_queue) |
| 1542 | 1525 | break; |
| 1543 | − | |
| 1544 | − blk_mq_tag_idle(hctx); | |
| 1545 | − | |
| 1546 | − if (set->ops->exit_hctx) | |
| 1547 | − set->ops->exit_hctx(hctx, i); | |
| 1548 | − | |
| 1549 | − blk_mq_unregister_cpu_notifier(&hctx->cpu_notifier); | |
| 1550 | − kfree(hctx->ctxs); | |
| 1551 | − blk_mq_free_bitmap(&hctx->ctx_map); | |
| 1526 | + blk_mq_exit_hctx(q, set, hctx, i); | |
| 1552 | 1527 | } |
| 1553 | − | |
| 1554 | 1528 | } |
| 1555 | 1529 | |
| 1556 | 1530 | static void blk_mq_free_hw_queues(struct request_queue *q, |
| @@ -1565,53 +1539,88 @@ static void blk_mq_free_hw_queues(struct request_queue *q, · 8 unmodified lines | ||
| 1565 | 1539 | } |
| 1566 | 1540 | } |
| 1567 | 1541 | |
| 1568 | −static int blk_mq_init_hw_queues(struct request_queue *q, | |
| 1569 | − struct blk_mq_tag_set *set) | |
| 1542 | +static int blk_mq_init_hctx(struct request_queue *q, | |
| 1543 | + struct blk_mq_tag_set *set, | |
| 1544 | + struct blk_mq_hw_ctx *hctx, unsigned hctx_idx) | |
| 1570 | 1545 | { |
| 1571 | − struct blk_mq_hw_ctx *hctx; | |
| 1572 | − unsigned int i; | |
| 1546 | + int node; | |
| 1547 | + unsigned flush_start_tag = set->queue_depth; | |
| 1548 | + | |
| 1549 | + node = hctx->numa_node; | |
| 1550 | + if (node == NUMA_NO_NODE) | |
| 1551 | + node = hctx->numa_node = set->numa_node; | |
| 1552 | + | |
| 1553 | + INIT_DELAYED_WORK(&hctx->run_work, blk_mq_run_work_fn); | |
| 1554 | + INIT_DELAYED_WORK(&hctx->delay_work, blk_mq_delay_work_fn); | |
| 1555 | + spin_lock_init(&hctx->lock); | |
| 1556 | + INIT_LIST_HEAD(&hctx->dispatch); | |
| 1557 | + hctx->queue = q; | |
| 1558 | + hctx->queue_num = hctx_idx; | |
| 1559 | + hctx->flags = set->flags; | |
| 1560 | + hctx->cmd_size = set->cmd_size; | |
| 1561 | + | |
| 1562 | + blk_mq_init_cpu_notifier(&hctx->cpu_notifier, | |
| 1563 | + blk_mq_hctx_notify, hctx); | |
| 1564 | + blk_mq_register_cpu_notifier(&hctx->cpu_notifier); | |
| 1565 | + | |
| 1566 | + hctx->tags = set->tags[hctx_idx]; | |
| 1573 | 1567 | |
| 1574 | 1568 | /* |
| 1575 | − * Initialize hardware queues | |
| 1569 | + * Allocate space for all possible cpus to avoid allocation at | |
| 1570 | + * runtime | |
| 1576 | 1571 | */ |
| 1577 | − queue_for_each_hw_ctx(q, hctx, i) { | |
| 1578 | − int node; | |
| 1572 | + hctx->ctxs = kmalloc_node(nr_cpu_ids * sizeof(void *), | |
| 1573 | + GFP_KERNEL, node); | |
| 1574 | + if (!hctx->ctxs) | |
| 1575 | + goto unregister_cpu_notifier; | |
| 1579 | 1576 | |
| 1580 | − node = hctx->numa_node; | |
| 1581 | − if (node == NUMA_NO_NODE) | |
| 1582 | − node = hctx->numa_node = set->numa_node; | |
| 1577 | + if (blk_mq_alloc_bitmap(&hctx->ctx_map, node)) | |
| 1578 | + goto free_ctxs; | |
| 1583 | 1579 | |
| 1584 | − INIT_DELAYED_WORK(&hctx->run_work, blk_mq_run_work_fn); | |
| 1585 | − INIT_DELAYED_WORK(&hctx->delay_work, blk_mq_delay_work_fn); | |
| 1586 | − spin_lock_init(&hctx->lock); | |
| 1587 | − INIT_LIST_HEAD(&hctx->dispatch); | |
| 1588 | − hctx->queue = q; | |
| 1589 | − hctx->queue_num = i; | |
| 1590 | − hctx->flags = set->flags; | |
| 1591 | − hctx->cmd_size = set->cmd_size; | |
| 1580 | + hctx->nr_ctx = 0; | |
| 1592 | 1581 | |
| 1593 | − blk_mq_init_cpu_notifier(&hctx->cpu_notifier, | |
| 1594 | − blk_mq_hctx_notify, hctx); | |
| 1595 | − blk_mq_register_cpu_notifier(&hctx->cpu_notifier); | |
| 1582 | + if (set->ops->init_hctx && | |
| 1583 | + set->ops->init_hctx(hctx, set->driver_data, hctx_idx)) | |
| 1584 | + goto free_bitmap; | |
| 1596 | 1585 | |
| 1597 | − hctx->tags = set->tags[i]; | |
| 1586 | + hctx->fq = blk_alloc_flush_queue(q, hctx->numa_node, set->cmd_size); | |
| 1587 | + if (!hctx->fq) | |
| 1588 | + goto exit_hctx; | |
| 1598 | 1589 | |
| 1599 | − /* | |
| 1600 | − * Allocate space for all possible cpus to avoid allocation at | |
| 1601 | − * runtime | |
| 1602 | − */ | |
| 1603 | − hctx->ctxs = kmalloc_node(nr_cpu_ids * sizeof(void *), | |
| 1604 | − GFP_KERNEL, node); | |
| 1605 | − if (!hctx->ctxs) | |
| 1606 | − break; | |
| 1590 | + if (set->ops->init_request && | |
| 1591 | + set->ops->init_request(set->driver_data, | |
| 1592 | + hctx->fq->flush_rq, hctx_idx, | |
| 1593 | + flush_start_tag + hctx_idx, node)) | |
| 1594 | + goto free_fq; | |
| 1607 | 1595 | |
| 1608 | − if (blk_mq_alloc_bitmap(&hctx->ctx_map, node)) | |
| 1609 | − break; | |
| 1596 | + return 0; | |
| 1610 | 1597 | |
| 1611 | − hctx->nr_ctx = 0; | |
| 1598 | + free_fq: | |
| 1599 | + kfree(hctx->fq); | |
| 1600 | + exit_hctx: | |
| 1601 | + if (set->ops->exit_hctx) | |
| 1602 | + set->ops->exit_hctx(hctx, hctx_idx); | |
| 1603 | + free_bitmap: | |
| 1604 | + blk_mq_free_bitmap(&hctx->ctx_map); | |
| 1605 | + free_ctxs: | |
| 1606 | + kfree(hctx->ctxs); | |
| 1607 | + unregister_cpu_notifier: | |
| 1608 | + blk_mq_unregister_cpu_notifier(&hctx->cpu_notifier); | |
| 1609 | + | |
| 1610 | + return -1; | |
| 1611 | +} | |
| 1612 | + | |
| 1613 | +static int blk_mq_init_hw_queues(struct request_queue *q, | |
| 1614 | + struct blk_mq_tag_set *set) | |
| 1615 | +{ | |
| 1616 | + struct blk_mq_hw_ctx *hctx; | |
| 1617 | + unsigned int i; | |
| 1612 | 1618 | |
| 1613 | − if (set->ops->init_hctx && | |
| 1614 | − set->ops->init_hctx(hctx, set->driver_data, i)) | |
| 1619 | + /* | |
| 1620 | + * Initialize hardware queues | |
| 1621 | + */ | |
| 1622 | + queue_for_each_hw_ctx(q, hctx, i) { | |
| 1623 | + if (blk_mq_init_hctx(q, set, hctx, i)) | |
| 1615 | 1624 | break; |
| 1616 | 1625 | } |
| 1617 | 1626 | |
| @@ -1765,6 +1774,16 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 147 unmodified lines | ||
| 1765 | 1774 | if (!ctx) |
| 1766 | 1775 | return ERR_PTR(-ENOMEM); |
| 1767 | 1776 | |
| 1777 | + /* | |
| 1778 | + * If a crashdump is active, then we are potentially in a very | |
| 1779 | + * memory constrained environment. Limit us to 1 queue and | |
| 1780 | + * 64 tags to prevent using too much memory. | |
| 1781 | + */ | |
| 1782 | + if (is_kdump_kernel()) { | |
| 1783 | + set->nr_hw_queues = 1; | |
| 1784 | + set->queue_depth = min(64U, set->queue_depth); | |
| 1785 | + } | |
| 1786 | + | |
| 1768 | 1787 | hctxs = kmalloc_node(set->nr_hw_queues * sizeof(*hctxs), GFP_KERNEL, |
| 1769 | 1788 | set->numa_node); |
| 1770 | 1789 | |
| @@ -1783,7 +1802,8 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 12 unmodified lines | ||
| 1783 | 1802 | if (!hctxs[i]) |
| 1784 | 1803 | goto err_hctxs; |
| 1785 | 1804 | |
| 1786 | − if (!zalloc_cpumask_var(&hctxs[i]->cpumask, GFP_KERNEL)) | |
| 1805 | + if (!zalloc_cpumask_var_node(&hctxs[i]->cpumask, GFP_KERNEL, | |
| 1806 | + node)) | |
| 1787 | 1807 | goto err_hctxs; |
| 1788 | 1808 | |
| 1789 | 1809 | atomic_set(&hctxs[i]->nr_active, 0); |
| @@ -1830,7 +1850,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 40 unmodified lines | ||
| 1830 | 1850 | else |
| 1831 | 1851 | blk_queue_make_request(q, blk_sq_make_request); |
| 1832 | 1852 | |
| 1833 | − blk_queue_rq_timed_out(q, blk_mq_rq_timed_out); | |
| 1834 | 1853 | if (set->timeout) |
| 1835 | 1854 | blk_queue_rq_timeout(q, set->timeout); |
| 1836 | 1855 | |
| @@ -1842,17 +1861,10 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 5 unmodified lines | ||
| 1842 | 1861 | if (set->ops->complete) |
| 1843 | 1862 | blk_queue_softirq_done(q, set->ops->complete); |
| 1844 | 1863 | |
| 1845 | − blk_mq_init_flush(q); | |
| 1846 | 1864 | blk_mq_init_cpu_queues(q, set->nr_hw_queues); |
| 1847 | 1865 | |
| 1848 | − q->flush_rq = kzalloc(round_up(sizeof(struct request) + | |
| 1849 | − set->cmd_size, cache_line_size()), | |
| 1850 | − GFP_KERNEL); | |
| 1851 | − if (!q->flush_rq) | |
| 1852 | − goto err_hw; | |
| 1853 | − | |
| 1854 | 1866 | if (blk_mq_init_hw_queues(q, set)) |
| 1855 | − goto err_flush_rq; | |
| 1867 | + goto err_hw; | |
| 1856 | 1868 | |
| 1857 | 1869 | mutex_lock(&all_q_mutex); |
| 1858 | 1870 | list_add_tail(&q->all_q_node, &all_q_list); |
| @@ -1864,8 +1876,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 5 unmodified lines | ||
| 1864 | 1876 | |
| 1865 | 1877 | return q; |
| 1866 | 1878 | |
| 1867 | −err_flush_rq: | |
| 1868 | − kfree(q->flush_rq); | |
| 1869 | 1879 | err_hw: |
| 1870 | 1880 | blk_cleanup_queue(q); |
| 1871 | 1881 | err_hctxs: |
| @@ -27,7 +27,6 @@ struct blk_mq_ctx { · 26 unmodified lines | ||
| 27 | 27 | |
| 28 | 28 | void __blk_mq_complete_request(struct request *rq); |
| 29 | 29 | void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async); |
| 30 | −void blk_mq_init_flush(struct request_queue *q); | |
| 31 | 30 | void blk_mq_freeze_queue(struct request_queue *q); |
| 32 | 31 | void blk_mq_free_queue(struct request_queue *q); |
| 33 | 32 | void blk_mq_clone_flush_request(struct request *flush_rq, |
| @@ -60,6 +59,8 @@ extern int blk_mq_hw_queue_to_node(unsigned int *map, unsigned int); · 26 unmodified lines | ||
| 60 | 59 | extern int blk_mq_sysfs_register(struct request_queue *q); |
| 61 | 60 | extern void blk_mq_sysfs_unregister(struct request_queue *q); |
| 62 | 61 | |
| 62 | +extern void blk_mq_rq_timed_out(struct request *req, bool reserved); | |
| 63 | + | |
| 63 | 64 | /* |
| 64 | 65 | * Basic implementation of sparser bitmap, allowing the user to spread |
| 65 | 66 | * the bits over more cachelines. |
| @@ -574,7 +574,7 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, · 573 unmodified lines | ||
| 574 | 574 | bottom = max(b->physical_block_size, b->io_min) + alignment; |
| 575 | 575 | |
| 576 | 576 | /* Verify that top and bottom intervals line up */ |
| 577 | − if (max(top, bottom) & (min(top, bottom) - 1)) { | |
| 577 | + if (max(top, bottom) % min(top, bottom)) { | |
| 578 | 578 | t->misaligned = 1; |
| 579 | 579 | ret = -1; |
| 580 | 580 | } |
| @@ -619,7 +619,7 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, · 38 unmodified lines | ||
| 619 | 619 | |
| 620 | 620 | /* Find lowest common alignment_offset */ |
| 621 | 621 | t->alignment_offset = lcm(t->alignment_offset, alignment) |
| 622 | − & (max(t->physical_block_size, t->io_min) - 1); | |
| 622 | + % max(t->physical_block_size, t->io_min); | |
| 623 | 623 | |
| 624 | 624 | /* Verify that new alignment_offset is on a logical block boundary */ |
| 625 | 625 | if (t->alignment_offset & (t->logical_block_size - 1)) { |
| @@ -519,8 +519,8 @@ static void blk_release_queue(struct kobject *kobj) · 518 unmodified lines | ||
| 519 | 519 | |
| 520 | 520 | if (q->mq_ops) |
| 521 | 521 | blk_mq_free_queue(q); |
| 522 | − | |
| 523 | − kfree(q->flush_rq); | |
| 522 | + else | |
| 523 | + blk_free_flush_queue(q->fq); | |
| 524 | 524 | |
| 525 | 525 | blk_trace_shutdown(q); |
| 526 | 526 | |
| @@ -90,10 +90,7 @@ static void blk_rq_timed_out(struct request *req) · 89 unmodified lines | ||
| 90 | 90 | switch (ret) { |
| 91 | 91 | case BLK_EH_HANDLED: |
| 92 | 92 | /* Can we use req->errors here? */ |
| 93 | − if (q->mq_ops) | |
| 94 | − __blk_mq_complete_request(req); | |
| 95 | − else | |
| 96 | − __blk_complete_request(req); | |
| 93 | + __blk_complete_request(req); | |
| 97 | 94 | break; |
| 98 | 95 | case BLK_EH_RESET_TIMER: |
| 99 | 96 | blk_add_timer(req); |
| @@ -113,7 +110,7 @@ static void blk_rq_timed_out(struct request *req) · 13 unmodified lines | ||
| 113 | 110 | } |
| 114 | 111 | } |
| 115 | 112 | |
| 116 | −void blk_rq_check_expired(struct request *rq, unsigned long *next_timeout, | |
| 113 | +static void blk_rq_check_expired(struct request *rq, unsigned long *next_timeout, | |
| 117 | 114 | unsigned int *next_set) |
| 118 | 115 | { |
| 119 | 116 | if (time_after_eq(jiffies, rq->deadline)) { |
| @@ -162,7 +159,10 @@ void blk_abort_request(struct request *req) · 42 unmodified lines | ||
| 162 | 159 | if (blk_mark_rq_complete(req)) |
| 163 | 160 | return; |
| 164 | 161 | blk_delete_timer(req); |
| 165 | − blk_rq_timed_out(req); | |
| 162 | + if (req->q->mq_ops) | |
| 163 | + blk_mq_rq_timed_out(req, false); | |
| 164 | + else | |
| 165 | + blk_rq_timed_out(req); | |
| 166 | 166 | } |
| 167 | 167 | EXPORT_SYMBOL_GPL(blk_abort_request); |
| 168 | 168 | |
| @@ -190,7 +190,8 @@ void blk_add_timer(struct request *req) · 21 unmodified lines | ||
| 190 | 190 | struct request_queue *q = req->q; |
| 191 | 191 | unsigned long expiry; |
| 192 | 192 | |
| 193 | − if (!q->rq_timed_out_fn) | |
| 193 | + /* blk-mq has its own handler, so we don't need ->rq_timed_out_fn */ | |
| 194 | + if (!q->mq_ops && !q->rq_timed_out_fn) | |
| 194 | 195 | return; |
| 195 | 196 | |
| 196 | 197 | BUG_ON(!list_empty(&req->timeout_list)); |
| @@ -2,6 +2,8 @@ · 1 unmodified line | ||
| 2 | 2 | #define BLK_INTERNAL_H |
| 3 | 3 | |
| 4 | 4 | #include <linux/idr.h> |
| 5 | +#include <linux/blk-mq.h> | |
| 6 | +#include "blk-mq.h" | |
| 5 | 7 | |
| 6 | 8 | /* Amount of time in which a process may batch requests */ |
| 7 | 9 | #define BLK_BATCH_TIME (HZ/50UL) |
| @@ -12,16 +14,44 @@ · 4 unmodified lines | ||
| 12 | 14 | /* Max future timer expiry for timeouts */ |
| 13 | 15 | #define BLK_MAX_TIMEOUT (5 * HZ) |
| 14 | 16 | |
| 17 | +struct blk_flush_queue { | |
| 18 | + unsigned int flush_queue_delayed:1; | |
| 19 | + unsigned int flush_pending_idx:1; | |
| 20 | + unsigned int flush_running_idx:1; | |
| 21 | + unsigned long flush_pending_since; | |
| 22 | + struct list_head flush_queue[2]; | |
| 23 | + struct list_head flush_data_in_flight; | |
| 24 | + struct request *flush_rq; | |
| 25 | + spinlock_t mq_flush_lock; | |
| 26 | +}; | |
| 27 | + | |
| 15 | 28 | extern struct kmem_cache *blk_requestq_cachep; |
| 16 | 29 | extern struct kmem_cache *request_cachep; |
| 17 | 30 | extern struct kobj_type blk_queue_ktype; |
| 18 | 31 | extern struct ida blk_queue_ida; |
| 19 | 32 | |
| 33 | +static inline struct blk_flush_queue *blk_get_flush_queue( | |
| 34 | + struct request_queue *q, struct blk_mq_ctx *ctx) | |
| 35 | +{ | |
| 36 | + struct blk_mq_hw_ctx *hctx; | |
| 37 | + | |
| 38 | + if (!q->mq_ops) | |
| 39 | + return q->fq; | |
| 40 | + | |
| 41 | + hctx = q->mq_ops->map_queue(q, ctx->cpu); | |
| 42 | + | |
| 43 | + return hctx->fq; | |
| 44 | +} | |
| 45 | + | |
| 20 | 46 | static inline void __blk_get_queue(struct request_queue *q) |
| 21 | 47 | { |
| 22 | 48 | kobject_get(&q->kobj); |
| 23 | 49 | } |
| 24 | 50 | |
| 51 | +struct blk_flush_queue *blk_alloc_flush_queue(struct request_queue *q, | |
| 52 | + int node, int cmd_size); | |
| 53 | +void blk_free_flush_queue(struct blk_flush_queue *q); | |
| 54 | + | |
| 25 | 55 | int blk_init_rl(struct request_list *rl, struct request_queue *q, |
| 26 | 56 | gfp_t gfp_mask); |
| 27 | 57 | void blk_exit_rl(struct request_list *rl); |
| @@ -38,8 +68,6 @@ bool __blk_end_bidi_request(struct request *rq, int error, · 10 unmodified lines | ||
| 38 | 68 | unsigned int nr_bytes, unsigned int bidi_bytes); |
| 39 | 69 | |
| 40 | 70 | void blk_rq_timed_out_timer(unsigned long data); |
| 41 | −void blk_rq_check_expired(struct request *rq, unsigned long *next_timeout, | |
| 42 | − unsigned int *next_set); | |
| 43 | 71 | unsigned long blk_rq_timeout(unsigned long timeout); |
| 44 | 72 | void blk_add_timer(struct request *req); |
| 45 | 73 | void blk_delete_timer(struct request *); |
| @@ -88,6 +116,7 @@ void blk_insert_flush(struct request *rq); · 42 unmodified lines | ||
| 88 | 116 | static inline struct request *__elv_next_request(struct request_queue *q) |
| 89 | 117 | { |
| 90 | 118 | struct request *rq; |
| 119 | + struct blk_flush_queue *fq = blk_get_flush_queue(q, NULL); | |
| 91 | 120 | |
| 92 | 121 | while (1) { |
| 93 | 122 | if (!list_empty(&q->queue_head)) { |
| @@ -110,9 +139,9 @@ static inline struct request *__elv_next_request(struct request_queue *q) · 16 unmodified lines | ||
| 110 | 139 | * should be restarted later. Please see flush_end_io() for |
| 111 | 140 | * details. |
| 112 | 141 | */ |
| 113 | − if (q->flush_pending_idx != q->flush_running_idx && | |
| 142 | + if (fq->flush_pending_idx != fq->flush_running_idx && | |
| 114 | 143 | !queue_flush_queueable(q)) { |
| 115 | − q->flush_queue_delayed = 1; | |
| 144 | + fq->flush_queue_delayed = 1; | |
| 116 | 145 | return NULL; |
| 117 | 146 | } |
| 118 | 147 | if (unlikely(blk_queue_bypass(q)) || |
| @@ -270,8 +270,8 @@ bsg_map_hdr(struct bsg_device *bd, struct sg_io_v4 *hdr, fmode_t has_write_perm, · 269 unmodified lines | ||
| 270 | 270 | * map scatter-gather elements separately and string them to request |
| 271 | 271 | */ |
| 272 | 272 | rq = blk_get_request(q, rw, GFP_KERNEL); |
| 273 | − if (!rq) | |
| 274 | − return ERR_PTR(-ENOMEM); | |
| 273 | + if (IS_ERR(rq)) | |
| 274 | + return rq; | |
| 275 | 275 | blk_rq_set_block_pc(rq); |
| 276 | 276 | |
| 277 | 277 | ret = blk_fill_sgv4_hdr_rq(q, rq, hdr, bd, has_write_perm); |
| @@ -285,8 +285,9 @@ bsg_map_hdr(struct bsg_device *bd, struct sg_io_v4 *hdr, fmode_t has_write_perm, · 7 unmodified lines | ||
| 285 | 285 | } |
| 286 | 286 | |
| 287 | 287 | next_rq = blk_get_request(q, READ, GFP_KERNEL); |
| 288 | − if (!next_rq) { | |
| 289 | − ret = -ENOMEM; | |
| 288 | + if (IS_ERR(next_rq)) { | |
| 289 | + ret = PTR_ERR(next_rq); | |
| 290 | + next_rq = NULL; | |
| 290 | 291 | goto out; |
| 291 | 292 | } |
| 292 | 293 | rq->next_rq = next_rq; |
| @@ -299,7 +299,7 @@ struct cfq_io_cq { · 298 unmodified lines | ||
| 299 | 299 | struct cfq_ttime ttime; |
| 300 | 300 | int ioprio; /* the current ioprio */ |
| 301 | 301 | #ifdef CONFIG_CFQ_GROUP_IOSCHED |
| 302 | − uint64_t blkcg_id; /* the current blkcg ID */ | |
| 302 | + uint64_t blkcg_serial_nr; /* the current blkcg serial */ | |
| 303 | 303 | #endif |
| 304 | 304 | }; |
| 305 | 305 | |
| @@ -3547,17 +3547,17 @@ static void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio) · 3241 unmodified lines | ||
| 3547 | 3547 | { |
| 3548 | 3548 | struct cfq_data *cfqd = cic_to_cfqd(cic); |
| 3549 | 3549 | struct cfq_queue *sync_cfqq; |
| 3550 | − uint64_t id; | |
| 3550 | + uint64_t serial_nr; | |
| 3551 | 3551 | |
| 3552 | 3552 | rcu_read_lock(); |
| 3553 | − id = bio_blkcg(bio)->id; | |
| 3553 | + serial_nr = bio_blkcg(bio)->css.serial_nr; | |
| 3554 | 3554 | rcu_read_unlock(); |
| 3555 | 3555 | |
| 3556 | 3556 | /* |
| 3557 | 3557 | * Check whether blkcg has changed. The condition may trigger |
| 3558 | 3558 | * spuriously on a newly created cic but there's no harm. |
| 3559 | 3559 | */ |
| 3560 | − if (unlikely(!cfqd) || likely(cic->blkcg_id == id)) | |
| 3560 | + if (unlikely(!cfqd) || likely(cic->blkcg_serial_nr == serial_nr)) | |
| 3561 | 3561 | return; |
| 3562 | 3562 | |
| 3563 | 3563 | sync_cfqq = cic_to_cfqq(cic, 1); |
| @@ -3571,7 +3571,7 @@ static void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio) · 7 unmodified lines | ||
| 3571 | 3571 | cfq_put_queue(sync_cfqq); |
| 3572 | 3572 | } |
| 3573 | 3573 | |
| 3574 | − cic->blkcg_id = id; | |
| 3574 | + cic->blkcg_serial_nr = serial_nr; | |
| 3575 | 3575 | } |
| 3576 | 3576 | #else |
| 3577 | 3577 | static inline void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio) { } |
| @@ -709,8 +709,6 @@ long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg) · 708 unmodified lines | ||
| 709 | 709 | if (!arg) |
| 710 | 710 | return -EINVAL; |
| 711 | 711 | bdi = blk_get_backing_dev_info(bdev); |
| 712 | − if (bdi == NULL) | |
| 713 | − return -ENOTTY; | |
| 714 | 712 | return compat_put_long(arg, |
| 715 | 713 | (bdi->ra_pages * PAGE_CACHE_SIZE) / 512); |
| 716 | 714 | case BLKROGET: /* compatible */ |
| @@ -731,8 +729,6 @@ long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg) · 14 unmodified lines | ||
| 731 | 729 | if (!capable(CAP_SYS_ADMIN)) |
| 732 | 730 | return -EACCES; |
| 733 | 731 | bdi = blk_get_backing_dev_info(bdev); |
| 734 | − if (bdi == NULL) | |
| 735 | − return -ENOTTY; | |
| 736 | 732 | bdi->ra_pages = (arg * 512) / PAGE_CACHE_SIZE; |
| 737 | 733 | return 0; |
| 738 | 734 | case BLKGETSIZE: |
| @@ -356,8 +356,6 @@ int blkdev_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd, · 355 unmodified lines | ||
| 356 | 356 | if (!arg) |
| 357 | 357 | return -EINVAL; |
| 358 | 358 | bdi = blk_get_backing_dev_info(bdev); |
| 359 | − if (bdi == NULL) | |
| 360 | − return -ENOTTY; | |
| 361 | 359 | return put_long(arg, (bdi->ra_pages * PAGE_CACHE_SIZE) / 512); |
| 362 | 360 | case BLKROGET: |
| 363 | 361 | return put_int(arg, bdev_read_only(bdev) != 0); |
| @@ -386,8 +384,6 @@ int blkdev_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd, · 22 unmodified lines | ||
| 386 | 384 | if(!capable(CAP_SYS_ADMIN)) |
| 387 | 385 | return -EACCES; |
| 388 | 386 | bdi = blk_get_backing_dev_info(bdev); |
| 389 | − if (bdi == NULL) | |
| 390 | − return -ENOTTY; | |
| 391 | 387 | bdi->ra_pages = (arg * 512) / PAGE_CACHE_SIZE; |
| 392 | 388 | return 0; |
| 393 | 389 | case BLKBSZSET: |
| @@ -81,7 +81,7 @@ int mac_partition(struct parsed_partitions *state) · 80 unmodified lines | ||
| 81 | 81 | be32_to_cpu(part->start_block) * (secsize/512), |
| 82 | 82 | be32_to_cpu(part->block_count) * (secsize/512)); |
| 83 | 83 | |
| 84 | − if (!strnicmp(part->type, "Linux_RAID", 10)) | |
| 84 | + if (!strncasecmp(part->type, "Linux_RAID", 10)) | |
| 85 | 85 | state->parts[slot].flags = ADDPART_FLAG_RAID; |
| 86 | 86 | #ifdef CONFIG_PPC_PMAC |
| 87 | 87 | /* |
| @@ -100,7 +100,7 @@ int mac_partition(struct parsed_partitions *state) · 12 unmodified lines | ||
| 100 | 100 | goodness++; |
| 101 | 101 | |
| 102 | 102 | if (strcasecmp(part->type, "Apple_UNIX_SVR2") == 0 |
| 103 | − || (strnicmp(part->type, "Linux", 5) == 0 | |
| 103 | + || (strncasecmp(part->type, "Linux", 5) == 0 | |
| 104 | 104 | && strcasecmp(part->type, "Linux_swap") != 0)) { |
| 105 | 105 | int i, l; |
| 106 | 106 | |
| @@ -109,13 +109,13 @@ int mac_partition(struct parsed_partitions *state) · 2 unmodified lines | ||
| 109 | 109 | if (strcmp(part->name, "/") == 0) |
| 110 | 110 | goodness++; |
| 111 | 111 | for (i = 0; i <= l - 4; ++i) { |
| 112 | − if (strnicmp(part->name + i, "root", | |
| 112 | + if (strncasecmp(part->name + i, "root", | |
| 113 | 113 | 4) == 0) { |
| 114 | 114 | goodness += 2; |
| 115 | 115 | break; |
| 116 | 116 | } |
| 117 | 117 | } |
| 118 | − if (strnicmp(part->name, "swap", 4) == 0) | |
| 118 | + if (strncasecmp(part->name, "swap", 4) == 0) | |
| 119 | 119 | goodness--; |
| 120 | 120 | } |
| 121 | 121 | |
| @@ -316,8 +316,8 @@ static int sg_io(struct request_queue *q, struct gendisk *bd_disk, · 315 unmodified lines | ||
| 316 | 316 | |
| 317 | 317 | ret = -ENOMEM; |
| 318 | 318 | rq = blk_get_request(q, writing ? WRITE : READ, GFP_KERNEL); |
| 319 | − if (!rq) | |
| 320 | − goto out; | |
| 319 | + if (IS_ERR(rq)) | |
| 320 | + return PTR_ERR(rq); | |
| 321 | 321 | blk_rq_set_block_pc(rq); |
| 322 | 322 | |
| 323 | 323 | if (hdr->cmd_len > BLK_MAX_CDB) { |
| @@ -387,7 +387,6 @@ out_free_cdb: · 63 unmodified lines | ||
| 387 | 387 | kfree(rq->cmd); |
| 388 | 388 | out_put_request: |
| 389 | 389 | blk_put_request(rq); |
| 390 | −out: | |
| 391 | 390 | return ret; |
| 392 | 391 | } |
| 393 | 392 | |
| @@ -457,8 +456,8 @@ int sg_scsi_ioctl(struct request_queue *q, struct gendisk *disk, fmode_t mode, · 63 unmodified lines | ||
| 457 | 456 | } |
| 458 | 457 | |
| 459 | 458 | rq = blk_get_request(q, in_len ? WRITE : READ, __GFP_WAIT); |
| 460 | − if (!rq) { | |
| 461 | − err = -ENOMEM; | |
| 459 | + if (IS_ERR(rq)) { | |
| 460 | + err = PTR_ERR(rq); | |
| 462 | 461 | goto error; |
| 463 | 462 | } |
| 464 | 463 | blk_rq_set_block_pc(rq); |
| @@ -548,6 +547,8 @@ static int __blk_send_generic(struct request_queue *q, struct gendisk *bd_disk, · 83 unmodified lines | ||
| 548 | 547 | int err; |
| 549 | 548 | |
| 550 | 549 | rq = blk_get_request(q, WRITE, __GFP_WAIT); |
| 550 | + if (IS_ERR(rq)) | |
| 551 | + return PTR_ERR(rq); | |
| 551 | 552 | blk_rq_set_block_pc(rq); |
| 552 | 553 | rq->timeout = BLK_DEFAULT_SG_TIMEOUT; |
| 553 | 554 | rq->cmd[0] = cmd; |
| @@ -0,0 +1,197 @@ | ||
| 1 | +/* | |
| 2 | + * t10_pi.c - Functions for generating and verifying T10 Protection | |
| 3 | + * Information. | |
| 4 | + * | |
| 5 | + * Copyright (C) 2007, 2008, 2014 Oracle Corporation | |
| 6 | + * Written by: Martin K. Petersen <martin.petersen@oracle.com> | |
| 7 | + * | |
| 8 | + * This program is free software; you can redistribute it and/or | |
| 9 | + * modify it under the terms of the GNU General Public License version | |
| 10 | + * 2 as published by the Free Software Foundation. | |
| 11 | + * | |
| 12 | + * This program is distributed in the hope that it will be useful, but | |
| 13 | + * WITHOUT ANY WARRANTY; without even the implied warranty of | |
| 14 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU | |
| 15 | + * General Public License for more details. | |
| 16 | + * | |
| 17 | + * You should have received a copy of the GNU General Public License | |
| 18 | + * along with this program; see the file COPYING. If not, write to | |
| 19 | + * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, | |
| 20 | + * USA. | |
| 21 | + * | |
| 22 | + */ | |
| 23 | + | |
| 24 | +#include <linux/t10-pi.h> | |
| 25 | +#include <linux/blkdev.h> | |
| 26 | +#include <linux/crc-t10dif.h> | |
| 27 | +#include <net/checksum.h> | |
| 28 | + | |
| 29 | +typedef __be16 (csum_fn) (void *, unsigned int); | |
| 30 | + | |
| 31 | +static const __be16 APP_ESCAPE = (__force __be16) 0xffff; | |
| 32 | +static const __be32 REF_ESCAPE = (__force __be32) 0xffffffff; | |
| 33 | + | |
| 34 | +static __be16 t10_pi_crc_fn(void *data, unsigned int len) | |
| 35 | +{ | |
| 36 | + return cpu_to_be16(crc_t10dif(data, len)); | |
| 37 | +} | |
| 38 | + | |
| 39 | +static __be16 t10_pi_ip_fn(void *data, unsigned int len) | |
| 40 | +{ | |
| 41 | + return (__force __be16)ip_compute_csum(data, len); | |
| 42 | +} | |
| 43 | + | |
| 44 | +/* | |
| 45 | + * Type 1 and Type 2 protection use the same format: 16 bit guard tag, | |
| 46 | + * 16 bit app tag, 32 bit reference tag. Type 3 does not define the ref | |
| 47 | + * tag. | |
| 48 | + */ | |
| 49 | +static int t10_pi_generate(struct blk_integrity_iter *iter, csum_fn *fn, | |
| 50 | + unsigned int type) | |
| 51 | +{ | |
| 52 | + unsigned int i; | |
| 53 | + | |
| 54 | + for (i = 0 ; i < iter->data_size ; i += iter->interval) { | |
| 55 | + struct t10_pi_tuple *pi = iter->prot_buf; | |
| 56 | + | |
| 57 | + pi->guard_tag = fn(iter->data_buf, iter->interval); | |
| 58 | + pi->app_tag = 0; | |
| 59 | + | |
| 60 | + if (type == 1) | |
| 61 | + pi->ref_tag = cpu_to_be32(lower_32_bits(iter->seed)); | |
| 62 | + else | |
| 63 | + pi->ref_tag = 0; | |
| 64 | + | |
| 65 | + iter->data_buf += iter->interval; | |
| 66 | + iter->prot_buf += sizeof(struct t10_pi_tuple); | |
| 67 | + iter->seed++; | |
| 68 | + } | |
| 69 | + | |
| 70 | + return 0; | |
| 71 | +} | |
| 72 | + | |
| 73 | +static int t10_pi_verify(struct blk_integrity_iter *iter, csum_fn *fn, | |
| 74 | + unsigned int type) | |
| 75 | +{ | |
| 76 | + unsigned int i; | |
| 77 | + | |
| 78 | + for (i = 0 ; i < iter->data_size ; i += iter->interval) { | |
| 79 | + struct t10_pi_tuple *pi = iter->prot_buf; | |
| 80 | + __be16 csum; | |
| 81 | + | |
| 82 | + switch (type) { | |
| 83 | + case 1: | |
| 84 | + case 2: | |
| 85 | + if (pi->app_tag == APP_ESCAPE) | |
| 86 | + goto next; | |
| 87 | + | |
| 88 | + if (be32_to_cpu(pi->ref_tag) != | |
| 89 | + lower_32_bits(iter->seed)) { | |
| 90 | + pr_err("%s: ref tag error at location %llu " \ | |
| 91 | + "(rcvd %u)\n", iter->disk_name, | |
| 92 | + (unsigned long long) | |
| 93 | + iter->seed, be32_to_cpu(pi->ref_tag)); | |
| 94 | + return -EILSEQ; | |
| 95 | + } | |
| 96 | + break; | |
| 97 | + case 3: | |
| 98 | + if (pi->app_tag == APP_ESCAPE && | |
| 99 | + pi->ref_tag == REF_ESCAPE) | |
| 100 | + goto next; | |
| 101 | + break; | |
| 102 | + } | |
| 103 | + | |
| 104 | + csum = fn(iter->data_buf, iter->interval); | |
| 105 | + | |
| 106 | + if (pi->guard_tag != csum) { | |
| 107 | + pr_err("%s: guard tag error at sector %llu " \ | |
| 108 | + "(rcvd %04x, want %04x)\n", iter->disk_name, | |
| 109 | + (unsigned long long)iter->seed, | |
| 110 | + be16_to_cpu(pi->guard_tag), be16_to_cpu(csum)); | |
| 111 | + return -EILSEQ; | |
| 112 | + } | |
| 113 | + | |
| 114 | +next: | |
| 115 | + iter->data_buf += iter->interval; | |
| 116 | + iter->prot_buf += sizeof(struct t10_pi_tuple); | |
| 117 | + iter->seed++; | |
| 118 | + } | |
| 119 | + | |
| 120 | + return 0; | |
| 121 | +} | |
| 122 | + | |
| 123 | +static int t10_pi_type1_generate_crc(struct blk_integrity_iter *iter) | |
| 124 | +{ | |
| 125 | + return t10_pi_generate(iter, t10_pi_crc_fn, 1); | |
| 126 | +} | |
| 127 | + | |
| 128 | +static int t10_pi_type1_generate_ip(struct blk_integrity_iter *iter) | |
| 129 | +{ | |
| 130 | + return t10_pi_generate(iter, t10_pi_ip_fn, 1); | |
| 131 | +} | |
| 132 | + | |
| 133 | +static int t10_pi_type1_verify_crc(struct blk_integrity_iter *iter) | |
| 134 | +{ | |
| 135 | + return t10_pi_verify(iter, t10_pi_crc_fn, 1); | |
| 136 | +} | |
| 137 | + | |
| 138 | +static int t10_pi_type1_verify_ip(struct blk_integrity_iter *iter) | |
| 139 | +{ | |
| 140 | + return t10_pi_verify(iter, t10_pi_ip_fn, 1); | |
| 141 | +} | |
| 142 | + | |
| 143 | +static int t10_pi_type3_generate_crc(struct blk_integrity_iter *iter) | |
| 144 | +{ | |
| 145 | + return t10_pi_generate(iter, t10_pi_crc_fn, 3); | |
| 146 | +} | |
| 147 | + | |
| 148 | +static int t10_pi_type3_generate_ip(struct blk_integrity_iter *iter) | |
| 149 | +{ | |
| 150 | + return t10_pi_generate(iter, t10_pi_ip_fn, 3); | |
| 151 | +} | |
| 152 | + | |
| 153 | +static int t10_pi_type3_verify_crc(struct blk_integrity_iter *iter) | |
| 154 | +{ | |
| 155 | + return t10_pi_verify(iter, t10_pi_crc_fn, 3); | |
| 156 | +} | |
| 157 | + | |
| 158 | +static int t10_pi_type3_verify_ip(struct blk_integrity_iter *iter) | |
| 159 | +{ | |
| 160 | + return t10_pi_verify(iter, t10_pi_ip_fn, 3); | |
| 161 | +} | |
| 162 | + | |
| 163 | +struct blk_integrity t10_pi_type1_crc = { | |
| 164 | + .name = "T10-DIF-TYPE1-CRC", | |
| 165 | + .generate_fn = t10_pi_type1_generate_crc, | |
| 166 | + .verify_fn = t10_pi_type1_verify_crc, | |
| 167 | + .tuple_size = sizeof(struct t10_pi_tuple), | |
| 168 | + .tag_size = 0, | |
| 169 | +}; | |
| 170 | +EXPORT_SYMBOL(t10_pi_type1_crc); | |
| 171 | + | |
| 172 | +struct blk_integrity t10_pi_type1_ip = { | |
| 173 | + .name = "T10-DIF-TYPE1-IP", | |
| 174 | + .generate_fn = t10_pi_type1_generate_ip, | |
| 175 | + .verify_fn = t10_pi_type1_verify_ip, | |
| 176 | + .tuple_size = sizeof(struct t10_pi_tuple), | |
| 177 | + .tag_size = 0, | |
| 178 | +}; | |
| 179 | +EXPORT_SYMBOL(t10_pi_type1_ip); | |
| 180 | + | |
| 181 | +struct blk_integrity t10_pi_type3_crc = { | |
| 182 | + .name = "T10-DIF-TYPE3-CRC", | |
| 183 | + .generate_fn = t10_pi_type3_generate_crc, | |
| 184 | + .verify_fn = t10_pi_type3_verify_crc, | |
| 185 | + .tuple_size = sizeof(struct t10_pi_tuple), | |
| 186 | + .tag_size = 0, | |
| 187 | +}; | |
| 188 | +EXPORT_SYMBOL(t10_pi_type3_crc); | |
| 189 | + | |
| 190 | +struct blk_integrity t10_pi_type3_ip = { | |
| 191 | + .name = "T10-DIF-TYPE3-IP", | |
| 192 | + .generate_fn = t10_pi_type3_generate_ip, | |
| 193 | + .verify_fn = t10_pi_type3_verify_ip, | |
| 194 | + .tuple_size = sizeof(struct t10_pi_tuple), | |
| 195 | + .tag_size = 0, | |
| 196 | +}; | |
| 197 | +EXPORT_SYMBOL(t10_pi_type3_ip); | |
| @@ -247,7 +247,7 @@ static void mtip_async_complete(struct mtip_port *port, · 246 unmodified lines | ||
| 247 | 247 | if (unlikely(cmd->unaligned)) |
| 248 | 248 | up(&port->cmd_slot_unal); |
| 249 | 249 | |
| 250 | − blk_mq_end_io(rq, status ? -EIO : 0); | |
| 250 | + blk_mq_end_request(rq, status ? -EIO : 0); | |
| 251 | 251 | } |
| 252 | 252 | |
| 253 | 253 | /* |
| @@ -3739,7 +3739,7 @@ static int mtip_submit_request(struct blk_mq_hw_ctx *hctx, struct request *rq) · 3485 unmodified lines | ||
| 3739 | 3739 | int err; |
| 3740 | 3740 | |
| 3741 | 3741 | err = mtip_send_trim(dd, blk_rq_pos(rq), blk_rq_sectors(rq)); |
| 3742 | − blk_mq_end_io(rq, err); | |
| 3742 | + blk_mq_end_request(rq, err); | |
| 3743 | 3743 | return 0; |
| 3744 | 3744 | } |
| 3745 | 3745 | |
| @@ -3775,13 +3775,16 @@ static bool mtip_check_unal_depth(struct blk_mq_hw_ctx *hctx, · 29 unmodified lines | ||
| 3775 | 3775 | return false; |
| 3776 | 3776 | } |
| 3777 | 3777 | |
| 3778 | −static int mtip_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq) | |
| 3778 | +static int mtip_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq, | |
| 3779 | + bool last) | |
| 3779 | 3780 | { |
| 3780 | 3781 | int ret; |
| 3781 | 3782 | |
| 3782 | 3783 | if (unlikely(mtip_check_unal_depth(hctx, rq))) |
| 3783 | 3784 | return BLK_MQ_RQ_QUEUE_BUSY; |
| 3784 | 3785 | |
| 3786 | + blk_mq_start_request(rq); | |
| 3787 | + | |
| 3785 | 3788 | ret = mtip_submit_request(hctx, rq); |
| 3786 | 3789 | if (likely(!ret)) |
| 3787 | 3790 | return BLK_MQ_RQ_QUEUE_OK; |
| @@ -177,7 +177,7 @@ static void end_cmd(struct nullb_cmd *cmd) · 176 unmodified lines | ||
| 177 | 177 | { |
| 178 | 178 | switch (queue_mode) { |
| 179 | 179 | case NULL_Q_MQ: |
| 180 | − blk_mq_end_io(cmd->rq, 0); | |
| 180 | + blk_mq_end_request(cmd->rq, 0); | |
| 181 | 181 | return; |
| 182 | 182 | case NULL_Q_RQ: |
| 183 | 183 | INIT_LIST_HEAD(&cmd->rq->queuelist); |
| @@ -313,13 +313,16 @@ static void null_request_fn(struct request_queue *q) · 129 unmodified lines | ||
| 313 | 313 | } |
| 314 | 314 | } |
| 315 | 315 | |
| 316 | −static int null_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq) | |
| 316 | +static int null_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq, | |
| 317 | + bool last) | |
| 317 | 318 | { |
| 318 | 319 | struct nullb_cmd *cmd = blk_mq_rq_to_pdu(rq); |
| 319 | 320 | |
| 320 | 321 | cmd->rq = rq; |
| 321 | 322 | cmd->nq = hctx->driver_data; |
| 322 | 323 | |
| 324 | + blk_mq_start_request(rq); | |
| 325 | + | |
| 323 | 326 | null_handle_cmd(cmd); |
| 324 | 327 | return BLK_MQ_RQ_QUEUE_OK; |
| 325 | 328 | } |
| @@ -722,6 +722,8 @@ static int pd_special_command(struct pd_unit *disk, · 721 unmodified lines | ||
| 722 | 722 | int err = 0; |
| 723 | 723 | |
| 724 | 724 | rq = blk_get_request(disk->gd->queue, READ, __GFP_WAIT); |
| 725 | + if (IS_ERR(rq)) | |
| 726 | + return PTR_ERR(rq); | |
| 725 | 727 | |
| 726 | 728 | rq->cmd_type = REQ_TYPE_SPECIAL; |
| 727 | 729 | rq->special = func; |
| @@ -704,6 +704,8 @@ static int pkt_generic_packet(struct pktcdvd_device *pd, struct packet_command * · 703 unmodified lines | ||
| 704 | 704 | |
| 705 | 705 | rq = blk_get_request(q, (cgc->data_direction == CGC_DATA_WRITE) ? |
| 706 | 706 | WRITE : READ, __GFP_WAIT); |
| 707 | + if (IS_ERR(rq)) | |
| 708 | + return PTR_ERR(rq); | |
| 707 | 709 | blk_rq_set_block_pc(rq); |
| 708 | 710 | |
| 709 | 711 | if (cgc->buflen) { |
| @@ -568,7 +568,7 @@ static struct carm_request *carm_get_special(struct carm_host *host) · 567 unmodified lines | ||
| 568 | 568 | return NULL; |
| 569 | 569 | |
| 570 | 570 | rq = blk_get_request(host->oob_q, WRITE /* bogus */, GFP_KERNEL); |
| 571 | − if (!rq) { | |
| 571 | + if (IS_ERR(rq)) { | |
| 572 | 572 | spin_lock_irqsave(&host->lock, flags); |
| 573 | 573 | carm_put_request(host, crq); |
| 574 | 574 | spin_unlock_irqrestore(&host->lock, flags); |
| @@ -129,7 +129,7 @@ static inline void virtblk_request_done(struct request *req) · 128 unmodified lines | ||
| 129 | 129 | req->errors = (error != 0); |
| 130 | 130 | } |
| 131 | 131 | |
| 132 | − blk_mq_end_io(req, error); | |
| 132 | + blk_mq_end_request(req, error); | |
| 133 | 133 | } |
| 134 | 134 | |
| 135 | 135 | static void virtblk_done(struct virtqueue *vq) |
| @@ -158,14 +158,14 @@ static void virtblk_done(struct virtqueue *vq) · 22 unmodified lines | ||
| 158 | 158 | spin_unlock_irqrestore(&vblk->vqs[qid].lock, flags); |
| 159 | 159 | } |
| 160 | 160 | |
| 161 | −static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req) | |
| 161 | +static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req, | |
| 162 | + bool last) | |
| 162 | 163 | { |
| 163 | 164 | struct virtio_blk *vblk = hctx->queue->queuedata; |
| 164 | 165 | struct virtblk_req *vbr = blk_mq_rq_to_pdu(req); |
| 165 | 166 | unsigned long flags; |
| 166 | 167 | unsigned int num; |
| 167 | 168 | int qid = hctx->queue_num; |
| 168 | − const bool last = (req->cmd_flags & REQ_END) != 0; | |
| 169 | 169 | int err; |
| 170 | 170 | bool notify = false; |
| 171 | 171 | |
| @@ -199,6 +199,8 @@ static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req) · 27 unmodified lines | ||
| 199 | 199 | } |
| 200 | 200 | } |
| 201 | 201 | |
| 202 | + blk_mq_start_request(req); | |
| 203 | + | |
| 202 | 204 | num = blk_rq_map_sg(hctx->queue, vbr->req, vbr->sg); |
| 203 | 205 | if (num) { |
| 204 | 206 | if (rq_data_dir(vbr->req) == WRITE) |
| @@ -2180,8 +2180,8 @@ static int cdrom_read_cdda_bpc(struct cdrom_device_info *cdi, __u8 __user *ubuf, · 2179 unmodified lines | ||
| 2180 | 2180 | len = nr * CD_FRAMESIZE_RAW; |
| 2181 | 2181 | |
| 2182 | 2182 | rq = blk_get_request(q, READ, GFP_KERNEL); |
| 2183 | − if (!rq) { | |
| 2184 | − ret = -ENOMEM; | |
| 2183 | + if (IS_ERR(rq)) { | |
| 2184 | + ret = PTR_ERR(rq); | |
| 2185 | 2185 | break; |
| 2186 | 2186 | } |
| 2187 | 2187 | blk_rq_set_block_pc(rq); |
| @@ -46,7 +46,7 @@ static void issue_park_cmd(ide_drive_t *drive, unsigned long timeout) · 45 unmodified lines | ||
| 46 | 46 | * timeout has expired, so power management will be reenabled. |
| 47 | 47 | */ |
| 48 | 48 | rq = blk_get_request(q, READ, GFP_NOWAIT); |
| 49 | − if (unlikely(!rq)) | |
| 49 | + if (IS_ERR(rq)) | |
| 50 | 50 | goto out; |
| 51 | 51 | |
| 52 | 52 | rq->cmd[0] = REQ_UNPARK_HEADS; |
| @@ -73,7 +73,6 @@ comment "SCSI support type (disk, tape, CD-ROM)" · 72 unmodified lines | ||
| 73 | 73 | config BLK_DEV_SD |
| 74 | 74 | tristate "SCSI disk support" |
| 75 | 75 | depends on SCSI |
| 76 | − select CRC_T10DIF if BLK_DEV_INTEGRITY | |
| 77 | 76 | ---help--- |
| 78 | 77 | If you want to use SCSI hard disks, Fibre Channel disks, |
| 79 | 78 | Serial ATA (SATA) or Parallel ATA (PATA) hard disks, |
| @@ -115,7 +115,7 @@ static struct request *get_alua_req(struct scsi_device *sdev, · 114 unmodified lines | ||
| 115 | 115 | |
| 116 | 116 | rq = blk_get_request(q, rw, GFP_NOIO); |
| 117 | 117 | |
| 118 | − if (!rq) { | |
| 118 | + if (IS_ERR(rq)) { | |
| 119 | 119 | sdev_printk(KERN_INFO, sdev, |
| 120 | 120 | "%s: blk_get_request failed\n", __func__); |
| 121 | 121 | return NULL; |
| @@ -275,7 +275,7 @@ static struct request *get_req(struct scsi_device *sdev, int cmd, · 274 unmodified lines | ||
| 275 | 275 | |
| 276 | 276 | rq = blk_get_request(sdev->request_queue, |
| 277 | 277 | (cmd != INQUIRY) ? WRITE : READ, GFP_NOIO); |
| 278 | − if (!rq) { | |
| 278 | + if (IS_ERR(rq)) { | |
| 279 | 279 | sdev_printk(KERN_INFO, sdev, "get_req: blk_get_request failed"); |
| 280 | 280 | return NULL; |
| 281 | 281 | } |
| @@ -117,7 +117,7 @@ static int hp_sw_tur(struct scsi_device *sdev, struct hp_sw_dh_data *h) · 116 unmodified lines | ||
| 117 | 117 | |
| 118 | 118 | retry: |
| 119 | 119 | req = blk_get_request(sdev->request_queue, WRITE, GFP_NOIO); |
| 120 | − if (!req) | |
| 120 | + if (IS_ERR(req)) | |
| 121 | 121 | return SCSI_DH_RES_TEMP_UNAVAIL; |
| 122 | 122 | |
| 123 | 123 | blk_rq_set_block_pc(req); |
| @@ -247,7 +247,7 @@ static int hp_sw_start_stop(struct hp_sw_dh_data *h) · 123 unmodified lines | ||
| 247 | 247 | struct request *req; |
| 248 | 248 | |
| 249 | 249 | req = blk_get_request(h->sdev->request_queue, WRITE, GFP_ATOMIC); |
| 250 | − if (!req) | |
| 250 | + if (IS_ERR(req)) | |
| 251 | 251 | return SCSI_DH_RES_TEMP_UNAVAIL; |
| 252 | 252 | |
| 253 | 253 | blk_rq_set_block_pc(req); |
| @@ -274,7 +274,7 @@ static struct request *get_rdac_req(struct scsi_device *sdev, · 273 unmodified lines | ||
| 274 | 274 | |
| 275 | 275 | rq = blk_get_request(q, rw, GFP_NOIO); |
| 276 | 276 | |
| 277 | − if (!rq) { | |
| 277 | + if (IS_ERR(rq)) { | |
| 278 | 278 | sdev_printk(KERN_INFO, sdev, |
| 279 | 279 | "get_rdac_req: blk_get_request failed.\n"); |
| 280 | 280 | return NULL; |
| @@ -1567,8 +1567,8 @@ static struct request *_make_request(struct request_queue *q, bool has_write, · 1566 unmodified lines | ||
| 1567 | 1567 | struct request *req; |
| 1568 | 1568 | |
| 1569 | 1569 | req = blk_get_request(q, has_write ? WRITE : READ, flags); |
| 1570 | − if (unlikely(!req)) | |
| 1571 | − return ERR_PTR(-ENOMEM); | |
| 1570 | + if (IS_ERR(req)) | |
| 1571 | + return req; | |
| 1572 | 1572 | |
| 1573 | 1573 | blk_rq_set_block_pc(req); |
| 1574 | 1574 | return req; |
| @@ -362,7 +362,7 @@ static int osst_execute(struct osst_request *SRpnt, const unsigned char *cmd, · 361 unmodified lines | ||
| 362 | 362 | int write = (data_direction == DMA_TO_DEVICE); |
| 363 | 363 | |
| 364 | 364 | req = blk_get_request(SRpnt->stp->device->request_queue, write, GFP_KERNEL); |
| 365 | − if (!req) | |
| 365 | + if (IS_ERR(req)) | |
| 366 | 366 | return DRIVER_ERROR << 24; |
| 367 | 367 | |
| 368 | 368 | blk_rq_set_block_pc(req); |
| @@ -1961,6 +1961,8 @@ static void scsi_eh_lock_door(struct scsi_device *sdev) · 1960 unmodified lines | ||
| 1961 | 1961 | * request becomes available |
| 1962 | 1962 | */ |
| 1963 | 1963 | req = blk_get_request(sdev->request_queue, READ, GFP_KERNEL); |
| 1964 | + if (IS_ERR(req)) | |
| 1965 | + return; | |
| 1964 | 1966 | |
| 1965 | 1967 | blk_rq_set_block_pc(req); |
| 1966 | 1968 | |
| @@ -221,7 +221,7 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd, · 220 unmodified lines | ||
| 221 | 221 | int ret = DRIVER_ERROR << 24; |
| 222 | 222 | |
| 223 | 223 | req = blk_get_request(sdev->request_queue, write, __GFP_WAIT); |
| 224 | − if (!req) | |
| 224 | + if (IS_ERR(req)) | |
| 225 | 225 | return ret; |
| 226 | 226 | blk_rq_set_block_pc(req); |
| 227 | 227 | |
| @@ -715,7 +715,7 @@ static bool scsi_end_request(struct request *req, int error, · 487 unmodified lines | ||
| 715 | 715 | |
| 716 | 716 | if (req->mq_ctx) { |
| 717 | 717 | /* |
| 718 | − * In the MQ case the command gets freed by __blk_mq_end_io, | |
| 718 | + * In the MQ case the command gets freed by __blk_mq_end_request, | |
| 719 | 719 | * so we have to do all cleanup that depends on it earlier. |
| 720 | 720 | * |
| 721 | 721 | * We also can't kick the queues from irq context, so we |
| @@ -723,7 +723,7 @@ static bool scsi_end_request(struct request *req, int error, · 1 unmodified line | ||
| 723 | 723 | */ |
| 724 | 724 | scsi_mq_uninit_cmd(cmd); |
| 725 | 725 | |
| 726 | − __blk_mq_end_io(req, error); | |
| 726 | + __blk_mq_end_request(req, error); | |
| 727 | 727 | |
| 728 | 728 | if (scsi_target(sdev)->single_lun || |
| 729 | 729 | !list_empty(&sdev->host->starved_list)) |
| @@ -1847,6 +1847,8 @@ static int scsi_mq_prep_fn(struct request *req) · 1117 unmodified lines | ||
| 1847 | 1847 | next_rq->special = bidi_sdb; |
| 1848 | 1848 | } |
| 1849 | 1849 | |
| 1850 | + blk_mq_start_request(req); | |
| 1851 | + | |
| 1850 | 1852 | return scsi_setup_cmnd(sdev, req); |
| 1851 | 1853 | } |
| 1852 | 1854 | |
| @@ -1856,7 +1858,8 @@ static void scsi_mq_done(struct scsi_cmnd *cmd) · 3 unmodified lines | ||
| 1856 | 1858 | blk_mq_complete_request(cmd->request); |
| 1857 | 1859 | } |
| 1858 | 1860 | |
| 1859 | −static int scsi_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req) | |
| 1861 | +static int scsi_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req, | |
| 1862 | + bool last) | |
| 1860 | 1863 | { |
| 1861 | 1864 | struct request_queue *q = req->q; |
| 1862 | 1865 | struct scsi_device *sdev = q->queuedata; |
| @@ -1880,11 +1883,14 @@ static int scsi_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req) · 17 unmodified lines | ||
| 1880 | 1883 | if (!scsi_host_queue_ready(q, shost, sdev)) |
| 1881 | 1884 | goto out_dec_target_busy; |
| 1882 | 1885 | |
| 1886 | + | |
| 1883 | 1887 | if (!(req->cmd_flags & REQ_DONTPREP)) { |
| 1884 | 1888 | ret = prep_to_mq(scsi_mq_prep_fn(req)); |
| 1885 | 1889 | if (ret) |
| 1886 | 1890 | goto out_dec_host_busy; |
| 1887 | 1891 | req->cmd_flags |= REQ_DONTPREP; |
| 1892 | + } else { | |
| 1893 | + blk_mq_start_request(req); | |
| 1888 | 1894 | } |
| 1889 | 1895 | |
| 1890 | 1896 | scsi_init_cmd_errh(cmd); |
| @@ -1931,6 +1937,14 @@ out: · 40 unmodified lines | ||
| 1931 | 1937 | return ret; |
| 1932 | 1938 | } |
| 1933 | 1939 | |
| 1940 | +static enum blk_eh_timer_return scsi_timeout(struct request *req, | |
| 1941 | + bool reserved) | |
| 1942 | +{ | |
| 1943 | + if (reserved) | |
| 1944 | + return BLK_EH_RESET_TIMER; | |
| 1945 | + return scsi_times_out(req); | |
| 1946 | +} | |
| 1947 | + | |
| 1934 | 1948 | static int scsi_init_request(void *data, struct request *rq, |
| 1935 | 1949 | unsigned int hctx_idx, unsigned int request_idx, |
| 1936 | 1950 | unsigned int numa_node) |
| @@ -2042,7 +2056,7 @@ static struct blk_mq_ops scsi_mq_ops = { · 105 unmodified lines | ||
| 2042 | 2056 | .map_queue = blk_mq_map_queue, |
| 2043 | 2057 | .queue_rq = scsi_queue_rq, |
| 2044 | 2058 | .complete = scsi_softirq_done, |
| 2045 | − .timeout = scsi_times_out, | |
| 2059 | + .timeout = scsi_timeout, | |
| 2046 | 2060 | .init_request = scsi_init_request, |
| 2047 | 2061 | .exit_request = scsi_exit_request, |
| 2048 | 2062 | }; |
| @@ -610,29 +610,44 @@ static void scsi_disk_put(struct scsi_disk *sdkp) · 609 unmodified lines | ||
| 610 | 610 | mutex_unlock(&sd_ref_mutex); |
| 611 | 611 | } |
| 612 | 612 | |
| 613 | −static void sd_prot_op(struct scsi_cmnd *scmd, unsigned int dif) | |
| 614 | −{ | |
| 615 | − unsigned int prot_op = SCSI_PROT_NORMAL; | |
| 616 | − unsigned int dix = scsi_prot_sg_count(scmd); | |
| 617 | − | |
| 618 | − if (scmd->sc_data_direction == DMA_FROM_DEVICE) { | |
| 619 | − if (dif && dix) | |
| 620 | − prot_op = SCSI_PROT_READ_PASS; | |
| 621 | − else if (dif && !dix) | |
| 622 | − prot_op = SCSI_PROT_READ_STRIP; | |
| 623 | − else if (!dif && dix) | |
| 624 | − prot_op = SCSI_PROT_READ_INSERT; | |
| 625 | − } else { | |
| 626 | − if (dif && dix) | |
| 627 | − prot_op = SCSI_PROT_WRITE_PASS; | |
| 628 | − else if (dif && !dix) | |
| 629 | − prot_op = SCSI_PROT_WRITE_INSERT; | |
| 630 | − else if (!dif && dix) | |
| 631 | − prot_op = SCSI_PROT_WRITE_STRIP; | |
| 613 | + | |
| 614 | + | |
| 615 | +static unsigned char sd_setup_protect_cmnd(struct scsi_cmnd *scmd, | |
| 616 | + unsigned int dix, unsigned int dif) | |
| 617 | +{ | |
| 618 | + struct bio *bio = scmd->request->bio; | |
| 619 | + unsigned int prot_op = sd_prot_op(rq_data_dir(scmd->request), dix, dif); | |
| 620 | + unsigned int protect = 0; | |
| 621 | + | |
| 622 | + if (dix) { /* DIX Type 0, 1, 2, 3 */ | |
| 623 | + if (bio_integrity_flagged(bio, BIP_IP_CHECKSUM)) | |
| 624 | + scmd->prot_flags |= SCSI_PROT_IP_CHECKSUM; | |
| 625 | + | |
| 626 | + if (bio_integrity_flagged(bio, BIP_CTRL_NOCHECK) == false) | |
| 627 | + scmd->prot_flags |= SCSI_PROT_GUARD_CHECK; | |
| 628 | + } | |
| 629 | + | |
| 630 | + if (dif != SD_DIF_TYPE3_PROTECTION) { /* DIX/DIF Type 0, 1, 2 */ | |
| 631 | + scmd->prot_flags |= SCSI_PROT_REF_INCREMENT; | |
| 632 | + | |
| 633 | + if (bio_integrity_flagged(bio, BIP_CTRL_NOCHECK) == false) | |
| 634 | + scmd->prot_flags |= SCSI_PROT_REF_CHECK; | |
| 635 | + } | |
| 636 | + | |
| 637 | + if (dif) { /* DIX/DIF Type 1, 2, 3 */ | |
| 638 | + scmd->prot_flags |= SCSI_PROT_TRANSFER_PI; | |
| 639 | + | |
| 640 | + if (bio_integrity_flagged(bio, BIP_DISK_NOCHECK)) | |
| 641 | + protect = 3 << 5; /* Disable target PI checking */ | |
| 642 | + else | |
| 643 | + protect = 1 << 5; /* Enable target PI checking */ | |
| 632 | 644 | } |
| 633 | 645 | |
| 634 | 646 | scsi_set_prot_op(scmd, prot_op); |
| 635 | 647 | scsi_set_prot_type(scmd, dif); |
| 648 | + scmd->prot_flags &= sd_prot_flag_mask(prot_op); | |
| 649 | + | |
| 650 | + return protect; | |
| 636 | 651 | } |
| 637 | 652 | |
| 638 | 653 | static void sd_config_discard(struct scsi_disk *sdkp, unsigned int mode) |
| @@ -893,7 +908,8 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) · 254 unmodified lines | ||
| 893 | 908 | sector_t block = blk_rq_pos(rq); |
| 894 | 909 | sector_t threshold; |
| 895 | 910 | unsigned int this_count = blk_rq_sectors(rq); |
| 896 | − int ret, host_dif; | |
| 911 | + unsigned int dif, dix; | |
| 912 | + int ret; | |
| 897 | 913 | unsigned char protect; |
| 898 | 914 | |
| 899 | 915 | ret = scsi_init_io(SCpnt, GFP_ATOMIC); |
| @@ -995,7 +1011,7 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) · 95 unmodified lines | ||
| 995 | 1011 | SCpnt->cmnd[0] = WRITE_6; |
| 996 | 1012 | |
| 997 | 1013 | if (blk_integrity_rq(rq)) |
| 998 | − sd_dif_prepare(rq, block, sdp->sector_size); | |
| 1014 | + sd_dif_prepare(SCpnt); | |
| 999 | 1015 | |
| 1000 | 1016 | } else if (rq_data_dir(rq) == READ) { |
| 1001 | 1017 | SCpnt->cmnd[0] = READ_6; |
| @@ -1010,14 +1026,15 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) · 8 unmodified lines | ||
| 1010 | 1026 | "writing" : "reading", this_count, |
| 1011 | 1027 | blk_rq_sectors(rq))); |
| 1012 | 1028 | |
| 1013 | − /* Set RDPROTECT/WRPROTECT if disk is formatted with DIF */ | |
| 1014 | − host_dif = scsi_host_dif_capable(sdp->host, sdkp->protection_type); | |
| 1015 | − if (host_dif) | |
| 1016 | − protect = 1 << 5; | |
| 1029 | + dix = scsi_prot_sg_count(SCpnt); | |
| 1030 | + dif = scsi_host_dif_capable(SCpnt->device->host, sdkp->protection_type); | |
| 1031 | + | |
| 1032 | + if (dif || dix) | |
| 1033 | + protect = sd_setup_protect_cmnd(SCpnt, dix, dif); | |
| 1017 | 1034 | else |
| 1018 | 1035 | protect = 0; |
| 1019 | 1036 | |
| 1020 | − if (host_dif == SD_DIF_TYPE2_PROTECTION) { | |
| 1037 | + if (protect && sdkp->protection_type == SD_DIF_TYPE2_PROTECTION) { | |
| 1021 | 1038 | SCpnt->cmnd = mempool_alloc(sd_cdb_pool, GFP_ATOMIC); |
| 1022 | 1039 | |
| 1023 | 1040 | if (unlikely(SCpnt->cmnd == NULL)) { |
| @@ -1102,10 +1119,6 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) · 78 unmodified lines | ||
| 1102 | 1119 | } |
| 1103 | 1120 | SCpnt->sdb.length = this_count * sdp->sector_size; |
| 1104 | 1121 | |
| 1105 | − /* If DIF or DIX is enabled, tell HBA how to handle request */ | |
| 1106 | − if (host_dif || scsi_prot_sg_count(SCpnt)) | |
| 1107 | − sd_prot_op(SCpnt, host_dif); | |
| 1108 | − | |
| 1109 | 1122 | /* |
| 1110 | 1123 | * We shouldn't disconnect in the middle of a sector, so with a dumb |
| 1111 | 1124 | * host adapter, it's safe to assume that we can at least transfer |
| @@ -166,6 +166,68 @@ enum sd_dif_target_protection_types { · 165 unmodified lines | ||
| 166 | 166 | SD_DIF_TYPE3_PROTECTION = 0x3, |
| 167 | 167 | }; |
| 168 | 168 | |
| 169 | +/* | |
| 170 | + * Look up the DIX operation based on whether the command is read or | |
| 171 | + * write and whether dix and dif are enabled. | |
| 172 | + */ | |
| 173 | +static inline unsigned int sd_prot_op(bool write, bool dix, bool dif) | |
| 174 | +{ | |
| 175 | + /* Lookup table: bit 2 (write), bit 1 (dix), bit 0 (dif) */ | |
| 176 | + const unsigned int ops[] = { /* wrt dix dif */ | |
| 177 | + SCSI_PROT_NORMAL, /* 0 0 0 */ | |
| 178 | + SCSI_PROT_READ_STRIP, /* 0 0 1 */ | |
| 179 | + SCSI_PROT_READ_INSERT, /* 0 1 0 */ | |
| 180 | + SCSI_PROT_READ_PASS, /* 0 1 1 */ | |
| 181 | + SCSI_PROT_NORMAL, /* 1 0 0 */ | |
| 182 | + SCSI_PROT_WRITE_INSERT, /* 1 0 1 */ | |
| 183 | + SCSI_PROT_WRITE_STRIP, /* 1 1 0 */ | |
| 184 | + SCSI_PROT_WRITE_PASS, /* 1 1 1 */ | |
| 185 | + }; | |
| 186 | + | |
| 187 | + return ops[write << 2 | dix << 1 | dif]; | |
| 188 | +} | |
| 189 | + | |
| 190 | +/* | |
| 191 | + * Returns a mask of the protection flags that are valid for a given DIX | |
| 192 | + * operation. | |
| 193 | + */ | |
| 194 | +static inline unsigned int sd_prot_flag_mask(unsigned int prot_op) | |
| 195 | +{ | |
| 196 | + const unsigned int flag_mask[] = { | |
| 197 | + [SCSI_PROT_NORMAL] = 0, | |
| 198 | + | |
| 199 | + [SCSI_PROT_READ_STRIP] = SCSI_PROT_TRANSFER_PI | | |
| 200 | + SCSI_PROT_GUARD_CHECK | | |
| 201 | + SCSI_PROT_REF_CHECK | | |
| 202 | + SCSI_PROT_REF_INCREMENT, | |
| 203 | + | |
| 204 | + [SCSI_PROT_READ_INSERT] = SCSI_PROT_REF_INCREMENT | | |
| 205 | + SCSI_PROT_IP_CHECKSUM, | |
| 206 | + | |
| 207 | + [SCSI_PROT_READ_PASS] = SCSI_PROT_TRANSFER_PI | | |
| 208 | + SCSI_PROT_GUARD_CHECK | | |
| 209 | + SCSI_PROT_REF_CHECK | | |
| 210 | + SCSI_PROT_REF_INCREMENT | | |
| 211 | + SCSI_PROT_IP_CHECKSUM, | |
| 212 | + | |
| 213 | + [SCSI_PROT_WRITE_INSERT] = SCSI_PROT_TRANSFER_PI | | |
| 214 | + SCSI_PROT_REF_INCREMENT, | |
| 215 | + | |
| 216 | + [SCSI_PROT_WRITE_STRIP] = SCSI_PROT_GUARD_CHECK | | |
| 217 | + SCSI_PROT_REF_CHECK | | |
| 218 | + SCSI_PROT_REF_INCREMENT | | |
| 219 | + SCSI_PROT_IP_CHECKSUM, | |
| 220 | + | |
| 221 | + [SCSI_PROT_WRITE_PASS] = SCSI_PROT_TRANSFER_PI | | |
| 222 | + SCSI_PROT_GUARD_CHECK | | |
| 223 | + SCSI_PROT_REF_CHECK | | |
| 224 | + SCSI_PROT_REF_INCREMENT | | |
| 225 | + SCSI_PROT_IP_CHECKSUM, | |
| 226 | + }; | |
| 227 | + | |
| 228 | + return flag_mask[prot_op]; | |
| 229 | +} | |
| 230 | + | |
| 169 | 231 | /* |
| 170 | 232 | * Data Integrity Field tuple. |
| 171 | 233 | */ |
| @@ -178,7 +240,7 @@ struct sd_dif_tuple { · 6 unmodified lines | ||
| 178 | 240 | #ifdef CONFIG_BLK_DEV_INTEGRITY |
| 179 | 241 | |
| 180 | 242 | extern void sd_dif_config_host(struct scsi_disk *); |
| 181 | −extern void sd_dif_prepare(struct request *rq, sector_t, unsigned int); | |
| 243 | +extern void sd_dif_prepare(struct scsi_cmnd *scmd); | |
| 182 | 244 | extern void sd_dif_complete(struct scsi_cmnd *, unsigned int); |
| 183 | 245 | |
| 184 | 246 | #else /* CONFIG_BLK_DEV_INTEGRITY */ |
| @@ -186,7 +248,7 @@ extern void sd_dif_complete(struct scsi_cmnd *, unsigned int); · 1 unmodified line | ||
| 186 | 248 | static inline void sd_dif_config_host(struct scsi_disk *disk) |
| 187 | 249 | { |
| 188 | 250 | } |
| 189 | −static inline int sd_dif_prepare(struct request *rq, sector_t s, unsigned int a) | |
| 251 | +static inline int sd_dif_prepare(struct scsi_cmnd *scmd) | |
| 190 | 252 | { |
| 191 | 253 | return 0; |
| 192 | 254 | } |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | */ |
| 22 | 22 | |
| 23 | 23 | #include <linux/blkdev.h> |
| 24 | −#include <linux/crc-t10dif.h> | |
| 24 | +#include <linux/t10-pi.h> | |
| 25 | 25 | |
| 26 | 26 | #include <scsi/scsi.h> |
| 27 | 27 | #include <scsi/scsi_cmnd.h> |
| @@ -33,268 +33,8 @@ · 5 unmodified lines | ||
| 33 | 33 | #include <scsi/scsi_ioctl.h> |
| 34 | 34 | #include <scsi/scsicam.h> |
| 35 | 35 | |
| 36 | −#include <net/checksum.h> | |
| 37 | − | |
| 38 | 36 | #include "sd.h" |
| 39 | 37 | |
| 40 | −typedef __u16 (csum_fn) (void *, unsigned int); | |
| 41 | − | |
| 42 | −static __u16 sd_dif_crc_fn(void *data, unsigned int len) | |
| 43 | −{ | |
| 44 | − return cpu_to_be16(crc_t10dif(data, len)); | |
| 45 | −} | |
| 46 | − | |
| 47 | −static __u16 sd_dif_ip_fn(void *data, unsigned int len) | |
| 48 | −{ | |
| 49 | − return ip_compute_csum(data, len); | |
| 50 | −} | |
| 51 | − | |
| 52 | −/* | |
| 53 | − * Type 1 and Type 2 protection use the same format: 16 bit guard tag, | |
| 54 | − * 16 bit app tag, 32 bit reference tag. | |
| 55 | − */ | |
| 56 | −static void sd_dif_type1_generate(struct blk_integrity_exchg *bix, csum_fn *fn) | |
| 57 | −{ | |
| 58 | − void *buf = bix->data_buf; | |
| 59 | − struct sd_dif_tuple *sdt = bix->prot_buf; | |
| 60 | − sector_t sector = bix->sector; | |
| 61 | − unsigned int i; | |
| 62 | − | |
| 63 | − for (i = 0 ; i < bix->data_size ; i += bix->sector_size, sdt++) { | |
| 64 | − sdt->guard_tag = fn(buf, bix->sector_size); | |
| 65 | − sdt->ref_tag = cpu_to_be32(sector & 0xffffffff); | |
| 66 | − sdt->app_tag = 0; | |
| 67 | − | |
| 68 | − buf += bix->sector_size; | |
| 69 | − sector++; | |
| 70 | − } | |
| 71 | −} | |
| 72 | − | |
| 73 | −static void sd_dif_type1_generate_crc(struct blk_integrity_exchg *bix) | |
| 74 | −{ | |
| 75 | − sd_dif_type1_generate(bix, sd_dif_crc_fn); | |
| 76 | −} | |
| 77 | − | |
| 78 | −static void sd_dif_type1_generate_ip(struct blk_integrity_exchg *bix) | |
| 79 | −{ | |
| 80 | − sd_dif_type1_generate(bix, sd_dif_ip_fn); | |
| 81 | −} | |
| 82 | − | |
| 83 | −static int sd_dif_type1_verify(struct blk_integrity_exchg *bix, csum_fn *fn) | |
| 84 | −{ | |
| 85 | − void *buf = bix->data_buf; | |
| 86 | − struct sd_dif_tuple *sdt = bix->prot_buf; | |
| 87 | − sector_t sector = bix->sector; | |
| 88 | − unsigned int i; | |
| 89 | − __u16 csum; | |
| 90 | − | |
| 91 | − for (i = 0 ; i < bix->data_size ; i += bix->sector_size, sdt++) { | |
| 92 | − /* Unwritten sectors */ | |
| 93 | − if (sdt->app_tag == 0xffff) | |
| 94 | − return 0; | |
| 95 | − | |
| 96 | − if (be32_to_cpu(sdt->ref_tag) != (sector & 0xffffffff)) { | |
| 97 | − printk(KERN_ERR | |
| 98 | − "%s: ref tag error on sector %lu (rcvd %u)\n", | |
| 99 | − bix->disk_name, (unsigned long)sector, | |
| 100 | − be32_to_cpu(sdt->ref_tag)); | |
| 101 | − return -EIO; | |
| 102 | − } | |
| 103 | − | |
| 104 | − csum = fn(buf, bix->sector_size); | |
| 105 | − | |
| 106 | − if (sdt->guard_tag != csum) { | |
| 107 | − printk(KERN_ERR "%s: guard tag error on sector %lu " \ | |
| 108 | − "(rcvd %04x, data %04x)\n", bix->disk_name, | |
| 109 | − (unsigned long)sector, | |
| 110 | − be16_to_cpu(sdt->guard_tag), be16_to_cpu(csum)); | |
| 111 | − return -EIO; | |
| 112 | − } | |
| 113 | − | |
| 114 | − buf += bix->sector_size; | |
| 115 | − sector++; | |
| 116 | − } | |
| 117 | − | |
| 118 | − return 0; | |
| 119 | −} | |
| 120 | − | |
| 121 | −static int sd_dif_type1_verify_crc(struct blk_integrity_exchg *bix) | |
| 122 | −{ | |
| 123 | − return sd_dif_type1_verify(bix, sd_dif_crc_fn); | |
| 124 | −} | |
| 125 | − | |
| 126 | −static int sd_dif_type1_verify_ip(struct blk_integrity_exchg *bix) | |
| 127 | −{ | |
| 128 | − return sd_dif_type1_verify(bix, sd_dif_ip_fn); | |
| 129 | −} | |
| 130 | − | |
| 131 | −/* | |
| 132 | − * Functions for interleaving and deinterleaving application tags | |
| 133 | − */ | |
| 134 | −static void sd_dif_type1_set_tag(void *prot, void *tag_buf, unsigned int sectors) | |
| 135 | −{ | |
| 136 | − struct sd_dif_tuple *sdt = prot; | |
| 137 | − u8 *tag = tag_buf; | |
| 138 | − unsigned int i, j; | |
| 139 | − | |
| 140 | − for (i = 0, j = 0 ; i < sectors ; i++, j += 2, sdt++) { | |
| 141 | − sdt->app_tag = tag[j] << 8 | tag[j+1]; | |
| 142 | − BUG_ON(sdt->app_tag == 0xffff); | |
| 143 | − } | |
| 144 | −} | |
| 145 | − | |
| 146 | −static void sd_dif_type1_get_tag(void *prot, void *tag_buf, unsigned int sectors) | |
| 147 | −{ | |
| 148 | − struct sd_dif_tuple *sdt = prot; | |
| 149 | − u8 *tag = tag_buf; | |
| 150 | − unsigned int i, j; | |
| 151 | − | |
| 152 | − for (i = 0, j = 0 ; i < sectors ; i++, j += 2, sdt++) { | |
| 153 | − tag[j] = (sdt->app_tag & 0xff00) >> 8; | |
| 154 | − tag[j+1] = sdt->app_tag & 0xff; | |
| 155 | − } | |
| 156 | −} | |
| 157 | − | |
| 158 | −static struct blk_integrity dif_type1_integrity_crc = { | |
| 159 | − .name = "T10-DIF-TYPE1-CRC", | |
| 160 | − .generate_fn = sd_dif_type1_generate_crc, | |
| 161 | − .verify_fn = sd_dif_type1_verify_crc, | |
| 162 | − .get_tag_fn = sd_dif_type1_get_tag, | |
| 163 | − .set_tag_fn = sd_dif_type1_set_tag, | |
| 164 | − .tuple_size = sizeof(struct sd_dif_tuple), | |
| 165 | − .tag_size = 0, | |
| 166 | −}; | |
| 167 | − | |
| 168 | −static struct blk_integrity dif_type1_integrity_ip = { | |
| 169 | − .name = "T10-DIF-TYPE1-IP", | |
| 170 | − .generate_fn = sd_dif_type1_generate_ip, | |
| 171 | − .verify_fn = sd_dif_type1_verify_ip, | |
| 172 | − .get_tag_fn = sd_dif_type1_get_tag, | |
| 173 | − .set_tag_fn = sd_dif_type1_set_tag, | |
| 174 | − .tuple_size = sizeof(struct sd_dif_tuple), | |
| 175 | − .tag_size = 0, | |
| 176 | −}; | |
| 177 | − | |
| 178 | − | |
| 179 | −/* | |
| 180 | − * Type 3 protection has a 16-bit guard tag and 16 + 32 bits of opaque | |
| 181 | − * tag space. | |
| 182 | − */ | |
| 183 | −static void sd_dif_type3_generate(struct blk_integrity_exchg *bix, csum_fn *fn) | |
| 184 | −{ | |
| 185 | − void *buf = bix->data_buf; | |
| 186 | − struct sd_dif_tuple *sdt = bix->prot_buf; | |
| 187 | − unsigned int i; | |
| 188 | − | |
| 189 | − for (i = 0 ; i < bix->data_size ; i += bix->sector_size, sdt++) { | |
| 190 | − sdt->guard_tag = fn(buf, bix->sector_size); | |
| 191 | − sdt->ref_tag = 0; | |
| 192 | − sdt->app_tag = 0; | |
| 193 | − | |
| 194 | − buf += bix->sector_size; | |
| 195 | − } | |
| 196 | −} | |
| 197 | − | |
| 198 | −static void sd_dif_type3_generate_crc(struct blk_integrity_exchg *bix) | |
| 199 | −{ | |
| 200 | − sd_dif_type3_generate(bix, sd_dif_crc_fn); | |
| 201 | −} | |
| 202 | − | |
| 203 | −static void sd_dif_type3_generate_ip(struct blk_integrity_exchg *bix) | |
| 204 | −{ | |
| 205 | − sd_dif_type3_generate(bix, sd_dif_ip_fn); | |
| 206 | −} | |
| 207 | − | |
| 208 | −static int sd_dif_type3_verify(struct blk_integrity_exchg *bix, csum_fn *fn) | |
| 209 | −{ | |
| 210 | − void *buf = bix->data_buf; | |
| 211 | − struct sd_dif_tuple *sdt = bix->prot_buf; | |
| 212 | − sector_t sector = bix->sector; | |
| 213 | − unsigned int i; | |
| 214 | − __u16 csum; | |
| 215 | − | |
| 216 | − for (i = 0 ; i < bix->data_size ; i += bix->sector_size, sdt++) { | |
| 217 | − /* Unwritten sectors */ | |
| 218 | − if (sdt->app_tag == 0xffff && sdt->ref_tag == 0xffffffff) | |
| 219 | − return 0; | |
| 220 | − | |
| 221 | − csum = fn(buf, bix->sector_size); | |
| 222 | − | |
| 223 | − if (sdt->guard_tag != csum) { | |
| 224 | − printk(KERN_ERR "%s: guard tag error on sector %lu " \ | |
| 225 | − "(rcvd %04x, data %04x)\n", bix->disk_name, | |
| 226 | − (unsigned long)sector, | |
| 227 | − be16_to_cpu(sdt->guard_tag), be16_to_cpu(csum)); | |
| 228 | − return -EIO; | |
| 229 | − } | |
| 230 | − | |
| 231 | − buf += bix->sector_size; | |
| 232 | − sector++; | |
| 233 | − } | |
| 234 | − | |
| 235 | − return 0; | |
| 236 | −} | |
| 237 | − | |
| 238 | −static int sd_dif_type3_verify_crc(struct blk_integrity_exchg *bix) | |
| 239 | −{ | |
| 240 | − return sd_dif_type3_verify(bix, sd_dif_crc_fn); | |
| 241 | −} | |
| 242 | − | |
| 243 | −static int sd_dif_type3_verify_ip(struct blk_integrity_exchg *bix) | |
| 244 | −{ | |
| 245 | − return sd_dif_type3_verify(bix, sd_dif_ip_fn); | |
| 246 | −} | |
| 247 | − | |
| 248 | −static void sd_dif_type3_set_tag(void *prot, void *tag_buf, unsigned int sectors) | |
| 249 | −{ | |
| 250 | − struct sd_dif_tuple *sdt = prot; | |
| 251 | − u8 *tag = tag_buf; | |
| 252 | − unsigned int i, j; | |
| 253 | − | |
| 254 | − for (i = 0, j = 0 ; i < sectors ; i++, j += 6, sdt++) { | |
| 255 | − sdt->app_tag = tag[j] << 8 | tag[j+1]; | |
| 256 | − sdt->ref_tag = tag[j+2] << 24 | tag[j+3] << 16 | | |
| 257 | − tag[j+4] << 8 | tag[j+5]; | |
| 258 | − } | |
| 259 | −} | |
| 260 | − | |
| 261 | −static void sd_dif_type3_get_tag(void *prot, void *tag_buf, unsigned int sectors) | |
| 262 | −{ | |
| 263 | − struct sd_dif_tuple *sdt = prot; | |
| 264 | − u8 *tag = tag_buf; | |
| 265 | − unsigned int i, j; | |
| 266 | − | |
| 267 | − for (i = 0, j = 0 ; i < sectors ; i++, j += 2, sdt++) { | |
| 268 | − tag[j] = (sdt->app_tag & 0xff00) >> 8; | |
| 269 | − tag[j+1] = sdt->app_tag & 0xff; | |
| 270 | − tag[j+2] = (sdt->ref_tag & 0xff000000) >> 24; | |
| 271 | − tag[j+3] = (sdt->ref_tag & 0xff0000) >> 16; | |
| 272 | − tag[j+4] = (sdt->ref_tag & 0xff00) >> 8; | |
| 273 | − tag[j+5] = sdt->ref_tag & 0xff; | |
| 274 | − BUG_ON(sdt->app_tag == 0xffff || sdt->ref_tag == 0xffffffff); | |
| 275 | − } | |
| 276 | −} | |
| 277 | − | |
| 278 | −static struct blk_integrity dif_type3_integrity_crc = { | |
| 279 | − .name = "T10-DIF-TYPE3-CRC", | |
| 280 | − .generate_fn = sd_dif_type3_generate_crc, | |
| 281 | − .verify_fn = sd_dif_type3_verify_crc, | |
| 282 | − .get_tag_fn = sd_dif_type3_get_tag, | |
| 283 | − .set_tag_fn = sd_dif_type3_set_tag, | |
| 284 | − .tuple_size = sizeof(struct sd_dif_tuple), | |
| 285 | − .tag_size = 0, | |
| 286 | −}; | |
| 287 | − | |
| 288 | −static struct blk_integrity dif_type3_integrity_ip = { | |
| 289 | − .name = "T10-DIF-TYPE3-IP", | |
| 290 | − .generate_fn = sd_dif_type3_generate_ip, | |
| 291 | − .verify_fn = sd_dif_type3_verify_ip, | |
| 292 | − .get_tag_fn = sd_dif_type3_get_tag, | |
| 293 | − .set_tag_fn = sd_dif_type3_set_tag, | |
| 294 | − .tuple_size = sizeof(struct sd_dif_tuple), | |
| 295 | − .tag_size = 0, | |
| 296 | −}; | |
| 297 | − | |
| 298 | 38 | /* |
| 299 | 39 | * Configure exchange of protection information between OS and HBA. |
| 300 | 40 | */ |
| @@ -316,22 +56,30 @@ void sd_dif_config_host(struct scsi_disk *sdkp) · 15 unmodified lines | ||
| 316 | 56 | return; |
| 317 | 57 | |
| 318 | 58 | /* Enable DMA of protection information */ |
| 319 | − if (scsi_host_get_guard(sdkp->device->host) & SHOST_DIX_GUARD_IP) | |
| 59 | + if (scsi_host_get_guard(sdkp->device->host) & SHOST_DIX_GUARD_IP) { | |
| 320 | 60 | if (type == SD_DIF_TYPE3_PROTECTION) |
| 321 | − blk_integrity_register(disk, &dif_type3_integrity_ip); | |
| 61 | + blk_integrity_register(disk, &t10_pi_type3_ip); | |
| 322 | 62 | else |
| 323 | − blk_integrity_register(disk, &dif_type1_integrity_ip); | |
| 324 | − else | |
| 63 | + blk_integrity_register(disk, &t10_pi_type1_ip); | |
| 64 | + | |
| 65 | + disk->integrity->flags |= BLK_INTEGRITY_IP_CHECKSUM; | |
| 66 | + } else | |
| 325 | 67 | if (type == SD_DIF_TYPE3_PROTECTION) |
| 326 | − blk_integrity_register(disk, &dif_type3_integrity_crc); | |
| 68 | + blk_integrity_register(disk, &t10_pi_type3_crc); | |
| 327 | 69 | else |
| 328 | − blk_integrity_register(disk, &dif_type1_integrity_crc); | |
| 70 | + blk_integrity_register(disk, &t10_pi_type1_crc); | |
| 329 | 71 | |
| 330 | 72 | sd_printk(KERN_NOTICE, sdkp, |
| 331 | 73 | "Enabling DIX %s protection\n", disk->integrity->name); |
| 332 | 74 | |
| 333 | 75 | /* Signal to block layer that we support sector tagging */ |
| 334 | − if (dif && type && sdkp->ATO) { | |
| 76 | + if (dif && type) { | |
| 77 | + | |
| 78 | + disk->integrity->flags |= BLK_INTEGRITY_DEVICE_CAPABLE; | |
| 79 | + | |
| 80 | + if (!sdkp) | |
| 81 | + return; | |
| 82 | + | |
| 335 | 83 | if (type == SD_DIF_TYPE3_PROTECTION) |
| 336 | 84 | disk->integrity->tag_size = sizeof(u16) + sizeof(u32); |
| 337 | 85 | else |
| @@ -358,50 +106,49 @@ void sd_dif_config_host(struct scsi_disk *sdkp) · 20 unmodified lines | ||
| 358 | 106 | * |
| 359 | 107 | * Type 3 does not have a reference tag so no remapping is required. |
| 360 | 108 | */ |
| 361 | −void sd_dif_prepare(struct request *rq, sector_t hw_sector, | |
| 362 | − unsigned int sector_sz) | |
| 109 | +void sd_dif_prepare(struct scsi_cmnd *scmd) | |
| 363 | 110 | { |
| 364 | − const int tuple_sz = sizeof(struct sd_dif_tuple); | |
| 111 | + const int tuple_sz = sizeof(struct t10_pi_tuple); | |
| 365 | 112 | struct bio *bio; |
| 366 | 113 | struct scsi_disk *sdkp; |
| 367 | − struct sd_dif_tuple *sdt; | |
| 114 | + struct t10_pi_tuple *pi; | |
| 368 | 115 | u32 phys, virt; |
| 369 | 116 | |
| 370 | − sdkp = rq->bio->bi_bdev->bd_disk->private_data; | |
| 117 | + sdkp = scsi_disk(scmd->request->rq_disk); | |
| 371 | 118 | |
| 372 | 119 | if (sdkp->protection_type == SD_DIF_TYPE3_PROTECTION) |
| 373 | 120 | return; |
| 374 | 121 | |
| 375 | − phys = hw_sector & 0xffffffff; | |
| 122 | + phys = scsi_prot_ref_tag(scmd); | |
| 376 | 123 | |
| 377 | − __rq_for_each_bio(bio, rq) { | |
| 124 | + __rq_for_each_bio(bio, scmd->request) { | |
| 125 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 378 | 126 | struct bio_vec iv; |
| 379 | 127 | struct bvec_iter iter; |
| 380 | 128 | unsigned int j; |
| 381 | 129 | |
| 382 | 130 | /* Already remapped? */ |
| 383 | − if (bio_flagged(bio, BIO_MAPPED_INTEGRITY)) | |
| 131 | + if (bip->bip_flags & BIP_MAPPED_INTEGRITY) | |
| 384 | 132 | break; |
| 385 | 133 | |
| 386 | − virt = bio->bi_integrity->bip_iter.bi_sector & 0xffffffff; | |
| 134 | + virt = bip_get_seed(bip) & 0xffffffff; | |
| 387 | 135 | |
| 388 | − bip_for_each_vec(iv, bio->bi_integrity, iter) { | |
| 389 | − sdt = kmap_atomic(iv.bv_page) | |
| 390 | − + iv.bv_offset; | |
| 136 | + bip_for_each_vec(iv, bip, iter) { | |
| 137 | + pi = kmap_atomic(iv.bv_page) + iv.bv_offset; | |
| 391 | 138 | |
| 392 | − for (j = 0; j < iv.bv_len; j += tuple_sz, sdt++) { | |
| 139 | + for (j = 0; j < iv.bv_len; j += tuple_sz, pi++) { | |
| 393 | 140 | |
| 394 | − if (be32_to_cpu(sdt->ref_tag) == virt) | |
| 395 | − sdt->ref_tag = cpu_to_be32(phys); | |
| 141 | + if (be32_to_cpu(pi->ref_tag) == virt) | |
| 142 | + pi->ref_tag = cpu_to_be32(phys); | |
| 396 | 143 | |
| 397 | 144 | virt++; |
| 398 | 145 | phys++; |
| 399 | 146 | } |
| 400 | 147 | |
| 401 | − kunmap_atomic(sdt); | |
| 148 | + kunmap_atomic(pi); | |
| 402 | 149 | } |
| 403 | 150 | |
| 404 | − bio->bi_flags |= (1 << BIO_MAPPED_INTEGRITY); | |
| 151 | + bip->bip_flags |= BIP_MAPPED_INTEGRITY; | |
| 405 | 152 | } |
| 406 | 153 | } |
| 407 | 154 | |
| @@ -411,11 +158,11 @@ void sd_dif_prepare(struct request *rq, sector_t hw_sector, · 3 unmodified lines | ||
| 411 | 158 | */ |
| 412 | 159 | void sd_dif_complete(struct scsi_cmnd *scmd, unsigned int good_bytes) |
| 413 | 160 | { |
| 414 | − const int tuple_sz = sizeof(struct sd_dif_tuple); | |
| 161 | + const int tuple_sz = sizeof(struct t10_pi_tuple); | |
| 415 | 162 | struct scsi_disk *sdkp; |
| 416 | 163 | struct bio *bio; |
| 417 | − struct sd_dif_tuple *sdt; | |
| 418 | − unsigned int j, sectors, sector_sz; | |
| 164 | + struct t10_pi_tuple *pi; | |
| 165 | + unsigned int j, intervals; | |
| 419 | 166 | u32 phys, virt; |
| 420 | 167 | |
| 421 | 168 | sdkp = scsi_disk(scmd->request->rq_disk); |
| @@ -423,39 +170,35 @@ void sd_dif_complete(struct scsi_cmnd *scmd, unsigned int good_bytes) · 1 unmodified line | ||
| 423 | 170 | if (sdkp->protection_type == SD_DIF_TYPE3_PROTECTION || good_bytes == 0) |
| 424 | 171 | return; |
| 425 | 172 | |
| 426 | − sector_sz = scmd->device->sector_size; | |
| 427 | − sectors = good_bytes / sector_sz; | |
| 428 | − | |
| 429 | − phys = blk_rq_pos(scmd->request) & 0xffffffff; | |
| 430 | − if (sector_sz == 4096) | |
| 431 | − phys >>= 3; | |
| 173 | + intervals = good_bytes / scsi_prot_interval(scmd); | |
| 174 | + phys = scsi_prot_ref_tag(scmd); | |
| 432 | 175 | |
| 433 | 176 | __rq_for_each_bio(bio, scmd->request) { |
| 177 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 434 | 178 | struct bio_vec iv; |
| 435 | 179 | struct bvec_iter iter; |
| 436 | 180 | |
| 437 | − virt = bio->bi_integrity->bip_iter.bi_sector & 0xffffffff; | |
| 181 | + virt = bip_get_seed(bip) & 0xffffffff; | |
| 438 | 182 | |
| 439 | − bip_for_each_vec(iv, bio->bi_integrity, iter) { | |
| 440 | − sdt = kmap_atomic(iv.bv_page) | |
| 441 | − + iv.bv_offset; | |
| 183 | + bip_for_each_vec(iv, bip, iter) { | |
| 184 | + pi = kmap_atomic(iv.bv_page) + iv.bv_offset; | |
| 442 | 185 | |
| 443 | − for (j = 0; j < iv.bv_len; j += tuple_sz, sdt++) { | |
| 186 | + for (j = 0; j < iv.bv_len; j += tuple_sz, pi++) { | |
| 444 | 187 | |
| 445 | − if (sectors == 0) { | |
| 446 | − kunmap_atomic(sdt); | |
| 188 | + if (intervals == 0) { | |
| 189 | + kunmap_atomic(pi); | |
| 447 | 190 | return; |
| 448 | 191 | } |
| 449 | 192 | |
| 450 | − if (be32_to_cpu(sdt->ref_tag) == phys) | |
| 451 | − sdt->ref_tag = cpu_to_be32(virt); | |
| 193 | + if (be32_to_cpu(pi->ref_tag) == phys) | |
| 194 | + pi->ref_tag = cpu_to_be32(virt); | |
| 452 | 195 | |
| 453 | 196 | virt++; |
| 454 | 197 | phys++; |
| 455 | − sectors--; | |
| 198 | + intervals--; | |
| 456 | 199 | } |
| 457 | 200 | |
| 458 | − kunmap_atomic(sdt); | |
| 201 | + kunmap_atomic(pi); | |
| 459 | 202 | } |
| 460 | 203 | } |
| 461 | 204 | } |
| @@ -1711,9 +1711,9 @@ sg_start_req(Sg_request *srp, unsigned char *cmd) · 1710 unmodified lines | ||
| 1711 | 1711 | } |
| 1712 | 1712 | |
| 1713 | 1713 | rq = blk_get_request(q, rw, GFP_ATOMIC); |
| 1714 | − if (!rq) { | |
| 1714 | + if (IS_ERR(rq)) { | |
| 1715 | 1715 | kfree(long_cmdp); |
| 1716 | − return -ENOMEM; | |
| 1716 | + return PTR_ERR(rq); | |
| 1717 | 1717 | } |
| 1718 | 1718 | |
| 1719 | 1719 | blk_rq_set_block_pc(rq); |
| @@ -490,7 +490,7 @@ static int st_scsi_execute(struct st_request *SRpnt, const unsigned char *cmd, · 489 unmodified lines | ||
| 490 | 490 | |
| 491 | 491 | req = blk_get_request(SRpnt->stp->device->request_queue, write, |
| 492 | 492 | GFP_KERNEL); |
| 493 | − if (!req) | |
| 493 | + if (IS_ERR(req)) | |
| 494 | 494 | return DRIVER_ERROR << 24; |
| 495 | 495 | |
| 496 | 496 | blk_rq_set_block_pc(req); |
| @@ -1050,7 +1050,7 @@ pscsi_execute_cmd(struct se_cmd *cmd) · 1049 unmodified lines | ||
| 1050 | 1050 | req = blk_get_request(pdv->pdv_sd->request_queue, |
| 1051 | 1051 | (data_direction == DMA_TO_DEVICE), |
| 1052 | 1052 | GFP_KERNEL); |
| 1053 | − if (!req) { | |
| 1053 | + if (IS_ERR(req)) { | |
| 1054 | 1054 | pr_err("PSCSI: blk_get_request() failed\n"); |
| 1055 | 1055 | ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE; |
| 1056 | 1056 | goto fail; |
| @@ -50,32 +50,22 @@ inline struct block_device *I_BDEV(struct inode *inode) · 49 unmodified lines | ||
| 50 | 50 | EXPORT_SYMBOL(I_BDEV); |
| 51 | 51 | |
| 52 | 52 | /* |
| 53 | − * Move the inode from its current bdi to a new bdi. If the inode is dirty we | |
| 54 | − * need to move it onto the dirty list of @dst so that the inode is always on | |
| 55 | − * the right list. | |
| 53 | + * Move the inode from its current bdi to a new bdi. Make sure the inode | |
| 54 | + * is clean before moving so that it doesn't linger on the old bdi. | |
| 56 | 55 | */ |
| 57 | 56 | static void bdev_inode_switch_bdi(struct inode *inode, |
| 58 | 57 | struct backing_dev_info *dst) |
| 59 | 58 | { |
| 60 | − struct backing_dev_info *old = inode->i_data.backing_dev_info; | |
| 61 | − bool wakeup_bdi = false; | |
| 62 | − | |
| 63 | − if (unlikely(dst == old)) /* deadlock avoidance */ | |
| 64 | − return; | |
| 65 | − bdi_lock_two(&old->wb, &dst->wb); | |
| 66 | − spin_lock(&inode->i_lock); | |
| 67 | − inode->i_data.backing_dev_info = dst; | |
| 68 | − if (inode->i_state & I_DIRTY) { | |
| 69 | − if (bdi_cap_writeback_dirty(dst) && !wb_has_dirty_io(&dst->wb)) | |
| 70 | − wakeup_bdi = true; | |
| 71 | − list_move(&inode->i_wb_list, &dst->wb.b_dirty); | |
| 59 | + while (true) { | |
| 60 | + spin_lock(&inode->i_lock); | |
| 61 | + if (!(inode->i_state & I_DIRTY)) { | |
| 62 | + inode->i_data.backing_dev_info = dst; | |
| 63 | + spin_unlock(&inode->i_lock); | |
| 64 | + return; | |
| 65 | + } | |
| 66 | + spin_unlock(&inode->i_lock); | |
| 67 | + WARN_ON_ONCE(write_inode_now(inode, true)); | |
| 72 | 68 | } |
| 73 | − spin_unlock(&inode->i_lock); | |
| 74 | − spin_unlock(&old->wb.list_lock); | |
| 75 | − spin_unlock(&dst->wb.list_lock); | |
| 76 | − | |
| 77 | − if (wakeup_bdi) | |
| 78 | − bdi_wakeup_thread_delayed(dst); | |
| 79 | 69 | } |
| 80 | 70 | |
| 81 | 71 | /* Kill _all_ buffers and pagecache , dirty or not.. */ |
| @@ -1179,8 +1169,6 @@ static int __blkdev_get(struct block_device *bdev, fmode_t mode, int for_part) · 1097 unmodified lines | ||
| 1179 | 1169 | if (!ret) { |
| 1180 | 1170 | bd_set_size(bdev,(loff_t)get_capacity(disk)<<9); |
| 1181 | 1171 | bdi = blk_get_backing_dev_info(bdev); |
| 1182 | − if (bdi == NULL) | |
| 1183 | − bdi = &default_backing_dev_info; | |
| 1184 | 1172 | bdev_inode_switch_bdi(bdev->bd_inode, bdi); |
| 1185 | 1173 | } |
| 1186 | 1174 | |
| @@ -1702,7 +1702,7 @@ static int btrfs_congested_fn(void *congested_data, int bdi_bits) · 1701 unmodified lines | ||
| 1702 | 1702 | if (!device->bdev) |
| 1703 | 1703 | continue; |
| 1704 | 1704 | bdi = blk_get_backing_dev_info(device->bdev); |
| 1705 | − if (bdi && bdi_congested(bdi, bdi_bits)) { | |
| 1705 | + if (bdi_congested(bdi, bdi_bits)) { | |
| 1706 | 1706 | ret = 1; |
| 1707 | 1707 | break; |
| 1708 | 1708 | } |
| @@ -220,11 +220,9 @@ ssize_t nfs_direct_IO(int rw, struct kiocb *iocb, struct iov_iter *iter, loff_t · 219 unmodified lines | ||
| 220 | 220 | #else |
| 221 | 221 | VM_BUG_ON(iocb->ki_nbytes != PAGE_SIZE); |
| 222 | 222 | |
| 223 | − if (rw == READ || rw == KERNEL_READ) | |
| 224 | − return nfs_file_direct_read(iocb, iter, pos, | |
| 225 | − rw == READ ? true : false); | |
| 226 | − return nfs_file_direct_write(iocb, iter, pos, | |
| 227 | − rw == WRITE ? true : false); | |
| 223 | + if (rw == READ) | |
| 224 | + return nfs_file_direct_read(iocb, iter, pos); | |
| 225 | + return nfs_file_direct_write(iocb, iter, pos); | |
| 228 | 226 | #endif /* CONFIG_NFS_SWAP */ |
| 229 | 227 | } |
| 230 | 228 | |
| @@ -510,7 +508,7 @@ static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq, · 279 unmodified lines | ||
| 510 | 508 | * cache. |
| 511 | 509 | */ |
| 512 | 510 | ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter, |
| 513 | − loff_t pos, bool uio) | |
| 511 | + loff_t pos) | |
| 514 | 512 | { |
| 515 | 513 | struct file *file = iocb->ki_filp; |
| 516 | 514 | struct address_space *mapping = file->f_mapping; |
| @@ -879,7 +877,7 @@ static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq, · 362 unmodified lines | ||
| 879 | 877 | * is no atomic O_APPEND write facility in the NFS protocol. |
| 880 | 878 | */ |
| 881 | 879 | ssize_t nfs_file_direct_write(struct kiocb *iocb, struct iov_iter *iter, |
| 882 | − loff_t pos, bool uio) | |
| 880 | + loff_t pos) | |
| 883 | 881 | { |
| 884 | 882 | ssize_t result = -EINVAL; |
| 885 | 883 | struct file *file = iocb->ki_filp; |
| @@ -172,7 +172,7 @@ nfs_file_read(struct kiocb *iocb, struct iov_iter *to) · 171 unmodified lines | ||
| 172 | 172 | ssize_t result; |
| 173 | 173 | |
| 174 | 174 | if (iocb->ki_filp->f_flags & O_DIRECT) |
| 175 | − return nfs_file_direct_read(iocb, to, iocb->ki_pos, true); | |
| 175 | + return nfs_file_direct_read(iocb, to, iocb->ki_pos); | |
| 176 | 176 | |
| 177 | 177 | dprintk("NFS: read(%pD2, %zu@%lu)\n", |
| 178 | 178 | iocb->ki_filp, |
| @@ -676,7 +676,7 @@ ssize_t nfs_file_write(struct kiocb *iocb, struct iov_iter *from) · 497 unmodified lines | ||
| 676 | 676 | return result; |
| 677 | 677 | |
| 678 | 678 | if (file->f_flags & O_DIRECT) |
| 679 | − return nfs_file_direct_write(iocb, from, pos, true); | |
| 679 | + return nfs_file_direct_write(iocb, from, pos); | |
| 680 | 680 | |
| 681 | 681 | dprintk("NFS: write(%pD2, %zu@%Ld)\n", |
| 682 | 682 | file, count, (long long) pos); |
| @@ -1670,8 +1670,6 @@ xfs_alloc_buftarg( · 1669 unmodified lines | ||
| 1670 | 1670 | btp->bt_dev = bdev->bd_dev; |
| 1671 | 1671 | btp->bt_bdev = bdev; |
| 1672 | 1672 | btp->bt_bdi = blk_get_backing_dev_info(bdev); |
| 1673 | − if (!btp->bt_bdi) | |
| 1674 | − goto error; | |
| 1675 | 1673 | |
| 1676 | 1674 | if (xfs_setsize_buftarg_early(btp, bdev)) |
| 1677 | 1675 | goto error; |
| @@ -28,12 +28,10 @@ struct dentry; · 27 unmodified lines | ||
| 28 | 28 | * Bits in backing_dev_info.state |
| 29 | 29 | */ |
| 30 | 30 | enum bdi_state { |
| 31 | − BDI_wb_alloc, /* Default embedded wb allocated */ | |
| 32 | 31 | BDI_async_congested, /* The async (write) queue is getting full */ |
| 33 | 32 | BDI_sync_congested, /* The sync queue is getting full */ |
| 34 | 33 | BDI_registered, /* bdi_register() was done */ |
| 35 | 34 | BDI_writeback_running, /* Writeback is in progress */ |
| 36 | − BDI_unused, /* Available bits start here */ | |
| 37 | 35 | }; |
| 38 | 36 | |
| 39 | 37 | typedef int (congested_fn)(void *, int); |
| @@ -50,7 +48,6 @@ enum bdi_stat_item { · 10 unmodified lines | ||
| 50 | 48 | |
| 51 | 49 | struct bdi_writeback { |
| 52 | 50 | struct backing_dev_info *bdi; /* our parent bdi */ |
| 53 | − unsigned int nr; | |
| 54 | 51 | |
| 55 | 52 | unsigned long last_old_flush; /* last old data flush */ |
| 56 | 53 | |
| @@ -124,7 +121,6 @@ void bdi_start_background_writeback(struct backing_dev_info *bdi); · 67 unmodified lines | ||
| 124 | 121 | void bdi_writeback_workfn(struct work_struct *work); |
| 125 | 122 | int bdi_has_dirty_io(struct backing_dev_info *bdi); |
| 126 | 123 | void bdi_wakeup_thread_delayed(struct backing_dev_info *bdi); |
| 127 | −void bdi_lock_two(struct bdi_writeback *wb1, struct bdi_writeback *wb2); | |
| 128 | 124 | |
| 129 | 125 | extern spinlock_t bdi_lock; |
| 130 | 126 | extern struct list_head bdi_list; |
| @@ -292,7 +292,24 @@ static inline unsigned bio_segments(struct bio *bio) · 291 unmodified lines | ||
| 292 | 292 | */ |
| 293 | 293 | #define bio_get(bio) atomic_inc(&(bio)->bi_cnt) |
| 294 | 294 | |
| 295 | +enum bip_flags { | |
| 296 | + BIP_BLOCK_INTEGRITY = 1 << 0, /* block layer owns integrity data */ | |
| 297 | + BIP_MAPPED_INTEGRITY = 1 << 1, /* ref tag has been remapped */ | |
| 298 | + BIP_CTRL_NOCHECK = 1 << 2, /* disable HBA integrity checking */ | |
| 299 | + BIP_DISK_NOCHECK = 1 << 3, /* disable disk integrity checking */ | |
| 300 | + BIP_IP_CHECKSUM = 1 << 4, /* IP checksum */ | |
| 301 | +}; | |
| 302 | + | |
| 295 | 303 | #if defined(CONFIG_BLK_DEV_INTEGRITY) |
| 304 | + | |
| 305 | +static inline struct bio_integrity_payload *bio_integrity(struct bio *bio) | |
| 306 | +{ | |
| 307 | + if (bio->bi_rw & REQ_INTEGRITY) | |
| 308 | + return bio->bi_integrity; | |
| 309 | + | |
| 310 | + return NULL; | |
| 311 | +} | |
| 312 | + | |
| 296 | 313 | /* |
| 297 | 314 | * bio integrity payload |
| 298 | 315 | */ |
| @@ -301,21 +318,40 @@ struct bio_integrity_payload { · 2 unmodified lines | ||
| 301 | 318 | |
| 302 | 319 | struct bvec_iter bip_iter; |
| 303 | 320 | |
| 304 | − /* kill - should just use bip_vec */ | |
| 305 | − void *bip_buf; /* generated integrity data */ | |
| 306 | − | |
| 307 | 321 | bio_end_io_t *bip_end_io; /* saved I/O completion fn */ |
| 308 | 322 | |
| 309 | 323 | unsigned short bip_slab; /* slab the bip came from */ |
| 310 | 324 | unsigned short bip_vcnt; /* # of integrity bio_vecs */ |
| 311 | 325 | unsigned short bip_max_vcnt; /* integrity bio_vec slots */ |
| 312 | − unsigned bip_owns_buf:1; /* should free bip_buf */ | |
| 326 | + unsigned short bip_flags; /* control flags */ | |
| 313 | 327 | |
| 314 | 328 | struct work_struct bip_work; /* I/O completion */ |
| 315 | 329 | |
| 316 | 330 | struct bio_vec *bip_vec; |
| 317 | 331 | struct bio_vec bip_inline_vecs[0];/* embedded bvec array */ |
| 318 | 332 | }; |
| 333 | + | |
| 334 | +static inline bool bio_integrity_flagged(struct bio *bio, enum bip_flags flag) | |
| 335 | +{ | |
| 336 | + struct bio_integrity_payload *bip = bio_integrity(bio); | |
| 337 | + | |
| 338 | + if (bip) | |
| 339 | + return bip->bip_flags & flag; | |
| 340 | + | |
| 341 | + return false; | |
| 342 | +} | |
| 343 | + | |
| 344 | +static inline sector_t bip_get_seed(struct bio_integrity_payload *bip) | |
| 345 | +{ | |
| 346 | + return bip->bip_iter.bi_sector; | |
| 347 | +} | |
| 348 | + | |
| 349 | +static inline void bip_set_seed(struct bio_integrity_payload *bip, | |
| 350 | + sector_t seed) | |
| 351 | +{ | |
| 352 | + bip->bip_iter.bi_sector = seed; | |
| 353 | +} | |
| 354 | + | |
| 319 | 355 | #endif /* CONFIG_BLK_DEV_INTEGRITY */ |
| 320 | 356 | |
| 321 | 357 | extern void bio_trim(struct bio *bio, int offset, int size); |
| @@ -342,6 +378,7 @@ static inline struct bio *bio_next_split(struct bio *bio, int sectors, · 20 unmodified lines | ||
| 342 | 378 | } |
| 343 | 379 | |
| 344 | 380 | extern struct bio_set *bioset_create(unsigned int, unsigned int); |
| 381 | +extern struct bio_set *bioset_create_nobvec(unsigned int, unsigned int); | |
| 345 | 382 | extern void bioset_free(struct bio_set *); |
| 346 | 383 | extern mempool_t *biovec_create_pool(int pool_entries); |
| 347 | 384 | |
| @@ -353,7 +390,6 @@ extern struct bio *bio_clone_fast(struct bio *, gfp_t, struct bio_set *); · 5 unmodified lines | ||
| 353 | 390 | extern struct bio *bio_clone_bioset(struct bio *, gfp_t, struct bio_set *bs); |
| 354 | 391 | |
| 355 | 392 | extern struct bio_set *fs_bio_set; |
| 356 | −unsigned int bio_integrity_tag_size(struct bio *bio); | |
| 357 | 393 | |
| 358 | 394 | static inline struct bio *bio_alloc(gfp_t gfp_mask, unsigned int nr_iovecs) |
| 359 | 395 | { |
| @@ -661,14 +697,10 @@ struct biovec_slab { · 301 unmodified lines | ||
| 661 | 697 | for_each_bio(_bio) \ |
| 662 | 698 | bip_for_each_vec(_bvl, _bio->bi_integrity, _iter) |
| 663 | 699 | |
| 664 | −#define bio_integrity(bio) (bio->bi_integrity != NULL) | |
| 665 | − | |
| 666 | 700 | extern struct bio_integrity_payload *bio_integrity_alloc(struct bio *, gfp_t, unsigned int); |
| 667 | 701 | extern void bio_integrity_free(struct bio *); |
| 668 | 702 | extern int bio_integrity_add_page(struct bio *, struct page *, unsigned int, unsigned int); |
| 669 | −extern int bio_integrity_enabled(struct bio *bio); | |
| 670 | −extern int bio_integrity_set_tag(struct bio *, void *, unsigned int); | |
| 671 | −extern int bio_integrity_get_tag(struct bio *, void *, unsigned int); | |
| 703 | +extern bool bio_integrity_enabled(struct bio *bio); | |
| 672 | 704 | extern int bio_integrity_prep(struct bio *); |
| 673 | 705 | extern void bio_integrity_endio(struct bio *, int); |
| 674 | 706 | extern void bio_integrity_advance(struct bio *, unsigned int); |
| @@ -680,14 +712,14 @@ extern void bio_integrity_init(void); · 5 unmodified lines | ||
| 680 | 712 | |
| 681 | 713 | #else /* CONFIG_BLK_DEV_INTEGRITY */ |
| 682 | 714 | |
| 683 | −static inline int bio_integrity(struct bio *bio) | |
| 715 | +static inline void *bio_integrity(struct bio *bio) | |
| 684 | 716 | { |
| 685 | − return 0; | |
| 717 | + return NULL; | |
| 686 | 718 | } |
| 687 | 719 | |
| 688 | −static inline int bio_integrity_enabled(struct bio *bio) | |
| 720 | +static inline bool bio_integrity_enabled(struct bio *bio) | |
| 689 | 721 | { |
| 690 | − return 0; | |
| 722 | + return false; | |
| 691 | 723 | } |
| 692 | 724 | |
| 693 | 725 | static inline int bioset_integrity_create(struct bio_set *bs, int pool_size) |
| @@ -733,6 +765,11 @@ static inline void bio_integrity_init(void) · 39 unmodified lines | ||
| 733 | 765 | return; |
| 734 | 766 | } |
| 735 | 767 | |
| 768 | +static inline bool bio_integrity_flagged(struct bio *bio, enum bip_flags flag) | |
| 769 | +{ | |
| 770 | + return false; | |
| 771 | +} | |
| 772 | + | |
| 736 | 773 | #endif /* CONFIG_BLK_DEV_INTEGRITY */ |
| 737 | 774 | |
| 738 | 775 | #endif /* CONFIG_BLOCK */ |
| @@ -4,6 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | #include <linux/blkdev.h> |
| 5 | 5 | |
| 6 | 6 | struct blk_mq_tags; |
| 7 | +struct blk_flush_queue; | |
| 7 | 8 | |
| 8 | 9 | struct blk_mq_cpu_notifier { |
| 9 | 10 | struct list_head list; |
| @@ -34,6 +35,7 @@ struct blk_mq_hw_ctx { · 24 unmodified lines | ||
| 34 | 35 | |
| 35 | 36 | struct request_queue *queue; |
| 36 | 37 | unsigned int queue_num; |
| 38 | + struct blk_flush_queue *fq; | |
| 37 | 39 | |
| 38 | 40 | void *driver_data; |
| 39 | 41 | |
| @@ -77,8 +79,9 @@ struct blk_mq_tag_set { · 37 unmodified lines | ||
| 77 | 79 | struct list_head tag_list; |
| 78 | 80 | }; |
| 79 | 81 | |
| 80 | −typedef int (queue_rq_fn)(struct blk_mq_hw_ctx *, struct request *); | |
| 82 | +typedef int (queue_rq_fn)(struct blk_mq_hw_ctx *, struct request *, bool); | |
| 81 | 83 | typedef struct blk_mq_hw_ctx *(map_queue_fn)(struct request_queue *, const int); |
| 84 | +typedef enum blk_eh_timer_return (timeout_fn)(struct request *, bool); | |
| 82 | 85 | typedef int (init_hctx_fn)(struct blk_mq_hw_ctx *, void *, unsigned int); |
| 83 | 86 | typedef void (exit_hctx_fn)(struct blk_mq_hw_ctx *, unsigned int); |
| 84 | 87 | typedef int (init_request_fn)(void *, struct request *, unsigned int, |
| @@ -86,6 +89,9 @@ typedef int (init_request_fn)(void *, struct request *, unsigned int, · 1 unmodified line | ||
| 86 | 89 | typedef void (exit_request_fn)(void *, struct request *, unsigned int, |
| 87 | 90 | unsigned int); |
| 88 | 91 | |
| 92 | +typedef void (busy_iter_fn)(struct blk_mq_hw_ctx *, struct request *, void *, | |
| 93 | + bool); | |
| 94 | + | |
| 89 | 95 | struct blk_mq_ops { |
| 90 | 96 | /* |
| 91 | 97 | * Queue request |
| @@ -100,7 +106,7 @@ struct blk_mq_ops { · 8 unmodified lines | ||
| 100 | 106 | /* |
| 101 | 107 | * Called on request timeout |
| 102 | 108 | */ |
| 103 | − rq_timed_out_fn *timeout; | |
| 109 | + timeout_fn *timeout; | |
| 104 | 110 | |
| 105 | 111 | softirq_done_fn *complete; |
| 106 | 112 | |
| @@ -115,6 +121,10 @@ struct blk_mq_ops { · 8 unmodified lines | ||
| 115 | 121 | /* |
| 116 | 122 | * Called for every command allocated by the block layer to allow |
| 117 | 123 | * the driver to set up driver specific data. |
| 124 | + * | |
| 125 | + * Tag greater than or equal to queue_depth is for setting up | |
| 126 | + * flush request. | |
| 127 | + * | |
| 118 | 128 | * Ditto for exit/teardown. |
| 119 | 129 | */ |
| 120 | 130 | init_request_fn *init_request; |
| @@ -160,8 +170,9 @@ struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag); · 39 unmodified lines | ||
| 160 | 170 | struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *, const int ctx_index); |
| 161 | 171 | struct blk_mq_hw_ctx *blk_mq_alloc_single_hw_queue(struct blk_mq_tag_set *, unsigned int, int); |
| 162 | 172 | |
| 163 | −void blk_mq_end_io(struct request *rq, int error); | |
| 164 | −void __blk_mq_end_io(struct request *rq, int error); | |
| 173 | +void blk_mq_start_request(struct request *rq); | |
| 174 | +void blk_mq_end_request(struct request *rq, int error); | |
| 175 | +void __blk_mq_end_request(struct request *rq, int error); | |
| 165 | 176 | |
| 166 | 177 | void blk_mq_requeue_request(struct request *rq); |
| 167 | 178 | void blk_mq_add_to_requeue_list(struct request *rq, bool at_head); |
| @@ -174,7 +185,8 @@ void blk_mq_stop_hw_queues(struct request_queue *q); · 6 unmodified lines | ||
| 174 | 185 | void blk_mq_start_hw_queues(struct request_queue *q); |
| 175 | 186 | void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async); |
| 176 | 187 | void blk_mq_delay_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs); |
| 177 | −void blk_mq_tag_busy_iter(struct blk_mq_tags *tags, void (*fn)(void *data, unsigned long *), void *data); | |
| 188 | +void blk_mq_tag_busy_iter(struct blk_mq_hw_ctx *hctx, busy_iter_fn *fn, | |
| 189 | + void *priv); | |
| 178 | 190 | |
| 179 | 191 | /* |
| 180 | 192 | * Driver command data is immediately after the request. So subtract request |
| @@ -78,9 +78,11 @@ struct bio { · 77 unmodified lines | ||
| 78 | 78 | struct io_context *bi_ioc; |
| 79 | 79 | struct cgroup_subsys_state *bi_css; |
| 80 | 80 | #endif |
| 81 | + union { | |
| 81 | 82 | #if defined(CONFIG_BLK_DEV_INTEGRITY) |
| 82 | − struct bio_integrity_payload *bi_integrity; /* data integrity */ | |
| 83 | + struct bio_integrity_payload *bi_integrity; /* data integrity */ | |
| 83 | 84 | #endif |
| 85 | + }; | |
| 84 | 86 | |
| 85 | 87 | unsigned short bi_vcnt; /* how many bio_vec's */ |
| 86 | 88 | |
| @@ -118,10 +120,8 @@ struct bio { · 31 unmodified lines | ||
| 118 | 120 | #define BIO_USER_MAPPED 6 /* contains user pages */ |
| 119 | 121 | #define BIO_EOPNOTSUPP 7 /* not supported */ |
| 120 | 122 | #define BIO_NULL_MAPPED 8 /* contains invalid user pages */ |
| 121 | −#define BIO_FS_INTEGRITY 9 /* fs owns integrity data, not block layer */ | |
| 122 | −#define BIO_QUIET 10 /* Make BIO Quiet */ | |
| 123 | −#define BIO_MAPPED_INTEGRITY 11/* integrity metadata has been remapped */ | |
| 124 | −#define BIO_SNAP_STABLE 12 /* bio data must be snapshotted during write */ | |
| 123 | +#define BIO_QUIET 9 /* Make BIO Quiet */ | |
| 124 | +#define BIO_SNAP_STABLE 10 /* bio data must be snapshotted during write */ | |
| 125 | 125 | |
| 126 | 126 | /* |
| 127 | 127 | * Flags starting here get preserved by bio_reset() - this includes |
| @@ -162,6 +162,7 @@ enum rq_flag_bits { · 34 unmodified lines | ||
| 162 | 162 | __REQ_WRITE_SAME, /* write same block many times */ |
| 163 | 163 | |
| 164 | 164 | __REQ_NOIDLE, /* don't anticipate more IO after this one */ |
| 165 | + __REQ_INTEGRITY, /* I/O includes block integrity payload */ | |
| 165 | 166 | __REQ_FUA, /* forced unit access */ |
| 166 | 167 | __REQ_FLUSH, /* request for cache flush */ |
| 167 | 168 | |
| @@ -186,9 +187,7 @@ enum rq_flag_bits { · 18 unmodified lines | ||
| 186 | 187 | __REQ_FLUSH_SEQ, /* request for flush sequence */ |
| 187 | 188 | __REQ_IO_STAT, /* account I/O stat */ |
| 188 | 189 | __REQ_MIXED_MERGE, /* merge of different types, fail separately */ |
| 189 | − __REQ_KERNEL, /* direct IO to kernel pages */ | |
| 190 | 190 | __REQ_PM, /* runtime pm request */ |
| 191 | − __REQ_END, /* last of chain of requests */ | |
| 192 | 191 | __REQ_HASHED, /* on IO scheduler merge hash */ |
| 193 | 192 | __REQ_MQ_INFLIGHT, /* track inflight for MQ */ |
| 194 | 193 | __REQ_NR_BITS, /* stops here */ |
| @@ -204,13 +203,14 @@ enum rq_flag_bits { · 9 unmodified lines | ||
| 204 | 203 | #define REQ_DISCARD (1ULL << __REQ_DISCARD) |
| 205 | 204 | #define REQ_WRITE_SAME (1ULL << __REQ_WRITE_SAME) |
| 206 | 205 | #define REQ_NOIDLE (1ULL << __REQ_NOIDLE) |
| 206 | +#define REQ_INTEGRITY (1ULL << __REQ_INTEGRITY) | |
| 207 | 207 | |
| 208 | 208 | #define REQ_FAILFAST_MASK \ |
| 209 | 209 | (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER) |
| 210 | 210 | #define REQ_COMMON_MASK \ |
| 211 | 211 | (REQ_WRITE | REQ_FAILFAST_MASK | REQ_SYNC | REQ_META | REQ_PRIO | \ |
| 212 | 212 | REQ_DISCARD | REQ_WRITE_SAME | REQ_NOIDLE | REQ_FLUSH | REQ_FUA | \ |
| 213 | − REQ_SECURE) | |
| 213 | + REQ_SECURE | REQ_INTEGRITY) | |
| 214 | 214 | #define REQ_CLONE_MASK REQ_COMMON_MASK |
| 215 | 215 | |
| 216 | 216 | #define BIO_NO_ADVANCE_ITER_MASK (REQ_DISCARD|REQ_WRITE_SAME) |
| @@ -240,9 +240,7 @@ enum rq_flag_bits { · 23 unmodified lines | ||
| 240 | 240 | #define REQ_IO_STAT (1ULL << __REQ_IO_STAT) |
| 241 | 241 | #define REQ_MIXED_MERGE (1ULL << __REQ_MIXED_MERGE) |
| 242 | 242 | #define REQ_SECURE (1ULL << __REQ_SECURE) |
| 243 | −#define REQ_KERNEL (1ULL << __REQ_KERNEL) | |
| 244 | 243 | #define REQ_PM (1ULL << __REQ_PM) |
| 245 | −#define REQ_END (1ULL << __REQ_END) | |
| 246 | 244 | #define REQ_HASHED (1ULL << __REQ_HASHED) |
| 247 | 245 | #define REQ_MQ_INFLIGHT (1ULL << __REQ_MQ_INFLIGHT) |
| 248 | 246 | |
| @@ -36,6 +36,7 @@ struct request; · 35 unmodified lines | ||
| 36 | 36 | struct sg_io_hdr; |
| 37 | 37 | struct bsg_job; |
| 38 | 38 | struct blkcg_gq; |
| 39 | +struct blk_flush_queue; | |
| 39 | 40 | |
| 40 | 41 | #define BLKDEV_MIN_RQ 4 |
| 41 | 42 | #define BLKDEV_MAX_RQ 128 /* Default maximum */ |
| @@ -455,14 +456,7 @@ struct request_queue { · 413 unmodified lines | ||
| 455 | 456 | */ |
| 456 | 457 | unsigned int flush_flags; |
| 457 | 458 | unsigned int flush_not_queueable:1; |
| 458 | − unsigned int flush_queue_delayed:1; | |
| 459 | − unsigned int flush_pending_idx:1; | |
| 460 | − unsigned int flush_running_idx:1; | |
| 461 | − unsigned long flush_pending_since; | |
| 462 | − struct list_head flush_queue[2]; | |
| 463 | − struct list_head flush_data_in_flight; | |
| 464 | − struct request *flush_rq; | |
| 465 | − spinlock_t mq_flush_lock; | |
| 459 | + struct blk_flush_queue *fq; | |
| 466 | 460 | |
| 467 | 461 | struct list_head requeue_list; |
| 468 | 462 | spinlock_t requeue_lock; |
| @@ -865,7 +859,7 @@ extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *, · 396 unmodified lines | ||
| 865 | 859 | |
| 866 | 860 | static inline struct request_queue *bdev_get_queue(struct block_device *bdev) |
| 867 | 861 | { |
| 868 | − return bdev->bd_disk->queue; | |
| 862 | + return bdev->bd_disk->queue; /* this is never NULL */ | |
| 869 | 863 | } |
| 870 | 864 | |
| 871 | 865 | /* |
| @@ -1285,10 +1279,9 @@ static inline int queue_alignment_offset(struct request_queue *q) · 413 unmodified lines | ||
| 1285 | 1279 | static inline int queue_limit_alignment_offset(struct queue_limits *lim, sector_t sector) |
| 1286 | 1280 | { |
| 1287 | 1281 | unsigned int granularity = max(lim->physical_block_size, lim->io_min); |
| 1288 | − unsigned int alignment = (sector << 9) & (granularity - 1); | |
| 1282 | + unsigned int alignment = sector_div(sector, granularity >> 9) << 9; | |
| 1289 | 1283 | |
| 1290 | − return (granularity + lim->alignment_offset - alignment) | |
| 1291 | − & (granularity - 1); | |
| 1284 | + return (granularity + lim->alignment_offset - alignment) % granularity; | |
| 1292 | 1285 | } |
| 1293 | 1286 | |
| 1294 | 1287 | static inline int bdev_alignment_offset(struct block_device *bdev) |
| @@ -1464,32 +1457,31 @@ static inline uint64_t rq_io_start_time_ns(struct request *req) · 169 unmodified lines | ||
| 1464 | 1457 | |
| 1465 | 1458 | #if defined(CONFIG_BLK_DEV_INTEGRITY) |
| 1466 | 1459 | |
| 1467 | −#define INTEGRITY_FLAG_READ 2 /* verify data integrity on read */ | |
| 1468 | −#define INTEGRITY_FLAG_WRITE 4 /* generate data integrity on write */ | |
| 1460 | +enum blk_integrity_flags { | |
| 1461 | + BLK_INTEGRITY_VERIFY = 1 << 0, | |
| 1462 | + BLK_INTEGRITY_GENERATE = 1 << 1, | |
| 1463 | + BLK_INTEGRITY_DEVICE_CAPABLE = 1 << 2, | |
| 1464 | + BLK_INTEGRITY_IP_CHECKSUM = 1 << 3, | |
| 1465 | +}; | |
| 1469 | 1466 | |
| 1470 | −struct blk_integrity_exchg { | |
| 1467 | +struct blk_integrity_iter { | |
| 1471 | 1468 | void *prot_buf; |
| 1472 | 1469 | void *data_buf; |
| 1473 | − sector_t sector; | |
| 1470 | + sector_t seed; | |
| 1474 | 1471 | unsigned int data_size; |
| 1475 | − unsigned short sector_size; | |
| 1472 | + unsigned short interval; | |
| 1476 | 1473 | const char *disk_name; |
| 1477 | 1474 | }; |
| 1478 | 1475 | |
| 1479 | −typedef void (integrity_gen_fn) (struct blk_integrity_exchg *); | |
| 1480 | −typedef int (integrity_vrfy_fn) (struct blk_integrity_exchg *); | |
| 1481 | −typedef void (integrity_set_tag_fn) (void *, void *, unsigned int); | |
| 1482 | −typedef void (integrity_get_tag_fn) (void *, void *, unsigned int); | |
| 1476 | +typedef int (integrity_processing_fn) (struct blk_integrity_iter *); | |
| 1483 | 1477 | |
| 1484 | 1478 | struct blk_integrity { |
| 1485 | − integrity_gen_fn *generate_fn; | |
| 1486 | − integrity_vrfy_fn *verify_fn; | |
| 1487 | − integrity_set_tag_fn *set_tag_fn; | |
| 1488 | − integrity_get_tag_fn *get_tag_fn; | |
| 1479 | + integrity_processing_fn *generate_fn; | |
| 1480 | + integrity_processing_fn *verify_fn; | |
| 1489 | 1481 | |
| 1490 | 1482 | unsigned short flags; |
| 1491 | 1483 | unsigned short tuple_size; |
| 1492 | − unsigned short sector_size; | |
| 1484 | + unsigned short interval; | |
| 1493 | 1485 | unsigned short tag_size; |
| 1494 | 1486 | |
| 1495 | 1487 | const char *name; |
| @@ -1504,10 +1496,10 @@ extern int blk_integrity_compare(struct gendisk *, struct gendisk *); · 8 unmodified lines | ||
| 1504 | 1496 | extern int blk_rq_map_integrity_sg(struct request_queue *, struct bio *, |
| 1505 | 1497 | struct scatterlist *); |
| 1506 | 1498 | extern int blk_rq_count_integrity_sg(struct request_queue *, struct bio *); |
| 1507 | −extern int blk_integrity_merge_rq(struct request_queue *, struct request *, | |
| 1508 | − struct request *); | |
| 1509 | −extern int blk_integrity_merge_bio(struct request_queue *, struct request *, | |
| 1510 | − struct bio *); | |
| 1499 | +extern bool blk_integrity_merge_rq(struct request_queue *, struct request *, | |
| 1500 | + struct request *); | |
| 1501 | +extern bool blk_integrity_merge_bio(struct request_queue *, struct request *, | |
| 1502 | + struct bio *); | |
| 1511 | 1503 | |
| 1512 | 1504 | static inline |
| 1513 | 1505 | struct blk_integrity *bdev_get_integrity(struct block_device *bdev) |
| @@ -1520,12 +1512,9 @@ static inline struct blk_integrity *blk_get_integrity(struct gendisk *disk) · 6 unmodified lines | ||
| 1520 | 1512 | return disk->integrity; |
| 1521 | 1513 | } |
| 1522 | 1514 | |
| 1523 | −static inline int blk_integrity_rq(struct request *rq) | |
| 1515 | +static inline bool blk_integrity_rq(struct request *rq) | |
| 1524 | 1516 | { |
| 1525 | − if (rq->bio == NULL) | |
| 1526 | − return 0; | |
| 1527 | − | |
| 1528 | − return bio_integrity(rq->bio); | |
| 1517 | + return rq->cmd_flags & REQ_INTEGRITY; | |
| 1529 | 1518 | } |
| 1530 | 1519 | |
| 1531 | 1520 | static inline void blk_queue_max_integrity_segments(struct request_queue *q, |
| @@ -1590,15 +1579,15 @@ static inline unsigned short queue_max_integrity_segments(struct request_queue * · 58 unmodified lines | ||
| 1590 | 1579 | { |
| 1591 | 1580 | return 0; |
| 1592 | 1581 | } |
| 1593 | −static inline int blk_integrity_merge_rq(struct request_queue *rq, | |
| 1594 | − struct request *r1, | |
| 1595 | − struct request *r2) | |
| 1582 | +static inline bool blk_integrity_merge_rq(struct request_queue *rq, | |
| 1583 | + struct request *r1, | |
| 1584 | + struct request *r2) | |
| 1596 | 1585 | { |
| 1597 | 1586 | return 0; |
| 1598 | 1587 | } |
| 1599 | −static inline int blk_integrity_merge_bio(struct request_queue *rq, | |
| 1600 | − struct request *r, | |
| 1601 | − struct bio *b) | |
| 1588 | +static inline bool blk_integrity_merge_bio(struct request_queue *rq, | |
| 1589 | + struct request *r, | |
| 1590 | + struct bio *b) | |
| 1602 | 1591 | { |
| 1603 | 1592 | return 0; |
| 1604 | 1593 | } |
| @@ -6,7 +6,8 @@ · 5 unmodified lines | ||
| 6 | 6 | #define CRC_T10DIF_DIGEST_SIZE 2 |
| 7 | 7 | #define CRC_T10DIF_BLOCK_SIZE 1 |
| 8 | 8 | |
| 9 | −__u16 crc_t10dif_generic(__u16 crc, const unsigned char *buffer, size_t len); | |
| 10 | −__u16 crc_t10dif(unsigned char const *, size_t); | |
| 9 | +extern __u16 crc_t10dif_generic(__u16 crc, const unsigned char *buffer, | |
| 10 | + size_t len); | |
| 11 | +extern __u16 crc_t10dif(unsigned char const *, size_t); | |
| 11 | 12 | |
| 12 | 13 | #endif |
| @@ -192,8 +192,6 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, · 191 unmodified lines | ||
| 192 | 192 | #define READ 0 |
| 193 | 193 | #define WRITE RW_MASK |
| 194 | 194 | #define READA RWA_MASK |
| 195 | −#define KERNEL_READ (READ|REQ_KERNEL) | |
| 196 | −#define KERNEL_WRITE (WRITE|REQ_KERNEL) | |
| 197 | 195 | |
| 198 | 196 | #define READ_SYNC (READ | REQ_SYNC) |
| 199 | 197 | #define WRITE_SYNC (WRITE | REQ_SYNC | REQ_NOIDLE) |
| @@ -448,10 +448,10 @@ static inline struct rpc_cred *nfs_file_cred(struct file *file) · 447 unmodified lines | ||
| 448 | 448 | extern ssize_t nfs_direct_IO(int, struct kiocb *, struct iov_iter *, loff_t); |
| 449 | 449 | extern ssize_t nfs_file_direct_read(struct kiocb *iocb, |
| 450 | 450 | struct iov_iter *iter, |
| 451 | − loff_t pos, bool uio); | |
| 451 | + loff_t pos); | |
| 452 | 452 | extern ssize_t nfs_file_direct_write(struct kiocb *iocb, |
| 453 | 453 | struct iov_iter *iter, |
| 454 | − loff_t pos, bool uio); | |
| 454 | + loff_t pos); | |
| 455 | 455 | |
| 456 | 456 | /* |
| 457 | 457 | * linux/fs/nfs/dir.c |
| @@ -0,0 +1,22 @@ | ||
| 1 | +#ifndef _LINUX_T10_PI_H | |
| 2 | +#define _LINUX_T10_PI_H | |
| 3 | + | |
| 4 | +#include <linux/types.h> | |
| 5 | +#include <linux/blkdev.h> | |
| 6 | + | |
| 7 | +/* | |
| 8 | + * T10 Protection Information tuple. | |
| 9 | + */ | |
| 10 | +struct t10_pi_tuple { | |
| 11 | + __be16 guard_tag; /* Checksum */ | |
| 12 | + __be16 app_tag; /* Opaque storage */ | |
| 13 | + __be32 ref_tag; /* Target LBA or indirect LBA */ | |
| 14 | +}; | |
| 15 | + | |
| 16 | + | |
| 17 | +extern struct blk_integrity t10_pi_type1_crc; | |
| 18 | +extern struct blk_integrity t10_pi_type1_ip; | |
| 19 | +extern struct blk_integrity t10_pi_type3_crc; | |
| 20 | +extern struct blk_integrity t10_pi_type3_ip; | |
| 21 | + | |
| 22 | +#endif | |
| @@ -10,9 +10,10 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <scsi/scsi_device.h> |
| 11 | 11 | |
| 12 | 12 | struct Scsi_Host; |
| 13 | −struct scsi_device; | |
| 14 | 13 | struct scsi_driver; |
| 15 | 14 | |
| 15 | +#include <scsi/scsi_device.h> | |
| 16 | + | |
| 16 | 17 | /* |
| 17 | 18 | * MAX_COMMAND_SIZE is: |
| 18 | 19 | * The longest fixed-length SCSI CDB as per the SCSI standard. |
| @@ -81,6 +82,7 @@ struct scsi_cmnd { · 62 unmodified lines | ||
| 81 | 82 | |
| 82 | 83 | unsigned char prot_op; |
| 83 | 84 | unsigned char prot_type; |
| 85 | + unsigned char prot_flags; | |
| 84 | 86 | |
| 85 | 87 | unsigned short cmd_len; |
| 86 | 88 | enum dma_data_direction sc_data_direction; |
| @@ -252,6 +254,14 @@ static inline unsigned char scsi_get_prot_op(struct scsi_cmnd *scmd) · 165 unmodified lines | ||
| 252 | 254 | return scmd->prot_op; |
| 253 | 255 | } |
| 254 | 256 | |
| 257 | +enum scsi_prot_flags { | |
| 258 | + SCSI_PROT_TRANSFER_PI = 1 << 0, | |
| 259 | + SCSI_PROT_GUARD_CHECK = 1 << 1, | |
| 260 | + SCSI_PROT_REF_CHECK = 1 << 2, | |
| 261 | + SCSI_PROT_REF_INCREMENT = 1 << 3, | |
| 262 | + SCSI_PROT_IP_CHECKSUM = 1 << 4, | |
| 263 | +}; | |
| 264 | + | |
| 255 | 265 | /* |
| 256 | 266 | * The controller usually does not know anything about the target it |
| 257 | 267 | * is communicating with. However, when DIX is enabled the controller |
| @@ -280,6 +290,17 @@ static inline sector_t scsi_get_lba(struct scsi_cmnd *scmd) · 22 unmodified lines | ||
| 280 | 290 | return blk_rq_pos(scmd->request); |
| 281 | 291 | } |
| 282 | 292 | |
| 293 | +static inline unsigned int scsi_prot_interval(struct scsi_cmnd *scmd) | |
| 294 | +{ | |
| 295 | + return scmd->device->sector_size; | |
| 296 | +} | |
| 297 | + | |
| 298 | +static inline u32 scsi_prot_ref_tag(struct scsi_cmnd *scmd) | |
| 299 | +{ | |
| 300 | + return blk_rq_pos(scmd->request) >> | |
| 301 | + (ilog2(scsi_prot_interval(scmd)) - 9) & 0xffffffff; | |
| 302 | +} | |
| 303 | + | |
| 283 | 304 | static inline unsigned scsi_prot_sg_count(struct scsi_cmnd *cmd) |
| 284 | 305 | { |
| 285 | 306 | return cmd->prot_sdb ? cmd->prot_sdb->table.nents : 0; |
| @@ -316,17 +337,12 @@ static inline void set_driver_byte(struct scsi_cmnd *cmd, char status) · 30 unmodified lines | ||
| 316 | 337 | static inline unsigned scsi_transfer_length(struct scsi_cmnd *scmd) |
| 317 | 338 | { |
| 318 | 339 | unsigned int xfer_len = scsi_out(scmd)->length; |
| 319 | − unsigned int prot_op = scsi_get_prot_op(scmd); | |
| 320 | − unsigned int sector_size = scmd->device->sector_size; | |
| 340 | + unsigned int prot_interval = scsi_prot_interval(scmd); | |
| 321 | 341 | |
| 322 | − switch (prot_op) { | |
| 323 | − case SCSI_PROT_NORMAL: | |
| 324 | − case SCSI_PROT_WRITE_STRIP: | |
| 325 | − case SCSI_PROT_READ_INSERT: | |
| 326 | − return xfer_len; | |
| 327 | − } | |
| 342 | + if (scmd->prot_flags & SCSI_PROT_TRANSFER_PI) | |
| 343 | + xfer_len += (xfer_len >> ilog2(prot_interval)) * 8; | |
| 328 | 344 | |
| 329 | − return xfer_len + (xfer_len >> ilog2(sector_size)) * 8; | |
| 345 | + return xfer_len; | |
| 330 | 346 | } |
| 331 | 347 | |
| 332 | 348 | #endif /* _SCSI_SCSI_CMND_H */ |
| @@ -40,7 +40,7 @@ LIST_HEAD(bdi_list); · 39 unmodified lines | ||
| 40 | 40 | /* bdi_wq serves all asynchronous writeback tasks */ |
| 41 | 41 | struct workqueue_struct *bdi_wq; |
| 42 | 42 | |
| 43 | −void bdi_lock_two(struct bdi_writeback *wb1, struct bdi_writeback *wb2) | |
| 43 | +static void bdi_lock_two(struct bdi_writeback *wb1, struct bdi_writeback *wb2) | |
| 44 | 44 | { |
| 45 | 45 | if (wb1 < wb2) { |
| 46 | 46 | spin_lock(&wb1->list_lock); |
| @@ -376,13 +376,7 @@ static void bdi_wb_shutdown(struct backing_dev_info *bdi) · 329 unmodified lines | ||
| 376 | 376 | mod_delayed_work(bdi_wq, &bdi->wb.dwork, 0); |
| 377 | 377 | flush_delayed_work(&bdi->wb.dwork); |
| 378 | 378 | WARN_ON(!list_empty(&bdi->work_list)); |
| 379 | − | |
| 380 | − /* | |
| 381 | − * This shouldn't be necessary unless @bdi for some reason has | |
| 382 | − * unflushed dirty IO after work_list is drained. Do it anyway | |
| 383 | − * just in case. | |
| 384 | − */ | |
| 385 | − cancel_delayed_work_sync(&bdi->wb.dwork); | |
| 379 | + WARN_ON(delayed_work_pending(&bdi->wb.dwork)); | |
| 386 | 380 | } |
| 387 | 381 | |
| 388 | 382 | /* |
| @@ -402,21 +396,15 @@ static void bdi_prune_sb(struct backing_dev_info *bdi) · 13 unmodified lines | ||
| 402 | 396 | |
| 403 | 397 | void bdi_unregister(struct backing_dev_info *bdi) |
| 404 | 398 | { |
| 405 | − struct device *dev = bdi->dev; | |
| 406 | − | |
| 407 | − if (dev) { | |
| 399 | + if (bdi->dev) { | |
| 408 | 400 | bdi_set_min_ratio(bdi, 0); |
| 409 | 401 | trace_writeback_bdi_unregister(bdi); |
| 410 | 402 | bdi_prune_sb(bdi); |
| 411 | 403 | |
| 412 | 404 | bdi_wb_shutdown(bdi); |
| 413 | 405 | bdi_debug_unregister(bdi); |
| 414 | − | |
| 415 | − spin_lock_bh(&bdi->wb_lock); | |
| 406 | + device_unregister(bdi->dev); | |
| 416 | 407 | bdi->dev = NULL; |
| 417 | − spin_unlock_bh(&bdi->wb_lock); | |
| 418 | − | |
| 419 | − device_unregister(dev); | |
| 420 | 408 | } |
| 421 | 409 | } |
| 422 | 410 | EXPORT_SYMBOL(bdi_unregister); |
| @@ -487,8 +475,17 @@ void bdi_destroy(struct backing_dev_info *bdi) · 64 unmodified lines | ||
| 487 | 475 | int i; |
| 488 | 476 | |
| 489 | 477 | /* |
| 490 | − * Splice our entries to the default_backing_dev_info, if this | |
| 491 | − * bdi disappears | |
| 478 | + * Splice our entries to the default_backing_dev_info. This | |
| 479 | + * condition shouldn't happen. @wb must be empty at this point and | |
| 480 | + * dirty inodes on it might cause other issues. This workaround is | |
| 481 | + * added by ce5f8e779519 ("writeback: splice dirty inode entries to | |
| 482 | + * default bdi on bdi_destroy()") without root-causing the issue. | |
| 483 | + * | |
| 484 | + * http://lkml.kernel.org/g/1253038617-30204-11-git-send-email-jens.axboe@oracle.com | |
| 485 | + * http://thread.gmane.org/gmane.linux.file-systems/35341/focus=35350 | |
| 486 | + * | |
| 487 | + * We should probably add WARN_ON() to find out whether it still | |
| 488 | + * happens and track it down if so. | |
| 492 | 489 | */ |
| 493 | 490 | if (bdi_has_dirty_io(bdi)) { |
| 494 | 491 | struct bdi_writeback *dst = &default_backing_dev_info.wb; |
| @@ -503,12 +500,7 @@ void bdi_destroy(struct backing_dev_info *bdi) · 8 unmodified lines | ||
| 503 | 500 | |
| 504 | 501 | bdi_unregister(bdi); |
| 505 | 502 | |
| 506 | − /* | |
| 507 | − * If bdi_unregister() had already been called earlier, the dwork | |
| 508 | − * could still be pending because bdi_prune_sb() can race with the | |
| 509 | − * bdi_wakeup_thread_delayed() calls from __mark_inode_dirty(). | |
| 510 | − */ | |
| 511 | − cancel_delayed_work_sync(&bdi->wb.dwork); | |
| 503 | + WARN_ON(delayed_work_pending(&bdi->wb.dwork)); | |
| 512 | 504 | |
| 513 | 505 | for (i = 0; i < NR_BDI_STAT_ITEMS; i++) |
| 514 | 506 | percpu_counter_destroy(&bdi->bdi_stat[i]); |