History

Merge branch 'for-3.18/core' of git://git.kernel.dk/linux-block

Pull core block layer changes from Jens Axboe: "This is the core block IO pull request for 3.18. Apart from the new and improved flush machinery for blk-mq, this is all mostly bug fixes and cleanups. - blk-mq timeout updates and fixes from Christoph. - Removal of REQ_END, also from Christoph. We pass it through the ->queue_rq() hook for blk-mq instead, freeing up one of the request bits. The space was overly tight on 32-bit, so Martin also killed REQ_KERNEL since it's no longer used. - blk integrity updates and fixes from Martin and Gu Zheng. - Update to the flush machinery for blk-mq from Ming Lei. Now we have a per hardware context flush request, which both cleans up the code should scale better for flush intensive workloads on blk-mq. - Improve the error printing, from Rob Elliott. - Backing device improvements and cleanups from Tejun. - Fixup of a misplaced rq_complete() tracepoint from Hannes. - Make blk_get_request() return error pointers, fixing up issues where we NULL deref when a device goes bad or missing. From Joe Lawrence. - Prep work for drastically reducing the memory consumption of dm devices from Junichi Nomura. This allows creating clone bio sets without preallocating a lot of memory. - Fix a blk-mq hang on certain combinations of queue depths and hardware queues from me. - Limit memory consumption for blk-mq devices for crash dump scenarios and drivers that use crazy high depths (certain SCSI shared tag setups). We now just use a single queue and limited depth for that" * 'for-3.18/core' of git://git.kernel.dk/linux-block: (58 commits) block: Remove REQ_KERNEL blk-mq: allocate cpumask on the home node bio-integrity: remove the needless fail handle of bip_slab creating block: include func name in __get_request prints block: make blk_update_request print prefix match ratelimited prefix blk-merge: don't compute bi_phys_segments from bi_vcnt for cloned bio block: fix alignment_offset math that assumes io_min is a power-of-2 blk-mq: Make bt_clear_tag() easier to read blk-mq: fix potential hang if rolling wakeup depth is too high block: add bioset_create_nobvec() block: use bio_clone_fast() in blk_rq_prep_clone() block: misplaced rq_complete tracepoint sd: Honor block layer integrity handling flags block: Replace strnicmp with strncasecmp block: Add T10 Protection Information functions block: Don't merge requests if integrity flags differ block: Integrity checksum flag block: Relocate bio integrity flags block: Add a disk flag to block integrity profile block: Add prefix to block integrity profile flags ...

Linus Torvalds committed 12 years ago

Commit d3dc3662 parents 511c41d e19a8a065 files · +1210 −1172Browse files

Unified · Split

65 files
  1. Documentation/ABI/testing/sysfs-block +8 −0
  2. Documentation/block/data-integrity.txt +5 −49
  3. block/Kconfig +1 −0
  4. block/Makefile +2 −2
  5. block/bio-integrity.c +68 −203
  6. block/bio.c +44 −17
  7. block/blk-cgroup.c +0 −2
  8. block/blk-cgroup.h +0 −3
  9. block/blk-core.c +31 −36
  10. block/blk-flush.c +98 −49
  11. block/blk-integrity.c +65 −38
  12. block/blk-merge.c +9 −5
  13. block/blk-mq-tag.c +22 −33
  14. block/blk-mq.c +187 −177
  15. block/blk-mq.h +2 −1
  16. block/blk-settings.c +2 −2
  17. block/blk-sysfs.c +2 −2
  18. block/blk-timeout.c +8 −7
  19. block/blk.h +33 −4
  20. block/bsg.c +5 −4
  21. block/cfq-iosched.c +5 −5
  22. block/compat_ioctl.c +0 −4
  23. block/ioctl.c +0 −4
  24. block/partitions/mac.c +4 −4
  25. block/scsi_ioctl.c +6 −5
  26. block/t10-pi.c +197 −0
  27. drivers/block/mtip32xx/mtip32xx.c +6 −3
  28. drivers/block/null_blk.c +5 −2
  29. drivers/block/paride/pd.c +2 −0
  30. drivers/block/pktcdvd.c +2 −0
  31. drivers/block/sx8.c +1 −1
  32. drivers/block/virtio_blk.c +5 −3
  33. drivers/cdrom/cdrom.c +2 −2
  34. drivers/ide/ide-park.c +1 −1
  35. drivers/scsi/Kconfig +0 −1
  36. drivers/scsi/device_handler/scsi_dh_alua.c +1 −1
  37. drivers/scsi/device_handler/scsi_dh_emc.c +1 −1
  38. drivers/scsi/device_handler/scsi_dh_hp_sw.c +2 −2
  39. drivers/scsi/device_handler/scsi_dh_rdac.c +1 −1
  40. drivers/scsi/osd/osd_initiator.c +2 −2
  41. drivers/scsi/osst.c +1 −1
  42. drivers/scsi/scsi_error.c +2 −0
  43. drivers/scsi/scsi_lib.c +19 −5
  44. drivers/scsi/sd.c +43 −30
  45. drivers/scsi/sd.h +64 −2
  46. drivers/scsi/sd_dif.c +48 −305
  47. drivers/scsi/sg.c +2 −2
  48. drivers/scsi/st.c +1 −1
  49. drivers/target/target_core_pscsi.c +1 −1
  50. fs/block_dev.c +11 −23
  51. fs/btrfs/disk-io.c +1 −1
  52. fs/nfs/direct.c +5 −7
  53. fs/nfs/file.c +2 −2
  54. fs/xfs/xfs_buf.c +0 −2
  55. include/linux/backing-dev.h +0 −4
  56. include/linux/bio.h +51 −14
  57. include/linux/blk-mq.h +17 −5
  58. include/linux/blk_types.h +8 −10
  59. include/linux/blkdev.h +30 −41
  60. include/linux/crc-t10dif.h +3 −2
  61. include/linux/fs.h +0 −2
  62. include/linux/nfs_fs.h +2 −2
  63. include/linux/t10-pi.h +22 −0
  64. include/scsi/scsi_cmnd.h +26 −10
  65. mm/backing-dev.c +16 −24
Documentation/ABI/testing/sysfs-block+8−0
@@ -53,6 +53,14 @@ Description: · 52 unmodified lines
5353 512 bytes of data.
5454
5555
56+What: /sys/block/<disk>/integrity/device_is_integrity_capable
57+Date: July 2014
58+Contact: Martin K. Petersen <martin.petersen@oracle.com>
59+Description:
60+ Indicates whether a storage device is capable of storing
61+ integrity metadata. Set if the device is T10 PI-capable.
62+
63+
5664 What: /sys/block/<disk>/integrity/write_generate
5765 Date: June 2008
5866 Contact: Martin K. Petersen <martin.petersen@oracle.com>
Documentation/block/data-integrity.txt+5−49
@@ -129,11 +129,11 @@ interface for this is being worked on. · 128 unmodified lines
129129 4.1 BIO
130130
131131 The data integrity patches add a new field to struct bio when
132−CONFIG_BLK_DEV_INTEGRITY is enabled. bio->bi_integrity is a pointer
133−to a struct bip which contains the bio integrity payload. Essentially
134−a bip is a trimmed down struct bio which holds a bio_vec containing
135−the integrity metadata and the required housekeeping information (bvec
136−pool, vector count, etc.)
132+CONFIG_BLK_DEV_INTEGRITY is enabled. bio_integrity(bio) returns a
133+pointer to a struct bip which contains the bio integrity payload.
134+Essentially a bip is a trimmed down struct bio which holds a bio_vec
135+containing the integrity metadata and the required housekeeping
136+information (bvec pool, vector count, etc.)
137137
138138 A kernel subsystem can enable data integrity protection on a bio by
139139 calling bio_integrity_alloc(bio). This will allocate and attach the
@@ -192,16 +192,6 @@ will require extra work due to the application tag. · 52 unmodified lines
192192 supported by the block device.
193193
194194
195− int bdev_integrity_enabled(block_device, int rw);
196−
197− bdev_integrity_enabled() will return 1 if the block device
198− supports integrity metadata transfer for the data direction
199− specified in 'rw'.
200−
201− bdev_integrity_enabled() honors the write_generate and
202− read_verify flags in sysfs and will respond accordingly.
203−
204−
205195 int bio_integrity_prep(bio);
206196
207197 To generate IMD for WRITE and to set up buffers for READ, the
@@ -216,36 +206,6 @@ will require extra work due to the application tag. · 8 unmodified lines
216206 bio_integrity_enabled() returned 1.
217207
218208
219− int bio_integrity_tag_size(bio);
220−
221− If the filesystem wants to use the application tag space it will
222− first have to find out how much storage space is available.
223− Because tag space is generally limited (usually 2 bytes per
224− sector regardless of sector size), the integrity framework
225− supports interleaving the information between the sectors in an
226− I/O.
227−
228− Filesystems can call bio_integrity_tag_size(bio) to find out how
229− many bytes of storage are available for that particular bio.
230−
231− Another option is bdev_get_tag_size(block_device) which will
232− return the number of available bytes per hardware sector.
233−
234−
235− int bio_integrity_set_tag(bio, void *tag_buf, len);
236−
237− After a successful return from bio_integrity_prep(),
238− bio_integrity_set_tag() can be used to attach an opaque tag
239− buffer to a bio. Obviously this only makes sense if the I/O is
240− a WRITE.
241−
242−
243− int bio_integrity_get_tag(bio, void *tag_buf, len);
244−
245− Similarly, at READ I/O completion time the filesystem can
246− retrieve the tag buffer using bio_integrity_get_tag().
247−
248−
249209 5.3 PASSING EXISTING INTEGRITY METADATA
250210
251211 Filesystems that either generate their own integrity metadata or
@@ -298,8 +258,6 @@ will require extra work due to the application tag. · 46 unmodified lines
298258 .name = "STANDARDSBODY-TYPE-VARIANT-CSUM",
299259 .generate_fn = my_generate_fn,
300260 .verify_fn = my_verify_fn,
301− .get_tag_fn = my_get_tag_fn,
302− .set_tag_fn = my_set_tag_fn,
303261 .tuple_size = sizeof(struct my_tuple_size),
304262 .tag_size = <tag bytes per hw sector>,
305263 };
@@ -321,7 +279,5 @@ will require extra work due to the application tag. · 15 unmodified lines
321279 are available per hardware sector. For DIF this is either 2 or
322280 0 depending on the value of the Control Mode Page ATO bit.
323281
324− See 6.2 for a description of get_tag_fn and set_tag_fn.
325−
326282 ----------------------------------------------------------------------
327283 2007-12-24 Martin K. Petersen <martin.petersen@oracle.com>
block/Kconfig+1−0
@@ -77,6 +77,7 @@ config BLK_DEV_BSGLIB · 76 unmodified lines
7777
7878 config BLK_DEV_INTEGRITY
7979 bool "Block layer data integrity support"
80+ select CRC_T10DIF if BLK_DEV_INTEGRITY
8081 ---help---
8182 Some storage devices allow extra information to be
8283 stored/retrieved to help protect the data. The block layer
block/Makefile+2−2
@@ -20,6 +20,6 @@ obj-$(CONFIG_IOSCHED_DEADLINE) += deadline-iosched.o · 19 unmodified lines
2020 obj-$(CONFIG_IOSCHED_CFQ) += cfq-iosched.o
2121
2222 obj-$(CONFIG_BLOCK_COMPAT) += compat_ioctl.o
23−obj-$(CONFIG_BLK_DEV_INTEGRITY) += blk-integrity.o
2423 obj-$(CONFIG_BLK_CMDLINE_PARSER) += cmdline-parser.o
25−obj-$(CONFIG_BLK_DEV_INTEGRITY) += bio-integrity.o
24+obj-$(CONFIG_BLK_DEV_INTEGRITY) += bio-integrity.o blk-integrity.o t10-pi.o
25+
block/bio-integrity.c+68−203
@@ -79,6 +79,7 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, · 78 unmodified lines
7979 bip->bip_slab = idx;
8080 bip->bip_bio = bio;
8181 bio->bi_integrity = bip;
82+ bio->bi_rw |= REQ_INTEGRITY;
8283
8384 return bip;
8485 err:
@@ -96,11 +97,12 @@ EXPORT_SYMBOL(bio_integrity_alloc); · 11 unmodified lines
9697 */
9798 void bio_integrity_free(struct bio *bio)
9899 {
99− struct bio_integrity_payload *bip = bio->bi_integrity;
100+ struct bio_integrity_payload *bip = bio_integrity(bio);
100101 struct bio_set *bs = bio->bi_pool;
101102
102− if (bip->bip_owns_buf)
103− kfree(bip->bip_buf);
103+ if (bip->bip_flags & BIP_BLOCK_INTEGRITY)
104+ kfree(page_address(bip->bip_vec->bv_page) +
105+ bip->bip_vec->bv_offset);
104106
105107 if (bs) {
106108 if (bip->bip_slab != BIO_POOL_NONE)
@@ -128,7 +130,7 @@ EXPORT_SYMBOL(bio_integrity_free); · 21 unmodified lines
128130 int bio_integrity_add_page(struct bio *bio, struct page *page,
129131 unsigned int len, unsigned int offset)
130132 {
131− struct bio_integrity_payload *bip = bio->bi_integrity;
133+ struct bio_integrity_payload *bip = bio_integrity(bio);
132134 struct bio_vec *iv;
133135
134136 if (bip->bip_vcnt >= bip->bip_max_vcnt) {
@@ -147,24 +149,6 @@ int bio_integrity_add_page(struct bio *bio, struct page *page, · 12 unmodified lines
147149 }
148150 EXPORT_SYMBOL(bio_integrity_add_page);
149151
150−static int bdev_integrity_enabled(struct block_device *bdev, int rw)
151−{
152− struct blk_integrity *bi = bdev_get_integrity(bdev);
153−
154− if (bi == NULL)
155− return 0;
156−
157− if (rw == READ && bi->verify_fn != NULL &&
158− (bi->flags & INTEGRITY_FLAG_READ))
159− return 1;
160−
161− if (rw == WRITE && bi->generate_fn != NULL &&
162− (bi->flags & INTEGRITY_FLAG_WRITE))
163− return 1;
164−
165− return 0;
166−}
167−
168152 /**
169153 * bio_integrity_enabled - Check whether integrity can be passed
170154 * @bio: bio to check
@@ -174,199 +158,92 @@ static int bdev_integrity_enabled(struct block_device *bdev, int rw) · 3 unmodified lines
174158 * set prior to calling. The functions honors the write_generate and
175159 * read_verify flags in sysfs.
176160 */
177−int bio_integrity_enabled(struct bio *bio)
161+bool bio_integrity_enabled(struct bio *bio)
178162 {
163+ struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
164+
179165 if (!bio_is_rw(bio))
180− return 0;
166+ return false;
181167
182168 /* Already protected? */
183169 if (bio_integrity(bio))
184− return 0;
170+ return false;
171+
172+ if (bi == NULL)
173+ return false;
174+
175+ if (bio_data_dir(bio) == READ && bi->verify_fn != NULL &&
176+ (bi->flags & BLK_INTEGRITY_VERIFY))
177+ return true;
178+
179+ if (bio_data_dir(bio) == WRITE && bi->generate_fn != NULL &&
180+ (bi->flags & BLK_INTEGRITY_GENERATE))
181+ return true;
185182
186− return bdev_integrity_enabled(bio->bi_bdev, bio_data_dir(bio));
183+ return false;
187184 }
188185 EXPORT_SYMBOL(bio_integrity_enabled);
189186
190187 /**
191− * bio_integrity_hw_sectors - Convert 512b sectors to hardware ditto
188+ * bio_integrity_intervals - Return number of integrity intervals for a bio
192189 * @bi: blk_integrity profile for device
193− * @sectors: Number of 512 sectors to convert
190+ * @sectors: Size of the bio in 512-byte sectors
194191 *
195192 * Description: The block layer calculates everything in 512 byte
196− * sectors but integrity metadata is done in terms of the hardware
197− * sector size of the storage device. Convert the block layer sectors
198− * to physical sectors.
193+ * sectors but integrity metadata is done in terms of the data integrity
194+ * interval size of the storage device. Convert the block layer sectors
195+ * to the appropriate number of integrity intervals.
199196 */
200−static inline unsigned int bio_integrity_hw_sectors(struct blk_integrity *bi,
201− unsigned int sectors)
197+static inline unsigned int bio_integrity_intervals(struct blk_integrity *bi,
198+ unsigned int sectors)
202199 {
203− /* At this point there are only 512b or 4096b DIF/EPP devices */
204− if (bi->sector_size == 4096)
205− return sectors >>= 3;
206−
207− return sectors;
200+ return sectors >> (ilog2(bi->interval) - 9);
208201 }
209202
210203 static inline unsigned int bio_integrity_bytes(struct blk_integrity *bi,
211204 unsigned int sectors)
212205 {
213− return bio_integrity_hw_sectors(bi, sectors) * bi->tuple_size;
206+ return bio_integrity_intervals(bi, sectors) * bi->tuple_size;
214207 }
215208
216209 /**
217− * bio_integrity_tag_size - Retrieve integrity tag space
218− * @bio: bio to inspect
219− *
220− * Description: Returns the maximum number of tag bytes that can be
221− * attached to this bio. Filesystems can use this to determine how
222− * much metadata to attach to an I/O.
223− */
224−unsigned int bio_integrity_tag_size(struct bio *bio)
225−{
226− struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
227−
228− BUG_ON(bio->bi_iter.bi_size == 0);
229−
230− return bi->tag_size * (bio->bi_iter.bi_size / bi->sector_size);
231−}
232−EXPORT_SYMBOL(bio_integrity_tag_size);
233−
234−static int bio_integrity_tag(struct bio *bio, void *tag_buf, unsigned int len,
235− int set)
236−{
237− struct bio_integrity_payload *bip = bio->bi_integrity;
238− struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
239− unsigned int nr_sectors;
240−
241− BUG_ON(bip->bip_buf == NULL);
242−
243− if (bi->tag_size == 0)
244− return -1;
245−
246− nr_sectors = bio_integrity_hw_sectors(bi,
247− DIV_ROUND_UP(len, bi->tag_size));
248−
249− if (nr_sectors * bi->tuple_size > bip->bip_iter.bi_size) {
250− printk(KERN_ERR "%s: tag too big for bio: %u > %u\n", __func__,
251− nr_sectors * bi->tuple_size, bip->bip_iter.bi_size);
252− return -1;
253− }
254−
255− if (set)
256− bi->set_tag_fn(bip->bip_buf, tag_buf, nr_sectors);
257− else
258− bi->get_tag_fn(bip->bip_buf, tag_buf, nr_sectors);
259−
260− return 0;
261−}
262−
263−/**
264− * bio_integrity_set_tag - Attach a tag buffer to a bio
265− * @bio: bio to attach buffer to
266− * @tag_buf: Pointer to a buffer containing tag data
267− * @len: Length of the included buffer
268− *
269− * Description: Use this function to tag a bio by leveraging the extra
270− * space provided by devices formatted with integrity protection. The
271− * size of the integrity buffer must be <= to the size reported by
272− * bio_integrity_tag_size().
273− */
274−int bio_integrity_set_tag(struct bio *bio, void *tag_buf, unsigned int len)
275−{
276− BUG_ON(bio_data_dir(bio) != WRITE);
277−
278− return bio_integrity_tag(bio, tag_buf, len, 1);
279−}
280−EXPORT_SYMBOL(bio_integrity_set_tag);
281−
282−/**
283− * bio_integrity_get_tag - Retrieve a tag buffer from a bio
284− * @bio: bio to retrieve buffer from
285− * @tag_buf: Pointer to a buffer for the tag data
286− * @len: Length of the target buffer
287− *
288− * Description: Use this function to retrieve the tag buffer from a
289− * completed I/O. The size of the integrity buffer must be <= to the
290− * size reported by bio_integrity_tag_size().
291− */
292−int bio_integrity_get_tag(struct bio *bio, void *tag_buf, unsigned int len)
293−{
294− BUG_ON(bio_data_dir(bio) != READ);
295−
296− return bio_integrity_tag(bio, tag_buf, len, 0);
297−}
298−EXPORT_SYMBOL(bio_integrity_get_tag);
299−
300−/**
301− * bio_integrity_generate_verify - Generate/verify integrity metadata for a bio
210+ * bio_integrity_process - Process integrity metadata for a bio
302211 * @bio: bio to generate/verify integrity metadata for
303− * @operate: operate number, 1 for generate, 0 for verify
212+ * @proc_fn: Pointer to the relevant processing function
304213 */
305−static int bio_integrity_generate_verify(struct bio *bio, int operate)
214+static int bio_integrity_process(struct bio *bio,
215+ integrity_processing_fn *proc_fn)
306216 {
307217 struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
308− struct blk_integrity_exchg bix;
218+ struct blk_integrity_iter iter;
309219 struct bio_vec *bv;
310− sector_t sector;
311− unsigned int sectors, ret = 0, i;
312− void *prot_buf = bio->bi_integrity->bip_buf;
313−
314− if (operate)
315− sector = bio->bi_iter.bi_sector;
316− else
317− sector = bio->bi_integrity->bip_iter.bi_sector;
220+ struct bio_integrity_payload *bip = bio_integrity(bio);
221+ unsigned int i, ret = 0;
222+ void *prot_buf = page_address(bip->bip_vec->bv_page) +
223+ bip->bip_vec->bv_offset;
318224
319− bix.disk_name = bio->bi_bdev->bd_disk->disk_name;
320− bix.sector_size = bi->sector_size;
225+ iter.disk_name = bio->bi_bdev->bd_disk->disk_name;
226+ iter.interval = bi->interval;
227+ iter.seed = bip_get_seed(bip);
228+ iter.prot_buf = prot_buf;
321229
322230 bio_for_each_segment_all(bv, bio, i) {
323231 void *kaddr = kmap_atomic(bv->bv_page);
324− bix.data_buf = kaddr + bv->bv_offset;
325− bix.data_size = bv->bv_len;
326− bix.prot_buf = prot_buf;
327− bix.sector = sector;
328−
329− if (operate)
330− bi->generate_fn(&bix);
331− else {
332− ret = bi->verify_fn(&bix);
333− if (ret) {
334− kunmap_atomic(kaddr);
335− return ret;
336− }
337− }
338232
339− sectors = bv->bv_len / bi->sector_size;
340− sector += sectors;
341− prot_buf += sectors * bi->tuple_size;
233+ iter.data_buf = kaddr + bv->bv_offset;
234+ iter.data_size = bv->bv_len;
235+
236+ ret = proc_fn(&iter);
237+ if (ret) {
238+ kunmap_atomic(kaddr);
239+ return ret;
240+ }
342241
343242 kunmap_atomic(kaddr);
344243 }
345244 return ret;
346245 }
347246
348−/**
349− * bio_integrity_generate - Generate integrity metadata for a bio
350− * @bio: bio to generate integrity metadata for
351− *
352− * Description: Generates integrity metadata for a bio by calling the
353− * block device's generation callback function. The bio must have a
354− * bip attached with enough room to accommodate the generated
355− * integrity metadata.
356− */
357−static void bio_integrity_generate(struct bio *bio)
358−{
359− bio_integrity_generate_verify(bio, 1);
360−}
361−
362−static inline unsigned short blk_integrity_tuple_size(struct blk_integrity *bi)
363−{
364− if (bi)
365− return bi->tuple_size;
366−
367− return 0;
368−}
369−
370247 /**
371248 * bio_integrity_prep - Prepare bio for integrity I/O
372249 * @bio: bio to prepare
@@ -387,17 +264,17 @@ int bio_integrity_prep(struct bio *bio) · 14 unmodified lines
387264 unsigned long start, end;
388265 unsigned int len, nr_pages;
389266 unsigned int bytes, offset, i;
390− unsigned int sectors;
267+ unsigned int intervals;
391268
392269 bi = bdev_get_integrity(bio->bi_bdev);
393270 q = bdev_get_queue(bio->bi_bdev);
394271 BUG_ON(bi == NULL);
395272 BUG_ON(bio_integrity(bio));
396273
397− sectors = bio_integrity_hw_sectors(bi, bio_sectors(bio));
274+ intervals = bio_integrity_intervals(bi, bio_sectors(bio));
398275
399276 /* Allocate kernel buffer for protection data */
400− len = sectors * blk_integrity_tuple_size(bi);
277+ len = intervals * bi->tuple_size;
401278 buf = kmalloc(len, GFP_NOIO | q->bounce_gfp);
402279 if (unlikely(buf == NULL)) {
403280 printk(KERN_ERR "could not allocate integrity buffer\n");
@@ -416,10 +293,12 @@ int bio_integrity_prep(struct bio *bio) · 12 unmodified lines
416293 return -EIO;
417294 }
418295
419− bip->bip_owns_buf = 1;
420− bip->bip_buf = buf;
296+ bip->bip_flags |= BIP_BLOCK_INTEGRITY;
421297 bip->bip_iter.bi_size = len;
422− bip->bip_iter.bi_sector = bio->bi_iter.bi_sector;
298+ bip_set_seed(bip, bio->bi_iter.bi_sector);
299+
300+ if (bi->flags & BLK_INTEGRITY_IP_CHECKSUM)
301+ bip->bip_flags |= BIP_IP_CHECKSUM;
423302
424303 /* Map it */
425304 offset = offset_in_page(buf);
@@ -455,25 +334,12 @@ int bio_integrity_prep(struct bio *bio) · 29 unmodified lines
455334
456335 /* Auto-generate integrity metadata if this is a write */
457336 if (bio_data_dir(bio) == WRITE)
458− bio_integrity_generate(bio);
337+ bio_integrity_process(bio, bi->generate_fn);
459338
460339 return 0;
461340 }
462341 EXPORT_SYMBOL(bio_integrity_prep);
463342
464−/**
465− * bio_integrity_verify - Verify integrity metadata for a bio
466− * @bio: bio to verify
467− *
468− * Description: This function is called to verify the integrity of a
469− * bio. The data in the bio io_vec is compared to the integrity
470− * metadata returned by the HBA.
471− */
472−static int bio_integrity_verify(struct bio *bio)
473−{
474− return bio_integrity_generate_verify(bio, 0);
475−}
476−
477343 /**
478344 * bio_integrity_verify_fn - Integrity I/O completion worker
479345 * @work: Work struct stored in bio to be verified
@@ -487,9 +353,10 @@ static void bio_integrity_verify_fn(struct work_struct *work) · 7 unmodified lines
487353 struct bio_integrity_payload *bip =
488354 container_of(work, struct bio_integrity_payload, bip_work);
489355 struct bio *bio = bip->bip_bio;
356+ struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
490357 int error;
491358
492− error = bio_integrity_verify(bio);
359+ error = bio_integrity_process(bio, bi->verify_fn);
493360
494361 /* Restore original bio completion handler */
495362 bio->bi_end_io = bip->bip_end_io;
@@ -510,7 +377,7 @@ static void bio_integrity_verify_fn(struct work_struct *work) · 14 unmodified lines
510377 */
511378 void bio_integrity_endio(struct bio *bio, int error)
512379 {
513− struct bio_integrity_payload *bip = bio->bi_integrity;
380+ struct bio_integrity_payload *bip = bio_integrity(bio);
514381
515382 BUG_ON(bip->bip_bio != bio);
516383
@@ -541,7 +408,7 @@ EXPORT_SYMBOL(bio_integrity_endio); · 24 unmodified lines
541408 */
542409 void bio_integrity_advance(struct bio *bio, unsigned int bytes_done)
543410 {
544− struct bio_integrity_payload *bip = bio->bi_integrity;
411+ struct bio_integrity_payload *bip = bio_integrity(bio);
545412 struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
546413 unsigned bytes = bio_integrity_bytes(bi, bytes_done >> 9);
547414
@@ -563,7 +430,7 @@ EXPORT_SYMBOL(bio_integrity_advance); · 15 unmodified lines
563430 void bio_integrity_trim(struct bio *bio, unsigned int offset,
564431 unsigned int sectors)
565432 {
566− struct bio_integrity_payload *bip = bio->bi_integrity;
433+ struct bio_integrity_payload *bip = bio_integrity(bio);
567434 struct blk_integrity *bi = bdev_get_integrity(bio->bi_bdev);
568435
569436 bio_integrity_advance(bio, offset << 9);
@@ -582,7 +449,7 @@ EXPORT_SYMBOL(bio_integrity_trim); · 12 unmodified lines
582449 int bio_integrity_clone(struct bio *bio, struct bio *bio_src,
583450 gfp_t gfp_mask)
584451 {
585− struct bio_integrity_payload *bip_src = bio_src->bi_integrity;
452+ struct bio_integrity_payload *bip_src = bio_integrity(bio_src);
586453 struct bio_integrity_payload *bip;
587454
588455 BUG_ON(bip_src == NULL);
@@ -646,6 +513,4 @@ void __init bio_integrity_init(void) · 57 unmodified lines
646513 sizeof(struct bio_integrity_payload) +
647514 sizeof(struct bio_vec) * BIP_INLINE_VECS,
648515 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL);
649− if (!bip_slab)
650− panic("Failed to create slab\n");
651516 }
block/bio.c+44−17
@@ -428,6 +428,9 @@ struct bio *bio_alloc_bioset(gfp_t gfp_mask, int nr_iovecs, struct bio_set *bs) · 427 unmodified lines
428428 front_pad = 0;
429429 inline_vecs = nr_iovecs;
430430 } else {
431+ /* should not use nobvec bioset for nr_iovecs > 0 */
432+ if (WARN_ON_ONCE(!bs->bvec_pool && nr_iovecs > 0))
433+ return NULL;
431434 /*
432435 * generic_make_request() converts recursion to iteration; this
433436 * means if we're running beneath it, any bios we allocate and
@@ -1900,20 +1903,9 @@ void bioset_free(struct bio_set *bs) · 1466 unmodified lines
19001903 }
19011904 EXPORT_SYMBOL(bioset_free);
19021905
1903−/**
1904− * bioset_create - Create a bio_set
1905− * @pool_size: Number of bio and bio_vecs to cache in the mempool
1906− * @front_pad: Number of bytes to allocate in front of the returned bio
1907− *
1908− * Description:
1909− * Set up a bio_set to be used with @bio_alloc_bioset. Allows the caller
1910− * to ask for a number of bytes to be allocated in front of the bio.
1911− * Front pad allocation is useful for embedding the bio inside
1912− * another structure, to avoid allocating extra data to go with the bio.
1913− * Note that the bio must be embedded at the END of that structure always,
1914− * or things will break badly.
1915− */
1916−struct bio_set *bioset_create(unsigned int pool_size, unsigned int front_pad)
1906+static struct bio_set *__bioset_create(unsigned int pool_size,
1907+ unsigned int front_pad,
1908+ bool create_bvec_pool)
19171909 {
19181910 unsigned int back_pad = BIO_INLINE_VECS * sizeof(struct bio_vec);
19191911 struct bio_set *bs;
@@ -1938,9 +1930,11 @@ struct bio_set *bioset_create(unsigned int pool_size, unsigned int front_pad) · 18 unmodified lines
19381930 if (!bs->bio_pool)
19391931 goto bad;
19401932
1941− bs->bvec_pool = biovec_create_pool(pool_size);
1942− if (!bs->bvec_pool)
1943− goto bad;
1933+ if (create_bvec_pool) {
1934+ bs->bvec_pool = biovec_create_pool(pool_size);
1935+ if (!bs->bvec_pool)
1936+ goto bad;
1937+ }
19441938
19451939 bs->rescue_workqueue = alloc_workqueue("bioset", WQ_MEM_RECLAIM, 0);
19461940 if (!bs->rescue_workqueue)
@@ -1951,8 +1945,41 @@ bad: · 4 unmodified lines
19511945 bioset_free(bs);
19521946 return NULL;
19531947 }
1948+
1949+/**
1950+ * bioset_create - Create a bio_set
1951+ * @pool_size: Number of bio and bio_vecs to cache in the mempool
1952+ * @front_pad: Number of bytes to allocate in front of the returned bio
1953+ *
1954+ * Description:
1955+ * Set up a bio_set to be used with @bio_alloc_bioset. Allows the caller
1956+ * to ask for a number of bytes to be allocated in front of the bio.
1957+ * Front pad allocation is useful for embedding the bio inside
1958+ * another structure, to avoid allocating extra data to go with the bio.
1959+ * Note that the bio must be embedded at the END of that structure always,
1960+ * or things will break badly.
1961+ */
1962+struct bio_set *bioset_create(unsigned int pool_size, unsigned int front_pad)
1963+{
1964+ return __bioset_create(pool_size, front_pad, true);
1965+}
19541966 EXPORT_SYMBOL(bioset_create);
19551967
1968+/**
1969+ * bioset_create_nobvec - Create a bio_set without bio_vec mempool
1970+ * @pool_size: Number of bio to cache in the mempool
1971+ * @front_pad: Number of bytes to allocate in front of the returned bio
1972+ *
1973+ * Description:
1974+ * Same functionality as bioset_create() except that mempool is not
1975+ * created for bio_vecs. Saving some memory for bio_clone_fast() users.
1976+ */
1977+struct bio_set *bioset_create_nobvec(unsigned int pool_size, unsigned int front_pad)
1978+{
1979+ return __bioset_create(pool_size, front_pad, false);
1980+}
1981+EXPORT_SYMBOL(bioset_create_nobvec);
1982+
19561983 #ifdef CONFIG_BLK_CGROUP
19571984 /**
19581985 * bio_associate_current - associate a bio with %current
block/blk-cgroup.c+0−2
@@ -822,7 +822,6 @@ static void blkcg_css_free(struct cgroup_subsys_state *css) · 821 unmodified lines
822822 static struct cgroup_subsys_state *
823823 blkcg_css_alloc(struct cgroup_subsys_state *parent_css)
824824 {
825− static atomic64_t id_seq = ATOMIC64_INIT(0);
826825 struct blkcg *blkcg;
827826
828827 if (!parent_css) {
@@ -836,7 +835,6 @@ blkcg_css_alloc(struct cgroup_subsys_state *parent_css) · 7 unmodified lines
836835
837836 blkcg->cfq_weight = CFQ_WEIGHT_DEFAULT;
838837 blkcg->cfq_leaf_weight = CFQ_WEIGHT_DEFAULT;
839− blkcg->id = atomic64_inc_return(&id_seq); /* root is 0, start from 1 */
840838 done:
841839 spin_lock_init(&blkcg->lock);
842840 INIT_RADIX_TREE(&blkcg->blkg_tree, GFP_ATOMIC);
block/blk-cgroup.h+0−3
@@ -50,9 +50,6 @@ struct blkcg { · 49 unmodified lines
5050 struct blkcg_gq *blkg_hint;
5151 struct hlist_head blkg_list;
5252
53− /* for policies to test whether associated blkcg has changed */
54− uint64_t id;
55−
5653 /* TODO: per-policy storage in blkcg */
5754 unsigned int cfq_weight; /* belongs to cfq */
5855 unsigned int cfq_leaf_weight;
block/blk-core.c+31−36
@@ -83,18 +83,14 @@ void blk_queue_congestion_threshold(struct request_queue *q) · 82 unmodified lines
8383 * @bdev: device
8484 *
8585 * Locates the passed device's request queue and returns the address of its
86− * backing_dev_info
87− *
88− * Will return NULL if the request queue cannot be located.
86+ * backing_dev_info. This function can only be called if @bdev is opened
87+ * and the return value is never NULL.
8988 */
9089 struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev)
9190 {
92− struct backing_dev_info *ret = NULL;
9391 struct request_queue *q = bdev_get_queue(bdev);
9492
95− if (q)
96− ret = &q->backing_dev_info;
97− return ret;
93+ return &q->backing_dev_info;
9894 }
9995 EXPORT_SYMBOL(blk_get_backing_dev_info);
10096
@@ -394,11 +390,13 @@ static void __blk_drain_queue(struct request_queue *q, bool drain_all) · 293 unmodified lines
394390 * be drained. Check all the queues and counters.
395391 */
396392 if (drain_all) {
393+ struct blk_flush_queue *fq = blk_get_flush_queue(q, NULL);
397394 drain |= !list_empty(&q->queue_head);
398395 for (i = 0; i < 2; i++) {
399396 drain |= q->nr_rqs[i];
400397 drain |= q->in_flight[i];
401− drain |= !list_empty(&q->flush_queue[i]);
398+ if (fq)
399+ drain |= !list_empty(&fq->flush_queue[i]);
402400 }
403401 }
404402
@@ -604,9 +602,6 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) · 199 unmodified lines
604602 #ifdef CONFIG_BLK_CGROUP
605603 INIT_LIST_HEAD(&q->blkg_list);
606604 #endif
607− INIT_LIST_HEAD(&q->flush_queue[0]);
608− INIT_LIST_HEAD(&q->flush_queue[1]);
609− INIT_LIST_HEAD(&q->flush_data_in_flight);
610605 INIT_DELAYED_WORK(&q->delay_work, blk_delay_work);
611606
612607 kobject_init(&q->kobj, &blk_queue_ktype);
@@ -709,8 +704,8 @@ blk_init_allocated_queue(struct request_queue *q, request_fn_proc *rfn, · 96 unmodified lines
709704 if (!q)
710705 return NULL;
711706
712− q->flush_rq = kzalloc(sizeof(struct request), GFP_KERNEL);
713− if (!q->flush_rq)
707+ q->fq = blk_alloc_flush_queue(q, NUMA_NO_NODE, 0);
708+ if (!q->fq)
714709 return NULL;
715710
716711 if (blk_init_rl(&q->root_rl, q, GFP_KERNEL))
@@ -746,7 +741,7 @@ blk_init_allocated_queue(struct request_queue *q, request_fn_proc *rfn, · 29 unmodified lines
746741 return q;
747742
748743 fail:
749− kfree(q->flush_rq);
744+ blk_free_flush_queue(q->fq);
750745 return NULL;
751746 }
752747 EXPORT_SYMBOL(blk_init_allocated_queue);
@@ -934,8 +929,8 @@ static struct io_context *rq_ioc(struct bio *bio) · 181 unmodified lines
934929 * pressure or if @q is dead.
935930 *
936931 * Must be called with @q->queue_lock held and,
937− * Returns %NULL on failure, with @q->queue_lock held.
938− * Returns !%NULL on success, with @q->queue_lock *not held*.
932+ * Returns ERR_PTR on failure, with @q->queue_lock held.
933+ * Returns request pointer on success, with @q->queue_lock *not held*.
939934 */
940935 static struct request *__get_request(struct request_list *rl, int rw_flags,
941936 struct bio *bio, gfp_t gfp_mask)
@@ -949,7 +944,7 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, · 7 unmodified lines
949944 int may_queue;
950945
951946 if (unlikely(blk_queue_dying(q)))
952− return NULL;
947+ return ERR_PTR(-ENODEV);
953948
954949 may_queue = elv_may_queue(q, rw_flags);
955950 if (may_queue == ELV_MQUEUE_NO)
@@ -974,7 +969,7 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, · 18 unmodified lines
974969 * process is not a "batcher", and not
975970 * exempted by the IO scheduler
976971 */
977− return NULL;
972+ return ERR_PTR(-ENOMEM);
978973 }
979974 }
980975 }
@@ -992,7 +987,7 @@ static struct request *__get_request(struct request_list *rl, int rw_flags, · 11 unmodified lines
992987 * allocated with any setting of ->nr_requests
993988 */
994989 if (rl->count[is_sync] >= (3 * q->nr_requests / 2))
995− return NULL;
990+ return ERR_PTR(-ENOMEM);
996991
997992 q->nr_rqs[is_sync]++;
998993 rl->count[is_sync]++;
@@ -1065,8 +1060,8 @@ fail_elvpriv: · 66 unmodified lines
10651060 * shouldn't stall IO. Treat this request as !elvpriv. This will
10661061 * disturb iosched and blkcg but weird is bettern than dead.
10671062 */
1068− printk_ratelimited(KERN_WARNING "%s: request aux data allocation failed, iosched may be disturbed\n",
1069− dev_name(q->backing_dev_info.dev));
1063+ printk_ratelimited(KERN_WARNING "%s: dev %s: request aux data allocation failed, iosched may be disturbed\n",
1064+ __func__, dev_name(q->backing_dev_info.dev));
10701065
10711066 rq->cmd_flags &= ~REQ_ELVPRIV;
10721067 rq->elv.icq = NULL;
@@ -1097,7 +1092,7 @@ fail_alloc: · 24 unmodified lines
10971092 rq_starved:
10981093 if (unlikely(rl->count[is_sync] == 0))
10991094 rl->starved[is_sync] = 1;
1100− return NULL;
1095+ return ERR_PTR(-ENOMEM);
11011096 }
11021097
11031098 /**
@@ -1111,8 +1106,8 @@ rq_starved: · 7 unmodified lines
11111106 * function keeps retrying under memory pressure and fails iff @q is dead.
11121107 *
11131108 * Must be called with @q->queue_lock held and,
1114− * Returns %NULL on failure, with @q->queue_lock held.
1115− * Returns !%NULL on success, with @q->queue_lock *not held*.
1109+ * Returns ERR_PTR on failure, with @q->queue_lock held.
1110+ * Returns request pointer on success, with @q->queue_lock *not held*.
11161111 */
11171112 static struct request *get_request(struct request_queue *q, int rw_flags,
11181113 struct bio *bio, gfp_t gfp_mask)
@@ -1125,12 +1120,12 @@ static struct request *get_request(struct request_queue *q, int rw_flags, · 6 unmodified lines
11251120 rl = blk_get_rl(q, bio); /* transferred to @rq on success */
11261121 retry:
11271122 rq = __get_request(rl, rw_flags, bio, gfp_mask);
1128− if (rq)
1123+ if (!IS_ERR(rq))
11291124 return rq;
11301125
11311126 if (!(gfp_mask & __GFP_WAIT) || unlikely(blk_queue_dying(q))) {
11321127 blk_put_rl(rl);
1133− return NULL;
1128+ return rq;
11341129 }
11351130
11361131 /* wait on @rl and retry */
@@ -1167,7 +1162,7 @@ static struct request *blk_old_get_request(struct request_queue *q, int rw, · 30 unmodified lines
11671162
11681163 spin_lock_irq(q->queue_lock);
11691164 rq = get_request(q, rw, NULL, gfp_mask);
1170− if (!rq)
1165+ if (IS_ERR(rq))
11711166 spin_unlock_irq(q->queue_lock);
11721167 /* q->queue_lock is unlocked at this point */
11731168
@@ -1219,8 +1214,8 @@ struct request *blk_make_request(struct request_queue *q, struct bio *bio, · 45 unmodified lines
12191214 {
12201215 struct request *rq = blk_get_request(q, bio_data_dir(bio), gfp_mask);
12211216
1222− if (unlikely(!rq))
1223− return ERR_PTR(-ENOMEM);
1217+ if (IS_ERR(rq))
1218+ return rq;
12241219
12251220 blk_rq_set_block_pc(rq);
12261221
@@ -1614,8 +1609,8 @@ get_rq: · 387 unmodified lines
16141609 * Returns with the queue unlocked.
16151610 */
16161611 req = get_request(q, rw_flags, bio, GFP_NOIO);
1617− if (unlikely(!req)) {
1618− bio_endio(bio, -ENODEV); /* @q is dead */
1612+ if (IS_ERR(req)) {
1613+ bio_endio(bio, PTR_ERR(req)); /* @q is dead */
16191614 goto out_unlock;
16201615 }
16211616
@@ -2405,11 +2400,11 @@ bool blk_update_request(struct request *req, int error, unsigned int nr_bytes) · 783 unmodified lines
24052400 {
24062401 int total_bytes;
24072402
2403+ trace_block_rq_complete(req->q, req, nr_bytes);
2404+
24082405 if (!req->bio)
24092406 return false;
24102407
2411− trace_block_rq_complete(req->q, req, nr_bytes);
2412−
24132408 /*
24142409 * For fs requests, rq is just carrier of independent bio's
24152410 * and each partial completion should be handled separately.
@@ -2449,8 +2444,8 @@ bool blk_update_request(struct request *req, int error, unsigned int nr_bytes) · 33 unmodified lines
24492444 error_type = "I/O";
24502445 break;
24512446 }
2452− printk_ratelimited(KERN_ERR "end_request: %s error, dev %s, sector %llu\n",
2453− error_type, req->rq_disk ?
2447+ printk_ratelimited(KERN_ERR "%s: %s error, dev %s, sector %llu\n",
2448+ __func__, error_type, req->rq_disk ?
24542449 req->rq_disk->disk_name : "?",
24552450 (unsigned long long)blk_rq_pos(req));
24562451
@@ -2931,7 +2926,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src, · 474 unmodified lines
29312926 blk_rq_init(NULL, rq);
29322927
29332928 __rq_for_each_bio(bio_src, rq_src) {
2934− bio = bio_clone_bioset(bio_src, gfp_mask, bs);
2929+ bio = bio_clone_fast(bio_src, gfp_mask, bs);
29352930 if (!bio)
29362931 goto free_and_out;
29372932
block/blk-flush.c+98−49
@@ -28,7 +28,7 @@ · 27 unmodified lines
2828 *
2929 * The actual execution of flush is double buffered. Whenever a request
3030 * needs to execute PRE or POSTFLUSH, it queues at
31− * q->flush_queue[q->flush_pending_idx]. Once certain criteria are met, a
31+ * fq->flush_queue[fq->flush_pending_idx]. Once certain criteria are met, a
3232 * flush is issued and the pending_idx is toggled. When the flush
3333 * completes, all the requests which were pending are proceeded to the next
3434 * step. This allows arbitrary merging of different types of FLUSH/FUA
@@ -91,7 +91,8 @@ enum { · 56 unmodified lines
9191 FLUSH_PENDING_TIMEOUT = 5 * HZ,
9292 };
9393
94−static bool blk_kick_flush(struct request_queue *q);
94+static bool blk_kick_flush(struct request_queue *q,
95+ struct blk_flush_queue *fq);
9596
9697 static unsigned int blk_flush_policy(unsigned int fflags, struct request *rq)
9798 {
@@ -126,8 +127,6 @@ static void blk_flush_restore_request(struct request *rq) · 28 unmodified lines
126127 /* make @rq a normal request */
127128 rq->cmd_flags &= ~REQ_FLUSH_SEQ;
128129 rq->end_io = rq->flush.saved_end_io;
129−
130− blk_clear_rq_complete(rq);
131130 }
132131
133132 static bool blk_flush_queue_rq(struct request *rq, bool add_front)
@@ -150,6 +149,7 @@ static bool blk_flush_queue_rq(struct request *rq, bool add_front) · 16 unmodified lines
150149 /**
151150 * blk_flush_complete_seq - complete flush sequence
152151 * @rq: FLUSH/FUA request being sequenced
152+ * @fq: flush queue
153153 * @seq: sequences to complete (mask of %REQ_FSEQ_*, can be zero)
154154 * @error: whether an error occurred
155155 *
@@ -157,16 +157,17 @@ static bool blk_flush_queue_rq(struct request *rq, bool add_front) · 1 unmodified line
157157 * completion and trigger the next step.
158158 *
159159 * CONTEXT:
160− * spin_lock_irq(q->queue_lock or q->mq_flush_lock)
160+ * spin_lock_irq(q->queue_lock or fq->mq_flush_lock)
161161 *
162162 * RETURNS:
163163 * %true if requests were added to the dispatch queue, %false otherwise.
164164 */
165−static bool blk_flush_complete_seq(struct request *rq, unsigned int seq,
166− int error)
165+static bool blk_flush_complete_seq(struct request *rq,
166+ struct blk_flush_queue *fq,
167+ unsigned int seq, int error)
167168 {
168169 struct request_queue *q = rq->q;
169− struct list_head *pending = &q->flush_queue[q->flush_pending_idx];
170+ struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
170171 bool queued = false, kicked;
171172
172173 BUG_ON(rq->flush.seq & seq);
@@ -182,12 +183,12 @@ static bool blk_flush_complete_seq(struct request *rq, unsigned int seq, · 9 unmodified lines
182183 case REQ_FSEQ_POSTFLUSH:
183184 /* queue for flush */
184185 if (list_empty(pending))
185− q->flush_pending_since = jiffies;
186+ fq->flush_pending_since = jiffies;
186187 list_move_tail(&rq->flush.list, pending);
187188 break;
188189
189190 case REQ_FSEQ_DATA:
190− list_move_tail(&rq->flush.list, &q->flush_data_in_flight);
191+ list_move_tail(&rq->flush.list, &fq->flush_data_in_flight);
191192 queued = blk_flush_queue_rq(rq, true);
192193 break;
193194
@@ -202,7 +203,7 @@ static bool blk_flush_complete_seq(struct request *rq, unsigned int seq, · 8 unmodified lines
202203 list_del_init(&rq->flush.list);
203204 blk_flush_restore_request(rq);
204205 if (q->mq_ops)
205− blk_mq_end_io(rq, error);
206+ blk_mq_end_request(rq, error);
206207 else
207208 __blk_end_request_all(rq, error);
208209 break;
@@ -211,7 +212,7 @@ static bool blk_flush_complete_seq(struct request *rq, unsigned int seq, · 2 unmodified lines
211212 BUG();
212213 }
213214
214− kicked = blk_kick_flush(q);
215+ kicked = blk_kick_flush(q, fq);
215216 return kicked | queued;
216217 }
217218
@@ -222,17 +223,18 @@ static void flush_end_io(struct request *flush_rq, int error) · 4 unmodified lines
222223 bool queued = false;
223224 struct request *rq, *n;
224225 unsigned long flags = 0;
226+ struct blk_flush_queue *fq = blk_get_flush_queue(q, flush_rq->mq_ctx);
225227
226228 if (q->mq_ops) {
227− spin_lock_irqsave(&q->mq_flush_lock, flags);
228− q->flush_rq->tag = -1;
229+ spin_lock_irqsave(&fq->mq_flush_lock, flags);
230+ flush_rq->tag = -1;
229231 }
230232
231− running = &q->flush_queue[q->flush_running_idx];
232− BUG_ON(q->flush_pending_idx == q->flush_running_idx);
233+ running = &fq->flush_queue[fq->flush_running_idx];
234+ BUG_ON(fq->flush_pending_idx == fq->flush_running_idx);
233235
234236 /* account completion of the flush request */
235− q->flush_running_idx ^= 1;
237+ fq->flush_running_idx ^= 1;
236238
237239 if (!q->mq_ops)
238240 elv_completed_request(q, flush_rq);
@@ -242,7 +244,7 @@ static void flush_end_io(struct request *flush_rq, int error) · 3 unmodified lines
242244 unsigned int seq = blk_flush_cur_seq(rq);
243245
244246 BUG_ON(seq != REQ_FSEQ_PREFLUSH && seq != REQ_FSEQ_POSTFLUSH);
245− queued |= blk_flush_complete_seq(rq, seq, error);
247+ queued |= blk_flush_complete_seq(rq, fq, seq, error);
246248 }
247249
248250 /*
@@ -256,71 +258,81 @@ static void flush_end_io(struct request *flush_rq, int error) · 7 unmodified lines
256258 * directly into request_fn may confuse the driver. Always use
257259 * kblockd.
258260 */
259− if (queued || q->flush_queue_delayed) {
261+ if (queued || fq->flush_queue_delayed) {
260262 WARN_ON(q->mq_ops);
261263 blk_run_queue_async(q);
262264 }
263− q->flush_queue_delayed = 0;
265+ fq->flush_queue_delayed = 0;
264266 if (q->mq_ops)
265− spin_unlock_irqrestore(&q->mq_flush_lock, flags);
267+ spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
266268 }
267269
268270 /**
269271 * blk_kick_flush - consider issuing flush request
270272 * @q: request_queue being kicked
273+ * @fq: flush queue
271274 *
272275 * Flush related states of @q have changed, consider issuing flush request.
273276 * Please read the comment at the top of this file for more info.
274277 *
275278 * CONTEXT:
276− * spin_lock_irq(q->queue_lock or q->mq_flush_lock)
279+ * spin_lock_irq(q->queue_lock or fq->mq_flush_lock)
277280 *
278281 * RETURNS:
279282 * %true if flush was issued, %false otherwise.
280283 */
281−static bool blk_kick_flush(struct request_queue *q)
284+static bool blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq)
282285 {
283− struct list_head *pending = &q->flush_queue[q->flush_pending_idx];
286+ struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
284287 struct request *first_rq =
285288 list_first_entry(pending, struct request, flush.list);
289+ struct request *flush_rq = fq->flush_rq;
286290
287291 /* C1 described at the top of this file */
288− if (q->flush_pending_idx != q->flush_running_idx || list_empty(pending))
292+ if (fq->flush_pending_idx != fq->flush_running_idx || list_empty(pending))
289293 return false;
290294
291295 /* C2 and C3 */
292− if (!list_empty(&q->flush_data_in_flight) &&
296+ if (!list_empty(&fq->flush_data_in_flight) &&
293297 time_before(jiffies,
294− q->flush_pending_since + FLUSH_PENDING_TIMEOUT))
298+ fq->flush_pending_since + FLUSH_PENDING_TIMEOUT))
295299 return false;
296300
297301 /*
298302 * Issue flush and toggle pending_idx. This makes pending_idx
299303 * different from running_idx, which means flush is in flight.
300304 */
301− q->flush_pending_idx ^= 1;
305+ fq->flush_pending_idx ^= 1;
302306
303− blk_rq_init(q, q->flush_rq);
304− if (q->mq_ops)
305− blk_mq_clone_flush_request(q->flush_rq, first_rq);
307+ blk_rq_init(q, flush_rq);
308+
309+ /*
310+ * Borrow tag from the first request since they can't
311+ * be in flight at the same time.
312+ */
313+ if (q->mq_ops) {
314+ flush_rq->mq_ctx = first_rq->mq_ctx;
315+ flush_rq->tag = first_rq->tag;
316+ }
306317
307− q->flush_rq->cmd_type = REQ_TYPE_FS;
308− q->flush_rq->cmd_flags = WRITE_FLUSH | REQ_FLUSH_SEQ;
309− q->flush_rq->rq_disk = first_rq->rq_disk;
310− q->flush_rq->end_io = flush_end_io;
318+ flush_rq->cmd_type = REQ_TYPE_FS;
319+ flush_rq->cmd_flags = WRITE_FLUSH | REQ_FLUSH_SEQ;
320+ flush_rq->rq_disk = first_rq->rq_disk;
321+ flush_rq->end_io = flush_end_io;
311322
312− return blk_flush_queue_rq(q->flush_rq, false);
323+ return blk_flush_queue_rq(flush_rq, false);
313324 }
314325
315326 static void flush_data_end_io(struct request *rq, int error)
316327 {
317328 struct request_queue *q = rq->q;
329+ struct blk_flush_queue *fq = blk_get_flush_queue(q, NULL);
318330
319331 /*
320332 * After populating an empty queue, kick it to avoid stall. Read
321333 * the comment in flush_end_io().
322334 */
323− if (blk_flush_complete_seq(rq, REQ_FSEQ_DATA, error))
335+ if (blk_flush_complete_seq(rq, fq, REQ_FSEQ_DATA, error))
324336 blk_run_queue_async(q);
325337 }
326338
@@ -328,20 +340,20 @@ static void mq_flush_data_end_io(struct request *rq, int error) · 1 unmodified line
328340 {
329341 struct request_queue *q = rq->q;
330342 struct blk_mq_hw_ctx *hctx;
331− struct blk_mq_ctx *ctx;
343+ struct blk_mq_ctx *ctx = rq->mq_ctx;
332344 unsigned long flags;
345+ struct blk_flush_queue *fq = blk_get_flush_queue(q, ctx);
333346
334− ctx = rq->mq_ctx;
335347 hctx = q->mq_ops->map_queue(q, ctx->cpu);
336348
337349 /*
338350 * After populating an empty queue, kick it to avoid stall. Read
339351 * the comment in flush_end_io().
340352 */
341− spin_lock_irqsave(&q->mq_flush_lock, flags);
342− if (blk_flush_complete_seq(rq, REQ_FSEQ_DATA, error))
353+ spin_lock_irqsave(&fq->mq_flush_lock, flags);
354+ if (blk_flush_complete_seq(rq, fq, REQ_FSEQ_DATA, error))
343355 blk_mq_run_hw_queue(hctx, true);
344− spin_unlock_irqrestore(&q->mq_flush_lock, flags);
356+ spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
345357 }
346358
347359 /**
@@ -361,6 +373,7 @@ void blk_insert_flush(struct request *rq) · 13 unmodified lines
361373 struct request_queue *q = rq->q;
362374 unsigned int fflags = q->flush_flags; /* may change, cache */
363375 unsigned int policy = blk_flush_policy(fflags, rq);
376+ struct blk_flush_queue *fq = blk_get_flush_queue(q, rq->mq_ctx);
364377
365378 /*
366379 * @policy now records what operations need to be done. Adjust
@@ -378,7 +391,7 @@ void blk_insert_flush(struct request *rq) · 11 unmodified lines
378391 */
379392 if (!policy) {
380393 if (q->mq_ops)
381− blk_mq_end_io(rq, 0);
394+ blk_mq_end_request(rq, 0);
382395 else
383396 __blk_end_bidi_request(rq, 0, 0, 0);
384397 return;
@@ -411,14 +424,14 @@ void blk_insert_flush(struct request *rq) · 26 unmodified lines
411424 if (q->mq_ops) {
412425 rq->end_io = mq_flush_data_end_io;
413426
414− spin_lock_irq(&q->mq_flush_lock);
415− blk_flush_complete_seq(rq, REQ_FSEQ_ACTIONS & ~policy, 0);
416− spin_unlock_irq(&q->mq_flush_lock);
427+ spin_lock_irq(&fq->mq_flush_lock);
428+ blk_flush_complete_seq(rq, fq, REQ_FSEQ_ACTIONS & ~policy, 0);
429+ spin_unlock_irq(&fq->mq_flush_lock);
417430 return;
418431 }
419432 rq->end_io = flush_data_end_io;
420433
421− blk_flush_complete_seq(rq, REQ_FSEQ_ACTIONS & ~policy, 0);
434+ blk_flush_complete_seq(rq, fq, REQ_FSEQ_ACTIONS & ~policy, 0);
422435 }
423436
424437 /**
@@ -474,7 +487,43 @@ int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask, · 49 unmodified lines
474487 }
475488 EXPORT_SYMBOL(blkdev_issue_flush);
476489
477−void blk_mq_init_flush(struct request_queue *q)
490+struct blk_flush_queue *blk_alloc_flush_queue(struct request_queue *q,
491+ int node, int cmd_size)
492+{
493+ struct blk_flush_queue *fq;
494+ int rq_sz = sizeof(struct request);
495+
496+ fq = kzalloc_node(sizeof(*fq), GFP_KERNEL, node);
497+ if (!fq)
498+ goto fail;
499+
500+ if (q->mq_ops) {
501+ spin_lock_init(&fq->mq_flush_lock);
502+ rq_sz = round_up(rq_sz + cmd_size, cache_line_size());
503+ }
504+
505+ fq->flush_rq = kzalloc_node(rq_sz, GFP_KERNEL, node);
506+ if (!fq->flush_rq)
507+ goto fail_rq;
508+
509+ INIT_LIST_HEAD(&fq->flush_queue[0]);
510+ INIT_LIST_HEAD(&fq->flush_queue[1]);
511+ INIT_LIST_HEAD(&fq->flush_data_in_flight);
512+
513+ return fq;
514+
515+ fail_rq:
516+ kfree(fq);
517+ fail:
518+ return NULL;
519+}
520+
521+void blk_free_flush_queue(struct blk_flush_queue *fq)
478522 {
479− spin_lock_init(&q->mq_flush_lock);
523+ /* bio based request queue hasn't flush queue */
524+ if (!fq)
525+ return;
526+
527+ kfree(fq->flush_rq);
528+ kfree(fq);
480529 }
block/blk-integrity.c+65−38
@@ -154,10 +154,10 @@ int blk_integrity_compare(struct gendisk *gd1, struct gendisk *gd2) · 153 unmodified lines
154154 if (!b1 || !b2)
155155 return -1;
156156
157− if (b1->sector_size != b2->sector_size) {
158− printk(KERN_ERR "%s: %s/%s sector sz %u != %u\n", __func__,
159− gd1->disk_name, gd2->disk_name,
160− b1->sector_size, b2->sector_size);
157+ if (b1->interval != b2->interval) {
158+ pr_err("%s: %s/%s protection interval %u != %u\n",
159+ __func__, gd1->disk_name, gd2->disk_name,
160+ b1->interval, b2->interval);
161161 return -1;
162162 }
163163
@@ -186,37 +186,53 @@ int blk_integrity_compare(struct gendisk *gd1, struct gendisk *gd2) · 22 unmodified lines
186186 }
187187 EXPORT_SYMBOL(blk_integrity_compare);
188188
189−int blk_integrity_merge_rq(struct request_queue *q, struct request *req,
190− struct request *next)
189+bool blk_integrity_merge_rq(struct request_queue *q, struct request *req,
190+ struct request *next)
191191 {
192− if (blk_integrity_rq(req) != blk_integrity_rq(next))
193− return -1;
192+ if (blk_integrity_rq(req) == 0 && blk_integrity_rq(next) == 0)
193+ return true;
194+
195+ if (blk_integrity_rq(req) == 0 || blk_integrity_rq(next) == 0)
196+ return false;
197+
198+ if (bio_integrity(req->bio)->bip_flags !=
199+ bio_integrity(next->bio)->bip_flags)
200+ return false;
194201
195202 if (req->nr_integrity_segments + next->nr_integrity_segments >
196203 q->limits.max_integrity_segments)
197− return -1;
204+ return false;
198205
199− return 0;
206+ return true;
200207 }
201208 EXPORT_SYMBOL(blk_integrity_merge_rq);
202209
203−int blk_integrity_merge_bio(struct request_queue *q, struct request *req,
204− struct bio *bio)
210+bool blk_integrity_merge_bio(struct request_queue *q, struct request *req,
211+ struct bio *bio)
205212 {
206213 int nr_integrity_segs;
207214 struct bio *next = bio->bi_next;
208215
216+ if (blk_integrity_rq(req) == 0 && bio_integrity(bio) == NULL)
217+ return true;
218+
219+ if (blk_integrity_rq(req) == 0 || bio_integrity(bio) == NULL)
220+ return false;
221+
222+ if (bio_integrity(req->bio)->bip_flags != bio_integrity(bio)->bip_flags)
223+ return false;
224+
209225 bio->bi_next = NULL;
210226 nr_integrity_segs = blk_rq_count_integrity_sg(q, bio);
211227 bio->bi_next = next;
212228
213229 if (req->nr_integrity_segments + nr_integrity_segs >
214230 q->limits.max_integrity_segments)
215− return -1;
231+ return false;
216232
217233 req->nr_integrity_segments += nr_integrity_segs;
218234
219− return 0;
235+ return true;
220236 }
221237 EXPORT_SYMBOL(blk_integrity_merge_bio);
222238
@@ -269,42 +285,48 @@ static ssize_t integrity_tag_size_show(struct blk_integrity *bi, char *page) · 46 unmodified lines
269285 return sprintf(page, "0\n");
270286 }
271287
272−static ssize_t integrity_read_store(struct blk_integrity *bi,
273− const char *page, size_t count)
288+static ssize_t integrity_verify_store(struct blk_integrity *bi,
289+ const char *page, size_t count)
274290 {
275291 char *p = (char *) page;
276292 unsigned long val = simple_strtoul(p, &p, 10);
277293
278294 if (val)
279− bi->flags |= INTEGRITY_FLAG_READ;
295+ bi->flags |= BLK_INTEGRITY_VERIFY;
280296 else
281− bi->flags &= ~INTEGRITY_FLAG_READ;
297+ bi->flags &= ~BLK_INTEGRITY_VERIFY;
282298
283299 return count;
284300 }
285301
286−static ssize_t integrity_read_show(struct blk_integrity *bi, char *page)
302+static ssize_t integrity_verify_show(struct blk_integrity *bi, char *page)
287303 {
288− return sprintf(page, "%d\n", (bi->flags & INTEGRITY_FLAG_READ) != 0);
304+ return sprintf(page, "%d\n", (bi->flags & BLK_INTEGRITY_VERIFY) != 0);
289305 }
290306
291−static ssize_t integrity_write_store(struct blk_integrity *bi,
292− const char *page, size_t count)
307+static ssize_t integrity_generate_store(struct blk_integrity *bi,
308+ const char *page, size_t count)
293309 {
294310 char *p = (char *) page;
295311 unsigned long val = simple_strtoul(p, &p, 10);
296312
297313 if (val)
298− bi->flags |= INTEGRITY_FLAG_WRITE;
314+ bi->flags |= BLK_INTEGRITY_GENERATE;
299315 else
300− bi->flags &= ~INTEGRITY_FLAG_WRITE;
316+ bi->flags &= ~BLK_INTEGRITY_GENERATE;
301317
302318 return count;
303319 }
304320
305−static ssize_t integrity_write_show(struct blk_integrity *bi, char *page)
321+static ssize_t integrity_generate_show(struct blk_integrity *bi, char *page)
322+{
323+ return sprintf(page, "%d\n", (bi->flags & BLK_INTEGRITY_GENERATE) != 0);
324+}
325+
326+static ssize_t integrity_device_show(struct blk_integrity *bi, char *page)
306327 {
307− return sprintf(page, "%d\n", (bi->flags & INTEGRITY_FLAG_WRITE) != 0);
328+ return sprintf(page, "%u\n",
329+ (bi->flags & BLK_INTEGRITY_DEVICE_CAPABLE) != 0);
308330 }
309331
310332 static struct integrity_sysfs_entry integrity_format_entry = {
@@ -317,23 +339,29 @@ static struct integrity_sysfs_entry integrity_tag_size_entry = { · 6 unmodified lines
317339 .show = integrity_tag_size_show,
318340 };
319341
320−static struct integrity_sysfs_entry integrity_read_entry = {
342+static struct integrity_sysfs_entry integrity_verify_entry = {
321343 .attr = { .name = "read_verify", .mode = S_IRUGO | S_IWUSR },
322− .show = integrity_read_show,
323− .store = integrity_read_store,
344+ .show = integrity_verify_show,
345+ .store = integrity_verify_store,
324346 };
325347
326−static struct integrity_sysfs_entry integrity_write_entry = {
348+static struct integrity_sysfs_entry integrity_generate_entry = {
327349 .attr = { .name = "write_generate", .mode = S_IRUGO | S_IWUSR },
328− .show = integrity_write_show,
329− .store = integrity_write_store,
350+ .show = integrity_generate_show,
351+ .store = integrity_generate_store,
352+};
353+
354+static struct integrity_sysfs_entry integrity_device_entry = {
355+ .attr = { .name = "device_is_integrity_capable", .mode = S_IRUGO },
356+ .show = integrity_device_show,
330357 };
331358
332359 static struct attribute *integrity_attrs[] = {
333360 &integrity_format_entry.attr,
334361 &integrity_tag_size_entry.attr,
335− &integrity_read_entry.attr,
336− &integrity_write_entry.attr,
362+ &integrity_verify_entry.attr,
363+ &integrity_generate_entry.attr,
364+ &integrity_device_entry.attr,
337365 NULL,
338366 };
339367
@@ -406,8 +434,8 @@ int blk_integrity_register(struct gendisk *disk, struct blk_integrity *template) · 66 unmodified lines
406434
407435 kobject_uevent(&bi->kobj, KOBJ_ADD);
408436
409− bi->flags |= INTEGRITY_FLAG_READ | INTEGRITY_FLAG_WRITE;
410− bi->sector_size = queue_logical_block_size(disk->queue);
437+ bi->flags |= BLK_INTEGRITY_VERIFY | BLK_INTEGRITY_GENERATE;
438+ bi->interval = queue_logical_block_size(disk->queue);
411439 disk->integrity = bi;
412440 } else
413441 bi = disk->integrity;
@@ -418,9 +446,8 @@ int blk_integrity_register(struct gendisk *disk, struct blk_integrity *template) · 4 unmodified lines
418446 bi->generate_fn = template->generate_fn;
419447 bi->verify_fn = template->verify_fn;
420448 bi->tuple_size = template->tuple_size;
421− bi->set_tag_fn = template->set_tag_fn;
422− bi->get_tag_fn = template->get_tag_fn;
423449 bi->tag_size = template->tag_size;
450+ bi->flags |= template->flags;
424451 } else
425452 bi->name = bi_unsupported_name;
426453
block/blk-merge.c+9−5
@@ -97,14 +97,18 @@ void blk_recalc_rq_segments(struct request *rq) · 96 unmodified lines
9797
9898 void blk_recount_segments(struct request_queue *q, struct bio *bio)
9999 {
100− if (test_bit(QUEUE_FLAG_NO_SG_MERGE, &q->queue_flags) &&
100+ bool no_sg_merge = !!test_bit(QUEUE_FLAG_NO_SG_MERGE,
101+ &q->queue_flags);
102+
103+ if (no_sg_merge && !bio_flagged(bio, BIO_CLONED) &&
101104 bio->bi_vcnt < queue_max_segments(q))
102105 bio->bi_phys_segments = bio->bi_vcnt;
103106 else {
104107 struct bio *nxt = bio->bi_next;
105108
106109 bio->bi_next = NULL;
107− bio->bi_phys_segments = __blk_recalc_rq_segments(q, bio, false);
110+ bio->bi_phys_segments = __blk_recalc_rq_segments(q, bio,
111+ no_sg_merge);
108112 bio->bi_next = nxt;
109113 }
110114
@@ -313,7 +317,7 @@ static inline int ll_new_hw_segment(struct request_queue *q, · 202 unmodified lines
313317 if (req->nr_phys_segments + nr_phys_segs > queue_max_segments(q))
314318 goto no_merge;
315319
316− if (bio_integrity(bio) && blk_integrity_merge_bio(q, req, bio))
320+ if (blk_integrity_merge_bio(q, req, bio) == false)
317321 goto no_merge;
318322
319323 /*
@@ -410,7 +414,7 @@ static int ll_merge_requests_fn(struct request_queue *q, struct request *req, · 90 unmodified lines
410414 if (total_phys_segments > queue_max_segments(q))
411415 return 0;
412416
413− if (blk_integrity_rq(req) && blk_integrity_merge_rq(q, req, next))
417+ if (blk_integrity_merge_rq(q, req, next) == false)
414418 return 0;
415419
416420 /* Merge is OK... */
@@ -590,7 +594,7 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) · 173 unmodified lines
590594 return false;
591595
592596 /* only merge integrity protected bio into ditto rq */
593− if (bio_integrity(bio) != blk_integrity_rq(rq))
597+ if (blk_integrity_merge_bio(rq->q, rq, bio) == false)
594598 return false;
595599
596600 /* must be using the same buffer */
block/blk-mq-tag.c+22−33
@@ -351,15 +351,12 @@ static void bt_clear_tag(struct blk_mq_bitmap_tags *bt, unsigned int tag) · 350 unmodified lines
351351 return;
352352
353353 wait_cnt = atomic_dec_return(&bs->wait_cnt);
354+ if (unlikely(wait_cnt < 0))
355+ wait_cnt = atomic_inc_return(&bs->wait_cnt);
354356 if (wait_cnt == 0) {
355−wake:
356357 atomic_add(bt->wake_cnt, &bs->wait_cnt);
357358 bt_index_atomic_inc(&bt->wake_index);
358359 wake_up(&bs->wait);
359− } else if (wait_cnt < 0) {
360− wait_cnt = atomic_inc_return(&bs->wait_cnt);
361− if (!wait_cnt)
362− goto wake;
363360 }
364361 }
365362
@@ -392,45 +389,37 @@ void blk_mq_put_tag(struct blk_mq_hw_ctx *hctx, unsigned int tag, · 26 unmodified lines
392389 __blk_mq_put_reserved_tag(tags, tag);
393390 }
394391
395−static void bt_for_each_free(struct blk_mq_bitmap_tags *bt,
396− unsigned long *free_map, unsigned int off)
392+static void bt_for_each(struct blk_mq_hw_ctx *hctx,
393+ struct blk_mq_bitmap_tags *bt, unsigned int off,
394+ busy_iter_fn *fn, void *data, bool reserved)
397395 {
398− int i;
396+ struct request *rq;
397+ int bit, i;
399398
400399 for (i = 0; i < bt->map_nr; i++) {
401400 struct blk_align_bitmap *bm = &bt->map[i];
402− int bit = 0;
403401
404− do {
405− bit = find_next_zero_bit(&bm->word, bm->depth, bit);
406− if (bit >= bm->depth)
407− break;
408−
409− __set_bit(bit + off, free_map);
410− bit++;
411− } while (1);
402+ for (bit = find_first_bit(&bm->word, bm->depth);
403+ bit < bm->depth;
404+ bit = find_next_bit(&bm->word, bm->depth, bit + 1)) {
405+ rq = blk_mq_tag_to_rq(hctx->tags, off + bit);
406+ if (rq->q == hctx->queue)
407+ fn(hctx, rq, data, reserved);
408+ }
412409
413410 off += (1 << bt->bits_per_word);
414411 }
415412 }
416413
417−void blk_mq_tag_busy_iter(struct blk_mq_tags *tags,
418− void (*fn)(void *, unsigned long *), void *data)
414+void blk_mq_tag_busy_iter(struct blk_mq_hw_ctx *hctx, busy_iter_fn *fn,
415+ void *priv)
419416 {
420− unsigned long *tag_map;
421− size_t map_size;
422−
423− map_size = ALIGN(tags->nr_tags, BITS_PER_LONG) / BITS_PER_LONG;
424− tag_map = kzalloc(map_size * sizeof(unsigned long), GFP_ATOMIC);
425− if (!tag_map)
426− return;
417+ struct blk_mq_tags *tags = hctx->tags;
427418
428− bt_for_each_free(&tags->bitmap_tags, tag_map, tags->nr_reserved_tags);
429419 if (tags->nr_reserved_tags)
430− bt_for_each_free(&tags->breserved_tags, tag_map, 0);
431−
432− fn(data, tag_map);
433− kfree(tag_map);
420+ bt_for_each(hctx, &tags->breserved_tags, 0, fn, priv, true);
421+ bt_for_each(hctx, &tags->bitmap_tags, tags->nr_reserved_tags, fn, priv,
422+ false);
434423 }
435424 EXPORT_SYMBOL(blk_mq_tag_busy_iter);
436425
@@ -463,8 +452,8 @@ static void bt_update_count(struct blk_mq_bitmap_tags *bt, · 26 unmodified lines
463452 }
464453
465454 bt->wake_cnt = BT_WAIT_BATCH;
466− if (bt->wake_cnt > depth / 4)
467− bt->wake_cnt = max(1U, depth / 4);
455+ if (bt->wake_cnt > depth / BT_WAIT_QUEUES)
456+ bt->wake_cnt = max(1U, depth / BT_WAIT_QUEUES);
468457
469458 bt->depth = depth;
470459 }
block/blk-mq.c+187−177
@@ -20,6 +20,7 @@ · 19 unmodified lines
2020 #include <linux/cache.h>
2121 #include <linux/sched/sysctl.h>
2222 #include <linux/delay.h>
23+#include <linux/crash_dump.h>
2324
2425 #include <trace/events/block.h>
2526
@@ -223,9 +224,11 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, gfp_t gfp, · 197 unmodified lines
223224 struct blk_mq_hw_ctx *hctx;
224225 struct request *rq;
225226 struct blk_mq_alloc_data alloc_data;
227+ int ret;
226228
227− if (blk_mq_queue_enter(q))
228− return NULL;
229+ ret = blk_mq_queue_enter(q);
230+ if (ret)
231+ return ERR_PTR(ret);
229232
230233 ctx = blk_mq_get_ctx(q);
231234 hctx = q->mq_ops->map_queue(q, ctx->cpu);
@@ -245,6 +248,8 @@ struct request *blk_mq_alloc_request(struct request_queue *q, int rw, gfp_t gfp, · 13 unmodified lines
245248 ctx = alloc_data.ctx;
246249 }
247250 blk_mq_put_ctx(ctx);
251+ if (!rq)
252+ return ERR_PTR(-EWOULDBLOCK);
248253 return rq;
249254 }
250255 EXPORT_SYMBOL(blk_mq_alloc_request);
@@ -276,27 +281,7 @@ void blk_mq_free_request(struct request *rq) · 25 unmodified lines
276281 __blk_mq_free_request(hctx, ctx, rq);
277282 }
278283
279−/*
280− * Clone all relevant state from a request that has been put on hold in
281− * the flush state machine into the preallocated flush request that hangs
282− * off the request queue.
283− *
284− * For a driver the flush request should be invisible, that's why we are
285− * impersonating the original request here.
286− */
287−void blk_mq_clone_flush_request(struct request *flush_rq,
288− struct request *orig_rq)
289−{
290− struct blk_mq_hw_ctx *hctx =
291− orig_rq->q->mq_ops->map_queue(orig_rq->q, orig_rq->mq_ctx->cpu);
292−
293− flush_rq->mq_ctx = orig_rq->mq_ctx;
294− flush_rq->tag = orig_rq->tag;
295− memcpy(blk_mq_rq_to_pdu(flush_rq), blk_mq_rq_to_pdu(orig_rq),
296− hctx->cmd_size);
297−}
298−
299−inline void __blk_mq_end_io(struct request *rq, int error)
284+inline void __blk_mq_end_request(struct request *rq, int error)
300285 {
301286 blk_account_io_done(rq);
302287
@@ -308,15 +293,15 @@ inline void __blk_mq_end_io(struct request *rq, int error) · 5 unmodified lines
308293 blk_mq_free_request(rq);
309294 }
310295 }
311−EXPORT_SYMBOL(__blk_mq_end_io);
296+EXPORT_SYMBOL(__blk_mq_end_request);
312297
313−void blk_mq_end_io(struct request *rq, int error)
298+void blk_mq_end_request(struct request *rq, int error)
314299 {
315300 if (blk_update_request(rq, error, blk_rq_bytes(rq)))
316301 BUG();
317− __blk_mq_end_io(rq, error);
302+ __blk_mq_end_request(rq, error);
318303 }
319−EXPORT_SYMBOL(blk_mq_end_io);
304+EXPORT_SYMBOL(blk_mq_end_request);
320305
321306 static void __blk_mq_complete_request_remote(void *data)
322307 {
@@ -356,7 +341,7 @@ void __blk_mq_complete_request(struct request *rq) · 33 unmodified lines
356341 struct request_queue *q = rq->q;
357342
358343 if (!q->softirq_done_fn)
359− blk_mq_end_io(rq, rq->errors);
344+ blk_mq_end_request(rq, rq->errors);
360345 else
361346 blk_mq_ipi_complete_request(rq);
362347 }
@@ -380,7 +365,7 @@ void blk_mq_complete_request(struct request *rq) · 17 unmodified lines
380365 }
381366 EXPORT_SYMBOL(blk_mq_complete_request);
382367
383−static void blk_mq_start_request(struct request *rq, bool last)
368+void blk_mq_start_request(struct request *rq)
384369 {
385370 struct request_queue *q = rq->q;
386371
@@ -417,35 +402,24 @@ static void blk_mq_start_request(struct request *rq, bool last) · 30 unmodified lines
417402 */
418403 rq->nr_phys_segments++;
419404 }
420−
421− /*
422− * Flag the last request in the series so that drivers know when IO
423− * should be kicked off, if they don't do it on a per-request basis.
424− *
425− * Note: the flag isn't the only condition drivers should do kick off.
426− * If drive is busy, the last request might not have the bit set.
427− */
428− if (last)
429− rq->cmd_flags |= REQ_END;
430405 }
406+EXPORT_SYMBOL(blk_mq_start_request);
431407
432408 static void __blk_mq_requeue_request(struct request *rq)
433409 {
434410 struct request_queue *q = rq->q;
435411
436412 trace_block_rq_requeue(q, rq);
437− clear_bit(REQ_ATOM_STARTED, &rq->atomic_flags);
438−
439− rq->cmd_flags &= ~REQ_END;
440413
441− if (q->dma_drain_size && blk_rq_bytes(rq))
442− rq->nr_phys_segments--;
414+ if (test_and_clear_bit(REQ_ATOM_STARTED, &rq->atomic_flags)) {
415+ if (q->dma_drain_size && blk_rq_bytes(rq))
416+ rq->nr_phys_segments--;
417+ }
443418 }
444419
445420 void blk_mq_requeue_request(struct request *rq)
446421 {
447422 __blk_mq_requeue_request(rq);
448− blk_clear_rq_complete(rq);
449423
450424 BUG_ON(blk_queued_rq(rq));
451425 blk_mq_add_to_requeue_list(rq, true);
@@ -514,78 +488,35 @@ void blk_mq_kick_requeue_list(struct request_queue *q) · 62 unmodified lines
514488 }
515489 EXPORT_SYMBOL(blk_mq_kick_requeue_list);
516490
517−static inline bool is_flush_request(struct request *rq, unsigned int tag)
491+static inline bool is_flush_request(struct request *rq,
492+ struct blk_flush_queue *fq, unsigned int tag)
518493 {
519494 return ((rq->cmd_flags & REQ_FLUSH_SEQ) &&
520− rq->q->flush_rq->tag == tag);
495+ fq->flush_rq->tag == tag);
521496 }
522497
523498 struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag)
524499 {
525500 struct request *rq = tags->rqs[tag];
501+ /* mq_ctx of flush rq is always cloned from the corresponding req */
502+ struct blk_flush_queue *fq = blk_get_flush_queue(rq->q, rq->mq_ctx);
526503
527− if (!is_flush_request(rq, tag))
504+ if (!is_flush_request(rq, fq, tag))
528505 return rq;
529506
530− return rq->q->flush_rq;
507+ return fq->flush_rq;
531508 }
532509 EXPORT_SYMBOL(blk_mq_tag_to_rq);
533510
534511 struct blk_mq_timeout_data {
535− struct blk_mq_hw_ctx *hctx;
536− unsigned long *next;
537− unsigned int *next_set;
512+ unsigned long next;
513+ unsigned int next_set;
538514 };
539515
540−static void blk_mq_timeout_check(void *__data, unsigned long *free_tags)
516+void blk_mq_rq_timed_out(struct request *req, bool reserved)
541517 {
542− struct blk_mq_timeout_data *data = __data;
543− struct blk_mq_hw_ctx *hctx = data->hctx;
544− unsigned int tag;
545−
546− /* It may not be in flight yet (this is where
547− * the REQ_ATOMIC_STARTED flag comes in). The requests are
548− * statically allocated, so we know it's always safe to access the
549− * memory associated with a bit offset into ->rqs[].
550− */
551− tag = 0;
552− do {
553− struct request *rq;
554−
555− tag = find_next_zero_bit(free_tags, hctx->tags->nr_tags, tag);
556− if (tag >= hctx->tags->nr_tags)
557− break;
558−
559− rq = blk_mq_tag_to_rq(hctx->tags, tag++);
560− if (rq->q != hctx->queue)
561− continue;
562− if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags))
563− continue;
564−
565− blk_rq_check_expired(rq, data->next, data->next_set);
566− } while (1);
567−}
568−
569−static void blk_mq_hw_ctx_check_timeout(struct blk_mq_hw_ctx *hctx,
570− unsigned long *next,
571− unsigned int *next_set)
572−{
573− struct blk_mq_timeout_data data = {
574− .hctx = hctx,
575− .next = next,
576− .next_set = next_set,
577− };
578−
579− /*
580− * Ask the tagging code to iterate busy requests, so we can
581− * check them for timeout.
582− */
583− blk_mq_tag_busy_iter(hctx->tags, blk_mq_timeout_check, &data);
584−}
585−
586−static enum blk_eh_timer_return blk_mq_rq_timed_out(struct request *rq)
587−{
588− struct request_queue *q = rq->q;
518+ struct blk_mq_ops *ops = req->q->mq_ops;
519+ enum blk_eh_timer_return ret = BLK_EH_RESET_TIMER;
589520
590521 /*
591522 * We know that complete is set at this point. If STARTED isn't set
@@ -596,21 +527,54 @@ static enum blk_eh_timer_return blk_mq_rq_timed_out(struct request *rq) · 4 unmodified lines
596527 * we both flags will get cleared. So check here again, and ignore
597528 * a timeout event with a request that isn't active.
598529 */
599− if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags))
600− return BLK_EH_NOT_HANDLED;
530+ if (!test_bit(REQ_ATOM_STARTED, &req->atomic_flags))
531+ return;
532+
533+ if (ops->timeout)
534+ ret = ops->timeout(req, reserved);
535+
536+ switch (ret) {
537+ case BLK_EH_HANDLED:
538+ __blk_mq_complete_request(req);
539+ break;
540+ case BLK_EH_RESET_TIMER:
541+ blk_add_timer(req);
542+ blk_clear_rq_complete(req);
543+ break;
544+ case BLK_EH_NOT_HANDLED:
545+ break;
546+ default:
547+ printk(KERN_ERR "block: bad eh return: %d\n", ret);
548+ break;
549+ }
550+}
551+
552+static void blk_mq_check_expired(struct blk_mq_hw_ctx *hctx,
553+ struct request *rq, void *priv, bool reserved)
554+{
555+ struct blk_mq_timeout_data *data = priv;
601556
602− if (!q->mq_ops->timeout)
603− return BLK_EH_RESET_TIMER;
557+ if (!test_bit(REQ_ATOM_STARTED, &rq->atomic_flags))
558+ return;
604559
605− return q->mq_ops->timeout(rq);
560+ if (time_after_eq(jiffies, rq->deadline)) {
561+ if (!blk_mark_rq_complete(rq))
562+ blk_mq_rq_timed_out(rq, reserved);
563+ } else if (!data->next_set || time_after(data->next, rq->deadline)) {
564+ data->next = rq->deadline;
565+ data->next_set = 1;
566+ }
606567 }
607568
608−static void blk_mq_rq_timer(unsigned long data)
569+static void blk_mq_rq_timer(unsigned long priv)
609570 {
610− struct request_queue *q = (struct request_queue *) data;
571+ struct request_queue *q = (struct request_queue *)priv;
572+ struct blk_mq_timeout_data data = {
573+ .next = 0,
574+ .next_set = 0,
575+ };
611576 struct blk_mq_hw_ctx *hctx;
612− unsigned long next = 0;
613− int i, next_set = 0;
577+ int i;
614578
615579 queue_for_each_hw_ctx(q, hctx, i) {
616580 /*
@@ -620,12 +584,12 @@ static void blk_mq_rq_timer(unsigned long data) · 3 unmodified lines
620584 if (!hctx->nr_ctx || !hctx->tags)
621585 continue;
622586
623− blk_mq_hw_ctx_check_timeout(hctx, &next, &next_set);
587+ blk_mq_tag_busy_iter(hctx, blk_mq_check_expired, &data);
624588 }
625589
626− if (next_set) {
627− next = blk_rq_timeout(round_jiffies_up(next));
628− mod_timer(&q->timeout, next);
590+ if (data.next_set) {
591+ data.next = blk_rq_timeout(round_jiffies_up(data.next));
592+ mod_timer(&q->timeout, data.next);
629593 } else {
630594 queue_for_each_hw_ctx(q, hctx, i)
631595 blk_mq_tag_idle(hctx);
@@ -751,9 +715,7 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx) · 119 unmodified lines
751715 rq = list_first_entry(&rq_list, struct request, queuelist);
752716 list_del_init(&rq->queuelist);
753717
754− blk_mq_start_request(rq, list_empty(&rq_list));
755−
756− ret = q->mq_ops->queue_rq(hctx, rq);
718+ ret = q->mq_ops->queue_rq(hctx, rq, list_empty(&rq_list));
757719 switch (ret) {
758720 case BLK_MQ_RQ_QUEUE_OK:
759721 queued++;
@@ -766,7 +728,7 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx) · 6 unmodified lines
766728 pr_err("blk-mq: bad return on queue: %d\n", ret);
767729 case BLK_MQ_RQ_QUEUE_ERROR:
768730 rq->errors = -EIO;
769− blk_mq_end_io(rq, rq->errors);
731+ blk_mq_end_request(rq, rq->errors);
770732 break;
771733 }
772734
@@ -1194,14 +1156,13 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio) · 421 unmodified lines
11941156 int ret;
11951157
11961158 blk_mq_bio_to_request(rq, bio);
1197− blk_mq_start_request(rq, true);
11981159
11991160 /*
12001161 * For OK queue, we are done. For error, kill it. Any other
12011162 * error (busy), just add it to our list as we previously
12021163 * would have done
12031164 */
1204− ret = q->mq_ops->queue_rq(data.hctx, rq);
1165+ ret = q->mq_ops->queue_rq(data.hctx, rq, true);
12051166 if (ret == BLK_MQ_RQ_QUEUE_OK)
12061167 goto done;
12071168 else {
@@ -1209,7 +1170,7 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio) · 1 unmodified line
12091170
12101171 if (ret == BLK_MQ_RQ_QUEUE_ERROR) {
12111172 rq->errors = -EIO;
1212− blk_mq_end_io(rq, rq->errors);
1173+ blk_mq_end_request(rq, rq->errors);
12131174 goto done;
12141175 }
12151176 }
@@ -1531,6 +1492,28 @@ static int blk_mq_hctx_notify(void *data, unsigned long action, · 315 unmodified lines
15311492 return NOTIFY_OK;
15321493 }
15331494
1495+static void blk_mq_exit_hctx(struct request_queue *q,
1496+ struct blk_mq_tag_set *set,
1497+ struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx)
1498+{
1499+ unsigned flush_start_tag = set->queue_depth;
1500+
1501+ blk_mq_tag_idle(hctx);
1502+
1503+ if (set->ops->exit_request)
1504+ set->ops->exit_request(set->driver_data,
1505+ hctx->fq->flush_rq, hctx_idx,
1506+ flush_start_tag + hctx_idx);
1507+
1508+ if (set->ops->exit_hctx)
1509+ set->ops->exit_hctx(hctx, hctx_idx);
1510+
1511+ blk_mq_unregister_cpu_notifier(&hctx->cpu_notifier);
1512+ blk_free_flush_queue(hctx->fq);
1513+ kfree(hctx->ctxs);
1514+ blk_mq_free_bitmap(&hctx->ctx_map);
1515+}
1516+
15341517 static void blk_mq_exit_hw_queues(struct request_queue *q,
15351518 struct blk_mq_tag_set *set, int nr_queue)
15361519 {
@@ -1540,17 +1523,8 @@ static void blk_mq_exit_hw_queues(struct request_queue *q, · 3 unmodified lines
15401523 queue_for_each_hw_ctx(q, hctx, i) {
15411524 if (i == nr_queue)
15421525 break;
1543−
1544− blk_mq_tag_idle(hctx);
1545−
1546− if (set->ops->exit_hctx)
1547− set->ops->exit_hctx(hctx, i);
1548−
1549− blk_mq_unregister_cpu_notifier(&hctx->cpu_notifier);
1550− kfree(hctx->ctxs);
1551− blk_mq_free_bitmap(&hctx->ctx_map);
1526+ blk_mq_exit_hctx(q, set, hctx, i);
15521527 }
1553−
15541528 }
15551529
15561530 static void blk_mq_free_hw_queues(struct request_queue *q,
@@ -1565,53 +1539,88 @@ static void blk_mq_free_hw_queues(struct request_queue *q, · 8 unmodified lines
15651539 }
15661540 }
15671541
1568−static int blk_mq_init_hw_queues(struct request_queue *q,
1569− struct blk_mq_tag_set *set)
1542+static int blk_mq_init_hctx(struct request_queue *q,
1543+ struct blk_mq_tag_set *set,
1544+ struct blk_mq_hw_ctx *hctx, unsigned hctx_idx)
15701545 {
1571− struct blk_mq_hw_ctx *hctx;
1572− unsigned int i;
1546+ int node;
1547+ unsigned flush_start_tag = set->queue_depth;
1548+
1549+ node = hctx->numa_node;
1550+ if (node == NUMA_NO_NODE)
1551+ node = hctx->numa_node = set->numa_node;
1552+
1553+ INIT_DELAYED_WORK(&hctx->run_work, blk_mq_run_work_fn);
1554+ INIT_DELAYED_WORK(&hctx->delay_work, blk_mq_delay_work_fn);
1555+ spin_lock_init(&hctx->lock);
1556+ INIT_LIST_HEAD(&hctx->dispatch);
1557+ hctx->queue = q;
1558+ hctx->queue_num = hctx_idx;
1559+ hctx->flags = set->flags;
1560+ hctx->cmd_size = set->cmd_size;
1561+
1562+ blk_mq_init_cpu_notifier(&hctx->cpu_notifier,
1563+ blk_mq_hctx_notify, hctx);
1564+ blk_mq_register_cpu_notifier(&hctx->cpu_notifier);
1565+
1566+ hctx->tags = set->tags[hctx_idx];
15731567
15741568 /*
1575− * Initialize hardware queues
1569+ * Allocate space for all possible cpus to avoid allocation at
1570+ * runtime
15761571 */
1577− queue_for_each_hw_ctx(q, hctx, i) {
1578− int node;
1572+ hctx->ctxs = kmalloc_node(nr_cpu_ids * sizeof(void *),
1573+ GFP_KERNEL, node);
1574+ if (!hctx->ctxs)
1575+ goto unregister_cpu_notifier;
15791576
1580− node = hctx->numa_node;
1581− if (node == NUMA_NO_NODE)
1582− node = hctx->numa_node = set->numa_node;
1577+ if (blk_mq_alloc_bitmap(&hctx->ctx_map, node))
1578+ goto free_ctxs;
15831579
1584− INIT_DELAYED_WORK(&hctx->run_work, blk_mq_run_work_fn);
1585− INIT_DELAYED_WORK(&hctx->delay_work, blk_mq_delay_work_fn);
1586− spin_lock_init(&hctx->lock);
1587− INIT_LIST_HEAD(&hctx->dispatch);
1588− hctx->queue = q;
1589− hctx->queue_num = i;
1590− hctx->flags = set->flags;
1591− hctx->cmd_size = set->cmd_size;
1580+ hctx->nr_ctx = 0;
15921581
1593− blk_mq_init_cpu_notifier(&hctx->cpu_notifier,
1594− blk_mq_hctx_notify, hctx);
1595− blk_mq_register_cpu_notifier(&hctx->cpu_notifier);
1582+ if (set->ops->init_hctx &&
1583+ set->ops->init_hctx(hctx, set->driver_data, hctx_idx))
1584+ goto free_bitmap;
15961585
1597− hctx->tags = set->tags[i];
1586+ hctx->fq = blk_alloc_flush_queue(q, hctx->numa_node, set->cmd_size);
1587+ if (!hctx->fq)
1588+ goto exit_hctx;
15981589
1599− /*
1600− * Allocate space for all possible cpus to avoid allocation at
1601− * runtime
1602− */
1603− hctx->ctxs = kmalloc_node(nr_cpu_ids * sizeof(void *),
1604− GFP_KERNEL, node);
1605− if (!hctx->ctxs)
1606− break;
1590+ if (set->ops->init_request &&
1591+ set->ops->init_request(set->driver_data,
1592+ hctx->fq->flush_rq, hctx_idx,
1593+ flush_start_tag + hctx_idx, node))
1594+ goto free_fq;
16071595
1608− if (blk_mq_alloc_bitmap(&hctx->ctx_map, node))
1609− break;
1596+ return 0;
16101597
1611− hctx->nr_ctx = 0;
1598+ free_fq:
1599+ kfree(hctx->fq);
1600+ exit_hctx:
1601+ if (set->ops->exit_hctx)
1602+ set->ops->exit_hctx(hctx, hctx_idx);
1603+ free_bitmap:
1604+ blk_mq_free_bitmap(&hctx->ctx_map);
1605+ free_ctxs:
1606+ kfree(hctx->ctxs);
1607+ unregister_cpu_notifier:
1608+ blk_mq_unregister_cpu_notifier(&hctx->cpu_notifier);
1609+
1610+ return -1;
1611+}
1612+
1613+static int blk_mq_init_hw_queues(struct request_queue *q,
1614+ struct blk_mq_tag_set *set)
1615+{
1616+ struct blk_mq_hw_ctx *hctx;
1617+ unsigned int i;
16121618
1613− if (set->ops->init_hctx &&
1614− set->ops->init_hctx(hctx, set->driver_data, i))
1619+ /*
1620+ * Initialize hardware queues
1621+ */
1622+ queue_for_each_hw_ctx(q, hctx, i) {
1623+ if (blk_mq_init_hctx(q, set, hctx, i))
16151624 break;
16161625 }
16171626
@@ -1765,6 +1774,16 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 147 unmodified lines
17651774 if (!ctx)
17661775 return ERR_PTR(-ENOMEM);
17671776
1777+ /*
1778+ * If a crashdump is active, then we are potentially in a very
1779+ * memory constrained environment. Limit us to 1 queue and
1780+ * 64 tags to prevent using too much memory.
1781+ */
1782+ if (is_kdump_kernel()) {
1783+ set->nr_hw_queues = 1;
1784+ set->queue_depth = min(64U, set->queue_depth);
1785+ }
1786+
17681787 hctxs = kmalloc_node(set->nr_hw_queues * sizeof(*hctxs), GFP_KERNEL,
17691788 set->numa_node);
17701789
@@ -1783,7 +1802,8 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 12 unmodified lines
17831802 if (!hctxs[i])
17841803 goto err_hctxs;
17851804
1786− if (!zalloc_cpumask_var(&hctxs[i]->cpumask, GFP_KERNEL))
1805+ if (!zalloc_cpumask_var_node(&hctxs[i]->cpumask, GFP_KERNEL,
1806+ node))
17871807 goto err_hctxs;
17881808
17891809 atomic_set(&hctxs[i]->nr_active, 0);
@@ -1830,7 +1850,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 40 unmodified lines
18301850 else
18311851 blk_queue_make_request(q, blk_sq_make_request);
18321852
1833− blk_queue_rq_timed_out(q, blk_mq_rq_timed_out);
18341853 if (set->timeout)
18351854 blk_queue_rq_timeout(q, set->timeout);
18361855
@@ -1842,17 +1861,10 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 5 unmodified lines
18421861 if (set->ops->complete)
18431862 blk_queue_softirq_done(q, set->ops->complete);
18441863
1845− blk_mq_init_flush(q);
18461864 blk_mq_init_cpu_queues(q, set->nr_hw_queues);
18471865
1848− q->flush_rq = kzalloc(round_up(sizeof(struct request) +
1849− set->cmd_size, cache_line_size()),
1850− GFP_KERNEL);
1851− if (!q->flush_rq)
1852− goto err_hw;
1853−
18541866 if (blk_mq_init_hw_queues(q, set))
1855− goto err_flush_rq;
1867+ goto err_hw;
18561868
18571869 mutex_lock(&all_q_mutex);
18581870 list_add_tail(&q->all_q_node, &all_q_list);
@@ -1864,8 +1876,6 @@ struct request_queue *blk_mq_init_queue(struct blk_mq_tag_set *set) · 5 unmodified lines
18641876
18651877 return q;
18661878
1867−err_flush_rq:
1868− kfree(q->flush_rq);
18691879 err_hw:
18701880 blk_cleanup_queue(q);
18711881 err_hctxs:
block/blk-mq.h+2−1
@@ -27,7 +27,6 @@ struct blk_mq_ctx { · 26 unmodified lines
2727
2828 void __blk_mq_complete_request(struct request *rq);
2929 void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
30−void blk_mq_init_flush(struct request_queue *q);
3130 void blk_mq_freeze_queue(struct request_queue *q);
3231 void blk_mq_free_queue(struct request_queue *q);
3332 void blk_mq_clone_flush_request(struct request *flush_rq,
@@ -60,6 +59,8 @@ extern int blk_mq_hw_queue_to_node(unsigned int *map, unsigned int); · 26 unmodified lines
6059 extern int blk_mq_sysfs_register(struct request_queue *q);
6160 extern void blk_mq_sysfs_unregister(struct request_queue *q);
6261
62+extern void blk_mq_rq_timed_out(struct request *req, bool reserved);
63+
6364 /*
6465 * Basic implementation of sparser bitmap, allowing the user to spread
6566 * the bits over more cachelines.
block/blk-settings.c+2−2
@@ -574,7 +574,7 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, · 573 unmodified lines
574574 bottom = max(b->physical_block_size, b->io_min) + alignment;
575575
576576 /* Verify that top and bottom intervals line up */
577− if (max(top, bottom) & (min(top, bottom) - 1)) {
577+ if (max(top, bottom) % min(top, bottom)) {
578578 t->misaligned = 1;
579579 ret = -1;
580580 }
@@ -619,7 +619,7 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, · 38 unmodified lines
619619
620620 /* Find lowest common alignment_offset */
621621 t->alignment_offset = lcm(t->alignment_offset, alignment)
622− & (max(t->physical_block_size, t->io_min) - 1);
622+ % max(t->physical_block_size, t->io_min);
623623
624624 /* Verify that new alignment_offset is on a logical block boundary */
625625 if (t->alignment_offset & (t->logical_block_size - 1)) {
block/blk-sysfs.c+2−2
@@ -519,8 +519,8 @@ static void blk_release_queue(struct kobject *kobj) · 518 unmodified lines
519519
520520 if (q->mq_ops)
521521 blk_mq_free_queue(q);
522−
523− kfree(q->flush_rq);
522+ else
523+ blk_free_flush_queue(q->fq);
524524
525525 blk_trace_shutdown(q);
526526
block/blk-timeout.c+8−7
@@ -90,10 +90,7 @@ static void blk_rq_timed_out(struct request *req) · 89 unmodified lines
9090 switch (ret) {
9191 case BLK_EH_HANDLED:
9292 /* Can we use req->errors here? */
93− if (q->mq_ops)
94− __blk_mq_complete_request(req);
95− else
96− __blk_complete_request(req);
93+ __blk_complete_request(req);
9794 break;
9895 case BLK_EH_RESET_TIMER:
9996 blk_add_timer(req);
@@ -113,7 +110,7 @@ static void blk_rq_timed_out(struct request *req) · 13 unmodified lines
113110 }
114111 }
115112
116−void blk_rq_check_expired(struct request *rq, unsigned long *next_timeout,
113+static void blk_rq_check_expired(struct request *rq, unsigned long *next_timeout,
117114 unsigned int *next_set)
118115 {
119116 if (time_after_eq(jiffies, rq->deadline)) {
@@ -162,7 +159,10 @@ void blk_abort_request(struct request *req) · 42 unmodified lines
162159 if (blk_mark_rq_complete(req))
163160 return;
164161 blk_delete_timer(req);
165− blk_rq_timed_out(req);
162+ if (req->q->mq_ops)
163+ blk_mq_rq_timed_out(req, false);
164+ else
165+ blk_rq_timed_out(req);
166166 }
167167 EXPORT_SYMBOL_GPL(blk_abort_request);
168168
@@ -190,7 +190,8 @@ void blk_add_timer(struct request *req) · 21 unmodified lines
190190 struct request_queue *q = req->q;
191191 unsigned long expiry;
192192
193− if (!q->rq_timed_out_fn)
193+ /* blk-mq has its own handler, so we don't need ->rq_timed_out_fn */
194+ if (!q->mq_ops && !q->rq_timed_out_fn)
194195 return;
195196
196197 BUG_ON(!list_empty(&req->timeout_list));
block/blk.h+33−4
@@ -2,6 +2,8 @@ · 1 unmodified line
22 #define BLK_INTERNAL_H
33
44 #include <linux/idr.h>
5+#include <linux/blk-mq.h>
6+#include "blk-mq.h"
57
68 /* Amount of time in which a process may batch requests */
79 #define BLK_BATCH_TIME (HZ/50UL)
@@ -12,16 +14,44 @@ · 4 unmodified lines
1214 /* Max future timer expiry for timeouts */
1315 #define BLK_MAX_TIMEOUT (5 * HZ)
1416
17+struct blk_flush_queue {
18+ unsigned int flush_queue_delayed:1;
19+ unsigned int flush_pending_idx:1;
20+ unsigned int flush_running_idx:1;
21+ unsigned long flush_pending_since;
22+ struct list_head flush_queue[2];
23+ struct list_head flush_data_in_flight;
24+ struct request *flush_rq;
25+ spinlock_t mq_flush_lock;
26+};
27+
1528 extern struct kmem_cache *blk_requestq_cachep;
1629 extern struct kmem_cache *request_cachep;
1730 extern struct kobj_type blk_queue_ktype;
1831 extern struct ida blk_queue_ida;
1932
33+static inline struct blk_flush_queue *blk_get_flush_queue(
34+ struct request_queue *q, struct blk_mq_ctx *ctx)
35+{
36+ struct blk_mq_hw_ctx *hctx;
37+
38+ if (!q->mq_ops)
39+ return q->fq;
40+
41+ hctx = q->mq_ops->map_queue(q, ctx->cpu);
42+
43+ return hctx->fq;
44+}
45+
2046 static inline void __blk_get_queue(struct request_queue *q)
2147 {
2248 kobject_get(&q->kobj);
2349 }
2450
51+struct blk_flush_queue *blk_alloc_flush_queue(struct request_queue *q,
52+ int node, int cmd_size);
53+void blk_free_flush_queue(struct blk_flush_queue *q);
54+
2555 int blk_init_rl(struct request_list *rl, struct request_queue *q,
2656 gfp_t gfp_mask);
2757 void blk_exit_rl(struct request_list *rl);
@@ -38,8 +68,6 @@ bool __blk_end_bidi_request(struct request *rq, int error, · 10 unmodified lines
3868 unsigned int nr_bytes, unsigned int bidi_bytes);
3969
4070 void blk_rq_timed_out_timer(unsigned long data);
41−void blk_rq_check_expired(struct request *rq, unsigned long *next_timeout,
42− unsigned int *next_set);
4371 unsigned long blk_rq_timeout(unsigned long timeout);
4472 void blk_add_timer(struct request *req);
4573 void blk_delete_timer(struct request *);
@@ -88,6 +116,7 @@ void blk_insert_flush(struct request *rq); · 42 unmodified lines
88116 static inline struct request *__elv_next_request(struct request_queue *q)
89117 {
90118 struct request *rq;
119+ struct blk_flush_queue *fq = blk_get_flush_queue(q, NULL);
91120
92121 while (1) {
93122 if (!list_empty(&q->queue_head)) {
@@ -110,9 +139,9 @@ static inline struct request *__elv_next_request(struct request_queue *q) · 16 unmodified lines
110139 * should be restarted later. Please see flush_end_io() for
111140 * details.
112141 */
113− if (q->flush_pending_idx != q->flush_running_idx &&
142+ if (fq->flush_pending_idx != fq->flush_running_idx &&
114143 !queue_flush_queueable(q)) {
115− q->flush_queue_delayed = 1;
144+ fq->flush_queue_delayed = 1;
116145 return NULL;
117146 }
118147 if (unlikely(blk_queue_bypass(q)) ||
block/bsg.c+5−4
@@ -270,8 +270,8 @@ bsg_map_hdr(struct bsg_device *bd, struct sg_io_v4 *hdr, fmode_t has_write_perm, · 269 unmodified lines
270270 * map scatter-gather elements separately and string them to request
271271 */
272272 rq = blk_get_request(q, rw, GFP_KERNEL);
273− if (!rq)
274− return ERR_PTR(-ENOMEM);
273+ if (IS_ERR(rq))
274+ return rq;
275275 blk_rq_set_block_pc(rq);
276276
277277 ret = blk_fill_sgv4_hdr_rq(q, rq, hdr, bd, has_write_perm);
@@ -285,8 +285,9 @@ bsg_map_hdr(struct bsg_device *bd, struct sg_io_v4 *hdr, fmode_t has_write_perm, · 7 unmodified lines
285285 }
286286
287287 next_rq = blk_get_request(q, READ, GFP_KERNEL);
288− if (!next_rq) {
289− ret = -ENOMEM;
288+ if (IS_ERR(next_rq)) {
289+ ret = PTR_ERR(next_rq);
290+ next_rq = NULL;
290291 goto out;
291292 }
292293 rq->next_rq = next_rq;
block/cfq-iosched.c+5−5
@@ -299,7 +299,7 @@ struct cfq_io_cq { · 298 unmodified lines
299299 struct cfq_ttime ttime;
300300 int ioprio; /* the current ioprio */
301301 #ifdef CONFIG_CFQ_GROUP_IOSCHED
302− uint64_t blkcg_id; /* the current blkcg ID */
302+ uint64_t blkcg_serial_nr; /* the current blkcg serial */
303303 #endif
304304 };
305305
@@ -3547,17 +3547,17 @@ static void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio) · 3241 unmodified lines
35473547 {
35483548 struct cfq_data *cfqd = cic_to_cfqd(cic);
35493549 struct cfq_queue *sync_cfqq;
3550− uint64_t id;
3550+ uint64_t serial_nr;
35513551
35523552 rcu_read_lock();
3553− id = bio_blkcg(bio)->id;
3553+ serial_nr = bio_blkcg(bio)->css.serial_nr;
35543554 rcu_read_unlock();
35553555
35563556 /*
35573557 * Check whether blkcg has changed. The condition may trigger
35583558 * spuriously on a newly created cic but there's no harm.
35593559 */
3560− if (unlikely(!cfqd) || likely(cic->blkcg_id == id))
3560+ if (unlikely(!cfqd) || likely(cic->blkcg_serial_nr == serial_nr))
35613561 return;
35623562
35633563 sync_cfqq = cic_to_cfqq(cic, 1);
@@ -3571,7 +3571,7 @@ static void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio) · 7 unmodified lines
35713571 cfq_put_queue(sync_cfqq);
35723572 }
35733573
3574− cic->blkcg_id = id;
3574+ cic->blkcg_serial_nr = serial_nr;
35753575 }
35763576 #else
35773577 static inline void check_blkcg_changed(struct cfq_io_cq *cic, struct bio *bio) { }
block/compat_ioctl.c+0−4
@@ -709,8 +709,6 @@ long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg) · 708 unmodified lines
709709 if (!arg)
710710 return -EINVAL;
711711 bdi = blk_get_backing_dev_info(bdev);
712− if (bdi == NULL)
713− return -ENOTTY;
714712 return compat_put_long(arg,
715713 (bdi->ra_pages * PAGE_CACHE_SIZE) / 512);
716714 case BLKROGET: /* compatible */
@@ -731,8 +729,6 @@ long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg) · 14 unmodified lines
731729 if (!capable(CAP_SYS_ADMIN))
732730 return -EACCES;
733731 bdi = blk_get_backing_dev_info(bdev);
734− if (bdi == NULL)
735− return -ENOTTY;
736732 bdi->ra_pages = (arg * 512) / PAGE_CACHE_SIZE;
737733 return 0;
738734 case BLKGETSIZE:
block/ioctl.c+0−4
@@ -356,8 +356,6 @@ int blkdev_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd, · 355 unmodified lines
356356 if (!arg)
357357 return -EINVAL;
358358 bdi = blk_get_backing_dev_info(bdev);
359− if (bdi == NULL)
360− return -ENOTTY;
361359 return put_long(arg, (bdi->ra_pages * PAGE_CACHE_SIZE) / 512);
362360 case BLKROGET:
363361 return put_int(arg, bdev_read_only(bdev) != 0);
@@ -386,8 +384,6 @@ int blkdev_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd, · 22 unmodified lines
386384 if(!capable(CAP_SYS_ADMIN))
387385 return -EACCES;
388386 bdi = blk_get_backing_dev_info(bdev);
389− if (bdi == NULL)
390− return -ENOTTY;
391387 bdi->ra_pages = (arg * 512) / PAGE_CACHE_SIZE;
392388 return 0;
393389 case BLKBSZSET:
block/partitions/mac.c+4−4
@@ -81,7 +81,7 @@ int mac_partition(struct parsed_partitions *state) · 80 unmodified lines
8181 be32_to_cpu(part->start_block) * (secsize/512),
8282 be32_to_cpu(part->block_count) * (secsize/512));
8383
84− if (!strnicmp(part->type, "Linux_RAID", 10))
84+ if (!strncasecmp(part->type, "Linux_RAID", 10))
8585 state->parts[slot].flags = ADDPART_FLAG_RAID;
8686 #ifdef CONFIG_PPC_PMAC
8787 /*
@@ -100,7 +100,7 @@ int mac_partition(struct parsed_partitions *state) · 12 unmodified lines
100100 goodness++;
101101
102102 if (strcasecmp(part->type, "Apple_UNIX_SVR2") == 0
103− || (strnicmp(part->type, "Linux", 5) == 0
103+ || (strncasecmp(part->type, "Linux", 5) == 0
104104 && strcasecmp(part->type, "Linux_swap") != 0)) {
105105 int i, l;
106106
@@ -109,13 +109,13 @@ int mac_partition(struct parsed_partitions *state) · 2 unmodified lines
109109 if (strcmp(part->name, "/") == 0)
110110 goodness++;
111111 for (i = 0; i <= l - 4; ++i) {
112− if (strnicmp(part->name + i, "root",
112+ if (strncasecmp(part->name + i, "root",
113113 4) == 0) {
114114 goodness += 2;
115115 break;
116116 }
117117 }
118− if (strnicmp(part->name, "swap", 4) == 0)
118+ if (strncasecmp(part->name, "swap", 4) == 0)
119119 goodness--;
120120 }
121121
block/scsi_ioctl.c+6−5
@@ -316,8 +316,8 @@ static int sg_io(struct request_queue *q, struct gendisk *bd_disk, · 315 unmodified lines
316316
317317 ret = -ENOMEM;
318318 rq = blk_get_request(q, writing ? WRITE : READ, GFP_KERNEL);
319− if (!rq)
320− goto out;
319+ if (IS_ERR(rq))
320+ return PTR_ERR(rq);
321321 blk_rq_set_block_pc(rq);
322322
323323 if (hdr->cmd_len > BLK_MAX_CDB) {
@@ -387,7 +387,6 @@ out_free_cdb: · 63 unmodified lines
387387 kfree(rq->cmd);
388388 out_put_request:
389389 blk_put_request(rq);
390−out:
391390 return ret;
392391 }
393392
@@ -457,8 +456,8 @@ int sg_scsi_ioctl(struct request_queue *q, struct gendisk *disk, fmode_t mode, · 63 unmodified lines
457456 }
458457
459458 rq = blk_get_request(q, in_len ? WRITE : READ, __GFP_WAIT);
460− if (!rq) {
461− err = -ENOMEM;
459+ if (IS_ERR(rq)) {
460+ err = PTR_ERR(rq);
462461 goto error;
463462 }
464463 blk_rq_set_block_pc(rq);
@@ -548,6 +547,8 @@ static int __blk_send_generic(struct request_queue *q, struct gendisk *bd_disk, · 83 unmodified lines
548547 int err;
549548
550549 rq = blk_get_request(q, WRITE, __GFP_WAIT);
550+ if (IS_ERR(rq))
551+ return PTR_ERR(rq);
551552 blk_rq_set_block_pc(rq);
552553 rq->timeout = BLK_DEFAULT_SG_TIMEOUT;
553554 rq->cmd[0] = cmd;
block/t10-pi.c+197−0
@@ -0,0 +1,197 @@
1+/*
2+ * t10_pi.c - Functions for generating and verifying T10 Protection
3+ * Information.
4+ *
5+ * Copyright (C) 2007, 2008, 2014 Oracle Corporation
6+ * Written by: Martin K. Petersen <martin.petersen@oracle.com>
7+ *
8+ * This program is free software; you can redistribute it and/or
9+ * modify it under the terms of the GNU General Public License version
10+ * 2 as published by the Free Software Foundation.
11+ *
12+ * This program is distributed in the hope that it will be useful, but
13+ * WITHOUT ANY WARRANTY; without even the implied warranty of
14+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
15+ * General Public License for more details.
16+ *
17+ * You should have received a copy of the GNU General Public License
18+ * along with this program; see the file COPYING. If not, write to
19+ * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139,
20+ * USA.
21+ *
22+ */
23+
24+#include <linux/t10-pi.h>
25+#include <linux/blkdev.h>
26+#include <linux/crc-t10dif.h>
27+#include <net/checksum.h>
28+
29+typedef __be16 (csum_fn) (void *, unsigned int);
30+
31+static const __be16 APP_ESCAPE = (__force __be16) 0xffff;
32+static const __be32 REF_ESCAPE = (__force __be32) 0xffffffff;
33+
34+static __be16 t10_pi_crc_fn(void *data, unsigned int len)
35+{
36+ return cpu_to_be16(crc_t10dif(data, len));
37+}
38+
39+static __be16 t10_pi_ip_fn(void *data, unsigned int len)
40+{
41+ return (__force __be16)ip_compute_csum(data, len);
42+}
43+
44+/*
45+ * Type 1 and Type 2 protection use the same format: 16 bit guard tag,
46+ * 16 bit app tag, 32 bit reference tag. Type 3 does not define the ref
47+ * tag.
48+ */
49+static int t10_pi_generate(struct blk_integrity_iter *iter, csum_fn *fn,
50+ unsigned int type)
51+{
52+ unsigned int i;
53+
54+ for (i = 0 ; i < iter->data_size ; i += iter->interval) {
55+ struct t10_pi_tuple *pi = iter->prot_buf;
56+
57+ pi->guard_tag = fn(iter->data_buf, iter->interval);
58+ pi->app_tag = 0;
59+
60+ if (type == 1)
61+ pi->ref_tag = cpu_to_be32(lower_32_bits(iter->seed));
62+ else
63+ pi->ref_tag = 0;
64+
65+ iter->data_buf += iter->interval;
66+ iter->prot_buf += sizeof(struct t10_pi_tuple);
67+ iter->seed++;
68+ }
69+
70+ return 0;
71+}
72+
73+static int t10_pi_verify(struct blk_integrity_iter *iter, csum_fn *fn,
74+ unsigned int type)
75+{
76+ unsigned int i;
77+
78+ for (i = 0 ; i < iter->data_size ; i += iter->interval) {
79+ struct t10_pi_tuple *pi = iter->prot_buf;
80+ __be16 csum;
81+
82+ switch (type) {
83+ case 1:
84+ case 2:
85+ if (pi->app_tag == APP_ESCAPE)
86+ goto next;
87+
88+ if (be32_to_cpu(pi->ref_tag) !=
89+ lower_32_bits(iter->seed)) {
90+ pr_err("%s: ref tag error at location %llu " \
91+ "(rcvd %u)\n", iter->disk_name,
92+ (unsigned long long)
93+ iter->seed, be32_to_cpu(pi->ref_tag));
94+ return -EILSEQ;
95+ }
96+ break;
97+ case 3:
98+ if (pi->app_tag == APP_ESCAPE &&
99+ pi->ref_tag == REF_ESCAPE)
100+ goto next;
101+ break;
102+ }
103+
104+ csum = fn(iter->data_buf, iter->interval);
105+
106+ if (pi->guard_tag != csum) {
107+ pr_err("%s: guard tag error at sector %llu " \
108+ "(rcvd %04x, want %04x)\n", iter->disk_name,
109+ (unsigned long long)iter->seed,
110+ be16_to_cpu(pi->guard_tag), be16_to_cpu(csum));
111+ return -EILSEQ;
112+ }
113+
114+next:
115+ iter->data_buf += iter->interval;
116+ iter->prot_buf += sizeof(struct t10_pi_tuple);
117+ iter->seed++;
118+ }
119+
120+ return 0;
121+}
122+
123+static int t10_pi_type1_generate_crc(struct blk_integrity_iter *iter)
124+{
125+ return t10_pi_generate(iter, t10_pi_crc_fn, 1);
126+}
127+
128+static int t10_pi_type1_generate_ip(struct blk_integrity_iter *iter)
129+{
130+ return t10_pi_generate(iter, t10_pi_ip_fn, 1);
131+}
132+
133+static int t10_pi_type1_verify_crc(struct blk_integrity_iter *iter)
134+{
135+ return t10_pi_verify(iter, t10_pi_crc_fn, 1);
136+}
137+
138+static int t10_pi_type1_verify_ip(struct blk_integrity_iter *iter)
139+{
140+ return t10_pi_verify(iter, t10_pi_ip_fn, 1);
141+}
142+
143+static int t10_pi_type3_generate_crc(struct blk_integrity_iter *iter)
144+{
145+ return t10_pi_generate(iter, t10_pi_crc_fn, 3);
146+}
147+
148+static int t10_pi_type3_generate_ip(struct blk_integrity_iter *iter)
149+{
150+ return t10_pi_generate(iter, t10_pi_ip_fn, 3);
151+}
152+
153+static int t10_pi_type3_verify_crc(struct blk_integrity_iter *iter)
154+{
155+ return t10_pi_verify(iter, t10_pi_crc_fn, 3);
156+}
157+
158+static int t10_pi_type3_verify_ip(struct blk_integrity_iter *iter)
159+{
160+ return t10_pi_verify(iter, t10_pi_ip_fn, 3);
161+}
162+
163+struct blk_integrity t10_pi_type1_crc = {
164+ .name = "T10-DIF-TYPE1-CRC",
165+ .generate_fn = t10_pi_type1_generate_crc,
166+ .verify_fn = t10_pi_type1_verify_crc,
167+ .tuple_size = sizeof(struct t10_pi_tuple),
168+ .tag_size = 0,
169+};
170+EXPORT_SYMBOL(t10_pi_type1_crc);
171+
172+struct blk_integrity t10_pi_type1_ip = {
173+ .name = "T10-DIF-TYPE1-IP",
174+ .generate_fn = t10_pi_type1_generate_ip,
175+ .verify_fn = t10_pi_type1_verify_ip,
176+ .tuple_size = sizeof(struct t10_pi_tuple),
177+ .tag_size = 0,
178+};
179+EXPORT_SYMBOL(t10_pi_type1_ip);
180+
181+struct blk_integrity t10_pi_type3_crc = {
182+ .name = "T10-DIF-TYPE3-CRC",
183+ .generate_fn = t10_pi_type3_generate_crc,
184+ .verify_fn = t10_pi_type3_verify_crc,
185+ .tuple_size = sizeof(struct t10_pi_tuple),
186+ .tag_size = 0,
187+};
188+EXPORT_SYMBOL(t10_pi_type3_crc);
189+
190+struct blk_integrity t10_pi_type3_ip = {
191+ .name = "T10-DIF-TYPE3-IP",
192+ .generate_fn = t10_pi_type3_generate_ip,
193+ .verify_fn = t10_pi_type3_verify_ip,
194+ .tuple_size = sizeof(struct t10_pi_tuple),
195+ .tag_size = 0,
196+};
197+EXPORT_SYMBOL(t10_pi_type3_ip);
drivers/block/mtip32xx/mtip32xx.c+6−3
@@ -247,7 +247,7 @@ static void mtip_async_complete(struct mtip_port *port, · 246 unmodified lines
247247 if (unlikely(cmd->unaligned))
248248 up(&port->cmd_slot_unal);
249249
250− blk_mq_end_io(rq, status ? -EIO : 0);
250+ blk_mq_end_request(rq, status ? -EIO : 0);
251251 }
252252
253253 /*
@@ -3739,7 +3739,7 @@ static int mtip_submit_request(struct blk_mq_hw_ctx *hctx, struct request *rq) · 3485 unmodified lines
37393739 int err;
37403740
37413741 err = mtip_send_trim(dd, blk_rq_pos(rq), blk_rq_sectors(rq));
3742− blk_mq_end_io(rq, err);
3742+ blk_mq_end_request(rq, err);
37433743 return 0;
37443744 }
37453745
@@ -3775,13 +3775,16 @@ static bool mtip_check_unal_depth(struct blk_mq_hw_ctx *hctx, · 29 unmodified lines
37753775 return false;
37763776 }
37773777
3778−static int mtip_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq)
3778+static int mtip_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq,
3779+ bool last)
37793780 {
37803781 int ret;
37813782
37823783 if (unlikely(mtip_check_unal_depth(hctx, rq)))
37833784 return BLK_MQ_RQ_QUEUE_BUSY;
37843785
3786+ blk_mq_start_request(rq);
3787+
37853788 ret = mtip_submit_request(hctx, rq);
37863789 if (likely(!ret))
37873790 return BLK_MQ_RQ_QUEUE_OK;
drivers/block/null_blk.c+5−2
@@ -177,7 +177,7 @@ static void end_cmd(struct nullb_cmd *cmd) · 176 unmodified lines
177177 {
178178 switch (queue_mode) {
179179 case NULL_Q_MQ:
180− blk_mq_end_io(cmd->rq, 0);
180+ blk_mq_end_request(cmd->rq, 0);
181181 return;
182182 case NULL_Q_RQ:
183183 INIT_LIST_HEAD(&cmd->rq->queuelist);
@@ -313,13 +313,16 @@ static void null_request_fn(struct request_queue *q) · 129 unmodified lines
313313 }
314314 }
315315
316−static int null_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq)
316+static int null_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *rq,
317+ bool last)
317318 {
318319 struct nullb_cmd *cmd = blk_mq_rq_to_pdu(rq);
319320
320321 cmd->rq = rq;
321322 cmd->nq = hctx->driver_data;
322323
324+ blk_mq_start_request(rq);
325+
323326 null_handle_cmd(cmd);
324327 return BLK_MQ_RQ_QUEUE_OK;
325328 }
drivers/block/paride/pd.c+2−0
@@ -722,6 +722,8 @@ static int pd_special_command(struct pd_unit *disk, · 721 unmodified lines
722722 int err = 0;
723723
724724 rq = blk_get_request(disk->gd->queue, READ, __GFP_WAIT);
725+ if (IS_ERR(rq))
726+ return PTR_ERR(rq);
725727
726728 rq->cmd_type = REQ_TYPE_SPECIAL;
727729 rq->special = func;
drivers/block/pktcdvd.c+2−0
@@ -704,6 +704,8 @@ static int pkt_generic_packet(struct pktcdvd_device *pd, struct packet_command * · 703 unmodified lines
704704
705705 rq = blk_get_request(q, (cgc->data_direction == CGC_DATA_WRITE) ?
706706 WRITE : READ, __GFP_WAIT);
707+ if (IS_ERR(rq))
708+ return PTR_ERR(rq);
707709 blk_rq_set_block_pc(rq);
708710
709711 if (cgc->buflen) {
drivers/block/sx8.c+1−1
@@ -568,7 +568,7 @@ static struct carm_request *carm_get_special(struct carm_host *host) · 567 unmodified lines
568568 return NULL;
569569
570570 rq = blk_get_request(host->oob_q, WRITE /* bogus */, GFP_KERNEL);
571− if (!rq) {
571+ if (IS_ERR(rq)) {
572572 spin_lock_irqsave(&host->lock, flags);
573573 carm_put_request(host, crq);
574574 spin_unlock_irqrestore(&host->lock, flags);
drivers/block/virtio_blk.c+5−3
@@ -129,7 +129,7 @@ static inline void virtblk_request_done(struct request *req) · 128 unmodified lines
129129 req->errors = (error != 0);
130130 }
131131
132− blk_mq_end_io(req, error);
132+ blk_mq_end_request(req, error);
133133 }
134134
135135 static void virtblk_done(struct virtqueue *vq)
@@ -158,14 +158,14 @@ static void virtblk_done(struct virtqueue *vq) · 22 unmodified lines
158158 spin_unlock_irqrestore(&vblk->vqs[qid].lock, flags);
159159 }
160160
161−static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req)
161+static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req,
162+ bool last)
162163 {
163164 struct virtio_blk *vblk = hctx->queue->queuedata;
164165 struct virtblk_req *vbr = blk_mq_rq_to_pdu(req);
165166 unsigned long flags;
166167 unsigned int num;
167168 int qid = hctx->queue_num;
168− const bool last = (req->cmd_flags & REQ_END) != 0;
169169 int err;
170170 bool notify = false;
171171
@@ -199,6 +199,8 @@ static int virtio_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req) · 27 unmodified lines
199199 }
200200 }
201201
202+ blk_mq_start_request(req);
203+
202204 num = blk_rq_map_sg(hctx->queue, vbr->req, vbr->sg);
203205 if (num) {
204206 if (rq_data_dir(vbr->req) == WRITE)
drivers/cdrom/cdrom.c+2−2
@@ -2180,8 +2180,8 @@ static int cdrom_read_cdda_bpc(struct cdrom_device_info *cdi, __u8 __user *ubuf, · 2179 unmodified lines
21802180 len = nr * CD_FRAMESIZE_RAW;
21812181
21822182 rq = blk_get_request(q, READ, GFP_KERNEL);
2183− if (!rq) {
2184− ret = -ENOMEM;
2183+ if (IS_ERR(rq)) {
2184+ ret = PTR_ERR(rq);
21852185 break;
21862186 }
21872187 blk_rq_set_block_pc(rq);
drivers/ide/ide-park.c+1−1
@@ -46,7 +46,7 @@ static void issue_park_cmd(ide_drive_t *drive, unsigned long timeout) · 45 unmodified lines
4646 * timeout has expired, so power management will be reenabled.
4747 */
4848 rq = blk_get_request(q, READ, GFP_NOWAIT);
49− if (unlikely(!rq))
49+ if (IS_ERR(rq))
5050 goto out;
5151
5252 rq->cmd[0] = REQ_UNPARK_HEADS;
drivers/scsi/Kconfig+0−1
@@ -73,7 +73,6 @@ comment "SCSI support type (disk, tape, CD-ROM)" · 72 unmodified lines
7373 config BLK_DEV_SD
7474 tristate "SCSI disk support"
7575 depends on SCSI
76− select CRC_T10DIF if BLK_DEV_INTEGRITY
7776 ---help---
7877 If you want to use SCSI hard disks, Fibre Channel disks,
7978 Serial ATA (SATA) or Parallel ATA (PATA) hard disks,
drivers/scsi/device_handler/scsi_dh_alua.c+1−1
@@ -115,7 +115,7 @@ static struct request *get_alua_req(struct scsi_device *sdev, · 114 unmodified lines
115115
116116 rq = blk_get_request(q, rw, GFP_NOIO);
117117
118− if (!rq) {
118+ if (IS_ERR(rq)) {
119119 sdev_printk(KERN_INFO, sdev,
120120 "%s: blk_get_request failed\n", __func__);
121121 return NULL;
drivers/scsi/device_handler/scsi_dh_emc.c+1−1
@@ -275,7 +275,7 @@ static struct request *get_req(struct scsi_device *sdev, int cmd, · 274 unmodified lines
275275
276276 rq = blk_get_request(sdev->request_queue,
277277 (cmd != INQUIRY) ? WRITE : READ, GFP_NOIO);
278− if (!rq) {
278+ if (IS_ERR(rq)) {
279279 sdev_printk(KERN_INFO, sdev, "get_req: blk_get_request failed");
280280 return NULL;
281281 }
drivers/scsi/device_handler/scsi_dh_hp_sw.c+2−2
@@ -117,7 +117,7 @@ static int hp_sw_tur(struct scsi_device *sdev, struct hp_sw_dh_data *h) · 116 unmodified lines
117117
118118 retry:
119119 req = blk_get_request(sdev->request_queue, WRITE, GFP_NOIO);
120− if (!req)
120+ if (IS_ERR(req))
121121 return SCSI_DH_RES_TEMP_UNAVAIL;
122122
123123 blk_rq_set_block_pc(req);
@@ -247,7 +247,7 @@ static int hp_sw_start_stop(struct hp_sw_dh_data *h) · 123 unmodified lines
247247 struct request *req;
248248
249249 req = blk_get_request(h->sdev->request_queue, WRITE, GFP_ATOMIC);
250− if (!req)
250+ if (IS_ERR(req))
251251 return SCSI_DH_RES_TEMP_UNAVAIL;
252252
253253 blk_rq_set_block_pc(req);
drivers/scsi/device_handler/scsi_dh_rdac.c+1−1
@@ -274,7 +274,7 @@ static struct request *get_rdac_req(struct scsi_device *sdev, · 273 unmodified lines
274274
275275 rq = blk_get_request(q, rw, GFP_NOIO);
276276
277− if (!rq) {
277+ if (IS_ERR(rq)) {
278278 sdev_printk(KERN_INFO, sdev,
279279 "get_rdac_req: blk_get_request failed.\n");
280280 return NULL;
drivers/scsi/osd/osd_initiator.c+2−2
@@ -1567,8 +1567,8 @@ static struct request *_make_request(struct request_queue *q, bool has_write, · 1566 unmodified lines
15671567 struct request *req;
15681568
15691569 req = blk_get_request(q, has_write ? WRITE : READ, flags);
1570− if (unlikely(!req))
1571− return ERR_PTR(-ENOMEM);
1570+ if (IS_ERR(req))
1571+ return req;
15721572
15731573 blk_rq_set_block_pc(req);
15741574 return req;
drivers/scsi/osst.c+1−1
@@ -362,7 +362,7 @@ static int osst_execute(struct osst_request *SRpnt, const unsigned char *cmd, · 361 unmodified lines
362362 int write = (data_direction == DMA_TO_DEVICE);
363363
364364 req = blk_get_request(SRpnt->stp->device->request_queue, write, GFP_KERNEL);
365− if (!req)
365+ if (IS_ERR(req))
366366 return DRIVER_ERROR << 24;
367367
368368 blk_rq_set_block_pc(req);
drivers/scsi/scsi_error.c+2−0
@@ -1961,6 +1961,8 @@ static void scsi_eh_lock_door(struct scsi_device *sdev) · 1960 unmodified lines
19611961 * request becomes available
19621962 */
19631963 req = blk_get_request(sdev->request_queue, READ, GFP_KERNEL);
1964+ if (IS_ERR(req))
1965+ return;
19641966
19651967 blk_rq_set_block_pc(req);
19661968
drivers/scsi/scsi_lib.c+19−5
@@ -221,7 +221,7 @@ int scsi_execute(struct scsi_device *sdev, const unsigned char *cmd, · 220 unmodified lines
221221 int ret = DRIVER_ERROR << 24;
222222
223223 req = blk_get_request(sdev->request_queue, write, __GFP_WAIT);
224− if (!req)
224+ if (IS_ERR(req))
225225 return ret;
226226 blk_rq_set_block_pc(req);
227227
@@ -715,7 +715,7 @@ static bool scsi_end_request(struct request *req, int error, · 487 unmodified lines
715715
716716 if (req->mq_ctx) {
717717 /*
718− * In the MQ case the command gets freed by __blk_mq_end_io,
718+ * In the MQ case the command gets freed by __blk_mq_end_request,
719719 * so we have to do all cleanup that depends on it earlier.
720720 *
721721 * We also can't kick the queues from irq context, so we
@@ -723,7 +723,7 @@ static bool scsi_end_request(struct request *req, int error, · 1 unmodified line
723723 */
724724 scsi_mq_uninit_cmd(cmd);
725725
726− __blk_mq_end_io(req, error);
726+ __blk_mq_end_request(req, error);
727727
728728 if (scsi_target(sdev)->single_lun ||
729729 !list_empty(&sdev->host->starved_list))
@@ -1847,6 +1847,8 @@ static int scsi_mq_prep_fn(struct request *req) · 1117 unmodified lines
18471847 next_rq->special = bidi_sdb;
18481848 }
18491849
1850+ blk_mq_start_request(req);
1851+
18501852 return scsi_setup_cmnd(sdev, req);
18511853 }
18521854
@@ -1856,7 +1858,8 @@ static void scsi_mq_done(struct scsi_cmnd *cmd) · 3 unmodified lines
18561858 blk_mq_complete_request(cmd->request);
18571859 }
18581860
1859−static int scsi_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req)
1861+static int scsi_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req,
1862+ bool last)
18601863 {
18611864 struct request_queue *q = req->q;
18621865 struct scsi_device *sdev = q->queuedata;
@@ -1880,11 +1883,14 @@ static int scsi_queue_rq(struct blk_mq_hw_ctx *hctx, struct request *req) · 17 unmodified lines
18801883 if (!scsi_host_queue_ready(q, shost, sdev))
18811884 goto out_dec_target_busy;
18821885
1886+
18831887 if (!(req->cmd_flags & REQ_DONTPREP)) {
18841888 ret = prep_to_mq(scsi_mq_prep_fn(req));
18851889 if (ret)
18861890 goto out_dec_host_busy;
18871891 req->cmd_flags |= REQ_DONTPREP;
1892+ } else {
1893+ blk_mq_start_request(req);
18881894 }
18891895
18901896 scsi_init_cmd_errh(cmd);
@@ -1931,6 +1937,14 @@ out: · 40 unmodified lines
19311937 return ret;
19321938 }
19331939
1940+static enum blk_eh_timer_return scsi_timeout(struct request *req,
1941+ bool reserved)
1942+{
1943+ if (reserved)
1944+ return BLK_EH_RESET_TIMER;
1945+ return scsi_times_out(req);
1946+}
1947+
19341948 static int scsi_init_request(void *data, struct request *rq,
19351949 unsigned int hctx_idx, unsigned int request_idx,
19361950 unsigned int numa_node)
@@ -2042,7 +2056,7 @@ static struct blk_mq_ops scsi_mq_ops = { · 105 unmodified lines
20422056 .map_queue = blk_mq_map_queue,
20432057 .queue_rq = scsi_queue_rq,
20442058 .complete = scsi_softirq_done,
2045− .timeout = scsi_times_out,
2059+ .timeout = scsi_timeout,
20462060 .init_request = scsi_init_request,
20472061 .exit_request = scsi_exit_request,
20482062 };
drivers/scsi/sd.c+43−30
@@ -610,29 +610,44 @@ static void scsi_disk_put(struct scsi_disk *sdkp) · 609 unmodified lines
610610 mutex_unlock(&sd_ref_mutex);
611611 }
612612
613−static void sd_prot_op(struct scsi_cmnd *scmd, unsigned int dif)
614−{
615− unsigned int prot_op = SCSI_PROT_NORMAL;
616− unsigned int dix = scsi_prot_sg_count(scmd);
617−
618− if (scmd->sc_data_direction == DMA_FROM_DEVICE) {
619− if (dif && dix)
620− prot_op = SCSI_PROT_READ_PASS;
621− else if (dif && !dix)
622− prot_op = SCSI_PROT_READ_STRIP;
623− else if (!dif && dix)
624− prot_op = SCSI_PROT_READ_INSERT;
625− } else {
626− if (dif && dix)
627− prot_op = SCSI_PROT_WRITE_PASS;
628− else if (dif && !dix)
629− prot_op = SCSI_PROT_WRITE_INSERT;
630− else if (!dif && dix)
631− prot_op = SCSI_PROT_WRITE_STRIP;
613+
614+
615+static unsigned char sd_setup_protect_cmnd(struct scsi_cmnd *scmd,
616+ unsigned int dix, unsigned int dif)
617+{
618+ struct bio *bio = scmd->request->bio;
619+ unsigned int prot_op = sd_prot_op(rq_data_dir(scmd->request), dix, dif);
620+ unsigned int protect = 0;
621+
622+ if (dix) { /* DIX Type 0, 1, 2, 3 */
623+ if (bio_integrity_flagged(bio, BIP_IP_CHECKSUM))
624+ scmd->prot_flags |= SCSI_PROT_IP_CHECKSUM;
625+
626+ if (bio_integrity_flagged(bio, BIP_CTRL_NOCHECK) == false)
627+ scmd->prot_flags |= SCSI_PROT_GUARD_CHECK;
628+ }
629+
630+ if (dif != SD_DIF_TYPE3_PROTECTION) { /* DIX/DIF Type 0, 1, 2 */
631+ scmd->prot_flags |= SCSI_PROT_REF_INCREMENT;
632+
633+ if (bio_integrity_flagged(bio, BIP_CTRL_NOCHECK) == false)
634+ scmd->prot_flags |= SCSI_PROT_REF_CHECK;
635+ }
636+
637+ if (dif) { /* DIX/DIF Type 1, 2, 3 */
638+ scmd->prot_flags |= SCSI_PROT_TRANSFER_PI;
639+
640+ if (bio_integrity_flagged(bio, BIP_DISK_NOCHECK))
641+ protect = 3 << 5; /* Disable target PI checking */
642+ else
643+ protect = 1 << 5; /* Enable target PI checking */
632644 }
633645
634646 scsi_set_prot_op(scmd, prot_op);
635647 scsi_set_prot_type(scmd, dif);
648+ scmd->prot_flags &= sd_prot_flag_mask(prot_op);
649+
650+ return protect;
636651 }
637652
638653 static void sd_config_discard(struct scsi_disk *sdkp, unsigned int mode)
@@ -893,7 +908,8 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) · 254 unmodified lines
893908 sector_t block = blk_rq_pos(rq);
894909 sector_t threshold;
895910 unsigned int this_count = blk_rq_sectors(rq);
896− int ret, host_dif;
911+ unsigned int dif, dix;
912+ int ret;
897913 unsigned char protect;
898914
899915 ret = scsi_init_io(SCpnt, GFP_ATOMIC);
@@ -995,7 +1011,7 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) · 95 unmodified lines
9951011 SCpnt->cmnd[0] = WRITE_6;
9961012
9971013 if (blk_integrity_rq(rq))
998− sd_dif_prepare(rq, block, sdp->sector_size);
1014+ sd_dif_prepare(SCpnt);
9991015
10001016 } else if (rq_data_dir(rq) == READ) {
10011017 SCpnt->cmnd[0] = READ_6;
@@ -1010,14 +1026,15 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) · 8 unmodified lines
10101026 "writing" : "reading", this_count,
10111027 blk_rq_sectors(rq)));
10121028
1013− /* Set RDPROTECT/WRPROTECT if disk is formatted with DIF */
1014− host_dif = scsi_host_dif_capable(sdp->host, sdkp->protection_type);
1015− if (host_dif)
1016− protect = 1 << 5;
1029+ dix = scsi_prot_sg_count(SCpnt);
1030+ dif = scsi_host_dif_capable(SCpnt->device->host, sdkp->protection_type);
1031+
1032+ if (dif || dix)
1033+ protect = sd_setup_protect_cmnd(SCpnt, dix, dif);
10171034 else
10181035 protect = 0;
10191036
1020− if (host_dif == SD_DIF_TYPE2_PROTECTION) {
1037+ if (protect && sdkp->protection_type == SD_DIF_TYPE2_PROTECTION) {
10211038 SCpnt->cmnd = mempool_alloc(sd_cdb_pool, GFP_ATOMIC);
10221039
10231040 if (unlikely(SCpnt->cmnd == NULL)) {
@@ -1102,10 +1119,6 @@ static int sd_setup_read_write_cmnd(struct scsi_cmnd *SCpnt) · 78 unmodified lines
11021119 }
11031120 SCpnt->sdb.length = this_count * sdp->sector_size;
11041121
1105− /* If DIF or DIX is enabled, tell HBA how to handle request */
1106− if (host_dif || scsi_prot_sg_count(SCpnt))
1107− sd_prot_op(SCpnt, host_dif);
1108−
11091122 /*
11101123 * We shouldn't disconnect in the middle of a sector, so with a dumb
11111124 * host adapter, it's safe to assume that we can at least transfer
drivers/scsi/sd.h+64−2
@@ -166,6 +166,68 @@ enum sd_dif_target_protection_types { · 165 unmodified lines
166166 SD_DIF_TYPE3_PROTECTION = 0x3,
167167 };
168168
169+/*
170+ * Look up the DIX operation based on whether the command is read or
171+ * write and whether dix and dif are enabled.
172+ */
173+static inline unsigned int sd_prot_op(bool write, bool dix, bool dif)
174+{
175+ /* Lookup table: bit 2 (write), bit 1 (dix), bit 0 (dif) */
176+ const unsigned int ops[] = { /* wrt dix dif */
177+ SCSI_PROT_NORMAL, /* 0 0 0 */
178+ SCSI_PROT_READ_STRIP, /* 0 0 1 */
179+ SCSI_PROT_READ_INSERT, /* 0 1 0 */
180+ SCSI_PROT_READ_PASS, /* 0 1 1 */
181+ SCSI_PROT_NORMAL, /* 1 0 0 */
182+ SCSI_PROT_WRITE_INSERT, /* 1 0 1 */
183+ SCSI_PROT_WRITE_STRIP, /* 1 1 0 */
184+ SCSI_PROT_WRITE_PASS, /* 1 1 1 */
185+ };
186+
187+ return ops[write << 2 | dix << 1 | dif];
188+}
189+
190+/*
191+ * Returns a mask of the protection flags that are valid for a given DIX
192+ * operation.
193+ */
194+static inline unsigned int sd_prot_flag_mask(unsigned int prot_op)
195+{
196+ const unsigned int flag_mask[] = {
197+ [SCSI_PROT_NORMAL] = 0,
198+
199+ [SCSI_PROT_READ_STRIP] = SCSI_PROT_TRANSFER_PI |
200+ SCSI_PROT_GUARD_CHECK |
201+ SCSI_PROT_REF_CHECK |
202+ SCSI_PROT_REF_INCREMENT,
203+
204+ [SCSI_PROT_READ_INSERT] = SCSI_PROT_REF_INCREMENT |
205+ SCSI_PROT_IP_CHECKSUM,
206+
207+ [SCSI_PROT_READ_PASS] = SCSI_PROT_TRANSFER_PI |
208+ SCSI_PROT_GUARD_CHECK |
209+ SCSI_PROT_REF_CHECK |
210+ SCSI_PROT_REF_INCREMENT |
211+ SCSI_PROT_IP_CHECKSUM,
212+
213+ [SCSI_PROT_WRITE_INSERT] = SCSI_PROT_TRANSFER_PI |
214+ SCSI_PROT_REF_INCREMENT,
215+
216+ [SCSI_PROT_WRITE_STRIP] = SCSI_PROT_GUARD_CHECK |
217+ SCSI_PROT_REF_CHECK |
218+ SCSI_PROT_REF_INCREMENT |
219+ SCSI_PROT_IP_CHECKSUM,
220+
221+ [SCSI_PROT_WRITE_PASS] = SCSI_PROT_TRANSFER_PI |
222+ SCSI_PROT_GUARD_CHECK |
223+ SCSI_PROT_REF_CHECK |
224+ SCSI_PROT_REF_INCREMENT |
225+ SCSI_PROT_IP_CHECKSUM,
226+ };
227+
228+ return flag_mask[prot_op];
229+}
230+
169231 /*
170232 * Data Integrity Field tuple.
171233 */
@@ -178,7 +240,7 @@ struct sd_dif_tuple { · 6 unmodified lines
178240 #ifdef CONFIG_BLK_DEV_INTEGRITY
179241
180242 extern void sd_dif_config_host(struct scsi_disk *);
181−extern void sd_dif_prepare(struct request *rq, sector_t, unsigned int);
243+extern void sd_dif_prepare(struct scsi_cmnd *scmd);
182244 extern void sd_dif_complete(struct scsi_cmnd *, unsigned int);
183245
184246 #else /* CONFIG_BLK_DEV_INTEGRITY */
@@ -186,7 +248,7 @@ extern void sd_dif_complete(struct scsi_cmnd *, unsigned int); · 1 unmodified line
186248 static inline void sd_dif_config_host(struct scsi_disk *disk)
187249 {
188250 }
189−static inline int sd_dif_prepare(struct request *rq, sector_t s, unsigned int a)
251+static inline int sd_dif_prepare(struct scsi_cmnd *scmd)
190252 {
191253 return 0;
192254 }
drivers/scsi/sd_dif.c+48−305
@@ -21,7 +21,7 @@ · 20 unmodified lines
2121 */
2222
2323 #include <linux/blkdev.h>
24−#include <linux/crc-t10dif.h>
24+#include <linux/t10-pi.h>
2525
2626 #include <scsi/scsi.h>
2727 #include <scsi/scsi_cmnd.h>
@@ -33,268 +33,8 @@ · 5 unmodified lines
3333 #include <scsi/scsi_ioctl.h>
3434 #include <scsi/scsicam.h>
3535
36−#include <net/checksum.h>
37−
3836 #include "sd.h"
3937
40−typedef __u16 (csum_fn) (void *, unsigned int);
41−
42−static __u16 sd_dif_crc_fn(void *data, unsigned int len)
43−{
44− return cpu_to_be16(crc_t10dif(data, len));
45−}
46−
47−static __u16 sd_dif_ip_fn(void *data, unsigned int len)
48−{
49− return ip_compute_csum(data, len);
50−}
51−
52−/*
53− * Type 1 and Type 2 protection use the same format: 16 bit guard tag,
54− * 16 bit app tag, 32 bit reference tag.
55− */
56−static void sd_dif_type1_generate(struct blk_integrity_exchg *bix, csum_fn *fn)
57−{
58− void *buf = bix->data_buf;
59− struct sd_dif_tuple *sdt = bix->prot_buf;
60− sector_t sector = bix->sector;
61− unsigned int i;
62−
63− for (i = 0 ; i < bix->data_size ; i += bix->sector_size, sdt++) {
64− sdt->guard_tag = fn(buf, bix->sector_size);
65− sdt->ref_tag = cpu_to_be32(sector & 0xffffffff);
66− sdt->app_tag = 0;
67−
68− buf += bix->sector_size;
69− sector++;
70− }
71−}
72−
73−static void sd_dif_type1_generate_crc(struct blk_integrity_exchg *bix)
74−{
75− sd_dif_type1_generate(bix, sd_dif_crc_fn);
76−}
77−
78−static void sd_dif_type1_generate_ip(struct blk_integrity_exchg *bix)
79−{
80− sd_dif_type1_generate(bix, sd_dif_ip_fn);
81−}
82−
83−static int sd_dif_type1_verify(struct blk_integrity_exchg *bix, csum_fn *fn)
84−{
85− void *buf = bix->data_buf;
86− struct sd_dif_tuple *sdt = bix->prot_buf;
87− sector_t sector = bix->sector;
88− unsigned int i;
89− __u16 csum;
90−
91− for (i = 0 ; i < bix->data_size ; i += bix->sector_size, sdt++) {
92− /* Unwritten sectors */
93− if (sdt->app_tag == 0xffff)
94− return 0;
95−
96− if (be32_to_cpu(sdt->ref_tag) != (sector & 0xffffffff)) {
97− printk(KERN_ERR
98− "%s: ref tag error on sector %lu (rcvd %u)\n",
99− bix->disk_name, (unsigned long)sector,
100− be32_to_cpu(sdt->ref_tag));
101− return -EIO;
102− }
103−
104− csum = fn(buf, bix->sector_size);
105−
106− if (sdt->guard_tag != csum) {
107− printk(KERN_ERR "%s: guard tag error on sector %lu " \
108− "(rcvd %04x, data %04x)\n", bix->disk_name,
109− (unsigned long)sector,
110− be16_to_cpu(sdt->guard_tag), be16_to_cpu(csum));
111− return -EIO;
112− }
113−
114− buf += bix->sector_size;
115− sector++;
116− }
117−
118− return 0;
119−}
120−
121−static int sd_dif_type1_verify_crc(struct blk_integrity_exchg *bix)
122−{
123− return sd_dif_type1_verify(bix, sd_dif_crc_fn);
124−}
125−
126−static int sd_dif_type1_verify_ip(struct blk_integrity_exchg *bix)
127−{
128− return sd_dif_type1_verify(bix, sd_dif_ip_fn);
129−}
130−
131−/*
132− * Functions for interleaving and deinterleaving application tags
133− */
134−static void sd_dif_type1_set_tag(void *prot, void *tag_buf, unsigned int sectors)
135−{
136− struct sd_dif_tuple *sdt = prot;
137− u8 *tag = tag_buf;
138− unsigned int i, j;
139−
140− for (i = 0, j = 0 ; i < sectors ; i++, j += 2, sdt++) {
141− sdt->app_tag = tag[j] << 8 | tag[j+1];
142− BUG_ON(sdt->app_tag == 0xffff);
143− }
144−}
145−
146−static void sd_dif_type1_get_tag(void *prot, void *tag_buf, unsigned int sectors)
147−{
148− struct sd_dif_tuple *sdt = prot;
149− u8 *tag = tag_buf;
150− unsigned int i, j;
151−
152− for (i = 0, j = 0 ; i < sectors ; i++, j += 2, sdt++) {
153− tag[j] = (sdt->app_tag & 0xff00) >> 8;
154− tag[j+1] = sdt->app_tag & 0xff;
155− }
156−}
157−
158−static struct blk_integrity dif_type1_integrity_crc = {
159− .name = "T10-DIF-TYPE1-CRC",
160− .generate_fn = sd_dif_type1_generate_crc,
161− .verify_fn = sd_dif_type1_verify_crc,
162− .get_tag_fn = sd_dif_type1_get_tag,
163− .set_tag_fn = sd_dif_type1_set_tag,
164− .tuple_size = sizeof(struct sd_dif_tuple),
165− .tag_size = 0,
166−};
167−
168−static struct blk_integrity dif_type1_integrity_ip = {
169− .name = "T10-DIF-TYPE1-IP",
170− .generate_fn = sd_dif_type1_generate_ip,
171− .verify_fn = sd_dif_type1_verify_ip,
172− .get_tag_fn = sd_dif_type1_get_tag,
173− .set_tag_fn = sd_dif_type1_set_tag,
174− .tuple_size = sizeof(struct sd_dif_tuple),
175− .tag_size = 0,
176−};
177−
178−
179−/*
180− * Type 3 protection has a 16-bit guard tag and 16 + 32 bits of opaque
181− * tag space.
182− */
183−static void sd_dif_type3_generate(struct blk_integrity_exchg *bix, csum_fn *fn)
184−{
185− void *buf = bix->data_buf;
186− struct sd_dif_tuple *sdt = bix->prot_buf;
187− unsigned int i;
188−
189− for (i = 0 ; i < bix->data_size ; i += bix->sector_size, sdt++) {
190− sdt->guard_tag = fn(buf, bix->sector_size);
191− sdt->ref_tag = 0;
192− sdt->app_tag = 0;
193−
194− buf += bix->sector_size;
195− }
196−}
197−
198−static void sd_dif_type3_generate_crc(struct blk_integrity_exchg *bix)
199−{
200− sd_dif_type3_generate(bix, sd_dif_crc_fn);
201−}
202−
203−static void sd_dif_type3_generate_ip(struct blk_integrity_exchg *bix)
204−{
205− sd_dif_type3_generate(bix, sd_dif_ip_fn);
206−}
207−
208−static int sd_dif_type3_verify(struct blk_integrity_exchg *bix, csum_fn *fn)
209−{
210− void *buf = bix->data_buf;
211− struct sd_dif_tuple *sdt = bix->prot_buf;
212− sector_t sector = bix->sector;
213− unsigned int i;
214− __u16 csum;
215−
216− for (i = 0 ; i < bix->data_size ; i += bix->sector_size, sdt++) {
217− /* Unwritten sectors */
218− if (sdt->app_tag == 0xffff && sdt->ref_tag == 0xffffffff)
219− return 0;
220−
221− csum = fn(buf, bix->sector_size);
222−
223− if (sdt->guard_tag != csum) {
224− printk(KERN_ERR "%s: guard tag error on sector %lu " \
225− "(rcvd %04x, data %04x)\n", bix->disk_name,
226− (unsigned long)sector,
227− be16_to_cpu(sdt->guard_tag), be16_to_cpu(csum));
228− return -EIO;
229− }
230−
231− buf += bix->sector_size;
232− sector++;
233− }
234−
235− return 0;
236−}
237−
238−static int sd_dif_type3_verify_crc(struct blk_integrity_exchg *bix)
239−{
240− return sd_dif_type3_verify(bix, sd_dif_crc_fn);
241−}
242−
243−static int sd_dif_type3_verify_ip(struct blk_integrity_exchg *bix)
244−{
245− return sd_dif_type3_verify(bix, sd_dif_ip_fn);
246−}
247−
248−static void sd_dif_type3_set_tag(void *prot, void *tag_buf, unsigned int sectors)
249−{
250− struct sd_dif_tuple *sdt = prot;
251− u8 *tag = tag_buf;
252− unsigned int i, j;
253−
254− for (i = 0, j = 0 ; i < sectors ; i++, j += 6, sdt++) {
255− sdt->app_tag = tag[j] << 8 | tag[j+1];
256− sdt->ref_tag = tag[j+2] << 24 | tag[j+3] << 16 |
257− tag[j+4] << 8 | tag[j+5];
258− }
259−}
260−
261−static void sd_dif_type3_get_tag(void *prot, void *tag_buf, unsigned int sectors)
262−{
263− struct sd_dif_tuple *sdt = prot;
264− u8 *tag = tag_buf;
265− unsigned int i, j;
266−
267− for (i = 0, j = 0 ; i < sectors ; i++, j += 2, sdt++) {
268− tag[j] = (sdt->app_tag & 0xff00) >> 8;
269− tag[j+1] = sdt->app_tag & 0xff;
270− tag[j+2] = (sdt->ref_tag & 0xff000000) >> 24;
271− tag[j+3] = (sdt->ref_tag & 0xff0000) >> 16;
272− tag[j+4] = (sdt->ref_tag & 0xff00) >> 8;
273− tag[j+5] = sdt->ref_tag & 0xff;
274− BUG_ON(sdt->app_tag == 0xffff || sdt->ref_tag == 0xffffffff);
275− }
276−}
277−
278−static struct blk_integrity dif_type3_integrity_crc = {
279− .name = "T10-DIF-TYPE3-CRC",
280− .generate_fn = sd_dif_type3_generate_crc,
281− .verify_fn = sd_dif_type3_verify_crc,
282− .get_tag_fn = sd_dif_type3_get_tag,
283− .set_tag_fn = sd_dif_type3_set_tag,
284− .tuple_size = sizeof(struct sd_dif_tuple),
285− .tag_size = 0,
286−};
287−
288−static struct blk_integrity dif_type3_integrity_ip = {
289− .name = "T10-DIF-TYPE3-IP",
290− .generate_fn = sd_dif_type3_generate_ip,
291− .verify_fn = sd_dif_type3_verify_ip,
292− .get_tag_fn = sd_dif_type3_get_tag,
293− .set_tag_fn = sd_dif_type3_set_tag,
294− .tuple_size = sizeof(struct sd_dif_tuple),
295− .tag_size = 0,
296−};
297−
29838 /*
29939 * Configure exchange of protection information between OS and HBA.
30040 */
@@ -316,22 +56,30 @@ void sd_dif_config_host(struct scsi_disk *sdkp) · 15 unmodified lines
31656 return;
31757
31858 /* Enable DMA of protection information */
319− if (scsi_host_get_guard(sdkp->device->host) & SHOST_DIX_GUARD_IP)
59+ if (scsi_host_get_guard(sdkp->device->host) & SHOST_DIX_GUARD_IP) {
32060 if (type == SD_DIF_TYPE3_PROTECTION)
321− blk_integrity_register(disk, &dif_type3_integrity_ip);
61+ blk_integrity_register(disk, &t10_pi_type3_ip);
32262 else
323− blk_integrity_register(disk, &dif_type1_integrity_ip);
324− else
63+ blk_integrity_register(disk, &t10_pi_type1_ip);
64+
65+ disk->integrity->flags |= BLK_INTEGRITY_IP_CHECKSUM;
66+ } else
32567 if (type == SD_DIF_TYPE3_PROTECTION)
326− blk_integrity_register(disk, &dif_type3_integrity_crc);
68+ blk_integrity_register(disk, &t10_pi_type3_crc);
32769 else
328− blk_integrity_register(disk, &dif_type1_integrity_crc);
70+ blk_integrity_register(disk, &t10_pi_type1_crc);
32971
33072 sd_printk(KERN_NOTICE, sdkp,
33173 "Enabling DIX %s protection\n", disk->integrity->name);
33274
33375 /* Signal to block layer that we support sector tagging */
334− if (dif && type && sdkp->ATO) {
76+ if (dif && type) {
77+
78+ disk->integrity->flags |= BLK_INTEGRITY_DEVICE_CAPABLE;
79+
80+ if (!sdkp)
81+ return;
82+
33583 if (type == SD_DIF_TYPE3_PROTECTION)
33684 disk->integrity->tag_size = sizeof(u16) + sizeof(u32);
33785 else
@@ -358,50 +106,49 @@ void sd_dif_config_host(struct scsi_disk *sdkp) · 20 unmodified lines
358106 *
359107 * Type 3 does not have a reference tag so no remapping is required.
360108 */
361−void sd_dif_prepare(struct request *rq, sector_t hw_sector,
362− unsigned int sector_sz)
109+void sd_dif_prepare(struct scsi_cmnd *scmd)
363110 {
364− const int tuple_sz = sizeof(struct sd_dif_tuple);
111+ const int tuple_sz = sizeof(struct t10_pi_tuple);
365112 struct bio *bio;
366113 struct scsi_disk *sdkp;
367− struct sd_dif_tuple *sdt;
114+ struct t10_pi_tuple *pi;
368115 u32 phys, virt;
369116
370− sdkp = rq->bio->bi_bdev->bd_disk->private_data;
117+ sdkp = scsi_disk(scmd->request->rq_disk);
371118
372119 if (sdkp->protection_type == SD_DIF_TYPE3_PROTECTION)
373120 return;
374121
375− phys = hw_sector & 0xffffffff;
122+ phys = scsi_prot_ref_tag(scmd);
376123
377− __rq_for_each_bio(bio, rq) {
124+ __rq_for_each_bio(bio, scmd->request) {
125+ struct bio_integrity_payload *bip = bio_integrity(bio);
378126 struct bio_vec iv;
379127 struct bvec_iter iter;
380128 unsigned int j;
381129
382130 /* Already remapped? */
383− if (bio_flagged(bio, BIO_MAPPED_INTEGRITY))
131+ if (bip->bip_flags & BIP_MAPPED_INTEGRITY)
384132 break;
385133
386− virt = bio->bi_integrity->bip_iter.bi_sector & 0xffffffff;
134+ virt = bip_get_seed(bip) & 0xffffffff;
387135
388− bip_for_each_vec(iv, bio->bi_integrity, iter) {
389− sdt = kmap_atomic(iv.bv_page)
390− + iv.bv_offset;
136+ bip_for_each_vec(iv, bip, iter) {
137+ pi = kmap_atomic(iv.bv_page) + iv.bv_offset;
391138
392− for (j = 0; j < iv.bv_len; j += tuple_sz, sdt++) {
139+ for (j = 0; j < iv.bv_len; j += tuple_sz, pi++) {
393140
394− if (be32_to_cpu(sdt->ref_tag) == virt)
395− sdt->ref_tag = cpu_to_be32(phys);
141+ if (be32_to_cpu(pi->ref_tag) == virt)
142+ pi->ref_tag = cpu_to_be32(phys);
396143
397144 virt++;
398145 phys++;
399146 }
400147
401− kunmap_atomic(sdt);
148+ kunmap_atomic(pi);
402149 }
403150
404− bio->bi_flags |= (1 << BIO_MAPPED_INTEGRITY);
151+ bip->bip_flags |= BIP_MAPPED_INTEGRITY;
405152 }
406153 }
407154
@@ -411,11 +158,11 @@ void sd_dif_prepare(struct request *rq, sector_t hw_sector, · 3 unmodified lines
411158 */
412159 void sd_dif_complete(struct scsi_cmnd *scmd, unsigned int good_bytes)
413160 {
414− const int tuple_sz = sizeof(struct sd_dif_tuple);
161+ const int tuple_sz = sizeof(struct t10_pi_tuple);
415162 struct scsi_disk *sdkp;
416163 struct bio *bio;
417− struct sd_dif_tuple *sdt;
418− unsigned int j, sectors, sector_sz;
164+ struct t10_pi_tuple *pi;
165+ unsigned int j, intervals;
419166 u32 phys, virt;
420167
421168 sdkp = scsi_disk(scmd->request->rq_disk);
@@ -423,39 +170,35 @@ void sd_dif_complete(struct scsi_cmnd *scmd, unsigned int good_bytes) · 1 unmodified line
423170 if (sdkp->protection_type == SD_DIF_TYPE3_PROTECTION || good_bytes == 0)
424171 return;
425172
426− sector_sz = scmd->device->sector_size;
427− sectors = good_bytes / sector_sz;
428−
429− phys = blk_rq_pos(scmd->request) & 0xffffffff;
430− if (sector_sz == 4096)
431− phys >>= 3;
173+ intervals = good_bytes / scsi_prot_interval(scmd);
174+ phys = scsi_prot_ref_tag(scmd);
432175
433176 __rq_for_each_bio(bio, scmd->request) {
177+ struct bio_integrity_payload *bip = bio_integrity(bio);
434178 struct bio_vec iv;
435179 struct bvec_iter iter;
436180
437− virt = bio->bi_integrity->bip_iter.bi_sector & 0xffffffff;
181+ virt = bip_get_seed(bip) & 0xffffffff;
438182
439− bip_for_each_vec(iv, bio->bi_integrity, iter) {
440− sdt = kmap_atomic(iv.bv_page)
441− + iv.bv_offset;
183+ bip_for_each_vec(iv, bip, iter) {
184+ pi = kmap_atomic(iv.bv_page) + iv.bv_offset;
442185
443− for (j = 0; j < iv.bv_len; j += tuple_sz, sdt++) {
186+ for (j = 0; j < iv.bv_len; j += tuple_sz, pi++) {
444187
445− if (sectors == 0) {
446− kunmap_atomic(sdt);
188+ if (intervals == 0) {
189+ kunmap_atomic(pi);
447190 return;
448191 }
449192
450− if (be32_to_cpu(sdt->ref_tag) == phys)
451− sdt->ref_tag = cpu_to_be32(virt);
193+ if (be32_to_cpu(pi->ref_tag) == phys)
194+ pi->ref_tag = cpu_to_be32(virt);
452195
453196 virt++;
454197 phys++;
455− sectors--;
198+ intervals--;
456199 }
457200
458− kunmap_atomic(sdt);
201+ kunmap_atomic(pi);
459202 }
460203 }
461204 }
drivers/scsi/sg.c+2−2
@@ -1711,9 +1711,9 @@ sg_start_req(Sg_request *srp, unsigned char *cmd) · 1710 unmodified lines
17111711 }
17121712
17131713 rq = blk_get_request(q, rw, GFP_ATOMIC);
1714− if (!rq) {
1714+ if (IS_ERR(rq)) {
17151715 kfree(long_cmdp);
1716− return -ENOMEM;
1716+ return PTR_ERR(rq);
17171717 }
17181718
17191719 blk_rq_set_block_pc(rq);
drivers/scsi/st.c+1−1
@@ -490,7 +490,7 @@ static int st_scsi_execute(struct st_request *SRpnt, const unsigned char *cmd, · 489 unmodified lines
490490
491491 req = blk_get_request(SRpnt->stp->device->request_queue, write,
492492 GFP_KERNEL);
493− if (!req)
493+ if (IS_ERR(req))
494494 return DRIVER_ERROR << 24;
495495
496496 blk_rq_set_block_pc(req);
drivers/target/target_core_pscsi.c+1−1
@@ -1050,7 +1050,7 @@ pscsi_execute_cmd(struct se_cmd *cmd) · 1049 unmodified lines
10501050 req = blk_get_request(pdv->pdv_sd->request_queue,
10511051 (data_direction == DMA_TO_DEVICE),
10521052 GFP_KERNEL);
1053− if (!req) {
1053+ if (IS_ERR(req)) {
10541054 pr_err("PSCSI: blk_get_request() failed\n");
10551055 ret = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
10561056 goto fail;
fs/block_dev.c+11−23
@@ -50,32 +50,22 @@ inline struct block_device *I_BDEV(struct inode *inode) · 49 unmodified lines
5050 EXPORT_SYMBOL(I_BDEV);
5151
5252 /*
53− * Move the inode from its current bdi to a new bdi. If the inode is dirty we
54− * need to move it onto the dirty list of @dst so that the inode is always on
55− * the right list.
53+ * Move the inode from its current bdi to a new bdi. Make sure the inode
54+ * is clean before moving so that it doesn't linger on the old bdi.
5655 */
5756 static void bdev_inode_switch_bdi(struct inode *inode,
5857 struct backing_dev_info *dst)
5958 {
60− struct backing_dev_info *old = inode->i_data.backing_dev_info;
61− bool wakeup_bdi = false;
62−
63− if (unlikely(dst == old)) /* deadlock avoidance */
64− return;
65− bdi_lock_two(&old->wb, &dst->wb);
66− spin_lock(&inode->i_lock);
67− inode->i_data.backing_dev_info = dst;
68− if (inode->i_state & I_DIRTY) {
69− if (bdi_cap_writeback_dirty(dst) && !wb_has_dirty_io(&dst->wb))
70− wakeup_bdi = true;
71− list_move(&inode->i_wb_list, &dst->wb.b_dirty);
59+ while (true) {
60+ spin_lock(&inode->i_lock);
61+ if (!(inode->i_state & I_DIRTY)) {
62+ inode->i_data.backing_dev_info = dst;
63+ spin_unlock(&inode->i_lock);
64+ return;
65+ }
66+ spin_unlock(&inode->i_lock);
67+ WARN_ON_ONCE(write_inode_now(inode, true));
7268 }
73− spin_unlock(&inode->i_lock);
74− spin_unlock(&old->wb.list_lock);
75− spin_unlock(&dst->wb.list_lock);
76−
77− if (wakeup_bdi)
78− bdi_wakeup_thread_delayed(dst);
7969 }
8070
8171 /* Kill _all_ buffers and pagecache , dirty or not.. */
@@ -1179,8 +1169,6 @@ static int __blkdev_get(struct block_device *bdev, fmode_t mode, int for_part) · 1097 unmodified lines
11791169 if (!ret) {
11801170 bd_set_size(bdev,(loff_t)get_capacity(disk)<<9);
11811171 bdi = blk_get_backing_dev_info(bdev);
1182− if (bdi == NULL)
1183− bdi = &default_backing_dev_info;
11841172 bdev_inode_switch_bdi(bdev->bd_inode, bdi);
11851173 }
11861174
fs/btrfs/disk-io.c+1−1
@@ -1702,7 +1702,7 @@ static int btrfs_congested_fn(void *congested_data, int bdi_bits) · 1701 unmodified lines
17021702 if (!device->bdev)
17031703 continue;
17041704 bdi = blk_get_backing_dev_info(device->bdev);
1705− if (bdi && bdi_congested(bdi, bdi_bits)) {
1705+ if (bdi_congested(bdi, bdi_bits)) {
17061706 ret = 1;
17071707 break;
17081708 }
fs/nfs/direct.c+5−7
@@ -220,11 +220,9 @@ ssize_t nfs_direct_IO(int rw, struct kiocb *iocb, struct iov_iter *iter, loff_t · 219 unmodified lines
220220 #else
221221 VM_BUG_ON(iocb->ki_nbytes != PAGE_SIZE);
222222
223− if (rw == READ || rw == KERNEL_READ)
224− return nfs_file_direct_read(iocb, iter, pos,
225− rw == READ ? true : false);
226− return nfs_file_direct_write(iocb, iter, pos,
227− rw == WRITE ? true : false);
223+ if (rw == READ)
224+ return nfs_file_direct_read(iocb, iter, pos);
225+ return nfs_file_direct_write(iocb, iter, pos);
228226 #endif /* CONFIG_NFS_SWAP */
229227 }
230228
@@ -510,7 +508,7 @@ static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq, · 279 unmodified lines
510508 * cache.
511509 */
512510 ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter,
513− loff_t pos, bool uio)
511+ loff_t pos)
514512 {
515513 struct file *file = iocb->ki_filp;
516514 struct address_space *mapping = file->f_mapping;
@@ -879,7 +877,7 @@ static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq, · 362 unmodified lines
879877 * is no atomic O_APPEND write facility in the NFS protocol.
880878 */
881879 ssize_t nfs_file_direct_write(struct kiocb *iocb, struct iov_iter *iter,
882− loff_t pos, bool uio)
880+ loff_t pos)
883881 {
884882 ssize_t result = -EINVAL;
885883 struct file *file = iocb->ki_filp;
fs/nfs/file.c+2−2
@@ -172,7 +172,7 @@ nfs_file_read(struct kiocb *iocb, struct iov_iter *to) · 171 unmodified lines
172172 ssize_t result;
173173
174174 if (iocb->ki_filp->f_flags & O_DIRECT)
175− return nfs_file_direct_read(iocb, to, iocb->ki_pos, true);
175+ return nfs_file_direct_read(iocb, to, iocb->ki_pos);
176176
177177 dprintk("NFS: read(%pD2, %zu@%lu)\n",
178178 iocb->ki_filp,
@@ -676,7 +676,7 @@ ssize_t nfs_file_write(struct kiocb *iocb, struct iov_iter *from) · 497 unmodified lines
676676 return result;
677677
678678 if (file->f_flags & O_DIRECT)
679− return nfs_file_direct_write(iocb, from, pos, true);
679+ return nfs_file_direct_write(iocb, from, pos);
680680
681681 dprintk("NFS: write(%pD2, %zu@%Ld)\n",
682682 file, count, (long long) pos);
fs/xfs/xfs_buf.c+0−2
@@ -1670,8 +1670,6 @@ xfs_alloc_buftarg( · 1669 unmodified lines
16701670 btp->bt_dev = bdev->bd_dev;
16711671 btp->bt_bdev = bdev;
16721672 btp->bt_bdi = blk_get_backing_dev_info(bdev);
1673− if (!btp->bt_bdi)
1674− goto error;
16751673
16761674 if (xfs_setsize_buftarg_early(btp, bdev))
16771675 goto error;
include/linux/backing-dev.h+0−4
@@ -28,12 +28,10 @@ struct dentry; · 27 unmodified lines
2828 * Bits in backing_dev_info.state
2929 */
3030 enum bdi_state {
31− BDI_wb_alloc, /* Default embedded wb allocated */
3231 BDI_async_congested, /* The async (write) queue is getting full */
3332 BDI_sync_congested, /* The sync queue is getting full */
3433 BDI_registered, /* bdi_register() was done */
3534 BDI_writeback_running, /* Writeback is in progress */
36− BDI_unused, /* Available bits start here */
3735 };
3836
3937 typedef int (congested_fn)(void *, int);
@@ -50,7 +48,6 @@ enum bdi_stat_item { · 10 unmodified lines
5048
5149 struct bdi_writeback {
5250 struct backing_dev_info *bdi; /* our parent bdi */
53− unsigned int nr;
5451
5552 unsigned long last_old_flush; /* last old data flush */
5653
@@ -124,7 +121,6 @@ void bdi_start_background_writeback(struct backing_dev_info *bdi); · 67 unmodified lines
124121 void bdi_writeback_workfn(struct work_struct *work);
125122 int bdi_has_dirty_io(struct backing_dev_info *bdi);
126123 void bdi_wakeup_thread_delayed(struct backing_dev_info *bdi);
127−void bdi_lock_two(struct bdi_writeback *wb1, struct bdi_writeback *wb2);
128124
129125 extern spinlock_t bdi_lock;
130126 extern struct list_head bdi_list;
include/linux/bio.h+51−14
@@ -292,7 +292,24 @@ static inline unsigned bio_segments(struct bio *bio) · 291 unmodified lines
292292 */
293293 #define bio_get(bio) atomic_inc(&(bio)->bi_cnt)
294294
295+enum bip_flags {
296+ BIP_BLOCK_INTEGRITY = 1 << 0, /* block layer owns integrity data */
297+ BIP_MAPPED_INTEGRITY = 1 << 1, /* ref tag has been remapped */
298+ BIP_CTRL_NOCHECK = 1 << 2, /* disable HBA integrity checking */
299+ BIP_DISK_NOCHECK = 1 << 3, /* disable disk integrity checking */
300+ BIP_IP_CHECKSUM = 1 << 4, /* IP checksum */
301+};
302+
295303 #if defined(CONFIG_BLK_DEV_INTEGRITY)
304+
305+static inline struct bio_integrity_payload *bio_integrity(struct bio *bio)
306+{
307+ if (bio->bi_rw & REQ_INTEGRITY)
308+ return bio->bi_integrity;
309+
310+ return NULL;
311+}
312+
296313 /*
297314 * bio integrity payload
298315 */
@@ -301,21 +318,40 @@ struct bio_integrity_payload { · 2 unmodified lines
301318
302319 struct bvec_iter bip_iter;
303320
304− /* kill - should just use bip_vec */
305− void *bip_buf; /* generated integrity data */
306−
307321 bio_end_io_t *bip_end_io; /* saved I/O completion fn */
308322
309323 unsigned short bip_slab; /* slab the bip came from */
310324 unsigned short bip_vcnt; /* # of integrity bio_vecs */
311325 unsigned short bip_max_vcnt; /* integrity bio_vec slots */
312− unsigned bip_owns_buf:1; /* should free bip_buf */
326+ unsigned short bip_flags; /* control flags */
313327
314328 struct work_struct bip_work; /* I/O completion */
315329
316330 struct bio_vec *bip_vec;
317331 struct bio_vec bip_inline_vecs[0];/* embedded bvec array */
318332 };
333+
334+static inline bool bio_integrity_flagged(struct bio *bio, enum bip_flags flag)
335+{
336+ struct bio_integrity_payload *bip = bio_integrity(bio);
337+
338+ if (bip)
339+ return bip->bip_flags & flag;
340+
341+ return false;
342+}
343+
344+static inline sector_t bip_get_seed(struct bio_integrity_payload *bip)
345+{
346+ return bip->bip_iter.bi_sector;
347+}
348+
349+static inline void bip_set_seed(struct bio_integrity_payload *bip,
350+ sector_t seed)
351+{
352+ bip->bip_iter.bi_sector = seed;
353+}
354+
319355 #endif /* CONFIG_BLK_DEV_INTEGRITY */
320356
321357 extern void bio_trim(struct bio *bio, int offset, int size);
@@ -342,6 +378,7 @@ static inline struct bio *bio_next_split(struct bio *bio, int sectors, · 20 unmodified lines
342378 }
343379
344380 extern struct bio_set *bioset_create(unsigned int, unsigned int);
381+extern struct bio_set *bioset_create_nobvec(unsigned int, unsigned int);
345382 extern void bioset_free(struct bio_set *);
346383 extern mempool_t *biovec_create_pool(int pool_entries);
347384
@@ -353,7 +390,6 @@ extern struct bio *bio_clone_fast(struct bio *, gfp_t, struct bio_set *); · 5 unmodified lines
353390 extern struct bio *bio_clone_bioset(struct bio *, gfp_t, struct bio_set *bs);
354391
355392 extern struct bio_set *fs_bio_set;
356−unsigned int bio_integrity_tag_size(struct bio *bio);
357393
358394 static inline struct bio *bio_alloc(gfp_t gfp_mask, unsigned int nr_iovecs)
359395 {
@@ -661,14 +697,10 @@ struct biovec_slab { · 301 unmodified lines
661697 for_each_bio(_bio) \
662698 bip_for_each_vec(_bvl, _bio->bi_integrity, _iter)
663699
664−#define bio_integrity(bio) (bio->bi_integrity != NULL)
665−
666700 extern struct bio_integrity_payload *bio_integrity_alloc(struct bio *, gfp_t, unsigned int);
667701 extern void bio_integrity_free(struct bio *);
668702 extern int bio_integrity_add_page(struct bio *, struct page *, unsigned int, unsigned int);
669−extern int bio_integrity_enabled(struct bio *bio);
670−extern int bio_integrity_set_tag(struct bio *, void *, unsigned int);
671−extern int bio_integrity_get_tag(struct bio *, void *, unsigned int);
703+extern bool bio_integrity_enabled(struct bio *bio);
672704 extern int bio_integrity_prep(struct bio *);
673705 extern void bio_integrity_endio(struct bio *, int);
674706 extern void bio_integrity_advance(struct bio *, unsigned int);
@@ -680,14 +712,14 @@ extern void bio_integrity_init(void); · 5 unmodified lines
680712
681713 #else /* CONFIG_BLK_DEV_INTEGRITY */
682714
683−static inline int bio_integrity(struct bio *bio)
715+static inline void *bio_integrity(struct bio *bio)
684716 {
685− return 0;
717+ return NULL;
686718 }
687719
688−static inline int bio_integrity_enabled(struct bio *bio)
720+static inline bool bio_integrity_enabled(struct bio *bio)
689721 {
690− return 0;
722+ return false;
691723 }
692724
693725 static inline int bioset_integrity_create(struct bio_set *bs, int pool_size)
@@ -733,6 +765,11 @@ static inline void bio_integrity_init(void) · 39 unmodified lines
733765 return;
734766 }
735767
768+static inline bool bio_integrity_flagged(struct bio *bio, enum bip_flags flag)
769+{
770+ return false;
771+}
772+
736773 #endif /* CONFIG_BLK_DEV_INTEGRITY */
737774
738775 #endif /* CONFIG_BLOCK */
include/linux/blk-mq.h+17−5
@@ -4,6 +4,7 @@ · 3 unmodified lines
44 #include <linux/blkdev.h>
55
66 struct blk_mq_tags;
7+struct blk_flush_queue;
78
89 struct blk_mq_cpu_notifier {
910 struct list_head list;
@@ -34,6 +35,7 @@ struct blk_mq_hw_ctx { · 24 unmodified lines
3435
3536 struct request_queue *queue;
3637 unsigned int queue_num;
38+ struct blk_flush_queue *fq;
3739
3840 void *driver_data;
3941
@@ -77,8 +79,9 @@ struct blk_mq_tag_set { · 37 unmodified lines
7779 struct list_head tag_list;
7880 };
7981
80−typedef int (queue_rq_fn)(struct blk_mq_hw_ctx *, struct request *);
82+typedef int (queue_rq_fn)(struct blk_mq_hw_ctx *, struct request *, bool);
8183 typedef struct blk_mq_hw_ctx *(map_queue_fn)(struct request_queue *, const int);
84+typedef enum blk_eh_timer_return (timeout_fn)(struct request *, bool);
8285 typedef int (init_hctx_fn)(struct blk_mq_hw_ctx *, void *, unsigned int);
8386 typedef void (exit_hctx_fn)(struct blk_mq_hw_ctx *, unsigned int);
8487 typedef int (init_request_fn)(void *, struct request *, unsigned int,
@@ -86,6 +89,9 @@ typedef int (init_request_fn)(void *, struct request *, unsigned int, · 1 unmodified line
8689 typedef void (exit_request_fn)(void *, struct request *, unsigned int,
8790 unsigned int);
8891
92+typedef void (busy_iter_fn)(struct blk_mq_hw_ctx *, struct request *, void *,
93+ bool);
94+
8995 struct blk_mq_ops {
9096 /*
9197 * Queue request
@@ -100,7 +106,7 @@ struct blk_mq_ops { · 8 unmodified lines
100106 /*
101107 * Called on request timeout
102108 */
103− rq_timed_out_fn *timeout;
109+ timeout_fn *timeout;
104110
105111 softirq_done_fn *complete;
106112
@@ -115,6 +121,10 @@ struct blk_mq_ops { · 8 unmodified lines
115121 /*
116122 * Called for every command allocated by the block layer to allow
117123 * the driver to set up driver specific data.
124+ *
125+ * Tag greater than or equal to queue_depth is for setting up
126+ * flush request.
127+ *
118128 * Ditto for exit/teardown.
119129 */
120130 init_request_fn *init_request;
@@ -160,8 +170,9 @@ struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag); · 39 unmodified lines
160170 struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *, const int ctx_index);
161171 struct blk_mq_hw_ctx *blk_mq_alloc_single_hw_queue(struct blk_mq_tag_set *, unsigned int, int);
162172
163−void blk_mq_end_io(struct request *rq, int error);
164−void __blk_mq_end_io(struct request *rq, int error);
173+void blk_mq_start_request(struct request *rq);
174+void blk_mq_end_request(struct request *rq, int error);
175+void __blk_mq_end_request(struct request *rq, int error);
165176
166177 void blk_mq_requeue_request(struct request *rq);
167178 void blk_mq_add_to_requeue_list(struct request *rq, bool at_head);
@@ -174,7 +185,8 @@ void blk_mq_stop_hw_queues(struct request_queue *q); · 6 unmodified lines
174185 void blk_mq_start_hw_queues(struct request_queue *q);
175186 void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async);
176187 void blk_mq_delay_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs);
177−void blk_mq_tag_busy_iter(struct blk_mq_tags *tags, void (*fn)(void *data, unsigned long *), void *data);
188+void blk_mq_tag_busy_iter(struct blk_mq_hw_ctx *hctx, busy_iter_fn *fn,
189+ void *priv);
178190
179191 /*
180192 * Driver command data is immediately after the request. So subtract request
include/linux/blk_types.h+8−10
@@ -78,9 +78,11 @@ struct bio { · 77 unmodified lines
7878 struct io_context *bi_ioc;
7979 struct cgroup_subsys_state *bi_css;
8080 #endif
81+ union {
8182 #if defined(CONFIG_BLK_DEV_INTEGRITY)
82− struct bio_integrity_payload *bi_integrity; /* data integrity */
83+ struct bio_integrity_payload *bi_integrity; /* data integrity */
8384 #endif
85+ };
8486
8587 unsigned short bi_vcnt; /* how many bio_vec's */
8688
@@ -118,10 +120,8 @@ struct bio { · 31 unmodified lines
118120 #define BIO_USER_MAPPED 6 /* contains user pages */
119121 #define BIO_EOPNOTSUPP 7 /* not supported */
120122 #define BIO_NULL_MAPPED 8 /* contains invalid user pages */
121−#define BIO_FS_INTEGRITY 9 /* fs owns integrity data, not block layer */
122−#define BIO_QUIET 10 /* Make BIO Quiet */
123−#define BIO_MAPPED_INTEGRITY 11/* integrity metadata has been remapped */
124−#define BIO_SNAP_STABLE 12 /* bio data must be snapshotted during write */
123+#define BIO_QUIET 9 /* Make BIO Quiet */
124+#define BIO_SNAP_STABLE 10 /* bio data must be snapshotted during write */
125125
126126 /*
127127 * Flags starting here get preserved by bio_reset() - this includes
@@ -162,6 +162,7 @@ enum rq_flag_bits { · 34 unmodified lines
162162 __REQ_WRITE_SAME, /* write same block many times */
163163
164164 __REQ_NOIDLE, /* don't anticipate more IO after this one */
165+ __REQ_INTEGRITY, /* I/O includes block integrity payload */
165166 __REQ_FUA, /* forced unit access */
166167 __REQ_FLUSH, /* request for cache flush */
167168
@@ -186,9 +187,7 @@ enum rq_flag_bits { · 18 unmodified lines
186187 __REQ_FLUSH_SEQ, /* request for flush sequence */
187188 __REQ_IO_STAT, /* account I/O stat */
188189 __REQ_MIXED_MERGE, /* merge of different types, fail separately */
189− __REQ_KERNEL, /* direct IO to kernel pages */
190190 __REQ_PM, /* runtime pm request */
191− __REQ_END, /* last of chain of requests */
192191 __REQ_HASHED, /* on IO scheduler merge hash */
193192 __REQ_MQ_INFLIGHT, /* track inflight for MQ */
194193 __REQ_NR_BITS, /* stops here */
@@ -204,13 +203,14 @@ enum rq_flag_bits { · 9 unmodified lines
204203 #define REQ_DISCARD (1ULL << __REQ_DISCARD)
205204 #define REQ_WRITE_SAME (1ULL << __REQ_WRITE_SAME)
206205 #define REQ_NOIDLE (1ULL << __REQ_NOIDLE)
206+#define REQ_INTEGRITY (1ULL << __REQ_INTEGRITY)
207207
208208 #define REQ_FAILFAST_MASK \
209209 (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER)
210210 #define REQ_COMMON_MASK \
211211 (REQ_WRITE | REQ_FAILFAST_MASK | REQ_SYNC | REQ_META | REQ_PRIO | \
212212 REQ_DISCARD | REQ_WRITE_SAME | REQ_NOIDLE | REQ_FLUSH | REQ_FUA | \
213− REQ_SECURE)
213+ REQ_SECURE | REQ_INTEGRITY)
214214 #define REQ_CLONE_MASK REQ_COMMON_MASK
215215
216216 #define BIO_NO_ADVANCE_ITER_MASK (REQ_DISCARD|REQ_WRITE_SAME)
@@ -240,9 +240,7 @@ enum rq_flag_bits { · 23 unmodified lines
240240 #define REQ_IO_STAT (1ULL << __REQ_IO_STAT)
241241 #define REQ_MIXED_MERGE (1ULL << __REQ_MIXED_MERGE)
242242 #define REQ_SECURE (1ULL << __REQ_SECURE)
243−#define REQ_KERNEL (1ULL << __REQ_KERNEL)
244243 #define REQ_PM (1ULL << __REQ_PM)
245−#define REQ_END (1ULL << __REQ_END)
246244 #define REQ_HASHED (1ULL << __REQ_HASHED)
247245 #define REQ_MQ_INFLIGHT (1ULL << __REQ_MQ_INFLIGHT)
248246
include/linux/blkdev.h+30−41
@@ -36,6 +36,7 @@ struct request; · 35 unmodified lines
3636 struct sg_io_hdr;
3737 struct bsg_job;
3838 struct blkcg_gq;
39+struct blk_flush_queue;
3940
4041 #define BLKDEV_MIN_RQ 4
4142 #define BLKDEV_MAX_RQ 128 /* Default maximum */
@@ -455,14 +456,7 @@ struct request_queue { · 413 unmodified lines
455456 */
456457 unsigned int flush_flags;
457458 unsigned int flush_not_queueable:1;
458− unsigned int flush_queue_delayed:1;
459− unsigned int flush_pending_idx:1;
460− unsigned int flush_running_idx:1;
461− unsigned long flush_pending_since;
462− struct list_head flush_queue[2];
463− struct list_head flush_data_in_flight;
464− struct request *flush_rq;
465− spinlock_t mq_flush_lock;
459+ struct blk_flush_queue *fq;
466460
467461 struct list_head requeue_list;
468462 spinlock_t requeue_lock;
@@ -865,7 +859,7 @@ extern void blk_execute_rq_nowait(struct request_queue *, struct gendisk *, · 396 unmodified lines
865859
866860 static inline struct request_queue *bdev_get_queue(struct block_device *bdev)
867861 {
868− return bdev->bd_disk->queue;
862+ return bdev->bd_disk->queue; /* this is never NULL */
869863 }
870864
871865 /*
@@ -1285,10 +1279,9 @@ static inline int queue_alignment_offset(struct request_queue *q) · 413 unmodified lines
12851279 static inline int queue_limit_alignment_offset(struct queue_limits *lim, sector_t sector)
12861280 {
12871281 unsigned int granularity = max(lim->physical_block_size, lim->io_min);
1288− unsigned int alignment = (sector << 9) & (granularity - 1);
1282+ unsigned int alignment = sector_div(sector, granularity >> 9) << 9;
12891283
1290− return (granularity + lim->alignment_offset - alignment)
1291− & (granularity - 1);
1284+ return (granularity + lim->alignment_offset - alignment) % granularity;
12921285 }
12931286
12941287 static inline int bdev_alignment_offset(struct block_device *bdev)
@@ -1464,32 +1457,31 @@ static inline uint64_t rq_io_start_time_ns(struct request *req) · 169 unmodified lines
14641457
14651458 #if defined(CONFIG_BLK_DEV_INTEGRITY)
14661459
1467−#define INTEGRITY_FLAG_READ 2 /* verify data integrity on read */
1468−#define INTEGRITY_FLAG_WRITE 4 /* generate data integrity on write */
1460+enum blk_integrity_flags {
1461+ BLK_INTEGRITY_VERIFY = 1 << 0,
1462+ BLK_INTEGRITY_GENERATE = 1 << 1,
1463+ BLK_INTEGRITY_DEVICE_CAPABLE = 1 << 2,
1464+ BLK_INTEGRITY_IP_CHECKSUM = 1 << 3,
1465+};
14691466
1470−struct blk_integrity_exchg {
1467+struct blk_integrity_iter {
14711468 void *prot_buf;
14721469 void *data_buf;
1473− sector_t sector;
1470+ sector_t seed;
14741471 unsigned int data_size;
1475− unsigned short sector_size;
1472+ unsigned short interval;
14761473 const char *disk_name;
14771474 };
14781475
1479−typedef void (integrity_gen_fn) (struct blk_integrity_exchg *);
1480−typedef int (integrity_vrfy_fn) (struct blk_integrity_exchg *);
1481−typedef void (integrity_set_tag_fn) (void *, void *, unsigned int);
1482−typedef void (integrity_get_tag_fn) (void *, void *, unsigned int);
1476+typedef int (integrity_processing_fn) (struct blk_integrity_iter *);
14831477
14841478 struct blk_integrity {
1485− integrity_gen_fn *generate_fn;
1486− integrity_vrfy_fn *verify_fn;
1487− integrity_set_tag_fn *set_tag_fn;
1488− integrity_get_tag_fn *get_tag_fn;
1479+ integrity_processing_fn *generate_fn;
1480+ integrity_processing_fn *verify_fn;
14891481
14901482 unsigned short flags;
14911483 unsigned short tuple_size;
1492− unsigned short sector_size;
1484+ unsigned short interval;
14931485 unsigned short tag_size;
14941486
14951487 const char *name;
@@ -1504,10 +1496,10 @@ extern int blk_integrity_compare(struct gendisk *, struct gendisk *); · 8 unmodified lines
15041496 extern int blk_rq_map_integrity_sg(struct request_queue *, struct bio *,
15051497 struct scatterlist *);
15061498 extern int blk_rq_count_integrity_sg(struct request_queue *, struct bio *);
1507−extern int blk_integrity_merge_rq(struct request_queue *, struct request *,
1508− struct request *);
1509−extern int blk_integrity_merge_bio(struct request_queue *, struct request *,
1510− struct bio *);
1499+extern bool blk_integrity_merge_rq(struct request_queue *, struct request *,
1500+ struct request *);
1501+extern bool blk_integrity_merge_bio(struct request_queue *, struct request *,
1502+ struct bio *);
15111503
15121504 static inline
15131505 struct blk_integrity *bdev_get_integrity(struct block_device *bdev)
@@ -1520,12 +1512,9 @@ static inline struct blk_integrity *blk_get_integrity(struct gendisk *disk) · 6 unmodified lines
15201512 return disk->integrity;
15211513 }
15221514
1523−static inline int blk_integrity_rq(struct request *rq)
1515+static inline bool blk_integrity_rq(struct request *rq)
15241516 {
1525− if (rq->bio == NULL)
1526− return 0;
1527−
1528− return bio_integrity(rq->bio);
1517+ return rq->cmd_flags & REQ_INTEGRITY;
15291518 }
15301519
15311520 static inline void blk_queue_max_integrity_segments(struct request_queue *q,
@@ -1590,15 +1579,15 @@ static inline unsigned short queue_max_integrity_segments(struct request_queue * · 58 unmodified lines
15901579 {
15911580 return 0;
15921581 }
1593−static inline int blk_integrity_merge_rq(struct request_queue *rq,
1594− struct request *r1,
1595− struct request *r2)
1582+static inline bool blk_integrity_merge_rq(struct request_queue *rq,
1583+ struct request *r1,
1584+ struct request *r2)
15961585 {
15971586 return 0;
15981587 }
1599−static inline int blk_integrity_merge_bio(struct request_queue *rq,
1600− struct request *r,
1601− struct bio *b)
1588+static inline bool blk_integrity_merge_bio(struct request_queue *rq,
1589+ struct request *r,
1590+ struct bio *b)
16021591 {
16031592 return 0;
16041593 }
include/linux/crc-t10dif.h+3−2
@@ -6,7 +6,8 @@ · 5 unmodified lines
66 #define CRC_T10DIF_DIGEST_SIZE 2
77 #define CRC_T10DIF_BLOCK_SIZE 1
88
9−__u16 crc_t10dif_generic(__u16 crc, const unsigned char *buffer, size_t len);
10−__u16 crc_t10dif(unsigned char const *, size_t);
9+extern __u16 crc_t10dif_generic(__u16 crc, const unsigned char *buffer,
10+ size_t len);
11+extern __u16 crc_t10dif(unsigned char const *, size_t);
1112
1213 #endif
include/linux/fs.h+0−2
@@ -192,8 +192,6 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset, · 191 unmodified lines
192192 #define READ 0
193193 #define WRITE RW_MASK
194194 #define READA RWA_MASK
195−#define KERNEL_READ (READ|REQ_KERNEL)
196−#define KERNEL_WRITE (WRITE|REQ_KERNEL)
197195
198196 #define READ_SYNC (READ | REQ_SYNC)
199197 #define WRITE_SYNC (WRITE | REQ_SYNC | REQ_NOIDLE)
include/linux/nfs_fs.h+2−2
@@ -448,10 +448,10 @@ static inline struct rpc_cred *nfs_file_cred(struct file *file) · 447 unmodified lines
448448 extern ssize_t nfs_direct_IO(int, struct kiocb *, struct iov_iter *, loff_t);
449449 extern ssize_t nfs_file_direct_read(struct kiocb *iocb,
450450 struct iov_iter *iter,
451− loff_t pos, bool uio);
451+ loff_t pos);
452452 extern ssize_t nfs_file_direct_write(struct kiocb *iocb,
453453 struct iov_iter *iter,
454− loff_t pos, bool uio);
454+ loff_t pos);
455455
456456 /*
457457 * linux/fs/nfs/dir.c
include/linux/t10-pi.h+22−0
@@ -0,0 +1,22 @@
1+#ifndef _LINUX_T10_PI_H
2+#define _LINUX_T10_PI_H
3+
4+#include <linux/types.h>
5+#include <linux/blkdev.h>
6+
7+/*
8+ * T10 Protection Information tuple.
9+ */
10+struct t10_pi_tuple {
11+ __be16 guard_tag; /* Checksum */
12+ __be16 app_tag; /* Opaque storage */
13+ __be32 ref_tag; /* Target LBA or indirect LBA */
14+};
15+
16+
17+extern struct blk_integrity t10_pi_type1_crc;
18+extern struct blk_integrity t10_pi_type1_ip;
19+extern struct blk_integrity t10_pi_type3_crc;
20+extern struct blk_integrity t10_pi_type3_ip;
21+
22+#endif
include/scsi/scsi_cmnd.h+26−10
@@ -10,9 +10,10 @@ · 9 unmodified lines
1010 #include <scsi/scsi_device.h>
1111
1212 struct Scsi_Host;
13−struct scsi_device;
1413 struct scsi_driver;
1514
15+#include <scsi/scsi_device.h>
16+
1617 /*
1718 * MAX_COMMAND_SIZE is:
1819 * The longest fixed-length SCSI CDB as per the SCSI standard.
@@ -81,6 +82,7 @@ struct scsi_cmnd { · 62 unmodified lines
8182
8283 unsigned char prot_op;
8384 unsigned char prot_type;
85+ unsigned char prot_flags;
8486
8587 unsigned short cmd_len;
8688 enum dma_data_direction sc_data_direction;
@@ -252,6 +254,14 @@ static inline unsigned char scsi_get_prot_op(struct scsi_cmnd *scmd) · 165 unmodified lines
252254 return scmd->prot_op;
253255 }
254256
257+enum scsi_prot_flags {
258+ SCSI_PROT_TRANSFER_PI = 1 << 0,
259+ SCSI_PROT_GUARD_CHECK = 1 << 1,
260+ SCSI_PROT_REF_CHECK = 1 << 2,
261+ SCSI_PROT_REF_INCREMENT = 1 << 3,
262+ SCSI_PROT_IP_CHECKSUM = 1 << 4,
263+};
264+
255265 /*
256266 * The controller usually does not know anything about the target it
257267 * is communicating with. However, when DIX is enabled the controller
@@ -280,6 +290,17 @@ static inline sector_t scsi_get_lba(struct scsi_cmnd *scmd) · 22 unmodified lines
280290 return blk_rq_pos(scmd->request);
281291 }
282292
293+static inline unsigned int scsi_prot_interval(struct scsi_cmnd *scmd)
294+{
295+ return scmd->device->sector_size;
296+}
297+
298+static inline u32 scsi_prot_ref_tag(struct scsi_cmnd *scmd)
299+{
300+ return blk_rq_pos(scmd->request) >>
301+ (ilog2(scsi_prot_interval(scmd)) - 9) & 0xffffffff;
302+}
303+
283304 static inline unsigned scsi_prot_sg_count(struct scsi_cmnd *cmd)
284305 {
285306 return cmd->prot_sdb ? cmd->prot_sdb->table.nents : 0;
@@ -316,17 +337,12 @@ static inline void set_driver_byte(struct scsi_cmnd *cmd, char status) · 30 unmodified lines
316337 static inline unsigned scsi_transfer_length(struct scsi_cmnd *scmd)
317338 {
318339 unsigned int xfer_len = scsi_out(scmd)->length;
319− unsigned int prot_op = scsi_get_prot_op(scmd);
320− unsigned int sector_size = scmd->device->sector_size;
340+ unsigned int prot_interval = scsi_prot_interval(scmd);
321341
322− switch (prot_op) {
323− case SCSI_PROT_NORMAL:
324− case SCSI_PROT_WRITE_STRIP:
325− case SCSI_PROT_READ_INSERT:
326− return xfer_len;
327− }
342+ if (scmd->prot_flags & SCSI_PROT_TRANSFER_PI)
343+ xfer_len += (xfer_len >> ilog2(prot_interval)) * 8;
328344
329− return xfer_len + (xfer_len >> ilog2(sector_size)) * 8;
345+ return xfer_len;
330346 }
331347
332348 #endif /* _SCSI_SCSI_CMND_H */
mm/backing-dev.c+16−24
@@ -40,7 +40,7 @@ LIST_HEAD(bdi_list); · 39 unmodified lines
4040 /* bdi_wq serves all asynchronous writeback tasks */
4141 struct workqueue_struct *bdi_wq;
4242
43−void bdi_lock_two(struct bdi_writeback *wb1, struct bdi_writeback *wb2)
43+static void bdi_lock_two(struct bdi_writeback *wb1, struct bdi_writeback *wb2)
4444 {
4545 if (wb1 < wb2) {
4646 spin_lock(&wb1->list_lock);
@@ -376,13 +376,7 @@ static void bdi_wb_shutdown(struct backing_dev_info *bdi) · 329 unmodified lines
376376 mod_delayed_work(bdi_wq, &bdi->wb.dwork, 0);
377377 flush_delayed_work(&bdi->wb.dwork);
378378 WARN_ON(!list_empty(&bdi->work_list));
379−
380− /*
381− * This shouldn't be necessary unless @bdi for some reason has
382− * unflushed dirty IO after work_list is drained. Do it anyway
383− * just in case.
384− */
385− cancel_delayed_work_sync(&bdi->wb.dwork);
379+ WARN_ON(delayed_work_pending(&bdi->wb.dwork));
386380 }
387381
388382 /*
@@ -402,21 +396,15 @@ static void bdi_prune_sb(struct backing_dev_info *bdi) · 13 unmodified lines
402396
403397 void bdi_unregister(struct backing_dev_info *bdi)
404398 {
405− struct device *dev = bdi->dev;
406−
407− if (dev) {
399+ if (bdi->dev) {
408400 bdi_set_min_ratio(bdi, 0);
409401 trace_writeback_bdi_unregister(bdi);
410402 bdi_prune_sb(bdi);
411403
412404 bdi_wb_shutdown(bdi);
413405 bdi_debug_unregister(bdi);
414−
415− spin_lock_bh(&bdi->wb_lock);
406+ device_unregister(bdi->dev);
416407 bdi->dev = NULL;
417− spin_unlock_bh(&bdi->wb_lock);
418−
419− device_unregister(dev);
420408 }
421409 }
422410 EXPORT_SYMBOL(bdi_unregister);
@@ -487,8 +475,17 @@ void bdi_destroy(struct backing_dev_info *bdi) · 64 unmodified lines
487475 int i;
488476
489477 /*
490− * Splice our entries to the default_backing_dev_info, if this
491− * bdi disappears
478+ * Splice our entries to the default_backing_dev_info. This
479+ * condition shouldn't happen. @wb must be empty at this point and
480+ * dirty inodes on it might cause other issues. This workaround is
481+ * added by ce5f8e779519 ("writeback: splice dirty inode entries to
482+ * default bdi on bdi_destroy()") without root-causing the issue.
483+ *
484+ * http://lkml.kernel.org/g/1253038617-30204-11-git-send-email-jens.axboe@oracle.com
485+ * http://thread.gmane.org/gmane.linux.file-systems/35341/focus=35350
486+ *
487+ * We should probably add WARN_ON() to find out whether it still
488+ * happens and track it down if so.
492489 */
493490 if (bdi_has_dirty_io(bdi)) {
494491 struct bdi_writeback *dst = &default_backing_dev_info.wb;
@@ -503,12 +500,7 @@ void bdi_destroy(struct backing_dev_info *bdi) · 8 unmodified lines
503500
504501 bdi_unregister(bdi);
505502
506− /*
507− * If bdi_unregister() had already been called earlier, the dwork
508− * could still be pending because bdi_prune_sb() can race with the
509− * bdi_wakeup_thread_delayed() calls from __mark_inode_dirty().
510− */
511− cancel_delayed_work_sync(&bdi->wb.dwork);
503+ WARN_ON(delayed_work_pending(&bdi->wb.dwork));
512504
513505 for (i = 0; i < NR_BDI_STAT_ITEMS; i++)
514506 percpu_counter_destroy(&bdi->bdi_stat[i]);

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t