History

Merge tag 'for-5.20-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux

Pull btrfs updates from David Sterba: "This brings some long awaited changes, the send protocol bump, otherwise lots of small improvements and fixes. The main core part is reworking bio handling, cleaning up the submission and endio and improving error handling. There are some changes outside of btrfs adding helpers or updating API, listed at the end of the changelog. Features: - sysfs: - export chunk size, in debug mode add tunable for setting its size - show zoned among features (was only in debug mode) - show commit stats (number, last/max/total duration) - send protocol updated to 2 - new commands: - ability write larger data chunks than 64K - send raw compressed extents (uses the encoded data ioctls), ie. no decompression on send side, no compression needed on receive side if supported - send 'otime' (inode creation time) among other timestamps - send file attributes (a.k.a file flags and xflags) - this is first version bump, backward compatibility on send and receive side is provided - there are still some known and wanted commands that will be implemented in the near future, another version bump will be needed, however we want to minimize that to avoid causing usability issues - print checksum type and implementation at mount time - don't print some messages at mount (mentioned as people asked about it), we want to print messages namely for new features so let's make some space for that - big metadata - this has been supported for a long time and is not a feature that's worth mentioning - skinny metadata - same reason, set by default by mkfs Performance improvements: - reduced amount of reserved metadata for delayed items - when inserted items can be batched into one leaf - when deleting batched directory index items - when deleting delayed items used for deletion - overall improved count of files/sec, decreased subvolume lock contention - metadata item access bounds checker micro-optimized, with a few percent of improved runtime for metadata-heavy operations - increase direct io limit for read to 256 sectors, improved throughput by 3x on sample workload Notable fixes: - raid56 - reduce parity writes, skip sectors of stripe when there are no data updates - restore reading from on-disk data instead of using stripe cache, this reduces chances to damage correct data due to RMW cycle - refuse to replay log with unknown incompat read-only feature bit set - zoned - fix page locking when COW fails in the middle of allocation - improved tracking of active zones, ZNS drives may limit the number and there are ENOSPC errors due to that limit and not actual lack of space - adjust maximum extent size for zone append so it does not cause late ENOSPC due to underreservation - mirror reading error messages show the mirror number - don't fallback to buffered IO for NOWAIT direct IO writes, we don't have the NOWAIT semantics for buffered io yet - send, fix sending link commands for existing file paths when there are deleted and created hardlinks for same files - repair all mirrors for profiles with more than 1 copy (raid1c34) - fix repair of compressed extents, unify where error detection and repair happen Core changes: - bio completion cleanups - don't double defer compression bios - simplify endio workqueues - add more data to btrfs_bio to avoid allocation for read requests - rework bio error handling so it's same what block layer does, the submission works and errors are consumed in endio - when asynchronous bio offload fails fall back to synchronous checksum calculation to avoid errors under writeback or memory pressure - new trace points - raid56 events - ordered extent operations - super block log_root_transid deprecated (never used) - mixed_backref and big_metadata sysfs feature files removed, they've been default for sufficiently long time, there are no known users and mixed_backref could be confused with mixed_groups Non-btrfs changes, API updates: - minor highmem API update to cover const arguments - switch all kmap/kmap_atomic to kmap_local - remove redundant flush_dcache_page() - address_space_operations::writepage callback removed - add bdev_max_segments() helper" * tag 'for-5.20-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux: (163 commits) btrfs: don't call btrfs_page_set_checked in finish_compressed_bio_read btrfs: fix repair of compressed extents btrfs: remove the start argument to check_data_csum and export btrfs: pass a btrfs_bio to btrfs_repair_one_sector btrfs: simplify the pending I/O counting in struct compressed_bio btrfs: repair all known bad mirrors btrfs: merge btrfs_dev_stat_print_on_error with its only caller btrfs: join running log transaction when logging new name btrfs: simplify error handling in btrfs_lookup_dentry btrfs: send: always use the rbtree based inode ref management infrastructure btrfs: send: fix sending link commands for existing file paths btrfs: send: introduce recorded_ref_alloc and recorded_ref_free btrfs: zoned: wait until zone is finished when allocation didn't progress btrfs: zoned: write out partially allocated region btrfs: zoned: activate necessary block group btrfs: zoned: activate metadata block group on flush_space btrfs: zoned: disable metadata overcommit for zoned btrfs: zoned: introduce space_info->active_total_bytes btrfs: zoned: finish least available block group on data bg allocation btrfs: let can_allocate_chunk return error ...

Linus Torvalds committed 4 years ago

Commit 353767e2 parents ab17c0c 0b078d957 files · +3842 −2829Browse files

Unified · Split

57 files
  1. arch/parisc/include/asm/cacheflush.h +3 −3
  2. arch/parisc/kernel/cache.c +1 −1
  3. fs/btrfs/async-thread.h +0 −1
  4. fs/btrfs/backref.c +49 −39
  5. fs/btrfs/backref.h +1 −2
  6. fs/btrfs/block-group.c +28 −6
  7. fs/btrfs/block-rsv.c +9 −12
  8. fs/btrfs/block-rsv.h +8 −7
  9. fs/btrfs/btrfs_inode.h +18 −7
  10. fs/btrfs/compression.c +102 −257
  11. fs/btrfs/compression.h +7 −11
  12. fs/btrfs/ctree.h +84 −21
  13. fs/btrfs/delalloc-space.c +3 −3
  14. fs/btrfs/delayed-inode.c +281 −114
  15. fs/btrfs/delayed-inode.h +11 −0
  16. fs/btrfs/delayed-ref.c +2 −2
  17. fs/btrfs/dev-replace.c +2 −1
  18. fs/btrfs/disk-io.c +88 −180
  19. fs/btrfs/disk-io.h +3 −14
  20. fs/btrfs/extent-tree.c +87 −62
  21. fs/btrfs/extent_io.c +392 −481
  22. fs/btrfs/extent_io.h +4 −11
  23. fs/btrfs/file.c +23 −6
  24. fs/btrfs/free-space-cache.c +2 −1
  25. fs/btrfs/inode.c +400 −364
  26. fs/btrfs/ioctl.c +79 −71
  27. fs/btrfs/lzo.c +12 −16
  28. fs/btrfs/ordered-data.c +29 −11
  29. fs/btrfs/ordered-data.h +3 −2
  30. fs/btrfs/raid56.c +371 −421
  31. fs/btrfs/raid56.h +151 −17
  32. fs/btrfs/reflink.c +12 −7
  33. fs/btrfs/scrub.c +26 −45
  34. fs/btrfs/send.c +533 −248
  35. fs/btrfs/send.h +103 −66
  36. fs/btrfs/space-info.c +99 −11
  37. fs/btrfs/space-info.h +7 −1
  38. fs/btrfs/struct-funcs.c +3 −8
  39. fs/btrfs/subpage.c +2 −2
  40. fs/btrfs/super.c +9 −27
  41. fs/btrfs/sysfs.c +177 −9
  42. fs/btrfs/tests/btrfs-tests.c +1 −0
  43. fs/btrfs/tests/extent-buffer-tests.c +2 −1
  44. fs/btrfs/transaction.c +24 −2
  45. fs/btrfs/tree-log.c +18 −11
  46. fs/btrfs/tree-log.h +3 −0
  47. fs/btrfs/volumes.c +172 −190
  48. fs/btrfs/volumes.h +43 −3
  49. fs/btrfs/zlib.c +18 −24
  50. fs/btrfs/zoned.c +128 −3
  51. fs/btrfs/zoned.h +18 −0
  52. fs/btrfs/zstd.c +13 −20
  53. include/linux/blkdev.h +5 −0
  54. include/linux/highmem-internal.h +5 −5
  55. include/trace/events/btrfs.h +158 −0
  56. include/uapi/linux/btrfs.h +9 −1
  57. mm/highmem.c +1 −1
arch/parisc/include/asm/cacheflush.h+3−3
@@ -22,7 +22,7 @@ void flush_kernel_icache_range_asm(unsigned long, unsigned long); · 21 unmodified lines
2222 void flush_user_dcache_range_asm(unsigned long, unsigned long);
2323 void flush_kernel_dcache_range_asm(unsigned long, unsigned long);
2424 void purge_kernel_dcache_range_asm(unsigned long, unsigned long);
25−void flush_kernel_dcache_page_asm(void *);
25+void flush_kernel_dcache_page_asm(const void *addr);
2626 void flush_kernel_icache_page(void *);
2727
2828 /* Cache flush operations */
@@ -31,7 +31,7 @@ void flush_cache_all_local(void); · 2 unmodified lines
3131 void flush_cache_all(void);
3232 void flush_cache_mm(struct mm_struct *mm);
3333
34−void flush_kernel_dcache_page_addr(void *addr);
34+void flush_kernel_dcache_page_addr(const void *addr);
3535
3636 #define flush_kernel_dcache_range(start,size) \
3737 flush_kernel_dcache_range_asm((start), (start)+(size));
@@ -75,7 +75,7 @@ void flush_dcache_page_asm(unsigned long phys_addr, unsigned long vaddr); · 37 unmodified lines
7575 void flush_anon_page(struct vm_area_struct *vma, struct page *page, unsigned long vmaddr);
7676
7777 #define ARCH_HAS_FLUSH_ON_KUNMAP
78−static inline void kunmap_flush_on_unmap(void *addr)
78+static inline void kunmap_flush_on_unmap(const void *addr)
7979 {
8080 flush_kernel_dcache_page_addr(addr);
8181 }
arch/parisc/kernel/cache.c+1−1
@@ -549,7 +549,7 @@ extern void purge_kernel_dcache_page_asm(unsigned long); · 548 unmodified lines
549549 extern void clear_user_page_asm(void *, unsigned long);
550550 extern void copy_user_page_asm(void *, void *, unsigned long);
551551
552−void flush_kernel_dcache_page_addr(void *addr)
552+void flush_kernel_dcache_page_addr(const void *addr)
553553 {
554554 unsigned long flags;
555555
fs/btrfs/async-thread.h+0−1
@@ -13,7 +13,6 @@ struct btrfs_fs_info; · 12 unmodified lines
1313 struct btrfs_workqueue;
1414 struct btrfs_work;
1515 typedef void (*btrfs_func_t)(struct btrfs_work *arg);
16−typedef void (*btrfs_work_func_t)(struct work_struct *arg);
1716
1817 struct btrfs_work {
1918 btrfs_func_t func;
fs/btrfs/backref.c+49−39
@@ -2028,10 +2028,29 @@ out: · 2027 unmodified lines
20282028 return ret;
20292029 }
20302030
2031+static int build_ino_list(u64 inum, u64 offset, u64 root, void *ctx)
2032+{
2033+ struct btrfs_data_container *inodes = ctx;
2034+ const size_t c = 3 * sizeof(u64);
2035+
2036+ if (inodes->bytes_left >= c) {
2037+ inodes->bytes_left -= c;
2038+ inodes->val[inodes->elem_cnt] = inum;
2039+ inodes->val[inodes->elem_cnt + 1] = offset;
2040+ inodes->val[inodes->elem_cnt + 2] = root;
2041+ inodes->elem_cnt += 3;
2042+ } else {
2043+ inodes->bytes_missing += c - inodes->bytes_left;
2044+ inodes->bytes_left = 0;
2045+ inodes->elem_missed += 3;
2046+ }
2047+
2048+ return 0;
2049+}
2050+
20312051 int iterate_inodes_from_logical(u64 logical, struct btrfs_fs_info *fs_info,
20322052 struct btrfs_path *path,
2033− iterate_extent_inodes_t *iterate, void *ctx,
2034− bool ignore_offset)
2053+ void *ctx, bool ignore_offset)
20352054 {
20362055 int ret;
20372056 u64 extent_item_pos;
@@ -2049,17 +2068,15 @@ int iterate_inodes_from_logical(u64 logical, struct btrfs_fs_info *fs_info, · 11 unmodified lines
20492068 extent_item_pos = logical - found_key.objectid;
20502069 ret = iterate_extent_inodes(fs_info, found_key.objectid,
20512070 extent_item_pos, search_commit_root,
2052− iterate, ctx, ignore_offset);
2071+ build_ino_list, ctx, ignore_offset);
20532072
20542073 return ret;
20552074 }
20562075
2057−typedef int (iterate_irefs_t)(u64 parent, u32 name_len, unsigned long name_off,
2058− struct extent_buffer *eb, void *ctx);
2076+static int inode_to_path(u64 inum, u32 name_len, unsigned long name_off,
2077+ struct extent_buffer *eb, struct inode_fs_paths *ipath);
20592078
2060−static int iterate_inode_refs(u64 inum, struct btrfs_root *fs_root,
2061− struct btrfs_path *path,
2062− iterate_irefs_t *iterate, void *ctx)
2079+static int iterate_inode_refs(u64 inum, struct inode_fs_paths *ipath)
20632080 {
20642081 int ret = 0;
20652082 int slot;
@@ -2068,6 +2085,8 @@ static int iterate_inode_refs(u64 inum, struct btrfs_root *fs_root, · 2 unmodified lines
20682085 u32 name_len;
20692086 u64 parent = 0;
20702087 int found = 0;
2088+ struct btrfs_root *fs_root = ipath->fs_root;
2089+ struct btrfs_path *path = ipath->btrfs_path;
20712090 struct extent_buffer *eb;
20722091 struct btrfs_inode_ref *iref;
20732092 struct btrfs_key found_key;
@@ -2103,8 +2122,8 @@ static int iterate_inode_refs(u64 inum, struct btrfs_root *fs_root, · 29 unmodified lines
21032122 "following ref at offset %u for inode %llu in tree %llu",
21042123 cur, found_key.objectid,
21052124 fs_root->root_key.objectid);
2106− ret = iterate(parent, name_len,
2107− (unsigned long)(iref + 1), eb, ctx);
2125+ ret = inode_to_path(parent, name_len,
2126+ (unsigned long)(iref + 1), eb, ipath);
21082127 if (ret)
21092128 break;
21102129 len = sizeof(*iref) + name_len;
@@ -2118,15 +2137,15 @@ static int iterate_inode_refs(u64 inum, struct btrfs_root *fs_root, · 7 unmodified lines
21182137 return ret;
21192138 }
21202139
2121−static int iterate_inode_extrefs(u64 inum, struct btrfs_root *fs_root,
2122− struct btrfs_path *path,
2123− iterate_irefs_t *iterate, void *ctx)
2140+static int iterate_inode_extrefs(u64 inum, struct inode_fs_paths *ipath)
21242141 {
21252142 int ret;
21262143 int slot;
21272144 u64 offset = 0;
21282145 u64 parent;
21292146 int found = 0;
2147+ struct btrfs_root *fs_root = ipath->fs_root;
2148+ struct btrfs_path *path = ipath->btrfs_path;
21302149 struct extent_buffer *eb;
21312150 struct btrfs_inode_extref *extref;
21322151 u32 item_size;
@@ -2162,8 +2181,8 @@ static int iterate_inode_extrefs(u64 inum, struct btrfs_root *fs_root, · 29 unmodified lines
21622181 extref = (struct btrfs_inode_extref *)(ptr + cur_offset);
21632182 parent = btrfs_inode_extref_parent(eb, extref);
21642183 name_len = btrfs_inode_extref_name_len(eb, extref);
2165− ret = iterate(parent, name_len,
2166− (unsigned long)&extref->name, eb, ctx);
2184+ ret = inode_to_path(parent, name_len,
2185+ (unsigned long)&extref->name, eb, ipath);
21672186 if (ret)
21682187 break;
21692188
@@ -2180,34 +2199,13 @@ static int iterate_inode_extrefs(u64 inum, struct btrfs_root *fs_root, · 10 unmodified lines
21802199 return ret;
21812200 }
21822201
2183−static int iterate_irefs(u64 inum, struct btrfs_root *fs_root,
2184− struct btrfs_path *path, iterate_irefs_t *iterate,
2185− void *ctx)
2186−{
2187− int ret;
2188− int found_refs = 0;
2189−
2190− ret = iterate_inode_refs(inum, fs_root, path, iterate, ctx);
2191− if (!ret)
2192− ++found_refs;
2193− else if (ret != -ENOENT)
2194− return ret;
2195−
2196− ret = iterate_inode_extrefs(inum, fs_root, path, iterate, ctx);
2197− if (ret == -ENOENT && found_refs)
2198− return 0;
2199−
2200− return ret;
2201−}
2202−
22032202 /*
22042203 * returns 0 if the path could be dumped (probably truncated)
22052204 * returns <0 in case of an error
22062205 */
22072206 static int inode_to_path(u64 inum, u32 name_len, unsigned long name_off,
2208− struct extent_buffer *eb, void *ctx)
2207+ struct extent_buffer *eb, struct inode_fs_paths *ipath)
22092208 {
2210− struct inode_fs_paths *ipath = ctx;
22112209 char *fspath;
22122210 char *fspath_min;
22132211 int i = ipath->fspath->elem_cnt;
@@ -2248,8 +2246,20 @@ static int inode_to_path(u64 inum, u32 name_len, unsigned long name_off, · 34 unmodified lines
22482246 */
22492247 int paths_from_inode(u64 inum, struct inode_fs_paths *ipath)
22502248 {
2251− return iterate_irefs(inum, ipath->fs_root, ipath->btrfs_path,
2252− inode_to_path, ipath);
2249+ int ret;
2250+ int found_refs = 0;
2251+
2252+ ret = iterate_inode_refs(inum, ipath);
2253+ if (!ret)
2254+ ++found_refs;
2255+ else if (ret != -ENOENT)
2256+ return ret;
2257+
2258+ ret = iterate_inode_extrefs(inum, ipath);
2259+ if (ret == -ENOENT && found_refs)
2260+ return 0;
2261+
2262+ return ret;
22532263 }
22542264
22552265 struct btrfs_data_container *init_data_container(u32 total_bytes)
fs/btrfs/backref.h+1−2
@@ -35,8 +35,7 @@ int iterate_extent_inodes(struct btrfs_fs_info *fs_info, · 34 unmodified lines
3535 bool ignore_offset);
3636
3737 int iterate_inodes_from_logical(u64 logical, struct btrfs_fs_info *fs_info,
38− struct btrfs_path *path,
39− iterate_extent_inodes_t *iterate, void *ctx,
38+ struct btrfs_path *path, void *ctx,
4039 bool ignore_offset);
4140
4241 int paths_from_inode(u64 inum, struct inode_fs_paths *ipath);
fs/btrfs/block-group.c+28−6
@@ -1051,8 +1051,13 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans, · 1050 unmodified lines
10511051 < block_group->zone_unusable);
10521052 WARN_ON(block_group->space_info->disk_total
10531053 < block_group->length * factor);
1054+ WARN_ON(block_group->zone_is_active &&
1055+ block_group->space_info->active_total_bytes
1056+ < block_group->length);
10541057 }
10551058 block_group->space_info->total_bytes -= block_group->length;
1059+ if (block_group->zone_is_active)
1060+ block_group->space_info->active_total_bytes -= block_group->length;
10561061 block_group->space_info->bytes_readonly -=
10571062 (block_group->length - block_group->zone_unusable);
10581063 block_group->space_info->bytes_zone_unusable -=
@@ -1816,11 +1821,10 @@ int btrfs_rmap_block(struct btrfs_fs_info *fs_info, u64 chunk_start, · 757 unmodified lines
18161821 stripe_nr = physical - map->stripes[i].physical;
18171822 stripe_nr = div64_u64_rem(stripe_nr, map->stripe_len, &offset);
18181823
1819− if (map->type & BTRFS_BLOCK_GROUP_RAID10) {
1824+ if (map->type & (BTRFS_BLOCK_GROUP_RAID0 |
1825+ BTRFS_BLOCK_GROUP_RAID10)) {
18201826 stripe_nr = stripe_nr * map->num_stripes + i;
18211827 stripe_nr = div_u64(stripe_nr, map->sub_stripes);
1822− } else if (map->type & BTRFS_BLOCK_GROUP_RAID0) {
1823− stripe_nr = stripe_nr * map->num_stripes + i;
18241828 }
18251829 /*
18261830 * The remaining case would be for RAID56, multiply by
@@ -2108,7 +2112,8 @@ static int read_one_block_group(struct btrfs_fs_info *info, · 281 unmodified lines
21082112 trace_btrfs_add_block_group(info, cache, 0);
21092113 btrfs_update_space_info(info, cache->flags, cache->length,
21102114 cache->used, cache->bytes_super,
2111− cache->zone_unusable, &space_info);
2115+ cache->zone_unusable, cache->zone_is_active,
2116+ &space_info);
21122117
21132118 cache->space_info = space_info;
21142119
@@ -2178,7 +2183,7 @@ static int fill_dummy_bgs(struct btrfs_fs_info *fs_info) · 63 unmodified lines
21782183 }
21792184
21802185 btrfs_update_space_info(fs_info, bg->flags, em->len, em->len,
2181− 0, 0, &space_info);
2186+ 0, 0, false, &space_info);
21822187 bg->space_info = space_info;
21832188 link_block_group(bg);
21842189
@@ -2559,7 +2564,7 @@ struct btrfs_block_group *btrfs_make_block_group(struct btrfs_trans_handle *tran · 374 unmodified lines
25592564 trace_btrfs_add_block_group(fs_info, cache, 1);
25602565 btrfs_update_space_info(fs_info, cache->flags, size, bytes_used,
25612566 cache->bytes_super, cache->zone_unusable,
2562− &cache->space_info);
2567+ cache->zone_is_active, &cache->space_info);
25632568 btrfs_update_global_block_rsv(fs_info);
25642569
25652570 link_block_group(cache);
@@ -2659,6 +2664,14 @@ int btrfs_inc_block_group_ro(struct btrfs_block_group *cache, · 93 unmodified lines
26592664 ret = btrfs_chunk_alloc(trans, alloc_flags, CHUNK_ALLOC_FORCE);
26602665 if (ret < 0)
26612666 goto out;
2667+ /*
2668+ * We have allocated a new chunk. We also need to activate that chunk to
2669+ * grant metadata tickets for zoned filesystem.
2670+ */
2671+ ret = btrfs_zoned_activate_one_bg(fs_info, cache->space_info, true);
2672+ if (ret < 0)
2673+ goto out;
2674+
26622675 ret = inc_block_group_ro(cache, 0);
26632676 if (ret == -ETXTBSY)
26642677 goto unlock_out;
@@ -3761,6 +3774,7 @@ int btrfs_chunk_alloc(struct btrfs_trans_handle *trans, u64 flags, · 1096 unmodified lines
37613774 * attempt.
37623775 */
37633776 wait_for_alloc = true;
3777+ force = CHUNK_ALLOC_NO_FORCE;
37643778 spin_unlock(&space_info->lock);
37653779 mutex_lock(&fs_info->chunk_mutex);
37663780 mutex_unlock(&fs_info->chunk_mutex);
@@ -3883,6 +3897,14 @@ static void reserve_chunk_space(struct btrfs_trans_handle *trans, · 116 unmodified lines
38833897 if (IS_ERR(bg)) {
38843898 ret = PTR_ERR(bg);
38853899 } else {
3900+ /*
3901+ * We have a new chunk. We also need to activate it for
3902+ * zoned filesystem.
3903+ */
3904+ ret = btrfs_zoned_activate_one_bg(fs_info, info, true);
3905+ if (ret < 0)
3906+ return;
3907+
38863908 /*
38873909 * If we fail to add the chunk item here, we end up
38883910 * trying again at phase 2 of chunk allocation, at
fs/btrfs/block-rsv.c+9−12
@@ -118,7 +118,7 @@ static u64 block_rsv_release_bytes(struct btrfs_fs_info *fs_info, · 117 unmodified lines
118118 if (block_rsv->reserved >= block_rsv->size) {
119119 num_bytes = block_rsv->reserved - block_rsv->size;
120120 block_rsv->reserved = block_rsv->size;
121− block_rsv->full = 1;
121+ block_rsv->full = true;
122122 } else {
123123 num_bytes = 0;
124124 }
@@ -142,7 +142,7 @@ static u64 block_rsv_release_bytes(struct btrfs_fs_info *fs_info, · 17 unmodified lines
142142 bytes_to_add = min(num_bytes, bytes_to_add);
143143 dest->reserved += bytes_to_add;
144144 if (dest->reserved >= dest->size)
145− dest->full = 1;
145+ dest->full = true;
146146 num_bytes -= bytes_to_add;
147147 }
148148 spin_unlock(&dest->lock);
@@ -171,7 +171,7 @@ int btrfs_block_rsv_migrate(struct btrfs_block_rsv *src, · 22 unmodified lines
171171 return 0;
172172 }
173173
174−void btrfs_init_block_rsv(struct btrfs_block_rsv *rsv, unsigned short type)
174+void btrfs_init_block_rsv(struct btrfs_block_rsv *rsv, enum btrfs_rsv_type type)
175175 {
176176 memset(rsv, 0, sizeof(*rsv));
177177 spin_lock_init(&rsv->lock);
@@ -180,7 +180,7 @@ void btrfs_init_block_rsv(struct btrfs_block_rsv *rsv, unsigned short type) · 2 unmodified lines
180180
181181 void btrfs_init_metadata_block_rsv(struct btrfs_fs_info *fs_info,
182182 struct btrfs_block_rsv *rsv,
183− unsigned short type)
183+ enum btrfs_rsv_type type)
184184 {
185185 btrfs_init_block_rsv(rsv, type);
186186 rsv->space_info = btrfs_find_space_info(fs_info,
@@ -188,7 +188,7 @@ void btrfs_init_metadata_block_rsv(struct btrfs_fs_info *fs_info, · 1 unmodified line
188188 }
189189
190190 struct btrfs_block_rsv *btrfs_alloc_block_rsv(struct btrfs_fs_info *fs_info,
191− unsigned short type)
191+ enum btrfs_rsv_type type)
192192 {
193193 struct btrfs_block_rsv *block_rsv;
194194
@@ -304,7 +304,7 @@ int btrfs_block_rsv_use_bytes(struct btrfs_block_rsv *block_rsv, u64 num_bytes) · 109 unmodified lines
304304 if (block_rsv->reserved >= num_bytes) {
305305 block_rsv->reserved -= num_bytes;
306306 if (block_rsv->reserved < block_rsv->size)
307− block_rsv->full = 0;
307+ block_rsv->full = false;
308308 ret = 0;
309309 }
310310 spin_unlock(&block_rsv->lock);
@@ -319,7 +319,7 @@ void btrfs_block_rsv_add_bytes(struct btrfs_block_rsv *block_rsv, · 8 unmodified lines
319319 if (update_size)
320320 block_rsv->size += num_bytes;
321321 else if (block_rsv->reserved >= block_rsv->size)
322− block_rsv->full = 1;
322+ block_rsv->full = true;
323323 spin_unlock(&block_rsv->lock);
324324 }
325325
@@ -341,7 +341,7 @@ int btrfs_cond_migrate_bytes(struct btrfs_fs_info *fs_info, · 15 unmodified lines
341341 }
342342 global_rsv->reserved -= num_bytes;
343343 if (global_rsv->reserved < global_rsv->size)
344− global_rsv->full = 0;
344+ global_rsv->full = false;
345345 spin_unlock(&global_rsv->lock);
346346
347347 btrfs_block_rsv_add_bytes(dest, num_bytes, true);
@@ -408,10 +408,7 @@ void btrfs_update_global_block_rsv(struct btrfs_fs_info *fs_info) · 60 unmodified lines
408408 btrfs_try_granting_tickets(fs_info, sinfo);
409409 }
410410
411− if (block_rsv->reserved == block_rsv->size)
412− block_rsv->full = 1;
413− else
414− block_rsv->full = 0;
411+ block_rsv->full = (block_rsv->reserved == block_rsv->size);
415412
416413 if (block_rsv->size >= sinfo->total_bytes)
417414 sinfo->force_alloc = CHUNK_ALLOC_FORCE;
fs/btrfs/block-rsv.h+8−7
@@ -9,7 +9,7 @@ enum btrfs_reserve_flush_enum; · 8 unmodified lines
99 /*
1010 * Types of block reserves
1111 */
12−enum {
12+enum btrfs_rsv_type {
1313 BTRFS_BLOCK_RSV_GLOBAL,
1414 BTRFS_BLOCK_RSV_DELALLOC,
1515 BTRFS_BLOCK_RSV_TRANS,
@@ -25,9 +25,10 @@ struct btrfs_block_rsv { · 9 unmodified lines
2525 u64 reserved;
2626 struct btrfs_space_info *space_info;
2727 spinlock_t lock;
28− unsigned short full;
29− unsigned short type;
30− unsigned short failfast;
28+ bool full;
29+ bool failfast;
30+ /* Block reserve type, one of BTRFS_BLOCK_RSV_* */
31+ enum btrfs_rsv_type type:8;
3132
3233 /*
3334 * Qgroup equivalent for @size @reserved
@@ -49,13 +50,13 @@ struct btrfs_block_rsv { · 15 unmodified lines
4950 u64 qgroup_rsv_reserved;
5051 };
5152
52−void btrfs_init_block_rsv(struct btrfs_block_rsv *rsv, unsigned short type);
53+void btrfs_init_block_rsv(struct btrfs_block_rsv *rsv, enum btrfs_rsv_type type);
5354 void btrfs_init_root_block_rsv(struct btrfs_root *root);
5455 struct btrfs_block_rsv *btrfs_alloc_block_rsv(struct btrfs_fs_info *fs_info,
55− unsigned short type);
56+ enum btrfs_rsv_type type);
5657 void btrfs_init_metadata_block_rsv(struct btrfs_fs_info *fs_info,
5758 struct btrfs_block_rsv *rsv,
58− unsigned short type);
59+ enum btrfs_rsv_type type);
5960 void btrfs_free_block_rsv(struct btrfs_fs_info *fs_info,
6061 struct btrfs_block_rsv *rsv);
6162 int btrfs_block_rsv_add(struct btrfs_fs_info *fs_info,
fs/btrfs/btrfs_inode.h+18−7
@@ -279,19 +279,31 @@ static inline void btrfs_insert_inode_hash(struct inode *inode) · 278 unmodified lines
279279 __insert_inode_hash(inode, h);
280280 }
281281
282+#if BITS_PER_LONG == 32
283+
284+/*
285+ * On 32 bit systems the i_ino of struct inode is 32 bits (unsigned long), so
286+ * we use the inode's location objectid which is a u64 to avoid truncation.
287+ */
282288 static inline u64 btrfs_ino(const struct btrfs_inode *inode)
283289 {
284290 u64 ino = inode->location.objectid;
285291
286− /*
287− * !ino: btree_inode
288− * type == BTRFS_ROOT_ITEM_KEY: subvol dir
289− */
290− if (!ino || inode->location.type == BTRFS_ROOT_ITEM_KEY)
292+ /* type == BTRFS_ROOT_ITEM_KEY: subvol dir */
293+ if (inode->location.type == BTRFS_ROOT_ITEM_KEY)
291294 ino = inode->vfs_inode.i_ino;
292295 return ino;
293296 }
294297
298+#else
299+
300+static inline u64 btrfs_ino(const struct btrfs_inode *inode)
301+{
302+ return inode->vfs_inode.i_ino;
303+}
304+
305+#endif
306+
295307 static inline void btrfs_i_size_write(struct btrfs_inode *inode, u64 size)
296308 {
297309 i_size_write(&inode->vfs_inode, size);
@@ -305,8 +317,7 @@ static inline bool btrfs_is_free_space_inode(struct btrfs_inode *inode) · 7 unmodified lines
305317 if (root == root->fs_info->tree_root &&
306318 btrfs_ino(inode) != BTRFS_BTREE_INODE_OBJECTID)
307319 return true;
308− if (inode->location.objectid == BTRFS_FREE_INO_OBJECTID)
309− return true;
320+
310321 return false;
311322 }
312323
fs/btrfs/compression.c+102−257
@@ -136,109 +136,14 @@ static int compression_decompress(int type, struct list_head *ws, · 135 unmodified lines
136136
137137 static int btrfs_decompress_bio(struct compressed_bio *cb);
138138
139−static inline int compressed_bio_size(struct btrfs_fs_info *fs_info,
140− unsigned long disk_size)
141−{
142− return sizeof(struct compressed_bio) +
143− (DIV_ROUND_UP(disk_size, fs_info->sectorsize)) * fs_info->csum_size;
144−}
145−
146−static int check_compressed_csum(struct btrfs_inode *inode, struct bio *bio,
147− u64 disk_start)
148−{
149− struct btrfs_fs_info *fs_info = inode->root->fs_info;
150− SHASH_DESC_ON_STACK(shash, fs_info->csum_shash);
151− const u32 csum_size = fs_info->csum_size;
152− const u32 sectorsize = fs_info->sectorsize;
153− struct page *page;
154− unsigned int i;
155− char *kaddr;
156− u8 csum[BTRFS_CSUM_SIZE];
157− struct compressed_bio *cb = bio->bi_private;
158− u8 *cb_sum = cb->sums;
159−
160− if ((inode->flags & BTRFS_INODE_NODATASUM) ||
161− test_bit(BTRFS_FS_STATE_NO_CSUMS, &fs_info->fs_state))
162− return 0;
163−
164− shash->tfm = fs_info->csum_shash;
165−
166− for (i = 0; i < cb->nr_pages; i++) {
167− u32 pg_offset;
168− u32 bytes_left = PAGE_SIZE;
169− page = cb->compressed_pages[i];
170−
171− /* Determine the remaining bytes inside the page first */
172− if (i == cb->nr_pages - 1)
173− bytes_left = cb->compressed_len - i * PAGE_SIZE;
174−
175− /* Hash through the page sector by sector */
176− for (pg_offset = 0; pg_offset < bytes_left;
177− pg_offset += sectorsize) {
178− kaddr = kmap_atomic(page);
179− crypto_shash_digest(shash, kaddr + pg_offset,
180− sectorsize, csum);
181− kunmap_atomic(kaddr);
182−
183− if (memcmp(&csum, cb_sum, csum_size) != 0) {
184− btrfs_print_data_csum_error(inode, disk_start,
185− csum, cb_sum, cb->mirror_num);
186− if (btrfs_bio(bio)->device)
187− btrfs_dev_stat_inc_and_print(
188− btrfs_bio(bio)->device,
189− BTRFS_DEV_STAT_CORRUPTION_ERRS);
190− return -EIO;
191− }
192− cb_sum += csum_size;
193− disk_start += sectorsize;
194− }
195− }
196− return 0;
197−}
198−
199−/*
200− * Reduce bio and io accounting for a compressed_bio with its corresponding bio.
201− *
202− * Return true if there is no pending bio nor io.
203− * Return false otherwise.
204− */
205−static bool dec_and_test_compressed_bio(struct compressed_bio *cb, struct bio *bio)
206−{
207− struct btrfs_fs_info *fs_info = btrfs_sb(cb->inode->i_sb);
208− unsigned int bi_size = 0;
209− bool last_io = false;
210− struct bio_vec *bvec;
211− struct bvec_iter_all iter_all;
212−
213− /*
214− * At endio time, bi_iter.bi_size doesn't represent the real bio size.
215− * Thus here we have to iterate through all segments to grab correct
216− * bio size.
217− */
218− bio_for_each_segment_all(bvec, bio, iter_all)
219− bi_size += bvec->bv_len;
220−
221− if (bio->bi_status)
222− cb->status = bio->bi_status;
223−
224− ASSERT(bi_size && bi_size <= cb->compressed_len);
225− last_io = refcount_sub_and_test(bi_size >> fs_info->sectorsize_bits,
226− &cb->pending_sectors);
227− /*
228− * Here we must wake up the possible error handler after all other
229− * operations on @cb finished, or we can race with
230− * finish_compressed_bio_*() which may free @cb.
231− */
232− wake_up_var(cb);
233−
234− return last_io;
235−}
236−
237139 static void finish_compressed_bio_read(struct compressed_bio *cb)
238140 {
239141 unsigned int index;
240142 struct page *page;
241143
144+ if (cb->status == BLK_STS_OK)
145+ cb->status = errno_to_blk_status(btrfs_decompress_bio(cb));
146+
242147 /* Release the compressed pages */
243148 for (index = 0; index < cb->nr_pages; index++) {
244149 page = cb->compressed_pages[index];
@@ -247,85 +152,63 @@ static void finish_compressed_bio_read(struct compressed_bio *cb) · 2 unmodified lines
247152 }
248153
249154 /* Do io completion on the original bio */
250− if (cb->status != BLK_STS_OK) {
155+ if (cb->status != BLK_STS_OK)
251156 cb->orig_bio->bi_status = cb->status;
252− bio_endio(cb->orig_bio);
253− } else {
254− struct bio_vec *bvec;
255− struct bvec_iter_all iter_all;
256−
257− /*
258− * We have verified the checksum already, set page checked so
259− * the end_io handlers know about it
260− */
261− ASSERT(!bio_flagged(cb->orig_bio, BIO_CLONED));
262− bio_for_each_segment_all(bvec, cb->orig_bio, iter_all) {
263− u64 bvec_start = page_offset(bvec->bv_page) +
264− bvec->bv_offset;
265−
266− btrfs_page_set_checked(btrfs_sb(cb->inode->i_sb),
267− bvec->bv_page, bvec_start,
268− bvec->bv_len);
269− }
270−
271− bio_endio(cb->orig_bio);
272− }
157+ bio_endio(cb->orig_bio);
273158
274159 /* Finally free the cb struct */
275160 kfree(cb->compressed_pages);
276161 kfree(cb);
277162 }
278163
279−/* when we finish reading compressed pages from the disk, we
280− * decompress them and then run the bio end_io routines on the
281− * decompressed pages (in the inode address space).
282− *
283− * This allows the checksumming and other IO error handling routines
284− * to work normally
285− *
286− * The compressed pages are freed here, and it must be run
287− * in process context
164+/*
165+ * Verify the checksums and kick off repair if needed on the uncompressed data
166+ * before decompressing it into the original bio and freeing the uncompressed
167+ * pages.
288168 */
289169 static void end_compressed_bio_read(struct bio *bio)
290170 {
291171 struct compressed_bio *cb = bio->bi_private;
292− struct inode *inode;
293− unsigned int mirror = btrfs_bio(bio)->mirror_num;
294− int ret = 0;
295−
296− if (!dec_and_test_compressed_bio(cb, bio))
297− goto out;
298−
299− /*
300− * Record the correct mirror_num in cb->orig_bio so that
301− * read-repair can work properly.
302− */
303− btrfs_bio(cb->orig_bio)->mirror_num = mirror;
304− cb->mirror_num = mirror;
305−
306− /*
307− * Some IO in this cb have failed, just skip checksum as there
308− * is no way it could be correct.
309− */
310− if (cb->status != BLK_STS_OK)
311− goto csum_failed;
312−
313− inode = cb->inode;
314− ret = check_compressed_csum(BTRFS_I(inode), bio,
315− bio->bi_iter.bi_sector << 9);
316− if (ret)
317− goto csum_failed;
172+ struct inode *inode = cb->inode;
173+ struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
174+ struct btrfs_inode *bi = BTRFS_I(inode);
175+ bool csum = !(bi->flags & BTRFS_INODE_NODATASUM) &&
176+ !test_bit(BTRFS_FS_STATE_NO_CSUMS, &fs_info->fs_state);
177+ blk_status_t status = bio->bi_status;
178+ struct btrfs_bio *bbio = btrfs_bio(bio);
179+ struct bvec_iter iter;
180+ struct bio_vec bv;
181+ u32 offset;
182+
183+ btrfs_bio_for_each_sector(fs_info, bv, bbio, iter, offset) {
184+ u64 start = bbio->file_offset + offset;
185+
186+ if (!status &&
187+ (!csum || !btrfs_check_data_csum(inode, bbio, offset,
188+ bv.bv_page, bv.bv_offset))) {
189+ clean_io_failure(fs_info, &bi->io_failure_tree,
190+ &bi->io_tree, start, bv.bv_page,
191+ btrfs_ino(bi), bv.bv_offset);
192+ } else {
193+ int ret;
194+
195+ refcount_inc(&cb->pending_ios);
196+ ret = btrfs_repair_one_sector(inode, bbio, offset,
197+ bv.bv_page, bv.bv_offset,
198+ btrfs_submit_data_read_bio);
199+ if (ret) {
200+ refcount_dec(&cb->pending_ios);
201+ status = errno_to_blk_status(ret);
202+ }
203+ }
204+ }
318205
319− /* ok, we're the last bio for this extent, lets start
320− * the decompression.
321− */
322− ret = btrfs_decompress_bio(cb);
206+ if (status)
207+ cb->status = status;
323208
324−csum_failed:
325− if (ret)
326− cb->status = errno_to_blk_status(ret);
327− finish_compressed_bio_read(cb);
328−out:
209+ if (refcount_dec_and_test(&cb->pending_ios))
210+ finish_compressed_bio_read(cb);
211+ btrfs_bio_free_csum(bbio);
329212 bio_put(bio);
330213 }
331214
@@ -403,6 +286,14 @@ static void finish_compressed_bio_write(struct compressed_bio *cb) · 71 unmodified lines
403286 kfree(cb);
404287 }
405288
289+static void btrfs_finish_compressed_write_work(struct work_struct *work)
290+{
291+ struct compressed_bio *cb =
292+ container_of(work, struct compressed_bio, write_end_work);
293+
294+ finish_compressed_bio_write(cb);
295+}
296+
406297 /*
407298 * Do the cleanup once all the compressed pages hit the disk. This will clear
408299 * writeback on the file pages and free the compressed pages.
@@ -414,29 +305,18 @@ static void end_compressed_bio_write(struct bio *bio) · 5 unmodified lines
414305 {
415306 struct compressed_bio *cb = bio->bi_private;
416307
417− if (!dec_and_test_compressed_bio(cb, bio))
418− goto out;
308+ if (bio->bi_status)
309+ cb->status = bio->bi_status;
419310
420− btrfs_record_physical_zoned(cb->inode, cb->start, bio);
311+ if (refcount_dec_and_test(&cb->pending_ios)) {
312+ struct btrfs_fs_info *fs_info = btrfs_sb(cb->inode->i_sb);
421313
422− finish_compressed_bio_write(cb);
423−out:
314+ btrfs_record_physical_zoned(cb->inode, cb->start, bio);
315+ queue_work(fs_info->compressed_write_workers, &cb->write_end_work);
316+ }
424317 bio_put(bio);
425318 }
426319
427−static blk_status_t submit_compressed_bio(struct btrfs_fs_info *fs_info,
428− struct bio *bio, int mirror_num)
429−{
430− blk_status_t ret;
431−
432− ASSERT(bio->bi_iter.bi_size);
433− ret = btrfs_bio_wq_end_io(fs_info, bio, BTRFS_WQ_ENDIO_DATA);
434− if (ret)
435− return ret;
436− ret = btrfs_map_bio(fs_info, bio, mirror_num);
437− return ret;
438−}
439−
440320 /*
441321 * Allocate a compressed_bio, which will be used to read/write on-disk
442322 * (aka, compressed) * data.
@@ -487,7 +367,7 @@ static struct bio *alloc_compressed_bio(struct compressed_bio *cb, u64 disk_byte · 44 unmodified lines
487367 return ERR_PTR(ret);
488368 }
489369 *next_stripe_start = disk_bytenr + geom.len;
490−
370+ refcount_inc(&cb->pending_ios);
491371 return bio;
492372 }
493373
@@ -514,26 +394,25 @@ blk_status_t btrfs_submit_compressed_write(struct btrfs_inode *inode, u64 start, · 20 unmodified lines
514394 struct compressed_bio *cb;
515395 u64 cur_disk_bytenr = disk_start;
516396 u64 next_stripe_start;
517− blk_status_t ret;
397+ blk_status_t ret = BLK_STS_OK;
518398 int skip_sum = inode->flags & BTRFS_INODE_NODATASUM;
519399 const bool use_append = btrfs_use_zone_append(inode, disk_start);
520400 const enum req_op bio_op = use_append ? REQ_OP_ZONE_APPEND : REQ_OP_WRITE;
521401
522402 ASSERT(IS_ALIGNED(start, fs_info->sectorsize) &&
523403 IS_ALIGNED(len, fs_info->sectorsize));
524− cb = kmalloc(compressed_bio_size(fs_info, compressed_len), GFP_NOFS);
404+ cb = kmalloc(sizeof(struct compressed_bio), GFP_NOFS);
525405 if (!cb)
526406 return BLK_STS_RESOURCE;
527− refcount_set(&cb->pending_sectors, compressed_len >> fs_info->sectorsize_bits);
407+ refcount_set(&cb->pending_ios, 1);
528408 cb->status = BLK_STS_OK;
529409 cb->inode = &inode->vfs_inode;
530410 cb->start = start;
531411 cb->len = len;
532− cb->mirror_num = 0;
533412 cb->compressed_pages = compressed_pages;
534413 cb->compressed_len = compressed_len;
535414 cb->writeback = writeback;
536− cb->orig_bio = NULL;
415+ INIT_WORK(&cb->write_end_work, btrfs_finish_compressed_write_work);
537416 cb->nr_pages = nr_pages;
538417
539418 if (blkcg_css)
@@ -554,8 +433,7 @@ blk_status_t btrfs_submit_compressed_write(struct btrfs_inode *inode, u64 start, · 14 unmodified lines
554433 &next_stripe_start);
555434 if (IS_ERR(bio)) {
556435 ret = errno_to_blk_status(PTR_ERR(bio));
557− bio = NULL;
558− goto finish_cb;
436+ break;
559437 }
560438 if (blkcg_css)
561439 bio->bi_opf |= REQ_CGROUP_PUNT;
@@ -599,44 +477,25 @@ blk_status_t btrfs_submit_compressed_write(struct btrfs_inode *inode, u64 start, · 37 unmodified lines
599477 if (submit) {
600478 if (!skip_sum) {
601479 ret = btrfs_csum_one_bio(inode, bio, start, true);
602− if (ret)
603− goto finish_cb;
480+ if (ret) {
481+ bio->bi_status = ret;
482+ bio_endio(bio);
483+ break;
484+ }
604485 }
605486
606− ret = submit_compressed_bio(fs_info, bio, 0);
607− if (ret)
608− goto finish_cb;
487+ ASSERT(bio->bi_iter.bi_size);
488+ btrfs_submit_bio(fs_info, bio, 0);
609489 bio = NULL;
610490 }
611491 cond_resched();
612492 }
613− if (blkcg_css)
614− kthread_associate_blkcg(NULL);
615−
616− return 0;
617493
618−finish_cb:
619494 if (blkcg_css)
620495 kthread_associate_blkcg(NULL);
621496
622− if (bio) {
623− bio->bi_status = ret;
624− bio_endio(bio);
625− }
626− /* Last byte of @cb is submitted, endio will free @cb */
627− if (cur_disk_bytenr == disk_start + compressed_len)
628− return ret;
629−
630− wait_var_event(cb, refcount_read(&cb->pending_sectors) ==
631− (disk_start + compressed_len - cur_disk_bytenr) >>
632− fs_info->sectorsize_bits);
633− /*
634− * Even with previous bio ended, we should still have io not yet
635− * submitted, thus need to finish manually.
636− */
637− ASSERT(refcount_read(&cb->pending_sectors));
638− /* Now we are the only one referring @cb, can finish it safely. */
639− finish_compressed_bio_write(cb);
497+ if (refcount_dec_and_test(&cb->pending_ios))
498+ finish_compressed_bio_write(cb);
640499 return ret;
641500 }
642501
@@ -765,7 +624,6 @@ static noinline int add_ra_bio_pages(struct inode *inode, · 122 unmodified lines
765624 int zeros;
766625 zeros = PAGE_SIZE - zero_offset;
767626 memzero_page(page, zero_offset, zeros);
768− flush_dcache_page(page);
769627 }
770628 }
771629
@@ -819,7 +677,6 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, · 47 unmodified lines
819677 blk_status_t ret;
820678 int ret2;
821679 int i;
822− u8 *sums;
823680
824681 em_tree = &BTRFS_I(inode)->extent_tree;
825682
@@ -837,17 +694,15 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, · 11 unmodified lines
837694
838695 ASSERT(em->compress_type != BTRFS_COMPRESS_NONE);
839696 compressed_len = em->block_len;
840− cb = kmalloc(compressed_bio_size(fs_info, compressed_len), GFP_NOFS);
697+ cb = kmalloc(sizeof(struct compressed_bio), GFP_NOFS);
841698 if (!cb) {
842699 ret = BLK_STS_RESOURCE;
843700 goto out;
844701 }
845702
846− refcount_set(&cb->pending_sectors, compressed_len >> fs_info->sectorsize_bits);
703+ refcount_set(&cb->pending_ios, 1);
847704 cb->status = BLK_STS_OK;
848705 cb->inode = inode;
849− cb->mirror_num = mirror_num;
850− sums = cb->sums;
851706
852707 cb->start = em->orig_start;
853708 em_len = em->len;
@@ -893,9 +748,8 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, · 39 unmodified lines
893748 REQ_OP_READ, end_compressed_bio_read,
894749 &next_stripe_start);
895750 if (IS_ERR(comp_bio)) {
896− ret = errno_to_blk_status(PTR_ERR(comp_bio));
897− comp_bio = NULL;
898− goto finish_cb;
751+ cb->status = errno_to_blk_status(PTR_ERR(comp_bio));
752+ break;
899753 }
900754 }
901755 /*
@@ -931,22 +785,33 @@ void btrfs_submit_compressed_read(struct inode *inode, struct bio *bio, · 29 unmodified lines
931785 submit = true;
932786
933787 if (submit) {
934− unsigned int nr_sectors;
788+ /* Save the original iter for read repair */
789+ if (bio_op(comp_bio) == REQ_OP_READ)
790+ btrfs_bio(comp_bio)->iter = comp_bio->bi_iter;
935791
936− ret = btrfs_lookup_bio_sums(inode, comp_bio, sums);
937− if (ret)
938− goto finish_cb;
792+ /*
793+ * Save the initial offset of this chunk, as there
794+ * is no direct correlation between compressed pages and
795+ * the original file offset. The field is only used for
796+ * priting error messages.
797+ */
798+ btrfs_bio(comp_bio)->file_offset = file_offset;
939799
940− nr_sectors = DIV_ROUND_UP(comp_bio->bi_iter.bi_size,
941− fs_info->sectorsize);
942− sums += fs_info->csum_size * nr_sectors;
800+ ret = btrfs_lookup_bio_sums(inode, comp_bio, NULL);
801+ if (ret) {
802+ comp_bio->bi_status = ret;
803+ bio_endio(comp_bio);
804+ break;
805+ }
943806
944− ret = submit_compressed_bio(fs_info, comp_bio, mirror_num);
945− if (ret)
946− goto finish_cb;
807+ ASSERT(comp_bio->bi_iter.bi_size);
808+ btrfs_submit_bio(fs_info, comp_bio, mirror_num);
947809 comp_bio = NULL;
948810 }
949811 }
812+
813+ if (refcount_dec_and_test(&cb->pending_ios))
814+ finish_compressed_bio_read(cb);
950815 return;
951816
952817 fail:
@@ -964,25 +829,6 @@ out: · 11 unmodified lines
964829 bio->bi_status = ret;
965830 bio_endio(bio);
966831 return;
967−finish_cb:
968− if (comp_bio) {
969− comp_bio->bi_status = ret;
970− bio_endio(comp_bio);
971− }
972− /* All bytes of @cb is submitted, endio will free @cb */
973− if (cur_disk_byte == disk_bytenr + compressed_len)
974− return;
975−
976− wait_var_event(cb, refcount_read(&cb->pending_sectors) ==
977− (disk_bytenr + compressed_len - cur_disk_byte) >>
978− fs_info->sectorsize_bits);
979− /*
980− * Even with previous bio ended, we should still have io not yet
981− * submitted, thus need to finish @cb manually.
982− */
983− ASSERT(refcount_read(&cb->pending_sectors));
984− /* Now we are the only one referring @cb, can finish it safely. */
985− finish_compressed_bio_read(cb);
986832 }
987833
988834 /*
@@ -1481,7 +1327,6 @@ int btrfs_decompress_buf2page(const char *buf, u32 buf_len, · 492 unmodified lines
14811327 ASSERT(copy_start - decompressed < buf_len);
14821328 memcpy_to_page(bvec.bv_page, bvec.bv_offset,
14831329 buf + copy_start - decompressed, copy_len);
1484− flush_dcache_page(bvec.bv_page);
14851330 cur_offset += copy_len;
14861331
14871332 bio_advance(orig_bio, copy_len);
fs/btrfs/compression.h+7−11
@@ -30,8 +30,8 @@ static_assert((BTRFS_MAX_COMPRESSED % PAGE_SIZE) == 0); · 29 unmodified lines
3030 #define BTRFS_ZLIB_DEFAULT_LEVEL 3
3131
3232 struct compressed_bio {
33− /* Number of sectors with unfinished IO (unsubmitted or unfinished) */
34− refcount_t pending_sectors;
33+ /* Number of outstanding bios */
34+ refcount_t pending_ios;
3535
3636 /* Number of compressed pages in the array */
3737 unsigned int nr_pages;
@@ -59,16 +59,12 @@ struct compressed_bio { · 21 unmodified lines
5959
6060 /* IO errors */
6161 blk_status_t status;
62− int mirror_num;
6362
64− /* for reads, this is the bio we are copying the data into */
65− struct bio *orig_bio;
66−
67− /*
68− * the start of a variable length array of checksums only
69− * used by reads
70− */
71− u8 sums[];
63+ union {
64+ /* For reads, this is the bio we are copying the data into */
65+ struct bio *orig_bio;
66+ struct work_struct write_end_work;
67+ };
7268 };
7369
7470 static inline unsigned int btrfs_compress_type(unsigned int type_level)
fs/btrfs/ctree.h+84−21
@@ -107,14 +107,6 @@ struct btrfs_ioctl_encoded_io_args; · 106 unmodified lines
107107 #define BTRFS_STAT_CURR 0
108108 #define BTRFS_STAT_PREV 1
109109
110−/*
111− * Count how many BTRFS_MAX_EXTENT_SIZE cover the @size
112− */
113−static inline u32 count_max_extents(u64 size)
114−{
115− return div_u64(size + BTRFS_MAX_EXTENT_SIZE - 1, BTRFS_MAX_EXTENT_SIZE);
116−}
117−
118110 static inline unsigned long btrfs_chunk_item_size(int num_stripes)
119111 {
120112 BUG_ON(num_stripes == 0);
@@ -229,6 +221,13 @@ struct btrfs_root_backup { · 108 unmodified lines
229221 #define BTRFS_SUPER_INFO_OFFSET SZ_64K
230222 #define BTRFS_SUPER_INFO_SIZE 4096
231223
224+/*
225+ * The reserved space at the beginning of each device.
226+ * It covers the primary super block and leaves space for potential use by other
227+ * tools like bootloaders or to lower potential damage of accidental overwrite.
228+ */
229+#define BTRFS_DEVICE_RANGE_RESERVED (SZ_1M)
230+
232231 /*
233232 * the super block basically lists the main trees of the FS
234233 * it currently lacks any block count etc etc
@@ -248,8 +247,12 @@ struct btrfs_super_block { · 13 unmodified lines
248247 __le64 chunk_root;
249248 __le64 log_root;
250249
251− /* this will help find the new super based on the log root */
252− __le64 log_root_transid;
250+ /*
251+ * This member has never been utilized since the very beginning, thus
252+ * it's always 0 regardless of kernel version. We always use
253+ * generation + 1 to read log tree root. So here we mark it deprecated.
254+ */
255+ __le64 __unused_log_root_transid;
253256 __le64 total_bytes;
254257 __le64 bytes_used;
255258 __le64 root_dir_objectid;
@@ -635,6 +638,9 @@ enum { · 379 unmodified lines
635638 /* Indicate we have half completed snapshot deletions pending. */
636639 BTRFS_FS_UNFINISHED_DROPS,
637640
641+ /* Indicate we have to finish a zone to do next allocation. */
642+ BTRFS_FS_NEED_ZONE_FINISH,
643+
638644 #if BITS_PER_LONG == 32
639645 /* Indicate if we have error/warn message printed on 32bit systems */
640646 BTRFS_FS_32BIT_ERROR,
@@ -656,6 +662,18 @@ enum btrfs_exclusive_operation { · 15 unmodified lines
656662 BTRFS_EXCLOP_SWAP_ACTIVATE,
657663 };
658664
665+/* Store data about transaction commits, exported via sysfs. */
666+struct btrfs_commit_stats {
667+ /* Total number of commits */
668+ u64 commit_count;
669+ /* The maximum commit duration so far in ns */
670+ u64 max_commit_dur;
671+ /* The last commit duration in ns */
672+ u64 last_commit_dur;
673+ /* The total commit duration in ns */
674+ u64 total_commit_dur;
675+};
676+
659677 struct btrfs_fs_info {
660678 u8 chunk_tree_uuid[BTRFS_UUID_SIZE];
661679 unsigned long flags;
@@ -850,11 +868,11 @@ struct btrfs_fs_info { · 188 unmodified lines
850868 struct btrfs_workqueue *hipri_workers;
851869 struct btrfs_workqueue *delalloc_workers;
852870 struct btrfs_workqueue *flush_workers;
853− struct btrfs_workqueue *endio_workers;
854− struct btrfs_workqueue *endio_meta_workers;
855− struct btrfs_workqueue *endio_raid56_workers;
871+ struct workqueue_struct *endio_workers;
872+ struct workqueue_struct *endio_meta_workers;
873+ struct workqueue_struct *endio_raid56_workers;
856874 struct workqueue_struct *rmw_workers;
857− struct btrfs_workqueue *endio_meta_write_workers;
875+ struct workqueue_struct *compressed_write_workers;
858876 struct btrfs_workqueue *endio_write_workers;
859877 struct btrfs_workqueue *endio_freespace_worker;
860878 struct btrfs_workqueue *caching_workers;
@@ -1032,6 +1050,12 @@ struct btrfs_fs_info { · 171 unmodified lines
10321050 u32 csums_per_leaf;
10331051 u32 stripesize;
10341052
1053+ /*
1054+ * Maximum size of an extent. BTRFS_MAX_EXTENT_SIZE on regular
1055+ * filesystem, on zoned it depends on the device constraints.
1056+ */
1057+ u64 max_extent_size;
1058+
10351059 /* Block groups and devices containing active swapfiles. */
10361060 spinlock_t swapfile_pins_lock;
10371061 struct rb_root swapfile_pins;
@@ -1047,6 +1071,8 @@ struct btrfs_fs_info { · 9 unmodified lines
10471071 */
10481072 u64 zone_size;
10491073
1074+ /* Max size to emit ZONE_APPEND write command */
1075+ u64 max_zone_append_size;
10501076 struct mutex zoned_meta_io_lock;
10511077 spinlock_t treelog_bg_lock;
10521078 u64 treelog_bg;
@@ -1063,6 +1089,11 @@ struct btrfs_fs_info { · 10 unmodified lines
10631089
10641090 spinlock_t zone_active_bgs_lock;
10651091 struct list_head zone_active_bgs;
1092+ /* Waiters when BTRFS_FS_NEED_ZONE_FINISH is set */
1093+ wait_queue_head_t zone_finish_wait;
1094+
1095+ /* Updates are not protected by any lock */
1096+ struct btrfs_commit_stats commit_stats;
10661097
10671098 #ifdef CONFIG_BTRFS_FS_REF_VERIFY
10681099 spinlock_t ref_verify_lock;
@@ -2475,8 +2506,6 @@ BTRFS_SETGET_STACK_FUNCS(super_chunk_root_level, struct btrfs_super_block, · 1406 unmodified lines
24752506 chunk_root_level, 8);
24762507 BTRFS_SETGET_STACK_FUNCS(super_log_root, struct btrfs_super_block,
24772508 log_root, 64);
2478−BTRFS_SETGET_STACK_FUNCS(super_log_root_transid, struct btrfs_super_block,
2479− log_root_transid, 64);
24802509 BTRFS_SETGET_STACK_FUNCS(super_log_root_level, struct btrfs_super_block,
24812510 log_root_level, 8);
24822511 BTRFS_SETGET_STACK_FUNCS(super_total_bytes, struct btrfs_super_block,
@@ -2733,8 +2762,16 @@ int btrfs_get_extent_inline_ref_type(const struct extent_buffer *eb, · 250 unmodified lines
27332762 enum btrfs_inline_ref_type is_data);
27342763 u64 hash_extent_data_ref(u64 root_objectid, u64 owner, u64 offset);
27352764
2765+static inline u8 *btrfs_csum_ptr(const struct btrfs_fs_info *fs_info, u8 *csums,
2766+ u64 offset)
2767+{
2768+ u64 offset_in_sectors = offset >> fs_info->sectorsize_bits;
2769+
2770+ return csums + offset_in_sectors * fs_info->csum_size;
2771+}
2772+
27362773 /*
2737− * Take the number of bytes to be checksummmed and figure out how many leaves
2774+ * Take the number of bytes to be checksummed and figure out how many leaves
27382775 * it would require to store the csums for that many bytes.
27392776 */
27402777 static inline u64 btrfs_csum_bytes_to_leaves(
@@ -3251,11 +3288,18 @@ void btrfs_inode_safe_disk_i_size_write(struct btrfs_inode *inode, u64 new_i_siz · 510 unmodified lines
32513288 u64 btrfs_file_extent_end(const struct btrfs_path *path);
32523289
32533290 /* inode.c */
3254−void btrfs_submit_data_bio(struct inode *inode, struct bio *bio,
3255− int mirror_num, enum btrfs_compression_type compress_type);
3291+void btrfs_submit_data_write_bio(struct inode *inode, struct bio *bio, int mirror_num);
3292+void btrfs_submit_data_read_bio(struct inode *inode, struct bio *bio,
3293+ int mirror_num, enum btrfs_compression_type compress_type);
3294+int btrfs_check_sector_csum(struct btrfs_fs_info *fs_info, struct page *page,
3295+ u32 pgoff, u8 *csum, const u8 * const csum_expected);
3296+int btrfs_check_data_csum(struct inode *inode, struct btrfs_bio *bbio,
3297+ u32 bio_offset, struct page *page, u32 pgoff);
32563298 unsigned int btrfs_verify_data_csum(struct btrfs_bio *bbio,
32573299 u32 bio_offset, struct page *page,
32583300 u64 start, u64 end);
3301+int btrfs_check_data_csum(struct inode *inode, struct btrfs_bio *bbio,
3302+ u32 bio_offset, struct page *page, u32 pgoff);
32593303 struct extent_map *btrfs_get_extent_fiemap(struct btrfs_inode *inode,
32603304 u64 start, u64 len);
32613305 noinline int can_nocow_extent(struct inode *inode, u64 offset, u64 *len,
@@ -3305,9 +3349,9 @@ void btrfs_new_inode_args_destroy(struct btrfs_new_inode_args *args); · 43 unmodified lines
33053349 struct inode *btrfs_new_subvol_inode(struct user_namespace *mnt_userns,
33063350 struct inode *dir);
33073351 void btrfs_set_delalloc_extent(struct inode *inode, struct extent_state *state,
3308− unsigned *bits);
3352+ u32 bits);
33093353 void btrfs_clear_delalloc_extent(struct inode *inode,
3310− struct extent_state *state, unsigned *bits);
3354+ struct extent_state *state, u32 bits);
33113355 void btrfs_merge_delalloc_extent(struct inode *inode, struct extent_state *new,
33123356 struct extent_state *other);
33133357 void btrfs_split_delalloc_extent(struct inode *inode,
@@ -3353,6 +3397,12 @@ int btrfs_writepage_cow_fixup(struct page *page); · 39 unmodified lines
33533397 void btrfs_writepage_endio_finish_ordered(struct btrfs_inode *inode,
33543398 struct page *page, u64 start,
33553399 u64 end, bool uptodate);
3400+int btrfs_encoded_io_compression_from_extent(struct btrfs_fs_info *fs_info,
3401+ int compress_type);
3402+int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode,
3403+ u64 file_offset, u64 disk_bytenr,
3404+ u64 disk_io_size,
3405+ struct page **pages);
33563406 ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter,
33573407 struct btrfs_ioctl_encoded_io_args *encoded);
33583408 ssize_t btrfs_do_encoded_write(struct kiocb *iocb, struct iov_iter *from,
@@ -4009,6 +4059,19 @@ static inline bool btrfs_is_zoned(const struct btrfs_fs_info *fs_info) · 650 unmodified lines
40094059 return fs_info->zone_size > 0;
40104060 }
40114061
4062+/*
4063+ * Count how many fs_info->max_extent_size cover the @size
4064+ */
4065+static inline u32 count_max_extents(struct btrfs_fs_info *fs_info, u64 size)
4066+{
4067+#ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
4068+ if (!fs_info)
4069+ return div_u64(size + BTRFS_MAX_EXTENT_SIZE - 1, BTRFS_MAX_EXTENT_SIZE);
4070+#endif
4071+
4072+ return div_u64(size + fs_info->max_extent_size - 1, fs_info->max_extent_size);
4073+}
4074+
40124075 static inline bool btrfs_is_data_reloc_root(const struct btrfs_root *root)
40134076 {
40144077 return root->root_key.objectid == BTRFS_DATA_RELOC_TREE_OBJECTID;
fs/btrfs/delalloc-space.c+3−3
@@ -273,7 +273,7 @@ static void calc_inode_reservations(struct btrfs_fs_info *fs_info, · 272 unmodified lines
273273 u64 num_bytes, u64 disk_num_bytes,
274274 u64 *meta_reserve, u64 *qgroup_reserve)
275275 {
276− u64 nr_extents = count_max_extents(num_bytes);
276+ u64 nr_extents = count_max_extents(fs_info, num_bytes);
277277 u64 csum_leaves = btrfs_csum_bytes_to_leaves(fs_info, disk_num_bytes);
278278 u64 inode_update = btrfs_calc_metadata_size(fs_info, 1);
279279
@@ -350,7 +350,7 @@ int btrfs_delalloc_reserve_metadata(struct btrfs_inode *inode, u64 num_bytes, · 70 unmodified lines
350350 * needs to free the reservation we just made.
351351 */
352352 spin_lock(&inode->lock);
353− nr_extents = count_max_extents(num_bytes);
353+ nr_extents = count_max_extents(fs_info, num_bytes);
354354 btrfs_mod_outstanding_extents(inode, nr_extents);
355355 inode->csum_bytes += disk_num_bytes;
356356 btrfs_calculate_inode_block_rsv_size(fs_info, inode);
@@ -413,7 +413,7 @@ void btrfs_delalloc_release_extents(struct btrfs_inode *inode, u64 num_bytes) · 56 unmodified lines
413413 unsigned num_extents;
414414
415415 spin_lock(&inode->lock);
416− num_extents = count_max_extents(num_bytes);
416+ num_extents = count_max_extents(fs_info, num_bytes);
417417 btrfs_mod_outstanding_extents(inode, -num_extents);
418418 btrfs_calculate_inode_block_rsv_size(fs_info, inode);
419419 spin_unlock(&inode->lock);
fs/btrfs/delayed-inode.c+281−114
@@ -52,18 +52,6 @@ static inline void btrfs_init_delayed_node( · 51 unmodified lines
5252 INIT_LIST_HEAD(&delayed_node->p_list);
5353 }
5454
55−static inline int btrfs_is_continuous_delayed_item(
56− struct btrfs_delayed_item *item1,
57− struct btrfs_delayed_item *item2)
58−{
59− if (item1->key.type == BTRFS_DIR_INDEX_KEY &&
60− item1->key.objectid == item2->key.objectid &&
61− item1->key.type == item2->key.type &&
62− item1->key.offset + 1 == item2->key.offset)
63− return 1;
64− return 0;
65−}
66−
6755 static struct btrfs_delayed_node *btrfs_get_delayed_node(
6856 struct btrfs_inode *btrfs_inode)
6957 {
@@ -398,8 +386,7 @@ static struct btrfs_delayed_item *__btrfs_lookup_delayed_insertion_item( · 328 unmodified lines
398386 }
399387
400388 static int __btrfs_add_delayed_item(struct btrfs_delayed_node *delayed_node,
401− struct btrfs_delayed_item *ins,
402− int action)
389+ struct btrfs_delayed_item *ins)
403390 {
404391 struct rb_node **p, *node;
405392 struct rb_node *parent_node = NULL;
@@ -408,9 +395,9 @@ static int __btrfs_add_delayed_item(struct btrfs_delayed_node *delayed_node, · 2 unmodified lines
408395 int cmp;
409396 bool leftmost = true;
410397
411− if (action == BTRFS_DELAYED_INSERTION_ITEM)
398+ if (ins->ins_or_del == BTRFS_DELAYED_INSERTION_ITEM)
412399 root = &delayed_node->ins_root;
413− else if (action == BTRFS_DELAYED_DELETION_ITEM)
400+ else if (ins->ins_or_del == BTRFS_DELAYED_DELETION_ITEM)
414401 root = &delayed_node->del_root;
415402 else
416403 BUG();
@@ -436,32 +423,19 @@ static int __btrfs_add_delayed_item(struct btrfs_delayed_node *delayed_node, · 19 unmodified lines
436423 rb_link_node(node, parent_node, p);
437424 rb_insert_color_cached(node, root, leftmost);
438425 ins->delayed_node = delayed_node;
439− ins->ins_or_del = action;
440426
441− if (ins->key.type == BTRFS_DIR_INDEX_KEY &&
442− action == BTRFS_DELAYED_INSERTION_ITEM &&
427+ /* Delayed items are always for dir index items. */
428+ ASSERT(ins->key.type == BTRFS_DIR_INDEX_KEY);
429+
430+ if (ins->ins_or_del == BTRFS_DELAYED_INSERTION_ITEM &&
443431 ins->key.offset >= delayed_node->index_cnt)
444− delayed_node->index_cnt = ins->key.offset + 1;
432+ delayed_node->index_cnt = ins->key.offset + 1;
445433
446434 delayed_node->count++;
447435 atomic_inc(&delayed_node->root->fs_info->delayed_root->items);
448436 return 0;
449437 }
450438
451−static int __btrfs_add_delayed_insertion_item(struct btrfs_delayed_node *node,
452− struct btrfs_delayed_item *item)
453−{
454− return __btrfs_add_delayed_item(node, item,
455− BTRFS_DELAYED_INSERTION_ITEM);
456−}
457−
458−static int __btrfs_add_delayed_deletion_item(struct btrfs_delayed_node *node,
459− struct btrfs_delayed_item *item)
460−{
461− return __btrfs_add_delayed_item(node, item,
462− BTRFS_DELAYED_DELETION_ITEM);
463−}
464−
465439 static void finish_one_item(struct btrfs_delayed_root *delayed_root)
466440 {
467441 int seq = atomic_inc_return(&delayed_root->items_seq);
@@ -573,7 +547,13 @@ static int btrfs_delayed_item_reserve_metadata(struct btrfs_trans_handle *trans, · 105 unmodified lines
573547 trace_btrfs_space_reservation(fs_info, "delayed_item",
574548 item->key.objectid,
575549 num_bytes, 1);
576− item->bytes_reserved = num_bytes;
550+ /*
551+ * For insertions we track reserved metadata space by accounting
552+ * for the number of leaves that will be used, based on the delayed
553+ * node's index_items_size field.
554+ */
555+ if (item->ins_or_del == BTRFS_DELAYED_DELETION_ITEM)
556+ item->bytes_reserved = num_bytes;
577557 }
578558
579559 return ret;
@@ -599,6 +579,21 @@ static void btrfs_delayed_item_release_metadata(struct btrfs_root *root, · 19 unmodified lines
599579 btrfs_block_rsv_release(fs_info, rsv, item->bytes_reserved, NULL);
600580 }
601581
582+static void btrfs_delayed_item_release_leaves(struct btrfs_delayed_node *node,
583+ unsigned int num_leaves)
584+{
585+ struct btrfs_fs_info *fs_info = node->root->fs_info;
586+ const u64 bytes = btrfs_calc_insert_metadata_size(fs_info, num_leaves);
587+
588+ /* There are no space reservations during log replay, bail out. */
589+ if (test_bit(BTRFS_FS_LOG_RECOVERING, &fs_info->flags))
590+ return;
591+
592+ trace_btrfs_space_reservation(fs_info, "delayed_item", node->inode_id,
593+ bytes, 0);
594+ btrfs_block_rsv_release(fs_info, &fs_info->delayed_block_rsv, bytes, NULL);
595+}
596+
602597 static int btrfs_delayed_inode_reserve_metadata(
603598 struct btrfs_trans_handle *trans,
604599 struct btrfs_root *root,
@@ -672,22 +667,53 @@ static void btrfs_delayed_inode_release_metadata(struct btrfs_fs_info *fs_info, · 67 unmodified lines
672667 }
673668
674669 /*
675− * Insert a single delayed item or a batch of delayed items that have consecutive
676− * keys if they exist.
670+ * Insert a single delayed item or a batch of delayed items, as many as possible
671+ * that fit in a leaf. The delayed items (dir index keys) are sorted by their key
672+ * in the rbtree, and if there's a gap between two consecutive dir index items,
673+ * then it means at some point we had delayed dir indexes to add but they got
674+ * removed (by btrfs_delete_delayed_dir_index()) before we attempted to flush them
675+ * into the subvolume tree. Dir index keys also have their offsets coming from a
676+ * monotonically increasing counter, so we can't get new keys with an offset that
677+ * fits within a gap between delayed dir index items.
677678 */
678679 static int btrfs_insert_delayed_item(struct btrfs_trans_handle *trans,
679680 struct btrfs_root *root,
680681 struct btrfs_path *path,
681682 struct btrfs_delayed_item *first_item)
682683 {
684+ struct btrfs_fs_info *fs_info = root->fs_info;
685+ struct btrfs_delayed_node *node = first_item->delayed_node;
683686 LIST_HEAD(item_list);
684687 struct btrfs_delayed_item *curr;
685688 struct btrfs_delayed_item *next;
686− const int max_size = BTRFS_LEAF_DATA_SIZE(root->fs_info);
689+ const int max_size = BTRFS_LEAF_DATA_SIZE(fs_info);
687690 struct btrfs_item_batch batch;
688691 int total_size;
689692 char *ins_data = NULL;
690693 int ret;
694+ bool continuous_keys_only = false;
695+
696+ lockdep_assert_held(&node->mutex);
697+
698+ /*
699+ * During normal operation the delayed index offset is continuously
700+ * increasing, so we can batch insert all items as there will not be any
701+ * overlapping keys in the tree.
702+ *
703+ * The exception to this is log replay, where we may have interleaved
704+ * offsets in the tree, so our batch needs to be continuous keys only in
705+ * order to ensure we do not end up with out of order items in our leaf.
706+ */
707+ if (test_bit(BTRFS_FS_LOG_RECOVERING, &fs_info->flags))
708+ continuous_keys_only = true;
709+
710+ /*
711+ * For delayed items to insert, we track reserved metadata bytes based
712+ * on the number of leaves that we will use.
713+ * See btrfs_insert_delayed_dir_index() and
714+ * btrfs_delayed_item_reserve_metadata()).
715+ */
716+ ASSERT(first_item->bytes_reserved == 0);
691717
692718 list_add_tail(&first_item->tree_list, &item_list);
693719 batch.total_data_size = first_item->data_len;
@@ -699,9 +725,19 @@ static int btrfs_insert_delayed_item(struct btrfs_trans_handle *trans, · 5 unmodified lines
699725 int next_size;
700726
701727 next = __btrfs_next_delayed_item(curr);
702− if (!next || !btrfs_is_continuous_delayed_item(curr, next))
728+ if (!next)
729+ break;
730+
731+ /*
732+ * We cannot allow gaps in the key space if we're doing log
733+ * replay.
734+ */
735+ if (continuous_keys_only &&
736+ (next->key.offset != curr->key.offset + 1))
703737 break;
704738
739+ ASSERT(next->bytes_reserved == 0);
740+
705741 next_size = next->data_len + sizeof(struct btrfs_item);
706742 if (total_size + next_size > max_size)
707743 break;
@@ -758,9 +794,41 @@ static int btrfs_insert_delayed_item(struct btrfs_trans_handle *trans, · 50 unmodified lines
758794 */
759795 btrfs_release_path(path);
760796
797+ ASSERT(node->index_item_leaves > 0);
798+
799+ /*
800+ * For normal operations we will batch an entire leaf's worth of delayed
801+ * items, so if there are more items to process we can decrement
802+ * index_item_leaves by 1 as we inserted 1 leaf's worth of items.
803+ *
804+ * However for log replay we may not have inserted an entire leaf's
805+ * worth of items, we may have not had continuous items, so decrementing
806+ * here would mess up the index_item_leaves accounting. For this case
807+ * only clean up the accounting when there are no items left.
808+ */
809+ if (next && !continuous_keys_only) {
810+ /*
811+ * We inserted one batch of items into a leaf a there are more
812+ * items to flush in a future batch, now release one unit of
813+ * metadata space from the delayed block reserve, corresponding
814+ * the leaf we just flushed to.
815+ */
816+ btrfs_delayed_item_release_leaves(node, 1);
817+ node->index_item_leaves--;
818+ } else if (!next) {
819+ /*
820+ * There are no more items to insert. We can have a number of
821+ * reserved leaves > 1 here - this happens when many dir index
822+ * items are added and then removed before they are flushed (file
823+ * names with a very short life, never span a transaction). So
824+ * release all remaining leaves.
825+ */
826+ btrfs_delayed_item_release_leaves(node, node->index_item_leaves);
827+ node->index_item_leaves = 0;
828+ }
829+
761830 list_for_each_entry_safe(curr, next, &item_list, tree_list) {
762831 list_del(&curr->tree_list);
763− btrfs_delayed_item_release_metadata(root, curr);
764832 btrfs_release_delayed_item(curr);
765833 }
766834 out:
@@ -796,62 +864,75 @@ static int btrfs_batch_delete_items(struct btrfs_trans_handle *trans, · 29 unmodified lines
796864 struct btrfs_path *path,
797865 struct btrfs_delayed_item *item)
798866 {
867+ struct btrfs_fs_info *fs_info = root->fs_info;
799868 struct btrfs_delayed_item *curr, *next;
800− struct extent_buffer *leaf;
801− struct btrfs_key key;
802− struct list_head head;
803− int nitems, i, last_item;
804− int ret = 0;
869+ struct extent_buffer *leaf = path->nodes[0];
870+ LIST_HEAD(batch_list);
871+ int nitems, slot, last_slot;
872+ int ret;
873+ u64 total_reserved_size = item->bytes_reserved;
805874
806− BUG_ON(!path->nodes[0]);
875+ ASSERT(leaf != NULL);
807876
808− leaf = path->nodes[0];
877+ slot = path->slots[0];
878+ last_slot = btrfs_header_nritems(leaf) - 1;
879+ /*
880+ * Our caller always gives us a path pointing to an existing item, so
881+ * this can not happen.
882+ */
883+ ASSERT(slot <= last_slot);
884+ if (WARN_ON(slot > last_slot))
885+ return -ENOENT;
809886
810− i = path->slots[0];
811− last_item = btrfs_header_nritems(leaf) - 1;
812− if (i > last_item)
813− return -ENOENT; /* FIXME: Is errno suitable? */
887+ nitems = 1;
888+ curr = item;
889+ list_add_tail(&curr->tree_list, &batch_list);
814890
815− next = item;
816− INIT_LIST_HEAD(&head);
817− btrfs_item_key_to_cpu(leaf, &key, i);
818− nitems = 0;
819891 /*
820− * count the number of the dir index items that we can delete in batch
892+ * Keep checking if the next delayed item matches the next item in the
893+ * leaf - if so, we can add it to the batch of items to delete from the
894+ * leaf.
821895 */
822− while (btrfs_comp_cpu_keys(&next->key, &key) == 0) {
823− list_add_tail(&next->tree_list, &head);
824− nitems++;
896+ while (slot < last_slot) {
897+ struct btrfs_key key;
825898
826− curr = next;
827899 next = __btrfs_next_delayed_item(curr);
828900 if (!next)
829901 break;
830902
831− if (!btrfs_is_continuous_delayed_item(curr, next))
832− break;
833−
834− i++;
835− if (i > last_item)
903+ slot++;
904+ btrfs_item_key_to_cpu(leaf, &key, slot);
905+ if (btrfs_comp_cpu_keys(&next->key, &key) != 0)
836906 break;
837− btrfs_item_key_to_cpu(leaf, &key, i);
907+ nitems++;
908+ curr = next;
909+ list_add_tail(&curr->tree_list, &batch_list);
910+ total_reserved_size += curr->bytes_reserved;
838911 }
839912
840− if (!nitems)
841− return 0;
842−
843913 ret = btrfs_del_items(trans, root, path, path->slots[0], nitems);
844914 if (ret)
845− goto out;
915+ return ret;
916+
917+ /* In case of BTRFS_FS_LOG_RECOVERING items won't have reserved space */
918+ if (total_reserved_size > 0) {
919+ /*
920+ * Check btrfs_delayed_item_reserve_metadata() to see why we
921+ * don't need to release/reserve qgroup space.
922+ */
923+ trace_btrfs_space_reservation(fs_info, "delayed_item",
924+ item->key.objectid, total_reserved_size,
925+ 0);
926+ btrfs_block_rsv_release(fs_info, &fs_info->delayed_block_rsv,
927+ total_reserved_size, NULL);
928+ }
846929
847− list_for_each_entry_safe(curr, next, &head, tree_list) {
848− btrfs_delayed_item_release_metadata(root, curr);
930+ list_for_each_entry_safe(curr, next, &batch_list, tree_list) {
849931 list_del(&curr->tree_list);
850932 btrfs_release_delayed_item(curr);
851933 }
852934
853−out:
854− return ret;
935+ return 0;
855936 }
856937
857938 static int btrfs_delete_delayed_items(struct btrfs_trans_handle *trans,
@@ -859,43 +940,52 @@ static int btrfs_delete_delayed_items(struct btrfs_trans_handle *trans, · 1 unmodified line
859940 struct btrfs_root *root,
860941 struct btrfs_delayed_node *node)
861942 {
862− struct btrfs_delayed_item *curr, *prev;
863943 int ret = 0;
864944
865−do_again:
866− mutex_lock(&node->mutex);
867− curr = __btrfs_first_delayed_deletion_item(node);
868− if (!curr)
869− goto delete_fail;
945+ while (ret == 0) {
946+ struct btrfs_delayed_item *item;
947+
948+ mutex_lock(&node->mutex);
949+ item = __btrfs_first_delayed_deletion_item(node);
950+ if (!item) {
951+ mutex_unlock(&node->mutex);
952+ break;
953+ }
954+
955+ ret = btrfs_search_slot(trans, root, &item->key, path, -1, 1);
956+ if (ret > 0) {
957+ /*
958+ * There's no matching item in the leaf. This means we
959+ * have already deleted this item in a past run of the
960+ * delayed items. We ignore errors when running delayed
961+ * items from an async context, through a work queue job
962+ * running btrfs_async_run_delayed_root(), and don't
963+ * release delayed items that failed to complete. This
964+ * is because we will retry later, and at transaction
965+ * commit time we always run delayed items and will
966+ * then deal with errors if they fail to run again.
967+ *
968+ * So just release delayed items for which we can't find
969+ * an item in the tree, and move to the next item.
970+ */
971+ btrfs_release_path(path);
972+ btrfs_release_delayed_item(item);
973+ ret = 0;
974+ } else if (ret == 0) {
975+ ret = btrfs_batch_delete_items(trans, root, path, item);
976+ btrfs_release_path(path);
977+ }
870978
871− ret = btrfs_search_slot(trans, root, &curr->key, path, -1, 1);
872− if (ret < 0)
873− goto delete_fail;
874− else if (ret > 0) {
875979 /*
876− * can't find the item which the node points to, so this node
877− * is invalid, just drop it.
980+ * We unlock and relock on each iteration, this is to prevent
981+ * blocking other tasks for too long while we are being run from
982+ * the async context (work queue job). Those tasks are typically
983+ * running system calls like creat/mkdir/rename/unlink/etc which
984+ * need to add delayed items to this delayed node.
878985 */
879− prev = curr;
880− curr = __btrfs_next_delayed_item(prev);
881− btrfs_release_delayed_item(prev);
882− ret = 0;
883− btrfs_release_path(path);
884− if (curr) {
885− mutex_unlock(&node->mutex);
886− goto do_again;
887− } else
888− goto delete_fail;
986+ mutex_unlock(&node->mutex);
889987 }
890988
891− btrfs_batch_delete_items(trans, root, path, curr);
892− btrfs_release_path(path);
893− mutex_unlock(&node->mutex);
894− goto do_again;
895−
896−delete_fail:
897− btrfs_release_path(path);
898− mutex_unlock(&node->mutex);
899989 return ret;
900990 }
901991
@@ -1354,9 +1444,13 @@ int btrfs_insert_delayed_dir_index(struct btrfs_trans_handle *trans, · 452 unmodified lines
13541444 struct btrfs_disk_key *disk_key, u8 type,
13551445 u64 index)
13561446 {
1447+ struct btrfs_fs_info *fs_info = trans->fs_info;
1448+ const unsigned int leaf_data_size = BTRFS_LEAF_DATA_SIZE(fs_info);
13571449 struct btrfs_delayed_node *delayed_node;
13581450 struct btrfs_delayed_item *delayed_item;
13591451 struct btrfs_dir_item *dir_item;
1452+ bool reserve_leaf_space;
1453+ u32 data_len;
13601454 int ret;
13611455
13621456 delayed_node = btrfs_get_or_create_delayed_node(dir);
@@ -1372,6 +1466,7 @@ int btrfs_insert_delayed_dir_index(struct btrfs_trans_handle *trans, · 9 unmodified lines
13721466 delayed_item->key.objectid = btrfs_ino(dir);
13731467 delayed_item->key.type = BTRFS_DIR_INDEX_KEY;
13741468 delayed_item->key.offset = index;
1469+ delayed_item->ins_or_del = BTRFS_DELAYED_INSERTION_ITEM;
13751470
13761471 dir_item = (struct btrfs_dir_item *)delayed_item->data;
13771472 dir_item->location = *disk_key;
@@ -1381,15 +1476,52 @@ int btrfs_insert_delayed_dir_index(struct btrfs_trans_handle *trans, · 3 unmodified lines
13811476 btrfs_set_stack_dir_type(dir_item, type);
13821477 memcpy((char *)(dir_item + 1), name, name_len);
13831478
1384− ret = btrfs_delayed_item_reserve_metadata(trans, dir->root, delayed_item);
1385− /*
1386− * we have reserved enough space when we start a new transaction,
1387− * so reserving metadata failure is impossible
1388− */
1389− BUG_ON(ret);
1479+ data_len = delayed_item->data_len + sizeof(struct btrfs_item);
13901480
13911481 mutex_lock(&delayed_node->mutex);
1392− ret = __btrfs_add_delayed_insertion_item(delayed_node, delayed_item);
1482+
1483+ if (delayed_node->index_item_leaves == 0 ||
1484+ delayed_node->curr_index_batch_size + data_len > leaf_data_size) {
1485+ delayed_node->curr_index_batch_size = data_len;
1486+ reserve_leaf_space = true;
1487+ } else {
1488+ delayed_node->curr_index_batch_size += data_len;
1489+ reserve_leaf_space = false;
1490+ }
1491+
1492+ if (reserve_leaf_space) {
1493+ ret = btrfs_delayed_item_reserve_metadata(trans, dir->root,
1494+ delayed_item);
1495+ /*
1496+ * Space was reserved for a dir index item insertion when we
1497+ * started the transaction, so getting a failure here should be
1498+ * impossible.
1499+ */
1500+ if (WARN_ON(ret)) {
1501+ mutex_unlock(&delayed_node->mutex);
1502+ btrfs_release_delayed_item(delayed_item);
1503+ goto release_node;
1504+ }
1505+
1506+ delayed_node->index_item_leaves++;
1507+ } else if (!test_bit(BTRFS_FS_LOG_RECOVERING, &fs_info->flags)) {
1508+ const u64 bytes = btrfs_calc_insert_metadata_size(fs_info, 1);
1509+
1510+ /*
1511+ * Adding the new dir index item does not require touching another
1512+ * leaf, so we can release 1 unit of metadata that was previously
1513+ * reserved when starting the transaction. This applies only to
1514+ * the case where we had a transaction start and excludes the
1515+ * transaction join case (when replaying log trees).
1516+ */
1517+ trace_btrfs_space_reservation(fs_info, "transaction",
1518+ trans->transid, bytes, 0);
1519+ btrfs_block_rsv_release(fs_info, trans->block_rsv, bytes, NULL);
1520+ ASSERT(trans->bytes_reserved >= bytes);
1521+ trans->bytes_reserved -= bytes;
1522+ }
1523+
1524+ ret = __btrfs_add_delayed_item(delayed_node, delayed_item);
13931525 if (unlikely(ret)) {
13941526 btrfs_err(trans->fs_info,
13951527 "err add delayed dir index item(name: %.*s) into the insertion tree of the delayed node(root id: %llu, inode id: %llu, errno: %d)",
@@ -1417,8 +1549,37 @@ static int btrfs_delete_delayed_insertion_item(struct btrfs_fs_info *fs_info, · 21 unmodified lines
14171549 return 1;
14181550 }
14191551
1420− btrfs_delayed_item_release_metadata(node->root, item);
1552+ /*
1553+ * For delayed items to insert, we track reserved metadata bytes based
1554+ * on the number of leaves that we will use.
1555+ * See btrfs_insert_delayed_dir_index() and
1556+ * btrfs_delayed_item_reserve_metadata()).
1557+ */
1558+ ASSERT(item->bytes_reserved == 0);
1559+ ASSERT(node->index_item_leaves > 0);
1560+
1561+ /*
1562+ * If there's only one leaf reserved, we can decrement this item from the
1563+ * current batch, otherwise we can not because we don't know which leaf
1564+ * it belongs to. With the current limit on delayed items, we rarely
1565+ * accumulate enough dir index items to fill more than one leaf (even
1566+ * when using a leaf size of 4K).
1567+ */
1568+ if (node->index_item_leaves == 1) {
1569+ const u32 data_len = item->data_len + sizeof(struct btrfs_item);
1570+
1571+ ASSERT(node->curr_index_batch_size >= data_len);
1572+ node->curr_index_batch_size -= data_len;
1573+ }
1574+
14211575 btrfs_release_delayed_item(item);
1576+
1577+ /* If we now have no more dir index items, we can release all leaves. */
1578+ if (RB_EMPTY_ROOT(&node->ins_root.rb_root)) {
1579+ btrfs_delayed_item_release_leaves(node, node->index_item_leaves);
1580+ node->index_item_leaves = 0;
1581+ }
1582+
14221583 mutex_unlock(&node->mutex);
14231584 return 0;
14241585 }
@@ -1451,6 +1612,7 @@ int btrfs_delete_delayed_dir_index(struct btrfs_trans_handle *trans, · 26 unmodified lines
14511612 }
14521613
14531614 item->key = item_key;
1615+ item->ins_or_del = BTRFS_DELAYED_DELETION_ITEM;
14541616
14551617 ret = btrfs_delayed_item_reserve_metadata(trans, dir->root, item);
14561618 /*
@@ -1465,7 +1627,7 @@ int btrfs_delete_delayed_dir_index(struct btrfs_trans_handle *trans, · 8 unmodified lines
14651627 }
14661628
14671629 mutex_lock(&node->mutex);
1468− ret = __btrfs_add_delayed_deletion_item(node, item);
1630+ ret = __btrfs_add_delayed_item(node, item);
14691631 if (unlikely(ret)) {
14701632 btrfs_err(trans->fs_info,
14711633 "err add delayed dir index item(index: %llu) into the deletion tree of the delayed node(root id: %llu, inode id: %llu, errno: %d)",
@@ -1833,12 +1995,17 @@ static void __btrfs_kill_delayed_node(struct btrfs_delayed_node *delayed_node) · 361 unmodified lines
18331995 mutex_lock(&delayed_node->mutex);
18341996 curr_item = __btrfs_first_delayed_insertion_item(delayed_node);
18351997 while (curr_item) {
1836− btrfs_delayed_item_release_metadata(root, curr_item);
18371998 prev_item = curr_item;
18381999 curr_item = __btrfs_next_delayed_item(prev_item);
18392000 btrfs_release_delayed_item(prev_item);
18402001 }
18412002
2003+ if (delayed_node->index_item_leaves > 0) {
2004+ btrfs_delayed_item_release_leaves(delayed_node,
2005+ delayed_node->index_item_leaves);
2006+ delayed_node->index_item_leaves = 0;
2007+ }
2008+
18422009 curr_item = __btrfs_first_delayed_deletion_item(delayed_node);
18432010 while (curr_item) {
18442011 btrfs_delayed_item_release_metadata(root, curr_item);
fs/btrfs/delayed-inode.h+11−0
@@ -58,6 +58,17 @@ struct btrfs_delayed_node { · 57 unmodified lines
5858 u64 index_cnt;
5959 unsigned long flags;
6060 int count;
61+ /*
62+ * The size of the next batch of dir index items to insert (if this
63+ * node is from a directory inode). Protected by @mutex.
64+ */
65+ u32 curr_index_batch_size;
66+ /*
67+ * Number of leaves reserved for inserting dir index items (if this
68+ * node belongs to a directory inode). This may be larger then the
69+ * actual number of leaves we end up using. Protected by @mutex.
70+ */
71+ u32 index_item_leaves;
6172 };
6273
6374 struct btrfs_delayed_item {
fs/btrfs/delayed-ref.c+2−2
@@ -132,7 +132,7 @@ void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans) · 131 unmodified lines
132132
133133 spin_lock(&delayed_rsv->lock);
134134 delayed_rsv->size += num_bytes;
135− delayed_rsv->full = 0;
135+ delayed_rsv->full = false;
136136 spin_unlock(&delayed_rsv->lock);
137137 trans->delayed_ref_updates = 0;
138138 }
@@ -175,7 +175,7 @@ void btrfs_migrate_to_delayed_refs_rsv(struct btrfs_fs_info *fs_info, · 36 unmodified lines
175175 if (num_bytes)
176176 delayed_refs_rsv->reserved += num_bytes;
177177 if (delayed_refs_rsv->reserved >= delayed_refs_rsv->size)
178− delayed_refs_rsv->full = 1;
178+ delayed_refs_rsv->full = true;
179179 spin_unlock(&delayed_refs_rsv->lock);
180180
181181 if (num_bytes)
fs/btrfs/dev-replace.c+2−1
@@ -587,7 +587,8 @@ bool btrfs_finish_block_group_to_copy(struct btrfs_device *srcdev, · 586 unmodified lines
587587 ASSERT(!IS_ERR(em));
588588 map = em->map_lookup;
589589
590− num_extents = cur_extent = 0;
590+ num_extents = 0;
591+ cur_extent = 0;
591592 for (i = 0; i < map->num_stripes; i++) {
592593 /* We have more device extent to copy */
593594 if (srcdev != map->stripes[i].dev)
fs/btrfs/disk-io.c+88−180
@@ -51,7 +51,6 @@ · 50 unmodified lines
5151 BTRFS_SUPER_FLAG_METADUMP |\
5252 BTRFS_SUPER_FLAG_METADUMP_V2)
5353
54−static void end_workqueue_fn(struct btrfs_work *work);
5554 static void btrfs_destroy_ordered_extents(struct btrfs_root *root);
5655 static int btrfs_destroy_delayed_refs(struct btrfs_transaction *trans,
5756 struct btrfs_fs_info *fs_info);
@@ -64,40 +63,6 @@ static int btrfs_destroy_pinned_extent(struct btrfs_fs_info *fs_info, · 6 unmodified lines
6463 static int btrfs_cleanup_transaction(struct btrfs_fs_info *fs_info);
6564 static void btrfs_error_commit_super(struct btrfs_fs_info *fs_info);
6665
67−/*
68− * btrfs_end_io_wq structs are used to do processing in task context when an IO
69− * is complete. This is used during reads to verify checksums, and it is used
70− * by writes to insert metadata for new file extents after IO is complete.
71− */
72−struct btrfs_end_io_wq {
73− struct bio *bio;
74− bio_end_io_t *end_io;
75− void *private;
76− struct btrfs_fs_info *info;
77− blk_status_t status;
78− enum btrfs_wq_endio_type metadata;
79− struct btrfs_work work;
80−};
81−
82−static struct kmem_cache *btrfs_end_io_wq_cache;
83−
84−int __init btrfs_end_io_wq_init(void)
85−{
86− btrfs_end_io_wq_cache = kmem_cache_create("btrfs_end_io_wq",
87− sizeof(struct btrfs_end_io_wq),
88− 0,
89− SLAB_MEM_SPREAD,
90− NULL);
91− if (!btrfs_end_io_wq_cache)
92− return -ENOMEM;
93− return 0;
94−}
95−
96−void __cold btrfs_end_io_wq_exit(void)
97−{
98− kmem_cache_destroy(btrfs_end_io_wq_cache);
99−}
100−
10166 static void btrfs_free_csum_hash(struct btrfs_fs_info *fs_info)
10267 {
10368 if (fs_info->csum_shash)
@@ -256,8 +221,8 @@ static int verify_parent_transid(struct extent_io_tree *io_tree, · 152 unmodified lines
256221 goto out;
257222 }
258223 btrfs_err_rl(eb->fs_info,
259− "parent transid verify failed on %llu wanted %llu found %llu",
260− eb->start,
224+"parent transid verify failed on logical %llu mirror %u wanted %llu found %llu",
225+ eb->start, eb->read_mirror,
261226 parent_transid, btrfs_header_generation(eb));
262227 ret = 1;
263228 clear_extent_buffer_uptodate(eb);
@@ -587,21 +552,23 @@ static int validate_extent_buffer(struct extent_buffer *eb) · 323 unmodified lines
587552
588553 found_start = btrfs_header_bytenr(eb);
589554 if (found_start != eb->start) {
590− btrfs_err_rl(fs_info, "bad tree block start, want %llu have %llu",
591− eb->start, found_start);
555+ btrfs_err_rl(fs_info,
556+ "bad tree block start, mirror %u want %llu have %llu",
557+ eb->read_mirror, eb->start, found_start);
592558 ret = -EIO;
593559 goto out;
594560 }
595561 if (check_tree_block_fsid(eb)) {
596− btrfs_err_rl(fs_info, "bad fsid on block %llu",
597− eb->start);
562+ btrfs_err_rl(fs_info, "bad fsid on logical %llu mirror %u",
563+ eb->start, eb->read_mirror);
598564 ret = -EIO;
599565 goto out;
600566 }
601567 found_level = btrfs_header_level(eb);
602568 if (found_level >= BTRFS_MAX_LEVEL) {
603− btrfs_err(fs_info, "bad tree block level %d on %llu",
604− (int)btrfs_header_level(eb), eb->start);
569+ btrfs_err(fs_info,
570+ "bad tree block level, mirror %u level %d on logical %llu",
571+ eb->read_mirror, btrfs_header_level(eb), eb->start);
605572 ret = -EIO;
606573 goto out;
607574 }
@@ -612,8 +579,8 @@ static int validate_extent_buffer(struct extent_buffer *eb) · 4 unmodified lines
612579
613580 if (memcmp(result, header_csum, csum_size) != 0) {
614581 btrfs_warn_rl(fs_info,
615− "checksum verify failed on %llu wanted " CSUM_FMT " found " CSUM_FMT " level %d",
616− eb->start,
582+"checksum verify failed on logical %llu mirror %u wanted " CSUM_FMT " found " CSUM_FMT " level %d",
583+ eb->start, eb->read_mirror,
617584 CSUM_FMT_VALUE(csum_size, header_csum),
618585 CSUM_FMT_VALUE(csum_size, result),
619586 btrfs_header_level(eb));
@@ -638,8 +605,8 @@ static int validate_extent_buffer(struct extent_buffer *eb) · 18 unmodified lines
638605 set_extent_buffer_uptodate(eb);
639606 else
640607 btrfs_err(fs_info,
641− "block=%llu read time tree block corruption detected",
642− eb->start);
608+ "read time tree block corruption detected on logical %llu mirror %u",
609+ eb->start, eb->read_mirror);
643610 out:
644611 return ret;
645612 }
@@ -740,58 +707,6 @@ err: · 94 unmodified lines
740707 return ret;
741708 }
742709
743−static void end_workqueue_bio(struct bio *bio)
744−{
745− struct btrfs_end_io_wq *end_io_wq = bio->bi_private;
746− struct btrfs_fs_info *fs_info;
747− struct btrfs_workqueue *wq;
748−
749− fs_info = end_io_wq->info;
750− end_io_wq->status = bio->bi_status;
751−
752− if (btrfs_op(bio) == BTRFS_MAP_WRITE) {
753− if (end_io_wq->metadata == BTRFS_WQ_ENDIO_METADATA)
754− wq = fs_info->endio_meta_write_workers;
755− else if (end_io_wq->metadata == BTRFS_WQ_ENDIO_FREE_SPACE)
756− wq = fs_info->endio_freespace_worker;
757− else if (end_io_wq->metadata == BTRFS_WQ_ENDIO_RAID56)
758− wq = fs_info->endio_raid56_workers;
759− else
760− wq = fs_info->endio_write_workers;
761− } else {
762− if (end_io_wq->metadata == BTRFS_WQ_ENDIO_RAID56)
763− wq = fs_info->endio_raid56_workers;
764− else if (end_io_wq->metadata)
765− wq = fs_info->endio_meta_workers;
766− else
767− wq = fs_info->endio_workers;
768− }
769−
770− btrfs_init_work(&end_io_wq->work, end_workqueue_fn, NULL, NULL);
771− btrfs_queue_work(wq, &end_io_wq->work);
772−}
773−
774−blk_status_t btrfs_bio_wq_end_io(struct btrfs_fs_info *info, struct bio *bio,
775− enum btrfs_wq_endio_type metadata)
776−{
777− struct btrfs_end_io_wq *end_io_wq;
778−
779− end_io_wq = kmem_cache_alloc(btrfs_end_io_wq_cache, GFP_NOFS);
780− if (!end_io_wq)
781− return BLK_STS_RESOURCE;
782−
783− end_io_wq->private = bio->bi_private;
784− end_io_wq->end_io = bio->bi_end_io;
785− end_io_wq->info = info;
786− end_io_wq->status = 0;
787− end_io_wq->bio = bio;
788− end_io_wq->metadata = metadata;
789−
790− bio->bi_private = end_io_wq;
791− bio->bi_end_io = end_workqueue_bio;
792− return 0;
793−}
794−
795710 static void run_one_async_start(struct btrfs_work *work)
796711 {
797712 struct async_submit_bio *async;
@@ -816,7 +731,6 @@ static void run_one_async_done(struct btrfs_work *work) · 18 unmodified lines
816731 {
817732 struct async_submit_bio *async;
818733 struct inode *inode;
819− blk_status_t ret;
820734
821735 async = container_of(work, struct async_submit_bio, work);
822736 inode = async->inode;
@@ -834,11 +748,7 @@ static void run_one_async_done(struct btrfs_work *work) · 11 unmodified lines
834748 * This changes nothing when cgroups aren't in use.
835749 */
836750 async->bio->bi_opf |= REQ_CGROUP_PUNT;
837− ret = btrfs_map_bio(btrfs_sb(inode->i_sb), async->bio, async->mirror_num);
838− if (ret) {
839− async->bio->bi_status = ret;
840− bio_endio(async->bio);
841− }
751+ btrfs_submit_bio(btrfs_sb(inode->i_sb), async->bio, async->mirror_num);
842752 }
843753
844754 static void run_one_async_free(struct btrfs_work *work)
@@ -849,16 +759,23 @@ static void run_one_async_free(struct btrfs_work *work) · 4 unmodified lines
849759 kfree(async);
850760 }
851761
852−blk_status_t btrfs_wq_submit_bio(struct inode *inode, struct bio *bio,
853− int mirror_num, u64 dio_file_offset,
854− extent_submit_bio_start_t *submit_bio_start)
762+/*
763+ * Submit bio to an async queue.
764+ *
765+ * Retrun:
766+ * - true if the work has been succesfuly submitted
767+ * - false in case of error
768+ */
769+bool btrfs_wq_submit_bio(struct inode *inode, struct bio *bio, int mirror_num,
770+ u64 dio_file_offset,
771+ extent_submit_bio_start_t *submit_bio_start)
855772 {
856773 struct btrfs_fs_info *fs_info = BTRFS_I(inode)->root->fs_info;
857774 struct async_submit_bio *async;
858775
859776 async = kmalloc(sizeof(*async), GFP_NOFS);
860777 if (!async)
861− return BLK_STS_RESOURCE;
778+ return false;
862779
863780 async->inode = inode;
864781 async->bio = bio;
@@ -876,7 +793,7 @@ blk_status_t btrfs_wq_submit_bio(struct inode *inode, struct bio *bio, · 11 unmodified lines
876793 btrfs_queue_work(fs_info->hipri_workers, &async->work);
877794 else
878795 btrfs_queue_work(fs_info->workers, &async->work);
879− return 0;
796+ return true;
880797 }
881798
882799 static blk_status_t btree_csum_one_bio(struct bio *bio)
@@ -902,7 +819,7 @@ static blk_status_t btree_submit_bio_start(struct inode *inode, struct bio *bio, · 19 unmodified lines
902819 {
903820 /*
904821 * when we're called for a write, we're already in the async
905− * submission context. Just jump into btrfs_map_bio
822+ * submission context. Just jump into btrfs_submit_bio.
906823 */
907824 return btree_csum_one_bio(bio);
908825 }
@@ -924,32 +841,29 @@ void btrfs_submit_metadata_bio(struct inode *inode, struct bio *bio, int mirror_ · 15 unmodified lines
924841 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
925842 blk_status_t ret;
926843
844+ bio->bi_opf |= REQ_META;
845+
927846 if (btrfs_op(bio) != BTRFS_MAP_WRITE) {
928− /*
929− * called for a read, do the setup so that checksum validation
930− * can happen in the async kernel threads
931− */
932− ret = btrfs_bio_wq_end_io(fs_info, bio,
933− BTRFS_WQ_ENDIO_METADATA);
934− if (!ret)
935− ret = btrfs_map_bio(fs_info, bio, mirror_num);
936− } else if (!should_async_write(fs_info, BTRFS_I(inode))) {
937− ret = btree_csum_one_bio(bio);
938− if (!ret)
939− ret = btrfs_map_bio(fs_info, bio, mirror_num);
940− } else {
941− /*
942− * kthread helpers are used to submit writes so that
943− * checksumming can happen in parallel across all CPUs
944− */
945− ret = btrfs_wq_submit_bio(inode, bio, mirror_num, 0,
946− btree_submit_bio_start);
847+ btrfs_submit_bio(fs_info, bio, mirror_num);
848+ return;
947849 }
948850
851+ /*
852+ * Kthread helpers are used to submit writes so that checksumming can
853+ * happen in parallel across all CPUs.
854+ */
855+ if (should_async_write(fs_info, BTRFS_I(inode)) &&
856+ btrfs_wq_submit_bio(inode, bio, mirror_num, 0, btree_submit_bio_start))
857+ return;
858+
859+ ret = btree_csum_one_bio(bio);
949860 if (ret) {
950861 bio->bi_status = ret;
951862 bio_endio(bio);
863+ return;
952864 }
865+
866+ btrfs_submit_bio(fs_info, bio, mirror_num);
953867 }
954868
955869 #ifdef CONFIG_MIGRATION
@@ -1870,7 +1784,7 @@ again: · 914 unmodified lines
18701784 fail:
18711785 /*
18721786 * If our caller provided us an anonymous device, then it's his
1873− * responsability to free it in case we fail. So we have to set our
1787+ * responsibility to free it in case we fail. So we have to set our
18741788 * root's anon_dev to 0 to avoid a double free, once by btrfs_put_root()
18751789 * and once again by our caller.
18761790 */
@@ -1953,25 +1867,6 @@ struct btrfs_root *btrfs_get_fs_root_commit_root(struct btrfs_fs_info *fs_info, · 76 unmodified lines
19531867 return root;
19541868 }
19551869
1956−/*
1957− * called by the kthread helper functions to finally call the bio end_io
1958− * functions. This is where read checksum verification actually happens
1959− */
1960−static void end_workqueue_fn(struct btrfs_work *work)
1961−{
1962− struct bio *bio;
1963− struct btrfs_end_io_wq *end_io_wq;
1964−
1965− end_io_wq = container_of(work, struct btrfs_end_io_wq, work);
1966− bio = end_io_wq->bio;
1967−
1968− bio->bi_status = end_io_wq->status;
1969− bio->bi_private = end_io_wq->private;
1970− bio->bi_end_io = end_io_wq->end_io;
1971− bio_endio(bio);
1972− kmem_cache_free(btrfs_end_io_wq_cache, end_io_wq);
1973−}
1974−
19751870 static int cleaner_kthread(void *arg)
19761871 {
19771872 struct btrfs_fs_info *fs_info = arg;
@@ -2278,10 +2173,14 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info) · 300 unmodified lines
22782173 btrfs_destroy_workqueue(fs_info->delalloc_workers);
22792174 btrfs_destroy_workqueue(fs_info->hipri_workers);
22802175 btrfs_destroy_workqueue(fs_info->workers);
2281− btrfs_destroy_workqueue(fs_info->endio_workers);
2282− btrfs_destroy_workqueue(fs_info->endio_raid56_workers);
2176+ if (fs_info->endio_workers)
2177+ destroy_workqueue(fs_info->endio_workers);
2178+ if (fs_info->endio_raid56_workers)
2179+ destroy_workqueue(fs_info->endio_raid56_workers);
22832180 if (fs_info->rmw_workers)
22842181 destroy_workqueue(fs_info->rmw_workers);
2182+ if (fs_info->compressed_write_workers)
2183+ destroy_workqueue(fs_info->compressed_write_workers);
22852184 btrfs_destroy_workqueue(fs_info->endio_write_workers);
22862185 btrfs_destroy_workqueue(fs_info->endio_freespace_worker);
22872186 btrfs_destroy_workqueue(fs_info->delayed_workers);
@@ -2295,8 +2194,8 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info) · 7 unmodified lines
22952194 * the queues used for metadata I/O, since tasks from those other work
22962195 * queues can do metadata I/O operations.
22972196 */
2298− btrfs_destroy_workqueue(fs_info->endio_meta_workers);
2299− btrfs_destroy_workqueue(fs_info->endio_meta_write_workers);
2197+ if (fs_info->endio_meta_workers)
2198+ destroy_workqueue(fs_info->endio_meta_workers);
23002199 }
23012200
23022201 static void free_root_extent_buffers(struct btrfs_root *root)
@@ -2426,7 +2325,9 @@ static void btrfs_init_btree_inode(struct btrfs_fs_info *fs_info) · 123 unmodified lines
24262325 extent_map_tree_init(&BTRFS_I(inode)->extent_tree);
24272326
24282327 BTRFS_I(inode)->root = btrfs_grab_root(fs_info->tree_root);
2429− memset(&BTRFS_I(inode)->location, 0, sizeof(struct btrfs_key));
2328+ BTRFS_I(inode)->location.objectid = BTRFS_BTREE_INODE_OBJECTID;
2329+ BTRFS_I(inode)->location.type = 0;
2330+ BTRFS_I(inode)->location.offset = 0;
24302331 set_bit(BTRFS_INODE_DUMMY, &BTRFS_I(inode)->runtime_flags);
24312332 btrfs_insert_inode_hash(inode);
24322333 }
@@ -2475,25 +2376,18 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info) · 42 unmodified lines
24752376 fs_info->fixup_workers =
24762377 btrfs_alloc_workqueue(fs_info, "fixup", flags, 1, 0);
24772378
2478− /*
2479− * endios are largely parallel and should have a very
2480− * low idle thresh
2481− */
24822379 fs_info->endio_workers =
2483− btrfs_alloc_workqueue(fs_info, "endio", flags, max_active, 4);
2380+ alloc_workqueue("btrfs-endio", flags, max_active);
24842381 fs_info->endio_meta_workers =
2485− btrfs_alloc_workqueue(fs_info, "endio-meta", flags,
2486− max_active, 4);
2487− fs_info->endio_meta_write_workers =
2488− btrfs_alloc_workqueue(fs_info, "endio-meta-write", flags,
2489− max_active, 2);
2382+ alloc_workqueue("btrfs-endio-meta", flags, max_active);
24902383 fs_info->endio_raid56_workers =
2491− btrfs_alloc_workqueue(fs_info, "endio-raid56", flags,
2492− max_active, 4);
2384+ alloc_workqueue("btrfs-endio-raid56", flags, max_active);
24932385 fs_info->rmw_workers = alloc_workqueue("btrfs-rmw", flags, max_active);
24942386 fs_info->endio_write_workers =
24952387 btrfs_alloc_workqueue(fs_info, "endio-write", flags,
24962388 max_active, 2);
2389+ fs_info->compressed_write_workers =
2390+ alloc_workqueue("btrfs-compressed-write", flags, max_active);
24972391 fs_info->endio_freespace_worker =
24982392 btrfs_alloc_workqueue(fs_info, "freespace-write", flags,
24992393 max_active, 0);
@@ -2508,7 +2402,7 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info) · 8 unmodified lines
25082402 if (!(fs_info->workers && fs_info->hipri_workers &&
25092403 fs_info->delalloc_workers && fs_info->flush_workers &&
25102404 fs_info->endio_workers && fs_info->endio_meta_workers &&
2511− fs_info->endio_meta_write_workers &&
2405+ fs_info->compressed_write_workers &&
25122406 fs_info->endio_write_workers && fs_info->endio_raid56_workers &&
25132407 fs_info->endio_freespace_worker && fs_info->rmw_workers &&
25142408 fs_info->caching_workers && fs_info->fixup_workers &&
@@ -2535,6 +2429,9 @@ static int btrfs_init_csum_hash(struct btrfs_fs_info *fs_info, u16 csum_type) · 20 unmodified lines
25352429
25362430 fs_info->csum_shash = csum_shash;
25372431
2432+ btrfs_info(fs_info, "using %s (%s) checksum algorithm",
2433+ btrfs_super_csum_name(csum_type),
2434+ crypto_shash_driver_name(csum_shash));
25382435 return 0;
25392436 }
25402437
@@ -3253,6 +3150,7 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info) · 712 unmodified lines
32533150 init_waitqueue_head(&fs_info->transaction_blocked_wait);
32543151 init_waitqueue_head(&fs_info->async_submit_wait);
32553152 init_waitqueue_head(&fs_info->delayed_iputs_wait);
3153+ init_waitqueue_head(&fs_info->zone_finish_wait);
32563154
32573155 /* Usable values until the real ones are cached from the superblock */
32583156 fs_info->nodesize = 4096;
@@ -3260,6 +3158,8 @@ void btrfs_init_fs_info(struct btrfs_fs_info *fs_info) · 1 unmodified line
32603158 fs_info->sectorsize_bits = ilog2(4096);
32613159 fs_info->stripesize = 4096;
32623160
3161+ fs_info->max_extent_size = BTRFS_MAX_EXTENT_SIZE;
3162+
32633163 spin_lock_init(&fs_info->swapfile_pins_lock);
32643164 fs_info->swapfile_pins = RB_ROOT;
32653165
@@ -3591,16 +3491,6 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device · 325 unmodified lines
35913491 */
35923492 fs_info->compress_type = BTRFS_COMPRESS_ZLIB;
35933493
3594− /*
3595− * Flag our filesystem as having big metadata blocks if they are bigger
3596− * than the page size.
3597− */
3598− if (btrfs_super_nodesize(disk_super) > PAGE_SIZE) {
3599− if (!(features & BTRFS_FEATURE_INCOMPAT_BIG_METADATA))
3600− btrfs_info(fs_info,
3601− "flagging fs with big metadata feature");
3602− features |= BTRFS_FEATURE_INCOMPAT_BIG_METADATA;
3603− }
36043494
36053495 /* Set up fs_info before parsing mount options */
36063496 nodesize = btrfs_super_nodesize(disk_super);
@@ -3638,8 +3528,12 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device · 31 unmodified lines
36383528 else if (fs_info->compress_type == BTRFS_COMPRESS_ZSTD)
36393529 features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD;
36403530
3641− if (features & BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA)
3642− btrfs_info(fs_info, "has skinny extents");
3531+ /*
3532+ * Flag our filesystem as having big metadata blocks if they are bigger
3533+ * than the page size.
3534+ */
3535+ if (btrfs_super_nodesize(disk_super) > PAGE_SIZE)
3536+ features |= BTRFS_FEATURE_INCOMPAT_BIG_METADATA;
36433537
36443538 /*
36453539 * mixed block groups end up with duplicate but slightly offset
@@ -3668,6 +3562,20 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device · 22 unmodified lines
36683562 err = -EINVAL;
36693563 goto fail_alloc;
36703564 }
3565+ /*
3566+ * We have unsupported RO compat features, although RO mounted, we
3567+ * should not cause any metadata write, including log replay.
3568+ * Or we could screw up whatever the new feature requires.
3569+ */
3570+ if (unlikely(features && btrfs_super_log_root(disk_super) &&
3571+ !btrfs_test_opt(fs_info, NOLOGREPLAY))) {
3572+ btrfs_err(fs_info,
3573+"cannot replay dirty log with unsupported compat_ro features (0x%llx), try rescue=nologreplay",
3574+ features);
3575+ err = -EINVAL;
3576+ goto fail_alloc;
3577+ }
3578+
36713579
36723580 if (sectorsize < PAGE_SIZE) {
36733581 struct btrfs_subpage_info *subpage_info;
fs/btrfs/disk-io.h+3−14
@@ -17,13 +17,6 @@ · 16 unmodified lines
1717 */
1818 #define BTRFS_BDEV_BLOCKSIZE (4096)
1919
20−enum btrfs_wq_endio_type {
21− BTRFS_WQ_ENDIO_DATA,
22− BTRFS_WQ_ENDIO_METADATA,
23− BTRFS_WQ_ENDIO_FREE_SPACE,
24− BTRFS_WQ_ENDIO_RAID56,
25−};
26−
2720 static inline u64 btrfs_sb_offset(int mirror)
2821 {
2922 u64 start = SZ_16K;
@@ -121,11 +114,9 @@ int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid, · 91 unmodified lines
121114 int atomic);
122115 int btrfs_read_extent_buffer(struct extent_buffer *buf, u64 parent_transid,
123116 int level, struct btrfs_key *first_key);
124−blk_status_t btrfs_bio_wq_end_io(struct btrfs_fs_info *info, struct bio *bio,
125− enum btrfs_wq_endio_type metadata);
126−blk_status_t btrfs_wq_submit_bio(struct inode *inode, struct bio *bio,
127− int mirror_num, u64 dio_file_offset,
128− extent_submit_bio_start_t *submit_bio_start);
117+bool btrfs_wq_submit_bio(struct inode *inode, struct bio *bio, int mirror_num,
118+ u64 dio_file_offset,
119+ extent_submit_bio_start_t *submit_bio_start);
129120 blk_status_t btrfs_submit_bio_done(void *private_data, struct bio *bio,
130121 int mirror_num);
131122 int btrfs_alloc_log_tree_node(struct btrfs_trans_handle *trans,
@@ -145,8 +136,6 @@ int btree_lock_page_hook(struct page *page, void *data, · 13 unmodified lines
145136 int btrfs_get_num_tolerated_disk_barrier_failures(u64 flags);
146137 int btrfs_get_free_objectid(struct btrfs_root *root, u64 *objectid);
147138 int btrfs_init_root_free_objectid(struct btrfs_root *root);
148−int __init btrfs_end_io_wq_init(void);
149−void __cold btrfs_end_io_wq_exit(void);
150139
151140 #ifdef CONFIG_DEBUG_LOCK_ALLOC
152141 void btrfs_set_buffer_lockdep_class(u64 objectid,
fs/btrfs/extent-tree.c+87−62
@@ -1269,7 +1269,7 @@ static int btrfs_issue_discard(struct block_device *bdev, u64 start, u64 len, · 1268 unmodified lines
12691269 return ret;
12701270 }
12711271
1272−static int do_discard_extent(struct btrfs_io_stripe *stripe, u64 *bytes)
1272+static int do_discard_extent(struct btrfs_discard_stripe *stripe, u64 *bytes)
12731273 {
12741274 struct btrfs_device *dev = stripe->dev;
12751275 struct btrfs_fs_info *fs_info = dev->fs_info;
@@ -1316,76 +1316,60 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, · 40 unmodified lines
13161316 u64 discarded_bytes = 0;
13171317 u64 end = bytenr + num_bytes;
13181318 u64 cur = bytenr;
1319− struct btrfs_io_context *bioc = NULL;
13201319
13211320 /*
1322− * Avoid races with device replace and make sure our bioc has devices
1323− * associated to its stripes that don't go away while we are discarding.
1321+ * Avoid races with device replace and make sure the devices in the
1322+ * stripes don't go away while we are discarding.
13241323 */
13251324 btrfs_bio_counter_inc_blocked(fs_info);
13261325 while (cur < end) {
1327− struct btrfs_io_stripe *stripe;
1326+ struct btrfs_discard_stripe *stripes;
1327+ unsigned int num_stripes;
13281328 int i;
13291329
13301330 num_bytes = end - cur;
1331− /* Tell the block device(s) that the sectors can be discarded */
1332− ret = btrfs_map_block(fs_info, BTRFS_MAP_DISCARD, cur,
1333− &num_bytes, &bioc, 0);
1334− /*
1335− * Error can be -ENOMEM, -ENOENT (no such chunk mapping) or
1336− * -EOPNOTSUPP. For any such error, @num_bytes is not updated,
1337− * thus we can't continue anyway.
1338− */
1339− if (ret < 0)
1340− goto out;
1331+ stripes = btrfs_map_discard(fs_info, cur, &num_bytes, &num_stripes);
1332+ if (IS_ERR(stripes)) {
1333+ ret = PTR_ERR(stripes);
1334+ if (ret == -EOPNOTSUPP)
1335+ ret = 0;
1336+ break;
1337+ }
13411338
1342− stripe = bioc->stripes;
1343− for (i = 0; i < bioc->num_stripes; i++, stripe++) {
1339+ for (i = 0; i < num_stripes; i++) {
1340+ struct btrfs_discard_stripe *stripe = stripes + i;
13441341 u64 bytes;
1345− struct btrfs_device *device = stripe->dev;
13461342
1347− if (!device->bdev) {
1343+ if (!stripe->dev->bdev) {
13481344 ASSERT(btrfs_test_opt(fs_info, DEGRADED));
13491345 continue;
13501346 }
13511347
1352− if (!test_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state))
1348+ if (!test_bit(BTRFS_DEV_STATE_WRITEABLE,
1349+ &stripe->dev->dev_state))
13531350 continue;
13541351
13551352 ret = do_discard_extent(stripe, &bytes);
1356− if (!ret) {
1357− discarded_bytes += bytes;
1358− } else if (ret != -EOPNOTSUPP) {
1353+ if (ret) {
13591354 /*
1360− * Logic errors or -ENOMEM, or -EIO, but
1361− * unlikely to happen.
1362− *
1363− * And since there are two loops, explicitly
1364− * go to out to avoid confusion.
1355+ * Keep going if discard is not supported by the
1356+ * device.
13651357 */
1366− btrfs_put_bioc(bioc);
1367− goto out;
1358+ if (ret != -EOPNOTSUPP)
1359+ break;
1360+ ret = 0;
1361+ } else {
1362+ discarded_bytes += bytes;
13681363 }
1369−
1370− /*
1371− * Just in case we get back EOPNOTSUPP for some reason,
1372− * just ignore the return value so we don't screw up
1373− * people calling discard_extent.
1374− */
1375− ret = 0;
13761364 }
1377− btrfs_put_bioc(bioc);
1365+ kfree(stripes);
1366+ if (ret)
1367+ break;
13781368 cur += num_bytes;
13791369 }
1380−out:
13811370 btrfs_bio_counter_dec(fs_info);
1382−
13831371 if (actual_bytes)
13841372 *actual_bytes = discarded_bytes;
1385−
1386−
1387− if (ret == -EOPNOTSUPP)
1388− ret = 0;
13891373 return ret;
13901374 }
13911375
@@ -3981,23 +3965,63 @@ static void found_extent(struct find_free_extent_ctl *ffe_ctl, · 2589 unmodified lines
39813965 }
39823966 }
39833967
3984−static bool can_allocate_chunk(struct btrfs_fs_info *fs_info,
3985− struct find_free_extent_ctl *ffe_ctl)
3968+static int can_allocate_chunk_zoned(struct btrfs_fs_info *fs_info,
3969+ struct find_free_extent_ctl *ffe_ctl)
3970+{
3971+ /* If we can activate new zone, just allocate a chunk and use it */
3972+ if (btrfs_can_activate_zone(fs_info->fs_devices, ffe_ctl->flags))
3973+ return 0;
3974+
3975+ /*
3976+ * We already reached the max active zones. Try to finish one block
3977+ * group to make a room for a new block group. This is only possible
3978+ * for a data block group because btrfs_zone_finish() may need to wait
3979+ * for a running transaction which can cause a deadlock for metadata
3980+ * allocation.
3981+ */
3982+ if (ffe_ctl->flags & BTRFS_BLOCK_GROUP_DATA) {
3983+ int ret = btrfs_zone_finish_one_bg(fs_info);
3984+
3985+ if (ret == 1)
3986+ return 0;
3987+ else if (ret < 0)
3988+ return ret;
3989+ }
3990+
3991+ /*
3992+ * If we have enough free space left in an already active block group
3993+ * and we can't activate any other zone now, do not allow allocating a
3994+ * new chunk and let find_free_extent() retry with a smaller size.
3995+ */
3996+ if (ffe_ctl->max_extent_size >= ffe_ctl->min_alloc_size)
3997+ return -ENOSPC;
3998+
3999+ /*
4000+ * Even min_alloc_size is not left in any block groups. Since we cannot
4001+ * activate a new block group, allocating it may not help. Let's tell a
4002+ * caller to try again and hope it progress something by writing some
4003+ * parts of the region. That is only possible for data block groups,
4004+ * where a part of the region can be written.
4005+ */
4006+ if (ffe_ctl->flags & BTRFS_BLOCK_GROUP_DATA)
4007+ return -EAGAIN;
4008+
4009+ /*
4010+ * We cannot activate a new block group and no enough space left in any
4011+ * block groups. So, allocating a new block group may not help. But,
4012+ * there is nothing to do anyway, so let's go with it.
4013+ */
4014+ return 0;
4015+}
4016+
4017+static int can_allocate_chunk(struct btrfs_fs_info *fs_info,
4018+ struct find_free_extent_ctl *ffe_ctl)
39864019 {
39874020 switch (ffe_ctl->policy) {
39884021 case BTRFS_EXTENT_ALLOC_CLUSTERED:
3989− return true;
4022+ return 0;
39904023 case BTRFS_EXTENT_ALLOC_ZONED:
3991− /*
3992− * If we have enough free space left in an already
3993− * active block group and we can't activate any other
3994− * zone now, do not allow allocating a new chunk and
3995− * let find_free_extent() retry with a smaller size.
3996− */
3997− if (ffe_ctl->max_extent_size >= ffe_ctl->min_alloc_size &&
3998− !btrfs_can_activate_zone(fs_info->fs_devices, ffe_ctl->flags))
3999− return false;
4000− return true;
4024+ return can_allocate_chunk_zoned(fs_info, ffe_ctl);
40014025 default:
40024026 BUG();
40034027 }
@@ -4079,8 +4103,9 @@ static int find_free_extent_update_loop(struct btrfs_fs_info *fs_info, · 75 unmodified lines
40794103 int exist = 0;
40804104
40814105 /*Check if allocation policy allows to create a new chunk */
4082− if (!can_allocate_chunk(fs_info, ffe_ctl))
4083− return -ENOSPC;
4106+ ret = can_allocate_chunk(fs_info, ffe_ctl);
4107+ if (ret)
4108+ return ret;
40844109
40854110 trans = current->journal_info;
40864111 if (trans)
@@ -5992,7 +6017,7 @@ int btrfs_error_unpin_extent_range(struct btrfs_fs_info *fs_info, · 1905 unmodified lines
59926017 */
59936018 static int btrfs_trim_free_extents(struct btrfs_device *device, u64 *trimmed)
59946019 {
5995− u64 start = SZ_1M, len = 0, end = 0;
6020+ u64 start = BTRFS_DEVICE_RANGE_RESERVED, len = 0, end = 0;
59966021 int ret;
59976022
59986023 *trimmed = 0;
@@ -6036,8 +6061,8 @@ static int btrfs_trim_free_extents(struct btrfs_device *device, u64 *trimmed) · 37 unmodified lines
60366061 break;
60376062 }
60386063
6039− /* Ensure we skip the reserved area in the first 1M */
6040− start = max_t(u64, start, SZ_1M);
6064+ /* Ensure we skip the reserved space on each device. */
6065+ start = max_t(u64, start, BTRFS_DEVICE_RANGE_RESERVED);
60416066
60426067 /*
60436068 * If find_first_clear_extent_bit find a range that spans the
fs/btrfs/extent_io.c+392−481
@@ -144,6 +144,7 @@ struct tree_entry { · 143 unmodified lines
144144 */
145145 struct btrfs_bio_ctrl {
146146 struct bio *bio;
147+ int mirror_num;
147148 enum btrfs_compression_type compress_type;
148149 u32 len_to_stripe_boundary;
149150 u32 len_to_oe_boundary;
@@ -178,61 +179,56 @@ static int add_extent_changeset(struct extent_state *state, u32 bits, · 28 unmodified lines
178179 return ret;
179180 }
180181
181−static void submit_one_bio(struct bio *bio, int mirror_num,
182− enum btrfs_compression_type compress_type)
182+static void submit_one_bio(struct btrfs_bio_ctrl *bio_ctrl)
183183 {
184− struct extent_io_tree *tree = bio->bi_private;
184+ struct bio *bio;
185+ struct bio_vec *bv;
186+ struct inode *inode;
187+ int mirror_num;
188+
189+ if (!bio_ctrl->bio)
190+ return;
185191
186− bio->bi_private = NULL;
192+ bio = bio_ctrl->bio;
193+ bv = bio_first_bvec_all(bio);
194+ inode = bv->bv_page->mapping->host;
195+ mirror_num = bio_ctrl->mirror_num;
187196
188197 /* Caller should ensure the bio has at least some range added */
189198 ASSERT(bio->bi_iter.bi_size);
190199
191− if (is_data_inode(tree->private_data))
192− btrfs_submit_data_bio(tree->private_data, bio, mirror_num,
193− compress_type);
194− else
195− btrfs_submit_metadata_bio(tree->private_data, bio, mirror_num);
196− /*
197− * Above submission hooks will handle the error by ending the bio,
198− * which will do the cleanup properly. So here we should not return
199− * any error, or the caller of submit_extent_page() will do cleanup
200− * again, causing problems.
201− */
202−}
200+ btrfs_bio(bio)->file_offset = page_offset(bv->bv_page) + bv->bv_offset;
203201
204−/* Cleanup unsubmitted bios */
205−static void end_write_bio(struct extent_page_data *epd, int ret)
206−{
207− struct bio *bio = epd->bio_ctrl.bio;
202+ if (!is_data_inode(inode))
203+ btrfs_submit_metadata_bio(inode, bio, mirror_num);
204+ else if (btrfs_op(bio) == BTRFS_MAP_WRITE)
205+ btrfs_submit_data_write_bio(inode, bio, mirror_num);
206+ else
207+ btrfs_submit_data_read_bio(inode, bio, mirror_num,
208+ bio_ctrl->compress_type);
208209
209− if (bio) {
210− bio->bi_status = errno_to_blk_status(ret);
211− bio_endio(bio);
212− epd->bio_ctrl.bio = NULL;
213− }
210+ /* The bio is owned by the bi_end_io handler now */
211+ bio_ctrl->bio = NULL;
214212 }
215213
216214 /*
217− * Submit bio from extent page data via submit_one_bio
218− *
219− * Return 0 if everything is OK.
220− * Return <0 for error.
215+ * Submit or fail the current bio in an extent_page_data structure.
221216 */
222−static void flush_write_bio(struct extent_page_data *epd)
217+static void submit_write_bio(struct extent_page_data *epd, int ret)
223218 {
224219 struct bio *bio = epd->bio_ctrl.bio;
225220
226− if (bio) {
227− submit_one_bio(bio, 0, 0);
228− /*
229− * Clean up of epd->bio is handled by its endio function.
230− * And endio is either triggered by successful bio execution
231− * or the error handler of submit bio hook.
232− * So at this point, no matter what happened, we don't need
233− * to clean up epd->bio.
234− */
221+ if (!bio)
222+ return;
223+
224+ if (ret) {
225+ ASSERT(ret < 0);
226+ bio->bi_status = errno_to_blk_status(ret);
227+ bio_endio(bio);
228+ /* The bio is owned by the bi_end_io handler now */
235229 epd->bio_ctrl.bio = NULL;
230+ } else {
231+ submit_one_bio(&epd->bio_ctrl);
236232 }
237233 }
238234
@@ -376,131 +372,121 @@ void free_extent_state(struct extent_state *state) · 137 unmodified lines
376372 }
377373 }
378374
379−static struct rb_node *tree_insert(struct rb_root *root,
380− struct rb_node *search_start,
381− u64 offset,
382− struct rb_node *node,
383− struct rb_node ***p_in,
384− struct rb_node **parent_in)
385−{
386− struct rb_node **p;
387− struct rb_node *parent = NULL;
388− struct tree_entry *entry;
389−
390− if (p_in && parent_in) {
391− p = *p_in;
392− parent = *parent_in;
393− goto do_insert;
394− }
395−
396− p = search_start ? &search_start : &root->rb_node;
397− while (*p) {
398− parent = *p;
399− entry = rb_entry(parent, struct tree_entry, rb_node);
400−
401− if (offset < entry->start)
402− p = &(*p)->rb_left;
403− else if (offset > entry->end)
404− p = &(*p)->rb_right;
405− else
406− return parent;
407− }
408−
409−do_insert:
410− rb_link_node(node, parent, p);
411− rb_insert_color(node, root);
412− return NULL;
413−}
414−
415375 /**
416376 * Search @tree for an entry that contains @offset. Such entry would have
417377 * entry->start <= offset && entry->end >= offset.
418378 *
419379 * @tree: the tree to search
420380 * @offset: offset that should fall within an entry in @tree
421− * @next_ret: pointer to the first entry whose range ends after @offset
422− * @prev_ret: pointer to the first entry whose range begins before @offset
423− * @p_ret: pointer where new node should be anchored (used when inserting an
381+ * @node_ret: pointer where new node should be anchored (used when inserting an
424382 * entry in the tree)
425383 * @parent_ret: points to entry which would have been the parent of the entry,
426384 * containing @offset
427385 *
428− * This function returns a pointer to the entry that contains @offset byte
429− * address. If no such entry exists, then NULL is returned and the other
430− * pointer arguments to the function are filled, otherwise the found entry is
431− * returned and other pointers are left untouched.
386+ * Return a pointer to the entry that contains @offset byte address and don't change
387+ * @node_ret and @parent_ret.
388+ *
389+ * If no such entry exists, return pointer to entry that ends before @offset
390+ * and fill parameters @node_ret and @parent_ret, ie. does not return NULL.
432391 */
433−static struct rb_node *__etree_search(struct extent_io_tree *tree, u64 offset,
434− struct rb_node **next_ret,
435− struct rb_node **prev_ret,
436− struct rb_node ***p_ret,
437− struct rb_node **parent_ret)
392+static inline struct rb_node *tree_search_for_insert(struct extent_io_tree *tree,
393+ u64 offset,
394+ struct rb_node ***node_ret,
395+ struct rb_node **parent_ret)
438396 {
439397 struct rb_root *root = &tree->state;
440− struct rb_node **n = &root->rb_node;
398+ struct rb_node **node = &root->rb_node;
441399 struct rb_node *prev = NULL;
442− struct rb_node *orig_prev = NULL;
443400 struct tree_entry *entry;
444− struct tree_entry *prev_entry = NULL;
445401
446− while (*n) {
447− prev = *n;
402+ while (*node) {
403+ prev = *node;
448404 entry = rb_entry(prev, struct tree_entry, rb_node);
449− prev_entry = entry;
450405
451406 if (offset < entry->start)
452− n = &(*n)->rb_left;
407+ node = &(*node)->rb_left;
453408 else if (offset > entry->end)
454− n = &(*n)->rb_right;
409+ node = &(*node)->rb_right;
455410 else
456− return *n;
411+ return *node;
457412 }
458413
459− if (p_ret)
460− *p_ret = n;
414+ if (node_ret)
415+ *node_ret = node;
461416 if (parent_ret)
462417 *parent_ret = prev;
463418
464− if (next_ret) {
465− orig_prev = prev;
466− while (prev && offset > prev_entry->end) {
467− prev = rb_next(prev);
468− prev_entry = rb_entry(prev, struct tree_entry, rb_node);
469− }
470− *next_ret = prev;
471− prev = orig_prev;
419+ /* Search neighbors until we find the first one past the end */
420+ while (prev && offset > entry->end) {
421+ prev = rb_next(prev);
422+ entry = rb_entry(prev, struct tree_entry, rb_node);
472423 }
473424
474− if (prev_ret) {
475− prev_entry = rb_entry(prev, struct tree_entry, rb_node);
476− while (prev && offset < prev_entry->start) {
477− prev = rb_prev(prev);
478− prev_entry = rb_entry(prev, struct tree_entry, rb_node);
479− }
480− *prev_ret = prev;
481− }
482− return NULL;
425+ return prev;
483426 }
484427
485−static inline struct rb_node *
486−tree_search_for_insert(struct extent_io_tree *tree,
487− u64 offset,
488− struct rb_node ***p_ret,
489− struct rb_node **parent_ret)
428+/*
429+ * Inexact rb-tree search, return the next entry if @offset is not found
430+ */
431+static inline struct rb_node *tree_search(struct extent_io_tree *tree, u64 offset)
490432 {
491− struct rb_node *next= NULL;
492− struct rb_node *ret;
493−
494− ret = __etree_search(tree, offset, &next, NULL, p_ret, parent_ret);
495− if (!ret)
496− return next;
497− return ret;
433+ return tree_search_for_insert(tree, offset, NULL, NULL);
498434 }
499435
500−static inline struct rb_node *tree_search(struct extent_io_tree *tree,
501− u64 offset)
436+/**
437+ * Search offset in the tree or fill neighbor rbtree node pointers.
438+ *
439+ * @tree: the tree to search
440+ * @offset: offset that should fall within an entry in @tree
441+ * @next_ret: pointer to the first entry whose range ends after @offset
442+ * @prev_ret: pointer to the first entry whose range begins before @offset
443+ *
444+ * Return a pointer to the entry that contains @offset byte address. If no
445+ * such entry exists, then return NULL and fill @prev_ret and @next_ret.
446+ * Otherwise return the found entry and other pointers are left untouched.
447+ */
448+static struct rb_node *tree_search_prev_next(struct extent_io_tree *tree,
449+ u64 offset,
450+ struct rb_node **prev_ret,
451+ struct rb_node **next_ret)
502452 {
503− return tree_search_for_insert(tree, offset, NULL, NULL);
453+ struct rb_root *root = &tree->state;
454+ struct rb_node **node = &root->rb_node;
455+ struct rb_node *prev = NULL;
456+ struct rb_node *orig_prev = NULL;
457+ struct tree_entry *entry;
458+
459+ ASSERT(prev_ret);
460+ ASSERT(next_ret);
461+
462+ while (*node) {
463+ prev = *node;
464+ entry = rb_entry(prev, struct tree_entry, rb_node);
465+
466+ if (offset < entry->start)
467+ node = &(*node)->rb_left;
468+ else if (offset > entry->end)
469+ node = &(*node)->rb_right;
470+ else
471+ return *node;
472+ }
473+
474+ orig_prev = prev;
475+ while (prev && offset > entry->end) {
476+ prev = rb_next(prev);
477+ entry = rb_entry(prev, struct tree_entry, rb_node);
478+ }
479+ *next_ret = prev;
480+ prev = orig_prev;
481+
482+ entry = rb_entry(prev, struct tree_entry, rb_node);
483+ while (prev && offset < entry->start) {
484+ prev = rb_prev(prev);
485+ entry = rb_entry(prev, struct tree_entry, rb_node);
486+ }
487+ *prev_ret = prev;
488+
489+ return NULL;
504490 }
505491
506492 /*
@@ -554,7 +540,7 @@ static void merge_state(struct extent_io_tree *tree, · 47 unmodified lines
554540 }
555541
556542 static void set_state_bits(struct extent_io_tree *tree,
557− struct extent_state *state, u32 *bits,
543+ struct extent_state *state, u32 bits,
558544 struct extent_changeset *changeset);
559545
560546 /*
@@ -568,36 +554,55 @@ static void set_state_bits(struct extent_io_tree *tree, · 7 unmodified lines
568554 * probably isn't what you want to call (see set/clear_extent_bit).
569555 */
570556 static int insert_state(struct extent_io_tree *tree,
571− struct extent_state *state, u64 start, u64 end,
572− struct rb_node ***p,
573− struct rb_node **parent,
574− u32 *bits, struct extent_changeset *changeset)
557+ struct extent_state *state,
558+ u32 bits, struct extent_changeset *changeset)
575559 {
576− struct rb_node *node;
577−
578− if (end < start) {
579− btrfs_err(tree->fs_info,
580− "insert state: end < start %llu %llu", end, start);
581− WARN_ON(1);
582− }
583− state->start = start;
584− state->end = end;
560+ struct rb_node **node;
561+ struct rb_node *parent;
562+ const u64 end = state->end;
585563
586564 set_state_bits(tree, state, bits, changeset);
587565
588− node = tree_insert(&tree->state, NULL, end, &state->rb_node, p, parent);
589− if (node) {
590− struct extent_state *found;
591− found = rb_entry(node, struct extent_state, rb_node);
592− btrfs_err(tree->fs_info,
593− "found node %llu %llu on insert of %llu %llu",
594− found->start, found->end, start, end);
595− return -EEXIST;
566+ node = &tree->state.rb_node;
567+ while (*node) {
568+ struct tree_entry *entry;
569+
570+ parent = *node;
571+ entry = rb_entry(parent, struct tree_entry, rb_node);
572+
573+ if (end < entry->start) {
574+ node = &(*node)->rb_left;
575+ } else if (end > entry->end) {
576+ node = &(*node)->rb_right;
577+ } else {
578+ btrfs_err(tree->fs_info,
579+ "found node %llu %llu on insert of %llu %llu",
580+ entry->start, entry->end, state->start, end);
581+ return -EEXIST;
582+ }
596583 }
584+
585+ rb_link_node(&state->rb_node, parent, node);
586+ rb_insert_color(&state->rb_node, &tree->state);
587+
597588 merge_state(tree, state);
598589 return 0;
599590 }
600591
592+/*
593+ * Insert state to @tree to the location given by @node and @parent.
594+ */
595+static void insert_state_fast(struct extent_io_tree *tree,
596+ struct extent_state *state, struct rb_node **node,
597+ struct rb_node *parent, unsigned bits,
598+ struct extent_changeset *changeset)
599+{
600+ set_state_bits(tree, state, bits, changeset);
601+ rb_link_node(&state->rb_node, parent, node);
602+ rb_insert_color(&state->rb_node, &tree->state);
603+ merge_state(tree, state);
604+}
605+
601606 /*
602607 * split a given extent state struct in two, inserting the preallocated
603608 * struct 'prealloc' as the newly created second half. 'split' indicates an
@@ -615,7 +620,8 @@ static int insert_state(struct extent_io_tree *tree, · 11 unmodified lines
615620 static int split_state(struct extent_io_tree *tree, struct extent_state *orig,
616621 struct extent_state *prealloc, u64 split)
617622 {
618− struct rb_node *node;
623+ struct rb_node *parent = NULL;
624+ struct rb_node **node;
619625
620626 if (tree->private_data && is_data_inode(tree->private_data))
621627 btrfs_split_delalloc_extent(tree->private_data, orig, split);
@@ -625,12 +631,27 @@ static int split_state(struct extent_io_tree *tree, struct extent_state *orig, · 3 unmodified lines
625631 prealloc->state = orig->state;
626632 orig->start = split;
627633
628− node = tree_insert(&tree->state, &orig->rb_node, prealloc->end,
629− &prealloc->rb_node, NULL, NULL);
630− if (node) {
631− free_extent_state(prealloc);
632− return -EEXIST;
634+ parent = &orig->rb_node;
635+ node = &parent;
636+ while (*node) {
637+ struct tree_entry *entry;
638+
639+ parent = *node;
640+ entry = rb_entry(parent, struct tree_entry, rb_node);
641+
642+ if (prealloc->end < entry->start) {
643+ node = &(*node)->rb_left;
644+ } else if (prealloc->end > entry->end) {
645+ node = &(*node)->rb_right;
646+ } else {
647+ free_extent_state(prealloc);
648+ return -EEXIST;
649+ }
633650 }
651+
652+ rb_link_node(&prealloc->rb_node, parent, node);
653+ rb_insert_color(&prealloc->rb_node, &tree->state);
654+
634655 return 0;
635656 }
636657
@@ -652,11 +673,11 @@ static struct extent_state *next_state(struct extent_state *state) · 15 unmodified lines
652673 */
653674 static struct extent_state *clear_state_bit(struct extent_io_tree *tree,
654675 struct extent_state *state,
655− u32 *bits, int wake,
676+ u32 bits, int wake,
656677 struct extent_changeset *changeset)
657678 {
658679 struct extent_state *next;
659− u32 bits_to_clear = *bits & ~EXTENT_CTLBITS;
680+ u32 bits_to_clear = bits & ~EXTENT_CTLBITS;
660681 int ret;
661682
662683 if ((bits_to_clear & EXTENT_DIRTY) && (state->state & EXTENT_DIRTY)) {
@@ -818,8 +839,7 @@ hit_next: · 155 unmodified lines
818839 if (err)
819840 goto out;
820841 if (state->end <= end) {
821− state = clear_state_bit(tree, state, &bits, wake,
822− changeset);
842+ state = clear_state_bit(tree, state, bits, wake, changeset);
823843 goto next;
824844 }
825845 goto search_again;
@@ -840,13 +860,13 @@ hit_next: · 14 unmodified lines
840860 if (wake)
841861 wake_up(&state->wq);
842862
843− clear_state_bit(tree, prealloc, &bits, wake, changeset);
863+ clear_state_bit(tree, prealloc, bits, wake, changeset);
844864
845865 prealloc = NULL;
846866 goto out;
847867 }
848868
849− state = clear_state_bit(tree, state, &bits, wake, changeset);
869+ state = clear_state_bit(tree, state, bits, wake, changeset);
850870 next:
851871 if (last_end == (u64)-1)
852872 goto out;
@@ -937,9 +957,9 @@ out: · 84 unmodified lines
937957
938958 static void set_state_bits(struct extent_io_tree *tree,
939959 struct extent_state *state,
940− u32 *bits, struct extent_changeset *changeset)
960+ u32 bits, struct extent_changeset *changeset)
941961 {
942− u32 bits_to_set = *bits & ~EXTENT_CTLBITS;
962+ u32 bits_to_set = bits & ~EXTENT_CTLBITS;
943963 int ret;
944964
945965 if (tree->private_data && is_data_inode(tree->private_data))
@@ -1033,11 +1053,9 @@ again: · 87 unmodified lines
10331053 if (!node) {
10341054 prealloc = alloc_extent_state_atomic(prealloc);
10351055 BUG_ON(!prealloc);
1036− err = insert_state(tree, prealloc, start, end,
1037− &p, &parent, &bits, changeset);
1038− if (err)
1039− extent_io_tree_panic(tree, err);
1040−
1056+ prealloc->start = start;
1057+ prealloc->end = end;
1058+ insert_state_fast(tree, prealloc, p, parent, bits, changeset);
10411059 cache_state(prealloc, cached_state);
10421060 prealloc = NULL;
10431061 goto out;
@@ -1060,7 +1078,7 @@ hit_next: · 16 unmodified lines
10601078 goto out;
10611079 }
10621080
1063− set_state_bits(tree, state, &bits, changeset);
1081+ set_state_bits(tree, state, bits, changeset);
10641082 cache_state(state, cached_state);
10651083 merge_state(tree, state);
10661084 if (last_end == (u64)-1)
@@ -1116,7 +1134,7 @@ hit_next: · 49 unmodified lines
11161134 if (err)
11171135 goto out;
11181136 if (state->end <= end) {
1119− set_state_bits(tree, state, &bits, changeset);
1137+ set_state_bits(tree, state, bits, changeset);
11201138 cache_state(state, cached_state);
11211139 merge_state(tree, state);
11221140 if (last_end == (u64)-1)
@@ -1150,8 +1168,9 @@ hit_next: · 27 unmodified lines
11501168 * Avoid to free 'prealloc' if it can be merged with
11511169 * the later extent.
11521170 */
1153− err = insert_state(tree, prealloc, start, this_end,
1154− NULL, NULL, &bits, changeset);
1171+ prealloc->start = start;
1172+ prealloc->end = this_end;
1173+ err = insert_state(tree, prealloc, bits, changeset);
11551174 if (err)
11561175 extent_io_tree_panic(tree, err);
11571176
@@ -1179,7 +1198,7 @@ hit_next: · 21 unmodified lines
11791198 if (err)
11801199 extent_io_tree_panic(tree, err);
11811200
1182− set_state_bits(tree, prealloc, &bits, changeset);
1201+ set_state_bits(tree, prealloc, bits, changeset);
11831202 cache_state(prealloc, cached_state);
11841203 merge_state(tree, prealloc);
11851204 prealloc = NULL;
@@ -1274,10 +1293,9 @@ again: · 88 unmodified lines
12741293 err = -ENOMEM;
12751294 goto out;
12761295 }
1277− err = insert_state(tree, prealloc, start, end,
1278− &p, &parent, &bits, NULL);
1279− if (err)
1280− extent_io_tree_panic(tree, err);
1296+ prealloc->start = start;
1297+ prealloc->end = end;
1298+ insert_state_fast(tree, prealloc, p, parent, bits, NULL);
12811299 cache_state(prealloc, cached_state);
12821300 prealloc = NULL;
12831301 goto out;
@@ -1294,9 +1312,9 @@ hit_next: · 10 unmodified lines
12941312 * Just lock what we found and keep going
12951313 */
12961314 if (state->start == start && state->end <= end) {
1297− set_state_bits(tree, state, &bits, NULL);
1315+ set_state_bits(tree, state, bits, NULL);
12981316 cache_state(state, cached_state);
1299− state = clear_state_bit(tree, state, &clear_bits, 0, NULL);
1317+ state = clear_state_bit(tree, state, clear_bits, 0, NULL);
13001318 if (last_end == (u64)-1)
13011319 goto out;
13021320 start = last_end + 1;
@@ -1335,10 +1353,9 @@ hit_next: · 32 unmodified lines
13351353 if (err)
13361354 goto out;
13371355 if (state->end <= end) {
1338− set_state_bits(tree, state, &bits, NULL);
1356+ set_state_bits(tree, state, bits, NULL);
13391357 cache_state(state, cached_state);
1340− state = clear_state_bit(tree, state, &clear_bits, 0,
1341− NULL);
1358+ state = clear_state_bit(tree, state, clear_bits, 0, NULL);
13421359 if (last_end == (u64)-1)
13431360 goto out;
13441361 start = last_end + 1;
@@ -1372,8 +1389,9 @@ hit_next: · 27 unmodified lines
13721389 * Avoid to free 'prealloc' if it can be merged with
13731390 * the later extent.
13741391 */
1375− err = insert_state(tree, prealloc, start, this_end,
1376− NULL, NULL, &bits, NULL);
1392+ prealloc->start = start;
1393+ prealloc->end = this_end;
1394+ err = insert_state(tree, prealloc, bits, NULL);
13771395 if (err)
13781396 extent_io_tree_panic(tree, err);
13791397 cache_state(prealloc, cached_state);
@@ -1398,9 +1416,9 @@ hit_next: · 18 unmodified lines
13981416 if (err)
13991417 extent_io_tree_panic(tree, err);
14001418
1401− set_state_bits(tree, prealloc, &bits, NULL);
1419+ set_state_bits(tree, prealloc, bits, NULL);
14021420 cache_state(prealloc, cached_state);
1403− clear_state_bit(tree, prealloc, &clear_bits, 0, NULL);
1421+ clear_state_bit(tree, prealloc, clear_bits, 0, NULL);
14041422 prealloc = NULL;
14051423 goto out;
14061424 }
@@ -1674,7 +1692,7 @@ void find_first_clear_extent_bit(struct extent_io_tree *tree, u64 start, · 267 unmodified lines
16741692
16751693 /* Find first extent with bits cleared */
16761694 while (1) {
1677− node = __etree_search(tree, start, &next, &prev, NULL, NULL);
1695+ node = tree_search_prev_next(tree, start, &prev, &next);
16781696 if (!node && !next && !prev) {
16791697 /*
16801698 * Tree is completely empty, send full range and let
@@ -2007,10 +2025,12 @@ noinline_for_stack bool find_lock_delalloc_range(struct inode *inode, · 326 unmodified lines
20072025 struct page *locked_page, u64 *start,
20082026 u64 *end)
20092027 {
2028+ struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
20102029 struct extent_io_tree *tree = &BTRFS_I(inode)->io_tree;
20112030 const u64 orig_start = *start;
20122031 const u64 orig_end = *end;
2013− u64 max_bytes = BTRFS_MAX_EXTENT_SIZE;
2032+ /* The sanity tests may not set a valid fs_info. */
2033+ u64 max_bytes = fs_info ? fs_info->max_extent_size : BTRFS_MAX_EXTENT_SIZE;
20142034 u64 delalloc_start;
20152035 u64 delalloc_end;
20162036 bool found;
@@ -2418,6 +2438,20 @@ int btrfs_repair_eb_io_failure(const struct extent_buffer *eb, int mirror_num) · 401 unmodified lines
24182438 return ret;
24192439 }
24202440
2441+static int next_mirror(const struct io_failure_record *failrec, int cur_mirror)
2442+{
2443+ if (cur_mirror == failrec->num_copies)
2444+ return cur_mirror + 1 - failrec->num_copies;
2445+ return cur_mirror + 1;
2446+}
2447+
2448+static int prev_mirror(const struct io_failure_record *failrec, int cur_mirror)
2449+{
2450+ if (cur_mirror == 1)
2451+ return failrec->num_copies;
2452+ return cur_mirror - 1;
2453+}
2454+
24212455 /*
24222456 * each time an IO finishes, we do a fast check in the IO failure tree
24232457 * to see if we need to process or clean up an io_failure_record
@@ -2430,7 +2464,7 @@ int clean_io_failure(struct btrfs_fs_info *fs_info, · 6 unmodified lines
24302464 u64 private;
24312465 struct io_failure_record *failrec;
24322466 struct extent_state *state;
2433− int num_copies;
2467+ int mirror;
24342468 int ret;
24352469
24362470 private = 0;
@@ -2454,20 +2488,19 @@ int clean_io_failure(struct btrfs_fs_info *fs_info, · 17 unmodified lines
24542488 EXTENT_LOCKED);
24552489 spin_unlock(&io_tree->lock);
24562490
2457− if (state && state->start <= failrec->start &&
2458− state->end >= failrec->start + failrec->len - 1) {
2459− num_copies = btrfs_num_copies(fs_info, failrec->logical,
2460− failrec->len);
2461− if (num_copies > 1) {
2462− repair_io_failure(fs_info, ino, start, failrec->len,
2463− failrec->logical, page, pg_offset,
2464− failrec->failed_mirror);
2465− }
2466− }
2491+ if (!state || state->start > failrec->start ||
2492+ state->end < failrec->start + failrec->len - 1)
2493+ goto out;
2494+
2495+ mirror = failrec->this_mirror;
2496+ do {
2497+ mirror = prev_mirror(failrec, mirror);
2498+ repair_io_failure(fs_info, ino, start, failrec->len,
2499+ failrec->logical, page, pg_offset, mirror);
2500+ } while (mirror != failrec->failed_mirror);
24672501
24682502 out:
24692503 free_io_failure(failure_tree, io_tree, failrec);
2470−
24712504 return 0;
24722505 }
24732506
@@ -2506,17 +2539,16 @@ void btrfs_free_io_failure_record(struct btrfs_inode *inode, u64 start, u64 end) · 32 unmodified lines
25062539 }
25072540
25082541 static struct io_failure_record *btrfs_get_io_failure_record(struct inode *inode,
2509− u64 start)
2542+ struct btrfs_bio *bbio,
2543+ unsigned int bio_offset)
25102544 {
25112545 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
2546+ u64 start = bbio->file_offset + bio_offset;
25122547 struct io_failure_record *failrec;
2513− struct extent_map *em;
25142548 struct extent_io_tree *failure_tree = &BTRFS_I(inode)->io_failure_tree;
25152549 struct extent_io_tree *tree = &BTRFS_I(inode)->io_tree;
2516− struct extent_map_tree *em_tree = &BTRFS_I(inode)->extent_tree;
25172550 const u32 sectorsize = fs_info->sectorsize;
25182551 int ret;
2519− u64 logical;
25202552
25212553 failrec = get_state_failrec(failure_tree, start);
25222554 if (!IS_ERR(failrec)) {
@@ -2528,7 +2560,8 @@ static struct io_failure_record *btrfs_get_io_failure_record(struct inode *inode · 5 unmodified lines
25282560 * (e.g. with a list for failed_mirror) to make
25292561 * clean_io_failure() clean all those errors at once.
25302562 */
2531−
2563+ ASSERT(failrec->this_mirror == bbio->mirror_num);
2564+ ASSERT(failrec->len == fs_info->sectorsize);
25322565 return failrec;
25332566 }
25342567
@@ -2538,41 +2571,28 @@ static struct io_failure_record *btrfs_get_io_failure_record(struct inode *inode · 3 unmodified lines
25382571
25392572 failrec->start = start;
25402573 failrec->len = sectorsize;
2541− failrec->this_mirror = 0;
2542− failrec->compress_type = BTRFS_COMPRESS_NONE;
2574+ failrec->failed_mirror = bbio->mirror_num;
2575+ failrec->this_mirror = bbio->mirror_num;
2576+ failrec->logical = (bbio->iter.bi_sector << SECTOR_SHIFT) + bio_offset;
25432577
2544− read_lock(&em_tree->lock);
2545− em = lookup_extent_mapping(em_tree, start, failrec->len);
2546− if (!em) {
2547− read_unlock(&em_tree->lock);
2548− kfree(failrec);
2549− return ERR_PTR(-EIO);
2550− }
2578+ btrfs_debug(fs_info,
2579+ "new io failure record logical %llu start %llu",
2580+ failrec->logical, start);
25512581
2552− if (em->start > start || em->start + em->len <= start) {
2553− free_extent_map(em);
2554− em = NULL;
2555− }
2556− read_unlock(&em_tree->lock);
2557− if (!em) {
2582+ failrec->num_copies = btrfs_num_copies(fs_info, failrec->logical, sectorsize);
2583+ if (failrec->num_copies == 1) {
2584+ /*
2585+ * We only have a single copy of the data, so don't bother with
2586+ * all the retry and error correction code that follows. No
2587+ * matter what the error is, it is very likely to persist.
2588+ */
2589+ btrfs_debug(fs_info,
2590+ "cannot repair logical %llu num_copies %d",
2591+ failrec->logical, failrec->num_copies);
25582592 kfree(failrec);
25592593 return ERR_PTR(-EIO);
25602594 }
25612595
2562− logical = start - em->start;
2563− logical = em->block_start + logical;
2564− if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags)) {
2565− logical = em->block_start;
2566− failrec->compress_type = em->compress_type;
2567− }
2568−
2569− btrfs_debug(fs_info,
2570− "Get IO Failure Record: (new) logical=%llu, start=%llu, len=%llu",
2571− logical, start, failrec->len);
2572−
2573− failrec->logical = logical;
2574− free_extent_map(em);
2575−
25762596 /* Set the bits in the private failure tree */
25772597 ret = set_extent_bits(failure_tree, start, start + sectorsize - 1,
25782598 EXTENT_LOCKED | EXTENT_DIRTY);
@@ -2589,65 +2609,16 @@ static struct io_failure_record *btrfs_get_io_failure_record(struct inode *inode · 10 unmodified lines
25892609 return failrec;
25902610 }
25912611
2592−static bool btrfs_check_repairable(struct inode *inode,
2593− struct io_failure_record *failrec,
2594− int failed_mirror)
2595−{
2596− struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
2597− int num_copies;
2598−
2599− num_copies = btrfs_num_copies(fs_info, failrec->logical, failrec->len);
2600− if (num_copies == 1) {
2601− /*
2602− * we only have a single copy of the data, so don't bother with
2603− * all the retry and error correction code that follows. no
2604− * matter what the error is, it is very likely to persist.
2605− */
2606− btrfs_debug(fs_info,
2607− "Check Repairable: cannot repair, num_copies=%d, next_mirror %d, failed_mirror %d",
2608− num_copies, failrec->this_mirror, failed_mirror);
2609− return false;
2610− }
2611−
2612− /* The failure record should only contain one sector */
2613− ASSERT(failrec->len == fs_info->sectorsize);
2614−
2615− /*
2616− * There are two premises:
2617− * a) deliver good data to the caller
2618− * b) correct the bad sectors on disk
2619− *
2620− * Since we're only doing repair for one sector, we only need to get
2621− * a good copy of the failed sector and if we succeed, we have setup
2622− * everything for repair_io_failure to do the rest for us.
2623− */
2624− ASSERT(failed_mirror);
2625− failrec->failed_mirror = failed_mirror;
2626− failrec->this_mirror++;
2627− if (failrec->this_mirror == failed_mirror)
2628− failrec->this_mirror++;
2629−
2630− if (failrec->this_mirror > num_copies) {
2631− btrfs_debug(fs_info,
2632− "Check Repairable: (fail) num_copies=%d, next_mirror %d, failed_mirror %d",
2633− num_copies, failrec->this_mirror, failed_mirror);
2634− return false;
2635− }
2636−
2637− return true;
2638−}
2639−
2640−int btrfs_repair_one_sector(struct inode *inode,
2641− struct bio *failed_bio, u32 bio_offset,
2642− struct page *page, unsigned int pgoff,
2643− u64 start, int failed_mirror,
2612+int btrfs_repair_one_sector(struct inode *inode, struct btrfs_bio *failed_bbio,
2613+ u32 bio_offset, struct page *page, unsigned int pgoff,
26442614 submit_bio_hook_t *submit_bio_hook)
26452615 {
2616+ u64 start = failed_bbio->file_offset + bio_offset;
26462617 struct io_failure_record *failrec;
26472618 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
26482619 struct extent_io_tree *tree = &BTRFS_I(inode)->io_tree;
26492620 struct extent_io_tree *failure_tree = &BTRFS_I(inode)->io_failure_tree;
2650− struct btrfs_bio *failed_bbio = btrfs_bio(failed_bio);
2621+ struct bio *failed_bio = &failed_bbio->bio;
26512622 const int icsum = bio_offset >> fs_info->sectorsize_bits;
26522623 struct bio *repair_bio;
26532624 struct btrfs_bio *repair_bbio;
@@ -2657,12 +2628,24 @@ int btrfs_repair_one_sector(struct inode *inode, · 3 unmodified lines
26572628
26582629 BUG_ON(bio_op(failed_bio) == REQ_OP_WRITE);
26592630
2660− failrec = btrfs_get_io_failure_record(inode, start);
2631+ failrec = btrfs_get_io_failure_record(inode, failed_bbio, bio_offset);
26612632 if (IS_ERR(failrec))
26622633 return PTR_ERR(failrec);
26632634
2664−
2665− if (!btrfs_check_repairable(inode, failrec, failed_mirror)) {
2635+ /*
2636+ * There are two premises:
2637+ * a) deliver good data to the caller
2638+ * b) correct the bad sectors on disk
2639+ *
2640+ * Since we're only doing repair for one sector, we only need to get
2641+ * a good copy of the failed sector and if we succeed, we have setup
2642+ * everything for repair_io_failure to do the rest for us.
2643+ */
2644+ failrec->this_mirror = next_mirror(failrec, failrec->this_mirror);
2645+ if (failrec->this_mirror == failrec->failed_mirror) {
2646+ btrfs_debug(fs_info,
2647+ "failed to repair num_copies %d this_mirror %d failed_mirror %d",
2648+ failrec->num_copies, failrec->this_mirror, failrec->failed_mirror);
26662649 free_io_failure(failure_tree, tree, failrec);
26672650 return -EIO;
26682651 }
@@ -2695,7 +2678,7 @@ int btrfs_repair_one_sector(struct inode *inode, · 26 unmodified lines
26952678 * will be handled by the endio on the repair_bio, so we can't return an
26962679 * error here.
26972680 */
2698− submit_bio_hook(inode, repair_bio, failrec->this_mirror, failrec->compress_type);
2681+ submit_bio_hook(inode, repair_bio, failrec->this_mirror, 0);
26992682 return BLK_STS_OK;
27002683 }
27012684
@@ -2727,21 +2710,35 @@ static void end_page_read(struct page *page, bool uptodate, u64 start, u32 len) · 25 unmodified lines
27272710 btrfs_subpage_end_reader(fs_info, page, start, len);
27282711 }
27292712
2730−static blk_status_t submit_data_read_repair(struct inode *inode,
2731− struct bio *failed_bio,
2732− u32 bio_offset, struct page *page,
2733− unsigned int pgoff,
2734− u64 start, u64 end,
2735− int failed_mirror,
2736− unsigned int error_bitmap)
2713+static void end_sector_io(struct page *page, u64 offset, bool uptodate)
2714+{
2715+ struct btrfs_inode *inode = BTRFS_I(page->mapping->host);
2716+ const u32 sectorsize = inode->root->fs_info->sectorsize;
2717+ struct extent_state *cached = NULL;
2718+
2719+ end_page_read(page, uptodate, offset, sectorsize);
2720+ if (uptodate)
2721+ set_extent_uptodate(&inode->io_tree, offset,
2722+ offset + sectorsize - 1, &cached, GFP_ATOMIC);
2723+ unlock_extent_cached_atomic(&inode->io_tree, offset,
2724+ offset + sectorsize - 1, &cached);
2725+}
2726+
2727+static void submit_data_read_repair(struct inode *inode,
2728+ struct btrfs_bio *failed_bbio,
2729+ u32 bio_offset, const struct bio_vec *bvec,
2730+ unsigned int error_bitmap)
27372731 {
2732+ const unsigned int pgoff = bvec->bv_offset;
27382733 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
2734+ struct page *page = bvec->bv_page;
2735+ const u64 start = page_offset(bvec->bv_page) + bvec->bv_offset;
2736+ const u64 end = start + bvec->bv_len - 1;
27392737 const u32 sectorsize = fs_info->sectorsize;
27402738 const int nr_bits = (end + 1 - start) >> fs_info->sectorsize_bits;
2741− int error = 0;
27422739 int i;
27432740
2744− BUG_ON(bio_op(failed_bio) == REQ_OP_WRITE);
2741+ BUG_ON(bio_op(&failed_bbio->bio) == REQ_OP_WRITE);
27452742
27462743 /* This repair is only for data */
27472744 ASSERT(is_data_inode(inode));
@@ -2753,12 +2750,11 @@ static blk_status_t submit_data_read_repair(struct inode *inode, · 5 unmodified lines
27532750 * We only get called on buffered IO, thus page must be mapped and bio
27542751 * must not be cloned.
27552752 */
2756− ASSERT(page->mapping && !bio_flagged(failed_bio, BIO_CLONED));
2753+ ASSERT(page->mapping && !bio_flagged(&failed_bbio->bio, BIO_CLONED));
27572754
27582755 /* Iterate through all the sectors in the range */
27592756 for (i = 0; i < nr_bits; i++) {
27602757 const unsigned int offset = i * sectorsize;
2761− struct extent_state *cached = NULL;
27622758 bool uptodate = false;
27632759 int ret;
27642760
@@ -2771,10 +2767,9 @@ static blk_status_t submit_data_read_repair(struct inode *inode, · 6 unmodified lines
27712767 goto next;
27722768 }
27732769
2774− ret = btrfs_repair_one_sector(inode, failed_bio,
2775− bio_offset + offset,
2776− page, pgoff + offset, start + offset,
2777− failed_mirror, btrfs_submit_data_bio);
2770+ ret = btrfs_repair_one_sector(inode, failed_bbio,
2771+ bio_offset + offset, page, pgoff + offset,
2772+ btrfs_submit_data_read_bio);
27782773 if (!ret) {
27792774 /*
27802775 * We have submitted the read repair, the page release
@@ -2785,24 +2780,12 @@ static blk_status_t submit_data_read_repair(struct inode *inode, · 4 unmodified lines
27852780 continue;
27862781 }
27872782 /*
2788− * Repair failed, just record the error but still continue.
2789− * Or the remaining sectors will not be properly unlocked.
2783+ * Continue on failed repair, otherwise the remaining sectors
2784+ * will not be properly unlocked.
27902785 */
2791− if (!error)
2792− error = ret;
27932786 next:
2794− end_page_read(page, uptodate, start + offset, sectorsize);
2795− if (uptodate)
2796− set_extent_uptodate(&BTRFS_I(inode)->io_tree,
2797− start + offset,
2798− start + offset + sectorsize - 1,
2799− &cached, GFP_ATOMIC);
2800− unlock_extent_cached_atomic(&BTRFS_I(inode)->io_tree,
2801− start + offset,
2802− start + offset + sectorsize - 1,
2803− &cached);
2787+ end_sector_io(page, start + offset, uptodate);
28042788 }
2805− return errno_to_blk_status(error);
28062789 }
28072790
28082791 /* lots and lots of room for performance fixes in the end_bio funcs */
@@ -3017,7 +3000,6 @@ static void end_bio_extent_readpage(struct bio *bio) · 208 unmodified lines
30173000 */
30183001 u32 bio_offset = 0;
30193002 int mirror;
3020− int ret;
30213003 struct bvec_iter_all iter_all;
30223004
30233005 ASSERT(!bio_flagged(bio, BIO_CLONED));
@@ -3028,6 +3010,7 @@ static void end_bio_extent_readpage(struct bio *bio) · 4 unmodified lines
30283010 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
30293011 const u32 sectorsize = fs_info->sectorsize;
30303012 unsigned int error_bitmap = (unsigned int)-1;
3013+ bool repair = false;
30313014 u64 start;
30323015 u64 end;
30333016 u32 len;
@@ -3065,57 +3048,23 @@ static void end_bio_extent_readpage(struct bio *bio) · 31 unmodified lines
30653048 if (is_data_inode(inode)) {
30663049 error_bitmap = btrfs_verify_data_csum(bbio,
30673050 bio_offset, page, start, end);
3068− ret = error_bitmap;
3051+ if (error_bitmap)
3052+ uptodate = false;
30693053 } else {
3070− ret = btrfs_validate_metadata_buffer(bbio,
3071− page, start, end, mirror);
3054+ if (btrfs_validate_metadata_buffer(bbio,
3055+ page, start, end, mirror))
3056+ uptodate = false;
30723057 }
3073− if (ret)
3074− uptodate = false;
3075− else
3076− clean_io_failure(BTRFS_I(inode)->root->fs_info,
3077− failure_tree, tree, start,
3078− page,
3079− btrfs_ino(BTRFS_I(inode)), 0);
30803058 }
30813059
3082− if (likely(uptodate))
3083− goto readpage_ok;
3084−
3085− if (is_data_inode(inode)) {
3086− /*
3087− * If we failed to submit the IO at all we'll have a
3088− * mirror_num == 0, in which case we need to just mark
3089− * the page with an error and unlock it and carry on.
3090− */
3091− if (mirror == 0)
3092− goto readpage_ok;
3093−
3094− /*
3095− * submit_data_read_repair() will handle all the good
3096− * and bad sectors, we just continue to the next bvec.
3097− */
3098− submit_data_read_repair(inode, bio, bio_offset, page,
3099− start - page_offset(page),
3100− start, end, mirror,
3101− error_bitmap);
3102−
3103− ASSERT(bio_offset + len > bio_offset);
3104− bio_offset += len;
3105− continue;
3106− } else {
3107− struct extent_buffer *eb;
3108−
3109− eb = find_extent_buffer_readpage(fs_info, page, start);
3110− set_bit(EXTENT_BUFFER_READ_ERR, &eb->bflags);
3111− eb->read_mirror = mirror;
3112− atomic_dec(&eb->io_pages);
3113− }
3114−readpage_ok:
31153060 if (likely(uptodate)) {
31163061 loff_t i_size = i_size_read(inode);
31173062 pgoff_t end_index = i_size >> PAGE_SHIFT;
31183063
3064+ clean_io_failure(BTRFS_I(inode)->root->fs_info,
3065+ failure_tree, tree, start, page,
3066+ btrfs_ino(BTRFS_I(inode)), 0);
3067+
31193068 /*
31203069 * Zero out the remaining part if this range straddles
31213070 * i_size.
@@ -3132,14 +3081,44 @@ readpage_ok: · 10 unmodified lines
31323081 zero_user_segment(page, zero_start,
31333082 offset_in_page(end) + 1);
31343083 }
3084+ } else if (is_data_inode(inode)) {
3085+ /*
3086+ * Only try to repair bios that actually made it to a
3087+ * device. If the bio failed to be submitted mirror
3088+ * is 0 and we need to fail it without retrying.
3089+ *
3090+ * This also includes the high level bios for compressed
3091+ * extents - these never make it to a device and repair
3092+ * is already handled on the lower compressed bio.
3093+ */
3094+ if (mirror > 0)
3095+ repair = true;
3096+ } else {
3097+ struct extent_buffer *eb;
3098+
3099+ eb = find_extent_buffer_readpage(fs_info, page, start);
3100+ set_bit(EXTENT_BUFFER_READ_ERR, &eb->bflags);
3101+ eb->read_mirror = mirror;
3102+ atomic_dec(&eb->io_pages);
3103+ }
3104+
3105+ if (repair) {
3106+ /*
3107+ * submit_data_read_repair() will handle all the good
3108+ * and bad sectors, we just continue to the next bvec.
3109+ */
3110+ submit_data_read_repair(inode, bbio, bio_offset, bvec,
3111+ error_bitmap);
3112+ } else {
3113+ /* Update page status and unlock */
3114+ end_page_read(page, uptodate, start, len);
3115+ endio_readpage_release_extent(&processed, BTRFS_I(inode),
3116+ start, end, PageUptodate(page));
31353117 }
3118+
31363119 ASSERT(bio_offset + len > bio_offset);
31373120 bio_offset += len;
31383121
3139− /* Update page status and unlock */
3140− end_page_read(page, uptodate, start, len);
3141− endio_readpage_release_extent(&processed, BTRFS_I(inode),
3142− start, end, PageUptodate(page));
31433122 }
31443123 /* Release the last extent */
31453124 endio_readpage_release_extent(&processed, NULL, 0, 0, false);
@@ -3208,19 +3187,6 @@ struct bio *btrfs_bio_alloc(unsigned int nr_iovecs) · 62 unmodified lines
32083187 return bio;
32093188 }
32103189
3211−struct bio *btrfs_bio_clone(struct block_device *bdev, struct bio *bio)
3212−{
3213− struct btrfs_bio *bbio;
3214− struct bio *new;
3215−
3216− /* Bio allocation backed by a bioset does not fail */
3217− new = bio_alloc_clone(bdev, bio, GFP_NOFS, &btrfs_bioset);
3218− bbio = btrfs_bio(new);
3219− btrfs_bio_init(bbio);
3220− bbio->iter = bio->bi_iter;
3221− return new;
3222−}
3223−
32243190 struct bio *btrfs_bio_clone_partial(struct bio *orig, u64 offset, u64 size)
32253191 {
32263192 struct bio *bio;
@@ -3380,7 +3346,6 @@ static int alloc_new_bio(struct btrfs_inode *inode, · 153 unmodified lines
33803346 bio_ctrl->bio = bio;
33813347 bio_ctrl->compress_type = compress_type;
33823348 bio->bi_end_io = end_io_func;
3383− bio->bi_private = &inode->io_tree;
33843349 bio->bi_opf = opf;
33853350 ret = calc_bio_boundaries(bio_ctrl, inode, file_offset);
33863351 if (ret < 0)
@@ -3445,7 +3410,6 @@ static int submit_extent_page(blk_opf_t opf, · 58 unmodified lines
34453410 struct page *page, u64 disk_bytenr,
34463411 size_t size, unsigned long pg_offset,
34473412 bio_end_io_t end_io_func,
3448− int mirror_num,
34493413 enum btrfs_compression_type compress_type,
34503414 bool force_bio_submit)
34513415 {
@@ -3457,10 +3421,8 @@ static int submit_extent_page(blk_opf_t opf, · 5 unmodified lines
34573421
34583422 ASSERT(pg_offset < PAGE_SIZE && size <= PAGE_SIZE &&
34593423 pg_offset + size <= PAGE_SIZE);
3460− if (force_bio_submit && bio_ctrl->bio) {
3461− submit_one_bio(bio_ctrl->bio, mirror_num, bio_ctrl->compress_type);
3462− bio_ctrl->bio = NULL;
3463− }
3424+ if (force_bio_submit)
3425+ submit_one_bio(bio_ctrl);
34643426
34653427 while (cur < pg_offset + size) {
34663428 u32 offset = cur - pg_offset;
@@ -3500,8 +3462,7 @@ static int submit_extent_page(blk_opf_t opf, · 33 unmodified lines
35003462 if (added < size - offset) {
35013463 /* The bio should contain some page(s) */
35023464 ASSERT(bio_ctrl->bio->bi_iter.bi_size);
3503− submit_one_bio(bio_ctrl->bio, mirror_num, bio_ctrl->compress_type);
3504− bio_ctrl->bio = NULL;
3465+ submit_one_bio(bio_ctrl);
35053466 }
35063467 cur += added;
35073468 }
@@ -3649,7 +3610,6 @@ static int btrfs_do_readpage(struct page *page, struct extent_map **em_cached, · 141 unmodified lines
36493610 if (zero_offset) {
36503611 iosize = PAGE_SIZE - zero_offset;
36513612 memzero_page(page, zero_offset, iosize);
3652− flush_dcache_page(page);
36533613 }
36543614 }
36553615 begin_page_read(fs_info, page);
@@ -3664,7 +3624,6 @@ static int btrfs_do_readpage(struct page *page, struct extent_map **em_cached, · 8 unmodified lines
36643624
36653625 iosize = PAGE_SIZE - pg_offset;
36663626 memzero_page(page, pg_offset, iosize);
3667− flush_dcache_page(page);
36683627 set_extent_uptodate(tree, cur, cur + iosize - 1,
36693628 &cached, GFP_NOFS);
36703629 unlock_extent_cached(tree, cur,
@@ -3748,7 +3707,6 @@ static int btrfs_do_readpage(struct page *page, struct extent_map **em_cached, · 77 unmodified lines
37483707 struct extent_state *cached = NULL;
37493708
37503709 memzero_page(page, pg_offset, iosize);
3751− flush_dcache_page(page);
37523710
37533711 set_extent_uptodate(tree, cur, cur + iosize - 1,
37543712 &cached, GFP_NOFS);
@@ -3781,10 +3739,8 @@ static int btrfs_do_readpage(struct page *page, struct extent_map **em_cached, · 26 unmodified lines
37813739
37823740 ret = submit_extent_page(REQ_OP_READ | read_flags, NULL,
37833741 bio_ctrl, page, disk_bytenr, iosize,
3784− pg_offset,
3785− end_bio_extent_readpage, 0,
3786− this_bio_flag,
3787− force_bio_submit);
3742+ pg_offset, end_bio_extent_readpage,
3743+ this_bio_flag, force_bio_submit);
37883744 if (ret) {
37893745 /*
37903746 * We have to unlock the remaining range, or the page
@@ -3817,8 +3773,7 @@ int btrfs_read_folio(struct file *file, struct folio *folio) · 26 unmodified lines
38173773 * If btrfs_do_readpage() failed we will want to submit the assembled
38183774 * bio to do the cleanup.
38193775 */
3820− if (bio_ctrl.bio)
3821− submit_one_bio(bio_ctrl.bio, 0, bio_ctrl.compress_type);
3776+ submit_one_bio(&bio_ctrl);
38223777 return ret;
38233778 }
38243779
@@ -4101,7 +4056,7 @@ static noinline_for_stack int __extent_writepage_io(struct btrfs_inode *inode, · 276 unmodified lines
41014056 disk_bytenr, iosize,
41024057 cur - page_offset(page),
41034058 end_bio_extent_writepage,
4104− 0, 0, false);
4059+ 0, false);
41054060 if (ret) {
41064061 has_error = true;
41074062 if (!saved_ret)
@@ -4166,10 +4121,8 @@ static int __extent_writepage(struct page *page, struct writeback_control *wbc, · 58 unmodified lines
41664121 return 0;
41674122 }
41684123
4169− if (page->index == end_index) {
4124+ if (page->index == end_index)
41704125 memzero_page(page, pg_offset, PAGE_SIZE - pg_offset);
4171− flush_dcache_page(page);
4172− }
41734126
41744127 ret = set_page_extent_mapped(page);
41754128 if (ret < 0) {
@@ -4278,7 +4231,7 @@ static noinline_for_stack int lock_extent_buffer_for_io(struct extent_buffer *eb · 102 unmodified lines
42784231 int ret = 0;
42794232
42804233 if (!btrfs_try_tree_write_lock(eb)) {
4281− flush_write_bio(epd);
4234+ submit_write_bio(epd, 0);
42824235 flush = 1;
42834236 btrfs_tree_lock(eb);
42844237 }
@@ -4288,7 +4241,7 @@ static noinline_for_stack int lock_extent_buffer_for_io(struct extent_buffer *eb · 3 unmodified lines
42884241 if (!epd->sync_io)
42894242 return 0;
42904243 if (!flush) {
4291− flush_write_bio(epd);
4244+ submit_write_bio(epd, 0);
42924245 flush = 1;
42934246 }
42944247 while (1) {
@@ -4335,7 +4288,7 @@ static noinline_for_stack int lock_extent_buffer_for_io(struct extent_buffer *eb · 40 unmodified lines
43354288
43364289 if (!trylock_page(p)) {
43374290 if (!flush) {
4338− flush_write_bio(epd);
4291+ submit_write_bio(epd, 0);
43394292 flush = 1;
43404293 }
43414294 lock_page(p);
@@ -4577,7 +4530,7 @@ static int write_one_subpage_eb(struct extent_buffer *eb, · 235 unmodified lines
45774530 {
45784531 struct btrfs_fs_info *fs_info = eb->fs_info;
45794532 struct page *page = eb->pages[0];
4580− blk_opf_t write_flags = wbc_to_write_flags(wbc) | REQ_META;
4533+ blk_opf_t write_flags = wbc_to_write_flags(wbc);
45814534 bool no_dirty_ebs = false;
45824535 int ret;
45834536
@@ -4596,7 +4549,7 @@ static int write_one_subpage_eb(struct extent_buffer *eb, · 12 unmodified lines
45964549 ret = submit_extent_page(REQ_OP_WRITE | write_flags, wbc,
45974550 &epd->bio_ctrl, page, eb->start, eb->len,
45984551 eb->start - page_offset(page),
4599− end_bio_subpage_eb_writepage, 0, 0, false);
4552+ end_bio_subpage_eb_writepage, 0, false);
46004553 if (ret) {
46014554 btrfs_subpage_clear_writeback(fs_info, page, eb->start, eb->len);
46024555 set_btree_ioerr(page, eb);
@@ -4622,7 +4575,7 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb, · 19 unmodified lines
46224575 {
46234576 u64 disk_bytenr = eb->start;
46244577 int i, num_pages;
4625− blk_opf_t write_flags = wbc_to_write_flags(wbc) | REQ_META;
4578+ blk_opf_t write_flags = wbc_to_write_flags(wbc);
46264579 int ret = 0;
46274580
46284581 prepare_eb_write(eb);
@@ -4637,7 +4590,7 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb, · 8 unmodified lines
46374590 &epd->bio_ctrl, p, disk_bytenr,
46384591 PAGE_SIZE, 0,
46394592 end_bio_extent_buffer_writepage,
4640− 0, 0, false);
4593+ 0, false);
46414594 if (ret) {
46424595 set_btree_ioerr(p, eb);
46434596 if (PageWriteback(p))
@@ -4751,7 +4704,7 @@ static int submit_eb_subpage(struct page *page, · 107 unmodified lines
47514704
47524705 cleanup:
47534706 /* We hit error, end bio for the submitted extent buffers */
4754− end_write_bio(epd, ret);
4707+ submit_write_bio(epd, ret);
47554708 return ret;
47564709 }
47574710
@@ -4930,10 +4883,6 @@ retry: · 172 unmodified lines
49304883 index = 0;
49314884 goto retry;
49324885 }
4933− if (ret < 0) {
4934− end_write_bio(&epd, ret);
4935− goto out;
4936− }
49374886 /*
49384887 * If something went wrong, don't allow any metadata write bio to be
49394888 * submitted.
@@ -4960,21 +4909,17 @@ retry: · 20 unmodified lines
49604909 * Now such dirty tree block will not be cleaned by any dirty
49614910 * extent io tree. Thus we don't want to submit such wild eb
49624911 * if the fs already has error.
4963− */
4964− if (!BTRFS_FS_ERROR(fs_info)) {
4965− flush_write_bio(&epd);
4966− } else {
4967− ret = -EROFS;
4968− end_write_bio(&epd, ret);
4969− }
4970−out:
4971− btrfs_zoned_meta_io_unlock(fs_info);
4972− /*
4912+ *
49734913 * We can get ret > 0 from submit_extent_page() indicating how many ebs
49744914 * were submitted. Reset it to 0 to avoid false alerts for the caller.
49754915 */
49764916 if (ret > 0)
49774917 ret = 0;
4918+ if (!ret && BTRFS_FS_ERROR(fs_info))
4919+ ret = -EROFS;
4920+ submit_write_bio(&epd, ret);
4921+
4922+ btrfs_zoned_meta_io_unlock(fs_info);
49784923 return ret;
49794924 }
49804925
@@ -5076,7 +5021,7 @@ retry: · 95 unmodified lines
50765021 * tmpfs file mapping
50775022 */
50785023 if (!trylock_page(page)) {
5079− flush_write_bio(epd);
5024+ submit_write_bio(epd, 0);
50805025 lock_page(page);
50815026 }
50825027
@@ -5087,7 +5032,7 @@ retry: · 4 unmodified lines
50875032
50885033 if (wbc->sync_mode != WB_SYNC_NONE) {
50895034 if (PageWriteback(page))
5090− flush_write_bio(epd);
5035+ submit_write_bio(epd, 0);
50915036 wait_on_page_writeback(page);
50925037 }
50935038
@@ -5127,7 +5072,7 @@ retry: · 33 unmodified lines
51275072 * page in our current bio, and thus deadlock, so flush the
51285073 * write bio here.
51295074 */
5130− flush_write_bio(epd);
5075+ submit_write_bio(epd, 0);
51315076 goto retry;
51325077 }
51335078
@@ -5138,26 +5083,6 @@ retry: · 4 unmodified lines
51385083 return ret;
51395084 }
51405085
5141−int extent_write_full_page(struct page *page, struct writeback_control *wbc)
5142−{
5143− int ret;
5144− struct extent_page_data epd = {
5145− .bio_ctrl = { 0 },
5146− .extent_locked = 0,
5147− .sync_io = wbc->sync_mode == WB_SYNC_ALL,
5148− };
5149−
5150− ret = __extent_writepage(page, wbc, &epd);
5151− ASSERT(ret <= 0);
5152− if (ret < 0) {
5153− end_write_bio(&epd, ret);
5154− return ret;
5155− }
5156−
5157− flush_write_bio(&epd);
5158− return ret;
5159−}
5160−
51615086 /*
51625087 * Submit the pages in the range to bio for call sites which delalloc range has
51635088 * already been ran (aka, ordered extent inserted) and all pages are still
@@ -5215,10 +5140,7 @@ int extent_write_locked_range(struct inode *inode, u64 start, u64 end) · 51 unmodified lines
52155140 cur = cur_end + 1;
52165141 }
52175142
5218− if (!found_error)
5219− flush_write_bio(&epd);
5220− else
5221− end_write_bio(&epd, ret);
5143+ submit_write_bio(&epd, found_error ? ret : 0);
52225144
52235145 wbc_detach_inode(&wbc_writepages);
52245146 if (found_error)
@@ -5243,13 +5165,7 @@ int extent_writepages(struct address_space *mapping, · 18 unmodified lines
52435165 */
52445166 btrfs_zoned_data_reloc_lock(BTRFS_I(inode));
52455167 ret = extent_write_cache_pages(mapping, wbc, &epd);
5246− ASSERT(ret <= 0);
5247− if (ret < 0) {
5248− btrfs_zoned_data_reloc_unlock(BTRFS_I(inode));
5249− end_write_bio(&epd, ret);
5250− return ret;
5251− }
5252− flush_write_bio(&epd);
5168+ submit_write_bio(&epd, ret);
52535169 btrfs_zoned_data_reloc_unlock(BTRFS_I(inode));
52545170 return ret;
52555171 }
@@ -5272,9 +5188,7 @@ void extent_readahead(struct readahead_control *rac) · 16 unmodified lines
52725188
52735189 if (em_cached)
52745190 free_extent_map(em_cached);
5275−
5276− if (bio_ctrl.bio)
5277− submit_one_bio(bio_ctrl.bio, 0, bio_ctrl.compress_type);
5191+ submit_one_bio(&bio_ctrl);
52785192 }
52795193
52805194 /*
@@ -6206,7 +6120,7 @@ static int check_eb_alignment(struct btrfs_fs_info *fs_info, u64 start) · 925 unmodified lines
62066120 return -EINVAL;
62076121 }
62086122 if (fs_info->nodesize >= PAGE_SIZE &&
6209− !IS_ALIGNED(start, PAGE_SIZE)) {
6123+ !PAGE_ALIGNED(start)) {
62106124 btrfs_err(fs_info,
62116125 "tree block is not page aligned, start %llu nodesize %u",
62126126 start, fs_info->nodesize);
@@ -6608,7 +6522,9 @@ static int read_extent_buffer_subpage(struct extent_buffer *eb, int wait, · 395 unmodified lines
66086522 struct btrfs_fs_info *fs_info = eb->fs_info;
66096523 struct extent_io_tree *io_tree;
66106524 struct page *page = eb->pages[0];
6611− struct btrfs_bio_ctrl bio_ctrl = { 0 };
6525+ struct btrfs_bio_ctrl bio_ctrl = {
6526+ .mirror_num = mirror_num,
6527+ };
66126528 int ret = 0;
66136529
66146530 ASSERT(!test_bit(EXTENT_BUFFER_UNMAPPED, &eb->bflags));
@@ -6640,11 +6556,10 @@ static int read_extent_buffer_subpage(struct extent_buffer *eb, int wait, · 25 unmodified lines
66406556 btrfs_subpage_clear_error(fs_info, page, eb->start, eb->len);
66416557
66426558 btrfs_subpage_start_reader(fs_info, page, eb->start, eb->len);
6643− ret = submit_extent_page(REQ_OP_READ | REQ_META, NULL, &bio_ctrl,
6559+ ret = submit_extent_page(REQ_OP_READ, NULL, &bio_ctrl,
66446560 page, eb->start, eb->len,
66456561 eb->start - page_offset(page),
6646− end_bio_extent_readpage, mirror_num, 0,
6647− true);
6562+ end_bio_extent_readpage, 0, true);
66486563 if (ret) {
66496564 /*
66506565 * In the endio function, if we hit something wrong we will
@@ -6653,10 +6568,7 @@ static int read_extent_buffer_subpage(struct extent_buffer *eb, int wait, · 2 unmodified lines
66536568 */
66546569 atomic_dec(&eb->io_pages);
66556570 }
6656− if (bio_ctrl.bio) {
6657− submit_one_bio(bio_ctrl.bio, mirror_num, 0);
6658− bio_ctrl.bio = NULL;
6659− }
6571+ submit_one_bio(&bio_ctrl);
66606572 if (ret || wait != WAIT_COMPLETE)
66616573 return ret;
66626574
@@ -6676,7 +6588,9 @@ int read_extent_buffer_pages(struct extent_buffer *eb, int wait, int mirror_num) · 13 unmodified lines
66766588 int all_uptodate = 1;
66776589 int num_pages;
66786590 unsigned long num_reads = 0;
6679− struct btrfs_bio_ctrl bio_ctrl = { 0 };
6591+ struct btrfs_bio_ctrl bio_ctrl = {
6592+ .mirror_num = mirror_num,
6593+ };
66806594
66816595 if (test_bit(EXTENT_BUFFER_UPTODATE, &eb->bflags))
66826596 return 0;
@@ -6747,10 +6661,10 @@ int read_extent_buffer_pages(struct extent_buffer *eb, int wait, int mirror_num) · 64 unmodified lines
67476661 }
67486662
67496663 ClearPageError(page);
6750− err = submit_extent_page(REQ_OP_READ | REQ_META, NULL,
6664+ err = submit_extent_page(REQ_OP_READ, NULL,
67516665 &bio_ctrl, page, page_offset(page),
67526666 PAGE_SIZE, 0, end_bio_extent_readpage,
6753− mirror_num, 0, false);
6667+ 0, false);
67546668 if (err) {
67556669 /*
67566670 * We failed to submit the bio so it's the
@@ -6767,10 +6681,7 @@ int read_extent_buffer_pages(struct extent_buffer *eb, int wait, int mirror_num) · 10 unmodified lines
67676681 }
67686682 }
67696683
6770− if (bio_ctrl.bio) {
6771− submit_one_bio(bio_ctrl.bio, mirror_num, bio_ctrl.compress_type);
6772− bio_ctrl.bio = NULL;
6773− }
6684+ submit_one_bio(&bio_ctrl);
67746685
67756686 if (ret || wait != WAIT_COMPLETE)
67766687 return ret;
fs/btrfs/extent_io.h+4−11
@@ -57,6 +57,7 @@ enum { · 56 unmodified lines
5757 #define BITMAP_LAST_BYTE_MASK(nbits) \
5858 (BYTE_MASK >> (-(nbits) & (BITS_PER_BYTE - 1)))
5959
60+struct btrfs_bio;
6061 struct btrfs_root;
6162 struct btrfs_inode;
6263 struct btrfs_io_bio;
@@ -142,15 +143,10 @@ static inline void extent_changeset_free(struct extent_changeset *changeset) · 79 unmodified lines
142143
143144 struct extent_map_tree;
144145
145−typedef struct extent_map *(get_extent_t)(struct btrfs_inode *inode,
146− struct page *page, size_t pg_offset,
147− u64 start, u64 len);
148−
149146 int try_release_extent_mapping(struct page *page, gfp_t mask);
150147 int try_release_extent_buffer(struct page *page);
151148
152149 int btrfs_read_folio(struct file *file, struct folio *folio);
153−int extent_write_full_page(struct page *page, struct writeback_control *wbc);
154150 int extent_write_locked_range(struct inode *inode, u64 start, u64 end);
155151 int extent_writepages(struct address_space *mapping,
156152 struct writeback_control *wbc);
@@ -247,7 +243,6 @@ void extent_clear_unlock_delalloc(struct btrfs_inode *inode, u64 start, u64 end, · 90 unmodified lines
247243
248244 int btrfs_alloc_page_array(unsigned int nr_pages, struct page **page_array);
249245 struct bio *btrfs_bio_alloc(unsigned int nr_iovecs);
250−struct bio *btrfs_bio_clone(struct block_device *bdev, struct bio *bio);
251246 struct bio *btrfs_bio_clone_partial(struct bio *orig, u64 offset, u64 size);
252247
253248 void end_extent_writepage(struct page *page, int err, u64 start, u64 end);
@@ -266,15 +261,13 @@ struct io_failure_record { · 12 unmodified lines
266261 u64 start;
267262 u64 len;
268263 u64 logical;
269− enum btrfs_compression_type compress_type;
270264 int this_mirror;
271265 int failed_mirror;
266+ int num_copies;
272267 };
273268
274−int btrfs_repair_one_sector(struct inode *inode,
275− struct bio *failed_bio, u32 bio_offset,
276− struct page *page, unsigned int pgoff,
277− u64 start, int failed_mirror,
269+int btrfs_repair_one_sector(struct inode *inode, struct btrfs_bio *failed_bbio,
270+ u32 bio_offset, struct page *page, unsigned int pgoff,
278271 submit_bio_hook_t *submit_bio_hook);
279272
280273 #ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
fs/btrfs/file.c+23−6
@@ -1954,11 +1954,25 @@ again: · 1953 unmodified lines
19541954
19551955 btrfs_inode_unlock(inode, ilock_flags);
19561956
1957− /* If 'err' is -ENOTBLK then it means we must fallback to buffered IO. */
1957+ /*
1958+ * If 'err' is -ENOTBLK or we have not written all data, then it means
1959+ * we must fallback to buffered IO.
1960+ */
19581961 if ((err < 0 && err != -ENOTBLK) || !iov_iter_count(from))
19591962 goto out;
19601963
19611964 buffered:
1965+ /*
1966+ * If we are in a NOWAIT context, then return -EAGAIN to signal the caller
1967+ * it must retry the operation in a context where blocking is acceptable,
1968+ * since we currently don't have NOWAIT semantics support for buffered IO
1969+ * and may block there for many reasons (reserving space for example).
1970+ */
1971+ if (iocb->ki_flags & IOCB_NOWAIT) {
1972+ err = -EAGAIN;
1973+ goto out;
1974+ }
1975+
19621976 pos = iocb->ki_pos;
19631977 written_buffered = btrfs_buffered_write(iocb, from);
19641978 if (written_buffered < 0) {
@@ -2041,9 +2055,11 @@ ssize_t btrfs_do_write_iter(struct kiocb *iocb, struct iov_iter *from, · 76 unmodified lines
20412055 num_written = btrfs_encoded_write(iocb, from, encoded);
20422056 num_sync = encoded->len;
20432057 } else if (iocb->ki_flags & IOCB_DIRECT) {
2044− num_written = num_sync = btrfs_direct_write(iocb, from);
2058+ num_written = btrfs_direct_write(iocb, from);
2059+ num_sync = num_written;
20452060 } else {
2046− num_written = num_sync = btrfs_buffered_write(iocb, from);
2061+ num_written = btrfs_buffered_write(iocb, from);
2062+ num_sync = num_written;
20472063 }
20482064
20492065 btrfs_set_inode_last_sub_trans(inode);
@@ -2291,7 +2307,7 @@ int btrfs_sync_file(struct file *file, loff_t start, loff_t end, int datasync) · 241 unmodified lines
22912307 btrfs_release_log_ctx_extents(&ctx);
22922308 if (ret < 0) {
22932309 /* Fallthrough and commit/free transaction. */
2294− ret = 1;
2310+ ret = BTRFS_LOG_FORCE_COMMIT;
22952311 }
22962312
22972313 /* we've logged all the items and now have a consistent
@@ -2717,7 +2733,7 @@ int btrfs_replace_file_extents(struct btrfs_inode *inode, · 419 unmodified lines
27172733 goto out;
27182734 }
27192735 rsv->size = btrfs_calc_insert_metadata_size(fs_info, 1);
2720− rsv->failfast = 1;
2736+ rsv->failfast = true;
27212737
27222738 /*
27232739 * 1 - update the inode
@@ -3083,7 +3099,8 @@ static int btrfs_punch_hole(struct file *file, loff_t offset, loff_t len) · 359 unmodified lines
30833099
30843100 ASSERT(trans != NULL);
30853101 inode_inc_iversion(inode);
3086− inode->i_mtime = inode->i_ctime = current_time(inode);
3102+ inode->i_mtime = current_time(inode);
3103+ inode->i_ctime = inode->i_mtime;
30873104 ret = btrfs_update_inode(trans, root, BTRFS_I(inode));
30883105 updated_inode = true;
30893106 btrfs_end_transaction(trans);
fs/btrfs/free-space-cache.c+2−1
@@ -3536,7 +3536,8 @@ int btrfs_find_space_cluster(struct btrfs_block_group *block_group, · 3535 unmodified lines
35363536 * data, keep it dense.
35373537 */
35383538 if (btrfs_test_opt(fs_info, SSD_SPREAD)) {
3539− cont1_bytes = min_bytes = bytes + empty_size;
3539+ cont1_bytes = bytes + empty_size;
3540+ min_bytes = cont1_bytes;
35403541 } else if (block_group->flags & BTRFS_BLOCK_GROUP_METADATA) {
35413542 cont1_bytes = bytes;
35423543 min_bytes = fs_info->sectorsize;
fs/btrfs/inode.c+400−364
@@ -114,21 +114,17 @@ struct kmem_cache *btrfs_free_space_bitmap_cachep; · 113 unmodified lines
114114
115115 static int btrfs_setsize(struct inode *inode, struct iattr *attr);
116116 static int btrfs_truncate(struct inode *inode, bool skip_writeback);
117−static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent);
118117 static noinline int cow_file_range(struct btrfs_inode *inode,
119118 struct page *locked_page,
120119 u64 start, u64 end, int *page_started,
121− unsigned long *nr_written, int unlock);
120+ unsigned long *nr_written, int unlock,
121+ u64 *done_offset);
122122 static struct extent_map *create_io_em(struct btrfs_inode *inode, u64 start,
123123 u64 len, u64 orig_start, u64 block_start,
124124 u64 block_len, u64 orig_block_len,
125125 u64 ram_bytes, int compress_type,
126126 int type);
127127
128−static void __endio_write_update_ordered(struct btrfs_inode *inode,
129− const u64 offset, const u64 bytes,
130− const bool uptodate);
131−
132128 /*
133129 * btrfs_inode_lock - lock inode i_rwsem based on arguments passed
134130 *
@@ -195,11 +191,14 @@ static inline void btrfs_cleanup_ordered_extents(struct btrfs_inode *inode, · 60 unmodified lines
195191 {
196192 unsigned long index = offset >> PAGE_SHIFT;
197193 unsigned long end_index = (offset + bytes - 1) >> PAGE_SHIFT;
198− u64 page_start = page_offset(locked_page);
199− u64 page_end = page_start + PAGE_SIZE - 1;
200−
194+ u64 page_start, page_end;
201195 struct page *page;
202196
197+ if (locked_page) {
198+ page_start = page_offset(locked_page);
199+ page_end = page_start + PAGE_SIZE - 1;
200+ }
201+
203202 while (index <= end_index) {
204203 /*
205204 * For locked page, we will call end_extent_writepage() on it
@@ -212,7 +211,7 @@ static inline void btrfs_cleanup_ordered_extents(struct btrfs_inode *inode, · 6 unmodified lines
212211 * btrfs_mark_ordered_io_finished() would skip the accounting
213212 * for the page range, and the ordered extent will never finish.
214213 */
215− if (index == (page_offset(locked_page) >> PAGE_SHIFT)) {
214+ if (locked_page && index == (page_start >> PAGE_SHIFT)) {
216215 index++;
217216 continue;
218217 }
@@ -223,7 +222,7 @@ static inline void btrfs_cleanup_ordered_extents(struct btrfs_inode *inode, · 4 unmodified lines
223222
224223 /*
225224 * Here we just clear all Ordered bits for every page in the
226− * range, then __endio_write_update_ordered() will handle
225+ * range, then btrfs_mark_ordered_io_finished() will handle
227226 * the ordered extent accounting for the range.
228227 */
229228 btrfs_page_clamp_clear_ordered(inode->root->fs_info, page,
@@ -231,20 +230,23 @@ static inline void btrfs_cleanup_ordered_extents(struct btrfs_inode *inode, · 1 unmodified line
231230 put_page(page);
232231 }
233232
234− /* The locked page covers the full range, nothing needs to be done */
235− if (bytes + offset <= page_offset(locked_page) + PAGE_SIZE)
236− return;
237− /*
238− * In case this page belongs to the delalloc range being instantiated
239− * then skip it, since the first page of a range is going to be
240− * properly cleaned up by the caller of run_delalloc_range
241− */
242− if (page_start >= offset && page_end <= (offset + bytes - 1)) {
243− bytes = offset + bytes - page_offset(locked_page) - PAGE_SIZE;
244− offset = page_offset(locked_page) + PAGE_SIZE;
233+ if (locked_page) {
234+ /* The locked page covers the full range, nothing needs to be done */
235+ if (bytes + offset <= page_start + PAGE_SIZE)
236+ return;
237+ /*
238+ * In case this page belongs to the delalloc range being
239+ * instantiated then skip it, since the first page of a range is
240+ * going to be properly cleaned up by the caller of
241+ * run_delalloc_range
242+ */
243+ if (page_start >= offset && page_end <= (offset + bytes - 1)) {
244+ bytes = offset + bytes - page_offset(locked_page) - PAGE_SIZE;
245+ offset = page_offset(locked_page) + PAGE_SIZE;
246+ }
245247 }
246248
247− return __endio_write_update_ordered(inode, offset, bytes, false);
249+ return btrfs_mark_ordered_io_finished(inode, NULL, offset, bytes, false);
248250 }
249251
250252 static int btrfs_dirty_inode(struct inode *inode);
@@ -332,9 +334,9 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans, · 81 unmodified lines
332334 cur_size = min_t(unsigned long, compressed_size,
333335 PAGE_SIZE);
334336
335− kaddr = kmap_atomic(cpage);
337+ kaddr = kmap_local_page(cpage);
336338 write_extent_buffer(leaf, kaddr, ptr, cur_size);
337− kunmap_atomic(kaddr);
339+ kunmap_local(kaddr);
338340
339341 i++;
340342 ptr += cur_size;
@@ -345,9 +347,9 @@ static int insert_inline_extent(struct btrfs_trans_handle *trans, · 4 unmodified lines
345347 } else {
346348 page = find_get_page(inode->vfs_inode.i_mapping, 0);
347349 btrfs_set_file_extent_compression(leaf, ei, 0);
348− kaddr = kmap_atomic(page);
350+ kaddr = kmap_local_page(page);
349351 write_extent_buffer(leaf, kaddr, ptr, size);
350− kunmap_atomic(kaddr);
352+ kunmap_local(kaddr);
351353 put_page(page);
352354 }
353355 btrfs_mark_buffer_dirty(leaf);
@@ -560,8 +562,8 @@ static inline int inode_need_compress(struct btrfs_inode *inode, u64 start, · 206 unmodified lines
560562 * will unlock the full page.
561563 */
562564 if (fs_info->sectorsize < PAGE_SIZE) {
563− if (!IS_ALIGNED(start, PAGE_SIZE) ||
564− !IS_ALIGNED(end + 1, PAGE_SIZE))
565+ if (!PAGE_ALIGNED(start) ||
566+ !PAGE_ALIGNED(end + 1))
565567 return 0;
566568 }
567569
@@ -678,8 +680,8 @@ again: · 110 unmodified lines
678680 * Thus we must also check against @actual_end, not just @end.
679681 */
680682 if (blocksize < PAGE_SIZE) {
681− if (!IS_ALIGNED(start, PAGE_SIZE) ||
682− !IS_ALIGNED(round_up(actual_end, blocksize), PAGE_SIZE))
683+ if (!PAGE_ALIGNED(start) ||
684+ !PAGE_ALIGNED(round_up(actual_end, blocksize)))
683685 goto cleanup_and_bail_uncompressed;
684686 }
685687
@@ -920,15 +922,25 @@ static int submit_uncompressed_range(struct btrfs_inode *inode, · 234 unmodified lines
920922 * can directly submit them without interruption.
921923 */
922924 ret = cow_file_range(inode, locked_page, start, end, &page_started,
923− &nr_written, 0);
925+ &nr_written, 0, NULL);
924926 /* Inline extent inserted, page gets unlocked and everything is done */
925927 if (page_started) {
926928 ret = 0;
927929 goto out;
928930 }
929931 if (ret < 0) {
930− if (locked_page)
932+ btrfs_cleanup_ordered_extents(inode, locked_page, start, end - start + 1);
933+ if (locked_page) {
934+ const u64 page_start = page_offset(locked_page);
935+ const u64 page_end = page_start + PAGE_SIZE - 1;
936+
937+ btrfs_page_set_error(inode->root->fs_info, locked_page,
938+ page_start, PAGE_SIZE);
939+ set_page_writeback(locked_page);
940+ end_page_writeback(locked_page);
941+ end_extent_writepage(locked_page, ret, page_start, page_end);
931942 unlock_page(locked_page);
943+ }
932944 goto out;
933945 }
934946
@@ -1133,15 +1145,39 @@ static u64 get_extent_allocation_hint(struct btrfs_inode *inode, u64 start, · 198 unmodified lines
11331145 * *page_started is set to one if we unlock locked_page and do everything
11341146 * required to start IO on it. It may be clean and already done with
11351147 * IO when we return.
1148+ *
1149+ * When unlock == 1, we unlock the pages in successfully allocated regions.
1150+ * When unlock == 0, we leave them locked for writing them out.
1151+ *
1152+ * However, we unlock all the pages except @locked_page in case of failure.
1153+ *
1154+ * In summary, page locking state will be as follow:
1155+ *
1156+ * - page_started == 1 (return value)
1157+ * - All the pages are unlocked. IO is started.
1158+ * - Note that this can happen only on success
1159+ * - unlock == 1
1160+ * - All the pages except @locked_page are unlocked in any case
1161+ * - unlock == 0
1162+ * - On success, all the pages are locked for writing out them
1163+ * - On failure, all the pages except @locked_page are unlocked
1164+ *
1165+ * When a failure happens in the second or later iteration of the
1166+ * while-loop, the ordered extents created in previous iterations are kept
1167+ * intact. So, the caller must clean them up by calling
1168+ * btrfs_cleanup_ordered_extents(). See btrfs_run_delalloc_range() for
1169+ * example.
11361170 */
11371171 static noinline int cow_file_range(struct btrfs_inode *inode,
11381172 struct page *locked_page,
11391173 u64 start, u64 end, int *page_started,
1140− unsigned long *nr_written, int unlock)
1174+ unsigned long *nr_written, int unlock,
1175+ u64 *done_offset)
11411176 {
11421177 struct btrfs_root *root = inode->root;
11431178 struct btrfs_fs_info *fs_info = root->fs_info;
11441179 u64 alloc_hint = 0;
1180+ u64 orig_start = start;
11451181 u64 num_bytes;
11461182 unsigned long ram_size;
11471183 u64 cur_alloc_size = 0;
@@ -1329,18 +1365,62 @@ out_reserve: · 181 unmodified lines
13291365 btrfs_dec_block_group_reservations(fs_info, ins.objectid);
13301366 btrfs_free_reserved_extent(fs_info, ins.objectid, ins.offset, 1);
13311367 out_unlock:
1368+ /*
1369+ * If done_offset is non-NULL and ret == -EAGAIN, we expect the
1370+ * caller to write out the successfully allocated region and retry.
1371+ */
1372+ if (done_offset && ret == -EAGAIN) {
1373+ if (orig_start < start)
1374+ *done_offset = start - 1;
1375+ else
1376+ *done_offset = start;
1377+ return ret;
1378+ } else if (ret == -EAGAIN) {
1379+ /* Convert to -ENOSPC since the caller cannot retry. */
1380+ ret = -ENOSPC;
1381+ }
1382+
1383+ /*
1384+ * Now, we have three regions to clean up:
1385+ *
1386+ * |-------(1)----|---(2)---|-------------(3)----------|
1387+ * `- orig_start `- start `- start + cur_alloc_size `- end
1388+ *
1389+ * We process each region below.
1390+ */
1391+
13321392 clear_bits = EXTENT_LOCKED | EXTENT_DELALLOC | EXTENT_DELALLOC_NEW |
13331393 EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV;
13341394 page_ops = PAGE_UNLOCK | PAGE_START_WRITEBACK | PAGE_END_WRITEBACK;
1395+
1396+ /*
1397+ * For the range (1). We have already instantiated the ordered extents
1398+ * for this region. They are cleaned up by
1399+ * btrfs_cleanup_ordered_extents() in e.g,
1400+ * btrfs_run_delalloc_range(). EXTENT_LOCKED | EXTENT_DELALLOC are
1401+ * already cleared in the above loop. And, EXTENT_DELALLOC_NEW |
1402+ * EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV are handled by the cleanup
1403+ * function.
1404+ *
1405+ * However, in case of unlock == 0, we still need to unlock the pages
1406+ * (except @locked_page) to ensure all the pages are unlocked.
1407+ */
1408+ if (!unlock && orig_start < start) {
1409+ if (!locked_page)
1410+ mapping_set_error(inode->vfs_inode.i_mapping, ret);
1411+ extent_clear_unlock_delalloc(inode, orig_start, start - 1,
1412+ locked_page, 0, page_ops);
1413+ }
1414+
13351415 /*
1336− * If we reserved an extent for our delalloc range (or a subrange) and
1337− * failed to create the respective ordered extent, then it means that
1338− * when we reserved the extent we decremented the extent's size from
1339− * the data space_info's bytes_may_use counter and incremented the
1340− * space_info's bytes_reserved counter by the same amount. We must make
1341− * sure extent_clear_unlock_delalloc() does not try to decrement again
1342− * the data space_info's bytes_may_use counter, therefore we do not pass
1343− * it the flag EXTENT_CLEAR_DATA_RESV.
1416+ * For the range (2). If we reserved an extent for our delalloc range
1417+ * (or a subrange) and failed to create the respective ordered extent,
1418+ * then it means that when we reserved the extent we decremented the
1419+ * extent's size from the data space_info's bytes_may_use counter and
1420+ * incremented the space_info's bytes_reserved counter by the same
1421+ * amount. We must make sure extent_clear_unlock_delalloc() does not try
1422+ * to decrement again the data space_info's bytes_may_use counter,
1423+ * therefore we do not pass it the flag EXTENT_CLEAR_DATA_RESV.
13441424 */
13451425 if (extent_reserved) {
13461426 extent_clear_unlock_delalloc(inode, start,
@@ -1350,12 +1430,19 @@ out_unlock: · 3 unmodified lines
13501430 page_ops);
13511431 start += cur_alloc_size;
13521432 if (start >= end)
1353− goto out;
1433+ return ret;
13541434 }
1435+
1436+ /*
1437+ * For the range (3). We never touched the region. In addition to the
1438+ * clear_bits above, we add EXTENT_CLEAR_DATA_RESV to release the data
1439+ * space_info's bytes_may_use counter, reserved in
1440+ * btrfs_check_data_free_space().
1441+ */
13551442 extent_clear_unlock_delalloc(inode, start, end, locked_page,
13561443 clear_bits | EXTENT_CLEAR_DATA_RESV,
13571444 page_ops);
1358− goto out;
1445+ return ret;
13591446 }
13601447
13611448 /*
@@ -1538,19 +1625,42 @@ static noinline int run_delalloc_zoned(struct btrfs_inode *inode, · 176 unmodified lines
15381625 u64 end, int *page_started,
15391626 unsigned long *nr_written)
15401627 {
1628+ u64 done_offset = end;
15411629 int ret;
1630+ bool locked_page_done = false;
15421631
1543− ret = cow_file_range(inode, locked_page, start, end, page_started,
1544− nr_written, 0);
1545− if (ret)
1546− return ret;
1632+ while (start <= end) {
1633+ ret = cow_file_range(inode, locked_page, start, end, page_started,
1634+ nr_written, 0, &done_offset);
1635+ if (ret && ret != -EAGAIN)
1636+ return ret;
15471637
1548− if (*page_started)
1549− return 0;
1638+ if (*page_started) {
1639+ ASSERT(ret == 0);
1640+ return 0;
1641+ }
1642+
1643+ if (ret == 0)
1644+ done_offset = end;
1645+
1646+ if (done_offset == start) {
1647+ struct btrfs_fs_info *info = inode->root->fs_info;
1648+
1649+ wait_var_event(&info->zone_finish_wait,
1650+ !test_bit(BTRFS_FS_NEED_ZONE_FINISH, &info->flags));
1651+ continue;
1652+ }
1653+
1654+ if (!locked_page_done) {
1655+ __set_page_dirty_nobuffers(locked_page);
1656+ account_page_redirty(locked_page);
1657+ }
1658+ locked_page_done = true;
1659+ extent_write_locked_range(&inode->vfs_inode, start, done_offset);
1660+
1661+ start = done_offset + 1;
1662+ }
15501663
1551− __set_page_dirty_nobuffers(locked_page);
1552− account_page_redirty(locked_page);
1553− extent_write_locked_range(&inode->vfs_inode, start, end);
15541664 *page_started = 1;
15551665
15561666 return 0;
@@ -1642,7 +1752,7 @@ static int fallback_to_cow(struct btrfs_inode *inode, struct page *locked_page, · 85 unmodified lines
16421752 }
16431753
16441754 return cow_file_range(inode, locked_page, start, end, page_started,
1645− nr_written, 1);
1755+ nr_written, 1, NULL);
16461756 }
16471757
16481758 struct can_nocow_file_extent_args {
@@ -2115,7 +2225,7 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct page *locked_page · 466 unmodified lines
21152225 page_started, nr_written);
21162226 else
21172227 ret = cow_file_range(inode, locked_page, start, end,
2118− page_started, nr_written, 1);
2228+ page_started, nr_written, 1, NULL);
21192229 } else {
21202230 set_bit(BTRFS_INODE_HAS_ASYNC_EXTENT, &inode->runtime_flags);
21212231 ret = cow_file_range_async(inode, wbc, locked_page, start, end,
@@ -2131,6 +2241,7 @@ int btrfs_run_delalloc_range(struct btrfs_inode *inode, struct page *locked_page · 9 unmodified lines
21312241 void btrfs_split_delalloc_extent(struct inode *inode,
21322242 struct extent_state *orig, u64 split)
21332243 {
2244+ struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
21342245 u64 size;
21352246
21362247 /* not delalloc, ignore it */
@@ -2138,7 +2249,7 @@ void btrfs_split_delalloc_extent(struct inode *inode, · 1 unmodified line
21382249 return;
21392250
21402251 size = orig->end - orig->start + 1;
2141− if (size > BTRFS_MAX_EXTENT_SIZE) {
2252+ if (size > fs_info->max_extent_size) {
21422253 u32 num_extents;
21432254 u64 new_size;
21442255
@@ -2147,10 +2258,10 @@ void btrfs_split_delalloc_extent(struct inode *inode, · 2 unmodified lines
21472258 * applies here, just in reverse.
21482259 */
21492260 new_size = orig->end - split + 1;
2150− num_extents = count_max_extents(new_size);
2261+ num_extents = count_max_extents(fs_info, new_size);
21512262 new_size = split - orig->start;
2152− num_extents += count_max_extents(new_size);
2153− if (count_max_extents(size) >= num_extents)
2263+ num_extents += count_max_extents(fs_info, new_size);
2264+ if (count_max_extents(fs_info, size) >= num_extents)
21542265 return;
21552266 }
21562267
@@ -2167,6 +2278,7 @@ void btrfs_split_delalloc_extent(struct inode *inode, · 10 unmodified lines
21672278 void btrfs_merge_delalloc_extent(struct inode *inode, struct extent_state *new,
21682279 struct extent_state *other)
21692280 {
2281+ struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
21702282 u64 new_size, old_size;
21712283 u32 num_extents;
21722284
@@ -2180,7 +2292,7 @@ void btrfs_merge_delalloc_extent(struct inode *inode, struct extent_state *new, · 7 unmodified lines
21802292 new_size = other->end - new->start + 1;
21812293
21822294 /* we're not bigger than the max, unreserve the space and go */
2183− if (new_size <= BTRFS_MAX_EXTENT_SIZE) {
2295+ if (new_size <= fs_info->max_extent_size) {
21842296 spin_lock(&BTRFS_I(inode)->lock);
21852297 btrfs_mod_outstanding_extents(BTRFS_I(inode), -1);
21862298 spin_unlock(&BTRFS_I(inode)->lock);
@@ -2206,10 +2318,10 @@ void btrfs_merge_delalloc_extent(struct inode *inode, struct extent_state *new, · 19 unmodified lines
22062318 * this case.
22072319 */
22082320 old_size = other->end - other->start + 1;
2209− num_extents = count_max_extents(old_size);
2321+ num_extents = count_max_extents(fs_info, old_size);
22102322 old_size = new->end - new->start + 1;
2211− num_extents += count_max_extents(old_size);
2212− if (count_max_extents(new_size) >= num_extents)
2323+ num_extents += count_max_extents(fs_info, old_size);
2324+ if (count_max_extents(fs_info, new_size) >= num_extents)
22132325 return;
22142326
22152327 spin_lock(&BTRFS_I(inode)->lock);
@@ -2274,21 +2386,21 @@ static void btrfs_del_delalloc_inode(struct btrfs_root *root, · 58 unmodified lines
22742386 * list of inodes that have pending delalloc work to be done.
22752387 */
22762388 void btrfs_set_delalloc_extent(struct inode *inode, struct extent_state *state,
2277− unsigned *bits)
2389+ u32 bits)
22782390 {
22792391 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
22802392
2281− if ((*bits & EXTENT_DEFRAG) && !(*bits & EXTENT_DELALLOC))
2393+ if ((bits & EXTENT_DEFRAG) && !(bits & EXTENT_DELALLOC))
22822394 WARN_ON(1);
22832395 /*
22842396 * set_bit and clear bit hooks normally require _irqsave/restore
22852397 * but in this case, we are only testing for the DELALLOC
22862398 * bit, which is only set or cleared with irqs on
22872399 */
2288− if (!(state->state & EXTENT_DELALLOC) && (*bits & EXTENT_DELALLOC)) {
2400+ if (!(state->state & EXTENT_DELALLOC) && (bits & EXTENT_DELALLOC)) {
22892401 struct btrfs_root *root = BTRFS_I(inode)->root;
22902402 u64 len = state->end + 1 - state->start;
2291− u32 num_extents = count_max_extents(len);
2403+ u32 num_extents = count_max_extents(fs_info, len);
22922404 bool do_list = !btrfs_is_free_space_inode(BTRFS_I(inode));
22932405
22942406 spin_lock(&BTRFS_I(inode)->lock);
@@ -2303,7 +2415,7 @@ void btrfs_set_delalloc_extent(struct inode *inode, struct extent_state *state, · 8 unmodified lines
23032415 fs_info->delalloc_batch);
23042416 spin_lock(&BTRFS_I(inode)->lock);
23052417 BTRFS_I(inode)->delalloc_bytes += len;
2306− if (*bits & EXTENT_DEFRAG)
2418+ if (bits & EXTENT_DEFRAG)
23072419 BTRFS_I(inode)->defrag_bytes += len;
23082420 if (do_list && !test_bit(BTRFS_INODE_IN_DELALLOC_LIST,
23092421 &BTRFS_I(inode)->runtime_flags))
@@ -2312,7 +2424,7 @@ void btrfs_set_delalloc_extent(struct inode *inode, struct extent_state *state, · 2 unmodified lines
23122424 }
23132425
23142426 if (!(state->state & EXTENT_DELALLOC_NEW) &&
2315− (*bits & EXTENT_DELALLOC_NEW)) {
2427+ (bits & EXTENT_DELALLOC_NEW)) {
23162428 spin_lock(&BTRFS_I(inode)->lock);
23172429 BTRFS_I(inode)->new_delalloc_bytes += state->end + 1 -
23182430 state->start;
@@ -2325,14 +2437,14 @@ void btrfs_set_delalloc_extent(struct inode *inode, struct extent_state *state, · 6 unmodified lines
23252437 * accounting happens.
23262438 */
23272439 void btrfs_clear_delalloc_extent(struct inode *vfs_inode,
2328− struct extent_state *state, unsigned *bits)
2440+ struct extent_state *state, u32 bits)
23292441 {
23302442 struct btrfs_inode *inode = BTRFS_I(vfs_inode);
23312443 struct btrfs_fs_info *fs_info = btrfs_sb(vfs_inode->i_sb);
23322444 u64 len = state->end + 1 - state->start;
2333− u32 num_extents = count_max_extents(len);
2445+ u32 num_extents = count_max_extents(fs_info, len);
23342446
2335− if ((state->state & EXTENT_DEFRAG) && (*bits & EXTENT_DEFRAG)) {
2447+ if ((state->state & EXTENT_DEFRAG) && (bits & EXTENT_DEFRAG)) {
23362448 spin_lock(&inode->lock);
23372449 inode->defrag_bytes -= len;
23382450 spin_unlock(&inode->lock);
@@ -2343,7 +2455,7 @@ void btrfs_clear_delalloc_extent(struct inode *vfs_inode, · 4 unmodified lines
23432455 * but in this case, we are only testing for the DELALLOC
23442456 * bit, which is only set or cleared with irqs on
23452457 */
2346− if ((state->state & EXTENT_DELALLOC) && (*bits & EXTENT_DELALLOC)) {
2458+ if ((state->state & EXTENT_DELALLOC) && (bits & EXTENT_DELALLOC)) {
23472459 struct btrfs_root *root = inode->root;
23482460 bool do_list = !btrfs_is_free_space_inode(inode);
23492461
@@ -2356,7 +2468,7 @@ void btrfs_clear_delalloc_extent(struct inode *vfs_inode, · 6 unmodified lines
23562468 * don't need to call delalloc_release_metadata if there is an
23572469 * error.
23582470 */
2359− if (*bits & EXTENT_CLEAR_META_RESV &&
2471+ if (bits & EXTENT_CLEAR_META_RESV &&
23602472 root != fs_info->tree_root)
23612473 btrfs_delalloc_release_metadata(inode, len, false);
23622474
@@ -2366,7 +2478,7 @@ void btrfs_clear_delalloc_extent(struct inode *vfs_inode, · 3 unmodified lines
23662478
23672479 if (!btrfs_is_data_reloc_root(root) &&
23682480 do_list && !(state->state & EXTENT_NORESERVE) &&
2369− (*bits & EXTENT_CLEAR_DATA_RESV))
2481+ (bits & EXTENT_CLEAR_DATA_RESV))
23702482 btrfs_free_reserved_data_space_noquota(fs_info, len);
23712483
23722484 percpu_counter_add_batch(&fs_info->delalloc_bytes, -len,
@@ -2381,11 +2493,11 @@ void btrfs_clear_delalloc_extent(struct inode *vfs_inode, · 8 unmodified lines
23812493 }
23822494
23832495 if ((state->state & EXTENT_DELALLOC_NEW) &&
2384− (*bits & EXTENT_DELALLOC_NEW)) {
2496+ (bits & EXTENT_DELALLOC_NEW)) {
23852497 spin_lock(&inode->lock);
23862498 ASSERT(inode->new_delalloc_bytes >= len);
23872499 inode->new_delalloc_bytes -= len;
2388− if (*bits & EXTENT_ADD_INODE_BYTES)
2500+ if (bits & EXTENT_ADD_INODE_BYTES)
23892501 inode_add_bytes(&inode->vfs_inode, len);
23902502 spin_unlock(&inode->lock);
23912503 }
@@ -2580,95 +2692,78 @@ out: · 188 unmodified lines
25802692 return errno_to_blk_status(ret);
25812693 }
25822694
2583−/*
2584− * extent_io.c submission hook. This does the right thing for csum calculation
2585− * on write, or reading the csums from the tree before a read.
2586− *
2587− * Rules about async/sync submit,
2588− * a) read: sync submit
2589− *
2590− * b) write without checksum: sync submit
2591− *
2592− * c) write with checksum:
2593− * c-1) if bio is issued by fsync: sync submit
2594− * (sync_writers != 0)
2595− *
2596− * c-2) if root is reloc root: sync submit
2597− * (only in case of buffered IO)
2598− *
2599− * c-3) otherwise: async submit
2600− */
2601−void btrfs_submit_data_bio(struct inode *inode, struct bio *bio,
2602− int mirror_num, enum btrfs_compression_type compress_type)
2695+void btrfs_submit_data_write_bio(struct inode *inode, struct bio *bio, int mirror_num)
26032696 {
26042697 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
2605− struct btrfs_root *root = BTRFS_I(inode)->root;
2606− enum btrfs_wq_endio_type metadata = BTRFS_WQ_ENDIO_DATA;
2607− blk_status_t ret = 0;
2608− int skip_sum;
2609− int async = !atomic_read(&BTRFS_I(inode)->sync_writers);
2610−
2611− skip_sum = (BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM) ||
2612− test_bit(BTRFS_FS_STATE_NO_CSUMS, &fs_info->fs_state);
2613−
2614− if (btrfs_is_free_space_inode(BTRFS_I(inode)))
2615− metadata = BTRFS_WQ_ENDIO_FREE_SPACE;
2698+ struct btrfs_inode *bi = BTRFS_I(inode);
2699+ blk_status_t ret;
26162700
26172701 if (bio_op(bio) == REQ_OP_ZONE_APPEND) {
2618− struct page *page = bio_first_bvec_all(bio)->bv_page;
2619− loff_t file_offset = page_offset(page);
2620−
2621− ret = extract_ordered_extent(BTRFS_I(inode), bio, file_offset);
2702+ ret = extract_ordered_extent(bi, bio,
2703+ page_offset(bio_first_bvec_all(bio)->bv_page));
26222704 if (ret)
26232705 goto out;
26242706 }
26252707
2626− if (btrfs_op(bio) != BTRFS_MAP_WRITE) {
2627− ret = btrfs_bio_wq_end_io(fs_info, bio, metadata);
2628− if (ret)
2629− goto out;
2630−
2631− if (compress_type != BTRFS_COMPRESS_NONE) {
2632− /*
2633− * btrfs_submit_compressed_read will handle completing
2634− * the bio if there were any errors, so just return
2635− * here.
2636− */
2637− btrfs_submit_compressed_read(inode, bio, mirror_num);
2708+ /*
2709+ * If we need to checksum, and the I/O is not issued by fsync and
2710+ * friends, that is ->sync_writers != 0, defer the submission to a
2711+ * workqueue to parallelize it.
2712+ *
2713+ * Csum items for reloc roots have already been cloned at this point,
2714+ * so they are handled as part of the no-checksum case.
2715+ */
2716+ if (!(bi->flags & BTRFS_INODE_NODATASUM) &&
2717+ !test_bit(BTRFS_FS_STATE_NO_CSUMS, &fs_info->fs_state) &&
2718+ !btrfs_is_data_reloc_root(bi->root)) {
2719+ if (!atomic_read(&bi->sync_writers) &&
2720+ btrfs_wq_submit_bio(inode, bio, mirror_num, 0,
2721+ btrfs_submit_bio_start))
26382722 return;
2639− } else {
2640− /*
2641− * Lookup bio sums does extra checks around whether we
2642− * need to csum or not, which is why we ignore skip_sum
2643− * here.
2644− */
2645− ret = btrfs_lookup_bio_sums(inode, bio, NULL);
2646− if (ret)
2647− goto out;
2648− }
2649− goto mapit;
2650− } else if (async && !skip_sum) {
2651− /* csum items have already been cloned */
2652− if (btrfs_is_data_reloc_root(root))
2653− goto mapit;
2654− /* we're doing a write, do the async checksumming */
2655− ret = btrfs_wq_submit_bio(inode, bio, mirror_num,
2656− 0, btrfs_submit_bio_start);
2657− goto out;
2658− } else if (!skip_sum) {
2659− ret = btrfs_csum_one_bio(BTRFS_I(inode), bio, (u64)-1, false);
2723+
2724+ ret = btrfs_csum_one_bio(bi, bio, (u64)-1, false);
26602725 if (ret)
26612726 goto out;
26622727 }
2728+ btrfs_submit_bio(fs_info, bio, mirror_num);
2729+ return;
2730+out:
2731+ if (ret) {
2732+ bio->bi_status = ret;
2733+ bio_endio(bio);
2734+ }
2735+}
26632736
2664−mapit:
2665− ret = btrfs_map_bio(fs_info, bio, mirror_num);
2737+void btrfs_submit_data_read_bio(struct inode *inode, struct bio *bio,
2738+ int mirror_num, enum btrfs_compression_type compress_type)
2739+{
2740+ struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
2741+ blk_status_t ret;
26662742
2667−out:
2743+ if (compress_type != BTRFS_COMPRESS_NONE) {
2744+ /*
2745+ * btrfs_submit_compressed_read will handle completing the bio
2746+ * if there were any errors, so just return here.
2747+ */
2748+ btrfs_submit_compressed_read(inode, bio, mirror_num);
2749+ return;
2750+ }
2751+
2752+ /* Save the original iter for read repair */
2753+ btrfs_bio(bio)->iter = bio->bi_iter;
2754+
2755+ /*
2756+ * Lookup bio sums does extra checks around whether we need to csum or
2757+ * not, which is why we ignore skip_sum here.
2758+ */
2759+ ret = btrfs_lookup_bio_sums(inode, bio, NULL);
26682760 if (ret) {
26692761 bio->bi_status = ret;
26702762 bio_endio(bio);
2763+ return;
26712764 }
2765+
2766+ btrfs_submit_bio(fs_info, bio, mirror_num);
26722767 }
26732768
26742769 /*
@@ -3075,8 +3170,10 @@ static int insert_ordered_extent_file_extent(struct btrfs_trans_handle *trans, · 400 unmodified lines
30753170 btrfs_set_stack_file_extent_disk_num_bytes(&stack_fi,
30763171 oe->disk_num_bytes);
30773172 btrfs_set_stack_file_extent_offset(&stack_fi, oe->offset);
3078− if (test_bit(BTRFS_ORDERED_TRUNCATED, &oe->flags))
3079− num_bytes = ram_bytes = oe->truncated_len;
3173+ if (test_bit(BTRFS_ORDERED_TRUNCATED, &oe->flags)) {
3174+ num_bytes = oe->truncated_len;
3175+ ram_bytes = num_bytes;
3176+ }
30803177 btrfs_set_stack_file_extent_num_bytes(&stack_fi, num_bytes);
30813178 btrfs_set_stack_file_extent_ram_bytes(&stack_fi, ram_bytes);
30823179 btrfs_set_stack_file_extent_compression(&stack_fi, oe->compress_type);
@@ -3102,7 +3199,7 @@ static int insert_ordered_extent_file_extent(struct btrfs_trans_handle *trans, · 19 unmodified lines
31023199 * an ordered extent if the range of bytes in the file it covers are
31033200 * fully written.
31043201 */
3105−static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)
3202+int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)
31063203 {
31073204 struct btrfs_inode *inode = BTRFS_I(ordered_extent->inode);
31083205 struct btrfs_root *root = inode->root;
@@ -3311,65 +3408,71 @@ out: · 202 unmodified lines
33113408 return ret;
33123409 }
33133410
3314−static void finish_ordered_fn(struct btrfs_work *work)
3315−{
3316− struct btrfs_ordered_extent *ordered_extent;
3317− ordered_extent = container_of(work, struct btrfs_ordered_extent, work);
3318− btrfs_finish_ordered_io(ordered_extent);
3319−}
3320−
33213411 void btrfs_writepage_endio_finish_ordered(struct btrfs_inode *inode,
33223412 struct page *page, u64 start,
33233413 u64 end, bool uptodate)
33243414 {
33253415 trace_btrfs_writepage_end_io_hook(inode, start, end, uptodate);
33263416
3327− btrfs_mark_ordered_io_finished(inode, page, start, end + 1 - start,
3328− finish_ordered_fn, uptodate);
3417+ btrfs_mark_ordered_io_finished(inode, page, start, end + 1 - start, uptodate);
3418+}
3419+
3420+/*
3421+ * Verify the checksum for a single sector without any extra action that depend
3422+ * on the type of I/O.
3423+ */
3424+int btrfs_check_sector_csum(struct btrfs_fs_info *fs_info, struct page *page,
3425+ u32 pgoff, u8 *csum, const u8 * const csum_expected)
3426+{
3427+ SHASH_DESC_ON_STACK(shash, fs_info->csum_shash);
3428+ char *kaddr;
3429+
3430+ ASSERT(pgoff + fs_info->sectorsize <= PAGE_SIZE);
3431+
3432+ shash->tfm = fs_info->csum_shash;
3433+
3434+ kaddr = kmap_local_page(page) + pgoff;
3435+ crypto_shash_digest(shash, kaddr, fs_info->sectorsize, csum);
3436+ kunmap_local(kaddr);
3437+
3438+ if (memcmp(csum, csum_expected, fs_info->csum_size))
3439+ return -EIO;
3440+ return 0;
33293441 }
33303442
33313443 /*
33323444 * check_data_csum - verify checksum of one sector of uncompressed data
33333445 * @inode: inode
3334− * @io_bio: btrfs_io_bio which contains the csum
3446+ * @bbio: btrfs_bio which contains the csum
33353447 * @bio_offset: offset to the beginning of the bio (in bytes)
33363448 * @page: page where is the data to be verified
33373449 * @pgoff: offset inside the page
3338− * @start: logical offset in the file
33393450 *
33403451 * The length of such check is always one sector size.
3452+ *
3453+ * When csum mismatch is detected, we will also report the error and fill the
3454+ * corrupted range with zero. (Thus it needs the extra parameters)
33413455 */
3342−static int check_data_csum(struct inode *inode, struct btrfs_bio *bbio,
3343− u32 bio_offset, struct page *page, u32 pgoff,
3344− u64 start)
3456+int btrfs_check_data_csum(struct inode *inode, struct btrfs_bio *bbio,
3457+ u32 bio_offset, struct page *page, u32 pgoff)
33453458 {
33463459 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
3347− SHASH_DESC_ON_STACK(shash, fs_info->csum_shash);
3348− char *kaddr;
33493460 u32 len = fs_info->sectorsize;
3350− const u32 csum_size = fs_info->csum_size;
3351− unsigned int offset_sectors;
33523461 u8 *csum_expected;
33533462 u8 csum[BTRFS_CSUM_SIZE];
33543463
33553464 ASSERT(pgoff + len <= PAGE_SIZE);
33563465
3357− offset_sectors = bio_offset >> fs_info->sectorsize_bits;
3358− csum_expected = ((u8 *)bbio->csum) + offset_sectors * csum_size;
3359−
3360− kaddr = kmap_atomic(page);
3361− shash->tfm = fs_info->csum_shash;
3362−
3363− crypto_shash_digest(shash, kaddr + pgoff, len, csum);
3364− kunmap_atomic(kaddr);
3466+ csum_expected = btrfs_csum_ptr(fs_info, bbio->csum, bio_offset);
33653467
3366− if (memcmp(csum, csum_expected, csum_size))
3468+ if (btrfs_check_sector_csum(fs_info, page, pgoff, csum, csum_expected))
33673469 goto zeroit;
3368−
33693470 return 0;
3471+
33703472 zeroit:
3371− btrfs_print_data_csum_error(BTRFS_I(inode), start, csum, csum_expected,
3372− bbio->mirror_num);
3473+ btrfs_print_data_csum_error(BTRFS_I(inode),
3474+ bbio->file_offset + bio_offset,
3475+ csum, csum_expected, bbio->mirror_num);
33733476 if (bbio->device)
33743477 btrfs_dev_stat_inc_and_print(bbio->device,
33753478 BTRFS_DEV_STAT_CORRUPTION_ERRS);
@@ -3401,11 +3504,6 @@ unsigned int btrfs_verify_data_csum(struct btrfs_bio *bbio, · 25 unmodified lines
34013504 u32 pg_off;
34023505 unsigned int result = 0;
34033506
3404− if (btrfs_page_test_checked(fs_info, page, start, end + 1 - start)) {
3405− btrfs_page_clear_checked(fs_info, page, start, end + 1 - start);
3406− return 0;
3407− }
3408−
34093507 /*
34103508 * This only happens for NODATASUM or compressed read.
34113509 * Normally this should be covered by above check for compressed read
@@ -3438,8 +3536,7 @@ unsigned int btrfs_verify_data_csum(struct btrfs_bio *bbio, · 26 unmodified lines
34383536 EXTENT_NODATASUM);
34393537 continue;
34403538 }
3441− ret = check_data_csum(inode, bbio, bio_offset, page, pg_off,
3442− page_offset(page) + pg_off);
3539+ ret = btrfs_check_data_csum(inode, bbio, bio_offset, page, pg_off);
34433540 if (ret < 0) {
34443541 const int nr_bit = (pg_off - offset_in_page(start)) >>
34453542 root->fs_info->sectorsize_bits;
@@ -4228,7 +4325,7 @@ skip_backref: · 782 unmodified lines
42284325 /*
42294326 * If we are in a rename context, we don't need to update anything in the
42304327 * log. That will be done later during the rename by btrfs_log_new_name().
4231− * Besides that, doing it here would only cause extra unncessary btree
4328+ * Besides that, doing it here would only cause extra unnecessary btree
42324329 * operations on the log tree, increasing latency for applications.
42334330 */
42344331 if (!rename_ctx) {
@@ -4256,8 +4353,9 @@ err: · 21 unmodified lines
42564353 btrfs_i_size_write(dir, dir->vfs_inode.i_size - name_len * 2);
42574354 inode_inc_iversion(&inode->vfs_inode);
42584355 inode_inc_iversion(&dir->vfs_inode);
4259− inode->vfs_inode.i_ctime = dir->vfs_inode.i_mtime =
4260− dir->vfs_inode.i_ctime = current_time(&inode->vfs_inode);
4356+ inode->vfs_inode.i_ctime = current_time(&inode->vfs_inode);
4357+ dir->vfs_inode.i_mtime = inode->vfs_inode.i_ctime;
4358+ dir->vfs_inode.i_ctime = inode->vfs_inode.i_ctime;
42614359 ret = btrfs_update_inode(trans, root, dir);
42624360 out:
42634361 return ret;
@@ -4419,7 +4517,8 @@ static int btrfs_unlink_subvol(struct btrfs_trans_handle *trans, · 155 unmodified lines
44194517
44204518 btrfs_i_size_write(BTRFS_I(dir), dir->i_size - name_len * 2);
44214519 inode_inc_iversion(dir);
4422− dir->i_mtime = dir->i_ctime = current_time(dir);
4520+ dir->i_mtime = current_time(dir);
4521+ dir->i_ctime = dir->i_mtime;
44234522 ret = btrfs_update_inode_fallback(trans, root, BTRFS_I(dir));
44244523 if (ret)
44254524 btrfs_abort_transaction(trans, ret);
@@ -4858,7 +4957,6 @@ again: · 432 unmodified lines
48584957 else
48594958 memzero_page(page, (block_start - page_offset(page)) + offset,
48604959 len);
4861− flush_dcache_page(page);
48624960 }
48634961 btrfs_page_clear_checked(fs_info, page, block_start,
48644962 block_end + 1 - block_start);
@@ -5061,9 +5159,10 @@ static int btrfs_setsize(struct inode *inode, struct iattr *attr) · 196 unmodified lines
50615159 */
50625160 if (newsize != oldsize) {
50635161 inode_inc_iversion(inode);
5064− if (!(mask & (ATTR_CTIME | ATTR_MTIME)))
5065− inode->i_ctime = inode->i_mtime =
5066− current_time(inode);
5162+ if (!(mask & (ATTR_CTIME | ATTR_MTIME))) {
5163+ inode->i_mtime = current_time(inode);
5164+ inode->i_ctime = inode->i_mtime;
5165+ }
50675166 }
50685167
50695168 if (newsize > oldsize) {
@@ -5371,7 +5470,7 @@ void btrfs_evict_inode(struct inode *inode) · 301 unmodified lines
53715470 if (!rsv)
53725471 goto no_delete;
53735472 rsv->size = btrfs_calc_metadata_size(fs_info, 1);
5374− rsv->failfast = 1;
5473+ rsv->failfast = true;
53755474
53765475 btrfs_i_size_write(BTRFS_I(inode), 0);
53775476
@@ -5763,14 +5862,14 @@ struct inode *btrfs_lookup_dentry(struct inode *dir, struct dentry *dentry) · 385 unmodified lines
57635862 if (ret != -ENOENT)
57645863 inode = ERR_PTR(ret);
57655864 else
5766− inode = new_simple_dir(dir->i_sb, &location, sub_root);
5865+ inode = new_simple_dir(dir->i_sb, &location, root);
57675866 } else {
57685867 inode = btrfs_iget(dir->i_sb, location.objectid, sub_root);
5769− }
5770− if (root != sub_root)
57715868 btrfs_put_root(sub_root);
57725869
5773− if (!IS_ERR(inode) && root != sub_root) {
5870+ if (IS_ERR(inode))
5871+ return inode;
5872+
57745873 down_read(&fs_info->cleanup_work_sem);
57755874 if (!sb_rdonly(inode->i_sb))
57765875 ret = btrfs_orphan_cleanup(sub_root);
@@ -6366,7 +6465,13 @@ int btrfs_create_new_inode(struct btrfs_trans_handle *trans, · 589 unmodified lines
63666465 }
63676466
63686467 btrfs_mark_buffer_dirty(path->nodes[0]);
6369− btrfs_release_path(path);
6468+ /*
6469+ * We don't need the path anymore, plus inheriting properties, adding
6470+ * ACLs, security xattrs, orphan item or adding the link, will result in
6471+ * allocating yet another path. So just free our path.
6472+ */
6473+ btrfs_free_path(path);
6474+ path = NULL;
63706475
63716476 if (args->subvol) {
63726477 struct inode *parent;
@@ -6423,8 +6528,7 @@ int btrfs_create_new_inode(struct btrfs_trans_handle *trans, · 50 unmodified lines
64236528 goto discard;
64246529 }
64256530
6426− ret = 0;
6427− goto out;
6531+ return 0;
64286532
64296533 discard:
64306534 /*
@@ -7506,7 +7610,8 @@ static int btrfs_get_blocks_direct_write(struct extent_map **map, · 1075 unmodified lines
75067610 btrfs_dec_nocow_writers(bg);
75077611 if (type == BTRFS_ORDERED_PREALLOC) {
75087612 free_extent_map(em);
7509− *map = em = em2;
7613+ *map = em2;
7614+ em = em2;
75107615 }
75117616
75127617 if (IS_ERR(em2)) {
@@ -7588,8 +7693,12 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start, · 75 unmodified lines
75887693 const u64 data_alloc_len = length;
75897694 bool unlock_extents = false;
75907695
7696+ /*
7697+ * Cap the size of reads to that usually seen in buffered I/O as we need
7698+ * to allocate a contiguous array for the checksums.
7699+ */
75917700 if (!write)
7592− len = min_t(u64, len, fs_info->sectorsize);
7701+ len = min_t(u64, len, fs_info->sectorsize * BTRFS_MAX_BIO_SECTORS);
75937702
75947703 lockstart = start;
75957704 lockend = start + len - 1;
@@ -7824,8 +7933,8 @@ static int btrfs_dio_iomap_end(struct inode *inode, loff_t pos, loff_t length, · 228 unmodified lines
78247933 pos += submitted;
78257934 length -= submitted;
78267935 if (write)
7827− __endio_write_update_ordered(BTRFS_I(inode), pos,
7828− length, false);
7936+ btrfs_mark_ordered_io_finished(BTRFS_I(inode), NULL,
7937+ pos, length, false);
78297938 else
78307939 unlock_extent(&BTRFS_I(inode)->io_tree, pos,
78317940 pos + length - 1);
@@ -7847,10 +7956,9 @@ static void btrfs_dio_private_put(struct btrfs_dio_private *dip) · 15 unmodified lines
78477956 return;
78487957
78497958 if (btrfs_op(&dip->bio) == BTRFS_MAP_WRITE) {
7850− __endio_write_update_ordered(BTRFS_I(dip->inode),
7851− dip->file_offset,
7852− dip->bytes,
7853− !dip->bio.bi_status);
7959+ btrfs_mark_ordered_io_finished(BTRFS_I(dip->inode), NULL,
7960+ dip->file_offset, dip->bytes,
7961+ !dip->bio.bi_status);
78547962 } else {
78557963 unlock_extent(&BTRFS_I(dip->inode)->io_tree,
78567964 dip->file_offset,
@@ -7870,12 +7978,8 @@ static void submit_dio_repair_bio(struct inode *inode, struct bio *bio, · 13 unmodified lines
78707978
78717979 BUG_ON(bio_op(bio) == REQ_OP_WRITE);
78727980
7873− if (btrfs_bio_wq_end_io(fs_info, bio, BTRFS_WQ_ENDIO_DATA))
7874− return;
7875−
78767981 refcount_inc(&dip->refs);
7877− if (btrfs_map_bio(fs_info, bio, mirror_num))
7878− refcount_dec(&dip->refs);
7982+ btrfs_submit_bio(fs_info, bio, mirror_num);
78797983 }
78807984
78817985 static blk_status_t btrfs_check_read_dio_bio(struct btrfs_dio_private *dip,
@@ -7884,56 +7988,35 @@ static blk_status_t btrfs_check_read_dio_bio(struct btrfs_dio_private *dip, · 2 unmodified lines
78847988 {
78857989 struct inode *inode = dip->inode;
78867990 struct btrfs_fs_info *fs_info = BTRFS_I(inode)->root->fs_info;
7887− const u32 sectorsize = fs_info->sectorsize;
78887991 struct extent_io_tree *failure_tree = &BTRFS_I(inode)->io_failure_tree;
78897992 struct extent_io_tree *io_tree = &BTRFS_I(inode)->io_tree;
78907993 const bool csum = !(BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM);
7891− struct bio_vec bvec;
7892− struct bvec_iter iter;
7893− u32 bio_offset = 0;
78947994 blk_status_t err = BLK_STS_OK;
7995+ struct bvec_iter iter;
7996+ struct bio_vec bv;
7997+ u32 offset;
7998+
7999+ btrfs_bio_for_each_sector(fs_info, bv, bbio, iter, offset) {
8000+ u64 start = bbio->file_offset + offset;
8001+
8002+ if (uptodate &&
8003+ (!csum || !btrfs_check_data_csum(inode, bbio, offset, bv.bv_page,
8004+ bv.bv_offset))) {
8005+ clean_io_failure(fs_info, failure_tree, io_tree, start,
8006+ bv.bv_page, btrfs_ino(BTRFS_I(inode)),
8007+ bv.bv_offset);
8008+ } else {
8009+ int ret;
78958010
7896− __bio_for_each_segment(bvec, &bbio->bio, iter, bbio->iter) {
7897− unsigned int i, nr_sectors, pgoff;
7898−
7899− nr_sectors = BTRFS_BYTES_TO_BLKS(fs_info, bvec.bv_len);
7900− pgoff = bvec.bv_offset;
7901− for (i = 0; i < nr_sectors; i++) {
7902− u64 start = bbio->file_offset + bio_offset;
7903−
7904− ASSERT(pgoff < PAGE_SIZE);
7905− if (uptodate &&
7906− (!csum || !check_data_csum(inode, bbio,
7907− bio_offset, bvec.bv_page,
7908− pgoff, start))) {
7909− clean_io_failure(fs_info, failure_tree, io_tree,
7910− start, bvec.bv_page,
7911− btrfs_ino(BTRFS_I(inode)),
7912− pgoff);
7913− } else {
7914− int ret;
7915−
7916− ret = btrfs_repair_one_sector(inode, &bbio->bio,
7917− bio_offset, bvec.bv_page, pgoff,
7918− start, bbio->mirror_num,
7919− submit_dio_repair_bio);
7920− if (ret)
7921− err = errno_to_blk_status(ret);
7922− }
7923− ASSERT(bio_offset + sectorsize > bio_offset);
7924− bio_offset += sectorsize;
7925− pgoff += sectorsize;
8011+ ret = btrfs_repair_one_sector(inode, bbio, offset,
8012+ bv.bv_page, bv.bv_offset,
8013+ submit_dio_repair_bio);
8014+ if (ret)
8015+ err = errno_to_blk_status(ret);
79268016 }
79278017 }
7928− return err;
7929−}
79308018
7931−static void __endio_write_update_ordered(struct btrfs_inode *inode,
7932− const u64 offset, const u64 bytes,
7933− const bool uptodate)
7934−{
7935− btrfs_mark_ordered_io_finished(inode, NULL, offset, bytes,
7936− finish_ordered_fn, uptodate);
8019+ return err;
79378020 }
79388021
79398022 static blk_status_t btrfs_submit_bio_start_direct_io(struct inode *inode,
@@ -7968,51 +8051,43 @@ static void btrfs_end_dio_bio(struct bio *bio) · 28 unmodified lines
79688051 btrfs_dio_private_put(dip);
79698052 }
79708053
7971−static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio,
7972− struct inode *inode, u64 file_offset, int async_submit)
8054+static void btrfs_submit_dio_bio(struct bio *bio, struct inode *inode,
8055+ u64 file_offset, int async_submit)
79738056 {
79748057 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
79758058 struct btrfs_dio_private *dip = bio->bi_private;
7976− bool write = btrfs_op(bio) == BTRFS_MAP_WRITE;
79778059 blk_status_t ret;
79788060
7979− /* Check btrfs_submit_bio_hook() for rules about async submit. */
7980− if (async_submit)
7981− async_submit = !atomic_read(&BTRFS_I(inode)->sync_writers);
7982−
7983− if (!write) {
7984− ret = btrfs_bio_wq_end_io(fs_info, bio, BTRFS_WQ_ENDIO_DATA);
7985− if (ret)
7986− goto err;
7987− }
8061+ /* Save the original iter for read repair */
8062+ if (btrfs_op(bio) == BTRFS_MAP_READ)
8063+ btrfs_bio(bio)->iter = bio->bi_iter;
79888064
79898065 if (BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM)
79908066 goto map;
79918067
7992− if (write && async_submit) {
7993− ret = btrfs_wq_submit_bio(inode, bio, 0, file_offset,
7994− btrfs_submit_bio_start_direct_io);
7995− goto err;
7996− } else if (write) {
8068+ if (btrfs_op(bio) == BTRFS_MAP_WRITE) {
8069+ /* Check btrfs_submit_data_write_bio() for async submit rules */
8070+ if (async_submit && !atomic_read(&BTRFS_I(inode)->sync_writers) &&
8071+ btrfs_wq_submit_bio(inode, bio, 0, file_offset,
8072+ btrfs_submit_bio_start_direct_io))
8073+ return;
8074+
79978075 /*
79988076 * If we aren't doing async submit, calculate the csum of the
79998077 * bio now.
80008078 */
80018079 ret = btrfs_csum_one_bio(BTRFS_I(inode), bio, file_offset, false);
8002− if (ret)
8003− goto err;
8080+ if (ret) {
8081+ bio->bi_status = ret;
8082+ bio_endio(bio);
8083+ return;
8084+ }
80048085 } else {
8005− u64 csum_offset;
8006−
8007− csum_offset = file_offset - dip->file_offset;
8008− csum_offset >>= fs_info->sectorsize_bits;
8009− csum_offset *= fs_info->csum_size;
8010− btrfs_bio(bio)->csum = dip->csums + csum_offset;
8086+ btrfs_bio(bio)->csum = btrfs_csum_ptr(fs_info, dip->csums,
8087+ file_offset - dip->file_offset);
80118088 }
80128089 map:
8013− ret = btrfs_map_bio(fs_info, bio, 0);
8014−err:
8015− return ret;
8090+ btrfs_submit_bio(fs_info, bio, 0);
80168091 }
80178092
80188093 static void btrfs_submit_direct(const struct iomap_iter *iter,
@@ -8125,14 +8200,7 @@ static void btrfs_submit_direct(const struct iomap_iter *iter, · 106 unmodified lines
81258200 async_submit = 1;
81268201 }
81278202
8128− status = btrfs_submit_dio_bio(bio, inode, file_offset,
8129− async_submit);
8130− if (status) {
8131− bio_put(bio);
8132− if (submit_len > 0)
8133− refcount_dec(&dip->refs);
8134− goto out_err_em;
8135− }
8203+ btrfs_submit_dio_bio(bio, inode, file_offset, async_submit);
81368204
81378205 dio_data->submitted += clone_len;
81388206 clone_offset += clone_len;
@@ -8181,31 +8249,6 @@ static int btrfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, · 42 unmodified lines
81818249 return extent_fiemap(BTRFS_I(inode), fieinfo, start, len);
81828250 }
81838251
8184−static int btrfs_writepage(struct page *page, struct writeback_control *wbc)
8185−{
8186− struct inode *inode = page->mapping->host;
8187− int ret;
8188−
8189− if (current->flags & PF_MEMALLOC) {
8190− redirty_page_for_writepage(wbc, page);
8191− unlock_page(page);
8192− return 0;
8193− }
8194−
8195− /*
8196− * If we are under memory pressure we will call this directly from the
8197− * VM, we need to make sure we have the inode referenced for the ordered
8198− * extent. If not just return like we didn't do anything.
8199− */
8200− if (!igrab(inode)) {
8201− redirty_page_for_writepage(wbc, page);
8202− return AOP_WRITEPAGE_ACTIVATE;
8203− }
8204− ret = extent_write_full_page(page, wbc);
8205− btrfs_add_delayed_iput(inode);
8206− return ret;
8207−}
8208−
82098252 static int btrfs_writepages(struct address_space *mapping,
82108253 struct writeback_control *wbc)
82118254 {
@@ -8503,7 +8546,7 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) · 291 unmodified lines
85038546 * Reserving delalloc space after obtaining the page lock can lead to
85048547 * deadlock. For example, if a dirty page is locked by this function
85058548 * and the call to btrfs_delalloc_reserve_space() ends up triggering
8506− * dirty page write out, then the btrfs_writepage() function could
8549+ * dirty page write out, then the btrfs_writepages() function could
85078550 * end up waiting indefinitely to get a lock on the page currently
85088551 * being processed by btrfs_page_mkwrite() function.
85098552 */
@@ -8594,10 +8637,9 @@ again: · 84 unmodified lines
85948637 else
85958638 zero_start = PAGE_SIZE;
85968639
8597− if (zero_start != PAGE_SIZE) {
8640+ if (zero_start != PAGE_SIZE)
85988641 memzero_page(page, zero_start, PAGE_SIZE - zero_start);
8599− flush_dcache_page(page);
8600− }
8642+
86018643 btrfs_page_clear_checked(fs_info, page, page_start, PAGE_SIZE);
86028644 btrfs_page_set_dirty(fs_info, page, page_start, end + 1 - page_start);
86038645 btrfs_page_set_uptodate(fs_info, page, page_start, end + 1 - page_start);
@@ -8680,7 +8722,7 @@ static int btrfs_truncate(struct inode *inode, bool skip_writeback) · 76 unmodified lines
86808722 if (!rsv)
86818723 return -ENOMEM;
86828724 rsv->size = min_size;
8683− rsv->failfast = 1;
8725+ rsv->failfast = true;
86848726
86858727 /*
86868728 * 1 for the truncate slack space
@@ -9201,8 +9243,10 @@ static int btrfs_rename_exchange(struct inode *old_dir, · 514 unmodified lines
92019243 inode_inc_iversion(new_dir);
92029244 inode_inc_iversion(old_inode);
92039245 inode_inc_iversion(new_inode);
9204− old_dir->i_ctime = old_dir->i_mtime = ctime;
9205− new_dir->i_ctime = new_dir->i_mtime = ctime;
9246+ old_dir->i_mtime = ctime;
9247+ old_dir->i_ctime = ctime;
9248+ new_dir->i_mtime = ctime;
9249+ new_dir->i_ctime = ctime;
92069250 old_inode->i_ctime = ctime;
92079251 new_inode->i_ctime = ctime;
92089252
@@ -9465,9 +9509,11 @@ static int btrfs_rename(struct user_namespace *mnt_userns, · 256 unmodified lines
94659509 inode_inc_iversion(old_dir);
94669510 inode_inc_iversion(new_dir);
94679511 inode_inc_iversion(old_inode);
9468− old_dir->i_ctime = old_dir->i_mtime =
9469− new_dir->i_ctime = new_dir->i_mtime =
9470− old_inode->i_ctime = current_time(old_dir);
9512+ old_dir->i_mtime = current_time(old_dir);
9513+ old_dir->i_ctime = old_dir->i_mtime;
9514+ new_dir->i_mtime = old_dir->i_mtime;
9515+ new_dir->i_ctime = old_dir->i_mtime;
9516+ old_inode->i_ctime = old_dir->i_mtime;
94719517
94729518 if (old_dentry->d_parent != new_dentry->d_parent)
94739519 btrfs_record_unlink_dir(trans, BTRFS_I(old_dir),
@@ -9555,15 +9601,21 @@ static int btrfs_rename2(struct user_namespace *mnt_userns, struct inode *old_di · 81 unmodified lines
95559601 struct dentry *old_dentry, struct inode *new_dir,
95569602 struct dentry *new_dentry, unsigned int flags)
95579603 {
9604+ int ret;
9605+
95589606 if (flags & ~(RENAME_NOREPLACE | RENAME_EXCHANGE | RENAME_WHITEOUT))
95599607 return -EINVAL;
95609608
95619609 if (flags & RENAME_EXCHANGE)
9562− return btrfs_rename_exchange(old_dir, old_dentry, new_dir,
9563− new_dentry);
9610+ ret = btrfs_rename_exchange(old_dir, old_dentry, new_dir,
9611+ new_dentry);
9612+ else
9613+ ret = btrfs_rename(mnt_userns, old_dir, old_dentry, new_dir,
9614+ new_dentry, flags);
95649615
9565− return btrfs_rename(mnt_userns, old_dir, old_dentry, new_dir,
9566− new_dentry, flags);
9616+ btrfs_btree_balance_dirty(BTRFS_I(new_dir)->root->fs_info);
9617+
9618+ return ret;
95679619 }
95689620
95699621 struct btrfs_delalloc_work {
@@ -10183,9 +10235,8 @@ void btrfs_set_range_writeback(struct btrfs_inode *inode, u64 start, u64 end) · 613 unmodified lines
1018310235 }
1018410236 }
1018510237
10186−static int btrfs_encoded_io_compression_from_extent(
10187− struct btrfs_fs_info *fs_info,
10188− int compress_type)
10238+int btrfs_encoded_io_compression_from_extent(struct btrfs_fs_info *fs_info,
10239+ int compress_type)
1018910240 {
1019010241 switch (compress_type) {
1019110242 case BTRFS_COMPRESS_NONE:
@@ -10308,7 +10359,6 @@ static blk_status_t submit_encoded_read_bio(struct btrfs_inode *inode, · 116 unmodified lines
1030810359 struct bio *bio, int mirror_num)
1030910360 {
1031010361 struct btrfs_encoded_read_private *priv = bio->bi_private;
10311− struct btrfs_bio *bbio = btrfs_bio(bio);
1031210362 struct btrfs_fs_info *fs_info = inode->root->fs_info;
1031310363 blk_status_t ret;
1031410364
@@ -10318,19 +10368,9 @@ static blk_status_t submit_encoded_read_bio(struct btrfs_inode *inode, · 3 unmodified lines
1031810368 return ret;
1031910369 }
1032010370
10321− ret = btrfs_bio_wq_end_io(fs_info, bio, BTRFS_WQ_ENDIO_DATA);
10322− if (ret) {
10323− btrfs_bio_free_csum(bbio);
10324− return ret;
10325− }
10326−
1032710371 atomic_inc(&priv->pending);
10328− ret = btrfs_map_bio(fs_info, bio, mirror_num);
10329− if (ret) {
10330− atomic_dec(&priv->pending);
10331− btrfs_bio_free_csum(bbio);
10332− }
10333− return ret;
10372+ btrfs_submit_bio(fs_info, bio, mirror_num);
10373+ return BLK_STS_OK;
1033410374 }
1033510375
1033610376 static blk_status_t btrfs_encoded_read_verify_csum(struct btrfs_bio *bbio)
@@ -10342,7 +10382,6 @@ static blk_status_t btrfs_encoded_read_verify_csum(struct btrfs_bio *bbio) · 5 unmodified lines
1034210382 u32 sectorsize = fs_info->sectorsize;
1034310383 struct bio_vec *bvec;
1034410384 struct bvec_iter_all iter_all;
10345− u64 start = priv->file_offset;
1034610385 u32 bio_offset = 0;
1034710386
1034810387 if (priv->skip_csum || !uptodate)
@@ -10355,10 +10394,9 @@ static blk_status_t btrfs_encoded_read_verify_csum(struct btrfs_bio *bbio) · 6 unmodified lines
1035510394 pgoff = bvec->bv_offset;
1035610395 for (i = 0; i < nr_sectors; i++) {
1035710396 ASSERT(pgoff < PAGE_SIZE);
10358− if (check_data_csum(&inode->vfs_inode, bbio, bio_offset,
10359− bvec->bv_page, pgoff, start))
10397+ if (btrfs_check_data_csum(&inode->vfs_inode, bbio, bio_offset,
10398+ bvec->bv_page, pgoff))
1036010399 return BLK_STS_IOERR;
10361− start += sectorsize;
1036210400 bio_offset += sectorsize;
1036310401 pgoff += sectorsize;
1036410402 }
@@ -10390,11 +10428,9 @@ static void btrfs_encoded_read_endio(struct bio *bio) · 25 unmodified lines
1039010428 bio_put(bio);
1039110429 }
1039210430
10393−static int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode,
10394− u64 file_offset,
10395− u64 disk_bytenr,
10396− u64 disk_io_size,
10397− struct page **pages)
10431+int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode,
10432+ u64 file_offset, u64 disk_bytenr,
10433+ u64 disk_io_size, struct page **pages)
1039810434 {
1039910435 struct btrfs_fs_info *fs_info = inode->root->fs_info;
1040010436 struct btrfs_encoded_read_private priv = {
@@ -10625,7 +10661,8 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, · 224 unmodified lines
1062510661 ret = -ENOBUFS;
1062610662 goto out_em;
1062710663 }
10628− disk_io_size = count = em->block_len;
10664+ disk_io_size = em->block_len;
10665+ count = em->block_len;
1062910666 encoded->unencoded_len = em->ram_bytes;
1063010667 encoded->unencoded_offset = iocb->ki_pos - em->orig_start;
1063110668 ret = btrfs_encoded_io_compression_from_extent(fs_info,
@@ -10788,15 +10825,15 @@ ssize_t btrfs_do_encoded_write(struct kiocb *iocb, struct iov_iter *from, · 156 unmodified lines
1078810825 ret = -ENOMEM;
1078910826 goto out_pages;
1079010827 }
10791− kaddr = kmap(pages[i]);
10828+ kaddr = kmap_local_page(pages[i]);
1079210829 if (copy_from_iter(kaddr, bytes, from) != bytes) {
10793− kunmap(pages[i]);
10830+ kunmap_local(kaddr);
1079410831 ret = -EFAULT;
1079510832 goto out_pages;
1079610833 }
1079710834 if (bytes < PAGE_SIZE)
1079810835 memset(kaddr + bytes, 0, PAGE_SIZE - bytes);
10799− kunmap(pages[i]);
10836+ kunmap_local(kaddr);
1080010837 }
1080110838
1080210839 for (;;) {
@@ -11425,7 +11462,6 @@ static const struct file_operations btrfs_dir_file_operations = { · 622 unmodified lines
1142511462 */
1142611463 static const struct address_space_operations btrfs_aops = {
1142711464 .read_folio = btrfs_read_folio,
11428− .writepage = btrfs_writepage,
1142911465 .writepages = btrfs_writepages,
1143011466 .readahead = btrfs_readahead,
1143111467 .direct_IO = noop_direct_IO,
fs/btrfs/ioctl.c+79−71
@@ -1230,16 +1230,18 @@ static struct extent_map *defrag_lookup_extent(struct inode *inode, u64 start, · 1229 unmodified lines
12301230 return em;
12311231 }
12321232
1233−static u32 get_extent_max_capacity(const struct extent_map *em)
1233+static u32 get_extent_max_capacity(const struct btrfs_fs_info *fs_info,
1234+ const struct extent_map *em)
12341235 {
12351236 if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags))
12361237 return BTRFS_MAX_COMPRESSED;
1237− return BTRFS_MAX_EXTENT_SIZE;
1238+ return fs_info->max_extent_size;
12381239 }
12391240
12401241 static bool defrag_check_next_extent(struct inode *inode, struct extent_map *em,
12411242 u32 extent_thresh, u64 newer_than, bool locked)
12421243 {
1244+ struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
12431245 struct extent_map *next;
12441246 bool ret = false;
12451247
@@ -1263,7 +1265,7 @@ static bool defrag_check_next_extent(struct inode *inode, struct extent_map *em, · 17 unmodified lines
12631265 * If the next extent is at its max capacity, defragging current extent
12641266 * makes no sense, as the total number of extents won't change.
12651267 */
1266− if (next->len >= get_extent_max_capacity(em))
1268+ if (next->len >= get_extent_max_capacity(fs_info, em))
12671269 goto out;
12681270 /* Skip older extent */
12691271 if (next->generation < newer_than)
@@ -1400,6 +1402,7 @@ static int defrag_collect_targets(struct btrfs_inode *inode, · 130 unmodified lines
14001402 bool locked, struct list_head *target_list,
14011403 u64 *last_scanned_ret)
14021404 {
1405+ struct btrfs_fs_info *fs_info = inode->root->fs_info;
14031406 bool last_is_target = false;
14041407 u64 cur = start;
14051408 int ret = 0;
@@ -1484,7 +1487,7 @@ static int defrag_collect_targets(struct btrfs_inode *inode, · 78 unmodified lines
14841487 * Skip extents already at its max capacity, this is mostly for
14851488 * compressed extents, which max cap is only 128K.
14861489 */
1487− if (em->len >= get_extent_max_capacity(em))
1490+ if (em->len >= get_extent_max_capacity(fs_info, em))
14881491 goto next;
14891492
14901493 /*
@@ -4243,26 +4246,6 @@ out: · 2752 unmodified lines
42434246 return ret;
42444247 }
42454248
4246−static int build_ino_list(u64 inum, u64 offset, u64 root, void *ctx)
4247−{
4248− struct btrfs_data_container *inodes = ctx;
4249− const size_t c = 3 * sizeof(u64);
4250−
4251− if (inodes->bytes_left >= c) {
4252− inodes->bytes_left -= c;
4253− inodes->val[inodes->elem_cnt] = inum;
4254− inodes->val[inodes->elem_cnt + 1] = offset;
4255− inodes->val[inodes->elem_cnt + 2] = root;
4256− inodes->elem_cnt += 3;
4257− } else {
4258− inodes->bytes_missing += c - inodes->bytes_left;
4259− inodes->bytes_left = 0;
4260− inodes->elem_missed += 3;
4261− }
4262−
4263− return 0;
4264−}
4265−
42664249 static long btrfs_ioctl_logical_to_ino(struct btrfs_fs_info *fs_info,
42674250 void __user *arg, int version)
42684251 {
@@ -4312,7 +4295,7 @@ static long btrfs_ioctl_logical_to_ino(struct btrfs_fs_info *fs_info, · 43 unmodified lines
43124295 }
43134296
43144297 ret = iterate_inodes_from_logical(loi->logical, fs_info, path,
4315− build_ino_list, inodes, ignore_offset);
4298+ inodes, ignore_offset);
43164299 if (ret == -EINVAL)
43174300 ret = -ENOENT;
43184301 if (ret < 0)
@@ -4355,13 +4338,79 @@ void btrfs_update_ioctl_balance_args(struct btrfs_fs_info *fs_info, · 36 unmodified lines
43554338 spin_unlock(&fs_info->balance_lock);
43564339 }
43574340
4341+/**
4342+ * Try to acquire fs_info::balance_mutex as well as set BTRFS_EXLCOP_BALANCE as
4343+ * required.
4344+ *
4345+ * @fs_info: the filesystem
4346+ * @excl_acquired: ptr to boolean value which is set to false in case balance
4347+ * is being resumed
4348+ *
4349+ * Return 0 on success in which case both fs_info::balance is acquired as well
4350+ * as exclusive ops are blocked. In case of failure return an error code.
4351+ */
4352+static int btrfs_try_lock_balance(struct btrfs_fs_info *fs_info, bool *excl_acquired)
4353+{
4354+ int ret;
4355+
4356+ /*
4357+ * Exclusive operation is locked. Three possibilities:
4358+ * (1) some other op is running
4359+ * (2) balance is running
4360+ * (3) balance is paused -- special case (think resume)
4361+ */
4362+ while (1) {
4363+ if (btrfs_exclop_start(fs_info, BTRFS_EXCLOP_BALANCE)) {
4364+ *excl_acquired = true;
4365+ mutex_lock(&fs_info->balance_mutex);
4366+ return 0;
4367+ }
4368+
4369+ mutex_lock(&fs_info->balance_mutex);
4370+ if (fs_info->balance_ctl) {
4371+ /* This is either (2) or (3) */
4372+ if (test_bit(BTRFS_FS_BALANCE_RUNNING, &fs_info->flags)) {
4373+ /* This is (2) */
4374+ ret = -EINPROGRESS;
4375+ goto out_failure;
4376+
4377+ } else {
4378+ mutex_unlock(&fs_info->balance_mutex);
4379+ /*
4380+ * Lock released to allow other waiters to
4381+ * continue, we'll reexamine the status again.
4382+ */
4383+ mutex_lock(&fs_info->balance_mutex);
4384+
4385+ if (fs_info->balance_ctl &&
4386+ !test_bit(BTRFS_FS_BALANCE_RUNNING, &fs_info->flags)) {
4387+ /* This is (3) */
4388+ *excl_acquired = false;
4389+ return 0;
4390+ }
4391+ }
4392+ } else {
4393+ /* This is (1) */
4394+ ret = BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS;
4395+ goto out_failure;
4396+ }
4397+
4398+ mutex_unlock(&fs_info->balance_mutex);
4399+ }
4400+
4401+out_failure:
4402+ mutex_unlock(&fs_info->balance_mutex);
4403+ *excl_acquired = false;
4404+ return ret;
4405+}
4406+
43584407 static long btrfs_ioctl_balance(struct file *file, void __user *arg)
43594408 {
43604409 struct btrfs_root *root = BTRFS_I(file_inode(file))->root;
43614410 struct btrfs_fs_info *fs_info = root->fs_info;
43624411 struct btrfs_ioctl_balance_args *bargs;
43634412 struct btrfs_balance_control *bctl;
4364− bool need_unlock; /* for mut. excl. ops lock */
4413+ bool need_unlock = true;
43654414 int ret;
43664415
43674416 if (!capable(CAP_SYS_ADMIN))
@@ -4378,53 +4427,12 @@ static long btrfs_ioctl_balance(struct file *file, void __user *arg) · 10 unmodified lines
43784427 goto out;
43794428 }
43804429
4381−again:
4382− if (btrfs_exclop_start(fs_info, BTRFS_EXCLOP_BALANCE)) {
4383− mutex_lock(&fs_info->balance_mutex);
4384− need_unlock = true;
4385− goto locked;
4386− }
4387−
4388− /*
4389− * mut. excl. ops lock is locked. Three possibilities:
4390− * (1) some other op is running
4391− * (2) balance is running
4392− * (3) balance is paused -- special case (think resume)
4393− */
4394− mutex_lock(&fs_info->balance_mutex);
4395− if (fs_info->balance_ctl) {
4396− /* this is either (2) or (3) */
4397− if (!test_bit(BTRFS_FS_BALANCE_RUNNING, &fs_info->flags)) {
4398− mutex_unlock(&fs_info->balance_mutex);
4399− /*
4400− * Lock released to allow other waiters to continue,
4401− * we'll reexamine the status again.
4402− */
4403− mutex_lock(&fs_info->balance_mutex);
4404−
4405− if (fs_info->balance_ctl &&
4406− !test_bit(BTRFS_FS_BALANCE_RUNNING, &fs_info->flags)) {
4407− /* this is (3) */
4408− need_unlock = false;
4409− goto locked;
4410− }
4411−
4412− mutex_unlock(&fs_info->balance_mutex);
4413− goto again;
4414− } else {
4415− /* this is (2) */
4416− mutex_unlock(&fs_info->balance_mutex);
4417− ret = -EINPROGRESS;
4418− goto out;
4419− }
4420− } else {
4421− /* this is (1) */
4422− mutex_unlock(&fs_info->balance_mutex);
4423− ret = BTRFS_ERROR_DEV_EXCL_RUN_IN_PROGRESS;
4430+ ret = btrfs_try_lock_balance(fs_info, &need_unlock);
4431+ if (ret)
44244432 goto out;
4425− }
44264433
4427−locked:
4434+ lockdep_assert_held(&fs_info->balance_mutex);
4435+
44284436 if (bargs->flags & BTRFS_BALANCE_RESUME) {
44294437 if (!fs_info->balance_ctl) {
44304438 ret = -ENOTCONN;
fs/btrfs/lzo.c+12−16
@@ -155,7 +155,7 @@ static int copy_compressed_data_to_page(char *compressed_data, · 154 unmodified lines
155155 out_pages[*cur_out / PAGE_SIZE] = cur_page;
156156 }
157157
158− kaddr = kmap(cur_page);
158+ kaddr = kmap_local_page(cur_page);
159159 write_compress_length(kaddr + offset_in_page(*cur_out),
160160 compressed_size);
161161 *cur_out += LZO_LEN;
@@ -167,7 +167,7 @@ static int copy_compressed_data_to_page(char *compressed_data, · 5 unmodified lines
167167 u32 copy_len = min_t(u32, sectorsize - *cur_out % sectorsize,
168168 orig_out + compressed_size - *cur_out);
169169
170− kunmap(cur_page);
170+ kunmap_local(kaddr);
171171
172172 if ((*cur_out / PAGE_SIZE) >= max_nr_page)
173173 return -E2BIG;
@@ -180,7 +180,7 @@ static int copy_compressed_data_to_page(char *compressed_data, · 6 unmodified lines
180180 return -ENOMEM;
181181 out_pages[*cur_out / PAGE_SIZE] = cur_page;
182182 }
183− kaddr = kmap(cur_page);
183+ kaddr = kmap_local_page(cur_page);
184184
185185 memcpy(kaddr + offset_in_page(*cur_out),
186186 compressed_data + *cur_out - orig_out, copy_len);
@@ -202,7 +202,7 @@ static int copy_compressed_data_to_page(char *compressed_data, · 15 unmodified lines
202202 *cur_out += sector_bytes_left;
203203
204204 out:
205− kunmap(cur_page);
205+ kunmap_local(kaddr);
206206 return 0;
207207 }
208208
@@ -248,12 +248,12 @@ int lzo_compress_pages(struct list_head *ws, struct address_space *mapping, · 39 unmodified lines
248248 /* Compress at most one sector of data each time */
249249 in_len = min_t(u32, start + len - cur_in, sectorsize - sector_off);
250250 ASSERT(in_len);
251− data_in = kmap(page_in);
251+ data_in = kmap_local_page(page_in);
252252 ret = lzo1x_1_compress(data_in +
253253 offset_in_page(cur_in), in_len,
254254 workspace->cbuf, &out_len,
255255 workspace->mem);
256− kunmap(page_in);
256+ kunmap_local(data_in);
257257 if (ret < 0) {
258258 pr_debug("BTRFS: lzo in loop returned %d\n", ret);
259259 ret = -EIO;
@@ -310,7 +310,6 @@ static void copy_compressed_segment(struct compressed_bio *cb, · 50 unmodified lines
310310 u32 orig_in = *cur_in;
311311
312312 while (*cur_in < orig_in + len) {
313− char *kaddr;
314313 struct page *cur_page;
315314 u32 copy_len = min_t(u32, PAGE_SIZE - offset_in_page(*cur_in),
316315 orig_in + len - *cur_in);
@@ -318,11 +317,8 @@ static void copy_compressed_segment(struct compressed_bio *cb, · 1 unmodified line
318317 ASSERT(copy_len);
319318 cur_page = cb->compressed_pages[*cur_in / PAGE_SIZE];
320319
321− kaddr = kmap(cur_page);
322− memcpy(dest + *cur_in - orig_in,
323− kaddr + offset_in_page(*cur_in),
324− copy_len);
325− kunmap(cur_page);
320+ memcpy_from_page(dest + *cur_in - orig_in, cur_page,
321+ offset_in_page(*cur_in), copy_len);
326322
327323 *cur_in += copy_len;
328324 }
@@ -342,9 +338,9 @@ int lzo_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 13 unmodified lines
342338 /* Bytes decompressed so far */
343339 u32 cur_out = 0;
344340
345− kaddr = kmap(cb->compressed_pages[0]);
341+ kaddr = kmap_local_page(cb->compressed_pages[0]);
346342 len_in = read_compress_length(kaddr);
347− kunmap(cb->compressed_pages[0]);
343+ kunmap_local(kaddr);
348344 cur_in += LZO_LEN;
349345
350346 /*
@@ -378,9 +374,9 @@ int lzo_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 27 unmodified lines
378374 (cur_in + LZO_LEN - 1) / sectorsize);
379375 cur_page = cb->compressed_pages[cur_in / PAGE_SIZE];
380376 ASSERT(cur_page);
381− kaddr = kmap(cur_page);
377+ kaddr = kmap_local_page(cur_page);
382378 seg_len = read_compress_length(kaddr + offset_in_page(cur_in));
383− kunmap(cur_page);
379+ kunmap_local(kaddr);
384380 cur_in += LZO_LEN;
385381
386382 if (seg_len > WORKSPACE_CBUF_LENGTH) {
fs/btrfs/ordered-data.c+29−11
@@ -272,25 +272,30 @@ void btrfs_add_ordered_sum(struct btrfs_ordered_extent *entry, · 271 unmodified lines
272272 spin_unlock_irq(&tree->lock);
273273 }
274274
275+static void finish_ordered_fn(struct btrfs_work *work)
276+{
277+ struct btrfs_ordered_extent *ordered_extent;
278+
279+ ordered_extent = container_of(work, struct btrfs_ordered_extent, work);
280+ btrfs_finish_ordered_io(ordered_extent);
281+}
282+
275283 /*
276284 * Mark all ordered extents io inside the specified range finished.
277285 *
278− * @page: The invovled page for the opeartion.
286+ * @page: The involved page for the operation.
279287 * For uncompressed buffered IO, the page status also needs to be
280288 * updated to indicate whether the pending ordered io is finished.
281289 * Can be NULL for direct IO and compressed write.
282290 * For these cases, callers are ensured they won't execute the
283291 * endio function twice.
284− * @finish_func: The function to be executed when all the IO of an ordered
285− * extent are finished.
286292 *
287293 * This function is called for endio, thus the range must have ordered
288− * extent(s) coveri it.
294+ * extent(s) covering it.
289295 */
290296 void btrfs_mark_ordered_io_finished(struct btrfs_inode *inode,
291− struct page *page, u64 file_offset,
292− u64 num_bytes, btrfs_func_t finish_func,
293− bool uptodate)
297+ struct page *page, u64 file_offset,
298+ u64 num_bytes, bool uptodate)
294299 {
295300 struct btrfs_ordered_inode_tree *tree = &inode->ordered_tree;
296301 struct btrfs_fs_info *fs_info = inode->root->fs_info;
@@ -401,8 +406,9 @@ void btrfs_mark_ordered_io_finished(struct btrfs_inode *inode, · 104 unmodified lines
401406 set_bit(BTRFS_ORDERED_IO_DONE, &entry->flags);
402407 cond_wake_up(&entry->wait);
403408 refcount_inc(&entry->refs);
409+ trace_btrfs_ordered_extent_mark_finished(inode, entry);
404410 spin_unlock_irqrestore(&tree->lock, flags);
405− btrfs_init_work(&entry->work, finish_func, NULL, NULL);
411+ btrfs_init_work(&entry->work, finish_ordered_fn, NULL, NULL);
406412 btrfs_queue_work(wq, &entry->work);
407413 spin_lock_irqsave(&tree->lock, flags);
408414 }
@@ -473,6 +479,7 @@ out: · 64 unmodified lines
473479 if (finished && cached && entry) {
474480 *cached = entry;
475481 refcount_inc(&entry->refs);
482+ trace_btrfs_ordered_extent_dec_test_pending(inode, entry);
476483 }
477484 spin_unlock_irqrestore(&tree->lock, flags);
478485 return finished;
@@ -807,8 +814,10 @@ struct btrfs_ordered_extent *btrfs_lookup_ordered_extent(struct btrfs_inode *ino · 328 unmodified lines
807814 entry = rb_entry(node, struct btrfs_ordered_extent, rb_node);
808815 if (!in_range(file_offset, entry->file_offset, entry->num_bytes))
809816 entry = NULL;
810− if (entry)
817+ if (entry) {
811818 refcount_inc(&entry->refs);
819+ trace_btrfs_ordered_extent_lookup(inode, entry);
820+ }
812821 out:
813822 spin_unlock_irqrestore(&tree->lock, flags);
814823 return entry;
@@ -848,8 +857,10 @@ struct btrfs_ordered_extent *btrfs_lookup_ordered_range( · 33 unmodified lines
848857 break;
849858 }
850859 out:
851− if (entry)
860+ if (entry) {
852861 refcount_inc(&entry->refs);
862+ trace_btrfs_ordered_extent_lookup_range(inode, entry);
863+ }
853864 spin_unlock_irq(&tree->lock);
854865 return entry;
855866 }
@@ -878,6 +889,7 @@ void btrfs_get_ordered_extents_for_logging(struct btrfs_inode *inode, · 22 unmodified lines
878889 ASSERT(list_empty(&ordered->log_list));
879890 list_add_tail(&ordered->log_list, list);
880891 refcount_inc(&ordered->refs);
892+ trace_btrfs_ordered_extent_lookup_for_logging(inode, ordered);
881893 }
882894 spin_unlock_irq(&tree->lock);
883895 }
@@ -901,6 +913,7 @@ btrfs_lookup_first_ordered_extent(struct btrfs_inode *inode, u64 file_offset) · 17 unmodified lines
901913
902914 entry = rb_entry(node, struct btrfs_ordered_extent, rb_node);
903915 refcount_inc(&entry->refs);
916+ trace_btrfs_ordered_extent_lookup_first(inode, entry);
904917 out:
905918 spin_unlock_irq(&tree->lock);
906919 return entry;
@@ -975,8 +988,11 @@ struct btrfs_ordered_extent *btrfs_lookup_first_ordered_range( · 68 unmodified lines
975988 /* No ordered extent in the range */
976989 entry = NULL;
977990 out:
978− if (entry)
991+ if (entry) {
979992 refcount_inc(&entry->refs);
993+ trace_btrfs_ordered_extent_lookup_first_range(inode, entry);
994+ }
995+
980996 spin_unlock_irq(&tree->lock);
981997 return entry;
982998 }
@@ -1055,6 +1071,8 @@ int btrfs_split_ordered_extent(struct btrfs_ordered_extent *ordered, u64 pre, · 72 unmodified lines
10551071 struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
10561072 int ret = 0;
10571073
1074+ trace_btrfs_ordered_extent_split(BTRFS_I(inode), ordered);
1075+
10581076 spin_lock_irq(&tree->lock);
10591077 /* Remove from tree once */
10601078 node = &ordered->rb_node;
fs/btrfs/ordered-data.h+3−2
@@ -180,13 +180,14 @@ btrfs_ordered_inode_tree_init(struct btrfs_ordered_inode_tree *t) · 179 unmodified lines
180180 t->last = NULL;
181181 }
182182
183+int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent);
184+
183185 void btrfs_put_ordered_extent(struct btrfs_ordered_extent *entry);
184186 void btrfs_remove_ordered_extent(struct btrfs_inode *btrfs_inode,
185187 struct btrfs_ordered_extent *entry);
186188 void btrfs_mark_ordered_io_finished(struct btrfs_inode *inode,
187189 struct page *page, u64 file_offset,
188− u64 num_bytes, btrfs_func_t finish_func,
189− bool uptodate);
190+ u64 num_bytes, bool uptodate);
190191 bool btrfs_dec_test_ordered_pending(struct btrfs_inode *inode,
191192 struct btrfs_ordered_extent **cached,
192193 u64 file_offset, u64 io_size);
fs/btrfs/raid56.c+371−421
@@ -63,137 +63,6 @@ struct sector_ptr { · 62 unmodified lines
6363 unsigned int uptodate:8;
6464 };
6565
66−enum btrfs_rbio_ops {
67− BTRFS_RBIO_WRITE,
68− BTRFS_RBIO_READ_REBUILD,
69− BTRFS_RBIO_PARITY_SCRUB,
70− BTRFS_RBIO_REBUILD_MISSING,
71−};
72−
73−struct btrfs_raid_bio {
74− struct btrfs_io_context *bioc;
75−
76− /* while we're doing rmw on a stripe
77− * we put it into a hash table so we can
78− * lock the stripe and merge more rbios
79− * into it.
80− */
81− struct list_head hash_list;
82−
83− /*
84− * LRU list for the stripe cache
85− */
86− struct list_head stripe_cache;
87−
88− /*
89− * for scheduling work in the helper threads
90− */
91− struct work_struct work;
92−
93− /*
94− * bio list and bio_list_lock are used
95− * to add more bios into the stripe
96− * in hopes of avoiding the full rmw
97− */
98− struct bio_list bio_list;
99− spinlock_t bio_list_lock;
100−
101− /* also protected by the bio_list_lock, the
102− * plug list is used by the plugging code
103− * to collect partial bios while plugged. The
104− * stripe locking code also uses it to hand off
105− * the stripe lock to the next pending IO
106− */
107− struct list_head plug_list;
108−
109− /*
110− * flags that tell us if it is safe to
111− * merge with this bio
112− */
113− unsigned long flags;
114−
115− /*
116− * set if we're doing a parity rebuild
117− * for a read from higher up, which is handled
118− * differently from a parity rebuild as part of
119− * rmw
120− */
121− enum btrfs_rbio_ops operation;
122−
123− /* Size of each individual stripe on disk */
124− u32 stripe_len;
125−
126− /* How many pages there are for the full stripe including P/Q */
127− u16 nr_pages;
128−
129− /* How many sectors there are for the full stripe including P/Q */
130− u16 nr_sectors;
131−
132− /* Number of data stripes (no p/q) */
133− u8 nr_data;
134−
135− /* Numer of all stripes (including P/Q) */
136− u8 real_stripes;
137−
138− /* How many pages there are for each stripe */
139− u8 stripe_npages;
140−
141− /* How many sectors there are for each stripe */
142− u8 stripe_nsectors;
143−
144− /* First bad stripe, -1 means no corruption */
145− s8 faila;
146−
147− /* Second bad stripe (for RAID6 use) */
148− s8 failb;
149−
150− /* Stripe number that we're scrubbing */
151− u8 scrubp;
152−
153− /*
154− * size of all the bios in the bio_list. This
155− * helps us decide if the rbio maps to a full
156− * stripe or not
157− */
158− int bio_list_bytes;
159−
160− int generic_bio_cnt;
161−
162− refcount_t refs;
163−
164− atomic_t stripes_pending;
165−
166− atomic_t error;
167− /*
168− * these are two arrays of pointers. We allocate the
169− * rbio big enough to hold them both and setup their
170− * locations when the rbio is allocated
171− */
172−
173− /* pointers to pages that we allocated for
174− * reading/writing stripes directly from the disk (including P/Q)
175− */
176− struct page **stripe_pages;
177−
178− /* Pointers to the sectors in the bio_list, for faster lookup */
179− struct sector_ptr *bio_sectors;
180−
181− /*
182− * For subpage support, we need to map each sector to above
183− * stripe_pages.
184− */
185− struct sector_ptr *stripe_sectors;
186−
187− /* Bitmap to record which horizontal stripe has data */
188− unsigned long *dbitmap;
189−
190− /* allocated with real_stripes-many pointers for finish_*() calls */
191− void **finish_pointers;
192−
193− /* Allocated with stripe_nsectors-many bits for finish_*() calls */
194− unsigned long *finish_pbitmap;
195−};
196−
19766 static int __raid56_parity_recover(struct btrfs_raid_bio *rbio);
19867 static noinline void finish_rmw(struct btrfs_raid_bio *rbio);
19968 static void rmw_work(struct work_struct *work);
@@ -347,6 +216,24 @@ static void index_stripe_sectors(struct btrfs_raid_bio *rbio) · 147 unmodified lines
347216 }
348217 }
349218
219+static void steal_rbio_page(struct btrfs_raid_bio *src,
220+ struct btrfs_raid_bio *dest, int page_nr)
221+{
222+ const u32 sectorsize = src->bioc->fs_info->sectorsize;
223+ const u32 sectors_per_page = PAGE_SIZE / sectorsize;
224+ int i;
225+
226+ if (dest->stripe_pages[page_nr])
227+ __free_page(dest->stripe_pages[page_nr]);
228+ dest->stripe_pages[page_nr] = src->stripe_pages[page_nr];
229+ src->stripe_pages[page_nr] = NULL;
230+
231+ /* Also update the sector->uptodate bits. */
232+ for (i = sectors_per_page * page_nr;
233+ i < sectors_per_page * page_nr + sectors_per_page; i++)
234+ dest->stripe_sectors[i].uptodate = true;
235+}
236+
350237 /*
351238 * Stealing an rbio means taking all the uptodate pages from the stripe array
352239 * in the source rbio and putting them into the destination rbio.
@@ -358,7 +245,6 @@ static void steal_rbio(struct btrfs_raid_bio *src, struct btrfs_raid_bio *dest) · 5 unmodified lines
358245 {
359246 int i;
360247 struct page *s;
361− struct page *d;
362248
363249 if (!test_bit(RBIO_CACHE_READY_BIT, &src->flags))
364250 return;
@@ -368,12 +254,7 @@ static void steal_rbio(struct btrfs_raid_bio *src, struct btrfs_raid_bio *dest) · 3 unmodified lines
368254 if (!s || !full_page_sectors_uptodate(src, i))
369255 continue;
370256
371− d = dest->stripe_pages[i];
372− if (d)
373− __free_page(d);
374−
375− dest->stripe_pages[i] = s;
376− src->stripe_pages[i] = NULL;
257+ steal_rbio_page(src, dest, i);
377258 }
378259 index_stripe_sectors(dest);
379260 index_stripe_sectors(src);
@@ -391,6 +272,9 @@ static void merge_rbio(struct btrfs_raid_bio *dest, · 11 unmodified lines
391272 {
392273 bio_list_merge(&dest->bio_list, &victim->bio_list);
393274 dest->bio_list_bytes += victim->bio_list_bytes;
275+ /* Also inherit the bitmaps from @victim. */
276+ bitmap_or(&dest->dbitmap, &victim->dbitmap, &dest->dbitmap,
277+ dest->stripe_nsectors);
394278 dest->generic_bio_cnt += victim->generic_bio_cnt;
395279 bio_list_init(&victim->bio_list);
396280 }
@@ -590,9 +474,9 @@ static int rbio_is_full(struct btrfs_raid_bio *rbio) · 193 unmodified lines
590474 int ret = 1;
591475
592476 spin_lock_irqsave(&rbio->bio_list_lock, flags);
593− if (size != rbio->nr_data * rbio->stripe_len)
477+ if (size != rbio->nr_data * BTRFS_STRIPE_LEN)
594478 ret = 0;
595− BUG_ON(size > rbio->nr_data * rbio->stripe_len);
479+ BUG_ON(size > rbio->nr_data * BTRFS_STRIPE_LEN);
596480 spin_unlock_irqrestore(&rbio->bio_list_lock, flags);
597481
598482 return ret;
@@ -932,6 +816,12 @@ static void rbio_orig_end_io(struct btrfs_raid_bio *rbio, blk_status_t err) · 333 unmodified lines
932816
933817 if (rbio->generic_bio_cnt)
934818 btrfs_bio_counter_sub(rbio->bioc->fs_info, rbio->generic_bio_cnt);
819+ /*
820+ * Clear the data bitmap, as the rbio may be cached for later usage.
821+ * do this before before unlock_stripe() so there will be no new bio
822+ * for this bio.
823+ */
824+ bitmap_clear(&rbio->dbitmap, 0, rbio->stripe_nsectors);
935825
936826 /*
937827 * At this moment, rbio->bio_list is empty, however since rbio does not
@@ -1023,29 +913,30 @@ static struct sector_ptr *sector_in_rbio(struct btrfs_raid_bio *rbio, · 85 unmodified lines
1023913 * this does not allocate any pages for rbio->pages.
1024914 */
1025915 static struct btrfs_raid_bio *alloc_rbio(struct btrfs_fs_info *fs_info,
1026− struct btrfs_io_context *bioc,
1027− u32 stripe_len)
916+ struct btrfs_io_context *bioc)
1028917 {
1029918 const unsigned int real_stripes = bioc->num_stripes - bioc->num_tgtdevs;
1030− const unsigned int stripe_npages = stripe_len >> PAGE_SHIFT;
919+ const unsigned int stripe_npages = BTRFS_STRIPE_LEN >> PAGE_SHIFT;
1031920 const unsigned int num_pages = stripe_npages * real_stripes;
1032− const unsigned int stripe_nsectors = stripe_len >> fs_info->sectorsize_bits;
921+ const unsigned int stripe_nsectors =
922+ BTRFS_STRIPE_LEN >> fs_info->sectorsize_bits;
1033923 const unsigned int num_sectors = stripe_nsectors * real_stripes;
1034924 struct btrfs_raid_bio *rbio;
1035− int nr_data = 0;
1036925 void *p;
1037926
1038− ASSERT(IS_ALIGNED(stripe_len, PAGE_SIZE));
1039927 /* PAGE_SIZE must also be aligned to sectorsize for subpage support */
1040928 ASSERT(IS_ALIGNED(PAGE_SIZE, fs_info->sectorsize));
929+ /*
930+ * Our current stripe len should be fixed to 64k thus stripe_nsectors
931+ * (at most 16) should be no larger than BITS_PER_LONG.
932+ */
933+ ASSERT(stripe_nsectors <= BITS_PER_LONG);
1041934
1042935 rbio = kzalloc(sizeof(*rbio) +
1043936 sizeof(*rbio->stripe_pages) * num_pages +
1044937 sizeof(*rbio->bio_sectors) * num_sectors +
1045938 sizeof(*rbio->stripe_sectors) * num_sectors +
1046− sizeof(*rbio->finish_pointers) * real_stripes +
1047− sizeof(*rbio->dbitmap) * BITS_TO_LONGS(stripe_nsectors) +
1048− sizeof(*rbio->finish_pbitmap) * BITS_TO_LONGS(stripe_nsectors),
939+ sizeof(*rbio->finish_pointers) * real_stripes,
1049940 GFP_NOFS);
1050941 if (!rbio)
1051942 return ERR_PTR(-ENOMEM);
@@ -1056,7 +947,6 @@ static struct btrfs_raid_bio *alloc_rbio(struct btrfs_fs_info *fs_info, · 4 unmodified lines
1056947 INIT_LIST_HEAD(&rbio->stripe_cache);
1057948 INIT_LIST_HEAD(&rbio->hash_list);
1058949 rbio->bioc = bioc;
1059− rbio->stripe_len = stripe_len;
1060950 rbio->nr_pages = num_pages;
1061951 rbio->nr_sectors = num_sectors;
1062952 rbio->real_stripes = real_stripes;
@@ -1081,18 +971,11 @@ static struct btrfs_raid_bio *alloc_rbio(struct btrfs_fs_info *fs_info, · 18 unmodified lines
1081971 CONSUME_ALLOC(rbio->bio_sectors, num_sectors);
1082972 CONSUME_ALLOC(rbio->stripe_sectors, num_sectors);
1083973 CONSUME_ALLOC(rbio->finish_pointers, real_stripes);
1084− CONSUME_ALLOC(rbio->dbitmap, BITS_TO_LONGS(stripe_nsectors));
1085− CONSUME_ALLOC(rbio->finish_pbitmap, BITS_TO_LONGS(stripe_nsectors));
1086974 #undef CONSUME_ALLOC
1087975
1088− if (bioc->map_type & BTRFS_BLOCK_GROUP_RAID5)
1089− nr_data = real_stripes - 1;
1090− else if (bioc->map_type & BTRFS_BLOCK_GROUP_RAID6)
1091− nr_data = real_stripes - 2;
1092− else
1093− BUG();
976+ ASSERT(btrfs_nr_parity_stripes(bioc->map_type));
977+ rbio->nr_data = real_stripes - btrfs_nr_parity_stripes(bioc->map_type);
1094978
1095− rbio->nr_data = nr_data;
1096979 return rbio;
1097980 }
1098981
@@ -1135,7 +1018,6 @@ static int rbio_add_io_sector(struct btrfs_raid_bio *rbio, · 36 unmodified lines
11351018 struct sector_ptr *sector,
11361019 unsigned int stripe_nr,
11371020 unsigned int sector_nr,
1138− unsigned long bio_max_len,
11391021 enum req_op op)
11401022 {
11411023 const u32 sectorsize = rbio->bioc->fs_info->sectorsize;
@@ -1180,7 +1062,8 @@ static int rbio_add_io_sector(struct btrfs_raid_bio *rbio, · 38 unmodified lines
11801062 }
11811063
11821064 /* put a new bio on the list */
1183− bio = bio_alloc(stripe->dev->bdev, max(bio_max_len >> PAGE_SHIFT, 1UL),
1065+ bio = bio_alloc(stripe->dev->bdev,
1066+ max(BTRFS_STRIPE_LEN >> PAGE_SHIFT, 1),
11841067 op, GFP_NOFS);
11851068 bio->bi_iter.bi_sector = disk_start >> 9;
11861069 bio->bi_private = rbio;
@@ -1215,9 +1098,6 @@ static void index_one_bio(struct btrfs_raid_bio *rbio, struct bio *bio) · 28 unmodified lines
12151098 u32 offset = (bio->bi_iter.bi_sector << SECTOR_SHIFT) -
12161099 rbio->bioc->raid_map[0];
12171100
1218− if (bio_flagged(bio, BIO_CLONED))
1219− bio->bi_iter = btrfs_bio(bio)->iter;
1220−
12211101 bio_for_each_segment(bvec, bio, iter) {
12221102 u32 bvec_offset;
12231103
@@ -1252,6 +1132,34 @@ static void index_rbio_pages(struct btrfs_raid_bio *rbio) · 28 unmodified lines
12521132 spin_unlock_irq(&rbio->bio_list_lock);
12531133 }
12541134
1135+static void bio_get_trace_info(struct btrfs_raid_bio *rbio, struct bio *bio,
1136+ struct raid56_bio_trace_info *trace_info)
1137+{
1138+ const struct btrfs_io_context *bioc = rbio->bioc;
1139+ int i;
1140+
1141+ ASSERT(bioc);
1142+
1143+ /* We rely on bio->bi_bdev to find the stripe number. */
1144+ if (!bio->bi_bdev)
1145+ goto not_found;
1146+
1147+ for (i = 0; i < bioc->num_stripes; i++) {
1148+ if (bio->bi_bdev != bioc->stripes[i].dev->bdev)
1149+ continue;
1150+ trace_info->stripe_nr = i;
1151+ trace_info->devid = bioc->stripes[i].dev->devid;
1152+ trace_info->offset = (bio->bi_iter.bi_sector << SECTOR_SHIFT) -
1153+ bioc->stripes[i].physical;
1154+ return;
1155+ }
1156+
1157+not_found:
1158+ trace_info->devid = -1;
1159+ trace_info->offset = -1;
1160+ trace_info->stripe_nr = -1;
1161+}
1162+
12551163 /*
12561164 * this is called from one of two situations. We either
12571165 * have a full stripe from the higher layers, or we've read all
@@ -1266,7 +1174,10 @@ static noinline void finish_rmw(struct btrfs_raid_bio *rbio) · 8 unmodified lines
12661174 const u32 sectorsize = bioc->fs_info->sectorsize;
12671175 void **pointers = rbio->finish_pointers;
12681176 int nr_data = rbio->nr_data;
1177+ /* The total sector number inside the full stripe. */
1178+ int total_sector_nr;
12691179 int stripe;
1180+ /* Sector number inside a stripe. */
12701181 int sectornr;
12711182 bool has_qstripe;
12721183 struct bio_list bio_list;
@@ -1282,6 +1193,9 @@ static noinline void finish_rmw(struct btrfs_raid_bio *rbio) · 9 unmodified lines
12821193 else
12831194 BUG();
12841195
1196+ /* We should have at least one data sector. */
1197+ ASSERT(bitmap_weight(&rbio->dbitmap, rbio->stripe_nsectors));
1198+
12851199 /* at this point we either have a full stripe,
12861200 * or we've read the full stripe from the drive.
12871201 * recalculate the parity and write the new results.
@@ -1348,55 +1262,71 @@ static noinline void finish_rmw(struct btrfs_raid_bio *rbio) · 60 unmodified lines
13481262 }
13491263
13501264 /*
1351− * time to start writing. Make bios for everything from the
1352− * higher layers (the bio_list in our rbio) and our p/q. Ignore
1353− * everything else.
1265+ * Start writing. Make bios for everything from the higher layers (the
1266+ * bio_list in our rbio) and our P/Q. Ignore everything else.
13541267 */
1355− for (stripe = 0; stripe < rbio->real_stripes; stripe++) {
1356− for (sectornr = 0; sectornr < rbio->stripe_nsectors; sectornr++) {
1357− struct sector_ptr *sector;
1358−
1359− if (stripe < rbio->nr_data) {
1360− sector = sector_in_rbio(rbio, stripe, sectornr, 1);
1361− if (!sector)
1362− continue;
1363− } else {
1364− sector = rbio_stripe_sector(rbio, stripe, sectornr);
1365− }
1268+ for (total_sector_nr = 0; total_sector_nr < rbio->nr_sectors;
1269+ total_sector_nr++) {
1270+ struct sector_ptr *sector;
1271+
1272+ stripe = total_sector_nr / rbio->stripe_nsectors;
1273+ sectornr = total_sector_nr % rbio->stripe_nsectors;
13661274
1367− ret = rbio_add_io_sector(rbio, &bio_list, sector, stripe,
1368− sectornr, rbio->stripe_len,
1369− REQ_OP_WRITE);
1370− if (ret)
1371− goto cleanup;
1275+ /* This vertical stripe has no data, skip it. */
1276+ if (!test_bit(sectornr, &rbio->dbitmap))
1277+ continue;
1278+
1279+ if (stripe < rbio->nr_data) {
1280+ sector = sector_in_rbio(rbio, stripe, sectornr, 1);
1281+ if (!sector)
1282+ continue;
1283+ } else {
1284+ sector = rbio_stripe_sector(rbio, stripe, sectornr);
13721285 }
1286+
1287+ ret = rbio_add_io_sector(rbio, &bio_list, sector, stripe,
1288+ sectornr, REQ_OP_WRITE);
1289+ if (ret)
1290+ goto cleanup;
13731291 }
13741292
13751293 if (likely(!bioc->num_tgtdevs))
13761294 goto write_data;
13771295
1378− for (stripe = 0; stripe < rbio->real_stripes; stripe++) {
1379− if (!bioc->tgtdev_map[stripe])
1380− continue;
1296+ for (total_sector_nr = 0; total_sector_nr < rbio->nr_sectors;
1297+ total_sector_nr++) {
1298+ struct sector_ptr *sector;
13811299
1382− for (sectornr = 0; sectornr < rbio->stripe_nsectors; sectornr++) {
1383− struct sector_ptr *sector;
1300+ stripe = total_sector_nr / rbio->stripe_nsectors;
1301+ sectornr = total_sector_nr % rbio->stripe_nsectors;
13841302
1385− if (stripe < rbio->nr_data) {
1386− sector = sector_in_rbio(rbio, stripe, sectornr, 1);
1387− if (!sector)
1388− continue;
1389− } else {
1390− sector = rbio_stripe_sector(rbio, stripe, sectornr);
1391− }
1303+ if (!bioc->tgtdev_map[stripe]) {
1304+ /*
1305+ * We can skip the whole stripe completely, note
1306+ * total_sector_nr will be increased by one anyway.
1307+ */
1308+ ASSERT(sectornr == 0);
1309+ total_sector_nr += rbio->stripe_nsectors - 1;
1310+ continue;
1311+ }
13921312
1393− ret = rbio_add_io_sector(rbio, &bio_list, sector,
1394− rbio->bioc->tgtdev_map[stripe],
1395− sectornr, rbio->stripe_len,
1396− REQ_OP_WRITE);
1397− if (ret)
1398− goto cleanup;
1313+ /* This vertical stripe has no data, skip it. */
1314+ if (!test_bit(sectornr, &rbio->dbitmap))
1315+ continue;
1316+
1317+ if (stripe < rbio->nr_data) {
1318+ sector = sector_in_rbio(rbio, stripe, sectornr, 1);
1319+ if (!sector)
1320+ continue;
1321+ } else {
1322+ sector = rbio_stripe_sector(rbio, stripe, sectornr);
13991323 }
1324+
1325+ ret = rbio_add_io_sector(rbio, &bio_list, sector,
1326+ rbio->bioc->tgtdev_map[stripe],
1327+ sectornr, REQ_OP_WRITE);
1328+ if (ret)
1329+ goto cleanup;
14001330 }
14011331
14021332 write_data:
@@ -1406,6 +1336,12 @@ write_data: · 3 unmodified lines
14061336 while ((bio = bio_list_pop(&bio_list))) {
14071337 bio->bi_end_io = raid_write_end_io;
14081338
1339+ if (trace_raid56_write_stripe_enabled()) {
1340+ struct raid56_bio_trace_info trace_info = { 0 };
1341+
1342+ bio_get_trace_info(rbio, bio, &trace_info);
1343+ trace_raid56_write_stripe(rbio, bio, &trace_info);
1344+ }
14091345 submit_bio(bio);
14101346 }
14111347 return;
@@ -1433,7 +1369,7 @@ static int find_bio_stripe(struct btrfs_raid_bio *rbio, · 21 unmodified lines
14331369
14341370 for (i = 0; i < rbio->bioc->num_stripes; i++) {
14351371 stripe = &rbio->bioc->stripes[i];
1436− if (in_range(physical, stripe->physical, rbio->stripe_len) &&
1372+ if (in_range(physical, stripe->physical, BTRFS_STRIPE_LEN) &&
14371373 stripe->dev->bdev && bio->bi_bdev == stripe->dev->bdev) {
14381374 return i;
14391375 }
@@ -1455,7 +1391,7 @@ static int find_logical_bio_stripe(struct btrfs_raid_bio *rbio, · 15 unmodified lines
14551391 for (i = 0; i < rbio->nr_data; i++) {
14561392 u64 stripe_start = rbio->bioc->raid_map[i];
14571393
1458− if (in_range(logical, stripe_start, rbio->stripe_len))
1394+ if (in_range(logical, stripe_start, BTRFS_STRIPE_LEN))
14591395 return i;
14601396 }
14611397 return -1;
@@ -1552,15 +1488,7 @@ static void set_bio_pages_uptodate(struct btrfs_raid_bio *rbio, struct bio *bio) · 90 unmodified lines
15521488 }
15531489 }
15541490
1555−/*
1556− * end io for the read phase of the rmw cycle. All the bios here are physical
1557− * stripe bios we've read from the disk so we can recalculate the parity of the
1558− * stripe.
1559− *
1560− * This will usually kick off finish_rmw once all the bios are read in, but it
1561− * may trigger parity reconstruction if we had any errors along the way
1562− */
1563−static void raid_rmw_end_io(struct bio *bio)
1491+static void raid56_bio_end_io(struct bio *bio)
15641492 {
15651493 struct btrfs_raid_bio *rbio = bio->bi_private;
15661494
@@ -1571,23 +1499,34 @@ static void raid_rmw_end_io(struct bio *bio) · 4 unmodified lines
15711499
15721500 bio_put(bio);
15731501
1574− if (!atomic_dec_and_test(&rbio->stripes_pending))
1575− return;
1502+ if (atomic_dec_and_test(&rbio->stripes_pending))
1503+ queue_work(rbio->bioc->fs_info->endio_raid56_workers,
1504+ &rbio->end_io_work);
1505+}
15761506
1577− if (atomic_read(&rbio->error) > rbio->bioc->max_errors)
1578− goto cleanup;
1507+/*
1508+ * End io handler for the read phase of the RMW cycle. All the bios here are
1509+ * physical stripe bios we've read from the disk so we can recalculate the
1510+ * parity of the stripe.
1511+ *
1512+ * This will usually kick off finish_rmw once all the bios are read in, but it
1513+ * may trigger parity reconstruction if we had any errors along the way
1514+ */
1515+static void raid56_rmw_end_io_work(struct work_struct *work)
1516+{
1517+ struct btrfs_raid_bio *rbio =
1518+ container_of(work, struct btrfs_raid_bio, end_io_work);
1519+
1520+ if (atomic_read(&rbio->error) > rbio->bioc->max_errors) {
1521+ rbio_orig_end_io(rbio, BLK_STS_IOERR);
1522+ return;
1523+ }
15791524
15801525 /*
1581− * this will normally call finish_rmw to start our write
1582− * but if there are any failed stripes we'll reconstruct
1583− * from parity first
1526+ * This will normally call finish_rmw to start our write but if there
1527+ * are any failed stripes we'll reconstruct from parity first.
15841528 */
15851529 validate_rbio_for_rmw(rbio);
1586− return;
1587−
1588−cleanup:
1589−
1590− rbio_orig_end_io(rbio, BLK_STS_IOERR);
15911530 }
15921531
15931532 /*
@@ -1598,9 +1537,9 @@ static int raid56_rmw_stripe(struct btrfs_raid_bio *rbio) · 4 unmodified lines
15981537 {
15991538 int bios_to_read = 0;
16001539 struct bio_list bio_list;
1540+ const int nr_data_sectors = rbio->stripe_nsectors * rbio->nr_data;
16011541 int ret;
1602− int sectornr;
1603− int stripe;
1542+ int total_sector_nr;
16041543 struct bio *bio;
16051544
16061545 bio_list_init(&bio_list);
@@ -1612,38 +1551,34 @@ static int raid56_rmw_stripe(struct btrfs_raid_bio *rbio) · 5 unmodified lines
16121551 index_rbio_pages(rbio);
16131552
16141553 atomic_set(&rbio->error, 0);
1615− /*
1616− * build a list of bios to read all the missing parts of this
1617− * stripe
1618− */
1619− for (stripe = 0; stripe < rbio->nr_data; stripe++) {
1620− for (sectornr = 0; sectornr < rbio->stripe_nsectors; sectornr++) {
1621− struct sector_ptr *sector;
1554+ /* Build a list of bios to read all the missing data sectors. */
1555+ for (total_sector_nr = 0; total_sector_nr < nr_data_sectors;
1556+ total_sector_nr++) {
1557+ struct sector_ptr *sector;
1558+ int stripe = total_sector_nr / rbio->stripe_nsectors;
1559+ int sectornr = total_sector_nr % rbio->stripe_nsectors;
16221560
1623− /*
1624− * We want to find all the sectors missing from the
1625− * rbio and read them from the disk. If * sector_in_rbio()
1626− * finds a page in the bio list we don't need to read
1627− * it off the stripe.
1628− */
1629− sector = sector_in_rbio(rbio, stripe, sectornr, 1);
1630− if (sector)
1631− continue;
1561+ /*
1562+ * We want to find all the sectors missing from the rbio and
1563+ * read them from the disk. If sector_in_rbio() finds a page
1564+ * in the bio list we don't need to read it off the stripe.
1565+ */
1566+ sector = sector_in_rbio(rbio, stripe, sectornr, 1);
1567+ if (sector)
1568+ continue;
16321569
1633− sector = rbio_stripe_sector(rbio, stripe, sectornr);
1634− /*
1635− * The bio cache may have handed us an uptodate page.
1636− * If so, be happy and use it.
1637− */
1638− if (sector->uptodate)
1639− continue;
1570+ sector = rbio_stripe_sector(rbio, stripe, sectornr);
1571+ /*
1572+ * The bio cache may have handed us an uptodate page. If so,
1573+ * use it.
1574+ */
1575+ if (sector->uptodate)
1576+ continue;
16401577
1641− ret = rbio_add_io_sector(rbio, &bio_list, sector,
1642− stripe, sectornr, rbio->stripe_len,
1643− REQ_OP_READ);
1644− if (ret)
1645− goto cleanup;
1646− }
1578+ ret = rbio_add_io_sector(rbio, &bio_list, sector,
1579+ stripe, sectornr, REQ_OP_READ);
1580+ if (ret)
1581+ goto cleanup;
16471582 }
16481583
16491584 bios_to_read = bio_list_size(&bio_list);
@@ -1662,11 +1597,16 @@ static int raid56_rmw_stripe(struct btrfs_raid_bio *rbio) · 12 unmodified lines
16621597 * touch it after that.
16631598 */
16641599 atomic_set(&rbio->stripes_pending, bios_to_read);
1600+ INIT_WORK(&rbio->end_io_work, raid56_rmw_end_io_work);
16651601 while ((bio = bio_list_pop(&bio_list))) {
1666− bio->bi_end_io = raid_rmw_end_io;
1602+ bio->bi_end_io = raid56_bio_end_io;
16671603
1668− btrfs_bio_wq_end_io(rbio->bioc->fs_info, bio, BTRFS_WQ_ENDIO_RAID56);
1604+ if (trace_raid56_read_partial_enabled()) {
1605+ struct raid56_bio_trace_info trace_info = { 0 };
16691606
1607+ bio_get_trace_info(rbio, bio, &trace_info);
1608+ trace_raid56_read_partial(rbio, bio, &trace_info);
1609+ }
16701610 submit_bio(bio);
16711611 }
16721612 /* the actual write will happen once the reads are done */
@@ -1833,27 +1773,53 @@ static void btrfs_raid_unplug(struct blk_plug_cb *cb, bool from_schedule) · 160 unmodified lines
18331773 run_plug(plug);
18341774 }
18351775
1776+/* Add the original bio into rbio->bio_list, and update rbio::dbitmap. */
1777+static void rbio_add_bio(struct btrfs_raid_bio *rbio, struct bio *orig_bio)
1778+{
1779+ const struct btrfs_fs_info *fs_info = rbio->bioc->fs_info;
1780+ const u64 orig_logical = orig_bio->bi_iter.bi_sector << SECTOR_SHIFT;
1781+ const u64 full_stripe_start = rbio->bioc->raid_map[0];
1782+ const u32 orig_len = orig_bio->bi_iter.bi_size;
1783+ const u32 sectorsize = fs_info->sectorsize;
1784+ u64 cur_logical;
1785+
1786+ ASSERT(orig_logical >= full_stripe_start &&
1787+ orig_logical + orig_len <= full_stripe_start +
1788+ rbio->nr_data * BTRFS_STRIPE_LEN);
1789+
1790+ bio_list_add(&rbio->bio_list, orig_bio);
1791+ rbio->bio_list_bytes += orig_bio->bi_iter.bi_size;
1792+
1793+ /* Update the dbitmap. */
1794+ for (cur_logical = orig_logical; cur_logical < orig_logical + orig_len;
1795+ cur_logical += sectorsize) {
1796+ int bit = ((u32)(cur_logical - full_stripe_start) >>
1797+ fs_info->sectorsize_bits) % rbio->stripe_nsectors;
1798+
1799+ set_bit(bit, &rbio->dbitmap);
1800+ }
1801+}
1802+
18361803 /*
18371804 * our main entry point for writes from the rest of the FS.
18381805 */
1839−int raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc, u32 stripe_len)
1806+void raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc)
18401807 {
18411808 struct btrfs_fs_info *fs_info = bioc->fs_info;
18421809 struct btrfs_raid_bio *rbio;
18431810 struct btrfs_plug_cb *plug = NULL;
18441811 struct blk_plug_cb *cb;
1845− int ret;
1812+ int ret = 0;
18461813
1847− rbio = alloc_rbio(fs_info, bioc, stripe_len);
1814+ rbio = alloc_rbio(fs_info, bioc);
18481815 if (IS_ERR(rbio)) {
18491816 btrfs_put_bioc(bioc);
1850− return PTR_ERR(rbio);
1817+ ret = PTR_ERR(rbio);
1818+ goto out_dec_counter;
18511819 }
1852− bio_list_add(&rbio->bio_list, bio);
1853− rbio->bio_list_bytes = bio->bi_iter.bi_size;
18541820 rbio->operation = BTRFS_RBIO_WRITE;
1821+ rbio_add_bio(rbio, bio);
18551822
1856− btrfs_bio_counter_inc_noblocked(fs_info);
18571823 rbio->generic_bio_cnt = 1;
18581824
18591825 /*
@@ -1863,8 +1829,8 @@ int raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc, u32 stri · 3 unmodified lines
18631829 if (rbio_is_full(rbio)) {
18641830 ret = full_stripe_write(rbio);
18651831 if (ret)
1866− btrfs_bio_counter_dec(fs_info);
1867− return ret;
1832+ goto out_dec_counter;
1833+ return;
18681834 }
18691835
18701836 cb = blk_check_plugged(btrfs_raid_unplug, fs_info, sizeof(*plug));
@@ -1875,13 +1841,18 @@ int raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc, u32 stri · 4 unmodified lines
18751841 INIT_LIST_HEAD(&plug->rbio_list);
18761842 }
18771843 list_add_tail(&rbio->plug_list, &plug->rbio_list);
1878− ret = 0;
18791844 } else {
18801845 ret = __raid56_parity_write(rbio);
18811846 if (ret)
1882− btrfs_bio_counter_dec(fs_info);
1847+ goto out_dec_counter;
18831848 }
1884− return ret;
1849+
1850+ return;
1851+
1852+out_dec_counter:
1853+ btrfs_bio_counter_dec(fs_info);
1854+ bio->bi_status = errno_to_blk_status(ret);
1855+ bio_endio(bio);
18851856 }
18861857
18871858 /*
@@ -1939,7 +1910,7 @@ static void __raid_recover_end_io(struct btrfs_raid_bio *rbio) · 51 unmodified lines
19391910 * which we have data when doing parity scrub.
19401911 */
19411912 if (rbio->operation == BTRFS_RBIO_PARITY_SCRUB &&
1942− !test_bit(sectornr, rbio->dbitmap))
1913+ !test_bit(sectornr, &rbio->dbitmap))
19431914 continue;
19441915
19451916 /*
@@ -2108,25 +2079,13 @@ cleanup_io: · 162 unmodified lines
21082079 }
21092080
21102081 /*
2111− * This is called only for stripes we've read from disk to
2112− * reconstruct the parity.
2082+ * This is called only for stripes we've read from disk to reconstruct the
2083+ * parity.
21132084 */
2114−static void raid_recover_end_io(struct bio *bio)
2085+static void raid_recover_end_io_work(struct work_struct *work)
21152086 {
2116− struct btrfs_raid_bio *rbio = bio->bi_private;
2117−
2118− /*
2119− * we only read stripe pages off the disk, set them
2120− * up to date if there were no errors
2121− */
2122− if (bio->bi_status)
2123− fail_bio_stripe(rbio, bio);
2124− else
2125− set_bio_pages_uptodate(rbio, bio);
2126− bio_put(bio);
2127−
2128− if (!atomic_dec_and_test(&rbio->stripes_pending))
2129− return;
2087+ struct btrfs_raid_bio *rbio =
2088+ container_of(work, struct btrfs_raid_bio, end_io_work);
21302089
21312090 if (atomic_read(&rbio->error) > rbio->bioc->max_errors)
21322091 rbio_orig_end_io(rbio, BLK_STS_IOERR);
@@ -2147,8 +2106,7 @@ static int __raid56_parity_recover(struct btrfs_raid_bio *rbio) · 14 unmodified lines
21472106 int bios_to_read = 0;
21482107 struct bio_list bio_list;
21492108 int ret;
2150− int sectornr;
2151− int stripe;
2109+ int total_sector_nr;
21522110 struct bio *bio;
21532111
21542112 bio_list_init(&bio_list);
@@ -2160,33 +2118,31 @@ static int __raid56_parity_recover(struct btrfs_raid_bio *rbio) · 5 unmodified lines
21602118 atomic_set(&rbio->error, 0);
21612119
21622120 /*
2163− * read everything that hasn't failed. Thanks to the
2164− * stripe cache, it is possible that some or all of these
2165− * pages are going to be uptodate.
2121+ * Read everything that hasn't failed. However this time we will
2122+ * not trust any cached sector.
2123+ * As we may read out some stale data but higher layer is not reading
2124+ * that stale part.
2125+ *
2126+ * So here we always re-read everything in recovery path.
21662127 */
2167− for (stripe = 0; stripe < rbio->real_stripes; stripe++) {
2128+ for (total_sector_nr = 0; total_sector_nr < rbio->nr_sectors;
2129+ total_sector_nr++) {
2130+ int stripe = total_sector_nr / rbio->stripe_nsectors;
2131+ int sectornr = total_sector_nr % rbio->stripe_nsectors;
2132+ struct sector_ptr *sector;
2133+
21682134 if (rbio->faila == stripe || rbio->failb == stripe) {
21692135 atomic_inc(&rbio->error);
2136+ /* Skip the current stripe. */
2137+ ASSERT(sectornr == 0);
2138+ total_sector_nr += rbio->stripe_nsectors - 1;
21702139 continue;
21712140 }
2172−
2173− for (sectornr = 0; sectornr < rbio->stripe_nsectors; sectornr++) {
2174− struct sector_ptr *sector;
2175−
2176− /*
2177− * the rmw code may have already read this
2178− * page in
2179− */
2180− sector = rbio_stripe_sector(rbio, stripe, sectornr);
2181− if (sector->uptodate)
2182− continue;
2183−
2184− ret = rbio_add_io_sector(rbio, &bio_list, sector,
2185− stripe, sectornr, rbio->stripe_len,
2186− REQ_OP_READ);
2187− if (ret < 0)
2188− goto cleanup;
2189− }
2141+ sector = rbio_stripe_sector(rbio, stripe, sectornr);
2142+ ret = rbio_add_io_sector(rbio, &bio_list, sector, stripe,
2143+ sectornr, REQ_OP_READ);
2144+ if (ret < 0)
2145+ goto cleanup;
21902146 }
21912147
21922148 bios_to_read = bio_list_size(&bio_list);
@@ -2209,11 +2165,16 @@ static int __raid56_parity_recover(struct btrfs_raid_bio *rbio) · 16 unmodified lines
22092165 * touch it after that.
22102166 */
22112167 atomic_set(&rbio->stripes_pending, bios_to_read);
2168+ INIT_WORK(&rbio->end_io_work, raid_recover_end_io_work);
22122169 while ((bio = bio_list_pop(&bio_list))) {
2213− bio->bi_end_io = raid_recover_end_io;
2170+ bio->bi_end_io = raid56_bio_end_io;
22142171
2215− btrfs_bio_wq_end_io(rbio->bioc->fs_info, bio, BTRFS_WQ_ENDIO_RAID56);
2172+ if (trace_raid56_scrub_read_recover_enabled()) {
2173+ struct raid56_bio_trace_info trace_info = { 0 };
22162174
2175+ bio_get_trace_info(rbio, bio, &trace_info);
2176+ trace_raid56_scrub_read_recover(rbio, bio, &trace_info);
2177+ }
22172178 submit_bio(bio);
22182179 }
22192180
@@ -2236,28 +2197,27 @@ cleanup: · 16 unmodified lines
22362197 * so we assume the bio they send down corresponds to a failed part
22372198 * of the drive.
22382199 */
2239−int raid56_parity_recover(struct bio *bio, struct btrfs_io_context *bioc,
2240− u32 stripe_len, int mirror_num, int generic_io)
2200+void raid56_parity_recover(struct bio *bio, struct btrfs_io_context *bioc,
2201+ int mirror_num, bool generic_io)
22412202 {
22422203 struct btrfs_fs_info *fs_info = bioc->fs_info;
22432204 struct btrfs_raid_bio *rbio;
2244− int ret;
22452205
22462206 if (generic_io) {
22472207 ASSERT(bioc->mirror_num == mirror_num);
22482208 btrfs_bio(bio)->mirror_num = mirror_num;
2209+ } else {
2210+ btrfs_get_bioc(bioc);
22492211 }
22502212
2251− rbio = alloc_rbio(fs_info, bioc, stripe_len);
2213+ rbio = alloc_rbio(fs_info, bioc);
22522214 if (IS_ERR(rbio)) {
2253− if (generic_io)
2254− btrfs_put_bioc(bioc);
2255− return PTR_ERR(rbio);
2215+ bio->bi_status = errno_to_blk_status(PTR_ERR(rbio));
2216+ goto out_end_bio;
22562217 }
22572218
22582219 rbio->operation = BTRFS_RBIO_READ_REBUILD;
2259− bio_list_add(&rbio->bio_list, bio);
2260− rbio->bio_list_bytes = bio->bi_iter.bi_size;
2220+ rbio_add_bio(rbio, bio);
22612221
22622222 rbio->faila = find_logical_bio_stripe(rbio, bio);
22632223 if (rbio->faila == -1) {
@@ -2265,18 +2225,13 @@ int raid56_parity_recover(struct bio *bio, struct btrfs_io_context *bioc, · 1 unmodified line
22652225 "%s could not find the bad stripe in raid56 so that we cannot recover any more (bio has logical %llu len %llu, bioc has map_type %llu)",
22662226 __func__, bio->bi_iter.bi_sector << 9,
22672227 (u64)bio->bi_iter.bi_size, bioc->map_type);
2268− if (generic_io)
2269− btrfs_put_bioc(bioc);
22702228 kfree(rbio);
2271− return -EIO;
2229+ bio->bi_status = BLK_STS_IOERR;
2230+ goto out_end_bio;
22722231 }
22732232
2274− if (generic_io) {
2275− btrfs_bio_counter_inc_noblocked(fs_info);
2233+ if (generic_io)
22762234 rbio->generic_bio_cnt = 1;
2277− } else {
2278− btrfs_get_bioc(bioc);
2279− }
22802235
22812236 /*
22822237 * Loop retry:
@@ -2295,24 +2250,20 @@ int raid56_parity_recover(struct bio *bio, struct btrfs_io_context *bioc, · 12 unmodified lines
22952250 rbio->failb--;
22962251 }
22972252
2298− ret = lock_stripe_add(rbio);
2253+ if (lock_stripe_add(rbio))
2254+ return;
22992255
23002256 /*
2301− * __raid56_parity_recover will end the bio with
2302− * any errors it hits. We don't want to return
2303− * its error value up the stack because our caller
2304− * will end up calling bio_endio with any nonzero
2305− * return
2306− */
2307− if (ret == 0)
2308− __raid56_parity_recover(rbio);
2309− /*
2310− * our rbio has been added to the list of
2311− * rbios that will be handled after the
2312− * currently lock owner is done
2257+ * This adds our rbio to the list of rbios that will be handled after
2258+ * the current lock owner is done.
23132259 */
2314− return 0;
2260+ __raid56_parity_recover(rbio);
2261+ return;
23152262
2263+out_end_bio:
2264+ btrfs_bio_counter_dec(fs_info);
2265+ btrfs_put_bioc(bioc);
2266+ bio_endio(bio);
23162267 }
23172268
23182269 static void rmw_work(struct work_struct *work)
@@ -2343,14 +2294,14 @@ static void read_rebuild_work(struct work_struct *work) · 24 unmodified lines
23432294
23442295 struct btrfs_raid_bio *raid56_parity_alloc_scrub_rbio(struct bio *bio,
23452296 struct btrfs_io_context *bioc,
2346− u32 stripe_len, struct btrfs_device *scrub_dev,
2297+ struct btrfs_device *scrub_dev,
23472298 unsigned long *dbitmap, int stripe_nsectors)
23482299 {
23492300 struct btrfs_fs_info *fs_info = bioc->fs_info;
23502301 struct btrfs_raid_bio *rbio;
23512302 int i;
23522303
2353− rbio = alloc_rbio(fs_info, bioc, stripe_len);
2304+ rbio = alloc_rbio(fs_info, bioc);
23542305 if (IS_ERR(rbio))
23552306 return NULL;
23562307 bio_list_add(&rbio->bio_list, bio);
@@ -2374,7 +2325,7 @@ struct btrfs_raid_bio *raid56_parity_alloc_scrub_rbio(struct bio *bio, · 17 unmodified lines
23742325 }
23752326 ASSERT(i < rbio->real_stripes);
23762327
2377− bitmap_copy(rbio->dbitmap, dbitmap, stripe_nsectors);
2328+ bitmap_copy(&rbio->dbitmap, dbitmap, stripe_nsectors);
23782329
23792330 /*
23802331 * We have already increased bio_counter when getting bioc, record it
@@ -2395,7 +2346,7 @@ void raid56_add_scrub_pages(struct btrfs_raid_bio *rbio, struct page *page, · 14 unmodified lines
23952346
23962347 ASSERT(logical >= rbio->bioc->raid_map[0]);
23972348 ASSERT(logical + sectorsize <= rbio->bioc->raid_map[0] +
2398− rbio->stripe_len * rbio->nr_data);
2349+ BTRFS_STRIPE_LEN * rbio->nr_data);
23992350 stripe_offset = (int)(logical - rbio->bioc->raid_map[0]);
24002351 index = stripe_offset / sectorsize;
24012352 rbio->bio_sectors[index].page = page;
@@ -2409,23 +2360,22 @@ void raid56_add_scrub_pages(struct btrfs_raid_bio *rbio, struct page *page, · 7 unmodified lines
24092360 static int alloc_rbio_essential_pages(struct btrfs_raid_bio *rbio)
24102361 {
24112362 const u32 sectorsize = rbio->bioc->fs_info->sectorsize;
2412− int stripe;
2413− int sectornr;
2414−
2415− for_each_set_bit(sectornr, rbio->dbitmap, rbio->stripe_nsectors) {
2416− for (stripe = 0; stripe < rbio->real_stripes; stripe++) {
2417− struct page *page;
2418− int index = (stripe * rbio->stripe_nsectors + sectornr) *
2419− sectorsize >> PAGE_SHIFT;
2363+ int total_sector_nr;
24202364
2421− if (rbio->stripe_pages[index])
2422− continue;
2365+ for (total_sector_nr = 0; total_sector_nr < rbio->nr_sectors;
2366+ total_sector_nr++) {
2367+ struct page *page;
2368+ int sectornr = total_sector_nr % rbio->stripe_nsectors;
2369+ int index = (total_sector_nr * sectorsize) >> PAGE_SHIFT;
24232370
2424− page = alloc_page(GFP_NOFS);
2425− if (!page)
2426− return -ENOMEM;
2427− rbio->stripe_pages[index] = page;
2428− }
2371+ if (!test_bit(sectornr, &rbio->dbitmap))
2372+ continue;
2373+ if (rbio->stripe_pages[index])
2374+ continue;
2375+ page = alloc_page(GFP_NOFS);
2376+ if (!page)
2377+ return -ENOMEM;
2378+ rbio->stripe_pages[index] = page;
24292379 }
24302380 index_stripe_sectors(rbio);
24312381 return 0;
@@ -2437,7 +2387,7 @@ static noinline void finish_parity_scrub(struct btrfs_raid_bio *rbio, · 5 unmodified lines
24372387 struct btrfs_io_context *bioc = rbio->bioc;
24382388 const u32 sectorsize = bioc->fs_info->sectorsize;
24392389 void **pointers = rbio->finish_pointers;
2440− unsigned long *pbitmap = rbio->finish_pbitmap;
2390+ unsigned long *pbitmap = &rbio->finish_pbitmap;
24412391 int nr_data = rbio->nr_data;
24422392 int stripe;
24432393 int sectornr;
@@ -2460,7 +2410,7 @@ static noinline void finish_parity_scrub(struct btrfs_raid_bio *rbio, · 16 unmodified lines
24602410
24612411 if (bioc->num_tgtdevs && bioc->tgtdev_map[rbio->scrubp]) {
24622412 is_replace = 1;
2463− bitmap_copy(pbitmap, rbio->dbitmap, rbio->stripe_nsectors);
2413+ bitmap_copy(pbitmap, &rbio->dbitmap, rbio->stripe_nsectors);
24642414 }
24652415
24662416 /*
@@ -2497,7 +2447,7 @@ static noinline void finish_parity_scrub(struct btrfs_raid_bio *rbio, · 30 unmodified lines
24972447 /* Map the parity stripe just once */
24982448 pointers[nr_data] = kmap_local_page(p_sector.page);
24992449
2500− for_each_set_bit(sectornr, rbio->dbitmap, rbio->stripe_nsectors) {
2450+ for_each_set_bit(sectornr, &rbio->dbitmap, rbio->stripe_nsectors) {
25012451 struct sector_ptr *sector;
25022452 void *parity;
25032453
@@ -2525,7 +2475,7 @@ static noinline void finish_parity_scrub(struct btrfs_raid_bio *rbio, · 21 unmodified lines
25252475 memcpy(parity, pointers[rbio->scrubp], sectorsize);
25262476 else
25272477 /* Parity is right, needn't writeback */
2528− bitmap_clear(rbio->dbitmap, sectornr, 1);
2478+ bitmap_clear(&rbio->dbitmap, sectornr, 1);
25292479 kunmap_local(parity);
25302480
25312481 for (stripe = nr_data - 1; stripe >= 0; stripe--)
@@ -2547,12 +2497,12 @@ writeback: · 15 unmodified lines
25472497 * higher layers (the bio_list in our rbio) and our p/q. Ignore
25482498 * everything else.
25492499 */
2550− for_each_set_bit(sectornr, rbio->dbitmap, rbio->stripe_nsectors) {
2500+ for_each_set_bit(sectornr, &rbio->dbitmap, rbio->stripe_nsectors) {
25512501 struct sector_ptr *sector;
25522502
25532503 sector = rbio_stripe_sector(rbio, rbio->scrubp, sectornr);
25542504 ret = rbio_add_io_sector(rbio, &bio_list, sector, rbio->scrubp,
2555− sectornr, rbio->stripe_len, REQ_OP_WRITE);
2505+ sectornr, REQ_OP_WRITE);
25562506 if (ret)
25572507 goto cleanup;
25582508 }
@@ -2566,7 +2516,7 @@ writeback: · 7 unmodified lines
25662516 sector = rbio_stripe_sector(rbio, rbio->scrubp, sectornr);
25672517 ret = rbio_add_io_sector(rbio, &bio_list, sector,
25682518 bioc->tgtdev_map[rbio->scrubp],
2569− sectornr, rbio->stripe_len, REQ_OP_WRITE);
2519+ sectornr, REQ_OP_WRITE);
25702520 if (ret)
25712521 goto cleanup;
25722522 }
@@ -2584,6 +2534,12 @@ submit_write: · 11 unmodified lines
25842534 while ((bio = bio_list_pop(&bio_list))) {
25852535 bio->bi_end_io = raid_write_end_io;
25862536
2537+ if (trace_raid56_scrub_write_stripe_enabled()) {
2538+ struct raid56_bio_trace_info trace_info = { 0 };
2539+
2540+ bio_get_trace_info(rbio, bio, &trace_info);
2541+ trace_raid56_scrub_write_stripe(rbio, bio, &trace_info);
2542+ }
25872543 submit_bio(bio);
25882544 }
25892545 return;
@@ -2671,24 +2627,14 @@ cleanup: · 81 unmodified lines
26712627 * This will usually kick off finish_rmw once all the bios are read in, but it
26722628 * may trigger parity reconstruction if we had any errors along the way
26732629 */
2674−static void raid56_parity_scrub_end_io(struct bio *bio)
2630+static void raid56_parity_scrub_end_io_work(struct work_struct *work)
26752631 {
2676− struct btrfs_raid_bio *rbio = bio->bi_private;
2677−
2678− if (bio->bi_status)
2679− fail_bio_stripe(rbio, bio);
2680− else
2681− set_bio_pages_uptodate(rbio, bio);
2682−
2683− bio_put(bio);
2684−
2685− if (!atomic_dec_and_test(&rbio->stripes_pending))
2686− return;
2632+ struct btrfs_raid_bio *rbio =
2633+ container_of(work, struct btrfs_raid_bio, end_io_work);
26872634
26882635 /*
2689− * this will normally call finish_rmw to start our write
2690− * but if there are any failed stripes we'll reconstruct
2691− * from parity first
2636+ * This will normally call finish_rmw to start our write, but if there
2637+ * are any failed stripes we'll reconstruct from parity first
26922638 */
26932639 validate_rbio_for_parity_scrub(rbio);
26942640 }
@@ -2698,8 +2644,7 @@ static void raid56_parity_scrub_stripe(struct btrfs_raid_bio *rbio) · 3 unmodified lines
26982644 int bios_to_read = 0;
26992645 struct bio_list bio_list;
27002646 int ret;
2701− int sectornr;
2702− int stripe;
2647+ int total_sector_nr;
27032648 struct bio *bio;
27042649
27052650 bio_list_init(&bio_list);
@@ -2709,37 +2654,38 @@ static void raid56_parity_scrub_stripe(struct btrfs_raid_bio *rbio) · 3 unmodified lines
27092654 goto cleanup;
27102655
27112656 atomic_set(&rbio->error, 0);
2712− /*
2713− * build a list of bios to read all the missing parts of this
2714− * stripe
2715− */
2716− for (stripe = 0; stripe < rbio->real_stripes; stripe++) {
2717− for_each_set_bit(sectornr , rbio->dbitmap, rbio->stripe_nsectors) {
2718− struct sector_ptr *sector;
2719− /*
2720− * We want to find all the sectors missing from the
2721− * rbio and read them from the disk. If * sector_in_rbio()
2722− * finds a sector in the bio list we don't need to read
2723− * it off the stripe.
2724− */
2725− sector = sector_in_rbio(rbio, stripe, sectornr, 1);
2726− if (sector)
2727− continue;
2657+ /* Build a list of bios to read all the missing parts. */
2658+ for (total_sector_nr = 0; total_sector_nr < rbio->nr_sectors;
2659+ total_sector_nr++) {
2660+ int sectornr = total_sector_nr % rbio->stripe_nsectors;
2661+ int stripe = total_sector_nr / rbio->stripe_nsectors;
2662+ struct sector_ptr *sector;
27282663
2729− sector = rbio_stripe_sector(rbio, stripe, sectornr);
2730− /*
2731− * The bio cache may have handed us an uptodate sector.
2732− * If so, be happy and use it.
2733− */
2734− if (sector->uptodate)
2735− continue;
2664+ /* No data in the vertical stripe, no need to read. */
2665+ if (!test_bit(sectornr, &rbio->dbitmap))
2666+ continue;
27362667
2737− ret = rbio_add_io_sector(rbio, &bio_list, sector,
2738− stripe, sectornr, rbio->stripe_len,
2739− REQ_OP_READ);
2740− if (ret)
2741− goto cleanup;
2742− }
2668+ /*
2669+ * We want to find all the sectors missing from the rbio and
2670+ * read them from the disk. If sector_in_rbio() finds a sector
2671+ * in the bio list we don't need to read it off the stripe.
2672+ */
2673+ sector = sector_in_rbio(rbio, stripe, sectornr, 1);
2674+ if (sector)
2675+ continue;
2676+
2677+ sector = rbio_stripe_sector(rbio, stripe, sectornr);
2678+ /*
2679+ * The bio cache may have handed us an uptodate sector. If so,
2680+ * use it.
2681+ */
2682+ if (sector->uptodate)
2683+ continue;
2684+
2685+ ret = rbio_add_io_sector(rbio, &bio_list, sector, stripe,
2686+ sectornr, REQ_OP_READ);
2687+ if (ret)
2688+ goto cleanup;
27432689 }
27442690
27452691 bios_to_read = bio_list_size(&bio_list);
@@ -2758,11 +2704,16 @@ static void raid56_parity_scrub_stripe(struct btrfs_raid_bio *rbio) · 12 unmodified lines
27582704 * touch it after that.
27592705 */
27602706 atomic_set(&rbio->stripes_pending, bios_to_read);
2707+ INIT_WORK(&rbio->end_io_work, raid56_parity_scrub_end_io_work);
27612708 while ((bio = bio_list_pop(&bio_list))) {
2762− bio->bi_end_io = raid56_parity_scrub_end_io;
2709+ bio->bi_end_io = raid56_bio_end_io;
27632710
2764− btrfs_bio_wq_end_io(rbio->bioc->fs_info, bio, BTRFS_WQ_ENDIO_RAID56);
2711+ if (trace_raid56_scrub_read_enabled()) {
2712+ struct raid56_bio_trace_info trace_info = { 0 };
27652713
2714+ bio_get_trace_info(rbio, bio, &trace_info);
2715+ trace_raid56_scrub_read(rbio, bio, &trace_info);
2716+ }
27662717 submit_bio(bio);
27672718 }
27682719 /* the actual write will happen once the reads are done */
@@ -2797,13 +2748,12 @@ void raid56_parity_submit_scrub_rbio(struct btrfs_raid_bio *rbio) · 28 unmodified lines
27972748 /* The following code is used for dev replace of a missing RAID 5/6 device. */
27982749
27992750 struct btrfs_raid_bio *
2800−raid56_alloc_missing_rbio(struct bio *bio, struct btrfs_io_context *bioc,
2801− u64 length)
2751+raid56_alloc_missing_rbio(struct bio *bio, struct btrfs_io_context *bioc)
28022752 {
28032753 struct btrfs_fs_info *fs_info = bioc->fs_info;
28042754 struct btrfs_raid_bio *rbio;
28052755
2806− rbio = alloc_rbio(fs_info, bioc, length);
2756+ rbio = alloc_rbio(fs_info, bioc);
28072757 if (IS_ERR(rbio))
28082758 return NULL;
28092759
fs/btrfs/raid56.h+151−17
@@ -7,45 +7,179 @@ · 6 unmodified lines
77 #ifndef BTRFS_RAID56_H
88 #define BTRFS_RAID56_H
99
10−static inline int nr_parity_stripes(const struct map_lookup *map)
11−{
12− if (map->type & BTRFS_BLOCK_GROUP_RAID5)
13− return 1;
14− else if (map->type & BTRFS_BLOCK_GROUP_RAID6)
15− return 2;
16− else
17− return 0;
18−}
10+#include <linux/workqueue.h>
11+#include "volumes.h"
12+
13+enum btrfs_rbio_ops {
14+ BTRFS_RBIO_WRITE,
15+ BTRFS_RBIO_READ_REBUILD,
16+ BTRFS_RBIO_PARITY_SCRUB,
17+ BTRFS_RBIO_REBUILD_MISSING,
18+};
19+
20+struct btrfs_raid_bio {
21+ struct btrfs_io_context *bioc;
22+
23+ /*
24+ * While we're doing RMW on a stripe we put it into a hash table so we
25+ * can lock the stripe and merge more rbios into it.
26+ */
27+ struct list_head hash_list;
28+
29+ /* LRU list for the stripe cache */
30+ struct list_head stripe_cache;
31+
32+ /* For scheduling work in the helper threads */
33+ struct work_struct work;
34+
35+ /*
36+ * bio_list and bio_list_lock are used to add more bios into the stripe
37+ * in hopes of avoiding the full RMW
38+ */
39+ struct bio_list bio_list;
40+ spinlock_t bio_list_lock;
41+
42+ /*
43+ * Also protected by the bio_list_lock, the plug list is used by the
44+ * plugging code to collect partial bios while plugged. The stripe
45+ * locking code also uses it to hand off the stripe lock to the next
46+ * pending IO.
47+ */
48+ struct list_head plug_list;
49+
50+ /* Flags that tell us if it is safe to merge with this bio. */
51+ unsigned long flags;
52+
53+ /*
54+ * Set if we're doing a parity rebuild for a read from higher up, which
55+ * is handled differently from a parity rebuild as part of RMW.
56+ */
57+ enum btrfs_rbio_ops operation;
58+
59+ /* How many pages there are for the full stripe including P/Q */
60+ u16 nr_pages;
61+
62+ /* How many sectors there are for the full stripe including P/Q */
63+ u16 nr_sectors;
64+
65+ /* Number of data stripes (no p/q) */
66+ u8 nr_data;
67+
68+ /* Numer of all stripes (including P/Q) */
69+ u8 real_stripes;
70+
71+ /* How many pages there are for each stripe */
72+ u8 stripe_npages;
73+
74+ /* How many sectors there are for each stripe */
75+ u8 stripe_nsectors;
76+
77+ /* First bad stripe, -1 means no corruption */
78+ s8 faila;
79+
80+ /* Second bad stripe (for RAID6 use) */
81+ s8 failb;
82+
83+ /* Stripe number that we're scrubbing */
84+ u8 scrubp;
85+
86+ /*
87+ * Size of all the bios in the bio_list. This helps us decide if the
88+ * rbio maps to a full stripe or not.
89+ */
90+ int bio_list_bytes;
91+
92+ int generic_bio_cnt;
93+
94+ refcount_t refs;
95+
96+ atomic_t stripes_pending;
97+
98+ atomic_t error;
99+
100+ struct work_struct end_io_work;
101+
102+ /* Bitmap to record which horizontal stripe has data */
103+ unsigned long dbitmap;
104+
105+ /* Allocated with stripe_nsectors-many bits for finish_*() calls */
106+ unsigned long finish_pbitmap;
107+
108+ /*
109+ * These are two arrays of pointers. We allocate the rbio big enough
110+ * to hold them both and setup their locations when the rbio is
111+ * allocated.
112+ */
113+
114+ /*
115+ * Pointers to pages that we allocated for reading/writing stripes
116+ * directly from the disk (including P/Q).
117+ */
118+ struct page **stripe_pages;
119+
120+ /* Pointers to the sectors in the bio_list, for faster lookup */
121+ struct sector_ptr *bio_sectors;
122+
123+ /*
124+ * For subpage support, we need to map each sector to above
125+ * stripe_pages.
126+ */
127+ struct sector_ptr *stripe_sectors;
128+
129+ /* Allocated with real_stripes-many pointers for finish_*() calls */
130+ void **finish_pointers;
131+};
132+
133+/*
134+ * For trace event usage only. Records useful debug info for each bio submitted
135+ * by RAID56 to each physical device.
136+ *
137+ * No matter signed or not, (-1) is always the one indicating we can not grab
138+ * the proper stripe number.
139+ */
140+struct raid56_bio_trace_info {
141+ u64 devid;
142+
143+ /* The offset inside the stripe. (<= STRIPE_LEN) */
144+ u32 offset;
145+
146+ /*
147+ * Stripe number.
148+ * 0 is the first data stripe, and nr_data for P stripe,
149+ * nr_data + 1 for Q stripe.
150+ * >= real_stripes for
151+ */
152+ u8 stripe_nr;
153+};
19154
20155 static inline int nr_data_stripes(const struct map_lookup *map)
21156 {
22− return map->num_stripes - nr_parity_stripes(map);
157+ return map->num_stripes - btrfs_nr_parity_stripes(map->type);
23158 }
159+
24160 #define RAID5_P_STRIPE ((u64)-2)
25161 #define RAID6_Q_STRIPE ((u64)-1)
26162
27163 #define is_parity_stripe(x) (((x) == RAID5_P_STRIPE) || \
28164 ((x) == RAID6_Q_STRIPE))
29165
30−struct btrfs_raid_bio;
31166 struct btrfs_device;
32167
33−int raid56_parity_recover(struct bio *bio, struct btrfs_io_context *bioc,
34− u32 stripe_len, int mirror_num, int generic_io);
35−int raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc, u32 stripe_len);
168+void raid56_parity_recover(struct bio *bio, struct btrfs_io_context *bioc,
169+ int mirror_num, bool generic_io);
170+void raid56_parity_write(struct bio *bio, struct btrfs_io_context *bioc);
36171
37172 void raid56_add_scrub_pages(struct btrfs_raid_bio *rbio, struct page *page,
38173 unsigned int pgoff, u64 logical);
39174
40175 struct btrfs_raid_bio *raid56_parity_alloc_scrub_rbio(struct bio *bio,
41− struct btrfs_io_context *bioc, u32 stripe_len,
176+ struct btrfs_io_context *bioc,
42177 struct btrfs_device *scrub_dev,
43178 unsigned long *dbitmap, int stripe_nsectors);
44179 void raid56_parity_submit_scrub_rbio(struct btrfs_raid_bio *rbio);
45180
46181 struct btrfs_raid_bio *
47−raid56_alloc_missing_rbio(struct bio *bio, struct btrfs_io_context *bioc,
48− u64 length);
182+raid56_alloc_missing_rbio(struct bio *bio, struct btrfs_io_context *bioc);
49183 void raid56_submit_missing_rbio(struct btrfs_raid_bio *rbio);
50184
51185 int btrfs_alloc_stripe_hash_table(struct btrfs_fs_info *info);
fs/btrfs/reflink.c+12−7
@@ -5,6 +5,7 @@ · 4 unmodified lines
55 #include "compression.h"
66 #include "ctree.h"
77 #include "delalloc-space.h"
8+#include "disk-io.h"
89 #include "reflink.h"
910 #include "transaction.h"
1011 #include "subpage.h"
@@ -22,8 +23,10 @@ static int clone_finish_inode_update(struct btrfs_trans_handle *trans, · 11 unmodified lines
2223 int ret;
2324
2425 inode_inc_iversion(inode);
25− if (!no_time_update)
26− inode->i_mtime = inode->i_ctime = current_time(inode);
26+ if (!no_time_update) {
27+ inode->i_mtime = current_time(inode);
28+ inode->i_ctime = inode->i_mtime;
29+ }
2730 /*
2831 * We round up to the block size at eof when determining which
2932 * extents to clone above, but shouldn't round up the file size.
@@ -110,7 +113,6 @@ static int copy_inline_to_page(struct btrfs_inode *inode, · 80 unmodified lines
110113 if (comp_type == BTRFS_COMPRESS_NONE) {
111114 memcpy_to_page(page, offset_in_page(file_offset), data_start,
112115 datal);
113− flush_dcache_page(page);
114116 } else {
115117 ret = btrfs_decompress(comp_type, data_start, page,
116118 offset_in_page(file_offset),
@@ -132,10 +134,8 @@ static int copy_inline_to_page(struct btrfs_inode *inode, · 15 unmodified lines
132134 *
133135 * So what's in the range [500, 4095] corresponds to zeroes.
134136 */
135− if (datal < block_size) {
137+ if (datal < block_size)
136138 memzero_page(page, datal, block_size - datal);
137− flush_dcache_page(page);
138− }
139139
140140 btrfs_page_set_uptodate(fs_info, page, file_offset, block_size);
141141 btrfs_page_clear_checked(fs_info, page, file_offset, block_size);
@@ -658,7 +658,8 @@ static void btrfs_double_mmap_unlock(struct inode *inode1, struct inode *inode2) · 516 unmodified lines
658658 static int btrfs_extent_same_range(struct inode *src, u64 loff, u64 len,
659659 struct inode *dst, u64 dst_loff)
660660 {
661− const u64 bs = BTRFS_I(src)->root->fs_info->sb->s_blocksize;
661+ struct btrfs_fs_info *fs_info = BTRFS_I(src)->root->fs_info;
662+ const u64 bs = fs_info->sb->s_blocksize;
662663 int ret;
663664
664665 /*
@@ -669,6 +670,8 @@ static int btrfs_extent_same_range(struct inode *src, u64 loff, u64 len, · 4 unmodified lines
669670 ret = btrfs_clone(src, dst, loff, len, ALIGN(len, bs), dst_loff, 1);
670671 btrfs_double_extent_unlock(src, loff, dst, dst_loff, len);
671672
673+ btrfs_btree_balance_dirty(fs_info);
674+
672675 return ret;
673676 }
674677
@@ -778,6 +781,8 @@ static noinline int btrfs_clone_files(struct file *file, struct file *file_src, · 103 unmodified lines
778781 round_down(destoff, PAGE_SIZE),
779782 round_up(destoff + len, PAGE_SIZE) - 1);
780783
784+ btrfs_btree_balance_dirty(fs_info);
785+
781786 return ret;
782787 }
783788
fs/btrfs/scrub.c+26−45
@@ -135,15 +135,13 @@ struct scrub_parity { · 134 unmodified lines
135135 struct work_struct work;
136136
137137 /* Mark the parity blocks which have data */
138− unsigned long *dbitmap;
138+ unsigned long dbitmap;
139139
140140 /*
141141 * Mark the parity blocks which have data, but errors happen when
142142 * read data or check data
143143 */
144− unsigned long *ebitmap;
145−
146− unsigned long bitmap[];
144+ unsigned long ebitmap;
147145 };
148146
149147 struct scrub_ctx {
@@ -1218,7 +1216,6 @@ static inline int scrub_nr_raid_mirrors(struct btrfs_io_context *bioc) · 1068 unmodified lines
12181216
12191217 static inline void scrub_stripe_index_and_offset(u64 logical, u64 map_type,
12201218 u64 *raid_map,
1221− u64 mapped_length,
12221219 int nstripes, int mirror,
12231220 int *stripe_index,
12241221 u64 *stripe_offset)
@@ -1233,7 +1230,7 @@ static inline void scrub_stripe_index_and_offset(u64 logical, u64 map_type, · 8 unmodified lines
12331230 continue;
12341231
12351232 if (logical >= raid_map[i] &&
1236− logical < raid_map[i] + mapped_length)
1233+ logical < raid_map[i] + BTRFS_STRIPE_LEN)
12371234 break;
12381235 }
12391236
@@ -1337,7 +1334,6 @@ leave_nomem: · 97 unmodified lines
13371334 scrub_stripe_index_and_offset(logical,
13381335 bioc->map_type,
13391336 bioc->raid_map,
1340− mapped_length,
13411337 bioc->num_stripes -
13421338 bioc->num_tgtdevs,
13431339 mirror_index,
@@ -1380,19 +1376,12 @@ static int scrub_submit_raid56_bio_wait(struct btrfs_fs_info *fs_info, · 36 unmodified lines
13801376 struct scrub_sector *sector)
13811377 {
13821378 DECLARE_COMPLETION_ONSTACK(done);
1383− int ret;
1384− int mirror_num;
13851379
13861380 bio->bi_iter.bi_sector = sector->logical >> 9;
13871381 bio->bi_private = &done;
13881382 bio->bi_end_io = scrub_bio_wait_endio;
1389−
1390− mirror_num = sector->sblock->sectors[0]->mirror_num;
1391− ret = raid56_parity_recover(bio, sector->recover->bioc,
1392− sector->recover->map_length,
1393− mirror_num, 0);
1394− if (ret)
1395− return ret;
1383+ raid56_parity_recover(bio, sector->recover->bioc,
1384+ sector->sblock->sectors[0]->mirror_num, false);
13961385
13971386 wait_for_completion_io(&done);
13981387 return blk_status_to_errno(bio->bi_status);
@@ -2197,7 +2186,7 @@ static void scrub_missing_raid56_pages(struct scrub_block *sblock) · 798 unmodified lines
21972186 bio->bi_private = sblock;
21982187 bio->bi_end_io = scrub_missing_raid56_end_io;
21992188
2200− rbio = raid56_alloc_missing_rbio(bio, bioc, length);
2189+ rbio = raid56_alloc_missing_rbio(bio, bioc);
22012190 if (!rbio)
22022191 goto rbio_out;
22032192
@@ -2406,13 +2395,13 @@ static inline void __scrub_mark_bitmap(struct scrub_parity *sparity, · 202 unmodified lines
24062395 static inline void scrub_parity_mark_sectors_error(struct scrub_parity *sparity,
24072396 u64 start, u32 len)
24082397 {
2409− __scrub_mark_bitmap(sparity, sparity->ebitmap, start, len);
2398+ __scrub_mark_bitmap(sparity, &sparity->ebitmap, start, len);
24102399 }
24112400
24122401 static inline void scrub_parity_mark_sectors_data(struct scrub_parity *sparity,
24132402 u64 start, u32 len)
24142403 {
2415− __scrub_mark_bitmap(sparity, sparity->dbitmap, start, len);
2404+ __scrub_mark_bitmap(sparity, &sparity->dbitmap, start, len);
24162405 }
24172406
24182407 static void scrub_block_complete(struct scrub_block *sblock)
@@ -2763,7 +2752,7 @@ static void scrub_free_parity(struct scrub_parity *sparity) · 344 unmodified lines
27632752 struct scrub_sector *curr, *next;
27642753 int nbits;
27652754
2766− nbits = bitmap_weight(sparity->ebitmap, sparity->nsectors);
2755+ nbits = bitmap_weight(&sparity->ebitmap, sparity->nsectors);
27672756 if (nbits) {
27682757 spin_lock(&sctx->stat_lock);
27692758 sctx->stat.read_errors += nbits;
@@ -2795,8 +2784,8 @@ static void scrub_parity_bio_endio(struct bio *bio) · 25 unmodified lines
27952784 struct btrfs_fs_info *fs_info = sparity->sctx->fs_info;
27962785
27972786 if (bio->bi_status)
2798− bitmap_or(sparity->ebitmap, sparity->ebitmap, sparity->dbitmap,
2799− sparity->nsectors);
2787+ bitmap_or(&sparity->ebitmap, &sparity->ebitmap,
2788+ &sparity->dbitmap, sparity->nsectors);
28002789
28012790 bio_put(bio);
28022791
@@ -2814,8 +2803,8 @@ static void scrub_parity_check_and_repair(struct scrub_parity *sparity) · 11 unmodified lines
28142803 u64 length;
28152804 int ret;
28162805
2817− if (!bitmap_andnot(sparity->dbitmap, sparity->dbitmap, sparity->ebitmap,
2818− sparity->nsectors))
2806+ if (!bitmap_andnot(&sparity->dbitmap, &sparity->dbitmap,
2807+ &sparity->ebitmap, sparity->nsectors))
28192808 goto out;
28202809
28212810 length = sparity->logic_end - sparity->logic_start;
@@ -2831,9 +2820,9 @@ static void scrub_parity_check_and_repair(struct scrub_parity *sparity) · 9 unmodified lines
28312820 bio->bi_private = sparity;
28322821 bio->bi_end_io = scrub_parity_bio_endio;
28332822
2834− rbio = raid56_parity_alloc_scrub_rbio(bio, bioc, length,
2823+ rbio = raid56_parity_alloc_scrub_rbio(bio, bioc,
28352824 sparity->scrub_dev,
2836− sparity->dbitmap,
2825+ &sparity->dbitmap,
28372826 sparity->nsectors);
28382827 if (!rbio)
28392828 goto rbio_out;
@@ -2847,7 +2836,7 @@ rbio_out: · 7 unmodified lines
28472836 bioc_out:
28482837 btrfs_bio_counter_dec(fs_info);
28492838 btrfs_put_bioc(bioc);
2850− bitmap_or(sparity->ebitmap, sparity->ebitmap, sparity->dbitmap,
2839+ bitmap_or(&sparity->ebitmap, &sparity->ebitmap, &sparity->dbitmap,
28512840 sparity->nsectors);
28522841 spin_lock(&sctx->stat_lock);
28532842 sctx->stat.malloc_errors++;
@@ -2856,11 +2845,6 @@ out: · 2 unmodified lines
28562845 scrub_free_parity(sparity);
28572846 }
28582847
2859−static inline int scrub_calc_parity_bitmap_len(int nsectors)
2860−{
2861− return DIV_ROUND_UP(nsectors, BITS_PER_LONG) * sizeof(long);
2862−}
2863−
28642848 static void scrub_parity_get(struct scrub_parity *sparity)
28652849 {
28662850 refcount_inc(&sparity->refs);
@@ -3131,7 +3115,6 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx, · 264 unmodified lines
31313115 int ret;
31323116 struct scrub_parity *sparity;
31333117 int nsectors;
3134− int bitmap_len;
31353118
31363119 path = btrfs_alloc_path();
31373120 if (!path) {
@@ -3145,9 +3128,8 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx, · 7 unmodified lines
31453128
31463129 ASSERT(map->stripe_len <= U32_MAX);
31473130 nsectors = map->stripe_len >> fs_info->sectorsize_bits;
3148− bitmap_len = scrub_calc_parity_bitmap_len(nsectors);
3149− sparity = kzalloc(sizeof(struct scrub_parity) + 2 * bitmap_len,
3150− GFP_NOFS);
3131+ ASSERT(nsectors <= BITS_PER_LONG);
3132+ sparity = kzalloc(sizeof(struct scrub_parity), GFP_NOFS);
31513133 if (!sparity) {
31523134 spin_lock(&sctx->stat_lock);
31533135 sctx->stat.malloc_errors++;
@@ -3165,8 +3147,6 @@ static noinline_for_stack int scrub_raid56_parity(struct scrub_ctx *sctx, · 11 unmodified lines
31653147 sparity->logic_end = logic_end;
31663148 refcount_set(&sparity->refs, 1);
31673149 INIT_LIST_HEAD(&sparity->sectors_list);
3168− sparity->dbitmap = sparity->bitmap;
3169− sparity->ebitmap = (void *)sparity->bitmap + bitmap_len;
31703150
31713151 ret = 0;
31723152 for (cur_logical = logic_start; cur_logical < logic_end;
@@ -3429,20 +3409,22 @@ static int scrub_simple_stripe(struct scrub_ctx *sctx, · 256 unmodified lines
34293409
34303410 static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
34313411 struct btrfs_block_group *bg,
3432− struct map_lookup *map,
3412+ struct extent_map *em,
34333413 struct btrfs_device *scrub_dev,
3434− int stripe_index, u64 dev_extent_len)
3414+ int stripe_index)
34353415 {
34363416 struct btrfs_path *path;
34373417 struct btrfs_fs_info *fs_info = sctx->fs_info;
34383418 struct btrfs_root *root;
34393419 struct btrfs_root *csum_root;
34403420 struct blk_plug plug;
3421+ struct map_lookup *map = em->map_lookup;
34413422 const u64 profile = map->type & BTRFS_BLOCK_GROUP_PROFILE_MASK;
34423423 const u64 chunk_logical = bg->start;
34433424 int ret;
34443425 u64 physical = map->stripes[stripe_index].physical;
3445− const u64 physical_end = physical + dev_extent_len;
3426+ const u64 dev_stripe_len = btrfs_calc_stripe_length(em);
3427+ const u64 physical_end = physical + dev_stripe_len;
34463428 u64 logical;
34473429 u64 logic_end;
34483430 /* The logical increment after finishing one stripe */
@@ -3569,8 +3551,8 @@ next: · 120 unmodified lines
35693551 physical += map->stripe_len;
35703552 spin_lock(&sctx->stat_lock);
35713553 if (stop_loop)
3572− sctx->stat.last_physical = map->stripes[stripe_index].physical +
3573− dev_extent_len;
3554+ sctx->stat.last_physical =
3555+ map->stripes[stripe_index].physical + dev_stripe_len;
35743556 else
35753557 sctx->stat.last_physical = physical;
35763558 spin_unlock(&sctx->stat_lock);
@@ -3639,8 +3621,7 @@ static noinline_for_stack int scrub_chunk(struct scrub_ctx *sctx, · 62 unmodified lines
36393621 for (i = 0; i < map->num_stripes; ++i) {
36403622 if (map->stripes[i].dev->bdev == scrub_dev->bdev &&
36413623 map->stripes[i].physical == dev_offset) {
3642− ret = scrub_stripe(sctx, bg, map, scrub_dev, i,
3643− dev_extent_len);
3624+ ret = scrub_stripe(sctx, bg, em, scrub_dev, i);
36443625 if (ret)
36453626 goto out;
36463627 }
fs/btrfs/send.c+533−248
@@ -17,6 +17,7 @@ · 16 unmodified lines
1717 #include <linux/crc32c.h>
1818
1919 #include "send.h"
20+#include "ctree.h"
2021 #include "backref.h"
2122 #include "locking.h"
2223 #include "disk-io.h"
@@ -82,8 +83,12 @@ struct send_ctx { · 59 unmodified lines
8283 char *send_buf;
8384 u32 send_size;
8485 u32 send_max_size;
85− u64 total_send_size;
86− u64 cmd_send_size[BTRFS_SEND_C_MAX + 1];
86+ /*
87+ * Whether BTRFS_SEND_A_DATA attribute was already added to current
88+ * command (since protocol v2, data must be the last attribute).
89+ */
90+ bool put_data;
91+ struct page **send_buf_pages;
8792 u64 flags; /* 'flags' member of btrfs_ioctl_send_args is u64 */
8893 /* Protocol version compatibility requested */
8994 u32 proto;
@@ -113,14 +118,14 @@ struct send_ctx { · 23 unmodified lines
113118 */
114119 u64 cur_ino;
115120 u64 cur_inode_gen;
116− int cur_inode_new;
117− int cur_inode_new_gen;
118− int cur_inode_deleted;
119121 u64 cur_inode_size;
120122 u64 cur_inode_mode;
121123 u64 cur_inode_rdev;
122124 u64 cur_inode_last_extent;
123125 u64 cur_inode_next_write_offset;
126+ bool cur_inode_new;
127+ bool cur_inode_new_gen;
128+ bool cur_inode_deleted;
124129 bool ignore_cur_inode;
125130
126131 u64 send_progress;
@@ -235,6 +240,9 @@ struct send_ctx { · 108 unmodified lines
235240 * Indexed by the inode number of the directory to be deleted.
236241 */
237242 struct rb_root orphan_dirs;
243+
244+ struct rb_root rbtree_new_refs;
245+ struct rb_root rbtree_deleted_refs;
238246 };
239247
240248 struct pending_dir_move {
@@ -335,8 +343,8 @@ __maybe_unused · 94 unmodified lines
335343 static bool proto_cmd_ok(const struct send_ctx *sctx, int cmd)
336344 {
337345 switch (sctx->proto) {
338− case 1: return cmd < __BTRFS_SEND_C_MAX_V1;
339− case 2: return cmd < __BTRFS_SEND_C_MAX_V2;
346+ case 1: return cmd <= BTRFS_SEND_C_MAX_V1;
347+ case 2: return cmd <= BTRFS_SEND_C_MAX_V2;
340348 default: return false;
341349 }
342350 }
@@ -577,15 +585,10 @@ static int write_buf(struct file *filp, const void *buf, u32 len, loff_t *off) · 234 unmodified lines
577585
578586 while (pos < len) {
579587 ret = kernel_write(filp, buf + pos, len - pos, off);
580− /* TODO handle that correctly */
581− /*if (ret == -ERESTARTSYS) {
582− continue;
583− }*/
584588 if (ret < 0)
585589 return ret;
586− if (ret == 0) {
590+ if (ret == 0)
587591 return -EIO;
588− }
589592 pos += ret;
590593 }
591594
@@ -598,6 +601,9 @@ static int tlv_put(struct send_ctx *sctx, u16 attr, const void *data, int len) · 6 unmodified lines
598601 int total_len = sizeof(*hdr) + len;
599602 int left = sctx->send_max_size - sctx->send_size;
600603
604+ if (WARN_ON_ONCE(sctx->put_data))
605+ return -EINVAL;
606+
601607 if (unlikely(left < total_len))
602608 return -EOVERFLOW;
603609
@@ -618,6 +624,7 @@ static int tlv_put(struct send_ctx *sctx, u16 attr, const void *data, int len) · 14 unmodified lines
618624 return tlv_put(sctx, attr, &__tmp, sizeof(__tmp)); \
619625 }
620626
627+TLV_PUT_DEFINE_INT(32)
621628 TLV_PUT_DEFINE_INT(64)
622629
623630 static int tlv_put_string(struct send_ctx *sctx, u16 attr,
@@ -693,8 +700,7 @@ static int send_header(struct send_ctx *sctx) · 69 unmodified lines
693700 struct btrfs_stream_header hdr;
694701
695702 strcpy(hdr.magic, BTRFS_SEND_STREAM_MAGIC);
696− hdr.version = cpu_to_le32(BTRFS_SEND_STREAM_VERSION);
697−
703+ hdr.version = cpu_to_le32(sctx->proto);
698704 return write_buf(sctx->send_filp, &hdr, sizeof(hdr),
699705 &sctx->send_off);
700706 }
@@ -734,9 +740,8 @@ static int send_cmd(struct send_ctx *sctx) · 33 unmodified lines
734740 ret = write_buf(sctx->send_filp, sctx->send_buf, sctx->send_size,
735741 &sctx->send_off);
736742
737− sctx->total_send_size += sctx->send_size;
738− sctx->cmd_send_size[get_unaligned_le16(&hdr->cmd)] += sctx->send_size;
739743 sctx->send_size = 0;
744+ sctx->put_data = false;
740745
741746 return ret;
742747 }
@@ -842,7 +847,7 @@ out: · 99 unmodified lines
842847 */
843848 static int __get_inode_info(struct btrfs_root *root, struct btrfs_path *path,
844849 u64 ino, u64 *size, u64 *gen, u64 *mode, u64 *uid,
845− u64 *gid, u64 *rdev)
850+ u64 *gid, u64 *rdev, u64 *fileattr)
846851 {
847852 int ret;
848853 struct btrfs_inode_item *ii;
@@ -872,6 +877,12 @@ static int __get_inode_info(struct btrfs_root *root, struct btrfs_path *path, · 23 unmodified lines
872877 *gid = btrfs_inode_gid(path->nodes[0], ii);
873878 if (rdev)
874879 *rdev = btrfs_inode_rdev(path->nodes[0], ii);
880+ /*
881+ * Transfer the unchanged u64 value of btrfs_inode_item::flags, that's
882+ * otherwise logically split to 32/32 parts.
883+ */
884+ if (fileattr)
885+ *fileattr = btrfs_inode_flags(path->nodes[0], ii);
875886
876887 return ret;
877888 }
@@ -879,7 +890,7 @@ static int __get_inode_info(struct btrfs_root *root, struct btrfs_path *path, · 1 unmodified line
879890 static int get_inode_info(struct btrfs_root *root,
880891 u64 ino, u64 *size, u64 *gen,
881892 u64 *mode, u64 *uid, u64 *gid,
882− u64 *rdev)
893+ u64 *rdev, u64 *fileattr)
883894 {
884895 struct btrfs_path *path;
885896 int ret;
@@ -888,7 +899,7 @@ static int get_inode_info(struct btrfs_root *root, · 2 unmodified lines
888899 if (!path)
889900 return -ENOMEM;
890901 ret = __get_inode_info(root, path, ino, size, gen, mode, uid, gid,
891− rdev);
902+ rdev, fileattr);
892903 btrfs_free_path(path);
893904 return ret;
894905 }
@@ -1634,7 +1645,7 @@ static int get_cur_inode_state(struct send_ctx *sctx, u64 ino, u64 gen) · 739 unmodified lines
16341645 u64 right_gen;
16351646
16361647 ret = get_inode_info(sctx->send_root, ino, NULL, &left_gen, NULL, NULL,
1637− NULL, NULL);
1648+ NULL, NULL, NULL);
16381649 if (ret < 0 && ret != -ENOENT)
16391650 goto out;
16401651 left_ret = ret;
@@ -1643,7 +1654,7 @@ static int get_cur_inode_state(struct send_ctx *sctx, u64 ino, u64 gen) · 2 unmodified lines
16431654 right_ret = -ENOENT;
16441655 } else {
16451656 ret = get_inode_info(sctx->parent_root, ino, NULL, &right_gen,
1646− NULL, NULL, NULL, NULL);
1657+ NULL, NULL, NULL, NULL, NULL);
16471658 if (ret < 0 && ret != -ENOENT)
16481659 goto out;
16491660 right_ret = ret;
@@ -1806,7 +1817,7 @@ static int get_first_ref(struct btrfs_root *root, u64 ino, · 156 unmodified lines
18061817
18071818 if (dir_gen) {
18081819 ret = get_inode_info(root, parent_dir, NULL, dir_gen, NULL,
1809− NULL, NULL, NULL);
1820+ NULL, NULL, NULL, NULL);
18101821 if (ret < 0)
18111822 goto out;
18121823 }
@@ -1878,7 +1889,7 @@ static int will_overwrite_ref(struct send_ctx *sctx, u64 dir, u64 dir_gen, · 65 unmodified lines
18781889 */
18791890 if (sctx->parent_root && dir != BTRFS_FIRST_FREE_OBJECTID) {
18801891 ret = get_inode_info(sctx->parent_root, dir, NULL, &gen, NULL,
1881− NULL, NULL, NULL);
1892+ NULL, NULL, NULL, NULL);
18821893 if (ret < 0 && ret != -ENOENT)
18831894 goto out;
18841895 if (ret) {
@@ -1906,7 +1917,7 @@ static int will_overwrite_ref(struct send_ctx *sctx, u64 dir, u64 dir_gen, · 21 unmodified lines
19061917 if (other_inode > sctx->send_progress ||
19071918 is_waiting_for_move(sctx, other_inode)) {
19081919 ret = get_inode_info(sctx->parent_root, other_inode, NULL,
1909− who_gen, who_mode, NULL, NULL, NULL);
1920+ who_gen, who_mode, NULL, NULL, NULL, NULL);
19101921 if (ret < 0)
19111922 goto out;
19121923
@@ -1945,7 +1956,7 @@ static int did_overwrite_ref(struct send_ctx *sctx, · 32 unmodified lines
19451956
19461957 if (dir != BTRFS_FIRST_FREE_OBJECTID) {
19471958 ret = get_inode_info(sctx->send_root, dir, NULL, &gen, NULL,
1948− NULL, NULL, NULL);
1959+ NULL, NULL, NULL, NULL);
19491960 if (ret < 0 && ret != -ENOENT)
19501961 goto out;
19511962 if (ret) {
@@ -1968,7 +1979,7 @@ static int did_overwrite_ref(struct send_ctx *sctx, · 16 unmodified lines
19681979 }
19691980
19701981 ret = get_inode_info(sctx->send_root, ow_inode, NULL, &gen, NULL, NULL,
1971− NULL, NULL);
1982+ NULL, NULL, NULL);
19721983 if (ret < 0)
19731984 goto out;
19741985
@@ -2184,7 +2195,7 @@ static int __get_cur_name_and_parent(struct send_ctx *sctx, · 209 unmodified lines
21842195 /*
21852196 * If the inode is not existent yet, add the orphan name and return 1.
21862197 * This should only happen for the parent dir that we determine in
2187− * __record_new_ref
2198+ * record_new_ref_if_needed().
21882199 */
21892200 ret = is_inode_existent(sctx, ino, gen);
21902201 if (ret < 0)
@@ -2499,6 +2510,39 @@ out: · 308 unmodified lines
24992510 return ret;
25002511 }
25012512
2513+static int send_fileattr(struct send_ctx *sctx, u64 ino, u64 gen, u64 fileattr)
2514+{
2515+ struct btrfs_fs_info *fs_info = sctx->send_root->fs_info;
2516+ int ret = 0;
2517+ struct fs_path *p;
2518+
2519+ if (sctx->proto < 2)
2520+ return 0;
2521+
2522+ btrfs_debug(fs_info, "send_fileattr %llu fileattr=%llu", ino, fileattr);
2523+
2524+ p = fs_path_alloc();
2525+ if (!p)
2526+ return -ENOMEM;
2527+
2528+ ret = begin_cmd(sctx, BTRFS_SEND_C_FILEATTR);
2529+ if (ret < 0)
2530+ goto out;
2531+
2532+ ret = get_cur_path(sctx, ino, gen, p);
2533+ if (ret < 0)
2534+ goto out;
2535+ TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, p);
2536+ TLV_PUT_U64(sctx, BTRFS_SEND_A_FILEATTR, fileattr);
2537+
2538+ ret = send_cmd(sctx);
2539+
2540+tlv_put_failure:
2541+out:
2542+ fs_path_free(p);
2543+ return ret;
2544+}
2545+
25022546 static int send_chown(struct send_ctx *sctx, u64 ino, u64 gen, u64 uid, u64 gid)
25032547 {
25042548 struct btrfs_fs_info *fs_info = sctx->send_root->fs_info;
@@ -2578,7 +2622,8 @@ static int send_utimes(struct send_ctx *sctx, u64 ino, u64 gen) · 73 unmodified lines
25782622 TLV_PUT_BTRFS_TIMESPEC(sctx, BTRFS_SEND_A_ATIME, eb, &ii->atime);
25792623 TLV_PUT_BTRFS_TIMESPEC(sctx, BTRFS_SEND_A_MTIME, eb, &ii->mtime);
25802624 TLV_PUT_BTRFS_TIMESPEC(sctx, BTRFS_SEND_A_CTIME, eb, &ii->ctime);
2581− /* TODO Add otime support when the otime patches get into upstream */
2625+ if (sctx->proto >= 2)
2626+ TLV_PUT_BTRFS_TIMESPEC(sctx, BTRFS_SEND_A_OTIME, eb, &ii->otime);
25822627
25832628 ret = send_cmd(sctx);
25842629
@@ -2612,7 +2657,7 @@ static int send_create_inode(struct send_ctx *sctx, u64 ino) · 27 unmodified lines
26122657
26132658 if (ino != sctx->cur_ino) {
26142659 ret = get_inode_info(sctx->send_root, ino, NULL, &gen, &mode,
2615− NULL, NULL, &rdev);
2660+ NULL, NULL, &rdev, NULL);
26162661 if (ret < 0)
26172662 goto out;
26182663 } else {
@@ -2751,48 +2796,50 @@ struct recorded_ref { · 132 unmodified lines
27512796 u64 dir;
27522797 u64 dir_gen;
27532798 int name_len;
2799+ struct rb_node node;
2800+ struct rb_root *root;
27542801 };
27552802
2756−static void set_ref_path(struct recorded_ref *ref, struct fs_path *path)
2803+static struct recorded_ref *recorded_ref_alloc(void)
27572804 {
2758− ref->full_path = path;
2759− ref->name = (char *)kbasename(ref->full_path->start);
2760− ref->name_len = ref->full_path->end - ref->name;
2805+ struct recorded_ref *ref;
2806+
2807+ ref = kzalloc(sizeof(*ref), GFP_KERNEL);
2808+ if (!ref)
2809+ return NULL;
2810+ RB_CLEAR_NODE(&ref->node);
2811+ INIT_LIST_HEAD(&ref->list);
2812+ return ref;
27612813 }
27622814
2763−/*
2764− * We need to process new refs before deleted refs, but compare_tree gives us
2765− * everything mixed. So we first record all refs and later process them.
2766− * This function is a helper to record one ref.
2767− */
2768−static int __record_ref(struct list_head *head, u64 dir,
2769− u64 dir_gen, struct fs_path *path)
2815+static void recorded_ref_free(struct recorded_ref *ref)
27702816 {
2771− struct recorded_ref *ref;
2772−
2773− ref = kmalloc(sizeof(*ref), GFP_KERNEL);
27742817 if (!ref)
2775− return -ENOMEM;
2818+ return;
2819+ if (!RB_EMPTY_NODE(&ref->node))
2820+ rb_erase(&ref->node, ref->root);
2821+ list_del(&ref->list);
2822+ fs_path_free(ref->full_path);
2823+ kfree(ref);
2824+}
27762825
2777− ref->dir = dir;
2778− ref->dir_gen = dir_gen;
2779− set_ref_path(ref, path);
2780− list_add_tail(&ref->list, head);
2781− return 0;
2826+static void set_ref_path(struct recorded_ref *ref, struct fs_path *path)
2827+{
2828+ ref->full_path = path;
2829+ ref->name = (char *)kbasename(ref->full_path->start);
2830+ ref->name_len = ref->full_path->end - ref->name;
27822831 }
27832832
27842833 static int dup_ref(struct recorded_ref *ref, struct list_head *list)
27852834 {
27862835 struct recorded_ref *new;
27872836
2788− new = kmalloc(sizeof(*ref), GFP_KERNEL);
2837+ new = recorded_ref_alloc();
27892838 if (!new)
27902839 return -ENOMEM;
27912840
27922841 new->dir = ref->dir;
27932842 new->dir_gen = ref->dir_gen;
2794− new->full_path = NULL;
2795− INIT_LIST_HEAD(&new->list);
27962843 list_add_tail(&new->list, list);
27972844 return 0;
27982845 }
@@ -2803,9 +2850,7 @@ static void __free_recorded_refs(struct list_head *head) · 4 unmodified lines
28032850
28042851 while (!list_empty(head)) {
28052852 cur = list_entry(head->next, struct recorded_ref, list);
2806− fs_path_free(cur->full_path);
2807− list_del(&cur->list);
2808− kfree(cur);
2853+ recorded_ref_free(cur);
28092854 }
28102855 }
28112856
@@ -3315,7 +3360,7 @@ finish: · 503 unmodified lines
33153360 * The parent inode might have been deleted in the send snapshot
33163361 */
33173362 ret = get_inode_info(sctx->send_root, cur->dir, NULL,
3318− NULL, NULL, NULL, NULL, NULL);
3363+ NULL, NULL, NULL, NULL, NULL, NULL);
33193364 if (ret == -ENOENT) {
33203365 ret = 0;
33213366 continue;
@@ -3490,11 +3535,11 @@ static int wait_for_dest_dir_move(struct send_ctx *sctx, · 168 unmodified lines
34903535 }
34913536
34923537 ret = get_inode_info(sctx->parent_root, di_key.objectid, NULL,
3493− &left_gen, NULL, NULL, NULL, NULL);
3538+ &left_gen, NULL, NULL, NULL, NULL, NULL);
34943539 if (ret < 0)
34953540 goto out;
34963541 ret = get_inode_info(sctx->send_root, di_key.objectid, NULL,
3497− &right_gen, NULL, NULL, NULL, NULL);
3542+ &right_gen, NULL, NULL, NULL, NULL, NULL);
34983543 if (ret < 0) {
34993544 if (ret == -ENOENT)
35003545 ret = 0;
@@ -3625,7 +3670,7 @@ static int is_ancestor(struct btrfs_root *root, · 124 unmodified lines
36253670 }
36263671
36273672 ret = get_inode_info(root, parent, NULL, &parent_gen,
3628− NULL, NULL, NULL, NULL);
3673+ NULL, NULL, NULL, NULL, NULL);
36293674 if (ret < 0)
36303675 goto out;
36313676 ret = check_ino_in_path(root, ino1, ino1_gen,
@@ -3717,7 +3762,7 @@ static int wait_for_parent_move(struct send_ctx *sctx, · 85 unmodified lines
37173762
37183763 ret = get_inode_info(sctx->parent_root, ino, NULL,
37193764 &parent_ino_gen, NULL, NULL, NULL,
3720− NULL);
3765+ NULL, NULL);
37213766 if (ret < 0)
37223767 goto out;
37233768 if (ino_gen == parent_ino_gen) {
@@ -4311,185 +4356,169 @@ out: · 587 unmodified lines
43114356 return ret;
43124357 }
43134358
4314−static int record_ref(struct btrfs_root *root, u64 dir, struct fs_path *name,
4315− void *ctx, struct list_head *refs)
4359+static int rbtree_ref_comp(const void *k, const struct rb_node *node)
4360+{
4361+ const struct recorded_ref *data = k;
4362+ const struct recorded_ref *ref = rb_entry(node, struct recorded_ref, node);
4363+ int result;
4364+
4365+ if (data->dir > ref->dir)
4366+ return 1;
4367+ if (data->dir < ref->dir)
4368+ return -1;
4369+ if (data->dir_gen > ref->dir_gen)
4370+ return 1;
4371+ if (data->dir_gen < ref->dir_gen)
4372+ return -1;
4373+ if (data->name_len > ref->name_len)
4374+ return 1;
4375+ if (data->name_len < ref->name_len)
4376+ return -1;
4377+ result = strcmp(data->name, ref->name);
4378+ if (result > 0)
4379+ return 1;
4380+ if (result < 0)
4381+ return -1;
4382+ return 0;
4383+}
4384+
4385+static bool rbtree_ref_less(struct rb_node *node, const struct rb_node *parent)
4386+{
4387+ const struct recorded_ref *entry = rb_entry(node, struct recorded_ref, node);
4388+
4389+ return rbtree_ref_comp(entry, parent) < 0;
4390+}
4391+
4392+static int record_ref_in_tree(struct rb_root *root, struct list_head *refs,
4393+ struct fs_path *name, u64 dir, u64 dir_gen,
4394+ struct send_ctx *sctx)
43164395 {
43174396 int ret = 0;
4318− struct send_ctx *sctx = ctx;
4319− struct fs_path *p;
4320− u64 gen;
4397+ struct fs_path *path = NULL;
4398+ struct recorded_ref *ref = NULL;
43214399
4322− p = fs_path_alloc();
4323− if (!p)
4324− return -ENOMEM;
4400+ path = fs_path_alloc();
4401+ if (!path) {
4402+ ret = -ENOMEM;
4403+ goto out;
4404+ }
43254405
4326− ret = get_inode_info(root, dir, NULL, &gen, NULL, NULL,
4327− NULL, NULL);
4328− if (ret < 0)
4406+ ref = recorded_ref_alloc();
4407+ if (!ref) {
4408+ ret = -ENOMEM;
43294409 goto out;
4410+ }
43304411
4331− ret = get_cur_path(sctx, dir, gen, p);
4412+ ret = get_cur_path(sctx, dir, dir_gen, path);
43324413 if (ret < 0)
43334414 goto out;
4334− ret = fs_path_add_path(p, name);
4415+ ret = fs_path_add_path(path, name);
43354416 if (ret < 0)
43364417 goto out;
43374418
4338− ret = __record_ref(refs, dir, gen, p);
4339−
4419+ ref->dir = dir;
4420+ ref->dir_gen = dir_gen;
4421+ set_ref_path(ref, path);
4422+ list_add_tail(&ref->list, refs);
4423+ rb_add(&ref->node, root, rbtree_ref_less);
4424+ ref->root = root;
43404425 out:
4341− if (ret)
4342− fs_path_free(p);
4426+ if (ret) {
4427+ if (path && (!ref || !ref->full_path))
4428+ fs_path_free(path);
4429+ recorded_ref_free(ref);
4430+ }
43434431 return ret;
43444432 }
43454433
4346−static int __record_new_ref(int num, u64 dir, int index,
4347− struct fs_path *name,
4348− void *ctx)
4349−{
4350− struct send_ctx *sctx = ctx;
4351− return record_ref(sctx->send_root, dir, name, ctx, &sctx->new_refs);
4352−}
4353−
4354−
4355−static int __record_deleted_ref(int num, u64 dir, int index,
4356− struct fs_path *name,
4357− void *ctx)
4434+static int record_new_ref_if_needed(int num, u64 dir, int index,
4435+ struct fs_path *name, void *ctx)
43584436 {
4437+ int ret = 0;
43594438 struct send_ctx *sctx = ctx;
4360− return record_ref(sctx->parent_root, dir, name, ctx,
4361− &sctx->deleted_refs);
4362−}
4363−
4364−static int record_new_ref(struct send_ctx *sctx)
4365−{
4366− int ret;
4439+ struct rb_node *node = NULL;
4440+ struct recorded_ref data;
4441+ struct recorded_ref *ref;
4442+ u64 dir_gen;
43674443
4368− ret = iterate_inode_ref(sctx->send_root, sctx->left_path,
4369− sctx->cmp_key, 0, __record_new_ref, sctx);
4444+ ret = get_inode_info(sctx->send_root, dir, NULL, &dir_gen, NULL,
4445+ NULL, NULL, NULL, NULL);
43704446 if (ret < 0)
43714447 goto out;
4372− ret = 0;
43734448
4449+ data.dir = dir;
4450+ data.dir_gen = dir_gen;
4451+ set_ref_path(&data, name);
4452+ node = rb_find(&data, &sctx->rbtree_deleted_refs, rbtree_ref_comp);
4453+ if (node) {
4454+ ref = rb_entry(node, struct recorded_ref, node);
4455+ recorded_ref_free(ref);
4456+ } else {
4457+ ret = record_ref_in_tree(&sctx->rbtree_new_refs,
4458+ &sctx->new_refs, name, dir, dir_gen,
4459+ sctx);
4460+ }
43744461 out:
43754462 return ret;
43764463 }
43774464
4378−static int record_deleted_ref(struct send_ctx *sctx)
4465+static int record_deleted_ref_if_needed(int num, u64 dir, int index,
4466+ struct fs_path *name, void *ctx)
43794467 {
4380− int ret;
4468+ int ret = 0;
4469+ struct send_ctx *sctx = ctx;
4470+ struct rb_node *node = NULL;
4471+ struct recorded_ref data;
4472+ struct recorded_ref *ref;
4473+ u64 dir_gen;
43814474
4382− ret = iterate_inode_ref(sctx->parent_root, sctx->right_path,
4383− sctx->cmp_key, 0, __record_deleted_ref, sctx);
4475+ ret = get_inode_info(sctx->parent_root, dir, NULL, &dir_gen, NULL,
4476+ NULL, NULL, NULL, NULL);
43844477 if (ret < 0)
43854478 goto out;
4386− ret = 0;
43874479
4480+ data.dir = dir;
4481+ data.dir_gen = dir_gen;
4482+ set_ref_path(&data, name);
4483+ node = rb_find(&data, &sctx->rbtree_new_refs, rbtree_ref_comp);
4484+ if (node) {
4485+ ref = rb_entry(node, struct recorded_ref, node);
4486+ recorded_ref_free(ref);
4487+ } else {
4488+ ret = record_ref_in_tree(&sctx->rbtree_deleted_refs,
4489+ &sctx->deleted_refs, name, dir,
4490+ dir_gen, sctx);
4491+ }
43884492 out:
43894493 return ret;
43904494 }
43914495
4392−struct find_ref_ctx {
4393− u64 dir;
4394− u64 dir_gen;
4395− struct btrfs_root *root;
4396− struct fs_path *name;
4397− int found_idx;
4398−};
4399−
4400−static int __find_iref(int num, u64 dir, int index,
4401− struct fs_path *name,
4402− void *ctx_)
4403−{
4404− struct find_ref_ctx *ctx = ctx_;
4405− u64 dir_gen;
4406− int ret;
4407−
4408− if (dir == ctx->dir && fs_path_len(name) == fs_path_len(ctx->name) &&
4409− strncmp(name->start, ctx->name->start, fs_path_len(name)) == 0) {
4410− /*
4411− * To avoid doing extra lookups we'll only do this if everything
4412− * else matches.
4413− */
4414− ret = get_inode_info(ctx->root, dir, NULL, &dir_gen, NULL,
4415− NULL, NULL, NULL);
4416− if (ret)
4417− return ret;
4418− if (dir_gen != ctx->dir_gen)
4419− return 0;
4420− ctx->found_idx = num;
4421− return 1;
4422− }
4423− return 0;
4424−}
4425−
4426−static int find_iref(struct btrfs_root *root,
4427− struct btrfs_path *path,
4428− struct btrfs_key *key,
4429− u64 dir, u64 dir_gen, struct fs_path *name)
4496+static int record_new_ref(struct send_ctx *sctx)
44304497 {
44314498 int ret;
4432− struct find_ref_ctx ctx;
4433−
4434− ctx.dir = dir;
4435− ctx.name = name;
4436− ctx.dir_gen = dir_gen;
4437− ctx.found_idx = -1;
4438− ctx.root = root;
44394499
4440− ret = iterate_inode_ref(root, path, key, 0, __find_iref, &ctx);
4500+ ret = iterate_inode_ref(sctx->send_root, sctx->left_path,
4501+ sctx->cmp_key, 0, record_new_ref_if_needed, sctx);
44414502 if (ret < 0)
4442− return ret;
4443−
4444− if (ctx.found_idx == -1)
4445− return -ENOENT;
4446−
4447− return ctx.found_idx;
4448−}
4449−
4450−static int __record_changed_new_ref(int num, u64 dir, int index,
4451− struct fs_path *name,
4452− void *ctx)
4453−{
4454− u64 dir_gen;
4455− int ret;
4456− struct send_ctx *sctx = ctx;
4457−
4458− ret = get_inode_info(sctx->send_root, dir, NULL, &dir_gen, NULL,
4459− NULL, NULL, NULL);
4460− if (ret)
4461− return ret;
4462−
4463− ret = find_iref(sctx->parent_root, sctx->right_path,
4464− sctx->cmp_key, dir, dir_gen, name);
4465− if (ret == -ENOENT)
4466− ret = __record_new_ref(num, dir, index, name, sctx);
4467− else if (ret > 0)
4468− ret = 0;
4503+ goto out;
4504+ ret = 0;
44694505
4506+out:
44704507 return ret;
44714508 }
44724509
4473−static int __record_changed_deleted_ref(int num, u64 dir, int index,
4474− struct fs_path *name,
4475− void *ctx)
4510+static int record_deleted_ref(struct send_ctx *sctx)
44764511 {
4477− u64 dir_gen;
44784512 int ret;
4479− struct send_ctx *sctx = ctx;
4480−
4481− ret = get_inode_info(sctx->parent_root, dir, NULL, &dir_gen, NULL,
4482− NULL, NULL, NULL);
4483− if (ret)
4484− return ret;
44854513
4486− ret = find_iref(sctx->send_root, sctx->left_path, sctx->cmp_key,
4487− dir, dir_gen, name);
4488− if (ret == -ENOENT)
4489− ret = __record_deleted_ref(num, dir, index, name, sctx);
4490− else if (ret > 0)
4491− ret = 0;
4514+ ret = iterate_inode_ref(sctx->parent_root, sctx->right_path,
4515+ sctx->cmp_key, 0, record_deleted_ref_if_needed,
4516+ sctx);
4517+ if (ret < 0)
4518+ goto out;
4519+ ret = 0;
44924520
4521+out:
44934522 return ret;
44944523 }
44954524
@@ -4498,11 +4527,11 @@ static int record_changed_ref(struct send_ctx *sctx) · 2 unmodified lines
44984527 int ret = 0;
44994528
45004529 ret = iterate_inode_ref(sctx->send_root, sctx->left_path,
4501− sctx->cmp_key, 0, __record_changed_new_ref, sctx);
4530+ sctx->cmp_key, 0, record_new_ref_if_needed, sctx);
45024531 if (ret < 0)
45034532 goto out;
45044533 ret = iterate_inode_ref(sctx->parent_root, sctx->right_path,
4505− sctx->cmp_key, 0, __record_changed_deleted_ref, sctx);
4534+ sctx->cmp_key, 0, record_deleted_ref_if_needed, sctx);
45064535 if (ret < 0)
45074536 goto out;
45084537 ret = 0;
@@ -4533,10 +4562,10 @@ static int process_all_refs(struct send_ctx *sctx, · 24 unmodified lines
45334562
45344563 if (cmd == BTRFS_COMPARE_TREE_NEW) {
45354564 root = sctx->send_root;
4536− cb = __record_new_ref;
4565+ cb = record_new_ref_if_needed;
45374566 } else if (cmd == BTRFS_COMPARE_TREE_DELETED) {
45384567 root = sctx->parent_root;
4539− cb = __record_deleted_ref;
4568+ cb = record_deleted_ref_if_needed;
45404569 } else {
45414570 btrfs_err(sctx->send_root->fs_info,
45424571 "Wrong command %d in process_all_refs", cmd);
@@ -4864,14 +4893,28 @@ static inline u64 max_send_read_size(const struct send_ctx *sctx) · 321 unmodified lines
48644893
48654894 static int put_data_header(struct send_ctx *sctx, u32 len)
48664895 {
4867− struct btrfs_tlv_header *hdr;
4896+ if (WARN_ON_ONCE(sctx->put_data))
4897+ return -EINVAL;
4898+ sctx->put_data = true;
4899+ if (sctx->proto >= 2) {
4900+ /*
4901+ * Since v2, the data attribute header doesn't include a length,
4902+ * it is implicitly to the end of the command.
4903+ */
4904+ if (sctx->send_max_size - sctx->send_size < sizeof(__le16) + len)
4905+ return -EOVERFLOW;
4906+ put_unaligned_le16(BTRFS_SEND_A_DATA, sctx->send_buf + sctx->send_size);
4907+ sctx->send_size += sizeof(__le16);
4908+ } else {
4909+ struct btrfs_tlv_header *hdr;
48684910
4869− if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len)
4870− return -EOVERFLOW;
4871− hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size);
4872− put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type);
4873− put_unaligned_le16(len, &hdr->tlv_len);
4874− sctx->send_size += sizeof(*hdr);
4911+ if (sctx->send_max_size - sctx->send_size < sizeof(*hdr) + len)
4912+ return -EOVERFLOW;
4913+ hdr = (struct btrfs_tlv_header *)(sctx->send_buf + sctx->send_size);
4914+ put_unaligned_le16(BTRFS_SEND_A_DATA, &hdr->tlv_type);
4915+ put_unaligned_le16(len, &hdr->tlv_len);
4916+ sctx->send_size += sizeof(*hdr);
4917+ }
48754918 return 0;
48764919 }
48774920
@@ -5014,7 +5057,7 @@ static int send_clone(struct send_ctx *sctx, · 136 unmodified lines
50145057
50155058 if (clone_root->root == sctx->send_root) {
50165059 ret = get_inode_info(sctx->send_root, clone_root->ino, NULL,
5017− &gen, NULL, NULL, NULL, NULL);
5060+ &gen, NULL, NULL, NULL, NULL, NULL);
50185061 if (ret < 0)
50195062 goto out;
50205063 ret = get_cur_path(sctx, clone_root->ino, gen, p);
@@ -5141,17 +5184,214 @@ tlv_put_failure: · 120 unmodified lines
51415184 return ret;
51425185 }
51435186
5144−static int send_extent_data(struct send_ctx *sctx,
5145− const u64 offset,
5146− const u64 len)
5187+static int send_encoded_inline_extent(struct send_ctx *sctx,
5188+ struct btrfs_path *path, u64 offset,
5189+ u64 len)
5190+{
5191+ struct btrfs_root *root = sctx->send_root;
5192+ struct btrfs_fs_info *fs_info = root->fs_info;
5193+ struct inode *inode;
5194+ struct fs_path *fspath;
5195+ struct extent_buffer *leaf = path->nodes[0];
5196+ struct btrfs_key key;
5197+ struct btrfs_file_extent_item *ei;
5198+ u64 ram_bytes;
5199+ size_t inline_size;
5200+ int ret;
5201+
5202+ inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root);
5203+ if (IS_ERR(inode))
5204+ return PTR_ERR(inode);
5205+
5206+ fspath = fs_path_alloc();
5207+ if (!fspath) {
5208+ ret = -ENOMEM;
5209+ goto out;
5210+ }
5211+
5212+ ret = begin_cmd(sctx, BTRFS_SEND_C_ENCODED_WRITE);
5213+ if (ret < 0)
5214+ goto out;
5215+
5216+ ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, fspath);
5217+ if (ret < 0)
5218+ goto out;
5219+
5220+ btrfs_item_key_to_cpu(leaf, &key, path->slots[0]);
5221+ ei = btrfs_item_ptr(leaf, path->slots[0], struct btrfs_file_extent_item);
5222+ ram_bytes = btrfs_file_extent_ram_bytes(leaf, ei);
5223+ inline_size = btrfs_file_extent_inline_item_len(leaf, path->slots[0]);
5224+
5225+ TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, fspath);
5226+ TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset);
5227+ TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN,
5228+ min(key.offset + ram_bytes - offset, len));
5229+ TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN, ram_bytes);
5230+ TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET, offset - key.offset);
5231+ ret = btrfs_encoded_io_compression_from_extent(fs_info,
5232+ btrfs_file_extent_compression(leaf, ei));
5233+ if (ret < 0)
5234+ goto out;
5235+ TLV_PUT_U32(sctx, BTRFS_SEND_A_COMPRESSION, ret);
5236+
5237+ ret = put_data_header(sctx, inline_size);
5238+ if (ret < 0)
5239+ goto out;
5240+ read_extent_buffer(leaf, sctx->send_buf + sctx->send_size,
5241+ btrfs_file_extent_inline_start(ei), inline_size);
5242+ sctx->send_size += inline_size;
5243+
5244+ ret = send_cmd(sctx);
5245+
5246+tlv_put_failure:
5247+out:
5248+ fs_path_free(fspath);
5249+ iput(inode);
5250+ return ret;
5251+}
5252+
5253+static int send_encoded_extent(struct send_ctx *sctx, struct btrfs_path *path,
5254+ u64 offset, u64 len)
5255+{
5256+ struct btrfs_root *root = sctx->send_root;
5257+ struct btrfs_fs_info *fs_info = root->fs_info;
5258+ struct inode *inode;
5259+ struct fs_path *fspath;
5260+ struct extent_buffer *leaf = path->nodes[0];
5261+ struct btrfs_key key;
5262+ struct btrfs_file_extent_item *ei;
5263+ u64 disk_bytenr, disk_num_bytes;
5264+ u32 data_offset;
5265+ struct btrfs_cmd_header *hdr;
5266+ u32 crc;
5267+ int ret;
5268+
5269+ inode = btrfs_iget(fs_info->sb, sctx->cur_ino, root);
5270+ if (IS_ERR(inode))
5271+ return PTR_ERR(inode);
5272+
5273+ fspath = fs_path_alloc();
5274+ if (!fspath) {
5275+ ret = -ENOMEM;
5276+ goto out;
5277+ }
5278+
5279+ ret = begin_cmd(sctx, BTRFS_SEND_C_ENCODED_WRITE);
5280+ if (ret < 0)
5281+ goto out;
5282+
5283+ ret = get_cur_path(sctx, sctx->cur_ino, sctx->cur_inode_gen, fspath);
5284+ if (ret < 0)
5285+ goto out;
5286+
5287+ btrfs_item_key_to_cpu(leaf, &key, path->slots[0]);
5288+ ei = btrfs_item_ptr(leaf, path->slots[0], struct btrfs_file_extent_item);
5289+ disk_bytenr = btrfs_file_extent_disk_bytenr(leaf, ei);
5290+ disk_num_bytes = btrfs_file_extent_disk_num_bytes(leaf, ei);
5291+
5292+ TLV_PUT_PATH(sctx, BTRFS_SEND_A_PATH, fspath);
5293+ TLV_PUT_U64(sctx, BTRFS_SEND_A_FILE_OFFSET, offset);
5294+ TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_FILE_LEN,
5295+ min(key.offset + btrfs_file_extent_num_bytes(leaf, ei) - offset,
5296+ len));
5297+ TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_LEN,
5298+ btrfs_file_extent_ram_bytes(leaf, ei));
5299+ TLV_PUT_U64(sctx, BTRFS_SEND_A_UNENCODED_OFFSET,
5300+ offset - key.offset + btrfs_file_extent_offset(leaf, ei));
5301+ ret = btrfs_encoded_io_compression_from_extent(fs_info,
5302+ btrfs_file_extent_compression(leaf, ei));
5303+ if (ret < 0)
5304+ goto out;
5305+ TLV_PUT_U32(sctx, BTRFS_SEND_A_COMPRESSION, ret);
5306+ TLV_PUT_U32(sctx, BTRFS_SEND_A_ENCRYPTION, 0);
5307+
5308+ ret = put_data_header(sctx, disk_num_bytes);
5309+ if (ret < 0)
5310+ goto out;
5311+
5312+ /*
5313+ * We want to do I/O directly into the send buffer, so get the next page
5314+ * boundary in the send buffer. This means that there may be a gap
5315+ * between the beginning of the command and the file data.
5316+ */
5317+ data_offset = ALIGN(sctx->send_size, PAGE_SIZE);
5318+ if (data_offset > sctx->send_max_size ||
5319+ sctx->send_max_size - data_offset < disk_num_bytes) {
5320+ ret = -EOVERFLOW;
5321+ goto out;
5322+ }
5323+
5324+ /*
5325+ * Note that send_buf is a mapping of send_buf_pages, so this is really
5326+ * reading into send_buf.
5327+ */
5328+ ret = btrfs_encoded_read_regular_fill_pages(BTRFS_I(inode), offset,
5329+ disk_bytenr, disk_num_bytes,
5330+ sctx->send_buf_pages +
5331+ (data_offset >> PAGE_SHIFT));
5332+ if (ret)
5333+ goto out;
5334+
5335+ hdr = (struct btrfs_cmd_header *)sctx->send_buf;
5336+ hdr->len = cpu_to_le32(sctx->send_size + disk_num_bytes - sizeof(*hdr));
5337+ hdr->crc = 0;
5338+ crc = btrfs_crc32c(0, sctx->send_buf, sctx->send_size);
5339+ crc = btrfs_crc32c(crc, sctx->send_buf + data_offset, disk_num_bytes);
5340+ hdr->crc = cpu_to_le32(crc);
5341+
5342+ ret = write_buf(sctx->send_filp, sctx->send_buf, sctx->send_size,
5343+ &sctx->send_off);
5344+ if (!ret) {
5345+ ret = write_buf(sctx->send_filp, sctx->send_buf + data_offset,
5346+ disk_num_bytes, &sctx->send_off);
5347+ }
5348+ sctx->send_size = 0;
5349+ sctx->put_data = false;
5350+
5351+tlv_put_failure:
5352+out:
5353+ fs_path_free(fspath);
5354+ iput(inode);
5355+ return ret;
5356+}
5357+
5358+static int send_extent_data(struct send_ctx *sctx, struct btrfs_path *path,
5359+ const u64 offset, const u64 len)
51475360 {
51485361 const u64 end = offset + len;
5362+ struct extent_buffer *leaf = path->nodes[0];
5363+ struct btrfs_file_extent_item *ei;
51495364 u64 read_size = max_send_read_size(sctx);
51505365 u64 sent = 0;
51515366
51525367 if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA)
51535368 return send_update_extent(sctx, offset, len);
51545369
5370+ ei = btrfs_item_ptr(leaf, path->slots[0],
5371+ struct btrfs_file_extent_item);
5372+ if ((sctx->flags & BTRFS_SEND_FLAG_COMPRESSED) &&
5373+ btrfs_file_extent_compression(leaf, ei) != BTRFS_COMPRESS_NONE) {
5374+ bool is_inline = (btrfs_file_extent_type(leaf, ei) ==
5375+ BTRFS_FILE_EXTENT_INLINE);
5376+
5377+ /*
5378+ * Send the compressed extent unless the compressed data is
5379+ * larger than the decompressed data. This can happen if we're
5380+ * not sending the entire extent, either because it has been
5381+ * partially overwritten/truncated or because this is a part of
5382+ * the extent that we couldn't clone in clone_range().
5383+ */
5384+ if (is_inline &&
5385+ btrfs_file_extent_inline_item_len(leaf,
5386+ path->slots[0]) <= len) {
5387+ return send_encoded_inline_extent(sctx, path, offset,
5388+ len);
5389+ } else if (!is_inline &&
5390+ btrfs_file_extent_disk_num_bytes(leaf, ei) <= len) {
5391+ return send_encoded_extent(sctx, path, offset, len);
5392+ }
5393+ }
5394+
51555395 if (sctx->cur_inode == NULL) {
51565396 struct btrfs_root *root = sctx->send_root;
51575397
@@ -5289,12 +5529,9 @@ out: · 131 unmodified lines
52895529 return ret;
52905530 }
52915531
5292−static int clone_range(struct send_ctx *sctx,
5293− struct clone_root *clone_root,
5294− const u64 disk_byte,
5295− u64 data_offset,
5296− u64 offset,
5297− u64 len)
5532+static int clone_range(struct send_ctx *sctx, struct btrfs_path *dst_path,
5533+ struct clone_root *clone_root, const u64 disk_byte,
5534+ u64 data_offset, u64 offset, u64 len)
52985535 {
52995536 struct btrfs_path *path;
53005537 struct btrfs_key key;
@@ -5318,7 +5555,7 @@ static int clone_range(struct send_ctx *sctx, · 17 unmodified lines
53185555 */
53195556 if (clone_root->offset == 0 &&
53205557 len == sctx->send_root->fs_info->sectorsize)
5321− return send_extent_data(sctx, offset, len);
5558+ return send_extent_data(sctx, dst_path, offset, len);
53225559
53235560 path = alloc_path_for_send();
53245561 if (!path)
@@ -5329,7 +5566,8 @@ static int clone_range(struct send_ctx *sctx, · 4 unmodified lines
53295566 * accept clones from these extents.
53305567 */
53315568 ret = __get_inode_info(clone_root->root, path, clone_root->ino,
5332− &clone_src_i_size, NULL, NULL, NULL, NULL, NULL);
5569+ &clone_src_i_size, NULL, NULL, NULL, NULL, NULL,
5570+ NULL);
53335571 btrfs_release_path(path);
53345572 if (ret < 0)
53355573 goto out;
@@ -5415,7 +5653,8 @@ static int clone_range(struct send_ctx *sctx, · 79 unmodified lines
54155653
54165654 if (hole_len > len)
54175655 hole_len = len;
5418− ret = send_extent_data(sctx, offset, hole_len);
5656+ ret = send_extent_data(sctx, dst_path, offset,
5657+ hole_len);
54195658 if (ret < 0)
54205659 goto out;
54215660
@@ -5488,14 +5727,16 @@ static int clone_range(struct send_ctx *sctx, · 66 unmodified lines
54885727 if (ret < 0)
54895728 goto out;
54905729 }
5491− ret = send_extent_data(sctx, offset + slen,
5730+ ret = send_extent_data(sctx, dst_path,
5731+ offset + slen,
54925732 clone_len - slen);
54935733 } else {
54945734 ret = send_clone(sctx, offset, clone_len,
54955735 clone_root);
54965736 }
54975737 } else {
5498− ret = send_extent_data(sctx, offset, clone_len);
5738+ ret = send_extent_data(sctx, dst_path, offset,
5739+ clone_len);
54995740 }
55005741
55015742 if (ret < 0)
@@ -5527,7 +5768,7 @@ next: · 25 unmodified lines
55275768 }
55285769
55295770 if (len > 0)
5530− ret = send_extent_data(sctx, offset, len);
5771+ ret = send_extent_data(sctx, dst_path, offset, len);
55315772 else
55325773 ret = 0;
55335774 out:
@@ -5558,10 +5799,10 @@ static int send_write_or_clone(struct send_ctx *sctx, · 24 unmodified lines
55585799 struct btrfs_file_extent_item);
55595800 disk_byte = btrfs_file_extent_disk_bytenr(path->nodes[0], ei);
55605801 data_offset = btrfs_file_extent_offset(path->nodes[0], ei);
5561− ret = clone_range(sctx, clone_root, disk_byte, data_offset,
5562− offset, end - offset);
5802+ ret = clone_range(sctx, path, clone_root, disk_byte,
5803+ data_offset, offset, end - offset);
55635804 } else {
5564− ret = send_extent_data(sctx, offset, end - offset);
5805+ ret = send_extent_data(sctx, path, offset, end - offset);
55655806 }
55665807 sctx->cur_inode_next_write_offset = end;
55675808 return ret;
@@ -6021,11 +6262,14 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end) · 453 unmodified lines
60216262 u64 left_mode;
60226263 u64 left_uid;
60236264 u64 left_gid;
6265+ u64 left_fileattr;
60246266 u64 right_mode;
60256267 u64 right_uid;
60266268 u64 right_gid;
6269+ u64 right_fileattr;
60276270 int need_chmod = 0;
60286271 int need_chown = 0;
6272+ bool need_fileattr = false;
60296273 int need_truncate = 1;
60306274 int pending_move = 0;
60316275 int refs_processed = 0;
@@ -6059,7 +6303,7 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end) · 27 unmodified lines
60596303 goto out;
60606304
60616305 ret = get_inode_info(sctx->send_root, sctx->cur_ino, NULL, NULL,
6062− &left_mode, &left_uid, &left_gid, NULL);
6306+ &left_mode, &left_uid, &left_gid, NULL, &left_fileattr);
60636307 if (ret < 0)
60646308 goto out;
60656309
@@ -6074,7 +6318,7 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end) · 8 unmodified lines
60746318
60756319 ret = get_inode_info(sctx->parent_root, sctx->cur_ino,
60766320 &old_size, NULL, &right_mode, &right_uid,
6077− &right_gid, NULL);
6321+ &right_gid, NULL, &right_fileattr);
60786322 if (ret < 0)
60796323 goto out;
60806324
@@ -6082,6 +6326,8 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end) · 1 unmodified line
60826326 need_chown = 1;
60836327 if (!S_ISLNK(sctx->cur_inode_mode) && left_mode != right_mode)
60846328 need_chmod = 1;
6329+ if (!S_ISLNK(sctx->cur_inode_mode) && left_fileattr != right_fileattr)
6330+ need_fileattr = true;
60856331 if ((old_size == sctx->cur_inode_size) ||
60866332 (sctx->cur_inode_size > old_size &&
60876333 sctx->cur_inode_next_write_offset == sctx->cur_inode_size))
@@ -6125,6 +6371,12 @@ static int finish_inode_if_needed(struct send_ctx *sctx, int at_end) · 37 unmodified lines
61256371 if (ret < 0)
61266372 goto out;
61276373 }
6374+ if (need_fileattr) {
6375+ ret = send_fileattr(sctx, sctx->cur_ino, sctx->cur_inode_gen,
6376+ left_fileattr);
6377+ if (ret < 0)
6378+ goto out;
6379+ }
61286380
61296381 ret = send_capabilities(sctx);
61306382 if (ret < 0)
@@ -6165,8 +6417,13 @@ static int record_parent_ref(int num, u64 dir, int index, struct fs_path *name, · 34 unmodified lines
61656417 {
61666418 struct parent_paths_ctx *ppctx = ctx;
61676419
6168− return record_ref(ppctx->sctx->parent_root, dir, name, ppctx->sctx,
6169− ppctx->refs);
6420+ /*
6421+ * Pass 0 as the generation for the directory, we don't care about it
6422+ * here as we have no new references to add, we just want to delete all
6423+ * references for an inode.
6424+ */
6425+ return record_ref_in_tree(&ppctx->sctx->rbtree_deleted_refs, ppctx->refs,
6426+ name, dir, 0, ppctx->sctx);
61706427 }
61716428
61726429 /*
@@ -6220,9 +6477,7 @@ static int btrfs_unlink_all_paths(struct send_ctx *sctx) · 47 unmodified lines
62206477 ret = send_unlink(sctx, ref->full_path);
62216478 if (ret < 0)
62226479 goto out;
6223− fs_path_free(ref->full_path);
6224− list_del(&ref->list);
6225− kfree(ref);
6480+ recorded_ref_free(ref);
62266481 }
62276482 ret = 0;
62286483 out:
@@ -6269,7 +6524,7 @@ static int changed_inode(struct send_ctx *sctx, · 40 unmodified lines
62696524 close_current_inode(sctx);
62706525
62716526 sctx->cur_ino = key->objectid;
6272− sctx->cur_inode_new_gen = 0;
6527+ sctx->cur_inode_new_gen = false;
62736528 sctx->cur_inode_last_extent = (u64)-1;
62746529 sctx->cur_inode_next_write_offset = 0;
62756530 sctx->ignore_cur_inode = false;
@@ -6310,7 +6565,7 @@ static int changed_inode(struct send_ctx *sctx, · 34 unmodified lines
63106565 */
63116566 if (left_gen != right_gen &&
63126567 sctx->cur_ino != BTRFS_FIRST_FREE_OBJECTID)
6313− sctx->cur_inode_new_gen = 1;
6568+ sctx->cur_inode_new_gen = true;
63146569 }
63156570
63166571 /*
@@ -6342,8 +6597,8 @@ static int changed_inode(struct send_ctx *sctx, · 25 unmodified lines
63426597
63436598 if (result == BTRFS_COMPARE_TREE_NEW) {
63446599 sctx->cur_inode_gen = left_gen;
6345− sctx->cur_inode_new = 1;
6346− sctx->cur_inode_deleted = 0;
6600+ sctx->cur_inode_new = true;
6601+ sctx->cur_inode_deleted = false;
63476602 sctx->cur_inode_size = btrfs_inode_size(
63486603 sctx->left_path->nodes[0], left_ii);
63496604 sctx->cur_inode_mode = btrfs_inode_mode(
@@ -6354,8 +6609,8 @@ static int changed_inode(struct send_ctx *sctx, · 4 unmodified lines
63546609 ret = send_create_inode_if_needed(sctx);
63556610 } else if (result == BTRFS_COMPARE_TREE_DELETED) {
63566611 sctx->cur_inode_gen = right_gen;
6357− sctx->cur_inode_new = 0;
6358− sctx->cur_inode_deleted = 1;
6612+ sctx->cur_inode_new = false;
6613+ sctx->cur_inode_deleted = true;
63596614 sctx->cur_inode_size = btrfs_inode_size(
63606615 sctx->right_path->nodes[0], right_ii);
63616616 sctx->cur_inode_mode = btrfs_inode_mode(
@@ -6373,8 +6628,8 @@ static int changed_inode(struct send_ctx *sctx, · 11 unmodified lines
63736628 * First, process the inode as if it was deleted.
63746629 */
63756630 sctx->cur_inode_gen = right_gen;
6376− sctx->cur_inode_new = 0;
6377− sctx->cur_inode_deleted = 1;
6631+ sctx->cur_inode_new = false;
6632+ sctx->cur_inode_deleted = true;
63786633 sctx->cur_inode_size = btrfs_inode_size(
63796634 sctx->right_path->nodes[0], right_ii);
63806635 sctx->cur_inode_mode = btrfs_inode_mode(
@@ -6388,8 +6643,8 @@ static int changed_inode(struct send_ctx *sctx, · 7 unmodified lines
63886643 * Now process the inode as if it was new.
63896644 */
63906645 sctx->cur_inode_gen = left_gen;
6391− sctx->cur_inode_new = 1;
6392− sctx->cur_inode_deleted = 0;
6646+ sctx->cur_inode_new = true;
6647+ sctx->cur_inode_deleted = false;
63936648 sctx->cur_inode_size = btrfs_inode_size(
63946649 sctx->left_path->nodes[0], left_ii);
63956650 sctx->cur_inode_mode = btrfs_inode_mode(
@@ -6421,9 +6676,9 @@ static int changed_inode(struct send_ctx *sctx, · 25 unmodified lines
64216676 goto out;
64226677 } else {
64236678 sctx->cur_inode_gen = left_gen;
6424− sctx->cur_inode_new = 0;
6425− sctx->cur_inode_new_gen = 0;
6426− sctx->cur_inode_deleted = 0;
6679+ sctx->cur_inode_new = false;
6680+ sctx->cur_inode_new_gen = false;
6681+ sctx->cur_inode_deleted = false;
64276682 sctx->cur_inode_size = btrfs_inode_size(
64286683 sctx->left_path->nodes[0], left_ii);
64296684 sctx->cur_inode_mode = btrfs_inode_mode(
@@ -6536,12 +6791,12 @@ static int dir_changed(struct send_ctx *sctx, u64 dir) · 106 unmodified lines
65366791 int ret;
65376792
65386793 ret = get_inode_info(sctx->send_root, dir, NULL, &new_gen, NULL, NULL,
6539− NULL, NULL);
6794+ NULL, NULL, NULL);
65406795 if (ret)
65416796 return ret;
65426797
65436798 ret = get_inode_info(sctx->parent_root, dir, NULL, &orig_gen, NULL,
6544− NULL, NULL, NULL);
6799+ NULL, NULL, NULL, NULL);
65456800 if (ret)
65466801 return ret;
65476802
@@ -7537,6 +7792,10 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) · 989 unmodified lines
75377792 } else {
75387793 sctx->proto = 1;
75397794 }
7795+ if ((arg->flags & BTRFS_SEND_FLAG_COMPRESSED) && sctx->proto < 2) {
7796+ ret = -EINVAL;
7797+ goto out;
7798+ }
75407799
75417800 sctx->send_filp = fget(arg->send_fd);
75427801 if (!sctx->send_filp) {
@@ -7556,8 +7815,31 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) · 13 unmodified lines
75567815
75577816 sctx->clone_roots_cnt = arg->clone_sources_count;
75587817
7559− sctx->send_max_size = BTRFS_SEND_BUF_SIZE;
7560− sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL);
7818+ if (sctx->proto >= 2) {
7819+ u32 send_buf_num_pages;
7820+
7821+ sctx->send_max_size = ALIGN(SZ_16K + BTRFS_MAX_COMPRESSED, PAGE_SIZE);
7822+ sctx->send_buf = vmalloc(sctx->send_max_size);
7823+ if (!sctx->send_buf) {
7824+ ret = -ENOMEM;
7825+ goto out;
7826+ }
7827+ send_buf_num_pages = sctx->send_max_size >> PAGE_SHIFT;
7828+ sctx->send_buf_pages = kcalloc(send_buf_num_pages,
7829+ sizeof(*sctx->send_buf_pages),
7830+ GFP_KERNEL);
7831+ if (!sctx->send_buf_pages) {
7832+ ret = -ENOMEM;
7833+ goto out;
7834+ }
7835+ for (i = 0; i < send_buf_num_pages; i++) {
7836+ sctx->send_buf_pages[i] =
7837+ vmalloc_to_page(sctx->send_buf + (i << PAGE_SHIFT));
7838+ }
7839+ } else {
7840+ sctx->send_max_size = BTRFS_SEND_BUF_SIZE_V1;
7841+ sctx->send_buf = kvmalloc(sctx->send_max_size, GFP_KERNEL);
7842+ }
75617843 if (!sctx->send_buf) {
75627844 ret = -ENOMEM;
75637845 goto out;
@@ -7566,6 +7848,8 @@ long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg) · 2 unmodified lines
75667848 sctx->pending_dir_moves = RB_ROOT;
75677849 sctx->waiting_dir_moves = RB_ROOT;
75687850 sctx->orphan_dirs = RB_ROOT;
7851+ sctx->rbtree_new_refs = RB_ROOT;
7852+ sctx->rbtree_deleted_refs = RB_ROOT;
75697853
75707854 sctx->clone_roots = kvcalloc(sizeof(*sctx->clone_roots),
75717855 arg->clone_sources_count + 1,
@@ -7750,6 +8034,7 @@ out: · 178 unmodified lines
77508034 fput(sctx->send_filp);
77518035
77528036 kvfree(sctx->clone_roots);
8037+ kfree(sctx->send_buf_pages);
77538038 kvfree(sctx->send_buf);
77548039
77558040 name_cache_free(sctx);
fs/btrfs/send.h+103−66
@@ -7,12 +7,19 @@ · 6 unmodified lines
77 #ifndef BTRFS_SEND_H
88 #define BTRFS_SEND_H
99
10−#include "ctree.h"
10+#include <linux/types.h>
1111
1212 #define BTRFS_SEND_STREAM_MAGIC "btrfs-stream"
13−#define BTRFS_SEND_STREAM_VERSION 1
13+#define BTRFS_SEND_STREAM_VERSION 2
1414
15−#define BTRFS_SEND_BUF_SIZE SZ_64K
15+/*
16+ * In send stream v1, no command is larger than 64K. In send stream v2, no limit
17+ * should be assumed.
18+ */
19+#define BTRFS_SEND_BUF_SIZE_V1 SZ_64K
20+
21+struct inode;
22+struct btrfs_ioctl_send_args;
1623
1724 enum btrfs_tlv_type {
1825 BTRFS_TLV_U8,
@@ -46,87 +53,117 @@ struct btrfs_tlv_header { · 27 unmodified lines
4653
4754 /* commands */
4855 enum btrfs_send_cmd {
49− BTRFS_SEND_C_UNSPEC,
56+ BTRFS_SEND_C_UNSPEC = 0,
5057
5158 /* Version 1 */
52− BTRFS_SEND_C_SUBVOL,
53− BTRFS_SEND_C_SNAPSHOT,
59+ BTRFS_SEND_C_SUBVOL = 1,
60+ BTRFS_SEND_C_SNAPSHOT = 2,
5461
55− BTRFS_SEND_C_MKFILE,
56− BTRFS_SEND_C_MKDIR,
57− BTRFS_SEND_C_MKNOD,
58− BTRFS_SEND_C_MKFIFO,
59− BTRFS_SEND_C_MKSOCK,
60− BTRFS_SEND_C_SYMLINK,
62+ BTRFS_SEND_C_MKFILE = 3,
63+ BTRFS_SEND_C_MKDIR = 4,
64+ BTRFS_SEND_C_MKNOD = 5,
65+ BTRFS_SEND_C_MKFIFO = 6,
66+ BTRFS_SEND_C_MKSOCK = 7,
67+ BTRFS_SEND_C_SYMLINK = 8,
6168
62− BTRFS_SEND_C_RENAME,
63− BTRFS_SEND_C_LINK,
64− BTRFS_SEND_C_UNLINK,
65− BTRFS_SEND_C_RMDIR,
69+ BTRFS_SEND_C_RENAME = 9,
70+ BTRFS_SEND_C_LINK = 10,
71+ BTRFS_SEND_C_UNLINK = 11,
72+ BTRFS_SEND_C_RMDIR = 12,
6673
67− BTRFS_SEND_C_SET_XATTR,
68− BTRFS_SEND_C_REMOVE_XATTR,
74+ BTRFS_SEND_C_SET_XATTR = 13,
75+ BTRFS_SEND_C_REMOVE_XATTR = 14,
6976
70− BTRFS_SEND_C_WRITE,
71− BTRFS_SEND_C_CLONE,
77+ BTRFS_SEND_C_WRITE = 15,
78+ BTRFS_SEND_C_CLONE = 16,
7279
73− BTRFS_SEND_C_TRUNCATE,
74− BTRFS_SEND_C_CHMOD,
75− BTRFS_SEND_C_CHOWN,
76− BTRFS_SEND_C_UTIMES,
80+ BTRFS_SEND_C_TRUNCATE = 17,
81+ BTRFS_SEND_C_CHMOD = 18,
82+ BTRFS_SEND_C_CHOWN = 19,
83+ BTRFS_SEND_C_UTIMES = 20,
7784
78− BTRFS_SEND_C_END,
79− BTRFS_SEND_C_UPDATE_EXTENT,
80− __BTRFS_SEND_C_MAX_V1,
85+ BTRFS_SEND_C_END = 21,
86+ BTRFS_SEND_C_UPDATE_EXTENT = 22,
87+ BTRFS_SEND_C_MAX_V1 = 22,
8188
8289 /* Version 2 */
83− __BTRFS_SEND_C_MAX_V2,
90+ BTRFS_SEND_C_FALLOCATE = 23,
91+ BTRFS_SEND_C_FILEATTR = 24,
92+ BTRFS_SEND_C_ENCODED_WRITE = 25,
93+ BTRFS_SEND_C_MAX_V2 = 25,
8494
8595 /* End */
86− __BTRFS_SEND_C_MAX,
96+ BTRFS_SEND_C_MAX = 25,
8797 };
88−#define BTRFS_SEND_C_MAX (__BTRFS_SEND_C_MAX - 1)
8998
9099 /* attributes in send stream */
91100 enum {
92− BTRFS_SEND_A_UNSPEC,
93−
94− BTRFS_SEND_A_UUID,
95− BTRFS_SEND_A_CTRANSID,
96−
97− BTRFS_SEND_A_INO,
98− BTRFS_SEND_A_SIZE,
99− BTRFS_SEND_A_MODE,
100− BTRFS_SEND_A_UID,
101− BTRFS_SEND_A_GID,
102− BTRFS_SEND_A_RDEV,
103− BTRFS_SEND_A_CTIME,
104− BTRFS_SEND_A_MTIME,
105− BTRFS_SEND_A_ATIME,
106− BTRFS_SEND_A_OTIME,
107−
108− BTRFS_SEND_A_XATTR_NAME,
109− BTRFS_SEND_A_XATTR_DATA,
110−
111− BTRFS_SEND_A_PATH,
112− BTRFS_SEND_A_PATH_TO,
113− BTRFS_SEND_A_PATH_LINK,
114−
115− BTRFS_SEND_A_FILE_OFFSET,
116− BTRFS_SEND_A_DATA,
117−
118− BTRFS_SEND_A_CLONE_UUID,
119− BTRFS_SEND_A_CLONE_CTRANSID,
120− BTRFS_SEND_A_CLONE_PATH,
121− BTRFS_SEND_A_CLONE_OFFSET,
122− BTRFS_SEND_A_CLONE_LEN,
123−
124− __BTRFS_SEND_A_MAX,
101+ BTRFS_SEND_A_UNSPEC = 0,
102+
103+ /* Version 1 */
104+ BTRFS_SEND_A_UUID = 1,
105+ BTRFS_SEND_A_CTRANSID = 2,
106+
107+ BTRFS_SEND_A_INO = 3,
108+ BTRFS_SEND_A_SIZE = 4,
109+ BTRFS_SEND_A_MODE = 5,
110+ BTRFS_SEND_A_UID = 6,
111+ BTRFS_SEND_A_GID = 7,
112+ BTRFS_SEND_A_RDEV = 8,
113+ BTRFS_SEND_A_CTIME = 9,
114+ BTRFS_SEND_A_MTIME = 10,
115+ BTRFS_SEND_A_ATIME = 11,
116+ BTRFS_SEND_A_OTIME = 12,
117+
118+ BTRFS_SEND_A_XATTR_NAME = 13,
119+ BTRFS_SEND_A_XATTR_DATA = 14,
120+
121+ BTRFS_SEND_A_PATH = 15,
122+ BTRFS_SEND_A_PATH_TO = 16,
123+ BTRFS_SEND_A_PATH_LINK = 17,
124+
125+ BTRFS_SEND_A_FILE_OFFSET = 18,
126+ /*
127+ * As of send stream v2, this attribute is special: it must be the last
128+ * attribute in a command, its header contains only the type, and its
129+ * length is implicitly the remaining length of the command.
130+ */
131+ BTRFS_SEND_A_DATA = 19,
132+
133+ BTRFS_SEND_A_CLONE_UUID = 20,
134+ BTRFS_SEND_A_CLONE_CTRANSID = 21,
135+ BTRFS_SEND_A_CLONE_PATH = 22,
136+ BTRFS_SEND_A_CLONE_OFFSET = 23,
137+ BTRFS_SEND_A_CLONE_LEN = 24,
138+
139+ BTRFS_SEND_A_MAX_V1 = 24,
140+
141+ /* Version 2 */
142+ BTRFS_SEND_A_FALLOCATE_MODE = 25,
143+
144+ /*
145+ * File attributes from the FS_*_FL namespace (i_flags, xflags),
146+ * translated to BTRFS_INODE_* bits (BTRFS_INODE_FLAG_MASK) and stored
147+ * in btrfs_inode_item::flags (represented by btrfs_inode::flags and
148+ * btrfs_inode::ro_flags).
149+ */
150+ BTRFS_SEND_A_FILEATTR = 26,
151+
152+ BTRFS_SEND_A_UNENCODED_FILE_LEN = 27,
153+ BTRFS_SEND_A_UNENCODED_LEN = 28,
154+ BTRFS_SEND_A_UNENCODED_OFFSET = 29,
155+ /*
156+ * COMPRESSION and ENCRYPTION default to NONE (0) if omitted from
157+ * BTRFS_SEND_C_ENCODED_WRITE.
158+ */
159+ BTRFS_SEND_A_COMPRESSION = 30,
160+ BTRFS_SEND_A_ENCRYPTION = 31,
161+ BTRFS_SEND_A_MAX_V2 = 31,
162+
163+ /* End */
164+ BTRFS_SEND_A_MAX = 31,
125165 };
126−#define BTRFS_SEND_A_MAX (__BTRFS_SEND_A_MAX - 1)
127166
128−#ifdef __KERNEL__
129167 long btrfs_ioctl_send(struct inode *inode, struct btrfs_ioctl_send_args *arg);
130−#endif
131168
132169 #endif
fs/btrfs/space-info.c+99−11
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include "ordered-data.h"
1010 #include "transaction.h"
1111 #include "block-group.h"
12+#include "zoned.h"
1213
1314 /*
1415 * HOW DOES SPACE RESERVATION WORK
@@ -187,6 +188,37 @@ void btrfs_clear_space_info_full(struct btrfs_fs_info *info) · 172 unmodified lines
187188 */
188189 #define BTRFS_DEFAULT_ZONED_RECLAIM_THRESH (75)
189190
191+/*
192+ * Calculate chunk size depending on volume type (regular or zoned).
193+ */
194+static u64 calc_chunk_size(const struct btrfs_fs_info *fs_info, u64 flags)
195+{
196+ if (btrfs_is_zoned(fs_info))
197+ return fs_info->zone_size;
198+
199+ ASSERT(flags & BTRFS_BLOCK_GROUP_TYPE_MASK);
200+
201+ if (flags & BTRFS_BLOCK_GROUP_DATA)
202+ return SZ_1G;
203+ else if (flags & BTRFS_BLOCK_GROUP_SYSTEM)
204+ return SZ_32M;
205+
206+ /* Handle BTRFS_BLOCK_GROUP_METADATA */
207+ if (fs_info->fs_devices->total_rw_bytes > 50ULL * SZ_1G)
208+ return SZ_1G;
209+
210+ return SZ_256M;
211+}
212+
213+/*
214+ * Update default chunk size.
215+ */
216+void btrfs_update_space_info_chunk_size(struct btrfs_space_info *space_info,
217+ u64 chunk_size)
218+{
219+ WRITE_ONCE(space_info->chunk_size, chunk_size);
220+}
221+
190222 static int create_space_info(struct btrfs_fs_info *info, u64 flags)
191223 {
192224
@@ -208,6 +240,7 @@ static int create_space_info(struct btrfs_fs_info *info, u64 flags) · 15 unmodified lines
208240 INIT_LIST_HEAD(&space_info->tickets);
209241 INIT_LIST_HEAD(&space_info->priority_tickets);
210242 space_info->clamp = 1;
243+ btrfs_update_space_info_chunk_size(space_info, calc_chunk_size(info, flags));
211244
212245 if (btrfs_is_zoned(info))
213246 space_info->bg_reclaim_threshold = BTRFS_DEFAULT_ZONED_RECLAIM_THRESH;
@@ -263,7 +296,7 @@ out: · 49 unmodified lines
263296 void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags,
264297 u64 total_bytes, u64 bytes_used,
265298 u64 bytes_readonly, u64 bytes_zone_unusable,
266− struct btrfs_space_info **space_info)
299+ bool active, struct btrfs_space_info **space_info)
267300 {
268301 struct btrfs_space_info *found;
269302 int factor;
@@ -274,6 +307,8 @@ void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags, · 4 unmodified lines
274307 ASSERT(found);
275308 spin_lock(&found->lock);
276309 found->total_bytes += total_bytes;
310+ if (active)
311+ found->active_total_bytes += total_bytes;
277312 found->disk_total += total_bytes * factor;
278313 found->bytes_used += bytes_used;
279314 found->disk_used += bytes_used * factor;
@@ -337,6 +372,22 @@ static u64 calc_available_free_space(struct btrfs_fs_info *fs_info, · 57 unmodified lines
337372 return avail;
338373 }
339374
375+static inline u64 writable_total_bytes(struct btrfs_fs_info *fs_info,
376+ struct btrfs_space_info *space_info)
377+{
378+ /*
379+ * On regular filesystem, all total_bytes are always writable. On zoned
380+ * filesystem, there may be a limitation imposed by max_active_zones.
381+ * For metadata allocation, we cannot finish an existing active block
382+ * group to avoid a deadlock. Thus, we need to consider only the active
383+ * groups to be writable for metadata space.
384+ */
385+ if (!btrfs_is_zoned(fs_info) || (space_info->flags & BTRFS_BLOCK_GROUP_DATA))
386+ return space_info->total_bytes;
387+
388+ return space_info->active_total_bytes;
389+}
390+
340391 int btrfs_can_overcommit(struct btrfs_fs_info *fs_info,
341392 struct btrfs_space_info *space_info, u64 bytes,
342393 enum btrfs_reserve_flush_enum flush)
@@ -349,9 +400,12 @@ int btrfs_can_overcommit(struct btrfs_fs_info *fs_info, · 6 unmodified lines
349400 return 0;
350401
351402 used = btrfs_space_info_used(space_info, true);
352− avail = calc_available_free_space(fs_info, space_info, flush);
403+ if (btrfs_is_zoned(fs_info) && (space_info->flags & BTRFS_BLOCK_GROUP_METADATA))
404+ avail = 0;
405+ else
406+ avail = calc_available_free_space(fs_info, space_info, flush);
353407
354− if (used + bytes < space_info->total_bytes + avail)
408+ if (used + bytes < writable_total_bytes(fs_info, space_info) + avail)
355409 return 1;
356410 return 0;
357411 }
@@ -387,7 +441,7 @@ again: · 29 unmodified lines
387441 ticket = list_first_entry(head, struct reserve_ticket, list);
388442
389443 /* Check and see if our ticket can be satisfied now. */
390− if ((used + ticket->bytes <= space_info->total_bytes) ||
444+ if ((used + ticket->bytes <= writable_total_bytes(fs_info, space_info)) ||
391445 btrfs_can_overcommit(fs_info, space_info, ticket->bytes,
392446 flush)) {
393447 btrfs_space_info_update_bytes_may_use(fs_info,
@@ -671,6 +725,18 @@ static void flush_space(struct btrfs_fs_info *fs_info, · 277 unmodified lines
671725 break;
672726 case ALLOC_CHUNK:
673727 case ALLOC_CHUNK_FORCE:
728+ /*
729+ * For metadata space on zoned filesystem, reaching here means we
730+ * don't have enough space left in active_total_bytes. Try to
731+ * activate a block group first, because we may have inactive
732+ * block group already allocated.
733+ */
734+ ret = btrfs_zoned_activate_one_bg(fs_info, space_info, false);
735+ if (ret < 0)
736+ break;
737+ else if (ret == 1)
738+ break;
739+
674740 trans = btrfs_join_transaction(root);
675741 if (IS_ERR(trans)) {
676742 ret = PTR_ERR(trans);
@@ -681,6 +747,23 @@ static void flush_space(struct btrfs_fs_info *fs_info, · 4 unmodified lines
681747 (state == ALLOC_CHUNK) ? CHUNK_ALLOC_NO_FORCE :
682748 CHUNK_ALLOC_FORCE);
683749 btrfs_end_transaction(trans);
750+
751+ /*
752+ * For metadata space on zoned filesystem, allocating a new chunk
753+ * is not enough. We still need to activate the block * group.
754+ * Active the newly allocated block group by (maybe) finishing
755+ * a block group.
756+ */
757+ if (ret == 1) {
758+ ret = btrfs_zoned_activate_one_bg(fs_info, space_info, true);
759+ /*
760+ * Revert to the original ret regardless we could finish
761+ * one block group or not.
762+ */
763+ if (ret >= 0)
764+ ret = 1;
765+ }
766+
684767 if (ret > 0 || ret == -ENOSPC)
685768 ret = 0;
686769 break;
@@ -718,6 +801,7 @@ btrfs_calc_reclaim_metadata_size(struct btrfs_fs_info *fs_info, · 31 unmodified lines
718801 {
719802 u64 used;
720803 u64 avail;
804+ u64 total;
721805 u64 to_reclaim = space_info->reclaim_size;
722806
723807 lockdep_assert_held(&space_info->lock);
@@ -732,8 +816,9 @@ btrfs_calc_reclaim_metadata_size(struct btrfs_fs_info *fs_info, · 8 unmodified lines
732816 * space. If that's the case add in our overage so we make sure to put
733817 * appropriate pressure on the flushing state machine.
734818 */
735− if (space_info->total_bytes + avail < used)
736− to_reclaim += used - (space_info->total_bytes + avail);
819+ total = writable_total_bytes(fs_info, space_info);
820+ if (total + avail < used)
821+ to_reclaim += used - (total + avail);
737822
738823 return to_reclaim;
739824 }
@@ -743,9 +828,12 @@ static bool need_preemptive_reclaim(struct btrfs_fs_info *fs_info, · 3 unmodified lines
743828 {
744829 u64 global_rsv_size = fs_info->global_block_rsv.reserved;
745830 u64 ordered, delalloc;
746− u64 thresh = div_factor_fine(space_info->total_bytes, 90);
831+ u64 total = writable_total_bytes(fs_info, space_info);
832+ u64 thresh;
747833 u64 used;
748834
835+ thresh = div_factor_fine(total, 90);
836+
749837 lockdep_assert_held(&space_info->lock);
750838
751839 /* If we're just plain full then async reclaim just slows us down. */
@@ -807,8 +895,8 @@ static bool need_preemptive_reclaim(struct btrfs_fs_info *fs_info, · 55 unmodified lines
807895 BTRFS_RESERVE_FLUSH_ALL);
808896 used = space_info->bytes_used + space_info->bytes_reserved +
809897 space_info->bytes_readonly + global_rsv_size;
810− if (used < space_info->total_bytes)
811− thresh += space_info->total_bytes - used;
898+ if (used < total)
899+ thresh += total - used;
812900 thresh >>= space_info->clamp;
813901
814902 used = space_info->bytes_pinned;
@@ -1280,7 +1368,7 @@ static void priority_reclaim_metadata_space(struct btrfs_fs_info *fs_info, · 465 unmodified lines
12801368 to_reclaim = btrfs_calc_reclaim_metadata_size(fs_info, space_info);
12811369 /*
12821370 * This is the priority reclaim path, so to_reclaim could be >0 still
1283− * because we may have only satisified the priority tickets and still
1371+ * because we may have only satisfied the priority tickets and still
12841372 * left non priority tickets on the list. We would then have
12851373 * to_reclaim but ->bytes == 0.
12861374 */
@@ -1525,7 +1613,7 @@ static int __reserve_bytes(struct btrfs_fs_info *fs_info, · 238 unmodified lines
15251613 * can_overcommit() to ensure we can overcommit to continue.
15261614 */
15271615 if (!pending_tickets &&
1528− ((used + orig_bytes <= space_info->total_bytes) ||
1616+ ((used + orig_bytes <= writable_total_bytes(fs_info, space_info)) ||
15291617 btrfs_can_overcommit(fs_info, space_info, orig_bytes, flush))) {
15301618 btrfs_space_info_update_bytes_may_use(fs_info, space_info,
15311619 orig_bytes);
fs/btrfs/space-info.h+7−1
@@ -19,12 +19,16 @@ struct btrfs_space_info { · 18 unmodified lines
1919 u64 bytes_may_use; /* number of bytes that may be used for
2020 delalloc/allocations */
2121 u64 bytes_readonly; /* total bytes that are read only */
22+ /* Total bytes in the space, but only accounts active block groups. */
23+ u64 active_total_bytes;
2224 u64 bytes_zone_unusable; /* total bytes that are unusable until
2325 resetting the device zone */
2426
2527 u64 max_extent_size; /* This will hold the maximum extent size of
2628 the space info if we had an ENOSPC in the
2729 allocator. */
30+ /* Chunk size in bytes */
31+ u64 chunk_size;
2832
2933 /*
3034 * Once a block group drops below this threshold (percents) we'll
@@ -122,7 +126,9 @@ int btrfs_init_space_info(struct btrfs_fs_info *fs_info); · 91 unmodified lines
122126 void btrfs_update_space_info(struct btrfs_fs_info *info, u64 flags,
123127 u64 total_bytes, u64 bytes_used,
124128 u64 bytes_readonly, u64 bytes_zone_unusable,
125− struct btrfs_space_info **space_info);
129+ bool active, struct btrfs_space_info **space_info);
130+void btrfs_update_space_info_chunk_size(struct btrfs_space_info *space_info,
131+ u64 chunk_size);
126132 struct btrfs_space_info *btrfs_find_space_info(struct btrfs_fs_info *info,
127133 u64 flags);
128134 u64 __pure btrfs_space_info_used(struct btrfs_space_info *s_info,
fs/btrfs/struct-funcs.c+3−8
@@ -12,15 +12,10 @@ static bool check_setget_bounds(const struct extent_buffer *eb, · 11 unmodified lines
1212 {
1313 const unsigned long member_offset = (unsigned long)ptr + off;
1414
15− if (member_offset > eb->len) {
15+ if (unlikely(member_offset + size > eb->len)) {
1616 btrfs_warn(eb->fs_info,
17− "bad eb member start: ptr 0x%lx start %llu member offset %lu size %d",
18− (unsigned long)ptr, eb->start, member_offset, size);
19− return false;
20− }
21− if (member_offset + size > eb->len) {
22− btrfs_warn(eb->fs_info,
23− "bad eb member end: ptr 0x%lx start %llu member offset %lu size %d",
17+ "bad eb member %s: ptr 0x%lx start %llu member offset %lu size %d",
18+ (member_offset > eb->len ? "start" : "end"),
2419 (unsigned long)ptr, eb->start, member_offset, size);
2520 return false;
2621 }
fs/btrfs/subpage.c+2−2
@@ -123,7 +123,7 @@ int btrfs_attach_subpage(const struct btrfs_fs_info *fs_info, · 122 unmodified lines
123123 struct btrfs_subpage *subpage;
124124
125125 /*
126− * We have cases like a dummy extent buffer page, which is not mappped
126+ * We have cases like a dummy extent buffer page, which is not mapped
127127 * and doesn't need to be locked.
128128 */
129129 if (page->mapping)
@@ -731,7 +731,7 @@ void btrfs_page_assert_not_dirty(const struct btrfs_fs_info *fs_info, · 601 unmodified lines
731731 * It should not have any subpage::writers count.
732732 * Can be unlocked by unlock_page().
733733 * This is the most common locked page for __extent_writepage() called
734− * inside extent_write_cache_pages() or extent_write_full_page().
734+ * inside extent_write_cache_pages().
735735 * Rarer cases include the @locked_page from extent_write_locked_range().
736736 *
737737 * - Page locked by lock_delalloc_pages()
fs/btrfs/super.c+9−27
@@ -48,6 +48,7 @@ · 47 unmodified lines
4848 #include "block-group.h"
4949 #include "discard.h"
5050 #include "qgroup.h"
51+#include "raid56.h"
5152 #define CREATE_TRACE_POINTS
5253 #include <trace/events/btrfs.h>
5354
@@ -72,7 +73,7 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data); · 18 unmodified lines
7273 #define STATE_STRING_BUF_LEN (sizeof(STATE_STRING_PREFACE) + BTRFS_FS_STATE_COUNT)
7374
7475 /*
75− * Characters to print to indicate error conditions or uncommon filesystem sate.
76+ * Characters to print to indicate error conditions or uncommon filesystem state.
7677 * RO is not an error.
7778 */
7879 static const char fs_state_chars[] = {
@@ -1931,10 +1932,6 @@ static void btrfs_resize_thread_pool(struct btrfs_fs_info *fs_info, · 1852 unmodified lines
19311932 btrfs_workqueue_set_max(fs_info->hipri_workers, new_pool_size);
19321933 btrfs_workqueue_set_max(fs_info->delalloc_workers, new_pool_size);
19331934 btrfs_workqueue_set_max(fs_info->caching_workers, new_pool_size);
1934− btrfs_workqueue_set_max(fs_info->endio_workers, new_pool_size);
1935− btrfs_workqueue_set_max(fs_info->endio_meta_workers, new_pool_size);
1936− btrfs_workqueue_set_max(fs_info->endio_meta_write_workers,
1937− new_pool_size);
19381935 btrfs_workqueue_set_max(fs_info->endio_write_workers, new_pool_size);
19391936 btrfs_workqueue_set_max(fs_info->endio_freespace_worker, new_pool_size);
19401937 btrfs_workqueue_set_max(fs_info->delayed_workers, new_pool_size);
@@ -2246,12 +2243,8 @@ static inline int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info, · 305 unmodified lines
22462243
22472244 if (type & BTRFS_BLOCK_GROUP_RAID0)
22482245 num_stripes = nr_devices;
2249− else if (type & BTRFS_BLOCK_GROUP_RAID1)
2250− num_stripes = 2;
2251− else if (type & BTRFS_BLOCK_GROUP_RAID1C3)
2252− num_stripes = 3;
2253− else if (type & BTRFS_BLOCK_GROUP_RAID1C4)
2254− num_stripes = 4;
2246+ else if (type & BTRFS_BLOCK_GROUP_RAID1_MASK)
2247+ num_stripes = rattr->ncopies;
22552248 else if (type & BTRFS_BLOCK_GROUP_RAID10)
22562249 num_stripes = 4;
22572250
@@ -2275,17 +2268,13 @@ static inline int btrfs_calc_avail_data_space(struct btrfs_fs_info *fs_info, · 17 unmodified lines
22752268 avail_space = rounddown(avail_space, BTRFS_STRIPE_LEN);
22762269
22772270 /*
2278− * In order to avoid overwriting the superblock on the drive,
2279− * btrfs starts at an offset of at least 1MB when doing chunk
2280− * allocation.
2281− *
2282− * This ensures we have at least min_stripe_size free space
2283− * after excluding 1MB.
2271+ * Ensure we have at least min_stripe_size on top of the
2272+ * reserved space on the device.
22842273 */
2285− if (avail_space <= SZ_1M + min_stripe_size)
2274+ if (avail_space <= BTRFS_DEVICE_RANGE_RESERVED + min_stripe_size)
22862275 continue;
22872276
2288− avail_space -= SZ_1M;
2277+ avail_space -= BTRFS_DEVICE_RANGE_RESERVED;
22892278
22902279 devices_info[i].dev = device;
22912280 devices_info[i].max_avail = avail_space;
@@ -2703,13 +2692,9 @@ static int __init init_btrfs_fs(void) · 411 unmodified lines
27032692 if (err)
27042693 goto free_delayed_ref;
27052694
2706− err = btrfs_end_io_wq_init();
2707− if (err)
2708− goto free_prelim_ref;
2709−
27102695 err = btrfs_interface_init();
27112696 if (err)
2712− goto free_end_io_wq;
2697+ goto free_prelim_ref;
27132698
27142699 btrfs_print_mod_info();
27152700
@@ -2725,8 +2710,6 @@ static int __init init_btrfs_fs(void) · 9 unmodified lines
27252710
27262711 unregister_ioctl:
27272712 btrfs_interface_exit();
2728−free_end_io_wq:
2729− btrfs_end_io_wq_exit();
27302713 free_prelim_ref:
27312714 btrfs_prelim_ref_exit();
27322715 free_delayed_ref:
@@ -2764,7 +2747,6 @@ static void __exit exit_btrfs_fs(void) · 31 unmodified lines
27642747 extent_state_cache_exit();
27652748 extent_io_exit();
27662749 btrfs_interface_exit();
2767− btrfs_end_io_wq_exit();
27682750 unregister_filesystem(&btrfs_fs_type);
27692751 btrfs_exit_sysfs();
27702752 btrfs_cleanup_fs_uuids();
fs/btrfs/sysfs.c+177−9
@@ -21,6 +21,7 @@ · 20 unmodified lines
2121 #include "space-info.h"
2222 #include "block-group.h"
2323 #include "qgroup.h"
24+#include "misc.h"
2425
2526 /*
2627 * Structure name Path
@@ -61,6 +62,10 @@ struct raid_kobject { · 34 unmodified lines
6162 .store = _store, \
6263 }
6364
65+#define BTRFS_ATTR_W(_prefix, _name, _store) \
66+ static struct kobj_attribute btrfs_attr_##_prefix##_##_name = \
67+ __INIT_KOBJ_ATTR(_name, 0200, NULL, _store)
68+
6469 #define BTRFS_ATTR_RW(_prefix, _name, _show, _store) \
6570 static struct kobj_attribute btrfs_attr_##_prefix##_##_name = \
6671 __INIT_KOBJ_ATTR(_name, 0644, _show, _store)
@@ -92,6 +97,7 @@ static struct btrfs_feature_attr btrfs_attr_features_##_name = { \ · 25 unmodified lines
9297
9398 static inline struct btrfs_fs_info *to_fs_info(struct kobject *kobj);
9499 static inline struct btrfs_fs_devices *to_fs_devs(struct kobject *kobj);
100+static struct kobject *get_btrfs_kobj(struct kobject *kobj);
95101
96102 static struct btrfs_feature_attr *to_btrfs_feature_attr(struct kobj_attribute *a)
97103 {
@@ -270,12 +276,10 @@ static umode_t btrfs_feature_visible(struct kobject *kobj, · 172 unmodified lines
270276 return mode;
271277 }
272278
273−BTRFS_FEAT_ATTR_INCOMPAT(mixed_backref, MIXED_BACKREF);
274279 BTRFS_FEAT_ATTR_INCOMPAT(default_subvol, DEFAULT_SUBVOL);
275280 BTRFS_FEAT_ATTR_INCOMPAT(mixed_groups, MIXED_GROUPS);
276281 BTRFS_FEAT_ATTR_INCOMPAT(compress_lzo, COMPRESS_LZO);
277282 BTRFS_FEAT_ATTR_INCOMPAT(compress_zstd, COMPRESS_ZSTD);
278−BTRFS_FEAT_ATTR_INCOMPAT(big_metadata, BIG_METADATA);
279283 BTRFS_FEAT_ATTR_INCOMPAT(extended_iref, EXTENDED_IREF);
280284 BTRFS_FEAT_ATTR_INCOMPAT(raid56, RAID56);
281285 BTRFS_FEAT_ATTR_INCOMPAT(skinny_metadata, SKINNY_METADATA);
@@ -283,9 +287,10 @@ BTRFS_FEAT_ATTR_INCOMPAT(no_holes, NO_HOLES); · 1 unmodified line
283287 BTRFS_FEAT_ATTR_INCOMPAT(metadata_uuid, METADATA_UUID);
284288 BTRFS_FEAT_ATTR_COMPAT_RO(free_space_tree, FREE_SPACE_TREE);
285289 BTRFS_FEAT_ATTR_INCOMPAT(raid1c34, RAID1C34);
286−#ifdef CONFIG_BTRFS_DEBUG
287−/* Remove once support for zoned allocation is feature complete */
290+#ifdef CONFIG_BLK_DEV_ZONED
288291 BTRFS_FEAT_ATTR_INCOMPAT(zoned, ZONED);
292+#endif
293+#ifdef CONFIG_BTRFS_DEBUG
289294 /* Remove once support for extent tree v2 is feature complete */
290295 BTRFS_FEAT_ATTR_INCOMPAT(extent_tree_v2, EXTENT_TREE_V2);
291296 #endif
@@ -296,17 +301,15 @@ BTRFS_FEAT_ATTR_COMPAT_RO(verity, VERITY); · 4 unmodified lines
296301 /*
297302 * Features which depend on feature bits and may differ between each fs.
298303 *
299− * /sys/fs/btrfs/features - all available features implemeted by this version
304+ * /sys/fs/btrfs/features - all available features implemented by this version
300305 * /sys/fs/btrfs/UUID/features - features of the fs which are enabled or
301306 * can be changed on a mounted filesystem.
302307 */
303308 static struct attribute *btrfs_supported_feature_attrs[] = {
304− BTRFS_FEAT_ATTR_PTR(mixed_backref),
305309 BTRFS_FEAT_ATTR_PTR(default_subvol),
306310 BTRFS_FEAT_ATTR_PTR(mixed_groups),
307311 BTRFS_FEAT_ATTR_PTR(compress_lzo),
308312 BTRFS_FEAT_ATTR_PTR(compress_zstd),
309− BTRFS_FEAT_ATTR_PTR(big_metadata),
310313 BTRFS_FEAT_ATTR_PTR(extended_iref),
311314 BTRFS_FEAT_ATTR_PTR(raid56),
312315 BTRFS_FEAT_ATTR_PTR(skinny_metadata),
@@ -314,8 +317,10 @@ static struct attribute *btrfs_supported_feature_attrs[] = { · 1 unmodified line
314317 BTRFS_FEAT_ATTR_PTR(metadata_uuid),
315318 BTRFS_FEAT_ATTR_PTR(free_space_tree),
316319 BTRFS_FEAT_ATTR_PTR(raid1c34),
317−#ifdef CONFIG_BTRFS_DEBUG
320+#ifdef CONFIG_BLK_DEV_ZONED
318321 BTRFS_FEAT_ATTR_PTR(zoned),
322+#endif
323+#ifdef CONFIG_BTRFS_DEBUG
319324 BTRFS_FEAT_ATTR_PTR(extent_tree_v2),
320325 #endif
321326 #ifdef CONFIG_FS_VERITY
@@ -709,6 +714,112 @@ static ssize_t btrfs_space_info_show_##field(struct kobject *kobj, \ · 387 unmodified lines
709714 } \
710715 BTRFS_ATTR(space_info, field, btrfs_space_info_show_##field)
711716
717+static ssize_t btrfs_chunk_size_show(struct kobject *kobj,
718+ struct kobj_attribute *a, char *buf)
719+{
720+ struct btrfs_space_info *sinfo = to_space_info(kobj);
721+
722+ return sysfs_emit(buf, "%llu\n", READ_ONCE(sinfo->chunk_size));
723+}
724+
725+/*
726+ * Store new chunk size in space info. Can be called on a read-only filesystem.
727+ *
728+ * If the new chunk size value is larger than 10% of free space it is reduced
729+ * to match that limit. Alignment must be to 256M and the system chunk size
730+ * cannot be set.
731+ */
732+static ssize_t btrfs_chunk_size_store(struct kobject *kobj,
733+ struct kobj_attribute *a,
734+ const char *buf, size_t len)
735+{
736+ struct btrfs_space_info *space_info = to_space_info(kobj);
737+ struct btrfs_fs_info *fs_info = to_fs_info(get_btrfs_kobj(kobj));
738+ char *retptr;
739+ u64 val;
740+
741+ if (!capable(CAP_SYS_ADMIN))
742+ return -EPERM;
743+
744+ if (!fs_info->fs_devices)
745+ return -EINVAL;
746+
747+ if (btrfs_is_zoned(fs_info))
748+ return -EINVAL;
749+
750+ /* System block type must not be changed. */
751+ if (space_info->flags & BTRFS_BLOCK_GROUP_SYSTEM)
752+ return -EPERM;
753+
754+ val = memparse(buf, &retptr);
755+ /* There could be trailing '\n', also catch any typos after the value */
756+ retptr = skip_spaces(retptr);
757+ if (*retptr != 0 || val == 0)
758+ return -EINVAL;
759+
760+ val = min(val, BTRFS_MAX_DATA_CHUNK_SIZE);
761+
762+ /* Limit stripe size to 10% of available space. */
763+ val = min(div_factor(fs_info->fs_devices->total_rw_bytes, 1), val);
764+
765+ /* Must be multiple of 256M. */
766+ val &= ~((u64)SZ_256M - 1);
767+
768+ /* Must be at least 256M. */
769+ if (val < SZ_256M)
770+ return -EINVAL;
771+
772+ btrfs_update_space_info_chunk_size(space_info, val);
773+
774+ return len;
775+}
776+
777+#ifdef CONFIG_BTRFS_DEBUG
778+/*
779+ * Request chunk allocation with current chunk size.
780+ */
781+static ssize_t btrfs_force_chunk_alloc_store(struct kobject *kobj,
782+ struct kobj_attribute *a,
783+ const char *buf, size_t len)
784+{
785+ struct btrfs_space_info *space_info = to_space_info(kobj);
786+ struct btrfs_fs_info *fs_info = to_fs_info(get_btrfs_kobj(kobj));
787+ struct btrfs_trans_handle *trans;
788+ bool val;
789+ int ret;
790+
791+ if (!capable(CAP_SYS_ADMIN))
792+ return -EPERM;
793+
794+ if (sb_rdonly(fs_info->sb))
795+ return -EROFS;
796+
797+ ret = kstrtobool(buf, &val);
798+ if (ret)
799+ return ret;
800+
801+ if (!val)
802+ return -EINVAL;
803+
804+ /*
805+ * This is unsafe to be called from sysfs context and may cause
806+ * unexpected problems.
807+ */
808+ trans = btrfs_start_transaction(fs_info->tree_root, 0);
809+ if (IS_ERR(trans))
810+ return PTR_ERR(trans);
811+ ret = btrfs_force_chunk_alloc(trans, space_info->flags);
812+ btrfs_end_transaction(trans);
813+
814+ if (ret == 1)
815+ return len;
816+
817+ return -ENOSPC;
818+}
819+BTRFS_ATTR_W(space_info, force_chunk_alloc, btrfs_force_chunk_alloc_store);
820+
821+#endif
822+
712823 SPACE_INFO_ATTR(flags);
713824 SPACE_INFO_ATTR(total_bytes);
714825 SPACE_INFO_ATTR(bytes_used);
@@ -719,6 +830,7 @@ SPACE_INFO_ATTR(bytes_readonly); · 4 unmodified lines
719830 SPACE_INFO_ATTR(bytes_zone_unusable);
720831 SPACE_INFO_ATTR(disk_used);
721832 SPACE_INFO_ATTR(disk_total);
833+BTRFS_ATTR_RW(space_info, chunk_size, btrfs_chunk_size_show, btrfs_chunk_size_store);
722834
723835 static ssize_t btrfs_sinfo_bg_reclaim_threshold_show(struct kobject *kobj,
724836 struct kobj_attribute *a,
@@ -773,6 +885,10 @@ static struct attribute *space_info_attrs[] = { · 48 unmodified lines
773885 BTRFS_ATTR_PTR(space_info, disk_used),
774886 BTRFS_ATTR_PTR(space_info, disk_total),
775887 BTRFS_ATTR_PTR(space_info, bg_reclaim_threshold),
888+ BTRFS_ATTR_PTR(space_info, chunk_size),
889+#ifdef CONFIG_BTRFS_DEBUG
890+ BTRFS_ATTR_PTR(space_info, force_chunk_alloc),
891+#endif
776892 NULL,
777893 };
778894 ATTRIBUTE_GROUPS(space_info);
@@ -871,6 +987,48 @@ static ssize_t btrfs_sectorsize_show(struct kobject *kobj, · 92 unmodified lines
871987
872988 BTRFS_ATTR(, sectorsize, btrfs_sectorsize_show);
873989
990+static ssize_t btrfs_commit_stats_show(struct kobject *kobj,
991+ struct kobj_attribute *a, char *buf)
992+{
993+ struct btrfs_fs_info *fs_info = to_fs_info(kobj);
994+
995+ return sysfs_emit(buf,
996+ "commits %llu\n"
997+ "last_commit_ms %llu\n"
998+ "max_commit_ms %llu\n"
999+ "total_commit_ms %llu\n",
1000+ fs_info->commit_stats.commit_count,
1001+ div_u64(fs_info->commit_stats.last_commit_dur, NSEC_PER_MSEC),
1002+ div_u64(fs_info->commit_stats.max_commit_dur, NSEC_PER_MSEC),
1003+ div_u64(fs_info->commit_stats.total_commit_dur, NSEC_PER_MSEC));
1004+}
1005+
1006+static ssize_t btrfs_commit_stats_store(struct kobject *kobj,
1007+ struct kobj_attribute *a,
1008+ const char *buf, size_t len)
1009+{
1010+ struct btrfs_fs_info *fs_info = to_fs_info(kobj);
1011+ unsigned long val;
1012+ int ret;
1013+
1014+ if (!fs_info)
1015+ return -EPERM;
1016+
1017+ if (!capable(CAP_SYS_RESOURCE))
1018+ return -EPERM;
1019+
1020+ ret = kstrtoul(buf, 10, &val);
1021+ if (ret)
1022+ return ret;
1023+ if (val)
1024+ return -EINVAL;
1025+
1026+ WRITE_ONCE(fs_info->commit_stats.max_commit_dur, 0);
1027+
1028+ return len;
1029+}
1030+BTRFS_ATTR_RW(, commit_stats, btrfs_commit_stats_show, btrfs_commit_stats_store);
1031+
8741032 static ssize_t btrfs_clone_alignment_show(struct kobject *kobj,
8751033 struct kobj_attribute *a, char *buf)
8761034 {
@@ -1110,6 +1268,7 @@ static const struct attribute *btrfs_attrs[] = { · 233 unmodified lines
11101268 BTRFS_ATTR_PTR(, generation),
11111269 BTRFS_ATTR_PTR(, read_policy),
11121270 BTRFS_ATTR_PTR(, bg_reclaim_threshold),
1271+ BTRFS_ATTR_PTR(, commit_stats),
11131272 NULL,
11141273 };
11151274
@@ -1140,6 +1299,16 @@ static inline struct btrfs_fs_info *to_fs_info(struct kobject *kobj) · 24 unmodified lines
11401299 return to_fs_devs(kobj)->fs_info;
11411300 }
11421301
1302+static struct kobject *get_btrfs_kobj(struct kobject *kobj)
1303+{
1304+ while (kobj) {
1305+ if (kobj->ktype == &btrfs_ktype)
1306+ return kobj;
1307+ kobj = kobj->parent;
1308+ }
1309+ return NULL;
1310+}
1311+
11431312 #define NUM_FEATURE_BITS 64
11441313 #define BTRFS_FEATURE_NAME_MAX 13
11451314 static char btrfs_unknown_feature_names[FEAT_MAX][NUM_FEATURE_BITS][BTRFS_FEATURE_NAME_MAX];
@@ -2106,4 +2275,3 @@ void __cold btrfs_exit_sysfs(void) · 960 unmodified lines
21062275 #endif
21072276 kset_unregister(btrfs_kset);
21082277 }
2109−
fs/btrfs/tests/btrfs-tests.c+1−0
@@ -59,6 +59,7 @@ struct inode *btrfs_new_test_inode(void) · 58 unmodified lines
5959 return NULL;
6060
6161 inode->i_mode = S_IFREG;
62+ inode->i_ino = BTRFS_FIRST_FREE_OBJECTID;
6263 BTRFS_I(inode)->location.type = BTRFS_INODE_ITEM_KEY;
6364 BTRFS_I(inode)->location.objectid = BTRFS_FIRST_FREE_OBJECTID;
6465 BTRFS_I(inode)->location.offset = 0;
fs/btrfs/tests/extent-buffer-tests.c+2−1
@@ -47,7 +47,8 @@ static int test_btrfs_split_item(u32 sectorsize, u32 nodesize) · 46 unmodified lines
4747 goto out;
4848 }
4949
50− path->nodes[0] = eb = alloc_dummy_extent_buffer(fs_info, nodesize);
50+ eb = alloc_dummy_extent_buffer(fs_info, nodesize);
51+ path->nodes[0] = eb;
5152 if (!eb) {
5253 test_std_err(TEST_ALLOC_EXTENT_BUFFER);
5354 ret = -ENOMEM;
fs/btrfs/transaction.c+24−2
@@ -10,6 +10,7 @@ · 9 unmodified lines
1010 #include <linux/pagemap.h>
1111 #include <linux/blkdev.h>
1212 #include <linux/uuid.h>
13+#include <linux/timekeeping.h>
1314 #include "misc.h"
1415 #include "ctree.h"
1516 #include "disk-io.h"
@@ -1831,8 +1832,8 @@ static noinline int create_pending_snapshot(struct btrfs_trans_handle *trans, · 1815 unmodified lines
18311832
18321833 btrfs_i_size_write(BTRFS_I(parent_inode), parent_inode->i_size +
18331834 dentry->d_name.len * 2);
1834− parent_inode->i_mtime = parent_inode->i_ctime =
1835− current_time(parent_inode);
1835+ parent_inode->i_mtime = current_time(parent_inode);
1836+ parent_inode->i_ctime = parent_inode->i_mtime;
18361837 ret = btrfs_update_inode_fallback(trans, parent_root, BTRFS_I(parent_inode));
18371838 if (ret) {
18381839 btrfs_abort_transaction(trans, ret);
@@ -2098,12 +2099,23 @@ static void add_pending_snapshot(struct btrfs_trans_handle *trans) · 259 unmodified lines
20982099 list_add(&trans->pending_snapshot->list, &cur_trans->pending_snapshots);
20992100 }
21002101
2102+static void update_commit_stats(struct btrfs_fs_info *fs_info, ktime_t interval)
2103+{
2104+ fs_info->commit_stats.commit_count++;
2105+ fs_info->commit_stats.last_commit_dur = interval;
2106+ fs_info->commit_stats.max_commit_dur =
2107+ max_t(u64, fs_info->commit_stats.max_commit_dur, interval);
2108+ fs_info->commit_stats.total_commit_dur += interval;
2109+}
2110+
21012111 int btrfs_commit_transaction(struct btrfs_trans_handle *trans)
21022112 {
21032113 struct btrfs_fs_info *fs_info = trans->fs_info;
21042114 struct btrfs_transaction *cur_trans = trans->transaction;
21052115 struct btrfs_transaction *prev_trans = NULL;
21062116 int ret;
2117+ ktime_t start_time;
2118+ ktime_t interval;
21072119
21082120 ASSERT(refcount_read(&trans->use_count) == 1);
21092121
@@ -2228,6 +2240,12 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans) · 118 unmodified lines
22282240 }
22292241 }
22302242
2243+ /*
2244+ * Get the time spent on the work done by the commit thread and not
2245+ * the time spent waiting on a previous commit
2246+ */
2247+ start_time = ktime_get_ns();
2248+
22312249 extwriter_counter_dec(cur_trans, trans->type);
22322250
22332251 ret = btrfs_start_delalloc_flush(fs_info);
@@ -2469,6 +2487,8 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans) · 235 unmodified lines
24692487
24702488 trace_btrfs_transaction_commit(fs_info);
24712489
2490+ interval = ktime_get_ns() - start_time;
2491+
24722492 btrfs_scrub_continue(fs_info);
24732493
24742494 if (current->journal_info == trans)
@@ -2476,6 +2496,8 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans) · 1 unmodified line
24762496
24772497 kmem_cache_free(btrfs_trans_handle_cachep, trans);
24782498
2499+ update_commit_stats(fs_info, interval);
2500+
24792501 return ret;
24802502
24812503 unlock_reloc:
fs/btrfs/tree-log.c+18−11
@@ -171,7 +171,7 @@ again: · 170 unmodified lines
171171 int index = (root->log_transid + 1) % 2;
172172
173173 if (btrfs_need_log_full_commit(trans)) {
174− ret = -EAGAIN;
174+ ret = BTRFS_LOG_FORCE_COMMIT;
175175 goto out;
176176 }
177177
@@ -194,7 +194,7 @@ again: · 16 unmodified lines
194194 * writing.
195195 */
196196 if (zoned && !created) {
197− ret = -EAGAIN;
197+ ret = BTRFS_LOG_FORCE_COMMIT;
198198 goto out;
199199 }
200200
@@ -2287,7 +2287,7 @@ static noinline int check_item_in_log(struct btrfs_trans_handle *trans, · 2086 unmodified lines
22872287 struct btrfs_key location;
22882288
22892289 /*
2290− * Currenly we only log dir index keys. Even if we replay a log created
2290+ * Currently we only log dir index keys. Even if we replay a log created
22912291 * by an older kernel that logged both dir index and dir item keys, all
22922292 * we need to do is process the dir index keys, we (and our caller) can
22932293 * safely ignore dir item keys (key type BTRFS_DIR_ITEM_KEY).
@@ -3121,7 +3121,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans, · 827 unmodified lines
31213121
31223122 /* bail out if we need to do a full commit */
31233123 if (btrfs_need_log_full_commit(trans)) {
3124− ret = -EAGAIN;
3124+ ret = BTRFS_LOG_FORCE_COMMIT;
31253125 mutex_unlock(&root->log_mutex);
31263126 goto out;
31273127 }
@@ -3222,7 +3222,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans, · 94 unmodified lines
32223222 }
32233223 btrfs_wait_tree_log_extents(log, mark);
32243224 mutex_unlock(&log_root_tree->log_mutex);
3225− ret = -EAGAIN;
3225+ ret = BTRFS_LOG_FORCE_COMMIT;
32263226 goto out;
32273227 }
32283228
@@ -3261,7 +3261,7 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans, · 32 unmodified lines
32613261 blk_finish_plug(&plug);
32623262 btrfs_wait_tree_log_extents(log, mark);
32633263 mutex_unlock(&log_root_tree->log_mutex);
3264− ret = -EAGAIN;
3264+ ret = BTRFS_LOG_FORCE_COMMIT;
32653265 goto out_wake_log_root;
32663266 }
32673267
@@ -5848,7 +5848,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans, · 2580 unmodified lines
58485848 inode_only == LOG_INODE_ALL &&
58495849 inode->last_unlink_trans >= trans->transid) {
58505850 btrfs_set_log_full_commit(trans);
5851− ret = 1;
5851+ ret = BTRFS_LOG_FORCE_COMMIT;
58525852 goto out_unlock;
58535853 }
58545854
@@ -6562,12 +6562,12 @@ static int btrfs_log_inode_parent(struct btrfs_trans_handle *trans, · 707 unmodified lines
65626562 bool log_dentries = false;
65636563
65646564 if (btrfs_test_opt(fs_info, NOTREELOG)) {
6565− ret = 1;
6565+ ret = BTRFS_LOG_FORCE_COMMIT;
65666566 goto end_no_trans;
65676567 }
65686568
65696569 if (btrfs_root_refs(&root->root_item) == 0) {
6570− ret = 1;
6570+ ret = BTRFS_LOG_FORCE_COMMIT;
65716571 goto end_no_trans;
65726572 }
65736573
@@ -6665,7 +6665,7 @@ static int btrfs_log_inode_parent(struct btrfs_trans_handle *trans, · 91 unmodified lines
66656665 end_trans:
66666666 if (ret < 0) {
66676667 btrfs_set_log_full_commit(trans);
6668− ret = 1;
6668+ ret = BTRFS_LOG_FORCE_COMMIT;
66696669 }
66706670
66716671 if (ret)
@@ -7029,8 +7029,15 @@ void btrfs_log_new_name(struct btrfs_trans_handle *trans, · 357 unmodified lines
70297029 * anyone from syncing the log until we have updated both inodes
70307030 * in the log.
70317031 */
7032+ ret = join_running_log_trans(root);
7033+ /*
7034+ * At least one of the inodes was logged before, so this should
7035+ * not fail, but if it does, it's not serious, just bail out and
7036+ * mark the log for a full commit.
7037+ */
7038+ if (WARN_ON_ONCE(ret < 0))
7039+ goto out;
70327040 log_pinned = true;
7033− btrfs_pin_log_trans(root);
70347041
70357042 path = btrfs_alloc_path();
70367043 if (!path) {
fs/btrfs/tree-log.h+3−0
@@ -12,6 +12,9 @@ · 11 unmodified lines
1212 /* return value for btrfs_log_dentry_safe that means we don't need to log it at all */
1313 #define BTRFS_NO_LOG_SYNC 256
1414
15+/* We can't use the tree log for whatever reason, force a transaction commit */
16+#define BTRFS_LOG_FORCE_COMMIT (1)
17+
1518 struct btrfs_log_ctx {
1619 int log_ret;
1720 int log_transid;
fs/btrfs/volumes.c+172−190
@@ -182,6 +182,13 @@ const char *btrfs_bg_type_to_raid_name(u64 flags) · 181 unmodified lines
182182 return btrfs_raid_array[index].raid_name;
183183 }
184184
185+int btrfs_nr_parity_stripes(u64 type)
186+{
187+ enum btrfs_raid_types index = btrfs_bg_flags_to_raid_index(type);
188+
189+ return btrfs_raid_array[index].nparity;
190+}
191+
185192 /*
186193 * Fill @buf with textual description of @bg_flags, no more than @size_buf
187194 * bytes including terminating null byte.
@@ -238,7 +245,6 @@ out_overflow:; · 50 unmodified lines
238245
239246 static int init_first_rw_device(struct btrfs_trans_handle *trans);
240247 static int btrfs_relocate_sys_chunks(struct btrfs_fs_info *fs_info);
241−static void btrfs_dev_stat_print_on_error(struct btrfs_device *dev);
242248 static void btrfs_dev_stat_print_on_load(struct btrfs_device *device);
243249 static int __btrfs_map_block(struct btrfs_fs_info *fs_info,
244250 enum btrfs_map_op op,
@@ -1396,12 +1402,7 @@ static u64 dev_extent_search_start(struct btrfs_device *device, u64 start) · 1151 unmodified lines
13961402 {
13971403 switch (device->fs_devices->chunk_alloc_policy) {
13981404 case BTRFS_CHUNK_ALLOC_REGULAR:
1399− /*
1400− * We don't want to overwrite the superblock on the drive nor
1401− * any area used by the boot loader (grub for example), so we
1402− * make sure to start at an offset of at least 1MB.
1403− */
1404− return max_t(u64, start, SZ_1M);
1405+ return max_t(u64, start, BTRFS_DEVICE_RANGE_RESERVED);
14051406 case BTRFS_CHUNK_ALLOC_ZONED:
14061407 /*
14071408 * We don't care about the starting region like regular
@@ -5071,26 +5072,16 @@ static void init_alloc_chunk_ctl_policy_regular( · 3663 unmodified lines
50715072 struct btrfs_fs_devices *fs_devices,
50725073 struct alloc_chunk_ctl *ctl)
50735074 {
5074− u64 type = ctl->type;
5075+ struct btrfs_space_info *space_info;
50755076
5076− if (type & BTRFS_BLOCK_GROUP_DATA) {
5077− ctl->max_stripe_size = SZ_1G;
5078− ctl->max_chunk_size = BTRFS_MAX_DATA_CHUNK_SIZE;
5079− } else if (type & BTRFS_BLOCK_GROUP_METADATA) {
5080− /* For larger filesystems, use larger metadata chunks */
5081− if (fs_devices->total_rw_bytes > 50ULL * SZ_1G)
5082− ctl->max_stripe_size = SZ_1G;
5083− else
5084− ctl->max_stripe_size = SZ_256M;
5085− ctl->max_chunk_size = ctl->max_stripe_size;
5086− } else if (type & BTRFS_BLOCK_GROUP_SYSTEM) {
5087− ctl->max_stripe_size = SZ_32M;
5088− ctl->max_chunk_size = 2 * ctl->max_stripe_size;
5089− ctl->devs_max = min_t(int, ctl->devs_max,
5090− BTRFS_MAX_DEVS_SYS_CHUNK);
5091− } else {
5092− BUG();
5093− }
5077+ space_info = btrfs_find_space_info(fs_devices->fs_info, ctl->type);
5078+ ASSERT(space_info);
5079+
5080+ ctl->max_chunk_size = READ_ONCE(space_info->chunk_size);
5081+ ctl->max_stripe_size = ctl->max_chunk_size;
5082+
5083+ if (ctl->type & BTRFS_BLOCK_GROUP_SYSTEM)
5084+ ctl->devs_max = min_t(int, ctl->devs_max, BTRFS_MAX_DEVS_SYS_CHUNK);
50945085
50955086 /* We don't want a chunk larger than 10% of writable space */
50965087 ctl->max_chunk_size = min(div_factor(fs_devices->total_rw_bytes, 1),
@@ -5720,7 +5711,8 @@ int btrfs_num_copies(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · 623 unmodified lines
57205711 {
57215712 struct extent_map *em;
57225713 struct map_lookup *map;
5723− int ret;
5714+ enum btrfs_raid_types index;
5715+ int ret = 1;
57245716
57255717 em = btrfs_get_chunk_map(fs_info, logical, len);
57265718 if (IS_ERR(em))
@@ -5733,10 +5725,11 @@ int btrfs_num_copies(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · 6 unmodified lines
57335725 return 1;
57345726
57355727 map = em->map_lookup;
5736− if (map->type & (BTRFS_BLOCK_GROUP_DUP | BTRFS_BLOCK_GROUP_RAID1_MASK))
5737− ret = map->num_stripes;
5738− else if (map->type & BTRFS_BLOCK_GROUP_RAID10)
5739− ret = map->sub_stripes;
5728+ index = btrfs_bg_flags_to_raid_index(map->type);
5729+
5730+ /* Non-RAID56, use their ncopies from btrfs_raid_array. */
5731+ if (!(map->type & BTRFS_BLOCK_GROUP_RAID56_MASK))
5732+ ret = btrfs_raid_array[index].ncopies;
57405733 else if (map->type & BTRFS_BLOCK_GROUP_RAID5)
57415734 ret = 2;
57425735 else if (map->type & BTRFS_BLOCK_GROUP_RAID6)
@@ -5748,8 +5741,6 @@ int btrfs_num_copies(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · 5 unmodified lines
57485741 * stripe under reconstruction.
57495742 */
57505743 ret = map->num_stripes;
5751− else
5752− ret = 1;
57535744 free_extent_map(em);
57545745
57555746 down_read(&fs_info->dev_replace.rwsem);
@@ -5768,6 +5759,9 @@ unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info, · 12 unmodified lines
57685759 struct map_lookup *map;
57695760 unsigned long len = fs_info->sectorsize;
57705761
5762+ if (!btrfs_fs_incompat(fs_info, RAID56))
5763+ return len;
5764+
57715765 em = btrfs_get_chunk_map(fs_info, logical, len);
57725766
57735767 if (!WARN_ON(IS_ERR(em))) {
@@ -5785,6 +5779,9 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, u64 logical, u64 len) · 11 unmodified lines
57855779 struct map_lookup *map;
57865780 int ret = 0;
57875781
5782+ if (!btrfs_fs_incompat(fs_info, RAID56))
5783+ return 0;
5784+
57885785 em = btrfs_get_chunk_map(fs_info, logical, len);
57895786
57905787 if(!WARN_ON(IS_ERR(em))) {
@@ -5917,18 +5914,17 @@ void btrfs_put_bioc(struct btrfs_io_context *bioc) · 126 unmodified lines
59175914 kfree(bioc);
59185915 }
59195916
5920−/* can REQ_OP_DISCARD be sent with other REQ like REQ_OP_WRITE? */
59215917 /*
59225918 * Please note that, discard won't be sent to target device of device
59235919 * replace.
59245920 */
5925−static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info,
5926− u64 logical, u64 *length_ret,
5927− struct btrfs_io_context **bioc_ret)
5921+struct btrfs_discard_stripe *btrfs_map_discard(struct btrfs_fs_info *fs_info,
5922+ u64 logical, u64 *length_ret,
5923+ u32 *num_stripes)
59285924 {
59295925 struct extent_map *em;
59305926 struct map_lookup *map;
5931− struct btrfs_io_context *bioc;
5927+ struct btrfs_discard_stripe *stripes;
59325928 u64 length = *length_ret;
59335929 u64 offset;
59345930 u64 stripe_nr;
@@ -5937,29 +5933,26 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, · 2 unmodified lines
59375933 u64 stripe_cnt;
59385934 u64 stripe_len;
59395935 u64 stripe_offset;
5940− u64 num_stripes;
59415936 u32 stripe_index;
59425937 u32 factor = 0;
59435938 u32 sub_stripes = 0;
59445939 u64 stripes_per_dev = 0;
59455940 u32 remaining_stripes = 0;
59465941 u32 last_stripe = 0;
5947− int ret = 0;
5942+ int ret;
59485943 int i;
59495944
5950− /* Discard always returns a bioc. */
5951− ASSERT(bioc_ret);
5952−
59535945 em = btrfs_get_chunk_map(fs_info, logical, length);
59545946 if (IS_ERR(em))
5955− return PTR_ERR(em);
5947+ return ERR_CAST(em);
59565948
59575949 map = em->map_lookup;
5950+
59585951 /* we don't discard raid56 yet */
59595952 if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) {
59605953 ret = -EOPNOTSUPP;
5961− goto out;
5962− }
5954+ goto out_free_map;
5955+}
59635956
59645957 offset = logical - em->start;
59655958 length = min_t(u64, em->start + em->len - logical, length);
@@ -5985,7 +5978,7 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, · 19 unmodified lines
59855978 * device we have to walk to find the data, and stripe_index is
59865979 * the number of our device in the stripe array
59875980 */
5988− num_stripes = 1;
5981+ *num_stripes = 1;
59895982 stripe_index = 0;
59905983 if (map->type & (BTRFS_BLOCK_GROUP_RAID0 |
59915984 BTRFS_BLOCK_GROUP_RAID10)) {
@@ -5995,7 +5988,7 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, · 3 unmodified lines
59955988 sub_stripes = map->sub_stripes;
59965989
59975990 factor = map->num_stripes / sub_stripes;
5998− num_stripes = min_t(u64, map->num_stripes,
5991+ *num_stripes = min_t(u64, map->num_stripes,
59995992 sub_stripes * stripe_cnt);
60005993 stripe_nr = div_u64_rem(stripe_nr, factor, &stripe_index);
60015994 stripe_index *= sub_stripes;
@@ -6005,31 +5998,30 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, · 3 unmodified lines
60055998 last_stripe *= sub_stripes;
60065999 } else if (map->type & (BTRFS_BLOCK_GROUP_RAID1_MASK |
60076000 BTRFS_BLOCK_GROUP_DUP)) {
6008− num_stripes = map->num_stripes;
6001+ *num_stripes = map->num_stripes;
60096002 } else {
60106003 stripe_nr = div_u64_rem(stripe_nr, map->num_stripes,
60116004 &stripe_index);
60126005 }
60136006
6014− bioc = alloc_btrfs_io_context(fs_info, num_stripes, 0);
6015− if (!bioc) {
6007+ stripes = kcalloc(*num_stripes, sizeof(*stripes), GFP_NOFS);
6008+ if (!stripes) {
60166009 ret = -ENOMEM;
6017− goto out;
6010+ goto out_free_map;
60186011 }
60196012
6020− for (i = 0; i < num_stripes; i++) {
6021− bioc->stripes[i].physical =
6013+ for (i = 0; i < *num_stripes; i++) {
6014+ stripes[i].physical =
60226015 map->stripes[stripe_index].physical +
60236016 stripe_offset + stripe_nr * map->stripe_len;
6024− bioc->stripes[i].dev = map->stripes[stripe_index].dev;
6017+ stripes[i].dev = map->stripes[stripe_index].dev;
60256018
60266019 if (map->type & (BTRFS_BLOCK_GROUP_RAID0 |
60276020 BTRFS_BLOCK_GROUP_RAID10)) {
6028− bioc->stripes[i].length = stripes_per_dev *
6029− map->stripe_len;
6021+ stripes[i].length = stripes_per_dev * map->stripe_len;
60306022
60316023 if (i / sub_stripes < remaining_stripes)
6032− bioc->stripes[i].length += map->stripe_len;
6024+ stripes[i].length += map->stripe_len;
60336025
60346026 /*
60356027 * Special for the first stripe and
@@ -6040,17 +6032,17 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, · 4 unmodified lines
60406032 * off end_off
60416033 */
60426034 if (i < sub_stripes)
6043− bioc->stripes[i].length -= stripe_offset;
6035+ stripes[i].length -= stripe_offset;
60446036
60456037 if (stripe_index >= last_stripe &&
60466038 stripe_index <= (last_stripe +
60476039 sub_stripes - 1))
6048− bioc->stripes[i].length -= stripe_end_offset;
6040+ stripes[i].length -= stripe_end_offset;
60496041
60506042 if (i == sub_stripes - 1)
60516043 stripe_offset = 0;
60526044 } else {
6053− bioc->stripes[i].length = length;
6045+ stripes[i].length = length;
60546046 }
60556047
60566048 stripe_index++;
@@ -6060,12 +6052,11 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, · 3 unmodified lines
60606052 }
60616053 }
60626054
6063− *bioc_ret = bioc;
6064− bioc->map_type = map->type;
6065− bioc->num_stripes = num_stripes;
6066−out:
60676055 free_extent_map(em);
6068− return ret;
6056+ return stripes;
6057+out_free_map:
6058+ free_extent_map(em);
6059+ return ERR_PTR(ret);
60696060 }
60706061
60716062 /*
@@ -6208,7 +6199,6 @@ static void handle_ops_on_dev_replace(enum btrfs_map_op op, · 136 unmodified lines
62086199 bioc->stripes + i;
62096200
62106201 new->physical = old->physical;
6211− new->length = old->length;
62126202 new->dev = dev_replace->tgtdev;
62136203 bioc->tgtdev_map[i] = index_where_to_add;
62146204 index_where_to_add++;
@@ -6249,8 +6239,6 @@ static void handle_ops_on_dev_replace(enum btrfs_map_op op, · 34 unmodified lines
62496239 bioc->stripes + num_stripes;
62506240
62516241 tgtdev_stripe->physical = physical_of_found;
6252− tgtdev_stripe->length =
6253− bioc->stripes[index_srcdev].length;
62546242 tgtdev_stripe->dev = dev_replace->tgtdev;
62556243 bioc->tgtdev_map[index_srcdev] = num_stripes;
62566244
@@ -6472,6 +6460,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, · 215 unmodified lines
64726460 }
64736461
64746462 } else if (map->type & BTRFS_BLOCK_GROUP_RAID56_MASK) {
6463+ ASSERT(map->stripe_len == BTRFS_STRIPE_LEN);
64756464 if (need_raid_map && (need_full_stripe(op) || mirror_num > 1)) {
64766465 /* push stripe_nr back to the start of the full stripe */
64776466 stripe_nr = div64_u64(raid56_full_stripe_start,
@@ -6479,9 +6468,12 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info, · 1 unmodified line
64796468
64806469 /* RAID[56] write or recovery. Return all stripes */
64816470 num_stripes = map->num_stripes;
6482− max_errors = nr_parity_stripes(map);
6471+ max_errors = btrfs_chunk_max_errors(map);
64836472
6484− *length = map->stripe_len;
6473+ /* Return the length to the full stripe end */
6474+ *length = min(logical + *length,
6475+ raid56_full_stripe_start + em->start +
6476+ data_stripes * stripe_len) - logical;
64856477 stripe_index = 0;
64866478 stripe_offset = 0;
64876479 } else {
@@ -6604,10 +6596,6 @@ int btrfs_map_block(struct btrfs_fs_info *fs_info, enum btrfs_map_op op, · 116 unmodified lines
66046596 u64 logical, u64 *length,
66056597 struct btrfs_io_context **bioc_ret, int mirror_num)
66066598 {
6607− if (op == BTRFS_MAP_DISCARD)
6608− return __btrfs_map_block_for_discard(fs_info, logical,
6609− length, bioc_ret);
6610−
66116599 return __btrfs_map_block(fs_info, op, logical, length, bioc_ret,
66126600 mirror_num, 0);
66136601 }
@@ -6620,77 +6608,106 @@ int btrfs_map_sblock(struct btrfs_fs_info *fs_info, enum btrfs_map_op op, · 6 unmodified lines
66206608 return __btrfs_map_block(fs_info, op, logical, length, bioc_ret, 0, 1);
66216609 }
66226610
6623−static inline void btrfs_end_bioc(struct btrfs_io_context *bioc, struct bio *bio)
6611+static struct workqueue_struct *btrfs_end_io_wq(struct btrfs_io_context *bioc)
6612+{
6613+ if (bioc->orig_bio->bi_opf & REQ_META)
6614+ return bioc->fs_info->endio_meta_workers;
6615+ return bioc->fs_info->endio_workers;
6616+}
6617+
6618+static void btrfs_end_bio_work(struct work_struct *work)
6619+{
6620+ struct btrfs_bio *bbio =
6621+ container_of(work, struct btrfs_bio, end_io_work);
6622+
6623+ bio_endio(&bbio->bio);
6624+}
6625+
6626+static void btrfs_end_bioc(struct btrfs_io_context *bioc, bool async)
66246627 {
6625− bio->bi_private = bioc->private;
6626− bio->bi_end_io = bioc->end_io;
6627− bio_endio(bio);
6628+ struct bio *orig_bio = bioc->orig_bio;
6629+ struct btrfs_bio *bbio = btrfs_bio(orig_bio);
6630+
6631+ bbio->mirror_num = bioc->mirror_num;
6632+ orig_bio->bi_private = bioc->private;
6633+ orig_bio->bi_end_io = bioc->end_io;
6634+
6635+ /*
6636+ * Only send an error to the higher layers if it is beyond the tolerance
6637+ * threshold.
6638+ */
6639+ if (atomic_read(&bioc->error) > bioc->max_errors)
6640+ orig_bio->bi_status = BLK_STS_IOERR;
6641+ else
6642+ orig_bio->bi_status = BLK_STS_OK;
6643+
6644+ if (btrfs_op(orig_bio) == BTRFS_MAP_READ && async) {
6645+ INIT_WORK(&bbio->end_io_work, btrfs_end_bio_work);
6646+ queue_work(btrfs_end_io_wq(bioc), &bbio->end_io_work);
6647+ } else {
6648+ bio_endio(orig_bio);
6649+ }
66286650
66296651 btrfs_put_bioc(bioc);
66306652 }
66316653
66326654 static void btrfs_end_bio(struct bio *bio)
66336655 {
6634− struct btrfs_io_context *bioc = bio->bi_private;
6635− int is_orig_bio = 0;
6656+ struct btrfs_io_stripe *stripe = bio->bi_private;
6657+ struct btrfs_io_context *bioc = stripe->bioc;
66366658
66376659 if (bio->bi_status) {
66386660 atomic_inc(&bioc->error);
66396661 if (bio->bi_status == BLK_STS_IOERR ||
66406662 bio->bi_status == BLK_STS_TARGET) {
6641− struct btrfs_device *dev = btrfs_bio(bio)->device;
6642−
6643− ASSERT(dev->bdev);
66446663 if (btrfs_op(bio) == BTRFS_MAP_WRITE)
6645− btrfs_dev_stat_inc_and_print(dev,
6664+ btrfs_dev_stat_inc_and_print(stripe->dev,
66466665 BTRFS_DEV_STAT_WRITE_ERRS);
66476666 else if (!(bio->bi_opf & REQ_RAHEAD))
6648− btrfs_dev_stat_inc_and_print(dev,
6667+ btrfs_dev_stat_inc_and_print(stripe->dev,
66496668 BTRFS_DEV_STAT_READ_ERRS);
66506669 if (bio->bi_opf & REQ_PREFLUSH)
6651− btrfs_dev_stat_inc_and_print(dev,
6670+ btrfs_dev_stat_inc_and_print(stripe->dev,
66526671 BTRFS_DEV_STAT_FLUSH_ERRS);
66536672 }
66546673 }
66556674
6656− if (bio == bioc->orig_bio)
6657− is_orig_bio = 1;
6675+ if (bio != bioc->orig_bio)
6676+ bio_put(bio);
66586677
66596678 btrfs_bio_counter_dec(bioc->fs_info);
6660−
6661− if (atomic_dec_and_test(&bioc->stripes_pending)) {
6662− if (!is_orig_bio) {
6663− bio_put(bio);
6664− bio = bioc->orig_bio;
6665− }
6666−
6667− btrfs_bio(bio)->mirror_num = bioc->mirror_num;
6668− /* only send an error to the higher layers if it is
6669− * beyond the tolerance of the btrfs bio
6670− */
6671− if (atomic_read(&bioc->error) > bioc->max_errors) {
6672− bio->bi_status = BLK_STS_IOERR;
6673− } else {
6674− /*
6675− * this bio is actually up to date, we didn't
6676− * go over the max number of errors
6677− */
6678− bio->bi_status = BLK_STS_OK;
6679− }
6680−
6681− btrfs_end_bioc(bioc, bio);
6682− } else if (!is_orig_bio) {
6683− bio_put(bio);
6684− }
6679+ if (atomic_dec_and_test(&bioc->stripes_pending))
6680+ btrfs_end_bioc(bioc, true);
66856681 }
66866682
6687−static void submit_stripe_bio(struct btrfs_io_context *bioc, struct bio *bio,
6688− u64 physical, struct btrfs_device *dev)
6683+static void submit_stripe_bio(struct btrfs_io_context *bioc,
6684+ struct bio *orig_bio, int dev_nr, bool clone)
66896685 {
66906686 struct btrfs_fs_info *fs_info = bioc->fs_info;
6687+ struct btrfs_device *dev = bioc->stripes[dev_nr].dev;
6688+ u64 physical = bioc->stripes[dev_nr].physical;
6689+ struct bio *bio;
6690+
6691+ if (!dev || !dev->bdev ||
6692+ test_bit(BTRFS_DEV_STATE_MISSING, &dev->dev_state) ||
6693+ (btrfs_op(orig_bio) == BTRFS_MAP_WRITE &&
6694+ !test_bit(BTRFS_DEV_STATE_WRITEABLE, &dev->dev_state))) {
6695+ atomic_inc(&bioc->error);
6696+ if (atomic_dec_and_test(&bioc->stripes_pending))
6697+ btrfs_end_bioc(bioc, false);
6698+ return;
6699+ }
6700+
6701+ if (clone) {
6702+ bio = bio_alloc_clone(dev->bdev, orig_bio, GFP_NOFS, &fs_bio_set);
6703+ } else {
6704+ bio = orig_bio;
6705+ bio_set_dev(bio, dev->bdev);
6706+ btrfs_bio(bio)->device = dev;
6707+ }
66916708
6692− bio->bi_private = bioc;
6693− btrfs_bio(bio)->device = dev;
6709+ bioc->stripes[dev_nr].bioc = bioc;
6710+ bio->bi_private = &bioc->stripes[dev_nr];
66946711 bio->bi_end_io = btrfs_end_bio;
66956712 bio->bi_iter.bi_sector = physical >> 9;
66966713 /*
@@ -6708,8 +6725,8 @@ static void submit_stripe_bio(struct btrfs_io_context *bioc, struct bio *bio, · 11 unmodified lines
67086725 }
67096726 }
67106727 btrfs_debug_in_rcu(fs_info,
6711− "btrfs_map_bio: rw %d 0x%x, sector=%llu, dev=%lu (%s id %llu), size=%u",
6712− bio_op(bio), bio->bi_opf, bio->bi_iter.bi_sector,
6728+ "%s: rw %d 0x%x, sector=%llu, dev=%lu (%s id %llu), size=%u",
6729+ __func__, bio_op(bio), bio->bi_opf, bio->bi_iter.bi_sector,
67136730 (unsigned long)dev->bdev->bd_dev, rcu_str_deref(dev->name),
67146731 dev->devid, bio->bi_iter.bi_size);
67156732
@@ -6719,66 +6736,39 @@ static void submit_stripe_bio(struct btrfs_io_context *bioc, struct bio *bio, · 3 unmodified lines
67196736 submit_bio(bio);
67206737 }
67216738
6722−static void bioc_error(struct btrfs_io_context *bioc, struct bio *bio, u64 logical)
6723−{
6724− atomic_inc(&bioc->error);
6725− if (atomic_dec_and_test(&bioc->stripes_pending)) {
6726− /* Should be the original bio. */
6727− WARN_ON(bio != bioc->orig_bio);
6728−
6729− btrfs_bio(bio)->mirror_num = bioc->mirror_num;
6730− bio->bi_iter.bi_sector = logical >> 9;
6731− if (atomic_read(&bioc->error) > bioc->max_errors)
6732− bio->bi_status = BLK_STS_IOERR;
6733− else
6734− bio->bi_status = BLK_STS_OK;
6735− btrfs_end_bioc(bioc, bio);
6736− }
6737−}
6738−
6739−blk_status_t btrfs_map_bio(struct btrfs_fs_info *fs_info, struct bio *bio,
6740− int mirror_num)
6739+void btrfs_submit_bio(struct btrfs_fs_info *fs_info, struct bio *bio, int mirror_num)
67416740 {
6742− struct btrfs_device *dev;
6743− struct bio *first_bio = bio;
67446741 u64 logical = bio->bi_iter.bi_sector << 9;
6745− u64 length = 0;
6746− u64 map_length;
6742+ u64 length = bio->bi_iter.bi_size;
6743+ u64 map_length = length;
67476744 int ret;
67486745 int dev_nr;
67496746 int total_devs;
67506747 struct btrfs_io_context *bioc = NULL;
67516748
6752− length = bio->bi_iter.bi_size;
6753− map_length = length;
6754−
67556749 btrfs_bio_counter_inc_blocked(fs_info);
67566750 ret = __btrfs_map_block(fs_info, btrfs_op(bio), logical,
67576751 &map_length, &bioc, mirror_num, 1);
67586752 if (ret) {
67596753 btrfs_bio_counter_dec(fs_info);
6760− return errno_to_blk_status(ret);
6754+ bio->bi_status = errno_to_blk_status(ret);
6755+ bio_endio(bio);
6756+ return;
67616757 }
67626758
67636759 total_devs = bioc->num_stripes;
6764− bioc->orig_bio = first_bio;
6765− bioc->private = first_bio->bi_private;
6766− bioc->end_io = first_bio->bi_end_io;
6767− atomic_set(&bioc->stripes_pending, bioc->num_stripes);
6760+ bioc->orig_bio = bio;
6761+ bioc->private = bio->bi_private;
6762+ bioc->end_io = bio->bi_end_io;
6763+ atomic_set(&bioc->stripes_pending, total_devs);
67686764
67696765 if ((bioc->map_type & BTRFS_BLOCK_GROUP_RAID56_MASK) &&
67706766 ((btrfs_op(bio) == BTRFS_MAP_WRITE) || (mirror_num > 1))) {
6771− /* In this case, map_length has been set to the length of
6772− a single stripe; not the whole write */
6773− if (btrfs_op(bio) == BTRFS_MAP_WRITE) {
6774− ret = raid56_parity_write(bio, bioc, map_length);
6775− } else {
6776− ret = raid56_parity_recover(bio, bioc, map_length,
6777− mirror_num, 1);
6778− }
6779−
6780− btrfs_bio_counter_dec(fs_info);
6781− return errno_to_blk_status(ret);
6767+ if (btrfs_op(bio) == BTRFS_MAP_WRITE)
6768+ raid56_parity_write(bio, bioc);
6769+ else
6770+ raid56_parity_recover(bio, bioc, mirror_num, true);
6771+ return;
67826772 }
67836773
67846774 if (map_length < length) {
@@ -6789,26 +6779,11 @@ blk_status_t btrfs_map_bio(struct btrfs_fs_info *fs_info, struct bio *bio, · 4 unmodified lines
67896779 }
67906780
67916781 for (dev_nr = 0; dev_nr < total_devs; dev_nr++) {
6792− dev = bioc->stripes[dev_nr].dev;
6793− if (!dev || !dev->bdev || test_bit(BTRFS_DEV_STATE_MISSING,
6794− &dev->dev_state) ||
6795− (btrfs_op(first_bio) == BTRFS_MAP_WRITE &&
6796− !test_bit(BTRFS_DEV_STATE_WRITEABLE, &dev->dev_state))) {
6797− bioc_error(bioc, first_bio, logical);
6798− continue;
6799− }
6800−
6801− if (dev_nr < total_devs - 1) {
6802− bio = btrfs_bio_clone(dev->bdev, first_bio);
6803− } else {
6804− bio = first_bio;
6805− bio_set_dev(bio, dev->bdev);
6806− }
6782+ const bool should_clone = (dev_nr < total_devs - 1);
68076783
6808− submit_stripe_bio(bioc, bio, bioc->stripes[dev_nr].physical, dev);
6784+ submit_stripe_bio(bioc, bio, dev_nr, should_clone);
68096785 }
68106786 btrfs_bio_counter_dec(fs_info);
6811− return BLK_STS_OK;
68126787 }
68136788
68146789 static bool dev_args_match_fs_devices(const struct btrfs_dev_lookup_args *args,
@@ -6966,11 +6941,12 @@ static void btrfs_report_missing_device(struct btrfs_fs_info *fs_info, · 151 unmodified lines
69666941 devid, uuid);
69676942 }
69686943
6969−static u64 calc_stripe_length(u64 type, u64 chunk_len, int num_stripes)
6944+u64 btrfs_calc_stripe_length(const struct extent_map *em)
69706945 {
6971− const int data_stripes = calc_data_stripes(type, num_stripes);
6946+ const struct map_lookup *map = em->map_lookup;
6947+ const int data_stripes = calc_data_stripes(map->type, map->num_stripes);
69726948
6973− return div_u64(chunk_len, data_stripes);
6949+ return div_u64(em->len, data_stripes);
69746950 }
69756951
69766952 #if BITS_PER_LONG == 32
@@ -7109,8 +7085,7 @@ static int read_one_chunk(struct btrfs_key *key, struct extent_buffer *leaf, · 132 unmodified lines
71097085 map->type = type;
71107086 map->sub_stripes = btrfs_chunk_sub_stripes(leaf, chunk);
71117087 map->verified_stripes = 0;
7112− em->orig_block_len = calc_stripe_length(type, em->len,
7113− map->num_stripes);
7088+ em->orig_block_len = btrfs_calc_stripe_length(em);
71147089 for (i = 0; i < num_stripes; i++) {
71157090 map->stripes[i].physical =
71167091 btrfs_stripe_offset_nr(leaf, chunk, i);
@@ -7236,7 +7211,8 @@ static int read_one_dev(struct extent_buffer *leaf, · 119 unmodified lines
72367211 u8 fs_uuid[BTRFS_FSID_SIZE];
72377212 u8 dev_uuid[BTRFS_UUID_SIZE];
72387213
7239− devid = args.devid = btrfs_device_id(leaf, dev_item);
7214+ devid = btrfs_device_id(leaf, dev_item);
7215+ args.devid = devid;
72407216 read_extent_buffer(leaf, dev_uuid, btrfs_device_uuid(dev_item),
72417217 BTRFS_UUID_SIZE);
72427218 read_extent_buffer(leaf, fs_uuid, btrfs_device_fsid(dev_item),
@@ -7865,11 +7841,7 @@ int btrfs_run_dev_stats(struct btrfs_trans_handle *trans) · 622 unmodified lines
78657841 void btrfs_dev_stat_inc_and_print(struct btrfs_device *dev, int index)
78667842 {
78677843 btrfs_dev_stat_inc(dev, index);
7868− btrfs_dev_stat_print_on_error(dev);
7869−}
78707844
7871−static void btrfs_dev_stat_print_on_error(struct btrfs_device *dev)
7872−{
78737845 if (!dev->dev_stats_valid)
78747846 return;
78757847 btrfs_err_rl_in_rcu(dev->fs_info,
@@ -8011,7 +7983,7 @@ static int verify_one_dev_extent(struct btrfs_fs_info *fs_info, · 135 unmodified lines
80117983 }
80127984
80137985 map = em->map_lookup;
8014− stripe_len = calc_stripe_length(map->type, em->len, map->num_stripes);
7986+ stripe_len = btrfs_calc_stripe_length(em);
80157987 if (physical_len != stripe_len) {
80167988 btrfs_err(fs_info,
80177989 "dev extent physical offset %llu on devid %llu length doesn't match chunk %llu, have %llu expect %llu",
@@ -8021,6 +7993,16 @@ static int verify_one_dev_extent(struct btrfs_fs_info *fs_info, · 3 unmodified lines
80217993 goto out;
80227994 }
80237995
7996+ /*
7997+ * Very old mkfs.btrfs (before v4.1) will not respect the reserved
7998+ * space. Although kernel can handle it without problem, better to warn
7999+ * the users.
8000+ */
8001+ if (physical_offset < BTRFS_DEVICE_RANGE_RESERVED)
8002+ btrfs_warn(fs_info,
8003+ "devid %llu physical %llu len %llu inside the reserved space",
8004+ devid, physical_offset, physical_len);
8005+
80248006 for (i = 0; i < map->num_stripes; i++) {
80258007 if (map->stripes[i].dev->devid == devid &&
80268008 map->stripes[i].physical == physical_offset) {
fs/btrfs/volumes.h+43−3
@@ -354,6 +354,13 @@ struct btrfs_fs_devices { · 353 unmodified lines
354354 - 2 * sizeof(struct btrfs_chunk)) \
355355 / sizeof(struct btrfs_stripe) + 1)
356356
357+/*
358+ * Maximum number of sectors for a single bio to limit the size of the
359+ * checksum array. This matches the number of bio_vecs per bio and thus the
360+ * I/O size for buffered I/O.
361+ */
362+#define BTRFS_MAX_BIO_SECTORS (256)
363+
357364 /*
358365 * Additional info to pass along bio.
359366 *
@@ -371,6 +378,9 @@ struct btrfs_bio { · 11 unmodified lines
371378 u8 csum_inline[BTRFS_BIO_INLINE_CSUM_SIZE];
372379 struct bvec_iter iter;
373380
381+ /* For read end I/O handling */
382+ struct work_struct end_io_work;
383+
374384 /*
375385 * This member must come last, bio_alloc_bioset will allocate enough
376386 * bytes for entire btrfs_bio but relies on bio being last.
@@ -391,10 +401,36 @@ static inline void btrfs_bio_free_csum(struct btrfs_bio *bbio) · 14 unmodified lines
391401 }
392402 }
393403
404+/*
405+ * Iterate through a btrfs_bio (@bbio) on a per-sector basis.
406+ *
407+ * bvl - struct bio_vec
408+ * bbio - struct btrfs_bio
409+ * iters - struct bvec_iter
410+ * bio_offset - unsigned int
411+ */
412+#define btrfs_bio_for_each_sector(fs_info, bvl, bbio, iter, bio_offset) \
413+ for ((iter) = (bbio)->iter, (bio_offset) = 0; \
414+ (iter).bi_size && \
415+ (((bvl) = bio_iter_iovec((&(bbio)->bio), (iter))), 1); \
416+ (bio_offset) += fs_info->sectorsize, \
417+ bio_advance_iter_single(&(bbio)->bio, &(iter), \
418+ (fs_info)->sectorsize))
419+
394420 struct btrfs_io_stripe {
421+ struct btrfs_device *dev;
422+ union {
423+ /* Block mapping */
424+ u64 physical;
425+ /* For the endio handler */
426+ struct btrfs_io_context *bioc;
427+ };
428+};
429+
430+struct btrfs_discard_stripe {
395431 struct btrfs_device *dev;
396432 u64 physical;
397− u64 length; /* only used for discard mappings */
433+ u64 length;
398434 };
399435
400436 /*
@@ -533,6 +569,9 @@ int btrfs_map_block(struct btrfs_fs_info *fs_info, enum btrfs_map_op op, · 132 unmodified lines
533569 int btrfs_map_sblock(struct btrfs_fs_info *fs_info, enum btrfs_map_op op,
534570 u64 logical, u64 *length,
535571 struct btrfs_io_context **bioc_ret);
572+struct btrfs_discard_stripe *btrfs_map_discard(struct btrfs_fs_info *fs_info,
573+ u64 logical, u64 *length_ret,
574+ u32 *num_stripes);
536575 int btrfs_get_io_geometry(struct btrfs_fs_info *fs_info, struct extent_map *map,
537576 enum btrfs_map_op op, u64 logical,
538577 struct btrfs_io_geometry *io_geom);
@@ -541,8 +580,7 @@ int btrfs_read_chunk_tree(struct btrfs_fs_info *fs_info); · 2 unmodified lines
541580 struct btrfs_block_group *btrfs_create_chunk(struct btrfs_trans_handle *trans,
542581 u64 type);
543582 void btrfs_mapping_tree_free(struct extent_map_tree *tree);
544−blk_status_t btrfs_map_bio(struct btrfs_fs_info *fs_info, struct bio *bio,
545− int mirror_num);
583+void btrfs_submit_bio(struct btrfs_fs_info *fs_info, struct bio *bio, int mirror_num);
546584 int btrfs_open_devices(struct btrfs_fs_devices *fs_devices,
547585 fmode_t flags, void *holder);
548586 struct btrfs_device *btrfs_scan_one_device(const char *path,
@@ -601,6 +639,8 @@ int btrfs_is_parity_mirror(struct btrfs_fs_info *fs_info, · 52 unmodified lines
601639 u64 logical, u64 len);
602640 unsigned long btrfs_full_stripe_len(struct btrfs_fs_info *fs_info,
603641 u64 logical);
642+u64 btrfs_calc_stripe_length(const struct extent_map *em);
643+int btrfs_nr_parity_stripes(u64 type);
604644 int btrfs_chunk_alloc_add_chunk_item(struct btrfs_trans_handle *trans,
605645 struct btrfs_block_group *bg);
606646 int btrfs_remove_chunk(struct btrfs_trans_handle *trans, u64 chunk_offset);
fs/btrfs/zlib.c+18−24
@@ -97,7 +97,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping, · 96 unmodified lines
9797 {
9898 struct workspace *workspace = list_entry(ws, struct workspace, list);
9999 int ret;
100− char *data_in;
100+ char *data_in = NULL;
101101 char *cpage_out;
102102 int nr_pages = 0;
103103 struct page *in_page = NULL;
@@ -126,7 +126,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping, · 22 unmodified lines
126126 ret = -ENOMEM;
127127 goto out;
128128 }
129− cpage_out = kmap(out_page);
129+ cpage_out = page_address(out_page);
130130 pages[0] = out_page;
131131 nr_pages = 1;
132132
@@ -148,26 +148,26 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping, · 15 unmodified lines
148148 int i;
149149
150150 for (i = 0; i < in_buf_pages; i++) {
151− if (in_page) {
152− kunmap(in_page);
151+ if (data_in) {
152+ kunmap_local(data_in);
153153 put_page(in_page);
154154 }
155155 in_page = find_get_page(mapping,
156156 start >> PAGE_SHIFT);
157− data_in = kmap(in_page);
157+ data_in = kmap_local_page(in_page);
158158 memcpy(workspace->buf + i * PAGE_SIZE,
159159 data_in, PAGE_SIZE);
160160 start += PAGE_SIZE;
161161 }
162162 workspace->strm.next_in = workspace->buf;
163163 } else {
164− if (in_page) {
165− kunmap(in_page);
164+ if (data_in) {
165+ kunmap_local(data_in);
166166 put_page(in_page);
167167 }
168168 in_page = find_get_page(mapping,
169169 start >> PAGE_SHIFT);
170− data_in = kmap(in_page);
170+ data_in = kmap_local_page(in_page);
171171 start += PAGE_SIZE;
172172 workspace->strm.next_in = data_in;
173173 }
@@ -196,9 +196,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping, · 22 unmodified lines
196196 * the stream end if required
197197 */
198198 if (workspace->strm.avail_out == 0) {
199− kunmap(out_page);
200199 if (nr_pages == nr_dest_pages) {
201− out_page = NULL;
202200 ret = -E2BIG;
203201 goto out;
204202 }
@@ -207,7 +205,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping, · 2 unmodified lines
207205 ret = -ENOMEM;
208206 goto out;
209207 }
210− cpage_out = kmap(out_page);
208+ cpage_out = page_address(out_page);
211209 pages[nr_pages] = out_page;
212210 nr_pages++;
213211 workspace->strm.avail_out = PAGE_SIZE;
@@ -234,9 +232,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping, · 20 unmodified lines
234232 goto out;
235233 } else if (workspace->strm.avail_out == 0) {
236234 /* get another page for the stream end */
237− kunmap(out_page);
238235 if (nr_pages == nr_dest_pages) {
239− out_page = NULL;
240236 ret = -E2BIG;
241237 goto out;
242238 }
@@ -245,7 +241,7 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping, · 2 unmodified lines
245241 ret = -ENOMEM;
246242 goto out;
247243 }
248− cpage_out = kmap(out_page);
244+ cpage_out = page_address(out_page);
249245 pages[nr_pages] = out_page;
250246 nr_pages++;
251247 workspace->strm.avail_out = PAGE_SIZE;
@@ -264,13 +260,11 @@ int zlib_compress_pages(struct list_head *ws, struct address_space *mapping, · 12 unmodified lines
264260 *total_in = workspace->strm.total_in;
265261 out:
266262 *out_pages = nr_pages;
267− if (out_page)
268− kunmap(out_page);
269−
270− if (in_page) {
271− kunmap(in_page);
263+ if (data_in) {
264+ kunmap_local(data_in);
272265 put_page(in_page);
273266 }
267+
274268 return ret;
275269 }
276270
@@ -287,7 +281,7 @@ int zlib_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 10 unmodified lines
287281 unsigned long buf_start;
288282 struct page **pages_in = cb->compressed_pages;
289283
290− data_in = kmap(pages_in[page_in_index]);
284+ data_in = kmap_local_page(pages_in[page_in_index]);
291285 workspace->strm.next_in = data_in;
292286 workspace->strm.avail_in = min_t(size_t, srclen, PAGE_SIZE);
293287 workspace->strm.total_in = 0;
@@ -309,7 +303,7 @@ int zlib_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 15 unmodified lines
309303
310304 if (Z_OK != zlib_inflateInit2(&workspace->strm, wbits)) {
311305 pr_warn("BTRFS: inflateInit failed\n");
312− kunmap(pages_in[page_in_index]);
306+ kunmap_local(data_in);
313307 return -EIO;
314308 }
315309 while (workspace->strm.total_in < srclen) {
@@ -336,13 +330,13 @@ int zlib_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 20 unmodified lines
336330
337331 if (workspace->strm.avail_in == 0) {
338332 unsigned long tmp;
339− kunmap(pages_in[page_in_index]);
333+ kunmap_local(data_in);
340334 page_in_index++;
341335 if (page_in_index >= total_pages_in) {
342336 data_in = NULL;
343337 break;
344338 }
345− data_in = kmap(pages_in[page_in_index]);
339+ data_in = kmap_local_page(pages_in[page_in_index]);
346340 workspace->strm.next_in = data_in;
347341 tmp = srclen - workspace->strm.total_in;
348342 workspace->strm.avail_in = min(tmp, PAGE_SIZE);
@@ -355,7 +349,7 @@ int zlib_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 6 unmodified lines
355349 done:
356350 zlib_inflateEnd(&workspace->strm);
357351 if (data_in)
358− kunmap(pages_in[page_in_index]);
352+ kunmap_local(data_in);
359353 if (!ret)
360354 zero_fill_bio(cb->orig_bio);
361355 return ret;
fs/btrfs/zoned.c+128−3
@@ -94,9 +94,9 @@ static int sb_write_pointer(struct block_device *bdev, struct blk_zone *zones, · 93 unmodified lines
9494 * Possible states of log buffer zones
9595 *
9696 * Empty[0] In use[0] Full[0]
97− * Empty[1] * x 0
98− * In use[1] 0 x 0
99− * Full[1] 1 1 C
97+ * Empty[1] * 0 1
98+ * In use[1] x x 1
99+ * Full[1] 0 0 C
100100 *
101101 * Log position:
102102 * *: Special case, no superblock is written
@@ -415,6 +415,16 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device, bool populate_cache) · 312 unmodified lines
415415 nr_sectors = bdev_nr_sectors(bdev);
416416 zone_info->zone_size_shift = ilog2(zone_info->zone_size);
417417 zone_info->nr_zones = nr_sectors >> ilog2(zone_sectors);
418+ /*
419+ * We limit max_zone_append_size also by max_segments *
420+ * PAGE_SIZE. Technically, we can have multiple pages per segment. But,
421+ * since btrfs adds the pages one by one to a bio, and btrfs cannot
422+ * increase the metadata reservation even if it increases the number of
423+ * extents, it is safe to stick with the limit.
424+ */
425+ zone_info->max_zone_append_size =
426+ min_t(u64, (u64)bdev_max_zone_append_sectors(bdev) << SECTOR_SHIFT,
427+ (u64)bdev_max_segments(bdev) << PAGE_SHIFT);
418428 if (!IS_ALIGNED(nr_sectors, zone_sectors))
419429 zone_info->nr_zones++;
420430
@@ -640,6 +650,7 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · 219 unmodified lines
640650 u64 zoned_devices = 0;
641651 u64 nr_devices = 0;
642652 u64 zone_size = 0;
653+ u64 max_zone_append_size = 0;
643654 const bool incompat_zoned = btrfs_fs_incompat(fs_info, ZONED);
644655 int ret = 0;
645656
@@ -674,6 +685,11 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · 28 unmodified lines
674685 ret = -EINVAL;
675686 goto out;
676687 }
688+ if (!max_zone_append_size ||
689+ (zone_info->max_zone_append_size &&
690+ zone_info->max_zone_append_size < max_zone_append_size))
691+ max_zone_append_size =
692+ zone_info->max_zone_append_size;
677693 }
678694 nr_devices++;
679695 }
@@ -723,7 +739,11 @@ int btrfs_check_zoned_mode(struct btrfs_fs_info *fs_info) · 43 unmodified lines
723739 }
724740
725741 fs_info->zone_size = zone_size;
742+ fs_info->max_zone_append_size = ALIGN_DOWN(max_zone_append_size,
743+ fs_info->sectorsize);
726744 fs_info->fs_devices->chunk_alloc_policy = BTRFS_CHUNK_ALLOC_ZONED;
745+ if (fs_info->max_zone_append_size < fs_info->max_extent_size)
746+ fs_info->max_extent_size = fs_info->max_zone_append_size;
727747
728748 /*
729749 * Check mount options here, because we might change fs_info->zoned
@@ -1829,6 +1849,7 @@ struct btrfs_device *btrfs_zoned_get_device(struct btrfs_fs_info *fs_info, · 1099 unmodified lines
18291849 bool btrfs_zone_activate(struct btrfs_block_group *block_group)
18301850 {
18311851 struct btrfs_fs_info *fs_info = block_group->fs_info;
1852+ struct btrfs_space_info *space_info = block_group->space_info;
18321853 struct map_lookup *map;
18331854 struct btrfs_device *device;
18341855 u64 physical;
@@ -1840,6 +1861,7 @@ bool btrfs_zone_activate(struct btrfs_block_group *block_group) · 5 unmodified lines
18401861
18411862 map = block_group->physical_map;
18421863
1864+ spin_lock(&space_info->lock);
18431865 spin_lock(&block_group->lock);
18441866 if (block_group->zone_is_active) {
18451867 ret = true;
@@ -1868,7 +1890,10 @@ bool btrfs_zone_activate(struct btrfs_block_group *block_group) · 22 unmodified lines
18681890
18691891 /* Successfully activated all the zones */
18701892 block_group->zone_is_active = 1;
1893+ space_info->active_total_bytes += block_group->length;
18711894 spin_unlock(&block_group->lock);
1895+ btrfs_try_granting_tickets(fs_info, space_info);
1896+ spin_unlock(&space_info->lock);
18721897
18731898 /* For the active block group list */
18741899 btrfs_get_block_group(block_group);
@@ -1881,6 +1906,7 @@ bool btrfs_zone_activate(struct btrfs_block_group *block_group) · 6 unmodified lines
18811906
18821907 out_unlock:
18831908 spin_unlock(&block_group->lock);
1909+ spin_unlock(&space_info->lock);
18841910 return ret;
18851911 }
18861912
@@ -1981,6 +2007,9 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ · 94 unmodified lines
19812007 /* For active_bg_list */
19822008 btrfs_put_block_group(block_group);
19832009
2010+ clear_bit(BTRFS_FS_NEED_ZONE_FINISH, &fs_info->flags);
2011+ wake_up_all(&fs_info->zone_finish_wait);
2012+
19842013 return 0;
19852014 }
19862015
@@ -2017,6 +2046,9 @@ bool btrfs_can_activate_zone(struct btrfs_fs_devices *fs_devices, u64 flags) · 30 unmodified lines
20172046 }
20182047 mutex_unlock(&fs_info->chunk_mutex);
20192048
2049+ if (!ret)
2050+ set_bit(BTRFS_FS_NEED_ZONE_FINISH, &fs_info->flags);
2051+
20202052 return ret;
20212053 }
20222054
@@ -2160,3 +2192,96 @@ out: · 137 unmodified lines
21602192 spin_unlock(&block_group->lock);
21612193 btrfs_put_block_group(block_group);
21622194 }
2195+
2196+int btrfs_zone_finish_one_bg(struct btrfs_fs_info *fs_info)
2197+{
2198+ struct btrfs_block_group *block_group;
2199+ struct btrfs_block_group *min_bg = NULL;
2200+ u64 min_avail = U64_MAX;
2201+ int ret;
2202+
2203+ spin_lock(&fs_info->zone_active_bgs_lock);
2204+ list_for_each_entry(block_group, &fs_info->zone_active_bgs,
2205+ active_bg_list) {
2206+ u64 avail;
2207+
2208+ spin_lock(&block_group->lock);
2209+ if (block_group->reserved ||
2210+ (block_group->flags & BTRFS_BLOCK_GROUP_SYSTEM)) {
2211+ spin_unlock(&block_group->lock);
2212+ continue;
2213+ }
2214+
2215+ avail = block_group->zone_capacity - block_group->alloc_offset;
2216+ if (min_avail > avail) {
2217+ if (min_bg)
2218+ btrfs_put_block_group(min_bg);
2219+ min_bg = block_group;
2220+ min_avail = avail;
2221+ btrfs_get_block_group(min_bg);
2222+ }
2223+ spin_unlock(&block_group->lock);
2224+ }
2225+ spin_unlock(&fs_info->zone_active_bgs_lock);
2226+
2227+ if (!min_bg)
2228+ return 0;
2229+
2230+ ret = btrfs_zone_finish(min_bg);
2231+ btrfs_put_block_group(min_bg);
2232+
2233+ return ret < 0 ? ret : 1;
2234+}
2235+
2236+int btrfs_zoned_activate_one_bg(struct btrfs_fs_info *fs_info,
2237+ struct btrfs_space_info *space_info,
2238+ bool do_finish)
2239+{
2240+ struct btrfs_block_group *bg;
2241+ int index;
2242+
2243+ if (!btrfs_is_zoned(fs_info) || (space_info->flags & BTRFS_BLOCK_GROUP_DATA))
2244+ return 0;
2245+
2246+ /* No more block groups to activate */
2247+ if (space_info->active_total_bytes == space_info->total_bytes)
2248+ return 0;
2249+
2250+ for (;;) {
2251+ int ret;
2252+ bool need_finish = false;
2253+
2254+ down_read(&space_info->groups_sem);
2255+ for (index = 0; index < BTRFS_NR_RAID_TYPES; index++) {
2256+ list_for_each_entry(bg, &space_info->block_groups[index],
2257+ list) {
2258+ if (!spin_trylock(&bg->lock))
2259+ continue;
2260+ if (btrfs_zoned_bg_is_full(bg) || bg->zone_is_active) {
2261+ spin_unlock(&bg->lock);
2262+ continue;
2263+ }
2264+ spin_unlock(&bg->lock);
2265+
2266+ if (btrfs_zone_activate(bg)) {
2267+ up_read(&space_info->groups_sem);
2268+ return 1;
2269+ }
2270+
2271+ need_finish = true;
2272+ }
2273+ }
2274+ up_read(&space_info->groups_sem);
2275+
2276+ if (!do_finish || !need_finish)
2277+ break;
2278+
2279+ ret = btrfs_zone_finish_one_bg(fs_info);
2280+ if (ret == 0)
2281+ break;
2282+ if (ret < 0)
2283+ return ret;
2284+ }
2285+
2286+ return 0;
2287+}
fs/btrfs/zoned.h+18−0
@@ -19,6 +19,7 @@ struct btrfs_zoned_device_info { · 18 unmodified lines
1919 */
2020 u64 zone_size;
2121 u8 zone_size_shift;
22+ u64 max_zone_append_size;
2223 u32 nr_zones;
2324 unsigned int max_active_zones;
2425 atomic_t active_zones_left;
@@ -79,6 +80,9 @@ void btrfs_free_zone_cache(struct btrfs_fs_info *fs_info); · 54 unmodified lines
7980 bool btrfs_zoned_should_reclaim(struct btrfs_fs_info *fs_info);
8081 void btrfs_zoned_release_data_reloc_bg(struct btrfs_fs_info *fs_info, u64 logical,
8182 u64 length);
83+int btrfs_zone_finish_one_bg(struct btrfs_fs_info *fs_info);
84+int btrfs_zoned_activate_one_bg(struct btrfs_fs_info *fs_info,
85+ struct btrfs_space_info *space_info, bool do_finish);
8286 #else /* CONFIG_BLK_DEV_ZONED */
8387 static inline int btrfs_get_dev_zone(struct btrfs_device *device, u64 pos,
8488 struct blk_zone *zone)
@@ -248,6 +252,20 @@ static inline bool btrfs_zoned_should_reclaim(struct btrfs_fs_info *fs_info) · 163 unmodified lines
248252
249253 static inline void btrfs_zoned_release_data_reloc_bg(struct btrfs_fs_info *fs_info,
250254 u64 logical, u64 length) { }
255+
256+static inline int btrfs_zone_finish_one_bg(struct btrfs_fs_info *fs_info)
257+{
258+ return 1;
259+}
260+
261+static inline int btrfs_zoned_activate_one_bg(struct btrfs_fs_info *fs_info,
262+ struct btrfs_space_info *space_info,
263+ bool do_finish)
264+{
265+ /* Consider all the block groups are active */
266+ return 0;
267+}
268+
251269 #endif
252270
253271 static inline bool btrfs_dev_is_sequential(struct btrfs_device *device, u64 pos)
fs/btrfs/zstd.c+13−20
@@ -403,7 +403,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, · 402 unmodified lines
403403
404404 /* map in the first page of input data */
405405 in_page = find_get_page(mapping, start >> PAGE_SHIFT);
406− workspace->in_buf.src = kmap(in_page);
406+ workspace->in_buf.src = kmap_local_page(in_page);
407407 workspace->in_buf.pos = 0;
408408 workspace->in_buf.size = min_t(size_t, len, PAGE_SIZE);
409409
@@ -415,7 +415,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, · 5 unmodified lines
415415 goto out;
416416 }
417417 pages[nr_pages++] = out_page;
418− workspace->out_buf.dst = kmap(out_page);
418+ workspace->out_buf.dst = page_address(out_page);
419419 workspace->out_buf.pos = 0;
420420 workspace->out_buf.size = min_t(size_t, max_out, PAGE_SIZE);
421421
@@ -450,9 +450,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, · 28 unmodified lines
450450 if (workspace->out_buf.pos == workspace->out_buf.size) {
451451 tot_out += PAGE_SIZE;
452452 max_out -= PAGE_SIZE;
453− kunmap(out_page);
454453 if (nr_pages == nr_dest_pages) {
455− out_page = NULL;
456454 ret = -E2BIG;
457455 goto out;
458456 }
@@ -462,7 +460,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, · 3 unmodified lines
462460 goto out;
463461 }
464462 pages[nr_pages++] = out_page;
465− workspace->out_buf.dst = kmap(out_page);
463+ workspace->out_buf.dst = page_address(out_page);
466464 workspace->out_buf.pos = 0;
467465 workspace->out_buf.size = min_t(size_t, max_out,
468466 PAGE_SIZE);
@@ -477,13 +475,12 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, · 8 unmodified lines
477475 /* Check if we need more input */
478476 if (workspace->in_buf.pos == workspace->in_buf.size) {
479477 tot_in += PAGE_SIZE;
480− kunmap(in_page);
478+ kunmap_local(workspace->in_buf.src);
481479 put_page(in_page);
482−
483480 start += PAGE_SIZE;
484481 len -= PAGE_SIZE;
485482 in_page = find_get_page(mapping, start >> PAGE_SHIFT);
486− workspace->in_buf.src = kmap(in_page);
483+ workspace->in_buf.src = kmap_local_page(in_page);
487484 workspace->in_buf.pos = 0;
488485 workspace->in_buf.size = min_t(size_t, len, PAGE_SIZE);
489486 }
@@ -510,9 +507,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, · 20 unmodified lines
510507
511508 tot_out += PAGE_SIZE;
512509 max_out -= PAGE_SIZE;
513− kunmap(out_page);
514510 if (nr_pages == nr_dest_pages) {
515− out_page = NULL;
516511 ret = -E2BIG;
517512 goto out;
518513 }
@@ -522,7 +517,7 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, · 3 unmodified lines
522517 goto out;
523518 }
524519 pages[nr_pages++] = out_page;
525− workspace->out_buf.dst = kmap(out_page);
520+ workspace->out_buf.dst = page_address(out_page);
526521 workspace->out_buf.pos = 0;
527522 workspace->out_buf.size = min_t(size_t, max_out, PAGE_SIZE);
528523 }
@@ -537,13 +532,10 @@ int zstd_compress_pages(struct list_head *ws, struct address_space *mapping, · 8 unmodified lines
537532 *total_out = tot_out;
538533 out:
539534 *out_pages = nr_pages;
540− /* Cleanup */
541− if (in_page) {
542− kunmap(in_page);
535+ if (workspace->in_buf.src) {
536+ kunmap_local(workspace->in_buf.src);
543537 put_page(in_page);
544538 }
545− if (out_page)
546− kunmap(out_page);
547539 return ret;
548540 }
549541
@@ -567,7 +559,7 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 17 unmodified lines
567559 goto done;
568560 }
569561
570− workspace->in_buf.src = kmap(pages_in[page_in_index]);
562+ workspace->in_buf.src = kmap_local_page(pages_in[page_in_index]);
571563 workspace->in_buf.pos = 0;
572564 workspace->in_buf.size = min_t(size_t, srclen, PAGE_SIZE);
573565
@@ -603,14 +595,15 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 29 unmodified lines
603595 break;
604596
605597 if (workspace->in_buf.pos == workspace->in_buf.size) {
606− kunmap(pages_in[page_in_index++]);
598+ kunmap_local(workspace->in_buf.src);
599+ page_in_index++;
607600 if (page_in_index >= total_pages_in) {
608601 workspace->in_buf.src = NULL;
609602 ret = -EIO;
610603 goto done;
611604 }
612605 srclen -= PAGE_SIZE;
613− workspace->in_buf.src = kmap(pages_in[page_in_index]);
606+ workspace->in_buf.src = kmap_local_page(pages_in[page_in_index]);
614607 workspace->in_buf.pos = 0;
615608 workspace->in_buf.size = min_t(size_t, srclen, PAGE_SIZE);
616609 }
@@ -619,7 +612,7 @@ int zstd_decompress_bio(struct list_head *ws, struct compressed_bio *cb) · 2 unmodified lines
619612 zero_fill_bio(cb->orig_bio);
620613 done:
621614 if (workspace->in_buf.src)
622− kunmap(pages_in[page_in_index]);
615+ kunmap_local(workspace->in_buf.src);
623616 return ret;
624617 }
625618
include/linux/blkdev.h+5−0
@@ -1167,6 +1167,11 @@ bdev_max_zone_append_sectors(struct block_device *bdev) · 1166 unmodified lines
11671167 return queue_max_zone_append_sectors(bdev_get_queue(bdev));
11681168 }
11691169
1170+static inline unsigned int bdev_max_segments(struct block_device *bdev)
1171+{
1172+ return queue_max_segments(bdev_get_queue(bdev));
1173+}
1174+
11701175 static inline unsigned queue_logical_block_size(const struct request_queue *q)
11711176 {
11721177 int retval = 512;
include/linux/highmem-internal.h+5−5
@@ -8,7 +8,7 @@ · 7 unmodified lines
88 #ifdef CONFIG_KMAP_LOCAL
99 void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot);
1010 void *__kmap_local_page_prot(struct page *page, pgprot_t prot);
11−void kunmap_local_indexed(void *vaddr);
11+void kunmap_local_indexed(const void *vaddr);
1212 void kmap_local_fork(struct task_struct *tsk);
1313 void __kmap_local_sched_out(void);
1414 void __kmap_local_sched_in(void);
@@ -89,7 +89,7 @@ static inline void *kmap_local_pfn(unsigned long pfn) · 74 unmodified lines
8989 return __kmap_local_pfn_prot(pfn, kmap_prot);
9090 }
9191
92−static inline void __kunmap_local(void *vaddr)
92+static inline void __kunmap_local(const void *vaddr)
9393 {
9494 kunmap_local_indexed(vaddr);
9595 }
@@ -121,7 +121,7 @@ static inline void *kmap_atomic_pfn(unsigned long pfn) · 25 unmodified lines
121121 return __kmap_local_pfn_prot(pfn, kmap_prot);
122122 }
123123
124−static inline void __kunmap_atomic(void *addr)
124+static inline void __kunmap_atomic(const void *addr)
125125 {
126126 kunmap_local_indexed(addr);
127127 pagefault_enable();
@@ -197,7 +197,7 @@ static inline void *kmap_local_pfn(unsigned long pfn) · 69 unmodified lines
197197 return kmap_local_page(pfn_to_page(pfn));
198198 }
199199
200−static inline void __kunmap_local(void *addr)
200+static inline void __kunmap_local(const void *addr)
201201 {
202202 #ifdef ARCH_HAS_FLUSH_ON_KUNMAP
203203 kunmap_flush_on_unmap(addr);
@@ -224,7 +224,7 @@ static inline void *kmap_atomic_pfn(unsigned long pfn) · 20 unmodified lines
224224 return kmap_atomic(pfn_to_page(pfn));
225225 }
226226
227−static inline void __kunmap_atomic(void *addr)
227+static inline void __kunmap_atomic(const void *addr)
228228 {
229229 #ifdef ARCH_HAS_FLUSH_ON_KUNMAP
230230 kunmap_flush_on_unmap(addr);
include/trace/events/btrfs.h+158−0
@@ -30,6 +30,8 @@ struct btrfs_qgroup; · 29 unmodified lines
3030 struct extent_io_tree;
3131 struct prelim_ref;
3232 struct btrfs_space_info;
33+struct btrfs_raid_bio;
34+struct raid56_bio_trace_info;
3335
3436 #define show_ref_type(type) \
3537 __print_symbolic(type, \
@@ -596,6 +598,70 @@ DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_put, · 560 unmodified lines
596598 TP_ARGS(inode, ordered)
597599 );
598600
601+DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_lookup,
602+
603+ TP_PROTO(const struct btrfs_inode *inode,
604+ const struct btrfs_ordered_extent *ordered),
605+
606+ TP_ARGS(inode, ordered)
607+);
608+
609+DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_lookup_range,
610+
611+ TP_PROTO(const struct btrfs_inode *inode,
612+ const struct btrfs_ordered_extent *ordered),
613+
614+ TP_ARGS(inode, ordered)
615+);
616+
617+DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_lookup_first_range,
618+
619+ TP_PROTO(const struct btrfs_inode *inode,
620+ const struct btrfs_ordered_extent *ordered),
621+
622+ TP_ARGS(inode, ordered)
623+);
624+
625+DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_lookup_for_logging,
626+
627+ TP_PROTO(const struct btrfs_inode *inode,
628+ const struct btrfs_ordered_extent *ordered),
629+
630+ TP_ARGS(inode, ordered)
631+);
632+
633+DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_lookup_first,
634+
635+ TP_PROTO(const struct btrfs_inode *inode,
636+ const struct btrfs_ordered_extent *ordered),
637+
638+ TP_ARGS(inode, ordered)
639+);
640+
641+DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_split,
642+
643+ TP_PROTO(const struct btrfs_inode *inode,
644+ const struct btrfs_ordered_extent *ordered),
645+
646+ TP_ARGS(inode, ordered)
647+);
648+
649+DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_dec_test_pending,
650+
651+ TP_PROTO(const struct btrfs_inode *inode,
652+ const struct btrfs_ordered_extent *ordered),
653+
654+ TP_ARGS(inode, ordered)
655+);
656+
657+DEFINE_EVENT(btrfs__ordered_extent, btrfs_ordered_extent_mark_finished,
658+
659+ TP_PROTO(const struct btrfs_inode *inode,
660+ const struct btrfs_ordered_extent *ordered),
661+
662+ TP_ARGS(inode, ordered)
663+);
664+
599665 DECLARE_EVENT_CLASS(btrfs__writepage,
600666
601667 TP_PROTO(const struct page *page, const struct inode *inode,
@@ -2258,6 +2324,98 @@ DEFINE_EVENT(btrfs__space_info_update, update_bytes_pinned, · 1656 unmodified lines
22582324 TP_ARGS(fs_info, sinfo, old, diff)
22592325 );
22602326
2327+DECLARE_EVENT_CLASS(btrfs_raid56_bio,
2328+
2329+ TP_PROTO(const struct btrfs_raid_bio *rbio,
2330+ const struct bio *bio,
2331+ const struct raid56_bio_trace_info *trace_info),
2332+
2333+ TP_ARGS(rbio, bio, trace_info),
2334+
2335+ TP_STRUCT__entry_btrfs(
2336+ __field( u64, full_stripe )
2337+ __field( u64, physical )
2338+ __field( u64, devid )
2339+ __field( u32, offset )
2340+ __field( u32, len )
2341+ __field( u8, opf )
2342+ __field( u8, total_stripes )
2343+ __field( u8, real_stripes )
2344+ __field( u8, nr_data )
2345+ __field( u8, stripe_nr )
2346+ ),
2347+
2348+ TP_fast_assign_btrfs(rbio->bioc->fs_info,
2349+ __entry->full_stripe = rbio->bioc->raid_map[0];
2350+ __entry->physical = bio->bi_iter.bi_sector << SECTOR_SHIFT;
2351+ __entry->len = bio->bi_iter.bi_size;
2352+ __entry->opf = bio_op(bio);
2353+ __entry->devid = trace_info->devid;
2354+ __entry->offset = trace_info->offset;
2355+ __entry->stripe_nr = trace_info->stripe_nr;
2356+ __entry->total_stripes = rbio->bioc->num_stripes;
2357+ __entry->real_stripes = rbio->real_stripes;
2358+ __entry->nr_data = rbio->nr_data;
2359+ ),
2360+ /*
2361+ * For type output, we need to output things like "DATA1"
2362+ * (the first data stripe), "DATA2" (the second data stripe),
2363+ * "PQ1" (P stripe),"PQ2" (Q stripe), "REPLACE0" (replace target device).
2364+ */
2365+ TP_printk_btrfs(
2366+"full_stripe=%llu devid=%lld type=%s%d offset=%d opf=0x%x physical=%llu len=%u",
2367+ __entry->full_stripe, __entry->devid,
2368+ (__entry->stripe_nr < __entry->nr_data) ? "DATA" :
2369+ ((__entry->stripe_nr < __entry->real_stripes) ? "PQ" :
2370+ "REPLACE"),
2371+ (__entry->stripe_nr < __entry->nr_data) ?
2372+ (__entry->stripe_nr + 1) :
2373+ ((__entry->stripe_nr < __entry->real_stripes) ?
2374+ (__entry->stripe_nr - __entry->nr_data + 1) : 0),
2375+ __entry->offset, __entry->opf, __entry->physical, __entry->len)
2376+);
2377+
2378+DEFINE_EVENT(btrfs_raid56_bio, raid56_read_partial,
2379+ TP_PROTO(const struct btrfs_raid_bio *rbio,
2380+ const struct bio *bio,
2381+ const struct raid56_bio_trace_info *trace_info),
2382+
2383+ TP_ARGS(rbio, bio, trace_info)
2384+);
2385+
2386+DEFINE_EVENT(btrfs_raid56_bio, raid56_write_stripe,
2387+ TP_PROTO(const struct btrfs_raid_bio *rbio,
2388+ const struct bio *bio,
2389+ const struct raid56_bio_trace_info *trace_info),
2390+
2391+ TP_ARGS(rbio, bio, trace_info)
2392+);
2393+
2394+
2395+DEFINE_EVENT(btrfs_raid56_bio, raid56_scrub_write_stripe,
2396+ TP_PROTO(const struct btrfs_raid_bio *rbio,
2397+ const struct bio *bio,
2398+ const struct raid56_bio_trace_info *trace_info),
2399+
2400+ TP_ARGS(rbio, bio, trace_info)
2401+);
2402+
2403+DEFINE_EVENT(btrfs_raid56_bio, raid56_scrub_read,
2404+ TP_PROTO(const struct btrfs_raid_bio *rbio,
2405+ const struct bio *bio,
2406+ const struct raid56_bio_trace_info *trace_info),
2407+
2408+ TP_ARGS(rbio, bio, trace_info)
2409+);
2410+
2411+DEFINE_EVENT(btrfs_raid56_bio, raid56_scrub_read_recover,
2412+ TP_PROTO(const struct btrfs_raid_bio *rbio,
2413+ const struct bio *bio,
2414+ const struct raid56_bio_trace_info *trace_info),
2415+
2416+ TP_ARGS(rbio, bio, trace_info)
2417+);
2418+
22612419 #endif /* _TRACE_BTRFS_H */
22622420
22632421 /* This part must be outside protection */
include/uapi/linux/btrfs.h+9−1
@@ -777,11 +777,19 @@ struct btrfs_ioctl_received_subvol_args { · 776 unmodified lines
777777 */
778778 #define BTRFS_SEND_FLAG_VERSION 0x8
779779
780+/*
781+ * Send compressed data using the ENCODED_WRITE command instead of decompressing
782+ * the data and sending it with the WRITE command. This requires protocol
783+ * version >= 2.
784+ */
785+#define BTRFS_SEND_FLAG_COMPRESSED 0x10
786+
780787 #define BTRFS_SEND_FLAG_MASK \
781788 (BTRFS_SEND_FLAG_NO_FILE_DATA | \
782789 BTRFS_SEND_FLAG_OMIT_STREAM_HEADER | \
783790 BTRFS_SEND_FLAG_OMIT_END_CMD | \
784− BTRFS_SEND_FLAG_VERSION)
791+ BTRFS_SEND_FLAG_VERSION | \
792+ BTRFS_SEND_FLAG_COMPRESSED)
785793
786794 struct btrfs_ioctl_send_args {
787795 __s64 send_fd; /* in */
mm/highmem.c+1−1
@@ -561,7 +561,7 @@ void *__kmap_local_page_prot(struct page *page, pgprot_t prot) · 560 unmodified lines
561561 }
562562 EXPORT_SYMBOL(__kmap_local_page_prot);
563563
564−void kunmap_local_indexed(void *vaddr)
564+void kunmap_local_indexed(const void *vaddr)
565565 {
566566 unsigned long addr = (unsigned long) vaddr & PAGE_MASK;
567567 pte_t *kmap_pte;

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t