History

Merge tag 'erofs-for-5.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs

Pull erofs updates from Gao Xiang: "There are some new features available for this cycle. Firstly, EROFS LZMA algorithm support, specifically called MicroLZMA, is available as an option for embedded devices, LiveCDs and/or as the secondary auxiliary compression algorithm besides the primary algorithm in one file. In order to better support the LZMA fixed-sized output compression, especially for 4KiB pcluster size (which has lowest memory pressure thus useful for memory-sensitive scenarios), Lasse introduced a new LZMA header/container format called MicroLZMA to minimize the original LZMA1 header (for example, we don't need to waste 4-byte dictionary size and another 8-byte uncompressed size, which can be calculated by fs directly, for each pcluster) and enable EROFS fixed-sized output compression. Note that MicroLZMA can also be later used by other things in addition to EROFS too where wasting minimal amount of space for headers is important and it can be only compiled by enabling XZ_DEC_MICROLZMA. MicroLZMA has been supported by the latest upstream XZ embedded [1] & XZ utils [2], apply the latest related XZ embedded upstream patches by the XZ author Lasse here. Secondly, multiple device is also supported in this cycle, which is designed for multi-layer container images. By working together with inter-layer data deduplication and compression, we can achieve the next high-performance container image solution. Our team will announce the new Nydus container image service [3] implementation with new RAFS v6 (EROFS-compatible) format in Open Source Summit 2021 China [4] soon. Besides, the secondary compression head support and readmore decompression strategy are also included in this cycle. There are also some minor bugfixes and cleanups, as always. Summary: - support multiple devices for multi-layer container images; - support the secondary compression head; - support readmore decompression strategy; - support new LZMA algorithm (specifically called MicroLZMA); - some bugfixes & cleanups" * tag 'erofs-for-5.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs: erofs: don't trigger WARN() when decompression fails erofs: get rid of ->lru usage erofs: lzma compression support erofs: rename some generic methods in decompressor lib/xz, lib/decompress_unxz.c: Fix spelling in comments lib/xz: Add MicroLZMA decoder lib/xz: Move s->lzma.len = 0 initialization to lzma_reset() lib/xz: Validate the value before assigning it to an enum variable lib/xz: Avoid overlapping memcpy() with invalid input with in-place decompression erofs: introduce readmore decompression strategy erofs: introduce the secondary compression head erofs: get compression algorithms directly on mapping erofs: add multiple device support erofs: decouple basic mount options from fs_context erofs: remove the fast path of per-CPU buffer decompression

Linus Torvalds committed 5 years ago

Commit 67a135b2 parents cd3e8ea a0961f325 files · +1281 −320Browse files

Unified · Split

25 files
  1. Documentation/filesystems/erofs.rst +8 −4
  2. fs/erofs/Kconfig +31 −9
  3. fs/erofs/Makefile +1 −0
  4. fs/erofs/compress.h +19 −9
  5. fs/erofs/data.c +60 −13
  6. fs/erofs/decompressor.c +44 −95
  7. fs/erofs/decompressor_lzma.c +290 −0
  8. fs/erofs/erofs_fs.h +50 −23
  9. fs/erofs/inode.c +1 −1
  10. fs/erofs/internal.h +94 −11
  11. fs/erofs/pcpubuf.c +3 −3
  12. fs/erofs/super.c +188 −43
  13. fs/erofs/utils.c +14 −5
  14. fs/erofs/xattr.c +2 −2
  15. fs/erofs/zdata.c +120 −55
  16. fs/erofs/zdata.h +0 −7
  17. fs/erofs/zmap.c +42 −23
  18. include/linux/xz.h +106 −0
  19. include/trace/events/erofs.h +1 −1
  20. lib/decompress_unxz.c +5 −5
  21. lib/xz/Kconfig +13 −0
  22. lib/xz/xz_dec_lzma2.c +175 −7
  23. lib/xz/xz_dec_stream.c +3 −3
  24. lib/xz/xz_dec_syms.c +8 −1
  25. lib/xz/xz_private.h +3 −0
Documentation/filesystems/erofs.rst+8−4
@@ -19,9 +19,10 @@ It is designed as a better filesystem solution for the following scenarios: · 18 unmodified lines
19 immutable and bit-for-bit identical to the official golden image for19 immutable and bit-for-bit identical to the official golden image for
20 their releases due to security and other considerations and20 their releases due to security and other considerations and
21 21
22 - hope to save some extra storage space with guaranteed end-to-end performance22 - hope to minimize extra storage space with guaranteed end-to-end performance
23 by using reduced metadata and transparent file compression, especially23 by using compact layout, transparent file compression and direct access,
24 for those embedded devices with limited memory (ex, smartphone);24 especially for those embedded devices with limited memory and high-density
25 hosts with numerous containers;
25 26
26Here is the main features of EROFS:27Here is the main features of EROFS:
27 28
@@ -51,7 +52,9 @@ Here is the main features of EROFS: · 23 unmodified lines
51 - Support POSIX.1e ACLs by using xattrs;52 - Support POSIX.1e ACLs by using xattrs;
52 53
53 - Support transparent data compression as an option:54 - Support transparent data compression as an option:
54 LZ4 algorithm with the fixed-sized output compression for high performance.55 LZ4 algorithm with the fixed-sized output compression for high performance;
56
57 - Multiple device support for multi-layer container images.
55 58
56The following git tree provides the file system user-space tools under59The following git tree provides the file system user-space tools under
57development (ex, formatting tool mkfs.erofs):60development (ex, formatting tool mkfs.erofs):
@@ -87,6 +90,7 @@ cache_strategy=%s Select a strategy for cached decompression from now on: · 29 unmodified lines
87dax={always,never} Use direct access (no page cache). See90dax={always,never} Use direct access (no page cache). See
88 Documentation/filesystems/dax.rst.91 Documentation/filesystems/dax.rst.
89dax A legacy option which is an alias for ``dax=always``.92dax A legacy option which is an alias for ``dax=always``.
93device=%s Specify a path to an extra device to be used together.
90=================== =========================================================94=================== =========================================================
91 95
92On-disk details96On-disk details
fs/erofs/Kconfig+31−9
@@ -6,16 +6,22 @@ config EROFS_FS · 5 unmodified lines
6 select FS_IOMAP6 select FS_IOMAP
7 select LIBCRC32C7 select LIBCRC32C
8 help8 help
9 EROFS (Enhanced Read-Only File System) is a lightweight9 EROFS (Enhanced Read-Only File System) is a lightweight read-only
10 read-only file system with modern designs (eg. page-sized10 file system with modern designs (e.g. no buffer heads, inline
11 blocks, inline xattrs/data, etc.) for scenarios which need11 xattrs/data, chunk-based deduplication, multiple devices, etc.) for
12 high-performance read-only requirements, e.g. Android OS12 scenarios which need high-performance read-only solutions, e.g.
13 for mobile phones and LIVECDs.13 smartphones with Android OS, LiveCDs and high-density hosts with
14 numerous containers;
14 15
15 It also provides fixed-sized output compression support,16 It also provides fixed-sized output compression support in order to
16 which improves storage density, keeps relatively higher17 improve storage density as well as keep relatively higher compression
17 compression ratios, which is more useful to achieve high18 ratios and implements in-place decompression to reuse the file page
18 performance for embedded devices with limited memory.19 for compressed data temporarily with proper strategies, which is
20 quite useful to ensure guaranteed end-to-end runtime decompression
21 performance under extremely memory pressure without extra cost.
22
23 See the documentation at <file:Documentation/filesystems/erofs.rst>
24 for more details.
19 25
20 If unsure, say N.26 If unsure, say N.
21 27
@@ -76,3 +82,19 @@ config EROFS_FS_ZIP · 54 unmodified lines
76 Enable fixed-sized output compression for EROFS.82 Enable fixed-sized output compression for EROFS.
77 83
78 If you don't want to enable compression feature, say N.84 If you don't want to enable compression feature, say N.
85
86config EROFS_FS_ZIP_LZMA
87 bool "EROFS LZMA compressed data support"
88 depends on EROFS_FS_ZIP
89 select XZ_DEC
90 select XZ_DEC_MICROLZMA
91 help
92 Saying Y here includes support for reading EROFS file systems
93 containing LZMA compressed data, specifically called microLZMA. it
94 gives better compression ratios than the LZ4 algorithm, at the
95 expense of more CPU overhead.
96
97 LZMA support is an experimental feature for now and so most file
98 systems will be readable without selecting this option.
99
100 If unsure, say N.
fs/erofs/Makefile+1−0
@@ -4,3 +4,4 @@ obj-$(CONFIG_EROFS_FS) += erofs.o · 3 unmodified lines
4erofs-objs := super.o inode.o data.o namei.o dir.o utils.o pcpubuf.o4erofs-objs := super.o inode.o data.o namei.o dir.o utils.o pcpubuf.o
5erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o5erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o
6erofs-$(CONFIG_EROFS_FS_ZIP) += decompressor.o zmap.o zdata.o6erofs-$(CONFIG_EROFS_FS_ZIP) += decompressor.o zmap.o zdata.o
7erofs-$(CONFIG_EROFS_FS_ZIP_LZMA) += decompressor_lzma.o
fs/erofs/compress.h+19−9
@@ -8,11 +8,6 @@ · 7 unmodified lines
8 8
9#include "internal.h"9#include "internal.h"
10 10
11enum {
12 Z_EROFS_COMPRESSION_SHIFTED = Z_EROFS_COMPRESSION_MAX,
13 Z_EROFS_COMPRESSION_RUNTIME_MAX
14};
15
16struct z_erofs_decompress_req {11struct z_erofs_decompress_req {
17 struct super_block *sb;12 struct super_block *sb;
18 struct page **in, **out;13 struct page **in, **out;
@@ -25,6 +20,12 @@ struct z_erofs_decompress_req { · 6 unmodified lines
25 bool inplace_io, partial_decoding;20 bool inplace_io, partial_decoding;
26};21};
27 22
23struct z_erofs_decompressor {
24 int (*decompress)(struct z_erofs_decompress_req *rq,
25 struct page **pagepool);
26 char *name;
27};
28
28/* some special page->private (unsigned long, see below) */29/* some special page->private (unsigned long, see below) */
29#define Z_EROFS_SHORTLIVED_PAGE (-1UL << 2)30#define Z_EROFS_SHORTLIVED_PAGE (-1UL << 2)
30#define Z_EROFS_PREALLOCATED_PAGE (-2UL << 2)31#define Z_EROFS_PREALLOCATED_PAGE (-2UL << 2)
@@ -63,7 +64,7 @@ static inline bool z_erofs_is_shortlived_page(struct page *page) · 32 unmodified lines
63 return true;64 return true;
64}65}
65 66
66static inline bool z_erofs_put_shortlivedpage(struct list_head *pagepool,67static inline bool z_erofs_put_shortlivedpage(struct page **pagepool,
67 struct page *page)68 struct page *page)
68{69{
69 if (!z_erofs_is_shortlived_page(page))70 if (!z_erofs_is_shortlived_page(page))
@@ -74,13 +75,22 @@ static inline bool z_erofs_put_shortlivedpage(struct list_head *pagepool, · 4 unmodified lines
74 put_page(page);75 put_page(page);
75 } else {76 } else {
76 /* follow the pcluster rule above. */77 /* follow the pcluster rule above. */
77 set_page_private(page, 0);78 erofs_pagepool_add(pagepool, page);
78 list_add(&page->lru, pagepool);
79 }79 }
80 return true;80 return true;
81}81}
82 82
83#define MNGD_MAPPING(sbi) ((sbi)->managed_cache->i_mapping)
84static inline bool erofs_page_is_managed(const struct erofs_sb_info *sbi,
85 struct page *page)
86{
87 return page->mapping == MNGD_MAPPING(sbi);
88}
89
83int z_erofs_decompress(struct z_erofs_decompress_req *rq,90int z_erofs_decompress(struct z_erofs_decompress_req *rq,
84 struct list_head *pagepool);91 struct page **pagepool);
85 92
93/* prototypes for specific algorithms */
94int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq,
95 struct page **pagepool);
86#endif96#endif
fs/erofs/data.c+60−13
@@ -89,6 +89,7 @@ static int erofs_map_blocks(struct inode *inode, · 88 unmodified lines
89 erofs_off_t pos;89 erofs_off_t pos;
90 int err = 0;90 int err = 0;
91 91
92 map->m_deviceid = 0;
92 if (map->m_la >= inode->i_size) {93 if (map->m_la >= inode->i_size) {
93 /* leave out-of-bound access unmapped */94 /* leave out-of-bound access unmapped */
94 map->m_flags = 0;95 map->m_flags = 0;
@@ -135,14 +136,8 @@ static int erofs_map_blocks(struct inode *inode, · 40 unmodified lines
135 map->m_flags = 0;136 map->m_flags = 0;
136 break;137 break;
137 default:138 default:
138 /* only one device is supported for now */139 map->m_deviceid = le16_to_cpu(idx->device_id) &
139 if (idx->device_id) {140 EROFS_SB(sb)->device_id_mask;
140 erofs_err(sb, "invalid device id %u @ %llu for nid %llu",
141 le16_to_cpu(idx->device_id),
142 chunknr, vi->nid);
143 err = -EFSCORRUPTED;
144 goto out_unlock;
145 }
146 map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));141 map->m_pa = blknr_to_addr(le32_to_cpu(idx->blkaddr));
147 map->m_flags = EROFS_MAP_MAPPED;142 map->m_flags = EROFS_MAP_MAPPED;
148 break;143 break;
@@ -155,11 +150,55 @@ out: · 6 unmodified lines
155 return err;150 return err;
156}151}
157 152
153int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *map)
154{
155 struct erofs_dev_context *devs = EROFS_SB(sb)->devs;
156 struct erofs_device_info *dif;
157 int id;
158
159 /* primary device by default */
160 map->m_bdev = sb->s_bdev;
161 map->m_daxdev = EROFS_SB(sb)->dax_dev;
162
163 if (map->m_deviceid) {
164 down_read(&devs->rwsem);
165 dif = idr_find(&devs->tree, map->m_deviceid - 1);
166 if (!dif) {
167 up_read(&devs->rwsem);
168 return -ENODEV;
169 }
170 map->m_bdev = dif->bdev;
171 map->m_daxdev = dif->dax_dev;
172 up_read(&devs->rwsem);
173 } else if (devs->extra_devices) {
174 down_read(&devs->rwsem);
175 idr_for_each_entry(&devs->tree, dif, id) {
176 erofs_off_t startoff, length;
177
178 if (!dif->mapped_blkaddr)
179 continue;
180 startoff = blknr_to_addr(dif->mapped_blkaddr);
181 length = blknr_to_addr(dif->blocks);
182
183 if (map->m_pa >= startoff &&
184 map->m_pa < startoff + length) {
185 map->m_pa -= startoff;
186 map->m_bdev = dif->bdev;
187 map->m_daxdev = dif->dax_dev;
188 break;
189 }
190 }
191 up_read(&devs->rwsem);
192 }
193 return 0;
194}
195
158static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,196static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length,
159 unsigned int flags, struct iomap *iomap, struct iomap *srcmap)197 unsigned int flags, struct iomap *iomap, struct iomap *srcmap)
160{198{
161 int ret;199 int ret;
162 struct erofs_map_blocks map;200 struct erofs_map_blocks map;
201 struct erofs_map_dev mdev;
163 202
164 map.m_la = offset;203 map.m_la = offset;
165 map.m_llen = length;204 map.m_llen = length;
@@ -168,8 +207,16 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length, · 2 unmodified lines
168 if (ret < 0)207 if (ret < 0)
169 return ret;208 return ret;
170 209
171 iomap->bdev = inode->i_sb->s_bdev;210 mdev = (struct erofs_map_dev) {
172 iomap->dax_dev = EROFS_I_SB(inode)->dax_dev;211 .m_deviceid = map.m_deviceid,
212 .m_pa = map.m_pa,
213 };
214 ret = erofs_map_dev(inode->i_sb, &mdev);
215 if (ret)
216 return ret;
217
218 iomap->bdev = mdev.m_bdev;
219 iomap->dax_dev = mdev.m_daxdev;
173 iomap->offset = map.m_la;220 iomap->offset = map.m_la;
174 iomap->length = map.m_llen;221 iomap->length = map.m_llen;
175 iomap->flags = 0;222 iomap->flags = 0;
@@ -188,15 +235,15 @@ static int erofs_iomap_begin(struct inode *inode, loff_t offset, loff_t length, · 12 unmodified lines
188 235
189 iomap->type = IOMAP_INLINE;236 iomap->type = IOMAP_INLINE;
190 ipage = erofs_get_meta_page(inode->i_sb,237 ipage = erofs_get_meta_page(inode->i_sb,
191 erofs_blknr(map.m_pa));238 erofs_blknr(mdev.m_pa));
192 if (IS_ERR(ipage))239 if (IS_ERR(ipage))
193 return PTR_ERR(ipage);240 return PTR_ERR(ipage);
194 iomap->inline_data = page_address(ipage) +241 iomap->inline_data = page_address(ipage) +
195 erofs_blkoff(map.m_pa);242 erofs_blkoff(mdev.m_pa);
196 iomap->private = ipage;243 iomap->private = ipage;
197 } else {244 } else {
198 iomap->type = IOMAP_MAPPED;245 iomap->type = IOMAP_MAPPED;
199 iomap->addr = map.m_pa;246 iomap->addr = mdev.m_pa;
200 }247 }
201 return 0;248 return 0;
202}249}
fs/erofs/decompressor.c+44−95
@@ -16,17 +16,6 @@ · 15 unmodified lines
16#define LZ4_DECOMPRESS_INPLACE_MARGIN(srcsize) (((srcsize) >> 8) + 32)16#define LZ4_DECOMPRESS_INPLACE_MARGIN(srcsize) (((srcsize) >> 8) + 32)
17#endif17#endif
18 18
19struct z_erofs_decompressor {
20 /*
21 * if destpages have sparsed pages, fill them with bounce pages.
22 * it also check whether destpages indicate continuous physical memory.
23 */
24 int (*prepare_destpages)(struct z_erofs_decompress_req *rq,
25 struct list_head *pagepool);
26 int (*decompress)(struct z_erofs_decompress_req *rq, u8 *out);
27 char *name;
28};
29
30int z_erofs_load_lz4_config(struct super_block *sb,19int z_erofs_load_lz4_config(struct super_block *sb,
31 struct erofs_super_block *dsb,20 struct erofs_super_block *dsb,
32 struct z_erofs_lz4_cfgs *lz4, int size)21 struct z_erofs_lz4_cfgs *lz4, int size)
@@ -63,8 +52,12 @@ int z_erofs_load_lz4_config(struct super_block *sb, · 30 unmodified lines
63 return erofs_pcpubuf_growsize(sbi->lz4.max_pclusterblks);52 return erofs_pcpubuf_growsize(sbi->lz4.max_pclusterblks);
64}53}
65 54
66static int z_erofs_lz4_prepare_destpages(struct z_erofs_decompress_req *rq,55/*
67 struct list_head *pagepool)56 * Fill all gaps with bounce pages if it's a sparse page list. Also check if
57 * all physical pages are consecutive, which can be seen for moderate CR.
58 */
59static int z_erofs_lz4_prepare_dstpages(struct z_erofs_decompress_req *rq,
60 struct page **pagepool)
68{61{
69 const unsigned int nr =62 const unsigned int nr =
70 PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT;63 PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT;
@@ -119,7 +112,7 @@ static int z_erofs_lz4_prepare_destpages(struct z_erofs_decompress_req *rq, · 48 unmodified lines
119 return kaddr ? 1 : 0;112 return kaddr ? 1 : 0;
120}113}
121 114
122static void *z_erofs_handle_inplace_io(struct z_erofs_decompress_req *rq,115static void *z_erofs_lz4_handle_inplace_io(struct z_erofs_decompress_req *rq,
123 void *inpage, unsigned int *inputmargin, int *maptype,116 void *inpage, unsigned int *inputmargin, int *maptype,
124 bool support_0padding)117 bool support_0padding)
125{118{
@@ -189,7 +182,8 @@ docopy: · 63 unmodified lines
189 return src;182 return src;
190}183}
191 184
192static int z_erofs_lz4_decompress(struct z_erofs_decompress_req *rq, u8 *out)185static int z_erofs_lz4_decompress_mem(struct z_erofs_decompress_req *rq,
186 u8 *out)
193{187{
194 unsigned int inputmargin;188 unsigned int inputmargin;
195 u8 *headpage, *src;189 u8 *headpage, *src;
@@ -216,8 +210,8 @@ static int z_erofs_lz4_decompress(struct z_erofs_decompress_req *rq, u8 *out) · 20 unmodified lines
216 }210 }
217 211
218 rq->inputsize -= inputmargin;212 rq->inputsize -= inputmargin;
219 src = z_erofs_handle_inplace_io(rq, headpage, &inputmargin, &maptype,213 src = z_erofs_lz4_handle_inplace_io(rq, headpage, &inputmargin,
220 support_0padding);214 &maptype, support_0padding);
221 if (IS_ERR(src))215 if (IS_ERR(src))
222 return PTR_ERR(src);216 return PTR_ERR(src);
223 217
@@ -233,7 +227,6 @@ static int z_erofs_lz4_decompress(struct z_erofs_decompress_req *rq, u8 *out) · 9 unmodified lines
233 erofs_err(rq->sb, "failed to decompress %d in[%u, %u] out[%u]",227 erofs_err(rq->sb, "failed to decompress %d in[%u, %u] out[%u]",
234 ret, rq->inputsize, inputmargin, rq->outputsize);228 ret, rq->inputsize, inputmargin, rq->outputsize);
235 229
236 WARN_ON(1);
237 print_hex_dump(KERN_DEBUG, "[ in]: ", DUMP_PREFIX_OFFSET,230 print_hex_dump(KERN_DEBUG, "[ in]: ", DUMP_PREFIX_OFFSET,
238 16, 1, src + inputmargin, rq->inputsize, true);231 16, 1, src + inputmargin, rq->inputsize, true);
239 print_hex_dump(KERN_DEBUG, "[out]: ", DUMP_PREFIX_OFFSET,232 print_hex_dump(KERN_DEBUG, "[out]: ", DUMP_PREFIX_OFFSET,
@@ -242,6 +235,8 @@ static int z_erofs_lz4_decompress(struct z_erofs_decompress_req *rq, u8 *out) · 2 unmodified lines
242 if (ret >= 0)235 if (ret >= 0)
243 memset(out + ret, 0, rq->outputsize - ret);236 memset(out + ret, 0, rq->outputsize - ret);
244 ret = -EIO;237 ret = -EIO;
238 } else {
239 ret = 0;
245 }240 }
246 241
247 if (maptype == 0) {242 if (maptype == 0) {
@@ -257,86 +252,25 @@ static int z_erofs_lz4_decompress(struct z_erofs_decompress_req *rq, u8 *out) · 9 unmodified lines
257 return ret;252 return ret;
258}253}
259 254
260static struct z_erofs_decompressor decompressors[] = {255static int z_erofs_lz4_decompress(struct z_erofs_decompress_req *rq,
261 [Z_EROFS_COMPRESSION_SHIFTED] = {256 struct page **pagepool)
262 .name = "shifted"
263 },
264 [Z_EROFS_COMPRESSION_LZ4] = {
265 .prepare_destpages = z_erofs_lz4_prepare_destpages,
266 .decompress = z_erofs_lz4_decompress,
267 .name = "lz4"
268 },
269};
270
271static void copy_from_pcpubuf(struct page **out, const char *dst,
272 unsigned short pageofs_out,
273 unsigned int outputsize)
274{
275 const char *end = dst + outputsize;
276 const unsigned int righthalf = PAGE_SIZE - pageofs_out;
277 const char *cur = dst - pageofs_out;
278
279 while (cur < end) {
280 struct page *const page = *out++;
281
282 if (page) {
283 char *buf = kmap_atomic(page);
284
285 if (cur >= dst) {
286 memcpy(buf, cur, min_t(uint, PAGE_SIZE,
287 end - cur));
288 } else {
289 memcpy(buf + pageofs_out, cur + pageofs_out,
290 min_t(uint, righthalf, end - cur));
291 }
292 kunmap_atomic(buf);
293 }
294 cur += PAGE_SIZE;
295 }
296}
297
298static int z_erofs_decompress_generic(struct z_erofs_decompress_req *rq,
299 struct list_head *pagepool)
300{257{
301 const unsigned int nrpages_out =258 const unsigned int nrpages_out =
302 PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT;259 PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT;
303 const struct z_erofs_decompressor *alg = decompressors + rq->alg;
304 unsigned int dst_maptype;260 unsigned int dst_maptype;
305 void *dst;261 void *dst;
306 int ret;262 int ret;
307 263
308 /* two optimized fast paths only for non bigpcluster cases yet */264 /* one optimized fast path only for non bigpcluster cases yet */
309 if (rq->inputsize <= PAGE_SIZE) {265 if (rq->inputsize <= PAGE_SIZE && nrpages_out == 1 && !rq->inplace_io) {
310 if (nrpages_out == 1 && !rq->inplace_io) {266 DBG_BUGON(!*rq->out);
311 DBG_BUGON(!*rq->out);267 dst = kmap_atomic(*rq->out);
312 dst = kmap_atomic(*rq->out);268 dst_maptype = 0;
313 dst_maptype = 0;269 goto dstmap_out;
314 goto dstmap_out;
315 }
316
317 /*
318 * For the case of small output size (especially much less
319 * than PAGE_SIZE), memcpy the decompressed data rather than
320 * compressed data is preferred.
321 */
322 if (rq->outputsize <= PAGE_SIZE * 7 / 8) {
323 dst = erofs_get_pcpubuf(1);
324 if (IS_ERR(dst))
325 return PTR_ERR(dst);
326
327 rq->inplace_io = false;
328 ret = alg->decompress(rq, dst);
329 if (!ret)
330 copy_from_pcpubuf(rq->out, dst, rq->pageofs_out,
331 rq->outputsize);
332
333 erofs_put_pcpubuf(dst);
334 return ret;
335 }
336 }270 }
337 271
338 /* general decoding path which can be used for all cases */272 /* general decoding path which can be used for all cases */
339 ret = alg->prepare_destpages(rq, pagepool);273 ret = z_erofs_lz4_prepare_dstpages(rq, pagepool);
340 if (ret < 0)274 if (ret < 0)
341 return ret;275 return ret;
342 if (ret) {276 if (ret) {
@@ -351,7 +285,7 @@ static int z_erofs_decompress_generic(struct z_erofs_decompress_req *rq, · 8 unmodified lines
351 dst_maptype = 2;285 dst_maptype = 2;
352 286
353dstmap_out:287dstmap_out:
354 ret = alg->decompress(rq, dst + rq->pageofs_out);288 ret = z_erofs_lz4_decompress_mem(rq, dst + rq->pageofs_out);
355 289
356 if (!dst_maptype)290 if (!dst_maptype)
357 kunmap_atomic(dst);291 kunmap_atomic(dst);
@@ -360,8 +294,8 @@ dstmap_out: · 2 unmodified lines
360 return ret;294 return ret;
361}295}
362 296
363static int z_erofs_shifted_transform(const struct z_erofs_decompress_req *rq,297static int z_erofs_shifted_transform(struct z_erofs_decompress_req *rq,
364 struct list_head *pagepool)298 struct page **pagepool)
365{299{
366 const unsigned int nrpages_out =300 const unsigned int nrpages_out =
367 PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT;301 PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT;
@@ -399,10 +333,25 @@ static int z_erofs_shifted_transform(const struct z_erofs_decompress_req *rq, · 31 unmodified lines
399 return 0;333 return 0;
400}334}
401 335
336static struct z_erofs_decompressor decompressors[] = {
337 [Z_EROFS_COMPRESSION_SHIFTED] = {
338 .decompress = z_erofs_shifted_transform,
339 .name = "shifted"
340 },
341 [Z_EROFS_COMPRESSION_LZ4] = {
342 .decompress = z_erofs_lz4_decompress,
343 .name = "lz4"
344 },
345#ifdef CONFIG_EROFS_FS_ZIP_LZMA
346 [Z_EROFS_COMPRESSION_LZMA] = {
347 .decompress = z_erofs_lzma_decompress,
348 .name = "lzma"
349 },
350#endif
351};
352
402int z_erofs_decompress(struct z_erofs_decompress_req *rq,353int z_erofs_decompress(struct z_erofs_decompress_req *rq,
403 struct list_head *pagepool)354 struct page **pagepool)
404{355{
405 if (rq->alg == Z_EROFS_COMPRESSION_SHIFTED)356 return decompressors[rq->alg].decompress(rq, pagepool);
406 return z_erofs_shifted_transform(rq, pagepool);
407 return z_erofs_decompress_generic(rq, pagepool);
408}357}
fs/erofs/decompressor_lzma.c+290−0
@@ -0,0 +1,290 @@
1// SPDX-License-Identifier: GPL-2.0-or-later
2#include <linux/xz.h>
3#include <linux/module.h>
4#include "compress.h"
5
6struct z_erofs_lzma {
7 struct z_erofs_lzma *next;
8 struct xz_dec_microlzma *state;
9 struct xz_buf buf;
10 u8 bounce[PAGE_SIZE];
11};
12
13/* considering the LZMA performance, no need to use a lockless list for now */
14static DEFINE_SPINLOCK(z_erofs_lzma_lock);
15static unsigned int z_erofs_lzma_max_dictsize;
16static unsigned int z_erofs_lzma_nstrms, z_erofs_lzma_avail_strms;
17static struct z_erofs_lzma *z_erofs_lzma_head;
18static DECLARE_WAIT_QUEUE_HEAD(z_erofs_lzma_wq);
19
20module_param_named(lzma_streams, z_erofs_lzma_nstrms, uint, 0444);
21
22void z_erofs_lzma_exit(void)
23{
24 /* there should be no running fs instance */
25 while (z_erofs_lzma_avail_strms) {
26 struct z_erofs_lzma *strm;
27
28 spin_lock(&z_erofs_lzma_lock);
29 strm = z_erofs_lzma_head;
30 if (!strm) {
31 spin_unlock(&z_erofs_lzma_lock);
32 DBG_BUGON(1);
33 return;
34 }
35 z_erofs_lzma_head = NULL;
36 spin_unlock(&z_erofs_lzma_lock);
37
38 while (strm) {
39 struct z_erofs_lzma *n = strm->next;
40
41 if (strm->state)
42 xz_dec_microlzma_end(strm->state);
43 kfree(strm);
44 --z_erofs_lzma_avail_strms;
45 strm = n;
46 }
47 }
48}
49
50int z_erofs_lzma_init(void)
51{
52 unsigned int i;
53
54 /* by default, use # of possible CPUs instead */
55 if (!z_erofs_lzma_nstrms)
56 z_erofs_lzma_nstrms = num_possible_cpus();
57
58 for (i = 0; i < z_erofs_lzma_nstrms; ++i) {
59 struct z_erofs_lzma *strm = kzalloc(sizeof(*strm), GFP_KERNEL);
60
61 if (!strm) {
62 z_erofs_lzma_exit();
63 return -ENOMEM;
64 }
65 spin_lock(&z_erofs_lzma_lock);
66 strm->next = z_erofs_lzma_head;
67 z_erofs_lzma_head = strm;
68 spin_unlock(&z_erofs_lzma_lock);
69 ++z_erofs_lzma_avail_strms;
70 }
71 return 0;
72}
73
74int z_erofs_load_lzma_config(struct super_block *sb,
75 struct erofs_super_block *dsb,
76 struct z_erofs_lzma_cfgs *lzma, int size)
77{
78 static DEFINE_MUTEX(lzma_resize_mutex);
79 unsigned int dict_size, i;
80 struct z_erofs_lzma *strm, *head = NULL;
81 int err;
82
83 if (!lzma || size < sizeof(struct z_erofs_lzma_cfgs)) {
84 erofs_err(sb, "invalid lzma cfgs, size=%u", size);
85 return -EINVAL;
86 }
87 if (lzma->format) {
88 erofs_err(sb, "unidentified lzma format %x, please check kernel version",
89 le16_to_cpu(lzma->format));
90 return -EINVAL;
91 }
92 dict_size = le32_to_cpu(lzma->dict_size);
93 if (dict_size > Z_EROFS_LZMA_MAX_DICT_SIZE || dict_size < 4096) {
94 erofs_err(sb, "unsupported lzma dictionary size %u",
95 dict_size);
96 return -EINVAL;
97 }
98
99 erofs_info(sb, "EXPERIMENTAL MicroLZMA in use. Use at your own risk!");
100
101 /* in case 2 z_erofs_load_lzma_config() race to avoid deadlock */
102 mutex_lock(&lzma_resize_mutex);
103
104 if (z_erofs_lzma_max_dictsize >= dict_size) {
105 mutex_unlock(&lzma_resize_mutex);
106 return 0;
107 }
108
109 /* 1. collect/isolate all streams for the following check */
110 for (i = 0; i < z_erofs_lzma_avail_strms; ++i) {
111 struct z_erofs_lzma *last;
112
113again:
114 spin_lock(&z_erofs_lzma_lock);
115 strm = z_erofs_lzma_head;
116 if (!strm) {
117 spin_unlock(&z_erofs_lzma_lock);
118 wait_event(z_erofs_lzma_wq,
119 READ_ONCE(z_erofs_lzma_head));
120 goto again;
121 }
122 z_erofs_lzma_head = NULL;
123 spin_unlock(&z_erofs_lzma_lock);
124
125 for (last = strm; last->next; last = last->next)
126 ++i;
127 last->next = head;
128 head = strm;
129 }
130
131 err = 0;
132 /* 2. walk each isolated stream and grow max dict_size if needed */
133 for (strm = head; strm; strm = strm->next) {
134 if (strm->state)
135 xz_dec_microlzma_end(strm->state);
136 strm->state = xz_dec_microlzma_alloc(XZ_PREALLOC, dict_size);
137 if (!strm->state)
138 err = -ENOMEM;
139 }
140
141 /* 3. push back all to the global list and update max dict_size */
142 spin_lock(&z_erofs_lzma_lock);
143 DBG_BUGON(z_erofs_lzma_head);
144 z_erofs_lzma_head = head;
145 spin_unlock(&z_erofs_lzma_lock);
146
147 z_erofs_lzma_max_dictsize = dict_size;
148 mutex_unlock(&lzma_resize_mutex);
149 return err;
150}
151
152int z_erofs_lzma_decompress(struct z_erofs_decompress_req *rq,
153 struct page **pagepool)
154{
155 const unsigned int nrpages_out =
156 PAGE_ALIGN(rq->pageofs_out + rq->outputsize) >> PAGE_SHIFT;
157 const unsigned int nrpages_in =
158 PAGE_ALIGN(rq->inputsize) >> PAGE_SHIFT;
159 unsigned int inputmargin, inlen, outlen, pageofs;
160 struct z_erofs_lzma *strm;
161 u8 *kin;
162 bool bounced = false;
163 int no, ni, j, err = 0;
164
165 /* 1. get the exact LZMA compressed size */
166 kin = kmap(*rq->in);
167 inputmargin = 0;
168 while (!kin[inputmargin & ~PAGE_MASK])
169 if (!(++inputmargin & ~PAGE_MASK))
170 break;
171
172 if (inputmargin >= PAGE_SIZE) {
173 kunmap(*rq->in);
174 return -EFSCORRUPTED;
175 }
176 rq->inputsize -= inputmargin;
177
178 /* 2. get an available lzma context */
179again:
180 spin_lock(&z_erofs_lzma_lock);
181 strm = z_erofs_lzma_head;
182 if (!strm) {
183 spin_unlock(&z_erofs_lzma_lock);
184 wait_event(z_erofs_lzma_wq, READ_ONCE(z_erofs_lzma_head));
185 goto again;
186 }
187 z_erofs_lzma_head = strm->next;
188 spin_unlock(&z_erofs_lzma_lock);
189
190 /* 3. multi-call decompress */
191 inlen = rq->inputsize;
192 outlen = rq->outputsize;
193 xz_dec_microlzma_reset(strm->state, inlen, outlen,
194 !rq->partial_decoding);
195 pageofs = rq->pageofs_out;
196 strm->buf.in = kin + inputmargin;
197 strm->buf.in_pos = 0;
198 strm->buf.in_size = min_t(u32, inlen, PAGE_SIZE - inputmargin);
199 inlen -= strm->buf.in_size;
200 strm->buf.out = NULL;
201 strm->buf.out_pos = 0;
202 strm->buf.out_size = 0;
203
204 for (ni = 0, no = -1;;) {
205 enum xz_ret xz_err;
206
207 if (strm->buf.out_pos == strm->buf.out_size) {
208 if (strm->buf.out) {
209 kunmap(rq->out[no]);
210 strm->buf.out = NULL;
211 }
212
213 if (++no >= nrpages_out || !outlen) {
214 erofs_err(rq->sb, "decompressed buf out of bound");
215 err = -EFSCORRUPTED;
216 break;
217 }
218 strm->buf.out_pos = 0;
219 strm->buf.out_size = min_t(u32, outlen,
220 PAGE_SIZE - pageofs);
221 outlen -= strm->buf.out_size;
222 if (rq->out[no])
223 strm->buf.out = kmap(rq->out[no]) + pageofs;
224 pageofs = 0;
225 } else if (strm->buf.in_pos == strm->buf.in_size) {
226 kunmap(rq->in[ni]);
227
228 if (++ni >= nrpages_in || !inlen) {
229 erofs_err(rq->sb, "compressed buf out of bound");
230 err = -EFSCORRUPTED;
231 break;
232 }
233 strm->buf.in_pos = 0;
234 strm->buf.in_size = min_t(u32, inlen, PAGE_SIZE);
235 inlen -= strm->buf.in_size;
236 kin = kmap(rq->in[ni]);
237 strm->buf.in = kin;
238 bounced = false;
239 }
240
241 /*
242 * Handle overlapping: Use bounced buffer if the compressed
243 * data is under processing; Otherwise, Use short-lived pages
244 * from the on-stack pagepool where pages share with the same
245 * request.
246 */
247 if (!bounced && rq->out[no] == rq->in[ni]) {
248 memcpy(strm->bounce, strm->buf.in, strm->buf.in_size);
249 strm->buf.in = strm->bounce;
250 bounced = true;
251 }
252 for (j = ni + 1; j < nrpages_in; ++j) {
253 struct page *tmppage;
254
255 if (rq->out[no] != rq->in[j])
256 continue;
257
258 DBG_BUGON(erofs_page_is_managed(EROFS_SB(rq->sb),
259 rq->in[j]));
260 tmppage = erofs_allocpage(pagepool,
261 GFP_KERNEL | __GFP_NOFAIL);
262 set_page_private(tmppage, Z_EROFS_SHORTLIVED_PAGE);
263 copy_highpage(tmppage, rq->in[j]);
264 rq->in[j] = tmppage;
265 }
266 xz_err = xz_dec_microlzma_run(strm->state, &strm->buf);
267 DBG_BUGON(strm->buf.out_pos > strm->buf.out_size);
268 DBG_BUGON(strm->buf.in_pos > strm->buf.in_size);
269
270 if (xz_err != XZ_OK) {
271 if (xz_err == XZ_STREAM_END && !outlen)
272 break;
273 erofs_err(rq->sb, "failed to decompress %d in[%u] out[%u]",
274 xz_err, rq->inputsize, rq->outputsize);
275 err = -EFSCORRUPTED;
276 break;
277 }
278 }
279 if (no < nrpages_out && strm->buf.out)
280 kunmap(rq->in[no]);
281 if (ni < nrpages_in)
282 kunmap(rq->in[ni]);
283 /* 4. push back LZMA stream context to the global list */
284 spin_lock(&z_erofs_lzma_lock);
285 strm->next = z_erofs_lzma_head;
286 z_erofs_lzma_head = strm;
287 spin_unlock(&z_erofs_lzma_lock);
288 wake_up(&z_erofs_lzma_wq);
289 return err;
290}
fs/erofs/erofs_fs.h+50−23
@@ -21,14 +21,29 @@ · 20 unmodified lines
21#define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x0000000221#define EROFS_FEATURE_INCOMPAT_COMPR_CFGS 0x00000002
22#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x0000000222#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002
23#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x0000000423#define EROFS_FEATURE_INCOMPAT_CHUNKED_FILE 0x00000004
24#define EROFS_FEATURE_INCOMPAT_DEVICE_TABLE 0x00000008
25#define EROFS_FEATURE_INCOMPAT_COMPR_HEAD2 0x00000008
24#define EROFS_ALL_FEATURE_INCOMPAT \26#define EROFS_ALL_FEATURE_INCOMPAT \
25 (EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \27 (EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \
26 EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \28 EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
27 EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \29 EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER | \
28 EROFS_FEATURE_INCOMPAT_CHUNKED_FILE)30 EROFS_FEATURE_INCOMPAT_CHUNKED_FILE | \
31 EROFS_FEATURE_INCOMPAT_DEVICE_TABLE | \
32 EROFS_FEATURE_INCOMPAT_COMPR_HEAD2)
29 33
30#define EROFS_SB_EXTSLOT_SIZE 1634#define EROFS_SB_EXTSLOT_SIZE 16
31 35
36struct erofs_deviceslot {
37 union {
38 u8 uuid[16]; /* used for device manager later */
39 u8 userdata[64]; /* digest(sha256), etc. */
40 } u;
41 __le32 blocks; /* total fs blocks of this device */
42 __le32 mapped_blkaddr; /* map starting at mapped_blkaddr */
43 u8 reserved[56];
44};
45#define EROFS_DEVT_SLOT_SIZE sizeof(struct erofs_deviceslot)
46
32/* erofs on-disk super block (currently 128 bytes) */47/* erofs on-disk super block (currently 128 bytes) */
33struct erofs_super_block {48struct erofs_super_block {
34 __le32 magic; /* file system magic number */49 __le32 magic; /* file system magic number */
@@ -54,7 +69,9 @@ struct erofs_super_block { · 19 unmodified lines
54 /* customized sliding window size instead of 64k by default */69 /* customized sliding window size instead of 64k by default */
55 __le16 lz4_max_distance;70 __le16 lz4_max_distance;
56 } __packed u1;71 } __packed u1;
57 __u8 reserved2[42];72 __le16 extra_devices; /* # of devices besides the primary device */
73 __le16 devt_slotoff; /* startoff = devt_slotoff * devt_slotsize */
74 __u8 reserved2[38];
58};75};
59 76
60/*77/*
@@ -238,7 +255,7 @@ static inline unsigned int erofs_xattr_entry_size(struct erofs_xattr_entry *e) · 177 unmodified lines
238/* 8-byte inode chunk indexes */255/* 8-byte inode chunk indexes */
239struct erofs_inode_chunk_index {256struct erofs_inode_chunk_index {
240 __le16 advise; /* always 0, don't care for now */257 __le16 advise; /* always 0, don't care for now */
241 __le16 device_id; /* back-end storage id, always 0 for now */258 __le16 device_id; /* back-end storage id (with bits masked) */
242 __le32 blkaddr; /* start block address of this inode chunk */259 __le32 blkaddr; /* start block address of this inode chunk */
243};260};
244 261
@@ -247,10 +264,11 @@ struct erofs_inode_chunk_index { · 2 unmodified lines
247 264
248/* available compression algorithm types (for h_algorithmtype) */265/* available compression algorithm types (for h_algorithmtype) */
249enum {266enum {
250 Z_EROFS_COMPRESSION_LZ4 = 0,267 Z_EROFS_COMPRESSION_LZ4 = 0,
268 Z_EROFS_COMPRESSION_LZMA = 1,
251 Z_EROFS_COMPRESSION_MAX269 Z_EROFS_COMPRESSION_MAX
252};270};
253#define Z_EROFS_ALL_COMPR_ALGS (1 << (Z_EROFS_COMPRESSION_MAX - 1))271#define Z_EROFS_ALL_COMPR_ALGS ((1 << Z_EROFS_COMPRESSION_MAX) - 1)
254 272
255/* 14 bytes (+ length field = 16 bytes) */273/* 14 bytes (+ length field = 16 bytes) */
256struct z_erofs_lz4_cfgs {274struct z_erofs_lz4_cfgs {
@@ -259,6 +277,15 @@ struct z_erofs_lz4_cfgs { · 2 unmodified lines
259 u8 reserved[10];277 u8 reserved[10];
260} __packed;278} __packed;
261 279
280/* 14 bytes (+ length field = 16 bytes) */
281struct z_erofs_lzma_cfgs {
282 __le32 dict_size;
283 __le16 format;
284 u8 reserved[8];
285} __packed;
286
287#define Z_EROFS_LZMA_MAX_DICT_SIZE (8 * Z_EROFS_PCLUSTER_MAX_SIZE)
288
262/*289/*
263 * bit 0 : COMPACTED_2B indexes (0 - off; 1 - on)290 * bit 0 : COMPACTED_2B indexes (0 - off; 1 - on)
264 * e.g. for 4k logical cluster size, 4B if compacted 2B is off;291 * e.g. for 4k logical cluster size, 4B if compacted 2B is off;
@@ -288,35 +315,34 @@ struct z_erofs_map_header { · 23 unmodified lines
288#define Z_EROFS_VLE_LEGACY_HEADER_PADDING 8315#define Z_EROFS_VLE_LEGACY_HEADER_PADDING 8
289 316
290/*317/*
291 * Fixed-sized output compression ondisk Logical Extent cluster type:318 * Fixed-sized output compression on-disk logical cluster type:
292 * 0 - literal (uncompressed) cluster319 * 0 - literal (uncompressed) lcluster
293 * 1 - compressed cluster (for the head logical cluster)320 * 1,3 - compressed lcluster (for HEAD lclusters)
294 * 2 - compressed cluster (for the other logical clusters)321 * 2 - compressed lcluster (for NONHEAD lclusters)
295 *322 *
296 * In detail,323 * In detail,
297 * 0 - literal (uncompressed) cluster,324 * 0 - literal (uncompressed) lcluster,
298 * di_advise = 0325 * di_advise = 0
299 * di_clusterofs = the literal data offset of the cluster326 * di_clusterofs = the literal data offset of the lcluster
300 * di_blkaddr = the blkaddr of the literal cluster327 * di_blkaddr = the blkaddr of the literal pcluster
301 *328 *
302 * 1 - compressed cluster (for the head logical cluster)329 * 1,3 - compressed lcluster (for HEAD lclusters)
303 * di_advise = 1330 * di_advise = 1 or 3
304 * di_clusterofs = the decompressed data offset of the cluster331 * di_clusterofs = the decompressed data offset of the lcluster
305 * di_blkaddr = the blkaddr of the compressed cluster332 * di_blkaddr = the blkaddr of the compressed pcluster
306 *333 *
307 * 2 - compressed cluster (for the other logical clusters)334 * 2 - compressed lcluster (for NONHEAD lclusters)
308 * di_advise = 2335 * di_advise = 2
309 * di_clusterofs =336 * di_clusterofs =
310 * the decompressed data offset in its own head cluster337 * the decompressed data offset in its own HEAD lcluster
311 * di_u.delta[0] = distance to its corresponding head cluster338 * di_u.delta[0] = distance to this HEAD lcluster
312 * di_u.delta[1] = distance to its corresponding tail cluster339 * di_u.delta[1] = distance to the next HEAD lcluster
313 * (di_advise could be 0, 1 or 2)
314 */340 */
315enum {341enum {
316 Z_EROFS_VLE_CLUSTER_TYPE_PLAIN = 0,342 Z_EROFS_VLE_CLUSTER_TYPE_PLAIN = 0,
317 Z_EROFS_VLE_CLUSTER_TYPE_HEAD = 1,343 Z_EROFS_VLE_CLUSTER_TYPE_HEAD1 = 1,
318 Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD = 2,344 Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD = 2,
319 Z_EROFS_VLE_CLUSTER_TYPE_RESERVED = 3,345 Z_EROFS_VLE_CLUSTER_TYPE_HEAD2 = 3,
320 Z_EROFS_VLE_CLUSTER_TYPE_MAX346 Z_EROFS_VLE_CLUSTER_TYPE_MAX
321};347};
322 348
@@ -384,6 +410,7 @@ static inline void erofs_check_ondisk_layout_definitions(void) · 61 unmodified lines
384 /* keep in sync between 2 index structures for better extendibility */410 /* keep in sync between 2 index structures for better extendibility */
385 BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) !=411 BUILD_BUG_ON(sizeof(struct erofs_inode_chunk_index) !=
386 sizeof(struct z_erofs_vle_decompressed_index));412 sizeof(struct z_erofs_vle_decompressed_index));
413 BUILD_BUG_ON(sizeof(struct erofs_deviceslot) != 128);
387 414
388 BUILD_BUG_ON(BIT(Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS) <415 BUILD_BUG_ON(BIT(Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS) <
389 Z_EROFS_VLE_CLUSTER_TYPE_MAX - 1);416 Z_EROFS_VLE_CLUSTER_TYPE_MAX - 1);
fs/erofs/inode.c+1−1
@@ -192,7 +192,7 @@ static struct page *erofs_read_inode(struct inode *inode, · 191 unmodified lines
192 inode->i_atime.tv_nsec = inode->i_ctime.tv_nsec;192 inode->i_atime.tv_nsec = inode->i_ctime.tv_nsec;
193 193
194 inode->i_flags &= ~S_DAX;194 inode->i_flags &= ~S_DAX;
195 if (test_opt(&sbi->ctx, DAX_ALWAYS) && S_ISREG(inode->i_mode) &&195 if (test_opt(&sbi->opt, DAX_ALWAYS) && S_ISREG(inode->i_mode) &&
196 vi->datalayout == EROFS_INODE_FLAT_PLAIN)196 vi->datalayout == EROFS_INODE_FLAT_PLAIN)
197 inode->i_flags |= S_DAX;197 inode->i_flags |= S_DAX;
198 if (!nblks)198 if (!nblks)
fs/erofs/internal.h+94−11
@@ -47,7 +47,16 @@ typedef u64 erofs_off_t; · 46 unmodified lines
47/* data type for filesystem-wide blocks number */47/* data type for filesystem-wide blocks number */
48typedef u32 erofs_blk_t;48typedef u32 erofs_blk_t;
49 49
50struct erofs_fs_context {50struct erofs_device_info {
51 char *path;
52 struct block_device *bdev;
53 struct dax_device *dax_dev;
54
55 u32 blocks;
56 u32 mapped_blkaddr;
57};
58
59struct erofs_mount_opts {
51#ifdef CONFIG_EROFS_FS_ZIP60#ifdef CONFIG_EROFS_FS_ZIP
52 /* current strategy of how to use managed cache */61 /* current strategy of how to use managed cache */
53 unsigned char cache_strategy;62 unsigned char cache_strategy;
@@ -60,6 +69,18 @@ struct erofs_fs_context { · 6 unmodified lines
60 unsigned int mount_opt;69 unsigned int mount_opt;
61};70};
62 71
72struct erofs_dev_context {
73 struct idr tree;
74 struct rw_semaphore rwsem;
75
76 unsigned int extra_devices;
77};
78
79struct erofs_fs_context {
80 struct erofs_mount_opts opt;
81 struct erofs_dev_context *devs;
82};
83
63/* all filesystem-wide lz4 configurations */84/* all filesystem-wide lz4 configurations */
64struct erofs_sb_lz4_info {85struct erofs_sb_lz4_info {
65 /* # of pages needed for EROFS lz4 rolling decompression */86 /* # of pages needed for EROFS lz4 rolling decompression */
@@ -69,6 +90,7 @@ struct erofs_sb_lz4_info { · 3 unmodified lines
69};90};
70 91
71struct erofs_sb_info {92struct erofs_sb_info {
93 struct erofs_mount_opts opt; /* options */
72#ifdef CONFIG_EROFS_FS_ZIP94#ifdef CONFIG_EROFS_FS_ZIP
73 /* list for all registered superblocks, mainly for shrinker */95 /* list for all registered superblocks, mainly for shrinker */
74 struct list_head list;96 struct list_head list;
@@ -85,12 +107,16 @@ struct erofs_sb_info { · 10 unmodified lines
85 107
86 struct erofs_sb_lz4_info lz4;108 struct erofs_sb_lz4_info lz4;
87#endif /* CONFIG_EROFS_FS_ZIP */109#endif /* CONFIG_EROFS_FS_ZIP */
110 struct erofs_dev_context *devs;
88 struct dax_device *dax_dev;111 struct dax_device *dax_dev;
89 u32 blocks;112 u64 total_blocks;
113 u32 primarydevice_blocks;
114
90 u32 meta_blkaddr;115 u32 meta_blkaddr;
91#ifdef CONFIG_EROFS_FS_XATTR116#ifdef CONFIG_EROFS_FS_XATTR
92 u32 xattr_blkaddr;117 u32 xattr_blkaddr;
93#endif118#endif
119 u16 device_id_mask; /* valid bits of device id to be used */
94 120
95 /* inode slot unit size in bit shift */121 /* inode slot unit size in bit shift */
96 unsigned char islotbits;122 unsigned char islotbits;
@@ -108,8 +134,6 @@ struct erofs_sb_info { · 11 unmodified lines
108 u8 volume_name[16]; /* volume name */134 u8 volume_name[16]; /* volume name */
109 u32 feature_compat;135 u32 feature_compat;
110 u32 feature_incompat;136 u32 feature_incompat;
111
112 struct erofs_fs_context ctx; /* options */
113};137};
114 138
115#define EROFS_SB(sb) ((struct erofs_sb_info *)(sb)->s_fs_info)139#define EROFS_SB(sb) ((struct erofs_sb_info *)(sb)->s_fs_info)
@@ -121,9 +145,9 @@ struct erofs_sb_info { · 5 unmodified lines
121#define EROFS_MOUNT_DAX_ALWAYS 0x00000040145#define EROFS_MOUNT_DAX_ALWAYS 0x00000040
122#define EROFS_MOUNT_DAX_NEVER 0x00000080146#define EROFS_MOUNT_DAX_NEVER 0x00000080
123 147
124#define clear_opt(ctx, option) ((ctx)->mount_opt &= ~EROFS_MOUNT_##option)148#define clear_opt(opt, option) ((opt)->mount_opt &= ~EROFS_MOUNT_##option)
125#define set_opt(ctx, option) ((ctx)->mount_opt |= EROFS_MOUNT_##option)149#define set_opt(opt, option) ((opt)->mount_opt |= EROFS_MOUNT_##option)
126#define test_opt(ctx, option) ((ctx)->mount_opt & EROFS_MOUNT_##option)150#define test_opt(opt, option) ((opt)->mount_opt & EROFS_MOUNT_##option)
127 151
128enum {152enum {
129 EROFS_ZIP_CACHE_DISABLED,153 EROFS_ZIP_CACHE_DISABLED,
@@ -237,6 +261,7 @@ static inline bool erofs_sb_has_##name(struct erofs_sb_info *sbi) \ · 107 unmodified lines
237EROFS_FEATURE_FUNCS(lz4_0padding, incompat, INCOMPAT_LZ4_0PADDING)261EROFS_FEATURE_FUNCS(lz4_0padding, incompat, INCOMPAT_LZ4_0PADDING)
238EROFS_FEATURE_FUNCS(compr_cfgs, incompat, INCOMPAT_COMPR_CFGS)262EROFS_FEATURE_FUNCS(compr_cfgs, incompat, INCOMPAT_COMPR_CFGS)
239EROFS_FEATURE_FUNCS(big_pcluster, incompat, INCOMPAT_BIG_PCLUSTER)263EROFS_FEATURE_FUNCS(big_pcluster, incompat, INCOMPAT_BIG_PCLUSTER)
264EROFS_FEATURE_FUNCS(device_table, incompat, INCOMPAT_DEVICE_TABLE)
240EROFS_FEATURE_FUNCS(sb_chksum, compat, COMPAT_SB_CHKSUM)265EROFS_FEATURE_FUNCS(sb_chksum, compat, COMPAT_SB_CHKSUM)
241 266
242/* atomic flag definitions */267/* atomic flag definitions */
@@ -307,6 +332,19 @@ static inline unsigned int erofs_inode_datalayout(unsigned int value) · 64 unmodified lines
307 EROFS_I_DATALAYOUT_BITS);332 EROFS_I_DATALAYOUT_BITS);
308}333}
309 334
335/*
336 * Different from grab_cache_page_nowait(), reclaiming is never triggered
337 * when allocating new pages.
338 */
339static inline
340struct page *erofs_grab_cache_page_nowait(struct address_space *mapping,
341 pgoff_t index)
342{
343 return pagecache_get_page(mapping, index,
344 FGP_LOCK|FGP_CREAT|FGP_NOFS|FGP_NOWAIT,
345 readahead_gfp_mask(mapping) & ~__GFP_RECLAIM);
346}
347
310extern const struct super_operations erofs_sops;348extern const struct super_operations erofs_sops;
311 349
312extern const struct address_space_operations erofs_raw_access_aops;350extern const struct address_space_operations erofs_raw_access_aops;
@@ -338,7 +376,7 @@ extern const struct address_space_operations z_erofs_aops; · 25 unmodified lines
338 * of the corresponding uncompressed data in the file.376 * of the corresponding uncompressed data in the file.
339 */377 */
340enum {378enum {
341 BH_Zipped = BH_PrivateStart,379 BH_Encoded = BH_PrivateStart,
342 BH_FullMapped,380 BH_FullMapped,
343};381};
344 382
@@ -346,8 +384,8 @@ enum { · 1 unmodified line
346#define EROFS_MAP_MAPPED (1 << BH_Mapped)384#define EROFS_MAP_MAPPED (1 << BH_Mapped)
347/* Located in metadata (could be copied from bd_inode) */385/* Located in metadata (could be copied from bd_inode) */
348#define EROFS_MAP_META (1 << BH_Meta)386#define EROFS_MAP_META (1 << BH_Meta)
349/* The extent has been compressed */387/* The extent is encoded */
350#define EROFS_MAP_ZIPPED (1 << BH_Zipped)388#define EROFS_MAP_ENCODED (1 << BH_Encoded)
351/* The length of extent is full */389/* The length of extent is full */
352#define EROFS_MAP_FULL_MAPPED (1 << BH_FullMapped)390#define EROFS_MAP_FULL_MAPPED (1 << BH_FullMapped)
353 391
@@ -355,6 +393,8 @@ struct erofs_map_blocks { · 1 unmodified line
355 erofs_off_t m_pa, m_la;393 erofs_off_t m_pa, m_la;
356 u64 m_plen, m_llen;394 u64 m_plen, m_llen;
357 395
396 unsigned short m_deviceid;
397 char m_algorithmformat;
358 unsigned int m_flags;398 unsigned int m_flags;
359 399
360 struct page *mpage;400 struct page *mpage;
@@ -367,6 +407,13 @@ struct erofs_map_blocks { · 6 unmodified lines
367 * approach instead if possible since it's more metadata lightweight.)407 * approach instead if possible since it's more metadata lightweight.)
368 */408 */
369#define EROFS_GET_BLOCKS_FIEMAP 0x0002409#define EROFS_GET_BLOCKS_FIEMAP 0x0002
410/* Used to map the whole extent if non-negligible data is requested for LZMA */
411#define EROFS_GET_BLOCKS_READMORE 0x0004
412
413enum {
414 Z_EROFS_COMPRESSION_SHIFTED = Z_EROFS_COMPRESSION_MAX,
415 Z_EROFS_COMPRESSION_RUNTIME_MAX
416};
370 417
371/* zmap.c */418/* zmap.c */
372extern const struct iomap_ops z_erofs_iomap_report_ops;419extern const struct iomap_ops z_erofs_iomap_report_ops;
@@ -386,9 +433,18 @@ static inline int z_erofs_map_blocks_iter(struct inode *inode, · 13 unmodified lines
386}433}
387#endif /* !CONFIG_EROFS_FS_ZIP */434#endif /* !CONFIG_EROFS_FS_ZIP */
388 435
436struct erofs_map_dev {
437 struct block_device *m_bdev;
438 struct dax_device *m_daxdev;
439
440 erofs_off_t m_pa;
441 unsigned int m_deviceid;
442};
443
389/* data.c */444/* data.c */
390extern const struct file_operations erofs_file_fops;445extern const struct file_operations erofs_file_fops;
391struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr);446struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr);
447int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *dev);
392int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,448int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
393 u64 start, u64 len);449 u64 start, u64 len);
394 450
@@ -443,7 +499,14 @@ void erofs_pcpubuf_init(void); · 48 unmodified lines
443void erofs_pcpubuf_exit(void);499void erofs_pcpubuf_exit(void);
444 500
445/* utils.c / zdata.c */501/* utils.c / zdata.c */
446struct page *erofs_allocpage(struct list_head *pool, gfp_t gfp);502struct page *erofs_allocpage(struct page **pagepool, gfp_t gfp);
503static inline void erofs_pagepool_add(struct page **pagepool,
504 struct page *page)
505{
506 set_page_private(page, (unsigned long)*pagepool);
507 *pagepool = page;
508}
509void erofs_release_pages(struct page **pagepool);
447 510
448#ifdef CONFIG_EROFS_FS_ZIP511#ifdef CONFIG_EROFS_FS_ZIP
449int erofs_workgroup_put(struct erofs_workgroup *grp);512int erofs_workgroup_put(struct erofs_workgroup *grp);
@@ -483,6 +546,26 @@ static inline int z_erofs_load_lz4_config(struct super_block *sb, · 33 unmodified lines
483}546}
484#endif /* !CONFIG_EROFS_FS_ZIP */547#endif /* !CONFIG_EROFS_FS_ZIP */
485 548
549#ifdef CONFIG_EROFS_FS_ZIP_LZMA
550int z_erofs_lzma_init(void);
551void z_erofs_lzma_exit(void);
552int z_erofs_load_lzma_config(struct super_block *sb,
553 struct erofs_super_block *dsb,
554 struct z_erofs_lzma_cfgs *lzma, int size);
555#else
556static inline int z_erofs_lzma_init(void) { return 0; }
557static inline int z_erofs_lzma_exit(void) { return 0; }
558static inline int z_erofs_load_lzma_config(struct super_block *sb,
559 struct erofs_super_block *dsb,
560 struct z_erofs_lzma_cfgs *lzma, int size) {
561 if (lzma) {
562 erofs_err(sb, "lzma algorithm isn't enabled");
563 return -EINVAL;
564 }
565 return 0;
566}
567#endif /* !CONFIG_EROFS_FS_ZIP */
568
486#define EFSCORRUPTED EUCLEAN /* Filesystem is corrupted */569#define EFSCORRUPTED EUCLEAN /* Filesystem is corrupted */
487 570
488#endif /* __EROFS_INTERNAL_H */571#endif /* __EROFS_INTERNAL_H */
fs/erofs/pcpubuf.c+3−3
@@ -49,7 +49,7 @@ int erofs_pcpubuf_growsize(unsigned int nrpages) · 48 unmodified lines
49{49{
50 static DEFINE_MUTEX(pcb_resize_mutex);50 static DEFINE_MUTEX(pcb_resize_mutex);
51 static unsigned int pcb_nrpages;51 static unsigned int pcb_nrpages;
52 LIST_HEAD(pagepool);52 struct page *pagepool = NULL;
53 int delta, cpu, ret, i;53 int delta, cpu, ret, i;
54 54
55 mutex_lock(&pcb_resize_mutex);55 mutex_lock(&pcb_resize_mutex);
@@ -102,13 +102,13 @@ int erofs_pcpubuf_growsize(unsigned int nrpages) · 46 unmodified lines
102 vunmap(old_ptr);102 vunmap(old_ptr);
103free_pagearray:103free_pagearray:
104 while (i)104 while (i)
105 list_add(&oldpages[--i]->lru, &pagepool);105 erofs_pagepool_add(&pagepool, oldpages[--i]);
106 kfree(oldpages);106 kfree(oldpages);
107 if (ret)107 if (ret)
108 break;108 break;
109 }109 }
110 pcb_nrpages = nrpages;110 pcb_nrpages = nrpages;
111 put_pages_list(&pagepool);111 erofs_release_pages(&pagepool);
112out:112out:
113 mutex_unlock(&pcb_resize_mutex);113 mutex_unlock(&pcb_resize_mutex);
114 return ret;114 return ret;
fs/erofs/super.c+188−43
@@ -225,6 +225,9 @@ static int erofs_load_compr_cfgs(struct super_block *sb, · 224 unmodified lines
225 case Z_EROFS_COMPRESSION_LZ4:225 case Z_EROFS_COMPRESSION_LZ4:
226 ret = z_erofs_load_lz4_config(sb, dsb, data, size);226 ret = z_erofs_load_lz4_config(sb, dsb, data, size);
227 break;227 break;
228 case Z_EROFS_COMPRESSION_LZMA:
229 ret = z_erofs_load_lzma_config(sb, dsb, data, size);
230 break;
228 default:231 default:
229 DBG_BUGON(1);232 DBG_BUGON(1);
230 ret = -EFAULT;233 ret = -EFAULT;
@@ -252,6 +255,79 @@ static int erofs_load_compr_cfgs(struct super_block *sb, · 21 unmodified lines
252}255}
253#endif256#endif
254 257
258static int erofs_init_devices(struct super_block *sb,
259 struct erofs_super_block *dsb)
260{
261 struct erofs_sb_info *sbi = EROFS_SB(sb);
262 unsigned int ondisk_extradevs;
263 erofs_off_t pos;
264 struct page *page = NULL;
265 struct erofs_device_info *dif;
266 struct erofs_deviceslot *dis;
267 void *ptr;
268 int id, err = 0;
269
270 sbi->total_blocks = sbi->primarydevice_blocks;
271 if (!erofs_sb_has_device_table(sbi))
272 ondisk_extradevs = 0;
273 else
274 ondisk_extradevs = le16_to_cpu(dsb->extra_devices);
275
276 if (ondisk_extradevs != sbi->devs->extra_devices) {
277 erofs_err(sb, "extra devices don't match (ondisk %u, given %u)",
278 ondisk_extradevs, sbi->devs->extra_devices);
279 return -EINVAL;
280 }
281 if (!ondisk_extradevs)
282 return 0;
283
284 sbi->device_id_mask = roundup_pow_of_two(ondisk_extradevs + 1) - 1;
285 pos = le16_to_cpu(dsb->devt_slotoff) * EROFS_DEVT_SLOT_SIZE;
286 down_read(&sbi->devs->rwsem);
287 idr_for_each_entry(&sbi->devs->tree, dif, id) {
288 erofs_blk_t blk = erofs_blknr(pos);
289 struct block_device *bdev;
290
291 if (!page || page->index != blk) {
292 if (page) {
293 kunmap(page);
294 unlock_page(page);
295 put_page(page);
296 }
297
298 page = erofs_get_meta_page(sb, blk);
299 if (IS_ERR(page)) {
300 up_read(&sbi->devs->rwsem);
301 return PTR_ERR(page);
302 }
303 ptr = kmap(page);
304 }
305 dis = ptr + erofs_blkoff(pos);
306
307 bdev = blkdev_get_by_path(dif->path,
308 FMODE_READ | FMODE_EXCL,
309 sb->s_type);
310 if (IS_ERR(bdev)) {
311 err = PTR_ERR(bdev);
312 goto err_out;
313 }
314 dif->bdev = bdev;
315 dif->dax_dev = fs_dax_get_by_bdev(bdev);
316 dif->blocks = le32_to_cpu(dis->blocks);
317 dif->mapped_blkaddr = le32_to_cpu(dis->mapped_blkaddr);
318 sbi->total_blocks += dif->blocks;
319 pos += EROFS_DEVT_SLOT_SIZE;
320 }
321err_out:
322 up_read(&sbi->devs->rwsem);
323 if (page) {
324 kunmap(page);
325 unlock_page(page);
326 put_page(page);
327 }
328 return err;
329}
330
255static int erofs_read_superblock(struct super_block *sb)331static int erofs_read_superblock(struct super_block *sb)
256{332{
257 struct erofs_sb_info *sbi;333 struct erofs_sb_info *sbi;
@@ -303,7 +379,7 @@ static int erofs_read_superblock(struct super_block *sb) · 45 unmodified lines
303 sbi->sb_size);379 sbi->sb_size);
304 goto out;380 goto out;
305 }381 }
306 sbi->blocks = le32_to_cpu(dsb->blocks);382 sbi->primarydevice_blocks = le32_to_cpu(dsb->blocks);
307 sbi->meta_blkaddr = le32_to_cpu(dsb->meta_blkaddr);383 sbi->meta_blkaddr = le32_to_cpu(dsb->meta_blkaddr);
308#ifdef CONFIG_EROFS_FS_XATTR384#ifdef CONFIG_EROFS_FS_XATTR
309 sbi->xattr_blkaddr = le32_to_cpu(dsb->xattr_blkaddr);385 sbi->xattr_blkaddr = le32_to_cpu(dsb->xattr_blkaddr);
@@ -330,6 +406,11 @@ static int erofs_read_superblock(struct super_block *sb) · 20 unmodified lines
330 ret = erofs_load_compr_cfgs(sb, dsb);406 ret = erofs_load_compr_cfgs(sb, dsb);
331 else407 else
332 ret = z_erofs_load_lz4_config(sb, dsb, NULL, 0);408 ret = z_erofs_load_lz4_config(sb, dsb, NULL, 0);
409 if (ret < 0)
410 goto out;
411
412 /* handle multiple devices */
413 ret = erofs_init_devices(sb, dsb);
333out:414out:
334 kunmap(page);415 kunmap(page);
335 put_page(page);416 put_page(page);
@@ -340,15 +421,15 @@ out: · 4 unmodified lines
340static void erofs_default_options(struct erofs_fs_context *ctx)421static void erofs_default_options(struct erofs_fs_context *ctx)
341{422{
342#ifdef CONFIG_EROFS_FS_ZIP423#ifdef CONFIG_EROFS_FS_ZIP
343 ctx->cache_strategy = EROFS_ZIP_CACHE_READAROUND;424 ctx->opt.cache_strategy = EROFS_ZIP_CACHE_READAROUND;
344 ctx->max_sync_decompress_pages = 3;425 ctx->opt.max_sync_decompress_pages = 3;
345 ctx->readahead_sync_decompress = false;426 ctx->opt.readahead_sync_decompress = false;
346#endif427#endif
347#ifdef CONFIG_EROFS_FS_XATTR428#ifdef CONFIG_EROFS_FS_XATTR
348 set_opt(ctx, XATTR_USER);429 set_opt(&ctx->opt, XATTR_USER);
349#endif430#endif
350#ifdef CONFIG_EROFS_FS_POSIX_ACL431#ifdef CONFIG_EROFS_FS_POSIX_ACL
351 set_opt(ctx, POSIX_ACL);432 set_opt(&ctx->opt, POSIX_ACL);
352#endif433#endif
353}434}
354 435
@@ -358,6 +439,7 @@ enum { · 3 unmodified lines
358 Opt_cache_strategy,439 Opt_cache_strategy,
359 Opt_dax,440 Opt_dax,
360 Opt_dax_enum,441 Opt_dax_enum,
442 Opt_device,
361 Opt_err443 Opt_err
362};444};
363 445
@@ -381,6 +463,7 @@ static const struct fs_parameter_spec erofs_fs_parameters[] = { · 17 unmodified lines
381 erofs_param_cache_strategy),463 erofs_param_cache_strategy),
382 fsparam_flag("dax", Opt_dax),464 fsparam_flag("dax", Opt_dax),
383 fsparam_enum("dax", Opt_dax_enum, erofs_dax_param_enums),465 fsparam_enum("dax", Opt_dax_enum, erofs_dax_param_enums),
466 fsparam_string("device", Opt_device),
384 {}467 {}
385};468};
386 469
@@ -392,12 +475,12 @@ static bool erofs_fc_set_dax_mode(struct fs_context *fc, unsigned int mode) · 5 unmodified lines
392 switch (mode) {475 switch (mode) {
393 case EROFS_MOUNT_DAX_ALWAYS:476 case EROFS_MOUNT_DAX_ALWAYS:
394 warnfc(fc, "DAX enabled. Warning: EXPERIMENTAL, use at your own risk");477 warnfc(fc, "DAX enabled. Warning: EXPERIMENTAL, use at your own risk");
395 set_opt(ctx, DAX_ALWAYS);478 set_opt(&ctx->opt, DAX_ALWAYS);
396 clear_opt(ctx, DAX_NEVER);479 clear_opt(&ctx->opt, DAX_NEVER);
397 return true;480 return true;
398 case EROFS_MOUNT_DAX_NEVER:481 case EROFS_MOUNT_DAX_NEVER:
399 set_opt(ctx, DAX_NEVER);482 set_opt(&ctx->opt, DAX_NEVER);
400 clear_opt(ctx, DAX_ALWAYS);483 clear_opt(&ctx->opt, DAX_ALWAYS);
401 return true;484 return true;
402 default:485 default:
403 DBG_BUGON(1);486 DBG_BUGON(1);
@@ -412,9 +495,10 @@ static bool erofs_fc_set_dax_mode(struct fs_context *fc, unsigned int mode) · 8 unmodified lines
412static int erofs_fc_parse_param(struct fs_context *fc,495static int erofs_fc_parse_param(struct fs_context *fc,
413 struct fs_parameter *param)496 struct fs_parameter *param)
414{497{
415 struct erofs_fs_context *ctx __maybe_unused = fc->fs_private;498 struct erofs_fs_context *ctx = fc->fs_private;
416 struct fs_parse_result result;499 struct fs_parse_result result;
417 int opt;500 struct erofs_device_info *dif;
501 int opt, ret;
418 502
419 opt = fs_parse(fc, erofs_fs_parameters, param, &result);503 opt = fs_parse(fc, erofs_fs_parameters, param, &result);
420 if (opt < 0)504 if (opt < 0)
@@ -424,9 +508,9 @@ static int erofs_fc_parse_param(struct fs_context *fc, · 3 unmodified lines
424 case Opt_user_xattr:508 case Opt_user_xattr:
425#ifdef CONFIG_EROFS_FS_XATTR509#ifdef CONFIG_EROFS_FS_XATTR
426 if (result.boolean)510 if (result.boolean)
427 set_opt(ctx, XATTR_USER);511 set_opt(&ctx->opt, XATTR_USER);
428 else512 else
429 clear_opt(ctx, XATTR_USER);513 clear_opt(&ctx->opt, XATTR_USER);
430#else514#else
431 errorfc(fc, "{,no}user_xattr options not supported");515 errorfc(fc, "{,no}user_xattr options not supported");
432#endif516#endif
@@ -434,16 +518,16 @@ static int erofs_fc_parse_param(struct fs_context *fc, · 1 unmodified line
434 case Opt_acl:518 case Opt_acl:
435#ifdef CONFIG_EROFS_FS_POSIX_ACL519#ifdef CONFIG_EROFS_FS_POSIX_ACL
436 if (result.boolean)520 if (result.boolean)
437 set_opt(ctx, POSIX_ACL);521 set_opt(&ctx->opt, POSIX_ACL);
438 else522 else
439 clear_opt(ctx, POSIX_ACL);523 clear_opt(&ctx->opt, POSIX_ACL);
440#else524#else
441 errorfc(fc, "{,no}acl options not supported");525 errorfc(fc, "{,no}acl options not supported");
442#endif526#endif
443 break;527 break;
444 case Opt_cache_strategy:528 case Opt_cache_strategy:
445#ifdef CONFIG_EROFS_FS_ZIP529#ifdef CONFIG_EROFS_FS_ZIP
446 ctx->cache_strategy = result.uint_32;530 ctx->opt.cache_strategy = result.uint_32;
447#else531#else
448 errorfc(fc, "compression not supported, cache_strategy ignored");532 errorfc(fc, "compression not supported, cache_strategy ignored");
449#endif533#endif
@@ -456,6 +540,25 @@ static int erofs_fc_parse_param(struct fs_context *fc, · 6 unmodified lines
456 if (!erofs_fc_set_dax_mode(fc, result.uint_32))540 if (!erofs_fc_set_dax_mode(fc, result.uint_32))
457 return -EINVAL;541 return -EINVAL;
458 break;542 break;
543 case Opt_device:
544 dif = kzalloc(sizeof(*dif), GFP_KERNEL);
545 if (!dif)
546 return -ENOMEM;
547 dif->path = kstrdup(param->string, GFP_KERNEL);
548 if (!dif->path) {
549 kfree(dif);
550 return -ENOMEM;
551 }
552 down_write(&ctx->devs->rwsem);
553 ret = idr_alloc(&ctx->devs->tree, dif, 0, 0, GFP_KERNEL);
554 up_write(&ctx->devs->rwsem);
555 if (ret < 0) {
556 kfree(dif->path);
557 kfree(dif);
558 return ret;
559 }
560 ++ctx->devs->extra_devices;
561 break;
459 default:562 default:
460 return -ENOPARAM;563 return -ENOPARAM;
461 }564 }
@@ -540,15 +643,19 @@ static int erofs_fc_fill_super(struct super_block *sb, struct fs_context *fc) · 78 unmodified lines
540 return -ENOMEM;643 return -ENOMEM;
541 644
542 sb->s_fs_info = sbi;645 sb->s_fs_info = sbi;
646 sbi->opt = ctx->opt;
543 sbi->dax_dev = fs_dax_get_by_bdev(sb->s_bdev);647 sbi->dax_dev = fs_dax_get_by_bdev(sb->s_bdev);
648 sbi->devs = ctx->devs;
649 ctx->devs = NULL;
650
544 err = erofs_read_superblock(sb);651 err = erofs_read_superblock(sb);
545 if (err)652 if (err)
546 return err;653 return err;
547 654
548 if (test_opt(ctx, DAX_ALWAYS) &&655 if (test_opt(&sbi->opt, DAX_ALWAYS) &&
549 !dax_supported(sbi->dax_dev, sb->s_bdev, EROFS_BLKSIZ, 0, bdev_nr_sectors(sb->s_bdev))) {656 !dax_supported(sbi->dax_dev, sb->s_bdev, EROFS_BLKSIZ, 0, bdev_nr_sectors(sb->s_bdev))) {
550 errorfc(fc, "DAX unsupported by block device. Turning off DAX.");657 errorfc(fc, "DAX unsupported by block device. Turning off DAX.");
551 clear_opt(ctx, DAX_ALWAYS);658 clear_opt(&sbi->opt, DAX_ALWAYS);
552 }659 }
553 sb->s_flags |= SB_RDONLY | SB_NOATIME;660 sb->s_flags |= SB_RDONLY | SB_NOATIME;
554 sb->s_maxbytes = MAX_LFS_FILESIZE;661 sb->s_maxbytes = MAX_LFS_FILESIZE;
@@ -557,13 +664,11 @@ static int erofs_fc_fill_super(struct super_block *sb, struct fs_context *fc) · 2 unmodified lines
557 sb->s_op = &erofs_sops;664 sb->s_op = &erofs_sops;
558 sb->s_xattr = erofs_xattr_handlers;665 sb->s_xattr = erofs_xattr_handlers;
559 666
560 if (test_opt(ctx, POSIX_ACL))667 if (test_opt(&sbi->opt, POSIX_ACL))
561 sb->s_flags |= SB_POSIXACL;668 sb->s_flags |= SB_POSIXACL;
562 else669 else
563 sb->s_flags &= ~SB_POSIXACL;670 sb->s_flags &= ~SB_POSIXACL;
564 671
565 sbi->ctx = *ctx;
566
567#ifdef CONFIG_EROFS_FS_ZIP672#ifdef CONFIG_EROFS_FS_ZIP
568 xa_init(&sbi->managed_pslots);673 xa_init(&sbi->managed_pslots);
569#endif674#endif
@@ -607,20 +712,44 @@ static int erofs_fc_reconfigure(struct fs_context *fc) · 37 unmodified lines
607 712
608 DBG_BUGON(!sb_rdonly(sb));713 DBG_BUGON(!sb_rdonly(sb));
609 714
610 if (test_opt(ctx, POSIX_ACL))715 if (test_opt(&ctx->opt, POSIX_ACL))
611 fc->sb_flags |= SB_POSIXACL;716 fc->sb_flags |= SB_POSIXACL;
612 else717 else
613 fc->sb_flags &= ~SB_POSIXACL;718 fc->sb_flags &= ~SB_POSIXACL;
614 719
615 sbi->ctx = *ctx;720 sbi->opt = ctx->opt;
616 721
617 fc->sb_flags |= SB_RDONLY;722 fc->sb_flags |= SB_RDONLY;
618 return 0;723 return 0;
619}724}
620 725
726static int erofs_release_device_info(int id, void *ptr, void *data)
727{
728 struct erofs_device_info *dif = ptr;
729
730 fs_put_dax(dif->dax_dev);
731 if (dif->bdev)
732 blkdev_put(dif->bdev, FMODE_READ | FMODE_EXCL);
733 kfree(dif->path);
734 kfree(dif);
735 return 0;
736}
737
738static void erofs_free_dev_context(struct erofs_dev_context *devs)
739{
740 if (!devs)
741 return;
742 idr_for_each(&devs->tree, &erofs_release_device_info, NULL);
743 idr_destroy(&devs->tree);
744 kfree(devs);
745}
746
621static void erofs_fc_free(struct fs_context *fc)747static void erofs_fc_free(struct fs_context *fc)
622{748{
623 kfree(fc->fs_private);749 struct erofs_fs_context *ctx = fc->fs_private;
750
751 erofs_free_dev_context(ctx->devs);
752 kfree(ctx);
624}753}
625 754
626static const struct fs_context_operations erofs_context_ops = {755static const struct fs_context_operations erofs_context_ops = {
@@ -632,15 +761,21 @@ static const struct fs_context_operations erofs_context_ops = { · 5 unmodified lines
632 761
633static int erofs_init_fs_context(struct fs_context *fc)762static int erofs_init_fs_context(struct fs_context *fc)
634{763{
635 fc->fs_private = kzalloc(sizeof(struct erofs_fs_context), GFP_KERNEL);764 struct erofs_fs_context *ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
636 if (!fc->fs_private)
637 return -ENOMEM;
638 765
639 /* set default mount options */766 if (!ctx)
640 erofs_default_options(fc->fs_private);767 return -ENOMEM;
768 ctx->devs = kzalloc(sizeof(struct erofs_dev_context), GFP_KERNEL);
769 if (!ctx->devs) {
770 kfree(ctx);
771 return -ENOMEM;
772 }
773 fc->fs_private = ctx;
641 774
775 idr_init(&ctx->devs->tree);
776 init_rwsem(&ctx->devs->rwsem);
777 erofs_default_options(ctx);
642 fc->ops = &erofs_context_ops;778 fc->ops = &erofs_context_ops;
643
644 return 0;779 return 0;
645}780}
646 781
@@ -659,6 +794,8 @@ static void erofs_kill_sb(struct super_block *sb) · 12 unmodified lines
659 sbi = EROFS_SB(sb);794 sbi = EROFS_SB(sb);
660 if (!sbi)795 if (!sbi)
661 return;796 return;
797
798 erofs_free_dev_context(sbi->devs);
662 fs_put_dax(sbi->dax_dev);799 fs_put_dax(sbi->dax_dev);
663 kfree(sbi);800 kfree(sbi);
664 sb->s_fs_info = NULL;801 sb->s_fs_info = NULL;
@@ -706,6 +843,10 @@ static int __init erofs_module_init(void) · 41 unmodified lines
706 if (err)843 if (err)
707 goto shrinker_err;844 goto shrinker_err;
708 845
846 err = z_erofs_lzma_init();
847 if (err)
848 goto lzma_err;
849
709 erofs_pcpubuf_init();850 erofs_pcpubuf_init();
710 err = z_erofs_init_zip_subsystem();851 err = z_erofs_init_zip_subsystem();
711 if (err)852 if (err)
@@ -720,6 +861,8 @@ static int __init erofs_module_init(void) · 8 unmodified lines
720fs_err:861fs_err:
721 z_erofs_exit_zip_subsystem();862 z_erofs_exit_zip_subsystem();
722zip_err:863zip_err:
864 z_erofs_lzma_exit();
865lzma_err:
723 erofs_exit_shrinker();866 erofs_exit_shrinker();
724shrinker_err:867shrinker_err:
725 kmem_cache_destroy(erofs_inode_cachep);868 kmem_cache_destroy(erofs_inode_cachep);
@@ -730,11 +873,13 @@ icache_err: · 4 unmodified lines
730static void __exit erofs_module_exit(void)873static void __exit erofs_module_exit(void)
731{874{
732 unregister_filesystem(&erofs_fs_type);875 unregister_filesystem(&erofs_fs_type);
733 z_erofs_exit_zip_subsystem();
734 erofs_exit_shrinker();
735 876
736 /* Ensure all RCU free inodes are safe before cache is destroyed. */877 /* Ensure all RCU free inodes / pclusters are safe to be destroyed. */
737 rcu_barrier();878 rcu_barrier();
879
880 z_erofs_exit_zip_subsystem();
881 z_erofs_lzma_exit();
882 erofs_exit_shrinker();
738 kmem_cache_destroy(erofs_inode_cachep);883 kmem_cache_destroy(erofs_inode_cachep);
739 erofs_pcpubuf_exit();884 erofs_pcpubuf_exit();
740}885}
@@ -748,7 +893,7 @@ static int erofs_statfs(struct dentry *dentry, struct kstatfs *buf) · 7 unmodified lines
748 893
749 buf->f_type = sb->s_magic;894 buf->f_type = sb->s_magic;
750 buf->f_bsize = EROFS_BLKSIZ;895 buf->f_bsize = EROFS_BLKSIZ;
751 buf->f_blocks = sbi->blocks;896 buf->f_blocks = sbi->total_blocks;
752 buf->f_bfree = buf->f_bavail = 0;897 buf->f_bfree = buf->f_bavail = 0;
753 898
754 buf->f_files = ULLONG_MAX;899 buf->f_files = ULLONG_MAX;
@@ -763,31 +908,31 @@ static int erofs_statfs(struct dentry *dentry, struct kstatfs *buf) · 8 unmodified lines
763static int erofs_show_options(struct seq_file *seq, struct dentry *root)908static int erofs_show_options(struct seq_file *seq, struct dentry *root)
764{909{
765 struct erofs_sb_info *sbi = EROFS_SB(root->d_sb);910 struct erofs_sb_info *sbi = EROFS_SB(root->d_sb);
766 struct erofs_fs_context *ctx = &sbi->ctx;911 struct erofs_mount_opts *opt = &sbi->opt;
767 912
768#ifdef CONFIG_EROFS_FS_XATTR913#ifdef CONFIG_EROFS_FS_XATTR
769 if (test_opt(ctx, XATTR_USER))914 if (test_opt(opt, XATTR_USER))
770 seq_puts(seq, ",user_xattr");915 seq_puts(seq, ",user_xattr");
771 else916 else
772 seq_puts(seq, ",nouser_xattr");917 seq_puts(seq, ",nouser_xattr");
773#endif918#endif
774#ifdef CONFIG_EROFS_FS_POSIX_ACL919#ifdef CONFIG_EROFS_FS_POSIX_ACL
775 if (test_opt(ctx, POSIX_ACL))920 if (test_opt(opt, POSIX_ACL))
776 seq_puts(seq, ",acl");921 seq_puts(seq, ",acl");
777 else922 else
778 seq_puts(seq, ",noacl");923 seq_puts(seq, ",noacl");
779#endif924#endif
780#ifdef CONFIG_EROFS_FS_ZIP925#ifdef CONFIG_EROFS_FS_ZIP
781 if (ctx->cache_strategy == EROFS_ZIP_CACHE_DISABLED)926 if (opt->cache_strategy == EROFS_ZIP_CACHE_DISABLED)
782 seq_puts(seq, ",cache_strategy=disabled");927 seq_puts(seq, ",cache_strategy=disabled");
783 else if (ctx->cache_strategy == EROFS_ZIP_CACHE_READAHEAD)928 else if (opt->cache_strategy == EROFS_ZIP_CACHE_READAHEAD)
784 seq_puts(seq, ",cache_strategy=readahead");929 seq_puts(seq, ",cache_strategy=readahead");
785 else if (ctx->cache_strategy == EROFS_ZIP_CACHE_READAROUND)930 else if (opt->cache_strategy == EROFS_ZIP_CACHE_READAROUND)
786 seq_puts(seq, ",cache_strategy=readaround");931 seq_puts(seq, ",cache_strategy=readaround");
787#endif932#endif
788 if (test_opt(ctx, DAX_ALWAYS))933 if (test_opt(opt, DAX_ALWAYS))
789 seq_puts(seq, ",dax=always");934 seq_puts(seq, ",dax=always");
790 if (test_opt(ctx, DAX_NEVER))935 if (test_opt(opt, DAX_NEVER))
791 seq_puts(seq, ",dax=never");936 seq_puts(seq, ",dax=never");
792 return 0;937 return 0;
793}938}
fs/erofs/utils.c+14−5
@@ -6,20 +6,29 @@ · 5 unmodified lines
6#include "internal.h"6#include "internal.h"
7#include <linux/pagevec.h>7#include <linux/pagevec.h>
8 8
9struct page *erofs_allocpage(struct list_head *pool, gfp_t gfp)9struct page *erofs_allocpage(struct page **pagepool, gfp_t gfp)
10{10{
11 struct page *page;11 struct page *page = *pagepool;
12 12
13 if (!list_empty(pool)) {13 if (page) {
14 page = lru_to_page(pool);
15 DBG_BUGON(page_ref_count(page) != 1);14 DBG_BUGON(page_ref_count(page) != 1);
16 list_del(&page->lru);15 *pagepool = (struct page *)page_private(page);
17 } else {16 } else {
18 page = alloc_page(gfp);17 page = alloc_page(gfp);
19 }18 }
20 return page;19 return page;
21}20}
22 21
22void erofs_release_pages(struct page **pagepool)
23{
24 while (*pagepool) {
25 struct page *page = *pagepool;
26
27 *pagepool = (struct page *)page_private(page);
28 put_page(page);
29 }
30}
31
23#ifdef CONFIG_EROFS_FS_ZIP32#ifdef CONFIG_EROFS_FS_ZIP
24/* global shrink count (for all mounted EROFS instances) */33/* global shrink count (for all mounted EROFS instances) */
25static atomic_long_t erofs_global_shrink_cnt;34static atomic_long_t erofs_global_shrink_cnt;
fs/erofs/xattr.c+2−2
@@ -429,7 +429,7 @@ static int shared_getxattr(struct inode *inode, struct getxattr_iter *it) · 428 unmodified lines
429 429
430static bool erofs_xattr_user_list(struct dentry *dentry)430static bool erofs_xattr_user_list(struct dentry *dentry)
431{431{
432 return test_opt(&EROFS_SB(dentry->d_sb)->ctx, XATTR_USER);432 return test_opt(&EROFS_SB(dentry->d_sb)->opt, XATTR_USER);
433}433}
434 434
435static bool erofs_xattr_trusted_list(struct dentry *dentry)435static bool erofs_xattr_trusted_list(struct dentry *dentry)
@@ -476,7 +476,7 @@ static int erofs_xattr_generic_get(const struct xattr_handler *handler, · 40 unmodified lines
476 476
477 switch (handler->flags) {477 switch (handler->flags) {
478 case EROFS_XATTR_INDEX_USER:478 case EROFS_XATTR_INDEX_USER:
479 if (!test_opt(&sbi->ctx, XATTR_USER))479 if (!test_opt(&sbi->opt, XATTR_USER))
480 return -EOPNOTSUPP;480 return -EOPNOTSUPP;
481 break;481 break;
482 case EROFS_XATTR_INDEX_TRUSTED:482 case EROFS_XATTR_INDEX_TRUSTED:
fs/erofs/zdata.c+120−55
@@ -236,7 +236,7 @@ static DEFINE_MUTEX(z_pagemap_global_lock); · 235 unmodified lines
236static void preload_compressed_pages(struct z_erofs_collector *clt,236static void preload_compressed_pages(struct z_erofs_collector *clt,
237 struct address_space *mc,237 struct address_space *mc,
238 enum z_erofs_cache_alloctype type,238 enum z_erofs_cache_alloctype type,
239 struct list_head *pagepool)239 struct page **pagepool)
240{240{
241 struct z_erofs_pcluster *pcl = clt->pcl;241 struct z_erofs_pcluster *pcl = clt->pcl;
242 bool standalone = true;242 bool standalone = true;
@@ -287,12 +287,10 @@ static void preload_compressed_pages(struct z_erofs_collector *clt, · 44 unmodified lines
287 if (!cmpxchg_relaxed(pages, NULL, tagptr_cast_ptr(t)))287 if (!cmpxchg_relaxed(pages, NULL, tagptr_cast_ptr(t)))
288 continue;288 continue;
289 289
290 if (page) {290 if (page)
291 put_page(page);291 put_page(page);
292 } else if (newpage) {292 else if (newpage)
293 set_page_private(newpage, 0);293 erofs_pagepool_add(pagepool, newpage);
294 list_add(&newpage->lru, pagepool);
295 }
296 }294 }
297 295
298 /*296 /*
@@ -476,6 +474,11 @@ static int z_erofs_register_collection(struct z_erofs_collector *clt, · 177 unmodified lines
476 struct erofs_workgroup *grp;474 struct erofs_workgroup *grp;
477 int err;475 int err;
478 476
477 if (!(map->m_flags & EROFS_MAP_ENCODED)) {
478 DBG_BUGON(1);
479 return -EFSCORRUPTED;
480 }
481
479 /* no available pcluster, let's allocate one */482 /* no available pcluster, let's allocate one */
480 pcl = z_erofs_alloc_pcluster(map->m_plen >> PAGE_SHIFT);483 pcl = z_erofs_alloc_pcluster(map->m_plen >> PAGE_SHIFT);
481 if (IS_ERR(pcl))484 if (IS_ERR(pcl))
@@ -483,16 +486,11 @@ static int z_erofs_register_collection(struct z_erofs_collector *clt, · 1 unmodified line
483 486
484 atomic_set(&pcl->obj.refcount, 1);487 atomic_set(&pcl->obj.refcount, 1);
485 pcl->obj.index = map->m_pa >> PAGE_SHIFT;488 pcl->obj.index = map->m_pa >> PAGE_SHIFT;
486 489 pcl->algorithmformat = map->m_algorithmformat;
487 pcl->length = (map->m_llen << Z_EROFS_PCLUSTER_LENGTH_BIT) |490 pcl->length = (map->m_llen << Z_EROFS_PCLUSTER_LENGTH_BIT) |
488 (map->m_flags & EROFS_MAP_FULL_MAPPED ?491 (map->m_flags & EROFS_MAP_FULL_MAPPED ?
489 Z_EROFS_PCLUSTER_FULL_LENGTH : 0);492 Z_EROFS_PCLUSTER_FULL_LENGTH : 0);
490 493
491 if (map->m_flags & EROFS_MAP_ZIPPED)
492 pcl->algorithmformat = Z_EROFS_COMPRESSION_LZ4;
493 else
494 pcl->algorithmformat = Z_EROFS_COMPRESSION_SHIFTED;
495
496 /* new pclusters should be claimed as type 1, primary and followed */494 /* new pclusters should be claimed as type 1, primary and followed */
497 pcl->next = clt->owned_head;495 pcl->next = clt->owned_head;
498 clt->mode = COLLECT_PRIMARY_FOLLOWED;496 clt->mode = COLLECT_PRIMARY_FOLLOWED;
@@ -643,7 +641,7 @@ static bool should_alloc_managed_pages(struct z_erofs_decompress_frontend *fe, · 144 unmodified lines
643}641}
644 642
645static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe,643static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe,
646 struct page *page, struct list_head *pagepool)644 struct page *page, struct page **pagepool)
647{645{
648 struct inode *const inode = fe->inode;646 struct inode *const inode = fe->inode;
649 struct erofs_sb_info *const sbi = EROFS_I_SB(inode);647 struct erofs_sb_info *const sbi = EROFS_I_SB(inode);
@@ -695,7 +693,7 @@ restart_now: · 45 unmodified lines
695 goto err_out;693 goto err_out;
696 694
697 /* preload all compressed pages (maybe downgrade role if necessary) */695 /* preload all compressed pages (maybe downgrade role if necessary) */
698 if (should_alloc_managed_pages(fe, sbi->ctx.cache_strategy, map->m_la))696 if (should_alloc_managed_pages(fe, sbi->opt.cache_strategy, map->m_la))
699 cache_strategy = TRYALLOC;697 cache_strategy = TRYALLOC;
700 else698 else
701 cache_strategy = DONTALLOC;699 cache_strategy = DONTALLOC;
@@ -796,7 +794,7 @@ static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io, · 94 unmodified lines
796 /* Use workqueue and sync decompression for atomic contexts only */794 /* Use workqueue and sync decompression for atomic contexts only */
797 if (in_atomic() || irqs_disabled()) {795 if (in_atomic() || irqs_disabled()) {
798 queue_work(z_erofs_workqueue, &io->u.work);796 queue_work(z_erofs_workqueue, &io->u.work);
799 sbi->ctx.readahead_sync_decompress = true;797 sbi->opt.readahead_sync_decompress = true;
800 return;798 return;
801 }799 }
802 z_erofs_decompressqueue_work(&io->u.work);800 z_erofs_decompressqueue_work(&io->u.work);
@@ -836,7 +834,7 @@ static void z_erofs_decompressqueue_endio(struct bio *bio) · 33 unmodified lines
836 834
837static int z_erofs_decompress_pcluster(struct super_block *sb,835static int z_erofs_decompress_pcluster(struct super_block *sb,
838 struct z_erofs_pcluster *pcl,836 struct z_erofs_pcluster *pcl,
839 struct list_head *pagepool)837 struct page **pagepool)
840{838{
841 struct erofs_sb_info *const sbi = EROFS_SB(sb);839 struct erofs_sb_info *const sbi = EROFS_SB(sb);
842 struct z_erofs_pagevec_ctor ctor;840 struct z_erofs_pagevec_ctor ctor;
@@ -1036,7 +1034,7 @@ out: · 193 unmodified lines
1036}1034}
1037 1035
1038static void z_erofs_decompress_queue(const struct z_erofs_decompressqueue *io,1036static void z_erofs_decompress_queue(const struct z_erofs_decompressqueue *io,
1039 struct list_head *pagepool)1037 struct page **pagepool)
1040{1038{
1041 z_erofs_next_pcluster_t owned = io->head;1039 z_erofs_next_pcluster_t owned = io->head;
1042 1040
@@ -1060,18 +1058,18 @@ static void z_erofs_decompressqueue_work(struct work_struct *work) · 17 unmodified lines
1060{1058{
1061 struct z_erofs_decompressqueue *bgq =1059 struct z_erofs_decompressqueue *bgq =
1062 container_of(work, struct z_erofs_decompressqueue, u.work);1060 container_of(work, struct z_erofs_decompressqueue, u.work);
1063 LIST_HEAD(pagepool);1061 struct page *pagepool = NULL;
1064 1062
1065 DBG_BUGON(bgq->head == Z_EROFS_PCLUSTER_TAIL_CLOSED);1063 DBG_BUGON(bgq->head == Z_EROFS_PCLUSTER_TAIL_CLOSED);
1066 z_erofs_decompress_queue(bgq, &pagepool);1064 z_erofs_decompress_queue(bgq, &pagepool);
1067 1065
1068 put_pages_list(&pagepool);1066 erofs_release_pages(&pagepool);
1069 kvfree(bgq);1067 kvfree(bgq);
1070}1068}
1071 1069
1072static struct page *pickup_page_for_submission(struct z_erofs_pcluster *pcl,1070static struct page *pickup_page_for_submission(struct z_erofs_pcluster *pcl,
1073 unsigned int nr,1071 unsigned int nr,
1074 struct list_head *pagepool,1072 struct page **pagepool,
1075 struct address_space *mc,1073 struct address_space *mc,
1076 gfp_t gfp)1074 gfp_t gfp)
1077{1075{
@@ -1173,7 +1171,7 @@ repeat: · 95 unmodified lines
1173out_allocpage:1171out_allocpage:
1174 page = erofs_allocpage(pagepool, gfp | __GFP_NOFAIL);1172 page = erofs_allocpage(pagepool, gfp | __GFP_NOFAIL);
1175 if (oldpage != cmpxchg(&pcl->compressed_pages[nr], oldpage, page)) {1173 if (oldpage != cmpxchg(&pcl->compressed_pages[nr], oldpage, page)) {
1176 list_add(&page->lru, pagepool);1174 erofs_pagepool_add(pagepool, page);
1177 cond_resched();1175 cond_resched();
1178 goto repeat;1176 goto repeat;
1179 }1177 }
@@ -1257,7 +1255,7 @@ static void move_to_bypass_jobqueue(struct z_erofs_pcluster *pcl, · 77 unmodified lines
1257 1255
1258static void z_erofs_submit_queue(struct super_block *sb,1256static void z_erofs_submit_queue(struct super_block *sb,
1259 struct z_erofs_decompress_frontend *f,1257 struct z_erofs_decompress_frontend *f,
1260 struct list_head *pagepool,1258 struct page **pagepool,
1261 struct z_erofs_decompressqueue *fgq,1259 struct z_erofs_decompressqueue *fgq,
1262 bool *force_fg)1260 bool *force_fg)
1263{1261{
@@ -1266,8 +1264,9 @@ static void z_erofs_submit_queue(struct super_block *sb, · 2 unmodified lines
1266 struct z_erofs_decompressqueue *q[NR_JOBQUEUES];1264 struct z_erofs_decompressqueue *q[NR_JOBQUEUES];
1267 void *bi_private;1265 void *bi_private;
1268 z_erofs_next_pcluster_t owned_head = f->clt.owned_head;1266 z_erofs_next_pcluster_t owned_head = f->clt.owned_head;
1269 /* since bio will be NULL, no need to initialize last_index */1267 /* bio is NULL initially, so no need to initialize last_{index,bdev} */
1270 pgoff_t last_index;1268 pgoff_t last_index;
1269 struct block_device *last_bdev;
1271 unsigned int nr_bios = 0;1270 unsigned int nr_bios = 0;
1272 struct bio *bio = NULL;1271 struct bio *bio = NULL;
1273 1272
@@ -1279,6 +1278,7 @@ static void z_erofs_submit_queue(struct super_block *sb, · 5 unmodified lines
1279 q[JQ_SUBMIT]->head = owned_head;1278 q[JQ_SUBMIT]->head = owned_head;
1280 1279
1281 do {1280 do {
1281 struct erofs_map_dev mdev;
1282 struct z_erofs_pcluster *pcl;1282 struct z_erofs_pcluster *pcl;
1283 pgoff_t cur, end;1283 pgoff_t cur, end;
1284 unsigned int i = 0;1284 unsigned int i = 0;
@@ -1290,7 +1290,13 @@ static void z_erofs_submit_queue(struct super_block *sb, · 5 unmodified lines
1290 1290
1291 pcl = container_of(owned_head, struct z_erofs_pcluster, next);1291 pcl = container_of(owned_head, struct z_erofs_pcluster, next);
1292 1292
1293 cur = pcl->obj.index;1293 /* no device id here, thus it will always succeed */
1294 mdev = (struct erofs_map_dev) {
1295 .m_pa = blknr_to_addr(pcl->obj.index),
1296 };
1297 (void)erofs_map_dev(sb, &mdev);
1298
1299 cur = erofs_blknr(mdev.m_pa);
1294 end = cur + pcl->pclusterpages;1300 end = cur + pcl->pclusterpages;
1295 1301
1296 /* close the main owned chain at first */1302 /* close the main owned chain at first */
@@ -1306,7 +1312,8 @@ static void z_erofs_submit_queue(struct super_block *sb, · 9 unmodified lines
1306 if (!page)1312 if (!page)
1307 continue;1313 continue;
1308 1314
1309 if (bio && cur != last_index + 1) {1315 if (bio && (cur != last_index + 1 ||
1316 last_bdev != mdev.m_bdev)) {
1310submit_bio_retry:1317submit_bio_retry:
1311 submit_bio(bio);1318 submit_bio(bio);
1312 bio = NULL;1319 bio = NULL;
@@ -1314,9 +1321,10 @@ submit_bio_retry: · 1 unmodified line
1314 1321
1315 if (!bio) {1322 if (!bio) {
1316 bio = bio_alloc(GFP_NOIO, BIO_MAX_VECS);1323 bio = bio_alloc(GFP_NOIO, BIO_MAX_VECS);
1317
1318 bio->bi_end_io = z_erofs_decompressqueue_endio;1324 bio->bi_end_io = z_erofs_decompressqueue_endio;
1319 bio_set_dev(bio, sb->s_bdev);1325
1326 bio_set_dev(bio, mdev.m_bdev);
1327 last_bdev = mdev.m_bdev;
1320 bio->bi_iter.bi_sector = (sector_t)cur <<1328 bio->bi_iter.bi_sector = (sector_t)cur <<
1321 LOG_SECTORS_PER_BLOCK;1329 LOG_SECTORS_PER_BLOCK;
1322 bio->bi_private = bi_private;1330 bio->bi_private = bi_private;
@@ -1355,7 +1363,7 @@ submit_bio_retry: · 32 unmodified lines
1355 1363
1356static void z_erofs_runqueue(struct super_block *sb,1364static void z_erofs_runqueue(struct super_block *sb,
1357 struct z_erofs_decompress_frontend *f,1365 struct z_erofs_decompress_frontend *f,
1358 struct list_head *pagepool, bool force_fg)1366 struct page **pagepool, bool force_fg)
1359{1367{
1360 struct z_erofs_decompressqueue io[NR_JOBQUEUES];1368 struct z_erofs_decompressqueue io[NR_JOBQUEUES];
1361 1369
@@ -1377,18 +1385,87 @@ static void z_erofs_runqueue(struct super_block *sb, · 15 unmodified lines
1377 z_erofs_decompress_queue(&io[JQ_SUBMIT], pagepool);1385 z_erofs_decompress_queue(&io[JQ_SUBMIT], pagepool);
1378}1386}
1379 1387
1388/*
1389 * Since partial uptodate is still unimplemented for now, we have to use
1390 * approximate readmore strategies as a start.
1391 */
1392static void z_erofs_pcluster_readmore(struct z_erofs_decompress_frontend *f,
1393 struct readahead_control *rac,
1394 erofs_off_t end,
1395 struct page **pagepool,
1396 bool backmost)
1397{
1398 struct inode *inode = f->inode;
1399 struct erofs_map_blocks *map = &f->map;
1400 erofs_off_t cur;
1401 int err;
1402
1403 if (backmost) {
1404 map->m_la = end;
1405 err = z_erofs_map_blocks_iter(inode, map,
1406 EROFS_GET_BLOCKS_READMORE);
1407 if (err)
1408 return;
1409
1410 /* expend ra for the trailing edge if readahead */
1411 if (rac) {
1412 loff_t newstart = readahead_pos(rac);
1413
1414 cur = round_up(map->m_la + map->m_llen, PAGE_SIZE);
1415 readahead_expand(rac, newstart, cur - newstart);
1416 return;
1417 }
1418 end = round_up(end, PAGE_SIZE);
1419 } else {
1420 end = round_up(map->m_la, PAGE_SIZE);
1421
1422 if (!map->m_llen)
1423 return;
1424 }
1425
1426 cur = map->m_la + map->m_llen - 1;
1427 while (cur >= end) {
1428 pgoff_t index = cur >> PAGE_SHIFT;
1429 struct page *page;
1430
1431 page = erofs_grab_cache_page_nowait(inode->i_mapping, index);
1432 if (!page)
1433 goto skip;
1434
1435 if (PageUptodate(page)) {
1436 unlock_page(page);
1437 put_page(page);
1438 goto skip;
1439 }
1440
1441 err = z_erofs_do_read_page(f, page, pagepool);
1442 if (err)
1443 erofs_err(inode->i_sb,
1444 "readmore error at page %lu @ nid %llu",
1445 index, EROFS_I(inode)->nid);
1446 put_page(page);
1447skip:
1448 if (cur < PAGE_SIZE)
1449 break;
1450 cur = (index << PAGE_SHIFT) - 1;
1451 }
1452}
1453
1380static int z_erofs_readpage(struct file *file, struct page *page)1454static int z_erofs_readpage(struct file *file, struct page *page)
1381{1455{
1382 struct inode *const inode = page->mapping->host;1456 struct inode *const inode = page->mapping->host;
1383 struct z_erofs_decompress_frontend f = DECOMPRESS_FRONTEND_INIT(inode);1457 struct z_erofs_decompress_frontend f = DECOMPRESS_FRONTEND_INIT(inode);
1458 struct page *pagepool = NULL;
1384 int err;1459 int err;
1385 LIST_HEAD(pagepool);
1386 1460
1387 trace_erofs_readpage(page, false);1461 trace_erofs_readpage(page, false);
1388
1389 f.headoffset = (erofs_off_t)page->index << PAGE_SHIFT;1462 f.headoffset = (erofs_off_t)page->index << PAGE_SHIFT;
1390 1463
1464 z_erofs_pcluster_readmore(&f, NULL, f.headoffset + PAGE_SIZE - 1,
1465 &pagepool, true);
1391 err = z_erofs_do_read_page(&f, page, &pagepool);1466 err = z_erofs_do_read_page(&f, page, &pagepool);
1467 z_erofs_pcluster_readmore(&f, NULL, 0, &pagepool, false);
1468
1392 (void)z_erofs_collector_end(&f.clt);1469 (void)z_erofs_collector_end(&f.clt);
1393 1470
1394 /* if some compressed cluster ready, need submit them anyway */1471 /* if some compressed cluster ready, need submit them anyway */
@@ -1400,8 +1477,7 @@ static int z_erofs_readpage(struct file *file, struct page *page) · 5 unmodified lines
1400 if (f.map.mpage)1477 if (f.map.mpage)
1401 put_page(f.map.mpage);1478 put_page(f.map.mpage);
1402 1479
1403 /* clean up the remaining free pages */1480 erofs_release_pages(&pagepool);
1404 put_pages_list(&pagepool);
1405 return err;1481 return err;
1406}1482}
1407 1483
@@ -1409,29 +1485,19 @@ static void z_erofs_readahead(struct readahead_control *rac) · 1 unmodified line
1409{1485{
1410 struct inode *const inode = rac->mapping->host;1486 struct inode *const inode = rac->mapping->host;
1411 struct erofs_sb_info *const sbi = EROFS_I_SB(inode);1487 struct erofs_sb_info *const sbi = EROFS_I_SB(inode);
1412
1413 unsigned int nr_pages = readahead_count(rac);
1414 bool sync = (sbi->ctx.readahead_sync_decompress &&
1415 nr_pages <= sbi->ctx.max_sync_decompress_pages);
1416 struct z_erofs_decompress_frontend f = DECOMPRESS_FRONTEND_INIT(inode);1488 struct z_erofs_decompress_frontend f = DECOMPRESS_FRONTEND_INIT(inode);
1417 struct page *page, *head = NULL;1489 struct page *pagepool = NULL, *head = NULL, *page;
1418 LIST_HEAD(pagepool);1490 unsigned int nr_pages;
1419
1420 trace_erofs_readpages(inode, readahead_index(rac), nr_pages, false);
1421 1491
1422 f.readahead = true;1492 f.readahead = true;
1423 f.headoffset = readahead_pos(rac);1493 f.headoffset = readahead_pos(rac);
1424 1494
1425 while ((page = readahead_page(rac))) {1495 z_erofs_pcluster_readmore(&f, rac, f.headoffset +
1426 prefetchw(&page->flags);1496 readahead_length(rac) - 1, &pagepool, true);
1427 1497 nr_pages = readahead_count(rac);
1428 /*1498 trace_erofs_readpages(inode, readahead_index(rac), nr_pages, false);
1429 * A pure asynchronous readahead is indicated if
1430 * a PG_readahead marked page is hitted at first.
1431 * Let's also do asynchronous decompression for this case.
1432 */
1433 sync &= !(PageReadahead(page) && !head);
1434 1499
1500 while ((page = readahead_page(rac))) {
1435 set_page_private(page, (unsigned long)head);1501 set_page_private(page, (unsigned long)head);
1436 head = page;1502 head = page;
1437 }1503 }
@@ -1450,16 +1516,15 @@ static void z_erofs_readahead(struct readahead_control *rac) · 12 unmodified lines
1450 page->index, EROFS_I(inode)->nid);1516 page->index, EROFS_I(inode)->nid);
1451 put_page(page);1517 put_page(page);
1452 }1518 }
1453 1519 z_erofs_pcluster_readmore(&f, rac, 0, &pagepool, false);
1454 (void)z_erofs_collector_end(&f.clt);1520 (void)z_erofs_collector_end(&f.clt);
1455 1521
1456 z_erofs_runqueue(inode->i_sb, &f, &pagepool, sync);1522 z_erofs_runqueue(inode->i_sb, &f, &pagepool,
1457 1523 sbi->opt.readahead_sync_decompress &&
1524 nr_pages <= sbi->opt.max_sync_decompress_pages);
1458 if (f.map.mpage)1525 if (f.map.mpage)
1459 put_page(f.map.mpage);1526 put_page(f.map.mpage);
1460 1527 erofs_release_pages(&pagepool);
1461 /* clean up the remaining free pages */
1462 put_pages_list(&pagepool);
1463}1528}
1464 1529
1465const struct address_space_operations z_erofs_aops = {1530const struct address_space_operations z_erofs_aops = {
fs/erofs/zdata.h+0−7
@@ -94,13 +94,6 @@ struct z_erofs_decompressqueue { · 93 unmodified lines
94 } u;94 } u;
95};95};
96 96
97#define MNGD_MAPPING(sbi) ((sbi)->managed_cache->i_mapping)
98static inline bool erofs_page_is_managed(const struct erofs_sb_info *sbi,
99 struct page *page)
100{
101 return page->mapping == MNGD_MAPPING(sbi);
102}
103
104#define Z_EROFS_ONLINEPAGE_COUNT_BITS 297#define Z_EROFS_ONLINEPAGE_COUNT_BITS 2
105#define Z_EROFS_ONLINEPAGE_COUNT_MASK ((1 << Z_EROFS_ONLINEPAGE_COUNT_BITS) - 1)98#define Z_EROFS_ONLINEPAGE_COUNT_MASK ((1 << Z_EROFS_ONLINEPAGE_COUNT_BITS) - 1)
106#define Z_EROFS_ONLINEPAGE_INDEX_SHIFT (Z_EROFS_ONLINEPAGE_COUNT_BITS)99#define Z_EROFS_ONLINEPAGE_INDEX_SHIFT (Z_EROFS_ONLINEPAGE_COUNT_BITS)
fs/erofs/zmap.c+42−23
@@ -28,7 +28,7 @@ static int z_erofs_fill_inode_lazy(struct inode *inode) · 27 unmodified lines
28{28{
29 struct erofs_inode *const vi = EROFS_I(inode);29 struct erofs_inode *const vi = EROFS_I(inode);
30 struct super_block *const sb = inode->i_sb;30 struct super_block *const sb = inode->i_sb;
31 int err;31 int err, headnr;
32 erofs_off_t pos;32 erofs_off_t pos;
33 struct page *page;33 struct page *page;
34 void *kaddr;34 void *kaddr;
@@ -68,9 +68,11 @@ static int z_erofs_fill_inode_lazy(struct inode *inode) · 33 unmodified lines
68 vi->z_algorithmtype[0] = h->h_algorithmtype & 15;68 vi->z_algorithmtype[0] = h->h_algorithmtype & 15;
69 vi->z_algorithmtype[1] = h->h_algorithmtype >> 4;69 vi->z_algorithmtype[1] = h->h_algorithmtype >> 4;
70 70
71 if (vi->z_algorithmtype[0] >= Z_EROFS_COMPRESSION_MAX) {71 headnr = 0;
72 erofs_err(sb, "unknown compression format %u for nid %llu, please upgrade kernel",72 if (vi->z_algorithmtype[0] >= Z_EROFS_COMPRESSION_MAX ||
73 vi->z_algorithmtype[0], vi->nid);73 vi->z_algorithmtype[++headnr] >= Z_EROFS_COMPRESSION_MAX) {
74 erofs_err(sb, "unknown HEAD%u format %u for nid %llu, please upgrade kernel",
75 headnr + 1, vi->z_algorithmtype[headnr], vi->nid);
74 err = -EOPNOTSUPP;76 err = -EOPNOTSUPP;
75 goto unmap_done;77 goto unmap_done;
76 }78 }
@@ -111,7 +113,7 @@ struct z_erofs_maprecorder { · 34 unmodified lines
111 113
112 unsigned long lcn;114 unsigned long lcn;
113 /* compression extent information gathered */115 /* compression extent information gathered */
114 u8 type;116 u8 type, headtype;
115 u16 clusterofs;117 u16 clusterofs;
116 u16 delta[2];118 u16 delta[2];
117 erofs_blk_t pblk, compressedlcs;119 erofs_blk_t pblk, compressedlcs;
@@ -178,7 +180,8 @@ static int legacy_load_cluster_from_disk(struct z_erofs_maprecorder *m, · 60 unmodified lines
178 m->clusterofs = 1 << vi->z_logical_clusterbits;180 m->clusterofs = 1 << vi->z_logical_clusterbits;
179 m->delta[0] = le16_to_cpu(di->di_u.delta[0]);181 m->delta[0] = le16_to_cpu(di->di_u.delta[0]);
180 if (m->delta[0] & Z_EROFS_VLE_DI_D0_CBLKCNT) {182 if (m->delta[0] & Z_EROFS_VLE_DI_D0_CBLKCNT) {
181 if (!(vi->z_advise & Z_EROFS_ADVISE_BIG_PCLUSTER_1)) {183 if (!(vi->z_advise & (Z_EROFS_ADVISE_BIG_PCLUSTER_1 |
184 Z_EROFS_ADVISE_BIG_PCLUSTER_2))) {
182 DBG_BUGON(1);185 DBG_BUGON(1);
183 return -EFSCORRUPTED;186 return -EFSCORRUPTED;
184 }187 }
@@ -189,7 +192,8 @@ static int legacy_load_cluster_from_disk(struct z_erofs_maprecorder *m, · 4 unmodified lines
189 m->delta[1] = le16_to_cpu(di->di_u.delta[1]);192 m->delta[1] = le16_to_cpu(di->di_u.delta[1]);
190 break;193 break;
191 case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:194 case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
192 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:195 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD1:
196 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD2:
193 m->clusterofs = le16_to_cpu(di->di_clusterofs);197 m->clusterofs = le16_to_cpu(di->di_clusterofs);
194 m->pblk = le32_to_cpu(di->di_u.blkaddr);198 m->pblk = le32_to_cpu(di->di_u.blkaddr);
195 break;199 break;
@@ -446,9 +450,9 @@ static int z_erofs_extent_lookback(struct z_erofs_maprecorder *m, · 250 unmodified lines
446 }450 }
447 return z_erofs_extent_lookback(m, m->delta[0]);451 return z_erofs_extent_lookback(m, m->delta[0]);
448 case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:452 case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
449 map->m_flags &= ~EROFS_MAP_ZIPPED;453 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD1:
450 fallthrough;454 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD2:
451 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:455 m->headtype = m->type;
452 map->m_la = (lcn << lclusterbits) | m->clusterofs;456 map->m_la = (lcn << lclusterbits) | m->clusterofs;
453 break;457 break;
454 default:458 default:
@@ -471,13 +475,18 @@ static int z_erofs_get_extent_compressedlen(struct z_erofs_maprecorder *m, · 16 unmodified lines
471 int err;475 int err;
472 476
473 DBG_BUGON(m->type != Z_EROFS_VLE_CLUSTER_TYPE_PLAIN &&477 DBG_BUGON(m->type != Z_EROFS_VLE_CLUSTER_TYPE_PLAIN &&
474 m->type != Z_EROFS_VLE_CLUSTER_TYPE_HEAD);478 m->type != Z_EROFS_VLE_CLUSTER_TYPE_HEAD1 &&
475 if (!(map->m_flags & EROFS_MAP_ZIPPED) ||479 m->type != Z_EROFS_VLE_CLUSTER_TYPE_HEAD2);
476 !(vi->z_advise & Z_EROFS_ADVISE_BIG_PCLUSTER_1)) {480 DBG_BUGON(m->type != m->headtype);
481
482 if (m->headtype == Z_EROFS_VLE_CLUSTER_TYPE_PLAIN ||
483 ((m->headtype == Z_EROFS_VLE_CLUSTER_TYPE_HEAD1) &&
484 !(vi->z_advise & Z_EROFS_ADVISE_BIG_PCLUSTER_1)) ||
485 ((m->headtype == Z_EROFS_VLE_CLUSTER_TYPE_HEAD2) &&
486 !(vi->z_advise & Z_EROFS_ADVISE_BIG_PCLUSTER_2))) {
477 map->m_plen = 1 << lclusterbits;487 map->m_plen = 1 << lclusterbits;
478 return 0;488 return 0;
479 }489 }
480
481 lcn = m->lcn + 1;490 lcn = m->lcn + 1;
482 if (m->compressedlcs)491 if (m->compressedlcs)
483 goto out;492 goto out;
@@ -499,7 +508,8 @@ static int z_erofs_get_extent_compressedlen(struct z_erofs_maprecorder *m, · 15 unmodified lines
499 508
500 switch (m->type) {509 switch (m->type) {
501 case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:510 case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
502 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:511 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD1:
512 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD2:
503 /*513 /*
504 * if the 1st NONHEAD lcluster is actually PLAIN or HEAD type514 * if the 1st NONHEAD lcluster is actually PLAIN or HEAD type
505 * rather than CBLKCNT, it's a 1 lcluster-sized pcluster.515 * rather than CBLKCNT, it's a 1 lcluster-sized pcluster.
@@ -554,7 +564,8 @@ static int z_erofs_get_extent_decompressedlen(struct z_erofs_maprecorder *m) · 48 unmodified lines
554 DBG_BUGON(!m->delta[1] &&564 DBG_BUGON(!m->delta[1] &&
555 m->clusterofs != 1 << lclusterbits);565 m->clusterofs != 1 << lclusterbits);
556 } else if (m->type == Z_EROFS_VLE_CLUSTER_TYPE_PLAIN ||566 } else if (m->type == Z_EROFS_VLE_CLUSTER_TYPE_PLAIN ||
557 m->type == Z_EROFS_VLE_CLUSTER_TYPE_HEAD) {567 m->type == Z_EROFS_VLE_CLUSTER_TYPE_HEAD1 ||
568 m->type == Z_EROFS_VLE_CLUSTER_TYPE_HEAD2) {
558 /* go on until the next HEAD lcluster */569 /* go on until the next HEAD lcluster */
559 if (lcn != headlcn)570 if (lcn != headlcn)
560 break;571 break;
@@ -609,16 +620,15 @@ int z_erofs_map_blocks_iter(struct inode *inode, · 48 unmodified lines
609 if (err)620 if (err)
610 goto unmap_out;621 goto unmap_out;
611 622
612 map->m_flags = EROFS_MAP_ZIPPED; /* by default, compressed */623 map->m_flags = EROFS_MAP_MAPPED | EROFS_MAP_ENCODED;
613 end = (m.lcn + 1ULL) << lclusterbits;624 end = (m.lcn + 1ULL) << lclusterbits;
614 625
615 switch (m.type) {626 switch (m.type) {
616 case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:627 case Z_EROFS_VLE_CLUSTER_TYPE_PLAIN:
617 if (endoff >= m.clusterofs)628 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD1:
618 map->m_flags &= ~EROFS_MAP_ZIPPED;629 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD2:
619 fallthrough;
620 case Z_EROFS_VLE_CLUSTER_TYPE_HEAD:
621 if (endoff >= m.clusterofs) {630 if (endoff >= m.clusterofs) {
631 m.headtype = m.type;
622 map->m_la = (m.lcn << lclusterbits) | m.clusterofs;632 map->m_la = (m.lcn << lclusterbits) | m.clusterofs;
623 break;633 break;
624 }634 }
@@ -650,13 +660,22 @@ int z_erofs_map_blocks_iter(struct inode *inode, · 25 unmodified lines
650 660
651 map->m_llen = end - map->m_la;661 map->m_llen = end - map->m_la;
652 map->m_pa = blknr_to_addr(m.pblk);662 map->m_pa = blknr_to_addr(m.pblk);
653 map->m_flags |= EROFS_MAP_MAPPED;
654 663
655 err = z_erofs_get_extent_compressedlen(&m, initial_lcn);664 err = z_erofs_get_extent_compressedlen(&m, initial_lcn);
656 if (err)665 if (err)
657 goto out;666 goto out;
658 667
659 if (flags & EROFS_GET_BLOCKS_FIEMAP) {668 if (m.headtype == Z_EROFS_VLE_CLUSTER_TYPE_PLAIN)
669 map->m_algorithmformat = Z_EROFS_COMPRESSION_SHIFTED;
670 else if (m.headtype == Z_EROFS_VLE_CLUSTER_TYPE_HEAD2)
671 map->m_algorithmformat = vi->z_algorithmtype[1];
672 else
673 map->m_algorithmformat = vi->z_algorithmtype[0];
674
675 if ((flags & EROFS_GET_BLOCKS_FIEMAP) ||
676 ((flags & EROFS_GET_BLOCKS_READMORE) &&
677 map->m_algorithmformat == Z_EROFS_COMPRESSION_LZMA &&
678 map->m_llen >= EROFS_BLKSIZ)) {
660 err = z_erofs_get_extent_decompressedlen(&m);679 err = z_erofs_get_extent_decompressedlen(&m);
661 if (!err)680 if (!err)
662 map->m_flags |= EROFS_MAP_FULL_MAPPED;681 map->m_flags |= EROFS_MAP_FULL_MAPPED;
include/linux/xz.h+106−0
@@ -233,6 +233,112 @@ XZ_EXTERN void xz_dec_reset(struct xz_dec *s); · 232 unmodified lines
233 */233 */
234XZ_EXTERN void xz_dec_end(struct xz_dec *s);234XZ_EXTERN void xz_dec_end(struct xz_dec *s);
235 235
236/*
237 * Decompressor for MicroLZMA, an LZMA variant with a very minimal header.
238 * See xz_dec_microlzma_alloc() below for details.
239 *
240 * These functions aren't used or available in preboot code and thus aren't
241 * marked with XZ_EXTERN. This avoids warnings about static functions that
242 * are never defined.
243 */
244/**
245 * struct xz_dec_microlzma - Opaque type to hold the MicroLZMA decoder state
246 */
247struct xz_dec_microlzma;
248
249/**
250 * xz_dec_microlzma_alloc() - Allocate memory for the MicroLZMA decoder
251 * @mode XZ_SINGLE or XZ_PREALLOC
252 * @dict_size LZMA dictionary size. This must be at least 4 KiB and
253 * at most 3 GiB.
254 *
255 * In contrast to xz_dec_init(), this function only allocates the memory
256 * and remembers the dictionary size. xz_dec_microlzma_reset() must be used
257 * before calling xz_dec_microlzma_run().
258 *
259 * The amount of allocated memory is a little less than 30 KiB with XZ_SINGLE.
260 * With XZ_PREALLOC also a dictionary buffer of dict_size bytes is allocated.
261 *
262 * On success, xz_dec_microlzma_alloc() returns a pointer to
263 * struct xz_dec_microlzma. If memory allocation fails or
264 * dict_size is invalid, NULL is returned.
265 *
266 * The compressed format supported by this decoder is a raw LZMA stream
267 * whose first byte (always 0x00) has been replaced with bitwise-negation
268 * of the LZMA properties (lc/lp/pb) byte. For example, if lc/lp/pb is
269 * 3/0/2, the first byte is 0xA2. This way the first byte can never be 0x00.
270 * Just like with LZMA2, lc + lp <= 4 must be true. The LZMA end-of-stream
271 * marker must not be used. The unused values are reserved for future use.
272 * This MicroLZMA header format was created for use in EROFS but may be used
273 * by others too.
274 */
275extern struct xz_dec_microlzma *xz_dec_microlzma_alloc(enum xz_mode mode,
276 uint32_t dict_size);
277
278/**
279 * xz_dec_microlzma_reset() - Reset the MicroLZMA decoder state
280 * @s Decoder state allocated using xz_dec_microlzma_alloc()
281 * @comp_size Compressed size of the input stream
282 * @uncomp_size Uncompressed size of the input stream. A value smaller
283 * than the real uncompressed size of the input stream can
284 * be specified if uncomp_size_is_exact is set to false.
285 * uncomp_size can never be set to a value larger than the
286 * expected real uncompressed size because it would eventually
287 * result in XZ_DATA_ERROR.
288 * @uncomp_size_is_exact This is an int instead of bool to avoid
289 * requiring stdbool.h. This should normally be set to true.
290 * When this is set to false, error detection is weaker.
291 */
292extern void xz_dec_microlzma_reset(struct xz_dec_microlzma *s,
293 uint32_t comp_size, uint32_t uncomp_size,
294 int uncomp_size_is_exact);
295
296/**
297 * xz_dec_microlzma_run() - Run the MicroLZMA decoder
298 * @s Decoder state initialized using xz_dec_microlzma_reset()
299 * @b: Input and output buffers
300 *
301 * This works similarly to xz_dec_run() with a few important differences.
302 * Only the differences are documented here.
303 *
304 * The only possible return values are XZ_OK, XZ_STREAM_END, and
305 * XZ_DATA_ERROR. This function cannot return XZ_BUF_ERROR: if no progress
306 * is possible due to lack of input data or output space, this function will
307 * keep returning XZ_OK. Thus, the calling code must be written so that it
308 * will eventually provide input and output space matching (or exceeding)
309 * comp_size and uncomp_size arguments given to xz_dec_microlzma_reset().
310 * If the caller cannot do this (for example, if the input file is truncated
311 * or otherwise corrupt), the caller must detect this error by itself to
312 * avoid an infinite loop.
313 *
314 * If the compressed data seems to be corrupt, XZ_DATA_ERROR is returned.
315 * This can happen also when incorrect dictionary, uncompressed, or
316 * compressed sizes have been specified.
317 *
318 * With XZ_PREALLOC only: As an extra feature, b->out may be NULL to skip over
319 * uncompressed data. This way the caller doesn't need to provide a temporary
320 * output buffer for the bytes that will be ignored.
321 *
322 * With XZ_SINGLE only: In contrast to xz_dec_run(), the return value XZ_OK
323 * is also possible and thus XZ_SINGLE is actually a limited multi-call mode.
324 * After XZ_OK the bytes decoded so far may be read from the output buffer.
325 * It is possible to continue decoding but the variables b->out and b->out_pos
326 * MUST NOT be changed by the caller. Increasing the value of b->out_size is
327 * allowed to make more output space available; one doesn't need to provide
328 * space for the whole uncompressed data on the first call. The input buffer
329 * may be changed normally like with XZ_PREALLOC. This way input data can be
330 * provided from non-contiguous memory.
331 */
332extern enum xz_ret xz_dec_microlzma_run(struct xz_dec_microlzma *s,
333 struct xz_buf *b);
334
335/**
336 * xz_dec_microlzma_end() - Free the memory allocated for the decoder state
337 * @s: Decoder state allocated using xz_dec_microlzma_alloc().
338 * If s is NULL, this function does nothing.
339 */
340extern void xz_dec_microlzma_end(struct xz_dec_microlzma *s);
341
236/*342/*
237 * Standalone build (userspace build or in-kernel build for boot time use)343 * Standalone build (userspace build or in-kernel build for boot time use)
238 * needs a CRC32 implementation. For normal in-kernel use, kernel's own344 * needs a CRC32 implementation. For normal in-kernel use, kernel's own
include/trace/events/erofs.h+1−1
@@ -24,7 +24,7 @@ struct erofs_map_blocks; · 23 unmodified lines
24#define show_mflags(flags) __print_flags(flags, "", \24#define show_mflags(flags) __print_flags(flags, "", \
25 { EROFS_MAP_MAPPED, "M" }, \25 { EROFS_MAP_MAPPED, "M" }, \
26 { EROFS_MAP_META, "I" }, \26 { EROFS_MAP_META, "I" }, \
27 { EROFS_MAP_ZIPPED, "Z" })27 { EROFS_MAP_ENCODED, "E" })
28 28
29TRACE_EVENT(erofs_lookup,29TRACE_EVENT(erofs_lookup,
30 30
lib/decompress_unxz.c+5−5
@@ -20,8 +20,8 @@ · 19 unmodified lines
20 *20 *
21 * The worst case for in-place decompression is that the beginning of21 * The worst case for in-place decompression is that the beginning of
22 * the file is compressed extremely well, and the rest of the file is22 * the file is compressed extremely well, and the rest of the file is
23 * uncompressible. Thus, we must look for worst-case expansion when the23 * incompressible. Thus, we must look for worst-case expansion when the
24 * compressor is encoding uncompressible data.24 * compressor is encoding incompressible data.
25 *25 *
26 * The structure of the .xz file in case of a compressed kernel is as follows.26 * The structure of the .xz file in case of a compressed kernel is as follows.
27 * Sizes (as bytes) of the fields are in parenthesis.27 * Sizes (as bytes) of the fields are in parenthesis.
@@ -58,7 +58,7 @@ · 30 unmodified lines
58 * uncompressed size of the payload is in practice never less than the58 * uncompressed size of the payload is in practice never less than the
59 * payload size itself. The LZMA2 format would allow uncompressed size59 * payload size itself. The LZMA2 format would allow uncompressed size
60 * to be less than the payload size, but no sane compressor creates such60 * to be less than the payload size, but no sane compressor creates such
61 * files. LZMA2 supports storing uncompressible data in uncompressed form,61 * files. LZMA2 supports storing incompressible data in uncompressed form,
62 * so there's never a need to create payloads whose uncompressed size is62 * so there's never a need to create payloads whose uncompressed size is
63 * smaller than the compressed size.63 * smaller than the compressed size.
64 *64 *
@@ -167,8 +167,8 @@ · 102 unmodified lines
167 * memeq and memzero are not used much and any remotely sane implementation167 * memeq and memzero are not used much and any remotely sane implementation
168 * is fast enough. memcpy/memmove speed matters in multi-call mode, but168 * is fast enough. memcpy/memmove speed matters in multi-call mode, but
169 * the kernel image is decompressed in single-call mode, in which only169 * the kernel image is decompressed in single-call mode, in which only
170 * memcpy speed can matter and only if there is a lot of uncompressible data170 * memmove speed can matter and only if there is a lot of incompressible data
171 * (LZMA2 stores uncompressible chunks in uncompressed form). Thus, the171 * (LZMA2 stores incompressible chunks in uncompressed form). Thus, the
172 * functions below should just be kept small; it's probably not worth172 * functions below should just be kept small; it's probably not worth
173 * optimizing for speed.173 * optimizing for speed.
174 */174 */
lib/xz/Kconfig+13−0
@@ -39,6 +39,19 @@ config XZ_DEC_SPARC · 38 unmodified lines
39 default y39 default y
40 select XZ_DEC_BCJ40 select XZ_DEC_BCJ
41 41
42config XZ_DEC_MICROLZMA
43 bool "MicroLZMA decoder"
44 default n
45 help
46 MicroLZMA is a header format variant where the first byte
47 of a raw LZMA stream (without the end of stream marker) has
48 been replaced with a bitwise-negation of the lc/lp/pb
49 properties byte. MicroLZMA was created to be used in EROFS
50 but can be used by other things too where wasting minimal
51 amount of space for headers is important.
52
53 Unless you know that you need this, say N.
54
42endif55endif
43 56
44config XZ_DEC_BCJ57config XZ_DEC_BCJ
lib/xz/xz_dec_lzma2.c+175−7
@@ -248,6 +248,10 @@ struct lzma2_dec { · 247 unmodified lines
248 * before the first LZMA chunk.248 * before the first LZMA chunk.
249 */249 */
250 bool need_props;250 bool need_props;
251
252#ifdef XZ_DEC_MICROLZMA
253 bool pedantic_microlzma;
254#endif
251};255};
252 256
253struct xz_dec_lzma2 {257struct xz_dec_lzma2 {
@@ -387,7 +391,14 @@ static void dict_uncompressed(struct dictionary *dict, struct xz_buf *b, · 133 unmodified lines
387 391
388 *left -= copy_size;392 *left -= copy_size;
389 393
390 memcpy(dict->buf + dict->pos, b->in + b->in_pos, copy_size);394 /*
395 * If doing in-place decompression in single-call mode and the
396 * uncompressed size of the file is larger than the caller
397 * thought (i.e. it is invalid input!), the buffers below may
398 * overlap and cause undefined behavior with memcpy().
399 * With valid inputs memcpy() would be fine here.
400 */
401 memmove(dict->buf + dict->pos, b->in + b->in_pos, copy_size);
391 dict->pos += copy_size;402 dict->pos += copy_size;
392 403
393 if (dict->full < dict->pos)404 if (dict->full < dict->pos)
@@ -397,7 +408,11 @@ static void dict_uncompressed(struct dictionary *dict, struct xz_buf *b, · 3 unmodified lines
397 if (dict->pos == dict->end)408 if (dict->pos == dict->end)
398 dict->pos = 0;409 dict->pos = 0;
399 410
400 memcpy(b->out + b->out_pos, b->in + b->in_pos,411 /*
412 * Like above but for multi-call mode: use memmove()
413 * to avoid undefined behavior with invalid input.
414 */
415 memmove(b->out + b->out_pos, b->in + b->in_pos,
401 copy_size);416 copy_size);
402 }417 }
403 418
@@ -408,6 +423,12 @@ static void dict_uncompressed(struct dictionary *dict, struct xz_buf *b, · 4 unmodified lines
408 }423 }
409}424}
410 425
426#ifdef XZ_DEC_MICROLZMA
427# define DICT_FLUSH_SUPPORTS_SKIPPING true
428#else
429# define DICT_FLUSH_SUPPORTS_SKIPPING false
430#endif
431
411/*432/*
412 * Flush pending data from dictionary to b->out. It is assumed that there is433 * Flush pending data from dictionary to b->out. It is assumed that there is
413 * enough space in b->out. This is guaranteed because caller uses dict_limit()434 * enough space in b->out. This is guaranteed because caller uses dict_limit()
@@ -421,8 +442,19 @@ static uint32_t dict_flush(struct dictionary *dict, struct xz_buf *b) · 7 unmodified lines
421 if (dict->pos == dict->end)442 if (dict->pos == dict->end)
422 dict->pos = 0;443 dict->pos = 0;
423 444
424 memcpy(b->out + b->out_pos, dict->buf + dict->start,445 /*
425 copy_size);446 * These buffers cannot overlap even if doing in-place
447 * decompression because in multi-call mode dict->buf
448 * has been allocated by us in this file; it's not
449 * provided by the caller like in single-call mode.
450 *
451 * With MicroLZMA, b->out can be NULL to skip bytes that
452 * the caller doesn't need. This cannot be done with XZ
453 * because it would break BCJ filters.
454 */
455 if (!DICT_FLUSH_SUPPORTS_SKIPPING || b->out != NULL)
456 memcpy(b->out + b->out_pos, dict->buf + dict->start,
457 copy_size);
426 }458 }
427 459
428 dict->start = dict->pos;460 dict->start = dict->pos;
@@ -488,7 +520,7 @@ static __always_inline void rc_normalize(struct rc_dec *rc) · 59 unmodified lines
488 * functions so that the compiler is supposed to be able to more easily avoid520 * functions so that the compiler is supposed to be able to more easily avoid
489 * an extra branch. In this particular version of the LZMA decoder, this521 * an extra branch. In this particular version of the LZMA decoder, this
490 * doesn't seem to be a good idea (tested with GCC 3.3.6, 3.4.6, and 4.3.3522 * doesn't seem to be a good idea (tested with GCC 3.3.6, 3.4.6, and 4.3.3
491 * on x86). Using a non-splitted version results in nicer looking code too.523 * on x86). Using a non-split version results in nicer looking code too.
492 *524 *
493 * NOTE: This must return an int. Do not make it return a bool or the speed525 * NOTE: This must return an int. Do not make it return a bool or the speed
494 * of the code generated by GCC 3.x decreases 10-15 %. (GCC 4.3 doesn't care,526 * of the code generated by GCC 3.x decreases 10-15 %. (GCC 4.3 doesn't care,
@@ -774,6 +806,7 @@ static void lzma_reset(struct xz_dec_lzma2 *s) · 279 unmodified lines
774 s->lzma.rep1 = 0;806 s->lzma.rep1 = 0;
775 s->lzma.rep2 = 0;807 s->lzma.rep2 = 0;
776 s->lzma.rep3 = 0;808 s->lzma.rep3 = 0;
809 s->lzma.len = 0;
777 810
778 /*811 /*
779 * All probabilities are initialized to the same value. This hack812 * All probabilities are initialized to the same value. This hack
@@ -1157,8 +1190,6 @@ XZ_EXTERN enum xz_ret xz_dec_lzma2_reset(struct xz_dec_lzma2 *s, uint8_t props) · 377 unmodified lines
1157 }1190 }
1158 }1191 }
1159 1192
1160 s->lzma.len = 0;
1161
1162 s->lzma2.sequence = SEQ_CONTROL;1193 s->lzma2.sequence = SEQ_CONTROL;
1163 s->lzma2.need_dict_reset = true;1194 s->lzma2.need_dict_reset = true;
1164 1195
@@ -1174,3 +1205,140 @@ XZ_EXTERN void xz_dec_lzma2_end(struct xz_dec_lzma2 *s) · 9 unmodified lines
1174 1205
1175 kfree(s);1206 kfree(s);
1176}1207}
1208
1209#ifdef XZ_DEC_MICROLZMA
1210/* This is a wrapper struct to have a nice struct name in the public API. */
1211struct xz_dec_microlzma {
1212 struct xz_dec_lzma2 s;
1213};
1214
1215enum xz_ret xz_dec_microlzma_run(struct xz_dec_microlzma *s_ptr,
1216 struct xz_buf *b)
1217{
1218 struct xz_dec_lzma2 *s = &s_ptr->s;
1219
1220 /*
1221 * sequence is SEQ_PROPERTIES before the first input byte,
1222 * SEQ_LZMA_PREPARE until a total of five bytes have been read,
1223 * and SEQ_LZMA_RUN for the rest of the input stream.
1224 */
1225 if (s->lzma2.sequence != SEQ_LZMA_RUN) {
1226 if (s->lzma2.sequence == SEQ_PROPERTIES) {
1227 /* One byte is needed for the props. */
1228 if (b->in_pos >= b->in_size)
1229 return XZ_OK;
1230
1231 /*
1232 * Don't increment b->in_pos here. The same byte is
1233 * also passed to rc_read_init() which will ignore it.
1234 */
1235 if (!lzma_props(s, ~b->in[b->in_pos]))
1236 return XZ_DATA_ERROR;
1237
1238 s->lzma2.sequence = SEQ_LZMA_PREPARE;
1239 }
1240
1241 /*
1242 * xz_dec_microlzma_reset() doesn't validate the compressed
1243 * size so we do it here. We have to limit the maximum size
1244 * to avoid integer overflows in lzma2_lzma(). 3 GiB is a nice
1245 * round number and much more than users of this code should
1246 * ever need.
1247 */
1248 if (s->lzma2.compressed < RC_INIT_BYTES
1249 || s->lzma2.compressed > (3U << 30))
1250 return XZ_DATA_ERROR;
1251
1252 if (!rc_read_init(&s->rc, b))
1253 return XZ_OK;
1254
1255 s->lzma2.compressed -= RC_INIT_BYTES;
1256 s->lzma2.sequence = SEQ_LZMA_RUN;
1257
1258 dict_reset(&s->dict, b);
1259 }
1260
1261 /* This is to allow increasing b->out_size between calls. */
1262 if (DEC_IS_SINGLE(s->dict.mode))
1263 s->dict.end = b->out_size - b->out_pos;
1264
1265 while (true) {
1266 dict_limit(&s->dict, min_t(size_t, b->out_size - b->out_pos,
1267 s->lzma2.uncompressed));
1268
1269 if (!lzma2_lzma(s, b))
1270 return XZ_DATA_ERROR;
1271
1272 s->lzma2.uncompressed -= dict_flush(&s->dict, b);
1273
1274 if (s->lzma2.uncompressed == 0) {
1275 if (s->lzma2.pedantic_microlzma) {
1276 if (s->lzma2.compressed > 0 || s->lzma.len > 0
1277 || !rc_is_finished(&s->rc))
1278 return XZ_DATA_ERROR;
1279 }
1280
1281 return XZ_STREAM_END;
1282 }
1283
1284 if (b->out_pos == b->out_size)
1285 return XZ_OK;
1286
1287 if (b->in_pos == b->in_size
1288 && s->temp.size < s->lzma2.compressed)
1289 return XZ_OK;
1290 }
1291}
1292
1293struct xz_dec_microlzma *xz_dec_microlzma_alloc(enum xz_mode mode,
1294 uint32_t dict_size)
1295{
1296 struct xz_dec_microlzma *s;
1297
1298 /* Restrict dict_size to the same range as in the LZMA2 code. */
1299 if (dict_size < 4096 || dict_size > (3U << 30))
1300 return NULL;
1301
1302 s = kmalloc(sizeof(*s), GFP_KERNEL);
1303 if (s == NULL)
1304 return NULL;
1305
1306 s->s.dict.mode = mode;
1307 s->s.dict.size = dict_size;
1308
1309 if (DEC_IS_MULTI(mode)) {
1310 s->s.dict.end = dict_size;
1311
1312 s->s.dict.buf = vmalloc(dict_size);
1313 if (s->s.dict.buf == NULL) {
1314 kfree(s);
1315 return NULL;
1316 }
1317 }
1318
1319 return s;
1320}
1321
1322void xz_dec_microlzma_reset(struct xz_dec_microlzma *s, uint32_t comp_size,
1323 uint32_t uncomp_size, int uncomp_size_is_exact)
1324{
1325 /*
1326 * comp_size is validated in xz_dec_microlzma_run().
1327 * uncomp_size can safely be anything.
1328 */
1329 s->s.lzma2.compressed = comp_size;
1330 s->s.lzma2.uncompressed = uncomp_size;
1331 s->s.lzma2.pedantic_microlzma = uncomp_size_is_exact;
1332
1333 s->s.lzma2.sequence = SEQ_PROPERTIES;
1334 s->s.temp.size = 0;
1335}
1336
1337void xz_dec_microlzma_end(struct xz_dec_microlzma *s)
1338{
1339 if (DEC_IS_MULTI(s->s.dict.mode))
1340 vfree(s->s.dict.buf);
1341
1342 kfree(s);
1343}
1344#endif
lib/xz/xz_dec_stream.c+3−3
@@ -402,12 +402,12 @@ static enum xz_ret dec_stream_header(struct xz_dec *s) · 401 unmodified lines
402 * we will accept other check types too, but then the check won't402 * we will accept other check types too, but then the check won't
403 * be verified and a warning (XZ_UNSUPPORTED_CHECK) will be given.403 * be verified and a warning (XZ_UNSUPPORTED_CHECK) will be given.
404 */404 */
405 if (s->temp.buf[HEADER_MAGIC_SIZE + 1] > XZ_CHECK_MAX)
406 return XZ_OPTIONS_ERROR;
407
405 s->check_type = s->temp.buf[HEADER_MAGIC_SIZE + 1];408 s->check_type = s->temp.buf[HEADER_MAGIC_SIZE + 1];
406 409
407#ifdef XZ_DEC_ANY_CHECK410#ifdef XZ_DEC_ANY_CHECK
408 if (s->check_type > XZ_CHECK_MAX)
409 return XZ_OPTIONS_ERROR;
410
411 if (s->check_type > XZ_CHECK_CRC32)411 if (s->check_type > XZ_CHECK_CRC32)
412 return XZ_UNSUPPORTED_CHECK;412 return XZ_UNSUPPORTED_CHECK;
413#else413#else
lib/xz/xz_dec_syms.c+8−1
@@ -15,8 +15,15 @@ EXPORT_SYMBOL(xz_dec_reset); · 14 unmodified lines
15EXPORT_SYMBOL(xz_dec_run);15EXPORT_SYMBOL(xz_dec_run);
16EXPORT_SYMBOL(xz_dec_end);16EXPORT_SYMBOL(xz_dec_end);
17 17
18#ifdef CONFIG_XZ_DEC_MICROLZMA
19EXPORT_SYMBOL(xz_dec_microlzma_alloc);
20EXPORT_SYMBOL(xz_dec_microlzma_reset);
21EXPORT_SYMBOL(xz_dec_microlzma_run);
22EXPORT_SYMBOL(xz_dec_microlzma_end);
23#endif
24
18MODULE_DESCRIPTION("XZ decompressor");25MODULE_DESCRIPTION("XZ decompressor");
19MODULE_VERSION("1.0");26MODULE_VERSION("1.1");
20MODULE_AUTHOR("Lasse Collin <lasse.collin@tukaani.org> and Igor Pavlov");27MODULE_AUTHOR("Lasse Collin <lasse.collin@tukaani.org> and Igor Pavlov");
21 28
22/*29/*
lib/xz/xz_private.h+3−0
@@ -37,6 +37,9 @@ · 36 unmodified lines
37# ifdef CONFIG_XZ_DEC_SPARC37# ifdef CONFIG_XZ_DEC_SPARC
38# define XZ_DEC_SPARC38# define XZ_DEC_SPARC
39# endif39# endif
40# ifdef CONFIG_XZ_DEC_MICROLZMA
41# define XZ_DEC_MICROLZMA
42# endif
40# define memeq(a, b, size) (memcmp(a, b, size) == 0)43# define memeq(a, b, size) (memcmp(a, b, size) == 0)
41# define memzero(buf, size) memset(buf, 0, size)44# define memzero(buf, size) memset(buf, 0, size)
42# endif45# endif

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t