History

Merge tag 'erofs-for-5.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs

Pull erofs updates from Gao Xiang: "There are some new features available for this cycle. Firstly, EROFS LZMA algorithm support, specifically called MicroLZMA, is available as an option for embedded devices, LiveCDs and/or as the secondary auxiliary compression algorithm besides the primary algorithm in one file. In order to better support the LZMA fixed-sized output compression, especially for 4KiB pcluster size (which has lowest memory pressure thus useful for memory-sensitive scenarios), Lasse introduced a new LZMA header/container format called MicroLZMA to minimize the original LZMA1 header (for example, we don't need to waste 4-byte dictionary size and another 8-byte uncompressed size, which can be calculated by fs directly, for each pcluster) and enable EROFS fixed-sized output compression. Note that MicroLZMA can also be later used by other things in addition to EROFS too where wasting minimal amount of space for headers is important and it can be only compiled by enabling XZ_DEC_MICROLZMA. MicroLZMA has been supported by the latest upstream XZ embedded [1] & XZ utils [2], apply the latest related XZ embedded upstream patches by the XZ author Lasse here. Secondly, multiple device is also supported in this cycle, which is designed for multi-layer container images. By working together with inter-layer data deduplication and compression, we can achieve the next high-performance container image solution. Our team will announce the new Nydus container image service [3] implementation with new RAFS v6 (EROFS-compatible) format in Open Source Summit 2021 China [4] soon. Besides, the secondary compression head support and readmore decompression strategy are also included in this cycle. There are also some minor bugfixes and cleanups, as always. Summary: - support multiple devices for multi-layer container images; - support the secondary compression head; - support readmore decompression strategy; - support new LZMA algorithm (specifically called MicroLZMA); - some bugfixes & cleanups" * tag 'erofs-for-5.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs: erofs: don't trigger WARN() when decompression fails erofs: get rid of ->lru usage erofs: lzma compression support erofs: rename some generic methods in decompressor lib/xz, lib/decompress_unxz.c: Fix spelling in comments lib/xz: Add MicroLZMA decoder lib/xz: Move s->lzma.len = 0 initialization to lzma_reset() lib/xz: Validate the value before assigning it to an enum variable lib/xz: Avoid overlapping memcpy() with invalid input with in-place decompression erofs: introduce readmore decompression strategy erofs: introduce the secondary compression head erofs: get compression algorithms directly on mapping erofs: add multiple device support erofs: decouple basic mount options from fs_context erofs: remove the fast path of per-CPU buffer decompression

Linus Torvalds committed 5 years ago

Commit 67a135b2 parents cd3e8ea a0961f325 files · +1281 −320Browse files

Showing the changes to include (2 of 25 files). Show all

Unified · Split

2 files
  1. include/linux/xz.h +106 −0
  2. include/trace/events/erofs.h +1 −1
include/linux/xz.h+106−0
@@ -233,6 +233,112 @@ XZ_EXTERN void xz_dec_reset(struct xz_dec *s); · 232 unmodified lines
233 */233 */
234XZ_EXTERN void xz_dec_end(struct xz_dec *s);234XZ_EXTERN void xz_dec_end(struct xz_dec *s);
235 235
236/*
237 * Decompressor for MicroLZMA, an LZMA variant with a very minimal header.
238 * See xz_dec_microlzma_alloc() below for details.
239 *
240 * These functions aren't used or available in preboot code and thus aren't
241 * marked with XZ_EXTERN. This avoids warnings about static functions that
242 * are never defined.
243 */
244/**
245 * struct xz_dec_microlzma - Opaque type to hold the MicroLZMA decoder state
246 */
247struct xz_dec_microlzma;
248
249/**
250 * xz_dec_microlzma_alloc() - Allocate memory for the MicroLZMA decoder
251 * @mode XZ_SINGLE or XZ_PREALLOC
252 * @dict_size LZMA dictionary size. This must be at least 4 KiB and
253 * at most 3 GiB.
254 *
255 * In contrast to xz_dec_init(), this function only allocates the memory
256 * and remembers the dictionary size. xz_dec_microlzma_reset() must be used
257 * before calling xz_dec_microlzma_run().
258 *
259 * The amount of allocated memory is a little less than 30 KiB with XZ_SINGLE.
260 * With XZ_PREALLOC also a dictionary buffer of dict_size bytes is allocated.
261 *
262 * On success, xz_dec_microlzma_alloc() returns a pointer to
263 * struct xz_dec_microlzma. If memory allocation fails or
264 * dict_size is invalid, NULL is returned.
265 *
266 * The compressed format supported by this decoder is a raw LZMA stream
267 * whose first byte (always 0x00) has been replaced with bitwise-negation
268 * of the LZMA properties (lc/lp/pb) byte. For example, if lc/lp/pb is
269 * 3/0/2, the first byte is 0xA2. This way the first byte can never be 0x00.
270 * Just like with LZMA2, lc + lp <= 4 must be true. The LZMA end-of-stream
271 * marker must not be used. The unused values are reserved for future use.
272 * This MicroLZMA header format was created for use in EROFS but may be used
273 * by others too.
274 */
275extern struct xz_dec_microlzma *xz_dec_microlzma_alloc(enum xz_mode mode,
276 uint32_t dict_size);
277
278/**
279 * xz_dec_microlzma_reset() - Reset the MicroLZMA decoder state
280 * @s Decoder state allocated using xz_dec_microlzma_alloc()
281 * @comp_size Compressed size of the input stream
282 * @uncomp_size Uncompressed size of the input stream. A value smaller
283 * than the real uncompressed size of the input stream can
284 * be specified if uncomp_size_is_exact is set to false.
285 * uncomp_size can never be set to a value larger than the
286 * expected real uncompressed size because it would eventually
287 * result in XZ_DATA_ERROR.
288 * @uncomp_size_is_exact This is an int instead of bool to avoid
289 * requiring stdbool.h. This should normally be set to true.
290 * When this is set to false, error detection is weaker.
291 */
292extern void xz_dec_microlzma_reset(struct xz_dec_microlzma *s,
293 uint32_t comp_size, uint32_t uncomp_size,
294 int uncomp_size_is_exact);
295
296/**
297 * xz_dec_microlzma_run() - Run the MicroLZMA decoder
298 * @s Decoder state initialized using xz_dec_microlzma_reset()
299 * @b: Input and output buffers
300 *
301 * This works similarly to xz_dec_run() with a few important differences.
302 * Only the differences are documented here.
303 *
304 * The only possible return values are XZ_OK, XZ_STREAM_END, and
305 * XZ_DATA_ERROR. This function cannot return XZ_BUF_ERROR: if no progress
306 * is possible due to lack of input data or output space, this function will
307 * keep returning XZ_OK. Thus, the calling code must be written so that it
308 * will eventually provide input and output space matching (or exceeding)
309 * comp_size and uncomp_size arguments given to xz_dec_microlzma_reset().
310 * If the caller cannot do this (for example, if the input file is truncated
311 * or otherwise corrupt), the caller must detect this error by itself to
312 * avoid an infinite loop.
313 *
314 * If the compressed data seems to be corrupt, XZ_DATA_ERROR is returned.
315 * This can happen also when incorrect dictionary, uncompressed, or
316 * compressed sizes have been specified.
317 *
318 * With XZ_PREALLOC only: As an extra feature, b->out may be NULL to skip over
319 * uncompressed data. This way the caller doesn't need to provide a temporary
320 * output buffer for the bytes that will be ignored.
321 *
322 * With XZ_SINGLE only: In contrast to xz_dec_run(), the return value XZ_OK
323 * is also possible and thus XZ_SINGLE is actually a limited multi-call mode.
324 * After XZ_OK the bytes decoded so far may be read from the output buffer.
325 * It is possible to continue decoding but the variables b->out and b->out_pos
326 * MUST NOT be changed by the caller. Increasing the value of b->out_size is
327 * allowed to make more output space available; one doesn't need to provide
328 * space for the whole uncompressed data on the first call. The input buffer
329 * may be changed normally like with XZ_PREALLOC. This way input data can be
330 * provided from non-contiguous memory.
331 */
332extern enum xz_ret xz_dec_microlzma_run(struct xz_dec_microlzma *s,
333 struct xz_buf *b);
334
335/**
336 * xz_dec_microlzma_end() - Free the memory allocated for the decoder state
337 * @s: Decoder state allocated using xz_dec_microlzma_alloc().
338 * If s is NULL, this function does nothing.
339 */
340extern void xz_dec_microlzma_end(struct xz_dec_microlzma *s);
341
236/*342/*
237 * Standalone build (userspace build or in-kernel build for boot time use)343 * Standalone build (userspace build or in-kernel build for boot time use)
238 * needs a CRC32 implementation. For normal in-kernel use, kernel's own344 * needs a CRC32 implementation. For normal in-kernel use, kernel's own
include/trace/events/erofs.h+1−1
@@ -24,7 +24,7 @@ struct erofs_map_blocks; · 23 unmodified lines
24#define show_mflags(flags) __print_flags(flags, "", \24#define show_mflags(flags) __print_flags(flags, "", \
25 { EROFS_MAP_MAPPED, "M" }, \25 { EROFS_MAP_MAPPED, "M" }, \
26 { EROFS_MAP_META, "I" }, \26 { EROFS_MAP_META, "I" }, \
27 { EROFS_MAP_ZIPPED, "Z" })27 { EROFS_MAP_ENCODED, "E" })
28 28
29TRACE_EVENT(erofs_lookup,29TRACE_EVENT(erofs_lookup,
30 30

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t