When fs-verity verifies data pages, currently it reads each Merkle tree page synchronously using read_mapping_page(). Therefore, when the Merkle tree pages aren't already cached, fs-verity causes an extra 4 KiB I/O request for every 512 KiB of data (assuming that the Merkle tree uses SHA-256 and 4 KiB blocks). This results in more I/O requests and performance loss than is strictly necessary. Therefore, implement readahead of the Merkle tree pages. For simplicity, we take advantage of the fact that the kernel already does readahead of the file's *data*, just like it does for any other file. Due to this, we don't really need a separate readahead state (struct file_ra_state) just for the Merkle tree, but rather we just need to piggy-back on the existing data readahead requests. We also only really need to bother with the first level of the Merkle tree, since the usual fan-out factor is 128, so normally over 99% of Merkle tree I/O requests are for the first level. Therefore, make fsverity_verify_bio() enable readahead of the first Merkle tree level, for up to 1/4 the number of pages in the bio, when it sees that the REQ_RAHEAD flag is set on the bio. The readahead size is then passed down to ->read_merkle_tree_page() for the filesystem to (optionally) implement if it sees that the requested page is uncached. While we're at it, also make build_merkle_tree_level() set the Merkle tree readahead size, since it's easy to do there. However, for now don't set the readahead size in fsverity_verify_page(), since currently it's only used to verify holes on ext4 and f2fs, and it would need parameters added to know how much to read ahead. This patch significantly improves fs-verity sequential read performance. Some quick benchmarks with 'cat'-ing a 250MB file after dropping caches: On an ARM64 phone (using sha256-ce): Before: 217 MB/s After: 263 MB/s (compare to sha256sum of non-verity file: 357 MB/s) In an x86_64 VM (using sha256-avx2): Before: 173 MB/s After: 215 MB/s (compare to sha256sum of non-verity file: 223 MB/s)
Showing the changes to include/linux (1 of 9 files). Show all
Unified · Split
| @@ -77,6 +77,10 @@ struct fsverity_operations { · 76 unmodified lines | ||
| 77 | 77 | * |
| 78 | 78 | * @inode: the inode |
| 79 | 79 | * @index: 0-based index of the page within the Merkle tree |
| 80 | + * @num_ra_pages: The number of Merkle tree pages that should be | |
| 81 | + * prefetched starting at @index if the page at @index | |
| 82 | + * isn't already cached. Implementations may ignore this | |
| 83 | + * argument; it's only a performance optimization. | |
| 80 | 84 | * |
| 81 | 85 | * This can be called at any time on an open verity file, as well as |
| 82 | 86 | * between ->begin_enable_verity() and ->end_enable_verity(). It may be |
| @@ -87,7 +91,8 @@ struct fsverity_operations { · 4 unmodified lines | ||
| 87 | 91 | * Return: the page on success, ERR_PTR() on failure |
| 88 | 92 | */ |
| 89 | 93 | struct page *(*read_merkle_tree_page)(struct inode *inode, |
| 90 | − pgoff_t index); | |
| 94 | + pgoff_t index, | |
| 95 | + unsigned long num_ra_pages); | |
| 91 | 96 | |
| 92 | 97 | /** |
| 93 | 98 | * Write a Merkle tree block to the given inode. |