================RAID 4/5/6 cache================ Raid 4/5/6 could include an extra disk for data cache besides normal RAIDdisks. The role of RAID disks isn't changed with the cache disk. The cache diskcaches data to the RAID disks. The cache can be in write-through (supportedsince 4.4) or write-back mode (supported since 4.10). mdadm (supported since3.4) has a new option '--write-journal' to create array with cache. Pleaserefer to mdadm manual for details. By default (RAID array starts), the cache isin write-through mode. A user can switch it to write-back mode by:: 	echo "write-back" > /sys/block/md0/md/journal_mode And switch it back to write-through mode by:: 	echo "write-through" > /sys/block/md0/md/journal_mode In both modes, all writes to the array will hit cache disk first. This meansthe cache disk must be fast and sustainable. write-through mode================== This mode mainly fixes the 'write hole' issue. For RAID 4/5/6 array, an uncleanshutdown can cause data in some stripes to not be in consistent state, eg, dataand parity don't match. The reason is that a stripe write involves several RAIDdisks and it's possible the writes don't hit all RAID disks yet before theunclean shutdown. We call an array degraded if it has inconsistent data. MDtries to resync the array to bring it back to normal state. But before theresync completes, any system crash will expose the chance of real datacorruption in the RAID array. This problem is called 'write hole'. The write-through cache will cache all data on cache disk first. After the datais safe on the cache disk, the data will be flushed onto RAID disks. Thetwo-step write will guarantee MD can recover correct data after uncleanshutdown even the array is degraded. Thus the cache can close the 'write hole'. In write-through mode, MD reports IO completion to upper layer (usuallyfilesystems) after the data is safe on RAID disks, so cache disk failuredoesn't cause data loss. Of course cache disk failure means the array isexposed to 'write hole' again. In write-through mode, the cache disk isn't required to be big. Severalhundreds megabytes are enough. write-back mode=============== write-back mode fixes the 'write hole' issue too, since all write data iscached on cache disk. But the main goal of 'write-back' cache is to speed upwrite. If a write crosses all RAID disks of a stripe, we call it full-stripewrite. For non-full-stripe writes, MD must read old data before the new paritycan be calculated. These synchronous reads hurt write throughput. Some writeswhich are sequential but not dispatched in the same time will suffer from thisoverhead too. Write-back cache will aggregate the data and flush the data toRAID disks only after the data becomes a full stripe write. This willcompletely avoid the overhead, so it's very helpful for some workloads. Atypical workload which does sequential write followed by fsync is an example. In write-back mode, MD reports IO completion to upper layer (usuallyfilesystems) right after the data hits cache disk. The data is flushed to raiddisks later after specific conditions met. So cache disk failure will causedata loss. In write-back mode, MD also caches data in memory. The memory cache includesthe same data stored on cache disk, so a power loss doesn't cause data loss.The memory cache size has performance impact for the array. It's recommendedthe size is big. A user can configure the size by:: 	echo "2048" > /sys/block/md0/md/stripe_cache_size Too small cache disk will make the write aggregation less efficient in thismode depending on the workloads. It's recommended to use a cache disk with atleast several gigabytes size in write-back mode. The implementation================== The write-through and write-back cache use the same disk format. The cache diskis organized as a simple write log. The log consists of 'meta data' and 'data'pairs. The meta data describes the data. It also includes checksum and sequenceID for recovery identification. Data can be IO data and parity data. Data ischecksummed too. The checksum is stored in the meta data ahead of the data. Thechecksum is an optimization because MD can write meta and data freely withoutworry about the order. MD superblock has a field pointed to the valid meta dataof log head. The log implementation is pretty straightforward. The difficult part is theorder in which MD writes data to cache disk and RAID disks. Specifically, inwrite-through mode, MD calculates parity for IO data, writes both IO data andparity to the log, writes the data and parity to RAID disks after the data andparity is settled down in log and finally the IO is finished. Read just readsfrom raid disks as usual. In write-back mode, MD writes IO data to the log and reports IO completion. Thedata is also fully cached in memory at that time, which means read must querymemory cache. If some conditions are met, MD will flush the data to RAID disks.MD will calculate parity for the data and write parity into the log. After thisis finished, MD will write both data and parity into RAID disks, then MD canrelease the memory cache. The flush conditions could be stripe becomes a fullstripe write, free cache disk space is low or free in-kernel memory cache spaceis low. After an unclean shutdown, MD does recovery. MD reads all meta data and datafrom the log. The sequence ID and checksum will help us detect corrupted metadata and data. If MD finds a stripe with data and valid parities (1 parity forraid4/5 and 2 for raid6), MD will write the data and parities to RAID disks. Ifparities are incompleted, they are discarded. If part of data is corrupted,they are discarded too. MD then loads valid data and writes them to RAID disksin normal way.