=====Cache===== Introduction============ dm-cache is a device mapper target written by Joe Thornber, HeinzMauelshagen, and Mike Snitzer. It aims to improve performance of a block device (eg, a spindle) bydynamically migrating some of its data to a faster, smaller device(eg, an SSD). This device-mapper solution allows us to insert this caching atdifferent levels of the dm stack, for instance above the data device fora thin-provisioning pool.  Caching solutions that are integrated moreclosely with the virtual memory system should give better performance. The target reuses the metadata library used in the thin-provisioninglibrary. The decision as to what data to migrate and when is left to a plug-inpolicy module.  Several of these have been written as we experiment,and we hope other people will contribute others for specific ioscenarios (eg. a vm image server). Glossary========   Migration	       Movement of the primary copy of a logical block from one	       device to the other.  Promotion	       Migration from slow device to fast device.  Demotion	       Migration from fast device to slow device. The origin device always contains a copy of the logical block, whichmay be out of date or kept in sync with the copy on the cache device(depending on policy). Design====== Sub-devices----------- The target is constructed by passing three devices to it (along withother parameters detailed later): 1. An origin device - the big, slow one. 2. A cache device - the small, fast one. 3. A small metadata device - records which blocks are in the cache,   which are dirty, and extra hints for use by the policy object.   This information could be put on the cache device, but having it   separate allows the volume manager to configure it differently,   e.g. as a mirror for extra robustness.  This metadata device may only   be used by a single cache device. Fixed block size---------------- The origin is divided up into blocks of a fixed size.  This block sizeis configurable when you first create the cache.  Typically we've beenusing block sizes of 256KB - 1024KB.  The block size must be between 64sectors (32KB) and 2097152 sectors (1GB) and a multiple of 64 sectors (32KB). Having a fixed block size simplifies the target a lot.  But it issomething of a compromise.  For instance, a small part of a block may begetting hit a lot, yet the whole block will be promoted to the cache.So large block sizes are bad because they waste cache space.  And smallblock sizes are bad because they increase the amount of metadata (bothin core and on disk). Cache operating modes--------------------- The cache has three operating modes: writeback, writethrough andpassthrough. If writeback, the default, is selected then a write to a block that iscached will go only to the cache and the block will be marked dirty inthe metadata. If writethrough is selected then a write to a cached block will notcomplete until it has hit both the origin and cache devices.  Cleanblocks should remain clean. If passthrough is selected, useful when the cache contents are not knownto be coherent with the origin device, then all reads are served fromthe origin device (all reads miss the cache) and all writes areforwarded to the origin device; additionally, write hits cause cacheblock invalidates.  To enable passthrough mode the cache must be clean.Passthrough mode allows a cache device to be activated without having toworry about coherency.  Coherency that exists is maintained, althoughthe cache will gradually cool as writes take place.  If the coherency ofthe cache can later be verified, or established through use of the"invalidate_cblocks" message, the cache device can be transitioned towritethrough or writeback mode while still warm.  Otherwise, the cachecontents can be discarded prior to transitioning to the desiredoperating mode. A simple cleaner policy is provided, which will clean (write back) alldirty blocks in a cache.  Useful for decommissioning a cache or whenshrinking a cache.  Shrinking the cache's fast device requires all cacheblocks, in the area of the cache being removed, to be clean.  If thearea being removed from the cache still contains dirty blocks the resizewill fail.  Care must be taken to never reduce the volume used for thecache's fast device until the cache is clean.  This is of particularimportance if writeback mode is used.  Writethrough and passthroughmodes already maintain a clean cache.  Future support to partially cleanthe cache, above a specified threshold, will allow for keeping the cachewarm and in writeback mode during resize. Migration throttling-------------------- Migrating data between the origin and cache device uses bandwidth.The user can set a throttle to prevent more than a certain amount ofmigration occurring at any one time.  Currently we're not taking anyaccount of normal io traffic going to the devices.  More work needsdoing here to avoid migrating during those peak io moments. For the time being, a message "migration_threshold <#sectors>"can be used to set the maximum number of sectors being migrated,the default being 2048 sectors (1MB). Updating on-disk metadata------------------------- On-disk metadata is committed every time a FLUSH or FUA bio is written.If no such requests are made then commits will occur every second.  Thismeans the cache behaves like a physical disk that has a volatile writecache.  If power is lost you may lose some recent writes.  The metadatashould always be consistent in spite of any crash. The 'dirty' state for a cache block changes far too frequently for usto keep updating it on the fly.  So we treat it as a hint.  In normaloperation it will be written when the dm device is suspended.  If thesystem crashes all cache blocks will be assumed dirty when restarted. Per-block policy hints---------------------- Policy plug-ins can store a chunk of data per cache block.  It's up tothe policy how big this chunk is, but it should be kept small.  Like thedirty flags this data is lost if there's a crash so a safe fallbackvalue should always be possible. Policy hints affect performance, not correctness. Policy messaging---------------- Policies will have different tunables, specific to each one, so weneed a generic way of getting and setting these.  Device-mappermessages are used.  Refer to cache-policies.txt. Discard bitset resolution------------------------- We can avoid copying data during migration if we know the block hasbeen discarded.  A prime example of this is when mkfs discards thewhole block device.  We store a bitset tracking the discard state ofblocks.  However, we allow this bitset to have a different block sizefrom the cache blocks.  This is because we need to track the discardstate for all of the origin device (compare with the dirty bitsetwhich is just for the smaller cache device). Target interface================ Constructor-----------   ::    cache <metadata dev> <cache dev> <origin dev> <block size>         <#feature args> [<feature arg>]*         <policy> <#policy args> [policy args]*  ================ ======================================================= metadata dev     fast device holding the persistent metadata cache dev	  fast device holding cached data blocks origin dev	  slow device holding original data blocks block size       cache unit size in sectors  #feature args    number of feature arguments passed feature args     writethrough or passthrough (The default is writeback.)  policy           the replacement policy to use #policy args     an even number of arguments corresponding to                  key/value pairs passed to the policy policy args      key/value pairs passed to the policy		  E.g. 'sequential_threshold 1024'		  See cache-policies.txt for details. ================ ======================================================= Optional feature arguments are:     ==================== ========================================================   writethrough		write through caching that prohibits cache block			content from being different from origin block content.			Without this argument, the default behaviour is to write			back cache block contents later for performance reasons,			so they may differ from the corresponding origin blocks.    passthrough		a degraded mode useful for various cache coherency			situations (e.g., rolling back snapshots of			underlying storage).	 Reads and writes always go to			the origin.	If a write goes to a cached origin			block, then the cache block is invalidated.			To enable passthrough mode the cache must be clean.    metadata2		use version 2 of the metadata.  This stores the dirty			bits in a separate btree, which improves speed of			shutting down the cache.    no_discard_passdown	disable passing down discards from the cache			to the origin's data device.   ==================== ======================================================== A policy called 'default' is always registered.  This is an alias forthe policy we currently think is giving best all round performance. As the default policy could vary between kernels, if you are relying onthe characteristics of a specific policy, always request it by name. Status------ ::   <metadata block size> <#used metadata blocks>/<#total metadata blocks>  <cache block size> <#used cache blocks>/<#total cache blocks>  <#read hits> <#read misses> <#write hits> <#write misses>  <#demotions> <#promotions> <#dirty> <#features> <features>*  <#core args> <core args>* <policy name> <#policy args> <policy args>*  <cache metadata mode>  ========================= =====================================================metadata block size	  Fixed block size for each metadata block in			  sectors#used metadata blocks	  Number of metadata blocks used#total metadata blocks	  Total number of metadata blockscache block size	  Configurable block size for the cache device			  in sectors#used cache blocks	  Number of blocks resident in the cache#total cache blocks	  Total number of cache blocks#read hits		  Number of times a READ bio has been mapped			  to the cache#read misses		  Number of times a READ bio has been mapped			  to the origin#write hits		  Number of times a WRITE bio has been mapped			  to the cache#write misses		  Number of times a WRITE bio has been			  mapped to the origin#demotions		  Number of times a block has been removed			  from the cache#promotions		  Number of times a block has been moved to			  the cache#dirty			  Number of blocks in the cache that differ			  from the origin#feature args		  Number of feature args to followfeature args		  'writethrough' (optional)#core args		  Number of core arguments (must be even)core args		  Key/value pairs for tuning the core			  e.g. migration_thresholdpolicy name		  Name of the policy#policy args		  Number of policy arguments to follow (must be even)policy args		  Key/value pairs e.g. sequential_thresholdcache metadata mode       ro if read-only, rw if read-write 			  In serious cases where even a read-only mode is			  deemed unsafe no further I/O will be permitted and			  the status will just contain the string 'Fail'.			  The userspace recovery tools should then be used.needs_check		  'needs_check' if set, '-' if not set			  A metadata operation has failed, resulting in the			  needs_check flag being set in the metadata's			  superblock.  The metadata device must be			  deactivated and checked/repaired before the			  cache can be made fully operational again.			  '-' indicates	needs_check is not set.========================= ===================================================== Messages-------- Policies will have different tunables, specific to each one, so weneed a generic way of getting and setting these.  Device-mappermessages are used.  (A sysfs interface would also be possible.) The message format is::    <key> <value> E.g.::    dmsetup message my_cache 0 sequential_threshold 1024  Invalidation is removing an entry from the cache without writing itback.  Cache blocks can be invalidated via the invalidate_cblocksmessage, which takes an arbitrary number of cblock ranges.  Each cblockrange's end value is "one past the end", meaning 5-10 expresses a rangeof values from 5 to 9.  Each cblock must be expressed as a decimalvalue, in the future a variant message that takes cblock rangesexpressed in hexadecimal may be needed to better support efficientinvalidation of larger caches.  The cache must be in passthrough modewhen invalidate_cblocks is used::    invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]* E.g.::    dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789 Examples======== The test suite can be found here: https://github.com/jthornber/device-mapper-test-suite ::   dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \	  /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0'  dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \	  /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \	  mq 4 sequential_threshold 1024 random_threshold 8'