.EQdelim $$.EN.TLThe Diskless Fileserver.AUErik Quanstromquanstro@coraid.com.ABThe Plan 9 Fileserver is structured as a multilevel cachefor direct-attached WORM storage.  I describe how theFileserver is being adapted for modern hardware usingnetwork-attached storage (AoE) over 10Gbps Ethernet.This structure allows for good performance and highreliability.  In addition it separates storage maintenancefrom Fileserver maintenance and provides automatic offsitebackup without performance penalty..AE.NHIntroduction.LPIn order to meet our growing performance and reliabilitydemands, I am in the process of rolling out a disklessFileserver.  The system consists of a diskless Intel-basedFileserver, a local AoE target and an offset AoE target.A backup Fileserver in “standby” mode is available in casethe main Fileserver should fail.The AoE targets are stock.I SR1521machines with added 10Gbps Ethernet cards.This configuration is pictured in \*(Fn..F1.PSscale=10u=2.5gap=19define sr |[	box "\f2SR1521\f1"	ht 3*u wid 19] |define fs |[	box $1		ht u wid 19] | A: fs("Fileserver")B: [sr] at A+(0, -4*u)C: spline <-> " 10Gbe" from B.nw+(0,-3/2*u)  left .75*u then up 4*u then right .5*u to A.sw+(0,u*.5)D: fs("backup Fileserver") at A+(gap+19, 0)E: [sr] at A+(gap+19, -4*u)F: spline <-> "Wireless" from A.ne+(0,-1/2*u) right gap/2 to E.nw+(0,-3*u/2)G: spline <-> "10Gbe " from E.ne+(0,-3/2*u)  right .75*u then up 4*u then left .5*u to D.se+(0,u*.5).PE.F2.F3.PPThe configuration string[2] for this Fileserver is.P1.CW "filsys main ce565.0{e565.1e545.1}" ..P2The configuration string for the backup Fileserver is.P1.CW "filsys main ce545.0e545.1" ..P2The targets.CW e565.\f2x\fPare connected to the Fileserver by a point-to-point 10GbpsEthernet link.  Except during a dump or in the event of afailure of.CW e565.1 ,all I/O is performed over this link.  The target.CW e545.0is in another building, connected by a shared 54Mbps wirelesslink..PPThe AoE targets are managed independently from theFileserver.  Maintenance tasks, like replacing failed drives,reconfiguring or adding storage, do not require knowledge ofthe Fileserver and may be performed without shutting downthe Fileserver.  Conversely, the Fileserver does not requireknowledge of how to perform maintenance on the AoE targets..NHFileserver Basics.LPThe Fileserver serves files via the Plan 9 file protocol,9P2000.  Requests that cannot be directly satisfied by thein-memory Block Cache are resolved by devices.  The BlockCache is indexed by device and device address.  The.CW cwdevice serves the WORM filesystem.  It is comprised of threeon-disk devices: cache, read-only and cachedWORM.  These devices are known as.CW c ,.CW w ,and .CW cw .All Blocks have a .CW w-address and a cache state.Blocks not in the cache are state.CW none .Freshly written blocks are state.CW write .Blocks on the .CW wdevice that are rewritten are state.CW dirty .A “dump,” a permanent snapshot of the filesystem, is takenby converting modified blocks to state.CW dump .This process takes just a few seconds.  Other activity onthe Fileserver is halted during the dump.  Copying takes placein the background and does not impact the performance of theFileserver.Once state.CW dumpblocks are copied to the WORM, their state is changed to.CW read or.CW none ,if it is dropped from the cache.  The copying phase of anynumber of dumps may overlap..LPThe implemented Fileserver has a Block Cache of402,197 8192-byte blocks (3137MB), a cache device,.CW e565.0 ,of 3,276,800 blocks (25GB) and a WORM device,.CW "{e565.1e545.1}" ,of 1.5TB.  The WORM device is the loose mirrorof AoE targets.CW e565.1and.CW e545.1 .Writes are preformed on the mirrored devices sequentiallybut data is read from the first device only.  Thus the wirelessconnection which limits dumps to ~1MB/s is not part ofthe client's I/O path..LPThe WORM filesystem is fully described in [1], [2] and [3]..NH“Standby” Mode.LPIt is not possible use both Fileservers at the same time.  Bothwill try to allocate.CW w-addresses without respect to the other.  To solve this problem aconfiguration item and command, both named.CW dumpctl were added.The main Fileserver is configured with.CW "dumpctl yes"and the backup Fileserver is configured with.CW "dumpctl no" .To prevent writes, attaches may be disallowed.In the event that the Fileserver fails, the command.CW "dumpctl yes"is executed on the backup Fileserver's console.And if disabled, attaches are allowed..LPWhile the backup Fileserver is running, it will notsee the new data written by the dump processon the main Fileserver.  The backup Fileserver mustbe halted each day after the dump on the Fileserverand the command.CW "recover main"must be typed at the.CW configprompt.  This will cause the cache to be flushed andthe filesystem to be initialized from the new dump..NHChanged Assumptions.LPIn the fifteen odd years since the Fileserver was developed,a few of its assumptions have ceased to hold.The most obvious is the.CW wormdevice is no longer a WORM. Even if we were to use WORMstorage, disk space is inexpensive enough that it would bepractical to keep an entire copy of the WORM on magneticstorage for performance reasons.  This means that thecache and the WORM devices have the same performance.Therefore it no longer makes sense to copy blocks in state.CW Creadto the cache device.  Blocks in state.CW Creadhave been read from the.CW wormdevice but not modified[3].  A new option,.CW conf.fastworm ,inhibits copying these blocks to the disk cache..PPA less obvious difference is in the structure of the cache.The cache device is structured as a hash table.  The hashfunction is simply modulo the number of hash lines andthe lines are written sequentially to disk.  If we let.I nbe the number of rows and.I lbe the number of columns in our hash, he functionis.P1    row = w % rows    c = column + row*n,.P2the blocks will belinearized onto the disk in the following order.P1    $0, n, 2n, ..., (l-1)n, 1, 1+n, 1+2n, ...$.P2Suppose that two blocks $w$ and $w+1$ are written to theFileserver with an empty cache.  Suppose further that $w+1~%~l~≠~0$.Then blocks $w$ and $w+1$ map to disk blocks $c$ and$c+"CEPERBK"$.  With a block size of 8192 bytes, currentFileserver parameters and 512 byte disk sectors, this worksout to 1072 sectors between “sequential” blocks..PPWith disk drives of the same era as the original Fileserver,disk transfer rates were limited by hardware buffer sizes andinterface bandwidth[5].  Assuming a transfer rate of 1MB/s anda seek time of 15ms, it would take 8ms to transfer 8192 bytesfrom the disk and less than 15ms to seek to another track orabout 347KB/s.On modern SATA drives, it would take 26µs to transfer 8192bytes from the disk and up to 9ms to seek.  this would onlyyield 890KB/s.  During testing about 2MB/s was observed.  Ifthis same ratio of calculated versus actual seek time wereto hold for older drives, the older drives would operate atnear rated bandwidth..PPWhen the formula was changed to .P1    n = w % rows    c = column*CEPERBK + n,.P2the blocks are linearized onto disk in the following order.P1    $0, 1, ..., l-1, l, l+1, ...$,.P2changing from row- to column-major ordering, performanceincreased to ~25MB/s.  Note that not caching blocks in state.CW Creadinsures that $w$ and $w+1$ will be stored sequentially ondisk, as.CW columnwill be the same for $w$ and $w+1$ unless $w+1~%~rows~=~0$.However, in this case the blocks will also be storedsequentially because row $r$ and row $r+1$ are alsosequential..NHAssumptions Redux.LP.DS IIf a cat can kill a rat in a minute, how long would it be killing 60,000 rats?Ah, how long, indeed!  My private opinion is that the rats would kill the cat..br	– Lewis Carroll.DEThe Fileserver's read-ahead system consists of a queue ofblocks to be read and a set of processes which read theminto the cache.  Although the original paper on theFileserver only lists one.CW rahprocess, the earliest Fileserver at the Labs' WORM started four.The Fourth Edition Fileserver again started one.CW rahprocess but attempted to sort the blocks by .CW w-\fRaddress\fRbefore processing.  This approach probably makes senseon slow, partitioned disks.  However, it has the disadvantageof processing blocks serially.  The more parallelism one canachieve among or within the Fileserver's devices, the greaterthe performance penalty of the sequential approach..PPTo test this idea, a 1GB file was created on the Fileserveron AoE storage.  The AoE driver has a maximum of 24outstanding frames per target.  After rebooting theFileserver to flush the Block Cache, it took 25.5s to readthe file.  Subsequent reads took and average of 13.72s.After changing the read-ahead algorithm to use 10independent.CW rahprocesses, the test was rerun.  It took 15.74s to read thefile.  Increasing the number of.CW rahprocesses to 20 reduced the uncached read time to 13.75s,the same as the cached read time.  Two concurrent readerscan each read the entire file in 15.17s, so the throughputappears to be limited by.CW 9P/ILlatency..NHCore Improvements.LPThe.CW portdirectory underwent some housecleaning.  The.CW 9p1protocol was removed.  The console code was rewritten to usethe.CW 9p2code.  The time zone code was replaced with the offset pairsfrom the CPU kernel to allow for arbitrary time zones.  A CECconsole was added to allow access without a serial console..PPMore significantly,.CW Lock swere changed from queueing locks to spin locks.  Since asignificant use of spin locks is to lock queues to add workand wake consumers,.CW unlockreschedules if the current process no longer holds any locksand has woken processes while it held locks.  Also, thescheduler takes care not to preempt a process with locksheld.  This improved the throughput of single-threaded readsby 25%.  These ideas were taken from the CPU kernel..PPNetworking was changed to allow interfaces with jumboMTUs.  This is not currently used by the IL code as it hasno MTU discovery mechanism.  .NHPC Architecture Improvements.LPBy far the largest change in the PC architecture wasto memory handling.  The primary goal was to be able tohandle most of the bottom 4GB of memory.  Thusthe definition of .CW KZEROneeded to be changed.  The PC port inherited its memorylayout from the MIPS port.  On the MIPS processor, the highbit indicated kernel mode.  Thus Fileserver memory was mapped from.CW 0x80000000to the top of memory.  Converting between a physical andvirtual address was done by inverting the high bit.  Whilesimple, this scheme allows for a maximum of only 2GB. Lowering.CW KZEROto.CW 0x30000000and mapping PCI space to.CW 0x20000000allows for.CW 3328MBmemory..PPUnfortunately, being able to recognize more memory puts usin greater danger of running into PCI space while sizingmemory, so another method is needed.  A BIOS.CW 0xe820scan was chosen.  Unfortunately, the processor must bein Real mode to perform the scan and the processor isalready in Protected mode when the Fileserver kernelis started.  So, Instead of switching back to real mode,.CW 9loadwas modified to perform the scan before turning onpaging[8]..PPSurprisingly, the preceding changes were not enoughto enable more memory.  The Fileserver faulted whenbuilding page tables.  It turned out this is becausethe 4MB temporary pagetables built by.CW 9loadwere not enough.  The BIOS scan of the testing machineyielded 3326MB of accessible memory.  This would require3.25MB of page tables.  Since the bottom megabyte of memoryis unusable, we don't have any room left for the kernel.The solution was to use 4MB pages.  This eliminates the needfor page tables, as the 1024-entry page directory has enoughspace to map 4GB of memory..PPOn 64-bit processors, it would be relatively easy to fill inmore memory from above 4GB by using the 40-bit extensionsto 4MB pages..NHThe AoE Driver.DS IIf you were plowing a field what would you rather use, 2 strong oxen or1024 chickens? .br	– Seymour Cray.DE.LPThis is the Fileserver's raison d'être.  The AoE driver is based onthe Plan 9 driver.  It is capable of sending jumbo or standardAoE frames.  It allows up to 24 outstanding frames per target.It also allows a many-to-many relationship between local interfacesand target interfaces. .PPWhen the AoE driver gets an I/O request, a .CW Srbstructure is allocated with.CW mballoc .Then the request is chopped up into .CW Framestructures as available  Each is sized to MTU of the chosen link.  Alink is chosen round-robin fashion first among localinterfaces which can see the target and then among thetarget's MAC addresses.  MTUs may be freely mixed.  Theframes are sent and the number of outstanding frames isappropriately incremented.  The driver then sleeps on the.CW Srb .When awoken, the process is repeated until all the bytes inthe request have been received..PPWhen an AoE frame is received that corresponds to I/O,the frame is copied into the buffer of the.CW Srband the number of outstanding frames is decremented.  Ifthere are no outstanding frame remaining, the.CW Srbis woken..PPSince the Myricom 10Gbe cards have an MTU of 9000 bytes,an entire 8192 byte block and the AoE header fit into a singleframe.  Thus sequential read performance depends onframe latency.  Performance was measured with a process runningthe following code.P1static voiddevcopy(Dcopy *d){	Iobuf *b; 	for(d->p = d->start; d->p < d->lim; d->p++){		b = getbuf(d->from, d->p, Bread);		if(b == 0)			continue;		putbuf(b);	}}.P2The latency for an frame with 8192 data bytesis 79µs giving 12,500pps or 103MB/s while two concurrent readsyield 201MB/s.  Testing beyond this level of performance hasnot been performed..NHSystem Performance.LPI measured both latency and throughputof reading and writing bytes between two processesfor a number of different paths.2007 measurements were made using an.I SR1521AoE target, an Intel Xeon-5000-based cpu server witha 1.6Ghz processor and a Xeon-5000-based Fileserverwith a 3.0Ghz processor.  1993 measurements arefrom [6].The latency is measured as the round trip timefor a byte sent from one process to another andback again.Throughput is measured using 16k writes fromone process to another..ps -2.DS C.TSbox, tab(:);c s s s sc | c | c | c | ca | n | n | n | n.Table 1 – Performance_test:93 throughput:93 latency:07 throughput:07 latency:MB/s:µs:MB/s:µs_pipes:8.15:255:2500:19_IL/ether:1.02:1420:78:72_URP/Datakit:0.22:1750:N/A\&:N/A\&_Cyclone; AoE:3.2:375:≥250:49.TE.DE.NL.LPRandom I/O was not tested for two reasons.  First, ~3GB ofrecient reads and writes are stored in the Block Cache andwhen new or newly modified files are reread from the cache,they are reread sequentially.  It is expected that theworking set of the Fileserver fit in the Block Cache.Second, since a single IL connection is latency limited,reads of highly fragmented files like.CW /sys/log/authfrom the WORM are not meaningfully slower (59MB/s) thanreads from the cache (62MB/s)..NHDiscussion.LPDecoupling storage from the Fileserver with AoE allows forautomatic offsite backup, affords good availability,scalablity and performance.  The Fileserver is not involvedin storage management.  It is possible to grow the existingWORM to 9TB without restarting the fileserver.  Byreconfiguring the Fileserver, essentially unlimited storagemay be added..LPThe size of the WORM and Block Cache have scaled by a factorof 1000 since [3] and single IL connections have scaled by afactor of 200 since [7].  The Block Cache is currently at apractical maximum for a kernel with 32-bit memory addresses.A kernel with 64-bit memory addresses in the next logical

Lines 1–500 of 550. Load more