.FP palatino.TM.TLPlan 9 on the Mikrotik RB450G Routerboard.AUGeoff Collyer.AI.MH.NH 1Motivation.LPI ported Plan 9 to the Routerboard mainly to verifythat Plan 9's MIPS-related code(compiler, assembler, loader,.CW libmach ,etc.) was still in working order and wouldwork on newer machines than the 1993-era ones that we last owned(MIPS Magnum, SGI Challenge, Carrera and the like).The verdict is that,with a few surprising exceptions, the code still works on newish machines(the MIPS 24K CPU in the Routerboard dates to about 2003 originally;this revision is from about 2005).So we now have amachine on which to test MIPS executables..LPThe other reason I did the port wasas an incremental step towardrunning Plan 9 on a MIPS64 machine (e.g., the dual-core, dual-issueCavium CN5020 in the Ubiquiti Edgerouter Lite 3)..NH 1The new MIPS world.LPThese newer MIPS systems are aimed at embedded applications, so theytypically lack FPUs and may also lack L2 caches or have small TLBs;the MIPS 24K in the Atheros 7161 SoC lacks FPU and L2 cache, and has a16-entry TLB.It is a MIPS32R2 architecture system and lacks the 64-bit instructionsof the R4000.These new MIPS systems are still big-endian,so provide a useful test case to expose byte-ordering bugs..NH 1Plan 9 changes and additions.NH 2CPU Bug Workarounds.LPThe Linux MIPS people cite MIPS 24K erratum 48:3 consecutive stores lose data.MIPS only distribute their errata lists under NDA and to theircorporate partners, so we have only the Linux report to go on.The fix requires.I bothwrite-through data cache andno more than two consecutive single-word stores in all executables.I have made a crude optional change to.I vlto generate a NOP before every third consecutive store.The fix could be better, in particular the technique forkeeping stores out of branch delay slots..NH 2Driver for Undocumented Ethernet Controller.LPThe FreeBSD Atheros.I argedriver(in.CW /usr/src/sys/mips/atheros )provided inspiration for our Gigabit Ethernet driver, since thehardware is otherwise largely undocumented.I haven't got the secondEthernet controller entirely working yet;it's perhaps complicated by having a switch attached to it (the Atheros 8316).At minimum, it probably needs MII or PHY initialisation..NH 2Floating-point Emulation.LPFloating-point emulation works but is.I veryslow:.I astrotakes about 8 seconds.I added an.CW fpemudebugcommand to.CW /dev/archctl ;ittakes a number as argument corresponding to the.CW Dbg*bits in.CW fpimips.c ,but requires the kernel to be compiled with.CW FPEMUDEBUGdefined..NH 3\&... in Locking Code.LPThe big surprises included that.CW /sys/src/libc/mips/lock.cread.CW FCR0tochoose the locking style.That's been broken out into.CW c_fcr0.sso that we can change it, but the kernel also emulates the.CW MOVW.CW FCR0,R1(and via a fast code path), to keep alive the possibility of runningold binaries from the dump..NH 2No 64-bit Instructions.LPThe other big surprise was that.CW /sys/src/libmp/mips/mpdigdiv.sused 64-bit instructions (SLLV, SRLV, ADDVU, DIVVU).For now I've resolved the problem by pushing it into asubdirectory (\c.CW r4k )and editing the.CW mkfile sto use the.CW portversion(and similarly in APE)..br.ne 8.NH 2Page Size vs TLB Faults.LPI started out with a 4K page size and reduced the number of TLBentries reserved for the kernel to 2, leaving 14 for user programs,but.CW /dev/sysstatwas reporting 6 times as many TLB faults as pagefaults, and the number increased at a furious rate..LPSo I switched toa 16K page size, adjusted.CW vl.CW -H2accordingly and recompiled the.CW /mipsworld.This reduced the TLB faults to just 10% more than the number of page faults.(That number is now around 15% more, due to a better soft-TLB hash functionthat makes the soft TLB more effective.)16K pages also produce consecutive (even recursive) page faultsfor the same address at the same PCand the system runs at about 10% of its normal speed,so 4K pages are currently the only sensible choice;we'll just live with the absurdly-high number of TLB faults(around 20k–30k per second).It probably doesn't help that one 16K page is half of the L1 data cacheand one quarter of the L1 instruction cache..LPPage size is controlled by.CW BIGPAGESin.CW mem.h ..NH 3Combined TLB Pool.LPI also changed.CW mmu.cto collapse the separate kernel and user TLB pools into one,once user processes start running,but that only helps to reduce TLB faults a little...br.ne 8..NH 1Remaining Problems.LPInterrupt-driven UART output isn't quite right.It can get stuck and then input makes it resume.The UART is apparently connected via the APB and requiresinterrupt unmasking in the APB (which we now do).There's some kludgey stuff in.CW uarti8250.cthat makes output work most of the time(characters do sometimes get dropped)..LPThe Ethernet driver currently does notdig out the MAC addresses from the hardware,so you'll need to edit the.CW rbconfiguration file for each Routerboard; the format should be obvious.I don't have the stomach to dig the MAC address out of the hardwarevia SPI or whatever vile interface it requires.