\chapter{Run Time Developments}Developing runtimes for processes on Plan~9 was challenging, because we did not opt for the traditional HPC approach of placing device drivers in user level libraries. From the beginning of this project, we had decided to have the kernel control communications and resource management tasks that have been relegated to user level runtimes for decades. Returning control of these tasks to the kernelmakes the HPC resource available to a much wider set of applications than just MPI programs. On many traditional HPC systems, high performance networks are not even visible to non-MPI applications. On Plan~9 on BG/P, even the command shell has access to the fastest networks: henceit is possible to run a shell pipeline between compute nodes, without rewriting all the programs involved to use MPI. Parallel programs become more portable, and much easier to write and use. Removing device drivers from libraries removed all the attendant complexity from those libraries; many user-level libraries have more complex code than the Plan~9 kernel, and even the smallest of those libraries is larger than our kernel. Our limited-function MPI supports many MPI applications in less than 5000 lines.User-level runtimes are not without their advantages. Most MPI libraries are thread-safe, and incorporate an activity scheduler that can choose activities with very low overhead. Because the device driver is contained in the process, pointers to data are valid for all the operations the runtime performs, including setting up DMA. There is no need to translate addresses, and on a simple system like Blue Gene, there is no need for complex page pinning software. These MPI runtimes in essence create a single-address-space operating system, complete with drivers and scheduling, that is "booted" by the host operating system and then left to run in control of thenode. Again, the advantages of the runtime include the implementation of system call functionality with function calls; and a singleaddress space, with the attendant elimination of address translations and page faults and elimination of unnecessary copies. Leaving device drivers in the kernel requires reduced overhead in some areas. A system call takes 1-2 microseconds, on average; this time must be reduced when latency-sensitive operations take place. User-level addresses must be mapped into the kernel address space. In interrupt handlers, user-level memory is not easily accessed in Plan~9, since interrupt handlers have no user context. In this section we describe the research we have undertaken to address these needs. \section{Unified Execution Model}In order to address the need for a more dynamic model of interacting with large scale machines,we built the unified execution model (UEM)~\cite{uem} which combined interfaces for logical provisioning and distributed command execution with integrated mechanisms for establishing and maintaining communication, synchronization, and control. The intent was to provide an environment whichwould allow the end-user to treat the supercomputer as a seamless extension of his desktopworkstation -- making elements of his local environment and file system available on the supercomputer as well as providing interfaces for interactive control and management of processes running on top of the high-performance system.Our initial attempt at providing the UEM infrastructure was based around an extension of some of our previous work in cluster workload management~\cite{xcpu}~\cite{xcpu2}. This initial infrastructure, named XCPU$^3$ was built on top of the Inferno virtual machine which could behosted on a variety of platforms (Windows, Linux, Mac, Plan 9), while providing secure communication channels and dynamic name space management. It functioned as a serverfor the purposes of providing access to the elements of the end-users desktop as wellas an gateway to the interfaces for job control and management on the supercomputer.The management interface was structured as a synthetic file system built into the Inferno virtual machine, which could be accessed by applications running on the hosteither directly via a library or by mounting a 9P exported name space (using the Linuxv9fs 9P client support built into the Linux kernel).A key aspect which allowed the UEM to scale was a hierarchical organization of nodeaggregation which on bluegene matched the topology of the collective network. It becameclear that at the core of the interaction of the UEM infrastructure were one to manycommunication pipelines which were used to broadcast control messages and aggregateresponse communication and reporting. We extracted that key functionality into a newsystem primitive called a multipipe~\cite{multipipe}. The result dramatically simplifiedthe infrastructure and improved overall system performance. It also became clear thatmultipipes were a useful primitive for the construction of applications and other systemservices. We extended the design to incorperate many to many communication as well assupport for collective operations and barriers.While Inferno provided a great environment on top of end-user desktops, we didn't wantto incur the overhead of running Inferno on all the compute nodes of the supercomputer.Using multipipes as the core primative, we built a pair of synthetic file systems for Plan 9 which provided an interface for initiating execution on a node named execfs anda second file system whose purpose was to coordinate groups of remote processes to allowfor fan-out computation and aggregate control named gangfs. The gangfs file systemswere built to organize themselves hierarchically in a fashion similar to XCPU$^3$. Thiscould then be mounted by front-end systems or end user workstations and interacted withusing Inferno as a gateway and server.In order to make interactions with the UEM file system a bit more straightforwardwe developed a new shell which incorporated support for multi-pipes and interactionswith the taskfs and gangfs interfaces. Push is a dataflow shell which uses two new pipe operators, fan out('|<') and fanin('>|') to create Directed Acyclic Graphs of processes.This allows construction of complicated computational workflows in the same fashion one would create UNIX shell script pipelines.
\chapter{Run Time Developments}
Developing runtimes for processes on Plan~9 was challenging, because we did not opt for the
traditional HPC approach of placing device drivers in user level libraries. From the beginning of this
project, we had decided to have the kernel control communications and
resource management tasks that have
been relegated to user level runtimes for decades. Returning control of these tasks to the kernel
makes the HPC resource available to a much wider set of applications than just MPI programs. On
many traditional HPC systems, high performance networks are not even visible to non-MPI
applications. On Plan~9 on BG/P, even the command shell has access to the fastest networks: hence
it is
possible to run a shell pipeline between compute nodes, without rewriting all the
programs involved to use MPI. Parallel programs become more portable, and much easier to
write and use.
Removing device drivers from libraries removed all the attendant complexity from those libraries;
many user-level libraries have more complex code than the Plan~9 kernel, and even the smallest
of those libraries is larger than our kernel. Our limited-function MPI supports many MPI applications
in less than 5000 lines.
User-level runtimes are not without their advantages. Most MPI libraries are thread-safe, and
incorporate an activity scheduler that can choose activities with very low overhead. Because the
device driver is contained in the process, pointers to data are valid for all the operations the runtime
performs, including setting up DMA. There is no need to translate addresses, and
on a simple system like Blue Gene, there is no need for complex page pinning software.
These MPI runtimes in essence create a single-address-space operating system, complete with drivers
and scheduling, that is "booted" by the host operating system and then left to run in control of the
node. Again, the advantages of the runtime include the implementation of system call functionality
with function calls; and a single
address space, with the attendant elimination of address translations and page faults and
elimination of unnecessary copies.
Leaving device drivers in the kernel requires reduced overhead in some areas.
A system call takes 1-2 microseconds, on average; this time must be
reduced when latency-sensitive operations take place. User-level
addresses must be mapped into the kernel address space. In interrupt handlers, user-level
memory is not easily accessed in Plan~9, since interrupt handlers have no user context.
In this section we describe the research we have undertaken to address these needs.
\section{Unified Execution Model}
In order to address the need for a more dynamic model of interacting with large scale machines,
we built the unified execution model (UEM)~\cite{uem} which combined interfaces for logical provisioning
and distributed command execution with integrated mechanisms for establishing and maintaining
communication, synchronization, and control. The intent was to provide an environment which
would allow the end-user to treat the supercomputer as a seamless extension of his desktop
workstation -- making elements of his local environment and file system available on the
supercomputer as well as providing interfaces for interactive control and management of
processes running on top of the high-performance system.
Our initial attempt at providing the UEM infrastructure was based around an extension of
some of our previous work in cluster workload management~\cite{xcpu}~\cite{xcpu2}. This
initial infrastructure, named XCPU$^3$ was built on top of the Inferno virtual machine which could be
hosted on a variety of platforms (Windows, Linux, Mac, Plan 9), while providing secure
communication channels and dynamic name space management. It functioned as a server
for the purposes of providing access to the elements of the end-users desktop as well
as an gateway to the interfaces for job control and management on the supercomputer.
The management interface was structured as a synthetic file system built into the
Inferno virtual machine, which could be accessed by applications running on the host
either directly via a library or by mounting a 9P exported name space (using the Linux
v9fs 9P client support built into the Linux kernel).
A key aspect which allowed the UEM to scale was a hierarchical organization of node
aggregation which on bluegene matched the topology of the collective network. It became
clear that at the core of the interaction of the UEM infrastructure were one to many
communication pipelines which were used to broadcast control messages and aggregate
response communication and reporting. We extracted that key functionality into a new
system primitive called a multipipe~\cite{multipipe}. The result dramatically simplified
the infrastructure and improved overall system performance. It also became clear that
multipipes were a useful primitive for the construction of applications and other system
services. We extended the design to incorperate many to many communication as well as
support for collective operations and barriers.
While Inferno provided a great environment on top of end-user desktops, we didn't want
to incur the overhead of running Inferno on all the compute nodes of the supercomputer.
Using multipipes as the core primative, we built a pair of synthetic file systems for
Plan 9 which provided an interface for initiating execution on a node named execfs and
a second file system whose purpose was to coordinate groups of remote processes to allow
for fan-out computation and aggregate control named gangfs. The gangfs file systems
were built to organize themselves hierarchically in a fashion similar to XCPU$^3$. This
could then be mounted by front-end systems or end user workstations and interacted with
using Inferno as a gateway and server.
In order to make interactions with the UEM file system a bit more straightforward
we developed a new shell which incorporated support for multi-pipes and interactions
with the taskfs and gangfs interfaces.
Push is a dataflow shell which uses two new pipe operators, fan out('|<') and
fanin('>|') to create Directed Acyclic Graphs of processes.
This allows construction of complicated computational workflows
in the same fashion one would create UNIX shell script pipelines.