\chapter{Implementation}\label{chap:implementation} \ifpdf \graphicspath{{Chapter3/Chapter3Figs/PDF/}{Chapter3/Chapter3Figs/PNG/} {Chapter3/Chapter3Figs/}}\else \graphicspath{{Chapter3/Chapter3Figs/EPS/}{Chapter3/Chapter3Figs/}}\fi This chapter discusses the implementation issues in more detail. It alsoexplains the filesystem interface and how it can be used. We will show howthe filesystem interface and independence of nodes gives much desiredflexibility. We will see some examples showing how easy it is to runapplications on this platform. This chapter concentrates on how XCPU3 wasimplemented and describes implementation decisions which have proven to beimportant. \section{Inferno}Before we start discussing our decision to use Inferno, let us explain whatInferno is. Inferno\cite{inferno} is an open source distributed operatingsystem developed and maintained by \textit{Vita Nuova}. It is a direct descendantof the \textbf{Plan 9} operating system. It runs natively on multiplehardware platforms and can also run as a user-space application on top of otheroperating systems. For the purpose of this project \textit{"hosted Inferno OS"} or\textit{"userspace Inferno"} refers to Inferno running as user-spaceapplication on top of other operating systems. So we will use these termsinterchangeably. As we are aiming for a flexible heterogeneous environment with differentoperating systems and different architectures, we found the Inferno operatingsystem to be an attractive platform. It allowed us to develop XCPU3 once andeasily deploy it on different platforms while enjoying the Plan 9 featureson all those platforms. This decision does come with the cost of some performance loss. Each node needsto run hosted Inferno in user-space which takes up some resources of the node. Also every communication with the filesystem involves invocation of the hostkernel which then passes the call to the guest inferno. The responsesfollow the same path. This does add latency in communication leading topoorer performance compared to filesystems implemented natively on hostoperating systems. But we are willing to accept this performance loss for achieving the portability to a large number of platforms. We also had a choice of developing XCPU3 as part of the Inferno kernel or as an Inferno user-spaceapplication. User-space development is comparatively easy and user-space toolscan be used for debugging any problems. On the other hand, kernel-space developmentcomes with more constraints and practically no support for debugging. Any bugin kernel-space code typically leads to a kernel panic giving very little informationabout the cause of the bug. The advantage of kernel-space development is that servinguser requests coming to XCPU3 will not involve context switches between Infernouser-space and kernel-space. As the choice between Inferno kernel-space and user-spacedoes not affect the portability, we decided to implement XCPU3 in Inferno kernel-spacefor better performance. We are also hoping that this decision will help ineasily developing a high performance native Plan 9 port for XCPU3 as the code-base ofPlan 9 and Inferno kernel are similar. Our choice of Inferno gave us the flexibility and features of the Plan 9 operatingsystem on other operating systems. It allowed us to quickly implement anddeploy our filesystem on multiple platforms easily at the cost of someperformance. \section{Implementation} XCPU3 is implemented as the filesystem in the Inferno kernel. Figure \ref{fig:XCPU3} shows the placement of XCPU3 in the context of applications, host operating system and the Inferno. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.4\textheight,width=0.6\textwidth] {XCPU3Structure} \fi \caption{XCPU3 Structure} \label{fig:XCPU3} \end{center}\end{figure} The XCPU3 filesystem is exported by Inferno using the \texttt{9P} protocol. This protocolis used by Plan 9 and Inferno extensively to access any file. Recently the Linux kernelhas added support for the \texttt{9P} protocol\cite{graverobbers}. This allows Linux to mount filesystemsexported over the 9P protocol. Other Unix based operating systems can use FUSE\cite{FUSE} for accessing the 9P based filesystems. As FUSE based filesystems run in user-space, they perform slower compared to native filesystems implemented in the kernel. But implementing FUSE based filesystems is much easier and faster. Applicationscan directly use the FUSE based filesystem just like a kernel implemented filesystem with the same API. Applications can also directly talk with filesystems exported over \texttt{9P} by implementingthe client side of the \texttt{9P} protocol. This way they can get better performanceby bypassing VFS and/or FUSE interfaces. In a typical setup, Inferno runs in the hosted environment on an operating system like Linuxor Plan 9. On its startup this hosted inferno will export the XCPU3 filesystem.This exported filesystem is mounted by the local host operating system. The applications interact with this mounted XCPU3 filesystem using the native interface.Any interaction with this XCPU3 filesystem is communicated to Inferno using 9P.The XCPU3 kernel module inside the inferno kernel then receives and interprets the user actions. If needed, it uses services from the host operating system or fromother XCPU3 filesystems deployed on remote locations. The XCPU3 filesystem then sends the prepared response over 9P. The host operating system will relay thisresponse back to the application via the local filesystem interface. As we discussed before in the Design chapter this XCPU3 is an independent unit, providingthe functionality of reservation, job management and computation. In the case of anisolated instantiation, any reservation request is satisfied using local resources.This way, the application can use the same semantics for both local and remote executions. \section{The big picture (connecting multiple XCPU3 nodes)}The real power of the XCPU3 is the ability to connect with each other and form biggerinstance of XCPU3. XCPU3 filesystem uses the \texttt{9P} protocol to connect with eachother. This protocol provides access to both local and remote filesystems in the same way. This allows every node to access the XCPU3 filesystem hosted on any nodewithout worrying about complexities of accessing remote filesystems. Thosecomplexities are handled by the \texttt{9P} protocol. \subsection{Central Services} The ability to configure many XCPU3 nodes into hierarchy is provided by the\textit{central services}. In contrary to the name, central services ishighly distributed and every XCPU3 runs an independent instance of the centralservices. The administrator or the user of each XCPU3 instance need toprovide only the information about its parent and children in the hierarchy andthe all XCPU3 nodes initiate these connections leading to the distributedcreation of the XCPU3 node hierarchy. Central services is aimed to make all the resources addressable in thenetwork-oblivious fashion. It also aims to have external entities(such as end-user workstations) participate in the hierarchy. It is also aimedto be able to work through multiple networks, NAT gateways and firewalls. Overall, central services is aimed to grants us a flexible facility forbuilding hierarchies across different types of network and networkboundaries in a distributed and secure fashion. \subsubsection{Implementation}The central services synthetic file server itself is quitesimple. It provides a simple hierarchy of directory mount pointsrepresenting remote nodes. Mounts of the remote nodes or binds ofpreviously mounted remote nodes are accomplished within this filesystem such that anyone who mounts our name space can also see (andaccess) anyone we have mounted transitively, in such a way a childnode can access a parent nodes, other children, or the parents nodesparent and so forth. The original implementation of the central services mechanism used anauto-mounter-like interface. When you referenced a name within thesynthetic file server for the first time, it would establish aconnection to that systems central-services server and establish a“duct”. A duct is essentially a two way pipeline that allows theclient to mount the sever and vice-versa -- allowing each side toaccess the others resources. Each would create an entry in theirrespective mount table based on the node-name. This allowed clientssitting behind network-address-translation gateways to contact aparent and the parent to access that child without having tore-traverse the gateway. A command-line option could be used whenstarting central services allowing it to connect to a single remoteresource over an ssh-connection establishing a tunnel capable ofcrossing firewalls. It proved difficult to debug and did not allow us to leveragepreviously mounted filesystem (such as the parent's root filesystemfrom which we get our root filesystem). A second lighter weightversion of the filesystem was created which allowed manipulation ofthe name space via a single control file at the top level with asimple name space oriented syntax: \begin{itemize} \item Establish a network connection and mount table.\begin{verbatim}mount <remote-address> <name>\end{verbatim} \item Bind previously mounted resources.\begin{verbatim}bind <path-to-remote-namespace> <name>\end{verbatim} \item Remove a resource.\begin{verbatim}remove <name>\end{verbatim} \end{itemize} Using this simple mechanism nodes could establish themselves within ahierarchy by binding a parent's central service directory to the name/csrv/parent and then tell the parent to back-mount their name space(allowing two way traversal). In this way children register withparents triggering the cross-mounts and establishing a graph whichspans the entire system. On the Blue Gene, the machine already has a natural aggregationtopology of compute nodes organized under IO-nodes which accessed fromfront-end servers. During boot, our compute nodes mount IO-nodeswhich mount name spaces from the front-end servers for the purposes ofestablishing a distributed filesystem. We leverage this sameaggregation topology for the purposes of central-services and theexecution model. \subsection{Example topology} The figure \ref{fig:XCPU3Topo} gives an example of how the XCPU3 filesystems can be be connected to form a larger system. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.4\textheight,width=0.6\textwidth] {XCPU3Topo} \fi \caption{Sample topology of XCPU3} \label{fig:XCPU3Topo} \end{center}\end{figure} This example shows a hierarchical topology with three levels and with a coupleof nodes behind a firewall. As ducts between XCPU3 filesystem are two-way, itallows node behind the firewall to participate in the larger topology as long as theyat least have one link with any other node. In our example, node F can be seenand accessed by any other node via node B. The interface for such access is explained in \textit{Filesystem interface} section. This way, central services play's key role in overcoming the partitioned networks and providing the view ofall resources in network oblivious manner. \chapter{Filesystem Interface} This chapter discusses the filesystem interface and how it can be used to accessany node. We will mainly concentrate on the interfaces for accessing the nodes,local resources, sessions and sub-sessions. \section{Interface for remote nodes}XCPU3 follows certain conventions which allows every node to easily interpret the filesystem view of the other nodes. This section describes the XCPU3 interface and the conventions for accessing the remote nodes. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.3\textheight,width=0.6\textwidth] {xcpu3FSTopo} \fi \caption{Sample filesystem interface for sample the topology in XCPU3} \label{fig:xcpu3FSTopo} \end{center}\end{figure} Figure \ref{fig:xcpu3FSTopo} tries to give a simple overview of how this syntheticfilesystem view is populated based on the underlying topology of the nodes. The nodesin the figure are arranged in the simple hierarchical topology. The XCPU3 filesystem starts withthe \texttt{[/csrv]} directory. This directory contains the synthetic contents dynamically generated by the XCPU3. The location \texttt{[/csrv/local/]} presents the local resourceswhereas \texttt{[/csrv/parent/]} presents the XCPU3 (ie \texttt{[/csrv/]}) filesystem of the node which is the parent of this node in the physical topology. All other directoriesin the \texttt{[/csrv/*]} represents the XCPU3 filesystem of the remote nodes which are the childrenin the physical topology. The \textit{App-1 view} is the XCPU3 filesystem view on the root node\texttt{[t]}. You can see that it has two directories in the \texttt{[/csrv/]}representing the local resources and the remote child L. The good part of this setup is that every node has to worry about only its childrenand the parent, other topology falls in the place automatically. In our sample case, the node \texttt{[t]} only needs to connect to node \texttt{[L]}. It can access other nodesfrom the filesystem view of the node \texttt{[L]}. In tern, the node \texttt{[L]} is responsiblefor connecting to the node \texttt{[t]} as parent and node \texttt{[l1]}, \texttt{[l2]} aschildren. These connections allow him to access the entire cluster. This way, responsibility of making the connections with other nodes is also \textbf{localized}leading to better scalability. The \textit{App-2 view} provides the \texttt{[/csrv/]} filesystem view at the leaf node \texttt{[c3]}. This node \texttt{[c3]} only connects to the node \texttt{[l2]} as it's parent and this single connection allows it to view the entire cluster. This particular view of resources is highly dependent on the underlying cluster.Nodes are directly connected to the neighbors and indirectly connected to othernodes, hence maintaining the relationship with underlying cluster. Mostapplications may not want a topology based view of underlying resources, butthe uniform view. For that reason we create an uniform overlay view separatelyfor every application when they request the reservation. This overlay viewbinds the resources from this underlying hierarchy and hence successfully hidesthe complexities created by underlying topology. As every application receivesits own session directory and all the requested resources are provided withinthe same session directory, the application can use all the resources withinits session without worrying about interfering with others or the physicallocation of resources. As each application lives in it's own session, thisdesign also enables the capability of running multiple applicationssimultaneously and independently. The XCPU3 infrastructure is designed to work by providing resources in a uniformway irrespective of the underlying cluster topology. But there is a class ofapplications and users that want to have the view of resources based on thecluster topology. The view provided by the \texttt{[/csrv/]} caters to the needof these applications as this view maps the resources to the underlying clustertopology. \subsection{Creating a global view}Even though the filesystem view at different nodes differ from each other, all nodes have access to the full topology. The \texttt{[/csrv/]} filesystem view encodes enough information within itself that any nodes can construct the global view and figure out their own position within the global view. Just because every node can construct the global view, does not mean that it must usethis global view for making any decision or performing a typical operation.The nodes mostly use only the local view for decision making and operations. Thislocal view includes the parent node and the children nodes. The ability to construct a global view is available to the applications running on any node which need this information. This operation is comparatively expensiveas it involves traversing the \texttt{[/csrv/]} filesystem view. As each directoryin \texttt{[/csrv/]} location except \texttt{[/csrv/local/]} is the XCPU3 filesystemof a remote node, any node can access the XCPU3 filesystem of other nodes using thistraversal and use this information to build the global view.The XCPU3 infrastructure provides the interface to generate such global views,and leaves the decision to use this information at the expense of performance to the application developers and users. \section{Interface for local resources}This section describes the interface for accessing and managing the local resources at any node. Again this interface is uniform across all the nodes enabling applicationsand other nodes to access it in the same way. All local resources can be accessedin the location \texttt{[/csrv/local/]}. Figure \ref{fig:xcpu3Local} shows the typical content of this location. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.4\textheight,width=0.4\textwidth] {xcpu3Local.png} \fi \caption{Sample filesystem interface for local resources in XCPU3} \label{fig:xcpu3Local} \end{center}\end{figure} Each file and directory in the \texttt{[/csrv/local/]} has a specific purpose in accessingand managing the local resources.%\begin{enumerate} \subsection{arch} \texttt{[/csrv/local/arch]} is a \textit{read-only} file. A read operationon this file returns the name of the host operating system and the hardware architecture.This information is non-destructive and can be read any number of times.The typical use-case of this file is\begin{verbatim}$ cat /csrv/local/archLinux 386$\end{verbatim}Typically, this information will be used by the scheduler to satisfy the constraintsimposed by the applications for the underlying operating system and architecture. \subsection{env} \texttt{[/csrv/local/env]} provides the interface for specifying the environmentalvariables. This file can be opened for writing and multiple tuples of \texttt{[Name Value]}pairs can be written into it. Following is the example of how this file can be used\begin{verbatim}$ echo "PATH /home/example/" > /csrv/local/env$\end{verbatim}The environment variables added this way are the default variables and they will be available to all the sessions started on this node. We do provide another interfacefor controlling the environmental variables within a session. We will discuss that interface in more details in the next section. \subsection{ns} \texttt{[/csrv/local/ns]} file provides the interface to assemble thenamespace that will be inherited by any application process created on this node.\begin{verbatim}$ echo "mount /exp/root/ /" > /csrv/local/ns$ echo "bind /some/path /other/path" > /csrv/local/ns$ \end{verbatim}This is the default namespace and can be overwritten for a particular session using anotherinterface, allowing more fine-grained control. \subsection{fs} \texttt{[/csrv/local/fs]} location gives the interface to the host filesystem.This interface allows the applications to access the local filesystem if needed. \subsection{net} \texttt{[/csrv/local/net]} location gives the interface to the host network.This interface is in the Plan 9 style filesystem which can be used create new networkconnections\cite{net}. The network connections created using this interface will use the localnetworking infrastructure. This is quite useful when the application wants to relyon the local networking interfaces instead of the abstractions provided by XCPU3. \subsection{status} \texttt{[/csrv/local/status]} is the read-only file providing information aboutthe available resources at local node and remote nodes. The information includesthe number of nodes for each combination of an operating system and an architecture. Thesestatistics include all the remote nodes which are the descendant of this node. Thesedescendants are traversed by avoiding the \textit{parent} links. Every node reads the \texttt{[/csrv/<child-node-name>/local/status]} file of all his children nodes and thenaggregate this information to produce its own status file. As all the children are XCPU3nodes themselves, when the parent reads their status file, they repeat the same actionof reading and aggregating the status files of their children. This way theprocess is recursively repeated till it hits the leaf nodes. These nodes terminatethe recursion by returning the information about themselves. Following is an example of the content of status file at a compute node of the Blue Genewhich is a leaf node.\begin{verbatim}$ cat /csrv/local/status1 Plan9 power$ \end{verbatim} This information is aggregated back till the root node which then returns to the user.Following is the example of an aggregated information on the Blue Gene computer.\begin{verbatim}$ cat /csrv/local/status1 Linux power65 Plan9 power$ \end{verbatim} The above description shows how nodes help each other to perform the operation that needsthe global information. Unfortunately, these global operations do involve lot of communication making them comparatively expensive. So, hereby we inform theusers about being careful in using these global operations. We have plans to reduce the cost of such global operations by caching the information of previous global operations. We expect that the global view of the system willnot change very frequently, allowing us to cache this information. We have notimplemented this yet but we plan to implement such caching in near future. \subsection{clone} \texttt{[/csrv/local/clone]} is a read-write file. This file is responsiblefor creating the sessions which can be used to run the applications. Whenever this fileis opened, it allocates the session to it. This session can be a new session with the newid or a previously used but free session. The session is identified by the session-id whichis a non-negative number and it is represented as a directory with the session-id value as the name.The directories \texttt{[/csrv/local/0/]},\texttt{[/csrv/local/1/]} and \texttt{[/csrv/local/n/]} are the session directories.Whenever the \texttt{[/csrv/local/clone]} file is opened, a session is allocated to it.And all subsequent reads on this file will return the session-id of the sessionallocated. Any writes on this opened file will be considered as writes on the \texttt{[/csrv/local/<session-ID>/ctl]} file. If this file is closed and if thereis no other open file within this session directory then this session is releasedfor future use. The typical way for using the clone semantic is \begin{enumerate}\item Open clone file in read-write mode.\item Read the session-id. \item Use the files within that session directory for starting and managing the computation.\item Once your computation is over, close all those files within session.\item Close the clone file to release the session and the resources allocated. \end{enumerate} \subsection{Sessions} \texttt{[/csrv/local/0/]},\texttt{[/csrv/local/1/]} and \texttt{[/csrv/local/n/]} are the session directories infigure \ref{fig:xcpu3Local}. New sessions are created as and when needed and the session-idvalue increases monotonically. Every session directory has its own files with its ownstructure. Next section will discuss the session interface in more details. \section{Session interface}This section explains the filesystem interface of the sessions in the XCPU3.These sessions are important because they are the single unit of execution which canbe entirely controlled by the users. The figure \ref{fig:xcpu3Local} shows the directory structure of a single session. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.44\textheight,width=0.4\textwidth] {xcpu3Session.png} \fi \caption{Sample filesystem interface for sessions in XCPU3} \label{fig:xcpu3Local} \end{center}\end{figure} \subsection{ctl} \texttt{[/csrv/local/<session-ID>/ctl]} is the read-write file whichprovides the control over the session. Just like the clone file, a read operation on this filewill return the session-id. Write operations on this file are treated as the management commands for this session. These commands can be used to do the reservations of the remote resources,requesting the execution or the termination of an ongoing execution. Lines 1–500 of 964. Load more
\chapter{Implementation} \label{chap:implementation} \ifpdf \graphicspath{{Chapter3/Chapter3Figs/PDF/}{Chapter3/Chapter3Figs/PNG/} {Chapter3/Chapter3Figs/}} \else \graphicspath{{Chapter3/Chapter3Figs/EPS/}{Chapter3/Chapter3Figs/}} \fi This chapter discusses the implementation issues in more detail. It also explains the filesystem interface and how it can be used. We will show how the filesystem interface and independence of nodes gives much desired flexibility. We will see some examples showing how easy it is to run applications on this platform. This chapter concentrates on how XCPU3 was implemented and describes implementation decisions which have proven to be important. \section{Inferno} Before we start discussing our decision to use Inferno, let us explain what Inferno is. Inferno\cite{inferno} is an open source distributed operating system developed and maintained by \textit{Vita Nuova}. It is a direct descendant of the \textbf{Plan 9} operating system. It runs natively on multiple hardware platforms and can also run as a user-space application on top of other operating systems. For the purpose of this project \textit{"hosted Inferno OS"} or \textit{"userspace Inferno"} refers to Inferno running as user-space application on top of other operating systems. So we will use these terms interchangeably. As we are aiming for a flexible heterogeneous environment with different operating systems and different architectures, we found the Inferno operating system to be an attractive platform. It allowed us to develop XCPU3 once and easily deploy it on different platforms while enjoying the Plan 9 features on all those platforms. This decision does come with the cost of some performance loss. Each node needs to run hosted Inferno in user-space which takes up some resources of the node. Also every communication with the filesystem involves invocation of the host kernel which then passes the call to the guest inferno. The responses follow the same path. This does add latency in communication leading to poorer performance compared to filesystems implemented natively on host operating systems. But we are willing to accept this performance loss for achieving the portability to a large number of platforms. We also had a choice of developing XCPU3 as part of the Inferno kernel or as an Inferno user-space application. User-space development is comparatively easy and user-space tools can be used for debugging any problems. On the other hand, kernel-space development comes with more constraints and practically no support for debugging. Any bug in kernel-space code typically leads to a kernel panic giving very little information about the cause of the bug. The advantage of kernel-space development is that serving user requests coming to XCPU3 will not involve context switches between Inferno user-space and kernel-space. As the choice between Inferno kernel-space and user-space does not affect the portability, we decided to implement XCPU3 in Inferno kernel-space for better performance. We are also hoping that this decision will help in easily developing a high performance native Plan 9 port for XCPU3 as the code-base of Plan 9 and Inferno kernel are similar. Our choice of Inferno gave us the flexibility and features of the Plan 9 operating system on other operating systems. It allowed us to quickly implement and deploy our filesystem on multiple platforms easily at the cost of some performance. \section{Implementation} XCPU3 is implemented as the filesystem in the Inferno kernel. Figure \ref{fig:XCPU3} shows the placement of XCPU3 in the context of applications, host operating system and the Inferno. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.4\textheight,width=0.6\textwidth] {XCPU3Structure} \fi \caption{XCPU3 Structure} \label{fig:XCPU3} \end{center} \end{figure} The XCPU3 filesystem is exported by Inferno using the \texttt{9P} protocol. This protocol is used by Plan 9 and Inferno extensively to access any file. Recently the Linux kernel has added support for the \texttt{9P} protocol\cite{graverobbers}. This allows Linux to mount filesystems exported over the 9P protocol. Other Unix based operating systems can use FUSE\cite{FUSE} for accessing the 9P based filesystems. As FUSE based filesystems run in user-space, they perform slower compared to native filesystems implemented in the kernel. But implementing FUSE based filesystems is much easier and faster. Applications can directly use the FUSE based filesystem just like a kernel implemented filesystem with the same API. Applications can also directly talk with filesystems exported over \texttt{9P} by implementing the client side of the \texttt{9P} protocol. This way they can get better performance by bypassing VFS and/or FUSE interfaces. In a typical setup, Inferno runs in the hosted environment on an operating system like Linux or Plan 9. On its startup this hosted inferno will export the XCPU3 filesystem. This exported filesystem is mounted by the local host operating system. The applications interact with this mounted XCPU3 filesystem using the native interface. Any interaction with this XCPU3 filesystem is communicated to Inferno using 9P. The XCPU3 kernel module inside the inferno kernel then receives and interprets the user actions. If needed, it uses services from the host operating system or from other XCPU3 filesystems deployed on remote locations. The XCPU3 filesystem then sends the prepared response over 9P. The host operating system will relay this response back to the application via the local filesystem interface. As we discussed before in the Design chapter this XCPU3 is an independent unit, providing the functionality of reservation, job management and computation. In the case of an isolated instantiation, any reservation request is satisfied using local resources. This way, the application can use the same semantics for both local and remote executions. \section{The big picture (connecting multiple XCPU3 nodes)} The real power of the XCPU3 is the ability to connect with each other and form bigger instance of XCPU3. XCPU3 filesystem uses the \texttt{9P} protocol to connect with each other. This protocol provides access to both local and remote filesystems in the same way. This allows every node to access the XCPU3 filesystem hosted on any node without worrying about complexities of accessing remote filesystems. Those complexities are handled by the \texttt{9P} protocol. \subsection{Central Services} The ability to configure many XCPU3 nodes into hierarchy is provided by the \textit{central services}. In contrary to the name, central services is highly distributed and every XCPU3 runs an independent instance of the central services. The administrator or the user of each XCPU3 instance need to provide only the information about its parent and children in the hierarchy and the all XCPU3 nodes initiate these connections leading to the distributed creation of the XCPU3 node hierarchy. Central services is aimed to make all the resources addressable in the network-oblivious fashion. It also aims to have external entities (such as end-user workstations) participate in the hierarchy. It is also aimed to be able to work through multiple networks, NAT gateways and firewalls. Overall, central services is aimed to grants us a flexible facility for building hierarchies across different types of network and network boundaries in a distributed and secure fashion. \subsubsection{Implementation} The central services synthetic file server itself is quite simple. It provides a simple hierarchy of directory mount points representing remote nodes. Mounts of the remote nodes or binds of previously mounted remote nodes are accomplished within this file system such that anyone who mounts our name space can also see (and access) anyone we have mounted transitively, in such a way a child node can access a parent nodes, other children, or the parents nodes parent and so forth. The original implementation of the central services mechanism used an auto-mounter-like interface. When you referenced a name within the synthetic file server for the first time, it would establish a connection to that systems central-services server and establish a “duct”. A duct is essentially a two way pipeline that allows the client to mount the sever and vice-versa -- allowing each side to access the others resources. Each would create an entry in their respective mount table based on the node-name. This allowed clients sitting behind network-address-translation gateways to contact a parent and the parent to access that child without having to re-traverse the gateway. A command-line option could be used when starting central services allowing it to connect to a single remote resource over an ssh-connection establishing a tunnel capable of crossing firewalls. It proved difficult to debug and did not allow us to leverage previously mounted filesystem (such as the parent's root filesystem from which we get our root filesystem). A second lighter weight version of the filesystem was created which allowed manipulation of the name space via a single control file at the top level with a simple name space oriented syntax: \begin{itemize} \item Establish a network connection and mount table. \begin{verbatim} mount <remote-address> <name> \end{verbatim} \item Bind previously mounted resources. \begin{verbatim} bind <path-to-remote-namespace> <name> \end{verbatim} \item Remove a resource. \begin{verbatim} remove <name> \end{verbatim} \end{itemize} Using this simple mechanism nodes could establish themselves within a hierarchy by binding a parent's central service directory to the name /csrv/parent and then tell the parent to back-mount their name space (allowing two way traversal). In this way children register with parents triggering the cross-mounts and establishing a graph which spans the entire system. On the Blue Gene, the machine already has a natural aggregation topology of compute nodes organized under IO-nodes which accessed from front-end servers. During boot, our compute nodes mount IO-nodes which mount name spaces from the front-end servers for the purposes of establishing a distributed filesystem. We leverage this same aggregation topology for the purposes of central-services and the execution model. \subsection{Example topology} The figure \ref{fig:XCPU3Topo} gives an example of how the XCPU3 filesystems can be be connected to form a larger system. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.4\textheight,width=0.6\textwidth] {XCPU3Topo} \fi \caption{Sample topology of XCPU3} \label{fig:XCPU3Topo} \end{center} \end{figure} This example shows a hierarchical topology with three levels and with a couple of nodes behind a firewall. As ducts between XCPU3 filesystem are two-way, it allows node behind the firewall to participate in the larger topology as long as they at least have one link with any other node. In our example, node F can be seen and accessed by any other node via node B. The interface for such access is explained in \textit{Filesystem interface} section. This way, central services play's key role in overcoming the partitioned networks and providing the view of all resources in network oblivious manner. \chapter{Filesystem Interface} This chapter discusses the filesystem interface and how it can be used to access any node. We will mainly concentrate on the interfaces for accessing the nodes, local resources, sessions and sub-sessions. \section{Interface for remote nodes} XCPU3 follows certain conventions which allows every node to easily interpret the filesystem view of the other nodes. This section describes the XCPU3 interface and the conventions for accessing the remote nodes. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.3\textheight,width=0.6\textwidth] {xcpu3FSTopo} \fi \caption{Sample filesystem interface for sample the topology in XCPU3} \label{fig:xcpu3FSTopo} \end{center} \end{figure} Figure \ref{fig:xcpu3FSTopo} tries to give a simple overview of how this synthetic filesystem view is populated based on the underlying topology of the nodes. The nodes in the figure are arranged in the simple hierarchical topology. The XCPU3 filesystem starts with the \texttt{[/csrv]} directory. This directory contains the synthetic contents dynamically generated by the XCPU3. The location \texttt{[/csrv/local/]} presents the local resources whereas \texttt{[/csrv/parent/]} presents the XCPU3 (ie \texttt{[/csrv/]}) filesystem of the node which is the parent of this node in the physical topology. All other directories in the \texttt{[/csrv/*]} represents the XCPU3 filesystem of the remote nodes which are the children in the physical topology. The \textit{App-1 view} is the XCPU3 filesystem view on the root node\texttt{[t]}. You can see that it has two directories in the \texttt{[/csrv/]} representing the local resources and the remote child L. The good part of this setup is that every node has to worry about only its children and the parent, other topology falls in the place automatically. In our sample case, the node \texttt{[t]} only needs to connect to node \texttt{[L]}. It can access other nodes from the filesystem view of the node \texttt{[L]}. In tern, the node \texttt{[L]} is responsible for connecting to the node \texttt{[t]} as parent and node \texttt{[l1]}, \texttt{[l2]} as children. These connections allow him to access the entire cluster. This way, responsibility of making the connections with other nodes is also \textbf{localized} leading to better scalability. The \textit{App-2 view} provides the \texttt{[/csrv/]} filesystem view at the leaf node \texttt{[c3]}. This node \texttt{[c3]} only connects to the node \texttt{[l2]} as it's parent and this single connection allows it to view the entire cluster. This particular view of resources is highly dependent on the underlying cluster. Nodes are directly connected to the neighbors and indirectly connected to other nodes, hence maintaining the relationship with underlying cluster. Most applications may not want a topology based view of underlying resources, but the uniform view. For that reason we create an uniform overlay view separately for every application when they request the reservation. This overlay view binds the resources from this underlying hierarchy and hence successfully hides the complexities created by underlying topology. As every application receives its own session directory and all the requested resources are provided within the same session directory, the application can use all the resources within its session without worrying about interfering with others or the physical location of resources. As each application lives in it's own session, this design also enables the capability of running multiple applications simultaneously and independently. The XCPU3 infrastructure is designed to work by providing resources in a uniform way irrespective of the underlying cluster topology. But there is a class of applications and users that want to have the view of resources based on the cluster topology. The view provided by the \texttt{[/csrv/]} caters to the need of these applications as this view maps the resources to the underlying cluster topology. \subsection{Creating a global view} Even though the filesystem view at different nodes differ from each other, all nodes have access to the full topology. The \texttt{[/csrv/]} filesystem view encodes enough information within itself that any nodes can construct the global view and figure out their own position within the global view. Just because every node can construct the global view, does not mean that it must use this global view for making any decision or performing a typical operation. The nodes mostly use only the local view for decision making and operations. This local view includes the parent node and the children nodes. The ability to construct a global view is available to the applications running on any node which need this information. This operation is comparatively expensive as it involves traversing the \texttt{[/csrv/]} filesystem view. As each directory in \texttt{[/csrv/]} location except \texttt{[/csrv/local/]} is the XCPU3 filesystem of a remote node, any node can access the XCPU3 filesystem of other nodes using this traversal and use this information to build the global view. The XCPU3 infrastructure provides the interface to generate such global views, and leaves the decision to use this information at the expense of performance to the application developers and users. \section{Interface for local resources} This section describes the interface for accessing and managing the local resources at any node. Again this interface is uniform across all the nodes enabling applications and other nodes to access it in the same way. All local resources can be accessed in the location \texttt{[/csrv/local/]}. Figure \ref{fig:xcpu3Local} shows the typical content of this location. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.4\textheight,width=0.4\textwidth] {xcpu3Local.png} \fi \caption{Sample filesystem interface for local resources in XCPU3} \label{fig:xcpu3Local} \end{center} \end{figure} Each file and directory in the \texttt{[/csrv/local/]} has a specific purpose in accessing and managing the local resources. %\begin{enumerate} \subsection{arch} \texttt{[/csrv/local/arch]} is a \textit{read-only} file. A read operation on this file returns the name of the host operating system and the hardware architecture. This information is non-destructive and can be read any number of times. The typical use-case of this file is \begin{verbatim} $ cat /csrv/local/arch Linux 386 $ \end{verbatim} Typically, this information will be used by the scheduler to satisfy the constraints imposed by the applications for the underlying operating system and architecture. \subsection{env} \texttt{[/csrv/local/env]} provides the interface for specifying the environmental variables. This file can be opened for writing and multiple tuples of \texttt{[Name Value]} pairs can be written into it. Following is the example of how this file can be used \begin{verbatim} $ echo "PATH /home/example/" > /csrv/local/env $ \end{verbatim} The environment variables added this way are the default variables and they will be available to all the sessions started on this node. We do provide another interface for controlling the environmental variables within a session. We will discuss that interface in more details in the next section. \subsection{ns} \texttt{[/csrv/local/ns]} file provides the interface to assemble the namespace that will be inherited by any application process created on this node. \begin{verbatim} $ echo "mount /exp/root/ /" > /csrv/local/ns $ echo "bind /some/path /other/path" > /csrv/local/ns $ \end{verbatim} This is the default namespace and can be overwritten for a particular session using another interface, allowing more fine-grained control. \subsection{fs} \texttt{[/csrv/local/fs]} location gives the interface to the host filesystem. This interface allows the applications to access the local filesystem if needed. \subsection{net} \texttt{[/csrv/local/net]} location gives the interface to the host network. This interface is in the Plan 9 style filesystem which can be used create new network connections\cite{net}. The network connections created using this interface will use the local networking infrastructure. This is quite useful when the application wants to rely on the local networking interfaces instead of the abstractions provided by XCPU3. \subsection{status} \texttt{[/csrv/local/status]} is the read-only file providing information about the available resources at local node and remote nodes. The information includes the number of nodes for each combination of an operating system and an architecture. These statistics include all the remote nodes which are the descendant of this node. These descendants are traversed by avoiding the \textit{parent} links. Every node reads the \texttt{[/csrv/<child-node-name>/local/status]} file of all his children nodes and then aggregate this information to produce its own status file. As all the children are XCPU3 nodes themselves, when the parent reads their status file, they repeat the same action of reading and aggregating the status files of their children. This way the process is recursively repeated till it hits the leaf nodes. These nodes terminate the recursion by returning the information about themselves. Following is an example of the content of status file at a compute node of the Blue Gene which is a leaf node. \begin{verbatim} $ cat /csrv/local/status 1 Plan9 power $ \end{verbatim} This information is aggregated back till the root node which then returns to the user. Following is the example of an aggregated information on the Blue Gene computer. \begin{verbatim} $ cat /csrv/local/status 1 Linux power 65 Plan9 power $ \end{verbatim} The above description shows how nodes help each other to perform the operation that needs the global information. Unfortunately, these global operations do involve lot of communication making them comparatively expensive. So, hereby we inform the users about being careful in using these global operations. We have plans to reduce the cost of such global operations by caching the information of previous global operations. We expect that the global view of the system will not change very frequently, allowing us to cache this information. We have not implemented this yet but we plan to implement such caching in near future. \subsection{clone} \texttt{[/csrv/local/clone]} is a read-write file. This file is responsible for creating the sessions which can be used to run the applications. Whenever this file is opened, it allocates the session to it. This session can be a new session with the new id or a previously used but free session. The session is identified by the session-id which is a non-negative number and it is represented as a directory with the session-id value as the name. The directories \texttt{[/csrv/local/0/]}, \texttt{[/csrv/local/1/]} and \texttt{[/csrv/local/n/]} are the session directories. Whenever the \texttt{[/csrv/local/clone]} file is opened, a session is allocated to it. And all subsequent reads on this file will return the session-id of the session allocated. Any writes on this opened file will be considered as writes on the \texttt{[/csrv/local/<session-ID>/ctl]} file. If this file is closed and if there is no other open file within this session directory then this session is released for future use. The typical way for using the clone semantic is \begin{enumerate} \item Open clone file in read-write mode. \item Read the session-id. \item Use the files within that session directory for starting and managing the computation. \item Once your computation is over, close all those files within session. \item Close the clone file to release the session and the resources allocated. \end{enumerate} \subsection{Sessions} \texttt{[/csrv/local/0/]}, \texttt{[/csrv/local/1/]} and \texttt{[/csrv/local/n/]} are the session directories in figure \ref{fig:xcpu3Local}. New sessions are created as and when needed and the session-id value increases monotonically. Every session directory has its own files with its own structure. Next section will discuss the session interface in more details. \section{Session interface} This section explains the filesystem interface of the sessions in the XCPU3. These sessions are important because they are the single unit of execution which can be entirely controlled by the users. The figure \ref{fig:xcpu3Local} shows the directory structure of a single session. \begin{figure}[h] \begin{center} \leavevmode \ifpdf \includegraphics[height=0.44\textheight,width=0.4\textwidth] {xcpu3Session.png} \fi \caption{Sample filesystem interface for sessions in XCPU3} \label{fig:xcpu3Local} \end{center} \end{figure} \subsection{ctl} \texttt{[/csrv/local/<session-ID>/ctl]} is the read-write file which provides the control over the session. Just like the clone file, a read operation on this file will return the session-id. Write operations on this file are treated as the management commands for this session. These commands can be used to do the reservations of the remote resources, requesting the execution or the termination of an ongoing execution.