\documentclass[conference]{sig-alternate} \ifCLASSINFOpdf \usepackage[pdftex]{graphicx}\else\fi\bibliographystyle{plain}   \begin{document}\title{Brasil: Basic Resource Aggregation System Infrastructure Layer}  \author{\IEEEauthorblockN{Pravin Shinde}\IEEEauthorblockA{Parallel and Distributed Systems(PDCS)\\Vrije Universiteit, Amsterdam\\Email: pse220@few.vu.nl}\and\IEEEauthorblockN{Eric Van hensbergen}\IEEEauthorblockA{IBM Research Lab\\Austin, USA\\Email: ericvanhensbergen@us.ibm.com}} \maketitle \begin{abstract}  As clusters and grids are reaching to domains like commercial application, itis becoming more clear that the existing workload distribution solutions whichare tailored for high performance scientific applications, are not efficientfor dynamic data-flow applications and many-task computing (MTC). Most of the existing workload distribution solutions that concentrate more onHPC applications and are not efficient in quickly deploying the dynamicworkload involving large number of small jobs which is the typical case indataflow applications. Also developing and deploying applications on existinginfrastructures still is a non-trivial task requiring special runtimes,middleware support and/or language dependence. We have explored an alternate approach for simplifying workloaddistribution and aggregation based on synthetic filesystems and the privatenamespace concepts of Plan 9.  Instead of sending workloads to a remotecluster, Brasil works by bringing the remote cluster to the user's desktop,leading to full control over the compute environment.   Brasil providesa filesystem interface, making it runtime and language agnostic.  It alsoallows executing multiple jobs simultaneously in isolation, leading tobetter resource utilization.  Brasil makes all nodes independent andequivalent in functionality leading to the ability to localize thedecision making and ability to handle dynamic workloads.  Brasil provides easy touse, flexible and scalable infrastructure for workload distribution andaggregation that can be used by dynamic workloads, many-task workloads anddataflow applications. \end{abstract} \IEEEpeerreviewmaketitle \section{Introduction} Due to the limits on the silicon technology, the trend is moving in thedirection of multicore and manycore architectures.  Having a large number ofless powerful computers is assumed to be economically more viable than having asingle more powerful computer.  The increasing acceptance of clusters andgrids coupled with the emergence of cloud computing and availability of computeresources as utility via \cite{ec2} \cite{azure} \cite{appengine} has increasedthe need for parallel programming and workflow management. The initial development around parallel programming and the traditionalinfrastructure around clusters and grids is designed to suit therequirements of the scientific community which tend to run large computeintensive applications for long durations.  With growing use of clusters andgrid, there is increasing interest in deploying new genre of applications likedataflow applications and many-task applications. Typical commercialapplications fall into the category of dataflow applications and involveexecuting lot of small and interdependent applications for small amount of time.Unfortunately, most of the existing workload deployment solutions are ill-suitedfor these applications. These solutions are efficient in deploying largecompute-intensive application which will run for long duration, but are highlyinefficient in deploying large number of small applications for small amount oftime. The major obstacle in widespread adoption is the expertise needed for writingparallel applications.  Also, dependence on a specific language or runtimeenvironment for parallelism dictates the need for major re-writing all existingapplications for exploiting the parallelism.  Another crucial property exhibited by dataflow applications is that the amountof resources needed for total computation is unpredictable in the beginning asit may depend on results of intermediate computations.  This leads to therequirement of dynamically allocating new resources whenever needed. Most of thecurrent job schedulers do not support such adaptability and flexibility.  In this paper, we target the problem of creating a scalable infrastructurewith the flexibility needed for running dataflow applications.  We primarilytarget the Blue Gene \cite{bgp} supercomputer as our testbed as it allows us totest our infrastructure on up to 64K nodes.  The challenge is to provide the flexibility needed by dataflow applicationsusing a very simple interface.  We need to provide an infrastructure withsupport for dynamically adjusting the resource reservation which can help incoping with unpredictable resource requirements.  This infrastructure needs toprovide quick job-startup time even for large numbers of jobs, allowing overallgood performance for dataflow applications.  It also needs to provide a simpleinterface for communication between nodes even when those nodes cannotcommunicate directly with each other. We do not want the application developmentfor this infrastructure to be constrained to a particular language, runtime,middle-ware or platform.  And we need to run most of the existing applicationswithout modification.  Our primary aim is not performance improvement, but to present a new and elegantway for workload distribution and aggregation.  It is aimed at demonstratingthat simple filesystem interfaces and a simple design based on the basicprinciples of the Plan 9 \cite{pike95plan} distributed operating system can beeasy, intuitive and scalable to a large number of nodes. This paper is organized as follow. The next section presents the backgroundwork which has inspired this project. The \textit{Design section} will discussthe design and the reasoning behind it. \textit{Implementation section} willdiscuss how system is implemented. The \textit{Filesystem interface section}explains the use of synthetic filesystem and presents few examples to show howeasily this interface can be used. Evaluations section presents some resultsshowing that this approach is feasible. Related work section positions Brasil inthe context of other related projects. Conclusion section will conclude thisproject. \section{Background}This section discusses the the previous work from which Brasil is evolved. \subsection{Plan 9}Plan 9 is an operating system which was specifically designed for dealing withnetworked systems.  It is based on principles like \textit{everything is file}and \textit{private namespaces}\cite{namespace}.  It provide applicationsability to build custom namespace by mounting and binding remote resources andthen run in network oblivious manner.  The usefulness of these features can beseen in CPU\cite{plan9-cpu} which is Plan 9's way for using remote computeresources.  Unlike Unix based remote login and SSH where remote processes runin the namespace and environment of remote node, CPU runs the remote processesin local namespace and environment.  One of the key advantages of this modelsimplifies the remote workload deployment as even though workload runs onremote resources, it runs in local environment without getting affected by theenvironment on remote node. \subsection{XCPU}XCPU\cite{ron-xcpu} \cite{lucho-xcpu} was an attempt to bring the functionalityof CPU into mainstream operating systems like Linux.  XCPU allows processmanagement using a \textbf{filesystem interface}.  As this underlying pseudofilesystem can be exported over network to other nodes, it simplifies thesharing of the compute resources.  XCPU does not provide private namespace,which is needed for transparently accessing remote filesystem.  XCPU worksaround this limitation by pushing the executable and it's dependencies neededfor requested remote execution. To achieve scalable pushing of programs and their dependencies, XCPU uses a fewcompute nodes for distributing the programs and dependencies during the programstartup time creating a tree hierarchy of the nodes.  This is known as the\textbf{tree spawn mechanism} and it improves the scalability for XCPU tolarger numbers of nodes. With automatic pushing and the tree spawn mechanism,XCPU not only simplifies the deployment of applications but it also scalesthem better with quicker job startup time. \subsection{XCPU2}XCPU2 \cite{lucho-xcpu2} is evolved from XCPU and is aimed to give moreflexibility by using \textit{private namespace} support added recently intoLinux kernel\cite{linuxns}.  This allows remote processes to execute in thenamespace which provides local filesystem view.  By controlling theenvironment and filesystem view of all the remote processes, every developeris free to use any libraries and development setup, regardless of what isinstalled on remote nodes.  As these processes run in the isolation provided bythe private namespace, they do not pollute the environment and filesystem viewof other processes.  This allows executing multiple processes of differentusers on same remote node without conflicting with each other. XCPU2 makes clear distinction between various nodes which are assigned withspecific responsibility.  The nodes are classified into \textit{control node}which is responsible for reserving the compute nodes, \textit{job control node}which are responsible remote job management and namespace management on computenodes. The \textit{compute nodes} are responsible for actual computationby running the processes on inside custom namespace on users behalf. The XCPU2 lacks fine-grained granularity of the control.  It only supportsmanaging all \textit{compute nodes} together in identical way, and does notgive any mechanism to control each \textit{compute node} individually.  Thislack of control on every node is a major deterrent for dataflow applications.This limitation was major motivation for us to develop Brasil for providing muchneeded flexibility and fine grained control over over all compute nodes usingsimple interface.  %%%%%%%%%%%%%%%%%%%%%%%%%%%% End of Introduction chapter %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Start of Design chapter %%%%%%%%%%%%%%%%%%%%%%% \section{Design} The key challenge in this design was to provide the flexibility needed bydataflow applications while achieving scalability to the large number of nodesavailable.  \subsection{Key design decisions} We decided to adopt a few guiding principles which will help us to reach ourgoal of flexibility with scalability.  We also decided to keep the design simple,and we believe that the simple design should lead to the simple and flexibleinterface. We also hope that the lack of complexity may help in improvingscalability. In this section we present those decisions which influenced mostaspects of system design and implementation. \subsubsection{Localization of decision making}The key requirement for us was scalability to a large number of nodes.  Weplanned to design the system without any central component which should haveknowledge of the entire system.  We plan to distribute and localize thecomputations as well as decisions like scheduling, job management andworkload distribution/aggregation. We avoid decision making based on global knowledge and promote use of localinformation.  We hope that if each node tries to attempt a local optimum, wewill reach the global optimum.  This may not be true in all cases, We hopethat in those cases, we hope to perform acceptably well if not optimal.  \subsubsection{Independence of every node}As we plan to distribute and localize all functionalities, it was essential toreplicate these functionalities at multiple levels making localizationpossible.  The granularity of functionality replication decides the granularity of control,and hence influences the flexibility provided by the system.  As we aim for maximum flexibility, we have decided for replicating the following threefunctionalities at each node.\begin{enumerate}\item \textbf{Resource reservation}: Each node should be able to reserve moreresources on it's own without involvement of any central entity. \item \textbf{Job management}: Every node should be capable of starting andmanaging new jobs using his reservations. \item \textbf{Computation}: Every node should be able to perform thecomputation by running the requested application in isolation and returningthe results.\end{enumerate} We intend to provide each node with the capability to perform all of these rolessimultaneously instead of binding them in one role at one time.  This designmakes the interface to every node a building block identical to each other, and provides the flexibility to build any structure with these building blocks. There are a few downsides in making every node independent.  With independenceof every node, each node can be a source of failures and faults.One will need to come up with better ways to deal with faults andfailures when so many sources of them are present.  This takes the Brasil inrealms of \textbf{Distributed Systems} opening many more possibilities andquestions. For purpose of this exploration, we avoid these complexities by assuming a verysimple model for handling failures.  Any failure anywhere in the system willresult in the failure of the entire operation.  We assume that failures willbe in-frequent, so aborting and restarting operation should be acceptable forsuch infrequent failures. \subsubsection{Filesystem Interface}We want to keep Brasil interface agnostic from language, runtime and middleware.Plan 9 has proven that the filesystem interface is very flexible and yetpowerful in the world of distributed applications.  We aim to follow the sameprinciple of \textbf{Everything is a file} from Plan 9. Every node will provide access to its services via filesystem interfaces.This interface will be exported as the filesystem over the network so that othernodes can use it. Other nodes can mount this filesystem and use it asinterface for interacting with that node.  Multiple remote nodes can beaggregated into the filesystem hierarchy providing a clean and easy way toaccess them. Multiple overlay views can be created by \textit{binding} the same filesystemat multiple locations with different names.  This ability of creating ad-hocoverlays allow users to arrange remote resources as per his needs withoutworrying about their actual locations. Other advantages of using filesystem interfaces are that \begin{enumerate}\item Existing tools/commands used with traditional filesystem can be directlyused with Brasil. \item filesystems come with their own mechanism for access control list forproviding the security.  We can leave the security to these already proven mechanisms instead of implementing our own. \item We inherit the ability to export, mount and bind the filesystemwithout writing any explicit code for it. \item Filesystem interfaces are simpler to program than socket interfaces.  Thiscan lead to simpler code and hence lesser bugs. \item Users don't need special privileges or administrative access to interactwith the filesystem.  This simplifies the user experience in running Brasil basedapplications.\end{enumerate} The filesystem interface has following limitations which affect our systemdesign. \begin{enumerate} \item The critical limitation concerns the POSIX standard for failurereporting in filesystem.  POSIX standard dictates that file operations shouldreport their success or failure in the form of a single number which may not beenough to provide the information about the reason behind failures.  Plan 9overcomes this limitation by returning a string which can provide moreverbose information instead of a single number.  But this breaks the POSIXcompatibility.  So, we need to find an alternate way to report informationabout failures in POSIX compatible way. \item Another drawback is the way a failure of remote services is detected.The filesystem interface relies on the underlying networking protocol fordetecting failures by waiting for timeouts, and then reporting them back tousers as an error.  This makes the filesystem interface less desirable wherequick failure detection and recovery is needed.  \end{enumerate} Our decision to choose the filesystem interface despite of it's drawbacks is thetrade-off we are willing to do for flexibility and simplicity.  We limitourself with the assumption that failures will be infrequent.  With thisassumption, we are willing to accept the delays in reporting failures at theremote end.  \section{Implementation} This section discusses the implementation issues in more detail.  This chapterconcentrates on how Brasil was implemented and discusses implementationdecisions which have proven to be important.  \subsection{Inferno}Inferno\cite{inferno} is an open source distributed operating system which is a direct descendant of the \textbf{Plan 9} operating system.  It runs natively onmultiple hardware platforms and can also run as a user-space application ontop of other operating systems.  For the purpose of this document\textit{"hosted Inferno OS"} or \textit{"userspace Inferno"} refers to Infernorunning as user-space application on top of other operating systems. As we are aiming for a flexible heterogeneous environment with differentoperating systems and different architectures, we found the Inferno operatingsystem to be an attractive platform.  It allowed us to develop Brasil once andeasily deploy it on different platforms while enjoying the Plan 9 featureson all those platforms. This decision does come with the cost of performance loss.  Each node needsto run hosted Inferno in user-space which takes up some resources of the node. Our choice of Inferno gave us the flexibility and features of the Plan 9operating system on other operating systems.  It allowed us to quicklyimplement and deploy our filesystem on multiple platforms easily at the costof some performance. \subsection{Implementation details} Brasil is implemented as the filesystem in the Inferno kernel.  Figure \ref{fig:Brasil} shows the placement of Brasil in the context of applications,host operating system and the Inferno. \begin{figure}[h]  \begin{center}    \leavevmode      \includegraphics[height=0.2\textheight,width=0.5\textwidth]		{./img/BrasilStructure}    \caption{Brasil Structure}    \label{fig:Brasil}  \end{center}\end{figure} The Brasil filesystem is exported by Inferno using the \texttt{9P} protocol. This protocol is used by Plan 9 and Inferno extensively to access any file. Recently the Linux kernel has added support for the \texttt{9P}protocol\cite{graverobbers}.  This allows Linux to mount filesystemsexported over the 9P protocol.  Other Unix based operating systems can useFUSE\cite{FUSE} for accessing the 9P based filesystems.  In a typical setup, Inferno runs in the hosted environment on an operatingsystem like Linux or Plan 9.  The hosted inferno will export the Brasilfilesystem which can be mounted by the local host operating system. The applications interact with this mounted Brasil filesystem using the nativefilesystem interfaces (e.g. VFS for Linux). Any interaction with this Brasilfilesystem is communicated to Inferno using \textit{9P}. The Brasil kernel moduleinside the inferno kernel then receives and interprets the user actions.  Ifneeded, it uses services from the host operating system or from other Brasilfilesystems deployed on remote locations.  The Brasil filesystem then sends theprepared response over 9P. The host operating system will relay this responseback to the application via the local filesystem interface.  \subsection{The big picture (connecting multiple Brasil nodes)}The real power of the Brasil is the ability to connect with each other and formbigger instance of Brasil.  Brasil filesystem is exported over \texttt{9P}providing same functionality and interface to both local and remoteapplications.  This allows Brasil instances to connect with other Brasilinstances and distribute some of their responsibilities among them.The \texttt{9P} protocol shields the Brasil from complications of remotefilesystem access and works as transparent glue for connecting multiple Brasilinstances together. \subsubsection{Central Services} The ability to configure many Brasil nodes into hierarchy is provided by the\textit{central services}.  In contrary to the name, central services ishighly distributed and every Brasil runs an independent instance of the centralservices.    %\subsubsection{Implementation}  The central services synthetic file server which provides a simple hierarchy ofdirectory mount points representing remote nodes.  Mounts of the remote nodesor binds of previously mounted remote nodes are accomplished within this filesystem such that anyone who mounts our name space can also see (andaccess) anyone we have mounted transitively,  in such a way a childnode can access a parent nodes, other children, or the parents nodesparent and so forth. Any node could establish themselves within ahierarchy by binding a parent's central service directory to the name\texttt{[/csrv/parent]} and then tell the parent to back-mount their name space(allowing two way traversal).  In this way children register with parentstriggering the cross-mounts and establishing a two way link between them. EachBrasil instance need to know only the information about its parent and childrenin the hierarchy and the all Brasil nodes initiate these connections leading tothe distributed creation of the Brasil node hierarchy. Figure \ref{fig:xcpu3FSTopo} tries to give a simple overview of how this synthetic filesystem view is populated based on the underlying mountconnections between the nodes. \begin{figure}[h]  \begin{center}    \leavevmode      \includegraphics[height=0.3\textheight,width=0.4\textwidth]		{./img/xcpu3FSTopo}    \caption{Sample filesystem interface for sample the topology in Brasil}    \label{fig:xcpu3FSTopo}  \end{center}\end{figure} Assuming the links between the nodes are created by the remote node mounts incentral services, this diagram shows how the filesystem views at differentnodes encompasses the whole network, even though each node is only connected toits neighbours. The Brasil filesystem starts with the \texttt{[/csrv]}directory. The location \texttt{[/csrv/local/]} presents the local resourceswhereas \texttt{[/csrv/parent/]} presents the Brasil filesystem of the parentnode. All other directories in the represents the Brasil filesystem of thechildren nodes.  It can be easily seen that both \textit{App-1} and\textit{App-2} have access to all the nodes even though they are running ondifferent nodes.  In this design, every node has to worry about only itschildren and the parent, other topology falls in the place automatically. Even though the filesystem view at different nodes differ from each other, all nodes have access to the full topology.  The \texttt{[/csrv/]} filesystemview encodes enough information within itself that any nodes can construct theglobal view and figure out their own position within the global view. Just because every node can construct the global view, does not mean that itmust use this global view for making any decision or performing a typicaloperation. The nodes mostly use only the local view for decision making andoperations. This local view includes the parent node and the children nodes.  \section{Filesystem Interface} This section discusses the filesystem interface and how it can be used formanaging the local and remote resources. Figure \ref{fig:xcpu3Local} gives thethe high-level view of the hierarchy of the Brasil filesystem.  This sectionwill discuss the overall hierarchy and few important files in the Brasil. \begin{figure}[h]  \begin{center}    \leavevmode      \includegraphics[height=0.25\textheight,width=0.4\textwidth]		{./img/local_session_subsessions}    \caption{Filesystem interface in Brasil}    \label{fig:xcpu3Local}  \end{center}\end{figure} The location \texttt{[/csrv/local/]} points to the local resources available onthe node and the files present in this directory allows managing these localresource. Few files here provide information about the system, like\texttt{[arch]} file tells the architecture and operating system of the nodewhile \texttt{[status]} file provides information about total amount ofresources available, including the remote resources which are direct orindirect decedents of this node.  Files like \texttt{[env]} and \texttt{[ns]}allowed controlling the default environment and the namespace of the processescreated on this node.  \texttt{[fs]} and \texttt{[net]} are links to the localfilesystem and networking resources available on the node.  The\texttt{[clone]} file provides the interface to create new sessions which isthe unit of the execution. The sessions are represented as directories with session-id number as a name. These sessions are self-contained and provide interfaces to manage theexecution of the session.  Files like \texttt{[env]} and \texttt{[ns]} arepresent is session directory also, and can be used to overwrite the defaultenvironment and namespace specifically for this session.  File \texttt{[ctl]} is

Lines 1–500 of 994. Load more