\chapter{Evaluation}\label{chap:evaluations} \ifpdf    \graphicspath{{Chapter4/Chapter4Figs/PDF/}{Chapter4/Chapter4Figs/PNG/}		{Chapter4/Chapter4Figs/}}\else    \graphicspath{{Chapter4/Chapter4Figs/EPS/}{Chapter4/Chapter4Figs/}}\fi This chapter presents the evaluation of the XCPU3 infrastructure from theperspective of job deployment.  XCPU3 is an implementation of a new methodologyfor the workload deployment for a new class of problems. Also, XCPU3concentrates on quicker deployment of a large number of small jobs while givingclean interfaces and abstractions.  Existing infrastructures do not concentratedirectly on this issue of deploying a large number of small jobs.  As XCPU3 isventuring into this unexplored domain, we do not have any fair ground forcomparing the performance of XCPU3. Another issue stopping us from performing extensive evaluations with otherinfrastructures is that the Blue Gene infrastructure is not very hospitable toother infrastructures which are primarily aimed at traditional HPC operatingsystems like Linux.  There are efforts going on in the form of theKittyhawk\cite{kittyhawk} project for using Linux for providing the traditionalenvironment on the Blue Gene.  Once the Kittyhawk infrastructure is up,comparisons with other solutions are  possible. \begin{figure}[h]  \begin{center}    \leavevmode    \ifpdf      \includegraphics[height=0.3\textheight,width=0.4\textwidth]		{contributions}    \fi    \caption{The big picture}    \label{fig:big_picture}  \end{center}\end{figure} XCPU3 is the part of the project HARE\cite{evh08hare}.  Figure\ref{fig:big_picture} shows the global picture and how these components workwith each other.  This is a layered design where XCPU3 uses the servicesprovided by \textbf{CSRV}. The services provided by XCPU3 can either be directlyused by an application or can be used by another infrastructure like PUSH forproviding targeted interfaces for dataflow applications.  As we are still working on this integration with PUSH, we will limit our performanceevaluations to simpler applications.  We plan to do more in-depth performance evaluation after the integration process is completed.   % The XCPU3 is the part of the project HARE\cite{evh08hare}.  We rely on other PUSH% for deployment of large scale dataflow applications on top of the XCPU3.  As we% are still working on this integration with PUSH, we will not be able to present% the performance evaluations for large dataflow applications.  \section{Experimental setup}We have performed our evaluations on a Blue Gene setup with 512 nodes.  Thissetup is visually presented in a figure \ref{fig:hare}.  We run hosted Infernoon all the compute nodes, IO nodes and the controller node.  The user interacts with the XCPU3 instance on the controller node for job submission.  \begin{figure}[h]  \begin{center}    \leavevmode    \ifpdf      \includegraphics[height=0.3\textheight,width=0.4\textwidth]		{EvaluationSetup}    \fi    \caption{Setup for evaluation}    \label{fig:hare}  \end{center}\end{figure}  \subsection{Scalability of XCPU3}Our first objective is to show how quickly we can do deployment and executionof a large number of small applications. We have avoided using largerapplications as the bigger runtimes of larger applications tend to amortize theoverhead in the deployment of the application.  We have used the \texttt{date}command as the application for deployment.  This is a small application and doesnot need any external inputs and produces small output. Each deployment involvessession creation, reservation, execution, output aggregation and terminationof the session.  We deployed varying numbers of execution of this application onthe cluster of 512 nodes.  The number of requested executions increasedexponentially from 1 to 2048 executions. \begin{figure}[h]  \begin{center}    \leavevmode    \ifpdf      \includegraphics[height=0.4\textheight,width=1.0\textwidth]		{linear}    \fi    \caption{Comparison if XCPU3 with sequential deployment}    \label{fig:sequential}  \end{center}\end{figure} Figure \ref{fig:sequential} gives an initial perspective of how XCPU3 performsrelative to sequential performance.  This graph plots the total time taken byXCPU3 and the hypothetical time it may take for performing the same amount ofwork on one machine.  This graph shows that the XCPU3 is successfully able toexploit the parallelism for deploying the jobs quickly. The XCPU3 deploys 2048jobs in 12.66 seconds whereas sequential execution would take upto 510 seconds.In the graph, the line showing sequential scaling looks exponential, but that isbecause number of requested executions increase exponentially.  \subsection{Job deployment without input}This section aims to further analyze the performance of XCPU3.  Again we are concentrating on similar deployment.  We have recorded the time taken by eachof the following stages in the deployment on the XCPU3 infrastructure. \begin{enumerate}\item Reservation: Create a new session, and request the reservation by writing\texttt{res n} into the session \texttt{[ctl]} file.  Here \textbf{n} variesfrom 1 to 2048 representing the number of executions requested.  \item Execution: Request the execution by writing \texttt{exec date} into thesession \texttt{[ctl]} file. \item Aggregation: Collect the output generated by all the executions by readingthe session \texttt{[stdio]} file. \item Termination: Closing all the files and terminating the session. \item Housekeeping: Additional time taken before, between and after above steps.\end{enumerate}Every deployment starts with the creation of the session followed by thereservation, execution, aggregation and then ending with termination of thesession.  We have run every deployment three times, and measuredthe time taken by each step.  We have taken the average value over thesemultiple runs for our analysis.  As the Blue gene setup is in controlledenvironment, we do not expect big variations in multiple runs.  Also, we haverepeated the entire experiment multiple times and always got similar results.We are discussing the measurements from one such experiment. \begin{figure}[h]  \begin{center}    \leavevmode    \ifpdf      \includegraphics[height=0.4\textheight,width=1.0\textwidth]		{date_graph}    \fi    \caption{Deployment without input}    \label{fig:date_graph}  \end{center}\end{figure} Figure \ref{fig:date_graph} presents the results of deployment of the datecommand in the form of graph.  This graph presents the breakup time for variousstages of the deployment using the XCPU3 infrastructure. From this graph we can observe that the session termination and the housekeepingoverheads are negligible compared to the time taken by reservation, executionand aggregation.  So, we can ignore these two overheads in our futureevaluations.  For jobs of up to 128 deployments, the reservation time dominateseverything else.  But for larger numbers of deployment execution andaggregation time increases rapidly while reservation time remains relativelyconstant. This shows that reservation time is not directly dependent onthe number of deployments, whereas execution and aggregation time are directlyproportional to the number of deployments. Now, let us try to analyze why reservation time is independent of the number ofdeployments.  The reservation process involves traversing the underlyingtopology tree of nodes till the reservation requirements are satisfied.  All thechildren on the same level are traversed parallelly at the same time.  This way,each level is traversed in the constant time, independent of number of nodes inthat level.  Another aspect of the reservation mechanism which helps here isthat the amount of data written and read from the \texttt{[ctl]} file and theamount of data exchanged between nodes for communicating reservation request isfixed in size and independent of the number of deployments requested.  Withthese two properties, the reservation time becomes directly proportional to thedepth of the tree and not with the number of nodes. We can observe the above relation in figure \ref{fig:date_graph}. The reservation timeremains relatively constant for deployment requests from 1 to 8.  Then it sharplyincreases between 8 to 16 and remains almost constant for all the requests between16 to 2048.  This can be attributed to underlying cluster topology.Figure \ref{fig:hare} shows the presence of the 8 IO nodes in the first level.This enables satisfying the requests which are smaller than 8 executions.For larger requests, one more level needs to be traversed in the topology,introducing delays.  The time taken for reservation remains almost constant between16 and 2048 executions as all these reservation requests essentially traverses the samedepth.  We can conclude from these observations that \textit{the time taken for the reservationis directly proportional to the depth of the tree}.  Now let us discuss, why the same property is not exhibited by execution time or aggregation time.We have discussed in the implementation chapter that all read and write requests are performed in parallel between all the nodes in the same level.  But the amount of dataexchanged for aggregation and execution is not constant.  This data is directly proportionalto the number of nodes involved.  With the increase in the number of requested deployments,the amount of data to be exchanged also increases, leading to larger aggregation time.The execution time is also similarly affected as all compute nodes will try to fetch thebinary of the executable from the initiating node leading to the copy of the data.These observations lead us to to conclusion that \textit{the time taken for the execution andaggregation is directly proportional to the number of deployments requested}. \subsection{Job deployment with input}Our next evaluation involve the deployment of an executable \texttt{wc} whichneeds input. This command counts the number of lines, words and characters inthe input file.  This is an interesting case for our infrastructure as thisdeployment involves the distribution of inputs to all the sessions.  Thisintroduces a new stage in the deployment process in addition to the 5 stages wedescribed in the above section.  This stage will be the \textbf{input} stage andinvolves distributing the input data to all the sessions which are responsiblefor execution.  By default XCPU3 will broadcast the input to all the computenodes, but we have plans to introduce filters in the PUSH shell which canpartition the input given to the compute  nodes.  Figure \ref{fig:wc_graph} presents the results of evaluations involving the distribution of the input.  \begin{figure}[h]  \begin{center}    \leavevmode    \ifpdf      \includegraphics[height=0.4\textheight,width=1.0\textwidth]		{wc_graph}    \fi    \caption{Deployment with input}    \label{fig:wc_graph}  \end{center}\end{figure} These results enforce our observations that reservation time is directlyproportional to the the depth of the tree whereas aggregation and execution timeare directly proportional to the number of deployments requested.  In additionto these observations, we can also observe that the input time exhibits behavior similar to the execution and aggregation time.  This observation can beattributed to the fact that input distribution implementation is similarto the output aggregation implementation.  And also the amount of data to beexchanged for input distribution depend on the number of deployments requestedas this data needs to reach all sessions responsible for execution.  Thisincreases the amount of data to be exchanged with any increase in the number ofthe deployments requested.  We can conclude with this observation that\textit{the input time is directly proportional to the number of deploymentsrequested}  \section{Dataflow workloads}The evaluations presented in this chapter helps us in understanding how fastXCPU3 can deploy small jobs and aggregate the results produced by them.  Nowlet us relate how these observations can justify our claims about dataflowapplications. Typical deployments of the dataflow applications are similar tothe above experiments as it involves starting up a large number of small jobs.But the similarity is over at this point. Dataflow deployment does not alwaysinvolve the input distribution and output aggregation stages.  These deploymentswork by feeding the output of one computation as input for other computation. At the end of the computation, a user needs to read the output of only selectedcompute sessions which does not need any aggregation.  What we can conclude fromthe above description is that XCPU3 performs really well in the stages likereservation which are important for dataflow deployments.  The stages like inputdistributions are output aggregation where XCPU3 is relatively slow are notneeded by dataflow deployments.  This makes XCPU3 ideal for rapid deployment ofdataflow workloads. Unfortunately we do not have concrete measurements and evaluations to back ourpredictions.  XCPU3 is one of the piece in the envisioned solution for  problemof efficient and easy deployment of large-scale dataflow applications.   We arestill working on the integration of userspace applications like PUSH with XCPU3which will further simplify the dataflow deployment. As the entire envisionedsolution is out of the scope of this document, we are leaving out the detailsabout it.  We are also working on reaching out the high performance computingcommunity for porting real life large scale dataflow applications to thisplatform for evaluation purpose. We are hopeful that we will have have extensiveperformance evaluations for real life dataflow applications on this platform innear future.  In this document, we have only attempted to provide a strong casefor the abilities of XCPU3.  XCPU3 is an infrastructure which provides the needed flexibility, speed and ease of use for dataflow workloads.  This chapter demonstrates the speed thatcan be achieved.  It is difficult to measure the properties like \textit{ease ofuse} and \textit{flexibility} but the examples presented in the filesysteminterface chapter should give some insights of all the possibilities opened upby the XCPU3.   % ------------------------------------------------------------------------ %%% Local Variables: %%% mode: latex%%% TeX-master: "../thesis"%%% End: