Shared Cluster How-To

Introduction

Our departmental cluster is composed of a head node which runs cluster management software and twelve computational nodes. Our cluster management software is Sun Grid Engine (SGE). It's also sometimes referred to as N1 Grid Engine. SGE manages and prioritizes computational jobs in a queue, then assigns the jobs to available processing slots. SGE represents each processor in the cluster as slot where a job can execute. The computational nodes have a total of 184 64-bit processors and 26 32-bit nodes, so when all nodes are functioning properly, there are 210 available slots for jobs. Jobs must be explicitly submitted to the cluster using the procedure described below. The IBM blade center is a 32-bit system and as such requires different software packages than the 64-bit Dell servers. To avoid confusion, access to the 32-bit slots is by request only; If you would like access to the 32 bit systems, please contact GS-ITS.

Connecting

First you must connect to the cluster's head node, sage.gs.washington.edu, using an ssh client. You will need a GSDS account and password to connect to sage. If you have trouble logging in, please check your account on our Self-Service page or contact GS-ITS.

Creating a Default Settings File

Default settings for jobs are stored in a .sge_request file in your home directory. Most options which can be provided on the command line when submitting a job, can be specified in this file. Below you will find a sample file with descriptions of helpful options. You can copy and paste the sample into a new .sge_request file in your home directory and adjust it to your liking or just use it as-is.

# Sample .sge_request file # # Set e-mail address -M REPLACEME@u.washington.edu # Uncomment to send e-mail to the above address when the job ends # If you use qlogin, it will also notify you # when those jobs are done, so you may want to put # this option in your job scripts instead. #-m e # Put job standard output into the sgeoutput directory in # your home directory. The filename will be named # [jobname].o[jobid] (ex. testjob2.sh.o10979). # If this option is not specified, the output file # will be created in your home directory. -o $HOME/sgeoutput # Put job standard error into the sgeoutput directory in # your home directory. The filename will be named # [jobname].e[jobid] (ex. testjob2.sh.e10979). # If this option is not specified, the output file # will be created in your home directory. -e $HOME/sgeoutput # Uncomment to direct standard error into the standard output file #-j y

If you elected to use the option to redirect your output and error files to a directory, create a that directory now.

[sjf4@sage ~]$ mkdir sgeoutput

Creating a Job Script

Job scripts setup the environment in which your application runs as well as setting job submission options. Below you will find an example job script. It will give you an idea about how to set up a job script for your application. You can copy and paste the below script into a new file called testjob2.sh and try it. In the fully commented (#) section at the top, there are some job submission options. This section is followed by some bash scripting and comments which illustrate some of the points covered later in this document. Please read through the sample job script below.

# Sample batch job script (testjob2.sh) # # Below you will find options used by SGE to affect how # your job is queued and run. # # Use the bash shell to interpret this job script #$ -S /bin/bash # # Remove one # to send an e-mail to the address # specified in .sge_request when this job ends. ##$ -m e # # Remove one # to only submit this job to nodes # that have at least 8GB of RAM free. ##$ -l mem_free=8.0G ## Put the hostname, current directory, and start date ## into variables, then write them to standard output. GSITSHOST=`/bin/hostname` GSITSPWD=`/bin/pwd` GSITSDATE=`/bin/date` echo "**** JOB STARTED ON $GSITSHOST AT $GSITSDATE" echo "**** JOB RUNNING IN $GSITSPWD" ## ## SGE jobs do not run in your login environment, so let's ## sort the variables and take a look at what is set. echo "######### START DUMP OF ENVIRONMENT VARIABLES #########" /usr/bin/printenv | sort echo "######### END DUMP OF ENVIRONMENT VARIABLES #########" ## ## Produce some standard error output. ls does not have a -j option, ## so it will produce an error. /bin/ls -j ## Write a message to standard output including the return ## code from ls. Programs typically exit 0 if there was ## no error. echo "ls exited with return code: $?" ## ## Sleep for 30s so we have time to look at qstat /bin/sleep 30 ## ## SGE creates a temporary directory on the local disk ## of the node processing your job. If your job makes heavy ## use of on disk data, you may want to consider using this ## space. Be sure to copy anything you'd like out the temp ## dir before your job ends, since the directory is deleted then. ## ## The bash code to copy the data is currently commented ## out. Remove the #s if you'd like to see how it works. /bin/ls > $TMPDIR/output #GSITSRESULT=$HOME/$JOB_NAME.$JOB_ID.result #echo "Saving data to $GSITSRESULT" #/bin/cp $TMPDIR/output $GSITSRESULT ## ## Put the current date into a variable and report it before we exit. GSITSENDDATE=`/bin/date` echo "**** JOB DONE, EXITING 0 AT $GSITSENDDATE" ## ## Exit with return code 0, otherwise the job would exit with the ## return code of the failed ls command. exit 0 ##

Submitting Your Batch Job

Once you've created your job script, submitting your job is easy. You can specify any of the options from the .sge_request file or the job script on the command line.

[sjf4@sage ~]$ qsub testjob2.sh Your job 10984 ("testjob2.sh") has been submitted

In the example below, the submitted job will only be assigned to nodes which have at least 8GB of RAM free.

[sjf4@sage ~]$ qsub -l mem_free=8.0G testjob2.sh Your job 10985 ("testjob2.sh") has been submitted

Checking Your Job's Status

Below is the status of a job immediately following submission. The job has not been queued yet since the priority ("prior" column) shows 0.00000. The "state" column shows qw which indicates the job is waiting in the queue. The -u option causes only jobs to be shown which have been submitted by the specified user.

[sjf4@sage ~]$ qstat -u sjf4 job-ID prior name user state submit/start at queue ------------------------------------------------------------------------ 10984 0.00000 testjob2.s sjf4 qw 07/23/2008 12:10:28

If there are available processing slots, the job should begin running within ten to fifteen seconds. The "state" column now shows r to indicate the job is running.

[sjf4@sage ~]$ qstat -u sjf4 job-ID prior name user state submit/start at queue ------------------------------------------------------------------------ 10984 10.55000 testjob2.s sjf4 r 07/23/2008 12:10:32 all.q@sage012

If your job does not start running after some time, you can view pending jobs in the queue to see where you are in line. The -s option causes only jobs of a certain state to be shown. In this example, we're using p for pending.

[sjf4@sage ~]$ qstat -s p job-ID prior name user state submit/start at queue ------------------------------------------------------------------------- 11163 10.55000 sleep.sh asafir qw 07/23/2008 14:20:13 11129 0.55004 sleep.sh sjf4 qw 07/23/2008 14:17:15 11130 0.55004 sleep.sh sjf4 qw 07/23/2008 14:17:15 11131 0.55004 sleep.sh sjf4 qw 07/23/2008 14:17:15

Since asafir has used less CPU time on the cluster than sjf4, he gets a higher priority. Priority is indicated by the "prior" column. Once a slot is freed up by another user's job ending, asafir's will be the next to run. Regardless of priority, a queued job cannot start until a running job has ended. If you would like more details on how priority is determined, check the Compute Cluster FAQ.

Checking Your Job's Output

If you created a directory to store the standard output and standard error for your jobs, cd to it. Unless you enabled combined standard output and standard error, two files will be produced, one for each type.

[sjf4@sage ~]$ cd sgeoutput [sjf4@sage sgeoutput]$ ls testjob2.sh.e10984 testjob2.sh.o10984

These files are updated as your job runs, so if your application ouputs information about its progress, you can view it in one of these two files.

Below you can find the trimmed standard output of our sample job. You should look at the output produced by the job you submitted so you can see your full list of environment variables. Jobs do not use the same environment as your login shell.

[sjf4@sage sgeoutput]$ cat testjob2.sh.o10984 **** JOB STARTED ON sage012.grid.gs.washington.edu AT Wed Jul 23 12:10:32 PDT 2008 **** JOB RUNNING IN /net/gs/vol1/home/sjf4 ######### START DUMP OF ENVIRONMENT VARIABLES ######### ARC=lx24-amd64 CONSOLE=/dev/console ENVIRONMENT=BATCH HOME=/net/gs/vol1/home/sjf4 ... OUTPUT TRIMMED ... TMPDIR=/tmp/10984.1.all.q TMP=/tmp/10984.1.all.q USER=sjf4 _=/usr/bin/printenv ######### END DUMP OF ENVIRONMENT VARIABLES ######### ls exited with return code: 2 **** JOB DONE, EXITING 0 AT Wed Jul 23 12:11:02 PDT 2008

Below you can view the standard error from our sample job. Since ls was the only command we ran that produced an error, only output from ls is in this file. Depending on your application though, warnings or informational messages may also appear in the standard error file.

[sjf4@sage sgeoutput]$ cat testjob2.sh.e10984 /bin/ls: invalid option -- j Try `/bin/ls --help' for more information.

Starting an Interactive Session

Some users prefer to run their jobs in interactive sessions. If your application can only be run through a GUI interface, you have a small number of jobs to run or you do not use the cluster often, it may be the best choice for you. Keep in mind though, as long as you have an interactive session open, another person cannot use that processing slot.

[sjf4@sage ~]$ qlogin Your job 11052 ("QLOGIN") has been submitted waiting for interactive job to be scheduled ... Your interactive job 11052 has been successfully scheduled. Establishing /net/gs/vol3/software/sge/util/qlogin_command session to host sage012 ...

Get More Information

Sun Grid Engine User Guide
The official user documentation provided by Sun. This book includes terminology and common tasks for SGE users. The book is available as both HTML and PDF.

Sun Grid Engine Manual Pages
Guides to SGE commands and configuration files. These documents are also available on sage using the man command, i.e. man qsub.

Sun Grid Engine Product Page
The official product page for Grid Engine. Here you can find links to additional information and documentation.