All production computing must be done on TRACE's compute nodes, NOT on TRACE's login nodes. The SLURM scheduler (Simple Linux Utility for Resource Management) manages and allocates all of TRACE's compute nodes. Several partitions, or job queues, have been set up in SLURM to allocate resources efficiently.
To run a job on TRACE you will decide which partition you want to use. Each project has a partition named for their project. There is also a general queue named "batch". Jobs sumitted to the "batch" queue will be cancelled if cores are needed for priority jobs.

Table of Contents

Batch jobs

Instead of working interactively on TRACE, you can instead run in batch. This means you will

create a file called a batch or job script
submit that script to a partition (queue) using the sbatch command
wait for the job's turn in the queue
if you like, check on the job's progress as it waits in the partition and as it is running
check the output file for results or any errors when it finishes

A simple example

This section outlines an example which submits a simple batch job. More detail on batch scripts, the sbatch command and its options follow.

Create a batch script

Use any editor you like to create your batch scripts. A simple batch script named hello.job which runs a "hello world" command is given here. Comments, which begin with '#', explain what each line does.
The first line of any batch script must indicate the shell to use for your batch job.
#!/bin/bash
# use the bash shell
set -x
# echo each command to standard out before running it
date
# run the Unix 'date' command
echo "Hello world, from TRACE-HPC!"
# run the Unix 'echo' command

Submit the batch script

Use the sbatch command to submit the hello.job script.
[joeuser@tracevm05 ~]$ sbatch hello.job
Submitted batch job 7408623
Note the jobid that is echoed back to you when the job is submitted. Here it is 7408623.

Check on the job progress

You can check on the job's progress in the partition by using the squeue command. By default you will get a list of all running and queued jobs. Use the -u option with your username to see only your jobs.
[joeuser@tracevm05 ~]$ squeue -u joeuser
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
7408623 RM hello.jo joeuser PD 0:08 1 r7320:00
The status "PD" (pending) in the output here shows that job 7408623 is waiting in the queue.
When the job is done, squeue will no longer show it:
[joeuser@tracevm05 ~]$ squeue -u joeuser
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

Check the output file when the job is done

By default, the standard output and error from a job are saved in a file with the name slurm-jobid.out, in the directory that the job was submitted from.
[joeuser@tracevm05 ~]$ more slurm-7408623.out
+ date
Sun Jan 19 10:27:06 EST 2020
+ echo 'Hello world, from TRACE-HPC!'
Hello world, from TRACE-HPC!
[joeuser@tracevm05 ~]$

Request a GPU for your job

If you are using a gpu in your job, add the -gpus option to your sbatch command.

[joeuser@tracevm05 ~]$ sbatch -gpus=1 hello.job

The sbatch command

To submit a batch job, use the sbatch command. The format is
sbatch -options batch-script
The options to sbatch can either be in your batch script or on the sbatch command line. Options in the command line override those in the batch script.
Note:.

In some cases, the options for sbatch differ from the options for interact or srun.
By default, sbatch submits jobs to your group's priority partition. Use the -p option forsbatch to direct your job to a different partition

Options to the sbatch command

For more information about these options and other useful sbatch options see the sbatch man page.

Option	Description	Default
-p partition	Partition requested	batch
-t HH:MM:SS	Walltime requested in HH:MM:SS	30 minutes
-N n	Number of nodes requested.	1
-n n	Number of cores requested in total.	None
--ntasks-per-node=n Note the "--" for this option	Request n cores be allocated per node.	1
-o filename	Save standard out and error in filename. This file will be written to the directory that the job was submitted from.	slurm-jobid.out
--gpus=type:n Note the "--" for this option	Specifies the number of GPUs requested. 'type' specifies the type of GPU you are requesting. See the GPU partitions section of this User Guide for information on the GPU types. 'n' is the total number of GPUs requested for this job.	N/A
-A account id	SLURM account id for the job. If not specified, your default account id is used. Find your default SLURM account id. Note: Files created during a job will be owned by the Unix group in effect when the job is submitted. This may be different than the account id used by the job. See the discussion of the newgrp command in the Account Administration section of this User Guide to see how to change the Unix group currently in effect.	Your default account id
-C constraints	Specifies constraints which the nodes allocated to this job must satisfy. Valid constraints are: PERF Turns on performance profiling. For use with performance profiling software.Note that the PERF option cannot be used in partitions in which user jobs can share a node, e.g., RM-shared.	N/A
--res reservation-name Note the "--" for this option	Use the reservation that has been set up for you. Use of --res does not automatically set any other options. You still need to specify the other options (partition, walltime, number of nodes) that you would in any sbatch command. If your reservation is not assigned to your default account then you will need to use the -A option to sbatch to specify the account.	N/A
--mail-type=type Note the "--" for this option	Send email when job events occur, where type can be BEGIN, END, FAIL or ALL.	N/A
--mail-user=username Note the "--" for this option	User to send email to as specified by -mail-type. Default is the user who submits the job.	N/A
-d=dependency-list	Set up dependencies between jobs, where dependency-list can be: after:job_id[:jobid...] This job can begin execution after the specified jobs have begun execution. afterany:job_id[:jobid...] This job can begin execution after the specified jobs have terminated. aftercorr:job_id[:jobid...] A task of this job array can begin execution after the corresponding task ID in the specified job has completed successfully (ran to completion with an exit code of zero). afternotok:job_id[:jobid...] This job can begin execution after the specified jobs have terminated in some failed state (non-zero exit code, node failure, timed out, etc). afterok:job_id[:jobid...] This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero). singleton This job can begin execution after any previously launched jobs sharing the same job name and user have terminated.	N/A
--no-requeue Note the "--" for this option	Specifies that your job will be not be requeued under any circumstances. If your job is running on a node that fails it will not be restarted. Note the "--" for this option.	N/A
--time-min=HH:MM:SS Note the "--" for this option.	Specifies a minimum walltime for your job in HH:MM:SS format. SLURM considers the walltime requested when deciding which job to start next. Free slots on the machine are defined by the number of nodes and how long those nodes are free until they will be needed by another job. By specifying a minimum walltime you allow the scheduler to reduce your walltime request to your specified minimum time when deciding whether to schedule your job. This could allow your job to start sooner. If you use this option your actual walltime assignment can vary between your minimum time and the time you specified with the -t option. If your job hits its actual walltime limit, it will be killed. When you use this option you should checkpoint your job frequently to save the results obtained to that point.	N/A
-h	Help, lists all the available command options

Requeuing Failed Jobs

If a job fails for any reason, the default scheduling policy for the cluster is to not reschedule the job. You will need to re-issue the sbatch command yourself. If you would like to have your job automatically put back in the queue after a failure, you can use the requeue option on the sbatch command to override the default. For example:

sbatch -p batch --requeue hello.job

will automatically put your hello.job job back in the queue. It will run when it is its turn.

Note that this will restart your job from the beginning. No state is maintained about the progress of your job. You should be cautious using automatic requeueing. Your batch script, including any executables, should be written to be aware that it may be requeued.

Reservations

A TRACE-HPC reservation dedicates nodes for your exclusive use for a specified time.
Things to keep in mind when submitting a reservation request:

State clearly the reason that you need the reservation. Requesting a reservation is not a guarantee that you will receive one.
Submit your request early, at least a week in advance. It is not always possible to drain the nodes for a reservation on short notice. The nodes may already be busy with jobs that will run for several days.

If you would like to request a reservation, please send an email to trace-it-help@andrew.cmu.edu.

If your request is approved, you wil get email from TRACE User Support with the details of your reservation, including the name of the reservation.
You can see and manage your reservation with these commands:
scontrol show reservation=reservation-name
Displays the details for the named reservation.
scontrol delete reservation=reservation-name
Ends the reservation.
[user@trace-hpc-login012 ~]$ scontrol show res=myreservation
ReservationName=myreservation StartTime=2021-08-17T09:10:53 EndTime=2021-08-17T10:10:53 Duration=01:00:00
Nodes=v007 NodeCnt=1 CoreCnt=40 Features=(null) PartitionName=GPU-shared Flags=
TRES=cpu=40
Users=(null) Groups=(null) Accounts=tracestaff Licenses=(null) State=ACTIVE BurstBuffer=(null) Watts=n/a
MaxStartDelay=(null)
[user@trace-hpc-login012 ~]$ scontrol delete res=myreservation
[user@trace-hpc-login012 ~]$ scontrol show res=myreservation
Reservation myreservation not found
[user@trace-hpc-login012 ~]$

Job status, node, and partition information

sinfo

The sinfo command displays information about the state of TRACE-HPC's nodes. The nodes can have several states:

alloc	Allocated to a job
down	Down
drain	Not available for scheduling
idle	Free
resv	Reserved

squeue

The squeue command displays information about the jobs in the partitions. Some useful options are:

-j jobid	Displays the information for the specified jobid
-u username	Restricts information to jobs belonging to the specified username
-p partition	Restricts information to the specified partition
-l	(long) Displays information including: time requested, time used, number of requested nodes, the nodes on which a job is running, job state and the reason why a job is waiting to run.

scancel

The scancel command is used to kill a job in a partition, whether it is running or still waiting to run. Specify the jobid for the job you want to kill. For example,
scancel 12345
kills job # 12345.

sacct

The sacct command can be used to display detailed information about jobs. It is especially useful in investigating why one of your jobs failed. The general format of the command is:
sacct -X -j nnnnnn -S MMDDYY --format parameter1,parameter2, ...

For 'nnnnnn' substitute the jobid of the job you are investigating.
The date given for the -S option is the date at which sacct begins searching for information about your job.
The commas between the parameters in the --format option cannot be followed by spaces.

The --format option determines what information to display about a job. Useful parameters are

JobID
Partition
Account - the account id
ExitCode - useful in determining why a job failed
State - useful in determining why a job failed
Start, End, Elapsed - start, end and elapsed time of the job
NodeList - list of nodes used in the job
NNodes - how many nodes the job was allocated
MaxRSS - how much memory the job used
AllocCPUs - how many cores the job was allocated

Monitoring memory usage

It can be useful to find the memory usage of your jobs. For example, you may want to find out if memory usage was a reason a job failed.
You can determine a job's memory usage whether it is still running or has finished. To determine if your job is still running, use the squeue command.
squeue -j nnnnnn -O state
where nnnnnn is the jobid.

For running jobs: srun and top or sstat

You can use the srun and top commands to determine the amount of memory being used.
srun --jobid=nnnnnn top -b -n 1 | grep userid
For nnnnnn substitute the jobid of your job. For 'userid' substitute your userid. The RES field in the output from top shows the actual amount of memory used by a process. The top man page can be used to identify the fields in the output of the top command.

You can also use the sstat command to determine the amount of memory being used in a running job
sstat -j nnnnnn.batch --format=JobID,MaxRss
where nnnnnn is your jobid.

For jobs that are finished: sacct or job_info

If you are checking within a day or two after your job has finished you can issue the command
sacct -j nnnnnn --format=JobID,MaxRss
If this command no longer shows a value for MaxRss, use the job_info command
job_info nnnnnn | grep max_rss
Substitute your jobid for nnnnnn in both of these commands.

More information

Online documentation for SLURM, including man pages for all the SLURM commands

Browser not supported