Slurm Usage

To use the new HPC Cluster

The new HPC cluster is available via a GUI to the SLURM interface, which can be found here:

https://hpc.grit.ucsb.edu/

Intro

SLURM (Simple Linux Utility for Resource Management) is a widely used open-source job scheduler that we use on GRIT HPC systems to allocate resources efficiently. This guide provides some basic information on how to use Slurm and create scripts for submitting jobs to the Slurm queue.

Typical Workflow

Develop your program (e.g. on your computer and a subset of data)
Update your program for use on HPC (e.g. change data paths if needed, etc.)
Create a slurm job file (see below)
Submit your job to the queue
Monitor the job status, wait for completion

Steps 3-5 are detailed below.

Limitations

The following limits are currently in place to prevent one user or group from saturating the cluster forever. These values are subject to change as the cluster grows or as needed.

Maximum running jobs per user: 150
Maximum submitted jobs per user: 300 (This includes both running and pending jobs.)
Maximum CPU per user: 300 cores (~25% of the cluster)
Maximum RAM per user: 1.2TB (~25% of the cluster)
Maximum job-array size: 1,001 tasks
Maximum jobs cluster-wide: 10,000
Maximum steps per job: 40,000
Maximum wall time: 30 days

Example Slurm job files

Slurm job files are writting in bash, which is a linux shell scripting language. Here's an example which uses one cpu on one computer to run a simple job, outputting any errors or other outputs to log files in the same directory. Note that on most GRIT HPC systems the main queue (aka partition in Slurm) is called 'grit_nodes'.

Some servers may be using "basic" in place of "grit_nodes"

#!/bin/bash
## SLURM REQUIRED SETTINGS  <--- two hashtags are a comment in Slurm
#SBATCH --partition=grit_nodes   
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

## Optional: Request GPU
#SBATCH --gres=gpu:1

## Specify RAM requested with units
#SBATCH --mem=4G 

## SLURM reads %x as the job name and %j as the job ID
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

# Job to run
./my_example_code.bash

Another Example:

#!/bin/bash
# 
#SBATCH -p grit_nodes # partition name (aka queue)
#SBATCH -c 1 # number of cores
#SBATCH --mem 100 # memory pool for all cores
#SBATCH -t 0-2:00 # time (D-HH:MM)
#SBATCH -o slurm.%N.%j.out # STDOUT
#SBATCH -e slurm.%N.%j.err # STDERR

# code or script to run 
for i in {1..100000}; do
echo $RANDOM >> SomeRandomNumbers.txt
donesort SomeRandomNumbers.txt

Python Example with Conda

The output goes to a file in your home directory called hello-python-*.out, which should contain a message from python.

#!/bin/bash

## SLURM REQUIRED SETTINGS1G
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1 

## SLURM reads %x as the job name and %j as the job ID
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

#SBATCH --job-name=hello-python     # create a short name for your job
#SBATCH --time=00:01:00          # total run time limit (HH:MM:SS)

## Example use of Conda:

# first source bashrc (with conda.sh), then conda can be used
source ~/.bashrc

# make sure conda base is activated
conda activate

# Other conda commands go here


## run python 
python hello.py

hello.py should be something like this:

print('Hello from python!')

Adding details to Slurm job files

These examples are all very simple, so here are some useful commands for adding more complexity, such as more memory, more CPU's etc. Adding these requires finding out facts about the computer for the job file:

Find the number of CPU cores on a computer from the command line:

[user@computer ~]$ grep 'cpu cores' /proc/cpuinfo | uniq
cpu cores	: 48                   <---- an example output

Find out how much memory a computer has:

[user@computer ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          1.5Ti       780Gi       721Gi       1.5Gi       8.6Gi       721Gi
Swap:          31Gi          0B        31Gi

For most of our use cases, one node and one task is all that is needed. More than this requires special code such as mpi4py (MPI = Message Passing Interface), or the Parallel computing toolbox such as with MATLAB (which uses --cpus-per-task). To request N cores for a job, just replace N with the number of cores you need in the Slurm job file, such as:

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=N

To get the max value for N for a computer:

[user@computer ~]$ scontrol show node | grep CPU
CPUAlloc=20 CPUTot=95 CPULoad=1.00

'CPUTot' is the max value for N.

Find the queue names:

[user@computer ~]$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
basic*       up   infinite      1   idle   <--- in this case the queue name is 'grit_nodes' and it's the default
                                                 as indicated by the *

Submitting your job to the queue

Assuming you have a Slurm job file named slurm_test.sh:

#  test a job submission (don't run)
[user@computer ~]$ sbatch --test-only slurm_test.sh

#  run a job
[user@computer ~]$ sbatch slurm_test.sh

Using the `preemptable_nodes` Slurm Queue

The preemptable_nodes queue allows jobs to use otherwise-idle resources throughout the GRIT HPC cluster. In exchange for broader access to available capacity, a job running on a non-GRIT node may be stopped and returned to the queue when that node's primary users need it.

How the queue works

Submit jobs to this partition with:

sbatch --partition=preemptable_nodes job.sh

The partition behaves as follows:

Where the job is running	Preemption behavior
GRIT-owned node	The job is not preempted by jobs in the normal `grit_nodes` partition.
A node belonging to another group	A job submitted to that node's primary partition can preempt it.
A node excluded by the administrators	The node is not available through `preemptable_nodes`.

If any part of a multi-node job is preempted, the whole job is requeued. Slurm may restart it on different nodes when sufficient resources become available.

When Slurm requeues a job:

The job keeps the same Slurm job ID.
The batch script starts again from its first line.
Program memory and unsaved work are not preserved.
The job returns to PENDING until resources are available again.
SLURM_RESTART_COUNT reports how many times the job has restarted.

The queue is best suited to workloads that can restart safely, checkpoint their progress, or be divided into small independent tasks.

Basic restart-from-the-beginning job

This is sufficient for a short job that can safely restart from the beginning:

#!/bin/bash
#SBATCH --job-name=example-preemptable
#SBATCH --partition=preemptable_nodes
#SBATCH --time=04:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --requeue
#SBATCH --open-mode=append
#SBATCH --output=slurm-%j.out

set -euo pipefail

echo "Job ID: $SLURM_JOB_ID"
echo "Restart count: ${SLURM_RESTART_COUNT:-0}"
echo "Running on: $SLURM_JOB_NODELIST"
echo "Started: $(date -Is)"

srun ./my_program --input input.dat --output results.dat

echo "Finished: $(date -Is)"

Submit it with:

sbatch job.sh

#SBATCH --requeue explicitly marks the job as eligible for requeue. #SBATCH --open-mode=append prevents a restarted job from truncating its existing Slurm output log.

The program must be safe to rerun. If it writes directly to results.dat, consider writing to a temporary file and renaming it only after successful completion:

srun ./my_program --input input.dat --output results.dat.tmp
mv results.dat.tmp results.dat

The mv operation prevents an interrupted run from leaving a partial file under the final output name.

Checkpoint-aware job

Long jobs should periodically save enough state to continue after a restart. Store checkpoints on a shared filesystem such as your home directory or HPC scratch space. Do not store the only copy in /tmp or another node-local directory because the restarted job may run on a different node.

The following example processes numbered work units and records the next unit only after the current one finishes successfully:

#!/bin/bash
#SBATCH --job-name=checkpoint-example
#SBATCH --partition=preemptable_nodes
#SBATCH --time=1-00:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --requeue
#SBATCH --open-mode=append
#SBATCH --output=slurm-%j.out

set -euo pipefail

checkpoint_dir="$HOME/slurm-checkpoints/$SLURM_JOB_ID"
state_file="$checkpoint_dir/next-item"
output_dir="$checkpoint_dir/results"
last_item=100

mkdir -p "$checkpoint_dir" "$output_dir"

if [[ -s "$state_file" ]]; then
    read -r next_item < "$state_file"
else
    next_item=1
fi

echo "Job ID: $SLURM_JOB_ID"
echo "Restart count: ${SLURM_RESTART_COUNT:-0}"
echo "Resuming at item: $next_item"

for item in $(seq "$next_item" "$last_item"); do
    final_output="$output_dir/item-${item}.dat"
    temporary_output="$output_dir/.item-${item}.tmp"

    # An interrupted task leaves only a temporary file. On restart, the same
    # item is safely attempted again.
    srun python3 process_item.py \
        --item "$item" \
        --output "$temporary_output"

    mv "$temporary_output" "$final_output"

    # Atomically record the next item after the output is complete.
    temporary_state="$state_file.tmp"
    printf '%s\n' "$((item + 1))" > "$temporary_state"
    mv "$temporary_state" "$state_file"
done

touch "$checkpoint_dir/complete"
echo "All work completed: $(date -Is)"

This pattern provides three useful guarantees:

Completed items are not repeated.
An interrupted item is attempted again after the job restarts.
A partial result is never mistaken for a completed result.

Applications with built-in checkpoint support

Many simulation, machine-learning, and analysis programs can periodically write a checkpoint. Configure the program to save frequently enough that losing the work since the last checkpoint is acceptable.

A typical wrapper looks like this:

#!/bin/bash
#SBATCH --job-name=training
#SBATCH --partition=preemptable_nodes
#SBATCH --time=2-00:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --gres=gpu:1
#SBATCH --requeue
#SBATCH --open-mode=append
#SBATCH --output=slurm-%j.out

set -euo pipefail

checkpoint="$HOME/checkpoints/my-training-job.ckpt"
arguments=(--checkpoint "$checkpoint" --checkpoint-every 300)

if [[ -f "$checkpoint" ]]; then
    echo "Restarting from $checkpoint"
    arguments+=(--resume)
else
    echo "Starting a new run"
fi

echo "Slurm restart count: ${SLURM_RESTART_COUNT:-0}"
srun python3 train.py "${arguments[@]}"

The exact checkpoint and resume options depend on the application. Slurm re-runs the batch script; it does not automatically create an application checkpoint.

Use job arrays when work can be divided

If a workload consists of many independent inputs, a job array often provides the simplest recovery model. Each array element handles one input, so only interrupted elements need to run again.

#!/bin/bash
#SBATCH --job-name=preemptable-array
#SBATCH --partition=preemptable_nodes
#SBATCH --array=1-1000%25
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --requeue
#SBATCH --open-mode=append
#SBATCH --output=slurm-%A_%a.out

set -euo pipefail

mkdir -p results

final_output="results/${SLURM_ARRAY_TASK_ID}.dat"
temporary_output="results/.${SLURM_ARRAY_TASK_ID}.tmp"

srun python3 process_item.py \
    --item "$SLURM_ARRAY_TASK_ID" \
    --output "$temporary_output"

mv "$temporary_output" "$final_output"

In --array=1-1000%25, the %25 limits the array to 25 simultaneously running tasks.

Arrayed or parallel jobs can also be run across multiple nodes. Jobs can be divided using the following options:

#SBATCH --nodes=4
#SBATCH --ntasks=32
#SBATCH --ntasks-per-node=8

In Slurm, an array controls how many copies of a job are submitted, while nodes, tasks, and CPUs per task control the resources allocated to each copy. The basic calculation for each running array element is:

CPUs per array element = nodes × tasks per node × CPUs per task

Slurm option	Meaning	Example
`--array=0-279%140`	Submit 280 copies and run no more than 140 simultaneously	Tasks `0–279`, with up to 140 running
`--nodes=10`	Allocate 10 physical compute nodes to each array element	Each running element occupies 10 nodes
`--ntasks-per-node=4`	Allocate four processes/tasks on each node	10 nodes × 4 tasks = 40 tasks
`--ntasks=1`	Allocate one process/task in total	Appropriate for one ordinary program
`--cpus-per-task=2`	Allocate two CPU cores to each process/task	Appropriate for one two-threaded process
`--mem=1G`	Allocate 1 GB of memory per node	With 10 nodes, this requests 10 GB total

For example:

Workload	Nodes	Tasks per node	CPUs per task	CPUs per array element	Maximum at `%140`
Original configuration	10	4	2	80	11,200 CPUs
One single-threaded model	1	1	1	1	140 CPUs
One two-threaded model	1	1	2	2	280 CPUs
MPI job with 32 processes	4	8	1	32	4,480 CPUs

Thus, --nodes=10, --ntasks-per-node=4, and --cpus-per-task=2 allocate 10 × 4 × 2 = 80 CPUs to every running array element. For a program that runs one two-threaded process per element, the appropriate request is --nodes=1, --ntasks=1, and --cpus-per-task=2. This allocates two CPUs per element and at most 280 CPUs across 140 concurrent elements. A distributed MPI application might instead use --nodes=4, --ntasks-per-node=8, and --cpus-per-task=1, allocating 32 processes across four nodes. Allocating tasks or nodes does not automatically parallelize a program: the script must use srun, MPI, or another distributed launcher to start work on those resources, while --cpus-per-task supplies the cores used by each individual multithreaded process.

Please note that ntasks-per-node must be used with either --nodes=# or --ntasks=#, otherwise the default value of 1 node will be selected.

#!/bin/bash
#SBATCH --job-name=multinode-test
#SBATCH --partition=grit_nodes
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --time=01:00:00
#SBATCH --mem=32G

srun ./my_parallel_program

Monitoring a job

Show your running and pending jobs:

squeue --me

Show detailed information about one job:

scontrol show job JOB_ID

Review accounting information:

sacct -j JOB_ID \
  --format=JobID,JobName,Partition,State,Elapsed,Start,End,NodeList,ExitCode

The job's log also reports ${SLURM_RESTART_COUNT:-0} when using the examples above.

Recommendations

Use #SBATCH --requeue even though the cluster permits requeue by default. It documents the intended behavior in the job script.
Use #SBATCH --open-mode=append so restarts do not erase earlier log output.
Save checkpoints on shared storage, not node-local temporary storage.
Checkpoint periodically; do not assume the application will receive enough warning to save state at preemption time.
Make output operations repeatable and safe. Write incomplete results to temporary names and atomically rename them when complete.
Keep individual work units reasonably short to reduce how much work can be lost.
Use job arrays for large collections of independent tasks.
Do not use #SBATCH --no-requeue in this partition. A job that cannot be requeued may be cancelled when preempted.

Quick template

#!/bin/bash
#SBATCH --job-name=replace-me
#SBATCH --partition=preemptable_nodes
#SBATCH --time=04:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --requeue
#SBATCH --open-mode=append
#SBATCH --output=slurm-%j.out

set -euo pipefail

echo "Job $SLURM_JOB_ID, restart ${SLURM_RESTART_COUNT:-0}"
srun YOUR_COMMAND_HERE

Scheduling a SLURM job

SLURM offers a crontab like utility called scrontab. To schedule your jobs in the slurm queue run:

scrontab -e

This will open an editor to your user's scrontab and allow you to schedule jobs:

# Welcome to scrontab, Slurm's cron-like interface.
#
# Edit this file to submit recurring jobs to be run by Slurm.
#
# Note that jobs will be run based on the Slurm controller's
# time and timezone.
#
# Lines must either be valid entries, comments (start with '#'),
# or blank.
#
# Lines starting with #SCRON will be parsed for options to use
# with the next cron line. E.g., "#SCRON --time 1" would request
# a one minute timelimit be applied. See the sbatch man page for
# options, although note that not all options are supported here.
#
# For example, the following line (when uncommented) would request
# a job be run at 5am each day.
# 0 5 * * * /my/script/to/run
#
# min hour day-of-month month day-of-week command
0 5 * * * /my/script/to/run

the other available flags for scrontab are:

-e
Edit the crontab. If a crontab does not exist already, a default example 
(without any defined entries) will be provided in the editor.

-l
List the crontab. (Prints directly to stdout.)

-r
Remove the crontab. Any currently running crontab-defined jobs will 
continue to run but will no longer recur. All other crontab-defined jobs will be cancelled.

Note that jobs scheduled via scrontab will be placed into the queue at the scheduled time. That means if there is available resources at the scheduled time it will run immediately, if resources are not available it will remain in the queue until the resources become available.

Monitoring your job

Email from Slurm when job is done:

Users can control the sending of email by adding the following to your job script:

#!/usr/bin/bash

#SBATCH --job-name=my_job                # Specify a job name
#SBATCH --mail-type=END                  # Send email when job ends
#SBATCH --mail-user=user@ucsb.edu      # Replace with your email address

You can test emailing capability by running this at the command line:

sbatch --wrap "sleep 30" --mail-type=ALL --mail-user=me@ucsb.edu

Note: If your user on striker matches your UCSB netid, you do not have to set the --mail-user option, it will send to the correct email address.

Examples of of other ways to monitor:

[user@computer ~]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            166626 grit_nodes my_code  username PD       0:00     1 (Resources)
            166627 grit_nodes my_code  username  R       3:04     1 anvil

In the above, 'R' denotes that the job is running, 'PD' denotes that Slurm is waiting for resources.

You can also monitor the output by watching the log files from the command line. This will show the last few lines of the log file and update as the log file changes:

[user@computer ~]$ tail -f log-file-name.txt

Cancel the job if needed:

[user@computer ~]$ scancel 22           # cancel job 22

You can get the job number from squeue (e.g. JOBID).

Useful Commands

sinfo                # general info about slurm
sinfo -lNe           # more detailed info reporting with long format and nodes listed individually
scontrol show job 2  # show control info on job 2

To find the number of cores per socket:

lscpu | grep "Core(s) per socket" | awk '{print $4}'

More Example Job scripts

An example with R

##!/bin/bash -l
## How long should I job run for
#SBATCH --time=01:00:00
## Number of CPU cores, in this case 1 core
#SBATCH --ntasks=1
## Number of compute nodes to use (always 1 on GRIT systems)
#SBATCH --nodes=1
## Name of the output log files to be created. If not specified the outputs will be joined
#SBATCH --output=%x.%j.out
#SBATCH --error=%x.%j.err

# The code you want to run your job
Rscript test_forge_r.R

Here's what was used in the test script test_forge_r.R:

# A simple R script to print hello world!
aString = "Hello World!"

print (aString)

References

https://www.carc.usc.edu/user-information/user-guides/hpc-basics/slurm-templates

https://docs.rc.fas.harvard.edu/kb/convenient-slurm-commands/

https://csc.cnsi.ucsb.edu/docs/slurm-job-scheduler

Python: https://rcpedia.stanford.edu/topicGuides/jobArrayPythonExample.html

https://login.scg.stanford.edu/faqs/cores/

https://stackoverflow.com/questions/65603381/slurm-nodes-tasks-cores-and-cpus

Regarding nodes vs tasks vs cpus vs cores: Here's a very good writeup: https://researchcomputing.princeton.edu/support/knowledge-base/scaling-analysis.

Jupyter Notebook

R Studio

JupyterNotebook Custom kernel

IDL Aliases

Troubleshooting

CLI Usage

Linux Desktop

MATLAB Usage

VS Code Server

ENVI

FAQ

Scratch Space

Paritions

GPU Resources

Fortran

Queue Setup

Slurm Usage

To use the new HPC Cluster

Intro

Typical Workflow

Limitations

Example Slurm job files

Another Example:

Python Example with Conda

Adding details to Slurm job files

Submitting your job to the queue

Using the preemptable_nodes Slurm Queue

How the queue works

Basic restart-from-the-beginning job

Checkpoint-aware job

Applications with built-in checkpoint support

Use job arrays when work can be divided

Monitoring a job

Recommendations

Quick template

Scheduling a SLURM job

Monitoring your job

Email from Slurm when job is done:

Examples of of other ways to monitor:

Useful Commands

More Example Job scripts

References

Using the `preemptable_nodes` Slurm Queue