Working with the Cluster

Architecture

The cluster consists of login nodes (where you connect via SSH), a scheduler (SLURM), and compute nodes (where jobs run). Never run computations on the login node — use it only to prepare, submit, and monitor jobs.

Storage

Two locations matter for most workflows:

Location	Variable	Size limit	Use for
Home directory	`$HOME`	60 GiB (soft)	Config files, scripts, small installs
Project directory	`$PROJECT`	10 TiB (varies by project)	Simulation data, software installations, container images

Check your current usage with show-quota.

Software

There are three ways to get software on the cluster:

Module system — the default for commonly available software (compilers, standard tools). Load with module load <name>. Fast, no installation needed.

module spider gcc     # search for available packages
module load gcc       # load a package
module list           # show loaded packages

Spack — use for scientific software not in the module system, or when you need a specific version. R is loaded this way (see Section 3).

module load spack
source $SPACK_ROOT/share/spack/setup-env.sh
spack find            # list installed packages
spack load r@4.5.2    # load a package (here: R version 4.5.2)

Apptainer — use when your software requires a complete custom environment, for example a Docker image with complex dependencies (see Section 4).

SLURM: Managing Jobs

Submit a job

sbatch submit.sh

Check status of your jobs

squeue --me

Output from each job is written to slurm-<jobid>.out in the directory where you submitted.

Cancel a job

scancel <jobid>      # cancel a specific job
scancel -u $USER     # cancel all your jobs

Review finished jobs

sacct -u $USER --units=G --format=JobID,JobName,Partition,Elapsed,AllocCPU,ReqMem,MaxRSS,State

SLURM aliases (optional)

Add to ~/.bashrc on the HPC for shorter commands:

alias jobs_own='squeue -u $USER --format="%.12i %.12j %.8u %.3P %.3q %.10l %.10M %.5D %.4C %.7m %16R %.8T"'
alias jobs_run='squeue -u $USER --states=RUNNING --format="%.12i %.12j %.8u %.3P %.3q %.10l %.10M %.5D %.4C %.7m %16R %.8T"'
alias jobs_pen='squeue -u $USER --states=PENDING --format="%.12i %.12j %.8u %.3P %.3q %.10l %.10M %.5D %.4C %.7m %16R %.8T"'
alias jobs_kill='scancel -u $USER'
alias jobs_info='sacct -u $USER --units=G --format=JobID,JobName,Partition,Elapsed,AllocCPU,ReqMem,MaxRSS,State'
alias jobs_n='squeue -u $USER --states=RUNNING | wc -l'
alias fairshare='sshare -U $USER'

Apply with source ~/.bashrc.

Parallelization

There are two fundamentally different ways to parallelize work on a cluster:

Between nodes — submitting many independent jobs that run simultaneously on different nodes. This is what SLURM and rslurm do. Use this when your tasks do not depend on each other (e.g. running the same simulation model with different parameter combinations).

Within a node — using multiple CPU cores on the same node for a single task. Use this when the task itself can be split into parallel parts (e.g. matrix operations or explicitly parallel algorithms).

For our group's typical workload — running many independent simulation runs with different parameters — between-node parallelism is almost always the right choice: request one CPU per job and let SLURM distribute jobs across available nodes.

If the jobs are not independent of each other and you need them to run on different nodes, you need to use MPI. Please talk to Sebastian about this.

How many jobs?

Submit one job per parameter combination. The scheduler has a limit on how many jobs can run simultaneously; check your current limit with:

sacctmgr -s show user $USER # max. 1000 parallel jobs at a time

If your parameter space exceeds the running job limit, rslurm handles the queuing automatically.

Caution: programs that use more CPUs than requested

Many programs and libraries use parallelization internally and will attempt to use all available CPU cores on a node, even if you only requested one core from SLURM. This includes R packages such as data.table, compiled code, and math libraries.

If your job uses more cores than requested, it competes with other users' jobs on the same node. Always limit parallelization explicitly for software you use. In your job script or R code:

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

In R, the parallel package's mclapply also defaults to using multiple cores — set mc.cores = 1 unless you have explicitly requested multiple CPUs per node from SLURM.

Data Transfer

Use rsync to copy files between your local machine and the HPC. The SSH config entry from Section 1 (Emmy-p3) works directly as the target host.

Copy a local folder to the HPC:

rsync -av _rslurm_myjob/ Emmy-p3:~/

Copy results back from the HPC:

rsync -avP Emmy-p3:~/_rslurm_myjob/ ./results/

Use -P for large transfers — it shows progress and allows resuming interrupted transfers. scp -r also works for simpler cases.