GWDG HPC
Introduction
The aim of this guide is to explain how to use a local R session and submit jobs to the GWDG High Performance Cluster (HPC) and retrieve the results in the same R session (requires GWDG account). The big advantage of this is that working locally in an IDE (e.g. RStudio) is way more convenient than working with Linux shell on the HPC. Also, it is not necessary to manually copy data to the HPC (and vice versa). The structure of this guide is heavily influenced by our own mistakes and we hope to make it somewhat easier for future HPC user.
In general, our setup should also work with any other HPC and the code snippets should be usable for other scheduling systems than SLURM (which is used by the GWDG) with slight modifications. To see the outdated version of this guide for LSF, there is a GitHub branch called LSF in this repo.
General setup
The user account must be activated to use the HPC. Therefore, an e-mail to hpc@gwdg.de asking to activate the account must be sent.
1. Check Linux shell
The Linux shell must be set to /bin/ksh (and not /bin/sh). It’s possible to set this in the user settings at www.gwdg.de.
2. SSH Key
For a convenient login and to use R via SSH connector, a SSH key is needed on the HPC (private key on local computer, public key on HPC).
Using Linux or macOS, it is straightforward to generate a SSH key and copy it to the HPC using the shell/terminal.
If copy is not installed by default, it can be added using the following command (macOS).
For Windows, this guide seems to cover how to connect via SSH. Using Windows 10, it should be also possible to use the new OpenSSH client.
If the SSH key works, it should be possible to login to the HPC frontend without a password using the following command.
3. Create .profile
Because .bashrc is not source by Scientific Linux by default (source), firstly a .profile file is created. The following code enables that .bashrc is sourced every time the HPC is accessed.
4. Create .bash-files
Now, the .bashrc file is sourced every time the HPC is accessed. This is convenient because it allows to set default settings, load aliases or modules. To keep everything organized, within .bashrc again different files are sourced (if they are present). This includes .bash_aliases and .bash_modules.
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# load aliases
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
# load modules
if [ -f ~/.bash_modules ]; then
. ~/.bash_modules
fiHere are some convenient aliases that could be saved in the .bash_aliases file (created using nano as before). .bash_aliases makes sure they are available on the HPC and allow an easier application of common SLURM commands. Of course, this list can be expanded by any alias required. This does not only include SLURM commands, but can also include Linux commands.
alias jobs_p='squeue'
alias jobs_med='squeue --partition=medium-fas,medium-fmz'
alias jobs_fat='squeue --partition=fat-fas,fat-fmz'
alias jobs_user='squeue -u $USER'
alias jobs_run='squeue -u $USER --states=RUNNING'
alias jobs_pen='squeue -u $USER --states=PENDING'
alias jobs_n='squeue -u $USER --states=RUNNING | wc -l'
alias jobs_kill='scancel -u $USER'
alias jobs_info='sacct -u $USER'
alias monitor='htop -u $USER'To be able to automatically import modules, .bash_modules (again, using nano) needs to be created. Modules provide a way to selectively activate and deactivate modifications to the user environment. This allows particular packages and versions to be found. First, the GNU compiler gcc is loaded. Second, the messaging library zeromq is loaded and used later to submit jobs to the HPC. The R module makes sure that always the most recent R version is used. Additionally, it’s possible to load any module which is needed (to see all available modules, use module available), such as e.g. R_GEO which allows to load certain spatial packages in R later.
GWDG HPC infrastructure
The HPC infrastructure is mostly accessible through partitions (formerly queues). A full overview can be found here: High-Performance Computing in Göttingen (outdated). There are two general base partitions that are of interest.
- medium - lots of cores, fewer memory, short pending time (formerly MPI)
- fat - fewer cores, lots of memory, long pending time
Both partitions have additional a “quality of service” (QOS), i.e. “normal”, “short” and “long” (formerly appended using “-”). This refers to the maximum walltime a job can run on the HPC. Of course this also relates to possible pending times. Generally, the possible pending time might be longer in the “long” QOS. However, once a job is running, it’s also allowed to use the partition longer.
There is a maximum number of jobs that can be submitted, as well as a maximum of simultaneously running jobs. The following command allows to check the limits on the HPC. The default is a submission of maximum 5000 jobs at a time and 1000 jobs running simultaneously.
More details can be found here.
sbatch command
The SLURM command sbatch is used to submit jobs to the HPC. This is important to remember to adjust submissions of jobs and understand error messages. Regular R syntax is only used as a proxy.
General SLURM commands
Here are the previously defined aliases for the most important SLURM commands to monitor and control jobs on the HPC. Of course, it is possible to change this list by modifying .bash_aliases. Also, it’s possible to combine all aliases with the default SLURM options or just use the original SLURM commands.
sinfo: General information about all partitionsjobs_p/jobs_med/jobs_fat: All jobs submitted to all/medium/fat partition(s)jobs_user/jobs_run/jobs_pen: Only own all/running/pending submitted jobsjobs_n: Number of own running jobs. Because the header is counted, actual count is n - 1jobs_kill: Kill all submitted jobsjobs_info: Information about finished jobs
Loading modules
As described above, modules help to select software versions needed to run code. The following command displays all available modules provided by the GWDG.
Modules can be loaded using the following command and the name of the corresponding module.
It is possible to list all loaded modules. This list should at least contain all modules specified in the .bash_modules file plus all later additionally loaded modules.
Disconnecting and reconnecting via ssh also removes all modules (with exception of those specified in the .bash_modules file). To detach selected modules or all modules at once without logging out, the purge command can be used.
Installing packages
All packages used in a local scripts must also be also installed on the HPC. This must be done for all R versions separately, i.e. if R is updated on the HPC, all packages must be re-installed. The easiest way to install packages on the HPC is to login to the frontend and start R.
Installing R packages works as expected. The only difference to installing packages locally is that R might asks to use the private library of the user (in which the user has writing rights).
To install packages from GitHub it is necessary to specify the path to the private library. Therefore, first the devtools and withr packages need to be installed. Now, the path can be provided where the package should be installed. It’s possible to get this path using .libPaths(). This path is wrapped around the install_github() function. Make sure to check the correct path each time you use a different R version.
Using the future package
One of the ways to use the GWDG HPC is the future package and framework. The advantage of future is that code can be run on the HPC with only minor changes. Furthermore, specifying how to parallelise the R code is straightforward, as to control how to distribute jobs over nodes and cores. A basic knowledge of future is advised.
1. Create template to submit jobs
A template file is used to control how jobs are submitted to the HPC. This allows to pass arguments as R syntax from the local R session to the SLURM system as sbatch commands. The template file, called e.g. future_slurm.tmpl, needs to be created on the HPC (same same nano future_slurm.tmpl). The following is an example for such a template file allowing to specify the most important sbatch options.
## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.
#!/bin/sh
#SBATCH --job-name <%= resources$job_name %> ## Name of the job
#SBATCH --ntasks <%= resources$n_cpu %> ## number of processes
#SBATCH --partition <%= resources$queue %> ## Job queue
#SBATCH --qos <%= resources$service %> ## QOS
#SBATCH --time <%= resources$walltime %> ## walltime in hh:mm:ss
#SBATCH --mem-per-cpu <%=resources$mem_cpu %> ## min memory per core
#SBATCH --nodes <%= resources$nodes %> ## if 1 put load on one node
#SBATCH --output <%= resources$log_file %> ## Output is sent to logfile, stdout + stderr by default
## Export value of DEBUGME environment var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>
module load gcc
module load R
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'2. First login
Every package from now on needs to be installed on both the local computer and the HPC!
The following R code sends every future to the HPC, as specified by the future plan.
# load the packages
library("future.batchtools")
library("future")
library("furrr")
# now we specify a future topology that fits our HPC
# login node -> cluster nodes -> core/ multiple cores
login <- tweak(remote, workers = "gwdu101.gwdg.de", user = "user_name_gwdg") # user = login credential
sbatch <- tweak(batchtools_slurm, template = "future_slurm.tmpl",
resources = list(job_name = "run_hpc", # name of the job
log_file = "run_hpc.log", # name of log file
queue = "medium", # which partition
service = "short", # which QOS
walltime = "00:05:00", # walltime <hh:mm:ss>
n_cpu = 12)) # number of cores
plan(list(
login,
sbatch,
multisession # how to run on nodes, could also be sequential
))3. Going down the future topology
Now it’s possible to reach the first level of the HPC (frontend). After logging in to gwdu101.gwdg.de, the function Sys.info() is executed on the frontend of the HPC.
# Before we start, despite that we have declared our future plan above,
# we are still working on our local machine if we do not use futures:
local_sysinfo <- Sys.info()
local_sysinfo
# To do something on the hpc, we actually have to use a future,
# in the following example with the future assignment %<-%
hpc_sysinfo %<-% Sys.info()
hpc_sysinfoThe second level, the cluster nodes, can be reached with a nested future. The furrr packages allows to use the purrr::map-family as futures. The future-operator %<-% is used to accesses the HPC frontend and future_map to accesses the second level cluster nodes. Because code should never be executed on the frontend, these two nested future levels are always required to submit jobs. The second level also controls the total number of jobs submitted. This becomes very important later for sequential job submission and exclusive jobs with future.
# do something on the cluster node level
hpc_sysinfo_on_nodes %<-% future_map(seq_len(10), ~ Sys.info())
hpc_sysinfo_on_nodesThe third level, the cores on each cluster node, is addressed by the settings specified in the second level. For example, in the second level multisession with processes = 12 can specified. This means demanding 12 cores on each cluster node and run the function parallel on each. Hence, the function could run on 12 cores and use the memory every core comes with. Here, the jobs are distributed over 10 nodes that have at least 12 available cores. In total, Sys.info() is executed 10 x 12 times. Therefore, hpc_sysinfo_on_node_cores is a two dimensional list, with 10 elements where each of these has again 12 elements. The 12 elements in each outer dimension of the list should be the same, but differ between the unique outer elements overall.
4. Submitting jobs efficiently
The future plans are crucial, as they specify how to distribute jobs. Therefore, before submitting jobs to the HPC it is important to ensure that the architecture of the plan and how the code is parallelised match. Otherwise, it may be possible to bypass the scheduling system and drain resources from other users.
The following code snippet explains how to access specific partitions, set the walltime and design a sequential plan. This setup makes sense if the jobs depend of each other, need a specific chip infrastructure or have memory needs that are only satisfied if whole cluster nodes are blocked and jobs are run there exclusively.
# login node -> cluster nodes -> core/ multiple cores
# there shouldn't be to much change for the login level of your plan
# you specify the address of the hpc and your user name
login <- tweak(remote, workers = "gwdu101.gwdg.de", user = "user_name_gwdg")
# the sbatch, or cluster node, level becomes the first stage
# where you could make adjustment to control you experiment.
# This tweak fills out the space between the curly brackets
# in the future_slurm.tmpl file we created on the HPC.
# (scroll to the right to read an explanation of every line)
sbatch <- tweak(batchtools_slurm, template = "future_slurm.tmpl",
resources = list(job_name = "run_hpc", # name of the job
log_file = "run_hpc.log", # name of log file
queue = "medium", # which partition
service = "short", # which QOS
walltime = "00:05:00", # walltime <hh:mm:ss>
n_cpu = 12)) # number of cores
plan(list(
login,
sbatch,
multisession # multisession because 12 cores were blocked on the sbatch level
))However, if jobs are independent of each other (e.g. repetitions of the same function) and do not need more memory than one core provides, it makes sense to submit a high number of very small jobs. Instead of blocking a whole node and running the function on all available nodes in parallel (submitted as one job), it is also possible to submit 12 single jobs only demanding one core per job. This has the advantage that the pending time will be decreased enormously. The reason for this is that the jobs will be submitted to any available free slot (1 core) on the cluster. The SLURM system automatically schedules these small jobs with a higher priority in order to use the capacities optimally.
# (scroll to the right to read an explanation of every line)
sbatch <- tweak(batchtools_slurm, template = "future_slurm.tmpl",
resources = list(job_name = "run_hpc", # name of the job
log_file = "run_hpc.log", # name of log file
queue = "medium", # which partition
service = "short", # which QOS
walltime = "00:05:00", # walltime <hh:mm:ss>
n_cpu = 1)) # number of cores
plan(list(
login,
sbatch,
sequential # we need sequential here, so that every job we submit only runs on a single core
))5. .future folder
future.batchtools creates a folder called .future/ on the HPC in the home directory. In this folder all submitted jobs are collected and the folder structure indicates the date and time of submission. If all jobs are collected successfully and retrieved on the local computer, the folder is empty and only contains a .sessioninfo.txt. However, all logs and (partial) results of failed jobs won’t be deleted. This also includes jobs that are killed by the user.
Note: It’s advisable to remove failed jobs from time to time in the .future/ folder. After a while, this folder can use up a lot of the disk quota on the HPC.
6. Example
Submit jobs to the HPC with the following requirements:
- 120 cores to simulate jobs
- More than 10 GB of RAM
- Functions need about 12 hours to run
# load the packages
library("future")
library("furrr")
library("tidyverse")
# login node -> cluster nodes -> core/ multiple cores
login <- tweak(remote, workers = "gwdu101.gwdg.de", user = "username")
# We submit to the multipurpose queue with a maximum
# walltime of 48 hours (we specify 15 so that we
# we are not pending that long).
# We furthermore specify 24 processes, which
# automatically means that we reserve nodes
# exclusively (the maximum number of cores on
# mpi nodes is 24, which means if we say 24
# here we wait until a node is not used by anyone).
sbatch <- tweak(batchtools_slurm, template = "future_slurm.tmpl",
resources = list(job_name = "run_hpc", # name of the job
log_file = "run_hpc.log", # name of log file
queue = "medium", # which partition
service = "normal", # which QOS
walltime = "00:15:00", # walltime <hh:mm:ss>
n_cpu = 12)) # number of cores
plan(list(
login,
sbatch,
multisession # Again: multisession to distribute over all the cores
))
# let's create an imaginary data frame to iterate over
mydata <- data.frame(x = rnorm(100), y = rnorm(100))
advanced_data <- list(mydata, mydata, mydata, mydata, mydata)
fancy_statistical_model <- future_map_dfr(advanced_data, function(x) {
future_map_dfr(seq_len(ncol(x)), function(y) {
single_row <- x[y, ]
tibble(single_row$x + single_row$y)
}, .id = "y")
}, .id = "x")Using the clustermq package
clustermq is another package that allows to submit jobs to the HPC via SSH connector. Compared to future (and “friends”), it has the advantage that it submits job arrays. Instead of gradually sending job after job, as done in the examples before, clustermq sends them all at once. This can be a huge time benefit. However, currently clustermq is only working on Linux and macOS
First, all necessary dependencies need to be installed on the local system if you use Linux. Loading the module zeromq should take care of this on the HPC.
# You can skip this step on Windows and macOS
brew install zeromq # Linuxbrew, Homebrew on macOS
conda install zeromq # Conda
sudo apt-get install libzmq3-dev # Ubuntu
sudo yum install zeromq3-devel # Fedora
pacman -S zeromq # ArchlinuxTo use clustermq, it has to be installed on both the local computer and the HPC.
# local pc
devtools::install_github("mschubert/clustermq")
# HPC
withr::with_libpaths("/home/uni08/user_name_gwdg/R/x86_64-pc-linux-gnu-library/3.5", devtools::install_github("mschubert/clustermq"))Afterwards, the local .Rprofile needs to be modified. This can be done with the usethis package.
options(
clustermq.scheduler = "ssh",
clustermq.ssh.host = "username_gwdg@gwdu101.gwdg.de", # use your user and host, obviously
clustermq.ssh.log = "~/clustermq_ssh.log" # log for easier debugging
)Also the .Rprofile on the HPC (nano .Rprofile in the home directory) needs to be modified.
options(
clustermq.scheduler = "slurm",
clustermq.template = "/home/uni08/username/clustermq_slurm.tmpl"
)Then, again an template to submit jobs is needed, e.g. nano clustermq_slurm.tmpl. The template submits jobs to the HPC specifying the options the scheduling system uses.
#!/bin/sh
#SBATCH --job-name={{ job_name }} # job name
#SBATCH --array=1-{{ n_jobs }} # number of processes
#SBATCH --partition={{ queue | medium }} # name of queue
#SBATCH --qos={{ service | normal }} # which special QOS (short/long)
#SBATCH --time={{ walltime | 12:00:00 }} # walltime in hh:mm:ss
#SBATCH --cpus-per-task={{ n_cpu | 1 }} # set cores per task
#SBATCH --mem-per-cpu={{ mem_cpu | 1024 }} # set min memory per core
#SBATCH --nodes={{ nodes | 1 }} # if 1 put load on one node
#SBATCH --output={{ log_file | /dev/null }}
#SBATCH --error={{ log_file | /dev/null }}
module load gcc/7.4.0
module load zeromq
module load R
module load R_GEO
ulimit -v $(( 1024 * {{ mem_cpu | 1024 }} ))
CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker("{{ master }}")'How to use clustermq
clustermq does not rely on plans as future. Submitting jobs becomes as easy as providing a function and a list or vector to iterate over. In the example, for each x = 1...5 a single job is submitted to the HPC. Contrastingly, y = 10 is hold constant for all submitted jobs. In case user-defined functions are needed, these can be specified by the the export argument of the Q() function. It’s also possible to iterate over rows of a data frame.
# Run a simple multiplication for data frame columns x and y on a worker node
fx = function (x, y) x * y
df = data.frame(x = 5, y = 10)
Q_rows(df, fx, job_size = 1)
# Q_rows also matches the names of a data frame with the function arguments
fx = function (x, y) x - y
df = data.frame(y = 5, x = 10)
Q_rows(df, fx, job_size = 1)All parameters within {{ }} in the template file can be specified using a list and the template argument of the Q() function. Therefore, it is very easy to e.g. submit to the short QOS and set the walltime to 5 minutes (
Tips and Tricks
1. Kill jobs/R sessions
There is an alias to kill already submitted jobs. In case something goes wrong while submitting jobs, there will be most likely several R sessions that do not close by themselves or are open and keep submitting jobs. A convenient way to kill these processes is to login to the HPC frontend and use the defined alias monitor to enter the process manger htop. With F9 it’s possible to select the R processes and kill them. F10 closes the process manager afterwards.
2. Better connection to the HPC
One inconvenience is that the connection to HPC from a local R session can not be interrupted. If that happens, jobs are lost (using future there might be chance to collect every result in the .future folder). This also means that running code from a local computer means that it has to be running until everything is finished. When code take several days, this can easily become a problem.
A way we explored for such cases is to use the GWDG Cloud Server. If you install there an instance of RStudio Server, you have a cloud based IDE that you can run from everywhere with access to the internet. By default, the RStudio Server unfortunately breaks the connection every now and then, so you have to tweak it a bit to be permanently accessible. To turn off the suspended session feature, must be specified in a ression.conf file. After the line session-timeout-minutes=0 is added to the file, the server must be restarted.
Also the SSH connection becomes a bit tricky from the server. Therefore, we first need to generate a SSH key pair and copy the public key information to the authorized_keys file on the HPC cluster. Most likely, you have to do this by hand.
Now, via the terminal of the RStudio Cloud Server, first connect to login.gwdg.de.
If this works, from the new console connect to gwdu101.gwdg.de.
If this works, we can automate the two steps. Navigate to the .ssh/ folder of the Cloud Server and create a config file and enter the following.
Host gwdu101.gwdg.de
ProxyCommand ssh gwdg_username@login.gwdg.de -W %h:%p
StrictHostKeyChecking no
ServerAliveInterval 30Now you should be able to login from the RStudio Cloud Server Terminal to the HPC directly as usual without any password identification. If so, you should be able to submit jobs to the cluster from the Cloud Server just as from your local machine. If you use clustermq, it’s not possible to logout and login again to the Cloud Server as long as jobs are running on the HPC. So either, you need to keep the browser window open as long as jobs are running on the HPC, or use another (local) terminal to check if still jobs are running and first login again on the Cloud Server once all jobs are finished.
In case you use another RStudio Server, the only thing you would have to change from the things we explained here is to open a VPN connection to the goenet.
Troubleshooting
Here, we are going to collect some issue we had during the setup and using the cluster at one point or the other. This issues might be rather specific, but maybe you run into the same problems. Thanks a lot to Jan Salecker who made us aware of the problems and also provided the solutions.
3.1 Problems with the .Rprofile
There was the problem that clustermq kept telling to specify the scheduler, even though the .Rprofile was set up correctly. In this case the problem was that there was no empty, new line at the end of the .Rprofile file.
3.2 Byte character sets
If you use a text editor to create all the template files, it might be possible that hidden characters (e.g. line break commands) are automatically added to the template files. This can also be a problem if you switch between different byte character sets. In case your jobs return errors and you see a lot of e.g. “” in your template files (or any other files), one possible solution is to run the following command in order to convert the file to a unix character set.
3.3 Kill old processes
We advise to make sure all processes are killed from time to time. This includes all submitted and not finished jobs using jobs_kill as well as all others processes. Therefore, you can use monitor to enter the htop and simply kill all processes you see that are somehow related to R. This is especially important if you had jobs that didn’t finish successfully before.
Links
Acknowledgments
Huge thanks to the GWDG HPC team that put quite some effort into installing R packages and explaining some HPC basics to us. Another big thanks to Henrik Bengtsson, Michael Schubert and Michel Lang for developing nice tools (future, clustermq, batchtools) to enable even ecologists to use cluster interfaces remo::ji("smile").