Skip to content

EFForTS-ABM–InVEST Coupling

This section covers the specifics of running the EFForTS-ABM–InVEST coupled model on the HPC. Read sections 01, 02, 03, and 04 first.

The workflow follows the Apptainer-based rslurm pattern from Section 4 with three additions specific to this model:

  1. The container image (ecomod/rstudio:netvest) bundles NetLogo, InVEST, nlrx, and a compatible R environment, but not the group-specific Refforts package. Refforts changes too frequently to bundle in the image — it is installed from a git clone into your personal R library and updated independently.
  2. InVEST writes output into the model directory during a run, so each compute node needs its own private copy of the model files.
  3. The PROJ_LIB environment variable must be set so InVEST can find its spatial projection database.

One-time Setup

Build the container image

Follow Section 4 — Building the Container Image. Use the netvest image:

Store the image in a named subdirectory of $PROJECT (e.g. $PROJECT/sebastian or $PROJECT/efforts). Use a recognisable name rather than $USER — the HPC account names (u12345) are not identifiable without looking them up. Create the folder if it doesn't exist yet:

mkdir -p $PROJECT/<name>

Then build the image:

srun --partition=int --pty bash
module load apptainer
apptainer build $PROJECT/<name>/netvest.sif docker://ecomod/rstudio:netvest

Find the PROJ_LIB path

Enter the container:

apptainer shell --bind $HOME,$PROJECT $PROJECT/<name>/netvest.sif

PROJ_LIB (required by InVEST):

find /usr -name proj.db
# note the output without the trailing /proj.db, typically /usr/share/proj

Save this value. Exit the container with exit.

Install Refforts

Refforts is not bundled in the netvest image and changes too frequently to rebuild the container image each time. It is installed directly into your personal R library in $HOME, which is bind-mounted into the container and persists between sessions. This is an explicit exception to the general rule in Section 4.

First, clone the repository onto the HPC:

git clone https://github.com/EFForTS-B10/Refforts.git ~/Refforts

Then install it via apptainer exec. The container's system R library is read-only and R_LIBS_USER is not set in this image, so specify the personal library path explicitly. Create the directory first (one-time):

mkdir -p ~/R/netvest_library
apptainer exec --bind $HOME,$PROJECT $PROJECT/<name>/netvest.sif Rscript -e 'install.packages(path.expand("~/Refforts"), repos=NULL, type="source", lib=path.expand("~/R/netvest_library"))'

Updating Refforts: when the branch or version changes, pull (or switch branch) and reinstall:

cd ~/Refforts && git checkout <new-branch> && git pull
apptainer exec --bind $HOME,$PROJECT $PROJECT/<name>/netvest.sif \
  Rscript -e 'install.packages(path.expand("~/Refforts"), repos=NULL, type="source", lib=path.expand("~/R/netvest_library"))'

If you rebuild netvest.sif: reinstall Refforts afterwards — the personal library is tied to the container's R version and will break if that changes.

Clone EFForTS-ABM onto the HPC

ssh Emmy-p3
git clone -b <branch> https://github.com/EFForTS-B10/EFForTS-ABM.git

The model is cloned into ~/EFForTS-ABM. To switch to a different branch later:

rm -r EFForTS-ABM
git clone -b <new-branch> https://github.com/EFForTS-B10/EFForTS-ABM.git

Note for sensitivity analyses: nlrx requires that the line set-rand-seed in EFForTS-ABM.nlogo is commented out, otherwise it ignores the seeds passed by nlrx. Edit the file on the HPC if needed:

nano EFForTS-ABM/01_EFForTS-ABM/EFForTS-ABM.nlogo

Job Template

Save the following as submit_template_netvest.sh locally (e.g. on your laptop). It extends the Section 4 template with three additions: loading apptainer, setting PROJ_LIB, and copying the model to a node-local directory before the job runs.

The per-node copy is necessary because InVEST writes output into the model directory. Without it, concurrent nodes would overwrite each other's files.

#!/bin/bash
#SBATCH --array=0-{{{max_node}}}{{{job_array_task_limit}}}
#SBATCH --cpus-per-task={{{cpus_per_node}}}
#SBATCH --job-name={{{jobname}}}
#SBATCH --output=slurm_%a.out
{{#flags}}
#SBATCH --{{{name}}}
{{/flags}}
{{#options}}
#SBATCH --{{{name}}}={{{value}}}
{{/options}}

module load apptainer

export R_LIBS_USER=$HOME/R/netvest_library
export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export PROJ_LIB=/usr/share/proj
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

cp -r ~/EFForTS-ABM/01_EFForTS-ABM $LOCAL_TMPDIR/01_EFForTS-ABM

{{{rscript}}} --vanilla slurm_run.R

Replace PROJ_LIB path with the value retrieved earlier.

rslurm Script

Note that $LOCAL_TMPDIR must be explicitly bind-mounted into the container (via the rscript_path) so R inside the container can access the per-node model copy. The modelpath in the simulation function then references it via Sys.getenv("LOCAL_TMPDIR").

library(rslurm)
library(Refforts)

sim_fun <- function(param_name, param_min, param_max) {
  library("Refforts")

  netlogopath <- "/opt/netlogo/6.2.1"   # path inside container — verify
  modelpath   <- file.path(
    Sys.getenv("LOCAL_TMPDIR"), "01_EFForTS-ABM", "EFForTS-ABM.nlogo"
  )

  nl_obj <- nl(
    nlversion = "6.2.1",
    nlpath    = netlogopath,
    modelpath = modelpath,
    jvmmem    = 1024
  )

  nl_obj@experiment <- experiment(
    expname     = "hpc",
    repetition  = 1,
    tickmetrics = "true",
    idsetup     = c("ca", "setup-with-external-maps"),
    idgo        = "go",
    runtime     = 61,
    evalticks   = 60,
    metrics     = get.abm.metrics(),
    constants   = get.abm.defaults()
  )

  nl_obj@simdesign <- simdesign_simple(nl_obj, nseeds = 1)

  return(run_nl_all(nl_obj))
}

# One row per parameter combination
params <- abm.table %>%
  select(name, min, max) %>%
  drop_na()

sjob <- slurm_apply(
  f             = sim_fun,
  params        = params,
  jobname       = "efforts_sim",
  nodes         = nrow(params),
  cpus_per_node = 1,
  rscript_path  = "apptainer exec --bind $HOME,$PROJECT,$LOCAL_TMPDIR $PROJECT/<name>/netvest.sif Rscript",
  sh_template   = "submit_template_netvest.sh",
  pkgs          = c("nlrx", "Refforts", "tidyr", "dplyr", "rslurm"),
  slurm_options = list(
    "partition"   = "scc-cpu",
    "time"        = "07:30:00",
    "mem-per-cpu" = "2G",
    "mail-type"   = "END,FAIL",
    "mail-user"   = "your@email.de"
  ),
  submit = FALSE
)

Transfer, submit, and retrieve results as in Section 3 — steps 3–5.

Sources

Open Questions