Creates n_workers tmux panes in the current window, tiles them, and starts
a worker loop in each one that claims and runs jobs from a file-backed queue
(queue_path). Control returns immediately to the master pane; all work
happens asynchronously inside the worker panes.
pane_mode)"killAndNewPane" (default)Each worker runs one job per R session, then exits. A fresh R session starts automatically for the next job, freeing all memory between runs.
localhost panes: After each job, tmuxRunWorkerLoop() calls
tmux respawn-pane -k, which replaces the current pane's process
in-place with a new Rscript invocation. No retiling needed.
Remote panes (cores = "hostname"): The local pane runs a bash
while-loop that repeatedly calls
ssh -t host bash -c 'exec env R_PROFILE_USER=<script> R --interactive'.
ssh -t allocates a PTY so R runs interactively (readline, OSC 2 title
updates, Ctrl+C propagation). A startup script injected via
R_PROFILE_USER runs one job then exits; q(status = 1L) (job done or
queue empty) lets the while-loop start a fresh R session, q() (status 0)
stops the loop. R_PROFILE_USER is unset inside R immediately
after startup so workers spawned by makeClusterPSOCK() do not inherit
it and inadvertently re-run the startup script.
cores)Supplying a hostname in cores triggers .setup_remote_machine() once per
unique host before any workers start. Steps run in this order:
Guard BASH_ENV – wraps the remote $BASH_ENV file's existing
content in a subshell (( ... ) 2>/dev/null || true) so that any exit
or failing command inside it cannot abort the non-interactive SSH shell
that carries setup commands.
Create remote directory; copy files – mkdir -p the remote working
directory (same relative path from ~ as on localhost), then scp
global_path, queue_path, and dots_path (if supplied) into it.
Rsync project R/ folder – syncs the R/ subdirectory next to
global_path to the remote with rsync --delete so user-defined
helper functions sourced by global.R are up to date.
Write ~/.Rprofile on remote – injects three lines (replacing any
previous versions): .libPaths(c(local_lib, ...)) so the project library
takes precedence over system libraries; options(repos = ...) including
the PredictiveEcology r-universe; and an SSL block that sets
CURL_CA_BUNDLE/SSL_CERT_FILE so HTTPS downloads work in non-login
SSH sessions where /etc/profile.d/ is not sourced.
Verify/install Require – compares the remote Require version and
git commit SHA to the local installation. If they differ, rsyncs the
installed directory (GitHub source) or runs install.packages("Require")
(CRAN source).
Install usethis on the remote via Require::Install().
Propagate GitHub credentials – reads the local token via
gitcreds::gitcreds_get() and pipes it into git credential approve on
the remote so private GitHub packages can be installed without interactive
setup. Falls back to checking whether the remote already has credentials;
errors if neither is true.
Install system libraries via
sudo -n apt-get install -y --no-install-recommends (non-interactive;
fails gracefully if passwordless sudo is not configured). Libraries
installed: spatial (libgdal-dev, libgeos-dev, libproj-dev,
libsqlite3-dev, libudunits2-dev), HTTP/TLS (libssl-dev,
libcurl4-openssl-dev), XML (libxml2-dev), archive (libarchive-dev),
git (libgit2-dev), fonts/graphics (libfontconfig1-dev,
libharfbuzz-dev, libfribidi-dev, libpng-dev, libjpeg-dev,
libtiff-dev, libfreetype6-dev), protobuf (libabsl-dev), and R
compilation headers (r-base-dev).
Ensure remote lib path exists – mkdir -p the project library path
on the remote (must match localhost exactly so installed file paths are
identical).
Rsync SpaDES.project – copies the locally installed SpaDES.project
directory to the same path on the remote. Both machines must share the
same platform and R version so compiled lazy-load databases are compatible.
Install SpaDES.project dependencies via Require::Install().
Spatial packages (terra, sf, rgdal, rgeos, lwgeom) are compiled
from source so they link against the remote's actual GDAL/GEOS/PROJ
versions. All other hard dependencies (Imports/Depends/LinkingTo) plus
any Suggests packages installed locally are installed as binaries via
Require::setLinuxBinaryRepo(). Common packages with strict version
requirements (purrr >= 1.2.1, rlang >= 1.1.7, cli >= 3.6.0,
vctrs >= 0.6.0) are pre-installed to the project library to avoid
stale system-library versions being picked up during compilation.
Rsync Require package cache (Require::cachePkgDir()) to the
remote to accelerate future package installations.
Rsync gargle OAuth cache (cache_path or
getOption("gargle_oauth_cache")) to the remote so the worker can
authenticate with Google APIs (Sheets, Drive) without a browser prompt.
Pane 1 starts immediately. Pane i > 1 waits
delay_before_source + (i - 2) * stagger_by seconds inside R before
claiming its first job, avoiding simultaneous queue contention at startup.
For remote workers in killAndNewPane mode the stagger only applies to the
first R session; subsequent while-loop iterations start immediately.
If a worker pane is manually interrupted (e.g. Ctrl+C) and drops to a shell
prompt, restart it by pressing (up-arrow) (up-arrow) in that pane and hitting Enter.
The full command is always in the pane's bash history:
localhost: Rscript -e "..." (re-enters tmuxRunWorkerLoop; in
killAndNewPane mode respawn-pane takes over from the first job onward).
remote: if setup && scp; then first_run; _st=$?; while [ $_st -ne 0 ]; do sleep 2; loop_run; _st=$?; done; fi
command (restarts the sh loop from scratch; plain POSIX – works in bash, dash, and sh).
At startup, experimentTmux sets the tmux session option
default-terminal = "tmux-256color". This ensures that all subsequently
created panes advertise a full-colour ANSI terminal, which is required for
R packages such as cli and crayon to render coloured/dynamic output
correctly. Without this, connections that arrive via Windows PowerShell
-> SSH -> tmux often inherit TERM=screen or no TERM at all, causing
R to fall back to plain-text output. The setting is applied globally to the
session (-g) and persists for the session's lifetime; it does not modify
~/.tmux.conf.
experimentTmux(
df,
global_path = "global.R",
cores = NULL,
n_workers = if (is.null(cores)) 4L else length(cores),
delay_after_split = 0.4,
delay_after_layout = 0.4,
delay_between_R_start = 0,
delay_before_source = 60,
stagger_by = delay_before_source,
set_mouse = TRUE,
statusCalculate = getOption("spades.statusCalculate"),
folderWithIterInFilename = getOption("spades.folderWithIterInFilename"),
activeRunningPath = getOption("spades.activeRunningPath"),
continue = TRUE,
queue_path = NULL,
on_interrupt = c("requeue", "fail"),
pane_mode = c("killAndNewPane", "reuse"),
ss_id = NULL,
forceLocalQueueToGS = FALSE,
enableGSSync = FALSE,
email = getOption("gargle_oauth_email"),
cache_path = getOption("gargle_oauth_cache"),
workersToMonitor = unique(if (is.null(cores)) "localhost" else cores),
runNameLabel = quote(colnames(q)[1:2]),
copyModules = FALSE,
...
)A data.frame of parameter combinations. Each row is one job.
Column names become object names in worker panes; values from each row
are assigned prior to sourcing global_path.
Character scalar. Absolute path to the script sourced for each job.
Character vector of machine hostnames, recycled to n_workers. Use
"localhost" for the local machine or a bare hostname (e.g. "sbw") for a remote
machine reachable via passwordless SSH. When any remote hosts are listed,
.setup_remote_machine() is called for each unique hostname before workers start.
Default NULL (all localhost).
Integer. Number of worker panes to spawn. Defaults to length(cores)
if cores is supplied, otherwise 4.
Numeric. Seconds to wait after each split-window. Default 2.
Numeric. Seconds to wait after select-layout. Default 0.2.
Numeric. Seconds to wait after starting R in each pane.
Default 0.1.
Numeric. Seconds panes 2..n wait before claiming their first
job. Default 60.
Numeric. Additional seconds per pane beyond pane 2:
pane i > 1 waits delay_before_source + (i - 2) * stagger_by. Default delay_before_source.
Logical. Enable tmux mouse support (pane selection, scroll). Default TRUE.
A quoted expression (optionally using runName) that evaluates
to a path containing job-status output files. Currently used by fireSense_SpreadFit.
Default getOption("spades.statusCalculate", NULL).
A quoted expression (optionally using runName) for a
folder whose filenames encode iteration info. Currently used by fireSense_SpreadFit.
Default getOption("spades.folderWithIterInFilename", NULL).
Directory for "running" flag files written while a job is
active. Must be cleaned up manually if a job crashes without removing its flag.
Default: file.path("logs/", basename(queue_path)).
Logical. Reserved for future single-shot mode; currently ignored.
Character. Path to the .rds queue file. Defaults to
file.path(dirname(global_path), "tmux_queue.rds").
"requeue" (default) or "fail". Action when a job errors:
requeue it for another worker, or mark it failed and stop this worker.
"killAndNewPane" (default) or "reuse". See Worker loop modes
above.
Optional Google Drive spreadsheet/folder ID for live status syncing via
googlesheets4. NULL disables syncing.
Logical. If TRUE, overwrite the Google Sheet
queue with the local df even if the sheet already contains rows.
Default FALSE.
Logical. If TRUE, start an additional tmux pane
that periodically syncs the local queue file to a Google Sheet (requires
ss_id). Default FALSE.
Optional email address for gargle/Google OAuth authentication.
Optional path to the gargle OAuth token cache directory.
Character vector of pane titles to monitor (currently unused).
A quoted expression evaluated against the queue data.frame to
produce a human-readable job label used in log files and Google Sheet status updates.
Logical. If TRUE and remote hosts are present, rsyncs the
directory given by getOption("spades.modulePath") to the same absolute path on
each remote host before workers start. Issues a warning and skips if the option is
unset. Default FALSE.
Additional arguments passed to .setup_remote_machine().
Invisibly returns a character vector of tmux pane IDs for the spawned workers.
Pass these to tmuxKillPanes() to tear down all workers at once.
if (FALSE) { # \dontrun{
# --- Minimal: build a tiny global.R, then run a 2 x 2 experiment ---
tdir <- file.path(tempdir(), "experimentTmux-demo")
dir.create(tdir, showWarnings = FALSE, recursive = TRUE)
writeLines(
'message("scenario=", .scenario, " rep=", .rep); Sys.sleep(2)',
file.path(tdir, "global.R")
)
expt <- expand.grid(.scenario = c("A", "B"), .rep = 1:2,
stringsAsFactors = FALSE)
workers <- experimentTmux(
df = expt,
global_path = file.path(tdir, "global.R"),
cores = rep("localhost", 2L),
queue_path = file.path(tdir, "queue.rds")
)
# --- Live inspection while panes run ---
experimentMonitor() # tmux pane scan (no args)
experimentMonitor(stats = TRUE) # adds CPU / RAM / state per pane
tmuxListPanes() # alias of experimentMonitor()
queueRead(file.path(tdir, "queue.rds")) # full queue snapshot
tmuxFindDuplicates(workers) # any double-claimed jobs?
tmuxRefreshQueueStatus(file.path(tdir, "queue.rds")) # reset stuck rows
# --- Basic local usage with explicit pane sizing ---
workers <- experimentTmux(
global_path = "/abs/path/to/global.R",
queue_path = "/abs/path/to/queue.rds",
n_workers = 4,
pane_mode = "killAndNewPane",
delay_before_source = 60,
stagger_by = 60,
set_mouse = TRUE
)
# --- Mixed local + remote ---
# Runs 2 workers on localhost and 2 on remote host "sbw".
# .setup_remote_machine("sbw", ...) is called automatically before workers start.
workers <- experimentTmux(
global_path = "/abs/path/to/global.R",
queue_path = "/abs/path/to/queue.rds",
cores = c("localhost", "localhost", "sbw", "sbw"),
pane_mode = "killAndNewPane",
email = "you@example.com",
cache_path = "/abs/path/to/.secret",
ss_id = "your-google-sheet-id"
)
# --- Tear down all workers ---
tmuxKillPanes(workers)
# --- Restart a single broken pane ---
# In the broken pane, press Up then Enter to re-run the last command.
} # }