R/librarySnapshot.R
librarySnapshot.RdEvery experiment backend (experimentTmux(), experimentFuture(),
experimentSBATCH()) runs many jobs against one project library. Nothing
stops that library from being modified while jobs use it, and R lazy-loads
package objects from R/<pkg>.rdb by file offset, so replacing a package
under a running session ends in
lazy-load database '.../R/<pkg>.rdb' is corrupt. Two functions make the
library safe:
syncProjectLibrary() installs or updates packages once per launch, in
a fresh R process (this session may itself have them loaded), and refuses
to run while any worker is alive. The backends call it when given
sync_library = <package specs>.
snapshotLibrary() gives one job its own copy of the library, made of
hardlinks (file.link(), so it costs seconds and no disk), falling back to
file.copy() across filesystems. A hardlink keeps the old inode alive: an
install into the shared library replaces files there and cannot touch
what the running job has open. Every job thereby also records exactly which
package versions it ran with. The backends take the snapshot in the worker
startup script, before any package is loaded, when snapshot_library = TRUE
(the default).
libraryInUse() reports the workers that make a sync unsafe;
releaseLibrarySnapshot() removes one snapshot (the worker loop does this at
exit) and sweepLibrarySnapshots() removes those whose owning process is gone.
syncProjectLibrary(
packages,
libPath = .libPaths()[1],
activeRunningPath = getOption("spades.activeRunningPath"),
requireIdle = TRUE,
installFun = NULL,
verbose = getOption("Require.verbose", 1)
)
libraryInUse(activeRunningPath = getOption("spades.activeRunningPath"))
snapshotLibrary(
libPath = .libPaths()[1],
where = getOption("spades.activeRunningPath"),
id = paste0(Sys.info()[["nodename"]], "_", Sys.getpid()),
setLibPaths = TRUE
)
releaseLibrarySnapshot(snapshot = Sys.getenv("SPADES_PROJECT_LIB_SNAPSHOT"))
sweepLibrarySnapshots(where = getOption("spades.activeRunningPath"))Character vector of package specifications as accepted by
Require::Install(), e.g.
"PredictiveEcology/reproducible@development (>= 3.2.1.9010)". Usually
the vector given to setupProject(packages = ), so the version floors
declared there are the single source of truth.
The shared library. Defaults to .libPaths()[1].
Where the experiment keeps its Running_* sentinels;
snapshots are created there as lib_<id>/ unless where says otherwise.
Logical. Refuse to install while libraryInUse() finds a
worker. Set to FALSE only when you know the workers are idle.
Function of (packages, libPath) performing the install;
defaults to Require::Install(). Exposed for tests.
Numeric.
Directory holding snapshots; defaults to activeRunningPath.
Snapshot id; defaults to <nodename>_<pid> so
sweepLibrarySnapshots() can tell a live owner from a dead one.
Logical. Put the snapshot first on .libPaths() in place
of libPath, and record it in SPADES_PROJECT_LIB_SNAPSHOT so the worker
loop can release it at exit.
A directory returned by snapshotLibrary().
syncProjectLibrary() returns TRUE invisibly after installing,
FALSE when packages is empty. libraryInUse() returns a character
vector describing live workers (sentinel files and ps lines), empty when
none. snapshotLibrary() returns the snapshot directory.
releaseLibrarySnapshot() returns TRUE invisibly if it removed one.
sweepLibrarySnapshots() returns the directories it removed.
if (FALSE) { # \dontrun{
# before launching an experiment, with no worker running:
syncProjectLibrary(c("PredictiveEcology/reproducible@development (>= 3.2.1.9010)",
"PredictiveEcology/SpaDES.core@development"))
# inside a worker, before loading anything from the library:
snapshotLibrary(where = "logs/my_queue.rds")
} # }