Download, untar, and load SpaDES sims from Google Drive

Source: R/download.R

Convenience wrapper around reGet(), reUntar(), and reLoad() – the inverse of outSaveTarUpload(). Operates on a batch: typically called with the multi-row dribble returned by outList() / outScenarios().

reGetUntarLoad(
  gFiles,
  destDir,
  pathRemap = NULL,
  projectPath = getwd(),
  method = c("loadSimList", "readRDS"),
  parse = TRUE,
  remote = FALSE,
  overwrite = FALSE,
  verbose = TRUE,
  skipExisting = TRUE,
  useCache = FALSE,
  pattern = NULL
)

Arguments

gFiles

Either a Google Drive dribble (e.g. the output of outList() / outScenarios()) or a character vector of Drive file IDs or URLs.

destDir

Character scalar. Local directory to write tarballs into. Created if it does not exist.

pathRemap

Optional named character vector of length 2, c(old = "/old/prefix", new = "/new/prefix"), applied to all tarballs. If NULL (default), files are extracted to their original absolute paths (tar --absolute-names).

projectPath

Character scalar. Passed to SpaDES.core::loadSimList() for relative-path resolution. Default getwd().

method

One of "loadSimList" (default) or "readRDS".

parse

Logical. If TRUE (default), module source code is re-parsed on load. FALSE forwards parse = FALSE to SpaDES.core::loadSimList(), which skips it – worthwhile because reparsing dominates the load time of a lazily saved simList (on a 19-module simulation, ~7 s of a ~9.5 s load). Objects are unaffected: user objects and each module's mod objects are still bound lazily. The result is inspect-only – it has no module code, so it cannot be passed to SpaDES.core::spades(). Ignored when method = "readRDS".

remote

Logical. If FALSE (default), the archive is downloaded and untarred as usual, and its objects are lazily read from the local sidecar. If TRUE, the archive is never downloaded: the simList shell, its manifest and the output files are fetched from within the remote archive by HTTP range request, and each object is fetched only when something touches it. Output files are fetched either way, since downstream code reads them through outputs(sim)$file and they are mostly needed. Requires an archive with an index beside it (see reIndex()) and method = "loadSimList".

overwrite

Logical. Force re-download even if the local file exists. Default FALSE.

verbose

Logical. Print elapsed time per download. Default TRUE.

skipExisting

Logical, passed to reUntar(). If TRUE (default), archives already extracted on disk are not extracted again.

useCache

Logical. If TRUE, reproducible::Cache() the metadata step (reGet()), skipping the remote round-trip on repeat calls in a new session. Only the small data.table of names and paths is cached – never the simLists, whose objects are delayedAssign() promises that saving would force, and so fully materialise. A hit is discarded if any path it names has since gone away. Default FALSE, since a hit cannot notice an archive re-uploaded under the same id.

pattern

Optional regex. When supplied, gFiles is taken to be the folder holding the archives and is listed with outList() using this pattern, rather than being the archives themselves. Either way, the _index.rds sidecars written by reIndex() are excluded.

Value

A named list of simList objects, one per row of gFiles, named by the archive's name (sans .tar.gz).