Reproducible figures
Every figure, table and number macro in a camera-ready paper is generated by tracked code from declared data. Each one is listed in the paper project's REPRODUCE.md, and a test enforces the list. A hand-made asset is allowed only when it is declared static and carries a provenance note.
The asset index
The index is a Markdown table. It has at least the columns asset, kind and generator; the columns label, config, data and provenance are read when present. Cells hold their values in backticks. Paths are relative to the project directory, and an asset may be a glob: * matches within a path segment and ** across segments.
| label | asset | kind | generator | config | data | provenance |
|---|---|---|---|---|---|---|
| `fig:teaser` | `papers/images/intro_overlay.png` | generated | `scripts/make_intro.jl` | `configs/figures/intro.toml` | images tier | |
| — | `papers/figures/stats.tex` | generated | `scripts/make_stats.jl` | `configs/figures/stats.toml` | promoted run | |
| `fig:arch` | `papers/images/architecture.pdf` | static | — | — | — | drawn in Inkscape by the authors from the network definition, 2026-05 |Generators and their configs
A generator writes its asset when it names the file, or when it declares it in a comment line reproduce: writes <path-or-glob>. It reads its config of record when it names the file, or when it declares reproduce: reads <path>. Both paths are relative to the project, and any comment syntax works.
Every parameter of a generator belongs in its config of record, not in the script. That covers frame picks, crops, seeds, panel lists, sizes and inputs.
The enforcing test
The project's suite holds the contract:
using Test
using VisualGeometryToolkitEval.Reproduce: reproduce_index_violations
vs = reproduce_index_violations(project_dir; papers = ["papers/main.tex", "papers/supp.tex"],
require_config = true)
foreach(println, vs)
@test isempty(vs)reproduce_index_violations reports each of these:
- an asset the paper reads that no entry covers;
- a missing asset;
- a reference whose path is not static;
- a stale entry;
- an unknown kind;
- a generated entry without an existing generator;
- a generator that does not write its asset;
- a generated entry without a config of record (with
require_config); - a missing config, or one that its generator does not read;
- a static entry without provenance;
- an entry filed under the label of another float;
- a number defined by a generated macro file and re-typed as a literal in the prose. This check is best effort. A coincidence is exempted with
% reproduce: literal-okon its line.
A paper built outside its own directory — a main.tex in paper/src/ compiled from paper/ that writes \input{src/sec/intro} — resolves its references through search_dirs (project-relative), the build's compile directory and TEXINPUTS: reproduce_index_violations(dir; papers, search_dirs = ["paper"]), or paper_project_violations(dir; papers, reproduce = (; search_dirs = ["paper"])).
Verify, deploy and the numbers file
A generator renders into its results, never into the paper. The last stages of a regeneration compare each render with the submitted copy and replace a submitted asset only with the author's sign-off:
using VisualGeometryToolkitEval.Reproduce
verify_assets(outdir, paperdir, names) # writes outdir/verify.tsv, prints each verdict
deploy_assets(outdir, paperdir, names; # verifies first; a differing submitted copy is
replace_submitted = false) # refused unless signed offA driver whose renders are the files of a keyed product gets the two as its last stages from asset_stages, the paper's figures directory paper_figures_dir and the sign-off its config's [deploy] replace_submitted (deploy_signoff):
const STAGES = [Stage(:figure, "the panel"; items = () -> [figure_product()], product = identity, run = publish),
asset_stages(figure_product, CFG, ASSETS)...] # :verify and :deployA .tex asset is compared by its numbers as well as its bytes (compare_fragments: its pgfplots coordinates and its macros), so a reformatted fragment reports "same numbers". The numbers file is one more asset under the same rule: gather_numbers writes it beside the generator's results from the drivers' fragments, one \newcommand per number, and deploy_assets puts it into the paper. uncited_macros lists the numbers the paper defines but never cites, over the files the paper reads (the ones paper_asset_graph visits), with an allow table of reasons.
Writing a driver's numbers
Every number a performance change could move is a macro: a number of the prose, each cell of a table row, a value a figure (its caption, its axis) reads. A driver writes its numbers with the one writer, into a fragment beside its results; gather_numbers gathers the fragments into the numbers file, and deploy_assets puts that into the paper only once the author has signed off a change (replace_submitted).
using VisualGeometryToolkitEval.Reproduce
using VisualGeometryToolkitEval.Experiments: read_provenance
nums = PaperNumbers(; driver = "scripts/hand_eye.jl", config = "configs/hand_eye.toml",
run_key = KEY, commit = read_provenance(OUTDIR)["gitcommit"]) # the run's commit
add_number!(nums, "heFrames", 240) # a number of the prose
add_number!(nums, number_macro_name("he", "rot_deg", "p90"), 0.0412; digits = 3)
add_number!(nums, "heShare", 0.256; percent = true, digits = 0) # 26\%
add_number!(nums, "teaserBoards", 12; label = "fig:teaser") # a value a figure reads
row = add_table_row!(nums, "heBaxter", ["frames" => 61, "transMm" => tex_number(1.234; digits = 2)];
label = "tab:handeye") # "\heBaxterFrames & \heBaxterTransMm"
write_numbers(joinpath(OUTDIR, "numbers.tex"), nums)The fragment holds, sorted by name, each macro after its provenance comment:
% provenance \heBaxterFrames: driver=scripts/hand_eye.jl; config=configs/hand_eye.toml; run=3f2a9c1e; commit=f11598f7; label=tab:handeye
\newcommand{\heBaxterFrames}{61}- Format (
tex_number):digitsfixed decimals (Printf's%.<d>f), orsigdigitssignificant digits without trailing zeros and without an exponent; an integer needs neither.percent,group_from(thin-space thousands),unit, and a range(lo, hi)with itsrange_separator. A NaN or an infinity is refused: a missing evaluation fails the stage. A string value is a body already typeset ("---","10^{-6}"), one value with balanced braces. - Names (
number_macro_name): letters only, never beginning withend; a driver prefix and name parts, each digit spelled out ("fx3_14mm"→FxThreeOneFourMm). - A table row is one macro per cell. The table's tex reads the row
add_table_row!returns, and the prose cites a cell by the same macro. One macro holdinga & b & cwould hide its numbers fromcompare_fragments,macro_literal_violationsanduncited_macros, which read a macro's value as one number. - Provenance: the driver, its config of record (project-relative), the run key and the run's commit (its recorded
gitcommit: the comment changes only when the run does). A number read from another run overrides fields withprovenance = (; run_key = …).gather_numberskeeps each comment, anddeploy_assetsdeploys a numbers file that differs from the submitted one in its% provenancelines alone (a rerun at a new run key or commit; the verify report'sprovenance_only) without sign-off. Any other change, a value, a hand-written comment or whitespace, still needs it. - Deterministic: the same numbers write the same bytes, whatever the order they were added in; the fragment is written whole (a temporary file renamed into place).
- Direction (
better)::lower(an error, a time),:higher(a rate, a score) or:none(a count, a setting), recorded in the provenance comment (; better=lower). A results branch's comparison calls a change of the number IMPROVED or DEGRADED by it; a number without one is CHANGED, with no verdict.add_table_row!takes one direction for the row orcolumn => directionpairs.
Results branches: a full reproduction under review
A full reproduction with updated code becomes a branch of the paper's repository, examined before anything reaches the paper:
using VisualGeometryToolkitEval.Reproduce
run = results_branch(projectdir(), [render => "paper/src/fig/teaser.png", numbers => "paper/src/latex/numbers.tex"];
run_keys, tiers = ["rc_baselines" => tier_dir], slurm_jobs, gpus, wall_times)
# run.branch == "results/20261111_021530-3de1a4f": the UTC stamp and the evaluated commit
accept_results(projectdir(), run.branch; replace_submitted = true) # one merge commit, tag accepted/20261111_021530
remove_results_worktree(projectdir(), run.branch) # after acceptance, or as the rejectionresults_branchcreates the branch from the base (the checked-out branch) in a worktree of its own under the scratch root (VGTK_RESULTS_WORKTREES, else<repository>-resultsbeside the repository); the user's checkout is not touched. Its one commit holds the renders at the paper's paths (macros, tables, figures: nothing large),RUN.toml(the code commit, each data tier's hash, the run keys under which the heavy results stay in the results root, the machine, GPUs (detected by nvidia-smi or CUDA.jl when not given), Slurm jobs and wall times; a run that gives no tiers, GPUs or wall times is warned about andmissingnames them) andCOMPARISON.md: every macro's base and new value, its delta and its verdict (a changed unit or percent sign is CHANGED), every asset's verify verdict, each changed PNG beside the base's copy. A failure after the worktree is created removes the worktree and the branch.accept_resultsis never automatic. It deploys the branch's assets into the base throughdeploy_assets(a changed asset needsreplace_submitted), as one merge commit made in a worktree of its own, then tags itaccepted/<stamp>, then moves the base to it: a checkout of the base fast-forwards (refused when an accepted asset has local changes: the tag is deleted again and the base stays), a checkout on another branch is untouched. The fast-forward is agit mergein that checkout and runs itspost-mergehook, the user's own; the merge commit and the results commit are made with--no-verify. A base that moved since the run is a warning naming the accepted assets it changed. The manifest and the comparison stay on the branch.remove_results_worktreeremoves the worktree and keeps the branch, or deletes it withdelete_branch = true.
A driver defines top-level constants meant for a process of its own; run_isolated includes it in a module of its own, so one regeneration script can run every driver in one warm session.
src/reproduce/reproduce_index.jl, src/reproduce/number_macros.jl and src/reproduce/paper_assets.jl depend on Base only (and the stdlib FileWatching, whose pid lock serializes the writers of a shared numbers file; number_macros.jl brings the package's one atomic write, src/write_atomic.jl, with it), so an environment that cannot load this package can include them by path, in that order (src/reproduce/results_branches.jl is not among them: it runs git and reads TOML).
One command: the runner
A paper project reproduces every result with one command: its stage graph of record (which launchers run which drivers on which configs of record, after what) run on a site (a machine: Slurm or local). The project keeps the graph (configs/reproduce.toml) and a one-line launcher per machine; the runner, the site profiles and their schema live here.
# from the repository root; a bare julia: the runner needs only the stdlibs
julia --startup-file=no -e 'include(ARGS[1]); include(ARGS[2]); exit(reproduce_main(ARGS[3:end]))' \
<Eval>/src/cli.jl <Eval>/src/reproduce/runner.jl \
--graph=<project>/configs/reproduce.toml --site=vcai [--only=a,b] [--no-deps] [--dry-run]The graph ([project], [sites.<site>], one [[stage]] per stage, [results_branch]; load_stage_graph documents every key):
[project]
name = "experiments"
workdir = "../.." # where the jobs are submitted from, relative to this file
julia_project = "experiments"
code = ["src", "experiments/src", "experiments/scripts"] # the project's code: its trees identify a run (below)
[sites.vcai] # where this project's data and results are on that machine
data_root = "/cluster/projects/me/paper-data"
results_root = "/cluster/projects/me/paper-results"
[[stage]]
name = "setup" # the environment: instantiated inside the scheduler
sbatch = "experiments/scripts/slurm/setup.sbatch"
gpu = true
[[stage]]
name = "refinement_chrome"
after = ["setup"]
sbatch = "experiments/scripts/slurm/refinement_chrome.sbatch" # the project's own launcher, submitted as is
driver = "experiments/scripts/refinement_chrome.jl"
config = "experiments/configs/experiments/refinement_chrome.toml"
gpu = true
array = { from = "cases", select_env = "CHROME_CASES", max_time = "pipeline.lm.max_time", time_margin_s = 10800 }
[[stage]]
name = "results_branch"
after = ["refinement_chrome"]
results_branch = true
[results_branch]
project = "experiments"
assets = [{ from = "plots/*.png", to = "paper/src/fig", submitted_only = true }]Selection. No flags: every
defaultstage.--only=a,b: those stages and, transitively, every stage they run after; with--no-deps, those alone (select_stages).--dry-runprints the job graph — per stage its items, dependencies, run key, the driver's own command line and the command the backend runs — and the missing inputs, and runs nothing (print_plan).Inputs first. Every
[inputs.<id>](tier,pathunder the data root) of every selected stage's configs is checked before anything runs; a missing one is a rowpath tier used by, the launch runs nothing and exits 3 (missing_inputs). An input that carriessha256(of a file, or a directory'stree_sha256) must have that content: another one is refused, naming the input, the expected and the actual hash (exit 2, the dry run too); without it the input is only checked to be there. Hashing streams the file and is cached by path, size, times and inode (file_sha256), so a large input is read once per process; and a large file's hash is kept across processes in the digest cache, so the many jobs of a launch read it once: see The digest cache below. A driver checks its config's inputs the same way withverify_inputs.Tier roots. An input's
pathis relative to its tier's root, which is the data root unless the machine puts that tier elsewhere: first the environment's<data_env>_<TIER>(the graph'sdata_env, defaultVGTK_DATA_ROOT:blobcalib_202609→VGTK_DATA_ROOT_BLOBCALIB_202609), then the graph's site table, then the data root, or where an artifact view mounts a tier the project's catalog lists (tier_data_root). A config never names a machine path; the machine's paths are its site's:[sites.vcai] data_root = "/cluster/projects/me/paper-data" results_root = "/cluster/projects/me/paper-results" tiers = { blobcalib_202609 = "/cluster/projects/me/datasets/202609_BlobCalib" }The runner gives its jobs the same roots (
<data_env>_<TIER>,VGTK_SITE), and a driver resolves one withdata_root(; tier), reading the graph with the runner's own reader. A tier root that is named but missing is reported as such.Directory inputs. A directory's
sha256is its tree digest,tree_sha256: the SHA-256 of its sorted(relative path, file sha256)lines, withsha256_excludeglobs left out; it is streamed and cached per file.The digest cache. The hash of every file of 64 MiB or more is kept across processes, in a small file per path under
digest_cache_dir(VGTK_DIGEST_CACHE, else$XDG_CACHE_HOME/vgtk/input-digests, else~/.cache/vgtk/input-digests; never in the data tree), with the file's size, mtime, ctime and inode. An unchanged file costs astat, so a directory input costs astatwalk; a changed file is rehashed and its entry rewritten (to a temporary file, then renamed: concurrent jobs are safe), and in a tree only the changed files are read again. A file changed in the last 3 s is not kept (a same-size rewrite within a coarse file system's time tick keeps its stat). A pin that the digest does not match is refused as before. The default is on: the launch's input check warms the cache for every job.VGTK_DIGEST_CACHE=offhashes every input in full, for a release gate or a run of record that wants the full verification. A site sets the directory in its profile's[env], which the launch's check and every stage use (an absolute path). The cache is per user: whoever can write it can make a changed input match its pin, so it is created private (0700), an entry another user owns is ignored, and it is never shared.Producers. A step that produces input data is produced under the results root and promoted,
promote_run(product_dir(product), tier), to<tier root>/<tier>/<name>, so its consumers'[inputs.<id>] path = "<tier>/<name>"finds it on every machine. A path in a tier that no config declares istier_path(tier, parts...).Unpublished inputs. Data of record that no artifact publishes (a capture kept on one machine) takes the reserved tier
unpublished(UNPUBLISHED_TIER): nothing fetches it, the dry run lists each such input apart, present or ABSENT, and a missing one is reported as never fetched. Its root resolves like any tier's (VGTK_DATA_ROOT_UNPUBLISHED,[sites.<site>.tiers]).Scratch outputs are no inputs. A run of
VGTK_SCRATCH=1recordsscratch = trueand keys apart from the runs of record; its output is refused as an input of record, an upstream run of record and a promoted run.One item upstream. An
afterentry"<stage>:<item>"waits on that item of an array stage alone: on a Slurm site the job runs after that array task (--dependency=afterok:<job>_<task>), not after the whole array (a texture recovery after the refinement'stexcalcase only). A local site runs the stages in order as before. The key folds the upstream stage's key either way: one key names every item's run, so a rerun of the stage reruns what waits on any of its items.Arrays.
array = { items = [...] }orarray = { from = "<dotted key>" }makes a stage an array, one task per item: the listed items, or the values at that key of the stage's config — a table's keys sorted by name (from = "cases",--timefrom each case'smax_time), a list of strings in its order (from = "grid.sets").select_envnames the variable that selects a subset and that the launcher reads.each = { from, env }loops inside each task: the task runs its command once per value atfromof its item's config, in order, withenvset to the value, and fails at the first value that fails (the same variable set in the launch's environment selects values). A grid of sets × variants is one task per set, each running every variant:[[stage]] name = "chrome_index_grid_index" sbatch = "experiments/scripts/slurm/chrome_index_grid_index.sbatch" config = "experiments/configs/experiments/chrome_index_grid.toml" array = { from = "grid.sets", select_env = "SETS", each = { from = "variants", env = "VARIANT" } }The values are part of the stage's key, so another selection is another run.
Backends.
slurm: each stage is onesbatchof the project's launcher (an array over its items,--timefrom the cases'max_time), after the jobs of the stages it depends on (--dependency=afterok), with the site's options over the launcher's#SBATCHlines; acommandstage is wrapped. Every Slurm stage is submitted through a wrapper the runner writes (with the launcher's#SBATCHdirectives and a copy of the launcher as submitted), which runs the stage and then writes each task's completion marker (below). A site's[slurm] job_name_prefix, or--job-name-prefix=<prefix>for one launch, is put before every job's name (the stage'sjob_name, else the launcher's#SBATCH --job-name, else<project>-<stage>): a label for the scheduler — a release run marked for a priority watcher — that changes no run key.local: every stage here in order, the launcher underbashwith the Slurm variables it reads (SLURM_ARRAY_TASK_ID,SLURM_SUBMIT_DIR,SLURM_CPUS_PER_TASK) and the site'sCUDA_VISIBLE_DEVICES, its output inlogs/<project>_<stage>[_<task>].log.Run manifest, reuse. Every launch writes
<results root>/<manifest_dir>/<UTC stamp>-<commit>.toml: the commit, the configs (sha256), the data and results roots, the site, the code trees (below), and per stage its run key, command, job id, state and code trees. A stage's run key hashes its launcher, driver and configs, its items, environment and command, and the run keys of the stages it runs after (of their last completed runs, for those a--no-depslaunch does not run: the project's own launches, in the order of the launch stamps that name their manifests, never by modification time), never the code and never its CPU count (--cpus-per-task,SLURM_CPUS_PER_TASK: the results are the same bits at any thread count, so a stage run at 8 CPUs is reused by a launch at 2; its device,gresandCUDA_VISIBLE_DEVICES, is keyed); a later launch reuses a stage of the same key that completed (and depends on one still in flight, by its job id) instead of running it again, while its code trees are the current ones and no stage it runs after is run again by that launch. A changed upstream stage thus reruns every stage downstream of it.Completion markers. Each task writes
<manifest without .toml>/<stage>/task-<t>.tomlwhen it ends — its state (completedorfailed), exit status, start and end, job id, commit and run key — atomically: a temporary file in that directory renamed into place (stage_markers_dir). Whether a stage finished is read from these markers and the queue (squeue), never from Slurm accounting, which a cluster may not keep (stage_run_state): completed when every task's marker says so (reused), in flight while its job is queued (depended on), failed by a marker (run again), and unknown when a task left no marker and its job is gone — a node failure, a kill past the time limit's grace — which is never taken as finished (run again). ATERM(scancel, the time limit) reaches the stage and is recorded as a failed marker; a task Slurm requeues under the same job id is in flight while the queue holds it, whatever its earlier attempt's marker says. A signal the launcher's own#SBATCH --signalasks for is forwarded to it and recorded. Ifsqueuefails for any reason other than an unknown job, the launch stops before submitting anything.Status.
reproduce_main(["status", …])reads the last manifest back: per stage its state from the markers and the queue, with its tasks counted by state, the log of its first failed or unknown task and that log's first error line; wheresacctworks, a task without a marker gets its accounting state beside it, as information only (reproduce_status).Results branch. A
results_branch = truestage runsresults_branch_of_runin the project's environment: the renders of[results_branch] assetson aresults_branch, its RUN.toml carrying the launch's commit, run keys, site, Slurm jobs and the data tiers (artifacts: an Artifacts.toml's git-tree-sha1s).
Code identity: when a stored result is reused
A result's run key names its configuration only, as DrWatson's savename and produce_or_load do: the runner's stage key, and a driver's run key (Experiments.run_key). The code a result was computed with is recorded beside it, by DrWatson's tag! (gitcommit, and gitpatch when the tree was dirty), together with the git tree hashes of that code (code_trees):
- the project's code paths: the graph's
codefor the runner,srcof the project for a driver's records (git rev-parse HEAD:<path>); - every stack package the project's environment loads (its
[deps]and what they reach in its Manifest, a workspace's included): a package loaded from a path ([sources],Pkg.develop) by the trees of itssrc,extandProject.tomlin its own repository, a released one by its Manifestgit-tree-sha1. - a project that is itself a registered install (the package's source in the depot, no git checkout: a registered package's example recording its results with
gitpath = pkgdir(ThePackage)): by the tree hash the active environment records for it (Pkg.dependencies()), with its name, version and uuid. It is committed code, content-addressed and immutable, and the stack packages are then those of the active project's environment.
A tree moves only when a file under it changes. A commit that touches only the docs, the tests or the paper changes no tree, so it reuses every result; a commit that changes src of the project or of a package it loads reruns what it computed.
The rule is one function, code_tree_changes(recorded, current): a stored result is reused only when it returns nothing. Every consumer goes through it — the runner's stage reuse and produced (a driver's keyed product, DrWatson's produce_or_load) — and each change it returns names the tree:
setup rerun: code changed: VisualGeometryToolkitCore: src 3fa29c01e2 → b71d0e44a9- A run of record runs committed code. Code whose files differ from its commit (an edit, a staged, deleted or untracked file in the code paths or in a package loaded from a path), or code that is in no git checkout and is no registered install, is refused with one error,
refuse_uncommitted_code: by a launch (exit 2; the dry run marks itDIRTY) and by a driver's records (write_provenance,write_run_records,produce). Scratch work setsVGTK_SCRATCH=1(scratch_run): its results are recorded with a warning, their trees marked+uncommitted, and never reused (the mark names HEAD, not the edits). A dirty paper or docs tree is no code and is not refused. A launch that identifies none of the project's own code (nosrc, no[project] code) is refused. - Results from before the trees were kept record none, and run once more; from then on they carry them. A launch recorded before the completion markers existed likewise reruns its completed Slurm stages once.
- A stage still in flight under other code stops the launch (exit 4): a second job would write the same results. Wait for it, or
scancelit.
For a paper project (RadiometricCamera's experiments, CaliBlobs' icra2026, BlobBoards' bmvc2026, RANSACScoringDoneRight, LocalOptimizationDoneRight) this needs nothing new: its drivers' products (produce, produced) record and compare the trees, and its graph's code lists the project's own code. List the code a stage runs there, not the docs: a path in code that holds the paper reruns every stage on a text edit.
A stage may set its own resources over the site's: cpus_per_task, mem_per_cpu and gres in its [[stage]] (a gres makes it a GPU stage). nice = <n> is its Slurm --nice, so a research stage yields to a release gate's jobs; like the CPU count and a job-name prefix it is a label for the scheduler and keys no run (the local backend runs every stage alike).
Phases: no GPU held for CPU work. A stage's phases splits it into jobs run one after another, each the stage's launcher (or the phase's own sbatch or command) with the phase's environment (and VGTK_PHASE=<name>), device (gpu = true: the stage's or the site's gres; a CPU phase none) and resources (cpus_per_task, mem_per_cpu, gres, time). A phase of an array stage is an array over the same items (per_item, default true), task k of the next phase running after its task k (--dependency=aftercorr); a phase with per_item = false is one task (afterok). The last phase is the stage's job (its completion markers); each earlier one writes its markers under <stage markers>/<phase>/, and status names a failed one as the cause of the stage that never starts. A phase's command and environment key the stage like its own; its scheduler labels (CPUs, nice) do not.
[[stage]] # GPU drivers, then the CPU numbers, verify and deploy
name = "regenerate_paper"
sbatch = "experiments/scripts/slurm/regenerate_paper.sbatch"
phases = [{ name = "drivers", gpu = true, env = { REGENERATE_PHASE = "drivers" } },
{ name = "numbers", env = { REGENERATE_PHASE = "numbers" } }]
[[stage]] # per set: decode on the CPU, then detect on a GPU (task k after task k)
name = "chrome_index_grid"
array = { from = "grid.sets", select_env = "SETS" }
phases = [{ name = "decode" }, { name = "detect", gpu = true }]On a Slurm site a stage with a GPU phase (or a GPU stage) gets one phase more, first: precompile, a CPU-only job (the stage's CPUs, memory and nice, JULIA_NUM_PRECOMPILE_TASKS its CPUs, the site's precompile_time) running julia --project=<julia_project> <julia_flags> -e 'using Pkg; Pkg.precompile()', so a GPU task only loads and runs. julia_flags are the flags the stage's jobs run julia with (julia_flags = ["--depwarn=error"]): a cache compiled under other flags is compiled again. precompile = false leaves it out (a stage whose launcher runs no julia). The precompile phase keys nothing: it is environment state (the depot's compile cache), not a result, so a fresh depot compiles again whatever an earlier launch did; the site's precompile_time is recorded in the manifest as a measurement. The dry run prints every phase (phase <name>:); the manifest records their job ids (phase_jobs). A local site runs the declared phases in order, each whole, and no precompile.
A project's bootstrap, scripts/reproduce/reproduce.sh, is generated, never written by hand: Reproduce.write_reproduce_script(repo_root; julia_project = "experiments") writes the one template shipped here (REPRODUCE_SCRIPT_TEMPLATE), and the project's suite checks its copy with reproduce_script. A driver reads an input of its config with input_path(cfg, id) (its tier's root, its sha256 checked).
src/reproduce/runner.jl depends on the stdlibs only (TOML, SHA, Dates; VisualGeometryToolkitCore's Base-only src/base/canonical.jl, the run keys' encoding, which the bootstrap includes by path before src/cli.jl; and src/cli.jl, included before it, which brings src/code_identity.jl), so a launcher runs it by path from a bare julia before the project's environment exists; the setup stage then instantiates the environment inside the scheduler.
Adding your machine
The shipped sites are configs/sites/desktop.toml (the owner's workstation: one RTX 4090, no scheduler, device 0; its [storage] puts every project's data and results and the artifact store under one tree of the home, data/vgtk/) and configs/sites/vcai.toml (the vcai-eaic-mds Slurm cluster: every job sources the project space's job environment; its [storage] puts every project's data and results and the artifact store in the NFS project space). The earlier dgx01 profile is retired: a project that launched there names a profile file of its own. For another machine:
- Copy
configs/sites/_template.toml, where every key is explained:[site]backend(slurmorlocal) and asetupshell snippet (module loads, an activated environment);[slurm]partition,gres(a GPU stage's),cpus_per_task,mem_per_cpu,gpu_limit,extraoptions,job_name_prefix;[local]cuda_visible_devices,cpus_per_task;[storage]data_root,results_root({project}the project's name; a path absolute or from the home,~/…,$HOME/…),artifact_storeandrefuse_depot(whether the depot may serve as the artifact store);[env]variables every stage gets (JULIA,JULIA_DEPOT_PATH, …). - Keep it with the project (the graph's
[project] sites_dir) or anywhere, and give--site=<name>or--site=path/to/mine.toml. - Say where the projects' data and results live on it: the profile's
[storage](the one place for a machine's paths; a project's own graph[sites.<name>] data_root/results_rootis still read before it, until the projects carry no machine paths), orVGTK_DATA_ROOT/VGTK_RESULTS_ROOT, or--data-root/--results-root. The jobs getVGTK_ARTIFACT_STOREfrom the profile'sartifact_store. - Check it with
--dry-run: the commands, the roots and the missing inputs, with nothing run.
A site holds no project's paths, and a project holds no machine's resources.