API: Experiments
The stage runner, run records, provenance and the data layer of a paper project.
Public names
VisualGeometryToolkitEval.Experiments — Module
ExperimentsThe paper-generic experiment-codification framework: the Stage table + run_stages entrypoint (--plan/--stage/--force), the run_driver shell-out and its in-process form run_isolated, require_input declarations, the keyed product (Product, produce: DrWatson's produce_or_load), the run records (the configuration-of-record layer), the executable codification contract (contract_violations) and the one data layer of every paper project: the root ladder and artifact view (set_project_root!, data_root, ArtifactView), the input write guard (assert_not_input), the provenance record (write_provenance, read_provenance) and the data policy (data_policy_violations).
This module holds what every paper uses unchanged: the runner mechanics and the contract rules. A paper's own decisions stay in that paper's project: its stage rosters and input paths, its driver wiring (set with set_driver_context!), its artifact destinations, splits and oracle constants.
VisualGeometryToolkitEval.Experiments.RUN_CONFIG_KEYS — Constant
RUN_CONFIG_KEYS :: Vector{Pair{String,Tuple}}Allowed keys of the run-config sections whose vocabulary THIS package defines, as dotted paths ("render.main" is the nested table). A section absent here belongs to the PROJECT that ships the config and is not key-checked — [source], [extract], [pool], [arm], [ransac] and the rest are read ad hoc by its drivers, and [source.recon_special] is keyed by scene name, so a fixed vocabulary would be wrong. [tuning.<name>] has its own, stricter check (Tuning _read_struct → VisualGeometryToolkitCore.check_keys, driven by the struct fields) and is therefore not restated here.
Nested tables are listed in their own right: a check on [render] alone sees main as one known key and nothing inside it.
VisualGeometryToolkitEval.Experiments.RUN_KEY_EXCLUDED — Constant
Top-level config sections that never touch a run's data and so never key it.
VisualGeometryToolkitEval.Experiments.ArtifactView — Type
ArtifactView(toml; dir, mount = name -> "", default = String[])A project's published input data as Julia artifacts: toml is the Artifacts.toml binding them (lazy, Zenodo-hosted tarballs), dir the view directory their trees are linked into (default <project>/data/exp_raw, so the view IS the project's DrWatson data directory), mount(name) the subdirectory of dir artifact name mounts at (trees overlap directories, so the view is per-file links), and default the artifacts data_root installs when asked for none in particular.
VisualGeometryToolkitEval.Experiments.DataPolicyAllow — Type
DataPolicyAllow(rule, file, match)One allowlisted (deferred) violation: rule "D1"…"D5", file relative to the scanned project root, and a substring match of the offending line. Each entry should name what fixes it, in a comment beside it.
VisualGeometryToolkitEval.Experiments.Produced — Type
Produced(record = nothing; inputs = String[], kwargs...)What a product's f returns to produce when it says more than its record: the record (saved by produceorload through DrWatson._wsave(file, ::Produced), atomically: a .jld2 output as JLD2's object "record", which takes no keywords; any other through VisualGeometryToolkitIO.save(file, record; kwargs...) — a camera's name, a length_unit, a provenance), or nothing when f wrote the product's output itself; and the inputs it read (paths, recorded in its provenance beside produce's own inputs). A bare record returned is Produced(record).
VisualGeometryToolkitEval.Experiments.Product — Type
Product(experiment, name, record; output, files = nothing, upstream = nothing, parts = nothing,
code = current_code_trees(project_root()))The keyed product of one step of a pipeline of experiment (a name, or a relative path of segments for a project whose results are laid out deeper: "ETH/sift/homography/fig4"): its name ("prepare", "start", "refine", …), the record it is keyed by (ONLY the config fields and inputs the product reads), its key and its output, the file under the product's directory (product_dir) that holds its result — the record produce saves, or the file a product that writes its own outputs writes last.
The key is THE dependency tracking of a pipeline (run_key(record; files, upstream, parts, code)): the record (a top-level section that a run key excludes, RUN_KEY_EXCLUDED's render, is refused in it: it would key nothing; a product that reads it keys it under another name); the products it reads (upstream: a Product, whose key AND its output's content hash, recorded when it was produced, are folded in — so an upstream produced again with another output re-keys this one, and produced checks it again; or a bare key, folded as it is: no output hash, so a bare key is not re-keyed when its product is produced again — give the Product); the content of the files it reads (files; a config's pinned inputs belong in its record, by their sha256); a choice made on a command line (parts); and the CODE it runs (code): the source-tree hash of every stack package its environment loads and the project's own src (current_code_trees: a path checkout's git trees of src, ext, Project.toml, a registered version's Manifest tree hash). A change of any of them is another key, so another directory, produced anew; an unchanged one is the same product, loaded. A change of a package re-keys every product that loads it (conservative). A product whose computation runs only some of the environment's packages is keyed by those alone — the per-package rule: code restricted to the packages its step uses (a step that runs an external tool adds the tool's pinned image to its inputs), never a driver's own source — so a commit elsewhere leaves it produced. A result of an earlier key stays where it is.
VisualGeometryToolkitEval.Experiments.Stage — Type
Stage(name::Symbol, desc::String; items, product, run)
Stage(name::Symbol, desc::String; check)One stage of a codified experiment. A stage that PRODUCES declares its items (items(): the cases, sets or names it treats — cheap: no GPU, no heavy loads), each item's keyed product (product(item), an experiment's Product) and its runner (run(item), which calls produce: a produced product loads, a pending one is produced through DrWatson's produce_or_load), so run is called for every item of the stage and a stage whose products are all produced costs their checks only. A stage that only CHECKS (its inputs present, a refusal, a deploy) declares check(), run every time. Declared in figure/table order and executed by run_stages.
The done-closure form Stage(name, desc, done, run) is retired: a product's own record says whether it is produced.
VisualGeometryToolkitEval.Experiments.TagRename — Type
TagRename(path::Regex, old, new)One tag rename: a string VALUE old whose dotted key path matches path becomes new. The path of a value is its keys joined by ., an array element adding [i] ("selectors[1]", "geometry.local_optimization_strategy.type"), so a rename can be scoped to the family that owns the tag: "fixed" is one tag in nuisance.scale and another in window.type.
VisualGeometryToolkitEval.Experiments.active_site — Method
active_site() -> Union{Nothing, Site}The site profile ENV["VGTK_SITE"] names (a shipped site's name, a profile file's path, or a name in the bound project's own sites directory, the stage graph's [project] sites_dir), read by the runner's load_site; nothing when VGTK_SITE is unset or empty, or names no profile (a site a project's stage graph alone knows, [sites.<site>]).
VisualGeometryToolkitEval.Experiments.artifact_view! — Method
artifact_view!(view, names) -> StringInstall the artifacts names of view (VisualGeometryToolkitCore's install_artifact) and link each tree into the view directory at its mount (link_tree!), marking the directory .artifact_view. Idempotent; returns the view directory.
VisualGeometryToolkitEval.Experiments.assert_active_project — Method
assert_active_project(marker) -> nothingThat DrWatson's projectdir() is the paper project the driver belongs to: the file marker (relative to the project, e.g. "src/BlobBoardsEval.jl") exists there. projectdir() resolves from the ACTIVE project, which survives any file move but is right only when the paper project is the active one; a driver run under the parent package's project fails here, naming both, instead of resolving its paths into the wrong tree.
VisualGeometryToolkitEval.Experiments.assert_not_input — Method
assert_not_input(path; producer = false)The runtime guard of every writer here: refuse a write at or under data_root — the read-only input tree — unless the caller declares itself a producer (a pool generator writing a NEW tree under the data root). A results root nested inside the data root (one author binding has data_root()/results/…) is not input. Nothing is guarded when no data root is bound. A tier root set elsewhere (VGTK_DATA_ROOT_<TIER>, data_root's tier) is input too.
VisualGeometryToolkitEval.Experiments.assert_primary_copy — Method
assert_primary_copy(what)Producers that write INTO the data tree (calibration generators, converters, packagers) call this first: an artifact view is read-only.
VisualGeometryToolkitEval.Experiments.cached_result — Method
Retired: produce(f, product) — DrWatson's produceorload on a keyed Product (a .jld2 output's record saved as JLD2's object "record"), read back with load_output.
VisualGeometryToolkitEval.Experiments.cancelled — Method
cancelled() -> BoolWhether a kaimon cancel_eval was requested for the running evaluation (Main.KaimonGate.is_cancelled()); false outside a kaimon session. Poll it from long loops, e.g. live_runs(...; cancelled = Experiments.cancelled).
VisualGeometryToolkitEval.Experiments.commit_drift — Method
commit_drift(prov; gitpath = project_root()) -> Union{Nothing, Tuple{String,String}}(recorded, current) when the commit a provenance record names is not the current one of gitpath, nothing when they agree or either is unknown. Compared as COMMITS, through DrWatson's gitdescribe(gitpath; dirty_suffix = ""): a describe string with an annotated tag (v0.7.0-12-g89f65bd), a bare sha and an exact tag all name their commit, and a dirty suffix is not a different commit.
VisualGeometryToolkitEval.Experiments.config_of_record — Method
config_of_record(name; root = project_root()) -> DictThe config of record name of the project at root (configs/<name>.toml), loaded as a run config (load_run_config: its [experiment] checked, unknown keys of the sections this package owns refused), not yet folded.
VisualGeometryToolkitEval.Experiments.contract_violations — Method
contract_violations(file) -> Vector{String}The codification-contract check for one experiment file: returns a human-readable violation per broken rule (empty ⇒ compliant). Support files (registry.jl and *common.jl, e.g. a paper's shared external-arm plumbing) are exempt from R3; *_worker.jl subroutines are exempt from its stage-runner requirement (they are driven by their parent experiment) but still carry a provenance line.
VisualGeometryToolkitEval.Experiments.current_code_trees — Function
current_code_trees(gitpath = project_root()) -> Dict{String,String}The code trees a result computed now records (code_trees of the top module): the project's src (of gitpath) and every stack package its environment (gitpath's, or the Manifest above it) loads. write_provenance stores them; a Product folds them into its key, so a product of other code is another product.
VisualGeometryToolkitEval.Experiments.data_policy_violations — Method
data_policy_violations(root; dirs = ["src", "scripts"], exts = [".jl", ".sh", ".sbatch"],
allow = DataPolicyAllow[])
-> Vector{NamedTuple{(:rule, :file, :line, :text)}}The data-policy findings (rules D1–D5, see the file header) in the dirs of project root (code, the exts files: every rule, comments skipped; their other text files: D1), and D1 also in its tests, examples, configs, docs and provenance (every line of their code, config and text files; a site profile, which holds a machine's paths by design, excepted) and in every Markdown file of the root and of data/ (README.md, REPRODUCE.md, data/README.md, …). In a git work tree the files git ignores are not read. The result is minus the allowed ones, plus a STALE finding for every allow entry that matched nothing. Empty ⇔ compliant.
VisualGeometryToolkitEval.Experiments.data_root — Method
data_root(; tiers = nothing, tier = nothing) -> StringThe project's input tree (read-only): ENV["VGTK_DATA_ROOT"] → the project's data/exp_raw (DrWatson datadir("exp_raw"); a link, a directory, or the artifact view) → the stage graph's [sites.<site>] data_root once it exists → the site profile's data_root once it exists (active_site, its {project} the project_name) → the project's fallback rung → its ArtifactView, installing tiers (default: the view's default). Configs state input paths RELATIVE to it; nothing under it is written except by a declared producer (assert_not_input).
With tier, the root of that input tier, which may live elsewhere on a machine (tier_data_root, the runner's order): ENV["<data_env>_<TIER>"] (the graph's data_env, default VGTK_DATA_ROOT) → the project's stage graph (configs/reproduce.toml, read by the runner's load_stage_graph) [sites.<site>.tiers], <site> from ENV["VGTK_SITE"] (the runner sets both for its jobs) → the one data root above, through the same resolver and the runner's view gate (catalog_view_mount): a tier the project's catalog (configs/data_artifacts.toml) lists is the view's, its root where the view mounts it (<data root>/<mount_dir>, a split entry's parts included), installed when the view's Artifacts.toml binds it; a tier the catalog does not list is only looked up. An [inputs.<id>] path is relative to its tier's root, so input_path fetches what it reads.
VisualGeometryToolkitEval.Experiments.experiment_dir — Method
Retired: a project's deeper layout is a Product's experiment path, Product(joinpath(dataset, matcher, problem, experiment), name, record; output).
VisualGeometryToolkitEval.Experiments.fresher — Method
Retired: a stage's done-predicate is its products' own record — Stage(name, desc; items, product, run), produced(product).
VisualGeometryToolkitEval.Experiments.hand_eye_mount_file — Method
hand_eye_mount_file(file) -> StringThe marker-mount record beside a hand-eye record file: <stem>_board_rig<ext>.
VisualGeometryToolkitEval.Experiments.input_path — Method
input_path(cfg, id; root = nothing) -> StringThe absolute path of the config's input [inputs.<id>]: its path under its tier's root (data_root(; tier): VGTK_DATA_ROOT_<TIER>, the stage graph's [sites.<site>.tiers], the results root for a tier the catalog mounts there (mount = "results": a run of record published as a tier), the data root), or under root when given; checked to be there and, with a sha256, to have that content (streamed, cached). The one reader a driver opens an input with; an ArgumentError for an id the config does not declare, a missing input or another content.
An input may instead name ANOTHER project's keyed product — its run of record, read where it lives, never copied:
[inputs.hand_eye]
project = "bmvc2026" # the producing project (its stage graph's [project] name)
product = "calibration/hand_eye" # the Product's <experiment>/<name>
key = "3f9a0c1d2e4b5a67" # its run key of record (the product's directory)
file = "hand_eye_bb.json" # optional: a file in the product's directory (default: the directory)
sha256 = "…" # its content (a directory's: tree_sha256)
tier = "bmvc2026_results" # optional: a results-mount tier of this project's catalog publishing themIt is read under the first root that holds <product>/<key>[/<file>]: VGTK_RESULTS_ROOT_<PROJECT>, then project_results_root (the active site profile's results_root for that project), then the tier's root (the fresh-machine form: the producer's results published as a tier, fetched by DataArtifacts); its record's run_key must be key (VisualGeometryToolkitEval's product_input, the runner's check too). With root, that root alone is the product's results root.
VisualGeometryToolkitEval.Experiments.is_artifact_view — Method
Whether dir is an artifact view (a directory carrying the .artifact_view marker).
VisualGeometryToolkitEval.Experiments.is_primary_copy — Method
is_primary_copy() -> BoolWhether the data root is a writable primary copy (bound by ENV, the project's data/exp_raw or its fallback) rather than an artifact view, whose trees are content-addressed and read-only. Same resolution as data_root.
VisualGeometryToolkitEval.Experiments.keyed_cell_dir — Method
Retired: product_dir(Product(experiment, name, record; output)) — DrWatson's datadir("exp_pro", experiment, name, key).
VisualGeometryToolkitEval.Experiments.keyed_data_dir — Method
Retired: a producer's product is a Product under the results root, made an input of record by promote_run(product_dir(s), tier).
VisualGeometryToolkitEval.Experiments.legacy_hash_key — Method
Retired: every key is run_key (SHA-256 of the canonical JSON), a Product's.
VisualGeometryToolkitEval.Experiments.link_tree! — Method
link_tree!(tree, dst) -> IntLink every file of tree into dst (per-file symlinks: artifact trees share directories, so directory links would not compose); an existing path — a link made earlier or an author's own file — is left alone. Returns the number of new links.
VisualGeometryToolkitEval.Experiments.load_hand_eye — Method
load_hand_eye(file) -> (X, mount)The hand-eye record pair save_hand_eye writes: X, the camera-mount pose body:camera_rig → ViewFrame{:optical}, and mount, the marker mount as a Rig{BodyFrame{:board_rig}}, both in plain millimetres, each pose Uncertain where the record states its covariance.
A record without the mount file — the 2026-09 hand-eye record, one vgio/transforms/v2 file that held the marker mount too, which VisualGeometryToolkitIO reads as the current transforms record — takes its mount from the file's target:<id> → body:board_rig poses. save_hand_eye writes the pair.
VisualGeometryToolkitEval.Experiments.load_output — Method
load_output(T, s::Product; kwargs...)The product's record: a .jld2 output's JLD2 object "record" (converted to T: a Dict{String, Vector{Float64}} reads as a Dict{String, Any}), any other VisualGeometryToolkitIO.load(T, <product_dir(s)>/<output>; kwargs...).
VisualGeometryToolkitEval.Experiments.load_run_config — Method
load_run_config(path) -> DictParse a run config: it must carry [experiment] with dataset, matcher, problem and pool_dir — or, for a project whose results are not laid out by pool, a layout (an array of path segments: the project's Product experiment path) — and an unknown key in a section this package owns (RUN_CONFIG_KEYS) is refused.
VisualGeometryToolkitEval.Experiments.oracle_floor_done — Method
oracle_floor_done(path, tag; source) -> BoolWhether cell tag has its floor in the sidecar path AND the floor was computed on source (oracle_floor_source): every key of source must equal the recorded one. Presence of the key alone is not enough: floors computed on a retired pool tree must not read as done.
VisualGeometryToolkitEval.Experiments.oracle_floor_source — Method
oracle_floor_source(pool_dir, problem, scenes) -> Dict{String,Any}The record of what an oracle floor was computed on: the pool tree, the problem family and the (sorted) scenes. write_oracle_floor stores it as the floor's source; oracle_floor_done compares it with the current one.
VisualGeometryToolkitEval.Experiments.oracle_floors_note — Method
oracle_floors_note() -> StringThe note stored in the oracle-floor sidecar. The mAA threshold in it is the configured one (VisualGeometryToolkitEval.Stats.maa_threshold), not a literal.
VisualGeometryToolkitEval.Experiments.oracle_floors_path — Function
The pseudo-GT oracle-floor sidecar of a results tree (the floors' producer writes it): <root>/oracle_floors.jld2.
VisualGeometryToolkitEval.Experiments.produce — Method
produce(f, s::Product; driver, config_source, inputs = String[], producer = false, fields = Dict(),
force = false, gitpath = project_root()) -> StringThe product s reused or produced — produced through DrWatson's produce_or_load on the product's output (loadfile = false). When it is produced (and not force) nothing runs, for a file and a directory output alike; otherwise its old provenance is removed (a failed product never reads as produced), f(dir) runs in the product's directory and returns the product's record — saved by produceorload atomically, a .jld2 output as JLD2's object "record", any other through VisualGeometryToolkitIO.save — or nothing when f wrote s.output itself, or a Produced (a record or nothing, with the inputs f read and the save's keywords); the product's records are written last (write_run_records: _config.toml, the record it is keyed by; _provenance.toml with run_key, driver, config_source, the declared inputs and f's, fields, output_sha256 (the content hash of every file the product's directory holds but its records: what re-keys its consumers) and upstream_outputs (its upstream products' as it read them); producer for a product that writes under the data root). A save that fails is an error. Returns the product's directory; read its record with load_output.
VisualGeometryToolkitEval.Experiments.produced — Method
produced(s::Product) -> BoolWhether the product is produced and reused: its output and its _provenance.toml with its key are there (the key holds the code it runs), its scratch flag is this run's (scratch_run) and no code tree it recorded was uncommitted (a scratch run's +uncommitted mark names HEAD, not the edits: never reused).
VisualGeometryToolkitEval.Experiments.produced_dir — Method
produced_dir(s::Product) -> StringThe product's directory (product_dir) when it is produced under its current key; an error naming the product otherwise (its producing stage has not run since the key changed: a commit re-keys a product). What a check stage that reads a product's outputs (Reproduce's asset_stages) reads them from.
VisualGeometryToolkitEval.Experiments.product_dir — Method
product_dir(s::Product) -> StringThe product's directory: <results_root()>/<experiment>/<name>/<key>, DrWatson's datadir("exp_pro", experiment, name, key) (results_root() is the project's data/exp_pro, or its binding VGTK_RESULTS_ROOT).
VisualGeometryToolkitEval.Experiments.product_output — Method
The file of the product's result: <product_dir(s)>/<output>.
VisualGeometryToolkitEval.Experiments.progress — Method
progress(msg) -> nothingReport msg as the running evaluation's progress (Main.KaimonGate.progress(msg)); a no-op outside a kaimon session.
VisualGeometryToolkitEval.Experiments.project_name — Method
project_name() -> StringThe bound project's name, the {project} of a site profile's roots: its stage graph's [project] name (configs/reproduce.toml), else the name of its directory.
VisualGeometryToolkitEval.Experiments.project_results_root — Method
project_results_root(project) -> Union{Nothing, String}Where ANOTHER project's results live on this machine's site: the active site profile's results_root for that project (active_site, its {project} the project's name: the runner's site_root(site, "results_root", project)); nothing without one. A config's product input ([inputs.<id>] project = …, input_path) is read under it, after VGTK_RESULTS_ROOT_<PROJECT>.
VisualGeometryToolkitEval.Experiments.project_root — Method
The project the drivers run from: set_project_root!'s root, else DrWatson's projectdir().
VisualGeometryToolkitEval.Experiments.promote_run — Method
promote_run(run_dir, tier; name = basename(run_dir)) -> StringA completed run of record (a directory with its _provenance.toml, under the results root) made an INPUT of record: copied to <data_root(; tier)>/<tier>/<name> (the tier's root resolved as a consumer's), where an [inputs.<id>] table names it (tier, path = "<tier>/<name>") and a data artifact of that tier publishes it (an ArtifactView installs it like any tier). A run read by another run belongs there, not in the results root and not in a directory named by a date. Refuses a run without its record, and a destination that holds a different run (another run_key in its record; for a run without a run_key, another record), and a scratch run's output; the same run again is left as it is. Returns the destination.
VisualGeometryToolkitEval.Experiments.provenance_inputs — Method
provenance_inputs(prov) -> Vector{String}The declared inputs of a provenance record as absolute paths: relative entries joined to the roots recorded beside them; a retired record's absolute list as is.
VisualGeometryToolkitEval.Experiments.read_provenance — Method
read_provenance(dir) -> Dict{String,Any}The provenance record of dir as the one flat dict of write_provenance, whatever layout wrote it: the current one; the VisualGeometryToolkitEval ≤ 0.1 one (its _dirty.patch read back as gitpatch); bmvc2026's [provenance] + [config] tables (merged; created read as timestamp). Errors when the record is absent.
VisualGeometryToolkitEval.Experiments.read_rekey_map — Method
read_rekey_map(path) -> Dict{String,String}The old-key → new-key map of a written re-key file (moved keys only).
VisualGeometryToolkitEval.Experiments.rekey_map — Method
rekey_map(configs; renames, keyed_table, key) -> Vector{NamedTuple}For each config path in configs: keyed_table(path) is the table the project's key function hashes, computed with the current code, and key(table) that hash. Returns one row per config — config, old_key, new_key, moved and the applied substitutions. A config whose keyed_table throws is recorded with error set and no keys (it is not keyed by this function).
VisualGeometryToolkitEval.Experiments.rename_tags — Method
rename_tags(table, renames) -> (renamed, applied)A deep copy of table with every matching rename applied, and the list of path => (old, new) substitutions made. A value matched by two renames is an ArgumentError (the renames must be unambiguous).
VisualGeometryToolkitEval.Experiments.require_input — Method
require_input(path; what, provenance, sha256 = nothing)Declared data input of an experiment: hard error (with the provenance recipe) when absent. Inputs are declared at the top of each experiment file so the dependency graph is readable without running anything. The check is the runner's and verify_inputs' one, inspect_input (src/codeidentity.jl): with sha256 (the input's content hash of record, 64 hex digits) another content is refused (ArgumentError: what, the path, expected and actual) — a file's `filesha256, a directory'streesha256— and so is a scratch run's output, unless this run is one ([scratchrun`](@ref)).
VisualGeometryToolkitEval.Experiments.results_root — Method
results_root() -> StringThe project's results tree: ENV["VGTK_RESULTS_ROOT"] → the project's data/exp_pro (DrWatson datadir("exp_pro"); a link or a directory) → the stage graph's [sites.<site>] results_root, then the site profile's results_root (active_site), each once it exists or when the project binds create_results = true; without any of them data/exp_pro. The order is the runner's launch's too (Reproduce resolve_roots), one ladder. The root is created when the project binds create_results = true; otherwise a missing data/exp_pro is an error.
VisualGeometryToolkitEval.Experiments.run_complete — Method
Retired: produced(product) (a Product's output, its provenance and its code trees).
VisualGeometryToolkitEval.Experiments.run_driver — Method
run_driver(script, args...; driver_dir = default_driver_dir(),
project = default_driver_project(),
env = Dict(), threads = get(ENV, "SLURM_CPUS_PER_TASK", "auto"))Shell out to a paper driver in a fresh Julia process (the drivers parse ARGS at include time, so in-process include is not composable). A relative script resolves against driver_dir; the child runs --project=<project>. Errors propagate (run throws on nonzero exit).
VisualGeometryToolkitEval.Experiments.run_isolated — Method
run_isolated(path; args = String[], env = Dict{String,String}(),
bindings = Dict{Symbol,Any}()) -> ModuleInclude the script path in a fresh module of its own, in this process: a paper driver defines top-level constants meant for a process of its own, so several drivers included into one session collide, while a throwaway module gives each the clean top level it expects and keeps the loaded packages warm (no new process, no recompilation). While the script's top level runs, ARGS is args and the environment holds env (both restored afterwards, also on error); bindings are constants defined in the module first (a name the script uses unqualified). The module's exit throws instead of ending the session. Returns the module: the functions the script defined live in a newer world, so call them through Base.invokelatest. A separate process (run_driver) is the other way to run a driver.
VisualGeometryToolkitEval.Experiments.run_key — Method
run_key(cfg; files = nothing, upstream = nothing, parts = nothing) -> StringThe run identity: the SHA-256 (hex, first 16) of the effective config's canonical JSON (_content_key: VisualGeometryToolkitCore's canonical_json, the one content key behind tuning_hash and the runner's stage key too), with the RUN_KEY_EXCLUDED sections removed ([render] holds panel knobs of record, not experiment inputs) and every [tuning.<name>] entry resolved to its COMPLETE table (resolve_tuning!: a default of this package's configs/tuning.toml that the config inherits keys the run like a stated one). Key order does not move it, nor a number's spelling (3 and 3.0 key alike, as do TOML's hex 0x0b10b5eed and 2970312429), the Julia version does not move it, and a TOML round trip of the effective config reproduces it, so run_key(TOML.parsefile("_config.toml")) names the directory that holds it.
What a run depends on beyond its config is folded in by name, so a project keys it here instead of in a private hash:
files:name => path, each file's content hash (file_sha256: streamed, cached by path and stat) — a calibration file, a board definition;upstream:name => key, the run keys of the runs it reads (a pose run's key in the key of the evaluation that reads its poses), orname => run directory, read for its recordedrun_key(a scratch run's is refused);parts:name => value(aDictor aNamedTuple, TOML values), a choice of the run that is not in its config (a frame limit of a smoke run);code:package => tree, the code version of what the run loads (current_code_trees: a path checkout's git trees, a registered version's Manifest tree hash) — aProduct's key always folds it.
They are keyed as the table keyed_with (files, upstream, parts, code, each present when given; and scratch in a scratch_run, so its outputs never share a run of record's directory), which a config therefore may not hold; without them the key is the config's alone, as before. A value of record belongs in the config: parts is for a choice made on the command line of a driver.
VisualGeometryToolkitEval.Experiments.run_stages — Method
run_stages(experiment::AbstractString, stages; argv = ARGS)The uniform experiment entrypoint: --plan prints the stage table (each stage's produced products of its items, or check) and exits (the structural check); otherwise runs the stages in declaration order (--stage=NAME restricts to one, or to a comma-separated subset — declaration order is still honored). A producing stage calls its run on EVERY one of its items, which loads a produced product and produces a pending one; its items are listed when the stage runs, so a product an earlier stage re-keyed is the new one. --force unmarks each product of the selected stages (its provenance removed) just before its item runs, so each is produced again and a failure leaves the later ones as they were. A stage failure aborts the run (later stages depend on earlier ones).
VisualGeometryToolkitEval.Experiments.save_hand_eye — Method
save_hand_eye(file, X, Y; provenance = (;)) -> (file, mount_file)A hand-eye calibration as its two records: file, a vgio/transforms record of X (body:camera_rig → camera_optical), and hand_eye_mount_file(file), a vgio/rig record of the marker mount Y (target id → its pose target:<id> → body:board_rig, each target a pose-only PlanarTarget named by its id). X and Y's poses are plain millimetres (an Uncertain pose keeps its covariance); provenance goes on the transforms record.
VisualGeometryToolkitEval.Experiments.set_driver_context! — Method
set_driver_context!(driver_dir; project = driver_dir)Register the default script directory (and Julia project) run_driver resolves against — called once by a paper's experiments/common.jl shim.
VisualGeometryToolkitEval.Experiments.set_project_root! — Function
set_project_root!(root = projectdir(); resolve_links = false, data_fallback = nothing,
artifacts = nothing, create_results = false)Bind the project the drivers run from (its git record, its data/exp_raw and data/exp_pro). The keywords are the per-project rungs of the ladder, kept so that every existing path resolves byte-identically:
resolve_links: returnrealpathof thedata/exp_rawlink rather than the link path itself (bmvc2026: its run keys hash paths under it);data_fallback: a() -> path | nothingrung after the link (bmvc2026'sBlobBoards.data_dir()preference);artifacts: theArtifactViewof last resort;create_results: createdata/exp_prowhen nothing binds a results root (accv2026 errors instead).
set_project_root!("") unbinds (the root then follows DrWatson's projectdir()).
VisualGeometryToolkitEval.Experiments.site_profile — Method
site_profile(spec) -> Union{Nothing, Site}The site profile spec names, as active_site reads VGTK_SITE: the runner's load_site over the bound project's own sites directory and the shipped profiles; nothing for a name no profile has (a malformed profile is an error). VisualGeometryToolkitCore's artifact_store reads its [storage] artifact_store through it (registered when this package loads).
VisualGeometryToolkitEval.Experiments.tier_path — Method
tier_path(tier, parts...) -> StringA path in the input tier tier: joinpath(data_root(; tier), parts...), the tier's root resolved (and installed from an artifact view) as for an input. The one reader of a dataset's own layout (a capture set, tier_path(tier, set)) where a config does not name the path; a config's declared input is input_path's, which also checks its sha256.
VisualGeometryToolkitEval.Experiments.verify_inputs — Method
verify_inputs(cfg; root = nothing) -> Dict{String,String}The [inputs.<id>] of a config, checked as the runner checks them (missing_inputs): each path (relative to its tier's root, data_root(; tier), or to root for every input when given) is there, and one with a sha256 has that content (streamed, cached: file_sha256), through the one check the runner makes (inspect_input). A scratch run's output is refused as an input, except to a scratch run (scratch_run). Returns id => absolute path; an ArgumentError names every missing input (id, tier, path), every other content (id, path, expected and actual hash) and every scratch output.
VisualGeometryToolkitEval.Experiments.warn_commit_drift — Method
warn_commit_drift(prov, what; gitpath = project_root()) -> BoolSkip-if-complete's WARNING (code is not keyed; the record restores it): warn when prov was recorded under another commit (commit_drift) — re-run with force if that change was result-affecting. Returns whether it warned.
VisualGeometryToolkitEval.Experiments.write_oracle_floor — Method
write_oracle_floor(path, tag, floor, maa; source, npairs, driver) -> pathRecord the pool-best floor floor (median model error) and ceiling maa (mean pool-best mAA) of cell tag in the sidecar path (oracle_floors_path), which holds "floors" (tag => floor, tag * "_maa" => maa), "sources" (tag => source plus npairs, driver and the time computed) and "note" (oracle_floors_note). The other cells of the file are kept. The read-modify-write is serialized by a pid lock (path * ".lock"), because the oracle stages of concurrent chains write one file.
VisualGeometryToolkitEval.Experiments.write_provenance — Function
write_provenance(dir, fields = Dict(); driver, inputs = nothing, sensor_model = nothing,
gitpath = project_root(), producer = false) -> StringWrite <dir>/_provenance.toml: fields (the caller's record — run key, config source, network hash, frame counts, …) plus driver, timestamp, hostname, julia, and DrWatson's tag! record of gitpath (gitcommit, gitpatch on a dirty tree, with uncommitted_files — the paths that patch touches) and code_trees, the git trees of the code the run loaded (current_code_trees). inputs (paths) are recorded relative to the results or data root they live under, the roots beside them (inputs = {data = […], data_root = …}). Refuses to write into the data root unless producer (assert_not_input), records scratch (whether this is a scratch_run; a run of record refuses inputs that are a scratch run's output, and a directory already holding a record with the other flag is refused), and refuses a run of record of uncommitted code (refuse_uncommitted_code, the runner's refusal: an edited, staged, deleted or untracked file in the project's src or a package it loads from a path, or code in no git checkout that is no registered install); a scratch_run records it with a warning, its trees marked +uncommitted. sensor_model (a run's SensorModel, or one per frame) records where its black, noise and defects came from — VisualGeometryToolkitCore's component_sources, one table per distinct model with its frame count, under sensor_model — so a table can state "measured" or "estimated" per run. Returns the record's path.
VisualGeometryToolkitEval.Experiments.write_rekey_map — Method
write_rekey_map(path, rows; project, renames, commits = Dict()) -> pathWrite a rekey_map as TOML: the project, the renames, the commits it was computed at, and per config its old and new keys and the substitutions.
VisualGeometryToolkitEval.Experiments.write_run_records — Method
write_run_records(outdir, cfg; driver, config_source, key, inputs = String[], producer = false, fields = Dict(),
gitpath = project_root())Write the run's _config.toml (the effective config, its tuning specs resolved to their complete tables) and its provenance record (write_provenance: config_source, run_key, the declared inputs relative to their roots, and DrWatson's git record). producer = true is a pool generator's declaration that outdir is a NEW tree under the data root. fields is a further payload of the record (a replay's own counts, a detection CV's folds): written beside config_source and run_key, which it may not name. gitpath is the repository whose code the provenance records (DrWatson's tag! and the code trees).
Internals
These names are not public; they are listed for contributors.