Paper projects
Every paper of the stack is produced by a paper project. A paper project is a DrWatson project, not a package. Each of its figures, tables and numbers comes from a tracked driver that runs on a config of record and reads declared data, and each result has a provenance record beside it.
The worked example is the project accv2026 of the ACCV 2026 paper (Asian Conference on Computer Vision), kept in LocalOptimizationDoneRight, with its REPRODUCE.md. One check enforces the rules below, and a template starts a project that already satisfies them.
Layout
| Path | Holds |
|---|---|
Project.toml | a DrWatson project depending on DrWatson and VisualGeometryToolkitEval; Manifest.toml is tracked, as the environment of record |
configs/ | the configs of record (TOML), one per driver or cell |
scripts/ | the drivers, one file per paper asset or cell, plus scripts/tools/ for what they launch |
src/ | the shared prelude that the drivers include with include(srcdir(…)): project code, never a data constant |
test/runtests.jl | the suite; it calls paper_project_violations |
paper/ | the manuscript, with generated assets under paper/figures/ |
data/exp_raw, data/exp_pro | the per-machine bindings of the input and results trees (links or directories, ignored by git); data/sims for smoke runs |
plots/ | the drivers' renders (ignored by git); a deploy stage copies the paper's into the paper |
REPRODUCE.md | the asset index (see Reproducible figures) |
Artifacts.toml, configs/data_artifacts.toml | the published input data and their catalog |
Paths come from DrWatson (projectdir, srcdir, datadir, plotsdir) and from the data layer of Experiments, never from a literal machine path.
Rules
- Configs of record. Every value that a result depends on is in a TOML file under
configs/that the driver reads. A driver holds no data constant: no literal seed, scene or frame list, data path or numeric table. A justified constant, such as a published test vector, is marked# paper-project: allow C2 <reason>. - Run keys from configs. A result directory is named by
run_key(cfg), the SHA-256 hash of the effective config without its[render]table (aProductrecord holding a top-levelrenderis refused: a product that reads it keys it under its own name). A changed config therefore gets a new directory and never overwrites an old one. A wall-clock stamp or a hand-bumped version suffix is never used. DrWatson'ssavenameis the equivalent for flat configs; run configs are nested, hence the hash. What a run depends on beyond its config is folded in, never hashed privately:run_key(cfg; files, upstream, parts)(a file's content, an upstream run's key, a frame limit). - Every step a keyed product, through
produce_or_load. Each step of a pipeline (decode, detect, index, a start state, a refinement, an evaluation, a figure) is aProduct(experiment, name, record; output, files, upstream, parts), keyed by ONLY what it reads — its own config section, its inputs, the products it reads (upstream = ("start" => start_product,): each one's key and its output's content hash, recorded when it was produced, so an upstream produced again with another output re-keys its consumers; build the products when their stage runs) — so a later step's setting never recomputes an earlier one.produce(f, product)produces the step's output once, through DrWatson'sproduce_or_load, and runs nothing whileproduced(product)holds (a file or a directory output alike); a recordfreturns is saved atomically (a temporary file renamed onto the output) — a.jld2output as JLD2's object"record"(any Julia value), any other throughVisualGeometryToolkitIO.save— and read withload_output; a product that writes its own outputs returnsnothing, itsoutputthe file (or directory) it writes last. Its directory is DrWatson'sdatadir("exp_pro", experiment, name, key)(results_root()isdatadir("exp_pro"),VGTK_RESULTS_ROOTrelocates it); a deeper layout is the experiment path (joinpath(dataset, matcher, problem, experiment)). A step of record another project reads is made an input of record withpromote_run(product_dir(product), tier). - Provenance per result. Every result directory carries
_config.tomland_provenance.toml, written bywrite_run_recordsandwrite_provenance. The provenance is DrWatson'stag!record: the commit, the patch of a dirty tree, and the declared inputs relative to their roots; a product's provenance is written last, so a half-written step never reads as produced. - Stage runner. A driver declares its inputs (
require_input) and its stages (Stage(name, desc; items, product, run), each item's keyed product, orStage(name, desc; check)), and it ends withrun_stages.--planprints each stage's produced products,--stage=NAMEruns one stage and--forceproduces its products again. - Data policy. Inputs under
data_root()are immutable, and the writers refuse them (assert_not_input). Results underresults_root()are add-only. No code, test, example, config, doc or tracked provenance holds a machine path, and no code anartifact"…"string macro or an artifact or download API of its own (data_policy_violations, rules D1 to D5). - Machine paths live in the site profiles. Where a machine keeps every project's inputs and results, and its artifact store, is written once, in VisualGeometryToolkitEval's site profiles (
configs/sites/<site>.toml,[storage]: see Machines and storage), selected byVGTK_SITE. A project names none. - Published data as Julia artifacts, in the artifact store. Input data are published on Zenodo (or Hugging Face) as lazy, hash-pinned artifacts (
Artifacts.toml). They are catalogued inconfigs/data_artifacts.toml(see The data catalog); every config input names its catalogued tier.scripts/fetch_data.jl(DataArtifacts'fetch_data!) installs them into the artifact store, never the Julia depot of a cluster, and links them into the artifact view thatdata_root()falls back to. A fresh machine therefore runs the drivers unchanged. Nothing underdata/is tracked. - REPRODUCE.md. Every figure, table and macro file that the paper reads is a row of the asset index. A row is either
generatedby a driver that declares it (# reproduce: writes <path>and# reproduce: reads <config>) orstaticwith a provenance note.
The enforcing test
using Test
using DrWatson: projectdir
using VisualGeometryToolkitEval.PaperProjects: paper_project_violations
@testset "paper-project conventions" begin
v = paper_project_violations(projectdir(); papers = ["paper/main.tex"])
foreach(x -> @info(sprint(show, x)), v)
@test isempty(v)
endpaper_project_violations returns a list of PaperProjectViolations, each with a check, a rule, a file, a line and a message:
| check | rules |
|---|---|
:layout | L1 a required path is missing · L2 Project.toml lacks DrWatson or VisualGeometryToolkitEval · L3 git does not ignore data/exp_raw and data/exp_pro · L4 the suite does not call this check · L5scripts/fetch_data.jl is missing or does not call fetch_data! · L6 git tracks data under data/ (anything but data/README.md) |
:data_policy | D1 a machine path — a machine root, a home-relative path (~/…, $HOME/…, homedir()) or a host name such as dgx01 other than a site selection's value — (code, launchers and the other text files of src/ and scripts/; tests, examples, configs, docs, provenance and the Markdown of the root and data/, comments included; in a git work tree the files git does not ignore; a site profile, the one place a machine path belongs, excepted) · D2 a write into the input tree · D3 an unkeyed output · D4 an artifact"…" in code · D5 an artifact or download API outside DataArtifacts (the data_policy keyword forwards keywords) |
:reproduce | every kind of reproduce_index_violations (the reproduce keyword forwards keywords) |
:configs | C1 a producing script reads no config of record · C2 it holds a data constant · C3 it names a config that does not exist |
:provenance | P1 no provenance through the shared writer (in the script, what it includes, or what it launches) · P2 no run key · S1 an index generator without run_stages |
:catalog | A1 the findings of catalog_violations, once the catalog or Artifacts.toml binds an artifact, and a produced entry whose producer is no stage of the graph or reads an input that is neither published nor produced (or a cycle of them) |
:storage | A2 a config's [inputs.<id>] names no tier the catalog lists, and no produced entry covers its path · A3 an artifact name bound to another tree in another project's Artifacts.toml (the stack_artifacts keyword) or in the artifact store's name record · A4 a link of the artifact view data/exp_raw that resolves outside the artifact store |
:release | opt-in, before a release: R1 a record not yet published (no Zenodo id, no Hugging Face revision) · R2 a Hugging Face record not stated public = true · R3 a binding whose URL is a placeholder or a local file |
:contract | opt-in: the contract_violations of every stage file |
A producing script is a .jl file under scripts/ that runs stages, declares an output, is an index generator or calls a result writer. The scans of configs and provenance are static and conservative: a necessary condition, not a proof.
A file opts out of one rule with # paper-project: allow <RULE> <reason>. For C2 and C3 the comment goes on the line or the line above; for C1, P1, P2 and S1 it may go anywhere in the file. An opt-out without a reason is rule X1. A project's deferred violations go in the allow keyword as DataPolicyAllow(rule, file, match) values, each with its fix in a comment beside it. An entry that matches nothing is reported as stale, so the list only shrinks.
Machines and storage
A machine is a site profile of VisualGeometryToolkitEval (configs/sites/<site>.toml; load_site), selected by VGTK_SITE (the runner sets it for every job). Its [storage] table is the only place a machine path is written:
[storage]
data_root = "/cluster/projects/vgtk/data/{project}" # a project's inputs; {project} its name
results_root = "/cluster/projects/vgtk/results/{project}" # a project's results
artifact_store = "/cluster/projects/vgtk/artifacts" # every artifact tree, shared
refuse_depot = true # never fall back to the node-local depotA path may start at the home (~/…, $HOME/…): the profile is the machine's and its user's, so the home is spelled out there and nowhere else (a project's configs name none, D1). {project} is the project's name (project_name: its stage graph's [project] name, else its directory's name). The roots resolve, the first that binds:
| Root | Order |
|---|---|
data_root() | VGTK_DATA_ROOT → data/exp_raw → the stage graph's [sites.<site>] data_root, then the site's data_root (each once it exists) → the project's fallback → the artifact view |
results_root() | VGTK_RESULTS_ROOT → data/exp_pro → the stage graph's [sites.<site>] results_root, then the site's results_root (each once it exists, or with create_results = true) |
| the artifact store | VGTK_ARTIFACT_STORE → the site's artifact_store → VisualGeometryToolkitCore's preference artifact_store → the depot (refused where the site sets refuse_depot = true; where no profile states it, refused where NODE_CACHE is set) |
The two roots follow one ladder, the runner's: a launch (resolve_roots) takes --data-root / --results-root first and has no data/exp_raw rung (it names the machine's roots, which its jobs get as VGTK_DATA_ROOT / VGTK_RESULTS_ROOT), otherwise the same order. A job gets VGTK_SITE as the site's name, or as its profile's file when the site was loaded by path, so a driver inside it finds the same site.
The artifact store (VisualGeometryToolkitCore's artifact_store) holds the trees of every published tier and every model's weights, content-addressed by their git-tree-sha1, so every project and package on a machine shares one copy. VisualGeometryToolkitCore's install_artifact installs into it: Pkg's with_artifacts_directory, its sha256 and tree-hash checks and its atomic move into place, under a lock per tree (safe on NFS), so jobs that start together download a tree once. A cluster's Julia depot (node-local and evicted, or a home with a quota) is refused by the site's [storage] refuse_depot = true; the shipped vcai site sets it and names the cluster's store. A machine whose depot already holds the trees moves them once with VisualGeometryToolkitCore.migrate_artifacts(toml) (each tree's hash checked; remove = true frees the depot).
The template
using VisualGeometryToolkitEval.PaperProjects: new_paper_project
new_paper_project("neurips2026"; papers = ["paper/main.tex"]) # git = false inside a package repositorynew_paper_project runs DrWatson's initialize_project with PAPER_PROJECT_TEMPLATE and writes the files of the layout above: Project.toml, README.md, CLAUDE.md (a pointer to these conventions), .gitignore, REPRODUCE.md with the index table, configs/example.toml, scripts/example_stage.jl, src/<name>.jl, test/runtests.jl, the papers, Artifacts.toml, the catalog stub and data/README.md. The example stage shows the stage runner, a config of record, a run key and provenance; it renders a figure, its caption and a macro file under plots/example/, and its deploy stage copies them into the paper (deploy_assets: a changed submitted asset needs --replace-submitted). With --results-branch it skips deploy and makes a results_branch of the renders instead, recording its synthetic draws as the data tier. Then:
cd neurips2026
julia --project=. -e 'using Pkg; Pkg.instantiate()'
julia --project=. scripts/example_stage.jl # the example's assets
julia --project=. test/runtests.jl # zero violationsThe data catalog
configs/data_artifacts.toml is read by load_catalog in the DataArtifacts module:
[catalog] # optional
artifacts_toml = "../Artifacts.toml" # the bindings, relative to the catalog
artifacts_toml_env = "ACCV2026_ARTIFACTS_TOML" # a variable naming another copy
[records.<group>] # one hosted record: title, placeholder, license, host (zenodo | huggingface) and its keys
[artifacts.<name>] # kind, mount, record, required, what, provenance, and the kind's keysThe variable named by artifacts_toml_env points to another copy of the bindings, with other URLs and the same hashes; the fresh-machine test uses it for a file:// copy. A record's host is where its tarballs are downloaded from. On Zenodo (the default) it has an id once the record exists; on Hugging Face it names its dataset repo and, once the tarballs are uploaded, the revision — a commit, never a branch, so a binding stays immutable while the dataset is edited. Until then the record carries its placeholder, an UPPERCASE token that `fetchartifacts!` refuses to download. A typical use: Hugging Face while the evaluation data is still changing, Zenodo (with its DOI) at publication — the bindings' hashes are the same on both, only the URLs move:
[records.rc]
title = "…"
license = "CC-BY-4.0"
host = "huggingface"
repo = "radiometric-camera/radiometric-camera-data"
placeholder = "HF_REVISION_RC"The kinds of artifact and their keys:
| kind | keys | holds |
|---|---|---|
pool_sources | config, splits | the raw files that the pool producer config reads for the scenes of splits |
pools | config, splits | the pool tree that config writes |
stored | config | the directory that the [stored] table of config names |
tree | dir, optional filter | every file under dir; with filter, a list of globs over paths relative to the mount root, the files an include glob matches (every file when there is none) and no ! glob matches |
A filter glob uses * for a run of characters other than /, ? for one of them, **/ for any run of whole directories (none included) and a trailing /** for everything below; every other character is literal. For example filter = ["**/*.ARW", "!archive/**"] with dir = "." selects every raw file outside archive/. pack_artifacts keeps the comment header of Artifacts.toml (its leading comment lines), which Pkg's writer drops.
A produced entry is a derived input of record that is never published: a stage of the project's graph regenerates it from the published tiers.
[artifacts.pt_e_pools_of_record]
kind = "produced"
producer = "pt_e_pools" # a stage of configs/reproduce.toml
covers = ["accv2026/data/PT-E-SIFT-4K_pf3"] # the config inputs it is, by their [inputs.<id>] path
tier = "derived" # the covered inputs' own tier (a path is relative to its tier's root)
what = "the PT-E scoring pools"
provenance = "regenerated from the published raw tiers by the stage pt_e_pools"An input of that tier whose path is one of covers, or lies under one, is this entry's, so the configs keep their inputs (and their run keys) as they are. The entry has no mount, record or binding: it is never packed, bound or fetched, and fetch_data! names its producer instead. The check (A1) requires the producer to be a stage of the graph with a config of record (each array item's), whose inputs are each a published tier of the catalog or covered by another produced entry, with no cycle; A2 accepts an input a produced entry covers.
A produced entry may name another project's producer, producer = "<project>:<stage>" ("accv2026:pt_e_pools"): A1 then requires that stage in that project's graph, the project found among the stack_artifacts of the check (another project's Artifacts.toml, with its configs/reproduce.toml beside it); that project's own check covers its inputs.
Another project's run of record
A config reads another project's keyed product — its run of record, where it lives, never copied — by naming the product rather than a tier's file:
[inputs.hand_eye]
project = "bmvc2026" # the producing project (its stage graph's [project] name)
product = "calibration/hand_eye" # the Product's <experiment>/<name> (its directory under that project's results)
key = "3f9a0c1d2e4b5a67" # the run key of record (the product's directory)
file = "hand_eye_bb.json" # optional: a file in the product's directory (default: the whole directory)
sha256 = "…" # its content (a directory's tree_sha256)
tier = "bmvc2026_results" # optional: a results-mount tier of THIS project's catalog that publishes theminput_path(cfg, "hand_eye") reads it under the first root that holds <product>/<key>[/<file>]: VGTK_RESULTS_ROOT_<PROJECT> (VGTK_RESULTS_ROOT_BMVC2026), then project_results_root (the active site profile's results_root for that project, {project} its name), then the tier's root. The product's record (_provenance.toml) must name key as its run key, a sha256 is verified, and a scratch run's output is refused. The runner's launch check (missing_inputs) makes the same check (product_input), with the site's results root and the launch's environment. A2 accepts such an input when it is well formed and any tier it names is catalogued.
On a fresh machine the producer's results come as a results-mount tier: an entry of this project's catalog with mount = "results" (a tree of the producer's results, published like any tier), which fetch_artifacts! links into the results root at its mount_dir. An input of that tier (tier = "…", path = "…") is read under <results root>/<mount_dir> (data_root(; tier), through tier_data_root's results rung), and a product input naming the tier falls back to it. The tree keeps the producer's layout, so <product>/<key>/<file> is the same path under it:
[artifacts.bmvc2026_results]
kind = "tree"
mount = "results"
mount_dir = "bmvc2026" # under this project's results root
dir = "."
record = "bb"
required = true
what = "BlobBoards bmvc2026's run of record: the hand-eye calibrations and the baseline poses"
provenance = "bmvc2026's results of record at <commit>"An entry too large to upload as one file can set split_by = "scene" (kinds pool_sources and pools). It is then bound as one artifact per scene, <name>_<scene>, plus <name>_meta for the files that no scene owns. The entry stays one logical artifact: required, fetch_artifacts! and pack_artifacts take its name and act on every part.
With split_by, the optional max_part_bytes = N caps one part at N uncompressed bytes. A scene whose files exceed it is bound as sub-parts <name>_<scene>_part1 to _partk. A file larger than N is cut into the fewest near-equal chunks <file>.chunk<i>of<m>, and the chunks are packed greedily into parts. fetch_artifacts! reassembles each chunked file in the artifact view, byte for byte. The author's file sizes derive the parts, and a consumer reads them from the bindings.
An entry published before the stack's terminology rules keeps its published file name, which a host such as Zenodo cannot rename, while its binding takes a current one: published_name = "<old name>" names the file the record holds (published_name). Its URL then ends in <old name>.tar.gz (a split entry's parts replace <name> by it), pack_artifacts stages the tarball under that name, and catalog_violations checks the URL's file against it instead of the binding name:
[artifacts.eth3d_splg_correspondences]
published_name = "<the file name the record holds>" # e.g. a name the record published in 2026-10Such a published name is data, not a binding: VisualGeometryToolkitCore's check_naming lists it among its foreign tokens and accepts it only in a quoted TOML value, a string literal or a data fixture (an Artifacts.toml).
Every artifact tree holds paths relative to its mount root: data is data_root() and results is results_root(). The author's tree, a packed artifact and a fetched view therefore share one layout. A project adds kinds of its own with the kinds keyword (kind => (cfg, entry, root) -> files).