API: Reproduce
The reproducible-figures contract. See Reproducible figures.
Public names
VisualGeometryToolkitEval.Reproduce — Module
ReproduceThe reproducible-figures contract: paper_asset_graph (what a paper reads), read_reproduce_index (what REPRODUCE.md declares) and reproduce_index_violations (where the two disagree, plus generators that do not write their asset, static assets without provenance, and generated numbers re-typed in prose); and the regeneration's last stages: verify_assets and deploy_assets (a submitted asset is replaced only with sign-off; asset_stages, the pair as a driver's stages over a keyed product), gather_numbers (the numbers file) and uncited_macros (numbers the paper defines but never cites); and the one writer of a driver's number macros: PaperNumbers, add_number!, add_table_row!, write_numbers, tex_number and number_macro_name, and where its numbers come from: checkout_commit and number_provenance; and a full reproduction as a branch of the paper for review: results_branch, accept_results and remove_results_worktree; and the runner that reproduces a project with one command: a stage graph of record (load_stage_graph, select_stages) on a site (load_site: Slurm or local), the inputs checked first (missing_inputs), a run manifest per launch, finished stages reused by run key, and the results branch at the end (results_branch_of_run); reproduce_main is its command line.
VisualGeometryToolkitEval.Reproduce.REPRODUCE_SCRIPT_TEMPLATE — Constant
REPRODUCE_SCRIPT_TEMPLATEThe template of a project's runner bootstrap, templates/reproduce/reproduce.sh in VisualGeometryToolkitEval: a bare julia finds the package's sources (fetching them on a fresh checkout), includes src/cli.jl and src/reproduce/runner.jl and runs reproduce_main on the project's graph. reproduce_script fills it.
VisualGeometryToolkitEval.Reproduce.REPRODUCE_USAGE — Constant
REPRODUCE_USAGEThe runner's usage text, printed by reproduce_main when --graph or --site is missing.
VisualGeometryToolkitEval.Reproduce.RESULTS_COMPARISON — Constant
RESULTS_COMPARISONThe comparison report a results branch carries at the paper project's root: "COMPARISON.md" (results_branch).
VisualGeometryToolkitEval.Reproduce.RESULTS_MANIFEST — Constant
RESULTS_MANIFESTThe manifest a results branch carries at the paper project's root: "RUN.toml" (results_branch).
VisualGeometryToolkitEval.Reproduce.SITES_DIR — Constant
SITES_DIRThe directory of the site profiles shipped with VisualGeometryToolkitEval (configs/sites/<name>.toml: the machines, not the projects). _template.toml there documents every key; a project may keep its own site files and name them by path, or list their directory as the graph's [project] sites_dir.
VisualGeometryToolkitEval.Reproduce.UNPUBLISHED_TIER — Constant
UNPUBLISHED_TIERThe reserved input tier "unpublished": data of record that is not (yet) published as a data artifact, so nothing can fetch it — a capture kept on one machine. An [inputs.<id>] of this tier (its root through VGTK_DATA_ROOT_UNPUBLISHED or [sites.<site>.tiers] like any tier) is checked to be there, and its sha256 when given; the dry run lists every such input apart, present or absent (print_plan), and a missing one is reported as never fetched.
VisualGeometryToolkitEval.Reproduce.VERIFY_REPORT — Constant
VERIFY_REPORTThe file name of the verify report verify_assets writes beside the renders: "verify.tsv".
VisualGeometryToolkitEval.Reproduce.IndexEntry — Type
IndexEntryOne row of a REPRODUCE.md asset index: the paper label it serves, the asset (project-relative path or glob), its kind (:generated, :static, or :invalid for an unrecognised kind cell), the generators (project-relative paths), the configs of record they read (project-relative; from an optional config column), the data it is generated from, the provenance note (required for :static), and the index line.
VisualGeometryToolkitEval.Reproduce.PaperAsset — Type
PaperAssetOne file the paper reads: path (relative to the project directory), kind (:tex for \input/\include, :graphics for \includegraphics, :data for a pgfplots/tikz data file), the referencing source file and line, the \labels of the enclosing float (inherited through \input), and whether the file exists.
VisualGeometryToolkitEval.Reproduce.PaperAssetGraph — Type
PaperAssetGraphWhat a set of paper sources reads: the project directory root, the visited tex files (project-relative), every PaperAsset, every \label key, and the references whose target could not be resolved statically (a macro or #1 in the path) as (source, line, text).
VisualGeometryToolkitEval.Reproduce.PaperNumbers — Type
PaperNumbers(; driver, config, run_key, commit)The number macros one driver writes, empty, each carrying its provenance: the driver and its config of record (project-relative paths), the run_key of the run the numbers are read from and the commit that computed it (the run's recorded gitcommit, not the checkout's: the commit moves only when the run does). A number added with its own provenance (one read from another run) overrides any of the four. Fill it with add_number! and add_table_row!; write it with write_numbers.
VisualGeometryToolkitEval.Reproduce.PlannedPhase — Type
PlannedPhaseA phase of a stage that runs before its last (StageSpecphases; on a Slurm site the runner's precompile): its name, the stage spec it runs as (spec: the stage's, with the phase's environment, device, resources and array), its backend command and its environment env.
VisualGeometryToolkitEval.Reproduce.PlannedStage — Type
PlannedStageOne stage of a launch: its spec, its items (and per item the values its task loops over, each: [] for none), the selected stages it runs after, its run key (every file and option the stage's jobs read, and the keys of the stages it runs after, their last completed runs' for those the launch does not run; never the code: a stage of the same key that finished before is reused while the code trees it recorded are the current ones, code_tree_changes), the environment its jobs get beyond the caller's (env), its command (on a Slurm site the sbatch argv without the dependency; on a local site the argv run once per item) and spec (its last phase: StageSpecphases), the phases before it (PlannedPhase), its --time in minutes (minutes, 0 = the launcher's), reuse (""; "completed", finished in an earlier launch; or the job id of an earlier launch's job still in flight, which this launch depends on instead of submitting again) and, for that job, reuse_markers, the directory its completion markers go to (stage_markers_dir); and rerun, why an earlier run of the same key is not reused ("" when there is none or it is: "code changed: VisualGeometryToolkitCore: src 3fa29c01e2 → …").
VisualGeometryToolkitEval.Reproduce.ReproducePlan — Type
ReproducePlanA launch of a stage graph on a site: the graph, the site, the code commit (and dirty: tracked files anywhere differ from it), the data_root and results_root, the stamp of the launch, the path of its run manifest, its stages in dependency order, the environment the launch was planned in (base_env: the stages' environment is it plus each stage's own), and the code it runs: code_trees (by name, the git trees of the graph's code paths and of every stack package the project's environment loads: code_trees) and dirty_code, those of them with uncommitted changes (a launch refuses to run them; a scratch launch runs them with a warning), and code_paths, the project's own paths among them (a launch that identifies none is refused too).
VisualGeometryToolkitEval.Reproduce.ReproduceViolation — Type
ReproduceViolationOne breach of the reproducible-figures contract: a kind (see reproduce_index_violations), the subject (an asset, generator or macro), a human-readable message and a location (file:line).
VisualGeometryToolkitEval.Reproduce.Site — Type
SiteA machine a reproduction runs on, read by load_site from a site profile: its backend (:slurm, sbatch arrays chained by --dependency=afterok; :local, every stage in order in this process's machine), the Slurm options every job gets (partition, a GPU stage's gres, cpus_per_task, mem_per_cpu, gpu_limit GPU array tasks at a time, extra options, and job_name_prefix, put before every job's name: the launch's label for the scheduler, never part of a run key), the local cuda_visible_devices and cpus_per_task, a shell setup run before every stage, the environment env every stage gets, and the machine's storage ([storage]): data_root and results_root, where a project's inputs and results live on it (absolute, or under the home: ~/…, $HOME/…; {project} in either is the project's name: site_root), artifact_store, the directory of the artifact store every project and package there installs into (VisualGeometryToolkitCore's artifact_store), and refuse_depot, whether a process there may fall back to the Julia depot as its store (true where the depot is node-local and evicted, or in a home with a quota; false where it may; nothing when the profile does not say, and VisualGeometryToolkitCore's NODE_CACHE heuristic decides). The site profiles are the ONLY place a machine path is written: a project's configs name none.
VisualGeometryToolkitEval.Reproduce.StageGraph — Type
StageGraphA project's stage graph of record, read by load_stage_graph: the project's name, its workdir (where every path of the graph is relative to and where the jobs are submitted from), its julia_project, the variables that bind its data and results roots (data_env, results_env), the directory of the run manifests under the results root (manifest_dir), each site's roots (site_roots), the project's own sites_dir, the code paths whose content keys every stage, the stages in dependency order and the [results_branch] table.
VisualGeometryToolkitEval.Reproduce.StageSpec — Type
StageSpecOne stage of a project's graph ([[stage]] of the graph TOML): its name, the stages it runs after (an entry "<stage>:<item>" waits on that item of an array stage alone: on a Slurm site its job runs after that array task, afterok:<job>_<task>, not the whole array; after_items holds them; the key folds the upstream stage's key either way, since one key names every item's run), what it is (what), and how it runs — exactly one of sbatch (the project's own launcher script: submitted as is on a Slurm site, run by bash on a local one with the Slurm variables it reads set), command (an argv, wrapped by sbatch --wrap on a Slurm site) or results_branch = true (the paper's results branch of the launch, results_branch_of_run). driver and config name the driver and its config of record (the config may hold {item}, one per array item): the run manifest hashes them, the input check reads the config's [inputs], and the dry run prints the driver's own command line. array makes the stage an array, one task per item: over items, or over the values at a dotted key from of the stage's config — a table's keys sorted by name (from = "cases", the config's [cases]; from = "variants"), a list of strings in its order (from = "grid.sets"). array.each = { from, env } loops inside each task: the task runs its command once per value at the dotted key from of its item's config (a table's keys sorted, a list in order), with the variable env set to the value, in order, and fails at the first value that fails (a set's task over every variant); inputs_from (tables {config, key, base}) names further configs whose [inputs] the stage reads, as the paths at a dotted key of config (a segment * is every key of that table; each path relative to base under the workdir), for a stage that runs other drivers' configs; select_env names the variable that selects a subset (comma-separated) and that the stage's launcher reads (each's env, set in the launch's environment, selects the values alike), and max_time (a dotted key of each entry of a from table, seconds; times the number of each values) plus time_margin_s sizes its --time. env is exported to the stage, job_name and time override the launcher's, gpu marks a GPU stage (gres, CUDA_VISIBLE_DEVICES, the site's gpu_limit), and default = false keeps it out of a run without --only. cpus_per_task, mem_per_cpu and gres are the stage's own resources, over the site's (0/"": the site's; validated: a count, 8G/4000M, gpu:4), and ntasks and nodes its tasks and nodes (a distributed stage; 0: one task, Slurm's default): a stage that needs 4 GPUs or 2 CPUs says so in the graph, not in its launcher; gres makes a stage a GPU stage. On a local site cpus_per_task is the SLURM_CPUS_PER_TASK its launcher reads. nice (a count, 0: none) is the stage's Slurm --nice: a research stage yields to a release gate's jobs. Like the CPU count and a job-name prefix it is a label for the scheduler and keys no run; the local backend runs every stage alike.
phases (tables {name, gpu, env, sbatch | command, cpus_per_task, mem_per_cpu, gres, time, per_item}) splits the stage into jobs run one after another, each the stage's launcher (or the phase's own sbatch/command) with the phase's environment (and VGTK_PHASE=<name>), device (gpu: a GPU phase gets the stage's or the site's gres, a CPU phase none) and resources; per_item (default: whether the stage is an array) makes a phase an array over the stage's items, task k of the next phase running after task k of this one (aftercorr), and false one task (afterok). So no GPU is held for CPU work: a GPU drivers phase then a CPU numbers phase, or a CPU decode phase then a GPU detect phase per item. The last phase is the stage's job (its completion markers, its job_id); each earlier one writes its markers under <stage markers>/<phase>/, and status names a failed one. A phase's command and environment key the stage like its own. On a Slurm site a stage with a GPU phase (or a GPU stage) gets one more, first: precompile, a CPU-only job running julia --project=<julia_project> <julia_flags> -e 'using Pkg; Pkg.precompile()' (julia_flags: the flags the stage's jobs run julia with, ["--depwarn=error"]: a cache compiled under other flags is compiled again), so a GPU task only loads and runs; precompile = false leaves it out (a stage that runs no julia).
VisualGeometryToolkitEval.Reproduce.accept_results — Method
accept_results(dir, branch; replace_submitted, scratch_root = <results_branch's>) -> NamedTupleBring the assets of the results branch (results_branch) of the paper project at dir into its base branch as one merge commit, and tag it accepted/<stamp>. The assets go through deploy_assets: an asset the base holds a different copy of is replaced only with the sign-off replace_submitted (true for every asset, or a collection of the assets' paths relative to dir); without it the acceptance is refused and nothing changes. The merge is made in a worktree of its own under scratch_root (removed at the end, whatever happens); the tag is made on it, then the base moves to it: a checkout of the base fast-forwards (one with local changes to an accepted asset refuses: the tag is deleted again and the base stays; an unrelated local change survives). That fast-forward is a git merge in the user's checkout, so it runs the checkout's own post-merge hook; the commits accept_results makes skip hooks (--no-verify). A base no checkout holds moves by update-ref, so a checkout on another branch is untouched. Only the assets are merged, never the branch's manifest or comparison. A base that moved since the run (the manifest's base_commit) is a warning naming the accepted assets it changed: the branch's comparison is against the older base.
Returns (; commit, tag, deployed): the merge commit, the tag and the assets copied (repository paths).
VisualGeometryToolkitEval.Reproduce.add_macro! — Method
add_macro!(nums, name, body; arity, label = "", provenance = (;)) -> numsAdd the macro \<name> with arity parameters (1 to 9) to nums: its body is TeX in which #1 … #<arity> stand for the arguments (a figure's equal-height panel cell, \parbox[c][\panelht][c]{#1}{\centering #2}). Written by write_numbers as \newcommand{\<name>}[<arity>]{<body>} after its provenance comment, gathered by gather_numbers as it is; it is not a value, so tex_macro_values does not return it. A parameter beyond the arity, an arity outside 1–9, unbalanced braces, an unescaped % and a name nums already holds are each an ArgumentError.
VisualGeometryToolkitEval.Reproduce.add_number! — Method
add_number!(nums, name, value; label = "", provenance = (;), better = nothing, format...) -> numsAdd the number macro \<name> to nums. A value that is a number is typeset by tex_number with the format keywords (digits, sigdigits, percent, group_from, unit); a pair (lo, hi) is a range; a string is its body as typeset (a label, "---" for a missing cell, 10^{-6}) and takes no format. label names what reads the number when it is not the prose: a figure's fig:… (its caption, its axis), a table's tab:…. It goes into the macro's provenance comment. provenance overrides fields of the driver's (PaperNumbers) for this number. better is the direction in which the number improves, :lower (an error, a time), :higher (a rate, a score) or :none (a count, a setting), as a Symbol or a String; it goes into the provenance comment (; better=lower), from which a results comparison (results_branch) calls a change IMPROVED or DEGRADED. Without it a change is reported CHANGED, with no verdict.
name is letters only, never beginning with end (number_macro_name builds one); a name nums already holds, a body of more than one value (an unescaped %, # or &, a row break, unbalanced braces or math shifts, a trailing backslash), a ; in the label or provenance (the comment's separator), a Bool and a NaN are each an ArgumentError.
VisualGeometryToolkitEval.Reproduce.add_table_row! — Method
add_table_row!(nums, row, cells; label = "", provenance = (;), better = nothing, format...) -> StringAdd a table row's numbers to nums, one macro per cell: cells is an ordered collection (a vector, a tuple, an OrderedDict) of column => value pairs in the table's column order; a Dict or a Set has no column order and is an ArgumentError. The cell's macro is \<row><Column> (row and each column letters only, the column capitalised). Each value is typeset as by add_number!, the format keywords shared by the row; better is one direction for every cell, or a collection of column => direction pairs for the cells that state one (a column the row lacks is an ArgumentError); a cell of another format is given as its tex_number string. Returns the row as the table's tex reads it, "\<row><A> & \<row><B> & …", without the row break.
One macro per cell, not one per row: the prose cites a table's number by the macro the table prints, and the gatherer's checks (compare_fragments, macro_literal_violations, uncited_macros) see each number.
VisualGeometryToolkitEval.Reproduce.asset_stages — Method
asset_stages(product, cfg, assets; paper = "paper/main.tex") -> Vector{Stage}The verify and deploy check stages (Experiments' Stage, for run_stages) of a driver whose renders assets (file names) are files of the keyed product product() — a function (any callable), so the product is built when its stage runs:
:verify, each render against the paper's copy inpaper_figures_dir(paper)(verify_assets: bytes, and a.texfragment's numbers), the report beside the renders;:deploy, into that directory where nothing changes, or where the config's[deploy] replace_submittedsigns the difference off (deploy_assets,deploy_signoff).
Each refuses a product that is not produced under its current key (Experiments' produced_dir: a commit re-keys it): its producing stage runs first. A driver lists them after its producing stages:
const STAGES = [Stage(:figure, …; items = () -> [figure_product()], …),
asset_stages(figure_product, CFG, ASSETS)...]VisualGeometryToolkitEval.Reproduce.assets_deployed — Method
assets_deployed(outdir, paperdir, names) -> BoolWhether every rendered outdir/<name> is byte-identical to the paper's paperdir/<name>: a cheap done predicate of a deploy stage.
VisualGeometryToolkitEval.Reproduce.checkout_commit — Function
checkout_commit(root = project_root()) -> StringThe commit of the checkout at root as DrWatson describes it (gitdescribe(root): <tag>-<n>-g<sha> or a sha, -dirty on an uncommitted tree; "unknown" outside a repository), a space or ; made _: the PaperNumberscommit of numbers a driver computes itself. Numbers read from a run carry the run's recorded commit instead (number_provenance).
VisualGeometryToolkitEval.Reproduce.compare_fragments — Method
compare_fragments(new, old) -> NamedTupleHow a regenerated tex fragment new differs from the submitted old (texts): identical (byte for byte); same_numbers, every coordinates block equal in order (fragment_coordinates) and every argument-free macro definition equal (a numbers file); the block counts blocks_new and blocks_old; and max_abs_diff, the largest absolute coordinate difference over blocks of equal length (Inf when the block structure differs).
VisualGeometryToolkitEval.Reproduce.declared_inputs — Method
declared_inputs(generator_path) -> Vector{String}The configs / inputs a generator declares, one per reproduce: reads <path> line in its source (in any comment syntax), project-relative.
VisualGeometryToolkitEval.Reproduce.declared_outputs — Method
declared_outputs(generator_path) -> Vector{String}The outputs a generator declares, one per reproduce: writes <path-or-glob> line in its source (in any comment syntax), project-relative.
VisualGeometryToolkitEval.Reproduce.deploy_assets — Method
deploy_assets(outdir, paperdir, names; replace_submitted = false) -> Vector{String}Copy each rendered outdir/<name> into paperdir when the paper has no copy or holds a different one, after verifying them all in one report (verify_assets writes a fresh outdir/verify.tsv first). A submitted copy that differs is replaced only with the author's sign-off on the difference the report states: replace_submitted = true signs off every asset of the call, a collection of names only those (per-asset sign-off, so one call deploys every asset and the report keeps every row). Without it the deploy is refused, as an error naming each differing asset and the report, and copies nothing. A numbers file that differs from the submitted one in its % provenance \<name>: … lines alone (a rerun at a new run key or commit, the same macro values; the report's provenance_only) changed no number: it is deployed without sign-off, its provenance updated. Any other difference, a hand-written comment or whitespace included, still needs sign-off. Returns the assets copied (an identical one is not copied).
VisualGeometryToolkitEval.Reproduce.deploy_signoff — Method
deploy_signoff(cfg) -> Union{Bool, Vector{String}}The author's sign-off of a driver's config on replacing submitted assets, its [deploy] replace_submitted: true for every asset, the names of the signed-off assets, or false (the default, also without a [deploy] table) — the replace_submitted of deploy_assets. Any other value is an ArgumentError.
VisualGeometryToolkitEval.Reproduce.fragment_coordinates — Method
fragment_coordinates(text) -> Vector{Vector{NTuple{2,Float64}}}Every coordinates {…} block of a pgfplots fragment, in order, parsed to numbers: the quantitative content of a panel, for comparing a regenerated fragment with the submitted one independently of its formatting.
VisualGeometryToolkitEval.Reproduce.gather_numbers — Method
gather_numbers(fragments, out; generator, root, root_label = "<results_root()>") -> outWrite the paper's numbers file out from the generators' number-macro fragments (tex files): every argument-free macro of every fragment, one \newcommand{\<name>}{<value>} per line, sorted by name, under a header naming the generator and each fragment as <root_label>/<path relative to root> (no machine path). A macro's provenance comment in its fragment (the % provenance \<name>: … line write_numbers writes) goes before it. Never \providecommand: a macro the manuscript also defines must clash. A macro with parameters that the writer wrote (add_macro!: its provenance comment before it) is gathered as written, its arity kept. A macro that two fragments define, and any other fragment macro with arguments (a number is a value, not a template), are an ArgumentError.
Write out beside the generator's results, not into the paper: a submitted numbers file is replaced under the same sign-off as a figure, by deploy_assets.
VisualGeometryToolkitEval.Reproduce.load_site — Method
load_site(spec; dirs = String[]) -> SiteThe site profile spec: a path to a TOML file (it ends in .toml or names an existing file), or a name looked up as <name>.toml in dirs (a project's own site files first) and then in SITES_DIR. An unknown table or key, a backend other than slurm or local, or an unknown name (the error lists the known ones) is refused.
VisualGeometryToolkitEval.Reproduce.load_stage_graph — Method
load_stage_graph(path) -> StageGraphA project's stage graph of record from its TOML (the project's configs/reproduce.toml): [project] (its name; workdir, relative to the graph file's directory, default "."; julia_project, relative to workdir; data_env and results_env, default VGTK_DATA_ROOT and VGTK_RESULTS_ROOT; manifest_dir, default "_reproduce"; sites_dir, the project's own site files; code, the paths, relative to workdir, of the code every stage runs, default src — their git trees, with those of every stack package the project's environment loads (code_trees), are the code a launch records: a finished stage is reused only while they are unchanged, so a change elsewhere, the paper's text or the docs, reuses it, and a launch with uncommitted changes in them is refused), [sites.<site>] (data_root, results_root, an env table, and a tiers table, <tier> = "<root>", the roots of the input tiers that are not under the data root on that machine: where this project's data and results are on that machine), one [[stage]] per stage (StageSpec), and [results_branch] (project, the paper project's directory relative to workdir; assets, tables {from, to, submitted_only}: the renders matching the glob from go to the directory to, both relative to project, only those with a submitted copy there when submitted_only; artifacts, an Artifacts.toml relative to project whose entries' git-tree-sha1 are the data tiers). Unknown keys, unknown dependencies and cycles are refused.
VisualGeometryToolkitEval.Reproduce.macro_literal_violations — Method
macro_literal_violations(prose_files, macro_files; root, ignore = String[],
min_integer_digits = 3) -> Vector{ReproduceViolation}Best-effort check that a number a generated macro file defines is not re-typed as a literal in the paper's prose: every argument-free macro of macro_files whose rendered value carries a number is searched for in the comment-stripped prose of prose_files, with number boundaries. A bare integer value is searched only from min_integer_digits digits (the default 3 skips the two-digit counts and percentages that coincide everywhere; 2 is strict and needs literal-ok marks). A TeX length is no literal: a number directly followed by a TeX unit (-0.3cm), or with a space where TeX reads a length (a whole braced argument, \hspace{0.3 cm}; after \kern, \hskip, …); in prose, "1.23 mm" is a number with a unit and is checked. A line holding reproduce: literal-ok (in a comment) is exempt, as is any macro named in ignore.
VisualGeometryToolkitEval.Reproduce.missing_inputs — Method
missing_inputs(g, stages; data_root, env = ENV, site = "", unpublished = nothing, results_root = "",
site_profile = nothing) -> Vector{NamedTuple}Every input of the selected stages that is not there, as data, before anything runs: one row (; path, tier, used_by) per missing path (used_by every stage and config that reads it). A stage's inputs are its configs of record (one per array item) and their [inputs.<id>] tables with a path (relative to data_root) and a tier; a missing config is a row of tier "config of record". With no data root every data input is missing (tier as declared).
Each input's path is relative to ITS TIER's root (tier_data_root): env's <data_env>_<TIER>, else the graph's [sites.<site>.tiers] table, else data_root. A tier root that is named but not there makes each of its inputs a row whose tier says so ("mocap (root /x, from VGTK_DATA_ROOT_MOCAP, is not there)"). Each input is checked by inspect_input (src/codeidentity.jl), the one check a driver's `verifyinputsmakes too: a scratch run's output is refused like another content, unless this launch is a scratch one (VGTKSCRATCH=1inenv, [scratchrun](@ref VisualGeometryToolkitEval.Experiments.scratch_run)). An input of the reserved tier [UNPUBLISHEDTIER](@ref) is reported missing as never fetched; withunpublished(a vector), each such input is also pushed there,(; path, present, usedby)`, for the dry run to list.
An input with a sha256 (64 hex digits: a file's file_sha256, a directory's tree_sha256) is verified: an input that is there with another content is REFUSED, an ArgumentError naming each such input (its id, config, path, the expected and the actual hash), so nothing runs on data other than the data of record. An input without one is not checked. The hash is streamed and cached by path, size, times and inode (file_sha256): a file is read once per process, and a launch that checks the same input from several configs reads it once; a large file's hash is kept across processes in the digest cache while its size, times and inode are unchanged (digest_cache_dir; VGTK_DIGEST_CACHE=off hashes in full), so the launch's check warms it for every job.
A tier the project's catalog mounts under its results (mount = "results", a run of record published as a tier) is read under results_root (catalog_results_mount). An input naming ANOTHER project's product (project, product, key, …: Experiments' input_path) is checked by product_input under env's <results_env>_<PROJECT> (VGTK_RESULTS_ROOT_<PROJECT> by default), then the site_profile's results root for that project (site_root), then its tier's root; a missing one is a row <project>:<product>/<key>[/<file>].
VisualGeometryToolkitEval.Reproduce.number_macro_name — Method
number_macro_name(prefix, parts...) -> StringThe name of a number macro (without its backslash) from a driver's prefix and name parts, the stack's naming rule: each part (a string, symbol or number) split into words at _, -, ., /, × and spaces, each word capitalised, each digit spelled out as a word of its own. number_macro_name("chromeTable", "kb_start", "median") is "chromeTableKbStartMedian"; number_macro_name("pose", "fx3_14mm") is "poseFxThreeOneFourMm". The prefix is letters only; a part holding any other character (a Greek letter, %, a comma) is an ArgumentError, as is a name LaTeX refuses (one beginning with end).
VisualGeometryToolkitEval.Reproduce.number_provenance — Method
number_provenance(run_dir; root = project_root()) -> NamedTupleThe provenance of numbers read from one run's artifacts, from its _provenance.toml (read_provenance): run_key, the commit that computed it (the recorded gitcommit) and the config it ran from (the recorded config_source, made relative to root), each only where the run recorded it, a space or ; made _. For add_number!'s provenance, or splatted into PaperNumbers: PaperNumbers(; driver, config, number_provenance(run_dir)...) (a field given after the splat wins).
VisualGeometryToolkitEval.Reproduce.paper_asset_graph — Method
paper_asset_graph(tex_files; root = dirname(first(tex_files)), search_dirs = String[]) -> PaperAssetGraphEvery file a paper reads, found by following \input/\include/ \subfile/\includestandalone (recursively), \includegraphics (with the \graphicspath directories and the default extensions tried in LaTeX's order), and pgfplots/tikz data files (\addplot table {file}, \pgfplotstableread{file}, \pgfplotstabletypeset{file}). Paths resolve against the directory of the top-level file, then against each of search_dirs in order (the compile directory and the TEXINPUTS of a build run elsewhere, e.g. ["paper"] for a main.tex in paper/src/ built from paper/ that writes \input{src/sec/intro}; a relative entry is relative to root), and are reported relative to root (the project directory). Graphics try the \graphicspath directories before search_dirs. Comments are ignored; a reference whose path holds a macro or a # parameter is recorded in unresolved, not guessed.
VisualGeometryToolkitEval.Reproduce.paper_figures_dir — Function
paper_figures_dir(paper = "paper/main.tex"; root = project_root()) -> StringThe directory the paper paper (its main .tex, relative to the project) reads its generated assets from: figures/ beside it, under the project root (Experiments' project_root(), the bound project). Where a driver's deploy stage puts its renders (deploy_assets, asset_stages) and where the template's papers keep them (PaperProjects' new_paper_project); root = "" gives it relative to the project.
VisualGeometryToolkitEval.Reproduce.plan_reproduction — Method
plan_reproduction(g, site; only = String[], deps = true, data_root = nothing, results_root = nothing,
job_name_prefix = nothing, env = ENV, stamp = <now, UTC>, queue_states = nothing) -> ReproducePlanThe launch of the selected stages (select_stages) of graph g on site, planned and nothing run: each stage's items, dependencies, environment, command and run key, and whether an earlier launch's run of the same key is reused (stage_run_state: completed, by its completion markers; or in flight on a Slurm site, its job still in the queue, queue_states(ids), by default slurm_queue_states on env's PATH; the earlier launches are the manifests under <results root>/<manifest_dir>), and only while the code trees that launch recorded are the current ones (code_tree_changes: a stage whose code changed runs again, rerun naming the change; one still in flight under other code is an error). job_name_prefix (default: the site's) is put before every job's name; the run keys are taken without it.
VisualGeometryToolkitEval.Reproduce.print_missing — Method
print_missing(io, rows; data_root, data_env)The missing-input report (missing_inputs): one tab-separated line path tier used by per missing path.
VisualGeometryToolkitEval.Reproduce.print_plan — Method
print_plan(io, plan; missing = nothing, unpublished = nothing)The job graph of a launch, as the dry run prints it: the commit, the roots, the site, then per stage its items, dependencies, the driver's own command line (when the stage names a driver and config), the command the backend runs (on a Slurm site the sbatch line, a dependency on a stage of this launch written $<stage>), and whether an earlier run is reused; then the missing inputs, and the inputs of the tier UNPUBLISHED_TIER, present or ABSENT (unpublished, from missing_inputs).
VisualGeometryToolkitEval.Reproduce.project_relative_path — Function
project_relative_path(path, root = project_root()) -> StringA path as a provenance comment cites it: relative to the project root when absolute, as given otherwise, made a provenance_token (number_provenance's config).
VisualGeometryToolkitEval.Reproduce.provenance_token — Method
provenance_token(s) -> StringA provenance field as a number macro's comment holds it (PaperNumbers's run_key, commit, config): every space and ; (the comment's separators) made _. The one spelling checkout_commit and number_provenance write; a project that records a field of its own uses it too.
VisualGeometryToolkitEval.Reproduce.read_reproduce_index — Method
read_reproduce_index(path) -> Vector{IndexEntry}The asset index of a REPRODUCE.md: every markdown table whose header has the columns asset, kind and generator (case-insensitive; label, config, data and provenance are read when present, any other column is ignored). A cell's backticked tokens are its values (several assets in one row expand to one entry each); a row without backticks uses the cell text. kind is the first word of its cell: generated or static.
VisualGeometryToolkitEval.Reproduce.remove_results_worktree — Method
remove_results_worktree(dir, branch; delete_branch = false) -> NothingRemove the worktree of the results branch (results_branch) of the paper project at dir, after its acceptance (accept_results) or as its rejection. The branch is kept, so a rejected run stays reviewable, unless delete_branch = true. A worktree with uncommitted changes is refused. Any branch not named results/… is an ArgumentError.
VisualGeometryToolkitEval.Reproduce.reproduce_index_violations — Method
reproduce_index_violations(project_dir; papers, index = "REPRODUCE.md", check_literals = true,
ignore_macros = String[], min_integer_digits = 3,
require_config = false, search_dirs = String[]) -> Vector{ReproduceViolation}The reproducible-figures contract of a paper project, as a list of violations (empty = compliant). papers are the top-level tex files and index the REPRODUCE.md, both relative to project_dir; every asset path in the index is project-relative. search_dirs (project-relative) are the directories a reference that does not resolve against a paper's own directory is looked up in (paper_asset_graph): the compile directory of a build run outside the paper's directory. Violation kinds:
:unindexed_asset— the paper reads a file no index entry covers:missing_asset— the paper reads a file that does not exist:unresolved_reference— a reference whose path is not static (macro,#1):unreferenced_entry— an index entry the paper does not read (stale row):invalid_kind— a kind other thangenerated/static:no_generator— a generated entry naming no generator:missing_generator— a named generator does not exist:generator_does_not_write— no named generator writes the asset: none declares it (reproduce: writes <glob>) and none mentions its file name:static_without_provenance— a static entry with no provenance note:no_config— withrequire_config = true, a generated entry naming no config of record (aconfigcolumn cell):missing_config— a named config does not exist:generator_does_not_read_config— no named generator reads the config: none declares it (reproduce: reads <path>) and none names its file:label_mismatch— the entry's label is a\labelof the paper, but not of the float the asset sits in:macro_literal— a number a generated macro file defines, re-typed as a literal in the prose (macro_literal_violations, withignore_macrosandmin_integer_digits); off withcheck_literals = false
VisualGeometryToolkitEval.Reproduce.reproduce_main — Function
reproduce_main(args = ARGS; io = stdout, env = ENV, queue_states = nothing, job_states = nothing) -> IntThe command line of the runner (REPRODUCE_USAGE); returns the exit code. A launch checks the inputs first and runs nothing when one is missing (missing_inputs, exit 3; --allow-missing runs anyway), writes the run manifest and then submits (Slurm) or runs (local) the stages. queue_states and job_states stand in for squeue and sacct (slurm_queue_states, slurm_job_states).
VisualGeometryToolkitEval.Reproduce.reproduce_script — Method
reproduce_script(; julia_project = ".", graph = "<julia_project>/configs/reproduce.toml",
path = "<julia_project>/scripts/reproduce/reproduce.sh") -> StringThe text of a project's runner bootstrap (REPRODUCE_SCRIPT_TEMPLATE filled): run from anywhere in the repository, it changes to the repository root and runs the runner with --project=<julia_project> on graph, every path relative to the repository root. A project's suite checks its copy against it (read(path, String) == reproduce_script(; …)), and write_reproduce_script writes it.
VisualGeometryToolkitEval.Reproduce.reproduce_status — Method
reproduce_status(manifest; queue_states = nothing, job_states = nothing, env = ENV) -> Vector{NamedTuple}The state of every stage of the launch a run manifest records: (; stage, job_id, state, log, error). The state is stage_run_state's, from the stages' completion markers and the queue (queue_states(ids), by default slurm_queue_states), with its tasks counted by state ("failed (completed 2, failed 1)"); a task with no marker that is no longer queued is unknown, and where Slurm keeps accounting (job_states(ids), by default slurm_job_states: empty when sacct fails) its accounting state is added for that task, as information only. The log is the first failed or unknown task's (else the first's), with its first error line. A stage whose earlier phase failed (StageSpecphases, the runner's precompile among them: read from that phase's completion markers) never starts (its dependency is never satisfied): it is failed (its <phase> phase, job <id>, failed …), with that phase's log and error line.
VisualGeometryToolkitEval.Reproduce.resolve_roots — Method
resolve_roots(g, site; data_root = nothing, results_root = nothing, env = ENV) -> (data, results)The data and results roots of a launch: the argument, else the graph's variables in env (data_env, results_env), else the graph's [sites.<site>] roots (a relative one relative to the workdir; a project's own table, kept until the projects carry no machine paths), else the site profile's [storage] roots for the graph's project (site_root); "" when none gives one; a relative root is relative to the graph's workdir. The precedence is the one a driver's data_root() and results_root() follow (Experiments), which add the project's own data/exp_raw and data/exp_pro after the variables: one ladder, owned by the runner.
VisualGeometryToolkitEval.Reproduce.results_branch — Method
results_branch(dir, assets; base = <the checked-out branch>, commit = <HEAD>,
scratch_root = ENV["VGTK_RESULTS_WORKTREES"] or "<repository>-results" beside it,
stamp = now(UTC), run_keys = String[], tiers = Pair[], machine = gethostname(),
gpus = <the GPUs seen: nvidia-smi, else CUDA.jl>, slurm_jobs = [ENV["SLURM_JOB_ID"]] or String[],
wall_times = Pair[]) -> NamedTupleMake a full reproduction of the paper project at dir (a git repository or a directory inside one) a branch of its repository for review: results/<UTC yyyymmdd_HHMMSS>-<short commit>, commit the repository HEAD the run evaluated (or any revision, resolved; DrWatson's <sha>-dirty record marks the run dirty), created from base in a worktree of its own under scratch_root. The user's checkout is not touched. A failure after the worktree is created removes the worktree and the branch, and the error is rethrown.
assets are the reproduction's paper-facing renders, source => path with path where the paper reads it, relative to dir (the number macro files write_numbers and gather_numbers write, the generated tables, the figures). The branch's one commit holds them and, at dir's place in the repository:
RESULTS_MANIFEST(RUN.toml): the branch, the stamp, the codecommit(and whether the checkout wasdirty), thebaseand its commit, therun_keys(the heavy results stay in the results root under them), each data tier's hash (tiers,name => source: an artifact's git-tree-sha1, a tier directory'sTIER.toml,MANIFEST.tsvorMANIFESTsha256, or a file's sha256), themachine,gpus,slurm_jobs,wall_times(name => seconds) and the assets, andmissing: which oftiers,gpusandwall_timesthe run did not give (each a warning);RESULTS_COMPARISON(COMPARISON.md): every macro of every number file against the base branch's copy (read bytex_macro_values's parser) with its delta and IMPROVED, DEGRADED or UNCHANGED by the direction its provenance comment records (add_number!(…; better)), CHANGED without one, NEW or REMOVED; every asset's verdict against the base (verify_assets' rows); each changed PNG beside the base's copy (kept undercomparison/base/).
Returns (; branch, worktree, commit, numbers, assets): commit the branch's, numbers the comparison's rows (name, file, base, new, delta, better, verdict), assets the verify rows. accept_results brings the assets into the base; remove_results_worktree removes the worktree.
VisualGeometryToolkitEval.Reproduce.results_branch_assets — Method
results_branch_assets(g::StageGraph) -> Vector{Pair{String,String}}The renders of a launch as results_branch assets, render => path in the paper project, from the graph's [results_branch] assets: every file of the directory of from (relative to the paper project project) whose name matches its glob goes to the directory to under its own name; with submitted_only = true only those whose submitted copy is already there.
VisualGeometryToolkitEval.Reproduce.results_branch_of_run — Method
results_branch_of_run(manifest; scratch_root = nothing) -> NamedTupleThe results branch of the launch a run manifest records (results_branch): the renders of the graph's [results_branch] assets (results_branch_assets), the commit the launch ran (-dirty when it was), every stage's run key, the data tiers (the git-tree-sha1 of each entry of [results_branch] artifacts, an Artifacts.toml), the site as the machine, the GPUs this job sees, and every Slurm job of the launch. The branch name is added to the manifest (results_branch), which the runner keeps when it rewrites it.
VisualGeometryToolkitEval.Reproduce.run_local! — Method
run_local!(plan; io = stdout) -> Dict{String, Any}Runs the plan's stages in order on this machine, each item of an array in turn (SLURM_ARRAY_TASK_ID set, so a project's sbatch launcher runs unchanged under bash; a task of an array.each stage once per value, its variable set to it), its output in logs/<project>_<stage>[_<task>].log under the workdir. A completed stage of an earlier launch is skipped, as is every task it completed; the first failing task stops the run (a later stage depends on it). The manifest and the task's completion marker (stage_markers_dir) are written after every task. Returns the records.
A local run is no Slurm job, whatever the launching process is: every SLURM_* variable of the environment the plan was made in is dropped, and a stage sees only the ones this backend sets (SLURM_SUBMIT_DIR, an array's SLURM_ARRAY_TASK_ID, the site's SLURM_CPUS_PER_TASK) — launched from inside a Slurm job (a test suite's array task), the job's own task id and submit directory would otherwise reach every stage.
VisualGeometryToolkitEval.Reproduce.run_manifests — Method
run_manifests(dir) -> Vector{String}The run manifests of earlier launches in dir (<results root>/<manifest_dir>), oldest launch first: by the UTC launch stamp that begins each one's name (<stamp>-<commit>[-dirty][_<n>].toml), never by modification time (the results-branch stage rewrites its launch's manifest).
VisualGeometryToolkitEval.Reproduce.select_stages — Method
select_stages(g; only = String[], deps = true) -> Vector{String}The stages a run of graph g covers, in dependency order: every default stage when only is empty, else the stages only names; with deps, also every stage they run after, transitively (a finished one is reused by its run key, not rerun). Without deps the named stages alone: their prerequisites are taken as done.
VisualGeometryToolkitEval.Reproduce.shipped_sites — Method
shipped_sites() -> Vector{String}The names of the site profiles shipped in SITES_DIR (the template excluded).
VisualGeometryToolkitEval.Reproduce.site_root — Method
site_root(site, key, project) -> StringThe site profile's [storage] root key ("data_root", "results_root" or "artifact_store") for the project named project: its {project} replaced by the name (the artifact store is shared: it names no project); "" when the profile states none.
VisualGeometryToolkitEval.Reproduce.slurm_job_states — Method
slurm_job_states(ids; env = ENV) -> Dict{String, Vector{String}}The accounting state of every task of the Slurm jobs ids (sacct, found on env's PATH), by job id: ["COMPLETED", "FAILED", …]. Empty when sacct is not there, fails (a cluster without accounting storage: "Slurm accounting storage is disabled") or knows none of them. An extra of reproduce_status only, for a task that left no completion marker; whether a stage finished is never read from it (stage_run_state).
VisualGeometryToolkitEval.Reproduce.slurm_queue_states — Method
slurm_queue_states(ids; env = ENV) -> Dict{String, Dict{Int, String}}The tasks of the Slurm jobs ids that are still in the queue (squeue, found on env's PATH; it needs no accounting), by job id and task: Dict(0 => "RUNNING", 1 => "PENDING") (task 0 for a job that is not an array; task -1 for a range of tasks squeue does not list one by one). Only the states of a job not yet finished (PENDING, RUNNING, COMPLETING, …); a job that left the queue is absent (squeue's "Invalid job id specified"), as is every job when squeue is not there, and so is a task pending on a dependency that can never be satisfied (an earlier stage failed: it will never run). Any other failure of squeue (a slurmctld timeout, a controller restart) is an error: the queue's state is then unknown, and reading its jobs as gone would submit a stage that is still running again.
VisualGeometryToolkitEval.Reproduce.stage_each — Method
stage_each(g, stage, item; env = ENV) -> Vector{String}The values a task of an array stage loops over (array.each): those at the dotted key each.from of its item's config (a table's keys sorted by name, a list in its order), selected by the comma-separated env[each.env] when set; [] for a stage without each (the task runs its command once).
VisualGeometryToolkitEval.Reproduce.stage_items — Method
stage_items(g, stage; env = ENV) -> Vector{String}The items of an array stage, in task order: its items, or the values at the dotted key array.from of its config (a table's keys sorted by name, [cases] among them; a list of strings in its order); the comma-separated env[select_env], when set, selects a subset (a table's keys sorted, a list's in the order given; a value the config does not hold is an error). [""] for a stage that is not an array.
VisualGeometryToolkitEval.Reproduce.stage_markers — Method
stage_markers(dir; job_id = nothing) -> Dict{Int, Dict{String, Any}}The completion markers in a stage's marker directory (stage_markers_dir), by task; with job_id, only those that job wrote (an earlier job of the same stage, resubmitted, is not this one).
VisualGeometryToolkitEval.Reproduce.stage_markers_dir — Method
stage_markers_dir(manifest, stage) -> StringThe directory of a stage's completion markers in the launch whose run manifest is manifest: <manifest without .toml>/<stage>/. Each task of the stage writes task-<t>.toml there when it ends (t its array task, 0 for a stage that is not an array): state ("completed" or "failed"), exit_status, start and end (UTC), job_id, commit, key (the stage's run key), stage, task, host, restart_count (Slurm's SLURM_RESTART_COUNT, 0 for a first attempt), and signal when the job received one (TERM, then state = "failed"; or a signal the launcher's --signal directive asks for, forwarded to it). On a Slurm site the job writes it itself (a wrapper the runner submits around the stage's launcher, kept in the same directory with the launcher's copy as submitted); the local backend writes it after each task. Every marker is written atomically (a temporary file in the directory, renamed into place): a reader sees a whole marker or none.
VisualGeometryToolkitEval.Reproduce.stage_run_state — Function
stage_run_state(record, queue = Dict()) -> (; state, tasks)What the run of a stage that a run manifest records (record, one of its [stages]) came to, read from the stage's completion markers (stage_markers) and the queue (queue: slurm_queue_states), never from accounting. tasks maps each task to its state: its job's state while the queue holds the task ("PENDING", "RUNNING", …), else its marker's ("completed", "failed"), else "unknown". The queue comes first because Slurm requeues a task under the same job id (preemption with PreemptMode=REQUEUE, scontrol requeue): the attempt it stopped left a "failed" marker, and the task is pending or running again. A task that is finishing (COMPLETING) reads its marker when it has one. state is
"failed"when a task's marker says so and the queue no longer holds it;"unknown"when a task has no marker and its job is no longer queued: it died without writing one (a node failure, a kill past the time limit's grace) or its markers are gone. Never taken as finished: a later launch runs the stage again;"in flight"when every task without a marker is still queued;"completed"when every task's marker says so.
A record of the local backend (no job id), which the runner itself writes after every task, is read as it stands: "completed" or "reused" is completed, "failed" failed, anything else (a run stopped mid-way) unknown.
VisualGeometryToolkitEval.Reproduce.submit_slurm! — Method
submit_slurm!(plan; io = stdout) -> Dict{String, Any}Submits the plan's stages in order with sbatch, each after the jobs of the stages it depends on (--dependency=afterok; a GPU array also after the GPU array submitted before it, afterany, when the site sets gpu_limit); a reused stage is not submitted (a completed one is no dependency, one in flight is depended on). A stage's earlier phases (StageSpecphases, the runner's precompile among them) are submitted first, each through the wrapper and after the one before it, the stage's job after the last of them; their ids are the record's phase_jobs. Each stage is submitted through a wrapper that runs it and writes each task's completion marker (stage_markers_dir): the stage's own launcher, copied as submitted, with its #SBATCH directives (a signal its --signal directive asks for is forwarded to it), named <project>-<stage> when neither the stage nor its launcher names it. sbatch runs without the SLURM_* variables of the process that launches it (SLURM_CONF kept): launched from inside a Slurm job, that job's array task and job id would otherwise reach every stage's job. Returns the manifest records; the manifest is written after every submission, so a failed sbatch leaves the record of what was submitted.
VisualGeometryToolkitEval.Reproduce.tex_macro_values — Method
tex_macro_values(path) -> Vector{Pair{String,String}}The argument-free macros a tex file defines (\newcommand, \renewcommand, \providecommand, \def), as name => value.
VisualGeometryToolkitEval.Reproduce.tex_number — Method
tex_number(x; digits = nothing, sigdigits = nothing, percent = false, group_from = 0,
unit = "") -> String
tex_number(lo, hi; range_separator = "--", <the same keywords>) -> StringA number as the paper prints it, the body of its macro. digits fixed decimals (Printf's %.<digits>f: the exact value rounded to nearest, ties to even); or sigdigits significant digits with trailing zeros dropped, never an exponent; an integer value (3, 3.0) needs neither. A zero never carries a sign (-0.001 at 2 digits is 0.00). percent prints 100x followed by \%; group_from groups the integer part in threes at thin spaces (\,) from that many digits on (4: 1\,234; 0: never); unit follows the number after a thin space (1.5\,mm). Two values are a range, lo and hi each typeset so, joined by range_separator (LaTeX's en dash by default; in math mode, "\text{--}"), the unit once after hi.
A NaN or an infinity is an ArgumentError: a missing evaluation fails its stage, it never reaches the manuscript. So is a non-integer x without digits or sigdigits, or both given.
VisualGeometryToolkitEval.Reproduce.uncited_macros — Method
uncited_macros(numbers_path, papers; root = dirname(first(papers)), search_dirs = String[],
allow = Dict{String,String}()) -> Vector{String}The macros the numbers file numbers_path defines that the paper does not cite: the tex files the paper reads are the ones paper_asset_graph visits from the top-level papers (with root and search_dirs as there), comments ignored, the numbers file itself excluded; a citation is the macro name not followed by a letter (\nA is not cited by \nAb). The allow table (macro name without backslash => reason) exempts a macro; an allow entry naming no macro of the file is returned as stale allow: <name>. Sorted. A paper that does not read numbers_path is an ArgumentError.
VisualGeometryToolkitEval.Reproduce.verify_assets — Method
verify_assets(outdir, paperdir, names) -> Vector{NamedTuple}Compare each rendered outdir/<name> with the submitted paperdir/<name> and write the report outdir/verify.tsv (VERIFY_REPORT): one row per asset with asset, submitted, identical, same_numbers, provenance_only (a .tex that differs from the submitted copy in its % provenance \<name>: … lines alone, which deploy_assets deploys without sign-off), blocks_new, blocks_old, max_abs_diff, bytes_new and bytes_old. A .tex asset is compared by its numbers as well as its bytes (compare_fragments); any other asset by its bytes. Each verdict is printed. A missing render is an error; a missing submitted copy is reported (submitted = false), not an error. Returns the rows.
VisualGeometryToolkitEval.Reproduce.write_numbers — Method
write_numbers(path, nums; shared = false, order = :name) -> pathWrite nums as a number-macro fragment at path (a file beside the driver's results, never in the paper): a header naming the driver, then per macro its provenance comment (% provenance \<name>: driver=…; config=…; run=…; commit=…[; label=…][; better=…]) and its \newcommand{\<name>}{<body>} ([<arity>] for a macro with parameters, add_macro!), sorted by name (order = :name) or in the order they were added (order = :added). The same numbers write the same bytes; the file is written whole (a temporary file renamed into place).
shared = true: path is ONE file several drivers write (CaliBlobs icra2026's trajectory macros). Each driver's entries are its own section, under its header; this call rewrites the driver's section alone and keeps every other section as it is, the sections ordered by driver, so the file is the same whatever order the drivers ran in; drivers that finish at once take turns (the read, the merge and the rename hold a pid lock, <path>.lock), so none drops another's section. A macro another section defines is an ArgumentError (nothing is written). gather_numbers gathers the fragments, comments included, into the paper's numbers file; tex_macro_values reads the macros back.
VisualGeometryToolkitEval.Reproduce.write_numbers — Method
write_numbers(io::IO, nums; order = :name) -> ioWrite nums to io exactly as write_numbers(path, nums; order) writes its file (one implementation): the header naming the driver, then per macro its provenance comment and its \newcommand, sorted by name or in the order they were added. The macros a figure defines in its own body (an add_macro! cell) are inlined this way: sprint(io -> write_numbers(io, nums; order = :added)). A file several drivers share is the path form's (shared = true).
VisualGeometryToolkitEval.Reproduce.write_reproduce_script — Method
write_reproduce_script(root; julia_project = ".", graph, path) -> StringWrite the runner bootstrap (reproduce_script) of the project in the repository root to <root>/<path> (by default <julia_project>/scripts/reproduce/reproduce.sh), executable. Returns its path.
Internals
These names are not public; they are listed for contributors.