API: DataArtifacts

Published input data as Julia artifacts. The catalog format is described in The data catalog.

Public names

VisualGeometryToolkitEval.DataArtifacts — Module
DataArtifacts

A paper project's published input data as lazy Julia artifacts. The catalog (load_catalog, DataCatalog) states what each artifact holds, where it mounts, which Zenodo record carries it and which config of record its file list derives from. The author packs and binds the artifacts (pack_artifacts); a consumer fetches them into the artifact view that the drivers read (fetch_artifacts!). catalog_violations checks a catalog against its bindings.

source
VisualGeometryToolkitEval.DataArtifacts.DataCatalog — Type
DataCatalog

A loaded data-artifact catalog (load_catalog): path (the catalog TOML), artifacts_toml (the Artifacts.toml its artifacts are bound in, after the artifacts_toml_env override), configs (the directory its config names resolve in: the catalog's own), records and artifacts (the tables).

source
VisualGeometryToolkitEval.DataArtifacts.artifact_files — Method
artifact_files(cat, name; kinds = Dict()) -> (; mount, root, files, pieces)

The files of artifact name — a catalog entry (all its parts), or one part of a split entry (artifact_parts): paths RELATIVE to its mount root root (dataroot() for mount = "data", resultsroot() for "results"), derived from the config of record the entry names (or, for tree, its directory). kinds adds project kinds, kind => (cfg, entry, root) -> files (cfg is nothing when the entry names no config). Every file must exist under root — the author's tree the artifact is packed from. pieces maps each of files (its path in the packed tree) to the bytes of the author's file it holds: the whole file, or one chunk of a sub-part (max_part_bytes).

source
VisualGeometryToolkitEval.DataArtifacts.artifact_parts — Method
artifact_parts(cat, name) -> Vector{String}

The Artifacts.toml names catalog entry name is bound as: [name], or for an entry with split_by = "scene" one artifact per scene of its splits, <name>_<scene>, preceded by <name>_meta (the files no scene owns) when the kind has such files (a pool tree's run records; a roma route's manifest). With max_part_bytes, a scene over the cap is <name>_<scene>_part1..k instead (the rule in the catalog header; the author's sizes derive k, a consumer reads it from the bindings).

source
VisualGeometryToolkitEval.DataArtifacts.catalog_view — Method
catalog_view(cat; dir = "", default = required_artifacts(cat)) -> ArtifactView

The project's artifact view of its catalog: its Artifacts.toml, each artifact mounted at its mount_dir, installing default when asked for no tier in particular. The binding a project gives Experiments.set_project_root!(…; artifacts = catalog_view(project_catalog())).

source
VisualGeometryToolkitEval.DataArtifacts.catalog_violations — Method
catalog_violations(cat; kinds = String[]) -> Vector{String}

Every inconsistency between the catalog and its Artifacts.toml, empty when none (a project's test, in one line): the two name the same artifacts; each entry has the common keys, a known kind (or one of the project's kinds), a filter only on a tree entry and as a non-empty list of globs, a known mount, a defined record and an existing config; each record has a placeholder (A_TOKEN_LIKE_THIS), a known host and its keys — on Zenodo, once it exists, a numeric id; on Hugging Face a repo and, once uploaded, a 40-hex commit revision; each binding is lazy, hash-pinned and downloads from its record's host (download_url) — the record's id or revision when it has one, else its placeholder; never a local path — under its published_name: an entry's published_name, when present, is a file name (no /, not the binding's own name, no other entry's name or published name), and the URL's file is checked against it instead of the binding name. A produced entry (kind = "produced") has exactly its keys (producer, covers — a non-empty list of relative input paths —, tier, what, provenance) and no binding; that its producer is a stage of the project's graph whose inputs are published or produced tiers is checked with the graph (PaperProjects' paper_project_violations, A1).

source
VisualGeometryToolkitEval.DataArtifacts.download_url — Method
download_url(cat, name; record_ids = Dict()) -> String

The download URL of artifact name on its record's host: zenodo_url for a host = "zenodo" record (the default), huggingface_url for a host = "huggingface" one. record_ids maps a record group to its Zenodo id or its Hugging Face revision (the author's rebind after an upload); without one, the record's id/revision, else its placeholder (which fetch_artifacts! refuses to download). A part of a split entry is hosted in its entry's record under its own name.

source
VisualGeometryToolkitEval.DataArtifacts.fetch_artifacts! — Function
fetch_artifacts!(cat, names = required_artifacts(cat)) -> Vector{NamedTuple}

Install names (VisualGeometryToolkitCore's install_artifact: download, verify, unpack into the artifact store, artifact_store(); a placeholder URL is refused with the way out) and link each tree into its mount root, at the entry's mount_dir: data artifacts into the project's data/exp_raw as an artifact view — skipped when a data root is already bound (VGTKDATAROOT, or an author's own data/expraw) — and results artifacts into resultsroot() (data/exppro is created when nothing binds one). Idempotent. One (; name, tree, mount, dst, linked) per artifact; dst is nothing where nothing was linked. A split entry ([`artifactparts](@ref)) installs and links every part, one row each (the parts' trees share directories; the view links per file); a part's own name fetches that part alone. A chunked file (maxpartbytes`) is reassembled in the view once all its chunks are linked: the chunks concatenated in order into the file (a regular file; the chunk links stay), each chunk already verified by its artifact's tree hash.

source
VisualGeometryToolkitEval.DataArtifacts.fetch_data! — Function
fetch_data!(cat = project_catalog(); argv = ARGS, io = stdout) -> Union{Vector{NamedTuple}, Nothing}

A paper project's scripts/fetch_data.jl, the new-machine setup step: the command line argv read as

(none)          the required artifacts ([`required_artifacts`](@ref))
--all           every catalogued artifact, required or not
--only=a,b      the named entries (or parts)
--list          the catalog, nothing fetched: per entry whether it is required,
                its mount and what it is, then each of its parts
                ([`artifact_parts`](@ref): a split entry's parts and sub-parts)
                with its download URL, or "(not bound)"

then fetch_artifacts! of them (a split entry all its parts, a chunked file reassembled), one line per artifact to io (its new links and where, or that its root is already bound), the two roots (data_root(), results_root()) and the artifact store, and the line FETCH_DATA_DONE. A produced entry (kind = "produced") is never fetched: --list and a request of it name its producing stage. Returns fetch_artifacts!'s rows (nothing for --list). A script is the one call fetch_data!(project_catalog()) after @quickactivate.

source
VisualGeometryToolkitEval.DataArtifacts.huggingface_url — Method
huggingface_url(cat, name; record_ids = Dict()) -> String

The Hugging Face URL of artifact name in its record's dataset repo repo: https://huggingface.co/datasets/<repo>/resolve/<revision>/<file>.tar.gz (<file> its published_name), with the revision record_ids gives the entry's record group, else the group's revision, else its placeholder. The revision is a commit, never a branch: a binding pins its bytes (its sha256), and a branch moves while the data is edited.

source
VisualGeometryToolkitEval.DataArtifacts.load_catalog — Method
load_catalog(path) -> DataCatalog

Read the catalog at path. Its bindings are [catalog] artifacts_toml (relative to the catalog; default ../Artifacts.toml, the project root's), unless the variable [catalog] artifacts_toml_env names another copy.

source
VisualGeometryToolkitEval.DataArtifacts.mount_dir — Method
mount_dir(cat, name) -> String

Where artifact (or part) name mounts under its mount root: the catalog entry's mount_dir (a relative directory, "absolute_pose_experiments"), "" (the root itself) when it names none. A tier's paths are relative to it (catalog_view, Experiments.data_root(; tier)), so a project carries no mount helper of its own.

source
VisualGeometryToolkitEval.DataArtifacts.pack_artifacts — Function
pack_artifacts(cat, names = sort(collect(keys(cat.artifacts))); stage,
               record_ids = Dict(), local_toml = "", kinds = Dict()) -> Vector{NamedTuple}

AUTHOR-SIDE: pack each artifact from the author's data/results roots (artifact_files) with Pkg.Artifacts — create_artifact in a scratch artifacts directory under stage (never the depot), archive_artifact to <stage>/tarballs/<name>.tar.gz — concurrently, then bind_artifact! each into the catalog's Artifacts.toml (lazy = true, the tree hash, the tarball's sha256, its record's download_url). The file's comment header (its leading comment and blank lines), which Pkg's writer drops, is kept. A staged tarball whose file listing is current is reused. local_toml also writes a copy of the bindings with file:// URLs to the staged tarballs (the fresh-machine test's; never committed).

source
VisualGeometryToolkitEval.DataArtifacts.project_catalog — Function
project_catalog(root = project_root()) -> DataCatalog

The catalog of the paper project at root (Experiments' project_root(), the bound project): its configs/data_artifacts.toml, loaded by load_catalog, so bound to the project's Artifacts.toml or the copy the catalog's artifacts_toml_env variable names.

source
VisualGeometryToolkitEval.DataArtifacts.published_name — Method
published_name(cat, name) -> String

The file name (without .tar.gz) under which the record's host publishes the bound artifact name (an entry or one part of a split entry): name itself, or, when its entry declares published_name, name with the entry's name replaced by it (a name published before the stack's terminology rules: the binding is renamed, the published file cannot be).

source
VisualGeometryToolkitEval.DataArtifacts.split_scenes — Method
split_scenes(side) -> Vector{String}

The scenes of one side of a registered scene split, side = "<split>:validation" or "<split>:test" (configs/splits.toml, Tuning.registered_split): how a catalog entry's (and a pool producer config's) splits names its scenes.

source

Internals

These names are not public; they are listed for contributors.

VisualGeometryToolkitEval.DataArtifacts._glob_regex — Method
_glob_regex(glob) -> Regex

The whole-path regex of a tree entry's filter glob over /-separated relative paths: **/ any run of whole directories (none included), a trailing /** everything below, * any run of characters but /, ? one of them; every other character literal.

source