The Artifact Store

Published artifact trees — a paper project's DataArtifacts tiers, a package's model weights — are installed into the artifact store, a directory of <git-tree-sha1>/ trees in Pkg's layout on storage that every job of a machine shares, and never into a cluster's Julia depot (node-local and evicted, or a home with a quota). artifact_store resolves it, the first rung that names one:

RungSet by
VGTK_ARTIFACT_STOREthe environment (a reproduction's jobs get it from their site)
the site profile VGTK_SITE namesits [storage] artifact_store (VisualGeometryToolkitEval's site profiles, registered with set_site_artifact_store!)
the preference artifact_storePreferences.set_preferences!(VisualGeometryToolkitCore, "artifact_store" => dir)
the depotPkg's default; refused where the site profile sets [storage] refuse_depot = true (VisualGeometryToolkitEval's vcai); where no profile states it, where NODE_CACHE is set (that cluster's job environment)

install_artifact installs by name from an Artifacts.toml into the store, inside with_artifact_store (Pkg's with_artifacts_directory under one lock), so Pkg's sha256 and git-tree-sha1 checks and its atomic move into place hold; a lock per tree (a directory made atomically, safe on a shared NFS store, taken over from a holder that is gone) makes the jobs that start together download a tree once. A model published as a single file goes in with store_file (verified by download_verified), and a tree Pkg cannot unpack itself (ONNX Runtime's CUDA build, merged from conda packages) is built into the store by install_built_artifact, checked against its pinned git-tree-sha1. A machine whose depot already holds trees moves them with migrate_artifacts, and store_name_clashes lists a name the store holds under two trees.

import VisualGeometryToolkitCore as VgtkCore
VgtkCore.artifact_store()                       # ArtifactStore(path, source)
tree = VgtkCore.install_artifact("tier_a", "Artifacts.toml")
VgtkCore.migrate_artifacts("Artifacts.toml")   # the depot's copies into the store, hash-checked
VisualGeometryToolkitCore.ArtifactStore — Type
ArtifactStore(path, source)

Where artifact trees are installed and read: path, the directory holding the <git-tree-sha1>/ trees (Pkg's layout), and source, the rung that named it — :env (VGTK_ARTIFACT_STORE), :site (the site profile VGTK_SITE names), :preference (this package's preference artifact_store) or :depot (the first depot's artifacts/, Pkg's default). artifact_store resolves it.

source
VisualGeometryToolkitCore.artifact_store — Method
artifact_store() -> ArtifactStore

The artifact store of this process, the first rung that names one:

  1. ENV["VGTK_ARTIFACT_STORE"];
  2. the site profile ENV["VGTK_SITE"] names, its [storage] artifact_store (VisualGeometryToolkitEval's site profiles; set_site_artifact_store!);
  3. this package's preference artifact_store (Preferences.set_preferences!( VisualGeometryToolkitCore, "artifact_store" => dir), LocalPreferences.toml);
  4. the depot (<first depot>/artifacts, Pkg's own) — refused where the site profile sets [storage] refuse_depot = true (VisualGeometryToolkitEval's vcai: its depot is node-local and evicted), allowed where it sets false, and, where no profile says (no VGTK_SITE, a site name and VisualGeometryToolkitEval not loaded, or a profile without the key), refused where NODE_CACHE is set (that cluster's job environment): one of the rungs above must name the store there.

Every rung but the depot names a directory on storage that outlives a job and is shared by its machines; its trees are content-addressed by their git-tree-sha1, so every project and package shares one copy of a tree.

source
VisualGeometryToolkitCore.download_verified — Method
download_verified(url, sha256, path) -> String

url downloaded to path, checked against its sha256 (hex): a mismatch is an error and leaves no file at path. The bytes land in a temporary file in path's directory and are moved into place, so no partial file is ever at path. The stack's one verified download of a single file (store_file's; VisualGeometryToolkitFeatures' ONNX Runtime packages).

source
VisualGeometryToolkitCore.install_artifact — Method
install_artifact(name, toml; store = artifact_store()) -> String

Install artifact name of toml into the artifact store (artifact_store: download, verify sha256 and git-tree-sha1, unpack, move into place atomically) unless present there, and return its tree. Processes installing the same tree into a shared store take turns (a lock per tree, safe on NFS: one downloads, the others then read it). A Hugging Face tarball is downloaded with the user's Hugging Face token when there is one (huggingface_token), so a private dataset installs for whoever has access to it. A placeholder URL (an UPPERCASE token in place of the Zenodo id or the Hugging Face revision: the record is not published or uploaded) and a failed download (an unpublished Zenodo draft, a private dataset without a token) are refused with the way out. A tree in the depot but not in the store is not read: move it with [`migrateartifacts`](@ref).

source
VisualGeometryToolkitCore.install_built_artifact — Method
install_built_artifact(build!, name, hash; store = artifact_store()) -> String

A tree Pkg cannot download and unpack itself (ONNX Runtime's CUDA build, merged from conda packages: VisualGeometryToolkitFeatures) installed into the artifact store under its pinned git-tree-sha1 hash (a Base.SHA1) unless present there, and its tree returned: build!(dir) fills a new directory in the store's directory, whose tree hash must be hash (another tree is removed, and is an error; a tree already in the store is never touched), and then moved into place; the build runs outside Pkg's directory override. Processes building the same tree into a shared store take turns under the tree's lock, as install_artifact's do (one builds, the others then read it), and the store records the tree under name (store_name_clashes).

source
VisualGeometryToolkitCore.is_published_url — Method
is_published_url(url) -> Bool

Whether url is a live download URL: a record not yet published or uploaded binds its placeholder, an UPPERCASE token (`ZENODOID), as a path segment where the Zenodo id or the Hugging Face revision goes (VisualGeometryToolkitEval's DataArtifacts). [install_artifact`](@ref) refuses such a URL, and a paper project's release check (VisualGeometryToolkitEval's R3) flags it.

source
VisualGeometryToolkitCore.migrate_artifacts — Method
migrate_artifacts(toml; store = artifact_store(), names = keys of toml, remove = false)
    -> Vector{Pair{String, Symbol}}

Move the artifacts of toml that the depot holds into the store, so a machine that installed them before the store existed does not download them again: each tree is copied into a temporary directory in the store, its git-tree-sha1 recomputed and checked, and moved into place atomically under the tree's install lock (the depots of DEPOT_PATH tried in order, the first copy that hashes to it migrated); with remove = true the depots' copies are deleted once the store holds the tree. Returns name => status per name: :migrated, :present (the store had it), :absent (no depot holds it either) or :mismatch (no depot copy hashes to its git-tree-sha1: left alone, never copied). The store must not be the depot.

source
VisualGeometryToolkitCore.set_site_artifact_store! — Method
set_site_artifact_store!(f) -> f

Register the site rung of artifact_store: f(site) returns the storage of the site profile site names (the value of VGTK_SITE) as (; artifact_store, refuse_depot) — its [storage] artifact_store ("" when it names none) and whether it refuses the depot ([storage] refuse_depot, nothing when the profile does not say) — or nothing when no profile has that name; a path string is read as the artifact store alone. VisualGeometryToolkitEval, which owns the site profiles, registers its reader when it loads; without it a VGTK_SITE that names a profile FILE (a path ending in .toml) is still read here (its [storage]), and a site NAME gives no rung: the runner gives its jobs VGTK_ARTIFACT_STORE from the profile, so a package without VisualGeometryToolkitEval (BlobBoards) installs into the site's store there too.

source
VisualGeometryToolkitCore.store_file — Method
store_file(url, sha256, file; depot_dir, store = artifact_store()) -> String

A single file that is not an artifact tree (a model's weights a release publishes as a file, LoMa's ONNX graphs) in the artifact store: downloaded once to <store>/files/<sha256>/<file>, checked against its sha256 (a mismatch is an error and leaves no file) and moved into place atomically, under the same per-file lock as a tree. With the depot store its home is depot_dir (the caller's scratch space), <depot_dir>/<file>.

source
VisualGeometryToolkitCore.store_name_clashes — Function
store_name_clashes(store = artifact_store()) -> Dict{String, Vector{String}}

The artifact names the store installed under more than one tree: name => its git-tree-sha1s. One name means one tree stack-wide (a tier re-cut under its old name is two data under one name); empty when every name has one tree.

source
VisualGeometryToolkitCore.with_artifact_store — Function
with_artifact_store(f, store = artifact_store())

Run f() with Pkg's artifact functions (artifact_exists, artifact_path, ensure_artifact_installed, create_artifact, …) reading and writing store (an ArtifactStore or a directory): Pkg's with_artifacts_directory, under one process-wide lock (Pkg's override is a global), nesting into the same store allowed: the block's task and the tasks it spawns enter it without the lock, and another store from inside a block is an error (it would wait on the lock the block holds). The depot store runs f without an override. The one wrapper of Pkg's override in the stack: DataArtifacts' packer stages through it.

source