The Artifact Store
Published artifact trees — a paper project's DataArtifacts tiers, a package's model weights — are installed into the artifact store, a directory of <git-tree-sha1>/ trees in Pkg's layout on storage that every job of a machine shares, and never into a cluster's Julia depot (node-local and evicted, or a home with a quota). artifact_store resolves it, the first rung that names one:
| Rung | Set by |
|---|---|
VGTK_ARTIFACT_STORE | the environment (a reproduction's jobs get it from their site) |
the site profile VGTK_SITE names | its [storage] artifact_store (VisualGeometryToolkitEval's site profiles, registered with set_site_artifact_store!) |
the preference artifact_store | Preferences.set_preferences!(VisualGeometryToolkitCore, "artifact_store" => dir) |
| the depot | Pkg's default; refused where the site profile sets [storage] refuse_depot = true (VisualGeometryToolkitEval's vcai); where no profile states it, where NODE_CACHE is set (that cluster's job environment) |
install_artifact installs by name from an Artifacts.toml into the store, inside with_artifact_store (Pkg's with_artifacts_directory under one lock), so Pkg's sha256 and git-tree-sha1 checks and its atomic move into place hold; a lock per tree (a directory made atomically, safe on a shared NFS store, taken over from a holder that is gone) makes the jobs that start together download a tree once. A model published as a single file goes in with store_file (verified by download_verified), and a tree Pkg cannot unpack itself (ONNX Runtime's CUDA build, merged from conda packages) is built into the store by install_built_artifact, checked against its pinned git-tree-sha1. A machine whose depot already holds trees moves them with migrate_artifacts, and store_name_clashes lists a name the store holds under two trees.
import VisualGeometryToolkitCore as VgtkCore
VgtkCore.artifact_store() # ArtifactStore(path, source)
tree = VgtkCore.install_artifact("tier_a", "Artifacts.toml")
VgtkCore.migrate_artifacts("Artifacts.toml") # the depot's copies into the store, hash-checkedVisualGeometryToolkitCore.ArtifactStore — Type
ArtifactStore(path, source)Where artifact trees are installed and read: path, the directory holding the <git-tree-sha1>/ trees (Pkg's layout), and source, the rung that named it — :env (VGTK_ARTIFACT_STORE), :site (the site profile VGTK_SITE names), :preference (this package's preference artifact_store) or :depot (the first depot's artifacts/, Pkg's default). artifact_store resolves it.
VisualGeometryToolkitCore.artifact_store — Method
artifact_store() -> ArtifactStoreThe artifact store of this process, the first rung that names one:
ENV["VGTK_ARTIFACT_STORE"];- the site profile
ENV["VGTK_SITE"]names, its[storage] artifact_store(VisualGeometryToolkitEval's site profiles;set_site_artifact_store!); - this package's preference
artifact_store(Preferences.set_preferences!( VisualGeometryToolkitCore, "artifact_store" => dir), LocalPreferences.toml); - the depot (
<first depot>/artifacts, Pkg's own) — refused where the site profile sets[storage] refuse_depot = true(VisualGeometryToolkitEval'svcai: its depot is node-local and evicted), allowed where it setsfalse, and, where no profile says (noVGTK_SITE, a site name and VisualGeometryToolkitEval not loaded, or a profile without the key), refused whereNODE_CACHEis set (that cluster's job environment): one of the rungs above must name the store there.
Every rung but the depot names a directory on storage that outlives a job and is shared by its machines; its trees are content-addressed by their git-tree-sha1, so every project and package shares one copy of a tree.
VisualGeometryToolkitCore.download_verified — Method
download_verified(url, sha256, path) -> Stringurl downloaded to path, checked against its sha256 (hex): a mismatch is an error and leaves no file at path. The bytes land in a temporary file in path's directory and are moved into place, so no partial file is ever at path. The stack's one verified download of a single file (store_file's; VisualGeometryToolkitFeatures' ONNX Runtime packages).
VisualGeometryToolkitCore.huggingface_token — Method
huggingface_token() -> Union{String, Nothing}The user's Hugging Face token: ENV["HF_TOKEN"], else the file hf auth login writes ($HF_HOME/token, HF_HOME defaulting to ~/.cache/huggingface), else nothing. install_artifact sends it to huggingface.co only.
VisualGeometryToolkitCore.install_artifact — Method
install_artifact(name, toml; store = artifact_store()) -> StringInstall artifact name of toml into the artifact store (artifact_store: download, verify sha256 and git-tree-sha1, unpack, move into place atomically) unless present there, and return its tree. Processes installing the same tree into a shared store take turns (a lock per tree, safe on NFS: one downloads, the others then read it). A Hugging Face tarball is downloaded with the user's Hugging Face token when there is one (huggingface_token), so a private dataset installs for whoever has access to it. A placeholder URL (an UPPERCASE token in place of the Zenodo id or the Hugging Face revision: the record is not published or uploaded) and a failed download (an unpublished Zenodo draft, a private dataset without a token) are refused with the way out. A tree in the depot but not in the store is not read: move it with [`migrateartifacts`](@ref).
VisualGeometryToolkitCore.install_built_artifact — Method
install_built_artifact(build!, name, hash; store = artifact_store()) -> StringA tree Pkg cannot download and unpack itself (ONNX Runtime's CUDA build, merged from conda packages: VisualGeometryToolkitFeatures) installed into the artifact store under its pinned git-tree-sha1 hash (a Base.SHA1) unless present there, and its tree returned: build!(dir) fills a new directory in the store's directory, whose tree hash must be hash (another tree is removed, and is an error; a tree already in the store is never touched), and then moved into place; the build runs outside Pkg's directory override. Processes building the same tree into a shared store take turns under the tree's lock, as install_artifact's do (one builds, the others then read it), and the store records the tree under name (store_name_clashes).
VisualGeometryToolkitCore.is_published_url — Method
is_published_url(url) -> BoolWhether url is a live download URL: a record not yet published or uploaded binds its placeholder, an UPPERCASE token (`ZENODOID), as a path segment where the Zenodo id or the Hugging Face revision goes (VisualGeometryToolkitEval's DataArtifacts). [install_artifact`](@ref) refuses such a URL, and a paper project's release check (VisualGeometryToolkitEval's R3) flags it.
VisualGeometryToolkitCore.migrate_artifacts — Method
migrate_artifacts(toml; store = artifact_store(), names = keys of toml, remove = false)
-> Vector{Pair{String, Symbol}}Move the artifacts of toml that the depot holds into the store, so a machine that installed them before the store existed does not download them again: each tree is copied into a temporary directory in the store, its git-tree-sha1 recomputed and checked, and moved into place atomically under the tree's install lock (the depots of DEPOT_PATH tried in order, the first copy that hashes to it migrated); with remove = true the depots' copies are deleted once the store holds the tree. Returns name => status per name: :migrated, :present (the store had it), :absent (no depot holds it either) or :mismatch (no depot copy hashes to its git-tree-sha1: left alone, never copied). The store must not be the depot.
VisualGeometryToolkitCore.set_site_artifact_store! — Method
set_site_artifact_store!(f) -> fRegister the site rung of artifact_store: f(site) returns the storage of the site profile site names (the value of VGTK_SITE) as (; artifact_store, refuse_depot) — its [storage] artifact_store ("" when it names none) and whether it refuses the depot ([storage] refuse_depot, nothing when the profile does not say) — or nothing when no profile has that name; a path string is read as the artifact store alone. VisualGeometryToolkitEval, which owns the site profiles, registers its reader when it loads; without it a VGTK_SITE that names a profile FILE (a path ending in .toml) is still read here (its [storage]), and a site NAME gives no rung: the runner gives its jobs VGTK_ARTIFACT_STORE from the profile, so a package without VisualGeometryToolkitEval (BlobBoards) installs into the site's store there too.
VisualGeometryToolkitCore.store_file — Method
store_file(url, sha256, file; depot_dir, store = artifact_store()) -> StringA single file that is not an artifact tree (a model's weights a release publishes as a file, LoMa's ONNX graphs) in the artifact store: downloaded once to <store>/files/<sha256>/<file>, checked against its sha256 (a mismatch is an error and leaves no file) and moved into place atomically, under the same per-file lock as a tree. With the depot store its home is depot_dir (the caller's scratch space), <depot_dir>/<file>.
VisualGeometryToolkitCore.store_name_clashes — Function
store_name_clashes(store = artifact_store()) -> Dict{String, Vector{String}}The artifact names the store installed under more than one tree: name => its git-tree-sha1s. One name means one tree stack-wide (a tier re-cut under its old name is two data under one name); empty when every name has one tree.
VisualGeometryToolkitCore.with_artifact_store — Function
with_artifact_store(f, store = artifact_store())Run f() with Pkg's artifact functions (artifact_exists, artifact_path, ensure_artifact_installed, create_artifact, …) reading and writing store (an ArtifactStore or a directory): Pkg's with_artifacts_directory, under one process-wide lock (Pkg's override is a global), nesting into the same store allowed: the block's task and the tasks it spawns enter it without the lock, and another store from inside a block is an error (it would wait on the lock the block holds). The depot store runs f without an override. The one wrapper of Pkg's override in the stack: DataArtifacts' packer stages through it.