Getting started

This page walks through the two things most users do first: summarize the pose errors of an evaluation, and tune a hyperparameter by cross-validation. The examples run without any data.

Each part of the package is a submodule, and its names are imported from it:

using VisualGeometryToolkitEval.Stats: finite_median, eligible_maa, bootstrap_ci, stable_seed
using VisualGeometryToolkitEval.Tuning: tuning_spec, grid_spec, grid
using VisualGeometryToolkitEval.Evaluation: coarse_fine_cv
using Random: Xoshiro
using TOML

Summarizing pose errors

An evaluation produces one pose error per image pair, in degrees. A pair where the estimator returned no model has the error NaN.

errors = [0.4, 1.2, NaN, 0.8, 3.5, 0.6, 2.1]
finite_median(errors)
1.0

finite_median drops the non-finite errors before it takes the median. The mean average accuracy (mAA) instead counts a failed pair as zero accuracy, because the pair was eligible to run:

eligible_maa(errors; max_error = 10.0)
0.7342857142857142

bootstrap_ci gives a bias-corrected and accelerated (BCa) bootstrap interval of any statistic. The random number generator (RNG) seeds the resampling, so the interval is reproducible:

bootstrap_ci(finite_median, Xoshiro(1), errors)
(0.4, 2.1)

Every random sample of the stack is seeded by VisualGeometryToolkitCore's stable_seed (Stats re-exports it), which derives a seed from a spec's seed and a name. Adding or removing another scene therefore leaves a scene's sample unchanged:

stable_seed(2026, "scene_a") == stable_seed(2026, "scene_a")
true

Tuning a threshold

A tuning is specified in a [tuning.<name>] table of a TOML config. The table names a scheme with type and states only what differs from the defaults in the package's configs/tuning.toml. Every spec names its data through a registered scene split; here, the PhotoTourism split of configs/splits.toml.

config = TOML.parse("""
    [tuning.threshold]
    type = "coarse_fine"
    coarse = { type = "spaced", window = { type = "fixed", lo = 0.1, hi = 10.0 }, spacing = { type = "log_per_decade", per_decade = 4 } }
    cross_validation = { scene_split = { type = "registered", name = "phototourism" } }
    """)
spec = tuning_spec(config, "threshold")
spec.scheme
CoarseFine(SpacedAxis(FixedWindow(0.1, 10.0), LogPerDecade(4)), GridExtension(4, 2), 1, LogCount(200), FiniteMedian(), CrossValidation{RegisteredSceneSplit, ExhaustiveSampling, ExhaustiveSampling}(RegisteredSceneSplit("phototourism", ["brandenburg_gate", "british_museum", "buckingham_palace", "colosseum_exterior", "florence_cathedral_side", "grand_place_brussels", "lincoln_memorial_statue", "milan_cathedral", "mount_rushmore", "trevi_fountain"], ["notre_dame_front_facade", "palace_of_westminster", "pantheon_exterior", "piazza_san_marco", "reichstag", "sacre_coeur", "sagrada_familia", "st_pauls_cathedral", "st_peters_square", "taj_mahal", "temple_nara_japan"]), ExhaustiveSampling(), ExhaustiveSampling()))

The coarse grid spans two decades at four points per decade:

collect(grid(grid_spec(spec.scheme.coarse)))
9-element Vector{Float64}:
  0.1
  0.1778279410038923
  0.31622776601683794
  0.5623413251903491
  1.0
  1.7782794100389228
  3.1622776601683795
  5.623413251903491
 10.0

coarse_fine_cv runs the coarse-to-fine search on an objective. The objective returns the validation errors at a threshold x; the spec's criterion (the median error, by default) reduces them to one number. Here a stand-in objective is smallest at x = 2:

validation_errors(x) = [abs(log(x / 2)) + 0.1 * k for k in 1:5]
result = coarse_fine_cv(validation_errors, grid_spec(spec.scheme.coarse), spec.scheme;
                        method_name = :threshold)
result[:x_star_fine]
2.0022003718155847

The search first evaluates the coarse grid. When the best point lies near an edge of the window, the grid extends on that side. A fine grid over the neighbours of the coarse optimum then gives the tuned value. A result whose optimum is still on the boundary after every extension is flagged, and the readers of tuned values refuse it:

result[:boundary_hit]
false

Where to go next