API: Stats

Summary statistics of pose errors and the seed derivation of every random sample.

Public names

VisualGeometryToolkitEval.Stats — Module
Stats

Summary statistics of the evaluation: finite_median and lower_median, the validation reference_scale and its robust extremes (scale_bounds), the mean average accuracy (mAA) under the eligibility rule (eligible_maa, pair_outcomes), bias-corrected and accelerated (BCa) bootstrap intervals (bootstrap_ci), means of per-scene medians (scene_mean_median, scene_mean_median_rows!) and the seed derivation of every random sample (stable_seed, VisualGeometryToolkitCore's, re-exported; every stream is StableRNGs' StableRNG), the quantile of a possibly empty sample (quantile_or) and the replay of persisted improvement traces (ImprovementTrace, trace_error_at, anytime_curve).

source
VisualGeometryToolkitEval.Stats.ImprovementTrace — Type
ImprovementTrace(offsets, t_ns, hyp, err)

The persisted improvement events of n pairs in flat columns: pair i owns the events offsets[i]:offsets[i+1]-1 (so length(offsets) == n + 1), each with the elapsed time t_ns (nanoseconds), the hypothesis count hyp (number of hypotheses drawn so far) and the ground-truth error err of the best model so far. Events of a pair are in time order.

source
VisualGeometryToolkitEval.Stats.anytime_curve — Method
anytime_curve(tr, tgrid; stat = eligible_maa) -> Vector

The accuracy-versus-time curve of a trace: for each budget t (seconds) of tgrid, stat of the per-pair errors trace_error_at(tr, i; t_ns = 1e9t). Pairs with no event by t enter as Inf, so a failure counts in stat (eligible_maa scores it 0; finite_median drops it; e -> count(isfinite, e) is the number of pairs reached).

source
VisualGeometryToolkitEval.Stats.bootstrap_ci — Method
bootstrap_ci(stat, rng, x; n_resamples, alpha) -> (lo, hi)

BCa (bias-corrected and accelerated) bootstrap CI of stat(x) via Bootstrap.jl — Shekhovtsov's scipy.stats.bootstrap(..., method='BCa') protocol. For a REAL sample the non-finite entries are dropped first and a constant sample returns its value as a degenerate interval. Any other element type is resampled as given (e.g. a scene-cluster bootstrap over per-scene error matrices, with stat reducing the resampled scenes). Fewer than 3 elements → (NaN, NaN) (BCa's jackknife needs them).

rng seeds the global Random stream before the Bootstrap.jl call (the library resamples from it and takes no rng), so a fresh seed per call keeps a run reproducible on one Julia version: the resampling itself is Julia's task-local default stream, whose algorithm Julia does not fix across versions (record julia_version beside a reported CI). n_resamples and alpha default to configs/default.toml [metrics] (bootstrap_resamples, ci_alpha).

source
VisualGeometryToolkitEval.Stats.eligible_maa — Function
eligible_maa(errors, eligible = nothing; max_error = maa_threshold()) -> Float64

The mAA@max_error of a set of pairs under the eligibility rule: the continuous AUC (VgtkCore mean_average_accuracy) over the ELIGIBLE pairs, where a non-finite error (a run that yielded no model) scores 0 and stays in the denominator, and an ineligible pair (eligible[i] == false: no ground truth, or fewer valid correspondences than a minimal sample) is excluded. eligible = nothing treats every pair as eligible. NaN when no pair is eligible. max_error defaults to maa_threshold.

source
VisualGeometryToolkitEval.Stats.finite_median — Method
finite_median(v; empty = NaN) -> Float64

Median over the finite entries of v. empty is returned when no finite entries exist — use the default NaN for REPORTING (propagates as missing) and Inf for ARGMIN OBJECTIVES (an all-NaN column must lose the argmin, never win it).

source
VisualGeometryToolkitEval.Stats.lower_median — Method
lower_median(v) -> eltype(v)

The ⌈n/2⌉-th order statistic of v under isless: for an even n the LOWER of the two middle values, never their mean, so the result is always an element of v. NaN sorts last under isless — filter first when it should not count. Empty v → NaN. The statistic of record of the PnP supplement tables and the pool-size audits (an observed value, not an interpolation).

source
VisualGeometryToolkitEval.Stats.maa_threshold — Method
maa_threshold() -> Float64

The mAA threshold of the reported statistics, in degrees: configs/default.toml [metrics] maa_threshold_deg (the 10 of mAA@10°). Every reader of that threshold calls this; none spells the number.

source
VisualGeometryToolkitEval.Stats.pair_outcomes — Function
pair_outcomes(errors, eligible = nothing) -> (; n_eligible, n_failed, n_skipped)

The counts behind eligible_maa: eligible pairs, eligible pairs whose run yielded no model (non-finite error), and skipped (ineligible) pairs — reported beside every mAA so a reader sees why runs fail.

source
VisualGeometryToolkitEval.Stats.quantile_or — Method
quantile_or(v, p; empty = NaN) -> Real

quantile(v, p), or empty when v has no element. The report cells' quantile: a sample can be empty (every pair failed), and the caller chooses what that reads as.

source
VisualGeometryToolkitEval.Stats.reference_scale — Method
reference_scale(s2) -> Float64

The validation reference scale s0 = √median σ̂² over the finite entries of the per-pair inlier variances s2 — the scale every tuning window is positioned at. Throws on an input with no finite entry: a window cannot be placed without a scale, and a NaN scale would silently select nothing.

source
VisualGeometryToolkitEval.Stats.scale_bounds — Method
scale_bounds(s2, q) -> (ŝ_min, ŝ_max)

Robust scale extremes of a validation pool: √ of the q = (q_lo, q_hi) quantiles of the per-pair noise variances s2 (σ̂², e.g. gt_variance; non-finite entries are dropped). Restricting drawable validation pairs to σ̂² ∈ [ŝ_min², ŝ_max²] makes the plug-in ŝ(n) = √median σ̂² over ANY drawn subset lie in [ŝ_min, ŝ_max], for every n. The coverage window rule (Tuning.CoverageWindow, which carries q) and the validation-eligible mask are built from these bounds, so a windowed tuner's window provably stays on its grid. q has no default: it is a parameter of the tuning spec.

source
VisualGeometryToolkitEval.Stats.scene_mean — Method
scene_mean(stat, scene_errors) -> Float64

The equal-weight MEAN over scenes of each scene's stat (scene_errors iterates the scenes' value vectors); a scene whose statistic is not finite is skipped, and the result is NaN when no scene has one.

The scene-level aggregation: a per-scene statistic weights scenes, not pairs, which is what a cross-validation split BY SCENE is asking for. isfinite is the right skip predicate for either statistic this takes — finite_median filters non-finite values first and so returns a finite value or NaN, while lower_median does not and can return ±Inf.

source
VisualGeometryToolkitEval.Stats.scene_mean_median_rows! — Method
scene_mean_median_rows!(rows, method, xs, cols, scene_ranges; seed, ci_min = [metrics] ci_min_scenes)

Shekhovtsov's "Mean median": per x, split the per-pair error view cols[j] by scene_ranges: point = scene_mean_median, CI = scene-clustered bootstrap_ci(mean, …) over the per-scene medians, resampled under StableRNG(stable_seed(seed, "scene_mean_median", j)) (the mAA's with key "scene_mean_maa"). Companion statistic: the per-scene eligible_maa (every stored pair is eligible — the pool only holds processable pairs), averaged over scenes. Schema (method, tuning_criterion, n, mean_med, lo95_mean_med, hi95_mean_med, mean_mAA, lo95_mean_mAA, hi95_mean_mAA); NOT pooled over all pairs.

source
VisualGeometryToolkitEval.Stats.trace_error_at — Method
trace_error_at(tr, i; t_ns = nothing, hyp = nothing) -> Float64

The ground-truth error of pair i's best model once the budget is spent: the error of its LAST event whose time is at most t_ns and whose hypothesis count is at most hyp (a budget left nothing does not bind). Inf when no event is within the budget: a pair with no model yet counts as a failure.

source

stable_seed is VisualGeometryToolkitCore's stable_seed, re-exported by Stats (the re-export goes in the next breaking release).

Internals

These names are not public; they are listed for contributors.