API: Evaluation

The coarse-to-fine search, the bootstrap tuning harness and the composition evaluator.

Public names

VisualGeometryToolkitEval.Evaluation — Module
Evaluation

The evaluators. The composition evaluator runs a scorer composed with a local optimizer (LO) on stored pool pairs (build_pair_ctx, refit_imc!, apply_lo_refits, evaluate_sample_budget, live_runs, Roster). It is written against VisualGeometryToolkitCore's scoring and local-optimization contracts only, so it runs any scorer with any optimizer. The module also holds the coarse-to-fine search of every cross-validation (coarse_fine_search, coarse_fine_cv) and the bootstrap tuning harness (WindowedGridMethod, compute_method_rows).

source
VisualGeometryToolkitEval.Evaluation.NativeAxis — Type
NativeAxis

The native width axes, Union{OutlierHalfwidth, LossWidth} (VisualGeometryToolkitCore WidthAxis): what a selector grid, a selection sweep or a refiner's own native lives on. EfficiencyConstant() is a width parameterization, never a native.

source
VisualGeometryToolkitEval.Evaluation.OPERATING_EFFICIENCY_CONSTANT — Constant
OPERATING_EFFICIENCY_CONSTANT

The refit protocol's operating efficiency constant, 2.5 by default: the scale multiple at which the outlier-half-width ↔ loss-width crossover is inverted when a loss-width-consuming refiner rides an outlier-half-width selector grid (consumed_native). A definition of the refit protocol, not a tuned value: configs/default.toml [refit] operating_efficiency_constant.

source
VisualGeometryToolkitEval.Evaluation.ComposedSpec — Type
ComposedSpec(base, sel, selection::WidthAxis, kind::WidthAxis, ctor; source = TunedWidth(),
             format6_constant = identity)

One composed score+LO pipeline. sel is the key of its selection sweep (the selections idxs_of[sel] a refit seeds from); selection is that sweep's native axis — VisualGeometryToolkitCore's width_axis of the selection scoring (OutlierHalfwidth() for the Student-t marginal, LossWidth() for a threshold score); kind is the arm's width parameterization; ctor(value) builds the LO strategy at a width value (an efficiency-constant arm bakes it in; the other kinds deliver it through their port, see width_delivery); source is where the width value comes from (WidthSource): a SelectionWidth arm requires kind == selection, and the tuning skips it.

format6_constant(m) is the arm's own reading of an efficiency constant m stored by a format-6 tuning sidecar, as the PLAIN multiple ctor takes now (applied by tuned_width, and only there). Format 6 stored the value the arm's ctor took then, in whatever currency that ctor read it, and only the roster that wrote the ctor knows it, so the roster declares it per arm. The default is the identity: a value the ctor already reads as the plain multiple, or one its own map converts. An arm whose ctor took a Geman–McClure half-weight depth directly (a bare Irls{GemanMcclureShape} at the stored value) declares format6_constant = m -> decision_width(GemanMcclureShape(), m).

source
VisualGeometryToolkitEval.Evaluation.FlatProtocol — Type
FlatProtocol(val_pair_idx, test_pair_idx)

Fixed deterministic pair-level split, bootstrap pairs only. Trial-level variance reflects pair-sampling variance only (no scene component). Used for ScanNet1500 and other "uniform sample of pairs" datasets.

source
VisualGeometryToolkitEval.Evaluation.LoRefitSpec — Type
LoRefitSpec(label, lo, axis, grid, idxs; bundle=false)

One post-LO refit of a selection sweep: refine the selected model idxs[p, t] with strategy lo at native parameter consumed_native(lo, axis, grid[t]). axis is the SELECTOR grid's native axis (a NativeAxis: OutlierHalfwidth() | LossWidth()) — cross-axis conversion happens per column inside the refit layer, so callers never pre-convert grids. lo = nothing records the selection imc (no refit).

Column t refits the model selected at column seedcols[t] of idxs at the EXPLICIT native value natives[t] when those fields are set (no consumed_native conversion — the value is already in the refiner's units). Both default to the standard coupling (seedcols[t] = t, natives === nothing ⇒ derive from grid[t]). This is what the tuning grids (one fixed width parameter per spec) and the fixed-width deployment evaluations (seed at the ζ-scaled column, refit at the exactly-scaled native) are built from.

widths[t] (an absolute loss width): refit at the loss width widths[t] whatever the refiner — the BITWISE-identical call solo_refit_imc! makes, so the tuning and the ladder's test side deliver the width identically. A native refiner sweeps its seed at the column's nat′ (from grid[t] — scoring only, never a width) under the context's solo template and deploys with_width(lo, fixed_width(lo, widths[t]²)); an external refiner refines at widths[t]. Mutually exclusive with natives.

columns restricts which grid columns are EVALUATED (default: all). The others keep the imc matrix's NaN fill. The matrix stays full-width and every column index keeps its meaning, so boundary detection, the evaluation's seed columns and the checkpoint are untouched — this only stops computing cells that a windowed tune never reads, because it argmins inside the standardized window and the grid is far wider than that window. Measured on the PT-E 5% run of 2026-10-02: 45,567 of 386,000 cells per pair were inside the searched windows; the other 88% were computed and discarded. (A cell outside the searched window is never read by a windowed argmin.)

fixed_s2s[t] (a fixed external scale): refit at the column's native exactly as the default path (the refit template), with the strategy deployed at that scale, with_width(lo, fixed_width(lo, fixed_s2s[t])) — the refit_imc!(…; s2) conduit — so its loss width is efficiency_constant·√s2 instead of the consensus set's efficiency_constant·ŝ. The consensus-anchoring ablation evaluation (miscal_evaluation_refit with plugin_scale) is built from it. Combines with natives; mutually exclusive with widths.

source
VisualGeometryToolkitEval.Evaluation.LossWidthSpec — Type
LossWidthSpec(name, scoring, loss_width, lo = nothing; refinement_loss_width = nothing)

One loss-width column of evaluate_sample_budget: selection by scoring, a loss-width family's template (its width_axis is LossWidth(): VisualGeometryToolkitCore's MsacScoring(dg) or RansacScoring(dg), or a paper's family with the same with_width and score! methods), built at loss_width (pool frame) with with_width(scoring, LossWidth(loss_width)). Its running best is refit by its own refiner lo (a paper's truncated refiner — an ExternalRefit strategy such as PoseLib's TLS-LM) at refinement_loss_width, the arm's tuned refinement loss width (nothing: the selection loss width). With lo = nothing the column is selection only: its LO checkpoints record the selected model's model_imc, the same statistic as its batch checkpoints, and cost no refit.

source
VisualGeometryToolkitEval.Evaluation.OracleSelector — Type
OracleSelector(name, outlier_halfwidth)

Oracle selector: the ground-truth-best SINGLE model by model_imc. Recomputes the imc per model (pruned by VisualGeometryToolkitCore's pool_imc, the stored lower bound on that imc) and refines that one model under LO (at a representative outlier half-width, the first method's, unless the config names the arm's refinement native).

source
VisualGeometryToolkitEval.Evaluation.PairCtx — Type
PairCtx

Everything the refit layer needs about one pair. Built once per pair by build_pair_ctx and reused across every (LO refit spec × grid column), it holds the pair itself (a light PoolPair), its pool-frame problem (ep), the optional RelposeBundleProblem (bp, nothing unless bundle=true), the geometry, the problem class (problem_type) and the refit scoring TEMPLATES: scoring, the scoring every native refit is built from at its own outlier half-width (with_width(scoring, OutlierHalfwidth(nat′))), and solo_scoring, the one an absolute-width (solo) refit sweeps its seed with (by default scoring).

source
VisualGeometryToolkitEval.Evaluation.PairGeom — Type
PairGeom

Per-pair calibration and ground truth, the context that scores a pool model's pose error. It scores any stored pool-frame fundamental's coherent imc pose error and holds the pool-frame correspondences cs (x1 => x2) as a concretely typed Correspondences.

source
VisualGeometryToolkitEval.Evaluation.RefitHalfwidth — Type
RefitHalfwidth(lo, outlier_halfwidth, scoring_of)
RefitHalfwidth(lo, outlier_halfwidth, template::AbstractScoring)

lo refitting under the refit scoring at ITS OWN outlier half-width instead of the selection's: the in-loop form of the refit layer's native port (refit_imc!'s nat′, refit_model). The self-scaled arms (Stu-tU IRLS, BCD) read their half-width from the scoring the loop hands them, so inside a RANSAC loop they would otherwise refit at the SELECTION half-width — not the refit half-width their sequential CV tuned at the fixed a*. scoring_of(problem) builds the refit scoring; the template form is with_width(template, OutlierHalfwidth(outlier_halfwidth)).

In the loop, each call re-scores the model under the refit scoring in the loop's own buffers and runs the loop's own Phase-2 step on that consensus set (VisualGeometryToolkitEstimation's locally_optimize_rescored!): at outlier_halfwidth = the selection's it is the score the loop just ran, so the wrapped arm reproduces the bare arm exactly. The refined model goes back to the loop, which re-scores it under the SELECTION scoring and keeps it only if it beats the incumbent.

source
VisualGeometryToolkitEval.Evaluation.Roster — Type
Roster(composed, solo, basin)

A paper's arm roster, the argument of every reader that rebuilds arms from a tuned sidecar (deployed_budget_settings, deployed_basin_settings, audit_operating_points). Each field is the paper's builder:

  • composed(problem::Symbol) -> Vector{ComposedSpec} — the composed score+LO roster (efficiency-constant / outlier-half-width / natively delivered loss-width arms) of the composed miscal and budget evaluations;
  • solo(problem::Symbol; role::Symbol) -> Vector{SoloSpec} — the solo roster;
  • basin(problem::Symbol) -> Vector{ComposedSpec} — the basin roster (loss-width and outlier-half-width arms only; an efficiency constant has no meaning on the basin axis).
source
VisualGeometryToolkitEval.Evaluation.SampleBudgetConfig — Type
SampleBudgetConfig

Configuration for the sample-budget selection protocol.

Fields (the schedule and the LO are REQUIRED: a sample-budget experiment

states its protocol of record, e.g. a paper's samplebudgetprotocol.toml)

  • sample_cap::Int — total minimal samples drawn per pair
  • batch_size::Int — selection (non-LO) checkpoint stride
  • lo_every::Int — LO checkpoint stride (after the first batch)
  • lo_strategy::Union{Nothing, AbstractOptimizationStrategy} — LO method (nothing: none)
  • budget_unit::Symbol — budget axis: :models counts essential matrices inspected (Shekhovtsov's unit); :samples counts minimal subsets.

Checkpoint schedule (both curves share one cumulative-sample x-axis)

  • Selection (running-best, no LO): every batch_size — 100, 200, …, cap.
  • LO (refit the running best): after the first batch, then the lo_every grid — {batch_size} ∪ {lo_every, 2·lo_every, …, cap} = 100, 500, 1000, ….
source
VisualGeometryToolkitEval.Evaluation.SampleBudgetMethodSpec — Type
SampleBudgetMethodSpec(name::Symbol, scoring)

A scoring-method column: a label and the scoring it selects with, at its validated width (a Student-t + uniform marginal at its prior and outlier half-width). Its running best is refit by the config's LO at the config's native, else at the scoring's outlier half-width. name is opaque to the evaluator.

source
VisualGeometryToolkitEval.Evaluation.SampleBudgetResults — Type
SampleBudgetResults

Per-pair error curves for each method column, plus LO-refined error at each LO checkpoint and the LO wall-clock accounting (see summarize_lo_timing).

Fields

  • methods::Vector{Symbol} — method labels (length M)
  • sample_counts::Vector{Int} — cumulative-sample axis for batch curves (B)
  • lo_counts::Vector{Int} — cumulative-sample axis for LO checkpoints (L)
  • pair_ids::Vector{String} — per-pair identifiers (P)
  • err_batch::Array{Float32,3} — (P, M, B) running-best imc [deg]
  • best_idx_batch::Array{Int32,3} — (P, M, B) pool index of running best
  • err_lo::Array{Float32,3} — (P, M, L) LO-refined imc [deg]
  • lo_ns_total/lo_calls_total/lo_startup_ns::Vector{Int64} — per-method warm LO wall-clock [ns] / call count / cold first-call [ns] (JIT + one call)
source
VisualGeometryToolkitEval.Evaluation.ScoreGuarded — Type
ScoreGuarded(inner::AbstractOptimizationStrategy)

LO-RANSAC best-so-far monotonicity wrapper for IRLS-family arms. The inner LO runs unchanged, but the refined model is kept only if its SELECTION score (the same marginal that seeded it) does not drop below the seed's (VisualGeometryToolkitEstimation's score_guard). It preserves score monotonicity while the optimizer descends a different objective.

Its RefitFamily and the external-refit bracket (external_workspace, prepare_external!, refine_external!) are inner's, as on VisualGeometryToolkitEstimation's CountedLO and TerminalLO: a guarded external arm is routed, prepared and refined as the bare arm. The guard is score_guard on the native path, so an external refit is forwarded unguarded.

source
VisualGeometryToolkitEval.Evaluation.ScoreSelector — Type
ScoreSelector(name, scoring, width, ws, scores; lo = nothing, refinement_width = nothing)

Scoring-based selector: the running argmax of scoring's score (VisualGeometryToolkitCore score! on the scorer's workspace ws, per-point scratch scores) over the pool models; width is the native width scoring selects at (its outlier half-width, or its threshold). Its running best is refit by the config's LO (lo = nothing) at the config's native (nat_ref, else width), or by its own LO lo at refinement_width (else the config's nat_ref, else width) — the MSAC + truncated-refiner columns.

source
VisualGeometryToolkitEval.Evaluation.SoloSpec — Type
SoloSpec(base, role, axis::WidthAxis, ctor)

One role (:selection | :refinement) validated in isolation at one width value on axis (see the block comment above); ctor(value) returns a pool scorer (selection) or an LO strategy (refinement).

source
VisualGeometryToolkitEval.Evaluation.WindowedGridMethod — Type
WindowedGridMethod(label, scene_errors, scene_sigma2, native_grid, window, to_native,
                   criterion)

Tunes on the standardized efficiency-constant axis (accv §common-cutoff, eq. 28) instead of the raw native grid, with the reference scale estimated from the validation draw so it tracks the validation cardinality n — the protocol of the windowed_bootstrap tuning scheme.

  • scene_errors[i] — precomputed (pairs × |native_grid|) post-LO error matrix on a fixed grid of native hyperparameter values (native_grid).
  • scene_sigma2[i][j] — pair (i,j)'s noise variance σ̂² (e.g. gt_variance), used to form the cardinality-tied scale σ_med(n).
  • to_native(efficiency_constant, scale) -> native — the eq-(28) map for this scoring, a closure over the caller's native_map(axis; d_g) so this package stays decoupled from the scoring-spec types.
  • window — the spec's EfficiencyConstantWindow; criterion — its objective (no default: both are parameters of the tuning spec).

Per trial, tune draws n val pairs, forms σ_med(n)=√median σ̂² over them, and takes the FREE ARGMIN of the val objective over every native_grid column inside the standardized window [to_native(lo, σ_med), to_native(hi, σ_med)] (the window's covered range at the drawn supplied scale — corrupting the scale slides the window, which is the miscalibration mechanism). evaluate reads the tuned column on the test pairs.

source
VisualGeometryToolkitEval.Evaluation.apply_lo_refits — Method
apply_lo_refits(problem_type, specs::Vector{LoRefitSpec}, pairs;
                scoring, solo_scoring = scoring,
                operating_efficiency_constant = OPERATING_EFFICIENCY_CONSTANT, on_fail = nothing)
    -> Dict{String, Matrix{Float64}}

Run every refit spec over pairs and return each spec's post-LO error matrix. scoring is the refit scoring TEMPLATE (rebuilt at each refit's own outlier half-width), solo_scoring the absolute-width (widths) columns' template. The run is threaded :greedy over the flat (pair × spec) product so neither a subsampled validation side nor the arms' very uneven per-spec cost leaves threads idle (see the comment in the body), with one PairCtx built per task at exactly that spec's bundle. pairs are light pool records (PoolPair): a refit never reads a residual matrix. on_fail = nothing ⇒ seed-error fallback (LO never worsens the running best); supply NaN for the ladder convention (failed refit ≠ success). Returns label => (P × T) imc matrix.

source
VisualGeometryToolkitEval.Evaluation.bootstrap_trials — Method
bootstrap_trials(method, proto::FlatProtocol, budget::BudgetSampling, n;
                 max_error = maa_threshold())
    -> (trial_meds, trial_maas, trial_idxs)

The per-draw bootstrap: each of the budget.n_draws draws tunes the hyperparameter on n val pairs of proto (tune) and evaluates the tuned column on its test pairs (evaluate): the draw's median test error (a pair without a model counts as Inf) and its mAA@max_error (the eligibility rule, eligible_maa). Draw r uses budget_rng(budget, n, r).

source
VisualGeometryToolkitEval.Evaluation.build_pair_ctx — Method
build_pair_ctx(problem_type, pair::PoolPair; scoring = nothing, solo_scoring = scoring, bundle = false) -> PairCtx

Build the PairCtx of one pool pair, once. It holds the family's pool-frame problem (VisualGeometryToolkitEstimation's EssentialProblem(pair), FundMatProblem(pair) or HomographyProblem(pair)), Kc, GT pose and pool-frame points; plus the Schur bundle problem when bundle = true. problem_type is the pool's track (problem_type(pool.problem)). scoring is the refit scoring template (any scoring with an outlier half-width: the caller's — a paper's Student-t + uniform marginal at its prior); nothing is a context that only reads selections (no native refit).

source
VisualGeometryToolkitEval.Evaluation.checkpoint — Function
checkpoint(s::SampleSelector, geom::PairGeom, pair) -> (imc::Float32, best_idx::Int32)

The running best's imc pose error [deg] and pool index, recorded at a checkpoint; (NaN32, 0) before any model was folded in.

source
VisualGeometryToolkitEval.Evaluation.coarse_fine_cv_batched — Method
coarse_fine_cv_batched(curvesfn, coarse::AbstractDict{K,<:SpacedGrid}, scheme::CoarseFine;
                       method_name = Symbol, label = string, param_name = :x, context_tag = "")
    -> Dict{K,Dict{Symbol,Any}}

coarse_fine_search of one scalar per cell, curvesfn(xs_of) -> Dict(k => [errors at x for x in xs_of[k]]). Per cell the record :coarse_spec, :x_star (coarse), :boundary_hit, :n_extensions, :curve_ys, :fine_spec, :fine_curve, :fine_idx, :fine_bracket, :x_star_fine (the tuned value) and the identity fields.

source
VisualGeometryToolkitEval.Evaluation.coarse_search — Method
coarse_search(errorsfn, cells::AbstractDict{K,<:NamedTuple}, scheme; label = string)
    -> Dict{K,NamedTuple}

Stage 1 of the coarse-fine search (module header): per search key k, cells[k] names each axis's coarse SpacedGrid (grid_spec of the spec's axis); the objective over their product, each axis extended on a side while the joint argmin lies within the spec's edge_margin of that face. scheme is a JointCoarseFine or a CoarseFine (one axis, named by the cells). Per key (; names, coarse, coarse_objective, n_extensions, coarse_argmin, boundary_hit, evaluated): the axis names, the final coarse grid specs, the objective over their product (an ndims = d array), the extension rounds (d × 2, lo and hi), the argmin and the per-axis boundary flags as NamedTuples, and every evaluated point => errors.

source
VisualGeometryToolkitEval.Evaluation.compute_method_rows — Method
compute_method_rows(method, budget::BudgetSampling; protocol, sizes = budget.sizes)
    -> (summary::Vector{NamedTuple}, trials::Vector{NamedTuple})

Per-method, per-n evaluation loop over the validation budgets sizes (the schedule's own by default; a miscal evaluation supplies its one fixed budget): budget.n_draws draws per budget (bootstrap_trials), the BCa CIs resampled with resampling_rng(budget, n). Returns two row vectors in canonical DataFrame-friendly shape:

  • summary: one row per n carrying the BCa CIs and per-trial argmin diagnostics that were the columns of the per-cell summary DataFrame.
  • trials : one row per (n, trial) carrying the raw (trial_med, trial_mAA, trial_idx) so CIs can be replayed at any α downstream.

protocol is the val/test split (FlatProtocol). (The produce_or_load cache and the decoupled tune_protocol path were pruned 2026-09-21 with their only caller, evaluate_val.)

source
VisualGeometryToolkitEval.Evaluation.consumed_axis — Method
consumed_axis(lo) -> NativeAxis

The native axis refiner lo consumes, by its refit family: a native refiner its scoring outlier half-width (OutlierHalfwidth()), an external one its truncated loss's width (LossWidth()).

source
VisualGeometryToolkitEval.Evaluation.consumed_native — Method
consumed_native(lo, axis::NativeAxis, native::Real;
                operating_efficiency_constant = OPERATING_EFFICIENCY_CONSTANT, d_g) -> Float64

The native width THIS refiner consumes (consumed_axis), given a selector grid value native on native axis axis. Same-axis is identity; cross-axis converts at the operating_efficiency_constant crossover (VisualGeometryToolkitCore outlier_halfwidth_to_loss_width / loss_width_to_outlier_halfwidth) for residual codimension d_g (codimension(problem_type): 1 for E/F, 2 for H).

source
VisualGeometryToolkitEval.Evaluation.deployed_basin_settings — Method
deployed_basin_settings(roster::Roster, path)
    -> Vector{base => (spec::SoloSpec, width::Float64)}

Each basin arm's tuned width from a basin tuning sidecar (mode = "basin"). The arms are rebuilt through roster.basin(problem) and repackaged as a refinement SoloSpec whose axis is the arm's parameterization, so tuning and test deliver the width identically via solo_refit_imc!. Refuses a non-basin sidecar. Same interior-argmin gate as the composed readers.

source
VisualGeometryToolkitEval.Evaluation.deployed_budget_settings — Method
deployed_budget_settings(roster::Roster, path; problem = nothing)
    -> Vector{label => (lo, sel, ref, kind)}

Per-arm tuned operating points of a composed tuning sidecar, rebuilt through the roster. The sidecar is read through VisualGeometryToolkitEval load_tuning_artifact, which refuses another format; each arm is rebuilt through roster.composed(problem): the LO instance (the tuned scale multiple baked in for efficiency-constant arms), the SELECTION native and the REFINEMENT native (efficiency-constant arms refit at the selection native — consensus-anchored; native parameterizations at their tuned value) and the arm's width parameterization kind — the roster's, the one source of an arm's kind (a sidecar's own kinds list is not read). problem defaults to the sidecar's own "problem".

INTERIOR-ARGMIN INVARIANT: a HARD ERROR on any arm whose persisted boundary_hits is non-empty — expand the window on the flagged side and re-run the tuning (a SelectionWidth arm has no width of its own to clip: its list carries the selection's axis alone, and its recorded width must be its selection's, operating_width). A basin sidecar is refused (deployed_basin_settings reads it).

A format-6 sidecar (read through its version step, load_tuning_artifact) stored an efficiency constant as the value its arm's ctor took then: it is read through the arm's format6_constant hook (tuned_width), so the deployed LO is the one the sidecar was tuned with.

source
VisualGeometryToolkitEval.Evaluation.deployed_setting — Method
deployed_setting(path; context = path, allow_edge = false) -> NamedTuple

The gated reader of a coarse-fine tuning artifact (save_coarse_fine_artifact): the selected value of every axis, as a NamedTuple. HARD ERROR when the spec is not a coarse-fine one or the search hit its window's edge on any axis (enforce_probe_interior; allow_edge = true downgrades to a warning for FORENSIC reads only). A tuned setting is deployed through this, never from a literal.

Also reads the one-scalar layout written before the joint search (a CV "record" and "x_star_fine"; BlobBoards' half-width run of record): (; <param_name> = x_star_fine).

source
VisualGeometryToolkitEval.Evaluation.enforce_probe_interior — Method
enforce_probe_interior(rec_or_flag; context, allow_edge = false)

Read-side gate of the coarse-fine search: HARD ERROR when a record (or an explicit boundary_hit::Bool) is flagged — the coarse window clipped the optimum even after the extension budget, so the tuned value is suspect and must never be silently consumed. Re-run the search (it expands itself; if it still hits the budget, the objective has no interior optimum on any reachable window and the method/cell needs attention). allow_edge = true downgrades to a loud warn — FORENSIC reads only, never production. Mirrors the LO engine's _enforce_interior_argmin gate on the product-grid readers.

source
VisualGeometryToolkitEval.Evaluation.essential_bearings — Method
essential_bearings(ctx::PairCtx) -> (f₁, f₂)

The unit bearings of an E context's problem (what a cheirality vote reads). Read from ctx.ep, which already back-projected every correspondence through the pool camera, rather than re-derived. Only an E context has bearings: on an F or H context ctx.ep holds pool pixels, so this refuses rather than hand 2-vectors to a ray test.

source
VisualGeometryToolkitEval.Evaluation.evaluate_sample_budget — Method
evaluate_sample_budget(problem_type, methods::Vector{<:SampleBudgetMethodSpec}, pairs, residuals;
                       pair_ids, loss_width_specs::Vector{<:LossWidthSpec}, cfg::SampleBudgetConfig,
                       sample_starts = nothing, refit_scoring = methods[1].scoring)
    -> SampleBudgetResults

Run the sample-budget protocol over the pool pairs pairs. The pairs are a flat vector of light pool records (PoolPair) with their residual matrices residuals (residuals[i] is pair i's; VisualGeometryToolkitEval load_pool_residuals, residual_matrix). A :samples budget (cfg.budget_unit) counts minimal samples through sample_starts[i], pair i's minimal-sample starts — VisualGeometryToolkitEval sample_starts(pool), the one reconstruction of sample boundaries; it is required then and ignored for a :models budget. Columns: the scoring methods, then one loss-width column per LossWidthSpec (MSAC, or a paper's loss-width family; selection only under no local optimization, nothing), then the oracle (last). Every checkpoint records the running best's model_imc. Every native refit is seeded under refit_scoring (a scoring TEMPLATE, rebuilt at each refit's own outlier half-width) — the methods' own: they must share it. Parallel over pairs; per-method scoring workspaces preallocated per thread.

source
VisualGeometryToolkitEval.Evaluation.evaluate_sample_budget_pair! — Method
evaluate_sample_budget_pair!(out_err_batch, out_best_idx_batch, out_err_lo,
                             out_lo_ns, out_lo_calls, selectors::Tuple,
                             problem_type, pair, R, cfg; refit_scoring = nothing,
                             sample_starts = nothing)

The per-pair worker of evaluate_sample_budget: walk one pair's pool and fill its slice of the results.

pair is the light pool record, R its residual matrix (numpoints × nummodels, Float32, the pool frame). sample_starts is the pair's minimal-sample starts (VisualGeometryToolkitEval sample_starts), required when the budget counts :samples. The selectors — a Tuple, so every per-model call is statically dispatched — are mutated too (reset per pair, then folded), so the problem type follows them. refit_scoring is the per-pair context's refit template — the one every refit's scoring is built from at its own native — so it must be the selectors' own.

source
VisualGeometryToolkitEval.Evaluation.fine_search — Method
fine_search(errorsfn, coarse::AbstractDict{K,<:NamedTuple}, scheme) -> Dict{K,NamedTuple}

Stage 2 of the coarse-fine search: per key, each axis's fine spacing over the two coarse neighbours of the coarse argmin on that axis, their product evaluated once (no extension), its free argmin the selection. Each record of coarse_search gains fine (the fine grid specs), fine_objective and selected (a NamedTuple of axis values); evaluated gains the fine cells. A spec's incumbent is evaluated (one more call) and the record gains incumbent, incumbent_errors, gain (objective, p_gain) and kept_incumbent; selected is the incumbent when kept.

source
VisualGeometryToolkitEval.Evaluation.guarded_refit — Method
guarded_refit(opt, score, on_fail; warn_label = nothing)
    -> (err::Float64, lo_ns::Int64)

The seed-fallback + timing bracket.

  • opt() -> Union{Nothing, M} — the bracketed optimizer call (data-in → result-out). Throws are caught; nothing signals bail / no consensus.
  • score(m::M) -> Float64 — GT scoring of the refined model, NOT timed. A non-finite score falls back to on_fail (a GT-free decision: the refit produced an unusable model).
  • on_fail policy is the caller's (refit_imc!'s fallback): budget/producer supply the seed's error (LO-RANSAC invariant — refinement never worsens the running best); the basin ladder supplies NaN (a failed refit did not travel).
  • lo_ns spans exactly opt().
source
VisualGeometryToolkitEval.Evaluation.live_row — Method
live_row(runs; max_error) -> NamedTuple

The summary of one live_runs result. Its fields are med the finite median error [deg], maa the mAA@max_error under the eligibility rule (VisualGeometryToolkitEval eligible_maa: every pool pair is eligible, a run without a model scores 0; max_error defaults to its maa_threshold()), med_trials the median trials, med_wall_ms the median wall [ms], and the outcome counts (n_eligible, n_failed, n_skipped).

source
VisualGeometryToolkitEval.Evaluation.live_runs — Method
live_runs(scoring, lo, ctxs; problem_of, seed_of, config = RansacConfig(), warmup = true,
          lo_of = nothing, on_improvement_of = nothing, score_guard = false,
          cancelled = () -> false, finalize = nothing)
    -> (; err, trials, wall_s)            # plus `extras` when `finalize !== nothing`

Run one live production ransac per pair context ctxs[j], threaded over pairs. Each run uses the caller's problem of the pair, problem_of(ctxs[j].pair), the selection scoring, the LO strategy lo, adaptive termination under config. Each pair's stream is StableRNG(seed_of(j)), handed to ransac explicitly, so a run is independent of the thread schedule, identical across calls that share seed_of (seed_of should be a stable_seed). Per pair: err[j] the imc pose error [deg] of the final model, Inf when the run yielded no model (found_model false) or the pair was cancelled; trials[j] the trials drawn; wall_s[j] the wall clock of the ransac call alone. warmup first runs one short solve on pair 1 (result discarded) so no bracket pays JIT.

The keywords below default so that the result is exactly the plain run's.

  • lo_of(j) returns the LO strategy of pair j, overriding lo. The warmup uses lo_of(1) (an LO holds no recorded state).
  • on_improvement_of(j) returns the ransac improvement callback (on_improvement(trial, score, ws), fired when the incumbent improves). It is called with j = 0 once for the warmup (return a throwaway callback of the same type as for real pairs, writing to scratch storage, so the loop compiles once and the warmup records nothing) and with j = 1:n for the pairs.
  • score_guard is forwarded to ransac: per-candidate ScoreGuarded acceptance.
  • cancelled() is polled before each pair; a cancelled pair is not run and keeps err = Inf, trials = 0, wall_s = NaN. Pairs are threaded, so a pair already started finishes. VisualGeometryToolkitEval.Experiments.cancelled is the kaimon-aware poll.
  • finalize(j, r) runs after pair j's ransac result r, on the pair's thread and outside its wall bracket; its values come back as extras::Vector{Any} (nothing for a pair not run). Without finalize the result has no extras field.
source
VisualGeometryToolkitEval.Evaluation.lo_iteration_budget — Method
lo_iteration_budget() -> Int

The equal LO model-update budget of the evaluation: configs/default.toml [local_optimization_strategy] iteration_budget, so arms compare at equal cost. A paper with its own protocol supplies its budget explicitly instead.

source
VisualGeometryToolkitEval.Evaluation.miscal_evaluation_refit — Method
miscal_evaluation_refit(arm, idxs_of, tuned, zetas, grids; d_g, plugin_scale = nothing) -> LoRefitSpec

The miscalibration evaluation refit of one arm, one column per ζ in zetas. Column j seeds at the selection column nearest ζj·selection* and refits at the exactly-scaled refinement native (ζj·width* for the native parameterizations — both formed from the corrupted plug-in — while an efficiency constant is consensus-anchored and NOT corrupted). tuned is the tuned point as NATIVE VALUES, (; selection, width), its width read through operating_width (a SelectionWidth arm: width is nothing or the selection, and column j refits at ζ_j·selection*, the selection's own corrupted native); grids supplies the arm's selection grid, which the seed columns are found in.

plugin_scale (the consensus-anchoring ABLATION, efficiency-constant arms only): the same operating point and seeds, the efficiency constant's loss width delivered at the FIXED external plug-in scale ζj·`pluginscaleinstead of the consensus set's ŝ —withwidth(lo, fixedwidth(lo, (ζj·pluginscale)²))through the LoRefitSpecfixed_s2s` column. A native parameterization's width already is the corrupted plug-in; it is refused.

source
VisualGeometryToolkitEval.Evaluation.model_imc — Method
model_imc(model, geom::PairGeom) -> Float64
model_imc(model, ctx::PairCtx)   -> Float64

The pose error [deg] of model against the pair's ground truth. By the model's type: a metric EssentialMat → max(Δr, Δt) of its best decomposition; a pool-frame FundMat → the same after the pool-frame calibration upgrade; a pool-frame HomographyMat → the optimistic decomposition ROTATION error.

source
VisualGeometryToolkitEval.Evaluation.problem_type — Method
problem_type(p::Symbol) -> Type

The RANSAC problem type of a two-view pool track: :E → EssentialProblem, :F → FundMatProblem, :H → HomographyProblem. The one map from a track's letter (a pool's problem) to its problem class — the value a PairCtx carries as problem_type; any other symbol is an ArgumentError. Inside a function, name the result P (a local problem_type would shadow this function).

source
VisualGeometryToolkitEval.Evaluation.refit_imc! — Method
refit_imc!(ctx, family::RefitFamily, lo, seed, nat′, scoring, fallback, bundle, s2)
    -> (imc::Float64, lo_ns::Int64)

The execution path of one RefitFamily (VisualGeometryToolkitCore's Holy trait: NativeRefit, ExternalRefit). refit_imc!(ctx, lo, seed, nat′; …) resolves the keywords and calls this with RefitFamily(lo). A new strategy plugs in through the contracts (locally_optimize!, or refine_external! with its external_workspace), never by a method here.

source
VisualGeometryToolkitEval.Evaluation.refit_imc! — Method
refit_imc!(ctx::PairCtx, lo, seed, nat′::Float64;
           scoring = ctx.scoring, on_fail = nothing, bundle::Bool = false,
           s2 = nothing) -> (imc::Float64, lo_ns::Int64)

Refine the seed model seed with strategy lo at ITS native parameter nat′ (see consumed_native) and return the refined model's imc error [deg] plus the bracketed optimizer wall-clock.

  • scoring (NativeRefit): the scoring TEMPLATE the refit is seeded under, rebuilt at nat′ (with_width(scoring, OutlierHalfwidth(nat′))) — the context's refit template by default; ctx.solo_scoring for an absolute-width (solo) refit.
  • on_fail: value on optimizer bail / non-finite refit. Default = the seed's own imc (never-worsen); the ladder supplies NaN.
  • bundle: route the native refit through ctx.bp (Schur reprojection bundle) instead of the pose-only ctx.ep.
  • s2: an externally supplied squared scale — the one conduit for a non-consensus width:
    • NativeRefit: the strategy is deployed at the absolute width with_width(lo, fixed_width(lo, s2)) (its efficiency constant at the scale √s2) — the role-isolated controls (solo CV, oracle-scale controls, the anchoring ablation). The seed sweep still runs; only the LO's width moves.
    • ExternalRefit: the fixed scale FixedS2(s2) replaces nat′ (by default the loss width √s2).
source
VisualGeometryToolkitEval.Evaluation.refit_model — Method
refit_model(problem, seed, lo, scoring; workspace = nothing, s2 = nothing)
    -> Union{Nothing, model}

The native (locally_optimize) refit of the refit layer, returning the refined MODEL — nothing when the strategy produced none. lo may be ScoreGuarded (the guard becomes score_guard). s2 deploys the same strategy at an absolute squared scale (with_width(lo, fixed_width(lo, s2)); the scoring sweep is untouched, only the LO's width moves); nothing keeps the strategy's own width mode. The refit layer's imc rows wrap exactly this in guarded_refit; a caller that needs the refined model itself (a polish warm start) calls it directly.

source
VisualGeometryToolkitEval.Evaluation.refit_seed — Method
refit_seed(ctx::PairCtx, model) -> seed

How a stored pool model becomes the seed of a refit in the pair's problem: an E pool's pool-frame fundamental is upgraded to the metric essential Kc₂ᵀ·F·Kc₁; F and H pool models seed as stored.

source
VisualGeometryToolkitEval.Evaluation.rigid_invariance_errors — Method
rigid_invariance_errors(poses; targets, units = sort!(collect(keys(poses))), min_obs = 3)
    -> (; translation::Matrix{Float64}, rotation::Matrix{Float64}, units, pairs)

The invariance errors of rigidly mounted targets. poses[u][m] is target m's estimated pose in unit u (a value with fields R, the rotation, and t, the translation: target → camera, x_cam = R·X + t, e.g. an Extrinsics3); a target absent from poses[u] was not observed there. targets lists every target that could be observed; pairs are its sorted pairs (m, n), m < n.

For each pair and each unit observing both targets, the relative pose T_m⁻¹ T_n is compared with a LEAVE-ONE-OUT reference over the other units observing that pair: the rotation MEDOID (the observation minimising summed geodesic distance to the others — robust to the two-fold flips a mean would be dragged by) and the componentwise MEDIAN translation. Leave-one-out is not optional: a reference containing the scored observation scores it zero by construction and rewards observing each pair in fewer units.

Returns length(units) × length(pairs) matrices: translation (the translation's distance from the reference, in the poses' units) and rotation (geodesic angle, degrees). An entry is Inf where the pair was not observed in that unit, or was observed in fewer than max(min_obs, 3) units (no reference) — so under eligible_maa (the mAA rule: a non-finite error scores 0 and stays in the denominator) a missed pair counts as a failure, and eligible_maa(e, isfinite.(e)) is the accuracy over what was observed.

source
VisualGeometryToolkitEval.Evaluation.save_coarse_fine_artifact — Method
save_coarse_fine_artifact(path, spec::TuningSpec, rec; unit, code, data...) -> path

Persist one search record rec (coarse_fine_search, one key's NamedTuple) under its coarse_fine / joint_coarse_fine spec as a tuning artifact (save_tuning_artifact): the grids coarse|<axis> and fine|<axis>, and

axes              the axis names, the search's order
selected          axis => the tuned value (the fine argmin)
coarse_argmin     axis => the coarse argmin
boundary_hit      axis => the coarse argmin within the margin of a face
                  after the extension budget (refused by the reader)
n_extensions      axis => [lo rounds, hi rounds]
coarse_objective, fine_objective   the objective over each product grid
cells             every evaluated cell, one row per cell (columns: axes)
unit_statistics   per cell, Tuning `unit_statistics` of its errors — when
                  every cell's errors are a `units × items` matrix
incumbent, kept_incumbent, gain, incumbent_unit_statistics
                  with a declared incumbent (`selected` is it when kept)

data are the caller's further keys (the bootstrap of the selection, say).

source
VisualGeometryToolkitEval.Evaluation.solo_refit_imc! — Method
solo_refit_imc!(ctx, spec::SoloSpec, seed_E, width, selection_halfwidth; on_fail = NaN)
    -> (imc°, lo_ns)

The width-delivery dispatch of a :refinement solo evaluation, one mechanical map from spec.axis to the refit call, shared by the basin tuning's refits and the ladder's solo mode so tuning and test deliver the width identically. The seed is swept under the context's solo template (ctx.solo_scoring):

  • LossWidth() — the absolute loss width width: a native refiner's strategy runs at it (with_width(lo, fixed_width(lo, width²)), its seed swept at selection_halfwidth — scoring only, never a width), an external refiner refines at it;
  • OutlierHalfwidth() — the LO at its own native nat′ = width.
source
VisualGeometryToolkitEval.Evaluation.stream_selections — Method
stream_selections(refits_of, T, problem_type, loaders, grids, scoring, backend, batch;
                  solo_scoring = scoring)
    -> (M::Dict{label, Vector{Matrix{T}}}, s2_by_scene, np_per)

The scene stream of every selection+LO experiment. loaders[i]() returns scene i's (pool, residuals) (VisualGeometryToolkitEval scene_loaders(dir, scenes; residuals = true)). Per scene: stage each batch of pairs once per staging kind (VisualGeometryToolkitCore stage_selection on backend, shared by the grids whose scorings declare the same selection_staging), evaluate every selection grid of grids (key => (widths, scoring): the scoring's native widths and the scoring itself — VisualGeometryToolkitCore select_on_grid!) on it, then one apply_lo_refits call over refits_of(idxs_of, scene_index) (idxs_of[key] = that grid's per-(pair, column) selections) on the light records. M[label] holds each refit's per-scene matrix in element type T; s2_by_scene the records' σ̂², np_per the pairs per scene. scoring and solo_scoring are the refits' scoring templates.

source
VisualGeometryToolkitEval.Evaluation.summarize_lo_timing — Method
summarize_lo_timing(res::SampleBudgetResults) -> Vector{NamedTuple}

Per-method wall-clock summary: mean warm LO call [µs] (Render's lo_us_per_call), total warm calls, and the cold startup cost [ms]. Per-call numbers are measured in-situ under the parallel run, so they include thread contention (run single-threaded for isolated per-call cost).

source
VisualGeometryToolkitEval.Evaluation.tune — Function
tune(rng, method::BootstrapMethod, val_pairs, n) -> params

Tune method on n validation pairs drawn by rng from val_pairs (the protocol's (scene, pair) indices); params is method-specific and is handed back to evaluate.

source
VisualGeometryToolkitEval.Evaluation.tuned_width — Method
tuned_width(art::TuningArtifact, arm::ComposedSpec) -> Float64

The refit width arm deploys from the tuning artifact art: its recorded tuned_width through operating_width (a SelectionWidth arm's is its tuned_selection, which the record must equal), in the CURRENT currency. The one reader of a tuned width; a caller that needs it raw (to refit, render or carry it into another artifact) calls this, never art["tuned_width"].

A format-7 file's width is returned as stored. A format-6 file (art.format == 6, read through its version step, load_tuning_artifact) stored an EfficiencyConstant (VisualGeometryToolkitCore's) arm's constant as the value its ctor took then: this applies the arm's own format6_constant hook (ComposedSpec; the identity unless the roster declares one), once, so arm.ctor(tuned_width(art, arm)) is the LO the sidecar was tuned with. No other arm and no format-7 file is converted.

So a format-6 width carried into a new artifact (a resumed run) is carried through this, and the format-7 file holds the plain value.

source
VisualGeometryToolkitEval.Evaluation.width_delivery — Method
width_delivery(P::WidthAxis, lo, value, n) -> NamedTuple

How a width value reaches the refit of lo over n columns, as LoRefitSpec keywords. An efficiency constant is baked into the LO object; an outlier half-width is the refit's native (natives); a loss width is an absolute width (widths), the solo delivery bitwise whatever the refiner.

source
VisualGeometryToolkitEval.Evaluation.window_columns — Method
window_columns(window::EfficiencyConstantWindow, grid, tonat, scale) -> Vector{Int}

Column indices of the native grid inside the standardized window [tonat(window.lo, scale), tonat(window.hi, scale)] — the candidate set of every windowed free argmin (the bootstrap tuner here, the miscalibration calibration tune of LocalOptimizationDoneRight). Falls back to the EDGE CLAMP (the grid boundary the window crossed) when the window slid entirely off the grid (extreme miscalibration on an unwidened grid); never a snapped candidate set.

source

Internals

These names are not public; they are listed for contributors.