API: Evaluation
The coarse-to-fine search, the bootstrap tuning harness and the composition evaluator.
Public names
VisualGeometryToolkitEval.Evaluation — Module
EvaluationThe evaluators. The composition evaluator runs a scorer composed with a local optimizer (LO) on stored pool pairs (build_pair_ctx, refit_imc!, apply_lo_refits, evaluate_sample_budget, live_runs, Roster). It is written against VisualGeometryToolkitCore's scoring and local-optimization contracts only, so it runs any scorer with any optimizer. The module also holds the coarse-to-fine search of every cross-validation (coarse_fine_search, coarse_fine_cv) and the bootstrap tuning harness (WindowedGridMethod, compute_method_rows).
VisualGeometryToolkitEval.Evaluation.NativeAxis — Type
NativeAxisThe native width axes, Union{OutlierHalfwidth, LossWidth} (VisualGeometryToolkitCore WidthAxis): what a selector grid, a selection sweep or a refiner's own native lives on. EfficiencyConstant() is a width parameterization, never a native.
VisualGeometryToolkitEval.Evaluation.OPERATING_EFFICIENCY_CONSTANT — Constant
OPERATING_EFFICIENCY_CONSTANTThe refit protocol's operating efficiency constant, 2.5 by default: the scale multiple at which the outlier-half-width ↔ loss-width crossover is inverted when a loss-width-consuming refiner rides an outlier-half-width selector grid (consumed_native). A definition of the refit protocol, not a tuned value: configs/default.toml [refit] operating_efficiency_constant.
VisualGeometryToolkitEval.Evaluation.BootstrapMethod — Type
BootstrapMethodA tuner the bootstrap harness runs (bootstrap_trials): it implements tune and evaluate. WindowedGridMethod is the one in this package.
VisualGeometryToolkitEval.Evaluation.ComposedSpec — Type
ComposedSpec(base, sel, selection::WidthAxis, kind::WidthAxis, ctor; source = TunedWidth(),
format6_constant = identity)One composed score+LO pipeline. sel is the key of its selection sweep (the selections idxs_of[sel] a refit seeds from); selection is that sweep's native axis — VisualGeometryToolkitCore's width_axis of the selection scoring (OutlierHalfwidth() for the Student-t marginal, LossWidth() for a threshold score); kind is the arm's width parameterization; ctor(value) builds the LO strategy at a width value (an efficiency-constant arm bakes it in; the other kinds deliver it through their port, see width_delivery); source is where the width value comes from (WidthSource): a SelectionWidth arm requires kind == selection, and the tuning skips it.
format6_constant(m) is the arm's own reading of an efficiency constant m stored by a format-6 tuning sidecar, as the PLAIN multiple ctor takes now (applied by tuned_width, and only there). Format 6 stored the value the arm's ctor took then, in whatever currency that ctor read it, and only the roster that wrote the ctor knows it, so the roster declares it per arm. The default is the identity: a value the ctor already reads as the plain multiple, or one its own map converts. An arm whose ctor took a Geman–McClure half-weight depth directly (a bare Irls{GemanMcclureShape} at the stored value) declares format6_constant = m -> decision_width(GemanMcclureShape(), m).
VisualGeometryToolkitEval.Evaluation.FlatProtocol — Type
FlatProtocol(val_pair_idx, test_pair_idx)Fixed deterministic pair-level split, bootstrap pairs only. Trial-level variance reflects pair-sampling variance only (no scene component). Used for ScanNet1500 and other "uniform sample of pairs" datasets.
VisualGeometryToolkitEval.Evaluation.LoRefitSpec — Type
LoRefitSpec(label, lo, axis, grid, idxs; bundle=false)One post-LO refit of a selection sweep: refine the selected model idxs[p, t] with strategy lo at native parameter consumed_native(lo, axis, grid[t]). axis is the SELECTOR grid's native axis (a NativeAxis: OutlierHalfwidth() | LossWidth()) — cross-axis conversion happens per column inside the refit layer, so callers never pre-convert grids. lo = nothing records the selection imc (no refit).
Column t refits the model selected at column seedcols[t] of idxs at the EXPLICIT native value natives[t] when those fields are set (no consumed_native conversion — the value is already in the refiner's units). Both default to the standard coupling (seedcols[t] = t, natives === nothing ⇒ derive from grid[t]). This is what the tuning grids (one fixed width parameter per spec) and the fixed-width deployment evaluations (seed at the ζ-scaled column, refit at the exactly-scaled native) are built from.
widths[t] (an absolute loss width): refit at the loss width widths[t] whatever the refiner — the BITWISE-identical call solo_refit_imc! makes, so the tuning and the ladder's test side deliver the width identically. A native refiner sweeps its seed at the column's nat′ (from grid[t] — scoring only, never a width) under the context's solo template and deploys with_width(lo, fixed_width(lo, widths[t]²)); an external refiner refines at widths[t]. Mutually exclusive with natives.
columns restricts which grid columns are EVALUATED (default: all). The others keep the imc matrix's NaN fill. The matrix stays full-width and every column index keeps its meaning, so boundary detection, the evaluation's seed columns and the checkpoint are untouched — this only stops computing cells that a windowed tune never reads, because it argmins inside the standardized window and the grid is far wider than that window. Measured on the PT-E 5% run of 2026-10-02: 45,567 of 386,000 cells per pair were inside the searched windows; the other 88% were computed and discarded. (A cell outside the searched window is never read by a windowed argmin.)
fixed_s2s[t] (a fixed external scale): refit at the column's native exactly as the default path (the refit template), with the strategy deployed at that scale, with_width(lo, fixed_width(lo, fixed_s2s[t])) — the refit_imc!(…; s2) conduit — so its loss width is efficiency_constant·√s2 instead of the consensus set's efficiency_constant·ŝ. The consensus-anchoring ablation evaluation (miscal_evaluation_refit with plugin_scale) is built from it. Combines with natives; mutually exclusive with widths.
VisualGeometryToolkitEval.Evaluation.LossWidthSpec — Type
LossWidthSpec(name, scoring, loss_width, lo = nothing; refinement_loss_width = nothing)One loss-width column of evaluate_sample_budget: selection by scoring, a loss-width family's template (its width_axis is LossWidth(): VisualGeometryToolkitCore's MsacScoring(dg) or RansacScoring(dg), or a paper's family with the same with_width and score! methods), built at loss_width (pool frame) with with_width(scoring, LossWidth(loss_width)). Its running best is refit by its own refiner lo (a paper's truncated refiner — an ExternalRefit strategy such as PoseLib's TLS-LM) at refinement_loss_width, the arm's tuned refinement loss width (nothing: the selection loss width). With lo = nothing the column is selection only: its LO checkpoints record the selected model's model_imc, the same statistic as its batch checkpoints, and cost no refit.
VisualGeometryToolkitEval.Evaluation.OracleSelector — Type
OracleSelector(name, outlier_halfwidth)Oracle selector: the ground-truth-best SINGLE model by model_imc. Recomputes the imc per model (pruned by VisualGeometryToolkitCore's pool_imc, the stored lower bound on that imc) and refines that one model under LO (at a representative outlier half-width, the first method's, unless the config names the arm's refinement native).
VisualGeometryToolkitEval.Evaluation.PairCtx — Type
PairCtxEverything the refit layer needs about one pair. Built once per pair by build_pair_ctx and reused across every (LO refit spec × grid column), it holds the pair itself (a light PoolPair), its pool-frame problem (ep), the optional RelposeBundleProblem (bp, nothing unless bundle=true), the geometry, the problem class (problem_type) and the refit scoring TEMPLATES: scoring, the scoring every native refit is built from at its own outlier half-width (with_width(scoring, OutlierHalfwidth(nat′))), and solo_scoring, the one an absolute-width (solo) refit sweeps its seed with (by default scoring).
VisualGeometryToolkitEval.Evaluation.PairGeom — Type
PairGeomPer-pair calibration and ground truth, the context that scores a pool model's pose error. It scores any stored pool-frame fundamental's coherent imc pose error and holds the pool-frame correspondences cs (x1 => x2) as a concretely typed Correspondences.
VisualGeometryToolkitEval.Evaluation.RefitHalfwidth — Type
RefitHalfwidth(lo, outlier_halfwidth, scoring_of)
RefitHalfwidth(lo, outlier_halfwidth, template::AbstractScoring)lo refitting under the refit scoring at ITS OWN outlier half-width instead of the selection's: the in-loop form of the refit layer's native port (refit_imc!'s nat′, refit_model). The self-scaled arms (Stu-tU IRLS, BCD) read their half-width from the scoring the loop hands them, so inside a RANSAC loop they would otherwise refit at the SELECTION half-width — not the refit half-width their sequential CV tuned at the fixed a*. scoring_of(problem) builds the refit scoring; the template form is with_width(template, OutlierHalfwidth(outlier_halfwidth)).
In the loop, each call re-scores the model under the refit scoring in the loop's own buffers and runs the loop's own Phase-2 step on that consensus set (VisualGeometryToolkitEstimation's locally_optimize_rescored!): at outlier_halfwidth = the selection's it is the score the loop just ran, so the wrapped arm reproduces the bare arm exactly. The refined model goes back to the loop, which re-scores it under the SELECTION scoring and keeps it only if it beats the incumbent.
VisualGeometryToolkitEval.Evaluation.Roster — Type
Roster(composed, solo, basin)A paper's arm roster, the argument of every reader that rebuilds arms from a tuned sidecar (deployed_budget_settings, deployed_basin_settings, audit_operating_points). Each field is the paper's builder:
composed(problem::Symbol) -> Vector{ComposedSpec}— the composed score+LO roster (efficiency-constant / outlier-half-width / natively delivered loss-width arms) of the composed miscal and budget evaluations;solo(problem::Symbol; role::Symbol) -> Vector{SoloSpec}— the solo roster;basin(problem::Symbol) -> Vector{ComposedSpec}— the basin roster (loss-width and outlier-half-width arms only; an efficiency constant has no meaning on the basin axis).
VisualGeometryToolkitEval.Evaluation.SampleBudgetConfig — Type
SampleBudgetConfigConfiguration for the sample-budget selection protocol.
Fields (the schedule and the LO are REQUIRED: a sample-budget experiment
states its protocol of record, e.g. a paper's samplebudgetprotocol.toml)
sample_cap::Int— total minimal samples drawn per pairbatch_size::Int— selection (non-LO) checkpoint stridelo_every::Int— LO checkpoint stride (after the first batch)lo_strategy::Union{Nothing, AbstractOptimizationStrategy}— LO method (nothing: none)budget_unit::Symbol— budget axis::modelscounts essential matrices inspected (Shekhovtsov's unit);:samplescounts minimal subsets.
Checkpoint schedule (both curves share one cumulative-sample x-axis)
- Selection (running-best, no LO): every
batch_size—100, 200, …, cap. - LO (refit the running best): after the first batch, then the
lo_everygrid —{batch_size} ∪ {lo_every, 2·lo_every, …, cap}=100, 500, 1000, ….
VisualGeometryToolkitEval.Evaluation.SampleBudgetMethodSpec — Type
SampleBudgetMethodSpec(name::Symbol, scoring)A scoring-method column: a label and the scoring it selects with, at its validated width (a Student-t + uniform marginal at its prior and outlier half-width). Its running best is refit by the config's LO at the config's native, else at the scoring's outlier half-width. name is opaque to the evaluator.
VisualGeometryToolkitEval.Evaluation.SampleBudgetResults — Type
SampleBudgetResultsPer-pair error curves for each method column, plus LO-refined error at each LO checkpoint and the LO wall-clock accounting (see summarize_lo_timing).
Fields
methods::Vector{Symbol}— method labels (length M)sample_counts::Vector{Int}— cumulative-sample axis for batch curves (B)lo_counts::Vector{Int}— cumulative-sample axis for LO checkpoints (L)pair_ids::Vector{String}— per-pair identifiers (P)err_batch::Array{Float32,3}— (P, M, B) running-best imc [deg]best_idx_batch::Array{Int32,3}— (P, M, B) pool index of running besterr_lo::Array{Float32,3}— (P, M, L) LO-refined imc [deg]lo_ns_total/lo_calls_total/lo_startup_ns::Vector{Int64}— per-method warm LO wall-clock [ns] / call count / cold first-call [ns] (JIT + one call)
VisualGeometryToolkitEval.Evaluation.SampleSelector — Type
SampleSelectorA running-best selector of the sample-budget engine. It folds pool models into a running best (update!), reports that best's pose error at a checkpoint (checkpoint) and refits it through the shared refit layer (lo_error!).
VisualGeometryToolkitEval.Evaluation.ScoreGuarded — Type
ScoreGuarded(inner::AbstractOptimizationStrategy)LO-RANSAC best-so-far monotonicity wrapper for IRLS-family arms. The inner LO runs unchanged, but the refined model is kept only if its SELECTION score (the same marginal that seeded it) does not drop below the seed's (VisualGeometryToolkitEstimation's score_guard). It preserves score monotonicity while the optimizer descends a different objective.
Its RefitFamily and the external-refit bracket (external_workspace, prepare_external!, refine_external!) are inner's, as on VisualGeometryToolkitEstimation's CountedLO and TerminalLO: a guarded external arm is routed, prepared and refined as the bare arm. The guard is score_guard on the native path, so an external refit is forwarded unguarded.
VisualGeometryToolkitEval.Evaluation.ScoreSelector — Type
ScoreSelector(name, scoring, width, ws, scores; lo = nothing, refinement_width = nothing)Scoring-based selector: the running argmax of scoring's score (VisualGeometryToolkitCore score! on the scorer's workspace ws, per-point scratch scores) over the pool models; width is the native width scoring selects at (its outlier half-width, or its threshold). Its running best is refit by the config's LO (lo = nothing) at the config's native (nat_ref, else width), or by its own LO lo at refinement_width (else the config's nat_ref, else width) — the MSAC + truncated-refiner columns.
VisualGeometryToolkitEval.Evaluation.SelectionWidth — Type
SelectionWidth()The arm's refit width IS its selection's tuned width, never tuned (see WidthSource).
VisualGeometryToolkitEval.Evaluation.SoloSpec — Type
SoloSpec(base, role, axis::WidthAxis, ctor)One role (:selection | :refinement) validated in isolation at one width value on axis (see the block comment above); ctor(value) returns a pool scorer (selection) or an LO strategy (refinement).
VisualGeometryToolkitEval.Evaluation.TunedWidth — Type
TunedWidth()The arm's own refit width, tuned at its selector's tuned selection width (see WidthSource).
VisualGeometryToolkitEval.Evaluation.WidthSource — Type
WidthSourceWhere a ComposedSpec's refit width comes from: TunedWidth or SelectionWidth (see the block comment above).
VisualGeometryToolkitEval.Evaluation.WindowedGridMethod — Type
WindowedGridMethod(label, scene_errors, scene_sigma2, native_grid, window, to_native,
criterion)Tunes on the standardized efficiency-constant axis (accv §common-cutoff, eq. 28) instead of the raw native grid, with the reference scale estimated from the validation draw so it tracks the validation cardinality n — the protocol of the windowed_bootstrap tuning scheme.
scene_errors[i]— precomputed (pairs × |native_grid|) post-LO error matrix on a fixed grid of native hyperparameter values (native_grid).scene_sigma2[i][j]— pair(i,j)'s noise varianceσ̂²(e.g.gt_variance), used to form the cardinality-tied scaleσ_med(n).to_native(efficiency_constant, scale) -> native— the eq-(28) map for this scoring, a closure over the caller'snative_map(axis; d_g)so this package stays decoupled from the scoring-spec types.window— the spec'sEfficiencyConstantWindow;criterion— its objective (no default: both are parameters of the tuning spec).
Per trial, tune draws n val pairs, forms σ_med(n)=√median σ̂² over them, and takes the FREE ARGMIN of the val objective over every native_grid column inside the standardized window [to_native(lo, σ_med), to_native(hi, σ_med)] (the window's covered range at the drawn supplied scale — corrupting the scale slides the window, which is the miscalibration mechanism). evaluate reads the tuned column on the test pairs.
VisualGeometryToolkitEval.Evaluation.apply_lo_refits — Method
apply_lo_refits(problem_type, specs::Vector{LoRefitSpec}, pairs;
scoring, solo_scoring = scoring,
operating_efficiency_constant = OPERATING_EFFICIENCY_CONSTANT, on_fail = nothing)
-> Dict{String, Matrix{Float64}}Run every refit spec over pairs and return each spec's post-LO error matrix. scoring is the refit scoring TEMPLATE (rebuilt at each refit's own outlier half-width), solo_scoring the absolute-width (widths) columns' template. The run is threaded :greedy over the flat (pair × spec) product so neither a subsampled validation side nor the arms' very uneven per-spec cost leaves threads idle (see the comment in the body), with one PairCtx built per task at exactly that spec's bundle. pairs are light pool records (PoolPair): a refit never reads a residual matrix. on_fail = nothing ⇒ seed-error fallback (LO never worsens the running best); supply NaN for the ladder convention (failed refit ≠ success). Returns label => (P × T) imc matrix.
VisualGeometryToolkitEval.Evaluation.audit_operating_points — Method
audit_operating_points(roster::Roster, path; problem = nothing) -> trueAudit the tuning sidecar at path through its gate-enforced reader. A basin sidecar goes through deployed_basin_settings, a composed one through deployed_budget_settings; it throws on any clipped arm (a boundary argmin never reaches a consumer) and on any other artifact format, and returns true. MANDATORY before a figure or table consumes the sidecar.
VisualGeometryToolkitEval.Evaluation.batch_sample_counts — Method
The cumulative sample counts of the selection checkpoints: batch_size:batch_size:sample_cap.
VisualGeometryToolkitEval.Evaluation.bootstrap_trials — Method
bootstrap_trials(method, proto::FlatProtocol, budget::BudgetSampling, n;
max_error = maa_threshold())
-> (trial_meds, trial_maas, trial_idxs)The per-draw bootstrap: each of the budget.n_draws draws tunes the hyperparameter on n val pairs of proto (tune) and evaluates the tuned column on its test pairs (evaluate): the draw's median test error (a pair without a model counts as Inf) and its mAA@max_error (the eligibility rule, eligible_maa). Draw r uses budget_rng(budget, n, r).
VisualGeometryToolkitEval.Evaluation.build_pair_ctx — Method
build_pair_ctx(problem_type, pair::PoolPair; scoring = nothing, solo_scoring = scoring, bundle = false) -> PairCtxBuild the PairCtx of one pool pair, once. It holds the family's pool-frame problem (VisualGeometryToolkitEstimation's EssentialProblem(pair), FundMatProblem(pair) or HomographyProblem(pair)), Kc, GT pose and pool-frame points; plus the Schur bundle problem when bundle = true. problem_type is the pool's track (problem_type(pool.problem)). scoring is the refit scoring template (any scoring with an outlier half-width: the caller's — a paper's Student-t + uniform marginal at its prior); nothing is a context that only reads selections (no native refit).
VisualGeometryToolkitEval.Evaluation.checkpoint — Function
checkpoint(s::SampleSelector, geom::PairGeom, pair) -> (imc::Float32, best_idx::Int32)The running best's imc pose error [deg] and pool index, recorded at a checkpoint; (NaN32, 0) before any model was folded in.
VisualGeometryToolkitEval.Evaluation.coarse_fine_cv — Method
coarse_fine_cv(evalfn, coarse::SpacedGrid, scheme::CoarseFine; method_name, label, kwargs...) -> Dictcoarse_fine_cv_batched of one cell and a point-wise evalfn(x).
VisualGeometryToolkitEval.Evaluation.coarse_fine_cv_batched — Method
coarse_fine_cv_batched(curvesfn, coarse::AbstractDict{K,<:SpacedGrid}, scheme::CoarseFine;
method_name = Symbol, label = string, param_name = :x, context_tag = "")
-> Dict{K,Dict{Symbol,Any}}coarse_fine_search of one scalar per cell, curvesfn(xs_of) -> Dict(k => [errors at x for x in xs_of[k]]). Per cell the record :coarse_spec, :x_star (coarse), :boundary_hit, :n_extensions, :curve_ys, :fine_spec, :fine_curve, :fine_idx, :fine_bracket, :x_star_fine (the tuned value) and the identity fields.
VisualGeometryToolkitEval.Evaluation.coarse_fine_search — Method
coarse_fine_search(errorsfn, cells, scheme; label = string) -> Dict{K,NamedTuple}The coarse-fine search, both stages: fine_search of coarse_search.
VisualGeometryToolkitEval.Evaluation.coarse_search — Method
coarse_search(errorsfn, cells::AbstractDict{K,<:NamedTuple}, scheme; label = string)
-> Dict{K,NamedTuple}Stage 1 of the coarse-fine search (module header): per search key k, cells[k] names each axis's coarse SpacedGrid (grid_spec of the spec's axis); the objective over their product, each axis extended on a side while the joint argmin lies within the spec's edge_margin of that face. scheme is a JointCoarseFine or a CoarseFine (one axis, named by the cells). Per key (; names, coarse, coarse_objective, n_extensions, coarse_argmin, boundary_hit, evaluated): the axis names, the final coarse grid specs, the objective over their product (an ndims = d array), the extension rounds (d × 2, lo and hi), the argmin and the per-axis boundary flags as NamedTuples, and every evaluated point => errors.
VisualGeometryToolkitEval.Evaluation.compute_method_rows — Method
compute_method_rows(method, budget::BudgetSampling; protocol, sizes = budget.sizes)
-> (summary::Vector{NamedTuple}, trials::Vector{NamedTuple})Per-method, per-n evaluation loop over the validation budgets sizes (the schedule's own by default; a miscal evaluation supplies its one fixed budget): budget.n_draws draws per budget (bootstrap_trials), the BCa CIs resampled with resampling_rng(budget, n). Returns two row vectors in canonical DataFrame-friendly shape:
summary: one row perncarrying the BCa CIs and per-trial argmin diagnostics that were the columns of the per-cellsummaryDataFrame.trials: one row per (n, trial) carrying the raw(trial_med, trial_mAA, trial_idx)so CIs can be replayed at any α downstream.
protocol is the val/test split (FlatProtocol). (The produce_or_load cache and the decoupled tune_protocol path were pruned 2026-09-21 with their only caller, evaluate_val.)
VisualGeometryToolkitEval.Evaluation.consumed_axis — Method
consumed_axis(lo) -> NativeAxisThe native axis refiner lo consumes, by its refit family: a native refiner its scoring outlier half-width (OutlierHalfwidth()), an external one its truncated loss's width (LossWidth()).
VisualGeometryToolkitEval.Evaluation.consumed_native — Method
consumed_native(lo, axis::NativeAxis, native::Real;
operating_efficiency_constant = OPERATING_EFFICIENCY_CONSTANT, d_g) -> Float64The native width THIS refiner consumes (consumed_axis), given a selector grid value native on native axis axis. Same-axis is identity; cross-axis converts at the operating_efficiency_constant crossover (VisualGeometryToolkitCore outlier_halfwidth_to_loss_width / loss_width_to_outlier_halfwidth) for residual codimension d_g (codimension(problem_type): 1 for E/F, 2 for H).
VisualGeometryToolkitEval.Evaluation.deployed_arm_settings — Method
deployed_arm_settings(roster::Roster, path; problem = nothing) -> Vector{label => (lo, fixed_native)}Each pipeline's DEPLOYED configuration: the LO strategy at its tuned width and its fixed native. It is the (lo, ref) projection of deployed_budget_settings, under the same interior-argmin gate.
VisualGeometryToolkitEval.Evaluation.deployed_basin_settings — Method
deployed_basin_settings(roster::Roster, path)
-> Vector{base => (spec::SoloSpec, width::Float64)}Each basin arm's tuned width from a basin tuning sidecar (mode = "basin"). The arms are rebuilt through roster.basin(problem) and repackaged as a refinement SoloSpec whose axis is the arm's parameterization, so tuning and test deliver the width identically via solo_refit_imc!. Refuses a non-basin sidecar. Same interior-argmin gate as the composed readers.
VisualGeometryToolkitEval.Evaluation.deployed_budget_settings — Method
deployed_budget_settings(roster::Roster, path; problem = nothing)
-> Vector{label => (lo, sel, ref, kind)}Per-arm tuned operating points of a composed tuning sidecar, rebuilt through the roster. The sidecar is read through VisualGeometryToolkitEval load_tuning_artifact, which refuses another format; each arm is rebuilt through roster.composed(problem): the LO instance (the tuned scale multiple baked in for efficiency-constant arms), the SELECTION native and the REFINEMENT native (efficiency-constant arms refit at the selection native — consensus-anchored; native parameterizations at their tuned value) and the arm's width parameterization kind — the roster's, the one source of an arm's kind (a sidecar's own kinds list is not read). problem defaults to the sidecar's own "problem".
INTERIOR-ARGMIN INVARIANT: a HARD ERROR on any arm whose persisted boundary_hits is non-empty — expand the window on the flagged side and re-run the tuning (a SelectionWidth arm has no width of its own to clip: its list carries the selection's axis alone, and its recorded width must be its selection's, operating_width). A basin sidecar is refused (deployed_basin_settings reads it).
A format-6 sidecar (read through its version step, load_tuning_artifact) stored an efficiency constant as the value its arm's ctor took then: it is read through the arm's format6_constant hook (tuned_width), so the deployed LO is the one the sidecar was tuned with.
VisualGeometryToolkitEval.Evaluation.deployed_setting — Method
deployed_setting(path; context = path, allow_edge = false) -> NamedTupleThe gated reader of a coarse-fine tuning artifact (save_coarse_fine_artifact): the selected value of every axis, as a NamedTuple. HARD ERROR when the spec is not a coarse-fine one or the search hit its window's edge on any axis (enforce_probe_interior; allow_edge = true downgrades to a warning for FORENSIC reads only). A tuned setting is deployed through this, never from a literal.
Also reads the one-scalar layout written before the joint search (a CV "record" and "x_star_fine"; BlobBoards' half-width run of record): (; <param_name> = x_star_fine).
VisualGeometryToolkitEval.Evaluation.enforce_probe_interior — Method
enforce_probe_interior(rec_or_flag; context, allow_edge = false)Read-side gate of the coarse-fine search: HARD ERROR when a record (or an explicit boundary_hit::Bool) is flagged — the coarse window clipped the optimum even after the extension budget, so the tuned value is suspect and must never be silently consumed. Re-run the search (it expands itself; if it still hits the budget, the objective has no interior optimum on any reachable window and the method/cell needs attention). allow_edge = true downgrades to a loud warn — FORENSIC reads only, never production. Mirrors the LO engine's _enforce_interior_argmin gate on the product-grid readers.
VisualGeometryToolkitEval.Evaluation.essential_bearings — Method
essential_bearings(ctx::PairCtx) -> (f₁, f₂)The unit bearings of an E context's problem (what a cheirality vote reads). Read from ctx.ep, which already back-projected every correspondence through the pool camera, rather than re-derived. Only an E context has bearings: on an F or H context ctx.ep holds pool pixels, so this refuses rather than hand 2-vectors to a ray test.
VisualGeometryToolkitEval.Evaluation.evaluate — Function
evaluate(method::BootstrapMethod, test_pairs, params) -> Vector{Float64}The test errors of method at the tuned params on test_pairs (the protocol's (scene, pair) indices).
VisualGeometryToolkitEval.Evaluation.evaluate_sample_budget — Method
evaluate_sample_budget(problem_type, methods::Vector{<:SampleBudgetMethodSpec}, pairs, residuals;
pair_ids, loss_width_specs::Vector{<:LossWidthSpec}, cfg::SampleBudgetConfig,
sample_starts = nothing, refit_scoring = methods[1].scoring)
-> SampleBudgetResultsRun the sample-budget protocol over the pool pairs pairs. The pairs are a flat vector of light pool records (PoolPair) with their residual matrices residuals (residuals[i] is pair i's; VisualGeometryToolkitEval load_pool_residuals, residual_matrix). A :samples budget (cfg.budget_unit) counts minimal samples through sample_starts[i], pair i's minimal-sample starts — VisualGeometryToolkitEval sample_starts(pool), the one reconstruction of sample boundaries; it is required then and ignored for a :models budget. Columns: the scoring methods, then one loss-width column per LossWidthSpec (MSAC, or a paper's loss-width family; selection only under no local optimization, nothing), then the oracle (last). Every checkpoint records the running best's model_imc. Every native refit is seeded under refit_scoring (a scoring TEMPLATE, rebuilt at each refit's own outlier half-width) — the methods' own: they must share it. Parallel over pairs; per-method scoring workspaces preallocated per thread.
VisualGeometryToolkitEval.Evaluation.evaluate_sample_budget_pair! — Method
evaluate_sample_budget_pair!(out_err_batch, out_best_idx_batch, out_err_lo,
out_lo_ns, out_lo_calls, selectors::Tuple,
problem_type, pair, R, cfg; refit_scoring = nothing,
sample_starts = nothing)The per-pair worker of evaluate_sample_budget: walk one pair's pool and fill its slice of the results.
pair is the light pool record, R its residual matrix (numpoints × nummodels, Float32, the pool frame). sample_starts is the pair's minimal-sample starts (VisualGeometryToolkitEval sample_starts), required when the budget counts :samples. The selectors — a Tuple, so every per-model call is statically dispatched — are mutated too (reset per pair, then folded), so the problem type follows them. refit_scoring is the per-pair context's refit template — the one every refit's scoring is built from at its own native — so it must be the selectors' own.
VisualGeometryToolkitEval.Evaluation.fine_search — Method
fine_search(errorsfn, coarse::AbstractDict{K,<:NamedTuple}, scheme) -> Dict{K,NamedTuple}Stage 2 of the coarse-fine search: per key, each axis's fine spacing over the two coarse neighbours of the coarse argmin on that axis, their product evaluated once (no extension), its free argmin the selection. Each record of coarse_search gains fine (the fine grid specs), fine_objective and selected (a NamedTuple of axis values); evaluated gains the fine cells. A spec's incumbent is evaluated (one more call) and the record gains incumbent, incumbent_errors, gain (objective, p_gain) and kept_incumbent; selected is the incumbent when kept.
VisualGeometryToolkitEval.Evaluation.fine_stage — Method
fine_stage(evalfn, rec, scheme::CoarseFine) -> (; x_star, idx, spec, grid, curve, bracket)The fine stage alone (fine_search) of a probe_curve record and a point-wise evalfn(x).
VisualGeometryToolkitEval.Evaluation.guarded_refit — Method
guarded_refit(opt, score, on_fail; warn_label = nothing)
-> (err::Float64, lo_ns::Int64)The seed-fallback + timing bracket.
opt() -> Union{Nothing, M}— the bracketed optimizer call (data-in → result-out). Throws are caught;nothingsignals bail / no consensus.score(m::M) -> Float64— GT scoring of the refined model, NOT timed. A non-finite score falls back toon_fail(a GT-free decision: the refit produced an unusable model).on_failpolicy is the caller's (refit_imc!'s fallback): budget/producer supply the seed's error (LO-RANSAC invariant — refinement never worsens the running best); the basin ladder suppliesNaN(a failed refit did not travel).lo_nsspans exactlyopt().
VisualGeometryToolkitEval.Evaluation.is_tuned — Method
Is arm's refit width tuned (a TunedWidth arm), rather than its selection's?
VisualGeometryToolkitEval.Evaluation.live_row — Method
live_row(runs; max_error) -> NamedTupleThe summary of one live_runs result. Its fields are med the finite median error [deg], maa the mAA@max_error under the eligibility rule (VisualGeometryToolkitEval eligible_maa: every pool pair is eligible, a run without a model scores 0; max_error defaults to its maa_threshold()), med_trials the median trials, med_wall_ms the median wall [ms], and the outcome counts (n_eligible, n_failed, n_skipped).
VisualGeometryToolkitEval.Evaluation.live_runs — Method
live_runs(scoring, lo, ctxs; problem_of, seed_of, config = RansacConfig(), warmup = true,
lo_of = nothing, on_improvement_of = nothing, score_guard = false,
cancelled = () -> false, finalize = nothing)
-> (; err, trials, wall_s) # plus `extras` when `finalize !== nothing`Run one live production ransac per pair context ctxs[j], threaded over pairs. Each run uses the caller's problem of the pair, problem_of(ctxs[j].pair), the selection scoring, the LO strategy lo, adaptive termination under config. Each pair's stream is StableRNG(seed_of(j)), handed to ransac explicitly, so a run is independent of the thread schedule, identical across calls that share seed_of (seed_of should be a stable_seed). Per pair: err[j] the imc pose error [deg] of the final model, Inf when the run yielded no model (found_model false) or the pair was cancelled; trials[j] the trials drawn; wall_s[j] the wall clock of the ransac call alone. warmup first runs one short solve on pair 1 (result discarded) so no bracket pays JIT.
The keywords below default so that the result is exactly the plain run's.
lo_of(j)returns the LO strategy of pairj, overridinglo. The warmup useslo_of(1)(an LO holds no recorded state).on_improvement_of(j)returns theransacimprovement callback (on_improvement(trial, score, ws), fired when the incumbent improves). It is called withj = 0once for the warmup (return a throwaway callback of the same type as for real pairs, writing to scratch storage, so the loop compiles once and the warmup records nothing) and withj = 1:nfor the pairs.score_guardis forwarded toransac: per-candidate ScoreGuarded acceptance.cancelled()is polled before each pair; a cancelled pair is not run and keepserr = Inf,trials = 0,wall_s = NaN. Pairs are threaded, so a pair already started finishes.VisualGeometryToolkitEval.Experiments.cancelledis the kaimon-aware poll.finalize(j, r)runs after pairj'sransacresultr, on the pair's thread and outside its wall bracket; its values come back asextras::Vector{Any}(nothingfor a pair not run). Withoutfinalizethe result has noextrasfield.
VisualGeometryToolkitEval.Evaluation.lo_error! — Function
lo_error!(ctx::PairCtx, s::SampleSelector, cfg) -> (imc_error::Float32, lo_ns::Int64)LO refit of the running best through the shared refit layer (refit_imc!). lo_ns is the bracketed optimizer wall-clock (0 when there is no model to refine).
VisualGeometryToolkitEval.Evaluation.lo_iteration_budget — Method
lo_iteration_budget() -> IntThe equal LO model-update budget of the evaluation: configs/default.toml [local_optimization_strategy] iteration_budget, so arms compare at equal cost. A paper with its own protocol supplies its budget explicitly instead.
VisualGeometryToolkitEval.Evaluation.lo_sample_counts — Method
lo_sample_counts(cfg::SampleBudgetConfig) -> Vector{Int}The cumulative sample counts of the LO checkpoints: an early point after the FIRST batch, then the regular lo_every grid up to sample_cap.
VisualGeometryToolkitEval.Evaluation.miscal_evaluation_refit — Method
miscal_evaluation_refit(arm, idxs_of, tuned, zetas, grids; d_g, plugin_scale = nothing) -> LoRefitSpecThe miscalibration evaluation refit of one arm, one column per ζ in zetas. Column j seeds at the selection column nearest ζj·selection* and refits at the exactly-scaled refinement native (ζj·width* for the native parameterizations — both formed from the corrupted plug-in — while an efficiency constant is consensus-anchored and NOT corrupted). tuned is the tuned point as NATIVE VALUES, (; selection, width), its width read through operating_width (a SelectionWidth arm: width is nothing or the selection, and column j refits at ζ_j·selection*, the selection's own corrupted native); grids supplies the arm's selection grid, which the seed columns are found in.
plugin_scale (the consensus-anchoring ABLATION, efficiency-constant arms only): the same operating point and seeds, the efficiency constant's loss width delivered at the FIXED external plug-in scale ζj·`pluginscaleinstead of the consensus set's ŝ —withwidth(lo, fixedwidth(lo, (ζj·pluginscale)²))through the LoRefitSpecfixed_s2s` column. A native parameterization's width already is the corrupted plug-in; it is refused.
VisualGeometryToolkitEval.Evaluation.model_imc — Method
model_imc(model, geom::PairGeom) -> Float64
model_imc(model, ctx::PairCtx) -> Float64The pose error [deg] of model against the pair's ground truth. By the model's type: a metric EssentialMat → max(Δr, Δt) of its best decomposition; a pool-frame FundMat → the same after the pool-frame calibration upgrade; a pool-frame HomographyMat → the optimistic decomposition ROTATION error.
VisualGeometryToolkitEval.Evaluation.n_batches — Method
Number of selection checkpoints of cfg: sample_cap ÷ batch_size.
VisualGeometryToolkitEval.Evaluation.n_los — Method
Number of LO checkpoints of cfg (see lo_sample_counts).
VisualGeometryToolkitEval.Evaluation.operating_width — Method
operating_width(arm::ComposedSpec, selection, tuned) -> Float64The refit width of arm at the tuned selection width selection. By its WidthSource: a TunedWidth arm's tuned value tuned; a SelectionWidth arm's selection itself — tuned is nothing (the tuning stage, which has none) or a recorded width, which must equal selection exactly (a sidecar read back).
VisualGeometryToolkitEval.Evaluation.perturbed_seed_E — Method
perturbed_seed_E(rng, R_gt, t_gt, δ) -> EssentialMatThe δ-perturbed metric essential seed [t]ₓR of VgtkCore.Synthetic.perturbed_pose — the seed model every basin-style E refit starts from.
VisualGeometryToolkitEval.Evaluation.probe_curve — Method
probe_curve(evalfn, coarse::SpacedGrid, scheme::CoarseFine; method_name, label, param_name = :x) -> DictThe coarse stage alone (coarse_search) of one cell and a point-wise evalfn(x): the record of coarse_fine_cv_batched without its fine keys.
VisualGeometryToolkitEval.Evaluation.problem_type — Method
problem_type(p::Symbol) -> TypeThe RANSAC problem type of a two-view pool track: :E → EssentialProblem, :F → FundMatProblem, :H → HomographyProblem. The one map from a track's letter (a pool's problem) to its problem class — the value a PairCtx carries as problem_type; any other symbol is an ArgumentError. Inside a function, name the result P (a local problem_type would shadow this function).
VisualGeometryToolkitEval.Evaluation.refit_imc! — Method
refit_imc!(ctx, family::RefitFamily, lo, seed, nat′, scoring, fallback, bundle, s2)
-> (imc::Float64, lo_ns::Int64)The execution path of one RefitFamily (VisualGeometryToolkitCore's Holy trait: NativeRefit, ExternalRefit). refit_imc!(ctx, lo, seed, nat′; …) resolves the keywords and calls this with RefitFamily(lo). A new strategy plugs in through the contracts (locally_optimize!, or refine_external! with its external_workspace), never by a method here.
VisualGeometryToolkitEval.Evaluation.refit_imc! — Method
refit_imc!(ctx::PairCtx, lo, seed, nat′::Float64;
scoring = ctx.scoring, on_fail = nothing, bundle::Bool = false,
s2 = nothing) -> (imc::Float64, lo_ns::Int64)Refine the seed model seed with strategy lo at ITS native parameter nat′ (see consumed_native) and return the refined model's imc error [deg] plus the bracketed optimizer wall-clock.
scoring(NativeRefit): the scoring TEMPLATE the refit is seeded under, rebuilt atnat′(with_width(scoring, OutlierHalfwidth(nat′))) — the context's refit template by default;ctx.solo_scoringfor an absolute-width (solo) refit.on_fail: value on optimizer bail / non-finite refit. Default = the seed's own imc (never-worsen); the ladder suppliesNaN.bundle: route the native refit throughctx.bp(Schur reprojection bundle) instead of the pose-onlyctx.ep.s2: an externally supplied squared scale — the one conduit for a non-consensus width:- NativeRefit: the strategy is deployed at the absolute width
with_width(lo, fixed_width(lo, s2))(its efficiency constant at the scale√s2) — the role-isolated controls (solo CV, oracle-scale controls, the anchoring ablation). The seed sweep still runs; only the LO's width moves. - ExternalRefit: the fixed scale
FixedS2(s2)replacesnat′(by default the loss width√s2).
- NativeRefit: the strategy is deployed at the absolute width
VisualGeometryToolkitEval.Evaluation.refit_model — Method
refit_model(problem, seed, lo, scoring; workspace = nothing, s2 = nothing)
-> Union{Nothing, model}The native (locally_optimize) refit of the refit layer, returning the refined MODEL — nothing when the strategy produced none. lo may be ScoreGuarded (the guard becomes score_guard). s2 deploys the same strategy at an absolute squared scale (with_width(lo, fixed_width(lo, s2)); the scoring sweep is untouched, only the LO's width moves); nothing keeps the strategy's own width mode. The refit layer's imc rows wrap exactly this in guarded_refit; a caller that needs the refined model itself (a polish warm start) calls it directly.
VisualGeometryToolkitEval.Evaluation.refit_seed — Method
refit_seed(ctx::PairCtx, model) -> seedHow a stored pool model becomes the seed of a refit in the pair's problem: an E pool's pool-frame fundamental is upgraded to the metric essential Kc₂ᵀ·F·Kc₁; F and H pool models seed as stored.
VisualGeometryToolkitEval.Evaluation.rigid_invariance_errors — Method
rigid_invariance_errors(poses; targets, units = sort!(collect(keys(poses))), min_obs = 3)
-> (; translation::Matrix{Float64}, rotation::Matrix{Float64}, units, pairs)The invariance errors of rigidly mounted targets. poses[u][m] is target m's estimated pose in unit u (a value with fields R, the rotation, and t, the translation: target → camera, x_cam = R·X + t, e.g. an Extrinsics3); a target absent from poses[u] was not observed there. targets lists every target that could be observed; pairs are its sorted pairs (m, n), m < n.
For each pair and each unit observing both targets, the relative pose T_m⁻¹ T_n is compared with a LEAVE-ONE-OUT reference over the other units observing that pair: the rotation MEDOID (the observation minimising summed geodesic distance to the others — robust to the two-fold flips a mean would be dragged by) and the componentwise MEDIAN translation. Leave-one-out is not optional: a reference containing the scored observation scores it zero by construction and rewards observing each pair in fewer units.
Returns length(units) × length(pairs) matrices: translation (the translation's distance from the reference, in the poses' units) and rotation (geodesic angle, degrees). An entry is Inf where the pair was not observed in that unit, or was observed in fewer than max(min_obs, 3) units (no reference) — so under eligible_maa (the mAA rule: a non-finite error scores 0 and stays in the denominator) a missed pair counts as a failure, and eligible_maa(e, isfinite.(e)) is the accuracy over what was observed.
VisualGeometryToolkitEval.Evaluation.save_coarse_fine_artifact — Method
save_coarse_fine_artifact(path, spec::TuningSpec, rec; unit, code, data...) -> pathPersist one search record rec (coarse_fine_search, one key's NamedTuple) under its coarse_fine / joint_coarse_fine spec as a tuning artifact (save_tuning_artifact): the grids coarse|<axis> and fine|<axis>, and
axes the axis names, the search's order
selected axis => the tuned value (the fine argmin)
coarse_argmin axis => the coarse argmin
boundary_hit axis => the coarse argmin within the margin of a face
after the extension budget (refused by the reader)
n_extensions axis => [lo rounds, hi rounds]
coarse_objective, fine_objective the objective over each product grid
cells every evaluated cell, one row per cell (columns: axes)
unit_statistics per cell, Tuning `unit_statistics` of its errors — when
every cell's errors are a `units × items` matrix
incumbent, kept_incumbent, gain, incumbent_unit_statistics
with a declared incumbent (`selected` is it when kept)data are the caller's further keys (the bootstrap of the selection, say).
VisualGeometryToolkitEval.Evaluation.solo_refit_imc! — Method
solo_refit_imc!(ctx, spec::SoloSpec, seed_E, width, selection_halfwidth; on_fail = NaN)
-> (imc°, lo_ns)The width-delivery dispatch of a :refinement solo evaluation, one mechanical map from spec.axis to the refit call, shared by the basin tuning's refits and the ladder's solo mode so tuning and test deliver the width identically. The seed is swept under the context's solo template (ctx.solo_scoring):
LossWidth()— the absolute loss widthwidth: a native refiner's strategy runs at it (with_width(lo, fixed_width(lo, width²)), its seed swept atselection_halfwidth— scoring only, never a width), an external refiner refines at it;OutlierHalfwidth()— the LO at its own nativenat′ = width.
VisualGeometryToolkitEval.Evaluation.stream_selections — Method
stream_selections(refits_of, T, problem_type, loaders, grids, scoring, backend, batch;
solo_scoring = scoring)
-> (M::Dict{label, Vector{Matrix{T}}}, s2_by_scene, np_per)The scene stream of every selection+LO experiment. loaders[i]() returns scene i's (pool, residuals) (VisualGeometryToolkitEval scene_loaders(dir, scenes; residuals = true)). Per scene: stage each batch of pairs once per staging kind (VisualGeometryToolkitCore stage_selection on backend, shared by the grids whose scorings declare the same selection_staging), evaluate every selection grid of grids (key => (widths, scoring): the scoring's native widths and the scoring itself — VisualGeometryToolkitCore select_on_grid!) on it, then one apply_lo_refits call over refits_of(idxs_of, scene_index) (idxs_of[key] = that grid's per-(pair, column) selections) on the light records. M[label] holds each refit's per-scene matrix in element type T; s2_by_scene the records' σ̂², np_per the pairs per scene. scoring and solo_scoring are the refits' scoring templates.
VisualGeometryToolkitEval.Evaluation.summarize_lo_timing — Method
summarize_lo_timing(res::SampleBudgetResults) -> Vector{NamedTuple}Per-method wall-clock summary: mean warm LO call [µs] (Render's lo_us_per_call), total warm calls, and the cold startup cost [ms]. Per-call numbers are measured in-situ under the parallel run, so they include thread contention (run single-threaded for isolated per-call cost).
VisualGeometryToolkitEval.Evaluation.tune — Function
tune(rng, method::BootstrapMethod, val_pairs, n) -> paramsTune method on n validation pairs drawn by rng from val_pairs (the protocol's (scene, pair) indices); params is method-specific and is handed back to evaluate.
VisualGeometryToolkitEval.Evaluation.tuned_width — Method
tuned_width(art::TuningArtifact, arm::ComposedSpec) -> Float64The refit width arm deploys from the tuning artifact art: its recorded tuned_width through operating_width (a SelectionWidth arm's is its tuned_selection, which the record must equal), in the CURRENT currency. The one reader of a tuned width; a caller that needs it raw (to refit, render or carry it into another artifact) calls this, never art["tuned_width"].
A format-7 file's width is returned as stored. A format-6 file (art.format == 6, read through its version step, load_tuning_artifact) stored an EfficiencyConstant (VisualGeometryToolkitCore's) arm's constant as the value its ctor took then: this applies the arm's own format6_constant hook (ComposedSpec; the identity unless the roster declares one), once, so arm.ctor(tuned_width(art, arm)) is the LO the sidecar was tuned with. No other arm and no format-7 file is converted.
So a format-6 width carried into a new artifact (a resumed run) is carried through this, and the format-7 file holds the plain value.
VisualGeometryToolkitEval.Evaluation.update! — Function
update!(s::SampleSelector, geom::PairGeom, pair, R, j, col_j, n)Fold pool model j (its residual column R[1:n, j], minimal sample col_j) into the selector's running best.
VisualGeometryToolkitEval.Evaluation.width_delivery — Method
width_delivery(P::WidthAxis, lo, value, n) -> NamedTupleHow a width value reaches the refit of lo over n columns, as LoRefitSpec keywords. An efficiency constant is baked into the LO object; an outlier half-width is the refit's native (natives); a loss width is an absolute width (widths), the solo delivery bitwise whatever the refiner.
VisualGeometryToolkitEval.Evaluation.window_columns — Method
window_columns(window::EfficiencyConstantWindow, grid, tonat, scale) -> Vector{Int}Column indices of the native grid inside the standardized window [tonat(window.lo, scale), tonat(window.hi, scale)] — the candidate set of every windowed free argmin (the bootstrap tuner here, the miscalibration calibration tune of LocalOptimizationDoneRight). Falls back to the EDGE CLAMP (the grid boundary the window crossed) when the window slid entirely off the grid (extreme miscalibration on an unwidened grid); never a snapped candidate set.
Internals
These names are not public; they are listed for contributors.
VisualGeometryToolkitEval.Evaluation._summarize_trials — Method
_summarize_trials(rng, meds, maas, idxs, T) -> NamedTupleBCa-bootstrap mean/std + 95% CI per criterion, plus per-trial argmin diagnostics (idx_mean, idx_std, edge_frac; NaN when idxs is empty or T == 0).