Correspondence API

Correspondence types and the solver layer built on them. Terminology: a correspondence (Correspondence) is a pair; a correspondence set (an n-tuple of mutually corresponding elements, e.g. multiview tracks) lives in VisualGeometryToolkitFeatures as CorrespondenceSet; the columnar collection of pair-correspondences is Correspondences.

Descriptor matching (NNMatch, match, strategies) lives in VisualGeometryToolkitFeatures.

Correspondences and Endpoint Lifts

VisualGeometryToolkitCore.Correspondences — Type
Correspondences(sources, targets; attrs...)

Columnar correspondence set: parallel sources/targets vectors plus any number of index-aligned attribute columns (score, ratio, weights, covariance, …) that filter and view IN LOCKSTEP with the endpoints.

cs[i] returns the i-th correspondence as a Correspondence (= Pair); cs[mask] / view(cs, idx) subset every column together — no more hand-written scored[keep]; weights[keep] co-filtering.

Attribute columns are reached by property: cs.score, cs.ratio; presence is queried with hasattr. Endpoint columns are reached through the protocol verbs sources / targets (NOT .first/.second — that duck-typing is retired).

Example

cs = Correspondences(src_pts, dst_pts; score = scores, ratio = ratios)
sources(cs)          # the source column
cs.score             # the score column
inliers = cs[mask]   # endpoints AND attributes filtered together
source
VisualGeometryToolkitCore.MappedColumn — Type
MappedColumn(f, parent::AbstractVector)

Lazy map(f, parent) view: m[i] == f(parent[i]), no allocation, no materialization. Element type is inferred via Base.promote_op.

Used by lift so that endpoint conversions (centroid extraction, z=0 planar lifts, bearing lifts) cost nothing until a solver actually indexes the column.

source
VisualGeometryToolkitCore.PointSet — Type
PointSet(points; attrs...)

A set of single-point observations (a contour's points for a conic, a circle, a line) with index-aligned attribute columns, the one-observation counterpart of Correspondences: xs[i] is the point, xs.Σ (the points' covariances), xs.sqrt_weights and any other column are reached by property, and xs[mask] / view(xs, idx) subset every column together. What the batched Sampson functions (sampson_residuals!) read per element; a plain vector of points is a PointSet with no columns, and withattrs(points; attrs...) makes one.

source
VisualGeometryToolkitCore.block_order — Method
block_order(groups::AbstractDict, given::AbstractDict...) -> Vector{K}

The block order of grouped data: the sorted keys of groups (the order a grouped estimator stacks its per-group unknowns and stores on its result). Every given dict — the per-group models a caller supplies, keyed by the same K — must have exactly the same key set: an ArgumentError otherwise.

source
VisualGeometryToolkitCore.hasattr — Method
hasattr(cs::Correspondences, k::Symbol) -> Bool

Whether the correspondence set carries the attribute column k (e.g. hasattr(cs, :score) before reading cs.score).

source
VisualGeometryToolkitCore.lift — Method
lift(cs; source = identity, target = identity) -> Correspondences

Lazy endpoint conversion: returns a Correspondences whose source/target columns apply source/target element-wise on access (no materialization); attribute columns are forwarded untouched. Works on any container that satisfies the sources/targets protocol.

This replaces eager extraction helpers (point_correspondences) and hand-written z=0 lifts: compose whatever endpoint conversion the solver needs at the call site, pay only when the solver indexes.

source
VisualGeometryToolkitCore.planar_lift — Method
planar_lift(cs) -> Correspondences

points with the source side additionally lifted onto the z=0 plane: 2D source centroids become Point3(x, y, 0) (the planar-board world convention), targets become 2D centroids. The correspondence set is then shaped for absolute-pose estimation.

source
VisualGeometryToolkitCore.sources — Method
sources(cs) -> AbstractVector

The source column of a correspondence container. Protocol verb — defined for Correspondences, the legacy StructArray{<:Pair} columnar layout, Attributed-wrapped (scored) correspondence vectors, and plain AbstractVector{<:Pair} (lazy view). Solver code MUST go through sources/targets rather than .first/.second.

source
VisualGeometryToolkitCore.split_blocks — Method
split_blocks(v::AbstractVector, ids::AbstractVector{K}, counts) -> Dict{K, Vector}

The inverse of stacking per-group vectors in block order: v's consecutive blocks of counts[j] entries, keyed by ids[j] (a copy per group). A DimensionMismatch unless ids and counts have one entry per group and the counts sum to length(v).

source
VisualGeometryToolkitCore.withattrs — Method
withattrs(cs::Correspondences; attrs...) -> Correspondences

A copy of cs with the given attribute columns merged in (same-name columns replaced). Endpoint columns are shared, not copied. The idiom for attaching derived per-correspondence data — e.g. IRLS weights after a fit — so it subsets in lockstep from then on.

source

Solver Layer: Frames, Verbs, Registry

VisualGeometryToolkitCore.Essential5pt — Type
Essential5pt(backend = NativeSolver())

Minimal 5-point essential-matrix solver (Nistér; solve verb over the essential_5pt kernel). sample_size 5, max_solutions 10.

Correspondence endpoints are bearing vectors (StaticVector{3}), and only those. A calibrated 2D point is a MethodError, not an input.

Why a 2-vector is refused rather than lifted

Until 0.29 a calibrated 2-vector was accepted and lifted to (x, y, 1). That lift places the ray on the z = 1 plane, which cannot represent a ray at or past 90° from the optical axis at all — so it silently excluded every fisheye and omnidirectional camera, which are the cameras this stack exists for. The failure was invisible: the lift always succeeded and returned a plausible EssentialMat, computed from a chart that could not hold the data.

Refusing by TYPE is the point. A caller holding pixels builds bearings with backproject — or backproject_with_jacobian when the pixel Jacobian is wanted too, which is what keeps a residual threshold in pixels.

The kernel computes in Float64; returned models are EssentialMat{Float64}. The epipolar constraint is scale-invariant per endpoint, so the algebra does not require unit norm — but unit bearings are the stack's convention and what backproject returns.

What the type refusal does NOT catch

It catches a 2-vector. It cannot catch a 3-vector that is still on the z = 1 plane — (x, y, 1) is a perfectly good StaticVector{3}, and because the epipolar algebra is scale-invariant per endpoint, such a point even produces the right answer for any ray the chart can represent. The exclusion of rays at and past 90° survives that lift unchanged; it has simply stopped being visible in the type.

So this signature is a NECESSARY condition, not a guarantee, and it should not be read as one. The guarantee has to come from where the vectors are BUILT — backproject(model, u), which is defined for every ray the camera can see — not from where they are consumed. A caller that reaches a 3-vector by dividing through by z has moved the same defect one layer up and past this gate.

source
VisualGeometryToolkitCore.Homography5ptRadial — Type
Homography5ptRadial(; backend = NativeSolver(), extra_check = false)

Minimal 5-point radially distorted homography solver — one-sided one-parameter division model on the TARGET side (solve verb over the homography_5pt_radial kernel; Wadenbäck et al. (2026)). sample_size 5, max_solutions 3 (at most 1 with extra_check, which keeps only the root most consistent with the redundant equation — the reference enables it inside its RANSAC wrapper).

Deliberately NOT entered in the minimal_solvers registry: the division coefficient k is only meaningful in roughly unit-scale target coordinates, a convention generic pixel correspondences don't carry (cf. the calibrated convention Essential5pt presumes). Opt in explicitly.

source
VisualGeometryToolkitCore.MinimalSolver — Type
MinimalSolver{M}

Abstract parent for exactly-determined minimal solvers producing models of type M. Each algorithm is its own singleton subtype (two solvers may share a model type but require different correspondence types — p3p vs. circle relative pose both yield poses).

Interface (all required):

  • solve(s, cs, idx) -> FixedModels{max_solutions(s), M} — ALWAYS a FixedModels; failure is count == 0, never nothing
  • sample_size(s) — minimal sample cardinality
  • max_solutions(s) — solution-count bound (the N in FixedModels)
  • model_type(s) == M

Solvers with pluggable numeric backends carry a backend::AbstractSolverBackend field (default NativeSolver()).

source
VisualGeometryToolkitCore.P3P — Type
P3P(backend = NativeSolver())

Perspective-3-point absolute-pose solver (solve verb over the p3p kernel). sample_size 3, max_solutions 4.

Correspondence contract: sources are 3D world points, targets are unit bearing vectors (pixel targets are lifted to bearings beforehand, e.g. lift(cs; target = u -> backproject(model, u))). Models are Extrinsics3 with x_cam = R·X + t.

source
VisualGeometryToolkitCore.Rotation2pt — Type
Rotation2pt()

Minimal 2-point rotation solver: the rotation R with vᵢ ≈ R·uᵢ from two pairs of bearing vectors (StaticVector{3}, any nonzero length — a direction is its unit vector) — the infinite homography of a calibrated camera under a pure rotation, or any two direction pairs. kabsch on the two unit pairs (solve verb); sample_size 2, max_solutions 1.

The rotation is exact on two consistent pairs (the two bearings of a view span a plane, and their cross product completes the frame). A sample whose two bearings are parallel or antiparallel in either view (sine of their angle below 1e-12) leaves a one-parameter family of rotations and returns count == 0. Two pairs whose angles differ (inconsistent with any rotation) still give the least-squares rotation; the residuals judge it.

Deliberately NOT entered in the minimal_solvers registry: its correspondence type, bearing ↔ bearing, is Essential5pt's too, and the registry answers one model per element-type pair.

source
VisualGeometryToolkitCore.induced_models — Method
induced_models(S, T)  /  induced_models(cs) -> Tuple{Vararg{Type}}

The model types estimable from correspondences with element types (S, T) (or from the container cs directly): model_type.(minimal_solvers(S, T)).

At (3D, 3D) this returns (Extrinsics3, EssentialMat), which are MUTUALLY EXCLUSIVE — a bare StaticVector{3} is both a world point and a bearing, and no dispatch can tell them apart. That is reported rather than resolved here; minimal_solver disambiguates by model type. See minimal_solvers for why the registry is built this way.

source
VisualGeometryToolkitCore.minimal_solver — Method
minimal_solver(::Type{M}, ::Type{S}, ::Type{T}) -> MinimalSolver{<:M}

Registry lookup: the minimal solver producing models of type M from (S, T) correspondences. Errors if the registry has no such solver.

source
VisualGeometryToolkitCore.minimal_solvers — Method
minimal_solvers(::Type{S}, ::Type{T}) -> Tuple

The minimal solvers applicable to correspondences with source element type S and target element type T. The registry is extended by adding a method — registering a new solver family requires NO edits to the verb layer or any problem struct.

Registered pairs:

  • 2D ↔ 2D points: Homography4pt, Fundamental7pt. Both are frame-covariant — a homography relates two planes, a fundamental matrix two images, in whatever 2D frame the caller holds — so here the element types really do determine that the solver applies.
  • 3D ↔ 3D: P3P and Essential5pt.

The 3D pair is ambiguous, and the registry says so rather than choosing

P3P and Essential5pt are MUTUALLY EXCLUSIVE and cannot be told apart by element type: P3P's source is a world POINT and its target a unit BEARING, while both of Essential5pt's endpoints are bearings. A bare StaticVector{3} is all three of those things, and no dispatch can separate them. What actually distinguishes them is the frame the caller built the data in, which lives in the frame tags — not in the element type.

So the two queries answer differently, on purpose:

  • minimal_solver(M, S, T) disambiguates by MODEL type and is exact: ask for EssentialMat and you get Essential5pt, ask for Extrinsics3 and you get P3P. This is the lookup consumers use.
  • induced_models(S, T) returns BOTH, because "what is estimable from a pair of 3-vectors" honestly has two answers and the types do not say which. It is not a claim that both apply to your data at once.

Why Essential5pt is not at (2D, 2D) any more

Until 0.29 it was, carrying the note that it "presumes calibrated coordinates — the element types cannot express that; it is the caller's frame convention". That was a claim about the CALLER'S DATA written down as a property of the code, and it was false in the way that matters: the lift it relied on sent a calibrated 2-vector to the z = 1 plane, which cannot represent a ray at or past 90°, silently excluding every fisheye and omnidirectional camera. Registering it there also made induced_models OFFER an essential matrix for plane data. The lift is deleted, the registration moved, and a 2-vector endpoint is now a MethodError at the solver.

The ambiguity that remains at (3D, 3D) is of a different kind: under-determined by the element type, but correct whenever the caller's data matches the solver's stated contract. The old (2D, 2D) entry was not under-determined — it was wrong.

source
VisualGeometryToolkitCore.solve — Function
solve(solver::MinimalSolver, cs, idx) -> FixedModels

Run a minimal solver on the sample idx drawn from the correspondence container cs (any container satisfying the sources/ targets protocol). ALWAYS returns FixedModels; a degenerate sample yields count == 0. This is the single public solve protocol — the positional kernels underneath are implementation detail.

source

Index