Correspondence API
Correspondence types and the solver layer built on them. Terminology: a correspondence (Correspondence) is a pair; a correspondence set (an n-tuple of mutually corresponding elements, e.g. multiview tracks) lives in VisualGeometryToolkitFeatures as CorrespondenceSet; the columnar collection of pair-correspondences is Correspondences.
Descriptor matching (NNMatch, match, strategies) lives in VisualGeometryToolkitFeatures.
Correspondences and Endpoint Lifts
VisualGeometryToolkitCore.Correspondences — Type
Correspondences(sources, targets; attrs...)Columnar correspondence set: parallel sources/targets vectors plus any number of index-aligned attribute columns (score, ratio, weights, covariance, …) that filter and view IN LOCKSTEP with the endpoints.
cs[i] returns the i-th correspondence as a Correspondence (= Pair); cs[mask] / view(cs, idx) subset every column together — no more hand-written scored[keep]; weights[keep] co-filtering.
Attribute columns are reached by property: cs.score, cs.ratio; presence is queried with hasattr. Endpoint columns are reached through the protocol verbs sources / targets (NOT .first/.second — that duck-typing is retired).
Example
cs = Correspondences(src_pts, dst_pts; score = scores, ratio = ratios)
sources(cs) # the source column
cs.score # the score column
inliers = cs[mask] # endpoints AND attributes filtered togetherVisualGeometryToolkitCore.MappedColumn — Type
MappedColumn(f, parent::AbstractVector)Lazy map(f, parent) view: m[i] == f(parent[i]), no allocation, no materialization. Element type is inferred via Base.promote_op.
Used by lift so that endpoint conversions (centroid extraction, z=0 planar lifts, bearing lifts) cost nothing until a solver actually indexes the column.
VisualGeometryToolkitCore.PointSet — Type
PointSet(points; attrs...)A set of single-point observations (a contour's points for a conic, a circle, a line) with index-aligned attribute columns, the one-observation counterpart of Correspondences: xs[i] is the point, xs.Σ (the points' covariances), xs.sqrt_weights and any other column are reached by property, and xs[mask] / view(xs, idx) subset every column together. What the batched Sampson functions (sampson_residuals!) read per element; a plain vector of points is a PointSet with no columns, and withattrs(points; attrs...) makes one.
VisualGeometryToolkitCore.block_order — Method
block_order(groups::AbstractDict, given::AbstractDict...) -> Vector{K}The block order of grouped data: the sorted keys of groups (the order a grouped estimator stacks its per-group unknowns and stores on its result). Every given dict — the per-group models a caller supplies, keyed by the same K — must have exactly the same key set: an ArgumentError otherwise.
VisualGeometryToolkitCore.hasattr — Method
hasattr(cs::Correspondences, k::Symbol) -> BoolWhether the correspondence set carries the attribute column k (e.g. hasattr(cs, :score) before reading cs.score).
VisualGeometryToolkitCore.lift — Method
lift(cs; source = identity, target = identity) -> CorrespondencesLazy endpoint conversion: returns a Correspondences whose source/target columns apply source/target element-wise on access (no materialization); attribute columns are forwarded untouched. Works on any container that satisfies the sources/targets protocol.
This replaces eager extraction helpers (point_correspondences) and hand-written z=0 lifts: compose whatever endpoint conversion the solver needs at the call site, pay only when the solver indexes.
VisualGeometryToolkitCore.planar_lift — Method
planar_lift(cs) -> Correspondencespoints with the source side additionally lifted onto the z=0 plane: 2D source centroids become Point3(x, y, 0) (the planar-board world convention), targets become 2D centroids. The correspondence set is then shaped for absolute-pose estimation.
VisualGeometryToolkitCore.points — Method
points(cs) -> Correspondenceslift both endpoints through centroid: detections, frames, discs, Attributed/Uncertain wrappers — anything with a centroid — become bare points, lazily.
VisualGeometryToolkitCore.sources — Method
sources(cs) -> AbstractVectorThe source column of a correspondence container. Protocol verb — defined for Correspondences, the legacy StructArray{<:Pair} columnar layout, Attributed-wrapped (scored) correspondence vectors, and plain AbstractVector{<:Pair} (lazy view). Solver code MUST go through sources/targets rather than .first/.second.
VisualGeometryToolkitCore.split_blocks — Method
split_blocks(v::AbstractVector, ids::AbstractVector{K}, counts) -> Dict{K, Vector}The inverse of stacking per-group vectors in block order: v's consecutive blocks of counts[j] entries, keyed by ids[j] (a copy per group). A DimensionMismatch unless ids and counts have one entry per group and the counts sum to length(v).
VisualGeometryToolkitCore.targets — Method
targets(cs) -> AbstractVectorThe target column of a correspondence container. See sources.
VisualGeometryToolkitCore.withattrs — Method
withattrs(cs::Correspondences; attrs...) -> CorrespondencesA copy of cs with the given attribute columns merged in (same-name columns replaced). Endpoint columns are shared, not copied. The idiom for attaching derived per-correspondence data — e.g. IRLS weights after a fit — so it subsets in lockstep from then on.
Solver Layer: Frames, Verbs, Registry
VisualGeometryToolkitCore.Essential5pt — Type
Essential5pt(backend = NativeSolver())Minimal 5-point essential-matrix solver (Nistér; solve verb over the essential_5pt kernel). sample_size 5, max_solutions 10.
Correspondence endpoints are bearing vectors (StaticVector{3}), and only those. A calibrated 2D point is a MethodError, not an input.
Why a 2-vector is refused rather than lifted
Until 0.29 a calibrated 2-vector was accepted and lifted to (x, y, 1). That lift places the ray on the z = 1 plane, which cannot represent a ray at or past 90° from the optical axis at all — so it silently excluded every fisheye and omnidirectional camera, which are the cameras this stack exists for. The failure was invisible: the lift always succeeded and returned a plausible EssentialMat, computed from a chart that could not hold the data.
Refusing by TYPE is the point. A caller holding pixels builds bearings with backproject — or backproject_with_jacobian when the pixel Jacobian is wanted too, which is what keeps a residual threshold in pixels.
The kernel computes in Float64; returned models are EssentialMat{Float64}. The epipolar constraint is scale-invariant per endpoint, so the algebra does not require unit norm — but unit bearings are the stack's convention and what backproject returns.
What the type refusal does NOT catch
It catches a 2-vector. It cannot catch a 3-vector that is still on the z = 1 plane — (x, y, 1) is a perfectly good StaticVector{3}, and because the epipolar algebra is scale-invariant per endpoint, such a point even produces the right answer for any ray the chart can represent. The exclusion of rays at and past 90° survives that lift unchanged; it has simply stopped being visible in the type.
So this signature is a NECESSARY condition, not a guarantee, and it should not be read as one. The guarantee has to come from where the vectors are BUILT — backproject(model, u), which is defined for every ray the camera can see — not from where they are consumed. A caller that reaches a 3-vector by dividing through by z has moved the same defect one layer up and past this gate.
VisualGeometryToolkitCore.Fundamental7pt — Type
Fundamental7pt(backend = NativeSolver())Minimal 7-point fundamental-matrix solver (solve verb over the fundamental_7pt kernel). sample_size 7, max_solutions 3.
VisualGeometryToolkitCore.Homography4pt — Type
Homography4pt(backend = NativeSolver())Minimal 4-point homography solver (solve verb over the homography_4pt kernel). sample_size 4, max_solutions 1.
VisualGeometryToolkitCore.Homography5ptRadial — Type
Homography5ptRadial(; backend = NativeSolver(), extra_check = false)Minimal 5-point radially distorted homography solver — one-sided one-parameter division model on the TARGET side (solve verb over the homography_5pt_radial kernel; Wadenbäck et al. (2026)). sample_size 5, max_solutions 3 (at most 1 with extra_check, which keeps only the root most consistent with the redundant equation — the reference enables it inside its RANSAC wrapper).
Deliberately NOT entered in the minimal_solvers registry: the division coefficient k is only meaningful in roughly unit-scale target coordinates, a convention generic pixel correspondences don't carry (cf. the calibrated convention Essential5pt presumes). Opt in explicitly.
VisualGeometryToolkitCore.MinimalSolver — Type
MinimalSolver{M}Abstract parent for exactly-determined minimal solvers producing models of type M. Each algorithm is its own singleton subtype (two solvers may share a model type but require different correspondence types — p3p vs. circle relative pose both yield poses).
Interface (all required):
solve(s, cs, idx) -> FixedModels{max_solutions(s), M}— ALWAYS aFixedModels; failure iscount == 0, nevernothingsample_size(s)— minimal sample cardinalitymax_solutions(s)— solution-count bound (theNinFixedModels)model_type(s) == M
Solvers with pluggable numeric backends carry a backend::AbstractSolverBackend field (default NativeSolver()).
VisualGeometryToolkitCore.P3P — Type
P3P(backend = NativeSolver())Perspective-3-point absolute-pose solver (solve verb over the p3p kernel). sample_size 3, max_solutions 4.
Correspondence contract: sources are 3D world points, targets are unit bearing vectors (pixel targets are lifted to bearings beforehand, e.g. lift(cs; target = u -> backproject(model, u))). Models are Extrinsics3 with x_cam = R·X + t.
VisualGeometryToolkitCore.Rotation2pt — Type
Rotation2pt()Minimal 2-point rotation solver: the rotation R with vᵢ ≈ R·uᵢ from two pairs of bearing vectors (StaticVector{3}, any nonzero length — a direction is its unit vector) — the infinite homography of a calibrated camera under a pure rotation, or any two direction pairs. kabsch on the two unit pairs (solve verb); sample_size 2, max_solutions 1.
The rotation is exact on two consistent pairs (the two bearings of a view span a plane, and their cross product completes the frame). A sample whose two bearings are parallel or antiparallel in either view (sine of their angle below 1e-12) leaves a one-parameter family of rotations and returns count == 0. Two pairs whose angles differ (inconsistent with any rotation) still give the least-squares rotation; the residuals judge it.
Deliberately NOT entered in the minimal_solvers registry: its correspondence type, bearing ↔ bearing, is Essential5pt's too, and the registry answers one model per element-type pair.
VisualGeometryToolkitCore.induced_models — Method
induced_models(S, T) / induced_models(cs) -> Tuple{Vararg{Type}}The model types estimable from correspondences with element types (S, T) (or from the container cs directly): model_type.(minimal_solvers(S, T)).
At (3D, 3D) this returns (Extrinsics3, EssentialMat), which are MUTUALLY EXCLUSIVE — a bare StaticVector{3} is both a world point and a bearing, and no dispatch can tell them apart. That is reported rather than resolved here; minimal_solver disambiguates by model type. See minimal_solvers for why the registry is built this way.
VisualGeometryToolkitCore.max_solutions — Function
max_solutions(s::MinimalSolver) -> IntUpper bound on the number of candidate models one minimal sample can produce (the N of the returned FixedModels{N,M}).
VisualGeometryToolkitCore.minimal_solver — Method
minimal_solver(::Type{M}, ::Type{S}, ::Type{T}) -> MinimalSolver{<:M}Registry lookup: the minimal solver producing models of type M from (S, T) correspondences. Errors if the registry has no such solver.
VisualGeometryToolkitCore.minimal_solvers — Method
minimal_solvers(::Type{S}, ::Type{T}) -> TupleThe minimal solvers applicable to correspondences with source element type S and target element type T. The registry is extended by adding a method — registering a new solver family requires NO edits to the verb layer or any problem struct.
Registered pairs:
- 2D ↔ 2D points:
Homography4pt,Fundamental7pt. Both are frame-covariant — a homography relates two planes, a fundamental matrix two images, in whatever 2D frame the caller holds — so here the element types really do determine that the solver applies. - 3D ↔ 3D:
P3PandEssential5pt.
The 3D pair is ambiguous, and the registry says so rather than choosing
P3P and Essential5pt are MUTUALLY EXCLUSIVE and cannot be told apart by element type: P3P's source is a world POINT and its target a unit BEARING, while both of Essential5pt's endpoints are bearings. A bare StaticVector{3} is all three of those things, and no dispatch can separate them. What actually distinguishes them is the frame the caller built the data in, which lives in the frame tags — not in the element type.
So the two queries answer differently, on purpose:
minimal_solver(M, S, T)disambiguates by MODEL type and is exact: ask forEssentialMatand you getEssential5pt, ask forExtrinsics3and you getP3P. This is the lookup consumers use.induced_models(S, T)returns BOTH, because "what is estimable from a pair of 3-vectors" honestly has two answers and the types do not say which. It is not a claim that both apply to your data at once.
Why Essential5pt is not at (2D, 2D) any more
Until 0.29 it was, carrying the note that it "presumes calibrated coordinates — the element types cannot express that; it is the caller's frame convention". That was a claim about the CALLER'S DATA written down as a property of the code, and it was false in the way that matters: the lift it relied on sent a calibrated 2-vector to the z = 1 plane, which cannot represent a ray at or past 90°, silently excluding every fisheye and omnidirectional camera. Registering it there also made induced_models OFFER an essential matrix for plane data. The lift is deleted, the registration moved, and a 2-vector endpoint is now a MethodError at the solver.
The ambiguity that remains at (3D, 3D) is of a different kind: under-determined by the element type, but correct whenever the caller's data matches the solver's stated contract. The old (2D, 2D) entry was not under-determined — it was wrong.
VisualGeometryToolkitCore.model_type — Method
model_type(s) -> TypeThe model type produced by a MinimalSolver.
VisualGeometryToolkitCore.sample_size — Function
sample_size(s::MinimalSolver) -> IntNumber of correspondences in the solver's minimal sample.
VisualGeometryToolkitCore.solve — Function
solve(solver::MinimalSolver, cs, idx) -> FixedModelsRun a minimal solver on the sample idx drawn from the correspondence container cs (any container satisfying the sources/ targets protocol). ALWAYS returns FixedModels; a degenerate sample yields count == 0. This is the single public solve protocol — the positional kernels underneath are implementation detail.
Index
VisualGeometryToolkitCore.CorrespondencesVisualGeometryToolkitCore.Essential5ptVisualGeometryToolkitCore.Fundamental7ptVisualGeometryToolkitCore.Homography4ptVisualGeometryToolkitCore.Homography5ptRadialVisualGeometryToolkitCore.MappedColumnVisualGeometryToolkitCore.MinimalSolverVisualGeometryToolkitCore.P3PVisualGeometryToolkitCore.PointSetVisualGeometryToolkitCore.Rotation2ptVisualGeometryToolkitCore.block_orderVisualGeometryToolkitCore.hasattrVisualGeometryToolkitCore.induced_modelsVisualGeometryToolkitCore.liftVisualGeometryToolkitCore.max_solutionsVisualGeometryToolkitCore.minimal_solverVisualGeometryToolkitCore.minimal_solversVisualGeometryToolkitCore.model_typeVisualGeometryToolkitCore.planar_liftVisualGeometryToolkitCore.pointsVisualGeometryToolkitCore.sample_sizeVisualGeometryToolkitCore.solveVisualGeometryToolkitCore.sourcesVisualGeometryToolkitCore.split_blocksVisualGeometryToolkitCore.targetsVisualGeometryToolkitCore.withattrs