Essential Matrix
RANSAC Problem
EssentialProblem is a type alias for MinimalSolveProblem{EssentialMat{T}, 5, T, ...} — a calibrated two-view problem with the 5-point minimal solver, on UNIT BEARINGS (3-vectors), not normalized 2-vectors. Build it from pixels and the two cameras, EssentialProblem(camera1, camera2, pixel_cs), which back-projects each view with its own camera and attaches the back-projection Jacobians M₁/M₂ that keep residuals and the threshold in PIXELS.
VisualGeometryToolkitEstimation.EssentialProblem — Type
EssentialProblem(camera1, camera2, pixel_cs; kw...)Build an EssentialProblem from PIXEL correspondences and the two cameras that produced them — the entry most callers want, and the only one that cannot get the frames wrong.
Each view is back-projected with its OWN camera through backproject_with_jacobian, so M₁ belongs with the source bearings and M₂ with the targets by construction rather than by the caller remembering. That matters more than it looks: swapping the two is norm-preserving and invisible to Monte-Carlo calibration, and shows only in the tangency contract.
Two cameras, not one, because they are genuinely independent — and because a single-camera convenience form would make the swap undetectable in exactly the tests that ought to catch it.
EssentialProblem(bearings; sampler=nothing, backend=NativeSolver())Construct an EssentialProblem from unit-bearing correspondences: 3-vector rays on S², carrying their back-projection Jacobians as the attribute columns M₁/M₂ (withattrs(cs; M₁, M₂), from backproject_with_jacobian).
Differs from FundMatProblem in coordinate space — bearings, not pixels — and in minimal sample size (5 vs 7).
Why 3-vectors, and why the length is enforced
The old contract took calibrated 2-vectors and implied z = 1. That is the gnomonic chart, and it does not exist for a ray at or past the horizon — which is exactly the rays a fisheye or omnidirectional camera produces. Bearings have no such gap.
The length is a type constraint, not a convention, because the failure it prevents is silent. A consumer that indexes components 1 and 2 of a bearing and appends 1.0 gets a nearly right answer whenever z ≈ 1, so it degrades with field of view rather than failing — the same signature as the defect this contract removes. A wrong-length input is therefore a MethodError here, and there is deliberately no method accepting either length.
Why M₁/M₂ are required rather than defaulted
M = ∂f/∂u is what carries the gate's and the scoring's tolerance into PIXELS: test_sample converts an angular separation as angle / σ_max(M). M picks the unit and the call cannot — an orthonormal tangent basis would put the same threshold in radians. With no columns there is no defensible unit, so their absence throws rather than falling back; a fallback is the path by which a pixel tolerance silently becomes an angular one.
Both are per-view: M₁ belongs with the source bearings and M₂ with the targets. Swapping them is norm-preserving and invisible to Monte-Carlo calibration — measured median ratio 1.018, best cell inside a 4 SE band — and shows only in the tangency contract, fᵀM = 0, which separates a correct wiring (~1e-19) from a swapped one (~1e-3) by sixteen orders of magnitude.
E takes no observation maps: a non-identity src_map/dst_map is an ArgumentError. Build the bearings with the cameras instead, EssentialProblem(camera1, camera2, pixel_cs).
EssentialProblem(pair::PoolPair{<:FundMat}; kw...) -> EssentialProblemThe pool-frame essential problem of a pool record: its correspondences (NormalizedImageFrame) back-projected to unit bearings by the pool-frame cameras CameraModel(Kcᵢ), Kcᵢ = Tᵢ·Kᵢ ([pool_frame_calibrations]). Residuals are the tangent Sampson whitened through Kc, so they are pool-frame quantities: the unit of the pool's stored distances, σ̂², outlier half-widths and loss widths. The sample gate's coincidence tolerance is 0.1 px in pool units (COINCIDENT_PX · pool_pixel_scale(pair)), always. kw go to EssentialProblem (e.g. a sampler).
A pool's Kc is a matrix, so this problem is pinhole by construction.
Fitting Pipeline
VisualGeometryToolkitEstimation.find_essential — Function
find_essential(camera1, camera2, pixel_cs; scoring,
max_trials, min_trials,
local_optimization_strategy, config=nothing)Robust essential-matrix estimation via 5-point RANSAC with MSAC scoring. Every default is configs/default.toml's ([ransac], [pipelines.essential]); a keyword overrides.
Takes PIXEL correspondences and the two cameras that produced them. Each view is back-projected with its OWN camera, so the per-view Jacobians land with the right endpoints by construction rather than by the caller remembering — a swap is norm-preserving and invisible to Monte-Carlo calibration, showing only in the tangency contract fᵀM = 0.
Two cameras rather than one because they are genuinely independent, and because a single-camera convenience form would make that swap undetectable in exactly the tests that ought to catch it.
DO NOT ADD a single-camera overload as an ergonomics improvement. Collapsing two distinct inputs into one erases the asymmetry a swap needs in order to be visible — the same reason an M-wiring harness needs two different focal lengths. A stereo pair with one camera repeated is written find_essential(cam, cam, cs), which costs three characters and keeps the asymmetry available.
Returns a RansacEstimate, the :no_model result (iterations 0) for fewer than 5 correspondences. iterations(result) is the number of trials (hypotheses drawn); test found_model(result) before reading the model (see RansacEstimate).
Arguments
camera1,camera2: the cameras of the source and target views (any camerabackproject_with_jacobianaccepts).pixel_cs: PIXEL correspondences,u₁ => u₂in the two cameras' images (Correspondences,StructArray{<:Pair}or a Pair vector); aCorrespondences:scorecolumn is the per-match prior (PROSAC; uniform otherwise) and travels to the bearing problem with every other attribute column.scoring: the scorer, its width in PIXELS (default[pipelines.essential] scoring, MSAC at 1 px).Mpicks the unit and the call cannot: the residual is whitened by the back-projection Jacobian, so the width it is compared against is in pixels; an orthonormal tangent basis would put the same number in radians. THE UNIT CHANGED — this was in calibrated coordinates until the bearing port, and no pixel value reproduces the old0.02across cameras because0.02·fis f-dependent (≈16 px at f = 800, ≈24 px at f = 1200). Values below VisualGeometryToolkitFeatures' 0.06 px refinement bound — a threshold scorer's threshold or the Student-t scorer's outlier half-width — are REFUSED (ArgumentError) rather than silently honoured, since they are almost always a calibrated leftover — and refused by every entry that scores anEssentialProblem: ascoringsupplied here, the class default, orransac/locally_optimizecalled directly.local_optimization_strategy: LO-RANSAC strategy (default[pipelines.essential] local_optimization_strategy, asfind_homography).- the other keywords as
find_homography.
Examples
result = find_essential(cam1, cam2, pixel_pairs)
result = find_essential(cam1, cam2, Correspondences(sources(pixel_pairs), targets(pixel_pairs); score = similarities))VisualGeometryToolkitEstimation.refine_essential — Function
refine_essential(E, camera1, camera2, pixel_cs; optimization_strategy, scale | width, workspace = nothing)
-> FitResult{M, RefineMetadata}The essential matrix E refined against the PIXEL correspondences pixel_cs of the cameras camera1, camera2, from E itself, on find_essential's bearing problem (each view back-projected with its own camera; the residual whitened into pixels, codimension 1): the keywords, the result and the refusals are those of every refine_*, documented on refine_homography (default strategy: [refine.essential] optimization_strategy). Fewer than 5 correspondences: the guess back unrefined, record (ReturnCode.InitialFailure, NoRule) (retcode_symbol:degenerate).
Thresholds and operating points
find_essential's default threshold, 1 px ([pipelines.essential] in configs/default.toml), is a convention (OpenCV's findEssentialMat), not a tuned value. The paper's cross-validated operating points — the per-class efficiency constants of the Welsch half-quadratic lift, with their provenance — are LocalOptimizationDoneRight's tuning constants (tuning_constant, WELSCH_HQ_EFFICIENCY_CONSTANT_ESSENTIAL, …) and are supplied with its strategy: find_essential(cam1, cam2, cs; scoring = …, local_optimization_strategy = HQ{WelschShape}(efficiency_constant = WELSCH_HQ_EFFICIENCY_CONSTANT_ESSENTIAL)).
The Pure Rotation
A camera that only rotates between two views induces the infinite homography, a rotation R with f₂ ∥ R·f₁ on unit bearings. RotationProblem is the bearing problem of that model, built like the essential problem from the cameras' back-projection with the M₁/M₂ Jacobian columns (the whole field of a fisheye takes part, past 90°): VisualGeometryToolkitCore's Rotation2pt on two pairs, the residual the Sampson distance of the joint observation (both images' pixels, at their noise blockdiag(Σ₁, Σ₂): unit isotropic for plain pixels, the detections' own for Uncertain ones) to the transfer constraint on bearings, in whitened pixels (VisualGeometryToolkitCore's constraint_with_jacobians(TransferForm(H), f₁, f₂; M₁, M₂) at H = R), and the chart-LM refit on VisualGeometryToolkitCore's SO3Manifold through the homography's transfer rows. find_rotation is its RANSAC; a scorer or a lift from another package plugs in as for find_essential. uncertain_rotation turns a pairwise result into a fit_rotation_averaging edge: the rotation with its absolute covariance (noise_scale² × solver_covariance in the so(3) chart, radians²).
VisualGeometryToolkitEstimation.RotationProblem — Type
RotationProblemThe RANSAC problem of a pure rotation between two calibrated views: the MinimalSolveProblem whose model is a RotMatrix{3, Float64}R with f₂ ∥ R·f₁ for unit bearings f₁ => f₂ (the infinite homography). Minimal sample 2 (VisualGeometryToolkitCore's Rotation2pt), codimension 2, a 3-dof chart. Build it with RotationProblem(camera1, camera2, pixel_cs) or from a bearing Correspondences carrying M₁/M₂ columns; the public estimator is find_rotation.
VisualGeometryToolkitEstimation.find_rotation — Function
find_rotation(camera1, camera2, pixel_cs; scoring, max_trials, min_trials,
local_optimization_strategy, config = nothing, rng)
find_rotation(bearings::Correspondences; kwargs...)Robust estimation of the rotation R between two views of a camera that only rotated — f₂ ∥ R·f₁ for the unit bearings of a correspondence, the infinite homography — by RANSAC on the RotationProblem: 2-point samples (VisualGeometryToolkitCore's Rotation2pt, kabsch on the pair), the Sampson distance of the joint observation (both images' pixels, Σ = I) to the transfer constraint on bearings, in PIXELS through the views' back-projection Jacobians, and the local optimization's chart-LM refit on the rotation chart. A sampler (it draws hypotheses and verifies them), so a find_*.
Every default is configs/default.toml's ([ransac], [pipelines.rotation]); a keyword overrides. The keywords are find_homography's; a scorer or local optimization from another package (RANSACScoringDoneRight's StudentTUniformScoring(RotationProblem, a), LocalOptimizationDoneRight's lifts selected by tag) plugs in through scoring and local_optimization_strategy.
camera1,camera2,pixel_cs: PIXEL correspondencesu₁ => u₂and the cameras of the two views, each view back-projected with its own camera (a rotating camera isfind_rotation(cam, cam, cs)); the whole field of a fisheye takes part, past 90°. A:scorecolumn is the PROSAC prior.bearings: unit bearings with theirM₁/M₂Jacobian columns (seeRotationProblem).scoring: the scorer, its width in PIXELS (the residual's unit, asfind_essential's); default[pipelines.rotation] scoring. Widths below VisualGeometryToolkitFeatures' 0.06 px refinement bound are refused.
Returns a RansacEstimate (its estimate a RotMatrix{3, Float64}), or the :no_model result (iterations 0) for fewer than 2 correspondences; test found_model(result) before reading the model.
VisualGeometryToolkitEstimation.refine_rotation — Function
refine_rotation(R, camera1, camera2, pixel_cs; optimization_strategy, scale | width, workspace = nothing)
refine_rotation(R, bearings::Correspondences; kwargs...)
-> FitResult{M, RefineMetadata}The rotation R of a camera that only rotated, refined against its correspondences from R itself, on find_rotation's RotationProblem (its forms; codimension 2): the keywords, the result and the refusals are those of every refine_*, documented on refine_homography (default strategy: [refine.rotation] optimization_strategy). Fewer than 2 correspondences: the guess back unrefined, record (ReturnCode.InitialFailure, NoRule) (retcode_symbol:degenerate).
VisualGeometryToolkitEstimation.uncertain_rotation — Function
uncertain_rotation(problem::RotationProblem, result::RansacEstimate; scale = result) -> Uncertain{RotMatrix{3, Float64}}The relative rotation of a pairwise rotation fit (find_rotation, refine_rotation on problem) with its covariance, the edge fit_rotation_averaging reads: Uncertain(R, CovMat(Σ_R)) with R = estimate(result) and
Σ_R = noise_scale² · Jᵀ·solver_covariance(problem, R, inliers)·J / 4in the so(3) chart of R (left exp, R → exp(δ)·R; J the chart's retract_jacobian, whose columns are orthogonal of norm √2), radians². The covariance is the implicit-function covariance of the problem's own whitened-constraint estimator over the result's inliers (solver_covariance), so it carries every detection's covariance when the pixels are Uncertain (RotationProblem(camera1, camera2, pixel_cs)), and is in whitened units: scale is the variance factor of those units — the scale input of the robust-estimation contract: by default the result's own noise_scale, the scale its scorer estimated on the consensus set (ScaleEstimated, in whitened units: ≈ 1 when the detections' covariances are right); or scale = σ, or another upstream result. A result that recorded no scale (ScaleNone) is refused. An edge covariance is absolute, so the averaging takes no further scale.
Pairs that share a detection across edges make their edges' errors correlated; the averaging treats the edges as independent, the first-order approximation.
An ArgumentError for a result with no estimate, or a singular sandwich (too few inliers).