Essential Matrix

RANSAC Problem

EssentialProblem is a type alias for MinimalSolveProblem{EssentialMat{T}, 5, T, ...} — a calibrated two-view problem with the 5-point minimal solver, on UNIT BEARINGS (3-vectors), not normalized 2-vectors. Build it from pixels and the two cameras, EssentialProblem(camera1, camera2, pixel_cs), which back-projects each view with its own camera and attaches the back-projection Jacobians M₁/M₂ that keep residuals and the threshold in PIXELS.

VisualGeometryToolkitEstimation.EssentialProblem — Type
EssentialProblem(camera1, camera2, pixel_cs; kw...)

Build an EssentialProblem from PIXEL correspondences and the two cameras that produced them — the entry most callers want, and the only one that cannot get the frames wrong.

Each view is back-projected with its OWN camera through backproject_with_jacobian, so M₁ belongs with the source bearings and M₂ with the targets by construction rather than by the caller remembering. That matters more than it looks: swapping the two is norm-preserving and invisible to Monte-Carlo calibration, and shows only in the tangency contract.

Two cameras, not one, because they are genuinely independent — and because a single-camera convenience form would make the swap undetectable in exactly the tests that ought to catch it.

source
EssentialProblem(bearings; sampler=nothing, backend=NativeSolver())

Construct an EssentialProblem from unit-bearing correspondences: 3-vector rays on S², carrying their back-projection Jacobians as the attribute columns M₁/M₂ (withattrs(cs; M₁, M₂), from backproject_with_jacobian).

Differs from FundMatProblem in coordinate space — bearings, not pixels — and in minimal sample size (5 vs 7).

Why 3-vectors, and why the length is enforced

The old contract took calibrated 2-vectors and implied z = 1. That is the gnomonic chart, and it does not exist for a ray at or past the horizon — which is exactly the rays a fisheye or omnidirectional camera produces. Bearings have no such gap.

The length is a type constraint, not a convention, because the failure it prevents is silent. A consumer that indexes components 1 and 2 of a bearing and appends 1.0 gets a nearly right answer whenever z ≈ 1, so it degrades with field of view rather than failing — the same signature as the defect this contract removes. A wrong-length input is therefore a MethodError here, and there is deliberately no method accepting either length.

Why M₁/M₂ are required rather than defaulted

M = ∂f/∂u is what carries the gate's and the scoring's tolerance into PIXELS: test_sample converts an angular separation as angle / σ_max(M). M picks the unit and the call cannot — an orthonormal tangent basis would put the same threshold in radians. With no columns there is no defensible unit, so their absence throws rather than falling back; a fallback is the path by which a pixel tolerance silently becomes an angular one.

Both are per-view: M₁ belongs with the source bearings and M₂ with the targets. Swapping them is norm-preserving and invisible to Monte-Carlo calibration — measured median ratio 1.018, best cell inside a 4 SE band — and shows only in the tangency contract, fᵀM = 0, which separates a correct wiring (~1e-19) from a swapped one (~1e-3) by sixteen orders of magnitude.

E takes no observation maps: a non-identity src_map/dst_map is an ArgumentError. Build the bearings with the cameras instead, EssentialProblem(camera1, camera2, pixel_cs).

source
EssentialProblem(pair::PoolPair{<:FundMat}; kw...) -> EssentialProblem

The pool-frame essential problem of a pool record: its correspondences (NormalizedImageFrame) back-projected to unit bearings by the pool-frame cameras CameraModel(Kcᵢ), Kcᵢ = Tᵢ·Kᵢ ([pool_frame_calibrations]). Residuals are the tangent Sampson whitened through Kc, so they are pool-frame quantities: the unit of the pool's stored distances, σ̂², outlier half-widths and loss widths. The sample gate's coincidence tolerance is 0.1 px in pool units (COINCIDENT_PX · pool_pixel_scale(pair)), always. kw go to EssentialProblem (e.g. a sampler).

A pool's Kc is a matrix, so this problem is pinhole by construction.

source

Fitting Pipeline

VisualGeometryToolkitEstimation.find_essential — Function
find_essential(camera1, camera2, pixel_cs; scoring,
    max_trials, min_trials,
    local_optimization_strategy, config=nothing)

Robust essential-matrix estimation via 5-point RANSAC with MSAC scoring. Every default is configs/default.toml's ([ransac], [pipelines.essential]); a keyword overrides.

Takes PIXEL correspondences and the two cameras that produced them. Each view is back-projected with its OWN camera, so the per-view Jacobians land with the right endpoints by construction rather than by the caller remembering — a swap is norm-preserving and invisible to Monte-Carlo calibration, showing only in the tangency contract fᵀM = 0.

Two cameras rather than one because they are genuinely independent, and because a single-camera convenience form would make that swap undetectable in exactly the tests that ought to catch it.

DO NOT ADD a single-camera overload as an ergonomics improvement. Collapsing two distinct inputs into one erases the asymmetry a swap needs in order to be visible — the same reason an M-wiring harness needs two different focal lengths. A stereo pair with one camera repeated is written find_essential(cam, cam, cs), which costs three characters and keeps the asymmetry available.

Returns a RansacEstimate, the :no_model result (iterations 0) for fewer than 5 correspondences. iterations(result) is the number of trials (hypotheses drawn); test found_model(result) before reading the model (see RansacEstimate).

Arguments

  • camera1, camera2: the cameras of the source and target views (any camera backproject_with_jacobian accepts).
  • pixel_cs: PIXEL correspondences, u₁ => u₂ in the two cameras' images (Correspondences, StructArray{<:Pair} or a Pair vector); a Correspondences:score column is the per-match prior (PROSAC; uniform otherwise) and travels to the bearing problem with every other attribute column.
  • scoring: the scorer, its width in PIXELS (default [pipelines.essential] scoring, MSAC at 1 px). M picks the unit and the call cannot: the residual is whitened by the back-projection Jacobian, so the width it is compared against is in pixels; an orthonormal tangent basis would put the same number in radians. THE UNIT CHANGED — this was in calibrated coordinates until the bearing port, and no pixel value reproduces the old 0.02 across cameras because 0.02·f is f-dependent (≈16 px at f = 800, ≈24 px at f = 1200). Values below VisualGeometryToolkitFeatures' 0.06 px refinement bound — a threshold scorer's threshold or the Student-t scorer's outlier half-width — are REFUSED (ArgumentError) rather than silently honoured, since they are almost always a calibrated leftover — and refused by every entry that scores an EssentialProblem: a scoring supplied here, the class default, or ransac / locally_optimize called directly.
  • local_optimization_strategy: LO-RANSAC strategy (default [pipelines.essential] local_optimization_strategy, as find_homography).
  • the other keywords as find_homography.

Examples

result = find_essential(cam1, cam2, pixel_pairs)
result = find_essential(cam1, cam2, Correspondences(sources(pixel_pairs), targets(pixel_pairs); score = similarities))
source
VisualGeometryToolkitEstimation.refine_essential — Function
refine_essential(E, camera1, camera2, pixel_cs; optimization_strategy, scale | width, workspace = nothing)
    -> FitResult{M, RefineMetadata}

The essential matrix E refined against the PIXEL correspondences pixel_cs of the cameras camera1, camera2, from E itself, on find_essential's bearing problem (each view back-projected with its own camera; the residual whitened into pixels, codimension 1): the keywords, the result and the refusals are those of every refine_*, documented on refine_homography (default strategy: [refine.essential] optimization_strategy). Fewer than 5 correspondences: the guess back unrefined, record (ReturnCode.InitialFailure, NoRule) (retcode_symbol:degenerate).

source

Thresholds and operating points

find_essential's default threshold, 1 px ([pipelines.essential] in configs/default.toml), is a convention (OpenCV's findEssentialMat), not a tuned value. The paper's cross-validated operating points — the per-class efficiency constants of the Welsch half-quadratic lift, with their provenance — are LocalOptimizationDoneRight's tuning constants (tuning_constant, WELSCH_HQ_EFFICIENCY_CONSTANT_ESSENTIAL, …) and are supplied with its strategy: find_essential(cam1, cam2, cs; scoring = …, local_optimization_strategy = HQ{WelschShape}(efficiency_constant = WELSCH_HQ_EFFICIENCY_CONSTANT_ESSENTIAL)).

The Pure Rotation

A camera that only rotates between two views induces the infinite homography, a rotation R with f₂ ∥ R·f₁ on unit bearings. RotationProblem is the bearing problem of that model, built like the essential problem from the cameras' back-projection with the M₁/M₂ Jacobian columns (the whole field of a fisheye takes part, past 90°): VisualGeometryToolkitCore's Rotation2pt on two pairs, the residual the Sampson distance of the joint observation (both images' pixels, at their noise blockdiag(Σ₁, Σ₂): unit isotropic for plain pixels, the detections' own for Uncertain ones) to the transfer constraint on bearings, in whitened pixels (VisualGeometryToolkitCore's constraint_with_jacobians(TransferForm(H), f₁, f₂; M₁, M₂) at H = R), and the chart-LM refit on VisualGeometryToolkitCore's SO3Manifold through the homography's transfer rows. find_rotation is its RANSAC; a scorer or a lift from another package plugs in as for find_essential. uncertain_rotation turns a pairwise result into a fit_rotation_averaging edge: the rotation with its absolute covariance (noise_scale² × solver_covariance in the so(3) chart, radians²).

VisualGeometryToolkitEstimation.RotationProblem — Type
RotationProblem

The RANSAC problem of a pure rotation between two calibrated views: the MinimalSolveProblem whose model is a RotMatrix{3, Float64}R with f₂ ∥ R·f₁ for unit bearings f₁ => f₂ (the infinite homography). Minimal sample 2 (VisualGeometryToolkitCore's Rotation2pt), codimension 2, a 3-dof chart. Build it with RotationProblem(camera1, camera2, pixel_cs) or from a bearing Correspondences carrying M₁/M₂ columns; the public estimator is find_rotation.

source
VisualGeometryToolkitEstimation.find_rotation — Function
find_rotation(camera1, camera2, pixel_cs; scoring, max_trials, min_trials,
    local_optimization_strategy, config = nothing, rng)
find_rotation(bearings::Correspondences; kwargs...)

Robust estimation of the rotation R between two views of a camera that only rotated — f₂ ∥ R·f₁ for the unit bearings of a correspondence, the infinite homography — by RANSAC on the RotationProblem: 2-point samples (VisualGeometryToolkitCore's Rotation2pt, kabsch on the pair), the Sampson distance of the joint observation (both images' pixels, Σ = I) to the transfer constraint on bearings, in PIXELS through the views' back-projection Jacobians, and the local optimization's chart-LM refit on the rotation chart. A sampler (it draws hypotheses and verifies them), so a find_*.

Every default is configs/default.toml's ([ransac], [pipelines.rotation]); a keyword overrides. The keywords are find_homography's; a scorer or local optimization from another package (RANSACScoringDoneRight's StudentTUniformScoring(RotationProblem, a), LocalOptimizationDoneRight's lifts selected by tag) plugs in through scoring and local_optimization_strategy.

  • camera1, camera2, pixel_cs: PIXEL correspondences u₁ => u₂ and the cameras of the two views, each view back-projected with its own camera (a rotating camera is find_rotation(cam, cam, cs)); the whole field of a fisheye takes part, past 90°. A :score column is the PROSAC prior.
  • bearings: unit bearings with their M₁/M₂ Jacobian columns (see RotationProblem).
  • scoring: the scorer, its width in PIXELS (the residual's unit, as find_essential's); default [pipelines.rotation] scoring. Widths below VisualGeometryToolkitFeatures' 0.06 px refinement bound are refused.

Returns a RansacEstimate (its estimate a RotMatrix{3, Float64}), or the :no_model result (iterations 0) for fewer than 2 correspondences; test found_model(result) before reading the model.

source
VisualGeometryToolkitEstimation.refine_rotation — Function
refine_rotation(R, camera1, camera2, pixel_cs; optimization_strategy, scale | width, workspace = nothing)
refine_rotation(R, bearings::Correspondences; kwargs...)
    -> FitResult{M, RefineMetadata}

The rotation R of a camera that only rotated, refined against its correspondences from R itself, on find_rotation's RotationProblem (its forms; codimension 2): the keywords, the result and the refusals are those of every refine_*, documented on refine_homography (default strategy: [refine.rotation] optimization_strategy). Fewer than 2 correspondences: the guess back unrefined, record (ReturnCode.InitialFailure, NoRule) (retcode_symbol:degenerate).

source
VisualGeometryToolkitEstimation.uncertain_rotation — Function
uncertain_rotation(problem::RotationProblem, result::RansacEstimate; scale = result) -> Uncertain{RotMatrix{3, Float64}}

The relative rotation of a pairwise rotation fit (find_rotation, refine_rotation on problem) with its covariance, the edge fit_rotation_averaging reads: Uncertain(R, CovMat(Σ_R)) with R = estimate(result) and

Σ_R = noise_scale² · Jᵀ·solver_covariance(problem, R, inliers)·J / 4

in the so(3) chart of R (left exp, R → exp(δ)·R; J the chart's retract_jacobian, whose columns are orthogonal of norm √2), radians². The covariance is the implicit-function covariance of the problem's own whitened-constraint estimator over the result's inliers (solver_covariance), so it carries every detection's covariance when the pixels are Uncertain (RotationProblem(camera1, camera2, pixel_cs)), and is in whitened units: scale is the variance factor of those units — the scale input of the robust-estimation contract: by default the result's own noise_scale, the scale its scorer estimated on the consensus set (ScaleEstimated, in whitened units: ≈ 1 when the detections' covariances are right); or scale = σ, or another upstream result. A result that recorded no scale (ScaleNone) is refused. An edge covariance is absolute, so the averaging takes no further scale.

Pairs that share a detection across edges make their edges' errors correlated; the averaging treats the edges as independent, the first-order approximation.

An ArgumentError for a result with no estimate, or a singular sandwich (too few inliers).

source