Pose and Rotation from Directions
Estimators whose data are DIRECTIONS — unit bearings from a camera's back-projection, star or gravity directions — rather than pixels. They work on the sphere throughout (VisualGeometryToolkitCore conventions/bearings.md), so rays at and past 90° from the optical axis, as a fisheye or omnidirectional camera sees them, are ordinary data. (A camera pose from a guess and pixels is refine_absolute_pose(pose, camera, pairs), on the PnP problem's pixel rows — P3P and Absolute Pose.)
Camera pose from bearings
refine_absolute_pose(pose, cs) with cs world points paired with UNIT bearings is the pose of a camera the caller back-projects itself — a grid or non-central lens the stack holds no CameraModel of — on the tangent-plane residual of the sphere, optionally from per-ray origins (a generalized camera) and whitened into pixels by the back-projection Jacobians. Its problem:
VisualGeometryToolkitEstimation.BearingPoseProblem — Type
BearingPoseProblem(cs; sampler = nothing)The absolute-pose problem on unit BEARINGS: cs pairs world points with the camera-frame unit bearings observing them, X => f (a Correspondences, a StructArray or a Pair vector), for a camera the caller back-projects itself. The model is the camera extrinsics (world → camera, x_c = R·X + t, the Extrinsics3 of AbsolutePoseProblem); the residual is the tangent- plane logarithm map of the sphere at f of the predicted bearing normalize(R·X + t − o), its norm the angle between the rays.
Optional attribute COLUMNS of cs:
origins— each ray's origin in the camera frame: a GENERALIZED (non-central) camera. Such a problem is refine-only: there is no generalized minimal solver, sosolverefuses it.M— each bearing's back-projection Jacobian (3 × 2, frombackproject_with_jacobian(camera, u)), tangent (fᵀM = 0, checked): the residuals are whitened into PIXELS,(BᵀM)⁻¹·r; without it they are radians.weights— fixed per-correspondence weights (finite, non-negative) on the refit rows; the scoring residual is unweighted.
A bearing of norm ≠ 1 (a z = 1 chart point) is an ArgumentError, as is a non-finite point, origin or bearing. The public estimator is the bearing form of refine_absolute_pose.
VisualGeometryToolkitEstimation.BearingPoseContext — Type
BearingPoseContextThe data of a BearingPoseProblem beside its correspondences: per correspondence the ray origin in the camera frame (zero for a central camera), the observation's tangent basis, the whitening (I₂, or (BᵀM)⁻¹ into pixels) and the caller's fixed √weight.
The problem's test of its M column is public: a caller that whitens bearing residuals by a camera's Jacobian asks it before building a row.
VisualGeometryToolkitEstimation.pixel_whitening_defect — Function
pixel_whitening_defect(f, M) -> Union{Nothing, Symbol}Why the back-projection Jacobian M = ∂f/∂u (3 × 2, backproject_with_jacobian) cannot whiten the tangent residual at the unit bearing f into the pixels it came from, or nothing when it can. The defects, in the order they are tested:
:nonfinite: an entry ofMis not finite (the camera does not back-project the pixel smoothly there);:not_tangent: a column ofMis not tangent tof(|fᵀM[:, k]| > 10⁻⁶ ‖M[:, k]‖): it is not this bearing's Jacobian. A swapped or untangent Jacobian moves the residual little and its descent direction a lot (bearings.md);:singular:BᵀMis not invertible on the tangent plane (Bthe tangent basis off,s2_tangent_basis_matrix), so no pixel step moves the bearing along some tangent direction.
It is the test of BearingPoseProblem's M column, which refuses a defective one by it, and the rotation-centre refine's (refine_camera_and_poses on RotationCenterViews), which leaves such a dot out and counts it. A caller that whitens bearing residuals by the camera's Jacobian asks it first:
f, M = backproject_with_jacobian(camera, u)
pixel_whitening_defect(normalize(f), M) === nothing || continue # a dot this camera cannot whitenRotation from direction pairs
fit_rotation is the robust rotation R with b ≈ R·a from direction pairs with gross outliers: VisualGeometryToolkitCore's kabsch, reweighted by a stack loss shape at ONE width, efficiency_constant × scale from the caller's scale (or its width), never re-estimated.
VisualGeometryToolkitEstimation.fit_rotation — Function
fit_rotation(cs; optimization_strategy, scale | width, tol)
-> FitResult{RotMatrix{3, Float64}, FitMetadata}The robust rotation R with bᵢ ≈ R·aᵢ from direction pairs cs (aᵢ => bᵢ, 3-vectors; normalized at entry, so any nonzero length is a direction), with a fraction of the pairs gross outliers. A fit — it seeds itself with VisualGeometryToolkitCore's kabsch — iterating a reweighted Kabsch at ONE loss width: residual angles rᵢ = ∠(R·aᵢ, bᵢ), the weights of the strategy's loss at that width, and the weighted Kabsch. The width is fixed by the caller's scale input and never re-estimated.
optimization_strategy— the robust weighting, anIrls{K}strategy (its lossK, its efficiency constant and itsmax_iter; default configs/default.toml[fit_rotation] optimization_strategy, Huber), ornothingfor the plain (prior-weighted) Kabsch.scale|width— with anIrlsstrategy, EXACTLY ONE of:scale = σ(the residual angles' noise scale per tangent component, radians),scale = result(an upstream robust result'snoise_scale), orwidth = w(the loss width, radians). The width isw, orc · σwithcthe strategy's efficiency constant, by default the shape's at codimension 2 (efficiency_constant(K, 2): Huber 1.5011); a shape with none (Irls{TLSShape}()) needs the strategy's or awidth. Neither, both, or one given withoptimization_strategy = nothingis anArgumentError.tol— converged when an iteration movesRby less thantolradians (default configs/default.toml [fit_rotation]).- A
weightscolumn oncs(withattrs(cs; weights)) multiplies every pair's loss weight (fixed prior weights).
retcode_symbol(result) is :converged, :max_iterations, :solved (least squares), or :degenerate when fewer than two pairs keep a positive weight. n is the support (the pairs of positive solve weight), rms the weight-weighted root mean square residual angle. The metadata (FitMetadata) carries the weights, the residual angles, the scale record and the loss width.
Accuracy: with a minority of gross outliers and the inliers' scale given, the estimate reaches the noise floor of the inliers' own least-squares rotation (≈ σ/√m for m inliers at direction noise σ); the suite pins 20% random outliers among 200 pairs at σ = 1e-3 rad to within 5e-4 rad, where the plain Kabsch is off by over 1e-2.
Its metadata is the reweighted fits' FitMetadata.
fit_rotation_averaging is the absolute rotations of a set of frames from the relative rotations of a view graph, some of them wrong: VisualGeometryToolkitCore's spectral seed, then a reweighted first-order Lie-algebra least squares over every edge (Chatterjee & Govindu), each a sparse linear solve, at one fixed loss width.
VisualGeometryToolkitEstimation.fit_rotation_averaging — Function
fit_rotation_averaging(relative; optimization_strategy, scale | width, tol, seed = :spectral)
fit_rotation_averaging(vg::ViewGraph; kwargs...)
-> FitResult{Dict{K, RotMatrix{3, Float64}}, RotationAveragingMetadata{K}}Robust rotation averaging on a view graph: the absolute rotations R_k of the frames (d = R_k·f_k, every frame's directions into the common frame, the first of the sorted frames the identity — the gauge) from the relative rotations relative, an AbstractDict keyed by frame pairs (i, j) (or the edges of a ViewGraph) whose value R_ij takes frame i's directions into frame j's (f_j = R_ij·f_i, so R_i = R_j·R_ij), some of them wrong. Each value is the rotation, or VisualGeometryToolkitCore's Attributed(R_ij, (; weight)) with a fixed non-negative edge weight (an inlier count); a bare rotation weighs 1. A fit: it seeds itself (below) and iterates a reweighted first-order Lie-algebra least squares over every edge (Chatterjee & Govindu 2013), each a sparse linear solve, at ONE loss width fixed by the caller's scale input.
- Every edge may instead carry its rotation's ABSOLUTE covariance (radians²):
Attributed(Uncertain(R_ij, CovMat(Σ)), (; weight))(or a bareUncertain(R_ij, CovMat(Σ))), Σ 3×3 in the so(3) chart ofR_ij(left exp,R_ij → exp(δ)·R_ij) or 9×9 overvec(R_ij).uncertain_rotationmakes one from a pairwise rotation fit:noise_scale² × solver_covariance, the detections' own covariances included when its pixels areUncertain. Each edge residual is whitened by its covariance (VisualGeometryToolkitCore'swhitening_factorofR_j·Σ·R_jᵀ, the edge's chart transported to the residual's, refreshed withR_j) before the loss and in the update, so a weak edge (few inliers, noisy detections) weighs less. The covariances carry the scale, so such a graph takes NOscale(the result records scale 1,ScaleUpstream); awidthis in whitened units. Edges that share a detection are treated as independent (their correlation ignored: the first-order approximation). A graph mixing Uncertain and plain edges is anArgumentError. - The edge weights: a zero weight drops its edge. The edges of positive weight must connect every frame (an
ArgumentErrorotherwise — a zero-weight bridge splits the graph), and all-zero weights are refused. optimization_strategy— the robust weighting, anIrls{K}strategy (its loss, its efficiency constant and itsmax_iter; default configs/default.toml[fit_rotation_averaging] optimization_strategy, Huber), ornothingfor the plain least squares: the same Lie-algebra iterations from the seed at unit loss weights (no robust loss), at most[fit_rotation_averaging] max_iter. The seed alone is VisualGeometryToolkitCore'sspectral_rotation_averaging.scale|width— with anIrlsstrategy and plain edges, EXACTLY ONE of:scale = σ(the edges' residual rotation noise scale per axis component, radians),scale = result(an upstream robust result'snoise_scale, e.g. the pairwisefind_rotationresults' scale), orwidth = w(the loss width, radians). The width isw, orc · σwithcthe strategy's efficiency constant, by default the shape's at codimension 3 (efficiency_constant(K, 3): Huber 1.628). Neither, both, or one given withoptimization_strategy = nothingis anArgumentError.tol— converged when an iteration moves every frame by less thantolradians (configs/default.toml [fitrotationaveraging]).seed— where the iteration starts::spectral(the default, VisualGeometryToolkitCore'sspectral_rotation_averagingover every edge),:spanning_tree(the rotations chained along the maximum spanning tree of the edges of positive weight, by weight: Theia's and OpenCV's initialization, seechain), or anAbstractDictframe => rotation given by the caller. Every seed is re-gauged so the first sorted frame is the identity.
When the loss's weights drop every edge of a frame (a redescending loss past its cutoff), that frame is held at its current rotation for the iteration rather than left free in a singular solve; more generally, in each part of the graph that the edges of positive weight no longer tie to the gauge frame, the first sorted frame is held and the rest move with it.
retcode_symbol(result) is :converged or :max_iterations; n the support (the edges of positive solve weight) and rms the weight-weighted root mean square residual angle over them. A disconnected graph is an ArgumentError, counting only the edges of positive weight; the ViewGraph form also refuses a view in no edge (require_connected).
VisualGeometryToolkitEstimation.RotationAveragingMetadata — Type
RotationAveragingMetadata{K, L, W}FitResult metadata of fit_rotation_averaging at the returned rotations (L the optimization_strategy's type, Nothing for least squares):
edges::Vector{Tuple{K, K}}— the edges, sorted; the order ofweights.weights::W— per edge, the loss weightρ′(r²)normalized to a maximum of 1 (IrlsWeights; a lazy all-ones vector,BinaryWeights, foroptimization_strategy = nothing). The solve's weight of an edge is its own weight times this.residuals::Dict{Tuple{K, K}, Float64}— per edge(i, j), the angle (radians) of the residual rotationR_j·R_ij·R_iᵀ.scale,scale_source— the scale record (radians;noise_scale(result)),NaNandScaleNoneforoptimization_strategy = nothingor awidth-only fit.loss_width— radians,NaNforoptimization_strategy = nothing.