Pose and Rotation from Directions

Estimators whose data are DIRECTIONS — unit bearings from a camera's back-projection, star or gravity directions — rather than pixels. They work on the sphere throughout (VisualGeometryToolkitCore conventions/bearings.md), so rays at and past 90° from the optical axis, as a fisheye or omnidirectional camera sees them, are ordinary data. (A camera pose from a guess and pixels is refine_absolute_pose(pose, camera, pairs), on the PnP problem's pixel rows — P3P and Absolute Pose.)

Camera pose from bearings

refine_absolute_pose(pose, cs) with cs world points paired with UNIT bearings is the pose of a camera the caller back-projects itself — a grid or non-central lens the stack holds no CameraModel of — on the tangent-plane residual of the sphere, optionally from per-ray origins (a generalized camera) and whitened into pixels by the back-projection Jacobians. Its problem:

VisualGeometryToolkitEstimation.BearingPoseProblem — Type
BearingPoseProblem(cs; sampler = nothing)

The absolute-pose problem on unit BEARINGS: cs pairs world points with the camera-frame unit bearings observing them, X => f (a Correspondences, a StructArray or a Pair vector), for a camera the caller back-projects itself. The model is the camera extrinsics (world → camera, x_c = R·X + t, the Extrinsics3 of AbsolutePoseProblem); the residual is the tangent- plane logarithm map of the sphere at f of the predicted bearing normalize(R·X + t − o), its norm the angle between the rays.

Optional attribute COLUMNS of cs:

  • origins — each ray's origin in the camera frame: a GENERALIZED (non-central) camera. Such a problem is refine-only: there is no generalized minimal solver, so solve refuses it.
  • M — each bearing's back-projection Jacobian (3 × 2, from backproject_with_jacobian(camera, u)), tangent (fᵀM = 0, checked): the residuals are whitened into PIXELS, (BᵀM)⁻¹·r; without it they are radians.
  • weights — fixed per-correspondence weights (finite, non-negative) on the refit rows; the scoring residual is unweighted.

A bearing of norm ≠ 1 (a z = 1 chart point) is an ArgumentError, as is a non-finite point, origin or bearing. The public estimator is the bearing form of refine_absolute_pose.

source

The problem's test of its M column is public: a caller that whitens bearing residuals by a camera's Jacobian asks it before building a row.

VisualGeometryToolkitEstimation.pixel_whitening_defect — Function
pixel_whitening_defect(f, M) -> Union{Nothing, Symbol}

Why the back-projection Jacobian M = ∂f/∂u (3 × 2, backproject_with_jacobian) cannot whiten the tangent residual at the unit bearing f into the pixels it came from, or nothing when it can. The defects, in the order they are tested:

  • :nonfinite: an entry of M is not finite (the camera does not back-project the pixel smoothly there);
  • :not_tangent: a column of M is not tangent to f (|fᵀM[:, k]| > 10⁻⁶ ‖M[:, k]‖): it is not this bearing's Jacobian. A swapped or untangent Jacobian moves the residual little and its descent direction a lot (bearings.md);
  • :singular: BᵀM is not invertible on the tangent plane (B the tangent basis of f, s2_tangent_basis_matrix), so no pixel step moves the bearing along some tangent direction.

It is the test of BearingPoseProblem's M column, which refuses a defective one by it, and the rotation-centre refine's (refine_camera_and_poses on RotationCenterViews), which leaves such a dot out and counts it. A caller that whitens bearing residuals by the camera's Jacobian asks it first:

f, M = backproject_with_jacobian(camera, u)
pixel_whitening_defect(normalize(f), M) === nothing || continue   # a dot this camera cannot whiten
source

Rotation from direction pairs

fit_rotation is the robust rotation R with b ≈ R·a from direction pairs with gross outliers: VisualGeometryToolkitCore's kabsch, reweighted by a stack loss shape at ONE width, efficiency_constant × scale from the caller's scale (or its width), never re-estimated.

VisualGeometryToolkitEstimation.fit_rotation — Function
fit_rotation(cs; optimization_strategy, scale | width, tol)
    -> FitResult{RotMatrix{3, Float64}, FitMetadata}

The robust rotation R with bᵢ ≈ R·aᵢ from direction pairs cs (aᵢ => bᵢ, 3-vectors; normalized at entry, so any nonzero length is a direction), with a fraction of the pairs gross outliers. A fit — it seeds itself with VisualGeometryToolkitCore's kabsch — iterating a reweighted Kabsch at ONE loss width: residual angles rᵢ = ∠(R·aᵢ, bᵢ), the weights of the strategy's loss at that width, and the weighted Kabsch. The width is fixed by the caller's scale input and never re-estimated.

  • optimization_strategy — the robust weighting, an Irls{K} strategy (its loss K, its efficiency constant and its max_iter; default configs/default.toml [fit_rotation] optimization_strategy, Huber), or nothing for the plain (prior-weighted) Kabsch.
  • scale | width — with an Irls strategy, EXACTLY ONE of: scale = σ (the residual angles' noise scale per tangent component, radians), scale = result (an upstream robust result's noise_scale), or width = w (the loss width, radians). The width is w, or c · σ with c the strategy's efficiency constant, by default the shape's at codimension 2 (efficiency_constant(K, 2): Huber 1.5011); a shape with none (Irls{TLSShape}()) needs the strategy's or a width. Neither, both, or one given with optimization_strategy = nothing is an ArgumentError.
  • tol — converged when an iteration moves R by less than tol radians (default configs/default.toml [fit_rotation]).
  • A weights column on cs (withattrs(cs; weights)) multiplies every pair's loss weight (fixed prior weights).

retcode_symbol(result) is :converged, :max_iterations, :solved (least squares), or :degenerate when fewer than two pairs keep a positive weight. n is the support (the pairs of positive solve weight), rms the weight-weighted root mean square residual angle. The metadata (FitMetadata) carries the weights, the residual angles, the scale record and the loss width.

Accuracy: with a minority of gross outliers and the inliers' scale given, the estimate reaches the noise floor of the inliers' own least-squares rotation (≈ σ/√m for m inliers at direction noise σ); the suite pins 20% random outliers among 200 pairs at σ = 1e-3 rad to within 5e-4 rad, where the plain Kabsch is off by over 1e-2.

source

Its metadata is the reweighted fits' FitMetadata.

fit_rotation_averaging is the absolute rotations of a set of frames from the relative rotations of a view graph, some of them wrong: VisualGeometryToolkitCore's spectral seed, then a reweighted first-order Lie-algebra least squares over every edge (Chatterjee & Govindu), each a sparse linear solve, at one fixed loss width.

VisualGeometryToolkitEstimation.fit_rotation_averaging — Function
fit_rotation_averaging(relative; optimization_strategy, scale | width, tol, seed = :spectral)
fit_rotation_averaging(vg::ViewGraph; kwargs...)
    -> FitResult{Dict{K, RotMatrix{3, Float64}}, RotationAveragingMetadata{K}}

Robust rotation averaging on a view graph: the absolute rotations R_k of the frames (d = R_k·f_k, every frame's directions into the common frame, the first of the sorted frames the identity — the gauge) from the relative rotations relative, an AbstractDict keyed by frame pairs (i, j) (or the edges of a ViewGraph) whose value R_ij takes frame i's directions into frame j's (f_j = R_ij·f_i, so R_i = R_j·R_ij), some of them wrong. Each value is the rotation, or VisualGeometryToolkitCore's Attributed(R_ij, (; weight)) with a fixed non-negative edge weight (an inlier count); a bare rotation weighs 1. A fit: it seeds itself (below) and iterates a reweighted first-order Lie-algebra least squares over every edge (Chatterjee & Govindu 2013), each a sparse linear solve, at ONE loss width fixed by the caller's scale input.

  • Every edge may instead carry its rotation's ABSOLUTE covariance (radians²): Attributed(Uncertain(R_ij, CovMat(Σ)), (; weight)) (or a bare Uncertain(R_ij, CovMat(Σ))), Σ 3×3 in the so(3) chart of R_ij (left exp, R_ij → exp(δ)·R_ij) or 9×9 over vec(R_ij). uncertain_rotation makes one from a pairwise rotation fit: noise_scale² × solver_covariance, the detections' own covariances included when its pixels are Uncertain. Each edge residual is whitened by its covariance (VisualGeometryToolkitCore's whitening_factor of R_j·Σ·R_jᵀ, the edge's chart transported to the residual's, refreshed with R_j) before the loss and in the update, so a weak edge (few inliers, noisy detections) weighs less. The covariances carry the scale, so such a graph takes NO scale (the result records scale 1, ScaleUpstream); a width is in whitened units. Edges that share a detection are treated as independent (their correlation ignored: the first-order approximation). A graph mixing Uncertain and plain edges is an ArgumentError.
  • The edge weights: a zero weight drops its edge. The edges of positive weight must connect every frame (an ArgumentError otherwise — a zero-weight bridge splits the graph), and all-zero weights are refused.
  • optimization_strategy — the robust weighting, an Irls{K} strategy (its loss, its efficiency constant and its max_iter; default configs/default.toml [fit_rotation_averaging] optimization_strategy, Huber), or nothing for the plain least squares: the same Lie-algebra iterations from the seed at unit loss weights (no robust loss), at most [fit_rotation_averaging] max_iter. The seed alone is VisualGeometryToolkitCore's spectral_rotation_averaging.
  • scale | width — with an Irls strategy and plain edges, EXACTLY ONE of: scale = σ (the edges' residual rotation noise scale per axis component, radians), scale = result (an upstream robust result's noise_scale, e.g. the pairwise find_rotation results' scale), or width = w (the loss width, radians). The width is w, or c · σ with c the strategy's efficiency constant, by default the shape's at codimension 3 (efficiency_constant(K, 3): Huber 1.628). Neither, both, or one given with optimization_strategy = nothing is an ArgumentError.
  • tol — converged when an iteration moves every frame by less than tol radians (configs/default.toml [fitrotationaveraging]).
  • seed — where the iteration starts: :spectral (the default, VisualGeometryToolkitCore's spectral_rotation_averaging over every edge), :spanning_tree (the rotations chained along the maximum spanning tree of the edges of positive weight, by weight: Theia's and OpenCV's initialization, see chain), or an AbstractDict frame => rotation given by the caller. Every seed is re-gauged so the first sorted frame is the identity.

When the loss's weights drop every edge of a frame (a redescending loss past its cutoff), that frame is held at its current rotation for the iteration rather than left free in a singular solve; more generally, in each part of the graph that the edges of positive weight no longer tie to the gauge frame, the first sorted frame is held and the rest move with it.

retcode_symbol(result) is :converged or :max_iterations; n the support (the edges of positive solve weight) and rms the weight-weighted root mean square residual angle over them. A disconnected graph is an ArgumentError, counting only the edges of positive weight; the ViewGraph form also refuses a view in no edge (require_connected).

source
VisualGeometryToolkitEstimation.RotationAveragingMetadata — Type
RotationAveragingMetadata{K, L, W}

FitResult metadata of fit_rotation_averaging at the returned rotations (L the optimization_strategy's type, Nothing for least squares):

  • edges::Vector{Tuple{K, K}} — the edges, sorted; the order of weights.
  • weights::W — per edge, the loss weight ρ′(r²) normalized to a maximum of 1 (IrlsWeights; a lazy all-ones vector, BinaryWeights, for optimization_strategy = nothing). The solve's weight of an edge is its own weight times this.
  • residuals::Dict{Tuple{K, K}, Float64} — per edge (i, j), the angle (radians) of the residual rotation R_j·R_ij·R_iᵀ.
  • scale, scale_source — the scale record (radians; noise_scale(result)), NaN and ScaleNone for optimization_strategy = nothing or a width-only fit.
  • loss_width — radians, NaN for optimization_strategy = nothing.
source